├── .gitignore ├── .gitbook.yaml ├── docs ├── assets │ ├── dashboard.png │ └── sourced-community-edition.png ├── quickstart │ ├── README.md │ ├── 2-install-sourced.md │ ├── 1-install-requirements.md │ ├── 3-init-sourced.md │ └── 4-explore-sourced.md ├── usage │ ├── README.md │ ├── bblfsh.md │ ├── multiple-datasets.md │ ├── examples.md │ └── commands.md ├── README.md ├── CONTRIBUTING.md └── learn-more │ ├── architecture.md │ ├── faq.md │ └── troubleshooting.md ├── MAINTAINERS ├── .github ├── pull_request_template.md └── CODE_OF_CONDUCT.md ├── cmd └── sourced │ ├── format │ └── colors.go │ ├── cmd │ ├── stop.go │ ├── start.go │ ├── restart.go │ ├── logs.go │ ├── sql.go │ ├── prune.go │ ├── root.go │ ├── compose.go │ ├── status.go │ ├── init.go │ └── web.go │ ├── main.go │ ├── release │ ├── release.go │ └── release_test.go │ ├── compose │ ├── workdir │ │ ├── workdir_test.go │ │ ├── factory_test.go │ │ ├── env_file_test.go │ │ ├── workdir.go │ │ ├── handler.go │ │ ├── handler_test.go │ │ └── factory.go │ ├── file │ │ └── file.go │ └── compose.go │ └── dir │ ├── dir_test.go │ └── dir.go ├── run-integration-tests.bat ├── NOTICE.md ├── test ├── commander.go ├── compose_test.go ├── common.go ├── init_local_test.go ├── superset.go └── init_orgs_test.go ├── .travis.yml ├── Makefile ├── DCO ├── go.mod ├── docker-compose.yml ├── README.md ├── CHANGELOG.md └── go.sum /.gitignore: -------------------------------------------------------------------------------- 1 | .ci/ 2 | build/ 3 | -------------------------------------------------------------------------------- /.gitbook.yaml: -------------------------------------------------------------------------------- 1 | root: ./docs 2 | 3 | structure: 4 | readme: ../README.md 5 | summary: README.md 6 | -------------------------------------------------------------------------------- /docs/assets/dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/src-d/sourced-ce/HEAD/docs/assets/dashboard.png -------------------------------------------------------------------------------- /MAINTAINERS: -------------------------------------------------------------------------------- 1 | David Pordomingo (@dpordomingo) 2 | Lou Marvin Caraig (@se7entyse7en) 3 | -------------------------------------------------------------------------------- /docs/assets/sourced-community-edition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/src-d/sourced-ce/HEAD/docs/assets/sourced-community-edition.png -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | --- 5 | 6 | 7 | 8 | * [ ] I have updated the CHANGELOG file according to the conventions in [keepachangelog.com](https://keepachangelog.com) 9 | * [ ] This PR contains changes that do not require a mention in the CHANGELOG file -------------------------------------------------------------------------------- /cmd/sourced/format/colors.go: -------------------------------------------------------------------------------- 1 | package format 2 | 3 | import ( 4 | "fmt" 5 | "runtime" 6 | ) 7 | 8 | // Color represents a color code 9 | type Color string 10 | 11 | const ( 12 | // Red for errors 13 | Red Color = "31" 14 | // Yellow for warnings 15 | Yellow Color = "33" 16 | ) 17 | 18 | // Colorize returns the passed string with the passed color 19 | func Colorize(color Color, s string) string { 20 | if runtime.GOOS == "windows" { 21 | return s 22 | } 23 | 24 | return fmt.Sprintf("\x1b[%sm%s\x1b[0m", color, s) 25 | } 26 | -------------------------------------------------------------------------------- /cmd/sourced/cmd/stop.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "context" 5 | 6 | "github.com/src-d/sourced-ce/cmd/sourced/compose" 7 | ) 8 | 9 | type stopCmd struct { 10 | Command `name:"stop" short-description:"Stop any running components" long-description:"Stop any running components without removing them.\nThey can be started again with 'start'."` 11 | } 12 | 13 | func (c *stopCmd) Execute(args []string) error { 14 | return compose.Run(context.Background(), "stop") 15 | } 16 | 17 | func init() { 18 | rootCmd.AddCommand(&stopCmd{}) 19 | } 20 | -------------------------------------------------------------------------------- /docs/quickstart/README.md: -------------------------------------------------------------------------------- 1 | # Quickstart 2 | 3 | This guide covers the full setup journey, from zero to populated dashboard with **source{d} Community Edition**. 4 | 5 | This process is divided into the following steps: 6 | 7 | 1. [Install **source{d} CE** Dependencies](./1-install-requirements.md) 8 | 1. [Install **source{d} CE**](./2-install-sourced.md) 9 | 1. [Initialize the Dataset](./3-init-sourced.md): 10 | - using local git data 11 | - using git repositories from your GitHub org 12 | 1. [Explore Your Dataset](./4-explore-sourced.md) 13 | -------------------------------------------------------------------------------- /cmd/sourced/cmd/start.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "context" 5 | "time" 6 | 7 | "github.com/src-d/sourced-ce/cmd/sourced/compose" 8 | ) 9 | 10 | type startCmd struct { 11 | Command `name:"start" short-description:"Start any stopped components" long-description:"Start any stopped components.\nThe containers must be initialized before with 'init'."` 12 | } 13 | 14 | func (c *startCmd) Execute(args []string) error { 15 | if err := compose.Run(context.Background(), "start"); err != nil { 16 | return err 17 | } 18 | 19 | return OpenUI(30 * time.Minute) 20 | 21 | } 22 | 23 | func init() { 24 | rootCmd.AddCommand(&startCmd{}) 25 | } 26 | -------------------------------------------------------------------------------- /run-integration-tests.bat: -------------------------------------------------------------------------------- 1 | :: we can't use makefile for windows because it depends on CI makefile which depends on shell 2 | 3 | :: compile sourced-ce 4 | go build -tags=forceposix -o build/sourced-ce_windows_amd64/sourced.exe ./cmd/sourced 5 | 6 | :: run tests 7 | set TMPDIR_INTEGRATION_TEST=C:\tmp 8 | :: see https://stackoverflow.com/questions/22948189/batch-getting-the-directory-is-not-empty-on-rmdir-command 9 | del /f /s /q %TMPDIR_INTEGRATION_TEST% 1>nul 10 | rmdir /s /q %TMPDIR_INTEGRATION_TEST% 11 | mkdir %TMPDIR_INTEGRATION_TEST% 12 | set TMP=%TMPDIR_INTEGRATION_TEST% 13 | go test -timeout 20m -parallel 1 -count 1 -tags="forceposix integration" github.com/src-d/sourced-ce/test/ -v 14 | -------------------------------------------------------------------------------- /cmd/sourced/cmd/restart.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "context" 5 | 6 | "github.com/src-d/sourced-ce/cmd/sourced/compose" 7 | ) 8 | 9 | type restartCmd struct { 10 | Command `name:"restart" short-description:"Update current installation according to the active docker compose file" long-description:"Update current installation according to the active docker compose file. It only recreates the component containers, keeping all your data, as charts, dashboards, repositories and GitHub metadata."` 11 | } 12 | 13 | func (c *restartCmd) Execute(args []string) error { 14 | return compose.Run(context.Background(), "up", "--force-recreate", "--detach") 15 | } 16 | 17 | func init() { 18 | rootCmd.AddCommand(&restartCmd{}) 19 | } 20 | -------------------------------------------------------------------------------- /NOTICE.md: -------------------------------------------------------------------------------- 1 | sourced-ce is the data platform for your software development life cycle 2 | 3 | Copyright (C) 2019 source{d} 4 | 5 | This program is free software: you can redistribute it and/or modify 6 | it under the terms of the GNU General Public License as published by 7 | the Free Software Foundation, either version 3 of the License, or 8 | (at your option) any later version. 9 | 10 | This program is distributed in the hope that it will be useful, 11 | but WITHOUT ANY WARRANTY; without even the implied warranty of 12 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 13 | GNU General Public License for more details. 14 | 15 | You should have received a copy of the GNU General Public License 16 | along with this program. If not, see . 17 | -------------------------------------------------------------------------------- /cmd/sourced/cmd/logs.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "context" 5 | 6 | "github.com/src-d/sourced-ce/cmd/sourced/compose" 7 | ) 8 | 9 | type logsCmd struct { 10 | Command `name:"logs" short-description:"Fetch the logs of source{d} components" long-description:"Fetch the logs of source{d} components"` 11 | 12 | Follow bool `short:"f" long:"follow" description:"Follow log output"` 13 | Args struct { 14 | Components []string `positional-arg-name:"component" description:"Component names from where to fetch logs"` 15 | } `positional-args:"yes"` 16 | } 17 | 18 | func (c *logsCmd) Execute(args []string) error { 19 | command := []string{"logs"} 20 | 21 | if c.Follow { 22 | command = append(command, "--follow") 23 | } 24 | 25 | if components := c.Args.Components; len(components) > 0 { 26 | command = append(command, components...) 27 | } 28 | 29 | return compose.Run(context.Background(), command...) 30 | } 31 | 32 | func init() { 33 | rootCmd.AddCommand(&logsCmd{}) 34 | } 35 | -------------------------------------------------------------------------------- /cmd/sourced/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | 6 | "github.com/src-d/sourced-ce/cmd/sourced/cmd" 7 | composefile "github.com/src-d/sourced-ce/cmd/sourced/compose/file" 8 | "github.com/src-d/sourced-ce/cmd/sourced/format" 9 | "github.com/src-d/sourced-ce/cmd/sourced/release" 10 | ) 11 | 12 | // this variable is rewritten during the CI build step 13 | var version = "master" 14 | var build = "dev" 15 | 16 | func main() { 17 | composefile.SetVersion(version) 18 | cmd.Init(version, build) 19 | 20 | checkUpdates() 21 | 22 | cmd.Execute() 23 | } 24 | 25 | func checkUpdates() { 26 | if version == "master" { 27 | return 28 | } 29 | 30 | update, latest, err := release.FindUpdates(version) 31 | if err != nil { 32 | return 33 | } 34 | 35 | if update { 36 | s := fmt.Sprintf( 37 | `There is a newer version. Current version: %s, latest version: %s 38 | Please go to https://github.com/src-d/sourced-ce/releases/latest to upgrade. 39 | `, version, latest) 40 | 41 | fmt.Println(format.Colorize(format.Yellow, s)) 42 | } 43 | } 44 | -------------------------------------------------------------------------------- /cmd/sourced/cmd/sql.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "context" 5 | "os" 6 | 7 | "github.com/src-d/sourced-ce/cmd/sourced/compose" 8 | 9 | "golang.org/x/crypto/ssh/terminal" 10 | ) 11 | 12 | type sqlCmd struct { 13 | Command `name:"sql" short-description:"Open a MySQL client connected to a SQL interface for Git" long-description:"Open a MySQL client connected to a SQL interface for Git"` 14 | 15 | Args struct { 16 | Query string `positional-arg-name:"query" description:"SQL query to be run by the SQL interface for Git"` 17 | } `positional-args:"yes"` 18 | } 19 | 20 | func (c *sqlCmd) Execute(args []string) error { 21 | command := []string{"exec"} 22 | if !terminal.IsTerminal(int(os.Stdout.Fd())) || !terminal.IsTerminal(int(os.Stdin.Fd())) { 23 | command = append(command, "-T") 24 | } 25 | command = append(command, "gitbase", "mysql") 26 | if c.Args.Query != "" { 27 | command = append(command, "--execute", c.Args.Query) 28 | } 29 | 30 | return compose.Run(context.Background(), command...) 31 | } 32 | 33 | func init() { 34 | rootCmd.AddCommand(&sqlCmd{}) 35 | } 36 | -------------------------------------------------------------------------------- /test/commander.go: -------------------------------------------------------------------------------- 1 | // +build integration 2 | 3 | package test 4 | 5 | import ( 6 | "fmt" 7 | "os" 8 | "time" 9 | 10 | "gotest.tools/icmd" 11 | ) 12 | 13 | type Commander struct { 14 | bin string 15 | sourcedDir string 16 | } 17 | 18 | // RunCmd runs a command with the appropriate SOURCED_DIR and returns a Result 19 | func (s *Commander) RunCmd(args []string, cmdOperators ...icmd.CmdOp) *icmd.Result { 20 | c := icmd.Command(s.bin, args...) 21 | 22 | newEnv := append(os.Environ(), 23 | fmt.Sprintf("SOURCED_DIR=%s", s.sourcedDir)) 24 | op := append(cmdOperators, icmd.WithEnv(newEnv...)) 25 | 26 | return icmd.RunCmd(c, op...) 27 | } 28 | 29 | // RunCommand is a convenience wrapper for RunCmd that accepts variadic arguments. 30 | // It runs a command with the appropriate SOURCED_DIR and returns a Result 31 | func (s *Commander) RunCommand(args ...string) *icmd.Result { 32 | return s.RunCmd(args) 33 | } 34 | 35 | // RunCommandWithTimeout runs a command with the given timeout 36 | func (s *Commander) RunCommandWithTimeout(timeout time.Duration, args ...string) *icmd.Result { 37 | return s.RunCmd(args, icmd.WithTimeout(timeout)) 38 | } 39 | -------------------------------------------------------------------------------- /docs/usage/README.md: -------------------------------------------------------------------------------- 1 | # Usage of source{d} Community Edition 2 | 3 | Once you know how to install and run **source{d} Community Edition**, you will find in this section some useful resources for guiding your first steps using this tool. 4 | 5 | - [`sourced` Command Reference](./commands.md) 6 | - [Using Multiple Datasets](./multiple-datasets.md) will show you how to analyze different datasets, no matter if they are stored locally or in GitHub. 7 | - [Some SQL Examples to Explore Your Dataset](./examples.md) 8 | - [Babelfish UAST](./bblfsh.md), about how to extract code features and understand code structure in a language-agnostic way". 9 | 10 | If you are interested in the different components of **source{d} Community Edition**, you can read more about it in the [docs about architecture](../learn-more/architecture.md) 11 | 12 | Some common questions are answered in the [FAQ](../learn-more/faq.md), and common problems and how to solve them in the [Troubleshooting](../learn-more/troubleshooting.md) guide. If you have any question about source{d} you can ask in our [source{d} Forum](https://forum.sourced.tech). If you spotted a bug, or have a feature request, please [open an issue](https://github.com/src-d/sourced-ce/issues) to let us know about it. 13 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | branches: 2 | only: 3 | - master 4 | - /^v\d+\.\d+(\.\d+)?(-\S*)?$/ 5 | 6 | dist: xenial 7 | sudo: required 8 | 9 | language: go 10 | go_import_path: github.com/src-d/sourced-ce 11 | go: 12 | - 1.13.x 13 | env: 14 | global: 15 | - SOURCED_GITHUB_TOKEN=$GITHUB_TOKEN 16 | 17 | matrix: 18 | fast_finish: true 19 | 20 | services: 21 | - docker 22 | 23 | stages: 24 | - name: tests 25 | - name: release 26 | if: tag IS present 27 | 28 | jobs: 29 | include: 30 | - stage: tests 31 | name: 'Go Unit Tests' 32 | script: 33 | - make packages 34 | - make test-coverage codecov 35 | 36 | - stage: tests 37 | name: 'Integration Tests Linux' 38 | script: 39 | # cannot use 'make test-integration' because 'make clean' fails with 40 | # GO111MODULE, see https://github.com/golang/go/issues/31002 41 | - make build 42 | - make test-integration-no-build 43 | 44 | - stage: release 45 | name: 'Release to GitHub' 46 | script: 47 | - make packages 48 | deploy: 49 | provider: releases 50 | api_key: $GITHUB_TOKEN 51 | file_glob: true 52 | file: 53 | - build/*.tar.gz 54 | skip_cleanup: true 55 | on: 56 | all_branches: true 57 | -------------------------------------------------------------------------------- /docs/quickstart/2-install-sourced.md: -------------------------------------------------------------------------------- 1 | # Install source{d} Community Edition 2 | 3 | Download the **[latest release](https://github.com/src-d/sourced-ce/releases/latest)** for your Linux, macOS (Darwin) or Windows. 4 | 5 | ## On Linux or macOS 6 | 7 | Extract `sourced` binary from the release you downloaded, and move it into your bin folder to make it executable from any directory: 8 | 9 | ```bash 10 | $ tar -xvf path/to/sourced-ce_REPLACE-VERSION_REPLACE-OS_amd64.tar.gz 11 | $ sudo mv path/to/sourced-ce_REPLACE-OS_amd64/sourced /usr/local/bin/ 12 | ``` 13 | 14 | ## On Windows 15 | 16 | *Please note that from now on we assume that the commands are executed in `powershell` and not in `cmd`.* 17 | 18 | Create a directory for `sourced.exe` and add it to your `$PATH`, running these commands in a powershell as administrator: 19 | ```powershell 20 | mkdir 'C:\Program Files\sourced' 21 | # Add the directory to the `%path%` to make it available from anywhere 22 | setx /M PATH "$($env:path);C:\Program Files\sourced" 23 | # Now open a new powershell to apply the changes 24 | ``` 25 | 26 | Extract the `sourced.exe` executable from the release you downloaded, and copy it into the directory you created in the previous step: 27 | ```powershell 28 | mv \path\to\sourced-ce_windows_amd64\sourced.exe 'C:\Program Files\sourced' 29 | ``` 30 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Package configuration 2 | PROJECT = sourced-ce 3 | COMMANDS = cmd/sourced 4 | PKG_OS ?= darwin linux windows 5 | 6 | # Including ci Makefile 7 | CI_REPOSITORY ?= https://github.com/src-d/ci.git 8 | CI_PATH ?= $(shell pwd)/.ci 9 | CI_VERSION ?= v1 10 | 11 | MAKEFILE := $(CI_PATH)/Makefile.main 12 | $(MAKEFILE): 13 | git clone --quiet --branch $(CI_VERSION) --depth 1 $(CI_REPOSITORY) $(CI_PATH); 14 | 15 | -include $(MAKEFILE) 16 | 17 | GOTEST_BASE = go test -v -timeout 20m -parallel 1 -count 1 -ldflags "$(LD_FLAGS)" 18 | GOTEST_INTEGRATION = $(GOTEST_BASE) -tags="forceposix integration" 19 | 20 | OS := $(shell uname) 21 | 22 | # override clean target from CI to avoid executing `go clean` 23 | # see https://github.com/src-d/sourced-ce/pull/154 24 | clean: 25 | rm -rf $(BUILD_PATH) $(BIN_PATH) $(VENDOR_PATH) 26 | 27 | ifeq ($(OS),Darwin) 28 | test-integration-clean: 29 | $(eval TMPDIR_INTEGRATION_TEST := $(PWD)/integration-test-tmp) 30 | $(eval GOTEST_INTEGRATION := TMPDIR=$(TMPDIR_INTEGRATION_TEST) $(GOTEST_INTEGRATION)) 31 | rm -rf $(TMPDIR_INTEGRATION_TEST) 32 | mkdir $(TMPDIR_INTEGRATION_TEST) 33 | else 34 | test-integration-clean: 35 | endif 36 | 37 | test-integration-no-build: test-integration-clean 38 | $(GOTEST_INTEGRATION) github.com/src-d/sourced-ce/test/ 39 | 40 | test-integration: clean build test-integration-no-build 41 | -------------------------------------------------------------------------------- /docs/README.md: -------------------------------------------------------------------------------- 1 | # Table of contents 2 | 3 | * [Introduction](../README.md) 4 | * [Quickstart](./quickstart/README.md) 5 | * [Dependencies](./quickstart/1-install-requirements.md) 6 | * [Install **source{d} CE**](./quickstart/2-install-sourced.md) 7 | * [Run **source{d} CE**](./quickstart/3-init-sourced.md) 8 | * [Explore Your Data](./quickstart/4-explore-sourced.md) 9 | * [Next steps](./usage/README.md) 10 | 11 | ## Usage 12 | * [sourced Command Reference](./usage/commands.md) 13 | * [Multiple Datasets](./usage/multiple-datasets.md) 14 | * [SQL Examples](./usage/examples.md) 15 | * [Babelfish UAST](./usage/bblfsh.md) 16 | 17 | ## Learn More 18 | 19 | * [FAQ](./learn-more/faq.md) 20 | * [Troubleshooting](./learn-more/troubleshooting.md) 21 | * [Architecture](./learn-more/architecture.md) 22 | * [Contribute](./CONTRIBUTING.md) 23 | * [Changelog](../CHANGELOG.md) 24 | * [License](../LICENSE.md) 25 | 26 | ## Resources 27 | 28 | * [GitHub Repository](https://github.com/src-d/sourced-ce) 29 | * [Product Page](https://www.sourced.tech) 30 | * [Book a Demo](https://go.sourced.tech/community-demo) 31 | * [Get in Touch With Us](http://go.sourced.tech/contact) 32 | * [Join Us on Slack](https://sourced-community.slack.com/join/shared_invite/enQtMjc4Njk5MzEyNzM2LTFjNzY4NjEwZGEwMzRiNTM4MzRlMzQ4MmIzZjkwZmZlM2NjODUxZmJjNDI1OTcxNDAyMmZlNmFjODZlNTg0YWM) 33 | * [source{d} Forum](https://forum.sourced.tech) 34 | -------------------------------------------------------------------------------- /test/compose_test.go: -------------------------------------------------------------------------------- 1 | // +build integration 2 | 3 | package test 4 | 5 | import ( 6 | "testing" 7 | 8 | "github.com/stretchr/testify/suite" 9 | ) 10 | 11 | type ComposeTestSuite struct { 12 | IntegrationSuite 13 | } 14 | 15 | func TestComposeTestSuite(t *testing.T) { 16 | itt := ComposeTestSuite{} 17 | suite.Run(t, &itt) 18 | } 19 | 20 | func (s *ComposeTestSuite) TestListComposeFiles() { 21 | r := s.RunCommand("compose", "list") 22 | s.Contains(r.Stdout(), "[0]* local") 23 | } 24 | 25 | func (s *ComposeTestSuite) TestSetComposeFile() { 26 | r := s.RunCommand("compose", "set", "0") 27 | s.Contains(r.Stdout(), "Active docker compose file was changed successfully") 28 | 29 | r = s.RunCommand("compose", "list") 30 | s.Contains(r.Stdout(), "[0]* local") 31 | } 32 | 33 | func (s *ComposeTestSuite) TestSetComposeFilIndexOutOfRange() { 34 | r := s.RunCommand("compose", "set", "5") 35 | s.Contains(r.Stderr(), "Index is out of range, please check the output of 'sourced compose list'") 36 | } 37 | 38 | func (s *ComposeTestSuite) TestSetComposeNotFound() { 39 | r := s.RunCommand("compose", "set", "NotFound") 40 | s.Error(r.Error) 41 | } 42 | 43 | func (s *ComposeTestSuite) TestSetComposeFilesWithStringIndex() { 44 | r := s.RunCommand("compose", "set", "local") 45 | s.Contains(r.Stdout(), "Active docker compose file was changed successfully") 46 | 47 | r = s.RunCommand("compose", "list") 48 | s.Contains(r.Stdout(), "[0]* local") 49 | } 50 | -------------------------------------------------------------------------------- /cmd/sourced/release/release.go: -------------------------------------------------------------------------------- 1 | // Package release deals with versioning and releases 2 | package release 3 | 4 | import ( 5 | "context" 6 | "net/http" 7 | "os" 8 | "path/filepath" 9 | 10 | "github.com/src-d/sourced-ce/cmd/sourced/dir" 11 | 12 | "github.com/blang/semver" 13 | "github.com/google/go-github/v25/github" 14 | "github.com/gregjones/httpcache" 15 | "github.com/gregjones/httpcache/diskcache" 16 | ) 17 | 18 | // FindUpdates calls the GitHub API to check the latest release tag. It returns 19 | // true if the latest stable release is newer than the current tag, and also 20 | // that latest tag name. 21 | func FindUpdates(current string) (update bool, latest string, err error) { 22 | currentV, err := semver.ParseTolerant(current) 23 | if err != nil { 24 | return false, "", err 25 | } 26 | 27 | diskcachePath := filepath.Join(dir.TmpPath(), "httpcache") 28 | err = os.MkdirAll(diskcachePath, os.ModePerm) 29 | if err != nil { 30 | return false, "", err 31 | } 32 | 33 | cache := diskcache.New(diskcachePath) 34 | client := github.NewClient(&http.Client{Transport: httpcache.NewTransport(cache)}) 35 | 36 | rel, _, err := client.Repositories.GetLatestRelease(context.Background(), "src-d", "sourced-ce") 37 | if err != nil { 38 | return false, "", err 39 | } 40 | 41 | latestV, err := semver.ParseTolerant(rel.GetTagName()) 42 | if err != nil { 43 | return false, "", err 44 | } 45 | 46 | update = latestV.GT(currentV) 47 | latest = latestV.String() 48 | 49 | return update, latest, nil 50 | } 51 | -------------------------------------------------------------------------------- /DCO: -------------------------------------------------------------------------------- 1 | Developer Certificate of Origin 2 | Version 1.1 3 | 4 | Copyright (C) 2004, 2006 The Linux Foundation and its contributors. 5 | 660 York Street, Suite 102, 6 | San Francisco, CA 94110 USA 7 | 8 | Everyone is permitted to copy and distribute verbatim copies of this 9 | license document, but changing it is not allowed. 10 | 11 | 12 | Developer's Certificate of Origin 1.1 13 | 14 | By making a contribution to this project, I certify that: 15 | 16 | (a) The contribution was created in whole or in part by me and I 17 | have the right to submit it under the open source license 18 | indicated in the file; or 19 | 20 | (b) The contribution is based upon previous work that, to the best 21 | of my knowledge, is covered under an appropriate open source 22 | license and I have the right under that license to submit that 23 | work with modifications, whether created in whole or in part 24 | by me, under the same open source license (unless I am 25 | permitted to submit under a different license), as indicated 26 | in the file; or 27 | 28 | (c) The contribution was provided directly to me by some other 29 | person who certified (a), (b) or (c) and I have not modified 30 | it. 31 | 32 | (d) I understand and agree that this project and the contribution 33 | are public and that a record of the contribution (including all 34 | personal information I submit with it, including my sign-off) is 35 | maintained indefinitely and may be redistributed consistent with 36 | this project or the open source license(s) involved. 37 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/src-d/sourced-ce 2 | 3 | go 1.13 4 | 5 | // See https://github.com/gotestyourself/gotest.tools/issues/156 6 | // replace gotest.tools => gotest.tools v2.3.0 7 | replace gotest.tools => gotest.tools v0.0.0-20181223230014-1083505acf35 8 | 9 | require ( 10 | github.com/PuerkitoBio/goquery v1.5.0 11 | github.com/blang/semver v3.5.1+incompatible 12 | github.com/golang/protobuf v1.3.1 // indirect 13 | github.com/google/btree v1.0.0 // indirect 14 | github.com/google/go-github/v25 v25.1.1 15 | github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc 16 | github.com/jessevdk/go-flags v1.4.0 // indirect 17 | github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d // indirect 18 | github.com/lib/pq v1.1.1 19 | github.com/mattn/go-colorable v0.1.1 // indirect 20 | github.com/mattn/go-isatty v0.0.7 // indirect 21 | github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b // indirect 22 | github.com/onsi/ginkgo v1.8.0 // indirect 23 | github.com/onsi/gomega v1.5.0 // indirect 24 | github.com/pbnjay/memory v0.0.0-20190104145345-974d429e7ae4 25 | github.com/peterbourgon/diskv v2.0.1+incompatible // indirect 26 | github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4 27 | github.com/pkg/errors v0.8.1 28 | github.com/serenize/snaker v0.0.0-20171204205717-a683aaf2d516 29 | github.com/sirupsen/logrus v1.4.1 // indirect 30 | github.com/src-d/envconfig v1.0.0 // indirect 31 | github.com/stretchr/testify v1.3.0 32 | github.com/x-cray/logrus-prefixed-formatter v0.5.2 // indirect 33 | golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734 34 | golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c 35 | gopkg.in/src-d/go-cli.v0 v0.0.0-20190422143124-3a646154da79 36 | gopkg.in/src-d/go-errors.v1 v1.0.0 37 | gopkg.in/src-d/go-log.v1 v1.0.2 // indirect 38 | gopkg.in/yaml.v2 v2.2.2 // indirect 39 | gotest.tools v0.0.0-00010101000000-000000000000 40 | ) 41 | -------------------------------------------------------------------------------- /docs/usage/bblfsh.md: -------------------------------------------------------------------------------- 1 | # Babelfish UAST 2 | 3 | _In the [Babelfish documentation](https://docs.sourced.tech/babelfish/), you will 4 | find detailed information about Babelfish specifications, usage, examples, etc._ 5 | 6 | One of the most important components of **source{d} CE** is the UAST, which stands for: 7 | [Universal Abstract Syntax Tree](https://docs.sourced.tech/babelfish/uast/uast-specification-v2). 8 | 9 | UASTs are a normalized form of a programming language's AST, annotated with language-agnostic roles and transformed with language-agnostic concepts (e.g. Functions, Imports, etc.). 10 | 11 | These enable an advanced static analysis of code and easy feature extraction for statistics or [Machine Learning on Code](https://github.com/src-d/awesome-machine-learning-on-source-code). 12 | 13 | 14 | ## UAST Usage 15 | 16 | From the web interface, you can use the `UAST` tab, to parse files by direct input, or you can also get UASTs from the `SQL Lab` tab, using the `UAST(content)` [gitbase function](https://docs.sourced.tech/gitbase/using-gitbase/functions). 17 | 18 | For the whole syntax about how to query the UASTs, you can refer to [How To Query UASTs With Babelfish](https://docs.sourced.tech/babelfish/using-babelfish/uast-querying) 19 | 20 | 21 | ## Supported Languages 22 | 23 | To see which languages are available, check the table of [Babelfish supported languages](https://docs.sourced.tech/babelfish/languages). 24 | 25 | 26 | ## Clients and Connectors 27 | 28 | The language parsing server (Babelfish) is available from the web interface, but you can also connect to the parsing server, deployed by **source{d} CE**, with several language clients, currently supported and maintained: 29 | 30 | - [Babelfish Go Client](https://github.com/bblfsh/go-client) 31 | - [Babelfish Python Client](https://github.com/bblfsh/client-python) 32 | - [Babelfish Scala Client](https://github.com/bblfsh/client-scala) 33 | -------------------------------------------------------------------------------- /docs/quickstart/1-install-requirements.md: -------------------------------------------------------------------------------- 1 | # Install source{d} Community Edition Dependencies 2 | 3 | ## Install Docker 4 | 5 | _Please note that Docker Toolbox is not supported. In case that you're running Docker Toolbox, please consider updating to newer Docker Desktop for Mac or Docker Desktop for Windows._ 6 | 7 | _For Linux installations, using Docker installed from a snap package is not supported._ 8 | 9 | Follow the instructions based on your OS: 10 | 11 | - [Docker for Ubuntu Linux](https://docs.docker.com/install/linux/docker-ce/ubuntu/#install-docker-ce-1) 12 | - [Docker for Arch Linux](https://wiki.archlinux.org/index.php/Docker#Installation) 13 | - [Docker for macOS](https://store.docker.com/editions/community/docker-ce-desktop-mac) 14 | - [Docker Desktop for Windows](https://hub.docker.com/editions/community/docker-ce-desktop-windows) (Make sure to read the [system requirements for Docker on Windows](https://docs.docker.com/docker-for-windows/install/).) 15 | 16 | Minimal supported version 18.02.0. 17 | 18 | ## Docker Compose 19 | 20 | **source{d} CE** is deployed as Docker containers, using [Docker Compose](https://docs.docker.com/compose), but it is not required to have a local installation of Docker Compose; if it is not found it will be downloaded from docker sources, and deployed inside a container. 21 | 22 | If you prefer a local installation of Docker Compose, or you have no access to internet to download it, you can follow the [Docker Compose install guide](https://docs.docker.com/compose/install) 23 | 24 | Minimal supported version 1.20.0. 25 | 26 | ## Internet Connection 27 | 28 | source{d} CE automatically fetches some resources from the Internet when they are not found locally, so in order to use all source{d} CE capacities, Internet connection is needed. 29 | 30 | For more details, you can refer to [Why Do I Need Internet Connection?](../learn-more/faq.md#why-do-i-need-internet-connection) 31 | -------------------------------------------------------------------------------- /cmd/sourced/cmd/prune.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "context" 5 | 6 | "github.com/src-d/sourced-ce/cmd/sourced/compose" 7 | "github.com/src-d/sourced-ce/cmd/sourced/compose/workdir" 8 | ) 9 | 10 | type pruneCmd struct { 11 | Command `name:"prune" short-description:"Stop and remove components and resources" long-description:"Stops containers and removes containers, networks, volumes and configuration created by 'init' for the current working directory.\nTo delete resources for all working directories pass --all flag.\nImages are not deleted unless you specify the --images flag."` 12 | 13 | All bool `short:"a" long:"all" description:"Remove containers and resources for all working directories"` 14 | Images bool `long:"images" description:"Remove docker images"` 15 | } 16 | 17 | func (c *pruneCmd) Execute(args []string) error { 18 | workdirHandler, err := workdir.NewHandler() 19 | if err != nil { 20 | return err 21 | } 22 | 23 | if !c.All { 24 | return c.pruneActive(workdirHandler) 25 | } 26 | 27 | wds, err := workdirHandler.List() 28 | if err != nil { 29 | return err 30 | } 31 | 32 | for _, wd := range wds { 33 | if err := workdirHandler.SetActive(wd); err != nil { 34 | return err 35 | } 36 | 37 | if err = c.pruneActive(workdirHandler); err != nil { 38 | return err 39 | } 40 | } 41 | 42 | return nil 43 | } 44 | 45 | func (c *pruneCmd) pruneActive(workdirHandler *workdir.Handler) error { 46 | a := []string{"down", "--volumes"} 47 | if c.Images { 48 | a = append(a, "--rmi", "all") 49 | } 50 | 51 | if err := compose.Run(context.Background(), a...); err != nil { 52 | return err 53 | } 54 | 55 | wd, err := workdirHandler.Active() 56 | if err != nil { 57 | return err 58 | } 59 | 60 | if err := workdirHandler.Remove(wd); err != nil { 61 | return err 62 | } 63 | 64 | return workdirHandler.UnsetActive() 65 | } 66 | 67 | func init() { 68 | rootCmd.AddCommand(&pruneCmd{}) 69 | } 70 | -------------------------------------------------------------------------------- /docs/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contribution Guidelines 2 | 3 | As all source{d} projects, this project follows the 4 | [source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md). 5 | 6 | 7 | # Additional Contribution Guidelines 8 | 9 | In addition to the [source{d} Contributing Guidelines](https://github.com/src-d/guide/blob/master/engineering/documents/CONTRIBUTING.md), this project follows the following guidelines. 10 | 11 | 12 | ## Changelog 13 | 14 | This project lists the important changes between releases in the [`CHANGELOG.md`](../CHANGELOG.md) file. 15 | 16 | If you open a PR, you should also add a brief summary in the `CHANGELOG.md` mentioning the new feature, change or bugfix that you proposed. 17 | 18 | 19 | ## How To Restore Dashboards and Charts to Defaults 20 | 21 | The official way to restore **source{d} CE** to its initial state, is to remove the running components with 22 | `sourced prune --all`, and then init again with `sourced init`. 23 | 24 | In some circumstances you need to restore only the state modified from the UI (charts, dashboards, saved queries, users, 25 | roles, etcetera), using the default ones for the version of **source{d} CE** that you're currently using, and preserve 26 | the repositories and metadata fetched from GitHub organizations. 27 | 28 | To do so, you only need to delete the docker volume containing the PostgreSQL database, and restart **source{d} CE**. 29 | It can be done following these steps if you already have [Docker Compose](https://docs.docker.com/compose/) installed: 30 | 31 | ```shell 32 | $ cd ~/.sourced/workdirs/__active__ 33 | $ source .env 34 | $ docker-compose stop postgres 35 | $ docker-compose rm -f postgres 36 | $ ENV_PREFIX=`awk '{print tolower($0)}' <<< ${COMPOSE_PROJECT_NAME}` 37 | $ docker volume rm ${ENV_PREFIX}_postgres 38 | $ docker-compose up -d postgres 39 | $ docker-compose exec -u superset sourced-ui bash -c 'sleep 10s && python bootstrap.py' 40 | ``` 41 | -------------------------------------------------------------------------------- /cmd/sourced/release/release_test.go: -------------------------------------------------------------------------------- 1 | package release 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | "io/ioutil" 7 | "net/http" 8 | "os" 9 | "path/filepath" 10 | "testing" 11 | 12 | "github.com/src-d/sourced-ce/cmd/sourced/dir" 13 | "github.com/stretchr/testify/assert" 14 | ) 15 | 16 | type testCase struct { 17 | responseTag string // tag returned by github 18 | current string 19 | update bool 20 | latest string 21 | } 22 | 23 | func TestFindUpdatesSuccess(t *testing.T) { 24 | os.RemoveAll(filepath.Join(dir.TmpPath(), "httpcache")) 25 | 26 | cases := []testCase{ 27 | { 28 | responseTag: "v0.14.0", 29 | current: "v0.14.0", 30 | update: false, 31 | latest: "0.14.0", 32 | }, 33 | { 34 | responseTag: "v0.11.0", 35 | current: "v0.14.0", 36 | update: false, 37 | latest: "0.11.0", 38 | }, 39 | { 40 | responseTag: "v0.14.0", 41 | current: "v0.13.0", 42 | update: true, 43 | latest: "0.14.0", 44 | }, 45 | { 46 | responseTag: "v0.14.0", 47 | current: "v0.13.1", 48 | update: true, 49 | latest: "0.14.0", 50 | }, 51 | } 52 | 53 | for _, c := range cases { 54 | name := fmt.Sprintf("%s_to_%s", c.current, c.responseTag) 55 | t.Run(name, func(t *testing.T) { 56 | restore := mockGithub(c.responseTag) 57 | defer restore() 58 | 59 | update, latest, err := FindUpdates(c.current) 60 | assert.Nil(t, err) 61 | assert.Equal(t, c.update, update) 62 | assert.Equal(t, c.latest, latest) 63 | }) 64 | } 65 | } 66 | 67 | func mockGithub(tag string) func() { 68 | originalTransport := http.DefaultTransport 69 | 70 | http.DefaultTransport = &ghTransport{tag: tag} 71 | return func() { 72 | http.DefaultTransport = originalTransport 73 | } 74 | } 75 | 76 | type ghTransport struct { 77 | tag string 78 | } 79 | 80 | func (t *ghTransport) RoundTrip(*http.Request) (*http.Response, error) { 81 | return &http.Response{ 82 | StatusCode: http.StatusOK, 83 | Header: make(http.Header), 84 | Body: ioutil.NopCloser(bytes.NewBufferString(fmt.Sprintf(`{"tag_name": "%s"}`, t.tag))), 85 | }, nil 86 | } 87 | -------------------------------------------------------------------------------- /cmd/sourced/compose/workdir/workdir_test.go: -------------------------------------------------------------------------------- 1 | package workdir 2 | 3 | import ( 4 | "os" 5 | "path" 6 | "testing" 7 | 8 | "github.com/stretchr/testify/assert" 9 | ) 10 | 11 | func TestBuilder(t *testing.T) { 12 | assert := assert.New(t) 13 | 14 | workdirsPath := path.Join(os.TempDir(), "builder") 15 | defer func() { 16 | os.RemoveAll(workdirsPath) 17 | }() 18 | 19 | b := builder{workdirsPath: workdirsPath} 20 | 21 | // incorrect: not in workdirsPath 22 | _, err := b.Build("/not/in/workdirs") 23 | assert.EqualError(err, "invalid workdir type for path /not/in/workdirs") 24 | 25 | // incorrect: unknown type 26 | unknownDir := path.Join(workdirsPath, "unknown") 27 | _, err = b.Build(unknownDir) 28 | assert.EqualError(err, "invalid workdir type for path "+unknownDir) 29 | 30 | // local 31 | name := "some" 32 | localDir := path.Join(workdirsPath, "local", encodeDirName(name)) 33 | wd, err := b.Build(localDir) 34 | assert.Nil(err) 35 | assert.Equal(Local, wd.Type) 36 | assert.Equal(name, wd.Name) 37 | assert.Equal(localDir, wd.Path) 38 | 39 | // org 40 | orgDir := path.Join(workdirsPath, "orgs", encodeDirName(name)) 41 | wd, err = b.Build(orgDir) 42 | assert.Nil(err) 43 | assert.Equal(Orgs, wd.Type) 44 | assert.Equal(name, wd.Name) 45 | assert.Equal(orgDir, wd.Path) 46 | } 47 | 48 | func TestIsEmptyFile(t *testing.T) { 49 | assert := assert.New(t) 50 | 51 | // not exist 52 | ok, err := isEmptyFile("/does/not/exist") 53 | assert.Nil(err) 54 | assert.True(ok) 55 | 56 | // empty 57 | emptyPath := path.Join(os.TempDir(), "empty") 58 | defer func() { 59 | os.RemoveAll(emptyPath) 60 | }() 61 | f, err := os.Create(emptyPath) 62 | assert.Nil(err) 63 | assert.Nil(f.Close()) 64 | 65 | ok, err = isEmptyFile(emptyPath) 66 | assert.Nil(err) 67 | assert.True(ok) 68 | 69 | // not empty 70 | nonEmptyPath := path.Join(os.TempDir(), "non-empty") 71 | defer func() { 72 | os.RemoveAll(nonEmptyPath) 73 | }() 74 | f, err = os.Create(nonEmptyPath) 75 | assert.Nil(err) 76 | _, err = f.Write([]byte("some content")) 77 | assert.Nil(err) 78 | assert.Nil(f.Close()) 79 | 80 | ok, err = isEmptyFile(nonEmptyPath) 81 | assert.Nil(err) 82 | assert.False(ok) 83 | } 84 | -------------------------------------------------------------------------------- /docs/usage/multiple-datasets.md: -------------------------------------------------------------------------------- 1 | # Working With Multiple Data Sets 2 | 3 | You can deploy more than one **source{d} CE** instance with different sets of organizations, or repositories, to analyze. 4 | 5 | For example, you may have initially started **source{d} CE** with the repositories in the `src-d` organization, with the command: 6 | ```shell 7 | $ sourced init orgs --token src-d 8 | ``` 9 | 10 | After a while, you may want to analyze the data on another set of repositories. You can run `sourced init` again with a different organization: 11 | ```shell 12 | $ sourced init orgs --token bblfsh 13 | ``` 14 | 15 | This command will then stop all the running containers used for the previous dataset, create an isolated environment for the new data, and create a new, clean deployment. 16 | 17 | Please note that each path will have an isolated deployment. This means that for example any chart or dashboard created for the deployment for `src-d` will not be available to the new deployment for `bblfsh`. 18 | 19 | Each isolated environment is persistent (unless you run `sourced prune`). Which means that if you decide to re-deploy **source{d} CE** using the original organization: 20 | ```shell 21 | $ sourced init orgs --token src-d 22 | ``` 23 | 24 | You will get back to the previous state, and things like charts and dashboards will be restored. 25 | 26 | These isolated environments also allow you to deploy **source{d} CE** using a local set of Git repositories. For example, if we want a third deployment to analyze repositories already existing in the `~/repos` directory, we just need to run `init` again: 27 | 28 | ```shell 29 | $ sourced init local ~/repos 30 | ``` 31 | 32 | You can list all the installed instances, and know which one is active at any moment by running `sourced status workdirs`. 33 | 34 | If you are familiar with Docker Compose and you want more control over the underlying resources, you can explore the contents of your `~/.sourced` directory. There you will find a `docker-compose.yml` and `.env` files for each set of repositories used by `sourced init`. 35 | 36 | _You can read more about how the environments are isolated in the **source{d} CE** 37 | [architecture docs](../learn-more/architecture.md)_ 38 | -------------------------------------------------------------------------------- /cmd/sourced/compose/workdir/factory_test.go: -------------------------------------------------------------------------------- 1 | package workdir 2 | 3 | import ( 4 | "os" 5 | "path" 6 | "path/filepath" 7 | "testing" 8 | 9 | "github.com/stretchr/testify/suite" 10 | ) 11 | 12 | type FactorySuite struct { 13 | suite.Suite 14 | 15 | originSrcdDir string 16 | } 17 | 18 | func TestFactorySuite(t *testing.T) { 19 | suite.Run(t, &FactorySuite{}) 20 | } 21 | 22 | func (s *FactorySuite) BeforeTest(suiteName, testName string) { 23 | s.originSrcdDir = os.Getenv("SOURCED_DIR") 24 | 25 | // on macOs os.TempDir returns symlink and tests fails 26 | tmpDir, _ := filepath.EvalSymlinks(os.TempDir()) 27 | srdPath := path.Join(tmpDir, testName) 28 | err := os.MkdirAll(srdPath, os.ModePerm) 29 | s.Nil(err) 30 | 31 | os.Setenv("SOURCED_DIR", srdPath) 32 | } 33 | 34 | func (s *FactorySuite) AfterTest(suiteName, testName string) { 35 | os.RemoveAll(os.Getenv("SOURCED_DIR")) 36 | os.Setenv("SOURCED_DIR", s.originSrcdDir) 37 | } 38 | 39 | func (s *FactorySuite) TestInitLocal() { 40 | reposdir := "some-dir" 41 | wd, err := InitLocal(reposdir) 42 | s.Nil(err) 43 | s.Equal(Local, wd.Type) 44 | s.Equal(reposdir, wd.Name) 45 | 46 | // check docker-compose.yml exists 47 | composeYmlPath := path.Join(wd.Path, "docker-compose.yml") 48 | _, err = os.Stat(composeYmlPath) 49 | s.Nil(err) 50 | 51 | // check .env file 52 | envPath := path.Join(wd.Path, ".env") 53 | _, err = os.Stat(envPath) 54 | s.Nil(err) 55 | 56 | envf := envFile{} 57 | s.Nil(readEnvFile(encodeDirName(reposdir), "local", &envf)) 58 | 59 | s.Equal(reposdir, envf.GitbaseVolumeSource) 60 | s.False(envf.NoForks) 61 | } 62 | 63 | func (s *FactorySuite) TestInitOrgs() { 64 | orgs := []string{"org2", "org1"} 65 | name := "org1,org2" 66 | token := "some-token" 67 | wd, err := InitOrgs(orgs, token, true) 68 | s.Nil(err) 69 | s.Equal(Orgs, wd.Type) 70 | s.Equal(name, wd.Name) 71 | 72 | // check docker-compose.yml exists 73 | composeYmlPath := path.Join(wd.Path, "docker-compose.yml") 74 | _, err = os.Stat(composeYmlPath) 75 | s.Nil(err) 76 | 77 | // check .env file 78 | envPath := path.Join(wd.Path, ".env") 79 | _, err = os.Stat(envPath) 80 | s.Nil(err) 81 | 82 | envf := envFile{} 83 | s.Nil(readEnvFile(encodeDirName(name), "orgs", &envf)) 84 | 85 | s.Equal("gitbase_repositories", envf.GitbaseVolumeSource) 86 | s.Equal(orgs, envf.GithubOrganizations) 87 | s.Equal(token, envf.GithubToken) 88 | s.False(envf.NoForks) 89 | } 90 | 91 | func (s *FactorySuite) TestReInitForksOrgs() { 92 | orgs := []string{"org2", "org1"} 93 | _, err := InitOrgs(orgs, "", false) 94 | s.Nil(err) 95 | 96 | _, err = InitOrgs(orgs, "", true) 97 | s.EqualError(err, "initialization failed: workdir was previously initialized with a different value for forks support") 98 | } 99 | -------------------------------------------------------------------------------- /docs/quickstart/3-init-sourced.md: -------------------------------------------------------------------------------- 1 | # Initialize source{d} Community Edition 2 | 3 | _For the full list of the sub-commands offered by `sourced`, please take a look 4 | at [the `sourced` sub-commands inventory](../usage/commands.md)._ 5 | 6 | **source{d} CE** can be initialized from 2 different data sources: GitHub organizations, or local Git repositories. 7 | 8 | Please note that you have to choose one data source to initialize **source{d} CE**, but you can have more than one isolated environment, and they can have different sources. See the guide about [Working With Multiple Data Sets](../usage/multiple-datasets.md) for more details. 9 | 10 | **source{d} CE** will download and install Docker images on demand. Therefore, the first time you run some of these commands, they might take a bit of time to start up. Subsequent runs will be faster. 11 | 12 | 13 | ## From GitHub Organizations 14 | 15 | When using GitHub organizations to populate the **source{d} CE** database you only need to provide a list of organization names and a [GitHub personal access token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/). If no scope is granted to the user token, only public 16 | data will be fetched. To let **source{d} CE** access also private repositories and hidden users, the token should 17 | have the following scopes enabled: 18 | 19 | - `repo` Full control of private repositories 20 | - `read:org` Read org and team membership, read org projects 21 | 22 | 23 | Use this command to initialize, e.g. 24 | 25 | ```shell 26 | $ sourced init orgs --token src-d,bblfsh 27 | ``` 28 | 29 | It will also download, in the background, the repositories of the passed GitHub organizations, and their metadata: pull requests, issues, users... 30 | 31 | 32 | ## From Local Repositories 33 | 34 | ```shell 35 | $ sourced init local 36 | ``` 37 | 38 | It will initialize **source{d} CE** to analyze the git repositories under the passed path, or under the current directory if no one is passed. The repositories will be found recursively. 39 | 40 | **Note for macOS:** 41 | Docker for Mac [requires enabling file sharing](https://docs.docker.com/docker-for-mac/troubleshoot/#volume-mounting-requires-file-sharing-for-any-project-directories-outside-of-users) for any path outside of `/Users`. 42 | 43 | **Note for Windows:** Docker for Windows [requires shared drives](https://docs.docker.com/docker-for-windows/#shared-drives). Other than that, it's important to use a working directory that doesn't include any sub-directory whose access is not readable by the user running `sourced`. For example, using `C:\Users` as workdir, will most probably not work. For more details see [this issue](https://github.com/src-d/engine/issues/250). 44 | 45 | 46 | ## What's Next? 47 | 48 | Once **source{d} CE** has been initialized, it will automatically open the web UI. 49 | If the UI is not opened automatically, you can use `sourced web` command, or visit http://127.0.0.1:8088. 50 | 51 | Use login: `admin` and password: `admin`, to access the web interface. 52 | -------------------------------------------------------------------------------- /test/common.go: -------------------------------------------------------------------------------- 1 | // +build integration 2 | 3 | package test 4 | 5 | import ( 6 | "fmt" 7 | "io/ioutil" 8 | "log" 9 | "os" 10 | "path/filepath" 11 | "runtime" 12 | "strings" 13 | "testing" 14 | 15 | "github.com/stretchr/testify/assert" 16 | "github.com/stretchr/testify/suite" 17 | ) 18 | 19 | // TODO (carlosms) this could be build/bin, workaround for https://github.com/src-d/ci/issues/97 20 | var srcdBin = fmt.Sprintf("../build/sourced-ce_%s_%s/sourced", runtime.GOOS, runtime.GOARCH) 21 | 22 | func init() { 23 | if os.Getenv("SOURCED_BIN") != "" { 24 | srcdBin = os.Getenv("SOURCED_BIN") 25 | } 26 | } 27 | 28 | type IntegrationSuite struct { 29 | suite.Suite 30 | *Commander 31 | TestDir string 32 | } 33 | 34 | func (s *IntegrationSuite) SetupTest() { 35 | testDir, err := ioutil.TempDir("", strings.Replace(s.T().Name(), "/", "_", -1)) 36 | if err != nil { 37 | log.Fatal(err) 38 | } 39 | 40 | if runtime.GOOS == "windows" { 41 | testDir, err = filepath.EvalSymlinks(testDir) 42 | if err != nil { 43 | log.Fatal(err) 44 | } 45 | } 46 | 47 | s.TestDir = testDir 48 | s.Commander = &Commander{bin: srcdBin, sourcedDir: filepath.Join(s.TestDir, "sourced")} 49 | 50 | // Instead of downloading the compose file, copy the local file 51 | err = os.MkdirAll(filepath.Join(s.sourcedDir, "compose-files", "local"), os.ModePerm) 52 | s.Require().NoError(err) 53 | 54 | p, _ := filepath.Abs(filepath.FromSlash("../docker-compose.yml")) 55 | bytes, err := ioutil.ReadFile(p) 56 | s.Require().NoError(err) 57 | err = ioutil.WriteFile(filepath.Join(s.sourcedDir, "compose-files", "local", "docker-compose.yml"), bytes, 0644) 58 | s.Require().NoError(err) 59 | 60 | r := s.RunCommand("compose", "set", "local") 61 | s.Require().NoError(r.Error, r.Combined()) 62 | } 63 | 64 | func (s *IntegrationSuite) TearDownTest() { 65 | // don't run prune on failed test to help debug. But stop the containers 66 | // to avoid port conflicts in the next test 67 | if s.T().Failed() { 68 | s.RunCommand("stop") 69 | s.T().Logf("Test failed. sourced data dir left in %s", s.TestDir) 70 | s.T().Logf("Probably there are also docker volumes left untouched") 71 | return 72 | } 73 | 74 | s.RunCommand("prune", "--all") 75 | 76 | os.RemoveAll(s.TestDir) 77 | } 78 | 79 | func (s *IntegrationSuite) testSQL() { 80 | testCases := []string{ 81 | "show tables", 82 | "show tables;", 83 | " show tables ; ", 84 | "/* comment */ show tables;", 85 | `/* multi line 86 | comment */ 87 | show tables;`, 88 | } 89 | 90 | showTablesOutput := 91 | `Table 92 | blobs 93 | commit_blobs 94 | commit_files 95 | commit_trees 96 | commits 97 | files 98 | ref_commits 99 | refs 100 | remotes 101 | repositories 102 | tree_entries 103 | ` 104 | 105 | for _, query := range testCases { 106 | s.T().Run(query, func(t *testing.T) { 107 | assert := assert.New(t) 108 | 109 | r := s.RunCommand("sql", query) 110 | assert.NoError(r.Error, r.Combined()) 111 | 112 | assert.Contains(r.Stdout(), showTablesOutput) 113 | }) 114 | } 115 | } 116 | -------------------------------------------------------------------------------- /cmd/sourced/cmd/root.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | "runtime" 7 | 8 | "github.com/src-d/sourced-ce/cmd/sourced/compose" 9 | "github.com/src-d/sourced-ce/cmd/sourced/compose/file" 10 | "github.com/src-d/sourced-ce/cmd/sourced/compose/workdir" 11 | "github.com/src-d/sourced-ce/cmd/sourced/dir" 12 | "github.com/src-d/sourced-ce/cmd/sourced/format" 13 | 14 | "gopkg.in/src-d/go-cli.v0" 15 | ) 16 | 17 | const name = "sourced" 18 | 19 | var version = "master" 20 | 21 | var rootCmd = cli.NewNoDefaults(name, "source{d} Community Edition & Enterprise Edition CLI client") 22 | 23 | // Init sets the version rewritten by the CI build and adds default sub commands 24 | func Init(v, build string) { 25 | version = v 26 | 27 | rootCmd.AddCommand(&cli.VersionCommand{ 28 | Name: name, 29 | Version: version, 30 | Build: build, 31 | }) 32 | 33 | if runtime.GOOS != "windows" { 34 | rootCmd.AddCommand(&cli.CompletionCommand{ 35 | Name: name, 36 | }, cli.InitCompletionCommand(name)) 37 | } 38 | } 39 | 40 | // Command implements the default group flags. It is meant to be embedded into 41 | // other application commands to provide default behavior for logging, config 42 | type Command struct { 43 | cli.PlainCommand 44 | cli.LogOptions `group:"Log Options"` 45 | } 46 | 47 | // Execute adds all child commands to the root command and sets flags appropriately. 48 | // This is called by main.main(). It only needs to happen once to the rootCmd. 49 | func Execute() { 50 | if err := dir.Prepare(); err != nil { 51 | fmt.Println(err) 52 | log(err) 53 | os.Exit(1) 54 | } 55 | 56 | if err := rootCmd.Run(os.Args); err != nil { 57 | log(err) 58 | os.Exit(1) 59 | } 60 | } 61 | 62 | func log(err error) { 63 | switch { 64 | case workdir.ErrMalformed.Is(err) || dir.ErrNotExist.Is(err): 65 | printRed("Cannot perform this action, source{d} needs to be initialized first with the 'init' sub command") 66 | case workdir.ErrInitFailed.Is(err): 67 | printRed("Cannot perform this action, full re-initialization is needed, run 'prune' command first") 68 | case dir.ErrNotValid.Is(err): 69 | printRed("Cannot perform this action, config directory is not valid") 70 | case compose.ErrComposeAlternative.Is(err): 71 | printRed("docker-compose is not installed, and there was an error while trying to use the container alternative") 72 | printRed(" see: https://docs.sourced.tech/community-edition/quickstart/1-install-requirements#docker-compose") 73 | case file.ErrConfigDownload.Is(err): 74 | printRed("The source{d} CE config file could not be set as active") 75 | case fmt.Sprintf("%T", err) == "*flags.Error": 76 | // syntax error is already logged by go-cli 77 | default: 78 | // unknown errors have no special message 79 | } 80 | 81 | switch { 82 | case dir.ErrNetwork.Is(err): 83 | // TODO(dpordomingo): if start using "https://golang.org/pkg/errors/", 84 | // we could do `var myErr ErrNetwork; errors.As(err, &myErr)` 85 | // to provide more info about the actual network error 86 | printRed("The resource could not be downloaded from the Internet") 87 | printRed(" see: https://docs.sourced.tech/community-edition/quickstart/1-install-requirements#internet-connection") 88 | case dir.ErrWrite.Is(err): 89 | // TODO(dpordomingo): see todo above 90 | printRed("could not write file") 91 | 92 | } 93 | } 94 | 95 | func printRed(message string) { 96 | fmt.Println(format.Colorize(format.Red, message)) 97 | } 98 | -------------------------------------------------------------------------------- /cmd/sourced/compose/workdir/env_file_test.go: -------------------------------------------------------------------------------- 1 | package workdir 2 | 3 | import ( 4 | "strings" 5 | "testing" 6 | 7 | "github.com/stretchr/testify/assert" 8 | ) 9 | 10 | // default limits depend on host system and can't be used in tests 11 | func setResourceLimits(f *envFile) { 12 | f.GitcollectorLimitCPU = 2.0 13 | f.GitbaseLimitCPU = 1.5 14 | f.GitbaseLimitMem = 100 15 | } 16 | 17 | const localContent = `COMPOSE_PROJECT_NAME=srcd-dir-name 18 | GITBASE_VOLUME_TYPE=bind 19 | GITBASE_VOLUME_SOURCE=repo-dir 20 | GITBASE_SIVA= 21 | GITHUB_ORGANIZATIONS= 22 | GITHUB_TOKEN= 23 | NO_FORKS= 24 | GITBASE_LIMIT_CPU=1.5 25 | GITCOLLECTOR_LIMIT_CPU=2 26 | GITBASE_LIMIT_MEM=100 27 | ` 28 | 29 | const orgsContent = `COMPOSE_PROJECT_NAME=srcd-dir-name 30 | GITBASE_VOLUME_TYPE=volume 31 | GITBASE_VOLUME_SOURCE=gitbase_repositories 32 | GITBASE_SIVA=true 33 | GITHUB_ORGANIZATIONS=org1,org2 34 | GITHUB_TOKEN=token 35 | NO_FORKS=true 36 | GITBASE_LIMIT_CPU=1.5 37 | GITCOLLECTOR_LIMIT_CPU=2 38 | GITBASE_LIMIT_MEM=100 39 | ` 40 | 41 | const emptyContent = `COMPOSE_PROJECT_NAME= 42 | GITBASE_VOLUME_TYPE= 43 | GITBASE_VOLUME_SOURCE= 44 | GITBASE_SIVA= 45 | GITHUB_ORGANIZATIONS= 46 | GITHUB_TOKEN= 47 | NO_FORKS= 48 | GITBASE_LIMIT_CPU=0 49 | GITCOLLECTOR_LIMIT_CPU=0 50 | GITBASE_LIMIT_MEM=0 51 | ` 52 | 53 | func TestEnvMarshal(t *testing.T) { 54 | assert := assert.New(t) 55 | 56 | f := newLocalEnvFile("dir-name", "repo-dir") 57 | setResourceLimits(&f) 58 | b, err := f.MarshalEnv() 59 | assert.Nil(err) 60 | assert.Equal(localContent, strings.ReplaceAll(string(b), "\r\n", "\n")) 61 | 62 | f = newOrgEnvFile("dir-name", []string{"org1", "org2"}, "token", false) 63 | setResourceLimits(&f) 64 | b, err = f.MarshalEnv() 65 | assert.Nil(err) 66 | assert.Equal(orgsContent, strings.ReplaceAll(string(b), "\r\n", "\n")) 67 | 68 | f = envFile{} 69 | b, err = f.MarshalEnv() 70 | assert.Nil(err) 71 | assert.Equal(emptyContent, strings.ReplaceAll(string(b), "\r\n", "\n")) 72 | } 73 | 74 | func TestEnvUnmarshal(t *testing.T) { 75 | assert := assert.New(t) 76 | 77 | b := []byte(localContent) 78 | f := envFile{} 79 | assert.Nil(f.UnmarshalEnv(b)) 80 | assert.Equal(envFile{ 81 | ComposeProjectName: "srcd-dir-name", 82 | GitbaseVolumeType: "bind", 83 | GitbaseVolumeSource: "repo-dir", 84 | 85 | GitcollectorLimitCPU: 2.0, 86 | GitbaseLimitCPU: 1.5, 87 | GitbaseLimitMem: 100, 88 | }, f) 89 | 90 | b = []byte(orgsContent) 91 | f = envFile{} 92 | assert.Nil(f.UnmarshalEnv(b)) 93 | assert.Equal(envFile{ 94 | ComposeProjectName: "srcd-dir-name", 95 | GitbaseVolumeType: "volume", 96 | GitbaseVolumeSource: "gitbase_repositories", 97 | GitbaseSiva: true, 98 | GithubOrganizations: []string{"org1", "org2"}, 99 | GithubToken: "token", 100 | NoForks: true, 101 | 102 | GitcollectorLimitCPU: 2.0, 103 | GitbaseLimitCPU: 1.5, 104 | GitbaseLimitMem: 100, 105 | }, f) 106 | 107 | b = []byte("") 108 | f = envFile{} 109 | assert.Nil(f.UnmarshalEnv(b)) 110 | 111 | b = []byte(" COMPOSE_PROJECT_NAME=srcd-dir-name \n\n GITBASE_VOLUME_TYPE=volume ") 112 | f = envFile{} 113 | assert.Nil(f.UnmarshalEnv(b)) 114 | assert.Equal(envFile{ 115 | ComposeProjectName: "srcd-dir-name", 116 | GitbaseVolumeType: "volume", 117 | }, f) 118 | 119 | b = []byte("UNKNOWN=1") 120 | f = envFile{} 121 | assert.Nil(f.UnmarshalEnv(b)) 122 | } 123 | -------------------------------------------------------------------------------- /.github/CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, gender identity and expression, level of experience, 9 | education, socio-economic status, nationality, personal appearance, race, 10 | religion, or sexual identity and orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies both within project spaces and in public spaces 49 | when an individual is representing the project or its community. Examples of 50 | representing a project or community include using an official project e-mail 51 | address, posting via an official social media account, or acting as an appointed 52 | representative at an online or offline event. Representation of a project may be 53 | further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at conduct@sourced.tech. All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ## Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 72 | 73 | [homepage]: https://www.contributor-covenant.org 74 | -------------------------------------------------------------------------------- /docs/usage/examples.md: -------------------------------------------------------------------------------- 1 | # SQL Examples to Analyze Your Data 2 | 3 | _If you want to know what the database schema looks like, you can refer to the [diagram about gitbase entities and relations](https://docs.sourced.tech/gitbase/using-gitbase/schema#database-diagram), or just use regular `SHOW` or `DESCRIBE` queries._ 4 | 5 | _In gitbase repository, you will find more [SQL examples of queries](https://docs.sourced.tech/gitbase/using-gitbase/examples)._ 6 | 7 | 8 | ## Index 9 | 10 | * [Queries For Repositories](#queries-for-repositories) 11 | * [Queries With Files](#queries-with-files) 12 | * [Queries With UASTs](#queries-with-uasts) 13 | * [Queries About Comitters](#queries-about-comitters) 14 | 15 | 16 | ## Queries For Repositories 17 | 18 | **Show me the repositories I am analyzing:** 19 | 20 | ```sql 21 | SELECT * FROM repositories; 22 | ``` 23 | 24 | **Last commit messages for HEAD for every repository** 25 | 26 | ```sql 27 | SELECT commit_message 28 | FROM refs 29 | NATURAL JOIN commits 30 | WHERE ref_name = 'HEAD'; 31 | ``` 32 | 33 | **Top 10 repositories by commit count from [HEAD](https://git-scm.com/book/en/v2/Git-Internals-Git-References#ref_the_ref):** 34 | 35 | ```sql 36 | SELECT repository_id,commit_count 37 | FROM ( 38 | SELECT 39 | repository_id, 40 | COUNT(*) AS commit_count 41 | FROM ref_commits 42 | WHERE ref_name = 'HEAD' 43 | GROUP BY repository_id 44 | ) AS q 45 | ORDER BY commit_count DESC 46 | LIMIT 10; 47 | ``` 48 | 49 | **10 top repos by file count at HEAD** 50 | 51 | ```sql 52 | SELECT repository_id, num_files FROM ( 53 | SELECT COUNT(*) num_files, repository_id 54 | FROM refs 55 | NATURAL JOIN commit_files 56 | WHERE ref_name = 'HEAD' 57 | GROUP BY repository_id 58 | ) AS t 59 | ORDER BY num_files DESC 60 | LIMIT 10; 61 | ``` 62 | 63 | 64 | ## Queries With Files 65 | 66 | **Query for all LICENSE & README files across history:** 67 | 68 | ```sql 69 | SELECT file_path, repository_id, blob_size 70 | FROM files 71 | WHERE 72 | file_path = 'LICENSE' 73 | OR file_path = 'README.md'; 74 | ``` 75 | 76 | **Query all files at [HEAD](https://git-scm.com/book/en/v2/Git-Internals-Git-References#ref_the_ref):** 77 | 78 | ```sql 79 | SELECT cf.file_path, f.blob_size 80 | FROM ref_commits rc 81 | NATURAL JOIN commit_files cf 82 | NATURAL JOIN files f 83 | WHERE 84 | rc.ref_name = 'HEAD' 85 | AND rc.history_index = 0; 86 | ``` 87 | 88 | 89 | ## Queries With UASTs 90 | 91 | _**Note**: UAST values are returned as binary blobs; they're best visualized in the web UI interface rather than the CLI where are seen as binary data._ 92 | 93 | **Retrieve the UAST for all files at [HEAD](https://git-scm.com/book/en/v2/Git-Internals-Git-References#ref_the_ref):** 94 | 95 | ```sql 96 | SELECT * FROM ( 97 | SELECT cf.file_path, 98 | UAST(f.blob_content, LANGUAGE(f.file_path, f.blob_content)) as uast 99 | FROM ref_commits r 100 | NATURAL JOIN commit_files cf 101 | NATURAL JOIN files f 102 | WHERE 103 | r.ref_name = 'HEAD' 104 | AND r.history_index = 0 105 | ) t WHERE uast != ''; 106 | ``` 107 | 108 | 109 | ## Queries About Comitters 110 | 111 | **Top committers per repository** 112 | 113 | ```sql 114 | SELECT * FROM ( 115 | SELECT 116 | commit_author_email as author, 117 | repository_id as id, 118 | count(*) as num_commits 119 | FROM commits 120 | GROUP BY commit_author_email, repository_id 121 | ) AS t 122 | ORDER BY num_commits DESC; 123 | ``` 124 | -------------------------------------------------------------------------------- /cmd/sourced/dir/dir_test.go: -------------------------------------------------------------------------------- 1 | package dir 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "net/http" 7 | "net/http/httptest" 8 | "os" 9 | "path" 10 | "testing" 11 | 12 | "github.com/pkg/errors" 13 | "github.com/stretchr/testify/assert" 14 | ) 15 | 16 | func TestValidate(t *testing.T) { 17 | assert := assert.New(t) 18 | 19 | err := validate("/does/not/exists") 20 | assert.True(ErrNotExist.Is(err)) 21 | assert.EqualError(err, "/does/not/exists does not exist") 22 | 23 | // with a file 24 | tmpFile := path.Join(os.TempDir(), "tmp-file") 25 | f, err := os.Create(tmpFile) 26 | assert.Nil(err) 27 | assert.Nil(f.Close()) 28 | defer func() { 29 | os.RemoveAll(tmpFile) 30 | }() 31 | 32 | err = validate(tmpFile) 33 | assert.True(ErrNotValid.Is(err)) 34 | assert.EqualError(err, tmpFile+" is not a valid config directory: it is not a directory") 35 | 36 | // with a dir 37 | tmpDir := path.Join(os.TempDir(), "tmp-dir") 38 | assert.Nil(os.Mkdir(tmpDir, os.ModePerm)) 39 | defer func() { 40 | os.RemoveAll(tmpDir) 41 | }() 42 | 43 | err = validate(tmpDir) 44 | assert.Nil(err) 45 | 46 | // read only 47 | assert.Nil(os.Chmod(tmpDir, 0444)) 48 | err = validate(tmpDir) 49 | assert.True(ErrNotValid.Is(err)) 50 | assert.EqualError(err, tmpDir+" is not a valid config directory: it has no read-write access") 51 | 52 | // write only 53 | assert.Nil(os.Chmod(tmpDir, 0222)) 54 | err = validate(tmpDir) 55 | assert.True(ErrNotValid.Is(err)) 56 | assert.EqualError(err, tmpDir+" is not a valid config directory: it has no read-write access") 57 | } 58 | 59 | func TestPrepare(t *testing.T) { 60 | assert := assert.New(t) 61 | 62 | tmpDir := path.Join(os.TempDir(), "tmp-dir") 63 | assert.Nil(os.Mkdir(tmpDir, os.ModePerm)) 64 | defer func() { 65 | os.RemoveAll(tmpDir) 66 | }() 67 | 68 | originSrcdDir := os.Getenv("SOURCED_DIR") 69 | defer func() { 70 | os.Setenv("SOURCED_DIR", originSrcdDir) 71 | }() 72 | 73 | os.Setenv("SOURCED_DIR", tmpDir) 74 | assert.Nil(Prepare()) 75 | 76 | toCreateDir := path.Join(os.TempDir(), "to-create-dir") 77 | defer func() { 78 | os.RemoveAll(toCreateDir) 79 | }() 80 | _, err := os.Stat(toCreateDir) 81 | assert.True(os.IsNotExist(err)) 82 | 83 | os.Setenv("SOURCED_DIR", toCreateDir) 84 | assert.Nil(Prepare()) 85 | _, err = os.Stat(toCreateDir) 86 | assert.Nil(err) 87 | } 88 | 89 | func TestDownloadURL(t *testing.T) { 90 | assert := assert.New(t) 91 | 92 | // success 93 | fileContext := []byte("hello") 94 | handler := func(w http.ResponseWriter, r *http.Request) { 95 | w.WriteHeader(http.StatusOK) 96 | w.Write(fileContext) 97 | } 98 | server := httptest.NewServer(http.HandlerFunc(handler)) 99 | dirPath := path.Join(os.TempDir(), "some-dir") 100 | filePath := path.Join(dirPath, "file-to-download") 101 | defer func() { 102 | os.RemoveAll(dirPath) 103 | }() 104 | 105 | assert.Nil(DownloadURL(server.URL, filePath)) 106 | _, err := os.Stat(filePath) 107 | assert.Nil(err) 108 | 109 | b, err := ioutil.ReadFile(filePath) 110 | assert.Nil(err) 111 | assert.Equal(fileContext, b) 112 | 113 | // error 114 | handler = func(w http.ResponseWriter, r *http.Request) { 115 | w.WriteHeader(http.StatusNotFound) 116 | } 117 | server = httptest.NewServer(http.HandlerFunc(handler)) 118 | err = DownloadURL(server.URL, "/dev/null") 119 | errExpected := errors.Wrapf( 120 | fmt.Errorf("HTTP status %v", "404 Not Found"), 121 | "network error downloading %s", server.URL, 122 | ) 123 | 124 | assert.EqualError(err, errExpected.Error()) 125 | } 126 | -------------------------------------------------------------------------------- /cmd/sourced/cmd/compose.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "fmt" 5 | "strconv" 6 | 7 | composefile "github.com/src-d/sourced-ce/cmd/sourced/compose/file" 8 | 9 | "gopkg.in/src-d/go-cli.v0" 10 | ) 11 | 12 | type composeCmd struct { 13 | cli.PlainCommand `name:"compose" short-description:"Manage source{d} docker compose files" long-description:"Manage source{d} docker compose files"` 14 | } 15 | 16 | type composeDownloadCmd struct { 17 | Command `name:"download" short-description:"Download docker compose files" long-description:"Download docker compose files. By default the command downloads the file for this binary version.\n\nUse the 'version' argument to choose a specific revision from\nthe https://github.com/src-d/sourced-ce repository, or to set a\nURL to a docker-compose.yml file.\n\nExamples:\n\nsourced compose download\nsourced compose download v0.0.1\nsourced compose download master\nsourced compose download https://raw.githubusercontent.com/src-d/sourced-ce/master/docker-compose.yml"` 18 | 19 | Args struct { 20 | Version string `positional-arg-name:"version" description:"Either a revision (tag, full sha1) or a URL to a docker-compose.yml file"` 21 | } `positional-args:"yes"` 22 | } 23 | 24 | func (c *composeDownloadCmd) Execute(args []string) error { 25 | v := c.Args.Version 26 | if v == "" { 27 | v = version 28 | } 29 | 30 | err := composefile.ActivateFromRemote(v) 31 | if err != nil { 32 | return err 33 | } 34 | 35 | fmt.Println("Docker compose file successfully downloaded to your ~/.sourced/compose-files directory. It is now the active compose file.") 36 | fmt.Println("To update your current installation use `sourced restart`") 37 | return nil 38 | } 39 | 40 | type composeListCmd struct { 41 | Command `name:"list" short-description:"List the downloaded docker compose files" long-description:"List the downloaded docker compose files"` 42 | } 43 | 44 | func (c *composeListCmd) Execute(args []string) error { 45 | active, err := composefile.Active() 46 | if err != nil { 47 | return err 48 | } 49 | 50 | files, err := composefile.List() 51 | if err != nil { 52 | return err 53 | } 54 | 55 | for index, file := range files { 56 | fmt.Printf("[%d]", index) 57 | if file == active { 58 | fmt.Printf("* %s\n", file) 59 | } else { 60 | fmt.Printf(" %s\n", file) 61 | } 62 | } 63 | 64 | return nil 65 | } 66 | 67 | type composeSetDefaultCmd struct { 68 | Command `name:"set" short-description:"Set the active docker compose file" long-description:"Set the active docker compose file"` 69 | 70 | Args struct { 71 | File string `positional-arg-name:"index/name" description:"Provide name or index of compose file on 'sourced compose list'"` 72 | } `positional-args:"yes" required:"yes"` 73 | } 74 | 75 | func (c *composeSetDefaultCmd) Execute(args []string) error { 76 | files, err := composefile.List() 77 | if err != nil { 78 | return err 79 | } 80 | 81 | index, err := strconv.Atoi(c.Args.File) 82 | 83 | if err == nil { 84 | if index >= 0 && index < len(files) { 85 | active := files[index] 86 | err = composefile.SetActive(active) 87 | } else { 88 | return fmt.Errorf("Index is out of range, please check the output of 'sourced compose list'") 89 | } 90 | 91 | } else { 92 | err := composefile.SetActive(c.Args.File) 93 | if err != nil { 94 | return err 95 | } 96 | } 97 | 98 | fmt.Println("Active docker compose file was changed successfully.") 99 | fmt.Println("To update your current installation use `sourced restart`") 100 | return nil 101 | } 102 | 103 | func init() { 104 | c := rootCmd.AddCommand(&composeCmd{}) 105 | c.AddCommand(&composeDownloadCmd{}) 106 | c.AddCommand(&composeListCmd{}) 107 | c.AddCommand(&composeSetDefaultCmd{}) 108 | } 109 | -------------------------------------------------------------------------------- /cmd/sourced/dir/dir.go: -------------------------------------------------------------------------------- 1 | // Package dir provides functions to manage the config directories. 2 | package dir 3 | 4 | import ( 5 | "fmt" 6 | "io" 7 | "net/http" 8 | "os" 9 | "path/filepath" 10 | 11 | "github.com/pkg/errors" 12 | goerrors "gopkg.in/src-d/go-errors.v1" 13 | ) 14 | 15 | // ErrNotExist is returned when config dir does not exists 16 | var ErrNotExist = goerrors.NewKind("%s does not exist") 17 | 18 | // ErrNotValid is returned when config dir is not valid 19 | var ErrNotValid = goerrors.NewKind("%s is not a valid config directory: %s") 20 | 21 | // ErrNetwork is returned when could not download 22 | var ErrNetwork = goerrors.NewKind("network error downloading %s") 23 | 24 | // ErrWrite is returned when could not write 25 | var ErrWrite = goerrors.NewKind("write error at %s") 26 | 27 | // Path returns the absolute path for $SOURCED_DIR, or $HOME/.sourced if unset 28 | // and returns an error if it does not exist or it could not be read. 29 | func Path() (string, error) { 30 | srcdDir, err := srcdPath() 31 | if err != nil { 32 | return "", err 33 | } 34 | 35 | if err := validate(srcdDir); err != nil { 36 | return "", err 37 | } 38 | 39 | return srcdDir, nil 40 | } 41 | 42 | func srcdPath() (string, error) { 43 | if d := os.Getenv("SOURCED_DIR"); d != "" { 44 | abs, err := filepath.Abs(d) 45 | if err != nil { 46 | return "", errors.Wrap(err, fmt.Sprintf("could not resolve SOURCED_DIR='%s'", d)) 47 | } 48 | 49 | return abs, nil 50 | } 51 | 52 | homedir, err := os.UserHomeDir() 53 | if err != nil { 54 | return "", errors.Wrap(err, "could not detect home directory") 55 | } 56 | 57 | return filepath.Join(homedir, ".sourced"), nil 58 | } 59 | 60 | // Prepare tries to create the config directory, returning an error if it could not 61 | // be created, or nil if already exist or was successfully created. 62 | func Prepare() error { 63 | srcdDir, err := srcdPath() 64 | if err != nil { 65 | return err 66 | } 67 | 68 | err = validate(srcdDir) 69 | if ErrNotExist.Is(err) { 70 | if err := os.MkdirAll(srcdDir, os.ModePerm); err != nil { 71 | return ErrNotValid.New(srcdDir, err) 72 | } 73 | 74 | return nil 75 | } 76 | 77 | return err 78 | } 79 | 80 | // validate validates that the passed config dir path is valid 81 | func validate(path string) error { 82 | info, err := os.Stat(path) 83 | if os.IsNotExist(err) { 84 | return ErrNotExist.New(path) 85 | } 86 | 87 | if err != nil { 88 | return ErrNotValid.New(path, err) 89 | } 90 | 91 | if !info.IsDir() { 92 | return ErrNotValid.New(path, "it is not a directory") 93 | } 94 | 95 | readWriteAccessMode := os.FileMode(0700) 96 | if info.Mode()&readWriteAccessMode != readWriteAccessMode { 97 | return ErrNotValid.New(path, "it has no read-write access") 98 | } 99 | 100 | return nil 101 | } 102 | 103 | // DownloadURL downloads the given url to a file to the 104 | // dst path, creating the directory if it's needed 105 | func DownloadURL(url, dst string) (err error) { 106 | resp, err := http.Get(url) 107 | if err != nil { 108 | return ErrNetwork.Wrap(err, url) 109 | } 110 | defer resp.Body.Close() 111 | 112 | if resp.StatusCode != http.StatusOK { 113 | return ErrNetwork.Wrap(fmt.Errorf("HTTP status %v", resp.Status), url) 114 | } 115 | 116 | if err := os.MkdirAll(filepath.Dir(dst), os.ModePerm); err != nil { 117 | return ErrWrite.Wrap(err, filepath.Dir(dst)) 118 | } 119 | 120 | out, err := os.Create(dst) 121 | if err != nil { 122 | return ErrWrite.Wrap(err, dst) 123 | } 124 | defer out.Close() 125 | 126 | _, err = io.Copy(out, resp.Body) 127 | if err != nil { 128 | return ErrWrite.Wrap(err, dst) 129 | } 130 | 131 | return nil 132 | } 133 | 134 | // TmpPath returns the absolute path for /tmp/srcd 135 | func TmpPath() string { 136 | return filepath.Join(os.TempDir(), "srcd") 137 | } 138 | -------------------------------------------------------------------------------- /test/init_local_test.go: -------------------------------------------------------------------------------- 1 | // +build integration 2 | 3 | package test 4 | 5 | import ( 6 | "fmt" 7 | "os" 8 | "os/exec" 9 | "path/filepath" 10 | "testing" 11 | 12 | "github.com/stretchr/testify/require" 13 | "github.com/stretchr/testify/suite" 14 | ) 15 | 16 | type InitLocalTestSuite struct { 17 | IntegrationSuite 18 | } 19 | 20 | func TestInitLocalTestSuite(t *testing.T) { 21 | itt := InitLocalTestSuite{} 22 | suite.Run(t, &itt) 23 | } 24 | 25 | func (s *InitLocalTestSuite) TestWithInvalidWorkdir() { 26 | require := s.Require() 27 | 28 | invalidWorkDir := filepath.Join(s.TestDir, "invalid-workdir") 29 | 30 | _, err := os.Create(invalidWorkDir) 31 | if err != nil { 32 | s.T().Fatal(err) 33 | } 34 | 35 | r := s.RunCommand("init", "local", invalidWorkDir) 36 | require.Error(r.Error) 37 | 38 | require.Equal( 39 | fmt.Sprintf("path '%s' is not a valid directory\n", invalidWorkDir), 40 | r.Stderr(), 41 | ) 42 | } 43 | 44 | func (s *InitLocalTestSuite) TestChangeWorkdir() { 45 | req := s.Require() 46 | 47 | r := s.RunCommand("status", "workdirs") 48 | req.Error(r.Error) 49 | 50 | // Create 2 workdirs, each with a repo 51 | workdirA := filepath.Join(s.TestDir, "workdir_a") 52 | workdirB := filepath.Join(s.TestDir, "workdir_b") 53 | pathA := filepath.Join(workdirA, "repo_a") 54 | pathB := filepath.Join(workdirB, "repo_b") 55 | 56 | s.initGitRepo(pathA) 57 | s.initGitRepo(pathB) 58 | 59 | // init with workdir A 60 | r = s.RunCommand("init", "local", workdirA) 61 | req.NoError(r.Error, r.Combined()) 62 | 63 | r = s.RunCommand("status", "workdirs") 64 | req.NoError(r.Error, r.Combined()) 65 | 66 | req.Equal(fmt.Sprintf("* %v\n", workdirA), r.Stdout()) 67 | 68 | r = s.RunCommand("sql", "select * from repositories") 69 | req.NoError(r.Error, r.Combined()) 70 | 71 | req.Contains(r.Stdout(), 72 | `repository_id 73 | repo_a 74 | `) 75 | 76 | // init with workdir B 77 | r = s.RunCommand("init", "local", workdirB) 78 | req.NoError(r.Error, r.Combined()) 79 | 80 | r = s.RunCommand("status", "workdirs") 81 | req.NoError(r.Error, r.Combined()) 82 | 83 | req.Equal(fmt.Sprintf(" %v\n* %v\n", workdirA, workdirB), r.Stdout()) 84 | 85 | r = s.RunCommand("sql", "select * from repositories") 86 | req.NoError(r.Error, r.Combined()) 87 | 88 | req.Contains(r.Stdout(), 89 | `repository_id 90 | repo_b 91 | `) 92 | 93 | // Test SQL queries. This should be a different test, but since starting 94 | // the environment takes a long time, it is bundled together here to speed up 95 | // the tests 96 | s.testSQL() 97 | 98 | client, err := newSupersetClient() 99 | req.NoError(err) 100 | 101 | // Test the list of dashboards created in superset 102 | s.T().Run("dashboard-list", func(t *testing.T) { 103 | req := require.New(t) 104 | 105 | links, err := client.dashboards() 106 | req.NoError(err) 107 | 108 | s.Equal([]string{ 109 | `Overview`, 110 | `Welcome`, 111 | }, links) 112 | }) 113 | 114 | // Test gitbase queries through superset 115 | s.T().Run("superset-gitbase", func(t *testing.T) { 116 | req := require.New(t) 117 | 118 | rows, err := client.gitbase("select * from repositories") 119 | req.NoError(err) 120 | 121 | s.Equal([]map[string]interface{}{ 122 | {"repository_id": "repo_b"}, 123 | }, rows) 124 | }) 125 | 126 | // Test bblfsh queries through superset 127 | s.T().Run("superset-bblfsh", func(t *testing.T) { 128 | req := require.New(t) 129 | 130 | lang, err := client.bblfsh("hello.js", `console.log("hello");`) 131 | req.NoError(err) 132 | req.Equal("javascript", lang) 133 | }) 134 | 135 | // Test gitbase can connect to bblfsh with a SQL query that uses UAST 136 | s.T().Run("gitbase-bblfsh", func(t *testing.T) { 137 | req := require.New(t) 138 | 139 | rows, err := client.gitbase( 140 | `SELECT UAST('console.log("hello");', 'javascript') AS uast`) 141 | req.NoError(err) 142 | 143 | req.Len(rows, 1) 144 | req.NotEmpty(rows[0]["uast"]) 145 | }) 146 | } 147 | 148 | func (s *InitLocalTestSuite) initGitRepo(path string) { 149 | s.T().Helper() 150 | 151 | err := os.MkdirAll(path, os.ModePerm) 152 | s.Require().NoError(err) 153 | 154 | cmd := exec.Command("git", "init", path) 155 | err = cmd.Run() 156 | s.Require().NoError(err) 157 | } 158 | -------------------------------------------------------------------------------- /cmd/sourced/cmd/status.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "io/ioutil" 7 | "os" 8 | "path/filepath" 9 | 10 | "github.com/src-d/sourced-ce/cmd/sourced/compose" 11 | "github.com/src-d/sourced-ce/cmd/sourced/compose/workdir" 12 | ) 13 | 14 | type statusCmd struct { 15 | Command `name:"status" short-description:"Show the list of working directories and the current deployment" long-description:"Show the list of working directories and the current deployment"` 16 | } 17 | 18 | type statusAllCmd struct { 19 | Command `name:"all" short-description:"Show all the available status information" long-description:"Show all the available status information"` 20 | } 21 | 22 | func (c *statusAllCmd) Execute(args []string) error { 23 | fmt.Print("List of all working directories:\n") 24 | 25 | err := printWorkdirsCmd() 26 | if err != nil { 27 | return err 28 | } 29 | 30 | active, err := activeWorkdir() 31 | if isNotExist(err) { 32 | // skip printing the config and components when there is no active dir 33 | return nil 34 | } 35 | 36 | if err != nil { 37 | return err 38 | } 39 | 40 | fmt.Print("\nConfiguration used for the active working directory:\n\n") 41 | 42 | err = printConfigCmd(active) 43 | if err != nil { 44 | return err 45 | } 46 | 47 | fmt.Print("\nStatus of all components:\n\n") 48 | 49 | err = printComponentsCmd() 50 | if err != nil { 51 | return err 52 | } 53 | 54 | return nil 55 | } 56 | 57 | type statusComponentsCmd struct { 58 | Command `name:"components" short-description:"Show the status of the components containers" long-description:"Show the status of the components containers"` 59 | } 60 | 61 | func (c *statusComponentsCmd) Execute(args []string) error { 62 | return printComponentsCmd() 63 | } 64 | 65 | func printComponentsCmd() error { 66 | return compose.Run(context.Background(), "ps") 67 | } 68 | 69 | type statusWorkdirsCmd struct { 70 | Command `name:"workdirs" short-description:"List all working directories" long-description:"List all the previously initialized working directories"` 71 | } 72 | 73 | func (c *statusWorkdirsCmd) Execute(args []string) error { 74 | return printWorkdirsCmd() 75 | } 76 | 77 | func printWorkdirsCmd() error { 78 | workdirHandler, err := workdir.NewHandler() 79 | if err != nil { 80 | return err 81 | } 82 | 83 | wds, err := workdirHandler.List() 84 | if err != nil { 85 | return err 86 | } 87 | 88 | activePath, err := activeWorkdir() 89 | // active directory does not necessarily exist 90 | if err != nil && !isNotExist(err) { 91 | return err 92 | } 93 | 94 | for _, wd := range wds { 95 | if wd.Path == activePath { 96 | fmt.Printf("* %s\n", wd.Name) 97 | } else { 98 | fmt.Printf(" %s\n", wd.Name) 99 | } 100 | } 101 | 102 | return nil 103 | } 104 | 105 | type statusConfigCmd struct { 106 | Command `name:"config" short-description:"Show the configuration for the active working directory" long-description:"Show the docker-compose environment variables configuration for the active working directory"` 107 | } 108 | 109 | func (c *statusConfigCmd) Execute(args []string) error { 110 | active, err := activeWorkdir() 111 | if err != nil { 112 | return err 113 | } 114 | 115 | return printConfigCmd(active) 116 | } 117 | 118 | func printConfigCmd(path string) error { 119 | content, err := ioutil.ReadFile(filepath.Join(path, ".env")) 120 | if err != nil { 121 | return err 122 | } 123 | 124 | fmt.Printf("%s\n", content) 125 | 126 | return nil 127 | } 128 | 129 | func isNotExist(err error) bool { 130 | if os.IsNotExist(err) { 131 | return true 132 | } 133 | 134 | if cause, ok := err.(causer); ok { 135 | return isNotExist(cause.Cause()) 136 | } 137 | 138 | return false 139 | } 140 | 141 | type causer interface { 142 | Cause() error 143 | } 144 | 145 | func activeWorkdir() (string, error) { 146 | workdirHandler, err := workdir.NewHandler() 147 | if err != nil { 148 | return "", err 149 | } 150 | 151 | active, err := workdirHandler.Active() 152 | if err != nil { 153 | return "", err 154 | } 155 | 156 | return active.Path, err 157 | } 158 | 159 | func init() { 160 | c := rootCmd.AddCommand(&statusCmd{}) 161 | 162 | c.AddCommand(&statusAllCmd{}) 163 | c.AddCommand(&statusComponentsCmd{}) 164 | c.AddCommand(&statusWorkdirsCmd{}) 165 | c.AddCommand(&statusConfigCmd{}) 166 | } 167 | -------------------------------------------------------------------------------- /docs/learn-more/architecture.md: -------------------------------------------------------------------------------- 1 | # source{d} Community Editon Architecture 2 | 3 | **source{d} Community Editon** provides a frictionless experience for trying 4 | source{d} for Code Analysis. 5 | 6 | 7 | ## Technical Architecture 8 | 9 | The `sourced` binary, a single CLI binary [written in Go](../../cmd/sourced/main.go), 10 | is the user's main interaction mechanism with **source{d} CE**. 11 | It is also the only piece (other than Docker) that the user will need to explicitly 12 | download on their machine to get started. 13 | 14 | The `sourced` binary manages the different installed environments and their 15 | configurations, acting as a wrapper of Docker Compose. 16 | 17 | The whole architecture is based on Docker containers, orchestrated by Docker Compose 18 | and managed by `sourced`. 19 | 20 | 21 | ## Components of source{d} 22 | 23 | **source{d} CE** relies on different components to handle different use cases 24 | and to cover different functionalities. Each component is implemented as a running 25 | Docker container. 26 | 27 | - `bblfsh`: parses source code into UASTs using [Babelfish](https://docs.sourced.tech/babelfish/); 28 | you can learn more about it in our [Babelfish UAST guide](usage/bblfsh.md) 29 | - `gitbase`: runs [gitbase](https://docs.sourced.tech/gitbase), a SQL database 30 | interface to Git repositories. 31 | - `gitcollector`: is responsible for fetching repositories from the organizations 32 | used to initialize **source{d} CE**. It uses [gitcollector](https://github.com/src-d/gitcollector). 33 | - `ghsync`: is responsible for fetching repository metadata from the organizations 34 | used to initialize **source{d} CE**. It uses [ghsync](https://github.com/src-d/ghsync) 35 | - `metadatadb`: runs the PostgreSQL database that stores the repositories 36 | metadata (users, pull requests, issues...) extracted by `ghsync`. 37 | - `postgres`: runs the PostgreSQL database that stores the state of the UI 38 | (charts, dashboards, users, saved queries and such). 39 | - `sourced-ui`: runs the **source{d} CE** Web Interface. This component queries 40 | data from `bblfsh`, `gitbase`, `metadatadb` and `postgres`. 41 | 42 | Some of these components can be accessed from the outside as described by 43 | [Docker Networking section](#docker-networking). 44 | 45 | 46 | ## Docker Set Up 47 | 48 | In order to make this work in the easiest way, some design decisions were made: 49 | 50 | ### Isolated Environments. 51 | 52 | _Read more in [Working With Multiple Data Sets](../usage/multiple-datasets.md)_ 53 | 54 | Each dataset runs in an isolated environment, and only one environment can run 55 | at the same time. 56 | Each environment is defined by one `docker-compose.yml` and one `.env`, stored 57 | in `~/.sourced`. 58 | 59 | ### Docker Naming 60 | 61 | All the Docker containers from the same environment share its prefix: 62 | `srcd-_` followed by the name of the service running inside, e.g 63 | `srcd-c3jjlwq_gitbase_1` and `srcd-c3jjlwq_bblfsh_1` will contain gitbase and 64 | babelfish for the same environment. 65 | 66 | ### Docker Networking 67 | 68 | In order to provide communication between the multiple containers started, all of 69 | them are attached to the same single bridge network. The network name also has 70 | the same prefix than the containers inside the same environment, e.g. 71 | `srcd-c3jjlwq_default`. 72 | 73 | Some environment services can be accessed from the outside, using their exposed 74 | port and connection values: 75 | - `bblfsh`: 76 | - port: `9432` 77 | - `gitbase`: 78 | - port: `3306` 79 | - database: `gitbase` 80 | - user: `root` 81 | - `metadatadb`: 82 | - port: `5433` 83 | - database: `metadata` 84 | - user: `metadata` 85 | - password: `metadata` 86 | - `sourced-ui`: 87 | - port: `8088` 88 | 89 | ### Persistence 90 | 91 | To prevent losing data when restarting services, or upgrading containers, the data 92 | is stored in volumes. These volumes also share the same prefix with the containers 93 | in the same environment, e.g. `srcd-c3jjlwq_gitbase_repositories`. 94 | 95 | These are the most relevant volumes: 96 | - `gitbase_repositories`, stores the repositories to be analyzed 97 | - `gitbase_indexes`, stores the gitbases indexes 98 | - `metadata`, stores the metadata from GitHub pull requests, issues, users... 99 | - `postgres`, stores the dashboards and charts used by the web interface 100 | -------------------------------------------------------------------------------- /cmd/sourced/compose/workdir/workdir.go: -------------------------------------------------------------------------------- 1 | package workdir 2 | 3 | import ( 4 | "encoding/base64" 5 | "fmt" 6 | "io/ioutil" 7 | "os" 8 | "path/filepath" 9 | "strings" 10 | 11 | "github.com/pkg/errors" 12 | goerrors "gopkg.in/src-d/go-errors.v1" 13 | 14 | datadir "github.com/src-d/sourced-ce/cmd/sourced/dir" 15 | ) 16 | 17 | const activeDir = "__active__" 18 | 19 | var ( 20 | // RequiredFiles list of required files in a directory to treat it as a working directory 21 | RequiredFiles = []string{".env", "docker-compose.yml"} 22 | 23 | // ErrMalformed is the returned error when the workdir is wrong 24 | ErrMalformed = goerrors.NewKind("workdir %s is not valid") 25 | 26 | // ErrInitFailed is an error returned on workdir initialization for custom cases 27 | ErrInitFailed = goerrors.NewKind("initialization failed") 28 | ) 29 | 30 | // Type defines the type of the workdir 31 | type Type int 32 | 33 | const ( 34 | // None refers to a failure in identifying the type of the workdir 35 | None Type = iota 36 | // Local refers to a workdir that has been initialized for local repos 37 | Local 38 | // Orgs refers to a workdir that has been initialized for organizations 39 | Orgs 40 | ) 41 | 42 | // Workdir represents a workdir associated with a local or an orgs initialization 43 | type Workdir struct { 44 | // Type is the type of working directory 45 | Type Type 46 | // Name is a human-friendly string to identify the workdir 47 | Name string 48 | // Path is the absolute path corresponding to the workdir 49 | Path string 50 | } 51 | 52 | type builder struct { 53 | workdirsPath string 54 | } 55 | 56 | // build returns the Workdir instance corresponding to the provided absolute path 57 | // the path must be inside `workdirsPath` 58 | func (b *builder) Build(path string) (*Workdir, error) { 59 | wdType, err := b.typeFromPath(path) 60 | if err != nil { 61 | return nil, err 62 | } 63 | 64 | if wdType == None { 65 | return nil, fmt.Errorf("invalid workdir type for path %s", path) 66 | } 67 | 68 | wdName, err := b.workdirName(wdType, path) 69 | if err != nil { 70 | return nil, err 71 | } 72 | 73 | return &Workdir{ 74 | Type: wdType, 75 | Name: wdName, 76 | Path: path, 77 | }, nil 78 | } 79 | 80 | // workdirName returns the workdir name given its type and absolute path 81 | func (b *builder) workdirName(wdType Type, path string) (string, error) { 82 | var subPath string 83 | switch wdType { 84 | case Local: 85 | subPath = "local" 86 | case Orgs: 87 | subPath = "orgs" 88 | } 89 | 90 | encoded, err := filepath.Rel(filepath.Join(b.workdirsPath, subPath), path) 91 | if err != nil { 92 | return "", err 93 | } 94 | 95 | decoded, err := base64.URLEncoding.DecodeString(encoded) 96 | if err == nil { 97 | return string(decoded), nil 98 | } 99 | 100 | return "", err 101 | } 102 | 103 | // typeFromPath returns the workdir type corresponding to the provided absolute path 104 | func (b *builder) typeFromPath(path string) (Type, error) { 105 | suffix, err := filepath.Rel(b.workdirsPath, path) 106 | if err != nil { 107 | return None, err 108 | } 109 | 110 | switch filepath.Dir(suffix) { 111 | case "local": 112 | return Local, nil 113 | case "orgs": 114 | return Orgs, nil 115 | default: 116 | return None, nil 117 | } 118 | } 119 | 120 | func hasContent(path, file string) bool { 121 | empty, err := isEmptyFile(filepath.Join(path, file)) 122 | return !empty && err == nil 123 | } 124 | 125 | // isEmptyFile returns true if the file does not exist or if it exists but 126 | // contains empty text 127 | func isEmptyFile(path string) (bool, error) { 128 | _, err := os.Stat(path) 129 | if err != nil { 130 | if !os.IsNotExist(err) { 131 | return false, err 132 | } 133 | 134 | return true, nil 135 | } 136 | 137 | contents, err := ioutil.ReadFile(path) 138 | if err != nil { 139 | return false, err 140 | } 141 | 142 | strContents := string(contents) 143 | return strings.TrimSpace(strContents) == "", nil 144 | } 145 | 146 | func link(linkTargetPath, linkPath string) error { 147 | _, err := os.Stat(linkPath) 148 | if err == nil { 149 | return nil 150 | } 151 | 152 | if !os.IsNotExist(err) { 153 | return errors.Wrap(err, "could not read the existing FILE_NAME file") 154 | } 155 | 156 | err = os.Symlink(linkTargetPath, linkPath) 157 | return errors.Wrap(err, fmt.Sprintf("could not create symlink to %s", linkTargetPath)) 158 | } 159 | 160 | func workdirsPath() (string, error) { 161 | path, err := datadir.Path() 162 | if err != nil { 163 | return "", err 164 | } 165 | 166 | return filepath.Join(path, "workdirs"), nil 167 | } 168 | -------------------------------------------------------------------------------- /docs/quickstart/4-explore-sourced.md: -------------------------------------------------------------------------------- 1 | # Explore source{d} CE Web Interface 2 | 3 | _If you have any problem running **source{d} CE** you can take a look to our [Troubleshooting](../learn-more/troubleshooting.md) section, and to our [source{d} Forum](https://forum.sourced.tech), where you can also ask for help when using **source{d} CE**. If you spotted a bug, or have a feature request, please [open an issue](https://github.com/src-d/sourced-ce/issues) to let us know abut it._ 4 | 5 | _In some circumstances, loading the data for the dashboards can take some time, and the UI can be frozen in the meanwhile. It can happen —on big datasets—, the first time you access the dashboards, or when they are refreshed. Please, take a look to our 6 | [Troubleshooting](../learn-more/troubleshooting.md#the-dashboard-takes-a-long-to-load-and-the-ui-freezes) 7 | to get more info about this exact issue._ 8 | 9 | Once **source{d} CE** has been [initialized with `sourced init`](./3-init-sourced.md), it will automatically open the web UI. If the UI is not automatically opened, you can use `sourced web` command, or visit http://127.0.0.1:8088. 10 | 11 | Use login: `admin` and password: `admin`, to access the web interface. 12 | 13 | If you [initialized **source{d} CE** from GitHub Organizations](./3-init-sourced.md#from-github-oganizations), its repositories and metadata will be downloaded on background, and it will be available graduatelly. You will find more info in the welcome dashboard once you log in. 14 | 15 | 16 | ## Sections 17 | 18 | The most relevant features that **source{d} CE** Web Interface offers are: 19 | - **[SQL Lab](#sql-lab-querying-code-and-metadata)**, to query your repositories and its GitHub metadata. 20 | - **[Babelfish web](#uast-parsing-code)**, web interface to parse files into UAST. 21 | - **[Dashboards](#dashboards)**, to aggregate charts for exploring and visualizing your data. 22 | - **Charts**, to see your data with a rich set of data visualizations. 23 | - A flexible UI to manage users, data sources, export data... 24 | 25 | The user interface is based in the open-sourced [Apache Superset](http://superset.incubator.apache.org), so you can also refer to their documentation for advanced usage of the web interface. 26 | 27 | 28 | ## SQL Lab. Querying Code and Metadata 29 | 30 | _If you prefer to work within the terminal via command line, you can open a SQL REPL running `sourced sql`_ 31 | 32 | Using the `SQL Lab` tab, from the web interface, you can analyze your dataset using SQL queries, and create charts from those queries with the `Explore` button. 33 | 34 | You can find some sample queries in the [examples](../usage/examples.md). 35 | 36 | If you want to know what the database schema looks like you can use either regular `SHOW` or `DESCRIBE` queries, or you can refer to the [diagram about gitbase entities and relations](https://docs.sourced.tech/gitbase/using-gitbase/schema#database-diagram). 37 | 38 | ```bash 39 | $ sourced sql "SHOW tables;" 40 | +--------------+ 41 | | TABLE | 42 | +--------------+ 43 | | blobs | 44 | | commit_blobs | 45 | | commit_files | 46 | | commit_trees | 47 | | commits | 48 | | files | 49 | | ref_commits | 50 | | refs | 51 | | remotes | 52 | | repositories | 53 | | tree_entries | 54 | +--------------+ 55 | ``` 56 | 57 | ```bash 58 | $ sourced sql "DESCRIBE TABLE commits;" 59 | +---------------------+-----------+ 60 | | NAME | TYPE | 61 | +---------------------+-----------+ 62 | | repository_id | TEXT | 63 | | commit_hash | TEXT | 64 | | commit_author_name | TEXT | 65 | | commit_author_email | TEXT | 66 | | commit_author_when | TIMESTAMP | 67 | | committer_name | TEXT | 68 | | committer_email | TEXT | 69 | | committer_when | TIMESTAMP | 70 | | commit_message | TEXT | 71 | | tree_hash | TEXT | 72 | | commit_parents | JSON | 73 | +---------------------+-----------+ 74 | ``` 75 | 76 | 77 | ## UAST. Parsing code 78 | 79 | _Please, refer to the [quick explanation about what Babelfish is](../usage/bblfsh.md) to know more about it._ 80 | 81 | You can get UASTs from the `UAST` tab (parsing files by direct input), or using the `UAST` gitbase function over blob contents on `SQL Lab` tab. 82 | 83 | 84 | ## Dashboards 85 | 86 | _Please, refer to [Superset Tutorial, creating your first dashboard](http://superset.incubator.apache.org/tutorial.html) for more details._ 87 | 88 | The dashboards let you aggregate custom charts to show in the same place different metrics for your repositories. 89 | 90 | You can create them: 91 | - From the `Dashboard` tab, adding a new one, and then selecting new charts. 92 | - From any chart view, the `Save` button will let you to add it into a new or existent one. 93 | -------------------------------------------------------------------------------- /test/superset.go: -------------------------------------------------------------------------------- 1 | // +build integration 2 | 3 | package test 4 | 5 | import ( 6 | "bytes" 7 | "encoding/json" 8 | "fmt" 9 | "io/ioutil" 10 | "log" 11 | "net/http" 12 | "net/http/cookiejar" 13 | "net/url" 14 | 15 | "github.com/PuerkitoBio/goquery" 16 | "golang.org/x/net/publicsuffix" 17 | ) 18 | 19 | type supersetClient struct { 20 | *http.Client 21 | csrf string 22 | } 23 | 24 | func newSupersetClient() (*supersetClient, error) { 25 | // Superset client needs a session cookie 26 | jar, err := cookiejar.New(&cookiejar.Options{PublicSuffixList: publicsuffix.List}) 27 | if err != nil { 28 | return nil, err 29 | } 30 | 31 | client := &http.Client{ 32 | Jar: jar, 33 | } 34 | 35 | // To POST in /login we need to first GET /login, read the hidden CSRF value 36 | // from the HTML, and send it back in the POST 37 | res, err := client.Get("http://127.0.0.1:8088/login/") 38 | if err != nil { 39 | return nil, err 40 | } 41 | 42 | doc, err := goquery.NewDocumentFromReader(res.Body) 43 | if err != nil { 44 | log.Fatal(err) 45 | } 46 | 47 | csrf, ok := doc.Find("input#csrf_token").First().Attr("value") 48 | if !ok { 49 | return nil, fmt.Errorf("CSRF token was not found in the /login page") 50 | } 51 | 52 | res, err = client.PostForm("http://127.0.0.1:8088/login/", url.Values{ 53 | "csrf_token": {csrf}, 54 | "username": {"admin"}, 55 | "password": {"admin"}, 56 | }) 57 | if err != nil { 58 | return nil, err 59 | } 60 | 61 | // After a successful POST the client is authenticated and can be used to call 62 | // the API 63 | return &supersetClient{client, csrf}, nil 64 | } 65 | 66 | func (c *supersetClient) dashboards() ([]string, error) { 67 | res, err := c.Get("http://127.0.0.1:8088/dashboard/api/read") 68 | if err != nil { 69 | return nil, err 70 | } 71 | 72 | var decoded struct { 73 | Result []struct { 74 | Link string `json:"dashboard_link"` 75 | } 76 | } 77 | 78 | err = json.NewDecoder(res.Body).Decode(&decoded) 79 | if err != nil { 80 | return nil, err 81 | } 82 | 83 | links := []string{} 84 | for _, result := range decoded.Result { 85 | links = append(links, result.Link) 86 | } 87 | 88 | return links, nil 89 | } 90 | 91 | func (c *supersetClient) sql(query, dbId, schema string) ([]map[string]interface{}, error) { 92 | res, err := c.PostForm("http://127.0.0.1:8088/superset/sql_json/", url.Values{ 93 | //"client_id": {""}, 94 | "database_id": {dbId}, 95 | "json": {"true"}, 96 | "runAsync": {"false"}, 97 | "schema": {schema}, 98 | "sql": {query}, 99 | //"sql_editor_id": {"jzl0KCm5Z"}, 100 | //"tab": {"Untitled Query 2"}, 101 | //"tmp_table_name": {""}, 102 | //"select_as_cta": {"false"}, 103 | //"templateParams": {"{}"}, 104 | //"queryLimit": {"1000"}, 105 | 106 | "csrf_token": {c.csrf}, 107 | }) 108 | if err != nil { 109 | return nil, err 110 | } 111 | 112 | var decoded struct { 113 | Status string 114 | Error string 115 | Data []map[string]interface{} 116 | } 117 | 118 | err = json.NewDecoder(res.Body).Decode(&decoded) 119 | if err != nil { 120 | return nil, err 121 | } 122 | 123 | if decoded.Status != "success" { 124 | return nil, fmt.Errorf("/sql_json endpoint returned an error: " + decoded.Error) 125 | } 126 | 127 | return decoded.Data, nil 128 | } 129 | 130 | func (c *supersetClient) gitbase(query string) ([]map[string]interface{}, error) { 131 | return c.sql(query, "1", "gitbase") 132 | } 133 | 134 | func (c *supersetClient) metadata(query string) ([]map[string]interface{}, error) { 135 | return c.sql(query, "2", "public") 136 | } 137 | 138 | func (c *supersetClient) bblfsh(filename, content string) (string, error) { 139 | var jsonStr = []byte(fmt.Sprintf( 140 | `{"mode":"semantic", "filename":%q, "content":%q, "query":""}`, 141 | filename, content)) 142 | 143 | req, err := http.NewRequest("POST", "http://127.0.0.1:8088/bblfsh/api/parse", bytes.NewBuffer(jsonStr)) 144 | if err != nil { 145 | return "", err 146 | } 147 | req.Header.Set("Content-Type", "application/json") 148 | 149 | res, err := c.Do(req) 150 | if err != nil { 151 | return "", err 152 | } 153 | 154 | var decoded struct { 155 | Status int 156 | Language string 157 | // Uast interface{} 158 | Errors []struct{ Message string } 159 | } 160 | 161 | err = json.NewDecoder(res.Body).Decode(&decoded) 162 | if err != nil { 163 | body := "" 164 | bytes, readErr := ioutil.ReadAll(res.Body) 165 | if readErr != nil { 166 | body = string(bytes) 167 | } 168 | 169 | return "", fmt.Errorf("could not decode the response body: %s, err: %s", body, err) 170 | } 171 | 172 | if decoded.Status != 0 { 173 | return "", fmt.Errorf("/bblfsh/api/parse endpoint returned an error: %v", decoded.Errors) 174 | } 175 | 176 | return decoded.Language, nil 177 | } 178 | -------------------------------------------------------------------------------- /cmd/sourced/compose/workdir/handler.go: -------------------------------------------------------------------------------- 1 | package workdir 2 | 3 | import ( 4 | "fmt" 5 | "io/ioutil" 6 | "os" 7 | "path/filepath" 8 | 9 | "github.com/pkg/errors" 10 | ) 11 | 12 | // Handler provides a way to interact with all the workdirs by exposing the following operations: 13 | // - read/set/unset active workdir, 14 | // - remove/validate a workdir, 15 | // - list workdirs. 16 | type Handler struct { 17 | workdirsPath string 18 | builder *builder 19 | } 20 | 21 | // NewHandler creates a handler that manages workdirs in the path returned by 22 | // the `workdirsPath` function 23 | func NewHandler() (*Handler, error) { 24 | path, err := workdirsPath() 25 | if err != nil { 26 | return nil, err 27 | } 28 | 29 | return &Handler{ 30 | workdirsPath: path, 31 | builder: &builder{workdirsPath: path}, 32 | }, nil 33 | } 34 | 35 | // SetActive creates a symlink from the fixed active workdir path to the prodived workdir 36 | func (h *Handler) SetActive(w *Workdir) error { 37 | path := h.activeAbsolutePath() 38 | 39 | if err := h.UnsetActive(); err != nil { 40 | return err 41 | } 42 | 43 | err := os.Symlink(w.Path, path) 44 | if os.IsExist(err) { 45 | return nil 46 | } 47 | 48 | return err 49 | } 50 | 51 | // UnsetActive removes symlink for active workdir 52 | func (h *Handler) UnsetActive() error { 53 | path := h.activeAbsolutePath() 54 | 55 | _, err := os.Lstat(path) 56 | if !os.IsNotExist(err) { 57 | err = os.Remove(path) 58 | if err != nil { 59 | return errors.Wrap(err, "could not delete the previous active workdir directory symlink") 60 | } 61 | } 62 | 63 | return nil 64 | } 65 | 66 | // Active returns active working directory 67 | func (h *Handler) Active() (*Workdir, error) { 68 | path := h.activeAbsolutePath() 69 | 70 | resolvedPath, err := filepath.EvalSymlinks(path) 71 | if os.IsNotExist(err) { 72 | return nil, ErrMalformed.Wrap(err, "active") 73 | } 74 | 75 | return h.builder.Build(resolvedPath) 76 | } 77 | 78 | // List returns array of working directories 79 | func (h *Handler) List() ([]*Workdir, error) { 80 | dirs := make([]string, 0) 81 | err := filepath.Walk(h.workdirsPath, func(path string, info os.FileInfo, err error) error { 82 | if err != nil { 83 | return err 84 | } 85 | if !info.IsDir() { 86 | return nil 87 | } 88 | for _, f := range RequiredFiles { 89 | if !hasContent(path, f) { 90 | return nil 91 | } 92 | } 93 | 94 | dirs = append(dirs, path) 95 | return nil 96 | }) 97 | 98 | if os.IsNotExist(err) { 99 | return nil, ErrMalformed.Wrap(err, h.workdirsPath) 100 | } 101 | 102 | if err != nil { 103 | return nil, err 104 | } 105 | 106 | wds := make([]*Workdir, 0, len(dirs)) 107 | for _, p := range dirs { 108 | wd, err := h.builder.Build(p) 109 | if err != nil { 110 | return nil, err 111 | } 112 | 113 | wds = append(wds, wd) 114 | } 115 | 116 | return wds, nil 117 | 118 | } 119 | 120 | // Validate validates that the passed working directoy is valid 121 | // It's path must be a directory (or a symlink) containing docker-compose.yml and .env files 122 | func (h *Handler) Validate(w *Workdir) error { 123 | pointedDir, err := filepath.EvalSymlinks(w.Path) 124 | if err != nil { 125 | return ErrMalformed.Wrap(fmt.Errorf("is not a directory"), w.Path) 126 | } 127 | 128 | if info, err := os.Lstat(pointedDir); err != nil || !info.IsDir() { 129 | return ErrMalformed.Wrap(fmt.Errorf("is not a directory"), pointedDir) 130 | } 131 | 132 | for _, f := range RequiredFiles { 133 | if !hasContent(pointedDir, f) { 134 | return ErrMalformed.Wrap(fmt.Errorf("%s not found", f), pointedDir) 135 | } 136 | } 137 | 138 | return nil 139 | } 140 | 141 | // Remove removes working directory by removing required and optional files, 142 | // and recursively removes directories up to the workdirs root as long as they are empty 143 | func (h *Handler) Remove(w *Workdir) error { 144 | path := w.Path 145 | var subPath string 146 | switch w.Type { 147 | case Local: 148 | subPath = "local" 149 | case Orgs: 150 | subPath = "orgs" 151 | } 152 | 153 | basePath := filepath.Join(h.workdirsPath, subPath) 154 | 155 | for _, f := range RequiredFiles { 156 | file := filepath.Join(path, f) 157 | if _, err := os.Stat(file); os.IsNotExist(err) { 158 | continue 159 | } 160 | 161 | if err := os.Remove(file); err != nil { 162 | return errors.Wrap(err, "could not remove from workdir directory") 163 | } 164 | } 165 | 166 | for { 167 | files, err := ioutil.ReadDir(path) 168 | if err != nil { 169 | return errors.Wrap(err, "could not read workdir directory") 170 | } 171 | if len(files) > 0 { 172 | return nil 173 | } 174 | 175 | if err := os.Remove(path); err != nil { 176 | return errors.Wrap(err, "could not delete workdir directory") 177 | } 178 | 179 | path = filepath.Dir(path) 180 | if path == basePath { 181 | return nil 182 | } 183 | } 184 | } 185 | 186 | func (h *Handler) activeAbsolutePath() string { 187 | return filepath.Join(h.workdirsPath, activeDir) 188 | } 189 | -------------------------------------------------------------------------------- /test/init_orgs_test.go: -------------------------------------------------------------------------------- 1 | // +build integration 2 | 3 | package test 4 | 5 | import ( 6 | "database/sql" 7 | "os" 8 | "testing" 9 | "time" 10 | 11 | _ "github.com/lib/pq" 12 | "github.com/stretchr/testify/require" 13 | "github.com/stretchr/testify/suite" 14 | ) 15 | 16 | type InitOrgsTestSuite struct { 17 | IntegrationSuite 18 | } 19 | 20 | func TestInitOrgsTestSuite(t *testing.T) { 21 | itt := InitOrgsTestSuite{} 22 | 23 | if os.Getenv("SOURCED_GITHUB_TOKEN") == "" { 24 | t.Skip("SOURCED_GITHUB_TOKEN is not set") 25 | return 26 | } 27 | 28 | suite.Run(t, &itt) 29 | } 30 | 31 | func checkGhsync(require *require.Assertions, repos int) { 32 | connStr := "user=metadata password=metadata dbname=metadata port=5433 sslmode=disable" 33 | db, err := sql.Open("postgres", connStr) 34 | require.NoError(err) 35 | defer db.Close() 36 | 37 | var id int 38 | var org, entity string 39 | var done, failed, total int 40 | 41 | // try for 2 minutes 42 | for i := 0; i < 24; i++ { 43 | time.Sleep(5 * time.Second) 44 | 45 | row := db.QueryRow("SELECT * FROM status") 46 | err = row.Scan(&id, &org, &entity, &done, &failed, &total) 47 | if err == sql.ErrNoRows { 48 | continue 49 | } 50 | require.NoError(err) 51 | 52 | if done == repos { 53 | break 54 | } 55 | } 56 | 57 | require.Equal(repos, done, 58 | "id = %v, org = %v, entity = %v, done = %v, failed = %v, total = %v", 59 | id, org, entity, done, failed, total) 60 | } 61 | 62 | func checkGitcollector(require *require.Assertions, repos int) { 63 | connStr := "user=superset password=superset dbname=superset port=5432 sslmode=disable" 64 | db, err := sql.Open("postgres", connStr) 65 | require.NoError(err) 66 | defer db.Close() 67 | 68 | var org string 69 | var discovered, downloaded, updated, failed int 70 | 71 | // try for 2 minutes 72 | for i := 0; i < 24; i++ { 73 | time.Sleep(5 * time.Second) 74 | 75 | row := db.QueryRow("SELECT * FROM gitcollector_metrics") 76 | err = row.Scan(&org, &discovered, &downloaded, &updated, &failed) 77 | if err == sql.ErrNoRows { 78 | continue 79 | } 80 | require.NoError(err) 81 | 82 | if downloaded == repos { 83 | break 84 | } 85 | } 86 | 87 | require.Equal(repos, downloaded, 88 | "org = %v, discovered = %v, downloaded = %v, updated = %v, failed = %v", 89 | org, discovered, downloaded, updated, failed) 90 | } 91 | 92 | func (s *InitOrgsTestSuite) TestOneOrg() { 93 | req := s.Require() 94 | 95 | // TODO will need to change with https://github.com/src-d/sourced-ce/issues/144 96 | r := s.RunCommand("status", "workdirs") 97 | req.Error(r.Error) 98 | 99 | r = s.RunCommand("init", "orgs", "golang-migrate") 100 | req.NoError(r.Error, r.Combined()) 101 | 102 | r = s.RunCommand("status", "workdirs") 103 | req.NoError(r.Error, r.Combined()) 104 | 105 | req.Equal("* golang-migrate\n", r.Stdout()) 106 | 107 | checkGhsync(req, 1) 108 | checkGitcollector(req, 1) 109 | 110 | // Check gitbase can also see the repositories 111 | r = s.RunCommand("sql", "select * from repositories where repository_id='github.com/golang-migrate/migrate'") 112 | req.NoError(r.Error, r.Combined()) 113 | 114 | req.Contains(r.Stdout(), 115 | `repository_id 116 | github.com/golang-migrate/migrate 117 | `) 118 | 119 | // Test SQL queries. This should be a different test, but since starting 120 | // the environment takes a long time, it is bundled together here to speed up 121 | // the tests 122 | s.testSQL() 123 | 124 | client, err := newSupersetClient() 125 | req.NoError(err) 126 | 127 | // Test the list of dashboards created in superset 128 | s.T().Run("dashboard-list", func(t *testing.T) { 129 | req := require.New(t) 130 | 131 | links, err := client.dashboards() 132 | req.NoError(err) 133 | 134 | s.Equal([]string{ 135 | `Overview`, 136 | `Welcome`, 137 | `Collaboration`, 138 | }, links) 139 | }) 140 | 141 | // Test gitbase queries through superset 142 | s.T().Run("superset-gitbase", func(t *testing.T) { 143 | req := require.New(t) 144 | 145 | rows, err := client.gitbase("select * from repositories") 146 | req.NoError(err) 147 | 148 | s.Equal([]map[string]interface{}{ 149 | {"repository_id": "github.com/golang-migrate/migrate"}, 150 | }, rows) 151 | }) 152 | 153 | // Test metadata queries through superset 154 | s.T().Run("superset-metadata", func(t *testing.T) { 155 | req := require.New(t) 156 | 157 | rows, err := client.metadata("select * from organizations") 158 | req.NoError(err) 159 | req.Len(rows, 1) 160 | 161 | s.Equal("golang-migrate", rows[0]["login"]) 162 | }) 163 | 164 | // Test bblfsh queries through superset 165 | s.T().Run("superset-bblfsh", func(t *testing.T) { 166 | req := require.New(t) 167 | 168 | lang, err := client.bblfsh("hello.js", `console.log("hello");`) 169 | req.NoError(err) 170 | req.Equal("javascript", lang) 171 | }) 172 | 173 | // Test gitbase can connect to bblfsh with a SQL query that uses UAST 174 | s.T().Run("gitbase-bblfsh", func(t *testing.T) { 175 | req := require.New(t) 176 | 177 | rows, err := client.gitbase( 178 | `SELECT UAST('console.log("hello");', 'javascript') AS uast`) 179 | req.NoError(err) 180 | 181 | req.Len(rows, 1) 182 | req.NotEmpty(rows[0]["uast"]) 183 | }) 184 | } 185 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3.4' 2 | 3 | x-superset-env: &superset-env 4 | SYNC_MODE: ${GITBASE_SIVA} 5 | ADMIN_LOGIN: admin 6 | ADMIN_FIRST_NAME: admin 7 | ADMIN_LAST_NAME: admin 8 | ADMIN_EMAIL: admin@example.com 9 | ADMIN_PASSWORD: admin 10 | POSTGRES_DB: superset 11 | POSTGRES_USER: superset 12 | POSTGRES_PASSWORD: superset 13 | POSTGRES_HOST: postgres 14 | POSTGRES_PORT: 5432 15 | REDIS_HOST: redis 16 | REDIS_PORT: 6379 17 | GITBASE_DB: gitbase 18 | GITBASE_USER: root 19 | GITBASE_PASSWORD: 20 | GITBASE_HOST: gitbase 21 | GITBASE_PORT: 3306 22 | METADATA_DB: metadata 23 | METADATA_USER: metadata 24 | METADATA_PASSWORD: metadata 25 | METADATA_HOST: metadatadb 26 | METADATA_PORT: 5432 27 | BBLFSH_WEB_HOST: bblfsh-web 28 | BBLFSH_WEB_PORT: 8080 29 | 30 | services: 31 | bblfsh: 32 | image: bblfsh/bblfshd:v2.15.0-drivers 33 | restart: unless-stopped 34 | privileged: true 35 | ports: 36 | - 9432:9432 37 | 38 | gitcollector: 39 | image: srcd/gitcollector:v0.0.4 40 | # wait for db 41 | command: ['/bin/sh', '-c', 'sleep 10s && gitcollector download'] 42 | environment: 43 | GITHUB_ORGANIZATIONS: ${GITHUB_ORGANIZATIONS-} 44 | GITHUB_TOKEN: ${GITHUB_TOKEN-} 45 | # use main db 46 | GITCOLLECTOR_METRICS_DB_URI: postgresql://superset:superset@postgres:5432/superset?sslmode=disable 47 | GITCOLLECTOR_NO_UPDATES: 'true' 48 | GITCOLLECTOR_NO_FORKS: ${NO_FORKS-true} 49 | LOG_LEVEL: ${LOG_LEVEL-info} 50 | depends_on: 51 | - postgres 52 | volumes: 53 | - type: ${GITBASE_VOLUME_TYPE} 54 | source: ${GITBASE_VOLUME_SOURCE} 55 | target: /library 56 | consistency: delegated 57 | deploy: 58 | resources: 59 | limits: 60 | cpus: ${GITCOLLECTOR_LIMIT_CPU-0.0} 61 | 62 | ghsync: 63 | image: srcd/ghsync:v0.2.0 64 | entrypoint: ['/bin/sh'] 65 | # wait for db to be created 66 | # we need to use something like https://github.com/vishnubob/wait-for-it 67 | # or implement wait in ghsync itself 68 | command: ['-c', 'sleep 10s && ghsync migrate && ghsync shallow'] 69 | depends_on: 70 | - metadatadb 71 | environment: 72 | GHSYNC_ORGS: ${GITHUB_ORGANIZATIONS-} 73 | GHSYNC_TOKEN: ${GITHUB_TOKEN-} 74 | GHSYNC_POSTGRES_DB: metadata 75 | GHSYNC_POSTGRES_USER: metadata 76 | GHSYNC_POSTGRES_PASSWORD: metadata 77 | GHSYNC_POSTGRES_HOST: metadatadb 78 | GHSYNC_POSTGRES_PORT: 5432 79 | GHSYNC_NO_FORKS: ${NO_FORKS-true} 80 | LOG_LEVEL: ${LOG_LEVEL-info} 81 | 82 | gitbase: 83 | image: srcd/gitbase:v0.23.1 84 | restart: unless-stopped 85 | ports: 86 | - 3306:3306 87 | environment: 88 | BBLFSH_ENDPOINT: bblfsh:9432 89 | SIVA: ${GITBASE_SIVA} 90 | GITBASE_LOG_LEVEL: ${LOG_LEVEL-info} 91 | depends_on: 92 | - bblfsh 93 | volumes: 94 | - type: ${GITBASE_VOLUME_TYPE} 95 | source: ${GITBASE_VOLUME_SOURCE} 96 | target: /opt/repos 97 | read_only: true 98 | consistency: delegated 99 | - gitbase_indexes:/var/lib/gitbase/index 100 | deploy: 101 | resources: 102 | limits: 103 | cpus: ${GITBASE_LIMIT_CPU-0.0} 104 | memory: ${GITBASE_LIMIT_MEM-0} 105 | 106 | bblfsh-web: 107 | image: bblfsh/web:v0.11.4 108 | restart: unless-stopped 109 | command: -bblfsh-addr bblfsh:9432 110 | ports: 111 | - 9999:8080 112 | depends_on: 113 | - bblfsh 114 | environment: 115 | LOG_LEVEL: ${LOG_LEVEL-info} 116 | 117 | redis: 118 | image: redis:5-alpine 119 | restart: unless-stopped 120 | ports: 121 | - 6379:6379 122 | volumes: 123 | - redis:/data 124 | 125 | postgres: 126 | image: postgres:10-alpine 127 | restart: unless-stopped 128 | environment: 129 | POSTGRES_DB: superset 130 | POSTGRES_PASSWORD: superset 131 | POSTGRES_USER: superset 132 | ports: 133 | - 5432:5432 134 | volumes: 135 | - postgres:/var/lib/postgresql/data 136 | 137 | metadatadb: 138 | image: postgres:10-alpine 139 | restart: unless-stopped 140 | environment: 141 | POSTGRES_DB: metadata 142 | POSTGRES_PASSWORD: metadata 143 | POSTGRES_USER: metadata 144 | ports: 145 | - 5433:5432 146 | volumes: 147 | - metadata:/var/lib/postgresql/data 148 | 149 | sourced-ui: 150 | image: srcd/sourced-ui:v0.8.1 151 | restart: unless-stopped 152 | environment: 153 | <<: *superset-env 154 | SUPERSET_ENV: production 155 | ports: 156 | - 8088:8088 157 | depends_on: 158 | - postgres 159 | - metadatadb 160 | - redis 161 | - gitbase 162 | - bblfsh-web 163 | 164 | sourced-ui-celery: 165 | image: srcd/sourced-ui:v0.8.1 166 | restart: unless-stopped 167 | environment: 168 | <<: *superset-env 169 | SUPERSET_ENV: celery 170 | depends_on: 171 | - postgres 172 | - metadatadb 173 | - redis 174 | - gitbase 175 | - sourced-ui 176 | 177 | volumes: 178 | gitbase_repositories: 179 | external: false 180 | gitbase_indexes: 181 | external: false 182 | metadata: 183 | external: false 184 | postgres: 185 | external: false 186 | redis: 187 | external: false 188 | -------------------------------------------------------------------------------- /cmd/sourced/cmd/init.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "net/http" 7 | "os" 8 | "path/filepath" 9 | "strings" 10 | "time" 11 | 12 | "github.com/src-d/sourced-ce/cmd/sourced/compose" 13 | "github.com/src-d/sourced-ce/cmd/sourced/compose/workdir" 14 | "gopkg.in/src-d/go-cli.v0" 15 | 16 | "github.com/pkg/errors" 17 | ) 18 | 19 | type initCmd struct { 20 | cli.PlainCommand `name:"init" short-description:"Initialize source{d} to work on local or GitHub orgs datasets" long-description:"Initialize source{d} to work on local or Github orgs datasets"` 21 | } 22 | 23 | type initLocalCmd struct { 24 | Command `name:"local" short-description:"Initialize source{d} to analyze local repositories" long-description:"Install, initialize, and start all the required docker containers, networks, volumes, and images.\n\nThe repos directory argument must point to a directory containing git repositories.\nIf it's not provided, the current working directory will be used."` 25 | 26 | Args struct { 27 | Reposdir string `positional-arg-name:"workdir"` 28 | } `positional-args:"yes"` 29 | } 30 | 31 | func (c *initLocalCmd) Execute(args []string) error { 32 | wdHandler, err := workdir.NewHandler() 33 | if err != nil { 34 | return err 35 | } 36 | 37 | reposdir, err := c.reposdirArg() 38 | if err != nil { 39 | return err 40 | } 41 | 42 | wd, err := workdir.InitLocal(reposdir) 43 | if err != nil { 44 | return err 45 | } 46 | 47 | if err := activate(wdHandler, wd); err != nil { 48 | return err 49 | } 50 | 51 | return OpenUI(60 * time.Minute) 52 | } 53 | 54 | func (c *initLocalCmd) reposdirArg() (string, error) { 55 | reposdir := c.Args.Reposdir 56 | reposdir = strings.TrimSpace(reposdir) 57 | 58 | var err error 59 | if reposdir == "" { 60 | reposdir, err = os.Getwd() 61 | } else { 62 | reposdir, err = filepath.Abs(reposdir) 63 | } 64 | 65 | if err != nil { 66 | return "", errors.Wrap(err, "could not get directory") 67 | } 68 | 69 | info, err := os.Stat(reposdir) 70 | if err != nil || !info.IsDir() { 71 | return "", fmt.Errorf("path '%s' is not a valid directory", reposdir) 72 | } 73 | 74 | return reposdir, nil 75 | } 76 | 77 | type initOrgsCmd struct { 78 | Command `name:"orgs" short-description:"Initialize source{d} to analyze GitHub organizations" long-description:"Install, initialize, and start all the required docker containers, networks, volumes, and images.\n\nThe orgs argument must a comma-separated list of GitHub organization names to be analyzed."` 79 | 80 | Token string `short:"t" long:"token" env:"SOURCED_GITHUB_TOKEN" description:"GitHub token for the passed organizations. It should be granted with 'repo' and 'read:org' scopes." required:"true"` 81 | WithForks bool `long:"with-forks" description:"Download GitHub forked repositories"` 82 | Args struct { 83 | Orgs []string `required:"yes"` 84 | } `positional-args:"yes" required:"1"` 85 | } 86 | 87 | func (c *initOrgsCmd) Execute(args []string) error { 88 | wdHandler, err := workdir.NewHandler() 89 | if err != nil { 90 | return err 91 | } 92 | 93 | orgs := c.orgsList() 94 | if err := c.validate(orgs); err != nil { 95 | return err 96 | } 97 | 98 | wd, err := workdir.InitOrgs(orgs, c.Token, c.WithForks) 99 | if err != nil { 100 | return err 101 | } 102 | 103 | if err := activate(wdHandler, wd); err != nil { 104 | return err 105 | } 106 | 107 | return OpenUI(60 * time.Minute) 108 | } 109 | 110 | // allows to pass organizations separated not only by a space 111 | // but by comma as well 112 | func (c *initOrgsCmd) orgsList() []string { 113 | orgs := c.Args.Orgs 114 | if len(c.Args.Orgs) == 1 { 115 | orgs = strings.Split(c.Args.Orgs[0], ",") 116 | } 117 | 118 | for i, org := range orgs { 119 | orgs[i] = strings.Trim(org, " ,") 120 | } 121 | 122 | return orgs 123 | } 124 | 125 | func (c *initOrgsCmd) validate(orgs []string) error { 126 | client := &http.Client{Transport: &authTransport{token: c.Token}} 127 | r, err := client.Get("https://api.github.com/user") 128 | if err != nil { 129 | return errors.Wrapf(err, "could not validate user token") 130 | } 131 | if r.StatusCode == http.StatusUnauthorized { 132 | return fmt.Errorf("github token is not valid") 133 | } 134 | 135 | for _, org := range orgs { 136 | r, err := client.Get("https://api.github.com/orgs/" + org) 137 | if err != nil { 138 | return errors.Wrapf(err, "could not validate organization") 139 | } 140 | if r.StatusCode == http.StatusNotFound { 141 | return fmt.Errorf("organization '%s' is not found", org) 142 | } 143 | } 144 | 145 | return nil 146 | } 147 | 148 | func activate(wdHandler *workdir.Handler, workdir *workdir.Workdir) error { 149 | // Before setting a new workdir, stop the current containers 150 | compose.Run(context.Background(), "stop") 151 | 152 | err := wdHandler.SetActive(workdir) 153 | if err != nil { 154 | return err 155 | } 156 | 157 | fmt.Printf("docker-compose working directory set to %s\n", workdir.Path) 158 | return compose.Run(context.Background(), "up", "--detach") 159 | } 160 | 161 | type authTransport struct { 162 | token string 163 | } 164 | 165 | func (t *authTransport) RoundTrip(r *http.Request) (*http.Response, error) { 166 | r.Header.Set("Authorization", "token "+t.token) 167 | return http.DefaultTransport.RoundTrip(r) 168 | } 169 | 170 | func init() { 171 | c := rootCmd.AddCommand(&initCmd{}) 172 | 173 | c.AddCommand(&initOrgsCmd{}) 174 | c.AddCommand(&initLocalCmd{}) 175 | } 176 | -------------------------------------------------------------------------------- /cmd/sourced/compose/workdir/handler_test.go: -------------------------------------------------------------------------------- 1 | package workdir 2 | 3 | import ( 4 | "os" 5 | "path" 6 | "path/filepath" 7 | "testing" 8 | 9 | "github.com/stretchr/testify/suite" 10 | ) 11 | 12 | type HandlerSuite struct { 13 | suite.Suite 14 | 15 | h *Handler 16 | originSrcdDir string 17 | } 18 | 19 | func TestHandlerSuite(t *testing.T) { 20 | suite.Run(t, &HandlerSuite{}) 21 | } 22 | 23 | func (s *HandlerSuite) BeforeTest(suiteName, testName string) { 24 | s.originSrcdDir = os.Getenv("SOURCED_DIR") 25 | 26 | // on macOs os.TempDir returns symlink and tests fails 27 | tmpDir, _ := filepath.EvalSymlinks(os.TempDir()) 28 | srdPath := path.Join(tmpDir, testName) 29 | err := os.MkdirAll(srdPath, os.ModePerm) 30 | s.Nil(err) 31 | 32 | os.Setenv("SOURCED_DIR", srdPath) 33 | 34 | s.h, err = NewHandler() 35 | s.Nil(err) 36 | } 37 | 38 | func (s *HandlerSuite) AfterTest(suiteName, testName string) { 39 | os.RemoveAll(filepath.Dir(s.h.workdirsPath)) 40 | os.Setenv("SOURCED_DIR", s.originSrcdDir) 41 | } 42 | 43 | // This tests only public interface without checking implementation (filesystem) details 44 | func (s *HandlerSuite) TestSuccessFlow() { 45 | wd := s.createWd("flow") 46 | 47 | s.Nil(s.h.Validate(wd)) 48 | s.Nil(s.h.SetActive(wd)) 49 | 50 | active, err := s.h.Active() 51 | s.Nil(err) 52 | s.Equal(wd, active) 53 | 54 | s.Nil(s.h.UnsetActive()) 55 | 56 | _, err = s.h.Active() 57 | s.True(ErrMalformed.Is(err)) 58 | 59 | wds, err := s.h.List() 60 | s.Nil(err) 61 | s.Len(wds, 1) 62 | s.Equal(wd, wds[0]) 63 | 64 | s.Nil(s.h.Remove(wd)) 65 | 66 | wds, err = s.h.List() 67 | s.Nil(err) 68 | s.Len(wds, 0) 69 | } 70 | 71 | // All tests below rely on implementation details to check error cases 72 | 73 | func (s *HandlerSuite) TestSetActiveOk() { 74 | wd := s.createWd("some") 75 | 76 | // non-active before 77 | s.Nil(s.h.SetActive(wd)) 78 | // re-activation should also work 79 | s.Nil(s.h.SetActive(wd)) 80 | 81 | // validate link points correctly 82 | target, err := filepath.EvalSymlinks(path.Join(s.h.workdirsPath, activeDir)) 83 | s.Nil(err) 84 | s.Equal(wd.Path, target) 85 | } 86 | 87 | func (s *HandlerSuite) TestSetActiveError() { 88 | wd := s.createWd("some") 89 | 90 | // break active path by making it dir with files 91 | activePath := path.Join(s.h.workdirsPath, activeDir) 92 | s.Nil(os.MkdirAll(activePath, os.ModePerm)) 93 | _, err := os.Create(path.Join(activePath, "some-file")) 94 | s.Nil(err) 95 | 96 | s.Error(s.h.SetActive(wd)) 97 | } 98 | 99 | func (s *HandlerSuite) TestUnsetActiveOk() { 100 | activePath := path.Join(s.h.workdirsPath, activeDir) 101 | s.Nil(os.MkdirAll(s.h.workdirsPath, os.ModePerm)) 102 | _, err := os.Create(activePath) 103 | s.Nil(err) 104 | 105 | s.Nil(s.h.UnsetActive()) 106 | // unset without active dir 107 | s.Nil(s.h.UnsetActive()) 108 | 109 | // validate we deleted the file 110 | _, err = os.Stat(activePath) 111 | s.True(os.IsNotExist(err)) 112 | } 113 | 114 | func (s *HandlerSuite) TestUnsetActiveError() { 115 | // break active path by making it dir with files 116 | activePath := path.Join(s.h.workdirsPath, activeDir) 117 | s.Nil(os.MkdirAll(activePath, os.ModePerm)) 118 | _, err := os.Create(path.Join(activePath, "some-file")) 119 | s.Nil(err) 120 | 121 | s.Error(s.h.UnsetActive()) 122 | } 123 | 124 | func (s *HandlerSuite) TestValidateError() { 125 | // dir doesn't exist 126 | wd, err := s.h.builder.Build(path.Join(s.h.workdirsPath, "local", "some")) 127 | s.Nil(err) 128 | err = s.h.Validate(wd) 129 | s.True(ErrMalformed.Is(err)) 130 | 131 | // dir is a file 132 | s.Nil(os.MkdirAll(path.Join(s.h.workdirsPath, "local"), os.ModePerm)) 133 | _, err = os.Create(wd.Path) 134 | s.Nil(err) 135 | 136 | err = s.h.Validate(wd) 137 | s.True(ErrMalformed.Is(err)) 138 | s.Nil(os.RemoveAll(wd.Path)) 139 | 140 | // dir is empty 141 | s.Nil(os.MkdirAll(wd.Path, os.ModePerm)) 142 | err = s.h.Validate(wd) 143 | s.True(ErrMalformed.Is(err)) 144 | } 145 | 146 | func (s *HandlerSuite) TestListOk() { 147 | s.Nil(os.MkdirAll(s.h.workdirsPath, os.ModePerm)) 148 | 149 | // empty results 150 | wds, err := s.h.List() 151 | s.Nil(err) 152 | s.Len(wds, 0) 153 | 154 | // multiple results 155 | s.createWd("one") 156 | s.createWd("two") 157 | 158 | wds, err = s.h.List() 159 | s.Nil(err) 160 | s.Len(wds, 2) 161 | 162 | s.Equal("one", wds[0].Name) 163 | s.Equal("two", wds[1].Name) 164 | 165 | // incorrect directory should be skipped 166 | wd, err := s.h.builder.Build(path.Join(s.h.workdirsPath, "local", "some")) 167 | s.Nil(err) 168 | s.Nil(os.MkdirAll(wd.Path, os.ModePerm)) 169 | 170 | wds, err = s.h.List() 171 | s.Nil(err) 172 | s.Len(wds, 2) 173 | } 174 | 175 | func (s *HandlerSuite) TestListError() { 176 | // workdirs dir doesn't exist 177 | _, err := s.h.List() 178 | s.True(ErrMalformed.Is(err)) 179 | } 180 | 181 | func (s *HandlerSuite) TestRemoveOk() { 182 | // local 183 | wd, err := InitLocal("local") 184 | s.Nil(err) 185 | s.Nil(s.h.Remove(wd)) 186 | _, err = os.Stat(wd.Path) 187 | s.True(os.IsNotExist(err)) 188 | 189 | // org 190 | wd, err = InitOrgs([]string{"some-org"}, "token", false) 191 | s.Nil(err) 192 | s.Nil(s.h.Remove(wd)) 193 | _, err = os.Stat(wd.Path) 194 | s.True(os.IsNotExist(err)) 195 | 196 | // skip deleting dir with extra files 197 | wd = s.createWd("some") 198 | _, err = os.Create(path.Join(wd.Path, "some-file")) 199 | s.Nil(err) 200 | 201 | s.Nil(s.h.Remove(wd)) 202 | _, err = os.Stat(wd.Path) 203 | s.Nil(err) 204 | } 205 | 206 | func (s *HandlerSuite) createWd(name string) *Workdir { 207 | wd, err := InitLocal(name) 208 | s.Nil(err) 209 | return wd 210 | } 211 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | source{d} Community Edition (CE) 3 | 4 | 5 | **source{d} Community Edition (CE) is the data platform for your software development life cycle.** 6 | 7 | [![GitHub version](https://badge.fury.io/gh/src-d%2Fsourced-ce.svg)](https://github.com/src-d/sourced-ce/releases) 8 | [![Build Status](https://travis-ci.com/src-d/sourced-ce.svg?branch=master)](https://travis-ci.com/src-d/sourced-ce) 9 | ![Beta](https://svg-badge.appspot.com/badge/stability/beta?color=D6604A) 10 | [![Go Report Card](https://goreportcard.com/badge/github.com/src-d/sourced-ce)](https://goreportcard.com/report/github.com/src-d/sourced-ce) 11 | [![GoDoc](https://godoc.org/github.com/src-d/sourced-ce?status.svg)](https://godoc.org/github.com/src-d/sourced-ce) 12 | 13 | [Website](https://www.sourced.tech) • 14 | [Documentation](https://docs.sourced.tech/community-edition) • 15 | [Blog](https://blog.sourced.tech) • 16 | [Slack](http://bit.ly/src-d-community) • 17 | [Twitter](https://twitter.com/sourcedtech) 18 | 19 | 20 | ![source{d} CE dashboard](docs/assets/dashboard.png) 21 | 22 | ## Introduction 23 | 24 | **source{d} Community Edition (CE)** helps you to manage all your code and engineering data in one place: 25 | 26 | - **Code Retrieval**: Retrieve and store the git history of the code of your organization as a dataset. 27 | - **Analysis in/for any Language**: Automatically identify languages, parse source code, and extract the pieces that matter in a language-agnostic way. 28 | - **History Analysis**: Extract information from the evolution, commits, and metadata of your codebase and from GitHub, generating detailed reports and insights. 29 | - **Familiar APIs**: Analyze your code through powerful SQL queries. Use tools you're familiar with to create reports and dashboards. 30 | 31 | This repository contains the code of **source{d} Community Edition (CE)** and its project documentation, which you can also see properly rendered at [docs.sourced.tech/community-edition](https://docs.sourced.tech/community-edition). 32 | 33 | 34 | ### Contents 35 | 36 | - [Introduction](README.md#introduction) 37 | - [Quick Start](README.md#quick-start) 38 | - [Architecture](README.md#architecture) 39 | - [Contributing](README.md#contributing) 40 | - [Community](README.md#community) 41 | - [Code of Conduct](README.md#code-of-conduct) 42 | - [License](README.md#license) 43 | 44 | ## Quick Start 45 | 46 | **source{d} CE** supports Linux, macOS, and Windows. 47 | 48 | To run it you only need: 49 | 50 | 1. To have Docker installed in your PC 51 | 1. Download `sourced` binary (for your OS) from [our releases](https://github.com/src-d/sourced-ce/releases) 52 | 1. Run it: 53 | ```bash 54 | $ sourced init orgs --token= 55 | ``` 56 | And log in into http://127.0.0.1:8088 with login: `admin`, and password: `admin`. 57 | 58 | If you want more details of each step, you will find in the [**Quick Start Guide**](docs/quickstart/README.md) all the steps to get started with **source{d} CE**, from the installation of its dependencies to running SQL queries to inspect git repositories. 59 | 60 | If you want to know more about **source{d} CE**, in the [next steps](docs/usage/README.md) section you will find some useful resources for guiding your experience using this tool. 61 | 62 | If you have any problem running **source{d} CE** you can take a look at our [Frequently Asked Questions](docs/learn-more/troubleshooting.md) or [Troubleshooting](docs/learn-more/troubleshooting.md) sections. You can also ask for help when using **source{d} CE** in our [source{d} Forum](https://forum.sourced.tech). If you spotted a bug, or you have a feature request, please [open an issue](https://github.com/src-d/sourced-ce/issues) to let us know about it. 63 | 64 | 65 | ## Architecture 66 | 67 | _For more details on the architecture of this project, read [docs/learn-more/architecture.md](docs/learn-more/architecture.md)._ 68 | 69 | **source{d} CE** is deployed as Docker containers, using Docker Compose. 70 | 71 | This tool is a wrapper for Docker Compose to manage the compose files and its containers easily. Moreover, `sourced` does not require a local installation of Docker Compose, if it is not found it will be deployed inside a container. 72 | 73 | The main entry point of **source{d} CE** is [sourced-ui](https://github.com/src-d/sourced-ui), the web interface from where you can access your data, create dashboards, run queries... 74 | 75 | The data exposed by the web interface is prepared and processed by the following services: 76 | 77 | - [babelfish](https://doc.bblf.sh): universal code parser. 78 | - [gitcollector](https://github.com/src-d/gitcollector): fetches the git repositories owned by your organization. 79 | - [ghsync](https://github.com/src-d/ghsync): fetches metadata from GitHub (users, pull requests, issues...). 80 | - [gitbase](https://github.com/src-d/gitbase): SQL database interface to Git repositories. 81 | 82 | 83 | ## Contributing 84 | 85 | [Contributions](https://github.com/src-d/sourced-ce/issues) are **welcome and very much appreciated** 🙌 86 | Please refer to [our Contribution Guide](docs/CONTRIBUTING.md) for more details. 87 | 88 | 89 | ## Community 90 | 91 | source{d} has an amazing community of developers and contributors who are interested in Code As Data and/or Machine Learning on Code. Please join us! 👋 92 | 93 | - [Community](https://sourced.tech/community/) 94 | - [Slack](http://bit.ly/src-d-community) 95 | - [Twitter](https://twitter.com/sourcedtech) 96 | - [Email](mailto:hello@sourced.tech) 97 | 98 | 99 | ## Code of Conduct 100 | 101 | All activities under source{d} projects are governed by the 102 | [source{d} code of conduct](https://github.com/src-d/guide/blob/master/.github/CODE_OF_CONDUCT.md). 103 | 104 | 105 | ## License 106 | 107 | GPL v3.0, see [LICENSE](LICENSE.md). 108 | -------------------------------------------------------------------------------- /cmd/sourced/compose/file/file.go: -------------------------------------------------------------------------------- 1 | // Package file provides functions to manage docker compose files inside the 2 | // $HOME/.sourced/compose-files directory 3 | package file 4 | 5 | import ( 6 | "encoding/base64" 7 | "fmt" 8 | "io/ioutil" 9 | "net/url" 10 | "os" 11 | "path/filepath" 12 | 13 | datadir "github.com/src-d/sourced-ce/cmd/sourced/dir" 14 | 15 | "github.com/pkg/errors" 16 | goerrors "gopkg.in/src-d/go-errors.v1" 17 | ) 18 | 19 | // ErrConfigDownload is returned when docker-compose.yml could not be downloaded 20 | var ErrConfigDownload = goerrors.NewKind("docker-compose.yml config file could not be downloaded") 21 | 22 | // ErrConfigActivation is returned when docker-compose.yml could not be set as active 23 | var ErrConfigActivation = goerrors.NewKind("docker-compose.yml could not be set as active") 24 | 25 | const ( 26 | orgName = "src-d" 27 | repoName = "sourced-ce" 28 | composeFileTmpl = "https://raw.githubusercontent.com/%s/%s/%s/docker-compose.yml" 29 | ) 30 | 31 | var version = "master" 32 | 33 | // activeDir is the name of the directory containing the symlink to the 34 | // active docker compose file 35 | const activeDir = "__active__" 36 | 37 | // RevOrURL is a revision (tag name, full sha1) or a valid URL to a 38 | // docker-compose.yml file 39 | type RevOrURL = string 40 | 41 | // composeFileURL returns the URL to download the raw github docker-compose.yml 42 | // file for the given revision (tag or full sha1) 43 | func composeFileURL(revision string) string { 44 | return fmt.Sprintf(composeFileTmpl, orgName, repoName, revision) 45 | } 46 | 47 | // SetVersion sets the version rewritten by the CI build 48 | func SetVersion(v string) { 49 | version = v 50 | } 51 | 52 | // InitDefault checks if there is an active docker compose file, and if there 53 | // isn't the file for this release is downloaded. 54 | // The current build version must be set with SetVersion. 55 | // It returns the absolute path to the active docker-compose.yml file 56 | func InitDefault() (string, error) { 57 | activeFilePath, err := path(activeDir) 58 | if err != nil { 59 | return "", err 60 | } 61 | 62 | _, err = os.Stat(activeFilePath) 63 | if err == nil { 64 | return activeFilePath, nil 65 | } 66 | 67 | if !os.IsNotExist(err) { 68 | return "", err 69 | } 70 | 71 | err = ActivateFromRemote(version) 72 | if err != nil { 73 | return "", err 74 | } 75 | 76 | return activeFilePath, nil 77 | } 78 | 79 | // ActivateFromRemote downloads the docker-compose.yml file from the given revision 80 | // or URL, and sets it as the active compose file. 81 | func ActivateFromRemote(revOrURL RevOrURL) (err error) { 82 | var url string 83 | if isURL(revOrURL) { 84 | url = revOrURL 85 | } else { 86 | url = composeFileURL(revOrURL) 87 | } 88 | 89 | outPath, err := path(revOrURL) 90 | if err != nil { 91 | return err 92 | } 93 | 94 | err = datadir.DownloadURL(url, outPath) 95 | if err != nil { 96 | return ErrConfigDownload.Wrap(err) 97 | } 98 | 99 | err = SetActive(revOrURL) 100 | if err != nil { 101 | return ErrConfigActivation.Wrap(err) 102 | } 103 | 104 | return nil 105 | } 106 | 107 | // SetActive makes a symlink from 108 | // $HOME/.sourced/compose-files/__active__/docker-compose.yml to the compose file 109 | // for the given revision or URL. 110 | func SetActive(revOrURL RevOrURL) error { 111 | filePath, err := path(revOrURL) 112 | if err != nil { 113 | return err 114 | } 115 | 116 | if _, err := os.Stat(filePath); err != nil { 117 | if os.IsNotExist(err) { 118 | return errors.Wrapf(err, "could not find a docker-compose.yml file in `%s`", filePath) 119 | } 120 | 121 | return err 122 | } 123 | 124 | activeFilePath, err := path(activeDir) 125 | if err != nil { 126 | return err 127 | } 128 | 129 | err = os.MkdirAll(filepath.Dir(activeFilePath), os.ModePerm) 130 | if err != nil { 131 | return errors.Wrapf(err, "error while creating directory for %s", activeFilePath) 132 | } 133 | 134 | if _, err := os.Lstat(activeFilePath); err == nil { 135 | if err := os.Remove(activeFilePath); err != nil { 136 | return errors.Wrap(err, "failed to unlink") 137 | } 138 | } 139 | 140 | return os.Symlink(filePath, activeFilePath) 141 | } 142 | 143 | // Active returns the revision (tag name, full sha1) or the URL of the active 144 | // docker compose file 145 | func Active() (RevOrURL, error) { 146 | activeFilePath, err := path(activeDir) 147 | if err != nil { 148 | return "", err 149 | } 150 | 151 | if _, err := os.Stat(activeFilePath); err != nil { 152 | if os.IsNotExist(err) { 153 | return "", nil 154 | } 155 | 156 | return "", err 157 | } 158 | 159 | dest, err := filepath.EvalSymlinks(activeFilePath) 160 | if err != nil { 161 | return "", err 162 | } 163 | 164 | _, name := filepath.Split(filepath.Dir(dest)) 165 | return composeName(name), nil 166 | } 167 | 168 | // List returns a list of installed docker compose files. Each name is the 169 | // revision (tag name, full sha1) or the URL 170 | func List() ([]RevOrURL, error) { 171 | list := []RevOrURL{} 172 | 173 | dir, err := dir() 174 | if err != nil { 175 | return list, err 176 | } 177 | 178 | if _, err := os.Stat(dir); err != nil { 179 | if os.IsNotExist(err) { 180 | return list, nil 181 | } 182 | 183 | return list, err 184 | } 185 | 186 | files, err := ioutil.ReadDir(dir) 187 | if err != nil { 188 | return list, err 189 | } 190 | 191 | for _, f := range files { 192 | entry := f.Name() 193 | if entry == activeDir { 194 | continue 195 | } 196 | 197 | list = append(list, composeName(entry)) 198 | } 199 | 200 | return list, nil 201 | } 202 | 203 | func composeName(rev string) string { 204 | if decoded, err := base64.URLEncoding.DecodeString(rev); err == nil { 205 | return string(decoded) 206 | } 207 | 208 | return rev 209 | } 210 | 211 | func isURL(revOrURL RevOrURL) bool { 212 | _, err := url.ParseRequestURI(revOrURL) 213 | return err == nil 214 | } 215 | 216 | // dir returns the absolute path for $HOME/.sourced/compose-files 217 | func dir() (string, error) { 218 | path, err := datadir.Path() 219 | if err != nil { 220 | return "", err 221 | } 222 | 223 | return filepath.Join(path, "compose-files"), nil 224 | } 225 | 226 | // path returns the absolute path to 227 | // $HOME/.sourced/compose-files/revOrURL/docker-compose.yml 228 | func path(revOrURL RevOrURL) (string, error) { 229 | composeDirPath, err := dir() 230 | if err != nil { 231 | return "", err 232 | } 233 | 234 | subPath := revOrURL 235 | if isURL(revOrURL) { 236 | subPath = base64.URLEncoding.EncodeToString([]byte(revOrURL)) 237 | } 238 | 239 | dirPath := filepath.Join(composeDirPath, subPath) 240 | 241 | return filepath.Join(dirPath, "docker-compose.yml"), nil 242 | } 243 | -------------------------------------------------------------------------------- /docs/usage/commands.md: -------------------------------------------------------------------------------- 1 | # List of `sourced` Sub-Commands 2 | 3 | `sourced` binary offers you different kinds of sub-commands: 4 | - [to manage their containers](#manage-containers) 5 | - [to manage **source{d} CE** configuration](#manage-configuration) 6 | - [to open interfaces to access its data](#open-interfaces) 7 | - [show info about the command](#others) 8 | 9 | Here is the list of all these commands and its description; you can get more info about each one 10 | adding `--help` when you run it. 11 | 12 | 13 | ## Manage Containers 14 | 15 | ### sourced init 16 | 17 | _There is a dedicated section to document this command in the quickstart about [how to initialize **source{d} CE**](../quickstart/3-init-sourced.md)_ 18 | 19 | This command installs and initializes **source{d} CE** docker containers, networks, and volumes, downloading its docker images if needed. 20 | 21 | It can work over a local repository or a list of GitHub organizations. 22 | 23 | **source{d} CE** will download and install Docker images on demand. Therefore, the first time you run some of these commands, they might take a bit of time to start up. Subsequent runs will be faster. 24 | 25 | Once **source{d} CE** has been initialized, it will automatically open the web UI. 26 | If the UI is not opened automatically, you can use [`sourced web`](#sourced-web) command, or visit http://127.0.0.1:8088. 27 | 28 | Use login: `admin` and password: `admin`, to access the web interface. 29 | 30 | #### sourced init orgs 31 | 32 | ```shell 33 | $ sourced init orgs --token=_USER_TOKEN_ [--with-forks] org1,org2... 34 | ``` 35 | 36 | Installs and initializes **source{d} CE** for a list of GitHub organizations, downloading their repositories and 37 | metadata: Users, PullRequests, Issues... 38 | 39 | The `orgs` argument must be a comma-separated list of GitHub organizations. 40 | 41 | The `--token` must contain a valid GitHub user token for the given organizations. It should be granted with 42 | 'repo' and'read:org' scopes. 43 | 44 | If `--with-forks` is passed, it will also fetch repositories who are marked as forks. 45 | 46 | #### sourced init local 47 | 48 | ```shell 49 | $ sourced init local [/path/to/repos] 50 | ``` 51 | 52 | Installs and initializes **source{d} CE** using a local directory containing the git repositories to be processed by **source{d} CE**. If the local path to the `workdir` is not provided, the current working directory will be used. 53 | 54 | ### sourced start 55 | 56 | Starts all the components that were initialized with `init` and then stopped with `stop`. 57 | 58 | ### sourced stop 59 | 60 | Stops all running containers without removing them. They can be started again with `start`. 61 | 62 | ### sourced prune 63 | 64 | Stops containers and removes containers, networks, volumes, and configurations created by `init` for the current working directory. 65 | 66 | To delete resources for all the installed working directories, add the `--all` flag. 67 | 68 | Container images are not deleted unless you specify the `--images` flag. 69 | 70 | If you want to completely uninstall `sourced` you must also delete the `~/.sourced` directory. 71 | 72 | ### sourced logs 73 | 74 | Show logs from source{d} components. 75 | 76 | If `--follow` is used the logs are shown as they are logged until you exit with `Ctrl+C`. 77 | 78 | You can optionally pass component names to see only their logs. 79 | 80 | ```shell 81 | $ sourced logs 82 | $ sourced logs --follow 83 | $ sourced logs --follow gitbase bblfsh 84 | ``` 85 | 86 | 87 | ## Manage Configuration 88 | 89 | ### sourced status 90 | 91 | Shows the status of **source{d} CE** components, the installed working directories and the current deployment. 92 | 93 | #### sourced status all 94 | 95 | Show all the available status information, from the `components`, `config` and `workdirs`, sub-commands below. 96 | 97 | #### sourced status components 98 | 99 | Shows the status of the components containers of the running working directory 100 | 101 | #### sourced status config 102 | 103 | Shows the docker-compose environment variables configuration for the active working directory 104 | 105 | #### sourced status workdirs 106 | 107 | Lists all the previously initialized working directories 108 | 109 | ### sourced compose 110 | 111 | Manages Docker Compose files in the `~/.sourced` directory with the following subcommands: 112 | 113 | ### sourced compose download 114 | 115 | Download the `docker-compose.yml` file to define **source{d} CE** services. By default, the command downloads the file for this binary version, but you can also download other version or any other custom one using its URL. 116 | 117 | Examples: 118 | ```shell 119 | $ sourced compose download 120 | $ sourced compose download v0.0.1 121 | $ sourced compose download master 122 | $ sourced compose download https://raw.githubusercontent.com/src-d/sourced-ce/master/docker-compose.yml 123 | ``` 124 | 125 | ### sourced compose list 126 | 127 | Lists the available `docker-compose.yml` files, and shows which one is active. 128 | You can activate any other with `compose set`. 129 | 130 | ### sourced compose set 131 | 132 | Sets the active `docker-compose.yml` file. Accepts either the name or index of the compose file as returned by 'compose list'. 133 | 134 | #### sourced restart 135 | 136 | Updates current installation according to the active docker compose file. 137 | 138 | It only recreates the component containers, keeping all your data, as charts, dashboards, repositories and GitHub metadata. 139 | 140 | 141 | ## Open Interfaces 142 | 143 | ### sourced sql 144 | 145 | Opens a MySQL client connected to gitbase. 146 | 147 | You can also pass a SQL query to be run by gitbase instead of opening the REPL, e.g. 148 | ```shell 149 | $ sourced sql "show databases" 150 | 151 | +----------+ 152 | | Database | 153 | +----------+ 154 | | gitbase | 155 | +----------+ 156 | ``` 157 | 158 | **source{d} CE** SQL supports a [UAST](#babelfish-uast) function that returns a Universal AST for the selected source text. UAST values are returned as binary blobs and are best visualized in the [SQL Lab, from the web interface](../quickstart/4-explore-sourced.md#sql-lab-querying-code) rather than the CLI where are seen as binary data. 159 | 160 | ### sourced web 161 | 162 | Opens the web interface in your browser. 163 | 164 | Use login: `admin` and password: `admin`, to access the web interface. 165 | 166 | 167 | ## Others 168 | 169 | ### sourced version 170 | 171 | Shows the version of the `sourced` command being used. 172 | 173 | ### sourced completion 174 | 175 | Prints a bash completion script for sourced; you can place its output in 176 | `/etc/bash_completion.d/sourced`, or add it to your `.bashrc` running: 177 | 178 | ```shell 179 | $ echo "source <(sourced completion)" >> ~/.bashrc 180 | ``` 181 | -------------------------------------------------------------------------------- /cmd/sourced/cmd/web.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "bytes" 5 | "context" 6 | "fmt" 7 | "net/http" 8 | "os" 9 | "os/exec" 10 | "regexp" 11 | "runtime" 12 | "strings" 13 | "time" 14 | 15 | "github.com/pkg/browser" 16 | "github.com/pkg/errors" 17 | "github.com/src-d/sourced-ce/cmd/sourced/compose" 18 | ) 19 | 20 | // The service name used in docker-compose.yml for the srcd/sourced-ui image 21 | const containerName = "sourced-ui" 22 | 23 | type webCmd struct { 24 | Command `name:"web" short-description:"Open the web interface in your browser." long-description:"Open the web interface in your browser, by default at: http://127.0.0.1:8088 user:admin pass:admin"` 25 | } 26 | 27 | func (c *webCmd) Execute(args []string) error { 28 | return OpenUI(2 * time.Second) 29 | } 30 | 31 | func init() { 32 | rootCmd.AddCommand(&webCmd{}) 33 | } 34 | 35 | func openUI(address string) error { 36 | // docker-compose returns 0.0.0.0 which is correct for the bind address 37 | // but incorrect as connect address 38 | url := fmt.Sprintf("http://%s", strings.Replace(address, "0.0.0.0", "127.0.0.1", 1)) 39 | 40 | for { 41 | client := http.Client{Timeout: time.Second} 42 | if _, err := client.Get(url); err == nil { 43 | break 44 | } 45 | 46 | time.Sleep(1 * time.Second) 47 | } 48 | 49 | if err := browser.OpenURL(url); err != nil { 50 | return errors.Wrap(err, "could not open the browser") 51 | } 52 | 53 | return nil 54 | } 55 | 56 | var stateExtractor = regexp.MustCompile(`(?m)^srcd-\w+.*(Up|Exit (\d+))`) 57 | 58 | func checkServiceStatus(service string) error { 59 | var stdout bytes.Buffer 60 | if err := compose.RunWithIO(context.Background(), 61 | os.Stdin, &stdout, nil, "ps", service); err != nil { 62 | return errors.Wrapf(err, "cannot get status service %s", service) 63 | } 64 | 65 | matches := stateExtractor.FindAllStringSubmatch(strings.TrimSpace(stdout.String()), -1) 66 | for _, match := range matches { 67 | state := match[1] 68 | 69 | if strings.HasPrefix(state, "Exit") { 70 | if service != "ghsync" && service != "gitcollector" { 71 | return fmt.Errorf("service '%s' is in state '%s'", service, state) 72 | } 73 | 74 | returnCode := state[len("Exit "):len(state)] 75 | if returnCode != "0" { 76 | return fmt.Errorf("service '%s' exited with return code: %s", service, returnCode) 77 | } 78 | 79 | continue 80 | } 81 | 82 | if state != "Up" { 83 | return fmt.Errorf("service '%s' is in state '%s'", service, state) 84 | } 85 | } 86 | 87 | return nil 88 | } 89 | 90 | // runMonitor checks the status of the containers in order to early exit in case 91 | // an unrecoverable error occurs. 92 | // The monitoring is performed by running `docker-compose ps ` for each 93 | // service returned by `docker-compose config --services`, and by grepping the 94 | // state from the stdout using a regex. 95 | // Getting the state of all the containers in a single pass by running `docker-compose ps` 96 | // and by using a multi-line regex to extract both service name and state is not reliable. 97 | // The reason is that the prefix of a container can be very long, especially for local 98 | // initialization, due to the value that we set for `COMPOSE_PROJECT_NAME` env var, and 99 | // docker-compose may split the name into multiple lines. 100 | // E.g.: 101 | // 102 | // Name Command State Ports 103 | // ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 104 | // srcd-l1vzzxjzl3nln2vudhlzztdlbi9qcm9qzwn0cy8uz28td29ya3nwywnll3nyyy9naxrodwiuy29tl3nln /bin/bblfsh-web -addr :808 ... Up 0.0.0.0:9999->8080/tcp 105 | // 2vudhlzztdlbg_bblfsh-web_1 106 | func runMonitor(ch chan<- error) { 107 | var servicesBuf bytes.Buffer 108 | if err := compose.RunWithIO(context.Background(), 109 | os.Stdin, &servicesBuf, nil, "config", "--services"); err != nil { 110 | ch <- errors.Wrap(err, "cannot get list of services") 111 | return 112 | } 113 | 114 | services := strings.Split(strings.TrimSpace(servicesBuf.String()), "\n") 115 | 116 | for _, service := range services { 117 | if err := checkServiceStatus(service); err != nil { 118 | ch <- err 119 | return 120 | } 121 | time.Sleep(time.Second) 122 | } 123 | } 124 | 125 | func getContainerPublicAddress(containerName, privatePort string) (string, error) { 126 | var stdout bytes.Buffer 127 | for { 128 | err := compose.RunWithIO(context.Background(), nil, &stdout, nil, "port", containerName, privatePort) 129 | if err == nil { 130 | break 131 | } 132 | // skip any unsuccessful command exits 133 | if _, ok := err.(*exec.ExitError); !ok { 134 | return "", err 135 | } 136 | 137 | time.Sleep(1 * time.Second) 138 | } 139 | 140 | address := strings.TrimSpace(stdout.String()) 141 | if address == "" { 142 | return "", fmt.Errorf("could not find the public port of %s", containerName) 143 | } 144 | 145 | return address, nil 146 | } 147 | 148 | // OpenUI opens the browser with the UI. 149 | func OpenUI(timeout time.Duration) error { 150 | ch := make(chan error) 151 | 152 | go func() { 153 | address, err := getContainerPublicAddress(containerName, "8088") 154 | if err != nil { 155 | ch <- err 156 | return 157 | } 158 | 159 | ch <- openUI(address) 160 | }() 161 | 162 | go runMonitor(ch) 163 | 164 | fmt.Println(` 165 | Once source{d} is fully initialized, the UI will be available, by default at: 166 | http://127.0.0.1:8088 167 | user:admin 168 | pass:admin 169 | `) 170 | 171 | if timeout > 5*time.Second { 172 | stopSpinner := startSpinner("Initializing source{d}...") 173 | defer stopSpinner() 174 | } 175 | 176 | select { 177 | case err := <-ch: 178 | return err 179 | case <-time.After(timeout): 180 | return fmt.Errorf("error opening the UI, the container is not running after %v", timeout) 181 | } 182 | } 183 | 184 | type spinner struct { 185 | msg string 186 | charset []int 187 | interval time.Duration 188 | 189 | stop chan bool 190 | } 191 | 192 | func startSpinner(msg string) func() { 193 | charset := []int{'⠋', '⠙', '⠹', '⠸', '⠼', '⠴', '⠦', '⠧', '⠇', '⠏'} 194 | if runtime.GOOS == "windows" { 195 | charset = []int{'|', '/', '-', '\\'} 196 | } 197 | 198 | s := &spinner{ 199 | msg: msg, 200 | charset: charset, 201 | interval: 200 * time.Millisecond, 202 | stop: make(chan bool), 203 | } 204 | s.Start() 205 | 206 | return s.Stop 207 | } 208 | 209 | func (s *spinner) Start() { 210 | go s.printLoop() 211 | } 212 | 213 | func (s *spinner) Stop() { 214 | s.stop <- true 215 | } 216 | 217 | func (s *spinner) printLoop() { 218 | i := 0 219 | for { 220 | select { 221 | case <-s.stop: 222 | fmt.Println(s.msg) 223 | return 224 | default: 225 | char := string(s.charset[i%len(s.charset)]) 226 | if runtime.GOOS == "windows" { 227 | fmt.Printf("\r%s %s", s.msg, char) 228 | } else { 229 | fmt.Printf("%s %s\n\033[A", s.msg, char) 230 | } 231 | 232 | time.Sleep(s.interval) 233 | } 234 | 235 | i++ 236 | if len(s.charset) == i { 237 | i = 0 238 | } 239 | } 240 | } 241 | -------------------------------------------------------------------------------- /cmd/sourced/compose/compose.go: -------------------------------------------------------------------------------- 1 | package compose 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "io" 7 | "os" 8 | "os/exec" 9 | "path/filepath" 10 | "regexp" 11 | "runtime" 12 | "strconv" 13 | "strings" 14 | 15 | "github.com/blang/semver" 16 | "github.com/src-d/sourced-ce/cmd/sourced/compose/workdir" 17 | "github.com/src-d/sourced-ce/cmd/sourced/dir" 18 | 19 | "github.com/pkg/errors" 20 | goerrors "gopkg.in/src-d/go-errors.v1" 21 | ) 22 | 23 | // v1.20.0 is the first version that supports `--compatibility` flag we rely on 24 | // there is no mention of it in changelog 25 | // the version has been found by trying downgrading unless it started to error 26 | var minDockerComposeVersion = semver.Version{ 27 | Major: 1, 28 | Minor: 20, 29 | Patch: 0, 30 | } 31 | 32 | // this version is choosen to be always compatible with docker-compose version 33 | // docker-compose v1.20.0 introduced compose files version 3.6 34 | // which requires Docker Engine 18.02.0 or above 35 | var minDockerVersion = semver.Version{ 36 | Major: 18, 37 | Minor: 2, 38 | Patch: 0, 39 | } 40 | 41 | // dockerComposeVersion is the version of docker-compose to download 42 | // if docker-compose isn't already present in the system 43 | const dockerComposeVersion = "1.24.0" 44 | 45 | var composeContainerURL = fmt.Sprintf("https://github.com/docker/compose/releases/download/%s/run.sh", dockerComposeVersion) 46 | 47 | // ErrComposeAlternative is returned when docker-compose alternative could not be installed 48 | var ErrComposeAlternative = goerrors.NewKind("error while trying docker-compose container alternative") 49 | 50 | type Compose struct { 51 | bin string 52 | workdirHandler *workdir.Handler 53 | } 54 | 55 | func (c *Compose) Run(ctx context.Context, arg ...string) error { 56 | return c.RunWithIO(ctx, os.Stdin, os.Stdout, os.Stderr, arg...) 57 | } 58 | 59 | func (c *Compose) RunWithIO(ctx context.Context, stdin io.Reader, 60 | stdout, stderr io.Writer, arg ...string) error { 61 | arg = append([]string{"--compatibility"}, arg...) 62 | cmd := exec.CommandContext(ctx, c.bin, arg...) 63 | 64 | wd, err := c.workdirHandler.Active() 65 | if err != nil { 66 | return err 67 | } 68 | 69 | if err := c.workdirHandler.Validate(wd); err != nil { 70 | return err 71 | } 72 | 73 | cmd.Dir = wd.Path 74 | cmd.Stdin = stdin 75 | cmd.Stdout = stdout 76 | cmd.Stderr = stderr 77 | 78 | return cmd.Run() 79 | } 80 | 81 | func newCompose() (*Compose, error) { 82 | // check docker first and exit fast 83 | dockerVersion, err := getDockerVersion() 84 | if err != nil { 85 | return nil, err 86 | } 87 | if !dockerVersion.GE(minDockerVersion) { 88 | return nil, fmt.Errorf("Minimal required docker version is %s but %s found", minDockerVersion, dockerVersion) 89 | } 90 | 91 | workdirHandler, err := workdir.NewHandler() 92 | if err != nil { 93 | return nil, err 94 | } 95 | 96 | bin, err := getOrInstallComposeBinary() 97 | if err != nil { 98 | return nil, err 99 | } 100 | dockerComposeVersion, err := getDockerComposeVersion(bin) 101 | if err != nil { 102 | return nil, err 103 | } 104 | if !dockerComposeVersion.GE(minDockerComposeVersion) { 105 | return nil, fmt.Errorf("Minimal required docker-compose version is %s but %s found", minDockerComposeVersion, dockerComposeVersion) 106 | } 107 | 108 | return &Compose{ 109 | bin: bin, 110 | workdirHandler: workdirHandler, 111 | }, nil 112 | } 113 | 114 | func getOrInstallComposeBinary() (string, error) { 115 | path, err := exec.LookPath("docker-compose") 116 | if err == nil { 117 | bin := strings.TrimSpace(path) 118 | if bin != "" { 119 | return bin, nil 120 | } 121 | } 122 | 123 | path, err = getOrInstallComposeContainer() 124 | if err != nil { 125 | return "", ErrComposeAlternative.Wrap(err) 126 | } 127 | 128 | return path, nil 129 | } 130 | 131 | func getOrInstallComposeContainer() (altPath string, err error) { 132 | datadir, err := dir.Path() 133 | if err != nil { 134 | return "", err 135 | } 136 | 137 | dirPath := filepath.Join(datadir, "bin") 138 | path := filepath.Join(dirPath, fmt.Sprintf("docker-compose-%s.sh", dockerComposeVersion)) 139 | 140 | readExecAccessMode := os.FileMode(0500) 141 | 142 | if info, err := os.Stat(path); err == nil { 143 | if info.Mode()&readExecAccessMode != readExecAccessMode { 144 | return "", fmt.Errorf("%s can not be run", path) 145 | } 146 | 147 | return path, nil 148 | } else if !os.IsNotExist(err) { 149 | return "", err 150 | } 151 | 152 | if err := downloadCompose(path); err != nil { 153 | return "", err 154 | } 155 | 156 | cmd := exec.CommandContext(context.Background(), "chmod", "+x", path) 157 | if err := cmd.Run(); err != nil { 158 | return "", errors.Wrapf(err, "cannot change permission to %s", path) 159 | } 160 | 161 | return path, nil 162 | } 163 | 164 | func downloadCompose(path string) error { 165 | if runtime.GOOS == "windows" { 166 | return fmt.Errorf("compose in container is not compatible with Windows") 167 | } 168 | 169 | return dir.DownloadURL(composeContainerURL, path) 170 | } 171 | 172 | func Run(ctx context.Context, arg ...string) error { 173 | comp, err := newCompose() 174 | if err != nil { 175 | return err 176 | } 177 | 178 | return comp.Run(ctx, arg...) 179 | } 180 | 181 | func RunWithIO(ctx context.Context, stdin io.Reader, stdout, stderr io.Writer, arg ...string) error { 182 | comp, err := newCompose() 183 | if err != nil { 184 | return err 185 | } 186 | 187 | return comp.RunWithIO(ctx, stdin, stdout, stderr, arg...) 188 | } 189 | 190 | var dockerVersionRe = regexp.MustCompile(`version (\d+).(\d+).(\d+)`) 191 | var dockerComposeVersionRe = regexp.MustCompile(`version (\d+.\d+.\d+)`) 192 | 193 | // docker doesn't use semver, so simple `semver.Parse` would fail 194 | // but semver.Version struct fits us to allow simple comparation 195 | func getDockerVersion() (*semver.Version, error) { 196 | if _, err := exec.LookPath("docker"); err != nil { 197 | return nil, err 198 | } 199 | 200 | out, err := exec.Command("docker", "--version").Output() 201 | if err != nil { 202 | return nil, err 203 | } 204 | 205 | submatches := dockerVersionRe.FindSubmatch(out) 206 | if len(submatches) != 4 { 207 | return nil, fmt.Errorf("can't parse docker version") 208 | } 209 | 210 | v := &semver.Version{} 211 | v.Major, err = strconv.ParseUint(string(submatches[1]), 10, 64) 212 | if err != nil { 213 | return nil, fmt.Errorf("can't parse docker version") 214 | } 215 | v.Minor, err = strconv.ParseUint(string(submatches[2]), 10, 64) 216 | if err != nil { 217 | return nil, fmt.Errorf("can't parse docker version") 218 | } 219 | v.Patch, err = strconv.ParseUint(string(submatches[3]), 10, 64) 220 | if err != nil { 221 | return nil, fmt.Errorf("can't parse docker version") 222 | } 223 | 224 | return v, nil 225 | } 226 | 227 | func getDockerComposeVersion(bin string) (*semver.Version, error) { 228 | out, err := exec.Command(bin, "--version").Output() 229 | if err != nil { 230 | return nil, err 231 | } 232 | 233 | submatches := dockerComposeVersionRe.FindSubmatch(out) 234 | if len(submatches) != 2 { 235 | return nil, fmt.Errorf("can't parse docker-compose version") 236 | } 237 | 238 | v, err := semver.ParseTolerant(string(submatches[1])) 239 | if err != nil { 240 | return nil, fmt.Errorf("can't parse docker-compose version: %s", err) 241 | } 242 | 243 | return &v, nil 244 | } 245 | -------------------------------------------------------------------------------- /docs/learn-more/faq.md: -------------------------------------------------------------------------------- 1 | # Frequently Asked Questions 2 | 3 | _For tips and advices to deal with unexpected errors, please refer to [Troubleshooting guide](./troubleshooting.md)_ 4 | 5 | ## Index 6 | 7 | - [Where Can I Find More Assistance to Run source{d} or Notify You About Any Issue or Suggestion?](#where-can-i-find-more-assistance-to-run-source-d-or-notify-you-about-any-issue-or-suggestion) 8 | - [How Can I Update My Version Of **source{d} CE**?](#how-can-i-update-my-version-of-source-d-ce) 9 | - [How to Update the Data from the Organizations That I'm Analyzing](#how-to-update-the-data-from-the-organizations-being-analyzed) 10 | - [Can I Query Gitbase or Babelfish with External Tools?](#can-i-query-gitbase-or-babelfish-with-external-tools) 11 | - [Where Can I Read More About the Web Interface?](#where-can-i-read-more-about-the-web-interface) 12 | - [I Get IOError Permission denied](#i-get-ioerror-permission-denied) 13 | - [Why Do I Need Internet Connection?](#why-do-i-need-internet-connection) 14 | 15 | 16 | ## Where Can I Find More Assistance to Run source{d} or Notify You About Any Issue or Suggestion? 17 | 18 | _If you're dealing with an error or something that you think that can be caused 19 | by an unexpected error, please refer to our [Troubleshooting guide](./troubleshooting.md). 20 | With the info that you can obtain following those steps, you could fix the problem 21 | or you will be able to explain it better in the following channels:_ 22 | 23 | * [open an issue](https://github.com/src-d/sourced-ce/issues), if you want to 24 | suggest a new feature, if you need assistance with a contribution, or if you 25 | found any bug. 26 | * [Visit the source{d} Forum](https://forum.sourced.tech) where users and community 27 | members discuss anything source{d} related. You will find there some common questions 28 | from other source{d} users, or ask yours. 29 | * [join our community on Slack](https://sourced-community.slack.com/join/shared_invite/enQtMjc4Njk5MzEyNzM2LTFjNzY4NjEwZGEwMzRiNTM4MzRlMzQ4MmIzZjkwZmZlM2NjODUxZmJjNDI1OTcxNDAyMmZlNmFjODZlNTg0YWM), 30 | and talk with some of our engineers. 31 | 32 | 33 | ## How Can I Update My Version Of source{d} CE? 34 | 35 | When there is a new release of **source{d} CE**, it is noticed every time a `sourced` 36 | command is called. When it happens you can download the new version from 37 | [src-d/sourced-ce/releases/latest](https://github.com/src-d/sourced-ce/releases/latest), 38 | and proceed as it follows: 39 | 40 | (You can also follow these steps if you want to update to any beta version, to 41 | downgrade, or to use your own built version of **source{d} CE**) 42 | 43 | 1. replace your current version of `sourced` from its current location with the 44 | one you're installing ([see Quickstart. Install](quickstart/2-install-sourced.md)), 45 | and confirm it was done by running `sourced version`. 46 | 1. run `sourced compose download` to download the new configuration. 47 | 1. run `sourced restart` to apply the new configuration. 48 | 49 | This process will reinstall **source{d} CE** with the new components, but it will 50 | keep your current data (repositories, metadata, charts, dashboards, etc) of your 51 | existent workdirs. 52 | 53 | If you want to replace all your current customizations —including charts and 54 | dashboards—, with the ones from the release that you just installed, the 55 | official way to proceed is to `prune` the running workdirs, and `init` them again. 56 | 57 | _**disclaimer:** pruning a workdir will delete all its data: its saved queries 58 | and charts, and if you were using repositories and metadata downloaded from a 59 | GitHub organization, they will be deleted, and downloaded again._ 60 | 61 | 1. `sourced status workdirs` to get the list of your current workdirs 62 | 1. Prune the workdirs you need, or prune all of them at once running 63 | `sourced prune --all` 64 | 1. `sourced init [local|orgs] ...` for each workdir again, to initialize them with 65 | the new configuration. 66 | 67 | 68 | ## How to Update the Data from the Organizations Being Analyzed 69 | 70 | There is no way to update imported data, and 71 | [when a scraper is restarted](./troubleshooting.md#how-can-i-restart-one-scraper), 72 | it procedes as it follows: 73 | 74 | ### gitcollector 75 | 76 | Organizations and repositories are downloaded independently, so if they fail, 77 | the process is not stopped until all the organizations and repositories have been 78 | iterated. 79 | 80 | If `gitcollector` is restarted, it will download more repositories, but it won’t 81 | update any of the already existent ones. You can see the progress of the new process 82 | in the welcome dashboard; since already existent repositories won't be updated, 83 | those will appear as `failed` in progress status. 84 | 85 | ### ghsync 86 | 87 | The way how metadata is imported by `ghsync` is a bit different, and it is done 88 | sequentially per each organization, so if any step fails, the whole importation 89 | will fail. 90 | 91 | Pull requests, issues, and users of the same organization, are imported in that 92 | order in separate transaction each one, and if one transaction fails, the process 93 | will be stopped so the next ones won't be processed. 94 | 95 | Once the three different entities have been imported, the organization will be 96 | considered as "done", and restarting `ghsync` won't cause to update its data. 97 | 98 | If `ghsync` is restarted, it will only import data from organizations that could 99 | not be finished considering the rules explained above. The process of `ghsync` 100 | will be updated in the welcome dashboard and if an organization was already 101 | imported, it will appear as "nothing imported" in the status chart. 102 | 103 | 104 | ## Can I Query Gitbase or Babelfish with External Tools? 105 | 106 | Yes, as explained in our docs about [**source{d} CE** Architecture](./architecture.md#docker-networking), 107 | these and other components are exposed to the host machine, to be used by third 108 | party tools like [Jupyter Notebook](https://jupyter.org/), 109 | [gitbase clients](https://docs.sourced.tech/gitbase/using-gitbase/supported-clients) 110 | and [Babelfish clients](https://docs.sourced.tech/babelfish/using-babelfish/clients). 111 | 112 | The connection values that you should use to connect to these components, are 113 | defined in the [`docker-compose.yml`](../docker-compose.yml), and sumarized in 114 | the [Architecture documentation](./architecture.md#docker-networking) 115 | 116 | 117 | ## Where Can I Read More About the Web Interface? 118 | 119 | The user interface is based in the open-sourced [Apache Superset](http://superset.apache.org), 120 | so you can also refer to [Superset tutorials](http://superset.apache.org/tutorial.html) 121 | for advanced usage of the web interface. 122 | 123 | 124 | ## I Get IOError Permission denied 125 | 126 | If you get this error message: 127 | 128 | ``` 129 | IOError: [Errno 13] Permission denied: u'./.env' 130 | ``` 131 | 132 | This may happen if you have installed Docker from a snap package. This installation mode is not supported, please install it following [the official documentation](./quickstart/1-install-requirements.md#install-docker) (See [#78](https://github.com/src-d/sourced-ce/issues/78)). 133 | 134 | 135 | ## Why Do I Need Internet Connection? 136 | 137 | source{d} CE automatically fetches some resources from the Internet when they are not found locally: 138 | 139 | - the source{d} CE configuration is fetched automatically when initializing it for the first time, using the proper version for the current version of `sourced`, e.g. if using `v0.16.0` it will automatically fetch `https://raw.githubusercontent.com/src-d/sourced-ce/v0.16.0/docker-compose.yml`. 140 | - to download the docker images of the source{d} CE components when initializing source{d} for the first time, or when initializing it after changing its configuration. 141 | - to download repositories and its metadata from GitHub when you initialize source{d} CE with `sourced init orgs`. 142 | - to download and install [Docker Compose alternative](#docker-compose) if there is no local installation of Docker Compose. 143 | 144 | If your connection to the network does not let source{d} CE to access to Internet, you should manually provide all these dependencies. 145 | -------------------------------------------------------------------------------- /cmd/sourced/compose/workdir/factory.go: -------------------------------------------------------------------------------- 1 | package workdir 2 | 3 | import ( 4 | "bufio" 5 | "bytes" 6 | "encoding/base64" 7 | "fmt" 8 | "io/ioutil" 9 | "os" 10 | "os/exec" 11 | "path/filepath" 12 | "reflect" 13 | "runtime" 14 | "sort" 15 | "strconv" 16 | "strings" 17 | 18 | "github.com/pbnjay/memory" 19 | "github.com/pkg/errors" 20 | "github.com/serenize/snaker" 21 | composefile "github.com/src-d/sourced-ce/cmd/sourced/compose/file" 22 | ) 23 | 24 | // InitLocal initializes the workdir for local path and returns the Workdir instance 25 | func InitLocal(reposdir string) (*Workdir, error) { 26 | dirName := encodeDirName(reposdir) 27 | envf := newLocalEnvFile(dirName, reposdir) 28 | 29 | return initialize(dirName, "local", envf) 30 | } 31 | 32 | // InitOrgs initializes the workdir for organizations and returns the Workdir instance 33 | func InitOrgs(orgs []string, token string, withForks bool) (*Workdir, error) { 34 | // be indifferent to the order of passed organizations 35 | sort.Strings(orgs) 36 | dirName := encodeDirName(strings.Join(orgs, ",")) 37 | 38 | envf := envFile{} 39 | err := readEnvFile(dirName, "orgs", &envf) 40 | if err == nil && envf.NoForks == withForks { 41 | return nil, ErrInitFailed.Wrap( 42 | fmt.Errorf("workdir was previously initialized with a different value for forks support")) 43 | } 44 | if err != nil && !os.IsNotExist(err) { 45 | return nil, err 46 | } 47 | 48 | // re-create env file to make sure all fields are updated 49 | envf = newOrgEnvFile(dirName, orgs, token, withForks) 50 | 51 | return initialize(dirName, "orgs", envf) 52 | } 53 | 54 | func readEnvFile(dirName string, subPath string, envf *envFile) error { 55 | workdir, err := workdirPath(dirName, subPath) 56 | if err != nil { 57 | return err 58 | } 59 | 60 | envPath := filepath.Join(workdir, ".env") 61 | b, err := ioutil.ReadFile(envPath) 62 | if err != nil { 63 | return err 64 | } 65 | 66 | return envf.UnmarshalEnv(b) 67 | } 68 | 69 | func encodeDirName(dirName string) string { 70 | return base64.URLEncoding.EncodeToString([]byte(dirName)) 71 | } 72 | 73 | func workdirPath(dirName string, subPath string) (string, error) { 74 | path, err := workdirsPath() 75 | if err != nil { 76 | return "", err 77 | } 78 | 79 | workdir := filepath.Join(path, subPath, dirName) 80 | if err != nil { 81 | return "", err 82 | } 83 | 84 | return workdir, nil 85 | } 86 | 87 | func initialize(dirName string, subPath string, envf envFile) (*Workdir, error) { 88 | path, err := workdirsPath() 89 | if err != nil { 90 | return nil, err 91 | } 92 | 93 | workdir, err := workdirPath(dirName, subPath) 94 | if err != nil { 95 | return nil, err 96 | } 97 | 98 | err = os.MkdirAll(workdir, 0755) 99 | if err != nil { 100 | return nil, errors.Wrap(err, "could not create working directory") 101 | } 102 | 103 | defaultFilePath, err := composefile.InitDefault() 104 | if err != nil { 105 | return nil, err 106 | } 107 | 108 | composePath := filepath.Join(workdir, "docker-compose.yml") 109 | if err := link(defaultFilePath, composePath); err != nil { 110 | return nil, err 111 | } 112 | 113 | envPath := filepath.Join(workdir, ".env") 114 | contents, err := envf.MarshalEnv() 115 | if err != nil { 116 | return nil, err 117 | } 118 | err = ioutil.WriteFile(envPath, contents, 0644) 119 | 120 | if err != nil { 121 | return nil, errors.Wrap(err, "could not write .env file") 122 | } 123 | 124 | b := &builder{workdirsPath: path} 125 | return b.Build(workdir) 126 | } 127 | 128 | type envFile struct { 129 | ComposeProjectName string 130 | 131 | GitbaseVolumeType string 132 | GitbaseVolumeSource string 133 | GitbaseSiva bool 134 | 135 | GithubOrganizations []string 136 | GithubToken string 137 | 138 | NoForks bool 139 | 140 | GitbaseLimitCPU float32 141 | GitcollectorLimitCPU float32 142 | GitbaseLimitMem uint64 143 | } 144 | 145 | func newLocalEnvFile(dirName, repoDir string) envFile { 146 | f := envFile{ 147 | ComposeProjectName: fmt.Sprintf("srcd-%s", dirName), 148 | 149 | GitbaseVolumeType: "bind", 150 | GitbaseVolumeSource: repoDir, 151 | } 152 | f.addResourceLimits() 153 | 154 | return f 155 | } 156 | 157 | func newOrgEnvFile(dirName string, orgs []string, token string, withForks bool) envFile { 158 | f := envFile{ 159 | ComposeProjectName: fmt.Sprintf("srcd-%s", dirName), 160 | 161 | GitbaseVolumeType: "volume", 162 | GitbaseVolumeSource: "gitbase_repositories", 163 | GitbaseSiva: true, 164 | 165 | GithubOrganizations: orgs, 166 | GithubToken: token, 167 | 168 | NoForks: !withForks, 169 | } 170 | f.addResourceLimits() 171 | 172 | return f 173 | } 174 | 175 | func (f *envFile) addResourceLimits() { 176 | // limit CPU for containers 177 | dockerCPUs, err := dockerNumCPU() 178 | if err != nil { // show warning 179 | fmt.Println(err) 180 | } 181 | // apply gitbase resource limits only when docker runs without any global limits 182 | // it's default behaviour on linux 183 | if runtime.NumCPU() == dockerCPUs { 184 | f.GitbaseLimitCPU = float32(dockerCPUs) - 0.1 185 | } 186 | // always apply gitcollector limit 187 | if dockerCPUs > 0 { 188 | halfCPUs := float32(dockerCPUs) / 2.0 189 | // let container consume more than a half if there is only one cpu available 190 | // otherwise it will be too slow 191 | if halfCPUs < 1 { 192 | halfCPUs = 1 193 | } 194 | f.GitcollectorLimitCPU = halfCPUs - 0.1 195 | } 196 | 197 | // limit memory for containers 198 | dockerMem, err := dockerTotalMem() 199 | if err != nil { // show warning 200 | fmt.Println(err) 201 | } 202 | // apply memory limits only when only when docker runs without any global limits 203 | // it's default behaviour on linux 204 | if dockerMem == memory.TotalMemory() { 205 | f.GitbaseLimitMem = uint64(float64(dockerMem) * 0.9) 206 | } 207 | } 208 | 209 | var newlineChar = "\n" 210 | 211 | func init() { 212 | if runtime.GOOS == "windows" { 213 | newlineChar = "\r\n" 214 | } 215 | } 216 | 217 | // implementation can be moved to separate package if we need to marshal any other structs 218 | // supports only simple types 219 | func (f envFile) MarshalEnv() ([]byte, error) { 220 | var b bytes.Buffer 221 | 222 | v := reflect.ValueOf(f) 223 | rType := v.Type() 224 | 225 | for i := 0; i < rType.NumField(); i++ { 226 | field := rType.Field(i) 227 | fieldEl := v.Field(i) 228 | if field.Anonymous { 229 | panic("struct composition isn't supported") 230 | } 231 | 232 | name := strings.ToUpper(snaker.CamelToSnake(field.Name)) 233 | switch field.Type.Kind() { 234 | case reflect.Slice: 235 | slice := make([]string, fieldEl.Len()) 236 | for i := 0; i < fieldEl.Len(); i++ { 237 | slice[i] = fmt.Sprintf("%v", fieldEl.Index(i).Interface()) 238 | } 239 | fmt.Fprintf(&b, "%s=%v%s", name, strings.Join(slice, ","), newlineChar) 240 | case reflect.Bool: 241 | // marshal false value as empty string instead of "false" string 242 | if fieldEl.Interface().(bool) { 243 | fmt.Fprintf(&b, "%s=true%s", name, newlineChar) 244 | } else { 245 | fmt.Fprintf(&b, "%s=%s", name, newlineChar) 246 | } 247 | default: 248 | fmt.Fprintf(&b, "%s=%v%s", name, fieldEl.Interface(), newlineChar) 249 | } 250 | } 251 | 252 | return b.Bytes(), nil 253 | } 254 | 255 | // implementation can be moved to separate package if we need to unmarshal any other structs 256 | // supports only simple types 257 | func (f *envFile) UnmarshalEnv(b []byte) error { 258 | v := reflect.ValueOf(f).Elem() 259 | 260 | r := bytes.NewReader(b) 261 | scanner := bufio.NewScanner(r) 262 | for scanner.Scan() { 263 | line := strings.TrimSpace(scanner.Text()) 264 | if line == "" || !strings.Contains(line, "=") { 265 | continue 266 | } 267 | 268 | parts := strings.SplitN(line, "=", 2) 269 | name := parts[0] 270 | value := parts[1] 271 | field := v.FieldByName(snaker.SnakeToCamel(strings.ToLower(name))) 272 | // skip unknown values 273 | if !field.IsValid() { 274 | continue 275 | } 276 | // skip empty values 277 | if value == "" { 278 | continue 279 | } 280 | switch field.Kind() { 281 | case reflect.String: 282 | field.SetString(value) 283 | case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: 284 | i, err := strconv.ParseInt(value, 10, 64) 285 | if err != nil { 286 | return fmt.Errorf("can't parse variable %s with value %s: %v", name, value, err) 287 | } 288 | field.SetInt(i) 289 | case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: 290 | i, err := strconv.ParseUint(value, 10, 64) 291 | if err != nil { 292 | return fmt.Errorf("can't parse variable %s with value %s: %v", name, value, err) 293 | } 294 | field.SetUint(i) 295 | case reflect.Float32, reflect.Float64: 296 | i, err := strconv.ParseFloat(value, 64) 297 | if err != nil { 298 | return fmt.Errorf("can't parse variable %s with value %s: %v", name, value, err) 299 | } 300 | field.SetFloat(i) 301 | case reflect.Bool: 302 | if value == "true" { 303 | field.SetBool(true) 304 | } else { 305 | field.SetBool(false) 306 | } 307 | case reflect.Slice: 308 | if field.Type().Elem().Kind() != reflect.String { 309 | panic("only slices of strings are supported") 310 | } 311 | vs := strings.Split(value, ",") 312 | slice := reflect.MakeSlice(field.Type(), len(vs), len(vs)) 313 | for i, v := range vs { 314 | slice.Index(i).SetString(v) 315 | } 316 | field.Set(slice) 317 | default: 318 | panic(fmt.Sprintf("unsupported type: %v", field.Kind())) 319 | } 320 | } 321 | 322 | return scanner.Err() 323 | } 324 | 325 | // returns number of CPUs available to docker 326 | func dockerNumCPU() (int, error) { 327 | // use cli instead of connection to docker server directly 328 | // in case server exposed by http or non-default socket path 329 | info, err := exec.Command("docker", "info", "--format", "{{.NCPU}}").Output() 330 | if err != nil { 331 | return 0, err 332 | } 333 | 334 | cpus, err := strconv.Atoi(strings.TrimSpace(string(info))) 335 | if err != nil || cpus == 0 { 336 | return 0, fmt.Errorf("Couldn't get number of available CPUs in docker") 337 | } 338 | 339 | return cpus, nil 340 | } 341 | 342 | // returns total memory in bytes available to docker 343 | func dockerTotalMem() (uint64, error) { 344 | info, err := exec.Command("docker", "info", "--format", "{{.MemTotal}}").Output() 345 | if err != nil { 346 | return 0, err 347 | } 348 | 349 | mem, err := strconv.ParseUint(strings.TrimSpace(string(info)), 10, 64) 350 | if err != nil || mem == 0 { 351 | return 0, fmt.Errorf("Couldn't get of available memory in docker") 352 | } 353 | 354 | return mem, nil 355 | } 356 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | All notable changes to this project will be documented in this file. 4 | 5 | The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), 6 | and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). 7 | 8 | The changes listed under `Unreleased` section have landed in master but are not yet released. 9 | 10 | 11 | ## [Unreleased] 12 | 13 | ### Components 14 | 15 | - `bblfsh/bblfshd` has been updated to [v2.15.0](https://github.com/bblfsh/bblfshd/releases/tag/v2.15.0). 16 | - `bblfsh/web` has been updated to [v0.11.4](https://github.com/bblfsh/web/releases/tag/v0.11.4). 17 | - Use the same logging level as the other components reading `LOG_LEVEL` enviroment value (default: `info`) (([#263](https://github.com/src-d/sourced-ce/pull/263)). 18 | - `srcd/sourced-ui` has been updated to [v0.8.1](https://github.com/src-d/sourced-ui/releases/tag/v0.8.1). 19 | 20 | 21 | ### Fixed 22 | 23 | - Identify and show errors for old unsupported version of docker/docker-compose ([#253](https://github.com/src-d/sourced-ce/issues/253)) 24 | 25 | ## [v0.17.0](https://github.com/src-d/sourced-ce/releases/tag/v0.17.0) - 2019-10-01 26 | 27 | ### Components 28 | 29 | - `srcd/sourced-ui` has been updated to [v0.7.0](https://github.com/src-d/sourced-ui/releases/tag/v0.7.0). 30 | - `srcd/gitcollector` has been updated to [v0.0.4](https://github.com/src-d/gitcollector/releases/tag/v0.0.4). 31 | 32 | ### Fixed 33 | 34 | - More detailed error messages for file downloads ([#245](https://github.com/src-d/sourced-ce/pull/245)). 35 | 36 | ### Changed 37 | 38 | - Make `sourced-ui` Superset celery workers run as separate containers ([#269](https://github.com/src-d/sourced-ui/issues/269)). 39 | - Remove need for `docker-compose.override.yml` ([#252](https://github.com/src-d/sourced-ui/issues/252)). 40 | 41 | ### Internal 42 | 43 | - Development and building of source{d} CE now requires `go 1.13` ([#242](https://github.com/src-d/sourced-ce/pull/242)). 44 | 45 | ### Upgrading 46 | 47 | Install the new `v0.17.0` binary, then run `sourced compose download`. Because of a change in the `docker-compose.yml` file version, you must delete the file `~/.sourced/compose-files/__active__/docker-compose.override.yml` manually. 48 | 49 | If you had a deployment running, you must re-deploy the containers with `sourced restart`. All your existing data will continue to work after the upgrade. 50 | 51 | ```shell 52 | $ sourced version 53 | sourced version v0.16.0 54 | 55 | $ rm ~/.sourced/compose-files/__active__/docker-compose.override.yml 56 | 57 | $ sourced compose download 58 | Docker compose file successfully downloaded to your ~/.sourced/compose-files directory. It is now the active compose file. 59 | To update your current installation use `sourced restart` 60 | 61 | $ sourced restart 62 | ``` 63 | 64 | 65 | ## [v0.16.0](https://github.com/src-d/sourced-ce/releases/tag/v0.16.0) - 2019-09-16 66 | 67 | ### Components 68 | 69 | - `srcd/sourced-ui` has been updated to [v0.6.0](https://github.com/src-d/sourced-ui/releases/tag/v0.6.0). 70 | - `bblfsh/web` has been updated to [v0.11.3](https://github.com/bblfsh/web/releases/tag/v0.11.3). 71 | 72 | ### Fixed 73 | 74 | - Increase the timeout for the `start` command ([#219](https://github.com/src-d/sourced-ce/pull/219)). 75 | 76 | ### Changed 77 | 78 | - `sourced compose list` shows an index number for each compose entry, and `sourced compose set` now accepts both the name or the index number (@cmbahadir) ([#199](https://github.com/src-d/sourced-ce/issues/199)). 79 | 80 | ### Upgrading 81 | 82 | Install the new `v0.16.0` binary, then run `sourced compose download`. If you had a deployment running, you can re-deploy the containers with `sourced restart`. 83 | 84 | Please note: `sourced-ui` contains changes to the color palettes for the default dashboard charts, and these changes will only be visible when you run `sourced init local/org` with a new path or organization. This is a cosmetic improvement that you can ignore safely. 85 | 86 | If you want to apply the new default dashboards over your existing deployment, you will need to run `sourced prune` (or `sourced prune --all`) and `sourced init local/org` again. 87 | 88 | Important: running `prune` will delete all your current data and customizations, including charts or dashboards. You can choose to not `prune` your existing deployments, keeping you previous default dashboards and charts. 89 | 90 | ```shell 91 | $ sourced version 92 | sourced version v0.16.0 93 | 94 | $ sourced compose download 95 | Docker compose file successfully downloaded to your ~/.sourced/compose-files directory. It is now the active compose file. 96 | To update your current installation use `sourced restart` 97 | 98 | $ sourced status workdirs 99 | bblfsh 100 | * src-d 101 | 102 | $ sourced prune --all 103 | $ sourced init orgs src-d 104 | $ sourced init orgs bblfsh 105 | ``` 106 | 107 | ## [v0.15.1](https://github.com/src-d/sourced-ce/releases/tag/v0.15.1) - 2019-08-27 108 | 109 | ### Fixed 110 | 111 | - Fix incompatibility of empty resource limits ([#227](https://github.com/src-d/sourced-ce/issues/227)). 112 | - Fix incorrect value for `GITCOLLECTOR_LIMIT_CPU` in some cases ([#225](https://github.com/src-d/sourced-ce/issues/225)). 113 | - Fix gitbase `LOG_LEVEL` environment variable in the compose file ([#228](https://github.com/src-d/sourced-ce/issues/228)). 114 | 115 | ### Removed 116 | 117 | - Remove the `completion` sub-command on Windows, as it only works for bash ([#169](https://github.com/src-d/sourced-ce/issues/169)). 118 | 119 | ### Upgrading 120 | 121 | Install the new `v0.15.1` binary, then run `sourced compose download`. 122 | 123 | For an upgrade from `v0.15.0`, you just need to run `sourced restart` to re-deploy the containers. 124 | 125 | For an upgrade from `v0.14.0`, please see the upgrade instructions in the release notes for `v0.15.0`. 126 | 127 | 128 | ## [v0.15.0](https://github.com/src-d/sourced-ce/releases/tag/v0.15.0) - 2019-08-21 129 | 130 | ### Components 131 | 132 | - `srcd/sourced-ui` has been updated to [v0.5.0](https://github.com/src-d/sourced-ui/releases/tag/v0.5.0). 133 | - `srcd/ghsync` has been updated to [v0.2.0](https://github.com/src-d/ghsync/releases/tag/v0.2.0). 134 | 135 | ### Added 136 | 137 | - Add a monitoring of containers state while waiting for the web UI to open during initialization ([#147](https://github.com/src-d/sourced-ce/issues/147)). 138 | - Exclude forks by default in `sourced init orgs`, adding a new flag `--with-forks` to include them if needed ([#109](https://github.com/src-d/sourced-ce/issues/109)). 139 | 140 | ### Changed 141 | 142 | - Refactor of the `status` command ([#203](https://github.com/src-d/sourced-ce/issues/203)): 143 | - `sourced status components` shows the previous output of `sourced status` 144 | - `sourced status workdirs` replaces `sourced workdirs` 145 | - `sourced status config` shows the contents of the Docker Compose environment variables. This is useful, for example, to check if the active working directory was configured to include or skip forks when downloading the data from GitHub 146 | - `sourced status all` shows all of the above 147 | 148 | ### Upgrading 149 | 150 | Install the new `v0.15.0` binary, then run `sourced compose download`. If you had a deployment running, you can re-deploy the containers with `sourced restart`. 151 | 152 | Please note: `sourced-ui` contains fixes for the default dashboard charts that will only be visible when you run `sourced init local/org` with a new path or organization. 153 | If you want to apply the new default dashboards over your existing deployment, you will need to run `sourced prune` (or `sourced prune --all`) and `sourced init local/org` again. 154 | 155 | Important: running `prune` will delete all your current data and customizations, including charts or dashboards. You can choose to not `prune` your existing deployments, keeping you previous default dashboards and charts. 156 | 157 | ```shell 158 | $ sourced version 159 | sourced version v0.15.0 build 08-21-2019_08_30_24 160 | 161 | $ sourced compose download 162 | Docker compose file successfully downloaded to your ~/.sourced/compose-files directory. It is now the active compose file. 163 | To update your current installation use `sourced restart` 164 | 165 | $ sourced status workdirs 166 | bblfsh 167 | * src-d 168 | 169 | $ sourced prune --all 170 | $ sourced init orgs src-d 171 | $ sourced init orgs bblfsh 172 | ``` 173 | 174 | ## [v0.14.0](https://github.com/src-d/sourced-ce/releases/tag/v0.14.0) - 2019-08-07 175 | 176 | Initial release of **source{d} Community Edition (CE)**, the data platform for your software development life cycle. 177 | 178 | The `sourced` binary is a wrapper for Docker Compose that downloads the `docker-compose.yml` file from this repository, and includes the following sub commands: 179 | 180 | - `init`: Initialize source{d} to work on local or GitHub organization datasets 181 | - `local`: Initialize source{d} to analyze local repositories 182 | - `orgs`: Initialize source{d} to analyze GitHub organizations 183 | - `status`: Show the status of all components 184 | - `stop`: Stop any running components 185 | - `start`: Start any stopped components 186 | - `logs`: Show logs from components 187 | - `web`: Open the web interface in your browser 188 | - `sql`: Open a MySQL client connected to a SQL interface for Git 189 | - `prune`: Stop and remove components and resources 190 | - `workdirs` List all working directories 191 | - `compose`: Manage source{d} docker compose files 192 | - `download`: Download docker compose files 193 | - `list`: List the downloaded docker compose files 194 | - `set`: Set the active docker compose file 195 | - `restart`: Update current installation according to the active docker compose file 196 | 197 | ### Known Issues 198 | 199 | - On Windows, if you use `sourced init local` on a directory with a long path, you may encounter the following error: 200 | ``` 201 | Can't find a suitable configuration file in this directory or any 202 | parent. Are you in the right directory? 203 | ``` 204 | 205 | This is caused by the [`MAX_PATH` limitation on windows](https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation). The only workaround is to move the target directory to a shorter path, closer to the root of your drive ([#191](https://github.com/src-d/sourced-ce/issues/191)). 206 | 207 | - Linux only: Docker installed from snap packages is not supported, please install it following [the official documentation](https://docs.docker.com/install/) ([#78](https://github.com/src-d/sourced-ce/issues/78)). 208 | 209 | ### Upgrading 210 | 211 | For internal releases we don't support upgrading. If you have a previous `sourced-ce` pre-release version installed, clean up all your data **before** downloading this release. This will delete everything, including the UI data for dashboards, charts, users, etc: 212 | 213 | ```shell 214 | sourced prune --all 215 | rm -rf ~/.sourced 216 | ``` 217 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/PuerkitoBio/goquery v1.5.0 h1:uGvmFXOA73IKluu/F84Xd1tt/z07GYm8X49XKHP7EJk= 2 | github.com/PuerkitoBio/goquery v1.5.0/go.mod h1:qD2PgZ9lccMbQlc7eEOjaeRlFQON7xY8kdmcsrnKqMg= 3 | github.com/andybalholm/cascadia v1.0.0 h1:hOCXnnZ5A+3eVDX8pvgl4kofXv2ELss0bKcqRySc45o= 4 | github.com/andybalholm/cascadia v1.0.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y= 5 | github.com/blang/semver v3.5.1+incompatible h1:cQNTCjp13qL8KC3Nbxr/y2Bqb63oX6wdnnjpJbkM4JQ= 6 | github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= 7 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 8 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 9 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 10 | github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I= 11 | github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= 12 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 13 | github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg= 14 | github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 15 | github.com/google/btree v1.0.0 h1:0udJVsspx3VBr5FwtLhQQtuAsVc79tTq0ocGIPAU6qo= 16 | github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= 17 | github.com/google/go-cmp v0.2.0 h1:+dTQ8DZQJz0Mb/HjFlkptS1FeQ4cWSnN941F8aEG4SQ= 18 | github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= 19 | github.com/google/go-github/v25 v25.1.1 h1:6eW++i/CXcR5GKfYaaJT7oJJtHNU+/iiw55noEPNVao= 20 | github.com/google/go-github/v25 v25.1.1/go.mod h1:6z5pC69qHtrPJ0sXPsj4BLnd82b+r6sLB7qcBoRZqpw= 21 | github.com/google/go-querystring v1.0.0 h1:Xkwi/a1rcvNg1PPYe5vI8GbeBY/jrVuDX5ASuANWTrk= 22 | github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck= 23 | github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc h1:f8eY6cV/x1x+HLjOp4r72s/31/V2aTUtg5oKRRPf8/Q= 24 | github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA= 25 | github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI= 26 | github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= 27 | github.com/jessevdk/go-flags v1.4.0 h1:4IU2WS7AumrZ/40jfhf4QVDMsQwqA7VEHozFRrGARJA= 28 | github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI= 29 | github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d h1:cVtBfNW5XTHiKQe7jDaDBSh/EVM4XLPutLAGboIXuM0= 30 | github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d/go.mod h1:P2viExyCEfeWGU259JnaQ34Inuec4R38JCyBx2edgD0= 31 | github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk= 32 | github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= 33 | github.com/lib/pq v1.1.1 h1:sJZmqHoEaY7f+NPP8pgLB/WxulyR3fewgCM2qaSlBb4= 34 | github.com/lib/pq v1.1.1/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo= 35 | github.com/mattn/go-colorable v0.1.1 h1:G1f5SKeVxmagw/IyvzvtZE4Gybcc4Tr1tf7I8z0XgOg= 36 | github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ= 37 | github.com/mattn/go-isatty v0.0.5 h1:tHXDdz1cpzGaovsTB+TVB8q90WEokoVmfMqoVcrLUgw= 38 | github.com/mattn/go-isatty v0.0.5/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= 39 | github.com/mattn/go-isatty v0.0.7 h1:UvyT9uN+3r7yLEYSlJsbQGdsaB/a0DlgWP3pql6iwOc= 40 | github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= 41 | github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b h1:j7+1HpAFS1zy5+Q4qx1fWh90gTKwiN4QCGoY9TWyyO4= 42 | github.com/mgutz/ansi v0.0.0-20170206155736-9520e82c474b/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE= 43 | github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 44 | github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w= 45 | github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 46 | github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo= 47 | github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= 48 | github.com/pbnjay/memory v0.0.0-20190104145345-974d429e7ae4 h1:MfIUBZ1bz7TgvQLVa/yPJZOGeKEgs6eTKUjz3zB4B+U= 49 | github.com/pbnjay/memory v0.0.0-20190104145345-974d429e7ae4/go.mod h1:RMU2gJXhratVxBDTFeOdNhd540tG57lt9FIUV0YLvIQ= 50 | github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI= 51 | github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU= 52 | github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4 h1:49lOXmGaUpV9Fz3gd7TFZY106KVlPVa5jcYD1gaQf98= 53 | github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4/go.mod h1:4OwLy04Bl9Ef3GJJCoec+30X3LQs/0/m4HFRt/2LUSA= 54 | github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 55 | github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= 56 | github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 57 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 58 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 59 | github.com/serenize/snaker v0.0.0-20171204205717-a683aaf2d516 h1:ofR1ZdrNSkiWcMsRrubK9tb2/SlZVWttAfqUjJi6QYc= 60 | github.com/serenize/snaker v0.0.0-20171204205717-a683aaf2d516/go.mod h1:Yow6lPLSAXx2ifx470yD/nUe22Dv5vBvxK/UK9UUTVs= 61 | github.com/sirupsen/logrus v1.4.1 h1:GL2rEmy6nsikmW0r8opw9JIRScdMF5hA8cOYLH7In1k= 62 | github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q= 63 | github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= 64 | github.com/src-d/envconfig v1.0.0 h1:/AJi6DtjFhZKNx3OB2qMsq7y4yT5//AeSZIe7rk+PX8= 65 | github.com/src-d/envconfig v1.0.0/go.mod h1:Q9YQZ7BKITldTBnoxsE5gOeB5y66RyPXeue/R4aaNBc= 66 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 67 | github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 68 | github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w= 69 | github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= 70 | github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q= 71 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 72 | github.com/x-cray/logrus-prefixed-formatter v0.5.2 h1:00txxvfBM9muc0jiLIEAkAcIMJzfthRT6usrui8uGmg= 73 | github.com/x-cray/logrus-prefixed-formatter v0.5.2/go.mod h1:2duySbKsL6M18s5GU7VPsoEPHyzalCE06qoARUCeBBE= 74 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 75 | golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734 h1:p/H982KKEjUnLJkM3tt/LemDnOc1GiZL5FCVlORJ5zo= 76 | golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 77 | golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 78 | golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 79 | golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 80 | golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 81 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3 h1:0GoQqolDA55aaLxZyTzK/Y2ePZzZTUrRacwib7cNsYQ= 82 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 83 | golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c h1:uOCk1iQW6Vc18bnC13MfzScl+wdKBmM9Y9kU7Z83/lw= 84 | golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 85 | golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= 86 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 87 | golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 88 | golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 89 | golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 90 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 91 | golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 92 | golang.org/x/sys v0.0.0-20190412213103-97732733099d h1:+R4KGOnez64A81RvjARKc4UT5/tI9ujCIVX+P5KiHuI= 93 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 94 | golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= 95 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 96 | golang.org/x/tools v0.0.0-20180810170437-e96c4e24768d/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 97 | google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= 98 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= 99 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 100 | gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4= 101 | gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= 102 | gopkg.in/src-d/go-cli.v0 v0.0.0-20190422143124-3a646154da79 h1:MBr8uUjT5gZe4udsLGNZ3nWy/+ck0LSB91mQjgpwLUI= 103 | gopkg.in/src-d/go-cli.v0 v0.0.0-20190422143124-3a646154da79/go.mod h1:z+K8VcOYVYcSwSjGebuDL6176A1XskgbtNl64NSg+n8= 104 | gopkg.in/src-d/go-errors.v1 v1.0.0 h1:cooGdZnCjYbeS1zb1s6pVAAimTdKceRrpn7aKOnNIfc= 105 | gopkg.in/src-d/go-errors.v1 v1.0.0/go.mod h1:q1cBlomlw2FnDBDNGlnh6X0jPihy+QxZfMMNxPCbdYg= 106 | gopkg.in/src-d/go-log.v1 v1.0.2 h1:dED4100pntH4l3qOTgD1xebQR6pVU8tuPbUCmqiMsb0= 107 | gopkg.in/src-d/go-log.v1 v1.0.2/go.mod h1:GN34hKP0g305ysm2/hctJ0Y8nWP3zxXXJ8GFabTyABE= 108 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= 109 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= 110 | gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 111 | gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= 112 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 113 | gotest.tools v0.0.0-20181223230014-1083505acf35 h1:zpdCK+REwbk+rqjJmHhiCN6iBIigrZ39glqSF0P3KF0= 114 | gotest.tools v0.0.0-20181223230014-1083505acf35/go.mod h1:R//lfYlUuTOTfblYI3lGoAAAebUdzjvbmQsuB7Ykd90= 115 | -------------------------------------------------------------------------------- /docs/learn-more/troubleshooting.md: -------------------------------------------------------------------------------- 1 | 2 | # Troubleshooting: 3 | 4 | _For commonly asked questions and their answers, you can refer to the [FAQ](./faq.md)_ 5 | 6 | Currently, **source{d} CE** does not expose nor log all errors directly into the 7 | UI. In the current stage of **source{d} CE**, following these steps is the 8 | better way to know if something is failing, why, and to know how to recover the 9 | app from some problems. The first two steps use to be always mandatory: 10 | 11 | 1. **[To see if any component is broken](#how-can-i-see-the-status-of-source-d-ce-components)** 12 | 1. **[To see the logs of the running components](#how-can-i-see-logs-of-the-running-components)** 13 | 1. [To know if scrapers finished their job](#how-can-i-see-what-happened-with-the-scrapers) 14 | - [To restart one scraper](#how-can-i-restart-one-scraper) 15 | 1. [To restart o initialize **source{d} CE** again](#how-to-restart-source-d-ce) 16 | 1. [To ask for help if the issue could not be solved](./faq.md#where-can-i-find-more-assistance-to-run-source-d-or-notify-you-about-any-issue-or-suggestion) 17 | 18 | Other issues that we detected, and which are strictly related to the UI are: 19 | 20 | - [When I Try to Create a Chart from a Query, Nothing Happens.](#when-i-try-to-create-a-chart-from-a-query-nothing-happens) 21 | - [When I Try to Export a Dashboard, Nothing Happens.](#when-i-try-to-export-a-dashboard-nothing-happens) 22 | - [The Dashboard Takes a Long to Load, and the UI Freezes.](#the-dashboard-takes-a-long-to-load-and-the-ui-freezes) 23 | 24 | 25 | ## source{d} CE Fails During Its Initialization 26 | 27 | The initialization can fail fast if there is any port conflict, or missing config 28 | file, etcetera; those errors are clearly logged in the terminal when they appear. 29 | 30 | If when initializing **source{d} CE**, all the required components appear as created, 31 | but the loading spinner keeps spinning forever (more than 1 minute can be symptomatic), 32 | there can be an underlying problem causing the UI not to be opened. In this 33 | situation you should: 34 | 35 | 1. **[See if any component is broken](#how-can-i-see-the-status-of-source-d-ce-components)** 36 | 1. **[See app logs or certain component logs](#how-can-i-see-logs-of-the-running-components)** 37 | 1. [Restart o initialize **source{d} CE** again](#how-to-restart-source-d-ce) 38 | 1. [To ask for help if the issue could not be solved](./faq.md#where-can-i-find-more-assistance-to-run-source-d-or-notify-you-about-any-issue-or-suggestion) 39 | 40 | 41 | ## How Can I See the Status of source{d} CE Components? 42 | 43 | To see the status of **source{d} CE** components, just run: 44 | 45 | ``` 46 | $ sourced status 47 | 48 | Name Command State Ports 49 | ------------------------------------------------------------------------------ 50 | srcd-xxx_sourced-ui_1 /entrypoint.sh Up (healthy) :8088->8088 51 | srcd-xxx_gitbase_1 ./init.sh Up :3306->3306 52 | srcd-xxx_bblfsh_1 /tini -- bblfshd Up :9432->9432 53 | srcd-xxx_bblfsh-web_1 /bin/bblfsh-web -addr ... Up :9999->8080 54 | srcd-xxx_metadatadb_1 docker-entrypoint.sh ... Up :5433->5432 55 | srcd-xxx_postgres_1 docker-entrypoint.sh ... Up :5432->5432 56 | srcd-xxx_redis_1 docker-entrypoint.sh ... Up :6379->6379 57 | srcd-xxx_ghsync_1 /bin/sh -c sleep 10s ... Exit 0 58 | srcd-xxx_gitcollector_1 /bin/dumb-init -- /bi ... Exit 0 59 | 60 | ``` 61 | 62 | It will report the status of all **source{d} CE** component. All components should 63 | be `Up`, but the scrapers: `ghsync` and `gitcollector`; these exceptions are 64 | explanined in [How Can I See What Happened with the Scrapers?](#how-can-i-see-what-happened-with-the-scrapers) 65 | 66 | If any component is not `Up` (but the scrapers), here are some key points to 67 | understand what might be happening: 68 | 69 | - All the components (but the scrapers) are restarted by Docker Compose 70 | automatically —process that can take some seconds—; if the component 71 | enters in a restart loop, something wrong is happening. 72 | - When any component is failing, or died, you should 73 | [see its logs to understand what is happening](#how-can-i-see-logs-of-the-running-components) 74 | 75 | When one of the required components fails, it uses to print an error in the UI, 76 | 77 | e.g. `lost connection to mysql server during query` while running a query might 78 | mean that `gitbase` went down. 79 | 80 | e.g. `unable to establish a connection with the bblfsh server: deadline exceeded` 81 | in SQL Lab might mean that `bblfsh` went down. 82 | 83 | If the failing component is not successfully restarted in a few seconds, or if it 84 | goes down when running certain queries, it could be a good idea to [open an issue](https://github.com/src-d/sourced-ce/issues) 85 | describing the problem. 86 | 87 | 88 | ## How Can I See Logs of The Running Components? 89 | 90 | ```shell 91 | $ sourced logs [-f] [components...] 92 | ``` 93 | 94 | Adding `-f` will keep the connection opened, and the logs will appear as they 95 | come instead of exiting after the last logged one. 96 | 97 | You can pass a space-separated list of component names to see only their logs 98 | (i.e. `sourced-ui`, `gitbase`, `bblfsh`, `gitcollector`, `ghsync`, `metadatadb`, `postgres`, `redis`). 99 | If you do not pass any component name, there will appear the logs of all of them. 100 | 101 | Currently, there is no way to filter by error level, so you could try with `grep`, 102 | e.g. 103 | 104 | ```shell 105 | sourced logs gitcollector | grep error 106 | ``` 107 | 108 | will output only log lines where `error` word appears. 109 | 110 | 111 | ## How Can I See What Happened with the Scrapers? 112 | 113 | _When **souece{d} CE** is initialized with `sourced init local`, the scrapers are 114 | not relevant because the repositories to analyze comes from your local data, so 115 | `ghsync` and `gitcollector` status is not relevant in this case._ 116 | 117 | When running **souece{d} CE** to analyze data from a list of GitHub organizations, 118 | `gitcollector` component is in charge of fetching GitHub repositories and `ghsync` 119 | component is in charge of fetching GitHub metadata (issues, pull requests...) 120 | 121 | Once the UI is opened, you can see the progress of the importation in the welcome 122 | dashboard, reporting the data imported, skipped, failed and completed. The process 123 | can take many minutes if the organization is big, so be patient. You can manually 124 | refresh both charts to confirm that the process is progressing, and it is not stuck. 125 | If you believe that there can be any problem during the process, the better way 126 | to find what is happening is: 127 | 128 | - **[check the components status](#how-can-i-see-the-status-of-source-d-ce-components) 129 | with `sourced status`**; `gitcollector` and `ghsync` should be `Up` (the process 130 | didn't finish yet), or `Exit 0` (the process finished succesfully). They are 131 | independent components, so they can finish on different order depending on how 132 | many repositories or metadata is needed to process. 133 | 134 | - **[check the logs](#how-can-i-see-logs-of-the-running-components) of the failing component with `sourced logs [-f] {gitcollector,ghsync}`** 135 | to get more info about the errors found. 136 | 137 | 138 | ## How Can I Restart One Scraper? 139 | 140 | _Restarting a scraper should be done to recover from temporal problems like 141 | connectivity loss, or lack of space in disc, not 142 | [to update the data you're analyzing](./faq.md#how-to-update-the-data-from-the-organizations-being-analyzed)_ 143 | 144 | **source{d} CE** does not provide way to start only one scraper. The recommended way 145 | to restart them would be [to restart the whole **source{d} CE**](#how-to-restart-source-d-ce), 146 | which is fast and safe for your data. In order to restart **source{d} CE**, run: 147 | 148 | ```shell 149 | $ sourced restart 150 | ``` 151 | 152 | _Read more about [which data will be imported after restarting a scraper](./faq.md#how-to-update-the-data-from-the-organizations-being-Analyzed)_ 153 | 154 | If you feel comfortable enough with Docker Compose, you could also try restarting 155 | each scraper separatelly, running: 156 | 157 | ```shell 158 | $ cd ~/.sourced/workdirs/__active__ 159 | $ docker-compose run gitcollector # to restart gitcollector 160 | $ docker-compose run ghsync # to restart ghsync 161 | ``` 162 | 163 | 164 | ## How to Restart source{d} CE 165 | 166 | Restarting **source{d} CE**, can fix some errors and is also the official way to 167 | restart the scrapers. It is also needed after downloading a new config (by running 168 | `sourced compose download`). **source{d} CE** is restarted with the command: 169 | 170 | ```shell 171 | $ sourced restart 172 | ``` 173 | 174 | It only recreates the component containers, keeping all your data, like charts, 175 | dashboards, repositories, and GitHub metadata. 176 | 177 | 178 | ## When I Try to Create a Chart from a Query, Nothing Happens. 179 | 180 | The charts can be created from the SQL Lab, using the `Explore` button once you 181 | run a query. If nothing happens, the browser may be blocking the new window that 182 | should be opened to edit the new chart. You should configure your browser to let 183 | source{d} UI to open pop-ups (e.g. in Chrome it is done allowing `127.0.0.1:8088` 184 | to handle `pop-ups and redirects` from the `Site Settings` menu). 185 | 186 | 187 | ## When I Try to Export a Dashboard, Nothing Happens. 188 | 189 | If nothing happens when pressing the `Export` button from the dashboard list, then 190 | you should configure your browser to let source{d} UI to open pop-ups (e.g. in 191 | Chrome it is done allowing `127.0.0.1:8088` to handle `pop-ups and redirects` 192 | from the `Site Settings` menu) 193 | 194 | 195 | ## The Dashboard Takes a Long to Load and the UI Freezes. 196 | 197 | _This is a known issue that we're trying to address, but here is more info about it._ 198 | 199 | In some circumstances, loading the data for the dashboards can take some time, 200 | and the UI can be frozen in the meanwhile. It can happen —on big datasets—, 201 | the first time you access the dashboards, or when they are refreshed. 202 | 203 | There are some limitations with how Apache Superset handles long-running SQL 204 | queries, which may affect the dashboard charts. Since most of the charts of the 205 | Overview dashboard loads its data from gitbase, its queries can take more time 206 | than the expected for the UI. 207 | 208 | When it happens, the UI can be frozen, or you can get this message in some charts: 209 | >_Query timeout - visualization queries are set to timeout at 300 seconds. 210 | Perhaps your data has grown, your database is under unusual load, or you are 211 | simply querying a data source that is too large to be processed within the timeout 212 | range. If that is the case, we recommend that you summarize your data further._ 213 | 214 | When it occurs, you should wait till the UI is responsive again, and separately 215 | refresh each failing chart with its `force refresh` option (on its top-right corner). 216 | With some big datasets, it took 3 refreshes and 15 minutes to get data for all charts. 217 | --------------------------------------------------------------------------------