├── .gitignore ├── AUTHORS ├── CONTRIBUTING.md ├── Gemfile ├── Gemfile.lock ├── LICENSE ├── README.md ├── _config.yml ├── assets ├── Selection_095.png ├── chaoss-logo.png ├── chaoss-logo.svg ├── eclipse.png ├── favicon │ ├── android-icon-144x144.png │ ├── android-icon-192x192.png │ ├── android-icon-36x36.png │ ├── android-icon-48x48.png │ ├── android-icon-72x72.png │ ├── android-icon-96x96.png │ ├── apple-icon-114x114.png │ ├── apple-icon-120x120.png │ ├── apple-icon-144x144.png │ ├── apple-icon-152x152.png │ ├── apple-icon-180x180.png │ ├── apple-icon-57x57.png │ ├── apple-icon-60x60.png │ ├── apple-icon-72x72.png │ ├── apple-icon-76x76.png │ ├── apple-icon-precomposed.png │ ├── apple-icon.png │ ├── browserconfig.xml │ ├── favicon-16x16.png │ ├── favicon-32x32.png │ ├── favicon-96x96.png │ ├── favicon.ico │ ├── manifest.json │ ├── ms-icon-144x144.png │ ├── ms-icon-150x150.png │ ├── ms-icon-310x310.png │ └── ms-icon-70x70.png ├── grimoirelab-3.png ├── grimoirelab-all-details.png ├── grimoirelab-all-details.svg ├── grimoirelab-all.png ├── grimoirelab-all.svg ├── grimoirelab-chaoss-banner.png ├── grimoirelab-chaoss.png ├── grimoirelab-logo.png ├── grimoirelab-logo.svg ├── grimoirelab.png ├── grimoirelab.svg └── howto-perceval-backends.png ├── docs ├── cases-chaoss │ ├── activity.md │ ├── dashboard_chaoss_community_activity-2y-perceval.png │ ├── dashboard_chaoss_community_activity-2y.png │ ├── dashboard_chaoss_community_activity.png │ ├── dashboards │ │ └── chaoss_community_activity.json │ ├── discover_indexes.png │ ├── index.md │ ├── index_patterns.png │ ├── meetup.md │ └── scripts │ │ └── retrieve.py ├── community-health │ ├── code-dev-activity.md │ ├── code-dev-efficiency.md │ ├── contributions.md │ ├── index.md │ ├── issue-resolution.md │ ├── people.md │ └── time.md ├── components │ ├── gelk │ │ ├── dashboards │ │ │ ├── git-dashboard.json │ │ │ ├── git-sortinghat.json │ │ │ └── github-dashboard.json │ │ ├── figs │ │ │ ├── dashboard-git-sortinghat.png │ │ │ ├── elasticsearch-index.png │ │ │ ├── kibana-dashboard.png │ │ │ └── meetup │ │ │ │ ├── bar-chart.png │ │ │ │ ├── bubbles-chart.png │ │ │ │ ├── groups-pie-chart.jpg │ │ │ │ ├── index-pattern-fields.png │ │ │ │ ├── kibana-metrics-viz.png │ │ │ │ ├── kibana-table-viz-def-1.png │ │ │ │ ├── kibana_index_pattern.png │ │ │ │ ├── meetings-evolution.png │ │ │ │ ├── meetings-table.jpg │ │ │ │ ├── meetup-stats-by-grimoirelab.jpg │ │ │ │ ├── meetup-stats.jpg │ │ │ │ ├── members-table.png │ │ │ │ ├── pie-chart-definition.png │ │ │ │ ├── url-field.png │ │ │ │ ├── visualize-kibana.jpg │ │ │ │ └── visualize-kibana.png │ │ ├── index.md │ │ ├── intro.md │ │ ├── kidash.md │ │ ├── simple.md │ │ └── sortinghat.md │ ├── graal │ │ ├── cocom.md │ │ ├── index.md │ │ ├── intro.md │ │ └── scripts │ │ │ └── graal_cocom_1.py │ ├── index.md │ ├── manuscripts │ │ ├── first.md │ │ └── index.md │ ├── sirmordred │ │ ├── container.md │ │ ├── dashboard.md │ │ ├── files │ │ │ ├── menu.yaml │ │ │ ├── mordred-simple.cfg │ │ │ └── projects.json │ │ ├── index.md │ │ ├── micro-mordred.md │ │ └── projects.md │ └── sortinghat-cli │ │ ├── basic.md │ │ ├── data.md │ │ └── index.md ├── contribute │ └── index.md ├── dashboards │ ├── access.md │ ├── add.md │ ├── assets │ │ ├── absolute-time-picker.png │ │ ├── add-button.png │ │ ├── customize.png │ │ ├── dashboard-option.png │ │ ├── dashboard.png │ │ ├── delete-dashboards.png │ │ ├── dev-output.png │ │ ├── dev-tool-interface.png │ │ ├── dev-tools.png │ │ ├── drag.gif │ │ ├── edit.png │ │ ├── edit_nav.png │ │ ├── empty_dashboard.png │ │ ├── fields.png │ │ ├── filter-active.png │ │ ├── filter.png │ │ ├── filters-and-time-picker.png │ │ ├── login-page.png │ │ ├── loginButton.png │ │ ├── navigate.png │ │ ├── new_dashboard.png │ │ ├── save-fields.png │ │ ├── save.png │ │ ├── time-picker.png │ │ └── visualize.png │ ├── edit.md │ ├── filter.md │ ├── index.md │ ├── navigate.md │ ├── query.md │ ├── remove.md │ └── visualization │ │ ├── add.md │ │ ├── assets │ │ ├── buckets_set.png │ │ ├── color.png │ │ ├── color_palette.png │ │ ├── delete-viz.png │ │ ├── index.png │ │ ├── metrics_set.png │ │ ├── save_visualization.png │ │ ├── visualization-type.png │ │ ├── visualization_interface.png │ │ ├── visualizations.png │ │ └── visualize.png │ │ ├── edit.md │ │ ├── index.md │ │ └── remove.md ├── data-sources │ ├── add-configurations.md │ ├── assets │ │ └── dashboard.png │ ├── configurations.md │ ├── index.md │ ├── schema.md │ └── supported.md ├── getting-started │ ├── assets │ │ ├── dashboard.png │ │ └── filters.png │ ├── configure.md │ ├── dev-setup.md │ ├── index.md │ ├── projects-json.md │ ├── setup-cfg.md │ ├── setup.md │ └── troubleshooting.md ├── install │ ├── dockerhub.md │ ├── figs │ │ ├── bitergia_analytics_screenshot.png │ │ ├── dashboard-grimoirelab.png │ │ └── kibana_welcome.png │ ├── index.md │ ├── install.md │ ├── quick.md │ └── supporting.md ├── python │ ├── es-dsl.md │ ├── es.md │ ├── index.md │ ├── indexes.md │ ├── pandas.md │ ├── querying.md │ ├── scripting.md │ └── scripts │ │ ├── cleanup.py │ │ ├── enriched_elasticsearch_1.py │ │ ├── pandas_1.py │ │ ├── perceval_elasticsearch_1.py │ │ ├── perceval_elasticsearch_2.py │ │ ├── perceval_elasticsearch_3.py │ │ ├── perceval_elasticsearch_4.py │ │ ├── perceval_elasticsearch_git_dsl.py │ │ ├── perceval_elasticsearch_git_dsl_2.py │ │ ├── perceval_elasticsearch_git_dsl_3.py │ │ ├── perceval_elasticsearch_mbox_1.py │ │ ├── perceval_elasticsearch_mbox_2.py │ │ └── perceval_elasticsearch_mbox_3.py ├── sortinghat │ ├── affiliations │ │ ├── add.md │ │ ├── assets │ │ │ ├── add-org.png │ │ │ ├── delete-org.png │ │ │ ├── edit-org.png │ │ │ └── un-enroll.png │ │ ├── edit.md │ │ ├── index.md │ │ ├── remove.md │ │ └── un-enroll.md │ ├── assets │ │ ├── sortinghat-no-identity.png │ │ ├── sortinghat-org.png │ │ ├── sortinghat-sameIdentity.png │ │ ├── sortinghat_individuals.png │ │ ├── sortinghat_orgs.png │ │ └── sortinghat_workspace.png │ ├── index.md │ ├── interface.md │ ├── profiles │ │ ├── add.md │ │ ├── assets │ │ │ ├── add-to-workspace.png │ │ │ ├── batch-delete.png │ │ │ ├── create-profile.png │ │ │ ├── drag-and-drop.png │ │ │ ├── edit-profile.png │ │ │ ├── expand.png │ │ │ ├── highlight-merge.png │ │ │ ├── is-bot.png │ │ │ ├── lock-profile.png │ │ │ ├── order-profiles.png │ │ │ ├── profiles.png │ │ │ ├── remove-profile.png │ │ │ ├── save-profile.png │ │ │ ├── set-bot.png │ │ │ ├── set-diff-bot.png │ │ │ ├── un-merge.png │ │ │ └── workspace-merge.png │ │ ├── bot.md │ │ ├── edit.md │ │ ├── filter.md │ │ ├── index.md │ │ ├── lock.md │ │ ├── merge.md │ │ ├── order.md │ │ ├── remove.md │ │ └── unmerge.md │ └── troubleshoot.md ├── terminology │ └── index.md ├── tools-and-tips │ ├── attraction-and-retention.md │ ├── copying-indexes.md │ ├── csv-from-jenkins-enriched-index.md │ ├── elasticsearch.md │ ├── html5-app-latest-activity.md │ ├── html5_app.png │ ├── html5_app_moving.gif │ ├── index.md │ ├── perceval.md │ └── scripts │ │ ├── C_dashboard.json │ │ ├── C_searches.json │ │ ├── C_visualizations.json │ │ ├── elastic_cp.py │ │ ├── elastic_last.py │ │ ├── elastic_projects_6.py │ │ ├── enriched_elasticsearch_git_pandas7.py │ │ ├── enriched_elasticsearch_jenkins.py │ │ ├── enriched_elasticsearch_newcomers.py │ │ ├── events.css │ │ ├── events.js │ │ ├── events.json │ │ ├── index.html │ │ └── perceval_git_counter.py └── workflow │ ├── assets │ ├── grimoirelab-all-complete.png │ ├── grimoirelab-all-complete.svg │ ├── grimoirelab-all-dashboard-noarthur-nomordred-nopanels-nosh.png │ ├── grimoirelab-all-dashboard-noarthur-nomordred-nopanels-nosh.svg │ ├── grimoirelab-all-dashboard-noarthur-nomordred-nopanels.png │ ├── grimoirelab-all-dashboard-noarthur-nomordred-nopanels.svg │ ├── grimoirelab-all-dashboard-noarthur-nomordred.png │ ├── grimoirelab-all-dashboard-noarthur-nomordred.svg │ ├── grimoirelab-all-dashboard-noarthur.png │ ├── grimoirelab-all-dashboard-noarthur.svg │ ├── grimoirelab-all-dashboard.png │ ├── grimoirelab-all-dashboard.svg │ ├── grimoirelab-all-everything.svg │ ├── grimoirelab-all-grimoireelk-enrich-sh.svg │ ├── grimoirelab-all-grimoireelk-enrich.svg │ ├── grimoirelab-all-grimoireelk-raw.png │ ├── grimoirelab-all-grimoireelk-raw.svg │ ├── grimoirelab-all-perceval-1.png │ ├── grimoirelab-all-perceval-1.svg │ ├── grimoirelab-all-perceval-n.png │ ├── grimoirelab-all-perceval-n.svg │ ├── grimoirelab-all-reports.png │ ├── grimoirelab-all-reports.svg │ ├── grimoirelab-fit-grimoireelk-enrich-sh.png │ ├── grimoirelab-fit-grimoireelk-enrich-sh.svg │ ├── grimoirelab-fit-grimoireelk-enrich.png │ ├── grimoirelab-fit-grimoireelk-enrich.svg │ ├── grimoirelab-fit-grimoireelk-raw.png │ ├── grimoirelab-fit-grimoireelk-raw.svg │ ├── grimoirelab-fit-perceval-1.png │ ├── grimoirelab-fit-perceval-1.svg │ ├── grimoirelab-fit-perceval-n.png │ └── grimoirelab-fit-perceval-n.svg │ ├── index.md │ ├── scenarios.md │ └── workflow.md ├── favicon.ico └── index.md /.gitignore: -------------------------------------------------------------------------------- 1 | # Jekyll 2 | _site/ 3 | .sass-cache/ 4 | .jekyll-cache/ 5 | .jekyll-metadata 6 | 7 | # Other 8 | .DS_Store 9 | .idea/ 10 | -------------------------------------------------------------------------------- /AUTHORS: -------------------------------------------------------------------------------- 1 | For a list of all our amazing authors please see the contributors page: 2 | https://github.com/chaoss/grimoirelab-tutorial/graphs/contributors 3 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source "https://rubygems.org" 2 | 3 | gem 'github-pages', group: :jekyll_plugins 4 | 5 | gem "just-the-docs" 6 | 7 | gem 'jekyll-redirect-from' 8 | 9 | gem "webrick", "~> 1.8" 10 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | # License 2 | 3 | This book is copyright of CHAOSS, 2017-2022. Some rights reserved. 4 | 5 | This book is licensed under the [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-sa/4.0/) (CC BY-SA 4.0). 6 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # GrimoireLab Tutorial 2 | 3 | [GrimoireLab](http://grimoirelab.github.io/), a 4 | [CHAOSS](http://chaoss.community) project, is free, open source software for 5 | software development analytics. It allows you to retrieve data from many kinds 6 | of systems with information related to software development, and produce 7 | analysis and visualizations with it. 8 | 9 | ![](assets/grimoirelab-chaoss-banner.png) 10 | 11 | GrimoireLab supports more than 20 different kinds of [data 12 | sources](https://gitlab.com/Bitergia/c/FAQ/tree/master/sources#supported-data-sources), 13 | from git repositories or GitHub projects, to issue trackers such as Jira or 14 | Bugzilla, including messaging systems such as IRC, Slack or mailing lists, or 15 | other types of systems such as StackOverflow or Jenkins. 16 | 17 | With the tools in the GrimoireLab toolset you can retrieve data, store it in 18 | databases, analyze it, and produce nice dashboards for interactively visualizing 19 | the resulting information. 20 | 21 | ![](assets/grimoirelab-3.png) 22 | 23 | Example dashboard produced with GrimoireLab: the [CHAOSS Development 24 | Dashboard](https://chaoss.biterg.io) 25 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | title: "GrimoireLab Tutorial" 2 | description: >- 3 | GrimoireLab Tutorial 4 | permalink: pretty 5 | markdown: kramdown 6 | baseurl: "/grimoirelab-tutorial" 7 | 8 | kramdown: 9 | toc_levels: 2..3 10 | 11 | exclude: 12 | - Gemfile 13 | - Gemgile.lock 14 | - node_mdules 15 | - .gitignore 16 | - vendor 17 | - .bundle 18 | - README.md 19 | 20 | # JUST THE DOCS CONFIG 21 | # theme: "just-the-docs" 22 | remote_theme: pmarsceill/just-the-docs 23 | # color_scheme: "dark" 24 | 25 | # Set a path/url to a logo that will be displayed instead of the title 26 | logo: "assets/grimoirelab.png" 27 | 28 | # Enable or disable the site search 29 | # Supports true (default) or false 30 | search_enabled: true 31 | 32 | # Enable support for hyphenated search words: 33 | search_tokenizer_separator: /[\s/]+/ 34 | 35 | # Aux links for the upper right navigation 36 | aux_links: 37 | "Website": 38 | - "//chaoss.github.io/grimoirelab" 39 | "GitHub": 40 | - "//github.com/chaoss/grimoirelab" 41 | "Mailing List": 42 | - "//lists.linuxfoundation.org/mailman/listinfo/grimoirelab-discussions" 43 | "Slack": 44 | - "//chaoss-workspace.slack.com/archives/C022NPTPC8Z" 45 | 46 | # Makes Aux links open in a new tab. Default is false 47 | aux_links_new_tab: true 48 | 49 | # Heading anchor links appear on hover over h1-h6 tags in page content 50 | # allowing users to deep link to a particular heading on a page. 51 | # 52 | # Supports true (default) or false/nil 53 | heading_anchors: true 54 | 55 | # Back to top link 56 | back_to_top: true 57 | back_to_top_text: "Back to top" 58 | 59 | # Footer content appears at the bottom of every page's main content 60 | footer_content: 'Copyright © CHAOSS; 2017-2022.
Distributed by an Creative Commons Attribution-ShareAlike 4.0 International License.' 61 | 62 | # Footer "Edit this page on GitHub" link text 63 | gh_edit_link: true # show or hide edit this page link 64 | gh_edit_link_text: "Edit this page on GitHub" 65 | gh_edit_repository: "https://github.com/chaoss/grimoirelab-tutorial" # the github URL for your repo 66 | gh_edit_branch: "master" # the branch that your docs is served from 67 | # gh_edit_source: docs # the source that your files originate from 68 | gh_edit_view_mode: "tree" # "tree" or "edit" if you want the user to jump into the editor immediately 69 | 70 | plugins: 71 | - jekyll-redirect-from 72 | 73 | # Google Analytics Tracking (optional) 74 | ga_tracking: 75 | -------------------------------------------------------------------------------- /assets/Selection_095.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/Selection_095.png -------------------------------------------------------------------------------- /assets/chaoss-logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/chaoss-logo.png -------------------------------------------------------------------------------- /assets/eclipse.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/eclipse.png -------------------------------------------------------------------------------- /assets/favicon/android-icon-144x144.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/android-icon-144x144.png -------------------------------------------------------------------------------- /assets/favicon/android-icon-192x192.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/android-icon-192x192.png -------------------------------------------------------------------------------- /assets/favicon/android-icon-36x36.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/android-icon-36x36.png -------------------------------------------------------------------------------- /assets/favicon/android-icon-48x48.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/android-icon-48x48.png -------------------------------------------------------------------------------- /assets/favicon/android-icon-72x72.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/android-icon-72x72.png -------------------------------------------------------------------------------- /assets/favicon/android-icon-96x96.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/android-icon-96x96.png -------------------------------------------------------------------------------- /assets/favicon/apple-icon-114x114.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/apple-icon-114x114.png -------------------------------------------------------------------------------- /assets/favicon/apple-icon-120x120.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/apple-icon-120x120.png -------------------------------------------------------------------------------- /assets/favicon/apple-icon-144x144.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/apple-icon-144x144.png -------------------------------------------------------------------------------- /assets/favicon/apple-icon-152x152.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/apple-icon-152x152.png -------------------------------------------------------------------------------- /assets/favicon/apple-icon-180x180.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/apple-icon-180x180.png -------------------------------------------------------------------------------- /assets/favicon/apple-icon-57x57.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/apple-icon-57x57.png -------------------------------------------------------------------------------- /assets/favicon/apple-icon-60x60.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/apple-icon-60x60.png -------------------------------------------------------------------------------- /assets/favicon/apple-icon-72x72.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/apple-icon-72x72.png -------------------------------------------------------------------------------- /assets/favicon/apple-icon-76x76.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/apple-icon-76x76.png -------------------------------------------------------------------------------- /assets/favicon/apple-icon-precomposed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/apple-icon-precomposed.png -------------------------------------------------------------------------------- /assets/favicon/apple-icon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/apple-icon.png -------------------------------------------------------------------------------- /assets/favicon/browserconfig.xml: -------------------------------------------------------------------------------- 1 | 2 | #ffffff -------------------------------------------------------------------------------- /assets/favicon/favicon-16x16.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/favicon-16x16.png -------------------------------------------------------------------------------- /assets/favicon/favicon-32x32.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/favicon-32x32.png -------------------------------------------------------------------------------- /assets/favicon/favicon-96x96.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/favicon-96x96.png -------------------------------------------------------------------------------- /assets/favicon/favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/favicon.ico -------------------------------------------------------------------------------- /assets/favicon/manifest.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "App", 3 | "icons": [ 4 | { 5 | "src": "\/android-icon-36x36.png", 6 | "sizes": "36x36", 7 | "type": "image\/png", 8 | "density": "0.75" 9 | }, 10 | { 11 | "src": "\/android-icon-48x48.png", 12 | "sizes": "48x48", 13 | "type": "image\/png", 14 | "density": "1.0" 15 | }, 16 | { 17 | "src": "\/android-icon-72x72.png", 18 | "sizes": "72x72", 19 | "type": "image\/png", 20 | "density": "1.5" 21 | }, 22 | { 23 | "src": "\/android-icon-96x96.png", 24 | "sizes": "96x96", 25 | "type": "image\/png", 26 | "density": "2.0" 27 | }, 28 | { 29 | "src": "\/android-icon-144x144.png", 30 | "sizes": "144x144", 31 | "type": "image\/png", 32 | "density": "3.0" 33 | }, 34 | { 35 | "src": "\/android-icon-192x192.png", 36 | "sizes": "192x192", 37 | "type": "image\/png", 38 | "density": "4.0" 39 | } 40 | ] 41 | } -------------------------------------------------------------------------------- /assets/favicon/ms-icon-144x144.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/ms-icon-144x144.png -------------------------------------------------------------------------------- /assets/favicon/ms-icon-150x150.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/ms-icon-150x150.png -------------------------------------------------------------------------------- /assets/favicon/ms-icon-310x310.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/ms-icon-310x310.png -------------------------------------------------------------------------------- /assets/favicon/ms-icon-70x70.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/favicon/ms-icon-70x70.png -------------------------------------------------------------------------------- /assets/grimoirelab-3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/grimoirelab-3.png -------------------------------------------------------------------------------- /assets/grimoirelab-all-details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/grimoirelab-all-details.png -------------------------------------------------------------------------------- /assets/grimoirelab-all.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/grimoirelab-all.png -------------------------------------------------------------------------------- /assets/grimoirelab-chaoss-banner.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/grimoirelab-chaoss-banner.png -------------------------------------------------------------------------------- /assets/grimoirelab-chaoss.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/grimoirelab-chaoss.png -------------------------------------------------------------------------------- /assets/grimoirelab-logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/grimoirelab-logo.png -------------------------------------------------------------------------------- /assets/grimoirelab.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/grimoirelab.png -------------------------------------------------------------------------------- /assets/howto-perceval-backends.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/assets/howto-perceval-backends.png -------------------------------------------------------------------------------- /docs/cases-chaoss/activity.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Community Activity 4 | nav_order: 1 5 | parent: Use Cases 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | ## Community Activity 12 | 13 | Once we have the indexes, let's create the first panel, for community activity. 14 | 15 | As usual with Kibana/Kibiter (from now own Kibiter) we will enter it in edit mode, import the produced indexes as index patterns, and proceed to create the visualizations and dashboards. 16 | 17 | Alternatively, you can instead use the [chaoss_community_activity.json](https://github.com/chaoss/grimoirelab-tutorial/blob/master/cases-chaoss/dashboards/chaoss_community_activity.json) file, which stores the configuration for all the index patterns, visualizations and dashboard. If you decide to shortcut this way, just install it with `kidash.py`: 18 | 19 | ```bash 20 | (gl) $ kidash.py --elastic_url http://localhost:9200 \ 21 | --import chaoss_community_activity.json --debug 22 | ``` 23 | 24 | This will import evertything into Kibana, and the dashboard will be now visible with the name `chaoss_community_activity'. 25 | 26 | Or you can produce everything yourself, as explained below. 27 | 28 | ### Index patterns 29 | 30 | In this case, it will be easy. As usual, click on `Management` on the left menu, and then on `Index patterns`. 31 | 32 | ![](/cases-chaoss/index_patterns.png) 33 | 34 | Once there, `Create new index`. You will do this twice: once for the index pattern `git`, and the other one for `github`, which are the enriched indexes we created with `p2o.py`. In both cases, specify `grimoire_creation_date` as the time field. 35 | 36 | Once this is done, you should be able of seeing both index patterns if you click on the `Discover` option of the left menu. 37 | 38 | ![](/cases-chaoss/discover_indexes.png) 39 | 40 | ### Visualizations 41 | 42 | This will be even easier. We will create a Markdown visualization for the text explaining the metric represented in this panel, and three visualizations for each of the three activity items to represent (git commits, GitHub issues and GitHub pull requests): 43 | 44 | * Metrics visualization, with the total number of items. For commits, use `Unique count` on the `hash` field, since we want to avoid double counting commits which happen to be in several repositories. For others, just use `Count`. 45 | * Horizontal bars visualization, with the time series of items over time. Again, use `Unique count` for commits, as above, and `Count` for the others. 46 | * Table visualization, with the items per repository, for filtering purposes. Just bucket by the `github_repo` field, and use `Count` as aggregator. 47 | 48 | For commits, you can just use the `git` index pattern as the base for the visualizations. But for issues and pull requests we need a little trick, since both are together in the `github` index pattern (the GitHub API Perceval uses provides all mixed). It is just to add a filter to the visualizations (use the `Add filter` button on the top left), setting it to `pull_request` (field) `is one of` `false`, for issues, and `true` for pull requests. 49 | 50 | And you are done. 51 | 52 | # Dashboard 53 | 54 | Producing the dashboard is just a matter of (still in Kibiter edit mode), click on the `Dashboards` option in the left menu, and create a new one. Add to it all the visualizations we just created, and ready. The result is as shown below. 55 | 56 | ![](/cases-chaoss/dashboard_chaoss_community_activity.png) 57 | 58 | The default period is 6 months, but since the dashboard is querying the database live, it is easy to select some other time frame. For example, go to the top right and select "last two years". 59 | 60 | ![](/cases-chaoss/dashboard_chaoss_community_activity-2y.png) 61 | 62 | Or youn can filter for a single repository (or everything except for a repository, or other combinations). For example, to examine the activity only in the Perceval repository, click on the `+` that appears when you hover the pointer over `grimoirelab/perceval` in any of the tables on the right. You'll get a filter (see in the top left) for that repository, and now all the metrics are only for it. 63 | 64 | ![](/cases-chaoss/dashboard_chaoss_community_activity-2y-perceval.png) 65 | 66 | Enjoy! 67 | -------------------------------------------------------------------------------- /docs/cases-chaoss/dashboard_chaoss_community_activity-2y-perceval.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/cases-chaoss/dashboard_chaoss_community_activity-2y-perceval.png -------------------------------------------------------------------------------- /docs/cases-chaoss/dashboard_chaoss_community_activity-2y.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/cases-chaoss/dashboard_chaoss_community_activity-2y.png -------------------------------------------------------------------------------- /docs/cases-chaoss/dashboard_chaoss_community_activity.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/cases-chaoss/dashboard_chaoss_community_activity.png -------------------------------------------------------------------------------- /docs/cases-chaoss/discover_indexes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/cases-chaoss/discover_indexes.png -------------------------------------------------------------------------------- /docs/cases-chaoss/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Use Cases 4 | permalink: /cases 5 | nav_order: 13 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Use Cases: CHAOSS Health Model 11 | 12 | The [CHAOSS Metrics Technical Committee](https://chaoss.community) is working on defining metrics for assessing open source communities' health and sustainability. In this chapter we use some of their definitions, as of September 2017, to produce visualizations based on the GrimoireLab stack. They will be used to compose a "CHAOSS Metrics Dashboard" produced with GrimoireLab technology. 13 | 14 | ## Data collection 15 | 16 | Before producing the visualizations, we need to grab the data and produce the corresponding raw indexes. Fortunately, most of the metrics the CHAOSS Metrics Committee is defining can be produced directly from GrimoireLab enriched indexes, so we will also produce enriched indexes. 17 | 18 | As data sources, we will use the main repositories the [GitHub GrimoireLab organization](https://github.com/grimoirelab): Perceval, Arthur, GrimoireELK, SortingHat, Panels, Mordred, and Training. It would be easy to use any other collection of repositories, and the result will be illustrative enough. 19 | 20 | The process shown here is completely manual, to minimize configuration and understanding of GrimoireLab components. It could be further automated with the use of Mordred and Arthur, especially if the data is to be updated continuously, but for now we will skip that part. In a first try we won't use SortingHat for simplicity: we can do that later, making some of the metrics more accurate. 21 | 22 | The process will include the installation of the GrimoireLab tools needed, and will assume that Python3, MySQL/MariaDB, ElasticSearch and Kibana are already installed (see [Installation section](/before-you-start/supporting-systems.md) for some hints on all of this. 23 | 24 | Let's start by installing GrimoireLab components: 25 | 26 | ```bash 27 | $ python3 -m venv gl 28 | $ source gl/bin/activate 29 | (gl) $ pip install grimoire-elk grimoire-kidash 30 | ``` 31 | 32 | Now, let's retrieve data for all the repositories to analyze (assuming ElasticSearch is local, available in its default port). We will use the following Python script, which will just run `p2o.py` for each repository to retrieve \([retrieve.py](/cases-chaoss/scripts/retrieve.py)\): 33 | 34 | ```python 35 | #! !/usr/bin/env python3 36 | # -*- coding: utf-8 -*- 37 | # retrieve.py 38 | 39 | import subprocess 40 | from sys import argv 41 | 42 | token = argv[1] 43 | repos = ['perceval', 'arthur', 'grimoireelk', 'sortinghat', 'mordred', 44 | 'panels', 'training'] 45 | 46 | for repo in repos: 47 | # Produce git and git_raw indexes from git repo 48 | subprocess.run(['p2o.py', '--enrich', '--index', 'git_raw', 49 | '--index-enrich', 'git', '-e', 'http://localhost:9200', 50 | '--no_inc', '--debug', 'git', 51 | 'https://github.com/grimoirelab/' + repo]) 52 | # Produce github and github_raw indexes from GitHub issues and prs 53 | subprocess.run(['p2o.py', '--enrich', '--index', 'github_raw', 54 | '--index-enrich', 'github', '-e', 'http://localhost:9200', 55 | '--no_inc', '--debug', 'github', 'grimoirelab', repo, 56 | '-t', token, '--sleep-for-rate']) 57 | ``` 58 | 59 | To run it, we need to specify a [personal GitHub token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) in the command line. That token will be provided to the GitHub API to speed things a bit. Be careful with your token, and don't include it in anything which may became public (your personal token allows for access to GitHub with your permissions). 60 | 61 | ``` 62 | (gl) $ python3 retrieve.py 63 | ``` 64 | 65 | After some minutes and a lot of messages, all the repositories will be retrieved, and the indexes will be produced. Now we can proceed to produce the metrics panels... 66 | -------------------------------------------------------------------------------- /docs/cases-chaoss/index_patterns.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/cases-chaoss/index_patterns.png -------------------------------------------------------------------------------- /docs/cases-chaoss/scripts/retrieve.py: -------------------------------------------------------------------------------- 1 | #! !/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | import subprocess 5 | from sys import argv 6 | 7 | token = argv[1] 8 | repos = ['perceval', 'arthur', 'grimoireelk', 'sortinghat', 'mordred', 9 | 'panels', 'training'] 10 | 11 | for repo in repos: 12 | # Produce git and git_raw indexes from git repo 13 | subprocess.run(['p2o.py', '--enrich', '--index', 'git_raw', 14 | '--index-enrich', 'git', '-e', 'http://localhost:9200', 15 | '--no_inc', '--debug', 'git', 16 | 'https://github.com/grimoirelab/' + repo]) 17 | # Produce github and github_raw indexes from GitHub issues and prs 18 | subprocess.run(['p2o.py', '--enrich', '--index', 'github_raw', 19 | '--index-enrich', 'github', '-e', 'http://localhost:9200', 20 | '--no_inc', '--debug', 'github', 'grimoirelab', repo, 21 | '-t', token, '--sleep-for-rate']) 22 | -------------------------------------------------------------------------------- /docs/community-health/code-dev-activity.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Code Development Activities 4 | nav_order: 4 5 | parent: Community Health 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Understand the types and frequency of activities involved in developing code 11 | 12 | The goal of this area is to identify and understand the source code changes over 13 | time and the frequency of those changes over time. For example, 14 | 15 | - What changes were made to the source code during a specified period ? 16 | - How projects manage lifecycle of their version control branches ? 17 | - What is the sum of the number of lines touches( lines addes plus lines removed 18 | ) in all changes to the source code during a certain period. ? -------------------------------------------------------------------------------- /docs/community-health/code-dev-efficiency.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Code Development Efficiency 4 | nav_order: 5 5 | parent: Community Health 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Understand how efficiently activities around code development get resolved. 11 | 12 | The goal of this area is to identify how efficiently any code related activity 13 | get resolved. For example, 14 | 15 | - What is the time taken to merge a change request ? 16 | - How many change requests are present in a code change ? 17 | - What reviews of change requests ended up being declined during a certain 18 | period ? 19 | - What is the ratio of change request accepted to rejected ? -------------------------------------------------------------------------------- /docs/community-health/contributions.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Contributions 4 | nav_order: 1 5 | parent: Community Health 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Identify what contributions are being made 11 | 12 | The goal of this area is to identify what kind of contributions (wiki edit, 13 | documentaion, bug fix) are being made to a project. -------------------------------------------------------------------------------- /docs/community-health/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Community Health 4 | permalink: /community-health 5 | nav_order: 6 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Understanding Community health 11 | 12 | GrimoireLab is the platform through which community health can be monitored but 13 | what is community health and how can we define it ? 14 | 15 | Community health is the potential that a project can continue developing quality 16 | software. There are different ways that it can be evaluated. For example, we can 17 | analyze the health of a community through the code quality, the activity in the 18 | community ( are the contributors active, are the issues being resolved, the 19 | diversity of contributors) and the resources needed by the project. 20 | 21 | Check out the following focus areas depending on your required metric. -------------------------------------------------------------------------------- /docs/community-health/issue-resolution.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Issue Resolution 4 | nav_order: 7 5 | parent: Community Health 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Identify how effective community is at adressing issues from community participants 11 | 12 | The goal of this area is to identify how effective the community is at 13 | addressing issues from community participants. For example, 14 | 15 | - What is the number of new issues created during a certain period ? 16 | - What is the count of issues that showed activity during a certain period ? 17 | - What is the count of issues that were closed during a certain period ? 18 | - What is the average time that open issues have been open ? 19 | - What is the time period between the opening of an issue and a response in the issue 20 | thread from another contributor ? 21 | - How much time taken to close an issue ? -------------------------------------------------------------------------------- /docs/community-health/people.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: People 4 | nav_order: 3 5 | parent: Community Health 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Understand personal engagement with projects 11 | 12 | The goal of this area is to identify and understand organizational and personal 13 | engagement with open source projects. For example, 14 | 15 | - Who are the contributors to your project ? 16 | - What is the geographical diversity of your contributors ? 17 | - What is the organizational diversity of your contributors ? -------------------------------------------------------------------------------- /docs/community-health/time.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Time 4 | nav_order: 2 5 | parent: Community Health 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Identify when contributions are made 11 | 12 | The goal of this area is to identify when does contributions from organizations 13 | and people happen. For example, 14 | 15 | - What are the timestamps of contribution activities ? 16 | - What is the duration for the review of a change request ? 17 | - How much time for an activity (issue, change request) to receive any response 18 | ? -------------------------------------------------------------------------------- /docs/components/gelk/figs/dashboard-git-sortinghat.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/dashboard-git-sortinghat.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/elasticsearch-index.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/elasticsearch-index.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/kibana-dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/kibana-dashboard.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/bar-chart.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/bar-chart.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/bubbles-chart.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/bubbles-chart.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/groups-pie-chart.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/groups-pie-chart.jpg -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/index-pattern-fields.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/index-pattern-fields.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/kibana-metrics-viz.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/kibana-metrics-viz.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/kibana-table-viz-def-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/kibana-table-viz-def-1.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/kibana_index_pattern.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/kibana_index_pattern.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/meetings-evolution.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/meetings-evolution.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/meetings-table.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/meetings-table.jpg -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/meetup-stats-by-grimoirelab.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/meetup-stats-by-grimoirelab.jpg -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/meetup-stats.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/meetup-stats.jpg -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/members-table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/members-table.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/pie-chart-definition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/pie-chart-definition.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/url-field.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/url-field.png -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/visualize-kibana.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/visualize-kibana.jpg -------------------------------------------------------------------------------- /docs/components/gelk/figs/meetup/visualize-kibana.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/components/gelk/figs/meetup/visualize-kibana.png -------------------------------------------------------------------------------- /docs/components/gelk/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: GrimoireELK 4 | permalink: /gelk 5 | nav_order: 1 6 | parent: Components 7 | has_children: true 8 | has_toc: true 9 | --- 10 | 11 | # GrimoireELK (Gelk) 12 | 13 | GrimoireELK is the component of GrimoireLab that interacts with the 14 | underlying (OpenSearch) database. Its goal is two-fold, first it aims 15 | at offering a convenient way to store the data coming from Perceval, 16 | second it processes and enriches the data in a format that can be 17 | used by the Dashboards. 18 | 19 | -------------------------------------------------------------------------------- /docs/components/gelk/intro.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Producing dashboards with GrimoireELK 4 | nav_order: 1 5 | parent: GrimoireELK 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | # Producing dashboards with GrimoireELK 11 | 12 | [GrimoireELK](http://github.com/chaoss/grimoirelab-elk) 13 | is an interim system oriented to produce Kibana-based dashboards with Perceval and friends. 14 | It provides a Python module (`grimoire_elk`) with facilities for driving 15 | Perceval, enriching data, and uploading / downloading it to / from ElasticSearch. It includes as well some scripts (mainly `p2o.py`) to retrieve data from repositories related to software development, and produce everything needed to have Elasticsearch indexes that provide the data for a nice Kibiter/Kibana dashboard. 16 | 17 | In summary, `p2o.py`: 18 | 19 | * drives Perceval to retrieve data from repositories, 20 | * uploads the resulting data as raw indexes (collections of JSON documents) to ElasticSearch 21 | * enriches those raw indexes (produce new data with fields suitable to be used by Kibana dashboards) 22 | * uploads that resulting data as enriched indexes to ElasticSearch 23 | 24 | In addition, `kidash`, 25 | available in the Python package of the same name, 26 | can upload dashboard definitions (including visualizations, searches and everything needed by them) to produce a Kibiter/Kibana dashboards. 27 | 28 | In this chapter we will explore how to use these tools to produce complete 29 | Kibiter/Kibana-based dashboards. 30 | Before following to the rest of the chapter, ensure that both 31 | `grimoire-elk` and `kidash` Python packages are installed 32 | (see [Installing GrimoireLab](../basics/install.html)). 33 | -------------------------------------------------------------------------------- /docs/components/gelk/kidash.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Managing dashboards with kidash 4 | nav_order: 4 5 | parent: GrimoireELK 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | ## Managing dashboards with kidash 12 | 13 | Kibana (or Kibiter, for that matter) offers a nice user interface to create and edit visualizations and dashboards. It allows to save and restore them as well. But if you're interested in backing up all the elements needed to produce a dashboard, you need some tools. That's why we developed Kidash. Then, we added some other features to it. And now Kidash is our tool of choice to back and restore dashboards with all their elements (visualizations, searches, and index patterns), in the same or in a different Kibana instance. 14 | 15 | Kidash works by managing the `.kibana` index in ElasticSearch, which is where Kibana stores its configuration. That index includes JSON documents corresponding to all elements in Kibana, from dashboards to index patterns. By retrieving documents from it, you can backup your elements. By creating or changing documents in it, you can create or change elements. kidash allows to save documents from that index in a file, in a comfortable way. That file is in JSON format, which allows for easy edition of it (if you know about how Kibana elements are stored in JSON documents). And kidash can upload the content of those files (be them edited or not) to the same or another Kibana instance. 16 | 17 | Kidash provides a Python script, `kidash`. 18 | It can be installed, with all its dependencies, using pip, 19 | as we saw in the [section on Installing GrimoireLab](../basics/install.html). 20 | 21 | ### Saving dashboards 22 | 23 | You can save a dashboard, with all its components, to a file, either for backup or for later uploading to a different Kibana instance. You will need access to the ElasticSearch instance that your Kibana is using, because kidash will be reading its `.kibana` index. Assuming that Kibana is in localhost, at port 9200, and that you want to backup the dashboard named "Git" to the file `/tmp/dashboard-git.json`, just run (from the virtual environment where you installed `grimoire-kidash`): 24 | 25 | 26 | ```bash 27 | kidash -e http://localhost:9200 --dashboard "Git" --export /tmp/dashboard-git.json 28 | ``` 29 | 30 | You can learn the name of the dashboard by looking at its top left corner, or by noting the name you use when opening it in Kibana. If the name includes spaces, use "-" instead. For example, for a dashboard named "Git History", use the line: 31 | 32 | ```bash 33 | kidash -e http://localhost:9200 --dashboard "Git-History" \ 34 | --export /tmp/dashboard-git.json 35 | ``` 36 | 37 | If you open the file created, 38 | you will see it is written in JSON format. 39 | In fact, it is a dictionary with one entry per kind of element 40 | (dashboards, visualizations, searches, index patterns). 41 | For each kind of element, you will find a list of the saved elements, 42 | as dictionaries with their characteristics. 43 | These are the same you can read in Kibana (only for dashboards, visualizations, and searches) 44 | in the "Management | Saved Objects" menu entry, if you edit one of the elements. 45 | `kidash` takes care of, given the name of the dashborad, 46 | find recursively the other elements needed to draw it. 47 | 48 | ### Restoring dashboards 49 | 50 | We already restored a dashboard in the 51 | [section on creating a simple dashboard](simple.html#uploading). 52 | We can restore from any file created with kidash. Assuming we have that file as `/tmp/dashboard-git.json`, we need to know the link to the ElasticSearch REST interface (same as for backing up). The format is, for example, as follows: 53 | 54 | ```bash 55 | kidash --elastic_url http://localhost:9200 \ 56 | --import /tmp/dashboard-git.json 57 | ``` 58 | 59 | This will restore all elements in the file, overwriting, if needed, elements with the same name in the corresponding Kibana instance (in fact, in the corresponding ElasticSearch `.kibana` index). Therefore, beware: restoring a JSON file with kidash may destroy your elements, because they may be overwritten by others in the file. Remember to backup before running the command. 60 | 61 | ### Other options 62 | 63 | Kidash has some more options. For a complete listing, use the `--help` argument: 64 | 65 | ```bash 66 | kidash --help 67 | ``` 68 | -------------------------------------------------------------------------------- /docs/components/graal/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Graal 4 | permalink: /graal 5 | nav_order: 5 6 | parent: Components 7 | has_children: true 8 | has_toc: true 9 | --- 10 | 11 | # Graal: a Generic Reporitory AnALyzer 12 | 13 | Graal leverages on the Git backend of Perceval and enhances it to set 14 | up ad-hoc source code analysis. Thus, it fetches the commits from a 15 | Git repository and provides a mechanism to plug third party 16 | tools/libraries focused on source code analysis. 17 | 18 | 19 | GrimoireELK is the component of GrimoireLab that interacts with the 20 | underlying (OpenSearch) database. Its goal is two-fold, first it aims 21 | at offering a convenient way to store the data coming from Perceval, 22 | second it processes and enriches the data in a format that can be 23 | used by the Dashboards. 24 | 25 | -------------------------------------------------------------------------------- /docs/components/graal/intro.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Intro 4 | nav_order: 1 5 | parent: Graal 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | # Graal 11 | 12 | - [Graal](https://github.com/chaoss/grimoirelab-graal) 13 | ( Generic Repository AnALyzer ) with the help of Perceval's 14 | [Git backend](https://github.com/chaoss/grimoirelab-perceval/blob/master/perceval/backends/core/git.py) 15 | fetches commits from a Git repository and provides a mechanism to 16 | plug third party tools/libraries focused on source code analysis. 17 | As of now, Graal produces analysis related to code complexity, 18 | quality, dependencies, vulnerability and licensing. In this chapter, 19 | you will learn the basics of working with Graal, including how to 20 | retrieve source code related analysis with the help of some of it's 21 | backends. 22 | 23 | - Before starting, ensure that you have Python3 ready, and the Graal 24 | module installed, as detailed in 25 | [Installing GrimoireLab Python modules](../basics/install.md). 26 | 27 | **Note:** As of now, Graal is not integrated in Grimoirelab toolchain, 28 | it will soon be added to support source code related metrics. 29 | -------------------------------------------------------------------------------- /docs/components/graal/scripts/graal_cocom_1.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | from graal.backends.core.cocom import CoCom 3 | 4 | # URL for the git repo to analyze 5 | repo_uri = "http://github.com/chaoss/grimoirelab-graal" 6 | 7 | # directory where to mirror the repo 8 | repo_dir = "/tmp/graal-cocom" 9 | 10 | # Cocom object initialization 11 | cc = CoCom(uri=repo_uri, git_path=repo_dir) 12 | 13 | # fetch all commits 14 | commits = [commit for commit in cc.fetch()] 15 | -------------------------------------------------------------------------------- /docs/components/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Components 4 | permalink: /components 5 | nav_order: 10 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | -------------------------------------------------------------------------------- /docs/components/manuscripts/first.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Reporting with Manuscripts 4 | nav_order: 1 5 | parent: Manuscripts 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | ## Reporting with Manuscripts 12 | 13 | [Manuscripts](https://github.com/chaoss/grimoirelab-manuscripts) is the GrimoireLab tool 14 | to produce reports. 15 | It uses data from GrimoireLab indexes in ElasticSearch, and produces PDF documents with charts and tables, useful for getting an idea of how the project is performing. 16 | 17 | ### Get your first report 18 | 19 | Reporting with GrimoireLab Manuscripts is easy. You need to have enriched ElasticSearch indexes, produced by GrimoireELK (see [Producing Kibana dashboards with GrimoireELK](/grimoireelk/intro.md)) or Mordred (see [Mordred: orchestrating everything](/mordred/intro.md)), and of course, having `grimoirelab-manuscripts` installed. The rest is pretty easy. 20 | 21 | For example, to produce a report about Git data in the standard GrimoireLab enriched index in my local ElasticSearch (accessible in the standard [http://localhost:9200](http://localhost:9200) location), you only need to run: 22 | 23 | ```bash 24 | manuscripts -d /tmp/reports -u http://localhost:9200 \ 25 | -n GrimoireLab --data-sources git 26 | ``` 27 | 28 | The report will be produced as `/tmp/reports/report.pdf`, using `GrimoireLab` as the name of the analyzed project, when presented in the report. 29 | 30 | > _Note:_ This requires pdflatex to be installed. 31 | -------------------------------------------------------------------------------- /docs/components/manuscripts/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Manuscripts 4 | permalink: /manuscripts 5 | nav_order: 4 6 | parent: Components 7 | has_children: true 8 | has_toc: true 9 | --- 10 | 11 | # Manuscripts 12 | 13 | Production of PDF documents reporting on the main characteristics of a 14 | project. 15 | -------------------------------------------------------------------------------- /docs/components/sirmordred/files/mordred-simple.cfg: -------------------------------------------------------------------------------- 1 | # Simple SirMordred config file 2 | # 3 | 4 | [general] 5 | # Name of the project to analyze (will appear in dashboard) 6 | short_name = TestProject 7 | # Update incrementally, forever 8 | update = true 9 | # Don't start a new update earlier than (since last update, seconds) 10 | min_update_delay = 50 11 | # Produce debugging data for the logs 12 | debug = true 13 | 14 | # Directory for logs 15 | logs_dir = logs 16 | 17 | 18 | [projects] 19 | # File with repositories to analyze 20 | projects_file = projects.json 21 | 22 | 23 | [es_collection] 24 | # Url to access raw indexes (Elasticsearch instance) 25 | url = http://localhost:9200 26 | # User and password to acces Elasticseach, if any 27 | user = 28 | password = 29 | 30 | 31 | [es_enrichment] 32 | # Url to access enriched indexes (Elasticsearch instance) 33 | url = http://127.0.0.1:9200 34 | # User and password to acces Elasticseach, if any 35 | user = 36 | password = 37 | 38 | # Refresh identities and projects for all items after enrichment 39 | autorefresh = true 40 | 41 | 42 | [sortinghat] 43 | # Host to access MariaDB / MySQL instance 44 | host = localhost 45 | # User and password to access MariaDB / MySQL instance 46 | user = mariadb_user 47 | password = XXX 48 | # Name of MariaDB / MySQL database 49 | database = grimoirelab_sh 50 | 51 | # Run affilation 52 | affiliate = True 53 | # How to match to unify 54 | matching = [email] 55 | # How long to sleep before running again, for identities tasks 56 | sleep_for = 10 57 | # Don't load a specific organizations file 58 | load_orgs = false 59 | # Name of group of identities with no affiliation 60 | unaffiliated_group = Unknown 61 | # How to autoprofile 62 | autoprofile = [TestProject:manual,git,github] 63 | 64 | [panels] 65 | # Kibitter / kibana entry point 66 | kibiter_url = http://localhost:5601 67 | # Dashboard: default time frame 68 | kibiter_time_from= "now-1y" 69 | # Dashboard: default index pattern 70 | kibiter_default_index= "git" 71 | 72 | [phases] 73 | # Data collection (build raw indexes) 74 | collection = true 75 | # Run SortingHat for identities 76 | identities = true 77 | # Build enriched indexes 78 | enrichment = true 79 | # Upload panels to Kibiter/Kibana 80 | panels = true 81 | 82 | [git] 83 | # Names for raw and enriched indexes 84 | raw_index = git_grimoirelab-raw 85 | enriched_index = git_grimoirelab 86 | 87 | -------------------------------------------------------------------------------- /docs/components/sirmordred/files/projects.json: -------------------------------------------------------------------------------- 1 | { 2 | "Test Project": { 3 | "git": [ 4 | "https://github.com/chaoss/grimoirelab-perceval", 5 | "https://github.com/chaoss/grimoirelab-elk" 6 | ] 7 | } 8 | } 9 | -------------------------------------------------------------------------------- /docs/components/sirmordred/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Sir Mordred rules them all 4 | permalink: /sirmordred 5 | nav_order: 2 6 | parent: Components 7 | has_children: true 8 | has_toc: true 9 | --- 10 | 11 | # Sir Mordred 12 | 13 | Mordred is the tool used to coordinate the execution of the GrimoireLab 14 | platform. 15 | -------------------------------------------------------------------------------- /docs/components/sirmordred/projects.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: The projects file 4 | nav_order: 4 5 | parent: Sir Mordred rules them all 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | ## The projects file 12 | 13 | An important part of the information provided to Mordred is the list of projects to analyze, and how they are structured in repositories. This is done in the projects file. 14 | 15 | The projects file is a JSON file with the following structure: 16 | 17 | ``` 18 | { 19 | "Project1": { 20 | "datasource1": [ list_of_repos_11 ], 21 | "datasource2": [ list_of_repos_12 ], 22 | ... 23 | "datasourcen": [ list_of_repos_1n ] 24 | }, 25 | "Project2": { 26 | ... 27 | }, 28 | ... 29 | "Projectk": { 30 | } 31 | } 32 | ``` 33 | 34 | That is, in Python terms it is a dictionary of projects, where the key for each project is the project name (as we want it to appear in the dashboard), and the value is a dictionary with the data sources for that project. 35 | 36 | In turn the dictionary with data sources has as keys the Perceval identifiers for the data sources of interest for the project (for example "git", "github", "jira", etc.). Values of that dictionary is a list of repositories, also in the format expected by Perceval. See `perceval --help`, or the [main Perceval README](https://github.com/chaoss/grimoirelab-perceval/blob/master/README.md) for a complete list of data sources supported, and their identifiers. For convenience, this is a close-to-complete list: 37 | 38 | * askbot: Questions and answers from Askbot site 39 | * bugzilla: Bugzilla server 40 | * bugzillarest: Bugzilla server (>=5.0) using its REST API 41 | * confluence: Confluence server 42 | * discourse: Discourse site 43 | * gerrit: Gerrit server 44 | * git: Commits from Git 45 | * github: Issues and PRs from GitHub 46 | * gmane: Gmane messages (not working now) 47 | * hyperkitty: Messages from a HyperKitty archiver 48 | * jenkins: Builds data from a Jenkins server 49 | * jira: Issues data from JIRA issue tracker 50 | * mbox: Messages from MBox files 51 | * mediawiki: Pages and revisions from a MediaWiki site 52 | * meetup: Events from a Meetup group 53 | * nntp: Articles from a NNTP news group 54 | * phabricator: Tasks from a Phabricator site 55 | * pipermail: Messages from a Pipermail archiver 56 | * redmine: Issues data from a Redmine server 57 | * rss: Entries from a RSS feed server 58 | * slack: Messages from a Slack channel 59 | * stackexchange: Questions, answers and comments from StackExchange sites 60 | * supybot: Messages from Supybot log files 61 | * telegram: Messages from the Telegram server 62 | 63 | As an example, this is an excerpt of the projects file for the CHAOSS dashboard: 64 | 65 | ``` 66 | { 67 | "GrimoireLab": { 68 | "git": [ 69 | "https://github.com/chaoss/grimoirelab-kingarthur", 70 | "https://github.com/chaoss/grimoirelab-elk", 71 | "https://github.com/chaoss/grimoirelab-sirmordred", 72 | ... 73 | ], 74 | "github": [ 75 | "https://github.com/chaoss/grimoirelab-kingarthur", 76 | "https://github.com/chaoss/grimoirelab-elk", 77 | "https://github.com/chaoss/grimoirelab/mordred", 78 | ... 79 | ], 80 | "pipermail": [ 81 | "https://lists.linuxfoundation.org/pipermail/grimoirelab-discussions" 82 | ] 83 | }, 84 | "GHData": { 85 | "git": [ 86 | "https://github.com/OSSHealth/ghdata" 87 | ], 88 | "github": [ 89 | "https://github.com/OSSHealth/ghdata" 90 | ] 91 | }, 92 | ... 93 | } 94 | ``` 95 | 96 | Note that for GitHub repos, we need entries both for "git" and "github", since two different Perceval backends are used: the first one for the git repository, the second one for GitHub issues and pull requests. 97 | 98 | Once we have completed the file, we have to specify its name in the `mordred.cfg` file (section "projects", parameter "projects_file"). For example, if the file is `/home/user/projects.json`, the lines in `mordred.cfg` would be: 99 | 100 | ``` 101 | [projects] 102 | projects_file = /home/user/projects.json 103 | ``` 104 | -------------------------------------------------------------------------------- /docs/components/sortinghat-cli/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: SortingHat (from the command line) 4 | permalink: /sortinghat-cli 5 | nav_order: 3 6 | parent: Components 7 | has_children: true 8 | has_toc: true 9 | --- 10 | 11 | # SortingHat 12 | 13 | This describes the usage of SortingHat from the command line. 14 | 15 | SortingHat has nice, modern web interface and the command line interface 16 | is considered legacy funcionality. 17 | -------------------------------------------------------------------------------- /docs/contribute/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Contribute to GrimoireLab 4 | redirect_to: https://github.com/chaoss/grimoirelab/blob/master/CONTRIBUTING.md 5 | nav_order: 7 6 | has_children: false 7 | has_toc: false 8 | --- -------------------------------------------------------------------------------- /docs/dashboards/access.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Public and Private Access 3 | nav_order: 1 4 | parent: Dashboards 5 | has_children: false 6 | has_toc: false 7 | --- 8 | 9 | # Who has access to the dashboards 10 | 11 | There are two main roles concerning dashboards access, 12 | - The user with permission to see the dashboards 13 | - The user with permission to edit the dasboards and accessing indexes. 14 | 15 | Lets take the example of [CHAOSS Community dashboards](http://chaoss.biterg.io). 16 | Anyone can access their GrimoireLab dashboards and view the information as it is 17 | publicly available. However, the CHAOSS Community dashboards has a login button 18 | where only people with proper permission can login to perform actions such as 19 | creating, editing and removing dashboards. 20 | 21 | ## How to login 22 | 23 | ##### Steps 24 | - Look for the `Login` icon at the bottom of the sidebar. 25 | 26 | ![login button](./assets/loginButton.png) 27 | 28 | - Clicking on it will bring you to the login page. 29 | 30 | ![login page](./assets/login-page.png) 31 | 32 | - Enter your credentials and press `Log in`. -------------------------------------------------------------------------------- /docs/dashboards/add.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Create a dashboard 4 | nav_order: 5 5 | parent: Dashboards 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | # How to create a dashboard 11 | 12 | Creating a dashboard is pretty intuitive in GrimoireLab. 13 | 14 | The first step is to open the dashboards interface which lists out all the 15 | created dashboards. Click on the `Dashboard` icon in the sidebar. It is the 16 | third one form the top. 17 | 18 | ![visualize](./assets/visualize.png) 19 | 20 | To create a new dashboard, click on the button labelled `Create new dashboard`. 21 | You can check out the existing dashboards below to get an idea of what kind of 22 | dashboard you need. 23 | 24 | ![new dashboard](./assets/new_dashboard.png) 25 | 26 | Once you've clicked the `Create new dashboard` button, you'll be presented with 27 | an empty dashboard which you can fill with the visualizations of your choice. 28 | Click on the `Add` button to open the visualization sidepanel and click of the 29 | needed visualizations to add them to your dashboard. 30 | 31 | ![empty dashboard](./assets/empty_dashboard.png) 32 | 33 | Once you've added your required visualization, you can save the dashboard by 34 | pressing the `Save` button. It is the third one on the left from the `Add` 35 | button. Make sure you're logged in before saving the dashboard. 36 | 37 | **Note: You need to be logged in to perform this action** 38 | 39 | You can learn more about the different accesses and the permissions associated 40 | with them in the [Private and Public Access](docs/dashboards/access) section -------------------------------------------------------------------------------- /docs/dashboards/assets/absolute-time-picker.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/absolute-time-picker.png -------------------------------------------------------------------------------- /docs/dashboards/assets/add-button.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/add-button.png -------------------------------------------------------------------------------- /docs/dashboards/assets/customize.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/customize.png -------------------------------------------------------------------------------- /docs/dashboards/assets/dashboard-option.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/dashboard-option.png -------------------------------------------------------------------------------- /docs/dashboards/assets/dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/dashboard.png -------------------------------------------------------------------------------- /docs/dashboards/assets/delete-dashboards.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/delete-dashboards.png -------------------------------------------------------------------------------- /docs/dashboards/assets/dev-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/dev-output.png -------------------------------------------------------------------------------- /docs/dashboards/assets/dev-tool-interface.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/dev-tool-interface.png -------------------------------------------------------------------------------- /docs/dashboards/assets/dev-tools.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/dev-tools.png -------------------------------------------------------------------------------- /docs/dashboards/assets/drag.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/drag.gif -------------------------------------------------------------------------------- /docs/dashboards/assets/edit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/edit.png -------------------------------------------------------------------------------- /docs/dashboards/assets/edit_nav.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/edit_nav.png -------------------------------------------------------------------------------- /docs/dashboards/assets/empty_dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/empty_dashboard.png -------------------------------------------------------------------------------- /docs/dashboards/assets/fields.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/fields.png -------------------------------------------------------------------------------- /docs/dashboards/assets/filter-active.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/filter-active.png -------------------------------------------------------------------------------- /docs/dashboards/assets/filter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/filter.png -------------------------------------------------------------------------------- /docs/dashboards/assets/filters-and-time-picker.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/filters-and-time-picker.png -------------------------------------------------------------------------------- /docs/dashboards/assets/login-page.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/login-page.png -------------------------------------------------------------------------------- /docs/dashboards/assets/loginButton.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/loginButton.png -------------------------------------------------------------------------------- /docs/dashboards/assets/navigate.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/navigate.png -------------------------------------------------------------------------------- /docs/dashboards/assets/new_dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/new_dashboard.png -------------------------------------------------------------------------------- /docs/dashboards/assets/save-fields.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/save-fields.png -------------------------------------------------------------------------------- /docs/dashboards/assets/save.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/save.png -------------------------------------------------------------------------------- /docs/dashboards/assets/time-picker.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/time-picker.png -------------------------------------------------------------------------------- /docs/dashboards/assets/visualize.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/assets/visualize.png -------------------------------------------------------------------------------- /docs/dashboards/edit.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Edit a dashboard 4 | nav_order: 6 5 | parent: Dashboards 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | # How to edit a dashboard ? 11 | 12 | A dashboard usually represent some visualization or table of some sort. 13 | GrimoireLab allows you to execute the following queries in regards to your 14 | dashboard. 15 | 16 | ## Edit panel's title 17 | 18 | Editing a dashboard's title is simple. 19 | - Steps 20 | 1. Click on `Edit` in the navigation bar of GrimoireLab. 21 | 22 | ![edit navigation](./assets/edit_nav.png) 23 | 24 | 2. Click on the `Settings` icon of the dashboard. This will open a dropdown. 25 | 26 | ![edit](./assets/edit.png) 27 | 28 | 3. Click on `Customize panel` and change your dashboard's panel. 29 | 30 | ![customize](./assets/customize.png) 31 | 32 | ## How to arrange your panels 33 | 34 | GrimoireLab provides you with the ability to move and display your panels in any 35 | order you want. 36 | 37 | - Steps 38 | 1. Look for the "Edit" button and click it. 39 | 40 | ![edit](./assets/edit_nav.png) 41 | 42 | 2. Once you've done the above, edit mode will be active. You can then drag any 43 | panels around and arrange it in any way you want. 44 | 45 | ![dragging panels](./assets/drag.gif) 46 | 47 | ## How to add a visualization to your dashboard 48 | 49 | Dashboards would not serve any purpose visualizations. In case you want to add a 50 | visualization to your dashboard, follow the steps below. 51 | 52 | - Steps 53 | 1. Click on `Edit` in the navigation bar of GrimoireLab. 54 | 55 | ![edit](./assets/edit_nav.png) 56 | 57 | 2. Click on `Add` on the same navigation bar. This should open the side 58 | panel titled `Add Panel`. 59 | 60 | ![add button](./assets/add-button.png) 61 | 62 | 3. Click on your visualization on choice and it will get added onto your 63 | dashboard. 64 | 65 | 4. Click on `Save` to save the changes to your dashboard. 66 | 67 | ![save](./assets/save.png) 68 | 69 | Note: If you want to edit your visualization, please refer to [this 70 | section](docs/dashboards/visualization/). Content on how to edit your 71 | visualization is covered there. 72 | -------------------------------------------------------------------------------- /docs/dashboards/filter.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Filtering data 4 | nav_order: 2 5 | parent: Dashboards 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | # How to filter data 11 | 12 | There are two places in GrimoireLab where we can change and add filters. 13 | 14 | - Time Picker 15 | - Normal Filters 16 | 17 | ![filter and time picker](./assets/filters-and-time-picker.png) 18 | 19 | ### Time Picker 20 | 21 | Most dashboards are based on item creation date. So you can filter according to 22 | a time frame. Time Picker allows you to visualize data according to a particular 23 | time frame. 24 | 25 | ![time picker](./assets/time-picker.png) 26 | 27 | You can also set a custom time picker if needed. Go to the section `Time picker 28 | quick ranges` in `Management` -> `Advances Settings` and add your quickRanges to 29 | the list. An example of quickRange is provided below. `from` and `to` contain 30 | date values, `display` is the title thay will be displayed in kibiter and 31 | `section` defines the column where the quickRange will be added. 32 | 33 | **Note: You need to be logged in to save your custom time picker. Check the 34 | [private and public access](/docs/dashboards/access) section 35 | to understand about permissions in GrimoireLab dashboards.** 36 | 37 | ``` 38 | { 39 | "from": "2020-03-21 00:00:00", 40 | "to": "2020-04-24 00:00:00", 41 | "display": "Most active period", 42 | "section": 3 43 | } 44 | ``` 45 | 46 | You can also set a custom time picker without actually looking into settings. 47 | 48 | 1. Click on Time Picker and choose "Absolute". 49 | 50 | ![absolute time picker](./assets/absolute-time-picker.png) 51 | 52 | 2. Set your time frame and press on "Go" to apply the filter. 53 | 54 | ### Normal Filters 55 | 56 | You can set a filter by clicking on **Add filter**. This filter allows you to 57 | set a particular key-value pair and display the visualizations according to that 58 | value. You can even set the operators to check for that particular value orevery 59 | other data except the one mentioned. 60 | 61 | #### Steps 62 | 63 | - Click on "Add filter" on top of your panels. 64 | 65 | ![filter](./assets/filter.png) 66 | 67 | - A popover will appear. Choose your field, the operator and the filtering 68 | value. 69 | 70 | ![fields](./assets/fields.png) 71 | 72 | - Once you've set your values, save it and the visualizations should re-ajust to 73 | fit your filter. 74 | 75 | ![save fields](./assets/save-fields.png) 76 | ![filter active](./assets/filter-active.png) -------------------------------------------------------------------------------- /docs/dashboards/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Dashboards 4 | permalink: /dashboards 5 | nav_order: 4 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Dashboards 11 | 12 | GrimoireLab provides a user or a community with the ability to monitor, evaluate 13 | and analyze the health of their project through several metrics and 14 | visualizations. The latters are displayed in a layout called a dashboard. 15 | You can create several dashboards to monitor different aspects of your project. 16 | 17 | In this section, we'll be talking about how to use the dashboards and interact 18 | with the features available. 19 | 20 | ![dashboards](./assets/dashboard.png) -------------------------------------------------------------------------------- /docs/dashboards/navigate.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Navigate through Dashboards 4 | nav_order: 4 5 | parent: Dashboards 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | # Navigate through your dashboards 11 | 12 | GrimoireLab allow its users to create various dashboards to monitor and analyze 13 | several aspects of their project and community. In order to navigate into your 14 | different dashboards, look for the dashboards navigation bar. 15 | 16 | ![navigation bar](./assets/navigate.png) 17 | 18 | You can also look at your dashboards using the dashboards option from the 19 | sidebar. 20 | 21 | ![dashboards option](./assets/dashboard-option.png) -------------------------------------------------------------------------------- /docs/dashboards/query.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Query data 4 | nav_order: 3 5 | parent: Dashboards 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | # How to query data in Elasticsearch using Dev tools 11 | 12 | GrimoireLab allows its users to interact with the data through its dev tools. 13 | 14 | ### Steps 15 | 16 | - Look for Dev Tools in the sidebar and open it. 17 | 18 | ![dev tools](./assets/dev-tools.png) 19 | 20 | - Once you have dev tools open, the following interface should be visible to 21 | you. The console is divided into two sections, an editor panel and a response 22 | panel. Type your query in the editor panel and press the green play button to 23 | submit your query. 24 | 25 | ![dev tool interface](./assets/dev-tool-interface.png) 26 | 27 | For example, the query below counts the number of unique authors on a git 28 | repository from 2018-01-01 to 2021-01-01 29 | 30 | ``` 31 | GET _search 32 | { 33 | "size":"0", 34 | "query": { 35 | "range": { 36 | "author_date": { 37 | "gte": "2018-01-01T00:00:00", 38 | "lte": "2021-01-01T00:00:00" 39 | } 40 | } 41 | }, 42 | "aggs": { 43 | "author_unique": { 44 | "cardinality": { 45 | "field": "author_id" 46 | } 47 | } 48 | } 49 | } 50 | ``` 51 | 52 | The `size` parameter specifies that we do not need the data fetched from the 53 | index, this is included as only count of authors is required and not data of 54 | authors. The `query` parameter includes the `range` of the field `author_date` 55 | for which the data is requested. The number of distinct author are calculated 56 | using `cardinality` of `author_id`. 57 | 58 | #### Output 59 | 60 | ``` 61 | { 62 | "took" : 130, 63 | "timed_out" : false, 64 | "_shards" : { 65 | "total" : 239, 66 | "successful" : 239, 67 | "skipped" : 204, 68 | "failed" : 0 69 | }, 70 | "hits" : { 71 | "total" : 47860, 72 | "max_score" : 0.0, 73 | "hits" : [ ] 74 | }, 75 | "aggregations" : { 76 | "author_unique" : { 77 | "value" : 493 78 | } 79 | } 80 | } 81 | ``` 82 | 83 | The `value` field hold the number of distinct authors for the time frame 84 | provided. 85 | 86 | ![dev output](./assets/dev-output.png) 87 | 88 | You can refer to the [ElasticSearch 89 | Docs](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-search.html) 90 | to get an overview on how to use queries. 91 | 92 | You can also refer to the [data source 93 | schemas](https://github.com/chaoss/grimoirelab-elk/tree/master/schema) to get an 94 | idea on their corresponding fields. -------------------------------------------------------------------------------- /docs/dashboards/remove.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Remove a dashboard 4 | nav_order: 7 5 | parent: Dashboards 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | # How to remove a dashboard ? 11 | 12 | **Note**: You need to be [logged in](/docs/dashboards/access) for perform this 13 | action. 14 | 15 | Steps 16 | 1. Click on the `Dashboard` icon in the sidebar. It is the third icon from 17 | the top. 18 | 2. Search for the dashboards to be removed and check the checkboxes. 19 | 3. Click on the `Delete Selected` button to delete the dashboard. 20 | 21 | ![delete dashboard](./assets/delete-dashboards.png) -------------------------------------------------------------------------------- /docs/dashboards/visualization/assets/buckets_set.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/visualization/assets/buckets_set.png -------------------------------------------------------------------------------- /docs/dashboards/visualization/assets/color.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/visualization/assets/color.png -------------------------------------------------------------------------------- /docs/dashboards/visualization/assets/color_palette.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/visualization/assets/color_palette.png -------------------------------------------------------------------------------- /docs/dashboards/visualization/assets/delete-viz.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/visualization/assets/delete-viz.png -------------------------------------------------------------------------------- /docs/dashboards/visualization/assets/index.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/visualization/assets/index.png -------------------------------------------------------------------------------- /docs/dashboards/visualization/assets/metrics_set.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/visualization/assets/metrics_set.png -------------------------------------------------------------------------------- /docs/dashboards/visualization/assets/save_visualization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/visualization/assets/save_visualization.png -------------------------------------------------------------------------------- /docs/dashboards/visualization/assets/visualization-type.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/visualization/assets/visualization-type.png -------------------------------------------------------------------------------- /docs/dashboards/visualization/assets/visualization_interface.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/visualization/assets/visualization_interface.png -------------------------------------------------------------------------------- /docs/dashboards/visualization/assets/visualizations.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/visualization/assets/visualizations.png -------------------------------------------------------------------------------- /docs/dashboards/visualization/assets/visualize.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/dashboards/visualization/assets/visualize.png -------------------------------------------------------------------------------- /docs/dashboards/visualization/edit.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Edit a visualization 4 | nav_order: 2 5 | parent: Visualizations 6 | grand_parent: Dashboards 7 | has_children: false 8 | has_toc: false 9 | --- 10 | 11 | # Edit a visualization 12 | 13 | In order to edit a visualization and save it, you need to be [logged 14 | in]({{ site.baseurl }}{% link docs/dashboards/access.md %}). 15 | - Steps 16 | 1. Click on `Edit` in the navigation bar of GrimoireLab. 17 | 2. Click on the `Settings` icon of the dashboard. This will open a dropdown. 18 | 3. Click on `Edit Visualization`. 19 | 4. Change/Add your Metrics and Buckets and press the play button to visualize it. 23 | 5. Once satisfied, `Save` your visualization. 24 | 25 | 26 | **Note**: Refer to [how to create a visualization]({{ site.baseurl }}{% link 27 | docs/dashboards/visualization/add.md %}) to understand the editing interface. 28 | 29 | ## How to change the color of visualizations 30 | 31 | Some visualizations make use of graphs, charts and tables to represent data. In 32 | order to represent the different data, several contrasting colors are used. 33 | 34 | - Steps 35 | 1. Find the panel in which you want to change the color. 36 | 2. Every panel that makes use of colors will have the sample color and the 37 | data it represents on the right side of the panel. In the case it is not 38 | visible, click on the arrow key on the right side of the panel. 39 | 40 | ![color selection](./assets/color.png) 41 | 42 | 3. Once you've clicked on the sample color, a palette of colors will be 43 | displayed. You just have to choose your preferred color. 44 | 45 | ![color palette](./assets/color_palette.png) 46 | -------------------------------------------------------------------------------- /docs/dashboards/visualization/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Visualizations 4 | nav_order: 8 5 | parent: Dashboards 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Visualizations 11 | 12 | The best way to understand your data is to visualize it. GrimoireLab allows 13 | users to pick one or more index patterns and create visualizations to bring 14 | clarity about their data. It can be displayed in the form of charts, tables, 15 | maps and many more which allow you to compare your data side-by-side to identify 16 | patterns and connections 17 | 18 | ![visualizations](./assets/visualizations.png) -------------------------------------------------------------------------------- /docs/dashboards/visualization/remove.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Remove a visualization 4 | nav_order: 3 5 | parent: Visualizations 6 | grand_parent: Dashboards 7 | has_children: false 8 | has_toc: false 9 | --- 10 | 11 | # Remove visualization from dashboard 12 | 13 | For this action, you need to [logged in]({{ site.baseurl }}{% link 14 | docs/dashboards/access.md %}) as well. 15 | - Steps 16 | 1. Click on `Edit` in the navigation bar of GrimoireLab. 17 | 2. Click on the `Settings` icon of the dashboard. This will open a dropdown. 18 | 3. Click on `Delete from dashboard`. 19 | 4. Press on `Save` in the navigation bar to save it. 20 | 21 | **Note**: The above action removes the visualization from the current dashboard 22 | only but the visualization would still be available in your visualizations 23 | lists. In case you want to remove a dashboard entirely, follow the next steps. 24 | 25 | ## Delete a visualization 26 | 27 | - Steps 28 | 1. Click on `Visualize` on the sidebar. It is the second icon from the top. 29 | 30 | ![visualize](./assets/visualize.png) 31 | 32 | 2. Search of the needed visualization and check it. 33 | 3. Once you've checked the required visualizations, you'll see a trash icon 34 | next to the search bar. 35 | 36 | ![delete visualization](./assets/delete-viz.png) 37 | 38 | 4. Click on the trash icon to delete the checked visualizations. 39 | 40 | Note: You need to be [logged in]({{ site.baseurl }}{% link 41 | docs/dashboards/access.md %}) to be able to perform the above action. -------------------------------------------------------------------------------- /docs/data-sources/add-configurations.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Add configurations 4 | nav_order: 3 5 | parent: Data Sources 6 | --- 7 | 8 | # How to analyze data from software development repositories 9 | 10 | GrimoireLab supports a variety of software development platforms. In the case 11 | you want to analyze data from one of those platforms, first check whether the 12 | platform in question is supported by GrimoireLab. You can check the [supported 13 | data sources]({{ site.baseurl }}{% link docs/data-sources/supported.md %}) 14 | section. 15 | 16 | Once you've confirmed your data source, you will have to do some changes in both 17 | `project.json` and `setup.cfg`. 18 | 19 | For example, let's just say you want to analyze the commits of one particular 20 | project from your github repo. 21 | 22 | ### projects.json 23 | 24 | Replace the below configurations in the `projects.json` file. 25 | 26 | ``` 27 | { 28 | "grimoirelab": { 29 | "git": [ 30 | "https://github.com/chaoss/grimoirelab-toolkit.git" 31 | ] 32 | } 33 | } 34 | ``` 35 | 36 | ### setup.cfg 37 | 38 | Replace the below configurations in the `setup.cfg` file. 39 | 40 | ``` 41 | [git] 42 | raw_index = git_raw 43 | enriched_index = git_enriched 44 | latest-items = true 45 | studies = [enrich_demography:git, enrich_git_branches:git, enrich_areas_of_code:git, enrich_onion:git, enrich_extra_data:git] # (optional) 46 | 47 | 48 | 49 | [enrich_demography:git] 50 | 51 | [enrich_git_branches:git] 52 | run_month_days = [1, 23] 53 | 54 | [enrich_areas_of_code:git] 55 | in_index = git_raw 56 | out_index = git-aoc_enriched 57 | 58 | [enrich_onion:git] 59 | in_index = git_enriched 60 | out_index = git-onion_enriched 61 | 62 | [enrich_forecast_activity] 63 | out_index = git_study_forecast 64 | ``` 65 | 66 | Once you have made the following changes, run your containers with 67 | docker-compose 68 | 69 | ```bash 70 | docker-compose up -d 71 | ``` 72 | 73 | Give it some time to gather the data and after a while your dashboard and data 74 | should be ready at `http://localhost:5601`. 75 | 76 | ![dashboard](./assets/dashboard.png) 77 | 78 | In the case you need to add another data source, please refer to [how to 79 | configure projects.json & setup.cfg]({{ site.baseurl }}{% link 80 | docs/getting-started/configure.md %}) files and also the 81 | [configurations]({{site.baseurl }}{% link docs/data-sources/configurations.md 82 | %}). 83 | -------------------------------------------------------------------------------- /docs/data-sources/assets/dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/data-sources/assets/dashboard.png -------------------------------------------------------------------------------- /docs/data-sources/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Data Sources 4 | permalink: /data-sources 5 | nav_order: 4 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Data Sources 11 | 12 | A data source in GrimoireLab refers to a platform (GitHub, Slack, Dockerhub 13 | etc...) from where GrimoireLab can collect data in order to provide you with a 14 | holistic picture of software development projects. Currently GrimoireLab 15 | supports **30+ different sources** and it keeps on increasing. 16 | 17 | In order to connect the data source of your choice to GrimoireLab, first check 18 | whether it is [supported in GrimoireLab]({{ site.baseurl }}{% link 19 | docs/data-sources/supported.md %}). 20 | -------------------------------------------------------------------------------- /docs/data-sources/schema.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Schema 4 | nav_order: 4 5 | parent: Data Sources 6 | --- 7 | 8 | # Schema 9 | 10 | Every supported data source has a data model that is used for the indexes and 11 | panels in GrimoireLab. The fields described in each data model are the fields 12 | that are expected from ElasticSearch after the enrichment process. 13 | 14 | You can check out the data models of the supported data sources 15 | [chaoss/grimoirelab-elk - 16 | schema](https://github.com/chaoss/grimoirelab-elk/tree/master/schema). Each CSV 17 | file will have minimum two columns, name and type. 18 | 19 | | name | type | 20 | 21 | `name` refers to the actual field. Eg `author_name`, `grimoire_creation_date` 22 | 23 | `type` refers to the category of the field. Eg `boolean`, `long` 24 | 25 | Some CSV file might have four columns, name, type, aggregatable and description. 26 | 27 | | name | type | aggregatable | description | 28 | 29 | `aggregatable` refers to whether the field can be aggregated. 30 | 31 | `description` gives a brief about the field. 32 | -------------------------------------------------------------------------------- /docs/data-sources/supported.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Supported 4 | nav_order: 1 5 | parent: Data Sources 6 | --- 7 | 8 | # Supported Data Sources 9 | 10 | For GrimoireLab, any system that stores data relevant for analyzing state of 11 | software development is a software development repository. GrimoireLab supports 12 | many of them, from simple chats to code review platforms and many more. 13 | 14 | - **Code**: [Git]({{site.baseurl}}/docs/data-sources/configurations/#git) 15 | - **Ticket/Issues**: 16 | [Bugzilla]({{site.baseurl}}/docs/data-sources/configurations/#bugzilla), 17 | [BugzillaREST]({{site.baseurl}}/docs/data-sources/configurations/#bugzillarest), 18 | [Github]({{site.baseurl}}/docs/data-sources/configurations/#github), 19 | [GitLab]({{site.baseurl}}/docs/data-sources/configurations/#gitlab), 20 | [Jira]({{site.baseurl}}/docs/data-sources/configurations/#jira), 21 | [Phabricator]({{site.baseurl}}/docs/data-sources/configurations/#phabricator), 22 | [Redmine]({{site.baseurl}}/docs/data-sources/configurations/#redmine), 23 | [Pagure]({{site.baseurl}}/docs/data-sources/configurations/#pagure) 24 | - **Code Review**: 25 | [Gerrit]({{site.baseurl}}/docs/data-sources/configurations/#gerrit) 26 | - **Containers/Packaging**: 27 | [DockerHub]({{site.baseurl}}/docs/data-sources/configurations/#dockerhub), 28 | [crates]({{site.baseurl}}/docs/data-sources/configurations/#crates) 29 | - **Continuous Integration**: 30 | [Jenkins]({{site.baseurl}}/docs/data-sources/configurations/#jenkins), 31 | [Puppetforge]({{site.baseurl}}/docs/data-sources/configurations/#puppetforge), 32 | [functest]({{site.baseurl}}/docs/data-sources/configurations/#functest) 33 | - **Wiki**: 34 | [Confluence]({{site.baseurl}}/docs/data-sources/configurations/#confluence), 35 | [MediaWiki]({{site.baseurl}}/docs/data-sources/configurations/#mediawiki) 36 | - **Questions and Answer Forums**: 37 | [Askbot]({{site.baseurl}}/docs/data-sources/configurations/#askbot), 38 | [Discourse]({{site.baseurl}}/docs/data-sources/configurations/#discourse), 39 | [Stack 40 | Exchange]({{site.baseurl}}/docs/data-sources/configurations/#stackexchange), 41 | [KitSune]({{site.baseurl}}/docs/data-sources/configurations/#kitsune) 42 | - **Mailing Lists**: 43 | [Pipermail]({{site.baseurl}}/docs/data-sources/configurations/#pipermail), 44 | [NNTP]({{site.baseurl}}/docs/data-sources/configurations/#nntp), 45 | [HyperKitty]({{site.baseurl}}/docs/data-sources/configurations/#hyperkitty), 46 | [Groupsio]({{site.baseurl}}/docs/data-sources/configurations/#groupsio), 47 | [mbox]({{site.baseurl}}/docs/data-sources/configurations/#mbox) 48 | - **Chats**: [Slack]({{site.baseurl}}/docs/data-sources/configurations/#slack), 49 | [Telegram]({{site.baseurl}}/docs/data-sources/configurations/#telegram), 50 | [Supybot]({{site.baseurl}}/docs/data-sources/configurations/#supybot), 51 | [RocketChat]({{site.baseurl}}/docs/data-sources/configurations/#rocketchat), 52 | [Mattermost]({{site.baseurl}}/docs/data-sources/configurations/#mattermost), 53 | [Gitter]({{site.baseurl}}/docs/data-sources/configurations/#gitter) 54 | - **Meetings/Events**: 55 | [Meetup]({{site.baseurl}}/docs/data-sources/configurations/#meetup), 56 | [Mozillaclub]({{site.baseurl}}/docs/data-sources/configurations/#mozillaclub), 57 | [remo]({{site.baseurl}}/docs/data-sources/configurations/#remo) 58 | - **Social Networks**: 59 | [Twitter]({{site.baseurl}}/docs/data-sources/configurations/#twitter) 60 | - **Others**: [RSS]({{site.baseurl}}/docs/data-sources/configurations/#rss), 61 | [Cocom]({{site.baseurl}}/docs/data-sources/configurations/#cocom), 62 | [Colic]({{site.baseurl}}/docs/data-sources/configurations/#colic), 63 | [Weblate]({{site.baseurl}}/docs/data-sources/configurations/#weblate), 64 | [google_hits]({{site.baseurl}}/docs/data-sources/configurations/#google_hits), 65 | [Dockerdeps]({{site.baseurl}}/docs/data-sources/configurations/#dockerdeps), 66 | [dockersmells]({{site.baseurl}}/docs/data-sources/configurations/#dockersmells) 67 | 68 | We also support other data sources. You can check it out [chaoss/grimoirelab - 69 | community 70 | components](https://github.com/chaoss/grimoirelab/blob/master/community_components.md#other-data-sources). 71 | -------------------------------------------------------------------------------- /docs/getting-started/assets/dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/getting-started/assets/dashboard.png -------------------------------------------------------------------------------- /docs/getting-started/assets/filters.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/getting-started/assets/filters.png -------------------------------------------------------------------------------- /docs/getting-started/configure.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Configure GrimoireLab 4 | nav_order: 2 5 | parent: Getting Started 6 | has_children: true 7 | has_toc: false 8 | --- 9 | 10 | # Configure GrimoireLab 11 | 12 | GrimoireLab can be setup through the use of two main configuration files, 13 | `projects.json` and `setup.cfg`. The role and usage of each one of those files 14 | are described in their corresponding sections. 15 | 16 | - [projects.json]({{ site.baseurl }}{% link 17 | docs/getting-started/projects-json.md %}) 18 | - [setup.cfg]({{ site.baseurl }}{% link docs/getting-started/setup-cfg.md %}) 19 | -------------------------------------------------------------------------------- /docs/getting-started/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Getting Started 4 | permalink: /getting-started 5 | nav_order: 3 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Getting Started 11 | 12 | GrimoireLab is a set of tools, and to ease the trouble with installation we are 13 | providing a default setup to analyze Git activity for the grimoirelab 14 | repository. In order to have this default setup running, follow the [setup 15 | grimoirelab]({{ site.baseurl }}{% link docs/getting-started/setup.md %}) 16 | section. 17 | 18 | In case you want to configure grimoirelab to include your own data source, refer 19 | to [How to add a data source]({{ site.baseurl }}{% link 20 | docs/data-sources/add-configurations.md %}). 21 | -------------------------------------------------------------------------------- /docs/getting-started/projects-json.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: projects.json 4 | nav_order: 1 5 | parent: Configure GrimoireLab 6 | grand_parent: Getting Started 7 | --- 8 | 9 | # projects.json 10 | 11 | The `projects.json` aims at describing the repositories grouped by project that 12 | will be shown on the dashboards. 13 | 14 | The project file enables the users to list the instances of the software 15 | development tools to analyse, such as local and remote Git repositories, the 16 | URLs of GitHub and GitLab issue trackers and the name of Slack channels. 17 | Furthermore, it also allows the users to organize these instances into nested 18 | groups, which structure is reflected in the visualization artifacts (i.e., 19 | documents and dashboards). Groups can be useful to represent projects within a 20 | single company, sub-projects within a large project such as Linux and Eclipse, 21 | or the organizations within a collaborative project. 22 | 23 | 1. **First level**: project names 24 | 2. **Second level**: data source and metadata 25 | 3. **Third level**: data source URLs 26 | 27 | There are some filters, labels, and a special section: 28 | 29 | - `--filter-no-collection=true`: This filter is used to show old enriched data 30 | within the dashboards from repositories that don't exist anymore in upstream. 31 | - `--filter-raw` and the section `unknown`: The data sources will only be 32 | collected at the section `unknown` but this allows user to add the same data 33 | source in different sections to enrich using the filter `--filter-raw`. 34 | - Label ` --labels=[example]`: The data source will have the label of `example` 35 | which can be used to create visualisations for specific sets of data 36 | - Section `unknown`: If the data source is only under this section it will be 37 | enriched as project `main`. 38 | 39 | ``` 40 | { 41 | "Chaoss":{ 42 | "gerrit":[ 43 | "gerrit.chaoss.org --filter-raw=data.projects:CHAOSS" 44 | ], 45 | "git":[ 46 | "https:/github.com/chaoss/grimoirelab-perceval", 47 | "https:/github.com/chaoss/grimoirelab-sirmordred" 48 | ], 49 | "github":[ 50 | "https:/github.com/chaoss/grimoirelab-perceval --filter-no-collection=true", 51 | "https:/github.com/chaoss/grimoirelab-sirmordred --labels=[example]" 52 | ] 53 | }, 54 | "GrimoireLab":{ 55 | "gerrit":[ 56 | "gerrit.chaoss.org --filter-raw=data.projects:GrimoireLab" 57 | ] 58 | }, 59 | "unknown":{ 60 | "gerrit":[ 61 | "gerrit.chaoss.org" 62 | ], 63 | "confluence":[ 64 | "https://wiki.chaoss.org" 65 | ] 66 | } 67 | } 68 | ``` 69 | 70 | In the `projects.json` above: 71 | 72 | - The data included in the repo `gerrit.chaoss.org` will be collected entirely 73 | since the repo is listed in the `unknown` section. However only the project 74 | `GrimoireLab` will be enriched as declared in the `GrimoireLab` section. 75 | - In the section `Chaoss` in the data source `github` the repository 76 | `grimoirelab-perceval` is not collected for raw index but it will enriched in 77 | the enriched index. 78 | - In the section `GrimoireLab` the metadata will showed in the enriched index as 79 | extra fields. 80 | - In the section `unknown` the data source `confluence` will be enriched as the 81 | project `main`. 82 | -------------------------------------------------------------------------------- /docs/getting-started/setup.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Setup GrimoireLab 4 | nav_order: 1 5 | parent: Getting Started 6 | --- 7 | 8 | # Setup GrimoireLab 9 | 10 | There are currently two ways to setup GrimoireLab, either through **docker** or 11 | **docker-compose**. In this tutorial, we'll be talking about **docker-compose 12 | only** due to the fact that it's currently the easiest and simplest method to 13 | get started with. 14 | 15 | ## Requirements 16 | 17 | - [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) 18 | - [Docker](https://docs.docker.com/engine/install/) 19 | - [Docker Compose](https://docs.docker.com/compose/install/) 20 | - At least 8GB RAM, 2 CPUs and 2GB virtual memory for ElasticSearch 21 | 22 | You can make sure that you have the above software and hardware requirements 23 | through the following means. 24 | 25 | ### Software 26 | 27 | ```bash 28 | git --version 29 | git version 2.32.0 30 | ``` 31 | ```bash 32 | docker --version 33 | Docker version 20.10.7, build f0df35096d 34 | ``` 35 | ```bash 36 | docker-compose --version 37 | docker-compose version 1.28.5, build c4eb3a1f 38 | ``` 39 | 40 | ### Hardware 41 | 42 | ```bash 43 | cat /proc/cpuinfo | grep processor | wc -l #View number of processors 44 | 4 45 | ``` 46 | ```bash 47 | grep MemTotal /proc/meminfo #View amount of RAM available 48 | MemTotal: 8029848 kB 49 | ``` 50 | ```bash 51 | sudo sysctl -w vm.max_map_count=262144 #Set virtual memory 52 | vm.max_map_count = 262144 53 | ``` 54 | 55 | The reason for allocating `262144` for memory is the check that ElasticSearch 56 | performs on boot. It ensures that the kernel allows at least 261144 memory 57 | mapped areas. 58 | 59 | ## Steps 60 | 61 | - Clone the GrimoireLab repo: 62 | 63 | ```bash 64 | git clone https://github.com/chaoss/grimoirelab 65 | ``` 66 | 67 | - Go to `docker-compose` folder and run the following command: 68 | 69 | ```bash 70 | cd grimoirelab/docker-compose 71 | sudo docker-compose up -d 72 | ``` 73 | 74 | Your dashboard will be ready after a while at `http://localhost:5601`. Usually, 75 | it takes about 10-15 minutes to show the data in the dashboards. But, the 76 | waiting time depends on the size of the repositories. 77 | 78 | ![dashboard](./assets/dashboard.png) 79 | 80 | Please check the [Troubleshooting]({{ site.baseurl }}{% link 81 | docs/getting-started/troubleshooting.md %}) page if you face any issues. 82 | -------------------------------------------------------------------------------- /docs/install/figs/bitergia_analytics_screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/install/figs/bitergia_analytics_screenshot.png -------------------------------------------------------------------------------- /docs/install/figs/dashboard-grimoirelab.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/install/figs/dashboard-grimoirelab.png -------------------------------------------------------------------------------- /docs/install/figs/kibana_welcome.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/install/figs/kibana_welcome.png -------------------------------------------------------------------------------- /docs/install/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Installation alternatives 4 | permalink: /install 5 | nav_order: 9 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Installation alternatives 11 | 12 | If the default installation method doesn't suite your use case here's a 13 | collection of alternative installation methods. 14 | 15 | The list and the details may be outdated. Feel free to contribute 16 | updates. 17 | -------------------------------------------------------------------------------- /docs/install/quick.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Installation for the impatient 4 | nav_order: 1 5 | parent: Installation alternatives 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | ## Installation for the impatient 11 | 12 | So you don't want to read all the details, 13 | you just want to install the stuff and follow on 14 | to other parts of this tutorial? 15 | 16 | * If you want to install GrimoireLab Python packages, 17 | and already know about Python virtual environments, `pip`, 18 | and that stuff, go ahead to [Install Python packages](#python). 19 | 20 | * If you want to use the GrimoireLab Docker images, 21 | and you already have Docker installed, know about 22 | `docker run` and all that, go ahead to [Run in a Docker container](#docker). 23 | 24 | In any case, have a look at the introduction of this chapter, 25 | so that you can get a basic idea of the requirements to run GrimoireLab, 26 | and to the section on What is GrimoireLab, where you will get 27 | a glimpse of the different components in the toolset. 28 | 29 | ### Install Python packages {#python} 30 | 31 | All GrimoireLab packages are pulled by a single Python package, 32 | available from [Pypi](https://pypi.org). If you have a recent version 33 | of Python3, very likely the following will just work 34 | (it is recommended to run it in a virtual environment): 35 | 36 | You may need to have some other non-Python packages installed previously. 37 | Please check the [section on installing non-Python 38 | packages](install.html#non-python-pkgs) if you have any trouble. 39 | 40 | ```bash 41 | (gl) $ pip3 install grimoirelab 42 | ``` 43 | 44 | If everything went well, you can just check the version that you installed: 45 | 46 | ```bash 47 | (gl) $ grimoirelab -v 48 | ``` 49 | 50 | And that's it. You can now skip the rest of this chapter 51 | (although it would be a good idea if you at least browse it, 52 | there is useful information in it). 53 | 54 | ### Run in a Docker container {#docker} 55 | 56 | All GrimoireLab packages can also be used from a Docker image, 57 | that can be pulled and run directly from [Docker Hub](https://hub.docker.com). 58 | Those are exactly the same packages that you get if you install 59 | from [Pypi](https://pypi.org), as shown above. 60 | But in this case, they are already installed on top of a 61 | standard Debian distro, so you can run those directly. 62 | 63 | To run that image, just type: 64 | 65 | ```bash 66 | docker run -p 127.0.0.1:9200:9200 \ 67 | -p 127.0.0.1:5601:5601 \ 68 | -p 127.0.0.1:3306:3306 \ 69 | -e RUN_MORDRED=NO \ 70 | -t grimoirelab/full 71 | ``` 72 | 73 | This will run the container, with the servers usually needed by 74 | some of GrimoireLab tools (Elasticsearch, Kibiter, MariaDB). 75 | It will also expose the ports for those servers 76 | (Elasticsearch is 9200, Kibiter is 5601, MariaDB is 3306), 77 | so that you can access them from the host machine. 78 | Make sure that you don't have other servers in those ports 79 | in the host machine (for example, an instance of MySQL, which 80 | also runs in 3306). 81 | 82 | Once the container is running, you can connect to it, 83 | and launch any GrimoireLab command or program in it: 84 | 85 | ```bash 86 | docker exec -it container_id env TERM=xterm /bin/bash 87 | ``` 88 | 89 | That container can be used also, as such, 90 | to produce a complete dashboard: see 91 | [Mordred in a container](../sirmordred/container.html). 92 | 93 | -------------------------------------------------------------------------------- /docs/python/es-dsl.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: ElasticSearch DSL 4 | nav_order: 2 5 | parent: Python scripting 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | ## ElasticSearch DSL 12 | 13 | The `elasticsearch` Python module may seem good enough to query ElasticSearch via its REST API. But for using it, we need to write full JSON documents with the intended queries. And these documents may become large, complex, and a burden to maintain and understand. Here is where the [`elasticsearch_dsl` Python module](http://elasticsearch-dsl.readthedocs.io) comes to the rescue. 14 | 15 | To install it, just use pip: 16 | 17 | ```bash 18 | pip install elasticsearch_dsl 19 | ``` 20 | 21 | It needs the `elasticsearch` Python module to work, but you'll have it already installed, or will be pulled in via dependencies, so don't worry about it. 22 | 23 | ### Get all contents in an index 24 | 25 | `elasticsearch_dsl` provides, among other goodies, a nice chainable API for building ElasticSearch requests (queries), and a convenient way to deal with responses. Let's start by showing a very simple example which works with the index we created for git commits in the chapter on Perceval (see the code as a file ready to run, 26 | [perceval_elasticsearch_git_dsl.py](scripts/perceval_elasticsearch_git_dsl.py)): 27 | 28 | ```python 29 | import elasticsearch 30 | import elasticsearch_dsl 31 | 32 | # ElasticSearch instance (url) 33 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 34 | 35 | # Build a DSL Search object on the 'commits' index, 'summary' document type 36 | request = elasticsearch_dsl.Search(using=es, index='commits', 37 | doc_type='summary') 38 | 39 | # Run the Search, using the scan interface to get all resuls 40 | response = request.scan() 41 | for commit in response: 42 | print(commit.hash, commit.author_date, commit.author) 43 | ``` 44 | 45 | After importing both modules, we create an object to connect to ElasticSearch intance, as we have always done with the `elasticsearch` module. Once we have it, new stuff enters into action. First, we create a 'request', by instantiating the `elasticsearch_dsl` `Search` class. It needs the object to connect, and the name of of the index of interest. In this case, we provide the document type as well (although that is not mandatory). 46 | 47 | Then, we obtain a reponse by calling the `scan()` method of the request. That call will produce real requests to the ElasticSearch REST API, using the scan interface. We use the scan interface because we want all documents, and on a potentially large index, this is the best way to do it. `scan()` returns a Python generator, taking care of sending new requests to ElasticSearch when needed. This, we can just iterate over it, getting all commits in the index, and printing them. 48 | 49 | ### Get only some fields 50 | 51 | This code can be made more efficient, by requesting only the fields we need, instead of getting all the data we have for documents in the `commits` index. For that, we can just chain a call to `source()` to the request we're using. The definition of the request is shown below (a complete script is available as 52 | [perceval_elasticsearch_git_dsl_2.py](scripts/perceval_elasticsearch_git_dsl_2.py)): 53 | 54 | ```python 55 | request = elasticsearch_dsl.Search(using=es, index='commits', 56 | doc_type='summary') 57 | request = request.source(['hash', 'author_date', 'author']) 58 | ``` 59 | 60 | The result is the same, but the bandwith needed, and the stress caused on the ElasticSearch server, are lower. Of course, the more unneeded data in the documents in the index, the more gain of this technique. 61 | 62 | ### Get the last commits 63 | 64 | DSL allows for specifying filters, order, bucketing, aggregations, and much more. And all of that is available via `elasticsearch_dsl`. For example, let's get the last 20 commits, by date of commit (for each commit, only hash, author date and author, as above). The complete script is available as 65 | 66 | [perceval_elasticsearch_git_dsl_3.py](scripts/perceval_elasticsearch_git_dsl_3.py), and the relevant fragment is below: 67 | 68 | ```python 69 | request = request.sort('-commit_date') 70 | request = request.source(['hash', 'author_date', 'author']) 71 | request = request[0:20] 72 | 73 | # Run the Search, using the execute interface to get ordered results 74 | response = request.execute() 75 | for commit in response: 76 | print(commit.hash, commit.author_date, commit.author) 77 | ``` 78 | 79 | Now, instead of `scan()`, we use `execute()` which allows for slicing (note the line where we slice `request`), and preserves order. 80 | -------------------------------------------------------------------------------- /docs/python/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Python scripting 4 | permalink: /python 5 | nav_order: 11 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Python scripting 11 | 12 | This describes the usage of Python scripts for data access and 13 | manipulation. 14 | -------------------------------------------------------------------------------- /docs/python/indexes.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Querying grimoire-elk indexes 4 | nav_order: 6 5 | parent: Python scripting 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | ## Querying grimoire-elk indexes 12 | 13 | As we discussed in section 14 | [A simple dashboard](../gelk/simple.html), 15 | for producing dashboards we can run tools that use the `grimoire-elk` Python module to produce 'enriched indexes'. Those indexes are designed to be used by Kibana visualizations, but they are also suitable for direct query. The main advantage of using these enriched indexes instead of the raw ones is that we have usually higher level data, usually closer to the data we look for most analysis. The drawback is that we don't have all the data available in the raw indexes, which in some cases mean that we miss the data we need. 16 | 17 | But when we have the data we need in them, they are easy to query, and the fact is that they can provide a good deal of quality information. Let's see how to query them using `elasticsearch_dsl`, using as an example the git enriched index. This is the one we produced for the the git Kibana dashboard in section [A simple dashboard](../gelk/simple.html). 18 | 19 | ### Common code for all the examples 20 | 21 | Before we can query the index, we need to import some modules, and declare the ElasticSearch instance we are going to use: 22 | 23 | ``` 24 | from datetime import datetime 25 | from pprint import pprint 26 | 27 | from elasticsearch import Elasticsearch 28 | from elasticsearch_dsl import Search 29 | 30 | es = Elasticsearch(["http://localhost:9200"]) 31 | index = 'git' 32 | ``` 33 | 34 | The first two lines import `datetime` and `pprint`, which we will use in the examples below. Then, we have the lines for importing `elasticsearch` and `elasticsearch_dsl` modules. Finally we define the ElasticSearch instance we're going to query (change that for your instance, if different), and the name of the index. In our case, we will use the name for the enriched git index, as created in section [A simple dashboard](../grimoireelk/a-simple-dashboard.md). 35 | 36 | You can see this code at the beginning of the file [enriched_elasticsearch_1.py](https://github.com/jgbarah/GrimoireLab-training/blob/master/python/scripts/enriched_elasticsearch_1.py), which includes the examples below ass well. 37 | 38 | ### Counting total number of commits 39 | 40 | ```python 41 | s = Search(using=es, index=index) 42 | total = s.count() 43 | print("Count of total number of commits in index: ", total) 44 | ``` 45 | 46 | # Counting number of unique commits 47 | 48 | ```python 49 | s = Search(using=es, index=index) 50 | s.aggs.metric('commits', 'cardinality', field='hash') 51 | unique = s.count() 52 | print("Count of unique commits in index: ", unique) 53 | ``` 54 | 55 | # Ignoring commits touching no files 56 | 57 | Counting number of unique commits, ignoring those touching no files. 58 | 59 | ```python 60 | s = Search(using=es, index=index) 61 | s = s.filter('range', files={'gt':0}) 62 | s.aggs.metric('commits', 'cardinality', field='hash') 63 | unique_no_empty = s.count() 64 | print("Count of unique commits in index: ", unique_no_empty) 65 | ``` 66 | 67 | # Counting commits newer than a date 68 | 69 | Counting number of unique commits, ignoring those touching no files, and newer than a certain date. 70 | 71 | ```python 72 | s = Search(using=es, index=index) 73 | s = s.filter('range', files={'gt':0}) 74 | s = s.filter('range', author_date={'gt': datetime(2016, 7, 1)}) 75 | s.aggs.metric('commits', 'cardinality', field='hash') 76 | unique_after = s.count() 77 | print("Count of unique commits in index, authored later than July 1st 2016: ", 78 | unique_after) 79 | ``` 80 | 81 | # Grouping commits by quarter 82 | 83 | Counting number of unique commits, ignoring those touching no files, and newer than a certain date, grouping them by quarter. 84 | 85 | ```python 86 | s = Search(using=es, index=index) 87 | s = s.filter('range', files={'gt':0}) 88 | s = s.filter('range', author_date={'gt': datetime(2016, 7, 1)}) 89 | s.aggs.metric('commits', 'cardinality', field='hash') 90 | s.aggs.bucket('histogram', 'date_histogram', 91 | field='author_date', interval='quarter') 92 | by_quarter = s.execute() 93 | print("Aggregations returned by quarter") 94 | pprint(by_quarter.to_dict()['aggregations']) 95 | for quarter in by_quarter.to_dict()['aggregations']['histogram']['buckets']: 96 | print("Unique commits for quarter starting on ", 97 | quarter['key_as_string'], 98 | ": ", quarter['doc_count']) 99 | ``` 100 | -------------------------------------------------------------------------------- /docs/python/querying.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Querying ElasticSearch 4 | nav_order: 5 5 | parent: Python scripting 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | # Querying ElasticSearch 12 | 13 | In the previous section we learned how to store Perceval data in ElasticSearch. Let's learn now how to query it. For this, we're going to use data from mailing lists. First, let's upload some messages to a fresh ElasticSearch index using the Perceval mbox backend \(script [perceval\_elasticsearch\_mbox\_1.py](scripts/perceval_elasticsearch_mbox_1.py)): 14 | 15 | ```python 16 | import email.utils 17 | 18 | from perceval.backends.core.mbox import MBox 19 | import elasticsearch 20 | 21 | # uri (label) for the mailing list to analyze 22 | mbox_uri = 'http://mail-archives.apache.org/mod_mbox/httpd-announce/' 23 | # directory for letting Perceval where mbox archives are 24 | # you need to have the archives to analyzed there before running the script 25 | mbox_dir = 'archives' 26 | # ElasticSearch instance (url) 27 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 28 | 29 | # Create the 'messages' index in ElasticSearch 30 | try: 31 | es.indices.create('messages') 32 | except elasticsearch.exceptions.RequestError: 33 | print('Index already exisits, remove it before running this script again.') 34 | exit() 35 | 36 | # create a mbox object, using mbox_uri as label, mbox_dir as directory to scan 37 | repo = MBox(uri=mbox_uri, dirpath=mbox_dir) 38 | 39 | # Fetch all commits as an iteratoir, and iterate it uploading to ElasticSearch 40 | print('Analyzing mbox archives...') 41 | # fetch all messages as an iteratoir 42 | for message in repo.fetch(): 43 | # Create the object (dictionary) to upload to ElasticSearch 44 | summary = { 45 | 'from': message['data']['From'], 46 | 'subject': message['data']['Subject'], 47 | 'date': email.utils.parsedate_to_datetime(message['data']['Date']) 48 | } 49 | print('.', end='') 50 | # Upload the object to ElasticSearch 51 | es.index(index='messages', doc_type='summary', body=summary) 52 | ``` 53 | 54 | And now, let's retrieve data from this recently created index. For example, we can get the sender and the subject for each stored message 55 | ([perceval_elasticsearch_mbox_2.py](scripts/perceval_elasticsearch_mbox_2.py)\): 56 | 57 | ```python 58 | import elasticsearch 59 | 60 | # ElasticSearch instance (url) 61 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 62 | 63 | # Query messages index, getting all items in it 64 | es_result = es.search(index="messages") 65 | # Print number of messages retrieved 66 | print("Found %d messages" % es_result['hits']['total']) 67 | for message in es_result['hits']['hits']: 68 | print("Sender: %s\n Subject: %s" % 69 | (message['_source']['from'], message['_source']['subject'])) 70 | ``` 71 | 72 | The key is the line calling `es.search`, where we are querying ElasticSearch. Since we only specify the index, we will get the whole index as a result. In fact, we get a dictionary, whose `hits` field includes several interesting fields: `total` for the total number of documents retrieved, and `hits`, for a list of the documents retrieved. 73 | 74 | For each of these documents, the data we uploaded is in the `_source` dictionary: we only need to retrieve the needed data from there \(in this case, `from` and `subject`\). 75 | 76 | The next script goes a step beyond, by searching for specific items \(documents\) in the index. In this case, we will ask for items whose `from` property matches (includes) the string `Jim` (see file 77 | [perceval_elasticsearch_mbox_3.py](scripts/perceval_elasticsearch_mbox_3.py)\): 78 | 79 | ```python 80 | import elasticsearch 81 | 82 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 83 | 84 | es_result = es.search(index="messages", doc_type='summary', 85 | body={"query": {"match": {"from": "Jim"}}}) 86 | 87 | print("Found %d messages" % es_result['hits']['total']) 88 | for message in es_result['hits']['hits']: 89 | print("Sender: %s\n Subject: %s" % 90 | (message['_source']['from'], message['_source']['subject'])) 91 | ``` 92 | 93 | The difference is in the parameters for the `es.search` function, which now include an specific `doc_type` \(`summary`,the one we used when uploading the items to ElasticSearch\), and a `body`, which will be interpreted as a query. 94 | 95 | -------------------------------------------------------------------------------- /docs/python/scripting.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Python scripting 4 | nav_order: 1 5 | parent: Python scripting 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | # Python scripting 11 | GrimoireLab is written in Python, and provides many Python modules that 12 | can be used from your scripts. We already saw how Perceval can be used 13 | from Python scripts. Now let's explore how we can walk that way into 14 | more detail. 15 | 16 | -------------------------------------------------------------------------------- /docs/python/scripts/cleanup.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | 3 | import elasticsearch 4 | from pprint import pprint 5 | 6 | indexes = ['github_grimoirelab', 'git_grimoirelab', 7 | 'github_grimoirelab-raw', 'git_grimoirelab-raw' ] 8 | 9 | es = elasticsearch.Elasticsearch(['http://localhost:9200']) 10 | 11 | aliases = es.indices.get_alias('*') 12 | pprint(aliases) 13 | 14 | all_schemas = es.indices.get('*') 15 | for index in all_schemas.keys(): 16 | print("Index: ", index) 17 | 18 | for index in indexes: 19 | print("Removing index: ", index) 20 | es.indices.delete(index=index, ignore=[400, 404]) 21 | 22 | all_schemas = es.indices.get('*') 23 | for index in all_schemas.keys(): 24 | print("Index: ", index) 25 | 26 | aliases = es.indices.get_alias('*') 27 | pprint(aliases) 28 | -------------------------------------------------------------------------------- /docs/python/scripts/enriched_elasticsearch_1.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2016 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | ## Some simple examples about how to query enriched indexes using 24 | ## elasticsearch_dsl 25 | 26 | from datetime import datetime 27 | from pprint import pprint 28 | 29 | from elasticsearch import Elasticsearch 30 | from elasticsearch_dsl import Search 31 | 32 | 33 | es = Elasticsearch(["http://localhost:9200"]) 34 | index = 'git' 35 | 36 | # Counting total number of commits 37 | s = Search(using=es, index=index) 38 | total = s.count() 39 | print("Count of total number of commits in index: ", total) 40 | 41 | # Counting number of unique commits 42 | s = Search(using=es, index=index) 43 | s.aggs.metric('commits', 'cardinality', field='hash') 44 | unique = s.count() 45 | print("Count of unique commits in index: ", unique) 46 | 47 | # Counting number of unique commits, ignoring those touching no files 48 | s = Search(using=es, index=index) 49 | s = s.filter('range', files={'gt':0}) 50 | s.aggs.metric('commits', 'cardinality', field='hash') 51 | unique_no_empty = s.count() 52 | print("Count of unique commits in index: ", unique_no_empty) 53 | 54 | # Counting number of unique commits, ignoring those touching no files 55 | # and newer than a date 56 | s = Search(using=es, index=index) 57 | s = s.filter('range', files={'gt':0}) 58 | s = s.filter('range', author_date={'gt': datetime(2016, 7, 1)}) 59 | s.aggs.metric('commits', 'cardinality', field='hash') 60 | unique_after = s.count() 61 | print("Count of unique commits in index, authored later than July 1st 2016: ", 62 | unique_after) 63 | 64 | # Counting number of unique commits, ignoring those touching no files 65 | # and newer than a date, grouping by quarter 66 | s = Search(using=es, index=index) 67 | s = s.filter('range', files={'gt':0}) 68 | s = s.filter('range', author_date={'gt': datetime(2016, 7, 1)}) 69 | s.aggs.metric('commits', 'cardinality', field='hash') 70 | s.aggs.bucket('histogram', 'date_histogram', 71 | field='author_date', interval='quarter') 72 | by_quarter = s.execute() 73 | print("Aggregations returned by quarter") 74 | pprint(by_quarter.to_dict()['aggregations']) 75 | for quarter in by_quarter.to_dict()['aggregations']['histogram']['buckets']: 76 | print("Unique commits for quarter starting on ", 77 | quarter['key_as_string'], 78 | ": ", quarter['doc_count']) 79 | -------------------------------------------------------------------------------- /docs/python/scripts/pandas_1.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2017 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | ## Some simple examples for exploring how to work with pandas 24 | ## and GrimoireLab indexes. 25 | 26 | from datetime import datetime 27 | 28 | from elasticsearch import Elasticsearch 29 | from elasticsearch_dsl import Search 30 | import pandas as pd 31 | 32 | es = Elasticsearch('http://localhost:9200', verify_certs=False) 33 | 34 | # Buckets by author name, finding first commit for each of them 35 | s = Search(using=es, index='git') 36 | s.aggs.bucket('by_authors', 'terms', field='author_name', size=10000) \ 37 | .metric('first_commit', 'min', field='author_date') 38 | s = s.sort("author_date") 39 | result = s.execute() 40 | 41 | # Uncomment the following two lines to see the resutls obtained 42 | # from the query 43 | #from pprint import pprint 44 | #pprint(result.to_dict()) 45 | 46 | # Get a dataframe with each author and their first commit 47 | buckets_result = result['aggregations']['by_authors']['buckets'] 48 | buckets = [] 49 | for bucket in buckets_result: 50 | first_commit = bucket['first_commit']['value']/1000 51 | buckets.append( 52 | {'first_commit': datetime.utcfromtimestamp(first_commit), 53 | 'author': bucket['key']} 54 | ) 55 | authors = pd.DataFrame.from_records(buckets) 56 | authors.sort_values(by='first_commit', ascending=False, inplace=True) 57 | 58 | # Uncomment the following line (and the import of pprint, above)ç 59 | # to print the dataframe 60 | #pprint(authors) 61 | 62 | # Get number of new authors per month 63 | by_month = authors['first_commit'] \ 64 | .groupby([authors.first_commit.dt.year, 65 | authors.first_commit.dt.month]) \ 66 | .agg('count') 67 | 68 | #pprint(by_month) 69 | 70 | # Produce csv files 71 | print("Creating CSV for new authors per month.") 72 | by_month.to_csv('authors_per_month.csv') 73 | print("Creating CSV for first date for authors.") 74 | authors.to_csv('authors_first.csv', 75 | columns=['first_commit', 'author'], 76 | index=False) 77 | -------------------------------------------------------------------------------- /docs/python/scripts/perceval_elasticsearch_1.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2016 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | 24 | from perceval.backends.core.git import Git 25 | import elasticsearch 26 | 27 | # Url for the git repo to analyze 28 | repo_url = 'http://github.com/grimoirelab/perceval.git' 29 | # Directory for letting Perceval clone the git repo 30 | repo_dir = '/tmp/perceval.git' 31 | # ElasticSearch instance (url) 32 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 33 | 34 | # Create the 'commits' index in ElasticSearch 35 | es.indices.create('commits') 36 | # Create a Git object, pointing to repo_url, using repo_dir for cloning 37 | repo = Git(uri=repo_url, gitpath=repo_dir) 38 | # Fetch all commits as an iteratoir, and iterate it uploading to ElasticSearch 39 | for commit in repo.fetch(): 40 | # Create the object (dictionary) to upload to ElasticSearch 41 | summary = {'hash': commit['data']['commit']} 42 | print(summary) 43 | # Upload the object to ElasticSearch 44 | es.index(index='commits', doc_type='summary', body=summary) 45 | -------------------------------------------------------------------------------- /docs/python/scripts/perceval_elasticsearch_2.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2016 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | 24 | from perceval.backends.core.git import Git 25 | import elasticsearch 26 | 27 | # Url for the git repo to analyze 28 | repo_url = 'http://github.com/grimoirelab/perceval.git' 29 | # Directory for letting Perceval clone the git repo 30 | repo_dir = '/tmp/perceval.git' 31 | # ElasticSearch instance (url) 32 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 33 | 34 | # Create the 'commits' index in ElasticSearch 35 | # if it already exisits, first delete it and then create it. 36 | try: 37 | es.indices.create('commits') 38 | except elasticsearch.exceptions.RequestError: 39 | es.indices.delete('commits') 40 | es.indices.create('commits') 41 | # Create a Git object, pointing to repo_url, using repo_dir for cloning 42 | repo = Git(uri=repo_url, gitpath=repo_dir) 43 | # Fetch all commits as an iteratoir, and iterate it uploading to ElasticSearch 44 | for commit in repo.fetch(): 45 | # Create the object (dictionary) to upload to ElasticSearch 46 | summary = {'hash': commit['data']['commit']} 47 | print(summary) 48 | # Upload the object to ElasticSearch 49 | es.index(index='commits', doc_type='summary', body=summary) 50 | -------------------------------------------------------------------------------- /docs/python/scripts/perceval_elasticsearch_3.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2016 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | 24 | from perceval.backends.core.git import Git 25 | import elasticsearch 26 | 27 | # Url for the git repo to analyze 28 | repo_url = 'http://github.com/grimoirelab/perceval.git' 29 | # Directory for letting Perceval clone the git repo 30 | repo_dir = '/tmp/perceval.git' 31 | # ElasticSearch instance (url) 32 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 33 | 34 | # Create the 'commits' index in ElasticSearch 35 | try: 36 | es.indices.create('commits') 37 | except elasticsearch.exceptions.RequestError: 38 | print('Index already exisits, remove it before running this script again.') 39 | exit() 40 | # Create a Git object, pointing to repo_url, using repo_dir for cloning 41 | repo = Git(uri=repo_url, gitpath=repo_dir) 42 | # Fetch all commits as an iteratoir, and iterate it uploading to ElasticSearch 43 | print('Analyzing git repo...') 44 | for commit in repo.fetch(): 45 | # Create the object (dictionary) to upload to ElasticSearch 46 | summary = { 47 | 'hash': commit['data']['commit'], 48 | 'author': commit['data']['Author'], 49 | 'author_date': commit['data']['AuthorDate'], 50 | 'commit': commit['data']['Commit'], 51 | 'commit_date': commit['data']['CommitDate'], 52 | 'files_no': len(commit['data']['files']) 53 | } 54 | print('.', end='') 55 | # Upload the object to ElasticSearch 56 | es.index(index='commits', doc_type='summary', body=summary) 57 | 58 | print('\nCreated new index with commits.') 59 | -------------------------------------------------------------------------------- /docs/python/scripts/perceval_elasticsearch_4.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2016 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | 24 | import datetime 25 | 26 | from perceval.backends.core.git import Git 27 | import elasticsearch 28 | 29 | # Url for the git repo to analyze 30 | repo_url = 'http://github.com/grimoirelab/perceval.git' 31 | # Directory for letting Perceval clone the git repo 32 | repo_dir = '/tmp/perceval.git' 33 | # ElasticSearch instance (url) 34 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 35 | 36 | # Create the 'commits' index in ElasticSearch 37 | try: 38 | es.indices.create('commits') 39 | except elasticsearch.exceptions.RequestError: 40 | print('Index already exisits, remove it before running this script again.') 41 | exit() 42 | # Create a Git object, pointing to repo_url, using repo_dir for cloning 43 | repo = Git(uri=repo_url, gitpath=repo_dir) 44 | # Fetch all commits as an iteratoir, and iterate it uploading to ElasticSearch 45 | print('Analyzing git repo...') 46 | for commit in repo.fetch(): 47 | # Create the object (dictionary) to upload to ElasticSearch 48 | summary = { 49 | 'hash': commit['data']['commit'], 50 | 'author': commit['data']['Author'], 51 | 'author_date': datetime.datetime.strptime(commit['data']['AuthorDate'], 52 | "%a %b %d %H:%M:%S %Y %z"), 53 | 'commit': commit['data']['Commit'], 54 | 'commit_date': datetime.datetime.strptime(commit['data']['CommitDate'], 55 | "%a %b %d %H:%M:%S %Y %z"), 56 | 'files_no': len(commit['data']['files']) 57 | } 58 | print('.', end='') 59 | # Upload the object to ElasticSearch 60 | es.index(index='commits', doc_type='summary', body=summary) 61 | 62 | print('\nCreated new index with commits.') 63 | -------------------------------------------------------------------------------- /docs/python/scripts/perceval_elasticsearch_git_dsl.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2016 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | 24 | import elasticsearch 25 | import elasticsearch_dsl 26 | 27 | # ElasticSearch instance (url) 28 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 29 | 30 | # Build a DSL Search object on the 'commits' index, 'summary' documents type 31 | request = elasticsearch_dsl.Search(using=es, index='commits', 32 | doc_type='summary') 33 | 34 | # Run the Search, using the scan interface to get all resuls 35 | response = request.scan() 36 | for commit in response: 37 | print(commit.hash, commit.author_date, commit.author) 38 | -------------------------------------------------------------------------------- /docs/python/scripts/perceval_elasticsearch_git_dsl_2.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2016 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | 24 | import elasticsearch 25 | import elasticsearch_dsl 26 | 27 | # ElasticSearch instance (url) 28 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 29 | 30 | # Build a DSL Search object on the 'commits' index, 'summary' documents type 31 | request = elasticsearch_dsl.Search(using=es, index='commits', 32 | doc_type='summary') 33 | # We need to request only some fields 34 | request = request.source(['hash', 'author_date', 'author']) 35 | 36 | # Run the Search, using the scan interface to get all resuls 37 | response = request.scan() 38 | for commit in response: 39 | print(commit.hash, commit.author_date, commit.author) 40 | -------------------------------------------------------------------------------- /docs/python/scripts/perceval_elasticsearch_git_dsl_3.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2016 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | 24 | import elasticsearch 25 | import elasticsearch_dsl 26 | 27 | # ElasticSearch instance (url) 28 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 29 | 30 | # Build a DSL Search object on the 'commits' index, 'summary' documents type 31 | request = elasticsearch_dsl.Search(using=es, index='commits', 32 | doc_type='summary') 33 | # We want the last 20 commits, by commit timestamp, with only some fields 34 | request = request.sort('-commit_date') 35 | request = request.source(['hash', 'author_date', 'author']) 36 | request = request[0:20] 37 | 38 | # Run the Search, using the execute interface to get ordered results 39 | response = request.execute() 40 | for commit in response: 41 | print(commit.hash, commit.author_date, commit.author) 42 | -------------------------------------------------------------------------------- /docs/python/scripts/perceval_elasticsearch_mbox_1.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2016 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | 24 | import email.utils 25 | 26 | from perceval.backends.core.mbox import MBox 27 | import elasticsearch 28 | 29 | # uri (label) for the mailing list to analyze 30 | mbox_uri = 'http://mail-archives.apache.org/mod_mbox/httpd-announce/' 31 | # directory for letting Perceval where mbox archives are 32 | # you need to have the archives to analyzed there before running the script 33 | mbox_dir = 'archives' 34 | # ElasticSearch instance (url) 35 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 36 | 37 | # Create the 'messages' index in ElasticSearch 38 | try: 39 | es.indices.create('messages') 40 | except elasticsearch.exceptions.RequestError: 41 | print('Index already exisits, remove it before running this script again.') 42 | exit() 43 | 44 | # create a mbox object, using mbox_uri as label, mbox_dir as directory to scan 45 | repo = MBox(uri=mbox_uri, dirpath=mbox_dir) 46 | 47 | # Fetch all commits as an iteratoir, and iterate it uploading to ElasticSearch 48 | print('Analyzing mbox archives...') 49 | # fetch all messages as an iteratoir 50 | for message in repo.fetch(): 51 | # Create the object (dictionary) to upload to ElasticSearch 52 | summary = { 53 | 'from': message['data']['From'], 54 | 'subject': message['data']['Subject'], 55 | 'date': email.utils.parsedate_to_datetime(message['data']['Date']) 56 | } 57 | print('.', end='') 58 | # Upload the object to ElasticSearch 59 | es.index(index='messages', doc_type='summary', body=summary) 60 | 61 | print('\nCreated new index with commits.') 62 | -------------------------------------------------------------------------------- /docs/python/scripts/perceval_elasticsearch_mbox_2.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2016 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | 24 | import elasticsearch 25 | 26 | # ElasticSearch instance (url) 27 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 28 | 29 | # Query messages index, getting all items in it 30 | es_result = es.search(index="messages") 31 | # Print number of messages retrieved 32 | print("Found %d messages" % es_result['hits']['total']) 33 | for message in es_result['hits']['hits']: 34 | print("Sender: %s\n Subject: %s" % 35 | (message['_source']['from'], message['_source']['subject'])) 36 | -------------------------------------------------------------------------------- /docs/python/scripts/perceval_elasticsearch_mbox_3.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2016 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | 24 | import elasticsearch 25 | 26 | # ElasticSearch instance (url) 27 | es = elasticsearch.Elasticsearch(['http://localhost:9200/']) 28 | 29 | # Query messages index, getting all items in it 30 | es_result = es.search(index="messages", doc_type='summary', 31 | body={"query": {"match": {"from": "Jim"}}}) 32 | # Print number of messages retrieved 33 | print("Found %d messages" % es_result['hits']['total']) 34 | for message in es_result['hits']['hits']: 35 | print("Sender: %s\n Subject: %s" % 36 | (message['_source']['from'], message['_source']['subject'])) 37 | -------------------------------------------------------------------------------- /docs/sortinghat/affiliations/add.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Add Orgs 4 | nav_order: 1 5 | parent: Affiliations in SortingHat 6 | grand_parent: SortingHat 7 | has_children: false 8 | has_toc: false 9 | --- 10 | 11 | # How to add organizations 12 | 13 | In order to add an organization in SortingHat, see the 14 | `Organizations` table on the rightmost side of the interface. 15 | 16 | 1. Click on the "Add" button to open the "Add organization" dialogue box. 17 | 2. Fill in the name of the organization (Bitergia) and the domains 18 | (www.bitergia.com). 19 | 3. Save it. 20 | 21 | ![add-org](../assets/add-org.png) -------------------------------------------------------------------------------- /docs/sortinghat/affiliations/assets/add-org.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/affiliations/assets/add-org.png -------------------------------------------------------------------------------- /docs/sortinghat/affiliations/assets/delete-org.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/affiliations/assets/delete-org.png -------------------------------------------------------------------------------- /docs/sortinghat/affiliations/assets/edit-org.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/affiliations/assets/edit-org.png -------------------------------------------------------------------------------- /docs/sortinghat/affiliations/assets/un-enroll.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/affiliations/assets/un-enroll.png -------------------------------------------------------------------------------- /docs/sortinghat/affiliations/edit.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Edit Orgs 4 | nav_order: 2 5 | parent: Affiliations in SortingHat 6 | grand_parent: SortingHat 7 | has_children: false 8 | has_toc: false 9 | --- 10 | 11 | # How to edit organizations 12 | 13 | Organization data consists of a name and domains. However only the domains are 14 | editable. You can either remove the exisiting domain or add new domains. In both 15 | cases, the process is pretty easy. Click on the edit icon to open up the Edit 16 | dialogue and press "Save" once done. 17 | 18 | ![edit-org](../assets/edit-org.png) -------------------------------------------------------------------------------- /docs/sortinghat/affiliations/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Affiliations in SortingHat 4 | nav_order: 3 5 | parent: SortingHat 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Affiliations in SortingHat 11 | 12 | SortingHat provides the ability the include the user's affiliation during the 13 | creation of the latter's profile. Details such as the `Date from`, `Date to` and 14 | `Organizaion` are included along with the profile. -------------------------------------------------------------------------------- /docs/sortinghat/affiliations/remove.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Remove Orgs 4 | nav_order: 3 5 | parent: Affiliations in SortingHat 6 | grand_parent: SortingHat 7 | has_children: false 8 | has_toc: false 9 | --- 10 | 11 | # How to delete organizations from registry 12 | 13 | To delete an organization, simple look over to the `Organizations` table and 14 | click on the Bin icon to delete the organization of your choice. 15 | 16 | ![delete-org](../assets/delete-org.png) -------------------------------------------------------------------------------- /docs/sortinghat/affiliations/un-enroll.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Un-enroll individual from org 4 | nav_order: 4 5 | parent: Affiliations in SortingHat 6 | grand_parent: SortingHat 7 | has_children: false 8 | has_toc: false 9 | --- 10 | 11 | # Un-enroll individual from org 12 | 13 | Simply go to an individual's profile and in the `Organizations` section, click 14 | on Bin icon to remove that organization from the profile. In the case, all of 15 | the affiliation need to be removed, click on the "Remove all" button in the 16 | Organizations section. 17 | 18 | ![un-enroll](../assets/un-enroll.png) -------------------------------------------------------------------------------- /docs/sortinghat/assets/sortinghat-no-identity.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/assets/sortinghat-no-identity.png -------------------------------------------------------------------------------- /docs/sortinghat/assets/sortinghat-org.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/assets/sortinghat-org.png -------------------------------------------------------------------------------- /docs/sortinghat/assets/sortinghat-sameIdentity.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/assets/sortinghat-sameIdentity.png -------------------------------------------------------------------------------- /docs/sortinghat/assets/sortinghat_individuals.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/assets/sortinghat_individuals.png -------------------------------------------------------------------------------- /docs/sortinghat/assets/sortinghat_orgs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/assets/sortinghat_orgs.png -------------------------------------------------------------------------------- /docs/sortinghat/assets/sortinghat_workspace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/assets/sortinghat_workspace.png -------------------------------------------------------------------------------- /docs/sortinghat/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: SortingHat 4 | permalink: /sortinghat 5 | nav_order: 6 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # SortingHat 11 | 12 | [SortingHat](https://github.com/chaoss/grimoirelab-sortinghat/tree/muggle) is a 13 | part of the GrimoireLab toolset and is used to gather data related to 14 | contributor information. Data about each individual is stored as a unique 15 | profile which may or may not include several identities belonging to the same 16 | contributor. Moreover each individual can be affiliated with a particular 17 | organisation where the latter was employed. 18 | 19 | SortingHat is currently being converted into a service so the documentation at 20 | the **[master](https://github.com/chaoss/grimoirelab-sortinghat)** branch would 21 | be totally inaccurate. The on-going development is carried out at the 22 | **[muggle](https://github.com/chaoss/grimoirelab-sortinghat/tree/muggle)** 23 | branch. -------------------------------------------------------------------------------- /docs/sortinghat/interface.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: SortingHat's interface 4 | nav_order: 1 5 | parent: SortingHat 6 | --- 7 | 8 | # SortingHat's new interface 9 | 10 | As stated before, SortingHat is currently being revamped into a service and as a 11 | result, its user-interface has been improved as well. 12 | 13 | - **Workspace** is for keeping track of an individual's profile on which some 14 | actions can be performed later on. This is in the case where you have numerous 15 | profiles but need to perform some action on a few of them. 16 | 17 | ![sortinghat workspace](./assets/sortinghat_workspace.png) 18 | 19 | - **Individuals** is, as the name suggests, the table where all the individual's 20 | profiles can be found. Information such as name, email, identities and 21 | affiliations can be found out from their respective profiles. 22 | 23 | ![sortinghat individuals](./assets/sortinghat_individuals.png) 24 | 25 | - **Organisation** is the list of affiliations where an individual can be 26 | employed at. 27 | 28 | ![sortinghat orgs](./assets/sortinghat_orgs.png) -------------------------------------------------------------------------------- /docs/sortinghat/profiles/add.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Creating profiles 4 | nav_order: 1 5 | parent: Profiles and Identities 6 | grand_parent: SortingHat 7 | has_children: false 8 | has_toc: false 9 | --- 10 | 11 | # How to create a profile 12 | 13 | Creating a profile is akin to adding an individual in SortingHat. 14 | 15 | To add an individual, click on the **Add** button in the `Individuals` table. 16 | This will open a dialogue titled `Add individual`. Fill in the individual's 17 | respective information and finally click on **Save** to save the information. 18 | 19 | ![create-profile](../assets/create-profile.png) 20 | 21 | In the **Add individual** dialogue, 22 | 23 | - `Name`: Name of the individual 24 | - `Email`: Email of the individual 25 | - `Username`: Username of individual with respect to the source 26 | - `Source`: Refers to the data source (github, Telegram, Dockerhub etc... 27 | - `Gender`: Gender of individual 28 | - `Country`: Country where individual resides 29 | - `Bot`: check it if individual is a bot, else leave it be 30 | - `Organization`: organization with which individual is affiliated 31 | - `Date from`: Date when individual started working for mentioned organization 32 | - `Date to`: Date when individual terminated his affiliation with mentioned 33 | organization 34 | 35 | ![save-profile](../assets/save-profile.png) 36 | 37 | Note: _Organization entered need to be registered in the Organization table 38 | before completing process to add individual. If organization is not present in 39 | organization's table, then individual's data will be saved, **excluding the 40 | organization's data**._ -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/add-to-workspace.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/add-to-workspace.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/batch-delete.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/batch-delete.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/create-profile.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/create-profile.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/drag-and-drop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/drag-and-drop.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/edit-profile.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/edit-profile.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/expand.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/expand.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/highlight-merge.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/highlight-merge.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/is-bot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/is-bot.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/lock-profile.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/lock-profile.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/order-profiles.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/order-profiles.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/profiles.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/profiles.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/remove-profile.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/remove-profile.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/save-profile.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/save-profile.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/set-bot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/set-bot.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/set-diff-bot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/set-diff-bot.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/un-merge.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/un-merge.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/assets/workspace-merge.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/sortinghat/profiles/assets/workspace-merge.png -------------------------------------------------------------------------------- /docs/sortinghat/profiles/bot.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Set profile as Bot 4 | nav_order: 6 5 | parent: Profiles and Identities 6 | grand_parent: SortingHat 7 | has_children: false 8 | has_toc: false 9 | --- 10 | 11 | # Set profile as Bot 12 | 13 | As we know, some identities can turn out to be bots. In order to remedy that, 14 | SortingHat provides the ability to mark profiles as Bot. It can be done either 15 | when creating a profile or after creating a profile. 16 | 17 | - When creating a profile, you'll have the option to mark a profile as Bot. 18 | Simply check that option and Save. 19 | 20 | ![setBot](../assets/set-bot.png) 21 | 22 | - After creating a profile, hover right next to the profile's name and a bunch 23 | of icons will appear. Click on the bot icon to mark it as bot. 24 | 25 | ![setBot2](../assets/set-diff-bot.png) -------------------------------------------------------------------------------- /docs/sortinghat/profiles/edit.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Editing profiles 4 | nav_order: 2 5 | parent: Profiles and Identities 6 | grand_parent: SortingHat 7 | has_children: false 8 | has_toc: false 9 | --- 10 | 11 | # How to edit profiles 12 | 13 | Once a profile is created, it is possible to edit some of its information. 14 | However some information is set during the profile creation phase and cannot be 15 | edited. 16 | 17 | A profile consists of two parts, **Identities** and **Organisations** data. 18 | 19 | - **Profile** information such as Name, Email, Gender and Country are editable. 20 | However **Source is not editable**. 21 | - **Identities** are **NOT editable** unless they are split up. Once split up, 22 | each identity is a standalone profile so the above described field would be 23 | editable. 24 | - **Organisation** data such as **Date from** and **Date to** are editable. 25 | Organisation data can also be removed as a whole. 26 | 27 | Tip: _If you hover on any field in profile and the edit icon appears, that means 28 | the filed is editable._ 29 | 30 | ![edit-profile](../assets/edit-profile.png) -------------------------------------------------------------------------------- /docs/sortinghat/profiles/filter.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Filter through profiles 4 | nav_order: 8 5 | parent: Profiles and Identities 6 | grand_parent: SortingHat 7 | has_children: false 8 | has_toc: false 9 | --- 10 | 11 | # Filter through profiles 12 | 13 | SortingHat provides several filters which can be used to filter through the 14 | lists of individuals to find the required one. You can filter through the 15 | profiles according to the following filters. 16 | 17 | - isBot 18 | 19 | Filter profiles marked as bots. For example `isBot: true` will return all profiles marked as bot and vice versa. 20 | 21 | ![is-bot](../assets/is-bot.png) 22 | 23 | - isLocked 24 | 25 | Filter profiles marked as locked. For example `isLocked: true` will return all profiles marked as locked and vice versa. 26 | 27 | - country 28 | 29 | Filter profiles according to country of residence. For example `country: "United States of America` or `country: USA` return individuals from the United States. 30 | 31 | - gender 32 | 33 | Filter profiles based on their gender. For example `gender: non binary` 34 | 35 | - source 36 | 37 | Filter profiles based on data source. For example `source: Github` 38 | 39 | - enrollment 40 | 41 | Filter profiles based on organizations. For example `enrollment: "Bitergia"` 42 | 43 | - isEnrolled 44 | 45 | Filter profiles based on enrollment status. For example `isEnrolled: true` will return all profiles currently enrolled at some organization and vice versa 46 | 47 | - enrollmentDate 48 | 49 | Filter profiles based on when they were affiliated to an organization. 50 | 51 | | Filter | Explanation | 52 | | -------------------------------------------------- | ---------------------------------------------------------------------------------------- | 53 | | `enrollmentDate:>YYYY-MM-DD` | Matches individuals that were affiliated to an organization after the given date. | 54 | | `enrollmentDate:>=YYYY-MM-DD` | Matches individuals that were affiliated to an organization on or after the given date. | 55 | | `enrollmentDate:lastUpdated 60 | 61 | Filter profiles based on when they were last updated. 62 | 63 | | Filter | Explanation | 64 | | ----------------------------------------------- | ----------------------------------------------------------------- | 65 | | `lastUpdated:>YYYY-MM-DD` | Matches individuals that were updated after the given date | 66 | | `lastUpdated:>=YYYY-MM-DD` | Matches individuals that were updated on or after the given date. | 67 | | `lastUpdated:split all of the identities or specifically pick the identity to be un-merged. 21 | 22 | ![un-merge](../assets/un-merge.png) -------------------------------------------------------------------------------- /docs/sortinghat/troubleshoot.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Troubleshoot 4 | nav_order: 4 5 | parent: SortingHat 6 | --- 7 | 8 | # Troubleshooting steps 9 | 10 | ### Profile creation issues 11 | 12 | - **Organisation not found** 13 | 14 | ![sortinghat-orgs](./assets/sortinghat-org.png) 15 | 16 | The above issue arises when the organisation submitted does not exist in the 17 | `Organisation` table. "Bitergia" here is taken as an example. In order to fix 18 | it, make sure to [add your organisation]({{ site.baseurl }}{% link 19 | docs/sortinghat/affiliations/add.md %}) first, then [create the individuals 20 | profile](). 21 | 22 | - **Profile already exist** 23 | 24 | ![sortinghat-sameIdentity](./assets/sortinghat-sameIdentity.png) 25 | 26 | The above issue arises if a existing profile has the exact same information 27 | (Name, Email, Username, Source etc...) as the profile being created. The ID 28 | `013d8db4d7fba708448c146c8fb54f8dcb974ed1` refers to the unique identifier of 29 | a profile. 30 | 31 | - **Fields cannot be empty** 32 | 33 | ![sortinghat-no-identity](./assets/sortinghat-no-identity.png) 34 | 35 | The above issue is pretty staightforward. In order to create a profile, a 36 | minimum amount of information is required which includes, primarily the 37 | `Source` and any other identity related info (`Name` or `Email` or 38 | `Username`). -------------------------------------------------------------------------------- /docs/terminology/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Terminology 4 | permalink: /terminology 5 | nav_order: 8 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | # Terminology used in GrimoireLab 11 | 12 | - **Dashboard** - refers to the collection of visualization under one name. For example, 13 | the Overview dashboard in CHAOSS contains visualization that give a general feel of the 14 | community and project health. 15 | 16 | - **Dashboards** - refers to all the dashboards present. You can access them through the 17 | dashboards option in the sidebar. It's the 3rd icon on the GrimoireLab sidebar. 18 | 19 | - **Visualization** - A visualization is a graphical view of the data pulled 20 | from an index. You can see all your visualization through the `Visualize` 21 | option in the sidebar. It's the 2nd icon on the GrimoireLab sidebar. 22 | 23 | - **Index** - An index is a collection of JSON documents related to one particular data 24 | source. 25 | 26 | - **Index-pattern** - An index pattern shows the different attributes contained in an 27 | index or a set of them. 28 | 29 | - **Data-Source** - GrimoireLab supports many data-sources. They represent the platforms and 30 | tools from where GrimoireLab can pull data to analyze. -------------------------------------------------------------------------------- /docs/tools-and-tips/attraction-and-retention.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Attraction & retention 4 | nav_order: 2 5 | parent: Tools and Tips 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | # Attraction & retention 12 | 13 | [ Warning: this is work in progress, use at your own risk! ] 14 | 15 | In this section we're going to use some Python scripting, including using Pandas data frames, for producing CSV files, ElasticSearch indexes and Kibana dashboards. We will do that to study how people join and leave projects. 16 | 17 | Some interesting data for learning about the evolution of a developing community is how people are entering and leaving it. From the point of view of the community, those are their attraction and retention rates: how they are attracting new developers, and how they are retaining (or not) active in the community over time. 18 | 19 | For doing this analysis with GrimoireLab, we will produce an index with one entry (document) per developer, annotated with the date for the first and the last activity. The first activity will show when they started to be active. The last activity will be an indicator of whether they are still active or not: if their last activity is, say, 6 months old, it is very likely that they are no longer active members of the community. 20 | 21 | The analysis can be done for each data source. To begin with, let's start with git. We will use a "standard" git index, such as those found in any standard GrimoireLab dashboards. So, this will be an execise of starting with some enriched GrimoireLab index, and producing a new one with specialized infomation. 22 | 23 | But even before producing an index, we're going to just produce some CSV files, with the same structure, so that we can first learn to do the analysis, and later how to manage with uploading data to ElasticSearch. In fact, the structure of the analysis is simple: 24 | 25 | * Get the data from ElasticSearch, using a query designed to produce data tailored for our analysis. 26 | * Store the data in a Pandas data frame. 27 | * Shape that data frame until we have the data we want. 28 | * Save the data frame in a CSV file (or later, in an ElasticSearch index). 29 | 30 | Scripts and JSON files mentioned here are available from the [tools&tips directory in the GitHub repo](https://github.com/jgbarah/GrimoireLab-training/tree/master/tools-and-tips/scripts). 31 | 32 | Use the `enriched_elasticsearch_git_pandas7.py` script as follows: 33 | 34 | ```bash 35 | ./enriched_elasticsearch_git_pandas7.py \ 36 | --es https://user:passwd@elasticsearch_url --es_index git \ 37 | --es_out https://user:passwd@elasticsearch_url --es_index_out git_demo \ 38 | --no_verify_certs 39 | ``` 40 | 41 | Then create in Kibana the index pattern `git_demo`, based on the index you just created, and upload to Kibana the following JSON files, which are the searches, visualizations and dashboards: 42 | 43 | * `C_dashboard.json` 44 | * `C_visualizations.json` 45 | * `C_searches.json` 46 | 47 | And point Kibana to load the dashboard `C_Git_Demo`. 48 | 49 | ## Final version of the script 50 | 51 | The final version of this script is [enriched_elasticsearch_newcomers.py](https://github.com/jgbarah/GrimoireLab-training/tree/master/tools-and-tips/scripts/enriched_elasticsearch_newcomers.py). 52 | -------------------------------------------------------------------------------- /docs/tools-and-tips/copying-indexes.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Copying indexes 4 | nav_order: 5 5 | parent: Tools and Tips 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | ## Copying indexes 12 | 13 | Python script to copy indexes: 14 | [`elastic_cp.py`](https://github.com/jgbarah/GrimoireLab-training/blob/master/tools-and-tips/scripts/elastic_cp.py). 15 | -------------------------------------------------------------------------------- /docs/tools-and-tips/csv-from-jenkins-enriched-index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Obtain CSV from Jenkins enriched indexes in ElasticSearch 4 | nav_order: 4 5 | parent: Tools and Tips 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | ## Obtain CSV from Jenkins enriched indexes in ElasticSearch 12 | 13 | To illustrate how to get data from an enriched index (produced using `grimoire_elk`), let's review the script [`enriched_elasticsearch_jenkins.py`](https://github.com/jgbarah/GrimoireLab-training/blob/master/tools-and-tips/scripts/enriched_elasticsearch_jenkins.py). This script will query a Jenkins enriched index, producing a CSV file with some of the fields in it, since a certain number of days. 14 | 15 | ### Installation 16 | 17 | To use it, we can create a new virtual environment for Python, and install the needed modules (including the script) in it. 18 | 19 | ```bash 20 | $ pyvenv ~/venv 21 | $ source ~/venv/bin/activate 22 | (venv) $ pip install elasticsearch 23 | (venv) $ pip install elasticsearch-dsl 24 | (venv) $ wget https://raw.githubusercontent.com/jgbarah/GrimoireLab-training/master/tools-and-tips/scripts/enriched_elasticsearch_jenkins.py 25 | (venv) $ python enriched_elasticsearch_jenkins.py --help 26 | ... 27 | ``` 28 | 29 | The last command should show the script help banner. 30 | 31 | ### Execution 32 | 33 | To use it, we will specify the url for the ElasticSearch instance we will use, the name of the Jenkins enriched index, and (optionally) a number of days (since when we want to get entries). If days are not specified, the script will produce entries for the last 90 days. Let's try it: 34 | 35 | ```bash 36 | (venv) $ python enriched_elasticsearch_jenkins.py --es elastic_url \ 37 | --es_index jenkins --days 2 > /tmp/jenkins.csv 38 | ``` 39 | 40 | The resulting file `/tmp/jenkins.csv` will be like: 41 | 42 | ``` 43 | job_build,build_date,duration(ms),result,builtOn,build,branch,loop,job_name 44 | ha-master/108,2017-02-23T06:42:30.575000+00:00,8800991,FAILURE,pod2,108,master,ha,ha-master 45 | ha-rel/107,2017-02-23T01:20:38.197000+00:00,6360746,FAILURE,pod2,107,rel,ha,ha-rel 46 | ... 47 | ``` 48 | 49 | The first line is the CSV header, and then we have one entry per line. Each entry shows several fields in the ElasticSearch document corresponding to a job build. 50 | 51 | ### Some details 52 | 53 | Let's have a look at some regions of the code of the script. 54 | 55 | ```python 56 | es = elasticsearch.Elasticsearch( 57 | [args.es], 58 | verify_certs=args.verify_certs 59 | ) 60 | ``` 61 | 62 | This will create an ElasticSearch object, which will be used to connect to ElasticSearch. As parameters, we pass a list of the urls to access the intended index (in our case, only one), and a parameter which will be true in case we don't want to verify SSL certificates (for instance, if they are not signed by the right UA, but we know they are reliable enough). 63 | 64 | ```python 65 | request = elasticsearch_dsl.Search(using=es, index=args.es_index) 66 | request = request.filter('range', 67 | build_date={'from': datetime.datetime.now() - datetime.timedelta(days=args.days)}) 68 | request = request.sort('-build_date') 69 | ``` 70 | 71 | The above code creates the request we will use, in several stages. First, we produce a Search request, specifying the index we want to query. Then, we filter it for a certain time period (since some days ago until now). Finally, we sort the results by build date, in descenting order. 72 | 73 | Once the request is complete, we'ready to execute it using the scan method, which can retrieve large collections of items from an index: 74 | 75 | ```python 76 | response = request.scan() 77 | print("job_build,build_date,duration(ms),result,builtOn,build,branch,loop,job_name") 78 | for job in response: 79 | print("{},{},{},{},{},{},{},{},{}".format( 80 | job['job_build'], job['build_date'], 81 | job['duration'], job['result'], 82 | job['builtOn'], job['build'], 83 | job['branch'], job['loop'], job['job_name'] 84 | )) 85 | ``` 86 | 87 | Once executed, we will just loop through the result, printing one line of the CSV file for each iteration. Before that, we will write the header of the CVS file, with names for all the fields to show. 88 | -------------------------------------------------------------------------------- /docs/tools-and-tips/elasticsearch.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: ElasticSearch REST API 4 | nav_order: 1 5 | parent: Tools and Tips 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | ## Elasticsearch REST API 12 | 13 | Some tips to work with Elasticsearch using its REST interface. 14 | 15 | In the examples below, let's assume that the url of the Elasticsearch instance is `https://elasticurl`. Remember that the full format for these urls, when they are protected by simple HTTP authentication (user and password), and are accessible through a non-root resource, is something like: 16 | 17 | ``` 18 | https://user:passwd@host:port/resource 19 | ``` 20 | 21 | ### Working with indexes 22 | 23 | To list all indexex stored by Elasticsearch: 24 | 25 | ```bash 26 | curl -XGET 'https://elasticurl/_cat/indices?v' 27 | ``` 28 | 29 | This returns for each index, its name, status (`open` comes to mean 'usable'), number of documents, deleted documents, and storage size used. 30 | 31 | ### Working with aliases 32 | 33 | Elasticsearch index aliases allow to work with a collection of indexes as if it were just a single index, or with a single index under a different name. In the following, we were refer as 'base indexes' to the real indexes for which we will create aliases. Find [more information about working with aliases in the Elasticsearch manual](https://www.elastic.co/guide/en/elasticsearch/guide/current/index-aliases.html). 34 | 35 | To list the base indexes corresponding to an index alias (assume the index alias is `alias_index`): 36 | 37 | ```bash 38 | curl -XGET 'https://elastic_url/alias_index/_alias/*' 39 | ``` 40 | 41 | The result will be similar to (being `base_index` the base index for the alias, and `alias_index`, `alias_index2` two aliases for that base index): 42 | 43 | ``` 44 | { 45 | "base_index" : { 46 | "aliases" : { 47 | "alias_index" : { }, 48 | "alias_index2" : { } 49 | } 50 | } 51 | } 52 | ``` 53 | 54 | To remove aliases, and create new ones, in an atomic operation: 55 | 56 | ```bash 57 | curl -XPOST 'https://elastic_url/_aliases' -d ' 58 | { 59 | "actions" : [ 60 | { "add" : { "index" : "base_index", "alias" : "alias_index" } }, 61 | { "add" : { "index" : "base_index2", "alias" : "alias_index" } }, 62 | { "remove" : { "index" : "old_base_index", "alias" : "alias_index" } }, 63 | { "remove" : { "index" : "old_base_index", "alias" : "alias_index" } } 64 | ] 65 | }' 66 | ``` 67 | -------------------------------------------------------------------------------- /docs/tools-and-tips/html5_app.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/tools-and-tips/html5_app.png -------------------------------------------------------------------------------- /docs/tools-and-tips/html5_app_moving.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/tools-and-tips/html5_app_moving.gif -------------------------------------------------------------------------------- /docs/tools-and-tips/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Tools and Tips 4 | permalink: /tools-tips 5 | nav_order: 12 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # Tools and tips 11 | 12 | This describes some useful tips, including the usage minor tools. 13 | 14 | Through this tutorial, we are using several systems that are not part 15 | of GrimoireLab. Find here some tips on how to profit from them. This 16 | content is probably useful beyond GrimoireLab, but in any case is a 17 | good context for it. 18 | -------------------------------------------------------------------------------- /docs/tools-and-tips/perceval.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Perceval Git Counter 4 | nav_order: 5 5 | parent: Tools and Tips 6 | has_children: false 7 | has_toc: false 8 | --- 9 | 10 | 11 | ## Perceval Git Counter 12 | 13 | This section shows some scripts using Perceval. 14 | 15 | ### Git commit counter. 16 | 17 | [perceval_git_counter](https://github.com/jgbarah/GrimoireLab-training/blob/master/tools-and-tips/scripts/perceval_git_counter.py) is a simple utility to count commits in a git repository. Just run it with the url of the repository to count, and a directory to clone, and you're done: 18 | 19 | ```bash 20 | python perceval_git_counter.py https://github.com/grimoirelab/perceval.git /tmp/ppp 21 | Number of commmits: 579. 22 | ``` 23 | 24 | You can get a help banner, including options, by running 25 | 26 | ```bash 27 | python perceval_git_counter.py --help 28 | ``` 29 | 30 | There is an option to print commit hashes for all commits in the repository: `--print`. 31 | 32 | This utility illusrates how the generator provided by Perceval classes for the different kinds of repositories can be used to run through all the items in them. 33 | -------------------------------------------------------------------------------- /docs/tools-and-tips/scripts/C_dashboard.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "_id": "C_Git_Demo", 4 | "_type": "dashboard", 5 | "_source": { 6 | "title": "C_Git_Demo", 7 | "hits": 0, 8 | "description": "", 9 | "panelsJSON": "[{\"col\":1,\"id\":\"C_Git_Demo_Table\",\"panelIndex\":1,\"row\":11,\"size_x\":8,\"size_y\":4,\"title\":\"New Authors\",\"type\":\"visualization\"},{\"col\":3,\"id\":\"C_Git_Demo_TS\",\"panelIndex\":2,\"row\":1,\"size_x\":6,\"size_y\":2,\"title\":\"# New Authors\",\"type\":\"visualization\"},{\"col\":1,\"id\":\"C_Git_Demo_Metrics\",\"panelIndex\":3,\"row\":1,\"size_x\":2,\"size_y\":4,\"title\":\"Summary\",\"type\":\"visualization\"},{\"col\":9,\"id\":\"C_Git_Demo_Repos_Table\",\"panelIndex\":4,\"row\":10,\"size_x\":4,\"size_y\":7,\"title\":\"Repos by New Authors\",\"type\":\"visualization\"},{\"col\":9,\"id\":\"C_Git_Demo_Orgs_Pie\",\"panelIndex\":5,\"row\":1,\"size_x\":4,\"size_y\":3,\"title\":\"New Authors per Organization\",\"type\":\"visualization\"},{\"col\":9,\"id\":\"C_Git_Demo_Repos_Pie\",\"panelIndex\":6,\"row\":7,\"size_x\":4,\"size_y\":3,\"title\":\"New Authors per First Repo\",\"type\":\"visualization\"},{\"col\":1,\"id\":\"C_Git_Demo_Org_TS\",\"panelIndex\":7,\"row\":5,\"size_x\":8,\"size_y\":3,\"title\":\"# New Authors by Top Organizations\",\"type\":\"visualization\"},{\"col\":1,\"id\":\"C_Git_Demo_Project_TS\",\"panelIndex\":8,\"row\":8,\"size_x\":8,\"size_y\":3,\"title\":\"# New Authors by Top First Projects\",\"type\":\"visualization\"},{\"col\":9,\"id\":\"C_Git_Demo_Projects_Pie\",\"panelIndex\":9,\"row\":4,\"size_x\":4,\"size_y\":3,\"title\":\"New Authors per First Project\",\"type\":\"visualization\"},{\"col\":3,\"id\":\"C_Git_Demo_Last_TS\",\"panelIndex\":10,\"row\":3,\"size_x\":6,\"size_y\":2,\"title\":\"# Authors by Last Commit\",\"type\":\"visualization\"},{\"col\":1,\"id\":\"C_Git_Demo_Last_Table\",\"panelIndex\":11,\"row\":15,\"size_x\":8,\"size_y\":4,\"title\":\"Authors With No Activity During Last 6 Months\",\"type\":\"visualization\"}]", 10 | "optionsJSON": "{\"darkTheme\":false}", 11 | "uiStateJSON": "{\"P-1\":{\"title\":\"New Authors\"},\"P-10\":{\"title\":\"# Authors by Last Commit\"},\"P-11\":{\"title\":\"Authors With No Activity During Last 6 Months\"},\"P-2\":{\"title\":\"# New Authors\"},\"P-3\":{\"title\":\"Summary\"},\"P-4\":{\"title\":\"Repos by New Authors\"},\"P-5\":{\"spy\":{\"mode\":{\"fill\":false,\"name\":null}},\"title\":\"New Authors per Organization\"},\"P-6\":{\"spy\":{\"mode\":{\"fill\":false,\"name\":null}},\"title\":\"New Authors per First Repo\"},\"P-7\":{\"title\":\"# New Authors by Top Organizations\"},\"P-8\":{\"title\":\"# New Authors by Top First Projects\"},\"P-9\":{\"title\":\"New Authors per First Project\"}}", 12 | "version": 1, 13 | "timeRestore": true, 14 | "timeTo": "now", 15 | "timeFrom": "now-15y", 16 | "kibanaSavedObjectMeta": { 17 | "searchSourceJSON": "{\"filter\":[{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}}}]}" 18 | } 19 | } 20 | } 21 | ] -------------------------------------------------------------------------------- /docs/tools-and-tips/scripts/C_searches.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "_id": "C_git_demo:last_commit-truncated", 4 | "_type": "search", 5 | "_source": { 6 | "title": "C_git_demo:last_commit-truncated", 7 | "description": "", 8 | "hits": 0, 9 | "columns": [ 10 | "_source" 11 | ], 12 | "sort": [ 13 | "first_commit", 14 | "desc" 15 | ], 16 | "version": 1, 17 | "kibanaSavedObjectMeta": { 18 | "searchSourceJSON": "{\"index\":\"git_demo\",\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}},\"filter\":[{\"$state\":{\"store\":\"appState\"},\"meta\":{\"alias\":null,\"disabled\":false,\"index\":\"git_demo\",\"key\":\"query\",\"negate\":false,\"value\":\"{\\\"range\\\":{\\\"last_commit\\\":{\\\"lt\\\":\\\"now-6M/M\\\"}}}\"},\"query\":{\"range\":{\"last_commit\":{\"lt\":\"now-6M/M\"}}}}],\"highlight\":{\"pre_tags\":[\"@kibana-highlighted-field@\"],\"post_tags\":[\"@/kibana-highlighted-field@\"],\"fields\":{\"*\":{}},\"require_field_match\":false,\"fragment_size\":2147483647}}" 19 | } 20 | } 21 | } 22 | ] -------------------------------------------------------------------------------- /docs/tools-and-tips/scripts/enriched_elasticsearch_jenkins.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | ## Copyright (C) 2017 Bitergia 5 | ## 6 | ## This program is free software; you can redistribute it and/or modify 7 | ## it under the terms of the GNU General Public License as published by 8 | ## the Free Software Foundation; either version 3 of the License, or 9 | ## (at your option) any later version. 10 | ## 11 | ## This program is distributed in the hope that it will be useful, 12 | ## but WITHOUT ANY WARRANTY; without even the implied warranty of 13 | ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 14 | ## GNU General Public License for more details. 15 | ## 16 | ## You should have received a copy of the GNU General Public License 17 | ## along with this program; if not, write to the Free Software 18 | ## Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. 19 | ## 20 | ## Authors: 21 | ## Jesus M. Gonzalez-Barahona 22 | ## 23 | 24 | import argparse 25 | import datetime 26 | import urllib3 27 | 28 | import elasticsearch 29 | import elasticsearch_dsl 30 | 31 | description = """Produce CSV from some fields in a Jenkins enriched index""" 32 | 33 | def parse_args (): 34 | 35 | parser = argparse.ArgumentParser(description = description) 36 | parser.add_argument("--es", type=str, required=True, 37 | help = "ElasticSearch instance") 38 | parser.add_argument("--es_index", type=str, required=True, 39 | help = "ElasticSearch index with Jenkins data") 40 | parser.add_argument("--days", type=int, required=False, 41 | default=90, 42 | help = "ElasticSearch index with Jenkins data") 43 | parser.add_argument("--verify_certs", dest="verify_certs", 44 | action="store_true", 45 | help = "Verify ssl certificates") 46 | parser.add_argument("--no_verify_certs", dest="verify_certs", 47 | action="store_false", 48 | help = "Do not verify ssl certificates") 49 | parser.set_defaults(verify_certs=True) 50 | args = parser.parse_args() 51 | return args 52 | 53 | def main(): 54 | # Disable warning about not verifying certificates 55 | urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) 56 | 57 | args = parse_args() 58 | 59 | es = elasticsearch.Elasticsearch([args.es], 60 | verify_certs=args.verify_certs) 61 | request = elasticsearch_dsl.Search(using=es, 62 | index=args.es_index) 63 | request = request.filter('range', 64 | build_date={'from': datetime.datetime.now() - datetime.timedelta(days=args.days)}) 65 | request = request.sort('-build_date') 66 | response = request.scan() 67 | 68 | print("job_build,build_date,duration(ms),result,builtOn,build,branch,loop,job_name") 69 | for job in response: 70 | print("{},{},{},{},{},{},{},{},{}".format( 71 | job['job_build'], job['build_date'], 72 | job['duration'], job['result'], 73 | job['builtOn'], job['build'], 74 | job['branch'], job['loop'], job['job_name'] 75 | )) 76 | 77 | if __name__ == "__main__": 78 | main() 79 | -------------------------------------------------------------------------------- /docs/tools-and-tips/scripts/events.css: -------------------------------------------------------------------------------- 1 | /* Inspired by css Zen Garden submission 175 - 2 | 'Business Style', by Gunta Klavina, http://www.klavina.com/ 3 | http://www.csszengarden.com/175/ 4 | */ 5 | 6 | body { 7 | margin: 0; 8 | padding: 0; 9 | color: #333; 10 | font: 12px Trebuchet MS, Lucida Sans Unicode, Arial, sans-serif; 11 | background: #E2EBED; 12 | text-align: center; 13 | } 14 | 15 | a { 16 | color: #307082; 17 | } 18 | 19 | a:hover { 20 | color: #D60808; 21 | } 22 | 23 | .page-wrapper { 24 | margin: 0 auto; 25 | width: 85%; 26 | position: relative; 27 | background: #fff; 28 | padding: 0 .5em 0 .5em; 29 | text-align: left; 30 | } 31 | 32 | header { 33 | position: absolute; 34 | top: 20px; 35 | left: 30px; 36 | } 37 | 38 | h1 39 | { 40 | height: 1.2em; 41 | margin: 0; 42 | font-size: 3em; 43 | white-space: nowrap; 44 | overflow: hidden; 45 | } 46 | 47 | h2 { 48 | font-size: 1.8em; 49 | margin: 0; 50 | } 51 | 52 | .summary { 53 | font-size: .9em; 54 | margin: 0; 55 | width: 250px; 56 | height: 1em; 57 | padding: 85px 90px 0 30px; 58 | font: 12px/150% Trebuchet MS; 59 | } 60 | 61 | .summary p { 62 | margin-top: 1em; 63 | width: 164px; 64 | /* height: 6em; */ 65 | padding: 1em 90px 1em 1em; 66 | font: 1.2em Trebuchet MS; 67 | } 68 | 69 | .summary p:first-child { 70 | background: grey; 71 | color: #fff; 72 | } 73 | .preamble { 74 | margin: 0 20px 0 243px; 75 | } 76 | 77 | .preamble p { 78 | line-height: 150%; 79 | margin: 10px 0 10px 0; 80 | } 81 | 82 | .preamble h3 { 83 | white-space: nowrap; 84 | overflow: hidden; 85 | width: 200px; 86 | height: 24px; 87 | margin: 40px 0 10px 0; 88 | } 89 | 90 | #events { 91 | width: 80%; 92 | height: 40em; 93 | margin: 1em auto; 94 | overflow: hidden; 95 | background: white; 96 | position: relative; 97 | box-sizing: border-box; 98 | } 99 | 100 | .event { 101 | background: grey; 102 | } 103 | 104 | .event p { 105 | margin-top: .2em; 106 | margin-bottom: .2em; 107 | } 108 | 109 | .head { 110 | background: white; 111 | padding-left: 1em; 112 | font-size: 1.2em; 113 | } 114 | .author, .message, .date { 115 | padding-left: 2em; 116 | } 117 | 118 | .author, .date { 119 | color: #fff; 120 | } 121 | 122 | .message { 123 | background: silver; 124 | padding-left: 3em; 125 | padding-bottom: 1em; 126 | } 127 | .marquee { 128 | margin: 0; 129 | padding: 0 1em; 130 | line-height: 1.4em; 131 | font: 1em 'Segoe UI', Tahoma, Helvetica, Sans-Serif; 132 | } 133 | 134 | .marquee { 135 | position: relative; 136 | box-sizing: border-box; 137 | } 138 | -------------------------------------------------------------------------------- /docs/tools-and-tips/scripts/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | Latest events 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 |
17 | 18 |
19 |

GrimoireLab Last Events Demo

20 |

Because knowledge empowers

21 |
22 | 23 |
24 |

A demonstration of how the latest activity of a project can be retrieved and shown.

25 |

Look in the GrimoireLab Training tutorial for details on how to produce something similar for your pet project(s).

26 |

Download the code for this demo. You'll need the file elastic_last.py to download commits from ElasticSearch indexes, and index.html, events.js and events.css for the webpage.

27 |
28 | 29 |
30 |
31 |
32 |
33 |
34 | 35 | 36 | -------------------------------------------------------------------------------- /docs/tools-and-tips/scripts/perceval_git_counter.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | # Count commits 3 | 4 | import argparse 5 | 6 | from perceval.backends.core.git import Git 7 | 8 | # Read command line arguments 9 | parser = argparse.ArgumentParser(description = "Count commits in a git repo") 10 | parser.add_argument("repo", help = "Repository url") 11 | parser.add_argument("dir", help = "Directory for cloning the repository") 12 | parser.add_argument("--print", action='store_true', help = "Print hashes") 13 | args = parser.parse_args() 14 | 15 | # create a Git object, and count commmits 16 | repo = Git(uri=args.repo, gitpath=args.dir) 17 | count = 0 18 | for commit in repo.fetch(): 19 | if args.print: 20 | print(commit['data']['commit']) 21 | count += 1 22 | print("Number of commmits: %d." % count) 23 | -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-all-complete.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-all-complete.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-all-dashboard-noarthur-nomordred-nopanels-nosh.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-all-dashboard-noarthur-nomordred-nopanels-nosh.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-all-dashboard-noarthur-nomordred-nopanels.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-all-dashboard-noarthur-nomordred-nopanels.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-all-dashboard-noarthur-nomordred.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-all-dashboard-noarthur-nomordred.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-all-dashboard-noarthur.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-all-dashboard-noarthur.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-all-dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-all-dashboard.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-all-grimoireelk-raw.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-all-grimoireelk-raw.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-all-perceval-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-all-perceval-1.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-all-perceval-n.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-all-perceval-n.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-all-reports.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-all-reports.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-fit-grimoireelk-enrich-sh.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-fit-grimoireelk-enrich-sh.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-fit-grimoireelk-enrich.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-fit-grimoireelk-enrich.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-fit-grimoireelk-raw.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-fit-grimoireelk-raw.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-fit-perceval-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-fit-perceval-1.png -------------------------------------------------------------------------------- /docs/workflow/assets/grimoirelab-fit-perceval-n.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/docs/workflow/assets/grimoirelab-fit-perceval-n.png -------------------------------------------------------------------------------- /docs/workflow/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Workflow 4 | permalink: /workflow 5 | nav_order: 2 6 | has_children: true 7 | has_toc: true 8 | --- 9 | 10 | # GrimoireLab Components 11 | 12 | The following figure summarizes the different tools used in GrimoreLab. 13 | 14 | ![](../../assets/grimoirelab-all-details.png) 15 | 16 | GrimoireLab is composed of different tools, which are listed below: 17 | 18 | ### Data retrieval related components: 19 | - [Perceval](https://github.com/chaoss/grimoirelab-perceval): retrieval of 20 | data from data sources 21 | - [Graal](https://github.com/chaoss/grimoirelab-graal): source data analysis 22 | with external tools 23 | - [KingArthur](https://github.com/chaoss/grimoirelab-kingarthur): batch 24 | processing for massive retrieval 25 | 26 | ### Data enrichment related components: 27 | - [GrimoireELK](https://github.com/chaoss/grimoirelab-elk): storage and 28 | enrichment of data 29 | - [Cereslib](https://github.com/chaoss/grimoirelab-cereslib): generic data 30 | processor 31 | - [SortingHat](https://github.com/chaoss/grimoirelab-sortinghat): identity 32 | management 33 | 34 | ### Data consumption related components: 35 | - [Kibiter](https://github.com/chaoss/grimoirelab-kibiter): dashboard, 36 | downstream version of Kibana 37 | - [Sigils](https://github.com/chaoss/grimoirelab-sigils): visualizations and 38 | dashboards 39 | - [Kidash](https://github.com/chaoss/grimoirelab-kidash): visualizations and 40 | dashboards manager 41 | - [Manuscripts](https://github.com/chaoss/grimoirelab-manuscripts): reporting 42 | 43 | ### Platform management, orchestration, and common utils: 44 | - [Mordred](https://github.com/chaoss/grimoirelab-mordred): orchestration 45 | - [GrimoireLab Toolkit](https://github.com/chaoss/grimoirelab-toolkit): common 46 | utilities 47 | - [Bestiary](https://github.com/chaoss/grimoirelab-bestiary): web-based user 48 | interface to manage repositories and projects for Mordred 49 | - [Hatstall](https://github.com/chaoss/grimoirelab-hatstall): web-based user 50 | interface to manage SortingHat identities 51 | -------------------------------------------------------------------------------- /docs/workflow/workflow.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: default 3 | title: Workflow 4 | nav_order: 1 5 | parent: Workflow 6 | --- 7 | 8 | # Workflow 9 | 10 | GrimoireLab is a platform for making it easy to apply analytics on data related 11 | to software development. It provides components for: 12 | 13 | * Automatic and incremental data retrieval from software development support 14 | systems (data sources). Those include source code management, issue tracking, 15 | communication, testing, code review systems, among others. 16 | * Automatic organization and enrichment of retrieved data, so that it can be 17 | analyzed more easily. 18 | * Management of different identities for the same person, and other data related 19 | to persons, such as affiliation to companies. 20 | * Visualization of the data in actionable dashboards. 21 | * Production of reports about specific aspects of software development 22 | processes. 23 | 24 | Most of the components in GrimoireLab are written in Python, and provide Python 25 | APIs and standalone programs. 26 | 27 | The main components of GrimoireLab are: 28 | 29 | * **Perceval**. Includes components for retrieving data from data sources. All 30 | components provide an uniform API for accessing their data sources, based on 31 | Python generators and data dictionaries (JSON documents). [Learn more about 32 | Perceval](https://perceval.readthedocs.io). 33 | * **Graal**. leverages on the Git backend of Perceval and enhances it to set up 34 | ad-hoc source code analysis. It fetches the commits from a Git repository and 35 | provides a mechanism to plug third-party tools/libraries focused on source 36 | code analysis. [Learn more about 37 | Graal](/grimoirelab-tutorial/graal/intro.html). 38 | * **KingArthur**. Organizes the data retrieval from large collections of 39 | repositories of different kinds, providing a uniform API to all of them, and 40 | to the different retrieval jobs running in any moment. [Learn more about 41 | KingArthur](https://github.com/chaoss/grimoirelab-kingarthur). 42 | * **GrimoireELK**. Collects data retrieved by Arthur, or directly by Perceval, 43 | and stores it in ElasticSearch indexes (raw indexes). It also organizes and 44 | transforms those raw indexes into other ElasticSearch indexes (enriched 45 | indexes), more suitable for visualization or reporting. [Learn more about 46 | GrimoireELK](/grimoirelab-tutorial/gelk/intro.html). 47 | * **SortingHat**. Manages identities for persons, including capabilities for 48 | merging the many identities corresponding to the same person, and other data 49 | related to them (such as affiliation to companies). [Learn more about 50 | SortingHat](https://sortinghat.readthedocs.io). 51 | * **Kibiter**. Visualization platform. It is a soft fork of 52 | [Kibana](https://www.elastic.co/products/kibana), which allows for the 53 | construction of actionable dashboards to show the data in enriched indexes. 54 | * **Sigils**. Definition of Kibiter dashboards. They encode the panels, 55 | visualizations and other details needed to produce dashboards in Kibiter or 56 | Kibana. [Learn more about 57 | Sigils](https://chaoss.github.io/grimoirelab-sigils/). 58 | * **Manuscripts**. Production of PDF documents reporting on the main 59 | characteristics of a project. [Learn more about 60 | Manuscripts](https://github.com/chaoss/grimoirelab-manuscripts). 61 | * **Mordred**. Manages the configuration of all the components needed to produce 62 | a dashboard, automating its production. [Learn more about 63 | Mordred](https://github.com/chaoss/grimoirelab-sirmordred). 64 | 65 | ![grimoirelab-all-complete.png](./assets/grimoirelab-all-complete.png) 66 | 67 | In the figure above, GrimoireLab components are represented in the pale green 68 | box. Bold arrows show the main data flow: from data sources to Perceval (which 69 | retrieves them), to Arthur (which schedules retrieval batches and stores results 70 | in Redis), to GrimoireELK (which stores retrieved items as raw indexes, and then 71 | uses them to produce enriched indexes, both in ElasticSearch), to Reports (to 72 | produce specialized reports) or Kibiter (to visualize in actionable dashboards). 73 | 74 | GrimoireELK uses SortingHat to store all the identities it finds in a MariaDB 75 | database. SortingHat uses lists of known identifiers (usually maintained in 76 | configuration files) and heuristics to merge identities corresponding to the 77 | same person, and related information (such as affiliation). 78 | 79 | All the process is configured and orchestrated by Mordred, which uses its own 80 | configuration about, for example, which data sources to use. 81 | -------------------------------------------------------------------------------- /favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chaoss/grimoirelab-tutorial/85bb9ad8da1c069700e62b83dff86b8ec6b5ba02/favicon.ico --------------------------------------------------------------------------------