├── .github └── ISSUE_TEMPLATE.md ├── .gitignore ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── docs ├── Makefile ├── _config.yml ├── conf.py ├── environment-doc.yml ├── examples │ ├── open-issues-prs-by-repo.ipynb │ ├── pandas-play.ipynb │ ├── report-repo-last-n-issues.ipynb │ └── test-request.ipynb ├── getting-started.md ├── graphql-notes.md ├── index.rst ├── intro.md ├── make.bat └── requirements-doc.txt ├── environment.yml ├── notebooks ├── open-issues-prs-by-repo.ipynb ├── pandas-play.ipynb ├── report-repo-last-n-issues.ipynb ├── traverse-json.ipynb ├── use-pyquery-ql-Copy1.ipynb ├── use-pyquery-ql-simplewalk.ipynb └── use-pyquery-ql.ipynb ├── nteract-open-issues-pr-counts-by-repo.ipynb ├── nteract-repos.ipynb ├── pyquery-ql ├── _version.py └── pyquery-ql.py ├── readthedocs.yml ├── requirements.txt └── test └── test-data ├── data.json ├── issue_data.csv └── mydata.csv /.github/ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | # Welcome 2 | 3 | Please leave your question or issue below. Providing as much detail as possible will be helpful too. 4 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | 57 | # Flask stuff: 58 | instance/ 59 | .webassets-cache 60 | 61 | # Scrapy stuff: 62 | .scrapy 63 | 64 | # Sphinx documentation 65 | docs/_build/ 66 | 67 | # PyBuilder 68 | target/ 69 | 70 | # Jupyter Notebook 71 | .ipynb_checkpoints 72 | 73 | # pyenv 74 | .python-version 75 | 76 | # celery beat schedule file 77 | celerybeat-schedule 78 | 79 | # SageMath parsed files 80 | *.sage.py 81 | 82 | # dotenv 83 | .env 84 | 85 | # virtualenv 86 | .venv 87 | venv/ 88 | ENV/ 89 | 90 | # Spyder project settings 91 | .spyderproject 92 | .spyproject 93 | 94 | # Rope project settings 95 | .ropeproject 96 | 97 | # mkdocs documentation 98 | /site 99 | 100 | # mypy 101 | .mypy_cache/ 102 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. 6 | 7 | ## Our Standards 8 | 9 | Examples of behavior that contributes to creating a positive environment include: 10 | 11 | * Using welcoming and inclusive language 12 | * Being respectful of differing viewpoints and experiences 13 | * Gracefully accepting constructive criticism 14 | * Focusing on what is best for the community 15 | * Showing empathy towards other community members 16 | 17 | Examples of unacceptable behavior by participants include: 18 | 19 | * The use of sexualized language or imagery and unwelcome sexual attention or advances 20 | * Trolling, insulting/derogatory comments, and personal or political attacks 21 | * Public or private harassment 22 | * Publishing others' private information, such as a physical or electronic address, without explicit permission 23 | * Other conduct which could reasonably be considered inappropriate in a professional setting 24 | 25 | ## Our Responsibilities 26 | 27 | Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. 28 | 29 | Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. 30 | 31 | ## Scope 32 | 33 | This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. 34 | 35 | ## Enforcement 36 | 37 | Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at GitHub issues. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. 38 | 39 | Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. 40 | 41 | ## Attribution 42 | 43 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version] 44 | 45 | [homepage]: http://contributor-covenant.org 46 | [version]: http://contributor-covenant.org/version/1/4/ 47 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | Welcome. 4 | 5 | We welcome contributors who abide by the Code of Conduct of this project. 6 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Carol Willing 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # pyquery-ql 2 | 3 | [![Documentation Status](http://readthedocs.org/projects/pyquery-ql/badge/?version=latest)](http://pyquery-ql.readthedocs.io/en/latest/?badge=latest) 4 | 5 | Query [GitHub API v4](https://developer.github.com/v4/) using a lightweight 6 | [GraphQL](http://graphql.org) client 7 | 8 | ## What is pyquery-ql? 9 | 10 | It's a repository for examples and tips when querying GraphQL APIs using 11 | [Python](https://python.org) and the popular [Jupyter](https://jupyter.org) 12 | notebook. It provides examples on using the 13 | elegant [requests](https://requests.org) library and *pandas* to do data 14 | collection and interactive 15 | data exploration. 16 | 17 | **pyquery-ql** is also a very early prototype for a lightweight Python 18 | GraphQL client. It's based on Python 3.6 or higher and takes advantage of 19 | f-strings (if you haven't tried them yet, please do). 20 | 21 | ## Why create a lightweight GraphQL client for Python? 22 | 23 | In data science and scientific programming, we do lots of queries for data 24 | from different sources. With the increased use of GraphQL, it's helpful to 25 | have a lightweight client to do quick queries. 26 | 27 | With [Jupyter](https://jupyter.org) and [nteract](https://nteract.io) 28 | notebooks, it's critical to one's workflow to be 29 | able to prototype and iterate from within the notebook. While full featured 30 | clients, like [Apollo Client](https://www.apollographql.com/client/), and 31 | explorers, such as [GitHub's GraphQL Explorer](https://developer.github.com/v4/explorer/), 32 | are wonderful, the user must leave the notebook to see their benefits. 33 | 34 | Using the Jupyter or nteract notebooks, a user explores data, and GraphQL is 35 | fun to use since it's very easy to introspect the data source. A lightweight 36 | Python client provides a simple way to bridge the data and 37 | the Python scientific stack, especially pandas, from within the interactive 38 | notebook. 39 | 40 | ## Installation 41 | 42 | Supports Python 3.6+ 43 | 44 | 1. Clone the repo. 45 | 46 | git clone https://github.com/willingc/pyquery-ql.git 47 | 48 | 2. Set up your GitHub API token to avoid rate limit and export to your 49 | environment: 50 | 51 | export GITHUB_API_TOKEN = '......' 52 | 53 | ### pip installation instructions 54 | 55 | 3. From the root of the cloned repo: 56 | 57 | pip install -r requirements.txt 58 | 59 | ### conda installation instructions 60 | 61 | 3. Create the conda environment: 62 | 63 | conda env create -f environment.yml 64 | 65 | 4. Activate the environment: 66 | 67 | source activate pyquery 68 | 69 | ## Usage 70 | 71 | To run: 72 | 73 | python3 pyquery-ql.py 74 | 75 | ## License 76 | 77 | MIT 78 | 79 | --- 80 | 81 | ## GraphQL info 82 | 83 | - [GraphQL](http://graphql.org) 84 | - describe your data 85 | - ask for what you want 86 | - get predictable results 87 | 88 | - A [GraphQL service](http://graphql.org/learn/queries/) defines **types** 89 | and **fields** on those types. A function is created for each field on 90 | each type. 91 | 92 | - **field** can be a specific type (like String) or an Object 93 | - **arguments** can be passed to fields when making a query 94 | ``` 95 | { 96 | human(id: "1000") { 97 | name 98 | height 99 | } 100 | } 101 | 102 | - GraphQL schema and types 103 | ``` 104 | type Character { 105 | name: String! 106 | appearsIn: [Episode]! 107 | } 108 | ``` 109 | 110 | - GraphQL Object type `Character` 111 | - fields `name` and `appearsIn` 112 | - Scalar type `String` 113 | - `!` non-nullable (will always return a value when queried) 114 | - array `[Episode]` 115 | 116 | - Arguments 117 | 118 | - every field can have 0 or more arguments 119 | - all arguments are named 120 | - all arguments are passed by name specifically (unlike JS or Python 121 | where functions take a list of ordered arguments 122 | - required or optional 123 | 124 | - Scalar types 125 | 126 | - fields that resolve to real data 127 | - leaves of the query 128 | - default scalar types: 129 | - `Int` 130 | - `Float` 131 | - `String` 132 | - `Boolean` 133 | - `ID` unique identifier 134 | - custom scalar types are possible i.e. `Date` 135 | - `Enums` 136 | 137 | - Modifiers: Non-Null and List (array) 138 | - List [] can be null 139 | - a member of the list can not be null or it will generate an error 140 | - []! the actual list can not be null though a member can be null 141 | 142 | - [**Introspection**](http://graphql.org/learn/introspection/) is a beautiful 143 | thing; ask for `__schema` for all the good stuff on the types 144 | 145 | ## GraphQL Client - deeper dive 146 | 147 | - [Pagination](http://graphql.org/learn/introspection/) 148 | - plurals 149 | - slicing 150 | - pagination and edges 151 | - Connections 152 | - [Caching](http://graphql.org/learn/caching/) 153 | - ID 154 | - pros and cons of globally unique IDs 155 | - server defined or client derived are both options 156 | 157 | ## GraphQL spec 158 | 159 | - [The Spec](http://facebook.github.io/graphql/October2016/) 160 | - [Response Format](http://facebook.github.io/graphql/October2016/#sec-Response-Format) 161 | - [Grammar Summary](http://facebook.github.io/graphql/October2016/#sec-Appendix-Grammar-Summary) -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = python -msphinx 7 | SPHINXPROJ = pyquery-ql 8 | SOURCEDIR = . 9 | BUILDDIR = _build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) -------------------------------------------------------------------------------- /docs/_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-minimal -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | # 4 | # pyquery-ql documentation build configuration file, created by 5 | # sphinx-quickstart on Thu Dec 14 13:38:04 2017. 6 | # 7 | # This file is execfile()d with the current directory set to its 8 | # containing dir. 9 | # 10 | # Note that not all possible configuration values are present in this 11 | # autogenerated file. 12 | # 13 | # All configuration values have a default; values that are commented out 14 | # serve to show the default. 15 | 16 | # If extensions (or modules to document with autodoc) are in another directory, 17 | # add these directories to sys.path here. If the directory is relative to the 18 | # documentation root, use os.path.abspath to make it absolute, like shown here. 19 | # 20 | import os 21 | # import sys 22 | # sys.path.insert(0, os.path.abspath('.')) 23 | 24 | # For conversion from markdown to html 25 | import recommonmark 26 | 27 | on_rtd = os.environ.get('READTHEDOCS', None) == 'True' 28 | if not on_rtd: 29 | import sphinx_rtd_theme 30 | 31 | # -- General configuration ------------------------------------------------ 32 | 33 | # If your documentation needs a minimal Sphinx version, state it here. 34 | # 35 | # needs_sphinx = '1.0' 36 | 37 | # Add any Sphinx extension module names here, as strings. They can be 38 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 39 | # ones. 40 | extensions = ['sphinx.ext.intersphinx', 41 | 'sphinx.ext.mathjax', 42 | 'nbsphinx'] 43 | 44 | # Add any paths that contain templates here, relative to this directory. 45 | templates_path = ['_templates'] 46 | 47 | # support markdown parser 48 | source_parsers = { 49 | '.md': 'recommonmark.parser.CommonMarkParser', 50 | } 51 | 52 | # The suffix(es) of source filenames. 53 | # You can specify multiple suffix as a list of string: 54 | # 55 | # source_suffix = ['.rst', '.md'] 56 | source_suffix = ['.rst', '.md', '.ipynb'] 57 | 58 | # The master toctree document. 59 | master_doc = 'index' 60 | 61 | # General information about the project. 62 | project = 'pyquery-ql' 63 | copyright = '2017, Carol Willing' 64 | author = 'Carol Willing' 65 | 66 | # The version info for the project you're documenting, acts as replacement for 67 | # |version| and |release|, also used in various other places throughout the 68 | # built documents. 69 | # 70 | # The short X.Y version. 71 | version = '0.1' 72 | # The full version, including alpha/beta/rc tags. 73 | release = '0.1' 74 | 75 | # The language for content autogenerated by Sphinx. Refer to documentation 76 | # for a list of supported languages. 77 | # 78 | # This is also used if you do content translation via gettext catalogs. 79 | # Usually you set "language" from the command line for these cases. 80 | language = None 81 | 82 | # List of patterns, relative to source directory, that match files and 83 | # directories to ignore when looking for source files. 84 | # This patterns also effect to html_static_path and html_extra_path 85 | exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] 86 | 87 | # The name of the Pygments (syntax highlighting) style to use. 88 | pygments_style = 'sphinx' 89 | 90 | # If true, `todo` and `todoList` produce output, else they produce nothing. 91 | todo_include_todos = False 92 | 93 | 94 | # -- Options for HTML output ---------------------------------------------- 95 | 96 | # The theme to use for HTML and HTML Help pages. See the documentation for 97 | # a list of builtin themes. 98 | # 99 | html_theme = 'sphinx_rtd_theme' 100 | 101 | # Theme options are theme-specific and customize the look and feel of a theme 102 | # further. For a list of options available for each theme, see the 103 | # documentation. 104 | # 105 | html_theme_options = { 106 | 'collapse_navigation': False, 107 | 'navigation_depth': 2, 108 | 'display_version': True, 109 | 'prev_next_buttons_location': 'both', 110 | } 111 | 112 | # Add any paths that contain custom static files (such as style sheets) here, 113 | # relative to this directory. They are copied after the builtin static files, 114 | # so a file named "default.css" will overwrite the builtin "default.css". 115 | html_static_path = ['_static'] 116 | 117 | # Custom sidebar templates, must be a dictionary that maps document names 118 | # to template names. 119 | # 120 | # This is required for the alabaster theme 121 | # refs: http://alabaster.readthedocs.io/en/latest/installation.html#sidebars 122 | html_sidebars = { 123 | '**': [ 124 | 'about.html', 125 | 'navigation.html', 126 | 'relations.html', # needs 'show_related': True theme option to display 127 | 'searchbox.html', 128 | 'donate.html', 129 | ] 130 | } 131 | 132 | 133 | # -- Options for HTMLHelp output ------------------------------------------ 134 | 135 | # Output file base name for HTML help builder. 136 | htmlhelp_basename = 'pyquery-qldoc' 137 | 138 | 139 | # -- Options for LaTeX output --------------------------------------------- 140 | 141 | latex_elements = { 142 | # The paper size ('letterpaper' or 'a4paper'). 143 | # 144 | # 'papersize': 'letterpaper', 145 | 146 | # The font size ('10pt', '11pt' or '12pt'). 147 | # 148 | # 'pointsize': '10pt', 149 | 150 | # Additional stuff for the LaTeX preamble. 151 | # 152 | # 'preamble': '', 153 | 154 | # Latex figure (float) alignment 155 | # 156 | # 'figure_align': 'htbp', 157 | } 158 | 159 | # Grouping the document tree into LaTeX files. List of tuples 160 | # (source start file, target name, title, 161 | # author, documentclass [howto, manual, or own class]). 162 | latex_documents = [ 163 | (master_doc, 'pyquery-ql.tex', 'pyquery-ql Documentation', 164 | 'Carol Willing', 'manual'), 165 | ] 166 | 167 | 168 | # -- Options for manual page output --------------------------------------- 169 | 170 | # One entry per manual page. List of tuples 171 | # (source start file, name, description, authors, manual section). 172 | man_pages = [ 173 | (master_doc, 'pyquery-ql', 'pyquery-ql Documentation', 174 | [author], 1) 175 | ] 176 | 177 | 178 | # -- Options for Texinfo output ------------------------------------------- 179 | 180 | # Grouping the document tree into Texinfo files. List of tuples 181 | # (source start file, target name, title, author, 182 | # dir menu entry, description, category) 183 | texinfo_documents = [ 184 | (master_doc, 'pyquery-ql', 'pyquery-ql Documentation', 185 | author, 'pyquery-ql', 'One line description of project.', 186 | 'Miscellaneous'), 187 | ] 188 | 189 | # Example configuration for intersphinx: refer to the Python standard library. 190 | intersphinx_mapping = {'https://docs.python.org/3/': None} 191 | -------------------------------------------------------------------------------- /docs/environment-doc.yml: -------------------------------------------------------------------------------- 1 | name: pyquery 2 | channels: 3 | - conda-forge 4 | - defaults 5 | dependencies: 6 | - python=3.6 7 | - jupyter 8 | - pandas 9 | - requests 10 | - Sphinx>=1.6 11 | - sphinx_rtd_theme 12 | - pip: 13 | - recommonmark==0.4 14 | - nbsphinx 15 | -------------------------------------------------------------------------------- /docs/examples/report-repo-last-n-issues.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Repo report: last `n` issues" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "import csv\n", 17 | "import json\n", 18 | "import os\n", 19 | "import pprint\n", 20 | "\n", 21 | "import pandas as pd\n", 22 | "import requests" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 2, 28 | "metadata": {}, 29 | "outputs": [], 30 | "source": [ 31 | "# get api token and set authorization\n", 32 | "api_token = os.environ['GITHUB_API_TOKEN']\n", 33 | "headers = {'Authorization': f'token {api_token}'}" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": 3, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "# set url to a graphql endpoint\n", 43 | "url = 'https://api.github.com/graphql'" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 4, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "# add a json query\n", 53 | "query = \"\"\"\n", 54 | "{\n", 55 | " organization(login: \"jupyterhub\") {\n", 56 | " name\n", 57 | " repository(name: \"binderhub\") {\n", 58 | " name\n", 59 | " issues(last: 5) {\n", 60 | " edges {\n", 61 | " node {\n", 62 | " number\n", 63 | " title\n", 64 | " }\n", 65 | " }\n", 66 | " } \n", 67 | " }\n", 68 | " }\n", 69 | "}\n", 70 | "\"\"\"" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 5, 76 | "metadata": {}, 77 | "outputs": [], 78 | "source": [ 79 | "# submit the request\n", 80 | "r = requests.post(url=url, json={'query': query}, headers=headers)" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 6, 86 | "metadata": {}, 87 | "outputs": [ 88 | { 89 | "data": { 90 | "text/plain": [ 91 | "{'data': {'organization': {'name': 'JupyterHub',\n", 92 | " 'repository': {'issues': {'edges': [{'node': {'number': 344,\n", 93 | " 'title': 'Improve CONTRIBUTING.md'}},\n", 94 | " {'node': {'number': 345,\n", 95 | " 'title': 'Upgrade to kubernetes client version 4.0'}},\n", 96 | " {'node': {'number': 349,\n", 97 | " 'title': 'race condition mounting docker-in-docker socket in build'}},\n", 98 | " {'node': {'number': 350,\n", 99 | " 'title': 'Consider *not* launching automatically'}},\n", 100 | " {'node': {'number': 351, 'title': 'Awesome bar functionality'}}]},\n", 101 | " 'name': 'binderhub'}}}}" 102 | ] 103 | }, 104 | "execution_count": 6, 105 | "metadata": {}, 106 | "output_type": "execute_result" 107 | } 108 | ], 109 | "source": [ 110 | "binderhub = r.json()\n", 111 | "binderhub" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 7, 117 | "metadata": {}, 118 | "outputs": [ 119 | { 120 | "data": { 121 | "text/plain": [ 122 | "[{'node': {'number': 344, 'title': 'Improve CONTRIBUTING.md'}},\n", 123 | " {'node': {'number': 345,\n", 124 | " 'title': 'Upgrade to kubernetes client version 4.0'}},\n", 125 | " {'node': {'number': 349,\n", 126 | " 'title': 'race condition mounting docker-in-docker socket in build'}},\n", 127 | " {'node': {'number': 350, 'title': 'Consider *not* launching automatically'}},\n", 128 | " {'node': {'number': 351, 'title': 'Awesome bar functionality'}}]" 129 | ] 130 | }, 131 | "execution_count": 7, 132 | "metadata": {}, 133 | "output_type": "execute_result" 134 | } 135 | ], 136 | "source": [ 137 | "issue_list = binderhub['data']['organization']['repository']['issues']['edges']\n", 138 | "issue_list" 139 | ] 140 | }, 141 | { 142 | "cell_type": "markdown", 143 | "metadata": {}, 144 | "source": [ 145 | "## Report: Last 5 issues for a repo" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 8, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "name": "stdout", 155 | "output_type": "stream", 156 | "text": [ 157 | "binderhub: last five issues submitted\n", 158 | " 344 Improve CONTRIBUTING.md \n", 159 | " 345 Upgrade to kubernetes client version 4.0\n", 160 | " 349 race condition mounting docker-in-docker socket in build\n", 161 | " 350 Consider *not* launching automatically\n", 162 | " 351 Awesome bar functionality \n" 163 | ] 164 | } 165 | ], 166 | "source": [ 167 | "print(f\"{binderhub['data']['organization']['repository']['name']}: last five issues submitted\")\n", 168 | "for issue in issue_list:\n", 169 | " print(f\"{issue['node']['number']:6} {issue['node']['title']:30}\")" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": {}, 175 | "source": [ 176 | "Let's try another repo\n", 177 | "- make it a little more generic using `response` as a variable\n", 178 | "- select last 10 issues " 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 9, 184 | "metadata": {}, 185 | "outputs": [], 186 | "source": [ 187 | "# add a json query\n", 188 | "query = \"\"\"\n", 189 | "{\n", 190 | " organization(login: \"jupyterhub\") {\n", 191 | " name\n", 192 | " repository(name: \"jupyterhub\") {\n", 193 | " name\n", 194 | " issues(last: 10) {\n", 195 | " edges {\n", 196 | " node {\n", 197 | " number\n", 198 | " title\n", 199 | " }\n", 200 | " }\n", 201 | " } \n", 202 | " }\n", 203 | " }\n", 204 | "}\n", 205 | "\"\"\"\n", 206 | "r = requests.post(url=url, json={'query': query}, headers=headers)\n", 207 | "response = r.json()\n", 208 | "issue_list = response['data']['organization']['repository']['issues']['edges']" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 10, 214 | "metadata": {}, 215 | "outputs": [ 216 | { 217 | "name": "stdout", 218 | "output_type": "stream", 219 | "text": [ 220 | "jupyterhub: last 10 issues submitted\n", 221 | " 1573 more activity tracking \n", 222 | " 1574 Jupyter cannot find Three.Js files - URL redirection not working *updated*\n", 223 | " 1576 anchor links on markdown docs pages are unreliable\n", 224 | " 1579 LocalAuthenticator question \n", 225 | " 1580 Specify Kubernetes volume claims in Jupyterhub spawner\n", 226 | " 1582 Jupyterhub shared service error\n", 227 | " 1583 c.JupyterHub.logo_file is not working on after Jupyterhub Version: 0.8.1 upgrade\n", 228 | " 1584 config c.LocalProcessSpawner.mem_limit Issue\n", 229 | " 1585 Prometheus metrics to add to JupyterHub\n", 230 | " 1586 Errno 8: Nodename nor servname provided\n" 231 | ] 232 | } 233 | ], 234 | "source": [ 235 | "print(f\"{response['data']['organization']['repository']['name']}: last 10 issues submitted\")\n", 236 | "for issue in issue_list:\n", 237 | " print(f\"{issue['node']['number']:6} {issue['node']['title']:30}\")" 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": null, 243 | "metadata": {}, 244 | "outputs": [], 245 | "source": [] 246 | } 247 | ], 248 | "metadata": { 249 | "kernelspec": { 250 | "display_name": "Python 3", 251 | "language": "python", 252 | "name": "python3" 253 | }, 254 | "language_info": { 255 | "codemirror_mode": { 256 | "name": "ipython", 257 | "version": 3 258 | }, 259 | "file_extension": ".py", 260 | "mimetype": "text/x-python", 261 | "name": "python", 262 | "nbconvert_exporter": "python", 263 | "pygments_lexer": "ipython3", 264 | "version": "3.6.3" 265 | } 266 | }, 267 | "nbformat": 4, 268 | "nbformat_minor": 2 269 | } 270 | -------------------------------------------------------------------------------- /docs/examples/test-request.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# test-request" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "import csv\n", 17 | "import json\n", 18 | "import os\n", 19 | "import pprint\n", 20 | "\n", 21 | "import pandas as pd\n", 22 | "import requests" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 2, 28 | "metadata": {}, 29 | "outputs": [], 30 | "source": [ 31 | "# get api token and set authorization\n", 32 | "api_token = os.environ['GITHUB_API_TOKEN']\n", 33 | "headers = {'Authorization': f'token {api_token}'}" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": 3, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "# set url to a graphql endpoint\n", 43 | "url = 'https://api.github.com/graphql'" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 4, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "# add a json query\n", 53 | "query = \"\"\"\n", 54 | "{\n", 55 | " organization(login: \"jupyterhub\") {\n", 56 | " name\n", 57 | " repository(name: \"binderhub\") {\n", 58 | " name\n", 59 | " issues(last: 5) {\n", 60 | " edges {\n", 61 | " node {\n", 62 | " number\n", 63 | " title\n", 64 | " }\n", 65 | " }\n", 66 | " } \n", 67 | " }\n", 68 | " }\n", 69 | "}\n", 70 | "\"\"\"" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 13, 76 | "metadata": {}, 77 | "outputs": [], 78 | "source": [ 79 | "# submit the request\n", 80 | "r = requests.post(url=url, json={'query': query}, headers=headers)" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 14, 86 | "metadata": {}, 87 | "outputs": [ 88 | { 89 | "data": { 90 | "text/plain": [ 91 | "{'data': {'organization': {'name': 'JupyterHub',\n", 92 | " 'repository': {'issues': {'edges': [{'node': {'number': 344,\n", 93 | " 'title': 'Improve CONTRIBUTING.md'}},\n", 94 | " {'node': {'number': 345,\n", 95 | " 'title': 'Upgrade to kubernetes client version 4.0'}},\n", 96 | " {'node': {'number': 349,\n", 97 | " 'title': 'race condition mounting docker-in-docker socket in build'}},\n", 98 | " {'node': {'number': 350,\n", 99 | " 'title': 'Consider *not* launching automatically'}},\n", 100 | " {'node': {'number': 351, 'title': 'Awesome bar functionality'}}]},\n", 101 | " 'name': 'binderhub'}}}}" 102 | ] 103 | }, 104 | "execution_count": 14, 105 | "metadata": {}, 106 | "output_type": "execute_result" 107 | } 108 | ], 109 | "source": [ 110 | "binderhub = r.json()\n", 111 | "binderhub" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 7, 117 | "metadata": {}, 118 | "outputs": [ 119 | { 120 | "data": { 121 | "text/plain": [ 122 | "[{'node': {'number': 344, 'title': 'Improve CONTRIBUTING.md'}},\n", 123 | " {'node': {'number': 345,\n", 124 | " 'title': 'Upgrade to kubernetes client version 4.0'}},\n", 125 | " {'node': {'number': 349,\n", 126 | " 'title': 'race condition mounting docker-in-docker socket in build'}},\n", 127 | " {'node': {'number': 350, 'title': 'Consider *not* launching automatically'}},\n", 128 | " {'node': {'number': 351, 'title': 'Awesome bar functionality'}}]" 129 | ] 130 | }, 131 | "execution_count": 7, 132 | "metadata": {}, 133 | "output_type": "execute_result" 134 | } 135 | ], 136 | "source": [ 137 | "issue_list = binderhub['data']['organization']['repository']['issues']['edges']\n", 138 | "issue_list" 139 | ] 140 | }, 141 | { 142 | "cell_type": "markdown", 143 | "metadata": {}, 144 | "source": [ 145 | "## Report: Last 5 issues for a repo" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 8, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "name": "stdout", 155 | "output_type": "stream", 156 | "text": [ 157 | "binderhub: last five issues submitted\n", 158 | " 344 Improve CONTRIBUTING.md \n", 159 | " 345 Upgrade to kubernetes client version 4.0\n", 160 | " 349 race condition mounting docker-in-docker socket in build\n", 161 | " 350 Consider *not* launching automatically\n", 162 | " 351 Awesome bar functionality \n" 163 | ] 164 | } 165 | ], 166 | "source": [ 167 | "print(f\"{binderhub['data']['organization']['repository']['name']}: last five issues submitted\")\n", 168 | "for issue in issue_list:\n", 169 | " print(f\"{issue['node']['number']:6} {issue['node']['title']:30}\")" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": {}, 175 | "source": [ 176 | "Let's try another repo\n", 177 | "- make it a little more generic using `response` as a variable\n", 178 | "- select last 10 issues " 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 9, 184 | "metadata": {}, 185 | "outputs": [], 186 | "source": [ 187 | "# add a json query\n", 188 | "query = \"\"\"\n", 189 | "{\n", 190 | " organization(login: \"jupyterhub\") {\n", 191 | " name\n", 192 | " repository(name: \"jupyterhub\") {\n", 193 | " name\n", 194 | " issues(last: 10) {\n", 195 | " edges {\n", 196 | " node {\n", 197 | " number\n", 198 | " title\n", 199 | " }\n", 200 | " }\n", 201 | " } \n", 202 | " }\n", 203 | " }\n", 204 | "}\n", 205 | "\"\"\"\n", 206 | "r = requests.post(url=url, json={'query': query}, headers=headers)\n", 207 | "response = r.json()\n", 208 | "issue_list = response['data']['organization']['repository']['issues']['edges']" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 10, 214 | "metadata": {}, 215 | "outputs": [ 216 | { 217 | "name": "stdout", 218 | "output_type": "stream", 219 | "text": [ 220 | "jupyterhub: last 10 issues submitted\n", 221 | " 1573 more activity tracking \n", 222 | " 1574 Jupyter cannot find Three.Js files - URL redirection not working *updated*\n", 223 | " 1576 anchor links on markdown docs pages are unreliable\n", 224 | " 1579 LocalAuthenticator question \n", 225 | " 1580 Specify Kubernetes volume claims in Jupyterhub spawner\n", 226 | " 1582 Jupyterhub shared service error\n", 227 | " 1583 c.JupyterHub.logo_file is not working on after Jupyterhub Version: 0.8.1 upgrade\n", 228 | " 1584 config c.LocalProcessSpawner.mem_limit Issue\n", 229 | " 1585 Prometheus metrics to add to JupyterHub\n", 230 | " 1586 Errno 8: Nodename nor servname provided\n" 231 | ] 232 | } 233 | ], 234 | "source": [ 235 | "print(f\"{response['data']['organization']['repository']['name']}: last 10 issues submitted\")\n", 236 | "for issue in issue_list:\n", 237 | " print(f\"{issue['node']['number']:6} {issue['node']['title']:30}\")" 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": null, 243 | "metadata": {}, 244 | "outputs": [], 245 | "source": [] 246 | } 247 | ], 248 | "metadata": { 249 | "kernelspec": { 250 | "display_name": "Python 3", 251 | "language": "python", 252 | "name": "python3" 253 | }, 254 | "language_info": { 255 | "codemirror_mode": { 256 | "name": "ipython", 257 | "version": 3 258 | }, 259 | "file_extension": ".py", 260 | "mimetype": "text/x-python", 261 | "name": "python", 262 | "nbconvert_exporter": "python", 263 | "pygments_lexer": "ipython3", 264 | "version": "3.6.3" 265 | } 266 | }, 267 | "nbformat": 4, 268 | "nbformat_minor": 2 269 | } 270 | -------------------------------------------------------------------------------- /docs/getting-started.md: -------------------------------------------------------------------------------- 1 | # Getting Started 2 | 3 | ## Installation 4 | 5 | Supports Python 3.6+ 6 | 7 | 1. Clone the repo. 8 | 9 | git clone https://github.com/willingc/pyquery-ql.git 10 | 11 | 2. Set up your GitHub API token to avoid rate limit and export to your 12 | environment: 13 | 14 | export GITHUB_API_TOKEN = '......' 15 | 16 | ### pip installation instructions 17 | 18 | 3. From the root of the cloned repo: 19 | 20 | pip install -r requirements.txt 21 | 22 | ### conda installation instructions 23 | 24 | 3. Create the conda environment: 25 | 26 | conda env create -f environment.yml 27 | 28 | 4. Activate the environment: 29 | 30 | source activate pyquery 31 | 32 | ## Usage 33 | 34 | To run: 35 | 36 | python3 pyquery-ql.py -------------------------------------------------------------------------------- /docs/graphql-notes.md: -------------------------------------------------------------------------------- 1 | # Notes about GraphQL 2 | 3 | ## GraphQL basics 4 | 5 | - [GraphQL](http://graphql.org) 6 | - describe your data 7 | - ask for what you want 8 | - get predictable results 9 | 10 | - A [GraphQL service](http://graphql.org/learn/queries/) defines **types** 11 | and **fields** on those types. A function is created for each field on 12 | each type. 13 | 14 | - **field** can be a specific type (like String) or an Object 15 | - **arguments** can be passed to fields when making a query 16 | ``` 17 | { 18 | human(id: "1000") { 19 | name 20 | height 21 | } 22 | } 23 | 24 | - GraphQL schema and types 25 | ``` 26 | type Character { 27 | name: String! 28 | appearsIn: [Episode]! 29 | } 30 | ``` 31 | 32 | - GraphQL Object type `Character` 33 | - fields `name` and `appearsIn` 34 | - Scalar type `String` 35 | - `!` non-nullable (will always return a value when queried) 36 | - array `[Episode]` 37 | 38 | - Arguments 39 | 40 | - every field can have 0 or more arguments 41 | - all arguments are named 42 | - all arguments are passed by name specifically (unlike JS or Python 43 | where functions take a list of ordered arguments 44 | - required or optional 45 | 46 | - Scalar types 47 | 48 | - fields that resolve to real data 49 | - leaves of the query 50 | - default scalar types: 51 | - `Int` 52 | - `Float` 53 | - `String` 54 | - `Boolean` 55 | - `ID` unique identifier 56 | - custom scalar types are possible i.e. `Date` 57 | - `Enums` 58 | 59 | - Modifiers: Non-Null and List (array) 60 | - List [] can be null 61 | - a member of the list can not be null or it will generate an error 62 | - []! the actual list can not be null though a member can be null 63 | 64 | - [**Introspection**](http://graphql.org/learn/introspection/) is a beautiful 65 | thing; ask for `__schema` for all the good stuff on the types 66 | 67 | ## GraphQL Client needs 68 | 69 | - [Pagination](http://graphql.org/learn/introspection/) 70 | - plurals 71 | - slicing 72 | - pagination and edges 73 | - Connections 74 | - [Caching](http://graphql.org/learn/caching/) 75 | - ID 76 | - pros and cons of globally unique IDs 77 | - server defined or client derived are both options 78 | 79 | ## GraphQL spec 80 | 81 | - [The Spec](http://facebook.github.io/graphql/October2016/) 82 | - [Response Format](http://facebook.github.io/graphql/October2016/#sec-Response-Format) 83 | - [Grammar Summary](http://facebook.github.io/graphql/October2016/#sec-Appendix-Grammar-Summary) -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. pyquery-ql documentation master file, created by 2 | sphinx-quickstart on Thu Dec 14 13:38:04 2017. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | Welcome to pyquery-ql 7 | ===================== 8 | 9 | Query `GitHub API v4 `_ using `pyquery-ql`_ 10 | a lightweight `GraphQL `_ client. 11 | 12 | .. toctree:: 13 | :maxdepth: 2 14 | :caption: Contents: 15 | 16 | intro.md 17 | getting-started.md 18 | graphql-notes.md 19 | 20 | 21 | Some sample notebooks demonstrate how to use GitHub GraphQL, Jupyter notebooks, 22 | and pandas with Python 3.6. 23 | 24 | .. toctree:: 25 | :maxdepth: 2 26 | :caption: Example notebooks: 27 | 28 | examples/test-request.ipynb 29 | examples/open-issues-prs-by-repo.ipynb 30 | examples/report-repo-last-n-issues.ipynb 31 | examples/pandas-play.ipynb 32 | 33 | 34 | License 35 | ------- 36 | 37 | MIT 38 | 39 | Indices and tables 40 | ------------------ 41 | 42 | * :ref:`genindex` 43 | * :ref:`modindex` 44 | * :ref:`search` 45 | 46 | 47 | .. _pyquery-ql: https://github.com/willingc/pyquery-ql 48 | -------------------------------------------------------------------------------- /docs/intro.md: -------------------------------------------------------------------------------- 1 | # Introduction 2 | 3 | ## What is pyquery-ql? 4 | 5 | It's a repository for examples and tips when querying GraphQL APIs using 6 | *Python* and the popular *Jupyter notebook*. It provides examples on using the 7 | elegant *requests* library and *pandas* to do data collection and interactive 8 | data exploration. 9 | 10 | **pyquery-ql** is also a very early prototype for a lightweight Python 11 | GraphQL client. It's based on Python 3.6 or higher and takes advantage of 12 | f-strings (if you haven't tried them yet, please do). 13 | 14 | ## Why create a lightweight GraphQL client for Python? 15 | 16 | In data science and scientific programming, we do lots of queries for data 17 | from different sources. With the increased use of GraphQL, it's helpful to 18 | have a lightweight client to do quick queries. 19 | 20 | With Jupyter and nteract notebooks, it's critical to one's workflow to be 21 | able to prototype and iterate from within the notebook. While full featured 22 | clients, like [Apollo Client](https://www.apollographql.com/client/), and 23 | explorers, such as [GitHub's GraphQL Explorer](https://developer.github.com/v4/explorer/), 24 | are wonderful, the user must leave the notebook to see their benefits. 25 | 26 | Using the Jupyter or nteract notebooks, a user explores data, and GraphQL is 27 | fun to use since it's very easy to introspect the data source. A lightweight 28 | Python client provides a simple way to bridge the data and 29 | the Python scientific stack, especially pandas, from within the interactive 30 | notebook. 31 | -------------------------------------------------------------------------------- /docs/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | pushd %~dp0 4 | 5 | REM Command file for Sphinx documentation 6 | 7 | if "%SPHINXBUILD%" == "" ( 8 | set SPHINXBUILD=python -msphinx 9 | ) 10 | set SOURCEDIR=. 11 | set BUILDDIR=_build 12 | set SPHINXPROJ=pyquery-ql 13 | 14 | if "%1" == "" goto help 15 | 16 | %SPHINXBUILD% >NUL 2>NUL 17 | if errorlevel 9009 ( 18 | echo. 19 | echo.The Sphinx module was not found. Make sure you have Sphinx installed, 20 | echo.then set the SPHINXBUILD environment variable to point to the full 21 | echo.path of the 'sphinx-build' executable. Alternatively you may add the 22 | echo.Sphinx directory to PATH. 23 | echo. 24 | echo.If you don't have Sphinx installed, grab it from 25 | echo.http://sphinx-doc.org/ 26 | exit /b 1 27 | ) 28 | 29 | %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% 30 | goto end 31 | 32 | :help 33 | %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% 34 | 35 | :end 36 | popd 37 | -------------------------------------------------------------------------------- /docs/requirements-doc.txt: -------------------------------------------------------------------------------- 1 | requests>=2.18 2 | Sphinx>=1.6 3 | sphinx_rtd_theme 4 | recommonmark==0.4 5 | nbsphinx 6 | -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- 1 | name: pyquery 2 | channels: 3 | - conda-forge 4 | - defaults 5 | dependencies: 6 | - python=3.6 7 | - jupyter 8 | - pandas 9 | - requests 10 | -------------------------------------------------------------------------------- /notebooks/report-repo-last-n-issues.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Repo report: last `n` issues" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "import csv\n", 17 | "import json\n", 18 | "import os\n", 19 | "import pprint\n", 20 | "\n", 21 | "import pandas as pd\n", 22 | "import requests" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 2, 28 | "metadata": {}, 29 | "outputs": [], 30 | "source": [ 31 | "# get api token and set authorization\n", 32 | "api_token = os.environ['GITHUB_API_TOKEN']\n", 33 | "headers = {'Authorization': f'token {api_token}'}" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": 3, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "# set url to a graphql endpoint\n", 43 | "url = 'https://api.github.com/graphql'" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 4, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "# add a json query\n", 53 | "query = \"\"\"\n", 54 | "{\n", 55 | " organization(login: \"jupyterhub\") {\n", 56 | " name\n", 57 | " repository(name: \"binderhub\") {\n", 58 | " name\n", 59 | " issues(last: 5) {\n", 60 | " edges {\n", 61 | " node {\n", 62 | " number\n", 63 | " title\n", 64 | " }\n", 65 | " }\n", 66 | " } \n", 67 | " }\n", 68 | " }\n", 69 | "}\n", 70 | "\"\"\"" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 5, 76 | "metadata": {}, 77 | "outputs": [], 78 | "source": [ 79 | "# submit the request\n", 80 | "r = requests.post(url=url, json={'query': query}, headers=headers)" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 6, 86 | "metadata": {}, 87 | "outputs": [ 88 | { 89 | "data": { 90 | "text/plain": [ 91 | "{'data': {'organization': {'name': 'JupyterHub',\n", 92 | " 'repository': {'issues': {'edges': [{'node': {'number': 344,\n", 93 | " 'title': 'Improve CONTRIBUTING.md'}},\n", 94 | " {'node': {'number': 345,\n", 95 | " 'title': 'Upgrade to kubernetes client version 4.0'}},\n", 96 | " {'node': {'number': 349,\n", 97 | " 'title': 'race condition mounting docker-in-docker socket in build'}},\n", 98 | " {'node': {'number': 350,\n", 99 | " 'title': 'Consider *not* launching automatically'}},\n", 100 | " {'node': {'number': 351, 'title': 'Awesome bar functionality'}}]},\n", 101 | " 'name': 'binderhub'}}}}" 102 | ] 103 | }, 104 | "execution_count": 6, 105 | "metadata": {}, 106 | "output_type": "execute_result" 107 | } 108 | ], 109 | "source": [ 110 | "binderhub = r.json()\n", 111 | "binderhub" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": 7, 117 | "metadata": {}, 118 | "outputs": [ 119 | { 120 | "data": { 121 | "text/plain": [ 122 | "[{'node': {'number': 344, 'title': 'Improve CONTRIBUTING.md'}},\n", 123 | " {'node': {'number': 345,\n", 124 | " 'title': 'Upgrade to kubernetes client version 4.0'}},\n", 125 | " {'node': {'number': 349,\n", 126 | " 'title': 'race condition mounting docker-in-docker socket in build'}},\n", 127 | " {'node': {'number': 350, 'title': 'Consider *not* launching automatically'}},\n", 128 | " {'node': {'number': 351, 'title': 'Awesome bar functionality'}}]" 129 | ] 130 | }, 131 | "execution_count": 7, 132 | "metadata": {}, 133 | "output_type": "execute_result" 134 | } 135 | ], 136 | "source": [ 137 | "issue_list = binderhub['data']['organization']['repository']['issues']['edges']\n", 138 | "issue_list" 139 | ] 140 | }, 141 | { 142 | "cell_type": "markdown", 143 | "metadata": {}, 144 | "source": [ 145 | "## Report: Last 5 issues for a repo" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": 8, 151 | "metadata": {}, 152 | "outputs": [ 153 | { 154 | "name": "stdout", 155 | "output_type": "stream", 156 | "text": [ 157 | "binderhub: last five issues submitted\n", 158 | " 344 Improve CONTRIBUTING.md \n", 159 | " 345 Upgrade to kubernetes client version 4.0\n", 160 | " 349 race condition mounting docker-in-docker socket in build\n", 161 | " 350 Consider *not* launching automatically\n", 162 | " 351 Awesome bar functionality \n" 163 | ] 164 | } 165 | ], 166 | "source": [ 167 | "print(f\"{binderhub['data']['organization']['repository']['name']}: last five issues submitted\")\n", 168 | "for issue in issue_list:\n", 169 | " print(f\"{issue['node']['number']:6} {issue['node']['title']:30}\")" 170 | ] 171 | }, 172 | { 173 | "cell_type": "markdown", 174 | "metadata": {}, 175 | "source": [ 176 | "Let's try another repo\n", 177 | "- make it a little more generic using `response` as a variable\n", 178 | "- select last 10 issues " 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 9, 184 | "metadata": {}, 185 | "outputs": [], 186 | "source": [ 187 | "# add a json query\n", 188 | "query = \"\"\"\n", 189 | "{\n", 190 | " organization(login: \"jupyterhub\") {\n", 191 | " name\n", 192 | " repository(name: \"jupyterhub\") {\n", 193 | " name\n", 194 | " issues(last: 10) {\n", 195 | " edges {\n", 196 | " node {\n", 197 | " number\n", 198 | " title\n", 199 | " }\n", 200 | " }\n", 201 | " } \n", 202 | " }\n", 203 | " }\n", 204 | "}\n", 205 | "\"\"\"\n", 206 | "r = requests.post(url=url, json={'query': query}, headers=headers)\n", 207 | "response = r.json()\n", 208 | "issue_list = response['data']['organization']['repository']['issues']['edges']" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 10, 214 | "metadata": {}, 215 | "outputs": [ 216 | { 217 | "name": "stdout", 218 | "output_type": "stream", 219 | "text": [ 220 | "jupyterhub: last 10 issues submitted\n", 221 | " 1573 more activity tracking \n", 222 | " 1574 Jupyter cannot find Three.Js files - URL redirection not working *updated*\n", 223 | " 1576 anchor links on markdown docs pages are unreliable\n", 224 | " 1579 LocalAuthenticator question \n", 225 | " 1580 Specify Kubernetes volume claims in Jupyterhub spawner\n", 226 | " 1582 Jupyterhub shared service error\n", 227 | " 1583 c.JupyterHub.logo_file is not working on after Jupyterhub Version: 0.8.1 upgrade\n", 228 | " 1584 config c.LocalProcessSpawner.mem_limit Issue\n", 229 | " 1585 Prometheus metrics to add to JupyterHub\n", 230 | " 1586 Errno 8: Nodename nor servname provided\n" 231 | ] 232 | } 233 | ], 234 | "source": [ 235 | "print(f\"{response['data']['organization']['repository']['name']}: last 10 issues submitted\")\n", 236 | "for issue in issue_list:\n", 237 | " print(f\"{issue['node']['number']:6} {issue['node']['title']:30}\")" 238 | ] 239 | }, 240 | { 241 | "cell_type": "code", 242 | "execution_count": null, 243 | "metadata": {}, 244 | "outputs": [], 245 | "source": [] 246 | } 247 | ], 248 | "metadata": { 249 | "kernelspec": { 250 | "display_name": "Python 3", 251 | "language": "python", 252 | "name": "python3" 253 | }, 254 | "language_info": { 255 | "codemirror_mode": { 256 | "name": "ipython", 257 | "version": 3 258 | }, 259 | "file_extension": ".py", 260 | "mimetype": "text/x-python", 261 | "name": "python", 262 | "nbconvert_exporter": "python", 263 | "pygments_lexer": "ipython3", 264 | "version": "3.6.3" 265 | } 266 | }, 267 | "nbformat": 4, 268 | "nbformat_minor": 2 269 | } 270 | -------------------------------------------------------------------------------- /notebooks/traverse-json.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Use pyquery-ql.py\n", 8 | "\n", 9 | "Send a graphql query to GitHub\n", 10 | "and pretty print output.\n", 11 | "\n", 12 | "Supports Python 3.6+" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 1, 18 | "metadata": {}, 19 | "outputs": [], 20 | "source": [ 21 | "import json\n", 22 | "import os\n", 23 | "import pprint\n", 24 | "\n", 25 | "import requests" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 2, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "# get api token and set authorization\n", 35 | "api_token = os.environ['GITHUB_API_TOKEN']\n", 36 | "headers = {'Authorization': f'token {api_token}'}" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 3, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "# set url to a graphql endpoint\n", 46 | "url = 'https://api.github.com/graphql'" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 4, 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "# add a json query\n", 56 | "query = \"\"\"\n", 57 | "{\n", 58 | " organization(login: \"jupyterhub\") {\n", 59 | " repositories(first: 30) {\n", 60 | " totalCount\n", 61 | " edges {\n", 62 | " node {\n", 63 | " name\n", 64 | " url\n", 65 | " issues(states: OPEN) {\n", 66 | " totalCount\n", 67 | " }\n", 68 | " pullRequests(states: OPEN) {\n", 69 | " totalCount\n", 70 | " }\n", 71 | " }\n", 72 | " }\n", 73 | " }\n", 74 | " }\n", 75 | "}\n", 76 | "\"\"\"" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": 5, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "# submit the request\n", 86 | "r = requests.post(url=url, json={'query': query}, headers=headers)" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 22, 92 | "metadata": {}, 93 | "outputs": [ 94 | { 95 | "data": { 96 | "text/plain": [ 97 | "{'data': {'organization': {'repositories': {'edges': [{'node': {'issues': {'totalCount': 143},\n", 98 | " 'name': 'jupyterhub',\n", 99 | " 'pullRequests': {'totalCount': 6},\n", 100 | " 'url': 'https://github.com/jupyterhub/jupyterhub'}},\n", 101 | " {'node': {'issues': {'totalCount': 10},\n", 102 | " 'name': 'configurable-http-proxy',\n", 103 | " 'pullRequests': {'totalCount': 0},\n", 104 | " 'url': 'https://github.com/jupyterhub/configurable-http-proxy'}},\n", 105 | " {'node': {'issues': {'totalCount': 13},\n", 106 | " 'name': 'oauthenticator',\n", 107 | " 'pullRequests': {'totalCount': 2},\n", 108 | " 'url': 'https://github.com/jupyterhub/oauthenticator'}},\n", 109 | " {'node': {'issues': {'totalCount': 22},\n", 110 | " 'name': 'dockerspawner',\n", 111 | " 'pullRequests': {'totalCount': 1},\n", 112 | " 'url': 'https://github.com/jupyterhub/dockerspawner'}},\n", 113 | " {'node': {'issues': {'totalCount': 2},\n", 114 | " 'name': 'sudospawner',\n", 115 | " 'pullRequests': {'totalCount': 0},\n", 116 | " 'url': 'https://github.com/jupyterhub/sudospawner'}},\n", 117 | " {'node': {'issues': {'totalCount': 12},\n", 118 | " 'name': 'batchspawner',\n", 119 | " 'pullRequests': {'totalCount': 5},\n", 120 | " 'url': 'https://github.com/jupyterhub/batchspawner'}},\n", 121 | " {'node': {'issues': {'totalCount': 25},\n", 122 | " 'name': 'kubespawner',\n", 123 | " 'pullRequests': {'totalCount': 3},\n", 124 | " 'url': 'https://github.com/jupyterhub/kubespawner'}},\n", 125 | " {'node': {'issues': {'totalCount': 18},\n", 126 | " 'name': 'ldapauthenticator',\n", 127 | " 'pullRequests': {'totalCount': 2},\n", 128 | " 'url': 'https://github.com/jupyterhub/ldapauthenticator'}},\n", 129 | " {'node': {'issues': {'totalCount': 9},\n", 130 | " 'name': 'jupyterhub-deploy-docker',\n", 131 | " 'pullRequests': {'totalCount': 1},\n", 132 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-docker'}},\n", 133 | " {'node': {'issues': {'totalCount': 13},\n", 134 | " 'name': 'jupyterhub-deploy-teaching',\n", 135 | " 'pullRequests': {'totalCount': 2},\n", 136 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-teaching'}},\n", 137 | " {'node': {'issues': {'totalCount': 5},\n", 138 | " 'name': 'jupyterhub-tutorial',\n", 139 | " 'pullRequests': {'totalCount': 0},\n", 140 | " 'url': 'https://github.com/jupyterhub/jupyterhub-tutorial'}},\n", 141 | " {'node': {'issues': {'totalCount': 2},\n", 142 | " 'name': 'jupyterhub-deploy-hpc',\n", 143 | " 'pullRequests': {'totalCount': 0},\n", 144 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-hpc'}},\n", 145 | " {'node': {'issues': {'totalCount': 4},\n", 146 | " 'name': 'systemdspawner',\n", 147 | " 'pullRequests': {'totalCount': 2},\n", 148 | " 'url': 'https://github.com/jupyterhub/systemdspawner'}},\n", 149 | " {'node': {'issues': {'totalCount': 5},\n", 150 | " 'name': 'wrapspawner',\n", 151 | " 'pullRequests': {'totalCount': 1},\n", 152 | " 'url': 'https://github.com/jupyterhub/wrapspawner'}},\n", 153 | " {'node': {'issues': {'totalCount': 4},\n", 154 | " 'name': 'jupyterlab-hub',\n", 155 | " 'pullRequests': {'totalCount': 0},\n", 156 | " 'url': 'https://github.com/jupyterhub/jupyterlab-hub'}},\n", 157 | " {'node': {'issues': {'totalCount': 2},\n", 158 | " 'name': 'nbserverproxy',\n", 159 | " 'pullRequests': {'totalCount': 1},\n", 160 | " 'url': 'https://github.com/jupyterhub/nbserverproxy'}},\n", 161 | " {'node': {'issues': {'totalCount': 3},\n", 162 | " 'name': 'jupyterhub-example-kerberos',\n", 163 | " 'pullRequests': {'totalCount': 0},\n", 164 | " 'url': 'https://github.com/jupyterhub/jupyterhub-example-kerberos'}},\n", 165 | " {'node': {'issues': {'totalCount': 10},\n", 166 | " 'name': 'hubshare',\n", 167 | " 'pullRequests': {'totalCount': 0},\n", 168 | " 'url': 'https://github.com/jupyterhub/hubshare'}},\n", 169 | " {'node': {'issues': {'totalCount': 4},\n", 170 | " 'name': 'nbrsessionproxy',\n", 171 | " 'pullRequests': {'totalCount': 2},\n", 172 | " 'url': 'https://github.com/jupyterhub/nbrsessionproxy'}},\n", 173 | " {'node': {'issues': {'totalCount': 0},\n", 174 | " 'name': 'tmpauthenticator',\n", 175 | " 'pullRequests': {'totalCount': 0},\n", 176 | " 'url': 'https://github.com/jupyterhub/tmpauthenticator'}},\n", 177 | " {'node': {'issues': {'totalCount': 90},\n", 178 | " 'name': 'zero-to-jupyterhub-k8s',\n", 179 | " 'pullRequests': {'totalCount': 3},\n", 180 | " 'url': 'https://github.com/jupyterhub/zero-to-jupyterhub-k8s'}},\n", 181 | " {'node': {'issues': {'totalCount': 3},\n", 182 | " 'name': 'helm-chart',\n", 183 | " 'pullRequests': {'totalCount': 0},\n", 184 | " 'url': 'https://github.com/jupyterhub/helm-chart'}},\n", 185 | " {'node': {'issues': {'totalCount': 75},\n", 186 | " 'name': 'binderhub',\n", 187 | " 'pullRequests': {'totalCount': 5},\n", 188 | " 'url': 'https://github.com/jupyterhub/binderhub'}},\n", 189 | " {'node': {'issues': {'totalCount': 24},\n", 190 | " 'name': 'mybinder.org-deploy',\n", 191 | " 'pullRequests': {'totalCount': 1},\n", 192 | " 'url': 'https://github.com/jupyterhub/mybinder.org-deploy'}},\n", 193 | " {'node': {'issues': {'totalCount': 9},\n", 194 | " 'name': 'binder',\n", 195 | " 'pullRequests': {'totalCount': 1},\n", 196 | " 'url': 'https://github.com/jupyterhub/binder'}},\n", 197 | " {'node': {'issues': {'totalCount': 0},\n", 198 | " 'name': 'nullauthenticator',\n", 199 | " 'pullRequests': {'totalCount': 0},\n", 200 | " 'url': 'https://github.com/jupyterhub/nullauthenticator'}},\n", 201 | " {'node': {'issues': {'totalCount': 2},\n", 202 | " 'name': 'team-compass',\n", 203 | " 'pullRequests': {'totalCount': 1},\n", 204 | " 'url': 'https://github.com/jupyterhub/team-compass'}}],\n", 205 | " 'totalCount': 27}}}}" 206 | ] 207 | }, 208 | "execution_count": 22, 209 | "metadata": {}, 210 | "output_type": "execute_result" 211 | } 212 | ], 213 | "source": [ 214 | "data = json.loads(r.text)\n", 215 | "data" 216 | ] 217 | }, 218 | { 219 | "cell_type": "code", 220 | "execution_count": null, 221 | "metadata": {}, 222 | "outputs": [], 223 | "source": [ 224 | "# pprint.pprint(data)" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "metadata": {}, 230 | "source": [ 231 | "Walk the JSON response contents" 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": 12, 237 | "metadata": {}, 238 | "outputs": [], 239 | "source": [ 240 | "def traverse(obj, path=None, callback=None):\n", 241 | " if path is None:\n", 242 | " path = []\n", 243 | "\n", 244 | " if isinstance(obj, dict):\n", 245 | " value = {k: traverse(v, path + [k], callback)\n", 246 | " for k, v in obj.items()}\n", 247 | " elif isinstance(obj, list):\n", 248 | " value = [traverse(elem, path + [[]], callback)\n", 249 | " for elem in obj]\n", 250 | " else:\n", 251 | " value = obj\n", 252 | "\n", 253 | " if callback is None: # if a callback is provided, call it to get the new value\n", 254 | " return value\n", 255 | " else:\n", 256 | " return callback(path, value)" 257 | ] 258 | }, 259 | { 260 | "cell_type": "code", 261 | "execution_count": 47, 262 | "metadata": {}, 263 | "outputs": [ 264 | { 265 | "data": { 266 | "text/plain": [ 267 | "{'data': {'organization': {'repositories': {'edges': [{'node': {'issues': {'totalCount': 143},\n", 268 | " 'name': 'jupyterhub',\n", 269 | " 'pullRequests': {'totalCount': 6},\n", 270 | " 'url': 'https://github.com/jupyterhub/jupyterhub'}},\n", 271 | " {'node': {'issues': {'totalCount': 10},\n", 272 | " 'name': 'configurable-http-proxy',\n", 273 | " 'pullRequests': {'totalCount': 0},\n", 274 | " 'url': 'https://github.com/jupyterhub/configurable-http-proxy'}},\n", 275 | " {'node': {'issues': {'totalCount': 13},\n", 276 | " 'name': 'oauthenticator',\n", 277 | " 'pullRequests': {'totalCount': 2},\n", 278 | " 'url': 'https://github.com/jupyterhub/oauthenticator'}},\n", 279 | " {'node': {'issues': {'totalCount': 22},\n", 280 | " 'name': 'dockerspawner',\n", 281 | " 'pullRequests': {'totalCount': 1},\n", 282 | " 'url': 'https://github.com/jupyterhub/dockerspawner'}},\n", 283 | " {'node': {'issues': {'totalCount': 2},\n", 284 | " 'name': 'sudospawner',\n", 285 | " 'pullRequests': {'totalCount': 0},\n", 286 | " 'url': 'https://github.com/jupyterhub/sudospawner'}},\n", 287 | " {'node': {'issues': {'totalCount': 12},\n", 288 | " 'name': 'batchspawner',\n", 289 | " 'pullRequests': {'totalCount': 5},\n", 290 | " 'url': 'https://github.com/jupyterhub/batchspawner'}},\n", 291 | " {'node': {'issues': {'totalCount': 25},\n", 292 | " 'name': 'kubespawner',\n", 293 | " 'pullRequests': {'totalCount': 3},\n", 294 | " 'url': 'https://github.com/jupyterhub/kubespawner'}},\n", 295 | " {'node': {'issues': {'totalCount': 18},\n", 296 | " 'name': 'ldapauthenticator',\n", 297 | " 'pullRequests': {'totalCount': 2},\n", 298 | " 'url': 'https://github.com/jupyterhub/ldapauthenticator'}},\n", 299 | " {'node': {'issues': {'totalCount': 9},\n", 300 | " 'name': 'jupyterhub-deploy-docker',\n", 301 | " 'pullRequests': {'totalCount': 1},\n", 302 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-docker'}},\n", 303 | " {'node': {'issues': {'totalCount': 13},\n", 304 | " 'name': 'jupyterhub-deploy-teaching',\n", 305 | " 'pullRequests': {'totalCount': 2},\n", 306 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-teaching'}},\n", 307 | " {'node': {'issues': {'totalCount': 5},\n", 308 | " 'name': 'jupyterhub-tutorial',\n", 309 | " 'pullRequests': {'totalCount': 0},\n", 310 | " 'url': 'https://github.com/jupyterhub/jupyterhub-tutorial'}},\n", 311 | " {'node': {'issues': {'totalCount': 2},\n", 312 | " 'name': 'jupyterhub-deploy-hpc',\n", 313 | " 'pullRequests': {'totalCount': 0},\n", 314 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-hpc'}},\n", 315 | " {'node': {'issues': {'totalCount': 4},\n", 316 | " 'name': 'systemdspawner',\n", 317 | " 'pullRequests': {'totalCount': 2},\n", 318 | " 'url': 'https://github.com/jupyterhub/systemdspawner'}},\n", 319 | " {'node': {'issues': {'totalCount': 5},\n", 320 | " 'name': 'wrapspawner',\n", 321 | " 'pullRequests': {'totalCount': 1},\n", 322 | " 'url': 'https://github.com/jupyterhub/wrapspawner'}},\n", 323 | " {'node': {'issues': {'totalCount': 4},\n", 324 | " 'name': 'jupyterlab-hub',\n", 325 | " 'pullRequests': {'totalCount': 0},\n", 326 | " 'url': 'https://github.com/jupyterhub/jupyterlab-hub'}},\n", 327 | " {'node': {'issues': {'totalCount': 2},\n", 328 | " 'name': 'nbserverproxy',\n", 329 | " 'pullRequests': {'totalCount': 1},\n", 330 | " 'url': 'https://github.com/jupyterhub/nbserverproxy'}},\n", 331 | " {'node': {'issues': {'totalCount': 3},\n", 332 | " 'name': 'jupyterhub-example-kerberos',\n", 333 | " 'pullRequests': {'totalCount': 0},\n", 334 | " 'url': 'https://github.com/jupyterhub/jupyterhub-example-kerberos'}},\n", 335 | " {'node': {'issues': {'totalCount': 10},\n", 336 | " 'name': 'hubshare',\n", 337 | " 'pullRequests': {'totalCount': 0},\n", 338 | " 'url': 'https://github.com/jupyterhub/hubshare'}},\n", 339 | " {'node': {'issues': {'totalCount': 4},\n", 340 | " 'name': 'nbrsessionproxy',\n", 341 | " 'pullRequests': {'totalCount': 2},\n", 342 | " 'url': 'https://github.com/jupyterhub/nbrsessionproxy'}},\n", 343 | " {'node': {'issues': {'totalCount': 0},\n", 344 | " 'name': 'tmpauthenticator',\n", 345 | " 'pullRequests': {'totalCount': 0},\n", 346 | " 'url': 'https://github.com/jupyterhub/tmpauthenticator'}},\n", 347 | " {'node': {'issues': {'totalCount': 90},\n", 348 | " 'name': 'zero-to-jupyterhub-k8s',\n", 349 | " 'pullRequests': {'totalCount': 3},\n", 350 | " 'url': 'https://github.com/jupyterhub/zero-to-jupyterhub-k8s'}},\n", 351 | " {'node': {'issues': {'totalCount': 3},\n", 352 | " 'name': 'helm-chart',\n", 353 | " 'pullRequests': {'totalCount': 0},\n", 354 | " 'url': 'https://github.com/jupyterhub/helm-chart'}},\n", 355 | " {'node': {'issues': {'totalCount': 75},\n", 356 | " 'name': 'binderhub',\n", 357 | " 'pullRequests': {'totalCount': 5},\n", 358 | " 'url': 'https://github.com/jupyterhub/binderhub'}},\n", 359 | " {'node': {'issues': {'totalCount': 24},\n", 360 | " 'name': 'mybinder.org-deploy',\n", 361 | " 'pullRequests': {'totalCount': 1},\n", 362 | " 'url': 'https://github.com/jupyterhub/mybinder.org-deploy'}},\n", 363 | " {'node': {'issues': {'totalCount': 9},\n", 364 | " 'name': 'binder',\n", 365 | " 'pullRequests': {'totalCount': 1},\n", 366 | " 'url': 'https://github.com/jupyterhub/binder'}},\n", 367 | " {'node': {'issues': {'totalCount': 0},\n", 368 | " 'name': 'nullauthenticator',\n", 369 | " 'pullRequests': {'totalCount': 0},\n", 370 | " 'url': 'https://github.com/jupyterhub/nullauthenticator'}},\n", 371 | " {'node': {'issues': {'totalCount': 2},\n", 372 | " 'name': 'team-compass',\n", 373 | " 'pullRequests': {'totalCount': 1},\n", 374 | " 'url': 'https://github.com/jupyterhub/team-compass'}}],\n", 375 | " 'totalCount': 27}}}}" 376 | ] 377 | }, 378 | "execution_count": 47, 379 | "metadata": {}, 380 | "output_type": "execute_result" 381 | } 382 | ], 383 | "source": [ 384 | "traverse(data)" 385 | ] 386 | }, 387 | { 388 | "cell_type": "code", 389 | "execution_count": 15, 390 | "metadata": {}, 391 | "outputs": [], 392 | "source": [ 393 | "def traverse_modify(obj, target_path, action):\n", 394 | " target_path = to_path(target_path) # converts 'foo.bar' to ['foo', 'bar']\n", 395 | "\n", 396 | " # This will get called for every path/value in the structure\n", 397 | " def transformer(path, value):\n", 398 | " if path == target_path:\n", 399 | " return action(value)\n", 400 | " else:\n", 401 | " return value\n", 402 | "\n", 403 | " return traverse(obj, callback=transformer)" 404 | ] 405 | }, 406 | { 407 | "cell_type": "code", 408 | "execution_count": 16, 409 | "metadata": {}, 410 | "outputs": [], 411 | "source": [ 412 | "def to_path(path):\n", 413 | " \"\"\"\n", 414 | " Helper function, converting path strings into path lists.\n", 415 | " >>> to_path('foo')\n", 416 | " ['foo']\n", 417 | " >>> to_path('foo.bar')\n", 418 | " ['foo', 'bar']\n", 419 | " >>> to_path('foo.bar[]')\n", 420 | " ['foo', 'bar', []]\n", 421 | " \"\"\"\n", 422 | " if isinstance(path, list):\n", 423 | " return path # already in list format\n", 424 | "\n", 425 | " def _iter_path(path):\n", 426 | " for parts in path.split('[]'):\n", 427 | " for part in parts.strip('.').split('.'):\n", 428 | " yield part\n", 429 | " yield []\n", 430 | "\n", 431 | " return list(_iter_path(path))[:-1]" 432 | ] 433 | }, 434 | { 435 | "cell_type": "code", 436 | "execution_count": 64, 437 | "metadata": {}, 438 | "outputs": [ 439 | { 440 | "data": { 441 | "text/plain": [ 442 | "{'data': {'organization': {'repositories': {'edges': [{'node': {'issues': {'totalCount': 143},\n", 443 | " 'name': 'jupyterhub',\n", 444 | " 'pullRequests': {'totalCount': 6},\n", 445 | " 'url': 'https://github.com/jupyterhub/jupyterhub'}},\n", 446 | " {'node': {'issues': {'totalCount': 10},\n", 447 | " 'name': 'configurable-http-proxy',\n", 448 | " 'pullRequests': {'totalCount': 0},\n", 449 | " 'url': 'https://github.com/jupyterhub/configurable-http-proxy'}},\n", 450 | " {'node': {'issues': {'totalCount': 13},\n", 451 | " 'name': 'oauthenticator',\n", 452 | " 'pullRequests': {'totalCount': 2},\n", 453 | " 'url': 'https://github.com/jupyterhub/oauthenticator'}},\n", 454 | " {'node': {'issues': {'totalCount': 22},\n", 455 | " 'name': 'dockerspawner',\n", 456 | " 'pullRequests': {'totalCount': 1},\n", 457 | " 'url': 'https://github.com/jupyterhub/dockerspawner'}},\n", 458 | " {'node': {'issues': {'totalCount': 2},\n", 459 | " 'name': 'sudospawner',\n", 460 | " 'pullRequests': {'totalCount': 0},\n", 461 | " 'url': 'https://github.com/jupyterhub/sudospawner'}},\n", 462 | " {'node': {'issues': {'totalCount': 12},\n", 463 | " 'name': 'batchspawner',\n", 464 | " 'pullRequests': {'totalCount': 5},\n", 465 | " 'url': 'https://github.com/jupyterhub/batchspawner'}},\n", 466 | " {'node': {'issues': {'totalCount': 25},\n", 467 | " 'name': 'kubespawner',\n", 468 | " 'pullRequests': {'totalCount': 3},\n", 469 | " 'url': 'https://github.com/jupyterhub/kubespawner'}},\n", 470 | " {'node': {'issues': {'totalCount': 18},\n", 471 | " 'name': 'ldapauthenticator',\n", 472 | " 'pullRequests': {'totalCount': 2},\n", 473 | " 'url': 'https://github.com/jupyterhub/ldapauthenticator'}},\n", 474 | " {'node': {'issues': {'totalCount': 9},\n", 475 | " 'name': 'jupyterhub-deploy-docker',\n", 476 | " 'pullRequests': {'totalCount': 1},\n", 477 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-docker'}},\n", 478 | " {'node': {'issues': {'totalCount': 13},\n", 479 | " 'name': 'jupyterhub-deploy-teaching',\n", 480 | " 'pullRequests': {'totalCount': 2},\n", 481 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-teaching'}},\n", 482 | " {'node': {'issues': {'totalCount': 5},\n", 483 | " 'name': 'jupyterhub-tutorial',\n", 484 | " 'pullRequests': {'totalCount': 0},\n", 485 | " 'url': 'https://github.com/jupyterhub/jupyterhub-tutorial'}},\n", 486 | " {'node': {'issues': {'totalCount': 2},\n", 487 | " 'name': 'jupyterhub-deploy-hpc',\n", 488 | " 'pullRequests': {'totalCount': 0},\n", 489 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-hpc'}},\n", 490 | " {'node': {'issues': {'totalCount': 4},\n", 491 | " 'name': 'systemdspawner',\n", 492 | " 'pullRequests': {'totalCount': 2},\n", 493 | " 'url': 'https://github.com/jupyterhub/systemdspawner'}},\n", 494 | " {'node': {'issues': {'totalCount': 5},\n", 495 | " 'name': 'wrapspawner',\n", 496 | " 'pullRequests': {'totalCount': 1},\n", 497 | " 'url': 'https://github.com/jupyterhub/wrapspawner'}},\n", 498 | " {'node': {'issues': {'totalCount': 4},\n", 499 | " 'name': 'jupyterlab-hub',\n", 500 | " 'pullRequests': {'totalCount': 0},\n", 501 | " 'url': 'https://github.com/jupyterhub/jupyterlab-hub'}},\n", 502 | " {'node': {'issues': {'totalCount': 2},\n", 503 | " 'name': 'nbserverproxy',\n", 504 | " 'pullRequests': {'totalCount': 1},\n", 505 | " 'url': 'https://github.com/jupyterhub/nbserverproxy'}},\n", 506 | " {'node': {'issues': {'totalCount': 3},\n", 507 | " 'name': 'jupyterhub-example-kerberos',\n", 508 | " 'pullRequests': {'totalCount': 0},\n", 509 | " 'url': 'https://github.com/jupyterhub/jupyterhub-example-kerberos'}},\n", 510 | " {'node': {'issues': {'totalCount': 10},\n", 511 | " 'name': 'hubshare',\n", 512 | " 'pullRequests': {'totalCount': 0},\n", 513 | " 'url': 'https://github.com/jupyterhub/hubshare'}},\n", 514 | " {'node': {'issues': {'totalCount': 4},\n", 515 | " 'name': 'nbrsessionproxy',\n", 516 | " 'pullRequests': {'totalCount': 2},\n", 517 | " 'url': 'https://github.com/jupyterhub/nbrsessionproxy'}},\n", 518 | " {'node': {'issues': {'totalCount': 0},\n", 519 | " 'name': 'tmpauthenticator',\n", 520 | " 'pullRequests': {'totalCount': 0},\n", 521 | " 'url': 'https://github.com/jupyterhub/tmpauthenticator'}},\n", 522 | " {'node': {'issues': {'totalCount': 90},\n", 523 | " 'name': 'zero-to-jupyterhub-k8s',\n", 524 | " 'pullRequests': {'totalCount': 3},\n", 525 | " 'url': 'https://github.com/jupyterhub/zero-to-jupyterhub-k8s'}},\n", 526 | " {'node': {'issues': {'totalCount': 3},\n", 527 | " 'name': 'helm-chart',\n", 528 | " 'pullRequests': {'totalCount': 0},\n", 529 | " 'url': 'https://github.com/jupyterhub/helm-chart'}},\n", 530 | " {'node': {'issues': {'totalCount': 75},\n", 531 | " 'name': 'binderhub',\n", 532 | " 'pullRequests': {'totalCount': 5},\n", 533 | " 'url': 'https://github.com/jupyterhub/binderhub'}},\n", 534 | " {'node': {'issues': {'totalCount': 24},\n", 535 | " 'name': 'mybinder.org-deploy',\n", 536 | " 'pullRequests': {'totalCount': 1},\n", 537 | " 'url': 'https://github.com/jupyterhub/mybinder.org-deploy'}},\n", 538 | " {'node': {'issues': {'totalCount': 9},\n", 539 | " 'name': 'binder',\n", 540 | " 'pullRequests': {'totalCount': 1},\n", 541 | " 'url': 'https://github.com/jupyterhub/binder'}},\n", 542 | " {'node': {'issues': {'totalCount': 0},\n", 543 | " 'name': 'nullauthenticator',\n", 544 | " 'pullRequests': {'totalCount': 0},\n", 545 | " 'url': 'https://github.com/jupyterhub/nullauthenticator'}},\n", 546 | " {'node': {'issues': {'totalCount': 2},\n", 547 | " 'name': 'team-compass',\n", 548 | " 'pullRequests': {'totalCount': 1},\n", 549 | " 'url': 'https://github.com/jupyterhub/team-compass'}}],\n", 550 | " 'totalCount': 27}}}}" 551 | ] 552 | }, 553 | "execution_count": 64, 554 | "metadata": {}, 555 | "output_type": "execute_result" 556 | } 557 | ], 558 | "source": [ 559 | "from operator import itemgetter\n", 560 | "\n", 561 | "def sort_nodes(names):\n", 562 | " \"\"\"Will sort a list of points.\"\"\"\n", 563 | " return sorted(names, reverse=True, key=itemgetter('node.name'))\n", 564 | "\n", 565 | "traverse_modify(data, 'data.organization.repositories.edges[].node{}', sort_nodes)" 566 | ] 567 | }, 568 | { 569 | "cell_type": "code", 570 | "execution_count": null, 571 | "metadata": {}, 572 | "outputs": [], 573 | "source": [ 574 | "issue_data = data['data']" 575 | ] 576 | }, 577 | { 578 | "cell_type": "code", 579 | "execution_count": null, 580 | "metadata": {}, 581 | "outputs": [], 582 | "source": [ 583 | "org = issue_data['organization']" 584 | ] 585 | }, 586 | { 587 | "cell_type": "code", 588 | "execution_count": null, 589 | "metadata": {}, 590 | "outputs": [], 591 | "source": [ 592 | "repos = org['repositories']" 593 | ] 594 | }, 595 | { 596 | "cell_type": "code", 597 | "execution_count": null, 598 | "metadata": {}, 599 | "outputs": [], 600 | "source": [ 601 | "edges = repos['edges']" 602 | ] 603 | }, 604 | { 605 | "cell_type": "code", 606 | "execution_count": null, 607 | "metadata": {}, 608 | "outputs": [], 609 | "source": [ 610 | "# edges[0]['node']" 611 | ] 612 | }, 613 | { 614 | "cell_type": "code", 615 | "execution_count": null, 616 | "metadata": {}, 617 | "outputs": [], 618 | "source": [ 619 | "# print(edges[0]['node']['name'] + '---' + edges[0]['node']['url'])" 620 | ] 621 | }, 622 | { 623 | "cell_type": "code", 624 | "execution_count": null, 625 | "metadata": {}, 626 | "outputs": [], 627 | "source": [ 628 | "# for edge in edges:\n", 629 | "# pprint.pprint(edge)" 630 | ] 631 | }, 632 | { 633 | "cell_type": "markdown", 634 | "metadata": {}, 635 | "source": [ 636 | "Bring into pandas" 637 | ] 638 | }, 639 | { 640 | "cell_type": "code", 641 | "execution_count": null, 642 | "metadata": {}, 643 | "outputs": [], 644 | "source": [ 645 | "import pandas as pd" 646 | ] 647 | }, 648 | { 649 | "cell_type": "code", 650 | "execution_count": null, 651 | "metadata": {}, 652 | "outputs": [], 653 | "source": [ 654 | "df = pd.DataFrame.from_records(edges)" 655 | ] 656 | }, 657 | { 658 | "cell_type": "code", 659 | "execution_count": null, 660 | "metadata": {}, 661 | "outputs": [], 662 | "source": [ 663 | "df.columns" 664 | ] 665 | }, 666 | { 667 | "cell_type": "code", 668 | "execution_count": null, 669 | "metadata": {}, 670 | "outputs": [], 671 | "source": [ 672 | "# df['node']" 673 | ] 674 | }, 675 | { 676 | "cell_type": "markdown", 677 | "metadata": {}, 678 | "source": [ 679 | "Generate basic report of total open issues" 680 | ] 681 | }, 682 | { 683 | "cell_type": "code", 684 | "execution_count": null, 685 | "metadata": {}, 686 | "outputs": [], 687 | "source": [ 688 | "print(f\"{'Repo':30} {'OpenIssues':11} {'OpenPRs':9} {'URL'}\")\n", 689 | "\n", 690 | "line = 26\n", 691 | "while line > 0:\n", 692 | " lineout = f\"{df['node'][line]['name']:30} {(df['node'][line]['issues']['totalCount']):8} {(df['node'][line]['pullRequests']['totalCount']):8} {df['node'][line]['url']}\"\n", 693 | " print(lineout)\n", 694 | " line -= 1" 695 | ] 696 | }, 697 | { 698 | "cell_type": "code", 699 | "execution_count": null, 700 | "metadata": {}, 701 | "outputs": [], 702 | "source": [ 703 | "df.dtypes" 704 | ] 705 | }, 706 | { 707 | "cell_type": "code", 708 | "execution_count": null, 709 | "metadata": {}, 710 | "outputs": [], 711 | "source": [ 712 | "df.head()" 713 | ] 714 | }, 715 | { 716 | "cell_type": "code", 717 | "execution_count": null, 718 | "metadata": {}, 719 | "outputs": [], 720 | "source": [ 721 | "df.index" 722 | ] 723 | }, 724 | { 725 | "cell_type": "code", 726 | "execution_count": null, 727 | "metadata": {}, 728 | "outputs": [], 729 | "source": [ 730 | "df.columns" 731 | ] 732 | }, 733 | { 734 | "cell_type": "code", 735 | "execution_count": null, 736 | "metadata": {}, 737 | "outputs": [], 738 | "source": [ 739 | "df.values" 740 | ] 741 | }, 742 | { 743 | "cell_type": "code", 744 | "execution_count": null, 745 | "metadata": {}, 746 | "outputs": [], 747 | "source": [ 748 | "df.sort_index" 749 | ] 750 | }, 751 | { 752 | "cell_type": "code", 753 | "execution_count": null, 754 | "metadata": {}, 755 | "outputs": [], 756 | "source": [ 757 | "# output data to a csv\n", 758 | "# df.to_csv('issue_data.csv')" 759 | ] 760 | } 761 | ], 762 | "metadata": { 763 | "kernelspec": { 764 | "display_name": "Python 3", 765 | "language": "python", 766 | "name": "python3" 767 | }, 768 | "language_info": { 769 | "codemirror_mode": { 770 | "name": "ipython", 771 | "version": 3 772 | }, 773 | "file_extension": ".py", 774 | "mimetype": "text/x-python", 775 | "name": "python", 776 | "nbconvert_exporter": "python", 777 | "pygments_lexer": "ipython3", 778 | "version": "3.6.3" 779 | } 780 | }, 781 | "nbformat": 4, 782 | "nbformat_minor": 2 783 | } 784 | -------------------------------------------------------------------------------- /notebooks/use-pyquery-ql-Copy1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Use pyquery-ql.py\n", 8 | "\n", 9 | "Send a graphql query to GitHub\n", 10 | "and pretty print output.\n", 11 | "\n", 12 | "Supports Python 3.6+" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 1, 18 | "metadata": {}, 19 | "outputs": [], 20 | "source": [ 21 | "import json\n", 22 | "import os\n", 23 | "import pprint\n", 24 | "\n", 25 | "import requests" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 2, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "# get api token and set authorization\n", 35 | "api_token = os.environ['GITHUB_API_TOKEN']\n", 36 | "headers = {'Authorization': f'token {api_token}'}" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 3, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "# set url to a graphql endpoint\n", 46 | "url = 'https://api.github.com/graphql'" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 4, 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "# add a json query\n", 56 | "query = \"\"\"\n", 57 | "{\n", 58 | " organization(login: \"jupyterhub\") {\n", 59 | " repositories(first: 30) {\n", 60 | " totalCount\n", 61 | " edges {\n", 62 | " node {\n", 63 | " name\n", 64 | " url\n", 65 | " issues(states: OPEN) {\n", 66 | " totalCount\n", 67 | " }\n", 68 | " pullRequests(states: OPEN) {\n", 69 | " totalCount\n", 70 | " }\n", 71 | " }\n", 72 | " }\n", 73 | " }\n", 74 | " }\n", 75 | "}\n", 76 | "\"\"\"" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": 5, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "# submit the request\n", 86 | "r = requests.post(url=url, json={'query': query}, headers=headers)" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 28, 92 | "metadata": {}, 93 | "outputs": [ 94 | { 95 | "data": { 96 | "text/plain": [ 97 | "'{\"data\":{\"organization\":{\"repositories\":{\"totalCount\":27,\"edges\":[{\"node\":{\"name\":\"jupyterhub\",\"url\":\"https://github.com/jupyterhub/jupyterhub\",\"issues\":{\"totalCount\":143},\"pullRequests\":{\"totalCount\":6}}},{\"node\":{\"name\":\"configurable-http-proxy\",\"url\":\"https://github.com/jupyterhub/configurable-http-proxy\",\"issues\":{\"totalCount\":10},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"oauthenticator\",\"url\":\"https://github.com/jupyterhub/oauthenticator\",\"issues\":{\"totalCount\":13},\"pullRequests\":{\"totalCount\":2}}},{\"node\":{\"name\":\"dockerspawner\",\"url\":\"https://github.com/jupyterhub/dockerspawner\",\"issues\":{\"totalCount\":22},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"sudospawner\",\"url\":\"https://github.com/jupyterhub/sudospawner\",\"issues\":{\"totalCount\":2},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"batchspawner\",\"url\":\"https://github.com/jupyterhub/batchspawner\",\"issues\":{\"totalCount\":12},\"pullRequests\":{\"totalCount\":5}}},{\"node\":{\"name\":\"kubespawner\",\"url\":\"https://github.com/jupyterhub/kubespawner\",\"issues\":{\"totalCount\":25},\"pullRequests\":{\"totalCount\":3}}},{\"node\":{\"name\":\"ldapauthenticator\",\"url\":\"https://github.com/jupyterhub/ldapauthenticator\",\"issues\":{\"totalCount\":18},\"pullRequests\":{\"totalCount\":2}}},{\"node\":{\"name\":\"jupyterhub-deploy-docker\",\"url\":\"https://github.com/jupyterhub/jupyterhub-deploy-docker\",\"issues\":{\"totalCount\":9},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"jupyterhub-deploy-teaching\",\"url\":\"https://github.com/jupyterhub/jupyterhub-deploy-teaching\",\"issues\":{\"totalCount\":13},\"pullRequests\":{\"totalCount\":2}}},{\"node\":{\"name\":\"jupyterhub-tutorial\",\"url\":\"https://github.com/jupyterhub/jupyterhub-tutorial\",\"issues\":{\"totalCount\":5},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"jupyterhub-deploy-hpc\",\"url\":\"https://github.com/jupyterhub/jupyterhub-deploy-hpc\",\"issues\":{\"totalCount\":2},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"systemdspawner\",\"url\":\"https://github.com/jupyterhub/systemdspawner\",\"issues\":{\"totalCount\":4},\"pullRequests\":{\"totalCount\":2}}},{\"node\":{\"name\":\"wrapspawner\",\"url\":\"https://github.com/jupyterhub/wrapspawner\",\"issues\":{\"totalCount\":5},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"jupyterlab-hub\",\"url\":\"https://github.com/jupyterhub/jupyterlab-hub\",\"issues\":{\"totalCount\":4},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"nbserverproxy\",\"url\":\"https://github.com/jupyterhub/nbserverproxy\",\"issues\":{\"totalCount\":2},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"jupyterhub-example-kerberos\",\"url\":\"https://github.com/jupyterhub/jupyterhub-example-kerberos\",\"issues\":{\"totalCount\":3},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"hubshare\",\"url\":\"https://github.com/jupyterhub/hubshare\",\"issues\":{\"totalCount\":10},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"nbrsessionproxy\",\"url\":\"https://github.com/jupyterhub/nbrsessionproxy\",\"issues\":{\"totalCount\":4},\"pullRequests\":{\"totalCount\":2}}},{\"node\":{\"name\":\"tmpauthenticator\",\"url\":\"https://github.com/jupyterhub/tmpauthenticator\",\"issues\":{\"totalCount\":0},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"zero-to-jupyterhub-k8s\",\"url\":\"https://github.com/jupyterhub/zero-to-jupyterhub-k8s\",\"issues\":{\"totalCount\":89},\"pullRequests\":{\"totalCount\":3}}},{\"node\":{\"name\":\"helm-chart\",\"url\":\"https://github.com/jupyterhub/helm-chart\",\"issues\":{\"totalCount\":3},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"binderhub\",\"url\":\"https://github.com/jupyterhub/binderhub\",\"issues\":{\"totalCount\":77},\"pullRequests\":{\"totalCount\":6}}},{\"node\":{\"name\":\"mybinder.org-deploy\",\"url\":\"https://github.com/jupyterhub/mybinder.org-deploy\",\"issues\":{\"totalCount\":24},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"binder\",\"url\":\"https://github.com/jupyterhub/binder\",\"issues\":{\"totalCount\":10},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"nullauthenticator\",\"url\":\"https://github.com/jupyterhub/nullauthenticator\",\"issues\":{\"totalCount\":0},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"team-compass\",\"url\":\"https://github.com/jupyterhub/team-compass\",\"issues\":{\"totalCount\":2},\"pullRequests\":{\"totalCount\":1}}}]}}}}'" 98 | ] 99 | }, 100 | "execution_count": 28, 101 | "metadata": {}, 102 | "output_type": "execute_result" 103 | } 104 | ], 105 | "source": [ 106 | "r.text" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 27, 112 | "metadata": {}, 113 | "outputs": [ 114 | { 115 | "data": { 116 | "text/plain": [ 117 | "{'data': {'organization': {'repositories': {'edges': [{'node': {'issues': {'totalCount': 143},\n", 118 | " 'name': 'jupyterhub',\n", 119 | " 'pullRequests': {'totalCount': 6},\n", 120 | " 'url': 'https://github.com/jupyterhub/jupyterhub'}},\n", 121 | " {'node': {'issues': {'totalCount': 10},\n", 122 | " 'name': 'configurable-http-proxy',\n", 123 | " 'pullRequests': {'totalCount': 0},\n", 124 | " 'url': 'https://github.com/jupyterhub/configurable-http-proxy'}},\n", 125 | " {'node': {'issues': {'totalCount': 13},\n", 126 | " 'name': 'oauthenticator',\n", 127 | " 'pullRequests': {'totalCount': 2},\n", 128 | " 'url': 'https://github.com/jupyterhub/oauthenticator'}},\n", 129 | " {'node': {'issues': {'totalCount': 22},\n", 130 | " 'name': 'dockerspawner',\n", 131 | " 'pullRequests': {'totalCount': 1},\n", 132 | " 'url': 'https://github.com/jupyterhub/dockerspawner'}},\n", 133 | " {'node': {'issues': {'totalCount': 2},\n", 134 | " 'name': 'sudospawner',\n", 135 | " 'pullRequests': {'totalCount': 0},\n", 136 | " 'url': 'https://github.com/jupyterhub/sudospawner'}},\n", 137 | " {'node': {'issues': {'totalCount': 12},\n", 138 | " 'name': 'batchspawner',\n", 139 | " 'pullRequests': {'totalCount': 5},\n", 140 | " 'url': 'https://github.com/jupyterhub/batchspawner'}},\n", 141 | " {'node': {'issues': {'totalCount': 25},\n", 142 | " 'name': 'kubespawner',\n", 143 | " 'pullRequests': {'totalCount': 3},\n", 144 | " 'url': 'https://github.com/jupyterhub/kubespawner'}},\n", 145 | " {'node': {'issues': {'totalCount': 18},\n", 146 | " 'name': 'ldapauthenticator',\n", 147 | " 'pullRequests': {'totalCount': 2},\n", 148 | " 'url': 'https://github.com/jupyterhub/ldapauthenticator'}},\n", 149 | " {'node': {'issues': {'totalCount': 9},\n", 150 | " 'name': 'jupyterhub-deploy-docker',\n", 151 | " 'pullRequests': {'totalCount': 1},\n", 152 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-docker'}},\n", 153 | " {'node': {'issues': {'totalCount': 13},\n", 154 | " 'name': 'jupyterhub-deploy-teaching',\n", 155 | " 'pullRequests': {'totalCount': 2},\n", 156 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-teaching'}},\n", 157 | " {'node': {'issues': {'totalCount': 5},\n", 158 | " 'name': 'jupyterhub-tutorial',\n", 159 | " 'pullRequests': {'totalCount': 0},\n", 160 | " 'url': 'https://github.com/jupyterhub/jupyterhub-tutorial'}},\n", 161 | " {'node': {'issues': {'totalCount': 2},\n", 162 | " 'name': 'jupyterhub-deploy-hpc',\n", 163 | " 'pullRequests': {'totalCount': 0},\n", 164 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-hpc'}},\n", 165 | " {'node': {'issues': {'totalCount': 4},\n", 166 | " 'name': 'systemdspawner',\n", 167 | " 'pullRequests': {'totalCount': 2},\n", 168 | " 'url': 'https://github.com/jupyterhub/systemdspawner'}},\n", 169 | " {'node': {'issues': {'totalCount': 5},\n", 170 | " 'name': 'wrapspawner',\n", 171 | " 'pullRequests': {'totalCount': 1},\n", 172 | " 'url': 'https://github.com/jupyterhub/wrapspawner'}},\n", 173 | " {'node': {'issues': {'totalCount': 4},\n", 174 | " 'name': 'jupyterlab-hub',\n", 175 | " 'pullRequests': {'totalCount': 0},\n", 176 | " 'url': 'https://github.com/jupyterhub/jupyterlab-hub'}},\n", 177 | " {'node': {'issues': {'totalCount': 2},\n", 178 | " 'name': 'nbserverproxy',\n", 179 | " 'pullRequests': {'totalCount': 1},\n", 180 | " 'url': 'https://github.com/jupyterhub/nbserverproxy'}},\n", 181 | " {'node': {'issues': {'totalCount': 3},\n", 182 | " 'name': 'jupyterhub-example-kerberos',\n", 183 | " 'pullRequests': {'totalCount': 0},\n", 184 | " 'url': 'https://github.com/jupyterhub/jupyterhub-example-kerberos'}},\n", 185 | " {'node': {'issues': {'totalCount': 10},\n", 186 | " 'name': 'hubshare',\n", 187 | " 'pullRequests': {'totalCount': 0},\n", 188 | " 'url': 'https://github.com/jupyterhub/hubshare'}},\n", 189 | " {'node': {'issues': {'totalCount': 4},\n", 190 | " 'name': 'nbrsessionproxy',\n", 191 | " 'pullRequests': {'totalCount': 2},\n", 192 | " 'url': 'https://github.com/jupyterhub/nbrsessionproxy'}},\n", 193 | " {'node': {'issues': {'totalCount': 0},\n", 194 | " 'name': 'tmpauthenticator',\n", 195 | " 'pullRequests': {'totalCount': 0},\n", 196 | " 'url': 'https://github.com/jupyterhub/tmpauthenticator'}},\n", 197 | " {'node': {'issues': {'totalCount': 89},\n", 198 | " 'name': 'zero-to-jupyterhub-k8s',\n", 199 | " 'pullRequests': {'totalCount': 3},\n", 200 | " 'url': 'https://github.com/jupyterhub/zero-to-jupyterhub-k8s'}},\n", 201 | " {'node': {'issues': {'totalCount': 3},\n", 202 | " 'name': 'helm-chart',\n", 203 | " 'pullRequests': {'totalCount': 0},\n", 204 | " 'url': 'https://github.com/jupyterhub/helm-chart'}},\n", 205 | " {'node': {'issues': {'totalCount': 77},\n", 206 | " 'name': 'binderhub',\n", 207 | " 'pullRequests': {'totalCount': 6},\n", 208 | " 'url': 'https://github.com/jupyterhub/binderhub'}},\n", 209 | " {'node': {'issues': {'totalCount': 24},\n", 210 | " 'name': 'mybinder.org-deploy',\n", 211 | " 'pullRequests': {'totalCount': 1},\n", 212 | " 'url': 'https://github.com/jupyterhub/mybinder.org-deploy'}},\n", 213 | " {'node': {'issues': {'totalCount': 10},\n", 214 | " 'name': 'binder',\n", 215 | " 'pullRequests': {'totalCount': 1},\n", 216 | " 'url': 'https://github.com/jupyterhub/binder'}},\n", 217 | " {'node': {'issues': {'totalCount': 0},\n", 218 | " 'name': 'nullauthenticator',\n", 219 | " 'pullRequests': {'totalCount': 0},\n", 220 | " 'url': 'https://github.com/jupyterhub/nullauthenticator'}},\n", 221 | " {'node': {'issues': {'totalCount': 2},\n", 222 | " 'name': 'team-compass',\n", 223 | " 'pullRequests': {'totalCount': 1},\n", 224 | " 'url': 'https://github.com/jupyterhub/team-compass'}}],\n", 225 | " 'totalCount': 27}}}}" 226 | ] 227 | }, 228 | "execution_count": 27, 229 | "metadata": {}, 230 | "output_type": "execute_result" 231 | } 232 | ], 233 | "source": [ 234 | "data = json.loads(r.text)" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": 1, 240 | "metadata": {}, 241 | "outputs": [], 242 | "source": [ 243 | "import requests\n", 244 | "import json\n", 245 | "from itertools import zip_longest\n", 246 | "import pandas as pd\n", 247 | "import numpy as np" 248 | ] 249 | }, 250 | { 251 | "cell_type": "code", 252 | "execution_count": 2, 253 | "metadata": {}, 254 | "outputs": [], 255 | "source": [ 256 | "#This is a little helper funciton that will take x of something, and let you chunk through a subset\n", 257 | "#Note that 'n' is the chunk size\n", 258 | "def _grouper(n, iterable, fillvalue=None):\n", 259 | " args = [iter(iterable)] * n\n", 260 | " return zip_longest(fillvalue=fillvalue, *args)" 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "execution_count": 22, 266 | "metadata": {}, 267 | "outputs": [], 268 | "source": [ 269 | "#Since you want a report by month, lets just create a function to get all the chat IDs for a two digit month and 4 digit year\n", 270 | "\n", 271 | "def get_ids(data):\n", 272 | " all_ids = []\n", 273 | " response = json.loads(data)\n", 274 | " for id in response['data']:\n", 275 | " all_ids.append('id')\n", 276 | " return all_ids" 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": 62, 282 | "metadata": {}, 283 | "outputs": [ 284 | { 285 | "name": "stdout", 286 | "output_type": "stream", 287 | "text": [ 288 | "organization\n", 289 | "repositories\n", 290 | "['organization', 'repositories']\n", 291 | "totalCount\n", 292 | "edges\n", 293 | "['organization', 'repositories', 'totalCount', 'edges']\n" 294 | ] 295 | }, 296 | { 297 | "ename": "KeyError", 298 | "evalue": "'edges'", 299 | "output_type": "error", 300 | "traceback": [ 301 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 302 | "\u001b[0;31mKeyError\u001b[0m Traceback (most recent call last)", 303 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 15\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mall_ids\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'data'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'edges'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", 304 | "\u001b[0;31mKeyError\u001b[0m: 'edges'" 305 | ] 306 | } 307 | ], 308 | "source": [ 309 | "all_ids = []\n", 310 | "for id in data['data']:\n", 311 | " print(id)\n", 312 | " all_ids.append(id)\n", 313 | " \n", 314 | " for id in data['data'][all_ids[0]]:\n", 315 | " print(id)\n", 316 | " all_ids.append(id)\n", 317 | " print(all_ids)\n", 318 | " \n", 319 | " for id in data['data'][all_ids[0]][all_ids[1]]:\n", 320 | " print(id)\n", 321 | " all_ids.append(id)\n", 322 | " \n", 323 | " print(all_ids)\n", 324 | " print(data['data']['edges'])\n" 325 | ] 326 | }, 327 | { 328 | "cell_type": "code", 329 | "execution_count": 4, 330 | "metadata": {}, 331 | "outputs": [], 332 | "source": [ 333 | "def build_csv_data(queried_id_list):\n", 334 | " all_chats_for_csv = []\n", 335 | " for chunk in _grouper(50, queried_id_list):\n", 336 | " filtered = [id for id in chunk if id is not None] #Only needed for last set of 50 that will have some sort of remainder filled with 'None' values by the grouper() function\n", 337 | " url = \"{0}/chats\".format(config['base_path'])\n", 338 | " stringified_ids = \",\".join(filtered)\n", 339 | " params = {\"ids\":stringified_ids}\n", 340 | " r = s.get(url, params=params)\n", 341 | " if r.status_code == 200:\n", 342 | " response = json.loads(r.text)\n", 343 | " docs = response['docs']\n", 344 | " for k in docs:\n", 345 | " doc_obj = docs[k] #This is the 'doc' object. Note that some keys may not exist for all items. Duration does not exist for offline messages for example\n", 346 | " all_chats_for_csv.append(doc_obj)\n", 347 | " else:\n", 348 | " print(\"error getting bulk chats\")\n", 349 | " return all_chats_for_csv" 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": 7, 355 | "metadata": {}, 356 | "outputs": [], 357 | "source": [ 358 | "#grab the ids for the specified range NOTE!! This is where you can change the year/month for the date range.\n", 359 | "queried_id_list = get_chat_ids_for_month(2016, 10)\n", 360 | "\n", 361 | "#turn those into raw objects\n", 362 | "csv_obj_data = build_csv_data(queried_id_list)\n", 363 | "\n", 364 | "#create a list to hold our rows\n", 365 | "all_rows_as_obj = []\n", 366 | "\n", 367 | "#turn the raw objects into nice and tasty ones that pandas can digest\n", 368 | "for record in csv_obj_data:\n", 369 | " try:\n", 370 | " response_time = record.get('response_time', {})\n", 371 | " count = record.get('count',{})\n", 372 | " csv_obj = {\n", 373 | " \"id\": record['id'],\n", 374 | " \"agent\": \",\".join(record.get('agent_names', \"N/A\")),\n", 375 | " \"visitor\": record['visitor']['name'],\n", 376 | " \"department\": record['department_name'],\n", 377 | " \"url\": record.get('webpath', \"N/A\"),\n", 378 | " \"missed\": record.get('missed', \"N/A\"),\n", 379 | " \"resp_first\": response_time.get('first', []),\n", 380 | " \"resp_max\": response_time.get('max', []),\n", 381 | " \"resp_avg\": response_time.get('avg', []),\n", 382 | " \"start timestamp\": record['session']['start_date'],\n", 383 | " \"end timestamp\": record['session']['end_date'],\n", 384 | " \"total messages\": count.get('total', 0),\n", 385 | " \"Agent Msg Count\": count.get('agent', 0),\n", 386 | " \"Visitor Msg Count\": count.get('visitor', 0),\n", 387 | " \"rating\": record.get('rating',[]),\n", 388 | " \"ticket_id\": record['zendesk_ticket_id']\n", 389 | " }\n", 390 | " except KeyError as e:\n", 391 | " print(e)\n", 392 | " print(\"key error occured for record with id: {}\".format(record['id']))\n", 393 | " print(record)\n", 394 | " pass\n", 395 | " all_rows_as_obj.append(csv_obj)\n", 396 | "\n", 397 | "#with prettier objects our column names can just come from a call to the first object's keys\n", 398 | "col_keys = all_rows_as_obj[0].keys()\n", 399 | "\n", 400 | "#create the dataframe, list comprehension creates an array of rows based on the column names we got\n", 401 | "#Let pandas do the hard part with csvs\n", 402 | "csv_frame = pd.DataFrame([[i[j] for j in col_keys] for i in all_rows_as_obj], columns=col_keys)\n", 403 | "\n", 404 | "#coerce ticket_id to be int not float\n", 405 | "csv_frame['ticket_id'] = csv_frame['ticket_id'].fillna(0).astype(np.int64)\n", 406 | "\n", 407 | "#fill blank values with the string 'N/A'\n", 408 | "csv_frame = csv_frame.fillna(\"N/A\")\n", 409 | "\n", 410 | "#output to a file\n", 411 | "csv_frame.to_csv('october_stats.csv')" 412 | ] 413 | }, 414 | { 415 | "cell_type": "markdown", 416 | "metadata": {}, 417 | "source": [ 418 | "Walk the JSON response contents" 419 | ] 420 | }, 421 | { 422 | "cell_type": "code", 423 | "execution_count": 8, 424 | "metadata": {}, 425 | "outputs": [], 426 | "source": [ 427 | "issue_data = data['data']" 428 | ] 429 | }, 430 | { 431 | "cell_type": "code", 432 | "execution_count": 9, 433 | "metadata": {}, 434 | "outputs": [], 435 | "source": [ 436 | "org = issue_data['organization']" 437 | ] 438 | }, 439 | { 440 | "cell_type": "code", 441 | "execution_count": 10, 442 | "metadata": {}, 443 | "outputs": [], 444 | "source": [ 445 | "repos = org['repositories']" 446 | ] 447 | }, 448 | { 449 | "cell_type": "code", 450 | "execution_count": 11, 451 | "metadata": {}, 452 | "outputs": [], 453 | "source": [ 454 | "edges = repos['edges']" 455 | ] 456 | }, 457 | { 458 | "cell_type": "code", 459 | "execution_count": 12, 460 | "metadata": {}, 461 | "outputs": [], 462 | "source": [ 463 | "# edges[0]['node']" 464 | ] 465 | }, 466 | { 467 | "cell_type": "code", 468 | "execution_count": 13, 469 | "metadata": {}, 470 | "outputs": [], 471 | "source": [ 472 | "# print(edges[0]['node']['name'] + '---' + edges[0]['node']['url'])" 473 | ] 474 | }, 475 | { 476 | "cell_type": "code", 477 | "execution_count": 14, 478 | "metadata": {}, 479 | "outputs": [], 480 | "source": [ 481 | "# for edge in edges:\n", 482 | "# pprint.pprint(edge)" 483 | ] 484 | }, 485 | { 486 | "cell_type": "markdown", 487 | "metadata": {}, 488 | "source": [ 489 | "Bring into pandas" 490 | ] 491 | }, 492 | { 493 | "cell_type": "code", 494 | "execution_count": 15, 495 | "metadata": {}, 496 | "outputs": [], 497 | "source": [ 498 | "import pandas as pd" 499 | ] 500 | }, 501 | { 502 | "cell_type": "code", 503 | "execution_count": 16, 504 | "metadata": {}, 505 | "outputs": [], 506 | "source": [ 507 | "df = pd.DataFrame.from_records(edges)" 508 | ] 509 | }, 510 | { 511 | "cell_type": "code", 512 | "execution_count": 17, 513 | "metadata": {}, 514 | "outputs": [ 515 | { 516 | "data": { 517 | "text/plain": [ 518 | "Index(['node'], dtype='object')" 519 | ] 520 | }, 521 | "execution_count": 17, 522 | "metadata": {}, 523 | "output_type": "execute_result" 524 | } 525 | ], 526 | "source": [ 527 | "df.columns" 528 | ] 529 | }, 530 | { 531 | "cell_type": "code", 532 | "execution_count": 18, 533 | "metadata": {}, 534 | "outputs": [], 535 | "source": [ 536 | "# df['node']" 537 | ] 538 | }, 539 | { 540 | "cell_type": "markdown", 541 | "metadata": {}, 542 | "source": [ 543 | "Generate basic report of total open issues" 544 | ] 545 | }, 546 | { 547 | "cell_type": "code", 548 | "execution_count": 19, 549 | "metadata": {}, 550 | "outputs": [ 551 | { 552 | "name": "stdout", 553 | "output_type": "stream", 554 | "text": [ 555 | "Repo OpenIssues OpenPRs URL\n", 556 | "team-compass 2 1 https://github.com/jupyterhub/team-compass\n", 557 | "nullauthenticator 0 0 https://github.com/jupyterhub/nullauthenticator\n", 558 | "binder 10 1 https://github.com/jupyterhub/binder\n", 559 | "mybinder.org-deploy 24 1 https://github.com/jupyterhub/mybinder.org-deploy\n", 560 | "binderhub 77 6 https://github.com/jupyterhub/binderhub\n", 561 | "helm-chart 3 0 https://github.com/jupyterhub/helm-chart\n", 562 | "zero-to-jupyterhub-k8s 89 3 https://github.com/jupyterhub/zero-to-jupyterhub-k8s\n", 563 | "tmpauthenticator 0 0 https://github.com/jupyterhub/tmpauthenticator\n", 564 | "nbrsessionproxy 4 2 https://github.com/jupyterhub/nbrsessionproxy\n", 565 | "hubshare 10 0 https://github.com/jupyterhub/hubshare\n", 566 | "jupyterhub-example-kerberos 3 0 https://github.com/jupyterhub/jupyterhub-example-kerberos\n", 567 | "nbserverproxy 2 1 https://github.com/jupyterhub/nbserverproxy\n", 568 | "jupyterlab-hub 4 0 https://github.com/jupyterhub/jupyterlab-hub\n", 569 | "wrapspawner 5 1 https://github.com/jupyterhub/wrapspawner\n", 570 | "systemdspawner 4 2 https://github.com/jupyterhub/systemdspawner\n", 571 | "jupyterhub-deploy-hpc 2 0 https://github.com/jupyterhub/jupyterhub-deploy-hpc\n", 572 | "jupyterhub-tutorial 5 0 https://github.com/jupyterhub/jupyterhub-tutorial\n", 573 | "jupyterhub-deploy-teaching 13 2 https://github.com/jupyterhub/jupyterhub-deploy-teaching\n", 574 | "jupyterhub-deploy-docker 9 1 https://github.com/jupyterhub/jupyterhub-deploy-docker\n", 575 | "ldapauthenticator 18 2 https://github.com/jupyterhub/ldapauthenticator\n", 576 | "kubespawner 25 3 https://github.com/jupyterhub/kubespawner\n", 577 | "batchspawner 12 5 https://github.com/jupyterhub/batchspawner\n", 578 | "sudospawner 2 0 https://github.com/jupyterhub/sudospawner\n", 579 | "dockerspawner 22 1 https://github.com/jupyterhub/dockerspawner\n", 580 | "oauthenticator 13 2 https://github.com/jupyterhub/oauthenticator\n", 581 | "configurable-http-proxy 10 0 https://github.com/jupyterhub/configurable-http-proxy\n" 582 | ] 583 | } 584 | ], 585 | "source": [ 586 | "print(f\"{'Repo':30} {'OpenIssues':11} {'OpenPRs':9} {'URL'}\")\n", 587 | "\n", 588 | "line = 26\n", 589 | "while line > 0:\n", 590 | " lineout = f\"{df['node'][line]['name']:30} {(df['node'][line]['issues']['totalCount']):8} {(df['node'][line]['pullRequests']['totalCount']):8} {df['node'][line]['url']}\"\n", 591 | " print(lineout)\n", 592 | " line -= 1" 593 | ] 594 | }, 595 | { 596 | "cell_type": "code", 597 | "execution_count": 20, 598 | "metadata": {}, 599 | "outputs": [ 600 | { 601 | "data": { 602 | "text/plain": [ 603 | "node object\n", 604 | "dtype: object" 605 | ] 606 | }, 607 | "execution_count": 20, 608 | "metadata": {}, 609 | "output_type": "execute_result" 610 | } 611 | ], 612 | "source": [ 613 | "df.dtypes" 614 | ] 615 | }, 616 | { 617 | "cell_type": "code", 618 | "execution_count": 21, 619 | "metadata": {}, 620 | "outputs": [ 621 | { 622 | "data": { 623 | "text/html": [ 624 | "
\n", 625 | "\n", 638 | "\n", 639 | " \n", 640 | " \n", 641 | " \n", 642 | " \n", 643 | " \n", 644 | " \n", 645 | " \n", 646 | " \n", 647 | " \n", 648 | " \n", 649 | " \n", 650 | " \n", 651 | " \n", 652 | " \n", 653 | " \n", 654 | " \n", 655 | " \n", 656 | " \n", 657 | " \n", 658 | " \n", 659 | " \n", 660 | " \n", 661 | " \n", 662 | " \n", 663 | " \n", 664 | " \n", 665 | " \n", 666 | " \n", 667 | "
node
0{'name': 'jupyterhub', 'url': 'https://github....
1{'name': 'configurable-http-proxy', 'url': 'ht...
2{'name': 'oauthenticator', 'url': 'https://git...
3{'name': 'dockerspawner', 'url': 'https://gith...
4{'name': 'sudospawner', 'url': 'https://github...
\n", 668 | "
" 669 | ], 670 | "text/plain": [ 671 | " node\n", 672 | "0 {'name': 'jupyterhub', 'url': 'https://github....\n", 673 | "1 {'name': 'configurable-http-proxy', 'url': 'ht...\n", 674 | "2 {'name': 'oauthenticator', 'url': 'https://git...\n", 675 | "3 {'name': 'dockerspawner', 'url': 'https://gith...\n", 676 | "4 {'name': 'sudospawner', 'url': 'https://github..." 677 | ] 678 | }, 679 | "execution_count": 21, 680 | "metadata": {}, 681 | "output_type": "execute_result" 682 | } 683 | ], 684 | "source": [ 685 | "df.head()" 686 | ] 687 | }, 688 | { 689 | "cell_type": "code", 690 | "execution_count": 22, 691 | "metadata": {}, 692 | "outputs": [ 693 | { 694 | "data": { 695 | "text/plain": [ 696 | "RangeIndex(start=0, stop=27, step=1)" 697 | ] 698 | }, 699 | "execution_count": 22, 700 | "metadata": {}, 701 | "output_type": "execute_result" 702 | } 703 | ], 704 | "source": [ 705 | "df.index" 706 | ] 707 | }, 708 | { 709 | "cell_type": "code", 710 | "execution_count": 23, 711 | "metadata": {}, 712 | "outputs": [ 713 | { 714 | "data": { 715 | "text/plain": [ 716 | "Index(['node'], dtype='object')" 717 | ] 718 | }, 719 | "execution_count": 23, 720 | "metadata": {}, 721 | "output_type": "execute_result" 722 | } 723 | ], 724 | "source": [ 725 | "df.columns" 726 | ] 727 | }, 728 | { 729 | "cell_type": "code", 730 | "execution_count": 24, 731 | "metadata": {}, 732 | "outputs": [ 733 | { 734 | "data": { 735 | "text/plain": [ 736 | "array([[ {'name': 'jupyterhub', 'url': 'https://github.com/jupyterhub/jupyterhub', 'issues': {'totalCount': 143}, 'pullRequests': {'totalCount': 6}}],\n", 737 | " [ {'name': 'configurable-http-proxy', 'url': 'https://github.com/jupyterhub/configurable-http-proxy', 'issues': {'totalCount': 10}, 'pullRequests': {'totalCount': 0}}],\n", 738 | " [ {'name': 'oauthenticator', 'url': 'https://github.com/jupyterhub/oauthenticator', 'issues': {'totalCount': 13}, 'pullRequests': {'totalCount': 2}}],\n", 739 | " [ {'name': 'dockerspawner', 'url': 'https://github.com/jupyterhub/dockerspawner', 'issues': {'totalCount': 22}, 'pullRequests': {'totalCount': 1}}],\n", 740 | " [ {'name': 'sudospawner', 'url': 'https://github.com/jupyterhub/sudospawner', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 0}}],\n", 741 | " [ {'name': 'batchspawner', 'url': 'https://github.com/jupyterhub/batchspawner', 'issues': {'totalCount': 12}, 'pullRequests': {'totalCount': 5}}],\n", 742 | " [ {'name': 'kubespawner', 'url': 'https://github.com/jupyterhub/kubespawner', 'issues': {'totalCount': 25}, 'pullRequests': {'totalCount': 3}}],\n", 743 | " [ {'name': 'ldapauthenticator', 'url': 'https://github.com/jupyterhub/ldapauthenticator', 'issues': {'totalCount': 18}, 'pullRequests': {'totalCount': 2}}],\n", 744 | " [ {'name': 'jupyterhub-deploy-docker', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-docker', 'issues': {'totalCount': 9}, 'pullRequests': {'totalCount': 1}}],\n", 745 | " [ {'name': 'jupyterhub-deploy-teaching', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-teaching', 'issues': {'totalCount': 13}, 'pullRequests': {'totalCount': 2}}],\n", 746 | " [ {'name': 'jupyterhub-tutorial', 'url': 'https://github.com/jupyterhub/jupyterhub-tutorial', 'issues': {'totalCount': 5}, 'pullRequests': {'totalCount': 0}}],\n", 747 | " [ {'name': 'jupyterhub-deploy-hpc', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-hpc', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 0}}],\n", 748 | " [ {'name': 'systemdspawner', 'url': 'https://github.com/jupyterhub/systemdspawner', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 2}}],\n", 749 | " [ {'name': 'wrapspawner', 'url': 'https://github.com/jupyterhub/wrapspawner', 'issues': {'totalCount': 5}, 'pullRequests': {'totalCount': 1}}],\n", 750 | " [ {'name': 'jupyterlab-hub', 'url': 'https://github.com/jupyterhub/jupyterlab-hub', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 0}}],\n", 751 | " [ {'name': 'nbserverproxy', 'url': 'https://github.com/jupyterhub/nbserverproxy', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 1}}],\n", 752 | " [ {'name': 'jupyterhub-example-kerberos', 'url': 'https://github.com/jupyterhub/jupyterhub-example-kerberos', 'issues': {'totalCount': 3}, 'pullRequests': {'totalCount': 0}}],\n", 753 | " [ {'name': 'hubshare', 'url': 'https://github.com/jupyterhub/hubshare', 'issues': {'totalCount': 10}, 'pullRequests': {'totalCount': 0}}],\n", 754 | " [ {'name': 'nbrsessionproxy', 'url': 'https://github.com/jupyterhub/nbrsessionproxy', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 2}}],\n", 755 | " [ {'name': 'tmpauthenticator', 'url': 'https://github.com/jupyterhub/tmpauthenticator', 'issues': {'totalCount': 0}, 'pullRequests': {'totalCount': 0}}],\n", 756 | " [ {'name': 'zero-to-jupyterhub-k8s', 'url': 'https://github.com/jupyterhub/zero-to-jupyterhub-k8s', 'issues': {'totalCount': 89}, 'pullRequests': {'totalCount': 3}}],\n", 757 | " [ {'name': 'helm-chart', 'url': 'https://github.com/jupyterhub/helm-chart', 'issues': {'totalCount': 3}, 'pullRequests': {'totalCount': 0}}],\n", 758 | " [ {'name': 'binderhub', 'url': 'https://github.com/jupyterhub/binderhub', 'issues': {'totalCount': 77}, 'pullRequests': {'totalCount': 6}}],\n", 759 | " [ {'name': 'mybinder.org-deploy', 'url': 'https://github.com/jupyterhub/mybinder.org-deploy', 'issues': {'totalCount': 24}, 'pullRequests': {'totalCount': 1}}],\n", 760 | " [ {'name': 'binder', 'url': 'https://github.com/jupyterhub/binder', 'issues': {'totalCount': 10}, 'pullRequests': {'totalCount': 1}}],\n", 761 | " [ {'name': 'nullauthenticator', 'url': 'https://github.com/jupyterhub/nullauthenticator', 'issues': {'totalCount': 0}, 'pullRequests': {'totalCount': 0}}],\n", 762 | " [ {'name': 'team-compass', 'url': 'https://github.com/jupyterhub/team-compass', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 1}}]], dtype=object)" 763 | ] 764 | }, 765 | "execution_count": 24, 766 | "metadata": {}, 767 | "output_type": "execute_result" 768 | } 769 | ], 770 | "source": [ 771 | "df.values" 772 | ] 773 | }, 774 | { 775 | "cell_type": "code", 776 | "execution_count": 25, 777 | "metadata": {}, 778 | "outputs": [ 779 | { 780 | "data": { 781 | "text/plain": [ 782 | "" 810 | ] 811 | }, 812 | "execution_count": 25, 813 | "metadata": {}, 814 | "output_type": "execute_result" 815 | } 816 | ], 817 | "source": [ 818 | "df.sort_index" 819 | ] 820 | }, 821 | { 822 | "cell_type": "code", 823 | "execution_count": 26, 824 | "metadata": {}, 825 | "outputs": [], 826 | "source": [ 827 | "# output data to a csv\n", 828 | "# df.to_csv('issue_data.csv')" 829 | ] 830 | } 831 | ], 832 | "metadata": { 833 | "kernelspec": { 834 | "display_name": "Python 3", 835 | "language": "python", 836 | "name": "python3" 837 | }, 838 | "language_info": { 839 | "codemirror_mode": { 840 | "name": "ipython", 841 | "version": 3 842 | }, 843 | "file_extension": ".py", 844 | "mimetype": "text/x-python", 845 | "name": "python", 846 | "nbconvert_exporter": "python", 847 | "pygments_lexer": "ipython3", 848 | "version": "3.6.3" 849 | } 850 | }, 851 | "nbformat": 4, 852 | "nbformat_minor": 2 853 | } 854 | -------------------------------------------------------------------------------- /notebooks/use-pyquery-ql-simplewalk.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Use pyquery-ql.py\n", 8 | "\n", 9 | "Send a graphql query to GitHub\n", 10 | "and pretty print output.\n", 11 | "\n", 12 | "Supports Python 3.6+" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 1, 18 | "metadata": {}, 19 | "outputs": [], 20 | "source": [ 21 | "import json\n", 22 | "import os\n", 23 | "import pprint\n", 24 | "\n", 25 | "import requests" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 2, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "# get api token and set authorization\n", 35 | "api_token = os.environ['GITHUB_API_TOKEN']\n", 36 | "headers = {'Authorization': f'token {api_token}'}" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 3, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "# set url to a graphql endpoint\n", 46 | "url = 'https://api.github.com/graphql'" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 4, 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "# add a json query\n", 56 | "query = \"\"\"\n", 57 | "{\n", 58 | " organization(login: \"jupyterhub\") {\n", 59 | " repositories(first: 30) {\n", 60 | " totalCount\n", 61 | " edges {\n", 62 | " node {\n", 63 | " name\n", 64 | " url\n", 65 | " issues(states: OPEN) {\n", 66 | " totalCount\n", 67 | " }\n", 68 | " pullRequests(states: OPEN) {\n", 69 | " totalCount\n", 70 | " }\n", 71 | " }\n", 72 | " }\n", 73 | " }\n", 74 | " }\n", 75 | "}\n", 76 | "\"\"\"" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": 5, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "# submit the request\n", 86 | "r = requests.post(url=url, json={'query': query}, headers=headers)" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 6, 92 | "metadata": {}, 93 | "outputs": [], 94 | "source": [ 95 | "with open('data.json', 'w') as f:\n", 96 | " json.dump(r.json(), f)" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": 7, 102 | "metadata": {}, 103 | "outputs": [], 104 | "source": [ 105 | "my_results = r.json()" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": 8, 111 | "metadata": {}, 112 | "outputs": [], 113 | "source": [ 114 | "my_list_nodes = my_results['data']['organization']['repositories']['edges']" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 9, 120 | "metadata": {}, 121 | "outputs": [], 122 | "source": [ 123 | "unpacked = []\n", 124 | "for node in my_list_nodes:\n", 125 | " unpacked.append(node['node'])" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": 10, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "headers = ['name', 'url', 'issues', 'prs']\n", 135 | "\n", 136 | "rows = []\n", 137 | "for obj in unpacked:\n", 138 | " new_dict = {'name':obj['name'], 'url':obj['url'], 'issues':obj['issues']['totalCount'], 'prs':obj['pullRequests']['totalCount']}\n", 139 | " rows.append(new_dict)" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": 11, 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "import csv\n", 149 | "\n", 150 | "with open('mydata.csv', 'w') as f:\n", 151 | " f_csv = csv.DictWriter(f, headers)\n", 152 | " f_csv.writeheader()\n", 153 | " f_csv.writerows(rows)" 154 | ] 155 | }, 156 | { 157 | "cell_type": "code", 158 | "execution_count": 12, 159 | "metadata": {}, 160 | "outputs": [ 161 | { 162 | "name": "stdout", 163 | "output_type": "stream", 164 | "text": [ 165 | "[{'issues': 141,\n", 166 | " 'name': 'jupyterhub',\n", 167 | " 'prs': 6,\n", 168 | " 'url': 'https://github.com/jupyterhub/jupyterhub'},\n", 169 | " {'issues': 11,\n", 170 | " 'name': 'configurable-http-proxy',\n", 171 | " 'prs': 0,\n", 172 | " 'url': 'https://github.com/jupyterhub/configurable-http-proxy'},\n", 173 | " {'issues': 13,\n", 174 | " 'name': 'oauthenticator',\n", 175 | " 'prs': 2,\n", 176 | " 'url': 'https://github.com/jupyterhub/oauthenticator'},\n", 177 | " {'issues': 22,\n", 178 | " 'name': 'dockerspawner',\n", 179 | " 'prs': 1,\n", 180 | " 'url': 'https://github.com/jupyterhub/dockerspawner'},\n", 181 | " {'issues': 2,\n", 182 | " 'name': 'sudospawner',\n", 183 | " 'prs': 0,\n", 184 | " 'url': 'https://github.com/jupyterhub/sudospawner'},\n", 185 | " {'issues': 12,\n", 186 | " 'name': 'batchspawner',\n", 187 | " 'prs': 5,\n", 188 | " 'url': 'https://github.com/jupyterhub/batchspawner'},\n", 189 | " {'issues': 25,\n", 190 | " 'name': 'kubespawner',\n", 191 | " 'prs': 3,\n", 192 | " 'url': 'https://github.com/jupyterhub/kubespawner'},\n", 193 | " {'issues': 18,\n", 194 | " 'name': 'ldapauthenticator',\n", 195 | " 'prs': 2,\n", 196 | " 'url': 'https://github.com/jupyterhub/ldapauthenticator'},\n", 197 | " {'issues': 9,\n", 198 | " 'name': 'jupyterhub-deploy-docker',\n", 199 | " 'prs': 1,\n", 200 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-docker'},\n", 201 | " {'issues': 13,\n", 202 | " 'name': 'jupyterhub-deploy-teaching',\n", 203 | " 'prs': 2,\n", 204 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-teaching'},\n", 205 | " {'issues': 5,\n", 206 | " 'name': 'jupyterhub-tutorial',\n", 207 | " 'prs': 0,\n", 208 | " 'url': 'https://github.com/jupyterhub/jupyterhub-tutorial'},\n", 209 | " {'issues': 2,\n", 210 | " 'name': 'jupyterhub-deploy-hpc',\n", 211 | " 'prs': 0,\n", 212 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-hpc'},\n", 213 | " {'issues': 4,\n", 214 | " 'name': 'systemdspawner',\n", 215 | " 'prs': 2,\n", 216 | " 'url': 'https://github.com/jupyterhub/systemdspawner'},\n", 217 | " {'issues': 5,\n", 218 | " 'name': 'wrapspawner',\n", 219 | " 'prs': 1,\n", 220 | " 'url': 'https://github.com/jupyterhub/wrapspawner'},\n", 221 | " {'issues': 4,\n", 222 | " 'name': 'jupyterlab-hub',\n", 223 | " 'prs': 0,\n", 224 | " 'url': 'https://github.com/jupyterhub/jupyterlab-hub'},\n", 225 | " {'issues': 2,\n", 226 | " 'name': 'nbserverproxy',\n", 227 | " 'prs': 1,\n", 228 | " 'url': 'https://github.com/jupyterhub/nbserverproxy'},\n", 229 | " {'issues': 3,\n", 230 | " 'name': 'jupyterhub-example-kerberos',\n", 231 | " 'prs': 0,\n", 232 | " 'url': 'https://github.com/jupyterhub/jupyterhub-example-kerberos'},\n", 233 | " {'issues': 10,\n", 234 | " 'name': 'hubshare',\n", 235 | " 'prs': 0,\n", 236 | " 'url': 'https://github.com/jupyterhub/hubshare'},\n", 237 | " {'issues': 4,\n", 238 | " 'name': 'nbrsessionproxy',\n", 239 | " 'prs': 2,\n", 240 | " 'url': 'https://github.com/jupyterhub/nbrsessionproxy'},\n", 241 | " {'issues': 0,\n", 242 | " 'name': 'tmpauthenticator',\n", 243 | " 'prs': 0,\n", 244 | " 'url': 'https://github.com/jupyterhub/tmpauthenticator'},\n", 245 | " {'issues': 89,\n", 246 | " 'name': 'zero-to-jupyterhub-k8s',\n", 247 | " 'prs': 3,\n", 248 | " 'url': 'https://github.com/jupyterhub/zero-to-jupyterhub-k8s'},\n", 249 | " {'issues': 3,\n", 250 | " 'name': 'helm-chart',\n", 251 | " 'prs': 2,\n", 252 | " 'url': 'https://github.com/jupyterhub/helm-chart'},\n", 253 | " {'issues': 79,\n", 254 | " 'name': 'binderhub',\n", 255 | " 'prs': 6,\n", 256 | " 'url': 'https://github.com/jupyterhub/binderhub'},\n", 257 | " {'issues': 25,\n", 258 | " 'name': 'mybinder.org-deploy',\n", 259 | " 'prs': 1,\n", 260 | " 'url': 'https://github.com/jupyterhub/mybinder.org-deploy'},\n", 261 | " {'issues': 9,\n", 262 | " 'name': 'binder',\n", 263 | " 'prs': 1,\n", 264 | " 'url': 'https://github.com/jupyterhub/binder'},\n", 265 | " {'issues': 0,\n", 266 | " 'name': 'nullauthenticator',\n", 267 | " 'prs': 0,\n", 268 | " 'url': 'https://github.com/jupyterhub/nullauthenticator'},\n", 269 | " {'issues': 2,\n", 270 | " 'name': 'team-compass',\n", 271 | " 'prs': 0,\n", 272 | " 'url': 'https://github.com/jupyterhub/team-compass'}]\n" 273 | ] 274 | } 275 | ], 276 | "source": [ 277 | "import pprint\n", 278 | "pprint.pprint(rows)" 279 | ] 280 | }, 281 | { 282 | "cell_type": "code", 283 | "execution_count": 13, 284 | "metadata": {}, 285 | "outputs": [ 286 | { 287 | "name": "stdout", 288 | "output_type": "stream", 289 | "text": [ 290 | "name url issues prs \n", 291 | "jupyterhub https://github.com/jupyterhub/jupyterhub 141 6\n", 292 | "configurable-http-proxy https://github.com/jupyterhub/configurable-http-proxy 11 0\n", 293 | "oauthenticator https://github.com/jupyterhub/oauthenticator 13 2\n", 294 | "dockerspawner https://github.com/jupyterhub/dockerspawner 22 1\n", 295 | "sudospawner https://github.com/jupyterhub/sudospawner 2 0\n", 296 | "batchspawner https://github.com/jupyterhub/batchspawner 12 5\n", 297 | "kubespawner https://github.com/jupyterhub/kubespawner 25 3\n", 298 | "ldapauthenticator https://github.com/jupyterhub/ldapauthenticator 18 2\n", 299 | "jupyterhub-deploy-docker https://github.com/jupyterhub/jupyterhub-deploy-docker 9 1\n", 300 | "jupyterhub-deploy-teaching https://github.com/jupyterhub/jupyterhub-deploy-teaching 13 2\n", 301 | "jupyterhub-tutorial https://github.com/jupyterhub/jupyterhub-tutorial 5 0\n", 302 | "jupyterhub-deploy-hpc https://github.com/jupyterhub/jupyterhub-deploy-hpc 2 0\n", 303 | "systemdspawner https://github.com/jupyterhub/systemdspawner 4 2\n", 304 | "wrapspawner https://github.com/jupyterhub/wrapspawner 5 1\n", 305 | "jupyterlab-hub https://github.com/jupyterhub/jupyterlab-hub 4 0\n", 306 | "nbserverproxy https://github.com/jupyterhub/nbserverproxy 2 1\n", 307 | "jupyterhub-example-kerberos https://github.com/jupyterhub/jupyterhub-example-kerberos 3 0\n", 308 | "hubshare https://github.com/jupyterhub/hubshare 10 0\n", 309 | "nbrsessionproxy https://github.com/jupyterhub/nbrsessionproxy 4 2\n", 310 | "tmpauthenticator https://github.com/jupyterhub/tmpauthenticator 0 0\n", 311 | "zero-to-jupyterhub-k8s https://github.com/jupyterhub/zero-to-jupyterhub-k8s 89 3\n", 312 | "helm-chart https://github.com/jupyterhub/helm-chart 3 2\n", 313 | "binderhub https://github.com/jupyterhub/binderhub 79 6\n", 314 | "mybinder.org-deploy https://github.com/jupyterhub/mybinder.org-deploy 25 1\n", 315 | "binder https://github.com/jupyterhub/binder 9 1\n", 316 | "nullauthenticator https://github.com/jupyterhub/nullauthenticator 0 0\n", 317 | "team-compass https://github.com/jupyterhub/team-compass 2 0\n" 318 | ] 319 | } 320 | ], 321 | "source": [ 322 | "print(f\"{'name':34} {'url':60} {'issues':8} {'prs':8}\")\n", 323 | "for obj in unpacked:\n", 324 | " print(f\"{obj['name']:30} {obj['url']:60} {obj['issues']['totalCount']:8} {obj['pullRequests']['totalCount']:8}\")" 325 | ] 326 | }, 327 | { 328 | "cell_type": "markdown", 329 | "metadata": {}, 330 | "source": [ 331 | "Bring into pandas" 332 | ] 333 | }, 334 | { 335 | "cell_type": "code", 336 | "execution_count": 14, 337 | "metadata": {}, 338 | "outputs": [], 339 | "source": [ 340 | "import pandas as pd" 341 | ] 342 | }, 343 | { 344 | "cell_type": "code", 345 | "execution_count": 15, 346 | "metadata": {}, 347 | "outputs": [], 348 | "source": [ 349 | "#df = pd.DataFrame.from_records(unpacked)\n", 350 | "df = pd.read_json('data.json')" 351 | ] 352 | }, 353 | { 354 | "cell_type": "code", 355 | "execution_count": 16, 356 | "metadata": {}, 357 | "outputs": [ 358 | { 359 | "data": { 360 | "text/plain": [ 361 | "Index(['data'], dtype='object')" 362 | ] 363 | }, 364 | "execution_count": 16, 365 | "metadata": {}, 366 | "output_type": "execute_result" 367 | } 368 | ], 369 | "source": [ 370 | "df.columns" 371 | ] 372 | }, 373 | { 374 | "cell_type": "code", 375 | "execution_count": 17, 376 | "metadata": {}, 377 | "outputs": [ 378 | { 379 | "data": { 380 | "text/html": [ 381 | "
\n", 382 | "\n", 395 | "\n", 396 | " \n", 397 | " \n", 398 | " \n", 399 | " \n", 400 | " \n", 401 | " \n", 402 | " \n", 403 | " \n", 404 | " \n", 405 | " \n", 406 | " \n", 407 | " \n", 408 | "
data
organization{'repositories': {'totalCount': 27, 'edges': [...
\n", 409 | "
" 410 | ], 411 | "text/plain": [ 412 | " data\n", 413 | "organization {'repositories': {'totalCount': 27, 'edges': [..." 414 | ] 415 | }, 416 | "execution_count": 17, 417 | "metadata": {}, 418 | "output_type": "execute_result" 419 | } 420 | ], 421 | "source": [ 422 | "df.head()" 423 | ] 424 | }, 425 | { 426 | "cell_type": "markdown", 427 | "metadata": {}, 428 | "source": [ 429 | "Generate basic report of total open issues" 430 | ] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "execution_count": 18, 435 | "metadata": {}, 436 | "outputs": [ 437 | { 438 | "data": { 439 | "text/plain": [ 440 | "data object\n", 441 | "dtype: object" 442 | ] 443 | }, 444 | "execution_count": 18, 445 | "metadata": {}, 446 | "output_type": "execute_result" 447 | } 448 | ], 449 | "source": [ 450 | "df.dtypes" 451 | ] 452 | }, 453 | { 454 | "cell_type": "code", 455 | "execution_count": 19, 456 | "metadata": {}, 457 | "outputs": [ 458 | { 459 | "data": { 460 | "text/html": [ 461 | "
\n", 462 | "\n", 475 | "\n", 476 | " \n", 477 | " \n", 478 | " \n", 479 | " \n", 480 | " \n", 481 | " \n", 482 | " \n", 483 | " \n", 484 | " \n", 485 | " \n", 486 | " \n", 487 | " \n", 488 | "
data
organization{'repositories': {'totalCount': 27, 'edges': [...
\n", 489 | "
" 490 | ], 491 | "text/plain": [ 492 | " data\n", 493 | "organization {'repositories': {'totalCount': 27, 'edges': [..." 494 | ] 495 | }, 496 | "execution_count": 19, 497 | "metadata": {}, 498 | "output_type": "execute_result" 499 | } 500 | ], 501 | "source": [ 502 | "df.head()" 503 | ] 504 | }, 505 | { 506 | "cell_type": "code", 507 | "execution_count": 20, 508 | "metadata": {}, 509 | "outputs": [ 510 | { 511 | "data": { 512 | "text/plain": [ 513 | "Index(['organization'], dtype='object')" 514 | ] 515 | }, 516 | "execution_count": 20, 517 | "metadata": {}, 518 | "output_type": "execute_result" 519 | } 520 | ], 521 | "source": [ 522 | "df.index" 523 | ] 524 | }, 525 | { 526 | "cell_type": "code", 527 | "execution_count": 21, 528 | "metadata": {}, 529 | "outputs": [ 530 | { 531 | "data": { 532 | "text/plain": [ 533 | "Index(['data'], dtype='object')" 534 | ] 535 | }, 536 | "execution_count": 21, 537 | "metadata": {}, 538 | "output_type": "execute_result" 539 | } 540 | ], 541 | "source": [ 542 | "df.columns" 543 | ] 544 | }, 545 | { 546 | "cell_type": "code", 547 | "execution_count": 22, 548 | "metadata": {}, 549 | "outputs": [ 550 | { 551 | "data": { 552 | "text/plain": [ 553 | "array([[ {'repositories': {'totalCount': 27, 'edges': [{'node': {'name': 'jupyterhub', 'url': 'https://github.com/jupyterhub/jupyterhub', 'issues': {'totalCount': 141}, 'pullRequests': {'totalCount': 6}}}, {'node': {'name': 'configurable-http-proxy', 'url': 'https://github.com/jupyterhub/configurable-http-proxy', 'issues': {'totalCount': 11}, 'pullRequests': {'totalCount': 0}}}, {'node': {'name': 'oauthenticator', 'url': 'https://github.com/jupyterhub/oauthenticator', 'issues': {'totalCount': 13}, 'pullRequests': {'totalCount': 2}}}, {'node': {'name': 'dockerspawner', 'url': 'https://github.com/jupyterhub/dockerspawner', 'issues': {'totalCount': 22}, 'pullRequests': {'totalCount': 1}}}, {'node': {'name': 'sudospawner', 'url': 'https://github.com/jupyterhub/sudospawner', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 0}}}, {'node': {'name': 'batchspawner', 'url': 'https://github.com/jupyterhub/batchspawner', 'issues': {'totalCount': 12}, 'pullRequests': {'totalCount': 5}}}, {'node': {'name': 'kubespawner', 'url': 'https://github.com/jupyterhub/kubespawner', 'issues': {'totalCount': 25}, 'pullRequests': {'totalCount': 3}}}, {'node': {'name': 'ldapauthenticator', 'url': 'https://github.com/jupyterhub/ldapauthenticator', 'issues': {'totalCount': 18}, 'pullRequests': {'totalCount': 2}}}, {'node': {'name': 'jupyterhub-deploy-docker', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-docker', 'issues': {'totalCount': 9}, 'pullRequests': {'totalCount': 1}}}, {'node': {'name': 'jupyterhub-deploy-teaching', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-teaching', 'issues': {'totalCount': 13}, 'pullRequests': {'totalCount': 2}}}, {'node': {'name': 'jupyterhub-tutorial', 'url': 'https://github.com/jupyterhub/jupyterhub-tutorial', 'issues': {'totalCount': 5}, 'pullRequests': {'totalCount': 0}}}, {'node': {'name': 'jupyterhub-deploy-hpc', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-hpc', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 0}}}, {'node': {'name': 'systemdspawner', 'url': 'https://github.com/jupyterhub/systemdspawner', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 2}}}, {'node': {'name': 'wrapspawner', 'url': 'https://github.com/jupyterhub/wrapspawner', 'issues': {'totalCount': 5}, 'pullRequests': {'totalCount': 1}}}, {'node': {'name': 'jupyterlab-hub', 'url': 'https://github.com/jupyterhub/jupyterlab-hub', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 0}}}, {'node': {'name': 'nbserverproxy', 'url': 'https://github.com/jupyterhub/nbserverproxy', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 1}}}, {'node': {'name': 'jupyterhub-example-kerberos', 'url': 'https://github.com/jupyterhub/jupyterhub-example-kerberos', 'issues': {'totalCount': 3}, 'pullRequests': {'totalCount': 0}}}, {'node': {'name': 'hubshare', 'url': 'https://github.com/jupyterhub/hubshare', 'issues': {'totalCount': 10}, 'pullRequests': {'totalCount': 0}}}, {'node': {'name': 'nbrsessionproxy', 'url': 'https://github.com/jupyterhub/nbrsessionproxy', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 2}}}, {'node': {'name': 'tmpauthenticator', 'url': 'https://github.com/jupyterhub/tmpauthenticator', 'issues': {'totalCount': 0}, 'pullRequests': {'totalCount': 0}}}, {'node': {'name': 'zero-to-jupyterhub-k8s', 'url': 'https://github.com/jupyterhub/zero-to-jupyterhub-k8s', 'issues': {'totalCount': 89}, 'pullRequests': {'totalCount': 3}}}, {'node': {'name': 'helm-chart', 'url': 'https://github.com/jupyterhub/helm-chart', 'issues': {'totalCount': 3}, 'pullRequests': {'totalCount': 2}}}, {'node': {'name': 'binderhub', 'url': 'https://github.com/jupyterhub/binderhub', 'issues': {'totalCount': 79}, 'pullRequests': {'totalCount': 6}}}, {'node': {'name': 'mybinder.org-deploy', 'url': 'https://github.com/jupyterhub/mybinder.org-deploy', 'issues': {'totalCount': 25}, 'pullRequests': {'totalCount': 1}}}, {'node': {'name': 'binder', 'url': 'https://github.com/jupyterhub/binder', 'issues': {'totalCount': 9}, 'pullRequests': {'totalCount': 1}}}, {'node': {'name': 'nullauthenticator', 'url': 'https://github.com/jupyterhub/nullauthenticator', 'issues': {'totalCount': 0}, 'pullRequests': {'totalCount': 0}}}, {'node': {'name': 'team-compass', 'url': 'https://github.com/jupyterhub/team-compass', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 0}}}]}}]], dtype=object)" 554 | ] 555 | }, 556 | "execution_count": 22, 557 | "metadata": {}, 558 | "output_type": "execute_result" 559 | } 560 | ], 561 | "source": [ 562 | "df.values" 563 | ] 564 | }, 565 | { 566 | "cell_type": "code", 567 | "execution_count": 23, 568 | "metadata": {}, 569 | "outputs": [ 570 | { 571 | "data": { 572 | "text/plain": [ 573 | "" 575 | ] 576 | }, 577 | "execution_count": 23, 578 | "metadata": {}, 579 | "output_type": "execute_result" 580 | } 581 | ], 582 | "source": [ 583 | "df.describe" 584 | ] 585 | }, 586 | { 587 | "cell_type": "code", 588 | "execution_count": 24, 589 | "metadata": {}, 590 | "outputs": [], 591 | "source": [ 592 | "# output data to a csv\n", 593 | "# df.to_csv('issue_data.csv')" 594 | ] 595 | } 596 | ], 597 | "metadata": { 598 | "kernelspec": { 599 | "display_name": "Python 3", 600 | "language": "python", 601 | "name": "python3" 602 | }, 603 | "language_info": { 604 | "codemirror_mode": { 605 | "name": "ipython", 606 | "version": 3 607 | }, 608 | "file_extension": ".py", 609 | "mimetype": "text/x-python", 610 | "name": "python", 611 | "nbconvert_exporter": "python", 612 | "pygments_lexer": "ipython3", 613 | "version": "3.6.3" 614 | } 615 | }, 616 | "nbformat": 4, 617 | "nbformat_minor": 2 618 | } 619 | -------------------------------------------------------------------------------- /notebooks/use-pyquery-ql.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Use pyquery-ql.py\n", 8 | "\n", 9 | "Send a graphql query to GitHub\n", 10 | "and pretty print output.\n", 11 | "\n", 12 | "Supports Python 3.6+" 13 | ] 14 | }, 15 | { 16 | "cell_type": "code", 17 | "execution_count": 1, 18 | "metadata": {}, 19 | "outputs": [], 20 | "source": [ 21 | "import json\n", 22 | "import os\n", 23 | "import pprint\n", 24 | "\n", 25 | "import requests" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 2, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "# get api token and set authorization\n", 35 | "api_token = os.environ['GITHUB_API_TOKEN']\n", 36 | "headers = {'Authorization': f'token {api_token}'}" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 3, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "# set url to a graphql endpoint\n", 46 | "url = 'https://api.github.com/graphql'" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": 4, 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "# add a json query\n", 56 | "query = \"\"\"\n", 57 | "{\n", 58 | " organization(login: \"jupyterhub\") {\n", 59 | " repositories(first: 30) {\n", 60 | " totalCount\n", 61 | " edges {\n", 62 | " node {\n", 63 | " name\n", 64 | " url\n", 65 | " issues(states: OPEN) {\n", 66 | " totalCount\n", 67 | " }\n", 68 | " pullRequests(states: OPEN) {\n", 69 | " totalCount\n", 70 | " }\n", 71 | " }\n", 72 | " }\n", 73 | " }\n", 74 | " }\n", 75 | "}\n", 76 | "\"\"\"" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": 5, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "# submit the request\n", 86 | "r = requests.post(url=url, json={'query': query}, headers=headers)" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 6, 92 | "metadata": {}, 93 | "outputs": [ 94 | { 95 | "data": { 96 | "text/plain": [ 97 | "{'data': {'organization': {'repositories': {'edges': [{'node': {'issues': {'totalCount': 141},\n", 98 | " 'name': 'jupyterhub',\n", 99 | " 'pullRequests': {'totalCount': 6},\n", 100 | " 'url': 'https://github.com/jupyterhub/jupyterhub'}},\n", 101 | " {'node': {'issues': {'totalCount': 11},\n", 102 | " 'name': 'configurable-http-proxy',\n", 103 | " 'pullRequests': {'totalCount': 0},\n", 104 | " 'url': 'https://github.com/jupyterhub/configurable-http-proxy'}},\n", 105 | " {'node': {'issues': {'totalCount': 13},\n", 106 | " 'name': 'oauthenticator',\n", 107 | " 'pullRequests': {'totalCount': 2},\n", 108 | " 'url': 'https://github.com/jupyterhub/oauthenticator'}},\n", 109 | " {'node': {'issues': {'totalCount': 22},\n", 110 | " 'name': 'dockerspawner',\n", 111 | " 'pullRequests': {'totalCount': 1},\n", 112 | " 'url': 'https://github.com/jupyterhub/dockerspawner'}},\n", 113 | " {'node': {'issues': {'totalCount': 2},\n", 114 | " 'name': 'sudospawner',\n", 115 | " 'pullRequests': {'totalCount': 0},\n", 116 | " 'url': 'https://github.com/jupyterhub/sudospawner'}},\n", 117 | " {'node': {'issues': {'totalCount': 12},\n", 118 | " 'name': 'batchspawner',\n", 119 | " 'pullRequests': {'totalCount': 5},\n", 120 | " 'url': 'https://github.com/jupyterhub/batchspawner'}},\n", 121 | " {'node': {'issues': {'totalCount': 25},\n", 122 | " 'name': 'kubespawner',\n", 123 | " 'pullRequests': {'totalCount': 3},\n", 124 | " 'url': 'https://github.com/jupyterhub/kubespawner'}},\n", 125 | " {'node': {'issues': {'totalCount': 18},\n", 126 | " 'name': 'ldapauthenticator',\n", 127 | " 'pullRequests': {'totalCount': 2},\n", 128 | " 'url': 'https://github.com/jupyterhub/ldapauthenticator'}},\n", 129 | " {'node': {'issues': {'totalCount': 9},\n", 130 | " 'name': 'jupyterhub-deploy-docker',\n", 131 | " 'pullRequests': {'totalCount': 1},\n", 132 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-docker'}},\n", 133 | " {'node': {'issues': {'totalCount': 13},\n", 134 | " 'name': 'jupyterhub-deploy-teaching',\n", 135 | " 'pullRequests': {'totalCount': 2},\n", 136 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-teaching'}},\n", 137 | " {'node': {'issues': {'totalCount': 5},\n", 138 | " 'name': 'jupyterhub-tutorial',\n", 139 | " 'pullRequests': {'totalCount': 0},\n", 140 | " 'url': 'https://github.com/jupyterhub/jupyterhub-tutorial'}},\n", 141 | " {'node': {'issues': {'totalCount': 2},\n", 142 | " 'name': 'jupyterhub-deploy-hpc',\n", 143 | " 'pullRequests': {'totalCount': 0},\n", 144 | " 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-hpc'}},\n", 145 | " {'node': {'issues': {'totalCount': 4},\n", 146 | " 'name': 'systemdspawner',\n", 147 | " 'pullRequests': {'totalCount': 2},\n", 148 | " 'url': 'https://github.com/jupyterhub/systemdspawner'}},\n", 149 | " {'node': {'issues': {'totalCount': 5},\n", 150 | " 'name': 'wrapspawner',\n", 151 | " 'pullRequests': {'totalCount': 1},\n", 152 | " 'url': 'https://github.com/jupyterhub/wrapspawner'}},\n", 153 | " {'node': {'issues': {'totalCount': 4},\n", 154 | " 'name': 'jupyterlab-hub',\n", 155 | " 'pullRequests': {'totalCount': 0},\n", 156 | " 'url': 'https://github.com/jupyterhub/jupyterlab-hub'}},\n", 157 | " {'node': {'issues': {'totalCount': 2},\n", 158 | " 'name': 'nbserverproxy',\n", 159 | " 'pullRequests': {'totalCount': 1},\n", 160 | " 'url': 'https://github.com/jupyterhub/nbserverproxy'}},\n", 161 | " {'node': {'issues': {'totalCount': 3},\n", 162 | " 'name': 'jupyterhub-example-kerberos',\n", 163 | " 'pullRequests': {'totalCount': 0},\n", 164 | " 'url': 'https://github.com/jupyterhub/jupyterhub-example-kerberos'}},\n", 165 | " {'node': {'issues': {'totalCount': 10},\n", 166 | " 'name': 'hubshare',\n", 167 | " 'pullRequests': {'totalCount': 0},\n", 168 | " 'url': 'https://github.com/jupyterhub/hubshare'}},\n", 169 | " {'node': {'issues': {'totalCount': 4},\n", 170 | " 'name': 'nbrsessionproxy',\n", 171 | " 'pullRequests': {'totalCount': 2},\n", 172 | " 'url': 'https://github.com/jupyterhub/nbrsessionproxy'}},\n", 173 | " {'node': {'issues': {'totalCount': 0},\n", 174 | " 'name': 'tmpauthenticator',\n", 175 | " 'pullRequests': {'totalCount': 0},\n", 176 | " 'url': 'https://github.com/jupyterhub/tmpauthenticator'}},\n", 177 | " {'node': {'issues': {'totalCount': 89},\n", 178 | " 'name': 'zero-to-jupyterhub-k8s',\n", 179 | " 'pullRequests': {'totalCount': 3},\n", 180 | " 'url': 'https://github.com/jupyterhub/zero-to-jupyterhub-k8s'}},\n", 181 | " {'node': {'issues': {'totalCount': 3},\n", 182 | " 'name': 'helm-chart',\n", 183 | " 'pullRequests': {'totalCount': 2},\n", 184 | " 'url': 'https://github.com/jupyterhub/helm-chart'}},\n", 185 | " {'node': {'issues': {'totalCount': 79},\n", 186 | " 'name': 'binderhub',\n", 187 | " 'pullRequests': {'totalCount': 6},\n", 188 | " 'url': 'https://github.com/jupyterhub/binderhub'}},\n", 189 | " {'node': {'issues': {'totalCount': 25},\n", 190 | " 'name': 'mybinder.org-deploy',\n", 191 | " 'pullRequests': {'totalCount': 1},\n", 192 | " 'url': 'https://github.com/jupyterhub/mybinder.org-deploy'}},\n", 193 | " {'node': {'issues': {'totalCount': 9},\n", 194 | " 'name': 'binder',\n", 195 | " 'pullRequests': {'totalCount': 1},\n", 196 | " 'url': 'https://github.com/jupyterhub/binder'}},\n", 197 | " {'node': {'issues': {'totalCount': 0},\n", 198 | " 'name': 'nullauthenticator',\n", 199 | " 'pullRequests': {'totalCount': 0},\n", 200 | " 'url': 'https://github.com/jupyterhub/nullauthenticator'}},\n", 201 | " {'node': {'issues': {'totalCount': 2},\n", 202 | " 'name': 'team-compass',\n", 203 | " 'pullRequests': {'totalCount': 0},\n", 204 | " 'url': 'https://github.com/jupyterhub/team-compass'}}],\n", 205 | " 'totalCount': 27}}}}" 206 | ] 207 | }, 208 | "execution_count": 6, 209 | "metadata": {}, 210 | "output_type": "execute_result" 211 | } 212 | ], 213 | "source": [ 214 | "r.json()" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": 7, 220 | "metadata": {}, 221 | "outputs": [ 222 | { 223 | "data": { 224 | "text/plain": [ 225 | "'{\"data\":{\"organization\":{\"repositories\":{\"totalCount\":27,\"edges\":[{\"node\":{\"name\":\"jupyterhub\",\"url\":\"https://github.com/jupyterhub/jupyterhub\",\"issues\":{\"totalCount\":141},\"pullRequests\":{\"totalCount\":6}}},{\"node\":{\"name\":\"configurable-http-proxy\",\"url\":\"https://github.com/jupyterhub/configurable-http-proxy\",\"issues\":{\"totalCount\":11},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"oauthenticator\",\"url\":\"https://github.com/jupyterhub/oauthenticator\",\"issues\":{\"totalCount\":13},\"pullRequests\":{\"totalCount\":2}}},{\"node\":{\"name\":\"dockerspawner\",\"url\":\"https://github.com/jupyterhub/dockerspawner\",\"issues\":{\"totalCount\":22},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"sudospawner\",\"url\":\"https://github.com/jupyterhub/sudospawner\",\"issues\":{\"totalCount\":2},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"batchspawner\",\"url\":\"https://github.com/jupyterhub/batchspawner\",\"issues\":{\"totalCount\":12},\"pullRequests\":{\"totalCount\":5}}},{\"node\":{\"name\":\"kubespawner\",\"url\":\"https://github.com/jupyterhub/kubespawner\",\"issues\":{\"totalCount\":25},\"pullRequests\":{\"totalCount\":3}}},{\"node\":{\"name\":\"ldapauthenticator\",\"url\":\"https://github.com/jupyterhub/ldapauthenticator\",\"issues\":{\"totalCount\":18},\"pullRequests\":{\"totalCount\":2}}},{\"node\":{\"name\":\"jupyterhub-deploy-docker\",\"url\":\"https://github.com/jupyterhub/jupyterhub-deploy-docker\",\"issues\":{\"totalCount\":9},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"jupyterhub-deploy-teaching\",\"url\":\"https://github.com/jupyterhub/jupyterhub-deploy-teaching\",\"issues\":{\"totalCount\":13},\"pullRequests\":{\"totalCount\":2}}},{\"node\":{\"name\":\"jupyterhub-tutorial\",\"url\":\"https://github.com/jupyterhub/jupyterhub-tutorial\",\"issues\":{\"totalCount\":5},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"jupyterhub-deploy-hpc\",\"url\":\"https://github.com/jupyterhub/jupyterhub-deploy-hpc\",\"issues\":{\"totalCount\":2},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"systemdspawner\",\"url\":\"https://github.com/jupyterhub/systemdspawner\",\"issues\":{\"totalCount\":4},\"pullRequests\":{\"totalCount\":2}}},{\"node\":{\"name\":\"wrapspawner\",\"url\":\"https://github.com/jupyterhub/wrapspawner\",\"issues\":{\"totalCount\":5},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"jupyterlab-hub\",\"url\":\"https://github.com/jupyterhub/jupyterlab-hub\",\"issues\":{\"totalCount\":4},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"nbserverproxy\",\"url\":\"https://github.com/jupyterhub/nbserverproxy\",\"issues\":{\"totalCount\":2},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"jupyterhub-example-kerberos\",\"url\":\"https://github.com/jupyterhub/jupyterhub-example-kerberos\",\"issues\":{\"totalCount\":3},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"hubshare\",\"url\":\"https://github.com/jupyterhub/hubshare\",\"issues\":{\"totalCount\":10},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"nbrsessionproxy\",\"url\":\"https://github.com/jupyterhub/nbrsessionproxy\",\"issues\":{\"totalCount\":4},\"pullRequests\":{\"totalCount\":2}}},{\"node\":{\"name\":\"tmpauthenticator\",\"url\":\"https://github.com/jupyterhub/tmpauthenticator\",\"issues\":{\"totalCount\":0},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"zero-to-jupyterhub-k8s\",\"url\":\"https://github.com/jupyterhub/zero-to-jupyterhub-k8s\",\"issues\":{\"totalCount\":89},\"pullRequests\":{\"totalCount\":3}}},{\"node\":{\"name\":\"helm-chart\",\"url\":\"https://github.com/jupyterhub/helm-chart\",\"issues\":{\"totalCount\":3},\"pullRequests\":{\"totalCount\":2}}},{\"node\":{\"name\":\"binderhub\",\"url\":\"https://github.com/jupyterhub/binderhub\",\"issues\":{\"totalCount\":79},\"pullRequests\":{\"totalCount\":6}}},{\"node\":{\"name\":\"mybinder.org-deploy\",\"url\":\"https://github.com/jupyterhub/mybinder.org-deploy\",\"issues\":{\"totalCount\":25},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"binder\",\"url\":\"https://github.com/jupyterhub/binder\",\"issues\":{\"totalCount\":9},\"pullRequests\":{\"totalCount\":1}}},{\"node\":{\"name\":\"nullauthenticator\",\"url\":\"https://github.com/jupyterhub/nullauthenticator\",\"issues\":{\"totalCount\":0},\"pullRequests\":{\"totalCount\":0}}},{\"node\":{\"name\":\"team-compass\",\"url\":\"https://github.com/jupyterhub/team-compass\",\"issues\":{\"totalCount\":2},\"pullRequests\":{\"totalCount\":0}}}]}}}}'" 226 | ] 227 | }, 228 | "execution_count": 7, 229 | "metadata": {}, 230 | "output_type": "execute_result" 231 | } 232 | ], 233 | "source": [ 234 | "r.text" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": 8, 240 | "metadata": {}, 241 | "outputs": [], 242 | "source": [ 243 | "data = json.loads(r.text)" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": 9, 249 | "metadata": {}, 250 | "outputs": [], 251 | "source": [ 252 | "# pprint.pprint(data)" 253 | ] 254 | }, 255 | { 256 | "cell_type": "markdown", 257 | "metadata": {}, 258 | "source": [ 259 | "Walk the JSON response contents" 260 | ] 261 | }, 262 | { 263 | "cell_type": "code", 264 | "execution_count": 10, 265 | "metadata": {}, 266 | "outputs": [], 267 | "source": [ 268 | "issue_data = data['data']" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": 11, 274 | "metadata": {}, 275 | "outputs": [], 276 | "source": [ 277 | "org = issue_data['organization']" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": 12, 283 | "metadata": {}, 284 | "outputs": [], 285 | "source": [ 286 | "repos = org['repositories']" 287 | ] 288 | }, 289 | { 290 | "cell_type": "code", 291 | "execution_count": 13, 292 | "metadata": {}, 293 | "outputs": [], 294 | "source": [ 295 | "edges = repos['edges']" 296 | ] 297 | }, 298 | { 299 | "cell_type": "code", 300 | "execution_count": 14, 301 | "metadata": {}, 302 | "outputs": [], 303 | "source": [ 304 | "# edges[0]['node']" 305 | ] 306 | }, 307 | { 308 | "cell_type": "code", 309 | "execution_count": 15, 310 | "metadata": {}, 311 | "outputs": [], 312 | "source": [ 313 | "# print(edges[0]['node']['name'] + '---' + edges[0]['node']['url'])" 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": 16, 319 | "metadata": {}, 320 | "outputs": [], 321 | "source": [ 322 | "# for edge in edges:\n", 323 | "# pprint.pprint(edge)" 324 | ] 325 | }, 326 | { 327 | "cell_type": "markdown", 328 | "metadata": {}, 329 | "source": [ 330 | "Bring into pandas" 331 | ] 332 | }, 333 | { 334 | "cell_type": "code", 335 | "execution_count": 17, 336 | "metadata": {}, 337 | "outputs": [], 338 | "source": [ 339 | "import pandas as pd" 340 | ] 341 | }, 342 | { 343 | "cell_type": "code", 344 | "execution_count": 18, 345 | "metadata": {}, 346 | "outputs": [], 347 | "source": [ 348 | "df = pd.DataFrame.from_records(edges)" 349 | ] 350 | }, 351 | { 352 | "cell_type": "code", 353 | "execution_count": 19, 354 | "metadata": {}, 355 | "outputs": [ 356 | { 357 | "data": { 358 | "text/plain": [ 359 | "Index(['node'], dtype='object')" 360 | ] 361 | }, 362 | "execution_count": 19, 363 | "metadata": {}, 364 | "output_type": "execute_result" 365 | } 366 | ], 367 | "source": [ 368 | "df.columns" 369 | ] 370 | }, 371 | { 372 | "cell_type": "code", 373 | "execution_count": 20, 374 | "metadata": {}, 375 | "outputs": [], 376 | "source": [ 377 | "# df['node']" 378 | ] 379 | }, 380 | { 381 | "cell_type": "markdown", 382 | "metadata": {}, 383 | "source": [ 384 | "Generate basic report of total open issues" 385 | ] 386 | }, 387 | { 388 | "cell_type": "code", 389 | "execution_count": 21, 390 | "metadata": {}, 391 | "outputs": [ 392 | { 393 | "name": "stdout", 394 | "output_type": "stream", 395 | "text": [ 396 | "Repo OpenIssues OpenPRs URL\n", 397 | "team-compass 2 0 https://github.com/jupyterhub/team-compass\n", 398 | "nullauthenticator 0 0 https://github.com/jupyterhub/nullauthenticator\n", 399 | "binder 9 1 https://github.com/jupyterhub/binder\n", 400 | "mybinder.org-deploy 25 1 https://github.com/jupyterhub/mybinder.org-deploy\n", 401 | "binderhub 79 6 https://github.com/jupyterhub/binderhub\n", 402 | "helm-chart 3 2 https://github.com/jupyterhub/helm-chart\n", 403 | "zero-to-jupyterhub-k8s 89 3 https://github.com/jupyterhub/zero-to-jupyterhub-k8s\n", 404 | "tmpauthenticator 0 0 https://github.com/jupyterhub/tmpauthenticator\n", 405 | "nbrsessionproxy 4 2 https://github.com/jupyterhub/nbrsessionproxy\n", 406 | "hubshare 10 0 https://github.com/jupyterhub/hubshare\n", 407 | "jupyterhub-example-kerberos 3 0 https://github.com/jupyterhub/jupyterhub-example-kerberos\n", 408 | "nbserverproxy 2 1 https://github.com/jupyterhub/nbserverproxy\n", 409 | "jupyterlab-hub 4 0 https://github.com/jupyterhub/jupyterlab-hub\n", 410 | "wrapspawner 5 1 https://github.com/jupyterhub/wrapspawner\n", 411 | "systemdspawner 4 2 https://github.com/jupyterhub/systemdspawner\n", 412 | "jupyterhub-deploy-hpc 2 0 https://github.com/jupyterhub/jupyterhub-deploy-hpc\n", 413 | "jupyterhub-tutorial 5 0 https://github.com/jupyterhub/jupyterhub-tutorial\n", 414 | "jupyterhub-deploy-teaching 13 2 https://github.com/jupyterhub/jupyterhub-deploy-teaching\n", 415 | "jupyterhub-deploy-docker 9 1 https://github.com/jupyterhub/jupyterhub-deploy-docker\n", 416 | "ldapauthenticator 18 2 https://github.com/jupyterhub/ldapauthenticator\n", 417 | "kubespawner 25 3 https://github.com/jupyterhub/kubespawner\n", 418 | "batchspawner 12 5 https://github.com/jupyterhub/batchspawner\n", 419 | "sudospawner 2 0 https://github.com/jupyterhub/sudospawner\n", 420 | "dockerspawner 22 1 https://github.com/jupyterhub/dockerspawner\n", 421 | "oauthenticator 13 2 https://github.com/jupyterhub/oauthenticator\n", 422 | "configurable-http-proxy 11 0 https://github.com/jupyterhub/configurable-http-proxy\n" 423 | ] 424 | } 425 | ], 426 | "source": [ 427 | "print(f\"{'Repo':30} {'OpenIssues':11} {'OpenPRs':9} {'URL'}\")\n", 428 | "\n", 429 | "line = 26\n", 430 | "while line > 0:\n", 431 | " lineout = f\"{df['node'][line]['name']:30} {(df['node'][line]['issues']['totalCount']):8} {(df['node'][line]['pullRequests']['totalCount']):8} {df['node'][line]['url']}\"\n", 432 | " print(lineout)\n", 433 | " line -= 1" 434 | ] 435 | }, 436 | { 437 | "cell_type": "code", 438 | "execution_count": 22, 439 | "metadata": {}, 440 | "outputs": [ 441 | { 442 | "data": { 443 | "text/plain": [ 444 | "node object\n", 445 | "dtype: object" 446 | ] 447 | }, 448 | "execution_count": 22, 449 | "metadata": {}, 450 | "output_type": "execute_result" 451 | } 452 | ], 453 | "source": [ 454 | "df.dtypes" 455 | ] 456 | }, 457 | { 458 | "cell_type": "code", 459 | "execution_count": 23, 460 | "metadata": {}, 461 | "outputs": [ 462 | { 463 | "data": { 464 | "text/html": [ 465 | "
\n", 466 | "\n", 479 | "\n", 480 | " \n", 481 | " \n", 482 | " \n", 483 | " \n", 484 | " \n", 485 | " \n", 486 | " \n", 487 | " \n", 488 | " \n", 489 | " \n", 490 | " \n", 491 | " \n", 492 | " \n", 493 | " \n", 494 | " \n", 495 | " \n", 496 | " \n", 497 | " \n", 498 | " \n", 499 | " \n", 500 | " \n", 501 | " \n", 502 | " \n", 503 | " \n", 504 | " \n", 505 | " \n", 506 | " \n", 507 | " \n", 508 | "
node
0{'name': 'jupyterhub', 'url': 'https://github....
1{'name': 'configurable-http-proxy', 'url': 'ht...
2{'name': 'oauthenticator', 'url': 'https://git...
3{'name': 'dockerspawner', 'url': 'https://gith...
4{'name': 'sudospawner', 'url': 'https://github...
\n", 509 | "
" 510 | ], 511 | "text/plain": [ 512 | " node\n", 513 | "0 {'name': 'jupyterhub', 'url': 'https://github....\n", 514 | "1 {'name': 'configurable-http-proxy', 'url': 'ht...\n", 515 | "2 {'name': 'oauthenticator', 'url': 'https://git...\n", 516 | "3 {'name': 'dockerspawner', 'url': 'https://gith...\n", 517 | "4 {'name': 'sudospawner', 'url': 'https://github..." 518 | ] 519 | }, 520 | "execution_count": 23, 521 | "metadata": {}, 522 | "output_type": "execute_result" 523 | } 524 | ], 525 | "source": [ 526 | "df.head()" 527 | ] 528 | }, 529 | { 530 | "cell_type": "code", 531 | "execution_count": 24, 532 | "metadata": {}, 533 | "outputs": [ 534 | { 535 | "data": { 536 | "text/plain": [ 537 | "RangeIndex(start=0, stop=27, step=1)" 538 | ] 539 | }, 540 | "execution_count": 24, 541 | "metadata": {}, 542 | "output_type": "execute_result" 543 | } 544 | ], 545 | "source": [ 546 | "df.index" 547 | ] 548 | }, 549 | { 550 | "cell_type": "code", 551 | "execution_count": 25, 552 | "metadata": {}, 553 | "outputs": [ 554 | { 555 | "data": { 556 | "text/plain": [ 557 | "Index(['node'], dtype='object')" 558 | ] 559 | }, 560 | "execution_count": 25, 561 | "metadata": {}, 562 | "output_type": "execute_result" 563 | } 564 | ], 565 | "source": [ 566 | "df.columns" 567 | ] 568 | }, 569 | { 570 | "cell_type": "code", 571 | "execution_count": 26, 572 | "metadata": {}, 573 | "outputs": [ 574 | { 575 | "data": { 576 | "text/plain": [ 577 | "array([[ {'name': 'jupyterhub', 'url': 'https://github.com/jupyterhub/jupyterhub', 'issues': {'totalCount': 141}, 'pullRequests': {'totalCount': 6}}],\n", 578 | " [ {'name': 'configurable-http-proxy', 'url': 'https://github.com/jupyterhub/configurable-http-proxy', 'issues': {'totalCount': 11}, 'pullRequests': {'totalCount': 0}}],\n", 579 | " [ {'name': 'oauthenticator', 'url': 'https://github.com/jupyterhub/oauthenticator', 'issues': {'totalCount': 13}, 'pullRequests': {'totalCount': 2}}],\n", 580 | " [ {'name': 'dockerspawner', 'url': 'https://github.com/jupyterhub/dockerspawner', 'issues': {'totalCount': 22}, 'pullRequests': {'totalCount': 1}}],\n", 581 | " [ {'name': 'sudospawner', 'url': 'https://github.com/jupyterhub/sudospawner', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 0}}],\n", 582 | " [ {'name': 'batchspawner', 'url': 'https://github.com/jupyterhub/batchspawner', 'issues': {'totalCount': 12}, 'pullRequests': {'totalCount': 5}}],\n", 583 | " [ {'name': 'kubespawner', 'url': 'https://github.com/jupyterhub/kubespawner', 'issues': {'totalCount': 25}, 'pullRequests': {'totalCount': 3}}],\n", 584 | " [ {'name': 'ldapauthenticator', 'url': 'https://github.com/jupyterhub/ldapauthenticator', 'issues': {'totalCount': 18}, 'pullRequests': {'totalCount': 2}}],\n", 585 | " [ {'name': 'jupyterhub-deploy-docker', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-docker', 'issues': {'totalCount': 9}, 'pullRequests': {'totalCount': 1}}],\n", 586 | " [ {'name': 'jupyterhub-deploy-teaching', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-teaching', 'issues': {'totalCount': 13}, 'pullRequests': {'totalCount': 2}}],\n", 587 | " [ {'name': 'jupyterhub-tutorial', 'url': 'https://github.com/jupyterhub/jupyterhub-tutorial', 'issues': {'totalCount': 5}, 'pullRequests': {'totalCount': 0}}],\n", 588 | " [ {'name': 'jupyterhub-deploy-hpc', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-hpc', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 0}}],\n", 589 | " [ {'name': 'systemdspawner', 'url': 'https://github.com/jupyterhub/systemdspawner', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 2}}],\n", 590 | " [ {'name': 'wrapspawner', 'url': 'https://github.com/jupyterhub/wrapspawner', 'issues': {'totalCount': 5}, 'pullRequests': {'totalCount': 1}}],\n", 591 | " [ {'name': 'jupyterlab-hub', 'url': 'https://github.com/jupyterhub/jupyterlab-hub', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 0}}],\n", 592 | " [ {'name': 'nbserverproxy', 'url': 'https://github.com/jupyterhub/nbserverproxy', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 1}}],\n", 593 | " [ {'name': 'jupyterhub-example-kerberos', 'url': 'https://github.com/jupyterhub/jupyterhub-example-kerberos', 'issues': {'totalCount': 3}, 'pullRequests': {'totalCount': 0}}],\n", 594 | " [ {'name': 'hubshare', 'url': 'https://github.com/jupyterhub/hubshare', 'issues': {'totalCount': 10}, 'pullRequests': {'totalCount': 0}}],\n", 595 | " [ {'name': 'nbrsessionproxy', 'url': 'https://github.com/jupyterhub/nbrsessionproxy', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 2}}],\n", 596 | " [ {'name': 'tmpauthenticator', 'url': 'https://github.com/jupyterhub/tmpauthenticator', 'issues': {'totalCount': 0}, 'pullRequests': {'totalCount': 0}}],\n", 597 | " [ {'name': 'zero-to-jupyterhub-k8s', 'url': 'https://github.com/jupyterhub/zero-to-jupyterhub-k8s', 'issues': {'totalCount': 89}, 'pullRequests': {'totalCount': 3}}],\n", 598 | " [ {'name': 'helm-chart', 'url': 'https://github.com/jupyterhub/helm-chart', 'issues': {'totalCount': 3}, 'pullRequests': {'totalCount': 2}}],\n", 599 | " [ {'name': 'binderhub', 'url': 'https://github.com/jupyterhub/binderhub', 'issues': {'totalCount': 79}, 'pullRequests': {'totalCount': 6}}],\n", 600 | " [ {'name': 'mybinder.org-deploy', 'url': 'https://github.com/jupyterhub/mybinder.org-deploy', 'issues': {'totalCount': 25}, 'pullRequests': {'totalCount': 1}}],\n", 601 | " [ {'name': 'binder', 'url': 'https://github.com/jupyterhub/binder', 'issues': {'totalCount': 9}, 'pullRequests': {'totalCount': 1}}],\n", 602 | " [ {'name': 'nullauthenticator', 'url': 'https://github.com/jupyterhub/nullauthenticator', 'issues': {'totalCount': 0}, 'pullRequests': {'totalCount': 0}}],\n", 603 | " [ {'name': 'team-compass', 'url': 'https://github.com/jupyterhub/team-compass', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 0}}]], dtype=object)" 604 | ] 605 | }, 606 | "execution_count": 26, 607 | "metadata": {}, 608 | "output_type": "execute_result" 609 | } 610 | ], 611 | "source": [ 612 | "df.values" 613 | ] 614 | }, 615 | { 616 | "cell_type": "code", 617 | "execution_count": 27, 618 | "metadata": {}, 619 | "outputs": [ 620 | { 621 | "data": { 622 | "text/plain": [ 623 | "" 651 | ] 652 | }, 653 | "execution_count": 27, 654 | "metadata": {}, 655 | "output_type": "execute_result" 656 | } 657 | ], 658 | "source": [ 659 | "df.sort_index" 660 | ] 661 | }, 662 | { 663 | "cell_type": "code", 664 | "execution_count": 28, 665 | "metadata": {}, 666 | "outputs": [], 667 | "source": [ 668 | "# output data to a csv\n", 669 | "# df.to_csv('issue_data.csv')" 670 | ] 671 | } 672 | ], 673 | "metadata": { 674 | "kernelspec": { 675 | "display_name": "Python 3", 676 | "language": "python", 677 | "name": "python3" 678 | }, 679 | "language_info": { 680 | "codemirror_mode": { 681 | "name": "ipython", 682 | "version": 3 683 | }, 684 | "file_extension": ".py", 685 | "mimetype": "text/x-python", 686 | "name": "python", 687 | "nbconvert_exporter": "python", 688 | "pygments_lexer": "ipython3", 689 | "version": "3.6.3" 690 | } 691 | }, 692 | "nbformat": 4, 693 | "nbformat_minor": 2 694 | } 695 | -------------------------------------------------------------------------------- /pyquery-ql/_version.py: -------------------------------------------------------------------------------- 1 | __version__ = '0.1.1dev' 2 | -------------------------------------------------------------------------------- /pyquery-ql/pyquery-ql.py: -------------------------------------------------------------------------------- 1 | """pyquery-ql.py 2 | 3 | Send a graphql query to GitHub 4 | and pretty print output. 5 | 6 | Usage: 7 | 8 | python3 pyquery-ql.py 9 | 10 | Supports Python 3.6+ 11 | 12 | """ 13 | import json 14 | import logging 15 | import os 16 | import pprint 17 | 18 | import requests 19 | 20 | logger = logging.getLogger(__name__) 21 | 22 | 23 | def send_query(url, query, headers): 24 | """Send a query. 25 | 26 | With GraphQL and Python requests, a `post` is used when querying, instead 27 | of a `get`. The `post` is used since the `query` must be sent to the 28 | GraphQL endpoint. This differs from REST where the API endpoint has the 29 | query in its url structure. 30 | 31 | Returns: 32 | result: query result 33 | 34 | """ 35 | result = requests.post(url=url, json={'query': query}, headers=headers) 36 | if result.status_code == 200: 37 | return result 38 | else: 39 | logger.warning(f"Request received error code: {result.status_code}") 40 | 41 | 42 | def pretty_print_query_response(response): 43 | """print the query response""" 44 | if response.text is not None: 45 | pprint.pprint(json.loads(response.text)) 46 | else: 47 | logger.warning('Response not valid.') 48 | 49 | 50 | def print_query_response(response): 51 | """print the query response""" 52 | if response.text is not None: 53 | print(json.loads(response.text)) 54 | else: 55 | logger.warning('Response not valid.') 56 | 57 | 58 | def create_auth_header(api_token): 59 | """create a dictionary for authorization header""" 60 | return {'Authorization': f'token {api_token}'} 61 | 62 | 63 | if __name__ == '__main__': 64 | 65 | api_token = os.environ['GITHUB_API_TOKEN'] 66 | 67 | # add authorization to headers 68 | headers = {} 69 | headers.update(create_auth_header(api_token)) 70 | 71 | url = 'https://api.github.com/graphql' 72 | 73 | query = """ 74 | { 75 | organization(login: "jupyterhub") { 76 | repositories(first: 26) { 77 | totalCount 78 | edges { 79 | node { 80 | name 81 | url 82 | issues(states: OPEN) { 83 | totalCount 84 | } 85 | pullRequests(states: OPEN) { 86 | totalCount 87 | } 88 | } 89 | } 90 | } 91 | } 92 | } 93 | """ 94 | 95 | response = send_query(url, query, headers) 96 | 97 | pretty_print_query_response(response) 98 | -------------------------------------------------------------------------------- /readthedocs.yml: -------------------------------------------------------------------------------- 1 | name: pyquery 2 | type: sphinx 3 | conda: 4 | file: docs/environment-doc.yml 5 | python: 6 | version: 3 7 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | requests>=2.18 -------------------------------------------------------------------------------- /test/test-data/data.json: -------------------------------------------------------------------------------- 1 | {"data": {"organization": {"repositories": {"nodes": [{"name": "sidecar", "url": "https://github.com/nteract/sidecar", "issues": {"totalCount": 6}, "pullRequests": {"totalCount": 1}}, {"name": "hydrogen", "url": "https://github.com/nteract/hydrogen", "issues": {"totalCount": 68}, "pullRequests": {"totalCount": 3}}, {"name": "jupyter-paths", "url": "https://github.com/nteract/jupyter-paths", "issues": {"totalCount": 2}, "pullRequests": {"totalCount": 0}}, {"name": "nteract", "url": "https://github.com/nteract/nteract", "issues": {"totalCount": 248}, "pullRequests": {"totalCount": 1}}, {"name": "nteract.github.io", "url": "https://github.com/nteract/nteract.github.io", "issues": {"totalCount": 5}, "pullRequests": {"totalCount": 0}}, {"name": "jupyter-display-area", "url": "https://github.com/nteract/jupyter-display-area", "issues": {"totalCount": 4}, "pullRequests": {"totalCount": 0}}, {"name": "transformime", "url": "https://github.com/nteract/transformime", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "transformime-jupyter-transformers", "url": "https://github.com/nteract/transformime-jupyter-transformers", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "transformime-commonmark", "url": "https://github.com/nteract/transformime-commonmark", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "jupyter-session", "url": "https://github.com/nteract/jupyter-session", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "jupyter-kernel-launcher", "url": "https://github.com/nteract/jupyter-kernel-launcher", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "jupyter-transport-wrapper", "url": "https://github.com/nteract/jupyter-transport-wrapper", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "ipython-paths", "url": "https://github.com/nteract/ipython-paths", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 1}}, {"name": "transformime-react", "url": "https://github.com/nteract/transformime-react", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "commutable", "url": "https://github.com/nteract/commutable", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "react-jupyter-display-area", "url": "https://github.com/nteract/react-jupyter-display-area", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "enchannel-zmq-backend", "url": "https://github.com/nteract/enchannel-zmq-backend", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "enchannel", "url": "https://github.com/nteract/enchannel", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 0}}, {"name": "notebook-test-data", "url": "https://github.com/nteract/notebook-test-data", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 0}}, {"name": "ick", "url": "https://github.com/nteract/ick", "issues": {"totalCount": 3}, "pullRequests": {"totalCount": 1}}, {"name": "terminal-sidecar", "url": "https://github.com/nteract/terminal-sidecar", "issues": {"totalCount": 3}, "pullRequests": {"totalCount": 0}}, {"name": "kernelspecs", "url": "https://github.com/nteract/kernelspecs", "issues": {"totalCount": 2}, "pullRequests": {"totalCount": 0}}, {"name": "spawnteract", "url": "https://github.com/nteract/spawnteract", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 0}}, {"name": "design", "url": "https://github.com/nteract/design", "issues": {"totalCount": 3}, "pullRequests": {"totalCount": 0}}, {"name": "specs", "url": "https://github.com/nteract/specs", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "kernel-relay", "url": "https://github.com/nteract/kernel-relay", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "enchannel-socketio-backend", "url": "https://github.com/nteract/enchannel-socketio-backend", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 0}}, {"name": "docs", "url": "https://github.com/nteract/docs", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 0}}, {"name": "logos", "url": "https://github.com/nteract/logos", "issues": {"totalCount": 2}, "pullRequests": {"totalCount": 1}}, {"name": "enchannel-in-memory", "url": "https://github.com/nteract/enchannel-in-memory", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "enchannel-notebook-backend", "url": "https://github.com/nteract/enchannel-notebook-backend", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 0}}, {"name": "ansi-to-react", "url": "https://github.com/nteract/ansi-to-react", "issues": {"totalCount": 3}, "pullRequests": {"totalCount": 1}}, {"name": "transformime-marked", "url": "https://github.com/nteract/transformime-marked", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "mathjax-electron", "url": "https://github.com/nteract/mathjax-electron", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 1}}, {"name": "assets", "url": "https://github.com/nteract/assets", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "associator", "url": "https://github.com/nteract/associator", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "notebook-preview", "url": "https://github.com/nteract/notebook-preview", "issues": {"totalCount": 3}, "pullRequests": {"totalCount": 1}}, {"name": "naming", "url": "https://github.com/nteract/naming", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "initiatives", "url": "https://github.com/nteract/initiatives", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 0}}, {"name": "term-launcher", "url": "https://github.com/nteract/term-launcher", "issues": {"totalCount": 2}, "pullRequests": {"totalCount": 0}}, {"name": "meeting-minutes", "url": "https://github.com/nteract/meeting-minutes", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "zmq-prebuilt-testing", "url": "https://github.com/nteract/zmq-prebuilt-testing", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 0}}, {"name": "oauth-server", "url": "https://github.com/nteract/oauth-server", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "nteract.io", "url": "https://github.com/nteract/nteract.io", "issues": {"totalCount": 9}, "pullRequests": {"totalCount": 1}}, {"name": "zmq-static", "url": "https://github.com/nteract/zmq-static", "issues": {"totalCount": 4}, "pullRequests": {"totalCount": 0}}, {"name": "inodejs", "url": "https://github.com/nteract/inodejs", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 1}}, {"name": "tick-tock", "url": "https://github.com/nteract/tick-tock", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "rx-jupyter", "url": "https://github.com/nteract/rx-jupyter", "issues": {"totalCount": 8}, "pullRequests": {"totalCount": 0}}, {"name": "libzmq-win", "url": "https://github.com/nteract/libzmq-win", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "commutable-perf", "url": "https://github.com/nteract/commutable-perf", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 0}}, {"name": "plotly.js", "url": "https://github.com/nteract/plotly.js", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "vega", "url": "https://github.com/nteract/vega", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "commuter", "url": "https://github.com/nteract/commuter", "issues": {"totalCount": 28}, "pullRequests": {"totalCount": 1}}, {"name": "snakestagram", "url": "https://github.com/nteract/snakestagram", "issues": {"totalCount": 8}, "pullRequests": {"totalCount": 0}}, {"name": "commutable-models", "url": "https://github.com/nteract/commutable-models", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "minimal-plotly", "url": "https://github.com/nteract/minimal-plotly", "issues": {"totalCount": 1}, "pullRequests": {"totalCount": 0}}, {"name": "desktop-manual", "url": "https://github.com/nteract/desktop-manual", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "papermill", "url": "https://github.com/nteract/papermill", "issues": {"totalCount": 11}, "pullRequests": {"totalCount": 1}}, {"name": "cabinet", "url": "https://github.com/nteract/cabinet", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 1}}, {"name": "papermillr", "url": "https://github.com/nteract/papermillr", "issues": {"totalCount": 0}, "pullRequests": {"totalCount": 0}}, {"name": "vdom", "url": "https://github.com/nteract/vdom", "issues": {"totalCount": 7}, "pullRequests": {"totalCount": 2}}]}}}} -------------------------------------------------------------------------------- /test/test-data/issue_data.csv: -------------------------------------------------------------------------------- 1 | ,node 2 | 0,"{'name': 'jupyterhub', 'url': 'https://github.com/jupyterhub/jupyterhub', 'issues': {'totalCount': 143}, 'pullRequests': {'totalCount': 6}}" 3 | 1,"{'name': 'configurable-http-proxy', 'url': 'https://github.com/jupyterhub/configurable-http-proxy', 'issues': {'totalCount': 10}, 'pullRequests': {'totalCount': 0}}" 4 | 2,"{'name': 'oauthenticator', 'url': 'https://github.com/jupyterhub/oauthenticator', 'issues': {'totalCount': 13}, 'pullRequests': {'totalCount': 2}}" 5 | 3,"{'name': 'dockerspawner', 'url': 'https://github.com/jupyterhub/dockerspawner', 'issues': {'totalCount': 22}, 'pullRequests': {'totalCount': 1}}" 6 | 4,"{'name': 'sudospawner', 'url': 'https://github.com/jupyterhub/sudospawner', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 0}}" 7 | 5,"{'name': 'batchspawner', 'url': 'https://github.com/jupyterhub/batchspawner', 'issues': {'totalCount': 12}, 'pullRequests': {'totalCount': 5}}" 8 | 6,"{'name': 'kubespawner', 'url': 'https://github.com/jupyterhub/kubespawner', 'issues': {'totalCount': 25}, 'pullRequests': {'totalCount': 3}}" 9 | 7,"{'name': 'ldapauthenticator', 'url': 'https://github.com/jupyterhub/ldapauthenticator', 'issues': {'totalCount': 18}, 'pullRequests': {'totalCount': 2}}" 10 | 8,"{'name': 'jupyterhub-deploy-docker', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-docker', 'issues': {'totalCount': 9}, 'pullRequests': {'totalCount': 1}}" 11 | 9,"{'name': 'jupyterhub-deploy-teaching', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-teaching', 'issues': {'totalCount': 13}, 'pullRequests': {'totalCount': 2}}" 12 | 10,"{'name': 'jupyterhub-tutorial', 'url': 'https://github.com/jupyterhub/jupyterhub-tutorial', 'issues': {'totalCount': 5}, 'pullRequests': {'totalCount': 0}}" 13 | 11,"{'name': 'jupyterhub-deploy-hpc', 'url': 'https://github.com/jupyterhub/jupyterhub-deploy-hpc', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 0}}" 14 | 12,"{'name': 'systemdspawner', 'url': 'https://github.com/jupyterhub/systemdspawner', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 2}}" 15 | 13,"{'name': 'wrapspawner', 'url': 'https://github.com/jupyterhub/wrapspawner', 'issues': {'totalCount': 5}, 'pullRequests': {'totalCount': 1}}" 16 | 14,"{'name': 'jupyterlab-hub', 'url': 'https://github.com/jupyterhub/jupyterlab-hub', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 0}}" 17 | 15,"{'name': 'nbserverproxy', 'url': 'https://github.com/jupyterhub/nbserverproxy', 'issues': {'totalCount': 2}, 'pullRequests': {'totalCount': 1}}" 18 | 16,"{'name': 'jupyterhub-example-kerberos', 'url': 'https://github.com/jupyterhub/jupyterhub-example-kerberos', 'issues': {'totalCount': 3}, 'pullRequests': {'totalCount': 0}}" 19 | 17,"{'name': 'hubshare', 'url': 'https://github.com/jupyterhub/hubshare', 'issues': {'totalCount': 10}, 'pullRequests': {'totalCount': 0}}" 20 | 18,"{'name': 'nbrsessionproxy', 'url': 'https://github.com/jupyterhub/nbrsessionproxy', 'issues': {'totalCount': 4}, 'pullRequests': {'totalCount': 2}}" 21 | 19,"{'name': 'tmpauthenticator', 'url': 'https://github.com/jupyterhub/tmpauthenticator', 'issues': {'totalCount': 0}, 'pullRequests': {'totalCount': 0}}" 22 | 20,"{'name': 'zero-to-jupyterhub-k8s', 'url': 'https://github.com/jupyterhub/zero-to-jupyterhub-k8s', 'issues': {'totalCount': 89}, 'pullRequests': {'totalCount': 3}}" 23 | 21,"{'name': 'helm-chart', 'url': 'https://github.com/jupyterhub/helm-chart', 'issues': {'totalCount': 3}, 'pullRequests': {'totalCount': 0}}" 24 | 22,"{'name': 'binderhub', 'url': 'https://github.com/jupyterhub/binderhub', 'issues': {'totalCount': 75}, 'pullRequests': {'totalCount': 5}}" 25 | 23,"{'name': 'mybinder.org-deploy', 'url': 'https://github.com/jupyterhub/mybinder.org-deploy', 'issues': {'totalCount': 24}, 'pullRequests': {'totalCount': 1}}" 26 | 24,"{'name': 'binder', 'url': 'https://github.com/jupyterhub/binder', 'issues': {'totalCount': 9}, 'pullRequests': {'totalCount': 1}}" 27 | 25,"{'name': 'nullauthenticator', 'url': 'https://github.com/jupyterhub/nullauthenticator', 'issues': {'totalCount': 0}, 'pullRequests': {'totalCount': 0}}" 28 | -------------------------------------------------------------------------------- /test/test-data/mydata.csv: -------------------------------------------------------------------------------- 1 | name,url,issues,prs 2 | sidecar,https://github.com/nteract/sidecar,6,1 3 | hydrogen,https://github.com/nteract/hydrogen,68,3 4 | jupyter-paths,https://github.com/nteract/jupyter-paths,2,0 5 | nteract,https://github.com/nteract/nteract,248,1 6 | nteract.github.io,https://github.com/nteract/nteract.github.io,5,0 7 | jupyter-display-area,https://github.com/nteract/jupyter-display-area,4,0 8 | transformime,https://github.com/nteract/transformime,0,0 9 | transformime-jupyter-transformers,https://github.com/nteract/transformime-jupyter-transformers,0,0 10 | transformime-commonmark,https://github.com/nteract/transformime-commonmark,0,0 11 | jupyter-session,https://github.com/nteract/jupyter-session,0,0 12 | jupyter-kernel-launcher,https://github.com/nteract/jupyter-kernel-launcher,0,0 13 | jupyter-transport-wrapper,https://github.com/nteract/jupyter-transport-wrapper,0,0 14 | ipython-paths,https://github.com/nteract/ipython-paths,1,1 15 | transformime-react,https://github.com/nteract/transformime-react,0,0 16 | commutable,https://github.com/nteract/commutable,0,0 17 | react-jupyter-display-area,https://github.com/nteract/react-jupyter-display-area,0,0 18 | enchannel-zmq-backend,https://github.com/nteract/enchannel-zmq-backend,0,0 19 | enchannel,https://github.com/nteract/enchannel,1,0 20 | notebook-test-data,https://github.com/nteract/notebook-test-data,1,0 21 | ick,https://github.com/nteract/ick,3,1 22 | terminal-sidecar,https://github.com/nteract/terminal-sidecar,3,0 23 | kernelspecs,https://github.com/nteract/kernelspecs,2,0 24 | spawnteract,https://github.com/nteract/spawnteract,1,0 25 | design,https://github.com/nteract/design,3,0 26 | specs,https://github.com/nteract/specs,0,0 27 | kernel-relay,https://github.com/nteract/kernel-relay,0,0 28 | enchannel-socketio-backend,https://github.com/nteract/enchannel-socketio-backend,1,0 29 | docs,https://github.com/nteract/docs,1,0 30 | logos,https://github.com/nteract/logos,2,1 31 | enchannel-in-memory,https://github.com/nteract/enchannel-in-memory,0,0 32 | enchannel-notebook-backend,https://github.com/nteract/enchannel-notebook-backend,1,0 33 | ansi-to-react,https://github.com/nteract/ansi-to-react,3,1 34 | transformime-marked,https://github.com/nteract/transformime-marked,0,0 35 | mathjax-electron,https://github.com/nteract/mathjax-electron,1,1 36 | assets,https://github.com/nteract/assets,0,0 37 | associator,https://github.com/nteract/associator,0,0 38 | notebook-preview,https://github.com/nteract/notebook-preview,3,1 39 | naming,https://github.com/nteract/naming,0,0 40 | initiatives,https://github.com/nteract/initiatives,1,0 41 | term-launcher,https://github.com/nteract/term-launcher,2,0 42 | meeting-minutes,https://github.com/nteract/meeting-minutes,0,0 43 | zmq-prebuilt-testing,https://github.com/nteract/zmq-prebuilt-testing,1,0 44 | oauth-server,https://github.com/nteract/oauth-server,0,0 45 | nteract.io,https://github.com/nteract/nteract.io,9,1 46 | zmq-static,https://github.com/nteract/zmq-static,4,0 47 | inodejs,https://github.com/nteract/inodejs,0,1 48 | tick-tock,https://github.com/nteract/tick-tock,0,0 49 | rx-jupyter,https://github.com/nteract/rx-jupyter,8,0 50 | libzmq-win,https://github.com/nteract/libzmq-win,0,0 51 | commutable-perf,https://github.com/nteract/commutable-perf,1,0 52 | plotly.js,https://github.com/nteract/plotly.js,0,0 53 | vega,https://github.com/nteract/vega,0,0 54 | commuter,https://github.com/nteract/commuter,28,1 55 | snakestagram,https://github.com/nteract/snakestagram,8,0 56 | commutable-models,https://github.com/nteract/commutable-models,0,0 57 | minimal-plotly,https://github.com/nteract/minimal-plotly,1,0 58 | desktop-manual,https://github.com/nteract/desktop-manual,0,0 59 | papermill,https://github.com/nteract/papermill,11,1 60 | cabinet,https://github.com/nteract/cabinet,0,1 61 | papermillr,https://github.com/nteract/papermillr,0,0 62 | vdom,https://github.com/nteract/vdom,7,2 63 | --------------------------------------------------------------------------------