├── .gitignore ├── LICENSE ├── README.md ├── accepted └── migration-to-python-3-only.md ├── community.md ├── companion-files.md ├── dashboards.md ├── declarativewidgets.md ├── deployment.md ├── documentation.md ├── ipython.md ├── ipywidgets.md ├── jupyterhub.md ├── jupyterlab.md ├── nbconvert.md ├── notebook.md ├── other.md ├── qtconsole.md ├── spark.md └── traitlets.md /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | 5 | # C extensions 6 | *.so 7 | 8 | # Distribution / packaging 9 | .Python 10 | env/ 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | *.egg-info/ 23 | .installed.cfg 24 | *.egg 25 | 26 | # PyInstaller 27 | # Usually these files are written by a python script from a template 28 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 29 | *.manifest 30 | *.spec 31 | 32 | # Installer logs 33 | pip-log.txt 34 | pip-delete-this-directory.txt 35 | 36 | # Unit test / coverage reports 37 | htmlcov/ 38 | .tox/ 39 | .coverage 40 | .coverage.* 41 | .cache 42 | nosetests.xml 43 | coverage.xml 44 | *,cover 45 | 46 | # Translations 47 | *.mo 48 | *.pot 49 | 50 | # Django stuff: 51 | *.log 52 | 53 | # Sphinx documentation 54 | docs/_build/ 55 | 56 | # PyBuilder 57 | target/ 58 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015, Project Jupyter 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | * Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | * Neither the name of roadmap nor the names of its 15 | contributors may be used to endorse or promote products derived from 16 | this software without specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 22 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 23 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 25 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 26 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | 29 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Project Jupyter Roadmap 2 | 3 | The following is a very rough outline of the roadmap for Project Jupyter and IPython. 4 | Historically, we have published major releases every 6-9 months. With our project now 5 | split across many subprojects, we expect to release subprojects separately and more 6 | often. 7 | 8 | This roadmap only includes mainline subprojects under the Jupyter 9 | (https://github.com/jupyter) and IPython (https://github.com/ipython) GitHub orgs. We 10 | also have a number of newer, experimental subprojects in the Jupyter Incubator 11 | (https://github.com/jupyter-incubator) organization. 12 | 13 | In this roadmap, we refer to the future major releases as follows: 14 | 15 | * Next (the next major release) 16 | * Next+1 (the major release after that) 17 | * Future (any release beyond the Next+1) 18 | 19 | This reflects the inherent uncertainty in releases that are more than about 1 year away. In 20 | some cases, if progress is made more quickly, features may be moved to earlier releases. 21 | 22 | ## Subprojects ## 23 | 24 | * [IPython](ipython.md) 25 | * [Jupyter Notebook](notebook.md) 26 | * [JupyterLab](jupyterlab.md) 27 | * [JupyterHub](jupyterhub.md) 28 | * [Documentation](documentation.md) 29 | * [Deployment](deployment.md) 30 | * [ipywidgets](ipywidgets.md) 31 | * [Declarative Widgets](declarativewidgets.md) 32 | * [nbconvert](nbconvert.md) 33 | * [qtconsole](qtconsole.md) 34 | * [Dashboards](dashboards.md) 35 | * [Traitlets](traitlets.md) 36 | * [Other Things](other.md) 37 | -------------------------------------------------------------------------------- /accepted/migration-to-python-3-only.md: -------------------------------------------------------------------------------- 1 | # Migration to Python 3 only 2 | 3 | This document discuss some of the various possibilities to migrate Jupyter, and 4 | its reference Kernel implementation IPython, toward a Python 3 only code base, 5 | without breaking installation for Python 2 users. 6 | 7 | The document is not meant to debate whether the Jupyter & IPython project should migrate 8 | to a Python 3 only code base, but the timeline of such a migration. 9 | 10 | 11 | # Too long, Didn't read (TL;DR:). 12 | 13 | - High Priority: Fix Pip to refuse to install Python 3 only packages on 14 | Python 2, or to accept the `py3` only suffix on source distributions. High 15 | priority as we want a new pip version to be released and propagate, but 16 | it does not have to be done before IPython 6.0. 17 | - June 2016 : Release IPython 5.0 LTS 18 | - Traitlets "LTS" 19 | - ipykernel LTS 20 | - ipython_genutils LTS 21 | - January 2017 : Release IPython 6.0 Python 3 only. 22 | - July 2017 : Release Notebook 6.0 Python 3 only. 23 | - December 2017: IPython 7.0 24 | - December 2017: End of active support IPython 5.x LTS 25 | - July 2019 : Complete end of support for IPython 5.x 26 | 27 | 28 | 29 | # Statement of the problem 30 | 31 | We recognize that many users of IPython and Jupyter still need to use a Python 2 32 | environment, and will still need to do so for quite some time. However, as an open 33 | source project, there is an overhead in maintaining Python 2 and 3 compatibility, 34 | and this slows progress on our goals. 35 | 36 | We also believe that Python 3 is sufficiently mature, and enough projects in 37 | the scientific community are ready, to make the commitment to stop having new 38 | release with active Python 2 support. 39 | 40 | We want to provide a seamless experience to current Python 3 users, without 41 | breaking any workflow and pipeline and installation for Python 2 users. Thus we 42 | want in a relatively short period of time to remove the burden of supporting 43 | Python 2 support from the core team, while preserving functionality for users 44 | of both Python 2 and Python 3. 45 | 46 | We also recognize that there are many companies and laboratories relying on 47 | Python 2, and we want to offer a solution that works for them too. 48 | 49 | # IPython 5.x LTS 50 | 51 | IPython is the reference kernel for Jupyter. This kernel relies on 52 | `ipykernel`, `traitlets` and `ipython_genutils`. 53 | 54 | We propose that the next major version cycle (5.x) of IPython be the last 55 | major version branch to support Python 2. As Python 2 is supported by the CPython core team 56 | until 2020, we propose to make the 5.x major cycle a Long Term Support version: 57 | 58 | - For the 18 months following the release of IPython 5.0, the core team will 59 | provide critical & security bug fixes for the 5.x branch, as well as 60 | regular releases in the 5.x series. 61 | This also applies to the documentation of the 5.x branch. 62 | 63 | - For the 36 months following the release of IPython 5.0, the core team will 64 | continue to review critical & security related patches, as well as 65 | improvements to the documentation, and make related release if necessary. The 66 | core team will make sure that the 5.x version of IPython is described in 67 | our installation instructions. 68 | 69 | - The core team will fix bugs affecting IPython in the necessary versions of 70 | traitlets, ipykernel and ipython_genutils - IPython's first-party 71 | dependencies. 72 | 73 | - Past 36 months after the IPython 5.0 release, IPython 5.x will be considered 74 | unsupported. 75 | 76 | - These time lines are a statement of intent; they may change in the future. 77 | However, we do not plan to support Python 2 beyond its end of life in 2020. 78 | 79 | - Third parties may support IPython on Python 2 for longer, either as a 80 | commercial deal or in the open. IPython is BSD licensed, so anyone may 81 | modify and redistribute the code. 82 | 83 | The above timeline would lead to a release of IPython 5.0 about mid 2016, 84 | leading to active 5.x LTS support until about winter 2017, and passive LTS 85 | support until mid 2019. 86 | 87 | ## Challenges: 88 | 89 | 90 | ### Dependencies 91 | 92 | IPython and ipykernel rely on a few dependencies maintained by the same groups 93 | of people, in particular: `ipykernel`, `traitlets`, and `ipython_genutils`. In 94 | order to provide LTS support for IPython, the IPython team will provide 95 | patches for the packages required by IPython 5.x if they 96 | affect the behavior of IPython. Otherwise our normal conventions apply, and only 97 | the current stable version of these packages will be supported. 98 | 99 | 100 | ### Source distribution installation. 101 | 102 | 103 | The Python package manager (`pip`) does not respect the `requires-python` 104 | metadata field, and `setuptools` does not provide a way to set a value of this 105 | metadata field as far as we know. This implies that if the IPython core team release a version of 106 | IPython that is not compatible with Python 2, a user installing IPython 107 | by running `pip install ipython` under Python 2 will end up with a 108 | broken installation. 109 | 110 | There is an open issue on pip to check compatibility with Python when installing 111 | packages: 112 | 113 | https://github.com/pypa/pip/issues/3182 114 | 115 | This will not be an issue before we actually release a source version 116 | of a package which is not Python 2 compatible, but we likely want to have 117 | this fixed in pip as soon as possible, so that users have a chance to upgrade to 118 | this version **before** upgrading IPython. 119 | 120 | There are alternative ways around that, like making `ipython` a metapackage which 121 | expresses a dependency on (say) `ipython_real` 122 | 123 | #### Multiple source distributions 124 | 125 | Alternatively, we could use a hidden feature of `pip`. Source distributions whose 126 | base name ends in `-pyX.Y` are considered for installation only on a matching 127 | Python version. Thus instead of publishing a single `.tar.gz`, we 128 | could publish multiple identical source distribution with various names. 129 | 130 | For supported python versions: 131 | 132 | - ipython-6.x-py3.4.tar.gz 133 | - ipython-6.x-py3.5.tar.gz 134 | 135 | For future python versions: 136 | 137 | - ipython-6.x-py3.6.tar.gz 138 | - ipython-6.x-py3.7.tar.gz 139 | 140 | 141 | It may be possible to patch pip to handle a more general form like 142 | `ipython-6.x-py3.tar.gz`, so we only need to publish a single sdist. 143 | 144 | # Notebook & Python 3 support. 145 | 146 | The Jupyter Protocol is designed to be language agnostic, so the language the 147 | Notebook server (or any frontend) is written in does not have to match the 148 | language of code within a notebook. 149 | 150 | For the next+1 version of the Notebook server (6.0), we propose to require Python 3. 151 | 152 | The rationale being that the notebook application is one of those that can 153 | benefit the most from recent Python improvement, such as asyncio or function annotations. 154 | 155 | Unlike for IPython, we do not believe a LTS support is necessary, as a Python 3 156 | only version of the notebook application can perfectly run a Python 2 kernel. 157 | 158 | In order to allow a progressive transition to Python 3 for users and downstream 159 | integrators, we propose to release the first Python 3 only version of the 160 | notebook 6 months after the first release of the Python 3 only version of IPython. 161 | 162 | We propose to clearly communicate on the next release of the notebook (5.0), that 163 | one of the next versions will require a Python 3 environment. 164 | 165 | Timelines: 166 | - Notebook 5.0 – Python 2 compatible – Summer/Fall 2016 167 | - IPython 6.0 – Python 3 Only – Early 2017 168 | - Notebook 6.0 (or 7.0) – Python 3 only – Mid 2017 169 | 170 | 171 | ## Challenges 172 | 173 | As for IPython above; the notebook package need to be able to express that it 174 | requires Python 3. This is mitigated by people installing `jupyter`: we can 175 | make that conditionally depend on different version of `notebook`, depending on 176 | the version of Python. 177 | 178 | In particular for Anaconda, one path forward would be to bundle the 179 | notebook with its own Python 3 interpreter. This can lead to some 180 | inconsistency for notebook server extensions, as on Python 2 Anaconda, the 181 | notebook server will be in a separate environment, and the notebook extensions 182 | and optional dependencies need to be installed in this environment as well. We 183 | are thus speaking of `nbconvert`, custom exporters, custom handler and server 184 | side extension. 185 | 186 | This is not a problem for most Linux package managers, where the norm is to 187 | treat Python 2 and 3 as distinct packages. In this context, the notebook will 188 | have a dependency on `python3`. 189 | 190 | # Not in this roadmap: 191 | 192 | 193 | - `nbconvert` CLI: Can likely be migrated to Python 3 on roughly the same 194 | schedule as the notebook. 195 | - `nbformat` : keep Python 2 support as long as we support IPython 196 | 5.x, maybe later, to help people work with newer notebook file format versions 197 | and upgrade older notebooks to newer formats. 198 | - `jupyter_client` : Suggest keeping some Python 2 compatibility as this might be helpful for some downstream project. 199 | - `ipyparallel`: Decision is on ipyparallel users/devs. 200 | - `ipywidgets` 201 | - `nbgrader` 202 | - `jupyterhub` is already Py3 only 203 | - `jupyter_console` 204 | 205 | 206 | # Alternatives not discussed: 207 | 208 | - keep Python 2 support: unless an external entity comes with sufficient 209 | contributions for a longer enough amount of time to IPython/Jupyter and offer 210 | to support Python 2, Python 2 support will be discontinued. 211 | 212 | - Is there a way to develop using Python 3 only and have a 3to2 conversion 213 | process to generate Python 2 compatible versions. 214 | 215 | 216 | # Misc 217 | 218 | We propose to move forward with signing something similar to "no more legacy python pledge" 219 | 220 | https://gist.github.com/brettcannon/2f6926dc6a7874478ccd 221 | 222 | Which text should be refined and get feedback from both Python 2 and 3 supporters. 223 | 224 | 225 | Scikit-bio rfc to drop Python 2 support: 226 | 227 | https://github.com/biocore/scikit-bio-rfcs/blob/master/accepted/002-py3-only.md 228 | 229 | 230 | - Make it explicit on website, PyPI, GitHub, and documentations that 5.x is LTS for Python 2. 231 | - Make it explicit on IPython 5.x install that it is LTS, and that IPython 6.0 is/will be Python 3 only. 232 | -------------------------------------------------------------------------------- /community.md: -------------------------------------------------------------------------------- 1 | ## Community Pipeline 2 | 3 | ### Next 4 | 5 | - Metrics: looking at contributions (e.g. percent from non-maintainers, 6 | distribution of number of contributions) 7 | - Create a basic "overview of Jupyter" page with at least a listing of all the 8 | projects in the Jupyter repo plus a short description of each 9 | - Create the common development guide for all Jupyter projects 10 | - Create the repo maintainer checklist and identify which repos need to add 11 | which things from the checklist. Carol will take a first pass at this. 12 | - Come up with a checklist/cheatsheet for developers. This should include things 13 | like the repo maintainer checklist as well as a guidelines for how to respond 14 | to issues and PRs, conduct, etc. This should be linked to from the Jupyter 15 | incubation instructions and from the development guide. 16 | - Create thoughtful replies for common reasons that a PR might be closed (e.g. 17 | pep8 changes). We can't actually use the GitHub saved replies for this because 18 | those are per-person rather per repo, but we can write a template for people 19 | to use at least. 20 | 21 | ### Next + 1 22 | 23 | - Continue fleshing out the "overview of Jupyter" page and include things like: 24 | * a dependency tree 25 | * better descriptions of how the projects relate to each other 26 | * status indicators 27 | - All projects should link to the common development guide and should satisfy 28 | all of the maintainer checklist requirements. 29 | - Grant access to 1 or 2 people who are not maintainers to be issue triagers -- 30 | this will just involve labeling issues as they come in and going back through 31 | old PRs/issues periodically. 32 | - Include in the newsletter both a "maintainer of the month" (someone with 33 | commit bits) and a "contributor of the month" (someone without commit bits) 34 | - Include in the newsletter a list of new contributors since the last newsletter 35 | 36 | ### Future 37 | 38 | - Create a bot to take actions on issues based on labels (e.g. "duplicate" gets 39 | closed automatically). 40 | - Create a mentorship mailing list (based on the Python Core Mentorship mailing 41 | list) for new contributors to ask questions 42 | -------------------------------------------------------------------------------- /companion-files.md: -------------------------------------------------------------------------------- 1 | ## Companion files 2 | 3 | Discussed Spring 2016, will require new versions of notebook and ?nbformat. 4 | 5 | We want to store more volatile information such as widget existence and state 6 | when saving notebooks. However, putting this kind of data in `.ipynb` files 7 | would make storing them in version control really inconvenient. We know that 8 | version controlling notebooks is already somewhat painful, and we don't want to 9 | make it worse. 10 | 11 | *Companion files* are how we intend to solve this. On save, if there is any data 12 | to store in it, a companion file will be written next to the `.ipynb` file. 13 | On load, if a companion file exists, its data will be recombined with the 14 | notebook file as far as possible. Where they do not match up, the `.ipynb` is 15 | considered canonical, and data from the companion file may be discarded. 16 | 17 | * The companion file will be used by ipywidgets to store widget state, but is 18 | also available to use for other data. 19 | * We may introduce an option to store large/binary(/all?) outputs in the 20 | companion file rather than base64 encoded in the `.ipynb` file. 21 | * Companion files are designed to be excluded from version control; if checked 22 | in, they should be treated as a blob of binary data, and not subject to 23 | line-based diffing and merging. 24 | * Each notebook has a single companion file. We discussed storing a directory 25 | of companion files: that may be better in some ways for version control, but 26 | directories can't be conveniently sent as email attachments. This is a 27 | compromise between the two use cases. 28 | * Companion files will be zip files: this provides a namespace to store 29 | different pieces of data, and easy access for other tools. We chose a binary 30 | format deliberately, so that tools like git do not try to diff or merge it as 31 | a text file. At the same time, it's possible that e.g. Github could offer a 32 | nicer UI showing the contents of the zip file. 33 | 34 | An alternate option could be to have the companion file be essentially the ipynb itself with the widget state in the metadata. On save, you would get something like: 35 | 36 | `foobar.ipynb` (no widget state) 37 | `foobar.full.ipynb` (contains widget state etc) 38 | 39 | To avoid the churn, we could also decide to remove anything that is output from the first one. A default `.gitignore` would ignore the full version, but you could still send it by email to someone with the inline figures an all. 40 | 41 | The widget state in the full file could be encoded in base64... 42 | 43 | ### Questions still to be clarified 44 | 45 | * What would the filename conventions be inside the zip file? e.g. companion.zip/jupyter-js-widgets.json and a companion.zip/metadata/ sub-folder for user things? 46 | 47 | ### Notes 48 | 49 | * In [PR 19](https://github.com/jupyter/roadmap/pull/19) we discussed the possibility of using [asar](https://github.com/electron/asar) instead of zip. For now, in the interest of simplicity of the main Python implementation, we'll stick to zip (which works out of the box in Python and is widely supported). We can revisit the asar idea later if zip proves problematic in practice. 50 | -------------------------------------------------------------------------------- /dashboards.md: -------------------------------------------------------------------------------- 1 | ## Dashboarding 2 | 3 | Discussed during the Spring 2016 dev meeting. 4 | 5 | ### Graduate jupyter/dashboards from Incubator 6 | 7 | The [dashboards wiki](https://github.com/jupyter-incubator/dashboards/wiki) summarizes the current scope of the dashboarding effort in jupyter incubator. We discussed graduation of these assets and agreed on the following rough steps. 8 | 9 | 1. Create an enhancement proposal to define an extensible layout metadata schema within the Jupyter notebook document format. 10 | * Use the jupyter-incubator/dashboards and Anaconda-server/nbpresent metadata as a guide for what needs to be supported. 11 | * https://github.com/jupyter/enhancement-proposals/pull/15 12 | 2. Update the incubator projects to work with the new schema and deprecate support for the older `urth.layout` schema. 13 | * https://github.com/jupyter-incubator/dashboards/wiki/Dashboard-Metadata-and-Rendering 14 | * https://github.com/jupyter-incubator/dashboards/issues/243 15 | 3. Submit an enhancement proposal to graduate the jupyter-incubator/dashboards project from incubator as a "classic" Jupyter Notebook extension. 16 | * Will be submitted when [v0.6.0 is released](https://github.com/jupyter-incubator/dashboards/milestones/0.6.0) 17 | 4. For now, keep the other two dashboard projects ([jupyter-incubator/dashboards_bunders](https://github.com/jupyter-incubator/dashboards_bundlers) and [jupyter-incubator/dashboards_server](https://github.com/jupyter-incubator/dashboards_server)) in incubator to continue exploration of the bundling and external deployment of notebooks as dashboards in incubator (e.g., security beyond trusted users). 18 | 19 | ### Dashboarding Support in Jupyter Lab 20 | 21 | We also discussed actions to be taken post-graduation, primarily with respect to dashboarding support in Jupyter Lab. We agreed on the following rough steps. 22 | 23 | 1. Implement a dashboard layout editor/viewer in Jupyter Lab compatible with the layout metadata schema and jupyter-incubator/dashboards use of that schema to layout dashboards. 24 | 2. Improve upon the layout authoring experience afforded in the "classic" Notebook by taking advantage of the multi-view capabilities of the Jupyter Lab interface. 25 | 3. Add a URL in the single user notebook server that shows a rendered dashboard given a notebook that follows the layout metadata schema. 26 | * TBD what renderer implementation does the rendering at the URL (e.g., new Jupyter Lab code?) 27 | 4. Possibly bundle the `jupyter/dashboards` extension in a Jupyter Notebook release. 28 | -------------------------------------------------------------------------------- /declarativewidgets.md: -------------------------------------------------------------------------------- 1 | ## Declarative widgets 2 | 3 | ### Next 4 | 5 | * Decide where to draw the line between Web Components specific functionality and generic code that should live closer to ipywidgets / jupyter-js-widgets to prevent future breakage. 6 | * Migrate generic, JS modules to jupyter-js-widgets (may be a no-op). 7 | * Graduate the declarativewidgets from the incubator. Specifically the move jupyter-incubator/declarativewidgets code to jupyter/jupyter-web-components or another appropriately named repo. 8 | * Investigate with the jupyter/notebook team about upstreaming a generic bower/npm restful API. 9 | * Determine the best location for kernel specific code. A possible options it to move Python bits under ipywidgets, move the R code into the IRkernel and the Scala code into Apache Toree or related project. 10 | -------------------------------------------------------------------------------- /deployment.md: -------------------------------------------------------------------------------- 1 | ## Deployment 2 | 3 | - provide reference "best practice" deployment of JuptyerHub, via Ansible scripts or otherwise 4 | -------------------------------------------------------------------------------- /documentation.md: -------------------------------------------------------------------------------- 1 | ## Documentation 2 | 3 | We list documentation as a top-level section in our roadmap to acknowledge the importance 4 | of it for developer communication, scaling and maintenance of deployments, and early user 5 | adoption of new releases. The state of the existing documentation has not kept pace with 6 | the rapid development in our projects, and effort will be focused on improving it. Because 7 | our documentation spans all subprojects, each of which have their own release cycles and 8 | versioning, the documentation roadmap doesn't have concrete version numbers associated with 9 | each milestone. 10 | 11 | ### Next 12 | 13 | * Hire full time and part time staff who will focus on improving our documentation. 14 | * Research existing tools for writing documentation (Readthedocs, Sphinx, Notebooks, etc.) 15 | and evaluate which tools we use to write documentation. 16 | * Develop a project wide outline of our existing and proposed documentation. 17 | * Perform observational and survey based user studies of our documentation to guide its 18 | design and development. 19 | * Explore and research the questions around how to maintain documentation across multiple 20 | subprojects. Do we centralize all or some of our documentation? Do we keep it in separate 21 | subproject repos? 22 | * Build publicly accessible API documentation on all commits for all JavaScript and 23 | TypeScript repositories. 24 | * Encode documentation in our project principles and community process as a core activity 25 | that is part of the everyday process of development, and adopt this in an integral 26 | fashion throughout the project for all project members. 27 | 28 | ### Next+1 29 | 30 | * Integrate Jupyter Notebooks more closely into our documentation tools. 31 | * Develop a formal documentation process and standard that includes technical review, 32 | organizational review, design, usability testing and copy-editing.* Where needed, develop 33 | additional tools for writing our documentation. 34 | * Work with each subproject to integrate our documentation tools and process into their 35 | workflow and roadmap. 36 | 37 | ### Future 38 | 39 | * Begin to block releases of subprojects until documentation meets our standards. 40 | * Using our new JavaScript packages and live kernels hosted in the cloud, enable 41 | documentation to be runnable in place. 42 | * Build documentation indexing and search into JupyterLab. 43 | * Build tools that make it easier for downstream projects to distribute notebook based 44 | documentation to users. 45 | -------------------------------------------------------------------------------- /ipython.md: -------------------------------------------------------------------------------- 1 | ## IPython 2 | 3 | IPython is comparatively mature, and there are consequently fewer major changes 4 | planned. However, we plan to: 5 | 6 | ### IPython 5.0 7 | 8 | Released planned mid 2016. 9 | 10 | #### Switch to prompt toolkit. 11 | 12 | Switch from readline to prompt_toolkit for terminal interaction; this should 13 | simplify installation and provide a richer experience in the terminal. 14 | The switch to `prompt_toolkit` will also imply dropping the readline interface. 15 | 16 | #### Last IPython version compatible with Python 2 and LTS 17 | 18 | See [Migration to Python 3 only document](accepted/migration-to-python-3-only.md), 19 | 20 | IPython 5.0 would be the last version to work on Python 2. Active long term support 21 | is planned until Fall/Winter 2017, and passive support until mid 2019. 22 | 23 | 24 | ### IPython 6.0 25 | 26 | 27 | #### Refactor tab completion: 28 | 29 | Refactor the completion machinery, and investigate using 30 | [Jedi](https://jedi.readthedocs.org/en/latest/) for completing in regular 31 | Python code (separate completion machinery is still needed for our special 32 | syntax). 33 | 34 | #### Drop Python 2 35 | 36 | - Deletion and removal of various Python 2 shims and code path. 37 | - Make sure upgrade process does not upgrade on Python 2. 38 | -------------------------------------------------------------------------------- /ipywidgets.md: -------------------------------------------------------------------------------- 1 | ## ipywidgets 2 | 3 | ### Next (5.0) 4 | 5 | * Improved and consistent visual style of built-in widgets. 6 | * Hands off approach to styling. ipywidgets only provides an API for 7 | adding/removing CSS classes and the Style API. The Style API provides the 8 | ability to set layout CSS traits in a mutable fashion. All other styling APIs 9 | are deprecated. 10 | * Include a new interact API that behaves like Sage's equivalent. 11 | * Use npm for packaging the JavaScript code as `jupyter-js-widgets`. 12 | * Use the new notebook nbextension mechanism, introduced in Jupyter notebook 4.2 13 | for the widgetsnbextension package, which will contain all of the notebook 14 | specific widget Javascript. 15 | * Include two mechanisms for exporting widgets to static formats, rasterized 16 | screencaptures of the widgets and an embed widget manager which is capable of 17 | rendering widgets outside of the notebook. 18 | * Add examples for using the widgets outside of the notebook. 19 | * Add narrative docs describing the widgets basic usage and low level widget 20 | spec information. 21 | 22 | ### Next+1 (6.0) 23 | 24 | * Unit tests for jupyter-js-widgets individual widgets 25 | * Unit tests for widgetsnbextension, especially manager.js 26 | * Enable a modern transpiler, either ES6 or TypeScript. Gradually transition 27 | the code as we make changes elsewhere. 28 | * Evaluate and possibly transition the codebase to use modern Javascript 29 | standards. Linting tools (i.e. eslint) should be used to assist developers 30 | into complying to said standard. 31 | * Integrate framework/foundation classes of the Declaritive Widget incubator 32 | directly into jupyter-js-widgets and ipywidgets. 33 | * Shrink the jupyter-js-widgets codebase using well adopted Javascript libraries 34 | installed via npm. 35 | * Treat widgets as regular output, and not have to have a separate widget area 36 | in the DOM. 37 | * Remove dependency on jQuery, jQuery UI, and bootstrap. 38 | * Integrate existing widgets with JupyterLab. 39 | * Explore mechanisms to allow Phosphor views to be used directly with 40 | jupyter-js-widget models. 41 | * Allow widgets to be used at the top-level of the JupyterLab UI to create dashboards. 42 | * Give the widgets a facelift. Maybe material-ui? Home spun design? 43 | * Add more synchronized events. In the process, investigate slicker ways to do 44 | cross context event synchronization. 45 | * Investigate making the model layer of widgets more symmetric, allowing 46 | widgets to be created, with traits, entirely in the front-end. 47 | 48 | ### Future 49 | 50 | * Extract the widget module synchronization layer into it's own generic project, 51 | maybe renaming to "XObject" or something similar along the way. The existing 52 | widgets will remain as-is, but will connect to this layer instead of to comms 53 | directly. 54 | 55 | * Regarding embeddable widgets, enable a mechanism for it to work with custom widget models and views to without having to manually put more with script tags on the page. We discussed multiple approaches during our meeting on the integration of a module version number so that we could compose unpkg urls for custom models. 56 | 57 | * Datetime and Date traits with serialization + Date picker and date time picker. The date traits should work with the time-zone issues between browser and backend as discussed earlier. 58 | 59 | * Use jupyter-js-outputarea for the ouput widget. 60 | 61 | -------------------------------------------------------------------------------- /jupyterhub.md: -------------------------------------------------------------------------------- 1 | ## JupyterHub 2 | 3 | The JupyterHub roadmaps are listed within their respective repositories. See the 4 | [roadmaps and milestones](https://jupyterhub-team-compass.readthedocs.io/en/latest/milestones.html) 5 | page on the [JupyterHub team compass](https://jupyterhub-team-compass.readthedocs.io) website 6 | for a list of the latest roadmaps and milestones for these projects. 7 | -------------------------------------------------------------------------------- /jupyterlab.md: -------------------------------------------------------------------------------- 1 | ## JupyterLab 2 | 3 | ## Beta Release 4 | * Flexible, customizable tabbed and paneled layout system. 5 | * All of JupyterLab will be a set of plugins that exchange runtime APIs. 6 | * Initial plugins: 7 | - Menu bar. 8 | - Overall page layout. 9 | - File browser. 10 | - Text editor. 11 | - Terminal. 12 | - Notebook. 13 | - Console. 14 | * Application infrastucture (keyboard shortcuts, commands, menus, layout), view 15 | layer, and other utilities (signals, properties, messages) provided by phosphorjs 16 | (http://phosphorjs.github.io/). 17 | * JupyterLab now has its own GitHub organization: https://github.com/jupyterlab 18 | * Movement towards having stable, documented JavaScript APIs. 19 | * Usability and UX User survey to help us guide the design process of JupyterLab. 20 | * Draggable standalone output area. 21 | * Ability to hook kernels up to text editors. 22 | * Cell drag and drop 23 | 24 | 25 | ## 1.0 Release 26 | * Feature parity with the classic notebook. We are tracking notebook feature parity issues with a combination of [labels](https://github.com/jupyterlab/jupyterlab/issues?q=is:open+is:issue+label:"tag:Feature+Parity"+label:pkg:Notebook). 27 | * Settings system. 28 | * Work on porting nbextensions to JupyterLab and building the bridge layers. 29 | * Theme switching. 30 | * UI for managing plugins. 31 | 32 | 33 | ## Future 34 | * [Variable inspector](https://github.com/jupyterlab/jupyterlab/issues/443). 35 | * Real-time collaboration on the notebook, text editor, and other plugins. This includes a server-side model of notebook and text documents. Discussion around the topic can be found [here](https://github.com/jupyterlab/jupyterlab/issues/5382). 36 | * [Dashboarding](https://github.com/jupyterlab/jupyterlab/issues/1640). 37 | * Perform and publish accesibility audit by running an automated tool. 38 | * Bring our core plugins up to the level of the Web Accessibility Standards 39 | (http://www.w3.org/standards/webdesign/accessibility) 40 | * Ability to hook kernels up to output areas. 41 | * Address the [2015 UX survey findings](https://github.com/jupyter/design/blob/master/surveys/2015-notebook-ux/analysis/report_dashboard.ipynb) 42 | * Version control (via git in particular) 43 | * Robust text and code editing (like in Emacs, Vim, Sublime, PyCharm) 44 | * Advanced code development tools (debugging, profiling, variable watching, code modularization) 45 | * Simpler export and deployment options (one-click transformations to slides, scripts, reports) 46 | * Improved installation guides, usage tutorials, and within-tool help 47 | -------------------------------------------------------------------------------- /nbconvert.md: -------------------------------------------------------------------------------- 1 | ## nbconvert 2 | 3 | ### Next (4.2) 4 | 5 | * Bring examples of nbconvert functionality that exists in locations like @takluyver's 6 | demos from last summer's Southampton workshop into the official docs. 7 | 8 | ### Future 9 | 10 | * Explore implementing nbconvert using Pandoc extensions. 11 | * Explore implementing the html output of nbconvert using node.js and the various npm packages we are creating to avoid reimplementing all of the HTML logic. 12 | * Implement an exporter for EPUB. 13 | -------------------------------------------------------------------------------- /notebook.md: -------------------------------------------------------------------------------- 1 | ## Notebook 2 | 3 | ### Next (4.1) 4 | 5 | * Multi-cell selection and actions. 6 | * Notebook wide find and replace. 7 | * Command palette. 8 | * Restart kernel and run all. 9 | 10 | ### Next+1 (5.0) 11 | * Ability to hide/show code and entire cells based on query syntax. 12 | * Plan for Phased Notebook/JupyterLab releases: 13 | * Phase 1: 14 | * A series of major releases starting with 5.0 that will have the existing notebook pages as the main UI with a JupyterLab button that takes people to the new JupyterLab page 15 | * These releases may have new features in the existing notebook pages, some API changes, focus on transitioning to JupyerLab. 16 | * These releases will have bug fixes and new features in JupyterLab designed to get JupyterLab to Phase 2. 17 | * During this phase, we will continue to work on porting nbextensions to JupyterLab and building the bridge layers. 18 | * Explicit that people can request help for porting extensions. 19 | * *Criteria for transition to phase 2:* 20 | 1. Jupyter Lab has all the primary functionality of the classic notebook interface. 21 | 1. Jupyter Lab offers some interesting features beyond the classic interface. 22 | 1. At least some important, popular extensions are ported / bridged to Jupyter Lab. 23 | 1. Documentation and examples exist for creating Jupyter Lab plug-ins. 24 | * Phase 2: 25 | * A series of major releases that will have JupyterLab as the main UI and the old notebook is available as a button 26 | * At this point, we start to discourage the development of new features in the existing notebook. 27 | * *Criteria for transition to phase 3:* 28 | 1. No major development or use of the classic UI. 29 | * Phase 3: 30 | * Starting with the z.0 major release. will remove the existing notebook pages. 31 | * Maybe offer the existing notebook as a server extension. 32 | 33 | * If, and only if, JupyterLab becomes a full replacement, deprecate existing pages in the notebook web-application. 34 | -------------------------------------------------------------------------------- /other.md: -------------------------------------------------------------------------------- 1 | ## nbgrader 2 | 3 | ## nbviewer 4 | 5 | ## Notebook document 6 | 7 | ## Kernel specification 8 | 9 | Implement a way for static resources (js, css, images, etc.) to be fetched based on the kernel for use by frontends (notebook, thebe, etc.). 10 | 11 | ## Message specification 12 | -------------------------------------------------------------------------------- /qtconsole.md: -------------------------------------------------------------------------------- 1 | ## Jupyter Qt Console 2 | 3 | - The functionality of the Qt Console is being implemented inside JupyterLab. 4 | - The goal of the initial version of the JupyterLab console is feature parity 5 | with the Qt Console application. 6 | - The JupyterLab console will also support any cells that a notebook can render. 7 | 8 | ### Next 9 | - There is currently no "save session" functionality. There are several ways the 10 | mechanics of saving the content of a console session might be implemented. A 11 | discussion of its semantics may be fruitful and may lead to an interesting 12 | "save" feature for the next version. 13 | - A full replacement of the Qt Console application may require wrapping the web 14 | version in a native application in the next version. 15 | -------------------------------------------------------------------------------- /spark.md: -------------------------------------------------------------------------------- 1 | # Improving Jupyter for Spark 2 | 3 | Exploring where Jupyter protocols, formats, and UI need to adapt to make it so that “integration with Spark” is seamless. The word “Spark” could likely be replaced with “TensorFlow” or “Dask” throughout all of this issue. 4 | 5 | This is a culmination of feedback, in progress initiatives, and further R&D that needs to be done to improve how Jupyter provides a wonderful ecosystem for Spark, Hadoop, and other cluster tooling. 6 | 7 | ## Asynchronous/Background Outputs ✅ 8 | 9 | For environments like Spark and Dask, when a job or multiple jobs are submitted, they provide a progress bar for the jobs with a way to cancel the overall job. They also provide links to external UI (Spark UI or Dask UI). When finished, you’re able to see the resulting table view. 10 | 11 | There’s a similar use case for Hive Jobs, Presto Jobs, etc. 12 | 13 | As far as Jupyter is concerned, there are things we can do to make updating a display like this easier without having to use widgets: 14 | - Routing background outputs: https://github.com/ipython/ipython/issues/9969 15 | - Ability to update prior displays: https://github.com/jupyter/jupyter_client/issues/209 16 | 17 | Progress: 18 | 19 | * [Outputs can now be updated by named display](https://github.com/jupyter/jupyter_client/issues/209) 20 | 21 | ## Progress Bar Primitives ✅ 22 | 23 | As discussed above in the Async Outputs section, we should seek to provide simple clean-cut library tooling for creating progress bars in the frontend. This is not to seek a replacement of e.g. [tqdm](https://github.com/noamraph/tqdm), but to provide the necessary additions to the messaging spec (whether an actual message or a specific versioned mimetype). 24 | 25 | The nested structure for Spark Jobs is something we can likely represent with a clean format for nested content with progress bars and external links. 26 | 27 | Approaches: 28 | 29 | - https://github.com/mozilla/jupyter-spark 30 | - [Progress monitor](https://github.com/jupyter/jupyter_client/issues/209#issuecomment-260789525) 31 | 32 | Progress: 33 | 34 | * [ProgressBar](https://github.com/ipython/ipython/blob/345d51130aec9644dbeb08c3e54d93f83babbf3f/IPython/core/display.py#L732) is now available in the IPython library 35 | * [VDOM](https://github.com/nteract/vdom) + updatable displays make this super easy, see https://github.com/nteract/nteract/blob/master/packages/desktop/example-notebooks/vdom.ipynb 36 | 37 | ## Cancellable execution 38 | 39 | The Jupyter protocols currently support interrupting execution on kernels. Spark focused notebook UIs, including DataBricks and Zeppelin both have a UI for cancelling execution within the cells and showing progress. 40 | 41 | ## Cluster/computational Context 42 | 43 | Users need a way to get context about their running spark setup, because you’re dealing with an extra remote resource beyond the kernels. They tend to want to know: 44 | - What cluster am I attached to? 45 | - What resources do I have (memory, CPU, Spark version, Hadoop version)? 46 | - How do I access the UI/logs for that cluster? 47 | 48 | Many users want to think in terms of “clusters” and a notebook that is attached to that cluster. The real distinction is 49 | 50 | notebook ← → kernel ← → cluster (or other background resources) 51 | 52 | This can be a cognitive burden for the user. One way of solving this is to have “kernels” that are assumed to be attached to clusters. 53 | 54 | Progress: 55 | 56 | * [Resource Info Request](https://github.com/jupyter/jupyter/issues/264) 57 | 58 | ## Kernel Startup and Background Resources 59 | 60 | Stages a spark user cares about when opening a notebook and running first cell: 61 | 62 | - Runtime started (kernel) 63 | - Preparing driver/Attaching to cluster $NAME/preparing Spark Context 64 | - Libraries uploading 65 | - Attached to cluster $NAME/Spark ready 66 | 67 | This seems to me like additional startup messages from the kernel (yes, new message spec) about driver context. 68 | 69 | We do have a banner for kernel_info, perhaps we can do banner updates with links to: 70 | 71 | - Cluster/driver logs 72 | - Spark UI/Dask UI 73 | 74 | ## Uploading libraries 75 | 76 | For a user of Spark, they need a way to upload JARs to the current spark cluster (and, many times, they need to restart the spark cluster). 77 | 78 | This is out of scope for the Jupyter protocols, would need to be implemented on the outside of the notebook. We need to be able to provide more context, likely through the kernel, about the operating environment and how the cluster is initialized for each kernel. 79 | 80 | Unsure of what/how we can handle this, needs exploration. 81 | 82 | ## Timing Data 83 | 84 | The transient information for how long a command took (as well as the user) are available in the message spec, but not persisted to the notebook document. We can certainly show this in the UI, but there is another reason for keeping this output consistent in serialized state: 85 | 86 | - Lifecycle management of notebooks -- need to be able to know if execution results are stale relative to data, APIs, etc. 87 | - In multi-user collaboration, this must be shared amongst all clients 88 | 89 | This is super useful, and part of why it’s in the message spec. An earlier implementation of IPython notebooks (2006) included and displayed all of this information. We found the information useful enough to keep it in the current protocol implementation, but not enough to put it in the UI or persist it to disk. 90 | 91 | Downsides of storing it in the document: 92 | 93 | - 100% chance of every re-run causing git changes, merge conflicts 94 | 95 | Options: opt-in ‘provenance mode’, where we persist a whole bunch of extra metadata. Super unpleasant for version control, but useful in notebooks that run as jobs, especially for analysts. 96 | 97 | ## Tables and Simple Charts 98 | 99 | For many people, being able to plot simple results before getting to deeper visualization frameworks is pretty important: e.g. in pandas we can use the `df.plot()`, `.hist()`, `.bar()`, etc. methods for quick and easy vis. 100 | 101 | There are a couple approaches to this right now, including mime types for plotly and vega being available and easy to use in both nteract and jupyterlab. For tables, there is an open discussion on Pandas: https://github.com/pandas-dev/pandas/issues/14386 102 | 103 | There's also the [spark magics incubation project](https://github.com/jupyter-incubator/sparkmagic). 104 | 105 | Progress: 106 | 107 | * nteract has direct support of custom media types for vega, vega-lite, and plotly's JSON spec. 108 | * There are JupyterLab extensions for vega, vega-lite, and Plotly 109 | * Pandas has adopted a well specified schema for exported tables (row-oriented JSON) within [pandas#14904](https://github.com/pandas-dev/pandas/pull/14904) 110 | * [nteract has a data explorer](https://blog.nteract.io/designing-the-nteract-data-explorer-f4476d53f897) that is continuing to be invested in 111 | 112 | 113 | ## Scala Kernel 114 | 115 | Massive effort needs to be put into making the Scala kernel(s) for Jupyter first class citizens. Competitive angle: It should be so good that Zeppelin or Beaker would adopt it (or create it) and contribute to it. Current contenders: 116 | 117 | - https://github.com/jupyter-scala/jupyter-scala 118 | - https://toree.incubator.apache.org/documentation/user/quick-start 119 | - https://github.com/mariusvniekerk/spylon-kernel 120 | 121 | ## Style of error reporting 122 | 123 | If you’ve ever worked with PySpark, you know the joys of having python tracebacks embedded in JVM tracebacks. In order to make development easier, we need to bring attention to the right lines to go straight to in a traceback. Spark centric notebooks do this, showing exactly the error text the user needs and a collapsible area for tracebacks. Jupyter has some open issues and discussion about improving error semantics - it’s really important for kernels and the libraries building on them - to be able to expose more useful context to users. 124 | 125 | At a minimum, we should expose links and/or popups to the spark driver’s stdout/stderr, Spark UI, and YARN application master where relevant. Without these essentials, a notebook user can’t debug issues that aren’t returned in the notebook output cell. 126 | 127 | ## POC ➡️ Production 128 | 129 | This part is more JVM centric and not language agnostic - people need a way to turn code segments into a Scala class, compile it, and generate a JAR. My knowledge of how this can be done is fairly old/outdated. 130 | 131 | ## Provide Architectural Recommendations for Jupyter and Spark 132 | 133 | Due to the particular networking requirements of Spark, running Jupyter with Spark tends to have a lot of moving pieces as well as recommendations. We need to provide solid narrative documentation that helps people determine precisely what they need. 134 | -------------------------------------------------------------------------------- /traitlets.md: -------------------------------------------------------------------------------- 1 | ## traitlets 2 | 3 | ### Next (4.1) 4 | 5 | * Much nicer, decorator based APIs for observing (`@observe`), validating 6 | (`@validate) and providing defaults (`@default) for traits. 7 | * Looking forward to its usage in Matplotlib. 8 | --------------------------------------------------------------------------------