├── requirements.txt ├── 47-jupyter-debugger-protocol ├── debugger-screencast.gif └── jupyter-debugger-protocol.md ├── 25-jupyter-enterprise-gateway-incorporation ├── jupyter_enterprise_gateway.gif ├── jupyter_enterprise_gateway_on_yarn.png ├── jupyter_enterprise_gateway_on_kubernetes.png └── jupyter-enterprise-gateway-incorporation.md ├── .github ├── ISSUE_TEMPLATE │ ├── jep-pre-proposal.md │ └── other.md ├── pull_request_template.md └── workflows │ ├── docs.yml │ └── monthly-jeps-report.yml ├── .readthedocs.yaml ├── _config.yml ├── .gitignore ├── noxfile.py ├── conf.py ├── 106-connectionfile-spec ├── connectionfile-spec.md └── connectionfile.schema.json ├── 105-kernelspec-spec ├── kernelspec-spec.md └── kernelspec.schema.json ├── LICENSE ├── _toc.yml ├── 92-jupyter-optional-features └── jupyter-optional-features.md ├── 80-kernel-info └── kernel-info.md ├── 93-debugger-info-copy-to-globals └── debugger-info-copy-to-globals.md ├── 118-restart-clarification └── restart-clarification.md ├── jupyter-enhancement-proposal-guidelines ├── JEP-TEMPLATE.md └── jupyter-enhancement-proposal-guidelines.md ├── 66-jupyter-handshaking └── jupyter-handshaking.md ├── 65-jupyter-xpub └── jupyter-xpub.md ├── 22-jupyter-dashboards-deployment-attic └── jupyter-dashboards-deployment-attic.md ├── 44-xeus-incorporation └── xeus-incorporation.md ├── 12-jupyter-kernel-gateway-incorporation └── jupyter-kernel-gateway-incorporation.md ├── 91-kernel-subshells └── kernel-subshells.md ├── 42-voila-incorporation └── voila-incorporation.md ├── README.md ├── 122-jupyter-book-incorporation └── repositoriestoinclude.csv ├── 17-jupyter-dashboards-extension-incorporation └── jupyter-dashboards-extension-incorporation.md ├── 108-jupyter-subdomain-for-schemas └── jupyter-subdomain-for-schemas.md ├── 18-jupyter-declarativewidgets-incorporation └── jupyter-declarativewidgets-extension-incorporation.md ├── 29-jep-process └── jep-process.md ├── 08-notebook-diff └── notebook-diff.md ├── 104-jep-process-v2 └── jep-process-v2.md └── 79-notebook-v7 └── notebook-v7.md /requirements.txt: -------------------------------------------------------------------------------- 1 | linkify-it-py 2 | myst-parser 3 | sphinx-book-theme 4 | sphinx-copybutton 5 | sphinx_external_toc 6 | sphinxcontrib-mermaid 7 | -------------------------------------------------------------------------------- /47-jupyter-debugger-protocol/debugger-screencast.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/enhancement-proposals/HEAD/47-jupyter-debugger-protocol/debugger-screencast.gif -------------------------------------------------------------------------------- /25-jupyter-enterprise-gateway-incorporation/jupyter_enterprise_gateway.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/enhancement-proposals/HEAD/25-jupyter-enterprise-gateway-incorporation/jupyter_enterprise_gateway.gif -------------------------------------------------------------------------------- /25-jupyter-enterprise-gateway-incorporation/jupyter_enterprise_gateway_on_yarn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/enhancement-proposals/HEAD/25-jupyter-enterprise-gateway-incorporation/jupyter_enterprise_gateway_on_yarn.png -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/jep-pre-proposal.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: JEP pre-proposal 3 | about: Open a JEP pre-proposal 4 | title: '' 5 | labels: JEP pre-proposal 6 | assignees: '' 7 | 8 | --- 9 | 10 | Describe your pre-proposal here 11 | -------------------------------------------------------------------------------- /.readthedocs.yaml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | build: 4 | os: ubuntu-22.04 5 | tools: 6 | python: "3.9" 7 | 8 | sphinx: 9 | configuration: conf.py 10 | 11 | python: 12 | install: 13 | - requirements: requirements.txt 14 | -------------------------------------------------------------------------------- /25-jupyter-enterprise-gateway-incorporation/jupyter_enterprise_gateway_on_kubernetes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/enhancement-proposals/HEAD/25-jupyter-enterprise-gateway-incorporation/jupyter_enterprise_gateway_on_kubernetes.png -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/other.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Other 3 | about: Open an issue that's not a JEP pre-proposal (discussion, suggestion for this 4 | repository, etc) 5 | title: '' 6 | labels: Discussion 7 | assignees: '' 8 | 9 | --- 10 | 11 | 12 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | title : Jupyter Enhancement Proposals 2 | author : Project Jupyter 3 | copyright : "2020" 4 | exclude_patterns : ["LICENSE.md", ".github"] 5 | home_page_in_navbar : false 6 | 7 | repository: 8 | url: https://github.com/jupyter/enhancement-proposals 9 | branch: master 10 | 11 | html: 12 | use_repository_button: true 13 | use_issues_button: true 14 | use_edit_page_button: true 15 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Logs 2 | logs 3 | *.log 4 | 5 | # Runtime data 6 | pids 7 | *.pid 8 | *.seed 9 | 10 | # Directory for instrumented libs generated by jscoverage/JSCover 11 | lib-cov 12 | 13 | # Coverage directory used by tools like istanbul 14 | coverage 15 | 16 | # Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files) 17 | .grunt 18 | 19 | # node-waf configuration 20 | .lock-wscript 21 | 22 | # Compiled binary addons (http://nodejs.org/api/addons.html) 23 | build/Release 24 | 25 | # Dependency directory 26 | # https://www.npmjs.org/doc/misc/npm-faq.html#should-i-check-my-node_modules-folder-into-git 27 | node_modules 28 | 29 | # Jupyter Book 30 | _build 31 | 32 | # Python files 33 | __pycache__ 34 | -------------------------------------------------------------------------------- /noxfile.py: -------------------------------------------------------------------------------- 1 | import nox 2 | 3 | nox.options.reuse_existing_virtualenvs = True 4 | 5 | build_command = ["-b", "html", ".", "_build/html"] 6 | 7 | @nox.session 8 | def docs(session): 9 | session.install("-r", "requirements.txt") 10 | session.run("sphinx-build", *build_command) 11 | 12 | @nox.session(name="docs-live") 13 | def docs_live(session): 14 | session.install("-r", "requirements.txt") 15 | session.install("sphinx-autobuild") 16 | 17 | AUTOBUILD_IGNORE = [ 18 | "_build", 19 | "build_assets", 20 | "tmp", 21 | ] 22 | cmd = ["sphinx-autobuild"] 23 | for folder in AUTOBUILD_IGNORE: 24 | cmd.extend(["--ignore", f"*/{folder}/*"]) 25 | cmd.extend(build_command) 26 | session.run(*cmd) 27 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | ✨Proposing a new JEP?✨ 2 | 3 | Make sure that you have at least done the following: 4 | 5 | 1. **Create a GitHub issue** in the Jupyter Enhancement Proposals repo that 6 | 1. Briefly outlines the proposal 7 | 2. Suggests a review team (optional) 8 | 3. Declares why it should be a JEP (See the “JEP / Not-a-JEP Rubric” below.) 9 | 2. **Identify a Shepherd** to see the process through. (Shepherds are assigned 10 | on a round-robin basis from a set of interested engaged volunteers). 11 | 3. **Decide if it's a JEP** according to the criteria listed above. The Shepherd decides if the JEP criteria have been met. 12 | 4. **Used the new JEP markdown template** to help structure your thoughts. You can find it in `jupyter-enhancement-proposal-guidelines/TEMPLATE.md` in this repository. 13 | -------------------------------------------------------------------------------- /conf.py: -------------------------------------------------------------------------------- 1 | author = "Project Jupyter" 2 | copyright = "2021" 3 | exclude_patterns = ["_build", ".nox", ".github"] 4 | extensions = [ 5 | "myst_parser", 6 | "sphinx_copybutton", 7 | "sphinx_external_toc", 8 | "sphinx.ext.intersphinx", 9 | "sphinx_book_theme", 10 | "sphinxcontrib.mermaid", 11 | ] 12 | html_baseurl = "https://jupyter.org/enhancement-proposals" 13 | html_favicon = "" 14 | html_logo = "" 15 | html_sourcelink_suffix = "" 16 | html_theme = "sphinx_book_theme" 17 | html_theme_options = { 18 | "search_bar_text": "Search the docs...", 19 | "repository_url": "https://github.com/jupyter/enhancement-proposals", 20 | "repository_branch": "master", 21 | "home_page_in_toc": True, 22 | "use_repository_button": True, 23 | "use_edit_page_button": True, 24 | "use_issues_button": True, 25 | } 26 | html_title = "Jupyter Enhancement Proposals" 27 | 28 | myst_enable_extensions = [ 29 | "colon_fence", 30 | "linkify", 31 | ] 32 | pygments_style = "sphinx" 33 | -------------------------------------------------------------------------------- /.github/workflows/docs.yml: -------------------------------------------------------------------------------- 1 | name: deploy-book 2 | 3 | # Only run this when the master branch changes 4 | on: 5 | push: 6 | branches: 7 | - master 8 | pull_request: 9 | branches: 10 | - master 11 | 12 | # Allow the bot to push to the repository 13 | permissions: 14 | contents: write 15 | 16 | # This job installs dependencies, build the book, and pushes it to `gh-pages` 17 | jobs: 18 | deploy-book: 19 | runs-on: ubuntu-latest 20 | steps: 21 | - uses: actions/checkout@v2 22 | 23 | # Install dependencies 24 | - name: Set up Python 3.9 25 | uses: actions/setup-python@v1 26 | with: 27 | python-version: 3.9 28 | 29 | - name: Build the docs 30 | run: | 31 | pip install nox 32 | nox -s docs 33 | 34 | # Push the book's HTML to github-pages 35 | - if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/master' }} 36 | name: GitHub Pages action 37 | uses: peaceiris/actions-gh-pages@v3.5.9 38 | with: 39 | github_token: ${{ secrets.GITHUB_TOKEN }} 40 | publish_dir: ./_build/html 41 | -------------------------------------------------------------------------------- /106-connectionfile-spec/connectionfile-spec.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: connection file specification 3 | authors: Johan Mabille 4 | issue-number: XX 5 | pr-number: 106 6 | date-started: "2023-04-19" 7 | --- 8 | 9 | # Specification of the connection file 10 | 11 | ## Problem 12 | 13 | The connection file is [documented](https://github.com/jupyter/jupyter_client/blob/main/docs/kernels.rst) aside the kernel protocol documentation, but it is not *specified*. 14 | 15 | ## Proposed Enhancement 16 | 17 | We propose to specify the connection file with the JSON schema joined in this PR. The specification would reflect 18 | [the current description of the connection file](https://jupyter-client.readthedocs.io/en/stable/kernels.html#connection-files). 19 | 20 | The documentation of the connection file will be stored along side [those of the kernel protocol](https://github.com/jupyter-standards/kernel-protocol) while its specification will be stored in the [Jupyter schema repo](https://github.com/jupyter/schema). 21 | 22 | ### Impact on existing implementations 23 | 24 | None, this JEP only specifies the current implementations. 25 | -------------------------------------------------------------------------------- /105-kernelspec-spec/kernelspec-spec.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: kernelspec specification 3 | authors: Johan Mabille 4 | issue-number: XX 5 | pr-number: 105 6 | date-started: "2023-04-19" 7 | --- 8 | 9 | # Specification of the kernelspec 10 | 11 | ## Problem 12 | 13 | The kernelspec configuration file is [documented](https://github.com/jupyter/jupyter_client/blob/main/docs/kernels.rst) aside the kernel protocol documentation, and an [implementation](https://github.com/jupyter/jupyter_client/blob/main/jupyter_client/kernelspec.py#L21) is available, but it is not *specified*. Besides, it might be unclear whether the kernelspec is part of the kernel protocol, or independent. 14 | 15 | ## Proposed Enhancement 16 | 17 | We propose to specify the kernelspec with the JSON schema joined in this PR. The specification would reflect 18 | [the current description of the kernelspec](https://jupyter-client.readthedocs.io/en/stable/kernels.html#kernel-specs), 19 | and adds an optional `kernel_protocol_version` field. 20 | 21 | The documentation of the kernelspec will be stored aside [that of the kernel protocol](https://github.com/jupyter-standards/kernel-protocol). The schema will be stored in the [dedicated repo for all Jupyter schemas](https://github.com/jupyter/schema). 22 | 23 | ### Impact on existing implementations 24 | 25 | None, this JEP only adds an optional field in the kernelspec. 26 | 27 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015, Project Jupyter 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | 10 | * Redistributions in binary form must reproduce the above copyright notice, 11 | this list of conditions and the following disclaimer in the documentation 12 | and/or other materials provided with the distribution. 13 | 14 | * Neither the name of enhancement-proposals nor the names of its 15 | contributors may be used to endorse or promote products derived from 16 | this software without specific prior written permission. 17 | 18 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 22 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 23 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 25 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 26 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | 29 | -------------------------------------------------------------------------------- /_toc.yml: -------------------------------------------------------------------------------- 1 | format: jb-book 2 | root: README 3 | parts: 4 | - caption: Information 5 | chapters: 6 | - file: jupyter-enhancement-proposal-guidelines/jupyter-enhancement-proposal-guidelines 7 | - url: https://github.com/jupyter/enhancement-proposals/blob/master/jupyter-enhancement-proposal-guidelines/JEP-TEMPLATE.md 8 | title: JEP Template 9 | 10 | - caption: Accepted JEPs 11 | chapters: 12 | - file: 72-language-server-protocol/language-server-protocol 13 | - file: 65-jupyter-xpub/jupyter-xpub 14 | - file: 91-kernel-subshells/kernel-subshells 15 | - file: 92-jupyter-optional-features/jupyter-optional-features 16 | - file: 93-debugger-info-copy-to-globals/debugger-info-copy-to-globals 17 | - file: 97-add-schema/add-schema-to-notebook-format 18 | - file: 104-jep-process-v2/jep-process-v2 19 | - file: 105-kernelspec-spec/kernelspec-spec 20 | - file: 106-connectionfile-spec/connectionfile-spec 21 | - file: 108-jupyter-subdomain-for-schemas/jupyter-subdomain-for-schemas 22 | - file: 118-restart-clarification/restart-clarification 23 | 24 | - caption: Implemented JEPs 25 | chapters: 26 | - file: 25-jupyter-enterprise-gateway-incorporation/jupyter-enterprise-gateway-incorporation 27 | - file: 22-jupyter-dashboards-deployment-attic/jupyter-dashboards-deployment-attic 28 | - file: 18-jupyter-declarativewidgets-incorporation/jupyter-declarativewidgets-extension-incorporation 29 | - file: 17-jupyter-dashboards-extension-incorporation/jupyter-dashboards-extension-incorporation 30 | - file: 12-jupyter-kernel-gateway-incorporation/jupyter-kernel-gateway-incorporation 31 | - file: 08-notebook-diff/notebook-diff 32 | - file: 62-cell-id/cell-id 33 | - file: 47-jupyter-debugger-protocol/jupyter-debugger-protocol 34 | - file: 44-xeus-incorporation/xeus-incorporation 35 | - file: 42-voila-incorporation/voila-incorporation 36 | - file: 29-jep-process/jep-process 37 | - file: 28-jupyter-server/jupyter-server 38 | - file: 79-notebook-v7/notebook-v7 39 | - file: 80-kernel-info/kernel-info 40 | -------------------------------------------------------------------------------- /92-jupyter-optional-features/jupyter-optional-features.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Jupyter Optional Features 3 | authors: Johan Mabille (@JohanMabille) 4 | issue-number: xxx 5 | pr-number: 92 6 | date-started: 2023-01-16 7 | --- 8 | 9 | # Jupyter Optional Features 10 | 11 | ## Summary 12 | 13 | This JEP introduces Jupyter optional features, which ease the way 14 | a kernel advertises which features it supports, without coupling it 15 | to the version of the protocol that it implements. 16 | 17 | ## Motivation 18 | 19 | Some of the features that were added (the debugger) or proposed (the subshells) 20 | may require a lot of work from kernel authors to implement. Besides, the 21 | changes they introduce on the protocol are self-contained; it is possible for a 22 | kernel to not support them without altering its exepected behavior regarding the 23 | rest of the protocol, and it is easy for a client to enable or disable such a 24 | feature as long as it knows whether the kernel supports it. By the way, the 25 | debugger is already optional, although it does not explicitly state it. 26 | 27 | The goal of this JEP is to introduce the notion of optional features in the protocol 28 | and to have an explicit list of such features. This way, we do not prevent kernel 29 | authors from upgrading to a more recent version of the protocol when we introduce 30 | a new feature that may be complicated to implement and not mandatory for them. 31 | 32 | ## Proposed Enhancement 33 | 34 | We propose to add a new `supported_features` field to the `kernel_info_reply` message. 35 | This field is a list of optional features that the kernel supports. The boolean field 36 | `debugger` should be deprecated, as it would duplicate a possible value of the list. 37 | 38 | An optional feature can be a list of additional messages and/or a list of additional 39 | fields in different existing messages. When a feature introduces new messages, it is 40 | its responsibility to specify the order of these messages when it makes sense. Under 41 | no circumstances this feature should alter the order of already existing messages, 42 | nor interleave new messages between already existing messages. 43 | 44 | The documentation should indicate which optional feature a message (or a field of a 45 | message) is linked to when it is relevant. This would ease the implementation of 46 | new kernels and the upgrade to new versions of the protocol. 47 | -------------------------------------------------------------------------------- /80-kernel-info/kernel-info.md: -------------------------------------------------------------------------------- 1 | # Support `kernel_info` request on the control channel 2 | 3 | ## Problem 4 | 5 | When connecting a new websocket to an existing kernel via the Jupyter server, if the kernel execution is 6 | busy (e.g. long running cell) or stopped (e.g. on a breakpoint), the messages sent over the websocket get no reply. This is because when 7 | establishing a new websocket connection, the Jupyter server will send `kernel_info` requests to the kernel 8 | and will prevent sending any other request until it receives a `kernel_info` reply. Since the `kernel_info` 9 | request is sent on the shell channel and the kernel execution is stopped, it cannot reply to that request. 10 | 11 | ## Proposed enhancement 12 | 13 | We propose to state in the Jupyter Messaging Protocol that the `kernel_info` request can be sent on both the shell and the control channels. Although both channels supports the `kernel_info` message, clients are encouraged to send it on the control channel as it will always be able to handle it, while the shell channel may be stopped. 14 | 15 | ### Impact on existing implementations 16 | 17 | This JEP impacts kernels since it requires them to support receiving `kernel_info_request` on the control channel in addition to receiving them on the shell channel. 18 | 19 | It also has an impact on the Jupyter Server. For example, the reference implementation of Jupyter Server will attempt to send a a `kernel_info` request on both channels and listen for a response from _either_ channel. Any response informs the UI that the kernel is connected. 20 | 21 | ## Relevant Resources (GitHub repositories, Issues, PRs) 22 | 23 | ### GitHub repositories 24 | 25 | - [Jupyter server](https://github.com/jupyter-server/jupyter_server): the backend to Jupyter web applications 26 | - [Jupyter client](https://github.com/jupyter/jupyter_client): Jupyter protocol client APIs 27 | - [JupyterLab](https://github.com/jupyterlab/jupyterlab): JupyterLab computational environment 28 | 29 | ### GitHub Issues 30 | 31 | - New websocket unable to communicate with kernel paused in debugging [#622](https://github.com/jupyter-server/jupyter_server/issues/622) 32 | - [Debugger] Active sessions cause the kernel to deadlock on page refresh [#10174](https://github.com/jupyterlab/jupyterlab/issues/10174) 33 | 34 | ### GitHub Pull Requests 35 | 36 | Nudge kernel with info request until we receive IOPub messages [#361](https://github.com/jupyter-server/jupyter_server/pull/361) 37 | 38 | -------------------------------------------------------------------------------- /93-debugger-info-copy-to-globals/debugger-info-copy-to-globals.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Debugger support to `copyToGlobals` 3 | authors: Nicolas Brichet (@brichet) 4 | issue-number: xxx 5 | pr-number: 93 6 | date-started: 2023-02-20 7 | --- 8 | 9 | # Debugger support to `copyToGlobals` 10 | 11 | ## Summary 12 | 13 | This JEP introduces a new field to the kernel debugger_info response. This new 14 | field will inform the UI that the debugger supports the `copyToGlobals` request. 15 | 16 | ## Motivation 17 | 18 | The `copyToGlobals` request has been introduced in 19 | [ipykernel](https://github.com/ipython/ipykernel/pull/1055) and in 20 | [xeus-python](https://github.com/jupyter-xeus/xeus-python/pull/562) to copy a local 21 | variable to the global scope during a breakpoint. It would be useful to inform the 22 | UI if this is supported by the kernel before displaying the corresponding menu entry. 23 | 24 | ## Proposed Enhancement 25 | 26 | We propose to add a new `copyToGlobals` boolean field to the `debugger_info` 27 | response which will inform the UI that this request is supported. 28 | 29 | ### Reference-level explanation 30 | 31 | This boolean flag should be included in the `debugger_info` response from the kernel 32 | which supports the feature. It is optional, assuming that its absence is understood 33 | as `false` from the client perspective. 34 | 35 | If the feature is supported, the kernel must provide a function for copying a variable 36 | from a local scope to the global scope. 37 | The communication between the UI and the kernel (request - response) will have the 38 | structures described at 39 | https://jupyter-client.readthedocs.io/en/latest/messaging.html#copytoglobals. 40 | 41 | - Request (from UI to kernel) 42 | 43 | ```python 44 | { 45 | 'type': 'request', 46 | 'command': 'copyToGlobals', 47 | 'arguments': { 48 | # the variable to copy from the frame corresponding to `srcFrameId` 49 | 'srcVariableName': str, 50 | 'srcFrameId': int, 51 | # the copied variable name in the global scope 52 | 'dstVariableName': str 53 | } 54 | } 55 | ``` 56 | 57 | - Response (from kernel to UI) 58 | 59 | ```python 60 | { 61 | 'type': 'response', 62 | 'success': bool, 63 | 'command': 'setExpression', 64 | 'body': { 65 | # string representation of the copied variable 66 | 'value': str, 67 | # type of the copied variable 68 | 'type': str, 69 | 'variablesReference': int 70 | } 71 | } 72 | -------------------------------------------------------------------------------- /106-connectionfile-spec/connectionfile.schema.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://json-schema.org/draft/2020-12/schema", 3 | "$id": "https://schema.jupyter.org/connectionfile-v1.0.schema.json", 4 | "title": "Jupyter Connection File", 5 | "description": "A description of the data required to connect and send messages to a Jupyter Kernel", 6 | "type": "object", 7 | 8 | "definitions": { 9 | "signature": { 10 | "type": "object", 11 | "required": ["signature_scheme", "key"], 12 | "properties": { 13 | "signature_scheme": { 14 | "enum": ["hmac-md5","hmac-sha256","hmac-sha512"], 15 | "description": "scheme used to sign the messages" 16 | }, 17 | "key": { 18 | "type": "string", 19 | "description": "key used to sign the messages" 20 | } 21 | } 22 | }, 23 | "kernel_network": { 24 | "type": "object", 25 | "required": ["transport", "ip", "shell_port", "control_port", "stdin_port", "hb_port", "iopub_port"], 26 | "properties": { 27 | "transport": { 28 | "type": "string", 29 | "description": "transport protocol", 30 | "enum": ["tcp", "ipc", "inproc"] 31 | }, 32 | "ip": { 33 | "type": "string", 34 | "description": "ip of the machine where the kernel runs" 35 | }, 36 | "shell_port": { 37 | "type": ["integer","string"], 38 | "description": "port used by the shell channel" 39 | }, 40 | "control_port": { 41 | "type": ["integer","string"], 42 | "description": "port used by the control channel" 43 | }, 44 | "stdin_port": { 45 | "type": ["integer","string"], 46 | "description": "port used by the stdin channel" 47 | }, 48 | "hb_port": { 49 | "type": ["integer","string"], 50 | "description": "port used by the heartbeat channel" 51 | }, 52 | "iopub_port": { 53 | "type": ["ingerer","string"], 54 | "description": "port used by the iopub channel" 55 | } 56 | } 57 | } 58 | }, 59 | "allOf": [ 60 | { "$ref": "#/definitions/kernel_network" }, 61 | { "$ref": "#/definitions/signature" } 62 | ] 63 | } 64 | 65 | -------------------------------------------------------------------------------- /118-restart-clarification/restart-clarification.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Restart Clarification 3 | authors: Marc Udoff (@mlucool) 4 | issue-number: [#117](https://github.com/jupyter/enhancement-proposals/issues) 5 | pr-number: [#118](https://github.com/jupyter/enhancement-proposals/pull/118) 6 | date-started: 2023-08-23 7 | --- 8 | 9 | # Restart Clarification 10 | 11 | ## Summary 12 | 13 | The jupyter client protocol around what "restart" means is ambiguous. Per Jupyter Client's [kernel shutdown documention](https://jupyter-client.readthedocs.io/en/latest/messaging.html#kernel-shutdown), the only guidance is: 14 | > 'restart' : bool # False if final shutdown, or True if shutdown precedes a restart 15 | 16 | This has led to a situation where certain jupyter subprojects (e.g. enterprise gateway) have interpreted this to mean a "hard restart", or restart the whole process tree. 17 | 18 | We propose to clarify this to always mean "restart-in-place", that is only restart the kernel itself. For 19 | most usage of Jupyter, this change is a no-op as subprocess kernels already act like this. 20 | 21 | ## Motivation 22 | 23 | This greatly improves the usability of remote kernels whose startup includes scheduling. By making this change, 24 | restarts for remote kernels will be nearly as fast as those of local kernels. It also matches what we 25 | believe to be the mental model of users when they click "restart". 26 | 27 | ## Guide-level explanation 28 | 29 | The [protocol](https://jupyter-client.readthedocs.io/en/latest/messaging.html#kernel-shutdown) would describe 30 | restart as optimally preserving as many resources outside the kernel as possible (e.g. restarting only the kernel process and its subprocess *not* any parent process). 31 | 32 | When the kernel is a toplevel process (e.g. local kernels), there is no change. 33 | 34 | When the kernel is not a toplevel process (e.g. when used in Enterprise Gateway), restart often means only the kernel restarts. To restart the whole progress group, the stop and start messages could be used. It's up to UIs 35 | for how to display this difference (if any). 36 | 37 | ## Reference-level explanation 38 | 39 | The `jupyter-client` messaging page would be updated to indicate this nuance. Any implementations that 40 | treat restart differently would be updated to match this clarification. 41 | 42 | ## Rationale and alternatives 43 | 44 | A new message could be added, as proposed in the pre-JEP. In the [Jupyter Server meeting](https://github.com/jupyter-server/team-compass/issues/45#issuecomment-1682582186), 45 | we concluded that is is likely most users want restart to only restart the kernel and not potentially reschedule resources. Therefore, a new message was not the best option. 46 | For the vast majority of kernels (local subprocesses), this change is a no-op because restart-in-place 47 | is the same as a hard restart since it is the toplevel kernel process. 48 | 49 | For users that want a hard restart, a stop followed by a start continues to be available. While this may be less convenient, a UI can trivially hide this two call process from the user. 50 | 51 | ## Prior art 52 | 53 | N/A 54 | 55 | ## Unresolved questions 56 | 57 | Only the exact wording changes as proposed in [jupyter_client](https://github.com/jupyter/jupyter_client/pull/966). 58 | 59 | ## Future possibilities 60 | 61 | We would make a service that implements a "hard restart" as discussed in the jupyter-server meeting. 62 | No one on the meeting had an immediate use case for it. 63 | -------------------------------------------------------------------------------- /.github/workflows/monthly-jeps-report.yml: -------------------------------------------------------------------------------- 1 | name: Monthly PR Report 2 | 3 | on: 4 | schedule: 5 | # Run on the first day of every month at 9:00 UTC 6 | - cron: '0 9 1 * *' 7 | workflow_dispatch: # Allow manual triggering 8 | 9 | permissions: 10 | issues: write 11 | pull-requests: read 12 | 13 | jobs: 14 | report-prs: 15 | runs-on: ubuntu-latest 16 | steps: 17 | - name: Generate PR Report 18 | uses: actions/github-script@v7 19 | with: 20 | script: | 21 | const issueNumber = 137; 22 | 23 | // Fetch all open pull requests 24 | const { data: allPRs } = await github.rest.pulls.list({ 25 | owner: context.repo.owner, 26 | repo: context.repo.repo, 27 | state: 'open', 28 | per_page: 100 29 | }); 30 | 31 | // Filter out PRs with SSC_propose_deferred label 32 | const prs = allPRs.filter(pr => 33 | !pr.labels.some(label => label.name === 'SSC_propose_deferred') 34 | ); 35 | 36 | // Group PRs by label 37 | const prsByLabel = new Map(); 38 | const nolabel = []; 39 | 40 | for (const pr of prs) { 41 | if (pr.labels.length === 0) { 42 | nolabel.push(pr); 43 | } else { 44 | for (const label of pr.labels) { 45 | // Skip the excluded label if it somehow wasn't filtered 46 | if (label.name === 'SSC_propose_deferred') continue; 47 | 48 | if (!prsByLabel.has(label.name)) { 49 | prsByLabel.set(label.name, []); 50 | } 51 | prsByLabel.get(label.name).push(pr); 52 | } 53 | } 54 | } 55 | 56 | // Sort labels alphabetically 57 | const sortedLabels = Array.from(prsByLabel.keys()).sort(); 58 | 59 | // Build the report 60 | let report = `Dear SSC members, here are the JEPs that require your attention this month:\n\n`; 61 | report += `## Monthly PR Report\n\n`; 62 | report += `Report generated on: ${new Date().toUTCString()}\n\n`; 63 | report += `**Total open PRs (excluding SSC_propose_deferred): ${prs.length}**\n\n`; 64 | report += `---\n\n`; 65 | 66 | // Add PRs grouped by label 67 | for (const label of sortedLabels) { 68 | const labelPRs = prsByLabel.get(label); 69 | report += `### 🏷️ ${label} (${labelPRs.length})\n\n`; 70 | 71 | for (const pr of labelPRs) { 72 | const age = Math.floor((Date.now() - new Date(pr.created_at)) / (1000 * 60 * 60 * 24)); 73 | report += `- #${pr.number} - ${pr.title} (@${pr.user.login}, ${age} days old)\n`; 74 | } 75 | report += `\n`; 76 | } 77 | 78 | // Add PRs without labels 79 | if (nolabel.length > 0) { 80 | report += `### 🔖 No Label (${nolabel.length})\n\n`; 81 | for (const pr of nolabel) { 82 | const age = Math.floor((Date.now() - new Date(pr.created_at)) / (1000 * 60 * 60 * 24)); 83 | report += `- #${pr.number} - ${pr.title} (@${pr.user.login}, ${age} days old)\n`; 84 | } 85 | report += `\n`; 86 | } 87 | 88 | // Post the report as a comment 89 | await github.rest.issues.createComment({ 90 | owner: context.repo.owner, 91 | repo: context.repo.repo, 92 | issue_number: issueNumber, 93 | body: report 94 | }); 95 | 96 | core.info(`Report posted to issue #${issueNumber}`); 97 | -------------------------------------------------------------------------------- /jupyter-enhancement-proposal-guidelines/JEP-TEMPLATE.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: 3 | authors: 4 | issue-number: 5 | pr-number: 6 | date-started: 7 | --- 8 | 9 | # Title 10 | 11 | ## Summary 12 | 13 | One paragraph explanation of the proposal. 14 | 15 | ## Motivation 16 | 17 | Why are we doing this? What use cases does it support? What is the expected outcome? 18 | 19 | ## Guide-level explanation 20 | 21 | Explain the proposal as if it was already implemented and you were 22 | explaining it to another community member. That generally means: 23 | 24 | - Introducing new named concepts. 25 | - Adding examples for how this proposal affects people's experience. 26 | - Explaining how others should *think* about the feature, and how it should impact the experience using Jupyter tools. It should explain the impact as concretely as possible. 27 | - If applicable, provide sample error messages, deprecation warnings, or migration guidance. 28 | - If applicable, describe the differences between teaching this to existing Jupyter members and new Jupyter members. 29 | 30 | For implementation-oriented JEPs, this section should focus on how other Jupyter 31 | developers should think about the change, and give examples of its concrete impact. For policy JEPs, this section should provide an example-driven introduction to the policy, and explain its impact in concrete terms. 32 | 33 | ## Reference-level explanation 34 | 35 | This is the technical portion of the JEP. Explain the design in 36 | sufficient detail that: 37 | 38 | - Its interaction with other features is clear. 39 | - It is reasonably clear how the feature would be implemented. 40 | - Corner cases are dissected by example. 41 | 42 | The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. 43 | 44 | ## Rationale and alternatives 45 | 46 | - Why is this choice the best in the space of possible designs? 47 | - What other designs have been considered and what is the rationale for not choosing them? 48 | - What is the impact of not doing this? 49 | 50 | ## Prior art 51 | 52 | Discuss prior art, both the good and the bad, in relation to this proposal. 53 | A few examples of what this can include are: 54 | 55 | - Does this feature exist in other tools or ecosystems, and what experience have their community had? 56 | - For community proposals: Is this done by some other community and what were their experiences with it? 57 | - For other teams: What lessons can we learn from what other communities have done here? 58 | - Papers: Are there any published papers or great posts that discuss this? If you have some relevant papers to refer to, this can serve as a more detailed theoretical background. 59 | 60 | This section is intended to encourage you as an author to think about the lessons from other languages, provide readers of your JEP with a fuller picture. 61 | If there is no prior art, that is fine - your ideas are interesting to us whether they are brand new or if it is an adaptation from other languages. 62 | 63 | 64 | ## Unresolved questions 65 | 66 | - What parts of the design do you expect to resolve through the JEP process before this gets merged? 67 | - What related issues do you consider out of scope for this JEP that could be addressed in the future independently of the solution that comes out of this JEP? 68 | 69 | ## Future possibilities 70 | 71 | Think about what the natural extension and evolution of your proposal would 72 | be and how it would affect the Jupyter community at-large. Try to use this section as a tool to more fully consider all possible 73 | interactions with the project and language in your proposal. 74 | Also consider how the this all fits into the roadmap for the project 75 | and of the relevant sub-team. 76 | 77 | This is also a good place to "dump ideas", if they are out of scope for the 78 | JEP you are writing but otherwise related. 79 | 80 | If you have tried and cannot think of any future possibilities, 81 | you may simply state that you cannot think of anything. 82 | 83 | Note that having something written down in the future-possibilities section 84 | is not a reason to accept the current or a future JEP; such notes should be 85 | in the section on motivation or rationale in this or subsequent JEPs. 86 | The section merely provides additional information. 87 | -------------------------------------------------------------------------------- /66-jupyter-handshaking/jupyter-handshaking.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Kernel Handshaking pattern 3 | authors: Johan Mabille (@JohanMabille) 4 | issue-number: 5 | pr-number: 66 6 | date-started: 2021-01-05 7 | --- 8 | 9 | # Kernel Handshaking pattern 10 | 11 | ## Problem 12 | 13 | The current implementation of Jupyter client makes it responsible for finding available ports and pass them to a new starting kernel. The issue is that a new process can start using one of these ports before the kernel has started, resulting in a ZMQError when the kernel starts. This is even more problematic when spawning a lot of kernels in a short laps of time, because the client may find available ports that have already been assigned to another kernel. 14 | 15 | A workaround has been implemented for the latter case, but it does not solve the former one. 16 | 17 | ## Proposed Enhancement 18 | 19 | We propose to implement a handshaking pattern: the client lets the kernel find free ports and communicate them back via a dedicated socket. It then connects to the kernel. More formally: 20 | 21 | - The kernel launcher is responsible for opening a dedicated socket for receiving connection information from kernels (channel ports). This socket will be referred as the **registration socket**. 22 | - When starting a new kernel, the launcher passes the connection information for this socket to the kernel. 23 | - The kernel starts, finds free ports to bind the shell, control, stdin, heartbeat and iopub sockets. It then connects to the registration socket and sends the connection information to the registration socket. 24 | - Upon reception of the connection information, the launcher sends an acknowledge receipt to the kernel, and the client connects to the kernel. 25 | 26 | The way the launcher passes the connection information for the registration socket to the kernel should be similar to that of passing the ports of the kernel socket in the current connection pattern: a connection file that can be read by local kernels or sent over the network for remote kernels (although this requires a custom kernel provisioner or "nanny"). This connection file should also contain the signature scheme and the key. 27 | 28 | Reagarding the registration socket lifetime: 29 | 30 | - The kernel launcher MAY close the registration socket after completing a kernel's registration. Therefore, the kernel should disconnect from the registration socket right after it has received the acknowledge receipt. A kernel should shutdown itself if it does not receive an acknowledge receipt after some time (the value of the time limit is let to the implementation). 31 | - To restart a kernel will require the registration socket again, so the kernel launcher SHOULD keep the registration socket open if it expects restarts to be possible, or open a new socket and pass the new registration socket URL to the new process. 32 | 33 | The kernel should write its connection information in a connection file so that other clients can connect to it. 34 | 35 | The kernel specifies whether it supports the handshake pattern via the "kernel_protocol_version" field in the kernelspec: 36 | - if the field is missing, or if its value if less than 5.5, the kernel supports passing ports only. 37 | - if the field value is >=5.5, the kernel supports both mechanisms. 38 | 39 | ### Remarks 40 | 41 | This pattern is **NOT** a replacement for the current connection pattern. It is an additional one and kernels will have to implement both of them to be conformant to the Jupyter Kernel Protocol specification. Which pattern should be used for the connection is decided by the kernel launcher, depending on the information passed in the initial connection file. 42 | 43 | 44 | A recommended implementation for a multi-kernel client (i.e. jupyter-server) is to have a single long-lived registration socket. 45 | 46 | ### Impact on existing implementations 47 | 48 | Although this enhancement requires changing all the existing kernels, the impact should be limited. Indeed, most of the kernels are based on the kernel wrapper approach, or on xeus. 49 | 50 | Most of the clients are based on `jupyter_client`. Therefore, the changes should only be limited to this repository or external kernel provisioners. 51 | 52 | ## Relevant Resources (GitHub repositories, Issues, PRs) 53 | 54 | ### GitHub repositories 55 | 56 | - Jupyter Client: https://github.com/jupyter/jupyter_client 57 | The Jupyter protocol client APIs 58 | - Voilà: https://github.com/voila-dashboards/voila 59 | Voilà turns Jupyter notebooks into standalone web applications 60 | - IPyKernel: https://github.com/ipython/ipykernel 61 | IPython kernel for Jupyter 62 | - Xeus: https://github.com/jupyter-xeus/xeus 63 | The C++ implementation of the Jupyter kernel protocol 64 | 65 | ### GitHub Issues 66 | 67 | - Spawning many kernels may result in ZMQError (https://github.com/jupyter/jupyter_client/issues/487) 68 | - Spawning ~20 requests at a time results in a ZMQError (https://github.com/voila-dashboards/voila/issues/408#issuecomment-539968325) 69 | 70 | ### GitHub Pull Requests 71 | 72 | - Prevent two kernels to have the same ports (https://github.com/jupyter/jupyter_client/pull/490) 73 | -------------------------------------------------------------------------------- /105-kernelspec-spec/kernelspec.schema.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://json-schema.org/draft/2020-12/schema", 3 | "$id": "https://schema.jupyter.org/kernelspec-v1.0.schema.json", 4 | "title": "Jupyter Kernelspec", 5 | "description": "A description of the data required to start and manage a Jupyter Kernel", 6 | "type": "object", 7 | 8 | "definitions": { 9 | "kernel_arguments": { 10 | "type": "object", 11 | "required": ["argv"], 12 | "properties": { 13 | "argv": { 14 | "description": "A list of command line arguments used to start the kernel. The text {connection_file} in any argument will be replaced with the path to the connection file.", 15 | "type": "array", 16 | "items": { 17 | "type": "string" 18 | }, 19 | "minItems": 1 20 | } 21 | } 22 | }, 23 | "display_name": { 24 | "type": "object", 25 | "required": ["display_name"], 26 | "properties": { 27 | "display_name": { 28 | "description": "The kernel’s name as it should be displayed in the UI. Unlike the kernel name used in the API, this can contain arbitrary unicode characters.", 29 | "type": "string" 30 | } 31 | } 32 | }, 33 | "language": { 34 | "type": "object", 35 | "required": ["language"], 36 | "properties": { 37 | "language": { 38 | "description": "The name of the language of the kernel. When loading notebooks, if no matching kernelspec key (may differ across machines) is found, a kernel with a matching language will be used. This allows a notebook written on any Python or Julia kernel to be properly associated with the user’s Python or Julia kernel, even if they aren’t listed under the same name as the author’s.", 39 | "type": "string" 40 | } 41 | } 42 | }, 43 | "kernel_protocol_version": { 44 | "type": "object", 45 | "required": ["kernel_protocol_version"], 46 | "properties": { 47 | "kernel_protocol_version": { 48 | "description": "The version of protocol this kernel implements. If not specified, the client will assume the version is <5.5 until it can get it via the kernel_info request. The kernel protocol uses semantic versioning (SemVer).", 49 | "type": "string", 50 | "pattern": "^(0|[1-9]\\d*)\\.(0|[1-9]\\d*)\\.(0|[1-9]\\d*)(?:-((?:0|[1-9]\\d*|\\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\\.(?:0|[1-9]\\d*|\\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\\+([0-9a-zA-Z-]+(?:\\.[0-9a-zA-Z-]+)*))?$" 51 | } 52 | } 53 | }, 54 | "interrupt_mode": { 55 | "type": "object", 56 | "required": ["interrupt_mode"], 57 | "properties": { 58 | "interrupt_mode": { 59 | "description": "May be either signal or message and specifies how a client is supposed to interrupt cell execution on this kernel, either by sending an interrupt signal via the operating system’s signalling facilities (e.g. SIGINT on POSIX systems), or by sending an interrupt_request message on the control channel (see Kernel interrupt). If this is not specified the client will default to signal mode.", 60 | "type": "string", 61 | "enum": ["signal", "message"] 62 | } 63 | } 64 | }, 65 | "env": { 66 | "type": "object", 67 | "required": ["env"], 68 | "properties": { 69 | "env": { 70 | "description": "A dictionary of environment variables to set for the kernel. These will be added to the current environment variables before the kernel is started. Existing environment variables can be referenced using ${} and will be substituted with the corresponding value. Administrators should note that use of ${} can expose sensitive variables and should use only in controlled circumstances.", 71 | "type": "object", 72 | "additionalProperties": {"type": "string" } 73 | } 74 | } 75 | }, 76 | "metadata": { 77 | "type": "object", 78 | "required": ["metadata"], 79 | "properties": { 80 | "metadata": { 81 | "description": "A dictionary of additional attributes about this kernel; used by clients to aid in kernel selection. Metadata added here should be namespaced for the tool reading and writing that metadata.", 82 | "type": "object", 83 | "additionalProperties": {"type": "object"} 84 | } 85 | } 86 | } 87 | }, 88 | "anyOf": [ 89 | { "$ref": "#/definitions/argv" }, 90 | { "$ref": "#/definitions/display_name" }, 91 | { "$ref": "#/definitions/language" }, 92 | { "$ref": "#/definitions/kernel_protocol_version" }, 93 | { "$ref": "#/definitions/interrupt_mode" }, 94 | { "$ref": "#/definitions/env" }, 95 | { "$ref": "#/definitions/metadata" } 96 | ], 97 | "required": ["argv", "display_name", "language"] 98 | } 99 | -------------------------------------------------------------------------------- /65-jupyter-xpub/jupyter-xpub.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Replace PUB socket with XPUB socket 3 | authors: Johan Mabille (@JohanMabille) 4 | issue-number: 5 | pr-number: 65 6 | date-started: 2021-01-05 7 | --- 8 | 9 | # Replace PUB socket with XPUB socket 10 | 11 | ## Problem 12 | 13 | A Jupyter kernel uses a PUB socket as a broadcast channel where it publishes various messages (incoming requests, outputs, events). Any client that wants to be notified of what happens in the kernel can simply subscribe to this channel via a SUB socket. The issue with this simple PUB - SUB pattern is that **there is no simple mechanism for clients to wait for their iopub subscription to be established.** 14 | 15 | This is particularly problematic when a client needs to send a bunch of execution requests just after starting a kernel (a typical use case is the "Restart Kernel and execute all" command of Jupyter Lab, or opening a Notebook with Voilà). In that case, the client needs to ensure that its SUB socket is connected to the PUB socket of the kernel before sending any execution request, otherwise it may miss some outputs. The kernel, on its side, will not emit any event until it receives a request. 16 | 17 | ## Current solution 18 | 19 | The current solution consists of sending `kernel_info` requests until a `kernel_info` reply is received on the SHELL and an idle status message is received on the SUB socket. If the client receives a `kernel_info` reply but not the idle status message before a timeout, this means that the SUB socket has not connected yet. The client discards the reply and send a new `kernel_info` request. 20 | 21 | This solution makes the implementation of a Jupyter client more complicated than required. Indeed, ZeroMQ provides a publisher socket that is able to detect new subscriptions. 22 | 23 | ## Proposed Enhancement 24 | 25 | We propose to replace the PUB socket of the kernel with an XPUB socket. XPUB sockets receive the following events: 26 | - subscribe (single frame messages with `\1{subscription-topic}`) 27 | - unsubscribe (single frame messages with `\0{subscription-topic}`) 28 | 29 | When the IOPub XPUB socket receives an event indicating a new subscription, it shall send an `iopub_welcome` message with no parent header and the received 30 | subscription in the `subscription` field, *with the given subscription topic*: 31 | 32 | ``` 33 | identity_prefix: ["subscription-topic"] 34 | parent_header: {} 35 | header: { 36 | "msg_type": "iopub_welcome" 37 | } 38 | content: { 39 | "subscription": "subscription-topic" 40 | } 41 | metadata: {} 42 | ``` 43 | 44 | Notes: 45 | 46 | - The identity prefix may contain extra information about the message, such as the kernel id, according to the convention used in the IPython kernel: 47 | ```kernel.{u-u-i-d}.subscription-topic```. 48 | - Subscriptions can be empty (this is almost always the case). An empty subscription means subscribing to all IOPub messages. 49 | - Subscriptions shall be UTF-8 encoded. Non-utf8 subscriptions shall be ignored, and not produce a welcome message. 50 | - Every subscribed client will receive every other client's welcome message, assuming they match existing subscriptions (e.g. empty strings) 51 | - Welcome messages do not and cannot identify the client whose subscription is being received. Receipt of an iopub_welcome message with your subscription does not mean it is in response to your own subscription. However, receiving a message does mean that a matching subscription has been registered for your client, otherwise no message will be received. So if only one subscription is registered, as is normally the case, receiving any welcome message is sufficient to indicate that your client's subscription is fully established. The gist is that receiving a welcome message is a sufficient condition to establish the subscription-propagation event, and additional welcome messages should be expected and ignored. 52 | - Unsubscribe events are ignored. 53 | 54 | ### Impact on existing implementations 55 | 56 | Although this enhancement requires changing all the existing kernels, the impact should be limited. Indeed, most of the kernels are based on the kernel wrapper approach, or on xeus. 57 | 58 | Regarding clients, this change is backward-incompatible only when they start assuming a welcome message will come. Clients not aware of 59 | the new message will just get an unrecognized iopub message they can safely ignore. 60 | 61 | Most of the clients are based on the Jupyter Server. Therefore, the changes should be limited to this repository and to the notebook. The new 62 | recommended starting procedure is: 63 | 64 | 1. always probe with a single `kernel_info_request` (required for protocol adaptation) 65 | 2. check protocol version in reply 66 | i. if welcome message is supported, wait for it on iopub 67 | ii. if not: 68 | a. use current request & wait loop (recommended, especially for current clients that already have this implemented) 69 | b. accept and warn about possible loss of early iopub messages for 'old' kernels ((increasingly common over time as protocol updates can be more safely assumed, especially for new clients 70 | 71 | 72 | ## Relevant Resources (GitHub repositories, Issues, PRs) 73 | 74 | ### GitHub repositories 75 | 76 | - Jupyter Client: https://github.com/jupyter/jupyter_client 77 | The Jupyter protocol client APIs 78 | - Jupyter Server: https://github.com/jupyter-server/jupyter_server 79 | The backend to Jupyter web applications 80 | - Jupyter Notebook: https://github.com/jupyter/notebook 81 | The Jupyter interactive notebook 82 | - IPyKernel: https://github.com/ipython/ipykernel 83 | IPython kernel for Jupyter 84 | - Xeus: https://github.com/jupyter-xeus/xeus 85 | The C++ implementation of the Jupyter kernel protocol 86 | 87 | ### GitHub Issues 88 | 89 | - Main issue in jupyter_client: SUB sockets take time to subscribe to the IOPub channel and miss important messages (https://github.com/jupyter/jupyter_client/issues/593) 90 | - Related issue in Xeus for implementation detail: Implement Last Value Cache pattern for iopub (https://github.com/jupyter-xeus/xeus/issues/266) 91 | 92 | ### GitHub Pull Requests 93 | 94 | - in jupyter_client: retry kernel_info_requests in wait_for_ready (https://github.com/jupyter/jupyter_client/pull/592) 95 | - in jupyter_server: Nudge kernel with info request until we receive IOPub messages (https://github.com/jupyter-server/jupyter_server/pull/361) 96 | - in notebook: ensure iopub subscriptions propagate prior to accepting websocket connections (https://github.com/jupyter/notebook/pull/5908) 97 | -------------------------------------------------------------------------------- /22-jupyter-dashboards-deployment-attic/jupyter-dashboards-deployment-attic.md: -------------------------------------------------------------------------------- 1 | # Move the Jupyter Dashboards Deployment Projects from Incubator to Attic 2 | 3 | ## Problem 4 | 5 | Jupyter users want to publish their work outside of their notebook authoring environment in a variety of formats. One such format is that of a standalone, interactive dashboard. We started the dashboards bundler and dashboards server projects to address this desire. The projects, collectively referred to as *dashboard deployment* hereafter, underwent significant development effort from late-2015 to mid-2016. Development has since stagnated. 6 | 7 | ## Proposal 8 | 9 | We should clearly signal to the broader community that work on the dashboard deployment incubator projects has effectively ceased by moving the following repositories to the [jupyter-attic organization on GitHub](https://github.com/jupyter-attic): 10 | 11 | * https://github.com/jupyter-incubator/dashboards_server 12 | * https://github.com/jupyter-incubator/dashboards_bundlers 13 | * https://github.com/jupyter-incubator/dashboards_setup 14 | * https://github.com/jupyter-incubator/showcase 15 | 16 | We should use the remainder of this document to capture: 17 | 18 | 1. Why the projects should move to the attic according to our governance criteria 19 | 2. Lessons learned from the incubation effort 20 | 3. Ideas for what to do next with respect to Jupyter and dashboards 21 | 22 | ## Why the attic? 23 | 24 | The [Jupyter Governance - New Subproject Process](https://github.com/jupyter/governance/blob/master/newsubprojects.md) lists eight criteria used to evaluate projects for incorporation into one of the official Jupyter organizations. The dashboard deployment projects have been failing to meet at least five of these criteria for some time. 25 | 26 | ### Have an active developer community that offers a sustainable model for future development. 27 | 28 | Members of the original development team have moved on from the project for a variety of reasons. While developers from the broader Jupyter community have opened a handful of pull requests (PRs) over the past year, there are no active maintainers who review PRs, respond to issues, upgrade components, maintain compatibility with other Jupyter projects, write documentation, fix failing tests, and so on. 29 | 30 | ### Have an active user community. 31 | 32 | There is [certainly user interest](https://github.com/jupyter-incubator/dashboards_server/issues/319) in a dashboard deployment solution for Jupyter notebooks. However, users who have tried the deployment projects [have met with limited success](https://github.com/jupyter-incubator/dashboards_server/issues) given the current complexity of [running the full project stack](https://github.com/jupyter/dashboards/wiki#get-started) due to lack of work on making it easier over the past year. 33 | 34 | ### Demonstrate continued growth and development. 35 | 36 | Development of new features has ceased. The original project roadmap ends at the current feature set and calls for work to continue on ["how dashboarding features materialize in JupyterLab."](https://github.com/jupyter/dashboards/wiki/Deployment-Roadmap#may-2016-update). 37 | 38 | ### Integrate well with other official Subprojects. 39 | 40 | The [dashboards layout extension](https://github.com/jupyter/dashboards) defines a [notebook metadata format](http://jupyter-dashboards-layout.readthedocs.io/en/latest/metadata.html) and uses it to persist information about on-screen notebook cell arrangements. The dashboard deployment projects depend on this metadata. No other Jupyter projects support it. 41 | 42 | Conversely, the dashboard deployment projects use ipywidgets and various JupyterLab components for interactivity and rendering. These two libraries have undergone major development over the past year as they've matured. The dashboard deployment projects have not kept pace, and currently rely on [back-level versions](https://github.com/jupyter-incubator/dashboards_server/blob/master/package.json#L48). 43 | 44 | ### Have a well-defined scope. 45 | 46 | The [dashboards incubator proposal](https://github.com/jupyter-incubator/proposals/blob/master/dashboards/proposal.md#scope) includes a section concerning project scope. The stated objective of "establish[ing] a baseline prototype that demonstrates how notebook authors can publish dashboards" is not, however, clearly documented outside the proposal nor refined in response to [evolving technical constraints](https://github.com/jupyter-incubator/dashboards_server/issues/302). Users are left guessing about the scope of "dashboard deployment" supported by the projects. 47 | 48 | ## What did we learn? 49 | 50 | Briefly: 51 | 52 | * The Jupyter community has a [variety of perspectives](https://github.com/jupyterlab/jupyterlab/issues/1640) on what a "dashboard" is, whether a dashboard is really the desired artifact for many use cases, how Jupyter users want to author dashboards, what "deploying" a dashboard means, and so on. Having a discussion and coming to some form of consensus on where to focus effort seems prudent. 53 | * The ability to write any notebook and deploy it anywhere as an interactive web app is too large an undertaking. Constraints on what libraries are supported and how much of the deployment is automated are necessary to set user expectations, manage project scope, and leave room for third-parties to develop solutions atop common specs. 54 | * We need to clearly note when projects in the incubator are experimental and we are not certain they will succeed. 55 | 56 | ## What might we do next? 57 | 58 | The following list consolidates ideas proposed across various projects. The list is most certainly non-exhaustive. 59 | 60 | * We should [continue the discussion about what the community wants to see in JupyterLab](https://github.com/jupyterlab/jupyterlab/issues/1640) with respect to dashboards and other notebook presentation formats. 61 | * We could [collaborate on a Jupyter nbformat spec](https://github.com/jupyterlab/jupyterlab/issues/1640#issuecomment-291883316) about how notebook documents should persist layout information in hopes of spurring development of multiple tools that use it. 62 | * We could create a [new user survey](https://github.com/jupyter/surveys) to gather information about how dashboard deployment, or, more broadly, notebook publication, fits into Jupyter-user workflows today and to solicit feedback on initial design choices. 63 | * We could [change stance on security in the jupyter/dashboard project](https://github.com/jupyter/dashboards/issues/279), and let the layout extension create the semblance of deployed dashboards in Jupyter Notebook by modifying the notebook interface and automatically executing trusted notebooks when opened. 64 | -------------------------------------------------------------------------------- /44-xeus-incorporation/xeus-incorporation.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Xeus Incorporation 3 | authors: SylvainCorlay 4 | issue-number: XX 5 | pr-number: 44 6 | date-started: "2019-11-30" 7 | --- 8 | 9 | # Xeus Incorporation 10 | 11 | ## Problem 12 | 13 | The [Xeus](https://github.com/QuantStack/xeus/) project is a C++ implementation of the [Jupyter kernel protocol](https://jupyter-client.readthedocs.io/en/stable/messaging.html). Xeus is not a kernel, but a library meant to facilitate the authoring of kernels. 14 | 15 | Several Jupyter kernels have been created with Xeus: 16 | 17 | - [xeus-cling](https://github.com/QuantStack/xeus-cling), a kernel for the C++ programming language, based on the Cling C++ interpreter. The [cling](https://github.com/root-project/cling) project comes from CERN and is at the foundation of the [ROOT](https://github.com/root-project/root.git) project. 18 | - [xeus-python](https://github.com/QuantStack/xeus-python), a kernel for the Python programming language, embedding the Python interpreter. 19 | - [xeus-calc](https://github.com/QuantStack/xeus-calc), a calculator kernel, meant as an educational example on how to make Jupyter kernels with Xeus. 20 | 21 | Beyond these three kernels built on top of Xeus by the Xeus authors, third-parties have developed other Jupyter kernels with Xeus: 22 | 23 | - [JuniperKernel](https://github.com/JuniperKernel/JuniperKernel), a kernel for the R programming language by Spencer Aiello. 24 | - [xeus-fift](https://github.com/atomex-me/xeus-fift), a kernel for the fift programming language by Michael Zaikin. The fift programming language was developed by Telegram to create TON blockchain contracts. 25 | - [SlicerJupyter](https://github.com/Slicer/SlicerJupyter), a kernel for the Python programming language by Kitware which integrates into the Qt event loop of the Kitware "Slicer" project. 26 | 27 | Finally, the xeus-python kernel is used as the backend for the [Jupyter debugger](https://github.com/jupyterlab/debugger/) project. Indeed, xeus-python enables the [Debug Adapter Protocol](https://microsoft.github.io/debug-adapter-protocol/) over the Control channel through new debug request/reply and debug event messages. 28 | 29 | - While Xeus started as a side project for QuantStack engineers, the project now has several stakeholders who depend on it. We think that moving the project to an open governance organization may be a better way to reflect this situation. 30 | - Both JuniperKernel and SlicerJupyter take advantage of Xeus' pluggable concurrency model, which makes it possible to override the concurrency model. The debugger use case required for the control channel messages to be processed while code is being executed by the kernel and makes use of threading for that purpose, and the SlicerJupyter kernel required plugging into another event loop. 31 | 32 | ## Proposed Enhancement 33 | 34 | We propose to make the Xeus project officially part of Jupyter, alongside other related projects from QuantStack, namely xeus-python, xeus-cling, and xeus-calc. Some utility packages used by these kernels may also be moved with xeus, such as [xhale](https://github.com/QuantStack/xhale) which is used by xeus-cling for the quick help mechanism. 35 | 36 | If this incorporation is accepted, the different projects will be modified to reflect this affiliation. Links to the governance documents, code of conduct, and contributing guidelines will be added to the readme and the documentation. 37 | 38 | Xeus and the projects listed above would be moved to the [jupyter-xeus](https://github.com/jupyter-xeus) GitHub organization. 39 | 40 | The [xwidgets](https://github.com/QuantStack/xwidgets), [xplot](https://github.com/QuantStack/xplot), [xleaflet](https://github.com/QuantStack/xleaflet), and [xthreejs](https://github.com/QuantStack/xthreejs) would also be moved to the same organization. Xwidgets is a C++ implementation of the Jupyter widgets protocol built upon xeus. xplot, xleaflet, and xthreejs are C++ backends to the bqplot, ipyleaflet, and PythreeJS Jupyter widget packages. 41 | 42 | ## Criteria for Incorporation 43 | 44 | The [Jupyter Governance - New Subproject Process](https://github.com/jupyter/governance/blob/master/newsubprojects.md) lists the criteria used to evaluate projects for incorporation into the official Jupyter organization. We address each of these criteria below. 45 | 46 | ### Have an active developer community 47 | 48 | **Contributors** 49 | 50 | The core Xeus package has 12 contributors, 524 GitHub stars. Xeus-cling is slightly more popular with over 1000 stars on GitHub. As mentioned earlier, there are several other kernels built upon Xeus by third parties. 51 | 52 | ### Have an active user community 53 | 54 | The Xeus project has a steadily growing user base. The C++ kernel, in particular, is very popular as it is used for teaching the C++ programming language at several universities. Most notably, the [Introduction to Computer Science](http://nicolas.thiery.name/Enseignement/Info111/) course by Nicolas Thiery at University Paris Sud has over 350 students every year who typed their first lines of C++ in a Jupyter notebook. 55 | 56 | ### Demonstrate continued growth and development 57 | 58 | Beyond the xeus-based kernels, there is a new push in the project motivated by the development of the JupyterLab visual debugger. We think that the core xeus library is fairly complete already and that most of the work already done will apply to other xeus-based language kernels such as xeus-cling. 59 | 60 | ### Integrate well with other official Subprojects. 61 | 62 | **Integrating with other Jupyter Components** 63 | 64 | - Xeus *is* an implementation of a Jupyter protocol. 65 | - Xeus-python and xeus-cling offer backends to the *Jupyter widgets protocol* and support *Jupyter rich mime type rendering*. 66 | - Xeus-python is used as a foundation for the JupyterLab debugger project. 67 | - Xeus-cling is co-developed with users who make use of it for teaching C++. A lot of the work in the latest versions has been done to enable the use of *nbgrader* with the C++ kernel. 68 | 69 | **The Jupyter philosophy: language agnosticism** 70 | 71 | We believe that offering a non-python implementation of the kernel protocol will help communicate about the language agnosticism of the Jupyter project. Unlike the one available in ipykernel, the implementation of the protocol of xeus is standalone and may serve as an example for other implementation in e.g. Java or C-Sharp. 72 | 73 | ### Use solid software engineering with documentation and tests hosted with appropriate technologies 74 | 75 | Xeus is continuously tested on multiple platforms with TravisCI and Appveyor. 76 | 77 | We also continuously build the documentation of Xeus which is hosted on ReadTheDocs at URL https://xeus.readthedocs.io. The same holds for xeus-python and xeus-cling. 78 | 79 | ### Have a well-defined scope. 80 | 81 | The scope of xeus is very well defined in that it is limited to the implementation of the Jupyter kernel protocol. 82 | 83 | ### License 84 | 85 | The Xeus project is licensed under the BSD-3-Clause license. 86 | 87 | We use a shared copyright model that enables all contributors to maintain the copyright on their contributions. 88 | 89 | -------------------------------------------------------------------------------- /12-jupyter-kernel-gateway-incorporation/jupyter-kernel-gateway-incorporation.md: -------------------------------------------------------------------------------- 1 | # Jupyter Kernel Gateway Incorporation 2 | 3 | ## Problem 4 | 5 | People are creating novel applications and libraries that use Jupyter kernels as interactive code execution engines. For example: 6 | 7 | * [pyxie](https://github.com/oreillymedia/pyxie-static) uses kernels to evaluate code from a text area on a web page 8 | * [Thebe](https://github.com/oreillymedia/thebe) uses kernels to evaluate code from multiple input areas on a web page 9 | * [gist exec](https://github.com/rgbkrk/gistexec) uses kernels to evaluate code from GitHub gists rendered as web pages 10 | * [Atom Notebook](https://github.com/jupyter/atom-notebook) uses kernels to enable a notebook experience in the Atom text editor 11 | * [Eclairjs](https://github.com/EclairJS/eclairjs-node) uses the Apache Toree kernel as a means of executing Apache Spark jobs defined in JavaScript 12 | * [Jupyter dashboard server](https://github.com/jupyter-incubator/dashboards_nodejs_app) uses kernels to evaluate code from notebooks rendered as interactive dashboards 13 | 14 | These applications have two important properties in common: 15 | 16 | 1. They spawn kernels using provisioning APIs that run separate, and often remote, from the user-facing applications themselves. 17 | 2. They communicate with kernels using Websockets rather than directly with ZeroMQ. 18 | 19 | Some of these application have spawned (or still do spawn) entire Jupyter Notebook servers on the backend in order to request kernels using notebook HTTP API and communicate with them over Websockets. This approach is less than ideal, however, for various reasons: 20 | 21 | 1. The notebook UI is exposed, but only the programmatic UI is used 22 | 2. The notebook authentication mechanism is form- and cookie-based and meant for humans, not programmatic clients 23 | 3. The notebook transport mechanisms and APIs serve the notebook user experience, and are not meant to be replaced or extended to support other clients 24 | 25 | ## Proposed Enhancement 26 | 27 | The incubating [Jupyter Kernel Gateway project](https://github.com/jupyter-incubator/kernel_gateway) has matured to the point where it addresses the issues noted above, and others. It should be considered for incorporation into the main Jupyter organization as an official Subproject. 28 | 29 | ## Detailed Explanation 30 | 31 | The kernel gateway is a web server that supports mechanisms for spawning and communicating with Jupyter kernels. Currently, it supports two such mechanisms: 32 | 33 | 1. A 100% Jupyter Notebook server compatible API for: 34 | * Creating, listing, and deleting Jupyter kernels via HTTP 35 | * Communicating with those kernels using the [Jupyter kernel protocol](http://jupyter-client.readthedocs.org/en/latest/messaging.html) via Websockets 36 | 2. A notebook-defined API that maps HTTP verbs (e.g., `GET`, `POST`) and resources (e.g., `/foo/:bar`) to code to execute on a kernel 37 | 38 | The server launches kernels in its local process/filesystem space. It can be containerized and scaled out using common technology like [tmpnb](https://github.com/jupyter/tmpnb), [Cloud Foundry](https://github.com/cloudfoundry), and [Kubernetes](http://kubernetes.io/). 39 | 40 | ### Current and Potential Use Cases 41 | 42 | * Provide a web-friendly, interactive API for defining, executing, and updating Apache Spark jobs 43 | * Enable non-notebook web clients to provision and use kernels via libraries like [jupyter-js-services](https://github.com/jupyter/jupyter-js-services)) 44 | * Scale Jupyter kernels independently from clients (e.g., via [tmpnb](https://github.com/jupyter/tmpnb), [Binder](https://mybinder.org), your favorite cluster manager) 45 | * Provision and connect to kernels on a remote cloud provider from a local Jupyter Notebook server 46 | * Deploy notebooks as [RESTful microservices](http://blog.ibmjstart.net/2016/01/28/jupyter-notebooks-as-restful-microservices/) 47 | 48 | ### Current Features 49 | 50 | The ["What It Gives You" section of the project README](https://github.com/jupyter-incubator/kernel_gateway#what-it-gives-you) details the currently implemented features. The [jupyter-incubator/kernel_gateway_demos](https://github.com/jupyter-incubator/kernel_gateway_demos) repository contains many runnable examples. 51 | 52 | ### Criteria for Incorporation 53 | 54 | #### Have an active developer community that offers a sustainable model for future development. 55 | 56 | The kernel gateway reuses and extends classes from the Jupyter `notebook` Python package. By virtue of this implementation, it is largely sustained by development of the `jupyter/notebook` project. Minimal maintenance is required to ensure the kernel gateway codebase continues to interoperate with future releases of the `notebook` package. 57 | 58 | #### Have an active user community. 59 | 60 | Kernel gateway is already a required component of the `jupyter/atom-notebook` and `EclairJS/eclairjs-node` projects, a key component letting notebooks with layout defined by `jupyter-incubator/dashboards` run as standalone web applications, and a swap-in replacement for the whole-notebook approach used by `oreillymedia/thebe` and `rgbkrk/gistexec`. 61 | 62 | #### Use solid software engineering with documentation and tests hosted with appropriate technologies. 63 | 64 | The kernel gateway has a suite of unit and integration tests that are run automatically on Travis on every PR and merge to master. 65 | 66 | The documentation for the kernel gateway currently resides in a single README. The document links to docs for various other Jupyter projects to explain the details of the Jupyter protocol and REST APIs. If and when the documentation becomes significantly larger, it can easily migrate to ReadTheDocs or another appropriate site. 67 | 68 | #### Demonstrate continued growth and development. 69 | 70 | See "Have an active developer community that offers a sustainable model for future development." 71 | 72 | #### Integrate well with other official Subprojects. 73 | 74 | The kernel gateway is a `jupyter/jupyter_core#Application` that uses programmatic APIs from `jupyter/notebook` to enable communication with Jupyter kernels like `ipython/ipykernel`. By definition, it integrates with other official Subprojects. 75 | 76 | #### Be developed according to the Jupyter governance and contribution model. 77 | 78 | The kernel gateway is in the Jupyter Incubator, and under the Jupyter governance and contribution model since its inception. 79 | 80 | #### Have a well-defined scope. 81 | 82 | The purpose of the kernel gateway is to provide web-friendly means of accessing kernels. Only features that fit this definition have been included. 83 | 84 | #### Be packaged using appropriate technologies such as pip, conda, npm, bower, docker, etc. 85 | 86 | The kernel gateway is packaged using setup tools, released in both source and wheel format on PyPI, and installable using `pip`. There is also a `jupyter/minimal-kernel` Docker image that can be used as a base image for combining the kernel gateway with any Jupyter kernel. 87 | 88 | ## Pros and Cons 89 | 90 | * Pro: All of the complex functionality for spawning and bridging Websocket to ZeroMQ comes from the Jupyter `notebook` package and is not reimplemented here. 91 | * Con: The kernel gateway does not yet support a clean interface for plugging in additional APIs, transports, and protocols for accessing kernels. The two existing modes are "baked in". 92 | 93 | ## Interested Contributors 94 | 95 | @parente, @rgbkrk 96 | -------------------------------------------------------------------------------- /91-kernel-subshells/kernel-subshells.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Jupyter kernel subshells 3 | authors: David Brochart (@davidbrochart), Sylvain Corlay (@SylvainCorlay), Johan Mabille (@JohanMabille), Ian Thomas (@ianthomas23) 4 | issue-number: XX 5 | pr-number: 91 6 | date-started: 2022-12-15 7 | --- 8 | 9 | # Jupyter kernel subshells 10 | 11 | ## Summary 12 | 13 | This JEP introduces kernel subshells to allow for concurrent shell requests. 14 | 15 | ## Motivation 16 | 17 | Users have been asking for ways to interact with a kernel while it is busy executing CPU-bound code, 18 | for the following reasons: 19 | - inspect the kernel's state to check the progress or debug a long-running computation (e.g. 20 | through a variable explorer). 21 | - visualize intermediary results before the final result is computed. 22 | - request [completion](https://jupyter-client.readthedocs.io/en/stable/messaging.html#completion) or 23 | [introspection](https://jupyter-client.readthedocs.io/en/stable/messaging.html#introspection). 24 | - process 25 | [Comm messages](https://jupyter-client.readthedocs.io/en/stable/messaging.html#custom-messages) 26 | immediately (e.g. for widgets). 27 | - execute arbitrary code in parallel. 28 | 29 | It is currently not possible to do so because the kernel processes shell requests sequentially. 30 | Since the control channel has had its own thread it has been possible to use the control channel 31 | for such interactions, but this is considered bad practice as it should only be used for control 32 | purposes, and the processing of those messages should be almost immediate. 33 | 34 | The goal of this JEP is to offer a way to process shell requests concurrently. 35 | 36 | ## Proposed enhancement: kernel subshells 37 | 38 | The proposal is to support extra threads within a kernel as a JEP 92 39 | [optional feature](https://github.com/jupyter/enhancement-proposals/blob/master/92-jupyter-optional-features/jupyter-optional-features.md) so that whilst the main thread is performing a long blocking task it 40 | will be possible for other threads to do something useful within the same process namespace. 41 | 42 | When a kernel that supports subshells is started it will have a single subshell and this is referred 43 | to as the parent subshell to distinguish it from the other optional subshells which are referred to 44 | as child subshells. 45 | 46 | A new child subshell thread is started using a new `create_subshell_request` control message rather 47 | than via the REST API. Each subshell has a `subshell_id` which is a unique identifier within that 48 | kernel. The `subshell_id` of a child subshell is generated when the subshell is created and 49 | returned in the `create_subshell_reply` message. The parent subshell has a `subshell_id` of `None`. 50 | [Shell messages](https://jupyter-client.readthedocs.io/en/stable/messaging.html#messages-on-the-shell-router-dealer-channel) 51 | include the `subshell_id` as an optional field in the message header to indicate which subshell the 52 | message should be sent to; if this is not specified or is `None` then the parent 53 | subshell is targeted. Use of a `subshell_id` that is not recognised will raise an error. 54 | Subshells are thus multiplexed on the shell channel through the `subshell_id`, and it is the 55 | responsibility of the kernel to route the messages to the target subshell according to the 56 | `subshell_id`. 57 | 58 | Note a kernel that does not support `subshell_id` will just ignore the field if it is present and 59 | run in the main thread. 60 | 61 | [Stdin messages](https://jupyter-client.readthedocs.io/en/stable/messaging.html#messages-on-the-stdin-router-dealer-channel) 62 | will also include the extra optional `subshell_id` field so that it is possible for a subshell to 63 | request and receive stdin independently of other subshells. 64 | 65 | Each subshell will store its own execution count and history. 66 | 67 | ### Modifications to existing messages 68 | 69 | #### Identify optional feature 70 | 71 | Clients identify if a kernel supports subshells via the 72 | [optional feature API](https://github.com/jupyter/enhancement-proposals/blob/master/92-jupyter-optional-features/jupyter-optional-features.md): 73 | 74 | Message type: `kernel_info_reply`: 75 | 76 | ```py 77 | content = { 78 | ... 79 | 'supported_features': [ 80 | 'kernel subshells', 81 | ... 82 | ] 83 | } 84 | ``` 85 | 86 | The full API for optional features is still to be determined, so the details here may change. 87 | In particular, there is probably the need for a version specifier here to allow future changes to 88 | the kernel subshells specification. 89 | 90 | ### New control channel messages 91 | 92 | #### Create subshell 93 | 94 | Message type: `create_subshell_request`: no content. 95 | 96 | Message type: `create_subshell_reply`: 97 | 98 | ```py 99 | content = { 100 | # 'ok' if the request succeeded or 'error', with error information as in all other replies. 101 | 'status': 'ok', 102 | 103 | # The ID of the subshell. 104 | 'subshell_id': str, 105 | } 106 | ``` 107 | 108 | #### Delete subshell 109 | 110 | Message type: `delete_subshell_request`: 111 | 112 | ```py 113 | content = { 114 | # The ID of the subshell. 115 | 'subshell_id': str 116 | } 117 | ``` 118 | 119 | Message type: `delete_subshell_reply`: 120 | 121 | ```py 122 | content = { 123 | # 'ok' if the request succeeded or 'error', with error information as in all other replies. 124 | 'status': 'ok', 125 | } 126 | ``` 127 | 128 | #### List subshells 129 | 130 | Message type: `list_subshell_request`: no content. 131 | 132 | Message type: `list_subshell_reply`: 133 | 134 | ```py 135 | content = { 136 | # A list of subshell IDs. 137 | 'subshell_id': [str] 138 | } 139 | ``` 140 | 141 | Note that the parent subshell (`subshell_id = None`) is not included in the returned list. 142 | 143 | ### New fields on existing messages 144 | 145 | #### Shell and stdin requests 146 | 147 | All shell and stdin messages will allow the optional `subshell_id` field in the request to identify 148 | which subshell should process that message: 149 | 150 | ```py 151 | content = { 152 | # Optional subshell to process request. 153 | 'subshell_id': str | None, 154 | } 155 | ``` 156 | 157 | This field is not in the corresponding reply message as it will be in the parent header. 158 | 159 | #### IOPub messages 160 | 161 | IOPub messages do not need an extra optional `subshell_id` field as this information is available 162 | in the parent header. 163 | 164 | ### Behavior 165 | 166 | #### Kernels supporting subshells 167 | 168 | A subshell request may be processed concurrently with other subshells. Within a an individual 169 | subshell, requests are processed sequentially. 170 | 171 | [Kernel shutdown](https://jupyter-client.readthedocs.io/en/stable/messaging.html#kernel-shutdown) 172 | and [kernel interrupt](https://jupyter-client.readthedocs.io/en/stable/messaging.html#kernel-interrupt) 173 | messages are handled at the kernel (process) rather than subshell (thread) level, and they do not 174 | include a `subshell_id` field. A child subshell can be individually shut down using a 175 | `delete_subshell_request` message. 176 | 177 | #### Kernels not supporting subshells 178 | 179 | These will not claim support for kernel subshells via the optional features API. Unrecognised shell 180 | request messages, such as the subshell request messages listed above, will be ignored as normal. 181 | Any use of a `subshell_id` field in a message will be ignored. Hence existing kernels that do not 182 | support kernel subshells will continue to work as they currently do and will not require any 183 | changes. 184 | 185 | ### Implications for other projects 186 | 187 | Kernel writers who wish to support subshells will need to write extra threading and socket 188 | management code. `ipykernel` will contain a reference implementation. 189 | 190 | Any client that wishes to create a subshell will have to issue a `create_subshell_request` control 191 | message, and pass the `subshell_id` in all relevant shell and stdin messages. 192 | 193 | There will need to be some sort of visual indicator for subshells in, for example, the JupyterLab 194 | UI, but this is not strictly speaking part of the JEP. 195 | -------------------------------------------------------------------------------- /42-voila-incorporation/voila-incorporation.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Voilà Incorporation 3 | authors: SylvainCorlay 4 | issue-number: XX 5 | pr-number: 42 6 | date-started: "2019-10-14" 7 | type: S - [Standards Track](https://www.python.org/dev/peps/#pep-types-key) 8 | --- 9 | 10 | # Voilà Incorporation 11 | 12 | ## Problem 13 | 14 | Interactive notebooks are not the best communication tool for *all* audiences. While they have proven invaluable to provide a *narrative* alongside the source, effectively implementing the *literate programming* concept, they are not ideal to address **non-technical readers** who may be put off by the presence of code cells or the need to run the notebook to see results. Besides, following the order of the code often results in the most interesting content to be at the end of the document. 15 | 16 | Another challenge with sharing notebooks is the **security model**. How can we offer the interactivity of a notebook making use of e.g. Jupyter widgets without allowing arbitrary code execution by the end-user? 17 | 18 | The goal of the [Voilà](https://github.com/voila-dashboards/voila) project is to solve these challenges by providing tools to transform interactive notebooks into standalone web applications. 19 | 20 | ## Proposed Enhancement 21 | 22 | We propose to make the Voilà project officially part of Jupyter, alongside the other projects of the voila-dashboards organization. 23 | 24 | If this incorporation is accepted, the different projects will be modified to reflect this affiliation. Links to the governance documents, code of conduct, and contributing guidelines will be added to the readme and the documentation. The voilà public meeting calendar will be added to the calendar of public Jupyter video meetings. 25 | 26 | Voilà and related projects would remain in the voila-dashboards organization that would effectively become a Jupyter-affiliated GitHub organization. 27 | 28 | ## Criteria for Incorporation 29 | 30 | The [Jupyter Governance - New Subproject Process](https://github.com/jupyter/governance/blob/master/newsubprojects.md) lists the criteria used to evaluate projects for incorporation into the official Jupyter organization. We address each of these criteria below. 31 | 32 | ### Have an active developer community 33 | 34 | **Contributors** 35 | 36 | The Voilà project is split across several repositories in the [voila-dashboards](https://github.com/voila-dashboards/) GitHub organization 37 | 38 | - The main [Voilà](https://github.com/voila-dashboards/voila) repository holds the core package, which had contributions from **25** people, with five regular contributors. 39 | - The [Gallery](https://github.com/voila-dashboards/gallery) project has **10** contributors. 40 | - We also have a [Compass](https://voila-dashboards.github.io/) project that holds the calendar for the weekly public team meeting and the minutes of previous meetings. 41 | 42 | Other repositories in the organization include "Voilà templates", and various utilities for authoring plugins for the Voilà project. We should mention the notable [voila-vuetify](https://github.com/voila-dashboards/voila-vuetify) project which holds a vuetifyjs-based template that produces applications suitable to mobile devices. 43 | 44 | **Growth of the contributor base** 45 | 46 | While the project was initially started by QuantStack, the team now comprises developers from Bloomberg, UC Berkeley, JP Morgan, and Cal Poly San Luis Obispo. OVH has been supportive of the project by kindly providing the free hosting of the gallery on their infrastructure. 47 | 48 | We believe that the *multi-stakeholder* nature of the Voilà project is well-suited for the Jupyter organization. 49 | 50 | ### Have an active user community 51 | 52 | The Voilà project has a very rapidly growing user base. The initial release of the project was announced on June 2019. As of today, (October 14th, 2019), the project has **1223 stars** on GitHub. 53 | 54 | The team strives to grow the user and developer communities with the organization of workshops and presentations at various venues. Since the initial announcement, we have presented Voilà at PyConDE, PyData Paris, Scipy Austin, EuroScipy, GeoPython, and CICM. 55 | 56 | We keep finding users at institutions when we visit them or meet them at conferences. 57 | 58 | ### Demonstrate continued growth and development 59 | 60 | The adoption and development of the Voilà project have only been accelerating since the original release, which was announced with a [medium post](https://blog.jupyter.org/and-voil%C3%A0-f6a2c08a4a93) in June 2019. 61 | 62 | Handling the growing adoption is one of the main motivations for moving to an open-governance model for the project - and we think that Jupyter is the natural home of the project. 63 | 64 | ### Integrate well with other official Subprojects. 65 | 66 | **Built upon Jupyter Components** 67 | 68 | The Voilà project is largely built upon Jupyter subprojects and standards. 69 | 70 | - The standard **notebook file format** is the main entry point to voilà. 71 | - **nbconvert** is used for the conversion to progressively-rendered HTML. 72 | - naturally, we use **jupyter_client** for handling the execution of notebook cells 73 | - **jupyter_server** is the default backend. 74 | - **jupyterhub** is at the foundation of the voila-gallery project. 75 | - **jupyterlab** components (mime renderers, input and output areas) are used in the front-end implementation. 76 | - **ipywidgets** and custom jupyter widget libraries such as bqplot, ipyvolume, ipyleaflets provide the bulk of the interactivity of Voilà applications. 77 | 78 | In fact, Voilà is more a *remix* of existing Jupyter components (with changes to enable that use case) than a completely new application. 79 | 80 | **Integrating improvements upstream** 81 | 82 | The Voilà team, which includes several maintainers of Jupyter subprojects strives to improve the underlying components to enable the Voilà use case instead of having Voilà bolted upon the project or duplicate features. 83 | 84 | - The Voilà gallery is a **JupyterHub** extension. 85 | - The Voilà front-end was designed as a **server extension** to jupyter_server or the classic notebook server. 86 | - The Voilà template system extends that of **nbconvert** and there is ongoing work to upstream the template system of voilà. 87 | - Voilà includes a "preview" extension for **JupyterLab** that lets users see what their notebook would look like in the form of a voilà dashboard. 88 | - The front-end of voilà itself reuses several javascript components from jupyterlab, including the theming capability. 89 | 90 | The Voilà use case motivated a lot of work in jupyter_server. Besides, the main developers of the projects are maintainers of Jupyter subprojects. 91 | 92 | **The Jupyter philosophy: language agnosticism** 93 | 94 | Built upon the Jupyter stack, the Voilà project remains **agnostic to the programming language**. This is demonstrated by the presence in the gallery of voilà applications based on the xeus-cling kernel and the xleaflet C++ backend to ipyleaflet. 95 | 96 | The language-agnosticism of Voilà makes it singular compared to other dashboarding systems such as Dash or Panel. 97 | 98 | **The relationship with jupyter-dashboards** 99 | 100 | The jupyter-dashboards project which provided dashboarding capabilities to jupyter is now unmaintained. The rich layout capability of jupyter-dashboards was based on the GridStackJS library. Unfortunately, jupyter-dashboards is not maintained anymore. 101 | 102 | The voila-dashboards GitHub organization includes a template for laying out notebook cells using the GridStackJS library, [voila-gridstack](https://github.com/voila-dashboards/voila-gridstack). We intend to make voila-gridstack able to read the same layout specification as the original jupyter-dashboards project. 103 | 104 | ### Use solid software engineering with documentation and tests hosted with appropriate technologies 105 | 106 | Voilà is continuously tested with TravisCI. 107 | 108 | We also continuously build the documentation of Voilà which is hosted on ReadTheDocs at URL https://voila.readthedocs.io. 109 | 110 | The documentation provides extensive resources on how to deploy Voilà on various services such as binder, Heroku, google app engine, etc. 111 | 112 | Similarly, the gallery project that powers the voila-gallery.org public instance can easily be deployed on various infrastructures. 113 | 114 | ### Have a well-defined scope. 115 | 116 | Voilà is a tool meant to *present* read-only notebooks, possibly with a different layout, but *without* the ability to execute arbitrary code. The scope is defined by the content that the notebook file format can hold. 117 | 118 | ### License 119 | 120 | The Voilà project is licensed under the BSD-3-Clause license. 121 | 122 | We use a shared copyright model that enables all contributors to maintain the copyright on their contributions. 123 | 124 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Jupyter Enhancement Proposals 2 | 3 | This repository contains enhancement proposals for the Jupyter ecosystem, known as Jupyter Enhancement Proposals or JEPs. Jupyter Enhancement Proposals will be used when presenting changes or additions that affect multiple components of the Jupyter ecosystem OR changes to a single key component. 4 | 5 | See [the Enhancement Proposal Guidelines](jupyter-enhancement-proposal-guidelines/jupyter-enhancement-proposal-guidelines.md) 6 | for more information. 7 | 8 | Nicely rendered version of the JEPs: 9 | 10 | ## Index of JEPs 11 | 12 | Below is a list of JEPs that have been Submitted in the past. 13 | 14 | If a JEP is in the **Submitted** state, a pull-request is currently open for conversation and comments. If a JEP 15 | is in the **Accepted** state, the JEP has been merged into this repository, and work is 16 | encouraged to commence on the topic. 17 | 18 | | Number | Status | Title | PR | 19 | |--------|--------|-------|----| 20 | | 0004 | Withdrawn | New Notebook Format for improved workflow integration | [#04](https://github.com/jupyter/enhancement-proposals/pull/4) | 21 | | 0007 | Withdrawn | Jupyter Extension Generator | [#07](https://github.com/jupyter/enhancement-proposals/pull/07) | 22 | | 0008 | Implemented | [Diffing and Merging Notebooks](08-notebook-diff/notebook-diff.md) | [#08](https://github.com/jupyter/enhancement-proposals/pull/08) | 23 | | 0012 | Implemented | [Kernel Gateway](12-jupyter-kernel-gateway-incorporation/jupyter-kernel-gateway-incorporation.md) | [#12](https://github.com/jupyter/enhancement-proposals/pull/12) | 24 | | 0014 | **Submitted** | Kernel Nanny | [#14](https://github.com/jupyter/enhancement-proposals/pull/14) | 25 | | 0015 | Withdrawn | Layout Namespaces and Discovery | [#15](https://github.com/jupyter/enhancement-proposals/pull/15) | 26 | | 0016 | **Submitted** | Notebook Translation and Localization | [#16](https://github.com/jupyter/enhancement-proposals/pull/16) | 27 | | 0017 | Implemented | [Dashboards Notebook Extension](17-jupyter-dashboards-extension-incorporation/jupyter-dashboards-extension-incorporation.md) | [#17](https://github.com/jupyter/enhancement-proposals/pull/17) | 28 | | 0018 | Implemented | [Declarative Widgets Extension](18-jupyter-declarativewidgets-incorporation/jupyter-declarativewidgets-extension-incorporation.md) | [#18](https://github.com/jupyter/enhancement-proposals/pull/18) | 29 | | 0022 | Implemented | [Move Dashboards Deployment Projects from Incubator to Attic](22-jupyter-dashboards-deployment-attic/jupyter-dashboards-deployment-attic.md) | [#22](https://github.com/jupyter/enhancement-proposals/pull/22) | 30 | | 0023 | **Submitted** | Jupyter Template as Metadata | [#23](https://github.com/jupyter/enhancement-proposals/pull/23) | 31 | | 0024 | **Submitted** | Simplifying Error Reporting in Jupyter Protocol | [#24](https://github.com/jupyter/enhancement-proposals/pull/24) | 32 | | 0025 | Implemented | [Enterprise Gateway](25-jupyter-enterprise-gateway-incorporation/jupyter-enterprise-gateway-incorporation.md) | [#25](https://github.com/jupyter/enhancement-proposals/pull/25) | 33 | | 0026 | Withdrawn | Add Language Server Support to Jupyter Server and jupyterlab-monaco | [#26](https://github.com/jupyter/enhancement-proposals/pull/26) | 34 | | 0028 | Implemented | [Standalone Jupyter Server](28-jupyter-server/jupyter-server.md) | [#28](https://github.com/jupyter/enhancement-proposals/pull/28) | 35 | | 0029 | Implemented | [Jupyter Enhancement Proposal updates](29-jep-process/jep-process.md) | [#29](https://github.com/jupyter/enhancement-proposals/pull/29) | 36 | | 0042 | Implemented | [Voila Incorporation](42-voila-incorporation/voila-incorporation.md) | [#43](https://github.com/jupyter/enhancement-proposals/pull/43) | 37 | | 0044 | Implemented | [Xeus Incorporation](44-xeus-incorporation/xeus-incorporation.md) | [#44](https://github.com/jupyter/enhancement-proposals/pull/44) | 38 | | 0047 | Implemented | [Jupyter Debugger Protocol](47-jupyter-debugger-protocol/jupyter-debugger-protocol.md) | [#47](https://github.com/jupyter/enhancement-proposals/pull/47) | 39 | | 0062 | Implemented | [Cell ID Addition to Notebook Format](62-cell-id/cell-id.md) | [#62](https://github.com/jupyter/enhancement-proposals/pull/62) | 40 | | 0065 | **Accepted** | [Replace PUB socket with XPUB socket](65-jupyter-xpub/jupyter-xpub.md) | [#65](https://github.com/jupyter/enhancement-proposals/pull/65) | 41 | | 0066 | **Accepted** | [Kernel Handshaking pattern](66-jupyter-handshaking/jupyter-handshaking.md) | [#66](https://github.com/jupyter/enhancement-proposals/pull/66) | 42 | | 0072 | **Accepted** | [Language server protocol (LSP)](72-language-server-protocol/language-server-protocol.md) | [#72](https://github.com/jupyter/enhancement-proposals/pull/72) | 43 | | 0079 | Implemented | [Build Jupyter Notebook v7 off of JupyterLab components](79-notebook-v7/notebook-v7.md) | [#79](https://github.com/jupyter/enhancement-proposals/pull/79) | 44 | | 0080 | **Accepted** | [Support `kernel_info` request on the control channel](80-kernel-info/kernel-info.md) | [#80](https://github.com/jupyter/enhancement-proposals/pull/80) | 45 | | 0091 | **Accepted** | [Jupyter kernel subshells](91-kernel-subshells/kernel-subshells.md) | [#91](https://github.com/jupyter/enhancement-proposals/pull/91) | 46 | | 0092 | **Accepted** | [Jupyter Optional Features](92-jupyter-optional-features/jupyter-optional-features.md) | [#92](https://github.com/jupyter/enhancement-proposals/pull/92) | 47 | | 0093 | **Accepted** | [Debugger support to `copyToGlobals`](93-debugger-info-copy-to-globals/debugger-info-copy-to-globals.md) | [#93](https://github.com/jupyter/enhancement-proposals/pull/93) | 48 | | 0097 | **Accepted** | [Add `$schema` to notebook format](97-add-schema/add-schema-to-notebook-format.md) | [#97](https://github.com/jupyter/enhancement-proposals/pull/97) | 49 | | 0104 | **Accepted** | [Jupyter Enhancement Proposal v2](104-jep-process-v2/jep-process-v2.md) | [#104](https://github.com/jupyter/enhancement-proposals/pull/104) | 50 | | 0105 | **Accepted** | [kernelspec specification](105-kernelspec-spec/kernelspec-spec.md) | [#105](https://github.com/jupyter/enhancement-proposals/pull/105) | 51 | | 0106 | **Accepted** | [Connection file specification](106-connectionfile-spec/connectionfile-spec.md) | [#106](https://github.com/jupyter/enhancement-proposals/pull/106) | 52 | | 0108 | **Accepted** | [Subdomain and repository for publishing schemas under jupyter.org](108-jupyter-subdomain-for-schemas/jupyter-subdomain-for-schemas.md) | [#108](https://github.com/jupyter/enhancement-proposals/pull/108) | 53 | | 0118 | **Accepted** | [Restart Clarification](118-restart-clarification/restart-clarification.md) | [#118](https://github.com/jupyter/enhancement-proposals/pull/118) | 54 | | 0122 | **Accepted** | [Incorporate Jupyter Book as a subproject](./122-jupyter-book-incorporation/jupyter-book-incorporation.md) | [#123](https://github.com/jupyter/enhancement-proposals/pull/123) | 55 | 56 | ## How do I submit a JEP? 57 | 58 | In order to submit a JEP, first read the [Jupyter Enhancement Proposal Submission Guidelines](jupyter-enhancement-proposal-guidelines/jupyter-enhancement-proposal-guidelines.md) which describes the JEP process. 59 | 60 | In addition, read 61 | [the JEP proposal template](jupyter-enhancement-proposal-guidelines/JEP-TEMPLATE.md) 62 | for guidance on the questions you should answer before officially submitting 63 | the JEP. 64 | 65 | ## Docs hosting 66 | 67 | We use a GitHub action to build the documentation with Sphinx and push it to the `gh-pages` branch of the repository. 68 | This is then hosted at `jupyter.org/enhancement-proposals`. 69 | 70 | We use a ReadTheDocs build to automatically generate **previews** of the documentation for Pull Requests. 71 | However this is not the publicly-hosted version of the documentation, it is just for PRs. 72 | 73 | ## Build the enhancement proposal docs 74 | 75 | The Enhancement Proposal documentation is structured as a [Sphinx documentation site](https://www.sphinx-doc.org/) that uses a them and configuration inspired by [Jupyter Book](https://jupyterbook.org). 76 | 77 | To build the documentation locally, use [the `nox` automation and environment management tool](https://nox.thea.codes/). 78 | Follow these steps: 79 | 80 | - Install `nox`: 81 | 82 | ```console 83 | $ pip install nox 84 | ``` 85 | - Build the docs from the `enhancement-proposals` folder: 86 | 87 | ```console 88 | $ nox -s docs 89 | ``` 90 | 91 | This will automatically install the environment needed to build the documentation, and then place the output HTML in the `_build/html` folder. 92 | 93 | To build the documentation with a live reload server, run: 94 | 95 | ```console 96 | $ nox -s docs-live 97 | ``` 98 | 99 | To manually install and build the documentation with Sphinx, install the requirements in `requirements.txt` and then run `sphinx-build . _build/html`. 100 | -------------------------------------------------------------------------------- /47-jupyter-debugger-protocol/jupyter-debugger-protocol.md: -------------------------------------------------------------------------------- 1 | # Jupyter debugger protocol 2 | 3 | | Item | Value | 4 | |------------|------------------------------------------------------------------------------------------------------------------------------| 5 | | Title | Jupyter Debugger Protocol | 6 | | Authors | Sylvain Corlay ([@SylvainCorlay](https://github.com/SylvainCorlay)) and Johan Mabille ([@JohanMabille](https://github.com/JohanMabille))| 7 | | Status | Draft | 8 | | Type | S - [Standards Track](https://www.python.org/dev/peps/#pep-types-key) JEP | 9 | | Created | January 1st, 2020 | 10 | 11 | ## Problem 12 | 13 | Jupyter users like to experiment in the notebook, but for more classical software development tasks such as the refactoring of a large codebase, they often switch to a general-purpose IDE. One of the main reasons for switching to other tools is the lack of a visual debugger in Jupyter, with the ability to set breakpoints, step into code, inspect variables, etc. A visual debugger for Jupyter has been a long-standing request from the community. 14 | 15 | One available feature in the case of the IPython kernel is [%debug magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-debug), which can be used both interactively, or for post-mortem analysis, and provides a command-line interface to pdb in Jupyter front-ends. 16 | 17 | The absence of a visual debugger was also one of the main pain points for users accustomed to classical IDEs, for whom the main way to enable an interactive workflow was to run their code in a debugger. 18 | 19 | The development of a visual debugger for Jupyter required both front-end and back-end work, including additions to the Jupyter kernel protocol. We have been making strides in all aspects of the project over the past few months and made a first release of the Jupyter debugger. This Jupyter Enhancement Proposal is meant to summarize the main changes, and more importantly, to **include the extensions to the Jupyter kernel protocol in the official specification**. 20 | 21 | ## Proposed Enhancement 22 | 23 | We propose to include the extensions to the Jupyter kernel protocol implemented in `jupyter-xeus/xeus-python` and in `jupyterlab/debugger` into the official specification for the Jupyter kernel protocol. 24 | 25 | ### New messages on the Control and IOPub channel 26 | 27 | Current communication channels with the kernel include the **Shell** channel, which is used for *e.g.* execution requests, and the **IOPub** channel, which is a one-directional communication channel from the kernel to the client, and is used to *e.g.* forward the content of the standard output stream (stdout and stderr). 28 | 29 | The **Control** channel is similar to Shell but operates on a **separate socket so that messages are not queued behind execution requests**, and have a higher priority. Control was already used for Interrupt and Shutdown requests, and we decided to use the same channel for the commands sent to the debugger. 30 | 31 | Two message types are added to the protocol: 32 | 33 | - the `debug_[request|reply]` to request specific actions to be performed by the debugger such as adding a breakpoint or stepping into a code, which is sent to the **Control** channel. 34 | - the `debug_event` uni-directional message used by kernels to send debugging events to the front-end, such as a breakpoint being reached in a thread. Debug events are sent over the **IOPub** channel. 35 | 36 | All debugging commands and events are communicated through these event types. 37 | 38 | ### The Debug Adapter Protocol 39 | 40 | A requirement for any addition to the Jupyter kernel protocol is to preserve the **language agnosticism** of the Jupyter architecture. 41 | 42 | A popular language-agnostic standard for debugging is Microsoft's [Debug Adapter Protocol](https://microsoft.github.io/debug-adapter-protocol/specification) (DAP), which is a JSON-based protocol underlying the debugger of Visual Studio Code, and for which there already exist multiple language back-ends. The Debug Adapter Protocol has three main message types: `Request`, `Response`, and `Event`. 43 | 44 | Our proposal is for the `content` attribute of the `debug_[request|reply]` to be the content of the `Request` and `Response` messages from the DAP, and for the `content` attribute of the `debug_event` to the `Event` message from the DAP. 45 | 46 | This approach will allow kernel authors to use one of the [many implementations of the DAP](https://microsoft.github.io/debug-adapter-protocol/implementors/adapters/). 47 | 48 | ### Additions to the Debug Adapter Protocol 49 | 50 | We make two notable additions to the Debug Adapter Protocol. 51 | 52 | - In order to support page reloading, or a client connecting at a later stage, Jupyter kernels must store the state of the debugger (breakpoints, whether the debugger is currently stopped). 53 | - In order to support the debugging of notebook cells and of Jupyter consoles, which are not based on source files, we also needed messages to submit code to the debugger to which breakpoints can be added. 54 | 55 | Both message types follow the same schema as the other `debug_[request|reply]` messages ported from the Debug Adapter Protocol. 56 | 57 | ### A new field in the `kernel_info_reply` message 58 | 59 | The kernel info reply has a new optional boolean field indicating whether the kernel implements the Jupyter Debug Protocol. 60 | 61 | ## Implementation 62 | 63 | We provide both a front-end and a kernel implementation for the Jupyter kernel protocol. The front-end is a JupyterLab extension, which is available at URL https://github.com/jupyterlab/debugger, and the back-end is implemented as part of the xeus-python kernel, which is a kernel for the Python programming language available at URL https://github.com/jupyter-xeus/xeus-python. 64 | 65 | Together, they provide a rather typical experience of a visual debugger in JupyterLab: 66 | 67 | ![screencast](debugger-screencast.gif) 68 | 69 | Typescript classes for the Debug Adapter Protocol message types are available in the MIT-licensed [vscode-debugadapter-node](https://github.com/microsoft/vscode-debugadapter-node/) package, which we were able to leverage in the debugger front-end implementation. 70 | 71 | ## Relevant Resources (GitHub repositories, Issues, PRs) 72 | 73 | ### GitHub repositories 74 | 75 | - The JupyterLab debugger extension: https://github.com/jupyterlab/debugger 76 | 77 | This is the first implementation of a debugger front-end supporting the described protocol 78 | 79 | - The Xeus-Python Jupyter kernel: https://github.com/jupyter-xeus/xeus-python 80 | 81 | This is the first Jupyter kernel implementing the proposed kernel protocol extensions. Xeus-python is a kernel for the Python programming language. 82 | 83 | ### GitHub Issues 84 | 85 | - Master issue by Johan Mabille ([@JohanMabille](https://github.com/johanmabille)) on the different steps towards supporting the Debug Adapter Protocol in Jupyter: https://github.com/jupyter/jupyter_client/issues/446 86 | 87 | ### GitHub Pull Requests 88 | 89 | In jupyter_client: 90 | 91 | - Adding `debug_[request|reply]` and `debug_event` to the Jupyter kernel protocol: https://github.com/jupyter/jupyter_client/pull/464 92 | - Debug adapter protocol: https://github.com/jupyter/jupyter_client/pull/502 93 | 94 | In jupyter_server and notebook: 95 | 96 | - Added control channel to ZMQChannelsHandler: https://github.com/jupyter/jupyter_server/pull/56 97 | - The same change in the classic notebook server: https://github.com/jupyter/notebook/pull/4672 98 | 99 | In JupyterLab: 100 | 101 | - Adds interface for messages sent on the Control channel: https://github.com/jupyterlab/jupyterlab/pull/6544 102 | - Adding debug messages: https://github.com/jupyterlab/jupyterlab/pull/6704 103 | 104 | In Xeus: 105 | 106 | - There has been a significant number of pull requests in the core xeus library for abstracting out the debug protocol and simplifying its support in other xeus-based kernels. 107 | 108 | Old resources for past discussions on the debugger: 109 | 110 | - The debugger repository in the jupyter-attic repository holds a number of discussions about the implementation of a visual debugger for Jupyter. The repository has been archived. https://github.com/jupyter-attic/debugger 111 | - The archived jupyterlab-debugger repository from the QuantStack organization holds the early front-end work by Jeremy Tuloup that was used for the first iterations on a JupyterLab extension: https://github.com/QuantStack/jupyterlab-debugger/ 112 | 113 | ## Interested Parties 114 | 115 | @SylvainCorlay, @JohanMabille, @Afshin, @jtpio, @martinRenou, @KsavinN 116 | -------------------------------------------------------------------------------- /25-jupyter-enterprise-gateway-incorporation/jupyter-enterprise-gateway-incorporation.md: -------------------------------------------------------------------------------- 1 | # Jupyter Enterprise Gateway Incorporation 2 | 3 | ## Problem 4 | 5 | Founded in academia, the Jupyter projects provide a rich and highly popular set of applications for 6 | interacting with and iterating on large and complex applications. It has been truly ground-breaking. 7 | However, when we first attempted to build a Notebook service that could enable a large number of data 8 | scientists to run frequent and large workloads against a large Apache Spark cluster, we identified 9 | several requirements that were not currently available in the Jupyter open source ecosystem. We tried 10 | to use the Jupyter Kernel Gateway project, but we quickly realized that the JKG server became the 11 | bottleneck because the co-located Spark driver application for these kinds of workloads (in this case, 12 | the kernel process running on behalf of notebook cells) were extremely resource intensive. In organizations 13 | with multiple data scientists, you can quickly saturate the compute resources of the Kernel Gateway server. 14 | 15 | Jupyter Enterprise Gateway enables Jupyter Notebook to launch and manage remote kernels in a distributed cluster, 16 | including Apache Spark managed by YARN, IBM Spectrum Conductor or Kubernetes. New platforms can be added via an 17 | extensibility layer that would handle specific capabilities of the underlying cluster manager. 18 | 19 | ## Proposed Enhancement 20 | 21 | The incubating [Jupyter Enterprise Gateway project](https://github.com/jupyter-incubator/enterprise_gateway) has 22 | matured to the point where it addresses the issues noted above, and others. It should be considered for incorporation 23 | into the main Jupyter organization as an official Subproject. 24 | 25 | ## Detailed Explanation 26 | 27 | Please see below detailed project information. 28 | 29 | ### Current and Potential Use Cases 30 | 31 | * Provision and manage kernels in a remote cluster. 32 | * Support kernels to be launched as a given user enabling multi-tenancy. 33 | 34 | ### Current Features 35 | 36 | Jupyter Enterprise Gateway is a web server that provides headless access to Jupyter kernels within 37 | an enterprise. Built directly upon Jupyter Kernel Gateway, Jupyter Enterprise Gateway leverages all 38 | of the Kernel Gateway functionality in addition to the following: 39 | 40 | * Adds support for remote kernels hosted throughout the enterprise where kernels can be launched in 41 | the following ways: 42 | * Local to the Enterprise Gateway server (today's Kernel Gateway behavior) 43 | * On specific nodes of the cluster utilizing a round-robin algorithm 44 | * On nodes identified by an associated resource manager 45 | * Provides support Apache Spark managed by YARN, IBM Spectrum Conductor or Kubernetes out of the box. 46 | Others can be configured via Enterprise Gateway's extensible framework. 47 | * Secure communication from the client, through the Enterprise Gateway server, to the kernels 48 | * Multi-tenant capabilities 49 | * Ability to associate profiles consisting of configuration settings to a kernel for a given user 50 | * Persistent kernel sessions 51 | 52 | Below are some more details on the supported cluster platforms and specific capabilities: 53 | 54 | #### Distributed Kernels in Apache Spark 55 | 56 | Jupyter Enterprise Gateway enables Jupyter Notebook to launch and manage remote kernels in a distributed cluster. It 57 | leverages different resource managers to enable distributed kernels in Apache Spark clusters. One example shown below 58 | describes kernels being launched in YARN cluster mode across all nodes of a cluster. 59 | 60 | 61 | ![Jupyter Enterprise Gateway leverages Apache Spark resource managers to distribute kernels](jupyter_enterprise_gateway.gif) 62 | 63 | Note that, Jupyter Enterprise Gateway also provides some other value added capabilities such as : enhanced security and multiuser support with user impersonation. 64 | 65 | ![Jupyter Enterprise Gateway provides Enhanced Security and Multiuser support with user Impersonation](jupyter_enterprise_gateway_on_yarn.png) 66 | 67 | #### Distributed Kernels in Kubernetes 68 | 69 | Jupyter Enterprise Gateway support for Kubernetes enables decoupling the Jupyter Notebook Server and its kernels into multiple pods. This enables running Notebook server pods with minimally necessary resources based on the workload being processed. 70 | 71 | ![Jupyter Enterprise Gateway enable remote kernels on Kubernetes cluster](jupyter_enterprise_gateway_on_kubernetes.png) 72 | 73 | 74 | ## Criteria for Incorporation 75 | 76 | ### Have an active developer community that offers a sustainable model for future development. 77 | 78 | The enterprise gateway reuses and extends classes from the Jupyter `kernel_gateway` and `notebook` Python package. By virtue of this implementation, it is largely sustained by development of the `jupyter/notebook` project. Minimal maintenance is required to ensure the enterprise gateway codebase continues to interoperate with future releases of the `notebook` package. 79 | 80 | ### Have an active user community. 81 | 82 | Enterprise gateway is a fundamental component in multiple IBM Cloud offerings, and has also been adopted in a few large companies that are providing Analytical and/or AI platform for it's internal/external customers. 83 | 84 | Other then that, below are some stats that have been collected from the Jupyter Enterprise Gateway GitHub repository from October 14th 2017 - current: 85 | 86 | - 7 releases 87 | - 10 contributors 88 | - 5 different organizations (based on current employment) 89 | - 205 commits (16,551 additions, 9,616 removals) 90 | - 60 Stars 91 | - 26 Forks 92 | - 10K+ pulls of primary docker image 93 | 94 | ### Use solid software engineering with documentation and tests hosted with appropriate technologies. 95 | 96 | The Enterprise Gateway has a suite of unit and integration tests that are run automatically on Travis on every PR and any commits to master. 97 | 98 | The Jupyter Enterprise Gateway community provides multiple resources that both users and contributors can use: 99 | 100 | - Source Code available at GitHub: https://github.com/jupyter-incubator/enterprise_gateway 101 | - Documentation available at ReadTheDocs: http://jupyter-enterprise-gateway.readthedocs.io/en/latest/ 102 | - Automated builds available at Travis.CI: https://travis-ci.org/jupyter-incubator/enterprise_gateway 103 | - Releases available at PyPi.org: https://pypi.org/project/jupyter_enterprise_gateway/ 104 | - Releases available at Conda Forge: https://github.com/conda-forge/jupyter_enterprise_gateway-feedstock 105 | - Related Docker Images available at Elyra organization at DockerHub: https://hub.docker.com/u/elyra/dashboard/ 106 | 107 | 108 | ### Demonstrate continued growth and development. 109 | 110 | Since entering incubation, the Jupyter Enterprise Gateway have added several new code contributors, performed 7 releases, and added support for two new resource managers - including support for Kubernetes. The community has also seen a recent increase on issues/questions submitted to the project, which implies more users are interested in, and deploying, Enterprise Gateway. 111 | 112 | In addition, we believe that by building upon the existing Jupyter stack, a majority of the necessary changes on Enterprise Gateway will be in the area of adding/maintaining support for resource managers (i.e., process proxy implementations). As a result Enterprise Gateway should continue to maintain compatibility/interoperability with new versions of Notebook components with little or no effort. 113 | 114 | ### Integrate well with other official subprojects. 115 | 116 | The Enterprise Gateway is a `jupyter/jupyter_core#Application` that uses programmatic APIs from `jupyter/notebook`, `jupyter/jupyter_client` and `jupyter/kernel_gateway` to enable communication with Jupyter kernels like `ipython/ipykernel`. By definition, it integrates with other official Subprojects. 117 | 118 | We are also looking for investigating deep integration with `JupyterHub` to decouple the kernel instances into specific pods in a kubernetes environment. 119 | 120 | ### Be developed according to the Jupyter governance and contribution model. 121 | 122 | The Enterprise Gateway is in the Jupyter Incubator, and under the Jupyter governance and contribution model since its inception. 123 | 124 | ### Have a well-defined scope. 125 | 126 | Jupyter Enterprise Gateway enables Jupyter Notebook to launch remote kernels in a distributed cluster, including Apache Spark managed by YARN, IBM Spectrum Conductor or Kubernetes. 127 | 128 | ### Be packaged using appropriate technologies such as pip, conda, npm, bower, docker, etc. 129 | 130 | The Enterprise Gateway is packaged using setup tools, released in both source and wheel format on PyPI, and installable using `pip`. It is also available in conda forge. 131 | 132 | ## Pros and Cons 133 | 134 | Pro: Extend Jupyter Stack to support distributed/remote Kernels 135 | 136 | Pro: The runtime can easily be extensible to support new cluster resource managers 137 | 138 | Con: Still requires couple extensions (e.g. NB2KG) to connect to the gateway 139 | 140 | ## Interested Contributors 141 | 142 | @parente, @rgbkrk 143 | -------------------------------------------------------------------------------- /122-jupyter-book-incorporation/repositoriestoinclude.csv: -------------------------------------------------------------------------------- 1 | Repository,Description,Last Updated,New Home 2 | myst-templates,A GitHub organization with templates for MyST renderers. Will remain a github org but move governance under the jupyter-book org.,2/28/2024,jupyter-book 3 | mystmd,Command line tools for working with MyST Markdown.,2/9/2024,jupyter-book 4 | myst-theme,Packages for creating MyST websites themes using React and Remix,2/8/2024,jupyter-book 5 | jupyter-book,"Create beautiful, publication-quality books and documents from computational content.",2/8/2024,jupyter-book 6 | jupyterlab-myst,Use MyST Markdown directly in Jupyter Lab,2/6/2024,jupyter-book 7 | jupyter-cache,A defined interface for working with a cache of executed jupyter notebooks,1/29/2024,jupyter-book 8 | jupyter-cache-upgrader,,1/25/2024,jupyter-book 9 | myst-spec,"MyST is designed to create publication-quality, computational documents written entirely in Markdown.",1/24/2024,jupyter-book 10 | thebe,Turn static HTML pages into live documents with Jupyter kernels.,1/20/2024,jupyter-book 11 | cookiecutter-jupyter-book,Cookiecutter template for a simple jupyter book,11/29/2023,jupyter-book 12 | github-activity,Simple markdown changelogs for GitHub repositories,11/22/2023,jupyter-book 13 | myst-enhancement-proposals,MyST Enhancement Proposals (MEPs),10/9/2023,jupyter-book 14 | mystmd.org,A splash page for the MyST Markdown ecosystem,8/2/2023,jupyter-book 15 | jupyterlab-myst-quickstart,Quickstart examples for working with MyST in Jupyter,7/5/2023,jupyter-book 16 | github-action-demo,A demonstration repository to build and host a book with GitHub Actions,6/21/2023,jupyter-book 17 | myst-theme-patches,A minimal repository containing essential patches for myst-theme deveopment,6/14/2023,jupyter-book 18 | thebe-binder-base,A minimal binder setup for thebe demos,4/27/2023,jupyter-book 19 | myst-vs-code,A syntax highlighter for the MyST Markdown format,3/6/2023,jupyter-book 20 | mystmd-quickstart,Repository to be used in the mystjs quickstart guide,1/25/2023,jupyter-book 21 | thebe-core,Typescript based Jupyter connectivity and code execution layer for thebe,8/22/2022,jupyter-book 22 | myst-demo,A web-component for interactive myst demos in documentation.,6/20/2022,jupyter-book 23 | unified-myst,A repository of packages for working with MyST in the https://unifiedjs.com/ ecosystem,4/30/2022,jupyter-book 24 | sphinx-book-theme,A clean book theme for scientific explanations and documentation with Sphinx,2/9/2024,executablebooks 25 | meta,A community dedicated to supporting tools for technical and scientific communication and interactive computing,2/8/2024,executablebooks 26 | sphinx-thebe,"A Sphinx extension to convert static code into interactive code cells with Jupyter, Thebe, and Binder.",2/7/2024,executablebooks 27 | mdformat-tables,An mdformat plugin for rendering tables,2/7/2024,executablebooks 28 | mdformat,CommonMark compliant Markdown formatter,2/6/2024,executablebooks 29 | mdit-py-plugins,Collection of core plugins for markdown-it-py,2/5/2024,executablebooks 30 | sphinx-external-toc,A sphinx extension that allows the site-map to be defined in a single YAML file,2/5/2024,executablebooks 31 | markdown-it-py,"Markdown parser, done right. 100% CommonMark support, extensions, syntax plugins & high speed. Now in Python!",2/5/2024,executablebooks 32 | MyST-NB,Parse and execute ipynb files in Sphinx,2/5/2024,executablebooks 33 | MyST-Parser,"An extended commonmark compliant parser, with bridges to docutils/sphinx",2/5/2024,executablebooks 34 | sphinx-design,"A sphinx extension for designing beautiful, screen-size responsive web components.",1/29/2024,executablebooks 35 | sphinx-copybutton,"Add a ""copy"" button to code blocks in Sphinx",1/29/2024,executablebooks 36 | rst-to-myst,Convert ReStructuredText to MyST Markdown,1/29/2024,executablebooks 37 | sphinx-tabs,Tabbed views for Sphinx,1/29/2024,executablebooks 38 | web-compile,"A CLI to compile/minify SCSS & JS, and associated pre-commit hook.",1/29/2024,executablebooks 39 | mdurl,URL utilities for markdown-it (a Python port),1/29/2024,executablebooks 40 | sphinx-togglebutton,Show and hide content with a button in Sphinx,12/22/2023,executablebooks 41 | sphinx-panels,A sphinx extension for creating panels in a grid layout,12/18/2023,executablebooks 42 | sphinx-jupyterbook-latex,Supporting LaTeX infrastructure for Jupyter Book,12/11/2023,executablebooks 43 | grant-admin,,12/6/2023,executablebooks 44 | .github,"Community health files: Contributing guidelines, Code of Conduct, ...",12/4/2023,executablebooks 45 | team-compass,Organizational policy and internal documentation for the executablebooks community,11/28/2023,executablebooks 46 | grant-application,Web publishing tool funding application,11/21/2023,executablebooks 47 | mdformat-plugin,A template for creating an mdformat parser extension plugin,11/3/2023,executablebooks 48 | sphinx-remove-toctrees,Speed up Sphinx builds by selectively removing toctrees from some pages,11/3/2023,executablebooks 49 | markdown-it-myst-extras,Additional plugins required for the MyST specification,9/26/2023,executablebooks 50 | markdown-it-dollarmath,A markdown-it plugin for $-delimited math,9/26/2023,executablebooks 51 | markdown-it-amsmath,A markdown-it plugin for amsmath LaTeX environments.,9/26/2023,executablebooks 52 | sphinx-proof,"A Sphinx extension for producing proof, theorem, lemma, definition, remark, conjecture, corollary and algorithm directives.",9/18/2023,executablebooks 53 | mdformat-myst,Mdformat plugin for MyST compatibility,8/26/2023,executablebooks 54 | mdformat-deflist,mdformat plugin for deflist,8/26/2023,executablebooks 55 | staged-recipes,A place to submit conda recipes before they become fully fledged conda-forge feedstocks,8/1/2023,executablebooks 56 | sphinx-comments,hypothes.is interaction layer with Sphinx,5/3/2023,executablebooks 57 | myst-article-theme,A light-weight theme for rendering scientific articles and associated notebooks,4/26/2023,executablebooks 58 | myst-book-theme,A lightweight MyST theme designed to mimic the look-and-feel of an interactive book.,4/26/2023,executablebooks 59 | markdown-it-docutils,A markdown-it plugin for implementing docutils style roles/directives.,3/6/2023,executablebooks 60 | jupyterlab-mystjs,TO ARCHIVE: Jupyterlab extension using the MySTjs parser,2/10/2023,executablebooks 61 | sphinx-exercise,A Sphinx extension for producing exercise and solution directives.,1/23/2023,executablebooks 62 | markdown-it-plugin-template,A template for creating a markdown-it plugin.,1/9/2023,executablebooks 63 | sphinx-examples,A Sphinx extension to create examples of source markdown and the result of rendering it.,5/3/2022,executablebooks 64 | mdformat-footnote,Footnote format addition for mdformat,2/16/2022,executablebooks 65 | quantecon-mini-example,A short example showing how to write a lecture series using Jupyter Book 2.0.,11/13/2021,executablebooks 66 | sphinx-multitoc-numbering,A Sphinx extension to support continuous numbering of sections across multiple tocs in HTML output.,7/6/2021,executablebooks 67 | myst-react,TO ARCHIVE: A web-based UI for rendering MyST Markdown,5/28/2021,executablebooks 68 | sphinx-tomyst,A sphinx translator for producing myst syntax files,5/26/2021,executablebooks 69 | myst-standard,TO ARCHIVE: A meta-repository to discuss and plan for the MyST standard,5/20/2021,executablebooks 70 | sphinx-yaml-config,,5/8/2021,executablebooks 71 | jupytext,"Jupyter Notebooks as Markdown Documents, Julia, Python or R scripts",4/25/2021,executablebooks 72 | quantecon-example,A demonstration of Jupyter Book functionality using QuantEcon Python programming source material.,10/30/2020,executablebooks 73 | sphinx-conditional-asset,A small extension for developers of Sphinx to conditionally add assets to a page.,10/25/2020,executablebooks 74 | mistletoe-ebp,"The EPB fork of mistletoe: A fast, extensible and spec-compliant Markdown parser in pure Python.",8/6/2020,executablebooks 75 | mappings,Work on Mappings between IPYNB-RST and IPYNB-MD++,7/29/2020,executablebooks 76 | rst2myst,Tools for converting RST files to MyST-NB files,6/23/2020,executablebooks 77 | python-pkg-cookiecutter,"A cookiecutter for python packages, with docs and GitHub Actions CI",6/3/2020,executablebooks 78 | sphinx-ext-autodoc,Auto-documentation for sphinx extension components,6/3/2020,executablebooks 79 | myst-nb.example-project,An example project for working with myst-nb and developing myst syntax,6/3/2020,executablebooks 80 | tcs_to_jb,For conversion of TCS to Jupyter Book format,5/26/2020,executablebooks 81 | cli,"馃洃馃洃DEPRECATED, see https://github.com/ExecutableBookProject/jupyter-book馃洃馃洃",5/15/2020,executablebooks 82 | multi-book-demo,,5/1/2020,executablebooks 83 | myst-parser.example-project,An self contained example project for testing and development of myst syntax using myst-parser,4/11/2020,executablebooks 84 | sphinx-jupyter-book-theme,An experimental Sphinx theme for Jupyter Book,3/5/2020,executablebooks 85 | sphinx-globaltoc,,2/29/2020,executablebooks 86 | write_ups,Write ups and publications that document the project,2/25/2020,executablebooks 87 | markup_test_cases,Markup test cases for source files,2/14/2020,executablebooks 88 | profile-sphinx-site,A tiny repository to profile a Sphinx site,1/23/2020,executablebooks 89 | myst,TO ARCHIVE: Myst - Markedly Structured Text,11/17/2019,executablebooks -------------------------------------------------------------------------------- /17-jupyter-dashboards-extension-incorporation/jupyter-dashboards-extension-incorporation.md: -------------------------------------------------------------------------------- 1 | # Jupyter Dashboards Extension Incorporation 2 | 3 | ## Problem 4 | 5 | > "Data science enables the creation of data products." – Mike Loukides in [What is data science?](https://www.oreilly.com/ideas/what-is-data-science) 6 | 7 | Alice is a Jupyter Notebook user. Alice prototypes data access, modeling, plotting, interactivity, etc. in a notebook. Now Alice needs to deliver a dynamic dashboard for non-notebook users. Today, Alice must step outside Jupyter Notebook and build a separate web application. Alice cannot simply do the following: 8 | 9 | * Select notebook content (widgets, plots, images, markdown, etc.) to show in a dashboard 10 | * Arrange the dashboard content in a grid- or report-style layout 11 | * Publish the dashboard layout as a standalone web application 12 | * Push updates to the published web app when the notebook changes 13 | * Secure the web app separate from the notebook authoring environment 14 | 15 | ## Proposed Enhancement 16 | 17 | The incubating [dashboards layout extension for Jupyter Notebook](https://github.com/jupyter-incubator/dashboards) (hereafter, *dashboards extension*) is one of three incubation projects that, together, address the issues above. The community should consider incorporating the dashboards extension as an official Jupyter Subproject as a first step toward a robust dashboarding solution. 18 | 19 | ## Detailed Explanation 20 | 21 | The full jupyter-incubator dashboards effort covers: 22 | 23 | 1. Arranging notebook outputs in a grid- or report-like layout (https://github.com/jupyter-incubator/dashboards) 24 | 2. Bundling notebooks and associated assets for deployment as dashboards https://github.com/jupyter-incubator/dashboards_bundlers) 25 | 3. Serving notebook-defined dashboards as standalone web apps (https://github.com/jupyter-incubator/dashboards_serve 26 | 27 | The effort has close ties to https://github.com/jupyter-incubator/declarativewidgets which provides one way (but not the only way) of enabling rich interactivity in notebook-defined dashboards. 28 | 29 | **N.B.** - Per the discussion at the spring 2016 dev meeting and steps noted in the [jupyter/roadmap](https://github.com/jupyter/roadmap#graduate-jupyterdashboards-from-incubator), this document covers the graduation of the `jupyter-incubator/dashboards` project alone. We will submit proposals for the other related projects in the near future. 30 | 31 | ### Current and Potential Use Cases 32 | 33 | The dashboards extension supports the following use cases: 34 | 35 | * Dashboard layout mode for arranging notebook cell outputs in a grid- or report-like fashion 36 | * Dashboard view mode for interacting with an assembled dashboard within the Jupyter Notebook 37 | * Ability to share notebooks that have dashboard layout metadata in them with other Jupyter Notebook users for layout and viewing 38 | 39 | When used with the other incubating dashboard projects, it supports the following workflow: 40 | 41 | 1. Alice authors a notebook document using Jupyter Notebook. She adds visualizations and interactive widgets. 42 | 2. Alice arranges her notebook cells in a grid- or report-like dashboard layout. 43 | 3. Alice one-click deploys her notebook and associated assets to a Jupyter Dashboards server. 44 | 4. Bob visits the dashboards server and interacts with Alice's notebook-turned-dashboard application. 45 | 5. Alice updates her notebook with new features and redeploys it to the dashboards server. 46 | 47 | ![End-to-end dashboards workflow](https://raw.githubusercontent.com/wiki/jupyter-incubator/dashboards/images/workflow.png) 48 | 49 | For example, 50 | [here is an animation](https://ibm.box.com/shared/static/t9zfbloipanirdm8u0vg3ggbmp5c1q8p.gif) of a user arrange portions of a notebook into a dashboard and deploying it to a dashboard server. 51 | 52 | ### Current Features 53 | 54 | The ["What It Gives You" section of the project README](https://github.com/jupyter-incubator/dashboards#what-it-gives-you) details the currently implemented features. The [notebooks directory](https://github.com/jupyter-incubator/dashboards/tree/master/etc/notebooks) in the project contains numerous examples. 55 | 56 | ### Design and Implementation 57 | 58 | The dashboards extension is a pure JavaScript extension for the Jupyter Notebook frontend. It adds a toolbar and menu items for switching between three views: notebook, dashboard layout, and dashboard preview. It also adds a set of menu items for quickly adding/removing all cells to/from the dashboard layout. Help for creating dashboard layouts appears in situ in the layout mode. 59 | 60 | The extension currently supports two types of layout: grid and report. The user moves, resizes, and shows/hides notebook cell widgets/outputs in a responsive grid in the former. The user simply shows/hides cell widgets/outputs in top-down notebook order in the latter. The extension [persists information about the layouts within the notebook document](https://github.com/jupyter-incubator/dashboards/wiki/Dashboard-Metadata-and-Rendering). 61 | 62 | Both the notebook extension and the notebook document specification are extensible. They may grow support for additional kinds of layout in the future (e.g., fixed grids, paged wizards). 63 | 64 | The extension is compatible with notebook version 4.0 through 4.2. Users install it using pip and manage it via the `jupyter` command line interface. 65 | 66 | ### Criteria for Incorporation 67 | 68 | The [Jupyter Governance - New Subproject Process](https://github.com/jupyter/governance/blob/master/newsubprojects.md) lists the criteria used to evaluate projects for incorporation in to the official Jupyter organization. We address each of these criteria below. 69 | 70 | #### Have an active developer community that offers a sustainable model for future development. 71 | 72 | The project has contributions from 12 developers on GitHub and has been under active development over the past 10 months. We foresee future development of the extension taking the following primary forms: 73 | 74 | 1. Fixes for reported issues 75 | 2. Compatibility updates for new versions of Jupyter Notebook (e.g., 4.3, 4.4, ...) 76 | 3. Contributed enhancements to the existing extension within the scope of the project 77 | 4. Enhancements to the metadata spec and layout authoring options to remain compatible with future dashboard features in Jupyter Lab 78 | 79 | The first two are straightforward and can be handled by current or future maintainers. The third, if and when it happens, can be integrated by current and future maintainers project as well. The fourth will be driven by future Jupyter Lab efforts to support dashboards. 80 | 81 | #### Have an active user community. 82 | 83 | The `jupyter-incubator/dashboards` repository has 151 stars and 30 watchers on GitHub. We have received questions, bug reports, and enhancement requests on GitHub, the Google Group, and Gitter. 84 | 85 | #### Use solid software engineering with documentation and tests hosted with appropriate technologies. 86 | 87 | The extension has a small suite of frontend smoke tests that use Selenium to drive the UI additions to the Notebook server. The tests run on SauceLabs via Travis CI on every PR merge. 88 | 89 | User and developer documentation about the extension resides on ReadTheDocs at http://jupyter-dashboards-layout.readthedocs.io. We plan to migrate additional information to these docs from https://github.com/jupyter-incubator/dashboards/wiki. 90 | 91 | #### Demonstrate continued growth and development. 92 | 93 | See "Have an active developer community that offers a sustainable model for future development." 94 | 95 | #### Integrate well with other official Subprojects. 96 | 97 | The dashboards extension is a [properly packaged Jupyter Notebook extension](http://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Distributing%20Jupyter%20Extensions%20as%20Python%20Packages.html) with [documentation about its extensible, versioned metadata](https://github.com/jupyter-incubator/dashboards/wiki/Dashboard-Metadata-and-Rendering) so that other Jupyter tools may render its dashboard layouts (e.g., [dashboards server](https://github.com/jupyter-incubator/dashboards_server) now, Jupyter Lab in the future). 98 | 99 | #### Be developed according to the Jupyter governance and contribution model. 100 | 101 | The dashboards extension is in the Jupyter Incubator, and under the Jupyter governance and contribution model since its inception. 102 | 103 | #### Have a well-defined scope. 104 | 105 | The purpose of the dashboards extension is to provide a user interface for arranging notebook cells into interactive dashboard layouts. We have separated features beyond this scope into other projects (e.g., serving dashboards as standalone web applications → jupyter-incubator/dashboards_server project) and have recommended similar actions for community requests (e.g., support for dashboard layouts when converting to PDF → add support for dashboard metadata to nbconvert). 106 | 107 | #### Be packaged using appropriate technologies such as pip, conda, npm, bower, docker, etc. 108 | 109 | The dashboards extension is packaged using setuptools, released on PyPI, and installable using `pip`. Users manage the extension after `pip install` via the `jupyter` CLI. 110 | 111 | ## Pros and Cons 112 | 113 | * Pro: Gives Jupyter users an avenue for creating dashboard layouts in Jupyter notebooks today. 114 | * Pro: Works with the other incubating projects to enable deployment of dashboard-notebooks as standalone web applications. 115 | * Pro: Defines a spec for storing layout metadata in notebook documents and steps for rendering those layouts. 116 | * Pro: Serves as a test bed and reference implementation for future dashboard efforts in Jupyter Lab. 117 | * Con: There is currently [no consensus on a light proposal](https://github.com/jupyter/enhancement-proposals/pull/15) for where to store layout metadata in notebooks, let alone a single layout metadata spec to be shared across all tools (including those beyond the dashboards use case). 118 | 119 | ## Interested Contributors 120 | 121 | @parente 122 | -------------------------------------------------------------------------------- /jupyter-enhancement-proposal-guidelines/jupyter-enhancement-proposal-guidelines.md: -------------------------------------------------------------------------------- 1 | # Jupyter Enhancement Proposals (JEP) Guidelines 2 | 3 | **Note**: This JEP repo is the **canonical "source of truth"** for individual JEPs, 4 | the JEP process, and activity on JEPs. 5 | 6 | ## Goals of the JEP process 7 | 8 | Project Jupyter uses the Jupyter Enhancement Proposal process (JEP) 9 | to address distributed collaboration and experimentation. 10 | The primary guiding principle of the JEP process is to 11 | encourage collaboration and discussion as early as possible in the lifecycle 12 | of a major proposed change in Jupyter, with the goal of preventing costly rework, 13 | competing ideas, and unnecessary forking or fragmentation of ideas. 14 | 15 | Several sub-goals exist for this process: 16 | 17 | - **Maximize success of large proposals** that get stalled in the wrong venue (e.g. a single PR comment thread) 18 | - Provide a **better alternative to “piecemeal” development** where multiple PRs 19 | to build an end-to-end set of functionality are split across multiple GitHub 20 | project without broad consensus, context, or guidance. 21 | - Provide a **clear, time-limited, and authoritative process for work proposals**, 22 | to facilitate funding conversations. (e.g. provide a concrete artifact to reference 23 | in a grant proposal, roadmap item, etc.) 24 | - Provide a **consolidated “stream” of all proposals across the entire Jupyter community** 25 | that contributors of all levels can monitor and selectively engage in. 26 | 27 | ## Tenets of the JEP process 28 | 29 | The JEP process operates under the following tenets: 30 | 31 | - **The JEP process is intended for changes of non-trivial scope.** 32 | “Non-trivial” is addressed below in the “JEP / Not-a-JEP Rubric” of this document. 33 | 34 | - **The JEP process naturally complements the PR process, but does not replace it.** 35 | A thoroughly-reviewed and approved JEP is a valuable reference during a PR to 36 | reduce friction, reduce time-consuming context sharing, and encapsulate decisions 37 | and other discussions. Moving a premature PR into a JEP should be a lightweight 38 | process that doesn’t cause friction for the contributor. 39 | 40 | For example, GitHub issue and PR templates across the entire Jupyter project should have 41 | references to the JEP process as a possible outcome of a given PR. 42 | 43 | - **There is one JEP repository, all Jupyter-governed projects must use it.** 44 | To faciliate the easiest possible adoption and high visibility of ideas, a 45 | single JEP repository will be used. Even if a JEP only applies to a single organization. 46 | 47 | - The JEP process **has multiple valid use cases**. Each use case might have a 48 | slightly different expected workflow. Some expected use cases include: 49 | 50 | - Non-trivial feature proposals within a single component that would benefit from 51 | process. (e.g., a non-trivial change to JupyterLab that would benefit from 52 | formal process within the JupyterLab project) 53 | - Non-trivial features or improvements that span multiple projects. 54 | - Any proposed changes to published APIs or core specifications (e.g., nbformat) 55 | - Changes to the JEP process itself. 56 | - Launching a major project in one of the Jupyter GitHub organizations. 57 | 58 | 59 | ## JEP Submission Workflow 60 | 61 | The following sections describe the major checkpoints in the JEP process. 62 | Note that at any time, the JEP author may withdraw their JEP (and if it has 63 | been added to the README, its status becomes "withdrawn"). 64 | 65 | ### Phase 1: Pre-proposal 66 | 67 | This is the least formal stage of any jupyter enhancement proposals. During this 68 | phase, discussions on the mailing list, community forum, in-person, on github issues 69 | are all fine to gauge community interest, feasibility, consequences, and to 70 | scope out technical impact and implementation details. 71 | 72 | In order to transition out of the pre-proposal stage, the following checklist must be complete: 73 | 74 | 1. **Create a GitHub issue** in the Jupyter Enhancement Proposals repo that 75 | 1. Briefly outlines the proposal 76 | 2. Suggests a review team (optional) 77 | 3. Declares why it should be a JEP (See the “JEP / Not-a-JEP Rubric” below.) 78 | 2. **Identify a Shepherd** to see the process through. (Shepherds are assigned 79 | on a round-robin basis from a set of interested engaged volunteers). 80 | 3. **Decide if it's a JEP** according to the criteria listed above. The Shepherd decides if the JEP criteria have been met. 81 | 82 | **If it's a JEP**. The JEP author creates a new PR with the contents of the JEP proposal. See [this markdown template](JEP-TEMPLATE.md) for a suggested structure of the pull-request. 83 | 84 | Subsequent discussion and decisions about the JEP will happen in that PR, and 85 | we now **Move to the RFC phase**. 86 | 87 | **If it’s not a JEP**. The shepherd provides a reason for why the issue 88 | doesn't meet JEP criteria, and closes the issue. 89 | 90 | ### Phase 2: Request for Comments for the JEP 91 | 92 | Phase two begins when **the JEP author submits a draft of the JEP as a PR to the JEP repository**. 93 | The Shepherd assigns a number to the JEP according to the pull-request number, and updates 94 | the README of the JEP repository with information about the now in-progress JEP. 95 | The Shepherd then works with the JEP author to 96 | **identify and notify relevant individuals in the Jupyter community to review and comment**. 97 | 98 | Once this group has been notified, the JEP process enters the RFC phase. 99 | 100 | During the **Request For Comment (RFC) phase**, the proposal is iterated on with 101 | feedback from the Review Team and the community at large. The Shepherd helps engage 102 | the Review Team and encourage productive discussion. After the Review Team members 103 | have signed off on the JEP, with the criteria that there are no major objections, 104 | and at least some of the Review Team are in favor, the Shepherd initiates the Final Comment Period. 105 | 106 | In the **Final Comment Period (FCP)**, the community at-large has at least 10 calendar days 107 | to provide any final comments on the JEP. A JEP may revert back to RFC if 108 | objections from the community are supported by the Review Team. 109 | 110 | At the end of the FCP, the JEP is either: 111 | 112 | * **approved**, in which case the PR is merged and work may commence on implementing the JEP (see Phase 3, below) 113 | * **rejected**, in which case the PR is closed 114 | * asked to return to the RFC phase. 115 | 116 | In each case, the JEP's status should be updated in the README of the 117 | `enhancement-proposals` repository. 118 | 119 | ### Phase 3: Work Commences 120 | 121 | Once a JEP has been merged into the `jupyter/enhancement-proposal` repository, 122 | implementation can commence on the JEP. The JEP author does not need to implement the JEP themselves. 123 | 124 | If the JEP requires one or more pull-requests to be implemented, the author of the PRs should 125 | provide a reference to the JEP so that reviewer has background context on the JEP. 126 | As the JEP is being implemented, the JEP text should be amended with with addendums to 127 | denote the progress of the implementation using the following stages. 128 | 129 | 1. In progress implementation via (list of PRs). 130 | 2. Fully implemented via (list of PRs). 131 | 132 | If in the course of implementation, it is discovered that the implementation needs to 133 | be radically different from what was defined in the original JEP, then a pull 134 | request is submitted to modify the original JEP with the new necessary implementation, 135 | and a note citing the need for a modification to the JEP. 136 | This pull request should be re-approved by the original review team. 137 | 138 | ### JEP Pathways and Status 139 | 140 | Below is a rough guide that describes the JEP process and its possible pathways. 141 | 142 | ``` 143 | +-----------+ 144 | | | 145 | | withdrawn | 146 | | | +-----------+ 147 | +-----------+ | | 148 | JEP may be withdrawn | rejected | 149 | at any stage | | 150 | +-----^-----+ 151 | | 152 | +------+------+ 153 | +--------------+ +-----------+ | | +-------------+ +-----------+ 154 | | | | | | Request for | | | | | 155 | | pre-proposal +---> submitted +-------------> Comments +------------> in progress +---> completed | 156 | | | | | identify | (RFC) | approved | | | | 157 | +--------------+ +-----------+ review | | +-------------+ +-----------+ 158 | team +------+------+ 159 | ``` 160 | 161 | 162 | ## What qualifies as a JEP? 163 | 164 | This section contains a set of principles to help determine when something is a JEP. 165 | These should be used to determine when something becomes a PR during the JEP 166 | pre-proposal stage, as well as to determine when a PR becomes a JEP at an individual repo level. 167 | 168 | **Principles to follow** 169 | 170 | Below are a few example guidelines to follow when deciding if an idea should include 171 | a JEP (If yes, it requires a JEP). Under each question is a relevant example proposal. 172 | 173 | - Does the proposal/implementation require PRs across multiple orgs? 174 | - **Example:** Defining a unique cell identifier 175 | - Does the proposal/implementation PR impact multiple orgs, or have widespread community impact? 176 | - **Example:** Updating nbformat 177 | - Does the proposal/implementation change an invariant in one or more orgs? 178 | - **Example:** Defining a unique cell identifier 179 | - **Example:** Deferred kernel startup 180 | - Does the proposal/implementation create a new concept that will impact multiple repositories? 181 | - **Example:** Sandboxed cell outputs 182 | - Does the proposal involve creating a new repository or sub-project? 183 | 184 | ## The JEP public archive 185 | 186 | A public website contains a readable archive of all JEP proposals. 187 | It contains list of all JEPs that have entered a "final" state 188 | (e.g., "Completed", "Withdrawn", "Rejected"). The content of each JEP will 189 | be displayed in a readable fashion. When a JEP enters into a final state, it 190 | is added to this website. 191 | 192 | Note that the JEPs themselves contain the content, while the website is just a 193 | quick way to display them in a reading-friendly format. 194 | 195 | ## Background 196 | 197 | For a background of the JEP process, and recent efforts to improve it, see 198 | [the meta-JEP readme](../29-jep-process/jep-process.md). 199 | 200 | -------------------------------------------------------------------------------- /108-jupyter-subdomain-for-schemas/jupyter-subdomain-for-schemas.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: subdomain and repository for publishing schemas under jupyter.org 3 | authors: Zach Sailer, Nick Bollweg, Tony Fast 4 | issue-number: "[#107](https://github.com/jupyter/enhancement-proposals/issues/107)" 5 | pr-number: 108 6 | date-started: 2023-04-24 7 | --- 8 | 9 | # jupyter.org subdomain and repository for publishing schemas 10 | 11 | 12 | ## Summary 13 | 14 | Create a subdomain under jupyter.org and a repository for publishing machine- and human-readable JSON Schemas provided by the core Jupyter ecosystem. 15 | 16 | ## Motivation 17 | 18 | Jupyter defines various specifications for interactive computing that are widely used in our world today. Examples include: 19 | * the notebook format 20 | * the kernel messaging specification 21 | * the kernel launching specification (a.k.a. kernelspec) 22 | * widget communication specification 23 | * the Jupyter Server REST API 24 | * the JupyterHub REST API 25 | * Jupyter configuration files 26 | * Jupyter Extension Proposal front matter 27 | 28 | Each of these specification have varying level of documentation or validation. Some specifications were never explicitly documented but, due to their critical nature and decade-long existence, have become implicitly fixed in time in reference implementations. 29 | 30 | There has been a surge of proposals aimed at backing these specifications with JSON Schemas (the motivation for schematizing these core specs is described in those JEPs). Each JSON schema (should) have an `$id` keyword defining a URI where that schema can be found, enabling other tools and schemas to reference it. 31 | 32 | **The URIs for a JSON schema should be static and "always" available.** This JEP aims to provide a single subdomain where all core "Jupyter-verified" schemas can be reliably hosted. 33 | 34 | ## Guide-level explanation 35 | 36 | Jupyter now hosts a subdomain for publishing schemas for humans and computers, schema.jupyter.org, where all core Jupyter JSON Schemas and related tools will be hosted. 37 | 38 | This site is built from a new repository, https://github.com/jupyter-standards/schemas, which collects and organizes these schema from across the Jupyter ecosystem. 39 | 40 | Any schemas currently hosted in core Jupyter subprojects can find a new home in this repository, updated with the new subdomain, and published under the new URL. 41 | 42 | New (and old) schemas can be submitted to the schemas repository by pull request. Members of the individual subprojects should have the authority to publish schemas for the project they represent. If a schema affects multiple subprojects, it should be reviewed and merged by representatives from those subprojects. 43 | 44 | There are some rules required of all schemas, where each schema must: 45 | 46 | - declare a `$schema` 47 | - must be a well-known JSON Schema draft, or a metaschema also in this repo 48 | - conform to declared schema 49 | - declare a `$id` which must 50 | - be unique across the repository 51 | - must start with `https://schema.jupyter.org/` 52 | - must be namespaced by the subproject name (if applicable) 53 | - must include a version 54 | 55 | For example, to create an "event" schema for Jupyter Server, the `$id` field might look like: 56 | ```json 57 | { 58 | "$schema": "http://json-schema.org/draft-07/schema", 59 | "$id": "https://schema.jupyter.org/jupyter_server/some-event/v1/event.json", 60 | ... 61 | } 62 | ``` 63 | 64 | ## Reference-level explanation 65 | 66 | The `@jupyter-standards/schemas` repository must provide, at a minimum: 67 | 68 | - a humane experience for authoring and reviewing schema-as-authored 69 | - a build/test tool chain which enforces the _schema rules_ mentioned above 70 | - a documentation tool chain for building a human-readable website, doubling as the fully-dereferenceable URL source for machine-readble schema 71 | 72 | To encourage the humane authoring of the notoriously-fickle JSON format, it is proposed schema-at-rest may be _authored_ in a number of _de facto_ file formats. 73 | 74 | For the most part, validation can be captured in an internal, not-neccessarily-published JSON Schema, with the exception of the `$id` uniqueness rule. This would be easily achieved with an existing testing framework, such as `pytest`, and executed either locally, on mybinder.org or in continuous integration, such as GitHub actions. 75 | 76 | The documentation site should likely be based on `sphinx`, which includes in its broad ecosystem good support for JSON Schema-based specifications such as [`sphinx-jsonschema`](https://pypi.org/project/sphinx-jsonschema/), [`sphinxcontrib-openapi`](https://github.com/sphinx-contrib/openapi), [`asyncapi-sphinx-ext`](https://asyncapi-sphinx-ext.readthedocs.io). These would further provide the ability for downstream documentation sites to unambiguously link to constructs in the schema website, via the well-known `objects.inv` file. ReadTheDocs would be an ideal candidate for review, providing many desirable features such as versioned releases, and per-PR review sites, while really any static host (such as GitLab Pages) would be suitable for the canonical URL. 77 | 78 | ### Notional Workflow 79 | 80 | Below, we describe a notional workflow for maturing a new pull request, driven by a Jupyter Extension Proposal, to publishing a new version of the documentation website and schemas. 81 | 82 | ```{mermaid} 83 | flowchart LR 84 | classDef external stroke-dasharray:4px 85 | JEP:::external --> authoring & examples 86 | subgraph "a pull request" 87 | subgraph authoring 88 | direction LR 89 | .schema.toml 90 | .schema.yaml 91 | .schema.json5 92 | end 93 | subgraph examples 94 | direction LR 95 | .good.example.* 96 | .bad.example.* 97 | .bad.example.expected.* 98 | end 99 | subgraph canonical 100 | .schema.json 101 | end 102 | subgraph documentation 103 | .schema.html 104 | .report.html 105 | objects.inv 106 | end 107 | normalize 108 | test 109 | sphinx 110 | lint 111 | link-check 112 | pass-fail 113 | review 114 | end 115 | authoring & examples --> normalize((normalize)) --> .schema.json 116 | canonical --> sphinx((sphinx)) --> .schema.html 117 | canonical --> lint((lint)) & test((test)) --> .report.html 118 | .schema.html --> link-check((link check)) 119 | review --> pass-fail 120 | lint & test & link-check --> pass-fail{OK?} 121 | pass-fail --> version 122 | subgraph website 123 | version --> latest & stable 124 | end 125 | ``` 126 | 127 | 128 | ## Rationale and alternatives 129 | 130 | Adopting standards-based representations of key Jupyter interfaces encourages broader adoption of, and adherance to, these protocols, improving the portability of Jupyter documents and interoperability of first- and third-party implementations. 131 | 132 | Especially in the [packaging](#packaging) case described below, having a single-source-of-truth which can be consumed by both sides of a protocol would increase the Jupyter community's ability to innovate in a way that does not negatively impact the user community. 133 | 134 | The primary alternatives is to _do nothing_, keeping interface definitions tightly-bound to their reference implementations. This would preserve, and likely compound, the challenges observed today, where it is unclear _where_ changes need to occur. 135 | 136 | 137 | ## Prior art 138 | 139 | The [Debug Adapter Protocol](https://github.com/microsoft/debug-adapter-protocol) (DAP) stores its source of truth in a canonical [JSON Schema](https://github.com/microsoft/debug-adapter-protocol/blob/main/debugAdapterProtocol.json), and from this, generates markdown-based documentation. 140 | 141 | In contrast to other protocols managed by the same parent organization, which rely on bespoke documentation and specification language, this approach made it relatively easy for Jupyter to previously integrate with DAP. 142 | 143 | ## Unresolved questions 144 | 145 | ## Future possibilities 146 | 147 | Once the schema repository and subdomain exists, a number of powerful features can be further enabled. 148 | 149 | ### Higher-order Specifications 150 | 151 | The [OpenAPI](https://openapis.org) and [AsyncAPI](https://asyncapi.com) specification formats provide tool-independent ways to document both REST-like systems as well as RPC-style systems. These augment JSON Schema with the understanding of how _inputs_ and _outputs_ are tied together logically, with concepts like URL paths and operations signatures, which can also be validated. 152 | 153 | While a number of Jupyter tools have started using OpenAPI, these are generally static, and are used to drive documentation, but may not be used to drive validation or testing. 154 | 155 | Either in the proposed schema repository, or in future siblings, these specifications could be used to document, test, and potentially partially implement, the correct functioning of the Jupyter ecosystem. 156 | 157 | ### Packaging 158 | 159 | As a widely-implemented standard, JSON Schema can be used to build packages, at various levels of specificity, that could be consumed by both Jupyter downstreams. 160 | 161 | By centralizing the definition, documentation, and testing of these packages, they can be delivered to Jupyter tools and others harmoniously and efficiently. 162 | 163 | #### Schema-at-rest 164 | 165 | Leveraging the Jupyter _well-known paths_ for declarative Jupyter configuration and asset discovery, all schema could be delivered via a canonical `jupyter-schemas` package, populating the `{sys.prefix}/share/jupyter/schemas` with a file tree identical to the published site. 166 | 167 | #### Typings 168 | 169 | A number of languages provide means for declaring the _type system_ embodied in a JSON Schema. Automating the creation, testing, publishing, and documentation of such type-only packages for a language, starting with the core Jupyter tool-authoring languages (Python and TypeScript), would allow for light-weight, static-analysis-friendly ways to keep first- and third-party tools in a conforming state, without incurring any additional runtime dependencies. 170 | 171 | Typings would _not_ be able to validate some properties of schema, such as string formats and regular expressions, but _would_ provide the general "shape" of the specification well enough to help a downstream decide if a new version of a particular schema will break their users' software. 172 | 173 | #### Validators 174 | 175 | In many languages, there are a small number of _de facto_ JSON Schema validator implementations, such as Python's `jsonschema` and TypeScript's `ajv`. With a slightly different template than the _typing_ approach, these could be provided with reasonable dependency constraints, for downstream tools to apply full validation, still leveraging the _typings_ above. 176 | 177 | #### Models 178 | 179 | A number of other validating, model-based systems can also be derived from schema in a way which would still conform to the _typings_ described above, but offer additional user and developer experience benefits. 180 | 181 | For example, in Python such generated packages could include `jupyter-schema-pydantic`, `jupyter-schema-attrs`, types compiled with `jupyter-schema-mypyc`, etc. as well as Jupyter's in-house `jupyter-schema-traitlets` or `jupyter-schema-widgets`. 182 | 183 | ### Composition 184 | 185 | A top-level schema describing the entire Jupyter _vocabulary_, compatible with more recent JSON Schema drafts, can be derived from the schema-as-built. 186 | 187 | ```json 188 | { 189 | ... 190 | "$vocabulary": { 191 | "{/subproject}{/schema_name}{/version}": True 192 | }, 193 | ... 194 | } 195 | ``` 196 | 197 | ## References 198 | 199 | - [JSON schema documentation](https://json-schema.org/) 200 | -------------------------------------------------------------------------------- /18-jupyter-declarativewidgets-incorporation/jupyter-declarativewidgets-extension-incorporation.md: -------------------------------------------------------------------------------- 1 | # Jupyter DeclarativeWidgets Extension Incorporation 2 | 3 | ## Problem 4 | 5 | > "Data science enables the creation of data products." – Mike Loukides in [What is data science?](https://www.oreilly.com/ideas/what-is-data-science) 6 | 7 | Alice is a Jupyter Notebook user. Alice has performed some data analysis in a Notebook and wants to make here results available in the form of an interactive dashboard or application. She does not want to leave the Notebook environment in order to build a rich UI for her intended audience. The UI that may require input from the user, interaction with plots, and other forms of rendering results. Currently some of solutions available for Alice are IPywidgets, Bokeh, and Plotly. These solutions suffer from the following drawbacks: 8 | 9 | 1. Most solutions are Python specific. Other language kernels has either limiting or no available solution in place. 10 | 2. Most solutions would require considerable development investment to make available on other language kernels. Their implementation rely heavily on kernel side code. 11 | 3. Most solutions are bound by a close widget system and are cumbersome to extend. 12 | 4. Layers of abstraction make debugging difficult, requiring "peeking under the hood". 13 | 14 | Depending on what language Alice is using for her Notebook, she will experience the following potential setbacks: 15 | 16 | 1. Her language has very limited support for plotting and widget (i.e. Scala), so she cannot complete here deliverable in the Notebook. She needs to use other tools and potentially seek help due to skill gaps. 17 | 2. A particular widget is not available. Either Alice combines other packages into the Notebook and hope they interoperate, or she hits a wall due to here gap in knowledge of web technologies. 18 | 19 | Alice cannot simply do the following: 20 | * Rely on a solution that is portable across different popular Data Science languages. 21 | * Use a simple web native approach to declare widgets and connect then to the data in the kernel. 22 | * Build interactivity through simple data bindings 23 | * Easily tap into a growing ecosystem of web components 24 | 25 | ## Proposed Enhancement 26 | 27 | The incubating [declarative widgets extension for Jupyter Notebook](https://github.com/jupyter-incubator/declarativewidgets) (hereafter, *declarativewidgets extension*) is the incubation projects that addresses the issues above. The community should consider incorporating the declarativewidgets extension as an official Jupyter Subproject that provides another option for building rich interactive Notebooks. 28 | 29 | ## Detailed Explanation 30 | 31 | 32 | The declarativewidets extension focuses on enabling the user to create UI areas of the Notebook that can connect and interact with the code and data in the kernel. 33 | 34 | In conjunction with the (dashboard extension)(https://github.com/jupyter-incubator/dashboards) and related sub-projects, it is possible to turn Notebooks into full-featured dashboards and applications. 35 | 36 | ### Current and Potential Use Cases 37 | 38 | The declarativewidgets extension supports the following use cases: 39 | 40 | * Invoking a function defined on the Kernel and getting the return value (if available) 41 | * Querying the data off a DataFrame on the Kernel. The DataFrame can be a: 42 | * pandas 43 | * pyspark 44 | * Scala Spark 45 | * R 46 | * sparkR 47 | * Installing and importing a necessary web-component into the Notebook 48 | * Creating a UI using HTML markup and connecting the elements using [Polymer's data binding)[https://www.polymer-project.org/1.0/docs/devguide/data-binding]. 49 | * Reacting to changes in data on the Kernel, for example, new data arriving on a Stream. 50 | 51 | ![taxi_declarative](https://cloud.githubusercontent.com/assets/2474841/15164018/a4e871fe-16d1-11e6-9f03-66c96b745cc4.gif) 52 | 53 | ### Current Features 54 | 55 | The following are the most relevant features: 56 | 57 | 1. A set of declarative elements and corresponding kernel side code that enables connecting and interacting with functions, DataFrames and arbitrary data on the Kernel. 58 | 2. A set of declarative elements for visualizing data in a variety of chart types. 59 | 3. An extension to the `