├── jupyter_ai └── __init__.py ├── .yarnrc.yml ├── docs ├── source │ ├── _static │ │ ├── vllm-api.png │ │ ├── vllm-chat.png │ │ ├── ai-settings.png │ │ ├── sample-code.png │ │ ├── vllm-serve.png │ │ ├── fix-response.png │ │ ├── jupyter_logo.png │ │ ├── chat-ask-command.png │ │ ├── chat-hello-world.png │ │ ├── chat-learn-docs.png │ │ ├── ollama-settings.png │ │ ├── openai-chat-vllm.png │ │ ├── openrouter-chat.png │ │ ├── sample-html-math.png │ │ ├── sample-markdown.png │ │ ├── vllm-aisettings.png │ │ ├── chat-generate-input.png │ │ ├── chat-learn-delete.png │ │ ├── chat-select-model.png │ │ ├── openai-chat-openai.png │ │ ├── openai-embeddings.png │ │ ├── bedrock-custom-models.png │ │ ├── bedrock-model-access.png │ │ ├── bedrock-model-select.png │ │ ├── chat-getting-started.png │ │ ├── jupyter-ai-screenshot.png │ │ ├── openai-chat-deepseek.png │ │ ├── bedrock-chat-basemodel.png │ │ ├── bedrock-finetuned-model.png │ │ ├── chat-explain-code-output.png │ │ ├── chat-history-context-1.png │ │ ├── chat-history-context-2.png │ │ ├── chat-icon-left-tab-bar.png │ │ ├── chat-interface-selection.png │ │ ├── chat-sagemaker-endpoints.png │ │ ├── fix-error-cell-selected.png │ │ ├── openrouter-model-setup-2.png │ │ ├── openrouter-model-setup.png │ │ ├── bedrock-chat-basemodel-arn.png │ │ ├── chat-select-model-complete.png │ │ ├── fix-no-error-cell-selected.png │ │ ├── bedrock-chat-custom-model-arn.png │ │ ├── chat-icon-left-tab-bar-custom.png │ │ ├── chat-replace-selection-input.png │ │ ├── bedrock-chat-basemodel-modelid.png │ │ ├── bedrock-cross-region-inference.png │ │ ├── chat-generate-command-response.png │ │ ├── chat-settings-choose-language-model.png │ │ └── css │ │ │ └── custom.css │ ├── developers │ │ ├── index.md │ │ └── entry_points_api │ │ │ ├── index.md │ │ │ ├── providing_entry_points.md │ │ │ ├── embeddings_providers_group.md │ │ │ ├── personas_group.md │ │ │ └── model_providers_group.md │ ├── index.md │ ├── users │ │ ├── vllm.md │ │ ├── openrouter.md │ │ ├── bedrock.md │ │ └── index.md │ ├── conf.py │ └── contributors │ │ └── index.md ├── requirements.txt ├── Makefile └── make.bat ├── dev-environment.yml ├── playground ├── README.md └── config.example.py ├── .github ├── workflows │ ├── enforce-label.yml │ ├── check-release.yml │ ├── prep-release.yml │ └── publish-release.yml └── pull_request_template.md ├── .readthedocs.yaml ├── scripts ├── ci-setup.sh ├── dev-setup.sh └── bump-version.py ├── LICENSE ├── .pre-commit-config.yaml ├── pyproject.toml ├── conftest.py ├── .gitignore └── README.md /jupyter_ai/__init__.py: -------------------------------------------------------------------------------- 1 | __version__ = "3.0.0b9" 2 | -------------------------------------------------------------------------------- /.yarnrc.yml: -------------------------------------------------------------------------------- 1 | nodeLinker: "node-modules" 2 | enableImmutableInstalls: false 3 | -------------------------------------------------------------------------------- /docs/source/_static/vllm-api.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/vllm-api.png -------------------------------------------------------------------------------- /docs/source/_static/vllm-chat.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/vllm-chat.png -------------------------------------------------------------------------------- /docs/source/_static/ai-settings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/ai-settings.png -------------------------------------------------------------------------------- /docs/source/_static/sample-code.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/sample-code.png -------------------------------------------------------------------------------- /docs/source/_static/vllm-serve.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/vllm-serve.png -------------------------------------------------------------------------------- /docs/source/_static/fix-response.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/fix-response.png -------------------------------------------------------------------------------- /docs/source/_static/jupyter_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/jupyter_logo.png -------------------------------------------------------------------------------- /docs/source/_static/chat-ask-command.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-ask-command.png -------------------------------------------------------------------------------- /docs/source/_static/chat-hello-world.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-hello-world.png -------------------------------------------------------------------------------- /docs/source/_static/chat-learn-docs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-learn-docs.png -------------------------------------------------------------------------------- /docs/source/_static/ollama-settings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/ollama-settings.png -------------------------------------------------------------------------------- /docs/source/_static/openai-chat-vllm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/openai-chat-vllm.png -------------------------------------------------------------------------------- /docs/source/_static/openrouter-chat.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/openrouter-chat.png -------------------------------------------------------------------------------- /docs/source/_static/sample-html-math.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/sample-html-math.png -------------------------------------------------------------------------------- /docs/source/_static/sample-markdown.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/sample-markdown.png -------------------------------------------------------------------------------- /docs/source/_static/vllm-aisettings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/vllm-aisettings.png -------------------------------------------------------------------------------- /docs/source/_static/chat-generate-input.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-generate-input.png -------------------------------------------------------------------------------- /docs/source/_static/chat-learn-delete.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-learn-delete.png -------------------------------------------------------------------------------- /docs/source/_static/chat-select-model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-select-model.png -------------------------------------------------------------------------------- /docs/source/_static/openai-chat-openai.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/openai-chat-openai.png -------------------------------------------------------------------------------- /docs/source/_static/openai-embeddings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/openai-embeddings.png -------------------------------------------------------------------------------- /docs/source/_static/bedrock-custom-models.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/bedrock-custom-models.png -------------------------------------------------------------------------------- /docs/source/_static/bedrock-model-access.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/bedrock-model-access.png -------------------------------------------------------------------------------- /docs/source/_static/bedrock-model-select.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/bedrock-model-select.png -------------------------------------------------------------------------------- /docs/source/_static/chat-getting-started.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-getting-started.png -------------------------------------------------------------------------------- /docs/source/_static/jupyter-ai-screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/jupyter-ai-screenshot.png -------------------------------------------------------------------------------- /docs/source/_static/openai-chat-deepseek.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/openai-chat-deepseek.png -------------------------------------------------------------------------------- /docs/source/_static/bedrock-chat-basemodel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/bedrock-chat-basemodel.png -------------------------------------------------------------------------------- /docs/source/_static/bedrock-finetuned-model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/bedrock-finetuned-model.png -------------------------------------------------------------------------------- /docs/source/_static/chat-explain-code-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-explain-code-output.png -------------------------------------------------------------------------------- /docs/source/_static/chat-history-context-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-history-context-1.png -------------------------------------------------------------------------------- /docs/source/_static/chat-history-context-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-history-context-2.png -------------------------------------------------------------------------------- /docs/source/_static/chat-icon-left-tab-bar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-icon-left-tab-bar.png -------------------------------------------------------------------------------- /docs/source/_static/chat-interface-selection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-interface-selection.png -------------------------------------------------------------------------------- /docs/source/_static/chat-sagemaker-endpoints.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-sagemaker-endpoints.png -------------------------------------------------------------------------------- /docs/source/_static/fix-error-cell-selected.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/fix-error-cell-selected.png -------------------------------------------------------------------------------- /docs/source/_static/openrouter-model-setup-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/openrouter-model-setup-2.png -------------------------------------------------------------------------------- /docs/source/_static/openrouter-model-setup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/openrouter-model-setup.png -------------------------------------------------------------------------------- /docs/source/_static/bedrock-chat-basemodel-arn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/bedrock-chat-basemodel-arn.png -------------------------------------------------------------------------------- /docs/source/_static/chat-select-model-complete.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-select-model-complete.png -------------------------------------------------------------------------------- /docs/source/_static/fix-no-error-cell-selected.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/fix-no-error-cell-selected.png -------------------------------------------------------------------------------- /docs/source/_static/bedrock-chat-custom-model-arn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/bedrock-chat-custom-model-arn.png -------------------------------------------------------------------------------- /docs/source/_static/chat-icon-left-tab-bar-custom.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-icon-left-tab-bar-custom.png -------------------------------------------------------------------------------- /docs/source/_static/chat-replace-selection-input.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-replace-selection-input.png -------------------------------------------------------------------------------- /docs/requirements.txt: -------------------------------------------------------------------------------- 1 | # nbsphinx-link requires older version of docutils 2 | docutils==0.20 3 | myst_parser 4 | nbsphinx 5 | nbsphinx-link 6 | pydata_sphinx_theme 7 | -------------------------------------------------------------------------------- /docs/source/_static/bedrock-chat-basemodel-modelid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/bedrock-chat-basemodel-modelid.png -------------------------------------------------------------------------------- /docs/source/_static/bedrock-cross-region-inference.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/bedrock-cross-region-inference.png -------------------------------------------------------------------------------- /docs/source/_static/chat-generate-command-response.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-generate-command-response.png -------------------------------------------------------------------------------- /docs/source/_static/chat-settings-choose-language-model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyterlab/jupyter-ai/HEAD/docs/source/_static/chat-settings-choose-language-model.png -------------------------------------------------------------------------------- /dev-environment.yml: -------------------------------------------------------------------------------- 1 | name: jaidev 2 | channels: 3 | - conda-forge 4 | dependencies: 5 | - python=3.11 6 | - nodejs=20 7 | - jupyterlab>=4.4 8 | - uv 9 | - pip 10 | -------------------------------------------------------------------------------- /playground/README.md: -------------------------------------------------------------------------------- 1 | # jupyter-ai playground 2 | 3 | Contents root that should be served by JupyterLab when developing Jupyter AI. An 4 | example configuration is provided in `config.example.py`. 5 | -------------------------------------------------------------------------------- /docs/source/_static/css/custom.css: -------------------------------------------------------------------------------- 1 | img.screenshot { 2 | /* Copied from div.admonition */ 3 | box-shadow: 0 .2rem .5rem var(--pst-color-shadow),0 0 .0625rem var(--pst-color-shadow); 4 | border-color: var(--pst-color-info); 5 | } 6 | -------------------------------------------------------------------------------- /playground/config.example.py: -------------------------------------------------------------------------------- 1 | # JupyterLab configuration file for Jupyter AI local development 2 | # Reference: https://jupyter-ai.readthedocs.io/en/latest/users/index.html#configuring-with-openai 3 | 4 | # Specify full path to the notebook dir if running jupyter lab from 5 | # outside of the jupyter-ai project root directory 6 | c.ServerApp.root_dir = "./playground" 7 | -------------------------------------------------------------------------------- /.github/workflows/enforce-label.yml: -------------------------------------------------------------------------------- 1 | name: Enforce PR label 2 | 3 | on: 4 | pull_request: 5 | types: [labeled, unlabeled, opened, edited, synchronize] 6 | 7 | jobs: 8 | enforce-label: 9 | runs-on: ubuntu-latest 10 | permissions: 11 | pull-requests: write 12 | steps: 13 | - name: enforce-triage-label 14 | uses: jupyterlab/maintainer-tools/.github/actions/enforce-label@v1 15 | -------------------------------------------------------------------------------- /.readthedocs.yaml: -------------------------------------------------------------------------------- 1 | # .readthedocs.yaml 2 | # Read the Docs configuration file 3 | # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details 4 | 5 | version: 2 6 | 7 | build: 8 | os: ubuntu-22.04 9 | tools: 10 | python: "3.11" 11 | apt_packages: 12 | - pandoc 13 | 14 | sphinx: 15 | configuration: docs/source/conf.py 16 | 17 | python: 18 | install: 19 | - requirements: docs/requirements.txt 20 | -------------------------------------------------------------------------------- /scripts/ci-setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # Setup script used to initialize CI environments in GitHub workflows. 4 | # Not intended for human use. 5 | # 6 | # NOTE: this script requires the `astral-sh/setup-uv` GitHub action to run 7 | # before being called in a GitHub workflow. See this page for more guidance on 8 | # running `uv` inside a GitHub workflow: 9 | # https://docs.astral.sh/uv/guides/integration/github/ 10 | 11 | set -eux 12 | 13 | # Install JupyterLab 14 | uv pip install "jupyterlab>=4.4" 15 | 16 | # Build & install packages 17 | jlpm install 18 | jlpm build 19 | jlpm dev:install 20 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line, and also 5 | # from the environment for the first two. 6 | SPHINXOPTS ?= 7 | SPHINXBUILD ?= sphinx-build 8 | SOURCEDIR = source 9 | BUILDDIR = build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 21 | -------------------------------------------------------------------------------- /docs/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | pushd %~dp0 4 | 5 | REM Command file for Sphinx documentation 6 | 7 | if "%SPHINXBUILD%" == "" ( 8 | set SPHINXBUILD=sphinx-build 9 | ) 10 | set SOURCEDIR=source 11 | set BUILDDIR=build 12 | 13 | %SPHINXBUILD% >NUL 2>NUL 14 | if errorlevel 9009 ( 15 | echo. 16 | echo.The 'sphinx-build' command was not found. Make sure you have Sphinx 17 | echo.installed, then set the SPHINXBUILD environment variable to point 18 | echo.to the full path of the 'sphinx-build' executable. Alternatively you 19 | echo.may add the Sphinx directory to PATH. 20 | echo. 21 | echo.If you don't have Sphinx installed, grab it from 22 | echo.https://www.sphinx-doc.org/ 23 | exit /b 1 24 | ) 25 | 26 | if "%1" == "" goto help 27 | 28 | %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% 29 | goto end 30 | 31 | :help 32 | %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% 33 | 34 | :end 35 | popd 36 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | 6 | 7 | ## Description 8 | 9 | 10 | 11 | 12 | 13 | ## Code changes 14 | 15 | 16 | 17 | ## User-facing changes 18 | 19 | 20 | 21 | 22 | 23 | ## Backwards-incompatible changes 24 | 25 | 26 | -------------------------------------------------------------------------------- /docs/source/developers/index.md: -------------------------------------------------------------------------------- 1 | # Developers 2 | 3 | This section describes the **developer API** in Jupyter AI that other Python 4 | packages can use to tailor Jupyter AI to their use-case. Here are some examples 5 | of what other packages can do with the developer API: 6 | 7 | - Add custom AI personas to Jupyter AI 8 | 9 | - Add custom model providers to Jupyter AI 10 | 11 | The developer API allows other packages to modify both the frontend & the 12 | backend. The developer API has two parts: the **Entry points API** and the 13 | **Lumino plugin API** (currently unused, but planned for v3). 14 | 15 | - The Entry points API allows packages to add certain objects to Jupyter AI's 16 | backend. This is available to any Python package. 17 | 18 | - The Lumino plugin API allows packages to add to, modify, or even override 19 | parts of Jupyter AI's frontend. This is only available to labextension packages. 20 | 21 | ```{toctree} Table of Contents 22 | :depth: 3 23 | 24 | entry_points_api/index.md 25 | ``` 26 | -------------------------------------------------------------------------------- /.github/workflows/check-release.yml: -------------------------------------------------------------------------------- 1 | name: Check Release 2 | 3 | on: 4 | push: 5 | branches: ["*"] 6 | pull_request: 7 | branches: ["*"] 8 | schedule: 9 | - cron: "0 0 * * *" 10 | 11 | jobs: 12 | check_release: 13 | runs-on: ubuntu-latest 14 | steps: 15 | - name: Checkout code 16 | uses: actions/checkout@v4 17 | 18 | - name: Setup environment 19 | uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1 20 | with: 21 | python_version: "3.11.x" 22 | 23 | - name: Check Release 24 | uses: jupyter-server/jupyter_releaser/.github/actions/check-release@v2 25 | with: 26 | token: ${{ secrets.GITHUB_TOKEN }} 27 | version_spec: "12.34.56" 28 | 29 | - name: Upload Distributions 30 | uses: actions/upload-artifact@v4 31 | with: 32 | name: jupyter-ai-jupyter-releaser-dist-${{ github.run_number }} 33 | path: .jupyter_releaser_checkout/dist 34 | 35 | - name: Runner debug info 36 | if: always() 37 | run: | 38 | echo "Node version: $(node --version)" 39 | echo "NPM version: $(npm --version)" 40 | echo "jlpm version: $(jlpm --version)" 41 | echo "Yarn version: $(yarn --version)" 42 | echo "Python version: $(python --version)" 43 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2023, author_a 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /docs/source/developers/entry_points_api/index.md: -------------------------------------------------------------------------------- 1 | # Entry points API 2 | 3 | The [**entry points API**][entry_points] allows other packages to add custom 4 | objects to Jupyter AI's backend. Jupyter AI defines a set of **entry point 5 | groups**. Each entry point group is reserved for a specific type of object, e.g. 6 | one group may be reserved for personas and another may be reserved for model 7 | providers. 8 | 9 | A package can *provide* an **entry point** to an entry point group to add an 10 | object of the type reserved by the group. For example, providing an entry point 11 | to the personas group will add a new AI persona to Jupyter AI. A package can 12 | also provide multiple entry points to the same group. 13 | 14 | The entry points API currently includes the following entry point groups: 15 | 16 | - `'jupyter_ai.personas'`: allows developers to add AI personas. 17 | 18 | - `'jupyter_ai.model_providers'`: allows developers to add model providers. 19 | 20 | - `'jupyter_ai.embeddings_model_providers'`: allows developers to add embedding model providers. 21 | 22 | Each entry point group used by Jupyter AI is documented in a section below, 23 | which describes the type of object expected and how to define a custom 24 | implementation. 25 | 26 | At the end, we describe how to provide an entry point using a custom 27 | implementation in your package. 28 | 29 | ```{toctree} 30 | :caption: Contents 31 | 32 | personas_group.md 33 | model_providers_group.md 34 | embeddings_providers_group.md 35 | providing_entry_points.md 36 | ``` 37 | 38 | [entry_points]: https://setuptools.pypa.io/en/latest/userguide/entry_point.html 39 | -------------------------------------------------------------------------------- /docs/source/developers/entry_points_api/providing_entry_points.md: -------------------------------------------------------------------------------- 1 | # Providing entry points 2 | 3 | To provide entry points, your custom Python package must use a `pyproject.toml` 4 | manifest. To provide an existing class as an entry point, fill out the following 5 | template and add it to the end of your package's 6 | `pyproject.toml` file. 7 | 8 | ``` 9 | [project.entry-points.] 10 | = ":" 11 | ``` 12 | 13 | where: 14 | 15 | - `` takes the name of an entry point group used by Jupyter 16 | AI, e.g. `'jupyter_ai.personas'`. Make sure `` is surrounded 17 | by single or double quotes. 18 | 19 | - `` can be any string, as long as it is unique within each 20 | `[projects.entry-points.*]` table. 21 | 22 | - `` takes the path to your module containing the class. 23 | 24 | - For example, if you are providing a persona defined in 25 | `my_custom_package/personas/custom.py`, then this field should be set to 26 | `my_custom_package.personas.custom`. 27 | 28 | - `` takes the name of the class defined in ``. 29 | 30 | Finally, for Jupyter AI to read from your package's entry points, your package 31 | must be installed in the same Python environment. Entry points are only read by 32 | Jupyter AI when the server starts, so you should also restart JupyterLab after 33 | installing your custom package to see the new changes. 34 | 35 | :::{note} 36 | After adding/removing an entry point, you will also need to re-install the 37 | package & restart JupyterLab for the changes to take effect. 38 | ::: 39 | -------------------------------------------------------------------------------- /.github/workflows/prep-release.yml: -------------------------------------------------------------------------------- 1 | name: "Step 1: Prep Release" 2 | on: 3 | workflow_dispatch: 4 | inputs: 5 | version_spec: 6 | description: "New Version Specifier" 7 | default: "next" 8 | required: false 9 | branch: 10 | description: "The branch to target" 11 | required: false 12 | post_version_spec: 13 | description: "Post Version Specifier" 14 | required: false 15 | since: 16 | description: "Use PRs with activity since this date or git reference" 17 | required: false 18 | since_last_stable: 19 | description: "Use PRs with activity since the last stable git tag" 20 | required: false 21 | type: boolean 22 | jobs: 23 | prep_release: 24 | runs-on: ubuntu-latest 25 | steps: 26 | - uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1 27 | with: 28 | python_version: "3.11.x" 29 | - name: Prep Release 30 | id: prep-release 31 | uses: jupyter-server/jupyter_releaser/.github/actions/prep-release@v2 32 | with: 33 | token: ${{ secrets.ADMIN_GITHUB_TOKEN }} 34 | version_spec: ${{ github.event.inputs.version_spec }} 35 | post_version_spec: ${{ github.event.inputs.post_version_spec }} 36 | target: ${{ github.event.inputs.target }} 37 | branch: ${{ github.event.inputs.branch }} 38 | since: ${{ github.event.inputs.since }} 39 | since_last_stable: ${{ github.event.inputs.since_last_stable }} 40 | 41 | - name: "** Next Step **" 42 | run: | 43 | echo "Optional): Review Draft Release: ${{ steps.prep-release.outputs.release_url }}" 44 | -------------------------------------------------------------------------------- /docs/source/developers/entry_points_api/embeddings_providers_group.md: -------------------------------------------------------------------------------- 1 | # `'jupyter_ai.embeddings_model_providers'` 2 | 3 | ```{contents} Contents 4 | ``` 5 | 6 | ## Summary 7 | 8 | This entry point group allows packages to add custom embedding model providers 9 | to power the retrieval-augmented generation (RAG) capabilities of Jupyter AI. 10 | 11 | This group expects an **embedding model provider class**, a subclass of 12 | `BaseEmbeddingsProvider` from `jupyter-ai` that also inherits from an 13 | `Embeddings` class from LangChain. Instructions on defining one are given in the 14 | next section. 15 | 16 | :::{warning} 17 | This is a v2 extension point that may be removed in v3. In v3, we may explore 18 | updating the model API to make it easier for developers to add custom models. 19 | ::: 20 | 21 | ## How-to: Define a custom embedding model provider 22 | 23 | ```python 24 | from jupyter_ai_magics import BaseEmbeddingsProvider 25 | from langchain.embeddings import FakeEmbeddings 26 | 27 | class MyEmbeddingsProvider(BaseEmbeddingsProvider, FakeEmbeddings): 28 | id = "my_embeddings_provider" 29 | name = "My Embeddings Provider" 30 | model_id_key = "model" 31 | models = ["my_model"] 32 | 33 | def __init__(self, **kwargs): 34 | super().__init__(size=300, **kwargs) 35 | ``` 36 | 37 | Jupyter AI uses entry points to discover embedding providers. 38 | In the `pyproject.toml` file, add your custom embedding provider to the 39 | `[project.entry-points."jupyter_ai.embeddings_model_providers"]` section: 40 | 41 | ```toml 42 | [project.entry-points."jupyter_ai.embeddings_model_providers"] 43 | my-provider = "my_provider:MyEmbeddingsProvider" 44 | ``` 45 | 46 | [Embeddings]: https://api.python.langchain.com/en/stable/embeddings/langchain_core.embeddings.Embeddings.html 47 | -------------------------------------------------------------------------------- /docs/source/index.md: -------------------------------------------------------------------------------- 1 | # Jupyter AI 2 | 3 | Welcome to Jupyter AI, which brings generative AI to Jupyter. Jupyter AI provides a user-friendly 4 | and powerful way to explore generative AI models in notebooks and improve your productivity 5 | in JupyterLab and the Jupyter Notebook. More specifically, Jupyter AI offers: 6 | 7 | * An `%%ai` magic that turns the Jupyter notebook into a reproducible generative AI playground. 8 | This works anywhere the IPython kernel runs (JupyterLab, Jupyter Notebook, Google Colab, VSCode, etc.). 9 | * A native chat UI in JupyterLab that enables you to work with generative AI as a conversational assistant. 10 | * Support for a wide range of generative model providers and models 11 | (AI21, Anthropic, Cohere, Gemini, Hugging Face, MistralAI, OpenAI, SageMaker, NVIDIA, etc.). 12 | 13 | A screenshot of Jupyter AI showing the chat interface and the magic commands 16 | 17 | ## JupyterLab support 18 | 19 | **Each major version of Jupyter AI supports *only one* major version of JupyterLab.** Jupyter AI 1.x supports 20 | JupyterLab 3.x, and Jupyter AI 2.x supports JupyterLab 4.x. The feature sets of versions 1.0.0 and 2.0.0 21 | are the same. We will maintain support for JupyterLab 3 for as long as it remains maintained. 22 | 23 | The `main` branch of Jupyter AI targets the newest supported major version of JupyterLab. All new features and most bug fixes will be 24 | committed to this branch. Features and bug fixes will be backported 25 | to work on JupyterLab 3 only if developers determine that they will add sufficient value. 26 | **We recommend that JupyterLab users who want the most advanced Jupyter AI functionality upgrade to JupyterLab 4.** 27 | 28 | ## Contents 29 | 30 | ```{toctree} 31 | --- 32 | maxdepth: 2 33 | --- 34 | 35 | users/index 36 | contributors/index 37 | developers/index 38 | examples/index 39 | ``` 40 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - repo: https://github.com/pre-commit/pre-commit-hooks 3 | rev: v5.0.0 4 | hooks: 5 | - id: end-of-file-fixer 6 | - id: check-case-conflict 7 | - id: check-executables-have-shebangs 8 | - id: requirements-txt-fixer 9 | - id: check-added-large-files 10 | - id: check-case-conflict 11 | - id: check-toml 12 | exclude: ^packages/jupyter-ai-module-cookiecutter 13 | - id: check-yaml 14 | exclude: ^packages/jupyter-ai-module-cookiecutter 15 | - id: debug-statements 16 | - id: forbid-new-submodules 17 | - id: check-builtin-literals 18 | - id: trailing-whitespace 19 | 20 | - repo: https://github.com/PyCQA/autoflake 21 | rev: v2.3.1 22 | hooks: 23 | - id: autoflake 24 | 25 | - repo: https://github.com/psf/black 26 | rev: 25.1.0 27 | hooks: 28 | - id: black 29 | 30 | - repo: https://github.com/PyCQA/isort 31 | rev: 6.0.1 32 | hooks: 33 | - id: isort 34 | args: ["--profile", "black"] 35 | files: \.py$ 36 | 37 | - repo: https://github.com/asottile/pyupgrade 38 | rev: v3.20.0 39 | hooks: 40 | - id: pyupgrade 41 | args: [--py39-plus] 42 | 43 | - repo: https://github.com/pycqa/flake8 44 | rev: 7.3.0 45 | hooks: 46 | - id: flake8 47 | additional_dependencies: 48 | [ 49 | "flake8-bugbear==20.1.4", 50 | "flake8-logging-format==0.6.0", 51 | "flake8-implicit-str-concat==0.2.0", 52 | ] 53 | stages: [manual] 54 | 55 | - repo: https://github.com/sirosen/check-jsonschema 56 | rev: 0.33.1 57 | hooks: 58 | - id: check-jsonschema 59 | name: "Check GitHub Workflows" 60 | files: ^\.github/workflows/ 61 | types: [yaml] 62 | args: ["--schemafile", "https://json.schemastore.org/github-workflow"] 63 | stages: [manual] 64 | -------------------------------------------------------------------------------- /docs/source/users/vllm.md: -------------------------------------------------------------------------------- 1 | # Using vLLM in Jupyter AI 2 | 3 | [(Return to the Chat Interface page)](index.md#vllm-usage) 4 | 5 | `vLLM` is a fast and easy-to-use library for LLM inference and serving. The [vLLM website](https://docs.vllm.ai/en/latest/) explains installation and usage. 6 | 7 | :::{note} 8 | To use `vLLM` via `OpenRouter` as described below you will need to upgrade to `jupyter-ai >= 2.29.1`. 9 | ::: 10 | 11 | 12 | Depending on your hardware set up you will install `vLLM` using these [instructions](https://docs.vllm.ai/en/latest/getting_started/installation/index.html). It is best to install it in a dedicated python environment. 13 | 14 | Once it is installed you may start serving any model with the command: 15 | ```python 16 | vllm serve 17 | ``` 18 | As an example, the deployment of the `Phi-3-mini-4k-instruct` model is shown below, with checks to make sure it is up and running: 19 | 20 | Screen shot of steps and checks in deploying a model using vllm. 23 | 24 | `vllm` serves up the model at the following URL: `http://:8000/v1` 25 | 26 | Start up Jupyter AI and update the AI Settings as follows (notice that we are using [OpenRouter](openrouter.md) as the provider, which is a unified interface for LLMs based on OpenAI's API interface): 27 | 28 | Screen shot of AI setting for using vllm. 31 | 32 | Since vLLM may be addressed using OpenAI's API, you can test if the model is available using the API call as shown: 33 | 34 | Screen shot of using vllm programmatically with its API. 37 | 38 | The model may be used in Jupyter AI's chat interface as shown in the example below: 39 | 40 | Screen shot of using vllm in Jupyter AI chat. 43 | 44 | [(Return to the Chat Interface page)](index.md#vllm-usage) 45 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["hatchling>=1.4.0"] 3 | build-backend = "hatchling.build" 4 | 5 | [project] 6 | name = "jupyter_ai" 7 | readme = "README.md" 8 | license = { file = "LICENSE" } 9 | requires-python = ">=3.9" 10 | description = "A set of extensions providing agentic AI in JupyterLab" 11 | authors = [{ name = "Project Jupyter", email = "jupyter@googlegroups.com" }] 12 | keywords = ["jupyter", "jupyterlab", "jupyterlab-extension", "jupyter-ai"] 13 | classifiers = [ 14 | "Framework :: Jupyter", 15 | "Framework :: Jupyter :: JupyterLab", 16 | "Framework :: Jupyter :: JupyterLab :: 4", 17 | "Framework :: Jupyter :: JupyterLab :: Extensions", 18 | "Framework :: Jupyter :: JupyterLab :: Extensions :: Prebuilt", 19 | "License :: OSI Approved :: BSD License", 20 | "Programming Language :: Python", 21 | "Programming Language :: Python :: 3", 22 | "Programming Language :: Python :: 3.9", 23 | "Programming Language :: Python :: 3.10", 24 | "Programming Language :: Python :: 3.11", 25 | "Programming Language :: Python :: 3.12", 26 | ] 27 | dynamic = ["version"] 28 | 29 | dependencies = [ 30 | "jupyterlab_chat>=0.19.0a0", 31 | "jupyter_server_documents>=0.1.0a8", 32 | "jupyter_ai_router>=0.0.2", 33 | "jupyter_ai_persona_manager>=0.0.4", 34 | "jupyter_ai_litellm>=0.0.1", 35 | "jupyter_ai_magic_commands>=0.0.2", 36 | "jupyter_ai_chat_commands>=0.0.4", 37 | "jupyter_ai_jupyternaut>=0.0.5", 38 | ] 39 | 40 | [project.optional-dependencies] 41 | test = [ 42 | # `jupyter_collaboration>=4` requires `jupyter_server_ydoc>=2.0.0`, 43 | # which requires `jupyter_server>=2.15.0`. 44 | "jupyter_server[test]>=2.15.0,<3", 45 | "coverage", 46 | "pytest", 47 | "pytest-asyncio", 48 | "pytest-cov", 49 | "pytest-tornasync", 50 | "pytest-jupyter", 51 | "syrupy~=4.0.8", 52 | "types-jsonschema", 53 | "mypy", 54 | ] 55 | 56 | [project.urls] 57 | Documentation = "https://jupyter-ai.readthedocs.io/en/v3/" 58 | Source = "https://github.com/jupyterlab/jupyter-ai" 59 | Tracker = "https://github.com/jupyterlab/jupyter-ai/issues" 60 | 61 | [tool.hatch.version] 62 | path = "jupyter_ai/__init__.py" 63 | -------------------------------------------------------------------------------- /.github/workflows/publish-release.yml: -------------------------------------------------------------------------------- 1 | name: "Step 2: Publish Release" 2 | on: 3 | workflow_dispatch: 4 | inputs: 5 | branch: 6 | description: "The target branch" 7 | required: false 8 | release_url: 9 | description: "The URL of the draft GitHub release" 10 | required: false 11 | steps_to_skip: 12 | description: "Comma separated list of steps to skip" 13 | required: false 14 | 15 | jobs: 16 | publish_release: 17 | runs-on: ubuntu-latest 18 | steps: 19 | - uses: jupyterlab/maintainer-tools/.github/actions/base-setup@v1 20 | with: 21 | python_version: "3.11.x" 22 | - name: Populate Release 23 | id: populate-release 24 | uses: jupyter-server/jupyter_releaser/.github/actions/populate-release@v2 25 | with: 26 | token: ${{ secrets.ADMIN_GITHUB_TOKEN }} 27 | target: ${{ github.event.inputs.target }} 28 | branch: ${{ github.event.inputs.branch }} 29 | release_url: ${{ github.event.inputs.release_url }} 30 | steps_to_skip: ${{ github.event.inputs.steps_to_skip }} 31 | 32 | - name: Finalize Release 33 | id: finalize-release 34 | env: 35 | PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }} 36 | PYPI_TOKEN_MAP: ${{ secrets.PYPI_TOKEN_MAP }} 37 | TWINE_USERNAME: __token__ 38 | NPM_TOKEN: ${{ secrets.NPM_TOKEN }} 39 | uses: jupyter-server/jupyter-releaser/.github/actions/finalize-release@v2 40 | with: 41 | token: ${{ secrets.ADMIN_GITHUB_TOKEN }} 42 | target: ${{ github.event.inputs.target }} 43 | release_url: ${{ steps.populate-release.outputs.release_url }} 44 | 45 | - name: "** Next Step **" 46 | if: ${{ success() }} 47 | run: | 48 | echo "Verify the final release" 49 | echo ${{ steps.finalize-release.outputs.release_url }} 50 | - name: "** Failure Message **" 51 | if: ${{ failure() }} 52 | run: | 53 | echo "Failed to Publish the Draft Release Url:" 54 | echo ${{ steps.populate-release.outputs.release_url }} 55 | -------------------------------------------------------------------------------- /docs/source/conf.py: -------------------------------------------------------------------------------- 1 | # Configuration file for the Sphinx documentation builder. 2 | # 3 | # For the full list of built-in configuration values, see the documentation: 4 | # https://www.sphinx-doc.org/en/master/usage/configuration.html 5 | 6 | # -- Project information ----------------------------------------------------- 7 | # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information 8 | 9 | import time 10 | 11 | project = "Jupyter AI" 12 | copyright = f"2023–{time.localtime().tm_year}, Project Jupyter" 13 | author = "Project Jupyter" 14 | 15 | # -- General configuration --------------------------------------------------- 16 | # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration 17 | 18 | extensions = ["myst_parser", "nbsphinx", "nbsphinx_link"] 19 | myst_enable_extensions = ["colon_fence"] 20 | 21 | templates_path = ["_templates"] 22 | exclude_patterns = [] 23 | 24 | # -- Options for HTML output ------------------------------------------------- 25 | # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output 26 | 27 | html_theme = "pydata_sphinx_theme" 28 | html_static_path = ["_static"] 29 | 30 | html_css_files = [ 31 | "css/custom.css", 32 | ] 33 | 34 | # -- Jupyter theming ------------------------------------------------- 35 | html_logo = "_static/jupyter_logo.png" 36 | 37 | html_theme_options = { 38 | "logo": { 39 | "text": "Jupyter AI", 40 | }, 41 | "icon_links": [ 42 | { 43 | # Label for this link 44 | "name": "GitHub", 45 | # URL where the link will redirect 46 | "url": "https://github.com/jupyterlab/jupyter-ai", # required 47 | # Icon class (if "type": "fontawesome"), or path to local image (if "type": "local") 48 | "icon": "fab fa-github-square", 49 | # The type of image to be used (see below for details) 50 | "type": "fontawesome", 51 | }, 52 | { 53 | "name": "jupyter.org", 54 | "url": "https://jupyter.org", 55 | "icon": "_static/jupyter_logo.png", 56 | "type": "local", 57 | }, 58 | ], 59 | } 60 | 61 | html_sidebars = {"**": []} 62 | -------------------------------------------------------------------------------- /conftest.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | import asyncio 3 | from pathlib import Path 4 | import pytest 5 | from traitlets.config import Config, LoggingConfigurable 6 | import logging 7 | from jupyter_server.services.contents.filemanager import AsyncFileContentsManager 8 | from typing import TYPE_CHECKING 9 | 10 | if TYPE_CHECKING: 11 | from jupyter_server.serverapp import ServerApp 12 | 13 | pytest_plugins = ("jupyter_server.pytest_plugin",) 14 | 15 | @pytest.fixture 16 | def jp_server_config(jp_server_config, tmp_path): 17 | return Config({"ServerApp": {"jpserver_extensions": {"jupyter_ai": True}}, "ContentsManager": {"root_dir": str(tmp_path)}}) 18 | 19 | 20 | @pytest.fixture(scope="session") 21 | def static_test_files_dir() -> Path: 22 | return ( 23 | Path(__file__).parent.resolve() 24 | / "packages" 25 | / "jupyter-ai" 26 | / "jupyter_ai" 27 | / "tests" 28 | / "static" 29 | ) 30 | 31 | class MockAiExtension(LoggingConfigurable): 32 | """Mock AiExtension class for testing purposes.""" 33 | 34 | serverapp: ServerApp 35 | 36 | def __init__(self, *args, serverapp: ServerApp, **kwargs): 37 | super().__init__(*args, **kwargs) 38 | self.serverapp = serverapp 39 | self._log = None 40 | 41 | @property 42 | def log(self) -> logging.Logger: 43 | return self.serverapp.log 44 | 45 | @property 46 | def event_loop(self) -> asyncio.AbstractEventLoop: 47 | return self.serverapp.io_loop.asyncio_loop 48 | 49 | @property 50 | def contents_manager(self) -> AsyncFileContentsManager: 51 | return self.serverapp.contents_manager 52 | 53 | 54 | @pytest.fixture 55 | def mock_ai_extension(jp_server_config, jp_configurable_serverapp) -> MockAiExtension: 56 | """ 57 | Returns a mocked `AiExtension` object that can be passed as the `parent` 58 | argument to objects normally initialized by `AiExtension`. This should be 59 | passed to most of the "manager singletons" like `ConfigManager`, 60 | `PersonaManager`, and `EnvSecretsManager`. 61 | 62 | See `MockAiExtension` in `conftest.py` for a complete description of the 63 | attributes, properties, and methods available. If something is missing, 64 | please feel free to add to it in your PR. 65 | 66 | Returns: 67 | A `MockAiExtension` instance that can be passed as the `parent` argument 68 | to objects normally initialized by `AiExtension`. 69 | """ 70 | serverapp = jp_configurable_serverapp() 71 | return MockAiExtension(config=jp_server_config, serverapp=serverapp) 72 | 73 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.bundle.* 2 | lib/ 3 | node_modules/ 4 | *.log 5 | .eslintcache 6 | .stylelintcache 7 | *.egg-info/ 8 | .ipynb_checkpoints 9 | *.tsbuildinfo 10 | 11 | # Integration tests 12 | ui-tests/test-results/ 13 | ui-tests/playwright-report/ 14 | 15 | # Created by https://www.gitignore.io/api/python 16 | # Edit at https://www.gitignore.io/?templates=python 17 | 18 | ### Python ### 19 | # Byte-compiled / optimized / DLL files 20 | __pycache__/ 21 | *.py[cod] 22 | *$py.class 23 | 24 | # C extensions 25 | *.so 26 | 27 | # Distribution / packaging 28 | .Python 29 | build/ 30 | develop-eggs/ 31 | dist/ 32 | downloads/ 33 | eggs/ 34 | .eggs/ 35 | lib/ 36 | lib64/ 37 | parts/ 38 | sdist/ 39 | var/ 40 | wheels/ 41 | pip-wheel-metadata/ 42 | share/python-wheels/ 43 | .installed.cfg 44 | *.egg 45 | MANIFEST 46 | 47 | # PyInstaller 48 | # Usually these files are written by a python script from a template 49 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 50 | *.manifest 51 | *.spec 52 | 53 | # Installer logs 54 | pip-log.txt 55 | pip-delete-this-directory.txt 56 | 57 | # Unit test / coverage reports 58 | htmlcov/ 59 | .tox/ 60 | .nox/ 61 | .coverage 62 | .coverage.* 63 | .cache 64 | nosetests.xml 65 | coverage/ 66 | coverage.xml 67 | *.cover 68 | .hypothesis/ 69 | .pytest_cache/ 70 | 71 | # Translations 72 | *.mo 73 | *.pot 74 | 75 | # Scrapy stuff: 76 | .scrapy 77 | 78 | # Sphinx documentation 79 | docs/_build/ 80 | 81 | # PyBuilder 82 | target/ 83 | 84 | # pyenv 85 | .python-version 86 | 87 | # celery beat schedule file 88 | celerybeat-schedule 89 | 90 | # SageMath parsed files 91 | *.sage.py 92 | 93 | # Spyder project settings 94 | .spyderproject 95 | .spyproject 96 | 97 | # Rope project settings 98 | .ropeproject 99 | 100 | # Mr Developer 101 | .mr.developer.cfg 102 | .project 103 | .pydevproject 104 | 105 | # mkdocs documentation 106 | /site 107 | 108 | # mypy 109 | .mypy_cache/ 110 | .dmypy.json 111 | dmypy.json 112 | 113 | # Pyre type checker 114 | .pyre/ 115 | 116 | # End of https://www.gitignore.io/api/python 117 | 118 | # OSX files 119 | .DS_Store 120 | 121 | # playground folder 122 | playground/ 123 | 124 | # jupyter releaser checkout 125 | .jupyter_releaser_checkout 126 | 127 | # reserve path for a dev script 128 | dev.sh 129 | 130 | .vscode 131 | 132 | .jupyter_ystore.db 133 | 134 | .yarn 135 | 136 | .conda/ 137 | 138 | # Version files are auto-generated by Hatchling and should not be committed to 139 | # the source repo. 140 | jupyter_ai/_version.py 141 | 142 | 143 | # Ignore local chat files & local .jupyter/ dir 144 | *.chat 145 | .jupyter/ 146 | 147 | # Ignore secrets in '.env' 148 | .env 149 | -------------------------------------------------------------------------------- /scripts/dev-setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # Development setup script for contributors. 4 | 5 | set -eu 6 | 7 | # Detect available Python environment manager (micromamba > mamba > conda) 8 | # command -v checks if a command exists in PATH and is executable (POSIX compliant) 9 | ENV_MANAGER="" 10 | if command -v micromamba >/dev/null 2>&1; then 11 | ENV_MANAGER="micromamba" 12 | elif command -v mamba >/dev/null 2>&1; then 13 | ENV_MANAGER="mamba" 14 | elif command -v conda >/dev/null 2>&1; then 15 | ENV_MANAGER="conda" 16 | else 17 | echo "Error: No Python environment manager found!" 18 | echo "Please install one of the following: micromamba, mamba, or conda" 19 | echo " - micromamba: https://micromamba.readthedocs.io/en/latest/installation.html" 20 | echo " - mamba: https://mamba.readthedocs.io/en/latest/installation.html" 21 | echo " - conda: https://docs.conda.io/en/latest/miniconda.html" 22 | exit 1 23 | fi 24 | echo "Using environment manager: $ENV_MANAGER" 25 | 26 | # Create `jaidev` environment if it does not exist 27 | if ! $ENV_MANAGER env list | grep -q "jaidev"; then 28 | echo "No Jupyter AI development environment named 'jaidev' found." 29 | echo "Creating 'jaidev' environment..." 30 | $ENV_MANAGER env create -f dev-environment.yml -y 31 | exit_code=$? 32 | if [ $exit_code -ne 0 ]; then 33 | echo "Error: Failed to create 'jaidev' environment." 34 | echo "Please report this issue to the maintainers on GitHub." 35 | exit $exit_code 36 | fi 37 | fi 38 | 39 | # Run environment manager shell hook. This is required for 40 | # `activate`/`deactivate` commands to work in scripts 41 | if [ "$ENV_MANAGER" = "conda" ]; then 42 | eval "$(conda shell.bash hook)" 43 | else 44 | # for mamba or micromamba 45 | eval "$($ENV_MANAGER shell hook --shell bash)" 46 | fi 47 | exit_code=$? 48 | if [ $exit_code -ne 0 ]; then 49 | echo "Error: Failed to run $ENV_MANAGER shell hook." 50 | echo "Please report this issue to the maintainers on GitHub." 51 | exit $exit_code 52 | fi 53 | 54 | # Activate `jaidev` environment 55 | echo "Activating Jupyter AI development environment 'jaidev'..." 56 | $ENV_MANAGER activate jaidev 57 | exit_code=$? 58 | if [ $exit_code -ne 0 ]; then 59 | echo "Error: Failed to activate 'jaidev' environment." 60 | echo "Please report this issue to the maintainers on GitHub." 61 | exit $exit_code 62 | fi 63 | 64 | # Install JS dependencies 65 | echo "Installing NPM dependencies..." 66 | jlpm install 67 | 68 | # Build JS assets 69 | echo "Building JavaScript assets..." 70 | jlpm build 71 | 72 | # Perform editable installation of `jupyter-ai` and `jupyter-ai-magics` locally 73 | echo "Running editable installation..." 74 | jlpm dev:install 75 | 76 | # Install documentation packages 77 | echo "Installing documentation build requirements..." 78 | uv pip install sphinx 79 | uv pip install -r docs/requirements.txt 80 | 81 | # Copy example config to `playground/` dir 82 | cp playground/config.example.py playground/config.py 83 | 84 | echo "The Jupyter AI development environment 'jaidev' is ready!" 85 | echo "Run 'jupyter lab' to start the JupyterLab server and test your development build of Jupyter AI." 86 | echo "Remember to run '{conda,mamba,micromamba} activate jaidev' each time you begin development from a new terminal." 87 | echo "See the contributor documentation for more help: https://jupyter-ai.readthedocs.io/en/latest/contributors/index.html" 88 | -------------------------------------------------------------------------------- /docs/source/users/openrouter.md: -------------------------------------------------------------------------------- 1 | # Using OpenRouter or OpenAI Interfaces in Jupyter AI 2 | 3 | [(Return to the Chat Interface page)](index.md#openrouter-usage) 4 | 5 | For models that are compatible with the OpenAI library, Jupyter AI provides configuration via OpenRouter. By supporting the configuration of parameters such as the api_key, base_url, and model, various large model services compatible with the OpenAI library call methods can be used. For more details on OpenRouter as a unified interface for LLMs, see https://openrouter.ai/. 6 | 7 | As an example, we walk through the steps needed to use models from [Deepseek](https://www.deepseek.com) via the OpenRouter provider. If you do not have `langchain-openai` installed, please install it and restart JupyterLab. This is necessary as it provides the SDK for accessing any OpenAI API. 8 | 9 | First, navigate to the `AI Settings` pane via the AI settings button in `v2` or via the dropdown in `v3` of Jupyter AI, as shown below: 10 | 11 | Screenshot of the dropdown where AI Settings is chosen and it opens tab in Jupyter AI where models are selected. 15 | 16 | Second, select the `OpenRouter :: *` model provider in the Jupyter AI settings. If you don't see this, please verify that you have installed `langchain-openai` and that you are using `jupyter_ai>=2.24.0`. Be sure to restart JupyterLab after upgrading or installing either package. 17 | 18 | Jupyter AI's settings page with the OpenRouter provider selected is shown below: 19 | 20 | Screenshot of the tab in Jupyter AI where OpenRouter model access is selected. 24 | 25 | Type in the model name and the API base URL corresponding to the model you wish to use. For Deepseek, you should use `https://api.deepseek.com` as the API base URL, and use `deepseek-chat` as the local model ID. 26 | 27 | If you have an OpenRouter account and wish to use their API and URL, it is also possible using the OpenRouter provider in Jupyter AI, as shown here: 28 | 29 | Screenshot of the tab in Jupyter AI where OpenRouter model access is selected with its own API and URL. 33 | 34 | If you are using OpenRouter for the first time it will also require entering the `OPENROUTER_API_KEY`. If you have used OpenRouter before with a different model provider, you will need to update the API key. After doing this, click "Save Changes" at the bottom to save your settings. 35 | 36 | You should now be able to use Deepseek! An example of usage is shown next: 37 | 38 | Screenshot of chat using Deepseek via the OpenRouter provider. 42 | 43 | In a similar manner, models may also be invoked directly using the OpenAI provider interface in Jupyter AI. First, you can choose the OpenAI provider and then enter in the model ID, as shown on the OpenAI [models page](https://platform.openai.com/docs/models). An example is shown below: 44 | 45 | Screenshot of chat using gpt-4o via the OpenAI provider. 49 | 50 | DeepSeek models may be used via the same interface, if the base API url is provided: 51 | 52 | Screenshot of chat using deepseek via the OpenAI provider. 56 | 57 | For DeepSeek models, enter the DeepSeek API for the OpenAI API key. 58 | 59 | Models deployed using vLLM may be used in a similar manner: 60 | 61 | Screenshot of chat using vllm via the OpenAI provider. 65 | 66 | Usage of models using vLLM and their deployment is discussed [here](vllm.md). 67 | 68 | For embedding models from OpenAI, you can generically choose them using the AI Settings interface as well: 69 | 70 | Screenshot of embedding use via the OpenAI provider. 74 | 75 | [(Return to the Chat Interface page)](index.md#openrouter-usage) 76 | -------------------------------------------------------------------------------- /scripts/bump-version.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) Jupyter Development Team. 2 | # Distributed under the terms of the Modified BSD License. 3 | 4 | """ 5 | The "bump version" script used by Jupyter Releaser to increment the version of 6 | Jupyter AI on manual release workflow runs. This script: 7 | 8 | - Accepts a *specified version* `spec_version` as the first positional argument. 9 | `spec_version` must be either a PEP-440 version string or "minor" literally. 10 | 11 | - Normal release version examples: "0.1.0", "1.2.3", "3.0.0" 12 | 13 | - Pre-release version examples: "3.0.0a0", "3.0.0b1", "3.0.0rc2" 14 | 15 | - **NOTE**: This script was not designed to support dev & post-releases for 16 | simplicity. By convention, our repo prefers patch releases over 17 | post-releases and pre-releases over dev releases. 18 | 19 | - Bumps `jupyter-ai` and `jupyter-ai-magics` to `spec_version`. If 20 | `spec_version` is "minor", then this script bumps the minor version of each 21 | package. 22 | 23 | - Updates `jupyter-ai`'s required version of `jupyter-ai-magics` to exactly 24 | match the specified version. In other words, this script ensures 25 | `jupyter-ai==x.y.z` always depends on `jupyter-ai-magics==x.y.z` exactly. 26 | 27 | - If `--skip-if-dirty` is passed, successive calls do nothing. This is a 28 | temporary workaround for 29 | https://github.com/jupyter-server/jupyter_releaser/issues/567. 30 | """ 31 | 32 | from pathlib import Path 33 | 34 | import click 35 | import tomlkit 36 | from jupyter_releaser.util import get_version, run 37 | from packaging.version import Version 38 | from pkg_resources import parse_version 39 | 40 | MONOREPO_ROOT = Path(__file__).parent.parent.resolve() 41 | LERNA_CMD = "npx -p lerna@6.4.1 -y lerna version --no-push --force-publish --no-git-tag-version -y" 42 | 43 | 44 | @click.command() 45 | @click.option("--ignore-dirty", default=False, is_flag=True) 46 | @click.option("--skip-if-dirty", default=False, is_flag=True) 47 | @click.argument("spec", nargs=1) 48 | def bump_version(ignore_dirty: bool, skip_if_dirty: bool, spec: str): 49 | is_dirty = len(run("git status --porcelain").strip()) > 0 50 | if is_dirty and not ignore_dirty: 51 | if skip_if_dirty: 52 | print( 53 | "Skipping this call as the repo is in a dirty state with untracked files." 54 | ) 55 | return 56 | raise Exception("Must be in a clean git state with no untracked files") 57 | 58 | next_version: Version = compute_next_version(spec) 59 | 60 | # convert the PyPI version string to a NPM version string 61 | next_version_npm = f"{next_version.major}.{next_version.minor}.{next_version.micro}" 62 | if next_version.pre: 63 | pre_type, pre_number = next_version.pre 64 | if pre_type == "a": 65 | pre_type = "alpha" 66 | elif pre_type == "b": 67 | pre_type = "beta" 68 | elif pre_type == "rc": 69 | pre_type = "rc" 70 | else: 71 | raise Exception(f"Unrecognized pre-release type: '{pre_type}'.") 72 | next_version_npm += f"-{pre_type}.{pre_number}" 73 | 74 | # bump the versions in NPM packages 75 | # 76 | # Note: `_version.py` files do not need to be updated manually. 77 | # `hatch-nodejs-version` updates those files automatically on build, setting 78 | # them to the version specified in the corresponding package.json file. 79 | lerna_cmd = f"{LERNA_CMD} {next_version_npm}" 80 | run(lerna_cmd) 81 | 82 | # bump the version of `jupyter-ai-magics` required by `jupyter-ai` 83 | jai_pyproject_path = MONOREPO_ROOT / "packages" / "jupyter-ai" / "pyproject.toml" 84 | jai_pyproject = tomlkit.parse(jai_pyproject_path.read_text()) 85 | jai_deps = jai_pyproject.get("project").get("dependencies") 86 | for i, dep in enumerate(jai_deps): 87 | if str(dep).startswith("jupyter_ai_magics"): 88 | next_major_version = f"{next_version.major + 1}.0.0" 89 | jai_deps[i] = ( 90 | f"jupyter_ai_magics>={str(next_version)},<{next_major_version}" 91 | ) 92 | break 93 | 94 | # write updated pyproject.toml file 95 | jai_pyproject_path.write_text(tomlkit.dumps(jai_pyproject)) 96 | 97 | 98 | def compute_next_version(spec: str) -> Version: 99 | if spec == "minor": 100 | curr_version = parse_version(get_version()) 101 | next_version = parse_version( 102 | f"{curr_version.major}.{curr_version.minor + 1}.{curr_version.micro}" 103 | ) 104 | else: 105 | next_version = parse_version(spec) 106 | 107 | return next_version 108 | 109 | 110 | if __name__ == "__main__": 111 | bump_version() 112 | -------------------------------------------------------------------------------- /docs/source/users/bedrock.md: -------------------------------------------------------------------------------- 1 | # Using Amazon Bedrock with Jupyter AI 2 | 3 | [(Return to the Chat Interface page)](index.md#amazon-bedrock-usage) 4 | 5 | Bedrock supports many language model providers such as AI21 Labs, Amazon, Anthropic, Cohere, Meta, and Mistral AI. To use the base models from any supported provider make sure to enable them in Amazon Bedrock by using the AWS console. You should also select embedding models in Bedrock in addition to language completion models if you intend to use retrieval augmented generation (RAG) on your documents. 6 | 7 | Go to Amazon Bedrock and select `Model Access` as shown here: 8 | 9 | Screenshot of the left panel in the AWS console where Bedrock model access is provided. 13 | 14 | Click through on `Model Access` and follow the instructions to grant access to the models you wish to use, as shown below. Make sure to accept the end user license (EULA) as required by each model. You may need your system administrator to grant access to your account if you do not have authority to do so. 15 | 16 | Screenshot of the Bedrock console where models may be selected. 20 | 21 | You should also select embedding models in addition to language completion models if you intend to use retrieval augmented generation (RAG) on your documents. 22 | 23 | You may now select a chosen Bedrock model from the drop-down menu box title `Completion model` in the chat interface. If RAG is going to be used then pick an embedding model that you chose from the Bedrock models as well. An example of these selections is shown below: 24 | 25 | Screenshot of the Jupyter AI chat panel where the base language model and embedding model is selected. 29 | 30 | If your provider requires an API key, please enter it in the box that will show for that provider. Make sure to click on `Save Changes` to ensure that the inputs have been saved. 31 | 32 | Bedrock also allows custom models to be trained from scratch or fine-tuned from a base model. Jupyter AI enables a custom model to be called in the chat panel using its `arn` (Amazon Resource Name). A fine-tuned model will have your 12-digit customer number in the ARN: 33 | 34 | Screenshot of the Jupyter AI chat panel where the custom model is selected using model arn. 38 | 39 | As with custom models, you can also call a base model by its `model id` or its `arn`. An example of using a base model with its `model id` through the custom model interface is shown below: 40 | 41 | Screenshot of the Jupyter AI chat panel where the base model is selected using model id. 45 | 46 | An example of using a base model using its `arn` through the custom model interface is shown below: 47 | 48 | Screenshot of the Jupyter AI chat panel where the base model is selected using its ARN. 52 | 53 | ## Fine-tuning in Bedrock 54 | 55 | To train a custom model in Amazon Bedrock, select `Custom models` in the Bedrock console as shown below, and then you may customize a base model by fine-tuning it or continuing to pre-train it: 56 | 57 | Screenshot of the Bedrock custom models access in the left panel of the Bedrock console. 61 | 62 | For details on fine-tuning a base model from Bedrock, see this [reference](https://aws.amazon.com/blogs/aws/customize-models-in-amazon-bedrock-with-your-own-data-using-fine-tuning-and-continued-pre-training/); with related [documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html). 63 | 64 | Once the model is fine-tuned, it will have its own `arn`, as shown below: 65 | 66 | Screenshot of the Bedrock fine-tuned model ARN in the Bedrock console. 70 | 71 | As seen above, you may click on `Purchase provisioned throughput` to buy inference units with which to call the custom model's API. Enter the model's `arn` in Jupyter AI's Language model user interface to use the provisioned model. 72 | 73 | ## Cross-Region Inference 74 | 75 | Amazon Bedrock now permits cross-region inference, where a model hosted in a different region than that of the user may be specified, see the [inference profiles documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/inference-profiles-support.html). Such models have IDs prefixed by a region identifier such as `us.meta.llama3-2-1b-instruct-v1:0`, for example. To use this feature, simply enter the Inference profile ID for the cross-region model instead of the ARN. 76 | 77 | Screenshot of Bedrock cross-region inference usage. 81 | 82 | ## Summary 83 | 84 | 1. Bedrock Base models: All available models will already be available in the drop down model list. The above interface also allows use of base model IDs or ARNs, though this is unnecessary as they are in the dropdown list. 85 | 2. Bedrock Custom models: If you have fine-tuned a Bedrock base model you may use the ARN for this custom model. Make sure to enter the correct provider information, such as `amazon`, `anthropic`, `cohere`, `meta`, `mistral` (always in lower case). 86 | 3. Provisioned Models: These are models that run on dedicated endpoints. Users can purchase Provisioned Throughput Model Units to get faster throughput. These may be base or custom models. Enter the ARN for these models in the Model ID field. 87 | 4. Cross-region Inference: Use the Inference profile ID for the cross-region model instead of the ARN. 88 | 89 | [(Return to the Chat Interface page)](index.md#amazon-bedrock-usage) 90 | -------------------------------------------------------------------------------- /docs/source/developers/entry_points_api/personas_group.md: -------------------------------------------------------------------------------- 1 | # `'jupyter_ai.personas'` 2 | 3 | ```{contents} Contents 4 | ``` 5 | 6 | ## Summary 7 | 8 | This entry point group allows packages to add custom AI personas to the chat. 9 | AI personas are analogous to "bots" in other chat applications. Every available 10 | persona will be available in every chat by default. If a chat has any other 11 | users besides the current user and a single AI persona, then AI personas will 12 | only respond when `@`-mentioned. Jupyternaut provides a single AI persona by 13 | default: `Jupyternaut`. 14 | 15 | For example, if your Jupyter AI instance has `Jupyternaut` and `MyCustomPersona` 16 | as 2 AI personas, then each persona will only respond when `@`-mentioned. 17 | 18 | - To call `Jupyternaut`, your message must include `@Jupyternaut`. 19 | 20 | - To call `MyCustomPersona`, your message must include `@MyCustomPersona`. 21 | 22 | - Multiple personas may be mentioned in a single message. Each mentioned persona 23 | will reply to the new message, allowing you to compare performance across AI 24 | personas. 25 | 26 | This group expects a **persona class**, a subclass of 27 | `jupyter_ai.personas:BasePersona`. Instructions on defining one are given in the 28 | next section. 29 | 30 | ## How-to: Define a custom AI persona 31 | 32 | Defining a custom AI persona is simple. One needs to define a new class that 33 | inherits from `BasePersona` and implements its two abstract methods. 34 | 35 | The two abstract methods required are the `defaults` property and the 36 | `process_message()` method. 37 | 38 | - `defaults` defines the default settings of the persona. This is mainly used to 39 | control the name and avatar shown in the Jupyter AI chat. 40 | 41 | - `process_message()` defines how your persona responds to new messages. 42 | 43 | We will dive into the `process_message()` method first, then discuss how to 44 | define the `defaults` property. 45 | 46 | ### Defining how an AI persona processes messages 47 | 48 | The `process_message()` method takes the signature: 49 | 50 | ```py 51 | @abstractmethod 52 | async def process_message(self, message: Message) -> None: 53 | """ 54 | Processes a new message. This method exclusively defines how new 55 | messages are handled by a persona, and should be considered the "main 56 | entry point" to this persona. Reading chat history and streaming a reply 57 | can be done through method calls to `self.ychat`. See 58 | `JupyternautPersona` for a reference implementation on how to do so. 59 | 60 | This is an abstract method that must be implemented by subclasses. 61 | """ 62 | ``` 63 | 64 | This method accepts a `Message` object that represents a new message from a 65 | human user. There are many attributes on this object, but the main one is 66 | `message.body`, which contains the content of the message as a string. 67 | A full reference can be found in `jupyterlab_chat.models:Message`. 68 | 69 | Subclasses may use the built-in methods on `BasePersona` to respond to the user. 70 | The two main methods are: 71 | 72 | - `send_message(body: str)`: Accepts a string and replies immediately. 73 | 74 | - `stream_message(stream: AsyncIterator[str])`: Accepts an async interator that 75 | yields string chunks, and streams the response to the chat. The output of the 76 | `astream()` method on LangChain models can be passed to this method directly. 77 | 78 | ### Using Jupyternaut's configured chat model 79 | 80 | The personas feature gives developers total freedom in how their persona 81 | responds to new messages. Personas do not need to use the same chat model as 82 | Jupyternaut, and can use any library of their choice, provided it is installed 83 | in the same environment. 84 | 85 | However, if your persona wants to use the same configured LangChain model used 86 | by Jupyternaut, you can access that through the `self.config: ConfigManager` 87 | attribute available to subclasses. 88 | 89 | Add and call this method on a persona to access the LangChain runnable used by 90 | Jupyternaut: 91 | 92 | ```py 93 | def build_runnable(self) -> Any: 94 | llm = self.config.lm_provider(**self.config.lm_provider_params) 95 | runnable = JUPYTERNAUT_PROMPT_TEMPLATE | llm | StrOutputParser() 96 | 97 | runnable = RunnableWithMessageHistory( 98 | runnable=runnable, # type:ignore[arg-type] 99 | get_session_history=lambda: YChatHistory(ychat=self.ychat, k=2), 100 | input_messages_key="input", 101 | history_messages_key="history", 102 | ) 103 | 104 | return runnable 105 | ``` 106 | 107 | See `jupyter_ai.personas.jupyternaut` for a complete reference. 108 | 109 | ### Defining AI persona defaults 110 | 111 | The `defaults` property takes the signature: 112 | 113 | ```py 114 | @property 115 | @abstractmethod 116 | def defaults(self) -> PersonaDefaults: 117 | """ 118 | Returns a `PersonaDefaults` data model that represents the default 119 | settings of this persona. 120 | 121 | This is an abstract method that must be implemented by subclasses. 122 | """ 123 | ``` 124 | 125 | This property should return a `PersonaDefaults` instance: 126 | 127 | ```py 128 | class PersonaDefaults(BaseModel): 129 | name: str # e.g. "Jupyternaut" 130 | description: str # e.g. "..." 131 | avatar_path: str # e.g. /avatars/jupyternaut.svg 132 | system_prompt: str # e.g. "You are a language model named..." 133 | ``` 134 | 135 | - The `name` field determines the name of the AI persona shown in chat. 136 | 137 | - The `description` field is currently reserved but unused. 138 | 139 | - The `avatar_path` field takes the URL path to an image served by the Jupyter 140 | Server, relative to the server's domain. This can be used to show custom avatars 141 | for your AI persona. 142 | 143 | - We may change this to pass base64-encoded images instead of URL paths in 144 | the future. 145 | 146 | - The `system_prompt` field is currently reserved but unused. 147 | 148 | ## Reference implementation 149 | 150 | This code defines a custom AI persona with the name `DebugPersona`, that always 151 | replies with `'Hello!'`. 152 | 153 | ```py 154 | class DebugPersona(BasePersona): 155 | """ 156 | The Jupyternaut persona, the main persona provided by Jupyter AI. 157 | """ 158 | 159 | def __init__(self, *args, **kwargs): 160 | super().__init__(*args, **kwargs) 161 | 162 | @property 163 | def defaults(self): 164 | return PersonaDefaults( 165 | name="DebugPersona", 166 | avatar_path="/api/ai/static/jupyternaut.svg", 167 | description="A mock persona used for debugging in local dev environments.", 168 | system_prompt="...", 169 | ) 170 | 171 | async def process_message(self, message: Message): 172 | self.send_message(NewMessage(body="Hello!", sender=self.id)) 173 | return 174 | ``` 175 | 176 | :::{note} 177 | To make a custom AI persona available to Jupyter AI, you must provide it as an 178 | entry point in your package. See the documentation on 179 | to learn more. 180 | ::: 181 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Jupyter AI 2 | 3 | **Jupyter AI is under incubation as part of the JupyterLab organization.** 4 | 5 | Jupyter AI connects generative AI with Jupyter notebooks. Jupyter AI provides a user-friendly 6 | and powerful way to explore generative AI models in notebooks and improve your productivity 7 | in JupyterLab and the Jupyter Notebook. More specifically, Jupyter AI offers: 8 | 9 | * An `%%ai` magic that turns the Jupyter notebook into a reproducible generative AI playground. 10 | This works anywhere the IPython kernel runs (JupyterLab, Jupyter Notebook, Google Colab, Kaggle, VSCode, etc.). 11 | * A native chat UI in JupyterLab that enables you to work with generative AI as a conversational assistant. 12 | * Support for a wide range of generative model providers, including AI21, Anthropic, AWS, Cohere, 13 | Gemini, Hugging Face, MistralAI, NVIDIA, and OpenAI. 14 | * Local model support through GPT4All and Ollama, enabling use of generative AI models on consumer grade machines 15 | with ease and privacy. 16 | 17 | Documentation is available on [ReadTheDocs](https://jupyter-ai.readthedocs.io/en/latest/). 18 | 19 | ![A screenshot of Jupyter AI showing the chat interface and the magic commands](docs/source/_static/jupyter-ai-screenshot.png) 20 | 21 | ## Requirements 22 | 23 | You will need to have installed the following software to use Jupyter AI: 24 | 25 | - Python 3.9 - 3.12 26 | - JupyterLab 4 or Notebook 7 27 | 28 | In addition, you will need access to at least one model provider. 29 | 30 | > [!IMPORTANT] 31 | > JupyterLab 3 reached its end of maintenance date on May 15, 2024. As a result, we will not backport new features to the v1 branch supporting JupyterLab 3. Fixes for critical issues will still be backported until December 31, 2024. If you are still using JupyterLab 3, we strongly encourage you to **upgrade to JupyterLab 4 as soon as possible**. For more information, see [JupyterLab 3 end of maintenance](https://blog.jupyter.org/jupyterlab-3-end-of-maintenance-879778927db2) on the Jupyter Blog. 32 | 33 | ## Setting Up Model Providers in a Notebook 34 | 35 | To use any AI model provider within this notebook, you'll need the appropriate credentials, such as API keys. 36 | 37 | Obtain the necessary credentials, such as API keys, from your model provider's platform. 38 | 39 | You can set your keys in a code cell in your notebook or using environment variables. 40 | In a code cell, you can set the credentials as follows without revealing your key in the notebook: 41 | 42 | ```python 43 | # NOTE: Replace 'PROVIDER_API_KEY' with the credential key's name, 44 | # and enter the API key when prompted by using the code shown below. 45 | 46 | import getpass 47 | 48 | # Enter your key 49 | key = getpass.getpass('Enter your PROVIDER API key: ') 50 | 51 | # Set the environment variable without displaying the full key 52 | os.environ['PROVIDER_API_KEY'] = key 53 | ``` 54 | 55 | :::{note} 56 | :name: using-env-key 57 | You may also set these keys directly using the `%env` magic command, but the key value may be echoed in the cell output. If you prefer to use `%env`, be sure to not share the notebook with people you don't trust, as this may leak your API keys. 58 | ``` 59 | %env PROVIDER_API_KEY=YOUR_API_KEY_HERE 60 | ``` 61 | ::: 62 | 63 | For more specific instructions for each model provider, refer to [the model providers documentation](https://jupyter-ai.readthedocs.io/en/latest/users/index.html#model-providers). 64 | 65 | ## Installation 66 | 67 | Below is a simplified overview of the installation and usage process. 68 | See [our official documentation](https://jupyter-ai.readthedocs.io/en/latest/users/index.html) 69 | for details on installing and using Jupyter AI. 70 | 71 | We offer 3 different ways to install Jupyter AI. You can read through each 72 | section to pick the installation method that works best for you. 73 | 74 | 1. Quick installation via `pip` (recommended) 75 | 2. Minimal installation via `pip` 76 | 3. Minimal installation via `conda` 77 | 78 | ### Quick installation via `pip` (recommended) 79 | 80 | If you want to install both the `%%ai` magic and the JupyterLab extension, you can run: 81 | 82 | $ pip install jupyter-ai[all] 83 | 84 | Then, restart JupyterLab. This will install every optional dependency, which 85 | provides access to all models currently supported by `jupyter-ai`. 86 | 87 | If you are not using JupyterLab and you only want to install the Jupyter AI 88 | `%%ai` magic, you can run: 89 | 90 | $ pip install jupyter-ai-magics[all] 91 | 92 | `jupyter-ai` depends on `jupyter-ai-magics`, so installing `jupyter-ai` 93 | automatically installs `jupyter-ai-magics`. 94 | 95 | ### Minimal installation via `pip` 96 | 97 | Most model providers in Jupyter AI require a specific dependency to be installed 98 | before they are available for use. These are called _provider dependencies_. 99 | Provider dependencies are optional to Jupyter AI, meaning that Jupyter AI can be 100 | installed with or without any provider dependencies installed. If a provider 101 | requires a dependency that is not installed, its models are not listed in the 102 | user interface which allows you to select a language model. 103 | 104 | To perform a minimal installation via `pip` without any provider dependencies, 105 | omit the `[all]` optional dependency group from the package name: 106 | 107 | ``` 108 | pip install jupyter-ai 109 | ``` 110 | 111 | By selectively installing provider dependencies, you can control which models 112 | are available in your Jupyter AI environment. 113 | 114 | For example, to install Jupyter AI with only added support for Anthropic models, run: 115 | 116 | ``` 117 | pip install jupyter-ai langchain-anthropic 118 | ``` 119 | 120 | For more information on model providers and which dependencies they require, see 121 | [the model provider table](https://jupyter-ai.readthedocs.io/en/latest/users/index.html#model-providers). 122 | 123 | ### Minimal installation via `conda` 124 | 125 | As an alternative to using `pip`, you can install `jupyter-ai` using 126 | [Conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html) 127 | from the `conda-forge` channel: 128 | 129 | $ conda install conda-forge::jupyter-ai 130 | 131 | Most model providers in Jupyter AI require a specific _provider dependency_ to 132 | be installed before they are available for use. Provider dependencies are 133 | not installed when installing `jupyter-ai` from Conda Forge, and should be 134 | installed separately as needed. 135 | 136 | For example, to install Jupyter AI with only added support for OpenAI models, run: 137 | 138 | ``` 139 | conda install conda-forge::jupyter-ai conda-forge::langchain-openai 140 | ``` 141 | 142 | For more information on model providers and which dependencies they require, see 143 | [the model provider table](https://jupyter-ai.readthedocs.io/en/latest/users/index.html#model-providers). 144 | 145 | ## The `%%ai` magic command 146 | 147 | The `%%ai` magic works anywhere the IPython kernel runs, including JupyterLab, Jupyter Notebook, Google Colab, and Visual Studio Code. 148 | 149 | Once you have installed the `%%ai` magic, you can enable it in any notebook or the IPython shell by running: 150 | 151 | %load_ext jupyter_ai_magics 152 | 153 | or: 154 | 155 | %load_ext jupyter_ai 156 | 157 | The screenshots below are from notebooks in the `examples/` directory of this package. 158 | 159 | Then, you can use the `%%ai` magic command to specify a model and natural language prompt: 160 | 161 | ![Sample with code generation](./docs/source/_static/sample-code.png) 162 | 163 | Jupyter AI can also generate HTML and math to be rendered as cell output. 164 | 165 | ![Sample with HTML and math generation](./docs/source/_static/sample-html-math.png) 166 | 167 | Jupyter AI can interpolate IPython expressions, allowing you to run prompts 168 | that include variable values. 169 | 170 | ![Sample with code interpolation and markdown output](./docs/source/_static/sample-markdown.png) 171 | 172 | ## JupyterLab extension 173 | 174 | The Jupyter AI extension for JupyterLab offers a native UI that enables multiple users 175 | to chat with the Jupyter AI conversational assistant. If you have JupyterLab installed, 176 | this should be installed and activated when you install the `jupyter_ai` package. 177 | 178 | ## Using 179 | 180 | For help with installing and using Jupyter AI, please see our 181 | [user documentation on ReadTheDocs](https://jupyter-ai.readthedocs.io/en/latest/users/index.html). 182 | 183 | ## Contributing 184 | 185 | If you would like to contribute to Jupyter AI, see our 186 | [contributor documentation on ReadTheDocs](https://jupyter-ai.readthedocs.io/en/latest/contributors/index.html). 187 | -------------------------------------------------------------------------------- /docs/source/contributors/index.md: -------------------------------------------------------------------------------- 1 | # Contributors 2 | 3 | This page is intended for people interested in building new or modified functionality for Jupyter AI. 4 | 5 | If you would like to build applications that enhance Jupyter AI, please see the {doc}`developer's guide `. 6 | 7 | ## Design principles 8 | 9 | Maintainers of Jupyter AI have adopted principles that contributors should also follow. These principles, which build on top of [the Zen of Python](https://peps.python.org/pep-0020/), are intended to earn users' trust by keeping their data under their control. The following list is non-exhaustive; maintainers have discretion to interpret and revise these principles. 10 | 11 | 1. Jupyter AI is **vendor-agnostic**. Jupyter AI does not discriminate between available models, and gives users a choice of model providers. A feature in Jupyter AI may be specific to one model or model provider if it cannot be used with other models or providers. 12 | 2. Jupyter AI **only responds to an explicit prompt**; it does not watch files and it does not send prompts automatically. Any change that watches user files must be opt-in only. 13 | 3. Jupyter AI is **transparent** with its chat prompts. The chat interface and magic commands use system messages and prompt templates that are open source, so that users know what gets sent to language models. 14 | 4. Jupyter AI is **traceable**; users know when it has been used to generate content. When Jupyter AI generates a notebook, the notebook says that it was generated by Jupyter AI. When a user runs a Jupyter AI magic command in a notebook, output cells say, in their metadata, that they were generated by Jupyter AI. 15 | 5. Jupyter AI uses a **human-centered design**. The chat interface should look and feel like chat applications that are generally available. The magic commands should look and work like other IPython magic commands. Settings screens should be used minimally, and wherever they are used, they should be readable and understandable, even for users not fluent in the user interface language. 16 | 17 | Issues and pull requests that violate the above principles may be declined. If you are unsure about whether your idea is a good fit for Jupyter AI, please [open an issue](https://github.com/jupyterlab/jupyter-ai/issues/new/choose) so that our maintainers can discuss it with you. 18 | 19 | ## Prerequisites 20 | 21 | You can develop Jupyter AI on any system that can run a supported Python version up to and including 3.12, including recent Windows, macOS, and Linux versions. 22 | 23 | You should have the newest supported version of JupyterLab installed. 24 | 25 | You will need [a supported version of node.js](https://github.com/nodejs/release#release-schedule) to use Jupyter AI. 26 | 27 | ## Automatic development setup (recommended) 28 | 29 | To get a development setup automatically, you must first install a Python 30 | environment manager. The development setup script currently supports 31 | `micromamba`, `mamba`, or `conda`. 32 | 33 | We recommend 34 | [micromamba](https://micromamba.readthedocs.io/en/latest/installation.html) for 35 | its speed and minimal footprint. Alternatively, you can install 36 | [mamba](https://mamba.readthedocs.io/en/latest/installation.html) or 37 | [conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html). 38 | 39 | Then, clone the repository and run the development setup script: 40 | 41 | ``` 42 | # Clone the repository 43 | git clone https://github.com/jupyterlab/jupyter-ai.git 44 | cd jupyter-ai 45 | 46 | # Run the development setup script 47 | ./scripts/dev-setup.sh 48 | ``` 49 | 50 | This script will automatically create and set up a Jupyter AI development 51 | environment named `jaidev` for you, using the environment manager available on 52 | your system. Specifically, this script will: 53 | 54 | - Detect your Python environment manager (micromamba, mamba, or conda), 55 | - Create a 'jaidev' Python environment with Python 3.11, Node.js 20, JupyterLab 4.4+, and `uv`, 56 | - Install all JavaScript dependencies, 57 | - Builds all frontend assets, 58 | - Perform editable installation of all Jupyter AI packages, and 59 | - Install documentation dependencies. 60 | 61 | After this is complete, you can start and launch JupyterLab in your default 62 | browser by running `jlpm dev` or simply `jupyter lab`. 63 | 64 | ## Manual development setup 65 | 66 | To perform a minimal development setup in another Python environment without 67 | using `dev-setup.sh`, first verify that the below prerequisites are installed in 68 | your environment: 69 | 70 | - Python 3.9 - 3.11 71 | 72 | - JupyterLab >=4.4 73 | 74 | - Any [maintained](https://github.com/nodejs/release#release-schedule) version of `nodejs` 75 | 76 | - [`uv`](https://docs.astral.sh/uv/) 77 | 78 | Then, run these commands: 79 | 80 | ```py 81 | # Install JS dependencies 82 | jlpm 83 | 84 | # Build frontend assets 85 | jlpm build 86 | 87 | # Perform editable installation of all Jupyter AI packages 88 | # This can also be done via the alias `jlpm di` 89 | jlpm dev:install 90 | ``` 91 | 92 | ## Development builds 93 | 94 | You can open a new terminal and use that to build local changes to the 95 | repository. Make sure to activate the `jaidev` environment each time you begin 96 | development from a new terminal: 97 | 98 | ``` 99 | {micromamba,mamba,conda} activate jaidev 100 | 101 | # Build frontend assets 102 | jlpm build 103 | ``` 104 | 105 | `jlpm build` only needs to be run after modifying anything related to the 106 | frontend (e.g. `*.ts`, `*.tsx`, `*.css` files). The browser has to be refreshed 107 | to reflect new changes to the frontend. 108 | 109 | To only build a subset of Jupyter AI packages, use the `--scope` argument that 110 | gets forwarded to Lerna: 111 | 112 | ``` 113 | # Builds frontend assets only for `packages/jupyter-ai` and its dependencies 114 | jlpm build --scope "@jupyter-ai/core" 115 | ``` 116 | 117 | Note that after making changes to anything backend-related (e.g. `*.py` files), 118 | you must restart the server for those changes to take effect. You can restart 119 | the server by first typing `Ctrl + C` in the terminal running it, then running 120 | `jlpm dev` or `jupyter lab` to restart it. 121 | 122 | ## Development reinstall 123 | 124 | To reinstall all Jupyter AI packages, run: 125 | 126 | ``` 127 | # reinstalls all Jupyter AI packages 128 | jlpm dev:reinstall 129 | 130 | # or, use the convenient alias: 131 | jlpm dr 132 | ``` 133 | 134 | This is generally necessary when updating `pyproject.toml` files. 135 | 136 | ## Building documentation 137 | 138 | The `./scripts/dev-setup.sh` should automatically install the documentation 139 | dependencies. You will need to install [pandoc](https://pandoc.org/) as well. You can install [pandoc from the conda-forge channel](https://anaconda.org/conda-forge/pandoc): 140 | 141 | ``` 142 | conda install -c conda-forge pandoc 143 | ``` 144 | 145 | Otherwise have a look at pandoc's [installation instructions](https://pandoc.org/installing.html). 146 | 147 | 148 | To build the documentation locally, run 149 | 150 | ``` 151 | cd docs/ 152 | make html 153 | ``` 154 | 155 | and open `file:///docs/build/html/index.html`, where 156 | `` is the absolute path of the Jupyter AI monorepo on 157 | your local filesystem. It is helpful to bookmark this path in your browser of 158 | choice to easily view your local documentation build. 159 | 160 | After making any changes, make sure to rebuild the documentation locally via 161 | `make html`, and then refresh your browser to verify the changes visually. 162 | 163 | 164 | ## Development uninstall 165 | 166 | To uninstall your Jupyter AI development environment, deactivate and remove the Conda environment: 167 | 168 | ``` 169 | conda deactivate 170 | conda env remove -n jupyter-ai 171 | ``` 172 | 173 | ## Testing 174 | 175 | ### Integration / E2E tests 176 | 177 | This extension uses Playwright for the integration / E2E tests (user-level tests). 178 | More precisely, the JupyterLab helper 179 | [Galata](https://github.com/jupyterlab/jupyterlab/tree/master/galata) is used to 180 | test the extension in JupyterLab. 181 | 182 | Install test dependencies (needed only once): 183 | 184 | ```sh 185 | cd ./packages/jupyter-ai/ui-tests/ 186 | jlpm install 187 | jlpm playwright install 188 | ``` 189 | 190 | Tests involve snapshot comparisons against a reference snapshots generated by the Github CI. If you are using an OS other than Linux, you will need to generate local snapshots before running the tests locally for the first time. To do this, execute the command: 191 | 192 | ```sh 193 | cd ./packages/jupyter-ai/ui-tests/ 194 | jlpm test:update 195 | ``` 196 | 197 | To execute tests, run: 198 | 199 | ```sh 200 | cd ./packages/jupyter-ai/ui-tests/ 201 | jlpm test 202 | ``` 203 | 204 | You can find more information in the 205 | [ui-tests](https://github.com/jupyterlab/jupyter-ai/tree/main/packages/jupyter-ai/ui-tests) 206 | README. 207 | -------------------------------------------------------------------------------- /docs/source/developers/entry_points_api/model_providers_group.md: -------------------------------------------------------------------------------- 1 | # `'jupyter_ai.model_providers'` 2 | 3 | ```{contents} Contents 4 | ``` 5 | 6 | ## Summary 7 | 8 | This entry point group allows packages to add custom model providers. 9 | 10 | This group expects a **model provider class**, a subclass of `BaseProvider` that 11 | also inherits from a `LLM` or `ChatModel` class from LangChain. Instructions on 12 | defining one are given in the next section. 13 | 14 | :::{warning} 15 | This is a v2 extension point that may be removed in v3. In v3, we may explore 16 | updating the model API to make it easier for developers to add custom models. 17 | ::: 18 | 19 | ## How-to: Define a custom model provider 20 | 21 | You can define new providers using the LangChain framework API. Custom providers 22 | inherit from both `jupyter-ai`'s `BaseProvider` and `langchain`'s [`LLM`][LLM]. 23 | You can either import a pre-defined model from [LangChain LLM list][langchain_llms], 24 | or define a [custom LLM][custom_llm]. 25 | In the example below, we define a provider with two models using 26 | a dummy `FakeListLLM` model, which returns responses from the `responses` 27 | keyword argument. 28 | 29 | ```python 30 | # my_package/my_provider.py 31 | from jupyter_ai_magics import BaseProvider 32 | from langchain_community.llms import FakeListLLM 33 | 34 | 35 | class MyProvider(BaseProvider, FakeListLLM): 36 | id = "my_provider" 37 | name = "My Provider" 38 | model_id_key = "model" 39 | models = [ 40 | "model_a", 41 | "model_b" 42 | ] 43 | def __init__(self, **kwargs): 44 | model = kwargs.get("model_id") 45 | kwargs["responses"] = ( 46 | ["This is a response from model 'a'"] 47 | if model == "model_a" else 48 | ["This is a response from model 'b'"] 49 | ) 50 | super().__init__(**kwargs) 51 | ``` 52 | 53 | 54 | If the new provider inherits from [`BaseChatModel`][BaseChatModel], it will be available 55 | both in the chat UI and with magic commands. Otherwise, users can only use the new provider 56 | with magic commands. 57 | 58 | To make the new provider available, you need to declare it as an [entry point][entry_points]: 59 | 60 | ```toml 61 | # my_package/pyproject.toml 62 | [project] 63 | name = "my_package" 64 | version = "0.0.1" 65 | 66 | [project.entry-points."jupyter_ai.model_providers"] 67 | my-provider = "my_provider:MyProvider" 68 | ``` 69 | 70 | To test that the above minimal provider package works, install it with: 71 | 72 | ```sh 73 | # from `my_package` directory 74 | pip install -e . 75 | ``` 76 | 77 | Then, restart JupyterLab. You should now see an info message in the log that mentions 78 | your new provider's `id`: 79 | 80 | ``` 81 | [I 2023-10-29 13:56:16.915 AiExtension] Registered model provider `my_provider`. 82 | ``` 83 | 84 | ### Adding custom API keys and fields 85 | 86 | You can add handle authentication via API keys, and configuration with 87 | custom parameters using an auth strategy and fields as shown in the example 88 | below. 89 | 90 | ```python 91 | from typing import ClassVar, List 92 | from jupyter_ai_magics import BaseProvider 93 | from jupyter_ai_magics.providers import EnvAuthStrategy, Field, TextField, MultilineTextField 94 | from langchain_community.llms import FakeListLLM 95 | 96 | 97 | class MyProvider(BaseProvider, FakeListLLM): 98 | id = "my_provider" 99 | name = "My Provider" 100 | model_id_key = "model" 101 | models = [ 102 | "model_a", 103 | "model_b" 104 | ] 105 | 106 | auth_strategy = EnvAuthStrategy( 107 | name="MY_API_KEY", keyword_param="my_api_key_param" 108 | ) 109 | 110 | fields: ClassVar[List[Field]] = [ 111 | TextField(key="my_llm_parameter", label="The name for my_llm_parameter to show in the UI"), 112 | MultilineTextField(key="custom_config", label="Custom Json Config", format="json"), 113 | ] 114 | 115 | def __init__(self, **kwargs): 116 | model = kwargs.get("model_id") 117 | kwargs["responses"] = ( 118 | ["This is a response from model 'a'"] 119 | if model == "model_a" else 120 | ["This is a response from model 'b'"] 121 | ) 122 | super().__init__(**kwargs) 123 | ``` 124 | 125 | The `auth_strategy` handles specifying API keys for providers and models. 126 | The example shows the `EnvAuthStrategy` which takes the API key from the 127 | environment variable with the name specified in `name` and be provided to the 128 | model's `__init__` as a kwarg with the name specified in `keyword_param`. 129 | This will also cause a field to be present in the configuration UI with the 130 | `name` of the environment variable as the label. 131 | 132 | Further configuration can be handled adding `fields` into the settings 133 | dialogue for your custom model by specifying a list of fields as shown in 134 | the example. These will be passed into the `__init__` as kwargs, with the 135 | key specified by the key in the field object. The label specified in the field 136 | object determines the text shown in the configuration section of the user 137 | interface. 138 | 139 | ## How-to: Define custom code completions 140 | 141 | Any model provider derived from `BaseProvider` can be used as a completion provider. 142 | However, some providers may benefit from customizing handling of completion requests. 143 | 144 | There are two asynchronous methods which can be overridden in subclasses of `BaseProvider`: 145 | 146 | - `generate_inline_completions`: takes a request (`InlineCompletionRequest`) and 147 | returns `InlineCompletionReply` 148 | 149 | - `stream_inline_completions`: takes a request and yields an initiating reply 150 | (`InlineCompletionReply`) with `isIncomplete` set to `True` followed by 151 | subsequent chunks (`InlineCompletionStreamChunk`) 152 | 153 | When streaming all replies and chunks for given invocation of the 154 | `stream_inline_completions()` method should include a constant and unique string 155 | token identifying the stream. All chunks except for the last chunk for a given 156 | item should have the `done` value set to `False`. 157 | 158 | The following example demonstrates a custom implementation of the completion 159 | provider with both a method for sending multiple completions in one go, and 160 | streaming multiple completions concurrently. The implementation and explanation 161 | for the `merge_iterators` function used in this example can be found 162 | [here](https://stackoverflow.com/q/72445371/4877269). 163 | 164 | ```python 165 | class MyCompletionProvider(BaseProvider, FakeListLLM): 166 | id = "my_provider" 167 | name = "My Provider" 168 | model_id_key = "model" 169 | models = ["model_a"] 170 | 171 | def __init__(self, **kwargs): 172 | kwargs["responses"] = ["This fake response will not be used for completion"] 173 | super().__init__(**kwargs) 174 | 175 | async def generate_inline_completions(self, request: InlineCompletionRequest): 176 | return InlineCompletionReply( 177 | list=InlineCompletionList(items=[ 178 | {"insertText": "An ant minding its own business"}, 179 | {"insertText": "A bug searching for a snack"} 180 | ]), 181 | reply_to=request.number, 182 | ) 183 | 184 | async def stream_inline_completions(self, request: InlineCompletionRequest): 185 | token_1 = f"t{request.number}s0" 186 | token_2 = f"t{request.number}s1" 187 | 188 | yield InlineCompletionReply( 189 | list=InlineCompletionList( 190 | items=[ 191 | {"insertText": "An ", "isIncomplete": True, "token": token_1}, 192 | {"insertText": "", "isIncomplete": True, "token": token_2} 193 | ] 194 | ), 195 | reply_to=request.number, 196 | ) 197 | 198 | # where merge_iterators 199 | async for reply in merge_iterators([ 200 | self._stream("elephant dancing in the rain", request.number, token_1, start_with="An"), 201 | self._stream("A flock of birds flying around a mountain", request.number, token_2) 202 | ]): 203 | yield reply 204 | 205 | async def _stream(self, sentence, request_number, token, start_with = ""): 206 | suggestion = start_with 207 | 208 | for fragment in sentence.split(): 209 | await asyncio.sleep(0.75) 210 | suggestion += " " + fragment 211 | yield InlineCompletionStreamChunk( 212 | type="stream", 213 | response={"insertText": suggestion, "token": token}, 214 | reply_to=request_number, 215 | done=False 216 | ) 217 | 218 | # finally, send a message confirming that we are done 219 | yield InlineCompletionStreamChunk( 220 | type="stream", 221 | response={"insertText": suggestion, "token": token}, 222 | reply_to=request_number, 223 | done=True, 224 | ) 225 | ``` 226 | 227 | ### Using the full notebook content for completions 228 | 229 | The `InlineCompletionRequest` contains the `path` of the current document (file or notebook). 230 | Inline completion providers can use this path to extract the content of the notebook from the disk, 231 | however such content may be outdated if the user has not saved the notebook recently. 232 | 233 | The accuracy of the suggestions can be slightly improved by combining the potentially outdated content of previous/following cells 234 | with the `prefix` and `suffix` which describe the up-to-date state of the current cell (identified by `cell_id`). 235 | 236 | Still, reading the full notebook from the disk may be slow for larger notebooks, which conflicts with the low latency requirement of inline completion. 237 | 238 | A better approach is to use the live copy of the notebook document that is persisted on the jupyter-server when *collaborative* document models are enabled. 239 | Two packages need to be installed to access the collaborative models: 240 | - `jupyter-server-ydoc` (>= 1.0) stores the collaborative models in the jupyter-server on runtime 241 | - `jupyter-docprovider` (>= 1.0) reconfigures JupyterLab/Notebook to use the collaborative models 242 | 243 | Both packages are automatically installed with `jupyter-collaboration` (in v3.0 or newer), however installing `jupyter-collaboration` is not required to take advantage of *collaborative* models. 244 | 245 | The snippet below demonstrates how to retrieve the content of all cells of a given type from the in-memory copy of the collaborative model (without additional disk reads). 246 | 247 | ```python 248 | from jupyter_ydoc import YNotebook 249 | 250 | 251 | class MyCompletionProvider(BaseProvider, FakeListLLM): 252 | id = "my_provider" 253 | name = "My Provider" 254 | model_id_key = "model" 255 | models = ["model_a"] 256 | 257 | def __init__(self, **kwargs): 258 | kwargs["responses"] = ["This fake response will not be used for completion"] 259 | super().__init__(**kwargs) 260 | 261 | async def _get_prefix_and_suffix(self, request: InlineCompletionRequest): 262 | prefix = request.prefix 263 | suffix = request.suffix.strip() 264 | 265 | server_ydoc = self.server_settings.get("jupyter_server_ydoc", None) 266 | if not server_ydoc: 267 | # fallback to prefix/suffix from single cell 268 | return prefix, suffix 269 | 270 | is_notebook = request.path.endswith("ipynb") 271 | document = await server_ydoc.get_document( 272 | path=request.path, 273 | content_type="notebook" if is_notebook else "file", 274 | file_format="json" if is_notebook else "text" 275 | ) 276 | if not document or not isinstance(document, YNotebook): 277 | return prefix, suffix 278 | 279 | cell_type = "markdown" if request.language == "markdown" else "code" 280 | 281 | is_before_request_cell = True 282 | before = [] 283 | after = [suffix] 284 | 285 | for cell in document.ycells: 286 | if is_before_request_cell and cell["id"] == request.cell_id: 287 | is_before_request_cell = False 288 | continue 289 | if cell["cell_type"] != cell_type: 290 | continue 291 | source = cell["source"].to_py() 292 | if is_before_request_cell: 293 | before.append(source) 294 | else: 295 | after.append(source) 296 | 297 | before.append(prefix) 298 | prefix = "\n\n".join(before) 299 | suffix = "\n\n".join(after) 300 | return prefix, suffix 301 | 302 | async def generate_inline_completions(self, request: InlineCompletionRequest): 303 | prefix, suffix = await self._get_prefix_and_suffix(request) 304 | 305 | return InlineCompletionReply( 306 | list=InlineCompletionList(items=[ 307 | {"insertText": your_llm_function(prefix, suffix)} 308 | ]), 309 | reply_to=request.number, 310 | ) 311 | ``` 312 | 313 | [entry_points]: https://setuptools.pypa.io/en/latest/userguide/entry_point.html 314 | [langchain_llms]: https://api.python.langchain.com/en/v0.0.339/api_reference.html#module-langchain.llms 315 | [custom_llm]: https://python.langchain.com/docs/modules/model_io/models/llms/custom_llm 316 | [LLM]: https://api.python.langchain.com/en/v0.0.339/llms/langchain.llms.base.LLM.html#langchain.llms.base.LLM 317 | [BaseChatModel]: https://api.python.langchain.com/en/v0.0.339/chat_models/langchain.chat_models.base.BaseChatModel.html 318 | -------------------------------------------------------------------------------- /docs/source/users/index.md: -------------------------------------------------------------------------------- 1 | # Users 2 | 3 | Welcome to the user documentation for Jupyter AI. 4 | 5 | If you are interested in contributing to Jupyter AI, 6 | please see our {doc}`contributor's guide `. 7 | 8 | If you would like to build applications that enhance Jupyter AI, 9 | please see the {doc}`developer's guide `. 10 | 11 | ## Prerequisites 12 | 13 | You can run Jupyter AI on any system that can run a supported Python version 14 | from 3.9 to 3.12, including recent Windows, macOS, and Linux versions. Python 15 | 3.8 support is also available in Jupyter AI v2.29.1 and below. 16 | 17 | If you use `conda`, you can install Python 3.12 in your environment by running: 18 | 19 | ``` 20 | conda install python=3.12 21 | ``` 22 | 23 | The `jupyter_ai` package, which provides the lab extension and user interface in 24 | JupyterLab, depends on JupyterLab 4. If upgrading to JupyterLab 4 is not 25 | possible in your environment, you should install `jupyter_ai` v1.x instead. 26 | See "Installation" for more details. 27 | 28 | :::{attention} 29 | :name: jupyter-lab-3-end-of-maintenance 30 | JupyterLab 3 reached its end of maintenance date on May 15, 2024. As a result, we will not backport new features to the v1 branch supporting JupyterLab 3. Fixes for critical issues will still be backported until December 31, 2024. If you are still using JupyterLab 3, we strongly encourage you to **upgrade to JupyterLab 4 as soon as possible**. For more information, see [JupyterLab 3 end of maintenance](https://blog.jupyter.org/jupyterlab-3-end-of-maintenance-879778927db2) on the Jupyter Blog. 31 | ::: 32 | 33 | You can install JupyterLab using `pip` or `conda`. 34 | 35 | 1. via `pip`: 36 | 37 | ``` 38 | # change 4.0 to 3.0 if you need JupyterLab 3 39 | pip install jupyterlab~=4.0 40 | ``` 41 | 42 | 2. via `conda`: 43 | 44 | ``` 45 | # change 4.0 to 3.0 if you need JupyterLab 3 46 | conda config --add channels conda-forge 47 | conda config --set channel_priority strict 48 | conda install jupyterlab~=4.0 49 | ``` 50 | 51 | You can also use Jupyter AI in Jupyter Notebook 7.2+. To install Jupyter Notebook 7.2: 52 | 53 | 1. via `pip`: 54 | 55 | ``` 56 | pip install notebook~=7.2 57 | ``` 58 | 59 | 2. via `conda`: 60 | ``` 61 | conda install notebook~=7.2 62 | ``` 63 | 64 | :::{note} 65 | To activate the chat interface in Jupyter Notebook, click on "View > Left sidebar > Show Jupyter AI Chat". 66 | ::: 67 | 68 | The `jupyter_ai_magics` package, which provides exclusively the IPython magics, 69 | does not depend on JupyterLab or `jupyter_ai`. You can install 70 | `jupyter_ai_magics` without installing `jupyterlab` or `jupyter_ai`. 71 | If you have both `jupyter_ai_magics` and `jupyter_ai` installed, you should 72 | have the same version of each, to avoid errors. 73 | 74 | Jupyter AI internally uses Pydantic v1 and should work with either Pydantic 75 | version 1 or version 2. For compatibility, developers using Pydantic V2 76 | should import classes using the `pydantic.v1` package. See the 77 | [LangChain Pydantic migration plan](https://python.langchain.com/docs/guides/pydantic_compatibility) 78 | for advice about how developers should use `v1` to avoid mixing v1 and v2 79 | classes in their code. 80 | 81 | ## Installation 82 | 83 | ### Setup: creating a Jupyter AI environment (recommended) 84 | 85 | Before installing Jupyter AI, we highly recommend first creating a separate 86 | Conda environment for Jupyter AI. This prevents the installation process from 87 | clobbering Python packages in your existing Python environment. 88 | 89 | To do so, install 90 | [conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html) 91 | and create an environment that uses Python 3.12 and the latest version of 92 | JupyterLab: 93 | 94 | $ conda create -n jupyter-ai python=3.12 jupyterlab 95 | $ conda activate jupyter-ai 96 | 97 | You can now choose how to install Jupyter AI. 98 | 99 | We offer 3 different ways to install Jupyter AI. You can read through each 100 | section to pick the installation method that works best for you. 101 | 102 | 1. Quick installation via `pip` (recommended) 103 | 2. Minimal installation via `pip` 104 | 3. Minimal installation via `conda` 105 | 106 | ### Quick installation via `pip` (recommended) 107 | 108 | If you want to install both the `%%ai` magic and the JupyterLab extension, you can run: 109 | 110 | $ pip install 'jupyter-ai[all]' 111 | 112 | Then, restart JupyterLab. This will install every optional dependency, which 113 | provides access to all models currently supported by `jupyter-ai`. 114 | 115 | If you are not using JupyterLab and you only want to install the Jupyter AI 116 | `%%ai` magic, you can run: 117 | 118 | $ pip install 'jupyter-ai-magics[all]' 119 | 120 | `jupyter-ai` depends on `jupyter-ai-magics`, so installing `jupyter-ai` 121 | automatically installs `jupyter-ai-magics`. 122 | 123 | :::{warning} 124 | :name: quoting-cli-arguments 125 | If running the above commands result in an error like `zsh: no matches found: jupyter-ai[all]`, this is because the `jupyter-ai[all]` argument must be surrounded by single or double quotes. Some shells reserve square brackets for pattern matching, so arguments containing square brackets must be quoted. 126 | ::: 127 | 128 | 129 | ### Minimal installation via `pip` 130 | 131 | Most model providers in Jupyter AI require a specific dependency to be installed 132 | before they are available for use. These are called _provider dependencies_. 133 | Provider dependencies are optional to Jupyter AI, meaning that Jupyter AI can be 134 | installed with or without any provider dependencies installed. If a provider 135 | requires a dependency that is not installed, its models are not listed in the 136 | user interface which allows you to select a language model. 137 | 138 | To perform a minimal installation via `pip` without any provider dependencies, 139 | omit the `[all]` optional dependency group from the package name: 140 | 141 | ``` 142 | pip install jupyter-ai 143 | ``` 144 | 145 | By selectively installing provider dependencies, you can control which models 146 | are available in your Jupyter AI environment. 147 | 148 | For example, to install Jupyter AI with only added support for Anthropic models, run: 149 | 150 | ``` 151 | pip install jupyter-ai langchain-anthropic 152 | ``` 153 | 154 | For more information on model providers and which dependencies they require, see 155 | [the model provider table](https://jupyter-ai.readthedocs.io/en/latest/users/index.html#model-providers). 156 | 157 | ### Minimal installation via `conda` 158 | 159 | As an alternative to using `pip`, you can install `jupyter-ai` using 160 | [Conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html) 161 | from the `conda-forge` channel: 162 | 163 | $ conda install conda-forge::jupyter-ai 164 | 165 | Most model providers in Jupyter AI require a specific _provider dependency_ to 166 | be installed before they are available for use. Provider dependencies are 167 | not installed when installing `jupyter-ai` from Conda Forge, and should be 168 | installed separately as needed. 169 | 170 | For example, to install Jupyter AI with only added support for OpenAI models, run: 171 | 172 | ``` 173 | conda install conda-forge::jupyter-ai conda-forge::langchain-openai 174 | ``` 175 | 176 | For more information on model providers and which dependencies they require, see 177 | [the model provider table](https://jupyter-ai.readthedocs.io/en/latest/users/index.html#model-providers). 178 | 179 | ## Uninstallation 180 | 181 | If you installed Jupyter AI using `pip`, to remove the extension, run: 182 | 183 | $ pip uninstall jupyter-ai 184 | 185 | or 186 | 187 | $ pip uninstall jupyter-ai-magics 188 | 189 | If you installed Jupyter AI using `conda`, you can remove it by running: 190 | 191 | $ conda remove jupyter-ai 192 | 193 | or 194 | 195 | $ conda remove jupyter-ai-magics 196 | 197 | ## Model providers 198 | 199 | Jupyter AI supports a wide range of model providers and models. To use Jupyter AI with a particular provider, you must install its Python packages and set its API key (or other credentials) in your environment or in the chat interface. 200 | 201 | Jupyter AI supports the following model providers: 202 | 203 | | Provider | Provider ID | Environment variable(s) | Python package(s) | 204 | |------------------------------|----------------------|----------------------------|-------------------------------------------| 205 | | AI21 | `ai21` | `AI21_API_KEY` | `ai21` | 206 | | Anthropic | `anthropic` | `ANTHROPIC_API_KEY` | `langchain-anthropic` | 207 | | Anthropic (chat) | `anthropic-chat` | `ANTHROPIC_API_KEY` | `langchain-anthropic` | 208 | | Bedrock | `bedrock` | N/A | `langchain-aws` | 209 | | Bedrock (chat) | `bedrock-chat` | N/A | `langchain-aws` | 210 | | Bedrock (custom/provisioned) | `bedrock-custom` | N/A | `langchain-aws` | 211 | | Cohere | `cohere` | `COHERE_API_KEY` | `langchain-cohere` | 212 | | ERNIE-Bot | `qianfan` | `QIANFAN_AK`, `QIANFAN_SK` | `qianfan` | 213 | | Gemini | `gemini` | `GOOGLE_API_KEY` | `langchain-google-genai` | 214 | | GPT4All | `gpt4all` | N/A | `gpt4all` | 215 | | Hugging Face Hub | `huggingface_hub` | `HUGGINGFACEHUB_API_TOKEN` | `huggingface_hub`, `ipywidgets`, `pillow` | 216 | | MistralAI | `mistralai` | `MISTRAL_API_KEY` | `langchain-mistralai` | 217 | | NVIDIA | `nvidia-chat` | `NVIDIA_API_KEY` | `langchain_nvidia_ai_endpoints` | 218 | | Ollama | `ollama` | N/A | `langchain-ollama` | 219 | | OpenAI | `openai` | `OPENAI_API_KEY` | `langchain-openai` | 220 | | OpenAI (chat) | `openai-chat` | `OPENAI_API_KEY` | `langchain-openai` | 221 | | SageMaker endpoint | `sagemaker-endpoint` | N/A | `langchain-aws` | 222 | 223 | The environment variable names shown above are also the names of the settings keys used when setting up the chat interface. 224 | If multiple variables are listed for a provider, **all** must be specified. 225 | 226 | To use the Bedrock models, you need access to the Bedrock service, and you will need to authenticate via [boto3](https://github.com/boto/boto3). For more information, see the [Amazon Bedrock Homepage](https://aws.amazon.com/bedrock/). 227 | 228 | You need the `pillow` Python package to use Hugging Face Hub's text-to-image models. 229 | 230 | You can find a list of Hugging Face's models at [https://huggingface.co/models](https://huggingface.co/models). 231 | 232 | To use NVIDIA models, create a free account with the [NVIDIA NGC service](https://catalog.ngc.nvidia.com/), which hosts AI solution catalogs, containers, models, and more. Navigate to Catalog > [AI Foundation Models](https://catalog.ngc.nvidia.com/ai-foundation-models), and select a model with an API endpoint. Click "API" on the model's detail page, and click "Generate Key". Save this key, and set it as the environment variable `NVIDIA_API_KEY` to access any of the model endpoints. 233 | 234 | SageMaker endpoint names are created when you deploy a model. For more information, see 235 | ["Create your endpoint and deploy your model"](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints-deployment.html) 236 | in the SageMaker documentation. 237 | 238 | To use SageMaker's models, you will need to authenticate via 239 | [boto3](https://github.com/boto/boto3). 240 | 241 | For example, to use OpenAI models, use the chat interface settings panel to choose the OpenAI language model: 242 | 243 | Screen shot of the chat settings interface with language model dropdown open 246 | 247 | Then, enter your API key in the 'API Keys' section. 248 | 249 | :::{attention} 250 | :name: open-ai-cost 251 | Model providers may charge users for API usage. Jupyter AI users are 252 | responsible for all charges they incur when they make API requests. Review your model 253 | provider's pricing information before submitting requests via Jupyter AI. 254 | ::: 255 | 256 | ## The chat interface 257 | 258 | The easiest way to get started with Jupyter AI is to use the chat interface. 259 | 260 | :::{attention} 261 | :name: open-ai-privacy-cost 262 | The chat interface sends data to generative AI models hosted by third parties. Please review your model provider's privacy policy to understand how it may use the data you send to it. Review its pricing model so that you understand your payment obligations when using the chat interface. 263 | ::: 264 | 265 | Once you have started JupyterLab, click the new "chat" icon in the left side panel to open the chat interface. You can right-click on the panel icon and move it to the other side, if you prefer. 266 | 267 | Screen shot of the setup interface 270 | 271 | The first time you open the chat interface, Jupyter AI will ask you which models you want to use as a language model and as an embedding model. Once you have made your selections, the UI may display text boxes for one or more settings keys. 272 | 273 | :::{admonition} Language models and embedding models 274 | :class: tip 275 | :name: language-models-and-embedding-models 276 | Users may select a language model and, optionally, an embedding model. You should select one of each so that you can use the full functionality of the chat interface. 277 | 278 | A **language model** responds to users' messages in the chat panel. It accepts a prompt and produces a response. Language models are typically *pre-trained*; they are ready to use, but their training sets are biased and incomplete, and users need to be aware of their biases when they use the chat interface. 279 | 280 | An **embedding model** is used when [learning and asking about local data](#learning-about-local-data). These models can transform your data, including documents and source code files, into vectors that can help Jupyter AI compose prompts to language models. 281 | 282 | Your language model and your embedding model do not need to be provided by the same vendor, but you will need authentication credentials for each model provider that you use. 283 | ::: 284 | 285 | 286 | Screen shot of the setup interface, showing model selections and key 289 | 290 | Before you can use the chat interface, you need to provide your API keys for the model providers that you have selected. Paste or type your keys into the boxes provided. 291 | 292 | Screen shot of the setup interface, showing model selections and key populated 295 | 296 | Once you have set all the necessary keys, click the "back" (left arrow) button in the upper-left corner of the Jupyter AI side panel. The chat interface now appears, with a help menu of available `/` (slash) commands, and you can ask a question using the message box at the bottom. 297 | 298 | Screen shot of the initial chat interface. 301 | 302 | You may customize the template of the chat interface from the default one. The steps are as follows: 303 | 1. Create a new `config.py` file in your current directory with the contents you want to see in the help message, by editing the template below: 304 | ``` 305 | c.AiExtension.help_message_template = """ 306 | Sup. I'm {persona_name}. This is a sassy custom help message. 307 | 308 | Here's the slash commands you can use. Use 'em or don't... I don't care. 309 | 310 | {slash_commands_list} 311 | """.strip() 312 | ``` 313 | 2. Start JupyterLab with the following command: 314 | ``` 315 | jupyter lab --config=config.py 316 | ``` 317 | The new help message will be used instead of the default, as shown below 318 | 319 | Screen shot of the custom chat interface. 322 | 323 | To compose a message, type it in the text box at the bottom of the chat interface and press ENTER to send it. You can press SHIFT+ENTER to add a new line. (These are the default keybindings; you can change them in the chat settings pane.) Once you have sent a message, you should see a response from Jupyternaut, the Jupyter AI chatbot. 324 | 325 | Screen shot of an example "Hello world" message sent to Jupyternaut, who responds with "Hello world, how are you today?" 328 | 329 | The chat backend remembers the last two exchanges in your conversation and passes them to the language model. You can ask follow up questions without repeating information from your previous conversations. Here is an example of a chat conversation with a follow up question: 330 | 331 | #### Initial question 332 | Screen shot of an example coding question sent to Jupyternaut, who responds with the code and explanation. 335 | 336 | #### Follow-up question 337 | Screen shot of an example follow up question sent to Jupyternaut, who responds with the improved code and explanation. 340 | 341 | 342 | ### Amazon Bedrock Usage 343 | 344 | Jupyter AI enables use of language models hosted on [Amazon Bedrock](https://aws.amazon.com/bedrock/) on AWS. Ensure that you have authentication to use AWS using the `boto3` SDK with credentials stored in the `default` profile. Guidance on how to do this can be found in the [`boto3` documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). 345 | 346 | For details on enabling model access in your AWS account, using cross-region inference, or invoking custom/provisioned models, please see our dedicated documentation page on [using Amazon Bedrock in Jupyter AI](bedrock.md). 347 | 348 | 349 | ### OpenRouter and OpenAI Interface Usage 350 | 351 | Jupyter AI enables use of language models accessible through [OpenRouter](https://openrouter.ai)'s unified interface. Examples of models that may be accessed via OpenRouter are: [Deepseek](https://openrouter.ai/deepseek/deepseek-chat), [Qwen](https://openrouter.ai/qwen/), [mistral](https://openrouter.ai/mistralai/), etc. OpenRouter enables usage of any model conforming to the OpenAI API. 352 | 353 | Likewise, for many models, you may directly choose the OpenAI provider in Jupyter AI instead of OpenRouter in the same way. 354 | 355 | For details on enabling model access via the AI Settings and using models via OpenRouter or OpenAI, please see the dedicated documentation page on using [OpenRouter and OpenAI providers in Jupyter AI](openrouter.md). 356 | 357 | 358 | ### SageMaker endpoints usage 359 | 360 | Jupyter AI supports language models hosted on SageMaker endpoints that use JSON 361 | schemas. The first step is to authenticate with AWS via the `boto3` SDK and have 362 | the credentials stored in the `default` profile. Guidance on how to do this can 363 | be found in the 364 | [`boto3` documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). 365 | 366 | When selecting the SageMaker provider in the settings panel, you will 367 | see the following interface: 368 | 369 | Screenshot of the settings panel with the SageMaker provider selected. 373 | 374 | Each of the additional fields under "Language model" is required. These fields 375 | should contain the following data: 376 | 377 | - **Endpoint name**: The name of your endpoint. This can be retrieved from the 378 | AWS Console at the URL 379 | `https://.console.aws.amazon.com/sagemaker/home?region=#/endpoints`. 380 | 381 | - **Region name**: The AWS region your SageMaker endpoint is hosted in, such as `us-west-2`. 382 | 383 | - **Request schema**: The JSON object the endpoint expects, with the prompt 384 | being substituted into any value that matches the string literal `""`. 385 | In this example, the request schema `{"text_inputs":""}` generates a JSON 386 | object with the prompt stored under the `text_inputs` key. 387 | 388 | - **Response path**: A [JSONPath](https://goessner.net/articles/JsonPath/index.html) 389 | string that retrieves the language model's output from the endpoint's JSON 390 | response. In this example, the endpoint returns an object with the schema 391 | `{"generated_texts":[""]}`, hence the response path is 392 | `generated_texts.[0]`. 393 | 394 | ### GPT4All usage (early-stage) 395 | 396 | Currently, we offer experimental support for GPT4All. To get started, first 397 | decide which models you will use. We currently offer the following models from GPT4All: 398 | 399 | | Model name | Model size | Model bin URL | 400 | |---------------------------------|------------|------------------------------------------------------------| 401 | | `ggml-gpt4all-l13b-snoozy` | 7.6 GB | `http://gpt4all.io/models/ggml-gpt4all-l13b-snoozy.bin` | 402 | | `ggml-gpt4all-j-v1.2-jazzy` | 3.8 GB | `https://gpt4all.io/models/ggml-gpt4all-j-v1.2-jazzy.bin` | 403 | | `ggml-gpt4all-j-v1.3-groovy` | 3.8 GB | `https://gpt4all.io/models/ggml-gpt4all-j-v1.3-groovy.bin` | 404 | | `mistral-7b-openorca.Q4_0` | 3.8 GB | `https://gpt4all.io/models/gguf/mistral-7b-openorca.Q4_0.gguf` | 405 | | `mistral-7b-instruct-v0.1.Q4_0` | 3.8 GB | `https://gpt4all.io/models/gguf/mistral-7b-instruct-v0.1.Q4_0.gguf` | 406 | | `gpt4all-falcon-q4_0` | 3.9 GB | `https://gpt4all.io/models/gguf/gpt4all-falcon-q4_0.gguf` | 407 | | `wizardlm-13b-v1.2.Q4_0` | 6.9 GB | `https://gpt4all.io/models/gguf/wizardlm-13b-v1.2.Q4_0.gguf` | 408 | | `nous-hermes-llama2-13b.Q4_0` | 6.9 GB | `https://gpt4all.io/models/gguf/nous-hermes-llama2-13b.Q4_0.gguf` | 409 | | `gpt4all-13b-snoozy-q4_0` | 6.9 GB | `https://gpt4all.io/models/gguf/gpt4all-13b-snoozy-q4_0.gguf` | 410 | | `mpt-7b-chat-merges-q4_0` | 3.5 GB | `https://gpt4all.io/models/gguf/mpt-7b-chat-merges-q4_0.gguf` | 411 | | `orca-mini-3b-gguf2-q4_0` | 1.8 GB | `https://gpt4all.io/models/gguf/orca-mini-3b-gguf2-q4_0.gguf` | 412 | | `starcoder-q4_0` | 8.4 GB | `https://gpt4all.io/models/gguf/starcoder-q4_0.gguf` | 413 | | `rift-coder-v0-7b-q4_0` | 3.6 GB | `https://gpt4all.io/models/gguf/rift-coder-v0-7b-q4_0.gguf` | 414 | | `all-MiniLM-L6-v2-f16` | 44 MB | `https://gpt4all.io/models/gguf/all-MiniLM-L6-v2-f16.gguf` | 415 | | `em_german_mistral_v01.Q4_0` | 3.8 GB | `https://huggingface.co/TheBloke/em_german_mistral_v01-GGUF/resolve/main/em_german_mistral_v01.Q4_0.gguf` | 416 | 417 | 418 | Note that each model comes with its own license, and that users are themselves 419 | responsible for verifying that their usage complies with the license. You can 420 | find licensing details on the [GPT4All official site](https://gpt4all.io/index.html). 421 | 422 | First, create a folder to store the model files. 423 | 424 | ``` 425 | mkdir ~/.cache/gpt4all 426 | ``` 427 | 428 | For each model you use, you will have to run the command 429 | 430 | ``` 431 | curl -LO --output-dir ~/.cache/gpt4all "" 432 | ``` 433 | 434 | , where `` should be substituted with the corresponding URL 435 | hosting the model binary (within the double quotes). After restarting the 436 | server, the GPT4All models installed in the previous step should be available to 437 | use in the chat interface. 438 | 439 | GPT4All support is still an early-stage feature, so some bugs may be encountered 440 | during usage. Our team is still actively improving support for locally-hosted 441 | models. 442 | 443 | ### Ollama usage 444 | 445 | To get started, follow the instructions on the [Ollama website](https://ollama.com/) to set up `ollama` and download the models locally. To select a model, enter the model name in the settings panel, for example `deepseek-coder-v2`. You can see all locally available models with `ollama list`. 446 | 447 | For the Ollama models to be available to JupyterLab-AI, your Ollama server _must_ be running. You can check that this is the case by calling `ollama serve` at the terminal, and should see something like: 448 | 449 | ``` 450 | $ ollama serve 451 | Error: listen tcp 127.0.0.1:11434: bind: address already in use 452 | ``` 453 | 454 | In some platforms (e.g. macOS or Windows), there may also be a graphical user interface or application that lets you start/stop the Ollama server from a menu. 455 | 456 | :::{tip} 457 | If you don't see Ollama listed as a model provider in the Jupyter-AI configuration box, despite confirming that your Ollama server is active, you may be missing the [`langchain-ollama` python package](https://pypi.org/project/langchain-ollama/) that is necessary for Jupyter-AI to interface with Ollama, as indicated in the [model providers](#model-providers) section above. 458 | 459 | You can install it with `pip install langchain-ollama` (as of Feb'2025 it is not available on conda-forge). 460 | ::: 461 | 462 | By default, Ollama is served on `127.0.0.1:11434` (locally on port `11434`), so Jupyter AI expects this by default. If you wish to use a remote Ollama server with a different IP address or a local Ollama server on a different port number, you have to configure this in advance. 463 | 464 | To configure this in the chat, set the "Base API URL" field in the AI settings page to your Ollama server's custom IP address and port number: 465 | 466 | Screenshot of the settings panel with Ollama on non-default port. 470 | 471 | 472 | To configure this in the magic commands, you should set the `OLLAMA_HOST` environment variable to the your Ollama server's custom IP address and port number (assuming you chose 11000) in a new code cell: 473 | 474 | ``` 475 | %load_ext jupyter_ai_magics 476 | os.environ["OLLAMA_HOST"] = "http://localhost:11000" 477 | ``` 478 | 479 | After running that cell, the AI magic command can then be used like so: 480 | 481 | ``` 482 | %%ai ollama:llama3.2 483 | What is a transformer? 484 | ``` 485 | 486 | ### vLLM usage 487 | 488 | `vLLM` is a fast and easy-to-use library for LLM inference and serving. The [vLLM website](https://docs.vllm.ai/en/latest/) explains installation and usage. To use `vLLM` in Jupyter AI, please see the dedicated documentation page on using [vLLM in Jupyter AI](vllm.md). 489 | 490 | ### Asking about something in your notebook 491 | 492 | Jupyter AI's chat interface can include a portion of your notebook in your prompt. 493 | 494 | :::{warning} 495 | :name: include-selection-cost 496 | When you choose to include the selection with your message, this may increase the 497 | number of tokens in your request, which may cause your request to cost more money. 498 | Review your model provider's cost policy before making large requests. 499 | ::: 500 | 501 | After highlighting a portion of your notebook, check "Include selection" in the chat panel, type your message, and then send your message. Your outgoing message will include your selection. 502 | 503 | Screen shot of JupyterLab with Jupyter AI's chat panel active. A Python function is selected, the user has "What does this code do?" as their prompt, and the user has chosen to include the selection with their message. 506 | 507 | Below your message, you will see Jupyternaut's response. 508 | 509 | Screen shot of Jupyter AI's chat panel, showing an answer to the question asked above. 512 | 513 | You can copy Jupyternaut's response to the clipboard so that you can paste it into your notebook, or into any other application. You can also choose to replace the selection with Jupyternaut's response by clicking "Replace selection" before you send your message. 514 | 515 | :::{warning} 516 | :name: replace-selection 517 | When you replace your selection, data is written immediately after Jupyter AI 518 | sends its response to your message. Review any generated code carefully before 519 | you run it. 520 | ::: 521 | 522 | Screen shot of Jupyter AI with a Python function selected, the user having typed "Rewrite this function to be iterative, not recursive" as their prompt, and with the user having chosen to include the selection with their message and to replace the selection with the response. 525 | 526 | After Jupyternaut sends a response, your notebook will be updated immediately with the response replacing the selection. You can also see the response in the chat panel. 527 | 528 | ### Generating a new notebook 529 | 530 | You can use Jupyter AI to generate an entire notebook from a text prompt. To get started, open the chat panel, and send it a message starting with `/generate`. 531 | 532 | Screen shot of a prompt reading "/generate A demonstration of how to use Matplotlib" in Jupyter AI 535 | 536 | Generating a notebook can take a substantial amount of time, so Jupyter AI will respond to your message immediately while it works. You can continue to ask it other questions in the meantime. 537 | 538 | Screen shot of Jupyternaut responding to a generate message with a message that it is working on a notebook. 541 | 542 | :::{note} 543 | :name: generate-progress 544 | Especially if your prompt is detailed, it may take several minutes to generate 545 | your notebook. During this time, you can still use JupyterLab and Jupyter AI 546 | as you would normally. Do not shut your JupyterLab instance down while 547 | Jupyter AI is working. 548 | ::: 549 | 550 | When Jupyter AI is done generating your notebook, it will send you another message with the filename that it generated. You can then open this file using the file browser. 551 | 552 | :::{warning} 553 | :name: generated-notebook 554 | Generated notebooks may contain errors and may have unintended side effects when 555 | you run the code contained in them. Please review all generated code carefully 556 | before you run it. 557 | ::: 558 | 559 | ### Learning about local data 560 | 561 | Using the `/learn` command, you can teach Jupyter AI about local data so that Jupyternaut can include it when answering your questions. This local data is embedded using the embedding model you selected in the settings panel. 562 | 563 | :::{warning} 564 | :name: learning-embedding-model 565 | If you are using an embedding model hosted by a third party, please review your 566 | model provider's policies before sending any confidential, sensitive, or 567 | privileged data to your embedding model. 568 | ::: 569 | 570 | To teach Jupyter AI about a folder full of documentation, for example, run `/learn docs/`. You will receive a response when Jupyter AI has indexed this documentation in a local vector database. 571 | 572 | Screen shot of "/learn docs/" command and a response. 575 | 576 | The `/learn` command also supports unix shell-style wildcard matching. This allows fine-grained file selection for learning. For example, to learn on only notebooks in all directories you can use `/learn **/*.ipynb` and all notebooks within your base (or preferred directory if set) will be indexed, while all other file extensions will be ignored. 577 | 578 | :::{warning} 579 | :name: unix shell-style wildcard matching 580 | Certain patterns may cause `/learn` to run more slowly. For instance `/learn **` may cause directories to be walked multiple times in search of files. 581 | ::: 582 | 583 | You can then use `/ask` to ask a question specifically about the data that you taught Jupyter AI with `/learn`. 584 | 585 | Screen shot of an "/ask" command and a response. 588 | 589 | To clear the local vector database, you can run `/learn -d` and Jupyter AI will forget all information that it learned from your `/learn` commands. 590 | 591 | Screen shot of a "/learn -d" command and a response. 594 | 595 | With the `/learn` command, some models work better with custom chunk size and chunk overlap values. To override the defaults, 596 | use the `-c` or `--chunk-size` option and the `-o` or `--chunk-overlap` option. 597 | 598 | ``` 599 | # default chunk size and chunk overlap 600 | /learn 601 | 602 | # chunk size of 500, and chunk overlap of 50 603 | /learn -c 500 -o 50 604 | 605 | # chunk size of 1000, and chunk overlap of 200 606 | /learn --chunk-size 1000 --chunk-overlap 200 607 | ``` 608 | 609 | By default, `/learn` will not read directories named `node_modules`, `lib`, or `build`, 610 | and will not read hidden files or hidden directories, where the file or directory name 611 | starts with a `.`. To force `/learn` to read all supported file types in all directories, 612 | use the `-a` or `--all-files` option. 613 | 614 | ``` 615 | # do not learn from hidden files, hidden directories, or node_modules, lib, or build directories 616 | /learn 617 | 618 | # learn from all supported files 619 | /learn -a 620 | ``` 621 | 622 | #### Supported files for the learn command 623 | 624 | Jupyter AI can only learn from files with the following file extensions: 625 | 626 | * .py 627 | * .md 628 | * .R 629 | * .Rmd 630 | * .jl 631 | * .sh 632 | * .ipynb 633 | * .js 634 | * .ts 635 | * .jsx 636 | * .tsx 637 | * .txt 638 | * .html 639 | * .pdf 640 | * .tex 641 | 642 | ### Learning arXiv files 643 | 644 | The `/learn` command also provides downloading and processing papers from the [arXiv](https://arxiv.org/) repository. You will need to install the `arxiv` python package for this feature to work. Run `pip install arxiv` to install the `arxiv` package. 645 | 646 | ``` 647 | /learn -r arxiv 2404.18558 648 | ``` 649 | 650 | ### Exporting chat history 651 | Use the `/export` command to export the chat history from the current session to a markdown file named `chat_history-YYYY-MM-DD-HH-mm-ss.md`. You can also specify a filename using `/export `. Each export will include the entire chat history up to that point in the session. 652 | 653 | 654 | ### Fixing a code cell with an error 655 | 656 | The `/fix` command can be used to fix any code cell with an error output in a 657 | Jupyter notebook file. To start, type `/fix` into the chat input. Jupyter AI 658 | will then prompt you to select a cell with error output before sending the 659 | request. 660 | 661 | Screenshot of the chat input containing `/fix` without a code cell with error output selected. 664 | 665 | Then click on a code cell with error output. A blue bar should appear 666 | immediately to the left of the code cell. 667 | 668 | Screenshot of a code cell with error output selected. 671 | 672 | After this, the Send button to the right of the chat input will be enabled, and 673 | you can use your mouse or keyboard to send `/fix` to Jupyternaut. The code cell 674 | and its associated error output are included in the message automatically. When 675 | complete, Jupyternaut will reply with suggested code that should fix the error. 676 | You can use the action toolbar under each code block to quickly replace the 677 | contents of the failing cell. 678 | 679 | Screenshot of a response from `/fix`, with the "Replace active cell" action hovered. 682 | 683 | 684 | ### Additional chat commands 685 | 686 | To start a new conversation, use the `/clear` command. This will clear the chat panel and reset the model's memory. 687 | 688 | ## The `%ai` and `%%ai` magic commands 689 | 690 | Jupyter AI can also be used in notebooks via Jupyter AI magics. This section 691 | provides guidance on how to use Jupyter AI magics effectively. The examples in 692 | this section are based on the [Jupyter AI example notebooks](https://github.com/jupyterlab/jupyter-ai/blob/main/examples/). 693 | 694 | If you already have `jupyter_ai` installed, the magics package 695 | `jupyter_ai_magics` is installed automatically. Otherwise, run 696 | 697 | pip install jupyter_ai_magics 698 | 699 | in your terminal to install the magics package. 700 | 701 | Before you send your first prompt to an AI model, load the IPython extension by 702 | running the following code in a notebook cell or IPython shell: 703 | 704 | ``` 705 | %load_ext jupyter_ai_magics 706 | ``` 707 | 708 | This command should not produce any output. 709 | 710 | :::{note} 711 | If you are using remote kernels, such as in Amazon SageMaker Studio, the above 712 | command will throw an error. You will need to install the magics package 713 | on your remote kernel separately, even if you already have `jupyter_ai_magics` 714 | installed in your server's environment. In a notebook, run 715 | 716 | ``` 717 | %pip install jupyter_ai_magics 718 | ``` 719 | 720 | and re-run `%load_ext jupyter_ai_magics`. 721 | ::: 722 | 723 | Once the extension has loaded, you can run `%%ai` cell magic commands and 724 | `%ai` line magic commands. Run `%%ai help` or `%ai help` for help with syntax. 725 | You can also pass `--help` as an argument to any line magic command (for example, 726 | `%ai list --help`) to learn about what the command does and how to use it. 727 | 728 | ### Choosing a provider and model 729 | 730 | The `%%ai` cell magic allows you to invoke a language model of your choice with 731 | a given prompt. The model is identified with a **global model ID**, which is a string with the 732 | syntax `:`, where `` is the ID of the 733 | provider and `` is the ID of the model scoped to that provider. 734 | The prompt begins on the second line of the cell. 735 | 736 | For example, to send a text prompt to the provider `anthropic` and the model ID 737 | `claude-v1.2`, enter the following code into a cell and run it: 738 | 739 | ``` 740 | %%ai anthropic:claude-v1.2 741 | Write a poem about C++. 742 | ``` 743 | 744 | We currently support the following language model providers: 745 | 746 | - `ai21` 747 | - `anthropic` 748 | - `anthropic-chat` 749 | - `bedrock` 750 | - `bedrock-chat` 751 | - `bedrock-custom` 752 | - `cohere` 753 | - `huggingface_hub` 754 | - `nvidia-chat` 755 | - `ollama` 756 | - `openai` 757 | - `openai-chat` 758 | - `sagemaker-endpoint` 759 | 760 | ### Configuring a default model 761 | 762 | To configure a default model you can use the IPython `%config` magic: 763 | 764 | ```python 765 | %config AiMagics.initial_language_model = "anthropic:claude-v1.2" 766 | ``` 767 | 768 | Then subsequent magics can be invoked without typing in the model: 769 | 770 | ``` 771 | %%ai 772 | Write a poem about C++. 773 | ``` 774 | 775 | You can configure the default model for all notebooks by specifying `c.AiMagics.initial_language_model` tratilet in `ipython_config.py`, for example: 776 | 777 | ```python 778 | c.AiMagics.initial_language_model = "anthropic:claude-v1.2" 779 | ``` 780 | 781 | The location of `ipython_config.py` file is documented in [IPython configuration reference](https://ipython.readthedocs.io/en/stable/config/intro.html). 782 | 783 | ### Listing available models 784 | 785 | Jupyter AI also includes multiple subcommands, which may be invoked via the 786 | `%ai` *line* magic. Jupyter AI uses subcommands to provide additional utilities 787 | in notebooks while keeping the same concise syntax for invoking a language model. 788 | 789 | The `%ai list` subcommand prints a list of available providers and models. Some 790 | providers explicitly define a list of supported models in their API. However, 791 | other providers, like Hugging Face Hub, lack a well-defined list of available 792 | models. In such cases, it's best to consult the provider's upstream 793 | documentation. The [Hugging Face website](https://huggingface.co/) includes a 794 | list of models, for example. 795 | 796 | Optionally, you can specify a provider ID as a positional argument to `%ai list` 797 | to get all models provided by one provider. For example, `%ai list openai` will 798 | display only models provided by the `openai` provider. 799 | 800 | ### Abbreviated syntax 801 | 802 | If your model ID is associated with only one provider, you can omit the `provider-id` and 803 | the colon from the first line. For example, because `ai21` is the only provider of the 804 | `j2-jumbo-instruct` model, you can either give the full provider and model, 805 | 806 | ``` 807 | %%ai ai21:j2-jumbo-instruct 808 | Write some JavaScript code that prints "hello world" to the console. 809 | ``` 810 | 811 | or just the model, 812 | 813 | ``` 814 | %%ai j2-jumbo-instruct # infers AI21 provider 815 | Write some JavaScript code that prints "hello world" to the console. 816 | ``` 817 | 818 | ### Formatting the output 819 | 820 | By default, Jupyter AI assumes that a model will output markdown, so the output of 821 | an `%%ai` command will be formatted as markdown by default. You can override this 822 | using the `-f` or `--format` argument to your magic command. Valid formats include: 823 | 824 | - `code` 825 | - `image` (for Hugging Face Hub's text-to-image models only) 826 | - `markdown` 827 | - `math` 828 | - `html` 829 | - `json` 830 | - `text` 831 | 832 | For example, to force the output of a command to be interpreted as HTML, you can run: 833 | 834 | ``` 835 | %%ai anthropic:claude-v1.2 -f html 836 | Create a square using SVG with a black border and white fill. 837 | ``` 838 | 839 | The following cell will produce output in IPython's `Math` format, which in a web browser 840 | will look like properly typeset equations. 841 | 842 | ``` 843 | %%ai chatgpt -f math 844 | Generate the 2D heat equation in LaTeX surrounded by `$$`. Do not include an explanation. 845 | ``` 846 | 847 | This prompt will produce output as a code cell below the input cell. 848 | 849 | :::{warning} 850 | :name: run-code 851 | **Please review any code that a generative AI model produces before you run it 852 | or distribute it.** 853 | The code that you get in response to a prompt may have negative side effects and may 854 | include calls to nonexistent (hallucinated) APIs. 855 | ::: 856 | 857 | ``` 858 | %%ai chatgpt -f code 859 | A function that computes the lowest common multiples of two integers, and 860 | a function that runs 5 test cases of the lowest common multiple function 861 | ``` 862 | 863 | ### Configuring the amount of history to include in the context 864 | 865 | By default, two previous Human/AI message exchanges are included in the context of the new prompt. 866 | You can change this using the IPython `%config` magic, for example: 867 | 868 | ```python 869 | %config AiMagics.max_history = 4 870 | ``` 871 | 872 | Note that old messages are still kept locally in memory, 873 | so they will be included in the context of the next prompt after raising the `max_history` value. 874 | 875 | You can configure the value for all notebooks 876 | by specifying `c.AiMagics.max_history` traitlet in `ipython_config.py`, for example: 877 | 878 | ```python 879 | c.AiMagics.max_history = 4 880 | ``` 881 | 882 | ### Clearing the chat history 883 | 884 | You can run the `%ai reset` line magic command to clear the chat history. After you do this, 885 | previous magic commands you've run will no longer be added as context in requests. 886 | 887 | ``` 888 | %ai reset 889 | ``` 890 | 891 | ### Interpolating in prompts 892 | 893 | Using curly brace syntax, you can include variables and other Python expressions in your 894 | prompt. This lets you execute a prompt using code that the IPython kernel knows about, 895 | but that is not in the current cell. 896 | 897 | For example, we can set a variable in one notebook cell: 898 | 899 | ```python 900 | poet = "Walt Whitman" 901 | ``` 902 | 903 | Then, we can use this same variable in an `%%ai` command in a later cell: 904 | 905 | ``` 906 | %%ai chatgpt 907 | Write a poem in the style of {poet} 908 | ``` 909 | 910 | When this cell runs, `{poet}` is interpolated as `Walt Whitman`, or as whatever `poet` 911 | is assigned to at that time. 912 | 913 | You can use the special `In` and `Out` list with interpolation syntax to explain code 914 | located elsewhere in a Jupyter notebook. For example, if you run the following code in 915 | a cell, and its input is assigned to `In[11]`: 916 | 917 | ```python 918 | for i in range(0, 5): 919 | print(i) 920 | ``` 921 | 922 | You can then refer to `In[11]` in an `%%ai` magic command, and it will be replaced 923 | with the code in question: 924 | 925 | ``` 926 | %%ai cohere:command-xlarge-nightly 927 | Please explain the code below: 928 | -- 929 | {In[11]} 930 | ``` 931 | 932 | You can also refer to the cell's output using the special `Out` list, with the same index. 933 | 934 | ``` 935 | %%ai cohere:command-xlarge-nightly 936 | Write code that would produce the following output: 937 | -- 938 | {Out[11]} 939 | ``` 940 | 941 | Jupyter AI also adds the special `Err` list, which uses the same indexes as `In` and `Out`. 942 | For example, if you run code in `In[3]` that produces an error, that error is captured in 943 | `Err[3]` so that you can request an explanation using a prompt such as: 944 | 945 | ``` 946 | %%ai chatgpt 947 | Explain the following Python error: 948 | -- 949 | {Err[3]} 950 | ``` 951 | 952 | The AI model that you use will then attempt to explain the error. You could also write a 953 | prompt that uses both `In` and `Err` to attempt to get an AI model to correct your code: 954 | 955 | ``` 956 | %%ai chatgpt --format code 957 | The following Python code: 958 | -- 959 | {In[3]} 960 | -- 961 | produced the following Python error: 962 | -- 963 | {Err[3]} 964 | -- 965 | Write a new version of this code that does not produce that error. 966 | ``` 967 | 968 | As a shortcut for explaining and fixing errors, you can use the `%ai fix` command, which will explain the most recent error using the model of your choice. 969 | 970 | ``` 971 | %ai fix anthropic:claude-v1.2 972 | ``` 973 | 974 | ### Creating and managing aliases 975 | 976 | You can create an alias for a model using the `%ai alias` command. For example, the command: 977 | 978 | ``` 979 | %ai alias claude anthropic:claude-v1.2 980 | ``` 981 | 982 | will register the alias `claude` as pointing to the `anthropic` provider's `claude-v1.2` model. You can then use this alias as you would use any other model name: 983 | 984 | ``` 985 | %%ai claude 986 | Write a poem about C++. 987 | ``` 988 | 989 | You can also define a custom LangChain chain: 990 | 991 | ```python 992 | from langchain.chains import LLMChain 993 | from langchain.prompts import PromptTemplate 994 | from langchain.llms import OpenAI 995 | 996 | llm = OpenAI(temperature=0.9) 997 | prompt = PromptTemplate( 998 | input_variables=["product"], 999 | template="What is a good name for a company that makes {product}?", 1000 | ) 1001 | chain = LLMChain(llm=llm, prompt=prompt) 1002 | ``` 1003 | 1004 | … and then use `%ai alias` to give it a name: 1005 | 1006 | ``` 1007 | %ai alias companyname chain 1008 | ``` 1009 | 1010 | You can change an alias's target using the `%ai update` command: 1011 | 1012 | ``` 1013 | %ai update claude anthropic:claude-instant-v1.0 1014 | ``` 1015 | 1016 | You can delete an alias using the `%ai dealias` command: 1017 | 1018 | ``` 1019 | %ai dealias claude 1020 | ``` 1021 | 1022 | You can see a list of all aliases by running the `%ai list` command. 1023 | 1024 | Aliases' names can contain ASCII letters (uppercase and lowercase), numbers, hyphens, underscores, and periods. They may not contain colons. They may also not override built-in commands — run `%ai help` for a list of these commands. 1025 | 1026 | Aliases must refer to models or `LLMChain` objects; they cannot refer to other aliases. 1027 | 1028 | To customize the aliases on startup you can set the `c.AiMagics.aliases` tratilet in `ipython_config.py`, for example: 1029 | 1030 | ```python 1031 | c.AiMagics.aliases = { 1032 | "my_custom_alias": "my_provider:my_model" 1033 | } 1034 | ``` 1035 | 1036 | The location of `ipython_config.py` file is documented in [IPython configuration reference](https://ipython.readthedocs.io/en/stable/config/intro.html). 1037 | 1038 | ### Using magic commands with SageMaker endpoints 1039 | 1040 | You can use magic commands with models hosted using Amazon SageMaker. 1041 | 1042 | First, make sure that you've set your `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables before starting JupyterLab as follows: 1043 | ``` 1044 | os.environ['AWS_ACCESS_KEY_ID'] = 1045 | os.environ['AWS_SECRET_ACCESS_KEY'] = 1046 | ``` 1047 | 1048 | You can also set the keys interactively and securely using the following code in your notebook: 1049 | 1050 | ```python 1051 | # NOTE: Enter the AWS access key id and the AWS secret access key when prompted by the code below 1052 | 1053 | import getpass 1054 | 1055 | # Enter your keys 1056 | access_key = getpass.getpass('Enter your AWS ACCESS KEY ID: ') 1057 | secret_access_key = getpass.getpass('Enter your AWS SECRET ACCESS KEY: ') 1058 | 1059 | # Set the environment variable without displaying the full key 1060 | os.environ['AWS_ACCESS_KEY_ID'] = access_key 1061 | os.environ['AWS_SECRET_ACCESS_KEY'] = secret_access_key 1062 | ``` 1063 | 1064 | :::{note} 1065 | :name: using-env-key 1066 | You may also set these keys directly using the `%env` magic command, but the key value may be echoed in the cell output. If you prefer to use `%env`, be sure to not share the notebook with people you don't trust, as this may leak your API keys. 1067 | ``` 1068 | %env AWS_ACCESS_KEY_ID = 1069 | %env AWS_SECRET_ACCESS_KEY = 1070 | ``` 1071 | ::: 1072 | 1073 | For more information about environment variables, see [Environment variables to configure the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) in AWS's documentation. 1074 | 1075 | Jupyter AI supports language models hosted on SageMaker endpoints that use JSON schemas. Authenticate with AWS via the `boto3` SDK and have the credentials stored in the `default` profile. Guidance on how to do this can be found in the [`boto3` documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). 1076 | 1077 | You will need to deploy a model in SageMaker, then provide it as the model name (as `sagemaker-endpoint:my-model-name`). See the [documentation on how to deploy a JumpStart model](https://docs.aws.amazon.com/sagemaker/latest/dg/jumpstart-deploy.html). 1078 | 1079 | All SageMaker endpoint requests require you to specify the `--region-name`, `--request-schema`, and `--response-path` options. The example below presumes that you have deployed a model called `jumpstart-dft-hf-text2text-flan-t5-xl`. 1080 | 1081 | ``` 1082 | %%ai sagemaker-endpoint:jumpstart-dft-hf-text2text-flan-t5-xl --region-name=us-east-1 --request-schema={"text_inputs":""} --response-path=generated_texts.[0] -f code 1083 | Write Python code to print "Hello world" 1084 | ``` 1085 | 1086 | The `--region-name` parameter is set to the [AWS region code](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) where the model is deployed, which in this case is `us-east-1`. 1087 | 1088 | The `--request-schema` parameter is the JSON object the endpoint expects as input, with the prompt being substituted into any value that matches the string literal `""`. For example, the request schema `{"text_inputs":""}` will submit a JSON object with the prompt stored under the `text_inputs` key. 1089 | 1090 | The `--response-path` option is a [JSONPath](https://goessner.net/articles/JsonPath/index.html) string that retrieves the language model's output from the endpoint's JSON response. For example, if your endpoint returns an object with the schema `{"generated_texts":[""]}`, its response path is `generated_texts.[0]`. 1091 | 1092 | 1093 | ## Configuration 1094 | 1095 | You can specify an allowlist, to only allow only a certain list of providers, or 1096 | a blocklist, to block some providers. 1097 | 1098 | ### Configuring default models and API keys 1099 | 1100 | This configuration allows for setting a default language and embedding models, and their corresponding API keys. 1101 | These values are offered as a starting point for users, so they don't have to select the models and API keys, however, 1102 | the selections they make in the settings panel will take precedence over these values. 1103 | 1104 | Specify default language model 1105 | ```bash 1106 | jupyter lab --AiExtension.initial_language_model=bedrock-chat:anthropic.claude-v2 1107 | ``` 1108 | 1109 | Specify default embedding model 1110 | ```bash 1111 | jupyter lab --AiExtension.default_embeddings_model=bedrock:amazon.titan-embed-text-v1 1112 | ``` 1113 | 1114 | Specify default completions model 1115 | ```bash 1116 | jupyter lab --AiExtension.default_completions_model=bedrock-chat:anthropic.claude-v2 1117 | ``` 1118 | 1119 | Specify default API keys 1120 | ```bash 1121 | jupyter lab --AiExtension.default_api_keys={'OPENAI_API_KEY': 'sk-abcd'} 1122 | ``` 1123 | 1124 | 1125 | ### Blocklisting providers 1126 | 1127 | This configuration allows for blocking specific providers in the settings panel. 1128 | This list takes precedence over the allowlist in the next section. 1129 | 1130 | ``` 1131 | jupyter lab --AiExtension.blocked_providers=openai 1132 | ``` 1133 | 1134 | To block more than one provider in the block-list, repeat the runtime 1135 | configuration. 1136 | 1137 | ``` 1138 | jupyter lab --AiExtension.blocked_providers=openai --AiExtension.blocked_providers=ai21 1139 | ``` 1140 | 1141 | ### Allowlisting providers 1142 | 1143 | This configuration allows for filtering the list of providers in the settings 1144 | panel to only an allowlisted set of providers. 1145 | 1146 | ``` 1147 | jupyter lab --AiExtension.allowed_providers=openai 1148 | ``` 1149 | 1150 | To allow more than one provider in the allowlist, repeat the runtime 1151 | configuration. 1152 | 1153 | ``` 1154 | jupyter lab --AiExtension.allowed_providers=openai --AiExtension.allowed_providers=ai21 1155 | ``` 1156 | 1157 | ### Chat memory size 1158 | 1159 | This configuration allows for setting the number of chat exchanges the model 1160 | uses as context when generating a response. 1161 | 1162 | One chat exchange corresponds to a user query message and its AI response, which counts as two messages. 1163 | k denotes one chat exchange, i.e., two messages. 1164 | The default value of k is 2, which corresponds to 4 messages. 1165 | 1166 | For example, if we want the default memory to be 4 exchanges, then use the following command line invocation when starting Jupyter Lab: 1167 | 1168 | ``` 1169 | jupyter lab --AiExtension.default_max_chat_history=4 1170 | ``` 1171 | 1172 | ### Model parameters 1173 | 1174 | This configuration allows specifying arbitrary parameters that are unpacked and 1175 | passed to the provider class. This is useful for passing parameters such as 1176 | model tuning that affect the response generation by the model. This is also an 1177 | appropriate place to pass in custom attributes required by certain 1178 | providers/models. 1179 | 1180 | The accepted value is a dictionary, with top level keys as the model id 1181 | (provider:model_id), and value should be any arbitrary dictionary which is 1182 | unpacked and passed as-is to the provider class. 1183 | 1184 | #### Configuring as a startup option 1185 | 1186 | In this sample, the `bedrock` provider will be created with the value for 1187 | `model_kwargs` when `ai21.j2-mid-v1` model is selected. 1188 | 1189 | ```bash 1190 | jupyter lab --AiExtension.model_parameters bedrock:ai21.j2-mid-v1='{"model_kwargs":{"maxTokens":200}}' 1191 | ``` 1192 | 1193 | Note the usage of single quotes surrounding the dictionary to escape the double 1194 | quotes. This is required in some shells. The above will result in the following 1195 | LLM class to be generated. 1196 | 1197 | ```python 1198 | BedrockProvider(model_kwargs={"maxTokens":200}, ...) 1199 | ``` 1200 | 1201 | Here is another example, where `anthropic` provider will be created with the 1202 | values for `max_tokens` and `temperature`, when `claude-2` model is selected. 1203 | 1204 | 1205 | ```bash 1206 | jupyter lab --AiExtension.model_parameters anthropic:claude-2='{"max_tokens":1024,"temperature":0.9}' 1207 | ``` 1208 | 1209 | The above will result in the following LLM class to be generated. 1210 | 1211 | ```python 1212 | AnthropicProvider(max_tokens=1024, temperature=0.9, ...) 1213 | ``` 1214 | 1215 | To pass multiple sets of model parameters for multiple models in the 1216 | command-line, you can append them as additional arguments to 1217 | `--AiExtension.model_parameters`, as shown below. 1218 | 1219 | ```bash 1220 | jupyter lab \ 1221 | --AiExtension.model_parameters bedrock:ai21.j2-mid-v1='{"model_kwargs":{"maxTokens":200}}' \ 1222 | --AiExtension.model_parameters anthropic:claude-2='{"max_tokens":1024,"temperature":0.9}' 1223 | ``` 1224 | 1225 | However, for more complex configuration, we highly recommend that you specify 1226 | this in a dedicated configuration file. We will describe how to do so in the 1227 | following section. 1228 | 1229 | #### Configuring as a config file 1230 | 1231 | This configuration can also be specified in a config file in json format. 1232 | 1233 | Here is an example for configuring the `bedrock` provider for `ai21.j2-mid-v1` 1234 | model. 1235 | 1236 | ```json 1237 | { 1238 | "AiExtension": { 1239 | "model_parameters": { 1240 | "bedrock:ai21.j2-mid-v1": { 1241 | "model_kwargs": { 1242 | "maxTokens": 200 1243 | } 1244 | } 1245 | } 1246 | } 1247 | } 1248 | ``` 1249 | 1250 | There are several ways to specify JupyterLab to pick this config. 1251 | 1252 | The first option is to save this config in a file and specifying the filepath at startup using the `--config` or `-c` option. 1253 | 1254 | ```bash 1255 | jupyter lab --config 1256 | ``` 1257 | 1258 | The second option is to drop it in a location that JupyterLab scans for configuration files. 1259 | The file should be named `jupyter_jupyter_ai_config.json` in this case. You can find these paths by running `jupyter --paths` 1260 | command, and picking one of the paths from the `config` section. 1261 | 1262 | Here is an example of running the `jupyter --paths` command. 1263 | 1264 | ```bash 1265 | (jupyter-ai-lab4) ➜ jupyter --paths 1266 | config: 1267 | /opt/anaconda3/envs/jupyter-ai-lab4/etc/jupyter 1268 | /Users/3coins/.jupyter 1269 | /Users/3coins/.local/etc/jupyter 1270 | /usr/3coins/etc/jupyter 1271 | /etc/jupyter 1272 | data: 1273 | /opt/anaconda3/envs/jupyter-ai-lab4/share/jupyter 1274 | /Users/3coins/Library/Jupyter 1275 | /Users/3coins/.local/share/jupyter 1276 | /usr/local/share/jupyter 1277 | /usr/share/jupyter 1278 | runtime: 1279 | /Users/3coins/Library/Jupyter/runtime 1280 | ``` 1281 | --------------------------------------------------------------------------------