├── test ├── __init__.py ├── tests │ ├── __init__.py │ ├── notebooks │ │ ├── PackageWithC │ │ │ ├── Sources │ │ │ │ ├── PackageWithC1 │ │ │ │ │ ├── include │ │ │ │ │ │ └── sillyfunction1.h │ │ │ │ │ └── sillyfunction1.c │ │ │ │ └── PackageWithC2 │ │ │ │ │ ├── include │ │ │ │ │ └── sillyfunction2.h │ │ │ │ │ └── sillyfunction2.c │ │ │ └── Package.swift │ │ ├── SimplePackage │ │ │ ├── Sources │ │ │ │ └── SimplePackage │ │ │ │ │ └── SimplePackage.swift │ │ │ └── Package.swift │ │ ├── install_package.ipynb │ │ ├── install_package_with_user_location.ipynb │ │ ├── simple_successful.ipynb │ │ ├── install_package_with_c.ipynb │ │ ├── intentional_compile_error.ipynb │ │ └── intentional_runtime_error.ipynb │ ├── tutorial_notebook_tests.py │ ├── simple_notebook_tests.py │ └── kernel_tests.py ├── fast_test.py ├── test.py ├── all_test_docker.py ├── all_test_local.py └── notebook_tester.py ├── .gitignore ├── .dockerignore ├── requirements.txt ├── screenshots ├── display_pandas.png └── display_matplotlib.png ├── docker ├── requirements_py_graphics.txt ├── run_jupyter.sh ├── requirements.txt └── Dockerfile ├── CONTRIBUTING ├── swift_shell └── __init__.py ├── parent_kernel.py ├── EnableIPythonDisplay.swift ├── KernelCommunicator.swift ├── register.py ├── LICENSE ├── README.md └── swift_kernel.py /test/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /test/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | venv 2 | 3 | **/*.pyc 4 | -------------------------------------------------------------------------------- /.dockerignore: -------------------------------------------------------------------------------- 1 | .git 2 | venv 3 | **/*.pyc 4 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | jupyter 2 | pandas 3 | matplotlib 4 | numpy 5 | -------------------------------------------------------------------------------- /test/tests/notebooks/PackageWithC/Sources/PackageWithC1/include/sillyfunction1.h: -------------------------------------------------------------------------------- 1 | int sillyfunction1(); 2 | -------------------------------------------------------------------------------- /test/tests/notebooks/PackageWithC/Sources/PackageWithC2/include/sillyfunction2.h: -------------------------------------------------------------------------------- 1 | int sillyfunction2(); 2 | -------------------------------------------------------------------------------- /screenshots/display_pandas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/0101011/swift-jupyter/master/screenshots/display_pandas.png -------------------------------------------------------------------------------- /test/tests/notebooks/PackageWithC/Sources/PackageWithC1/sillyfunction1.c: -------------------------------------------------------------------------------- 1 | int sillyfunction1() { 2 | return 42; 3 | } 4 | -------------------------------------------------------------------------------- /screenshots/display_matplotlib.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/0101011/swift-jupyter/master/screenshots/display_matplotlib.png -------------------------------------------------------------------------------- /test/tests/notebooks/SimplePackage/Sources/SimplePackage/SimplePackage.swift: -------------------------------------------------------------------------------- 1 | public let publicIntThatIsInSimplePackage: Int = 42 2 | -------------------------------------------------------------------------------- /test/tests/notebooks/PackageWithC/Sources/PackageWithC2/sillyfunction2.c: -------------------------------------------------------------------------------- 1 | int sillyfunction2() { 2 | return MACRO_DEFINED_IN_COMPILER_FLAG; 3 | } 4 | -------------------------------------------------------------------------------- /docker/requirements_py_graphics.txt: -------------------------------------------------------------------------------- 1 | # Not required to run the kernel, but required for inline graphics. 2 | ipykernel 3 | matplotlib 4 | numpy 5 | pandas 6 | -------------------------------------------------------------------------------- /test/fast_test.py: -------------------------------------------------------------------------------- 1 | """Runs fast tests.""" 2 | 3 | import unittest 4 | 5 | from tests.kernel_tests import SwiftKernelTests, OwnKernelTests 6 | from tests.simple_notebook_tests import * 7 | 8 | 9 | if __name__ == '__main__': 10 | unittest.main() 11 | -------------------------------------------------------------------------------- /test/test.py: -------------------------------------------------------------------------------- 1 | """Copy of "all_test_docker.py", for backwards-compatibility with CI scripts 2 | that call "test.py". 3 | 4 | TODO: Delete this after updating CI scripts. 5 | """ 6 | 7 | import unittest 8 | 9 | from tests.kernel_tests import * 10 | from tests.simple_notebook_tests import * 11 | from tests.tutorial_notebook_tests import * 12 | 13 | 14 | if __name__ == '__main__': 15 | unittest.main() 16 | -------------------------------------------------------------------------------- /test/tests/notebooks/SimplePackage/Package.swift: -------------------------------------------------------------------------------- 1 | // swift-tools-version:4.2 2 | 3 | import PackageDescription 4 | 5 | let package = Package( 6 | name: "SimplePackage", 7 | products: [ 8 | .library(name: "SimplePackage", targets: ["SimplePackage"]), 9 | ], 10 | dependencies: [], 11 | targets: [ 12 | .target( 13 | name: "SimplePackage", 14 | dependencies: []), 15 | ] 16 | ) 17 | -------------------------------------------------------------------------------- /test/all_test_docker.py: -------------------------------------------------------------------------------- 1 | """Runs all tests that work in the docker image. 2 | 3 | Specifically, this includes the SwiftKernelTestsPython27 test that requires a 4 | special kernel named 'swift-with-python-2.7'. 5 | """ 6 | 7 | import unittest 8 | 9 | from tests.kernel_tests import * 10 | from tests.simple_notebook_tests import * 11 | from tests.tutorial_notebook_tests import * 12 | 13 | 14 | if __name__ == '__main__': 15 | unittest.main() 16 | -------------------------------------------------------------------------------- /test/all_test_local.py: -------------------------------------------------------------------------------- 1 | """Runs all tests that work locally. 2 | 3 | Specifically, this excludes the SwiftKernelTestsPython27 test that requires a 4 | special kernel named 'swift-with-python-2.7'. 5 | """ 6 | 7 | import unittest 8 | 9 | from tests.kernel_tests import SwiftKernelTests, OwnKernelTests 10 | from tests.simple_notebook_tests import * 11 | from tests.tutorial_notebook_tests import * 12 | 13 | 14 | if __name__ == '__main__': 15 | unittest.main() 16 | -------------------------------------------------------------------------------- /test/tests/notebooks/PackageWithC/Package.swift: -------------------------------------------------------------------------------- 1 | // swift-tools-version:4.2 2 | 3 | import PackageDescription 4 | 5 | let package = Package( 6 | name: "PackageWithC", 7 | products: [ 8 | .library(name: "PackageWithC", targets: ["PackageWithC1", "PackageWithC2"]), 9 | ], 10 | dependencies: [], 11 | targets: [ 12 | .target( 13 | name: "PackageWithC1", 14 | dependencies: []), 15 | .target( 16 | name: "PackageWithC2", 17 | dependencies: []), 18 | ] 19 | ) 20 | -------------------------------------------------------------------------------- /docker/run_jupyter.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # Copyright 2015 The TensorFlow Authors. All Rights Reserved. 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | # ============================================================================== 16 | 17 | 18 | python2 -m jupyter notebook "$@" 19 | -------------------------------------------------------------------------------- /test/tests/notebooks/install_package.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%install '.package(path: \"$cwd/SimplePackage\")' SimplePackage" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import SimplePackage" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "print(publicIntThatIsInSimplePackage)" 28 | ] 29 | } 30 | ], 31 | "metadata": { 32 | "kernelspec": { 33 | "display_name": "Swift", 34 | "language": "swift", 35 | "name": "swift" 36 | } 37 | }, 38 | "nbformat": 4, 39 | "nbformat_minor": 2 40 | } 41 | -------------------------------------------------------------------------------- /test/tests/notebooks/install_package_with_user_location.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%install-location $cwd/swift-modules\n", 10 | "%install '.package(path: \"$cwd/SimplePackage\")' SimplePackage" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "import SimplePackage" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": {}, 26 | "outputs": [], 27 | "source": [ 28 | "print(publicIntThatIsInSimplePackage)" 29 | ] 30 | } 31 | ], 32 | "metadata": { 33 | "kernelspec": { 34 | "display_name": "Swift", 35 | "language": "swift", 36 | "name": "swift" 37 | }, 38 | "language_info": { 39 | "file_extension": ".swift", 40 | "mimetype": "text/x-swift", 41 | "name": "swift", 42 | "version": "" 43 | } 44 | }, 45 | "nbformat": 4, 46 | "nbformat_minor": 2 47 | } 48 | -------------------------------------------------------------------------------- /docker/requirements.txt: -------------------------------------------------------------------------------- 1 | # These requirements are pinned to versions that Colab uses, so that our tests 2 | # are more likely to catch problems that show up in the Colab environment. 3 | backcall==0.1.0 4 | bleach==3.1.0 5 | decorator==4.3.2 6 | defusedxml==0.5.0 7 | entrypoints==0.3 8 | ipykernel==4.6.1 9 | ipython==5.5.0 10 | ipython-genutils==0.2.0 11 | ipywidgets==7.4.2 12 | jedi==0.13.2 13 | Jinja2==2.10 14 | jsonschema==2.6.0 15 | jupyter==1.0.0 16 | jupyter-client==5.2.4 17 | jupyter-console==6.0.0 18 | jupyter-core==4.4.0 19 | jupyter-kernel-test==0.3 20 | MarkupSafe==1.1.0 21 | mistune==0.8.4 22 | nbconvert==5.4.0 23 | nbformat==4.4.0 24 | nose==1.3.7 25 | notebook==5.2.2 26 | pandocfilters==1.4.2 27 | pexpect==4.6.0 28 | pickleshare==0.7.5 29 | prometheus-client==0.5.0 30 | prompt-toolkit==1.0.15 31 | ptyprocess==0.6.0 32 | Pygments==2.1.3 33 | python-dateutil==2.5.3 34 | pyzmq==17.0.0 35 | qtconsole==4.4.3 36 | Send2Trash==1.5.0 37 | six==1.11.0 38 | terminado==0.8.1 39 | testpath==0.4.2 40 | tornado==4.5.3 41 | traitlets==4.3.2 42 | wcwidth==0.1.7 43 | webencodings==0.5.1 44 | widgetsnbextension==3.4.2 45 | -------------------------------------------------------------------------------- /test/tests/tutorial_notebook_tests.py: -------------------------------------------------------------------------------- 1 | """Checks that tutorial notebooks behave as expected. 2 | """ 3 | 4 | import unittest 5 | import os 6 | import shutil 7 | import tempfile 8 | 9 | from notebook_tester import NotebookTestRunner 10 | 11 | 12 | class TutorialNotebookTests(unittest.TestCase): 13 | @classmethod 14 | def setUpClass(cls): 15 | cls.tmp_dir = tempfile.mkdtemp() 16 | git_url = 'https://github.com/tensorflow/swift.git' 17 | os.system('git clone %s %s -b nightly-notebooks' % (git_url, cls.tmp_dir)) 18 | 19 | @classmethod 20 | def tearDownClass(cls): 21 | shutil.rmtree(cls.tmp_dir) 22 | 23 | def test_iris(self): 24 | notebook = os.path.join(self.tmp_dir, 'docs', 'site', 'tutorials', 25 | 'model_training_walkthrough.ipynb') 26 | runner = NotebookTestRunner(notebook, verbose=False) 27 | runner.run() 28 | self.assertEqual([], runner.unexpected_errors) 29 | all_stdout = '\n\n'.join(runner.stdout) 30 | self.assertIn('Epoch 100:', all_stdout) 31 | self.assertIn('Example 2 prediction:', all_stdout) 32 | -------------------------------------------------------------------------------- /CONTRIBUTING: -------------------------------------------------------------------------------- 1 | # How to Contribute 2 | 3 | We'd love to accept your patches and contributions to this project. There are 4 | just a few small guidelines you need to follow. 5 | 6 | ## Contributor License Agreement 7 | 8 | Contributions to this project must be accompanied by a Contributor License 9 | Agreement. You (or your employer) retain the copyright to your contribution; 10 | this simply gives us permission to use and redistribute your contributions as 11 | part of the project. Head over to to see 12 | your current agreements on file or to sign a new one. 13 | 14 | You generally only need to submit a CLA once, so if you've already submitted one 15 | (even if it was for a different project), you probably don't need to do it 16 | again. 17 | 18 | ## Code reviews 19 | 20 | All submissions, including submissions by project members, require review. We 21 | use GitHub pull requests for this purpose. Consult 22 | [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more 23 | information on using pull requests. 24 | 25 | ## Community Guidelines 26 | 27 | This project follows [Google's Open Source Community 28 | Guidelines](https://opensource.google.com/conduct/). 29 | 30 | -------------------------------------------------------------------------------- /test/tests/notebooks/simple_successful.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "let x = 1" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 2, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "let y = 2" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 3, 24 | "metadata": {}, 25 | "outputs": [ 26 | { 27 | "name": "stdout", 28 | "output_type": "stream", 29 | "text": [ 30 | "Hello World: 3\r\n" 31 | ] 32 | } 33 | ], 34 | "source": [ 35 | "print(\"Hello World: \\(x+y)\")" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": null, 41 | "metadata": {}, 42 | "outputs": [], 43 | "source": [] 44 | } 45 | ], 46 | "metadata": { 47 | "kernelspec": { 48 | "display_name": "Swift", 49 | "language": "swift", 50 | "name": "swift" 51 | }, 52 | "language_info": { 53 | "file_extension": ".swift", 54 | "mimetype": "text/x-swift", 55 | "name": "swift", 56 | "version": "" 57 | } 58 | }, 59 | "nbformat": 4, 60 | "nbformat_minor": 2 61 | } 62 | -------------------------------------------------------------------------------- /test/tests/notebooks/install_package_with_c.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%install-swiftpm-flags -Xcc -DMACRO_DEFINED_IN_COMPILER_FLAG=1337\n", 10 | "%install '.package(path: \"$cwd/PackageWithC\")' PackageWithC" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "import PackageWithC1\n", 20 | "import PackageWithC2" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": null, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "print(sillyfunction1())" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "print(sillyfunction2())" 39 | ] 40 | } 41 | ], 42 | "metadata": { 43 | "kernelspec": { 44 | "display_name": "Swift", 45 | "language": "swift", 46 | "name": "swift" 47 | }, 48 | "language_info": { 49 | "file_extension": ".swift", 50 | "mimetype": "text/x-swift", 51 | "name": "swift", 52 | "version": "" 53 | } 54 | }, 55 | "nbformat": 4, 56 | "nbformat_minor": 2 57 | } 58 | -------------------------------------------------------------------------------- /test/tests/notebooks/intentional_compile_error.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stdout", 10 | "output_type": "stream", 11 | "text": [ 12 | "Hello World\r\n" 13 | ] 14 | } 15 | ], 16 | "source": [ 17 | "print(\"Hello World\")" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [ 25 | { 26 | "ename": "", 27 | "evalue": "", 28 | "output_type": "error", 29 | "traceback": [ 30 | "error: :1:1: error: use of unresolved identifier 'intentionalCompileError'\nintentionalCompileError\n^~~~~~~~~~~~~~~~~~~~~~~\n\n" 31 | ] 32 | } 33 | ], 34 | "source": [ 35 | "intentionalCompileError" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": null, 41 | "metadata": {}, 42 | "outputs": [], 43 | "source": [] 44 | } 45 | ], 46 | "metadata": { 47 | "kernelspec": { 48 | "display_name": "Swift", 49 | "language": "swift", 50 | "name": "swift" 51 | }, 52 | "language_info": { 53 | "file_extension": ".swift", 54 | "mimetype": "text/x-swift", 55 | "name": "swift", 56 | "version": "" 57 | } 58 | }, 59 | "nbformat": 4, 60 | "nbformat_minor": 2 61 | } 62 | -------------------------------------------------------------------------------- /swift_shell/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright 2018 Google LLC 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from ipykernel.zmqshell import ZMQInteractiveShell 16 | from jupyter_client.session import Session 17 | 18 | 19 | class CapturingSocket: 20 | """Simulates a ZMQ socket, saving messages instead of sending them. 21 | 22 | We use this to capture display messages. 23 | """ 24 | 25 | def __init__(self): 26 | self.messages = [] 27 | 28 | def send_multipart(self, msg, **kwargs): 29 | self.messages.append(msg) 30 | 31 | 32 | class SwiftShell(ZMQInteractiveShell): 33 | """An IPython shell, modified to work within Swift.""" 34 | 35 | def enable_gui(self, gui): 36 | """Disable the superclass's `enable_gui`. 37 | 38 | `enable_matplotlib("inline")` calls this method, and the superclass's 39 | method fails because it looks for a kernel that doesn't exist. I don't 40 | know what this method is supposed to do, but everything seems to work 41 | after I disable it. 42 | """ 43 | pass 44 | 45 | 46 | def create_shell(username, session_id, key): 47 | """Instantiates a CapturingSocket and SwiftShell and hooks them up. 48 | 49 | After you call this, the returned CapturingSocket should capture all 50 | IPython display messages. 51 | """ 52 | socket = CapturingSocket() 53 | session = Session(username=username, session=session_id, key=key) 54 | shell = SwiftShell.instance() 55 | shell.display_pub.session = session 56 | shell.display_pub.pub_socket = socket 57 | return (socket, shell) 58 | -------------------------------------------------------------------------------- /parent_kernel.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # 3 | # Copyright 2019 Google LLC 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | # Does some intialization that must happen before "swift_kernel.py" runs, and 18 | # then launches "swift_kernel.py" as a subprocess. 19 | 20 | import os 21 | import signal 22 | import subprocess 23 | import sys 24 | import tempfile 25 | 26 | # The args to launch "swift_kernel.py" are the same as the args we received, 27 | # except with "parent_kernel.py" replaced with "swift_kernel.py". 28 | args = [ 29 | sys.executable, 30 | os.path.join(os.path.dirname(sys.argv[0]), 'swift_kernel.py') 31 | ] 32 | args += sys.argv[1:] 33 | 34 | # Construct a temporary directory for package installation scratchwork. This 35 | # must happen in the parent process because we need to set the 36 | # SWIFT_IMPORT_SEARCH_PATH environment in the child to tell LLDB where module 37 | # files go. 38 | package_install_scratchwork_base = tempfile.mkdtemp() 39 | package_install_scratchwork_base = os.path.join(package_install_scratchwork_base, 'swift-install') 40 | swift_import_search_path = os.path.join(package_install_scratchwork_base, 41 | 'modules') 42 | 43 | # Launch "swift_kernel.py". 44 | process = subprocess.Popen( 45 | args, env=dict(os.environ, 46 | SWIFT_IMPORT_SEARCH_PATH=swift_import_search_path)) 47 | 48 | # Forward SIGINT to the subprocess so that it can handle interrupt requests 49 | # from Jupyter. 50 | def handle_sigint(sig, frame): 51 | process.send_signal(signal.SIGINT) 52 | signal.signal(signal.SIGINT, handle_sigint) 53 | 54 | process.wait() 55 | -------------------------------------------------------------------------------- /test/tests/notebooks/intentional_runtime_error.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stdout", 10 | "output_type": "stream", 11 | "text": [ 12 | "Hello World\r\n" 13 | ] 14 | } 15 | ], 16 | "source": [ 17 | "print(\"Hello World\")" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": {}, 24 | "outputs": [ 25 | { 26 | "name": "stdout", 27 | "output_type": "stream", 28 | "text": [ 29 | "Fatal error: intentional runtime error: file , line 1\r\n", 30 | "Current stack trace:\r\n", 31 | "0 libswiftCore.so 0x00007f01af2f9320 _swift_stdlib_reportFatalErrorInFile + 115\r\n", 32 | "1 libswiftCore.so 0x00007f01af240dec + 3079660\r\n", 33 | "2 libswiftCore.so 0x00007f01af240ede + 3079902\r\n", 34 | "3 libswiftCore.so 0x00007f01af088752 + 1275730\r\n", 35 | "4 libswiftCore.so 0x00007f01af20b0c2 + 2859202\r\n", 36 | "5 libswiftCore.so 0x00007f01af087b99 + 1272729\r\n" 37 | ] 38 | }, 39 | { 40 | "ename": "", 41 | "evalue": "", 42 | "output_type": "error", 43 | "traceback": [ 44 | "Current stack trace:", 45 | "\tframe #2: 0x00007f01ba316d8b $__lldb_expr22`main at :1:1" 46 | ] 47 | } 48 | ], 49 | "source": [ 50 | "fatalError(\"intentional runtime error\")" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": null, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [] 59 | } 60 | ], 61 | "metadata": { 62 | "kernelspec": { 63 | "display_name": "Swift", 64 | "language": "swift", 65 | "name": "swift" 66 | }, 67 | "language_info": { 68 | "file_extension": ".swift", 69 | "mimetype": "text/x-swift", 70 | "name": "swift", 71 | "version": "" 72 | } 73 | }, 74 | "nbformat": 4, 75 | "nbformat_minor": 2 76 | } 77 | -------------------------------------------------------------------------------- /test/tests/simple_notebook_tests.py: -------------------------------------------------------------------------------- 1 | """Checks that simple notebooks behave as expected. 2 | """ 3 | 4 | import unittest 5 | import os 6 | 7 | from notebook_tester import ExecuteError 8 | from notebook_tester import NotebookTestRunner 9 | 10 | 11 | THIS_DIR = os.path.dirname(os.path.abspath(__file__)) 12 | NOTEBOOK_DIR = os.path.join(THIS_DIR, 'notebooks') 13 | 14 | 15 | class SimpleNotebookTests(unittest.TestCase): 16 | def test_simple_successful(self): 17 | notebook = os.path.join(NOTEBOOK_DIR, 'simple_successful.ipynb') 18 | runner = NotebookTestRunner(notebook, verbose=False) 19 | runner.run() 20 | self.assertEqual([], runner.unexpected_errors) 21 | self.assertIn('Hello World: 3', runner.stdout[2]) 22 | 23 | def test_intentional_compile_error(self): 24 | notebook = os.path.join(NOTEBOOK_DIR, 'intentional_compile_error.ipynb') 25 | runner = NotebookTestRunner(notebook, verbose=False) 26 | runner.run() 27 | self.assertEqual(1, len(runner.unexpected_errors)) 28 | self.assertIsInstance(runner.unexpected_errors[0]['error'], 29 | ExecuteError) 30 | self.assertEqual(1, runner.unexpected_errors[0]['error'].cell_index) 31 | 32 | def test_intentional_runtime_error(self): 33 | notebook = os.path.join(NOTEBOOK_DIR, 'intentional_runtime_error.ipynb') 34 | runner = NotebookTestRunner(notebook, verbose=False) 35 | runner.run() 36 | self.assertEqual(1, len(runner.unexpected_errors)) 37 | self.assertIsInstance(runner.unexpected_errors[0]['error'], 38 | ExecuteError) 39 | self.assertEqual(1, runner.unexpected_errors[0]['error'].cell_index) 40 | 41 | def test_install_package(self): 42 | notebook = os.path.join(NOTEBOOK_DIR, 'install_package.ipynb') 43 | runner = NotebookTestRunner(notebook, char_step=0, verbose=False) 44 | runner.run() 45 | self.assertIn('Installation complete', runner.stdout[0]) 46 | self.assertIn('42', runner.stdout[2]) 47 | 48 | def test_install_package_with_c(self): 49 | notebook = os.path.join(NOTEBOOK_DIR, 'install_package_with_c.ipynb') 50 | runner = NotebookTestRunner(notebook, char_step=0, verbose=False) 51 | runner.run() 52 | self.assertIn('Installation complete', runner.stdout[0]) 53 | self.assertIn('42', runner.stdout[2]) 54 | self.assertIn('1337', runner.stdout[3]) 55 | -------------------------------------------------------------------------------- /docker/Dockerfile: -------------------------------------------------------------------------------- 1 | # TODO: We should have a job that creates a S4TF base image so that 2 | #we don't have to duplicate the installation everywhere. 3 | FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04 4 | 5 | # Allows the caller to specify the toolchain to use. 6 | ARG swift_tf_url=https://storage.googleapis.com/s4tf-kokoro-artifact-testing/latest/swift-tensorflow-DEVELOPMENT-cuda10.0-cudnn7-ubuntu18.04.tar.gz 7 | 8 | # Install Swift deps. 9 | ENV DEBIAN_FRONTEND=noninteractive 10 | RUN apt-get update && apt-get install -y --no-install-recommends \ 11 | build-essential \ 12 | ca-certificates \ 13 | curl \ 14 | git \ 15 | python \ 16 | python-dev \ 17 | python-pip \ 18 | python-setuptools \ 19 | python-tk \ 20 | python3 \ 21 | python3-pip \ 22 | python3-setuptools \ 23 | clang \ 24 | libcurl4-openssl-dev \ 25 | libicu-dev \ 26 | libpython-dev \ 27 | libpython3-dev \ 28 | libncurses5-dev \ 29 | libxml2 \ 30 | libblocksruntime-dev 31 | 32 | # Upgrade pips 33 | RUN pip2 install --upgrade pip 34 | RUN pip3 install --upgrade pip 35 | 36 | # Install swift-jupyter's dependencies in python3 because we run the kernel in python3. 37 | WORKDIR /swift-jupyter 38 | COPY docker/requirements*.txt ./ 39 | RUN pip3 install -r requirements.txt 40 | 41 | # Install some python libraries that are useful to call from swift. Since 42 | # swift can interoperate with python2 and python3, install them in both. 43 | RUN pip2 install -r requirements_py_graphics.txt 44 | RUN pip3 install -r requirements_py_graphics.txt 45 | 46 | # Copy the kernel into the container 47 | WORKDIR /swift-jupyter 48 | COPY . . 49 | 50 | # Download and extract S4TF 51 | WORKDIR /swift-tensorflow-toolchain 52 | ADD $swift_tf_url swift.tar.gz 53 | RUN mkdir usr \ 54 | && tar -xzf swift.tar.gz --directory=usr --strip-components=1 \ 55 | && rm swift.tar.gz 56 | 57 | # Register the kernel with jupyter 58 | WORKDIR /swift-jupyter 59 | RUN python3 register.py --user --swift-toolchain /swift-tensorflow-toolchain --swift-python-version 2.7 --kernel-name "Swift (with Python 2.7)" && \ 60 | python3 register.py --user --swift-toolchain /swift-tensorflow-toolchain --swift-python-library /usr/lib/x86_64-linux-gnu/libpython3.6m.so --kernel-name "Swift" 61 | 62 | # Configure cuda 63 | RUN echo "/usr/local/cuda-10.0/targets/x86_64-linux/lib/stubs" > /etc/ld.so.conf.d/cuda-10.0-stubs.conf && \ 64 | ldconfig 65 | 66 | # Run jupyter on startup 67 | EXPOSE 8888 68 | RUN mkdir /notebooks 69 | WORKDIR /notebooks 70 | CMD ["/swift-jupyter/docker/run_jupyter.sh", "--allow-root", "--no-browser", "--ip=0.0.0.0", "--port=8888", "--NotebookApp.custom_display_url=http://127.0.0.1:8888"] 71 | -------------------------------------------------------------------------------- /EnableIPythonDisplay.swift: -------------------------------------------------------------------------------- 1 | // Copyright 2018 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | /// Hooks IPython to the KernelCommunicator, so that it can send display 16 | /// messages to Jupyter. 17 | 18 | import Python 19 | 20 | enum IPythonDisplay { 21 | static var socket: PythonObject = Python.None 22 | static var shell: PythonObject = Python.None 23 | 24 | // Tracks whether the Python version that we are interoperating with has a 25 | // "real" bytes type that is an array of bytes, rather than Python2's "fake" 26 | // bytes type that is just an alias of str. 27 | private static var hasRealBytesType: Bool = false 28 | } 29 | 30 | extension IPythonDisplay { 31 | private static func bytes(_ py: PythonObject) -> KernelCommunicator.BytesReference { 32 | // TODO: Replace with a faster implementation that reads bytes directly 33 | // from the python object's memory. 34 | if hasRealBytesType { 35 | let bytes = py.lazy.map { CChar(bitPattern: UInt8($0)!) } 36 | return KernelCommunicator.BytesReference(bytes) 37 | } 38 | let bytes = py.lazy.map { CChar(bitPattern: UInt8(Python.ord($0))!) } 39 | return KernelCommunicator.BytesReference(bytes) 40 | } 41 | 42 | private static func updateParentMessage(to parentMessage: KernelCommunicator.ParentMessage) { 43 | let json = Python.import("json") 44 | IPythonDisplay.shell.set_parent(json.loads(parentMessage.json)) 45 | } 46 | 47 | private static func consumeDisplayMessages() -> [KernelCommunicator.JupyterDisplayMessage] { 48 | let displayMessages = IPythonDisplay.socket.messages.map { 49 | KernelCommunicator.JupyterDisplayMessage(parts: $0.map { bytes($0) }) 50 | } 51 | IPythonDisplay.socket.messages = [] 52 | return displayMessages 53 | } 54 | 55 | static func enable() { 56 | if IPythonDisplay.shell != Python.None { 57 | print("Warning: IPython display already enabled.") 58 | return 59 | } 60 | 61 | hasRealBytesType = Bool(Python.isinstance(PythonObject("t").encode("utf8")[0], Python.int))! 62 | 63 | let swift_shell = Python.import("swift_shell") 64 | let socketAndShell = swift_shell.create_shell( 65 | username: JupyterKernel.communicator.jupyterSession.username, 66 | session_id: JupyterKernel.communicator.jupyterSession.id, 67 | key: PythonObject(JupyterKernel.communicator.jupyterSession.key).encode("utf8")) 68 | IPythonDisplay.socket = socketAndShell[0] 69 | IPythonDisplay.shell = socketAndShell[1] 70 | 71 | JupyterKernel.communicator.handleParentMessage(updateParentMessage) 72 | JupyterKernel.communicator.afterSuccessfulExecution(run: consumeDisplayMessages) 73 | } 74 | } 75 | 76 | extension PythonObject { 77 | func display() { 78 | Python.import("IPython.display").display(pythonObject) 79 | } 80 | } 81 | 82 | IPythonDisplay.enable() 83 | 84 | func display(base64EncodedPNG: String) { 85 | let displayImage = Python.import("IPython.display") 86 | let codecs = Python.import("codecs") 87 | let imageData = codecs.decode(Python.bytes(base64EncodedPNG, encoding: "utf8"), encoding: "base64") 88 | displayImage.Image(data: imageData, format: "png").display() 89 | } 90 | -------------------------------------------------------------------------------- /KernelCommunicator.swift: -------------------------------------------------------------------------------- 1 | // Copyright 2018 Google LLC 2 | // 3 | // Licensed under the Apache License, Version 2.0 (the "License"); 4 | // you may not use this file except in compliance with the License. 5 | // You may obtain a copy of the License at 6 | // 7 | // http://www.apache.org/licenses/LICENSE-2.0 8 | // 9 | // Unless required by applicable law or agreed to in writing, software 10 | // distributed under the License is distributed on an "AS IS" BASIS, 11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | // See the License for the specific language governing permissions and 13 | // limitations under the License. 14 | 15 | /// A struct with functions that the kernel and the code running inside the 16 | /// kernel use to talk to each other. 17 | /// 18 | /// Note that it would be more Jupyter-y for the communication to happen over 19 | /// ZeroMQ. This is not currently possible, because ZeroMQ sends messages 20 | /// asynchronously using IO threads, and LLDB pauses those IO threads, which 21 | /// prevents them from sending the messages. 22 | public struct KernelCommunicator { 23 | private var afterSuccessfulExecutionHandlers: [() -> [JupyterDisplayMessage]] 24 | private var parentMessageHandlers: [(ParentMessage) -> ()] 25 | 26 | public let jupyterSession: JupyterSession 27 | 28 | private var previousDisplayMessages: [JupyterDisplayMessage] = [] 29 | 30 | init(jupyterSession: JupyterSession) { 31 | self.afterSuccessfulExecutionHandlers = [] 32 | self.parentMessageHandlers = [] 33 | self.jupyterSession = jupyterSession 34 | } 35 | 36 | /// Register a handler to run after the kernel successfully executes a cell 37 | /// of user code. The handler may return messages. These messages will be 38 | /// sent to the Jupyter client. 39 | public mutating func afterSuccessfulExecution( 40 | run handler: @escaping () -> [JupyterDisplayMessage]) { 41 | afterSuccessfulExecutionHandlers.append(handler) 42 | } 43 | 44 | /// Register a handler to run when the parent message changes. 45 | public mutating func handleParentMessage(_ handler: @escaping (ParentMessage) -> ()) { 46 | parentMessageHandlers.append(handler) 47 | } 48 | 49 | /// The kernel calls this after successfully executing a cell of user code. 50 | /// Returns an array of messages, where each message is returned as an array 51 | /// of parts, where each part is returned as an `UnsafeBufferPointer` 52 | /// to the memory containing the part's bytes. 53 | public mutating func triggerAfterSuccessfulExecution() -> [[UnsafeBufferPointer]] { 54 | // Keep a reference to the messages, so that their `.unsafeBufferPointer` 55 | // stays valid while the kernel is reading from them. 56 | previousDisplayMessages = afterSuccessfulExecutionHandlers.flatMap { $0() } 57 | return previousDisplayMessages.map { $0.parts.map { $0.unsafeBufferPointer } } 58 | } 59 | 60 | /// The kernel calls this when the parent message changes. 61 | public mutating func updateParentMessage(to parentMessage: ParentMessage) { 62 | for parentMessageHandler in parentMessageHandlers { 63 | parentMessageHandler(parentMessage) 64 | } 65 | } 66 | 67 | /// A single serialized display message for the Jupyter client. 68 | /// Corresponds to a ZeroMQ "multipart message". 69 | public struct JupyterDisplayMessage { 70 | let parts: [BytesReference] 71 | } 72 | 73 | /// A reference to memory containing bytes. 74 | /// 75 | /// As long as there is a strong reference to an instance, that instance's 76 | /// `unsafeBufferPointer` refers to memory containing the bytes passed to 77 | /// that instance's constructor. 78 | /// 79 | /// We use this so that we can give the kernel a memory location that it can 80 | /// read bytes from. 81 | public class BytesReference { 82 | private var bytes: ContiguousArray 83 | 84 | init(_ bytes: S) where S.Element == CChar { 85 | // Construct our own array and copy `bytes` into it, so that no one 86 | // else aliases the underlying memory. 87 | self.bytes = [] 88 | self.bytes.append(contentsOf: bytes) 89 | } 90 | 91 | public var unsafeBufferPointer: UnsafeBufferPointer { 92 | // We have tried very hard to make the pointer stay valid outside the 93 | // closure: 94 | // - No one else aliases the underlying memory. 95 | // - The comment on this class reminds users that the memory may become 96 | // invalid after all references to the BytesReference instance are 97 | // released. 98 | return bytes.withUnsafeBufferPointer { $0 } 99 | } 100 | } 101 | 102 | /// ParentMessage identifies the request that causes things to happen. 103 | /// This lets Jupyter, for example, know which cell to display graphics 104 | /// messages in. 105 | public struct ParentMessage { 106 | let json: String 107 | } 108 | 109 | /// The data necessary to identify and sign outgoing jupyter messages. 110 | public struct JupyterSession { 111 | let id: String 112 | let key: String 113 | let username: String 114 | } 115 | } 116 | -------------------------------------------------------------------------------- /register.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # 3 | # Copyright 2018 Google LLC 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import argparse 18 | import json 19 | import os 20 | import platform 21 | import sys 22 | 23 | from jupyter_client.kernelspec import KernelSpecManager 24 | from IPython.utils.tempdir import TemporaryDirectory 25 | from glob import glob 26 | 27 | kernel_code_name_allowed_chars = "-." 28 | 29 | 30 | def get_kernel_code_name(kernel_name): 31 | """ 32 | Returns a valid kernel code name (like `swift-for-tensorflow`) 33 | from a kernel display name (like `Swift for TensorFlow`). 34 | """ 35 | 36 | kernel_code_name = kernel_name.lower().replace(" ", kernel_code_name_allowed_chars[0]) 37 | kernel_code_name = "".join(list(filter(lambda x: x.isalnum() or x in kernel_code_name_allowed_chars, kernel_code_name))) 38 | return kernel_code_name 39 | 40 | 41 | def linux_lldb_python_lib_subdir(): 42 | return 'lib/python%d.%d/site-packages' % (sys.version_info[0], 43 | sys.version_info[1]) 44 | 45 | 46 | def make_kernel_env(args): 47 | """Returns environment varialbes that tell the kernel where things are.""" 48 | 49 | kernel_env = {} 50 | 51 | if args.swift_toolchain is not None: 52 | # Use a prebuilt Swift toolchain. 53 | if platform.system() == 'Linux': 54 | kernel_env['PYTHONPATH'] = '%s/usr/%s' % ( 55 | args.swift_toolchain, linux_lldb_python_lib_subdir()) 56 | kernel_env['LD_LIBRARY_PATH'] = '%s/usr/lib/swift/linux' % args.swift_toolchain 57 | kernel_env['REPL_SWIFT_PATH'] = '%s/usr/bin/repl_swift' % args.swift_toolchain 58 | kernel_env['SWIFT_BUILD_PATH'] = '%s/usr/bin/swift-build' % args.swift_toolchain 59 | kernel_env['SWIFT_PACKAGE_PATH'] = '%s/usr/bin/swift-package' % args.swift_toolchain 60 | elif platform.system() == 'Darwin': 61 | kernel_env['PYTHONPATH'] = '%s/System/Library/PrivateFrameworks/LLDB.framework/Resources/Python' % args.swift_toolchain 62 | kernel_env['LD_LIBRARY_PATH'] = '%s/usr/lib/swift/macosx' % args.swift_toolchain 63 | kernel_env['REPL_SWIFT_PATH'] = '%s/System/Library/PrivateFrameworks/LLDB.framework/Resources/repl_swift' % args.swift_toolchain 64 | else: 65 | raise Exception('Unknown system %s' % platform.system()) 66 | 67 | elif args.swift_build is not None: 68 | # Use a build dir created by build-script. 69 | 70 | # TODO: Make this work on macos 71 | if platform.system() != 'Linux': 72 | raise Exception('build-script build dir only implemented on Linux') 73 | 74 | swift_build_dir = '%s/swift-linux-x86_64' % args.swift_build 75 | lldb_build_dir = '%s/lldb-linux-x86_64' % args.swift_build 76 | 77 | kernel_env['PYTHONPATH'] = '%s/%s' % (lldb_build_dir, 78 | linux_lldb_python_lib_subdir()) 79 | kernel_env['LD_LIBRARY_PATH'] = '%s/lib/swift/linux' % swift_build_dir 80 | kernel_env['REPL_SWIFT_PATH'] = '%s/bin/repl_swift' % lldb_build_dir 81 | 82 | elif args.xcode_path is not None: 83 | # Use an Xcode provided Swift toolchain. 84 | 85 | if platform.system() != 'Darwin': 86 | raise Exception('Xcode support is only available on Darwin') 87 | 88 | lldb_framework = '%s/Contents/SharedFrameworks/LLDB.framework' % args.xcode_path 89 | xcode_toolchain = '%s/Contents/Developer/Toolchains/XcodeDefault.xctoolchain' % args.xcode_path 90 | 91 | kernel_env['PYTHONPATH'] = '%s/Resources/Python' % lldb_framework 92 | kernel_env['REPL_SWIFT_PATH'] = '%s/Resources/repl_swift' % lldb_framework 93 | kernel_env['LD_LIBRARY_PATH'] = '%s/usr/lib/swift/macosx' % xcode_toolchain 94 | 95 | if args.swift_python_version is not None: 96 | kernel_env['PYTHON_VERSION'] = args.swift_python_version 97 | if args.swift_python_library is not None: 98 | kernel_env['PYTHON_LIBRARY'] = args.swift_python_library 99 | if args.swift_python_use_conda: 100 | libpython = glob(sys.prefix+'/lib/libpython*.so')[0] 101 | kernel_env['PYTHON_LIBRARY'] = libpython 102 | 103 | if args.use_conda_shared_libs: 104 | kernel_env['LD_LIBRARY_PATH'] += ':' + sys.prefix + '/lib' 105 | 106 | return kernel_env 107 | 108 | 109 | def validate_kernel_env(kernel_env): 110 | """Validates that the env vars refer to things that actually exist.""" 111 | 112 | if not os.path.isfile(kernel_env['PYTHONPATH'] + '/lldb/_lldb.so'): 113 | raise Exception('lldb python libs not found at %s' % 114 | kernel_env['PYTHONPATH']) 115 | if not os.path.isfile(kernel_env['REPL_SWIFT_PATH']): 116 | raise Exception('repl_swift binary not found at %s' % 117 | kernel_env['REPL_SWIFT_PATH']) 118 | if 'SWIFT_BUILD_PATH' in kernel_env and \ 119 | not os.path.isfile(kernel_env['SWIFT_BUILD_PATH']): 120 | raise Exception('swift-build binary not found at %s' % 121 | kernel_env['SWIFT_BUILD_PATH']) 122 | if 'SWIFT_PACKAGE_PATH' in kernel_env and \ 123 | not os.path.isfile(kernel_env['SWIFT_PACKAGE_PATH']): 124 | raise Exception('swift-package binary not found at %s' % 125 | kernel_env['SWIFT_PACKAGE_PATH']) 126 | if 'PYTHON_LIBRARY' in kernel_env and \ 127 | not os.path.isfile(kernel_env['PYTHON_LIBRARY']): 128 | raise Exception('python library not found at %s' % 129 | kernel_env['PYTHON_LIBRARY']) 130 | 131 | lib_paths = kernel_env['LD_LIBRARY_PATH'].split(':') 132 | for index, lib_path in enumerate(lib_paths): 133 | if os.path.isdir(lib_path): 134 | continue 135 | # First LD_LIBRARY_PATH should contain the swift toolchain libs. 136 | if index == 0: 137 | raise Exception('swift libs not found at %s' % lib_path) 138 | # Other LD_LIBRARY_PATHs may be appended for other libs. 139 | raise Exception('shared lib dir not found at %s' % lib_path) 140 | 141 | 142 | def main(): 143 | args = parse_args() 144 | kernel_env = make_kernel_env(args) 145 | validate_kernel_env(kernel_env) 146 | 147 | script_dir = os.path.dirname(os.path.realpath(sys.argv[0])) 148 | kernel_json = { 149 | 'argv': [ 150 | sys.executable, 151 | '%s/parent_kernel.py' % script_dir, 152 | '-f', 153 | '{connection_file}', 154 | ], 155 | 'display_name': args.kernel_name, 156 | 'language': 'swift', 157 | 'env': kernel_env, 158 | } 159 | 160 | print('kernel.json:\n%s\n' % json.dumps(kernel_json, indent=2)) 161 | 162 | kernel_code_name = get_kernel_code_name(args.kernel_name) 163 | 164 | with TemporaryDirectory() as td: 165 | os.chmod(td, 0o755) 166 | with open(os.path.join(td, 'kernel.json'), 'w') as f: 167 | json.dump(kernel_json, f, indent=2) 168 | KernelSpecManager().install_kernel_spec( 169 | td, kernel_code_name, user=args.user, prefix=args.prefix) 170 | 171 | print('Registered kernel \'{}\' as \'{}\'!'.format(args.kernel_name, kernel_code_name)) 172 | 173 | 174 | def parse_args(): 175 | parser = argparse.ArgumentParser( 176 | description='Register KernelSpec for Swift Kernel') 177 | 178 | parser.add_argument( 179 | '--kernel-name', 180 | help='Kernel display name', 181 | default='Swift' 182 | ) 183 | 184 | prefix_locations = parser.add_mutually_exclusive_group() 185 | prefix_locations.add_argument( 186 | '--user', 187 | help='Register KernelSpec in user homedirectory', 188 | action='store_true') 189 | prefix_locations.add_argument( 190 | '--sys-prefix', 191 | help='Register KernelSpec in sys.prefix. Useful in conda / virtualenv', 192 | action='store_true', 193 | dest='sys_prefix') 194 | prefix_locations.add_argument( 195 | '--prefix', 196 | help='Register KernelSpec in this prefix', 197 | default=None) 198 | 199 | swift_locations = parser.add_mutually_exclusive_group(required=True) 200 | swift_locations.add_argument( 201 | '--swift-toolchain', 202 | help='Path to a prebuilt swift toolchain') 203 | swift_locations.add_argument( 204 | '--swift-build', 205 | help='Path to build-script build directory, containing swift and lldb') 206 | swift_locations.add_argument( 207 | '--xcode-path', 208 | help='Path to Xcode app bundle') 209 | 210 | python_locations = parser.add_mutually_exclusive_group() 211 | python_locations.add_argument( 212 | '--swift-python-version', 213 | help='direct Swift\'s Python interop library to use this version of ' + 214 | 'Python') 215 | python_locations.add_argument( 216 | '--swift-python-library', 217 | help='direct Swift\'s Python interop library to use this Python ' + 218 | 'library') 219 | python_locations.add_argument( 220 | '--swift-python-use-conda', 221 | action='store_true', 222 | help='direct Swift\'s Python interop library to use the Python ' 223 | 'from the current conda environment') 224 | 225 | parser.add_argument( 226 | '--use-conda-shared-libs', 227 | action='store_true', 228 | help='set LD_LIBRARY_PATH to search for shared libs installed in ' 229 | 'the current conda environment') 230 | 231 | args = parser.parse_args() 232 | if args.sys_prefix: 233 | args.prefix = sys.prefix 234 | if args.swift_toolchain is not None: 235 | args.swift_toolchain = os.path.realpath(args.swift_toolchain) 236 | if args.swift_build is not None: 237 | args.swift_build = os.path.realpath(args.swift_build) 238 | if args.xcode_path is not None: 239 | args.xcode_path = os.path.realpath(args.xcode_path) 240 | return args 241 | 242 | 243 | if __name__ == '__main__': 244 | main() 245 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Swift-Jupyter 2 | 3 | This is a Jupyter Kernel for Swift, intended to make it possible to use Jupyter 4 | with the [Swift for TensorFlow](https://github.com/tensorflow/swift) project. 5 | 6 | # Installation Instructions 7 | 8 | ## Option 1: Using a Swift for TensorFlow toolchain and Virtualenv 9 | 10 | ### Requirements 11 | 12 | Operating system: 13 | 14 | * Ubuntu 18.04 (64-bit); OR 15 | * other operating systems may work, but you will have to build Swift from 16 | sources. 17 | 18 | Dependencies: 19 | 20 | * Python 3 (Ubuntu 18.04 package name: `python3`) 21 | * Python 3 Virtualenv (Ubuntu 18.04 package name: `python3-venv`) 22 | 23 | ### Installation 24 | 25 | swift-jupyter requires a Swift toolchain with LLDB Python3 support. Currently, the only prebuilt toolchains with LLDB Python3 support are the [Swift for TensorFlow Ubuntu 18.04 Nightly Builds](https://github.com/tensorflow/swift/blob/master/Installation.md#pre-built-packages). Alternatively, you can build a toolchain from sources (see the section below for instructions). 26 | 27 | Extract the Swift toolchain somewhere. 28 | 29 | Create a virtualenv, install the requirements in it, and register the kernel in 30 | it: 31 | 32 | ```bash 33 | python3 -m venv venv 34 | . venv/bin/activate 35 | pip install -r requirements.txt 36 | python register.py --sys-prefix --swift-toolchain 37 | ``` 38 | 39 | Finally, run Jupyter: 40 | 41 | ```bash 42 | . venv/bin/activate 43 | jupyter notebook 44 | ``` 45 | 46 | You should be able to create Swift notebooks. Installation is done! 47 | 48 | ## Option 2: Using a Swift for TensorFlow toolchain and Conda 49 | 50 | ### Requirements 51 | 52 | Operating system: 53 | 54 | * Ubuntu 18.04 (64-bit); OR 55 | * other operating systems may work, but you will have to build Swift from 56 | sources. 57 | 58 | ### Installation 59 | 60 | #### 1. Get toolchain 61 | 62 | swift-jupyter requires a Swift toolchain with LLDB Python3 support. Currently, the only prebuilt toolchains with LLDB Python3 support are the [Swift for TensorFlow Ubuntu 18.04 Nightly Builds](https://github.com/tensorflow/swift/blob/master/Installation.md#pre-built-packages). Alternatively, you can build a toolchain from sources (see the section below for instructions). 63 | 64 | Extract the Swift toolchain somewhere. 65 | 66 | Important note about CUDA/CUDNN: If you are using a CUDA toolchain, then you should install CUDA and CUDNN on your system 67 | without using Conda, because Conda's CUDNN is too old to work with the Swift toolchain's TensorFlow. (As of 2019-04-08, 68 | Swift for TensorFlow requires CUDNN 7.5, but Conda only has CUDNN 7.3). 69 | 70 | #### 2. Initialize environment 71 | 72 | Create a Conda environment and install some packages in it: 73 | 74 | ```bash 75 | conda create -n swift-tensorflow python==3.6 76 | conda activate swift-tensorflow 77 | conda install jupyter numpy matplotlib 78 | ``` 79 | 80 | #### 3. Register kernel 81 | 82 | Register the Swift kernel with Jupyter: 83 | 84 | ```bash 85 | python register.py --sys-prefix --swift-python-use-conda --use-conda-shared-libs \ 86 | --swift-toolchain 87 | ``` 88 | 89 | Finally, run Jupyter: 90 | 91 | ```bash 92 | jupyter notebook 93 | ``` 94 | 95 | You should be able to create Swift notebooks. Installation is done! 96 | 97 | ## Option 3: Using the Docker Container 98 | 99 | This repository also includes a dockerfile which can be used to run a Jupyter Notebook instance which includes this Swift kernel. To build the container, the following command may be used: 100 | 101 | ```bash 102 | # from inside the directory of this repository 103 | docker build -f docker/Dockerfile -t swift-jupyter . 104 | ``` 105 | 106 | The resulting container comes with the latest Swift for TensorFlow toolchain installed, along with Jupyter and the Swift kernel contained in this repository. 107 | 108 | This container can now be run with the following command: 109 | 110 | ```bash 111 | docker run -p 8888:8888 --cap-add SYS_PTRACE -v /my/host/notebooks:/notebooks swift-jupyter 112 | ``` 113 | 114 | The functions of these parameters are: 115 | 116 | - `-p 8888:8888` exposes the port on which Jupyter is running to the host. 117 | 118 | - `--cap-add SYS_PTRACE` adjusts the privileges with which this container is run, which is required for the Swift REPL. 119 | 120 | - `-v :/notebooks` bind mounts a host directory as a volume where notebooks created in the container will be stored. If this command is omitted, any notebooks created using the container will not be persisted when the container is stopped. 121 | 122 | ## (optional) Building toolchain with LLDB Python3 support 123 | 124 | Follow the 125 | [Building Swift for TensorFlow](https://github.com/apple/swift/tree/tensorflow#building-swift-for-tensorflow) 126 | instructions, with some modifications: 127 | 128 | * Also install the Python 3 development headers. (For Ubuntu 18.04, 129 | `sudo apt-get install libpython3-dev`). The LLDB build will automatically 130 | find these and build with Python 3 support. 131 | * Instead of running `utils/build-script`, run 132 | `SWIFT_PACKAGE=tensorflow_linux,no_test ./swift/utils/build-toolchain local.swift` 133 | or `SWIFT_PACKAGE=tensorflow_linux ./swift/utils/build-toolchain local.swift,gpu,no_test` 134 | (depending on whether you want to build tensorflow with GPU support). 135 | 136 | This will create a tar file containing the full toolchain. You can now proceed 137 | with the installation instructions from the previous section. 138 | 139 | # Usage Instructions 140 | 141 | ## Rich output 142 | 143 | You can call Python libraries using [Swift's Python interop] to display rich 144 | output in your Swift notebooks. (Eventually, we'd like to support Swift 145 | libraries that produce rich output too!) 146 | 147 | Prerequisites: 148 | 149 | * You must use a Swift toolchain that has Python interop. As of February 2019, 150 | only the Swift for TensorFlow toolchains have Python interop. 151 | 152 | After taking care of the prerequisites, run 153 | `%include "EnableIPythonDisplay.swift"` in your Swift notebook. Now you should 154 | be able to display rich output! For example: 155 | 156 | ```swift 157 | let np = Python.import("numpy") 158 | let plt = Python.import("matplotlib.pyplot") 159 | IPythonDisplay.shell.enable_matplotlib("inline") 160 | ``` 161 | 162 | ```swift 163 | let time = np.arange(0, 10, 0.01) 164 | let amplitude = np.exp(-0.1 * time) 165 | let position = amplitude * np.sin(3 * time) 166 | 167 | plt.figure(figsize: [15, 10]) 168 | 169 | plt.plot(time, position) 170 | plt.plot(time, amplitude) 171 | plt.plot(time, -amplitude) 172 | 173 | plt.xlabel("time (s)") 174 | plt.ylabel("position (m)") 175 | plt.title("Oscillations") 176 | 177 | plt.show() 178 | ``` 179 | 180 | ![Screenshot of running the above two snippets of code in Jupyter](./screenshots/display_matplotlib.png) 181 | 182 | ```swift 183 | let display = Python.import("IPython.display") 184 | let pd = Python.import("pandas") 185 | ``` 186 | 187 | ```swift 188 | display.display(pd.DataFrame.from_records([["col 1": 3, "col 2": 5], ["col 1": 8, "col 2": 2]])) 189 | ``` 190 | 191 | ![Screenshot of running the above two snippets of code in Jupyter](./screenshots/display_pandas.png) 192 | 193 | [Swift's Python interop]: https://github.com/tensorflow/swift/blob/master/docs/PythonInteroperability.md 194 | 195 | ## %install directives 196 | 197 | `%install` directives let you install SwiftPM packages so that your notebook 198 | can import them: 199 | 200 | ```swift 201 | // Specify SwiftPM flags to use during package installation. 202 | %install-swiftpm-flags -c release 203 | 204 | // Install the DeckOfPlayingCards package from GitHub. 205 | %install '.package(url: "https://github.com/NSHipster/DeckOfPlayingCards", from: "4.0.0")' DeckOfPlayingCards 206 | 207 | // Install the SimplePackage package that's in the kernel's working directory. 208 | %install '.package(path: "$cwd/SimplePackage")' SimplePackage 209 | ``` 210 | 211 | The first argument to `%install` is a [SwiftPM package dependency specification](https://github.com/apple/swift-package-manager/blob/master/Documentation/PackageDescriptionV4.md#dependencies). 212 | The next argument(s) to `%install` are the products that you want to install from the package. 213 | 214 | `%install` directives currently have some limitations: 215 | 216 | * You must install all your packages in the first cell that you execute. (It 217 | will refuse to install packages, and print out an error message explaining 218 | why, if you try to install packages in later cells.) 219 | * `%install-swiftpm-flags` apply to all packages that you are installing; there 220 | is no way to specify different flags for different packages. 221 | * Packages that use system libraries may require you to manually specify some 222 | header search paths. See the `%install-extra-include-command` section below. 223 | 224 | ### Troubleshooting %installs 225 | 226 | If you get "expression failed to parse, unknown error" when you try to import a 227 | package that you installed, there is a way to get a more detailed error 228 | message. 229 | 230 | The cell with the "%install" directives has something like "Working in: 231 | /tmp/xyzxyzxyzxyz/swift-install" in its output. There is a binary 232 | `usr/bin/swift` where you extracted the toolchain. Start the binary as follows: 233 | 234 | ``` 235 | SWIFT_IMPORT_SEARCH_PATH=/tmp/xyzxyzxyzxyz/swift-install/modules /usr/bin/swift 236 | ``` 237 | 238 | This gives you an interactive Swift REPL. In the REPL, do: 239 | ``` 240 | import Glibc 241 | dlopen("/tmp/xyzxyzxyzxyz/swift-install/package/.build/debug/libjupyterInstalledPackages.so", RTLD_NOW) 242 | 243 | import TheModuleThatYouHaveTriedToInstall 244 | ``` 245 | 246 | This should give you a useful error message. If the error message says that 247 | some header files can't be found, see the section below about 248 | `%install-extra-include-command`. 249 | 250 | ### %install-extra-include-command 251 | 252 | You can specify extra header files to be put on the header search path. Add a 253 | directive `%install-extra-include-command`, followed by a shell command that 254 | prints "-I/path/to/extra/include/files". For example, 255 | 256 | ``` 257 | // Puts the headers in /usr/include/glib-2.0 on the header search path. 258 | %install-extra-include-command echo -I/usr/include/glib-2.0 259 | 260 | // Puts the headers returned by `pkg-config` on the header search path. 261 | %install-extra-include-command pkg-config --cflags-only-I glib-2.0 262 | ``` 263 | 264 | In principle, swift-jupyter should be able to infer the necessary header search 265 | paths without you needing to manually specify them, but this hasn't been 266 | implemented yet. See [this forum 267 | thread](https://forums.fast.ai/t/cant-import-swiftvips/44833/21?u=marcrasi) for 268 | more information. 269 | 270 | ## %include directives 271 | 272 | `%include` directives let you include code from files. To use them, put a line 273 | `%include ""` in your cell. The kernel will preprocess your cell and 274 | replace the `%include` directive with the contents of the file before sending 275 | your cell to the Swift interpreter. 276 | 277 | `` must be relative to the directory containing `swift_kernel.py`. 278 | We'll probably add more search paths later. 279 | 280 | # Running tests 281 | 282 | ## Locally 283 | 284 | Install swift-jupyter locally using the above installation instructions. Now 285 | you can activate the virtualenv and run the tests: 286 | 287 | ``` 288 | . venv/bin/activate 289 | python test/fast_test.py # Fast tests, should complete in 1-2 min 290 | python test/all_test_local.py # Much slower, 10+ min 291 | python test/all_test_local.py SimpleNotebookTests.test_simple_successful # Invoke specific test method 292 | ``` 293 | 294 | You might also be interested in manually invoking the notebook tester on 295 | specific notebooks. See its `--help` documentation: 296 | 297 | ``` 298 | python test/notebook_tester.py --help 299 | ``` 300 | 301 | ## In Docker 302 | 303 | After building the docker image according to the instructions above, 304 | 305 | ``` 306 | docker run --cap-add SYS_PTRACE swift-jupyter python3 /swift-jupyter/test/all_test_docker.py 307 | ``` 308 | -------------------------------------------------------------------------------- /test/notebook_tester.py: -------------------------------------------------------------------------------- 1 | """Runs notebooks. 2 | 3 | See --help text for more information. 4 | """ 5 | 6 | import argparse 7 | import nbformat 8 | import numpy 9 | import os 10 | import sys 11 | import time 12 | 13 | from collections import defaultdict 14 | from jupyter_client.manager import start_new_kernel 15 | 16 | 17 | # Exception for problems that occur while executing cell. 18 | class ExecuteException(Exception): 19 | def __init__(self, cell_index): 20 | self.cell_index = cell_index 21 | 22 | 23 | # There was an error (that did not crash the kernel) while executing the cell. 24 | class ExecuteError(ExecuteException): 25 | def __init__(self, cell_index, reply, stdout): 26 | super(ExecuteError, self).__init__(cell_index) 27 | self.reply = reply 28 | self.stdout = stdout 29 | 30 | def __str__(self): 31 | return 'ExecuteError at cell %d, reply:\n%s\n\nstdout:\n%s' % ( 32 | self.cell_index, self.reply, self.stdout) 33 | 34 | 35 | # The kernel crashed while executing the cell. 36 | class ExecuteCrash(ExecuteException): 37 | def __init__(self, cell_index): 38 | super(ExecuteCrash, self).__init__(cell_index) 39 | 40 | def __str__(self): 41 | return 'ExecuteCrash at cell %d' % self.cell_index 42 | 43 | 44 | # Exception for problems that occur during a completion request. 45 | class CompleteException(Exception): 46 | def __init__(self, cell_index, char_index): 47 | self.cell_index = cell_index 48 | self.char_index = char_index 49 | 50 | 51 | # There was an error (that did not crash the kernel) while processing a 52 | # completion request. 53 | class CompleteError(CompleteException): 54 | def __init__(self, cell_index, char_index): 55 | super(CompleteError, self).__init__(cell_index, char_index) 56 | 57 | def __str__(self): 58 | return 'CompleteError at cell %d, char %d' % (self.cell_index, 59 | self.char_index) 60 | 61 | 62 | # The kernel crashed while processing a completion request. 63 | class CompleteCrash(CompleteException): 64 | def __init__(self, cell_index, char_index): 65 | super(CompleteCrash, self).__init__(cell_index, char_index) 66 | 67 | def __str__(self): 68 | return 'CompleteCrash at cell %d, char %d' % (self.cell_index, 69 | self.char_index) 70 | 71 | 72 | class NotebookTestRunner: 73 | def __init__(self, notebook, char_step=1, repeat_times=1, 74 | execute_timeout=30, complete_timeout=5, verbose=True): 75 | """ 76 | noteboook - path to a notebook to run the test on 77 | char_step - number of chars to step per completion request. 0 disables 78 | repeat_times - run the notebook this many times, in the same kernel 79 | instance 80 | execute_timeout - number of seconds to wait for cell execution 81 | complete_timeout - number of seconds to wait for completion 82 | verbose - print progress, statistics, and errors 83 | """ 84 | 85 | self.char_step = char_step 86 | self.repeat_times = repeat_times 87 | self.execute_timeout = execute_timeout 88 | self.complete_timeout = complete_timeout 89 | self.verbose = verbose 90 | 91 | notebook_dir = os.path.dirname(notebook) 92 | os.chdir(notebook_dir) 93 | nb = nbformat.read(notebook, as_version=4) 94 | 95 | self.code_cells = [cell for cell in nb.cells 96 | if cell.cell_type == 'code' \ 97 | and not cell.source.startswith('#@title')] 98 | 99 | self.stdout = [] 100 | self.unexpected_errors = [] 101 | 102 | def _execute_cell(self, cell_index): 103 | code = self.code_cells[cell_index].source 104 | self._execute_code(code, cell_index) 105 | 106 | def _execute_code(self, code, cell_index=-1): 107 | self.kc.execute(code) 108 | 109 | # Consume all the iopub messages that the execution produced. 110 | stdout = '' 111 | while True: 112 | try: 113 | reply = self.kc.get_iopub_msg(timeout=self.execute_timeout) 114 | except TimeoutError: 115 | # Timeout usually means that the kernel has crashed. 116 | raise ExecuteCrash(cell_index) 117 | if reply['header']['msg_type'] == 'stream' and \ 118 | reply['content']['name'] == 'stdout': 119 | stdout += reply['content']['text'] 120 | if reply['header']['msg_type'] == 'status' and \ 121 | reply['content']['execution_state'] == 'idle': 122 | break 123 | 124 | # Consume the shell message that the execution produced. 125 | try: 126 | reply = self.kc.get_shell_msg(timeout=self.execute_timeout) 127 | except TimeoutError: 128 | # Timeout usually means that the kernel has crashed. 129 | raise ExecuteCrash(cell_index) 130 | if reply['content']['status'] != 'ok': 131 | raise ExecuteError(cell_index, reply, stdout) 132 | 133 | if cell_index >= 0: 134 | self.stdout.append(stdout) 135 | 136 | return stdout 137 | 138 | def _complete(self, cell_index, char_index): 139 | code = self.code_cells[cell_index].source[:char_index] 140 | try: 141 | reply = self.kc.complete(code, reply=True, timeout=self.complete_timeout) 142 | except TimeoutError: 143 | # Timeout usually means that the kernel has crashed. 144 | raise CompleteCrash(cell_index, char_index) 145 | if reply['content']['status'] != 'ok': 146 | raise CompleteError(cell_index, char_index) 147 | 148 | # Consume all the iopub messages that the completion produced. 149 | while True: 150 | try: 151 | reply = self.kc.get_iopub_msg(timeout=self.execute_timeout) 152 | except TimeoutError: 153 | # Timeout usually means that the kernel has crashed. 154 | raise CompleteCrash(cell_index, char_index) 155 | if reply['header']['msg_type'] == 'status' and \ 156 | reply['content']['execution_state'] == 'idle': 157 | break 158 | 159 | def _init_kernel(self): 160 | km, kc = start_new_kernel(kernel_name='swift') 161 | self.km = km 162 | self.kc = kc 163 | 164 | # Runs each code cell in order, asking for completions in each cell along 165 | # the way. Raises an exception if there is an error or crash. Otherwise, 166 | # returns. 167 | def _run_notebook_once(self, failed_completions): 168 | for cell_index, cell in enumerate(self.code_cells): 169 | completion_times = [] 170 | 171 | # Don't do completions when `char_step` is 0. 172 | # Don't do completions when we already have 3 completion failures 173 | # in this cell. 174 | # Otherwise, ask for a completion every `char_step` chars. 175 | if self.char_step > 0 and \ 176 | len(failed_completions[cell_index]) < 3: 177 | for char_index in range(0, len(cell.source), self.char_step): 178 | if char_index in failed_completions[cell_index]: 179 | continue 180 | if self.verbose: 181 | print('Cell %d/%d: completing char %d/%d' % ( 182 | cell_index, len(self.code_cells), char_index, 183 | len(cell.source)), 184 | end='\r') 185 | start_time = time.time() 186 | self._complete(cell_index, char_index) 187 | completion_times.append(1000 * (time.time() - start_time)) 188 | 189 | 190 | # Execute the cell. 191 | if self.verbose: 192 | print('Cell %d/%d: executing ' % ( 193 | cell_index, len(self.code_cells)), 194 | end='\r') 195 | start_time = time.time() 196 | self._execute_cell(cell_index) 197 | execute_time = 1000 * (time.time() - start_time) 198 | 199 | # Report the results. 200 | report = 'Cell %d/%d: done' % (cell_index, len(self.code_cells)) 201 | report += ' - execute %.0f ms' % execute_time 202 | if len(failed_completions[cell_index]) > 0: 203 | # Don't report completion timings in cells with failed 204 | # completions, because they might be misleading. 205 | report += ' - completion error(s) occurred' 206 | elif len(completion_times) == 0: 207 | report += ' - no completions performed' 208 | else: 209 | report += ' - complete p50 %.0f ms' % ( 210 | numpy.percentile(completion_times, 50)) 211 | report += ' - complete p90 %.0f ms' % ( 212 | numpy.percentile(completion_times, 90)) 213 | report += ' - complete p99 %.0f ms' % ( 214 | numpy.percentile(completion_times, 99)) 215 | if self.verbose: 216 | print(report) 217 | 218 | def _record_error(self, e): 219 | cell = self.code_cells[e.cell_index] 220 | if hasattr(e, 'char_index'): 221 | code = cell.source[:e.char_index] 222 | else: 223 | code = cell.source 224 | 225 | error_description = { 226 | 'error': e, 227 | 'code': code, 228 | } 229 | 230 | if self.verbose: 231 | print('ERROR!\n%s\n\nCode:\n%s\n' % (e, code)) 232 | self.unexpected_errors.append(error_description) 233 | 234 | def run(self): 235 | # map from cell index to set of char indexes where completions failed 236 | failed_completions = defaultdict(set) 237 | 238 | while True: 239 | self._init_kernel() 240 | try: 241 | for _ in range(self.repeat_times): 242 | self._run_notebook_once(failed_completions) 243 | break 244 | except ExecuteException as ee: 245 | # Execution exceptions can't be recovered, so take note of the 246 | # error and stop the stress test. 247 | self._record_error(ee) 248 | break 249 | except CompleteException as ce: 250 | # Completion exceptions can be recovered! Restart the kernel 251 | # and don't ask for the broken completion next time. 252 | self._record_error(ce) 253 | failed_completions[ce.cell_index].add(ce.char_index) 254 | finally: 255 | self.km.shutdown_kernel(now=True) 256 | 257 | 258 | def parse_args(): 259 | parser = argparse.ArgumentParser( 260 | description='Executes all the cells in a Jupyter notebook, and ' 261 | 'requests completions along the way. Records and ' 262 | 'reports errors and kernel crashes that occur.') 263 | parser.add_argument('notebook', 264 | help='path to a notebook to run the test on') 265 | parser.add_argument('--char-step', type=int, default=1, 266 | help='number of chars to step per completion request. ' 267 | '0 disables completion requests') 268 | parser.add_argument('--repeat-times', type=int, default=1, 269 | help='run the notebook this many times, in the same ' 270 | 'kernel instance') 271 | parser.add_argument('--execute-timeout', type=int, default=15, 272 | help='number of seconds to wait for cell execution') 273 | parser.add_argument('--complete-timeout', type=int, default=5, 274 | help='number of seconds to wait for completion') 275 | return parser.parse_args() 276 | 277 | 278 | def _main(): 279 | args = parse_args() 280 | runner = NotebookTestRunner(**args.__dict__) 281 | runner.run() 282 | print(runner.unexpected_errors) 283 | 284 | 285 | if __name__ == '__main__': 286 | _main() 287 | -------------------------------------------------------------------------------- /test/tests/kernel_tests.py: -------------------------------------------------------------------------------- 1 | """Manually crafted tests testing specific features of the kernel. 2 | """ 3 | 4 | import unittest 5 | import jupyter_kernel_test 6 | import time 7 | 8 | from jupyter_client.manager import start_new_kernel 9 | 10 | # This superclass defines tests but does not run them against kernels, so that 11 | # we can subclass this to run the same tests against different kernels. 12 | # 13 | # In particular, the subclasses run against kernels that interop with differnt 14 | # versions of Python, so that we test that graphics work with different versions 15 | # of Python. 16 | class SwiftKernelTestsBase: 17 | language_name = 'swift' 18 | 19 | code_hello_world = 'print("hello, world!")' 20 | 21 | code_execute_result = [ 22 | {'code': 'let x = 2; x', 'result': '2\n'} 23 | ] 24 | 25 | code_generate_error = 'varThatIsntDefined' 26 | 27 | def setUp(self): 28 | self.flush_channels() 29 | 30 | def test_graphics_matplotlib(self): 31 | reply, output_msgs = self.execute_helper(code=""" 32 | %include "EnableIPythonDisplay.swift" 33 | """) 34 | self.assertEqual(reply['content']['status'], 'ok') 35 | 36 | reply, output_msgs = self.execute_helper(code=""" 37 | let np = Python.import("numpy") 38 | let plt = Python.import("matplotlib.pyplot") 39 | IPythonDisplay.shell.enable_matplotlib("inline") 40 | """) 41 | self.assertEqual(reply['content']['status'], 'ok') 42 | 43 | reply, output_msgs = self.execute_helper(code=""" 44 | let ys = np.arange(0, 10, 0.01) 45 | plt.plot(ys) 46 | plt.show() 47 | """) 48 | self.assertEqual(reply['content']['status'], 'ok') 49 | self.assertIn('image/png', output_msgs[0]['content']['data']) 50 | 51 | def test_extensions(self): 52 | reply, output_msgs = self.execute_helper(code=""" 53 | struct Foo{} 54 | """) 55 | self.assertEqual(reply['content']['status'], 'ok') 56 | reply, output_msgs = self.execute_helper(code=""" 57 | extension Foo { func f() -> Int { return 1 } } 58 | """) 59 | self.assertEqual(reply['content']['status'], 'ok') 60 | reply, output_msgs = self.execute_helper(code=""" 61 | print("Value of Foo().f() is", Foo().f()) 62 | """) 63 | self.assertEqual(reply['content']['status'], 'ok') 64 | self.assertIn("Value of Foo().f() is 1", output_msgs[0]['content']['text']) 65 | reply, output_msgs = self.execute_helper(code=""" 66 | extension Foo { func f() -> Int { return 2 } } 67 | """) 68 | self.assertEqual(reply['content']['status'], 'ok') 69 | reply, output_msgs = self.execute_helper(code=""" 70 | print("Value of Foo().f() is", Foo().f()) 71 | """) 72 | self.assertEqual(reply['content']['status'], 'ok') 73 | self.assertIn("Value of Foo().f() is 2", output_msgs[0]['content']['text']) 74 | 75 | def test_gradient_across_cells_error(self): 76 | reply, output_msgs = self.execute_helper(code=""" 77 | func square(_ x : Float) -> Float { return x * x } 78 | """) 79 | self.assertEqual(reply['content']['status'], 'ok') 80 | reply, output_msgs = self.execute_helper(code=""" 81 | print("5^2 is", square(5)) 82 | """) 83 | self.assertEqual(reply['content']['status'], 'ok') 84 | self.assertIn("5^2 is 25.0", output_msgs[0]['content']['text']) 85 | reply, output_msgs = self.execute_helper(code=""" 86 | print("gradient of square at 5 is", gradient(at: 5, in: square)) 87 | """) 88 | self.assertEqual(reply['content']['status'], 'error') 89 | self.assertIn("cannot differentiate functions that have not been marked '@differentiable'", 90 | reply['content']['traceback'][0]) 91 | 92 | def test_gradient_across_cells(self): 93 | reply, output_msgs = self.execute_helper(code=""" 94 | @differentiable 95 | func square(_ x : Float) -> Float { return x * x } 96 | """) 97 | self.assertEqual(reply['content']['status'], 'ok') 98 | reply, output_msgs = self.execute_helper(code=""" 99 | print("5^2 is", square(5)) 100 | """) 101 | self.assertEqual(reply['content']['status'], 'ok') 102 | self.assertIn("5^2 is 25.0", output_msgs[0]['content']['text']) 103 | reply, output_msgs = self.execute_helper(code=""" 104 | print("gradient of square at 5 is", gradient(at: 5, in: square)) 105 | """) 106 | self.assertEqual(reply['content']['status'], 'ok') 107 | self.assertIn("gradient of square at 5 is 10.0", output_msgs[0]['content']['text']) 108 | 109 | def test_error_runtime(self): 110 | reply, output_msgs = self.execute_helper(code=""" 111 | func a() { fatalError("oops") } 112 | """) 113 | self.assertEqual(reply['content']['status'], 'ok') 114 | a_cell = reply['content']['execution_count'] 115 | reply, output_msgs = self.execute_helper(code=""" 116 | print("hello") 117 | print("world") 118 | func b() { a() } 119 | """) 120 | self.assertEqual(reply['content']['status'], 'ok') 121 | b_cell = reply['content']['execution_count'] 122 | reply, output_msgs = self.execute_helper(code=""" 123 | b() 124 | """) 125 | self.assertEqual(reply['content']['status'], 'error') 126 | call_cell = reply['content']['execution_count'] 127 | 128 | stdout = output_msgs[0]['content']['text'] 129 | self.assertIn('Fatal error: oops', stdout) 130 | traceback = output_msgs[1]['content']['traceback'] 131 | all_tracebacks = '\n'.join(traceback) 132 | self.assertIn('Current stack trace:', all_tracebacks) 133 | self.assertIn('a() at :2:24' % a_cell, all_tracebacks) 134 | # TODO(TF-495): Reenable this assertion. 135 | # self.assertIn('b() at :4:24' % b_cell, all_tracebacks) 136 | self.assertIn('main at :2:13' % call_cell, all_tracebacks) 137 | 138 | def test_interrupt_execution(self): 139 | # Execute something to trigger debugger initialization, so that the 140 | # next cell executes quickly. 141 | self.execute_helper(code='1 + 1') 142 | 143 | msg_id = self.kc.execute(code="""while true {}""") 144 | 145 | # Give the kernel some time to actually start execution, because it 146 | # ignores interrupts that arrive when it's not actually executing. 147 | time.sleep(1) 148 | 149 | msg = self.kc.iopub_channel.get_msg(timeout=1) 150 | self.assertEqual(msg['content']['execution_state'], 'busy') 151 | 152 | self.km.interrupt_kernel() 153 | reply = self.kc.get_shell_msg(timeout=1) 154 | self.assertEqual(reply['content']['status'], 'error') 155 | 156 | while True: 157 | msg = self.kc.iopub_channel.get_msg(timeout=1) 158 | if msg['msg_type'] == 'status': 159 | self.assertEqual(msg['content']['execution_state'], 'idle') 160 | break 161 | 162 | # Check that the kernel can still execute things after handling an 163 | # interrupt. 164 | reply, output_msgs = self.execute_helper( 165 | code="""print("Hello world")""") 166 | self.assertEqual(reply['content']['status'], 'ok') 167 | for msg in output_msgs: 168 | if msg['msg_type'] == 'stream' and \ 169 | msg['content']['name'] == 'stdout': 170 | self.assertIn('Hello world', msg['content']['text']) 171 | break 172 | 173 | def test_async_stdout(self): 174 | # Execute something to trigger debugger initialization, so that the 175 | # next cell executes quickly. 176 | self.execute_helper(code='1 + 1') 177 | 178 | # Test that we receive stdout while execution is happening by printing 179 | # something and then entering an infinite loop. 180 | msg_id = self.kc.execute(code=""" 181 | print("some stdout") 182 | while true {} 183 | """) 184 | 185 | # Give the kernel some time to send out the stdout. 186 | time.sleep(1) 187 | 188 | # Check that the kernel has sent out the stdout. 189 | while True: 190 | msg = self.kc.iopub_channel.get_msg(timeout=1) 191 | if msg['msg_type'] == 'stream' and \ 192 | msg['content']['name'] == 'stdout': 193 | self.assertIn('some stdout', msg['content']['text']) 194 | break 195 | 196 | # Interrupt execution and consume all messages, so that subsequent 197 | # tests can run. (All the tests in this class run against the same 198 | # instance of the kernel.) 199 | self.km.interrupt_kernel() 200 | self.kc.get_shell_msg(timeout=1) 201 | while True: 202 | msg = self.kc.iopub_channel.get_msg(timeout=1) 203 | if msg['msg_type'] == 'status': 204 | break 205 | 206 | def test_swift_completion(self): 207 | reply, output_msgs = self.execute_helper(code=""" 208 | func aFunctionToComplete() {} 209 | """) 210 | self.assertEqual(reply['content']['status'], 'ok') 211 | 212 | self.kc.complete('aFunctionToC') 213 | reply = self.kc.get_shell_msg() 214 | self.assertEqual(reply['content']['matches'], 215 | ['aFunctionToComplete()']) 216 | self.flush_channels() 217 | 218 | reply, output_msgs = self.execute_helper(code=""" 219 | %disableCompletion 220 | """) 221 | self.assertEqual(reply['content']['status'], 'ok') 222 | 223 | self.kc.complete('aFunctionToC') 224 | reply = self.kc.get_shell_msg() 225 | self.assertEqual(reply['content']['matches'], []) 226 | self.flush_channels() 227 | 228 | reply, output_msgs = self.execute_helper(code=""" 229 | %enableCompletion 230 | """) 231 | self.assertEqual(reply['content']['status'], 'ok') 232 | 233 | self.kc.complete('aFunctionToC') 234 | reply = self.kc.get_shell_msg() 235 | self.assertEqual(reply['content']['matches'], 236 | ['aFunctionToComplete()']) 237 | self.flush_channels() 238 | 239 | 240 | class SwiftKernelTestsPython27(SwiftKernelTestsBase, 241 | jupyter_kernel_test.KernelTests): 242 | kernel_name = 'swift-with-python-2.7' 243 | 244 | 245 | class SwiftKernelTests(SwiftKernelTestsBase, 246 | jupyter_kernel_test.KernelTests): 247 | kernel_name = 'swift' 248 | 249 | 250 | # Class for tests that need their own kernel. (`SwiftKernelTestsBase` uses one 251 | # kernel for all the tests.) 252 | class OwnKernelTests(unittest.TestCase): 253 | def test_process_killed(self): 254 | km, kc = start_new_kernel(kernel_name='swift') 255 | kc.execute(""" 256 | import Glibc 257 | exit(0) 258 | """) 259 | messages = self.wait_for_idle(kc) 260 | 261 | had_error = False 262 | for message in messages: 263 | if message['header']['msg_type'] == 'error': 264 | had_error = True 265 | self.assertEqual(['Process killed'], 266 | message['content']['traceback']) 267 | self.assertTrue(had_error) 268 | 269 | def test_install_after_execute(self): 270 | # The kernel is supposed to refuse to install package after executing 271 | # code. 272 | 273 | km, kc = start_new_kernel(kernel_name='swift') 274 | kc.execute('1 + 1') 275 | self.wait_for_idle(kc) 276 | kc.execute(""" 277 | %install DummyPackage DummyPackage 278 | """) 279 | messages = self.wait_for_idle(kc) 280 | 281 | had_error = False 282 | for message in messages: 283 | if message['header']['msg_type'] == 'error': 284 | had_error = True 285 | self.assertIn('Install Error: Packages can only be installed ' 286 | 'during the first cell execution.', 287 | message['content']['traceback'][0]) 288 | self.assertTrue(had_error) 289 | 290 | def test_install_after_execute_blank(self): 291 | # If the user executes blank code, the kernel is supposed to try 292 | # to install packages. In particular, Colab sends a blank execution 293 | # request to the kernel when it starts up, and it's important that this 294 | # doesn't block package installation. 295 | 296 | km, kc = start_new_kernel(kernel_name='swift') 297 | kc.execute('\n\n\n') 298 | self.wait_for_idle(kc) 299 | kc.execute(""" 300 | %install DummyPackage DummyPackage 301 | """) 302 | messages = self.wait_for_idle(kc) 303 | 304 | # DummyPackage doesn't exist, so package installation won't actually 305 | # succeed. So we just assert that the kernel tries to install it. 306 | stdout = '' 307 | for message in messages: 308 | if message['header']['msg_type'] == 'stream' and \ 309 | message['content']['name'] == 'stdout': 310 | stdout += message['content']['text'] 311 | self.assertIn('Installing packages:', stdout) 312 | 313 | def wait_for_idle(self, kc): 314 | messages = [] 315 | while True: 316 | message = kc.get_iopub_msg(timeout=30) 317 | messages.append(message) 318 | if message['header']['msg_type'] == 'status' and \ 319 | message['content']['execution_state'] == 'idle': 320 | break 321 | return messages 322 | -------------------------------------------------------------------------------- /swift_kernel.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | # 3 | # Copyright 2018 Google LLC 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | import glob 18 | import json 19 | import lldb 20 | import os 21 | import stat 22 | import re 23 | import shlex 24 | import shutil 25 | import signal 26 | import string 27 | import subprocess 28 | import sys 29 | import tempfile 30 | import textwrap 31 | import time 32 | import threading 33 | import sqlite3 34 | import json 35 | 36 | from ipykernel.kernelbase import Kernel 37 | from jupyter_client.jsonutil import squash_dates 38 | from tornado import ioloop 39 | 40 | 41 | class ExecutionResult: 42 | """Base class for the result of executing code.""" 43 | pass 44 | 45 | 46 | class ExecutionResultSuccess(ExecutionResult): 47 | """Base class for the result of successfully executing code.""" 48 | pass 49 | 50 | 51 | class ExecutionResultError(ExecutionResult): 52 | """Base class for the result of unsuccessfully executing code.""" 53 | def description(self): 54 | raise NotImplementedError() 55 | 56 | 57 | class SuccessWithoutValue(ExecutionResultSuccess): 58 | """The code executed successfully, and did not produce a value.""" 59 | def __repr__(self): 60 | return 'SuccessWithoutValue()' 61 | 62 | 63 | class SuccessWithValue(ExecutionResultSuccess): 64 | """The code executed successfully, and produced a value.""" 65 | def __init__(self, result): 66 | self.result = result # SBValue 67 | 68 | def __repr__(self): 69 | return 'SuccessWithValue(result=%s, description=%s)' % ( 70 | repr(self.result), repr(self.result.description)) 71 | 72 | 73 | class PreprocessorError(ExecutionResultError): 74 | """There was an error preprocessing the code.""" 75 | def __init__(self, exception): 76 | self.exception = exception # PreprocessorException 77 | 78 | def description(self): 79 | return str(self.exception) 80 | 81 | def __repr__(self): 82 | return 'PreprocessorError(exception=%s)' % repr(self.exception) 83 | 84 | 85 | class PreprocessorException(Exception): 86 | pass 87 | 88 | 89 | class PackageInstallException(Exception): 90 | pass 91 | 92 | 93 | class SwiftError(ExecutionResultError): 94 | """There was a compile or runtime error.""" 95 | def __init__(self, result): 96 | self.result = result # SBValue 97 | 98 | def description(self): 99 | return self.result.error.description 100 | 101 | def __repr__(self): 102 | return 'SwiftError(result=%s, description=%s)' % ( 103 | repr(self.result), repr(self.description())) 104 | 105 | 106 | class SIGINTHandler(threading.Thread): 107 | """Interrupts currently-executing code whenever the process receives a 108 | SIGINT.""" 109 | 110 | daemon = True 111 | 112 | def __init__(self, kernel): 113 | super(SIGINTHandler, self).__init__() 114 | self.kernel = kernel 115 | 116 | def run(self): 117 | try: 118 | while True: 119 | signal.sigwait([signal.SIGINT]) 120 | self.kernel.process.SendAsyncInterrupt() 121 | except Exception as e: 122 | self.kernel.log.error('Exception in SIGINTHandler: %s' % str(e)) 123 | 124 | 125 | class StdoutHandler(threading.Thread): 126 | """Collects stdout from the Swift process and sends it to the client.""" 127 | 128 | daemon = True 129 | 130 | def __init__(self, kernel): 131 | super(StdoutHandler, self).__init__() 132 | self.kernel = kernel 133 | self.stop_event = threading.Event() 134 | self.had_stdout = False 135 | 136 | def _get_stdout(self): 137 | while True: 138 | BUFFER_SIZE = 1000 139 | stdout_buffer = self.kernel.process.GetSTDOUT(BUFFER_SIZE) 140 | if len(stdout_buffer) == 0: 141 | break 142 | yield stdout_buffer 143 | 144 | def _get_and_send_stdout(self): 145 | stdout = ''.join([buf for buf in self._get_stdout()]) 146 | if len(stdout) > 0: 147 | self.had_stdout = True 148 | self.kernel.send_response(self.kernel.iopub_socket, 'stream', { 149 | 'name': 'stdout', 150 | 'text': stdout 151 | }) 152 | 153 | def run(self): 154 | try: 155 | while True: 156 | if self.stop_event.wait(0.1): 157 | break 158 | self._get_and_send_stdout() 159 | self._get_and_send_stdout() 160 | except Exception as e: 161 | self.kernel.log.error('Exception in StdoutHandler: %s' % str(e)) 162 | 163 | 164 | class SwiftKernel(Kernel): 165 | implementation = 'SwiftKernel' 166 | implementation_version = '0.1' 167 | banner = '' 168 | 169 | language_info = { 170 | 'name': 'swift', 171 | 'mimetype': 'text/x-swift', 172 | 'file_extension': '.swift', 173 | 'version': '', 174 | } 175 | 176 | def __init__(self, **kwargs): 177 | super(SwiftKernel, self).__init__(**kwargs) 178 | 179 | # We don't initialize Swift yet, so that the user has a chance to 180 | # "%install" packages before Swift starts. (See doc comment in 181 | # `_init_swift`). 182 | 183 | # Whether to do code completion. Since the debugger is not yet 184 | # initialized, we can't do code completion yet. 185 | self.completion_enabled = False 186 | 187 | def _init_swift(self): 188 | """Initializes Swift so that it's ready to start executing user code. 189 | 190 | This must happen after package installation, because the ClangImporter 191 | does not see modulemap files that appear after it has started.""" 192 | 193 | self._init_repl_process() 194 | self._init_kernel_communicator() 195 | self._init_int_bitwidth() 196 | self._init_sigint_handler() 197 | 198 | # We do completion by default when the toolchain has the 199 | # SBTarget.CompleteCode API. 200 | # The user can disable/enable using "%disableCompletion" and 201 | # "%enableCompletion". 202 | self.completion_enabled = hasattr(self.target, 'CompleteCode') 203 | 204 | def _init_repl_process(self): 205 | self.debugger = lldb.SBDebugger.Create() 206 | if not self.debugger: 207 | raise Exception('Could not start debugger') 208 | self.debugger.SetAsync(False) 209 | 210 | # LLDB crashes while trying to load some Python stuff on Mac. Maybe 211 | # something is misconfigured? This works around the problem by telling 212 | # LLDB not to load the Python scripting stuff, which we don't use 213 | # anyways. 214 | self.debugger.SetScriptLanguage(lldb.eScriptLanguageNone) 215 | 216 | repl_swift = os.environ['REPL_SWIFT_PATH'] 217 | self.target = self.debugger.CreateTargetWithFileAndArch(repl_swift, '') 218 | if not self.target: 219 | raise Exception('Could not create target %s' % repl_swift) 220 | 221 | self.main_bp = self.target.BreakpointCreateByName( 222 | 'repl_main', self.target.GetExecutable().GetFilename()) 223 | if not self.main_bp: 224 | raise Exception('Could not set breakpoint') 225 | 226 | repl_env = [] 227 | script_dir = os.path.dirname(os.path.realpath(sys.argv[0])) 228 | repl_env.append('PYTHONPATH=%s' % script_dir) 229 | env_var_blacklist = [ 230 | 'PYTHONPATH', 231 | 'REPL_SWIFT_PATH' 232 | ] 233 | for key in os.environ: 234 | if key in env_var_blacklist: 235 | continue 236 | repl_env.append('%s=%s' % (key, os.environ[key])) 237 | 238 | self.process = self.target.LaunchSimple(None, 239 | repl_env, 240 | os.getcwd()) 241 | if not self.process: 242 | raise Exception('Could not launch process') 243 | 244 | self.expr_opts = lldb.SBExpressionOptions() 245 | self.swift_language = lldb.SBLanguageRuntime.GetLanguageTypeFromString( 246 | 'swift') 247 | self.expr_opts.SetLanguage(self.swift_language) 248 | self.expr_opts.SetREPLMode(True) 249 | self.expr_opts.SetUnwindOnError(False) 250 | self.expr_opts.SetGenerateDebugInfo(True) 251 | 252 | # Sets an infinite timeout so that users can run aribtrarily long 253 | # computations. 254 | self.expr_opts.SetTimeoutInMicroSeconds(0) 255 | 256 | self.main_thread = self.process.GetThreadAtIndex(0) 257 | 258 | def _init_kernel_communicator(self): 259 | result = self._preprocess_and_execute( 260 | '%include "KernelCommunicator.swift"') 261 | if isinstance(result, ExecutionResultError): 262 | raise Exception('Error initing KernelCommunicator: %s' % result) 263 | 264 | session_key = self.session.key.decode('utf8') 265 | decl_code = """ 266 | enum JupyterKernel { 267 | static var communicator = KernelCommunicator( 268 | jupyterSession: KernelCommunicator.JupyterSession( 269 | id: %s, key: %s, username: %s)) 270 | } 271 | """ % (json.dumps(self.session.session), json.dumps(session_key), 272 | json.dumps(self.session.username)) 273 | result = self._preprocess_and_execute(decl_code) 274 | if isinstance(result, ExecutionResultError): 275 | raise Exception('Error declaring JupyterKernel: %s' % result) 276 | 277 | def _init_int_bitwidth(self): 278 | result = self._execute('Int.bitWidth') 279 | if not isinstance(result, SuccessWithValue): 280 | raise Exception('Expected value from Int.bitWidth, but got: %s' % 281 | result) 282 | self._int_bitwidth = int(result.result.description) 283 | 284 | def _init_sigint_handler(self): 285 | self.sigint_handler = SIGINTHandler(self) 286 | self.sigint_handler.start() 287 | 288 | def _file_name_for_source_location(self): 289 | return '' % self.execution_count 290 | 291 | def _preprocess_and_execute(self, code): 292 | try: 293 | preprocessed = self._preprocess(code) 294 | except PreprocessorException as e: 295 | return PreprocessorError(e) 296 | 297 | return self._execute(preprocessed) 298 | 299 | def _preprocess(self, code): 300 | lines = code.split('\n') 301 | preprocessed_lines = [ 302 | self._preprocess_line(i, line) for i, line in enumerate(lines)] 303 | return '\n'.join(preprocessed_lines) 304 | 305 | def _handle_disable_completion(self): 306 | self.completion_enabled = False 307 | self.send_response(self.iopub_socket, 'stream', { 308 | 'name': 'stdout', 309 | 'text': 'Completion disabled!\n' 310 | }) 311 | 312 | def _handle_enable_completion(self): 313 | if not hasattr(self.target, 'CompleteCode'): 314 | self.send_response(self.iopub_socket, 'stream', { 315 | 'name': 'stdout', 316 | 'text': 'Completion NOT enabled because toolchain does not ' + 317 | 'have CompleteCode API.\n' 318 | }) 319 | return 320 | 321 | self.completion_enabled = True 322 | self.send_response(self.iopub_socket, 'stream', { 323 | 'name': 'stdout', 324 | 'text': 'Completion enabled!\n' 325 | }) 326 | 327 | def _preprocess_line(self, line_index, line): 328 | """Returns the preprocessed line. 329 | 330 | Does not process "%install" directives, because those need to be 331 | handled before everything else.""" 332 | 333 | include_match = re.match(r'^\s*%include (.*)$', line) 334 | if include_match is not None: 335 | return self._read_include(line_index, include_match.group(1)) 336 | 337 | disable_completion_match = re.match(r'^\s*%disableCompletion\s*$', line) 338 | if disable_completion_match is not None: 339 | self._handle_disable_completion() 340 | return '' 341 | 342 | enable_completion_match = re.match(r'^\s*%enableCompletion\s*$', line) 343 | if enable_completion_match is not None: 344 | self._handle_enable_completion() 345 | return '' 346 | 347 | return line 348 | 349 | def _read_include(self, line_index, rest_of_line): 350 | name_match = re.match(r'^\s*"([^"]+)"\s*$', rest_of_line) 351 | if name_match is None: 352 | raise PreprocessorException( 353 | 'Line %d: %%include must be followed by a name in quotes' % ( 354 | line_index + 1)) 355 | name = name_match.group(1) 356 | 357 | include_paths = [ 358 | os.path.dirname(os.path.realpath(sys.argv[0])), 359 | os.path.realpath("."), 360 | ] 361 | 362 | code = None 363 | for include_path in include_paths: 364 | try: 365 | with open(os.path.join(include_path, name), 'r') as f: 366 | code = f.read() 367 | except IOError: 368 | continue 369 | 370 | if code is None: 371 | raise PreprocessorException( 372 | 'Line %d: Could not find "%s". Searched %s.' % ( 373 | line_index + 1, name, include_paths)) 374 | 375 | return '\n'.join([ 376 | '#sourceLocation(file: "%s", line: 1)' % name, 377 | code, 378 | '#sourceLocation(file: "%s", line: %d)' % ( 379 | self._file_name_for_source_location(), line_index + 1), 380 | '' 381 | ]) 382 | 383 | def _process_installs(self, code): 384 | """Handles all "%install" directives, and returns `code` with all 385 | "%install" directives removed.""" 386 | processed_lines = [] 387 | all_packages = [] 388 | all_swiftpm_flags = [] 389 | extra_include_commands = [] 390 | user_install_location = None 391 | for index, line in enumerate(code.split('\n')): 392 | line, install_location = self._process_install_location_line(line) 393 | line, swiftpm_flags = self._process_install_swiftpm_flags_line( 394 | line) 395 | all_swiftpm_flags += swiftpm_flags 396 | line, packages = self._process_install_line(index, line) 397 | line, extra_include_command = \ 398 | self._process_extra_include_command_line(line) 399 | if extra_include_command: 400 | extra_include_commands.append(extra_include_command) 401 | processed_lines.append(line) 402 | all_packages += packages 403 | if install_location: user_install_location = install_location 404 | 405 | self._install_packages(all_packages, all_swiftpm_flags, 406 | extra_include_commands, 407 | user_install_location) 408 | return '\n'.join(processed_lines) 409 | 410 | def _process_install_location_line(self, line): 411 | install_location_match = re.match( 412 | r'^\s*%install-location (.*)$', line) 413 | if install_location_match is None: 414 | return line, None 415 | 416 | install_location = install_location_match.group(1) 417 | try: 418 | install_location = string.Template(install_location).substitute({"cwd": os.getcwd()}) 419 | except KeyError as e: 420 | raise PackageInstallException( 421 | 'Line %d: Invalid template argument %s' % (line_index + 1, 422 | str(e))) 423 | except ValueError as e: 424 | raise PackageInstallException( 425 | 'Line %d: %s' % (line_index + 1, str(e))) 426 | 427 | return '', install_location 428 | 429 | def _process_extra_include_command_line(self, line): 430 | extra_include_command_match = re.match( 431 | r'^\s*%install-extra-include-command (.*)$', line) 432 | if extra_include_command_match is None: 433 | return line, None 434 | 435 | extra_include_command = extra_include_command_match.group(1) 436 | 437 | return '', extra_include_command 438 | 439 | def _process_install_swiftpm_flags_line(self, line): 440 | install_swiftpm_flags_match = re.match( 441 | r'^\s*%install-swiftpm-flags (.*)$', line) 442 | if install_swiftpm_flags_match is None: 443 | return line, [] 444 | flags = shlex.split(install_swiftpm_flags_match.group(1)) 445 | return '', flags 446 | 447 | def _process_install_line(self, line_index, line): 448 | install_match = re.match(r'^\s*%install (.*)$', line) 449 | if install_match is None: 450 | return line, [] 451 | 452 | parsed = shlex.split(install_match.group(1)) 453 | if len(parsed) < 2: 454 | raise PackageInstallException( 455 | 'Line %d: %%install usage: SPEC PRODUCT [PRODUCT ...]' % ( 456 | line_index + 1)) 457 | try: 458 | spec = string.Template(parsed[0]).substitute({"cwd": os.getcwd()}) 459 | except KeyError as e: 460 | raise PackageInstallException( 461 | 'Line %d: Invalid template argument %s' % (line_index + 1, 462 | str(e))) 463 | except ValueError as e: 464 | raise PackageInstallException( 465 | 'Line %d: %s' % (line_index + 1, str(e))) 466 | 467 | return '', [{ 468 | 'spec': spec, 469 | 'products': parsed[1:], 470 | }] 471 | 472 | def _link_extra_includes(self, swift_import_search_path, include_dir): 473 | for include_file in os.listdir(include_dir): 474 | link_name = os.path.join(swift_import_search_path, include_file) 475 | target = os.path.join(include_dir, include_file) 476 | try: 477 | if stat.S_ISLNK(os.lstat(link_name).st_mode): 478 | os.unlink(link_name) 479 | except FileNotFoundError as e: 480 | pass 481 | except Error as e: 482 | raise PackageInstallException( 483 | 'Failed to stat scratchwork base path: %s' % str(e)) 484 | os.symlink(target, link_name) 485 | 486 | def _install_packages(self, packages, swiftpm_flags, extra_include_commands, 487 | user_install_location): 488 | if len(packages) == 0 and len(swiftpm_flags) == 0: 489 | return 490 | 491 | if hasattr(self, 'debugger'): 492 | raise PackageInstallException( 493 | 'Install Error: Packages can only be installed during the ' 494 | 'first cell execution. Restart the kernel to install ' 495 | 'packages.') 496 | 497 | swift_build_path = os.environ.get('SWIFT_BUILD_PATH') 498 | if swift_build_path is None: 499 | raise PackageInstallException( 500 | 'Install Error: Cannot install packages because ' 501 | 'SWIFT_BUILD_PATH is not specified.') 502 | 503 | swift_package_path = os.environ.get('SWIFT_PACKAGE_PATH') 504 | if swift_package_path is None: 505 | raise PackageInstallException( 506 | 'Install Error: Cannot install packages because ' 507 | 'SWIFT_PACKAGE_PATH is not specified.') 508 | 509 | swift_import_search_path = os.environ.get('SWIFT_IMPORT_SEARCH_PATH') 510 | if swift_import_search_path is None: 511 | raise PackageInstallException( 512 | 'Install Error: Cannot install packages because ' 513 | 'SWIFT_IMPORT_SEARCH_PATH is not specified.') 514 | 515 | scratchwork_base_path = os.path.dirname(swift_import_search_path) 516 | package_base_path = os.path.join(scratchwork_base_path, 'package') 517 | 518 | # If the user has specified a custom install location, make a link from 519 | # the scratchwork base path to it. 520 | if user_install_location is not None: 521 | # symlink to the specified location 522 | # Remove existing base if it is already a symlink 523 | os.makedirs(user_install_location, exist_ok=True) 524 | try: 525 | if stat.S_ISLNK(os.lstat(scratchwork_base_path).st_mode): 526 | os.unlink(scratchwork_base_path) 527 | except FileNotFoundError as e: 528 | pass 529 | except Error as e: 530 | raise PackageInstallException( 531 | 'Failed to stat scratchwork base path: %s' % str(e)) 532 | os.symlink(user_install_location, scratchwork_base_path, 533 | target_is_directory=True) 534 | 535 | # Make the directory containing our synthesized package. 536 | os.makedirs(package_base_path, exist_ok=True) 537 | 538 | # Make the directory containing our built modules and other includes. 539 | os.makedirs(swift_import_search_path, exist_ok=True) 540 | 541 | # Make links from the install location to extra includes. 542 | for include_command in extra_include_commands: 543 | result = subprocess.run(include_command, shell=True, 544 | stdout=subprocess.PIPE, 545 | stderr=subprocess.PIPE) 546 | if result.returncode != 0: 547 | raise PackageInstallException( 548 | '%%install-extra-include-command returned nonzero ' 549 | 'exit code: %d\nStdout:\n%s\nStderr:\n%s\n' % ( 550 | result.returncode, 551 | result.stdout.decode('utf8'), 552 | result.stderr.decode('utf8'))) 553 | include_dirs = shlex.split(result.stdout.decode('utf8')) 554 | for include_dir in include_dirs: 555 | if include_dir[0:2] != '-I': 556 | self.log.warn( 557 | 'Non "-I" output from ' 558 | '%%install-extra-include-command: %s' % include_dir) 559 | continue 560 | include_dir = include_dir[2:] 561 | self._link_extra_includes(swift_import_search_path, include_dir) 562 | 563 | # Summary of how this works: 564 | # - create a SwiftPM package that depends on all the packages that 565 | # the user requested 566 | # - ask SwiftPM to build that package 567 | # - copy all the .swiftmodule and module.modulemap files that SwiftPM 568 | # created to SWIFT_IMPORT_SEARCH_PATH 569 | # - dlopen the .so file that SwiftPM created 570 | 571 | # == Create the SwiftPM package == 572 | 573 | package_swift_template = textwrap.dedent("""\ 574 | // swift-tools-version:4.2 575 | import PackageDescription 576 | let package = Package( 577 | name: "jupyterInstalledPackages", 578 | products: [ 579 | .library( 580 | name: "jupyterInstalledPackages", 581 | type: .dynamic, 582 | targets: ["jupyterInstalledPackages"]), 583 | ], 584 | dependencies: [%s], 585 | targets: [ 586 | .target( 587 | name: "jupyterInstalledPackages", 588 | dependencies: [%s], 589 | path: ".", 590 | sources: ["jupyterInstalledPackages.swift"]), 591 | ]) 592 | """) 593 | 594 | packages_specs = '' 595 | packages_products = '' 596 | packages_human_description = '' 597 | for package in packages: 598 | packages_specs += '%s,\n' % package['spec'] 599 | packages_human_description += '\t%s\n' % package['spec'] 600 | for target in package['products']: 601 | packages_products += '%s,\n' % json.dumps(target) 602 | packages_human_description += '\t\t%s\n' % target 603 | 604 | self.send_response(self.iopub_socket, 'stream', { 605 | 'name': 'stdout', 606 | 'text': 'Installing packages:\n%s' % packages_human_description 607 | }) 608 | self.send_response(self.iopub_socket, 'stream', { 609 | 'name': 'stdout', 610 | 'text': 'With SwiftPM flags: %s\n' % str(swiftpm_flags) 611 | }) 612 | self.send_response(self.iopub_socket, 'stream', { 613 | 'name': 'stdout', 614 | 'text': 'Working in: %s\n' % scratchwork_base_path 615 | }) 616 | 617 | package_swift = package_swift_template % (packages_specs, 618 | packages_products) 619 | 620 | with open('%s/Package.swift' % package_base_path, 'w') as f: 621 | f.write(package_swift) 622 | with open('%s/jupyterInstalledPackages.swift' % package_base_path, 'w') as f: 623 | f.write("// intentionally blank\n") 624 | 625 | # == Ask SwiftPM to build the package == 626 | 627 | build_p = subprocess.Popen([swift_build_path] + swiftpm_flags, 628 | stdout=subprocess.PIPE, 629 | stderr=subprocess.STDOUT, 630 | cwd=package_base_path) 631 | for build_output_line in iter(build_p.stdout.readline, b''): 632 | self.send_response(self.iopub_socket, 'stream', { 633 | 'name': 'stdout', 634 | 'text': build_output_line.decode('utf8') 635 | }) 636 | build_returncode = build_p.wait() 637 | if build_returncode != 0: 638 | raise PackageInstallException( 639 | 'Install Error: swift-build returned nonzero exit code ' 640 | '%d.' % build_returncode) 641 | 642 | show_bin_path_result = subprocess.run( 643 | [swift_build_path, '--show-bin-path'] + swiftpm_flags, 644 | stdout=subprocess.PIPE, stderr=subprocess.PIPE, 645 | cwd=package_base_path) 646 | bin_dir = show_bin_path_result.stdout.decode('utf8').strip() 647 | lib_filename = os.path.join(bin_dir, 'libjupyterInstalledPackages.so') 648 | 649 | # == Copy .swiftmodule and modulemap files to SWIFT_IMPORT_SEARCH_PATH == 650 | 651 | build_db_file = os.path.join(package_base_path, '.build', 'build.db') 652 | if not os.path.exists(build_db_file): 653 | raise PackageInstallException('build.db is missing') 654 | 655 | # Execute swift-package show-dependencies to get all dependencies' paths 656 | dependencies_result = subprocess.run( 657 | [swift_package_path, 'show-dependencies', '--format', 'json'], 658 | stdout=subprocess.PIPE, stderr=subprocess.PIPE, 659 | cwd=package_base_path) 660 | dependencies_json = dependencies_result.stdout.decode('utf8') 661 | dependencies_obj = json.loads(dependencies_json) 662 | 663 | def flatten_deps_paths(dep): 664 | paths = [] 665 | paths.append(dep["path"]) 666 | if dep["dependencies"]: 667 | for d in dep["dependencies"]: 668 | paths.extend(flatten_deps_paths(d)) 669 | return paths 670 | 671 | # Make list of paths where we expect .swiftmodule and .modulemap files of dependencies 672 | dependencies_paths = [package_base_path] 673 | dependencies_paths = flatten_deps_paths(dependencies_obj) 674 | dependencies_paths = list(set(dependencies_paths)) 675 | 676 | def is_valid_dependency(path): 677 | for p in dependencies_paths: 678 | if path.startswith(p): return True 679 | return False 680 | 681 | # Query to get build files list from build.db 682 | # SUBSTR because string starts with "N" (why?) 683 | SQL_FILES_SELECT = "SELECT SUBSTR(key, 2) FROM 'key_names' WHERE key LIKE ?" 684 | 685 | # Connect to build.db 686 | db_connection = sqlite3.connect(build_db_file) 687 | cursor = db_connection.cursor() 688 | 689 | # Process *.swiftmodules files 690 | cursor.execute(SQL_FILES_SELECT, ['%.swiftmodule']) 691 | swift_modules = [row[0] for row in cursor.fetchall() if is_valid_dependency(row[0])] 692 | for filename in swift_modules: 693 | shutil.copy(filename, swift_import_search_path) 694 | 695 | # Process modulemap files 696 | cursor.execute(SQL_FILES_SELECT, ['%/module.modulemap']) 697 | modulemap_files = [row[0] for row in cursor.fetchall() if is_valid_dependency(row[0])] 698 | for index, filename in enumerate(modulemap_files): 699 | # Create a separate directory for each modulemap file because the 700 | # ClangImporter requires that they are all named 701 | # "module.modulemap". 702 | # Use the module name to prevent two modulema[s for the same 703 | # depndency ending up in multiple directories after several 704 | # installations, causing the kernel to end up in a bad state. 705 | # Make all relative header paths in module.modulemap absolute 706 | # because we copy file to different location. 707 | 708 | src_folder, src_filename = os.path.split(filename) 709 | with open(filename, encoding='utf8') as file: 710 | modulemap_contents = file.read() 711 | modulemap_contents = re.sub( 712 | r'header\s+"(.*?)"', 713 | lambda m: 'header "%s"' % 714 | (m.group(1) if os.path.isabs(m.group(1)) else os.path.abspath(os.path.join(src_folder, m.group(1)))), 715 | modulemap_contents 716 | ) 717 | 718 | module_match = re.match(r'module\s+([^\s]+)\s.*{', modulemap_contents) 719 | module_name = module_match.group(1) if module_match is not None else str(index) 720 | modulemap_dest = os.path.join(swift_import_search_path, 'modulemap-%s' % module_name) 721 | os.makedirs(modulemap_dest, exist_ok=True) 722 | dst_path = os.path.join(modulemap_dest, src_filename) 723 | 724 | with open(dst_path, 'w', encoding='utf8') as outfile: 725 | outfile.write(modulemap_contents) 726 | 727 | # == dlopen the shared lib == 728 | 729 | self.send_response(self.iopub_socket, 'stream', { 730 | 'name': 'stdout', 731 | 'text': 'Initializing Swift...\n' 732 | }) 733 | self._init_swift() 734 | 735 | dynamic_load_code = textwrap.dedent("""\ 736 | import func Glibc.dlopen 737 | dlopen(%s, RTLD_NOW) 738 | """ % json.dumps(lib_filename)) 739 | dynamic_load_result = self._execute(dynamic_load_code) 740 | if not isinstance(dynamic_load_result, SuccessWithValue): 741 | raise PackageInstallException( 742 | 'Install Error: dlopen error: %s' % \ 743 | str(dynamic_load_result)) 744 | if dynamic_load_result.result.description.strip() == 'nil': 745 | raise PackageInstallException('Install Error: dlopen error. Run ' 746 | '`String(cString: dlerror())` to see ' 747 | 'the error message.') 748 | 749 | self.send_response(self.iopub_socket, 'stream', { 750 | 'name': 'stdout', 751 | 'text': 'Installation complete!\n' 752 | }) 753 | self.already_installed_packages = True 754 | 755 | def _execute(self, code): 756 | locationDirective = '#sourceLocation(file: "%s", line: 1)' % ( 757 | self._file_name_for_source_location()) 758 | codeWithLocationDirective = locationDirective + '\n' + code 759 | result = self.target.EvaluateExpression( 760 | codeWithLocationDirective, self.expr_opts) 761 | 762 | if result.error.type == lldb.eErrorTypeInvalid: 763 | return SuccessWithValue(result) 764 | elif result.error.type == lldb.eErrorTypeGeneric: 765 | return SuccessWithoutValue() 766 | else: 767 | return SwiftError(result) 768 | 769 | def _after_successful_execution(self): 770 | result = self._execute( 771 | 'JupyterKernel.communicator.triggerAfterSuccessfulExecution()') 772 | if not isinstance(result, SuccessWithValue): 773 | self.log.error( 774 | 'Expected value from triggerAfterSuccessfulExecution(), ' 775 | 'but got: %s' % result) 776 | return 777 | 778 | messages = self._read_jupyter_messages(result.result) 779 | self._send_jupyter_messages(messages) 780 | 781 | def _read_jupyter_messages(self, sbvalue): 782 | return { 783 | 'display_messages': [ 784 | self._read_display_message(display_message_sbvalue) 785 | for display_message_sbvalue 786 | in sbvalue 787 | ] 788 | } 789 | 790 | def _read_display_message(self, sbvalue): 791 | return [self._read_byte_array(part) for part in sbvalue] 792 | 793 | def _read_byte_array(self, sbvalue): 794 | get_position_error = lldb.SBError() 795 | position = sbvalue \ 796 | .GetChildMemberWithName('_position') \ 797 | .GetData() \ 798 | .GetAddress(get_position_error, 0) 799 | if get_position_error.Fail(): 800 | raise Exception('getting position: %s' % str(get_position_error)) 801 | 802 | get_count_error = lldb.SBError() 803 | count_data = sbvalue \ 804 | .GetChildMemberWithName('count') \ 805 | .GetData() 806 | if self._int_bitwidth == 32: 807 | count = count_data.GetSignedInt32(get_count_error, 0) 808 | elif self._int_bitwidth == 64: 809 | count = count_data.GetSignedInt64(get_count_error, 0) 810 | else: 811 | raise Exception('Unsupported integer bitwidth %d' % 812 | self._int_bitwidth) 813 | if get_count_error.Fail(): 814 | raise Exception('getting count: %s' % str(get_count_error)) 815 | 816 | # ReadMemory requires that count is positive, so early-return an empty 817 | # byte array when count is 0. 818 | if count == 0: 819 | return bytes() 820 | 821 | get_data_error = lldb.SBError() 822 | data = self.process.ReadMemory(position, count, get_data_error) 823 | if get_data_error.Fail(): 824 | raise Exception('getting data: %s' % str(get_data_error)) 825 | 826 | return data 827 | 828 | def _send_jupyter_messages(self, messages): 829 | for display_message in messages['display_messages']: 830 | self.iopub_socket.send_multipart(display_message) 831 | 832 | def _set_parent_message(self): 833 | result = self._execute(""" 834 | JupyterKernel.communicator.updateParentMessage( 835 | to: KernelCommunicator.ParentMessage(json: %s)) 836 | """ % json.dumps(json.dumps(squash_dates(self._parent_header)))) 837 | if isinstance(result, ExecutionResultError): 838 | raise Exception('Error setting parent message: %s' % result) 839 | 840 | def _get_pretty_main_thread_stack_trace(self): 841 | stack_trace = [] 842 | for frame in self.main_thread: 843 | # Do not include frames without source location information. These 844 | # are frames in libraries and frames that belong to the LLDB 845 | # expression execution implementation. 846 | if not frame.line_entry.file: 847 | continue 848 | # Do not include frames. These are 849 | # specializations of library functions. 850 | if frame.line_entry.file.fullpath == '': 851 | continue 852 | stack_trace.append(str(frame)) 853 | return stack_trace 854 | 855 | def _make_error_message(self, traceback): 856 | return { 857 | 'status': 'error', 858 | 'execution_count': self.execution_count, 859 | 'ename': '', 860 | 'evalue': '', 861 | 'traceback': traceback 862 | } 863 | 864 | def _send_exception_report(self, while_doing, e): 865 | error_message = self._make_error_message([ 866 | 'Kernel is in a bad state. Try restarting the kernel.', 867 | '', 868 | 'Exception in `%s`:' % while_doing, 869 | str(e) 870 | ]) 871 | self.send_response(self.iopub_socket, 'error', error_message) 872 | return error_message 873 | 874 | def _execute_cell(self, code): 875 | self._set_parent_message() 876 | result = self._preprocess_and_execute(code) 877 | if isinstance(result, ExecutionResultSuccess): 878 | self._after_successful_execution() 879 | return result 880 | 881 | def do_execute(self, code, silent, store_history=True, 882 | user_expressions=None, allow_stdin=False): 883 | 884 | # Return early if the code is empty or whitespace, to avoid 885 | # initializing Swift and preventing package installs. 886 | if len(code) == 0 or code.isspace(): 887 | return { 888 | 'status': 'ok', 889 | 'execution_count': self.execution_count, 890 | 'payload': [], 891 | 'user_expressions': {} 892 | } 893 | 894 | # Package installs must be done before initializing Swift (see doc 895 | # comment in `_init_swift`). 896 | try: 897 | code = self._process_installs(code) 898 | except PackageInstallException as e: 899 | error_message = self._make_error_message([str(e)]) 900 | self.send_response(self.iopub_socket, 'error', error_message) 901 | return error_message 902 | except Exception as e: 903 | self._send_exception_report('_process_installs', e) 904 | raise e 905 | 906 | if not hasattr(self, 'debugger'): 907 | self._init_swift() 908 | 909 | # Start up a new thread to collect stdout. 910 | stdout_handler = StdoutHandler(self) 911 | stdout_handler.start() 912 | 913 | # Execute the cell, handle unexpected exceptions, and make sure to 914 | # always clean up the stdout handler. 915 | try: 916 | result = self._execute_cell(code) 917 | except Exception as e: 918 | self._send_exception_report('_execute_cell', e) 919 | raise e 920 | finally: 921 | stdout_handler.stop_event.set() 922 | stdout_handler.join() 923 | 924 | # Send values/errors and status to the client. 925 | if isinstance(result, SuccessWithValue): 926 | self.send_response(self.iopub_socket, 'execute_result', { 927 | 'execution_count': self.execution_count, 928 | 'data': { 929 | 'text/plain': result.result.description 930 | }, 931 | 'metadata': {} 932 | }) 933 | return { 934 | 'status': 'ok', 935 | 'execution_count': self.execution_count, 936 | 'payload': [], 937 | 'user_expressions': {} 938 | } 939 | elif isinstance(result, SuccessWithoutValue): 940 | return { 941 | 'status': 'ok', 942 | 'execution_count': self.execution_count, 943 | 'payload': [], 944 | 'user_expressions': {} 945 | } 946 | elif isinstance(result, ExecutionResultError): 947 | if not self.process.is_alive: 948 | error_message = self._make_error_message(['Process killed']) 949 | self.send_response(self.iopub_socket, 'error', error_message) 950 | 951 | # Exit the kernel because there is no way to recover from a 952 | # killed process. The UI will tell the user that the kernel has 953 | # died and the UI will automatically restart the kernel. 954 | # We do the exit in a callback so that this execute request can 955 | # cleanly finish before the kernel exits. 956 | loop = ioloop.IOLoop.current() 957 | loop.add_timeout(time.time()+0.1, loop.stop) 958 | 959 | return error_message 960 | 961 | if stdout_handler.had_stdout: 962 | # When there is stdout, it is a runtime error. Stdout, which we 963 | # have already sent to the client, contains the error message 964 | # (plus some other ugly traceback that we should eventually 965 | # figure out how to suppress), so this block of code only needs 966 | # to add a traceback. 967 | traceback = [] 968 | traceback.append('Current stack trace:') 969 | traceback += [ 970 | '\t%s' % frame 971 | for frame in self._get_pretty_main_thread_stack_trace() 972 | ] 973 | 974 | error_message = self._make_error_message(traceback) 975 | self.send_response(self.iopub_socket, 'error', error_message) 976 | return error_message 977 | 978 | # There is no stdout, so it must be a compile error. Simply return 979 | # the error without trying to get a stack trace. 980 | error_message = self._make_error_message([result.description()]) 981 | self.send_response(self.iopub_socket, 'error', error_message) 982 | return error_message 983 | 984 | def do_complete(self, code, cursor_pos): 985 | if not self.completion_enabled: 986 | return { 987 | 'status': 'ok', 988 | 'matches': [], 989 | 'cursor_start': cursor_pos, 990 | 'cursor_end': cursor_pos, 991 | } 992 | 993 | code_to_cursor = code[:cursor_pos] 994 | sbresponse = self.target.CompleteCode( 995 | self.swift_language, None, code_to_cursor) 996 | prefix = sbresponse.GetPrefix() 997 | insertable_matches = [] 998 | for i in range(sbresponse.GetNumMatches()): 999 | sbmatch = sbresponse.GetMatchAtIndex(i) 1000 | insertable_match = prefix + sbmatch.GetInsertable() 1001 | if insertable_match.startswith("_"): 1002 | continue 1003 | insertable_matches.append(insertable_match) 1004 | return { 1005 | 'status': 'ok', 1006 | 'matches': insertable_matches, 1007 | 'cursor_start': cursor_pos - len(prefix), 1008 | 'cursor_end': cursor_pos, 1009 | } 1010 | 1011 | if __name__ == '__main__': 1012 | # Jupyter sends us SIGINT when the user requests execution interruption. 1013 | # Here, we block all threads from receiving the SIGINT, so that we can 1014 | # handle it in a specific handler thread. 1015 | signal.pthread_sigmask(signal.SIG_BLOCK, [signal.SIGINT]) 1016 | 1017 | from ipykernel.kernelapp import IPKernelApp 1018 | # We pass the kernel name as a command-line arg, since Jupyter gives those 1019 | # highest priority (in particular overriding any system-wide config). 1020 | IPKernelApp.launch_instance( 1021 | argv=sys.argv + ['--IPKernelApp.kernel_class=__main__.SwiftKernel']) 1022 | --------------------------------------------------------------------------------