├── tests ├── __init__.py ├── resources │ ├── utils.py │ ├── doctest_failed.ipynb │ ├── unittest_failed.ipynb │ ├── runtime_error.ipynb │ └── basic.ipynb ├── resources-2 │ ├── a1.ipynb │ └── b1.ipynb ├── test_execution.py └── test_treon.py ├── treon ├── __init__.py ├── __main__.py ├── task.py ├── test_execution.py └── treon.py ├── MANIFEST.in ├── requirements-dev.txt ├── .pylintrc ├── Makefile ├── images ├── doctest.png └── unittest.png ├── .travis.yml ├── LICENSE ├── setup.py ├── .gitignore └── README.md /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /treon/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include LICENSE -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | pylint 2 | pytest 3 | -------------------------------------------------------------------------------- /.pylintrc: -------------------------------------------------------------------------------- 1 | [MESSAGE CONTROL] 2 | disable=missing-docstring 3 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | 2 | lint: 3 | pylint treon tests 4 | 5 | test: 6 | pytest tests 7 | -------------------------------------------------------------------------------- /images/doctest.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ReviewNB/treon/HEAD/images/doctest.png -------------------------------------------------------------------------------- /images/unittest.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ReviewNB/treon/HEAD/images/unittest.png -------------------------------------------------------------------------------- /treon/__main__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | 4 | from .treon import main 5 | main() 6 | -------------------------------------------------------------------------------- /tests/resources/utils.py: -------------------------------------------------------------------------------- 1 | # -*- encoding: utf-8 -*- 2 | 3 | 4 | def guess_number(): 5 | """If notebooks can import this function, the Python path is set up correctly""" 6 | return 42 7 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - "3.5" 4 | - "3.6" 5 | 6 | install: 7 | - pip install . 8 | - pip install -r requirements-dev.txt 9 | 10 | script: 11 | - make lint 12 | - make test 13 | -------------------------------------------------------------------------------- /tests/resources-2/a1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "" 12 | ] 13 | } 14 | ], 15 | "metadata": { 16 | "kernelspec": { 17 | "display_name": "Python 3", 18 | "language": "python", 19 | "name": "python3" 20 | }, 21 | "language_info": { 22 | "codemirror_mode": { 23 | "name": "ipython", 24 | "version": 2 25 | }, 26 | "file_extension": ".py", 27 | "mimetype": "text/x-python", 28 | "name": "python", 29 | "nbconvert_exporter": "python", 30 | "pygments_lexer": "ipython2", 31 | "version": "2.7.6" 32 | } 33 | }, 34 | "nbformat": 4, 35 | "nbformat_minor": 0 36 | } -------------------------------------------------------------------------------- /tests/resources-2/b1.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "" 12 | ] 13 | } 14 | ], 15 | "metadata": { 16 | "kernelspec": { 17 | "display_name": "Python 3", 18 | "language": "python", 19 | "name": "python3" 20 | }, 21 | "language_info": { 22 | "codemirror_mode": { 23 | "name": "ipython", 24 | "version": 2 25 | }, 26 | "file_extension": ".py", 27 | "mimetype": "text/x-python", 28 | "name": "python", 29 | "nbconvert_exporter": "python", 30 | "pygments_lexer": "ipython2", 31 | "version": "2.7.6" 32 | } 33 | }, 34 | "nbformat": 4, 35 | "nbformat_minor": 0 36 | } -------------------------------------------------------------------------------- /tests/resources/doctest_failed.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "def function_failed_doctest():\n", 10 | " \"\"\"\n", 11 | " >>> 1 + 1\n", 12 | " 42\n", 13 | " \"\"\"\n", 14 | " pass" 15 | ] 16 | } 17 | ], 18 | "metadata": { 19 | "kernelspec": { 20 | "display_name": "Python 3", 21 | "language": "python", 22 | "name": "python3" 23 | }, 24 | "language_info": { 25 | "codemirror_mode": { 26 | "name": "ipython", 27 | "version": 3 28 | }, 29 | "file_extension": ".py", 30 | "mimetype": "text/x-python", 31 | "name": "python", 32 | "nbconvert_exporter": "python", 33 | "pygments_lexer": "ipython3", 34 | "version": "3.6.3" 35 | } 36 | }, 37 | "nbformat": 4, 38 | "nbformat_minor": 2 39 | } 40 | -------------------------------------------------------------------------------- /tests/resources/unittest_failed.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import unittest\n", 10 | "\n", 11 | "class FailingTest(unittest.TestCase):\n", 12 | " \n", 13 | " def test_fail(self):\n", 14 | " self.assertEqual(-1, 1)\n" 15 | ] 16 | } 17 | ], 18 | "metadata": { 19 | "kernelspec": { 20 | "display_name": "Python 3", 21 | "language": "python", 22 | "name": "python3" 23 | }, 24 | "language_info": { 25 | "codemirror_mode": { 26 | "name": "ipython", 27 | "version": 3 28 | }, 29 | "file_extension": ".py", 30 | "mimetype": "text/x-python", 31 | "name": "python", 32 | "nbconvert_exporter": "python", 33 | "pygments_lexer": "ipython3", 34 | "version": "3.6.3" 35 | } 36 | }, 37 | "nbformat": 4, 38 | "nbformat_minor": 2 39 | } 40 | -------------------------------------------------------------------------------- /tests/test_execution.py: -------------------------------------------------------------------------------- 1 | # -*- encoding: utf-8 -*- 2 | import os 3 | import pytest 4 | 5 | from nbconvert.preprocessors import CellExecutionError 6 | from treon.test_execution import execute_notebook 7 | 8 | 9 | def test_successful_execution(): 10 | successful, _ = _run('resources/basic.ipynb') 11 | 12 | assert successful 13 | 14 | 15 | def test_failed_execution(): 16 | with pytest.raises(CellExecutionError) as exc_info: 17 | _run('resources/runtime_error.ipynb') 18 | 19 | assert 'ZeroDivisionError' in exc_info.value.traceback 20 | 21 | 22 | def test_failed_unittest(): 23 | successful, output = _run('resources/unittest_failed.ipynb') 24 | 25 | assert not successful 26 | assert 'AssertionError' in output 27 | 28 | 29 | def test_failed_doctest(): 30 | successful, output = _run('resources/doctest_failed.ipynb') 31 | 32 | assert not successful 33 | assert 'Test Failed' in output 34 | 35 | 36 | def _run(notebook): 37 | path = os.path.join(os.path.dirname(os.path.abspath(__file__)), notebook) 38 | return execute_notebook(path) 39 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 ReviewNB 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | try: 2 | from setuptools import setup 3 | except ImportError: 4 | from distutils.core import setup 5 | 6 | import re 7 | 8 | version = re.search( 9 | '^__version__\s*=\s*"(.*)"', 10 | open('treon/treon.py').read(), 11 | re.M 12 | ).group(1) 13 | 14 | 15 | with open("README.md", "rb") as f: 16 | long_descr = f.read().decode("utf-8") 17 | 18 | 19 | setup( 20 | name = "treon", 21 | packages = ["treon"], 22 | python_requires='>=3', 23 | entry_points = { 24 | "console_scripts": ['treon = treon.treon:main'] 25 | }, 26 | version = version, 27 | description = "Testing framework for Jupyter Notebooks", 28 | long_description = long_descr, 29 | long_description_content_type="text/markdown", 30 | author = "Amit Rathi", 31 | author_email = "amit@reviewnb.com", 32 | url = "https://github.com/reviewNB/treon", 33 | license='MIT', 34 | keywords=['test', 'jupyter', 'notebook', 'jupyter test', 'notebook test', 'unittest', 'doctest'], 35 | install_requires=[ 36 | 'nbconvert', 37 | 'jupyter_client', 38 | 'jupyter', 39 | 'docopt' 40 | ] 41 | ) 42 | -------------------------------------------------------------------------------- /tests/resources/runtime_error.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "This notebook will not execute successfully." 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": {}, 14 | "outputs": [ 15 | { 16 | "ename": "ZeroDivisionError", 17 | "evalue": "division by zero", 18 | "output_type": "error", 19 | "traceback": [ 20 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 21 | "\u001b[0;31mZeroDivisionError\u001b[0m Traceback (most recent call last)", 22 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;36m1\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", 23 | "\u001b[0;31mZeroDivisionError\u001b[0m: division by zero" 24 | ] 25 | } 26 | ], 27 | "source": [ 28 | "1 / 0" 29 | ] 30 | } 31 | ], 32 | "metadata": { 33 | "kernelspec": { 34 | "display_name": "Python 3", 35 | "language": "python", 36 | "name": "python3" 37 | }, 38 | "language_info": { 39 | "codemirror_mode": { 40 | "name": "ipython", 41 | "version": 3 42 | }, 43 | "file_extension": ".py", 44 | "mimetype": "text/x-python", 45 | "name": "python", 46 | "nbconvert_exporter": "python", 47 | "pygments_lexer": "ipython3", 48 | "version": "3.6.3" 49 | } 50 | }, 51 | "nbformat": 4, 52 | "nbformat_minor": 2 53 | } 54 | -------------------------------------------------------------------------------- /treon/task.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import textwrap 3 | 4 | from nbconvert.preprocessors import CellExecutionError 5 | from nbconvert.utils.exceptions import ConversionException 6 | 7 | from .test_execution import execute_notebook 8 | 9 | LOG = logging.getLogger('treon.task') 10 | 11 | 12 | def _is_verbose(): 13 | return LOG.isEnabledFor(logging.DEBUG) 14 | 15 | 16 | class Task: 17 | def __init__(self, file_path): 18 | self.file_path = file_path 19 | self.is_successful = False 20 | 21 | def run_tests(self): 22 | LOG.info("Triggered test for %s", self.file_path) 23 | 24 | try: 25 | self.is_successful, console_output = execute_notebook(self.file_path) 26 | result = self.result_string() 27 | 28 | if not self.is_successful or _is_verbose(): 29 | result += console_output 30 | 31 | LOG.info(result) 32 | except CellExecutionError as cell_error: 33 | LOG.error(self.error_string(cell_error.traceback)) 34 | except ConversionException: 35 | LOG.exception("Execution of notebook '%s' failed", self.file_path) 36 | 37 | def result_string(self): 38 | if self.is_successful: 39 | return '\n{file_path} -- PASSED \n'.format(file_path=self.file_path) 40 | 41 | return '\n{file_path} -- FAILED \n'.format(file_path=self.file_path) 42 | 43 | def error_string(self, stack_trace): 44 | variables = { 45 | 'file_path': self.file_path, 46 | 'stack_trace': stack_trace 47 | } 48 | error_string = textwrap.dedent("""\ 49 | ERROR in testing {file_path} 50 | 51 | {stack_trace} 52 | \n 53 | """).format(**variables) 54 | 55 | return error_string 56 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | pip-wheel-metadata/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | 53 | # Translations 54 | *.mo 55 | *.pot 56 | 57 | # Django stuff: 58 | *.log 59 | local_settings.py 60 | db.sqlite3 61 | db.sqlite3-journal 62 | 63 | # Flask stuff: 64 | instance/ 65 | .webassets-cache 66 | 67 | # Scrapy stuff: 68 | .scrapy 69 | 70 | # Sphinx documentation 71 | docs/_build/ 72 | 73 | # PyBuilder 74 | target/ 75 | 76 | # Jupyter Notebook 77 | .ipynb_checkpoints 78 | 79 | # IPython 80 | profile_default/ 81 | ipython_config.py 82 | 83 | # pyenv 84 | .python-version 85 | 86 | # pipenv 87 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 88 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 89 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 90 | # install all needed dependencies. 91 | Pipfile.lock 92 | 93 | # celery beat schedule file 94 | celerybeat-schedule 95 | 96 | # SageMath parsed files 97 | *.sage.py 98 | 99 | # Environments 100 | .env 101 | .venv 102 | env/ 103 | venv/ 104 | ENV/ 105 | env.bak/ 106 | venv.bak/ 107 | 108 | # Spyder project settings 109 | .spyderproject 110 | .spyproject 111 | 112 | # Rope project settings 113 | .ropeproject 114 | 115 | # mkdocs documentation 116 | /site 117 | 118 | # mypy 119 | .mypy_cache/ 120 | .dmypy.json 121 | dmypy.json 122 | 123 | # Pyre type checker 124 | .pyre/ 125 | 126 | .DS_Store 127 | .vscode 128 | -------------------------------------------------------------------------------- /tests/resources/basic.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Basic Notebook\n", 8 | "\n", 9 | "This notebook showcases normal execution within treon, without errors." 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 5, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "import utils # Test notebook-relative import\n", 19 | "assert utils.guess_number() == 42" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 1, 25 | "metadata": {}, 26 | "outputs": [ 27 | { 28 | "name": "stdout", 29 | "output_type": "stream", 30 | "text": [ 31 | "hello world\n" 32 | ] 33 | } 34 | ], 35 | "source": [ 36 | "print('hello world')" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 3, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "assert 1 + 2 == 3" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 2, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "import unittest\n", 55 | "\n", 56 | "class SpamTest(unittest.TestCase):\n", 57 | " \n", 58 | " def test_eggs(self):\n", 59 | " assert 2 + 2 == 4" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 4, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "def function_with_doctests():\n", 69 | " \"\"\"\n", 70 | " >>> 1 + 2\n", 71 | " 3\n", 72 | " >>> 1 / 0\n", 73 | " Traceback (most recent call last):\n", 74 | " ZeroDivisionError: division by zero\n", 75 | " \"\"\"\n", 76 | " pass" 77 | ] 78 | } 79 | ], 80 | "metadata": { 81 | "kernelspec": { 82 | "display_name": "Python 3", 83 | "language": "python", 84 | "name": "python3" 85 | }, 86 | "language_info": { 87 | "codemirror_mode": { 88 | "name": "ipython", 89 | "version": 3 90 | }, 91 | "file_extension": ".py", 92 | "mimetype": "text/x-python", 93 | "name": "python", 94 | "nbconvert_exporter": "python", 95 | "pygments_lexer": "ipython3", 96 | "version": "3.7.3" 97 | } 98 | }, 99 | "nbformat": 4, 100 | "nbformat_minor": 2 101 | } 102 | -------------------------------------------------------------------------------- /treon/test_execution.py: -------------------------------------------------------------------------------- 1 | import os 2 | import textwrap 3 | import nbformat 4 | 5 | from nbformat.v4 import new_code_cell 6 | from nbconvert.preprocessors import ExecutePreprocessor 7 | 8 | 9 | def execute_notebook(path): 10 | notebook = nbformat.read(path, as_version=4) 11 | notebook.cells.extend([unittest_cell(), doctest_cell()]) 12 | processor = ExecutePreprocessor(timeout=-1, kernel_name='python3') 13 | processor.preprocess(notebook, metadata(path)) 14 | return parse_test_result(notebook.cells) 15 | 16 | 17 | def metadata(path): 18 | return {'metadata': { 19 | 'path': os.path.dirname(path) 20 | }} 21 | 22 | 23 | def parse_test_result(cells): 24 | is_successful, console_output = True, '' 25 | unittest_output = cells[-2].outputs 26 | doctest_output = cells[-1].outputs 27 | 28 | if unittest_output: 29 | has_unittest_passed, unittest_output = parse_unittest_output(unittest_output) 30 | is_successful = is_successful and has_unittest_passed 31 | console_output += unittest_output 32 | 33 | if doctest_output: 34 | has_doctest_passed, doctest_output = parse_doctest_output(doctest_output) 35 | is_successful = is_successful and has_doctest_passed 36 | console_output += doctest_output 37 | 38 | return is_successful, console_output 39 | 40 | 41 | def parse_unittest_output(outputs): 42 | has_passed, text = False, '' 43 | 44 | for output in outputs: 45 | text += output.text + '\n' 46 | 47 | if 'OK' in text[-25:]: 48 | has_passed = True 49 | 50 | return has_passed, text 51 | 52 | 53 | def parse_doctest_output(outputs): 54 | has_passed, text = False, '' 55 | 56 | for output in outputs: 57 | text += output.text + '\n' 58 | 59 | if 'Test passed.' in text[-25:]: 60 | has_passed = True 61 | 62 | return has_passed, text 63 | 64 | 65 | def unittest_cell(): 66 | source = textwrap.dedent(""" 67 | from IPython.display import clear_output 68 | import unittest 69 | 70 | r = unittest.main(argv=[''], verbosity=2, exit=False) 71 | 72 | if r.result.testsRun == 0: 73 | clear_output() 74 | """) 75 | return new_code_cell(source=source) 76 | 77 | 78 | def doctest_cell(): 79 | source = textwrap.dedent(""" 80 | from IPython.display import clear_output 81 | import doctest 82 | 83 | r = doctest.testmod(verbose=True) 84 | 85 | if r.attempted == 0: 86 | clear_output() 87 | """) 88 | return new_code_cell(source=source) 89 | -------------------------------------------------------------------------------- /tests/test_treon.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import mock 3 | from treon import treon 4 | 5 | def test_filter_results_file(): 6 | args = { 7 | "--exclude": ['resources/basic.ipynb', 8 | 'failed'] 9 | } 10 | results = ['resources/basic.ipynb', 11 | 'resources/doctest_failed.ipynb', 12 | 'resources/runtime_error.ipynb', 13 | 'resources/unittest_failed.ipynb', 14 | 'other/resources.ipynb'] 15 | 16 | filtered = treon.filter_results(results=results, args=args) 17 | expected = ['resources/doctest_failed.ipynb', 18 | 'resources/runtime_error.ipynb', 19 | 'resources/unittest_failed.ipynb', 20 | 'other/resources.ipynb'] 21 | assert filtered == expected 22 | 23 | 24 | def test_filter_results_folder(): 25 | args = {"--exclude": ['resources']} 26 | results = ['resources/basic.ipynb', 27 | 'resources/doctest_failed.ipynb', 28 | 'resources/runtime_error.ipynb', 29 | 'resources/unittest_failed.ipynb', 30 | 'other/resources.ipynb'] 31 | 32 | filtered = treon.filter_results(results=results, args=args) 33 | expected = ['other/resources.ipynb'] 34 | assert filtered == expected 35 | 36 | 37 | def test_filter_results_empty(): 38 | args = {"--exclude": ['resources']} 39 | results = ['resources/basic.ipynb'] 40 | filtered = treon.filter_results(results=results, args=args) 41 | expected = [] 42 | assert filtered == expected 43 | 44 | 45 | def test_filter_results_homedir(): 46 | args = {"--exclude": ['~/resources']} 47 | results = [os.path.join(os.path.expanduser("~"), "resources/basic.ipynb")] 48 | filtered = treon.filter_results(results=results, args=args) 49 | expected = [] 50 | assert filtered == expected 51 | 52 | 53 | @mock.patch('os.path.isdir') 54 | def test_filter_results_exclude_is_dir(mock_isdir): 55 | mock_isdir.return_value = True 56 | args = {"--exclude": ["./notebook"]} 57 | results = ["./notebook/1.pynb", "./notebook2/1.pynb"] 58 | filtered = treon.filter_results(results=results, args=args) 59 | expected = ["./notebook2/1.pynb"] 60 | assert filtered == expected 61 | 62 | 63 | @mock.patch('os.path.isdir') 64 | def test_filter_results_exclude_is_not_dir(mock_isdir): 65 | mock_isdir.return_value = False 66 | args = {"--exclude": ["./notebook"]} 67 | results = ["./notebook1/1.pynb", "./notebook2/1.pynb"] 68 | filtered = treon.filter_results(results=results, args=args) 69 | expected = [] 70 | assert filtered == expected 71 | 72 | 73 | def test_get_notebooks_to_test_with_multiple_paths(): 74 | notebook_files = [ 75 | os.path.join(os.path.dirname(__file__), "resources/basic.ipynb"), 76 | os.path.join(os.path.dirname(__file__), "resources/unittest_failed.ipynb"), 77 | ] 78 | args = { 79 | "PATH": list(notebook_files), 80 | "--exclude": [], 81 | } 82 | notebooks = treon.get_notebooks_to_test(args=args) 83 | expected = notebook_files 84 | assert notebooks == expected 85 | 86 | 87 | def test_get_notebooks_to_test_with_multiple_dir_paths(): 88 | cwd = os.path.dirname(__file__) 89 | notebook_folders = [ 90 | os.path.join(cwd, "resources"), 91 | os.path.join(cwd, "resources-2") 92 | ] 93 | expected = [os.path.join(cwd, 'resources/unittest_failed.ipynb'), 94 | os.path.join(cwd, 'resources/runtime_error.ipynb'), 95 | os.path.join(cwd, 'resources/basic.ipynb'), 96 | os.path.join(cwd, 'resources/doctest_failed.ipynb'), 97 | os.path.join(cwd, 'resources-2/a1.ipynb'), 98 | os.path.join(cwd, 'resources-2/b1.ipynb')] 99 | args = { 100 | "PATH": list(notebook_folders), 101 | "--exclude": [], 102 | } 103 | notebooks = treon.get_notebooks_to_test(args=args) 104 | assert notebooks == expected 105 | 106 | 107 | def test_get_notebooks_to_test_with_multiple_dir_paths_with_exclude_files(): 108 | cwd = os.path.dirname(__file__) 109 | notebook_folders = [ 110 | os.path.join(cwd, "resources"), 111 | os.path.join(cwd, "resources-2") 112 | ] 113 | expected = [os.path.join(cwd, 'resources/unittest_failed.ipynb'), 114 | os.path.join(cwd, 'resources/runtime_error.ipynb'), 115 | os.path.join(cwd, 'resources/basic.ipynb'), 116 | os.path.join(cwd, 'resources-2/a1.ipynb')] 117 | args = { 118 | "PATH": list(notebook_folders), 119 | "--exclude": [os.path.join(cwd, 'resources/doctest_failed.ipynb'), 120 | os.path.join(cwd, 'resources-2/b1.ipynb')] 121 | } 122 | notebooks = treon.get_notebooks_to_test(args=args) 123 | assert notebooks == expected 124 | 125 | 126 | def test_get_notebooks_to_test_with_multiple_dir_paths_with_exclude_folder(): 127 | cwd = os.path.dirname(__file__) 128 | notebook_folders = [ 129 | os.path.join(cwd, "resources"), 130 | os.path.join(cwd, "resources-2") 131 | ] 132 | expected = [os.path.join(cwd, 'resources-2/a1.ipynb'), 133 | os.path.join(cwd, 'resources-2/b1.ipynb')] 134 | args = { 135 | "PATH": list(notebook_folders), 136 | "--exclude": [os.path.join(cwd, 'resources')], 137 | } 138 | notebooks = treon.get_notebooks_to_test(args=args) 139 | assert notebooks == expected 140 | -------------------------------------------------------------------------------- /treon/treon.py: -------------------------------------------------------------------------------- 1 | # pylint: disable=line-too-long 2 | """ 3 | Usage: 4 | treon 5 | treon [PATH]... [--threads=] [-v] [--exclude=]... 6 | 7 | Arguments: 8 | PATH File or directory path to find notebooks to test. Searches recursively for directory paths. [default: current working directory] 9 | 10 | Options: 11 | --threads= Number of parallel threads. Each thread processes one notebook file at a time. [default: 10] 12 | -e= --exclude= Option for excluding files or entire directories from testing. All files whose 13 | absolute path starts with the specified string are excluded from testing. This option can be 14 | specified more than once to exclude multiple files or directories. If the exclude path is 15 | a valid directory name, only this directory is excluded. 16 | -v --verbose Print detailed output for debugging. 17 | -h --help Show this screen. 18 | --version Show version. 19 | 20 | """ 21 | 22 | __version__ = "0.1.4" 23 | 24 | 25 | import sys 26 | import os 27 | import glob 28 | import logging 29 | import textwrap 30 | from multiprocessing.dummy import Pool as ThreadPool 31 | from docopt import docopt, DocoptExit 32 | 33 | from .task import Task 34 | 35 | DEFAULT_THREAD_COUNT = 10 36 | 37 | LOG = logging.getLogger('treon') 38 | 39 | 40 | def main(): 41 | try: 42 | arguments = docopt(__doc__, version=__version__) 43 | except DocoptExit: 44 | sys.exit(__doc__) 45 | 46 | setup_logging(arguments) 47 | LOG.info('Executing treon version %s', __version__) 48 | thread_count = arguments['--threads'] or DEFAULT_THREAD_COUNT 49 | notebooks = get_notebooks_to_test(arguments) 50 | tasks = [Task(notebook) for notebook in notebooks] 51 | print_test_collection(notebooks) 52 | trigger_tasks(tasks, thread_count) 53 | has_failed = print_test_result(tasks) 54 | 55 | if has_failed: 56 | sys.exit(-1) 57 | 58 | 59 | def setup_logging(arguments): 60 | logging.basicConfig(format='%(message)s') 61 | LOG.setLevel(loglevel(arguments)) 62 | 63 | 64 | def loglevel(arguments): 65 | verbose = arguments['--verbose'] 66 | return logging.DEBUG if verbose else logging.INFO 67 | 68 | 69 | def trigger_tasks(tasks, thread_count): 70 | pool = ThreadPool(int(thread_count)) 71 | pool.map(Task.run_tests, tasks) 72 | 73 | 74 | def print_test_result(tasks): 75 | has_failed = False 76 | succeeded = [t.file_path for t in tasks if t.is_successful] 77 | failed = [t.file_path for t in tasks if not t.is_successful] 78 | variables = { 79 | 'succeeded_count': len(succeeded), 80 | 'failed_count': len(failed), 81 | 'total': len(tasks) 82 | } 83 | 84 | message = textwrap.dedent(""" 85 | ----------------------------------------------------------------------- 86 | TEST RESULT 87 | -----------------------------------------------------------------------\n""") 88 | 89 | if succeeded: 90 | message += ' -- PASSED \n'.join(succeeded) + ' -- PASSED \n' 91 | 92 | if failed: 93 | has_failed = True 94 | message += ' -- FAILED \n'.join(failed) + ' -- FAILED \n' 95 | 96 | message += '-----------------------------------------------------------------------\n' 97 | message += '{succeeded_count} succeeded, {failed_count} failed, out of {total} notebooks tested.\n'.format(**variables) 98 | message += '-----------------------------------------------------------------------\n' 99 | LOG.info(message) 100 | return has_failed 101 | 102 | 103 | def print_test_collection(notebooks): 104 | message = textwrap.dedent(""" 105 | ----------------------------------------------------------------------- 106 | Collected following Notebooks for testing 107 | -----------------------------------------------------------------------\n""") 108 | message += '\n'.join(notebooks) + '\n' 109 | message += '-----------------------------------------------------------------------\n' 110 | LOG.debug(message) 111 | 112 | 113 | def filter_results(results, args): 114 | for exclude_str in args['--exclude']: 115 | exclude_abs = os.path.abspath(os.path.expanduser(exclude_str)) 116 | if os.path.isdir(exclude_abs): 117 | exclude_abs += os.sep 118 | results = [file for file in results if not os.path.abspath(file).startswith(exclude_abs)] 119 | return results 120 | 121 | 122 | def get_notebooks_to_test(args): 123 | paths = args['PATH'] or [os.getcwd()] 124 | result = [] 125 | 126 | for path in paths: 127 | current_result = [] 128 | 129 | if os.path.isdir(path): 130 | LOG.info('Recursively scanning %s for notebooks...', path) 131 | path = os.path.join(path, '') # adds trailing slash (/) if it's missing 132 | glob_path = path + '**/*.ipynb' 133 | current_result = glob.glob(glob_path, recursive=True) 134 | elif os.path.isfile(path): 135 | if path.lower().endswith('.ipynb'): 136 | LOG.debug('Testing notebook %s', path) 137 | current_result = [path] 138 | else: 139 | sys.exit('{path} is not a Notebook'.format(path=path)) 140 | else: 141 | sys.exit('{path} is not a valid path'.format(path=path)) 142 | 143 | if not current_result: 144 | sys.exit('No notebooks to test in {path}'.format(path=path)) 145 | 146 | result.extend(current_result) 147 | 148 | return filter_results(result, args) 149 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # treon 2 | 3 | [![PyPI version](https://badge.fury.io/py/treon.svg)](https://badge.fury.io/py/treon) 4 | [![Build Status](https://travis-ci.org/ReviewNB/treon.svg?branch=master)](https://travis-ci.org/ReviewNB/treon) 5 | 6 | Easy to use test framework for Jupyter Notebooks. 7 | * Runs notebook top to bottom and flags execution errors if any 8 | * Runs [unittest](https://docs.python.org/2/library/unittest.html) present in your notebook code cells 9 | * Runs [doctest](https://docs.python.org/2/library/doctest.html) present in your notebook code cells 10 | 11 | ### Why should you use it? 12 | * Start testing notebooks without writing a single line of test code 13 | * Multithreaded execution for quickly testing a set of notebooks 14 | * Executes every Notebook in a fresh kernel to avoid hidden state problems 15 | * Primarily a command line tool that can be used easily in any Continuous Integration (CI) system 16 | 17 | 18 | ## Installation 19 | ``` 20 | pip install treon 21 | ``` 22 | 23 | ## Usage 24 | Treon will execute notebook from top to bottom and the test fails if any code cell returns an error. Additionally, one can write unittest & doctest to test specific behaviour (examples shown below). 25 | 26 | ``` 27 | $ treon 28 | Executing treon version 0.1.4 29 | Recursively scanning /workspace/treon/tmp/docs/site/ru/guide for Notebooks... 30 | 31 | ----------------------------------------------------------------------- 32 | Collected following Notebooks for testing 33 | ----------------------------------------------------------------------- 34 | /workspace/treon/tmp/docs/site/ru/guide/keras.ipynb 35 | /workspace/treon/tmp/docs/site/ru/guide/eager.ipynb 36 | ----------------------------------------------------------------------- 37 | 38 | Triggered test for /workspace/treon/tmp/docs/site/ru/guide/keras.ipynb 39 | Triggered test for /workspace/treon/tmp/docs/site/ru/guide/eager.ipynb 40 | 41 | test_sum (__main__.TestNotebook) ... 42 | ok 43 | test_sum (__main__.TestNotebook2) ... 44 | ok 45 | test_sum (__main__.TestNotebook3) ... 46 | ok 47 | 48 | ---------------------------------------------------------------------- 49 | Ran 3 tests in 0.004s 50 | 51 | OK 52 | 53 | ----------------------------------------------------------------------- 54 | TEST RESULT 55 | ----------------------------------------------------------------------- 56 | /workspace/treon/tmp/docs/site/ru/guide/keras.ipynb -- PASSED 57 | /workspace/treon/tmp/docs/site/ru/guide/eager.ipynb -- PASSED 58 | ----------------------------------------------------------------------- 59 | 2 succeeded, 0 failed, out of 2 notebooks tested. 60 | ----------------------------------------------------------------------- 61 | ``` 62 | 63 | ## Command line arguments 64 | ``` 65 | Usage: 66 | treon 67 | treon [PATH] [--threads=] [-v] [--exclude=]... 68 | 69 | Arguments: 70 | PATH File or directory path to find notebooks to test. Searches recursively for directory paths. [default: current working directory] 71 | 72 | Options: 73 | --threads= Number of parallel threads. Each thread processes one notebook file at a time. [default: 10] 74 | -e= --exclude= Option for excluding files or entire directories from testing. All files whose 75 | absolute path starts with the specified string are excluded from testing. This option can be 76 | specified more than once to exclude multiple files or directories. If the exclude path is 77 | a valid directory name, only this directory is excluded. 78 | -v --verbose Print detailed output for debugging. 79 | -h --help Show this screen. 80 | --version Show version. 81 | ``` 82 | 83 | ## unittest example 84 | You just need to add tests as shown below & treon would execute them and report the result on the console. See [this](https://docs.python.org/2/library/unittest.html) for more details on how to write unittest. 85 | 86 | ![](images/unittest.png) 87 | 88 | ## doctest example 89 | You just need to add tests as shown below & treon would execute them and report the result on the console. See [this](https://docs.python.org/2/library/doctest.html) for more details on how to write doctest. 90 | 91 | ![](images/doctest.png) 92 | 93 | ## Note about dependencies 94 | * You need to run treon from environment (virtualenv/pipenv etc.) that has all the dependencies required for Notebooks under test 95 | * treon only works with python3+ environments and uses python3 kernel for executing notebooks 96 | 97 | ## Development 98 | For development, you may use below to create a Python interpreter that resides in `venv` in the current working directory, and to install all of treon's dependencies: 99 | 100 | ``` 101 | $ virtualenv venv 102 | $ source venv/bin/activate 103 | $ pip install -e . 104 | $ pip install -r requirements-dev.txt 105 | $ treon --help # should work 106 | ``` 107 | 108 | Because the script installs the package as editable, you can make changes in the source tree and use the `treon` command to immediately validate them. If this does not appear to work, check that you are using a the proper virtual environment, and that the package is indeed installed in editable mode: 109 | 110 | ``` 111 | $ which treon # should point into your virtualenv 112 | /path/to/my/venv/bin/treon 113 | $ pip list --local | grep treon # should point to the source tree 114 | treon 0.1.4 /workspace/treon 115 | ``` 116 | 117 | Please refer to the `Makefile` for supplementary development tasks. 118 | In particular, the following targets may be relevant when validating changes before committing: 119 | 120 | ``` 121 | $ make lint # check treon's source for code style errors 122 | $ make test # run all tests 123 | ``` 124 | 125 | ## Motivation 126 | Our aim at [ReviewNB](https://www.reviewnb.com/) is to make notebooks a first class entity in the production workflow. We've built a code review system for Notebooks. The next step is to [build a CI pipeline](https://github.com/ReviewNB/support/issues/19) & treon is the core tool in that effort. It is licensed liberally (MIT) & I foresee it being used as an independent tool as well. You can use it locally and/or integrate with CI system of your choice. 127 | 128 | For motivation, checkout [Netflix's blog](https://medium.com/netflix-techblog/scheduling-notebooks-348e6c14cfd6) to see how notebooks are graduating from scratchpad to a part of production workflow. 129 | 130 | ## Contribute 131 | If you see any problem, open an issue or send a pull request. You can write to team@reviewnb.com for any questions. 132 | --------------------------------------------------------------------------------