├── .gitignore ├── Index.ipynb ├── Part-1 ├── Embedding │ ├── Index.ipynb │ ├── embed_class_long.py │ ├── embed_class_short.py │ ├── embed_function.py │ ├── inprocess_qtconsole.py │ ├── inprocess_terminal.py │ ├── internal_ipkernel.py │ ├── ipkernel_qtapp.py │ └── ipkernel_wxapp.py ├── IPython Kernel │ ├── Animations Using clear_output.ipynb │ ├── Background Jobs.ipynb │ ├── Beyond Plain Python.ipynb │ ├── Capturing Output.ipynb │ ├── Cell Magics.ipynb │ ├── Custom Display Logic.ipynb │ ├── Index.ipynb │ ├── Input in the Notebook.ipynb │ ├── Plotting in the Notebook.ipynb │ ├── Rich Output.ipynb │ ├── Script Magics.ipynb │ ├── SymPy.ipynb │ ├── Terminal Usage.ipynb │ ├── Third Party Rich Output.ipynb │ ├── Working With External Code.ipynb │ ├── data │ │ └── flare.json │ ├── example-demo.py │ ├── gui │ │ ├── gui-glut.py │ │ ├── gui-gtk.py │ │ ├── gui-gtk3.py │ │ ├── gui-pyglet.py │ │ ├── gui-qt.py │ │ ├── gui-tk.py │ │ └── gui-wx.py │ ├── ipython-completion.bash │ ├── ipython-get-history.py │ ├── ipython-qtconsole.desktop │ └── ipython.desktop ├── Index.ipynb ├── Notebook │ ├── Configuring the Notebook and Server.ipynb │ ├── Connecting with the Qt Console.ipynb │ ├── Converting Notebooks With nbconvert.ipynb │ ├── Custom Keyboard Shortcuts.ipynb │ ├── Importing Notebooks.ipynb │ ├── Index.ipynb │ ├── JavaScript Notebook Extensions.ipynb │ ├── Notebook Basics.ipynb │ ├── Running Code.ipynb │ ├── Typesetting Equations.ipynb │ ├── Using nbconvert as a Library.ipynb │ ├── What is the Jupyter Notebook.ipynb │ ├── Working With Markdown Cells.ipynb │ ├── images │ │ ├── command_mode.png │ │ ├── dashboard_files_tab.png │ │ ├── dashboard_files_tab_btns.png │ │ ├── dashboard_files_tab_new.png │ │ ├── dashboard_files_tab_run.png │ │ ├── dashboard_running_tab.png │ │ ├── edit_mode.png │ │ ├── menubar_toolbar.png │ │ └── nbconvert_arch.png │ └── nbpackage │ │ ├── __init__.py │ │ ├── mynotebook.ipynb │ │ └── nbs │ │ ├── __init__.py │ │ └── other.ipynb ├── exercises │ ├── MonteCarloPi-solution.ipynb │ ├── MonteCarloPi.ipynb │ ├── NumericalChaos-solution.ipynb │ ├── NumericalChaos.ipynb │ ├── Wallis-Pi-solution.ipynb │ ├── Wallis-Pi.ipynb │ ├── WordFrequencies-solution.ipynb │ ├── WordFrequencies.ipynb │ ├── circle_integrate.png │ ├── circle_integrate.svg │ ├── data │ │ ├── CallRingingIn.wav │ │ ├── HISTORY.gz │ │ ├── dessert.png │ │ ├── moon_denoise.png │ │ ├── moonlanding.png │ │ ├── stained_glass_barcelona.png │ │ └── stations.txt │ ├── fourier_introduction-solution.ipynb │ ├── fourier_introduction.ipynb │ ├── fourier_moon_denoise-solution.ipynb │ ├── fourier_moon_denoise.ipynb │ ├── fourier_sound_compress.ipynb │ ├── mapping_seismic_stations-solution.ipynb │ ├── mapping_seismic_stations.ipynb │ ├── mapping_seismic_stations_interactive.ipynb │ ├── sound_wavfiles.ipynb │ └── understanding_images.ipynb └── images │ ├── FrontendKernel.graffle │ ├── data.plist │ └── image1.png │ ├── FrontendKernel.png │ ├── animation.m4v │ ├── ipython_logo.png │ └── python_logo.svg ├── Part-2 ├── Index.ipynb ├── Nbviewer.ipynb ├── Notebook file format.ipynb ├── Using Nbconvert.ipynb ├── images │ ├── jupyter_logo.png │ ├── nbconvert_file_menu.png │ ├── nbval_cell_tag.png │ └── nbviewer_download_link.png ├── nbconvert_templates │ ├── Nbconvert templates.ipynb │ ├── README.md │ ├── Stock display.ipynb │ ├── foldcode.tpl │ ├── makeitpop.tpl │ ├── nbconvert_template_structure.html │ └── solutions │ │ ├── hidecode.tplx │ │ └── hidecode_selective.tplx ├── nbdime-after.ipynb ├── nbdime-before.ipynb ├── nbdime.ipynb └── nbval.ipynb ├── Part-3 ├── Beat Frequencies.ipynb ├── Custom Widget - Hello World.ipynb ├── Date Picker Widget.ipynb ├── Exploring Graphs.ipynb ├── Export As (nbconvert).html ├── Export As (nbconvert).ipynb ├── File Upload Widget.ipynb ├── Image Browser.ipynb ├── Index.ipynb ├── Lorenz Differential Equations.ipynb ├── Nonblocking Console.ipynb ├── Using Interact.ipynb ├── Variable Inspector.ipynb ├── Widget Basics.ipynb ├── Widget Events.ipynb ├── Widget List.ipynb ├── Widget Styling.ipynb ├── exercises │ ├── Image Processing-Solution.ipynb │ ├── Image Processing.ipynb │ ├── ImageCompare-Solution.ipynb │ ├── ImageCompare.ipynb │ ├── Interact with matplotlib figures.ipynb │ ├── InteractBasics-Solution.ipynb │ ├── InteractBasics.ipynb │ ├── University_of_Sheffield_coat_of_arms_new.png │ ├── University_of_Sheffield_coat_of_arms_old.png │ ├── WidgetBasics.ipynb │ └── matplotlib_beyond_basics.ipynb ├── images │ ├── MultilanguageKernels.graffle │ ├── MultilanguageKernels.png │ ├── ParallelKernels.graffle │ ├── ParallelKernels.png │ ├── VizInteractCompute.graffle │ ├── VizInteractCompute.png │ ├── WidgetArch.graffle │ ├── WidgetArch.png │ ├── WidgetModelView.graffle │ ├── WidgetModelView.png │ └── ipython_logo.png └── mpltune.ipynb ├── Part-4 ├── Index.ipynb ├── Overview.ipynb ├── Performance.ipynb ├── Summary.ipynb ├── download-images.ipynb ├── examples │ ├── Counting Words.ipynb │ ├── DAG Dependencies.ipynb │ ├── MC Options.ipynb │ ├── MPI Broadcast.ipynb │ ├── Monitoring MPI.ipynb │ ├── Parallel face detection.ipynb │ ├── Parallel image processing.ipynb │ ├── memmap.ipynb │ └── wikipedia │ │ ├── Wikipedia.ipynb │ │ ├── eventful_dict.py │ │ ├── eventful_graph.py │ │ ├── widget_forcedirectedgraph.js │ │ └── widget_forcedirectedgraph.py ├── exercises │ ├── Monte Carlo π.ipynb │ └── Remote Iteration.ipynb ├── figs │ ├── allconnections.png │ ├── darts.png │ ├── latency.png │ ├── latency2.png │ ├── map.png │ ├── throughput1.png │ ├── throughput2.png │ └── wideView.png ├── hints.py ├── hubble-tiles.py ├── images_common.py ├── soln │ ├── matmul.py │ ├── mcpi.py │ ├── nestedloop.py │ ├── ngrams.py │ ├── remote_iter.py │ └── remote_iter_hint.py └── tutorial │ ├── All Together.ipynb │ ├── Load-Balancing.ipynb │ ├── Multiplexing.ipynb │ ├── Parallel Magics.ipynb │ ├── Remote Execution.ipynb │ └── myscript.py ├── README.md ├── utils ├── list_pyfiles.ipy └── list_subdirs.ipy ├── version_check.py └── website_summary.rst /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints 2 | Part-4/images 3 | Part-4/examples/*.txt 4 | Part-4/examples/gutenberg* 5 | Part-4/*.jpg 6 | Part-4/hubble-*x* 7 | __pycache__ 8 | .DS_Store -------------------------------------------------------------------------------- /Index.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "collapsed": true 7 | }, 8 | "source": [ 9 | "# Jupyter and IPython" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "To follow this tutorial, you should have Jupyter installed on your laptop. If you don't have it, please install the *Anaconda* bundle from [www.continuum.io/downloads](https://www.continuum.io/downloads). Get the Python 3.6 version.\n", 17 | "\n", 18 | "If you're having installation trouble, please ask for help!\n", 19 | "\n", 20 | "Clone or download the tutorial materials from https://github.com/jupyter/ngcm-tutorial" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "* **Day 1**\n", 28 | " * [Morning: Introducing Jupyter notebooks and IPython](Part-1/Index.ipynb)\n", 29 | " * [Afternoon: Working with notebook files](Part-2/Index.ipynb)\n", 30 | "* Day 2:\n", 31 | " * [Morning: Interactive 'widgets' in the notebook](Part-3/Index.ipynb)\n", 32 | " * [Afternoon: Interactive Parallel computing](Part-4/Index.ipynb)" 33 | ] 34 | } 35 | ], 36 | "metadata": { 37 | "kernelspec": { 38 | "display_name": "Python 3", 39 | "language": "python", 40 | "name": "python3" 41 | }, 42 | "language_info": { 43 | "codemirror_mode": { 44 | "name": "ipython", 45 | "version": 3 46 | }, 47 | "file_extension": ".py", 48 | "mimetype": "text/x-python", 49 | "name": "python", 50 | "nbconvert_exporter": "python", 51 | "pygments_lexer": "ipython3", 52 | "version": "3.5.2" 53 | } 54 | }, 55 | "nbformat": 4, 56 | "nbformat_minor": 1 57 | } 58 | -------------------------------------------------------------------------------- /Part-1/Embedding/Index.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "Back to the main [Index](../Index.ipynb)" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "# Embedding IPython Into Other Applications" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "The architecture of IPython is built with reusable components. These components include:\n", 29 | "\n", 30 | "* The configuration system for processing command line arguments and configuration files\n", 31 | "* The IPython `InteractiveShell` object that provides the core interactive features across the entire code base\n", 32 | "* The IPython kernel, which provides the capabilities of the `InteractiveShell` object over a ZeroMQ/JSON based message protocol to various frontends\n", 33 | "* The IPython frontends (Notebook, Qt Console, Console, Terminal)\n", 34 | "\n", 35 | "These components can be embedded into other applications." 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "## Non-notebook examples" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": {}, 48 | "source": [ 49 | "This directory also contains examples that are regular Python (`.py`) files.\n", 50 | "\n", 51 | "These files demonstrate how to embed the core functionality of IPython into various kinds of scenarios, from a terminal based script to a GUI application:" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": 1, 57 | "metadata": { 58 | "collapsed": false 59 | }, 60 | "outputs": [ 61 | { 62 | "name": "stdout", 63 | "output_type": "stream", 64 | "text": [ 65 | "\u001b[31membed_class_long.py\u001b[m\u001b[m* inprocess_qtconsole.py \u001b[31mipkernel_qtapp.py\u001b[m\u001b[m*\r\n", 66 | "embed_class_short.py inprocess_terminal.py \u001b[31mipkernel_wxapp.py\u001b[m\u001b[m*\r\n", 67 | "embed_function.py internal_ipkernel.py\r\n" 68 | ] 69 | } 70 | ], 71 | "source": [ 72 | "ls *py" 73 | ] 74 | } 75 | ], 76 | "metadata": { 77 | "kernelspec": { 78 | "display_name": "Python 3", 79 | "language": "python", 80 | "name": "python3" 81 | }, 82 | "language_info": { 83 | "codemirror_mode": { 84 | "name": "ipython", 85 | "version": 3 86 | }, 87 | "file_extension": ".py", 88 | "mimetype": "text/x-python", 89 | "name": "python", 90 | "nbconvert_exporter": "python", 91 | "pygments_lexer": "ipython3", 92 | "version": "3.4.3" 93 | } 94 | }, 95 | "nbformat": 4, 96 | "nbformat_minor": 0 97 | } 98 | -------------------------------------------------------------------------------- /Part-1/Embedding/embed_class_long.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """An example of how to embed an IPython shell into a running program. 3 | 4 | Please see the documentation in the IPython.Shell module for more details. 5 | 6 | The accompanying file embed_class_short.py has quick code fragments for 7 | embedding which you can cut and paste in your code once you understand how 8 | things work. 9 | 10 | The code in this file is deliberately extra-verbose, meant for learning.""" 11 | from __future__ import print_function 12 | 13 | # The basics to get you going: 14 | 15 | # IPython injects get_ipython into builtins, so you can know if you have nested 16 | # copies running. 17 | 18 | # Try running this code both at the command line and from inside IPython (with 19 | # %run example-embed.py) 20 | from IPython.config.loader import Config 21 | try: 22 | get_ipython 23 | except NameError: 24 | nested = 0 25 | cfg = Config() 26 | prompt_config = cfg.PromptManager 27 | prompt_config.in_template = 'In <\\#>: ' 28 | prompt_config.in2_template = ' .\\D.: ' 29 | prompt_config.out_template = 'Out<\\#>: ' 30 | else: 31 | print("Running nested copies of IPython.") 32 | print("The prompts for the nested copy have been modified") 33 | cfg = Config() 34 | nested = 1 35 | 36 | # First import the embeddable shell class 37 | from IPython.terminal.embed import InteractiveShellEmbed 38 | 39 | # Now create an instance of the embeddable shell. The first argument is a 40 | # string with options exactly as you would type them if you were starting 41 | # IPython at the system command line. Any parameters you want to define for 42 | # configuration can thus be specified here. 43 | ipshell = InteractiveShellEmbed(config=cfg, 44 | banner1 = 'Dropping into IPython', 45 | exit_msg = 'Leaving Interpreter, back to program.') 46 | 47 | # Make a second instance, you can have as many as you want. 48 | cfg2 = cfg.copy() 49 | prompt_config = cfg2.PromptManager 50 | prompt_config.in_template = 'In2<\\#>: ' 51 | if not nested: 52 | prompt_config.in_template = 'In2<\\#>: ' 53 | prompt_config.in2_template = ' .\\D.: ' 54 | prompt_config.out_template = 'Out<\\#>: ' 55 | ipshell2 = InteractiveShellEmbed(config=cfg, 56 | banner1 = 'Second IPython instance.') 57 | 58 | print('\nHello. This is printed from the main controller program.\n') 59 | 60 | # You can then call ipshell() anywhere you need it (with an optional 61 | # message): 62 | ipshell('***Called from top level. ' 63 | 'Hit Ctrl-D to exit interpreter and continue program.\n' 64 | 'Note that if you use %kill_embedded, you can fully deactivate\n' 65 | 'This embedded instance so it will never turn on again') 66 | 67 | print('\nBack in caller program, moving along...\n') 68 | 69 | #--------------------------------------------------------------------------- 70 | # More details: 71 | 72 | # InteractiveShellEmbed instances don't print the standard system banner and 73 | # messages. The IPython banner (which actually may contain initialization 74 | # messages) is available as get_ipython().banner in case you want it. 75 | 76 | # InteractiveShellEmbed instances print the following information everytime they 77 | # start: 78 | 79 | # - A global startup banner. 80 | 81 | # - A call-specific header string, which you can use to indicate where in the 82 | # execution flow the shell is starting. 83 | 84 | # They also print an exit message every time they exit. 85 | 86 | # Both the startup banner and the exit message default to None, and can be set 87 | # either at the instance constructor or at any other time with the 88 | # by setting the banner and exit_msg attributes. 89 | 90 | # The shell instance can be also put in 'dummy' mode globally or on a per-call 91 | # basis. This gives you fine control for debugging without having to change 92 | # code all over the place. 93 | 94 | # The code below illustrates all this. 95 | 96 | 97 | # This is how the global banner and exit_msg can be reset at any point 98 | ipshell.banner = 'Entering interpreter - New Banner' 99 | ipshell.exit_msg = 'Leaving interpreter - New exit_msg' 100 | 101 | def foo(m): 102 | s = 'spam' 103 | ipshell('***In foo(). Try %whos, or print s or m:') 104 | print('foo says m = ',m) 105 | 106 | def bar(n): 107 | s = 'eggs' 108 | ipshell('***In bar(). Try %whos, or print s or n:') 109 | print('bar says n = ',n) 110 | 111 | # Some calls to the above functions which will trigger IPython: 112 | print('Main program calling foo("eggs")\n') 113 | foo('eggs') 114 | 115 | # The shell can be put in 'dummy' mode where calls to it silently return. This 116 | # allows you, for example, to globally turn off debugging for a program with a 117 | # single call. 118 | ipshell.dummy_mode = True 119 | print('\nTrying to call IPython which is now "dummy":') 120 | ipshell() 121 | print('Nothing happened...') 122 | # The global 'dummy' mode can still be overridden for a single call 123 | print('\nOverriding dummy mode manually:') 124 | ipshell(dummy=False) 125 | 126 | # Reactivate the IPython shell 127 | ipshell.dummy_mode = False 128 | 129 | print('You can even have multiple embedded instances:') 130 | ipshell2() 131 | 132 | print('\nMain program calling bar("spam")\n') 133 | bar('spam') 134 | 135 | print('Main program finished. Bye!') 136 | -------------------------------------------------------------------------------- /Part-1/Embedding/embed_class_short.py: -------------------------------------------------------------------------------- 1 | """Quick code snippets for embedding IPython into other programs. 2 | 3 | See embed_class_long.py for full details, this file has the bare minimum code for 4 | cut and paste use once you understand how to use the system.""" 5 | 6 | #--------------------------------------------------------------------------- 7 | # This code loads IPython but modifies a few things if it detects it's running 8 | # embedded in another IPython session (helps avoid confusion) 9 | 10 | try: 11 | get_ipython 12 | except NameError: 13 | banner=exit_msg='' 14 | else: 15 | banner = '*** Nested interpreter ***' 16 | exit_msg = '*** Back in main IPython ***' 17 | 18 | # First import the embed function 19 | from IPython.terminal.embed import InteractiveShellEmbed 20 | # Now create the IPython shell instance. Put ipshell() anywhere in your code 21 | # where you want it to open. 22 | ipshell = InteractiveShellEmbed(banner1=banner, exit_msg=exit_msg) 23 | 24 | #--------------------------------------------------------------------------- 25 | # This code will load an embeddable IPython shell always with no changes for 26 | # nested embededings. 27 | 28 | from IPython import embed 29 | # Now embed() will open IPython anywhere in the code. 30 | 31 | #--------------------------------------------------------------------------- 32 | # This code loads an embeddable shell only if NOT running inside 33 | # IPython. Inside IPython, the embeddable shell variable ipshell is just a 34 | # dummy function. 35 | 36 | try: 37 | get_ipython 38 | except NameError: 39 | from IPython.terminal.embed import InteractiveShellEmbed 40 | ipshell = InteractiveShellEmbed() 41 | # Now ipshell() will open IPython anywhere in the code 42 | else: 43 | # Define a dummy ipshell() so the same code doesn't crash inside an 44 | # interactive IPython 45 | def ipshell(): pass 46 | -------------------------------------------------------------------------------- /Part-1/Embedding/embed_function.py: -------------------------------------------------------------------------------- 1 | """Embed IPython using the simple embed function rather than the class API.""" 2 | 3 | from IPython import embed 4 | 5 | a = 10 6 | b = 20 7 | 8 | embed(header='First time', banner1='') 9 | 10 | c = 30 11 | d = 40 12 | 13 | try: 14 | raise Exception('adsfasdf') 15 | except: 16 | embed(header='The second time') 17 | -------------------------------------------------------------------------------- /Part-1/Embedding/inprocess_qtconsole.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import os 3 | 4 | from IPython.qt.console.rich_ipython_widget import RichIPythonWidget 5 | from IPython.qt.inprocess import QtInProcessKernelManager 6 | from IPython.lib import guisupport 7 | 8 | 9 | def print_process_id(): 10 | print('Process ID is:', os.getpid()) 11 | 12 | 13 | def main(): 14 | # Print the ID of the main process 15 | print_process_id() 16 | 17 | app = guisupport.get_app_qt4() 18 | 19 | # Create an in-process kernel 20 | # >>> print_process_id() 21 | # will print the same process ID as the main process 22 | kernel_manager = QtInProcessKernelManager() 23 | kernel_manager.start_kernel() 24 | kernel = kernel_manager.kernel 25 | kernel.gui = 'qt4' 26 | kernel.shell.push({'foo': 43, 'print_process_id': print_process_id}) 27 | 28 | kernel_client = kernel_manager.client() 29 | kernel_client.start_channels() 30 | 31 | def stop(): 32 | kernel_client.stop_channels() 33 | kernel_manager.shutdown_kernel() 34 | app.exit() 35 | 36 | control = RichIPythonWidget() 37 | control.kernel_manager = kernel_manager 38 | control.kernel_client = kernel_client 39 | control.exit_requested.connect(stop) 40 | control.show() 41 | 42 | guisupport.start_event_loop_qt4(app) 43 | 44 | 45 | if __name__ == '__main__': 46 | main() 47 | -------------------------------------------------------------------------------- /Part-1/Embedding/inprocess_terminal.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import os 3 | 4 | from IPython.kernel.inprocess import InProcessKernelManager 5 | from IPython.terminal.console.interactiveshell import ZMQTerminalInteractiveShell 6 | 7 | 8 | def print_process_id(): 9 | print('Process ID is:', os.getpid()) 10 | 11 | 12 | def main(): 13 | print_process_id() 14 | 15 | # Create an in-process kernel 16 | # >>> print_process_id() 17 | # will print the same process ID as the main process 18 | kernel_manager = InProcessKernelManager() 19 | kernel_manager.start_kernel() 20 | kernel = kernel_manager.kernel 21 | kernel.gui = 'qt4' 22 | kernel.shell.push({'foo': 43, 'print_process_id': print_process_id}) 23 | client = kernel_manager.client() 24 | client.start_channels() 25 | 26 | shell = ZMQTerminalInteractiveShell(manager=kernel_manager, client=client) 27 | shell.mainloop() 28 | 29 | 30 | if __name__ == '__main__': 31 | main() 32 | -------------------------------------------------------------------------------- /Part-1/Embedding/internal_ipkernel.py: -------------------------------------------------------------------------------- 1 | #----------------------------------------------------------------------------- 2 | # Imports 3 | #----------------------------------------------------------------------------- 4 | 5 | import sys 6 | 7 | from IPython.lib.kernel import connect_qtconsole 8 | from IPython.kernel.zmq.kernelapp import IPKernelApp 9 | 10 | #----------------------------------------------------------------------------- 11 | # Functions and classes 12 | #----------------------------------------------------------------------------- 13 | def mpl_kernel(gui): 14 | """Launch and return an IPython kernel with matplotlib support for the desired gui 15 | """ 16 | kernel = IPKernelApp.instance() 17 | kernel.initialize(['python', '--matplotlib=%s' % gui, 18 | #'--log-level=10' 19 | ]) 20 | return kernel 21 | 22 | 23 | class InternalIPKernel(object): 24 | 25 | def init_ipkernel(self, backend): 26 | # Start IPython kernel with GUI event loop and mpl support 27 | self.ipkernel = mpl_kernel(backend) 28 | # To create and track active qt consoles 29 | self.consoles = [] 30 | 31 | # This application will also act on the shell user namespace 32 | self.namespace = self.ipkernel.shell.user_ns 33 | 34 | # Example: a variable that will be seen by the user in the shell, and 35 | # that the GUI modifies (the 'Counter++' button increments it): 36 | self.namespace['app_counter'] = 0 37 | #self.namespace['ipkernel'] = self.ipkernel # dbg 38 | 39 | def print_namespace(self, evt=None): 40 | print("\n***Variables in User namespace***") 41 | for k, v in self.namespace.items(): 42 | if not k.startswith('_'): 43 | print('%s -> %r' % (k, v)) 44 | sys.stdout.flush() 45 | 46 | def new_qt_console(self, evt=None): 47 | """start a new qtconsole connected to our kernel""" 48 | return connect_qtconsole(self.ipkernel.connection_file, profile=self.ipkernel.profile) 49 | 50 | def count(self, evt=None): 51 | self.namespace['app_counter'] += 1 52 | 53 | def cleanup_consoles(self, evt=None): 54 | for c in self.consoles: 55 | c.kill() 56 | -------------------------------------------------------------------------------- /Part-1/Embedding/ipkernel_qtapp.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """Example integrating an IPython kernel into a GUI App. 3 | 4 | This trivial GUI application internally starts an IPython kernel, to which Qt 5 | consoles can be connected either by the user at the command line or started 6 | from the GUI itself, via a button. The GUI can also manipulate one variable in 7 | the kernel's namespace, and print the namespace to the console. 8 | 9 | Play with it by running the script and then opening one or more consoles, and 10 | pushing the 'Counter++' and 'Namespace' buttons. 11 | 12 | Upon exit, it should automatically close all consoles opened from the GUI. 13 | 14 | Consoles attached separately from a terminal will not be terminated, though 15 | they will notice that their kernel died. 16 | """ 17 | #----------------------------------------------------------------------------- 18 | # Imports 19 | #----------------------------------------------------------------------------- 20 | 21 | from PyQt4 import Qt 22 | 23 | from internal_ipkernel import InternalIPKernel 24 | 25 | #----------------------------------------------------------------------------- 26 | # Functions and classes 27 | #----------------------------------------------------------------------------- 28 | class SimpleWindow(Qt.QWidget, InternalIPKernel): 29 | 30 | def __init__(self, app): 31 | Qt.QWidget.__init__(self) 32 | self.app = app 33 | self.add_widgets() 34 | self.init_ipkernel('qt') 35 | 36 | def add_widgets(self): 37 | self.setGeometry(300, 300, 400, 70) 38 | self.setWindowTitle('IPython in your app') 39 | 40 | # Add simple buttons: 41 | console = Qt.QPushButton('Qt Console', self) 42 | console.setGeometry(10, 10, 100, 35) 43 | self.connect(console, Qt.SIGNAL('clicked()'), self.new_qt_console) 44 | 45 | namespace = Qt.QPushButton('Namespace', self) 46 | namespace.setGeometry(120, 10, 100, 35) 47 | self.connect(namespace, Qt.SIGNAL('clicked()'), self.print_namespace) 48 | 49 | count = Qt.QPushButton('Count++', self) 50 | count.setGeometry(230, 10, 80, 35) 51 | self.connect(count, Qt.SIGNAL('clicked()'), self.count) 52 | 53 | # Quit and cleanup 54 | quit = Qt.QPushButton('Quit', self) 55 | quit.setGeometry(320, 10, 60, 35) 56 | self.connect(quit, Qt.SIGNAL('clicked()'), Qt.qApp, Qt.SLOT('quit()')) 57 | 58 | self.app.connect(self.app, Qt.SIGNAL("lastWindowClosed()"), 59 | self.app, Qt.SLOT("quit()")) 60 | 61 | self.app.aboutToQuit.connect(self.cleanup_consoles) 62 | 63 | #----------------------------------------------------------------------------- 64 | # Main script 65 | #----------------------------------------------------------------------------- 66 | 67 | if __name__ == "__main__": 68 | app = Qt.QApplication([]) 69 | # Create our window 70 | win = SimpleWindow(app) 71 | win.show() 72 | 73 | # Very important, IPython-specific step: this gets GUI event loop 74 | # integration going, and it replaces calling app.exec_() 75 | win.ipkernel.start() 76 | -------------------------------------------------------------------------------- /Part-1/Embedding/ipkernel_wxapp.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """Example integrating an IPython kernel into a GUI App. 3 | 4 | This trivial GUI application internally starts an IPython kernel, to which Qt 5 | consoles can be connected either by the user at the command line or started 6 | from the GUI itself, via a button. The GUI can also manipulate one variable in 7 | the kernel's namespace, and print the namespace to the console. 8 | 9 | Play with it by running the script and then opening one or more consoles, and 10 | pushing the 'Counter++' and 'Namespace' buttons. 11 | 12 | Upon exit, it should automatically close all consoles opened from the GUI. 13 | 14 | Consoles attached separately from a terminal will not be terminated, though 15 | they will notice that their kernel died. 16 | 17 | Ref: Modified from wxPython source code wxPython/samples/simple/simple.py 18 | """ 19 | #----------------------------------------------------------------------------- 20 | # Imports 21 | #----------------------------------------------------------------------------- 22 | import sys 23 | 24 | import wx 25 | 26 | from internal_ipkernel import InternalIPKernel 27 | 28 | #----------------------------------------------------------------------------- 29 | # Functions and classes 30 | #----------------------------------------------------------------------------- 31 | 32 | class MyFrame(wx.Frame, InternalIPKernel): 33 | """ 34 | This is MyFrame. It just shows a few controls on a wxPanel, 35 | and has a simple menu. 36 | """ 37 | 38 | def __init__(self, parent, title): 39 | wx.Frame.__init__(self, parent, -1, title, 40 | pos=(150, 150), size=(350, 285)) 41 | 42 | # Create the menubar 43 | menuBar = wx.MenuBar() 44 | 45 | # and a menu 46 | menu = wx.Menu() 47 | 48 | # add an item to the menu, using \tKeyName automatically 49 | # creates an accelerator, the third param is some help text 50 | # that will show up in the statusbar 51 | menu.Append(wx.ID_EXIT, "E&xit\tAlt-X", "Exit this simple sample") 52 | 53 | # bind the menu event to an event handler 54 | self.Bind(wx.EVT_MENU, self.OnTimeToClose, id=wx.ID_EXIT) 55 | 56 | # and put the menu on the menubar 57 | menuBar.Append(menu, "&File") 58 | self.SetMenuBar(menuBar) 59 | 60 | self.CreateStatusBar() 61 | 62 | # Now create the Panel to put the other controls on. 63 | panel = wx.Panel(self) 64 | 65 | # and a few controls 66 | text = wx.StaticText(panel, -1, "Hello World!") 67 | text.SetFont(wx.Font(14, wx.SWISS, wx.NORMAL, wx.BOLD)) 68 | text.SetSize(text.GetBestSize()) 69 | qtconsole_btn = wx.Button(panel, -1, "Qt Console") 70 | ns_btn = wx.Button(panel, -1, "Namespace") 71 | count_btn = wx.Button(panel, -1, "Count++") 72 | close_btn = wx.Button(panel, -1, "Quit") 73 | 74 | # bind the button events to handlers 75 | self.Bind(wx.EVT_BUTTON, self.new_qt_console, qtconsole_btn) 76 | self.Bind(wx.EVT_BUTTON, self.print_namespace, ns_btn) 77 | self.Bind(wx.EVT_BUTTON, self.count, count_btn) 78 | self.Bind(wx.EVT_BUTTON, self.OnTimeToClose, close_btn) 79 | 80 | # Use a sizer to layout the controls, stacked vertically and with 81 | # a 10 pixel border around each 82 | sizer = wx.BoxSizer(wx.VERTICAL) 83 | for ctrl in [text, qtconsole_btn, ns_btn, count_btn, close_btn]: 84 | sizer.Add(ctrl, 0, wx.ALL, 10) 85 | panel.SetSizer(sizer) 86 | panel.Layout() 87 | 88 | # Start the IPython kernel with gui support 89 | self.init_ipkernel('wx') 90 | 91 | def OnTimeToClose(self, evt): 92 | """Event handler for the button click.""" 93 | print("See ya later!") 94 | sys.stdout.flush() 95 | self.cleanup_consoles(evt) 96 | self.Close() 97 | # Not sure why, but our IPython kernel seems to prevent normal WX 98 | # shutdown, so an explicit exit() call is needed. 99 | sys.exit() 100 | 101 | 102 | class MyApp(wx.App): 103 | def OnInit(self): 104 | frame = MyFrame(None, "Simple wxPython App") 105 | self.SetTopWindow(frame) 106 | frame.Show(True) 107 | self.ipkernel = frame.ipkernel 108 | return True 109 | 110 | #----------------------------------------------------------------------------- 111 | # Main script 112 | #----------------------------------------------------------------------------- 113 | 114 | if __name__ == '__main__': 115 | app = MyApp(redirect=False, clearSigInt=False) 116 | 117 | # Very important, IPython-specific step: this gets GUI event loop 118 | # integration going, and it replaces calling app.MainLoop() 119 | app.ipkernel.start() 120 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/Index.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "Back to the main [Index](../Index.ipynb)" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "# IPython Kernel" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "IPython provides extensions to the Python programming language that make working interactively convenient and efficient. These extensions are implemented in the IPython Kernel and are available in all of the IPython Frontends (Notebook, Terminal, Console and Qt Console) when running this kernel." 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "## Tutorials" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "* [Cell Magics](Cell Magics.ipynb)\n", 43 | "* [Script Magics](Script Magics.ipynb)\n", 44 | "* [Rich Output](Rich Output.ipynb)\n", 45 | "* [Custom Display Logic](Custom Display Logic.ipynb)\n", 46 | "* [Plotting in the Notebook](Plotting in the Notebook.ipynb)\n", 47 | "* [Capturing Output](Capturing Output.ipynb)" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "## Examples" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "* [Background Jobs](Background Jobs.ipynb)\n", 62 | "* [Trapezoid Rule](Trapezoid Rule.ipynb)\n", 63 | "* [SymPy](SymPy.ipynb)\n", 64 | "* [Raw Input in the Notebook](Raw Input in the Notebook.ipynb)" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "metadata": {}, 70 | "source": [ 71 | "## Non-notebook examples" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "metadata": {}, 77 | "source": [ 78 | "This directory also contains examples that are regular Python (`.py`) files." 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 1, 84 | "metadata": { 85 | "collapsed": false 86 | }, 87 | "outputs": [ 88 | { 89 | "data": { 90 | "text/html": [ 91 | "example-demo.py
" 92 | ], 93 | "text/plain": [ 94 | "/Users/minrk/dev/ip/mine/examples/IPython Kernel/example-demo.py" 95 | ] 96 | }, 97 | "metadata": {}, 98 | "output_type": "display_data" 99 | }, 100 | { 101 | "data": { 102 | "text/html": [ 103 | "ipython-get-history.py
" 104 | ], 105 | "text/plain": [ 106 | "/Users/minrk/dev/ip/mine/examples/IPython Kernel/ipython-get-history.py" 107 | ] 108 | }, 109 | "metadata": {}, 110 | "output_type": "display_data" 111 | } 112 | ], 113 | "source": [ 114 | "%run ../../utils/list_pyfiles.ipy" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "There are also a set of examples that show how to integrate IPython with different GUI event loops:" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 2, 127 | "metadata": { 128 | "collapsed": false 129 | }, 130 | "outputs": [ 131 | { 132 | "data": { 133 | "text/html": [ 134 | "data/
\n", 135 | "  flare.json
" 136 | ], 137 | "text/plain": [ 138 | "data/\n", 139 | " flare.json" 140 | ] 141 | }, 142 | "metadata": {}, 143 | "output_type": "display_data" 144 | }, 145 | { 146 | "data": { 147 | "text/html": [ 148 | "gui/
\n", 149 | "  gui-glut.py
\n", 150 | "  gui-gtk.py
\n", 151 | "  gui-gtk3.py
\n", 152 | "  gui-pyglet.py
\n", 153 | "  gui-qt.py
\n", 154 | "  gui-tk.py
\n", 155 | "  gui-wx.py
" 156 | ], 157 | "text/plain": [ 158 | "gui/\n", 159 | " gui-glut.py\n", 160 | " gui-gtk.py\n", 161 | " gui-gtk3.py\n", 162 | " gui-pyglet.py\n", 163 | " gui-qt.py\n", 164 | " gui-tk.py\n", 165 | " gui-wx.py" 166 | ] 167 | }, 168 | "metadata": {}, 169 | "output_type": "display_data" 170 | } 171 | ], 172 | "source": [ 173 | "%run ../../utils/list_subdirs.ipy" 174 | ] 175 | } 176 | ], 177 | "metadata": { 178 | "kernelspec": { 179 | "display_name": "Python 3", 180 | "language": "python", 181 | "name": "python3" 182 | }, 183 | "language_info": { 184 | "codemirror_mode": { 185 | "name": "ipython", 186 | "version": 3 187 | }, 188 | "file_extension": ".py", 189 | "mimetype": "text/x-python", 190 | "name": "python", 191 | "nbconvert_exporter": "python", 192 | "pygments_lexer": "ipython3", 193 | "version": "3.4.3" 194 | } 195 | }, 196 | "nbformat": 4, 197 | "nbformat_minor": 0 198 | } 199 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/example-demo.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """A simple interactive demo to illustrate the use of IPython's Demo class. 3 | 4 | Any python script can be run as a demo, but that does little more than showing 5 | it on-screen, syntax-highlighted in one shot. If you add a little simple 6 | markup, you can stop at specified intervals and return to the ipython prompt, 7 | resuming execution later. 8 | 9 | This is a unicode test, åäö 10 | """ 11 | from __future__ import print_function 12 | 13 | print('Hello, welcome to an interactive IPython demo.') 14 | print('Executing this block should require confirmation before proceeding,') 15 | print('unless auto_all has been set to true in the demo object') 16 | 17 | # The mark below defines a block boundary, which is a point where IPython will 18 | # stop execution and return to the interactive prompt. 19 | # --- stop --- 20 | 21 | x = 1 22 | y = 2 23 | 24 | # --- stop --- 25 | 26 | # the mark below makes this block as silent 27 | # silent 28 | 29 | print('This is a silent block, which gets executed but not printed.') 30 | 31 | # --- stop --- 32 | # auto 33 | print('This is an automatic block.') 34 | print('It is executed without asking for confirmation, but printed.') 35 | z = x+y 36 | 37 | print('z=',x) 38 | 39 | # --- stop --- 40 | # This is just another normal block. 41 | print('z is now:', z) 42 | 43 | print('bye!') 44 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/gui/gui-glut.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """Simple GLUT example to manually test event loop integration. 3 | 4 | This is meant to run tests manually in ipython as: 5 | 6 | In [5]: %gui glut 7 | 8 | In [6]: %run gui-glut.py 9 | 10 | In [7]: gl.glClearColor(1,1,1,1) 11 | """ 12 | 13 | #!/usr/bin/env python 14 | import sys 15 | import OpenGL.GL as gl 16 | import OpenGL.GLUT as glut 17 | 18 | def close(): 19 | glut.glutDestroyWindow(glut.glutGetWindow()) 20 | 21 | def display(): 22 | gl.glClear (gl.GL_COLOR_BUFFER_BIT | gl.GL_DEPTH_BUFFER_BIT) 23 | glut.glutSwapBuffers() 24 | 25 | def resize(width,height): 26 | gl.glViewport(0, 0, width, height+4) 27 | gl.glMatrixMode(gl.GL_PROJECTION) 28 | gl.glLoadIdentity() 29 | gl.glOrtho(0, width, 0, height+4, -1, 1) 30 | gl.glMatrixMode(gl.GL_MODELVIEW) 31 | 32 | if glut.glutGetWindow() > 0: 33 | interactive = True 34 | glut.glutInit(sys.argv) 35 | glut.glutInitDisplayMode(glut.GLUT_DOUBLE | 36 | glut.GLUT_RGBA | 37 | glut.GLUT_DEPTH) 38 | else: 39 | interactive = False 40 | 41 | glut.glutCreateWindow('gui-glut') 42 | glut.glutDisplayFunc(display) 43 | glut.glutReshapeFunc(resize) 44 | # This is necessary on osx to be able to close the window 45 | # (else the close button is disabled) 46 | if sys.platform == 'darwin' and not bool(glut.HAVE_FREEGLUT): 47 | glut.glutWMCloseFunc(close) 48 | gl.glClearColor(0,0,0,1) 49 | 50 | if not interactive: 51 | glut.glutMainLoop() 52 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/gui/gui-gtk.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """Simple GTK example to manually test event loop integration. 3 | 4 | This is meant to run tests manually in ipython as: 5 | 6 | In [5]: %gui gtk 7 | 8 | In [6]: %run gui-gtk.py 9 | """ 10 | 11 | import pygtk 12 | pygtk.require('2.0') 13 | import gtk 14 | 15 | 16 | def hello_world(wigdet, data=None): 17 | print("Hello World") 18 | 19 | def delete_event(widget, event, data=None): 20 | return False 21 | 22 | def destroy(widget, data=None): 23 | gtk.main_quit() 24 | 25 | window = gtk.Window(gtk.WINDOW_TOPLEVEL) 26 | window.connect("delete_event", delete_event) 27 | window.connect("destroy", destroy) 28 | button = gtk.Button("Hello World") 29 | button.connect("clicked", hello_world, None) 30 | 31 | window.add(button) 32 | button.show() 33 | window.show() 34 | 35 | try: 36 | from IPython.lib.inputhook import enable_gui 37 | enable_gui('gtk') 38 | except ImportError: 39 | gtk.main() 40 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/gui/gui-gtk3.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """Simple Gtk example to manually test event loop integration. 3 | 4 | This is meant to run tests manually in ipython as: 5 | 6 | In [1]: %gui gtk3 7 | 8 | In [2]: %run gui-gtk3.py 9 | """ 10 | 11 | from gi.repository import Gtk 12 | 13 | 14 | def hello_world(wigdet, data=None): 15 | print("Hello World") 16 | 17 | def delete_event(widget, event, data=None): 18 | return False 19 | 20 | def destroy(widget, data=None): 21 | Gtk.main_quit() 22 | 23 | window = Gtk.Window(Gtk.WindowType.TOPLEVEL) 24 | window.connect("delete_event", delete_event) 25 | window.connect("destroy", destroy) 26 | button = Gtk.Button("Hello World") 27 | button.connect("clicked", hello_world, None) 28 | 29 | window.add(button) 30 | button.show() 31 | window.show() 32 | 33 | try: 34 | from IPython.lib.inputhook import enable_gui 35 | enable_gui('gtk3') 36 | except ImportError: 37 | Gtk.main() 38 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/gui/gui-pyglet.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """Simple pyglet example to manually test event loop integration. 3 | 4 | This is meant to run tests manually in ipython as: 5 | 6 | In [5]: %gui pyglet 7 | 8 | In [6]: %run gui-pyglet.py 9 | """ 10 | 11 | import pyglet 12 | 13 | 14 | window = pyglet.window.Window() 15 | label = pyglet.text.Label('Hello, world', 16 | font_name='Times New Roman', 17 | font_size=36, 18 | x=window.width//2, y=window.height//2, 19 | anchor_x='center', anchor_y='center') 20 | @window.event 21 | def on_close(): 22 | window.close() 23 | 24 | @window.event 25 | def on_draw(): 26 | window.clear() 27 | label.draw() 28 | 29 | try: 30 | from IPython.lib.inputhook import enable_gui 31 | enable_gui('pyglet') 32 | except ImportError: 33 | pyglet.app.run() 34 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/gui/gui-qt.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """Simple Qt4 example to manually test event loop integration. 3 | 4 | This is meant to run tests manually in ipython as: 5 | 6 | In [5]: %gui qt 7 | 8 | In [6]: %run gui-qt.py 9 | 10 | Ref: Modified from http://zetcode.com/tutorials/pyqt4/firstprograms/ 11 | """ 12 | 13 | from PyQt4 import QtGui, QtCore 14 | 15 | class SimpleWindow(QtGui.QWidget): 16 | def __init__(self, parent=None): 17 | QtGui.QWidget.__init__(self, parent) 18 | 19 | self.setGeometry(300, 300, 200, 80) 20 | self.setWindowTitle('Hello World') 21 | 22 | quit = QtGui.QPushButton('Close', self) 23 | quit.setGeometry(10, 10, 60, 35) 24 | 25 | self.connect(quit, QtCore.SIGNAL('clicked()'), 26 | self, QtCore.SLOT('close()')) 27 | 28 | if __name__ == '__main__': 29 | app = QtCore.QCoreApplication.instance() 30 | if app is None: 31 | app = QtGui.QApplication([]) 32 | 33 | sw = SimpleWindow() 34 | sw.show() 35 | 36 | try: 37 | from IPython.lib.guisupport import start_event_loop_qt4 38 | start_event_loop_qt4(app) 39 | except ImportError: 40 | app.exec_() 41 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/gui/gui-tk.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """Simple Tk example to manually test event loop integration. 3 | 4 | This is meant to run tests manually in ipython as: 5 | 6 | In [5]: %gui tk 7 | 8 | In [6]: %run gui-tk.py 9 | """ 10 | 11 | try: 12 | from tkinter import * # Python 3 13 | except ImportError: 14 | from Tkinter import * # Python 2 15 | 16 | class MyApp: 17 | 18 | def __init__(self, root): 19 | frame = Frame(root) 20 | frame.pack() 21 | 22 | self.button = Button(frame, text="Hello", command=self.hello_world) 23 | self.button.pack(side=LEFT) 24 | 25 | def hello_world(self): 26 | print("Hello World!") 27 | 28 | root = Tk() 29 | 30 | app = MyApp(root) 31 | 32 | try: 33 | from IPython.lib.inputhook import enable_gui 34 | enable_gui('tk', root) 35 | except ImportError: 36 | root.mainloop() 37 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/gui/gui-wx.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """ 3 | A Simple wx example to test IPython's event loop integration. 4 | 5 | To run this do: 6 | 7 | In [5]: %gui wx # or start IPython with '--gui wx' 8 | 9 | In [6]: %run gui-wx.py 10 | 11 | Ref: Modified from wxPython source code wxPython/samples/simple/simple.py 12 | """ 13 | 14 | import wx 15 | 16 | 17 | class MyFrame(wx.Frame): 18 | """ 19 | This is MyFrame. It just shows a few controls on a wxPanel, 20 | and has a simple menu. 21 | """ 22 | def __init__(self, parent, title): 23 | wx.Frame.__init__(self, parent, -1, title, 24 | pos=(150, 150), size=(350, 200)) 25 | 26 | # Create the menubar 27 | menuBar = wx.MenuBar() 28 | 29 | # and a menu 30 | menu = wx.Menu() 31 | 32 | # add an item to the menu, using \tKeyName automatically 33 | # creates an accelerator, the third param is some help text 34 | # that will show up in the statusbar 35 | menu.Append(wx.ID_EXIT, "E&xit\tAlt-X", "Exit this simple sample") 36 | 37 | # bind the menu event to an event handler 38 | self.Bind(wx.EVT_MENU, self.OnTimeToClose, id=wx.ID_EXIT) 39 | 40 | # and put the menu on the menubar 41 | menuBar.Append(menu, "&File") 42 | self.SetMenuBar(menuBar) 43 | 44 | self.CreateStatusBar() 45 | 46 | # Now create the Panel to put the other controls on. 47 | panel = wx.Panel(self) 48 | 49 | # and a few controls 50 | text = wx.StaticText(panel, -1, "Hello World!") 51 | text.SetFont(wx.Font(14, wx.SWISS, wx.NORMAL, wx.BOLD)) 52 | text.SetSize(text.GetBestSize()) 53 | btn = wx.Button(panel, -1, "Close") 54 | funbtn = wx.Button(panel, -1, "Just for fun...") 55 | 56 | # bind the button events to handlers 57 | self.Bind(wx.EVT_BUTTON, self.OnTimeToClose, btn) 58 | self.Bind(wx.EVT_BUTTON, self.OnFunButton, funbtn) 59 | 60 | # Use a sizer to layout the controls, stacked vertically and with 61 | # a 10 pixel border around each 62 | sizer = wx.BoxSizer(wx.VERTICAL) 63 | sizer.Add(text, 0, wx.ALL, 10) 64 | sizer.Add(btn, 0, wx.ALL, 10) 65 | sizer.Add(funbtn, 0, wx.ALL, 10) 66 | panel.SetSizer(sizer) 67 | panel.Layout() 68 | 69 | 70 | def OnTimeToClose(self, evt): 71 | """Event handler for the button click.""" 72 | print("See ya later!") 73 | self.Close() 74 | 75 | def OnFunButton(self, evt): 76 | """Event handler for the button click.""" 77 | print("Having fun yet?") 78 | 79 | 80 | class MyApp(wx.App): 81 | def OnInit(self): 82 | frame = MyFrame(None, "Simple wxPython App") 83 | self.SetTopWindow(frame) 84 | 85 | print("Print statements go to this stdout window by default.") 86 | 87 | frame.Show(True) 88 | return True 89 | 90 | 91 | if __name__ == '__main__': 92 | 93 | app = wx.GetApp() 94 | if app is None: 95 | app = MyApp(redirect=False, clearSigInt=False) 96 | else: 97 | frame = MyFrame(None, "Simple wxPython App") 98 | app.SetTopWindow(frame) 99 | print("Print statements go to this stdout window by default.") 100 | frame.Show(True) 101 | 102 | try: 103 | from IPython.lib.inputhook import enable_gui 104 | enable_gui('wx', app) 105 | except ImportError: 106 | app.MainLoop() 107 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/ipython-completion.bash: -------------------------------------------------------------------------------- 1 | # load with: . ipython-completion.bash 2 | 3 | if [[ -n ${ZSH_VERSION-} ]]; then 4 | autoload -Uz bashcompinit && bashcompinit 5 | fi 6 | 7 | _ipython_get_flags() 8 | { 9 | local url=$1 10 | local var=$2 11 | local dash=$3 12 | if [[ "$url $var" == $__ipython_complete_last ]]; then 13 | opts=$__ipython_complete_last_res 14 | return 15 | fi 16 | # matplotlib and profile don't need the = and the 17 | # version without simplifies the special cased completion 18 | opts=$(ipython ${url} --help-all | grep -E "^-{1,2}[^-]" | sed -e "s/<.*//" -e "s/[^=]$/& /" -e "s/^--matplotlib=$//" -e "s/^--profile=$/--profile /" -e "$ s/^/\n-h\n--help\n--help-all\n/") 19 | __ipython_complete_last="$url $var" 20 | __ipython_complete_last_res="$opts" 21 | } 22 | 23 | _ipython() 24 | { 25 | local cur=${COMP_WORDS[COMP_CWORD]} 26 | local prev=${COMP_WORDS[COMP_CWORD - 1]} 27 | local subcommands="notebook qtconsole console kernel profile locate history nbconvert kernelspec install-nbextension trust " 28 | local opts="help" 29 | if [ -z "$__ipython_complete_baseopts" ]; then 30 | _ipython_get_flags baseopts 31 | __ipython_complete_baseopts="${opts}" 32 | fi 33 | local baseopts="$__ipython_complete_baseopts" 34 | local mode="" 35 | for i in "${COMP_WORDS[@]}"; do 36 | [ "$cur" = "$i" ] && break 37 | if [[ ${subcommands} == *${i}* ]]; then 38 | mode="$i" 39 | break 40 | elif [[ ${i} == "--"* ]]; then 41 | mode="nosubcommand" 42 | break 43 | fi 44 | done 45 | 46 | 47 | if [[ ${cur} == -* ]]; then 48 | case $mode in 49 | "notebook" | "qtconsole" | "console" | "kernel" | "nbconvert") 50 | _ipython_get_flags $mode 51 | opts=$"${opts} ${baseopts}" 52 | ;; 53 | "locate" | "profile" | "install-nbextension" | "trust") 54 | _ipython_get_flags $mode 55 | ;; 56 | "history" | "kernelspec") 57 | if [[ $COMP_CWORD -ge 3 ]]; then 58 | # 'history trim' and 'history clear' covered by next line 59 | _ipython_get_flags $mode\ "${COMP_WORDS[2]}" 60 | else 61 | _ipython_get_flags $mode 62 | 63 | fi 64 | opts=$"${opts}" 65 | ;; 66 | *) 67 | opts=$baseopts 68 | esac 69 | # don't drop the trailing space 70 | local IFS=$'\t\n' 71 | COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) ) 72 | return 0 73 | elif [[ $mode == "profile" ]]; then 74 | opts="list create locate " 75 | local IFS=$'\t\n' 76 | COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) ) 77 | elif [[ $mode == "history" ]]; then 78 | if [[ $COMP_CWORD -ge 3 ]]; then 79 | # drop into flags 80 | opts="--" 81 | else 82 | opts="trim clear " 83 | fi 84 | local IFS=$'\t\n' 85 | COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) ) 86 | elif [[ $mode == "kernelspec" ]]; then 87 | if [[ $COMP_CWORD -ge 3 ]]; then 88 | # drop into flags 89 | opts="--" 90 | else 91 | opts="list install " 92 | fi 93 | local IFS=$'\t\n' 94 | COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) ) 95 | elif [[ $mode == "locate" ]]; then 96 | if [[ $COMP_CWORD -ge 3 ]]; then 97 | # drop into flags 98 | opts="--" 99 | else 100 | opts="profile " 101 | fi 102 | local IFS=$'\t\n' 103 | COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) ) 104 | elif [[ ${prev} == "--matplotlib"* ]] || [[ ${prev} == "--gui"* ]]; then 105 | if [ -z "$__ipython_complete_matplotlib" ]; then 106 | __ipython_complete_matplotlib=`cat < 2: 25 | dest = open(sys.argv[2], "w") 26 | raw = not sys.argv[2].endswith('.py') 27 | else: 28 | dest = sys.stdout 29 | raw = True 30 | dest.write("# coding: utf-8\n") 31 | 32 | # Profiles other than 'default' can be specified here with a profile= argument: 33 | hist = HistoryAccessor() 34 | 35 | for session, lineno, cell in hist.get_range(session=session_number, raw=raw): 36 | cell = cell.encode('utf-8') # This line is only needed on Python 2. 37 | dest.write(cell + '\n') 38 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/ipython-qtconsole.desktop: -------------------------------------------------------------------------------- 1 | # If you want ipython to appear in a linux app launcher ("start menu"), install this by doing: 2 | # sudo desktop-file-install ipython-qtconsole.desktop 3 | 4 | [Desktop Entry] 5 | Comment=Enhanced interactive Python qtconsole 6 | Exec=ipython qtconsole 7 | GenericName[en_US]=Python shell 8 | GenericName=Python shell 9 | Icon=gnome-netstatus-idle 10 | Name[en_US]=IPython Qt console 11 | Name=IPython Qt console 12 | Categories=Development;Utility; 13 | StartupNotify=false 14 | Terminal=false 15 | Type=Application 16 | Actions=Matplotlib;Matplotlibinline; 17 | 18 | [Desktop Action Matplotlib] 19 | Name=Matplotlib 20 | Exec=ipython qtconsole --matplotlib 21 | 22 | [Desktop Action Matplotlibinline] 23 | Name=Matplotlib (inline plots) 24 | Exec=ipython qtconsole --matplotlib=inline 25 | -------------------------------------------------------------------------------- /Part-1/IPython Kernel/ipython.desktop: -------------------------------------------------------------------------------- 1 | # If you want ipython to appear in a linux app launcher ("start menu"), install this by doing: 2 | # sudo desktop-file-install ipython.desktop 3 | 4 | [Desktop Entry] 5 | Comment=Enhanced interactive Python shell 6 | Exec=ipython 7 | GenericName[en_US]=IPython 8 | GenericName=IPython 9 | Icon=gnome-netstatus-idle 10 | Name[en_US]=ipython 11 | Name=ipython 12 | Categories=Development;Utility; 13 | StartupNotify=false 14 | Terminal=true 15 | Type=Application 16 | -------------------------------------------------------------------------------- /Part-1/Index.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# Day 1 morning: Jupyter and IPython overview" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "In this day, we will cover the core parts of Jupyter and IPython, including how to use the various frontends, the Jupyter notebook, and how the IPython kernel goes beyond the plain Python language.\n", 22 | "\n", 23 | "**Jupyter** includes the notebook interface and various other pieces which work with multiple programming languages. **IPython** is the Jupyter backend for running Python code." 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "## Getting started\n", 31 | "\n", 32 | "* [What is the Jupyter Notebook](Notebook/What is the Jupyter Notebook.ipynb)\n", 33 | "* [Notebook Basics](Notebook/Notebook Basics.ipynb)\n", 34 | "* [Running Code](Notebook/Running Code.ipynb)\n", 35 | "* [Markdown Cells](Notebook/Working With Markdown Cells.ipynb)\n", 36 | "\n", 37 | "#### Exercises\n", 38 | "\n", 39 | "* [Counting word frequencies](exercises/WordFrequencies.ipynb)\n", 40 | "* [Monte Carlo integration](exercises/MonteCarloPi.ipynb)\n", 41 | "\n", 42 | "## Magics, specialized display and other tricks of IPython\n", 43 | "\n", 44 | "* [IPython extra language features](IPython Kernel/Beyond Plain Python.ipynb)\n", 45 | "* [Plotting in the Notebook](IPython Kernel/Plotting in the Notebook.ipynb)\n", 46 | " - **Exercise:** [Wallis' formula](exercises/Wallis-Pi.ipynb)\n", 47 | " - **Exercise:** [Numerical chaos](exercises/NumericalChaos.ipynb)\n", 48 | " - **Exercise:** [Image de-noising](exercises/fourier_moon_denoise.ipynb)\n", 49 | "* [Rich Display System](IPython Kernel/Rich Output.ipynb)\n", 50 | "* [Cell Magics](IPython Kernel/Cell Magics.ipynb)\n", 51 | "* [Embedding](Embedding/Index.ipynb): Embedding and reusing IPython's components into other applications\n", 52 | "\n", 53 | "#### Exercise\n", 54 | "\n", 55 | "* [Mapping seismic stations](exercises/mapping_seismic_stations.ipynb)\n", 56 | "\n", 57 | "-------\n", 58 | "\n", 59 | "# Additional materials for supplementary reading\n", 60 | "\n", 61 | "* [Using input and %debug in the Notebook](IPython Kernel/Input%20in%20the%20Notebook.ipynb)\n", 62 | "* [Simple Animations Using clear_output](IPython%20Kernel/Animations%20Using%20clear_output.ipynb)\n", 63 | "* [Simple interactive bacgkround jobs with IPython](IPython%20Kernel/Background%20Jobs.ipynb)\n", 64 | "* [A few things that work best/only at the IPython terminal or Qt console clients](IPython%20Kernel/Terminal%20Usage.ipynb)\n", 65 | "* [Calling arbitrary system scripts with cell magics](IPython%20Kernel/Script%20Magics.ipynb)\n", 66 | "* [Custom Display Logic](IPython Kernel/Custom Display Logic.ipynb)\n", 67 | "* [Third Party Libraries With Rich Output](IPython%20Kernel/Third%20Party%20Rich%20Output.ipynb)\n", 68 | "* [Combining symbolic and numerical computing in the IPython notebook with SymPy and NumPy](IPython%20Kernel/SymPy.ipynb)\n", 69 | "* [Configuring the Notebook and Server](Notebook/Configuring the Notebook and Server.ipynb)\n", 70 | "* [Custom Keyboard Shortcuts](Notebook/Custom Keyboard Shortcuts.ipynb)\n", 71 | "* [JavaScript Notebook Extensions](Notebook/JavaScript Notebook Extensions.ipynb)\n", 72 | "* [Converting Notebooks With nbconvert](Notebook/Converting Notebooks With nbconvert.ipynb)\n", 73 | "* [Using nbconvert as a Library](Notebook/Using nbconvert as a Library.ipynb)\n", 74 | "* [Typesetting Equations](Notebook/Typesetting%20Equations.ipynb)" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": null, 80 | "metadata": { 81 | "collapsed": true 82 | }, 83 | "outputs": [], 84 | "source": [] 85 | } 86 | ], 87 | "metadata": { 88 | "kernelspec": { 89 | "display_name": "Python 3", 90 | "language": "python", 91 | "name": "python3" 92 | }, 93 | "language_info": { 94 | "codemirror_mode": { 95 | "name": "ipython", 96 | "version": 3 97 | }, 98 | "file_extension": ".py", 99 | "mimetype": "text/x-python", 100 | "name": "python", 101 | "nbconvert_exporter": "python", 102 | "pygments_lexer": "ipython3", 103 | "version": "3.5.2" 104 | } 105 | }, 106 | "nbformat": 4, 107 | "nbformat_minor": 1 108 | } 109 | -------------------------------------------------------------------------------- /Part-1/Notebook/Connecting with the Qt Console.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Connecting to an existing IPython kernel using the Qt Console" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## The Frontend/Kernel Model" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "The traditional IPython (`ipython`) consists of a single process that combines a terminal based UI with the process that runs the users code.\n", 22 | "\n", 23 | "While this traditional application still exists, the modern IPython consists of two processes:\n", 24 | "\n", 25 | "* Kernel: this is the process that runs the users code.\n", 26 | "* Frontend: this is the process that provides the user interface where the user types code and sees results.\n", 27 | "\n", 28 | "IPython currently has 3 frontends:\n", 29 | "\n", 30 | "* Terminal Console (`ipython console`)\n", 31 | "* Qt Console (`ipython qtconsole`)\n", 32 | "* Notebook (`ipython notebook`)\n", 33 | "\n", 34 | "The Kernel and Frontend communicate over a ZeroMQ/JSON based messaging protocol, which allows multiple Frontends (even of different types) to communicate with a single Kernel. This opens the door for all sorts of interesting things, such as connecting a Console or Qt Console to a Notebook's Kernel. For example, you may want to connect a Qt console to your Notebook's Kernel and use it as a help\n", 35 | "browser, calling `??` on objects in the Qt console (whose pager is more flexible than the\n", 36 | "one in the notebook). \n", 37 | "\n", 38 | "This Notebook describes how you would connect another Frontend to a Kernel that is associated with a Notebook." 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "## Manual connection" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "To connect another Frontend to a Kernel manually, you first need to find out the connection information for the Kernel using the `%connect_info` magic:" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": { 59 | "collapsed": false 60 | }, 61 | "outputs": [], 62 | "source": [ 63 | "%connect_info" 64 | ] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "metadata": {}, 69 | "source": [ 70 | "You can see that this magic displays everything you need to connect to this Notebook's Kernel." 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "## Automatic connection using a new Qt Console" 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": {}, 83 | "source": [ 84 | "You can also start a new Qt Console connected to your current Kernel by using the `%qtconsole` magic. This will detect the necessary connection\n", 85 | "information and start the Qt Console for you automatically." 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": { 92 | "collapsed": false 93 | }, 94 | "outputs": [], 95 | "source": [ 96 | "a = 10" 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": null, 102 | "metadata": { 103 | "collapsed": false 104 | }, 105 | "outputs": [], 106 | "source": [ 107 | "%qtconsole" 108 | ] 109 | } 110 | ], 111 | "metadata": { 112 | "kernelspec": { 113 | "display_name": "Python 3", 114 | "language": "python", 115 | "name": "python3" 116 | }, 117 | "language_info": { 118 | "codemirror_mode": { 119 | "name": "ipython", 120 | "version": 3 121 | }, 122 | "file_extension": ".py", 123 | "mimetype": "text/x-python", 124 | "name": "python", 125 | "nbconvert_exporter": "python", 126 | "pygments_lexer": "ipython3", 127 | "version": "3.4.3" 128 | } 129 | }, 130 | "nbformat": 4, 131 | "nbformat_minor": 0 132 | } 133 | -------------------------------------------------------------------------------- /Part-1/Notebook/Custom Keyboard Shortcuts.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Keyboard Shortcut Customization" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "Starting with IPython 2.0 keyboard shortcuts in command and edit mode are fully customizable. These customizations are made using the IPython JavaScript API. Here is an example that makes the `r` key available for running a cell:" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": { 21 | "collapsed": false 22 | }, 23 | "outputs": [], 24 | "source": [ 25 | "%%javascript\n", 26 | "\n", 27 | "IPython.keyboard_manager.command_shortcuts.add_shortcut('r', {\n", 28 | " help : 'run cell',\n", 29 | " help_index : 'zz',\n", 30 | " handler : function (event) {\n", 31 | " IPython.notebook.execute_cell();\n", 32 | " return false;\n", 33 | " }}\n", 34 | ");" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": {}, 40 | "source": [ 41 | "There are a couple of points to mention about this API:\n", 42 | "\n", 43 | "* The `help_index` field is used to sort the shortcuts in the Keyboard Shortcuts help dialog. It defaults to `zz`.\n", 44 | "* When a handler returns `false` it indicates that the event should stop propagating and the default action should not be performed. For further details about the `event` object or event handling, see the jQuery docs.\n", 45 | "* If you don't need a `help` or `help_index` field, you can simply pass a function as the second argument to `add_shortcut`." 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": { 52 | "collapsed": false 53 | }, 54 | "outputs": [], 55 | "source": [ 56 | "%%javascript\n", 57 | "\n", 58 | "IPython.keyboard_manager.command_shortcuts.add_shortcut('r', function (event) {\n", 59 | " IPython.notebook.execute_cell();\n", 60 | " return false;\n", 61 | "});" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "Likewise, to remove a shortcut, use `remove_shortcut`:" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": { 75 | "collapsed": false 76 | }, 77 | "outputs": [], 78 | "source": [ 79 | "%%javascript\n", 80 | "\n", 81 | "IPython.keyboard_manager.command_shortcuts.remove_shortcut('r');" 82 | ] 83 | }, 84 | { 85 | "cell_type": "markdown", 86 | "metadata": {}, 87 | "source": [ 88 | "If you want your keyboard shortcuts to be active for all of your notebooks, put the above API calls into your `/static/custom/custom.js` file." 89 | ] 90 | } 91 | ], 92 | "metadata": { 93 | "kernelspec": { 94 | "display_name": "Python 3", 95 | "language": "python", 96 | "name": "python3" 97 | }, 98 | "language_info": { 99 | "codemirror_mode": { 100 | "name": "ipython", 101 | "version": 3 102 | }, 103 | "file_extension": ".py", 104 | "mimetype": "text/x-python", 105 | "name": "python", 106 | "nbconvert_exporter": "python", 107 | "pygments_lexer": "ipython3", 108 | "version": "3.4.3" 109 | } 110 | }, 111 | "nbformat": 4, 112 | "nbformat_minor": 0 113 | } 114 | -------------------------------------------------------------------------------- /Part-1/Notebook/Index.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "Back to the main [Index](../Index.ipynb)" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "# Notebook" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "The IPython Notebook is a web-based interactive computing system that enables users to author documents that include live code, narrative text, LaTeX equations, HTML, images and video. These documents contain a full record of a computation and its results and can be shared on email, [Dropbox](http://dropbox.com), version control systems (like git/[GitHub](http://github.com)) or [nbviewer.ipython.org](http://nbviewer.ipython.org)." 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "## Tutorials" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "* [What is the IPython Notebook](What is the IPython Notebook.ipynb)\n", 43 | "* [Notebook Basics](Notebook Basics.ipynb)\n", 44 | "* [Running Code](Running Code.ipynb)\n", 45 | "* [Working With Markdown Cells](Working With Markdown Cells.ipynb)\n", 46 | "* [Configuring the Notebook and Server](Configuring the Notebook and Server.ipynb)\n", 47 | "* [Custom Keyboard Shortcuts](Custom Keyboard Shortcuts.ipynb)\n", 48 | "* [JavaScript Notebook Extensions](JavaScript Notebook Extensions.ipynb)\n", 49 | "* [Converting Notebooks With nbconvert](Converting Notebooks With nbconvert.ipynb)\n", 50 | "* [Using nbconvert as a Library](Using nbconvert as a Library.ipynb)" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "## Examples" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "metadata": {}, 63 | "source": [ 64 | "* [Importing Notebooks](Importing Notebooks.ipynb)\n", 65 | "* [Connecting with the Qt Console](Connecting with the Qt Console.ipynb)\n", 66 | "* [Typesetting Equations](Typesetting Equations.ipynb)" 67 | ] 68 | } 69 | ], 70 | "metadata": { 71 | "kernelspec": { 72 | "display_name": "Python 3", 73 | "language": "python", 74 | "name": "python3" 75 | }, 76 | "language_info": { 77 | "codemirror_mode": { 78 | "name": "ipython", 79 | "version": 3 80 | }, 81 | "file_extension": ".py", 82 | "mimetype": "text/x-python", 83 | "name": "python", 84 | "nbconvert_exporter": "python", 85 | "pygments_lexer": "ipython3", 86 | "version": "3.4.3" 87 | } 88 | }, 89 | "nbformat": 4, 90 | "nbformat_minor": 0 91 | } 92 | -------------------------------------------------------------------------------- /Part-1/Notebook/images/command_mode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/Notebook/images/command_mode.png -------------------------------------------------------------------------------- /Part-1/Notebook/images/dashboard_files_tab.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/Notebook/images/dashboard_files_tab.png -------------------------------------------------------------------------------- /Part-1/Notebook/images/dashboard_files_tab_btns.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/Notebook/images/dashboard_files_tab_btns.png -------------------------------------------------------------------------------- /Part-1/Notebook/images/dashboard_files_tab_new.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/Notebook/images/dashboard_files_tab_new.png -------------------------------------------------------------------------------- /Part-1/Notebook/images/dashboard_files_tab_run.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/Notebook/images/dashboard_files_tab_run.png -------------------------------------------------------------------------------- /Part-1/Notebook/images/dashboard_running_tab.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/Notebook/images/dashboard_running_tab.png -------------------------------------------------------------------------------- /Part-1/Notebook/images/edit_mode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/Notebook/images/edit_mode.png -------------------------------------------------------------------------------- /Part-1/Notebook/images/menubar_toolbar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/Notebook/images/menubar_toolbar.png -------------------------------------------------------------------------------- /Part-1/Notebook/images/nbconvert_arch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/Notebook/images/nbconvert_arch.png -------------------------------------------------------------------------------- /Part-1/Notebook/nbpackage/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/Notebook/nbpackage/__init__.py -------------------------------------------------------------------------------- /Part-1/Notebook/nbpackage/mynotebook.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# My Notebook" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 1, 13 | "metadata": { 14 | "collapsed": false 15 | }, 16 | "outputs": [], 17 | "source": [ 18 | "def foo():\n", 19 | " return \"foo\"" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": 2, 25 | "metadata": { 26 | "collapsed": false 27 | }, 28 | "outputs": [], 29 | "source": [ 30 | "def has_ip_syntax():\n", 31 | " listing = !ls\n", 32 | " return listing" 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 4, 38 | "metadata": { 39 | "collapsed": false 40 | }, 41 | "outputs": [], 42 | "source": [ 43 | "def whatsmyname():\n", 44 | " return __name__" 45 | ] 46 | } 47 | ], 48 | "metadata": { 49 | "kernelspec": { 50 | "display_name": "Python 3", 51 | "language": "python", 52 | "name": "python3" 53 | }, 54 | "language_info": { 55 | "codemirror_mode": { 56 | "name": "ipython", 57 | "version": 3 58 | }, 59 | "file_extension": ".py", 60 | "mimetype": "text/x-python", 61 | "name": "python", 62 | "nbconvert_exporter": "python", 63 | "pygments_lexer": "ipython3", 64 | "version": "3.4.2" 65 | } 66 | }, 67 | "nbformat": 4, 68 | "nbformat_minor": 0 69 | } 70 | -------------------------------------------------------------------------------- /Part-1/Notebook/nbpackage/nbs/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/Notebook/nbpackage/nbs/__init__.py -------------------------------------------------------------------------------- /Part-1/Notebook/nbpackage/nbs/other.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "This notebook just defines `bar`" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 2, 13 | "metadata": { 14 | "collapsed": false 15 | }, 16 | "outputs": [], 17 | "source": [ 18 | "def bar(x):\n", 19 | " return \"bar\" * x" 20 | ] 21 | } 22 | ], 23 | "metadata": { 24 | "kernelspec": { 25 | "display_name": "Python 3", 26 | "language": "python", 27 | "name": "python3" 28 | }, 29 | "language_info": { 30 | "codemirror_mode": { 31 | "name": "ipython", 32 | "version": 3 33 | }, 34 | "file_extension": ".py", 35 | "mimetype": "text/x-python", 36 | "name": "python", 37 | "nbconvert_exporter": "python", 38 | "pygments_lexer": "ipython3", 39 | "version": "3.4.2" 40 | } 41 | }, 42 | "nbformat": 4, 43 | "nbformat_minor": 0 44 | } 45 | -------------------------------------------------------------------------------- /Part-1/exercises/MonteCarloPi-solution.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Simple generation of $\\pi$ via MonteCarlo integration.\n", 8 | "\n", 9 | "We compute $\\pi$ as the area of a unit circle, and we compute this area by\n", 10 | "integration:\n", 11 | "\n", 12 | "$$\n", 13 | "\\pi = 4\\int_0^1{\\sqrt{1-x^2}}\n", 14 | "$$\n", 15 | "\n", 16 | "This integral can be then done via MonteCarlo integration.\n", 17 | "\n", 18 | "Define a function that approximates this integral by averaging N random samples of the integrand.\n", 19 | "\n", 20 | "*Hint* use the math and random modules from the std library." 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": null, 26 | "metadata": { 27 | "collapsed": true 28 | }, 29 | "outputs": [], 30 | "source": [ 31 | "import math\n", 32 | "import random\n", 33 | "\n", 34 | "def m_pi(n = 100000):\n", 35 | " \"\"\"Approximate pi via monte carlo integration\"\"\"\n", 36 | "\n", 37 | " rand = random.random\n", 38 | " sqrt = math.sqrt\n", 39 | " sm = 0.0\n", 40 | " for i in range(n):\n", 41 | " sm += sqrt(1.0-rand()**2)\n", 42 | " return 4.0*sm/n" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": {}, 48 | "source": [ 49 | "Print the results for the default sampling of N=100,000:" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": 16, 55 | "metadata": { 56 | "collapsed": false 57 | }, 58 | "outputs": [ 59 | { 60 | "name": "stdout", 61 | "output_type": "stream", 62 | "text": [ 63 | "π = 3.141592653589793\n", 64 | "π ~ 3.1443644521879284\n", 65 | "relative error: 0.0882291 % \n" 66 | ] 67 | } 68 | ], 69 | "source": [ 70 | "print('π =', math.pi)\n", 71 | "approx = m_pi()\n", 72 | "print('π ~', approx)\n", 73 | "print('relative error: %g %% ' % (100*abs((approx-math.pi)/math.pi)) )" 74 | ] 75 | } 76 | ], 77 | "metadata": { 78 | "kernelspec": { 79 | "display_name": "Python 3", 80 | "language": "python", 81 | "name": "python3" 82 | }, 83 | "language_info": { 84 | "codemirror_mode": { 85 | "name": "ipython", 86 | "version": 3 87 | }, 88 | "file_extension": ".py", 89 | "mimetype": "text/x-python", 90 | "name": "python", 91 | "nbconvert_exporter": "python", 92 | "pygments_lexer": "ipython3", 93 | "version": "3.4.3" 94 | } 95 | }, 96 | "nbformat": 4, 97 | "nbformat_minor": 0 98 | } 99 | -------------------------------------------------------------------------------- /Part-1/exercises/MonteCarloPi.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Simple generation of $\\pi$ via MonteCarlo integration\n", 8 | "\n", 9 | "We compute $\\pi$ as the area of a unit circle (radius=1), and we compute this area by\n", 10 | "integration:\n", 11 | "\n", 12 | "$$\n", 13 | "\\pi = 4\\int_0^1{\\sqrt{1-x^2}}\n", 14 | "$$\n", 15 | "\n", 16 | "What does this mean? We're taking thin vertical slices of a quarter-circle, and measuring the height of each to see how much area the segment covers.\n", 17 | "\n", 18 | "![quarter circle with vertical slices](circle_integrate.png)\n", 19 | "\n", 20 | "This integral can be then done via Monte Carlo integration: pick random values of $x$, calculate $\\sqrt{1-x^2}$ for each, and average these values.\n", 21 | "\n", 22 | "Define a function that approximates pi by averaging $N$ random samples of the integrand.\n", 23 | "\n", 24 | "*Hint* use the math and random modules from the std library." 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": null, 30 | "metadata": {}, 31 | "outputs": [], 32 | "source": [ 33 | "import math\n", 34 | "import random\n", 35 | "\n", 36 | "def m_pi(n = 100000):\n", 37 | " \"\"\"Approximate pi via monte carlo integration\"\"\"\n", 38 | " # Your code here" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "Print the results for the default sampling of $N=100,000$, and check the accuracy:" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "print('π =', math.pi)\n", 55 | "approx = m_pi()\n", 56 | "print('π ~', approx)\n", 57 | "print('relative error: %g %% ' % (100*abs((approx-math.pi)/math.pi)) )" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "metadata": {}, 63 | "source": [ 64 | "Think about extending this to multiple dimensions. The volume of a hypersphere,\n", 65 | "\n", 66 | "$$\n", 67 | " V_D = \\int dx_1 \\int \\dots \\int dx_D \\, \\sqrt{1 - \\sum_{i=1}^D x_i^2},\n", 68 | "$$\n", 69 | "\n", 70 | "can be [computed in closed form](https://en.wikipedia.org/wiki/Volume_of_an_n-ball#The_volume). How would you do this with Monte Carlo integration? What would be the advantage?" 71 | ] 72 | } 73 | ], 74 | "metadata": { 75 | "kernelspec": { 76 | "display_name": "Python 3", 77 | "language": "python", 78 | "name": "python3" 79 | }, 80 | "language_info": { 81 | "codemirror_mode": { 82 | "name": "ipython", 83 | "version": 3 84 | }, 85 | "file_extension": ".py", 86 | "mimetype": "text/x-python", 87 | "name": "python", 88 | "nbconvert_exporter": "python", 89 | "pygments_lexer": "ipython3", 90 | "version": "3.5.2" 91 | } 92 | }, 93 | "nbformat": 4, 94 | "nbformat_minor": 1 95 | } 96 | -------------------------------------------------------------------------------- /Part-1/exercises/NumericalChaos.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Numerical chaos in the logistic map and floating point error\n", 8 | "\n", 9 | "One of the most classic examples of chaotic behavior in non-linear\n", 10 | "systems is the iteration of the logistic map\n", 11 | "\n", 12 | "$$\n", 13 | "x_{n+1} = f(x_n) = r x_n (1-x_n)\n", 14 | "$$\n", 15 | "\n", 16 | "which for $x \\in (0,1)$ and $r \\in (0,4)$ can produce very surprising\n", 17 | "behavior. For now we use this system to illustrate numerical\n", 18 | "roundoff error.\n", 19 | "\n", 20 | "Computers, when performing almost any floating point operation, must by\n", 21 | "necessity throw away information from the digits that can't be stored at\n", 22 | "any finite precision. This has a simple implication that is nonetheless\n", 23 | "often overlooked: algebraically equivalent forms of the same expression\n", 24 | "aren't necessarily always numerically equivalent. A simple illustration\n", 25 | "shows the problem very easily:\n", 26 | "\n", 27 | "For this exercise, try to find three different ways to express $f(x)$\n", 28 | "in the logistic map and compute the evolution of the same initial condition\n", 29 | "after a few hundred iterations. For this problem, it will be extremely\n", 30 | "useful to look at your results graphically; simply build lists of\n", 31 | "numbers and call matplotlib's `plot` function to look at how each trace\n", 32 | "evolves.\n", 33 | "\n", 34 | "The following snippet can be used as a starting point, and it includes\n", 35 | "some hints of what values of $r$ to look at:" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": 7, 41 | "metadata": { 42 | "collapsed": true 43 | }, 44 | "outputs": [], 45 | "source": [ 46 | "\"\"\"Illustrating error propagation by iterating the logistic map.\n", 47 | "\n", 48 | "f(x) = r*x*(1-x)\n", 49 | "\n", 50 | "Write the above function in three algebraically equivalent forms, and study\n", 51 | "their behavior under iteration. See for what values of r all forms evolve\n", 52 | "identically and for which ones they don't.\n", 53 | "\"\"\"\n", 54 | "\n", 55 | "import matplotlib.pyplot as plt\n", 56 | "\n", 57 | "# Interesting values to try for r:\n", 58 | "# [1.9, 2.9, 3.1, 3.5, 3.9]\n", 59 | "r = 3.9 # global default\n", 60 | "x0 = 0.6 # any number in [0,1] will do here\n", 61 | "num_points = 100 # total number of points to compute\n", 62 | "drop_points = 0 # don't display the first drop_points" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": {}, 68 | "source": [ 69 | "The three algebraically, but not numerically, equivalent forms of $f(x)$:" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": 4, 75 | "metadata": { 76 | "collapsed": true 77 | }, 78 | "outputs": [], 79 | "source": [ 80 | "def f1(x): return # Your code here\n", 81 | "def f2(x): return # Your code here\n", 82 | "def f3(x): return # Your code here" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": {}, 88 | "source": [ 89 | "Now, we compute and plot results with these three forms of $f(x)$:" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": 1, 95 | "metadata": { 96 | "collapsed": false 97 | }, 98 | "outputs": [], 99 | "source": [ 100 | "%matplotlib inline\n", 101 | "\n", 102 | " # Your code here" 103 | ] 104 | } 105 | ], 106 | "metadata": { 107 | "kernelspec": { 108 | "display_name": "Python 3", 109 | "language": "python", 110 | "name": "python3" 111 | }, 112 | "language_info": { 113 | "codemirror_mode": { 114 | "name": "ipython", 115 | "version": 3 116 | }, 117 | "file_extension": ".py", 118 | "mimetype": "text/x-python", 119 | "name": "python", 120 | "nbconvert_exporter": "python", 121 | "pygments_lexer": "ipython3", 122 | "version": "3.4.3" 123 | } 124 | }, 125 | "nbformat": 4, 126 | "nbformat_minor": 0 127 | } 128 | -------------------------------------------------------------------------------- /Part-1/exercises/Wallis-Pi.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Wallis' formula for $\\pi$\n", 8 | "\n", 9 | "Wallis' formula is a slowly converging infinite product that approximates pi as\n", 10 | "\n", 11 | "$$\n", 12 | " \\pi = \\lim_{n \\rightarrow \\infty} 2 \\prod_{i=1}^{n}\\frac{4i^2}{4i^2-1}.\n", 13 | "$$\n", 14 | "\n", 15 | "While this isn't a particularly good way of computing $\\pi$ from a numerical\n", 16 | "standpoint, it provides for an excellent illustration of how Python's integers\n", 17 | "are more flexible and powerful than those typically found by default in\n", 18 | "compiled languages like C and Fortran. The problem is that for\n", 19 | "this formula to be even remotely accurate, one must evaluate it for fairly\n", 20 | "large values of $n$, where both the numerator and the denominator will easily\n", 21 | "overflow the limits of 64-bit integers. It is only after taking the ratio of\n", 22 | "these two huge numbers that the value is small (close to $\\pi$).\n", 23 | "\n", 24 | "Fortunately for us, Python integers automatically allocate as many digits as\n", 25 | "necessary (within the limits of physically available memory) to hold their\n", 26 | "result. So while implementing the above in C or Fortran (without auxilliary\n", 27 | "libraries like [GMP](http://gmplib.org)) would be fairly tricky, in Python it's very\n", 28 | "straightforward.\n", 29 | "\n", 30 | "For this exercise, write a program that implements the above formula. Note\n", 31 | "that Python's `math` module already contains $\\pi$ in double precision, so\n", 32 | "you can use this value to compare your results:" 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 1, 38 | "metadata": { 39 | "collapsed": false 40 | }, 41 | "outputs": [ 42 | { 43 | "data": { 44 | "text/plain": [ 45 | "3.141592653589793" 46 | ] 47 | }, 48 | "execution_count": 1, 49 | "metadata": {}, 50 | "output_type": "execute_result" 51 | } 52 | ], 53 | "source": [ 54 | "import math\n", 55 | "math.pi" 56 | ] 57 | }, 58 | { 59 | "cell_type": "markdown", 60 | "metadata": {}, 61 | "source": [ 62 | "Plot how the accuracy depends on $n$." 63 | ] 64 | } 65 | ], 66 | "metadata": { 67 | "kernelspec": { 68 | "display_name": "Python 3", 69 | "language": "python", 70 | "name": "python3" 71 | }, 72 | "language_info": { 73 | "codemirror_mode": { 74 | "name": "ipython", 75 | "version": 3 76 | }, 77 | "file_extension": ".py", 78 | "mimetype": "text/x-python", 79 | "name": "python", 80 | "nbconvert_exporter": "python", 81 | "pygments_lexer": "ipython3", 82 | "version": "3.4.3" 83 | } 84 | }, 85 | "nbformat": 4, 86 | "nbformat_minor": 0 87 | } 88 | -------------------------------------------------------------------------------- /Part-1/exercises/WordFrequencies.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Dictionaries for counting words\n", 8 | "\n", 9 | "A common task in text processing is to produce a count of word\n", 10 | "frequencies. While NumPy has a builtin histogram function for doing\n", 11 | "numerical histograms, it won't work out of the box for counting discrete\n", 12 | "items, since it is a binning histogram for a range of real values.\n", 13 | "\n", 14 | "But the Python language provides very powerful string manipulation\n", 15 | "capabilities, as well as a very flexible and efficiently implemented\n", 16 | "builtin data type, the *dictionary*, that makes this task a very simple\n", 17 | "one.\n", 18 | "\n", 19 | "In this problem, you will need to count the frequencies of all the words\n", 20 | "contained in a compressed text file supplied as input. Load and read the\n", 21 | "data file `data/HISTORY.gz` (without uncompressing it on the filesystem\n", 22 | "separately), and then use a dictionary count the frequency of each word\n", 23 | "in the file. Then, display the 20 most and 20 least frequent words in\n", 24 | "the text.\n", 25 | "\n", 26 | "## Hints\n", 27 | "\n", 28 | "- To read the compressed file `HISTORY.gz` without uncompressing it\n", 29 | " first, see the [gzip module](https://docs.python.org/3/library/gzip.html).\n", 30 | "- Consider 'words' as the result of splitting the input text into\n", 31 | " a list, using any form of whitespace as a separator. This is\n", 32 | " obviously a very naive definition of 'word', but it shall suffice\n", 33 | " for the purposes of this exercise.\n", 34 | "- Python strings have a `.split()` method that allows for very\n", 35 | " flexible splitting. You can easily get more details on it in\n", 36 | " IPython:\n", 37 | "\n", 38 | "\n", 39 | "```\n", 40 | " In [2]: a = 'somestring'\n", 41 | "\n", 42 | " In [3]: a.split?\n", 43 | " Type: builtin_function_or_method\n", 44 | " Base Class: \n", 45 | " Namespace: Interactive\n", 46 | " Docstring:\n", 47 | " S.split([sep [,maxsplit]]) -> list of strings\n", 48 | "\n", 49 | " Return a list of the words in the string S, using sep as the\n", 50 | " delimiter string. If maxsplit is given, at most maxsplit\n", 51 | " splits are done. If sep is not specified or is None, any\n", 52 | " whitespace string is a separator.\n", 53 | "```\n", 54 | "\n", 55 | "The complete set of methods of Python strings can be viewed by hitting\n", 56 | "the TAB key in IPython after typing `a.`, and each of them can be\n", 57 | "similarly queried with the `?` operator as above. For more details on\n", 58 | "Python strings and their companion sequence types, see\n", 59 | "[the Python documentation](https://docs.python.org/3/library/stdtypes.html#sequence-types-list-tuple-range).\n" 60 | ] 61 | } 62 | ], 63 | "metadata": { 64 | "kernelspec": { 65 | "display_name": "Python 3", 66 | "language": "python", 67 | "name": "python3" 68 | }, 69 | "language_info": { 70 | "codemirror_mode": { 71 | "name": "ipython", 72 | "version": 3 73 | }, 74 | "file_extension": ".py", 75 | "mimetype": "text/x-python", 76 | "name": "python", 77 | "nbconvert_exporter": "python", 78 | "pygments_lexer": "ipython3", 79 | "version": "3.5.2" 80 | } 81 | }, 82 | "nbformat": 4, 83 | "nbformat_minor": 1 84 | } 85 | -------------------------------------------------------------------------------- /Part-1/exercises/circle_integrate.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/exercises/circle_integrate.png -------------------------------------------------------------------------------- /Part-1/exercises/circle_integrate.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 22 | 24 | 46 | 48 | 49 | 51 | image/svg+xml 52 | 54 | 55 | 56 | 57 | 58 | 63 | 70 | 75 | 80 | 85 | 96 | 97 | 98 | -------------------------------------------------------------------------------- /Part-1/exercises/data/CallRingingIn.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/exercises/data/CallRingingIn.wav -------------------------------------------------------------------------------- /Part-1/exercises/data/HISTORY.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/exercises/data/HISTORY.gz -------------------------------------------------------------------------------- /Part-1/exercises/data/dessert.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/exercises/data/dessert.png -------------------------------------------------------------------------------- /Part-1/exercises/data/moon_denoise.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/exercises/data/moon_denoise.png -------------------------------------------------------------------------------- /Part-1/exercises/data/moonlanding.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/exercises/data/moonlanding.png -------------------------------------------------------------------------------- /Part-1/exercises/data/stained_glass_barcelona.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/exercises/data/stained_glass_barcelona.png -------------------------------------------------------------------------------- /Part-1/exercises/data/stations.txt: -------------------------------------------------------------------------------- 1 | station lat long elev 2 | BIRA 26.4840 87.2670 0.0120 3 | BUNG 27.8771 85.8909 1.1910 4 | GAIG 26.8380 86.6318 0.1660 5 | HILE 27.0482 87.3242 2.0880 6 | ILAM 26.9102 87.9227 1.1810 7 | JIRI 27.6342 86.2303 1.8660 8 | NAMC 27.8027 86.7146 3.5230 9 | PHAP 27.5150 86.5842 2.4880 10 | PHID 27.1501 87.7645 1.1760 11 | RUMJ 27.3038 86.5482 1.3190 12 | SIND 27.2107 85.9088 0.4650 13 | THAK 27.5996 85.5566 1.5510 14 | TUML 27.3208 87.1950 0.3600 15 | LAZE 29.1403 87.5922 4.0110 16 | SAJA 28.9093 88.0209 4.3510 17 | ONRN 29.3020 87.2440 4.3500 18 | SSAN 29.4238 86.7290 4.5850 19 | SAGA 29.3292 85.2321 4.5240 20 | DINX 28.6646 87.1157 4.3740 21 | RBSH 28.1955 86.8280 5.1000 22 | NAIL 28.6597 86.4126 4.3780 23 | MNBU 28.7558 86.1610 4.5000 24 | NLMU 28.1548 85.9777 3.8890 25 | YALA 28.4043 86.1133 4.4340 26 | XIXI 28.7409 85.6904 4.6600 27 | RC14 29.4972 86.4373 4.7560 28 | MAZA 28.6713 87.8553 4.3670 29 | JANA 26.7106 85.9242 0.0770 30 | SUKT 27.7057 85.7611 0.7450 31 | -------------------------------------------------------------------------------- /Part-1/exercises/fourier_moon_denoise-solution.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Simple image denoising example using 2-dimensional FFT\n", 8 | "\n", 9 | "In this example we illustrate a very elementary type of image denoising \n", 10 | "with the two-dimensional Fourier Transform." 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "metadata": { 17 | "collapsed": true 18 | }, 19 | "outputs": [], 20 | "source": [ 21 | "%matplotlib inline\n", 22 | "\n", 23 | "import sys\n", 24 | "import numpy as np\n", 25 | "import matplotlib.pyplot as plt" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": null, 31 | "metadata": { 32 | "collapsed": true 33 | }, 34 | "outputs": [], 35 | "source": [ 36 | "def plot_spectrum(F, amplify=1000, ax=None):\n", 37 | " \"\"\"Normalise, amplify and plot an amplitude spectrum.\"\"\"\n", 38 | "\n", 39 | " # Note: the problem here is that we have a spectrum whose histogram is\n", 40 | " # *very* sharply peaked at small values. To get a meaningful display, a\n", 41 | " # simple strategy to improve the display quality consists of simply\n", 42 | " # amplifying the values in the array and then clipping.\n", 43 | "\n", 44 | " # Compute the magnitude of the input F (call it mag). Then, rescale mag by\n", 45 | " # amplify/maximum_of_mag.\n", 46 | " mag = abs(F) \n", 47 | " mag *= amplify/mag.max() \n", 48 | " \n", 49 | " # Next, clip all values larger than one to one.\n", 50 | " mag[mag > 1] = 1 \n", 51 | "\n", 52 | " if ax is None: ax = plt.gca()\n", 53 | " ax.imshow(mag, plt.cm.Blues)" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "Read in original image, convert to floating point for further\n", 61 | "manipulation; imread returns a MxNx4 RGBA image. Since the image is\n", 62 | "grayscale, just extract the 1st channel\n", 63 | "\n", 64 | "**Hints:**\n", 65 | "\n", 66 | "* use plt.imread() to load the file\n", 67 | "* convert to a float array with the .astype() method\n", 68 | "* extract all rows, all columns, 0-th plane to get the first\n", 69 | " channel\n", 70 | "* the resulting array should have 2 dimensions only" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": null, 76 | "metadata": { 77 | "collapsed": false 78 | }, 79 | "outputs": [], 80 | "source": [ 81 | "fname = 'data/moonlanding.png'\n", 82 | "im = plt.imread(fname).astype(float) \n", 83 | "print(\"Image shape: %s\" % str(im.shape))" 84 | ] 85 | }, 86 | { 87 | "cell_type": "markdown", 88 | "metadata": {}, 89 | "source": [ 90 | "Compute the 2d FFT of the input image\n", 91 | "\n", 92 | "Hint: Look for a 2-d FFT in np.fft.\n", 93 | "\n", 94 | "Note: call this variable 'F', which is the name we'll be using below.\n", 95 | "\n", 96 | "In the lines following, we'll make a copy of the original spectrum and\n", 97 | "truncate coefficients. " 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": { 104 | "collapsed": true 105 | }, 106 | "outputs": [], 107 | "source": [ 108 | "F = np.fft.fft2(im) \n", 109 | "\n", 110 | "# Define the fraction of coefficients (in each direction) we keep\n", 111 | "keep_fraction = 0.1\n", 112 | "\n", 113 | "# Call ff a copy of the original transform. Numpy arrays have a copy\n", 114 | "# method for this purpose.\n", 115 | "ff = F.copy() \n", 116 | "\n", 117 | "# Set r and c to be the number of rows and columns of the array.\n", 118 | "r,c = ff.shape \n", 119 | "\n", 120 | "# Set to zero all rows with indices between r*keep_fraction and\n", 121 | "# r*(1-keep_fraction):\n", 122 | "ff[r*keep_fraction:r*(1-keep_fraction)] = 0 \n", 123 | "\n", 124 | "# Similarly with the columns:\n", 125 | "ff[:, c*keep_fraction:c*(1-keep_fraction)] = 0 " 126 | ] 127 | }, 128 | { 129 | "cell_type": "markdown", 130 | "metadata": {}, 131 | "source": [ 132 | "Reconstruct the denoised image from the filtered spectrum, keep only the\n", 133 | "real part for display.\n", 134 | "Hint: There's an inverse 2d fft in the np.fft module as well (don't\n", 135 | "forget that you only want the real part).\n", 136 | "Call the result im_new and plot the results" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": null, 142 | "metadata": { 143 | "collapsed": false 144 | }, 145 | "outputs": [], 146 | "source": [ 147 | "im_new = np.fft.ifft2(ff).real \n", 148 | "\n", 149 | "fig, ax = plt.subplots(2, 2, figsize=(10,7))\n", 150 | "\n", 151 | "ax[0,0].set_title('Original image')\n", 152 | "ax[0,0].imshow(im, plt.cm.gray)\n", 153 | "\n", 154 | "ax[0,1].set_title('Fourier transform')\n", 155 | "plot_spectrum(F, ax=ax[0,1])\n", 156 | "\n", 157 | "ax[1,1].set_title('Filtered Spectrum')\n", 158 | "plot_spectrum(ff, ax=ax[1,1])\n", 159 | "\n", 160 | "ax[1,0].set_title('Reconstructed Image')\n", 161 | "ax[1,0].imshow(im_new, plt.cm.gray);" 162 | ] 163 | } 164 | ], 165 | "metadata": { 166 | "kernelspec": { 167 | "display_name": "Python 3", 168 | "language": "python", 169 | "name": "python3" 170 | }, 171 | "language_info": { 172 | "codemirror_mode": { 173 | "name": "ipython", 174 | "version": 3 175 | }, 176 | "file_extension": ".py", 177 | "mimetype": "text/x-python", 178 | "name": "python", 179 | "nbconvert_exporter": "python", 180 | "pygments_lexer": "ipython3", 181 | "version": "3.4.3" 182 | } 183 | }, 184 | "nbformat": 4, 185 | "nbformat_minor": 0 186 | } 187 | -------------------------------------------------------------------------------- /Part-1/exercises/fourier_moon_denoise.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# FFT Image Denoising\n", 8 | "\n", 9 | "\n", 10 | "**Illustrates**: 2-d image denoising, use of the numpy FFT library,\n", 11 | "array manipulations, image plotting.\n", 12 | "\n", 13 | "The convolution of an input with a linear filter in the temporal\n", 14 | "or spatial domain is equivalent to multiplication by the Fourier\n", 15 | "transforms of the input and the filter in the spectral domain. This\n", 16 | "provides a conceptually simple way to think about filtering: transform\n", 17 | "your signal into the frequency domain, dampen the frequencies you are\n", 18 | "not interested in by multiplying the frequency spectrum by the desired\n", 19 | "weights, and then apply the inverse transform to the modified spectrum,\n", 20 | "back into the original domain. In the example below, we will simply set\n", 21 | "the weights of the frequencies we are uninterested in (the high\n", 22 | "frequency noise) to zero rather than dampening them with a smoothly\n", 23 | "varying function. Although this is not usually the best thing to do,\n", 24 | "since sharp edges in one domain usually introduce artifacts in another\n", 25 | "(eg high frequency \"ringing\"), it is easy to do and sometimes provides\n", 26 | "satisfactory results.\n", 27 | "\n", 28 | "![Figure](data/moon_denoise.png)\n", 29 | "\n", 30 | "High frequency noise filtering of a 2D image in the Fourier domain.\n", 31 | "The upper panels show the original image (left) and spectral power\n", 32 | "(right) and the lower panels show the same data with the high frequency\n", 33 | "power set to zero. Although the input and output images are grayscale,\n", 34 | "you can provide colormaps to `imshow` to plot them in pseudo-color, with\n", 35 | "the `cmap` argument to `imshow`, which accepts any of the colormaps found\n", 36 | "in the `matplotlib.cm` module.\n", 37 | "\n", 38 | "You can use the function `imread` (in the module `matplotlib.pyplot`)\n", 39 | "to read the image file `data/moonlanding.png` into a `numpy.array`.\n", 40 | "After doing this, try to produce images like the ones in the figure.\n", 41 | "We will describe the process here and provide some hints as to what\n", 42 | "you need to think about.\n", 43 | "\n", 44 | "The image in the upper left panel of the Figure is a grayscale photo of\n", 45 | "the moon landing. There is a banded pattern of high frequency noise\n", 46 | "polluting the image. In the upper right panel we see the 2D spatial\n", 47 | "frequency spectrum. The FFT output in the `numpy.fft` module is packed\n", 48 | "with the lower freqeuencies starting in the upper left, and proceeding\n", 49 | "to higher frequencies as one moves to the center of the spectrum (this\n", 50 | "is the most efficient way numerically to fill the output of the FFT\n", 51 | "algorithm). Because the input signal is real, the output spectrum is\n", 52 | "complex and symmetrical: the transformation values beyond the midpoint\n", 53 | "of the frequency spectrum (the Nyquist frequency) correspond to the\n", 54 | "values for negative frequencies and are simply the mirror image of the\n", 55 | "positive frequencies below the Nyquist (this is true for the 1D, 2D and\n", 56 | "ND FFTs in `numpy`).\n", 57 | "\n", 58 | "You should compute the 2D spatial frequency spectra of the luminance\n", 59 | "image, zero out the high frequency components, and inverse transform\n", 60 | "back into the spatial domain. You can plot the input and output images\n", 61 | "with `plt.imshow`, but you should observe that if you show the power\n", 62 | "spectrum (the absolute value of the FFT) directly, you will only see\n", 63 | "white, and not the image in the Figure's upper right panel. This is due\n", 64 | "to the fact that the power spectrum has a small number of pixels with\n", 65 | "extremely high amplitude, which completely swamp the contrast (you can\n", 66 | "verify this by playing with a histogram of the data). You will thus need\n", 67 | "to clip the limits of the color range to a small range of values where\n", 68 | "most of the power actually lives (say 95%).\n", 69 | "\n", 70 | "Hints\n", 71 | "-----\n", 72 | "\n", 73 | "- The upper right panel is obtained after finding the range of values\n", 74 | " that contain 95% of the power. For this, the `numpy.percentile` function\n", 75 | " will be useful.\n", 76 | "- The `numpy.fft` module contains the necessary FFT routines for this\n", 77 | " exercise.\n", 78 | "- In Python, a complex number `z` has `z.real` and `z.imag` attributes\n", 79 | " for its real and imaginary parts." 80 | ] 81 | } 82 | ], 83 | "metadata": { 84 | "kernelspec": { 85 | "display_name": "Python 3", 86 | "language": "python", 87 | "name": "python3" 88 | }, 89 | "language_info": { 90 | "codemirror_mode": { 91 | "name": "ipython", 92 | "version": 3 93 | }, 94 | "file_extension": ".py", 95 | "mimetype": "text/x-python", 96 | "name": "python", 97 | "nbconvert_exporter": "python", 98 | "pygments_lexer": "ipython3", 99 | "version": "3.4.3" 100 | } 101 | }, 102 | "nbformat": 4, 103 | "nbformat_minor": 0 104 | } 105 | -------------------------------------------------------------------------------- /Part-1/exercises/mapping_seismic_stations_interactive.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Mapping seismic stations in the Himalayas with Numpy and Matplotlib\n", 8 | "## Part II - Interactive event handling" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "metadata": {}, 14 | "source": [ 15 | "Here, we extend the plot above by adding an event handler that prints the location (four-letter string) of the station you click on.\n", 16 | "\n", 17 | "We use a threshold for distance, and discriminate between a click below threshold (considered to be 'on') vs a miss, in which case we indicate what the closest station is, its coordinates and the distance to it from the click.\n", 18 | "\n", 19 | "In order to get interactive plot windows that support event handling, we must restart the kernel and activate pylab but *not* in `inline` mode." 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": { 26 | "collapsed": false 27 | }, 28 | "outputs": [], 29 | "source": [ 30 | "%matplotlib\n", 31 | "import sys\n", 32 | "import numpy as np\n", 33 | "import matplotlib.pyplot as plt" 34 | ] 35 | }, 36 | { 37 | "cell_type": "markdown", 38 | "metadata": {}, 39 | "source": [ 40 | "We quickly reload the data using the same approach as in the previous exercise" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "metadata": { 47 | "collapsed": true 48 | }, 49 | "outputs": [], 50 | "source": [ 51 | "# Data descriptor to make a proper array.\n", 52 | "dt = [('station','S4'), ('lat',np.float32), ('lon',np.float32), ('elev',np.float32) ]\n", 53 | "data_fname = os.path.join('data', 'stations.txt')\n", 54 | "tab = np.loadtxt(data_fname, dt).view(np.recarray)" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "We start by defining our `StationPicker` object that will do the job:" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "metadata": { 68 | "collapsed": false 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "class StationPicker(object):\n", 73 | " def __init__(self, figure, stations, eps=0.10, axis=None):\n", 74 | " self.figure = figure\n", 75 | " self.stations = stations\n", 76 | " self.cid = figure.canvas.mpl_connect('button_press_event', self)\n", 77 | " if axis is None:\n", 78 | " axis = figure.axes[0]\n", 79 | " self.axis = axis\n", 80 | " self.eps = eps\n", 81 | "\n", 82 | " def __call__(self, event):\n", 83 | " #print 'click', event # dbg\n", 84 | " if event.inaxes != self.axis:\n", 85 | " return\n", 86 | " self.figure.canvas.draw()\n", 87 | " # Compute the distance from the click to all stations\n", 88 | " lats = self.stations['lat']\n", 89 | " longs = self.stations['lon']\n", 90 | " click_lat, click_long = event.xdata, event.ydata\n", 91 | " lat_d = lats - click_lat\n", 92 | " lon_d = longs - click_long\n", 93 | " dist = np.sqrt(lat_d**2 + lon_d**2)\n", 94 | " nearest_i = dist.argmin()\n", 95 | " near_dist = dist[nearest_i]\n", 96 | " nearest = self.stations[nearest_i]\n", 97 | " #print 'Nearest distance:', near_dist # dbg\n", 98 | " if near_dist < self.eps:\n", 99 | " print (\"HIT! You clicked on\", nearest['station'])\n", 100 | " else:\n", 101 | " print (\"No hit, nearest is:\", nearest['station'])\n", 102 | " print(\"It is at:\", nearest['lat'], nearest['lon'])\n", 103 | " print (\"Distance to it:\", near_dist)\n", 104 | " sys.stdout.flush()" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": { 111 | "collapsed": false 112 | }, 113 | "outputs": [], 114 | "source": [ 115 | "fig, ax = plt.subplots()\n", 116 | "ax.scatter(tab['lat'], tab['lon'], 40*(tab['elev']+1), c=tab['elev'] )\n", 117 | "# We can now make a picker with that binds the figure and the data\n", 118 | "StationPicker(fig, tab)" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": null, 124 | "metadata": { 125 | "collapsed": true 126 | }, 127 | "outputs": [], 128 | "source": [] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": null, 133 | "metadata": { 134 | "collapsed": true 135 | }, 136 | "outputs": [], 137 | "source": [] 138 | } 139 | ], 140 | "metadata": { 141 | "kernelspec": { 142 | "display_name": "Python 3", 143 | "language": "python", 144 | "name": "python3" 145 | }, 146 | "language_info": { 147 | "codemirror_mode": { 148 | "name": "ipython", 149 | "version": 3 150 | }, 151 | "file_extension": ".py", 152 | "mimetype": "text/x-python", 153 | "name": "python", 154 | "nbconvert_exporter": "python", 155 | "pygments_lexer": "ipython3", 156 | "version": "3.4.3" 157 | } 158 | }, 159 | "nbformat": 4, 160 | "nbformat_minor": 0 161 | } 162 | -------------------------------------------------------------------------------- /Part-1/images/FrontendKernel.graffle/image1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/images/FrontendKernel.graffle/image1.png -------------------------------------------------------------------------------- /Part-1/images/FrontendKernel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/images/FrontendKernel.png -------------------------------------------------------------------------------- /Part-1/images/animation.m4v: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/images/animation.m4v -------------------------------------------------------------------------------- /Part-1/images/ipython_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-1/images/ipython_logo.png -------------------------------------------------------------------------------- /Part-2/Index.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# Day 1 (Afternoon): Working with notebook files\n", 15 | "\n", 16 | "Jupyter notebooks are themselves files that can be manipulated programatically and can be converted to a variety of formats with customizations.\n", 17 | "\n", 18 | "This section is a whistle-stop tour of four different tools to help you work with and share notebook files: **nbconvert**, **nbviewer**, **nbdime** and **nbval**.\n", 19 | "\n", 20 | "## Notebook files and Nbconvert\n", 21 | "\n", 22 | "* [Notebook file format](Notebook file format.ipynb)\n", 23 | "* [Converting notebooks to other formats](Using Nbconvert.ipynb)\n", 24 | "* [Nbviewer: Sharing notebooks on the web](Nbviewer.ipynb)\n", 25 | "\n", 26 | "## Nbdime\n", 27 | "\n", 28 | "* [Notebooks in version control](nbdime.ipynb)\n", 29 | "\n", 30 | "## Nbval\n", 31 | "\n", 32 | "* [Testing notebooks](nbval.ipynb)\n", 33 | "\n", 34 | "## Material for extra reading\n", 35 | "\n", 36 | "* [Nbconvert and templates](nbconvert_templates/Nbconvert templates.ipynb)\n", 37 | "\n" 38 | ] 39 | } 40 | ], 41 | "metadata": { 42 | "kernelspec": { 43 | "display_name": "Python 3", 44 | "language": "python", 45 | "name": "python3" 46 | }, 47 | "language_info": { 48 | "codemirror_mode": { 49 | "name": "ipython", 50 | "version": 3 51 | }, 52 | "file_extension": ".py", 53 | "mimetype": "text/x-python", 54 | "name": "python", 55 | "nbconvert_exporter": "python", 56 | "pygments_lexer": "ipython3", 57 | "version": "3.6.0" 58 | }, 59 | "widgets": { 60 | "state": {}, 61 | "version": "1.1.1" 62 | } 63 | }, 64 | "nbformat": 4, 65 | "nbformat_minor": 1 66 | } 67 | -------------------------------------------------------------------------------- /Part-2/Nbviewer.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Nbviewer\n", 8 | "\n", 9 | "
\n", 10 | "https://nbviewer.jupyter.org/\n", 11 | "
\n", 12 | "\n", 13 | "Nbviewer is a service to share notebooks on the web. You upload the `.ipynb` notebook file somewhere and put the URL into nbviewer; it gives you a link with an HTML view of the notebook.\n", 14 | "\n", 15 | "For example:\n", 16 | "\n", 17 | "* Notebook file: http://norvig.com/ipython/Countdown.ipynb\n", 18 | "* Nbviewer link: http://nbviewer.jupyter.org/url/norvig.com/ipython/Countdown.ipynb" 19 | ] 20 | }, 21 | { 22 | "cell_type": "markdown", 23 | "metadata": {}, 24 | "source": [ 25 | "Sharing notebooks like this has a key advantage over converting notebooks to HTML yourself:\n", 26 | "\n", 27 | "![nbviewer header showing download button](images/nbviewer_download_link.png)\n", 28 | "\n", 29 | "Nbviewer automatically links back to the notebook file. So if someone is reading your notebook on nbviewer and finds it interesting, they can download the notebook file, run your code, and experiment with their own changes.\n", 30 | "\n", 31 | "This is good for *reproducible research*: you're not just telling people about your computational methods, you're letting them run those methods." 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "## Nbviewer and GitHub\n", 39 | "\n", 40 | "GitHub is one of the most popular places to host notebooks online. This tutorial is hosted on GitHub, for example.\n", 41 | "\n", 42 | "GitHub integrates nbviewer, so when you look at a notebook on `github.com`, you see the HTML view instead of the raw file. For instance, here's this notebook in the GitHub interface:\n", 43 | "\n", 44 | "\n", 49 | "\n", 50 | "Some content doesn't work in the GitHub view for security reasons. If you want to see this, you can still pass GitHub URLs into nbviewer." 51 | ] 52 | } 53 | ], 54 | "metadata": { 55 | "kernelspec": { 56 | "display_name": "Python 3", 57 | "language": "python", 58 | "name": "python3" 59 | }, 60 | "language_info": { 61 | "codemirror_mode": { 62 | "name": "ipython", 63 | "version": 3 64 | }, 65 | "file_extension": ".py", 66 | "mimetype": "text/x-python", 67 | "name": "python", 68 | "nbconvert_exporter": "python", 69 | "pygments_lexer": "ipython3", 70 | "version": "3.5.2" 71 | } 72 | }, 73 | "nbformat": 4, 74 | "nbformat_minor": 2 75 | } 76 | -------------------------------------------------------------------------------- /Part-2/Notebook file format.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Notebook file format\n", 8 | "\n", 9 | "Notebooks are stored on disk as JSON files. JSON is a really simple way of representing data: it looks exactly like Python lists and dictionaries, so you already know how to read it.\n", 10 | "\n", 11 | "```javascript\n", 12 | "{\n", 13 | " \"key\": \"value\",\n", 14 | " \"ultimate answer\": 42,\n", 15 | " \"lists\": [\"like\", \"this\"]\n", 16 | "}\n", 17 | "```\n", 18 | "\n", 19 | "At the top level of the notebook file there are four fields:\n", 20 | "\n", 21 | "* `nbformat` & `nbformat_minor`: The version of the format this notebook is stored in. The current version is 4.2.\n", 22 | "* `metadata`: Information about the notebook, like the language it's written in.\n", 23 | "* `cells`: List of cells with the notebook content\n", 24 | "\n", 25 | "```javascript\n", 26 | "{\n", 27 | " \"nbformat\": 4,\n", 28 | " \"nbformat_minor\": 2,\n", 29 | " \"metadata\": {\n", 30 | " \"language_info\": {\n", 31 | " \"name\": \"python\"\n", 32 | " ...\n", 33 | " }\n", 34 | " },\n", 35 | " \"cells\": [...]\n", 36 | "}\n", 37 | "``` " 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "## Challenges\n", 45 | "\n", 46 | "Open the notebooks in this directory in your text editor and look at the structure.\n", 47 | "\n", 48 | "1. What distinguishes a markdown cell from a code cell?\n", 49 | "2. How many different kinds of output can you see?\n", 50 | "\n", 51 | "The [notebook format documentation](http://ipython.org/ipython-doc/3/notebook/nbformat.html) has the answers. It describes the structure of notebook files in detail." 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "metadata": {}, 57 | "source": [ 58 | "## Manipulating notebooks in Python code\n", 59 | "\n", 60 | "The `IPython.nbformat` package has functions to load and save notebooks, convert between different versions of the format, and validate notebooks against the specification." 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 1, 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "from IPython import nbformat\n", 70 | "nb = nbformat.read('Notebook file format.ipynb', as_version=4)" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 2, 76 | "metadata": {}, 77 | "outputs": [ 78 | { 79 | "name": "stdout", 80 | "output_type": "stream", 81 | "text": [ 82 | "## Manipulating notebooks in Python code\n", 83 | "\n", 84 | "The `IPython.nbformat` package has functions to load and save notebooks, convert between different versions of the format, and validate notebooks against the specification.\n" 85 | ] 86 | } 87 | ], 88 | "source": [ 89 | "print(nb.cells[2].source)" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": null, 95 | "metadata": { 96 | "collapsed": true 97 | }, 98 | "outputs": [], 99 | "source": [ 100 | "# Run this, then reload the page to see the change\n", 101 | "nb.cells.append(nbformat.v4.new_markdown_cell('**Look at me!**'))\n", 102 | "nbformat.write(nb, 'Notebook file format.ipynb')" 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": {}, 108 | "source": [ 109 | "Next: [nbconvert](Nbconvert%20templates.ipynb)" 110 | ] 111 | } 112 | ], 113 | "metadata": { 114 | "kernelspec": { 115 | "display_name": "Python 3", 116 | "language": "python", 117 | "name": "python3" 118 | }, 119 | "language_info": { 120 | "codemirror_mode": { 121 | "name": "ipython", 122 | "version": 3 123 | }, 124 | "file_extension": ".py", 125 | "mimetype": "text/x-python", 126 | "name": "python", 127 | "nbconvert_exporter": "python", 128 | "pygments_lexer": "ipython3", 129 | "version": "3.5.2" 130 | } 131 | }, 132 | "nbformat": 4, 133 | "nbformat_minor": 1 134 | } 135 | -------------------------------------------------------------------------------- /Part-2/images/jupyter_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-2/images/jupyter_logo.png -------------------------------------------------------------------------------- /Part-2/images/nbconvert_file_menu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-2/images/nbconvert_file_menu.png -------------------------------------------------------------------------------- /Part-2/images/nbval_cell_tag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-2/images/nbval_cell_tag.png -------------------------------------------------------------------------------- /Part-2/images/nbviewer_download_link.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-2/images/nbviewer_download_link.png -------------------------------------------------------------------------------- /Part-2/nbconvert_templates/README.md: -------------------------------------------------------------------------------- 1 | # Custom templates for nbconvert 2 | 3 | This directory contains three nbconvert templates to hide code cells: 4 | 5 | - `hidecode.tplx` : Latex, hide all code cells 6 | - `hidecode_selective.tplx` : Latex, hide only code cells indicated by metadata 7 | - `foldcode.tpl` : HTML, hide code cells indicated by metadata, and add buttons to 8 | show and hide each code cell individually. 9 | 10 | For the latter two, you can mark code cells that should be hidden by adding this 11 | JSON to their metadata (select the 'Edit Metadata' cell toolbar): 12 | 13 | ```javascript 14 | "nbconvert": { 15 | "hide_code": true 16 | } 17 | ``` 18 | 19 | To use the templates at the command line: 20 | 21 | ```shell 22 | ipython nbconvert --to latex --template hidecode.tplx MyNotebook.ipynb 23 | 24 | ipython nbconvert --to latex --template hidecode_selective.tplx MyNotebook.ipynb 25 | 26 | ipython nbconvert --to html --template foldcode.tpl MyNotebook.ipynb 27 | ``` 28 | -------------------------------------------------------------------------------- /Part-2/nbconvert_templates/foldcode.tpl: -------------------------------------------------------------------------------- 1 | {%- extends 'full.tpl' -%} 2 | 3 | {# HTML template that selectively hides code, with show/hide buttons. #} 4 | 5 | {%- block header -%} 6 | 7 | 8 | 9 | 10 | 11 | {{resources['metadata']['name']}} 12 | 13 | 14 | 15 | 16 | {% for css in resources.inlining.css -%} 17 | 20 | {% endfor %} 21 | 22 | 58 | 59 | 64 | 65 | 66 | 67 | 68 | 69 | {{ mathjax() }} 70 | 71 | 72 | {%- endblock header -%} 73 | 74 | {# There must be a better way of giving each code input an HTML ID. 75 | This is rather a hack using the primitives Jinja gives us. 76 | #} 77 | {% set code_cell_numbers = cycler(*range(nb.cells | count)) %} 78 | 79 | {% block in_prompt -%} 80 | {% set code_cell_number = (code_cell_numbers).__next__() %} 81 |
82 | {# I don't think the +1 should be needed here, but it is. #} 83 | ±± 85 |
86 | {%- endblock in_prompt %} 87 | 88 | {% block input %} 89 | {% set code_cell_number = (code_cell_numbers).current %} 90 |
91 |
93 | {{ cell.source | highlight_code(metadata=cell.metadata) }} 94 |
95 |
96 | 97 | {% endblock input %} 98 | -------------------------------------------------------------------------------- /Part-2/nbconvert_templates/makeitpop.tpl: -------------------------------------------------------------------------------- 1 | {# This is a comment #} 2 | 3 | {# First we need to say which template we're extending. 4 | full.tpl is the default template for HTML output. #} 5 | {%- extends 'full.tpl' -%} 6 | 7 | {# Now to override some blocks. #} 8 | 9 | {%- block markdowncell -%} 10 |
11 | {# super() means 'put here whatever the parent template does for this block' #} 12 | {{ super() }} 13 |
14 | {%- endblock markdowncell -%} 15 | -------------------------------------------------------------------------------- /Part-2/nbconvert_templates/nbconvert_template_structure.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | Nbconvert template structure 7 | 8 | 27 | 28 | 29 | 30 | 31 |

Nbconvert template structure

32 | 33 | Nbconvert templates consist of a set of nested blocks. When defining a new 34 | template, you extend an existing template by overriding some of the blocks. 35 | 36 | All the templates shipped in nbconvert have the basic structure described here, 37 | though some may define additional blocks. 38 | 39 |

Main page

40 |
header
41 | 42 |
body 43 |
any_cell 44 |
codecell 45 |
input_group 46 |
in_prompt
47 |
input
48 |
49 |
output_group 50 |
output_prompt
51 |
outputs (see below)
52 |
53 |
54 |
55 |
any_cell 56 |
markdowncell
57 |
58 |
any_cell 59 |
rawcell
60 |
61 |
any_cell 62 |
unknowncell
63 |
64 |
⋮
65 |
66 | 67 |
footer
68 | 69 |

Outputs

70 | 71 |
outputs 72 |
output 73 |
execute_result
74 |
75 |
output 76 |
stream_stdout
77 |
78 |
output 79 |
stream_stderr
80 |
81 |
output 82 |
display_data 83 |
data_priority 84 |
data_pdf / data_svg / data_png / 85 | data_html / data_markdown / data_jpg / data_text / 86 | data_latex / data_javascript / data_other 87 |
88 |
89 |
90 |
91 |
output 92 |
error 93 |
traceback_line
94 |
⋮
95 |
96 |
97 |
⋮
98 |
99 | 100 | 101 | 102 | 103 | 104 | 105 | -------------------------------------------------------------------------------- /Part-2/nbconvert_templates/solutions/hidecode.tplx: -------------------------------------------------------------------------------- 1 | ((*- extends 'article.tplx' -*)) 2 | 3 | ((* block input_group *)) 4 | ((* endblock input_group *)) 5 | -------------------------------------------------------------------------------- /Part-2/nbconvert_templates/solutions/hidecode_selective.tplx: -------------------------------------------------------------------------------- 1 | ((*- extends 'article.tplx' -*)) 2 | 3 | ((* block input_group *)) 4 | ((*- if not cell.metadata.get('nbconvert', {}).get('hide_code', False) -*)) 5 | ((( super() ))) 6 | ((*- endif -*)) 7 | ((* endblock input_group *)) 8 | -------------------------------------------------------------------------------- /Part-2/nbdime.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# nbdime - Notebook Diffing and Merging\n", 8 | "\n", 9 | "nbdime is a Python package for comparing the differences between two Notebooks. It can give you informative reports about changes at the command-line, and can also show a side-by-side comparison in the browser. When you have Notebooks in version control (e.g. git), it can also help you deal with resolving conflicting changes:\n", 10 | " - It uses the knowledge of the notebook format to make informed decisions about which changes are actually conflicting.\n", 11 | " - It ensures that conflicts can be presented in a workable Notebook (naive conflict resolution can often make the Notebook file invalid).\n", 12 | " \n", 13 | "This intro should help get you set up with nbdime, and how and when to use it." 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": {}, 19 | "source": [ 20 | "## Installation\n", 21 | "\n", 22 | "The preferred way of installing nbdime is with `pip`:" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "metadata": { 29 | "tags": [ 30 | "nbval-skip" 31 | ] 32 | }, 33 | "outputs": [], 34 | "source": [ 35 | "!pip install nbdime" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "## Usage\n", 43 | "\n", 44 | "After installing nbdime, the following commands will be available from your command-line:\n", 45 | " - `nbshow`: This will print a structured view of your Notebook to the terminal. It also allows you to filter on source / outputs for a tidier presentation.\n", 46 | " - `nbdiff `: This will compare two Notebooks, and print a structured report about the changes to the terminal.\n", 47 | " - `nbdiff-web `: This will show you a side-by-side comparison of two Notebooks in the web-browser:" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "A notebook is a JSON document that's not the easiest to read raw:" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "with open('nbdime-before.ipynb') as f:\n", 64 | " for line in f.readlines()[:64]:\n", 65 | " line = line.rstrip()\n", 66 | " if len(line) > 80:\n", 67 | " line = line[:64] + '...(%i chars)' % len(line)\n", 68 | " print(line)" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "`nbshow` gives you a nice readable format, and can focus on particular information,\n", 76 | "such as showing only source code with `-s`:" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "metadata": {}, 83 | "outputs": [], 84 | "source": [ 85 | "!nbshow nbdime-after.ipynb -s" 86 | ] 87 | }, 88 | { 89 | "cell_type": "markdown", 90 | "metadata": {}, 91 | "source": [ 92 | "nbdime also provides diffing commands. `nbdiff` shows you a diff in the terminal, similarly formatted to `nbshow`:" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": null, 98 | "metadata": {}, 99 | "outputs": [], 100 | "source": [ 101 | "!nbdiff nbdime-before.ipynb nbdime-after.ipynb" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "And `nbdiff-web` opens a browser to show a rendered diff, including side-by-side images:" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "metadata": {}, 115 | "outputs": [], 116 | "source": [ 117 | "!nbdiff-web nbdime-before.ipynb nbdime-after.ipynb" 118 | ] 119 | }, 120 | { 121 | "cell_type": "markdown", 122 | "metadata": {}, 123 | "source": [ 124 | "## Git integration\n", 125 | "\n", 126 | "If you want nbdime to integrate with git, you will have to tell git about how to use nbdime. You can configure this manually, but for convenience nbdime includes some commands to register itself in the most common configuration. Starting out, you simply need to run:" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": { 133 | "collapsed": true 134 | }, 135 | "outputs": [], 136 | "source": [ 137 | "!nbdime config-git --enable --global" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "With the default configuration, nbdime will automatically be used for `.ipynb` files when:\n", 145 | " - you call `git diff` on the command-line\n", 146 | " - git is trying to resolve conflicting changes (*merge conflicts*)\n", 147 | " \n", 148 | "It is worth noting that `nbdiff` and `nbdiff-web` can also be used to show you the changes to Notebooks in a git repository. Simply calling `nbdiff-web` without any arguments in your repository will show you the diff for all the Notebooks that have changed:" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": null, 154 | "metadata": { 155 | "collapsed": true 156 | }, 157 | "outputs": [], 158 | "source": [ 159 | "!nbdiff-web origin/master nbdime.ipynb" 160 | ] 161 | } 162 | ], 163 | "metadata": { 164 | "kernelspec": { 165 | "display_name": "Python 3", 166 | "language": "python", 167 | "name": "python3" 168 | }, 169 | "language_info": { 170 | "codemirror_mode": { 171 | "name": "ipython", 172 | "version": 3 173 | }, 174 | "file_extension": ".py", 175 | "mimetype": "text/x-python", 176 | "name": "python", 177 | "nbconvert_exporter": "python", 178 | "pygments_lexer": "ipython3", 179 | "version": "3.6.0" 180 | } 181 | }, 182 | "nbformat": 4, 183 | "nbformat_minor": 2 184 | } 185 | -------------------------------------------------------------------------------- /Part-3/Beat Frequencies.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Exploring Beat Frequencies using the `Audio` Object" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "This example uses the `Audio` object and Matplotlib to explore the phenomenon of beat frequencies." 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": { 21 | "collapsed": false 22 | }, 23 | "outputs": [], 24 | "source": [ 25 | "%matplotlib inline\n", 26 | "import matplotlib.pyplot as plt\n", 27 | "import numpy as np" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": null, 33 | "metadata": { 34 | "collapsed": false 35 | }, 36 | "outputs": [], 37 | "source": [ 38 | "from ipywidgets import interactive\n", 39 | "from IPython.display import Audio, display\n", 40 | "import numpy as np" 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "metadata": { 47 | "collapsed": false 48 | }, 49 | "outputs": [], 50 | "source": [ 51 | "def beat_freq(f1=220.0, f2=224.0):\n", 52 | " max_time = 3\n", 53 | " rate = 8000\n", 54 | " times = np.linspace(0,max_time,rate*max_time)\n", 55 | " signal = np.sin(2*np.pi*f1*times) + np.sin(2*np.pi*f2*times)\n", 56 | " print(f1, f2, abs(f1-f2))\n", 57 | " display(Audio(data=signal, rate=rate, autoplay=True))\n", 58 | " return signal" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": null, 64 | "metadata": { 65 | "collapsed": false 66 | }, 67 | "outputs": [], 68 | "source": [ 69 | "v = interactive(beat_freq, f1=(200.0,300.0), f2=(200.0,300.0))\n", 70 | "display(v)" 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": null, 76 | "metadata": { 77 | "collapsed": false 78 | }, 79 | "outputs": [], 80 | "source": [ 81 | "v.kwargs" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": { 88 | "collapsed": false 89 | }, 90 | "outputs": [], 91 | "source": [ 92 | "f1, f2 = v.children\n", 93 | "f1.value = 255\n", 94 | "f2.value = 260\n", 95 | "plt.plot(v.result[0:6000]);" 96 | ] 97 | } 98 | ], 99 | "metadata": { 100 | "kernelspec": { 101 | "display_name": "Python 3", 102 | "language": "python", 103 | "name": "python3" 104 | }, 105 | "language_info": { 106 | "codemirror_mode": { 107 | "name": "ipython", 108 | "version": 3 109 | }, 110 | "file_extension": ".py", 111 | "mimetype": "text/x-python", 112 | "name": "python", 113 | "nbconvert_exporter": "python", 114 | "pygments_lexer": "ipython3", 115 | "version": "3.5.1" 116 | }, 117 | "widgets": { 118 | "state": {}, 119 | "version": "1.1.1" 120 | } 121 | }, 122 | "nbformat": 4, 123 | "nbformat_minor": 0 124 | } 125 | -------------------------------------------------------------------------------- /Part-3/Exploring Graphs.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Explore Random Graphs Using NetworkX" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "In this example, we build a simple UI for exploring random graphs with [NetworkX](http://networkx.github.io/)." 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": { 21 | "collapsed": false 22 | }, 23 | "outputs": [], 24 | "source": [ 25 | "from ipywidgets import interact" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": null, 31 | "metadata": { 32 | "collapsed": false 33 | }, 34 | "outputs": [], 35 | "source": [ 36 | "%matplotlib inline\n", 37 | "import matplotlib.pyplot as plt" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": null, 43 | "metadata": { 44 | "collapsed": false 45 | }, 46 | "outputs": [], 47 | "source": [ 48 | "import networkx as nx" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": null, 54 | "metadata": { 55 | "collapsed": false 56 | }, 57 | "outputs": [], 58 | "source": [ 59 | "# wrap a few graph generation functions so they have the same signature\n", 60 | "\n", 61 | "def random_lobster(n, m, k, p):\n", 62 | " return nx.random_lobster(n, p, p / m)\n", 63 | "\n", 64 | "def powerlaw_cluster(n, m, k, p):\n", 65 | " return nx.powerlaw_cluster_graph(n, m, p)\n", 66 | "\n", 67 | "def erdos_renyi(n, m, k, p):\n", 68 | " return nx.erdos_renyi_graph(n, p)\n", 69 | "\n", 70 | "def newman_watts_strogatz(n, m, k, p):\n", 71 | " return nx.newman_watts_strogatz_graph(n, k, p)\n", 72 | "\n", 73 | "def plot_random_graph(n, m, k, p, generator):\n", 74 | " g = generator(n, m, k, p)\n", 75 | " nx.draw(g)\n", 76 | " plt.show()" 77 | ] 78 | }, 79 | { 80 | "cell_type": "code", 81 | "execution_count": null, 82 | "metadata": { 83 | "collapsed": false 84 | }, 85 | "outputs": [], 86 | "source": [ 87 | "interact(plot_random_graph, n=(2,30), m=(1,10), k=(1,10), p=(0.0, 1.0, 0.001),\n", 88 | " generator={'lobster': random_lobster,\n", 89 | " 'power law': powerlaw_cluster,\n", 90 | " 'Newman-Watts-Strogatz': newman_watts_strogatz,\n", 91 | " u'Erdős-Rényi': erdos_renyi,\n", 92 | " });" 93 | ] 94 | } 95 | ], 96 | "metadata": { 97 | "kernelspec": { 98 | "display_name": "Python 3", 99 | "language": "python", 100 | "name": "python3" 101 | }, 102 | "language_info": { 103 | "codemirror_mode": { 104 | "name": "ipython", 105 | "version": 3 106 | }, 107 | "file_extension": ".py", 108 | "mimetype": "text/x-python", 109 | "name": "python", 110 | "nbconvert_exporter": "python", 111 | "pygments_lexer": "ipython3", 112 | "version": "3.4.0" 113 | } 114 | }, 115 | "nbformat": 4, 116 | "nbformat_minor": 0 117 | } 118 | -------------------------------------------------------------------------------- /Part-3/Export As (nbconvert).ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": false 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "# Widget related imports\n", 12 | "import ipywidgets as widgets\n", 13 | "from IPython.display import display, clear_output, Javascript\n", 14 | "from traitlets import Unicode\n", 15 | "\n", 16 | "# nbconvert related imports\n", 17 | "from IPython.nbconvert import get_export_names, export_by_name\n", 18 | "from IPython.nbconvert.writers import FilesWriter\n", 19 | "from IPython.nbformat import read, NO_CONVERT\n", 20 | "from IPython.nbconvert.utils.exceptions import ConversionException" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "This notebook shows a really roundabout way to get the name of the notebook file using widgets. The true purpose of this demo is to demonstrate how Javascript and Python widget models are related by `id`." 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "Create a text Widget without displaying it. The widget will be used to store the notebook's name which is otherwise only available in the front-end." 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": { 41 | "collapsed": false 42 | }, 43 | "outputs": [], 44 | "source": [ 45 | "notebook_name = widgets.Text()" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "Get the current notebook's name by pushing JavaScript to the browser that sets the notebook name in a string widget." 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": { 59 | "collapsed": false 60 | }, 61 | "outputs": [], 62 | "source": [ 63 | "js = \"\"\"IPython.notebook.kernel.widget_manager.get_model('%s').then(function(model) {\n", 64 | " model.set('value', IPython.notebook.notebook_name);\n", 65 | " model.save();\n", 66 | "});\n", 67 | "\"\"\" % notebook_name.model_id\n", 68 | "display(Javascript(data=js))" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": { 75 | "collapsed": false 76 | }, 77 | "outputs": [], 78 | "source": [ 79 | "filename = notebook_name.value\n", 80 | "filename" 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "Create the widget that will allow the user to Export the current notebook." 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": { 94 | "collapsed": false 95 | }, 96 | "outputs": [], 97 | "source": [ 98 | "exporter_names = widgets.Dropdown(options=get_export_names(), value='html')\n", 99 | "export_button = widgets.Button(description=\"Export\")\n", 100 | "download_link = widgets.HTML(visible=False)" 101 | ] 102 | }, 103 | { 104 | "cell_type": "markdown", 105 | "metadata": {}, 106 | "source": [ 107 | "Export the notebook when the export button is clicked." 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": null, 113 | "metadata": { 114 | "collapsed": false 115 | }, 116 | "outputs": [], 117 | "source": [ 118 | "file_writer = FilesWriter()\n", 119 | "\n", 120 | "def export(name, nb):\n", 121 | " \n", 122 | " # Get a unique key for the notebook and set it in the resources object.\n", 123 | " notebook_name = name[:name.rfind('.')]\n", 124 | " resources = {}\n", 125 | " resources['unique_key'] = notebook_name\n", 126 | " resources['output_files_dir'] = '%s_files' % notebook_name\n", 127 | "\n", 128 | " # Try to export\n", 129 | " try:\n", 130 | " output, resources = export_by_name(exporter_names.value, nb)\n", 131 | " except ConversionException as e:\n", 132 | " download_link.value = \"
Could not export notebook!\"\n", 133 | " else:\n", 134 | " write_results = file_writer.write(output, resources, notebook_name=notebook_name)\n", 135 | " \n", 136 | " download_link.value = \"
Results: \\\"{filename}\\\"\".format(filename=write_results)\n", 137 | " download_link.visible = True\n", 138 | " \n", 139 | "def handle_export(widget):\n", 140 | " with open(filename, 'r') as f:\n", 141 | " export(filename, read(f, NO_CONVERT))\n", 142 | " \n", 143 | "export_button.on_click(handle_export)" 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "Display the controls." 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": null, 156 | "metadata": { 157 | "collapsed": false 158 | }, 159 | "outputs": [], 160 | "source": [ 161 | "display(exporter_names, export_button, download_link)" 162 | ] 163 | } 164 | ], 165 | "metadata": { 166 | "kernelspec": { 167 | "display_name": "Python 3", 168 | "language": "python", 169 | "name": "python3" 170 | }, 171 | "language_info": { 172 | "codemirror_mode": { 173 | "name": "ipython", 174 | "version": 3 175 | }, 176 | "file_extension": ".py", 177 | "mimetype": "text/x-python", 178 | "name": "python", 179 | "nbconvert_exporter": "python", 180 | "pygments_lexer": "ipython3", 181 | "version": "3.4.0" 182 | } 183 | }, 184 | "nbformat": 4, 185 | "nbformat_minor": 0 186 | } 187 | -------------------------------------------------------------------------------- /Part-3/Image Browser.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Image Browser" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "This example shows how to browse through a set of images with a slider." 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": { 21 | "collapsed": false 22 | }, 23 | "outputs": [], 24 | "source": [ 25 | "%matplotlib inline\n", 26 | "import matplotlib.pyplot as plt" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": null, 32 | "metadata": { 33 | "collapsed": false 34 | }, 35 | "outputs": [], 36 | "source": [ 37 | "from ipywidgets import interact" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": null, 43 | "metadata": { 44 | "collapsed": false 45 | }, 46 | "outputs": [], 47 | "source": [ 48 | "from sklearn import datasets" 49 | ] 50 | }, 51 | { 52 | "cell_type": "markdown", 53 | "metadata": {}, 54 | "source": [ 55 | "We will use the digits dataset from [scikit-learn](http://scikit-learn.org/stable/)." 56 | ] 57 | }, 58 | { 59 | "cell_type": "code", 60 | "execution_count": null, 61 | "metadata": { 62 | "collapsed": false 63 | }, 64 | "outputs": [], 65 | "source": [ 66 | "digits = datasets.load_digits()" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "metadata": { 73 | "collapsed": false 74 | }, 75 | "outputs": [], 76 | "source": [ 77 | "def browse_images(digits):\n", 78 | " n = len(digits.images)\n", 79 | " def view_image(i):\n", 80 | " plt.imshow(digits.images[i], cmap=plt.cm.gray_r, interpolation='nearest')\n", 81 | " plt.title('Training: %s' % digits.target[i])\n", 82 | " plt.show()\n", 83 | " interact(view_image, i=(0,n-1))" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": { 90 | "collapsed": false 91 | }, 92 | "outputs": [], 93 | "source": [ 94 | "browse_images(digits)" 95 | ] 96 | } 97 | ], 98 | "metadata": { 99 | "kernelspec": { 100 | "display_name": "Python 3", 101 | "language": "python", 102 | "name": "python3" 103 | }, 104 | "language_info": { 105 | "codemirror_mode": { 106 | "name": "ipython", 107 | "version": 3 108 | }, 109 | "file_extension": ".py", 110 | "mimetype": "text/x-python", 111 | "name": "python", 112 | "nbconvert_exporter": "python", 113 | "pygments_lexer": "ipython3", 114 | "version": "3.4.0" 115 | } 116 | }, 117 | "nbformat": 4, 118 | "nbformat_minor": 0 119 | } 120 | -------------------------------------------------------------------------------- /Part-3/Index.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# Day 2 (Morning): Interactive Widgets\n", 15 | "\n", 16 | "IPython includes an architecture for interactive widgets that tie together Python code running in the kernel and JavaScript/HTML/CSS running in the browser. These widgets enable users to explore their code and data interactively.\n", 17 | "\n", 18 | "## Interactive HTML Widgets\n", 19 | "\n", 20 | "- [Using Interact](Using Interact.ipynb)\n", 21 | "- [Widget Basics](Widget Basics.ipynb) \n", 22 | "- [Widget List](Widget List.ipynb)\n", 23 | "- [Widget Events](Widget Events.ipynb) \n", 24 | "\n", 25 | "## Exercises\n", 26 | "\n", 27 | "* [Interact Basics](exercises/InteractBasics.ipynb)\n", 28 | "* [Image Processing](exercises/Image Processing.ipynb)\n", 29 | "* [Comparing two images](exercises/ImageCompare.ipynb)\n", 30 | "* [Basics of Widgets](exercises/WidgetBasics.ipynb)\n", 31 | "* [Interacting with Matplotlib](exercises/Interact%20with%20matplotlib%20figures.ipynb)\n", 32 | "\n", 33 | "---------\n", 34 | "\n", 35 | "# Additional materials for further reading\n", 36 | "\n", 37 | "## More Examples using `interact`/`interactive`\n", 38 | "\n", 39 | "* [Beat Frequencies](Beat Frequencies.ipynb)\n", 40 | "* [Exploring Graphs](Exploring Graphs.ipynb)\n", 41 | "* [Image Browser](Image Browser.ipynb)\n", 42 | "* [Lorenz Differential Equations](Lorenz Differential Equations.ipynb)\n", 43 | "\n", 44 | "## Examples of custom widgets\n", 45 | "\n", 46 | "- [Widget Styling](Widget Styling.ipynb)\n", 47 | "- [Custom Widget](Custom Widget - Hello World.ipynb)\n", 48 | "- [Variable Inspector](Variable Inspector.ipynb) \n", 49 | "- [Export As (nbconvert)](Export As (nbconvert%29.ipynb) \n", 50 | "- [Nonblocking Console](Nonblocking Console.ipynb) \n", 51 | "- [File Upload Widget](File Upload Widget.ipynb)" 52 | ] 53 | } 54 | ], 55 | "metadata": { 56 | "kernelspec": { 57 | "display_name": "Python 3", 58 | "language": "python", 59 | "name": "python3" 60 | }, 61 | "language_info": { 62 | "codemirror_mode": { 63 | "name": "ipython", 64 | "version": 3 65 | }, 66 | "file_extension": ".py", 67 | "mimetype": "text/x-python", 68 | "name": "python", 69 | "nbconvert_exporter": "python", 70 | "pygments_lexer": "ipython3", 71 | "version": "3.5.2" 72 | }, 73 | "widgets": { 74 | "state": {}, 75 | "version": "1.1.1" 76 | } 77 | }, 78 | "nbformat": 4, 79 | "nbformat_minor": 1 80 | } 81 | -------------------------------------------------------------------------------- /Part-3/exercises/Image Processing.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Image Manipulation with skimage" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "In this exercise, we build a simple UI for performing basic image manipulation with [scikit-image](http://scikit-image.org/)." 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": {}, 21 | "outputs": [], 22 | "source": [ 23 | "from ipywidgets import interact, interactive, fixed\n", 24 | "from IPython.display import display\n", 25 | "\n", 26 | "import skimage\n", 27 | "from skimage import data, filters, io" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "Scikit-image provides some pre-loaded example data, let's load the nice espresso cup:" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": {}, 41 | "outputs": [], 42 | "source": [ 43 | "%matplotlib inline\n", 44 | "i = data.coffee()\n", 45 | "io.imshow(i)" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "The image `i`, which we see as a colored coffee cup, is to the computer an array:" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "i.shape" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "The first two dimensions are the XxY spatial dimensions of the image, and the third dimension indicates that there's actually three \"layers\", or \"channels\". In fact, while we see it in color, that's because we interpret it as such when rendering it on a monitor; in the computer, what we have are three 400x600 arrays of numbers that represent color intensities in each of these channels, one for red, one for green, one for blue.\n", 69 | "\n", 70 | "A very rudimentary \"image editor\" can be constructed by modifying the amount of each of these channels that the final image is composed of. We can multiply the orignal value at each pixel by a multiplier, altering the color balance of the original image.\n", 71 | "\n", 72 | "In addition, we'll toss in a blurring effect, courtesy of the fact that scikit-image provides out of the box a filter that convolves our image with a Gaussian of a specified width, effectively blurring the original image.\n", 73 | "\n", 74 | "Putting together these operations (multiplying each color channel by a certain amount and convolving the image with a Gaussian), we have our little image editor in the following function:" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": null, 80 | "metadata": {}, 81 | "outputs": [], 82 | "source": [ 83 | "def edit_image(image, sigma=0.1, r=1.0, g=1.0, b=1.0):\n", 84 | " new_image = filters.gaussian_filter(image, sigma=sigma, multichannel=True)\n", 85 | " new_image[:,:,0] = r*new_image[:,:,0]\n", 86 | " new_image[:,:,1] = g*new_image[:,:,1]\n", 87 | " new_image[:,:,2] = b*new_image[:,:,2]\n", 88 | " io.imshow(new_image)" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "metadata": {}, 94 | "source": [ 95 | "We can for example blur our image with a gaussian of radius 5 pixels and knock out the red channel by 50%:" 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": null, 101 | "metadata": {}, 102 | "outputs": [], 103 | "source": [ 104 | "edit_image(i, sigma=5, r=0.5)" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": {}, 110 | "source": [ 111 | "# Exercise\n", 112 | "\n", 113 | "Using IPython's `interact`, provide a slider for `sigma` that ranges from 0.1 to 10, as well as sliders for the red, green and blue channels that range from 0 to 1." 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": null, 119 | "metadata": {}, 120 | "outputs": [], 121 | "source": [ 122 | "# Your code here" 123 | ] 124 | }, 125 | { 126 | "cell_type": "markdown", 127 | "metadata": {}, 128 | "source": [ 129 | "## Extra credit\n", 130 | "\n", 131 | "How would you have to modify the function to accept R, G, B sliders that could go beyond one?" 132 | ] 133 | } 134 | ], 135 | "metadata": { 136 | "kernelspec": { 137 | "display_name": "Python 3", 138 | "language": "python", 139 | "name": "python3" 140 | }, 141 | "language_info": { 142 | "codemirror_mode": { 143 | "name": "ipython", 144 | "version": 3 145 | }, 146 | "file_extension": ".py", 147 | "mimetype": "text/x-python", 148 | "name": "python", 149 | "nbconvert_exporter": "python", 150 | "pygments_lexer": "ipython3", 151 | "version": "3.5.2" 152 | } 153 | }, 154 | "nbformat": 4, 155 | "nbformat_minor": 1 156 | } 157 | -------------------------------------------------------------------------------- /Part-3/exercises/InteractBasics.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "nbgrader": {} 7 | }, 8 | "source": [ 9 | "# Interact Basics" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": { 15 | "nbgrader": {} 16 | }, 17 | "source": [ 18 | "## Import" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "nbgrader": {} 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "from ipywidgets import interact, interactive, fixed\n", 30 | "from IPython.display import display" 31 | ] 32 | }, 33 | { 34 | "cell_type": "markdown", 35 | "metadata": { 36 | "nbgrader": {} 37 | }, 38 | "source": [ 39 | "## Adding numbers" 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": { 45 | "nbgrader": {} 46 | }, 47 | "source": [ 48 | "Write a `print_sum` function that `prints` the sum of its arguments `a` and `b`." 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": null, 54 | "metadata": { 55 | "collapsed": true, 56 | "nbgrader": {} 57 | }, 58 | "outputs": [], 59 | "source": [ 60 | "def print_sum(a, b):\n", 61 | " \"\"\"Print the sum of the arguments a and b.\"\"\"" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": { 67 | "nbgrader": {} 68 | }, 69 | "source": [ 70 | "Use the `interact` function to interact with the `print_sum` function.\n", 71 | "\n", 72 | "* `a` should be a floating point slider over the interval `[-10., 10.]` with step sizes of `0.1`\n", 73 | "* `b` should be an integer slider the interval [-8, 8] with step sizes of `2`." 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": null, 79 | "metadata": { 80 | "nbgrader": { 81 | "solution": true 82 | } 83 | }, 84 | "outputs": [], 85 | "source": [ 86 | "# Your code here" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": { 92 | "nbgrader": {} 93 | }, 94 | "source": [ 95 | "## Optional flags\n", 96 | "Write a function named `print_string` that prints a string and additionally prints the length of that string if a boolean parameter is `True`." 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": null, 102 | "metadata": { 103 | "collapsed": true, 104 | "nbgrader": {} 105 | }, 106 | "outputs": [], 107 | "source": [ 108 | "def print_string(s, length=False):\n", 109 | " \"\"\"Print the string s and optionally its length.\"\"\"" 110 | ] 111 | }, 112 | { 113 | "cell_type": "markdown", 114 | "metadata": { 115 | "nbgrader": {} 116 | }, 117 | "source": [ 118 | "Use the `interact` function to interact with the `print_string` function.\n", 119 | "\n", 120 | "* `s` should be a textbox with the initial value `\"Hello World!\"`.\n", 121 | "* `length` should be a checkbox with an initial value of `True`." 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "metadata": { 128 | "nbgrader": { 129 | "solution": true 130 | } 131 | }, 132 | "outputs": [], 133 | "source": [ 134 | "# Your code here" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "# Using SymPy to play with Polynomials" 142 | ] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "metadata": {}, 147 | "source": [ 148 | "We now show that IPython's `interact` capabilities are not limited to basic Python types, with an exercise that uses [SymPy](http://sympy.org/en/index.html) to factor polynomials. \n", 149 | "\n", 150 | "We begin by initializing SymPy to display its results with proper mathematical formatting in the browser:" 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": null, 156 | "metadata": {}, 157 | "outputs": [], 158 | "source": [ 159 | "from sympy import Symbol, Eq, factor, init_printing\n", 160 | "init_printing(use_latex='mathjax')" 161 | ] 162 | }, 163 | { 164 | "cell_type": "markdown", 165 | "metadata": {}, 166 | "source": [ 167 | "Now, we create a symbolic variable, $x$, to build symbolic polynomials with:" 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": null, 173 | "metadata": {}, 174 | "outputs": [], 175 | "source": [ 176 | "x = Symbol('x')" 177 | ] 178 | }, 179 | { 180 | "cell_type": "markdown", 181 | "metadata": {}, 182 | "source": [ 183 | "We can then, for example, factor a polynomial:" 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": null, 189 | "metadata": {}, 190 | "outputs": [], 191 | "source": [ 192 | "factor(x**5-1)" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "## Exercise\n", 200 | "Write a function that takes an argument $n$, and will display both the polynomial $x^n-1$ and its factorization.\n", 201 | "\n", 202 | "Then, use IPython's `interact` to provide a control to visualize how the result varies with a slider control that allows $n$ to vary over the range from 2 to 40.\n", 203 | "\n", 204 | "*Hint:* Look up the help for the sympy function `Eq`." 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": null, 210 | "metadata": {}, 211 | "outputs": [], 212 | "source": [ 213 | "def factorit(n):\n", 214 | "# Your code here" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": null, 220 | "metadata": {}, 221 | "outputs": [], 222 | "source": [ 223 | "# Your code here" 224 | ] 225 | } 226 | ], 227 | "metadata": { 228 | "celltoolbar": "Create Assignment", 229 | "kernelspec": { 230 | "display_name": "Python 3", 231 | "language": "python", 232 | "name": "python3" 233 | }, 234 | "language_info": { 235 | "codemirror_mode": { 236 | "name": "ipython", 237 | "version": 3 238 | }, 239 | "file_extension": ".py", 240 | "mimetype": "text/x-python", 241 | "name": "python", 242 | "nbconvert_exporter": "python", 243 | "pygments_lexer": "ipython3", 244 | "version": "3.5.2" 245 | }, 246 | "name": "_merged" 247 | }, 248 | "nbformat": 4, 249 | "nbformat_minor": 1 250 | } 251 | -------------------------------------------------------------------------------- /Part-3/exercises/University_of_Sheffield_coat_of_arms_new.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-3/exercises/University_of_Sheffield_coat_of_arms_new.png -------------------------------------------------------------------------------- /Part-3/exercises/University_of_Sheffield_coat_of_arms_old.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-3/exercises/University_of_Sheffield_coat_of_arms_old.png -------------------------------------------------------------------------------- /Part-3/exercises/WidgetBasics.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Widget Exercises" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": null, 13 | "metadata": { 14 | "collapsed": true 15 | }, 16 | "outputs": [], 17 | "source": [ 18 | "from ipywidgets import *\n", 19 | "from IPython.display import display\n", 20 | "from traitlets import link" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "### Text Widget" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "Create and display a `Text` widget. Change the Text widget's font color and background color. Set the properties in the constructor of the widget." 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": { 41 | "collapsed": true 42 | }, 43 | "outputs": [], 44 | "source": [] 45 | }, 46 | { 47 | "cell_type": "markdown", 48 | "metadata": {}, 49 | "source": [ 50 | "Do the same, but this time set the properties after the widget is constructed." 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": null, 56 | "metadata": { 57 | "collapsed": true 58 | }, 59 | "outputs": [], 60 | "source": [] 61 | }, 62 | { 63 | "cell_type": "markdown", 64 | "metadata": {}, 65 | "source": [ 66 | "### Link" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "Use a link to link the values of a `Textarea` and an `HTML` widget. Display the widgets and try typing HTML in the textarea. " 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": null, 79 | "metadata": {}, 80 | "outputs": [], 81 | "source": [] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "### Popup" 88 | ] 89 | }, 90 | { 91 | "cell_type": "markdown", 92 | "metadata": {}, 93 | "source": [ 94 | "Display a widget in the `Popup` widget." 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": { 101 | "collapsed": true 102 | }, 103 | "outputs": [], 104 | "source": [] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "## Widget events" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "### on_submit event" 118 | ] 119 | }, 120 | { 121 | "cell_type": "markdown", 122 | "metadata": {}, 123 | "source": [ 124 | "Create and display a `Text`. Use the `on_submit` event to print the value of the textbox just before you clear the textbox. \n", 125 | "\n", 126 | "*Hint: The `on_submit` callback must accept one argument, the `sender`.*" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": {}, 133 | "outputs": [], 134 | "source": [] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "metadata": {}, 139 | "source": [ 140 | "### on_trait_change event" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "metadata": {}, 146 | "source": [ 147 | "Create and display a `Text`. Use the `on_trait_change` method to register a callback that prints the value of the textbox without clearing it. Observe the difference in behavior to Exercise 1." 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": null, 153 | "metadata": {}, 154 | "outputs": [], 155 | "source": [] 156 | }, 157 | { 158 | "cell_type": "markdown", 159 | "metadata": {}, 160 | "source": [ 161 | "## Widget styling" 162 | ] 163 | }, 164 | { 165 | "cell_type": "markdown", 166 | "metadata": {}, 167 | "source": [ 168 | "### Colored text" 169 | ] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "metadata": {}, 174 | "source": [ 175 | "Create and display an `HTML` with a value of your choice (i.e. \"Hello World\"). Change the widget's background color and font color." 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": null, 181 | "metadata": {}, 182 | "outputs": [], 183 | "source": [] 184 | } 185 | ], 186 | "metadata": { 187 | "kernelspec": { 188 | "display_name": "Python 3", 189 | "language": "python", 190 | "name": "python3" 191 | }, 192 | "language_info": { 193 | "codemirror_mode": { 194 | "name": "ipython", 195 | "version": 3 196 | }, 197 | "file_extension": ".py", 198 | "mimetype": "text/x-python", 199 | "name": "python", 200 | "nbconvert_exporter": "python", 201 | "pygments_lexer": "ipython3", 202 | "version": "3.5.2" 203 | } 204 | }, 205 | "nbformat": 4, 206 | "nbformat_minor": 1 207 | } 208 | -------------------------------------------------------------------------------- /Part-3/images/MultilanguageKernels.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-3/images/MultilanguageKernels.png -------------------------------------------------------------------------------- /Part-3/images/ParallelKernels.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-3/images/ParallelKernels.png -------------------------------------------------------------------------------- /Part-3/images/VizInteractCompute.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-3/images/VizInteractCompute.png -------------------------------------------------------------------------------- /Part-3/images/WidgetArch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-3/images/WidgetArch.png -------------------------------------------------------------------------------- /Part-3/images/WidgetModelView.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-3/images/WidgetModelView.png -------------------------------------------------------------------------------- /Part-3/images/ipython_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-3/images/ipython_logo.png -------------------------------------------------------------------------------- /Part-4/Index.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Day 2 (Afternoon): ipyparallel\n", 8 | "\n", 9 | "## Interactive (parallel) Python" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "# Installation and dependencies\n", 17 | "\n", 18 | "You will need ipyparallel >= 5.x, and pyzmq ≥ 13. To use the demo notebooks, you will also need tornado ≥ 4. I will also make use of numpy and matplotlib. If you have Canopy or Anaconda, you already have all of these.\n", 19 | "\n", 20 | "Quick one-line install for IPython and its dependencies:\n", 21 | " \n", 22 | " pip install ipyparallel\n", 23 | " \n", 24 | "Or get everything for the tutorial with conda:\n", 25 | "\n", 26 | " conda install ipyparallel\n", 27 | "\n", 28 | "For those who prefer pip or otherwise manual package installation, the following packages will be used:\n", 29 | "\n", 30 | "ipython\n", 31 | "ipyparallel\n", 32 | "numpy\n", 33 | "matplotlib\n", 34 | "networkx\n", 35 | "scikit-image\n", 36 | "requests\n", 37 | "beautifulsoup\n", 38 | "mpi4py\n", 39 | "\n", 40 | "\n", 41 | "Optional dependencies: I will use [NetworkX](http://networkx.lanl.gov/)\n", 42 | "for one demo, and `scikit-image` for another, but they are not critical. Both packages are in in Anaconda.\n", 43 | "\n", 44 | "For the image-related demos, all you need are some images on your computer. The notebooks will try to fetch images from Wikimedia Commons, but since the networks can be untrustworty, we have [bundled some images here](http://s3.amazonaws.com/ipython-parallel-data/images.zip)." 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "## Outline\n", 52 | "\n", 53 | "- [Motivating Example](examples/Parallel%20image%20processing.ipynb)\n", 54 | "- [Overview](Overview.ipynb)\n", 55 | "- [Tutorial](tutorial)\n", 56 | " - [Remote Execution](tutorial/Remote%20Execution.ipynb)\n", 57 | " - [Multiplexing](tutorial/Multiplexing.ipynb)\n", 58 | " - [Load-Balancing](tutorial/Load-Balancing.ipynb)\n", 59 | " - [Both!](tutorial/All%20Together.ipynb)\n", 60 | " - [Parallel Magics](tutorial/Parallel%20Magics.ipynb)\n", 61 | "- [Examples](examples)\n", 62 | "- [Exercises](exercises)\n" 63 | ] 64 | } 65 | ], 66 | "metadata": { 67 | "kernelspec": { 68 | "display_name": "Python 3", 69 | "language": "python", 70 | "name": "python3" 71 | }, 72 | "language_info": { 73 | "codemirror_mode": { 74 | "name": "ipython", 75 | "version": 3 76 | }, 77 | "file_extension": ".py", 78 | "mimetype": "text/x-python", 79 | "name": "python", 80 | "nbconvert_exporter": "python", 81 | "pygments_lexer": "ipython3", 82 | "version": "3.6.0" 83 | }, 84 | "widgets": { 85 | "state": {}, 86 | "version": "1.1.1" 87 | } 88 | }, 89 | "nbformat": 4, 90 | "nbformat_minor": 1 91 | } 92 | -------------------------------------------------------------------------------- /Part-4/Summary.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "metadata": { 3 | "name": "" 4 | }, 5 | "nbformat": 3, 6 | "nbformat_minor": 0, 7 | "worksheets": [ 8 | { 9 | "cells": [ 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# Summary of what we have covered\n", 15 | "\n", 16 | "* Simple remote execution of functions, or statements\n", 17 | "* Distribution of data\n", 18 | "* Parallel map\n", 19 | "* Multiplexed execution\n", 20 | "* Load-Balanced task farming\n", 21 | "* Asynchronous task submission and result retrieval\n", 22 | "* More efficient data movememnt with memmap and MPI" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": {}, 28 | "source": [ 29 | "# And further things we have *not* covered\n", 30 | "\n", 31 | "* Functional Task Dependencies\n", 32 | "* Configuring the IPython cluster (working with profiles, SGE, MongoDB, etc.)\n", 33 | "* Tuning the IPython Cluster (TaskScheduler.hwm, custom serialization, etc.)\n", 34 | "* Inter-engine communication with PyZMQ or MPI\n", 35 | "* decorators for parallel and remote functions\n", 36 | "* Result caching, and task metadata\n", 37 | "* The TaskResult Database\n", 38 | "* Task resubmission\n", 39 | "* Queue monitoring and management\n", 40 | "\n", 41 | "See the [IPython.parallel docs](http://ipython.org/ipython-doc/dev/parallel) for more detailed coverage of these topics, and check out our [examples](https://www.github.com/ipython/ipython/tree/master/docs/examples/parallel)." 42 | ] 43 | } 44 | ], 45 | "metadata": {} 46 | } 47 | ] 48 | } 49 | -------------------------------------------------------------------------------- /Part-4/download-images.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Downloading images from Wikimedia Commons" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "This notebook defines a few functions to download images with specified search terms from Wikimedia, via the mediawiki search API.\n", 15 | "\n", 16 | "The results are used in the [Parallel face detection notebook](Parallel%20face%20detection.ipynb).\n", 17 | "\n", 18 | "For some caching, you can use requests_cache:\n", 19 | "\n", 20 | " pip install requests_cache" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": null, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "import sys, os\n", 30 | "\n", 31 | "import requests\n", 32 | "try:\n", 33 | " import requests_cache\n", 34 | "except ImportError:\n", 35 | " print(\"no cache, no worries\")\n", 36 | "else:\n", 37 | " requests_cache.install_cache(\"mediawiki\")\n", 38 | "\n", 39 | "api_url = \"http://commons.wikimedia.org/w/api.php\"" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": null, 45 | "metadata": {}, 46 | "outputs": [], 47 | "source": [ 48 | "def api_request(**kwargs):\n", 49 | " \"\"\"Make a request of the Wikimedia Commons API\n", 50 | " \n", 51 | " Returns data after parsing JSON\n", 52 | " \"\"\"\n", 53 | " sys.stdout.write('.')\n", 54 | " sys.stdout.flush()\n", 55 | " \n", 56 | "\n", 57 | " params = dict(\n", 58 | " action='query',\n", 59 | " format='json',\n", 60 | " )\n", 61 | " params.update(kwargs)\n", 62 | " r = requests.get(api_url, params=params)\n", 63 | " r.raise_for_status()\n", 64 | " return r.json()\n", 65 | "\n", 66 | "\n", 67 | "import json\n", 68 | "\n", 69 | "def search_images(search, limit=100, size_limit=2e6):\n", 70 | " \"\"\"search wikimedia commons for a given term\n", 71 | " \n", 72 | " returns a list of `limit` URLs for images\n", 73 | " \"\"\"\n", 74 | " continue_params = {}\n", 75 | " while limit > 0:\n", 76 | " data = api_request(\n", 77 | " srnamespace=6,\n", 78 | " prop='imageinfo',\n", 79 | " list='search',\n", 80 | " srsearch=search,\n", 81 | " srlimit=min(limit, 50),\n", 82 | " **continue_params\n", 83 | " )\n", 84 | "# continue_params = data['query-continue']['search']\n", 85 | " continue_params = data['continue']\n", 86 | "\n", 87 | " total = data['query']['searchinfo']['totalhits']\n", 88 | " results = data['query']['search']\n", 89 | " for r in results:\n", 90 | " title = r['title']\n", 91 | " \n", 92 | " data = api_request(\n", 93 | " prop='imageinfo',\n", 94 | " titles=title,\n", 95 | " iiprop='url|size|mime')\n", 96 | " imageinfo = list(data['query']['pages'].values())[0]['imageinfo'][0]\n", 97 | " if imageinfo['mime'] in ('image/png', 'image/jpeg') and imageinfo['size'] <= size_limit:\n", 98 | " yield imageinfo['url']\n", 99 | " limit -= 1\n", 100 | " if limit <= 0:\n", 101 | " return\n", 102 | "# else:\n", 103 | "# import pprint\n", 104 | "# pprint.pprint(imageinfo)\n" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "def download_images(search, n):\n", 114 | " \"\"\"download images from mediawiki commons to folders based on the search term\"\"\"\n", 115 | " if not os.path.exists('images'):\n", 116 | " os.mkdir('images')\n", 117 | " tagdir = os.path.join('images', search)\n", 118 | " if not os.path.exists(tagdir):\n", 119 | " os.mkdir(tagdir)\n", 120 | " for url in search_images(search, n):\n", 121 | " r = requests.get(url)\n", 122 | " fname = url.rsplit('/')[-1]\n", 123 | " dest = os.path.join(tagdir, fname)\n", 124 | " # print(\"downloading %s => %s\" % (url, dest))\n", 125 | " sys.stdout.write('+')\n", 126 | " sys.stdout.flush()\n", 127 | " with open(dest, 'wb') as f:\n", 128 | " f.write(r.content)" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": null, 134 | "metadata": {}, 135 | "outputs": [], 136 | "source": [ 137 | "download_images('castle', 100)" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": null, 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "download_images('portrait', 100)" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": null, 152 | "metadata": {}, 153 | "outputs": [], 154 | "source": [ 155 | "download_images('face', 100)" 156 | ] 157 | }, 158 | { 159 | "cell_type": "code", 160 | "execution_count": null, 161 | "metadata": { 162 | "collapsed": true 163 | }, 164 | "outputs": [], 165 | "source": [ 166 | "download_images('child', 100)" 167 | ] 168 | } 169 | ], 170 | "metadata": { 171 | "kernelspec": { 172 | "display_name": "Python 3", 173 | "language": "python", 174 | "name": "python3" 175 | }, 176 | "language_info": { 177 | "codemirror_mode": { 178 | "name": "ipython", 179 | "version": 3 180 | }, 181 | "file_extension": ".py", 182 | "mimetype": "text/x-python", 183 | "name": "python", 184 | "nbconvert_exporter": "python", 185 | "pygments_lexer": "ipython3", 186 | "version": "3.6.0" 187 | }, 188 | "widgets": { 189 | "state": {}, 190 | "version": "1.1.1" 191 | } 192 | }, 193 | "nbformat": 4, 194 | "nbformat_minor": 1 195 | } 196 | -------------------------------------------------------------------------------- /Part-4/examples/MPI Broadcast.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# More efficient data movement with MPI" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "Just like [we did](memmap.ipynb) manually with memmap,\n", 15 | "you can move data more efficiently with MPI by sending it to just one engine,\n", 16 | "and using MPI to broadcast it to the rest of the engines.\n" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 1, 22 | "metadata": { 23 | "collapsed": false 24 | }, 25 | "outputs": [], 26 | "source": [ 27 | "import socket\n", 28 | "import os, sys, re\n", 29 | "\n", 30 | "import numpy as np\n", 31 | "\n", 32 | "import ipyparallel as parallel" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "metadata": {}, 38 | "source": [ 39 | "For this demo, I will connect to a cluster with engines started with MPI.\n", 40 | "If you have MPI and mpi4py on your machine, you can start a local cluster with MPI with:\n", 41 | "\n", 42 | " ipcluster start -n 8 --engines=MPI --profile mpi" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": null, 48 | "metadata": { 49 | "collapsed": false 50 | }, 51 | "outputs": [], 52 | "source": [ 53 | "mpi_profile = 'dirac'\n", 54 | "rc = parallel.Client(profile=mpi_profile)\n", 55 | "eall = rc[:]\n", 56 | "root = rc[-1]" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": { 63 | "collapsed": false 64 | }, 65 | "outputs": [], 66 | "source": [ 67 | "%px from mpi4py.MPI import COMM_WORLD as MPI" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "metadata": { 74 | "collapsed": false 75 | }, 76 | "outputs": [], 77 | "source": [ 78 | "mpi_ranks = eall.apply_async(lambda : MPI.Get_rank()).get_dict()\n", 79 | "root_rank = root.apply_sync(lambda : MPI.Get_rank())\n", 80 | "mpi_ranks" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": null, 86 | "metadata": { 87 | "collapsed": false 88 | }, 89 | "outputs": [], 90 | "source": [ 91 | "sz = 256\n", 92 | "data = np.random.random((sz, sz))\n", 93 | "data = data.dot(data.T)" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": { 100 | "collapsed": false 101 | }, 102 | "outputs": [], 103 | "source": [ 104 | "%%time \n", 105 | "ar = eall.push({'data': data}, block=False)\n", 106 | "ar.wait_interactive()" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": { 113 | "collapsed": false 114 | }, 115 | "outputs": [], 116 | "source": [ 117 | "@parallel.interactive\n", 118 | "def _bcast(key, root_rank):\n", 119 | " \"\"\"function to run on engines as part of broadcast\"\"\"\n", 120 | " g = globals()\n", 121 | " obj = g.get(key, None)\n", 122 | " obj = MPI.bcast(obj, root_rank)\n", 123 | " g[key] = obj\n", 124 | "\n", 125 | "def broadcast(key, obj, dv, root, root_rank):\n", 126 | " \"\"\"More efficient broadcast by doing push to root,\n", 127 | " and MPI broadcast to other engines.\n", 128 | " \n", 129 | " Still O(N) messages, but all but one message is always small.\n", 130 | " \"\"\"\n", 131 | " root.push({key : obj}, block=False)\n", 132 | " return dv.apply_async(_bcast, key, root_rank)" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": null, 138 | "metadata": { 139 | "collapsed": false 140 | }, 141 | "outputs": [], 142 | "source": [ 143 | "%%time\n", 144 | "ar = broadcast('data', data, eall, root, root_rank)\n", 145 | "ar.wait_interactive()" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": null, 151 | "metadata": { 152 | "collapsed": false 153 | }, 154 | "outputs": [], 155 | "source": [ 156 | "eall.apply_sync(np.linalg.norm, parallel.Reference('data'), 2)" 157 | ] 158 | } 159 | ], 160 | "metadata": { 161 | "kernelspec": { 162 | "display_name": "Python 3", 163 | "language": "python", 164 | "name": "python3" 165 | }, 166 | "language_info": { 167 | "codemirror_mode": { 168 | "name": "ipython", 169 | "version": 3 170 | }, 171 | "file_extension": ".py", 172 | "mimetype": "text/x-python", 173 | "name": "python", 174 | "nbconvert_exporter": "python", 175 | "pygments_lexer": "ipython3", 176 | "version": "3.4.3" 177 | } 178 | }, 179 | "nbformat": 4, 180 | "nbformat_minor": 0 181 | } 182 | -------------------------------------------------------------------------------- /Part-4/examples/wikipedia/eventful_dict.py: -------------------------------------------------------------------------------- 1 | import time 2 | 3 | class EventfulDict(dict): 4 | """Eventful dictionary""" 5 | 6 | def __init__(self, *args, **kwargs): 7 | """Sleep is an optional float that allows you to tell the 8 | dictionary to hang for the given amount of seconds on each 9 | event. This is usefull for animations.""" 10 | self._sleep = kwargs.pop('sleep', 0.0) 11 | self._add_callbacks = [] 12 | self._del_callbacks = [] 13 | self._set_callbacks = [] 14 | dict.__init__(self, *args, **kwargs) 15 | 16 | def on_add(self, callback, remove=False): 17 | self._register_callback(self._add_callbacks, callback, remove) 18 | def on_del(self, callback, remove=False): 19 | self._register_callback(self._del_callbacks, callback, remove) 20 | def on_set(self, callback, remove=False): 21 | self._register_callback(self._set_callbacks, callback, remove) 22 | def _register_callback(self, callback_list, callback, remove=False): 23 | if callable(callback): 24 | if remove and callback in callback_list: 25 | callback_list.remove(callback) 26 | elif not remove and not callback in callback_list: 27 | callback_list.append(callback) 28 | else: 29 | raise Exception('Callback must be callable.') 30 | 31 | def _handle_add(self, key, value): 32 | self._try_callbacks(self._add_callbacks, key, value) 33 | self._try_sleep() 34 | def _handle_del(self, key): 35 | self._try_callbacks(self._del_callbacks, key) 36 | self._try_sleep() 37 | def _handle_set(self, key, value): 38 | self._try_callbacks(self._set_callbacks, key, value) 39 | self._try_sleep() 40 | def _try_callbacks(self, callback_list, *pargs, **kwargs): 41 | for callback in callback_list: 42 | callback(*pargs, **kwargs) 43 | 44 | def _try_sleep(self): 45 | if self._sleep > 0.0: 46 | time.sleep(self._sleep) 47 | 48 | def __setitem__(self, key, value): 49 | return_val = None 50 | exists = False 51 | if key in self: 52 | exists = True 53 | 54 | # If the user sets the property to a new dict, make the dict 55 | # eventful and listen to the changes of it ONLY if it is not 56 | # already eventful. Any modification to this new dict will 57 | # fire a set event of the parent dict. 58 | if isinstance(value, dict) and not isinstance(value, EventfulDict): 59 | new_dict = EventfulDict(value) 60 | 61 | def handle_change(*pargs, **kwargs): 62 | self._try_callbacks(self._set_callbacks, key, dict.__getitem__(self, key)) 63 | 64 | new_dict.on_add(handle_change) 65 | new_dict.on_del(handle_change) 66 | new_dict.on_set(handle_change) 67 | return_val = dict.__setitem__(self, key, new_dict) 68 | else: 69 | return_val = dict.__setitem__(self, key, value) 70 | 71 | if exists: 72 | self._handle_set(key, value) 73 | else: 74 | self._handle_add(key, value) 75 | return return_val 76 | 77 | def __delitem__(self, key): 78 | return_val = dict.__delitem__(self, key) 79 | self._handle_del(key) 80 | return return_val 81 | 82 | def pop(self, key): 83 | return_val = dict.pop(self, key) 84 | if key in self: 85 | self._handle_del(key) 86 | return return_val 87 | 88 | def popitem(self): 89 | popped = dict.popitem(self) 90 | if popped is not None and popped[0] is not None: 91 | self._handle_del(popped[0]) 92 | return popped 93 | 94 | def update(self, other_dict): 95 | for (key, value) in other_dict.items(): 96 | self[key] = value 97 | 98 | def clear(self): 99 | for key in list(self.keys()): 100 | del self[key] -------------------------------------------------------------------------------- /Part-4/examples/wikipedia/eventful_graph.py: -------------------------------------------------------------------------------- 1 | """NetworkX graphs do not have events that can be listened to. In order to 2 | watch the NetworkX graph object for changes a custom eventful graph object must 3 | be created. The custom eventful graph object will inherit from the base graph 4 | object and use special eventful dictionaries instead of standard Python dict 5 | instances. Because NetworkX nests dictionaries inside dictionaries, it's 6 | important that the eventful dictionary is capable of recognizing when a 7 | dictionary value is set to another dictionary instance. When this happens, the 8 | eventful dictionary needs to also make the new dictionary an eventful 9 | dictionary. This allows the eventful dictionary to listen to changes made to 10 | dictionaries within dictionaries.""" 11 | import networkx 12 | from networkx.generators.classic import empty_graph 13 | 14 | from eventful_dict import EventfulDict 15 | 16 | class EventfulGraph(networkx.Graph): 17 | 18 | _constructed_callback = None 19 | 20 | @staticmethod 21 | def on_constructed(callback): 22 | """Register a callback to be called when a graph is constructed.""" 23 | if callback is None or callable(callback): 24 | EventfulGraph._constructed_callback = callback 25 | 26 | def __init__(self, *pargs, **kwargs): 27 | """Initialize a graph with edges, name, graph attributes. 28 | 29 | Parameters 30 | sleep: float 31 | optional float that allows you to tell the 32 | dictionary to hang for the given amount of seconds on each 33 | event. This is usefull for animations.""" 34 | super(EventfulGraph, self).__init__(*pargs, **kwargs) 35 | 36 | # Override internal dictionaries with custom eventful ones. 37 | sleep = kwargs.get('sleep', 0.0) 38 | self.graph = EventfulDict(self.graph, sleep=sleep) 39 | self.node = EventfulDict(self.node, sleep=sleep) 40 | self.adj = EventfulDict(self.adj, sleep=sleep) 41 | 42 | # Notify callback of construction event. 43 | if EventfulGraph._constructed_callback: 44 | EventfulGraph._constructed_callback(self) 45 | 46 | 47 | def empty_eventfulgraph_hook(*pargs, **kwargs): 48 | def wrapped(*wpargs, **wkwargs): 49 | """Wrapper for networkx.generators.classic.empty_graph(...)""" 50 | wkwargs['create_using'] = EventfulGraph(*pargs, **kwargs) 51 | return empty_graph(*wpargs, **wkwargs) 52 | return wrapped 53 | -------------------------------------------------------------------------------- /Part-4/examples/wikipedia/widget_forcedirectedgraph.py: -------------------------------------------------------------------------------- 1 | import ipywidgets as widgets # Widget definitions 2 | from traitlets import Unicode, CInt, CFloat # Import the base Widget class and the traitlets Unicode class. 3 | from IPython.display import display, Javascript 4 | 5 | def publish_js(): 6 | with open('./widget_forcedirectedgraph.js', 'r') as f: 7 | display(Javascript(data=f.read())) 8 | 9 | 10 | # Define our ForceDirectedGraphWidget and its target model and default view. 11 | class ForceDirectedGraphWidget(widgets.DOMWidget): 12 | _view_name = Unicode('D3ForceDirectedGraphView').tag(sync=True) 13 | 14 | width = CInt(400).tag(sync=True) 15 | height = CInt(300).tag(sync=True) 16 | charge = CFloat(270.).tag(sync=True) 17 | distance = CInt(30.).tag(sync=True) 18 | strength = CInt(0.3).tag(sync=True) 19 | 20 | def __init__(self, eventful_graph, *pargs, **kwargs): 21 | widgets.DOMWidget.__init__(self, *pargs, **kwargs) 22 | 23 | self._eventful_graph = eventful_graph 24 | self._send_dict_changes(eventful_graph.graph, 'graph') 25 | self._send_dict_changes(eventful_graph.node, 'node') 26 | self._send_dict_changes(eventful_graph.adj, 'adj') 27 | 28 | def _ipython_display_(self, *pargs, **kwargs): 29 | 30 | # Show the widget, then send the current state 31 | widgets.DOMWidget._ipython_display_(self, *pargs, **kwargs) 32 | for (key, value) in self._eventful_graph.graph.items(): 33 | self.send({'dict': 'graph', 'action': 'add', 'key': key, 'value': value}) 34 | for (key, value) in self._eventful_graph.node.items(): 35 | self.send({'dict': 'node', 'action': 'add', 'key': key, 'value': value}) 36 | for (key, value) in self._eventful_graph.adj.items(): 37 | self.send({'dict': 'adj', 'action': 'add', 'key': key, 'value': value}) 38 | 39 | def _send_dict_changes(self, eventful_dict, dict_name): 40 | def key_add(key, value): 41 | self.send({'dict': dict_name, 'action': 'add', 'key': key, 'value': value}) 42 | def key_set(key, value): 43 | self.send({'dict': dict_name, 'action': 'set', 'key': key, 'value': value}) 44 | def key_del(key): 45 | self.send({'dict': dict_name, 'action': 'del', 'key': key}) 46 | eventful_dict.on_add(key_add) 47 | eventful_dict.on_set(key_set) 48 | eventful_dict.on_del(key_del) 49 | -------------------------------------------------------------------------------- /Part-4/exercises/Monte Carlo π.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Multiplexing Exercise - Monte Carlo π\n", 8 | "\n", 9 | "A simple toy problem to get a handle on multiple engines is a Monte\n", 10 | "Carlo approximation of π.\n", 11 | "\n", 12 | "Let's say we have a dartboard with a round target inscribed on a square\n", 13 | "board. If you threw darts randomly, and they land evenly distributed on\n", 14 | "the square board, how many darts would you expect to hit the target?\n", 15 | "\n", 16 | "" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "$$\n", 24 | "\\frac{A_c}{A_{sq}} = \\frac{\\pi r^2}{(2r)^2} = \\frac{\\pi}{4}\n", 25 | "$$" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": null, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "from __future__ import print_function\n", 35 | "\n", 36 | "from random import random\n", 37 | "from math import pi\n", 38 | "\n", 39 | "def mcpi(nsamples):\n", 40 | " s = 0\n", 41 | " for i in range(nsamples):\n", 42 | " x = random()\n", 43 | " y = random()\n", 44 | " if x*x + y*y <= 1:\n", 45 | " s+=1\n", 46 | " return 4.*s/nsamples" 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": null, 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "for n in [10, 100, 1000, 10000, 100000, 1000000]:\n", 56 | " print(\"%8i\" % n, end=' ')\n", 57 | " for i in range(3):\n", 58 | " print(\"%.5f\" % mcpi(n), end=' ')\n", 59 | " print()" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "%timeit mcpi(1000000)" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "It takes a lot of samples to get a good approximation. Can you write a\n", 76 | "function that will use your engines to break up the work?\n", 77 | "\n", 78 | "```python\n", 79 | "def multi_mcpi(dview, nsamples):\n", 80 | " raise NotImplementedError(\"you write this\")\n", 81 | "```" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "import ipyparallel as ipp\n", 91 | "rc = ipp.Client()\n", 92 | "\n", 93 | "view = rc[:]" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": { 100 | "collapsed": true 101 | }, 102 | "outputs": [], 103 | "source": [ 104 | "%load ../soln/mcpi.py" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "multi_mcpi(view, 10000000)" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": null, 119 | "metadata": { 120 | "collapsed": true 121 | }, 122 | "outputs": [], 123 | "source": [ 124 | "# bonus: plot convergence like we did for Monte Carlo integration" 125 | ] 126 | } 127 | ], 128 | "metadata": { 129 | "kernelspec": { 130 | "display_name": "Python 3", 131 | "language": "python", 132 | "name": "python3" 133 | }, 134 | "language_info": { 135 | "codemirror_mode": { 136 | "name": "ipython", 137 | "version": 3 138 | }, 139 | "file_extension": ".py", 140 | "mimetype": "text/x-python", 141 | "name": "python", 142 | "nbconvert_exporter": "python", 143 | "pygments_lexer": "ipython3", 144 | "version": "3.6.0" 145 | }, 146 | "widgets": { 147 | "state": {}, 148 | "version": "1.1.1" 149 | } 150 | }, 151 | "nbformat": 4, 152 | "nbformat_minor": 1 153 | } 154 | -------------------------------------------------------------------------------- /Part-4/figs/allconnections.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-4/figs/allconnections.png -------------------------------------------------------------------------------- /Part-4/figs/darts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-4/figs/darts.png -------------------------------------------------------------------------------- /Part-4/figs/latency.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-4/figs/latency.png -------------------------------------------------------------------------------- /Part-4/figs/latency2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-4/figs/latency2.png -------------------------------------------------------------------------------- /Part-4/figs/map.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-4/figs/map.png -------------------------------------------------------------------------------- /Part-4/figs/throughput1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-4/figs/throughput1.png -------------------------------------------------------------------------------- /Part-4/figs/throughput2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-4/figs/throughput2.png -------------------------------------------------------------------------------- /Part-4/figs/wideView.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jupyter/ngcm-tutorial/e2acd3365c8a18cb4360ee7ecd4b74a9e1d9ce27/Part-4/figs/wideView.png -------------------------------------------------------------------------------- /Part-4/hints.py: -------------------------------------------------------------------------------- 1 | from IPython.display import display, HTML 2 | 3 | def mmhint(): 4 | display(HTML(""" 5 | 6 |
7 | Remember - multiply rows of one by the columns of the other. 8 |
9 | 10 |
11 | 12 |
13 | easiest implementation involves one each of: push, scatter, execute, gather 14 |
15 | 19 | """)) 20 | 21 | def nesthint(): 22 | display(HTML(""" 23 | 26 | 27 |
28 | `itertools.product` and `zip` will be helpful. 29 |
30 | 33 | """)) -------------------------------------------------------------------------------- /Part-4/hubble-tiles.py: -------------------------------------------------------------------------------- 1 | """Construct tiles of a high-res hubble image 2 | for sample data 3 | """ 4 | 5 | import os, sys 6 | 7 | import requests 8 | import skimage.io 9 | 10 | from ipywidgets import IntProgress 11 | from IPython.display import display 12 | 13 | url = 'http://imgsrc.hubblesite.org/hvi/uploads/image_file/image_attachment/9077/full_jpg.jpg' 14 | here = os.path.dirname(__file__) 15 | full_size_file = os.path.join(here, 'hubble.jpg') 16 | 17 | def download_original(): 18 | p = IntProgress(max=1, description="Downloading") 19 | display(p) 20 | if os.path.exists(full_size_file): 21 | print("Already have %s" % full_size_file) 22 | p.value = p.max 23 | else: 24 | r = requests.get(url, stream=True) 25 | content_length = r.headers.get('content-length', int(1e8)) 26 | print("Downloading %s" % url) 27 | p.max = content_length 28 | r.raise_for_status() 29 | 30 | with open(full_size_file, 'wb') as f: 31 | for chunk in r.iter_content(chunk_size=8096): 32 | p.value += len(chunk) 33 | f.write(chunk) 34 | p.value = p.max 35 | 36 | def tile_image(full_size_path, tile_size=400): 37 | p = IntProgress(description='Tiling', max=1) 38 | display(p) 39 | root_dir = os.path.dirname(full_size_path) 40 | name, ext = os.path.splitext(os.path.basename(full_size_path)) 41 | tile_dir = os.path.join(root_dir, '{0}-{1}x{1}'.format(name, tile_size)) 42 | if not os.path.exists(tile_dir): 43 | os.mkdir(tile_dir) 44 | tpl = os.path.join(tile_dir, '{name}-{i:02}-{j:02}{ext}') 45 | full_size = skimage.io.imread(full_size_path) 46 | X, Y = full_size.shape[:2] 47 | total_tiles = (X // tile_size) * (Y // tile_size) 48 | p.max = total_tiles 49 | print("Creating %i tiles in %s" % (total_tiles, tile_dir)) 50 | 51 | for i in range(X // tile_size): 52 | for j in range(Y // tile_size): 53 | tile = full_size[ 54 | i * tile_size: (i+1) * tile_size, 55 | j * tile_size: (j+1) * tile_size, 56 | : 57 | ] 58 | fname = tpl.format(**locals()) 59 | p.value += 1 60 | skimage.io.imsave(fname, tile) 61 | p.value = p.max 62 | 63 | if __name__ == '__main__': 64 | download_original() 65 | tile_image(full_size_file) 66 | -------------------------------------------------------------------------------- /Part-4/images_common.py: -------------------------------------------------------------------------------- 1 | import os 2 | import matplotlib.pyplot as plt 3 | 4 | from skimage.io import imread 5 | 6 | def plot_corners(img, corners, show=True): 7 | """Display the image and plot all contours found""" 8 | plt.imshow(img, cmap='gray') 9 | plt.plot(corners[:,1], corners[:,0], 'r+', markeredgewidth=1.5, markersize=8) # Plot corners 10 | plt.axis('image') 11 | plt.xticks([]) 12 | plt.yticks([]) 13 | if show: 14 | plt.show() 15 | 16 | def find_corners(path, min_distance=5): 17 | """Find corners in an image at path 18 | 19 | Returns the image and the corner lists. 20 | """ 21 | from skimage.feature import corner_harris, corner_peaks 22 | img = imread(path, flatten=True) 23 | corners = corner_peaks(corner_harris(img), min_distance=min_distance) 24 | return img, corners 25 | 26 | def get_corners_image(path): 27 | """Given a path, return a PNG of the image with contour lines 28 | 29 | Calls both find_contours and plot_contours 30 | """ 31 | from IPython.core.pylabtools import print_figure 32 | 33 | img, corners = find_corners(path) 34 | plot_corners(img, corners, show=False) 35 | fig = plt.gcf() 36 | pngdata = print_figure(fig) 37 | plt.close(fig) 38 | return pngdata 39 | 40 | def get_pictures(pictures_dir): 41 | """Return a list of picture files found in pictures_dir""" 42 | 43 | pictures = [] 44 | for directory, subdirs, files in os.walk(pictures_dir): 45 | for fname in files: 46 | if fname.lower().endswith(('.jpg', '.png')): 47 | pictures.append(os.path.join(directory, fname)) 48 | 49 | return pictures 50 | 51 | 52 | def find_blobs(img_path): 53 | """Find the blobs in an image""" 54 | from skimage.color import rgb2gray 55 | from skimage.feature import blob_doh 56 | from skimage.io import imread 57 | 58 | image = imread(img_path) 59 | image_gray = rgb2gray(image) 60 | return image, blob_doh(image_gray, max_sigma=30, threshold=.01) 61 | 62 | 63 | def plot_blobs(image, blobs): 64 | """Plot the blobs in an image""" 65 | fig, ax = plt.subplots() 66 | ax.grid(False) 67 | ax.imshow(image, interpolation='nearest') 68 | ax.set_xticks([]) 69 | ax.set_yticks([]) 70 | for blob in blobs: 71 | y, x, r = blob 72 | c = plt.Circle((x, y), r, color='r', linewidth=2, fill=False) 73 | ax.add_patch(c) 74 | 75 | -------------------------------------------------------------------------------- /Part-4/soln/matmul.py: -------------------------------------------------------------------------------- 1 | def pdot(v, A, B): 2 | v['B'] = B # push B everywhere 3 | v.scatter('A', A) # scatter A 4 | v.execute('C=A.dot(B)') # compute the dot-product 5 | return v.gather('C', block=True) # gather the resulting sub-arrays 6 | -------------------------------------------------------------------------------- /Part-4/soln/mcpi.py: -------------------------------------------------------------------------------- 1 | def mcpi(nsamples): 2 | from random import random 3 | s = 0 4 | for i in range(nsamples): 5 | x = random() 6 | y = random() 7 | if x*x + y*y <= 1: 8 | s += 1 9 | return 4. * s / nsamples 10 | 11 | def multi_mcpi(view, nsamples): 12 | p = len(view.targets) 13 | if nsamples % p: 14 | # ensure even divisibility 15 | nsamples += p - (nsamples % p) 16 | 17 | subsamples = nsamples // p 18 | 19 | ar = view.apply_async(mcpi, subsamples) 20 | return sum(ar) / p -------------------------------------------------------------------------------- /Part-4/soln/nestedloop.py: -------------------------------------------------------------------------------- 1 | # To parallelize every call with map, you just need to get a list for each argument. 2 | # You can use `itertools.product` + `zip` to get this: 3 | 4 | 5 | import itertools 6 | 7 | product = list(itertools.product(widths, heights)) 8 | # [(1, 6), (1, 7), (2, 6), (2, 7), (3, 6), (3, 7)] 9 | 10 | # So we have a "list of pairs", 11 | # but what we really want is a single list for each argument, i.e. a "pair of lists". 12 | # This is exactly what the slightly weird `zip(*product)` syntax gets us: 13 | 14 | allwidths, allheights = zip(*itertools.product(widths, heights)) 15 | 16 | print(" widths", allwidths) 17 | print("heights", allheights) 18 | 19 | # Now we just map our function onto those two lists, to parallelize nested for loops: 20 | 21 | ar = lview.map_async(area, allwidths, allheights) 22 | -------------------------------------------------------------------------------- /Part-4/soln/ngrams.py: -------------------------------------------------------------------------------- 1 | def ngrams_parallel(view, fnames, n=1): 2 | """Compute ngrams in parallel 3 | 4 | view - An IPython DirectView 5 | fnames - The filenames containing the split data. 6 | """ 7 | 8 | ar = view.map_async(ngrams, fnames, [n] * len(fnames)) 9 | counts = {} 10 | for engine_count in ar: 11 | for gram, count in engine_count.items(): 12 | if gram not in counts: 13 | counts[gram] = 0 14 | counts[gram] += count 15 | return counts 16 | -------------------------------------------------------------------------------- /Part-4/soln/remote_iter.py: -------------------------------------------------------------------------------- 1 | import ipyparallel as parallel 2 | 3 | def remote_iterator(view, name): 4 | """Return an iterator on an object living on a remote engine.""" 5 | it_name = '_%s_iter' % name 6 | view.execute('%s = iter(%s)' % (it_name,name), block=True) 7 | ref = parallel.Reference(it_name) 8 | while True: 9 | try: 10 | yield view.apply_sync(next, ref) 11 | # This causes the StopIteration exception to be raised. 12 | except parallel.RemoteError as e: 13 | if e.ename == 'StopIteration': 14 | raise StopIteration 15 | else: 16 | raise e 17 | -------------------------------------------------------------------------------- /Part-4/soln/remote_iter_hint.py: -------------------------------------------------------------------------------- 1 | from IPython.display import display 2 | 3 | t_minus = range(10,0,-1) 4 | 5 | def lazy_iterator(name): 6 | seq = eval(name) 7 | it = iter(seq) 8 | while True: 9 | try: 10 | yield next(it) 11 | # this looks silly locally, but it will be useful for the remote version: 12 | except StopIteration: 13 | raise StopIteration 14 | 15 | lzit = lazy_iterator('t_minus') 16 | display(lzit) 17 | list(lzit) -------------------------------------------------------------------------------- /Part-4/tutorial/myscript.py: -------------------------------------------------------------------------------- 1 | import math 2 | import numpy 3 | import sys 4 | 5 | a=5 6 | 7 | def mysquare(x): 8 | return x*x -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # IPython/Jupyter Workshop at the NGCM Summer Academy 2 | 3 | Materials for the IPython/Jupyter workshop at the Next-Generation Computational 4 | Modeling Summer Academy: 5 | 6 | - 2-day course 7 | - 27 and 28 June 2017 8 | - At Southampton University, Boldrewood campus 9 | 10 | Course URL: http://ngcm.soton.ac.uk/summer-academy/ipython.html 11 | 12 | Teaching is from 10am to 6pm, broken up by half-hour tea breaks morning and 13 | afternoon, and an hour lunch break. See 14 | [the programme](http://ngcm.soton.ac.uk/summer-academy/programme.html) for 15 | details. 16 | 17 | ## Course content 18 | 19 | Jupyter and IPython provide tools for interactive and parallel computing that are 20 | widely used in scientific computing. We will show some uses of IPython 21 | for scientific applications, focusing on exciting recent developments, 22 | web-based notebooks with code, graphics, and rich HTML. 23 | 24 | Day 1 morning: Core Jupyter and IPython 25 | 26 | - Notebook Basics 27 | - IPython - beyond plain python 28 | - Markdown Cells 29 | - Rich Display System 30 | - Beyond Python: the Jupyter architecture with Julia and R 31 | 32 | Day 1 afternoon: Working with notebook files 33 | 34 | - Converting notebooks to other formats with nbconvert 35 | - Using notebooks in version control, with git and nbdime 36 | - Sharing notebooks online using nbviewer 37 | - Notebooks in continuous integration with nbval 38 | 39 | Day 2 morning: Interactive widgets in notebooks 40 | 41 | - Using interact() to explore a function 42 | - Creating widgets manually and connecting them to Python functions 43 | - Laying out widgets on the page 44 | - The architecture of interactive widgets 45 | 46 | Day 2 afternoon: Parallel computing with IPython 47 | 48 | - Overview of the ipyparallel model 49 | - Controller and engines 50 | - Basics of remote execution 51 | - Direct vs task execution 52 | - Integration with MPI codes 53 | - Handling dependencies between tasks 54 | - Performance considerations 55 | 56 | 57 | ## Software Requirements 58 | 59 | - Python 3.x 60 | - Jupyter, including the Notebook and IPython. It should 61 | be available through the usual distribution channels, such as 62 | [Anaconda](http://continuum.io/downloads). 63 | - Your favorite text editor. 64 | - If you have trouble installing Anaconda, [this blog 65 | entry](http://www.southampton.ac.uk/~fangohr/blog/installation-of-python-spyder-numpy-sympy-scipy-pytest-matplotlib-via-anaconda.html) 66 | may help. 67 | - For the material related to `nbconvert`, the `pandoc` package, together with a `latex` installation, would be useful. 68 | 69 | To install the packages required for this course and the Pandas course 70 | in a new environment with Anaconda, run: 71 | 72 | ```bash 73 | conda create -n ngcm python=3 numpy scipy jupyter ipywidgets pandas matplotlib requests scikit-image sympy 74 | ``` 75 | 76 | Then, to use this environment, enter: 77 | 78 | ```bash 79 | source activate ngcm 80 | ``` 81 | 82 | On Windows, this command is just `activate ngcm`. 83 | 84 | ## Checking your installation 85 | 86 | You can download and run this 87 | [version_check.py](https://github.com/jupyter/ngcm-tutorial/raw/master/version_check.py) 88 | script, and execute it using `python version_check.py` to check 89 | you have fulfilled the installation requirements. 90 | 91 | 92 | ## Required knowledge 93 | 94 | - Basic Python, 95 | - some vague notion of html would be great. 96 | 97 | 98 | ## The trainers 99 | 100 | - [Thomas Kluyver](http://cmg.soton.ac.uk/people/tk2e15/) 101 | - [MinRK](http://github.com/minrk) 102 | 103 | 104 | ## Infrastructure 105 | 106 | - [Etherpad](https://public.etherpad-mozilla.org/p/ngcm-2017-jupyter) 107 | - [Slack channel "ipython"](https://ngcmsummeracademy2016.slack.com/messages/ipython/) 108 | -------------------------------------------------------------------------------- /utils/list_pyfiles.ipy: -------------------------------------------------------------------------------- 1 | # A simple IPython script that provides Notebook links to .py files in the cwd 2 | 3 | from IPython.display import FileLink, display 4 | files =!ls *.py 5 | for f in files: 6 | display(FileLink(f)) -------------------------------------------------------------------------------- /utils/list_subdirs.ipy: -------------------------------------------------------------------------------- 1 | # A simple IPython script that lists files in all subdirs 2 | 3 | from IPython.display import FileLinks, display 4 | dirs =!ls -d */ 5 | for d in dirs: 6 | display(FileLinks(d)) -------------------------------------------------------------------------------- /version_check.py: -------------------------------------------------------------------------------- 1 | # check Python version 2 | 3 | import sys 4 | assert sys.version_info.major >= 3, "You need Python 3." 5 | print("Python version is {} -> okay".format(sys.version.splitlines()[0])) 6 | 7 | # check IPython version 8 | 9 | import IPython 10 | assert IPython.version_info[0] >= 4, "You need IPython >= 4.0" 11 | print("IPython version is {} -> okay".format(IPython.__version__)) 12 | 13 | import notebook 14 | assert notebook.version_info >= (4,2), "You need notebook >= 4.2" 15 | print("notebook version is {} -> okay".format(notebook.__version__)) 16 | 17 | import ipywidgets 18 | assert 4 <= ipywidgets.version_info[0] < 6, "You need ipywidgets 4.x-5.x" 19 | print("ipywidgets version is {} -> okay".format(ipywidgets.__version__)) 20 | 21 | import ipyparallel 22 | assert ipyparallel.version_info[0] > 4, "You need ipyparallel >= 4" 23 | print("ipyparallel version is {} -> okay".format(ipyparallel.__version__)) 24 | 25 | import traitlets 26 | assert traitlets.version_info >= (4,2), "You need traitlets >= 4.2" 27 | print("traitlets version is {} -> okay".format(traitlets.__version__)) 28 | 29 | print("You are all set for the IPython course at the NGCM Summer academy 2016.") 30 | -------------------------------------------------------------------------------- /website_summary.rst: -------------------------------------------------------------------------------- 1 | Course content 2 | -------------- 3 | 4 | Jupyter and IPython provide tools for interactive and parallel computing that are 5 | widely used in scientific computing. We will show some uses of IPython 6 | for scientific applications, focusing on exciting recent developments, 7 | web-based notebooks with code, graphics, and rich HTML. 8 | 9 | Day 1 morning: Core Jupyter and IPython 10 | 11 | - Notebook Basics 12 | - IPython - beyond plain python 13 | - Markdown Cells 14 | - Rich Display System 15 | - Beyond Python: the Jupyter architecture with Julia and R 16 | 17 | Day 1 afternoon: Working with notebook files 18 | 19 | - Converting notebooks to other formats with nbconvert 20 | - Using notebooks in version control, with git and nbdime 21 | - Sharing notebooks online using nbviewer 22 | - Notebooks in continuous integration with nbval 23 | 24 | Day 2 morning: Interactive widgets in notebooks 25 | 26 | - Using interact() to explore a function 27 | - Creating widgets manually and connecting them to Python functions 28 | - Laying out widgets on the page 29 | - The architecture of interactive widgets 30 | 31 | Day 2 afternoon: Parallel computing with IPython 32 | 33 | - Overview of the ipyparallel model 34 | - Controller and engines 35 | - Basics of remote execution 36 | - Direct vs task execution 37 | - Integration with MPI codes 38 | - Handling dependencies between tasks 39 | - Performance considerations 40 | 41 | Software Requirements 42 | --------------------- 43 | 44 | - Python 3.x 45 | - Jupyter, including the Notebook and IPython. It should 46 | be available through the usual distribution channels, such as 47 | `Anaconda `__. 48 | - Your favorite text editor. 49 | - If you have trouble installing Anaconda, `this blog 50 | entry `__ 51 | may help. 52 | - For the material related to ``nbconvert``, the ``pandoc`` package, together with a ``latex`` installation, would be useful. 53 | 54 | To install the packages required for this course and the Pandas course 55 | in a new environment with Anaconda, run:: 56 | 57 | conda create -n ngcm python=3 numpy scipy jupyter ipywidgets pandas matplotlib requests scikit-image sympy 58 | 59 | Then, to use this environment, enter:: 60 | 61 | source activate ngcm 62 | 63 | On Windows, this command is just ``activate ngcm``. 64 | 65 | Checking your installation 66 | -------------------------- 67 | 68 | You can download and run this 69 | `version_check.py `__ 70 | script, and execute it using ``python version_check.py`` to check 71 | you have fulfilled the installation requirements. 72 | 73 | Required knowledge 74 | ------------------ 75 | 76 | - Basic Python, 77 | - some vague notion of html would be great. 78 | 79 | The trainers 80 | ------------ 81 | 82 | - `Thomas Kluyver `__ 83 | - `MinRK `__ 84 | 85 | Infrastructure 86 | -------------- 87 | 88 | - `Etherpad `__ 89 | - `Slack channel "ipython" `__ 90 | --------------------------------------------------------------------------------