├── .gitignore ├── 0_dask_tutorial ├── 01-dask.delayed.ipynb ├── 02-dask-arrays.ipynb ├── README.md ├── prep_data.py ├── solutions │ ├── 01-delayed-control-flow.py │ ├── 01-delayed-loop.py │ ├── 02-dask-arrays-blocked-mean.py │ ├── 02-dask-arrays-make-arrays.py │ ├── 02-dask-arrays-stacked.py │ ├── 02-dask-arrays-store.py │ ├── 02-dask-arrays-weather-difference.py │ └── 02-dask-arrays-weather-mean.py └── static │ ├── fail-case.gif │ ├── ml-dimensions-color.png │ ├── ml-dimensions.png │ ├── sklearn-parallel-dask.png │ └── sklearn-parallel.png ├── 1_dask_array ├── 1_multicore_array.ipynb ├── 2_multicore_array_outofcore.ipynb ├── 3_multinode_distributed_array.ipynb ├── 4_dask_graphs.ipynb ├── dask.delayed.ipynb ├── mydask.png ├── processes.ipynb └── threads.ipynb ├── 2_digits_of_pi ├── 1_serial_digits_of_pi.py ├── 2_multicore_digits_of_pi.py ├── 3_dask_multicore_digits_of_pi.py ├── 4_dask_multinode_digits_of_pi.py └── run_digits_of_pi_scripts.ipynb ├── GIL_counter.py ├── README.md ├── dask_slurm ├── dask_workers.slrm ├── launch_scheduler.sh └── launch_worker.sh ├── intro.ipynb ├── notebook_singularity.slrm └── numba_groupby_pixels.ipynb /.gitignore: -------------------------------------------------------------------------------- 1 | mydask.png 2 | dask-worker-space 3 | -------------------------------------------------------------------------------- /0_dask_tutorial/01-dask.delayed.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "\"Dask\n", 11 | "\n", 12 | "# Parallelize code with `dask.delayed`\n", 13 | "\n", 14 | "In this section we parallelize simple for-loop style code with Dask and `dask.delayed`.\n", 15 | "\n", 16 | "This is a simple way to use `dask` to parallelize existing codebases or build [complex systems](http://matthewrocklin.com/blog/work/2018/02/09/credit-models-with-dask). This will also help us to develop an understanding for later sections." 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "## Basics\n", 24 | "\n", 25 | "First let's make some toy functions, `inc` and `add`, that sleep for a while to simulate work. We'll then time running these functions normally.\n", 26 | "\n", 27 | "In the next section we'll parallelize this code." 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": null, 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "from time import sleep\n", 37 | "\n", 38 | "def inc(x):\n", 39 | " sleep(1)\n", 40 | " return x + 1\n", 41 | "\n", 42 | "def add(x, y):\n", 43 | " sleep(1)\n", 44 | " return x + y" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "%%time\n", 54 | "# This takes three seconds to run because we call each\n", 55 | "# function sequentially, one after the other\n", 56 | "\n", 57 | "x = inc(1)\n", 58 | "y = inc(2)\n", 59 | "z = add(x, y)" 60 | ] 61 | }, 62 | { 63 | "cell_type": "markdown", 64 | "metadata": {}, 65 | "source": [ 66 | "### Parallelize with the `dask.delayed` decorator\n", 67 | "\n", 68 | "Those two increment calls *could* be called in parallel.\n", 69 | "\n", 70 | "We'll wrap the `inc` and `add` functions in the `dask.delayed` decorator. When we call the delayed version by passing the arguments, the original function isn't actually called yet.\n", 71 | "Instead, a *task graph* is built up, representing the *delayed* function call." 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": null, 77 | "metadata": {}, 78 | "outputs": [], 79 | "source": [ 80 | "from dask import delayed" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": null, 86 | "metadata": {}, 87 | "outputs": [], 88 | "source": [ 89 | "%%time\n", 90 | "# This runs immediately, all it does is build a graph\n", 91 | "\n", 92 | "x = delayed(inc)(1)\n", 93 | "y = delayed(inc)(2)\n", 94 | "z = delayed(add)(x, y)" 95 | ] 96 | }, 97 | { 98 | "cell_type": "markdown", 99 | "metadata": {}, 100 | "source": [ 101 | "This ran immediately, since nothing has really happened yet.\n", 102 | "\n", 103 | "To get the result, call `compute`." 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": {}, 110 | "outputs": [], 111 | "source": [ 112 | "%%time\n", 113 | "# This actually runs our computation using a local thread pool\n", 114 | "\n", 115 | "z.compute()" 116 | ] 117 | }, 118 | { 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "## What just happened?\n", 123 | "\n", 124 | "The `z` object is a lazy `Delayed` object. This object holds everything we need to compute the final result. We can compute the result with `.compute()` as above or we can visualize the task graph for this value with `.visualize()`." 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": null, 130 | "metadata": {}, 131 | "outputs": [], 132 | "source": [ 133 | "z" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": null, 139 | "metadata": {}, 140 | "outputs": [], 141 | "source": [ 142 | "# Look at the task graph for `z`\n", 143 | "z.visualize()" 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "### Some questions to consider:\n", 151 | "\n", 152 | "- Why did we go from 3s to 2s? Why weren't we able to parallelize down to 1s?\n", 153 | "- What would have happened if the inc and add functions didn't include the `sleep(1)`? Would Dask still be able to speed up this code?\n", 154 | "- What if we have multiple outputs or also want to get access to x or y?" 155 | ] 156 | }, 157 | { 158 | "cell_type": "markdown", 159 | "metadata": {}, 160 | "source": [ 161 | "## Exercise: Parallelize a for loop\n", 162 | "\n", 163 | "`for` loops are one of the most common things that we want to parallelize. Use `dask.delayed` on `inc` and `sum` to parallelize the computation below:" 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": null, 169 | "metadata": {}, 170 | "outputs": [], 171 | "source": [ 172 | "data = [1, 2, 3, 4, 5, 6, 7, 8]" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": null, 178 | "metadata": {}, 179 | "outputs": [], 180 | "source": [ 181 | "%%time\n", 182 | "# Sequential code\n", 183 | "\n", 184 | "results = []\n", 185 | "for x in data:\n", 186 | " y = inc(x)\n", 187 | " results.append(y)\n", 188 | " \n", 189 | "total = sum(results)" 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": null, 195 | "metadata": {}, 196 | "outputs": [], 197 | "source": [ 198 | "total" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": null, 204 | "metadata": {}, 205 | "outputs": [], 206 | "source": [ 207 | "%%time\n", 208 | "# Your parallel code here..." 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": null, 214 | "metadata": {}, 215 | "outputs": [], 216 | "source": [ 217 | "%load solutions/01-delayed-loop.py" 218 | ] 219 | }, 220 | { 221 | "cell_type": "markdown", 222 | "metadata": {}, 223 | "source": [ 224 | "## Exercise: Parallelizing a for-loop code with control flow\n", 225 | "\n", 226 | "Often we want to delay only *some* functions, running a few of them immediately. This is especially helpful when those functions are fast and help us to determine what other slower functions we should call. This decision, to delay or not to delay, is usually where we need to be thoughtful when using `dask.delayed`.\n", 227 | "\n", 228 | "In the example below we iterate through a list of inputs. If that input is even then we want to call `inc`. If the input is odd then we want to call `double`. This `is_even` decision to call `inc` or `double` has to be made immediately (not lazily) in order for our graph-building Python code to proceed." 229 | ] 230 | }, 231 | { 232 | "cell_type": "code", 233 | "execution_count": null, 234 | "metadata": {}, 235 | "outputs": [], 236 | "source": [ 237 | "def double(x):\n", 238 | " sleep(1)\n", 239 | " return 2 * x\n", 240 | "\n", 241 | "def is_even(x):\n", 242 | " return not x % 2\n", 243 | "\n", 244 | "data = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]" 245 | ] 246 | }, 247 | { 248 | "cell_type": "code", 249 | "execution_count": null, 250 | "metadata": {}, 251 | "outputs": [], 252 | "source": [ 253 | "%%time\n", 254 | "# Sequential code\n", 255 | "\n", 256 | "results = []\n", 257 | "for x in data:\n", 258 | " if is_even(x):\n", 259 | " y = double(x)\n", 260 | " else:\n", 261 | " y = inc(x)\n", 262 | " results.append(y)\n", 263 | " \n", 264 | "total = sum(results)\n", 265 | "print(total)" 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": null, 271 | "metadata": {}, 272 | "outputs": [], 273 | "source": [ 274 | "%%time\n", 275 | "# Your parallel code here...\n", 276 | "# TODO: parallelize the sequential code above using dask.delayed\n", 277 | "# You will need to delay some functions, but not all" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": null, 283 | "metadata": {}, 284 | "outputs": [], 285 | "source": [ 286 | "%load solutions/01-delayed-control-flow.py" 287 | ] 288 | }, 289 | { 290 | "cell_type": "code", 291 | "execution_count": null, 292 | "metadata": {}, 293 | "outputs": [], 294 | "source": [ 295 | "%time total.compute()" 296 | ] 297 | }, 298 | { 299 | "cell_type": "code", 300 | "execution_count": null, 301 | "metadata": {}, 302 | "outputs": [], 303 | "source": [ 304 | "total.visualize()" 305 | ] 306 | }, 307 | { 308 | "cell_type": "markdown", 309 | "metadata": {}, 310 | "source": [ 311 | "### Some questions to consider:\n", 312 | "\n", 313 | "- What are other examples of control flow where we can't use delayed?\n", 314 | "- What would have happened if we had delayed the evaluation of `is_even(x)` in the example above?\n", 315 | "- What are your thoughts on delaying `sum`? This function is both computational but also fast to run." 316 | ] 317 | } 318 | ], 319 | "metadata": { 320 | "kernelspec": { 321 | "display_name": "Python 3", 322 | "language": "python", 323 | "name": "python3" 324 | }, 325 | "language_info": { 326 | "codemirror_mode": { 327 | "name": "ipython", 328 | "version": 3 329 | }, 330 | "file_extension": ".py", 331 | "mimetype": "text/x-python", 332 | "name": "python", 333 | "nbconvert_exporter": "python", 334 | "pygments_lexer": "ipython3", 335 | "version": "3.6.5" 336 | } 337 | }, 338 | "nbformat": 4, 339 | "nbformat_minor": 2 340 | } 341 | -------------------------------------------------------------------------------- /0_dask_tutorial/02-dask-arrays.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "\"Dask" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "# Table of Contents\n", 18 | "* [Arrays](#Arrays)\n", 19 | "\t* [Blocked Algorithms](#Blocked-Algorithms)\n", 20 | "\t\t* [Exercise: Compute the mean using a blocked algorithm](#Exercise:--Compute-the-mean-using-a-blocked-algorithm)\n", 21 | "\t\t* [Exercise: Compute the mean](#Exercise:--Compute-the-mean)" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "# Arrays" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "\n", 36 | "\n", 37 | "Dask array provides a parallel, larger-than-memory, n-dimensional array using blocked algorithms. Simply put: distributed Numpy.\n", 38 | "\n", 39 | "* **Parallel**: Uses all of the cores on your computer\n", 40 | "* **Larger-than-memory**: Lets you work on datasets that are larger than your available memory by breaking up your array into many small pieces, operating on those pieces in an order that minimizes the memory footprint of your computation, and effectively streaming data from disk.\n", 41 | "* **Blocked Algorithms**: Perform large computations by performing many smaller computations\n", 42 | "\n", 43 | "**Related Documentation**\n", 44 | "\n", 45 | "* http://dask.readthedocs.io/en/latest/array.html\n", 46 | "* http://dask.readthedocs.io/en/latest/array-api.html" 47 | ] 48 | }, 49 | { 50 | "cell_type": "markdown", 51 | "metadata": {}, 52 | "source": [ 53 | "## Blocked Algorithms" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "A *blocked algorithm* executes on a large dataset by breaking it up into many small blocks.\n", 61 | "\n", 62 | "For example, consider taking the sum of a billion numbers. We might instead break up the array into 1,000 chunks, each of size 1,000,000, take the sum of each chunk, and then take the sum of the intermediate sums.\n", 63 | "\n", 64 | "We achieve the intended result (one sum on one billion numbers) by performing many smaller results (one thousand sums on one million numbers each, followed by another sum of a thousand numbers.)\n", 65 | "\n", 66 | "We do exactly this with Python and NumPy in the following example:" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "**Create random dataset**" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": null, 79 | "metadata": {}, 80 | "outputs": [], 81 | "source": [ 82 | "# create data if it doesn't already exist\n", 83 | "# Run this to create the data inside the local `data` folder if you are not on Comet\n", 84 | "# from prep_data import random_array\n", 85 | "# random_array() \n", 86 | "\n", 87 | "# Load data with h5py\n", 88 | "# this gives the load prescription, but does no real work.\n", 89 | "import h5py\n", 90 | "import os\n", 91 | "f = h5py.File(os.path.join('/oasis/scratch/comet/zonca/temp_project', 'random.hdf5'), mode='r')\n", 92 | "dset = f['/x']" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "**Compute sum using blocked algorithm**" 100 | ] 101 | }, 102 | { 103 | "cell_type": "markdown", 104 | "metadata": {}, 105 | "source": [ 106 | "Here we compute the sum of this large array on disk by \n", 107 | "\n", 108 | "1. Computing the sum of each 1,000,000 sized chunk of the array\n", 109 | "2. Computing the sum of the 1,000 intermediate sums\n", 110 | "\n", 111 | "Note that we are fetching every partial result from the cluster and summing them here, in the notebook kernel." 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": null, 117 | "metadata": {}, 118 | "outputs": [], 119 | "source": [ 120 | "# Compute sum of large array, one million numbers at a time\n", 121 | "sums = []\n", 122 | "for i in range(0, 1000000000, 1000000):\n", 123 | " chunk = dset[i: i + 1000000] # pull out numpy array\n", 124 | " sums.append(chunk.sum())\n", 125 | "\n", 126 | "total = sum(sums)\n", 127 | "print(total)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": {}, 133 | "source": [ 134 | "### Exercise: Compute the mean using a blocked algorithm" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "Now that we've seen the simple example above try doing a slightly more complicated problem, compute the mean of the array. You can do this by changing the code above with the following alterations:\n", 142 | "\n", 143 | "1. Compute the sum of each block\n", 144 | "2. Compute the length of each block\n", 145 | "3. Compute the sum of the 1,000 intermediate sums and the sum of the 1,000 intermediate lengths and divide one by the other\n", 146 | "\n", 147 | "This approach is overkill for our case but does nicely generalize if we don't know the size of the array or individual blocks beforehand." 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": null, 153 | "metadata": {}, 154 | "outputs": [], 155 | "source": [ 156 | "# Compute the mean of the array" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": null, 162 | "metadata": {}, 163 | "outputs": [], 164 | "source": [ 165 | "# %load solutions/02-dask-arrays-blocked-mean.py\n", 166 | "sums = []\n", 167 | "lengths = []\n", 168 | "for i in range(0, 1000000000, 1000000):\n", 169 | " chunk = dset[i: i + 1000000] # pull out numpy array\n", 170 | " sums.append(chunk.sum())\n", 171 | " lengths.append(len(chunk))\n", 172 | "\n", 173 | "total = sum(sums)\n", 174 | "length = sum(lengths)\n", 175 | "print(total / length)" 176 | ] 177 | }, 178 | { 179 | "cell_type": "markdown", 180 | "metadata": {}, 181 | "source": [ 182 | "`dask.array` contains these algorithms\n", 183 | "--------------------------------------------\n", 184 | "\n", 185 | "Dask.array is a NumPy-like library that does these kinds of tricks to operate on large datasets that don't fit into memory. It extends beyond the linear problems discussed above to full N-Dimensional algorithms and a decent subset of the NumPy interface." 186 | ] 187 | }, 188 | { 189 | "cell_type": "markdown", 190 | "metadata": {}, 191 | "source": [ 192 | "**Create `dask.array` object**" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "You can create a `dask.array` `Array` object with the `da.from_array` function. This function accepts\n", 200 | "\n", 201 | "1. `data`: Any object that supports NumPy slicing, like `dset`\n", 202 | "2. `chunks`: A chunk size to tell us how to block up our array, like `(1000000,)`" 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": null, 208 | "metadata": {}, 209 | "outputs": [], 210 | "source": [ 211 | "import dask.array as da\n", 212 | "x = da.from_array(dset, chunks=(1000000,))" 213 | ] 214 | }, 215 | { 216 | "cell_type": "markdown", 217 | "metadata": {}, 218 | "source": [ 219 | "** Manipulate `dask.array` object as you would a numpy array**" 220 | ] 221 | }, 222 | { 223 | "cell_type": "markdown", 224 | "metadata": {}, 225 | "source": [ 226 | "Now that we have an `Array` we perform standard numpy-style computations like arithmetic, mathematics, slicing, reductions, etc..\n", 227 | "\n", 228 | "The interface is familiar, but the actual work is different. dask_array.sum() does not do the same thing as numpy_array.sum()." 229 | ] 230 | }, 231 | { 232 | "cell_type": "markdown", 233 | "metadata": {}, 234 | "source": [ 235 | "**What's the difference?**" 236 | ] 237 | }, 238 | { 239 | "cell_type": "markdown", 240 | "metadata": {}, 241 | "source": [ 242 | "`dask_array.sum()` builds an expression of the computation. It does not do the computation yet. `numpy_array.sum()` computes the sum immediately." 243 | ] 244 | }, 245 | { 246 | "cell_type": "markdown", 247 | "metadata": {}, 248 | "source": [ 249 | "*Why the difference?*" 250 | ] 251 | }, 252 | { 253 | "cell_type": "markdown", 254 | "metadata": {}, 255 | "source": [ 256 | "Dask arrays are split into chunks. Each chunk must have computations run on that chunk explicitly. If the desired answer comes from a small slice of the entire dataset, running the computation over all data would be wasteful of CPU and memory." 257 | ] 258 | }, 259 | { 260 | "cell_type": "code", 261 | "execution_count": null, 262 | "metadata": {}, 263 | "outputs": [], 264 | "source": [ 265 | "result = x.sum()\n", 266 | "result" 267 | ] 268 | }, 269 | { 270 | "cell_type": "markdown", 271 | "metadata": {}, 272 | "source": [ 273 | "**Compute result**" 274 | ] 275 | }, 276 | { 277 | "cell_type": "markdown", 278 | "metadata": {}, 279 | "source": [ 280 | "Dask.array objects are lazily evaluated. Operations like `.sum` build up a graph of blocked tasks to execute. \n", 281 | "\n", 282 | "We ask for the final result with a call to `.compute()`. This triggers the actual computation." 283 | ] 284 | }, 285 | { 286 | "cell_type": "code", 287 | "execution_count": null, 288 | "metadata": {}, 289 | "outputs": [], 290 | "source": [ 291 | "result.compute()" 292 | ] 293 | }, 294 | { 295 | "cell_type": "markdown", 296 | "metadata": {}, 297 | "source": [ 298 | "### Exercise: Compute the mean" 299 | ] 300 | }, 301 | { 302 | "cell_type": "markdown", 303 | "metadata": {}, 304 | "source": [ 305 | "And the variance, std, etc.. This should be a trivial change to the example above.\n", 306 | "\n", 307 | "Look at what other operations you can do with the Jupyter notebook's tab-completion." 308 | ] 309 | }, 310 | { 311 | "cell_type": "code", 312 | "execution_count": null, 313 | "metadata": {}, 314 | "outputs": [], 315 | "source": [] 316 | }, 317 | { 318 | "cell_type": "markdown", 319 | "metadata": {}, 320 | "source": [ 321 | "Does this match your result from before?" 322 | ] 323 | }, 324 | { 325 | "cell_type": "markdown", 326 | "metadata": {}, 327 | "source": [ 328 | "Performance and Parallelism\n", 329 | "-------------------------------\n", 330 | "\n", 331 | "\n", 332 | "\n", 333 | "In our first examples we used `for` loops to walk through the array one block at a time. For simple operations like `sum` this is optimal. However for complex operations we may want to traverse through the array differently. In particular we may want the following:\n", 334 | "\n", 335 | "1. Use multiple cores in parallel\n", 336 | "2. Chain operations on a single blocks before moving on to the next one\n", 337 | "\n", 338 | "Dask.array translates your array operations into a graph of inter-related tasks with data dependencies between them. Dask then executes this graph in parallel with multiple threads. We'll discuss more about this in the next section.\n", 339 | "\n" 340 | ] 341 | }, 342 | { 343 | "cell_type": "markdown", 344 | "metadata": {}, 345 | "source": [ 346 | "### Example" 347 | ] 348 | }, 349 | { 350 | "cell_type": "markdown", 351 | "metadata": {}, 352 | "source": [ 353 | "1. Construct a 20000x20000 array of normally distributed random values broken up into 1000x1000 sized chunks\n", 354 | "2. Take the mean along one axis\n", 355 | "3. Take every 100th element" 356 | ] 357 | }, 358 | { 359 | "cell_type": "code", 360 | "execution_count": null, 361 | "metadata": {}, 362 | "outputs": [], 363 | "source": [ 364 | "import numpy as np\n", 365 | "import dask.array as da\n", 366 | "\n", 367 | "x = da.random.normal(10, 0.1, size=(200000, 200000), # 400 million element array \n", 368 | " chunks=(10000, 10000)) # Cut into 1000x1000 sized chunks\n", 369 | "y = x.mean(axis=0) # Perform NumPy-style operations" 370 | ] 371 | }, 372 | { 373 | "cell_type": "code", 374 | "execution_count": null, 375 | "metadata": {}, 376 | "outputs": [], 377 | "source": [ 378 | "x.numblocks" 379 | ] 380 | }, 381 | { 382 | "cell_type": "code", 383 | "execution_count": null, 384 | "metadata": {}, 385 | "outputs": [], 386 | "source": [ 387 | "x.nbytes / 1e9 # Gigabytes of the input processed lazily" 388 | ] 389 | }, 390 | { 391 | "cell_type": "code", 392 | "execution_count": null, 393 | "metadata": {}, 394 | "outputs": [], 395 | "source": [ 396 | "%%time\n", 397 | "y.compute() # Time to compute the result" 398 | ] 399 | } 400 | ], 401 | "metadata": { 402 | "anaconda-cloud": {}, 403 | "kernelspec": { 404 | "display_name": "Python 3", 405 | "language": "python", 406 | "name": "python3" 407 | }, 408 | "language_info": { 409 | "codemirror_mode": { 410 | "name": "ipython", 411 | "version": 3 412 | }, 413 | "file_extension": ".py", 414 | "mimetype": "text/x-python", 415 | "name": "python", 416 | "nbconvert_exporter": "python", 417 | "pygments_lexer": "ipython3", 418 | "version": "3.6.5" 419 | } 420 | }, 421 | "nbformat": 4, 422 | "nbformat_minor": 2 423 | } 424 | -------------------------------------------------------------------------------- /0_dask_tutorial/README.md: -------------------------------------------------------------------------------- 1 | ## Dask tutorial 2 | 3 | See the full tutorial material at: 4 | 5 | and the video recording of the 3 hours tutorial at Scipy 2018: 6 | -------------------------------------------------------------------------------- /0_dask_tutorial/prep_data.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | 3 | import os 4 | import numpy as np 5 | import pandas as pd 6 | import tarfile 7 | import urllib.request 8 | import zipfile 9 | from glob import glob 10 | 11 | data_dir = 'data' 12 | 13 | 14 | def flights(): 15 | flights_raw = os.path.join(data_dir, 'nycflights.tar.gz') 16 | flightdir = os.path.join(data_dir, 'nycflights') 17 | jsondir = os.path.join(data_dir, 'flightjson') 18 | 19 | if not os.path.exists(data_dir): 20 | os.mkdir(data_dir) 21 | 22 | if not os.path.exists(flights_raw): 23 | print("- Downloading NYC Flights dataset... ", end='', flush=True) 24 | url = "https://storage.googleapis.com/dask-tutorial-data/nycflights.tar.gz" 25 | urllib.request.urlretrieve(url, flights_raw) 26 | print("done", flush=True) 27 | 28 | if not os.path.exists(flightdir): 29 | print("- Extracting flight data... ", end='', flush=True) 30 | tar_path = os.path.join('data', 'nycflights.tar.gz') 31 | with tarfile.open(tar_path, mode='r:gz') as flights: 32 | flights.extractall('data/') 33 | print("done", flush=True) 34 | 35 | if not os.path.exists(jsondir): 36 | print("- Creating json data... ", end='', flush=True) 37 | os.mkdir(jsondir) 38 | for path in glob(os.path.join('data', 'nycflights', '*.csv')): 39 | prefix = os.path.splitext(os.path.basename(path))[0] 40 | # Just take the first 10000 rows for the demo 41 | df = pd.read_csv(path).iloc[:10000] 42 | df.to_json(os.path.join('data', 'flightjson', prefix + '.json'), 43 | orient='records', lines=True) 44 | print("done", flush=True) 45 | 46 | print("** Finished! **") 47 | 48 | 49 | def random_array(): 50 | if os.path.exists(os.path.join('data', 'random.hdf5')): 51 | return 52 | 53 | print("Create random data for array exercise") 54 | import h5py 55 | 56 | with h5py.File(os.path.join('data', 'random.hdf5')) as f: 57 | dset = f.create_dataset('/x', shape=(1000000000,), dtype='f4') 58 | for i in range(0, 1000000000, 1000000): 59 | dset[i: i + 1000000] = np.random.exponential(size=1000000) 60 | 61 | 62 | def weather(growth=3200): 63 | url = 'https://storage.googleapis.com/dask-tutorial-data/weather-small.zip' 64 | weather_zip = os.path.join('data', 'weather-small.zip') 65 | weather_small = os.path.join('data', 'weather-small') 66 | 67 | if not os.path.exists(weather_zip): 68 | print("Downloading weather data.") 69 | urllib.request.urlretrieve(url, weather_zip) 70 | 71 | if not os.path.exists(weather_small): 72 | print("Extracting to {}".format(weather_small)) 73 | zf = zipfile.ZipFile(weather_zip) 74 | zf.extractall(data_dir) 75 | 76 | filenames = sorted(glob(os.path.join('data', 'weather-small', '*.hdf5'))) 77 | 78 | if not os.path.exists(os.path.join('data', 'weather-big')): 79 | os.mkdir(os.path.join('data', 'weather-big')) 80 | 81 | if all(os.path.exists(fn.replace('small', 'big')) for fn in filenames): 82 | return 83 | 84 | from skimage.transform import resize 85 | import h5py 86 | 87 | for fn in filenames: 88 | with h5py.File(fn, mode='r') as f: 89 | x = f['/t2m'][:] 90 | 91 | new_shape = tuple(s * growth // 100 for s in x.shape) 92 | 93 | y = resize(x, new_shape, mode='constant') 94 | 95 | out_fn = os.path.join('data', 'weather-big', os.path.split(fn)[-1]) 96 | 97 | try: 98 | with h5py.File(out_fn) as f: 99 | f.create_dataset('/t2m', data=y, chunks=(500, 500)) 100 | except: 101 | pass 102 | 103 | 104 | def main(): 105 | print("Setting up data directory") 106 | print("-------------------------") 107 | 108 | flights() 109 | random_array() 110 | weather() 111 | 112 | print('Finished!') 113 | 114 | 115 | if __name__ == '__main__': 116 | main() 117 | -------------------------------------------------------------------------------- /0_dask_tutorial/solutions/01-delayed-control-flow.py: -------------------------------------------------------------------------------- 1 | results = [] 2 | for x in data: 3 | if is_even(x): # even 4 | y = delayed(double)(x) 5 | else: # odd 6 | y = delayed(inc)(x) 7 | results.append(y) 8 | 9 | total = delayed(sum)(results) -------------------------------------------------------------------------------- /0_dask_tutorial/solutions/01-delayed-loop.py: -------------------------------------------------------------------------------- 1 | results = [] 2 | 3 | for x in data: 4 | y = delayed(inc)(x) 5 | results.append(y) 6 | 7 | total = delayed(sum)(results) 8 | print("Before computing:", total) # Let's see what type of thing total is 9 | result = total.compute() 10 | print("After computing :", result) # After it's computed -------------------------------------------------------------------------------- /0_dask_tutorial/solutions/02-dask-arrays-blocked-mean.py: -------------------------------------------------------------------------------- 1 | sums = [] 2 | lengths = [] 3 | for i in range(0, 1000000000, 1000000): 4 | chunk = dset[i: i + 1000000] # pull out numpy array 5 | sums.append(chunk.sum()) 6 | lengths.append(len(chunk)) 7 | 8 | total = sum(sums) 9 | length = sum(lengths) 10 | print(total / length) -------------------------------------------------------------------------------- /0_dask_tutorial/solutions/02-dask-arrays-make-arrays.py: -------------------------------------------------------------------------------- 1 | arrays = [da.from_array(dset, chunks=(500, 500)) for dset in dsets] 2 | arrays -------------------------------------------------------------------------------- /0_dask_tutorial/solutions/02-dask-arrays-stacked.py: -------------------------------------------------------------------------------- 1 | x = da.stack(arrays, axis=0) 2 | x -------------------------------------------------------------------------------- /0_dask_tutorial/solutions/02-dask-arrays-store.py: -------------------------------------------------------------------------------- 1 | result = x[:, ::2, ::2] 2 | da.to_hdf5(os.path.join('data', 'myfile.hdf5'), '/output', result) -------------------------------------------------------------------------------- /0_dask_tutorial/solutions/02-dask-arrays-weather-difference.py: -------------------------------------------------------------------------------- 1 | result = x[0] - x.mean(axis=0) 2 | fig = plt.figure(figsize=(16, 8)) 3 | plt.imshow(result, cmap='RdBu_r') -------------------------------------------------------------------------------- /0_dask_tutorial/solutions/02-dask-arrays-weather-mean.py: -------------------------------------------------------------------------------- 1 | result = x.mean(axis=0) 2 | fig = plt.figure(figsize=(16, 8)) 3 | plt.imshow(result, cmap='RdBu_r') -------------------------------------------------------------------------------- /0_dask_tutorial/static/fail-case.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zonca/python_hpc_tutorial/7e76e59e8c48b1ef5b2f2e232c0c5b18009a6fc4/0_dask_tutorial/static/fail-case.gif -------------------------------------------------------------------------------- /0_dask_tutorial/static/ml-dimensions-color.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zonca/python_hpc_tutorial/7e76e59e8c48b1ef5b2f2e232c0c5b18009a6fc4/0_dask_tutorial/static/ml-dimensions-color.png -------------------------------------------------------------------------------- /0_dask_tutorial/static/ml-dimensions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zonca/python_hpc_tutorial/7e76e59e8c48b1ef5b2f2e232c0c5b18009a6fc4/0_dask_tutorial/static/ml-dimensions.png -------------------------------------------------------------------------------- /0_dask_tutorial/static/sklearn-parallel-dask.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zonca/python_hpc_tutorial/7e76e59e8c48b1ef5b2f2e232c0c5b18009a6fc4/0_dask_tutorial/static/sklearn-parallel-dask.png -------------------------------------------------------------------------------- /0_dask_tutorial/static/sklearn-parallel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zonca/python_hpc_tutorial/7e76e59e8c48b1ef5b2f2e232c0c5b18009a6fc4/0_dask_tutorial/static/sklearn-parallel.png -------------------------------------------------------------------------------- /1_dask_array/1_multicore_array.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Array Multicore\n", 8 | "\n", 9 | "Instead of trivially parallel independent tasks here we want to use multiple threads to process simultaneously different parts of the same array. `dask` automatically provides this feature by replacing the `numpy` function with `dask` functions. The key concept is a chunk, each chunk of data is executed separately by different threads. For example for a matrix we define a 2D block size and each of those blocks can be executed independently and then the results accumulated to get to the final answer. See " 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "metadata": { 16 | "collapsed": true 17 | }, 18 | "outputs": [], 19 | "source": [ 20 | "import numpy as np\n", 21 | "import dask.array as da" 22 | ] 23 | }, 24 | { 25 | "cell_type": "code", 26 | "execution_count": null, 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "np.__config__.show()" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": { 37 | "collapsed": true 38 | }, 39 | "outputs": [], 40 | "source": [ 41 | "A = np.random.rand(20000,40000)" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": null, 47 | "metadata": {}, 48 | "outputs": [], 49 | "source": [ 50 | "A.size / 1024**3" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": null, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "%time B = A**2 + np.sin(A) * A * np.log(A)" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "from numba import njit" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "def compute_B(A):\n", 78 | " return A**2 + np.sin(A) * A * np.log(A)" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "%time B = njit(compute_B, parallel=True)(A)" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": { 94 | "collapsed": true 95 | }, 96 | "outputs": [], 97 | "source": [ 98 | "A_dask = da.from_array(A, chunks=(5000, 5000))" 99 | ] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "execution_count": null, 104 | "metadata": {}, 105 | "outputs": [], 106 | "source": [ 107 | "A_dask.numblocks" 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": null, 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "%time B_dask = (A_dask**2 + da.sin(A_dask) * A_dask * da.log(A_dask)).compute()" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": null, 122 | "metadata": { 123 | "collapsed": true 124 | }, 125 | "outputs": [], 126 | "source": [ 127 | "assert np.allclose(B, B_dask)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": {}, 133 | "source": [ 134 | "Just last year this was an example of single node parallelization with threads, now this is not necessary, a lot of `numpy` internal functions are now already parallelized, so check that first!" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": null, 140 | "metadata": {}, 141 | "outputs": [], 142 | "source": [] 143 | } 144 | ], 145 | "metadata": { 146 | "anaconda-cloud": {}, 147 | "kernelspec": { 148 | "display_name": "Python 3", 149 | "language": "python", 150 | "name": "python3" 151 | }, 152 | "language_info": { 153 | "codemirror_mode": { 154 | "name": "ipython", 155 | "version": 3 156 | }, 157 | "file_extension": ".py", 158 | "mimetype": "text/x-python", 159 | "name": "python", 160 | "nbconvert_exporter": "python", 161 | "pygments_lexer": "ipython3", 162 | "version": "3.7.3" 163 | } 164 | }, 165 | "nbformat": 4, 166 | "nbformat_minor": 2 167 | } 168 | -------------------------------------------------------------------------------- /1_dask_array/2_multicore_array_outofcore.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Array multicore outofcore\n", 8 | "\n", 9 | "Outofcore computing refers to processing data larger than memory by reading it in chunks from disk or creating chunks on the fly.\n", 10 | "\n", 11 | "Dask supports it natively, in this example we are still processing an array with multiple threads but the array would never fit in memory, we are creating chunks on the fly and processing them and then free the memory." 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": null, 17 | "metadata": { 18 | "collapsed": true 19 | }, 20 | "outputs": [], 21 | "source": [ 22 | "import numpy as np\n", 23 | "import dask.array as da" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": null, 29 | "metadata": { 30 | "collapsed": true 31 | }, 32 | "outputs": [], 33 | "source": [ 34 | "# AA = da.random.normal(0,1,size=(200000,100000), chunks=(1000, 1000))" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": { 41 | "collapsed": true 42 | }, 43 | "outputs": [], 44 | "source": [ 45 | "AA = da.ones((200000,100000), chunks=(10000, 10000))" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "AA.numblocks" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "AA.nbytes / 1024**3" 64 | ] 65 | }, 66 | { 67 | "cell_type": "code", 68 | "execution_count": null, 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "%time AA.sum().compute()" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "%time AA.sum().compute(num_workers=4)" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": { 88 | "collapsed": true 89 | }, 90 | "outputs": [], 91 | "source": [ 92 | "# Don't do this!\n", 93 | "# Just know you can save to disk data that does NOT fit in memory\n", 94 | "# AA.to_hdf5(\"AA.h5\", \"/data\"" 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "metadata": { 101 | "collapsed": true 102 | }, 103 | "outputs": [], 104 | "source": [] 105 | } 106 | ], 107 | "metadata": { 108 | "anaconda-cloud": {}, 109 | "kernelspec": { 110 | "display_name": "Python 3", 111 | "language": "python", 112 | "name": "python3" 113 | }, 114 | "language_info": { 115 | "codemirror_mode": { 116 | "name": "ipython", 117 | "version": 3 118 | }, 119 | "file_extension": ".py", 120 | "mimetype": "text/x-python", 121 | "name": "python", 122 | "nbconvert_exporter": "python", 123 | "pygments_lexer": "ipython3", 124 | "version": "3.6.5" 125 | } 126 | }, 127 | "nbformat": 4, 128 | "nbformat_minor": 2 129 | } 130 | -------------------------------------------------------------------------------- /1_dask_array/3_multinode_distributed_array.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Array multinode" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": null, 13 | "metadata": { 14 | "collapsed": true 15 | }, 16 | "outputs": [], 17 | "source": [ 18 | "import numpy as np\n", 19 | "import dask.array as da" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": { 26 | "collapsed": true 27 | }, 28 | "outputs": [], 29 | "source": [ 30 | "# AA = da.random.normal(0,1,size=(200000,100000), chunks=(1000, 1000))" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "address = !hostname\n", 40 | "address = address[0]\n", 41 | "print(address)" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": null, 47 | "metadata": {}, 48 | "outputs": [], 49 | "source": [ 50 | "from distributed import Client\n", 51 | "client = Client(f'{address}:8786')" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": null, 57 | "metadata": {}, 58 | "outputs": [], 59 | "source": [ 60 | "print(f\"Connect using your browser to: \\n \\n http://{address}:8787/status\")" 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "metadata": {}, 66 | "source": [ 67 | "## Launch a dask distributed cluster\n", 68 | "\n", 69 | "The easiest way is to use `dask-jobqueue`, see its configuration for Comet: https://jobqueue.dask.org/en/latest/configurations.html#sdsc-comet\n", 70 | "\n", 71 | "However we are running inside a Singularity container which has no access to SLURM commands, therefore we manually launch the schedule in the Jupyterlab terminal with `launch_scheduler.sh` and submit a job for the workers using a terminal on the Comet login node." 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": null, 77 | "metadata": { 78 | "collapsed": true 79 | }, 80 | "outputs": [], 81 | "source": [ 82 | "AA = da.ones((200000,200000), chunks=(10000, 10000))" 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": null, 88 | "metadata": {}, 89 | "outputs": [], 90 | "source": [ 91 | "AA.numblocks" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": null, 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "AA.nbytes / 1024**3" 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": null, 106 | "metadata": {}, 107 | "outputs": [], 108 | "source": [ 109 | "%time AA.sum().compute()" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": null, 115 | "metadata": { 116 | "collapsed": true 117 | }, 118 | "outputs": [], 119 | "source": [] 120 | } 121 | ], 122 | "metadata": { 123 | "anaconda-cloud": {}, 124 | "kernelspec": { 125 | "display_name": "Python 3", 126 | "language": "python", 127 | "name": "python3" 128 | }, 129 | "language_info": { 130 | "codemirror_mode": { 131 | "name": "ipython", 132 | "version": 3 133 | }, 134 | "file_extension": ".py", 135 | "mimetype": "text/x-python", 136 | "name": "python", 137 | "nbconvert_exporter": "python", 138 | "pygments_lexer": "ipython3", 139 | "version": "3.6.5" 140 | } 141 | }, 142 | "nbformat": 4, 143 | "nbformat_minor": 2 144 | } 145 | -------------------------------------------------------------------------------- /1_dask_array/4_dask_graphs.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "import dask.array as da" 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "The key element of `dask` is the scheduler which builds a Direct Acyclic Graph of all the operations to be executed on each chunk of data to compute the final result." 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": true 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "x = da.ones(15, chunks=(5,))" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "`visualize()` calls `graphviz` to create a graphical representation of the graph" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "x.visualize()" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "(x+1).visualize()" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "(x+1).sum().visualize()" 64 | ] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "metadata": {}, 69 | "source": [ 70 | "Let's try with a more complex example." 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": null, 76 | "metadata": { 77 | "collapsed": true 78 | }, 79 | "outputs": [], 80 | "source": [ 81 | "m = da.ones((15, 15), chunks=(5,5))" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "m.numblocks" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": {}, 97 | "outputs": [], 98 | "source": [ 99 | "(m.T + 1).visualize()" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": null, 105 | "metadata": {}, 106 | "outputs": [], 107 | "source": [ 108 | "(m.T + m).visualize()" 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "metadata": {}, 115 | "outputs": [], 116 | "source": [ 117 | "(m.dot(m.T + 1) - m.mean(axis=0)).visualize()" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "(m.dot(m.T + 1) - m.mean(axis=0)).compute()" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": { 133 | "collapsed": true 134 | }, 135 | "outputs": [], 136 | "source": [] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": null, 141 | "metadata": {}, 142 | "outputs": [], 143 | "source": [] 144 | } 145 | ], 146 | "metadata": { 147 | "kernelspec": { 148 | "display_name": "Python 3", 149 | "language": "python", 150 | "name": "python3" 151 | }, 152 | "language_info": { 153 | "codemirror_mode": { 154 | "name": "ipython", 155 | "version": 3 156 | }, 157 | "file_extension": ".py", 158 | "mimetype": "text/x-python", 159 | "name": "python", 160 | "nbconvert_exporter": "python", 161 | "pygments_lexer": "ipython3", 162 | "version": "3.6.5" 163 | } 164 | }, 165 | "nbformat": 4, 166 | "nbformat_minor": 2 167 | } 168 | -------------------------------------------------------------------------------- /1_dask_array/dask.delayed.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "from time import sleep" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 2, 16 | "metadata": {}, 17 | "outputs": [ 18 | { 19 | "name": "stderr", 20 | "output_type": "stream", 21 | "text": [ 22 | "/opt/conda/lib/python3.7/site-packages/dask/config.py:168: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n", 23 | " data = yaml.load(f.read()) or {}\n" 24 | ] 25 | } 26 | ], 27 | "source": [ 28 | "import dask" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": 3, 34 | "metadata": {}, 35 | "outputs": [], 36 | "source": [ 37 | "def read_data(filename):\n", 38 | " print(\"Reading file\", filename)\n", 39 | " for progress in range(0, 100, 25):\n", 40 | " print(f\"Read {progress}%\")\n", 41 | " sleep(.5)\n", 42 | " print(\"Reading completed\")" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 4, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "def perform_computation():\n", 52 | " print(\"Computation started\")\n", 53 | " for progress in range(0, 100, 25):\n", 54 | " print(f\"Computed {progress}%\")\n", 55 | " sleep(.5)\n", 56 | " print(\"Computation completed\")" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": 5, 62 | "metadata": {}, 63 | "outputs": [ 64 | { 65 | "name": "stdout", 66 | "output_type": "stream", 67 | "text": [ 68 | "Reading file a.fits\n", 69 | "Read 0%\n", 70 | "Read 25%\n", 71 | "Read 50%\n", 72 | "Read 75%\n", 73 | "Reading completed\n", 74 | "CPU times: user 2 ms, sys: 1 ms, total: 3 ms\n", 75 | "Wall time: 2 s\n" 76 | ] 77 | } 78 | ], 79 | "source": [ 80 | "%time read_data(\"a.fits\")" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 6, 86 | "metadata": {}, 87 | "outputs": [ 88 | { 89 | "name": "stdout", 90 | "output_type": "stream", 91 | "text": [ 92 | "Computation started\n", 93 | "Computed 0%\n", 94 | "Computed 25%\n", 95 | "Computed 50%\n", 96 | "Computed 75%\n", 97 | "Computation completed\n", 98 | "CPU times: user 4 ms, sys: 0 ns, total: 4 ms\n", 99 | "Wall time: 2 s\n" 100 | ] 101 | } 102 | ], 103 | "source": [ 104 | "%time perform_computation()" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 7, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "t1 = dask.delayed(read_data)(\"a.fits\")\n", 114 | "t2 = dask.delayed(perform_computation)()" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": 8, 120 | "metadata": {}, 121 | "outputs": [], 122 | "source": [ 123 | "import dask.multiprocessing" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 12, 129 | "metadata": {}, 130 | "outputs": [ 131 | { 132 | "name": "stdout", 133 | "output_type": "stream", 134 | "text": [ 135 | "Computation startedReading file a.fits\n", 136 | "Read 0%\n", 137 | "\n", 138 | "Computed 0%\n", 139 | "Read 25%\n", 140 | "Computed 25%\n", 141 | "Read 50%\n", 142 | "Computed 50%\n", 143 | "Read 75%\n", 144 | "Computed 75%\n", 145 | "Reading completed\n", 146 | "Computation completed\n", 147 | "CPU times: user 7 ms, sys: 5 ms, total: 12 ms\n", 148 | "Wall time: 2.01 s\n" 149 | ] 150 | }, 151 | { 152 | "data": { 153 | "text/plain": [ 154 | "(None, None)" 155 | ] 156 | }, 157 | "execution_count": 12, 158 | "metadata": {}, 159 | "output_type": "execute_result" 160 | } 161 | ], 162 | "source": [ 163 | "%%time\n", 164 | "\n", 165 | "dask.compute(t1, t2, scheduler=\"threading\")" 166 | ] 167 | }, 168 | { 169 | "cell_type": "markdown", 170 | "metadata": {}, 171 | "source": [ 172 | "Repeat many times, see if you can get different orders of execution" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": 13, 178 | "metadata": {}, 179 | "outputs": [ 180 | { 181 | "ename": "TypeError", 182 | "evalue": "The get= keyword has been removed.\n\nPlease use the scheduler= keyword instead with the name of\nthe desired scheduler like 'threads' or 'processes'\n\n x.compute(scheduler='single-threaded')\n x.compute(scheduler='threads')\n x.compute(scheduler='processes')\n\nor with a function that takes the graph and keys\n\n x.compute(scheduler=my_scheduler_function)\n\nor with a Dask client\n\n x.compute(scheduler=client)", 183 | "output_type": "error", 184 | "traceback": [ 185 | "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", 186 | "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", 187 | "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n", 188 | "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/dask/base.py\u001b[0m in \u001b[0;36mcompute\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 391\u001b[0m schedule = get_scheduler(scheduler=kwargs.pop('scheduler', None),\n\u001b[1;32m 392\u001b[0m \u001b[0mcollections\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcollections\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 393\u001b[0;31m get=kwargs.pop('get', None))\n\u001b[0m\u001b[1;32m 394\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 395\u001b[0m \u001b[0mdsk\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcollections_to_dsk\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcollections\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimize_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 189 | "\u001b[0;32m/opt/conda/lib/python3.7/site-packages/dask/base.py\u001b[0m in \u001b[0;36mget_scheduler\u001b[0;34m(get, scheduler, collections, cls)\u001b[0m\n\u001b[1;32m 861\u001b[0m \"\"\"\n\u001b[1;32m 862\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mget\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 863\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mget_err_msg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 864\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 865\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mscheduler\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", 190 | "\u001b[0;31mTypeError\u001b[0m: The get= keyword has been removed.\n\nPlease use the scheduler= keyword instead with the name of\nthe desired scheduler like 'threads' or 'processes'\n\n x.compute(scheduler='single-threaded')\n x.compute(scheduler='threads')\n x.compute(scheduler='processes')\n\nor with a function that takes the graph and keys\n\n x.compute(scheduler=my_scheduler_function)\n\nor with a Dask client\n\n x.compute(scheduler=client)" 191 | ] 192 | } 193 | ], 194 | "source": [ 195 | "%%time\n", 196 | "\n", 197 | "dask.compute(t1, t2, scheduler=\"processes\")" 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": null, 203 | "metadata": {}, 204 | "outputs": [], 205 | "source": [] 206 | } 207 | ], 208 | "metadata": { 209 | "kernelspec": { 210 | "display_name": "Python 3", 211 | "language": "python", 212 | "name": "python3" 213 | }, 214 | "language_info": { 215 | "codemirror_mode": { 216 | "name": "ipython", 217 | "version": 3 218 | }, 219 | "file_extension": ".py", 220 | "mimetype": "text/x-python", 221 | "name": "python", 222 | "nbconvert_exporter": "python", 223 | "pygments_lexer": "ipython3", 224 | "version": "3.7.3" 225 | } 226 | }, 227 | "nbformat": 4, 228 | "nbformat_minor": 2 229 | } 230 | -------------------------------------------------------------------------------- /1_dask_array/mydask.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zonca/python_hpc_tutorial/7e76e59e8c48b1ef5b2f2e232c0c5b18009a6fc4/1_dask_array/mydask.png -------------------------------------------------------------------------------- /1_dask_array/processes.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "from time import sleep" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "import multiprocessing" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": {}, 26 | "outputs": [], 27 | "source": [ 28 | "def read_data(filename):\n", 29 | " print(\"Reading file\", filename)\n", 30 | " for progress in range(0, 100, 25):\n", 31 | " print(f\"Read {progress}%\")\n", 32 | " sleep(.5)\n", 33 | " print(\"Reading completed\")" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "def perform_computation():\n", 43 | " print(\"Computation started\")\n", 44 | " for progress in range(0, 100, 25):\n", 45 | " print(f\"Computed {progress}%\")\n", 46 | " sleep(.5)\n", 47 | " print(\"Computation completed\")" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "metadata": {}, 54 | "outputs": [], 55 | "source": [ 56 | "%time read_data(\"a.fits\")" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "%time perform_computation()" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": null, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "t1 = multiprocessing.Process(target=read_data, args=(\"a.fits\",))\n", 75 | "t2 = multiprocessing.Process(target=perform_computation)" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": null, 81 | "metadata": {}, 82 | "outputs": [], 83 | "source": [ 84 | "%%time\n", 85 | "\n", 86 | "t1.start()\n", 87 | "t2.start()\n", 88 | "\n", 89 | "t1.join()\n", 90 | "t2.join()" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "metadata": {}, 97 | "outputs": [], 98 | "source": [] 99 | } 100 | ], 101 | "metadata": { 102 | "kernelspec": { 103 | "display_name": "Python 3", 104 | "language": "python", 105 | "name": "python3" 106 | }, 107 | "language_info": { 108 | "codemirror_mode": { 109 | "name": "ipython", 110 | "version": 3 111 | }, 112 | "file_extension": ".py", 113 | "mimetype": "text/x-python", 114 | "name": "python", 115 | "nbconvert_exporter": "python", 116 | "pygments_lexer": "ipython3", 117 | "version": "3.6.5" 118 | } 119 | }, 120 | "nbformat": 4, 121 | "nbformat_minor": 2 122 | } 123 | -------------------------------------------------------------------------------- /1_dask_array/threads.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import numpy as np\n", 10 | "from time import sleep" 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": null, 16 | "metadata": {}, 17 | "outputs": [], 18 | "source": [ 19 | "import threading" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": {}, 26 | "outputs": [], 27 | "source": [ 28 | "def read_data(filename):\n", 29 | " print(\"Reading file\", filename)\n", 30 | " for progress in range(0, 100, 25):\n", 31 | " print(f\"Read {progress}%\")\n", 32 | " sleep(.5)\n", 33 | " print(\"Reading completed\")" 34 | ] 35 | }, 36 | { 37 | "cell_type": "code", 38 | "execution_count": null, 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "def perform_computation():\n", 43 | " print(\"Computation started\")\n", 44 | " for progress in range(0, 100, 25):\n", 45 | " print(f\"Computed {progress}%\")\n", 46 | " sleep(.5)\n", 47 | " print(\"Computation completed\")" 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "metadata": {}, 54 | "outputs": [], 55 | "source": [ 56 | "%time read_data(\"a.fits\")" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "%time perform_computation()" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": null, 71 | "metadata": {}, 72 | "outputs": [], 73 | "source": [ 74 | "t1 = threading.Thread(target=read_data, args=(\"a.fits\",))\n", 75 | "t2 = threading.Thread(target=perform_computation)" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": null, 81 | "metadata": {}, 82 | "outputs": [], 83 | "source": [ 84 | "%%time\n", 85 | "\n", 86 | "t1.start()\n", 87 | "t2.start()\n", 88 | "\n", 89 | "t1.join()\n", 90 | "t2.join()" 91 | ] 92 | } 93 | ], 94 | "metadata": { 95 | "kernelspec": { 96 | "display_name": "Python 3", 97 | "language": "python", 98 | "name": "python3" 99 | }, 100 | "language_info": { 101 | "codemirror_mode": { 102 | "name": "ipython", 103 | "version": 3 104 | }, 105 | "file_extension": ".py", 106 | "mimetype": "text/x-python", 107 | "name": "python", 108 | "nbconvert_exporter": "python", 109 | "pygments_lexer": "ipython3", 110 | "version": "3.6.5" 111 | } 112 | }, 113 | "nbformat": 4, 114 | "nbformat_minor": 2 115 | } 116 | -------------------------------------------------------------------------------- /2_digits_of_pi/1_serial_digits_of_pi.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import sys 3 | import numpy as np 4 | 5 | def inside_circle(total_count): 6 | 7 | x = np.random.uniform(size=total_count) 8 | y = np.random.uniform(size=total_count) 9 | 10 | radii_square = x**2 + y**2 11 | 12 | count = (radii_square<=1.0).sum() 13 | 14 | return count 15 | 16 | def estimate_pi(n_samples): 17 | 18 | return (4.0 * inside_circle(n_samples) / n_samples) 19 | 20 | if __name__=='__main__': 21 | 22 | n_samples = 10000 23 | if len(sys.argv) > 1: 24 | n_samples = int(sys.argv[1]) 25 | 26 | my_pi = estimate_pi(n_samples) 27 | sizeof = np.dtype(np.float64).itemsize 28 | 29 | print("required memory {:.3f} MB".format(n_samples*sizeof*3/(1024*1024))) 30 | print("pi is {} from {} samples".format(my_pi,n_samples)) 31 | print("error is {:.3e}".format(abs(my_pi - np.pi))) 32 | -------------------------------------------------------------------------------- /2_digits_of_pi/2_multicore_digits_of_pi.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import sys 3 | import numpy as np 4 | 5 | def inside_circle(total_count): 6 | 7 | x = np.random.uniform(size=total_count) 8 | y = np.random.uniform(size=total_count) 9 | 10 | radii_square = x**2 + y**2 11 | 12 | count = (radii_square<=1.0).sum() 13 | 14 | return count 15 | 16 | def estimate_pi(n_samples, executor): 17 | 18 | n_workers = executor._max_workers 19 | samples_per_worker, remainder = divmod(n_samples, n_workers) 20 | samples_per_worker_array = np.array([samples_per_worker] * n_workers) 21 | samples_per_worker_array[:remainder] += 1 22 | n_samples_inside = np.sum(list(executor.map(inside_circle, samples_per_worker_array))) 23 | 24 | return (4.0 * n_samples_inside / n_samples) 25 | 26 | if __name__=='__main__': 27 | 28 | n_samples = 10000 29 | n_workers = None 30 | if len(sys.argv) > 1: 31 | n_samples = int(sys.argv[1]) 32 | n_workers = int(sys.argv[2]) 33 | sizeof = np.dtype(np.float64).itemsize 34 | print("required memory {:.3f} MB".format(n_samples*sizeof*3/(1024*1024))) 35 | 36 | 37 | import concurrent.futures 38 | 39 | executor = concurrent.futures.ProcessPoolExecutor(max_workers = n_workers) 40 | 41 | my_pi = estimate_pi(n_samples, executor) 42 | 43 | print("pi is {} from {} samples".format(my_pi,n_samples)) 44 | print("error is {:.3e}".format(abs(my_pi - np.pi))) 45 | -------------------------------------------------------------------------------- /2_digits_of_pi/3_dask_multicore_digits_of_pi.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import sys 3 | import numpy as np 4 | import dask.array as da 5 | import dask 6 | 7 | def inside_circle(total_count): 8 | 9 | x = da.random.uniform(size=total_count, chunks=total_count//12) 10 | y = da.random.uniform(size=total_count, chunks=total_count//12) 11 | 12 | radii_square = x**2 + y**2 13 | 14 | count = (radii_square<=1.0).sum() 15 | 16 | return count 17 | 18 | def estimate_pi(n_samples): 19 | 20 | return (4.0 * inside_circle(n_samples) / n_samples) 21 | 22 | if __name__=='__main__': 23 | n_samples = 10000 24 | if len(sys.argv) > 1: 25 | n_samples = int(sys.argv[1]) 26 | 27 | my_pi = estimate_pi(n_samples).compute(num_workers=1) 28 | sizeof = np.dtype(np.float64).itemsize 29 | 30 | print("pi is {} from {} samples".format(my_pi,n_samples)) 31 | print("error is {:.3e}".format(abs(my_pi - np.pi))) 32 | -------------------------------------------------------------------------------- /2_digits_of_pi/4_dask_multinode_digits_of_pi.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import sys 3 | import numpy as np 4 | import dask.array as da 5 | 6 | def inside_circle(total_count): 7 | 8 | x = da.random.uniform(size=total_count, chunks=total_count//48) 9 | y = da.random.uniform(size=total_count, chunks=total_count//48) 10 | 11 | radii_square = x**2 + y**2 12 | 13 | count = (radii_square<=1.0).sum().compute() 14 | 15 | return count 16 | 17 | def estimate_pi(n_samples): 18 | 19 | return (4.0 * inside_circle(n_samples) / n_samples) 20 | 21 | if __name__=='__main__': 22 | 23 | n_samples = 10000 24 | if len(sys.argv) > 1: 25 | n_samples = int(sys.argv[1]) 26 | 27 | from distributed import Client 28 | import socket 29 | hostname = socket.gethostname() 30 | client = Client(f'{hostname}:8786') 31 | 32 | my_pi = estimate_pi(n_samples) 33 | sizeof = np.dtype(np.float64).itemsize 34 | 35 | print("pi is {} from {} samples".format(my_pi,n_samples)) 36 | print("error is {:.3e}".format(abs(my_pi - np.pi))) 37 | -------------------------------------------------------------------------------- /2_digits_of_pi/run_digits_of_pi_scripts.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "%ls *.py" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "metadata": { 16 | "collapsed": true 17 | }, 18 | "outputs": [], 19 | "source": [ 20 | "SAMPLES = int(1e9)" 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": null, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "%%time\n", 30 | "\n", 31 | "%run 1_serial_digits_of_pi.py $SAMPLES" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": null, 37 | "metadata": {}, 38 | "outputs": [], 39 | "source": [ 40 | "%%time\n", 41 | "\n", 42 | "%run 2_multicore_digits_of_pi.py $SAMPLES 12" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": null, 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "%%time\n", 52 | "\n", 53 | "%run 3_dask_multicore_digits_of_pi.py $SAMPLES" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "%%time\n", 63 | "\n", 64 | "%run 4_dask_multinode_digits_of_pi.py $SAMPLES" 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": {}, 71 | "outputs": [], 72 | "source": [] 73 | } 74 | ], 75 | "metadata": { 76 | "kernelspec": { 77 | "display_name": "Python 3", 78 | "language": "python", 79 | "name": "python3" 80 | }, 81 | "language_info": { 82 | "codemirror_mode": { 83 | "name": "ipython", 84 | "version": 3 85 | }, 86 | "file_extension": ".py", 87 | "mimetype": "text/x-python", 88 | "name": "python", 89 | "nbconvert_exporter": "python", 90 | "pygments_lexer": "ipython3", 91 | "version": "3.6.5" 92 | } 93 | }, 94 | "nbformat": 4, 95 | "nbformat_minor": 2 96 | } 97 | -------------------------------------------------------------------------------- /GIL_counter.py: -------------------------------------------------------------------------------- 1 | import threading 2 | import sys 3 | import time 4 | 5 | numSecondsToRun = 2 6 | 7 | class CounterThread(threading.Thread): 8 | def __init__(self): 9 | threading.Thread.__init__(self) 10 | self._counter = 0 11 | self._endTime = time.time() + numSecondsToRun 12 | 13 | def run(self): 14 | # Simulate a computation on the CPU 15 | while(time.time() < self._endTime): 16 | self._counter += 1 17 | 18 | if __name__ == "__main__": 19 | if len(sys.argv) < 2: 20 | print("Usage: python counter 5") 21 | sys.exit(5) 22 | 23 | numThreads = int(sys.argv[1]) 24 | print("Spawning %i counting threads for %i seconds..." % (numThreads, numSecondsToRun)) 25 | 26 | threads = [] 27 | for i in range(0,numThreads): 28 | t = CounterThread() 29 | t.start() 30 | threads.append(t) 31 | 32 | totalCounted = 0 33 | for t in threads: 34 | t.join() 35 | totalCounted += t._counter 36 | print("Total amount counted was {:.2f} millions".format(totalCounted/1e6)) 37 | print("Average count per thread {:.2f} millions".format(totalCounted/1e6/numThreads)) 38 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Webinar recording 2 | 3 | https://www.sdsc.edu/Events/training/webinars/distributed_parallel_computing_with_python_2019/recording/ 4 | 5 | # Python for HPC 6 | 7 | This webinar provides an introduction to distributed computing with Python, we will show how to modify a standard Python script to use multiple CPU cores using the concurrent.futures module from the Python standard library and then the dask package. Then we will leverage dask workers running on other machines to distribute a data processing task and monitor its execution through the live dask dashboard. You will understand the difference between threads and processes, how the Global Interpreter Lock works and principles of distributed computing. All material will be available as Jupyter Notebooks. 8 | 9 | * `notebook_singularity.slrm`: SLURM script to launch a Jupyter Notebook job through Singularity 10 | * `digits_of_pi/`: example of parallel programming estimating the digits of pi, using `concurrent.futures` and `dask` 11 | * `dask` scripts: `launch_workers.sh` and `launch_scheduler.sh` and `dask_workers.slrm` launch all the components of `dask` `distributed` 12 | * [Slides on Google Docs](https://docs.google.com/presentation/d/1hcgwy6S7QXVCIZHI0_Rb7_9FPHQP8HH6iWmaqCKpdIU/edit?usp=sharing) 13 | 14 | -------------------------------------------------------------------------------- /dask_slurm/dask_workers.slrm: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #SBATCH --job-name="dask-workers" 3 | #SBATCH --output="dask-workers.%j.%N.out" 4 | #SBATCH --partition=compute 5 | #SBATCH --nodes=2 6 | #SBATCH --ntasks-per-node=24 7 | #SBATCH --export=ALL 8 | #SBATCH -t 0:20:00 9 | 10 | module load singularity 11 | 12 | SINGULARITY_IMAGE="/oasis/scratch/comet/zonca/temp_project/zonca-singularity-comet-master-2019-05.simg" 13 | COMMAND="bash ./launch_worker.sh" 14 | export SINGULARITY_BINDPATH="/oasis" 15 | 16 | ibrun --npernode=1 singularity exec $SINGULARITY_IMAGE $COMMAND 17 | -------------------------------------------------------------------------------- /dask_slurm/launch_scheduler.sh: -------------------------------------------------------------------------------- 1 | echo "Launching dask scheduler" 2 | # fix for ubuntu container 3 | export LC_ALL=C.UTF-8 4 | export LANG=C.UTF-8 5 | dask-scheduler --scheduler-file ~/.dask_scheduler.json --host $(hostname) 6 | -------------------------------------------------------------------------------- /dask_slurm/launch_worker.sh: -------------------------------------------------------------------------------- 1 | echo "Launching dask worker" 2 | # fix for ubuntu container 3 | export LC_ALL=C.UTF-8 4 | export LANG=C.UTF-8 5 | MEM_GB=100 6 | # memory limit is in bytes 7 | MEM=$(( $MEM_GB*1024**3 )) 8 | /opt/conda/bin/dask-worker --scheduler-file ~/.dask_scheduler.json --memory-limit $MEM --nprocs 1 9 | -------------------------------------------------------------------------------- /intro.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [ 8 | { 9 | "name": "stdout", 10 | "output_type": "stream", 11 | "text": [ 12 | "Hello world\n" 13 | ] 14 | } 15 | ], 16 | "source": [ 17 | "print(\"Hello world\")" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 2, 23 | "metadata": { 24 | "collapsed": true 25 | }, 26 | "outputs": [], 27 | "source": [ 28 | "a = 3" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": 4, 34 | "metadata": {}, 35 | "outputs": [ 36 | { 37 | "data": { 38 | "text/plain": [ 39 | "9" 40 | ] 41 | }, 42 | "execution_count": 4, 43 | "metadata": {}, 44 | "output_type": "execute_result" 45 | } 46 | ], 47 | "source": [ 48 | "a ** 2\n", 49 | "\n" 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "# Python for HPC\n", 57 | "\n", 58 | "* dfsafds\n", 59 | "* fdsafdsaf\n", 60 | "\n", 61 | "$\\chi^2$" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 5, 67 | "metadata": { 68 | "collapsed": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "%matplotlib inline" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 6, 78 | "metadata": { 79 | "collapsed": true 80 | }, 81 | "outputs": [], 82 | "source": [ 83 | "import matplotlib.pyplot as plt" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 7, 89 | "metadata": {}, 90 | "outputs": [ 91 | { 92 | "data": { 93 | "text/plain": [ 94 | "[]" 95 | ] 96 | }, 97 | "execution_count": 7, 98 | "metadata": {}, 99 | "output_type": "execute_result" 100 | }, 101 | { 102 | "data": { 103 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XdYVHfaxvHvwwBiwQp2sUSNsRcETLLpxVTTNoItFsBs\nyibZvNnN5t1NNptsNpvsm94EWyxYNtEkm967IGDvPXbFhhUp87x/MNmLNSCjDBxm5vlcF5fDOb+Z\nuY9Hb4YzZ85PVBVjjDHBI8TpAMYYY2qWFb8xxgQZK35jjAkyVvzGGBNkrPiNMSbIWPEbY0yQseI3\nxpggY8VvjDFBxorfGGOCTKjTAcoTFRWlHTp0cDqGMcb4jdzc3H2qGu3N2FpZ/B06dCAnJ8fpGMYY\n4zdE5Cdvx9qhHmOMCTJW/MYYE2Ss+I0xJshY8RtjTJCx4jfGmCBTafGLSISILBSRpSKyUkQeL2eM\niMhLIrJBRJaJSP8y6waLyFrPuod9vQHGGGPOjDev+E8Cl6lqH6AvMFhEEk4Zcw3QxfOVCrwOICIu\n4FXP+u5Akoh091F2Y4wxZ6HS4tdSRz3fhnm+Tp2vcQgwzTM2E2gsIq2AOGCDqm5S1UJgtmesMcYh\nbrcyN3sb2w8edzqKcYhXx/hFxCUiS4C9wGeqmnXKkDbAtjLfb/csq2h5ec+RKiI5IpKTl5fnbX5j\nzBkocSsPvbWM37+9jNteX8DGvKOV38kEHK+KX1VLVLUv0BaIE5Gevg6iqmmqGquqsdHRXn3q2Bhz\nBopL3DwwZwlvL9rOHYPaU+x2M3RCJuv3HHE6mqlhZ3RWj6oeAr4CBp+yagfQrsz3bT3LKlpujKlB\nhcVu7p21mPeW7uTha7rx+JCezE5NIEQgMS2T1bsOOx3R1CBvzuqJFpHGntt1gSuBNacMew8Y5Tm7\nJwHIV9VdQDbQRUQ6ikg4kOgZa4ypISeLS7hrZi4frdjNn6/vzp0XnwNA5+aRzBk/iPDQEJLSM1mx\nI9/hpKamePOKvxXwlYgso7TIP1PV90XkThG50zPmQ2ATsAFIB+4CUNVi4B7gE2A1MFdVV/p4G4wx\nFSgoKiF1Wi6fr97LEzf1ZNyFHf9rfceo+swdP4j64aEkpWeyeOtBh5KamiSqp56g47zY2Fi1q3Ma\nUzUnCktImZbDDxv38fQtvRg6MKbCsTsOnSApLZMDxwqZOmYgsR2a1mBS4wsikquqsd6MtU/uGhOA\njp0sZvSUhfy4cR//vK3PaUsfoE3juswdP4jmkXUYNXkhCzbur6GkxglW/MYEmMMFRYyavJCcnw7y\nQmI/bh3Q1qv7tWwUwezxCbRpXJcxUxfy/fp91ZzUOMWK35gAkn+8iJETs1i67RCvJPXjxj6tz+j+\nzSMjmJ2aQIdm9Rn7ZjZfrdlbTUmNk6z4jQkQB48VMmxiJqt3HeGNEQO4plers3qcZg3qMCslga4t\nGpA6PYdPV+72cVLjNCt+YwLAvqMnSUrPZP3eo6SNGsAV3VtU6fGa1A9nZnICPVo34q6Zi/hw+S4f\nJTW1gRW/MX5u7+ECEtMy2bL/GFNGD+SSc5v75HEb1Q1j+rg4+rZrzL2zFvPuEvvsZaCw4jfGj+3K\nP8HQtEx2HjrB1DFxXNA5yqePHxkRxptj4xjYoQn3z1nCW7nbffr4xhlW/Mb4qe0HjzN0Qib7jpxk\n+rg4Ejo1q5bnqV8nlCmj47iwcxQPvbWUWQu3VsvzmJpjxW+MH9q6v7T0Dx0vZEZyPAPaV+8HruqG\nu0gfFcslXaP547zlTFuwpVqfz1QvK35j/MymvKPcPmEBxwqLyUhJoE+7xjXyvBFhLt4YOYAru7fg\n0XdXMvG7TTXyvMb3rPiN8SPr9xxhaFomRSVuZqcm0LNNoxp9/jqhLl4b3p9re7XkyQ9W89rXG2r0\n+Y1vhDodwBjjndW7DjNiYhYhIcLs1AS6tIh0JEeYK4SXEvsR5lrKMx+vpahY+e3lnRERR/KYM2fF\nb4wfWLEjnxGTsogIdZGREk+n6AaO5gl1hfDc7X0Jc4Xw/OfrKCpx8+BVXa38/YQVvzG13JJthxg1\nKYvIiDBmpSQQ06ye05EAcIUIz9zamzCX8MpXGygscfPHa7pZ+fsBK35jarHcnw5wx+RsmtYPJyMl\nnrZNakfp/ywkRPjbTb0Ic4WQ9u0mCovdPHZDdyv/Wq7S4heRdsA0oAWgQJqqvnjKmIeA4WUe8zwg\nWlUPiMgW4AhQAhR7e71oY4Jd5qb9jJ2aTcuGEcxMiadVo7pORypXSIjw+I09CHeFMPH7zRSWuHly\nSE9CQqz8aytvXvEXAw+q6iIRiQRyReQzVV318wBVfRZ4FkBEbgAeUNUDZR7jUlW1a7wa46Xv1+8j\neVo27ZrUY2ZyPM0bRjgd6bREhP+97jzCQ0N47euNFBW7efrW3ris/GulSovfM3fuLs/tIyKyGmgD\nrKrgLknALJ8lNCbIfL12L6nTc+kUVZ8ZyfFENajjdCSviAgPXX0uYa4QXvxiPUUlbv756z6Euuys\n8drmjI7xi0gHoB+QVcH6esBgSufZ/ZkCn4tICTBBVdPOKqkxQeDzVXu4a+YiurRowIxx8TSpH+50\npDMiIjxwZVfCQ0N49pO1FLmVF4aWnv1jag+vi19EGgBvA/er6uEKht0A/HDKYZ4LVXWHiDQHPhOR\nNar6bTmPnwqkAsTEnH6aOGMC0UfLd3HvrMX0aNOIaWPiaFQvzOlIZ+3uSzsT7grhbx+uprjEzctJ\n/QkPtfKvLbzaEyISRmnpz1TVeacZmsgph3lUdYfnz73AfCCuvDuqapqqxqpqbHR0tDexjAkY7y7Z\nwT2zFtOnXWNmjPPv0v9ZykWd+MsN3flk5R7unJFLQVGJ05GMR6XFL6XnZU0CVqvqc6cZ1wi4GHi3\nzLL6njeEEZH6wFXAiqqGNiaQvJW7nQfmLCG2fROmjY0jMsL/S/9noy/oyFM39+LLNXtJmZZj5V9L\neHOo5wJgJLBcRJZ4lj0CxACo6hueZTcDn6rqsTL3bQHM95zTGwpkqOrHvghuTCCYvXArf5y/nAvO\niSJ9VCx1w11OR/K5YfExhLqEP7y9jDFTspk0OpZ64fYRIieJqjqd4RdiY2M1JyfH6RjGVKvpC7bw\n53dXcnHXaCaMHEBEWOCVflnzF2/nwblLiW3flMljBtKgjpW/L4lIrrefk7J3W4xxwKTvN/Pnd1dy\nxXktSBsV+KUPcHO/tryU1I/crQcZOSmL/BNFTkcKWlb8xtSw17/eyBPvr+Kani15bXh/6oQGfun/\n7PrerXlteP/Si85NzOLQ8UKnIwUlK35jatBLX6znHx+v4cY+rXk5qV9QnuJ4dY+WTBg5gLW7j5CU\nnsWBY1b+NS34/tUZ4wBV5f8+Xctzn63jlv5teH5o36D+ROtl3Vow8Y5YNuUdJTFtAXlHTjodKagE\n7788Y2qIqvL0R2t4+csNJA5sxz9v62PXsAEu6hrNlNED2XbgBIlpC9hzuMDpSEHDit+YaqSq/PX9\nVUz4dhMjE9rz1M297KqVZZzfOYo3x8axO7+AoRMWsPPQCacjBQUrfmOqidut/OmdFUz5YQvjLuzI\nX4f0sNIvR1zHpkxPjmf/sUJun7CAbQeOOx0p4FnxG1MNStzKw/OWMTNrK7+55Bz+dN15NjnJafSP\nacLM5HiOFBQzdMICtuw7VvmdzFmz4jfGx4pL3Dz0r6XMzdnOby/vwu+vPtdK3wu92zYmIyWegmI3\nQ9MWsGHvUacjBSwrfmN8qKjEzf1zljBv8Q7+56qu/O5Km4D8TPRo3YhZKQmUuCExLZO1u484HSkg\nWfEb4yOFxW7uyVjE+8t28ci13bjnsi5OR/JL57aMZHZqAiECSemZrNpZ0VXgzdmy4jfGBwqKSvjN\njFw+WbmHx27oTupF5zgdya91bt6AueMHEREaQlJ6Jsu35zsdKaBY8RtTRQVFJaROz+WLNXt58qae\njLmgo9ORAkKHqPrMGT+IyIhQhk3MZNHWg05HChhW/MZUwfHCYsZOzea79Xk8c2tvRiS0dzpSQGnX\ntB5zxg+iaf1wRk7MInvLgcrvZCplxW/MWTp6spjRk7PJ3LSf527vw+0D2zkdKSC1aVyXueMH0aJR\nBKMmLeTHjfucjuT3rPiNOQuHC4oYNSmL3K0HeTGxHzf3a+t0pIDWomEEc1IH0a5pXcZMyebbdXlO\nR/Jr3ky92E5EvhKRVSKyUkTuK2fMJSKSLyJLPF+Pllk3WETWisgGEXnY1xtgTE07dLyQEROzWL4j\nn1eH9eeGPq2djhQUoiPrMCslgU7RDUh+M4cv1+xxOpLf8uYVfzHwoKp2BxKAu0WkeznjvlPVvp6v\nvwKIiAt4FbgG6A4kVXBfY/zCgWOFDEvPYs2uI7wxYgCDe7Z0OlJQadagDrNS4jm3ZSTjp+fyycrd\nTkfyS5UWv6ruUtVFnttHgNVAGy8fPw7YoKqbVLUQmA0MOduwxjgp78hJktIy2Zh3lPQ7Yrn8vBZO\nRwpKjeuFMyM5np5tGnH3zEV8sGyX05H8zhkd4xeRDkA/IKuc1eeLyDIR+UhEeniWtQG2lRmzHe9/\naBhTa+w5XEBi2gK2HjjOlNEDubhrtNORglqjumFMHxdPv5jG3DtrEe8s3uF0JL/idfGLSAPgbeB+\nVT31o3SLgBhV7Q28DLxzpkFEJFVEckQkJy/P3rgxtcfOQycYOmEBu/MLeHNsHOd3jnI6kgEa1Anl\nzbFxxHdsxgNzl/CvnG2V38kAXha/iIRRWvozVXXeqetV9bCqHvXc/hAIE5EoYAdQ9hy3tp5lv6Cq\naaoaq6qx0dH2asrUDtsOHGdo2gL2Hy1k2rh44jo2dTqSKaNeeCiTRw/kws5RPPTWMjKytjodyS94\nc1aPAJOA1ar6XAVjWnrGISJxnsfdD2QDXUSko4iEA4nAe74Kb0x1+mn/MYZOWED+8SJmJMczoH0T\npyOZctQNd5E+KpbLujXnkfnLefPHLU5HqvVCvRhzATASWC4iSzzLHgFiAFT1DeA24DciUgycABJV\nVYFiEbkH+ARwAZNVdaWPt8EYn9uYd5Rh6ZkUFruZlZpAj9aNnI5kTiMizMUbIwZwT8YiHntvJYXF\nblIu6uR0rFpLSvu5domNjdWcnBynY5ggtW7PEYalZwHKzOQEzm0Z6XQk46WfL4v9wbJdPHT1udx9\naWenI9UYEclV1Vhvxnrzit+YoLFq52FGTMoiNETISBlE5+YNnI5kzkCYK4QXh/YlLER49pO1FBa7\nuf+KLjYnwims+I3xWL49nxGTsqgX7iIjJYGOUfWdjmTOQqgrhP+7vS+hrhBe/GI9RSVuHrJZ0P6L\nFb8xwOKtBxk1eSENI8KYnZpAu6b1nI5kqsAVIjxza2/CQ0N47euNFBa7+V+b9/g/rPhN0MvecoAx\nU7Jp1iCcjJQE2jSu63Qk4wMhIcLfbupJuCuEid9vpqjEzWM39CAkxMrfit8EtQUb9zPuzWxaNoog\nIzmBlo0inI5kfEhEeOyG7oS5hPTvNlNYovztpp5BX/5W/CZofbc+j5RpObRrUo+ZKfE0j7TSD0Qi\nwiPXnkd4aAivfrWRohI3/7i1N64gLn8rfhOUvlqzl/EzcukUVZ+ZyfE0a1DH6UimGokI/3PVuYS7\nXDz/+TqKStz836/7EOoKzilJrPhN0Pl05W7uzljEuS0jmT42nib1w52OZGqAiHDfFV0ICxWe+Xgt\nxSXKC4l9CQvC8rfiN0Hlg2W7uG/2Ynq2acSbY+NoVDfM6Uimht11SWfCXSE8+cFqCkvcvDKsH3VC\nXU7HqlHB96POBK13l+zg3lmL6BfTmOnjrPSDWfKvOvHXIT34bNUe7pyeS0FRidORapQVvwkK/8rZ\nxv1zlhDXsSlTx8QRGWGlH+xGDerAUzf34ut1pW/ynygMnvK34jcBLyNrKw+9tYwLO0cxZXQc9evY\nEU5Talh8DM/c2pvvN+xjzNSFHDtZ7HSkGmHFbwLatAVbeGT+ci49N5r0UbHUDQ+uY7mmcr+ObccL\nQ/uSveUgd0xeyJGCIqcjVTsrfhOwJn63iUffXcmV3VvwxsgBRIRZ6ZvyDenbhpcS+7Fk2yFGTlpI\n/onALn8rfhOQXv1qA09+sJrrerXiteH9g+6sDXPmrutd+m9l5c58hk/M5NDxQqcjVRsrfhNQVJUX\nPl/Hs5+sZUjf1rwYpOdpm7NzVY+WpI2MZd2eoySmZbL/6EmnI1ULb6ZebCciX4nIKhFZKSL3lTNm\nuIgsE5HlIvKjiPQps26LZ/kSEbHZVUy1UVX++elaXvh8PbcNaMtznkvzGnMmLu3WnEl3xLJl/zES\n0zLZe6TA6Ug+583/imLgQVXtDiQAd4tI91PGbAYuVtVewBNA2inrL1XVvt7ODmPMmVJVnvpwNa9+\ntZGkuNIzNYL5Wiyman7VJZopo+PYcegEiRMy2Z0fWOVfafGr6i5VXeS5fQRYDbQ5ZcyPqnrQ820m\n0NbXQY2piKry+L9Xkf7dZu4Y1J6nbrarL5qqG3ROM6aNjWPvkZMMTVvAjkMnnI7kM2f0e7CIdAD6\nAVmnGTYO+KjM9wp8LiK5IpJ6msdOFZEcEcnJy8s7k1gmiLndyiPzVzD1xy2k/Kojf7mxh022YXwm\ntkNTpo+L48CxQoZOWMC2A8edjuQTXhe/iDQA3gbuV9XDFYy5lNLi/0OZxReqal/gGkoPE11U3n1V\nNU1VY1U1Njo62usNMMGrxK38/u1lzFq4lbsuOYdHrrUZlozv9YtpQkZyAkdPFnP7hAVs3nfM6UhV\n5lXxi0gYpaU/U1XnVTCmNzARGKKq+39erqo7PH/uBeYDcVUNbUxxiZsH5y7hrdzt3H9FF5tT1VSr\nXm0bkZGcwMliN0MnLGDD3iNOR6oSb87qEWASsFpVn6tgTAwwDxipquvKLK8vIpE/3wauAlb4IrgJ\nXkUlbu6bvYR3luzkoavP5f4rulrpm2rXvXVDZqcm4FZITMtk7W7/LX9vXvFfAIwELvOckrlERK4V\nkTtF5E7PmEeBZsBrp5y22QL4XkSWAguBD1T1Y19vhAkeJ4tLuHvmIj5Yvos/XXced1/a2elIJoh0\nbRHJnPEJuEKExLQFrNyZ73SksyKq6nSGX4iNjdWcHDvl3/y3gqISfjMjl6/W5vH4jT244/wOTkcy\nQWrLvmMMS8/kWGEJ08fF0bttY6cjISK53p4yb59uMX7hRGEJKdNy+GptHk/d3MtK3ziqQ1R95owf\nRMO6oQxPzyL3p4OV36kWseI3td7xwmLGTs3m+w37eOa23gyLj3E6kjG0a1qPOamDiIqsw6hJWWRt\n2l/5nWoJK35Tqx0pKOKOyQvJ2ryf52/vy+2x7ZyOZMx/tG5cl9mpCbRsFMHoKdn8sGGf05G8YsVv\naq38E0WMnLSQxVsP8XJSf27q16byOxlTw1o0jGB26iBimtZj7NRsvllX+z+AasVvaqVDxwsZMTGL\nlTvzeXV4f67r3crpSMZUKDqyDrNSEzgnugEpb+bwxeo9Tkc6LSt+U+vsP3qSpPQs1u45woSRA7i6\nR0unIxlTqab1w8lIiadbq0junJHLxyt2Ox2pQlb8plbZe6SApPRMNuUdZeKoWC7r1sLpSMZ4rXG9\ncGYkx9OrTSPuzljEv5fudDpSuaz4Ta2xO7+AxLRMth04wZQxA7moq12zyfifhhFhTBsXz4CYJtw3\nezHzF293OtIvWPGbWmHHoRMMTVvAnvwCpo2L4/xzopyOZMxZa1AnlKljB5LQqRm/m7uUudnbnI70\nX6z4jeO2HTjO0AkLOHCskOnJ8Qzs0NTpSMZUWb3wUCaPHsivukTz+7eXMSPzJ6cj/YcVv3HUln3H\nuH3CAo4UFJORnED/mCZORzLGZyLCXKSNHMDl3Zrzp3dWMOWHzU5HAqz4jYM27D3K7RMWcLLYzayU\nBHq1beR0JGN8LiLMxesjBnB1jxY8/u9VTPhmo9ORrPiNM9buPkJi2gLcCrNTE+jeuqHTkYypNuGh\nIbwyrD/X927F3z9awytfrnc0T6ijz26C0sqd+YyYmEV4aAgZKaUfejEm0IW5QnhhaF/CXSH889N1\nFJYoD1zRxZG5JKz4TY1atv0QIyctpH64i4yUBDpE1Xc6kjE1JtQVwrO/7kOoS3jpi/UUFrv5w+Ca\nnz3Omxm42onIVyKySkRWish95YwREXlJRDaIyDIR6V9m3WARWetZ97CvN8D4j0VbDzI8PYvIiFDm\njB9kpW+CkitEePqW3gyPj+GNbzbyxPurqel5Ubx5xV8MPKiqizzTKOaKyGequqrMmGuALp6veOB1\nIF5EXMCrwJXAdiBbRN475b4mCCzcfIAxUxYSHVmHjJQEWjeu63QkYxwTEiI8eVNPwlwhTP5hM0Ul\nbh6/sQchITXzyr/S4lfVXcAuz+0jIrIaaAOULe8hwDQt/bGVKSKNRaQV0AHYoKqbAERktmesFX8Q\n+XHDPsa9mUPrxhFkpCTQomGE05GMcZyI8NgN3akTGsKEbzdRVOLmqZt71Uj5n9ExfhHpAPQDsk5Z\n1QYo+9G07Z5l5S2PP9OQxn99uy6PlGk5tG9Wj5nJCURH1nE6kjG1hojw8DXdCA8N4eUvN1BUojxz\nW29c1Vz+Xhe/iDQA3gbuV9XDvg4iIqlAKkBMjM2wFAi+WL2H38xYxDnNGzBjXBzNGljpG3MqEeHB\nq84lzBXC9xv2UVjspm64q1qf06vz+EUkjNLSn6mq88oZsgMoOzVSW8+yipb/gqqmqWqsqsZGR9vF\nufzdxyt2c+eMXLq1imRWSryVvjGV+O3lXZgxLr7aSx+8O6tHgEnAalV9roJh7wGjPGf3JAD5nvcG\nsoEuItJRRMKBRM9YE8DeX7aTuzMW0bNNI2Ykx9O4XrjTkYzxC+GhNfOZWm8O9VwAjASWi8gSz7JH\ngBgAVX0D+BC4FtgAHAfGeNYVi8g9wCeAC5isqit9ugWmVpm/eDsPzl3KgPZNmDImjgZ17KMixtQ2\n3pzV8z1w2ncaPGfz3F3Bug8p/cFgAtzcnG384e1lJHRsxqTRsdQLt9I3pjaya/UYn5iZ9RO/f2sZ\nF3aOYvLogVb6xtRi9r/TVNnUHzbzl3+v4rJuzXlteH8iwqr/zSljzNmz4jdVkvbtRp76cA1X92jB\ny0n9a+zNKWPM2bPiN2ftlS/X889P13Fd71a8MLQvYS4rfWP8gRW/OWOqyvOfr+elL9Zzc782PHtb\nb0Kt9I3xG1b85oyoKs98spbXv97Irwe05elbq//j5cYY37LiN15TVZ78YDWTvt/M8PgYnhjSs8au\nJmiM8R0rfuMVt1v5y79XMm3BT4w+vwOP3dDdkZmDjDFVZ8VvKuV2K//7znJmLdxG6kWd+OM13az0\njfFjVvzmtErcyu/fWsbbi7Zzz6WdefCqrlb6xvg5K35ToeISNw/+aynvLtnJ767sym8v7+J0JGOM\nD1jxm3IVlbi5b/ZiPly+m98PPpe7LunsdCRjjI9Y8ZtfOFlcwt0zF/P56j386brzSP5VJ6cjGWN8\nyIrf/JeCohLunJHL12vz+OuQHowa1MHpSMYYH7PiN/9xorCElGk5/LBxH3+/pRdJcTYFpjGByIrf\nAHDsZDHj3sxm4eYDPHtbH24b0NbpSMaYalJp8YvIZOB6YK+q9ixn/UPA8DKPdx4QraoHRGQLcAQo\nAYpVNdZXwY3vHCkoYsyUbBZvO8TzQ/sypG8bpyMZY6qRN1fWmgoMrmilqj6rqn1VtS/wR+AbVT1Q\nZsilnvVW+rVQ/okiRkxayJJth3g5qZ+VvjFBoNLiV9VvgQOVjfNIAmZVKZGpMQePFTJ8Yiardubz\n2vD+XNurldORjDE1wGfX0hWRepT+ZvB2mcUKfC4iuSKS6qvnMlW37+hJktIzWbfnKGmjYrmqR0un\nIxljaogv39y9AfjhlMM8F6rqDhFpDnwmIms8v0H8gucHQypATIydTVKd9h4uYPjELLYdPM7kOwZy\nYZcopyMZY2qQL2fPSOSUwzyqusPz515gPhBX0Z1VNU1VY1U1Njo62oexTFm78wtITMtkx6ETTBkd\nZ6VvTBDySfGLSCPgYuDdMsvqi0jkz7eBq4AVvng+c3Z2HDrB0LQF7D1ykmlj4xh0TjOnIxljHODN\n6ZyzgEuAKBHZDjwGhAGo6hueYTcDn6rqsTJ3bQHM91zJMRTIUNWPfRfdnImt+4+TlJ7J4YIipo+L\no19ME6cjGWMcUmnxq2qSF2OmUnraZ9llm4A+ZxvM+M7mfccYlp7JiaISZqUk0LNNI6cjGWMcZJ/c\nDXAb9h4hKT2LErcyKyWB81o1dDqSMcZhVvwBbM3uwwxPz0JEmJ2aQNcWkU5HMsbUAlb8AWrFjnxG\nTsoiPDSEjJQEzolu4HQkY0wtYcUfgJZuO8TISVlERoSRkRJP+2b1nY5kjKlFrPgDTO5PBxg9OZvG\n9cPISE6gXdN6TkcyxtQyVvwBJGvTfsZOzaZ5wwhmJsfTunFdpyMZY2ohK/4A8cOGfSS/mUPrxhHM\nSkmgecMIpyMZY2opX16ywTjkm3V5jJ2aTUzTesxOHWSlb4w5LXvF7+c+X7WHu2YuonPzBsxIjqdp\n/XCnIxljajl7xe/HPl6xiztn5NKtVSQZKVb6xhjv2Ct+P/XvpTu5f84S+rRtxNSxcTSMCHM6kjHG\nT1jx+6F5i7bzP/9aSmyHpkwePZAGdWw3GmO8Z43hZ+Zmb+MP85YxqFMzJt4RS71w24XGmDNjreFH\npmf+xJ/fWcFFXaNJGzmAiDCX05GMMX7Iit9PTP5+M399fxWXd2vOq8P7W+kbY86aFb8fmPDNRv7+\n0RoG92jJS0n9CA+1k7GMMWev0gYRkckisldEyp02UUQuEZF8EVni+Xq0zLrBIrJWRDaIyMO+DB4s\nXv5iPX//aA039GnNy8Os9I0xVefNK/6pwCvAtNOM+U5Vry+7QERcwKvAlcB2IFtE3lPVVWeZNaio\nKs9/to6XvtzALf3a8MxtvQl1WekbY6qu0iZR1W+BA2fx2HHABlXdpKqFwGxgyFk8TtBRVf7x8Vpe\n+nIDt8cwxafaAAALXUlEQVS25dlf97HSN8b4jK/a5HwRWSYiH4lID8+yNsC2MmO2e5aVS0RSRSRH\nRHLy8vJ8FMv/qCpPvL+aN77ZyIiEGJ6+pTeuEHE6ljEmgPii+BcBMaraG3gZeOdsHkRV01Q1VlVj\no6OjfRDL/7jdyqPvrmTyD5sZc0EHnhjSkxArfWOMj1W5+FX1sKoe9dz+EAgTkShgB9CuzNC2nmWm\nHG638sj85UzP/InxF3fi0eu7I2Klb4zxvSqfzikiLYE9qqoiEkfpD5P9wCGgi4h0pLTwE4FhVX2+\nQFTiVh56aynzFu3g3ss687sru1rpG2OqTaXFLyKzgEuAKBHZDjwGhAGo6hvAbcBvRKQYOAEkqqoC\nxSJyD/AJ4AImq+rKatkKP1Zc4uaBuUv599Kd/O7Krvz28i5ORzLGBDgp7ejaJTY2VnNycpyOUe0K\ni93cN3sxH63YzcPXdOPOi89xOpIxxk+JSK6qxnoz1j6565CTxSXcPXMRn6/ey5+v7864Czs6HckY\nEySs+B1QUFTC+Om5fLMujyeG9GDkoA5ORzLGBBEr/hp2orCE5GnZ/LhxP0/f0ovEuBinIxljgowV\nfw06drKYsVOzyd5ygH/e1odbB7R1OpIxJghZ8deQwwVFjJmSzZJth3ghsR839mntdCRjTJCy4q8B\n+ceLGDU5i5U7D/NKUj+u6dXK6UjGmCBmxV/NDhwrZOSkLNbvOcrrIwZwZfcWTkcyxgQ5K/5qtO/o\nSUZMzGLTvmOkjRrAJec2dzqSMcZY8VeXvYcLGDYxi+0HjzNl9EAu6BzldCRjjAGs+KvFrvwTDEvP\nYs/hAqaOiSOhUzOnIxljzH9Y8fvY9oPHGZaexYFjhUwfF8eA9k2djmSMMf/Fit+Htu4/TlJ6JkcK\nipiRHE/fdo2djmSMMb9gxe8jm/KOMiw9i4LiEjJSEujZppHTkYwxplxW/D6wfs8Rhk3Mwu1WZqcm\n0K1lQ6cjGWNMhWwG7ypaveswiWmZAFb6xhi/YMVfBSt25JOUnkmYK4Q5qQl0aRHpdCRjjKlUpcUv\nIpNFZK+IrKhg/XARWSYiy0XkRxHpU2bdFs/yJSISUDOrLNl2iGHpmdQPD2XO+AQ6RTdwOpIxxnjF\nm1f8U4HBp1m/GbhYVXsBTwBpp6y/VFX7ejszjD/I/ekAIyZm0bheOHPGJ9C+WX2nIxljjNcqfXNX\nVb8VkQ6nWf9jmW8zgYC+1nDmpv2MnZpNy4YRzEyJp1Wjuk5HMsaYM+LrY/zjgI/KfK/A5yKSKyKp\np7ujiKSKSI6I5OTl5fk4lm98v34fo6cspHXjusxOTbDSN8b4JZ+dzikil1Ja/BeWWXyhqu4QkebA\nZyKyRlW/Le/+qpqG5zBRbGxsrZsB/uu1e0mdnkunqPrMSI4nqkEdpyMZY8xZ8ckrfhHpDUwEhqjq\n/p+Xq+oOz597gflAnC+er6Z9vmoPqdNy6dK8AbNSEqz0jTF+rcrFLyIxwDxgpKquK7O8vohE/nwb\nuAoo98yg2uyj5bu4c0Yu57VuSEZyAk3qhzsdyRhjqqTSQz0iMgu4BIgSke3AY0AYgKq+ATwKNANe\nExGAYs8ZPC2A+Z5loUCGqn5cDdtQbd5dsoPfzV1K33aNmTJmIA0jwpyOZIwxVebNWT1JlaxPBpLL\nWb4J6PPLe/iHt3O389BbS4nt0JTJowfSoI5d3cIYExiszcoxJ3srD89bzvnnNCN9VCz1wu2vyRgT\nOKzRTjF9wRb+/O5KLu4azYSRA4gIczkdyRhjfMqKv4xJ32/mifdXccV5LXh1eD/qhFrpG2MCjxW/\nx+tfb+QfH6/hmp4teTGxH+Ghdv06Y0xgsuIHXvpiPc99to4b+rTm+dv7EOqy0jfGBK6gLn5V5bnP\n1vHylxu4pX8bnr2tD64QcTqWMcZUq6AtflXl6Y/WMOHbTSQObMdTN/cixErfGBMEgrL4VZW/vr+K\nKT9sYWRCex6/sYeVvjEmaARd8bvdyqPvrWBG5lbGXtCRP19/Hp5PFxtjTFAIquIvcSuPzFvOnJxt\n3HnxOfxh8LlW+saYoBM0xV9c4ub3by1j3uId/PbyLjxwRRcrfWNMUAqK4i8qcfPAnCW8v2wXD17Z\nlXsv7+J0JGOMcUzAF39hsZt7Zy3ik5V7+OM13Rh/8TlORzLGGEcFdPGfLC7hrhmL+GLNXh69vjtj\nL+zodCRjjHFcwBZ/QVEJqdNz+XZdHk/e1JMRCe2djmSMMbVCpdcmEJHJIrJXRMqdPUtKvSQiG0Rk\nmYj0L7NusIis9ax72JfBT+d4YTFjp2bz3fo8nrm1t5W+McaU4c1FaaYCg0+z/hqgi+crFXgdQERc\nwKue9d2BJBHpXpWw3jh6spjRk7PJ3LSf527vw+0D21X3UxpjjF+ptPhV9VvgwGmGDAGmaalMoLGI\ntKJ0YvUNqrpJVQuB2Z6x1eZwQRGjJmWRu/UgLyb24+Z+bavz6Ywxxi/54jKUbYBtZb7f7llW0fJq\ncbigiJETs1i+I59Xh/Xjhj6tq+upjDHGr9WaN3dFJJXSQ0XExMSc8f3rhbnoGFWfey/rwhXdW/g6\nnjHGBAxfFP8OoOyB9LaeZWEVLC+XqqYBaQCxsbF6piFCXSG8kNjvTO9mjDFBxxeHet4DRnnO7kkA\n8lV1F5ANdBGRjiISDiR6xhpjjHFQpa/4RWQWcAkQJSLbgccofTWPqr4BfAhcC2wAjgNjPOuKReQe\n4BPABUxW1ZXVsA3GGGPOQKXFr6pJlaxX4O4K1n1I6Q8GY4wxtYRNLmuMMUHGit8YY4KMFb8xxgQZ\nK35jjAkyVvzGGBNkpPSknNpFRPKAn87y7lHAPh/GcVKgbEugbAfYttRGgbIdULVtaa+q0d4MrJXF\nXxUikqOqsU7n8IVA2ZZA2Q6wbamNAmU7oOa2xQ71GGNMkLHiN8aYIBOIxZ/mdAAfCpRtCZTtANuW\n2ihQtgNqaFsC7hi/McaY0wvEV/zGGGNOwy+Lv7JJ3E83AXxt48W2XCIi+SKyxPP1qBM5KyMik0Vk\nr4isqGC9P+2TyrbFX/ZJOxH5SkRWichKEbmvnDF+sV+83BZ/2S8RIrJQRJZ6tuXxcsZU735RVb/6\novQSzxuBTkA4sBTofsqYa4GPAAESgCync1dhWy4B3nc6qxfbchHQH1hRwXq/2Cdebou/7JNWQH/P\n7UhgnR//X/FmW/xlvwjQwHM7DMgCEmpyv/jjK35vJnGvaAL42qbGJ6SvLqr6LXDgNEP8ZZ94sy1+\nQVV3qeoiz+0jwGp+Oe+1X+wXL7fFL3j+ro96vg3zfJ36Zmu17hd/LH5vJnGv0Yneq8DbnOd7ft37\nSER61Ew0n/OXfeItv9onItIB6Efpq8uy/G6/nGZbwE/2i4i4RGQJsBf4TFVrdL/UmsnWTYUWATGq\nelRErgXeAbo4nCnY+dU+EZEGwNvA/ap62Ok8VVHJtvjNflHVEqCviDQG5otIT1Ut9z2l6uCPr/gr\nmtz9TMfUBpXmVNXDP/9aqKUzmoWJSFTNRfQZf9knlfKnfSIiYZQW5UxVnVfOEL/ZL5Vtiz/tl5+p\n6iHgK2DwKauqdb/4Y/F7M4l7RRPA1zaVbouItBQR8dyOo3Sf7a/xpFXnL/ukUv6yTzwZJwGrVfW5\nCob5xX7xZlv8aL9Ee17pIyJ1gSuBNacMq9b94neHerSCSdxF5E7P+gongK9tvNyW24DfiEgxcAJI\nVM/b/rWJiMyi9KyKKBHZDjxG6ZtWfrVPwKtt8Yt9AlwAjASWe44nAzwCxIDf7RdvtsVf9ksr4E0R\ncVH6w2muqr5fkx1mn9w1xpgg44+HeowxxlSBFb8xxgQZK35jjAkyVvzGGBNkrPiNMSbIWPEbY0yQ\nseI3xpggY8VvjDFB5v8B6zDXe54vk6EAAAAASUVORK5CYII=\n", 104 | "text/plain": [ 105 | "" 106 | ] 107 | }, 108 | "metadata": {}, 109 | "output_type": "display_data" 110 | } 111 | ], 112 | "source": [ 113 | "plt.plot([1,2,3,2])" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": 8, 119 | "metadata": {}, 120 | "outputs": [ 121 | { 122 | "name": "stdout", 123 | "output_type": "stream", 124 | "text": [ 125 | "comet-07-18.sdsc.edu\r\n" 126 | ] 127 | } 128 | ], 129 | "source": [ 130 | "!hostname" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": null, 136 | "metadata": { 137 | "collapsed": true 138 | }, 139 | "outputs": [], 140 | "source": [] 141 | } 142 | ], 143 | "metadata": { 144 | "kernelspec": { 145 | "display_name": "Python 3", 146 | "language": "python", 147 | "name": "python3" 148 | }, 149 | "language_info": { 150 | "codemirror_mode": { 151 | "name": "ipython", 152 | "version": 3 153 | }, 154 | "file_extension": ".py", 155 | "mimetype": "text/x-python", 156 | "name": "python", 157 | "nbconvert_exporter": "python", 158 | "pygments_lexer": "ipython3", 159 | "version": "3.6.1" 160 | } 161 | }, 162 | "nbformat": 4, 163 | "nbformat_minor": 2 164 | } 165 | -------------------------------------------------------------------------------- /notebook_singularity.slrm: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #SBATCH --job-name="jupyter-notebook" 3 | #SBATCH --output="jupyter-notebook.%j.%N.out" 4 | #SBATCH --partition=compute 5 | #SBATCH --nodes=1 6 | #SBATCH --ntasks-per-node=24 7 | #SBATCH --export=ALL 8 | #SBATCH --time=04:00:00 9 | 10 | module load singularity 11 | 12 | SINGULARITY_IMAGE="/oasis/scratch/comet/zonca/temp_project/zonca-singularity-comet-master-2019-05.simg" 13 | COMMAND="/opt/conda/bin/jupyter lab --no-browser --ip=$(hostname)" 14 | export SINGULARITY_BINDPATH="/oasis" 15 | 16 | singularity exec $SINGULARITY_IMAGE $COMMAND 17 | -------------------------------------------------------------------------------- /numba_groupby_pixels.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## Astrophysics background\n", 8 | "\n", 9 | "It is very common in Astrophysics to work with sky pixels. The sky is tassellated in patches with specific properties and a sky map is then a collection of intensity values for each pixel. The most common pixelization used in Cosmology is [HEALPix](http://healpix.jpl.nasa.gov)." 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "Measurements from telescopes are then represented as an array of pixels that encode the pointing of the instrument at each timestamp and the measurement output." 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "## Sample timeline" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 1, 29 | "metadata": { 30 | "collapsed": true 31 | }, 32 | "outputs": [], 33 | "source": [ 34 | "import pandas as pd\n", 35 | "import numba\n", 36 | "import numpy as np" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "For simplicity let's assume we have a sky with 50K pixels:" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 2, 49 | "metadata": { 50 | "collapsed": true 51 | }, 52 | "outputs": [], 53 | "source": [ 54 | "NPIX = 50000" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "And we have 50 million measurement from our instrument:" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 3, 67 | "metadata": { 68 | "collapsed": true 69 | }, 70 | "outputs": [], 71 | "source": [ 72 | "NTIME = int(50 * 1e6)" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": {}, 78 | "source": [ 79 | "The pointing of our instrument is an array of pixels, random in our sample case:" 80 | ] 81 | }, 82 | { 83 | "cell_type": "code", 84 | "execution_count": 4, 85 | "metadata": { 86 | "collapsed": true 87 | }, 88 | "outputs": [], 89 | "source": [ 90 | "pixels = np.random.randint(0, NPIX-1, NTIME)" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": {}, 96 | "source": [ 97 | "Our data are also random:" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": 5, 103 | "metadata": { 104 | "collapsed": true 105 | }, 106 | "outputs": [], 107 | "source": [ 108 | "timeline = np.random.randn(NTIME)" 109 | ] 110 | }, 111 | { 112 | "cell_type": "markdown", 113 | "metadata": {}, 114 | "source": [ 115 | "## Create a map of the sky with pandas" 116 | ] 117 | }, 118 | { 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "One of the most common operations is to sum all of our measurements in a sky map, so the value of each pixel in our sky map will be the sum of each individual measurement.\n", 123 | "The easiest way is to use the `groupby` operation in `pandas`:" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 6, 129 | "metadata": { 130 | "collapsed": true 131 | }, 132 | "outputs": [], 133 | "source": [ 134 | "timeline_pandas = pd.Series(timeline, index=pixels)" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": 7, 140 | "metadata": {}, 141 | "outputs": [ 142 | { 143 | "data": { 144 | "text/plain": [ 145 | "11105 -0.498438\n", 146 | "16106 -1.103009\n", 147 | "44723 0.392057\n", 148 | "16687 0.086918\n", 149 | "15197 1.946979\n", 150 | "dtype: float64" 151 | ] 152 | }, 153 | "execution_count": 7, 154 | "metadata": {}, 155 | "output_type": "execute_result" 156 | } 157 | ], 158 | "source": [ 159 | "timeline_pandas.head()" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": 8, 165 | "metadata": {}, 166 | "outputs": [ 167 | { 168 | "name": "stdout", 169 | "output_type": "stream", 170 | "text": [ 171 | "CPU times: user 2.49 s, sys: 604 ms, total: 3.09 s\n", 172 | "Wall time: 3.1 s\n" 173 | ] 174 | } 175 | ], 176 | "source": [ 177 | "%time m = timeline_pandas.groupby(level=0).sum()" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": {}, 183 | "source": [ 184 | "## Create a map of the sky with numba" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": {}, 190 | "source": [ 191 | "We would like to improve the performance of this operation using `numba`, which allows to produce automatically C-speed compiled code from pure python functions." 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "metadata": {}, 197 | "source": [ 198 | "First we need to develop a pure python version of the code, test it, and then have `numba` optimize it:" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": 9, 204 | "metadata": { 205 | "collapsed": true 206 | }, 207 | "outputs": [], 208 | "source": [ 209 | "def groupby_python(index, value, output):\n", 210 | " for i in range(index.shape[0]):\n", 211 | " output[index[i]] += value[i]" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": 10, 217 | "metadata": { 218 | "collapsed": true 219 | }, 220 | "outputs": [], 221 | "source": [ 222 | "m_python = np.zeros_like(m)" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": 11, 228 | "metadata": {}, 229 | "outputs": [ 230 | { 231 | "name": "stdout", 232 | "output_type": "stream", 233 | "text": [ 234 | "CPU times: user 25.8 s, sys: 5 ms, total: 25.8 s\n", 235 | "Wall time: 25.8 s\n" 236 | ] 237 | } 238 | ], 239 | "source": [ 240 | "%time groupby_python(pixels, timeline, m_python)" 241 | ] 242 | }, 243 | { 244 | "cell_type": "code", 245 | "execution_count": 12, 246 | "metadata": { 247 | "collapsed": true 248 | }, 249 | "outputs": [], 250 | "source": [ 251 | "np.testing.assert_allclose(m_python, m)" 252 | ] 253 | }, 254 | { 255 | "cell_type": "markdown", 256 | "metadata": {}, 257 | "source": [ 258 | "Pure Python is slower than the `pandas` version implemented in `cython`." 259 | ] 260 | }, 261 | { 262 | "cell_type": "markdown", 263 | "metadata": {}, 264 | "source": [ 265 | "### Optimize the function with numba.jit\n", 266 | "\n", 267 | "`numba.jit` gets an input function and creates an compiled version with does not depend on slow Python calls, this is enforced by `nopython=True`, `numba` would throw an error if it would not be possible to run in `nopython` mode." 268 | ] 269 | }, 270 | { 271 | "cell_type": "code", 272 | "execution_count": 13, 273 | "metadata": { 274 | "collapsed": true 275 | }, 276 | "outputs": [], 277 | "source": [ 278 | "groupby_numba = numba.jit(groupby_python, nopython=True)" 279 | ] 280 | }, 281 | { 282 | "cell_type": "code", 283 | "execution_count": 14, 284 | "metadata": { 285 | "collapsed": true 286 | }, 287 | "outputs": [], 288 | "source": [ 289 | "m_numba = np.zeros_like(m)" 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": 15, 295 | "metadata": {}, 296 | "outputs": [ 297 | { 298 | "name": "stdout", 299 | "output_type": "stream", 300 | "text": [ 301 | "CPU times: user 411 ms, sys: 7 ms, total: 418 ms\n", 302 | "Wall time: 441 ms\n" 303 | ] 304 | } 305 | ], 306 | "source": [ 307 | "%time groupby_numba(pixels, timeline, m_numba)" 308 | ] 309 | }, 310 | { 311 | "cell_type": "code", 312 | "execution_count": 16, 313 | "metadata": { 314 | "collapsed": true 315 | }, 316 | "outputs": [], 317 | "source": [ 318 | "np.testing.assert_allclose(m_numba, m)" 319 | ] 320 | }, 321 | { 322 | "cell_type": "markdown", 323 | "metadata": {}, 324 | "source": [ 325 | "Performance improvement is about 50x compared to Python and up to 10x compared to Pandas, pretty good!" 326 | ] 327 | }, 328 | { 329 | "cell_type": "markdown", 330 | "metadata": {}, 331 | "source": [ 332 | "## Use numba.jit as a decorator" 333 | ] 334 | }, 335 | { 336 | "cell_type": "markdown", 337 | "metadata": {}, 338 | "source": [ 339 | "The exact same result is obtained if we use `numba.jit` as a decorator:" 340 | ] 341 | }, 342 | { 343 | "cell_type": "code", 344 | "execution_count": null, 345 | "metadata": { 346 | "collapsed": true 347 | }, 348 | "outputs": [], 349 | "source": [ 350 | "@numba.jit(nopython=True)\n", 351 | "def groupby_numba(index, value, output):\n", 352 | " for i in range(index.shape[0]):\n", 353 | " output[index[i]] += value[i]" 354 | ] 355 | } 356 | ], 357 | "metadata": { 358 | "kernelspec": { 359 | "display_name": "Python 3", 360 | "language": "python", 361 | "name": "python3" 362 | }, 363 | "language_info": { 364 | "codemirror_mode": { 365 | "name": "ipython", 366 | "version": 3 367 | }, 368 | "file_extension": ".py", 369 | "mimetype": "text/x-python", 370 | "name": "python", 371 | "nbconvert_exporter": "python", 372 | "pygments_lexer": "ipython3", 373 | "version": "3.6.5" 374 | } 375 | }, 376 | "nbformat": 4, 377 | "nbformat_minor": 1 378 | } 379 | --------------------------------------------------------------------------------