├── 00-Setup.ipynb
├── 01-Flux-digits.ipynb
├── 02-DP-linear-fit.ipynb
├── 03-Differentiable-trebuchet-Zygote.ipynb
├── 03-Differentiable-trebuchet.ipynb
├── LICENSE
├── README.md
└── msr_2019_06_03
├── Differentiable Trebuchet.ipynb
├── Manifest.toml
├── Project.toml
├── Zygote IR Example.ipynb
├── Zygote Linear Regression.ipynb
└── Zygote Optimizing Raytracer.ipynb
/00-Setup.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "# Only if using JuliaBox\n",
10 | "pop!(DEPOT_PATH)\n",
11 | "push!(DEPOT_PATH, \"/home/jrun/.julia2/\")"
12 | ]
13 | },
14 | {
15 | "cell_type": "code",
16 | "execution_count": null,
17 | "metadata": {},
18 | "outputs": [],
19 | "source": [
20 | "# Install packages\n",
21 | "using Pkg\n",
22 | "Pkg.add([\"Flux\",\"Zygote\",\"Trebuchet#master\",\"Images\",\"ImageMagick\"])"
23 | ]
24 | },
25 | {
26 | "cell_type": "code",
27 | "execution_count": null,
28 | "metadata": {},
29 | "outputs": [],
30 | "source": [
31 | "# On Julia v1.3rc5, use Flux#master"
32 | ]
33 | },
34 | {
35 | "cell_type": "code",
36 | "execution_count": null,
37 | "metadata": {},
38 | "outputs": [],
39 | "source": [
40 | "# Load them so they are precompiled\n",
41 | "using Flux, Zygote, Trebuchet, Images, ImageMagick"
42 | ]
43 | },
44 | {
45 | "cell_type": "code",
46 | "execution_count": null,
47 | "metadata": {},
48 | "outputs": [],
49 | "source": []
50 | }
51 | ],
52 | "metadata": {
53 | "@webio": {
54 | "lastCommId": null,
55 | "lastKernelId": null
56 | },
57 | "kernelspec": {
58 | "display_name": "Julia 1.0.3",
59 | "language": "julia",
60 | "name": "julia-1.0"
61 | },
62 | "language_info": {
63 | "file_extension": ".jl",
64 | "mimetype": "application/julia",
65 | "name": "julia",
66 | "version": "1.0.3"
67 | }
68 | },
69 | "nbformat": 4,
70 | "nbformat_minor": 2
71 | }
72 |
--------------------------------------------------------------------------------
/01-Flux-digits.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "# Only if using JuliaBox\n",
10 | "pop!(DEPOT_PATH)\n",
11 | "push!(DEPOT_PATH, \"/home/jrun/.julia2/\")"
12 | ]
13 | },
14 | {
15 | "cell_type": "markdown",
16 | "metadata": {},
17 | "source": [
18 | "
\n",
19 | "# Recognizing handwritten digits using a neural network"
20 | ]
21 | },
22 | {
23 | "cell_type": "markdown",
24 | "metadata": {},
25 | "source": [
26 | "We have now reached the point where we can tackle a very interesting task: applying the knowledge we have gained with machine learning in general, and `Flux.jl` in particular, to create a neural network that can recognize handwritten digits! The data are from a data set called MNIST, which has become a classic in the machine learning world.\n",
27 | "\n",
28 | "[We could also try to apply the techniques to the original images of fruit instead. However, the fruit images are much larger than the MNIST images, which makes the learning a suitable neural network too slow.]"
29 | ]
30 | },
31 | {
32 | "cell_type": "markdown",
33 | "metadata": {},
34 | "source": [
35 | "## Data munging"
36 | ]
37 | },
38 | {
39 | "cell_type": "markdown",
40 | "metadata": {},
41 | "source": [
42 | "As we know, the first difficulty with any new data set is locating it, understanding what format it is stored in, reading it in and decoding it into a useful data structure in Julia.\n",
43 | "\n",
44 | "The original MNIST data is available [here](http://yann.lecun.com/exdb/mnist); see also the [Wikipedia page](https://en.wikipedia.org/wiki/MNIST_database). However, the format that the data is stored in is rather obscure.\n",
45 | "\n",
46 | "Fortunately, various packages in Julia provide nicer interfaces to access it. We will use the one provided by `Flux.jl`.\n",
47 | "\n",
48 | "The data are images of handwritten digits, and the corresponding labels that were determined by hand (i.e. by humans). Our job will be to get the computer to **learn** to recognize digits by learning, as usual, the function that relates the input and output data."
49 | ]
50 | },
51 | {
52 | "cell_type": "markdown",
53 | "metadata": {},
54 | "source": [
55 | "### Loading and examining the data"
56 | ]
57 | },
58 | {
59 | "cell_type": "markdown",
60 | "metadata": {},
61 | "source": [
62 | "First we load the required packages:"
63 | ]
64 | },
65 | {
66 | "cell_type": "code",
67 | "execution_count": null,
68 | "metadata": {},
69 | "outputs": [],
70 | "source": [
71 | "using Flux, Flux.Data.MNIST, Images"
72 | ]
73 | },
74 | {
75 | "cell_type": "markdown",
76 | "metadata": {},
77 | "source": [
78 | "Now we read in the data:"
79 | ]
80 | },
81 | {
82 | "cell_type": "code",
83 | "execution_count": null,
84 | "metadata": {},
85 | "outputs": [],
86 | "source": [
87 | "labels = MNIST.labels();\n",
88 | "images = MNIST.images();"
89 | ]
90 | },
91 | {
92 | "cell_type": "markdown",
93 | "metadata": {},
94 | "source": [
95 | "We see that `images` is a `Vector`, i.e. an `Array{T, 1}` with a complicated parameter `T`. It has length"
96 | ]
97 | },
98 | {
99 | "cell_type": "code",
100 | "execution_count": null,
101 | "metadata": {},
102 | "outputs": [],
103 | "source": [
104 | "length(labels)"
105 | ]
106 | },
107 | {
108 | "cell_type": "markdown",
109 | "metadata": {},
110 | "source": [
111 | "But we can just look at the first handful to get a sense of the contents"
112 | ]
113 | },
114 | {
115 | "cell_type": "code",
116 | "execution_count": null,
117 | "metadata": {},
118 | "outputs": [],
119 | "source": [
120 | "images[1:5]"
121 | ]
122 | },
123 | {
124 | "cell_type": "code",
125 | "execution_count": null,
126 | "metadata": {},
127 | "outputs": [],
128 | "source": [
129 | "labels[1:5]' # transposed to match the above"
130 | ]
131 | },
132 | {
133 | "cell_type": "markdown",
134 | "metadata": {},
135 | "source": [
136 | "So the $i$th entry of the array is the data for the $i$th image."
137 | ]
138 | },
139 | {
140 | "cell_type": "code",
141 | "execution_count": null,
142 | "metadata": {},
143 | "outputs": [],
144 | "source": [
145 | "typeof(images[1])"
146 | ]
147 | },
148 | {
149 | "cell_type": "markdown",
150 | "metadata": {},
151 | "source": [
152 | "As with the fruit images from the start of the course, the image is an array of color blocks, except that now each pixel just has a grey scale.\n",
153 | "\n",
154 | "To see the actual numeric content of the image, we can do, for example"
155 | ]
156 | },
157 | {
158 | "cell_type": "code",
159 | "execution_count": null,
160 | "metadata": {},
161 | "outputs": [],
162 | "source": [
163 | "Float64.(images[1])"
164 | ]
165 | },
166 | {
167 | "cell_type": "markdown",
168 | "metadata": {},
169 | "source": [
170 | "### Setting up a neural network"
171 | ]
172 | },
173 | {
174 | "cell_type": "markdown",
175 | "metadata": {},
176 | "source": [
177 | "In the previous notebooks, we arranged the input data for Flux as a `Vector` of `Vector`s.\n",
178 | "Now we will use an alternative arrangement, as a matrix, since that allows `Flux` to use matrix operations, which are more efficient.\n",
179 | "\n",
180 | "The column $i$ of the matrix is a vector consisting of the $i$th data point $\\mathbf{x}^{(i)}$. Similarly, the desired outputs are given as a matrix, with the $i$th column being the desired output $\\mathbf{y}^{(i)}$."
181 | ]
182 | },
183 | {
184 | "cell_type": "code",
185 | "execution_count": null,
186 | "metadata": {},
187 | "outputs": [],
188 | "source": [
189 | "n_inputs = unique(length.(images))[]"
190 | ]
191 | },
192 | {
193 | "cell_type": "code",
194 | "execution_count": null,
195 | "metadata": {},
196 | "outputs": [],
197 | "source": [
198 | "n_outputs = length(unique(labels))"
199 | ]
200 | },
201 | {
202 | "cell_type": "markdown",
203 | "metadata": {},
204 | "source": [
205 | "#### Creating the features\n",
206 | "\n",
207 | "We want to create a vector of feature vectors, each with the floating point values of the 784 pixels.\n",
208 | "\n",
209 | "An image is a matrix of colours, but now we need a vector of floating point numbers instead. To do so, we just arrange all of the elements of the matrix in a certain way into a single list; fortunately, Julia already provides the function `vec` to do so!"
210 | ]
211 | },
212 | {
213 | "cell_type": "markdown",
214 | "metadata": {},
215 | "source": [
216 | "Let's use a subset of $N=5,000$ of the total $60,000$ images available in order to hold out test images that our model hasn't been trained on."
217 | ]
218 | },
219 | {
220 | "cell_type": "code",
221 | "execution_count": null,
222 | "metadata": {},
223 | "outputs": [],
224 | "source": [
225 | "preprocess(img) = vec(Float64.(img))"
226 | ]
227 | },
228 | {
229 | "cell_type": "code",
230 | "execution_count": null,
231 | "metadata": {},
232 | "outputs": [],
233 | "source": [
234 | "xs = preprocess.(images[1:5000]);"
235 | ]
236 | },
237 | {
238 | "cell_type": "markdown",
239 | "metadata": {},
240 | "source": [
241 | "#### Creating the labels\n",
242 | "\n",
243 | "We can just use `Flux.onehot` to create them:"
244 | ]
245 | },
246 | {
247 | "cell_type": "code",
248 | "execution_count": null,
249 | "metadata": {},
250 | "outputs": [],
251 | "source": [
252 | "ys = [Flux.onehot(label, 0:9) for label in labels[1:5000]]"
253 | ]
254 | },
255 | {
256 | "cell_type": "markdown",
257 | "metadata": {},
258 | "source": [
259 | "#### Create the batched matrices for efficiency"
260 | ]
261 | },
262 | {
263 | "cell_type": "markdown",
264 | "metadata": {},
265 | "source": [
266 | "Let's also create a function so we can easily create independent batches from arbitrary segments of the original dataset."
267 | ]
268 | },
269 | {
270 | "cell_type": "code",
271 | "execution_count": null,
272 | "metadata": {},
273 | "outputs": [],
274 | "source": [
275 | "function create_batch(r)\n",
276 | " xs = [preprocess(img) for img in images[r]]\n",
277 | " ys = [Flux.onehot(label, 0:9) for label in labels[r]]\n",
278 | " return (Flux.batch(xs), Flux.batch(ys))\n",
279 | "end"
280 | ]
281 | },
282 | {
283 | "cell_type": "markdown",
284 | "metadata": {},
285 | "source": [
286 | "We'll train our model on the first 5000 images."
287 | ]
288 | },
289 | {
290 | "cell_type": "code",
291 | "execution_count": null,
292 | "metadata": {},
293 | "outputs": [],
294 | "source": [
295 | "trainbatch = create_batch(1:5000)"
296 | ]
297 | },
298 | {
299 | "cell_type": "code",
300 | "execution_count": null,
301 | "metadata": {},
302 | "outputs": [],
303 | "source": [
304 | "testbatch = create_batch(5001:10000);"
305 | ]
306 | },
307 | {
308 | "cell_type": "markdown",
309 | "metadata": {},
310 | "source": [
311 | "## Setting up the neural network"
312 | ]
313 | },
314 | {
315 | "cell_type": "markdown",
316 | "metadata": {},
317 | "source": [
318 | "Now we must set up a neural network. Since the data is complicated, we may expect to need several layers.\n",
319 | "But we can start with a single layer.\n",
320 | "\n",
321 | "- The network will take as inputs the vectors $\\mathbf{x}^{(i)}$, so the input layer has $n$ nodes.\n",
322 | "\n",
323 | "- The output will be a one-hot vector encoding the digit from 1 to 9 or 0 that is desired. There are 10 possible categories, so we need an output layer of size 10.\n",
324 | "\n",
325 | "It is then our task as neural network designers to insert layers between these input and output layers, whose weights will be tuned during the learning process. *This is an art, not a science*! But major advances have come from finding a good structure for the network."
326 | ]
327 | },
328 | {
329 | "cell_type": "code",
330 | "execution_count": null,
331 | "metadata": {},
332 | "outputs": [],
333 | "source": [
334 | "model = Chain(Dense(n_inputs, n_outputs, identity), softmax)"
335 | ]
336 | },
337 | {
338 | "cell_type": "code",
339 | "execution_count": null,
340 | "metadata": {},
341 | "outputs": [],
342 | "source": [
343 | "L(x,y) = Flux.crossentropy(model(x), y)"
344 | ]
345 | },
346 | {
347 | "cell_type": "code",
348 | "execution_count": null,
349 | "metadata": {},
350 | "outputs": [],
351 | "source": [
352 | "opt = Descent()"
353 | ]
354 | },
355 | {
356 | "cell_type": "markdown",
357 | "metadata": {},
358 | "source": [
359 | "## Training"
360 | ]
361 | },
362 | {
363 | "cell_type": "code",
364 | "execution_count": null,
365 | "metadata": {},
366 | "outputs": [],
367 | "source": [
368 | "@time Flux.train!(L, params(model), [trainbatch], opt)\n",
369 | "@time Flux.train!(L, params(model), [trainbatch], opt)"
370 | ]
371 | },
372 | {
373 | "cell_type": "markdown",
374 | "metadata": {},
375 | "source": [
376 | "We can see what the total current loss is (after training just a handful to times above):"
377 | ]
378 | },
379 | {
380 | "cell_type": "code",
381 | "execution_count": null,
382 | "metadata": {},
383 | "outputs": [],
384 | "source": [
385 | "L(trainbatch...)"
386 | ]
387 | },
388 | {
389 | "cell_type": "code",
390 | "execution_count": null,
391 | "metadata": {},
392 | "outputs": [],
393 | "source": [
394 | "@time Flux.train!(L, params(model), [trainbatch], opt)"
395 | ]
396 | },
397 | {
398 | "cell_type": "code",
399 | "execution_count": null,
400 | "metadata": {},
401 | "outputs": [],
402 | "source": [
403 | "L(trainbatch...)"
404 | ]
405 | },
406 | {
407 | "cell_type": "markdown",
408 | "metadata": {},
409 | "source": [
410 | "### Using callbacks"
411 | ]
412 | },
413 | {
414 | "cell_type": "markdown",
415 | "metadata": {},
416 | "source": [
417 | "The `train!` function can take an optional keyword argument, `cb` (short for \"**c**all**b**ack\"). A callback function is a function that you provide as an argument to a function `f`, which \"calls back\" your function every so often.\n",
418 | "\n",
419 | "This provides the possibility to provide a function that is called at each step or every so often during the training process.\n",
420 | "A common use case is to provide a visual trace of the training process by printing out the current value of the `loss` function:"
421 | ]
422 | },
423 | {
424 | "cell_type": "code",
425 | "execution_count": null,
426 | "metadata": {},
427 | "outputs": [],
428 | "source": [
429 | "callback() = @show(L(trainbatch...))\n",
430 | "\n",
431 | "Flux.train!(L, params(model), Iterators.repeated(trainbatch, 3), opt; cb = callback)"
432 | ]
433 | },
434 | {
435 | "cell_type": "markdown",
436 | "metadata": {},
437 | "source": [
438 | "However, it is expensive to calculate the complete `loss` function and it is not necessary to output it every step. So `Flux` also provides a function `throttle`, that provides a mechanism to call a given function at most once every certain number of seconds:"
439 | ]
440 | },
441 | {
442 | "cell_type": "code",
443 | "execution_count": null,
444 | "metadata": {},
445 | "outputs": [],
446 | "source": [
447 | "Flux.train!(L, params(model), Iterators.repeated(trainbatch, 40), opt; cb = Flux.throttle(callback, 1))"
448 | ]
449 | },
450 | {
451 | "cell_type": "markdown",
452 | "metadata": {},
453 | "source": [
454 | "Of course, that's just measuring the loss over the same data it's training on. It'd be more representative to test against novel data. In fact, let's track the performance of both as we continue training our model. In order to do so, we need to create a batch of test data."
455 | ]
456 | },
457 | {
458 | "cell_type": "code",
459 | "execution_count": null,
460 | "metadata": {},
461 | "outputs": [],
462 | "source": [
463 | "using Printf\n",
464 | "function show_loss()\n",
465 | " train_loss = L(trainbatch...).data\n",
466 | " test_loss = L(testbatch...).data\n",
467 | " @printf(\"train loss = %.3f, test loss = %.3f\\n\", train_loss, test_loss)\n",
468 | "end"
469 | ]
470 | },
471 | {
472 | "cell_type": "code",
473 | "execution_count": null,
474 | "metadata": {},
475 | "outputs": [],
476 | "source": [
477 | "Flux.train!(L, params(model), Iterators.repeated(trainbatch, 100), opt;\n",
478 | " cb = Flux.throttle(show_loss, 1))"
479 | ]
480 | },
481 | {
482 | "cell_type": "markdown",
483 | "metadata": {},
484 | "source": [
485 | "## Testing phase"
486 | ]
487 | },
488 | {
489 | "cell_type": "markdown",
490 | "metadata": {},
491 | "source": [
492 | "We now have trained a model, i.e. we have found the parameters `W` and `b` for the network layer(s). In order to **test** if the learning procedure was really successful, we check how well the resulting trained network performs when we test it with images that the network has not yet seen!\n",
493 | "\n",
494 | "Often, a dataset is split up into \"training data\" and \"test (or validation) data\" for this purpose, and indeed the MNIST data set has a separate pool of training data. We can instead use the images that we have not included in our reduced training process."
495 | ]
496 | },
497 | {
498 | "cell_type": "code",
499 | "execution_count": null,
500 | "metadata": {},
501 | "outputs": [],
502 | "source": [
503 | "i = 5001\n",
504 | "display(images[i])\n",
505 | "display(labels[i])\n",
506 | "model(preprocess(images[i]))"
507 | ]
508 | },
509 | {
510 | "cell_type": "markdown",
511 | "metadata": {},
512 | "source": [
513 | "## Evaluation\n",
514 | "\n",
515 | "What percent of images are we correctly classifying if we take the highest element to be the chosen answer?"
516 | ]
517 | },
518 | {
519 | "cell_type": "code",
520 | "execution_count": null,
521 | "metadata": {},
522 | "outputs": [],
523 | "source": [
524 | "prediction(i) = findmax(model(preprocess(images[i])))[2]-1 # returns (max_value, index)"
525 | ]
526 | },
527 | {
528 | "cell_type": "code",
529 | "execution_count": null,
530 | "metadata": {},
531 | "outputs": [],
532 | "source": [
533 | "sum(prediction(i) == labels[i] for i in 1:5000)/5000"
534 | ]
535 | },
536 | {
537 | "cell_type": "code",
538 | "execution_count": null,
539 | "metadata": {},
540 | "outputs": [],
541 | "source": [
542 | "sum(prediction(i) == labels[i] for i in 5001:10000)/5000"
543 | ]
544 | },
545 | {
546 | "cell_type": "markdown",
547 | "metadata": {},
548 | "source": [
549 | "## Improving the prediction"
550 | ]
551 | },
552 | {
553 | "cell_type": "markdown",
554 | "metadata": {},
555 | "source": [
556 | "So far we have used a single layer. In order to improve the prediction, we probably need to use more layers. Try adding more layers yourself and see how the performance changes."
557 | ]
558 | },
559 | {
560 | "cell_type": "code",
561 | "execution_count": null,
562 | "metadata": {},
563 | "outputs": [],
564 | "source": [
565 | "n_hidden = 20\n",
566 | "model = Chain(Dense(n_inputs, n_hidden, relu),\n",
567 | " Dense(n_hidden, n_outputs, identity), softmax)\n",
568 | "L(x,y) = Flux.crossentropy(model(x), y)\n",
569 | "opt = ADAM()"
570 | ]
571 | },
572 | {
573 | "cell_type": "code",
574 | "execution_count": null,
575 | "metadata": {},
576 | "outputs": [],
577 | "source": [
578 | "train_loss = Float64[]\n",
579 | "test_loss = Float64[]\n",
580 | "Flux.train!(L, params(model), Iterators.repeated(trainbatch, 500), opt;\n",
581 | " cb = Flux.throttle(show_loss, 1))"
582 | ]
583 | },
584 | {
585 | "cell_type": "code",
586 | "execution_count": null,
587 | "metadata": {},
588 | "outputs": [],
589 | "source": [
590 | "display(images[7010])\n",
591 | "labels[7010]"
592 | ]
593 | },
594 | {
595 | "cell_type": "code",
596 | "execution_count": null,
597 | "metadata": {},
598 | "outputs": [],
599 | "source": [
600 | "model(preprocess(images[7010]))"
601 | ]
602 | },
603 | {
604 | "cell_type": "markdown",
605 | "metadata": {},
606 | "source": [
607 | "## What about image structure?"
608 | ]
609 | },
610 | {
611 | "cell_type": "markdown",
612 | "metadata": {},
613 | "source": [
614 | "As a final note, notice that our model doesn't take into account any aspect of the image's connected-ness."
615 | ]
616 | },
617 | {
618 | "cell_type": "code",
619 | "execution_count": null,
620 | "metadata": {},
621 | "outputs": [],
622 | "source": [
623 | "using Random\n",
624 | "p = randperm(28)\n",
625 | "images[1][p,p]"
626 | ]
627 | },
628 | {
629 | "cell_type": "code",
630 | "execution_count": null,
631 | "metadata": {},
632 | "outputs": [],
633 | "source": [
634 | "show(preprocess(images[1]))"
635 | ]
636 | },
637 | {
638 | "cell_type": "code",
639 | "execution_count": null,
640 | "metadata": {},
641 | "outputs": [],
642 | "source": [
643 | "?Conv"
644 | ]
645 | }
646 | ],
647 | "metadata": {
648 | "@webio": {
649 | "lastCommId": null,
650 | "lastKernelId": null
651 | },
652 | "kernelspec": {
653 | "display_name": "Julia 1.0.3",
654 | "language": "julia",
655 | "name": "julia-1.0"
656 | },
657 | "language_info": {
658 | "file_extension": ".jl",
659 | "mimetype": "application/julia",
660 | "name": "julia",
661 | "version": "1.0.3"
662 | }
663 | },
664 | "nbformat": 4,
665 | "nbformat_minor": 3
666 | }
667 |
--------------------------------------------------------------------------------
/02-DP-linear-fit.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "# Only if using JuliaBox\n",
10 | "pop!(DEPOT_PATH)\n",
11 | "push!(DEPOT_PATH, \"/home/jrun/.julia2/\")"
12 | ]
13 | },
14 | {
15 | "cell_type": "markdown",
16 | "metadata": {},
17 | "source": [
18 | "## Linear fit by differentiable programming"
19 | ]
20 | },
21 | {
22 | "cell_type": "code",
23 | "execution_count": null,
24 | "metadata": {},
25 | "outputs": [],
26 | "source": [
27 | "using Zygote, LinearAlgebra"
28 | ]
29 | },
30 | {
31 | "cell_type": "code",
32 | "execution_count": null,
33 | "metadata": {},
34 | "outputs": [],
35 | "source": [
36 | "# Define a data type representing our model (optional)\n",
37 | "struct LinearRegression\n",
38 | " weights::Matrix\n",
39 | " bias::Array{Float64,0}\n",
40 | "end\n",
41 | "LinearRegression(nparams) = LinearRegression(randn(1, nparams), fill(0.0))"
42 | ]
43 | },
44 | {
45 | "cell_type": "code",
46 | "execution_count": null,
47 | "metadata": {},
48 | "outputs": [],
49 | "source": [
50 | "# Define how to \"run\" the model forwards to generate a prediction\n",
51 | "function predict(model::LinearRegression, X)\n",
52 | " return model.weights * X .+ model.bias[]\n",
53 | "end"
54 | ]
55 | },
56 | {
57 | "cell_type": "code",
58 | "execution_count": null,
59 | "metadata": {},
60 | "outputs": [],
61 | "source": [
62 | "# Compute the \"loss\" to be minimized\n",
63 | "function loss(model::LinearRegression, X, Y)\n",
64 | " return norm(predict(model, X) .- Y, 2)\n",
65 | "end"
66 | ]
67 | },
68 | {
69 | "cell_type": "code",
70 | "execution_count": null,
71 | "metadata": {},
72 | "outputs": [],
73 | "source": [
74 | "# Set up a sample problem.\n",
75 | "# \"Ground truth\" values we will try to learn/recover\n",
76 | "weights_gt = [1.0, 2.7, 0.3, 1.2]'\n",
77 | "bias_gt = 0.4"
78 | ]
79 | },
80 | {
81 | "cell_type": "code",
82 | "execution_count": null,
83 | "metadata": {},
84 | "outputs": [],
85 | "source": [
86 | "# Generate a fake dataset of many observations\n",
87 | "X = randn(length(weights_gt), 10000)\n",
88 | "Y = weights_gt * X .+ bias_gt\n",
89 | "# Add a little bit of noise to `X` so that we do not have an exact solution,\n",
90 | "# but must instead do a least-squares fit\n",
91 | "X .+= 0.01 .* randn(size(X))"
92 | ]
93 | },
94 | {
95 | "cell_type": "code",
96 | "execution_count": null,
97 | "metadata": {},
98 | "outputs": [],
99 | "source": [
100 | "# Start with a random model to train\n",
101 | "model = LinearRegression(size(X, 1))"
102 | ]
103 | },
104 | {
105 | "cell_type": "code",
106 | "execution_count": null,
107 | "metadata": {},
108 | "outputs": [],
109 | "source": [
110 | "# Calculate gradient on `model` for the first example in our training set\n",
111 | "grads = Zygote.gradient(m -> loss(m, X[:, 1], Y[1]), model)"
112 | ]
113 | },
114 | {
115 | "cell_type": "code",
116 | "execution_count": null,
117 | "metadata": {},
118 | "outputs": [],
119 | "source": [
120 | "# The `grads` object is a tuple containing one element per argument to\n",
121 | "# `gradient`, so we take the first one to get the gradient on `model`\n",
122 | "grads = grads[1]"
123 | ]
124 | },
125 | {
126 | "cell_type": "code",
127 | "execution_count": null,
128 | "metadata": {},
129 | "outputs": [],
130 | "source": [
131 | "# Define an update rule that will allow us to modify the weights\n",
132 | "# of our model a bit according to the gradients\n",
133 | "function sgd_update!(model::LinearRegression, grads, η = 0.001)\n",
134 | " model.weights .-= η .* grads.weights\n",
135 | " model.bias .-= η .* grads.bias\n",
136 | "end"
137 | ]
138 | },
139 | {
140 | "cell_type": "code",
141 | "execution_count": null,
142 | "metadata": {},
143 | "outputs": [],
144 | "source": [
145 | "for idx in 1:size(X, 2)\n",
146 | " grads = Zygote.gradient(m -> loss(m, X[:, idx], Y[idx]), model)[1]\n",
147 | " sgd_update!(model, grads)\n",
148 | "end"
149 | ]
150 | },
151 | {
152 | "cell_type": "code",
153 | "execution_count": null,
154 | "metadata": {},
155 | "outputs": [],
156 | "source": [
157 | "model"
158 | ]
159 | },
160 | {
161 | "cell_type": "code",
162 | "execution_count": null,
163 | "metadata": {},
164 | "outputs": [],
165 | "source": [
166 | "weights_gt"
167 | ]
168 | },
169 | {
170 | "cell_type": "code",
171 | "execution_count": null,
172 | "metadata": {},
173 | "outputs": [],
174 | "source": []
175 | }
176 | ],
177 | "metadata": {
178 | "@webio": {
179 | "lastCommId": null,
180 | "lastKernelId": null
181 | },
182 | "kernelspec": {
183 | "display_name": "Julia 1.0.3",
184 | "language": "julia",
185 | "name": "julia-1.0"
186 | },
187 | "language_info": {
188 | "file_extension": ".jl",
189 | "mimetype": "application/julia",
190 | "name": "julia",
191 | "version": "1.0.3"
192 | }
193 | },
194 | "nbformat": 4,
195 | "nbformat_minor": 2
196 | }
197 |
--------------------------------------------------------------------------------
/03-Differentiable-trebuchet.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "# Only if using JuliaBox\n",
10 | "pop!(DEPOT_PATH)\n",
11 | "push!(DEPOT_PATH, \"/home/jrun/.julia2/\")"
12 | ]
13 | },
14 | {
15 | "cell_type": "markdown",
16 | "metadata": {},
17 | "source": [
18 | "# How To Aim Your Flagon"
19 | ]
20 | },
21 | {
22 | "cell_type": "markdown",
23 | "metadata": {},
24 | "source": [
25 | "## Loading your Trebuchet"
26 | ]
27 | },
28 | {
29 | "cell_type": "markdown",
30 | "metadata": {},
31 | "source": [
32 | "Today we practice the ancient medieval art of throwing stuff. First up, we load our trebuchet simulator, Trebuchet.jl."
33 | ]
34 | },
35 | {
36 | "cell_type": "code",
37 | "execution_count": null,
38 | "metadata": {
39 | "scrolled": false
40 | },
41 | "outputs": [],
42 | "source": [
43 | "using Trebuchet"
44 | ]
45 | },
46 | {
47 | "cell_type": "markdown",
48 | "metadata": {},
49 | "source": [
50 | "We can see what the trebuchet looks like, by explicitly creating a trebuchet state, running a simulation, and visualising the trajectory."
51 | ]
52 | },
53 | {
54 | "cell_type": "code",
55 | "execution_count": null,
56 | "metadata": {},
57 | "outputs": [],
58 | "source": [
59 | "t = TrebuchetState()\n",
60 | "simulate(t)\n",
61 | "visualise(t)"
62 | ]
63 | },
64 | {
65 | "cell_type": "markdown",
66 | "metadata": {},
67 | "source": [
68 | "For training and optimisation, we don't need the whole visualisation, just a simple function that accepts and produces numbers. The `shoot` function just takes a wind speed, angle of release and counterweight mass, and tells us how far the projectile got."
69 | ]
70 | },
71 | {
72 | "cell_type": "code",
73 | "execution_count": null,
74 | "metadata": {},
75 | "outputs": [],
76 | "source": [
77 | "function shoot(wind, angle, weight)\n",
78 | " Trebuchet.shoot((wind, Trebuchet.deg2rad(angle), weight))[2]\n",
79 | "end"
80 | ]
81 | },
82 | {
83 | "cell_type": "code",
84 | "execution_count": null,
85 | "metadata": {},
86 | "outputs": [],
87 | "source": [
88 | "shoot(0, 30, 400)"
89 | ]
90 | },
91 | {
92 | "cell_type": "markdown",
93 | "metadata": {},
94 | "source": [
95 | "It's worth playing with these parameters to see the impact they have. How far can you throw the projectile, tweaking only the angle of release?"
96 | ]
97 | },
98 | {
99 | "cell_type": "markdown",
100 | "metadata": {},
101 | "source": [
102 | "There's actually a much better way of aiming the trebuchet. Let's load up a machine learning library, Flux, and see what we can do."
103 | ]
104 | },
105 | {
106 | "cell_type": "code",
107 | "execution_count": null,
108 | "metadata": {},
109 | "outputs": [],
110 | "source": [
111 | "pathof(Trebuchet)"
112 | ]
113 | },
114 | {
115 | "cell_type": "code",
116 | "execution_count": null,
117 | "metadata": {},
118 | "outputs": [],
119 | "source": [
120 | "using Flux, Trebuchet\n",
121 | "using Flux.Tracker: gradient, forwarddiff"
122 | ]
123 | },
124 | {
125 | "cell_type": "markdown",
126 | "metadata": {},
127 | "source": [
128 | "Firstly, we're going to wrap `shoot` to take a _parameter vector_ (just a list of the three numbers we're interested in). There's also a call to `forwarddiff` here, which tells Flux to differentiate the trebuchet itself using forward mode. The number of parameters is small, so forward mode will be the most efficient way to do it. Otherwise Flux defaults to reverse mode."
129 | ]
130 | },
131 | {
132 | "cell_type": "code",
133 | "execution_count": null,
134 | "metadata": {},
135 | "outputs": [],
136 | "source": [
137 | "shoot(ps) = forwarddiff(p -> shoot(p...), ps)"
138 | ]
139 | },
140 | {
141 | "cell_type": "markdown",
142 | "metadata": {},
143 | "source": [
144 | "We can get a distance as usual."
145 | ]
146 | },
147 | {
148 | "cell_type": "code",
149 | "execution_count": null,
150 | "metadata": {},
151 | "outputs": [],
152 | "source": [
153 | "shoot([0, 45, 200])"
154 | ]
155 | },
156 | {
157 | "cell_type": "markdown",
158 | "metadata": {},
159 | "source": [
160 | "But we can also get something much more interesting: *gradients* for each of those parameters with respect to distance."
161 | ]
162 | },
163 | {
164 | "cell_type": "code",
165 | "execution_count": null,
166 | "metadata": {},
167 | "outputs": [],
168 | "source": [
169 | "gradient(shoot, [0, 45, 200])"
170 | ]
171 | },
172 | {
173 | "cell_type": "markdown",
174 | "metadata": {},
175 | "source": [
176 | "What does these numbers mean? The gradient tells us, very roughly, that if we increase a parameter – let's say we make wind speed 1 m/s stronger – distance will also increase by about 4 metres. Let's try that."
177 | ]
178 | },
179 | {
180 | "cell_type": "code",
181 | "execution_count": null,
182 | "metadata": {},
183 | "outputs": [],
184 | "source": [
185 | "shoot([1, 45, 200])"
186 | ]
187 | },
188 | {
189 | "cell_type": "markdown",
190 | "metadata": {},
191 | "source": [
192 | "Lo and behold, this is indeed about four metres further!"
193 | ]
194 | },
195 | {
196 | "cell_type": "code",
197 | "execution_count": null,
198 | "metadata": {},
199 | "outputs": [],
200 | "source": [
201 | "shoot([1, 45, 200]) - shoot([0, 45, 200])"
202 | ]
203 | },
204 | {
205 | "cell_type": "markdown",
206 | "metadata": {},
207 | "source": [
208 | "So this seems like very useful information if we're trying to aim, or maximise distance. Notice that our gradient for the release angle is negative – increasing angle will decrease distance, so in other words we should probably *decrease* angle if we want more distance. Let's try that."
209 | ]
210 | },
211 | {
212 | "cell_type": "code",
213 | "execution_count": null,
214 | "metadata": {},
215 | "outputs": [],
216 | "source": [
217 | "shoot([0, 10, 200])"
218 | ]
219 | },
220 | {
221 | "cell_type": "markdown",
222 | "metadata": {},
223 | "source": [
224 | "Oh no, this is actually *less* far than before!"
225 | ]
226 | },
227 | {
228 | "cell_type": "markdown",
229 | "metadata": {},
230 | "source": [
231 | "So if the angle is too shallow, the projectile doesn't spend enough time in the air to gain any distance before hitting the ground. But if it's too high, the projectile doesn't have enough horizontal speed even with lots of time in the air. So we'll have to find a middle ground.\n",
232 | "\n",
233 | "More generally, the lesson here is that the gradient only gives you limited information; it helps us take a small step towards a better aim, and we can keep iterating to get to the best possible aim. For example, we choose a starting angle:"
234 | ]
235 | },
236 | {
237 | "cell_type": "code",
238 | "execution_count": null,
239 | "metadata": {},
240 | "outputs": [],
241 | "source": [
242 | "angle = 45\n",
243 | "shoot([0, angle, 200])"
244 | ]
245 | },
246 | {
247 | "cell_type": "markdown",
248 | "metadata": {},
249 | "source": [
250 | "Get a gradient for `angle` alone:"
251 | ]
252 | },
253 | {
254 | "cell_type": "code",
255 | "execution_count": null,
256 | "metadata": {},
257 | "outputs": [],
258 | "source": [
259 | "dangle = gradient(angle -> shoot(Tracker.collect([0, angle, 200])), angle)[1] |> Flux.data"
260 | ]
261 | },
262 | {
263 | "cell_type": "markdown",
264 | "metadata": {},
265 | "source": [
266 | "Update the angle, using the learning rate η:"
267 | ]
268 | },
269 | {
270 | "cell_type": "code",
271 | "execution_count": null,
272 | "metadata": {},
273 | "outputs": [],
274 | "source": [
275 | "η = 10\n",
276 | "angle += η*dangle"
277 | ]
278 | },
279 | {
280 | "cell_type": "code",
281 | "execution_count": null,
282 | "metadata": {},
283 | "outputs": [],
284 | "source": [
285 | "shoot([0, angle, 200])"
286 | ]
287 | },
288 | {
289 | "cell_type": "markdown",
290 | "metadata": {},
291 | "source": [
292 | "Now we just lather, rinse and repeat! Ok, maybe we should write a loop to automate this a bit."
293 | ]
294 | },
295 | {
296 | "cell_type": "code",
297 | "execution_count": null,
298 | "metadata": {},
299 | "outputs": [],
300 | "source": [
301 | "for i = 1:10\n",
302 | " dangle = gradient(angle -> shoot(Tracker.collect([0, angle, 200])), angle)[1] |> Flux.data\n",
303 | " angle += η*dangle\n",
304 | " @show angle\n",
305 | "end\n",
306 | "shoot([0, angle, 200])"
307 | ]
308 | },
309 | {
310 | "cell_type": "markdown",
311 | "metadata": {},
312 | "source": [
313 | "Notice how the change in the angle slows down as things converge. Turns out the best angle is about 30 degrees, and we can hit about 90 metres.\n",
314 | "\n",
315 | "We can make this nicely repeatable and get the best angle for any given wind speed."
316 | ]
317 | },
318 | {
319 | "cell_type": "code",
320 | "execution_count": null,
321 | "metadata": {},
322 | "outputs": [],
323 | "source": [
324 | "function best_angle(wind)\n",
325 | " angle = 45\n",
326 | " objective(angle) = shoot(Tracker.collect([wind, angle, 200]))\n",
327 | " for i = 1:10\n",
328 | " dangle = gradient(objective, angle)[1] |> Flux.data\n",
329 | " angle += η*dangle\n",
330 | " end\n",
331 | " return angle\n",
332 | "end"
333 | ]
334 | },
335 | {
336 | "cell_type": "code",
337 | "execution_count": null,
338 | "metadata": {},
339 | "outputs": [],
340 | "source": [
341 | "best_angle(0)"
342 | ]
343 | },
344 | {
345 | "cell_type": "code",
346 | "execution_count": null,
347 | "metadata": {},
348 | "outputs": [],
349 | "source": [
350 | "best_angle(10)"
351 | ]
352 | },
353 | {
354 | "cell_type": "code",
355 | "execution_count": null,
356 | "metadata": {},
357 | "outputs": [],
358 | "source": [
359 | "best_angle(-10)"
360 | ]
361 | },
362 | {
363 | "cell_type": "markdown",
364 | "metadata": {},
365 | "source": [
366 | "It turns out that if the wind is on our side, we should just throw the projectile upwards and let it get blown along. If the wind is strong against us, just chuck that stone right into it."
367 | ]
368 | },
369 | {
370 | "cell_type": "code",
371 | "execution_count": null,
372 | "metadata": {},
373 | "outputs": [],
374 | "source": [
375 | "t = TrebuchetState(release_angle = deg2rad(19), wind_speed = -10)\n",
376 | "simulate(t)\n",
377 | "visualise(t)"
378 | ]
379 | },
380 | {
381 | "cell_type": "markdown",
382 | "metadata": {},
383 | "source": [
384 | "## Accuracy Matters"
385 | ]
386 | },
387 | {
388 | "cell_type": "markdown",
389 | "metadata": {},
390 | "source": [
391 | "In optimisation terms, we just created an objective (distance) and tried to maximise that objective. Flinging boulders as far as possible has its moments, but lacks a certain subtlety. What if we instead want to hit a precise target?"
392 | ]
393 | },
394 | {
395 | "cell_type": "code",
396 | "execution_count": null,
397 | "metadata": {},
398 | "outputs": [],
399 | "source": [
400 | "t = TrebuchetState()\n",
401 | "simulate(t)\n",
402 | "visualise(t, 50)"
403 | ]
404 | },
405 | {
406 | "cell_type": "markdown",
407 | "metadata": {},
408 | "source": [
409 | "The way to do this is to state the problem in terms of maximising, or minisming, some number – the objective. In this case, an easy way to come up with an objective is to take the difference from our target (gets closer to 0 as aim gets better) and square it (so it's always positive: 0 is the lowest *and* best possible score)."
410 | ]
411 | },
412 | {
413 | "cell_type": "markdown",
414 | "metadata": {},
415 | "source": [
416 | "Here's a modified `best_angle` function that takes a target and tells us the distance it acheived."
417 | ]
418 | },
419 | {
420 | "cell_type": "code",
421 | "execution_count": null,
422 | "metadata": {},
423 | "outputs": [],
424 | "source": [
425 | "η = 0.1\n",
426 | "function best_angle(wind, target)\n",
427 | " angle = 45\n",
428 | " objective(angle) = (shoot(Tracker.collect([wind, angle, 200])) - target)^2\n",
429 | " for i = 1:30\n",
430 | " dangle = gradient(objective, angle)[1] |> Flux.data\n",
431 | " angle -= η*dangle\n",
432 | " end\n",
433 | " return angle, shoot([wind, angle, 200])\n",
434 | "end"
435 | ]
436 | },
437 | {
438 | "cell_type": "markdown",
439 | "metadata": {},
440 | "source": [
441 | "It's pretty accurate!"
442 | ]
443 | },
444 | {
445 | "cell_type": "code",
446 | "execution_count": null,
447 | "metadata": {},
448 | "outputs": [],
449 | "source": [
450 | "best_angle(0, 50)"
451 | ]
452 | },
453 | {
454 | "cell_type": "markdown",
455 | "metadata": {},
456 | "source": [
457 | "Even when we try to push it, by making wind really strong."
458 | ]
459 | },
460 | {
461 | "cell_type": "code",
462 | "execution_count": null,
463 | "metadata": {
464 | "scrolled": true
465 | },
466 | "outputs": [],
467 | "source": [
468 | "best_angle(-20, 35)"
469 | ]
470 | },
471 | {
472 | "cell_type": "code",
473 | "execution_count": null,
474 | "metadata": {},
475 | "outputs": [],
476 | "source": [
477 | "t = TrebuchetState(release_angle = deg2rad(21.8), weight = 200, wind_speed = -20)\n",
478 | "simulate(t)\n",
479 | "visualise(t, 35)"
480 | ]
481 | },
482 | {
483 | "cell_type": "markdown",
484 | "metadata": {},
485 | "source": [
486 | "## Siege Weapon Autopilot"
487 | ]
488 | },
489 | {
490 | "cell_type": "markdown",
491 | "metadata": {},
492 | "source": [
493 | "Finally, we go one level more meta by training a neural network to aim the trebuchet for us. Rather than solving a whole optimisation problem every time we want to aim, we can just ask the network for good parameters and get them in constant time.\n",
494 | "\n",
495 | "Here's a simple multi layer perceptron. Its input is two parameters (wind speed and target) and its output is two more (release angle and counterweight mass)."
496 | ]
497 | },
498 | {
499 | "cell_type": "code",
500 | "execution_count": null,
501 | "metadata": {},
502 | "outputs": [],
503 | "source": [
504 | "model = Chain(Dense(2, 16, σ),\n",
505 | " Dense(16, 64, σ),\n",
506 | " Dense(64, 16, σ),\n",
507 | " Dense(16, 2)) |> f64\n",
508 | "\n",
509 | "θ = params(model)\n",
510 | "\n",
511 | "function aim(wind, target)\n",
512 | " angle, weight = model([wind, target])\n",
513 | " angle = σ(angle)*90\n",
514 | " weight = weight + 200\n",
515 | " angle, weight\n",
516 | "end\n",
517 | "\n",
518 | "distance(wind, target) = shoot(Tracker.collect([wind, aim(wind, target)...]))"
519 | ]
520 | },
521 | {
522 | "cell_type": "markdown",
523 | "metadata": {},
524 | "source": [
525 | "The model's initial guesses will be fairly random, and miss the mark."
526 | ]
527 | },
528 | {
529 | "cell_type": "code",
530 | "execution_count": null,
531 | "metadata": {},
532 | "outputs": [],
533 | "source": [
534 | "aim(0, 70)"
535 | ]
536 | },
537 | {
538 | "cell_type": "code",
539 | "execution_count": null,
540 | "metadata": {},
541 | "outputs": [],
542 | "source": [
543 | "distance(0, 70)"
544 | ]
545 | },
546 | {
547 | "cell_type": "markdown",
548 | "metadata": {},
549 | "source": [
550 | "However, just as before, we can define an objective – or loss – and get gradients."
551 | ]
552 | },
553 | {
554 | "cell_type": "code",
555 | "execution_count": null,
556 | "metadata": {},
557 | "outputs": [],
558 | "source": [
559 | "function loss(wind, target)\n",
560 | " try\n",
561 | " (distance(wind, target) - target)^2\n",
562 | " catch e\n",
563 | " # Roots.jl sometimes give convergence errors, ignore them\n",
564 | " param(0)\n",
565 | " end\n",
566 | "end\n",
567 | "\n",
568 | "loss(0, 70)"
569 | ]
570 | },
571 | {
572 | "cell_type": "markdown",
573 | "metadata": {},
574 | "source": [
575 | "This time, though, we'll get gradients for the *model parameters*, and updating these will improve the network's accuracy. This works because we're able to differentiate the *whole program*; the backwards pass propagates errors through the trebuchet simulator and then through the ML model."
576 | ]
577 | },
578 | {
579 | "cell_type": "code",
580 | "execution_count": null,
581 | "metadata": {},
582 | "outputs": [],
583 | "source": [
584 | "dθ = gradient(θ) do\n",
585 | " loss(0, 70)\n",
586 | "end\n",
587 | "dθ[model[1].W]"
588 | ]
589 | },
590 | {
591 | "cell_type": "code",
592 | "execution_count": null,
593 | "metadata": {},
594 | "outputs": [],
595 | "source": [
596 | "DIST = (20, 100) # Maximum target distance\n",
597 | "SPEED = 5 # Maximum wind speed\n",
598 | "\n",
599 | "lerp(x, lo, hi) = x*(hi-lo)+lo\n",
600 | "\n",
601 | "randtarget() = (randn() * SPEED, lerp(rand(), DIST...))"
602 | ]
603 | },
604 | {
605 | "cell_type": "code",
606 | "execution_count": null,
607 | "metadata": {},
608 | "outputs": [],
609 | "source": [
610 | "using Statistics\n",
611 | "\n",
612 | "meanloss() = mean(sqrt(loss(randtarget()...)) for i = 1:100)\n",
613 | "\n",
614 | "opt = ADAM()\n",
615 | "\n",
616 | "dataset = (randtarget() for i = 1:10_000)\n",
617 | "\n",
618 | "Flux.train!(loss, θ, dataset, opt, cb = Flux.throttle(() -> @show(meanloss()), 10))"
619 | ]
620 | },
621 | {
622 | "cell_type": "markdown",
623 | "metadata": {},
624 | "source": [
625 | "After only a few minutes of training, we're getting solid accuracy, even on hard wind speeds and targets. You can run the training loop again to improve the accuracy even further."
626 | ]
627 | },
628 | {
629 | "cell_type": "code",
630 | "execution_count": null,
631 | "metadata": {},
632 | "outputs": [],
633 | "source": [
634 | "wind, target = -10, 50\n",
635 | "angle, mass = Flux.data.(aim(wind, target))\n",
636 | "t = TrebuchetState(release_angle = deg2rad(angle), weight = mass, wind_speed = wind)\n",
637 | "simulate(t)\n",
638 | "visualise(t, target)"
639 | ]
640 | },
641 | {
642 | "cell_type": "markdown",
643 | "metadata": {},
644 | "source": [
645 | "Notice that aiming with a neural net in one shot is significantly faster than solving the optimisation problem; and we only have a small loss in accuracy."
646 | ]
647 | },
648 | {
649 | "cell_type": "code",
650 | "execution_count": null,
651 | "metadata": {},
652 | "outputs": [],
653 | "source": [
654 | "@time aim(wind, target)"
655 | ]
656 | },
657 | {
658 | "cell_type": "code",
659 | "execution_count": null,
660 | "metadata": {},
661 | "outputs": [],
662 | "source": [
663 | "@time best_angle(wind, target)"
664 | ]
665 | },
666 | {
667 | "cell_type": "code",
668 | "execution_count": null,
669 | "metadata": {},
670 | "outputs": [],
671 | "source": []
672 | }
673 | ],
674 | "metadata": {
675 | "@webio": {
676 | "lastCommId": "38f60cc138e94c4a8adb6511edcf9aaa",
677 | "lastKernelId": "0eaaa3e6-be49-4e62-b783-5e324970c443"
678 | },
679 | "kernelspec": {
680 | "display_name": "Julia 1.0.3",
681 | "language": "julia",
682 | "name": "julia-1.0"
683 | },
684 | "language_info": {
685 | "file_extension": ".jl",
686 | "mimetype": "application/julia",
687 | "name": "julia",
688 | "version": "1.0.3"
689 | }
690 | },
691 | "nbformat": 4,
692 | "nbformat_minor": 2
693 | }
694 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2019 Julia Computing, Inc.
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # ODSC2019
2 | Material for "Differentiable programming with Flux"
3 |
--------------------------------------------------------------------------------
/msr_2019_06_03/Manifest.toml:
--------------------------------------------------------------------------------
1 | [[AbstractFFTs]]
2 | deps = ["LinearAlgebra"]
3 | git-tree-sha1 = "380e36c66edfa099cd90116b24c1ce8cafccac40"
4 | uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c"
5 | version = "0.4.1"
6 |
7 | [[AbstractTrees]]
8 | deps = ["Markdown", "Test"]
9 | git-tree-sha1 = "6621d9645702c1c4e6970cc6a3eae440c768000b"
10 | uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c"
11 | version = "0.2.1"
12 |
13 | [[Adapt]]
14 | deps = ["LinearAlgebra", "Test"]
15 | git-tree-sha1 = "53d8fec4f662088c1202530e338a11a919407f3b"
16 | uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e"
17 | version = "0.4.2"
18 |
19 | [[ArrayInterface]]
20 | deps = ["Requires", "Test"]
21 | git-tree-sha1 = "6a1a371393e56f5e8d5657fe4da4b11aea0bfbae"
22 | uuid = "4fba245c-0d91-5ea0-9b3e-6abc04ee57a9"
23 | version = "0.1.1"
24 |
25 | [[AssetRegistry]]
26 | deps = ["Distributed", "JSON", "Pidfile", "SHA", "Test"]
27 | git-tree-sha1 = "b25e88db7944f98789130d7b503276bc34bc098e"
28 | uuid = "bf4720bc-e11a-5d0c-854e-bdca1663c893"
29 | version = "0.1.0"
30 |
31 | [[AxisAlgorithms]]
32 | deps = ["LinearAlgebra", "Random", "SparseArrays", "WoodburyMatrices"]
33 | git-tree-sha1 = "a4d07a1c313392a77042855df46c5f534076fab9"
34 | uuid = "13072b0f-2c55-5437-9ae7-d433b7a33950"
35 | version = "1.0.0"
36 |
37 | [[AxisArrays]]
38 | deps = ["Compat", "Dates", "IntervalSets", "IterTools", "Random", "RangeArrays", "Test"]
39 | git-tree-sha1 = "2e2536e9e6f27c4f8d09d8442b61a7ae0b910c28"
40 | uuid = "39de3d68-74b9-583c-8d2d-e117c070f3a9"
41 | version = "0.3.0"
42 |
43 | [[Base64]]
44 | uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
45 |
46 | [[BinDeps]]
47 | deps = ["Compat", "Libdl", "SHA", "URIParser"]
48 | git-tree-sha1 = "12093ca6cdd0ee547c39b1870e0c9c3f154d9ca9"
49 | uuid = "9e28174c-4ba2-5203-b857-d8d62c4213ee"
50 | version = "0.8.10"
51 |
52 | [[BinaryProvider]]
53 | deps = ["Libdl", "SHA"]
54 | git-tree-sha1 = "c7361ce8a2129f20b0e05a89f7070820cfed6648"
55 | uuid = "b99e7846-7c00-51b0-8f62-c81ae34c0232"
56 | version = "0.5.4"
57 |
58 | [[CSSUtil]]
59 | deps = ["Colors", "Compat", "JSON", "Measures", "Pkg", "WebIO"]
60 | git-tree-sha1 = "ff13fd99e4dd54f56eb064815f843bc992a871a2"
61 | uuid = "70588ee8-6100-5070-97c1-3cb50ed05fe8"
62 | version = "0.1.0"
63 |
64 | [[CSTParser]]
65 | deps = ["LibGit2", "Test", "Tokenize"]
66 | git-tree-sha1 = "437c93bc191cd55957b3f8dee7794b6131997c56"
67 | uuid = "00ebfdb7-1f24-5e51-bd34-a7502290713f"
68 | version = "0.5.2"
69 |
70 | [[Calculus]]
71 | deps = ["Compat"]
72 | git-tree-sha1 = "f60954495a7afcee4136f78d1d60350abd37a409"
73 | uuid = "49dc2e85-a5d0-5ad3-a950-438e2897f1b9"
74 | version = "0.4.1"
75 |
76 | [[CatIndices]]
77 | deps = ["CustomUnitRanges", "OffsetArrays", "Test"]
78 | git-tree-sha1 = "254cf73ea369d2e39bfd6c5eb27a2296cfaed68c"
79 | uuid = "aafaddc9-749c-510e-ac4f-586e18779b91"
80 | version = "0.2.0"
81 |
82 | [[CodecZlib]]
83 | deps = ["BinaryProvider", "Libdl", "Test", "TranscodingStreams"]
84 | git-tree-sha1 = "36bbf5374c661054d41410dc53ff752972583b9b"
85 | uuid = "944b1d66-785c-5afd-91f1-9de20f533193"
86 | version = "0.5.2"
87 |
88 | [[ColorTypes]]
89 | deps = ["FixedPointNumbers", "Random"]
90 | git-tree-sha1 = "10050a24b09e8e41b951e9976b109871ce98d965"
91 | uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f"
92 | version = "0.8.0"
93 |
94 | [[ColorVectorSpace]]
95 | deps = ["ColorTypes", "Colors", "FixedPointNumbers", "LinearAlgebra", "SpecialFunctions", "Statistics", "StatsBase"]
96 | git-tree-sha1 = "d626b542f9c3aa22b4f82fe3d36a0ff56fc58601"
97 | uuid = "c3611d14-8923-5661-9e6a-0046d554d3a4"
98 | version = "0.7.0"
99 |
100 | [[Colors]]
101 | deps = ["ColorTypes", "FixedPointNumbers", "InteractiveUtils", "Printf", "Reexport", "Test"]
102 | git-tree-sha1 = "9f0a0210450acb91c730b730a994f8eef1d3d543"
103 | uuid = "5ae59095-9a9b-59fe-a467-6f913c188581"
104 | version = "0.9.5"
105 |
106 | [[CommonSubexpressions]]
107 | deps = ["Test"]
108 | git-tree-sha1 = "efdaf19ab11c7889334ca247ff4c9f7c322817b0"
109 | uuid = "bbf7d656-a473-5ed7-a52c-81e309532950"
110 | version = "0.2.0"
111 |
112 | [[Compat]]
113 | deps = ["Base64", "Dates", "DelimitedFiles", "Distributed", "InteractiveUtils", "LibGit2", "Libdl", "LinearAlgebra", "Markdown", "Mmap", "Pkg", "Printf", "REPL", "Random", "Serialization", "SharedArrays", "Sockets", "SparseArrays", "Statistics", "Test", "UUIDs", "Unicode"]
114 | git-tree-sha1 = "84aa74986c5b9b898b0d1acaf3258741ee64754f"
115 | uuid = "34da2185-b29b-5c13-b0c7-acf172513d20"
116 | version = "2.1.0"
117 |
118 | [[ComputationalResources]]
119 | deps = ["Test"]
120 | git-tree-sha1 = "89e7e7ed20af73d9f78877d2b8d1194e7b6ff13d"
121 | uuid = "ed09eef8-17a6-5b46-8889-db040fac31e3"
122 | version = "0.3.0"
123 |
124 | [[Conda]]
125 | deps = ["Compat", "JSON", "VersionParsing"]
126 | git-tree-sha1 = "b625d802587c2150c279a40a646fba63f9bd8187"
127 | uuid = "8f4d0f93-b110-5947-807f-2305c1781a2d"
128 | version = "1.2.0"
129 |
130 | [[CoordinateTransformations]]
131 | deps = ["Compat", "Rotations", "StaticArrays"]
132 | git-tree-sha1 = "47f05d0b7f4999609f92e657147df000818c1f24"
133 | uuid = "150eb455-5306-5404-9cee-2592286d6298"
134 | version = "0.5.0"
135 |
136 | [[Crayons]]
137 | deps = ["Test"]
138 | git-tree-sha1 = "f621b8ef51fd2004c7cf157ea47f027fdeac5523"
139 | uuid = "a8cc5b0e-0ffa-5ad4-8c14-923d3ee1735f"
140 | version = "4.0.0"
141 |
142 | [[CustomUnitRanges]]
143 | deps = ["Test"]
144 | git-tree-sha1 = "0a106457a1831555857e18ac9617279c22fc393b"
145 | uuid = "dc8bdbbb-1ca9-579f-8c36-e416f6a65cce"
146 | version = "0.2.0"
147 |
148 | [[DataStructures]]
149 | deps = ["InteractiveUtils", "OrderedCollections", "Random", "Serialization", "Test"]
150 | git-tree-sha1 = "ca971f03e146cf144a9e2f2ce59674f5bf0e8038"
151 | uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
152 | version = "0.15.0"
153 |
154 | [[Dates]]
155 | deps = ["Printf"]
156 | uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"
157 |
158 | [[DelimitedFiles]]
159 | deps = ["Mmap"]
160 | uuid = "8bb1440f-4735-579b-a4ab-409b98df4dab"
161 |
162 | [[DiffEqBase]]
163 | deps = ["Compat", "DocStringExtensions", "IterativeSolvers", "IteratorInterfaceExtensions", "LinearAlgebra", "MuladdMacro", "Parameters", "RecipesBase", "RecursiveArrayTools", "RecursiveFactorization", "Requires", "Roots", "SparseArrays", "StaticArrays", "Statistics", "SuiteSparse", "TableTraits", "TreeViews"]
164 | git-tree-sha1 = "efc7b2774ca1e962e9b529bb89d2f2f4e72b7516"
165 | uuid = "2b5f629d-d688-5b77-993f-72d75c75574e"
166 | version = "5.8.1"
167 |
168 | [[DiffEqDiffTools]]
169 | deps = ["LinearAlgebra"]
170 | git-tree-sha1 = "14c4ec4f8796e71ba034f5089b6186876574a49d"
171 | uuid = "01453d9d-ee7c-5054-8395-0335cb756afa"
172 | version = "0.10.1"
173 |
174 | [[DiffEqOperators]]
175 | deps = ["DiffEqBase", "ForwardDiff", "LinearAlgebra", "SparseArrays", "StaticArrays", "SuiteSparse"]
176 | git-tree-sha1 = "2884a79a72aac38347b247615ac42eda41aa36e0"
177 | uuid = "9fdde737-9c7f-55bf-ade8-46b3f136cc48"
178 | version = "3.5.0"
179 |
180 | [[DiffResults]]
181 | deps = ["Compat", "StaticArrays"]
182 | git-tree-sha1 = "34a4a1e8be7bc99bc9c611b895b5baf37a80584c"
183 | uuid = "163ba53b-c6d8-5494-b064-1a9d43ac40c5"
184 | version = "0.0.4"
185 |
186 | [[DiffRules]]
187 | deps = ["Random", "Test"]
188 | git-tree-sha1 = "dc0869fb2f5b23466b32ea799bd82c76480167f7"
189 | uuid = "b552c78f-8df3-52c6-915a-8e097449b14b"
190 | version = "0.0.10"
191 |
192 | [[Distances]]
193 | deps = ["LinearAlgebra", "Printf", "Random", "Statistics", "Test"]
194 | git-tree-sha1 = "a135c7c062023051953141da8437ed74f89d767a"
195 | uuid = "b4f34e82-e78d-54a5-968a-f98e89d6e8f7"
196 | version = "0.8.0"
197 |
198 | [[Distributed]]
199 | deps = ["LinearAlgebra", "Random", "Serialization", "Sockets"]
200 | uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b"
201 |
202 | [[DocStringExtensions]]
203 | deps = ["LibGit2", "Markdown", "Pkg", "Test"]
204 | git-tree-sha1 = "4d30e889c9f106a51ffa4791a88ffd4765bf20c3"
205 | uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae"
206 | version = "0.7.0"
207 |
208 | [[ExponentialUtilities]]
209 | deps = ["LinearAlgebra", "Printf", "Random", "SparseArrays", "Test"]
210 | git-tree-sha1 = "6fad21cd7637d0ad6a7661f8abea1149922d6c9c"
211 | uuid = "d4d017d3-3776-5f7e-afef-a10c40355c18"
212 | version = "1.4.0"
213 |
214 | [[FFTViews]]
215 | deps = ["CustomUnitRanges", "FFTW", "Test"]
216 | git-tree-sha1 = "9d7993227ca7c0fdb6b31deef193adbba11c8f4e"
217 | uuid = "4f61f5a4-77b1-5117-aa51-3ab5ef4ef0cd"
218 | version = "0.2.0"
219 |
220 | [[FFTW]]
221 | deps = ["AbstractFFTs", "BinaryProvider", "Compat", "Conda", "Libdl", "LinearAlgebra", "Reexport", "Test"]
222 | git-tree-sha1 = "29cda58afbf62f35b1a094882ad6c745a47b2eaa"
223 | uuid = "7a1cc6ca-52ef-59f5-83cd-3a7055c09341"
224 | version = "0.2.4"
225 |
226 | [[FileIO]]
227 | deps = ["Pkg", "Random", "Test"]
228 | git-tree-sha1 = "da32159d4a2e526338506685e280e39ed2f18961"
229 | uuid = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549"
230 | version = "1.0.6"
231 |
232 | [[FileWatching]]
233 | uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee"
234 |
235 | [[FixedPointNumbers]]
236 | git-tree-sha1 = "d14a6fa5890ea3a7e5dcab6811114f132fec2b4b"
237 | uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93"
238 | version = "0.6.1"
239 |
240 | [[Flux]]
241 | deps = ["AbstractTrees", "Adapt", "CodecZlib", "Colors", "DelimitedFiles", "Juno", "LinearAlgebra", "MacroTools", "NNlib", "Pkg", "Printf", "Random", "Reexport", "Requires", "SHA", "Statistics", "StatsBase", "Tracker", "ZipFile"]
242 | git-tree-sha1 = "08212989c2856f95f90709ea5fd824bd27b34514"
243 | uuid = "587475ba-b771-5e3f-ad9e-33799f191a9c"
244 | version = "0.8.3"
245 |
246 | [[ForwardDiff]]
247 | deps = ["CommonSubexpressions", "DiffResults", "DiffRules", "InteractiveUtils", "LinearAlgebra", "NaNMath", "Random", "SparseArrays", "SpecialFunctions", "StaticArrays", "Test"]
248 | git-tree-sha1 = "4c4d727f1b7e0092134fabfab6396b8945c1ea5b"
249 | uuid = "f6369f11-7733-5829-9624-2563aa707210"
250 | version = "0.10.3"
251 |
252 | [[FunctionalCollections]]
253 | deps = ["Test"]
254 | git-tree-sha1 = "04cb9cfaa6ba5311973994fe3496ddec19b6292a"
255 | uuid = "de31a74c-ac4f-5751-b3fd-e18cd04993ca"
256 | version = "0.5.0"
257 |
258 | [[GenericSVD]]
259 | deps = ["LinearAlgebra", "Random", "Test"]
260 | git-tree-sha1 = "8aa93c3f3d81562a8962047eafcc5712af0a0f59"
261 | uuid = "01680d73-4ee2-5a08-a1aa-533608c188bb"
262 | version = "0.2.1"
263 |
264 | [[GeometryTypes]]
265 | deps = ["ColorTypes", "FixedPointNumbers", "IterTools", "LinearAlgebra", "StaticArrays"]
266 | git-tree-sha1 = "05796e9ac5a8fb8adfdcdf842fb8737d1cb5924e"
267 | uuid = "4d00f742-c7ba-57c2-abde-4428a4b178cb"
268 | version = "0.7.4"
269 |
270 | [[Graphics]]
271 | deps = ["Colors", "Compat", "NaNMath"]
272 | git-tree-sha1 = "e3ead4211073d4117a0d2ef7d1efc5c8092c8412"
273 | uuid = "a2bd30eb-e257-5431-a919-1863eab51364"
274 | version = "0.4.0"
275 |
276 | [[IRTools]]
277 | deps = ["InteractiveUtils", "MacroTools", "Test"]
278 | git-tree-sha1 = "42b54182e1764f908636157353825707c1aab764"
279 | uuid = "7869d1d1-7146-5819-86e3-90919afe41df"
280 | version = "0.2.1"
281 |
282 | [[IdentityRanges]]
283 | deps = ["OffsetArrays", "Test"]
284 | git-tree-sha1 = "b8c36c6083fd14e2a82c5974225702126e894f23"
285 | uuid = "bbac6d45-d8f3-5730-bfe4-7a449cd117ca"
286 | version = "0.3.0"
287 |
288 | [[ImageAxes]]
289 | deps = ["AxisArrays", "Colors", "FixedPointNumbers", "ImageCore", "MappedArrays", "Reexport", "SimpleTraits"]
290 | git-tree-sha1 = "8109bb9a28deca29a5b938ebfa7d0a75319d8776"
291 | uuid = "2803e5a7-5153-5ecf-9a86-9b4c37f5f5ac"
292 | version = "0.6.0"
293 |
294 | [[ImageCore]]
295 | deps = ["ColorTypes", "Colors", "FFTW", "FixedPointNumbers", "Graphics", "MappedArrays", "OffsetArrays", "PaddedViews", "Reexport"]
296 | git-tree-sha1 = "653a098434f6d76fd42796e48c42f2c2c4bd451b"
297 | uuid = "a09fc81d-aa75-5fe9-8630-4744c3626534"
298 | version = "0.8.3"
299 |
300 | [[ImageDistances]]
301 | deps = ["ColorVectorSpace", "Colors", "Distances", "FixedPointNumbers", "ImageCore"]
302 | git-tree-sha1 = "aa69ce81260bcb5e950a5e3b48ccca15447c6d8c"
303 | uuid = "51556ac3-7006-55f5-8cb3-34580c88182d"
304 | version = "0.2.4"
305 |
306 | [[ImageFiltering]]
307 | deps = ["CatIndices", "ColorVectorSpace", "Colors", "ComputationalResources", "DataStructures", "FFTViews", "FFTW", "FixedPointNumbers", "ImageCore", "LinearAlgebra", "MappedArrays", "OffsetArrays", "Requires", "StaticArrays", "Statistics", "TiledIteration"]
308 | git-tree-sha1 = "ea26aea659b889da49f51afe2886358990ef9732"
309 | uuid = "6a3955dd-da59-5b1f-98d4-e7296123deb5"
310 | version = "0.6.3"
311 |
312 | [[ImageMetadata]]
313 | deps = ["AxisArrays", "ColorVectorSpace", "Colors", "FixedPointNumbers", "ImageAxes", "ImageCore", "IndirectArrays"]
314 | git-tree-sha1 = "631c3e1dc2b645ed3bdee14418f8bbfc38b9b7f0"
315 | uuid = "bc367c6b-8a6b-528e-b4bd-a4b897500b49"
316 | version = "0.7.0"
317 |
318 | [[ImageMorphology]]
319 | deps = ["Colors", "FixedPointNumbers", "ImageCore"]
320 | git-tree-sha1 = "2ea6e55a36166321ca735b8eaf74936180e2ad5d"
321 | uuid = "787d08f9-d448-5407-9aad-5290dd7ab264"
322 | version = "0.2.4"
323 |
324 | [[ImageShow]]
325 | deps = ["Base64", "ColorTypes", "Colors", "FileIO", "FixedPointNumbers", "ImageCore", "OffsetArrays", "Requires"]
326 | git-tree-sha1 = "c23323afc82b6b553e6b2244d531e50858ea392c"
327 | uuid = "4e3cecfd-b093-5904-9786-8bbb286a6a31"
328 | version = "0.2.0"
329 |
330 | [[ImageTransformations]]
331 | deps = ["AxisAlgorithms", "ColorTypes", "ColorVectorSpace", "Colors", "CoordinateTransformations", "FixedPointNumbers", "IdentityRanges", "ImageCore", "Interpolations", "OffsetArrays", "StaticArrays"]
332 | git-tree-sha1 = "4cf03fc72d8877d5e58c1c72d176340ad4e28fda"
333 | uuid = "02fcd773-0e25-5acc-982a-7f6622650795"
334 | version = "0.8.0"
335 |
336 | [[Images]]
337 | deps = ["AxisArrays", "Base64", "ColorTypes", "ColorVectorSpace", "Colors", "FileIO", "FixedPointNumbers", "Graphics", "ImageAxes", "ImageCore", "ImageDistances", "ImageFiltering", "ImageMetadata", "ImageMorphology", "ImageShow", "ImageTransformations", "IndirectArrays", "MappedArrays", "OffsetArrays", "Reexport", "SparseArrays", "StaticArrays", "Statistics", "StatsBase", "TiledIteration"]
338 | git-tree-sha1 = "1aae59fc0e34e47dfb111113bb6e9a65d7bafe81"
339 | uuid = "916415d5-f1e6-5110-898d-aaa5f9f070e0"
340 | version = "0.18.0"
341 |
342 | [[IndirectArrays]]
343 | deps = ["Compat", "Test"]
344 | git-tree-sha1 = "b6e249be10a3381b2c72ac82f2d13d70067cb2bd"
345 | uuid = "9b13fd28-a010-5f03-acff-a1bbcff69959"
346 | version = "0.5.0"
347 |
348 | [[Interact]]
349 | deps = ["CSSUtil", "InteractBase", "JSON", "Knockout", "Observables", "OrderedCollections", "Reexport", "Test", "WebIO", "Widgets"]
350 | git-tree-sha1 = "f6531bc554990a6659966afcb8f05543c032068b"
351 | uuid = "c601a237-2ae4-5e1e-952c-7a85b0c7eef1"
352 | version = "0.10.2"
353 |
354 | [[InteractBase]]
355 | deps = ["Base64", "CSSUtil", "Colors", "Dates", "JSExpr", "JSON", "Knockout", "Observables", "OrderedCollections", "Random", "Test", "WebIO", "Widgets"]
356 | git-tree-sha1 = "83fc45f21bfec97e5cd56e790f6945497ab0f095"
357 | uuid = "d3863d7c-f0c8-5437-a7b4-3ae773c01009"
358 | version = "0.10.1"
359 |
360 | [[InteractiveUtils]]
361 | deps = ["LinearAlgebra", "Markdown"]
362 | uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240"
363 |
364 | [[Interpolations]]
365 | deps = ["AxisAlgorithms", "LinearAlgebra", "OffsetArrays", "Random", "Ratios", "SharedArrays", "SparseArrays", "StaticArrays", "WoodburyMatrices"]
366 | git-tree-sha1 = "e1bac96b5ef3ea23b50e801b4a988ec21861a47f"
367 | uuid = "a98d9a8b-a2ab-59e6-89dd-64a1c18fca59"
368 | version = "0.12.2"
369 |
370 | [[IntervalSets]]
371 | deps = ["Compat"]
372 | git-tree-sha1 = "9dc556002f23740de13946e8c2e41798e09a9249"
373 | uuid = "8197267c-284f-5f27-9208-e0e47529a953"
374 | version = "0.3.1"
375 |
376 | [[IterTools]]
377 | deps = ["SparseArrays", "Test"]
378 | git-tree-sha1 = "79246285c43602384e6f1943b3554042a3712056"
379 | uuid = "c8e1da08-722c-5040-9ed9-7db0dc04731e"
380 | version = "1.1.1"
381 |
382 | [[IterativeSolvers]]
383 | deps = ["LinearAlgebra", "Printf", "Random", "RecipesBase", "SparseArrays", "Test"]
384 | git-tree-sha1 = "5687f68018b4f14c0da54d402bb23eecaec17f37"
385 | uuid = "42fd0dbc-a981-5370-80f2-aaf504508153"
386 | version = "0.8.1"
387 |
388 | [[IteratorInterfaceExtensions]]
389 | git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856"
390 | uuid = "82899510-4779-5014-852e-03e436cf321d"
391 | version = "1.0.0"
392 |
393 | [[JSExpr]]
394 | deps = ["JSON", "MacroTools", "Observables", "Test", "WebIO"]
395 | git-tree-sha1 = "013bc2143a2e84ea489365cf30db3407deb540c2"
396 | uuid = "97c1335a-c9c5-57fe-bc5d-ec35cebe8660"
397 | version = "0.5.0"
398 |
399 | [[JSON]]
400 | deps = ["Dates", "Distributed", "Mmap", "Sockets", "Test", "Unicode"]
401 | git-tree-sha1 = "1f7a25b53ec67f5e9422f1f551ee216503f4a0fa"
402 | uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
403 | version = "0.20.0"
404 |
405 | [[Juno]]
406 | deps = ["Base64", "Logging", "Media", "Profile", "Test"]
407 | git-tree-sha1 = "4e4a8d43aa7ecec66cadaf311fbd1e5c9d7b9175"
408 | uuid = "e5e0dc1b-0480-54bc-9374-aad01c23163d"
409 | version = "0.7.0"
410 |
411 | [[Knockout]]
412 | deps = ["JSExpr", "JSON", "Observables", "Pkg", "Test", "WebIO"]
413 | git-tree-sha1 = "5cca7f070f85392cd0f42220c8ea5f7be96dacf4"
414 | uuid = "bcebb21b-c2e3-54f8-a781-646b90f6d2cc"
415 | version = "0.2.2"
416 |
417 | [[LibGit2]]
418 | uuid = "76f85450-5226-5b5a-8eaa-529ad045b433"
419 |
420 | [[Libdl]]
421 | uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb"
422 |
423 | [[LineSearches]]
424 | deps = ["LinearAlgebra", "NLSolversBase", "NaNMath", "Parameters", "Printf", "Test"]
425 | git-tree-sha1 = "54eb90e8dbe745d617c78dee1d6ae95c7f6f5779"
426 | uuid = "d3d80556-e9d4-5f37-9878-2ab0fcc64255"
427 | version = "7.0.1"
428 |
429 | [[LinearAlgebra]]
430 | deps = ["Libdl"]
431 | uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
432 |
433 | [[Logging]]
434 | uuid = "56ddb016-857b-54e1-b83d-db4d58db5568"
435 |
436 | [[MacroTools]]
437 | deps = ["CSTParser", "Compat", "DataStructures", "Test"]
438 | git-tree-sha1 = "daecd9e452f38297c686eba90dba2a6d5da52162"
439 | uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09"
440 | version = "0.5.0"
441 |
442 | [[MappedArrays]]
443 | deps = ["Test"]
444 | git-tree-sha1 = "923441c5ac942b60bd3a842d5377d96646bcbf46"
445 | uuid = "dbb5928d-eab1-5f90-85c2-b9b0edb7c900"
446 | version = "0.2.1"
447 |
448 | [[Markdown]]
449 | deps = ["Base64"]
450 | uuid = "d6f4376e-aef5-505a-96c1-9c027394607a"
451 |
452 | [[Measures]]
453 | deps = ["Test"]
454 | git-tree-sha1 = "ddfd6d13e330beacdde2c80de27c1c671945e7d9"
455 | uuid = "442fdcdd-2543-5da2-b0f3-8c86c306513e"
456 | version = "0.3.0"
457 |
458 | [[Media]]
459 | deps = ["MacroTools", "Test"]
460 | git-tree-sha1 = "75a54abd10709c01f1b86b84ec225d26e840ed58"
461 | uuid = "e89f7d12-3494-54d1-8411-f7d8b9ae1f27"
462 | version = "0.5.0"
463 |
464 | [[MeshIO]]
465 | deps = ["ColorTypes", "FileIO", "GeometryTypes", "Printf", "Test"]
466 | git-tree-sha1 = "3e01e12c4c626578a9d47fac0d15c9fe8dad5139"
467 | uuid = "7269a6da-0436-5bbc-96c2-40638cbb6118"
468 | version = "0.3.1"
469 |
470 | [[Missings]]
471 | deps = ["SparseArrays", "Test"]
472 | git-tree-sha1 = "f0719736664b4358aa9ec173077d4285775f8007"
473 | uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28"
474 | version = "0.4.1"
475 |
476 | [[Mmap]]
477 | uuid = "a63ad114-7e13-5084-954f-fe012c677804"
478 |
479 | [[MuladdMacro]]
480 | deps = ["MacroTools", "Test"]
481 | git-tree-sha1 = "41e6e7c4b448afeaddaac7f496b414854f83b848"
482 | uuid = "46d2c3a1-f734-5fdb-9937-b9b9aeba4221"
483 | version = "0.2.1"
484 |
485 | [[NLSolversBase]]
486 | deps = ["Calculus", "DiffEqDiffTools", "DiffResults", "Distributed", "ForwardDiff", "LinearAlgebra", "Random", "SparseArrays", "Test"]
487 | git-tree-sha1 = "0c6f0e7f2178f78239cfb75310359eed10f2cacb"
488 | uuid = "d41bc354-129a-5804-8e4c-c37616107c6c"
489 | version = "7.3.1"
490 |
491 | [[NLsolve]]
492 | deps = ["DiffEqDiffTools", "Distances", "ForwardDiff", "LineSearches", "LinearAlgebra", "NLSolversBase", "Printf", "Random", "Reexport", "SparseArrays", "Test"]
493 | git-tree-sha1 = "413e54f04a4cbe9804089794eec6b06b2a43bc47"
494 | uuid = "2774e3e8-f4cf-5e23-947b-6d7e65073b56"
495 | version = "4.0.0"
496 |
497 | [[NNlib]]
498 | deps = ["Libdl", "LinearAlgebra", "Requires", "Statistics", "TimerOutputs"]
499 | git-tree-sha1 = "0c667371391fc6bb31f7f12f96a56a17098b3de8"
500 | uuid = "872c559c-99b0-510c-b3b7-b6c96a88d5cd"
501 | version = "0.6.0"
502 |
503 | [[NaNMath]]
504 | deps = ["Compat"]
505 | git-tree-sha1 = "ce3b85e484a5d4c71dd5316215069311135fa9f2"
506 | uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3"
507 | version = "0.3.2"
508 |
509 | [[Observables]]
510 | deps = ["Test"]
511 | git-tree-sha1 = "dc02cec22747d1d10d9f70d8a1c03432b5bfbcd0"
512 | uuid = "510215fc-4207-5dde-b226-833fc4488ee2"
513 | version = "0.2.3"
514 |
515 | [[OffsetArrays]]
516 | deps = ["DelimitedFiles"]
517 | git-tree-sha1 = "49a6d9b5b3dedec18035e4d97ce77e2b2a182c05"
518 | uuid = "6fe1bfb0-de20-5000-8ca7-80f57d26f881"
519 | version = "0.11.0"
520 |
521 | [[OrderedCollections]]
522 | deps = ["Random", "Serialization", "Test"]
523 | git-tree-sha1 = "c4c13474d23c60d20a67b217f1d7f22a40edf8f1"
524 | uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
525 | version = "1.1.0"
526 |
527 | [[OrdinaryDiffEq]]
528 | deps = ["DataStructures", "DiffEqBase", "DiffEqDiffTools", "DiffEqOperators", "ExponentialUtilities", "ForwardDiff", "GenericSVD", "LinearAlgebra", "Logging", "MuladdMacro", "NLsolve", "Parameters", "RecursiveArrayTools", "Reexport", "StaticArrays"]
529 | git-tree-sha1 = "300118c9211d3a11dad71e66bc9eb4a158c39c16"
530 | uuid = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"
531 | version = "5.7.0"
532 |
533 | [[PaddedViews]]
534 | deps = ["OffsetArrays", "Test"]
535 | git-tree-sha1 = "7da3e7e1a58cffbf10177553ae95f17b92516912"
536 | uuid = "5432bcbf-9aad-5242-b902-cca2824c8663"
537 | version = "0.4.2"
538 |
539 | [[Parameters]]
540 | deps = ["Markdown", "OrderedCollections", "REPL", "Test"]
541 | git-tree-sha1 = "70bdbfb2bceabb15345c0b54be4544813b3444e4"
542 | uuid = "d96e819e-fc66-5662-9728-84c9c7592b0a"
543 | version = "0.10.3"
544 |
545 | [[Pidfile]]
546 | deps = ["FileWatching", "Test"]
547 | git-tree-sha1 = "1ffd82728498b5071cde851bbb7abd780d4445f3"
548 | uuid = "fa939f87-e72e-5be4-a000-7fc836dbe307"
549 | version = "1.1.0"
550 |
551 | [[Pkg]]
552 | deps = ["Dates", "LibGit2", "Markdown", "Printf", "REPL", "Random", "SHA", "UUIDs"]
553 | uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
554 |
555 | [[Printf]]
556 | deps = ["Unicode"]
557 | uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"
558 |
559 | [[Profile]]
560 | deps = ["Printf"]
561 | uuid = "9abbd945-dff8-562f-b5e8-e1ebf5ef1b79"
562 |
563 | [[QuartzImageIO]]
564 | deps = ["ColorTypes", "ColorVectorSpace", "FileIO", "FixedPointNumbers", "ImageCore", "Libdl"]
565 | git-tree-sha1 = "813432448a8f06a1316943c3456dcf043afacd96"
566 | uuid = "dca85d43-d64c-5e67-8c65-017450d5d020"
567 | version = "0.6.0"
568 |
569 | [[REPL]]
570 | deps = ["InteractiveUtils", "Markdown", "Sockets"]
571 | uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb"
572 |
573 | [[Random]]
574 | deps = ["Serialization"]
575 | uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
576 |
577 | [[RangeArrays]]
578 | deps = ["Compat"]
579 | git-tree-sha1 = "d925adfd5b01cb46fde89dc9548d167b3b136f4a"
580 | uuid = "b3c3ace0-ae52-54e7-9d0b-2c1406fd6b9d"
581 | version = "0.3.1"
582 |
583 | [[Ratios]]
584 | deps = ["Compat"]
585 | git-tree-sha1 = "cdbbe0f350581296f3a2e3e7a91b214121934407"
586 | uuid = "c84ed2f1-dad5-54f0-aa8e-dbefe2724439"
587 | version = "0.3.1"
588 |
589 | [[RayTracer]]
590 | deps = ["FileIO", "Flux", "IRTools", "Images", "MeshIO", "Zygote"]
591 | git-tree-sha1 = "b08a0b54ef8c92acd65fdbd983620c1fb6b26ae9"
592 | repo-rev = "master"
593 | repo-url = "https://github.com/avik-pal/RayTracer.jl"
594 | uuid = "60dacb86-48ff-11e9-0f01-03ab8794bbc9"
595 | version = "0.1.0"
596 |
597 | [[RecipesBase]]
598 | deps = ["Random", "Test"]
599 | git-tree-sha1 = "0b3cb370ee4dc00f47f1193101600949f3dcf884"
600 | uuid = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
601 | version = "0.6.0"
602 |
603 | [[RecursiveArrayTools]]
604 | deps = ["ArrayInterface", "RecipesBase", "Requires", "StaticArrays", "Statistics", "Test"]
605 | git-tree-sha1 = "187ea7dd541955102c7035a6668613bdf52022ca"
606 | uuid = "731186ca-8d62-57ce-b412-fbd966d074cd"
607 | version = "0.20.0"
608 |
609 | [[RecursiveFactorization]]
610 | deps = ["LinearAlgebra", "Random", "Test"]
611 | git-tree-sha1 = "54410ebd72cbb84d7b7678eb3da643f8e71181fc"
612 | uuid = "f2c3362d-daeb-58d1-803e-2bc74f2840b4"
613 | version = "0.0.1"
614 |
615 | [[Reexport]]
616 | deps = ["Pkg"]
617 | git-tree-sha1 = "7b1d07f411bc8ddb7977ec7f377b97b158514fe0"
618 | uuid = "189a3867-3050-52da-a836-e630ba90ab69"
619 | version = "0.2.0"
620 |
621 | [[Requires]]
622 | deps = ["Test"]
623 | git-tree-sha1 = "f6fbf4ba64d295e146e49e021207993b6b48c7d1"
624 | uuid = "ae029012-a4dd-5104-9daa-d747884805df"
625 | version = "0.5.2"
626 |
627 | [[Roots]]
628 | deps = ["Printf", "Statistics", "Test"]
629 | git-tree-sha1 = "7228278e31d6d0e22a1ae0b41ea9a0df2859f33d"
630 | uuid = "f2b01f46-fcfa-551c-844a-d8ac1e96c665"
631 | version = "0.8.1"
632 |
633 | [[Rotations]]
634 | deps = ["LinearAlgebra", "Random", "StaticArrays", "Statistics", "Test"]
635 | git-tree-sha1 = "dfb3ceb177a59f25fee4e2f26c1aeb92b73d3a0e"
636 | uuid = "6038ab10-8711-5258-84ad-4b1120ba62dc"
637 | version = "0.11.1"
638 |
639 | [[SHA]]
640 | uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
641 |
642 | [[Serialization]]
643 | uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
644 |
645 | [[SharedArrays]]
646 | deps = ["Distributed", "Mmap", "Random", "Serialization"]
647 | uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383"
648 |
649 | [[SimpleTraits]]
650 | deps = ["InteractiveUtils", "MacroTools", "Test"]
651 | git-tree-sha1 = "c0a542b8d5e369b179ccd296b2ca987f6da5da0a"
652 | uuid = "699a6c99-e7fa-54fc-8d76-47d257e15c1d"
653 | version = "0.8.0"
654 |
655 | [[Sockets]]
656 | uuid = "6462fe0b-24de-5631-8697-dd941f90decc"
657 |
658 | [[SortingAlgorithms]]
659 | deps = ["DataStructures", "Random", "Test"]
660 | git-tree-sha1 = "03f5898c9959f8115e30bc7226ada7d0df554ddd"
661 | uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c"
662 | version = "0.3.1"
663 |
664 | [[SparseArrays]]
665 | deps = ["LinearAlgebra", "Random"]
666 | uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf"
667 |
668 | [[SpecialFunctions]]
669 | deps = ["BinDeps", "BinaryProvider", "Libdl", "Test"]
670 | git-tree-sha1 = "0b45dc2e45ed77f445617b99ff2adf0f5b0f23ea"
671 | uuid = "276daf66-3868-5448-9aa4-cd146d93841b"
672 | version = "0.7.2"
673 |
674 | [[StaticArrays]]
675 | deps = ["InteractiveUtils", "LinearAlgebra", "Random", "Statistics", "Test"]
676 | git-tree-sha1 = "3841b39ed5f047db1162627bf5f80a9cd3e39ae2"
677 | uuid = "90137ffa-7385-5640-81b9-e52037218182"
678 | version = "0.10.3"
679 |
680 | [[Statistics]]
681 | deps = ["LinearAlgebra", "SparseArrays"]
682 | uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
683 |
684 | [[StatsBase]]
685 | deps = ["DataStructures", "LinearAlgebra", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics"]
686 | git-tree-sha1 = "8a0f4b09c7426478ab677245ab2b0b68552143c7"
687 | uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
688 | version = "0.30.0"
689 |
690 | [[SuiteSparse]]
691 | deps = ["Libdl", "LinearAlgebra", "SparseArrays"]
692 | uuid = "4607b0f0-06f3-5cda-b6b1-a6196a1729e9"
693 |
694 | [[TableTraits]]
695 | deps = ["IteratorInterfaceExtensions"]
696 | git-tree-sha1 = "b1ad568ba658d8cbb3b892ed5380a6f3e781a81e"
697 | uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c"
698 | version = "1.0.0"
699 |
700 | [[Test]]
701 | deps = ["Distributed", "InteractiveUtils", "Logging", "Random"]
702 | uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
703 |
704 | [[TiledIteration]]
705 | deps = ["OffsetArrays", "Test"]
706 | git-tree-sha1 = "58f6f07d3b54a363ec283a8f5fc9fb4ecebde656"
707 | uuid = "06e1c1a7-607b-532d-9fad-de7d9aa2abac"
708 | version = "0.2.3"
709 |
710 | [[TimerOutputs]]
711 | deps = ["Crayons", "Printf", "Test", "Unicode"]
712 | git-tree-sha1 = "b80671c06f8f8bae08c55d67b5ce292c5ae2660c"
713 | uuid = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f"
714 | version = "0.5.0"
715 |
716 | [[Tokenize]]
717 | deps = ["Printf", "Test"]
718 | git-tree-sha1 = "3e83f60b74911d3042d3550884ca2776386a02b8"
719 | uuid = "0796e94c-ce3b-5d07-9a54-7f471281c624"
720 | version = "0.5.3"
721 |
722 | [[Tracker]]
723 | deps = ["Adapt", "DiffRules", "ForwardDiff", "LinearAlgebra", "MacroTools", "NNlib", "NaNMath", "Printf", "Random", "Requires", "SpecialFunctions", "Statistics", "Test"]
724 | git-tree-sha1 = "327342fec6e09f68ced0c2dc5731ed475e4b696b"
725 | uuid = "9f7883ad-71c0-57eb-9f7f-b5c9e6d3789c"
726 | version = "0.2.2"
727 |
728 | [[TranscodingStreams]]
729 | deps = ["Random", "Test"]
730 | git-tree-sha1 = "a25d8e5a28c3b1b06d3859f30757d43106791919"
731 | uuid = "3bb67fe8-82b1-5028-8e26-92a6c54297fa"
732 | version = "0.9.4"
733 |
734 | [[Trebuchet]]
735 | deps = ["DiffEqBase", "JSExpr", "OrdinaryDiffEq", "WebIO"]
736 | git-tree-sha1 = "c1daf264ca0c050381be6258b5e311615ea5f250"
737 | uuid = "98b73d46-197d-11e9-11eb-69a6ff759d3a"
738 | version = "0.1.0"
739 |
740 | [[TreeViews]]
741 | deps = ["Test"]
742 | git-tree-sha1 = "8d0d7a3fe2f30d6a7f833a5f19f7c7a5b396eae6"
743 | uuid = "a2a6695c-b41b-5b7d-aed9-dbfdeacea5d7"
744 | version = "0.3.0"
745 |
746 | [[URIParser]]
747 | deps = ["Test", "Unicode"]
748 | git-tree-sha1 = "6ddf8244220dfda2f17539fa8c9de20d6c575b69"
749 | uuid = "30578b45-9adc-5946-b283-645ec420af67"
750 | version = "0.4.0"
751 |
752 | [[UUIDs]]
753 | deps = ["Random"]
754 | uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
755 |
756 | [[Unicode]]
757 | uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"
758 |
759 | [[VersionParsing]]
760 | deps = ["Compat"]
761 | git-tree-sha1 = "c9d5aa108588b978bd859554660c8a5c4f2f7669"
762 | uuid = "81def892-9a0e-5fdd-b105-ffc91e053289"
763 | version = "1.1.3"
764 |
765 | [[WebIO]]
766 | deps = ["AssetRegistry", "Base64", "Compat", "Distributed", "FunctionalCollections", "JSON", "Logging", "Observables", "Random", "Requires", "Sockets", "Test", "UUIDs", "Widgets"]
767 | git-tree-sha1 = "125dc746e5b36424c6a7e694889a7f3d434a160a"
768 | uuid = "0f1e0344-ec1d-5b48-a673-e5cf874b6c29"
769 | version = "0.8.1"
770 |
771 | [[Widgets]]
772 | deps = ["Colors", "Dates", "Observables", "OrderedCollections", "Test"]
773 | git-tree-sha1 = "c53befc70c6b91eaa2a9888c2f6ac2d92720a81b"
774 | uuid = "cc8bc4a8-27d6-5769-a93b-9d913e69aa62"
775 | version = "0.6.1"
776 |
777 | [[WoodburyMatrices]]
778 | deps = ["LinearAlgebra", "Random", "SparseArrays", "Test"]
779 | git-tree-sha1 = "21772c33b447757ec7d3e61fcdfb9ea5c47eedcf"
780 | uuid = "efce3f68-66dc-5838-9240-27a6d6f5f9b6"
781 | version = "0.4.1"
782 |
783 | [[ZipFile]]
784 | deps = ["BinaryProvider", "Libdl", "Printf"]
785 | git-tree-sha1 = "580ce62b6c14244916cc28ad54f8a2e2886f843d"
786 | uuid = "a5390f91-8eb1-5f08-bee0-b1d1ffed6cea"
787 | version = "0.8.3"
788 |
789 | [[Zygote]]
790 | deps = ["DiffRules", "ForwardDiff", "IRTools", "InteractiveUtils", "LinearAlgebra", "MacroTools", "NNlib", "NaNMath", "Random", "Requires", "SpecialFunctions", "Statistics"]
791 | git-tree-sha1 = "40aa7d4724b3ab508f6200a885b6e92cc6647f14"
792 | uuid = "e88e6eb3-aa80-5325-afca-941959d7151f"
793 | version = "0.3.1"
794 |
--------------------------------------------------------------------------------
/msr_2019_06_03/Project.toml:
--------------------------------------------------------------------------------
1 | [deps]
2 | Flux = "587475ba-b771-5e3f-ad9e-33799f191a9c"
3 | Images = "916415d5-f1e6-5110-898d-aaa5f9f070e0"
4 | Interact = "c601a237-2ae4-5e1e-952c-7a85b0c7eef1"
5 | QuartzImageIO = "dca85d43-d64c-5e67-8c65-017450d5d020"
6 | RayTracer = "60dacb86-48ff-11e9-0f01-03ab8794bbc9"
7 | Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
8 | Trebuchet = "98b73d46-197d-11e9-11eb-69a6ff759d3a"
9 | Zygote = "e88e6eb3-aa80-5325-afca-941959d7151f"
10 |
--------------------------------------------------------------------------------
/msr_2019_06_03/Zygote IR Example.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 1,
6 | "metadata": {},
7 | "outputs": [
8 | {
9 | "name": "stderr",
10 | "output_type": "stream",
11 | "text": [
12 | "┌ Info: activating environment at `~/src/msr_talk/Project.toml`.\n",
13 | "└ @ Pkg.API /Users/sabae/tmp/julia-build/julia-release-1.2/usr/share/julia/stdlib/v1.2/Pkg/src/API.jl:564\n"
14 | ]
15 | },
16 | {
17 | "name": "stdout",
18 | "output_type": "stream",
19 | "text": [
20 | "\u001b[32m\u001b[1m Updating\u001b[22m\u001b[39m registry at `~/.julia/registries/General`\n",
21 | "\u001b[32m\u001b[1m Updating\u001b[22m\u001b[39m git-repo `https://github.com/JuliaRegistries/General.git`\n",
22 | "\u001b[?25l\u001b[2K\u001b[?25h"
23 | ]
24 | }
25 | ],
26 | "source": [
27 | "# Initialize environment in current directory, to load\n",
28 | "import Pkg; Pkg.activate(@__DIR__); Pkg.instantiate()\n",
29 | "using Zygote"
30 | ]
31 | },
32 | {
33 | "cell_type": "code",
34 | "execution_count": 31,
35 | "metadata": {},
36 | "outputs": [
37 | {
38 | "data": {
39 | "text/plain": [
40 | "iterative_pow (generic function with 1 method)"
41 | ]
42 | },
43 | "execution_count": 31,
44 | "metadata": {},
45 | "output_type": "execute_result"
46 | }
47 | ],
48 | "source": [
49 | "function iterative_pow(x, p::Int)\n",
50 | " while p > 1\n",
51 | " x = x*x\n",
52 | " p -= 1\n",
53 | " end\n",
54 | " return x\n",
55 | "end"
56 | ]
57 | },
58 | {
59 | "cell_type": "code",
60 | "execution_count": 32,
61 | "metadata": {},
62 | "outputs": [
63 | {
64 | "name": "stdout",
65 | "output_type": "stream",
66 | "text": [
67 | "y = 65536.0\n",
68 | "J(1.0) = (524288.0, nothing)\n"
69 | ]
70 | },
71 | {
72 | "data": {
73 | "text/plain": [
74 | "(524288.0, nothing)"
75 | ]
76 | },
77 | "execution_count": 32,
78 | "metadata": {},
79 | "output_type": "execute_result"
80 | }
81 | ],
82 | "source": [
83 | "y, J = Zygote.forward(iterative_pow, 2.0, 5)\n",
84 | "\n",
85 | "@show y\n",
86 | "@show J(1.0)"
87 | ]
88 | },
89 | {
90 | "cell_type": "code",
91 | "execution_count": 35,
92 | "metadata": {
93 | "scrolled": false
94 | },
95 | "outputs": [
96 | {
97 | "name": "stdout",
98 | "output_type": "stream",
99 | "text": [
100 | "\t.section\t__TEXT,__text,regular,pure_instructions\n",
101 | "; ┌ @ interface.jl:38 within `#34'\n",
102 | "\tpushq\t%rbp\n",
103 | "\tmovq\t%rsp, %rbp\n",
104 | "\tpushq\t%r15\n",
105 | "\tpushq\t%r14\n",
106 | "\tpushq\t%r13\n",
107 | "\tpushq\t%r12\n",
108 | "\tpushq\t%rbx\n",
109 | "\tandq\t$-32, %rsp\n",
110 | "\tsubq\t$96, %rsp\n",
111 | "\tvmovsd\t%xmm0, 64(%rsp)\n",
112 | "\tmovq\t%rdi, %rbx\n",
113 | "\tvxorps\t%xmm0, %xmm0, %xmm0\n",
114 | "\tvmovaps\t%ymm0, 32(%rsp)\n",
115 | "\tmovabsq\t$jl_get_ptls_states_fast, %rax\n",
116 | "\tvzeroupper\n",
117 | "\tcallq\t*%rax\n",
118 | "\tmovq\t%rax, %r14\n",
119 | "\tmovq\t$4, 32(%rsp)\n",
120 | "\tmovq\t(%r14), %rax\n",
121 | "\tmovq\t%rax, 40(%rsp)\n",
122 | "\tleaq\t32(%rsp), %rax\n",
123 | "\tmovq\t%rax, (%r14)\n",
124 | "\tmovq\t(%rbx), %rdi\n",
125 | "\tmovabsq\t$Pullback, %rax\n",
126 | "\tvmovsd\t64(%rsp), %xmm0 ## xmm0 = mem[0],zero\n",
127 | "\tcallq\t*%rax\n",
128 | "\tmovq\t%rax, %rbx\n",
129 | "\tmovq\t%rbx, 48(%rsp)\n",
130 | "; │┌ @ interface.jl:34 within `tailmemaybe'\n",
131 | "; ││┌ @ essentials.jl:201 within `tail'\n",
132 | "\tmovq\t%rbx, 16(%rsp)\n",
133 | "\tmovabsq\t$4548353680, %rax ## imm = 0x10F1A5E90\n",
134 | "\tmovq\t%rax, 24(%rsp)\n",
135 | "\tmovabsq\t$jl_f_getfield, %r13\n",
136 | "\tleaq\t16(%rsp), %r15\n",
137 | "\txorl\t%edi, %edi\n",
138 | "\tmovl\t$2, %edx\n",
139 | "\tmovq\t%r15, %rsi\n",
140 | "\tcallq\t*%r13\n",
141 | "\tmovq\t%rax, %r12\n",
142 | "\tmovq\t%r12, 56(%rsp)\n",
143 | "\tmovq\t%rbx, 16(%rsp)\n",
144 | "\tmovabsq\t$4548353744, %rax ## imm = 0x10F1A5ED0\n",
145 | "\tmovq\t%rax, 24(%rsp)\n",
146 | "\txorl\t%edi, %edi\n",
147 | "\tmovl\t$2, %edx\n",
148 | "\tmovq\t%r15, %rsi\n",
149 | "\tcallq\t*%r13\n",
150 | "\tmovq\t%rax, 48(%rsp)\n",
151 | "\tmovq\t%r12, 16(%rsp)\n",
152 | "\tmovq\t%rax, 24(%rsp)\n",
153 | "\tmovabsq\t$jl_f_tuple, %rax\n",
154 | "\txorl\t%edi, %edi\n",
155 | "\tmovl\t$2, %edx\n",
156 | "\tmovq\t%r15, %rsi\n",
157 | "\tcallq\t*%rax\n",
158 | "; │└└\n",
159 | "\tmovq\t40(%rsp), %rcx\n",
160 | "\tmovq\t%rcx, (%r14)\n",
161 | "\tleaq\t-40(%rbp), %rsp\n",
162 | "\tpopq\t%rbx\n",
163 | "\tpopq\t%r12\n",
164 | "\tpopq\t%r13\n",
165 | "\tpopq\t%r14\n",
166 | "\tpopq\t%r15\n",
167 | "\tpopq\t%rbp\n",
168 | "\tretq\n",
169 | "\tnopw\t%cs:(%rax,%rax)\n",
170 | "; └\n"
171 | ]
172 | }
173 | ],
174 | "source": [
175 | "@code_native J(1.0)"
176 | ]
177 | },
178 | {
179 | "cell_type": "code",
180 | "execution_count": null,
181 | "metadata": {},
182 | "outputs": [],
183 | "source": []
184 | }
185 | ],
186 | "metadata": {
187 | "kernelspec": {
188 | "display_name": "Julia 1.2.0-rc1",
189 | "language": "julia",
190 | "name": "julia-1.2"
191 | },
192 | "language_info": {
193 | "file_extension": ".jl",
194 | "mimetype": "application/julia",
195 | "name": "julia",
196 | "version": "1.2.0"
197 | }
198 | },
199 | "nbformat": 4,
200 | "nbformat_minor": 2
201 | }
202 |
--------------------------------------------------------------------------------
/msr_2019_06_03/Zygote Linear Regression.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# An Introduction to Zygote: Linear Regression\n",
8 | "\n",
9 | "In this notebook, we will define Linear Regression in Zygote from scratch, showing how easy it is to take derivatives of custom code."
10 | ]
11 | },
12 | {
13 | "cell_type": "code",
14 | "execution_count": 1,
15 | "metadata": {},
16 | "outputs": [
17 | {
18 | "name": "stderr",
19 | "output_type": "stream",
20 | "text": [
21 | "┌ Info: activating environment at `~/src/msr_talk/Project.toml`.\n",
22 | "└ @ Pkg.API /Users/sabae/tmp/julia-build/julia-release-1.2/usr/share/julia/stdlib/v1.2/Pkg/src/API.jl:564\n"
23 | ]
24 | },
25 | {
26 | "name": "stdout",
27 | "output_type": "stream",
28 | "text": [
29 | "\u001b[32m\u001b[1m Updating\u001b[22m\u001b[39m registry at `~/.julia/registries/General`\n",
30 | "\u001b[32m\u001b[1m Updating\u001b[22m\u001b[39m git-repo `https://github.com/JuliaRegistries/General.git`\n",
31 | "\u001b[2K\u001b[?25h[1mFetching:\u001b[22m\u001b[39m [========================================>] 100.0 %.0 %"
32 | ]
33 | },
34 | {
35 | "name": "stderr",
36 | "output_type": "stream",
37 | "text": [
38 | "┌ Info: Recompiling stale cache file /Users/sabae/.julia/compiled/v1.2/Zygote/4kbLI.ji for Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]\n",
39 | "└ @ Base loading.jl:1240\n"
40 | ]
41 | }
42 | ],
43 | "source": [
44 | "# Initialize environment in current directory, to load\n",
45 | "import Pkg; Pkg.activate(@__DIR__); Pkg.instantiate()\n",
46 | "using Zygote, LinearAlgebra"
47 | ]
48 | },
49 | {
50 | "cell_type": "markdown",
51 | "metadata": {},
52 | "source": [
53 | "This example will showcase how we do a simple linear fit with Zygote, making use of complex datastructures, a home-grown stochastic gradient descent optimizer, and some good old-fashioned math. We start with the problem\n",
54 | "statement: We wish to learn the mapping `f(X) -> Y`, where `X` is a matrix of vector observations, `f()` is a linear mapping function and `Y` is a vector of scalar observations.\n",
55 | "\n",
56 | "Because we like complex objects, we will define our linear regression as the following object:"
57 | ]
58 | },
59 | {
60 | "cell_type": "code",
61 | "execution_count": 4,
62 | "metadata": {},
63 | "outputs": [
64 | {
65 | "data": {
66 | "text/plain": [
67 | "LinearRegression"
68 | ]
69 | },
70 | "execution_count": 4,
71 | "metadata": {},
72 | "output_type": "execute_result"
73 | }
74 | ],
75 | "source": [
76 | "# LinearRegression object, containing multiple fields, some of which will be learned.\n",
77 | "mutable struct LinearRegression\n",
78 | " # These values will be implicitly learned\n",
79 | " weights::Matrix\n",
80 | " bias::Float64\n",
81 | "\n",
82 | " # These values will not be learned\n",
83 | " name::String\n",
84 | "end\n",
85 | "\n",
86 | "LinearRegression(nparams, name) = LinearRegression(randn(1, nparams), 0.0, name)"
87 | ]
88 | },
89 | {
90 | "cell_type": "markdown",
91 | "metadata": {},
92 | "source": [
93 | "We will define two verbs to act upon a `LinearRegression` object; `predict()`, to perform the linear regression, and `loss()` to measure the $\\ell_2$ norm between a target and our current prediction."
94 | ]
95 | },
96 | {
97 | "cell_type": "code",
98 | "execution_count": 5,
99 | "metadata": {},
100 | "outputs": [
101 | {
102 | "data": {
103 | "text/plain": [
104 | "loss (generic function with 1 method)"
105 | ]
106 | },
107 | "execution_count": 5,
108 | "metadata": {},
109 | "output_type": "execute_result"
110 | }
111 | ],
112 | "source": [
113 | "# Our linear regression looks very familiar; w*X + b\n",
114 | "function predict(model::LinearRegression, X)\n",
115 | " return model.weights * X .+ model.bias\n",
116 | "end\n",
117 | "\n",
118 | "# Our \"loss\" that must be minimized is the l2 norm between our current\n",
119 | "# prediction and our ground-truth Y\n",
120 | "function loss(model::LinearRegression, X, Y)\n",
121 | " return norm(predict(model, X) .- Y, 2)\n",
122 | "end\n"
123 | ]
124 | },
125 | {
126 | "cell_type": "code",
127 | "execution_count": 6,
128 | "metadata": {},
129 | "outputs": [
130 | {
131 | "data": {
132 | "text/plain": [
133 | "4×10000 Array{Float64,2}:\n",
134 | " -1.19203 1.91616 0.843466 … -1.57857 0.246021 -0.597997\n",
135 | " -0.48234 1.32798 -0.899407 0.043473 1.44814 -0.89648 \n",
136 | " 1.55866 0.0718622 1.89751 0.994072 0.638415 -0.273722\n",
137 | " -0.558112 -1.68074 0.954623 0.442166 -0.187531 -0.92237 "
138 | ]
139 | },
140 | "execution_count": 6,
141 | "metadata": {},
142 | "output_type": "execute_result"
143 | }
144 | ],
145 | "source": [
146 | "# Our \"ground truth\" values (that we will learn, to prove that this works)\n",
147 | "weights_gt = [1.0, 2.7, 0.3, 1.2]'\n",
148 | "bias_gt = 0.4\n",
149 | "\n",
150 | "# Generate a dataset of many observations\n",
151 | "X = randn(length(weights_gt), 10000)\n",
152 | "Y = weights_gt * X .+ bias_gt\n",
153 | "\n",
154 | "# Add a little bit of noise to `X` so that we do not have an exact solution,\n",
155 | "# but must instead do a least-squares fit:\n",
156 | "X .+= 0.001.*randn(size(X))"
157 | ]
158 | },
159 | {
160 | "cell_type": "code",
161 | "execution_count": 8,
162 | "metadata": {},
163 | "outputs": [
164 | {
165 | "data": {
166 | "text/plain": [
167 | "(Base.RefValue{Any}((weights = [-1.192026114794742 -0.482340240894484 1.5586551748882012 -0.5581118308574825], bias = 1.0, name = nothing)),)"
168 | ]
169 | },
170 | "execution_count": 8,
171 | "metadata": {},
172 | "output_type": "execute_result"
173 | }
174 | ],
175 | "source": [
176 | "# Now we begin our \"training loop\", where we take examples from `X`,\n",
177 | "# calculate loss with respect to the corresponding entry in `Y`, find the\n",
178 | "# gradient upon our model, update the model, and continue. Before we jump\n",
179 | "# in, let's look at what `Zygote.gradient()` gives us:\n",
180 | "model = LinearRegression(size(X, 1), \"Example\")\n",
181 | "\n",
182 | "# Calculate gradient upon `model` for the first example in our training set\n",
183 | "grads = Zygote.gradient(model) do m\n",
184 | " return loss(m, X[:,1], Y[1])\n",
185 | "end"
186 | ]
187 | },
188 | {
189 | "cell_type": "markdown",
190 | "metadata": {},
191 | "source": [
192 | "The `grads` object is a Tuple containing one element per argument to `gradient()`, so we take the first one to get the gradient upon `model`:"
193 | ]
194 | },
195 | {
196 | "cell_type": "code",
197 | "execution_count": 9,
198 | "metadata": {},
199 | "outputs": [
200 | {
201 | "data": {
202 | "text/plain": [
203 | "Base.RefValue{Any}((weights = [-1.192026114794742 -0.482340240894484 1.5586551748882012 -0.5581118308574825], bias = 1.0, name = nothing))"
204 | ]
205 | },
206 | "execution_count": 9,
207 | "metadata": {},
208 | "output_type": "execute_result"
209 | }
210 | ],
211 | "source": [
212 | "grads = grads[1]"
213 | ]
214 | },
215 | {
216 | "cell_type": "markdown",
217 | "metadata": {},
218 | "source": [
219 | "Because our LinearRegression object is mutable, the gradient holds a reference to it, which we peel via `grads[]`:\n"
220 | ]
221 | },
222 | {
223 | "cell_type": "code",
224 | "execution_count": 10,
225 | "metadata": {},
226 | "outputs": [
227 | {
228 | "data": {
229 | "text/plain": [
230 | "(weights = [-1.192026114794742 -0.482340240894484 1.5586551748882012 -0.5581118308574825], bias = 1.0, name = nothing)"
231 | ]
232 | },
233 | "execution_count": 10,
234 | "metadata": {},
235 | "output_type": "execute_result"
236 | }
237 | ],
238 | "source": [
239 | "grads = grads[]"
240 | ]
241 | },
242 | {
243 | "cell_type": "markdown",
244 | "metadata": {},
245 | "source": [
246 | "We now get a `NamedTuple` so we can now do things like `grads.weights`. Note that while `weights` and `bias` have gradients, `name` just naturally has a gradient of `nothing`, because it was not involved in the calculation of the output loss."
247 | ]
248 | },
249 | {
250 | "cell_type": "code",
251 | "execution_count": 11,
252 | "metadata": {},
253 | "outputs": [
254 | {
255 | "data": {
256 | "text/plain": [
257 | "1×4 Array{Float64,2}:\n",
258 | " -1.19203 -0.48234 1.55866 -0.558112"
259 | ]
260 | },
261 | "execution_count": 11,
262 | "metadata": {},
263 | "output_type": "execute_result"
264 | }
265 | ],
266 | "source": [
267 | "grads.weights"
268 | ]
269 | },
270 | {
271 | "cell_type": "markdown",
272 | "metadata": {},
273 | "source": [
274 | "Next, we will define an update rule that will allow us to modify the weights of our model according to the gradients, using the simplest gradient descent update rule. We'll then run a training loop to update our weights with the loss from the training set, as we would expect:"
275 | ]
276 | },
277 | {
278 | "cell_type": "code",
279 | "execution_count": 12,
280 | "metadata": {},
281 | "outputs": [
282 | {
283 | "data": {
284 | "text/plain": [
285 | "sgd_update! (generic function with 2 methods)"
286 | ]
287 | },
288 | "execution_count": 12,
289 | "metadata": {},
290 | "output_type": "execute_result"
291 | }
292 | ],
293 | "source": [
294 | "# Let's define \n",
295 | "function sgd_update!(model::LinearRegression, grads, η = 0.001)\n",
296 | " model.weights .-= η .* grads.weights\n",
297 | " model.bias -= η * grads.bias\n",
298 | "end"
299 | ]
300 | },
301 | {
302 | "cell_type": "code",
303 | "execution_count": 13,
304 | "metadata": {},
305 | "outputs": [
306 | {
307 | "name": "stderr",
308 | "output_type": "stream",
309 | "text": [
310 | "┌ Info: Running train loop for 10000 iterations\n",
311 | "└ @ Main In[13]:2\n"
312 | ]
313 | }
314 | ],
315 | "source": [
316 | "# Now let's do that for each example in our training set:\n",
317 | "@info(\"Running train loop for $(size(X,2)) iterations\")\n",
318 | "for idx in 1:size(X, 2)\n",
319 | " grads = Zygote.gradient(m -> loss(m, X[:, idx], Y[idx]), model)[1][]\n",
320 | " sgd_update!(model, grads)\n",
321 | "end"
322 | ]
323 | },
324 | {
325 | "cell_type": "code",
326 | "execution_count": 14,
327 | "metadata": {},
328 | "outputs": [
329 | {
330 | "data": {
331 | "text/plain": [
332 | "1×4 Adjoint{Float64,Array{Float64,1}}:\n",
333 | " 1.0 2.7 0.3 1.2"
334 | ]
335 | },
336 | "execution_count": 14,
337 | "metadata": {},
338 | "output_type": "execute_result"
339 | }
340 | ],
341 | "source": [
342 | "weights_gt"
343 | ]
344 | },
345 | {
346 | "cell_type": "code",
347 | "execution_count": 15,
348 | "metadata": {},
349 | "outputs": [
350 | {
351 | "data": {
352 | "text/plain": [
353 | "1×4 Array{Float64,2}:\n",
354 | " 1.00142 2.70157 0.300252 1.20033"
355 | ]
356 | },
357 | "execution_count": 15,
358 | "metadata": {},
359 | "output_type": "execute_result"
360 | }
361 | ],
362 | "source": [
363 | "model.weights"
364 | ]
365 | },
366 | {
367 | "cell_type": "code",
368 | "execution_count": 16,
369 | "metadata": {},
370 | "outputs": [
371 | {
372 | "data": {
373 | "text/plain": [
374 | "0.4"
375 | ]
376 | },
377 | "execution_count": 16,
378 | "metadata": {},
379 | "output_type": "execute_result"
380 | }
381 | ],
382 | "source": [
383 | "bias_gt"
384 | ]
385 | },
386 | {
387 | "cell_type": "code",
388 | "execution_count": 17,
389 | "metadata": {},
390 | "outputs": [
391 | {
392 | "data": {
393 | "text/plain": [
394 | "0.3980000000000003"
395 | ]
396 | },
397 | "execution_count": 17,
398 | "metadata": {},
399 | "output_type": "execute_result"
400 | }
401 | ],
402 | "source": [
403 | "model.bias"
404 | ]
405 | }
406 | ],
407 | "metadata": {
408 | "@webio": {
409 | "lastCommId": null,
410 | "lastKernelId": null
411 | },
412 | "kernelspec": {
413 | "display_name": "Julia 1.2.0-rc1",
414 | "language": "julia",
415 | "name": "julia-1.2"
416 | },
417 | "language_info": {
418 | "file_extension": ".jl",
419 | "mimetype": "application/julia",
420 | "name": "julia",
421 | "version": "1.2.0"
422 | }
423 | },
424 | "nbformat": 4,
425 | "nbformat_minor": 2
426 | }
427 |
--------------------------------------------------------------------------------