├── .gitignore ├── .nojekyll ├── CODE_OF_CONDUCT.md ├── Deep-Learning-and-Scientific-Computing-with-R-torch.Rproj ├── Deep-Learning-and-Scientific-Computing-with-R-torch.tex ├── README.md ├── _quarto.yml ├── audio_classification.qmd ├── autograd.qmd ├── basics_overview.qmd ├── data.qmd ├── dl_overview.qmd ├── docs ├── .nojekyll ├── audio_classification.html ├── autograd.html ├── basics_overview.html ├── data.html ├── dl_overview.html ├── fourier_transform_dft.html ├── fourier_transform_fft.html ├── image_classification_1.html ├── image_classification_2.html ├── image_segmentation.html ├── images │ ├── audio-alluvial-baseline.png │ ├── audio-alluvial-complex.png │ ├── audio-alluvial-mel.png │ ├── audio-bird-dft.png │ ├── audio-bird-waveform.png │ ├── audio-fit-baseline.png │ ├── audio-fit-complex.png │ ├── audio-fit-mel.png │ ├── audio-lr-finder-baseline.png │ ├── audio-lr-finder-complex.png │ ├── audio-lr-finder-mel.png │ ├── audio-mel-filterbank.png │ ├── audio-mel-spectrogram.png │ ├── audio-spectrogram.png │ ├── autograd-compgraph.png │ ├── autograd-paraboloid.png │ ├── dft-cos-1-rev-dft.png │ ├── dft-cos-1-rev.png │ ├── dft-cos-32-rev.png │ ├── dft-cos-4-rev.png │ ├── dft-cos-mult-amplitude.png │ ├── dft-cos-phase-pi12.png │ ├── dft-dial-dft.png │ ├── dft-dial-fft.png │ ├── dft-dial-ifft.png │ ├── dft-dial-spectrogram.png │ ├── dft-dial-waveform.png │ ├── dft-mix-of-sinuisoids.png │ ├── efficiency-mnist-lr-finder.png │ ├── fft-perf-jit.png │ ├── fft-perf.png │ ├── images-conv-arithmetic-dilation.png │ ├── images-conv-arithmetic-padding.png │ ├── images-conv-arithmetic-strides.png │ ├── images-cross-correlation.png │ ├── images-feature-visualization.png │ ├── images-square-bottom.png │ ├── images-square-left.png │ ├── images-square-right.png │ ├── images-square-top.png │ ├── images-square.png │ ├── images2-lr-finder-batchnorm.png │ ├── images2-lr-finder-dropout.png │ ├── images2-lr-finder-resnet.png │ ├── least-squares-benchmark.png │ ├── optim-1-rosenbrock.png │ ├── optimizers-adagrad.png │ ├── optimizers-adam.png │ ├── optimizers-momentum.png │ ├── optimizers-rmsprop.png │ ├── optimizers-steepest-descent-elliptic.png │ ├── optimizers-steepest-descent-symmetric.png │ ├── overfitting-mnist-mixup-0.5.png │ ├── overfitting-mnist-mixup-0.7.png │ ├── overfitting-mnist-mixup-0.9.png │ ├── overfitting-mnist-test.png │ ├── overfitting-mnist-transforms.png │ ├── segmentation-conv-arithmetic-transposed.png │ ├── segmentation-lr-finder.png │ ├── segmentation-segmentation.png │ ├── segmentation-unet.png │ ├── tabular-heart-disease-biplot.png │ ├── tabular-heart-disease-fit.png │ ├── tabular-heart-disease-pca.png │ ├── tensors-dimensions.png │ ├── timeseries-vic-elec-fit-mlp.png │ ├── timeseries-vic-elec-fit.png │ ├── timeseries-vic-elec-lr-finder-mlp.png │ ├── timeseries-vic-elec-lr-finder.png │ ├── timeseries-vic-elec-preds-mlp.png │ ├── timeseries-vic-elec-preds.png │ ├── timeseries-vic-elec-stl-month.png │ ├── timeseries-vic-elec-stl.png │ ├── wav-K-omega.png │ ├── wav-chaffinch-fft.png │ ├── wav-chaffinch-spectrogram.png │ ├── wav-chaffinch-waveletdiag.png │ ├── wav-chaffinch.png │ ├── wav-example-long-diag1.png │ ├── wav-example-long-diag2.png │ ├── wav-example-long-diag3.png │ ├── wav-example-long-diag4.png │ ├── wav-example-long-diag5.png │ ├── wav-example-long-diag6.png │ ├── wav-example-long-transform.png │ ├── wav-example-long-transform2.png │ ├── wav-morlet-example.png │ ├── wav-signal-long.png │ ├── wav-signal-short-transform-largeK.png │ ├── wav-signal-short-transform.png │ └── wav-signal-short.png ├── index.html ├── loss_functions.html ├── matrix_computations_convolution.html ├── matrix_computations_leastsquares.html ├── modules.html ├── network_1.html ├── network_2.html ├── optim_1.html ├── optim_2.html ├── optimizers.html ├── other_overview.html ├── overfitting.html ├── references.html ├── search.json ├── site_libs │ ├── bootstrap │ │ ├── bootstrap-icons.css │ │ ├── bootstrap-icons.woff │ │ ├── bootstrap.min.css │ │ └── bootstrap.min.js │ ├── clipboard │ │ └── clipboard.min.js │ ├── quarto-html │ │ ├── anchor.min.js │ │ ├── popper.min.js │ │ ├── quarto-syntax-highlighting.css │ │ ├── quarto.js │ │ ├── tippy.css │ │ └── tippy.umd.min.js │ ├── quarto-nav │ │ ├── headroom.min.js │ │ └── quarto-nav.js │ └── quarto-search │ │ ├── autocomplete.umd.js │ │ ├── fuse.min.js │ │ └── quarto-search.js ├── tabular_data.html ├── tensors.html ├── time_series.html ├── training_efficiency.html ├── training_with_luz.html ├── wavelets.html └── what_is_torch.html ├── errata.md ├── fourier_transform_dft.qmd ├── fourier_transform_fft.qmd ├── image_classification_1.qmd ├── image_classification_2.qmd ├── image_segmentation.qmd ├── images ├── adagrad.ggb ├── adam.ggb ├── audio-alluvial-baseline.png ├── audio-alluvial-complex.png ├── audio-alluvial-mel.png ├── audio-bird-dft.png ├── audio-bird-waveform.png ├── audio-fit-baseline.png ├── audio-fit-complex.png ├── audio-fit-mel.png ├── audio-lr-finder-baseline.png ├── audio-lr-finder-complex.png ├── audio-lr-finder-mel.png ├── audio-mel-filterbank.png ├── audio-mel-spectrogram.png ├── audio-spectrogram-zoom.png ├── audio-spectrogram.png ├── autograd-compgraph.png ├── autograd-paraboloid.png ├── compgraph.odg ├── cross-correlation.ggb ├── cross-correlation.ods ├── dft-cos-1-rev-dft.png ├── dft-cos-1-rev.png ├── dft-cos-32-rev.png ├── dft-cos-4-rev.png ├── dft-cos-mult-amplitude.png ├── dft-cos-phase-pi12.png ├── dft-dial-dft.png ├── dft-dial-fft.png ├── dft-dial-ifft.png ├── dft-dial-spectrogram.png ├── dft-dial-waveform.png ├── dft-mix-of-sinuisoids.png ├── dimensions.ggb ├── dimensions.xcf ├── efficiency-mnist-lr-finder.png ├── fft-perf-jit.png ├── fft-perf.png ├── images-conv-arithmetic-dilation.png ├── images-conv-arithmetic-padding.png ├── images-conv-arithmetic-strides.png ├── images-cross-correlation.png ├── images-feature-visualization.png ├── images-square-bottom.png ├── images-square-filters.png ├── images-square-left.png ├── images-square-right.png ├── images-square-top.png ├── images-square.png ├── images2-lr-finder-batchnorm.png ├── images2-lr-finder-dropout.png ├── images2-lr-finder-resnet.png ├── least-squares-benchmark.png ├── momentum.ggb ├── nesterov.ggb ├── nesterov_rosenbrock.ggb ├── optim-1-rosenbrock.png ├── optimizers-adagrad.png ├── optimizers-adam.png ├── optimizers-momentum.png ├── optimizers-rmsprop.png ├── optimizers-steepest-descent-elliptic.png ├── optimizers-steepest-descent-symmetric.png ├── overfitting-mnist-mixup-0.5.png ├── overfitting-mnist-mixup-0.7.png ├── overfitting-mnist-mixup-0.9.png ├── overfitting-mnist-test.png ├── overfitting-mnist-transforms.png ├── paraboloid.ggb ├── rmsprop.ggb ├── segmentation-conv-arithmetic-transposed.png ├── segmentation-lr-finder.png ├── segmentation-segmentation.png ├── segmentation-unet.png ├── square.xcf ├── steepest_descent_elliptic.ggb ├── steepest_descent_symmetric.ggb ├── tabular-heart-disease-biplot.png ├── tabular-heart-disease-fit.png ├── tabular-heart-disease-pca.png ├── tensors-dimensions.png ├── timeseries-vic-elec-fit-mlp.png ├── timeseries-vic-elec-fit.png ├── timeseries-vic-elec-lr-finder-mlp.png ├── timeseries-vic-elec-lr-finder.png ├── timeseries-vic-elec-preds-mlp.png ├── timeseries-vic-elec-preds.png ├── timeseries-vic-elec-stl-month.png ├── timeseries-vic-elec-stl.png ├── wav-K-omega.png ├── wav-chaffinch-fft.png ├── wav-chaffinch-spectrogram.png ├── wav-chaffinch-waveletdiag.png ├── wav-chaffinch.png ├── wav-example-long-diag1.png ├── wav-example-long-diag2.png ├── wav-example-long-diag3.png ├── wav-example-long-diag4.png ├── wav-example-long-diag5.png ├── wav-example-long-diag6.png ├── wav-example-long-transform.png ├── wav-example-long-transform2.png ├── wav-morlet-example.png ├── wav-signal-long.png ├── wav-signal-short-transform-largeK.png ├── wav-signal-short-transform.png └── wav-signal-short.png ├── index.qmd ├── intro.qmd ├── krantz.cls ├── loss_functions.qmd ├── matrix_computations_convolution.qmd ├── matrix_computations_leastsquares.qmd ├── modules.qmd ├── network_1.qmd ├── network_2.qmd ├── optim_1.qmd ├── optim_2.qmd ├── optimizers.qmd ├── other_overview.qmd ├── overfitting.qmd ├── references.bib ├── references.qmd ├── resources ├── adagrad.tex ├── adam.tex ├── chaffinch.wav ├── dft-dial.wav ├── dimensions.tex ├── matrix-weather.csv ├── momentum.tex ├── paraboloid.tex ├── rmsprop.tex ├── steepest-descent-elliptic.tex ├── steepest-descent-symmetric.tex └── tabular-heart.csv ├── tabular_data.qmd ├── tensors.qmd ├── time_series.qmd ├── training_efficiency.qmd ├── training_with_luz.qmd ├── wavelets.qmd └── what_is_torch.qmd /.gitignore: -------------------------------------------------------------------------------- 1 | .Rproj.user 2 | .Rhistory 3 | .RData 4 | .Ruserdata 5 | .data 6 | .models 7 | BookProposalForm.odt 8 | index.log 9 | tmp 10 | _book 11 | /.quarto/ 12 | *_cache/ 13 | resources/*.log 14 | resources/*.pdf 15 | resources/*.aux 16 | resources/*synctex.gz 17 | crc 18 | docs/Deep-Learning-and-Scientific-Computing-with-R-torch.pdf 19 | 20 | -------------------------------------------------------------------------------- /.nojekyll: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/.nojekyll -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | --- 2 | editor: 3 | markdown: 4 | wrap: 72 5 | --- 6 | 7 | # Contributor Code of Conduct 8 | 9 | As contributors and maintainers of this project, we pledge to respect 10 | all people who contribute through reporting issues, posting feature 11 | requests, updating documentation, submitting pull requests or patches, 12 | and other activities. 13 | 14 | We are committed to making participation in this project a 15 | harassment-free experience for everyone, regardless of level of 16 | experience, gender, gender identity and expression, sexual orientation, 17 | disability, personal appearance, body size, race, ethnicity, age, or 18 | religion. 19 | 20 | Examples of unacceptable behavior by participants include the use of 21 | sexual language or imagery, derogatory comments or personal attacks, 22 | trolling, public or private harassment, insults, or other unprofessional 23 | conduct. 24 | 25 | Project maintainers have the right and responsibility to remove, edit, 26 | or reject comments, commits, code, wiki edits, issues, and other 27 | contributions that are not aligned to this Code of Conduct. Project 28 | maintainers who do not follow the Code of Conduct may be removed from 29 | the project team. 30 | 31 | Instances of abusive, harassing, or otherwise unacceptable behavior may 32 | be reported by opening an issue or contacting one or more of the project 33 | maintainers. 34 | 35 | This Code of Conduct is adapted from the Contributor Covenant 36 | (https://www.contributor-covenant.org), version 1.0.0, available at 37 | https://contributor-covenant.org/version/1/0/0/. 38 | -------------------------------------------------------------------------------- /Deep-Learning-and-Scientific-Computing-with-R-torch.Rproj: -------------------------------------------------------------------------------- 1 | Version: 1.0 2 | 3 | RestoreWorkspace: Default 4 | SaveWorkspace: Default 5 | AlwaysSaveHistory: Default 6 | 7 | EnableCodeIndexing: Yes 8 | UseSpacesForTab: Yes 9 | NumSpacesForTab: 2 10 | Encoding: UTF-8 11 | 12 | RnwWeave: Sweave 13 | LaTeX: pdfLaTeX 14 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | This is the repository for the book [Deep Learning and Scientific Computing with R torch](https://skeydan.github.io/Deep-Learning-and-Scientific-Computing-with-R-torch/), written by [Sigrid Keydana](https://divergences.xyz/) and published by [CRC Press](https://doi.org/10.1201/9781003275923). 2 | 3 | A list of errata may be found in `errata.md`. 4 | 5 | To cite using `bibtex`, you could use 6 | 7 | ``` 8 | @book{keydana2023deep, 9 | title={Deep Learning and Scientific Computing with R torch}, 10 | author={Keydana, Sigrid}, 11 | year={2023}, 12 | publisher={CRC Press} 13 | } 14 | ``` 15 | 16 | Please note that this work is written under a Contributor Code of Conduct, and the online version is licensed under a [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-nc-sa/4.0/). By participating in this project (for example, by submitting an issue with suggestions or edits) you agree to abide by its terms. 17 | -------------------------------------------------------------------------------- /_quarto.yml: -------------------------------------------------------------------------------- 1 | project: 2 | type: book 3 | output-dir: docs 4 | book: 5 | title: "Deep Learning and Scientific Computing with R torch" 6 | author: "Sigrid Keydana" 7 | date: April, 2023 8 | chapters: 9 | - index.qmd 10 | - part: "Getting familiar with torch" 11 | chapters: 12 | - basics_overview.qmd 13 | - what_is_torch.qmd 14 | - tensors.qmd 15 | - autograd.qmd 16 | - optim_1.qmd 17 | - network_1.qmd 18 | - modules.qmd 19 | - optimizers.qmd 20 | - loss_functions.qmd 21 | - optim_2.qmd 22 | - network_2.qmd 23 | - part: "Deep learning with torch" 24 | chapters: 25 | - dl_overview.qmd 26 | - data.qmd 27 | - training_with_luz.qmd 28 | - image_classification_1.qmd 29 | - overfitting.qmd 30 | - training_efficiency.qmd 31 | - image_classification_2.qmd 32 | - image_segmentation.qmd 33 | - tabular_data.qmd 34 | - time_series.qmd 35 | - audio_classification.qmd 36 | - part: "Other things to do with torch: Matrices, Fourier Transform, and Wavelets" 37 | chapters: 38 | - other_overview.qmd 39 | - matrix_computations_leastsquares.qmd 40 | - matrix_computations_convolution.qmd 41 | - fourier_transform_dft.qmd 42 | - fourier_transform_fft.qmd 43 | - wavelets.qmd 44 | - references.qmd 45 | bibliography: references.bib 46 | format: 47 | html: 48 | theme: minty 49 | pdf: 50 | keep-tex: true 51 | documentclass: krantz 52 | toc-title: "Contents" 53 | lof: true 54 | fig-pos: "H" 55 | highlight-style: none 56 | colorlinks: false 57 | links-as-notes: true 58 | include-in-header: 59 | text: | 60 | \usepackage{makeidx} 61 | \makeindex 62 | include-after-body: 63 | text: | 64 | \printindex 65 | execute: 66 | eval: false 67 | crossref: 68 | fig-prefix: fig. 69 | tbl-prefix: tbl. 70 | 71 | 72 | 73 | 74 | -------------------------------------------------------------------------------- /autograd.qmd: -------------------------------------------------------------------------------- 1 | # Autograd {#sec:autograd} 2 | 3 | In the last chapter, we have seen how to manipulate tensors, and encountered a sample of mathematical operations one can perform on them. If those operations, numerous though they may be, were all there was to core `torch`, you would not be reading this book. Frameworks like `torch` are so popular because of what you can do with them: deep learning, machine learning, optimization, large-scale scientific computation in general. Most of these application areas involve minimizing some *loss function*. This, in turn, entails computing function *derivatives*. Now imagine that as a user, you had to specify the functional form of each derivative yourself. Especially with neural networks, this could get cumbersome pretty fast! 4 | 5 | In fact, `torch` does not work out, nor store, functional representations of derivatives either. Instead, it implements what is called *automatic differentiation*\index{automatic differentiation}. In automatic differentiation, and more specifically, its often-used *reverse-mode* variant, derivatives are computed and combined on a *backward pass* through the graph of tensor operations. We'll look at an example for this in a minute. But first, let's quickly backtrack and talk about *why* we would want to compute derivatives. 6 | 7 | ## Why compute derivatives? 8 | 9 | In supervised machine learning, we have at our disposal a *training set*, where the variable we're hoping to predict is known. This is the target, or *ground truth*. We now develop and train a prediction algorithm, based on a set of input variables, the *predictors*. This training, or learning, process, is based on comparing the algorithm's predictions with the ground truth, a comparison that leads to a number capturing how good or bad the current predictions are. To provide this number is the job of the *loss function*\index{loss function}. 10 | 11 | Once it is aware of the current loss, an algorithm can adjust its parameters -- the *weights*, in a neural network -- in order to deliver better predictions. It just has to know in which direction to adjust them. This information is made available by the *gradient*\index{gradient}, the vector of derivatives\index{derivative(s)}. 12 | 13 | As an example, we imagine a loss function that looks like this (@fig-autograd-paraboloid): 14 | 15 | ![Hypothetical loss function (a paraboloid).](images/autograd-paraboloid.png){#fig-autograd-paraboloid fig-alt="A paraboloid in two dimensions that has a minimum at (0,0)."} 16 | 17 | This is a quadratic function of two variables: $f(x_1, x_2) = 0.2 {x_1}^2 + 0.2 {x_2}^2 - 5$. It has its minimum at `(0,0)`, and this is the point we'd like to be at. As humans, standing at the location designated by the white dot, and looking at the landscape, we have a pretty clear idea how to go downhill fast (assuming we're not scared by the slope). To find the best direction computationally, however, we compute the gradient. 18 | 19 | Take the $x_1$ direction. The derivative of the function with respect to $x_1$ indicates how its value varies as $x_1$ varies. As we know the function in closed form, we can compute that: $\frac{\partial f}{\partial x_1} = 0.4 x_1$. This tells us that as $x_1$ increases, loss increases, and how fast. But we want loss to *decrease*, so we have to go in the opposite direction. 20 | 21 | The same holds for the $x_2$-axis. We compute the derivative ($\frac{\partial f}{\partial x_2} = 0.4 x_2$). Again, we want to take the direction opposite to where the derivative points. Overall, this yields a descent direction of $\begin{bmatrix}-0.4x_1\\-0.4x_2 \end{bmatrix}$. 22 | 23 | Descriptively, this strategy is called *steepest descent*\index{descent!steepest}. Commonly referred to as *gradient descent*\index{descent!gradient}, it is the most basic optimization algorithm in deep learning. Perhaps unintuitively, it is not always the most efficient way. And there's another question: Can we assume that this direction, computed at the starting point, will remain optimal as we continue descending? Maybe we'd better regularly recompute directions instead? Questions like this will be addressed in later chapters. 24 | 25 | ## Automatic differentiation example 26 | 27 | Now that we know why we need derivatives, let's see how automatic differentiation (AD) would compute them. 28 | 29 | This (@fig-autograd-compgraph) is how our above function could be represented in a computational graph. `x1` and `x2` are input nodes, corresponding to function parameters $x_1$ and $x_2$. `x7` is the function's output; all other nodes are intermediate ones, necessary to ensure correct order of execution. (We could have given the constants, `-5` , `0.2`, and `2`, their own nodes as well; but as they're remaining, well, constant anyway, we're not too interested in them and prefer having a simpler graph.) 30 | 31 | ![Example of a computational graph.](images/autograd-compgraph.png){#fig-autograd-compgraph fig-alt="A directed graph where nodes represent data, and arrows, mathematical operations. There are two input nodes, four intermediate nodes, and one output node. Operations used are exponentiation, multiplication, and addition."} 32 | 33 | In reverse-mode AD, the flavor of automatic differentiation implemented by `torch`, the first thing that happens is to calculate the function's output value. This corresponds to a forward pass through the graph. Then, a backward pass is performed to calculate the gradient of the output with respect to both inputs, `x1` and `x2`. In this process, information becomes available, and is built up, from the right: 34 | 35 | - At `x7`, we calculate partial derivatives with respect to `x5` and `x6`. Basically, the equation to differentiate looks like this: $f(x_5, x_6) = x_5 + x_6 - 5$. Thus, both partial derivatives are 1. 36 | 37 | - From `x5`, we move to the left to see how it depends on `x3`. We find that $\frac{\partial x_5}{\partial x_3} = 0.2$. At this point, applying the chain rule of calculus, we already know how the output depends on `x3`: $\frac{\partial f}{\partial x_3} = 0.2 * 1 = 0.2$. 38 | 39 | - From `x3`, we take the final step to `x`. We learn that $\frac{\partial x_3}{\partial x_1} = 2 x_1$. Now, we again apply the chain rule, and are able to formulate how the function depends on its first input: $\frac{\partial f}{\partial x_1} = 2 x_1 * 0.2 * 1 = 0.4 x_1$. 40 | 41 | - Analogously, we determine the second partial derivative, and thus, already have the gradient available: $\nabla f = \frac{\partial f}{\partial x_1} + \frac{\partial f}{\partial x_2} = 0.4 x_1 + 0.4 x_2$. 42 | 43 | That is the principle. In practice, different frameworks implement reverse-mode automatic differentiation differently. We'll catch a glimpse of how `torch` does it in the next section. 44 | 45 | ## Automatic differentiation with `torch` *autograd* 46 | 47 | First, a quick note on terminology. In `torch`, the AD engine is usually referred to as *autograd*, and that is the way you'll see it denoted in most of the rest of this book. Now, back to the task. 48 | 49 | To construct the above computational graph with `torch`, we create "source" tensors `x1` and `x2`. These will mimic the parameters whose impact we're interested in. However, if we just proceed "as usual", creating the tensors the way we've been doing so far, `torch` will not prepare for AD. Instead, we need to pass in `requires_grad = TRUE` when instantiating those tensors: 50 | 51 | ```{r} 52 | library(torch) 53 | 54 | x1 <- torch_tensor(2, requires_grad = TRUE) 55 | x2 <- torch_tensor(2, requires_grad = TRUE) 56 | ``` 57 | 58 | (By the way, the value `2` for both tensors was chosen completely arbitrarily.) 59 | 60 | Now, to create "invisible" nodes `x3` to `x6` , we square and multiply accordingly. Then `x7` stores the final result. 61 | 62 | ```{r} 63 | x3 <- x1$square() 64 | x5 <- x3 * 0.2 65 | 66 | x4 <- x2$square() 67 | x6 <- x4 * 0.2 68 | 69 | x7 <- x5 + x6 - 5 70 | x7 71 | ``` 72 | 73 | torch_tensor 74 | -3.4000 75 | [ CPUFloatType{1} ][ grad_fn = ] 76 | 77 | Note that we have to add `requires_grad = TRUE` when creating the "source" tensors only. All dependent nodes in the graph inherit this property. For example: 78 | 79 | ```{r} 80 | x7$requires_grad 81 | ``` 82 | 83 | [1] TRUE 84 | 85 | Now, all prerequisites are fulfilled to see automatic differentiation at work. All we need to do to determine how `x7` depends on `x1` and `x2` is call `backward()`: 86 | 87 | ```{r} 88 | x7$backward() 89 | ``` 90 | 91 | Due to this call, the `$grad` fields have been populated in `x1` and `x2`: 92 | 93 | ```{r} 94 | x1$grad 95 | x2$grad 96 | ``` 97 | 98 | 0.8000 99 | [ CPUFloatType{1} ] 100 | torch_tensor 101 | 0.8000 102 | [ CPUFloatType{1} ] 103 | 104 | These are the partial derivatives of `x7` with respect to `x1` and `x2`, respectively. Conforming to our manual calculations above, both amount to 0.8, that is, 0.4 times the tensor values 2 and 2. 105 | 106 | How about the accumulation process we said was needed to build up those end-to-end derivatives? Can we "follow" the end-to-end derivative as it's being built up? For example, can we see how the final output depends on `x3`? 107 | 108 | ```{r} 109 | x3$grad 110 | ``` 111 | 112 | [W TensorBody.h:470] Warning: The .grad attribute of a Tensor 113 | that is not a leaf Tensor is being accessed. Its .grad attribute 114 | won't be populated during autograd.backward(). 115 | If you indeed want the .grad field to be populated for a 116 | non-leaf Tensor, use .retain_grad() on the non-leaf Tensor.[...] 117 | 118 | torch_tensor 119 | [ Tensor (undefined) ] 120 | 121 | The field does not seem to be populated. In fact, while it *has* to compute them, `torch` throws away the intermediate aggregates once they are no longer needed, to save memory. We can, however, ask it to keep them, using `retain_grad = TRUE`: 122 | 123 | ```{r} 124 | x3 <- x1$square() 125 | x3$retain_grad() 126 | 127 | x5 <- x3 * 0.2 128 | x5$retain_grad() 129 | 130 | x4 <- x2$square() 131 | x4$retain_grad() 132 | 133 | x6 <- x4 * 0.2 134 | x6$retain_grad() 135 | 136 | x7 <- x5 + x6 - 5 137 | x7$backward() 138 | ``` 139 | 140 | Now, we find that `x3`'s `grad` field *is* populated: 141 | 142 | ```{r} 143 | x3$grad 144 | ``` 145 | 146 | torch_tensor 147 | 0.2000 148 | [ CPUFloatType{1} ] 149 | 150 | The same goes for `x4`, `x5`, and `x6`: 151 | 152 | ```{r} 153 | x4$grad 154 | x5$grad 155 | x6$grad 156 | ``` 157 | 158 | torch_tensor 159 | 0.2000 160 | [ CPUFloatType{1} ] 161 | torch_tensor 162 | 1 163 | [ CPUFloatType{1} ] 164 | torch_tensor 165 | 1 166 | [ CPUFloatType{1} ] 167 | 168 | There is one remaining thing we might be curious about. We've managed to catch a glimpse of the gradient-accumulation process from the "running gradient" point of view, in a sense; but how about the individual derivatives that need to be taken in order to proceed with accumulation? For example, what `x3$grad` tells us is how the output depends on the intermediate state at `x3`; how do we get from there to `x1`, the actual input node? 169 | 170 | It turns out that of that aspect, too, we can get an idea. During the forward pass, `torch` already takes a note on what it will have to do, later, to calculate the individual derivatives. This "recipe" is stored in a tensor's `grad_fn` field. For `x3`, this adds the "missing link" to `x1`: 171 | 172 | ```{r} 173 | x3$grad_fn 174 | ``` 175 | 176 | PowBackward0 177 | 178 | The same works for `x4`, `x5`, and `x6`: 179 | 180 | ```{r} 181 | x4$grad_fn 182 | x5$grad_fn 183 | x6$grad_fn 184 | ``` 185 | 186 | PowBackward0 187 | MulBackward1 188 | MulBackward1 189 | 190 | And there we are! We've seen how `torch` computes derivatives for us, and we've even caught a glimpse of how it does it. Now, we are ready to play around with our first two applied tasks. 191 | -------------------------------------------------------------------------------- /basics_overview.qmd: -------------------------------------------------------------------------------- 1 | # Overview {#sec:basics-overview} 2 | 3 | This book has three sections. The second and third will explore various deep learning applications and essential scientific computation techniques, respectively. Before though, in this first part, we are going to learn about `torch`'s basic building blocks: tensors, automatic differentiation, optimizers, and modules. I'd call this part "`torch` basics", or, following a common template, "Getting started with `torch`", were it not for a certain false impression this could create. These are basics, true, but basics in the sense of *foundations*: Having worked through the next chapters, you'll have solid conceptions about how `torch` works, and you'll have seen enough code to feel comfortable experimenting with the more involved examples encountered in later sections. In other words, you'll be, to some degree, *fluent* in `torch`. 4 | 5 | In addition, you'll have coded a neural network from scratch -- twice, even: One version will involve just raw tensors and their in-built capabilities, while the other will make use of dedicated `torch` structures that encapsulate, in an object-oriented way, functionality essential to neural network training. As a consequence, you'll be excellently equipped for part two, where we'll see how to apply deep learning to different tasks and domains. 6 | -------------------------------------------------------------------------------- /data.qmd: -------------------------------------------------------------------------------- 1 | # Loading data {#sec:data} 2 | 3 | Our toy example, which we'll see a third (and last) version of in the next chapter, had the model train on a tiny set of data -- small enough to pass all observations to the model in one go. What if that wasn't the case? Say we had 10,000 items instead, and every item was an RGB image of size 256 x 256 pixels. Even on very powerful hardware, we could not possibly train a model on the complete data all at once. 4 | 5 | For that reason, deep-learning frameworks like `torch` include an input pipeline that lets you pass data to the model in *batches* -- that is, subsets of observations. Involved in this process are two classes: `dataset()` and `dataloader()`. Before we look at how to construct instances of these, let's characterize them by what they're *for*. 6 | 7 | ## Data vs. `dataset()`\index{\texttt{dataset()}} vs. `dataloader()`\index{\texttt{dataloader()}} -- what's the difference? 8 | 9 | In this book, "dataset" (variable-width font, no parentheses), or just "the data", usually refers to things like R matrices, `data.frame`s, and what's contained therein. A `dataset()` (fixed-width font, parentheses), however, is a `torch` object that knows how to do one thing: *deliver to the caller a* *single item.* That item, usually, will be a list, consisting of one input and one target tensor. (It could be anything, though -- whatever makes sense for the task. For example, it could be a single tensor, if input and target are the same. Or more than two tensors, in case different inputs should be passed to different modules.) 10 | 11 | As long as it fulfills the above-stated contract, a `dataset()` is free to do whatever needs to be done. It could, for example, download data from the internet, store them in some temporary location, do some pre-processing, and when asked, return bite-sized chunks of data in just the shape expected by a certain class of models. No matter what it does in the background, all its caller cares about is that it return a single item. Its caller, that's the `dataloader()`. 12 | 13 | A `dataloader()`'s role is to feed input to the model in *batches*. One immediate reason is computer memory: Most `dataset()`s will be far too large to pass them to the model in one go. But there are additional benefits to batching. Since gradients are computed (and model weights updated) once per *batch*, there is an inherent stochasticity to the process, a stochasticity that helps with model training. We'll talk more about that in an upcoming chapter. 14 | 15 | ## Using `dataset()`s 16 | 17 | `dataset()`s come in all flavors, from ready-to-use -- and brought to you by some package, `torchvision` or `torchdatasets`, say, or any package that chooses to provide access to data in `torch`-ready form -- to fully customized (made by you, that is). Creating `dataset()`s is straightforward, since they are R6 objects, and there's just three methods to be implemented. These methods are: 18 | 19 | 1. `initialize(...)`. Parameters to `initialize()` are passed when a `dataset()` is instantiated. Possibilities include, but are not limited to, references to R `data.frame`s, filesystem paths, download URLs, and any configurations and parameterizations expected by the `dataset()`. 20 | 2. `.getitem(i)`. This is the method responsible for fulfilling the contract. Whatever it returns counts as a single item. The parameter, `i`, is an index that, in many cases, will be used to determine the starting position in the underlying data structure (a `data.frame` of file system paths, for example). However, the `dataset()` is not *obliged* to actually make use of that parameter. With extremely huge `dataset()`s, for example, or given serious class imbalance, it could instead decide to return items based on *sampling*. 21 | 3. `.length()`. This, usually, is a one-liner, its only purpose being to inform about the number of available items in a `dataset()`. 22 | 23 | Here is a blueprint for creating a `dataset()`: 24 | 25 | ```{r} 26 | ds <- dataset()( 27 | initialize = function(...) { 28 | ... 29 | }, 30 | .getitem = function(index) { 31 | ... 32 | }, 33 | .length = function() { 34 | ... 35 | } 36 | ) 37 | ``` 38 | 39 | That said, let's compare three ways of obtaining a `dataset()` to work with, from tailor-made to maximally effortless. 40 | 41 | ### A self-built `dataset()` 42 | 43 | Let's say we wanted to build a classifier based on the popular `iris` alternative, `palmerpenguins`. 44 | 45 | ```{r} 46 | library(torch) 47 | library(palmerpenguins) 48 | library(dplyr) 49 | 50 | penguins %>% glimpse() 51 | ``` 52 | 53 | $ species Adelie, Adelie, Adelie, Adelie,... 54 | $ island Torgersen, Torgersen, Torgersen,... 55 | $ bill_length_mm 39.1, 39.5, 40.3, NA, 36.7, 39.3,... 56 | $ bill_depth_mm 18.7, 17.4, 18.0, NA, 19.3, 20.6,... 57 | $ flipper_length_mm 181, 186, 195, NA, 193, 190, 181,... 58 | $ body_mass_g 3750, 3800, 3250, NA, 3450, 3650,... 59 | $ sex male, female, female, NA, female,... 60 | $ year 2007, 2007, 2007, 2007, 2007,... 61 | 62 | In predicting `species`, we want to make use of just a subset of columns: `bill_length_mm`, `bill_depth_mm`, `flipper_length_mm`, and `body_mass_g`. We build a `dataset()` that returns exactly what is needed: 63 | 64 | ```{r} 65 | penguins_dataset <- dataset( 66 | name = "penguins_dataset()", 67 | initialize = function(df) { 68 | df <- na.omit(df) 69 | self$x <- as.matrix(df[, 3:6]) %>% torch_tensor() 70 | self$y <- torch_tensor( 71 | as.numeric(df$species) 72 | )$to(torch_long()) 73 | }, 74 | .getitem = function(i) { 75 | list(x = self$x[i, ], y = self$y[i]) 76 | }, 77 | .length = function() { 78 | dim(self$x)[1] 79 | } 80 | ) 81 | ``` 82 | 83 | Once we've instantiated a `penguins_dataset()`, we should immediately perform some checks. First, does it have the expected length? 84 | 85 | ```{r} 86 | ds <- penguins_dataset(penguins) 87 | length(ds) 88 | ``` 89 | 90 | [1] 333 91 | 92 | And second, do individual elements have the expected shape and data type? Conveniently, we can access `dataset()` items like tensor values, through indexing: 93 | 94 | ```{r} 95 | ds[1] 96 | 97 | ``` 98 | 99 | $x 100 | torch_tensor 101 | 39.1000 102 | 18.7000 103 | 181.0000 104 | 3750.0000 105 | [ CPUFloatType{4} ] 106 | 107 | $y 108 | torch_tensor 109 | 1 110 | [ CPULongType{} ] 111 | 112 | This also works for items "further down" in the `dataset()` -- it has to: When indexing into a `dataset()`, what happens in the background is a call to `.getitem(i)`, passing along the desired position `i`. 113 | 114 | Truth be told, in this case we didn't really have to build our own `dataset()`. With so little pre-processing to be done, there is an alternative: `tensor_dataset()`. 115 | 116 | ### `tensor_dataset()` 117 | 118 | When you already have a tensor around, or something that's readily converted to one, you can make use of a built-in `dataset()` generator: `tensor_dataset()`. This function can be passed any number of tensors; each batch item then is a list of tensor values: 119 | 120 | ```{r} 121 | three <- tensor_dataset( 122 | torch_randn(10), torch_randn(10), torch_randn(10) 123 | ) 124 | three[1] 125 | ``` 126 | 127 | [[1]] 128 | torch_tensor 129 | 0.522735 130 | [ CPUFloatType{} ] 131 | 132 | [[2]] 133 | torch_tensor 134 | -0.976477 135 | [ CPUFloatType{} ] 136 | 137 | [[3]] 138 | torch_tensor 139 | -1.14685 140 | [ CPUFloatType{} ] 141 | 142 | In our `penguins` scenario, we end up with two lines of code: 143 | 144 | ```{r} 145 | penguins <- na.omit(penguins) 146 | ds <- tensor_dataset( 147 | torch_tensor(as.matrix(penguins[, 3:6])), 148 | torch_tensor( 149 | as.numeric(penguins$species) 150 | )$to(torch_long()) 151 | ) 152 | 153 | ds[1] 154 | ``` 155 | 156 | Admittedly though, we have not made use of all the dataset's columns. The more pre-processing you need a `dataset()` to do, the more likely you are to want to code your own. 157 | 158 | Thirdly and finally, here is the most effortless possible way. 159 | 160 | ### `torchvision::mnist_dataset()` 161 | 162 | When you're working with packages in the `torch` ecosystem, chances are that they already include some `dataset()`s, be it for demonstration purposes or for the sake of the data themselves. `torchvision`, for example, packages a number of classic image datasets -- among those, that archetype of archetypes, MNIST. 163 | 164 | Since we're going to talk about image processing in a later chapter, I won't comment on the arguments to `mnist_dataset()` here; we do, however, include a quick check that the data delivered conform to what we'd expect: 165 | 166 | ```{r} 167 | library(torchvision) 168 | 169 | dir <- "~/.torch-datasets" 170 | 171 | ds <- mnist_dataset( 172 | root = dir, 173 | train = TRUE, # default 174 | download = TRUE, 175 | transform = function(x) { 176 | x %>% transform_to_tensor() 177 | } 178 | ) 179 | 180 | first <- ds[1] 181 | cat("Image shape: ", first$x$shape, " Label: ", first$y, "\n") 182 | ``` 183 | 184 | Image shape: 1 28 28 Label: 6 185 | 186 | At this point, that is all we need to know about `dataset()`s -- we'll encounter plenty of them in the course of this book. Now, we move on from the one to the many. 187 | 188 | ## Using `dataloader()`s 189 | 190 | Continuing to work with the newly created MNIST `dataset()`, we instantiate a `dataloader()` for it. The `dataloader()` will deliver pairs of images and labels in batches: thirty-two at a time. In every epoch, it will return them in different order (`shuffle = TRUE`): 191 | 192 | ```{r} 193 | dl <- dataloader(ds, batch_size = 32, shuffle = TRUE) 194 | ``` 195 | 196 | Just like `dataset()`s, `dataloader()`s can be queried about their length: 197 | 198 | ```{r} 199 | length(dl) 200 | ``` 201 | 202 | [1] 1875 203 | 204 | This time, though, the returned value is not the number of items; it is the number of batches. 205 | 206 | To loop over batches, we first obtain an iterator, an object that knows how to traverse the elements in this `dataloader()`. Calling `dataloader_next()`, we can then access successive batches, one by one: 207 | 208 | ```{r} 209 | first_batch <- dl %>% 210 | # obtain an iterator for this dataloader 211 | dataloader_make_iter() %>% 212 | dataloader_next() 213 | 214 | dim(first_batch$x) 215 | dim(first_batch$y) 216 | ``` 217 | 218 | [1] 32 1 28 28 219 | [1] 32 220 | 221 | If you compare the batch shape of `x` -- the image part -- with the shape of an individual image (as inspected above), you see that now, there is an additional dimension in front, reflecting the number of images in a batch. 222 | 223 | The next step is passing the batches to a model. This -- in fact, this as well as the complete, end-to-end deep-learning workflow -- is what the next chapter is about. 224 | -------------------------------------------------------------------------------- /dl_overview.qmd: -------------------------------------------------------------------------------- 1 | # Overview {#sec:dl-overview} 2 | 3 | This part of the book is completely dedicated to applications of deep learning. There will be two categories of things to dive into: topics workflow-related, and topics related to domain adaptation. 4 | 5 | Regarding workflow, we'll see how to: 6 | 7 | - prepare the input data in a form the model can work with; 8 | - effectively and efficiently train a model, monitoring progress and adjusting hyper-parameters on the fly; 9 | - save and load models; 10 | - making models generalize beyond the training data; 11 | - speed up training; 12 | - and more. 13 | 14 | Secondly -- beyond an efficient workflow -- the task in question matters. Compositions of linear layers, of the type we used to learn `torch` in the first part, will not suffice when our goal is to model images or time series. Successful use of deep learning means tailoring model architecture to the domain in question. To that end, we start from concrete tasks, and present applicable architectures directly by example. 15 | 16 | Concretely, the plan is the following. The upcoming two chapters will introduce you to workflow-related techniques that are indispensable in practice. You'll encounter another package, `luz`, that endows `torch` with an important layer of abstraction, and significantly streamlines the workflow. Once you know how to use it, we're all set to look at a first application: image classification. To improve on our initial results, we then back up and explore two more advanced workflow-related topics: how to improve generalization, and how to speed up training. Equipped with that knowledge, we first return to images, before extending our domain-related skills to tabular data, time series, and audio. 17 | -------------------------------------------------------------------------------- /docs/.nojekyll: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/.nojekyll -------------------------------------------------------------------------------- /docs/images/audio-alluvial-baseline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-alluvial-baseline.png -------------------------------------------------------------------------------- /docs/images/audio-alluvial-complex.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-alluvial-complex.png -------------------------------------------------------------------------------- /docs/images/audio-alluvial-mel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-alluvial-mel.png -------------------------------------------------------------------------------- /docs/images/audio-bird-dft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-bird-dft.png -------------------------------------------------------------------------------- /docs/images/audio-bird-waveform.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-bird-waveform.png -------------------------------------------------------------------------------- /docs/images/audio-fit-baseline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-fit-baseline.png -------------------------------------------------------------------------------- /docs/images/audio-fit-complex.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-fit-complex.png -------------------------------------------------------------------------------- /docs/images/audio-fit-mel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-fit-mel.png -------------------------------------------------------------------------------- /docs/images/audio-lr-finder-baseline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-lr-finder-baseline.png -------------------------------------------------------------------------------- /docs/images/audio-lr-finder-complex.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-lr-finder-complex.png -------------------------------------------------------------------------------- /docs/images/audio-lr-finder-mel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-lr-finder-mel.png -------------------------------------------------------------------------------- /docs/images/audio-mel-filterbank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-mel-filterbank.png -------------------------------------------------------------------------------- /docs/images/audio-mel-spectrogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-mel-spectrogram.png -------------------------------------------------------------------------------- /docs/images/audio-spectrogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/audio-spectrogram.png -------------------------------------------------------------------------------- /docs/images/autograd-compgraph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/autograd-compgraph.png -------------------------------------------------------------------------------- /docs/images/autograd-paraboloid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/autograd-paraboloid.png -------------------------------------------------------------------------------- /docs/images/dft-cos-1-rev-dft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-cos-1-rev-dft.png -------------------------------------------------------------------------------- /docs/images/dft-cos-1-rev.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-cos-1-rev.png -------------------------------------------------------------------------------- /docs/images/dft-cos-32-rev.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-cos-32-rev.png -------------------------------------------------------------------------------- /docs/images/dft-cos-4-rev.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-cos-4-rev.png -------------------------------------------------------------------------------- /docs/images/dft-cos-mult-amplitude.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-cos-mult-amplitude.png -------------------------------------------------------------------------------- /docs/images/dft-cos-phase-pi12.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-cos-phase-pi12.png -------------------------------------------------------------------------------- /docs/images/dft-dial-dft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-dial-dft.png -------------------------------------------------------------------------------- /docs/images/dft-dial-fft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-dial-fft.png -------------------------------------------------------------------------------- /docs/images/dft-dial-ifft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-dial-ifft.png -------------------------------------------------------------------------------- /docs/images/dft-dial-spectrogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-dial-spectrogram.png -------------------------------------------------------------------------------- /docs/images/dft-dial-waveform.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-dial-waveform.png -------------------------------------------------------------------------------- /docs/images/dft-mix-of-sinuisoids.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/dft-mix-of-sinuisoids.png -------------------------------------------------------------------------------- /docs/images/efficiency-mnist-lr-finder.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/efficiency-mnist-lr-finder.png -------------------------------------------------------------------------------- /docs/images/fft-perf-jit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/fft-perf-jit.png -------------------------------------------------------------------------------- /docs/images/fft-perf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/fft-perf.png -------------------------------------------------------------------------------- /docs/images/images-conv-arithmetic-dilation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images-conv-arithmetic-dilation.png -------------------------------------------------------------------------------- /docs/images/images-conv-arithmetic-padding.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images-conv-arithmetic-padding.png -------------------------------------------------------------------------------- /docs/images/images-conv-arithmetic-strides.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images-conv-arithmetic-strides.png -------------------------------------------------------------------------------- /docs/images/images-cross-correlation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images-cross-correlation.png -------------------------------------------------------------------------------- /docs/images/images-feature-visualization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images-feature-visualization.png -------------------------------------------------------------------------------- /docs/images/images-square-bottom.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images-square-bottom.png -------------------------------------------------------------------------------- /docs/images/images-square-left.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images-square-left.png -------------------------------------------------------------------------------- /docs/images/images-square-right.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images-square-right.png -------------------------------------------------------------------------------- /docs/images/images-square-top.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images-square-top.png -------------------------------------------------------------------------------- /docs/images/images-square.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images-square.png -------------------------------------------------------------------------------- /docs/images/images2-lr-finder-batchnorm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images2-lr-finder-batchnorm.png -------------------------------------------------------------------------------- /docs/images/images2-lr-finder-dropout.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images2-lr-finder-dropout.png -------------------------------------------------------------------------------- /docs/images/images2-lr-finder-resnet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/images2-lr-finder-resnet.png -------------------------------------------------------------------------------- /docs/images/least-squares-benchmark.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/least-squares-benchmark.png -------------------------------------------------------------------------------- /docs/images/optim-1-rosenbrock.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/optim-1-rosenbrock.png -------------------------------------------------------------------------------- /docs/images/optimizers-adagrad.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/optimizers-adagrad.png -------------------------------------------------------------------------------- /docs/images/optimizers-adam.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/optimizers-adam.png -------------------------------------------------------------------------------- /docs/images/optimizers-momentum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/optimizers-momentum.png -------------------------------------------------------------------------------- /docs/images/optimizers-rmsprop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/optimizers-rmsprop.png -------------------------------------------------------------------------------- /docs/images/optimizers-steepest-descent-elliptic.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/optimizers-steepest-descent-elliptic.png -------------------------------------------------------------------------------- /docs/images/optimizers-steepest-descent-symmetric.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/optimizers-steepest-descent-symmetric.png -------------------------------------------------------------------------------- /docs/images/overfitting-mnist-mixup-0.5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/overfitting-mnist-mixup-0.5.png -------------------------------------------------------------------------------- /docs/images/overfitting-mnist-mixup-0.7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/overfitting-mnist-mixup-0.7.png -------------------------------------------------------------------------------- /docs/images/overfitting-mnist-mixup-0.9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/overfitting-mnist-mixup-0.9.png -------------------------------------------------------------------------------- /docs/images/overfitting-mnist-test.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/overfitting-mnist-test.png -------------------------------------------------------------------------------- /docs/images/overfitting-mnist-transforms.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/overfitting-mnist-transforms.png -------------------------------------------------------------------------------- /docs/images/segmentation-conv-arithmetic-transposed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/segmentation-conv-arithmetic-transposed.png -------------------------------------------------------------------------------- /docs/images/segmentation-lr-finder.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/segmentation-lr-finder.png -------------------------------------------------------------------------------- /docs/images/segmentation-segmentation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/segmentation-segmentation.png -------------------------------------------------------------------------------- /docs/images/segmentation-unet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/segmentation-unet.png -------------------------------------------------------------------------------- /docs/images/tabular-heart-disease-biplot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/tabular-heart-disease-biplot.png -------------------------------------------------------------------------------- /docs/images/tabular-heart-disease-fit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/tabular-heart-disease-fit.png -------------------------------------------------------------------------------- /docs/images/tabular-heart-disease-pca.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/tabular-heart-disease-pca.png -------------------------------------------------------------------------------- /docs/images/tensors-dimensions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/tensors-dimensions.png -------------------------------------------------------------------------------- /docs/images/timeseries-vic-elec-fit-mlp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/timeseries-vic-elec-fit-mlp.png -------------------------------------------------------------------------------- /docs/images/timeseries-vic-elec-fit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/timeseries-vic-elec-fit.png -------------------------------------------------------------------------------- /docs/images/timeseries-vic-elec-lr-finder-mlp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/timeseries-vic-elec-lr-finder-mlp.png -------------------------------------------------------------------------------- /docs/images/timeseries-vic-elec-lr-finder.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/timeseries-vic-elec-lr-finder.png -------------------------------------------------------------------------------- /docs/images/timeseries-vic-elec-preds-mlp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/timeseries-vic-elec-preds-mlp.png -------------------------------------------------------------------------------- /docs/images/timeseries-vic-elec-preds.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/timeseries-vic-elec-preds.png -------------------------------------------------------------------------------- /docs/images/timeseries-vic-elec-stl-month.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/timeseries-vic-elec-stl-month.png -------------------------------------------------------------------------------- /docs/images/timeseries-vic-elec-stl.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/timeseries-vic-elec-stl.png -------------------------------------------------------------------------------- /docs/images/wav-K-omega.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-K-omega.png -------------------------------------------------------------------------------- /docs/images/wav-chaffinch-fft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-chaffinch-fft.png -------------------------------------------------------------------------------- /docs/images/wav-chaffinch-spectrogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-chaffinch-spectrogram.png -------------------------------------------------------------------------------- /docs/images/wav-chaffinch-waveletdiag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-chaffinch-waveletdiag.png -------------------------------------------------------------------------------- /docs/images/wav-chaffinch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-chaffinch.png -------------------------------------------------------------------------------- /docs/images/wav-example-long-diag1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-example-long-diag1.png -------------------------------------------------------------------------------- /docs/images/wav-example-long-diag2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-example-long-diag2.png -------------------------------------------------------------------------------- /docs/images/wav-example-long-diag3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-example-long-diag3.png -------------------------------------------------------------------------------- /docs/images/wav-example-long-diag4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-example-long-diag4.png -------------------------------------------------------------------------------- /docs/images/wav-example-long-diag5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-example-long-diag5.png -------------------------------------------------------------------------------- /docs/images/wav-example-long-diag6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-example-long-diag6.png -------------------------------------------------------------------------------- /docs/images/wav-example-long-transform.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-example-long-transform.png -------------------------------------------------------------------------------- /docs/images/wav-example-long-transform2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-example-long-transform2.png -------------------------------------------------------------------------------- /docs/images/wav-morlet-example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-morlet-example.png -------------------------------------------------------------------------------- /docs/images/wav-signal-long.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-signal-long.png -------------------------------------------------------------------------------- /docs/images/wav-signal-short-transform-largeK.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-signal-short-transform-largeK.png -------------------------------------------------------------------------------- /docs/images/wav-signal-short-transform.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-signal-short-transform.png -------------------------------------------------------------------------------- /docs/images/wav-signal-short.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/images/wav-signal-short.png -------------------------------------------------------------------------------- /docs/site_libs/bootstrap/bootstrap-icons.woff: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/docs/site_libs/bootstrap/bootstrap-icons.woff -------------------------------------------------------------------------------- /docs/site_libs/clipboard/clipboard.min.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * clipboard.js v2.0.11 3 | * https://clipboardjs.com/ 4 | * 5 | * Licensed MIT © Zeno Rocha 6 | */ 7 | !function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return n={686:function(t,e,n){"use strict";n.d(e,{default:function(){return b}});var e=n(279),i=n.n(e),e=n(370),u=n.n(e),e=n(817),r=n.n(e);function c(t){try{return document.execCommand(t)}catch(t){return}}var a=function(t){t=r()(t);return c("cut"),t};function o(t,e){var n,o,t=(n=t,o="rtl"===document.documentElement.getAttribute("dir"),(t=document.createElement("textarea")).style.fontSize="12pt",t.style.border="0",t.style.padding="0",t.style.margin="0",t.style.position="absolute",t.style[o?"right":"left"]="-9999px",o=window.pageYOffset||document.documentElement.scrollTop,t.style.top="".concat(o,"px"),t.setAttribute("readonly",""),t.value=n,t);return e.container.appendChild(t),e=r()(t),c("copy"),t.remove(),e}var f=function(t){var e=1.anchorjs-link,.anchorjs-link:focus{opacity:1}",u.sheet.cssRules.length),u.sheet.insertRule("[data-anchorjs-icon]::after{content:attr(data-anchorjs-icon)}",u.sheet.cssRules.length),u.sheet.insertRule('@font-face{font-family:anchorjs-icons;src:url(data:n/a;base64,AAEAAAALAIAAAwAwT1MvMg8yG2cAAAE4AAAAYGNtYXDp3gC3AAABpAAAAExnYXNwAAAAEAAAA9wAAAAIZ2x5ZlQCcfwAAAH4AAABCGhlYWQHFvHyAAAAvAAAADZoaGVhBnACFwAAAPQAAAAkaG10eASAADEAAAGYAAAADGxvY2EACACEAAAB8AAAAAhtYXhwAAYAVwAAARgAAAAgbmFtZQGOH9cAAAMAAAAAunBvc3QAAwAAAAADvAAAACAAAQAAAAEAAHzE2p9fDzz1AAkEAAAAAADRecUWAAAAANQA6R8AAAAAAoACwAAAAAgAAgAAAAAAAAABAAADwP/AAAACgAAA/9MCrQABAAAAAAAAAAAAAAAAAAAAAwABAAAAAwBVAAIAAAAAAAIAAAAAAAAAAAAAAAAAAAAAAAMCQAGQAAUAAAKZAswAAACPApkCzAAAAesAMwEJAAAAAAAAAAAAAAAAAAAAARAAAAAAAAAAAAAAAAAAAAAAQAAg//0DwP/AAEADwABAAAAAAQAAAAAAAAAAAAAAIAAAAAAAAAIAAAACgAAxAAAAAwAAAAMAAAAcAAEAAwAAABwAAwABAAAAHAAEADAAAAAIAAgAAgAAACDpy//9//8AAAAg6cv//f///+EWNwADAAEAAAAAAAAAAAAAAAAACACEAAEAAAAAAAAAAAAAAAAxAAACAAQARAKAAsAAKwBUAAABIiYnJjQ3NzY2MzIWFxYUBwcGIicmNDc3NjQnJiYjIgYHBwYUFxYUBwYGIwciJicmNDc3NjIXFhQHBwYUFxYWMzI2Nzc2NCcmNDc2MhcWFAcHBgYjARQGDAUtLXoWOR8fORYtLTgKGwoKCjgaGg0gEhIgDXoaGgkJBQwHdR85Fi0tOAobCgoKOBoaDSASEiANehoaCQkKGwotLXoWOR8BMwUFLYEuehYXFxYugC44CQkKGwo4GkoaDQ0NDXoaShoKGwoFBe8XFi6ALjgJCQobCjgaShoNDQ0NehpKGgobCgoKLYEuehYXAAAADACWAAEAAAAAAAEACAAAAAEAAAAAAAIAAwAIAAEAAAAAAAMACAAAAAEAAAAAAAQACAAAAAEAAAAAAAUAAQALAAEAAAAAAAYACAAAAAMAAQQJAAEAEAAMAAMAAQQJAAIABgAcAAMAAQQJAAMAEAAMAAMAAQQJAAQAEAAMAAMAAQQJAAUAAgAiAAMAAQQJAAYAEAAMYW5jaG9yanM0MDBAAGEAbgBjAGgAbwByAGoAcwA0ADAAMABAAAAAAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAH//wAP) format("truetype")}',u.sheet.cssRules.length)),u=document.querySelectorAll("[id]"),t=[].map.call(u,function(A){return A.id}),i=0;i\]./()*\\\n\t\b\v\u00A0]/g,"-").replace(/-{2,}/g,"-").substring(0,this.options.truncate).replace(/^-+|-+$/gm,"").toLowerCase()},this.hasAnchorJSLink=function(A){var e=A.firstChild&&-1<(" "+A.firstChild.className+" ").indexOf(" anchorjs-link "),A=A.lastChild&&-1<(" "+A.lastChild.className+" ").indexOf(" anchorjs-link ");return e||A||!1}}}); 9 | // @license-end -------------------------------------------------------------------------------- /docs/site_libs/quarto-html/quarto-syntax-highlighting.css: -------------------------------------------------------------------------------- 1 | /* quarto syntax highlight colors */ 2 | :root { 3 | --quarto-hl-ot-color: #003B4F; 4 | --quarto-hl-at-color: #657422; 5 | --quarto-hl-ss-color: #20794D; 6 | --quarto-hl-an-color: #5E5E5E; 7 | --quarto-hl-fu-color: #4758AB; 8 | --quarto-hl-st-color: #20794D; 9 | --quarto-hl-cf-color: #003B4F; 10 | --quarto-hl-op-color: #5E5E5E; 11 | --quarto-hl-er-color: #AD0000; 12 | --quarto-hl-bn-color: #AD0000; 13 | --quarto-hl-al-color: #AD0000; 14 | --quarto-hl-va-color: #111111; 15 | --quarto-hl-bu-color: inherit; 16 | --quarto-hl-ex-color: inherit; 17 | --quarto-hl-pp-color: #AD0000; 18 | --quarto-hl-in-color: #5E5E5E; 19 | --quarto-hl-vs-color: #20794D; 20 | --quarto-hl-wa-color: #5E5E5E; 21 | --quarto-hl-do-color: #5E5E5E; 22 | --quarto-hl-im-color: #00769E; 23 | --quarto-hl-ch-color: #20794D; 24 | --quarto-hl-dt-color: #AD0000; 25 | --quarto-hl-fl-color: #AD0000; 26 | --quarto-hl-co-color: #5E5E5E; 27 | --quarto-hl-cv-color: #5E5E5E; 28 | --quarto-hl-cn-color: #8f5902; 29 | --quarto-hl-sc-color: #5E5E5E; 30 | --quarto-hl-dv-color: #AD0000; 31 | --quarto-hl-kw-color: #003B4F; 32 | } 33 | 34 | /* other quarto variables */ 35 | :root { 36 | --quarto-font-monospace: SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace; 37 | } 38 | 39 | pre > code.sourceCode > span { 40 | color: #003B4F; 41 | } 42 | 43 | code span { 44 | color: #003B4F; 45 | } 46 | 47 | code.sourceCode > span { 48 | color: #003B4F; 49 | } 50 | 51 | div.sourceCode, 52 | div.sourceCode pre.sourceCode { 53 | color: #003B4F; 54 | } 55 | 56 | code span.ot { 57 | color: #003B4F; 58 | font-style: inherit; 59 | } 60 | 61 | code span.at { 62 | color: #657422; 63 | font-style: inherit; 64 | } 65 | 66 | code span.ss { 67 | color: #20794D; 68 | font-style: inherit; 69 | } 70 | 71 | code span.an { 72 | color: #5E5E5E; 73 | font-style: inherit; 74 | } 75 | 76 | code span.fu { 77 | color: #4758AB; 78 | font-style: inherit; 79 | } 80 | 81 | code span.st { 82 | color: #20794D; 83 | font-style: inherit; 84 | } 85 | 86 | code span.cf { 87 | color: #003B4F; 88 | font-style: inherit; 89 | } 90 | 91 | code span.op { 92 | color: #5E5E5E; 93 | font-style: inherit; 94 | } 95 | 96 | code span.er { 97 | color: #AD0000; 98 | font-style: inherit; 99 | } 100 | 101 | code span.bn { 102 | color: #AD0000; 103 | font-style: inherit; 104 | } 105 | 106 | code span.al { 107 | color: #AD0000; 108 | font-style: inherit; 109 | } 110 | 111 | code span.va { 112 | color: #111111; 113 | font-style: inherit; 114 | } 115 | 116 | code span.bu { 117 | font-style: inherit; 118 | } 119 | 120 | code span.ex { 121 | font-style: inherit; 122 | } 123 | 124 | code span.pp { 125 | color: #AD0000; 126 | font-style: inherit; 127 | } 128 | 129 | code span.in { 130 | color: #5E5E5E; 131 | font-style: inherit; 132 | } 133 | 134 | code span.vs { 135 | color: #20794D; 136 | font-style: inherit; 137 | } 138 | 139 | code span.wa { 140 | color: #5E5E5E; 141 | font-style: italic; 142 | } 143 | 144 | code span.do { 145 | color: #5E5E5E; 146 | font-style: italic; 147 | } 148 | 149 | code span.im { 150 | color: #00769E; 151 | font-style: inherit; 152 | } 153 | 154 | code span.ch { 155 | color: #20794D; 156 | font-style: inherit; 157 | } 158 | 159 | code span.dt { 160 | color: #AD0000; 161 | font-style: inherit; 162 | } 163 | 164 | code span.fl { 165 | color: #AD0000; 166 | font-style: inherit; 167 | } 168 | 169 | code span.co { 170 | color: #5E5E5E; 171 | font-style: inherit; 172 | } 173 | 174 | code span.cv { 175 | color: #5E5E5E; 176 | font-style: italic; 177 | } 178 | 179 | code span.cn { 180 | color: #8f5902; 181 | font-style: inherit; 182 | } 183 | 184 | code span.sc { 185 | color: #5E5E5E; 186 | font-style: inherit; 187 | } 188 | 189 | code span.dv { 190 | color: #AD0000; 191 | font-style: inherit; 192 | } 193 | 194 | code span.kw { 195 | color: #003B4F; 196 | font-style: inherit; 197 | } 198 | 199 | .prevent-inlining { 200 | content: ".tippy-arrow{bottom:0}.tippy-box[data-placement^=top]>.tippy-arrow:before{bottom:-7px;left:0;border-width:8px 8px 0;border-top-color:initial;transform-origin:center top}.tippy-box[data-placement^=bottom]>.tippy-arrow{top:0}.tippy-box[data-placement^=bottom]>.tippy-arrow:before{top:-7px;left:0;border-width:0 8px 8px;border-bottom-color:initial;transform-origin:center bottom}.tippy-box[data-placement^=left]>.tippy-arrow{right:0}.tippy-box[data-placement^=left]>.tippy-arrow:before{border-width:8px 0 8px 8px;border-left-color:initial;right:-7px;transform-origin:center left}.tippy-box[data-placement^=right]>.tippy-arrow{left:0}.tippy-box[data-placement^=right]>.tippy-arrow:before{left:-7px;border-width:8px 8px 8px 0;border-right-color:initial;transform-origin:center right}.tippy-box[data-inertia][data-state=visible]{transition-timing-function:cubic-bezier(.54,1.5,.38,1.11)}.tippy-arrow{width:16px;height:16px;color:#333}.tippy-arrow:before{content:"";position:absolute;border-color:transparent;border-style:solid}.tippy-content{position:relative;padding:5px 9px;z-index:1} -------------------------------------------------------------------------------- /docs/site_libs/quarto-nav/headroom.min.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * headroom.js v0.12.0 - Give your page some headroom. Hide your header until you need it 3 | * Copyright (c) 2020 Nick Williams - http://wicky.nillia.ms/headroom.js 4 | * License: MIT 5 | */ 6 | 7 | !function(t,n){"object"==typeof exports&&"undefined"!=typeof module?module.exports=n():"function"==typeof define&&define.amd?define(n):(t=t||self).Headroom=n()}(this,function(){"use strict";function t(){return"undefined"!=typeof window}function d(t){return function(t){return t&&t.document&&function(t){return 9===t.nodeType}(t.document)}(t)?function(t){var n=t.document,o=n.body,s=n.documentElement;return{scrollHeight:function(){return Math.max(o.scrollHeight,s.scrollHeight,o.offsetHeight,s.offsetHeight,o.clientHeight,s.clientHeight)},height:function(){return t.innerHeight||s.clientHeight||o.clientHeight},scrollY:function(){return void 0!==t.pageYOffset?t.pageYOffset:(s||o.parentNode||o).scrollTop}}}(t):function(t){return{scrollHeight:function(){return Math.max(t.scrollHeight,t.offsetHeight,t.clientHeight)},height:function(){return Math.max(t.offsetHeight,t.clientHeight)},scrollY:function(){return t.scrollTop}}}(t)}function n(t,s,e){var n,o=function(){var n=!1;try{var t={get passive(){n=!0}};window.addEventListener("test",t,t),window.removeEventListener("test",t,t)}catch(t){n=!1}return n}(),i=!1,r=d(t),l=r.scrollY(),a={};function c(){var t=Math.round(r.scrollY()),n=r.height(),o=r.scrollHeight();a.scrollY=t,a.lastScrollY=l,a.direction=ls.tolerance[a.direction],e(a),l=t,i=!1}function h(){i||(i=!0,n=requestAnimationFrame(c))}var u=!!o&&{passive:!0,capture:!1};return t.addEventListener("scroll",h,u),c(),{destroy:function(){cancelAnimationFrame(n),t.removeEventListener("scroll",h,u)}}}function o(t){return t===Object(t)?t:{down:t,up:t}}function s(t,n){n=n||{},Object.assign(this,s.options,n),this.classes=Object.assign({},s.options.classes,n.classes),this.elem=t,this.tolerance=o(this.tolerance),this.offset=o(this.offset),this.initialised=!1,this.frozen=!1}return s.prototype={constructor:s,init:function(){return s.cutsTheMustard&&!this.initialised&&(this.addClass("initial"),this.initialised=!0,setTimeout(function(t){t.scrollTracker=n(t.scroller,{offset:t.offset,tolerance:t.tolerance},t.update.bind(t))},100,this)),this},destroy:function(){this.initialised=!1,Object.keys(this.classes).forEach(this.removeClass,this),this.scrollTracker.destroy()},unpin:function(){!this.hasClass("pinned")&&this.hasClass("unpinned")||(this.addClass("unpinned"),this.removeClass("pinned"),this.onUnpin&&this.onUnpin.call(this))},pin:function(){this.hasClass("unpinned")&&(this.addClass("pinned"),this.removeClass("unpinned"),this.onPin&&this.onPin.call(this))},freeze:function(){this.frozen=!0,this.addClass("frozen")},unfreeze:function(){this.frozen=!1,this.removeClass("frozen")},top:function(){this.hasClass("top")||(this.addClass("top"),this.removeClass("notTop"),this.onTop&&this.onTop.call(this))},notTop:function(){this.hasClass("notTop")||(this.addClass("notTop"),this.removeClass("top"),this.onNotTop&&this.onNotTop.call(this))},bottom:function(){this.hasClass("bottom")||(this.addClass("bottom"),this.removeClass("notBottom"),this.onBottom&&this.onBottom.call(this))},notBottom:function(){this.hasClass("notBottom")||(this.addClass("notBottom"),this.removeClass("bottom"),this.onNotBottom&&this.onNotBottom.call(this))},shouldUnpin:function(t){return"down"===t.direction&&!t.top&&t.toleranceExceeded},shouldPin:function(t){return"up"===t.direction&&t.toleranceExceeded||t.top},addClass:function(t){this.elem.classList.add.apply(this.elem.classList,this.classes[t].split(" "))},removeClass:function(t){this.elem.classList.remove.apply(this.elem.classList,this.classes[t].split(" "))},hasClass:function(t){return this.classes[t].split(" ").every(function(t){return this.classList.contains(t)},this.elem)},update:function(t){t.isOutOfBounds||!0!==this.frozen&&(t.top?this.top():this.notTop(),t.bottom?this.bottom():this.notBottom(),this.shouldUnpin(t)?this.unpin():this.shouldPin(t)&&this.pin())}},s.options={tolerance:{up:0,down:0},offset:0,scroller:t()?window:null,classes:{frozen:"headroom--frozen",pinned:"headroom--pinned",unpinned:"headroom--unpinned",top:"headroom--top",notTop:"headroom--not-top",bottom:"headroom--bottom",notBottom:"headroom--not-bottom",initial:"headroom"}},s.cutsTheMustard=!!(t()&&function(){}.bind&&"classList"in document.documentElement&&Object.assign&&Object.keys&&requestAnimationFrame),s}); 8 | -------------------------------------------------------------------------------- /docs/site_libs/quarto-nav/quarto-nav.js: -------------------------------------------------------------------------------- 1 | const headroomChanged = new CustomEvent("quarto-hrChanged", { 2 | detail: {}, 3 | bubbles: true, 4 | cancelable: false, 5 | composed: false, 6 | }); 7 | 8 | window.document.addEventListener("DOMContentLoaded", function () { 9 | let init = false; 10 | 11 | // Manage the back to top button, if one is present. 12 | let lastScrollTop = window.pageYOffset || document.documentElement.scrollTop; 13 | const scrollDownBuffer = 5; 14 | const scrollUpBuffer = 35; 15 | const btn = document.getElementById("quarto-back-to-top"); 16 | const hideBackToTop = () => { 17 | btn.style.display = "none"; 18 | }; 19 | const showBackToTop = () => { 20 | btn.style.display = "inline-block"; 21 | }; 22 | if (btn) { 23 | window.document.addEventListener( 24 | "scroll", 25 | function () { 26 | const currentScrollTop = 27 | window.pageYOffset || document.documentElement.scrollTop; 28 | 29 | // Shows and hides the button 'intelligently' as the user scrolls 30 | if (currentScrollTop - scrollDownBuffer > lastScrollTop) { 31 | hideBackToTop(); 32 | lastScrollTop = currentScrollTop <= 0 ? 0 : currentScrollTop; 33 | } else if (currentScrollTop < lastScrollTop - scrollUpBuffer) { 34 | showBackToTop(); 35 | lastScrollTop = currentScrollTop <= 0 ? 0 : currentScrollTop; 36 | } 37 | 38 | // Show the button at the bottom, hides it at the top 39 | if (currentScrollTop <= 0) { 40 | hideBackToTop(); 41 | } else if ( 42 | window.innerHeight + currentScrollTop >= 43 | document.body.offsetHeight 44 | ) { 45 | showBackToTop(); 46 | } 47 | }, 48 | false 49 | ); 50 | } 51 | 52 | function throttle(func, wait) { 53 | var timeout; 54 | return function () { 55 | const context = this; 56 | const args = arguments; 57 | const later = function () { 58 | clearTimeout(timeout); 59 | timeout = null; 60 | func.apply(context, args); 61 | }; 62 | 63 | if (!timeout) { 64 | timeout = setTimeout(later, wait); 65 | } 66 | }; 67 | } 68 | 69 | function headerOffset() { 70 | // Set an offset if there is are fixed top navbar 71 | const headerEl = window.document.querySelector("header.fixed-top"); 72 | if (headerEl) { 73 | return headerEl.clientHeight; 74 | } else { 75 | return 0; 76 | } 77 | } 78 | 79 | function footerOffset() { 80 | const footerEl = window.document.querySelector("footer.footer"); 81 | if (footerEl) { 82 | return footerEl.clientHeight; 83 | } else { 84 | return 0; 85 | } 86 | } 87 | 88 | function updateDocumentOffsetWithoutAnimation() { 89 | updateDocumentOffset(false); 90 | } 91 | 92 | function updateDocumentOffset(animated) { 93 | // set body offset 94 | const topOffset = headerOffset(); 95 | const bodyOffset = topOffset + footerOffset(); 96 | const bodyEl = window.document.body; 97 | bodyEl.setAttribute("data-bs-offset", topOffset); 98 | bodyEl.style.paddingTop = topOffset + "px"; 99 | 100 | // deal with sidebar offsets 101 | const sidebars = window.document.querySelectorAll( 102 | ".sidebar, .headroom-target" 103 | ); 104 | sidebars.forEach((sidebar) => { 105 | if (!animated) { 106 | sidebar.classList.add("notransition"); 107 | // Remove the no transition class after the animation has time to complete 108 | setTimeout(function () { 109 | sidebar.classList.remove("notransition"); 110 | }, 201); 111 | } 112 | 113 | if (window.Headroom && sidebar.classList.contains("sidebar-unpinned")) { 114 | sidebar.style.top = "0"; 115 | sidebar.style.maxHeight = "100vh"; 116 | } else { 117 | sidebar.style.top = topOffset + "px"; 118 | sidebar.style.maxHeight = "calc(100vh - " + topOffset + "px)"; 119 | } 120 | }); 121 | 122 | // allow space for footer 123 | const mainContainer = window.document.querySelector(".quarto-container"); 124 | if (mainContainer) { 125 | mainContainer.style.minHeight = "calc(100vh - " + bodyOffset + "px)"; 126 | } 127 | 128 | // link offset 129 | let linkStyle = window.document.querySelector("#quarto-target-style"); 130 | if (!linkStyle) { 131 | linkStyle = window.document.createElement("style"); 132 | linkStyle.setAttribute("id", "quarto-target-style"); 133 | window.document.head.appendChild(linkStyle); 134 | } 135 | while (linkStyle.firstChild) { 136 | linkStyle.removeChild(linkStyle.firstChild); 137 | } 138 | if (topOffset > 0) { 139 | linkStyle.appendChild( 140 | window.document.createTextNode(` 141 | section:target::before { 142 | content: ""; 143 | display: block; 144 | height: ${topOffset}px; 145 | margin: -${topOffset}px 0 0; 146 | }`) 147 | ); 148 | } 149 | if (init) { 150 | window.dispatchEvent(headroomChanged); 151 | } 152 | init = true; 153 | } 154 | 155 | // initialize headroom 156 | var header = window.document.querySelector("#quarto-header"); 157 | if (header && window.Headroom) { 158 | const headroom = new window.Headroom(header, { 159 | tolerance: 5, 160 | onPin: function () { 161 | const sidebars = window.document.querySelectorAll( 162 | ".sidebar, .headroom-target" 163 | ); 164 | sidebars.forEach((sidebar) => { 165 | sidebar.classList.remove("sidebar-unpinned"); 166 | }); 167 | updateDocumentOffset(); 168 | }, 169 | onUnpin: function () { 170 | const sidebars = window.document.querySelectorAll( 171 | ".sidebar, .headroom-target" 172 | ); 173 | sidebars.forEach((sidebar) => { 174 | sidebar.classList.add("sidebar-unpinned"); 175 | }); 176 | updateDocumentOffset(); 177 | }, 178 | }); 179 | headroom.init(); 180 | 181 | let frozen = false; 182 | window.quartoToggleHeadroom = function () { 183 | if (frozen) { 184 | headroom.unfreeze(); 185 | frozen = false; 186 | } else { 187 | headroom.freeze(); 188 | frozen = true; 189 | } 190 | }; 191 | } 192 | 193 | window.addEventListener( 194 | "hashchange", 195 | function (e) { 196 | window.scrollTo(0, window.pageYOffset - headerOffset()); 197 | }, 198 | false 199 | ); 200 | 201 | // Observe size changed for the header 202 | const headerEl = window.document.querySelector("header.fixed-top"); 203 | if (headerEl && window.ResizeObserver) { 204 | const observer = new window.ResizeObserver( 205 | updateDocumentOffsetWithoutAnimation 206 | ); 207 | observer.observe(headerEl, { 208 | attributes: true, 209 | childList: true, 210 | characterData: true, 211 | }); 212 | } else { 213 | window.addEventListener( 214 | "resize", 215 | throttle(updateDocumentOffsetWithoutAnimation, 50) 216 | ); 217 | } 218 | setTimeout(updateDocumentOffsetWithoutAnimation, 250); 219 | 220 | // fixup index.html links if we aren't on the filesystem 221 | if (window.location.protocol !== "file:") { 222 | const links = window.document.querySelectorAll("a"); 223 | for (let i = 0; i < links.length; i++) { 224 | if (links[i].href) { 225 | links[i].href = links[i].href.replace(/\/index\.html/, "/"); 226 | } 227 | } 228 | 229 | // Fixup any sharing links that require urls 230 | // Append url to any sharing urls 231 | const sharingLinks = window.document.querySelectorAll( 232 | "a.sidebar-tools-main-item" 233 | ); 234 | for (let i = 0; i < sharingLinks.length; i++) { 235 | const sharingLink = sharingLinks[i]; 236 | const href = sharingLink.getAttribute("href"); 237 | if (href) { 238 | sharingLink.setAttribute( 239 | "href", 240 | href.replace("|url|", window.location.href) 241 | ); 242 | } 243 | } 244 | 245 | // Scroll the active navigation item into view, if necessary 246 | const navSidebar = window.document.querySelector("nav#quarto-sidebar"); 247 | if (navSidebar) { 248 | // Find the active item 249 | const activeItem = navSidebar.querySelector("li.sidebar-item a.active"); 250 | if (activeItem) { 251 | // Wait for the scroll height and height to resolve by observing size changes on the 252 | // nav element that is scrollable 253 | const resizeObserver = new ResizeObserver((_entries) => { 254 | // The bottom of the element 255 | const elBottom = activeItem.offsetTop; 256 | const viewBottom = navSidebar.scrollTop + navSidebar.clientHeight; 257 | 258 | // The element height and scroll height are the same, then we are still loading 259 | if (viewBottom !== navSidebar.scrollHeight) { 260 | // Determine if the item isn't visible and scroll to it 261 | if (elBottom >= viewBottom) { 262 | navSidebar.scrollTop = elBottom; 263 | } 264 | 265 | // stop observing now since we've completed the scroll 266 | resizeObserver.unobserve(navSidebar); 267 | } 268 | }); 269 | resizeObserver.observe(navSidebar); 270 | } 271 | } 272 | } 273 | }); 274 | -------------------------------------------------------------------------------- /errata.md: -------------------------------------------------------------------------------- 1 | ## Errata 2 | 3 | - Training with `luz`, line [486](https://github.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/blob/a9a6a0f8d2e32cd5ef8a059bfc6f885834578663/training_with_luz.qmd#L483): replace `nn_mse_loss()` with `nnf_mse_loss()` [issue](https://github.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/issues/2) 4 | - Training with `luz`, line [494](https://github.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/blob/a9a6a0f8d2e32cd5ef8a059bfc6f885834578663/training_with_luz.qmd#L494): replace `nn_mse_loss()` with `nnf_mse_loss()` [issue](https://github.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/issues/2) 5 | -------------------------------------------------------------------------------- /images/adagrad.ggb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/adagrad.ggb -------------------------------------------------------------------------------- /images/adam.ggb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/adam.ggb -------------------------------------------------------------------------------- /images/audio-alluvial-baseline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-alluvial-baseline.png -------------------------------------------------------------------------------- /images/audio-alluvial-complex.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-alluvial-complex.png -------------------------------------------------------------------------------- /images/audio-alluvial-mel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-alluvial-mel.png -------------------------------------------------------------------------------- /images/audio-bird-dft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-bird-dft.png -------------------------------------------------------------------------------- /images/audio-bird-waveform.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-bird-waveform.png -------------------------------------------------------------------------------- /images/audio-fit-baseline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-fit-baseline.png -------------------------------------------------------------------------------- /images/audio-fit-complex.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-fit-complex.png -------------------------------------------------------------------------------- /images/audio-fit-mel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-fit-mel.png -------------------------------------------------------------------------------- /images/audio-lr-finder-baseline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-lr-finder-baseline.png -------------------------------------------------------------------------------- /images/audio-lr-finder-complex.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-lr-finder-complex.png -------------------------------------------------------------------------------- /images/audio-lr-finder-mel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-lr-finder-mel.png -------------------------------------------------------------------------------- /images/audio-mel-filterbank.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-mel-filterbank.png -------------------------------------------------------------------------------- /images/audio-mel-spectrogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-mel-spectrogram.png -------------------------------------------------------------------------------- /images/audio-spectrogram-zoom.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-spectrogram-zoom.png -------------------------------------------------------------------------------- /images/audio-spectrogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/audio-spectrogram.png -------------------------------------------------------------------------------- /images/autograd-compgraph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/autograd-compgraph.png -------------------------------------------------------------------------------- /images/autograd-paraboloid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/autograd-paraboloid.png -------------------------------------------------------------------------------- /images/compgraph.odg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/compgraph.odg -------------------------------------------------------------------------------- /images/cross-correlation.ggb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/cross-correlation.ggb -------------------------------------------------------------------------------- /images/cross-correlation.ods: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/cross-correlation.ods -------------------------------------------------------------------------------- /images/dft-cos-1-rev-dft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-cos-1-rev-dft.png -------------------------------------------------------------------------------- /images/dft-cos-1-rev.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-cos-1-rev.png -------------------------------------------------------------------------------- /images/dft-cos-32-rev.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-cos-32-rev.png -------------------------------------------------------------------------------- /images/dft-cos-4-rev.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-cos-4-rev.png -------------------------------------------------------------------------------- /images/dft-cos-mult-amplitude.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-cos-mult-amplitude.png -------------------------------------------------------------------------------- /images/dft-cos-phase-pi12.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-cos-phase-pi12.png -------------------------------------------------------------------------------- /images/dft-dial-dft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-dial-dft.png -------------------------------------------------------------------------------- /images/dft-dial-fft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-dial-fft.png -------------------------------------------------------------------------------- /images/dft-dial-ifft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-dial-ifft.png -------------------------------------------------------------------------------- /images/dft-dial-spectrogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-dial-spectrogram.png -------------------------------------------------------------------------------- /images/dft-dial-waveform.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-dial-waveform.png -------------------------------------------------------------------------------- /images/dft-mix-of-sinuisoids.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dft-mix-of-sinuisoids.png -------------------------------------------------------------------------------- /images/dimensions.ggb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dimensions.ggb -------------------------------------------------------------------------------- /images/dimensions.xcf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/dimensions.xcf -------------------------------------------------------------------------------- /images/efficiency-mnist-lr-finder.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/efficiency-mnist-lr-finder.png -------------------------------------------------------------------------------- /images/fft-perf-jit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/fft-perf-jit.png -------------------------------------------------------------------------------- /images/fft-perf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/fft-perf.png -------------------------------------------------------------------------------- /images/images-conv-arithmetic-dilation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images-conv-arithmetic-dilation.png -------------------------------------------------------------------------------- /images/images-conv-arithmetic-padding.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images-conv-arithmetic-padding.png -------------------------------------------------------------------------------- /images/images-conv-arithmetic-strides.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images-conv-arithmetic-strides.png -------------------------------------------------------------------------------- /images/images-cross-correlation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images-cross-correlation.png -------------------------------------------------------------------------------- /images/images-feature-visualization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images-feature-visualization.png -------------------------------------------------------------------------------- /images/images-square-bottom.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images-square-bottom.png -------------------------------------------------------------------------------- /images/images-square-filters.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images-square-filters.png -------------------------------------------------------------------------------- /images/images-square-left.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images-square-left.png -------------------------------------------------------------------------------- /images/images-square-right.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images-square-right.png -------------------------------------------------------------------------------- /images/images-square-top.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images-square-top.png -------------------------------------------------------------------------------- /images/images-square.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images-square.png -------------------------------------------------------------------------------- /images/images2-lr-finder-batchnorm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images2-lr-finder-batchnorm.png -------------------------------------------------------------------------------- /images/images2-lr-finder-dropout.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images2-lr-finder-dropout.png -------------------------------------------------------------------------------- /images/images2-lr-finder-resnet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/images2-lr-finder-resnet.png -------------------------------------------------------------------------------- /images/least-squares-benchmark.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/least-squares-benchmark.png -------------------------------------------------------------------------------- /images/momentum.ggb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/momentum.ggb -------------------------------------------------------------------------------- /images/nesterov.ggb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/nesterov.ggb -------------------------------------------------------------------------------- /images/nesterov_rosenbrock.ggb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/nesterov_rosenbrock.ggb -------------------------------------------------------------------------------- /images/optim-1-rosenbrock.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/optim-1-rosenbrock.png -------------------------------------------------------------------------------- /images/optimizers-adagrad.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/optimizers-adagrad.png -------------------------------------------------------------------------------- /images/optimizers-adam.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/optimizers-adam.png -------------------------------------------------------------------------------- /images/optimizers-momentum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/optimizers-momentum.png -------------------------------------------------------------------------------- /images/optimizers-rmsprop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/optimizers-rmsprop.png -------------------------------------------------------------------------------- /images/optimizers-steepest-descent-elliptic.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/optimizers-steepest-descent-elliptic.png -------------------------------------------------------------------------------- /images/optimizers-steepest-descent-symmetric.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/optimizers-steepest-descent-symmetric.png -------------------------------------------------------------------------------- /images/overfitting-mnist-mixup-0.5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/overfitting-mnist-mixup-0.5.png -------------------------------------------------------------------------------- /images/overfitting-mnist-mixup-0.7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/overfitting-mnist-mixup-0.7.png -------------------------------------------------------------------------------- /images/overfitting-mnist-mixup-0.9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/overfitting-mnist-mixup-0.9.png -------------------------------------------------------------------------------- /images/overfitting-mnist-test.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/overfitting-mnist-test.png -------------------------------------------------------------------------------- /images/overfitting-mnist-transforms.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/overfitting-mnist-transforms.png -------------------------------------------------------------------------------- /images/paraboloid.ggb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/paraboloid.ggb -------------------------------------------------------------------------------- /images/rmsprop.ggb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/rmsprop.ggb -------------------------------------------------------------------------------- /images/segmentation-conv-arithmetic-transposed.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/segmentation-conv-arithmetic-transposed.png -------------------------------------------------------------------------------- /images/segmentation-lr-finder.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/segmentation-lr-finder.png -------------------------------------------------------------------------------- /images/segmentation-segmentation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/segmentation-segmentation.png -------------------------------------------------------------------------------- /images/segmentation-unet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/segmentation-unet.png -------------------------------------------------------------------------------- /images/square.xcf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/square.xcf -------------------------------------------------------------------------------- /images/steepest_descent_elliptic.ggb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/steepest_descent_elliptic.ggb -------------------------------------------------------------------------------- /images/steepest_descent_symmetric.ggb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/steepest_descent_symmetric.ggb -------------------------------------------------------------------------------- /images/tabular-heart-disease-biplot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/tabular-heart-disease-biplot.png -------------------------------------------------------------------------------- /images/tabular-heart-disease-fit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/tabular-heart-disease-fit.png -------------------------------------------------------------------------------- /images/tabular-heart-disease-pca.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/tabular-heart-disease-pca.png -------------------------------------------------------------------------------- /images/tensors-dimensions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/tensors-dimensions.png -------------------------------------------------------------------------------- /images/timeseries-vic-elec-fit-mlp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/timeseries-vic-elec-fit-mlp.png -------------------------------------------------------------------------------- /images/timeseries-vic-elec-fit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/timeseries-vic-elec-fit.png -------------------------------------------------------------------------------- /images/timeseries-vic-elec-lr-finder-mlp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/timeseries-vic-elec-lr-finder-mlp.png -------------------------------------------------------------------------------- /images/timeseries-vic-elec-lr-finder.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/timeseries-vic-elec-lr-finder.png -------------------------------------------------------------------------------- /images/timeseries-vic-elec-preds-mlp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/timeseries-vic-elec-preds-mlp.png -------------------------------------------------------------------------------- /images/timeseries-vic-elec-preds.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/timeseries-vic-elec-preds.png -------------------------------------------------------------------------------- /images/timeseries-vic-elec-stl-month.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/timeseries-vic-elec-stl-month.png -------------------------------------------------------------------------------- /images/timeseries-vic-elec-stl.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/timeseries-vic-elec-stl.png -------------------------------------------------------------------------------- /images/wav-K-omega.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-K-omega.png -------------------------------------------------------------------------------- /images/wav-chaffinch-fft.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-chaffinch-fft.png -------------------------------------------------------------------------------- /images/wav-chaffinch-spectrogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-chaffinch-spectrogram.png -------------------------------------------------------------------------------- /images/wav-chaffinch-waveletdiag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-chaffinch-waveletdiag.png -------------------------------------------------------------------------------- /images/wav-chaffinch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-chaffinch.png -------------------------------------------------------------------------------- /images/wav-example-long-diag1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-example-long-diag1.png -------------------------------------------------------------------------------- /images/wav-example-long-diag2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-example-long-diag2.png -------------------------------------------------------------------------------- /images/wav-example-long-diag3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-example-long-diag3.png -------------------------------------------------------------------------------- /images/wav-example-long-diag4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-example-long-diag4.png -------------------------------------------------------------------------------- /images/wav-example-long-diag5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-example-long-diag5.png -------------------------------------------------------------------------------- /images/wav-example-long-diag6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-example-long-diag6.png -------------------------------------------------------------------------------- /images/wav-example-long-transform.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-example-long-transform.png -------------------------------------------------------------------------------- /images/wav-example-long-transform2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-example-long-transform2.png -------------------------------------------------------------------------------- /images/wav-morlet-example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-morlet-example.png -------------------------------------------------------------------------------- /images/wav-signal-long.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-signal-long.png -------------------------------------------------------------------------------- /images/wav-signal-short-transform-largeK.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-signal-short-transform-largeK.png -------------------------------------------------------------------------------- /images/wav-signal-short-transform.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-signal-short-transform.png -------------------------------------------------------------------------------- /images/wav-signal-short.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/images/wav-signal-short.png -------------------------------------------------------------------------------- /index.qmd: -------------------------------------------------------------------------------- 1 | ::: {.content-visible when-format="html"} 2 |

Welcome!

3 | 4 | **This is the on-line edition of *Deep Learning and Scientific Computing with R `torch`*, written by [Sigrid Keydana](https://divergences.xyz/). Visit the [GitHub repository for this site](https://github.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch), or buy a physical copy from the publisher, [CRC Press](https://doi.org/10.1201/9781003275923). You'll also find the book at the usual outlets, e.g., [Amazon](https://www.amazon.com/Learning-Scientific-Computing-torch-Chapman/dp/1032231394).** 5 | 6 | **This on-line work is licensed under a** **Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License****.** 7 | ::: 8 | 9 | # Preface {.unnumbered .unlisted} 10 | 11 | This is a book about `torch`, the R interface to PyTorch. PyTorch, as of this writing, is one of the major deep-learning and scientific-computing frameworks, widely used across industries and areas of research. With `torch`, you get to access its rich functionality directly from R, with no need to install, let alone learn, Python. Though still "young" as a project, `torch` already has a vibrant community of users and developers; the latter not just extending the core framework, but also, building on it in their own packages. 12 | 13 | In this text, I'm attempting to attain three goals, corresponding to the book's three major sections. 14 | 15 | The first is a thorough introduction to core `torch`: the basic structures without whom nothing would work. Even though, in future work, you'll likely go with higher-level syntactic constructs when possible, it is important to know what it is they take care of, and to have understood the core concepts. What's more, from a practical point of view, you just need to be "fluent" in `torch` to some degree, so you don't have to resort to "trial-and-error-programming" too often. 16 | 17 | In the second section, basics explained, we proceed to explore various applications of deep learning, ranging from image recognition over time series and tabular data to audio classification. Here, too, the focus is on conceptual explanation. In addition, each chapter presents an approach you can use as a "template" for your own applications. Whenever adequate, I also try to point out the importance of incorporating domain knowledge, as opposed to the not-uncommon "big data, big models, big compute" approach. 18 | 19 | The third section is special in that it highlights some of the non-deep-learning things you can do with `torch`: matrix computations (e.g., various ways of solving linear-regression problems), calculating the Discrete Fourier Transform, and wavelet analysis. Here, more than anywhere else, the conceptual approach is very important to me. Let me explain. 20 | 21 | For one, I expect that in terms of educational background, my readers will vary quite a bit. With R being increasingly taught, and used, in the natural sciences, as well as other areas close to applied mathematics, there will be those who feel they can't benefit much from a conceptual (though formula-guided!) explanation of how, say, the Discrete Fourier Transform works. To others, however, much of this may be uncharted territory, never to be entered if all goes its normal way. This may hold, for example, for people with a humanist, not-traditionally-empirically-oriented background, such as literature, cultural studies, or the philologies. Of course, chances are that if you're among the latter, you may find my explanations, though concept-focused, still highly (or: too) mathematical. In that case, please rest assured that, to the understanding of these things (like many others worthwhile of understanding), it is a long way; but we have a life's time. 22 | 23 | Secondly, even though deep learning has been "the" paradigm of the last decade, recent developments seem to indicate that interest in mathematical/domain-based foundations is (*again* -- this being a recurring phenomenon) on the rise (Consider, for example, the Geometric Deep Learning approach, systematically explained in @abs-2104-13478, and conceptually introduced in [*Beyond alchemy: A first look at geometric deep learning*](https://blogs.rstudio.com/ai/posts/2021-08-26-geometric-deep-learning/).) In the future, I assume that we'll likely see more and more "hybrid" approaches that integrate deep-learning techniques and domain knowledge. The Fourier Transform is not going away. 24 | 25 | Last but not least, on this topic, let me make clear that, of course, all chapters have `torch` code. In case of the Fourier Transform, for example, you'll see not just the official way of doing this, using dedicated functionality, but also, various ways of coding the algorithm yourself -- in a surprisingly small number of lines, and with highly impressive performance. 26 | 27 | This, in a nutshell, is what to expect from the book. Before I close, there is one thing I absolutely need to say, all the more since even though I'd have liked to, I did not find occasion to address it much in the book, given the technicality of the content. In our societies, as adoption of machine/deep learning ("AI") is growing, so are opportunities for misuse, by governments as well as private organizations. Often, harm may not even be intended; but still, outcomes can be catastrophic, especially for people belonging to minorities, or groups already at a disadvantage. Like that, even the inevitable, in most of today's political systems, drive to make profits results in, at the very least, societies imbued with highly questionable features (think: surveillance, and the "quantification of everything"); and most likely, in discrimination, unfairness, and severe harm. Here, I cannot do more than draw attention to this problem, point you to an introductory blog post that perhaps you'll find useful: [*Starting to think about AI Fairness*](https://blogs.rstudio.com/ai/posts/2021-07-15-ai-fairness/), and just ask you to, please, be actively aware of this problem in public life as well as your own work and applications. 28 | 29 | Finally, let me end with saying thank you. There are far too many people to thank that I could ever be sure I haven't left anyone out; so instead I'll keep this short. I'm extremely grateful to my publisher, CRC Press (first and foremost, David Grubbs and Curtis Hill) for the extraordinarily pleasant interactions during all of the writing and editing phases. And *very* special thanks, for their support related to this book as well as their respective roles in the process, go to Daniel Falbel, the creator and maintainer of `torch`, who in-depth reviewed this book and helped me with many technical issues; Tracy Teal, my manager, who supported and encouraged me in every possible way; and Posit (formerly, RStudio), my employer, who lets me do things like this for a living. 30 | 31 | Sigrid Keydana 32 | -------------------------------------------------------------------------------- /intro.qmd: -------------------------------------------------------------------------------- 1 | # Running torch {#sec:running-torch} 2 | 3 | 4 | - installation 5 | -------------------------------------------------------------------------------- /modules.qmd: -------------------------------------------------------------------------------- 1 | # Modules {#sec:modules} 2 | 3 | In the last chapter, we built a neural network for a regression task. There were two distinct types of operations: linear and non-linear. 4 | 5 | In the non-linear category, we had ReLU activation, expressed as a straightforward function call: `nnf_relu()`. Activation functions are *functions*: Given input $\mathbf{x}$, they return output $\mathbf{y}$ every time. In other words, they are deterministic. It's different with the linear part, though. 6 | 7 | The linear part in the regression network was implemented as multiplication by a matrix -- the weight matrix -- and addition of a vector (the bias vector). With operations like that, results inevitably depend on the actual values stored in the respective tensors. Put differently, the operation is *stateful*. 8 | 9 | Whenever there is state involved, it helps to encapsulate it in an object, freeing the user from manual management. This is what `torch`'s *modules* do. 10 | 11 | Note that term, *modules*\index{module (terminology)}. In `torch`, a module can be of any complexity, ranging from basic *layers* -- like the `nn_linear()` we are going to introduce in a minute -- to complete *models* consisting of many such layers. Code-wise, there is no difference between "layers"\index{layer (terminology)} and "models"\index{model (terminology)}. This is why in some texts, you'll see "module" used throughout. In this book, I'll mostly stay with the common terminology of layers and models, as it maps more closely to how things appear conceptually. 12 | 13 | Back to the *why* of modules. In addition to encapsulation, there is another reason for providing layer objects: Not all often-used layers are as light-weight as `nn_linear()` is. We'll quickly mention a few others at the end of the next section, reserving a complete introduction to later chapters of this book. 14 | 15 | ## Built-in `nn_module()`s 16 | 17 | In `torch`, a linear layer is created using `nn_linear()`. `nn_linear()` expects (at least) two arguments: `in_features` and `out_features`. Let's say your input data has fifty observations with five features each; that is, it is of size 50 x 5. You want to build a hidden layer with sixteen units. Then `in_features` is 5, and `out_features` is 16. (The same 5 and 16 would constitute the number of rows/columns in the weight matrix if you built one yourself.) 18 | 19 | ```{r} 20 | library(torch) 21 | l <- nn_linear(in_features = 5, out_features = 16) 22 | ``` 23 | 24 | Once created, the module readily informs you about its parameters: 25 | 26 | ```{r} 27 | l 28 | ``` 29 | 30 | An `nn_module` containing 96 parameters. 31 | Parameters 32 | weight: Float [1:16, 1:5] 33 | bias: Float [1:16] 34 | 35 | Encapsulation doesn't keep us from inspecting the weight and bias tensors: 36 | 37 | ```{r} 38 | l$weight 39 | ``` 40 | 41 | torch_tensor 42 | -0.2079 -0.1920 0.2926 0.0036 -0.0897 43 | 0.3658 0.0076 -0.0671 0.3981 -0.4215 44 | 0.2568 0.3648 -0.0374 -0.2778 -0.1662 45 | 0.4444 0.3851 -0.1225 0.1678 -0.3443 46 | -0.3998 0.0207 -0.0767 0.4323 0.1653 47 | 0.3997 0.0647 -0.2823 -0.1639 -0.0225 48 | 0.0479 0.0207 -0.3426 -0.1567 0.2830 49 | 0.0925 -0.4324 0.0448 -0.0039 0.1531 50 | -0.2924 -0.0009 -0.1841 0.2028 0.1586 51 | -0.3064 -0.4006 -0.0553 -0.0067 0.2575 52 | -0.0472 0.1238 -0.3583 0.4426 -0.0269 53 | -0.0275 -0.0295 -0.2687 0.2236 0.3787 54 | -0.2617 -0.2221 0.1503 -0.0627 0.1094 55 | 0.0122 0.2041 0.4466 0.4112 0.4168 56 | -0.4362 -0.3390 0.3679 -0.3045 0.1358 57 | 0.2979 0.0023 0.0695 -0.1906 -0.1526 58 | [ CPUFloatType{16,5} ] 59 | 60 | ```{r} 61 | l$bias 62 | ``` 63 | 64 | torch_tensor 65 | -0.2314 66 | 0.2942 67 | 0.0567 68 | -0.1728 69 | -0.3220 70 | -0.1553 71 | -0.4149 72 | -0.2103 73 | -0.1769 74 | 0.4219 75 | -0.3368 76 | 0.0689 77 | 0.3625 78 | -0.1391 79 | -0.1411 80 | -0.2014 81 | [ CPUFloatType{16} ] 82 | 83 | At this point, I need to ask for your indulgence. You've probably noticed that `torch` reports the weight matrix as being of size 16 x 5, not 5 x 16, like we said you'd create it when coding from scratch. This is due to an implementation detail inherited from the underlying C++ implementation, `libtorch`. For performance reasons, `libtorch`'s linear module stores the weight and bias tensors in *transposed* form. On the R side, all we can do is explicitly point you to it and thereby, hopefully, alleviate the confusion. 84 | 85 | Let's go on. To apply this module to input data, just "call" it like a function: 86 | 87 | ```{r} 88 | x <- torch_randn(50, 5) 89 | output <- l(x) 90 | output$size() 91 | ``` 92 | 93 | [1] 50 16 94 | 95 | So that's the forward pass. How about gradient computation? Previously, when creating a tensor we wanted to figure as a "source" in gradient computation, we had to let `torch` know explicitly, passing `requires_grad = TRUE`. No such thing is required for built-in `nn_module()`s. We can immediately check that `output` knows what to do on `backward()`: 96 | 97 | ```{r} 98 | output$grad_fn 99 | ``` 100 | 101 | AddmmBackward0 102 | 103 | To be sure though, let's calculate some "dummy" loss based on `output`, and call `backward()`. We see that now, the linear module's `weight` tensor has its `grad` field populated: 104 | 105 | ```{r} 106 | loss <- output$mean() 107 | loss$backward() 108 | l$weight$grad 109 | ``` 110 | 111 | torch_tensor 112 | 0.01 * 113 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 114 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 115 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 116 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 117 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 118 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 119 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 120 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 121 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 122 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 123 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 124 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 125 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 126 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 127 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 128 | -0.3064 2.4118 -0.6095 0.3419 -1.6131 129 | [ CPUFloatType{16,5} ] 130 | 131 | Thus, once you work with `nn_module`s, `torch` automatically assumes that you'll want gradients computed. 132 | 133 | `nn_linear()`, straightforward though it may be, is an essential building block encountered in most every model architecture. Others include: 134 | 135 | - `nn_conv1d()`, `nn_conv2d(), and nn_conv3d()`, the so-called *convolutional* layers that apply filters to input data of varying dimensionality, 136 | 137 | - `nn_lstm()` and `nn_gru()` , the *recurrent* layers that carry through a state, 138 | 139 | - `nn_embedding()` that is used to embed categorical data in high-dimensional space, 140 | 141 | - and more. 142 | 143 | ## Building up a model 144 | 145 | The built-in `nn_module()`s give us *layers*, in usual speak. How do we combine those into *models*? Using the "factory function" `nn_module()`, we can define models of arbitrary complexity. But we may not always need to go that way. 146 | 147 | ### Models as sequences of layers: `nn_sequential()`index{`nn_sequential()`} 148 | 149 | If all our model should do is propagate straight through the layers, we can use `nn_sequential()` to build it. Models consisting of all linear layers are known as *Multi-Layer Perceptrons*index{Multi-Layer Perceptron (MLP)} (MLPs). Here is one: 150 | 151 | ```{r} 152 | mlp <- nn_sequential( 153 | nn_linear(10, 32), 154 | nn_relu(), 155 | nn_linear(32, 64), 156 | nn_relu(), 157 | nn_linear(64, 1) 158 | ) 159 | ``` 160 | 161 | Take a close look at the layers involved. We've already seen `nnf_relu()`, the *function* that implements ReLU activation. (The `f` in `nnf_` stands for functional.) Below, `nn_relu`, like `nn_linear()`, is a module, that is, an object. This is because `nn_sequential()` expects all its arguments to be modules. 162 | 163 | Just like the built-in modules, you can apply this model to data by just *calling* it: 164 | 165 | ```{r} 166 | mlp(torch_randn(5, 10)) 167 | ``` 168 | 169 | torch_tensor 170 | 0.01 * 171 | -7.8097 172 | -9.0363 173 | -38.3282 174 | 5.3959 175 | -16.4837 176 | [ CPUFloatType{5,1} ][ grad_fn = ] 177 | 178 | The single call triggered a complete forward pass through the network. Analogously, calling `backward()` will back-propagate\index{backpropagation} through all the layers. 179 | 180 | What if you need the model to chain execution steps in a non-sequential way? 181 | 182 | ### Models with custom logic 183 | 184 | As already hinted at, this is where you use `nn_module()`. 185 | 186 | `nn_module()` creates constructors for custom-made R6 objects. Below, `my_linear()` is such a constructor. When called, it will return a linear module similar to the built-in `nn_linear()`. 187 | 188 | Two methods should be implemented in defining a constructor: `initialize()` and `forward()`. `initialize()` creates the module object's fields, that is, the objects or values it "owns" and can access from inside any of its methods. `forward()` defines what should happen when the module is called on the input: 189 | 190 | ```{r} 191 | my_linear <- nn_module( 192 | initialize = function(in_features, out_features) { 193 | self$w <- nn_parameter(torch_randn( 194 | in_features, out_features 195 | )) 196 | self$b <- nn_parameter(torch_zeros(out_features)) 197 | }, 198 | forward = function(input) { 199 | input$mm(self$w) + self$b 200 | } 201 | ) 202 | ``` 203 | 204 | Note the use of `nn_parameter()`. `nn_parameter()` makes sure that the passed-in tensor is registered as a module *parameter*, and thus, is subject to backpropagation by default. 205 | 206 | To instantiate the newly-defined module, call its constructor: 207 | 208 | ```{r} 209 | l <- my_linear(7, 1) 210 | l 211 | ``` 212 | 213 | An `nn_module` containing 8 parameters. 214 | 215 | Parameters ──────────────────────────────────────────────────────────────────────────────────────────── 216 | ● w: Float [1:7, 1:1] 217 | ● b: Float [1:1] 218 | 219 | Granted, in this example, there really is no *custom logic* we needed to define our own module for. But here, you have a template applicable to any use case. Later, we'll see definitions of `initialize()` and `forward()` that are more complex, and we'll encounter additional methods defined on modules. But the basic mechanism will remain the same. 220 | 221 | At this point, you may feel like you'd like to rewrite last chapter's neural network using modules. Feel free to do so! Or maybe wait until, in the next chapter, we'll have learned about *optimizer*s, and built-in loss functions. Once we're done, we'll return to our two examples, function minimization and the regression network. Then, we'll be removing all do-it-yourself pieces rendered superfluous by `torch`. 222 | -------------------------------------------------------------------------------- /network_2.qmd: -------------------------------------------------------------------------------- 1 | # Modularizing the neural network {#sec:network-2} 2 | 3 | Let's recall the network we built a few chapters ago. Its purpose was regression, but its method was not *linear*. Instead, an activation function (ReLU, for "rectified linear unit") introduced a nonlinearity, located between the single hidden layer and the output layer. The "layers", in this original implementation, were just tensors: weights and biases. You won't be surprised to hear that these will be replaced by *modules*. 4 | 5 | How will the training process change? Conceptually, we can distinguish four phases: the forward pass, loss computation, backpropagation of gradients, and weight updating. Let's think about where our new tools will fit in: 6 | 7 | - The forward pass, instead of calling functions on tensors, will call the model. 8 | 9 | - In computing the loss, we now make use of `torch`'s `nnf_mse_loss()`. 10 | 11 | - Backpropagation of gradients is, in fact, the only operation that remains unchanged. 12 | 13 | - Weight updating is taken care of by the optimizer. 14 | 15 | Once we've made those changes, the code will be more modular, and a lot more readable. 16 | 17 | ## Data 18 | 19 | As a prerequisite, we generate the data, same as last time. 20 | 21 | ```{r} 22 | # input dimensionality (number of input features) 23 | d_in <- 3 24 | # number of observations in training set 25 | n <- 100 26 | 27 | x <- torch_randn(n, d_in) 28 | coefs <- c(0.2, -1.3, -0.5) 29 | y <- x$matmul(coefs)$unsqueeze(2) + torch_randn(n, 1) 30 | 31 | 32 | ``` 33 | 34 | ## Network 35 | 36 | With two linear layers connected via ReLU activation, the easiest choice is a sequential module, very similar to the one we saw in the introduction to modules: 37 | 38 | ```{r} 39 | # dimensionality of hidden layer 40 | d_hidden <- 32 41 | # output dimensionality (number of predicted features) 42 | d_out <- 1 43 | 44 | net <- nn_sequential( 45 | nn_linear(d_in, d_hidden), 46 | nn_relu(), 47 | nn_linear(d_hidden, d_out) 48 | ) 49 | ``` 50 | 51 | ## Training 52 | 53 | Here is the updated training process. We use the Adam optimizer, a popular choice. 54 | 55 | ```{r} 56 | 57 | opt <- optim_adam(net$parameters) 58 | 59 | ### training loop -------------------------------------- 60 | 61 | for (t in 1:200) { 62 | 63 | ### -------- Forward pass -------- 64 | y_pred <- net(x) 65 | 66 | ### -------- Compute loss -------- 67 | loss <- nnf_mse_loss(y_pred, y) 68 | if (t %% 10 == 0) 69 | cat("Epoch: ", t, " Loss: ", loss$item(), "\n") 70 | 71 | ### -------- Backpropagation -------- 72 | opt$zero_grad() 73 | loss$backward() 74 | 75 | ### -------- Update weights -------- 76 | opt$step() 77 | 78 | } 79 | ``` 80 | 81 | Epoch: 10 Loss: 2.549933 82 | Epoch: 20 Loss: 2.422556 83 | Epoch: 30 Loss: 2.298053 84 | Epoch: 40 Loss: 2.173909 85 | Epoch: 50 Loss: 2.0489 86 | Epoch: 60 Loss: 1.924003 87 | Epoch: 70 Loss: 1.800404 88 | Epoch: 80 Loss: 1.678221 89 | Epoch: 90 Loss: 1.56143 90 | Epoch: 100 Loss: 1.453637 91 | Epoch: 110 Loss: 1.355832 92 | Epoch: 120 Loss: 1.269234 93 | Epoch: 130 Loss: 1.195116 94 | Epoch: 140 Loss: 1.134008 95 | Epoch: 150 Loss: 1.085828 96 | Epoch: 160 Loss: 1.048921 97 | Epoch: 170 Loss: 1.021384 98 | Epoch: 180 Loss: 1.0011 99 | Epoch: 190 Loss: 0.9857832 100 | Epoch: 200 Loss: 0.973796 101 | 102 | In addition to shortening and streamlining the code, our changes have made a big difference performance-wise. 103 | 104 | ## What's to come 105 | 106 | You now know a lot about how `torch` works, and how to use it to minimize a cost function in various settings: for example, to train a neural network. But for real-world applications, there is a lot more `torch` has to offer. The next -- and most voluminous -- part of the book focuses on deep learning. 107 | -------------------------------------------------------------------------------- /optim_1.qmd: -------------------------------------------------------------------------------- 1 | # Function minimization with *autograd* {#sec:optim-1} 2 | 3 | In the last two chapters, we've learned about tensors and automatic differentiation. In the upcoming two, we take a break from studying `torch` mechanics and, instead, find out what we're able to do with what we already have. Using nothing but tensors, and supported by nothing but *autograd*, we can already do two things: 4 | 5 | - minimize a function (i.e., perform numerical optimization), and 6 | 7 | - build and train a neural network. 8 | 9 | In this chapter, we start with minimization, and leave the network to the next one. 10 | 11 | ## An optimization classic 12 | 13 | In optimization\index{optimization} research, the *Rosenbrock function* is a classic. It is a function of two variables; its minimum is at `(1,1)`. If you take a look at its contours, you see that the minimum lies inside a stretched-out, narrow valley (@fig-optim-1-rosenbrock): 14 | 15 | ![Rosenbrock function.](images/optim-1-rosenbrock.png){#fig-optim-1-rosenbrock fig-alt="Contour plot of a function in two variables, where the small function values lie inside a stretched-out, narrow valley."} 16 | 17 | Here is the function definition. `a` and `b` are parameters that can be freely chosen; the values we use here are a frequent choice. 18 | 19 | ```{r} 20 | a <- 1 21 | b <- 5 22 | 23 | rosenbrock <- function(x) { 24 | x1 <- x[1] 25 | x2 <- x[2] 26 | (a - x1)^2 + b * (x2 - x1^2)^2 27 | } 28 | ``` 29 | 30 | ## Minimization from scratch 31 | 32 | The scenario is the following. We start at some given point `(x1,x2)`, and set out to find the location where the Rosenbrock function has its minimum. 33 | 34 | We follow the strategy outlined in the previous chapter: compute the function's gradient at our current position, and use it to go the opposite way. We don't know how far to go; if we take too big a big step we may easily overshoot. (If you look back at the contour plot, you see that if you were standing at one of the steep cliffs east or west of the minimum, this could happen very fast.) 35 | 36 | Thus, it is best to proceed iteratively, taking moderate steps and re-evaluating the gradient every time. 37 | 38 | In a nutshell, the optimization procedure then looks somewhat like this: 39 | 40 | ```{r} 41 | library(torch) 42 | 43 | # attention: this is not the correct procedure yet! 44 | 45 | for (i in 1:num_iterations) { 46 | 47 | # call function, passing in current parameter value 48 | value <- rosenbrock(x) 49 | 50 | # compute gradient of value w.r.t. parameter 51 | value$backward() 52 | 53 | # manually update parameter, subtracting a fraction 54 | # of the gradient 55 | # this is not quite correct yet! 56 | x$sub_(lr * x$grad) 57 | } 58 | ``` 59 | 60 | As written, this code snippet demonstrates our intentions, but it's not quite correct (yet). It is also missing a few prerequisites: Neither the tensor `x` nor the variables `lr` and `num_iterations` have been defined. Let's make sure we have those ready first. `lr`, for learning rate, is the fraction of the gradient to subtract on every step, and `num_iterations` is the number of steps to take. Both are a matter of experimentation. 61 | 62 | ```{r} 63 | lr <- 0.01 64 | 65 | num_iterations <- 1000 66 | ``` 67 | 68 | `x` is the parameter to optimize, that is, it is the function input that hopefully, at the end of the process, will yield the minimum possible function value. This makes it the tensor *with respect to which* we want to compute the function value's derivative. And that, in turn, means we need to create it with `requires_grad = TRUE`: 69 | 70 | ```{r} 71 | x <- torch_tensor(c(-1, 1), requires_grad = TRUE) 72 | ``` 73 | 74 | The starting point, `(-1,1)`, here has been chosen arbitrarily. 75 | 76 | Now, all that remains to be done is apply a small fix to the optimization loop. With *autograd* enabled on `x`, `torch` will record all operations performed on that tensor, meaning that whenever we call `backward()`, it will compute all required derivatives. However, when we subtract a fraction of the gradient, this is not something we want a derivative to be calculated for! We need to tell `torch` not to record this action, and that we can do by wrapping it in `with_no_grad()`. 77 | 78 | There's one other thing we have to tell it. By default, `torch` accumulates the gradients stored in `grad` fields. We need to zero them out for every new calculation, using `grad$zero_()`. 79 | 80 | Taking into account these considerations, the parameter update should look like this: 81 | 82 | ```{r} 83 | with_no_grad({ 84 | x$sub_(lr * x$grad) 85 | x$grad$zero_() 86 | }) 87 | ``` 88 | 89 | Here is the complete code, enhanced with logging statements that make it easier to see what is going on. 90 | 91 | ```{r} 92 | num_iterations <- 1000 93 | 94 | lr <- 0.01 95 | 96 | x <- torch_tensor(c(-1, 1), requires_grad = TRUE) 97 | 98 | for (i in 1:num_iterations) { 99 | if (i %% 100 == 0) cat("Iteration: ", i, "\n") 100 | 101 | value <- rosenbrock(x) 102 | if (i %% 100 == 0) { 103 | cat("Value is: ", as.numeric(value), "\n") 104 | } 105 | 106 | value$backward() 107 | if (i %% 100 == 0) { 108 | cat("Gradient is: ", as.matrix(x$grad), "\n") 109 | } 110 | 111 | with_no_grad({ 112 | x$sub_(lr * x$grad) 113 | x$grad$zero_() 114 | }) 115 | } 116 | ``` 117 | 118 | Iteration: 100 119 | Value is: 0.3502924 120 | Gradient is: -0.667685 -0.5771312 121 | 122 | Iteration: 200 123 | Value is: 0.07398106 124 | Gradient is: -0.1603189 -0.2532476 125 | 126 | Iteration: 300 127 | Value is: 0.02483024 128 | Gradient is: -0.07679074 -0.1373911 129 | 130 | Iteration: 400 131 | Value is: 0.009619333 132 | Gradient is: -0.04347242 -0.08254051 133 | 134 | Iteration: 500 135 | Value is: 0.003990697 136 | Gradient is: -0.02652063 -0.05206227 137 | 138 | Iteration: 600 139 | Value is: 0.001719962 140 | Gradient is: -0.01683905 -0.03373682 141 | 142 | Iteration: 700 143 | Value is: 0.0007584976 144 | Gradient is: -0.01095017 -0.02221584 145 | 146 | Iteration: 800 147 | Value is: 0.0003393509 148 | Gradient is: -0.007221781 -0.01477957 149 | 150 | Iteration: 900 151 | Value is: 0.0001532408 152 | Gradient is: -0.004811743 -0.009894371 153 | 154 | Iteration: 1000 155 | Value is: 6.962555e-05 156 | Gradient is: -0.003222887 -0.006653666 157 | 158 | After thousand iterations, we have reached a function value lower than 0.0001. What is the corresponding `(x1,x2)`-position? 159 | 160 | ```{r} 161 | x 162 | ``` 163 | 164 | torch_tensor 165 | 0.9918 166 | 0.9830 167 | [ CPUFloatType{2} ] 168 | 169 | This is rather close to the true minimum of `(1,1)`. If you feel like, play around a little, and try to find out what kind of difference the learning rate makes. For example, try 0.001 and 0.1, respectively. 170 | 171 | In the next chapter, we will build a neural network from scratch. There, the function we minimize will be a *loss function*, namely, the mean squared error arising from a regression problem. 172 | -------------------------------------------------------------------------------- /optim_2.qmd: -------------------------------------------------------------------------------- 1 | # Function minimization with L-BFGS {#sec:optim-2} 2 | 3 | Now that we've become acquainted with `torch` modules and optimizers, we can go back to the two tasks we already approached without either: function minimization, and training a neural network. Again, we start with minimization, and leave the network to the next chapter. 4 | 5 | Thinking back to what we did when minimizing the Rosenbrock function, in essence it was this: 6 | 7 | 1. Define a tensor to hold the parameter to be optimized, namely, the $\mathbf{x}$-position where the function attains its minimum. 8 | 9 | 2. Iteratively update the parameter, subtracting a fraction of the current gradient. 10 | 11 | While as a strategy, this was straightforward, a problem remained: How big a fraction of the gradient should we subtract? It's exactly here that optimizers come in useful. 12 | 13 | ## Meet L-BFGS\index{L-BFGS (optimizer)} 14 | 15 | So far, we've only talked about the kinds of optimizers often used in deep learning -- stochastic gradient descent (SGD), SGD with momentum, and a few classics from the *adaptive* *learning rate* family: RMSProp, Adadelta, Adagrad, Adam. All these have in common one thing: They only make use of the *gradient*, that is, the vector of first derivatives. Accordingly, they are all *first-order* algorithms. This means, however, that they are missing out on helpful information provided by the *Hessian*, the matrix of second derivatives. 16 | 17 | ### Changing slopes 18 | 19 | First derivatives tell us about the *slope* of the landscape: Does it go up? Does it go down? How much so? Going a step further, second derivatives encode how much that slope *changes*. 20 | 21 | Why should that be important? 22 | 23 | Assume we're at point $\mathbf{x}_n$, and have just decided on a suitable descent direction. We take a step, of length determined by some pre-chosen learning rate, all set to arrive at point $\mathbf{x}_{n+1}$. What we don't know is how the slope will have changed by the time we'll have gotten there. Maybe it's become much flatter in the meantime: In this case, we'll have gone way too far, overshooting and winding up in a far-off area where anything could have happened in-between (including the slope going *up* again!). 24 | 25 | We can illustrate this on a function of a single variable. Take a parabola, such as 26 | 27 | $$ 28 | y = 10x^2 29 | $$ 30 | 31 | Its derivative is $\frac{dy}{dx} = 20x$. If our current $x$ is, say, $3$, and we work with a learning rate of $0.1$, we'll subtract $20 * 3 * 0.1= 6$, winding up at $-3$. 32 | 33 | But say we had slowed down at $2$ and inspected the current slope. We'd have seen that there, the slope was less steep; in fact, when at that point, we should just have subtracted $20 * 2 * 0.1= 4$. 34 | 35 | By sheer luck, this "close-your-eyes-and-jump" strategy can still work out -- *if* we happen to be using just the right learning rate for the function in question. (At the chosen learning rate, this would have been the case for a different parabola, $y = 5x^2$, for example.) But wouldn't it make sense to include second derivatives in the decision from the outset? 36 | 37 | Algorithms that do this form the family of Newton methods. First, we look at their "purest" specimen, which best illustrates the principle but seldom is feasible in practice. 38 | 39 | ### Exact Newton method 40 | 41 | In higher dimensions, the exact Newton method multiplies the gradient by the inverse of the Hessian, thus scaling the descent direction coordinate-by-coordinate. Our current example has just a single independent variable; so this means for us: take the first derivative, and divide by the second. 42 | 43 | We now have a scaled gradient -- but what portion of it should we subtract? In its original version, the exact Newton method does not make use of a learning rate, thus freeing us of the familiar trial-and-error game. Let's see, then: In our example, the second derivative is $20$, meaning that at $x=3$ we have to subtract $(20 * 3)/20=3$. Voilà, we end up at $0$, the location of the minimum, in a single step. 44 | 45 | Seeing how that turned out just great, why don't we do it all the time? For one, it will work perfectly only with quadratic functions, like the one we chose for the demonstration. In other cases, it, too, will normally need some "tuning", for example, by using a learning rate here as well. 46 | 47 | But the main reason is another one. In more realistic applications, and certainly in the areas of machine learning and deep learning, computing the inverse of the Hessian at every step is way too costly. (It may, in fact, not even be possible.) This is where *approximate*, a.k.a. *Quasi-Newton*, methods come in. 48 | 49 | ### Approximate Newton: BFGS and L-BFGS 50 | 51 | Among approximate Newton methods, probably the most-used is the *Broyden-Goldfarb-Fletcher-Shanno* algorithm, or *BFGS*. Instead of continually computing the exact inverse of the Hessian, it keeps an iteratively-updated approximation of that inverse. BFGS is often implemented in a more memory-friendly version, referred to as *Limited-Memory BFGS* (*L-BFGS*). This is the one provided as part of the core `torch` optimizers. 52 | 53 | Before we get there, though, there is one last conceptual thing to discuss. 54 | 55 | ### Line search 56 | 57 | Like their exact counterpart, approximate Newton methods can work without a learning rate. In that case, they compute a descent direction and follow the scaled gradient as-is. We already talked about how, depending on the function in question, this can work more or less well. When it does not, there are two things one could do: Firstly, take small steps, or put differently, introduce a learning rate. And secondly, do a *line search*. 58 | 59 | With line search, we spend some time evaluating how far to follow the descent direction. There are two principal ways of doing this. 60 | 61 | The first, *exact* line search, involves yet another optimization problem: Take the current point, compute the descent direction, and hard-code them as givens in a *second* function that depends on the learning rate only. Then, differentiate this function to find *its* minimum. The solution will be the learning rate that optimizes the step length taken. 62 | 63 | The alternative strategy is to do an approximate search. By now, you're probably not surprised: Just as approximate Newton is more realistically-feasible than exact Newton, approximate line search is more practicable than exact line search. 64 | 65 | For line search, approximating the best solution means following a set of proven heuristics. Essentially, we look for something that is *just* *good enough*. Among the most established heuristics are the *Strong Wolfe conditions*, and this is the strategy implemented in `torch`'s `optim_lbfgs()`. In the next section, we'll see how to use `optim_lbfgs()` to minimize the Rosenbrock function, both with and without line search. 66 | 67 | ## Minimizing the Rosenbrock function with `optim_lbfgs()` 68 | 69 | Here is the Rosenbrock function again: 70 | 71 | ```{r} 72 | library(torch) 73 | 74 | a <- 1 75 | b <- 5 76 | 77 | rosenbrock <- function(x) { 78 | x1 <- x[1] 79 | x2 <- x[2] 80 | (a - x1)^2 + b * (x2 - x1^2)^2 81 | } 82 | ``` 83 | 84 | In our manual minimization efforts, the procedure was the following. A one-time action, we first defined the parameter tensor destined to hold the current $\mathbf{x}$: 85 | 86 | ```{r} 87 | x <- torch_tensor(c(-1, 1), requires_grad = TRUE) 88 | ``` 89 | 90 | Then, we iteratively executed the following operations: 91 | 92 | 1. Calculate the function value at the current $\mathbf{x}$. 93 | 94 | 2. Compute the gradient of that value at the position in question. 95 | 96 | 3. Subtract a fraction of the gradient from the current $\mathbf{x}$. 97 | 98 | How, if so, does that blueprint change? 99 | 100 | The first step remains unchanged. We still have 101 | 102 | ```{r} 103 | value <- rosenbrock(x) 104 | ``` 105 | 106 | The second step stays the same, as well. We still call `backward()` directly on the output tensor: 107 | 108 | ```{r} 109 | value$backward() 110 | ``` 111 | 112 | This is because an optimizer does not *compute* gradients; it *decides what to do with the gradient* once it's been computed. 113 | 114 | What changes, thus, is the third step, the one that also was the most cumbersome. Now, it is the optimizer that applies the update. To be able to do that, there is a prerequisite: Prior to starting the loop, the optimizer will need to be told which parameter it is supposed to work on. In fact, this is so important that you can't even create an optimizer without passing it that parameter: 115 | 116 | ```{r} 117 | opt <- optim_lbfgs(x) 118 | ``` 119 | 120 | In the loop, we now call the `step()` method on the optimizer object to update the parameter. There is just one part from our manual procedure that needs to get carried over to the new way: We still need to zero out the gradient on each iteration. Just this time, not on the parameter tensor, `x`, but the optimizer object itself. *In principle*, this then yields the following actions to be performed on each iteration: 121 | 122 | ```{r} 123 | value <- rosenbrock(x) 124 | 125 | opt$zero_grad() 126 | value$backward() 127 | 128 | opt$step() 129 | ``` 130 | 131 | Why "in principle"? In fact, this is what we'd write for every optimizer *but* `optim_lbfgs()`. 132 | 133 | For `optim_lbfgs()`, `step()` needs to be called passing in an anonymous function, a closure. Zeroing of previous gradients, function call, and gradient calculation, all these happen inside the closure: 134 | 135 | ```{r} 136 | calc_loss <- function() { 137 | optimizer$zero_grad() 138 | value <- rosenbrock(x_star) 139 | value$backward() 140 | value 141 | } 142 | 143 | 144 | ``` 145 | 146 | Having executed those actions, the closure returns the function value. Here is how it is called by `step()`: 147 | 148 | ```{r} 149 | for (i in 1:num_iterations) { 150 | optimizer$step(calc_loss) 151 | } 152 | ``` 153 | 154 | Now we put it all together, add some logging output, and compare what happens with and without line search. 155 | 156 | ### `optim_lbfgs()` default behavior 157 | 158 | As a baseline, we first run without line search. Two iterations are enough. In the below output, you can see that in each iteration, the closure is evaluated several times. This is the technical reason we had to create it in the first place. 159 | 160 | ```{r} 161 | num_iterations <- 2 162 | 163 | x <- torch_tensor(c(-1, 1), requires_grad = TRUE) 164 | 165 | optimizer <- optim_lbfgs(x) 166 | 167 | calc_loss <- function() { 168 | optimizer$zero_grad() 169 | 170 | value <- rosenbrock(x) 171 | cat("Value is: ", as.numeric(value), "\n") 172 | 173 | value$backward() 174 | value 175 | } 176 | 177 | for (i in 1:num_iterations) { 178 | cat("\nIteration: ", i, "\n") 179 | optimizer$step(calc_loss) 180 | } 181 | 182 | 183 | ``` 184 | 185 | Iteration: 1 186 | Value is: 4 187 | Value is: 6 188 | Value is: 318.0431 189 | Value is: 5.146369 190 | Value is: 4.443705 191 | Value is: 0.8787204 192 | Value is: 0.8543001 193 | Value is: 2.001667 194 | Value is: 0.5656172 195 | Value is: 0.400589 196 | Value is: 7.726219 197 | Value is: 0.3388008 198 | Value is: 0.2861604 199 | Value is: 1.951176 200 | Value is: 0.2071857 201 | Value is: 0.150776 202 | Value is: 0.411357 203 | Value is: 0.08056168 204 | Value is: 0.04880721 205 | Value is: 0.0302862 206 | 207 | Iteration: 2 208 | Value is: 0.01697086 209 | Value is: 0.01124081 210 | Value is: 0.0006622815 211 | Value is: 3.300996e-05 212 | Value is: 1.35731e-07 213 | Value is: 1.111701e-09 214 | Value is: 4.547474e-12 215 | 216 | To make sure we really have found the minimum, we check `x`: 217 | 218 | ```{r} 219 | x 220 | ``` 221 | 222 | torch_tensor 223 | 1.0000 224 | 1.0000 225 | [ CPUFloatType{2} ] 226 | 227 | Can this still be improved upon? 228 | 229 | ### `optim_lbfgs()` with line search 230 | 231 | Let's see. Below, the only line that's changed is the one where we construct the optimizer. 232 | 233 | ```{r} 234 | num_iterations <- 2 235 | 236 | x <- torch_tensor(c(-1, 1), requires_grad = TRUE) 237 | 238 | optimizer <- optim_lbfgs(x, line_search_fn = "strong_wolfe") 239 | 240 | calc_loss <- function() { 241 | optimizer$zero_grad() 242 | 243 | value <- rosenbrock(x) 244 | cat("Value is: ", as.numeric(value), "\n") 245 | 246 | value$backward() 247 | value 248 | } 249 | 250 | for (i in 1:num_iterations) { 251 | cat("\nIteration: ", i, "\n") 252 | optimizer$step(calc_loss) 253 | } 254 | ``` 255 | 256 | Iteration: 1 257 | Value is: 4 258 | Value is: 6 259 | Value is: 3.802412 260 | Value is: 3.680712 261 | Value is: 2.883048 262 | Value is: 2.5165 263 | Value is: 2.064779 264 | Value is: 1.38384 265 | Value is: 1.073063 266 | Value is: 0.8844351 267 | Value is: 0.5554555 268 | Value is: 0.2501077 269 | Value is: 0.8948895 270 | Value is: 0.1619074 271 | Value is: 0.06823064 272 | Value is: 0.01653575 273 | Value is: 0.004060207 274 | Value is: 0.00353789 275 | Value is: 0.000391416 276 | Value is: 4.303527e-06 277 | Value is: 2.036851e-08 278 | Value is: 6.870948e-12 279 | 280 | Iteration: 2 281 | Value is: 6.870948e-12 282 | 283 | With line search, a single iteration is sufficient to reach the minimum. Inspecting the individual losses, we also see that the algorithm reduces the loss nearly every time it probes the function, which without line search, had not been the case. 284 | -------------------------------------------------------------------------------- /other_overview.qmd: -------------------------------------------------------------------------------- 1 | # Overview {#sec:other-overview} 2 | 3 | By now, we've talked a lot about deep learning. But `torch` is fruitfully employed in other kinds of tasks, as well -- scientific applications, for example, that rely on mathematical methods to discover patterns, relations, and structure. 4 | 5 | In this section, we concentrate on three topics. The first is matrix computations -- a subject whose importance is hard to call into question, seeing how *all* computations in scientific computation and machine learning are matrix computations (tensors just being higher-order matrices). Concretely, we'll solve a least-squares problem by means of matrix factorization, making use of functions like `linalg_cholesky()`, `linalg_qr()`, and `linalg_svd()`. In addition, we'll take a short look at how convolution (in its original, signal-processing sense) can be implemented efficiently. 6 | 7 | Next, we move on to a famous mathematical method we've already made (indirect, but highly beneficial) use of: the Discrete Fourier Transform (DFT). This time, though, we don't just *use* it; instead, we aim to understand *why* and *how* it works. Once we have that understanding, a straightforward implementation is a matter of just a few lines of code. A second chapter is then dedicated to implementing the DFT efficiently, by means of the Fast Fourier Transform (FFT). Again, we start by analyzing its workings, and go on to code it from scratch. You'll see one of the hand-coded methods coming surprisingly close, in performance, to `torch`'s own `torch_fft_fft()`. 8 | 9 | Finally, we explore an idea that is far more recent than Fourier methods; namely, the Wavelet Transform. This transform is widely used in data analysis, and we'll understand clearly why that's the case. In `torch`, there is no dedicated method to compute the Wavelet Transform; but we'll see how repeated use of `torch_fft_fft()` results in an efficient implementation. 10 | -------------------------------------------------------------------------------- /references.bib: -------------------------------------------------------------------------------- 1 | @article{olah2017feature, 2 | author = {Olah, Chris and Mordvintsev, Alexander and Schubert, Ludwig}, 3 | title = {Feature Visualization}, 4 | journal = {Distill}, 5 | year = {2017}, 6 | note = {https://distill.pub/2017/feature-visualization}, 7 | doi = {10.23915/distill.00007} 8 | } 9 | 10 | @ARTICLE{2016arXiv160307285D, 11 | author = {{Dumoulin}, Vincent and {Visin}, Francesco}, 12 | title = "{A guide to convolution arithmetic for deep learning}", 13 | journal = {arXiv e-prints}, 14 | keywords = {Statistics - Machine Learning, Computer Science - Machine Learning, Computer Science - Neural and Evolutionary Computing}, 15 | year = 2016, 16 | month = mar, 17 | eid = {arXiv:1603.07285}, 18 | pages = {arXiv:1603.07285}, 19 | archivePrefix = {arXiv}, 20 | eprint = {1603.07285}, 21 | primaryClass = {stat.ML}, 22 | adsurl = {https://ui.adsabs.harvard.edu/abs/2016arXiv160307285D}, 23 | adsnote = {Provided by the SAO/NASA Astrophysics Data System} 24 | } 25 | 26 | 27 | @article{RonnebergerFB15, 28 | author = {Olaf Ronneberger and 29 | Philipp Fischer and 30 | Thomas Brox}, 31 | title = {U-Net: Convolutional Networks for Biomedical Image Segmentation}, 32 | journal = {CoRR}, 33 | volume = {abs/1505.04597}, 34 | year = {2015}, 35 | url = {http://arxiv.org/abs/1505.04597}, 36 | eprinttype = {arXiv}, 37 | eprint = {1505.04597}, 38 | timestamp = {Mon, 13 Aug 2018 16:46:52 +0200}, 39 | biburl = {https://dblp.org/rec/journals/corr/RonnebergerFB15.bib}, 40 | bibsource = {dblp computer science bibliography, https://dblp.org} 41 | } 42 | 43 | @article{abs-1801-04381, 44 | author = {Mark Sandler and 45 | Andrew G. Howard and 46 | Menglong Zhu and 47 | Andrey Zhmoginov and 48 | Liang{-}Chieh Chen}, 49 | title = {Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, 50 | Detection and Segmentation}, 51 | journal = {CoRR}, 52 | volume = {abs/1801.04381}, 53 | year = {2018}, 54 | url = {http://arxiv.org/abs/1801.04381}, 55 | eprinttype = {arXiv}, 56 | eprint = {1801.04381}, 57 | timestamp = {Tue, 12 Jan 2021 15:30:06 +0100}, 58 | biburl = {https://dblp.org/rec/journals/corr/abs-1801-04381.bib}, 59 | bibsource = {dblp computer science bibliography, https://dblp.org} 60 | } 61 | 62 | 63 | @ARTICLE{2017arXiv171009412Z, 64 | author = {{Zhang}, Hongyi and {Cisse}, Moustapha and {Dauphin}, Yann N. and {Lopez-Paz}, David}, 65 | title = "{mixup: Beyond Empirical Risk Minimization}", 66 | journal = {arXiv e-prints}, 67 | keywords = {Computer Science - Machine Learning, Statistics - Machine Learning}, 68 | year = 2017, 69 | month = oct, 70 | eid = {arXiv:1710.09412}, 71 | pages = {arXiv:1710.09412}, 72 | archivePrefix = {arXiv}, 73 | eprint = {1710.09412}, 74 | primaryClass = {cs.LG}, 75 | adsurl = {https://ui.adsabs.harvard.edu/abs/2017arXiv171009412Z}, 76 | adsnote = {Provided by the SAO/NASA Astrophysics Data System} 77 | } 78 | 79 | @book{ISLR2, 80 | title={An Introduction to Statistical Learning: with Applications in R}, 81 | author={Gareth James and Daniela Witten and Trevor Hastie and Robert Tibshirani}, 82 | publisher={Springer}, 83 | year={2021} 84 | } 85 | 86 | @article{10.5555/2627435.2670313, 87 | author = {Srivastava, Nitish and Hinton, Geoffrey and Krizhevsky, Alex and Sutskever, Ilya and Salakhutdinov, Ruslan}, 88 | title = {Dropout: A Simple Way to Prevent Neural Networks from Overfitting}, 89 | year = {2014}, 90 | issue_date = {January 2014}, 91 | publisher = {JMLR.org}, 92 | volume = {15}, 93 | number = {1}, 94 | issn = {1532-4435}, 95 | journal = {J. Mach. Learn. Res.}, 96 | month = {jan}, 97 | pages = {1929–1958}, 98 | numpages = {30}, 99 | keywords = {neural networks, model combination, regularization, deep learning} 100 | } 101 | 102 | @misc{ioffe2015batch, 103 | title={Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift}, author={Sergey Ioffe and Christian Szegedy}, 104 | year={2015}, 105 | eprint={1502.03167}, 106 | archivePrefix={arXiv}, 107 | primaryClass={cs.LG} 108 | } 109 | 110 | @article{Smith15a, 111 | author = {Leslie N. Smith}, 112 | title = {No More Pesky Learning Rate Guessing Games}, 113 | journal = {CoRR}, 114 | volume = {abs/1506.01186}, 115 | year = {2015}, 116 | url = {http://arxiv.org/abs/1506.01186}, 117 | archivePrefix = {arXiv}, 118 | eprint = {1506.01186}, 119 | timestamp = {Mon, 13 Aug 2018 16:47:53 +0200}, 120 | biburl = {https://dblp.org/rec/journals/corr/Smith15a.bib}, 121 | bibsource = {dblp computer science bibliography, https://dblp.org} 122 | } 123 | 124 | 125 | @article{LoshchilovH16a, 126 | author = {Ilya Loshchilov and 127 | Frank Hutter}, 128 | title = {{SGDR:} Stochastic Gradient Descent with Restarts}, 129 | journal = {CoRR}, 130 | volume = {abs/1608.03983}, 131 | year = {2016}, 132 | url = {http://arxiv.org/abs/1608.03983}, 133 | archivePrefix = {arXiv}, 134 | eprint = {1608.03983}, 135 | timestamp = {Mon, 13 Aug 2018 16:48:29 +0200}, 136 | biburl = {https://dblp.org/rec/journals/corr/LoshchilovH16a.bib}, 137 | bibsource = {dblp computer science bibliography, https://dblp.org} 138 | } 139 | 140 | @article{abs-1708-07120, 141 | author = {Leslie N. Smith and 142 | Nicholay Topin}, 143 | title = {Super-Convergence: Very Fast Training of Residual Networks Using Large 144 | Learning Rates}, 145 | journal = {CoRR}, 146 | volume = {abs/1708.07120}, 147 | year = {2017}, 148 | url = {http://arxiv.org/abs/1708.07120}, 149 | archivePrefix = {arXiv}, 150 | eprint = {1708.07120}, 151 | timestamp = {Mon, 13 Aug 2018 16:48:13 +0200}, 152 | biburl = {https://dblp.org/rec/journals/corr/abs-1708-07120.bib}, 153 | bibsource = {dblp computer science bibliography, https://dblp.org} 154 | } 155 | 156 | @article{HeZRS15, 157 | author = {Kaiming He and 158 | Xiangyu Zhang and 159 | Shaoqing Ren and 160 | Jian Sun}, 161 | title = {Deep Residual Learning for Image Recognition}, 162 | journal = {CoRR}, 163 | volume = {abs/1512.03385}, 164 | year = {2015}, 165 | url = {http://arxiv.org/abs/1512.03385}, 166 | eprinttype = {arXiv}, 167 | eprint = {1512.03385}, 168 | timestamp = {Wed, 17 Apr 2019 17:23:45 +0200}, 169 | biburl = {https://dblp.org/rec/journals/corr/HeZRS15.bib}, 170 | bibsource = {dblp computer science bibliography, https://dblp.org} 171 | } 172 | 173 | @article{hochreiter1997long, 174 | added-at = {2016-11-15T08:49:43.000+0100}, 175 | author = {Hochreiter, Sepp and Schmidhuber, J{\"u}rgen}, 176 | biburl = {https://www.bibsonomy.org/bibtex/2a4a80026d24955b267cae636aa8abe4a/dallmann}, 177 | interhash = {0692b471c4b9ae65d00affebc09fb467}, 178 | intrahash = {a4a80026d24955b267cae636aa8abe4a}, 179 | journal = {Neural computation}, 180 | keywords = {lstm rnn}, 181 | number = 8, 182 | pages = {1735--1780}, 183 | publisher = {MIT Press}, 184 | timestamp = {2016-11-15T08:49:43.000+0100}, 185 | title = {Long short-term memory}, 186 | volume = 9, 187 | year = 1997 188 | } 189 | 190 | @article{ChoMGBSB14, 191 | author = {Kyunghyun Cho and 192 | Bart van Merrienboer and 193 | {\c{C}}aglar G{\"{u}}l{\c{c}}ehre and 194 | Fethi Bougares and 195 | Holger Schwenk and 196 | Yoshua Bengio}, 197 | title = {Learning Phrase Representations using {RNN} Encoder-Decoder for Statistical 198 | Machine Translation}, 199 | journal = {CoRR}, 200 | volume = {abs/1406.1078}, 201 | year = {2014}, 202 | url = {http://arxiv.org/abs/1406.1078}, 203 | eprinttype = {arXiv}, 204 | eprint = {1406.1078}, 205 | timestamp = {Mon, 13 Aug 2018 16:46:44 +0200}, 206 | biburl = {https://dblp.org/rec/journals/corr/ChoMGBSB14.bib}, 207 | bibsource = {dblp computer science bibliography, https://dblp.org} 208 | } 209 | 210 | @article{abs-1804-03209, 211 | author = {Pete Warden}, 212 | title = {Speech Commands: {A} Dataset for Limited-Vocabulary Speech Recognition}, 213 | journal = {CoRR}, 214 | volume = {abs/1804.03209}, 215 | year = {2018}, 216 | url = {http://arxiv.org/abs/1804.03209}, 217 | eprinttype = {arXiv}, 218 | eprint = {1804.03209}, 219 | timestamp = {Mon, 13 Aug 2018 16:48:32 +0200}, 220 | biburl = {https://dblp.org/rec/journals/corr/abs-1804-03209.bib}, 221 | bibsource = {dblp computer science bibliography, https://dblp.org} 222 | } 223 | 224 | @article{2019EA000740, 225 | author = {Cho, Dongjin and Yoo, Cheolhee and Im, Jungho and Cha, Dong-Hyun}, 226 | title = {Comparative Assessment of Various Machine Learning-Based Bias Correction Methods for Numerical Weather Prediction Model Forecasts of Extreme Air Temperatures in Urban Areas}, 227 | journal = {Earth and Space Science}, 228 | volume = {7}, 229 | number = {4}, 230 | pages = {e2019EA000740}, 231 | keywords = {Air temperature forecast, bias correction, random forest, support vector regression, artificial neural networks, multi-model ensemble}, 232 | doi = {https://doi.org/10.1029/2019EA000740}, 233 | url = {https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019EA000740}, 234 | eprint = {https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2019EA000740}, 235 | note = {e2019EA000740 2019EA000740}, 236 | year = {2020} 237 | } 238 | 239 | @book{Trefethen, 240 | title={Numerical linear algebra}, 241 | author={Lloyd N. Trefethen and David Bau}, 242 | publisher={SIAM}, 243 | year={1997} 244 | } 245 | 246 | @book{Osgood, 247 | title={Lectures on the Fourier Transform and Its Applications}, 248 | author={Brad Osgood}, 249 | publisher={American Mathematical Society}, 250 | year={2019} 251 | } 252 | 253 | 254 | @book{waves, 255 | title={Physics of Oscillations and Waves. With use of Matlab and Python}, 256 | author={Arnt Inge Vistnes}, 257 | publisher={Springer}, 258 | year={2018} 259 | } 260 | 261 | 262 | 263 | @article{abs-2104-13478, 264 | author = {Michael M. Bronstein and 265 | Joan Bruna and 266 | Taco Cohen and 267 | Petar Velickovic}, 268 | title = {Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges}, 269 | journal = {CoRR}, 270 | volume = {abs/2104.13478}, 271 | year = {2021}, 272 | url = {https://arxiv.org/abs/2104.13478}, 273 | eprinttype = {arXiv}, 274 | eprint = {2104.13478}, 275 | timestamp = {Tue, 04 May 2021 15:12:43 +0200}, 276 | biburl = {https://dblp.org/rec/journals/corr/abs-2104-13478.bib}, 277 | bibsource = {dblp computer science bibliography, https://dblp.org} 278 | } 279 | -------------------------------------------------------------------------------- /references.qmd: -------------------------------------------------------------------------------- 1 | # References {.unnumbered .unlisted} 2 | 3 | ::: {#refs} 4 | ::: 5 | -------------------------------------------------------------------------------- /resources/adagrad.tex: -------------------------------------------------------------------------------- 1 | \documentclass [border = .2cm] {standalone} 2 | 3 | % Required packages 4 | \usepackage{pgfplots} 5 | \pgfplotsset{compat = newest} 6 | \usetikzlibrary {backgrounds} 7 | 8 | \begin{document} 9 | 10 | \begin{tikzpicture} 11 | [background rectangle/.style= 12 | {bottom color=white!70!gray!}, 13 | show background rectangle] 14 | 15 | \begin{axis}[ 16 | view = {15}{20}, 17 | colormap={blackwhite}{gray(0cm)=(0.5); gray(1cm)=(1)}, 18 | thick, 19 | axis lines = center, 20 | axis on top, 21 | color=black, 22 | xmax = 10, 23 | xmin = -10, 24 | ymax = 3, 25 | ymin = -3, 26 | zmax = 12, 27 | zmin = -6, 28 | xticklabels=\empty, 29 | yticklabels=\empty, 30 | zticklabels=\empty, 31 | xtick=false, 32 | ytick=false, 33 | ztick=false, 34 | ] 35 | 36 | \addplot3 [ 37 | domain=-10:10, 38 | domain y = -3:3, 39 | samples = 50, 40 | samples y = 50, 41 | surf, 42 | mesh, 43 | ultra thin, 44 | shader = interp, 45 | no marks, 46 | ] {0.2 * x^2 + 2 * y^2 - 5}; 47 | 48 | 49 | 50 | \addplot3 [ 51 | color=gray!50!black!, 52 | thin, 53 | mark=*, 54 | mark size=1pt, 55 | draw=gray!50!black!, 56 | ] coordinates 57 | { 58 | (5.1, 2.3, 10.78) 59 | (4.28400000000000, -1.38000000000000, 2.47933120000000) 60 | (3.59856000000000, 0.828000000000001, -1.03890518528000) 61 | (3.02279040000000, -0.496800000000000, -2.67892715953357) 62 | (2.53914393600000, 0.298080000000000, -3.53284624165489) 63 | (2.13288090624000, -0.178848000000000, -4.02619059375137) 64 | (1.79161996124160, 0.107308800000000, -4.33498922578125) 65 | (1.50496076744295, -0.0643852800000001, -4.53872768913016) 66 | (1.26416704465207, 0.0386311680000001, -4.67739160236105) 67 | (1.06190031750774, -0.0231787008000000, -4.77339903879384) 68 | (0.891996266706503, 0.0139072204800000, -4.84048171047338) 69 | (0.749276864033463, -0.00834433228800002, -4.88757758044218) 70 | (0.629392565788109, 0.00500659937280001, -4.92072286755158) 71 | (0.528689755262011, -0.00300395962368001, -4.94407938098936) 72 | (0.444099394420089, 0.00180237577420801, -4.96054864845828) 73 | (0.373043491312875, -0.00108142546452480, -4.97216537175575) 74 | (0.313356532702815, 0.000648855278714882, -4.98036069465615) 75 | (0.263219487470365, -0.000389313167228929, -4.98614279715369) 76 | (0.221104369475106, 0.000233587900337358, -4.99022246243319) 77 | (0.185727670359089, -0.000140152740202415, -4.99310100720702) 78 | (0.156011243101635, 0.0000840916441214488, -4.99513208426237) 79 | (0.131049444205373, -0.0000504549864728693, -4.99656520354329) 80 | (0.110081533132514, 0.0000302729918837216, -4.99757640937974) 81 | (0.0924684878313114, -0.0000181637951302329, -4.99828991509180) 82 | (0.0776735297783016, 0.0000108982770781398, -4.99879336431682) 83 | (0.0652457650137734, -6.53896624688386E-6, -4.99914859794404) 84 | (0.0548064426115696, 3.92337974813032E-6, -4.99939925073887) 85 | (0.0460374117937185, -2.35402784887819E-6, -4.99957611133199) 86 | (0.0386714259067235, 1.41241670932691E-6, -4.99970090415968) 87 | (0.0324839977616478, -8.47450025596149E-7, -4.99978895797645) 88 | (0.0272865581197841, 5.08470015357689E-7, -4.99985108874868) 89 | (0.0229207088206187, -3.05082009214614E-7, -4.99989492822125) 90 | (0.0192533954093197, 1.83049205528768E-7, -4.99992586135298) 91 | (0.0161728521438285, -1.09829523317261E-7, -4.99994768777069) 92 | (0.0135851958008160, 6.58977139903566E-8, -4.99996308849101) 93 | (0.0114115644726854, -3.95386283942140E-8, -4.99997395523926) 94 | (0.00958571415705574, 2.37231770365284E-8, -4.99998162281682) 95 | (0.00805199989192682, -1.42339062219170E-8, -4.99998703305955) 96 | (0.00676367990921853, 8.54034373315022E-9, -4.99999085052682) 97 | (0.00568149112374356, -5.12420623989013E-9, -4.99999354413173) 98 | (0.00477245254394459, 3.07452374393408E-9, -4.99999544473935) 99 | 100 | }; 101 | 102 | \addplot3 [ 103 | color=white, 104 | thick, 105 | mark=*, 106 | mark size=1.5pt, 107 | draw=white, 108 | ] coordinates 109 | { 110 | (5.1, 2.3, 10.78) 111 | (1.40000181372460, -1.39999959782613, -0.688001236486907) 112 | (0.420548264150597, 0.523803523826846, -4.41588756835715) 113 | (0.127254829139659, -0.182733715958336, -4.92997801979625) 114 | (0.0385318995324591, 0.0632032462803674, -4.99171375786294) 115 | (0.0116679060510461, -0.0218380283970381, -4.99901897302514) 116 | (0.00353319693096165, 0.00754456381439479, -4.99988366241760) 117 | (0.00106989950621102, -0.00260644449389353, -4.99998618395722) 118 | }; 119 | 120 | \addplot3 [ 121 | color=gray!30!black!, 122 | mark=*, 123 | mark size=1.5pt, 124 | ] coordinates 125 | { 126 | (5.1, 2.3, 10.78) 127 | 128 | }; 129 | 130 | 131 | \end{axis} 132 | 133 | \end{tikzpicture} 134 | 135 | \end{document} -------------------------------------------------------------------------------- /resources/adam.tex: -------------------------------------------------------------------------------- 1 | \documentclass [border = .2cm] {standalone} 2 | 3 | % Required packages 4 | \usepackage{pgfplots} 5 | \pgfplotsset{compat = newest} 6 | \usetikzlibrary {backgrounds} 7 | 8 | \begin{document} 9 | 10 | \begin{tikzpicture} 11 | [background rectangle/.style= 12 | {bottom color=white!70!gray!}, 13 | show background rectangle] 14 | 15 | \begin{axis}[ 16 | view = {15}{20}, 17 | colormap={blackwhite}{gray(0cm)=(0.5); gray(1cm)=(1)}, 18 | thick, 19 | axis lines = center, 20 | axis on top, 21 | color=black, 22 | xmax = 10, 23 | xmin = -10, 24 | ymax = 3, 25 | ymin = -3, 26 | zmax = 12, 27 | zmin = -6, 28 | xticklabels=\empty, 29 | yticklabels=\empty, 30 | zticklabels=\empty, 31 | xtick=false, 32 | ytick=false, 33 | ztick=false, 34 | ] 35 | 36 | \addplot3 [ 37 | domain=-10:10, 38 | domain y = -3:3, 39 | samples = 50, 40 | samples y = 50, 41 | surf, 42 | mesh, 43 | ultra thin, 44 | shader = interp, 45 | no marks, 46 | ] {0.2 * x^2 + 2 * y^2 - 5}; 47 | 48 | 49 | 50 | \addplot3 [ 51 | color=gray!50!black!, 52 | thin, 53 | mark=*, 54 | mark size=1pt, 55 | draw=gray!50!black!, 56 | ] coordinates 57 | { 58 | (5.1, 2.3, 10.78) 59 | (4.28400000000000, -1.38000000000000, 2.47933120000000) 60 | (3.59856000000000, 0.828000000000001, -1.03890518528000) 61 | (3.02279040000000, -0.496800000000000, -2.67892715953357) 62 | (2.53914393600000, 0.298080000000000, -3.53284624165489) 63 | (2.13288090624000, -0.178848000000000, -4.02619059375137) 64 | (1.79161996124160, 0.107308800000000, -4.33498922578125) 65 | (1.50496076744295, -0.0643852800000001, -4.53872768913016) 66 | (1.26416704465207, 0.0386311680000001, -4.67739160236105) 67 | (1.06190031750774, -0.0231787008000000, -4.77339903879384) 68 | (0.891996266706503, 0.0139072204800000, -4.84048171047338) 69 | (0.749276864033463, -0.00834433228800002, -4.88757758044218) 70 | (0.629392565788109, 0.00500659937280001, -4.92072286755158) 71 | (0.528689755262011, -0.00300395962368001, -4.94407938098936) 72 | (0.444099394420089, 0.00180237577420801, -4.96054864845828) 73 | (0.373043491312875, -0.00108142546452480, -4.97216537175575) 74 | (0.313356532702815, 0.000648855278714882, -4.98036069465615) 75 | (0.263219487470365, -0.000389313167228929, -4.98614279715369) 76 | (0.221104369475106, 0.000233587900337358, -4.99022246243319) 77 | (0.185727670359089, -0.000140152740202415, -4.99310100720702) 78 | (0.156011243101635, 0.0000840916441214488, -4.99513208426237) 79 | (0.131049444205373, -0.0000504549864728693, -4.99656520354329) 80 | (0.110081533132514, 0.0000302729918837216, -4.99757640937974) 81 | (0.0924684878313114, -0.0000181637951302329, -4.99828991509180) 82 | (0.0776735297783016, 0.0000108982770781398, -4.99879336431682) 83 | (0.0652457650137734, -6.53896624688386E-6, -4.99914859794404) 84 | (0.0548064426115696, 3.92337974813032E-6, -4.99939925073887) 85 | (0.0460374117937185, -2.35402784887819E-6, -4.99957611133199) 86 | (0.0386714259067235, 1.41241670932691E-6, -4.99970090415968) 87 | (0.0324839977616478, -8.47450025596149E-7, -4.99978895797645) 88 | (0.0272865581197841, 5.08470015357689E-7, -4.99985108874868) 89 | (0.0229207088206187, -3.05082009214614E-7, -4.99989492822125) 90 | (0.0192533954093197, 1.83049205528768E-7, -4.99992586135298) 91 | (0.0161728521438285, -1.09829523317261E-7, -4.99994768777069) 92 | (0.0135851958008160, 6.58977139903566E-8, -4.99996308849101) 93 | (0.0114115644726854, -3.95386283942140E-8, -4.99997395523926) 94 | (0.00958571415705574, 2.37231770365284E-8, -4.99998162281682) 95 | (0.00805199989192682, -1.42339062219170E-8, -4.99998703305955) 96 | (0.00676367990921853, 8.54034373315022E-9, -4.99999085052682) 97 | (0.00568149112374356, -5.12420623989013E-9, -4.99999354413173) 98 | (0.00477245254394459, 3.07452374393408E-9, -4.99999544473935) 99 | 100 | }; 101 | 102 | \addplot3 [ 103 | color=white, 104 | thick, 105 | mark=*, 106 | mark size=1.5pt, 107 | draw=white, 108 | ] coordinates 109 | { 110 | (5.1, 2.3, 10.78) 111 | (2.10000000000000, -0.700000000000001, -3.13800000000000) 112 | (0.687648728879729, -0.178024002514379, -4.84204275419154) 113 | (0.181964715659077, 0.0833831259894776, -4.97947227705147) 114 | (0.0349012262360075, 0.0115102552579141, -4.99949140892925) 115 | (0.00167514117333054, -0.00921905415838821, -4.99982945686126) 116 | (-0.00253810619313966, -0.000444093945468965, -4.99999831716453) 117 | (-0.00161202680509502, 0.000956709780018702, -4.99999764968671) 118 | (-0.000663033879759380, -0.0000301171398955555, -4.99999991026314) 119 | 120 | }; 121 | 122 | \addplot3 [ 123 | color=gray!30!black!, 124 | mark=*, 125 | mark size=1.5pt, 126 | ] coordinates 127 | { 128 | (5.1, 2.3, 10.78) 129 | 130 | }; 131 | 132 | 133 | \end{axis} 134 | 135 | \end{tikzpicture} 136 | 137 | \end{document} -------------------------------------------------------------------------------- /resources/chaffinch.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/resources/chaffinch.wav -------------------------------------------------------------------------------- /resources/dft-dial.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/skeydan/Deep-Learning-and-Scientific-Computing-with-R-torch/e3c7bfeae59deab61dbb78083665523627399bd9/resources/dft-dial.wav -------------------------------------------------------------------------------- /resources/dimensions.tex: -------------------------------------------------------------------------------- 1 | \documentclass [border = .2cm] {standalone} 2 | 3 | % Required packages 4 | \usepackage{pgfplots} 5 | \pgfplotsset{compat = newest} 6 | \usepackage{tikz-3dplot} 7 | \usetikzlibrary {backgrounds} 8 | 9 | \begin{document} 10 | 11 | \tdplotsetmaincoords{80}{40} 12 | \begin{tikzpicture} 13 | [ 14 | background rectangle/.style={bottom color=white!70!gray!}, 15 | show background rectangle, 16 | tdplot_main_coords, 17 | grid/.style={very thin,gray}, 18 | axis/.style={->,black,thin}, 19 | cube/.style={opacity=.3,thick,fill=gray}] 20 | \foreach \x in {0,1,...,6} 21 | \foreach \y in {0,1,...,6} 22 | { 23 | \draw[grid] (\x,0) -- (\x,6); 24 | \draw[grid] (0,\y) -- (6,\y); 25 | } 26 | 27 | %draw the axes 28 | \draw[axis] (0,0,0) -- (6,0,0) node[anchor=west]{$x$}; 29 | \draw[axis] (0,0,0) -- (0,6,0) node[anchor=west]{$y$}; 30 | \draw[axis] (0,0,0) -- (0,0,4) node[anchor=west]{$z$}; 31 | 32 | %draw the bottom of the cube 33 | \draw[cube] (0,0,0) -- (0,3,0) -- (4,3,0) -- (4,0,0) -- cycle; 34 | 35 | %draw the back-right of the cube 36 | \draw[cube] (0,0,0) -- (0,3,0) -- (0,3,2) -- (0,0,2) -- cycle; 37 | 38 | %draw the back-left of the cube 39 | \draw[cube] (0,0,0) -- (4,0,0) -- (4,0,2) -- (0,0,2) -- cycle; 40 | 41 | 42 | %draw the front-right of the cube 43 | \draw[cube] (4,0,0) -- (4,3,0) -- (4,3,2) -- (4,0,2) -- cycle; 44 | 45 | %draw the front-left of the cube 46 | \draw[cube] (0,3,0) -- (4,3,0) -- (4,3,2) -- (0,3,2) -- cycle; 47 | 48 | %draw the top of the cube 49 | \draw[cube] (0,0,2) -- (0,3,2) -- (4,3,2) -- (4,0,2) -- cycle; 50 | 51 | \end{tikzpicture} 52 | 53 | \end{document} 54 | -------------------------------------------------------------------------------- /resources/momentum.tex: -------------------------------------------------------------------------------- 1 | \documentclass [border = .2cm] {standalone} 2 | 3 | % Required packages 4 | \usepackage{pgfplots} 5 | \pgfplotsset{compat = newest} 6 | \usetikzlibrary {backgrounds} 7 | 8 | \begin{document} 9 | 10 | \begin{tikzpicture} 11 | [background rectangle/.style= 12 | {bottom color=white!70!gray!}, 13 | show background rectangle] 14 | 15 | \begin{axis}[ 16 | view = {15}{20}, 17 | colormap={blackwhite}{gray(0cm)=(0.5); gray(1cm)=(1)}, 18 | thick, 19 | axis lines = center, 20 | axis on top, 21 | color=black, 22 | xmax = 10, 23 | xmin = -10, 24 | ymax = 3, 25 | ymin = -3, 26 | zmax = 12, 27 | zmin = -6, 28 | xticklabels=\empty, 29 | yticklabels=\empty, 30 | zticklabels=\empty, 31 | xtick=false, 32 | ytick=false, 33 | ztick=false, 34 | ] 35 | 36 | \addplot3 [ 37 | domain=-10:10, 38 | domain y = -3:3, 39 | samples = 50, 40 | samples y = 50, 41 | surf, 42 | mesh, 43 | ultra thin, 44 | shader = interp, 45 | no marks, 46 | ] {0.2 * x^2 + 2 * y^2 - 5}; 47 | 48 | 49 | 50 | \addplot3 [ 51 | color=gray!50!black!, 52 | thin, 53 | mark=*, 54 | mark size=1pt, 55 | draw=gray!50!black!, 56 | ] coordinates 57 | { 58 | (5.1, 2.3, 10.78) 59 | (4.28400000000000, -1.38000000000000, 2.47933120000000) 60 | (3.59856000000000, 0.828000000000001, -1.03890518528000) 61 | (3.02279040000000, -0.496800000000000, -2.67892715953357) 62 | (2.53914393600000, 0.298080000000000, -3.53284624165489) 63 | (2.13288090624000, -0.178848000000000, -4.02619059375137) 64 | (1.79161996124160, 0.107308800000000, -4.33498922578125) 65 | (1.50496076744295, -0.0643852800000001, -4.53872768913016) 66 | (1.26416704465207, 0.0386311680000001, -4.67739160236105) 67 | (1.06190031750774, -0.0231787008000000, -4.77339903879384) 68 | (0.891996266706503, 0.0139072204800000, -4.84048171047338) 69 | (0.749276864033463, -0.00834433228800002, -4.88757758044218) 70 | (0.629392565788109, 0.00500659937280001, -4.92072286755158) 71 | (0.528689755262011, -0.00300395962368001, -4.94407938098936) 72 | (0.444099394420089, 0.00180237577420801, -4.96054864845828) 73 | (0.373043491312875, -0.00108142546452480, -4.97216537175575) 74 | (0.313356532702815, 0.000648855278714882, -4.98036069465615) 75 | (0.263219487470365, -0.000389313167228929, -4.98614279715369) 76 | (0.221104369475106, 0.000233587900337358, -4.99022246243319) 77 | (0.185727670359089, -0.000140152740202415, -4.99310100720702) 78 | (0.156011243101635, 0.0000840916441214488, -4.99513208426237) 79 | (0.131049444205373, -0.0000504549864728693, -4.99656520354329) 80 | (0.110081533132514, 0.0000302729918837216, -4.99757640937974) 81 | (0.0924684878313114, -0.0000181637951302329, -4.99828991509180) 82 | (0.0776735297783016, 0.0000108982770781398, -4.99879336431682) 83 | (0.0652457650137734, -6.53896624688386E-6, -4.99914859794404) 84 | (0.0548064426115696, 3.92337974813032E-6, -4.99939925073887) 85 | (0.0460374117937185, -2.35402784887819E-6, -4.99957611133199) 86 | (0.0386714259067235, 1.41241670932691E-6, -4.99970090415968) 87 | (0.0324839977616478, -8.47450025596149E-7, -4.99978895797645) 88 | (0.0272865581197841, 5.08470015357689E-7, -4.99985108874868) 89 | (0.0229207088206187, -3.05082009214614E-7, -4.99989492822125) 90 | (0.0192533954093197, 1.83049205528768E-7, -4.99992586135298) 91 | (0.0161728521438285, -1.09829523317261E-7, -4.99994768777069) 92 | (0.0135851958008160, 6.58977139903566E-8, -4.99996308849101) 93 | (0.0114115644726854, -3.95386283942140E-8, -4.99997395523926) 94 | (0.00958571415705574, 2.37231770365284E-8, -4.99998162281682) 95 | (0.00805199989192682, -1.42339062219170E-8, -4.99998703305955) 96 | (0.00676367990921853, 8.54034373315022E-9, -4.99999085052682) 97 | (0.00568149112374356, -5.12420623989013E-9, -4.99999354413173) 98 | (0.00477245254394459, 3.07452374393408E-9, -4.99999544473935) 99 | 100 | }; 101 | 102 | \addplot3 [ 103 | color=white, 104 | thick, 105 | mark=*, 106 | mark size=1.5pt, 107 | draw=white, 108 | ] coordinates 109 | { 110 | (5.1, 2.3, 10.78) 111 | (4.08000000000000, -2.30000000000000, 8.90928000000001) 112 | (3.16200000000000, 1.84000000000000, 3.77084880000000) 113 | (2.43780000000000, -1.42600000000000, 0.255525767999999) 114 | (1.87782000000000, 1.09940000000000, -1.87739768952000) 115 | (1.44625800000000, -0.846860000000001, -3.14732384028720) 116 | (1.11385020000000, 0.652234000000001, -3.90104916488000) 117 | (0.857839380000001, -0.502324600000000, -4.34816231209393) 118 | (0.660670422000001, 0.386868740000000, -4.61336807472050) 119 | (0.508819441800000, -0.297949406000000, -4.77067285805777) 120 | (0.391870455420000, 0.229467591400000, -4.86397675822796) 121 | (0.301801465698000, -0.176725891660000, -4.91931909349447) 122 | (0.232434273586200, 0.136106543354000, -4.95214487940495) 123 | (0.179010699657780, -0.104823299852600, -4.97161518549764) 124 | (0.137866202333382, 0.0807303155319401, -4.98316381435907) 125 | (0.106178512134266, -0.0621749539934861, -4.99001377490401) 126 | (0.0817740406875011, 0.0478844270409435, -4.99407676454785) 127 | (0.0629787854053244, -0.0368784889375005, -4.99648668862515) 128 | (0.0485035027960418, 0.0284021973396561, -4.99791611241587) 129 | }; 130 | 131 | \addplot3 [ 132 | color=gray!30!black!, 133 | mark=*, 134 | mark size=1.5pt, 135 | ] coordinates 136 | { 137 | (5.1, 2.3, 10.78) 138 | 139 | }; 140 | 141 | 142 | \end{axis} 143 | 144 | \end{tikzpicture} 145 | 146 | \end{document} -------------------------------------------------------------------------------- /resources/paraboloid.tex: -------------------------------------------------------------------------------- 1 | \documentclass [border = .2cm] {standalone} 2 | 3 | % Required packages 4 | \usepackage{pgfplots} 5 | \pgfplotsset{compat = newest} 6 | \usetikzlibrary {backgrounds} 7 | 8 | \begin{document} 9 | 10 | \begin{tikzpicture} 11 | [background rectangle/.style= 12 | {bottom color=white!70!gray!}, 13 | show background rectangle] 14 | 15 | \begin{axis}[ 16 | view = {15}{20}, 17 | colormap={blackwhite}{gray(0cm)=(0.5); gray(1cm)=(1)}, 18 | thick, 19 | axis lines = center, 20 | axis on top, 21 | color=black, 22 | xmax = 10, 23 | xmin = -10, 24 | ymax = 10, 25 | ymin = -10, 26 | zmax = 12, 27 | zmin = -6, 28 | xticklabels=\empty, 29 | yticklabels=\empty, 30 | zticklabels=\empty, 31 | xtick=false, 32 | ytick=false, 33 | ztick=false, 34 | ] 35 | 36 | \addplot3 [ 37 | domain=-10:10, 38 | domain y = -10:10, 39 | samples = 50, 40 | samples y = 50, 41 | surf, 42 | mesh, 43 | ultra thin, 44 | shader = interp, 45 | no marks, 46 | ] {0.2 * x^2 + 0.2 * y^2 - 9.4}; 47 | 48 | 49 | 50 | \coordinate (A) at (-6,6,9.4); 51 | \coordinate (B) at (-5,5,5); 52 | 53 | \draw [fill=white, white] (A) circle (1pt) node [left] {}; 54 | 55 | \draw [-latex, white, thick] (A) -- (B); 56 | 57 | 58 | \end{axis} 59 | 60 | \end{tikzpicture} 61 | 62 | \end{document} -------------------------------------------------------------------------------- /resources/rmsprop.tex: -------------------------------------------------------------------------------- 1 | \documentclass [border = .2cm] {standalone} 2 | 3 | % Required packages 4 | \usepackage{pgfplots} 5 | \pgfplotsset{compat = newest} 6 | \usetikzlibrary {backgrounds} 7 | 8 | \begin{document} 9 | 10 | \begin{tikzpicture} 11 | [background rectangle/.style= 12 | {bottom color=white!70!gray!}, 13 | show background rectangle] 14 | 15 | \begin{axis}[ 16 | view = {15}{20}, 17 | colormap={blackwhite}{gray(0cm)=(0.5); gray(1cm)=(1)}, 18 | thick, 19 | axis lines = center, 20 | axis on top, 21 | color=black, 22 | xmax = 10, 23 | xmin = -10, 24 | ymax = 3, 25 | ymin = -3, 26 | zmax = 12, 27 | zmin = -6, 28 | xticklabels=\empty, 29 | yticklabels=\empty, 30 | zticklabels=\empty, 31 | xtick=false, 32 | ytick=false, 33 | ztick=false, 34 | ] 35 | 36 | \addplot3 [ 37 | domain=-10:10, 38 | domain y = -3:3, 39 | samples = 50, 40 | samples y = 50, 41 | surf, 42 | mesh, 43 | ultra thin, 44 | shader = interp, 45 | no marks, 46 | ] {0.2 * x^2 + 2 * y^2 - 5}; 47 | 48 | 49 | 50 | \addplot3 [ 51 | color=gray!50!black!, 52 | thin, 53 | mark=*, 54 | mark size=1pt, 55 | draw=gray!50!black!, 56 | ] coordinates 57 | { 58 | (5.1, 2.3, 10.78) 59 | (4.28400000000000, -1.38000000000000, 2.47933120000000) 60 | (3.59856000000000, 0.828000000000001, -1.03890518528000) 61 | (3.02279040000000, -0.496800000000000, -2.67892715953357) 62 | (2.53914393600000, 0.298080000000000, -3.53284624165489) 63 | (2.13288090624000, -0.178848000000000, -4.02619059375137) 64 | (1.79161996124160, 0.107308800000000, -4.33498922578125) 65 | (1.50496076744295, -0.0643852800000001, -4.53872768913016) 66 | (1.26416704465207, 0.0386311680000001, -4.67739160236105) 67 | (1.06190031750774, -0.0231787008000000, -4.77339903879384) 68 | (0.891996266706503, 0.0139072204800000, -4.84048171047338) 69 | (0.749276864033463, -0.00834433228800002, -4.88757758044218) 70 | (0.629392565788109, 0.00500659937280001, -4.92072286755158) 71 | (0.528689755262011, -0.00300395962368001, -4.94407938098936) 72 | (0.444099394420089, 0.00180237577420801, -4.96054864845828) 73 | (0.373043491312875, -0.00108142546452480, -4.97216537175575) 74 | (0.313356532702815, 0.000648855278714882, -4.98036069465615) 75 | (0.263219487470365, -0.000389313167228929, -4.98614279715369) 76 | (0.221104369475106, 0.000233587900337358, -4.99022246243319) 77 | (0.185727670359089, -0.000140152740202415, -4.99310100720702) 78 | (0.156011243101635, 0.0000840916441214488, -4.99513208426237) 79 | (0.131049444205373, -0.0000504549864728693, -4.99656520354329) 80 | (0.110081533132514, 0.0000302729918837216, -4.99757640937974) 81 | (0.0924684878313114, -0.0000181637951302329, -4.99828991509180) 82 | (0.0776735297783016, 0.0000108982770781398, -4.99879336431682) 83 | (0.0652457650137734, -6.53896624688386E-6, -4.99914859794404) 84 | (0.0548064426115696, 3.92337974813032E-6, -4.99939925073887) 85 | (0.0460374117937185, -2.35402784887819E-6, -4.99957611133199) 86 | (0.0386714259067235, 1.41241670932691E-6, -4.99970090415968) 87 | (0.0324839977616478, -8.47450025596149E-7, -4.99978895797645) 88 | (0.0272865581197841, 5.08470015357689E-7, -4.99985108874868) 89 | (0.0229207088206187, -3.05082009214614E-7, -4.99989492822125) 90 | (0.0192533954093197, 1.83049205528768E-7, -4.99992586135298) 91 | (0.0161728521438285, -1.09829523317261E-7, -4.99994768777069) 92 | (0.0135851958008160, 6.58977139903566E-8, -4.99996308849101) 93 | (0.0114115644726854, -3.95386283942140E-8, -4.99997395523926) 94 | (0.00958571415705574, 2.37231770365284E-8, -4.99998162281682) 95 | (0.00805199989192682, -1.42339062219170E-8, -4.99998703305955) 96 | (0.00676367990921853, 8.54034373315022E-9, -4.99999085052682) 97 | (0.00568149112374356, -5.12420623989013E-9, -4.99999354413173) 98 | (0.00477245254394459, 3.07452374393408E-9, -4.99999544473935) 99 | 100 | }; 101 | 102 | \addplot3 [ 103 | color=white, 104 | thick, 105 | mark=*, 106 | mark size=1.5pt, 107 | draw=white, 108 | ] coordinates 109 | { 110 | (5.1, 2.3, 10.78) 111 | (0.840000588235007, -0.699999673913080, -3.87888071539614) 112 | (0.323992599577153, 0.257517355868665, -4.84637538193668) 113 | (0.114519640056023, -0.113506237009797, -4.97160971872805) 114 | (0.0364899734793467, 0.0588479371880704, -4.99280753694452) 115 | (0.0102827085373594, -0.0353389971081824, -4.99748116374781) 116 | (0.00249815885326874, 0.0242798816003804, -4.99881972653942) 117 | (0.000504620064207610, -0.00231538033469132, -4.99998922709953) 118 | }; 119 | 120 | \addplot3 [ 121 | color=gray!30!black!, 122 | mark=*, 123 | mark size=1.5pt, 124 | ] coordinates 125 | { 126 | (5.1, 2.3, 10.78) 127 | 128 | }; 129 | 130 | 131 | \end{axis} 132 | 133 | \end{tikzpicture} 134 | 135 | \end{document} -------------------------------------------------------------------------------- /resources/steepest-descent-elliptic.tex: -------------------------------------------------------------------------------- 1 | \documentclass [border = .2cm] {standalone} 2 | 3 | % Required packages 4 | \usepackage{pgfplots} 5 | \pgfplotsset{compat = newest} 6 | \usetikzlibrary {backgrounds} 7 | 8 | \begin{document} 9 | 10 | \begin{tikzpicture} 11 | [background rectangle/.style= 12 | {bottom color=white!70!gray!}, 13 | show background rectangle] 14 | 15 | \begin{axis}[ 16 | view = {15}{20}, 17 | colormap={blackwhite}{gray(0cm)=(0.5); gray(1cm)=(1)}, 18 | thick, 19 | axis lines = center, 20 | axis on top, 21 | color=black, 22 | xmax = 10, 23 | xmin = -10, 24 | ymax = 3, 25 | ymin = -3, 26 | zmax = 12, 27 | zmin = -6, 28 | xticklabels=\empty, 29 | yticklabels=\empty, 30 | zticklabels=\empty, 31 | xtick=false, 32 | ytick=false, 33 | ztick=false, 34 | ] 35 | 36 | \addplot3 [ 37 | domain=-10:10, 38 | domain y = -3:3, 39 | samples = 50, 40 | samples y = 50, 41 | surf, 42 | mesh, 43 | ultra thin, 44 | shader = interp, 45 | no marks, 46 | ] {0.2 * x^2 + 2 * y^2 - 5}; 47 | 48 | 49 | \addplot3 [ 50 | color=white!75!gray!, 51 | thick, 52 | mark=*, 53 | mark size=1pt, 54 | draw=white!75!gray!, 55 | ] coordinates 56 | { 57 | (5.1, 2.3, 10.78) 58 | (4.05960000000000, -2.39200000000000, 9.73939843200001) 59 | (3.23144160000000, 2.48768000000000, 9.46554652764211) 60 | (2.57222751360000, -2.58718720000000, 9.71034609203185) 61 | (2.04749310082560, 2.69067468800000, 10.3179061528735) 62 | (1.62980450825718, -2.79830167552000, 11.1922370814632) 63 | (1.29732438857271, 2.91023374254080, 12.2755309862832) 64 | (1.03267021330388, -3.02664309224243, 13.5344183695267) 65 | (0.822005489789889, 3.14770881593213, 14.9512801848426) 66 | }; 67 | 68 | \addplot3 [ 69 | color=white!65!gray!, 70 | thick, 71 | mark=*, 72 | mark size=1pt, 73 | draw=white!65!gray!, 74 | ] coordinates 75 | { 76 | (5.1, 2.3, 10.78) 77 | (4.09020000000000, -2.25400000000000, 8.50697920800000) 78 | (3.28034040000000, 2.20892000000000, 6.91078176077443) 79 | (2.63083300080000, -2.16474160000000, 5.75646884516078) 80 | (2.10992806664160, 2.12144676800000, 4.89143206819568) 81 | (1.69216230944656, -2.07901783264000, 4.21731295317255) 82 | (1.35711417217615, 2.03743747598720, 3.67065471237845) 83 | (1.08840556608527, -1.99668872646746, 3.21045707606154) 84 | (0.872901264000385, 1.95675495193811, 2.81017120720729) 85 | (0.700066813728309, -1.91761985289935, 2.45255050920415) 86 | (0.561453584610104, 1.87926745584136, 2.12633836670320) 87 | (0.450285774857303, -1.84168210672453, 1.82413742026637) 88 | (0.361129191435557, 1.80484846459004, 1.54103881884742) 89 | (0.289625611531317, -1.76875149529824, 1.27374030321049) 90 | (0.232279740448116, 1.73337646539228, 1.01997871711615) 91 | (0.186288351839389, -1.69870893608443, 0.778164769072387) 92 | (0.149403258175190, 1.66473475736274, 0.547147891453830) 93 | (0.119821413056502, -1.63144006221549, 0.326064787408698) 94 | (0.0960967732713150, 1.59881126097118, 0.114241814383106) 95 | (0.0770696121635946, -1.56683503575175, -0.0888679964578084) 96 | (0.0618098289552029, 1.53549833503672, -0.283725635207852) 97 | (0.0495714828220727, -1.50478836833598, -0.470732466659632) 98 | }; 99 | 100 | \addplot3 [ 101 | color=white, 102 | thick, 103 | mark=*, 104 | mark size=1pt, 105 | draw=white, 106 | ] coordinates 107 | { 108 | (5.1, 2.3, 10.78) 109 | (4.28400000000000, -1.38000000000000, 2.47933120000000) 110 | (3.59856000000000, 0.828000000000001, -1.03890518528000) 111 | (3.02279040000000, -0.496800000000000, -2.67892715953357) 112 | (2.53914393600000, 0.298080000000000, -3.53284624165489) 113 | (2.13288090624000, -0.178848000000000, -4.02619059375137) 114 | (1.79161996124160, 0.107308800000000, -4.33498922578125) 115 | (1.50496076744295, -0.0643852800000001, -4.53872768913016) 116 | (1.26416704465207, 0.0386311680000001, -4.67739160236105) 117 | (1.06190031750774, -0.0231787008000000, -4.77339903879384) 118 | (0.891996266706503, 0.0139072204800000, -4.84048171047338) 119 | (0.749276864033463, -0.00834433228800002, -4.88757758044218) 120 | (0.629392565788109, 0.00500659937280001, -4.92072286755158) 121 | (0.528689755262011, -0.00300395962368001, -4.94407938098936) 122 | (0.444099394420089, 0.00180237577420801, -4.96054864845828) 123 | (0.373043491312875, -0.00108142546452480, -4.97216537175575) 124 | (0.313356532702815, 0.000648855278714882, -4.98036069465615) 125 | (0.263219487470365, -0.000389313167228929, -4.98614279715369) 126 | (0.221104369475106, 0.000233587900337358, -4.99022246243319) 127 | (0.185727670359089, -0.000140152740202415, -4.99310100720702) 128 | (0.156011243101635, 0.0000840916441214488, -4.99513208426237) 129 | (0.131049444205373, -0.0000504549864728693, -4.99656520354329) 130 | (0.110081533132514, 0.0000302729918837216, -4.99757640937974) 131 | (0.0924684878313114, -0.0000181637951302329, -4.99828991509180) 132 | (0.0776735297783016, 0.0000108982770781398, -4.99879336431682) 133 | (0.0652457650137734, -6.53896624688386E-6, -4.99914859794404) 134 | (0.0548064426115696, 3.92337974813032E-6, -4.99939925073887) 135 | (0.0460374117937185, -2.35402784887819E-6, -4.99957611133199) 136 | (0.0386714259067235, 1.41241670932691E-6, -4.99970090415968) 137 | (0.0324839977616478, -8.47450025596149E-7, -4.99978895797645) 138 | (0.0272865581197841, 5.08470015357689E-7, -4.99985108874868) 139 | (0.0229207088206187, -3.05082009214614E-7, -4.99989492822125) 140 | (0.0192533954093197, 1.83049205528768E-7, -4.99992586135298) 141 | (0.0161728521438285, -1.09829523317261E-7, -4.99994768777069) 142 | (0.0135851958008160, 6.58977139903566E-8, -4.99996308849101) 143 | (0.0114115644726854, -3.95386283942140E-8, -4.99997395523926) 144 | (0.00958571415705574, 2.37231770365284E-8, -4.99998162281682) 145 | (0.00805199989192682, -1.42339062219170E-8, -4.99998703305955) 146 | (0.00676367990921853, 8.54034373315022E-9, -4.99999085052682) 147 | (0.00568149112374356, -5.12420623989013E-9, -4.99999354413173) 148 | (0.00477245254394459, 3.07452374393408E-9, -4.99999544473935) 149 | 150 | }; 151 | 152 | \addplot3 [ 153 | color=gray!30!black!, 154 | mark=*, 155 | mark size=1.5pt, 156 | ] coordinates 157 | { 158 | (5.1, 2.3, 10.78) 159 | 160 | }; 161 | 162 | 163 | \end{axis} 164 | 165 | \end{tikzpicture} 166 | 167 | \end{document} -------------------------------------------------------------------------------- /resources/steepest-descent-symmetric.tex: -------------------------------------------------------------------------------- 1 | \documentclass [border = .2cm] {standalone} 2 | 3 | % Required packages 4 | \usepackage{pgfplots} 5 | \pgfplotsset{compat = newest} 6 | \usetikzlibrary {backgrounds} 7 | 8 | \begin{document} 9 | 10 | \begin{tikzpicture} 11 | [background rectangle/.style= 12 | {bottom color=white!70!gray!}, 13 | show background rectangle] 14 | 15 | \begin{axis}[ 16 | view = {15}{20}, 17 | colormap={blackwhite}{gray(0cm)=(0.5); gray(1cm)=(1)}, 18 | thick, 19 | axis lines = center, 20 | axis on top, 21 | color=black, 22 | xmax = 10, 23 | xmin = -10, 24 | ymax = 10, 25 | ymin = -10, 26 | zmax = 12, 27 | zmin = -6, 28 | xticklabels=\empty, 29 | yticklabels=\empty, 30 | zticklabels=\empty, 31 | xtick=false, 32 | ytick=false, 33 | ztick=false, 34 | ] 35 | 36 | \addplot3 [ 37 | domain=-10:10, 38 | domain y = -10:10, 39 | samples = 50, 40 | samples y = 50, 41 | surf, 42 | mesh, 43 | ultra thin, 44 | shader = interp, 45 | no marks, 46 | ] {0.2 * x^2 + 0.2 * y^2 - 5}; 47 | 48 | 49 | \addplot3 [ 50 | color=white!85!gray!, 51 | thick, 52 | mark=*, 53 | mark size=1pt, 54 | draw=white!85!gray!, 55 | ] coordinates 56 | { 57 | (6, 6, 9.4) 58 | (3.60000000000000, 3.60000000000000, 0.183999999999999) 59 | (2.16000000000000, 2.16000000000000, -3.13376000000000) 60 | (1.29600000000000, 1.29600000000000, -4.32815360000000) 61 | (0.777600000000001, 0.777600000000001, -4.75813529600001) 62 | (0.466560000000000, 0.466560000000000, -4.91292870656001) 63 | (0.279936000000000, 0.279936000000000, -4.96865433436160) 64 | (0.167961600000000, 0.167961600000000, -4.98871556037018) 65 | (0.100776960000000, 0.100776960000000, -4.99593760173327) 66 | (0.0604661760000000, 0.0604661760000000, -4.99853753662398) 67 | (0.0362797056000000, 0.0362797056000000, -4.99947351318464) 68 | (0.0217678233600000, 0.0217678233600000, -4.99981046474647) 69 | (0.0130606940160000, 0.0130606940160000, -4.99993176730873) 70 | (0.00783641640960000, 0.00783641640960000, -4.99997543623115) 71 | (0.00470184984576000, 0.00470184984576000, -4.99999115704322) 72 | }; 73 | 74 | \addplot3 [ 75 | color=white, 76 | thick, 77 | mark=*, 78 | mark size=1pt, 79 | draw=white, 80 | ] coordinates 81 | { 82 | (6, 6, 9.4) 83 | (0,0,-5) 84 | }; 85 | 86 | \addplot3 [ 87 | color=gray!30!black!, 88 | mark=*, 89 | mark size=1.5pt, 90 | ] coordinates 91 | { 92 | (6, 6, 9.4) 93 | 94 | }; 95 | 96 | 97 | \end{axis} 98 | 99 | \end{tikzpicture} 100 | 101 | \end{document} -------------------------------------------------------------------------------- /what_is_torch.qmd: -------------------------------------------------------------------------------- 1 | # On `torch`, and how to get it {#sec:basics-torch} 2 | 3 | ## In `torch` world 4 | 5 | `torch` is an R port of PyTorch, one of the two (as of this writing) most-employed deep learning frameworks in industry and research. By its design, it is also an excellent tool to use in various types of scientific computation tasks (a subset of which you'll encounter in the book's final part). It is written entirely in R and C++ (including a bit of C). No Python installation is required to use it. 6 | 7 | On the Python (PyTorch) side, the ecosystem appears as a set of concentric cycles. In the middle, there's PyTorch\index{PyTorch} itself, the core library without which nothing could work. Surrounding it, we have the inner circle of what could be called framework libraries, dedicated to special types of data (images, sound, text ...), or centered on workflow tasks, like deployment. Then, there is the broader ecosystem of add-ons, specializations, and libraries for whom PyTorch is a building block, or a tool. 8 | 9 | On the R side, we have the same "heart" -- all depends on core `torch` -- and we do have the same types of libraries; but the categories, the "circles", appear less clearly set off from each other. There are no strict boundaries. There's just a vibrant community of developers, of diverse origin and with diverse goals, working to further develop and extend `torch`, so it can help more and more people accomplish their various tasks. The ecosystem growing so quickly, I'll refrain from naming individual packages -- at any time, visit [the `torch` website](https://torch.mlverse.org/packages/) to see a featured subset. 10 | 11 | There are three packages, though, that I *will* name here, since they are used in the book: `torchvision` , `torchaudio`, and `luz`. The former two bundle domain-specific transformations, deep learning models, datasets, and utilities for images (incl. video) and audio data, respectively. The third is a high-level, intuitive, nice-to-use interface to `torch`, allowing to define, train, and evaluate a neural network in just a few lines of code. Like `torch` itself, all three packages can be installed from CRAN. 12 | 13 | ## Installing and running torch 14 | 15 | `torch` is available for Windows, MacOS, and Linux. If you have a compatible GPU, and the necessary NVidia software installed, you can benefit from significant speedup, a speedup that will depend on the type of model trained. All examples in this book, though, have been chosen so they can be run on the CPU, without posing taxing demands on your patience. 16 | 17 | Due to their often-transient character, I won't elaborate on compatibility issues here, in the book; analogously, I'll refrain from listing concrete installation\index{installation} instructions. At any time, you'll find up-to-date information in the [vignette](https://cran.r-project.org/web/packages/torch/vignettes/installation.html); and you're more than welcome, should you encounter problems or have questions, to open an issue in the `torch` GitHub repository. 18 | --------------------------------------------------------------------------------