├── .gitignore ├── 00_Installation.ipynb ├── 00_Introduction.ipynb ├── 01_Variables_Control_Packages.ipynb ├── 02_Functions_Types_Dispatch.ipynb ├── 03_Arrays_Parametric_Types.ipynb ├── 04_Getting_Help.ipynb ├── 05_Specialisation_Speed.ipynb ├── 06_Composability_Code_Reuse.ipynb ├── 07_Linear_Algebra_Profiling.ipynb ├── 08_Multithreading_Basics.ipynb ├── 09_Useful_Packages.ipynb ├── LICENSE ├── Manifest.toml ├── Project.toml ├── Projects ├── 01_Dancing_Particles.ipynb ├── 02_Types_Specialisation.ipynb ├── 03_Performance_Engineering.ipynb ├── 04_Solving_Differential_Equations.ipynb ├── 05_Statistical_Learning.ipynb └── Solutions_to_Exercises.ipynb ├── README.md ├── Solutions_to_Exercises.ipynb ├── img └── from_source_to_native.png ├── install-manual.jl └── install.jl /.gitignore: -------------------------------------------------------------------------------- 1 | **/.ipynb_checkpoints 2 | *~ 3 | .*.swp 4 | -------------------------------------------------------------------------------- /00_Installation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "16b28ead", 6 | "metadata": {}, 7 | "source": [ 8 | "# Installation and Setup" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "id": "8829affd", 14 | "metadata": {}, 15 | "source": [ 16 | "## Software and material\n", 17 | "What you need for the workshop (quick overview):\n", 18 | "\n", 19 | "- [Julia 1.8](https://julialang.org/downloads/)\n", 20 | "- [Jupyter](https://jupyter.org/) and [IJulia.jl](https://github.com/JuliaLang/IJulia.jl)\n", 21 | "- This repository of workshop materials\n", 22 | "- All required dependencies (Julia packages) for the workshop\n", 23 | "\n", 24 | "### Getting Julia\n", 25 | "For following the course you will need at least Julia 1.6, but for full compatibility **Julia 1.8** is recommended.\n", 26 | "Julia can be easily obtained in binary form from [Julia downloads](https://julialang.org/downloads/).\n", 27 | "\n", 28 | "### Getting all the rest\n", 29 | "The easiest way to get the remaining files and dependencies\n", 30 | "is to [download the install.jl script](https://github.com/mfherbst/2022-rwth-julia-workshop/blob/master/install.jl) and run it from julia. See the instructions [on the github repository](https://github.com/mfherbst/2022-rwth-julia-workshop).\n", 31 | "\n", 32 | "As an alternative you can also also run the following commands manually\n", 33 | "(this requires to have `git` and `julia` available from the commandline):\n", 34 | "```\n", 35 | "git clone https://github.com/mfherbst/2022-rwth-julia-workshop\n", 36 | "cd 2022-rwth-julia-workshop\n", 37 | "julia install-manual.jl\n", 38 | "```\n", 39 | "\n", 40 | "### Troubleshooting\n", 41 | "If you are facing issues, check out\n", 42 | "the [great troubleshooting section](https://carstenbauer.github.io/WorkshopWizard.jl/dev/troubleshooting/)\n", 43 | "from the WorkshopWizard package by Carsten Bauer (which `install.jl` is using)." 44 | ] 45 | }, 46 | { 47 | "cell_type": "markdown", 48 | "id": "d3d75953", 49 | "metadata": {}, 50 | "source": [ 51 | "## Verifying everything works\n", 52 | "\n", 53 | "A few quick commands to run:" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "id": "7bb04662", 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "using Plots\n", 64 | "\n", 65 | "p = plot(1, xlim=(0, 3π), ylim=(-1.5, 1.5), title=\"Sine\", marker=2)\n", 66 | "N = 100\n", 67 | "@gif for i=1:N\n", 68 | " x = (i-1) * 3π / N\n", 69 | " push!(p, x, sin(x))\n", 70 | "end" 71 | ] 72 | } 73 | ], 74 | "metadata": { 75 | "kernelspec": { 76 | "display_name": "Julia 1.7.2", 77 | "language": "julia", 78 | "name": "julia-1.7" 79 | }, 80 | "language_info": { 81 | "file_extension": ".jl", 82 | "mimetype": "application/julia", 83 | "name": "julia", 84 | "version": "1.7.2" 85 | } 86 | }, 87 | "nbformat": 4, 88 | "nbformat_minor": 5 89 | } 90 | -------------------------------------------------------------------------------- /00_Introduction.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Introduction" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## About this course\n", 15 | "\n", 16 | "With this course I want to provide a condensed, high-level overview of key concepts of the Julia programming language and provide a few practical use cases, which show what the language is capable of.\n", 17 | "\n", 18 | "We have two sessions of three hours and similarly the material is split into two parts as well. In the first session to be held on Thursday I will explain you the basics and key concepts. In the second you will work on smaller projects in groups, which allows you to explore selected topics in more detail. For more details see also the [course website](https://michael-herbst.com/teaching/2022-rwth-julia-workshop/).\n", 19 | "\n", 20 | "In the material I have frequently tried to collect links to the Julia documentation as well as other material, such that you can start exploring on your own. In particular I took a great deal of inspiration from the excellent workshop material of Carsten Bauer, for example [JuliaNRWSS21](https://github.com/carstenbauer/JuliaNRWSS21)." 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "## About Julia\n", 28 | "\n", 29 | "Before we [start digging](01_Variables_Control_Packages.ipynb) in the lovely world of Julia, let me loose a few words about the language itself.\n", 30 | "\n", 31 | "- This monday Julia has [turned 10 years](https://julialang.org/blog/2022/02/10years/).\n", 32 | "- Project started by Jeff Bezanson, Stefan Karpinski, Viral B. Shah, and Alan Edelman over the frustration of the state of scientific software\n", 33 | "\n", 34 | "- What is a good scientific programming language:\n", 35 | " - Fast and scalable\n", 36 | " - Interactive\n", 37 | " - Easy to write and read\n", 38 | " - Vanishing division between users and developers\n", 39 | "\n", 40 | "- But there are already so many languages, so why Julia?\n", 41 | "\n", 42 | "- Traditional divide in languages:\n", 43 | " - Compiled / interpreted\n", 44 | " - Dynamic / static typing\n", 45 | " - Prototyping / production\n", 46 | " - Users / developers\n", 47 | " \n", 48 | "- Two language problem\n", 49 | " - [language share of numpy](https://github.com/numpy/numpy)\n", 50 | " - [language share of Julia](https://github.com/JuliaLang/julia)\n", 51 | "\n", 52 | "### The point of Julia\n", 53 | "\n", 54 | "- **Walks like Python, talks like Lisp, runs like FORTRAN**\n", 55 | " - Development dynamic like writing python code\n", 56 | " - Less object-oriented more function-oriented (like Lisp)\n", 57 | " - Same speed as FORTRAN or C code (Petaflop club)\n", 58 | "- *Both* fast and high-level!\n", 59 | "- Extremely good ratio between coding time and running time.\n", 60 | "\n", 61 | "- Key concepts:\n", 62 | " - **Just-in-time compilation** (JIT)\n", 63 | " - Compilation down to *native code* (running on CPU and GPU, ...)\n", 64 | " - Strong type system to encode exploitable properties (e.g. symmetries, sparsities, storage location etc.)\n", 65 | " - Multiple dispatch and strong composability of packages (Details later)\n", 66 | " - Facilitates code reuse and features for free.\n", 67 | "\n", 68 | "- Additional value:\n", 69 | " - **Multi-platform** (x86, ARM, PowerPC, Apple's M1)\n", 70 | " - **Cross-language support:** Easily use C++, FORTRAN, python, R, Java ... from Julia (and in some cases the other way round, too)\n", 71 | " - **Builtin package manager**\n", 72 | " - **Unicode support** inside the core language\n", 73 | " - **Green threading** and **distributed computing**\n", 74 | " - **Macros** are part of the language\n", 75 | " - **Rich set of packages**: GPU, Threading, Distributed computing, Plotting, Visualisation, Optimisation, Statistics, Data science, Automatic differentiation, Machine Learning, PDEs ...\n", 76 | "\n", 77 | "- Established language and community\n", 78 | " - Global community of users and developers\n", 79 | " - Used in over 30 universities for teaching\n", 80 | " - Multiply awarded: James H. Wilkinson Prize, 30 under 30 (Keno Fisher), IEEE Babbage Prize (Alan Edelman)" 81 | ] 82 | } 83 | ], 84 | "metadata": { 85 | "kernelspec": { 86 | "display_name": "Julia 1.7.2", 87 | "language": "julia", 88 | "name": "julia-1.7" 89 | }, 90 | "language_info": { 91 | "file_extension": ".jl", 92 | "mimetype": "application/julia", 93 | "name": "julia", 94 | "version": "1.7.2" 95 | } 96 | }, 97 | "nbformat": 4, 98 | "nbformat_minor": 2 99 | } 100 | -------------------------------------------------------------------------------- /01_Variables_Control_Packages.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Variables, Control Structures & Packages" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Variables and elementary types\n", 15 | "\n", 16 | "Defining variables in Julia works like in most languages:" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": null, 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "int = 4 # An integer" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": null, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "str = \"Hi\" # A string" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": {}, 41 | "outputs": [], 42 | "source": [ 43 | "float = 1.2 # A floating-point number" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "bool = true # A boolean (also false)" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "The type is automatically inferred:" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "typeof(int)" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "typeof(str)" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": null, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "typeof(float)" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "Julia supports a large range of integer and floating-point types out of the box, for example:" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": {}, 100 | "outputs": [], 101 | "source": [ 102 | "x = UInt8(1) # 8-bit wide unsigned integer\n", 103 | "y = Int32(-1) # 32-bit wide signed integer\n", 104 | "z = Float32(0.2) # single precision\n", 105 | "α = Float16(-1.0) # half precision\n", 106 | "β = ComplexF64(2. + 8im) # Complex number (composed of two Float64)" 107 | ] 108 | }, 109 | { 110 | "cell_type": "markdown", 111 | "metadata": {}, 112 | "source": [ 113 | "**Operations** between these types may involve **automatic promotion**." 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": null, 119 | "metadata": {}, 120 | "outputs": [], 121 | "source": [ 122 | "println(typeof(1 + 1.2))\n", 123 | "println(typeof(x * y)) # x was Uint8, y was Int32" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": null, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "x = Float32(1.)\n", 133 | "4x # No \"*\" needed" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": null, 139 | "metadata": {}, 140 | "outputs": [], 141 | "source": [ 142 | "println(5 % 2) # Remainder operator" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": null, 148 | "metadata": {}, 149 | "outputs": [], 150 | "source": [ 151 | "5^6" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": null, 157 | "metadata": {}, 158 | "outputs": [], 159 | "source": [ 160 | "2.3^6" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": null, 166 | "metadata": {}, 167 | "outputs": [], 168 | "source": [ 169 | "println(4 - Float32(2))\n", 170 | "println(8 / 3) # Note conversion to Float64\n", 171 | "typeof(8 / 3)" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "metadata": {}, 177 | "source": [ 178 | "This can be avoided using a rational:" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": null, 184 | "metadata": {}, 185 | "outputs": [], 186 | "source": [ 187 | "x = 6 // 3\n", 188 | "println(typeof(x), \" \", Int(x))" 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": null, 194 | "metadata": {}, 195 | "outputs": [], 196 | "source": [ 197 | "√2 # == sqrt(2)" 198 | ] 199 | }, 200 | { 201 | "cell_type": "markdown", 202 | "metadata": {}, 203 | "source": [ 204 | "Notice the `*` operation for **string concatenation** ..." 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": null, 210 | "metadata": {}, 211 | "outputs": [], 212 | "source": [ 213 | "\"Concatenating \" * \"strings is done \" * \"using multiplication\"" 214 | ] 215 | }, 216 | { 217 | "cell_type": "markdown", 218 | "metadata": {}, 219 | "source": [ 220 | "**Type conversion** is possible, for example using constructors:" 221 | ] 222 | }, 223 | { 224 | "cell_type": "code", 225 | "execution_count": null, 226 | "metadata": {}, 227 | "outputs": [], 228 | "source": [ 229 | "Int(2.0)" 230 | ] 231 | }, 232 | { 233 | "cell_type": "code", 234 | "execution_count": null, 235 | "metadata": {}, 236 | "outputs": [], 237 | "source": [ 238 | "x = Float32(1.0)\n", 239 | "Float64(x)" 240 | ] 241 | }, 242 | { 243 | "cell_type": "code", 244 | "execution_count": null, 245 | "metadata": {}, 246 | "outputs": [], 247 | "source": [ 248 | "Int(2.3)" 249 | ] 250 | }, 251 | { 252 | "cell_type": "code", 253 | "execution_count": null, 254 | "metadata": {}, 255 | "outputs": [], 256 | "source": [ 257 | "UInt8(-1)" 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": null, 263 | "metadata": {}, 264 | "outputs": [], 265 | "source": [ 266 | "Float64(2.0 + 1im)" 267 | ] 268 | }, 269 | { 270 | "cell_type": "markdown", 271 | "metadata": {}, 272 | "source": [ 273 | "Sometimes a special conversion function or syntax exists:" 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "execution_count": null, 279 | "metadata": {}, 280 | "outputs": [], 281 | "source": [ 282 | "string(1.2)" 283 | ] 284 | }, 285 | { 286 | "cell_type": "code", 287 | "execution_count": null, 288 | "metadata": {}, 289 | "outputs": [], 290 | "source": [ 291 | "ceil(Int, 2.3)" 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "execution_count": null, 297 | "metadata": {}, 298 | "outputs": [], 299 | "source": [ 300 | "round(Int32, 4.5)" 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "execution_count": null, 306 | "metadata": {}, 307 | "outputs": [], 308 | "source": [ 309 | "x = 1.2\n", 310 | "println(\"x is currently $x.\")\n", 311 | "println(\"two less is $(x - 2)\")" 312 | ] 313 | }, 314 | { 315 | "cell_type": "markdown", 316 | "metadata": {}, 317 | "source": [ 318 | "**Arbitrary precision** is supported out of the box:" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": null, 324 | "metadata": {}, 325 | "outputs": [], 326 | "source": [ 327 | "# arbitrary precision\n", 328 | "b = big\"12\"; println(b, \" is a \", typeof(b))\n", 329 | "f = big\"-1.2\"; println(f, \" is a \", typeof(f))" 330 | ] 331 | }, 332 | { 333 | "cell_type": "code", 334 | "execution_count": null, 335 | "metadata": {}, 336 | "outputs": [], 337 | "source": [ 338 | "# Or wir an explicit cast:\n", 339 | "b = BigInt(12)\n", 340 | "f = BigFloat(-1.2) # Note the difference to the above result" 341 | ] 342 | }, 343 | { 344 | "cell_type": "markdown", 345 | "metadata": {}, 346 | "source": [ 347 | "Julia is garbage-collected, meaning that memory occupied by a variable value will be freed automatically if no longer needed, e.g.:" 348 | ] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": null, 353 | "metadata": {}, 354 | "outputs": [], 355 | "source": [ 356 | "x = 15\n", 357 | "println(x)\n", 358 | "x = \"abc\"; # x newly assigned, memory by the \"15\" will be freed at some point in the future" 359 | ] 360 | }, 361 | { 362 | "cell_type": "markdown", 363 | "metadata": {}, 364 | "source": [ 365 | "### Exercise\n", 366 | "Compute $2^{100}$:\n", 367 | "- As fast as possible\n", 368 | "- As exact as possible" 369 | ] 370 | }, 371 | { 372 | "cell_type": "markdown", 373 | "metadata": {}, 374 | "source": [ 375 | "##### For more details\n", 376 | "- https://docs.julialang.org/en/v1/manual/integers-and-floating-point-numbers/\n", 377 | "- https://docs.julialang.org/en/v1/manual/mathematical-operations/\n", 378 | "- https://docs.julialang.org/en/v1/manual/strings/\n", 379 | "- https://docs.julialang.org/en/v1/manual/conversion-and-promotion/" 380 | ] 381 | }, 382 | { 383 | "cell_type": "markdown", 384 | "metadata": {}, 385 | "source": [ 386 | "## Basic control structures\n", 387 | "\n", 388 | "Unsurprisingly Julia has the standard `for` and `while` loops and the `if` conditionals." 389 | ] 390 | }, 391 | { 392 | "cell_type": "code", 393 | "execution_count": null, 394 | "metadata": {}, 395 | "outputs": [], 396 | "source": [ 397 | "x = 3; y = 5\n", 398 | "\n", 399 | "if x < y\n", 400 | " println(\"x is less than y\")\n", 401 | "elseif x > y\n", 402 | " println(\"x is greater than y\")\n", 403 | "else\n", 404 | " println(\"x is equal to y\")\n", 405 | "end" 406 | ] 407 | }, 408 | { 409 | "cell_type": "code", 410 | "execution_count": null, 411 | "metadata": {}, 412 | "outputs": [], 413 | "source": [ 414 | "x = 5\n", 415 | "while x > 3\n", 416 | " println(\"Now x is $x\")\n", 417 | " x -= 1\n", 418 | "end" 419 | ] 420 | }, 421 | { 422 | "cell_type": "code", 423 | "execution_count": null, 424 | "metadata": {}, 425 | "outputs": [], 426 | "source": [ 427 | "for i in 1:5\n", 428 | " println(\"Hello from number $i\")\n", 429 | "end" 430 | ] 431 | }, 432 | { 433 | "cell_type": "markdown", 434 | "metadata": {}, 435 | "source": [ 436 | "Where the `1:5` is a `Range`-object, but it could be any iterable. There are a few syntax variations, including:" 437 | ] 438 | }, 439 | { 440 | "cell_type": "code", 441 | "execution_count": null, 442 | "metadata": {}, 443 | "outputs": [], 444 | "source": [ 445 | "accu = 0\n", 446 | "# To get ∈ use \\in\n", 447 | "for j ∈ 1:0.5:3 # Note the step parameter\n", 448 | " accu += j\n", 449 | "end\n", 450 | "println(accu)" 451 | ] 452 | }, 453 | { 454 | "cell_type": "markdown", 455 | "metadata": {}, 456 | "source": [ 457 | "Worth mentioning are also `||`, `&&` and `?:`" 458 | ] 459 | }, 460 | { 461 | "cell_type": "code", 462 | "execution_count": null, 463 | "metadata": {}, 464 | "outputs": [], 465 | "source": [ 466 | "true || println(\"The RHS of || is only run\")\n", 467 | "false || println(\"if the LHS is false\")" 468 | ] 469 | }, 470 | { 471 | "cell_type": "code", 472 | "execution_count": null, 473 | "metadata": {}, 474 | "outputs": [], 475 | "source": [ 476 | "iseven(3) && println(\"The RHS of || is only run\")\n", 477 | "isodd(3) && println(\"if the LHS is true\")" 478 | ] 479 | }, 480 | { 481 | "cell_type": "code", 482 | "execution_count": null, 483 | "metadata": {}, 484 | "outputs": [], 485 | "source": [ 486 | "x = 3\n", 487 | "x < 5 ? \"smaller than 5\" : \"larger or equal 5\"" 488 | ] 489 | }, 490 | { 491 | "cell_type": "markdown", 492 | "metadata": {}, 493 | "source": [ 494 | "Generally expressions can be made one block using `begin` and `end` or `(` and `)`. The last statement determines the returned value." 495 | ] 496 | }, 497 | { 498 | "cell_type": "code", 499 | "execution_count": null, 500 | "metadata": {}, 501 | "outputs": [], 502 | "source": [ 503 | "z = begin\n", 504 | " x = 5\n", 505 | " y = -3\n", 506 | " x + y\n", 507 | "end" 508 | ] 509 | }, 510 | { 511 | "cell_type": "code", 512 | "execution_count": null, 513 | "metadata": {}, 514 | "outputs": [], 515 | "source": [ 516 | "# Or equivalently\n", 517 | "z = (x = 5; y = -3; x + y)" 518 | ] 519 | }, 520 | { 521 | "cell_type": "markdown", 522 | "metadata": {}, 523 | "source": [ 524 | "### Exercise\n", 525 | "Compute\n", 526 | "$$ 15! \\qquad 100! \\qquad \\left(\\begin{array}{c} 100 \\\\ 15 \\end{array}\\right) $$\n", 527 | "with the Julia you know so far." 528 | ] 529 | }, 530 | { 531 | "cell_type": "markdown", 532 | "metadata": {}, 533 | "source": [ 534 | "##### For more details\n", 535 | "- https://docs.julialang.org/en/v1/manual/control-flow/" 536 | ] 537 | }, 538 | { 539 | "cell_type": "markdown", 540 | "metadata": {}, 541 | "source": [ 542 | "## Julia packages\n", 543 | "\n", 544 | "The Julia standard library is already surprisingly rich in functionality. Features are grouped into smaller packages according to topics (e.g. `LinearAlgebra`, `SparseArrays`, `Test`, ...). The most basic part of the standard library resides in the special package `Base` and is always automatically to the programmer. If anything beyond `Base` is needed, that is both packages of the standard library or any other third-party package, this needs to be explicitly **installed** and **imported** before use." 545 | ] 546 | }, 547 | { 548 | "cell_type": "markdown", 549 | "metadata": {}, 550 | "source": [ 551 | "For example. The `randn` function produces a list or array of random numbers. It is already available in `Base` and we can thus directly use it to create 4 random numbers:" 552 | ] 553 | }, 554 | { 555 | "cell_type": "code", 556 | "execution_count": null, 557 | "metadata": {}, 558 | "outputs": [], 559 | "source": [ 560 | "d = randn(4)" 561 | ] 562 | }, 563 | { 564 | "cell_type": "markdown", 565 | "metadata": {}, 566 | "source": [ 567 | "The returned object in `d` is an `Array`, a type from `Base`, which we will discuss later. We wish to create a diagonal Matrix with these four vaules on the diagonal. For this we need the `LinearAlgebra.Diagonal` datastructure. We make `LinearAlgebra` available to our scope with a `using` statement:" 568 | ] 569 | }, 570 | { 571 | "cell_type": "code", 572 | "execution_count": null, 573 | "metadata": {}, 574 | "outputs": [], 575 | "source": [ 576 | "using LinearAlgebra" 577 | ] 578 | }, 579 | { 580 | "cell_type": "markdown", 581 | "metadata": {}, 582 | "source": [ 583 | "This (might) cause an error, because `LinearAlgebra` is not yet installed. We resolve this as recommended:" 584 | ] 585 | }, 586 | { 587 | "cell_type": "code", 588 | "execution_count": null, 589 | "metadata": {}, 590 | "outputs": [], 591 | "source": [ 592 | "import Pkg; Pkg.add(\"LinearAlgebra\")" 593 | ] 594 | }, 595 | { 596 | "cell_type": "markdown", 597 | "metadata": {}, 598 | "source": [ 599 | "Note, that `import` is roughly the same as `using` except the package is not brought into scope. We will neglect this detail for this course and just use `using` for all packages we use." 600 | ] 601 | }, 602 | { 603 | "cell_type": "markdown", 604 | "metadata": {}, 605 | "source": [ 606 | "Having installed the package, we are able to use LinearAlgebra and make the diagonal matrix:" 607 | ] 608 | }, 609 | { 610 | "cell_type": "code", 611 | "execution_count": null, 612 | "metadata": {}, 613 | "outputs": [], 614 | "source": [ 615 | "using LinearAlgebra\n", 616 | "Diagonal(d)" 617 | ] 618 | }, 619 | { 620 | "cell_type": "markdown", 621 | "metadata": {}, 622 | "source": [ 623 | "## The Julia REPL\n", 624 | "\n", 625 | "Apart from interacting with Julia via a webbrowser and Jupyter (as we will do it in this course). There is also the alternative option to use the Julia REPL. You can get to the REPL by starting `julia` from a terminal. The result is a prompt similar to\n", 626 | "```\n", 627 | " _\n", 628 | " _ _ _(_)_ | Documentation: https://docs.julialang.org\n", 629 | " (_) | (_) (_) |\n", 630 | " _ _ _| |_ __ _ | Type \"?\" for help, \"]?\" for Pkg help.\n", 631 | " | | | | | | |/ _` | |\n", 632 | " | | |_| | | | (_| | | Version 1.7.2 (2022-02-06)\n", 633 | " _/ |\\__'_|_|_|\\__'_| | Official https://julialang.org/ release\n", 634 | "|__/ |\n", 635 | "\n", 636 | "julia> \n", 637 | "\n", 638 | "```" 639 | ] 640 | }, 641 | { 642 | "cell_type": "markdown", 643 | "metadata": {}, 644 | "source": [ 645 | "In this prompt you can directly start typing julia expressions, which will be executed upon `[Enter]`. The REPL is very nice to prototype things and with some optional customisations becomes a very powerful tool. We will not use the REPL much in this course, but still I want to mention some basic aspects to get you started. Mostly this is for people who prefer the commandline over graphics (like me):\n", 646 | "\n", 647 | "- Exiting the shell can be achieved using `exit()` or Ctrl + D.\n", 648 | "- Inside the shell tab completion is available.\n", 649 | "- The REPL comes in multiple modes. The most useful ones are the (default) Julian mode, Pkg mode (for installing packages), shell mode (for running one-shot shell commands) and help mode. You can get from Julian mode to another by typing a magic key, see the list below. You get back to Julian mode by `[Backspace]` on an empty prompt.\n", 650 | "\n", 651 | "| Magic key | Prompt | Mode |\n", 652 | "|---------------|-------------------------|------------------------------|\n", 653 | "| `[Backspace]` | `julia>` | Julian mode (Command mode) |\n", 654 | "| `]` | `(v1.7) pkg>` | Pkg mode |\n", 655 | "| `?` | `help>` | Help mode |\n", 656 | "| `;` | `shell>` | Shell command mode |\n", 657 | "\n", 658 | "\n", 659 | "- To understand Pkg mode better, type `help` once the `(v1.7) pkg>` prompt shows.\n", 660 | "- In each mode Ctrl + R allows to search backwards in history for a previously entered command." 661 | ] 662 | }, 663 | { 664 | "cell_type": "markdown", 665 | "metadata": {}, 666 | "source": [ 667 | "##### For more details\n", 668 | "- https://jupyter-notebook.readthedocs.io/en/stable/\n", 669 | "- https://docs.julialang.org/en/v1/stdlib/Pkg/\n", 670 | "- https://docs.julialang.org/en/v1/stdlib/REPL/" 671 | ] 672 | }, 673 | { 674 | "cell_type": "markdown", 675 | "metadata": {}, 676 | "source": [ 677 | "## Takaways\n", 678 | "- Overview of basic types in Julia\n", 679 | "- Control structures `if`, `for`, `while`\n", 680 | "- Installing packages" 681 | ] 682 | } 683 | ], 684 | "metadata": { 685 | "@webio": { 686 | "lastCommId": null, 687 | "lastKernelId": null 688 | }, 689 | "kernelspec": { 690 | "display_name": "Julia 1.7.2", 691 | "language": "julia", 692 | "name": "julia-1.7" 693 | }, 694 | "language_info": { 695 | "file_extension": ".jl", 696 | "mimetype": "application/julia", 697 | "name": "julia", 698 | "version": "1.7.2" 699 | } 700 | }, 701 | "nbformat": 4, 702 | "nbformat_minor": 2 703 | } 704 | -------------------------------------------------------------------------------- /03_Arrays_Parametric_Types.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Parametric types and arrays\n", 8 | "\n", 9 | "Arrays are a good example to get some idea of the Julian way of defining interfaces." 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## Parametric types\n", 17 | "\n", 18 | "Defining arrays is rather intuitive, for example:" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "i = [1, 2, 3, 4] # An integer array" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": null, 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "f = [1.2, 3.4, 21, π] # A float array ... note the implicit conversion" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "s = [\"abc\", \"def\", \"ghi\"] # A string array" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "Notice, that the type of arrays is `Array{T, N}`, where `T` is a type and `N` is the number of dimensions. This type is an example of a **parametric type**, i.e. a type, which itself is parametrised by other values or types. For convenience:" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "Vector{String} === Array{String, 1}" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "Arrays do not need to be of one type only, for example ..." 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "m = [1, 3.4, \"abc\"] # A mixed array" 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": {}, 83 | "source": [ 84 | "and we can explicitly fix their types" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": null, 90 | "metadata": {}, 91 | "outputs": [], 92 | "source": [ 93 | "m = Float32[1, 3.4, 4]" 94 | ] 95 | }, 96 | { 97 | "cell_type": "markdown", 98 | "metadata": {}, 99 | "source": [ 100 | "Comprehensions also work in Julia:\n" 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": null, 106 | "metadata": {}, 107 | "outputs": [], 108 | "source": [ 109 | "arr = randn(10) # Array of 10 random values\n", 110 | "[e for e in arr if e > 0]" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "Another example of a parametric type is the `Tuple`." 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "(1, 2.0, \"3\")" 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "metadata": {}, 133 | "outputs": [], 134 | "source": [ 135 | "typeof((1, 2.0, \"3\"))" 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "metadata": {}, 141 | "source": [ 142 | "### Type parameters in function signatures" 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "metadata": {}, 148 | "source": [ 149 | "Since `Vector{Real} === Array{Real,1}` one would naively define" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": null, 155 | "metadata": {}, 156 | "outputs": [], 157 | "source": [ 158 | "myfunc(v::Vector{Real}) = \"Got a real vector!\"" 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": {}, 164 | "source": [ 165 | "However ..." 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": null, 171 | "metadata": {}, 172 | "outputs": [], 173 | "source": [ 174 | "myfunc([1.0, 2.0, 3.0])" 175 | ] 176 | }, 177 | { 178 | "cell_type": "markdown", 179 | "metadata": {}, 180 | "source": [ 181 | "Why is this?" 182 | ] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "metadata": {}, 187 | "source": [ 188 | "Note although we have" 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": null, 194 | "metadata": {}, 195 | "outputs": [], 196 | "source": [ 197 | "Float64 <: Real" 198 | ] 199 | }, 200 | { 201 | "cell_type": "markdown", 202 | "metadata": {}, 203 | "source": [ 204 | "parametric types have the following (perhaps somewhat counterintuitive) property" 205 | ] 206 | }, 207 | { 208 | "cell_type": "code", 209 | "execution_count": null, 210 | "metadata": {}, 211 | "outputs": [], 212 | "source": [ 213 | "Vector{Float64} <: Vector{Real}" 214 | ] 215 | }, 216 | { 217 | "cell_type": "code", 218 | "execution_count": null, 219 | "metadata": {}, 220 | "outputs": [], 221 | "source": [ 222 | "[1.0, 2.0, 3.0] isa Vector{Real}" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": {}, 228 | "source": [ 229 | "How can we understand the behavior above? The crucial point is that `Vector{Real}` is a **concrete** container type despite the fact that `Real` is an abstract type. Specifically, it describes a **heterogeneous** vector of values that individually can be of any type `T <: Real`." 230 | ] 231 | }, 232 | { 233 | "cell_type": "code", 234 | "execution_count": null, 235 | "metadata": {}, 236 | "outputs": [], 237 | "source": [ 238 | "isconcretetype(Vector{Real})" 239 | ] 240 | }, 241 | { 242 | "cell_type": "code", 243 | "execution_count": null, 244 | "metadata": {}, 245 | "outputs": [], 246 | "source": [ 247 | "[1.0, 2.0]" 248 | ] 249 | }, 250 | { 251 | "cell_type": "code", 252 | "execution_count": null, 253 | "metadata": {}, 254 | "outputs": [], 255 | "source": [ 256 | "Real[1, 2.2, 13f0]" 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": {}, 262 | "source": [ 263 | "As we have learned above, concrete types are the leafes of the type tree and **cannot** have any subtypes. Hence it is only consistent to have..." 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": null, 269 | "metadata": {}, 270 | "outputs": [], 271 | "source": [ 272 | "Vector{Float64} <: Vector{Real}" 273 | ] 274 | }, 275 | { 276 | "cell_type": "markdown", 277 | "metadata": {}, 278 | "source": [ 279 | "What we often actually *mean* when writing `myfunc(v::Vector{Real}) = ...` is" 280 | ] 281 | }, 282 | { 283 | "cell_type": "code", 284 | "execution_count": null, 285 | "metadata": {}, 286 | "outputs": [], 287 | "source": [ 288 | "myfunc(v::Vector{T}) where T<:Real = \"I'm a real vector!\"" 289 | ] 290 | }, 291 | { 292 | "cell_type": "code", 293 | "execution_count": null, 294 | "metadata": {}, 295 | "outputs": [], 296 | "source": [ 297 | "myfunc([1.0, 2.0, 3.0])" 298 | ] 299 | }, 300 | { 301 | "cell_type": "markdown", 302 | "metadata": {}, 303 | "source": [ 304 | "It works! But what does it mean exactly? First of all, we see that" 305 | ] 306 | }, 307 | { 308 | "cell_type": "code", 309 | "execution_count": null, 310 | "metadata": {}, 311 | "outputs": [], 312 | "source": [ 313 | "Vector{Float64} <: Vector{T} where T<:Real" 314 | ] 315 | }, 316 | { 317 | "cell_type": "markdown", 318 | "metadata": {}, 319 | "source": [ 320 | "Here, `Vector{T} where T <: Real` describes the **set** of concrete `Vector` types whose elements are of any specific single type `T` that is a subtype of `Real`.\n", 321 | "\n", 322 | "Think of it as representing `{{ Vector{Float64}, Vector{Int64}, Vector{Int32}, Vector{AbstractFloat}, ... }}`." 323 | ] 324 | }, 325 | { 326 | "cell_type": "code", 327 | "execution_count": null, 328 | "metadata": {}, 329 | "outputs": [], 330 | "source": [ 331 | "Vector{Int64} <: Vector{T} where T<:Real" 332 | ] 333 | }, 334 | { 335 | "cell_type": "code", 336 | "execution_count": null, 337 | "metadata": {}, 338 | "outputs": [], 339 | "source": [ 340 | "Vector{AbstractFloat} <: Vector{T} where T<:Real" 341 | ] 342 | }, 343 | { 344 | "cell_type": "code", 345 | "execution_count": null, 346 | "metadata": {}, 347 | "outputs": [], 348 | "source": [ 349 | "[1.0, 2.0, 3.0] isa Vector{T} where T<:Real" 350 | ] 351 | }, 352 | { 353 | "cell_type": "markdown", 354 | "metadata": {}, 355 | "source": [ 356 | "We can also use the `where` notation to write out our naive `Vector{Real}` from above in a more explicit way:" 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": null, 362 | "metadata": {}, 363 | "outputs": [], 364 | "source": [ 365 | "Vector{Real} === Vector{T where T<:Real}" 366 | ] 367 | }, 368 | { 369 | "cell_type": "markdown", 370 | "metadata": {}, 371 | "source": [ 372 | "Note that the crucial difference is the position of the `where T<:Real` piece, i.e. whether it is inside or outside of the curly braces." 373 | ] 374 | }, 375 | { 376 | "cell_type": "code", 377 | "execution_count": null, 378 | "metadata": {}, 379 | "outputs": [], 380 | "source": [ 381 | "Vector{T where T<:Real} <: Vector{T} where T<:Real" 382 | ] 383 | }, 384 | { 385 | "cell_type": "code", 386 | "execution_count": null, 387 | "metadata": {}, 388 | "outputs": [], 389 | "source": [ 390 | "(Vector{T} where T<:Real) <: Vector{T where T<:Real}" 391 | ] 392 | }, 393 | { 394 | "cell_type": "markdown", 395 | "metadata": {}, 396 | "source": [ 397 | "## Basic functions for arrays" 398 | ] 399 | }, 400 | { 401 | "cell_type": "code", 402 | "execution_count": null, 403 | "metadata": {}, 404 | "outputs": [], 405 | "source": [ 406 | "A = randn(Float32, 4, 5)" 407 | ] 408 | }, 409 | { 410 | "cell_type": "code", 411 | "execution_count": null, 412 | "metadata": {}, 413 | "outputs": [], 414 | "source": [ 415 | "ndims(A) # Get the number of dimensions" 416 | ] 417 | }, 418 | { 419 | "cell_type": "code", 420 | "execution_count": null, 421 | "metadata": {}, 422 | "outputs": [], 423 | "source": [ 424 | "eltype(A) # Get the type of the array elements" 425 | ] 426 | }, 427 | { 428 | "cell_type": "code", 429 | "execution_count": null, 430 | "metadata": {}, 431 | "outputs": [], 432 | "source": [ 433 | "length(A) # Return the number of elements" 434 | ] 435 | }, 436 | { 437 | "cell_type": "code", 438 | "execution_count": null, 439 | "metadata": {}, 440 | "outputs": [], 441 | "source": [ 442 | "size(A) # Get the size of the array" 443 | ] 444 | }, 445 | { 446 | "cell_type": "code", 447 | "execution_count": null, 448 | "metadata": {}, 449 | "outputs": [], 450 | "source": [ 451 | "size(A, 1) # Get the size along an axis" 452 | ] 453 | }, 454 | { 455 | "cell_type": "code", 456 | "execution_count": null, 457 | "metadata": {}, 458 | "outputs": [], 459 | "source": [ 460 | "reshape(A, 2, 5, 2) # Return an array with the shape changed" 461 | ] 462 | }, 463 | { 464 | "cell_type": "code", 465 | "execution_count": null, 466 | "metadata": {}, 467 | "outputs": [], 468 | "source": [ 469 | "A[:, 1] # Access first column" 470 | ] 471 | }, 472 | { 473 | "cell_type": "code", 474 | "execution_count": null, 475 | "metadata": {}, 476 | "outputs": [], 477 | "source": [ 478 | "A[2, :] # Access second row" 479 | ] 480 | }, 481 | { 482 | "cell_type": "markdown", 483 | "metadata": {}, 484 | "source": [ 485 | "Julia provides the `push!` and `append!` functions to add additional elements to an existing array.\n", 486 | "For example:" 487 | ] 488 | }, 489 | { 490 | "cell_type": "code", 491 | "execution_count": null, 492 | "metadata": {}, 493 | "outputs": [], 494 | "source": [ 495 | "A = Vector{Float64}() # Create an empty Float64 array" 496 | ] 497 | }, 498 | { 499 | "cell_type": "code", 500 | "execution_count": null, 501 | "metadata": {}, 502 | "outputs": [], 503 | "source": [ 504 | "push!(A, 4.)" 505 | ] 506 | }, 507 | { 508 | "cell_type": "code", 509 | "execution_count": null, 510 | "metadata": {}, 511 | "outputs": [], 512 | "source": [ 513 | "append!(A, [5, 6, 7])" 514 | ] 515 | }, 516 | { 517 | "cell_type": "markdown", 518 | "metadata": {}, 519 | "source": [ 520 | "Notice, that the `!` is part of the name of the function. In Julia the `!` is a convention to indicate that the respective function *mutates* the content of at least one of the passed arrays." 521 | ] 522 | }, 523 | { 524 | "cell_type": "markdown", 525 | "metadata": {}, 526 | "source": [ 527 | "Very helpful functions as we will see are:\n", 528 | "\n", 529 | "- `zero`, which allocates an array of zeros of the same element type" 530 | ] 531 | }, 532 | { 533 | "cell_type": "code", 534 | "execution_count": null, 535 | "metadata": {}, 536 | "outputs": [], 537 | "source": [ 538 | "A = randn(Float32, 3, 4)\n", 539 | "zero(A)" 540 | ] 541 | }, 542 | { 543 | "cell_type": "markdown", 544 | "metadata": {}, 545 | "source": [ 546 | "- `similar`, which returns an uninitialised array, which is similar to the passed array. This means that by default array type, element type and size are all kept." 547 | ] 548 | }, 549 | { 550 | "cell_type": "code", 551 | "execution_count": null, 552 | "metadata": {}, 553 | "outputs": [], 554 | "source": [ 555 | "similar(A)" 556 | ] 557 | }, 558 | { 559 | "cell_type": "markdown", 560 | "metadata": {}, 561 | "source": [ 562 | "- One may also change these parameters easily:" 563 | ] 564 | }, 565 | { 566 | "cell_type": "code", 567 | "execution_count": null, 568 | "metadata": {}, 569 | "outputs": [], 570 | "source": [ 571 | "similar(A, (3, 2)) # Keep element type and array type" 572 | ] 573 | }, 574 | { 575 | "cell_type": "code", 576 | "execution_count": null, 577 | "metadata": {}, 578 | "outputs": [], 579 | "source": [ 580 | "similar(A, Float64) # Change element type" 581 | ] 582 | }, 583 | { 584 | "cell_type": "code", 585 | "execution_count": null, 586 | "metadata": {}, 587 | "outputs": [], 588 | "source": [ 589 | "similar(A, Float64, (1, 2)) # Change element type and shape" 590 | ] 591 | }, 592 | { 593 | "cell_type": "markdown", 594 | "metadata": {}, 595 | "source": [ 596 | "## Vector operations and vectorised operations\n", 597 | "\n", 598 | "Array addition (`+`, `-`) and scalar multiplication are directly available on arrays (of any dimension):" 599 | ] 600 | }, 601 | { 602 | "cell_type": "code", 603 | "execution_count": null, 604 | "metadata": {}, 605 | "outputs": [], 606 | "source": [ 607 | "x = [1, 2, 3]\n", 608 | "y = [4, 5, 6]" 609 | ] 610 | }, 611 | { 612 | "cell_type": "code", 613 | "execution_count": null, 614 | "metadata": {}, 615 | "outputs": [], 616 | "source": [ 617 | "x + 2.0y" 618 | ] 619 | }, 620 | { 621 | "cell_type": "markdown", 622 | "metadata": {}, 623 | "source": [ 624 | "For element-wise operations the vectorisation syntax is used:" 625 | ] 626 | }, 627 | { 628 | "cell_type": "code", 629 | "execution_count": null, 630 | "metadata": {}, 631 | "outputs": [], 632 | "source": [ 633 | "x .* y # elementwise mulitplication" 634 | ] 635 | }, 636 | { 637 | "cell_type": "code", 638 | "execution_count": null, 639 | "metadata": {}, 640 | "outputs": [], 641 | "source": [ 642 | "x .^ y # Elementwise exponentiation" 643 | ] 644 | }, 645 | { 646 | "cell_type": "markdown", 647 | "metadata": {}, 648 | "source": [ 649 | "Note, that the `.`-syntax continues to *all* functions in Julia. That includes base Julia ..." 650 | ] 651 | }, 652 | { 653 | "cell_type": "code", 654 | "execution_count": null, 655 | "metadata": {}, 656 | "outputs": [], 657 | "source": [ 658 | "sqrt.(cos.(2π .* x) .+ sin.(2π * x))" 659 | ] 660 | }, 661 | { 662 | "cell_type": "code", 663 | "execution_count": null, 664 | "metadata": {}, 665 | "outputs": [], 666 | "source": [ 667 | "@. sqrt(cos(2π * x) + sin(2π * x))" 668 | ] 669 | }, 670 | { 671 | "cell_type": "markdown", 672 | "metadata": {}, 673 | "source": [ 674 | "... custom functions ..." 675 | ] 676 | }, 677 | { 678 | "cell_type": "code", 679 | "execution_count": null, 680 | "metadata": {}, 681 | "outputs": [], 682 | "source": [ 683 | "myfun(x) = x * x + x\n", 684 | "myfun.(y)" 685 | ] 686 | }, 687 | { 688 | "cell_type": "markdown", 689 | "metadata": {}, 690 | "source": [ 691 | "... and may be easily chained " 692 | ] 693 | }, 694 | { 695 | "cell_type": "code", 696 | "execution_count": null, 697 | "metadata": {}, 698 | "outputs": [], 699 | "source": [ 700 | "@. exp(cos(x^2))" 701 | ] 702 | }, 703 | { 704 | "cell_type": "markdown", 705 | "metadata": {}, 706 | "source": [ 707 | "### Exercise\n", 708 | "Create the following arrays using Julia code:\n", 709 | "$$\\left(\\begin{array}{ccccc}\n", 710 | " 2&2&2&2&2 \\\\\n", 711 | " 2&2&2&2&2 \\\\\n", 712 | " 2&2&2&2&2 \\\\\n", 713 | " \\end{array}\\right) \\qquad\n", 714 | " \\left(\\begin{array}{cccc}\n", 715 | " 0.1&0.5&0.9&1.3\\\\\n", 716 | " 0.2&0.6&1.0&1.4\\\\\n", 717 | " 0.3&0.7&1.1&1.5\\\\\n", 718 | " 0.4&0.8&1.2&1.6\\\\\n", 719 | " \\end{array}\\right)\n", 720 | " $$" 721 | ] 722 | }, 723 | { 724 | "cell_type": "markdown", 725 | "metadata": {}, 726 | "source": [ 727 | "##### More details on Arrays\n", 728 | "- https://docs.julialang.org/en/v1/manual/arrays/\n", 729 | "- https://docs.julialang.org/en/v1/base/arrays/\n", 730 | "- https://docs.julialang.org/en/v1/manual/interfaces/#man-interface-array-1" 731 | ] 732 | }, 733 | { 734 | "cell_type": "markdown", 735 | "metadata": {}, 736 | "source": [ 737 | "## `UnitRange` as an `AbstractArray`" 738 | ] 739 | }, 740 | { 741 | "cell_type": "code", 742 | "execution_count": null, 743 | "metadata": {}, 744 | "outputs": [], 745 | "source": [ 746 | "x = 1:30" 747 | ] 748 | }, 749 | { 750 | "cell_type": "code", 751 | "execution_count": null, 752 | "metadata": {}, 753 | "outputs": [], 754 | "source": [ 755 | "typeof(x)" 756 | ] 757 | }, 758 | { 759 | "cell_type": "code", 760 | "execution_count": null, 761 | "metadata": {}, 762 | "outputs": [], 763 | "source": [ 764 | "typeof(x) <: AbstractArray" 765 | ] 766 | }, 767 | { 768 | "cell_type": "markdown", 769 | "metadata": {}, 770 | "source": [ 771 | "Because it is a subtype of `AbstractArray` we can do array-like things with it (it should basically behave like an array!)" 772 | ] 773 | }, 774 | { 775 | "cell_type": "code", 776 | "execution_count": null, 777 | "metadata": {}, 778 | "outputs": [], 779 | "source": [ 780 | "x[3]" 781 | ] 782 | }, 783 | { 784 | "cell_type": "code", 785 | "execution_count": null, 786 | "metadata": {}, 787 | "outputs": [], 788 | "source": [ 789 | "size(x)" 790 | ] 791 | }, 792 | { 793 | "cell_type": "code", 794 | "execution_count": null, 795 | "metadata": {}, 796 | "outputs": [], 797 | "source": [ 798 | "eltype(x)" 799 | ] 800 | }, 801 | { 802 | "cell_type": "markdown", 803 | "metadata": {}, 804 | "source": [ 805 | "However, it's not implemented like a regular `Array` at all.\n", 806 | "\n", 807 | "In fact, it's just two numbers! We can see this by looking at it's fields:" 808 | ] 809 | }, 810 | { 811 | "cell_type": "code", 812 | "execution_count": null, 813 | "metadata": {}, 814 | "outputs": [], 815 | "source": [ 816 | "fieldnames(typeof(x))" 817 | ] 818 | }, 819 | { 820 | "cell_type": "markdown", 821 | "metadata": {}, 822 | "source": [ 823 | "or just by inspecting the source code" 824 | ] 825 | }, 826 | { 827 | "cell_type": "code", 828 | "execution_count": null, 829 | "metadata": {}, 830 | "outputs": [], 831 | "source": [ 832 | "@which UnitRange(1, 30)" 833 | ] 834 | }, 835 | { 836 | "cell_type": "markdown", 837 | "metadata": {}, 838 | "source": [ 839 | "It is an `immutable` type which just holds the start and stop values.\n", 840 | "\n", 841 | "This means that indexing, `A[i]`, is not just a look-up but a (small) function (try `@which getindex(x, 4)`).\n", 842 | "\n", 843 | "What's nice about this is that we can use it in calculations and no array, containing the numbers from 1 to 30, is ever created.\n", 844 | "\n", 845 | "Julia is pretty smart here, for example:" 846 | ] 847 | }, 848 | { 849 | "cell_type": "code", 850 | "execution_count": null, 851 | "metadata": {}, 852 | "outputs": [], 853 | "source": [ 854 | "(1:10) .+ 3" 855 | ] 856 | }, 857 | { 858 | "cell_type": "markdown", 859 | "metadata": {}, 860 | "source": [ 861 | "Allocating memory is typically costly." 862 | ] 863 | }, 864 | { 865 | "cell_type": "code", 866 | "execution_count": null, 867 | "metadata": {}, 868 | "outputs": [], 869 | "source": [ 870 | "collect(1:10)" 871 | ] 872 | }, 873 | { 874 | "cell_type": "code", 875 | "execution_count": null, 876 | "metadata": {}, 877 | "outputs": [], 878 | "source": [ 879 | "@time collect(1:10000000);" 880 | ] 881 | }, 882 | { 883 | "cell_type": "markdown", 884 | "metadata": {}, 885 | "source": [ 886 | "But creating an immutable type of two numbers is essentially free, no matter what those two numbers are:" 887 | ] 888 | }, 889 | { 890 | "cell_type": "code", 891 | "execution_count": null, 892 | "metadata": {}, 893 | "outputs": [], 894 | "source": [ 895 | "@time 1:10000000;" 896 | ] 897 | }, 898 | { 899 | "cell_type": "markdown", 900 | "metadata": {}, 901 | "source": [ 902 | "Yet, in code they *act* the same way." 903 | ] 904 | }, 905 | { 906 | "cell_type": "markdown", 907 | "metadata": {}, 908 | "source": [ 909 | "## Takeaways\n", 910 | "\n", 911 | "- Parametric types ar types that by themselves have parameters (e.g. `Vector{Float64}`). The notation `T where T <: SuperType` exists to denote sets of types.\n", 912 | "- Arrays are deeply built into Julia\n", 913 | "- Julia has many clever Array types (like `UnitRange`) to speed up array operations." 914 | ] 915 | } 916 | ], 917 | "metadata": { 918 | "@webio": { 919 | "lastCommId": null, 920 | "lastKernelId": null 921 | }, 922 | "kernelspec": { 923 | "display_name": "Julia 1.7.2", 924 | "language": "julia", 925 | "name": "julia-1.7" 926 | }, 927 | "language_info": { 928 | "file_extension": ".jl", 929 | "mimetype": "application/julia", 930 | "name": "julia", 931 | "version": "1.7.2" 932 | } 933 | }, 934 | "nbformat": 4, 935 | "nbformat_minor": 4 936 | } 937 | -------------------------------------------------------------------------------- /04_Getting_Help.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Getting help" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": null, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "using LinearAlgebra" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "## For the standard library and installed packages\n", 24 | "\n", 25 | "For displaying a short hint (docstring) about a function or type, use the `?`:" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": null, 31 | "metadata": {}, 32 | "outputs": [], 33 | "source": [ 34 | "?eachcol" 35 | ] 36 | }, 37 | { 38 | "cell_type": "code", 39 | "execution_count": null, 40 | "metadata": {}, 41 | "outputs": [], 42 | "source": [ 43 | "?tr" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "?Symmetric" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | "A frequent question one asks when trying to understand the structure of a new library or code is: What can I actually do with this type? This question is answered by the command `methodswith`, which lists all methods which has at least one argument of the specified type:" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": { 66 | "scrolled": true 67 | }, 68 | "outputs": [], 69 | "source": [ 70 | "methodswith(Diagonal)" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "metadata": {}, 76 | "source": [ 77 | "To search more specifically, one may optionally specify a module or function:" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": null, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "methodswith(Diagonal, tr)" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "We already saw how to answer the reverse question, namely what concrete methods do I have for one function:" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": { 100 | "scrolled": true 101 | }, 102 | "outputs": [], 103 | "source": [ 104 | "methods(transpose)" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": {}, 110 | "source": [ 111 | "Again a more specific version, which searches for methods with respect to a particular type is" 112 | ] 113 | }, 114 | { 115 | "cell_type": "code", 116 | "execution_count": null, 117 | "metadata": {}, 118 | "outputs": [], 119 | "source": [ 120 | "methods(transpose, [Symmetric])" 121 | ] 122 | }, 123 | { 124 | "cell_type": "markdown", 125 | "metadata": {}, 126 | "source": [ 127 | "If one does not remember the name of a function, the `apropos` function can be very helpful:" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": null, 133 | "metadata": {}, 134 | "outputs": [], 135 | "source": [ 136 | "apropos(\"invert\")" 137 | ] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "metadata": {}, 142 | "source": [ 143 | "### Exercise\n", 144 | "Explore the following functions and types using the tools mentioned above\n", 145 | "- `fill`\n", 146 | "- `ones`\n", 147 | "- `cholesky`\n", 148 | "- `eigen`\n", 149 | "- `qr`\n", 150 | "- `factorize`\n", 151 | "- `Tridiagonal`\n", 152 | "- `BunchKaufman`\n", 153 | "- `UpperHessenberg`" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "## Julia Documentation\n", 161 | "\n", 162 | "The [Julia documentation](https://docs.julialang.org) is the starting point for finding answers about the **julia language**, any **`Base` package** or the **standard library**. If you suspect something is wrong with Julia itself, looking at the [list of issues](https://github.com/JuliaLang/julia/issues) or [list of Pull requests](https://github.com/JuliaLang/julia/issues) can be helpful.\n", 163 | "\n", 164 | "Most **Julia packages** have adopted similar standards to Base Julia with respect to documentation. They are usually developed on [github](https://github.com/topics/julia) and that's also a very good resource to find the project's homepage, installation tips or quickstart guides. Most have their own documentation somewhere linked as well (look out for an icon like ![](https://img.shields.io/badge/docs-stable-blue.svg)). If you have trouble with the package, it's also worth checking out the list of issues or pull requests to see if others have the same problem or people are already working on a fix. If not generally maintainers are more than happy if you get in touch there. Even if you are a Julia newbie: Be not afraid to send a patch or open a PR! Most maintainers are happy to help if you make the first step ;)." 165 | ] 166 | }, 167 | { 168 | "cell_type": "markdown", 169 | "metadata": {}, 170 | "source": [ 171 | "## Julia Community\n", 172 | "\n", 173 | "The Julia community is open and friendly. The primary channel of communication is the [Julia Discourse](https://discourse.julialang.org/). There's also a [Julia Slack](https://julialang.slack.com/)." 174 | ] 175 | } 176 | ], 177 | "metadata": { 178 | "kernelspec": { 179 | "display_name": "Julia 1.7.2", 180 | "language": "julia", 181 | "name": "julia-1.7" 182 | }, 183 | "language_info": { 184 | "file_extension": ".jl", 185 | "mimetype": "application/julia", 186 | "name": "julia", 187 | "version": "1.7.2" 188 | } 189 | }, 190 | "nbformat": 4, 191 | "nbformat_minor": 2 192 | } 193 | -------------------------------------------------------------------------------- /05_Specialisation_Speed.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Speed by specialisation\n", 8 | "\n", 9 | "In the previous notebooks we discussed Julia's elaborate type system and how multiple dispatch allows functions to decide on the method to dispatch really from looking at all the arguments. Intuitively **more specialised code**, i.e. code that is allowed to exploit known structure, **is faster.** In this notebook we will explore how Julia is able to exploit this while retaining generic code ... thanks to multiple dispatch." 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## The speed of Julia\n", 17 | "\n", 18 | "Now let us first try to address the question *Is Julia fast?*\n", 19 | "\n", 20 | "To some extend this is a bit of a mismatching question, since one is able to write slow code in any language ... so let's try do address something else instead: *Can Julia be fast?*" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "## A simple example: Sums\n", 28 | "\n", 29 | "Let's compare for a simple example ..." 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "function mysum(v)\n", 39 | " result = zero(eltype(v))\n", 40 | " for i in 1:length(v)\n", 41 | " result += v[i]\n", 42 | " end\n", 43 | " result\n", 44 | "end" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "v = randn(10^7); # Large vector of numbers" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "using BenchmarkTools" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "# Compile some C code and call it from julia ...\n", 72 | "using Libdl\n", 73 | "code = \"\"\"\n", 74 | "#include \n", 75 | "double c_sum(size_t n, double *v) {\n", 76 | " double accu = 0.0;\n", 77 | " for (size_t i = 0; i < n; ++i) {\n", 78 | " accu += v[i];\n", 79 | " }\n", 80 | " return accu;\n", 81 | "}\n", 82 | "\"\"\"\n", 83 | "\n", 84 | "# Compile to a shared library (with fast maths and machine-specific)\n", 85 | "const Clib = tempname()\n", 86 | "open(`gcc -fPIC -O3 -march=native -xc -shared -ffast-math -o $(Clib * \".\" * Libdl.dlext) -`, \"w\") do f\n", 87 | " print(f, code) \n", 88 | "end\n", 89 | "\n", 90 | "# define a Julia function that calls the C function:\n", 91 | "c_sum(v::Array{Float64}) = ccall((\"c_sum\", Clib), Float64, (Csize_t, Ptr{Float64}), length(v), v)" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": null, 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "bench = @benchmark c_sum($v)" 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": null, 106 | "metadata": {}, 107 | "outputs": [], 108 | "source": [ 109 | "times = Dict()\n", 110 | "times[\"C (naive)\"] = minimum(bench.times) / 1e6\n", 111 | "times" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "An explicitly and fully vectorised C++ code:" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": null, 124 | "metadata": {}, 125 | "outputs": [], 126 | "source": [ 127 | "# Compile some C code and call it from julia ...\n", 128 | "using Libdl\n", 129 | "code = \"\"\"\n", 130 | "#include \n", 131 | "#include \n", 132 | "\n", 133 | "double c_sum_vector(size_t n, double *v)\n", 134 | "{\n", 135 | " size_t i;\n", 136 | " double result;\n", 137 | " double tmp[4] __attribute__ ((aligned(64)));\n", 138 | "\n", 139 | " __m256d sums1 = _mm256_setzero_pd();\n", 140 | " __m256d sums2 = _mm256_setzero_pd();\n", 141 | " for ( i = 0; i + 7 < n; i += 8 )\n", 142 | " {\n", 143 | " sums1 = _mm256_add_pd( sums1, _mm256_loadu_pd(v+i ) );\n", 144 | " sums2 = _mm256_add_pd( sums2, _mm256_loadu_pd(v+i+4) );\n", 145 | " }\n", 146 | " _mm256_store_pd( tmp, _mm256_add_pd(sums1, sums2) );\n", 147 | "\n", 148 | " return tmp[0] + tmp[1] + tmp[2] + tmp[3]; \n", 149 | "}\n", 150 | "\"\"\"\n", 151 | "\n", 152 | "# Compile to a shared library (with fast maths and machine-specific)\n", 153 | "const Clib = tempname()\n", 154 | "open(`gcc -fPIC -O2 -march=native -xc -shared -ffast-math -o $(Clib * \".\" * Libdl.dlext) -`, \"w\") do f\n", 155 | " print(f, code) \n", 156 | "end\n", 157 | "\n", 158 | "# define a Julia function that calls the C function:\n", 159 | "c_sum(v::Array{Float64}) = ccall((\"c_sum_vector\", Clib), Float64, (Csize_t, Ptr{Float64}), length(v), v)" 160 | ] 161 | }, 162 | { 163 | "cell_type": "code", 164 | "execution_count": null, 165 | "metadata": {}, 166 | "outputs": [], 167 | "source": [ 168 | "bench = @benchmark c_sum($v)" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": null, 174 | "metadata": {}, 175 | "outputs": [], 176 | "source": [ 177 | "times[\"C (vectorised)\"] = minimum(bench.times) / 1e6" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": {}, 183 | "source": [ 184 | "How does our version do?" 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": null, 190 | "metadata": {}, 191 | "outputs": [], 192 | "source": [ 193 | "bench = @benchmark mysum($v)" 194 | ] 195 | }, 196 | { 197 | "cell_type": "code", 198 | "execution_count": null, 199 | "metadata": {}, 200 | "outputs": [], 201 | "source": [ 202 | "times[\"Julia (naive)\"] = minimum(bench.times) / 1e6" 203 | ] 204 | }, 205 | { 206 | "cell_type": "markdown", 207 | "metadata": {}, 208 | "source": [ 209 | "But unlike C we have not yet tried all tricks!" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": null, 215 | "metadata": {}, 216 | "outputs": [], 217 | "source": [ 218 | "function fastsum(v)\n", 219 | " result = zero(eltype(v))\n", 220 | " @simd for i in 1:length(v) # @simd enforces vectorisation in the loop\n", 221 | " @inbounds result += v[i] # @inbounds suppresses bound checks\n", 222 | " end\n", 223 | " result\n", 224 | "end\n", 225 | "\n", 226 | "# Still nicely readable code ..." 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": null, 232 | "metadata": {}, 233 | "outputs": [], 234 | "source": [ 235 | "bench = @benchmark fastsum($v)" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": null, 241 | "metadata": {}, 242 | "outputs": [], 243 | "source": [ 244 | "times[\"Julia (simd)\"] = minimum(bench.times) / 1e6" 245 | ] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": {}, 250 | "source": [ 251 | "Compare with python ..." 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": null, 257 | "metadata": {}, 258 | "outputs": [], 259 | "source": [ 260 | "using PyCall\n", 261 | "numpysum(v) = pyimport(\"numpy\").sum(v)\n", 262 | "\n", 263 | "bench = @benchmark numpysum($v)" 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": null, 269 | "metadata": {}, 270 | "outputs": [], 271 | "source": [ 272 | "times[\"Numpy\"] = minimum(bench.times) / 1e6" 273 | ] 274 | }, 275 | { 276 | "cell_type": "code", 277 | "execution_count": null, 278 | "metadata": {}, 279 | "outputs": [], 280 | "source": [ 281 | "py\"\"\"\n", 282 | "def py_sum(A):\n", 283 | " s = 0.0\n", 284 | " for a in A:\n", 285 | " s += a\n", 286 | " return s\n", 287 | "\"\"\"\n", 288 | "\n", 289 | "pysum = py\"py_sum\"\n", 290 | "\n", 291 | "bench = @benchmark pysum($v)" 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "execution_count": null, 297 | "metadata": {}, 298 | "outputs": [], 299 | "source": [ 300 | "times[\"Python (naive)\"] = minimum(bench.times) / 1e6" 301 | ] 302 | }, 303 | { 304 | "cell_type": "markdown", 305 | "metadata": {}, 306 | "source": [ 307 | "Overview ..." 308 | ] 309 | }, 310 | { 311 | "cell_type": "code", 312 | "execution_count": null, 313 | "metadata": {}, 314 | "outputs": [], 315 | "source": [ 316 | "times" 317 | ] 318 | }, 319 | { 320 | "cell_type": "markdown", 321 | "metadata": {}, 322 | "source": [ 323 | "## A more complicated example: Vandermonde matrices\n", 324 | "(modified from [Steven's Julia intro](https://web.mit.edu/18.06/www/Fall17/1806/julia/Julia-intro.pdf))" 325 | ] 326 | }, 327 | { 328 | "cell_type": "markdown", 329 | "metadata": {}, 330 | "source": [ 331 | "\\begin{align}V=\\begin{bmatrix}1&\\alpha _{1}&\\alpha _{1}^{2}&\\dots &\\alpha _{1}^{n-1}\\\\1&\\alpha _{2}&\\alpha _{2}^{2}&\\dots &\\alpha _{2}^{n-1}\\\\1&\\alpha _{3}&\\alpha _{3}^{2}&\\dots &\\alpha _{3}^{n-1}\\\\\\vdots &\\vdots &\\vdots &\\ddots &\\vdots \\\\1&\\alpha _{m}&\\alpha _{m}^{2}&\\dots &\\alpha _{m}^{n-1}\\end{bmatrix}\\end{align}" 332 | ] 333 | }, 334 | { 335 | "cell_type": "code", 336 | "execution_count": null, 337 | "metadata": {}, 338 | "outputs": [], 339 | "source": [ 340 | "using PyCall\n", 341 | "np = pyimport(\"numpy\")" 342 | ] 343 | }, 344 | { 345 | "cell_type": "code", 346 | "execution_count": null, 347 | "metadata": {}, 348 | "outputs": [], 349 | "source": [ 350 | "np.vander(1:5, increasing=true)" 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "metadata": {}, 356 | "source": [ 357 | "[The source code for this function](https://github.com/numpy/numpy/blob/v1.16.1/numpy/lib/twodim_base.py#L475-L563) calls `np.multiply.accumulate` [which is implemented in C](https://github.com/numpy/numpy/blob/deea4983aedfa96905bbaee64e3d1de84144303f/numpy/core/src/umath/ufunc_object.c#L3678). However, this code doesn't actually perform the computation, it basically only checks types and stuff. The actual kernel is [implemented here](https://github.com/numpy/numpy/blob/deea4983aedfa96905bbaee64e3d1de84144303f/numpy/core/src/umath/loops.c.src#L1742), which is not even C code, but only a template that gets transformed to type-specific kernels. Still only a limited set of types `Float64`, `Float32`, and so forth are supported." 358 | ] 359 | }, 360 | { 361 | "cell_type": "markdown", 362 | "metadata": {}, 363 | "source": [ 364 | "Here is a type-generic Julia implementation:" 365 | ] 366 | }, 367 | { 368 | "cell_type": "code", 369 | "execution_count": null, 370 | "metadata": {}, 371 | "outputs": [], 372 | "source": [ 373 | "function vander(x::AbstractVector{T}) where T\n", 374 | " m = length(x)\n", 375 | " V = Matrix{T}(undef, m, m)\n", 376 | " for j = 1:m\n", 377 | " V[j,1] = one(x[j])\n", 378 | " end\n", 379 | " for i= 2:m\n", 380 | " for j = 1:m\n", 381 | " V[j,i] = x[j] * V[j,i-1]\n", 382 | " end\n", 383 | " end\n", 384 | " return V\n", 385 | "end" 386 | ] 387 | }, 388 | { 389 | "cell_type": "code", 390 | "execution_count": null, 391 | "metadata": {}, 392 | "outputs": [], 393 | "source": [ 394 | "vander(1:5)" 395 | ] 396 | }, 397 | { 398 | "cell_type": "markdown", 399 | "metadata": {}, 400 | "source": [ 401 | "### Quick speed comparison" 402 | ] 403 | }, 404 | { 405 | "cell_type": "code", 406 | "execution_count": null, 407 | "metadata": {}, 408 | "outputs": [], 409 | "source": [ 410 | "using BenchmarkTools\n", 411 | "using Plots\n", 412 | "ns = exp10.(range(1, 4, length=30));\n", 413 | "\n", 414 | "tnp = Float64[]\n", 415 | "tjl = Float64[]\n", 416 | "for n in ns\n", 417 | " x = collect(1:n)\n", 418 | " push!(tnp, @belapsed np.vander($x) samples=3 evals=1)\n", 419 | " push!(tjl, @belapsed vander($x) samples=3 evals=1)\n", 420 | "end\n", 421 | "plot(ns, tnp./tjl, m=:circle, xscale=:log10, yscale=:log10, ylims=[1, Inf],\n", 422 | " xlab=\"matrix size\", ylab=\"NumPy time / Julia time\", legend=:false)" 423 | ] 424 | }, 425 | { 426 | "cell_type": "markdown", 427 | "metadata": {}, 428 | "source": [ 429 | "Notably the clean and concise Julia implementation is **faster than numpy**'s C implementation for small matrices and **as fast** for large matrix sizes. Still it works for **arbitrary types**." 430 | ] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "execution_count": null, 435 | "metadata": {}, 436 | "outputs": [], 437 | "source": [ 438 | "vander(Int32[4, 8, 16, 32])" 439 | ] 440 | }, 441 | { 442 | "cell_type": "markdown", 443 | "metadata": {}, 444 | "source": [ 445 | "This includes non-numerical types ... the only assumption is that the type induces a multiplicative group, i.e. has a `one` function to yield the identity element and an apropriate `*` defined.\n", 446 | "\n", 447 | "A rather unusual one is `String`, which works since `one(String) == \"\"`." 448 | ] 449 | }, 450 | { 451 | "cell_type": "code", 452 | "execution_count": null, 453 | "metadata": {}, 454 | "outputs": [], 455 | "source": [ 456 | "vander([\"this\", \"is\", \"a\", \"test\"])" 457 | ] 458 | }, 459 | { 460 | "cell_type": "markdown", 461 | "metadata": {}, 462 | "source": [ 463 | "## How does this work?\n", 464 | "\n", 465 | "\n", 466 | "\n", 467 | " \n", 468 | "- AST = Abstract Syntax Tree\n", 469 | "- SSA = Static Single Assignment\n", 470 | "- [LLVM](https://de.wikipedia.org/wiki/LLVM) = Low Level Virtual Machine" 471 | ] 472 | }, 473 | { 474 | "cell_type": "markdown", 475 | "metadata": {}, 476 | "source": [ 477 | "Julia compiles **ahead of time**, i.e. Julia waits until the first function call for a particular set of input types is issued and then compiles the full call stack all the way down to machine code specialising as much as possible along the way." 478 | ] 479 | }, 480 | { 481 | "cell_type": "code", 482 | "execution_count": null, 483 | "metadata": {}, 484 | "outputs": [], 485 | "source": [ 486 | "x = [1, 3, 5] # Vector{Int64}\n", 487 | "\n", 488 | "@time mysum(x)\n", 489 | "@time mysum(x)" 490 | ] 491 | }, 492 | { 493 | "cell_type": "markdown", 494 | "metadata": {}, 495 | "source": [ 496 | "As we saw if thes change, Julia compiles a new specialization of the function!" 497 | ] 498 | }, 499 | { 500 | "cell_type": "markdown", 501 | "metadata": {}, 502 | "source": [ 503 | "To see what happens under the hood, we have a bunch of inspection macros:\n", 504 | "* The AST after parsing (**`@macroexpand`**)\n", 505 | "* The AST after lowering (**`@code_typed`**, **`@code_warntype`**)\n", 506 | "* The AST after type inference and optimization (**`@code_lowered`**)\n", 507 | "* The LLVM IR (**`@code_llvm`**)\n", 508 | "* The assembly machine code (**`@code_native`**)" 509 | ] 510 | }, 511 | { 512 | "cell_type": "code", 513 | "execution_count": null, 514 | "metadata": {}, 515 | "outputs": [], 516 | "source": [ 517 | "@code_typed mysum(1)" 518 | ] 519 | }, 520 | { 521 | "cell_type": "code", 522 | "execution_count": null, 523 | "metadata": {}, 524 | "outputs": [], 525 | "source": [ 526 | "@code_lowered mysum(1)" 527 | ] 528 | }, 529 | { 530 | "cell_type": "code", 531 | "execution_count": null, 532 | "metadata": {}, 533 | "outputs": [], 534 | "source": [ 535 | "@code_llvm mysum(1)" 536 | ] 537 | }, 538 | { 539 | "cell_type": "markdown", 540 | "metadata": {}, 541 | "source": [ 542 | "We can remove the comments (lines starting with `;` using `debuginfo=:none`)." 543 | ] 544 | }, 545 | { 546 | "cell_type": "code", 547 | "execution_count": null, 548 | "metadata": {}, 549 | "outputs": [], 550 | "source": [ 551 | "@code_llvm debuginfo=:none mysum(1.2)" 552 | ] 553 | }, 554 | { 555 | "cell_type": "code", 556 | "execution_count": null, 557 | "metadata": {}, 558 | "outputs": [], 559 | "source": [ 560 | "@code_native debuginfo=:none mysum(1.2)" 561 | ] 562 | }, 563 | { 564 | "cell_type": "markdown", 565 | "metadata": {}, 566 | "source": [ 567 | "Let's compare this to `Float64` input." 568 | ] 569 | }, 570 | { 571 | "cell_type": "code", 572 | "execution_count": null, 573 | "metadata": {}, 574 | "outputs": [], 575 | "source": [ 576 | "@code_native debuginfo=:none mysum(1)" 577 | ] 578 | }, 579 | { 580 | "cell_type": "markdown", 581 | "metadata": {}, 582 | "source": [ 583 | "## Types and code specialisation\n", 584 | "\n", 585 | "As a rule of thumb: The Julia compiler only uses the types to specialise, not the values. Therefore using special types that match the possible assumptions most closely is crucial.\n", 586 | "\n", 587 | "As an example we consider the determinant of a 2x2 matrix." 588 | ] 589 | }, 590 | { 591 | "cell_type": "code", 592 | "execution_count": null, 593 | "metadata": {}, 594 | "outputs": [], 595 | "source": [ 596 | "# Fast custom implementation ... note: No type annotation\n", 597 | "det_custom(M) = @inbounds (M[1, 1] * M[2, 2] - M[1, 2] * M[2, 1])" 598 | ] 599 | }, 600 | { 601 | "cell_type": "code", 602 | "execution_count": null, 603 | "metadata": {}, 604 | "outputs": [], 605 | "source": [ 606 | "M = randn(2, 2)\n", 607 | "@btime det_custom(M);" 608 | ] 609 | }, 610 | { 611 | "cell_type": "code", 612 | "execution_count": null, 613 | "metadata": {}, 614 | "outputs": [], 615 | "source": [ 616 | "using LinearAlgebra\n", 617 | "@btime det(M);" 618 | ] 619 | }, 620 | { 621 | "cell_type": "markdown", 622 | "metadata": {}, 623 | "source": [ 624 | "Now that is a drastic difference ... but we actually did not tell the compiler anything about the size of `M`:" 625 | ] 626 | }, 627 | { 628 | "cell_type": "code", 629 | "execution_count": null, 630 | "metadata": {}, 631 | "outputs": [], 632 | "source": [ 633 | "typeof(M)" 634 | ] 635 | }, 636 | { 637 | "cell_type": "markdown", 638 | "metadata": {}, 639 | "source": [ 640 | "Now let's help along ... by using a static array type:" 641 | ] 642 | }, 643 | { 644 | "cell_type": "code", 645 | "execution_count": null, 646 | "metadata": {}, 647 | "outputs": [], 648 | "source": [ 649 | "using StaticArrays\n", 650 | "\n", 651 | "S = @SMatrix randn(2, 2)" 652 | ] 653 | }, 654 | { 655 | "cell_type": "code", 656 | "execution_count": null, 657 | "metadata": {}, 658 | "outputs": [], 659 | "source": [ 660 | "@btime det(S);" 661 | ] 662 | }, 663 | { 664 | "cell_type": "markdown", 665 | "metadata": {}, 666 | "source": [ 667 | "Got it! Now ... what happens under the hood:" 668 | ] 669 | }, 670 | { 671 | "cell_type": "code", 672 | "execution_count": null, 673 | "metadata": {}, 674 | "outputs": [], 675 | "source": [ 676 | "@code_typed debuginfo=:none det(S)" 677 | ] 678 | }, 679 | { 680 | "cell_type": "markdown", 681 | "metadata": {}, 682 | "source": [ 683 | "in other words\n", 684 | "```\n", 685 | "%2 = S[1, 1]\n", 686 | "%4 = S[2, 2]\n", 687 | "%5 = S[1, 1] * S[2, 2]\n", 688 | "%7 = S[2, 1]\n", 689 | "%9 = S[1, 2]\n", 690 | "%10 = S[2, 1] * S[1, 2]\n", 691 | "%11 = S[1, 1] * S[2, 2] - S[2, 1] * S[1, 2]\n", 692 | "```\n", 693 | "exactly what we had hand-optimised before!" 694 | ] 695 | }, 696 | { 697 | "cell_type": "markdown", 698 | "metadata": {}, 699 | "source": [ 700 | "Now one should add here that part of the magic has not been the compiler being smart, but much rather the `StaticArrays` package, which has a number of carfully optimised implementations for standard operations:" 701 | ] 702 | }, 703 | { 704 | "cell_type": "markdown", 705 | "metadata": {}, 706 | "source": [ 707 | "```\n", 708 | "============================================\n", 709 | " Benchmarks for 3×3 Float64 matrices\n", 710 | "============================================\n", 711 | "Matrix multiplication -> 5.9x speedup\n", 712 | "Matrix multiplication (mutating) -> 1.8x speedup\n", 713 | "Matrix addition -> 33.1x speedup\n", 714 | "Matrix addition (mutating) -> 2.5x speedup\n", 715 | "Matrix determinant -> 112.9x speedup\n", 716 | "Matrix inverse -> 67.8x speedup\n", 717 | "Matrix symmetric eigendecomposition -> 25.0x speedup\n", 718 | "Matrix Cholesky decomposition -> 8.8x speedup\n", 719 | "Matrix LU decomposition -> 6.1x speedup\n", 720 | "Matrix QR decomposition -> 65.0x speedup\n", 721 | "```" 722 | ] 723 | }, 724 | { 725 | "cell_type": "markdown", 726 | "metadata": {}, 727 | "source": [ 728 | "##### More details\n", 729 | "- https://github.com/JuliaCI/BenchmarkTools.jl\n", 730 | "- https://github.com/JuliaArrays/StaticArrays.jl/\n", 731 | "- https://docs.julialang.org/en/v1/devdocs/compiler/" 732 | ] 733 | }, 734 | { 735 | "cell_type": "markdown", 736 | "metadata": {}, 737 | "source": [ 738 | "## Takeaways\n", 739 | "\n", 740 | "- Julia can be fast\n", 741 | "- Code is compiled down to the metal\n", 742 | "- Stages can be inspected using macros like `@code_warntype`\n", 743 | "- Typing is crucial to exploit specialised structure\n", 744 | "- Type annotations are irrelevant for performance" 745 | ] 746 | } 747 | ], 748 | "metadata": { 749 | "@webio": { 750 | "lastCommId": null, 751 | "lastKernelId": null 752 | }, 753 | "kernelspec": { 754 | "display_name": "Julia 1.7.2", 755 | "language": "julia", 756 | "name": "julia-1.7" 757 | }, 758 | "language_info": { 759 | "file_extension": ".jl", 760 | "mimetype": "application/julia", 761 | "name": "julia", 762 | "version": "1.7.2" 763 | } 764 | }, 765 | "nbformat": 4, 766 | "nbformat_minor": 4 767 | } 768 | -------------------------------------------------------------------------------- /06_Composability_Code_Reuse.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Composable software from generic code\n", 8 | "\n", 9 | "In the previous notebook we produced the following sum function, which was basically en par in speed with C or numpy:" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": null, 15 | "metadata": {}, 16 | "outputs": [], 17 | "source": [ 18 | "function fastsum(v)\n", 19 | " result = zero(eltype(v))\n", 20 | " @simd for i in 1:length(v) # @simd enforces vectorisation in the loop\n", 21 | " @inbounds result += v[i] # @inbounds suppresses bound checks\n", 22 | " end\n", 23 | " result\n", 24 | "end" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": {}, 30 | "source": [ 31 | "But unlike a C implementation, we are not at all restricted to using a particular data type ... and this let's us do crazy things **even though the code is equally fast**:" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "## Composability\n", 39 | "\n", 40 | "... or the real power of Julia.\n", 41 | "\n", 42 | "\n", 43 | "(a) **Elevated precision** ... let's consider a nasty case:" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": null, 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "# A vector that sums to one by construction\n", 53 | "function generate(N)\n", 54 | " x = randn(N) .* exp.(10 .* randn(N))\n", 55 | " x = [x; -x; 1.0]\n", 56 | " x[sortperm(rand(length(x)))]\n", 57 | "end\n", 58 | " \n", 59 | "v = generate(10000);" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "fastsum(v)" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "... looks wrong, let's try elevated precision:" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": null, 81 | "metadata": {}, 82 | "outputs": [], 83 | "source": [ 84 | "fastsum(big.(v))" 85 | ] 86 | }, 87 | { 88 | "cell_type": "markdown", 89 | "metadata": {}, 90 | "source": [ 91 | "(b) **Tracking numerical error**" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": null, 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "import IntervalArithmetic\n", 101 | "\n", 102 | "fastsum(IntervalArithmetic.Interval.(v))" 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": {}, 108 | "source": [ 109 | "... ups clearly something wrong here! But now we know." 110 | ] 111 | }, 112 | { 113 | "cell_type": "markdown", 114 | "metadata": {}, 115 | "source": [ 116 | "(c) **Error propagation**" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": null, 122 | "metadata": {}, 123 | "outputs": [], 124 | "source": [ 125 | "using Measurements\n", 126 | "v = [1 ± 1e-10, 1.1 ± 1e-12, 1.2 ± 1e-10]\n", 127 | "fastsum(v)" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": {}, 133 | "source": [ 134 | "(d) **Unlocking new features**\n", 135 | "\n", 136 | "Let us solve:\n", 137 | "\n", 138 | "$$\\frac{du(t)}{dt} = -cu(t)$$" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": null, 144 | "metadata": {}, 145 | "outputs": [], 146 | "source": [ 147 | "using OrdinaryDiffEq\n", 148 | "\n", 149 | "# Half-life of Carbon-14 is 5730 years.\n", 150 | "c = 5.730\n", 151 | "\n", 152 | "# Setup\n", 153 | "u0 = 1.0\n", 154 | "tspan = (0.0, 1.0)\n", 155 | "\n", 156 | "# Define the problem\n", 157 | "radioactivedecay(u, p, t) = -c*u\n", 158 | "\n", 159 | "# Pass to solver\n", 160 | "prob = ODEProblem(radioactivedecay, u0, tspan)\n", 161 | "sol = solve(prob, Tsit5(), reltol=1e-8, abstol=1e-8);" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": null, 167 | "metadata": {}, 168 | "outputs": [], 169 | "source": [ 170 | "using Plots\n", 171 | "\n", 172 | "plot(sol.t, sol.u, ylabel=\"u(t)\", xlabel=\"t\", lw=2, legend=false)" 173 | ] 174 | }, 175 | { 176 | "cell_type": "markdown", 177 | "metadata": {}, 178 | "source": [ 179 | "Now what if we have measurement errors ..." 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [ 188 | "using OrdinaryDiffEq, Measurements, Plots\n", 189 | "\n", 190 | "# Half-life of Carbon-14 is 5730 years.\n", 191 | "c = 5.730 ± 2\n", 192 | "\n", 193 | "#Setup\n", 194 | "u0 = 1.0 ± 0.1\n", 195 | "tspan = (0.0, 1.0)\n", 196 | "\n", 197 | "#Define the problem\n", 198 | "radioactivedecay(u,p,t) = -c*u\n", 199 | "\n", 200 | "#Pass to solver\n", 201 | "prob = ODEProblem(radioactivedecay,u0,tspan)\n", 202 | "sol = solve(prob, Tsit5(), reltol=1e-8, abstol=1e-8);\n", 203 | "\n", 204 | "plot(sol.t, sol.u, ylabel=\"u(t)\", xlabel=\"t\", lw=2, legend=false)" 205 | ] 206 | }, 207 | { 208 | "cell_type": "markdown", 209 | "metadata": {}, 210 | "source": [ 211 | "Note that, in some sense, **Julia implemented that feature by itself**.\n", 212 | "\n", 213 | "The authors of Measurements.jl and DifferentialEquations.jl never had any collaboration on this.\n", 214 | "\n", 215 | "It **just works**." 216 | ] 217 | }, 218 | { 219 | "cell_type": "markdown", 220 | "metadata": {}, 221 | "source": [ 222 | "## Generic programming + multiple dispatch\n", 223 | "\n", 224 | "So how does this work?\n", 225 | "\n", 226 | "Julia's ecosystem is centred around convenient abstractions *by convention* (**duck typing**).\n", 227 | "\n", 228 | "- As a **user of a type** we can easily pick within the type hierarchy the most generic type, which satisfies all our needs. If we do not need the precise details of the `Array` implementation, we indicate our **algorithm** works with `AbstractArray`s. The resulting **generic implementation** immediately gives us access to a large range of datatypes.\n", 229 | "\n", 230 | "- As the **provider of a type** we only specify that our object behaves like an `AbstractArray`. Based on providing only a small set of functions in whatever optimised way we want our **new type works with a large range of algorithms**.\n", 231 | "\n", 232 | "In combination with **multiple dispatch** this leads to an ([unreasonable](https://www.youtube.com/watch?v=kc9HwsxE1OY)) amount of code reuse, for example:\n", 233 | "\n", 234 | "1. **Sharing types** (DataStructures.jl, OrderedCollections.jl, StaticArrays.jl, Colors.jl, Measurements.jl)\n", 235 | "2. **Sharing algorithms** (StatsBase.jl, SortingAlgorithms.jl, GenericLinearAlgebra.jl)" 236 | ] 237 | }, 238 | { 239 | "cell_type": "markdown", 240 | "metadata": {}, 241 | "source": [ 242 | "### Code decoupling by multiple dispatch\n", 243 | "\n", 244 | "Now you might rightfully wonder why I emphasised *multiple dispatch* here. After all this sounds very much just like the generic programming available in other languages. Let me illustrate this with an example." 245 | ] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": {}, 250 | "source": [ 251 | "- We consider the case of a sparse compressed-stored column matrix type `SparseMatrixCSC`, which is available in the `SparseArrays` Julia standard library. Since we care about precision we want to use the `Double64` floating-point type (from `DoubleFloats`), which implements a fast datatype for higher-precision floating-point arithmetic.\n", 252 | "\n", 253 | "- To make these two datastructures interoperate, we need a set of methods. We consider here the scalar-matrix multiplication `*(factor::Double64, matrix::SparseMatrixCSC)`.\n", 254 | "\n", 255 | "- In many languages such as Python, implementing such a binary operator translates to a special operator function. In the Python case this is `__mul__`, which needs to be implemented for the LHS type, i.e. `Double64`. In other words which method to call for `*` is decided solely by considering the type of the *first* argument, hence the name *single dispatch*.\n", 256 | "\n", 257 | "- This is not always the most clever thing to do. In our case one could well argue that `SparseMatrixCSC` should be the place where the operator should be implemented, because the net effect will be to scale this matrix. Amongst other reasons this is why Python rolled out a second set of functions, called `__rmul__`, which dispatch on the *second* argument. Problem solved?\n", 258 | "\n", 259 | "- Well, not really. The issue is that the algorithm in either case ends up being closely coupled to one of the data structures. Moreover since these datastructures live in third-party packages (that know nothing from each other) one either needs to either\n", 260 | " 1. Implement `__rmul__` in `SparseMatrixCSC` and add a dependency `SparseArrays` -> `Double64`.\n", 261 | " 2. Implement `__mul__` in `Double64`: Same thing in reverse\n", 262 | " 3. Derive off e.g. `SparseMatrixCSC` in my code and implement there. Works, but now my code likely ends up in close coupling to the internals of the `SparseArrays` package.\n", 263 | "\n", 264 | "The aforementioned problem is known as the *binary method problem* and is elegantly solved by multiple dispatch, because both arguments are treated on the same footing. So in Julia I just need to do\n", 265 | "```\n", 266 | "import Base: *\n", 267 | "*(n::Double64, str::SparseMatrixCSC) = ...\n", 268 | "```\n", 269 | "in my code, without changing any `Double64` or `SparseMatrixCSC` code. Notice how again algorithm and data types are well-separated and can be independently distributed as packages.\n", 270 | "\n", 271 | "- As an asided In C++ the very problem of implementing multiplication operators is not so prominent, because `operator*` can also be a free function. But: For other binary operations (like type conversion) the problem is exactly the same: As soon as two non-standard third-party types are involved and my code needs to connect them, I have to make one more special than the other." 272 | ] 273 | }, 274 | { 275 | "cell_type": "markdown", 276 | "metadata": {}, 277 | "source": [ 278 | "### Workflow advantages by multiple dispatch\n", 279 | "\n", 280 | "Commonly in Julia for each algorithm there exists a generic fallback implementation, which is slow, but always works. An example is the `det`-function we saw earlier. For specific types, where more structure is known, appropriate specialisations are available (like for `StaticArray`s or `Tridiagonal` matrices),\n", 281 | "which more efficient.\n", 282 | "\n", 283 | "Let's assume now we program an algorithm, which involves a determinant, e.g. the simple normalisation function\n", 284 | "```\n", 285 | "normalise(A) = A / det(A)\n", 286 | "```\n", 287 | "(where of course we assume `A` to be nonsingular). We note:\n", 288 | "- As a user, if I know the structure of my matrix is special, I mark it e.g. with the `Tridiagonal` type. When running `normalise` on it, the `det`-method specialised for `Tridiagonal` will be used, so that the best possible performance results.\n", 289 | "- As a user, if I do not know the structure of the matrix, I leave it general. Then the slow `det` will still give me a sensible result.\n", 290 | "- As a programmer, writing the `normalise` algorithm I do not need to know what kind of matrix my users will pass me, since the dispatch of `det` will assure the best thing happens. Most notably even if I have standard matrices in mind when writing the code, my `normalise` will still work if a passed matrix `A` is non-standard, because the fallback `det` will do the job. \n", 291 | "- As a programmer, writing a custom matrix `A` which adheres to all conventions of the `AbstractMatrix` interface, I instantly have a working `normalise` function even if I could be more clever about it. This way I can rapidly prototype my fancy new matrix type and only later implement the `det` function once I see it's necessary to achieve better performance.\n", 292 | "- This allows for a gradual performance increase in codes by **making only local changes** where profiling flags up issues.\n", 293 | "\n", 294 | "- This leads to the general Julia mantra:\n", 295 | "\n", 296 | "
First get it to work, then get it to work fast.
" 297 | ] 298 | }, 299 | { 300 | "cell_type": "markdown", 301 | "metadata": {}, 302 | "source": [ 303 | "##### More details\n", 304 | "- PhD Thesis from Jeff Bezanson\n", 305 | "- https://www.youtube.com/watch?v=kc9HwsxE1OY" 306 | ] 307 | }, 308 | { 309 | "cell_type": "markdown", 310 | "metadata": {}, 311 | "source": [ 312 | "## Takeaways\n", 313 | "\n", 314 | "- Generic code in julia can be fast\n", 315 | "- Everyone can write generic code and everyone should do it.\n", 316 | "- Multiple dispatch: Code reuse and emergent features\n", 317 | "- Multiple dispatch allows gradual, local performance engineering." 318 | ] 319 | } 320 | ], 321 | "metadata": { 322 | "@webio": { 323 | "lastCommId": null, 324 | "lastKernelId": null 325 | }, 326 | "kernelspec": { 327 | "display_name": "Julia 1.7.2", 328 | "language": "julia", 329 | "name": "julia-1.7" 330 | }, 331 | "language_info": { 332 | "file_extension": ".jl", 333 | "mimetype": "application/julia", 334 | "name": "julia", 335 | "version": "1.7.2" 336 | } 337 | }, 338 | "nbformat": 4, 339 | "nbformat_minor": 4 340 | } 341 | -------------------------------------------------------------------------------- /07_Linear_Algebra_Profiling.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "initial-movement", 6 | "metadata": {}, 7 | "source": [ 8 | "# Linear Algebra, Precision and Profiling\n", 9 | "\n", 10 | "Julias generic way of implementing algorithms often makes it easy to explore different storage schemes, elevated or reduced precision or to try acceleration hardware like a GPU. I want to present a few illustrating examples on a real-world iterative algorithm to show you how little effort is needed to give these things a try in Julia. I will also show in one example how one can track performance by profiling and understand what should be done to improve an algorithm at hand." 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "id": "883d1c41", 16 | "metadata": {}, 17 | "source": [ 18 | "## Linear Algebra\n", 19 | "\n", 20 | "For dense and sparse arrays, all important linear algebra routines are available in the `LinearAlgebra`. This includes common tasks such as\n", 21 | "- `qr` (also pivoted)\n", 22 | "- `cholesky` (also pivoted)\n", 23 | "- `eigen`, `eigvals`, `eigvecs` (compute eigenpairs, values, vectors)\n", 24 | "- `factorize` (for computing matrix factorisations)\n", 25 | "- `inv` (invert a matrix)\n", 26 | "\n", 27 | "All these methods are both implemented for generic matrices (all `AbstractMatrices`) and specialised for specific kinds. For example `factorize` is intended to compute a clever factorisation for solving linear systems. What it does depends on the matrix properties:" 28 | ] 29 | }, 30 | { 31 | "cell_type": "code", 32 | "execution_count": null, 33 | "id": "f4589840", 34 | "metadata": {}, 35 | "outputs": [], 36 | "source": [ 37 | "using LinearAlgebra\n", 38 | "using SparseArrays" 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": null, 44 | "id": "a5d91451", 45 | "metadata": { 46 | "scrolled": true 47 | }, 48 | "outputs": [], 49 | "source": [ 50 | "# Random real matrix -> will do an LU\n", 51 | "A = randn(10, 10)\n", 52 | "@show typeof(factorize(A))\n", 53 | "\n", 54 | "# Real-symmetric matrix -> will do a Bunch-Kaufman\n", 55 | "Am = Symmetric(A + A')\n", 56 | "@show typeof(factorize(Am))\n", 57 | "\n", 58 | "# Symmetric tridiagonal -> will do a LDLt\n", 59 | "Am = SymTridiagonal(A + A')\n", 60 | "@show typeof(factorize(Am))\n", 61 | "\n", 62 | "# Random sparse matrix -> will do sparse LU\n", 63 | "S = sprandn(50, 50, 0.3)\n", 64 | "@show typeof(factorize(S))\n", 65 | "\n", 66 | "# ... and so on ..." 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "id": "aeedb7c2", 72 | "metadata": {}, 73 | "source": [ 74 | "The all share a common interface, such that an algorithm like" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": null, 80 | "id": "c9f31e67", 81 | "metadata": {}, 82 | "outputs": [], 83 | "source": [ 84 | "function solve_many(A, xs)\n", 85 | " F = factorize(A)\n", 86 | " [F \\ rhs for rhs in xs]\n", 87 | "end" 88 | ] 89 | }, 90 | { 91 | "cell_type": "markdown", 92 | "id": "e201cf42", 93 | "metadata": {}, 94 | "source": [ 95 | "will automatically work for sparse arrays and dense arrays and is furthermore independent of the floating-point type.\n", 96 | "\n", 97 | "##### More details\n", 98 | "- https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/" 99 | ] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "id": "2ba0c729", 104 | "metadata": {}, 105 | "source": [ 106 | "## Use case: A generic Davidson\n", 107 | "\n", 108 | "Let's try this in a more realistic algorithm.\n", 109 | "A simple Davidson algorithm can be implemented quite concisely:" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": null, 115 | "id": "general-computer", 116 | "metadata": {}, 117 | "outputs": [], 118 | "source": [ 119 | "using LinearAlgebra\n", 120 | "\n", 121 | "qrortho(X::Array) = Array(qr(X).Q)\n", 122 | "qrortho(X, Y) = qrortho(X - Y * Y'X)\n", 123 | "\n", 124 | "function rayleigh_ritz(X::Array, AX::Array, N)\n", 125 | " F = eigen(Hermitian(X'AX))\n", 126 | " F.values[1:N], F.vectors[:,1:N]\n", 127 | "end\n", 128 | "\n", 129 | "function davidson(A, SS::AbstractArray; tol=1e-5, maxsubspace=8size(SS, 2), verbose=true)\n", 130 | " m = size(SS, 2)\n", 131 | " for i in 1:100\n", 132 | " Ass = A * SS\n", 133 | " rvals, rvecs = rayleigh_ritz(SS, Ass, m)\n", 134 | " Ax = Ass * rvecs\n", 135 | "\n", 136 | " R = Ax - SS * rvecs * Diagonal(rvals)\n", 137 | " if norm(R) < tol\n", 138 | " return rvals, SS * rvecs\n", 139 | " end\n", 140 | "\n", 141 | " verbose && println(i, \" \", size(SS, 2), \" \", norm(R))\n", 142 | "\n", 143 | " # Use QR to orthogonalise the subspace.\n", 144 | " if size(SS, 2) + m > maxsubspace\n", 145 | " SS = qrortho([SS*rvecs R])\n", 146 | " else\n", 147 | " SS = qrortho([SS R])\n", 148 | " end\n", 149 | " end\n", 150 | " error(\"not converged.\")\n", 151 | "end" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": null, 157 | "id": "accompanied-homework", 158 | "metadata": {}, 159 | "outputs": [], 160 | "source": [ 161 | "nev = 2\n", 162 | "A = randn(50, 50); A = A + A' + 5I;\n", 163 | "\n", 164 | "# Generate two random orthogonal guess vectors\n", 165 | "x0 = qrortho(randn(size(A, 2), nev))\n", 166 | "\n", 167 | "# Run the problem\n", 168 | "davidson(A, x0)" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": null, 174 | "id": "competent-homework", 175 | "metadata": {}, 176 | "outputs": [], 177 | "source": [ 178 | "# Mixed precision!\n", 179 | "using GenericLinearAlgebra\n", 180 | "\n", 181 | "λ, v = davidson(Matrix{Float32}(A), Float32.(x0), tol=1e-3)\n", 182 | "println()\n", 183 | "λ, v = davidson(Matrix{Float64}(A), v, tol=1e-13)\n", 184 | "println()\n", 185 | "λ, v = davidson(Matrix{BigFloat}(A), v, tol=1e-25)\n", 186 | "λ" 187 | ] 188 | }, 189 | { 190 | "cell_type": "code", 191 | "execution_count": null, 192 | "id": "b242c18f", 193 | "metadata": {}, 194 | "outputs": [], 195 | "source": [ 196 | "using SparseArrays\n", 197 | "nev = 2\n", 198 | "spA = sprandn(100, 100, 0.3); spA = spA + spA' + 2I\n", 199 | "\n", 200 | "spA" 201 | ] 202 | }, 203 | { 204 | "cell_type": "code", 205 | "execution_count": null, 206 | "id": "separate-moore", 207 | "metadata": {}, 208 | "outputs": [], 209 | "source": [ 210 | "spx0 = randn(size(spA, 2), nev)\n", 211 | "spx0 = Array(qr(spx0).Q)\n", 212 | "\n", 213 | "davidson(spA, spx0, tol=1e-6)" 214 | ] 215 | }, 216 | { 217 | "cell_type": "code", 218 | "execution_count": null, 219 | "id": "tribal-chase", 220 | "metadata": {}, 221 | "outputs": [], 222 | "source": [ 223 | "# ... runs with GPUs !\n", 224 | "using CUDA\n", 225 | "\n", 226 | "qrortho(X::CuArray) = CuArray(qr(X).Q)\n", 227 | "function rayleigh_ritz(X::CuArray, AX::CuArray, N)\n", 228 | " values, vectors = CUDA.CUSOLVER.syevd!('V', 'U', X'AX)\n", 229 | " values[1:N], vectors[:, 1:N]\n", 230 | "end" 231 | ] 232 | }, 233 | { 234 | "cell_type": "code", 235 | "execution_count": null, 236 | "id": "centered-elite", 237 | "metadata": {}, 238 | "outputs": [], 239 | "source": [ 240 | "davidson(cu(A), cu(x0))" 241 | ] 242 | }, 243 | { 244 | "cell_type": "markdown", 245 | "id": "4a02a6dd", 246 | "metadata": {}, 247 | "source": [ 248 | "... but actually the performance is overall not that good out of the box, because we're doing a lot of copying and elementwise access in our naive algorithm, which is especially bad for the GPU version." 249 | ] 250 | }, 251 | { 252 | "cell_type": "markdown", 253 | "id": "9d0a6e28", 254 | "metadata": {}, 255 | "source": [ 256 | "## Iterative methods\n", 257 | "\n", 258 | "Instead of implementing iterative methods such as the Davidson diagonalisation ourselves, we can also build upon existing packages for standard linear algebra, such as [IterativeSolvers.jl](https://github.com/JuliaLinearAlgebra/IterativeSolvers.jl), [KrylovKit.jl](https://github.com/Jutho/KrylovKit.jl), [Krylov.jl](https://github.com/JuliaSmoothOptimizers/Krylov.jl).\n", 259 | "\n", 260 | "For example instead of hand-coding a Davidson, we could use IterativeSolvers' LOBPCG implementation." 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "execution_count": null, 266 | "id": "d822e8d7", 267 | "metadata": {}, 268 | "outputs": [], 269 | "source": [ 270 | "using IterativeSolvers" 271 | ] 272 | }, 273 | { 274 | "cell_type": "code", 275 | "execution_count": null, 276 | "id": "2689d0e0", 277 | "metadata": {}, 278 | "outputs": [], 279 | "source": [ 280 | "largest = false\n", 281 | "IterativeSolvers.lobpcg(A, largest, x0)" 282 | ] 283 | }, 284 | { 285 | "cell_type": "markdown", 286 | "id": "aba4370b", 287 | "metadata": {}, 288 | "source": [ 289 | "which works seamlessly with GPUs as well:" 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": null, 295 | "id": "2932ce33", 296 | "metadata": {}, 297 | "outputs": [], 298 | "source": [ 299 | "largest = false\n", 300 | "IterativeSolvers.lobpcg(cu(A), largest, cu(x0))" 301 | ] 302 | }, 303 | { 304 | "cell_type": "markdown", 305 | "id": "2181b3d5", 306 | "metadata": {}, 307 | "source": [ 308 | "## Profiling and timing measurements\n", 309 | "\n", 310 | "Now how would one go about improving this piece of code?\n", 311 | "\n", 312 | "The best way forward is to obtain an idea *where* the computational time is spent and then think where we could *locally* improve. We already saw the `@btime` macro (from [BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) for getting accurate timing measurements on single instructions. Let's see what other options there are.\n", 313 | "\n", 314 | "For our tests we will use this piece of code:" 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": null, 320 | "id": "f12fa526", 321 | "metadata": {}, 322 | "outputs": [], 323 | "source": [ 324 | "function myfunction(n)\n", 325 | " for i = 1:n\n", 326 | " A = randn(100,100,20)\n", 327 | " m = maximum(A)\n", 328 | " Am = mapslices(sum, A; dims=2)\n", 329 | " B = A[:,:,5]\n", 330 | " Bsort = mapslices(sort, B; dims=1)\n", 331 | " b = rand(100)\n", 332 | " C = B.*b\n", 333 | " end\n", 334 | "end" 335 | ] 336 | }, 337 | { 338 | "cell_type": "markdown", 339 | "id": "8be8465c", 340 | "metadata": {}, 341 | "source": [ 342 | "### Profiling\n", 343 | "\n", 344 | "To profile this piece of code we will use Julia's builtin `Profile` package in combination with `ProfileView` as a grapical viewer. Some Julia editors (like VSCode) also have their own plugins to integrate with Julia's profiling capabilities, so worth to look out for this in your favourite editor!" 345 | ] 346 | }, 347 | { 348 | "cell_type": "code", 349 | "execution_count": null, 350 | "id": "e5b758fe", 351 | "metadata": {}, 352 | "outputs": [], 353 | "source": [ 354 | "using Profile\n", 355 | "using ProfileView" 356 | ] 357 | }, 358 | { 359 | "cell_type": "code", 360 | "execution_count": null, 361 | "id": "e0e032d8", 362 | "metadata": {}, 363 | "outputs": [], 364 | "source": [ 365 | "# Run once to compile everything ... this should be ignored\n", 366 | "ProfileView.@profview myfunction(1);" 367 | ] 368 | }, 369 | { 370 | "cell_type": "code", 371 | "execution_count": null, 372 | "id": "a79dbdd2", 373 | "metadata": { 374 | "scrolled": true 375 | }, 376 | "outputs": [], 377 | "source": [ 378 | "ProfileView.@profview myfunction(10);" 379 | ] 380 | }, 381 | { 382 | "cell_type": "markdown", 383 | "id": "e0adef51", 384 | "metadata": {}, 385 | "source": [ 386 | "**Note:** ProfileView does not always work so well with Jupyter Notebooks or Jupyterlab (but it's great from the REPL). An alternative is ProfileSVG:" 387 | ] 388 | }, 389 | { 390 | "cell_type": "code", 391 | "execution_count": null, 392 | "id": "1577a951", 393 | "metadata": {}, 394 | "outputs": [], 395 | "source": [ 396 | "using ProfileSVG" 397 | ] 398 | }, 399 | { 400 | "cell_type": "code", 401 | "execution_count": null, 402 | "id": "ea77234c", 403 | "metadata": { 404 | "scrolled": false 405 | }, 406 | "outputs": [], 407 | "source": [ 408 | "ProfileSVG.@profview myfunction(1)\n", 409 | "ProfileSVG.@profview myfunction(10)" 410 | ] 411 | }, 412 | { 413 | "cell_type": "markdown", 414 | "id": "d26af1a1", 415 | "metadata": {}, 416 | "source": [ 417 | "So how should one interpret this?\n", 418 | "- The horizontal direction is the time spent.\n", 419 | "- The vertical diretcion is the depth of the call stack.\n", 420 | "\n", 421 | "What do we learn:\n", 422 | "- The `mapslices` calls are clearly the most expensive parts of the function we should worry most\n", 423 | "- The first call (`mapslices(sum, A; dims=2)`) is more expensive as it works on more data than `mapslices(sort, B; dims=1)`\n", 424 | "- There is a stack of calls to functions in sort.jl on the right. This is because in Julia sorting is implemented recursively (sort functions call themselves)\n", 425 | "\n", 426 | "It is worth noting that red is a special colour in these graphs, highlighting a runtime dispatch, which can be an indicator for a type instability (more details and how to cure this in [Projects/03_Performance_Engineering.ipynb](Projects/03_Performance_Engineering.ipynb).\n", 427 | "\n", 428 | "For more details, take a look at the [ProfileView](https://github.com/timholy/ProfileView.jl) website." 429 | ] 430 | }, 431 | { 432 | "cell_type": "markdown", 433 | "id": "5589da60", 434 | "metadata": {}, 435 | "source": [ 436 | "### High-level timings in TimerOutputs.jl\n", 437 | "\n", 438 | "[TimerOutputs.jl](https://github.com/KristofferC/TimerOutputs.jl) is great package to get a rough overview where time is spent. The idea is to annotate the code with simple tags, where timings are taken while the code is running. This is not for free, but if done at a high level cheap enough to be \"always on\":" 439 | ] 440 | }, 441 | { 442 | "cell_type": "code", 443 | "execution_count": null, 444 | "id": "5d218782", 445 | "metadata": {}, 446 | "outputs": [], 447 | "source": [ 448 | "using TimerOutputs\n", 449 | "\n", 450 | "const to = TimerOutput()\n", 451 | "function annotated_function(n)\n", 452 | " @timeit to \"loop\" for i = 1:n\n", 453 | " @timeit to \"initialisation\" A = randn(100,100,20)\n", 454 | " m = maximum(A)\n", 455 | " @timeit to \"mapslices on A\" Am = mapslices(sum, A; dims=2)\n", 456 | " B = A[:,:,5]\n", 457 | " @timeit to \"mapslices on B\" Bsort = mapslices(sort, B; dims=1)\n", 458 | " b = rand(100)\n", 459 | " C = B.*b\n", 460 | " end\n", 461 | "end" 462 | ] 463 | }, 464 | { 465 | "cell_type": "code", 466 | "execution_count": null, 467 | "id": "6b5db0b3", 468 | "metadata": {}, 469 | "outputs": [], 470 | "source": [ 471 | "reset_timer!(to)\n", 472 | "annotated_function(10)\n", 473 | "to" 474 | ] 475 | }, 476 | { 477 | "cell_type": "markdown", 478 | "id": "6050b458", 479 | "metadata": {}, 480 | "source": [ 481 | "##### More details\n", 482 | "- https://docs.julialang.org/en/v1/manual/profile/\n", 483 | "- https://github.com/kimikage/ProfileSVG.jl\n", 484 | "- https://github.com/JuliaCI/BenchmarkTools.jl\n", 485 | "- https://github.com/KristofferC/TimerOutputs.jl" 486 | ] 487 | }, 488 | { 489 | "cell_type": "markdown", 490 | "id": "dc0c746f", 491 | "metadata": {}, 492 | "source": [ 493 | "## Takeaways\n", 494 | "\n", 495 | "- Julia has extensive builtin support for standard dense linear algebra (SVG, Diagonalisation, Linear systems)\n", 496 | "- Plenty of packages complement this by iterative methods\n", 497 | "- Profiling and timing can take place at various levels (From benchmarking individual statements to profiling a whole code)" 498 | ] 499 | } 500 | ], 501 | "metadata": { 502 | "@webio": { 503 | "lastCommId": null, 504 | "lastKernelId": null 505 | }, 506 | "kernelspec": { 507 | "display_name": "Julia 1.7.2", 508 | "language": "julia", 509 | "name": "julia-1.7" 510 | }, 511 | "language_info": { 512 | "file_extension": ".jl", 513 | "mimetype": "application/julia", 514 | "name": "julia", 515 | "version": "1.7.2" 516 | } 517 | }, 518 | "nbformat": 4, 519 | "nbformat_minor": 5 520 | } 521 | -------------------------------------------------------------------------------- /08_Multithreading_Basics.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Multithreading" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "By default Julia only using a single thread. To start it with multiple threads we must tell it explicitly:\n", 15 | "\n", 16 | "##### Command line argument\n", 17 | "\n", 18 | "```bash\n", 19 | "julia -t 4\n", 20 | "```\n", 21 | "\n", 22 | "or\n", 23 | "\n", 24 | "```bash\n", 25 | "julia --threads 4\n", 26 | "```\n", 27 | "\n", 28 | "##### Environmental variable\n", 29 | "\n", 30 | "On Linux/MacOS:\n", 31 | "\n", 32 | "```bash\n", 33 | "export JULIA_NUM_THREADS=4\n", 34 | "```\n", 35 | "\n", 36 | "On Windows:\n", 37 | "\n", 38 | "```bash\n", 39 | "set JULIA_NUM_THREADS=4\n", 40 | "```\n", 41 | "\n", 42 | "Afterwards start julia *in the same terminal*.\n", 43 | "\n", 44 | "##### Jupyter kernel\n", 45 | "\n", 46 | "You can also create a *Jupyter kernel* for multithreaded Julia:\n", 47 | "\n", 48 | "```julia\n", 49 | "using IJulia\n", 50 | "installkernel(\"Julia (4 threads)\", \"--project=@.\", env=Dict(\"JULIA_NUM_THREADS\"=>\"4\"))\n", 51 | "```\n", 52 | "\n", 53 | "*Note:* This has to be manually redone for every new Julia version and you have to restart your Jupyter process to see an effect." 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "To check this has worked we we use:" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": null, 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "Threads.nthreads()" 70 | ] 71 | }, 72 | { 73 | "cell_type": "markdown", 74 | "metadata": {}, 75 | "source": [ 76 | "## `Threads.@spawn`" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "The `Threads.@spawn` macro dynamically spawns a new thread to execute a command in the background. Programmatically, it creates a `Task` and puts it on the todo-list. Whenever a thread is free, the task is dynamically assigned to a thread and executing the work starts." 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": {}, 90 | "outputs": [], 91 | "source": [ 92 | "Threads.@spawn println(\"test\")" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "**Important:** `Threads.@spawn` returns the created task *immediately*, but we might have to wait until the task is done and fetch the result later:" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": null, 105 | "metadata": {}, 106 | "outputs": [], 107 | "source": [ 108 | "t = Threads.@spawn begin\n", 109 | " sleep(3)\n", 110 | " 4\n", 111 | "end\n", 112 | "# We immediately get here\n", 113 | "@time fetch(t) # This waits until the task is done" 114 | ] 115 | }, 116 | { 117 | "cell_type": "markdown", 118 | "metadata": {}, 119 | "source": [ 120 | "To prevent the immediate return, we need to explicitly synchronise the execution using an `@sync` macro barrier.\n", 121 | "For example:" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "metadata": {}, 128 | "outputs": [], 129 | "source": [ 130 | "@sync begin\n", 131 | " t = Threads.@spawn begin\n", 132 | " sleep(3)\n", 133 | " 4\n", 134 | " end\n", 135 | "end\n", 136 | "\n", 137 | "@time fetch(t) # No need to wait, the task is already done" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "## Filling an array in parallel\n", 145 | "\n", 146 | "Now, let's use this to actually parallelise something: We will fill an array in parallel:" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": null, 152 | "metadata": {}, 153 | "outputs": [], 154 | "source": [ 155 | "function fill_array_parallel(a)\n", 156 | " @sync for i in 1:length(a)\n", 157 | " Threads.@spawn a[i] = Threads.threadid()\n", 158 | " end\n", 159 | " a\n", 160 | "end\n", 161 | "\n", 162 | "a = zeros(Threads.nthreads()*10);\n", 163 | "fill_array_parallel(a)" 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": null, 169 | "metadata": {}, 170 | "outputs": [], 171 | "source": [ 172 | "@show count(a .== 1.0)\n", 173 | "@show count(a .== 2.0)\n", 174 | "@show count(a .== 3.0)\n", 175 | "@show count(a .== 4.0)" 176 | ] 177 | }, 178 | { 179 | "cell_type": "markdown", 180 | "metadata": {}, 181 | "source": [ 182 | "Note: Due to the **dynamic scheduling** some threads actually do more work (more values of i) than others!" 183 | ] 184 | }, 185 | { 186 | "cell_type": "markdown", 187 | "metadata": {}, 188 | "source": [ 189 | "## Nesting threading\n", 190 | "\n", 191 | "A key motion in the Julia ecosystem is to support **nested threading**:" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": null, 197 | "metadata": {}, 198 | "outputs": [], 199 | "source": [ 200 | "function threaded_fun()\n", 201 | " x = Threads.threadid()\n", 202 | " Threads.@spawn println(\"job1\", \" (spawned from $x, processed by $(Threads.threadid()))\")\n", 203 | " Threads.@spawn println(\"job2\", \" (spawned from $x, processed by $(Threads.threadid()))\")\n", 204 | " Threads.@spawn println(\"job3\", \" (spawned from $x, processed by $(Threads.threadid()))\")\n", 205 | "end" 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": {}, 212 | "outputs": [], 213 | "source": [ 214 | "@sync for i in 1:Threads.nthreads()\n", 215 | " Threads.@spawn threaded_fun()\n", 216 | "end" 217 | ] 218 | }, 219 | { 220 | "cell_type": "markdown", 221 | "metadata": {}, 222 | "source": [ 223 | "The key point about this is that in this way the threading of different layers of functions does not interfer by causing more threads to be spawned than there are workers (CPU cores).\n", 224 | "\n", 225 | "The issue happens rather easily whenever a parallelised routine like `threaded_fun` (e.g. a numerical integration routine) is again called from a parallelised outer loop (e.g. a solver). To avoid the problem one needs to introduce some kind of coupling between the routines to communicate to the inner routine (`threaded_fun`) how many threads it may use. To avoid the need to do this explicitly, Julia implemented has decided to base its threading mostly on dynamic scheduling and the `@spawn` formalism." 226 | ] 227 | }, 228 | { 229 | "cell_type": "markdown", 230 | "metadata": {}, 231 | "source": [ 232 | "## Threading takes extra care: Parallel summation\n", 233 | "\n", 234 | "We consider the case of a parallel summation" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": null, 240 | "metadata": {}, 241 | "outputs": [], 242 | "source": [ 243 | "function mysum(xs)\n", 244 | " s = zero(eltype(xs))\n", 245 | " for x in xs\n", 246 | " s += x\n", 247 | " end\n", 248 | " s\n", 249 | "end" 250 | ] 251 | }, 252 | { 253 | "cell_type": "code", 254 | "execution_count": null, 255 | "metadata": {}, 256 | "outputs": [], 257 | "source": [ 258 | "function mysum_parallel_naive(xs)\n", 259 | " s = zero(eltype(xs))\n", 260 | " @sync for x in xs\n", 261 | " Threads.@spawn (s += x)\n", 262 | " end\n", 263 | " s\n", 264 | "end" 265 | ] 266 | }, 267 | { 268 | "cell_type": "code", 269 | "execution_count": null, 270 | "metadata": {}, 271 | "outputs": [], 272 | "source": [ 273 | "xs = rand(100_000);" 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "execution_count": null, 279 | "metadata": {}, 280 | "outputs": [], 281 | "source": [ 282 | "@show sum(xs);\n", 283 | "@show mysum(xs);\n", 284 | "@show mysum_parallel_naive(xs);" 285 | ] 286 | }, 287 | { 288 | "cell_type": "markdown", 289 | "metadata": {}, 290 | "source": [ 291 | "Hmmm ... the problem is a so-called **race condition**, a clash due to the parallel writing access from multiple threads.\n", 292 | "\n", 293 | "One way to solve this is by using [Atomic Operations](https://docs.julialang.org/en/v1/manual/multi-threading/#Atomic-Operations):" 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": null, 299 | "metadata": {}, 300 | "outputs": [], 301 | "source": [ 302 | "import Base.Threads: Atomic, atomic_add!\n", 303 | "\n", 304 | "function mysum_parallel_atomics(xs)\n", 305 | " T = eltype(xs)\n", 306 | " s = Atomic{T}(zero(T))\n", 307 | " @sync for x in xs\n", 308 | " Threads.@spawn atomic_add!(s, x)\n", 309 | " end\n", 310 | " s[]\n", 311 | "end" 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": null, 317 | "metadata": {}, 318 | "outputs": [], 319 | "source": [ 320 | "@show mysum(xs);\n", 321 | "@show mysum_parallel_atomics(xs);" 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": null, 327 | "metadata": {}, 328 | "outputs": [], 329 | "source": [ 330 | "@btime mysum($xs);\n", 331 | "@btime mysum_parallel_atomics($xs);\n", 332 | "@btime mysum_parallel_naive($xs);" 333 | ] 334 | }, 335 | { 336 | "cell_type": "markdown", 337 | "metadata": {}, 338 | "source": [ 339 | "**Note:** Atomics are generally bad. Don't use this paradigm in production unless you know what you are doing. Use FLoops.jl (see below)." 340 | ] 341 | }, 342 | { 343 | "cell_type": "markdown", 344 | "metadata": {}, 345 | "source": [ 346 | "## Is there no static scheduling option in Julia?\n", 347 | "\n", 348 | "Yes there is and it can sometimes be faster than dynamic threading:" 349 | ] 350 | }, 351 | { 352 | "cell_type": "code", 353 | "execution_count": null, 354 | "metadata": {}, 355 | "outputs": [], 356 | "source": [ 357 | "function mysum_parallel_threads(xs)\n", 358 | " T = eltype(xs)\n", 359 | " s = Atomic{T}(zero(T))\n", 360 | " Threads.@threads :static for x in xs\n", 361 | " atomic_add!(s, x)\n", 362 | " end\n", 363 | " s[]\n", 364 | "end" 365 | ] 366 | }, 367 | { 368 | "cell_type": "code", 369 | "execution_count": null, 370 | "metadata": {}, 371 | "outputs": [], 372 | "source": [ 373 | "@btime mysum_parallel_atomics($xs);\n", 374 | "@btime mysum_parallel_threads($xs);" 375 | ] 376 | }, 377 | { 378 | "cell_type": "markdown", 379 | "metadata": {}, 380 | "source": [ 381 | "While on a first look this has advantages in form of a 10-fold reduced speed, the disadvantages are that there is no nested threading and there can be severe load imbalancing since work is split statically at startup of the loop." 382 | ] 383 | }, 384 | { 385 | "cell_type": "markdown", 386 | "metadata": {}, 387 | "source": [ 388 | "## FLoops.jl: Easy and fast dynamic threads\n", 389 | "\n", 390 | "As a way out the Julia ecosystem has brought forward a number of carefully optimised packages for threaded execution based on *dynamic* scheduling. One example is [FLoops.jl](https://github.com/JuliaFolds/FLoops.jl). Our `mysum` function is parallelised using FLoops by just adding two macros:" 391 | ] 392 | }, 393 | { 394 | "cell_type": "code", 395 | "execution_count": null, 396 | "metadata": {}, 397 | "outputs": [], 398 | "source": [ 399 | "using FLoops\n", 400 | "\n", 401 | "function mysum_parallel_floops(xs)\n", 402 | " s = zero(eltype(xs))\n", 403 | " @floop for x in xs\n", 404 | " @reduce s += x\n", 405 | " end\n", 406 | " s\n", 407 | "end" 408 | ] 409 | }, 410 | { 411 | "cell_type": "markdown", 412 | "metadata": {}, 413 | "source": [ 414 | "Still it gives the right result and is faster than our statically scheduled `@threads` version:" 415 | ] 416 | }, 417 | { 418 | "cell_type": "code", 419 | "execution_count": null, 420 | "metadata": {}, 421 | "outputs": [], 422 | "source": [ 423 | "@show mysum(xs);\n", 424 | "@show mysum_parallel_floops(xs);" 425 | ] 426 | }, 427 | { 428 | "cell_type": "code", 429 | "execution_count": null, 430 | "metadata": {}, 431 | "outputs": [], 432 | "source": [ 433 | "@btime mysum_parallel_threads($xs);\n", 434 | "@btime mysum_parallel_floops($xs);" 435 | ] 436 | }, 437 | { 438 | "cell_type": "markdown", 439 | "metadata": {}, 440 | "source": [ 441 | "**Note:** The fact that `FLoops` is faster is a little misleading at first sight, but illustrates an important point nevertheless:\n", 442 | "\n", 443 | "- If *perfectly* written *statically scheduled* threads are faster than dynamically scheduled threads\n", 444 | "- But this requires deep insight to obtain optimal load balancing, careful use of atomics etc.\n", 445 | "- If you are not a parallelisation expert carefully optimised packages based on *dynamical scheduling* will likely be faster for your use case. The plain reason is that the *learning time* to understand all the neccessary tricks and the time needed to *fix all the subtle bugs* is not to be underestimated." 446 | ] 447 | }, 448 | { 449 | "cell_type": "markdown", 450 | "metadata": {}, 451 | "source": [ 452 | "### Takeaways\n", 453 | "- Julia's thread infrastructure is mostly based on *dynamic* threading\n", 454 | "- The advantages are thread nesting and better load balancing in cases where load per iteration is not uniform.\n", 455 | "- The disadvantage is a larger startup time per thread\n", 456 | "- Packages like FLoops.jl make it easy to write fast parallel code." 457 | ] 458 | }, 459 | { 460 | "cell_type": "markdown", 461 | "metadata": {}, 462 | "source": [ 463 | "##### More details\n", 464 | "- https://juliafolds.github.io/data-parallelism/" 465 | ] 466 | } 467 | ], 468 | "metadata": { 469 | "@webio": { 470 | "lastCommId": null, 471 | "lastKernelId": null 472 | }, 473 | "kernelspec": { 474 | "display_name": "Julia (4 threads) 1.7.2", 475 | "language": "julia", 476 | "name": "julia-(4-threads)-1.7" 477 | }, 478 | "language_info": { 479 | "file_extension": ".jl", 480 | "mimetype": "application/julia", 481 | "name": "julia", 482 | "version": "1.7.2" 483 | } 484 | }, 485 | "nbformat": 4, 486 | "nbformat_minor": 4 487 | } 488 | -------------------------------------------------------------------------------- /09_Useful_Packages.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Useful Packages\n", 8 | "\n", 9 | "The number of registered packages in the [General](https://github.com/JuliaRegistries/General/blob/master/Registry.toml) registry is quite vast (about 7100 on **16/02/2022**) and growing every day. So if you have a reasonably standard problem you want to solve, there is a good chance that someone has already made an effort. Even if that's not the case the composability between packages enabled by the Julia language implies that often building blocks from other packages can be easily reused as well. There is also a [great search engine](https://juliahub.com/ui/Packages) for Julia packages.\n", 10 | "\n", 11 | "To give you some ideas about the community and what is available, here is a non-exhaustive list, which is naturally influenced by my own background. I'm happy to adapt it, if you feel something is wrong / missing.\n", 12 | "\n", 13 | "Without further ado:" 14 | ] 15 | }, 16 | { 17 | "cell_type": "markdown", 18 | "metadata": {}, 19 | "source": [ 20 | "## Development and coding workflow\n", 21 | "\n", 22 | "See also the [workflow tips](https://docs.julialang.org/en/v1/manual/workflow-tips/)." 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": {}, 28 | "source": [ 29 | "- [Revise](https://github.com/timholy/Revise.jl) is one of the absolute must-have packages. It watches your filesystem for changes to a Julia package or script and reloads it automatically in the current REPL / Jupyter kernel. If you are the developer of a Julia package or usually find yourself coding in one terminal and using the code in another, this package is indispensable.\n", 30 | "- [JuliaFormatter](https://github.com/domluna/JuliaFormatter.jl): Basic automatic source-code formatter for Julia.\n", 31 | "- [Infiltrator](https://github.com/JuliaDebug/Infiltrator.jl): Drop into an interactive REPL at an arbitrary place in your code. Great for debugging and testing.\n", 32 | "- [Debugger](https://github.com/JuliaDebug/Debugger.jl): A gdb-like Julia debugger, which I use every day.\n", 33 | "- [Rebugger](https://github.com/timholy/Rebugger.jl): Another debugger. Preferred over Debugger by some people.\n", 34 | "- [Julia-VSCode](https://www.julia-vscode.org/): Not actually a package, but the development environment endorsed by Julia computing.\n", 35 | "- Surely there is Julia support for [your favourite editor](https://github.com/JuliaEditorSupport), too." 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "-----" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": {}, 48 | "source": [ 49 | "## Reports and source-code documentation\n", 50 | "- [Documenter](https://github.com/JuliaDocs/Documenter.jl) is the goto source code documentation tool for Julia.\n", 51 | "- [Literate](https://github.com/fredrikekre/Literate.jl/) allows to mix markdown and Julia, where the Julia code is automatically executed and resulting plots, data, tables are included into the final document. Supports generation of HTML files or Jupyter notebooks as well.\n", 52 | "- [IJulia](https://github.com/JuliaLang/IJulia.jl): Integration of Julia with Jupyter notebooks\n", 53 | "- [Pluto](https://github.com/fonsp/Pluto.jl): Interactive reactive notebooks" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "## More linear algebra\n", 61 | "\n", 62 | "- [IterativeSolvers](https://juliamath.github.io/IterativeSolvers.jl/dev/) is a rich library for iterative methods with the most important iterative solver algorithms (Conjugate Gradient, GMRES, BiCGStab, Chebychev iteration, ...)\n", 63 | "- [StaticArrays](https://github.com/JuliaArrays/StaticArrays.jl): Arrays of compile-time known size. Has a huge speed improvement for example when used for 3D vectors or 3D rotations.\n", 64 | "- [BlockArrays](https://github.com/JuliaArrays/BlockArrays.jl): Special array for block-wise storing or treating arrays \n", 65 | "- [LazyArrays](https://github.com/JuliaArrays/LazyArrays.jl): Makes array operations lazy.\n", 66 | "- [Tullio](https://github.com/mcabbott/Tullio.jl/): Extremely flexible package for higher-order tensor operations in an intuitive syntax. Support for GPU and AD.\n", 67 | "- [TensorOperations](https://github.com/Jutho/TensorOperations.jl): Similar to Tullio with a slightly varying feature set. Sometimes one, sometimes the other is better." 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": {}, 73 | "source": [ 74 | "## Automatic Differentiation (AD)\n", 75 | "- [Zygote](https://github.com/FluxML/Zygote.jl): Adjoint-mode AD\n", 76 | "- [ForwardDiff](https://github.com/JuliaDiff/ForwardDiff.jl): Forward-mode AD\n", 77 | "- [FiniteDifferences](https://github.com/JuliaDiff/FiniteDifferences.jl): Classic finite-difference approximations\n", 78 | "- [ChainRules](https://github.com/JuliaDiff/ChainRules.jl): Generic interface for automatice differentiation in Julia, allows to couple different kinds of AD" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": {}, 84 | "source": [ 85 | "## Visualisation and interaction \n", 86 | "- [Plots](https://github.com/JuliaPlots/Plots.jl) (Generic plotting interface metapackage)\n", 87 | "- [PyPlot](https://github.com/JuliaPy/PyPlot.jl) (Plotting via python and matplotlib)\n", 88 | "[PGFPlotsX](https://github.com/KristofferC/PGFPlotsX.jl) (Plotting via TeX, TikZ and pgf)\n", 89 | "- [Gladify](https://github.com/GiovineItalia/Gadfly.jl) (Pure Julia plotting implementation, widespread)\n", 90 | "- [PlotlyJS](https://github.com/sglyon/PlotlyJS.jl) (Ploting via Javascript and Plotly)\n", 91 | "- [GR](https://github.com/jheinen/GR.jl) (Plotting based on the GR framework, pretty fast)\n", 92 | "- [Makie](https://github.com/JuliaPlots/Makie.jl) (Extremely feature-rich; can plot on the GPU directly)" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "## Symbolic manipulation\n", 100 | "- [SymPy](https://github.com/JuliaPy/SymPy.jl): Julia wrapper of the popular python package\n", 101 | "- [Symbolics](https://juliasymbolics.org/): Pure-Julia Computer Algebra System (CAS)" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "## Standard file formats\n", 109 | "\n", 110 | "Plenty of standard file formats in scientific computing and elsewhere can be used in Julia:\n", 111 | "- [JSON](https://github.com/JuliaIO/JSON.jl) and [YAML](https://github.com/BioJulia/YAML.jl) (TOML is in the standard library)\n", 112 | "- [HDF5](https://github.com/JuliaIO/HDF5.jl)\n", 113 | "- [MAT](https://github.com/JuliaIO/MAT.jl): Support for Matlab's mat files\n", 114 | "- [NCDatasets](https://github.com/Alexander-Barth/NCDatasets.jl): Support for netcdf\n", 115 | "- [JLD](https://github.com/JuliaIO/JLD.jl) and [JLD2](https://github.com/JuliaIO/JLD2.jl): Julia's own file format to reproducably dump and load any Julia object." 116 | ] 117 | }, 118 | { 119 | "cell_type": "markdown", 120 | "metadata": {}, 121 | "source": [ 122 | "-----" 123 | ] 124 | }, 125 | { 126 | "cell_type": "markdown", 127 | "metadata": {}, 128 | "source": [ 129 | "## Chemistry and Materials Science\n", 130 | "\n", 131 | "\n", 132 | "- [DFTK](https://github.com/JuliaMolSim/DFTK.jl): Toolkit for density-functional theory and related models in extended systems\n", 133 | "- [Fermi](https://github.com/FermiQC/Fermi.jl): Small toolkit for molecular Post-HF calculations using Gaussian basis sets.\n", 134 | "- [AtomsBase](https://github.com/JuliaMolSim/AtomsBase.jl): Interfacing framework for molecular and periodic structures" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "metadata": {}, 140 | "source": [ 141 | "## PDEs and ODEs\n", 142 | "- Most of the tooling for solving differential equations in Julia is collected in the [JuliaDiffEq](https://juliadiffeq.org/) organisation. The most important package is [DifferentialEquations](https://github.com/JuliaDiffEq/DifferentialEquations.jl), where basically all the strings are collected. The package mostly concentrates on ordinary differential equations in various kinnds (random, stochastic, algebraic, mixed discrete / continuous ...)\n", 143 | "- For PDEs there are a couple of emerging implementations. We already briefly talked about [Gridap](https://github.com/gridap/Gridap.jl). But there also is [Ferrite](https://github.com/Ferrite-FEM/Ferrite.jl), which seems to have a more hackable design." 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "## Optimisation-related packages\n", 151 | "\n", 152 | "The optimisation community has adopted Julia rather early and the respective Julia libraries are at a very good shape.\n", 153 | "\n", 154 | "- The driving force behind the Julia optimisation community is the [JuliaOpt](https://www.juliaopt.org/) organisation with its working horse [JuMP](https://github.com/JuliaOpt/JuMP.jl). This package defines a metalanguage for optimisation problems, which can be combined with about 20 open-source or commercial optimisation solvers. A pretty exhaustive set of problem classes are supported: Linear programming, (mixed) integer programming, semidefinite programming, nonlinear programming, ...\n", 155 | "\n", 156 | "- For certain highly specific use cases a number of specialised packages have emerged:\n", 157 | " - [Optim](https://github.com/JuliaNLSolvers/Optim.jl) for continuous optimisation problems including simple manifolds\n", 158 | " - [LsqFit](https://github.com/JuliaNLSolvers/LsqFit.jl) for least-squares problems\n", 159 | " - [Roots](https://github.com/JuliaMath/Roots.jl) for finding roots of functions\n", 160 | " - [NLsolve](https://github.com/JuliaNLSolvers/NLsolve.jl) for general non-linear systems of equations" 161 | ] 162 | }, 163 | { 164 | "cell_type": "markdown", 165 | "metadata": {}, 166 | "source": [ 167 | "## Data science and statistics\n", 168 | "Apart from a few examples, the most important statistics and data science packages are associated with the [JuliaStats](https://github.com/JuliaStats) and [JuliaData](https://github.com/JuliaData) organisations. In particular there are:\n", 169 | "- `Statistics` in the standard library.\n", 170 | "- [DataFrames](https://github.com/JuliaData/DataFrames.jl): Package for working with tabular data (\"the pandas of Julia\")\n", 171 | "- [Queryverse](http://www.queryverse.org/): Metapackage for datascience and large-scale queries\n", 172 | "- [PrettyTables](https://github.com/ronisbr/PrettyTables.jl): Nicely formatted tables\n", 173 | "- [StatsKit](https://github.com/JuliaStats/StatsKit.jl): Metapackage loading a full suite of Statistics-related tools\n", 174 | "- [Distributions](https://github.com/JuliaStats/Distributions.jl): Defines standard distributions and their properties\n", 175 | "- [Turing](https://github.com/TuringLang/Turing.jl): Bayesian inference and probabilistic programming\n", 176 | "- [BayesianOptimization](https://github.com/jbrea/BayesianOptimization.jl)\n", 177 | "- [GaussianProcesses](https://github.com/STOR-i/GaussianProcesses.jl): Fit and work with Gaussian Processes\n", 178 | "- [Stheno](https://github.com/JuliaGaussianProcesses/Stheno.jl): Another Gaussian process package\n", 179 | "- [BAT.jl](https://github.com/bat/BAT.jl): Bayesian Analysis Toolkit (MCMC sampling, ...)\n", 180 | "- [BayesianLinearRegressors](https://github.com/JuliaGaussianProcesses/BayesianLinearRegressors.jl): Bayesian Linear Regression" 181 | ] 182 | }, 183 | { 184 | "cell_type": "markdown", 185 | "metadata": {}, 186 | "source": [ 187 | "## Machine Learning\n", 188 | "\n", 189 | "The most important machine-learning framework in Julia is [Flux](https://fluxml.ai/). It's ecosystem features packages such as:\n", 190 | "- [Flux.jl](https://github.com/FluxML/Flux.jl): The Flux backend\n", 191 | "- [FluxTraining.jl](https://github.com/FluxML/FluxTraining.jl): Tools for setting up and training neural nets.\n", 192 | "- See also the [Flux ecosystem](https://fluxml.ai/ecosystem.html) overview" 193 | ] 194 | } 195 | ], 196 | "metadata": { 197 | "@webio": { 198 | "lastCommId": "0bba521938544a5096c2349d28e65280", 199 | "lastKernelId": "e13bd870-ac07-446d-a160-212f94b7a565" 200 | }, 201 | "kernelspec": { 202 | "display_name": "Julia 1.7.2", 203 | "language": "julia", 204 | "name": "julia-1.7" 205 | }, 206 | "language_info": { 207 | "file_extension": ".jl", 208 | "mimetype": "application/julia", 209 | "name": "julia", 210 | "version": "1.7.2" 211 | } 212 | }, 213 | "nbformat": 4, 214 | "nbformat_minor": 2 215 | } 216 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2022 Michael F. Herbst 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so, 10 | subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 17 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 18 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 19 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 20 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /Project.toml: -------------------------------------------------------------------------------- 1 | [deps] 2 | AbstractGPs = "99985d1d-32ba-4be9-9821-2ec096f28918" 3 | AbstractTrees = "1520ce14-60c1-5f80-bbc7-55ef81b5835c" 4 | BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf" 5 | CUDA = "052768ef-5323-5732-b1bb-66c8b64840ba" 6 | DFTK = "acf6eb54-70d9-11e9-0013-234b7a5f5337" 7 | DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0" 8 | FLoops = "cc61a311-1640-44b5-9fba-1b764f453329" 9 | Flux = "587475ba-b771-5e3f-ad9e-33799f191a9c" 10 | FluxTraining = "7bf95e4d-ca32-48da-9824-f0dc5310474f" 11 | GenericLinearAlgebra = "14197337-ba66-59df-a3e3-ca00e7dcff7a" 12 | Gridap = "56d4f2e9-7ea1-5844-9cf6-b9c51ca7ce8e" 13 | IJulia = "7073ff75-c697-5162-941a-fcdaad2a7d2a" 14 | IntervalArithmetic = "d1acc4aa-44c8-5952-acd4-ba5d80a2a253" 15 | IterativeSolvers = "42fd0dbc-a981-5370-80f2-aaf504508153" 16 | LibGit2 = "76f85450-5226-5b5a-8eaa-529ad045b433" 17 | Libdl = "8f399da3-3557-5675-b5ff-fb832c97cbdb" 18 | LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" 19 | MLDataPattern = "9920b226-0b2a-5f5f-9153-9aa70a013f8b" 20 | MLDatasets = "eb30cadb-4394-5ae3-aed4-317e484a6458" 21 | Measurements = "eff96d63-e80a-5855-80a2-b1b0885c5ab7" 22 | ModelingToolkit = "961ee093-0014-501f-94e3-6117800e7a78" 23 | Optim = "429524aa-4258-5aef-a3af-852621145aeb" 24 | OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed" 25 | Pipe = "b98c9c47-44ae-5843-9183-064241ee97a0" 26 | Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80" 27 | Printf = "de0858da-6303-5e67-8744-51eddeeeb8d7" 28 | Profile = "9abbd945-dff8-562f-b5e8-e1ebf5ef1b79" 29 | ProfileSVG = "132c30aa-f267-4189-9183-c8a63c7e05e6" 30 | ProfileView = "c46f51b8-102a-5cf2-8d2c-8597cb0e0da7" 31 | PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0" 32 | SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" 33 | StaticArrays = "90137ffa-7385-5640-81b9-e52037218182" 34 | Stheno = "8188c328-b5d6-583d-959b-9690869a5511" 35 | TimerOutputs = "a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f" 36 | Traceur = "37b6cedf-1f77-55f8-9503-c64b63398394" 37 | Zygote = "e88e6eb3-aa80-5325-afca-941959d7151f" 38 | -------------------------------------------------------------------------------- /Projects/01_Dancing_Particles.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Dancing particles\n", 8 | "\n", 9 | "This project will be concerned with developing a simple code for the classical dynamics of interacting particles. Notice that there are already a number of Julia packages (such as [Molly](https://github.com/JuliaMolSim/Molly.jl) or [NBodySimulator](https://github.com/SciML/NBodySimulator.jl), which are able to do this in various contexts, so in practice one should rather resort to these packages instead of starting from scratch. Still, why not just get our hands dirty with what we know so far ;)." 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## Propagating a particle in time\n", 17 | "\n", 18 | "Assume we are given a potential $V(x)$ for a single\n", 19 | "particle with unit mass at position $x$. We want to propagate the particle in time.\n", 20 | "That's integrating the famous Newton's equation of motion\n", 21 | "$$ -\\frac{dV}{dx} = \\frac{d^2x}{dt^2}$$\n", 22 | "in time. Defining the force map\n", 23 | "$$ F_V = -\\frac{dV}{dx}$$\n", 24 | "this may be written equivalently:\n", 25 | "$$\\left\\{\\begin{array}{l}\n", 26 | "\\frac{dv}{dt} = F_V \\\\\n", 27 | "\\frac{dx}{dt} = v\n", 28 | "\\end{array}\\right. $$" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "At first we will just apply forward-Euler: We discrete in time using the interval\n", 36 | "$\\Delta t$, which leads to sequences $t_{n} = n \\Delta t + t_{0}$ and similarly $v_{n}$, $x_{n}$, etc:\n", 37 | "$$\\left\\{\\begin{array}{l}\n", 38 | "v_{n+1} = v_{n} + F_V(x_{n}) \\Delta t\\\\\n", 39 | "x_{n+1} = x_{n} + v_{n} \\Delta t\\\\\n", 40 | "\\end{array}\\right. .$$\n", 41 | "This is transformed to a Julia by pretty much copy-n-paste of the equations" 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": null, 47 | "metadata": {}, 48 | "outputs": [], 49 | "source": [ 50 | "function euler(F, Δt, xₙ, vₙ) \n", 51 | " xₙ₊₁ = xₙ + vₙ * Δt # Type as u \\_n \\_+ \\_1 ...\n", 52 | " vₙ₊₁ = vₙ + F(xₙ) * Δt\n", 53 | " xₙ₊₁, vₙ₊₁ # Return next position and velocity as a tuple\n", 54 | "end" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "metadata": {}, 60 | "source": [ 61 | "where `F` is a function to be passed to `euler`, for example:" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "metadata": {}, 68 | "outputs": [], 69 | "source": [ 70 | "F(x) = -x\n", 71 | "Δt = 0.1\n", 72 | "x₀ = 0\n", 73 | "v₀ = 1\n", 74 | "x₁, v₁ = euler(F, 0.1, x₀, v₀)" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": {}, 80 | "source": [ 81 | "Of course we can repeat this now and get a particle moving in phase space:" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "Δt = 0.1\n", 91 | "x, v = 0, 1\n", 92 | "for n = 1:10\n", 93 | " @show x, v\n", 94 | " x, v = euler(F, Δt, x, v)\n", 95 | "end " 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "## Taking derivatives without headaches\n", 103 | "\n", 104 | "First we consider the Harmonic potential. We plot the function and its derivative" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "using Plots" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": null, 119 | "metadata": {}, 120 | "outputs": [], 121 | "source": [ 122 | "Vho(r, a=0.5) = (r - a)^2 # Shifted harmonic oscillator\n", 123 | "dVho(r, a=0.5) = 2r -2a # Derivative\n", 124 | "\n", 125 | "p = plot(xlims=(-5, 5), xaxis=\"relative radial distance\")\n", 126 | "plot!(p, Vho, label=\"Vho a=0.5\") # Note: Plots figures out adaptively\n", 127 | "plot!(p, dVho, label=\"∂Vho\") # where to sample the functions" 128 | ] 129 | }, 130 | { 131 | "cell_type": "markdown", 132 | "metadata": {}, 133 | "source": [ 134 | "Now, taking the derivative for the harmonic oscillator this is kind of ok, but for more complicated potentials $V$ this quickly becomes error prone and most importantly quite *boring*. The solution is **algorithmic differentiation**, where the Julia compiler is taught how to automatically trace the code and not only produce the function value, but also produce a derivative.\n", 135 | "\n", 136 | "Without going into further details here, let us use the `Zygote` package to take the derivative on `Vho` as defined above:" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": null, 142 | "metadata": {}, 143 | "outputs": [], 144 | "source": [ 145 | "using Zygote\n", 146 | "\n", 147 | "p = plot(xlims=(-5, 5), xaxis=\"relative radial distance\")\n", 148 | "plot!(p, Vho, label=\"Vho a=0.5\")\n", 149 | "plot!(p, Vho', label=\"∂Vho\") # Notice the dash" 150 | ] 151 | }, 152 | { 153 | "cell_type": "markdown", 154 | "metadata": {}, 155 | "source": [ 156 | "With ease we add the second derivative to the plot:" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": null, 162 | "metadata": {}, 163 | "outputs": [], 164 | "source": [ 165 | "plot!(p, Vho'', label=\"∂∂Vho\")" 166 | ] 167 | }, 168 | { 169 | "cell_type": "markdown", 170 | "metadata": {}, 171 | "source": [ 172 | "An important thing to note is that this **really generates the derivative code** instead of numerical differentiation (i.e. finite-differences):" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": null, 178 | "metadata": {}, 179 | "outputs": [], 180 | "source": [ 181 | "@code_llvm debuginfo=:none Vho'(1.0)" 182 | ] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "metadata": {}, 187 | "source": [ 188 | "Or written out more nicely (Take `%0 = x`):\n", 189 | "```\n", 190 | " %1 = x + 0.5\n", 191 | " %2 = 2 * (x + 0.5)\n", 192 | " ret %2 = return 2 * (x + 0.5)\n", 193 | "```" 194 | ] 195 | }, 196 | { 197 | "cell_type": "markdown", 198 | "metadata": {}, 199 | "source": [ 200 | "Now let us consider more complicated potentials, for example the **Morse potential**, which is a common model for a chemical bond:\n", 201 | "$$ V_\\text{Morse} = D (1 - \\exp(-\\alpha (r - r_0)))^2 - D$$" 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": null, 207 | "metadata": { 208 | "scrolled": true 209 | }, 210 | "outputs": [], 211 | "source": [ 212 | "Vmorse(r; r₀=0.7, α=1.3, D=10) = D*(1 - exp(-α * (r - r₀)))^2 - D" 213 | ] 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": null, 218 | "metadata": {}, 219 | "outputs": [], 220 | "source": [ 221 | "p = plot(xlims=(0, 7), ylims=(-25, 25), xaxis=\"Radial distance\")\n", 222 | "plot!(Vmorse, label=\"Vmorse\")\n", 223 | "plot!(p, Vmorse', label=\"Vmorse'\")\n", 224 | "plot!(p, Vmorse'', label=\"Vmorse''\")" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "metadata": {}, 230 | "source": [ 231 | "This works for higher dimensions and more complicated expressions, too, we will use this in a sec." 232 | ] 233 | }, 234 | { 235 | "cell_type": "markdown", 236 | "metadata": {}, 237 | "source": [ 238 | "## Inspecting Euler dynamics in 1D\n", 239 | "\n", 240 | "Now we actually want to see things! We define a plot function to plot the potential and animate the particle over time:" 241 | ] 242 | }, 243 | { 244 | "cell_type": "code", 245 | "execution_count": null, 246 | "metadata": {}, 247 | "outputs": [], 248 | "source": [ 249 | "using Printf\n", 250 | "\n", 251 | "# The arguments after the ; are so-called keyword arguments, they can be omitted\n", 252 | "# and then the default value after the = is used\n", 253 | "function plot_dynamics_line(V, Δt, n_steps; x₀=0.0, v₀=randn(), integrator=euler,\n", 254 | " xlim=(-5, 5), ylim=(0, 10))\n", 255 | " t, x, v = 0.0, x₀, v₀ # Initialise state\n", 256 | " \n", 257 | " # Compute potential values (for plotting)\n", 258 | " xrange = xlim[1]:0.1:xlim[2]\n", 259 | " Vrange = V.(xrange)\n", 260 | " \n", 261 | " # @gif is a macro to \"record\" an animation of the dynamics,\n", 262 | " # each loop iteration gives a frame\n", 263 | " @gif for i in 1:n_steps\n", 264 | " # Propagate dynamics (notice the derivative)\n", 265 | " x, v = integrator(x -> -V'(x), Δt, x, v)\n", 266 | " t += Δt\n", 267 | "\n", 268 | " # Plot potential\n", 269 | " p = plot(xrange, Vrange; label=\"Potential\", xlim, ylim)\n", 270 | " \n", 271 | " # Plot the particle and its velocity (as an arrow)\n", 272 | " timestr = @sprintf \"%.2f\" t # Format time as a nice string\n", 273 | " scatter!(p, [x], [V(x)], label=\"Particle at t=$timestr\")\n", 274 | " plot!(p, [(x, V(x)), (x + 0.5v, V(x))], arrow=1.0,\n", 275 | " label=\"particle velocity / 2\")\n", 276 | " end\n", 277 | "end" 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": null, 283 | "metadata": {}, 284 | "outputs": [], 285 | "source": [ 286 | "# Now let's actually see it ....\n", 287 | "plot_dynamics_line(Vho, 0.1, 200)" 288 | ] 289 | }, 290 | { 291 | "cell_type": "markdown", 292 | "metadata": {}, 293 | "source": [ 294 | "Wow ... that's strange. \n", 295 | "Our plot points at the well-known problem that energy is not conserved for a simple forward Euler scheme.\n", 296 | "We can also diagnose this using a phase-space plot, where the time evolution of particle position $x$ and particle velocity $v$ is shown. This should be a closed curve if energy is conserved, so let's see ..." 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": null, 302 | "metadata": {}, 303 | "outputs": [], 304 | "source": [ 305 | "function plot_phase_space(F, Δt, n_steps; x₀=0.0, v₀=randn(), integrator=euler,\n", 306 | " xlim=(-7, 7), ylim=(-7, 7))\n", 307 | " x, v = x₀, v₀\n", 308 | " p = plot([x], [v]; xlim, ylim, label=\"\", xaxis=\"position\", yaxis=\"velocity\")\n", 309 | " @gif for N in 1:n_steps\n", 310 | " x, v = integrator(F, Δt, x, v)\n", 311 | " push!(p, x, v) # Add new positions to the plot ...\n", 312 | " end\n", 313 | "end" 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": null, 319 | "metadata": {}, 320 | "outputs": [], 321 | "source": [ 322 | "plot_phase_space(x -> -Vho'(x), 0.1, 200)" 323 | ] 324 | }, 325 | { 326 | "cell_type": "markdown", 327 | "metadata": {}, 328 | "source": [ 329 | "**Exercise:**\n", 330 | "\n", 331 | "A much more stable integrator than the `euler` we used so far is the verlocity Verlet:\n", 332 | "\n", 333 | "$$\\left\\{\\begin{array}{l}\n", 334 | "x_{n+1} = x_{n} + v_{n} \\Delta t + \\frac{F_V(x_{n})}{2} \\Delta t^2\\\\\n", 335 | "v_{n+1} = v_{n} + \\frac{F_V(x_{n)} + F_V(x_{n+1})}{2} \\Delta t\\\\\n", 336 | "\\end{array}\\right. $$\n", 337 | "\n", 338 | "- Program this algorithm, taking care that it supports multi-dimensional positions and velocities as well. In practice we would like to avoid recomputing $F_V(x)$ as much as possible, since this is usually the expensive step of the dynamics. For our purposes there is no need to keep an eye on that for now.\n", 339 | "- How does the previous dynamics look like in this example. Does this algorithm conserve energy (phase-space plot)?\n", 340 | "- Also look at the Morse potential" 341 | ] 342 | }, 343 | { 344 | "cell_type": "code", 345 | "execution_count": null, 346 | "metadata": {}, 347 | "outputs": [], 348 | "source": [ 349 | "# solution\n", 350 | "\n", 351 | "function verlet(F, Δt, xₙ, vₙ)\n", 352 | " # You're code here ...\n", 353 | "end" 354 | ] 355 | }, 356 | { 357 | "cell_type": "code", 358 | "execution_count": null, 359 | "metadata": {}, 360 | "outputs": [], 361 | "source": [ 362 | "# Some example plots and parameters:\n", 363 | "# plot_dynamics_line(Vho, 0.1, 200, integrator=verlet, ylim=(0, 2.5), xlim=(-1, 3))\n", 364 | "# plot_phase_space(x -> -Vho'(x), 0.1, 200, integrator=verlet, xlim=(-2, 2), ylim=(-2, 2))\n", 365 | "# plot_dynamics_line(Vmorse, 0.03, 200, integrator=verlet, xlim=(-1, 4), ylim=(-10, 5), x₀=2.0)" 366 | ] 367 | }, 368 | { 369 | "cell_type": "markdown", 370 | "metadata": {}, 371 | "source": [ 372 | "# Multiple particles" 373 | ] 374 | }, 375 | { 376 | "cell_type": "markdown", 377 | "metadata": {}, 378 | "source": [ 379 | "Now we want to simulate multiple identical partices in 2D. For a system of $N$ particles in 2D, we define the particle positions as the matrix\n", 380 | "$$ \\textbf{x} = \\left(\n", 381 | " \\begin{array}{cccc}\n", 382 | " x_{1x} & x_{2x} & \\cdots & x_{Nx} \\\\\n", 383 | " x_{1y} & x_{2y} & \\cdots & x_{Ny}\n", 384 | " \\end{array}\n", 385 | " \\right) = \\left( \\vec{x}_1 \\vec{x}_2 \\cdots \\vec{x}_N \\right). $$\n", 386 | "with the individual particle vectors as columns.\n", 387 | "We assume our particles interact via the idential pair potential $V_\\text{pair}(r)$\n", 388 | "depending only on particle distance $r$. The total potential is therefore\n", 389 | "$$ V_\\text{tot}(\\textbf{x}) = \\sum_{i=1}^N \\sum_{j=i+1}^N V_\\text{pair}(\\| \\vec{x}_i - \\vec{x}_j \\|). $$" 390 | ] 391 | }, 392 | { 393 | "cell_type": "markdown", 394 | "metadata": {}, 395 | "source": [ 396 | "### Exercise\n", 397 | "Program the total potential function for a matrix $\\textbf{x}$. A useful function is `norm` from the `LinearAlgebra` package." 398 | ] 399 | }, 400 | { 401 | "cell_type": "code", 402 | "execution_count": null, 403 | "metadata": {}, 404 | "outputs": [], 405 | "source": [ 406 | "using LinearAlgebra\n", 407 | "\n", 408 | "function Vtot(Vpair, x)\n", 409 | " # You're code here ...\n", 410 | "end" 411 | ] 412 | }, 413 | { 414 | "cell_type": "markdown", 415 | "metadata": {}, 416 | "source": [ 417 | "Now we want to plot the dynamics in a plane. In the following function the force is computed using automatically generated derivatives." 418 | ] 419 | }, 420 | { 421 | "cell_type": "code", 422 | "execution_count": null, 423 | "metadata": {}, 424 | "outputs": [], 425 | "source": [ 426 | "function plot_dynamics_plane(Vpair, Δt, n_steps; x₀=randn(2, 2), v₀=zero(x₀), integrator=verlet,\n", 427 | " xlim=(-3, 3), ylim=(-3, 3))\n", 428 | " # Total force via automatic differentiation\n", 429 | " V(x) = Vtot(Vpair, x)\n", 430 | " Ftot(x) = -V'(x)\n", 431 | " \n", 432 | " t, x, v = 0, x₀, v₀ # Initialise state\n", 433 | " @gif for i in 1:n_steps\n", 434 | " # Propagate dynamics\n", 435 | " x, v = integrator(Ftot, Δt, x, v)\n", 436 | " t += Δt\n", 437 | " timestr = @sprintf \"%.2f\" t # Format time\n", 438 | " \n", 439 | " # Plot the particles and their velocities\n", 440 | " p = scatter(x[1, :], x[2, :]; xlim, ylim, label=\"Particles at t=$timestr\")\n", 441 | " label = \"Velocity / 4\"\n", 442 | " for (xᵢ, vᵢ) in zip(eachcol(x), eachcol(v))\n", 443 | " plot!(p, [Tuple(xᵢ), Tuple(xᵢ + 0.25vᵢ)]; arrow=1.0, colour=:red, label)\n", 444 | " label = \"\"\n", 445 | " end\n", 446 | " end\n", 447 | "end" 448 | ] 449 | }, 450 | { 451 | "cell_type": "code", 452 | "execution_count": null, 453 | "metadata": {}, 454 | "outputs": [], 455 | "source": [ 456 | "Δt = 0.07\n", 457 | "n_steps = 300\n", 458 | "x₀ = [[0.; 0.] [1.; 0.] [-1.; 0.0]]\n", 459 | "plot_dynamics_plane(Vmorse, Δt, n_steps; x₀)" 460 | ] 461 | }, 462 | { 463 | "cell_type": "markdown", 464 | "metadata": {}, 465 | "source": [ 466 | "## A few nice examples" 467 | ] 468 | }, 469 | { 470 | "cell_type": "code", 471 | "execution_count": null, 472 | "metadata": {}, 473 | "outputs": [], 474 | "source": [ 475 | "Δt = 0.07\n", 476 | "n_steps = 300\n", 477 | "x₀ = [[0.; 0.] [1.; 0.] [-1.; 0.15]]\n", 478 | "plot_dynamics_plane(Vmorse, Δt, n_steps; x₀)" 479 | ] 480 | }, 481 | { 482 | "cell_type": "code", 483 | "execution_count": null, 484 | "metadata": {}, 485 | "outputs": [], 486 | "source": [ 487 | "Δt = 0.05\n", 488 | "n_steps = 300\n", 489 | "x₀ = [[0.; 1.] [1.; 0.] [-1.; 0] [0; -1.2]]\n", 490 | "plot_dynamics_plane(Vmorse, Δt, n_steps; x₀, ylim=(-10, 10))" 491 | ] 492 | }, 493 | { 494 | "cell_type": "code", 495 | "execution_count": null, 496 | "metadata": {}, 497 | "outputs": [], 498 | "source": [ 499 | "Δt = 0.05\n", 500 | "n_steps = 300\n", 501 | "x₀ = 4randn(2, 10)\n", 502 | "plot_dynamics_plane(Vmorse, Δt, n_steps; x₀, xlim=(-10, 10), ylim=(-10, 10))" 503 | ] 504 | }, 505 | { 506 | "cell_type": "markdown", 507 | "metadata": {}, 508 | "source": [ 509 | "## Open-ended exercise\n", 510 | "\n", 511 | "There are many ways one could go from here. Some suggestions (roughly ordered by difficulty):\n", 512 | "\n", 513 | "#### 1. Performance improvements\n", 514 | "Take a look at [03_Performance_Engineering.ipynb](03_Performance_Engineering.ipynb) and afterwards try to get your evaluation of `Vtot` as fast as possible. For example, try to reduce the timings of:" 515 | ] 516 | }, 517 | { 518 | "cell_type": "code", 519 | "execution_count": null, 520 | "metadata": {}, 521 | "outputs": [], 522 | "source": [ 523 | "using BenchmarkTools\n", 524 | "x = randn(2, 5000)\n", 525 | "@btime Vtot($Vmorse, $x)" 526 | ] 527 | }, 528 | { 529 | "cell_type": "markdown", 530 | "metadata": {}, 531 | "source": [ 532 | "#### 2. Parallelism and speed:\n", 533 | " * The expensive step in the simulation is the double loop over particles needed to compute Vtot.\n", 534 | " * Some suggestions were discussed in [../08_Multithreading_Basics.ipynb](../08_Multithreading_Basics.ipynb)\n", 535 | " \n", 536 | " \n", 537 | " \n", 538 | "#### 3. GPU support:\n", 539 | " * Since all operations we need are just linear algebra, just replacing standard arrays by `CuArray`s\n", 540 | " should be everything what is needed (see [../07_Linear_Algebra_Profiling.ipynb](../07_Linear_Algebra_Profiling.ipynb)\n", 541 | " * *Disclaimer:* I did not test this." 542 | ] 543 | } 544 | ], 545 | "metadata": { 546 | "@webio": { 547 | "lastCommId": "0ee45cb602ba4de49f9d8b52e7e3ca3d", 548 | "lastKernelId": "15fb271e-7c73-4f9c-be84-edddd39fe586" 549 | }, 550 | "kernelspec": { 551 | "display_name": "Julia 1.7.2", 552 | "language": "julia", 553 | "name": "julia-1.7" 554 | }, 555 | "language_info": { 556 | "file_extension": ".jl", 557 | "mimetype": "application/julia", 558 | "name": "julia", 559 | "version": "1.7.2" 560 | } 561 | }, 562 | "nbformat": 4, 563 | "nbformat_minor": 2 564 | } 565 | -------------------------------------------------------------------------------- /Projects/02_Types_Specialisation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Custom Types and Specialisation\n", 8 | "\n", 9 | "The purpose of this small project is to learn how to define custom data types. We will discuss how little effort is needed to integrate them into the Julia ecosystem (and benefit from pre-defined fallback functions) and how we can step-by-step exploit known structure to make computation faster." 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## Defining data types\n", 17 | "\n", 18 | "Apart from the built-in types, Julia also offers to declare custom types (i.e. data structures), for example:" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "struct MyType end" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "Note: By convention types are written in [camel case](https://en.wikipedia.org/wiki/Camel_case)." 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": {}, 40 | "source": [ 41 | "To obtain an obejct of type `MyType` one conventionally uses functions of the same name as the datatype\n", 42 | "([constructors](https://docs.julialang.org/en/v1/manual/constructors/)).\n", 43 | "\n", 44 | "A trivial default constructor is created by Julia automatically:" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "methods(MyType)" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "m = MyType()" 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "metadata": {}, 69 | "outputs": [], 70 | "source": [ 71 | "typeof(m)" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": null, 77 | "metadata": {}, 78 | "outputs": [], 79 | "source": [ 80 | "m isa MyType" 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "Even though such empty data structures have an important use case in Julia (as a marker *singleton type* for dispatch), more frequently we will need types to hold some data." 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [ 96 | "struct A\n", 97 | " x::Int64 # A field\n", 98 | "end" 99 | ] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "metadata": {}, 104 | "source": [ 105 | "The default constructor expects values for all fields, in the order of appearance:" 106 | ] 107 | }, 108 | { 109 | "cell_type": "code", 110 | "execution_count": null, 111 | "metadata": {}, 112 | "outputs": [], 113 | "source": [ 114 | "a = A(3)" 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": null, 120 | "metadata": {}, 121 | "outputs": [], 122 | "source": [ 123 | "a.x" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": null, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "a.x = 2" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "A key difference from Julia structs compared to equivalent constructs in other languages is that `struct`s are immutable, i.e. their fields cannot be changed. To make a struct mutable, add the `mutable` keyword:" 140 | ] 141 | }, 142 | { 143 | "cell_type": "code", 144 | "execution_count": null, 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "mutable struct B\n", 149 | " x::Int64\n", 150 | "end\n", 151 | "\n", 152 | "b = B(3)\n", 153 | "b.x = 4\n", 154 | "b.x" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": null, 160 | "metadata": {}, 161 | "outputs": [], 162 | "source": [ 163 | "struct C\n", 164 | " x::Vector{Int64}\n", 165 | "end" 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": null, 171 | "metadata": {}, 172 | "outputs": [], 173 | "source": [ 174 | "c = C([1, 2, 3])" 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": null, 180 | "metadata": {}, 181 | "outputs": [], 182 | "source": [ 183 | "c.x" 184 | ] 185 | }, 186 | { 187 | "cell_type": "code", 188 | "execution_count": null, 189 | "metadata": {}, 190 | "outputs": [], 191 | "source": [ 192 | "c.x[1] = -1" 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": null, 198 | "metadata": {}, 199 | "outputs": [], 200 | "source": [ 201 | "c.x" 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": null, 207 | "metadata": {}, 208 | "outputs": [], 209 | "source": [ 210 | "c.x = [4, 5, 6]" 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": null, 216 | "metadata": {}, 217 | "outputs": [], 218 | "source": [ 219 | "c.x .= [4, 5, 6] # dot to perform the assignment element-wise" 220 | ] 221 | }, 222 | { 223 | "cell_type": "markdown", 224 | "metadata": {}, 225 | "source": [ 226 | "Defining and using abstract types:" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": null, 232 | "metadata": {}, 233 | "outputs": [], 234 | "source": [ 235 | "abstract type MyAbstractType end # No fields! Just an informal interface" 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": null, 241 | "metadata": {}, 242 | "outputs": [], 243 | "source": [ 244 | "struct MyConcreteType <: MyAbstractType # subtype\n", 245 | " somefield::String\n", 246 | "end" 247 | ] 248 | }, 249 | { 250 | "cell_type": "code", 251 | "execution_count": null, 252 | "metadata": {}, 253 | "outputs": [], 254 | "source": [ 255 | "c = MyConcreteType(\"test\")" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": null, 261 | "metadata": {}, 262 | "outputs": [], 263 | "source": [ 264 | "c isa MyAbstractType" 265 | ] 266 | }, 267 | { 268 | "cell_type": "code", 269 | "execution_count": null, 270 | "metadata": {}, 271 | "outputs": [], 272 | "source": [ 273 | "supertype(MyConcreteType)" 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "execution_count": null, 279 | "metadata": {}, 280 | "outputs": [], 281 | "source": [ 282 | "subtypes(MyAbstractType)" 283 | ] 284 | }, 285 | { 286 | "cell_type": "markdown", 287 | "metadata": {}, 288 | "source": [ 289 | "## Example: Diagonal Matrix" 290 | ] 291 | }, 292 | { 293 | "cell_type": "code", 294 | "execution_count": null, 295 | "metadata": {}, 296 | "outputs": [], 297 | "source": [ 298 | "struct DiagMat\n", 299 | " diag::Vector{Float64}\n", 300 | "end" 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "execution_count": null, 306 | "metadata": {}, 307 | "outputs": [], 308 | "source": [ 309 | "DiagMat([1.2, 4.3, 5.0])" 310 | ] 311 | }, 312 | { 313 | "cell_type": "markdown", 314 | "metadata": {}, 315 | "source": [ 316 | "### Arithmetic" 317 | ] 318 | }, 319 | { 320 | "cell_type": "code", 321 | "execution_count": null, 322 | "metadata": {}, 323 | "outputs": [], 324 | "source": [ 325 | "import Base: +, -, *, /\n", 326 | "\n", 327 | "+(Da::DiagMat, Db::DiagMat) = DiagMat(Da.diag + Db.diag)\n", 328 | "-(Da::DiagMat, Db::DiagMat) = DiagMat(Da.diag - Db.diag)\n", 329 | "*(Da::DiagMat, Db::DiagMat) = DiagMat(Da.diag .* Db.diag)\n", 330 | "/(Da::DiagMat, Db::DiagMat) = DiagMat(Da.diag ./ Db.diag)" 331 | ] 332 | }, 333 | { 334 | "cell_type": "code", 335 | "execution_count": null, 336 | "metadata": {}, 337 | "outputs": [], 338 | "source": [ 339 | "D1 = DiagMat([1,2,3])\n", 340 | "D2 = DiagMat([2.4,1.9,5.7])" 341 | ] 342 | }, 343 | { 344 | "cell_type": "code", 345 | "execution_count": null, 346 | "metadata": {}, 347 | "outputs": [], 348 | "source": [ 349 | "D1 + D2" 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": null, 355 | "metadata": {}, 356 | "outputs": [], 357 | "source": [ 358 | "D1 - D2" 359 | ] 360 | }, 361 | { 362 | "cell_type": "code", 363 | "execution_count": null, 364 | "metadata": {}, 365 | "outputs": [], 366 | "source": [ 367 | "D1 * D2" 368 | ] 369 | }, 370 | { 371 | "cell_type": "code", 372 | "execution_count": null, 373 | "metadata": {}, 374 | "outputs": [], 375 | "source": [ 376 | "D1 / D2" 377 | ] 378 | }, 379 | { 380 | "cell_type": "code", 381 | "execution_count": null, 382 | "metadata": {}, 383 | "outputs": [], 384 | "source": [ 385 | "# Number\n", 386 | "*(x::Number, D::DiagMat) = DiagMat(x * D.diag)\n", 387 | "*(D::DiagMat, x::Number) = DiagMat(D.diag * x)\n", 388 | "/(D::DiagMat, x::Number) = DiagMat(D.diag / x)\n", 389 | "\n", 390 | "# Vector\n", 391 | "*(D::DiagMat, V::AbstractVector) = D.diag .* V" 392 | ] 393 | }, 394 | { 395 | "cell_type": "code", 396 | "execution_count": null, 397 | "metadata": {}, 398 | "outputs": [], 399 | "source": [ 400 | "D1 * 2" 401 | ] 402 | }, 403 | { 404 | "cell_type": "code", 405 | "execution_count": null, 406 | "metadata": {}, 407 | "outputs": [], 408 | "source": [ 409 | "D1 * rand(3)" 410 | ] 411 | }, 412 | { 413 | "cell_type": "markdown", 414 | "metadata": {}, 415 | "source": [ 416 | "Note that `Base` Julia's generic fallback implementations give us some functionality for free:" 417 | ] 418 | }, 419 | { 420 | "cell_type": "code", 421 | "execution_count": null, 422 | "metadata": {}, 423 | "outputs": [], 424 | "source": [ 425 | "2 * D1" 426 | ] 427 | }, 428 | { 429 | "cell_type": "code", 430 | "execution_count": null, 431 | "metadata": {}, 432 | "outputs": [], 433 | "source": [ 434 | "sum([D1, D2])" 435 | ] 436 | }, 437 | { 438 | "cell_type": "markdown", 439 | "metadata": {}, 440 | "source": [ 441 | "### Parameterization and `AbstractArray` interface\n", 442 | "\n", 443 | "That's a good start, but we can do better, because" 444 | ] 445 | }, 446 | { 447 | "cell_type": "code", 448 | "execution_count": null, 449 | "metadata": {}, 450 | "outputs": [], 451 | "source": [ 452 | "DiagMat([1, 2, 3]) # implicit conversion to Vector{Float64} => BAD" 453 | ] 454 | }, 455 | { 456 | "cell_type": "code", 457 | "execution_count": null, 458 | "metadata": {}, 459 | "outputs": [], 460 | "source": [ 461 | "DiagMat([1+3im, 4-2im, im]) # No complex number support?" 462 | ] 463 | }, 464 | { 465 | "cell_type": "code", 466 | "execution_count": null, 467 | "metadata": {}, 468 | "outputs": [], 469 | "source": [ 470 | "DiagMat([\"Why\", \"not\", \"support\", \"strings?\"]) # Would be cool, wouldn't it?" 471 | ] 472 | }, 473 | { 474 | "cell_type": "markdown", 475 | "metadata": {}, 476 | "source": [ 477 | "with actually *less* lines of code we can get a more generic version and fully integrate into Julia's type hierarchy. We do this by defining a *parametric type*, which is subtyping `AbstractMatrix`:" 478 | ] 479 | }, 480 | { 481 | "cell_type": "code", 482 | "execution_count": null, 483 | "metadata": {}, 484 | "outputs": [], 485 | "source": [ 486 | "struct DiagonalMatrix{T, V<:AbstractVector{T}} <: AbstractMatrix{T}\n", 487 | " diag::V\n", 488 | "end" 489 | ] 490 | }, 491 | { 492 | "cell_type": "markdown", 493 | "metadata": {}, 494 | "source": [ 495 | "To integrate properly we implement the [`AbstractArray` interface convention](https://docs.julialang.org/en/v1/manual/interfaces/#man-interface-array-1):" 496 | ] 497 | }, 498 | { 499 | "cell_type": "code", 500 | "execution_count": null, 501 | "metadata": {}, 502 | "outputs": [], 503 | "source": [ 504 | "# implement AbstractArray interface\n", 505 | "Base.size(D::DiagonalMatrix) = (length(D.diag), length(D.diag))\n", 506 | "\n", 507 | "function Base.getindex(D::DiagonalMatrix{T,V}, i::Int, j::Int) where {T, V}\n", 508 | " if i == j\n", 509 | " r = D.diag[i]\n", 510 | " else\n", 511 | " r = zero(T)\n", 512 | " end\n", 513 | " r\n", 514 | "end\n", 515 | "\n", 516 | "function setindex!(D::DiagonalMatrix, v, i::Int, j::Int)\n", 517 | " if i == j\n", 518 | " D.diag[i] = v\n", 519 | " else\n", 520 | " throw(ArgumentError(\"cannot set off-diagonal entry ($i, $j)\"))\n", 521 | " end\n", 522 | " return v\n", 523 | "end" 524 | ] 525 | }, 526 | { 527 | "cell_type": "markdown", 528 | "metadata": {}, 529 | "source": [ 530 | "Now we can:" 531 | ] 532 | }, 533 | { 534 | "cell_type": "code", 535 | "execution_count": null, 536 | "metadata": {}, 537 | "outputs": [], 538 | "source": [ 539 | "DiagonalMatrix([1.0, 2.0, 3.0])" 540 | ] 541 | }, 542 | { 543 | "cell_type": "code", 544 | "execution_count": null, 545 | "metadata": {}, 546 | "outputs": [], 547 | "source": [ 548 | "D = DiagonalMatrix([1, 2, 3])" 549 | ] 550 | }, 551 | { 552 | "cell_type": "markdown", 553 | "metadata": {}, 554 | "source": [ 555 | "Note the fancy pretty-pringing :)" 556 | ] 557 | }, 558 | { 559 | "cell_type": "code", 560 | "execution_count": null, 561 | "metadata": {}, 562 | "outputs": [], 563 | "source": [ 564 | "D * D" 565 | ] 566 | }, 567 | { 568 | "cell_type": "code", 569 | "execution_count": null, 570 | "metadata": {}, 571 | "outputs": [], 572 | "source": [ 573 | "D + D" 574 | ] 575 | }, 576 | { 577 | "cell_type": "code", 578 | "execution_count": null, 579 | "metadata": {}, 580 | "outputs": [], 581 | "source": [ 582 | "D - D" 583 | ] 584 | }, 585 | { 586 | "cell_type": "code", 587 | "execution_count": null, 588 | "metadata": {}, 589 | "outputs": [], 590 | "source": [ 591 | "D / D" 592 | ] 593 | }, 594 | { 595 | "cell_type": "markdown", 596 | "metadata": {}, 597 | "source": [ 598 | "Similarly basic arithmetic works without additional effort:" 599 | ] 600 | }, 601 | { 602 | "cell_type": "code", 603 | "execution_count": null, 604 | "metadata": {}, 605 | "outputs": [], 606 | "source": [ 607 | "sin.(D)" 608 | ] 609 | }, 610 | { 611 | "cell_type": "code", 612 | "execution_count": null, 613 | "metadata": {}, 614 | "outputs": [], 615 | "source": [ 616 | "sum([D, D, D])" 617 | ] 618 | }, 619 | { 620 | "cell_type": "code", 621 | "execution_count": null, 622 | "metadata": {}, 623 | "outputs": [], 624 | "source": [ 625 | "using LinearAlgebra\n", 626 | "eigvals(D) # Compute the eigenvalues" 627 | ] 628 | }, 629 | { 630 | "cell_type": "markdown", 631 | "metadata": {}, 632 | "source": [ 633 | "Of note, these functions work, but they do not exploit the diagonal structure. So a few functions we should define, such that the compiler can make use of what we know about our type." 634 | ] 635 | }, 636 | { 637 | "cell_type": "code", 638 | "execution_count": null, 639 | "metadata": {}, 640 | "outputs": [], 641 | "source": [ 642 | "@which D + D" 643 | ] 644 | }, 645 | { 646 | "cell_type": "code", 647 | "execution_count": null, 648 | "metadata": {}, 649 | "outputs": [], 650 | "source": [ 651 | "+(Da::DiagonalMatrix, Db::DiagonalMatrix) = DiagonalMatrix(Da.diag + Db.diag)\n", 652 | "*(x::Number, D::DiagonalMatrix) = DiagonalMatrix(x * D.diag)" 653 | ] 654 | }, 655 | { 656 | "cell_type": "code", 657 | "execution_count": null, 658 | "metadata": {}, 659 | "outputs": [], 660 | "source": [ 661 | "@which D + D" 662 | ] 663 | }, 664 | { 665 | "cell_type": "markdown", 666 | "metadata": {}, 667 | "source": [ 668 | "Certainly, defining all possible options is again a considerable effort. But we don't need to. In practice we would profile our code to identify bottle necks, e.g." 669 | ] 670 | }, 671 | { 672 | "cell_type": "code", 673 | "execution_count": null, 674 | "metadata": {}, 675 | "outputs": [], 676 | "source": [ 677 | "Dbig = DiagonalMatrix(randn(1000))\n", 678 | "@time eigvals(Dbig); # Ouch!" 679 | ] 680 | }, 681 | { 682 | "cell_type": "code", 683 | "execution_count": null, 684 | "metadata": {}, 685 | "outputs": [], 686 | "source": [ 687 | "# Define faster version to improve this part of the code\n", 688 | "LinearAlgebra.eigvals(D::DiagonalMatrix) = D.diag" 689 | ] 690 | }, 691 | { 692 | "cell_type": "code", 693 | "execution_count": null, 694 | "metadata": {}, 695 | "outputs": [], 696 | "source": [ 697 | "@time eigvals(Dbig); # Much better" 698 | ] 699 | }, 700 | { 701 | "cell_type": "markdown", 702 | "metadata": {}, 703 | "source": [ 704 | "Actually this implementation is pretty much the same to the [`Diagonal` implementation](https://github.com/JuliaLang/julia/blob/master/stdlib/LinearAlgebra/src/diagonal.jl#L5) in the `LinearAlgebra` package. " 705 | ] 706 | }, 707 | { 708 | "cell_type": "markdown", 709 | "metadata": {}, 710 | "source": [ 711 | "## Project: One-hot vector\n", 712 | "\n", 713 | "[One-hot encoding](https://en.wikipedia.org/wiki/One-hot) is useful for classification problems in machine learning.\n", 714 | "\n", 715 | "It simply means that among a group of bits (all either `0` or `1`) only one is hot (`1`) while all others are cold (`0`). Usually this is stored as a vector\n", 716 | "\n", 717 | "```julia\n", 718 | " v = [0, 0, 0, 0, 0, 1, 0, 0, 0]\n", 719 | "```\n", 720 | "\n", 721 | "1. What information does such a one-hot vector actually need to store?\n", 722 | "\n", 723 | "2. Define a `OneHot` type which represents a vector with only a single hot (i.e. `== 1`) bit.\n", 724 | "\n", 725 | "3. Extend all the necessary `Base` functions such that the following computation works\n", 726 | " for a matrix `A` and a vector of `OneHot` vectors `vs` (i.e. `vs isa Vector{OneHot}`):\n", 727 | " \n", 728 | " ```julia\n", 729 | " function innersum(A, vs)\n", 730 | " t = zero(eltype(A))\n", 731 | " for v in vs\n", 732 | " y = A*v\n", 733 | " for i in 1:length(vs[1])\n", 734 | " t += v[i] * y[i]\n", 735 | " end\n", 736 | " end\n", 737 | " t\n", 738 | " end\n", 739 | "\n", 740 | " A = rand(3,3)\n", 741 | " vs = [rand(3) for i in 1:10] # This should be replaced by a `Vector{OneHot}`\n", 742 | "\n", 743 | " innersum(A, vs)\n", 744 | "\n", 745 | " ```\n", 746 | "\n", 747 | "4. Benchmark the speed of `innersum` when called with a vector of `OneHot` vectors (i.e. `vs = [OneHot(3, rand(1:3)) for i in 1:10]`) and when called with a vector of `Vector{Float64}` vectors, respectively. Do you observe a speed up?\n", 748 | "\n", 749 | "\n", 750 | "5. Define a new `OneHotVector` type which is identical to `OneHot` but is declared as a subtype of `AbstractVector{Bool}`. Extend only the interface-defining functions `Base.getindex(v::OneHotVector, i::Int)` and `Base.size(v::OneHotVector)`. Here, the function `size` should return a `Tuple{Int64}` indicating the length of the vector, i.e. `(3,)` for a one-hot vector of length 3.\n", 751 | " \n", 752 | "\n", 753 | "6. Try to create a single `OneHotVector` and try to run the `innersum` function using the new `OneHotVector` type.\n", 754 | " * What changes do you observe?\n", 755 | " * Do you have to implement any further methods?" 756 | ] 757 | } 758 | ], 759 | "metadata": { 760 | "@webio": { 761 | "lastCommId": null, 762 | "lastKernelId": null 763 | }, 764 | "kernelspec": { 765 | "display_name": "Julia 1.7.2", 766 | "language": "julia", 767 | "name": "julia-1.7" 768 | }, 769 | "language_info": { 770 | "file_extension": ".jl", 771 | "mimetype": "application/julia", 772 | "name": "julia", 773 | "version": "1.7.2" 774 | } 775 | }, 776 | "nbformat": 4, 777 | "nbformat_minor": 4 778 | } 779 | -------------------------------------------------------------------------------- /Projects/04_Solving_Differential_Equations.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "f38403f1", 6 | "metadata": {}, 7 | "source": [ 8 | "# Solving Differential Equations\n", 9 | "\n", 10 | "This notebook is less a project for you to work on, much rather a show case of multiple packages, which allow you to solve partial differential equations in Julia. Based on \"classic\" example in the respective fields I invite you to start playing by yourself. Resources for the respective packages are given." 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "id": "documented-kidney", 16 | "metadata": {}, 17 | "source": [ 18 | "## Using ModelingToolkit and OrdinaryDiffEq to solve an ODE system\n", 19 | "\n", 20 | "Consider the Lorenz attractor:\n", 21 | "\n", 22 | "$$\n", 23 | "\\begin{align}\n", 24 | " \\dot{x} &= \\sigma y - \\sigma x\\\\\n", 25 | " \\dot{y} &= \\rho x - xz - y\\\\\n", 26 | " \\dot{z} &= xy - \\beta z\n", 27 | "\\end{align}\n", 28 | "$$" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": null, 34 | "id": "painful-ghana", 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "using ModelingToolkit\n", 39 | "using OrdinaryDiffEq\n", 40 | "using Plots\n", 41 | "\n", 42 | "# Define the system to be solved\n", 43 | "@parameters t σ ρ β\n", 44 | "@variables x(t) y(t) z(t)\n", 45 | "D = Differential(t)\n", 46 | "\n", 47 | "eqs = [D(x) ~ σ*y - σ*x,\n", 48 | " D(y) ~ x*ρ - x*z - y,\n", 49 | " D(z) ~ x*y - β*z]\n", 50 | "@named lorenz = ODESystem(eqs)\n", 51 | "\n", 52 | "# Define the inital conditions\n", 53 | "u0 = [x => 1.0,\n", 54 | " y => 1.0,\n", 55 | " z => 1.0]\n", 56 | "\n", 57 | "# Define the values of the parameters\n", 58 | "p = [σ => 10.0,\n", 59 | " ρ => 28.0,\n", 60 | " β => 8/3]\n", 61 | "\n", 62 | "# Timespan for propagation\n", 63 | "tspan = (0.0, 30.0)\n", 64 | "\n", 65 | "# Solve it\n", 66 | "prob = ODEProblem(lorenz, u0, tspan, p, jac=true)\n", 67 | "sol = OrdinaryDiffEq.solve(prob, Tsit5())\n", 68 | "\n", 69 | "# and plot solution\n", 70 | "plot(sol, vars=(x,y))" 71 | ] 72 | }, 73 | { 74 | "cell_type": "markdown", 75 | "id": "082fc157", 76 | "metadata": {}, 77 | "source": [ 78 | "##### More details\n", 79 | "- https://github.com/SciML/OrdinaryDiffEq.jl\n", 80 | "- https://github.com/SciML/DifferentialEquations.jl\n", 81 | "- https://github.com/SciML/ModelingToolkit.jl" 82 | ] 83 | }, 84 | { 85 | "cell_type": "markdown", 86 | "id": "entitled-verse", 87 | "metadata": {}, 88 | "source": [ 89 | "## Solving PDEs using the finite-element method\n", 90 | "\n", 91 | "Classic example:\n", 92 | "\n", 93 | "$$\n", 94 | "\\begin{align}\n", 95 | " - \\Delta u &= f &&\\text{in $\\Omega$} \\\\\n", 96 | " u &= g &&\\text{on $\\partial \\Omega$}\n", 97 | "\\end{align}\n", 98 | "$$\n", 99 | "\n", 100 | "with $f(x) = 1.0, g(x) = 2.0$" 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": null, 106 | "id": "pressed-worth", 107 | "metadata": {}, 108 | "outputs": [], 109 | "source": [ 110 | "using Gridap\n", 111 | "\n", 112 | "# Create cartesian mesh of 10 regularly-spaced cells\n", 113 | "L = 1.0\n", 114 | "domain = (0.0, L, 0.0, L, 0.0, L)\n", 115 | "n = 10\n", 116 | "partition = (n, n, n)\n", 117 | "model = CartesianDiscreteModel(domain, partition)\n", 118 | "\n", 119 | "# Dirichlet BCs everywhere\n", 120 | "labels = get_face_labeling(model)\n", 121 | "add_tag_from_tags!(labels, \"dirichlet\", collect(1:8))\n", 122 | "\n", 123 | "# Test space built from linear Lagrangian elements\n", 124 | "order = 1\n", 125 | "reffe = ReferenceFE(lagrangian, Float64, order)\n", 126 | "V0 = TestFESpace(model, reffe; conformity=:H1, dirichlet_tags=\"dirichlet\")\n", 127 | "\n", 128 | "# Trial space\n", 129 | "g(x) = 2.0\n", 130 | "Ug = TrialFESpace(V0, g)\n", 131 | "\n", 132 | "# Integration\n", 133 | "degree = 2\n", 134 | "Ω = Triangulation(model)\n", 135 | "dΩ = Measure(Ω, degree)\n", 136 | "\n", 137 | "# Weak form\n", 138 | "f(x) = 1.0\n", 139 | "h(x) = 3.0\n", 140 | "a(u, v) = ∫( ∇(v)⋅∇(u) )*dΩ\n", 141 | "b(v) = ∫( v*f )*dΩ\n", 142 | "\n", 143 | "# Build and solve FE problem\n", 144 | "op = AffineFEOperator(a, b, Ug, V0)\n", 145 | "ls = LUSolver()\n", 146 | "solver = LinearFESolver(ls)\n", 147 | "uh = Gridap.solve(solver, op)\n", 148 | "\n", 149 | "# Write to VTK, to be able to look at the function from Paraview\n", 150 | "writevtk(Ω, \"results\", cellfields=[\"uh\" => uh])" 151 | ] 152 | }, 153 | { 154 | "cell_type": "markdown", 155 | "id": "f9e8e839", 156 | "metadata": {}, 157 | "source": [ 158 | "##### More details\n", 159 | "- https://github.com/gridap/Gridap.jl\n", 160 | "- https://gridap.github.io/Gridap.jl/dev/" 161 | ] 162 | }, 163 | { 164 | "cell_type": "markdown", 165 | "id": "facial-composition", 166 | "metadata": {}, 167 | "source": [ 168 | "## Solving PDEs using plane-wave discretisations\n", 169 | "\n", 170 | "Consider solving the Gross-Pitaevskii equation\n", 171 | "\n", 172 | "$$\n", 173 | "H ψ = \\left(-\\frac12 Δ + V + 2 C |ψ|^2\\right) ψ = μ ψ \\qquad \\|ψ\\|_{L^2} = 1\n", 174 | "$$\n", 175 | "\n", 176 | "with $C = 1$ in a periodic 1D \"box\" of length $10$ with confining potential $V(x) = (x - 5)^2$.\n", 177 | "\n", 178 | "This is a common mean-field model for Bose-Einstein condensates, where $\\psi$ is the quantum-mechanical state of condensation and $\\rho = |\\psi|^2$ is the particle density." 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": null, 184 | "id": "b6da8d49", 185 | "metadata": {}, 186 | "outputs": [], 187 | "source": [ 188 | "using DFTK\n", 189 | "using Plots\n", 190 | "\n", 191 | "# Periodic boundary condition ... 1D problem\n", 192 | "lattice = 10 .* [[1 0 0.];\n", 193 | " [0 0 0.];\n", 194 | " [0 0 0.]]\n", 195 | "\n", 196 | "# Setup the model to be solved\n", 197 | "V(x) = (x - 5)^2\n", 198 | "C = 1\n", 199 | "terms = [Kinetic(), # -½Δ (Kinetic energy operator in quantum physics)\n", 200 | " ExternalFromReal(r -> V(r[1])),\n", 201 | " LocalNonlinearity(ρ -> 2C * ρ^2), # 2 * |ψ|^2\n", 202 | "]\n", 203 | "model = Model(lattice; n_electrons=1, terms=terms,\n", 204 | " spin_polarization=:spinless) # use \"spinless electrons\"\n", 205 | "\n", 206 | "# Discretise problem in a plane-wave basis\n", 207 | "basis = PlaneWaveBasis(model; Ecut=500, kgrid=(1, 1, 1))" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": null, 213 | "id": "municipal-teens", 214 | "metadata": {}, 215 | "outputs": [], 216 | "source": [ 217 | "# Solve it using a constrained preconditioned LBFGS:\n", 218 | "scfres = direct_minimization(basis, tol=1e-8);" 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": null, 224 | "id": "continuing-lying", 225 | "metadata": {}, 226 | "outputs": [], 227 | "source": [ 228 | "# Plot the resulting particle density |ψ|^2\n", 229 | "x = 10vec(first.(DFTK.r_vectors(basis)))\n", 230 | "plot(x, scfres.ρ[:, 1, 1], label=\"ρ\")\n", 231 | "plot!(x, V.(x) / 10, label=\"V\")" 232 | ] 233 | }, 234 | { 235 | "cell_type": "markdown", 236 | "id": "85b64f86", 237 | "metadata": {}, 238 | "source": [ 239 | "##### More details\n", 240 | "- https://dftk.org\n", 241 | "- https://docs.dftk.org" 242 | ] 243 | } 244 | ], 245 | "metadata": { 246 | "@webio": { 247 | "lastCommId": null, 248 | "lastKernelId": null 249 | }, 250 | "kernelspec": { 251 | "display_name": "Julia 1.7.2", 252 | "language": "julia", 253 | "name": "julia-1.7" 254 | }, 255 | "language_info": { 256 | "file_extension": ".jl", 257 | "mimetype": "application/julia", 258 | "name": "julia", 259 | "version": "1.7.2" 260 | } 261 | }, 262 | "nbformat": 4, 263 | "nbformat_minor": 5 264 | } 265 | -------------------------------------------------------------------------------- /Projects/05_Statistical_Learning.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "f254a551", 6 | "metadata": {}, 7 | "source": [ 8 | "# Statistical learning\n", 9 | "\n", 10 | "In this project we briefly discuss a few packages to do statistical learning in Julia.\n", 11 | "\n", 12 | "*Aside:* I'm not an expert in this field, so if some of you has more insights or better tricks / setups to try, I'm happy for any comments!" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "id": "enabling-metro", 18 | "metadata": {}, 19 | "source": [ 20 | "## Stheno - Gaussian process regression\n", 21 | "\n", 22 | "This first section considers Gaussian process regression and inference as implemented in the [Stheno](https://juliagaussianprocesses.github.io/Stheno.jl) package." 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "id": "bc7ea7bd", 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "using AbstractGPs\n", 33 | "using Stheno\n", 34 | "using Plots\n", 35 | "using LinearAlgebra" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": null, 41 | "id": "690830d5", 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "# Generate some synthetic data\n", 46 | "N = 50\n", 47 | "xdata = 2π * rand(N)\n", 48 | "y = sin.(xdata) + 0.05*randn(N);" 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": null, 54 | "id": "focused-tumor", 55 | "metadata": { 56 | "scrolled": true 57 | }, 58 | "outputs": [], 59 | "source": [ 60 | "# Setup zero-mean Gaussian Process with a squared exponential kernel σ² exp(-||x-x'||^2 / 2l²)\n", 61 | "l = 0.4\n", 62 | "σ² = 0.6\n", 63 | "f = @gppp let\n", 64 | " # Build a Gaussian Process Programming Problem, but only with one GP\n", 65 | " # (could add multiple GPs here with parameters that depend on each other etc.)\n", 66 | " f1 = σ² * stretch(GP(SEKernel()), 1 / l)\n", 67 | "end\n", 68 | "\n", 69 | "# Sample from the GP f1 (with some IID observation noise)\n", 70 | "# which is put on the diagonal of the Kernel\n", 71 | "const x = GPPPInput(:f1, xdata)\n", 72 | "σ²ₙ = 0.05\n", 73 | "fx = f(x, σ²ₙ)\n", 74 | "\n", 75 | "# Compute posterior over f given data y\n", 76 | "f_posterior = posterior(fx, y)\n", 77 | "\n", 78 | "# Plot data\n", 79 | "plt = plot();\n", 80 | "scatter!(plt, x.x, y; color=:red, label=\"\");\n", 81 | "\n", 82 | "# Plot posterior\n", 83 | "x_plot = range(-0.5, 6.7; length=1000);\n", 84 | "xp = GPPPInput(:f1, x_plot)\n", 85 | "plot!(plt, x_plot, f_posterior(xp); label=\"\", color=:blue,\n", 86 | " fillalpha=0.2, linewidth=2) \n", 87 | "plot!(\n", 88 | " plt, x_plot, rand(f_posterior(xp, 1e-9), 10);\n", 89 | " samples=10, markersize=1, alpha=0.3, label=\"\", color=:blue,\n", 90 | ")" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "id": "micro-winning", 97 | "metadata": { 98 | "scrolled": true 99 | }, 100 | "outputs": [], 101 | "source": [ 102 | "# Hyperparameter optimisation\n", 103 | "using Optim\n", 104 | "using Zygote\n", 105 | "\n", 106 | "# Unpack hyperparameters (given in log scale)\n", 107 | "# Fudge term to avoid nasty zeros ...\n", 108 | "unpack(θ) = exp.(θ) .+ 1e-6\n", 109 | "\n", 110 | "# nlml = negative log marginal likelihood (of θ)\n", 111 | "function nlml(θ)\n", 112 | " l, σ², σ²ₙ = unpack(θ)\n", 113 | " f = @gppp let\n", 114 | " f1 = σ² * stretch(GP(SEKernel()), 1 / l)\n", 115 | " end\n", 116 | " -logpdf(f(x, σ²ₙ), y)\n", 117 | "end\n", 118 | "\n", 119 | "# Optimise using BFGS with adjoint-mode autodiff gradient on nlml\n", 120 | "θ0 = randn(3);\n", 121 | "results = Optim.optimize(nlml, nlml', θ0, BFGS(); inplace=false)\n", 122 | "l_opt, σ²_opt, σ²ₙ_opt = unpack(results.minimizer)\n", 123 | "\n", 124 | "@show σ²_opt\n", 125 | "@show l_opt\n", 126 | "@show σ²ₙ_opt\n", 127 | "\n", 128 | "# Optimal GP and posterior\n", 129 | "f_opt = @gppp let\n", 130 | " f1 = σ²_opt * stretch(GP(SEKernel()), 1 / l_opt)\n", 131 | "end\n", 132 | "f_posterior_opt = posterior(f(x, σ²ₙ_opt), y)\n", 133 | "\n", 134 | "# Add to plot ...\n", 135 | "plot!(plt, x_plot, f_posterior_opt(xp); label=\"\", color=:orange,\n", 136 | " fillalpha=0.2, linewidth=2)\n", 137 | "plot!(\n", 138 | " plt, x_plot, rand(f_posterior(xp, 1e-9), 10);\n", 139 | " samples=10, markersize=1, alpha=0.3, label=\"\", color=:orange,\n", 140 | ")" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "id": "d7b20ad3", 146 | "metadata": {}, 147 | "source": [ 148 | "## Flux - The ML library that doesn't make you tensor\n", 149 | "\n", 150 | "[Flux](https://fluxml.ai/) has become the de-facto standard for neural-network-based statistical learning in Julia. A good starting point for any NN-based learning problem is the [Flux model zoo](https://github.com/FluxML/model-zoo/).\n", 151 | "\n", 152 | "What we will do here is handwriting recognition based on the `MNIST` data set." 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": null, 158 | "id": "97c6d3e2", 159 | "metadata": {}, 160 | "outputs": [], 161 | "source": [ 162 | "using Flux\n", 163 | "using CUDA\n", 164 | "using Plots\n", 165 | "using PyCall\n", 166 | "using MLDatasets: MNIST\n", 167 | "using MLDataPattern\n", 168 | "using Flux.Data: DataLoader\n", 169 | "using FluxTraining\n", 170 | "\n", 171 | "# \"Accept\" terms when downloading MNIST image data\n", 172 | "ENV[\"DATADEPS_ALWAYS_ACCEPT\"] = true;" 173 | ] 174 | }, 175 | { 176 | "cell_type": "code", 177 | "execution_count": null, 178 | "id": "d4bb05a4", 179 | "metadata": {}, 180 | "outputs": [], 181 | "source": [ 182 | "# Load handwritten digits dataset\n", 183 | "x_data = reshape(MNIST.traintensor(Float32), 28, 28, 1, :)\n", 184 | "y_data = Flux.onehotbatch(MNIST.trainlabels(), 0:9)\n", 185 | "\n", 186 | "# Split into training, validation, testing\n", 187 | "traindata, valdata, testdata = splitobs((x_data, y_data), at=(0.9, 0.05))\n", 188 | "\n", 189 | "# Show sizes\n", 190 | "@show size(traindata[1]) size(valdata[1]) size(testdata[1])\n", 191 | "@show size(traindata[2]) size(valdata[2]) size(testdata[2]);" 192 | ] 193 | }, 194 | { 195 | "cell_type": "markdown", 196 | "id": "1aae9d27", 197 | "metadata": {}, 198 | "source": [ 199 | "Now let us get a quick idea what we are dealing with. We plot a few of the images:" 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": null, 205 | "id": "45a0f1b7", 206 | "metadata": {}, 207 | "outputs": [], 208 | "source": [ 209 | "plots = map(1:25) do i\n", 210 | " p = plot(Gray.(1 .- testdata[1][:, :, 1, i])'; yaxis=nothing, xaxis=nothing)\n", 211 | " label = Flux.onecold(testdata[2][:, i], 0:9)\n", 212 | " title!(p, string(label), titlefontsize=8)\n", 213 | "end\n", 214 | "plot(plots...)" 215 | ] 216 | }, 217 | { 218 | "cell_type": "markdown", 219 | "id": "9e82b13b", 220 | "metadata": {}, 221 | "source": [ 222 | "Ok ... so a bunch of handwritten numbers, labelled with the appropriate image." 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "id": "a98e2547", 228 | "metadata": {}, 229 | "source": [ 230 | "Next we define a convolutional neural network to be trained. Each Image has 28x28 pixels, that's the input size. Output is the 10 probabilities for being one of the numbers 0 to 9.\n", 231 | "\n", 232 | "For more details on the layers and models, see\n", 233 | "- https://fluxml.ai/Flux.jl/stable/models/layers/\n", 234 | "- https://fluxml.ai/Flux.jl/stable/models/nnlib/" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": null, 240 | "id": "d590c036", 241 | "metadata": {}, 242 | "outputs": [], 243 | "source": [ 244 | "model = Chain(\n", 245 | " Conv((3, 3), 1 => 6, relu), # Convolution layer, 6 convolution kernels of size 3x3, relu activation\n", 246 | " AdaptiveMaxPool((13, 13)), # In each layer keep only the largest of 2x2 windows\n", 247 | " Conv((3, 3), 6 => 16, relu), # Another convolution ...\n", 248 | " AdaptiveMaxPool((5, 5)), # ... and another pooling layer\n", 249 | " flatten, # Flatten all the data\n", 250 | " Dense(400, 100, relu), # down to 100 outputs by a dense layer\n", 251 | " Dense(100, 84, relu), # to 84\n", 252 | " Dense(84, 10), # to 10\n", 253 | " softmax\n", 254 | ")" 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "id": "ff6f1f91", 260 | "metadata": {}, 261 | "source": [ 262 | "Now we train (This will take a while ...)" 263 | ] 264 | }, 265 | { 266 | "cell_type": "code", 267 | "execution_count": null, 268 | "id": "d7edaf75", 269 | "metadata": { 270 | "scrolled": true 271 | }, 272 | "outputs": [], 273 | "source": [ 274 | "loss = Flux.Losses.crossentropy\n", 275 | "opt = Flux.Optimise.ADAM()\n", 276 | "\n", 277 | "train_iterator = DataLoader(traindata, batchsize=250, shuffle=true)\n", 278 | "valid_iterator = DataLoader(valdata, batchsize=250, shuffle=true)\n", 279 | "learner = Learner(model, (train_iterator, valid_iterator),\n", 280 | " opt, loss, Metrics(accuracy), ToGPU()) # Use GPU if available\n", 281 | "\n", 282 | "nepochs = 10\n", 283 | "FluxTraining.fit!(learner, nepochs)" 284 | ] 285 | }, 286 | { 287 | "cell_type": "markdown", 288 | "id": "12c5ec4c", 289 | "metadata": {}, 290 | "source": [ 291 | "To verify our training did not overfit, we check the learning rate comparing the training and validation loss over the learning epochs:" 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "execution_count": null, 297 | "id": "e36425f5", 298 | "metadata": {}, 299 | "outputs": [], 300 | "source": [ 301 | "training_loss = learner.cbstate.metricsepoch[TrainingPhase()][:Loss].values\n", 302 | "validation_loss = learner.cbstate.metricsepoch[ValidationPhase()][:Loss].values\n", 303 | "\n", 304 | "p = plot(training_loss, label=\"Training loss\")\n", 305 | "p = plot!(p, validation_loss, label=\"Validation loss\")" 306 | ] 307 | }, 308 | { 309 | "cell_type": "markdown", 310 | "id": "98f6ceaf", 311 | "metadata": {}, 312 | "source": [ 313 | "As we see the rates for training and validation are nicely paralell, so we did not yet overfit.\n", 314 | "\n", 315 | "Let's assume we are happy with our results (we could do another 10 epochs of training if we we were not). After all training is done we check how good our model is with the test data:" 316 | ] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": null, 321 | "id": "72022c51", 322 | "metadata": {}, 323 | "outputs": [], 324 | "source": [ 325 | "# Evaluate the model using the test data\n", 326 | "test_pred = model(testdata[1])\n", 327 | "\n", 328 | "@show loss(test_pred, testdata[2])\n", 329 | "@show accuracy(test_pred, testdata[2]);" 330 | ] 331 | }, 332 | { 333 | "cell_type": "markdown", 334 | "id": "9640420d", 335 | "metadata": {}, 336 | "source": [ 337 | "Not too bad, let's see what this means in practice:" 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "execution_count": null, 343 | "id": "8934a8f3", 344 | "metadata": { 345 | "scrolled": false 346 | }, 347 | "outputs": [], 348 | "source": [ 349 | "plots = map(1:25) do i\n", 350 | " p = plot(Gray.(1 .- testdata[1][:, :, 1, i])'; yaxis=nothing, xaxis=nothing)\n", 351 | " prediction = findmax(vec(model(testdata[1][:, :, :, i:i])))\n", 352 | " title!(p, \"$(prediction[2]-1) -> $(round(100prediction[1], digits=1))%\",\n", 353 | " titlefontsize=8)\n", 354 | "end\n", 355 | "plot(plots...)" 356 | ] 357 | } 358 | ], 359 | "metadata": { 360 | "@webio": { 361 | "lastCommId": null, 362 | "lastKernelId": null 363 | }, 364 | "kernelspec": { 365 | "display_name": "Julia 1.7.2", 366 | "language": "julia", 367 | "name": "julia-1.7" 368 | }, 369 | "language_info": { 370 | "file_extension": ".jl", 371 | "mimetype": "application/julia", 372 | "name": "julia", 373 | "version": "1.7.2" 374 | } 375 | }, 376 | "nbformat": 4, 377 | "nbformat_minor": 5 378 | } 379 | -------------------------------------------------------------------------------- /Projects/Solutions_to_Exercises.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "cd9ae8be", 6 | "metadata": {}, 7 | "source": [ 8 | "# [01_Dancing_Particles](01_Dancing_Particles.ipynb)\n", 9 | "\n", 10 | "\n", 11 | "**Exercise:**\n", 12 | "\n", 13 | "A much more stable integrator than the `euler` we used so far is the verlocity Verlet:\n", 14 | "\n", 15 | "$$\\left\\{\\begin{array}{l}\n", 16 | "x_{n+1} = x_{n} + v_{n} \\Delta t + \\frac{F_V(x_{n})}{2} \\Delta t^2\\\\\n", 17 | "v_{n+1} = v_{n} + \\frac{F_V(x_{n)} + F_V(x_{n+1})}{2} \\Delta t\\\\\n", 18 | "\\end{array}\\right. $$\n", 19 | "\n", 20 | "- Program this algorithm, taking care that it supports multi-dimensional positions and velocities as well. In practice we would like to avoid recomputing $F_V(x)$ as much as possible, since this is usually the expensive step of the dynamics. For our purposes there is no need to keep an eye on that for now.\n", 21 | "- How does the previous dynamics look like in this example. Does this algorithm conserve energy (phase-space plot)?\n", 22 | "- Also look at the Morse potential" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "id": "c9038d69", 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "# One way to code the verlet function is:\n", 33 | "function verlet(F, Δt, xₙ, vₙ)\n", 34 | " Fₙ = F(xₙ)\n", 35 | " xₙ₊₁ = xₙ + vₙ * Δt + Fₙ/2 * Δt^2\n", 36 | " vₙ₊₁ = vₙ + (Fₙ + F(xₙ₊₁)) / 2 * Δt\n", 37 | " xₙ₊₁, vₙ₊₁\n", 38 | "end" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "id": "9fb8266c", 44 | "metadata": {}, 45 | "source": [ 46 | "### Exercise\n", 47 | "Program the total potential function for a matrix $\\textbf{x}$. A useful function is `norm` from the `LinearAlgebra` package." 48 | ] 49 | }, 50 | { 51 | "cell_type": "code", 52 | "execution_count": null, 53 | "id": "29fdaa34", 54 | "metadata": {}, 55 | "outputs": [], 56 | "source": [ 57 | "# One solution:\n", 58 | "function Vtot(Vpair, x)\n", 59 | " n_particles = size(x, 2)\n", 60 | " accu = zero(eltype(x)) # Get a zero of the appropriate type\n", 61 | " for i in 1:n_particles, j in i+1:n_particles\n", 62 | " accu += Vpair(norm(x[:, i] .- x[:, j]))\n", 63 | " end\n", 64 | " accu\n", 65 | "end" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "id": "df56d00c", 71 | "metadata": {}, 72 | "source": [ 73 | "## Open-ended exercise\n", 74 | "\n", 75 | "#### 1. Performance improvements\n", 76 | "\n", 77 | "The most immediate performance improvements are obtained by using views and disabling the bounds checks, for example:" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": null, 83 | "id": "ec99a8f0", 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "function Vtot(Vpair, x)\n", 88 | " n_particles = size(x, 2)\n", 89 | " accu = zero(eltype(x)) # Get a zero of the appropriate type\n", 90 | " @views @inbounds begin\n", 91 | " for i in 1:n_particles, j in i+1:n_particles\n", 92 | " accu += Vpair(norm(x[:, i] .- x[:, j]))\n", 93 | " end\n", 94 | " end\n", 95 | " accu\n", 96 | "end" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "id": "fd7ea0bc", 102 | "metadata": {}, 103 | "source": [ 104 | "To go beyond that more global changes of the MD algorithm would be needed. For example one could store the positions and velocities of the particles as `SVector` (from `StaticArrays`), allowing the compiler to unroll the inner loops for computing the distances. This leads to a code such as:" 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "id": "05d63a02", 111 | "metadata": {}, 112 | "outputs": [], 113 | "source": [ 114 | "function Vtot_static(Vpair, x)\n", 115 | " accum = zero(eltype(x))\n", 116 | " for (i, xi) in enumerate(x), j in 1:i\n", 117 | " xj = @inbounds x[j]\n", 118 | " accum += Vpair(norm(xi .- xj))\n", 119 | " end\n", 120 | " accum\n", 121 | "end" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "id": "3fbf1241", 128 | "metadata": {}, 129 | "outputs": [], 130 | "source": [ 131 | "# Which we can time as such:\n", 132 | "using StaticArrays\n", 133 | "xstatic = [@SVector randn(2) for _ in 1:1000]\n", 134 | "@btime Vtot_mfh($Vmorse, $xstatic)" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "id": "9dce8335", 140 | "metadata": {}, 141 | "source": [ 142 | "In practice one would furthermore employ a neighbour list (i.e. a list of all atoms within a certain cutoff range) to break the double loop over particles (`O(N^2)`) into a single loop over the neighbour list." 143 | ] 144 | }, 145 | { 146 | "cell_type": "markdown", 147 | "id": "1e296447", 148 | "metadata": {}, 149 | "source": [ 150 | "#### 2. Parallelism and speed:\n", 151 | "\n", 152 | "Again the simplest way to parallelise the loop is using `FLoops`:" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": null, 158 | "id": "89bc3b2c", 159 | "metadata": {}, 160 | "outputs": [], 161 | "source": [ 162 | "using FLoops\n", 163 | "\n", 164 | "function Vtot(Vpair, x)\n", 165 | " n_particles = size(x, 2)\n", 166 | " accu = zero(eltype(x)) # Get a zero of the appropriate type\n", 167 | " @floop for i in 1:n_particles\n", 168 | " @init local_accu = zero(eltype(x)) # Initialise thread-local accumulator\n", 169 | " for j in i+1:n_particles\n", 170 | " # Note: Since we accumulate into thread-local storage,\n", 171 | " # we don't need to use the blocking `@reduce` macro here.\n", 172 | " local_accu += @inbounds @views Vpair(norm(x[:, i] .- x[:, j]))\n", 173 | " end\n", 174 | " @reduce accu += local_accu # Accumulate\n", 175 | " end\n", 176 | " accu\n", 177 | "end" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "id": "b432dc44", 183 | "metadata": {}, 184 | "source": [ 185 | "Again, here also more clever improvements would be possible (e.g. better chunking of the data etc.), but the coding effort will be larger." 186 | ] 187 | }, 188 | { 189 | "cell_type": "markdown", 190 | "id": "c0ea3e7a", 191 | "metadata": {}, 192 | "source": [ 193 | "# [02_Types_Specialisation](02_Types_Specialisation.ipynb)\n", 194 | "\n", 195 | "1. It only needs to store the length and the position of the one-bit.\n", 196 | "\n", 197 | "2. A possible definition of the type is:" 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": null, 203 | "id": "8bfeaf11", 204 | "metadata": {}, 205 | "outputs": [], 206 | "source": [ 207 | "struct OneHot\n", 208 | " len::Int # Length\n", 209 | " ind::Int # Index of the one-bit\n", 210 | "end " 211 | ] 212 | }, 213 | { 214 | "cell_type": "markdown", 215 | "id": "352fb8ba", 216 | "metadata": {}, 217 | "source": [ 218 | "3. To support the indicated piece of code we need the following functions frome base:" 219 | ] 220 | }, 221 | { 222 | "cell_type": "code", 223 | "execution_count": null, 224 | "id": "2457ffe0", 225 | "metadata": {}, 226 | "outputs": [], 227 | "source": [ 228 | "import Base: *, getindex, length\n", 229 | "\n", 230 | "length(v::OneHot) = v.len\n", 231 | "getindex(v::OneHot, i::Int) = i == v.ind\n", 232 | "*(A::AbstractMatrix, v::OneHot) = A[:, v.ind]" 233 | ] 234 | }, 235 | { 236 | "cell_type": "markdown", 237 | "id": "edab8598", 238 | "metadata": {}, 239 | "source": [ 240 | "4. We benchmark the `innersum` for both indicated cases:" 241 | ] 242 | }, 243 | { 244 | "cell_type": "code", 245 | "execution_count": null, 246 | "id": "86ed1abe", 247 | "metadata": {}, 248 | "outputs": [], 249 | "source": [ 250 | "function innersum(A, vs)\n", 251 | " t = zero(eltype(A))\n", 252 | " for v in vs\n", 253 | " y = A * v\n", 254 | " for i in 1:length(vs[1])\n", 255 | " t += v[i] * y[i]\n", 256 | " end\n", 257 | " end\n", 258 | " t\n", 259 | "end" 260 | ] 261 | }, 262 | { 263 | "cell_type": "code", 264 | "execution_count": null, 265 | "id": "cab1a5b8", 266 | "metadata": {}, 267 | "outputs": [], 268 | "source": [ 269 | "using BenchmarkTools\n", 270 | "A = rand(3, 3)\n", 271 | "vs_float = [rand(3) for i in 1:10]\n", 272 | "vs_onehot = [OneHot(3, rand(1:3)) for _ in 1:10]\n", 273 | "\n", 274 | "@btime innersum($A, $vs_float);\n", 275 | "@btime innersum($A, $vs_onehot);" 276 | ] 277 | }, 278 | { 279 | "cell_type": "markdown", 280 | "id": "13a47ed7", 281 | "metadata": {}, 282 | "source": [ 283 | "In my benchmarks the speedup is about a factor of 2." 284 | ] 285 | }, 286 | { 287 | "cell_type": "markdown", 288 | "id": "ac4774d6", 289 | "metadata": {}, 290 | "source": [ 291 | "5. One way to define the `OneHotVector`:" 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "execution_count": null, 297 | "id": "70905276", 298 | "metadata": {}, 299 | "outputs": [], 300 | "source": [ 301 | "struct OneHotVector <: AbstractVector{Bool}\n", 302 | " len::Int\n", 303 | " ind::Int\n", 304 | "end\n", 305 | "\n", 306 | "Base.getindex(v::OneHotVector, i::Integer) = i == v.ind\n", 307 | "Base.size(v::OneHotVector) = (v.len, )" 308 | ] 309 | }, 310 | { 311 | "cell_type": "markdown", 312 | "id": "9eb121ff", 313 | "metadata": {}, 314 | "source": [ 315 | "6. Creating a single vector works ..." 316 | ] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": null, 321 | "id": "0f999f7b", 322 | "metadata": {}, 323 | "outputs": [], 324 | "source": [ 325 | "OneHotVector(5, 3)" 326 | ] 327 | }, 328 | { 329 | "cell_type": "markdown", 330 | "id": "4433a5ad", 331 | "metadata": {}, 332 | "source": [ 333 | "... and gives us a nice visualisation. Also, without any additional effort, the `innersum` just works:" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": null, 339 | "id": "ae7f1e09", 340 | "metadata": {}, 341 | "outputs": [], 342 | "source": [ 343 | "A = rand(3, 3)\n", 344 | "vs = [OneHotVector(3, rand(1:3)) for _ in 1:10]\n", 345 | "\n", 346 | "innersum(A, vs)" 347 | ] 348 | }, 349 | { 350 | "cell_type": "markdown", 351 | "id": "e6b799b5", 352 | "metadata": {}, 353 | "source": [ 354 | "# [03_Performance_Engineering](03_Performance_Engineering.ipynb)" 355 | ] 356 | }, 357 | { 358 | "cell_type": "markdown", 359 | "id": "d8e47a48", 360 | "metadata": {}, 361 | "source": [ 362 | "## Optimisation project 1" 363 | ] 364 | }, 365 | { 366 | "cell_type": "code", 367 | "execution_count": null, 368 | "id": "7f1deefe", 369 | "metadata": {}, 370 | "outputs": [], 371 | "source": [ 372 | "using BenchmarkTools\n", 373 | "\n", 374 | "N = 100\n", 375 | "A = rand(N, N)\n", 376 | "b = rand(N)\n", 377 | "c = 1.23;" 378 | ] 379 | }, 380 | { 381 | "cell_type": "markdown", 382 | "id": "6e37df68", 383 | "metadata": {}, 384 | "source": [ 385 | "#### Unoptimised code" 386 | ] 387 | }, 388 | { 389 | "cell_type": "code", 390 | "execution_count": null, 391 | "id": "8b9a9595", 392 | "metadata": {}, 393 | "outputs": [], 394 | "source": [ 395 | "function work!(A, N)\n", 396 | " D = zeros(N, N)\n", 397 | " for i in 1:N\n", 398 | " D = b[i] * c * A\n", 399 | " b[i] = sum(D)\n", 400 | " end\n", 401 | "end\n", 402 | "\n", 403 | "@btime work!($A, $N);" 404 | ] 405 | }, 406 | { 407 | "cell_type": "markdown", 408 | "id": "f7269cb1", 409 | "metadata": {}, 410 | "source": [ 411 | "First we run `@code_warntype` to check for type instabilities:" 412 | ] 413 | }, 414 | { 415 | "cell_type": "code", 416 | "execution_count": null, 417 | "id": "32e61639", 418 | "metadata": {}, 419 | "outputs": [], 420 | "source": [ 421 | "@code_warntype work!(A, N)" 422 | ] 423 | }, 424 | { 425 | "cell_type": "markdown", 426 | "id": "5127ea33", 427 | "metadata": {}, 428 | "source": [ 429 | "`D` is of type `Any`, because it depends on the global variables `b` and `c`. We fix that first:" 430 | ] 431 | }, 432 | { 433 | "cell_type": "markdown", 434 | "id": "03390ce0", 435 | "metadata": {}, 436 | "source": [ 437 | "#### Avoiding globals" 438 | ] 439 | }, 440 | { 441 | "cell_type": "code", 442 | "execution_count": null, 443 | "id": "0a9744c1", 444 | "metadata": {}, 445 | "outputs": [], 446 | "source": [ 447 | "function work2!(A, N, b, c)\n", 448 | " D = zeros(N, N)\n", 449 | " for i in 1:N\n", 450 | " D = b[i] * c * A\n", 451 | " b[i] = sum(D)\n", 452 | " end\n", 453 | "end\n", 454 | "\n", 455 | "@btime work2!($A, $N, $b, $c);" 456 | ] 457 | }, 458 | { 459 | "cell_type": "code", 460 | "execution_count": null, 461 | "id": "56209a4f", 462 | "metadata": {}, 463 | "outputs": [], 464 | "source": [ 465 | "@code_warntype work2!(A, N, b, c)" 466 | ] 467 | }, 468 | { 469 | "cell_type": "markdown", 470 | "id": "2e2ad1ac", 471 | "metadata": {}, 472 | "source": [ 473 | "#### Avoiding allocations\n", 474 | "\n", 475 | "That's fixed. Next we use vectorised operations to avoid allocations and avoid bounds checks:" 476 | ] 477 | }, 478 | { 479 | "cell_type": "code", 480 | "execution_count": null, 481 | "id": "aff0ba5d", 482 | "metadata": {}, 483 | "outputs": [], 484 | "source": [ 485 | "function work3!(A, N, b, c)\n", 486 | " D = zeros(N, N)\n", 487 | " @inbounds for i in 1:N\n", 488 | " @. D = b[i] * c * A\n", 489 | " b[i] = sum(D)\n", 490 | " end\n", 491 | "end\n", 492 | "\n", 493 | "@btime work3!($A, $N, $b, $c);" 494 | ] 495 | }, 496 | { 497 | "cell_type": "markdown", 498 | "id": "ae5cff6e", 499 | "metadata": {}, 500 | "source": [ 501 | "#### Improving the algorithm\n", 502 | "\n", 503 | "The multiplication by `b[i]` and `c` can be factored out:" 504 | ] 505 | }, 506 | { 507 | "cell_type": "code", 508 | "execution_count": null, 509 | "id": "0e866347", 510 | "metadata": {}, 511 | "outputs": [], 512 | "source": [ 513 | "function work4!(A, N, b, c)\n", 514 | " b .*= c * sum(A)\n", 515 | "end\n", 516 | "\n", 517 | "@btime work4!($A, $N, $b, $c);" 518 | ] 519 | }, 520 | { 521 | "cell_type": "markdown", 522 | "id": "5bcaf8e3", 523 | "metadata": {}, 524 | "source": [ 525 | "## Optimisation project 2" 526 | ] 527 | }, 528 | { 529 | "cell_type": "code", 530 | "execution_count": null, 531 | "id": "6f5e8833", 532 | "metadata": {}, 533 | "outputs": [], 534 | "source": [ 535 | "using BenchmarkTools\n", 536 | "\n", 537 | "N = 4000\n", 538 | "A = zeros(N,N)\n", 539 | "B = rand(N,N)\n", 540 | "v = rand(Int, N);" 541 | ] 542 | }, 543 | { 544 | "cell_type": "markdown", 545 | "id": "ec715357", 546 | "metadata": {}, 547 | "source": [ 548 | "#### Unoptimized code" 549 | ] 550 | }, 551 | { 552 | "cell_type": "code", 553 | "execution_count": null, 554 | "id": "4fcc1feb", 555 | "metadata": {}, 556 | "outputs": [], 557 | "source": [ 558 | "function work!(A, B, v, N)\n", 559 | " val = 0\n", 560 | " for i in 1:N\n", 561 | " for j in 1:N\n", 562 | " val = mod(v[i], 256);\n", 563 | " A[i, j] = B[i, j] * (sin(val) * sin(val) - cos(val) * cos(val));\n", 564 | " end\n", 565 | " end\n", 566 | "end\n", 567 | "\n", 568 | "runtime = @belapsed work!($A, $B, $v, $N);\n", 569 | "println(\"Performance: $(N^2 * 1e-6 / runtime) MIt/s\")" 570 | ] 571 | }, 572 | { 573 | "cell_type": "markdown", 574 | "id": "aa102a38", 575 | "metadata": {}, 576 | "source": [ 577 | "#### Simplification\n", 578 | "\n", 579 | "Notice:\n", 580 | "$$\n", 581 | " \\sin(x) \\sin(x) - \\cos(x) \\cos(x) = 1 - 2 \\cos(x) \\cos(x) = - \\cos(2x)\n", 582 | "$$" 583 | ] 584 | }, 585 | { 586 | "cell_type": "code", 587 | "execution_count": null, 588 | "id": "027a6d07", 589 | "metadata": {}, 590 | "outputs": [], 591 | "source": [ 592 | "x = rand()\n", 593 | "sin(x)*sin(x) - cos(x)*cos(x) ≈ -cos(2x)" 594 | ] 595 | }, 596 | { 597 | "cell_type": "code", 598 | "execution_count": null, 599 | "id": "0a580f84", 600 | "metadata": {}, 601 | "outputs": [], 602 | "source": [ 603 | "function work2!(A, B, v, N)\n", 604 | " for i in 1:N\n", 605 | " for j in 1:N \n", 606 | " val = -cos(2mod(v[i], 256))\n", 607 | " A[i, j] = B[i, j] * val\n", 608 | " end\n", 609 | " end\n", 610 | "end\n", 611 | "\n", 612 | "runtime = @belapsed work2!($A, $B, $v, $N);\n", 613 | "println(\"Performance: $(N^2 * 1e-6 / runtime) MIt/s\")" 614 | ] 615 | }, 616 | { 617 | "cell_type": "markdown", 618 | "id": "6b91c2bc", 619 | "metadata": {}, 620 | "source": [ 621 | "#### Avoiding recomputation in the inner loop\n", 622 | "\n", 623 | "We move the computation of the the second factor out of the inner loop:" 624 | ] 625 | }, 626 | { 627 | "cell_type": "code", 628 | "execution_count": null, 629 | "id": "9893eabd", 630 | "metadata": {}, 631 | "outputs": [], 632 | "source": [ 633 | "function work3!(A, B, v, N)\n", 634 | " for i in 1:N\n", 635 | " val = -cos(2mod(v[i], 256))\n", 636 | " for j in 1:N \n", 637 | " A[i, j] = B[i, j] * val\n", 638 | " end\n", 639 | " end\n", 640 | "end\n", 641 | "\n", 642 | "runtime = @belapsed work3!($A, $B, $v, $N);\n", 643 | "println(\"Performance: $(N^2 * 1e-6 / runtime) MIt/s\")" 644 | ] 645 | }, 646 | { 647 | "cell_type": "markdown", 648 | "id": "a4dbdbc5", 649 | "metadata": {}, 650 | "source": [ 651 | "#### Precompute val factor" 652 | ] 653 | }, 654 | { 655 | "cell_type": "code", 656 | "execution_count": null, 657 | "id": "cedc1221", 658 | "metadata": {}, 659 | "outputs": [], 660 | "source": [ 661 | "function work4!(A, B, v, N)\n", 662 | " val = -cos.(2mod.(v, 256))\n", 663 | " for i in 1:N\n", 664 | " for j in 1:N \n", 665 | " A[i, j] = B[i, j] * val[i]\n", 666 | " end\n", 667 | " end\n", 668 | "end\n", 669 | "runtime = @belapsed work4!($A, $B, $v, $N);\n", 670 | "println(\"Performance: $(N^2 * 1e-6 / runtime) MIt/s\")" 671 | ] 672 | }, 673 | { 674 | "cell_type": "markdown", 675 | "id": "57a7c388", 676 | "metadata": {}, 677 | "source": [ 678 | "#### Switch loop order" 679 | ] 680 | }, 681 | { 682 | "cell_type": "code", 683 | "execution_count": null, 684 | "id": "d32deda3", 685 | "metadata": {}, 686 | "outputs": [], 687 | "source": [ 688 | "function work5!(A, B, v, N)\n", 689 | " val = -cos.(2mod.(v, 256))\n", 690 | " for j in 1:N\n", 691 | " for i in 1:N \n", 692 | " A[i, j] = B[i, j] * val[i]\n", 693 | " end\n", 694 | " end\n", 695 | "end\n", 696 | "runtime = @belapsed work5!($A, $B, $v, $N);\n", 697 | "println(\"Performance: $(N^2 * 1e-6 / runtime) MIt/s\")" 698 | ] 699 | }, 700 | { 701 | "cell_type": "markdown", 702 | "id": "04955f96", 703 | "metadata": {}, 704 | "source": [ 705 | "#### Inbounds" 706 | ] 707 | }, 708 | { 709 | "cell_type": "code", 710 | "execution_count": null, 711 | "id": "91d4f11c", 712 | "metadata": {}, 713 | "outputs": [], 714 | "source": [ 715 | "function work6!(A, B, v, N)\n", 716 | " val = -cos.(2mod.(v, 256))\n", 717 | " @inbounds for j in 1:N\n", 718 | " for i in 1:N \n", 719 | " A[i, j] = B[i, j] * val[i]\n", 720 | " end\n", 721 | " end\n", 722 | "end\n", 723 | "runtime = @belapsed work6!($A, $B, $v, $N);\n", 724 | "println(\"Performance: $(N^2 * 1e-6 / runtime) MIt/s\")" 725 | ] 726 | } 727 | ], 728 | "metadata": { 729 | "@webio": { 730 | "lastCommId": null, 731 | "lastKernelId": null 732 | }, 733 | "kernelspec": { 734 | "display_name": "Julia (4 threads) 1.7.2", 735 | "language": "julia", 736 | "name": "julia-(4-threads)-1.7" 737 | }, 738 | "language_info": { 739 | "file_extension": ".jl", 740 | "mimetype": "application/julia", 741 | "name": "julia", 742 | "version": "1.7.2" 743 | } 744 | }, 745 | "nbformat": 4, 746 | "nbformat_minor": 5 747 | } 748 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Introduction to the Julia programming language 2 | 3 | Material for the RWTH Julia workshop taking place on 17th and 18th February 2022. 4 | For more details [check out the course website](https://michael-herbst.com/teaching/2022-rwth-julia-workshop/). 5 | 6 | ## Software and material 7 | What you need for the workshop (quick overview): 8 | 9 | - [Julia 1.8](https://julialang.org/downloads/) 10 | - [Jupyter](https://jupyter.org/) and [IJulia.jl](https://github.com/JuliaLang/IJulia.jl) 11 | - This repository of workshop materials 12 | - All required dependencies (Julia packages) for the workshop 13 | 14 | ### Getting Julia 15 | For following the course you will need at least **Julia 1.8**. 16 | Julia can be easily obtained in binary form from [Julia downloads](https://julialang.org/downloads/). 17 | 18 | ### Getting all the rest 19 | To get the remaining files and dependencies 20 | start up `julia` and in the resulting REPL shell, 21 | copy and paste the following: 22 | 23 | ```julia 24 | import Downloads 25 | script = Downloads.download("https://raw.githubusercontent.com/mfherbst/2022-rwth-julia-workshop/master/install.jl") 26 | include(script) 27 | ``` 28 | This [downloads the install.jl script](https://raw.githubusercontent.com/mfherbst/2022-rwth-julia-workshop/master/install.jl) 29 | and runs it from julia. 30 | Follow the instructions on the screen and start the Jupyter notebook server 31 | with the command that will be printed. 32 | 33 | As an alternative you can also also run the following commands manually 34 | (this requires to have `git` and `julia` available from the commandline): 35 | ``` 36 | git clone https://github.com/mfherbst/2022-rwth-julia-workshop 37 | cd 2022-rwth-julia-workshop 38 | julia install-manual.jl 39 | ``` 40 | 41 | ### Troubleshooting 42 | If you are facing issues, check out 43 | the [great troubleshooting section](https://carstenbauer.github.io/WorkshopWizard.jl/dev/troubleshooting/) 44 | from the WorkshopWizard package by Carsten Bauer (which `install.jl` is using). 45 | 46 | ### Check everything works 47 | There is a section in the [00_Installation](00_Installation.ipynb) notebook 48 | with a few quick commands to check everything works as expected. 49 | Please run these **before the course**. 50 | 51 | ## Working with these notes online (Beta) 52 | Click on the [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/mfherbst/2022-rwth-julia-workshop/master?labpath=00_Introduction.ipynb) 53 | badge to work with these notes online (without a local Julia installation). 54 | Note that for some of the exercises the computational performance available on 55 | binder might not be sufficient. 56 | -------------------------------------------------------------------------------- /Solutions_to_Exercises.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# [01_Variables_Control_Packages](01_Variables_Control_Packages.ipynb)\n", 8 | "\n", 9 | "Compute $2^{100}$:\n", 10 | "- As fast as possible\n", 11 | "- As exact as possible" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": null, 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "# This will give the wrong result:\n", 21 | "2^100" 22 | ] 23 | }, 24 | { 25 | "cell_type": "code", 26 | "execution_count": null, 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "# An exact way to compute the answer is using arbitrary-precision integers\n", 31 | "BigInt(2)^100" 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": null, 37 | "metadata": {}, 38 | "outputs": [], 39 | "source": [ 40 | "# Usually faster than arbitrary-precision computations are\n", 41 | "# floating point operations (might not be true for this simple case, however)\n", 42 | "2.0^100" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": {}, 48 | "source": [ 49 | "### Exercise\n", 50 | "Compute\n", 51 | "$$ 15! \\qquad 100! \\qquad \\left(\\begin{array}{c} 100 \\\\ 15 \\end{array}\\right) $$\n", 52 | "with the Julia you know so far." 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "res15 = 1\n", 62 | "for i in 1:15\n", 63 | " res15 = res15 * i\n", 64 | "end\n", 65 | "\n", 66 | "res100 = BigInt(1)\n", 67 | "for i in 1:100\n", 68 | " res100 = res100 * i\n", 69 | "end\n", 70 | "\n", 71 | "res85 = BigInt(1)\n", 72 | "for i in 1:(100 - 15)\n", 73 | " res85 = res85 * i\n", 74 | "end\n", 75 | "\n", 76 | "res100 / res85 / res15" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "# [02_Functions_Types_Dispatch](02_Functions_Types_Dispatch.ipynb)\n", 84 | "\n", 85 | "### Exercise\n", 86 | "Which of the following type are subtypes of another?\n", 87 | "Try to guess first and then verify by using the operator `<:`.\n", 88 | "\n", 89 | "```julia\n", 90 | "Float64 AbstractFloat Integer\n", 91 | "Number AbstractArray Complex\n", 92 | "Real Any Nothing\n", 93 | "```" 94 | ] 95 | }, 96 | { 97 | "cell_type": "markdown", 98 | "metadata": {}, 99 | "source": [ 100 | "The following type chains (subtype ``<:`` supertype) are true:" 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": null, 106 | "metadata": {}, 107 | "outputs": [], 108 | "source": [ 109 | "Float64 <: AbstractFloat <: Real <: Number <: Any" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": null, 115 | "metadata": {}, 116 | "outputs": [], 117 | "source": [ 118 | "Integer <: Real <: Number <: Any" 119 | ] 120 | }, 121 | { 122 | "cell_type": "code", 123 | "execution_count": null, 124 | "metadata": {}, 125 | "outputs": [], 126 | "source": [ 127 | "Complex <: Number <: Any" 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": null, 133 | "metadata": {}, 134 | "outputs": [], 135 | "source": [ 136 | "Nothing <: Any" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": null, 142 | "metadata": {}, 143 | "outputs": [], 144 | "source": [ 145 | "AbstractArray <: Any" 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "metadata": {}, 151 | "source": [ 152 | "An elegant way to find this information is:" 153 | ] 154 | }, 155 | { 156 | "cell_type": "code", 157 | "execution_count": null, 158 | "metadata": {}, 159 | "outputs": [], 160 | "source": [ 161 | "types = (\n", 162 | " Float64, AbstractFloat, Integer,\n", 163 | " Number, AbstractArray, Complex,\n", 164 | " Real, Any, Nothing,\n", 165 | ")\n", 166 | "\n", 167 | "using Printf\n", 168 | "for u in types, v in types\n", 169 | " @printf \"%15s <: %15s = %s\" \"$u\" \"$v\" \"$(u <: v)\\n\"\n", 170 | "end" 171 | ] 172 | }, 173 | { 174 | "cell_type": "markdown", 175 | "metadata": {}, 176 | "source": [ 177 | "# [03_Arrays_Parametric_Types](03_Arrays_Parametric_Types.ipynb)\n", 178 | "\n", 179 | "### Exercise\n", 180 | "Create the following arrays using Julia code:\n", 181 | "$$\\left(\\begin{array}{ccccc}\n", 182 | " 2&2&2&2&2 \\\\\n", 183 | " 2&2&2&2&2 \\\\\n", 184 | " 2&2&2&2&2 \\\\\n", 185 | " \\end{array}\\right) \\qquad\n", 186 | " \\left(\\begin{array}{cccc}\n", 187 | " 0.1&0.5&0.9&1.3\\\\\n", 188 | " 0.2&0.6&1.0&1.4\\\\\n", 189 | " 0.3&0.7&1.1&1.5\\\\\n", 190 | " 0.4&0.8&1.2&1.6\\\\\n", 191 | " \\end{array}\\right)\n", 192 | " $$" 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": null, 198 | "metadata": {}, 199 | "outputs": [], 200 | "source": [ 201 | "ones(Int, 3, 5) + ones(Int, 3, 5)\n", 202 | "\n", 203 | "# or \n", 204 | "\n", 205 | "2ones(Int, 3, 5)" 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": {}, 212 | "outputs": [], 213 | "source": [ 214 | "reshape(1:16, 4, 4) / 10" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": null, 220 | "metadata": {}, 221 | "outputs": [], 222 | "source": [ 223 | "reshape(0.1:0.1:1.6, 4, 4)" 224 | ] 225 | } 226 | ], 227 | "metadata": { 228 | "kernelspec": { 229 | "display_name": "Julia 1.7.2", 230 | "language": "julia", 231 | "name": "julia-1.7" 232 | }, 233 | "language_info": { 234 | "file_extension": ".jl", 235 | "mimetype": "application/julia", 236 | "name": "julia", 237 | "version": "1.7.2" 238 | } 239 | }, 240 | "nbformat": 4, 241 | "nbformat_minor": 2 242 | } 243 | -------------------------------------------------------------------------------- /img/from_source_to_native.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mfherbst/2022-rwth-julia-workshop/f606ab4cb6964d5dbe0ec2da1855c542b7a7a9ae/img/from_source_to_native.png -------------------------------------------------------------------------------- /install-manual.jl: -------------------------------------------------------------------------------- 1 | using Pkg 2 | println("Activating environment in $(pwd())...") 3 | Pkg.activate(".") 4 | println("Installing packages...") 5 | flush(stdout) 6 | Pkg.instantiate() 7 | Pkg.precompile() 8 | 9 | println("Done!") 10 | -------------------------------------------------------------------------------- /install.jl: -------------------------------------------------------------------------------- 1 | using Pkg 2 | Pkg.add(url="https://github.com/carstenbauer/WorkshopWizard.jl/") 3 | using WorkshopWizard 4 | 5 | if Sys.iswindows() 6 | path = joinpath(homedir(), "Desktop") 7 | else 8 | path = homedir() 9 | end 10 | success = WorkshopWizard.install(;repo="https://github.com/mfherbst/2022-rwth-julia-workshop", 11 | path, 12 | global_IJulia=true, 13 | auto_overwrite=false) 14 | 15 | if success 16 | @info "That's it. Start the notebook server with 'using IJulia; notebook(dir=\"$path/2022-rwth-julia-workshop\")' ..." 17 | end 18 | --------------------------------------------------------------------------------