├── DataFrames ├── build.sh ├── datatable.pdf ├── slides.md └── slides.pdf ├── GridOfResistors ├── GridOfResistors.md ├── battery.gif ├── grid2d-comp.jl ├── grid2d-devec-fast.jl ├── grid2d-devec-slow.jl ├── grid2d-vec.jl ├── grid2d.c └── grid2d.m ├── LightningRound ├── IAP_2013_Lightning.pdf ├── IAP_2013_Lightning.pptx ├── dice └── lightning.jl ├── NumericalOptimization ├── ExampleSolution.jl ├── Tutorial.jl ├── activityplotdeterm.pdf ├── activityplotnorm.pdf ├── presentation.pdf ├── presentation.tex ├── refs.bib ├── tutorial.pdf └── tutorial.tex ├── README.md ├── Stats ├── header.tex ├── slides.md └── slides.pdf └── Vision └── vision.pdf /DataFrames/build.sh: -------------------------------------------------------------------------------- 1 | pandoc -t beamer slides.md -o slides.pdf 2 | #pandoc slides.md -o slides.tex 3 | -------------------------------------------------------------------------------- /DataFrames/datatable.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JuliaAttic/julia-tutorial/4acd63163586a9016a31b3fad2ba28a32c14c84b/DataFrames/datatable.pdf -------------------------------------------------------------------------------- /DataFrames/slides.md: -------------------------------------------------------------------------------- 1 | % Working with Data in Julia 2 | % John Myles White 3 | % \today 4 | 5 | --- 6 | 7 | julia> Pkg.add("DataFrames") 8 | julia> Pkg.add("RDatasets") 9 | julia> using DataFrames 10 | julia> using RDatasets 11 | 12 | --- 13 | 14 | \begin{center} 15 | \bf{How do we cope with missing data?} 16 | \end{center} 17 | 18 | --- 19 | 20 | v = [x1, x2, x3, x4, x5] 21 | 22 | mean(v) 23 | 24 | --- 25 | 26 | v = [0.5, 0.6, 0.7, 0.8, 0.9] 27 | 28 | mean(v) 29 | 30 | --- 31 | 32 | v = [0.5, 0.6, 0.7, NA, 0.9] 33 | 34 | mean(v) 35 | 36 | --- 37 | 38 | The `NA` type: 39 | 40 | * Represents a missing value 41 | * Like `NULL` in some systems 42 | * Poisons other values 43 | * Like `NaN` for floating point numbers 44 | 45 | --- 46 | 47 | julia> 1 + NA 48 | NA 49 | 50 | julia> 1 > NA 51 | NA 52 | 53 | julia> isna(NA) 54 | true 55 | 56 | --- 57 | 58 | * `DataArray{T}` adds NA's to `Array{T}` 59 | * `DataArray{T}` can store `T` or `NA` 60 | 61 | --- 62 | 63 | julia> dv = DataArray([1, 2, 3]) 64 | 3-element Int64 DataArray: 65 | 1 66 | 2 67 | 3 68 | 69 | julia> dv[1] = NA 70 | NA 71 | 72 | julia> join(dv, "::") 73 | "NA::2::3" 74 | 75 | --- 76 | 77 | Convenience constructors: 78 | 79 | * `datazeros()` 80 | * `dataones()` 81 | * `datafalses()` 82 | * `datatrues()` 83 | * `dataeye()` 84 | * `datadiagm()` 85 | 86 | --- 87 | 88 | julia> dm = dataeye(2) 89 | 2x2 Float64 DataArray: 90 | 1.0 0.0 91 | 0.0 1.0 92 | 93 | julia> svd(dm) 94 | ( 95 | 2x2 Float64 Array: 96 | 1.0 0.0 97 | 0.0 1.0, 98 | [1.0, 1.0], 99 | 2x2 Float64 Array: 100 | 1.0 0.0 101 | 0.0 1.0) 102 | 103 | --- 104 | 105 | Convenience converters: 106 | 107 | * `dataint()` 108 | * `datafloat()` 109 | * `databool()` 110 | 111 | --- 112 | 113 | julia> dataint([1.0, 2.0, 3.0]) 114 | 3-element Int64 DataArray: 115 | 1 116 | 2 117 | 3 118 | 119 | julia> databool([1, 0, 1]) 120 | 3-element Bool DataArray: 121 | true 122 | false 123 | true 124 | 125 | --- 126 | 127 | \begin{center} 128 | \bf{How do we store data efficiently?} 129 | \end{center} 130 | 131 | --- 132 | 133 | `PooledDataArray{T}` compresses `DataArray{T}` 134 | 135 | --- 136 | 137 | julia> pda = PooledDataArray(["AA", "BB", "BB", "AA"]) 138 | 4-element ASCIIString PooledDataArray: 139 | "AA" 140 | "BB" 141 | "BB" 142 | "AA" 143 | 144 | julia> pda[1] = NA 145 | NA 146 | 147 | --- 148 | 149 | julia> levels(pda) 150 | 3-element ASCIIString DataArray: 151 | "AA" 152 | "BB" 153 | NA 154 | 155 | --- 156 | 157 | julia> pda.refs 158 | 4-element Uint16 Array: 159 | 0x0000 160 | 0x0002 161 | 0x0002 162 | 0x0001 163 | 164 | julia> pda.pool 165 | 2-element ASCIIString Array: 166 | "AA" 167 | "BB 168 | 169 | --- 170 | 171 | \begin{center} 172 | \bf{How do we cope with heteregeneous data?} 173 | \end{center} 174 | 175 | --- 176 | 177 | \begin{center} 178 | \includegraphics[scale = 0.5]{datatable.pdf} 179 | \end{center} 180 | 181 | --- 182 | 183 | df = DataFrame() 184 | df["Name"] = DataVector["John Smith", "Jane Doe"] 185 | df["Height"] = DataVector[73.0, 68.0] 186 | df["Weight"] = DataVector[NA, 130] 187 | df["Gender"] = DataVector["Male", "Female"] 188 | 189 | --- 190 | 191 | julia> df 192 | 2x4 DataFrame: 193 | Name Height Weight Gender 194 | [1,] "John Smith" 73.0 NA "Male" 195 | [2,] "Jane Doe" 68.0 130 "Female" 196 | 197 | --- 198 | 199 | julia> df["Weight"] 200 | 2-element Int64 DataArray: 201 | NA 202 | 130 203 | 204 | --- 205 | 206 | * A `DataFrame` is a list of `DataVector`'s 207 | * `DataFrame`'s allow mixed indexing: 208 | * Columns by number 209 | * Columns by name 210 | * Rows + Columns by number + number 211 | * Rows + Columns by number + name 212 | 213 | --- 214 | 215 | julia> df = DataFrame() 216 | 0x0 DataFrame: 217 | 218 | julia> df["A"] = dataones(3) 219 | 3-element Float64 DataArray: 220 | 1.0 221 | 1.0 222 | 1.0 223 | 224 | julia> df[2] = datazeros(Int, 3) 225 | 3-element Int64 DataArray: 226 | 0 227 | 0 228 | 0 229 | 230 | --- 231 | 232 | julia> df 233 | 3x2 DataFrame: 234 | A x2 235 | [1,] 1.0 0 236 | [2,] 1.0 0 237 | [3,] 1.0 0 238 | 239 | julia> df[1, 1] 240 | 1.0 241 | 242 | julia> df[1, "A"] 243 | 1.0 244 | 245 | --- 246 | 247 | julia> path = joinpath(julia_pkgdir(), 248 | "DataFrames", 249 | "test", "data", "types.csv") 250 | "/Users/johnmyleswhite/.julia/DataFrames/te... 251 | 252 | --- 253 | 254 | julia> types = read_table(path) 255 | 3x5 DataFrame: 256 | IntColumn IntlikeColumn FloatColumn BoolColumn... 257 | [1,] 1 1.0 3.1 true... 258 | [2,] 2 7.0 -3.1e8 false... 259 | [3,] -1 7.0 -3.1e-8 false... 260 | 261 | --- 262 | 263 | julia> head(types, 2) 264 | 3x5 DataFrame: 265 | IntColumn IntlikeColumn FloatColumn BoolColumn... 266 | [1,] 1 1.0 3.1 true... 267 | [2,] 2 7.0 -3.1e8 false... 268 | 269 | --- 270 | 271 | julia> tail(types, 2) 272 | 3x5 DataFrame: 273 | IntColumn IntlikeColumn FloatColumn BoolColumn... 274 | [2,] 2 7.0 -3.1e8 false... 275 | [3,] -1 7.0 -3.1e-8 false... 276 | 277 | --- 278 | 279 | julia> types[1:3, 1:5] 280 | 3x5 DataFrame: 281 | IntColumn IntlikeColumn FloatColumn BoolColumn... 282 | [1,] 1 1.0 3.1 true... 283 | [2,] 2 7.0 -3.1e8 false... 284 | [3,] -1 7.0 -3.1e-8 false... 285 | 286 | --- 287 | 288 | julia> iris = data("datasets", "iris") 289 | 150x6 DataFrame: 290 | Sepal.Length Sepal.Width Petal.Length... 291 | [1,] 1 5.1 3.5 1.4... 292 | [2,] 2 4.9 3.0 1.4... 293 | ... 294 | [149,] 149 6.2 3.4 5.4... 295 | [150,] 150 5.9 3.0 5.1... 296 | 297 | --- 298 | 299 | julia> RDatasets.datasets() 300 | 570-element Any Array: 301 | ["COUNT", "affairs"] 302 | ["COUNT", "azdrg112"] 303 | ... 304 | ["vcd", "WeldonDice"] 305 | ["vcd", "WomenQueue"] 306 | 307 | --- 308 | 309 | julia> colnames(iris) 310 | 6-element Union(ASCIIString,UTF8String) Array: 311 | "" 312 | "Sepal.Length" 313 | "Sepal.Width" 314 | "Petal.Length" 315 | "Petal.Width" 316 | "Species" 317 | 318 | --- 319 | 320 | julia> coltypes(iris) 321 | 6-element Any Array: 322 | Int64 323 | Float64 324 | Float64 325 | Float64 326 | Float64 327 | UTF8String 328 | 329 | --- 330 | 331 | julia> clean_colnames!(iris) 332 | 333 | julia> colnames(iris) 334 | 6-element Union(ASCIIString,UTF8String) Array: 335 | "" 336 | "Sepal_Length" 337 | "Sepal_Width" 338 | "Petal_Length" 339 | "Petal_Width" 340 | "Species" 341 | 342 | --- 343 | 344 | julia> size(iris) 345 | (150,6) 346 | 347 | julia> nrow(iris) 348 | 150 349 | 350 | julia> ncol(iris) 351 | 6 352 | 353 | --- 354 | 355 | julia> vcat(iris, iris) 356 | ... 357 | 358 | julia> hcat(iris, iris) 359 | ... 360 | 361 | julia> rbind(iris, iris) 362 | ... 363 | 364 | julia> cbind(iris, iris) 365 | ... 366 | 367 | --- 368 | 369 | julia> iris[1, 1] = NA 370 | NA 371 | 372 | julia> head(iris) 373 | 6x6 DataFrame: 374 | Sepal_Length Sepal_Width Petal_Length... 375 | [1,] NA 5.1 3.5 1.4... 376 | [2,] 2 4.9 3.0 1.4... 377 | [3,] 3 4.7 3.2 1.3... 378 | [4,] 4 4.6 3.1 1.5... 379 | [5,] 5 5.0 3.6 1.4... 380 | [6,] 6 5.4 3.9 1.7... 381 | 382 | --- 383 | 384 | julia> complete_cases(iris) 385 | 150-element Bool Array: 386 | false 387 | true 388 | ... 389 | true 390 | true 391 | 392 | --- 393 | 394 | julia> iris[complete_cases(iris), :] 395 | ... 396 | 397 | --- 398 | 399 | julia> complete_cases!(iris) 400 | 401 | julia> head(iris) 402 | 6x6 DataFrame: 403 | Sepal_Length Sepal_Width Petal_Length... 404 | [1,] 2 4.9 3.0 1.4... 405 | [2,] 3 4.7 3.2 1.3... 406 | [3,] 4 4.6 3.1 1.5... 407 | [4,] 5 5.0 3.6 1.4... 408 | [5,] 6 5.4 3.9 1.7... 409 | [6,] 7 4.6 3.4 1.4... 410 | 411 | --- 412 | 413 | julia> any(duplicated(iris)) 414 | false 415 | 416 | julia> any(duplicated(rbind(iris, iris))) 417 | true 418 | 419 | --- 420 | 421 | julia> new_iris = rbind(iris, iris) 422 | ... 423 | 424 | julia> drop_duplicates!(new_iris) 425 | 426 | julia> nrow(new_iris) 427 | 149 428 | 429 | --- 430 | 431 | julia> vector(iris["Species"]) 432 | 149-element UTF8String Array: 433 | "setosa" 434 | "setosa" 435 | ... 436 | "virginica" 437 | "virginica" 438 | 439 | --- 440 | 441 | julia> vector(iris["Species"], Any) 442 | 149-element Any Array: 443 | "setosa" 444 | "setosa" 445 | ... 446 | "virginica" 447 | "virginica" 448 | 449 | --- 450 | 451 | julia> matrix(iris) 452 | julia> matrix(iris[:, 2:3]) 453 | julia> matrix(iris[:, 1:3]) 454 | julia> matrix(iris[:, 1:3], Any) 455 | 456 | --- 457 | 458 | julia> with(iris, :(Petal_Length .* Petal_Width)) 459 | 149-element Float64 DataArray: 460 | 0.28 461 | 0.26 462 | ... 463 | 12.42 464 | 9.18 465 | 466 | --- 467 | 468 | julia> within!(iris, 469 | :(Petal_Area = Petal_Length .* Petal_Width)) 470 | julia> head(iris) 471 | 6x7 DataFrame: 472 | Sepal_Length Sepal_Width Petal_Length... 473 | [1,] 2 4.9 3.0 1.4... 474 | [2,] 3 4.7 3.2 1.3... 475 | [3,] 4 4.6 3.1 1.5... 476 | [4,] 5 5.0 3.6 1.4... 477 | [5,] 6 5.4 3.9 1.7... 478 | [6,] 7 4.6 3.4 1.4... 479 | 480 | --- 481 | 482 | Database operations on DataFrames: 483 | 484 | * subset 485 | * merge 486 | * groupby 487 | 488 | --- 489 | 490 | julia> subset(iris, :(Species .== "setosa")) 491 | julia> nrow(subset(iris, :(Species .== "setosa"))) 492 | 49 493 | 494 | --- 495 | 496 | julia> df1 = DataFrame({"a" => [1, 2, 3], 497 | "b" => ["America", "Europe", 498 | "Africa"]}) 499 | julia> df2 = DataFrame({"a" => [1, 2, 4], 500 | "c" => ["New World", "Old World", 501 | "New World"]}) 502 | 503 | --- 504 | 505 | julia> merge(df1, df2) 506 | 2x3 DataFrame: 507 | a b c 508 | [1,] 1 "America" "New World" 509 | [2,] 2 "Europe" "Old World" 510 | 511 | --- 512 | 513 | julia> merge(df1, df2, "a", "inner") 514 | 2x3 DataFrame: 515 | a b c 516 | [1,] 1 "America" "New World" 517 | [2,] 2 "Europe" "Old World" 518 | 519 | --- 520 | 521 | julia> merge(df1, df2, "a", "left") 522 | 3x3 DataFrame: 523 | a b c 524 | [1,] 1 "America" "New World" 525 | [2,] 2 "Europe" "Old World" 526 | [3,] 3 "Africa" NA 527 | 528 | --- 529 | 530 | julia> merge(df1, df2, "a", "right") 531 | 3x3 DataFrame: 532 | a b c 533 | [1,] 1 "America" "New World" 534 | [2,] 2 "Europe" "Old World" 535 | [3,] NA NA "New World" 536 | 537 | --- 538 | 539 | julia> merge(df1, df2, "a", "outer") 540 | 4x3 DataFrame: 541 | a b c 542 | [1,] 1 "America" "New World" 543 | [2,] 2 "Europe" "Old World" 544 | [3,] 3 "Africa" NA 545 | [4,] NA NA "New World" 546 | 547 | --- 548 | 549 | The Split-Apply-Combine Strategy: 550 | 551 | * Segment data into groups 552 | * Apply a function to each group independently 553 | * Combine results into a single DataFrame 554 | 555 | --- 556 | 557 | julia> movies = data("ggplot2", "movies") 558 | ... 559 | 560 | --- 561 | 562 | julia> by(movies, "year", nrow) 563 | 113x2 DataFrame: 564 | year x1 565 | [1,] 1893 1 566 | [2,] 1894 9 567 | ... 568 | [112,] 2004 1945 569 | [113,] 2005 349 570 | 571 | --- 572 | 573 | julia> subset(movies, :(year .== 1893)) 574 | 1x25 SubDataFrame: 575 | title year length budget... 576 | [1,] 6076 "Blacksmith Scene" 1893 1 NA... 577 | 578 | --- 579 | 580 | julia> by(movies, ["Action", "year"], nrow) 581 | 205x3 DataFrame: 582 | Action year x1 583 | [1,] 0 1893 1 584 | [2,] 0 1894 9 585 | ... 586 | [204,] 0 2005 306 587 | [205,] 1 2005 43 588 | 589 | --- 590 | 591 | # Exercises 592 | 593 | * Create some DataArray's 594 | * A DataArray containing the first five primes 595 | * A DataArray containing only NA's 596 | * A DataArray containing DataArray's 597 | 598 | --- 599 | 600 | * Create some PooledDataArray's 601 | * A PooledDataArray of all strings 602 | * A PooledDataArray of repeated ComplexPair's 603 | 604 | --- 605 | 606 | * Create some DataFrame's 607 | * A 4x3 DataFrame w/: 608 | * 4 String's 609 | * 4 Int's 610 | * 4 ComplexPair's 611 | 612 | --- 613 | 614 | * Load more datasets from RDatasets: 615 | * Search using `RDatasets.datasets()` 616 | 617 | --- 618 | 619 | * Do some Split-Apply-Combine Working on `iris`: 620 | * Find the median petal area 621 | * Find the variance of the petal area 622 | * Find the centroid of the sepal and petal dimensions 623 | 624 | --- 625 | 626 | * Do some modeling with other packages 627 | * Clustering package 628 | * kNN package 629 | * DecisionTree package 630 | 631 | --- 632 | 633 | Pkg.add("Clustering") 634 | using Clustering 635 | iris = data("datasets", "iris") 636 | k_means(matrix(iris[:, 2:5]), 3) 637 | iris["Cluster"] = 638 | k_means(matrix(iris[:, 2:5]), 3).assignments 639 | by(iris, ["Cluster", "Species"], nrow) 640 | 641 | --- 642 | 643 | Pkg.add("kNN") 644 | Pkg.add("Resampling") 645 | using kNN, Resampling 646 | iris = data("datasets", "iris") 647 | train, test = splitrandom(iris, 0.80) 648 | test["Guess"] = knn(matrix(train[:, 2:5]), 649 | matrix(test[:, 2:5]), 650 | vector(train[:, 6]), 651 | 10) 652 | by(test, ["Species", "Guess"], nrow) 653 | mean(test["Guess"] .== test["Species"]) 654 | 655 | --- 656 | 657 | Pkg.add("DecisionTree") 658 | using DecisionTree, Resampling 659 | iris = data("datasets", "iris") 660 | train, test = splitrandom(iris, 0.80) 661 | train_features = matrix(train[:, 2:5]) 662 | train_labels = vector(train[:, "Species"]) 663 | t = build_tree(train_features, train_labels) 664 | preds = apply_tree(t, matrix(test[:, 2:5])) 665 | mean(preds .== test["Species"]) 666 | -------------------------------------------------------------------------------- /DataFrames/slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JuliaAttic/julia-tutorial/4acd63163586a9016a31b3fad2ba28a32c14c84b/DataFrames/slides.pdf -------------------------------------------------------------------------------- /GridOfResistors/GridOfResistors.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 |

Grid of Resistors

6 | 7 |

Description

8 | 9 | The problem is to compute the voltages and the effective 10 | resistance of a 2n+1 by 2n+2 grid of 1 ohm resistors if 11 | a battery is connected to the two center points. This is a discrete 12 | version of finding the lines of force using iron filings for a magnet. 13 | The picture below describes the two dimensional problem. 14 | 15 | 16 | 17 | The method of solution that we will use here is successive 18 | overrelaxation (SOR) with red-black ordering. This is certainly 19 | not the fastest way to solve the problem, but it does illustrate many 20 | important programming ideas. 21 | 22 | It is not so important that you know the details of SOR. Some of the 23 | basic ideas may be found on pages 407-409 of Gil 24 | Strang's Introduction 25 | to Applied Mathematics. A somewhat more in-depth discussion may be 26 | found in any serious numerical analysis text such as Stoer and 27 | Bulirsch's Introduction to Numerical Analysis. What is 28 | important is that you see that the nodes are divided in half into red 29 | nodes and black nodes. During the first pass, the red nodes obtain the 30 | voltages as a weighted average of their original voltage, the input 31 | (if any) and the four surrounding black nodes. During the 32 | second pass, the black nodes obtain voltages from the four surrounding 33 | red nodes. The process converges in the limit to the correct answer 34 | for the finite grid. 35 | 36 | A C program is provided in `sor2d.c` and a MATLAB version in 37 | `sor2d.m`. Implement the following versions in julia and compare the 38 | timings of the different approaches: 39 | 40 | 1. A vectorized version. 41 | 2. A devectorized version. 42 | 3. Create a stencil function and refactor the stencil part of the computation. 43 | 4. Use the stencil function and implement using comprehensions. 44 | 5. Use a macro for the node updates. 45 | 46 | 47 | 48 | -------------------------------------------------------------------------------- /GridOfResistors/battery.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JuliaAttic/julia-tutorial/4acd63163586a9016a31b3fad2ba28a32c14c84b/GridOfResistors/battery.gif -------------------------------------------------------------------------------- /GridOfResistors/grid2d-comp.jl: -------------------------------------------------------------------------------- 1 | # grid2d.jl - Grid of Resistors problem in two dimensions using array comprehensions 2 | 3 | n = length(ARGS) >= 1 ? int(ARGS[1]) : 1000 4 | niter = length(ARGS) >= 2 ? int(ARGS[2]) : 10 5 | 6 | stencil(v,i,j,om) = (1-om)*v[i,j] + om*0.25(v[i+1,j]+v[i-1,j]+v[i,j+1]+v[i,j-1]) 7 | 8 | function do_sor(n, niter) 9 | 10 | v = zeros(2n+1,2n+2) 11 | 12 | mu = (cos(pi/(2n))+cos(pi/(2n+1)))/2 13 | om = 2(1-sqrt(1-mu^2))/mu^2 14 | 15 | tic() 16 | for k = 1:niter 17 | v[2:2:2n , 2:2:2n ] = [ stencil(v, i, j, om) for i=2:2:2n, j=2:2:2n ] 18 | v[3:2:2n-1, 3:2:2n+1] = [ stencil(v, i, j, om) for i=3:2:2n-1, j=3:2:2n+1 ] 19 | v[n+1,n+1] += 0.25om 20 | 21 | v[2:2:2n , 3:2:2n+1] = [ stencil(v, i, j, om) for i=2:2:2n, j=3:2:2n+1 ] 22 | v[3:2:2n-1, 2:2:2n ] = [ stencil(v, i, j, om) for i=3:2:2n-1, j=2:2:2n ] 23 | v[n+1,n+2] -= 0.25om 24 | 25 | r = 2v[n+1,n+1] 26 | println("Iter = $k, r = $r") 27 | end 28 | toq() 29 | 30 | end 31 | 32 | tottime = do_sor(n, niter) 33 | println("Time/iteration = $(tottime/niter) s") 34 | -------------------------------------------------------------------------------- /GridOfResistors/grid2d-devec-fast.jl: -------------------------------------------------------------------------------- 1 | # grid2d.jl - Grid of Resistors problem in two dimensions 2 | # Devectorized version that is fast due to stride 1 array accesses 3 | 4 | n = length(ARGS) >= 1 ? int(ARGS[1]) : 1000 5 | niter = length(ARGS) >= 2 ? int(ARGS[2]) : 10 6 | 7 | function do_sor(n, niter) 8 | 9 | v = zeros(2n+1,2n+2) 10 | 11 | mu = (cos(pi/(2n))+cos(pi/(2n+1)))/2 12 | om = 2(1-sqrt(1-mu^2))/mu^2 13 | 14 | tic() 15 | for k = 1:niter 16 | for je = 2:2:2n, ie = 2:2:2n 17 | v[ie,je] = (1-om)*v[ie,je] + om*0.25(v[ie+1,je]+v[ie-1,je]+v[ie,je+1]+v[ie,je-1]) 18 | end 19 | 20 | for jo = 3:2:2n+1, io = 3:2:2n-1 21 | v[io,jo] = (1-om)*v[io,jo] + om*0.25(v[io+1,jo]+v[io-1,jo]+v[io,jo+1]+v[io,jo-1]) 22 | end 23 | v[n+1,n+1] = v[n+1,n+1]+0.25om 24 | 25 | for jo = 3:2:2n+1, ie = 2:2:2n 26 | v[ie,jo] = (1-om)*v[ie,jo] + om*0.25(v[ie+1,jo]+v[ie-1,jo]+v[ie,jo+1]+v[ie,jo-1]) 27 | end 28 | 29 | for je = 2:2:2n, io = 3:2:2n-1 30 | v[io,je] = (1-om)*v[io,je] + om*0.25(v[io+1,je]+v[io-1,je]+v[io,je+1]+v[io,je-1]) 31 | end 32 | v[n+1,n+2] = v[n+1,n+2]-0.25om 33 | 34 | r = 2v[n+1,n+1] 35 | println("Iter = $k, r = $r") 36 | end 37 | toq() 38 | 39 | end 40 | 41 | tottime = do_sor(n, niter) 42 | println("Time/iteration = $(tottime/niter) s") 43 | -------------------------------------------------------------------------------- /GridOfResistors/grid2d-devec-slow.jl: -------------------------------------------------------------------------------- 1 | # grid2d-devec-slow.jl - Grid of Resistors problem in two dimensions 2 | # Devectorized version that is slow because array accesses are not stride 1. 3 | 4 | n = length(ARGS) >= 1 ? int(ARGS[1]) : 1000 5 | niter = length(ARGS) >= 2 ? int(ARGS[2]) : 10 6 | 7 | function do_sor(n, niter) 8 | 9 | v = zeros(2n+1,2n+2) 10 | 11 | mu = (cos(pi/(2n))+cos(pi/(2n+1)))/2 12 | om = 2(1-sqrt(1-mu^2))/mu^2 13 | 14 | tic() 15 | for k = 1:niter 16 | for ie = 2:2:2n, je = 2:2:2n 17 | v[ie,je] = (1-om)*v[ie,je] + om*0.25(v[ie+1,je]+v[ie-1,je]+v[ie,je+1]+v[ie,je-1]) 18 | end 19 | 20 | for io = 3:2:2n-1, jo = 3:2:2n+1 21 | v[io,jo] = (1-om)*v[io,jo] + om*0.25(v[io+1,jo]+v[io-1,jo]+v[io,jo+1]+v[io,jo-1]) 22 | end 23 | v[n+1,n+1] = v[n+1,n+1]+0.25om 24 | 25 | for ie = 2:2:2n, jo = 3:2:2n+1 26 | v[ie,jo] = (1-om)*v[ie,jo] + om*0.25(v[ie+1,jo]+v[ie-1,jo]+v[ie,jo+1]+v[ie,jo-1]) 27 | end 28 | 29 | for io = 3:2:2n-1, je = 2:2:2n 30 | v[io,je] = (1-om)*v[io,je] + om*0.25(v[io+1,je]+v[io-1,je]+v[io,je+1]+v[io,je-1]) 31 | end 32 | v[n+1,n+2] = v[n+1,n+2]-0.25om 33 | 34 | r = 2v[n+1,n+1] 35 | println("Iter = $k, r = $r") 36 | end 37 | toq() 38 | 39 | end 40 | 41 | tottime = do_sor(n, niter) 42 | println("Time/iteration = $(tottime/niter) s") 43 | -------------------------------------------------------------------------------- /GridOfResistors/grid2d-vec.jl: -------------------------------------------------------------------------------- 1 | # grid2d-vec.jl - Grid of Resistors problem in two dimensions - vectorized version 2 | 3 | n = length(ARGS) >= 1 ? int(ARGS[1]) : 1000 4 | niter = length(ARGS) >= 2 ? int(ARGS[2]) : 10 5 | 6 | function do_sor(n, niter) 7 | mu = (cos(pi/(2n))+cos(pi/(2n+1)))/2 8 | om = 2(1-sqrt(1-mu^2))/mu^2 9 | 10 | v = zeros(2n+1,2n+2) 11 | 12 | ie = 2:2:2n 13 | io = 3:2:2n-1 14 | je = 2:2:2n 15 | jo = 3:2:2n+1 16 | 17 | tic() 18 | for k = 1:niter 19 | v[ie,je] = (1-om)*v[ie,je] + om*0.25(v[ie+1,je]+v[ie-1,je]+v[ie,je+1]+v[ie,je-1]) 20 | v[io,jo] = (1-om)*v[io,jo] + om*0.25(v[io+1,jo]+v[io-1,jo]+v[io,jo+1]+v[io,jo-1]) 21 | v[n+1,n+1] = v[n+1,n+1]+0.25om 22 | 23 | v[ie,jo] = (1-om)*v[ie,jo] + om*0.25(v[ie+1,jo]+v[ie-1,jo]+v[ie,jo+1]+v[ie,jo-1]) 24 | v[io,je] = (1-om)*v[io,je] + om*0.25(v[io+1,je]+v[io-1,je]+v[io,je+1]+v[io,je-1]) 25 | v[n+1,n+2] = v[n+1,n+2]-0.25om 26 | 27 | r = 2v[n+1,n+1] 28 | println("Iter = $k, r = $r") 29 | end 30 | tottime = toq(); 31 | end 32 | 33 | tottime = do_sor(n, niter) 34 | println("Time/iteration = $(tottime/niter) s") 35 | -------------------------------------------------------------------------------- /GridOfResistors/grid2d.c: -------------------------------------------------------------------------------- 1 | // gcc -O2 grid2d.c -o grid2d 2 | 3 | /* grid2d.c - Grid of Resistors problem in two dimensions */ 4 | 5 | /* MIT 18.337 - Applied Parallel Computing, Spring 2004 */ 6 | /* Per-Olof Persson */ 7 | 8 | #include 9 | #include 10 | #include 11 | #include 12 | 13 | #define n 1000 14 | #define niter 10 15 | 16 | #define V(i,j) v[(i)+(2*n+1)*(j)] 17 | #define SQR(x) ((x)*(x)) 18 | 19 | int main(int argc, char **argv) 20 | { 21 | double mu,om; 22 | double *v; 23 | int it,i,j; 24 | 25 | printf("\nSize: n = %d\n\n",n); 26 | 27 | mu=0.5*(cos(M_PI/(2.0*n))+cos(M_PI/(2.0*n+1.0))); 28 | om=2.0*(1.0-sqrt(1.0-SQR(mu)))/SQR(mu); 29 | 30 | v=(double*)calloc((2*n+2)*(2*n+1),sizeof(double)); 31 | 32 | int t1=clock(); 33 | 34 | for (it=1; it<=niter; it++) { 35 | /* Update red nodes */ 36 | for (j=1; j<2*n; j+=2) { 37 | for (i=1; i<2*n-2; i+=2) { 38 | V(i,j)=(1-om)*V(i,j)+om*0.25*(V(i+1,j)+V(i-1,j)+V(i,j+1)+V(i,j-1)); 39 | V(i+1,j+1)=(1-om)*V(i+1,j+1)+om*0.25*(V(i+2,j+1)+V(i,j+1)+V(i+1,j+2)+V(i+1,j)); 40 | } 41 | i=2*n-1; 42 | V(i,j)=(1-om)*V(i,j)+om*0.25*(V(i+1,j)+V(i-1,j)+V(i,j+1)+V(i,j-1)); 43 | } 44 | /* RHS */ 45 | V(n,n)+=om*0.25; 46 | 47 | /* Update black nodes */ 48 | for (j=1; j<2*n; j+=2) { 49 | for (i=1; i<2*n-2; i+=2) { 50 | V(i,j+1)=(1-om)*V(i,j+1)+om*0.25*(V(i+1,j+1)+V(i-1,j+1)+V(i,j+2)+V(i,j)); 51 | V(i+1,j)=(1-om)*V(i+1,j)+om*0.25*(V(i+2,j)+V(i,j)+V(i+1,j+1)+V(i+1,j-1)); 52 | } 53 | i=2*n-1; 54 | V(i,j+1)=(1-om)*V(i,j+1)+om*0.25*(V(i+1,j+1)+V(i-1,j+1)+V(i,j+2)+V(i,j)); 55 | } 56 | /* RHS */ 57 | V(n,n+1)-=om*0.25; 58 | 59 | if (1) /* Change to 1 for printing (slower) */ 60 | printf("Iter = %4d, r = %.16f\n",it,2*V(n,n)); 61 | } 62 | 63 | int t2 = clock(); 64 | 65 | printf("time per iter = %.5f r = %.5f\n\n", ((t2-t1)/(double)CLOCKS_PER_SEC)/niter, 2*V(n,n)); 66 | 67 | free(v); 68 | return 0; 69 | } 70 | -------------------------------------------------------------------------------- /GridOfResistors/grid2d.m: -------------------------------------------------------------------------------- 1 | % sor2d.m - Grid of Resistors problem in two dimensions 2 | 3 | % MIT 18.337 - Applied Parallel Computing, Spring 2004 4 | % Per-Olof Persson 5 | 6 | n=1000; 7 | niter=10; 8 | 9 | mu = (cos(pi/(2*n))+cos(pi/(2*n+1)))/2; 10 | om = 2*(1-sqrt(1-mu^2))/mu^2; 11 | 12 | v=zeros(2*n+1,2*n+2); 13 | 14 | ie=2:2:2*n; 15 | io=3:2:2*n-1; 16 | je=2:2:2*n; 17 | jo=3:2:2*n+1; 18 | 19 | tic 20 | for k=1:niter 21 | v(ie,je)=(1-om)*v(ie,je)+ ... 22 | om*0.25*(v(ie+1,je)+v(ie-1,je)+v(ie,je+1)+v(ie,je-1)); 23 | v(io,jo)=(1-om)*v(io,jo)+ ... 24 | om*0.25*(v(io+1,jo)+v(io-1,jo)+v(io,jo+1)+v(io,jo-1)); 25 | v(n+1,n+1)=v(n+1,n+1)+om*0.25; 26 | 27 | v(ie,jo)=(1-om)*v(ie,jo)+ ... 28 | om*0.25*(v(ie+1,jo)+v(ie-1,jo)+v(ie,jo+1)+v(ie,jo-1)); 29 | v(io,je)=(1-om)*v(io,je)+ ... 30 | om*0.25*(v(io+1,je)+v(io-1,je)+v(io,je+1)+v(io,je-1)); 31 | v(n+1,n+2)=v(n+1,n+2)-om*0.25; 32 | 33 | r=2*v(n+1,n+1); 34 | fprintf('Iter = %4d, r = %.16f\n',k,r); 35 | end 36 | tottime=toc; 37 | fprintf('Time/iteration = %.5f s',tottime/niter); 38 | -------------------------------------------------------------------------------- /LightningRound/IAP_2013_Lightning.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JuliaAttic/julia-tutorial/4acd63163586a9016a31b3fad2ba28a32c14c84b/LightningRound/IAP_2013_Lightning.pdf -------------------------------------------------------------------------------- /LightningRound/IAP_2013_Lightning.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JuliaAttic/julia-tutorial/4acd63163586a9016a31b3fad2ba28a32c14c84b/LightningRound/IAP_2013_Lightning.pptx -------------------------------------------------------------------------------- /LightningRound/dice: -------------------------------------------------------------------------------- 1 | #!/Users/stefan/projects/julia/julia 2 | 3 | n = int(ARGS[1]) 4 | println(randn(n,n)) 5 | count = 0 6 | for i = 1:n 7 | a = randi(6,3) 8 | count += (3==length(unique(a))) 9 | end 10 | println(count/n) 11 | -------------------------------------------------------------------------------- /LightningRound/lightning.jl: -------------------------------------------------------------------------------- 1 | # This file contains all the examples from the presentation 2 | 3 | # This file is not executable, and contains examples that will produce 4 | # errors. It is provided for convenience for students to copy and paste 5 | # code on the julia prompt. 6 | 7 | A = rand(5,5) 8 | A[1,1] 9 | rand(5,5)[1,1] 10 | 11 | [ i for i=1:5 ] 12 | [ trace(rand(n,n)) for n=1:5 ] 13 | x = rand(10) 14 | [ x[i]+x[i+1] for i=1:9 ] 15 | { eye(n) for n=1:5 } 16 | [ i+j for i=1:5, j=1:5 ] 17 | 18 | A = rand(5,6) 19 | svd(A) 20 | (u,s,v) = svd(A) 21 | ndims(u), typeof(u) 22 | ndims(s), typeof(s) 23 | 24 | sgn(x) = x > 0 ? 1 : -1 25 | sgn(x) = x > 0 ? 1 : x < 0 ? -1 : 0 26 | 27 | im 28 | typeof(2im) 29 | typeof(2.0im) 30 | complex(3,4) 31 | complex(3,4.0) 32 | sqrt(-1) 33 | sqrt(complex(-1)) 34 | 35 | A = rand(5,5) 36 | v = rand(5) 37 | typeof(v) 38 | typeof(1.0:5) 39 | w = 1:5 40 | A*w 41 | A*[w] 42 | ones(5) 43 | eye(5) 44 | 45 | run(`cal`) 46 | run(`cal` | `grep Sa`) 47 | 48 | ccall(:clock, Int32, ()) 49 | bytestring(ccall(:ctime, Ptr{Uint8}, ())) 50 | 51 | Pkg.add("Calendar") 52 | using Calendar 53 | Calendar.now() 54 | now() 55 | 56 | n = now() 57 | typeof(n) 58 | n.tz 59 | n.millis 60 | z = convert(Array, @parallel [ Calendar.now().millis for x=1:10 ]) 61 | z - mean(z) 62 | 63 | strang(n) = SymTridiagonal(2*ones(n),-ones(n-1)) 64 | lit = strang(500) 65 | big = full(lit) 66 | @time eigvals(lit) 67 | @time eigvals(big) 68 | big + big 69 | lit + lit 70 | import Base.+ 71 | xdump(lit) 72 | +(a::SymTridiagonal,b::SymTridiagonal) = SymTridiagonal(a.dv+b.dv,a.ev+b.ev) 73 | lit + lit 74 | 75 | function stepbystep() 76 | for n=1:3 77 | produce(n^2) 78 | end 79 | end 80 | p = Task(stepbystep) 81 | consume(p) 82 | consume(p) 83 | consume(p) 84 | consume(p) 85 | 86 | sqrt(-1) 87 | anothersqrt(x) = x < 0 ? sqrt(complex(x)) : sqrt(x) 88 | [ anothersqrt(x) for x=-2:3 ] 89 | -------------------------------------------------------------------------------- /NumericalOptimization/ExampleSolution.jl: -------------------------------------------------------------------------------- 1 | require("Optim") 2 | using Winston 3 | 4 | @everywhere busytime = Array((Float64,Float64),0) 5 | 6 | # f_i(x) = 0.5(x-i)^2 7 | # returns subderivative in the form 8 | # (a,b) -> ax' + b = f'(x)*x' + (f(x)-x*f'(x)) (from f(x) + f'(x)(x'-x)) 9 | @everywhere function f(input) 10 | x,i = input 11 | t = time() 12 | #sleep(0.56) 13 | sleep(max(0.1,0.2+1*randn())) # simulate computation time 14 | push!(busytime,(t,time())) 15 | return ((x-i),0.5(x-i)^2-x*(x-i)) 16 | end 17 | 18 | function evalmodel(x, subderivatives::Vector{Vector{(Float64,Float64)}}) 19 | N = length(subderivatives) 20 | if 0 <= x <= N 21 | modelval = 0. 22 | for i in 1:length(subderivatives) # for each f_i 23 | maxval = -Inf 24 | for (a,b) in subderivatives[i] # for each stored subderivative of f_i 25 | maxval = max(a*x+b,maxval) 26 | end 27 | modelval += maxval 28 | end 29 | return modelval 30 | elseif x < 0 # repel away from here 31 | return -100000x+100000 32 | else 33 | return 100000x 34 | end 35 | end 36 | 37 | function staticmapversion(N::Integer) 38 | cur = [1/N] 39 | subderivatives = Array(Vector{(Float64,Float64)},N) 40 | for i in 1:N 41 | subderivatives[i] = Array((Float64,Float64),0) 42 | end 43 | niter = 0 44 | while abs(cur[1]-(N+1)/2) > 1e-4 45 | for (i,result) in enumerate(Base.pmap_static(f,[(cur[1],i) for i in 1:N])) 46 | push!(subderivatives[i],fetch(result)) 47 | end 48 | modelresults = Optim.optimize(x->evalmodel(x[1],subderivatives),cur) 49 | #println("Model obj: ", modelresults.f_minimum) 50 | println("Model x: ", modelresults.minimum) 51 | cur = modelresults.minimum 52 | niter += 1 53 | 54 | 55 | end 56 | println("Converged in $niter iterations") 57 | 58 | end 59 | 60 | function pmapversion(N::Integer) 61 | cur = [1/N] 62 | subderivatives = Array(Vector{(Float64,Float64)},N) 63 | for i in 1:N 64 | subderivatives[i] = Array((Float64,Float64),0) 65 | end 66 | niter = 0 67 | mastertime = 0. 68 | while abs(cur[1]-(N+1)/2) > 1e-4 69 | results = pmap(f,[(cur[1],i) for i in 1:N]) 70 | for i in 1:N 71 | push!(subderivatives[i],results[i]) 72 | end 73 | t = time() 74 | modelresults = Optim.optimize(x->evalmodel(x[1],subderivatives),cur) 75 | mastertime += time() - t 76 | #println("Model obj: ", modelresults.f_minimum) 77 | println("Model x: ", modelresults.minimum) 78 | cur = modelresults.minimum 79 | niter += 1 80 | points = linspace(0,N,200) 81 | modelval = map(x->evalmodel(x,subderivatives),points) 82 | fval = map(x->sum([0.5(x-i)^2 for i in 1:N]),points) 83 | p = FramedPlot() 84 | add(p, Curve(points,modelval)) 85 | add(p, Curve(points,fval,"type","dash")) 86 | file(p, "iter$niter.pdf") 87 | end 88 | println("Converged in $niter iterations, mastertime: $mastertime") 89 | 90 | end 91 | 92 | function asyncversion(N::Integer,asyncparam::Float64) 93 | np = nprocs() 94 | cur = [1/N] 95 | subderivatives = Array(Vector{(Float64,Float64)},N) 96 | for i in 1:N 97 | subderivatives[i] = Array((Float64,Float64),0) 98 | end 99 | converged = false 100 | set_converged() = (converged = true) 101 | is_converged() = (converged) 102 | mastertime = 0. 103 | nmaster = 0 104 | increment_mastertime(t) = (mastertime += t; nmaster += 1) 105 | nback = Int[] 106 | didtrigger = Bool[] 107 | nsubproblems = 0 108 | increment_solved() = (nsubproblems += 1) 109 | tasks = Array((Float64,Int,Int),0) 110 | 111 | function pushtask(x) 112 | for i in 1:N 113 | push!(tasks,(x[1],i,nmaster+1)) 114 | end 115 | push!(didtrigger, false) 116 | push!(nback, 0) 117 | end 118 | 119 | pushtask(cur) 120 | 121 | @sync for p in 1:np 122 | if p != myid() || np == 1 123 | @spawnlocal while !is_converged() 124 | if length(tasks) == 0 125 | yield() 126 | continue 127 | end 128 | mytask = shift!(tasks) 129 | result = remote_call_fetch(p,f,mytask[1:2]) 130 | push!(subderivative[mytask[2]],result) 131 | @assert length(nback) >= mytask[3] 132 | @assert length(didtrigger) >= mytask[3] 133 | nback[mytask[3]] += 1 134 | resolve = false 135 | if nback[mytask[3]] >= asyncparam*N && didtrigger[mytask[3]] == false 136 | didtrigger[mytask[3]] = true 137 | resolve = true 138 | #elseif nback[mytask[3]] == N 139 | # resolve = true 140 | end 141 | if resolve 142 | # generate new candidate point 143 | t = time() 144 | modelresults = Optim.optimize(x->evalmodel(x[1],subderivatives),cur) 145 | increment_mastertime(time() - t) 146 | #println("Model obj: ", modelresults.f_minimum) 147 | println("Model x: ", modelresults.minimum) 148 | cur[1] = modelresults.minimum[1] 149 | if abs(cur[1]-(N+1)/2) <= 1e-4 150 | println("Converged!") 151 | set_converged() 152 | else 153 | pushtask(cur) 154 | end 155 | end 156 | end 157 | end 158 | end 159 | println("Converged in $nmaster master solves, $(sum(nback)) subproblem evaluations of f, mastertime: $mastertime") 160 | end 161 | 162 | 163 | 164 | 165 | time0 = time() 166 | #@time staticmapversion(100) 167 | @time pmapversion(100) 168 | #@time asyncversion(100,0.9) 169 | 170 | # gather busy intervals from each process and plot 171 | if nprocs() > 1 172 | busytimes = [ remote_call_fetch(p,()->busytime) for p in 1:nprocs() ] 173 | p = FramedPlot() 174 | for i in 1:nprocs() 175 | for (starttime,endtime) in busytimes[i] 176 | add(p, Curve([starttime-time0,endtime-time0],[i,i])) 177 | end 178 | end 179 | 180 | setattr(p.frame, "draw_axis", false) 181 | setattr(p.y1, "ticks", [float(i) for i in 2:nprocs()]) 182 | setattr(p.x1, "ticks", [0.]) 183 | setattr(p.x1, "ticklabels", [""]) 184 | setattr(p.x1, "range", (0.,45.)) # adjust as necessary 185 | file(p, "activityplot.pdf") 186 | end 187 | 188 | 189 | -------------------------------------------------------------------------------- /NumericalOptimization/Tutorial.jl: -------------------------------------------------------------------------------- 1 | require("Optim") 2 | 3 | # f_i(x) = 0.5(x-i)^2 4 | # returns subderivative in the form 5 | # (a,b) -> ax' + b = f'(x)*x' + (f(x)-x*f'(x)) (from f(x) + f'(x)(x'-x)) 6 | function f(input) 7 | x,i = input 8 | sleep(max(0.1,0.2+1*randn())) # simulate computation time 9 | return ((x-i),0.5(x-i)^2-x*(x-i)) 10 | end 11 | 12 | function evalmodel(x, subderivatives::Vector{Vector{(Float64,Float64)}}) 13 | N = length(subderivatives) 14 | if 0 <= x <= N 15 | modelval = 0. 16 | for i in 1:length(subderivatives) # for each f_i 17 | maxval = -Inf 18 | for (a,b) in subderivatives[i] # for each stored subderivative of f_i 19 | maxval = max(a*x+b,maxval) 20 | end 21 | modelval += maxval 22 | end 23 | return modelval 24 | elseif x < 0 # repel away from here 25 | return -100000x+100000 26 | else 27 | return 100000x 28 | end 29 | end 30 | 31 | function cpserial(N::Integer) 32 | cur = [1/2] 33 | println("Solving model with n = $N, initial solution: $cur") 34 | println("Optimal solutuon should be $((N+1)/2)") 35 | subderivatives = Array(Vector{(Float64,Float64)},N) 36 | for i in 1:N 37 | subderivatives[i] = Array((Float64,Float64),0) 38 | end 39 | niter = 0 40 | while abs(cur[1]-(N+1)/2) > 1e-4 # we're cheating because we know the answer 41 | results = map(f,[(cur[1],i) for i in 1:N]) 42 | for i in 1:N 43 | push!(subderivatives[i],results[i]) 44 | end 45 | modelresults = Optim.optimize(x->evalmodel(x[1],subderivatives),cur) 46 | println("Model minimizer: ", modelresults.minimum) 47 | cur = modelresults.minimum 48 | niter += 1 49 | end 50 | println("Converged in $niter iterations") 51 | 52 | end 53 | 54 | function asyncversion(N::Integer,asyncparam::Float64) 55 | np = nprocs() 56 | cur = [1/2] 57 | subderivatives = Array(Vector{(Float64,Float64)},N) 58 | for i in 1:N 59 | subderivatives[i] = Array((Float64,Float64),0) 60 | end 61 | converged = false 62 | set_converged() = (converged = true) 63 | is_converged() = (converged) 64 | nmaster = 0 65 | nback = Int[] 66 | didtrigger = Bool[] 67 | nsubproblems = 0 68 | increment_solved() = (nsubproblems += 1) 69 | tasks = Array((Float64,Int,Int),0) 70 | 71 | function pushtask(x) 72 | for i in 1:N 73 | push!(tasks,(x[1],i,nmaster+1)) 74 | end 75 | push!(didtrigger, false) 76 | push!(nback, 0) 77 | end 78 | 79 | pushtask(cur) 80 | 81 | @sync for p in 1:np 82 | if p != myid() || np == 1 83 | @spawnlocal while !is_converged() 84 | if length(tasks) == 0 85 | yield() 86 | continue 87 | end 88 | mytask = shift!(tasks) 89 | result = remote_call_fetch(p,f,mytask[1:2]) 90 | push!(subderivatives[mytask[2]],result) 91 | @assert length(nback) >= mytask[3] 92 | @assert length(didtrigger) >= mytask[3] 93 | nback[mytask[3]] += 1 94 | resolve = false 95 | if nback[mytask[3]] >= asyncparam*N && didtrigger[mytask[3]] == false 96 | didtrigger[mytask[3]] = true 97 | resolve = true 98 | #elseif nback[mytask[3]] == N 99 | # resolve = true 100 | end 101 | if resolve 102 | # generate new candidate point 103 | modelresults = Optim.optimize(x->evalmodel(x[1],subderivatives),cur) 104 | #println("Model obj: ", modelresults.f_minimum) 105 | println("Model minimizer: ", modelresults.minimum) 106 | cur[1] = modelresults.minimum[1] 107 | if abs(cur[1]-(N+1)/2) <= 1e-4 108 | println("Converged!") 109 | set_converged() 110 | else 111 | pushtask(cur) 112 | end 113 | end 114 | end 115 | end 116 | end 117 | println("Converged in $nmaster master solves, $(sum(nback)) subproblem evaluations of f") 118 | end 119 | 120 | 121 | @time cpserial(10) 122 | -------------------------------------------------------------------------------- /NumericalOptimization/activityplotdeterm.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JuliaAttic/julia-tutorial/4acd63163586a9016a31b3fad2ba28a32c14c84b/NumericalOptimization/activityplotdeterm.pdf -------------------------------------------------------------------------------- /NumericalOptimization/activityplotnorm.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JuliaAttic/julia-tutorial/4acd63163586a9016a31b3fad2ba28a32c14c84b/NumericalOptimization/activityplotnorm.pdf -------------------------------------------------------------------------------- /NumericalOptimization/presentation.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JuliaAttic/julia-tutorial/4acd63163586a9016a31b3fad2ba28a32c14c84b/NumericalOptimization/presentation.pdf -------------------------------------------------------------------------------- /NumericalOptimization/presentation.tex: -------------------------------------------------------------------------------- 1 | \documentclass{beamer} 2 | 3 | \usepackage{url} 4 | \usepackage{tikz} 5 | \usetheme{Madrid} 6 | 7 | \title{Julia for Linear Programming} 8 | \author{Iain Dunning and Miles Lubin} 9 | \date{IAP -- January 16, 2013} 10 | \institute{MIT} 11 | 12 | \setbeamercovered{invisible} 13 | \setbeamertemplate{navigation symbols}{} 14 | \setbeamertemplate{footline}[page number] 15 | 16 | \begin{document} 17 | 18 | \frame{\titlepage} 19 | 20 | \begin{frame} 21 | \frametitle{What is Linear Programming (LP)} 22 | \begin{align*} 23 | \text{Maximize} \quad & c^T x \quad \text{over } x \in \mathbb{R}^N \\ 24 | \text{Subject to} \quad & a^T_i x \leq b_i \quad \forall i=1,\ldots,M 25 | \end{align*} 26 | \begin{itemize} 27 | \item Widely used - airline scheduling, production planning, TeX hyphenation... 28 | \item Simplex algorithm for LP named in "Top 10 Algo. of 20th Century" by SIAM 29 | \end{itemize} 30 | \end{frame} 31 | 32 | \begin{frame}[fragile] 33 | \frametitle{How are they solved - Modeling} 34 | \begin{itemize} 35 | \item Why use modeling languages? 36 | \end{itemize} 37 | \begin{verbatim} 38 | maximize Obj: 39 | sum {j in 1..N} profit[j] * x[j]; 40 | 41 | subject to CapacityCons: 42 | sum {j in 1..N} weight[j] * x[j] <= Capacity; 43 | \end{verbatim} 44 | \begin{itemize} 45 | \item Options 46 | \begin{itemize} 47 | \item Commercial - e.g. AMPL, GAMS 48 | \begin{itemize} 49 | \item Specialized - so fast 50 | \item Not general purpose language 51 | \end{itemize} 52 | \item Open-source - e.g. PuLP, Pyomo, CVX, YALMIP 53 | \begin{itemize} 54 | \item Built on Python or MATLAB 55 | \item Use operator overloading - slower 56 | \end{itemize} 57 | \end{itemize} 58 | \end{itemize} 59 | \end{frame} 60 | 61 | \begin{frame}[fragile] 62 | \frametitle{Modeling in Julia} 63 | \begin{center} 64 | \url{http://github.com/IainNZ/Julp} 65 | \end{center} 66 | \begin{itemize} 67 | \item Julia replaces domain-specific language 68 | \item Use macros to avoid issues with operator overloading 69 | \end{itemize} 70 | {\small 71 | \begin{verbatim} 72 | m = Model("max") 73 | x = [ Variable(m) for j = 1:N ] 74 | profit = rand(N); weight = rand(N); 75 | 76 | setObjective(m, 77 | @sumExpr([ profit[j] * x[j] for j = 1:N ]) 78 | ) 79 | 80 | addConstraint(m, 81 | @sumExpr([ weight[j] * x[j] for j = 1:N ]) 82 | ) 83 | \end{verbatim} 84 | } 85 | \end{frame} 86 | 87 | \begin{frame}[fragile] 88 | \frametitle{Macro and Benchmark} 89 | \begin{itemize} 90 | \item Macro only 3 (long-ish) lines 91 | \item Breaks \texttt{[c[i] * x[i] for i = 1:N]} into... 92 | \begin{itemize} 93 | \item \texttt{[c[i] for i = 1:N]} 94 | \item \texttt{[x[i] for i = 1:N]} 95 | \end{itemize} 96 | \item ... which is how constraints are stored 97 | \item Benchmark times (in seconds): 98 | \end{itemize} 99 | \begin{table} 100 | \begin{tabular}{|c|c|c|} 101 | \hline 102 | Lang. & N=5000 & N=10000 \\ 103 | \hline 104 | AMPL & 4 & 6 \\ 105 | Julia (Julp) & 6.44 & 16.29 \\ 106 | PyPy2 (PuLP) & 26.62 & 53.45 \\ 107 | Python (PuLP) & 111.80 & 222.95 \\ 108 | \hline 109 | \end{tabular} 110 | \end{table} 111 | \end{frame} 112 | 113 | \begin{frame} 114 | \frametitle{How are they solved -- Algorithms} 115 | \begin{itemize} 116 | \item Dantzig's simplex algorithm most used method. 117 | \item Computationally very challenging to implement efficiently 118 | \begin{itemize} 119 | \item Naturally not vectorizable -- specialized sparse linear algebra 120 | \item Typically memory bound -- cache misses 121 | \end{itemize} 122 | \item Matlab implementations too slow to be used in practice 123 | \begin{itemize} 124 | \item High-quality open-source codes exist in C++ 125 | \end{itemize} 126 | \item Can Julia compete? 127 | \end{itemize} 128 | \end{frame} 129 | 130 | \begin{frame} 131 | \frametitle{Simplex Benchmarks} 132 | \begin{center} 133 | \url{https://github.com/mlubin/SimplexBenchmarks} 134 | \end{center} 135 | \begin{itemize} 136 | \item Benchmark of some important operations: 137 | \end{itemize} 138 | \begin{table}\begin{center} 139 | \begin{tabular}{|c|c|c|c|c|c|c|} 140 | \hline 141 | & Julia & C++ & C++bnd &Matlab & PyPy & Python \\ 142 | \hline 143 | Sp.\ mat-sp.\ vec& 1.29 & 0.90 & 1.00 & 5.79 & 19.20 & 417.16\\ 144 | Sp.\ vector scan & 1.59 & 0.96 & 1.00 & 13.98 & 13.81 & 48.39\\ 145 | Sp.\ \texttt{axpy}& 1.85 & 0.70 & 1.00 & 19.12 & 9.21 & 78.65\\ 146 | \hline 147 | \end{tabular}\end{center} 148 | \end{table} 149 | \begin{itemize} 150 | \item C++bnd = C++ with bounds checking 151 | \item Execution times relative to C++bnd 152 | \end{itemize} 153 | \end{frame} 154 | 155 | \begin{frame} 156 | \frametitle{Tutorial} 157 | \begin{center} 158 | \url{https://github.com/JuliaLang/IAP2013/blob/master/NumericalOptimization/tutorial.pdf} 159 | 160 | \begin{tikzpicture}[xscale=3,yscale=6] 161 | \draw [gray,ultra thick,dashed,domain=0:2] plot (\x, {0.5*(\x-1)^2}); 162 | %\draw [dashed,domain=0:1] plot (\x, {-1*\x + 0.5}); 163 | %\draw [dashed,domain=0:2] plot (\x, {-\x/2 + 3/8}); 164 | %\draw [dashed,domain=1:2] plot (\x, {(3/4)*\x-1.03125}); 165 | \draw [thick,domain=0:2,samples=200] plot (\x, {max(-1*\x + 0.5,max(-\x/2 + 3/8,(3/4)*\x-1.03125))}); 166 | \node () at (1,0) {\textbullet}; 167 | \node [font=\large] () at (1,0.05) {$f_\text{min}$}; 168 | \node () at (1.125,-0.1875) {\textbullet}; 169 | \node [font=\large] () at (1.25,-0.1875) {$m_\text{min}^1$}; 170 | \draw [dashed,domain=0:2] plot (\x, {0.125*\x - 0.132813}); 171 | \node () at (0.812501,-0.0312504) {\textbullet}; 172 | \node [font=\large] () at (0.8,-0.07) {$m_\text{min}^2$}; 173 | \draw (current bounding box.north east) rectangle (current bounding box.south west); 174 | \end{tikzpicture}\end{center} 175 | \begin{itemize} 176 | \item Tutorial for implementing a parallel asynchronous optimization algorithm in Julia using master-worker paradigm 177 | \item No background in optimization needed 178 | \end{itemize} 179 | 180 | \end{frame} 181 | 182 | \end{document} 183 | -------------------------------------------------------------------------------- /NumericalOptimization/refs.bib: -------------------------------------------------------------------------------- 1 | @BOOK{HiriartLemarechal93book2, 2 | author = {Hiriart-Urruty, J. B. and Lemar{\'e}chal, C.}, 3 | title = {Convex Analysis and Minimization Algorithms}, 4 | publisher = {Springer-Verlag}, 5 | year = {1993}, 6 | address = {Germany}, 7 | edition = {}, 8 | volume = {I-II} 9 | } 10 | 11 | @ARTICLE{Linderoth03, 12 | author = {Linderoth, Jeff and Wright, Stephen}, 13 | title = {Decomposition algorithms for stochastic programming on a computational grid}, 14 | journal = {Computational Optimization and Applications}, 15 | year = {2003}, 16 | volume = {24}, 17 | number = {2}, 18 | pages = {207-250} 19 | } -------------------------------------------------------------------------------- /NumericalOptimization/tutorial.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JuliaAttic/julia-tutorial/4acd63163586a9016a31b3fad2ba28a32c14c84b/NumericalOptimization/tutorial.pdf -------------------------------------------------------------------------------- /NumericalOptimization/tutorial.tex: -------------------------------------------------------------------------------- 1 | \documentclass[12pt]{article} 2 | \usepackage{tikz} 3 | \usepackage{amsmath,amsthm,amssymb} 4 | \usepackage{algorithmic,algorithm} 5 | \usepackage{hyperref} 6 | \hypersetup{ 7 | colorlinks=true 8 | } 9 | \usepackage{fancyvrb} 10 | \renewcommand{\algorithmicrequire}{\textbf{Input:}} 11 | \theoremstyle{definition} 12 | \newtheorem*{defn}{Definition} 13 | \newtheorem*{advex}{Advanced Exercise} 14 | \newtheorem*{optex}{Optional Exercise} 15 | \newtheorem{ex}{Exercise} 16 | 17 | \title{The cutting-plane algorithm: synchronous and asynchronous master-worker parallelism in Julia} 18 | \author{Miles Lubin\\MIT Operations Research Center} 19 | \date{Julia IAP Tutorial, January 15-16, 2013} 20 | \begin{document} 21 | \maketitle 22 | 23 | \section{The cutting plane algorithm} 24 | This section is meant to be a self-contained presentation of the cutting-plane algorithm for convex optimization. It assumes only a background in calculus. We present the one-dimensional case for simplicity. Most of this section can be skipped or skimmed; the fun starts in the next section. The essential parts of this section are the statement of Algorithm \ref{algserial} and the paragraph that follows it.\nocite{HiriartLemarechal93book2} 25 | 26 | \begin{defn} 27 | A function $f : \mathbb{R} \to \mathbb{R}$ is \textit{subdifferentiable} if for all $x' \in \mathbb{R}$, there exists a scalar $g_{x'}$ such that for all $x \in \mathbb{R}$, 28 | \[ 29 | f(x) \ge g_{x'}(x-x') + f(x') = g_{x'}x + (f(x')-g_{x'}x'). 30 | \] 31 | Any such $g_{x'}$ is called a \textit{subderivative} of $f$ at $x'$. 32 | \end{defn} 33 | The property of subdifferentiability is nearly equivalent to \textit{convexity}. The above definition means that at every point in the domain of $f$, we can draw a line tangent to the graph of $f$ which will remain under the graph at all points. If $f$ is also differentiable, then $g_{x'}$ is exactly the derivative $f'(x)$, but the definition holds for functions which may not be differentiable everywhere. An example is the absolute value function $f(x) = |x|$. For fun, you may want to find the set of valid $g_{x'}$ for $f(x) = |x|$ at $x' = 0$. 34 | 35 | We move on immediately to the field of convex optimization. Let $f_1,f_2,\ldots,f_n$ be given subdifferentiable functions. The problem we are interested in solving is 36 | \begin{equation} 37 | \operatorname{minimize}_{x \in \mathbb{R}} \sum_{i=1}^n f_i(x), 38 | \end{equation} 39 | that is, find a value of $x \in \mathbb{R}$ that minimizes the function $f(x) := \sum_{i=1}^n f_i(x)$. 40 | 41 | Suppose we have a black box procedure for each $f_i$ that, given any point $x$, returns a tuple $(f_i(x),g_{i,x})$; that is, it computes both the value of $f$ at $x$ and a corresponding subderivative. Now suppose that we've called this procedure at a sequence of points $x^1,x^2,\ldots,x^k$ for each $i$. Using the set of subderivatives, we build a \textit{model} of each $f_i$, which we define as 42 | \begin{equation}\label{eq:mod} 43 | m^k_i(x) := \max\{g_{i,x^1}(x-x^1) + f_i(x^1),g_{i,x^2}(x-x^2) + f_i(x^2),\ldots,g_{i,x^k}(x-x^k) + f_i(x^k)\}. 44 | \end{equation} 45 | This model is piecewise linear and is a lower estimate of $f_i$. The idea of the \textit{cutting-plane} algorithm is to use the minimum of the model, something which is easy to compute, as a guess for the minimum of $f$. In particular, the next trial point $x^{k+1}$ is set to the minimizer of $\sum_{i=1}^n m_i^k(x)$, the black boxes are then called again, and we repeat until we're satisfied with our solution. An iteration of the algorithm is illustrated in Figure~\ref{fig:cut}, and the algorithm is formally stated in Algorithm \ref{algserial}. 46 | 47 | \begin{figure}[ht] 48 | \centering 49 | \begin{tikzpicture}[xscale=5,yscale=10] 50 | \draw [gray,ultra thick,dashed,domain=0:2] plot (\x, {0.5*(\x-1)^2}); 51 | %\draw [dashed,domain=0:1] plot (\x, {-1*\x + 0.5}); 52 | %\draw [dashed,domain=0:2] plot (\x, {-\x/2 + 3/8}); 53 | %\draw [dashed,domain=1:2] plot (\x, {(3/4)*\x-1.03125}); 54 | \draw [thick,domain=0:2,samples=200] plot (\x, {max(-1*\x + 0.5,max(-\x/2 + 3/8,(3/4)*\x-1.03125))}); 55 | \node () at (1,0) {\textbullet}; 56 | \node [font=\large] () at (1,0.05) {$f_\text{min}$}; 57 | \node () at (1.125,-0.1875) {\textbullet}; 58 | \node [font=\large] () at (1.25,-0.1875) {$m_\text{min}^1$}; 59 | \draw [dashed,domain=0:2] plot (\x, {0.125*\x - 0.132813}); 60 | \node () at (0.812501,-0.0312504) {\textbullet}; 61 | \node [font=\large] () at (0.8,-0.07) {$m_\text{min}^2$}; 62 | \draw (current bounding box.north east) rectangle (current bounding box.south west); 63 | \end{tikzpicture} 64 | \caption{An iteration of the cutting-plane algorithm applied to $f = x^2$ ($n = 1$). The point $m_\text{min}^1$ is the minimizer of the current model. We then re-evaluate $f$ and add a \textit{cut} (dashed line) to the model corresponding to the new (sub)derivative. The point $m_\text{min}^2$ minimizes the new model. The point $f_\text{min}$ is the true minimizer.}\label{fig:cut} 65 | \end{figure} 66 | \begin{algorithm} 67 | \caption{Cuting plane algorithm -- Serial}\label{algserial} 68 | \begin{algorithmic}[1] 69 | \REQUIRE Subdifferentiable functions $f_1,\ldots,f_n$. Starting point $x^1$. 70 | \STATE $k \leftarrow 1$ 71 | \REPEAT 72 | \FOR{$i=1$ \TO $n$} 73 | \STATE Compute function value and subderivative $(f_i(x^k),g_{i,x^k})$ at $x^k$. 74 | \ENDFOR 75 | \STATE $x^{k+1} \leftarrow \operatorname{argmin}_x \sum_{i=1}^n m^k_i(x)$, with $m^k_i(x)$ defined by \eqref{eq:mod} 76 | \STATE $k \leftarrow k + 1$ 77 | \UNTIL{convergence} 78 | \end{algorithmic} 79 | \end{algorithm} 80 | 81 | An important observation is that the computation of the function values and subderivatives may be performed independently for each $i$ in the \textbf{for} loop. In many applications this computation is expensive, hence there is a real potential for speedup if we parallelize this loop. We will now investigate implementing and parallelizing this algorithm in Julia. 82 | 83 | \section{Julia implementation} 84 | In the previous section we presented the mathematical algorithm. Here we'll work with an actual implementation on a small toy problem and experiment with parallel computing in Julia. First, retrieve the code from \href{https://github.com/JuliaLang/IAP2013/tree/master/NumericalOptimization}{here}. 85 | The \texttt{Tutorial.jl} file contains the code we will be referring to, while \texttt{ExampleSolution.jl} contains potential solutions to some of the exercises. 86 | 87 | Our toy problem will be based on the functions $f_i(x) = \frac{1}{2}(x-i)^2$. With $f(x) = \sum_{i=1}^n f_i(x)$ as before, we have $f'(x) = \sum_{i=1}^n (x-i)$. Using basic calculus, we know already that the minimizer of $f$ is at $x = \frac{1}{n}\sum_{i=1}^n i = (n+1)/2$. Before running, install the Optim package, \texttt{Pkg.add("Optim")}. We can now run the the code by \texttt{julia Tutorial.jl}. The output should appear as follows: 88 | \begin{center} 89 | \begin{SaveVerbatim}{verb2} 90 | Solving model with n = 10, initial solution: [0.5] 91 | Optimal solutuon should be 5.5 92 | Model minimizer: [10.0] 93 | Model minimizer: [5.25] 94 | Model minimizer: [7.625] 95 | ... 96 | Model minimizer: [5.49995] 97 | Converged in 15 iterations 98 | elapsed time: 78.94296097755432 seconds 99 | \end{SaveVerbatim} 100 | \fbox{\BUseVerbatim{verb2}} 101 | \end{center} 102 | 103 | Let's open up the code. The function \texttt{f} implements the black-box calculation of the subderivatives. Notice the line with \texttt{sleep}, which we use to simulate a variably expensive black-box. The \texttt{evalmodel} function evaluates $\sum_{i=1}^n m_i^k(x)$ at \texttt{x} given the stored \texttt{subderivatives}. 104 | 105 | The \texttt{cpserial} function implements Algorithm~\ref{algserial} in serial. The parameter $N$ is our cursive $n$. The line \texttt{results = map(f,[(cur[1],i) for i in 1:N])} corresponds to loop of lines 3--5 of the algorithm. The call to \texttt{Optim.optimize} corresponds to line 6. Now for the fun stuff. 106 | 107 | \begin{ex} 108 | Insert code to plot $\sum_{i=1}^n m_i^k(x)$ versus $\sum_{i=1}^n f_i(x)$ at each iteration. You may use \href{https://github.com/nolta/Winston.jl}{Winston} or any other package. 109 | \end{ex} 110 | 111 | \begin{ex} 112 | Review the Julia documentation on \href{http://docs.julialang.org/en/latest/manual/parallel-computing/}{parallel computing}. Modify the code so that the subderivatives are computed in parallel. \textit{(Hint: it only requires a small change.)} 113 | \end{ex} 114 | 115 | Now try running \texttt{julia -p 2 Tutorial.jl}. If you get an error like 116 | \begin{center} 117 | \begin{BVerbatim} 118 | From worker 2: exception on 2: f not defined 119 | \end{BVerbatim} 120 | \end{center} 121 | this means that \texttt{f} was only defined on process 1, but you're trying to call it on process 2. Check the documentation or \texttt{ExampleSolution.jl} for a fix. 122 | 123 | \begin{ex}\label{ex:par} 124 | Change $n$ to 20 and run \texttt{julia -p nproc Tutorial.jl} for nproc = 1, 2, 3, 5, 10 on a machine with sufficiently many cores. Because of the explicit randomness in computing times, you may want to repeat and take average execution times. Do you observe speedup? Compute the parallel efficiency (observed speedup divided by perfect speedup) from 1 to 10 processes. 125 | \end{ex} 126 | 127 | \begin{ex} 128 | With \texttt{nproc} $= 1$, record and print out the total time spent in the \textit{serial bottleneck} of calling \texttt{Optim.optimize}. Use \href{http://en.wikipedia.org/wiki/Amdahl's\_law}{Amdahl's law} to compute the theoretical maximum possible speedup. Was this achieved in the previous exercise? 129 | \end{ex} 130 | 131 | \begin{advex} 132 | Replace \texttt{Optim.optimize} with a faster approach for minimizing the model function. Hint: this problem can be solved efficiently by using Linear Programming. \textit{(Note: you should complete the rest of the tutorial before attempting this.)} 133 | \end{advex} 134 | 135 | \begin{ex} 136 | Change the random \texttt{sleep} time inside \texttt{f} to a deterministic time, say 0.6 seconds. Repeat Exercise~\ref{ex:par}. Does the parallel efficiency change? 137 | \end{ex} 138 | 139 | You should have observed that the imbalance in computation time does have a significant effect on the parallel speedups observed. Let's visualize this effect. 140 | 141 | \begin{ex} 142 | Modify the code so that each process records the intervals during which it spends inside the function \texttt{f}. Plot these intervals in some reasonable form, using Figure \ref{fig:act} as an example (source code in \texttt{ExampleSolution.jl}). This figure could be improved by plotting the time spent inside \texttt{Optim.optimize} as well. 143 | \end{ex} 144 | 145 | 146 | \begin{figure}[ht] 147 | \includegraphics[scale=0.4]{activityplotnorm} 148 | \includegraphics[scale=0.4]{activityplotdeterm} 149 | \caption{Plots of process activity over time running on 5 processes (4 worker processes). Variable subproblem evaluation time on left, deterministic on right. Black lines denote intervals of activity, white space indicates inactivity. }\label{fig:act} 150 | \end{figure} 151 | 152 | \begin{optex} 153 | Try using \texttt{Base.pmap\_static} (take a look at its definition in \texttt{julia/base/multi.jl}). How does this affect performance and the previous discussion? 154 | \end{optex} 155 | 156 | \section{Asynchronous algorithm} 157 | The asynchronous variant of the cutting-plane algorithm aims to reduce the idle time of the worker processes by eliminating the bottleneck of resolving the model. While previously the parallelism was hidden by high-level Julia functions, now we will need to explicitly consider the master process and the set of workers. 158 | 159 | The idea of the asynchronous algorithm is to minimize the model function using \textit{incomplete information} in order to generate new tasks to feed to workers. That is, instead of waiting for all subderivatives to be computed at a given candidate solution, we generate a new candidate solution (and a full set of new subgradient evaluation tasks) once some proportion $\sigma$ of the subderivatives have been computed. These tasks may be immediately fed to workers, hence workers will spend less time waiting for new tasks. Further description is beyond the scope of this tutorial; see \cite{Linderoth03}. 160 | 161 | The asynchronous algorithm is implemented as \texttt{asyncversion} in \texttt{Tutorial.jl}. It is worth spending a few minutes trying to understand the code. The use of \href{http://docs.julialang.org/en/latest/manual/control-flow/#man-tasks}{Julia tasks} with \texttt{@spawnlocal} and \texttt{@sync} is not intuitively obvious (at least to me). As a starting point, compare with the implementation of \texttt{pmap} in \texttt{julia/base/multi.jl}. 162 | 163 | \begin{ex} 164 | Run both the original parallel code and the asynchronous code with $n = 100$ and nprocs $= 5, 10, 20$, under the original random computation time model (\texttt{max(0.1,0.2+1*randn())}). Is the asynchronous code faster? Note that because less information is used to generate the iterates, the asynchronous version typically requires more iterations. Compare both the total execution time and the average rate of subproblems solved per second. 165 | \end{ex} 166 | 167 | \begin{ex} 168 | Experiment with different random models for computation time. What happens if you increase or decrease the variance (the coefficient of \texttt{randn()})? 169 | \end{ex} 170 | 171 | \begin{advex} 172 | Currently \texttt{Optim.optimize} is called by the same process that manages the workers. Modify the code so that new tasks can be distributed in the middle of calls to \texttt{Optim.optimize}. 173 | \end{advex} 174 | 175 | 176 | \bibliography{refs}{} 177 | \bibliographystyle{plain} 178 | 179 | \end{document} -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Julia Tutorial from MIT IAP 2013 2 | ================================ 3 | 4 | **NOTE:** This is an old tutorial that has not been kept updated with new versions of Julia. 5 | 6 | The material here was used in the Julia tutorial at MIT IAP 2013. It contains the following: 7 | 8 | 1. `LightningRound` is a rapid introduction to Julia, with a number of quick examples designed to give a feel for the language. The presentation and example codes are both included. 9 | 10 | 2. `Vision` contains a presentation on the overall julia vision. 11 | 12 | 3. `DataFrames` contains a presentation on using the Julia DataFrames package, with a number of examples and exercises. 13 | 14 | 4. `Stats` contains a presentation on doing statistics with Julia. 15 | 16 | 5. `GridOfResistors` is an exercise where the students compute the effective resistance in a grid of resistors, and highlights various performance characteristics and language features. 17 | 18 | 6. `NumericalOptimization` is an exercise where students explore different types of parallelism in julia in the context of a numerical optimization problem. 19 | -------------------------------------------------------------------------------- /Stats/header.tex: -------------------------------------------------------------------------------- 1 | \usepackage{bm} 2 | -------------------------------------------------------------------------------- /Stats/slides.md: -------------------------------------------------------------------------------- 1 | % Statistical Models in Julia 2 | % Douglas Bates 3 | % \today 4 | 5 | --- 6 | 7 | # Leveraging types and multiple dispatch in statistics 8 | 9 | * The *R* community is accustomed to using generic functions, methods 10 | and composite types, especially for representing statistical models. 11 | In fact, generics and methods were introduced in *R*'s predecessor, 12 | *S*, with the "white book" ("Statistical Models in S", 1988) 13 | * The concept was sound; the implementation rather casual. `S3' 14 | classes are simply tags on a general type of structure and the 15 | generics allowed for single dispatch only. 16 | * The `S4' system of classes, generics and methods are closer in 17 | design to Julia types and multiple dispatch but have not gained much 18 | of a following nor are they well supported. 19 | * The best known *R* packages that use S4 are the `Matrix' package and 20 | the packages from [Bioconductor](http://www.bioconductor.org). 21 | 22 | --- 23 | 24 | # Statistical models, probability distributions & parameters 25 | 26 | * Statistical models express the distribution of the response 27 | vector as depending on the values of parameters and of covariates. 28 | * Given the observed data we estimate values of the parameters by 29 | optimizing an objective function (e.g. log-likelihood, posterior 30 | density) with respect to the parameters. 31 | * For example, in a linear regression model, the vector of responses, 32 | $\bm y$, is expressed as 33 | $$ \bm y = \bm X\bm\beta+\bm\epsilon,\quad\bm\epsilon\sim\mathcal{N}(\bm0,\sigma^2\bm I)$$ 34 | where the $n\times p$ model matrix, $\bm X$, is derived from the 35 | values of covariates. 36 | * The maximum likelihood estimates of $\bm\beta$ are the least squares 37 | estimates 38 | $$ \widehat{\bm\beta}=\arg\min_{\bm\beta}\|\bm y-\bm X\bm\beta\|^2 $$ 39 | 40 | --- 41 | 42 | # A fitted linear regression model in R and extractors 43 | 44 | > fm = lm(optden ~ carb, Formaldehyde) 45 | > class(fm) 46 | [1] "lm" 47 | > coef(fm) 48 | (Intercept) carb 49 | 0.005085714 0.876285714 50 | > residuals(fm) 51 | 1 2 3 4 5 6 52 | -0.006714286 0.001028571 0.002771429 0.007142857 0.007514286 -0.011742857 53 | > fitted(fm) 54 | 1 2 3 4 5 6 55 | 0.09271429 0.26797143 0.44322857 0.53085714 0.61848571 0.79374286 56 | > deviance(fm) # residual sum of squares 57 | [1] 0.0002992 58 | 59 | --- 60 | 61 | ``` 62 | > summary(fm) 63 | 64 | Call: 65 | lm(formula = optden ~ carb, data = Formaldehyde) 66 | 67 | Residuals: 68 | 1 2 3 4 5 6 69 | -0.006714 0.001029 0.002771 0.007143 0.007514 -0.011743 70 | 71 | Coefficients: 72 | Estimate Std. Error t value Pr(>|t|) 73 | (Intercept) 0.005086 0.007834 0.649 0.552 74 | carb 0.876286 0.013535 64.744 3.41e-07 *** 75 | --- 76 | Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 77 | 78 | Residual standard error: 0.008649 on 4 degrees of freedom 79 | Multiple R-squared: 0.999, Adjusted R-squared: 0.9988 80 | F-statistic: 4192 on 1 and 4 DF, p-value: 3.409e-07 81 | ``` 82 | 83 | --- 84 | 85 | # Methods with `lm' as the class of the first argument 86 | 87 | ``` 88 | > methods(class = "lm") 89 | [1] add1.lm* alias.lm* anova.lm case.names.lm* 90 | [5] confint.lm* cooks.distance.lm* deviance.lm* dfbeta.lm* 91 | [9] dfbetas.lm* drop1.lm* dummy.coef.lm* effects.lm* 92 | [13] extractAIC.lm* family.lm* formula.lm* hatvalues.lm 93 | [17] influence.lm* kappa.lm labels.lm* logLik.lm* 94 | [21] model.frame.lm model.matrix.lm nobs.lm* plot.lm 95 | [25] predict.lm print.lm proj.lm* qr.lm* 96 | [29] residuals.lm rstandard.lm rstudent.lm simulate.lm* 97 | [33] summary.lm variable.names.lm* vcov.lm* 98 | ``` 99 | 100 | --- 101 | 102 | # Categorical covariates in linear models 103 | 104 | ``` 105 | > str(InsectSprays) 106 | 'data.frame': 72 obs. of 2 variables: 107 | $ count: num 10 7 20 14 14 12 10 23 17 20 ... 108 | $ spray: Factor w/ 6 levels "A","B","C","D",..: 1 1 1 1 1 1 1 1 1 1 ... 109 | > anova(fm2 <- lm(count ~ spray, InsectSprays)) 110 | Analysis of Variance Table 111 | 112 | Response: count 113 | Df Sum Sq Mean Sq F value Pr(>F) 114 | spray 5 2668.8 533.77 34.702 < 2.2e-16 115 | Residuals 66 1015.2 15.38 116 | ``` 117 | 118 | --- 119 | 120 | # Generalized linear models 121 | 122 | * The Gaussian (or "normal") distribution has many special 123 | properties. When generalizing to other distributions it helps to 124 | re-write the model as 125 | $$\mathcal{Y}\sim\mathcal{N}(\bm X\bm\beta, \sigma^2\bm I)$$ 126 | * A Poisson regression model for count data (as in the last example) 127 | $$\bm y\sim\mathcal{P}(\exp(\bm X\bm\beta))$$ 128 | where the $\exp$ of the vector is element-wise, $\mu_i=\exp(\eta_i)$ 129 | * For historical reasons, the mapping from $\bm\mu$ to $\bm\eta=\bm 130 | X\bm\beta$ is called the `link function' so this model has a 131 | log-link. 132 | * `logistic regression' for a binary response uses a logistic or 133 | log-odds link, $\eta_i=\log(\mu_i/(1-\mu_i))$ 134 | 135 | --- 136 | 137 | # Fitting GLM's 138 | 139 | * Nelder and Wedderburn realized that models defined by a 140 | distribution, a linear predictor and a link function could be fit by 141 | iteratively re-weight least squares (IRLS). 142 | * As the name implies, IRLS involves repeatedly solving a weighted 143 | least squares problem, updating the weights and the residual and 144 | iterating. 145 | * This is similar to iterative methods for nonlinear least squares. 146 | In fact, the underlying operations of nonlinear regression and IRLS 147 | can be the same. 148 | 149 | --- 150 | 151 | # Linear mixed-effects models (LMM's) 152 | 153 | * In a mixed-effects model some of the coefficients in the linear 154 | predictor apply to particular 'subjects' sampled from a population 155 | and we are interested in the distribution of these 'effects'. 156 | * The model is defined by the conditional distribution 157 | $$(\mathcal{Y}|\mathcal{B}=\bm b)\sim\mathcal{N}(\bm X\bm\beta+\bm 158 | Z\bm b, \sigma^2\bm I)$$ 159 | and the unconditional distribution, $\mathcal{B}\sim\mathcal{N}(\bm 160 | 0,\bm\Sigma)$ 161 | * The dimension of $\bm b$ can be very large (in the millions is not 162 | uncommon), but $\bm Z$ is very sparse, as is $\bm\Sigma$. 163 | * Because $\bm\Sigma$ is positive (semi-)definite we can express it as 164 | $\bm\Sigma=\sigma^2\bm\Lambda\bm\Lambda^T$. $\bm\Lambda$ is a 165 | function of $\bm\theta$ a parameter vector whose dimension is small. 166 | 167 | --- 168 | 169 | # Fitting LMM's 170 | 171 | * By solving a penalized least squares (PLS) problem 172 | $$\begin{bmatrix} 173 | \bm\Lambda^T\bm Z^T\bm Z\bm\Lambda+\bm I&\bm\Lambda^T\bm Z^T\bm X\\ 174 | \bm X^T\bm Z\bm\Lambda&\bm X^T\bm X 175 | \end{bmatrix} 176 | \begin{bmatrix}\widehat{\bm\beta}\\ \tilde{\bm b}\end{bmatrix}= 177 | \begin{bmatrix}\bm\Lambda^T\bm Z^T\bm y&&\bm X^T\bm 178 | y\end{bmatrix}$$ 179 | we can evaluate a 'profiled log-likelihood' that can 180 | be optimized with respect to $\bm\theta$. 181 | * This can be extended to generalized linear mixed-effects models 182 | (GLMMM's) requiring penalized iteratively reweigthed least squares 183 | (PIRLS) or nonlinear mixed-effects models (NLMMs) requiring 184 | penalized nonlinear least squares (PNLS), etc. 185 | * In the goriest forms these kinds of optimizations are nested within 186 | yet another optimization problem taking into account time-series 187 | dependencies or spatial dependencies. 188 | 189 | --- 190 | 191 | # lm and glm in Julia 192 | 193 | Add the **DataFrames**, **Distributions**, **GLM** and **RDataSets** 194 | packages if you don't already have them 195 | 196 | ``` 197 | julia> form = data("datasets", "Formaldehyde") 198 | 6x3 DataFrame: 199 | carb optden 200 | [1,] 1 0.1 0.086 201 | [2,] 2 0.3 0.269 202 | [3,] 3 0.5 0.446 203 | [4,] 4 0.6 0.538 204 | [5,] 5 0.7 0.626 205 | [6,] 6 0.9 0.782 206 | ``` 207 | 208 | --- 209 | 210 | ``` 211 | julia> fm = lm(:(optden ~ carb), form) 212 | Formula: optden ~ carb 213 | 214 | Coefficients: 215 | Term Estimate Std. Error t value Pr(>|t|) 216 | (Intercept) 0.00509 0.00783 0.649 0.552 217 | carb 0.87629 0.01353 64.744 0.000 *** 218 | 219 | Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 220 | 221 | R-squared: 0.0000 222 | 223 | julia> deviance(fm) 224 | 0.0002992000000000012 225 | ``` 226 | 227 | --- 228 | 229 | This morning *dump* of a model frame is broken so we will look at 230 | components 231 | 232 | ``` 233 | julia> typeof(fm) 234 | LmMod 235 | julia> typeof(fm).names 236 | (fr,mm,rr,pp) 237 | julia> dump(fm.rr) # the response component 238 | LmResp 239 | mu: Array(Float64,(6,)) [0.0927143, 0.267971, 0.443229, 0.530857, 0.618486, 0.793743] 240 | offset: Array(Float64,(6,)) [0.0, 0.0, 0.0, 0.0, 0.0, 0.0] 241 | wts: Array(Float64,(6,)) [1.0, 1.0, 1.0, 1.0, 1.0, 1.0] 242 | y: Array(Float64,(6,)) [0.086, 0.269, 0.446, 0.538, 0.626, 0.782] 243 | ``` 244 | 245 | --- 246 | 247 | ``` 248 | julia> dump(fm.pp) # the predictor component 249 | DensePredQR 250 | X: Array(Float64,(6,2)) 6x2 Float64 Array: 251 | 1.0 0.1 252 | 1.0 0.3 253 | 1.0 0.5 254 | 1.0 0.6 255 | 1.0 0.7 256 | 1.0 0.9 257 | beta0: Array(Float64,(2,)) [0.00508571, 0.876286] 258 | delbeta: Array(Float64,(2,)) [0.0, 0.0] 259 | qr: QRDense{Float64} 260 | hh: Array(Float64,(6,2)) 6x2 Float64 Array: 261 | -2.44949 -1.26557 262 | 0.289898 0.63901 263 | 0.289898 -0.141688 264 | 0.289898 -0.277763 265 | 0.289898 -0.413839 266 | 0.289898 -0.68599 267 | tau: Array(Float64,(2,)) [1.40825, 1.15004] 268 | ``` 269 | 270 | --- 271 | 272 | # Multiple linear regression 273 | 274 | ``` 275 | julia> LifeCycleSavings = data("datasets", "LifeCycleSavings") 276 | ... 277 | julia> fm2 = lm(:(sr ~ pop15 + pop75 + dpi + ddpi), LifeCycleSavings) 278 | Formula: sr ~ :(+(pop15,pop75,dpi,ddpi)) 279 | 280 | Coefficients: 281 | Term Estimate Std. Error t value Pr(>|t|) 282 | (Intercept) 28.56609 7.35452 3.884 0.000 *** 283 | pop15 -0.46119 0.14464 -3.189 0.003 ** 284 | pop75 -1.69150 1.08360 -1.561 0.126 285 | dpi -0.00034 0.00093 -0.362 0.719 286 | ddpi 0.40969 0.19620 2.088 0.042 * 287 | ``` 288 | 289 | --- 290 | 291 | # Generalized Linear models 292 | 293 | ``` 294 | julia> dobson = DataFrame({[18.,17,15,20,10,20,25,13,12], 295 | gl(3,1,9), gl(3,3)}, 296 | ["counts","outcome","treatment"]) 297 | 9x3 DataFrame: 298 | counts outcome treatment 299 | [1,] 18.0 1 1 300 | [2,] 17.0 2 1 301 | [3,] 15.0 3 1 302 | [4,] 20.0 1 2 303 | [5,] 10.0 2 2 304 | [6,] 20.0 3 2 305 | [7,] 25.0 1 3 306 | [8,] 13.0 2 3 307 | [9,] 12.0 3 3 308 | ``` 309 | 310 | --- 311 | 312 | ``` 313 | julia> fm3 = glm(:(counts ~ outcome + treatment), dobson, Poisson()) 314 | 1: 46.81189638788046, Inf 315 | 2: 46.76132443472076, 0.0010814910349747592 316 | 3: 46.761318401957794, 1.2901182375636843e-7 317 | 318 | Formula: counts ~ :(+(outcome,treatment)) 319 | 320 | Coefficients: 321 | Term Estimate Std. Error t value Pr(>|t|) 322 | (Intercept) 3.04452 0.17089 17.815 0.000 *** 323 | outcome:2 -0.45426 0.20215 -2.247 0.025 * 324 | outcome:3 -0.29299 0.19273 -1.520 0.128 325 | treatment:2 0.00000 0.19998 0.000 1.000 326 | treatment:3 0.00000 0.19999 0.000 1.000 327 | ``` 328 | 329 | 330 | 331 | --- 332 | 333 | # An LmMod and a GlmMod differ mostly in the response object 334 | 335 | ``` 336 | julia> typeof(fm3) 337 | GlmMod 338 | julia> dump(fm3.rr) 339 | GlmResp 340 | d: Poisson 341 | lambda: Float64 1.0 342 | l: LogLink 343 | eta: Array(Float64,(9,)) [3.04452, 2.59027, 2.75154, 3.04452, 2.59027, 2.75154, 3.04452, 2.59027, 2.75154] 344 | mu: Array(Float64,(9,)) [21.0, 13.3333, 15.6667, 21.0, 13.3333, 15.6667, 21.0, 13.3333, 15.6667] 345 | offset: Array(Float64,(9,)) [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0] 346 | wts: Array(Float64,(9,)) [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] 347 | y: Array(Float64,(9,)) [18.0, 17.0, 15.0, 20.0, 10.0, 20.0, 25.0, 13.0, 12.0] 348 | ``` 349 | 350 | --- 351 | 352 | * Response types: LmResp, GlmResp (to be added NlsResp) 353 | * Predictor types: LinPred, DensePred, DensePredChol, DensePredQR 354 | * to be added: DensePredCholPiv, DensePredQRPiv, DDensePred, 355 | DDensePredChol, SparsePred, 356 | SparsePredChol, SparsePredQR, MixedPred, MixedPredChol, 357 | MixedPredDiag, ... 358 | * Distribution types: Bernoulli, Beta, Binomial, Categorical, Cauchy, 359 | Chisq, Dirichlet, Exponential, FDist, Gamma, Geometric, HyperGeometric, 360 | Logistic, logNormal, Multinomial, NegativeBinomial, NoncentralBeta, 361 | NoncentralChisq, NoncentralF, NoncentralT, Normal, Poisson, TDist, Uniform, 362 | Weibull 363 | * Link types: CauchitLink, CloglogLink, IdentityLink, InverseLink, 364 | LogitLink, LogLink, ProbitLink 365 | 366 | --- 367 | 368 | # What does Julia offer that R doesn't 369 | 370 | * One language providing flexibility and efficiency 371 | -------------------------------------------------------------------------------- /Stats/slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JuliaAttic/julia-tutorial/4acd63163586a9016a31b3fad2ba28a32c14c84b/Stats/slides.pdf -------------------------------------------------------------------------------- /Vision/vision.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/JuliaAttic/julia-tutorial/4acd63163586a9016a31b3fad2ba28a32c14c84b/Vision/vision.pdf --------------------------------------------------------------------------------