├── day1.md ├── day1 ├── regex.md ├── unix.md └── vim.md ├── day2.md ├── day2 ├── networks.md ├── package.json ├── post.txt ├── server.js ├── stream.md ├── stream │ ├── cat.js │ ├── core-loud.js │ ├── dup │ │ ├── api.js │ │ └── run.js │ ├── echo.js │ ├── gunzip.js │ ├── hash.js │ ├── http-get-client.js │ ├── http-post-client.js │ ├── http.js │ ├── line-count.js │ ├── loud.js │ ├── obj.js │ ├── p2p │ │ └── swarm.js │ ├── proxy.js │ ├── ps.js │ ├── r.js │ ├── rpc │ │ ├── client.js │ │ └── server.js │ ├── vpn-client.js │ ├── vpn.js │ ├── w.js │ └── wsock │ │ ├── client.js │ │ ├── nclient.js │ │ ├── public │ │ └── index.html │ │ └── server.js └── userver.js ├── day3.md ├── day3 ├── batch.js ├── blob │ ├── addimg.js │ ├── list-files.js │ ├── read-file.js │ └── server.js ├── browser │ └── main.js ├── crypto.md ├── inc.js ├── kappa.md ├── leveldb.md ├── log │ ├── add.js │ ├── dex.js │ ├── ix-list.js │ ├── list.js │ └── rep.js ├── package.json ├── posts │ ├── mkpost.js │ ├── postlist.js │ ├── posts-by-author.js │ ├── posts-by-time.js │ ├── useradd.js │ └── userlist.js ├── sub.js └── wiki │ └── main.js ├── day4.md ├── day4 ├── assert │ └── test.js ├── cool-test.js ├── hyperx.js ├── package.json ├── tape │ ├── index.js │ └── test │ │ ├── browser-test │ │ └── foo.js │ │ └── one.js ├── template.js ├── testing.md ├── web.md ├── web │ ├── package.json │ ├── public │ │ └── index.html │ ├── reduce.js │ ├── server.js │ └── yo.js └── webgl │ ├── main.js │ └── song.js ├── notes.md └── workshops.md /day1.md: -------------------------------------------------------------------------------- 1 | # day 1: command-line 2 | 3 | Learn the command-line. 4 | 5 | So many reference materials, open source tools, blog posts, and job descriptions 6 | assume command-line proficiency, but the resources for learning the command-line 7 | effectively are scattered and largely unhelpful for people who aren't already 8 | familiar with how to get by in a unix system. 9 | 10 | Join James Halliday (substack) as we dive into the command-line. You will: 11 | 12 | * Become comfortable with the unix command-line using the bash shell. 13 | * Log in to remote servers to edit files and run commands. 14 | * Edit files with vim, a command-line text editor. 15 | * Search and match patterns using regular expressions. 16 | * Automate command-line tasks with their own shell scripts. 17 | * Administer user permissions and system services. 18 | 19 | ## schedule 20 | 21 | * 08:30 - system check, hello 22 | * 09:00 - commands, input/output, and some history 23 | * 09:30 - directories and files, environment variables, flags 24 | * 10:00 - pipes, redirects, scripts, interpolation 25 | * 10:30 - permissions, signals, job control 26 | * 11:00 - user accounts, ssh, public keys 27 | * 11:30 - screen 28 | * 12:00 - lunch etc 29 | * 13:00 - services, init scripts, cron 30 | * 13:30 - regular expressions, grep, sed 31 | * 14:30 - vim basics and practice 32 | * 16:00 - closing notes and wrap up 33 | 34 | We will start out the day by covering basic command-line concepts hands-on. 35 | These skills will build up into more advanced techniques and we will begin to 36 | apply our growing knowledge of the command-line and common utility programs by 37 | writing shell scripts, connecting to remote servers over ssh, and administering 38 | user accounts, permissions, and system services. 39 | 40 | After lunch, we will use regular expressions to search for patterns and format 41 | text. We will edit text files on the command-line with vim. 42 | 43 | ## who is this for? 44 | 45 | This workshop is for people who want to become effective at the command-line. 46 | Students should be comfortable computer users, but no programming or prior 47 | command-line experience is required. 48 | 49 | If you've recently learned some programming, plan to learn programming soon, or 50 | have been programming for a while but never got around to learning the 51 | command-line and a unix system in depth, this workshop is for you! 52 | 53 | ## prerequisites 54 | 55 | Bring a computer running a unix operating system such as GNU/Linux or MacOSX. 56 | 57 | If you have a Windows computer, please install Linux. Most Linux installers will 58 | let you dual boot your computer into both Linux and Windows if you want to keep 59 | Windows around. If you don't know where to start, download Ubuntu from 60 | https://www.ubuntu.com/ and install from a bootable USB stick. 61 | 62 | -------------------------------------------------------------------------------- /day1/regex.md: -------------------------------------------------------------------------------- 1 | # regular expressions 2 | 3 | regex is a pattern-matching language 4 | 5 | many programming languages and system tools 6 | have a regex engine 7 | 8 | --- 9 | # system tools with regex 10 | 11 | * grep 12 | * sed 13 | * perl 14 | * vim 15 | * less 16 | 17 | --- 18 | # regex in javascript 19 | 20 | * str.split(re) 21 | * str.match(re) 22 | * str.replace(re, 23 | * re.test(str) 24 | * re.exec(str) 25 | 26 | --- 27 | # common regex format 28 | 29 | * `/PATTERN/` - match 30 | * `s/PATTERN/REP/` - replace 31 | 32 | You can also have FLAGS at the end: 33 | 34 | * `/PATTERN/FLAGS` 35 | * `s/PATTERN/REP/FLAGS` 36 | 37 | --- 38 | # sometimes you need the slashes 39 | 40 | ``` 41 | $ echo cat cabbage | sed 's/a/@/g' 42 | c@t c@bb@ge 43 | ``` 44 | 45 | --- 46 | # sometimes the slashes are implied 47 | 48 | ``` 49 | $ echo -e 'one\ntwo\nthree' | grep ^t 50 | two 51 | three 52 | ``` 53 | 54 | --- 55 | # sometimes the slashes are not slashes at all 56 | 57 | ``` 58 | $ echo 'xyz party' | sed 's!xyz!cat!' 59 | cat party 60 | ``` 61 | 62 | --- 63 | # replacing in javascript 64 | 65 | ``` js 66 | '1 two three\n'.replace(/1/, 'one') 67 | ``` 68 | 69 | similar to: 70 | 71 | ``` sh 72 | echo '1 two three' | sed 's/1/one/' 73 | ``` 74 | 75 | --- 76 | # match testing in javascript 77 | 78 | ``` js 79 | if (/^-(h|-help)$/.test(process.argv[2])) { 80 | console.log('usage: ...') 81 | } 82 | ``` 83 | --- 84 | # capturing in javascript 85 | 86 | ``` js 87 | var m = /^hello (\S+)/.test('hello cat') 88 | console.log(m[1]) // cat 89 | ``` 90 | 91 | --- 92 | # splitting in javascript 93 | 94 | ``` js 95 | > 'one_two-three'.split(/[_-]/) 96 | [ 'one', 'two', 'three' ] 97 | ``` 98 | 99 | --- 100 | # flags 101 | 102 | ``` 103 | /PATTERN/FLAGS 104 | s/PATTERN/REP/FLAGS 105 | ``` 106 | 107 | * i - case insensitive 108 | * g - match all occurences (global) 109 | * m - treat string as multiple lines 110 | * s - treat string as a single line 111 | 112 | --- 113 | # metacharacters 114 | 115 | * `.` matches any character 116 | * `?` - zero or one time 117 | * `*` - zero or more times 118 | * `+` - one or more times 119 | 120 | * `[]` - character class 121 | * `^` - anchor at the beginning 122 | * `$` - anchor to the end 123 | 124 | * `(a|b)` - match a or b 125 | 126 | * `()` - capture group 127 | * `(?:)` non capture group 128 | 129 | * `\d` - digit `[0-9]` 130 | * `\w` - word `[A-Za-z0-9_]` 131 | * `\s` - whitespace `[ \t\r\n\f]` 132 | 133 | --- 134 | `.` matches any character 135 | 136 | ``` 137 | $ echo hello beep boop | sed 's/b..p/XXXX/g' 138 | hello XXXX XXXX 139 | ``` 140 | 141 | --- 142 | # quantifiers 143 | 144 | * `?` - zero or one time 145 | * `*` - zero or more times 146 | * `+` - one or more times 147 | 148 | ``` 149 | $ echo 'dog and doge' | sed 's/doge\?/DOGE/g' 150 | DOGE and DOGE 151 | $ echo 'beep bp beeeeep' | sed 's/be*p/BEEP/g' 152 | BEEP BEEP BEEP 153 | $ echo 'beep bp beeeeep' | sed 's/be\+p/BEEP/g' 154 | BEEP bp BEEP 155 | ``` 156 | 157 | --- 158 | # when to escape metacharacters 159 | 160 | In some engines, you need to escape metacharacters 161 | such as `+` and `?`. In others, you don't. 162 | 163 | In javascript and perl, you generally don't need to 164 | escape metacharacters. To use sed and grep in a 165 | similar way, use: 166 | 167 | * `sed -r` 168 | * `grep -E` 169 | 170 | --- 171 | # character class 172 | 173 | `[...]` 174 | 175 | Any characters inside the square brackets will 176 | match. 177 | 178 | For example, to match a vowel character: `[aeiou]`. 179 | 180 | ``` 181 | $ echo 'beep and boop' \ 182 | | sed 's/b[aeiou]\+p/BXXP/g' 183 | BXXP and BXXP 184 | ``` 185 | 186 | --- 187 | # character class ranges 188 | 189 | `[A-Z]` 190 | 191 | You can use `-` to specify ranges. 192 | 193 | ``` 194 | $ echo 'beep and boop' | sed 's/[a-f]/X/g' 195 | XXXp XnX Xoop 196 | ``` 197 | 198 | --- 199 | # negated character class 200 | 201 | `[^...]` 202 | 203 | Put a `^` after the opening square bracket in a 204 | character class to negate it. 205 | 206 | For example, to match a non-vowel character: `[^aeiou]` 207 | 208 | ``` 209 | $ echo 'beep boop' | sed 's/[^aeiou]/Z/g' 210 | ZeeZZZooZ 211 | ``` 212 | 213 | --- 214 | # character class sequences 215 | 216 | Regex engines provide many pre-defined character class sequences: 217 | 218 | * `\w` - word character: `[A-Za-z0-9_]` 219 | * `\W` - non-word character: `[^A-Za-z0-9_]` 220 | * `\s` - whitespace: `[ \t\r\n\f]` 221 | * `\S` - non-whitespace: `[^ \t\r\n\f]` 222 | * `\d` - digit: `[0-9]` 223 | * `\D` - non-digit: `[^0-9]` 224 | 225 | --- 226 | # anchors 227 | 228 | * `^` - anchor at the beginning 229 | * `$` - anchor to the end 230 | 231 | --- 232 | # groups 233 | 234 | (a|b) - match a or b 235 | 236 | * `()` capture group 237 | * `(?:)` non capture group 238 | 239 | --- 240 | # capture groups in sed 241 | 242 | ``` 243 | $ echo 'hey whatever' | sed -r 's/<([^>]+)>/(\1)/g' 244 | hey (cool) whatever 245 | ``` 246 | 247 | --- 248 | # back references in sed 249 | 250 | ``` 251 | $ echo 'hey cool cool beans' | sed -r 's/(\S+) \1/REPEATED/' 252 | hey REPEATED beans 253 | ``` 254 | 255 | --- 256 | # capture groups in javascript 257 | 258 | ``` js 259 | var str = 'hey whatever' 260 | var m = /<([^]+)>/.exec(str) 261 | console.log(m[1]) // cool 262 | ``` 263 | 264 | ``` js 265 | var str = 'hey whatever' 266 | console.log(str.replace(/<([^]+)>/,'MATHEMATICAL')) 267 | ``` 268 | 269 | -------------------------------------------------------------------------------- /day1/unix.md: -------------------------------------------------------------------------------- 1 | # unix 2 | 3 | introduction to the unix command-line 4 | 5 | --- 6 | # unix 7 | 8 | UNIX was an operating system developed at AT&T 9 | Bell Labs in the 1960s through the 1980s. 10 | 11 | https://www.youtube.com/watch?v=tc4ROCJYbm0 12 | 13 | GNU/Linux, MacOSX, and Android are all based on ideas 14 | and specifications created by UNIX. 15 | 16 | https://en.wikipedia.org/wiki/File:Unix_history-simple.svg 17 | 18 | --- 19 | # time-sharing 20 | 21 | UNIX was originally built for large mainframe 22 | computers that many people would use at the same 23 | time. 24 | 25 | --- 26 | # terminals and teleprinters 27 | 28 | * teleprinters printed program output on paper 29 | * terminals displayed output on a CRT monitor 30 | 31 | Neither device had processing power of their own. 32 | 33 | Connected to the mainframe over cables or by telephone. 34 | 35 | --- 36 | # teletype legacy: standard input and output 37 | 38 | Every program on a UNIX system can read input from 39 | the standard input device (stdin) and write to 40 | standard output (stdout). 41 | 42 | By default, stdin comes from the keyboard and 43 | stdout gets "printed" to the graphical display. 44 | 45 | --- 46 | # organization 47 | 48 | The UNIX operating system is a collection of 49 | programs, each with a special role: 50 | 51 | * kernel 52 | * shell 53 | * utilities 54 | 55 | --- 56 | # kernel 57 | 58 | mediate access between user programs and system resources 59 | 60 | * CPU scheduling 61 | * I/O to computer hardware 62 | * memory 63 | 64 | Programs request resources by making a syscall. 65 | 66 | --- 67 | # shell 68 | 69 | A shell is a computer program that can execute 70 | other programs from a text-based interface. 71 | 72 | In a text-based interface, you interact with a 73 | program completely from the command-line with text 74 | commands and text output. 75 | 76 | Most modern shells are strongly influenced by the 77 | first UNIX shells. 78 | 79 | --- 80 | # shells through the ages 81 | 82 | * thompson shell - Ken Thompson 1971 83 | * pwb (mashey) shell - John Mashey 1975 84 | * bourne shell - Stephen Bourne 1977 85 | * c shell (csh) - Bill Joy 1978 86 | * tcsh - Ken Greer and Mike Ellis 1983 87 | * korn shell - David Korn 1983 88 | * bourne again shell (bash) - Brian Fox 1987 89 | * almquist shell (ash) - Kenneth Almquist 1989 90 | * debian almquist shell (dash) - Herbert Xu 1997 91 | 92 | incomplete list of popular or influential shells 93 | 94 | --- 95 | # utilities 96 | 97 | Any distribution of UNIX will come with dozens of 98 | other programs that perform narrow single-purpose 99 | tasks. 100 | 101 | The available utilities on a given system vary 102 | widely but some utilities are very common. 103 | 104 | For example, there is a command to make new 105 | directories and another to move files. 106 | 107 | --- 108 | # why UNIX still matters 109 | 110 | * portable to many kinds of hardware 111 | * consistent conventions 112 | * vast software ecosystem 113 | * text! 114 | 115 | --- 116 | # places you can find a unix command-line 117 | 118 | * wifi routers 119 | * dsl and cable modems 120 | * raspberry pi, beaglebone, nvidia jetson 121 | * android phones 122 | * linux laptop or desktop 123 | * Mac OSX computer 124 | * web server 125 | 126 | You can take your command-line skills with 127 | you to all of these platforms and more! 128 | 129 | --- 130 | # text interface 131 | 132 | To remotely access a UNIX system, you can use the 133 | same command-line tools and interface that you use 134 | locally. You can remotely access devices without a 135 | display. 136 | 137 | Text is easy to read so you can poke around more 138 | easily to figure out what's going on. Many aspects 139 | of computer programming involve shuffling text 140 | around. UNIX excels at these kinds of tasks. 141 | 142 | --- 143 | # unix philosophy 144 | 145 | The unix philosophy is a set of design principles for 146 | how programs relate to each other. 147 | 148 | * each program should do one thing well 149 | * the output of a program can become the input of another 150 | 151 | Unix Programming Environment 1984 152 | Brian Kernighan and Rob Pike 153 | 154 | --- 155 | # let's learn the command line! 156 | 157 | bash is a popular shell for UNIX-like systems. 158 | 159 | Open up a bash shell for the next sections to 160 | follow along. 161 | 162 | If you're not sure which shell you're in, type: 163 | 164 | echo $SHELL 165 | 166 | If you're in bash, you sould see something like: 167 | 168 | /bin/bash 169 | 170 | --- 171 | # list files 172 | 173 | You can use the `ls` command to show all the files 174 | in the current directory. 175 | 176 | Type `ls` and you should see something like: 177 | 178 | ~ $ ls 179 | doc media notes.txt projects 180 | 181 | --- 182 | # arguments 183 | 184 | By default, `ls` lists files from the current directory. 185 | 186 | You can list files from another directory by 187 | giving `ls` an argument. An argument is just 188 | another piece of text after the `ls`. 189 | 190 | --- 191 | For example, to list the files in `/` (the root) 192 | we can do: 193 | 194 | ``` 195 | ~ $ ls / 196 | bin etc lib media proc sbin sys var 197 | boot home lib64 mnt root selinux tmp vmlinuz 198 | dev initrd.img lost+found opt run srv usr 199 | ``` 200 | 201 | In this example, `ls` is the command and `/` is the argument. 202 | 203 | Commands can have multiple arguments separated by 204 | spaces or no arguments. 205 | 206 | --- 207 | # print the current directory 208 | 209 | To display the current directory, you can use the 210 | `pwd` command: 211 | 212 | $ pwd 213 | /home/substack 214 | 215 | pwd stands for print working directory. 216 | 217 | 218 | --- 219 | # change directory 220 | 221 | To change the current working directory, use the 222 | `cd` command. The `cd` command takes a single 223 | argument: the directory to move to. 224 | 225 | After changing the current directory, list the 226 | files again with `ls`. 227 | 228 | --- 229 | ``` 230 | ~ $ ls 231 | doc media notes.txt projects 232 | ~ $ cd media 233 | ~/media $ ls 234 | 3d audio avatars vector warp 235 | ~/media $ cd warp 236 | ~/media/warp $ ls 237 | mac.sh* mac_startup.mp3 mac_warped.mp3 watch.js 238 | Mac Startup-i9qOJqNjalE.mp4 mac_startup.wav mac_warp.mp3 239 | ``` 240 | 241 | --- 242 | # special directories 243 | 244 | There are some special directories: 245 | 246 | * `..` - the parent directory 247 | * `.` - the current directory 248 | * `~` - your home directory 249 | 250 | To navigate back up to the parent directory, do 251 | `cd ..`. 252 | 253 | ``` 254 | ~/media/warp $ cd .. 255 | ~/media $ 256 | ``` 257 | 258 | --- 259 | You can also list the parent directory without 260 | changing the current directory by doing `ls ..`: 261 | 262 | ``` 263 | ~/media $ ls .. 264 | doc notes.txt media projects 265 | ``` 266 | 267 | You can add paths after `..` too: 268 | 269 | ``` 270 | ~/media $ ls ../projects/workshops 271 | computers.markdown unix.markdown 272 | ``` 273 | 274 | --- 275 | Or `ls .` is the same as `ls`: 276 | 277 | ``` 278 | ~/media $ ls . 279 | 3d audio avatars vector warp 280 | ``` 281 | 282 | Jump back to your home directory at any time by 283 | typing `cd` with no arguments. 284 | 285 | --- 286 | # cat 287 | 288 | cat was originally written to concatenate all the 289 | files from its arguments: 290 | 291 | ~/doc $ cat beep.txt boop.txt 292 | BEEP 293 | BOOP 294 | 295 | but it also a handy way to display single text 296 | files on the command-line: 297 | 298 | ~/doc $ cat beep.txt 299 | BEEP 300 | 301 | --- 302 | # cp 303 | 304 | Copy a file to another directory or file name. You 305 | can copy a single file to make a new duplicate 306 | file: 307 | 308 | ~/doc $ ls 309 | a.txt 310 | 311 | We can copy a.txt to b.txt: 312 | 313 | ~/doc $ cp a.txt b.txt 314 | 315 | 316 | --- 317 | Now there are 2 identical files, `a.txt` and `b.txt`: 318 | 319 | ``` 320 | ~/doc $ ls 321 | a.txt b.txt 322 | ``` 323 | 324 | --- 325 | # cp 326 | 327 | You can copy a file or a directory too. Here we'll 328 | copy `a.txt` to the directory called `wow`: 329 | 330 | ~/doc $ mkdir wow 331 | ~/doc $ ls 332 | a.txt b.txt wow 333 | ~/doc $ cp a.txt wow 334 | 335 | now `wow/` has an `a.txt` file in it: 336 | 337 | ~/doc $ ls wow 338 | a.txt 339 | 340 | --- 341 | You can copy to a specific destination file: 342 | 343 | ``` 344 | ~/doc $ cp a.txt wow/whatever.txt 345 | ~/doc $ ls wow 346 | a.txt whatever.txt 347 | ``` 348 | 349 | --- 350 | # cp (multiple files) 351 | 352 | You can even copy multiple files at once to a new 353 | place: 354 | 355 | ``` 356 | ~/doc $ mkdir xyz 357 | ~/doc $ cp a.txt b.txt xyz/ 358 | ~/doc $ ls xyz 359 | a.txt b.txt 360 | ``` 361 | 362 | The last argument is the destination file or 363 | directory and the other arguments are the source 364 | files. 365 | 366 | --- 367 | 368 | # cp -r 369 | 370 | If you have a directory full of files and 371 | directories you want to copy to a new place, you 372 | can use `cp -r` to recursively copy a directory 373 | and all its subdirectories to a new location: 374 | 375 | ``` 376 | ~/doc $ mkdir xyz/123 377 | ~/doc $ cp a.txt xyz/123/ 378 | ~/doc $ cp -r xyz newxyz 379 | ~/doc $ ls newxyz/ 380 | 123 a.txt b.txt 381 | ~/doc $ ls newxyz/123 382 | a.txt 383 | ``` 384 | 385 | --- 386 | Likewise, there is a `-R` for the `ls` command that recursively lists 387 | subdirectories: 388 | 389 | ``` 390 | ~/doc $ ls -R newxyz 391 | newxyz: 392 | 123 a.txt b.txt 393 | 394 | newxyz/123: 395 | a.txt 396 | ``` 397 | 398 | --- 399 | # mv 400 | 401 | The `mv` command is used to rename and overwrite 402 | files and directories. 403 | 404 | To rename a file, set the first argument to the 405 | original file name and the second argument to the 406 | new file name or destination directory. 407 | 408 | --- 409 | We can rename `a.txt` to be `pigeon.txt`: 410 | 411 | ``` 412 | ~/doc $ mv a.txt pigeon.txt 413 | ~/doc $ ls 414 | b.txt newxyz pigeon.txt xyz 415 | ``` 416 | 417 | --- 418 | Or we can move a file to a new directory: 419 | 420 | ``` 421 | ~/doc $ mv pigeon.txt xyz 422 | ~/doc $ ls xyz 423 | 123 a.txt b.txt pigeon.txt 424 | ``` 425 | 426 | --- 427 | We can rename directories just the same as files: 428 | 429 | ``` 430 | ~/doc $ mv xyz woo 431 | ~/doc $ ls 432 | b.txt newxyz woo 433 | ~/doc $ ls woo 434 | 123 a.txt b.txt pigeon.txt 435 | ``` 436 | 437 | --- 438 | # mkdir 439 | 440 | To make a new directory, just execute the `mkdir` 441 | command with a list of new directory names to make 442 | as arguments: 443 | 444 | $ mkdir hooray 445 | 446 | and now a new directory called `hooray` exists. 447 | 448 | --- 449 | You can create multiple directories at once: 450 | 451 | $ mkdir one two 452 | 453 | and now two new directories, `one` and `two`, 454 | exist. 455 | 456 | --- 457 | # mkdir -p 458 | 459 | Suppose we want to make the following nested 460 | directory structure: 461 | 462 | foo/ 463 | bar/ 464 | baz/ 465 | qrs/ 466 | 467 | --- 468 | Instead of doing: 469 | 470 | ``` 471 | ~ $ mkdir foo foo/bar foo/bar/baz foo/bar/qrs 472 | ``` 473 | 474 | We can just do: 475 | 476 | ``` 477 | ~/doc $ mkdir -p foo/bar/baz foo/bar/qrs 478 | ``` 479 | 480 | and the necessary parent directories `foo/` and 481 | `foo/bar/` will be created automatically. 482 | 483 | --- 484 | # brace expansion 485 | 486 | There is a handy syntax built into bash for 487 | expanding patterns that would be repetitive to 488 | type out by hand. 489 | 490 | Instead of doing something like: 491 | 492 | ~/doc $ mkdir -p foo/bar/baz foo/bar/qrs 493 | 494 | we can use a list of items between curly braces: 495 | 496 | ~/doc $ mkdir -p foo/bar/{baz,qrs} 497 | 498 | which expands to the same command as before. 499 | 500 | --- 501 | To prove this you can use `echo` to see what the 502 | expansion is: 503 | 504 | ~ $ echo mkdir -p foo/bar/{baz,qrs} 505 | mkdir -p foo/bar/baz foo/bar/qrs 506 | 507 | The items that a brace expansion generates are 508 | separated by spaces as if you had typed out those 509 | words by hand. 510 | 511 | --- 512 | You can have as many items as you like in a list: 513 | 514 | ``` 515 | ~ $ echo robot-{one,two,three,four}-x 516 | robot-one-x robot-two-x robot-three-x robot-four-x 517 | ``` 518 | 519 | --- 520 | With brace expansions, you can have multiple expansions: 521 | 522 | ``` 523 | ~/doc $ echo robot/{c3po,r2d2}/{sound.mp3,info.txt} 524 | robot/c3po/sound.mp3 robot/c3po/info.txt robot/r2d2/sound.mp3 robot/r2d2/info.txt 525 | ``` 526 | 527 | You can even nest the expansions! 528 | 529 | ``` 530 | ~/doc $ echo x-{wing,b{ee,oo}p} 531 | x-wing x-beep x-boop 532 | ``` 533 | 534 | --- 535 | # brace expansion sequences 536 | 537 | It can be tedious to type out numerical lists by hand. 538 | 539 | Brace expansions can help with that: 540 | 541 | ``` 542 | ~/doc $ echo wow{1..10} 543 | wow1 wow2 wow3 wow4 wow5 wow6 wow7 wow8 wow9 wow10 544 | ``` 545 | 546 | and you can even specify an amount to skip: 547 | 548 | ``` 549 | ~/doc $ echo img{0..100..10} 550 | img0 img10 img20 img30 img40 img50 img60 img70 img80 img90 img100 551 | ``` 552 | 553 | --- 554 | # rm 555 | 556 | To remove a file, just do: 557 | 558 | ~/doc $ rm b.txt 559 | 560 | You can remove multiple files at once: 561 | 562 | ~/doc $ rm newxyz/a.txt newxyz/b.txt 563 | 564 | and you can remove entire directories including subdirectories with: 565 | 566 | ~/doc $ rm -r newxyz 567 | 568 | Be very careful with `-r`. You might accidentally 569 | delete much more than you meant to delete! 570 | 571 | --- 572 | # wc 573 | 574 | The `wc` command computes the number of lines, 575 | words, and bytes in a file: 576 | 577 | ~ $ wc notes.txt 578 | 3 7 35 /home/substack/notes.txt 579 | 580 | To see each field independently, you can use 581 | different options: arguments that start with a `-` 582 | or `--` followed by a letter or word. 583 | 584 | --- 585 | To get just the word counts, we can use `-w`: 586 | 587 | ~ $ wc -w notes.txt 588 | 7 notes.txt 589 | 590 | --- 591 | To get just the number of lines in a file, use `-l`: 592 | 593 | ~ $ wc -l notes.txt 594 | 3 notes.txt 595 | 596 | --- 597 | To get just the number of bytes in a file, use `-c`: 598 | 599 | ~ $ wc -c notes.txt 600 | 35 notes.txt 601 | 602 | --- 603 | If you don't specify a file, `wc` will read from 604 | stdin. Type Ctrl+D (^D) to end the input. 605 | 606 | ~ $ wc -l 607 | one 608 | two 609 | three 610 | four 611 | ^D 612 | 4 613 | 614 | --- 615 | # man 616 | 617 | All of these command options are a lot to remember! 618 | 619 | You can pull up documentation at any time in your 620 | shell by typing `man foo` for any command `foo`. 621 | 622 | For example to read up on all the options you can 623 | give to the `wc` command, do: 624 | 625 | ~ $ man wc 626 | 627 | The help page will open up in your `$PAGER`. Type 628 | `q` to exit back to your shell. 629 | 630 | --- 631 | # more on options 632 | 633 | Options (also called flags or switches) are 634 | special arguments that start with a `-` or `--` 635 | followed by a letter or word. 636 | 637 | --- 638 | Generally speaking, they are distinct from other 639 | arguments in that their order usually doesn't 640 | matter. For example: 641 | 642 | grep -i wow 643 | 644 | is the same as 645 | 646 | grep wow -i 647 | 648 | where `-i` just informs the `grep` command to 649 | perform a case-insensitive search. 650 | 651 | --- 652 | Sometimes options have a value that follows: 653 | 654 | head -n 1 655 | 656 | means that `-n` has the value `1`. 657 | 658 | Sometimes you can omit the space: 659 | 660 | head -n1 661 | 662 | but each program individually decides how to 663 | interpret its arguments. 664 | 665 | --- 666 | # absolute and relative paths 667 | 668 | Paths that start with `.` or `..` are relative paths. 669 | Paths that start with `/` are absolute paths. 670 | 671 | --- 672 | Relative paths are resolved according to the 673 | current working directory: 674 | 675 | ``` 676 | ~/doc $ cat ../media/warp/mac.sh 677 | #!/bin/bash 678 | youtube-dl 'https://www.youtube.com/watch?v=i9qOJqNjalE' 679 | ffmpeg -i *.mp4 -vn mac_startup.wav 680 | sox mac_startup.wav mac_warp.mp3 chorus 0.6 0.9 25 0.9 1 8 -s \ 681 | echos 0.8 0.7 40 0.25 63 0.3 phaser 1 0.7 3 0.7 0.5 -t 682 | play mac_startup.wav 683 | ``` 684 | 685 | --- 686 | Absolute paths are the same no matter what the 687 | current working directory is: 688 | 689 | ~/projects/workshops $ cat /etc/issue 690 | Debian GNU/Linux 7 \n \l 691 | 692 | --- 693 | # echo 694 | 695 | The echo command just prints text from its arguments: 696 | 697 | ~ $ echo wow cool 698 | wow cool 699 | 700 | This is not very useful by itself, but becomes 701 | useful when combined with redirects and pipes. 702 | 703 | --- 704 | # write to a file 705 | 706 | Using the `>` character, you can write the output 707 | of a command to a file. 708 | 709 | For example, to make a new file `greetings.txt` 710 | with the contents "ahoy thar", we can do: 711 | 712 | ~ $ echo ahoy thar > greetings.txt 713 | 714 | and to print the contents of greetings.txt, use `cat`: 715 | 716 | ~ $ cat greetings.txt 717 | ahoy thar 718 | 719 | --- 720 | You can redirect the output of any program to a file: 721 | 722 | ~ $ ls / > list.txt 723 | 724 | --- 725 | # append to a file 726 | 727 | The `>` redirect operator will overwrite a file 728 | with new contents if it already exists. 729 | 730 | There is a `>>` operator that appends to the end 731 | of a file if it already exists: 732 | 733 | ~ $ echo wow > cool.txt 734 | ~ $ ls >> cool.txt 735 | ~ $ cat cool.txt 736 | wow 737 | cool.txt 738 | doc 739 | media 740 | notes.txt 741 | projects 742 | 743 | --- 744 | # read from a file 745 | 746 | You can read a file into the stdin of a program 747 | with `<`. 748 | 749 | Remember that if `wc` doesn't get a file as an 750 | argument, it will read from stdin. We can load a 751 | file in to `wc` with `<` instead: 752 | 753 | ~ $ wc -c < notes.txt 754 | 35 755 | 756 | --- 757 | # pipes! 758 | 759 | The last but most important kind of redirect is 760 | the pipe operator `|`. 761 | 762 | With `|` you can feed the output of one program to 763 | the input of the next. 764 | 765 | For example, the `ls -1` command will list files, 766 | one per line, to stdout. The `wc -l` command, 767 | meanwhile, will count the number of lines. 768 | 769 | --- 770 | By piping these two programs together, we can 771 | count the number of files and subdirectories in a 772 | directory: 773 | 774 | ~ $ ls -1 | wc -l 775 | 5 776 | 777 | and indeed, there are five files and 778 | subdirectories in this directory: 779 | 780 | ~ $ ls -1 781 | cool.txt 782 | doc 783 | media 784 | notes.txt 785 | projects 786 | 787 | You can chain together commands with `|` as much 788 | as you like. 789 | 790 | --- 791 | Here's an example using two new commands `curl` 792 | and `sed` that will fetch Moby Dick from Project 793 | Gutenberg and count the number of occurences of 794 | "whale", case-insensitive: 795 | 796 | ``` 797 | ~ $ curl -s http://www.gutenberg.org/cache/epub/2701/pg2701.txt 798 | | sed -r 's/\s+/\n/g' | grep -i whale | wc -l 799 | 1691 800 | ``` 801 | 802 | --- 803 | We can even save that number of a file. Just add 804 | `> whale_count.txt` to the end of the pipeline: 805 | 806 | ``` 807 | ~ $ curl -s http://www.gutenberg.org/cache/epub/2701/pg2701.txt | 808 | sed -r 's/\s+/\n/g' | grep -i whale | wc -l > whalecount.txt 809 | ``` 810 | 811 | --- 812 | # pipeline breakdown: curl 813 | 814 | Here's a breakdown of each part of the pipeline 815 | and what it does: 816 | 817 | ``` 818 | curl -s http://www.gutenberg.org/cache/epub/2701/pg2701.txt 819 | ``` 820 | 821 | fetches Moby Dick from Project Gutenberg and 822 | prints the results to stdout. 823 | 824 | --- 825 | # pipeline breakdown: sed 826 | 827 | sed -r 's/\s+/\n/g' 828 | 829 | converts all whitespace (tabs, spaces, newlines) 830 | into newlines. 831 | 832 | This means that each word gets its own line: 833 | 834 | ``` 835 | ~ $ echo one two three beep boop | sed -r 's/\s+/\n/g' 836 | one 837 | two 838 | three 839 | beep 840 | boop 841 | ``` 842 | 843 | --- 844 | # pipeline breakdown: grep 845 | 846 | grep -i whale 847 | 848 | filters the output so that only lines that contain 849 | the word "whale" will be shown. `-i` makes the 850 | search case-insensitive. 851 | 852 | --- 853 | For example if we have a file `tale.txt`: 854 | 855 | Wow 856 | such 857 | a 858 | whale. 859 | A 860 | whale 861 | of 862 | a 863 | WHALE! 864 | 865 | --- 866 | then our grep command will show: 867 | 868 | ~ $ grep -i whale < tale.txt 869 | whale. 870 | whale 871 | WHALE! 872 | 873 | --- 874 | # pipeline breakdown: wc -l 875 | 876 | wc -l 877 | 878 | counts the number of lines from stdin and prints the result. 879 | 880 | --- 881 | # head 882 | 883 | The head command prints the first part of a file. 884 | 885 | If a file isn't given, `head` reads from stdin. 886 | 887 | Read the first 3 lines of a file with `head -n3`: 888 | 889 | ``` 890 | $ head -n3 mobydick.txt 891 | The Project Gutenberg EBook of Moby Dick; or The Whale, by Herman Melville 892 | 893 | This eBook is for the use of anyone anywhere at no cost and with 894 | ``` 895 | 896 | --- 897 | Read the first 20 bytes of a file with `head -c20`: 898 | 899 | ``` 900 | ~ $ head -c20 mobydick.txt 901 | The Project Guten 902 | ``` 903 | 904 | --- 905 | 906 | # tail 907 | 908 | The tail command prints the last part of a file. 909 | 910 | If a file isn't given, `tail` reads from stdin. 911 | 912 | Read the last 4 lines of a file with `tail -n4`: 913 | 914 | ``` 915 | ~ $ tail -n4 mobydick.txt 916 | This Web site includes information about Project Gutenberg-tm, 917 | including how to make donations to the Project Gutenberg Literary 918 | Archive Foundation, how to help produce our new eBooks, and how to 919 | subscribe to our email newsletter to hear about new eBooks. 920 | ``` 921 | 922 | --- 923 | Read the last 9 bytes of a file with `tail -c9`: 924 | 925 | ``` 926 | ~ $ tail -c9 mobydick.txt 927 | eBooks. 928 | ``` 929 | 930 | --- 931 | # cal 932 | 933 | If you need a handy text calendar, just type `cal`: 934 | 935 | ``` 936 | ~ $ cal 937 | December 2014 938 | Su Mo Tu We Th Fr Sa 939 | 1 2 3 4 5 6 940 | 7 8 9 10 11 12 13 941 | 14 15 16 17 18 19 20 942 | 21 22 23 24 25 26 27 943 | 28 29 30 31 944 | ``` 945 | 946 | --- 947 | You can show the current, previous, and next month: 948 | 949 | ``` 950 | ~ $ cal -3 951 | November 2014 December 2014 January 2015 952 | Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa 953 | 1 1 2 3 4 5 6 1 2 3 954 | 2 3 4 5 6 7 8 7 8 9 10 11 12 13 4 5 6 7 8 9 10 955 | 9 10 11 12 13 14 15 14 15 16 17 18 19 20 11 12 13 14 15 16 17 956 | 16 17 18 19 20 21 22 21 22 23 24 25 26 27 18 19 20 21 22 23 24 957 | 23 24 25 26 27 28 29 28 29 30 31 25 26 27 28 29 30 31 958 | 30 959 | ``` 960 | 961 | --- 962 | Or you can show a whole year: 963 | 964 | ``` 965 | ~ $ cal 2015 966 | 2015 967 | January February March 968 | Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa 969 | 1 2 3 1 2 3 4 5 6 7 1 2 3 4 5 6 7 970 | 4 5 6 7 8 9 10 8 9 10 11 12 13 14 8 9 10 11 12 13 14 971 | 11 12 13 14 15 16 17 15 16 17 18 19 20 21 15 16 17 18 19 20 21 972 | 18 19 20 21 22 23 24 22 23 24 25 26 27 28 22 23 24 25 26 27 28 973 | 25 26 27 28 29 30 31 29 30 31 974 | 975 | 976 | April May June 977 | Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa 978 | 1 2 3 4 1 2 1 2 3 4 5 6 979 | 5 6 7 8 9 10 11 3 4 5 6 7 8 9 7 8 9 10 11 12 13 980 | 12 13 14 15 16 17 18 10 11 12 13 14 15 16 14 15 16 17 18 19 20 981 | 19 20 21 22 23 24 25 17 18 19 20 21 22 23 21 22 23 24 25 26 27 982 | 26 27 28 29 30 24 25 26 27 28 29 30 28 29 30 983 | 31 984 | 985 | July August September 986 | Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa 987 | 1 2 3 4 1 1 2 3 4 5 988 | 5 6 7 8 9 10 11 2 3 4 5 6 7 8 6 7 8 9 10 11 12 989 | 12 13 14 15 16 17 18 9 10 11 12 13 14 15 13 14 15 16 17 18 19 990 | 19 20 21 22 23 24 25 16 17 18 19 20 21 22 20 21 22 23 24 25 26 991 | 26 27 28 29 30 31 23 24 25 26 27 28 29 27 28 29 30 992 | 30 31 993 | 994 | October November December 995 | Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa 996 | 1 2 3 1 2 3 4 5 6 7 1 2 3 4 5 997 | 4 5 6 7 8 9 10 8 9 10 11 12 13 14 6 7 8 9 10 11 12 998 | 11 12 13 14 15 16 17 15 16 17 18 19 20 21 13 14 15 16 17 18 19 999 | 18 19 20 21 22 23 24 22 23 24 25 26 27 28 20 21 22 23 24 25 26 1000 | 25 26 27 28 29 30 31 29 30 27 28 29 30 31 1001 | 1002 | ``` 1003 | 1004 | --- 1005 | # date 1006 | 1007 | To print the date, just do: 1008 | 1009 | ``` 1010 | ~ $ date 1011 | Sat Dec 27 20:43:13 PST 2014 1012 | ``` 1013 | 1014 | --- 1015 | You can format the date however you like: 1016 | 1017 | ``` 1018 | ~ $ date +'%Y-%m-%d %H:%M:%S' 1019 | 2014-12-27 20:45:07 1020 | ``` 1021 | 1022 | Check out the manual page (`man date`) for more info about what options are 1023 | available for date strings. 1024 | 1025 | --- 1026 | # fold 1027 | 1028 | Sometimes it's handy to break long lines into 1029 | shorter lines. 1030 | 1031 | --- 1032 | We can use the fold command to break some text at 1033 | 30 characters: 1034 | 1035 | ``` 1036 | ~ $ head -n250 mobydick.txt | tail -n3 | fold -w 30 1037 | can see a whale, for the first 1038 | discoverer has a ducat for hi 1039 | s pains.... 1040 | I was told of a whale taken ne 1041 | ar Shetland, that had above a 1042 | barrel of 1043 | herrings in his belly.... One 1044 | of our harpooneers told me tha 1045 | t he caught 1046 | ``` 1047 | 1048 | --- 1049 | or to break on spaces instead, use `-s`: 1050 | 1051 | ``` 1052 | ~ $ head -n250 mobydick.txt | tail -n3 | fold -sw 30 1053 | can see a whale, for the 1054 | first discoverer has a ducat 1055 | for his pains.... 1056 | I was told of a whale taken 1057 | near Shetland, that had above 1058 | a barrel of 1059 | herrings in his belly.... One 1060 | of our harpooneers told me 1061 | that he caught 1062 | ``` 1063 | 1064 | --- 1065 | # curl 1066 | 1067 | curl is a handy little tool for making HTTP 1068 | requests. 1069 | 1070 | Here's a simple snippet to fetch my most recent 1071 | RSA public key from github, wrapping the output to 1072 | 75 character lines: 1073 | 1074 | --- 1075 | ``` 1076 | ~ $ curl -s https://github.com/substack.keys | tail -n1 | fold -w75 1077 | ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC7wF3cwpH+NVG+qNz0PLjEg9IqaNyXeeITme9 1078 | fksfJx/rTyoFAWW+JrJVKLPNBCe63JYvp3pTvPqJRg/8hEb/+uFlIIzUNhHAUSaS1mmKgnTHbb+ 1079 | 1d8CkFAZiZnDhFOKRkEXh5R9F9htNQVIjyzD3XL/H69mb8YzICItq/9klVTZ66s1thp7r3V5+qE 1080 | hbLB4yH6stXyuj2SZiMS+CJeVBJ8Pf/CCUH66NK2o7l1zliJwme+UJry1RtxWQBfEChj9qe36B/ 1081 | bR3HACtx6ANMdYJsOxZm0znUjn/XJ9jxy22nVJY5otwZNeIZSSyA1lZB2mZRzTTWzPPx62VWdgH 1082 | eQdOmnqBP0YWpxPBSMJwn4kFt6aGImrm7WTU5sHwqqxRgNvcrecxPWgbdLcV+x/OWF5bug3s096 1083 | AWcP4wQI101w7QtI3cc5+JKHSGssuY17jyyNaHttE7GafBu3pbK93YolgNAMyYUHVicgK+uY6o+ 1084 | sH4gcRx+RyQ4OkO7Js49wJi0AXPGhp5QRmIFpua/vVzhMTwMhqW+6luWgfPeAVqn95erc49cY+W 1085 | 2B83ZgaDVSuRfDafVCSjUl+oXG/1KxzP2F/ZhGmNGmBRnF5N4OLHW6/KtVgxCpf3+1bcgye+yiq 1086 | NQuM5/NNWZRw3NJhk0XEppd5Ai4JpvguDLhWZ19/+XEvFj9kwKRMRbxf1M7hWDutAE46sQc9x4M 1087 | 135M/SyuHW9asHBDCJPgD3nBAjYpMV0fQxIbcNiYWF+JsH6NzhRpLnsTNUvsfUcC/FQqX3VD0Xu 1088 | IEoYmKwDesv6PU60pQNEi6p4u+PnFHS/vvRASYLo/4s+99GQDWxqzi0jjYVWheQW9RLnTU+ghud 1089 | A+xPp7CK/tH8/RAutDdk3k0HdsNTsjHFN/HvM23UIHOpuY07yohayQididHt023IAZdys6m2daQ 1090 | RUKXM8cfaFdQqoj/vaby7pxBPWzO6tuXy1tI6gQ+nolZaXQfXUBHF1uBXo1UQI0dp8J5tCppty6 1091 | NvXmvv90PBGVXOlplyhXB9q0JXBInidATeT8zlgM4Iq1X6ZVlXN2OIU5CiWVA1NYmf05709e6SK 1092 | P0kK2oh19gA+qg1oPOw0WTpZGKz/9NCCw2ywK2/yNJRWuIbSE4RAv6N8v7qtPObwAU5Lohj8oQV 1093 | yC/bbLF6VuVJo6V/nfvP+EJKtsXlBBPBzdsmV1hikkGLJx7Up1s7WTZCwSeSGFPXCe7RdElz2mQ 1094 | YB6dwEbhaGl48MhuiIeER7KZqzQFOu74G0u5tyyCUeEc90BkeUcf/EhrxfS8R9ZRJ9ce7IpYQ4+ 1095 | 9wTBKFzVc1HinCSUwJTu7m+UHLaaNbK+WCIF+2fFvM1IJmTh2pWSMb 1096 | ``` 1097 | 1098 | --- 1099 | # grep 1100 | 1101 | You can search for patterns in files or from stdin 1102 | with the `grep` command. 1103 | 1104 | The first argument is the pattern to search for as 1105 | a regular expression. 1106 | 1107 | Regular expressions are a language for pattern 1108 | matching. 1109 | 1110 | --- 1111 | Here we can search for all lines matching 1112 | "whaling" or "fishing": 1113 | 1114 | ``` 1115 | ~ $ grep -iE '(whal|fish)ing' mobydick.txt | tail -n5 1116 | Equatorial fishing-ground, and in the deep darkness that goes before the 1117 | the whaling season? See, Flask, only see how pale he looks--pale in the 1118 | preliminary cruise, Ahab,--all other whaling waters swept--seemed to 1119 | fixed upon her broad beams, called shears, which, in some whaling-ships, 1120 | years of continual whaling! forty years of privation, and peril, and 1121 | ``` 1122 | 1123 | Check out the other workshop about regular 1124 | expressions to learn more! 1125 | 1126 | --- 1127 | # backticks 1128 | 1129 | Sometimes it's useful to include the output of a 1130 | program in the arguments list of another. 1131 | 1132 | For example with the date command we can print the 1133 | current year: 1134 | 1135 | ``` 1136 | date +%Y 1137 | ``` 1138 | 1139 | --- 1140 | and we can use this value in a message with echo: 1141 | 1142 | ``` 1143 | ~ $ echo Greetings from the year `date +%Y`. 1144 | Greetings from the year 2014. 1145 | ``` 1146 | 1147 | --- 1148 | # arithmetic 1149 | 1150 | With `$((...))` expressions, you can do simple 1151 | arithmetic on the command line! 1152 | 1153 | ``` 1154 | ~ $ echo $((4*5+1)) 1155 | 21 1156 | ``` 1157 | 1158 | I wouldn't go overboard with this feature, but 1159 | it's handy sometimes. 1160 | 1161 | --- 1162 | 1163 | # environment variables 1164 | 1165 | Environment variables are defined by the shell and 1166 | shell scripts. 1167 | 1168 | To list the current environment variables, type 1169 | `export`: 1170 | 1171 | --- 1172 | ``` 1173 | ~ $ export 1174 | declare -x DISPLAY=":0" 1175 | declare -x HOME="/home/substack" 1176 | declare -x HUSHLOGIN="FALSE" 1177 | declare -x LANG="en_US.UTF-8" 1178 | declare -x LD_LIBRARY_PATH="/home/substack/prefix/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:/usr/lib:/lib64:/lib" 1179 | declare -x LIBGL_DRIVERS_PATH="/usr/lib/i386-linux-gnu/dri:/usr/lib/x86_64-linux-gnu/dri" 1180 | declare -x LOGNAME="substack" 1181 | declare -x MAIL="/var/mail/substack" 1182 | declare -x OLDPWD="/home/substack/projects/workshops" 1183 | declare -x PATH="/home/substack/prefix/bin:/usr/local/bin:/usr/bin:/bin:/sbin:/usr/sbin:/usr/local/games:/usr/games" 1184 | declare -x PREFIX="/home/substack/prefix" 1185 | declare -x PWD="/home/substack" 1186 | declare -x ROXTERM_ID="0x43962f0" 1187 | declare -x ROXTERM_NUM="15" 1188 | declare -x ROXTERM_PID="2521" 1189 | declare -x SHELL="/bin/bash" 1190 | declare -x SHLVL="3" 1191 | declare -x TERM="xterm" 1192 | declare -x USER="substack" 1193 | declare -x WINDOWID="8684328" 1194 | declare -x WINDOWPATH="7" 1195 | declare -x XAUTHORITY="/home/substack/.Xauthority" 1196 | ``` 1197 | 1198 | --- 1199 | You can use any environment variable by refering to its `$NAME`. 1200 | 1201 | For example to print the value of `$HOME` do: 1202 | 1203 | ``` 1204 | ~ $ echo $HOME 1205 | /home/substack 1206 | ``` 1207 | 1208 | --- 1209 | You can use environment variables in any command: 1210 | 1211 | ``` 1212 | ~ $ ls /home/$USER 1213 | doc media notes.txt projects 1214 | ``` 1215 | 1216 | --- 1217 | To define your own environment variable, just put 1218 | its name followed by an equal sign (with no 1219 | spaces) followed by its value: 1220 | 1221 | ``` 1222 | ~ $ ANIMAL=cats 1223 | ~ $ echo $ANIMAL 1224 | cats 1225 | ``` 1226 | 1227 | --- 1228 | Environment variables are almost always 1229 | capitalized to distinguish them from variables in 1230 | shell scripts but lower-case variables work too. 1231 | 1232 | --- 1233 | # quotes 1234 | 1235 | If you want to use characters like `<` or `>` in 1236 | the arguments to a program, you will need to use 1237 | quotes so that the shell doesn't try to interpret 1238 | them. 1239 | 1240 | For example, to echo the string `wow` we 1241 | can use single quotes: 1242 | 1243 | ``` 1244 | ~ $ echo 'wow' 1245 | wow 1246 | ``` 1247 | 1248 | --- 1249 | Double quotes are similar but environment variables and backticks will be 1250 | interpolated in-place (replaced with their value): 1251 | 1252 | ``` 1253 | ~ $ echo "There's no place like $HOME." 1254 | There's no place like /home/substack. 1255 | ~ $ echo "So long `date +%Y`..." 1256 | So long 2014... 1257 | ~ $ echo "So long `date +%Y`... next stop $((`date +%Y`+1))"'!' 1258 | So long 2014... next stop 2015! 1259 | ``` 1260 | 1261 | --- 1262 | You will also need to use quotes if one of the 1263 | arguments you want to give has a whitespace 1264 | character in it, because whitespace is otherwise 1265 | used to split arguments. 1266 | 1267 | --- 1268 | 1269 | # scripts 1270 | 1271 | Whenever you find yourself typing the same 1272 | sequence of commands several times, consider 1273 | making a script! 1274 | 1275 | Just put the commands you would normally type into 1276 | a file and add `#!/bin/bash` to the top of the 1277 | file: 1278 | 1279 | ``` sh 1280 | #!/bin/bash 1281 | mkdir wow 1282 | cd wow 1283 | echo "yay" > zing.txt 1284 | ``` 1285 | 1286 | --- 1287 | Now make your script file executable: 1288 | 1289 | ~ $ chmod +x yourscript.sh 1290 | 1291 | And now you can do: 1292 | 1293 | ~ $ ./yourscript.sh 1294 | 1295 | to run the commands from your file! 1296 | 1297 | --- 1298 | # script arguments 1299 | 1300 | When you execute a script with arguments on the 1301 | command-line, special environment variables `$1`, 1302 | `$2`, `$3`... will be defined for each argument. 1303 | 1304 | For example, if our script is: 1305 | 1306 | ``` sh 1307 | #!/bin/bash 1308 | echo first=$1 1309 | echo second=$2 1310 | ``` 1311 | 1312 | --- 1313 | Then we print out the first and second arguments: 1314 | 1315 | ``` 1316 | ~ $ ./yourscript.sh beep boop 1317 | first=beep 1318 | second=boop 1319 | ``` 1320 | 1321 | --- 1322 | There is a special variable `$*` that contains all 1323 | the arguments separated by spaces. For a script 1324 | of: 1325 | 1326 | ``` sh 1327 | #!/bin/bash 1328 | echo The arguments are: $* 1329 | ``` 1330 | 1331 | --- 1332 | And now we can get at all the arguments in one place: 1333 | 1334 | ``` 1335 | ~ $ ./args.sh cats dogs ducks lizards 1336 | The arguments are: cats dogs ducks lizards 1337 | ``` 1338 | 1339 | --- 1340 | # $PATH 1341 | 1342 | There is a special environment variable called `$PATH`: 1343 | 1344 | ``` 1345 | ~ $ echo $PATH 1346 | /usr/local/bin:/usr/bin:/bin:/sbin:/usr/sbin:/usr/local/games:/usr/games 1347 | ``` 1348 | 1349 | This variable contains a list of places separated 1350 | by `:` that bash will look when you type a 1351 | command. 1352 | 1353 | --- 1354 | If you put an executable file in one of the 1355 | directories in your `$PATH`, you can make your own 1356 | commands without needing to specify a relative or 1357 | absolute path! 1358 | 1359 | `/usr/local/bin` is the customary place to put 1360 | system-specific scripts that are not managed by 1361 | your system distribution. If you do: 1362 | 1363 | ``` 1364 | ~ $ sudo cp yourscript.sh /usr/local/bin 1365 | ``` 1366 | 1367 | --- 1368 | Then you'll be able to type `yourscript.sh` from 1369 | any directory on the command-line! 1370 | 1371 | You can rename that command that you type by 1372 | renaming the file: 1373 | 1374 | ``` 1375 | ~ $ sudo mv /usr/local/bin/{yourscript.sh,whatever} 1376 | ``` 1377 | 1378 | and now the command is called `whatever`. 1379 | 1380 | --- 1381 | # ~/.bashrc 1382 | 1383 | There is a special bash script called `~/.bashrc` 1384 | that runs whenever you start bash. You can edit 1385 | this file to set up aliases, environment 1386 | variables, and run commands when you start a new 1387 | terminal session. 1388 | 1389 | --- 1390 | At the bottom of your `~/.bashrc` file, try adding 1391 | a command: 1392 | 1393 | ``` 1394 | echo Greetings $USER. Nice to see you again. 1395 | ``` 1396 | 1397 | Now open a new terminal and you should see a friendly new message! 1398 | 1399 | --- 1400 | # permissions 1401 | 1402 | Each file on a UNIX system belongs to a user and a 1403 | group. 1404 | 1405 | users are accounts on the system, like the one you 1406 | log in with. groups are collections of users. 1407 | 1408 | --- 1409 | To see what groups you belong to, just type 1410 | `groups`: 1411 | 1412 | ``` 1413 | ~ $ groups 1414 | substack cdrom floppy audio dip video plugdev 1415 | ``` 1416 | 1417 | --- 1418 | To inspect the permissions on a file, use `ls -l`: 1419 | 1420 | ``` 1421 | ~/doc $ ls -l b.txt 1422 | -rw-r--r-- 1 substack whatever 14 Dec 28 00:29 b.txt 1423 | ``` 1424 | 1425 | --- 1426 | Here we can see that the file `b.txt` is owned by 1427 | the user `substack` and the group `whatever`. 1428 | There's also this part on the left: 1429 | 1430 | ``` 1431 | -rw-r--r-- 1432 | ``` 1433 | 1434 | This string describes the permissions of the file. 1435 | 1436 | --- 1437 | The first character is reserved for some fancy 1438 | uses, but after that there are 3 groups of 3 1439 | characters: 1440 | 1441 | ``` 1442 | rwxrwxrwx 1443 | ``` 1444 | 1445 | --- 1446 | Each character describes a permission: (r)ead, 1447 | (w)rite, and e(x)ecute. A `-` in place of one of 1448 | those letters means the permission is not 1449 | available. 1450 | 1451 | If the e(x)ecute bit is enabled on a file for a 1452 | user, it means the user can execute the file. 1453 | 1454 | If the e(x)ecute bit is enabled on a directory for 1455 | a user, it means the user can list the files in 1456 | that directory. 1457 | 1458 | --- 1459 | * The first `rwx` says what the owner can do. 1460 | * The second `rwx` says what users in the group can do. 1461 | * The third `rwx` says what everyone else can do. 1462 | 1463 | These three categories are called user (u), 1464 | group (g), and other (o). 1465 | 1466 | --- 1467 | # chmod 1468 | 1469 | To change the permissions on a file, first figure 1470 | out which capabilities you want to grant or revoke 1471 | (rwx) from which categories of users (ugo). 1472 | 1473 | --- 1474 | To allow the owner of a file to execute a script you can do: 1475 | 1476 | ~ $ chmod u+x script.sh 1477 | 1478 | which is the same as: 1479 | 1480 | ~ $ chmod +x script.sh 1481 | 1482 | because the `u` is implied if not specified. 1483 | 1484 | --- 1485 | You can also revoke permissions with a `-`. To 1486 | make it so that other users can't write to a 1487 | file: 1488 | 1489 | ~ $ chmod o-w wow.txt 1490 | 1491 | --- 1492 | You can grant and revoke permissions at the same 1493 | time. Here we're adding read and execute 1494 | permissions to the user while simultaneously 1495 | revoking read and write from the group: 1496 | 1497 | ~ $ chmod u+rxg-rw status.sh 1498 | 1499 | You can change the owner of a file with `chown` 1500 | and the group with `chgrp`. 1501 | 1502 | --- 1503 | # job control 1504 | 1505 | Bash is built to handle multiple programs running 1506 | in parallel. 1507 | 1508 | --- 1509 | # time cat 1510 | 1511 | Type `time cat` and then hit ctrl-c before one 1512 | second, as close as possible without going over: 1513 | 1514 | $ time cat 1515 | ^C 1516 | 1517 | real 0m0.920s 1518 | user 0m0.004s 1519 | sys 0m0.000s 1520 | 1521 | --- 1522 | # ctrl-c 1523 | 1524 | Terminate a process in the foreground. 1525 | 1526 | --- 1527 | # ctrl-z 1528 | 1529 | Put a process in the background. 1530 | 1531 | --- 1532 | # fg JOB 1533 | 1534 | Move a process from the background to the 1535 | foreground by its JOB. 1536 | 1537 | ~ $ cat 1538 | ^Z 1539 | [1]+ Stopped cat 1540 | ~ $ echo wow 1541 | wow 1542 | ~ $ fg %1 1543 | cat 1544 | cool 1545 | cool 1546 | 1547 | --- 1548 | # job syntax 1549 | 1550 | When you background a process with ctrl-z, the 1551 | shell prints a message with `[N]`. `N` is the job 1552 | id. Use `%N` to refer to a particular job or: 1553 | 1554 | * `%%` - the most recent job 1555 | 1556 | --- 1557 | # & 1558 | 1559 | Another way to background a process is to use `&`: 1560 | 1561 | $ ~ node & 1562 | [1] 29877 1563 | 1564 | The job id of `node` is 1 and the process id is 1565 | 29877. Job ids are local to a shell session, but 1566 | process ids are global across the entire system. 1567 | 1568 | --- 1569 | ~ $ perl & 1570 | [1] 29870 1571 | ~ $ pgrep perl 1572 | 29870 1573 | ~ $ kill %1 1574 | [1]+ Terminated perl 1575 | 1576 | --- 1577 | # pgrep 1578 | 1579 | Search for a process by its name. 1580 | 1581 | --- 1582 | # kill ID 1583 | 1584 | Kill a process by its process or job id. 1585 | 1586 | --- 1587 | # screen 1588 | 1589 | You can use screen to run command-line programs and keep 1590 | them running, even when you go away. 1591 | 1592 | --- 1593 | # install screen 1594 | 1595 | $ sudo apt-get install screen 1596 | 1597 | --- 1598 | # create a new named screen 1599 | 1600 | $ screen -S website 1601 | 1602 | --- 1603 | # list screens 1604 | 1605 | $ screen -list 1606 | 1607 | --- 1608 | # connect to a screen 1609 | 1610 | $ screen -x website 1611 | 1612 | --- 1613 | # detach from a screen 1614 | 1615 | From inside of a screen, press CTRL+A then `d`. 1616 | 1617 | --- 1618 | # create a new window inside screen 1619 | 1620 | CTRL+A c 1621 | 1622 | --- 1623 | # go to the next window 1624 | 1625 | CTRL+A n 1626 | 1627 | --- 1628 | # go to the previous window 1629 | 1630 | CTRL+A p 1631 | 1632 | --- 1633 | # close a window 1634 | 1635 | Just type `exit` to close a window. 1636 | 1637 | --- 1638 | # irc from the command-line 1639 | 1640 | Install irssi: 1641 | 1642 | $ sudo apt-get install irssi 1643 | 1644 | Then create a screen for irc: 1645 | 1646 | $ screen -S irc 1647 | 1648 | --- 1649 | Then in a screen, you can run `irssi` to use irc from the 1650 | command line. 1651 | 1652 | * `/nick robowizard` - set your nickname on the server 1653 | * `/connect irc.freenode.net` - connect to irc.freenode.net 1654 | * `/join #cyberwizard` - join the channel called cyberwizard 1655 | * ESC+N or `/win N` - to jump to the window at number N 1656 | 1657 | Once you're in a channel, type text like usual. 1658 | `CTRL+A d` to detach and `screen -x irc` to resume. 1659 | 1660 | --- 1661 | # run a web server 1662 | 1663 | Make a web server.js: 1664 | 1665 | ``` js 1666 | var http = require('http'); 1667 | var server = http.createServer(function (req, res) { 1668 | res.end("YOU'RE A WIZARD.\n"); 1669 | }); 1670 | server.listen(8000); 1671 | ``` 1672 | 1673 | now run your server with node from inside a screen: 1674 | 1675 | ``` 1676 | $ node server.js 1677 | ``` 1678 | 1679 | then detach the screen with CTRL+A d. 1680 | 1681 | -------------------------------------------------------------------------------- /day1/vim.md: -------------------------------------------------------------------------------- 1 | # vim 2 | 3 | vim is a popular command-line text editor 4 | 5 | --- 6 | # vim cheat sheet 7 | 8 | Keep this handy as you experiment with vim: 9 | 10 | http://www.fprintf.net/vimCheatSheet.html 11 | 12 | Here is another guide that covers the commands 13 | incrementally: 14 | 15 | http://yannesposito.com/Scratch/en/blog/Learn-Vim-Progressively/ 16 | 17 | --- 18 | # why vim? 19 | 20 | * edit files on a remote server over ssh 21 | * works without a graphical desktop environment 22 | * many programming-specific features 23 | * really fast editing 24 | 25 | --- 26 | # why vim - remote editing 27 | 28 | When you log in over ssh to administer a server, 29 | all you've got is a command-line interface. 30 | 31 | If you are comfortable with vim, you can work on 32 | the remote system with the same ease and 33 | familiarity as your local environment. 34 | 35 | --- 36 | # why vim - graphics not required 37 | 38 | Not all systems have graphical environments! 39 | 40 | What do you do if the graphical environment on 41 | your computer stops working? 42 | 43 | What if you want to configure a device that 44 | doesn't have a graphics card? 45 | 46 | --- 47 | # why vim - programming-specific features 48 | 49 | vim is very carefully tuned to be effective for 50 | programming. 51 | 52 | * easily change the indentation on blocks of text 53 | * syntax highlighting for many programming languages 54 | * fluid interface with the system shell 55 | 56 | --- 57 | # why vim - really fast editing 58 | 59 | vim is designed from the ground-up to be very fast 60 | to use once you've learned its terse commands 61 | 62 | Many of the commands are designed to keep your 63 | fingers on the home row of the keyboard so that 64 | you can drift seemlessly between editing and 65 | typing. 66 | 67 | --- 68 | # alternatives 69 | 70 | Here are some other command-line text editors: 71 | 72 | * nano 73 | * emacs 74 | * vi 75 | 76 | nano is much easier to learn than vim because it 77 | doesn't have many features. 78 | 79 | emacs has a huge number of features and is very 80 | configurable 81 | 82 | vi is a precursor to vim from 1983. vi's features 83 | are a subset of vim, and vi tends to already be 84 | installed on many systems. 85 | 86 | --- 87 | # interface 88 | 89 | Unlike many command-line programs, vim is 90 | interactive. 91 | 92 | vim uses ANSI codes to control a cursor and 93 | position blocks of text on the screen. 94 | 95 | --- 96 | # ansi codes 97 | 98 | ANSI codes are special instructions that your 99 | terminal interprets and renders. 100 | 101 | ANSI codes can: 102 | 103 | * move the text cursor around 104 | * change colors 105 | * set modes 106 | 107 | --- 108 | # ansi codes - colors! 109 | 110 | Try this command: 111 | 112 | ``` 113 | $ echo -e '\x1b[38;5;44mwow' 114 | ``` 115 | 116 | also try changing "44" to some other values. 117 | 118 | Try stacking multiple color codes: 119 | 120 | ``` 121 | $ echo -e '\x1b[38;5;44mso \x1b[38;5;33mcool' 122 | ``` 123 | 124 | --- 125 | brighter version: 126 | 127 | ``` 128 | $ echo -e '\x1b[1m\x1b[38;5;44mso\x1b[38;5;1mcool' 129 | ``` 130 | 131 | Play around! Here is a list of some codes: 132 | https://en.wikipedia.org/wiki/ANSI_escape_code#CSI_codes 133 | 134 | --- 135 | # ansi codes 136 | 137 | Applications like vim make heavy use of ansi codes. 138 | 139 | --- 140 | # and now: let's learn vim! 141 | 142 | First: type `vim` 143 | 144 | If that doesn't work, install vim. 145 | 146 | On a debian or ubuntu system do: 147 | 148 | ``` 149 | sudo apt-get install vim vim-common 150 | ``` 151 | 152 | to get vim plus extras like syntax hilighting. 153 | 154 | --- 155 | # now we're in vim! 156 | 157 | Type `i` to go into insert mode. 158 | 159 | Now you can type normally. 160 | 161 | --- 162 | # saving and quitting 163 | 164 | Hit `esc` to get out of insert mode. 165 | 166 | Now type: 167 | 168 | :w foo.txt 169 | 170 | to save your file as `foo.txt`. 171 | 172 | --- 173 | You can go back into insert mode by typing `i` 174 | again or you can quit by typing: 175 | 176 | :q 177 | 178 | Once you've quit, you can open your file back up 179 | again by running: 180 | 181 | $ vim foo.txt 182 | 183 | --- 184 | or you can do just `vim` and then from command 185 | mode do: 186 | 187 | :o foo.txt 188 | 189 | If you've opened a file already, you can just type 190 | `:w` to save the file, you don't need to type its 191 | name every time. 192 | 193 | --- 194 | # modes 195 | 196 | Time to recap what just happened! 197 | 198 | The first thing you'll notice is how we used 2 199 | modes: command mode and insert mode. 200 | 201 | If you're in command mode, press `i` to go into 202 | insert mode. 203 | 204 | If you're in insert mode, press `esc` to go into 205 | command mode. 206 | 207 | --- 208 | If it says `-- INSERT --` at the bottom left of 209 | your terminal, you're in insert mode! 210 | 211 | Otherwise you're in command mode. 212 | 213 | --- 214 | # vim is a language 215 | 216 | Next, let's combine some commands. 217 | 218 | Try `:wq` to save and then quit. 219 | 220 | Try `:q!` to quit without saving. 221 | 222 | --- 223 | # moving around - hjkl 224 | 225 | In insert mode, the arrow keys do work, but you 226 | should practice not using them! 227 | 228 | Instead, in command mode: 229 | 230 | * h - moves left one character 231 | * j - moves down one line 232 | * k - moves up one line 233 | * l - moves right one character 234 | 235 | --- 236 | # hjkl elsewhere... 237 | 238 | * the `less` command uses j and k for up and down 239 | * https://twitter.com - j and k 240 | * many tiling window managers such as xmonad 241 | 242 | --- 243 | # moving around - even more! 244 | 245 | You can move all kinds of places quickly in 246 | command mode: 247 | 248 | * ^ or 0 - move to the start of the current line 249 | * $ - move to the end of the current line 250 | * gg - jump to the beginning of the file 251 | * G - jump to the end of the file 252 | 253 | --- 254 | # delete 255 | 256 | There are so many ways to delete! 257 | 258 | * x - delete the character under the cursor 259 | * dd - delete the current line 260 | * d$ or D - delete from the cursor to the end of 261 | the current line 262 | * d0 or d^ - delete from the cursor to the start 263 | of the current line 264 | 265 | --- 266 | You'll notice that we've already seen `0` and `$` 267 | before! You can repurpose each of the moving 268 | around commands to delete text. 269 | These all work: 270 | 271 | * dG - delete from the current position to the end 272 | of the file 273 | * dgg - delete from the current position to the 274 | start of the file 275 | * dj - delete the current line and the line below 276 | * dk - delete the current line and the line above 277 | * 2dd, 3dd etc - delete the next N lines 278 | 279 | --- 280 | Even `dl` and `dh` work! 281 | 282 | Remember that vim is a language! 283 | 284 | --- 285 | # searching 286 | 287 | You can search for text using regular expressions. 288 | 289 | * /PATTERN - search forward for PATTERN 290 | * ?PATTERN - search backward for PATTERN 291 | 292 | Press: 293 | 294 | * n - jump to the next match 295 | * N - jump to the previous match 296 | 297 | --- 298 | PATTERN is a regular expression, but you can just 299 | treat it as an ordinary text match for the most 300 | part. 301 | 302 | You can combine searching with deleting too: 303 | 304 | * d/PATTERN - delete to the next match of PATTERN 305 | * d?PATTERN - delete to the previous match of PATTERN 306 | * dn - delete to the next already matched pattern 307 | * dN - delete to the previous already matched pattern 308 | 309 | --- 310 | # jumping 311 | 312 | You can also skip ahead to individual characters 313 | in a simple way on the current line: 314 | 315 | * f + CHAR - search forward on the current line to CHAR 316 | * t + CHAR - search forward on the current line to 317 | the character before CHAR 318 | * F + CHAR - search backward on the current line to CHAR 319 | * T + CHAR - search backward on the current line 320 | to the character after CHAR 321 | 322 | --- 323 | These are very useful in combination with the 324 | delete operators! They combine as you might 325 | expect: 326 | 327 | * df + CHAR - delete forward on the current line to CHAR 328 | * dt + CHAR - delete forward on the current line 329 | to the character before CHAR 330 | * dF + CHAR - delete backward on the current line to CHAR 331 | * dT + CHAR - delete backward on the current line 332 | to the character after CHAR 333 | 334 | --- 335 | # search and replace 336 | 337 | :s/PATTERN/REPLACEMENT/FLAGS 338 | 339 | Try these on a line with the string cats: 340 | 341 | :s/cat/dog/ 342 | :s/cat/dog/g 343 | :s/cat/dog/i 344 | 345 | --- 346 | # replace everything 347 | 348 | :%s/cat/dog/ig 349 | 350 | Replaces "cat" with dog everywhere in the entire 351 | file, case insensitively. 352 | 353 | --- 354 | # regex flags 355 | 356 | * i - case insensitive 357 | * g - global replace (per line) 358 | 359 | --- 360 | # visual select 361 | 362 | Press `v` to go into visual select mode. 363 | Move the cursor around to select text. 364 | 365 | Once you've selected a block, you can press: 366 | 367 | * `y` - "yank" the text into the paste buffer 368 | * `x` or `d` - delete the selected text 369 | * `>>` - indent the text right by shiftwidth 370 | * `<<` - indent the text left by shiftwidth 371 | 372 | --- 373 | # paste 374 | 375 | Once you've populated the paste buffer by yanking 376 | or deleting, press `p` to paste. 377 | 378 | --- 379 | # visual modes 380 | 381 | * `v` - select by characters 382 | * `V` - select by lines 383 | * ctrl-`v` - select in a block 384 | 385 | --- 386 | # more insert modes 387 | 388 | There are more ways to insert mode than just `i`: 389 | 390 | * `o` - go into insert mode, inserting a new line 391 | below the current line 392 | * `O` - go into insert mode, inserting a new line 393 | * above the current line 394 | * `a` - go into insert mode at one character to 395 | the right 396 | * `A` - go into insert mode at the end of the 397 | current line 398 | 399 | --- 400 | # fancy odds and ends 401 | 402 | * `J` - move the next line to the end of the 403 | current line 404 | * (backtick)+`.` - jump to the last edit 405 | 406 | --- 407 | # insert a file 408 | 409 | You can insert a file at the cursor position with: 410 | 411 | ``` 412 | :r otherfile.txt 413 | ``` 414 | 415 | --- 416 | # insert with a command in place 417 | 418 | You can insert the output of a command at the 419 | cursor position with `:r!`. 420 | 421 | For example, to insert the output of the `pwd` 422 | command: 423 | 424 | ``` 425 | :r!pwd 426 | ``` 427 | 428 | --- 429 | # .vimrc 430 | 431 | * autoindento 432 | * expandtab 433 | * tabstop 434 | * shiftwidth (sw) 435 | 436 | my vimrc: 437 | 438 | https://gist.github.com/substack/7745bb6ff9ad58d4805d 439 | 440 | --- 441 | # set -o vi 442 | 443 | You can use vi shorthand in bash too! 444 | 445 | Just do: 446 | 447 | $ set -o vi 448 | 449 | now press esc and hjkl your way around! 450 | 451 | ---- 452 | # escape is too far away! 453 | 454 | It's common for vim users to remap their 455 | keyboards. 456 | 457 | One common thing to do is swap the caps lock key 458 | with the escape key because escape is such a 459 | common key in vim. 460 | 461 | --- 462 | # xmodmap for escape 463 | 464 | In linux you can use xmodmap to remap your keys. 465 | 466 | Save this text to a file called `.xmodmap` in your 467 | home directory: 468 | 469 | remove Lock = Caps_Lock 470 | keysym Escape = Caps_Lock 471 | keysym Caps_Lock = Escape 472 | add Lock = Caps_Lock 473 | 474 | --- 475 | now run `xmodmap ~/.xmodmap` to enable your 476 | swapped keys. 477 | 478 | Add this command to your login scripts so that 479 | each time you log in you won't need to remember to 480 | run the command every time you log in. 481 | 482 | --- 483 | # built-in escape alternative 484 | 485 | You can also use ctrl+`]` to get out of insert mode. 486 | 487 | --- 488 | # EOF 489 | -------------------------------------------------------------------------------- /day2.md: -------------------------------------------------------------------------------- 1 | # day 2: networking and streams 2 | 3 | Learn about networking and node.js streams. 4 | 5 | Streams let you glue together sources and sinks of I/O with backpressure to 6 | produce effective data pipelines for processing data. How do streams relate to 7 | network protocols such as TCP, HTTP, and websockets? 8 | 9 | Join James Halliday (substack) as we dive into networking with streams. You 10 | will: 11 | 12 | * Use curl and netcat to send and receive network requests. 13 | * Learn the stream types: readable, writable, transform, duplex. 14 | * Use stream modules from npm to build streaming pipelines. 15 | * Learn about text-based network protocols such as http, irc, and email. 16 | * Write tcp, http, and websocket servers and clients using node.js. 17 | * Build symmetric streaming protocols. 18 | 19 | ## schedule 20 | 21 | * 08:30 - system check, hello 22 | * 09:00 - TCP, UDP, and netcat 23 | * 09:30 - text-based network protocols 24 | * 10:00 - http servers, headers, and curl 25 | * 11:00 - streaming interface basics 26 | * 11:30 - basic stream modules 27 | * 12:00 - lunch etc 28 | * 13:00 - streaming transports galore: websockets, webrtc, p2p 29 | * 13:30 - advanced streaming modules 30 | * 14:00 - implementing core streams 31 | * 15:00 - rpc, multiplexing, and symmetric protocols 32 | * 16:00 - closing notes and wrap up 33 | 34 | ## who is this for? 35 | 36 | This workshop is for people who want to learn more about streams in node.js and 37 | the network protocols and fundamentals that streams sit on top of. 38 | 39 | If you know some node.js or frontend js but want to dive deeper into how a 40 | server and client work and how to write glue around IO, this workshop is for 41 | you! 42 | 43 | ## prerequisites 44 | 45 | You should have some familiarity with javascript and the command-line. 46 | 47 | -------------------------------------------------------------------------------- /day2/networks.md: -------------------------------------------------------------------------------- 1 | # networking 2 | 3 | servers, clients, and protocols! 4 | 5 | --- 6 | # servers and clients 7 | 8 | Any networked computer can be a server, 9 | any networked computer can be a client! 10 | 11 | --- 12 | # packets 13 | 14 | tiny chunks of data 15 | 16 | For example, if we have a payload: 17 | 18 | ``` 19 | This is the message we want to send. It contains 20 | information. XXXX YYyY ZZzZ QRSTUV ABCDEFG BLAH BLAH. 21 | ``` 22 | 23 | it might get broken up into multiple packets: 24 | 25 | This is the message we want to send. It conta 26 | 27 | and: 28 | 29 | ins 30 | information. XXXX YYyY ZZzZ QRSTUV ABCDEFG BLAH BLAH. 31 | 32 | --- 33 | # tcp vs udp 34 | 35 | TCP - reliable transport: if a packet is not acknowledged 36 | (ACK) on the other end, it gets resent 37 | 38 | UDP - unreliable transport: packets are sent but there is no 39 | confirmation that the packet was received at the other end 40 | 41 | --- 42 | # tcp vs udp uses 43 | 44 | UDP - sometimes used for streaming video and audio, some games 45 | 46 | TCP - everything else 47 | 48 | --- 49 | # protocols 50 | 51 | the language that computer programs speak to each other 52 | 53 | Examples of network protocols: 54 | 55 | * HTTP - browse web pages 56 | * HTTPS - browse web pages with encryption 57 | * SMTP - send and receive emails 58 | * IMAP, POP3 - load emails from an inbox 59 | * IRC - chat 60 | * FTP - file transfer 61 | * SSH - remote shell over an encrypted connection 62 | * SSL - low-level secure data transfer (used by HTTPS) 63 | 64 | --- 65 | # ports 66 | 67 | Each computer can have many services. 68 | 69 | A port is a number between 1 and 65535 that 70 | differentiates among the services on a system. 71 | 72 | --- 73 | # customary ports 74 | 75 | Any service can listen on any port, but there are customary 76 | ports for many protocols: 77 | 78 | * 21 - ftp (control port) 79 | * 22 - ssh 80 | * 25 - smtp 81 | * 80 - http 82 | * 443 - https 83 | * 3306 - mysql 84 | * 5432 - postgresql 85 | * 5984 - couchdb 86 | * 6667 - irc 87 | 88 | --- 89 | # port and permissions 90 | 91 | By default, systems can only listen to ports below 1024 as 92 | the root user: 93 | 94 | ``` 95 | $ nc -lp 1024 96 | ^C 97 | $ nc -lp 1023 98 | Can't grab 0.0.0.0:1023 with bind : Permission denied 99 | ``` 100 | 101 | --- 102 | # servers 103 | 104 | Sometimes when people say "server" they mean a computer 105 | program that listens for incoming connections. 106 | 107 | Other times when people say "server" they mean a computer 108 | that is configured to run server programs. 109 | 110 | Any computer can be a server! 111 | 112 | --- 113 | # clients 114 | 115 | Clients are computer programs that connect to servers. 116 | 117 | initiate a connection 118 | 119 | Any computer can be a client! 120 | 121 | --- 122 | # peer to peer 123 | 124 | Aside from servers and clients, there is a third role in 125 | computer networks: peer. 126 | 127 | In a peer to peer network, clients establish connections 128 | directly to other clients. Nodes in the network are 129 | symmetric with no fixed central servers. 130 | 131 | Examples of peer to peer protocols: 132 | 133 | * bittorrent 134 | * webrtc 135 | 136 | --- 137 | # netcat 138 | 139 | netcat can create tcp and udp connections and servers 140 | 141 | ``` 142 | sudo apt-get install netcat 143 | ``` 144 | 145 | With netcat you can speak tcp directly. 146 | 147 | --- 148 | # netcat server and client 149 | 150 | ``` 151 | $ nc -lp 5000 152 | ``` 153 | 154 | or if that doesn't work try: 155 | 156 | ``` 157 | $ nc -l 5000 158 | ``` 159 | 160 | then connect to your server in another terminal: 161 | 162 | ``` 163 | $ nc localhost 5000 164 | ``` 165 | 166 | Type messages in each session and see the messages on the 167 | other session. 168 | 169 | --- 170 | # http 171 | 172 | hypertext transfer protocol 173 | 174 | how web servers and web browsers communicate 175 | 176 | --- 177 | # http verbs 178 | 179 | HTTP requests begin with a VERB. 180 | Here are some things each VERB is used for: 181 | 182 | * GET - fetch a document 183 | * POST - submit a form 184 | * HEAD - fetch metadata about a document 185 | * PUT - upload a file 186 | 187 | --- 188 | # http headers 189 | 190 | Next come the headers. 191 | 192 | Headers have a key followed by a colon followed by a value: 193 | 194 | ``` 195 | key: value 196 | ``` 197 | 198 | --- 199 | # http 200 | 201 | ``` 202 | $ nc google.com 80 203 | GET / HTTP/1.0 204 | Host: google.com 205 | 206 | 207 | ``` 208 | 209 | Make sure to hit `return` twice after `Host: google.com`. 210 | 211 | --- 212 | The server responds with a version and status code 213 | (301 means redirect) followed by some headers. 214 | 215 | The body is separated from the headers by an empty line: 216 | 217 | ``` 218 | HTTP/1.0 301 Moved Permanently 219 | Location: http://www.google.com/ 220 | Content-Type: text/html; charset=UTF-8 221 | Date: Mon, 12 Jan 2015 01:26:19 GMT 222 | Expires: Wed, 11 Feb 2015 01:26:19 GMT 223 | Cache-Control: public, max-age=2592000 224 | Server: gws 225 | Content-Length: 219 226 | X-XSS-Protection: 1; mode=block 227 | X-Frame-Options: SAMEORIGIN 228 | Alternate-Protocol: 80:quic,p=0.02 229 | 230 | 232 | 301 Moved 233 |

301 Moved

234 | The document has moved 235 | here. 236 | 237 | ``` 238 | 239 | --- 240 | ``` 241 | $ nc google.com 80 242 | GET / HTTP/1.0 243 | Host: www.google.com 244 | 245 | HTTP/1.0 200 OK 246 | Date: Mon, 12 Jan 2015 01:26:34 GMT 247 | Expires: -1 248 | Cache-Control: private, max-age=0 249 | Content-Type: text/html; charset=ISO-8859-1 250 | Set-Cookie: PREF=ID=ef742f69ec142ebf:FF=0:TM=1421025994:LM=1421025994:S=F1aTCyHJIJ82Pk1n; expires=Wed, 11-Jan-2017 01:26:34 GMT; path=/; domain=.google.com 251 | Set-Cookie: NID=67=bsMXxsgStx4qF_9eM34aG2sYr_-tJpQsh2IW0CUQx3I2K8-HAfbAm1LKcuHZUMfFwupYqdrthJAN-PguV9ftUtEtr5Tb3NUvJ6zIutDXtEQxb_SDoSpYiHpYrPpkJW1x; expires=Tue, 14-Jul-2015 01:26:34 GMT; path=/; domain=.google.com; HttpOnly 252 | P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info." 253 | Server: gws 254 | X-XSS-Protection: 1; mode=block 255 | X-Frame-Options: SAMEORIGIN 256 | Alternate-Protocol: 80:quic,p=0.02 257 | Accept-Ranges: none 258 | Vary: Accept-Encoding 259 | ``` 260 | --- 261 | # http post 262 | 263 | Forms in html are often delivered with a POST: 264 | 265 | ``` 266 | POST /form HTTP/1.1 267 | Host: localhost 268 | Content-Length: 51 269 | Content-Type: application/x-www-form-urlencoded 270 | 271 | title=whatever&date=1421044443&body=beep%20boop%21 272 | ``` 273 | --- 274 | using this simple node http server, 275 | we can decode the POST body: 276 | 277 | ``` js 278 | var http = require('http') 279 | var parseform = require('body/any') 280 | 281 | var server = http.createServer(function (req, res) { 282 | console.log(req.method, req.url, req.heders) 283 | parseform(req, res, function (err, params) { 284 | console.log(params) 285 | res.end('ok\n') 286 | }) 287 | }) 288 | server.listen(5000) 289 | ``` 290 | --- 291 | When the POST payload arrives and is decoded, we get: 292 | 293 | ``` 294 | $ node server.js 295 | { title: 'whatever', date: '1421044443', body: 'beep boop!\n' } 296 | ``` 297 | --- 298 | and the server responds with: 299 | 300 | ``` 301 | HTTP/1.1 200 OK 302 | Date: Mon, 12 Jan 2015 06:37:51 GMT 303 | Connection: keep-alive 304 | Transfer-Encoding: chunked 305 | 306 | 3 307 | ok 308 | 309 | 0 310 | ``` 311 | --- 312 | # curl 313 | 314 | You can also send http requests with the curl command: 315 | 316 | ``` 317 | $ curl -s http://substack.net 318 | ``` 319 | 320 | issues a GET request to substack.net and prints the body. 321 | 322 | To see just the headers, use `-I`: 323 | 324 | ``` 325 | $ curl -sI http://substack.net 326 | ``` 327 | 328 | The `-s` gets rid of annoying progress output. 329 | 330 | --- 331 | # curl: issuing a POST 332 | 333 | Use `-X` to set the http verb (POST) and `-d` to set 334 | form parameters: 335 | 336 | ``` 337 | $ curl -X POST http://localhost:5000 -d title=whatever \ 338 | -d date=1421044443 -d body='beep boop!' 339 | ``` 340 | 341 | --- 342 | # smtp 343 | 344 | smtp is the protocol used to deliver email messages. 345 | 346 | Here we can send an email from `trump@whitehouse.gov` to 347 | `substack@localhost`. 348 | 349 | The lines that start with a number are messages from the 350 | server. 351 | 352 | --- 353 | ``` 354 | $ nc localhost 25 355 | 220 zzz ESMTP Exim 4.84_2 Tue, 02 May 2017 21:29:36 -0700 356 | helo localhost 357 | 250 zzz Hello localhost [127.0.0.1] 358 | mail from: trump@whitehouse.gov 359 | 250 OK 360 | rcpt to: substack@localhost 361 | 250 Accepted 362 | data 363 | 354 Enter message, ending with "." on a line by itself 364 | Subject: FAKE NEWS 365 | 366 | You're fired. 367 | . 368 | 250 OK id=1d5lvL-00026H-DW 369 | quit 370 | 221 zzz closing connection 371 | ``` 372 | --- 373 | Since this email was sent locally, I can read the message 374 | with the `mail` command: 375 | 376 | ``` 377 | $ mail 378 | Mail version 8.1.2 01/15/2001. Type ? for help. 379 | "/var/mail/substack": 1 message 1 new 380 | >N 1 trump@whitehouse. Tue May 02 21:30 16/491 FAKE NEWS 381 | ``` 382 | --- 383 | Seems legit: 384 | 385 | ``` 386 | & n 387 | Message 1: 388 | From trump@whitehouse.gov Tue May 02 21:30:09 2017 389 | Envelope-to: substack@localhost 390 | Delivery-date: Tue, 02 May 2017 21:30:09 -0700 391 | Subject: FAKE NEWS 392 | From: trump@whitehouse.gov 393 | Date: Tue, 02 May 2017 21:30:05 -0700 394 | 395 | You're fired. 396 | 397 | ``` 398 | --- 399 | # irc 400 | 401 | irc is an ancient text-based chat protocol that is still 402 | very popular among programmers. 403 | 404 | To play around, pick a nick (like whatevz) and hop on a 405 | server (such as irc.freenode.net): 406 | 407 | ``` 408 | $ nc irc.freenode.net 6667 409 | nick whatevz 410 | user whatevz whatevz irc.freenode.net :whatevz 411 | join #cyberwizard 412 | privmsg #cyberwizard :hack the planet! 413 | ``` 414 | 415 | --- 416 | # irc commands 417 | 418 | * nick - identify as a user 419 | * user - also identify as a user 420 | * join - join a channel 421 | * privmsg - send a message to a channel 422 | 423 | --- 424 | # text protocols 425 | 426 | So far, we've seen a number of text protocols: 427 | 428 | * http 429 | * smtp 430 | * irc 431 | 432 | These are nice protocols to implement because you can 433 | inspect the data going over the wire visually and type 434 | requests using the keyboard. 435 | 436 | --- 437 | # binary protocols 438 | 439 | In binary protocols, you can't just type messages with the 440 | keyboard like we've been doing. You've got to write programs 441 | that unpack the incoming bytes and pack outgoing bytes 442 | according to the specification. 443 | 444 | --- 445 | # ssh 446 | 447 | ``` 448 | $ nc substack.net 22 449 | SSH-2.0-OpenSSH_6.0p1 Debian-4+deb7u2 450 | help 451 | Protocol mismatch. 452 | ``` 453 | 454 | Aside from the initial greeting, the rest of the ssh 455 | protocol expects binary. 456 | 457 | --- 458 | Luckily, the ssh command does the work of speaking the 459 | protocol for us: 460 | 461 | ``` 462 | $ ssh substack.net 463 | substack@origin : ~ $ 464 | ``` 465 | 466 | --- 467 | # inspecting protocols 468 | 469 | To inspect protocols, you can capture everything coming out 470 | of and into your wireless or ethernet card using: 471 | 472 | * wireshark for a graphical tool 473 | * tcpdump for a command-line tool 474 | 475 | --- 476 | # tcpdump 477 | 478 | First install tcpdump: 479 | 480 | sudo apt-get install tcpdump 481 | 482 | then do: 483 | 484 | ``` 485 | $ sudo tcpdump -X 486 | ``` 487 | 488 | to see each packet with a hexadecimal representation in the 489 | middle and ascii on the right. 490 | 491 | --- 492 | To filter the output so that we only see HTTP traffic, we 493 | can filter the output to only show traffic on port 80: 494 | 495 | ``` 496 | $ sudo tcpdump 'tcp port 80' -X 497 | ``` 498 | --- 499 | If you look carefully, there are HTTP requests: 500 | 501 | ``` 502 | 23:07:37.655461 IP 10.0.0.2.40646 > 104.131.0.235.http: Flags [P.], seq 1:77, ack 1, win 115, options [nop,nop,TS val 62572874 ecr 2756441271], length 76 503 | 0x0000: 4500 0080 0231 4000 4006 c4d7 0a00 0002 E....1@.@....... 504 | 0x0010: 6883 00eb 9ec6 0050 ece3 cb1c 857a efc5 h......P.....z.. 505 | 0x0020: 8018 0073 15ec 0000 0101 080a 03ba c94a ...s...........J 506 | 0x0030: a44b f4b7 4745 5420 2f20 4854 5450 2f31 .K..GET./.HTTP/1 507 | 0x0040: 2e31 0d0a 5573 6572 2d41 6765 6e74 3a20 .1..User-Agent:. 508 | 0x0050: 6375 726c 2f37 2e32 362e 300d 0a48 6f73 curl/7.26.0..Hos 509 | 0x0060: 743a 2073 7562 7374 6163 6b2e 6e65 740d t:.substack.net. 510 | 0x0070: 0a41 6363 6570 743a 202a 2f2a 0d0a 0d0a .Accept:.*/*.... 511 | ``` 512 | --- 513 | and HTTP responses: 514 | 515 | ``` 516 | 0x0030: 03bb 7d44 4854 5450 2f31 2e31 2032 3030 ..}DHTTP/1.1.200 517 | 0x0040: 204f 4b0d 0a63 6f6e 7465 6e74 2d74 7970 .OK..content-typ 518 | 0x0050: 653a 2074 6578 742f 6874 6d6c 0d0a 4461 e:.text/html..Da 519 | 0x0060: 7465 3a20 4d6f 6e2c 2031 3220 4a61 6e20 te:.Mon,.12.Jan. 520 | 0x0070: 3230 3135 2030 373a 3130 3a32 3420 474d 2015.07:10:24.GM 521 | 0x0080: 540d 0a43 6f6e 6e65 6374 696f 6e3a 206b T..Connection:.k 522 | 0x0090: 6565 702d 616c 6976 650d 0a54 7261 6e73 eep-alive..Trans 523 | ``` 524 | --- 525 | and HTML: 526 | 527 | ``` 528 | 0x01a0: 0d0a 3137 3136 0d0a 3c68 3120 6964 3d22 ..1716..offline.dec 533 | 0x01f0: 656e 7472 616c 697a 6564 2073 696e 676c entralized.singl 534 | 0x0200: 6520 7369 676e 2d6f 6e20 696e 2074 6865 e.sign-on.in.the 535 | 0x0210: 2062 726f 7773 6572 3c2f 6831 3e0a 3c70 .browser.

greetz.txt 73 | $ node greetz.js 74 | beep boop 75 | ``` 76 | 77 | --- 78 | 79 | now let's transform the data before we print it out! 80 | 81 | --- 82 | # fs 83 | 84 | You can chain `.pipe()` calls together just like the `|` 85 | operator in bash: 86 | 87 | ``` js 88 | var fs = require('fs') 89 | 90 | fs.createReadStream('greetz.txt') 91 | .pipe(...) 92 | .pipe(process.stdout) 93 | ``` 94 | 95 | --- 96 | # fs 97 | 98 | ``` js 99 | var fs = require('fs') 100 | var through = require('through2') 101 | 102 | fs.createReadStream('greetz.txt') 103 | .pipe(through(toUpper)) 104 | .pipe(process.stdout) 105 | 106 | function toUpper (buf, enc, next) { 107 | next(null, buf.toString().toUpperCase()) 108 | } 109 | ``` 110 | 111 | --- 112 | # fs 113 | 114 | ``` js 115 | var fs = require('fs'); 116 | var through = require('through2'); 117 | 118 | fs.createReadStream('greetz.txt') 119 | .pipe(through(toUpper)) 120 | .pipe(process.stdout) 121 | 122 | function toUpper (buf, enc, next) { 123 | next(null, buf.toString().toUpperCase()) 124 | } 125 | ``` 126 | 127 | --- 128 | # stdin 129 | 130 | 131 | Instead of reading from a file, we could read from stdin: 132 | 133 | ``` js 134 | var through = require('through2'); 135 | 136 | process.stdin 137 | .pipe(through(toUpper)) 138 | .pipe(process.stdout) 139 | 140 | function toUpper (buf, enc, next) { 141 | next(null, buf.toString().toUpperCase()) 142 | } 143 | ``` 144 | 145 | --- 146 | # through2 147 | 148 | through2 is a module you can install with npm: 149 | 150 | ``` 151 | $ npm install through2 152 | ``` 153 | 154 | --- 155 | 156 | or you can use stream.Transform from node core: 157 | 158 | ``` js 159 | var Transform = require('stream').Transform 160 | var toUpper = new Transform({ 161 | transform: function (buf, enc, next) { 162 | next(null, buf.toString().toUpperCase()) 163 | } 164 | }) 165 | 166 | process.stdin 167 | .pipe(toUpper) 168 | .pipe(process.stdout) 169 | ``` 170 | 171 | --- 172 | # through2 vs stream.Transform 173 | 174 | * `through2(opts={...}, write, end)` 175 | * `new Transform({ transform: write, flush: end, ... })` 176 | 177 | --- 178 | # through(write, end) 179 | 180 | With through there are 2 parameters: `write` and `end`. 181 | Both are optional. 182 | 183 | * `function write (buf, enc, next) {}` 184 | * `function end () {}` 185 | 186 | Call `next()` when you're ready for the next chunk. 187 | If you don't call `next()`, your stream will hang! 188 | 189 | Call `this.push(VALUE)` inside the callback to put VALUE 190 | into the stream's output. 191 | 192 | Use a `VALUE` of `null` to end the stream. 193 | 194 | --- 195 | # through() 196 | 197 | If you don't give through any arguments, these are the 198 | default values for write and end: 199 | 200 | * `function write (buf, enc, next) { this.push(buf); next() }` 201 | * `function end () { this.push(null) }` 202 | 203 | This means that `through()` with no arguments will pass 204 | everything written as input directly through to its output. 205 | 206 | --- 207 | # concat-stream 208 | 209 | `npm install concat-stream` 210 | 211 | concat-stream buffers up all the data in the stream: 212 | 213 | ``` 214 | var concat = require('concat-stream') 215 | process.stdin.pipe(concat(function (body) { 216 | console.log(body.length) 217 | })) 218 | ``` 219 | 220 | You can only write to a concat-stream. You can't read from a 221 | concat-stream. 222 | 223 | Keep in mind that all the data will be in memory. 224 | 225 | --- 226 | # stream types 227 | 228 | There are many kinds of streams. We've seen two types 229 | already: transform (through2) and writable (concat-stream). 230 | 231 | * readable - produces data: you can pipe FROM it 232 | * writable - consumes data: you can pipe TO it 233 | * transform - consumes data, producing transformed data 234 | * duplex - consumes data separately from producing data 235 | 236 | --- 237 | # stream types in code 238 | 239 | * readable: `readable.pipe(A)` 240 | * writable: `A.pipe(writable)` 241 | * transform: `A.pipe(transform).pipe(B)` 242 | * duplex: `A.pipe(duplex).pipe(A)` 243 | 244 | --- 245 | # writable stream methods 246 | 247 | We've seen `.pipe()` which is a method of all readable 248 | streams (readable, transform, and duplex). 249 | 250 | Any stream you can write to (writable, transform, and duplex 251 | streams) has these methods: 252 | 253 | * `.write(buf)` 254 | * `.end()` 255 | * `.end(buf)` 256 | * `.on('finish', function () {})` 257 | * (...).pipe(stream) 258 | 259 | --- 260 | # readable stream methods 261 | 262 | * `stream.pipe(...)` 263 | * `stream.once('end', function () {})` 264 | 265 | you probably won't need to call these very often: 266 | 267 | * `stream.read()` 268 | * `stream.on('readable', function () {})` 269 | 270 | you can let a module or `.pipe()` take care of calling those 271 | 272 | --- 273 | # readable: paused mode 274 | 275 | default behavior with automatic backpressure 276 | 277 | --- 278 | # readable: flowing mode 279 | 280 | data is consumed as soon as chunks are available (no backpressure) 281 | 282 | turn on flowing mode with: 283 | 284 | * `stream.resume()` 285 | * `stream.on('data', function (buf) {})` 286 | 287 | --- 288 | # transform 289 | 290 | readable + writable stream where: 291 | 292 | ``` 293 | input -> transform -> output 294 | ``` 295 | 296 | All the readable AND writable methods are available. 297 | 298 | --- 299 | # duplex 300 | 301 | readable + writable stream where 302 | input is decoupled from output 303 | 304 | like a telephone! 305 | 306 | ``` 307 | input -> duplex 308 | duplex -> output 309 | ``` 310 | 311 | All the readable AND writable methods are available. 312 | 313 | --- 314 | # duplex 315 | 316 | If you see: 317 | 318 | a.pipe(stream).pipe(a) 319 | 320 | then you are dealing with a duplex stream. 321 | 322 | --- 323 | # object streams 324 | 325 | Normally you can only read and write buffers and strings 326 | with streams. However, if you initialize a stream in 327 | `objectMode`, you can use any kind of object (except for 328 | `null`): 329 | 330 | ``` js 331 | var through = require('through2') 332 | var tr = through.obj(function (row, enc, next) { 333 | next(null, (row.n * 1000) + '\n') 334 | }) 335 | tr.pipe(process.stdout) 336 | tr.write({ n: 5 }) 337 | tr.write({ n: 10 }) 338 | tr.write({ n: 3 }) 339 | tr.end() 340 | ``` 341 | 342 | --- 343 | output: 344 | 345 | ``` 346 | 5000 347 | 10000 348 | 3000 349 | ``` 350 | 351 | --- 352 | # core streams in node 353 | 354 | many of the APIs in node core provide stream interfaces: 355 | 356 | * fs.createReadStream() 357 | * fs.createWriteStream() 358 | * process.stdin, process.stderr 359 | * ps.stdin, ps.stdout, ps.stderr 360 | * net.connect(), tls.connect() 361 | * net.createServer(function (stream) {}) 362 | * tls.createServer(opts, function (stream) {}) 363 | 364 | ... 365 | 366 | --- 367 | # http core streams 368 | 369 | ``` js 370 | // req: readable, res: writable 371 | http.createServer(function (req, res) {}) 372 | 373 | // req: writable, res: readable 374 | var req = http.request(opts, function (res) {}) 375 | ``` 376 | 377 | --- 378 | # http 379 | 380 | ``` js 381 | var http = require('http') 382 | var server = http.createServer(function (req, res) { 383 | req.pipe(process.stdout) 384 | res.end('hello thar!\n') 385 | }) 386 | server.listen(5000) 387 | ``` 388 | 389 | --- 390 | # crypto core streams 391 | 392 | ``` 393 | * `crtypo.createCipher(algo, password)` - transform stream to encrypt 394 | * `crtypo.createDecipher(algo, password)` - transform stream to decrypt 395 | * `crypto.createCipheriv(algo, key, iv)` - transform stream to encrypt with iv 396 | * `crypto.createDecipheriv(algo, key, iv)` - transform stream to decrypt with 397 | * iv 398 | * `crypto.createHash(algo)` - transform stream to output cryptographic hash 399 | * `crypto.createHMAC(algo, key)` - transform stream to output HMAC digest 400 | * `crypto.createSign(algo)` - writable stream to sign messages 401 | * `crypto.createVerify(algo)` - writable stream to verify signatures 402 | ``` 403 | 404 | --- 405 | # zlib core streams 406 | 407 | ``` 408 | * `zlib.createGzip(opts)` - transform stream to compress with gzip 409 | * `zlib.createGunzip(opts)` - transform stream to uncompress with gzip 410 | * `zlib.createDeflate(opts)` - transform stream to compress with deflate 411 | * `zlib.createDeflateRaw(opts)` - transform stream to compress with raw deflate 412 | * `zlib.createInflate(opts)` - transform stream to uncompress with deflate 413 | * `zlib.createInflateRaw(opts)` - transform stream to uncompress with raw deflate 414 | * `zlib.createUnzip(opts)` - transform stream to uncompress gzip and deflate 415 | ``` 416 | 417 | --- 418 | # split2 419 | 420 | split input on newlines 421 | 422 | This program counts the number of lines of input, like `wc -l`: 423 | 424 | ``` js 425 | var split = require('split2') 426 | var through = require('through2') 427 | 428 | var count = 0 429 | process.stdin.pipe(split()) 430 | .pipe(through(write, end)) 431 | 432 | function write (buf, enc, next) { 433 | count++ 434 | next() 435 | } 436 | function end () { 437 | console.log(count) 438 | } 439 | ``` 440 | 441 | --- 442 | # split2 443 | 444 | A note about split2: 445 | 446 | In each line, the trailing `'\n'` is removed. 447 | 448 | --- 449 | # split2 450 | 451 | You can give `split()` a custom string or regex to split on: 452 | 453 | This program splits on all whitespace: 454 | 455 | ``` js 456 | var split = require('split2') 457 | process.stdin.pipe(split(/\s+/)) 458 | .pipe(through(function (buf, enc, next) { 459 | console.log('word=' + buf.toString()) 460 | next() 461 | })) 462 | ``` 463 | 464 | --- 465 | # websocket-stream 466 | 467 | streaming websockets in node and the browser 468 | 469 | --- 470 | # websocket-stream: server 471 | 472 | ``` js 473 | var http = require('http') 474 | var wsock = require('websocket-stream') 475 | var through = require('through2') 476 | 477 | var server = http.createServer(function (req, res) { 478 | res.end('not found\n') 479 | }).listen(5000) 480 | wsock.createServer({ server: server }, function (stream) { 481 | stream.pipe(louder()).pipe(stream) 482 | }) 483 | 484 | function louder () { 485 | return through(function (buf, enc, next) { 486 | next(null, buf.toString().toUpperCase()) 487 | }) 488 | } 489 | ``` 490 | 491 | --- 492 | # websocket-stream: node client 493 | 494 | ``` js 495 | var wsock = require('websocket-stream') 496 | var stream = wsock('ws://localhost:5000') 497 | process.stdin.pipe(stream).pipe(process.stdout) 498 | ``` 499 | 500 | --- 501 | # websocket-stream: browser client 502 | 503 | ``` js 504 | var wsock = require('websocket-stream') 505 | var to = require('to2') 506 | var html = require('yo-yo') 507 | var root = document.body.appendChild(document.createElement('div')) 508 | var output = [] 509 | update() 510 | 511 | var stream = wsock('ws://localhost:5000') 512 | stream.pipe(to(function (buf, enc, next) { 513 | output.push(buf.toString()) 514 | update() 515 | next() 516 | })) 517 | 518 | function update () { 519 | html.update(root, html`

520 |
521 | 522 |
523 | 
${output.join('')}
524 |
`) 525 | function onsubmit (ev) { 526 | ev.preventDefault() 527 | stream.write(this.elements.msg.value + '\n') 528 | this.reset() 529 | } 530 | } 531 | ``` 532 | 533 | --- 534 | # collect-stream 535 | 536 | collect a stream's output into a single buffer 537 | 538 | for object streams, collect output into an array of objects 539 | 540 | ``` js 541 | var collect = require('collect-stream') 542 | var split = require('split2') 543 | 544 | var sp = process.stdin.pipe(split(JSON.parse)) 545 | collect(sp, function (err, rows) { 546 | if (err) console.error(err) 547 | else console.log(rows) 548 | }) 549 | ``` 550 | 551 | This module is very useful for unit tests. 552 | 553 | --- 554 | # from2 555 | 556 | create a readable stream with a pull function 557 | 558 | ``` js 559 | var from = require('from2') 560 | var messages = [ 'hello', ' world\n', null ] 561 | 562 | from(function (size, next) { 563 | next(null, messages.shift()) 564 | }).pipe(process.stdout) 565 | ``` 566 | 567 | --- 568 | # to2 569 | 570 | create a writable stream with a write and flush function 571 | 572 | ``` js 573 | var to = require('to2') 574 | var split = require('split2') 575 | 576 | process.stdin.pipe(split()).pipe(to(function (buf, next) { 577 | console.log(buf.length) 578 | next() 579 | })) 580 | ``` 581 | 582 | --- 583 | # duplexify 584 | 585 | ``` js 586 | var duplexify = require('duplexify') 587 | var d = duplexify() 588 | 589 | d.setReadable(...) 590 | d.setWritable(...) 591 | ``` 592 | 593 | --- 594 | # pump 595 | 596 | ``` js 597 | var pump = require('pump') 598 | 599 | pump(stream1, stream2, stream3, ...) 600 | ``` 601 | 602 | --- 603 | # pumpify 604 | 605 | ``` js 606 | var pump = require('pumpify') 607 | 608 | var stream = pump(stream1, stream2, stream3, ...) 609 | ``` 610 | 611 | --- 612 | # end-of-stream 613 | 614 | reliably detect when a stream is finished 615 | 616 | ``` js 617 | var onend = require('end-of-stream') 618 | var net = require('net') 619 | 620 | var server = net.createServer(function (stream) { 621 | var iv = setInterval(function () { 622 | stream.write(Date.now() + '\n') 623 | }, 1000) 624 | onend(stream, function () { 625 | clearInterval(iv) 626 | }) 627 | }) 628 | server.listen(5000) 629 | ``` 630 | 631 | --- 632 | # rpc-stream 633 | 634 | call methods defined by a remote endpoint 635 | 636 | --- 637 | # rpc-stream: server.js 638 | 639 | ``` js 640 | var net = require('net') 641 | var rpc = require('rpc-stream') 642 | 643 | net.createServer(function (stream) { 644 | stream.pipe(rpc({ 645 | hello: function (name, cb) { 646 | cb(null, 'howdy ' + name) 647 | } 648 | })).pipe(stream) 649 | }).listen(5000) 650 | ``` 651 | 652 | --- 653 | # rpc-stream: client.js 654 | 655 | ``` js 656 | var net = require('net') 657 | var rpc = require('rpc-stream') 658 | 659 | var client = rpc() 660 | client.pipe(net.connect(5000, 'localhost')).pipe(client) 661 | var remote = client.wrap(['hello']) 662 | 663 | remote.hello(process.env.USER, function (err, msg) { 664 | if (err) return console.error(err) 665 | console.log(msg) 666 | client.end() 667 | }) 668 | ``` 669 | 670 | --- 671 | # multiplex 672 | 673 | pack multiple streams into a single stream 674 | 675 | --- 676 | # multiplex: server 677 | 678 | ``` js 679 | var net = require('net') 680 | var multiplex = require('multiplex') 681 | var rpc = require('rpc-stream') 682 | var fs = require('fs') 683 | 684 | net.createServer(function (stream) { 685 | var plex = multiplex() 686 | stream.pipe(plex).pipe(stream) 687 | var client = rpc({ 688 | read: function (name, cb) { 689 | if (!/^[\w-]+$/.test(name)) { 690 | return cb(new Error('file not allowed')) 691 | } 692 | var r = fs.createReadStream('files/' + name) 693 | r.on('error', cb) 694 | r.pipe(plex.createStream('file-' + name)).pipe(r) 695 | cb(null) 696 | } 697 | }) 698 | client.pipe(plex.createSharedStream('rpc')).pipe(client) 699 | }).listen(5000) 700 | ``` 701 | 702 | --- 703 | # multiplex: client 704 | 705 | ``` js 706 | var net = require('net') 707 | var multiplex = require('multiplex') 708 | var rpc = require('rpc-stream') 709 | 710 | var plex = multiplex(function (stream, id) { 711 | if (/^file-/.test(id)) { 712 | console.log('RECEIVED FILE ' + id.replace(/^file-/,'')) 713 | stream.pipe(process.stdout) 714 | } 715 | }) 716 | plex.pipe(net.connect(5000)).pipe(plex) 717 | 718 | var client = rpc() 719 | client.pipe(plex.createSharedStream('rpc')).pipe(client) 720 | 721 | var remote = client.wrap(['read']) 722 | remote.read(process.argv[2], function (err) { 723 | if (err) console.error(err) 724 | }) 725 | ``` 726 | 727 | -------------------------------------------------------------------------------- /day2/stream/cat.js: -------------------------------------------------------------------------------- 1 | var concat = require('concat-stream') 2 | var through = require('through2') 3 | var http = require('http') 4 | var qs = require('querystring') 5 | 6 | var server = http.createServer(function (req, res) { 7 | req 8 | .pipe(counter()) 9 | .pipe(concat({ encoding: 'string' }, onbody)) 10 | 11 | function counter () { 12 | var size = 0 13 | return through(function (buf, enc, next) { 14 | size += buf.length 15 | if (size > 20) { 16 | res.end('ETOOBIG\n') 17 | //next(null, null) 18 | } else next(null, buf) 19 | }) 20 | } 21 | function onbody (body) { 22 | var params = qs.parse(body) 23 | console.log(params) 24 | res.end('ok\n') 25 | } 26 | }) 27 | server.listen(5000) 28 | -------------------------------------------------------------------------------- /day2/stream/core-loud.js: -------------------------------------------------------------------------------- 1 | var fs = require('fs') 2 | var Transform = require('stream').Transform 3 | process.stdin 4 | .pipe(toupper()) 5 | .pipe(process.stdout) 6 | 7 | function toupper () { 8 | return new Transform({ 9 | transform: function (buf, enc, next) { 10 | next(null, buf.toString().toUpperCase()) 11 | } 12 | }) 13 | } 14 | -------------------------------------------------------------------------------- /day2/stream/dup/api.js: -------------------------------------------------------------------------------- 1 | var duplexify = require('duplexify') 2 | var mkdirp = require('mkdirp') 3 | var fs = require('fs') 4 | 5 | module.exports = function (name) { 6 | var d = duplexify() 7 | mkdirp('logs', function (err) { 8 | var w = fs.createWriteStream('logs/' + name + '.log') 9 | d.setWritable(w) 10 | }) 11 | return d 12 | } 13 | -------------------------------------------------------------------------------- /day2/stream/dup/run.js: -------------------------------------------------------------------------------- 1 | var log = require('./api.js') 2 | 3 | var stream = log() 4 | var n = 0 5 | var iv = setInterval(function () { 6 | stream.write(Date.now() + '\n') 7 | if (n++ === 5) { 8 | clearInterval(iv) 9 | stream.end() 10 | } 11 | }, 100) 12 | -------------------------------------------------------------------------------- /day2/stream/echo.js: -------------------------------------------------------------------------------- 1 | var net = require('net') 2 | net.createServer(function (stream) { 3 | stream.pipe(stream) 4 | }).listen(5000) 5 | -------------------------------------------------------------------------------- /day2/stream/gunzip.js: -------------------------------------------------------------------------------- 1 | var createGunzip = require('zlib').createGunzip 2 | var createHash = require('crypto').createHash 3 | process.stdin 4 | .pipe(createGunzip()) 5 | .pipe(createHash('sha512', { encoding: 'hex' })) 6 | .pipe(process.stdout) 7 | -------------------------------------------------------------------------------- /day2/stream/hash.js: -------------------------------------------------------------------------------- 1 | var createHash = require('crypto').createHash 2 | 3 | process.stdin.pipe(createHash('sha512', { encoding: 'hex' })) 4 | .pipe(process.stdout) 5 | -------------------------------------------------------------------------------- /day2/stream/http-get-client.js: -------------------------------------------------------------------------------- 1 | var http = require('http') 2 | var req = http.request({ 3 | method: 'GET', 4 | host: 'localhost', 5 | port: 5000, 6 | url: '/' 7 | }, function (res) { 8 | console.log(res.statusCode) 9 | res.pipe(process.stdout) 10 | }) 11 | req.end() 12 | -------------------------------------------------------------------------------- /day2/stream/http-post-client.js: -------------------------------------------------------------------------------- 1 | var http = require('http') 2 | var req = http.request({ 3 | method: 'POST', 4 | host: 'localhost', 5 | port: 5000, 6 | url: '/' 7 | }, function (res) { 8 | console.log(res.statusCode) 9 | res.pipe(process.stdout) 10 | }) 11 | req.end('HELLO\n') 12 | -------------------------------------------------------------------------------- /day2/stream/http.js: -------------------------------------------------------------------------------- 1 | var http = require('http') 2 | var fs = require('fs') 3 | 4 | var server = http.createServer(function (req, res) { 5 | if (req.method === 'POST') { 6 | req.pipe(process.stdout) 7 | req.once('end', function () { 8 | res.end('ok\n') 9 | }) 10 | } else { 11 | res.setHeader('content-type','text/plain') 12 | fs.createReadStream('hello.txt').pipe(res) 13 | } 14 | }) 15 | server.listen(5000) 16 | -------------------------------------------------------------------------------- /day2/stream/line-count.js: -------------------------------------------------------------------------------- 1 | var split = require('split2') 2 | var to = require('to2') 3 | var lineCount = 0 4 | 5 | process.stdin 6 | .pipe(split()) 7 | .pipe(to(write,end)) 8 | 9 | function write (buf, enc, next) { 10 | lineCount++ 11 | next() 12 | } 13 | function end (next) { 14 | console.log(lineCount) 15 | next() 16 | } 17 | -------------------------------------------------------------------------------- /day2/stream/loud.js: -------------------------------------------------------------------------------- 1 | var fs = require('fs') 2 | var through = require('through2') 3 | process.stdin 4 | .pipe(toupper()) 5 | .pipe(process.stdout) 6 | 7 | function toupper () { 8 | return through(function (buf, enc, next) { 9 | next(null, buf.toString().toUpperCase()) 10 | }) 11 | } 12 | -------------------------------------------------------------------------------- /day2/stream/obj.js: -------------------------------------------------------------------------------- 1 | var through = require('through2') 2 | var size = 0 3 | process.stdin 4 | .pipe(through.obj(write1)) 5 | .pipe(through.obj(write2, end)) 6 | 7 | function write1 (buf, enc, next) { 8 | next(null, { length: buf.length }) 9 | } 10 | 11 | function write2 (obj, enc, next) { 12 | size += obj.length 13 | next() 14 | } 15 | function end () { 16 | console.log('size=', size) 17 | } 18 | -------------------------------------------------------------------------------- /day2/stream/p2p/swarm.js: -------------------------------------------------------------------------------- 1 | var swarm = require('webrtc-swarm') 2 | var signalhub = require('signalhub') 3 | var onend = require('end-of-stream') 4 | var through = require('through2') 5 | var to = require('to2') 6 | var html = require('yo-yo') 7 | var root = document.body.appendChild(document.createElement('div')) 8 | 9 | var hub = signalhub('streams-day', [ 'https://signalhub.mafintosh.com' ]) 10 | var sw = swarm(hub) 11 | var output = [] 12 | var peers = {} 13 | update() 14 | 15 | function update () { 16 | html.update(root, html`
17 |
18 | 19 |
20 | 
${output.join('')}
21 |
`) 22 | function onsubmit (ev) { 23 | ev.preventDefault() 24 | var msg = this.elements.msg.value 25 | Object.keys(peers).forEach(function (id) { 26 | peers[id].write(msg + '\n') 27 | }) 28 | this.reset() 29 | } 30 | } 31 | 32 | sw.on('peer', function (peer, id) { 33 | peers[id] = peer 34 | onend(peer, function () { 35 | delete peers[id] 36 | }) 37 | peer.pipe(to(function (buf, enc, next) { 38 | output.push(buf.toString()) 39 | update() 40 | next() 41 | })) 42 | }) 43 | 44 | function toupper () { 45 | return through(function (buf, enc, next) { 46 | next(null, buf.toString().toUpperCase()) 47 | }) 48 | } 49 | -------------------------------------------------------------------------------- /day2/stream/proxy.js: -------------------------------------------------------------------------------- 1 | var net = require('net') 2 | net.createServer(function (stream) { 3 | stream.pipe(net.connect(5000,'localhost')).pipe(stream) 4 | }).listen(5005) 5 | -------------------------------------------------------------------------------- /day2/stream/ps.js: -------------------------------------------------------------------------------- 1 | var spawn = require('child_process').spawn 2 | var ps = spawn('grep',['potato']) 3 | ps.stdout.pipe(process.stdout) 4 | ps.stdin.write('cheese\n') 5 | ps.stdin.write('carrots\n') 6 | ps.stdin.write('carrot potatoes\n') 7 | ps.stdin.write('potato!\n') 8 | ps.stdin.end() 9 | -------------------------------------------------------------------------------- /day2/stream/r.js: -------------------------------------------------------------------------------- 1 | var fs = require('fs') 2 | var r = fs.createReadStream(process.argv[2]) 3 | r.pipe(process.stdout) 4 | -------------------------------------------------------------------------------- /day2/stream/rpc/client.js: -------------------------------------------------------------------------------- 1 | var net = require('net') 2 | var rpc = require('rpc-stream') 3 | 4 | var client = rpc() 5 | client.pipe(net.connect(5000, 'localhost')).pipe(client) 6 | var remote = client.wrap(['hello']) 7 | 8 | remote.hello(process.env.USER, function (err, msg) { 9 | if (err) return console.error(err) 10 | console.log(msg) 11 | client.end() 12 | }) 13 | -------------------------------------------------------------------------------- /day2/stream/rpc/server.js: -------------------------------------------------------------------------------- 1 | var net = require('net') 2 | var rpc = require('rpc-stream') 3 | 4 | net.createServer(function (stream) { 5 | stream.pipe(rpc({ 6 | hello: function (name, cb) { 7 | cb(null, 'howdy ' + name) 8 | } 9 | })).pipe(stream) 10 | }).listen(5000) 11 | -------------------------------------------------------------------------------- /day2/stream/vpn-client.js: -------------------------------------------------------------------------------- 1 | var net = require('net') 2 | var crypto = require('crypto') 3 | var pw = 'abc123' 4 | 5 | var stream = net.connect(5005,'localhost') 6 | process.stdin 7 | .pipe(crypto.createCipher('aes192',pw)) 8 | .pipe(stream) 9 | .pipe(crypto.createDecipher('aes192',pw)) 10 | .pipe(process.stdout) 11 | -------------------------------------------------------------------------------- /day2/stream/vpn.js: -------------------------------------------------------------------------------- 1 | var net = require('net') 2 | var crypto = require('crypto') 3 | var pump = require('pump') 4 | var pw = 'abc123' 5 | 6 | net.createServer(function (stream) { 7 | pump( 8 | stream, 9 | crypto.createDecipher('aes192',pw), 10 | net.connect(5000,'localhost'), 11 | crypto.createCipher('aes192',pw), 12 | stream, 13 | function (err) { 14 | console.error(err) 15 | } 16 | ) 17 | }).listen(5005) 18 | -------------------------------------------------------------------------------- /day2/stream/w.js: -------------------------------------------------------------------------------- 1 | var fs = require('fs') 2 | var w = fs.createWriteStream('cool.txt') 3 | w.on('finish', function () { 4 | console.log('FINISHED') 5 | }) 6 | w.write('hi\n') 7 | w.write('wow\n') 8 | w.end() 9 | -------------------------------------------------------------------------------- /day2/stream/wsock/client.js: -------------------------------------------------------------------------------- 1 | var wsock = require('websocket-stream') 2 | var through = require('through2') 3 | var stream = wsock('ws://' + location.host) 4 | var html = require('yo-yo') 5 | var root = document.body.appendChild(document.createElement('div')) 6 | var output = [] 7 | update() 8 | 9 | stream.pipe(through(function (buf, enc, next) { 10 | output.push(buf.toString()) 11 | update() 12 | next() 13 | })) 14 | 15 | function update () { 16 | html.update(root, html`
17 |
18 | 19 |
20 | 
${output.join('')}
21 |
`) 22 | function onsubmit (ev) { 23 | ev.preventDefault() 24 | stream.write(this.elements.msg.value + '\n') 25 | this.reset() 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /day2/stream/wsock/nclient.js: -------------------------------------------------------------------------------- 1 | var wsock = require('websocket-stream') 2 | var stream = wsock('ws://localhost:5000') 3 | process.stdin.pipe(stream).pipe(process.stdout) 4 | -------------------------------------------------------------------------------- /day2/stream/wsock/public/index.html: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /day2/stream/wsock/server.js: -------------------------------------------------------------------------------- 1 | var http = require('http') 2 | var ecstatic = require('ecstatic') 3 | var through = require('through2') 4 | 5 | var server = http.createServer(ecstatic(__dirname + '/public')) 6 | server.listen(5000) 7 | 8 | var wsock = require('websocket-stream') 9 | wsock.createServer({ server: server }, function (stream) { 10 | stream.pipe(loud()).pipe(stream) 11 | }) 12 | 13 | function loud () { 14 | return through(function (buf, enc, next) { 15 | next(null, buf.toString().toUpperCase()) 16 | }) 17 | } 18 | -------------------------------------------------------------------------------- /day2/userver.js: -------------------------------------------------------------------------------- 1 | var http = require('http') 2 | var concat = require('concat-stream') 3 | var qs = require('querystring') 4 | 5 | var server = http.createServer(function (req, res) { 6 | console.log(req.method, req.url, req.headers) 7 | // todo: make sure the user can only upload 8 | // a finite amount of data 9 | req.pipe(concat(function (body) { 10 | var params = qs.parse(body) 11 | console.log(params) 12 | res.end('ok\n') 13 | })) 14 | }) 15 | server.listen(5000) 16 | -------------------------------------------------------------------------------- /day3.md: -------------------------------------------------------------------------------- 1 | # day 3: leveldb and crypto 2 | 3 | Learn about crypto basics and leveldb, a modular database. 4 | 5 | Cryptography provides a foundation for secure communications and distributed systems. 6 | LevelDB provides a modular ecosystem to persist and query data in node.js and 7 | the browser. These techniques will let you build completely different kinds of 8 | webapps that are fault tolerant, work offline, and can even replicate p2p. 9 | 10 | Join James Halliday (substack) as we dive into crypto fundamentals and leveldb. 11 | You will: 12 | 13 | * Sign, verify, and encrypt with libsodium/nacl. 14 | * Use cryptographic hashes to build secure data structures. 15 | * Store data in the browser using IndexedDB wrappers. 16 | * Design modular data schemas over lexicographic keys. 17 | * Build data pipelines for live, streaming data. 18 | * Synchronize data and build indexes using the kappa architecture 19 | 20 | All of the content relies on libraries that work in both node.js and the 21 | browser. 22 | 23 | ## schedule 24 | 25 | * 08:30 - system check, hello 26 | * 09:00 - querying and updating data 27 | * 09:30 - thinking lexicographically 28 | * 10:00 - modular database design 29 | * 10:30 - leveldb in the browser 30 | * 11:00 - hashes and signing 31 | * 11:30 - merkle DAGs and secure data structures 32 | * 12:00 - lunch etc 33 | * 13:00 - asymmetric and symmetric encryption 34 | * 13:30 - logs, materialized views, and the kappa architecture 35 | * 14:30 - project: verified secure gossip network 36 | * 16:00 - closing notes and wrap up 37 | 38 | ## who is this for? 39 | 40 | This workshop is for people who are proficient in javascript and want to explore 41 | some advanced concepts and architectures. 42 | 43 | If you know your way around node.js or frontend and you want to know how to 44 | build offline-first, secure applications, this workshop is for you! 45 | 46 | A knowledge of node.js streams will be beneficial but is not strictly required. 47 | -------------------------------------------------------------------------------- /day3/batch.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('batch.db', { valueEncoding: 'json' }) 3 | var to = require('to2') 4 | 5 | db.createReadStream({ gt: 'n5', lt: 'n8' }) 6 | .pipe(to.obj(function (row, enc, next) { 7 | console.log(row) 8 | next() 9 | })) 10 | 11 | /* 12 | var batch = [] 13 | for (var i = 0; i < 10; i++) { 14 | batch.push({ key: 'n' + i, value: i*1000 }) 15 | } 16 | db.batch(batch, function (err) { 17 | if (err) console.error(err) 18 | }) 19 | */ 20 | -------------------------------------------------------------------------------- /day3/blob/addimg.js: -------------------------------------------------------------------------------- 1 | var blobs = require('content-addressable-blob-store') 2 | var store = blobs('img.blob') 3 | var level = require('level') 4 | var db = level('img.db', { valueEncoding: 'json' }) 5 | 6 | var w = store.createWriteStream(function (err) { 7 | if (err) return console.error(err) 8 | var key = 'img!' + w.key 9 | var doc = { 10 | time: Date.now() 11 | } 12 | db.put(key, doc, function (err) { 13 | if (err) return console.error(err) 14 | }) 15 | }) 16 | process.stdin.pipe(w) 17 | -------------------------------------------------------------------------------- /day3/blob/list-files.js: -------------------------------------------------------------------------------- 1 | var blobs = require('content-addressable-blob-store') 2 | var store = blobs('img.blob') 3 | var level = require('level') 4 | var db = level('img.db', { valueEncoding: 'json' }) 5 | var to = require('to2') 6 | 7 | db.createReadStream() 8 | .pipe(to.obj(function (row, enc, next) { 9 | console.log({ 10 | key: row.key.split('!')[1], 11 | value: row.value 12 | }) 13 | next() 14 | })) 15 | -------------------------------------------------------------------------------- /day3/blob/read-file.js: -------------------------------------------------------------------------------- 1 | var blobs = require('content-addressable-blob-store') 2 | var store = blobs('img.blob') 3 | 4 | var hash = process.argv[2] 5 | store.createReadStream(hash) 6 | .pipe(process.stdout) 7 | -------------------------------------------------------------------------------- /day3/blob/server.js: -------------------------------------------------------------------------------- 1 | var http = require('http') 2 | var blobs = require('content-addressable-blob-store') 3 | var store = blobs('img.blob') 4 | var level = require('level') 5 | var db = level('img.db', { valueEncoding: 'json' }) 6 | var through = require('through2') 7 | 8 | var server = http.createServer(function (req, res) { 9 | if (req.url === '/') { 10 | db.createReadStream() 11 | .pipe(through.obj(function (row, enc, next) { 12 | next(null, `
13 |

${row.value.time}

14 | 15 |
`) 16 | })) 17 | .pipe(res) 18 | } else if (/^\/image\//.test(req.url)) { 19 | var hash = req.url.replace(/^\/image\//,'') 20 | res.setHeader('content-type','image/jpg') 21 | store.createReadStream(hash).pipe(res) 22 | } else res.end('not found\n') 23 | }) 24 | server.listen(5000) 25 | -------------------------------------------------------------------------------- /day3/browser/main.js: -------------------------------------------------------------------------------- 1 | var level = require('level-browserify') 2 | var db = level('whatever', { valueEncoding: 'json' }) 3 | var html = require('yo-yo') 4 | var root = document.body.appendChild(document.createElement('div')) 5 | var count = '?' 6 | update() 7 | db.get('count', function (err, value) { 8 | count = value || 0 9 | update() 10 | }) 11 | 12 | function update () { 13 | html.update(root, html`
14 |

${count}

15 | 16 |
`) 17 | function onclick (ev) { 18 | count++ 19 | db.put('count', count, function (err) { 20 | if (err) console.error(err) 21 | else update() 22 | }) 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /day3/crypto.md: -------------------------------------------------------------------------------- 1 | # crypto 2 | 3 | --- 4 | # hashes 5 | 6 | ``` js 7 | var createHash = require('crypto').createHash 8 | var createHash = require('createHash') 9 | 10 | var stream = createHash(algorithm) 11 | ``` 12 | 13 | algorithms: 14 | 15 | * sha1 16 | * sha256 17 | * sha512 18 | * md5 19 | 20 | --- 21 | # symmetric ciphers 22 | 23 | * requires a shared secret (like a password) 24 | 25 | --- 26 | # asymmetric crypto 27 | 28 | * public/private keypairs 29 | * need to know somebody's public key to send them a secret message 30 | 31 | --- 32 | # random number generator 33 | 34 | secure entropy needed for generating keys, nonces 35 | 36 | ``` js 37 | window.crypto.getRandomValues(new Uint8Array(8)) 38 | ``` 39 | 40 | If your RNG is bad, your crypto will be bad. 41 | 42 | --- 43 | # don't roll your own crypto™ 44 | 45 | very easy to mess something up 46 | 47 | * replay attacks 48 | * timing attacks 49 | * padding/compression oracle attacks 50 | * side channels 51 | * downgrade attacks 52 | 53 | so many: https://en.wikipedia.org/wiki/Category:Cryptographic_attacks 54 | 55 | --- 56 | # use libsodium/nacl 57 | 58 | * works in node and the browser 59 | * `require('chloride')` 60 | * uses only good crypto algorithms 61 | * resists timing attacks 62 | 63 | --- 64 | # sodium generate keypairs 65 | 66 | ``` 67 | var sodium = require('chloride') 68 | console.log(sodium.crypto_sign_keypair()) 69 | console.log(sodium.crypto_box_keypair()) 70 | ``` 71 | 72 | --- 73 | # combined vs detached 74 | 75 | * combined mode - output contains the original message + signature 76 | * detached mode - output contains only the signature 77 | 78 | --- 79 | # sodium sign/verify combined 80 | 81 | ``` js 82 | var signedData = sodium.crypto_sign(msg,secretKey) 83 | var msg = sodium.crypto_sign_open(signedData,publicKey) 84 | // msg is undefined if the signature failed 85 | ``` 86 | 87 | --- 88 | # sodium sign/verify detached 89 | 90 | ``` js 91 | var sig = sodium.crypto_sign_detached(msg, secretKey) 92 | var ok = sodium.crypto_sign_verify_detached(sig,msg,publicKey) 93 | ``` 94 | 95 | --- 96 | # sodium authenticated encryption combined 97 | 98 | symmetric cipher with message authentication code (MAC) 99 | to prevent tampering 100 | 101 | ``` js 102 | var nonce = crypto.randomBytes(24) 103 | var key = crypto.randomBytes(32) 104 | var cipherText = sodium.crypto_secretbox_easy(msg, nonce, key) 105 | var clearText = sodium.crypto_secretbox_open_easy(cipherText, nonce, key) 106 | ``` 107 | 108 | --- 109 | # sodium public key encrypt combined 110 | 111 | ``` js 112 | sodium.crypto_box_easy() 113 | sodium.crypto_box_open_easy() 114 | ``` 115 | 116 | etc 117 | 118 | --- 119 | # secret connection 120 | 121 | npm install secret-handshake pull-stream-to-stream 122 | 123 | --- 124 | # merkle DAGs 125 | 126 | * hash every document 127 | * point to other docs by their hash inside the doc itself 128 | 129 | examples: 130 | 131 | * git 132 | * ipfs 133 | * secure scuttlebutt 134 | * dat/hypercore 135 | 136 | --- 137 | # merkle DAGs 138 | 139 | security properties: 140 | 141 | * tamper-proof: changing a doc changes every hash that points at it 142 | * with signing, docs are also authenticated 143 | 144 | you can receive merkle DAG nodes from untrusted peers over gossip protocols 145 | 146 | --- 147 | # merkle DAGs 148 | 149 | demo: DIY merkle DAG using shasum 150 | 151 | -------------------------------------------------------------------------------- /day3/inc.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('inc.db', { valueEncoding: 'json' }) 3 | 4 | db.get('count', function (err, value) { 5 | var n = (value || 0) + 1 6 | db.put('count', n, function (err) { 7 | if (err) console.error(err) 8 | else console.log(n) 9 | }) 10 | }) 11 | -------------------------------------------------------------------------------- /day3/kappa.md: -------------------------------------------------------------------------------- 1 | # kappa architecture 2 | 3 | enterprise architecture 4 | 5 | * immutable, append-only logs are the source of truth 6 | * materialized views built from the logs 7 | 8 | also good for p2p! 9 | 10 | --- 11 | # append-only logs 12 | 13 | * use hashes to build a merkle DAG 14 | * trivial naive replication: concatenate 15 | 16 | --- 17 | # hyperlog 18 | 19 | append-only merkle DAG log store 20 | 21 | * link to other documents by cryptographic hash 22 | * hooks for cryptographic signing and vertification 23 | 24 | ``` js 25 | var level = require('level') 26 | var hyperlog = require('hyperlog') 27 | var log = hyperlog(level('log.db'), { valueEncoding: 'json' }) 28 | ``` 29 | 30 | --- 31 | # hyperlog-index 32 | 33 | build materialized views on top of a hyperlog 34 | 35 | ``` js 36 | var indexer = require('hyperlog-index') 37 | var dex = indexer({ 38 | log: log, 39 | db: db, 40 | map: function (row, next) { 41 | // store data for row then call next() 42 | } 43 | }) 44 | ``` 45 | 46 | --- 47 | # hyperkv 48 | 49 | * p2p key/value store as a materialized view over a hyperlog 50 | * multi-value register conflict strategy 51 | 52 | ``` js 53 | var hyperkv = require('hyperkv') 54 | var kv = hyperkv({ log: log, db: db }) 55 | ``` 56 | 57 | --- 58 | # multi-value register 59 | 60 | instead of picking a "winner", accept plurality 61 | 62 | for any given key, there could be many legitimate values 63 | 64 | --- 65 | # hyperkv 66 | 67 | ``` js 68 | kv.put(key, value, { links: links }, cb) 69 | kv.get(key, function (err, values) { 70 | // could be many forks! 71 | }) 72 | ``` 73 | 74 | --- 75 | # hyperlog-sodium 76 | 77 | more easily configure hyperlog for cryptographic signing 78 | 79 | --- 80 | # kappa architecture examples 81 | 82 | * secure scuttlebutt - p2p social database 83 | * osm-p2p - map database built on hyperlog, hyperkv, hyperlog-kdb-index 84 | * dat/hyperdrive - p2p file sync over an append-only merkle DAG log 85 | 86 | --- 87 | # webrtc swarm with hyperlogs 88 | 89 | ``` 90 | var swarm = require('webrtc-swarm') 91 | var signalhub = require('signalhub') 92 | var level = require('level-browserify') 93 | var hyperlog = require('hyperlog') 94 | var log = hyperlog(level('whatever'), { valueEncoding: 'json' }) 95 | var sw = swarm(signalhub('cool-swarm')) 96 | sw.on('peer', function (stream, id) { 97 | stream.pipe(log.replicate({ live: true })).pipe(stream) 98 | }) 99 | ``` 100 | 101 | --- 102 | # project idea 103 | 104 | p2p wiki in the browser with cryptographic signing 105 | over a webrtc swarm using: 106 | 107 | * hyperlog 108 | * hyperkv 109 | * chloride 110 | * level-browserify 111 | * webrtc-swarm 112 | * images using hypercore? (and multiplexing the replication stream) 113 | 114 | -------------------------------------------------------------------------------- /day3/leveldb.md: -------------------------------------------------------------------------------- 1 | # leveldb! 2 | 3 | an embedded key/value database 4 | 5 | --- 6 | # embedded vs standalone 7 | 8 | * embedded - in-process library 9 | * standalone - separate service 10 | 11 | --- 12 | # embedded databases 13 | 14 | * leveldb 15 | * sqlite 16 | * berkeleydb 17 | 18 | --- 19 | # standalone databases 20 | 21 | * postgresql 22 | * mysql 23 | * mongodb 24 | * couchdb 25 | 26 | --- 27 | # install 28 | 29 | Since leveldb is a standalone databse, 30 | you can install it with npm: 31 | 32 | ``` 33 | npm install level 34 | ``` 35 | 36 | --- 37 | # and then 38 | 39 | ``` js 40 | var level = require('level') 41 | var db = level('./whatever.db') 42 | ``` 43 | 44 | --- 45 | # valueEncoding 46 | 47 | ``` js 48 | var level = require('level') 49 | var db = level('./whatever.db', { valueEncoding: 'json' }) 50 | ``` 51 | 52 | --- 53 | # what leveldb is good for 54 | 55 | * running the same database in node and the browser 56 | * when your data isn't very relational 57 | * building your own kappa architecture 58 | 59 | --- 60 | # level methods 61 | 62 | * `db.get()` 63 | * `db.put()` 64 | * `db.del()` 65 | * `db.batch()` 66 | * `db.createReadStream()` 67 | 68 | --- 69 | # put 70 | 71 | set a value for a key with `.put()` 72 | 73 | ``` js 74 | var level = require('level') 75 | var db = level('./whatever.db') 76 | db.put('key', 'value', function (err) { 77 | if (err) console.error(err) 78 | }) 79 | ``` 80 | 81 | --- 82 | # get 83 | 84 | load a value for a key with `.get()` 85 | 86 | ``` js 87 | var level = require('level') 88 | var db = level('./whatever.db') 89 | db.get('key', function (err, value) { 90 | if (err) console.error(err) 91 | else console.log('value=', value) 92 | }) 93 | ``` 94 | 95 | --- 96 | # del 97 | 98 | delete a value at a key with `.del()`: 99 | 100 | --- 101 | # atomicity 102 | 103 | either all transactions succeed 104 | or all transactions fail 105 | 106 | --- 107 | # consistency 108 | 109 | atomicity is important to enforce consistency 110 | 111 | Suppose a user has just signed up. 112 | We might need to create: 113 | 114 | * a record for their 115 | * a record for their login username and password 116 | 117 | --- 118 | # batch 119 | 120 | insert multiple records at a time, atomically 121 | 122 | ``` js 123 | db.batch([ 124 | {"key":"foo","value":"123"}, 125 | {"key":"bar","value":"456"} 126 | ], function (err) { }) 127 | ``` 128 | 129 | --- 130 | # createReadStream 131 | 132 | `db.createReadStream(opts)`: 133 | 134 | Returns a readable objectMode stream. 135 | 136 | * opts.gte - greater than or equal to 137 | * opts.gt - greater than 138 | * opts.lte - less than or equal to 139 | * opts.lt - less than 140 | * opts.limit - maximum number of results 141 | * opts.reverse - higher keys before lower keys 142 | 143 | --- 144 | # createReadStream 145 | 146 | ``` 147 | db.createReadStream({ 148 | gt: "a", 149 | lt: "m" 150 | }) 151 | ``` 152 | 153 | * `gt: "a"` - greater than "a" 154 | * `lt: "m"` - less than "m" 155 | 156 | --- 157 | # thinking lexicographically 158 | 159 | keys are sorted by their string values: 160 | 161 | * aaaaa 162 | * bb 163 | * ccccc 164 | 165 | --- 166 | # numbers get converted into strings! 167 | 168 | * "1" 169 | * "12" 170 | * "3" 171 | * "4" 172 | * "555" 173 | * "6" 174 | 175 | --- 176 | # organizing your keys 177 | 178 | key/value structure we might use for 179 | a user/post system: 180 | 181 | ``` json 182 | [{"key":"user!substack","value":{"bio":"beep boop"}}, 183 | {"key":"user!maxogden","value":{"bio":"cats."}}, 184 | {"key":"post!substack!2015-01-04 11:45","value":"cool beans"}] 185 | {"key":"post!maxogden!2015-01-03 17:33","value":"soup."}] 186 | ``` 187 | 188 | --- 189 | 190 | This will let us efficiently query for a user's posts: 191 | 192 | ``` js 193 | db.createReadStream({ 194 | gt: "post!substack", 195 | lt: "post!substack!~" 196 | }) 197 | ``` 198 | 199 | --- 200 | 201 | In either case, 202 | what if we want to get ALL the posts on the system? 203 | 204 | --- 205 | # secondary indexes 206 | 207 | We can use `.batch()` to create multiple keys for each post: 208 | 209 | ``` js 210 | var now = new Date().toISOString() 211 | var id = crypto.randomBytes(16).toString('hex') 212 | var subkey = now + '!' + id 213 | db.batch([ 214 | {type:'post',key:'post!substack!'+subkey,value:msg}, 215 | {type:'post',key:'post!'+subkey,value:msg}, 216 | ]) 217 | ``` 218 | 219 | --- 220 | # querying our indexes 221 | 222 | now to get all the posts system-wide sorted by date, 223 | we can do: 224 | 225 | ``` js 226 | db.createReadStream({ 227 | start: "post!", 228 | end: "post!~" 229 | }) 230 | ``` 231 | 232 | --- 233 | # subleveldown 234 | 235 | we can create nested sub-databases with subleveldown: 236 | 237 | ``` js 238 | var level = require('level') 239 | var sublevel = require('subleveldown') 240 | var db = level('whatever.db') 241 | 242 | var catsdb = sublevel(db, 'cats') 243 | var robodb = sublevel(db, 'robots') 244 | 245 | catsdb.put('msg', 'meow') 246 | robodb.put('msg', 'beep boop') 247 | ``` 248 | 249 | and `catsdb` and `robodb` will each key a unique namespace 250 | for a `msg` key. 251 | 252 | --- 253 | # level-livefeed 254 | 255 | subscribe to a live feed of changes to the database 256 | 257 | ``` js 258 | var liveStream = require('level-livefeed') 259 | var stream = liveStream(db, {}) 260 | ``` 261 | 262 | --- 263 | # designing modules 264 | 265 | * only do `require('level')` at the outermost layer 266 | * your module should expect to be passed a db instance 267 | 268 | --- 269 | # leveldb in the browser 270 | 271 | * `require('level-browserify')` instead of `require('level')` 272 | 273 | Uses IndexedDB internally to present the levelup API. 274 | 275 | --- 276 | # what to store and not store in level 277 | 278 | * best for tiny documents 279 | * documents can point at binary data by hash 280 | 281 | some good modules for blob storage: 282 | 283 | * content-addressable-blob-store 284 | * hypercore (can also replicate!) 285 | * webtorrent 286 | * torrent-blob-store 287 | 288 | -------------------------------------------------------------------------------- /day3/log/add.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('log.db') 3 | var hyperlog = require('hyperlog') 4 | var log = hyperlog(db, { valueEncoding: 'json' }) 5 | 6 | var msg = process.argv[2] 7 | var links = process.argv.slice(3) 8 | log.add(links, { message: msg, time: Date.now() }, function (err, node) { 9 | if (err) console.error(err) 10 | else console.log(node) 11 | }) 12 | -------------------------------------------------------------------------------- /day3/log/dex.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('log.db') 3 | var idb = level('ix.db') 4 | var hyperlog = require('hyperlog') 5 | var indexer = require('hyperlog-index') 6 | var strftime = require('strftime') 7 | var log = hyperlog(db, { valueEncoding: 'json' }) 8 | 9 | var dex = indexer({ 10 | log: log, 11 | db: idb, 12 | map: function (row, next) { 13 | var key = strftime('%F %T', new Date(row.value.time)) 14 | console.log('key=', key) 15 | console.log('value=', row.key) 16 | idb.put(key, row.key, next) 17 | } 18 | }) 19 | -------------------------------------------------------------------------------- /day3/log/ix-list.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('ix.db') 3 | var to = require('to2') 4 | 5 | db.createReadStream() 6 | .pipe(to.obj(function (row, enc, next) { 7 | console.log(row) 8 | next() 9 | })) 10 | -------------------------------------------------------------------------------- /day3/log/list.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('log.db') 3 | var hyperlog = require('hyperlog') 4 | var log = hyperlog(db, { valueEncoding: 'json' }) 5 | var to = require('to2') 6 | 7 | log.createReadStream() 8 | .pipe(to.obj(function (row, enc, next) { 9 | console.log(row) 10 | next() 11 | })) 12 | -------------------------------------------------------------------------------- /day3/log/rep.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db1 = level('log.db') 3 | var db2 = level('log2.db') 4 | var hyperlog = require('hyperlog') 5 | var log1 = hyperlog(db1, { valueEncoding: 'json' }) 6 | var log2 = hyperlog(db2, { valueEncoding: 'json' }) 7 | 8 | var r1 = log1.replicate() 9 | var r2 = log2.replicate() 10 | r1.pipe(r2).pipe(r1) 11 | -------------------------------------------------------------------------------- /day3/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "dependencies": { 3 | "collect-stream": "^1.2.1", 4 | "content-addressable-blob-store": "^4.3.0", 5 | "hypercore": "^6.2.4", 6 | "hyperkv": "^2.0.3", 7 | "hyperlog": "^4.12.1", 8 | "hyperlog-index": "^5.2.0", 9 | "hyperlog-sodium": "^2.0.0", 10 | "level": "^1.6.0", 11 | "level-browserify": "^1.1.0", 12 | "level-livefeed": "^0.2.0", 13 | "lexicographic-integer": "^1.1.0", 14 | "pull-stream-to-stream": "^1.3.4", 15 | "secret-handshake": "^1.1.7", 16 | "signalhub": "^4.8.0", 17 | "strftime": "^0.10.0", 18 | "subleveldown": "^2.1.0", 19 | "through2": "^2.0.3", 20 | "to2": "^1.0.0", 21 | "webrtc-swarm": "^2.8.0", 22 | "yo-yo": "^1.4.0" 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /day3/posts/mkpost.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('posts.db', { valueEncoding: 'json' }) 3 | var strftime = require('strftime') 4 | var randomBytes = require('crypto').randomBytes 5 | 6 | var name = process.argv[2] 7 | var msg = process.argv.slice(3).join(' ') 8 | var time = strftime('%F %T') 9 | 10 | var id = randomBytes(16).toString('hex') 11 | var batch = [ 12 | { key: 'post!' + id, value: { name: name, time: time, body: msg } }, 13 | { key: 'post-name!' + name + '!' + time + '!' + id, value: 0 }, 14 | { key: 'post-time!' + time + '!' + name + '!' + id, value: 0 } 15 | ] 16 | db.batch(batch, function (err) { 17 | if (err) console.error(err) 18 | }) 19 | -------------------------------------------------------------------------------- /day3/posts/postlist.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('posts.db', { valueEncoding: 'json' }) 3 | var to = require('to2') 4 | 5 | db.createReadStream({ gt: 'post!', lt: 'post!~' }) 6 | .pipe(to.obj(function (row, enc, next) { 7 | var id = row.key.split('!')[0] 8 | var name = row.value.name 9 | var time = row.value.time 10 | var body = row.value.body 11 | console.log(time + ' <' + name + '> ' + body) 12 | next() 13 | })) 14 | -------------------------------------------------------------------------------- /day3/posts/posts-by-author.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('posts.db', { valueEncoding: 'json' }) 3 | var to = require('to2') 4 | 5 | var name = process.argv[2] 6 | var opts = { 7 | gt: 'post-name!' + name + '!', 8 | lt: 'post-name!' + name + '!~' 9 | } 10 | db.createReadStream(opts) 11 | .pipe(to.obj(function (row, enc, next) { 12 | var id = row.key.split('!').slice(-1)[0] 13 | db.get('post!' + id, function (err, doc) { 14 | var name = doc.name 15 | var time = doc.time 16 | var body = doc.body 17 | console.log(time + ' <' + name + '> ' + body) 18 | next() 19 | }) 20 | })) 21 | -------------------------------------------------------------------------------- /day3/posts/posts-by-time.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('posts.db', { valueEncoding: 'json' }) 3 | var to = require('to2') 4 | 5 | var opts = { 6 | gt: 'post-time!', 7 | lt: 'post-time!~' 8 | } 9 | db.createReadStream(opts) 10 | .pipe(to.obj(function (row, enc, next) { 11 | var id = row.key.split('!').slice(-1)[0] 12 | db.get('post!' + id, function (err, doc) { 13 | var name = doc.name 14 | var time = doc.time 15 | var body = doc.body 16 | console.log(time + ' <' + name + '> ' + body) 17 | next() 18 | }) 19 | })) 20 | -------------------------------------------------------------------------------- /day3/posts/useradd.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('posts.db', { valueEncoding: 'json' }) 3 | 4 | var name = process.argv[2] 5 | db.put('user!' + name, {}, function (err) { 6 | if (err) console.error(err) 7 | }) 8 | -------------------------------------------------------------------------------- /day3/posts/userlist.js: -------------------------------------------------------------------------------- 1 | var level = require('level') 2 | var db = level('posts.db', { valueEncoding: 'json' }) 3 | var to = require('to2') 4 | 5 | db.createReadStream({ gt: 'user!', lt: 'user!~' }) 6 | .pipe(to.obj(function (row, enc, next) { 7 | console.log(row.key.split('!')[1]) 8 | next() 9 | })) 10 | -------------------------------------------------------------------------------- /day3/sub.js: -------------------------------------------------------------------------------- 1 | var sublevel = require('subleveldown') 2 | var level = require('level') 3 | var db = level('sub.db') 4 | 5 | var adb = sublevel(db, 'a') 6 | var bdb = sublevel(db, 'b') 7 | 8 | adb.get('count', function (err, value) { 9 | var n = Number(value||0)+1 10 | adb.put('count', n, function (err) { 11 | if (err) console.error(err) 12 | else console.log('a:count', n) 13 | }) 14 | }) 15 | bdb.get('count', function (err, value) { 16 | var n = Number(value||0)+10 17 | bdb.put('count', n, function (err) { 18 | if (err) console.error(err) 19 | else console.log('b:count', n) 20 | }) 21 | }) 22 | -------------------------------------------------------------------------------- /day3/wiki/main.js: -------------------------------------------------------------------------------- 1 | var level = require('level-browserify') 2 | var sub = require('subleveldown') 3 | var db = level('wiki.db') 4 | var hyperlog = require('hyperlog') 5 | var log = hyperlog(sub(db,'log'), { valueEncoding: 'json' }) 6 | var html = require('yo-yo') 7 | var hyperkv = require('hyperkv') 8 | var kv = hyperkv({ log: log, db: sub(db,'kv') }) 9 | var to = require('to2') 10 | 11 | var root = document.body.appendChild(document.createElement('div')) 12 | var docs = {} 13 | 14 | var wswarm = require('webrtc-swarm') 15 | var signalhub = require('signalhub') 16 | 17 | var swarm = wswarm(signalhub('cool-wiki',['http://localhost:5008'])) 18 | swarm.on('peer', function (peer, id) { 19 | console.log('PEER',id) 20 | peer.pipe(log.replicate({ live: true })).pipe(peer) 21 | }) 22 | update() 23 | getlist() 24 | setInterval(function () { 25 | getlist() 26 | }, 1000) 27 | 28 | function update () { 29 | html.update(root, html`
30 |
31 |
32 |
33 |
34 |
`) 35 | function onsubmit (ev) { 36 | ev.preventDefault() 37 | var title = this.elements.title.value 38 | var content = this.elements.content.value 39 | kv.put(title, { body: content }, function (err, node) { 40 | if (err) return console.error(err) 41 | console.log('node=',node) 42 | }) 43 | this.reset() 44 | } 45 | } 46 | 47 | function getlist () { 48 | kv.createReadStream() 49 | .pipe(to.obj(function (row, enc, next) { 50 | console.log('row=',row) 51 | next() 52 | })) 53 | } 54 | -------------------------------------------------------------------------------- /day4.md: -------------------------------------------------------------------------------- 1 | # day 4: testing and modular frontend 2 | 3 | Learn how to write unit tests for node.js and the browser and build up a modular 4 | frontend brick by brick. 5 | 6 | Join James Halliday (substack) as we dive into javascript testing fundamentals 7 | and modular frontend development. You will: 8 | 9 | * Write unit tests that run in node.js and the browser. 10 | * Set up code coverage, build scripts, and continuous integration. 11 | * Learn cutting edge front-end architectures. 12 | * Build a modern webapp from zero. 13 | * Run the same code on the server and the client. 14 | * Automate tasks with npm scripts. 15 | 16 | ## schedule 17 | 18 | * 08:30 - system check, hello 19 | * 09:00 - assert and the test anything protocol 20 | * 09:30 - testing in the browser 21 | * 10:00 - code coverage and the AST 22 | * 10:30 - testing workflows 23 | * 11:00 - package.json scripts 24 | * 11:30 - template strings 25 | * 12:00 - lunch etc 26 | * 13:00 - routing and servers 27 | * 13:30 - diy redux 28 | * 14:00 - realtime modules 29 | * 14:30 - browser api modules 30 | * 15:00 - webaudio, webgl, etc! 31 | * 16:00 - closing notes and wrap up 32 | 33 | ## who is this for? 34 | 35 | This workshop is for people who want to know how to start testing and how to 36 | quickly whip up modern web apps without having to wade through a bunch of 37 | boilerplate or configuration. 38 | 39 | If you know javascript but you haven't started writing tests or if you want to 40 | make webdev fun again, this workshop is for you! 41 | 42 | ## prerequisites 43 | 44 | You should have some familiarity with javascript. 45 | 46 | -------------------------------------------------------------------------------- /day4/assert/test.js: -------------------------------------------------------------------------------- 1 | var assert = require('assert') 2 | var fs = require('fs') 3 | 4 | assert.equal(1+2,3) 5 | countLines(function (err, n) { 6 | assert.ifError(err) 7 | assert.equal(n, 3) 8 | }) 9 | 10 | function countLines (cb) { 11 | fs.readFile('file.txt', 'utf8', function (err, src) { 12 | if (err) cb(err) 13 | else cb(null, src.trim().split('\n').length) 14 | }) 15 | } 16 | -------------------------------------------------------------------------------- /day4/cool-test.js: -------------------------------------------------------------------------------- 1 | var test = require('tape') 2 | var h = require('hyperscript') 3 | var hyperx = require('../') 4 | var hx = hyperx(h) 5 | 6 | test(function (t) { 7 | var title = 'world' 8 | var wow = [1,2,3] 9 | var tree = hx`
10 |

hello ${title}!

11 | ${hx`cool`} 12 | wow 13 | ${wow.map(function (w) { 14 | return hx`${w}\n` 15 | })} 16 |
` 17 | t.equal(trim(tree.outerHTML), trim(` 18 |
19 |

hello world!

20 | cool 21 | wow 22 | 123 23 |
`)) 24 | t.end() 25 | function trim (s) { 26 | return s.replace(/\s+/g,'') 27 | } 28 | }) 29 | -------------------------------------------------------------------------------- /day4/hyperx.js: -------------------------------------------------------------------------------- 1 | var hyperx = require('hyperx') 2 | var html = hyperx(function (tagName, props, children) { 3 | console.log(tagName, props, children) 4 | // ... 5 | }) 6 | var n = 3 7 | console.log(html`
8 |

${n*1000}

9 |
`) 10 | -------------------------------------------------------------------------------- /day4/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "dependencies": { 3 | "acorn": "^5.0.3", 4 | "angle-normals": "^1.0.0", 5 | "bankai": "^7.6.1", 6 | "baudio": "^2.1.2", 7 | "browser-run": "^4.0.2", 8 | "choo": "^5.4.0", 9 | "concat-stream": "^1.6.0", 10 | "coverify": "^1.4.1", 11 | "ecstatic": "^2.1.0", 12 | "falafel": "^2.1.0", 13 | "glsl-dither": "^1.0.1", 14 | "glsl-hsl2rgb": "^1.1.0", 15 | "glsl-noise": "0.0.0", 16 | "glslify": "^6.0.2", 17 | "hyperx": "^2.3.0", 18 | "icosphere": "^1.0.0", 19 | "morphdom": "^2.3.2", 20 | "nyc": "^10.3.0", 21 | "regl": "^1.3.0", 22 | "regl-camera": "^1.1.2", 23 | "regl-feedback": "^1.0.0", 24 | "routes": "^2.1.0", 25 | "sheetify": "^6.0.2", 26 | "strftime": "^0.10.0", 27 | "tap": "^10.3.2", 28 | "tape": "^4.6.3", 29 | "uglify-js": "^2.8.22", 30 | "webaudio": "^2.0.0", 31 | "websocket-stream": "^4.0.0", 32 | "webworkify": "^1.4.0", 33 | "xhr": "^2.4.0", 34 | "yo-yo": "^1.4.0" 35 | } 36 | } 37 | -------------------------------------------------------------------------------- /day4/tape/index.js: -------------------------------------------------------------------------------- 1 | module.exports = function (n, cb) { 2 | setTimeout(function () { 3 | if (n == 5) cb(null, 555) 4 | else cb(null, n*111) 5 | }, 500) 6 | } 7 | -------------------------------------------------------------------------------- /day4/tape/test/browser-test/foo.js: -------------------------------------------------------------------------------- 1 | var test = require('tape') 2 | 3 | test('whatever', function (t) { 4 | t.ok(/chrome/i.test(window.navigator.appVersion)) 5 | t.equal(1+1,2) 6 | t.equal(555,5*111) 7 | t.end() 8 | }) 9 | -------------------------------------------------------------------------------- /day4/tape/test/one.js: -------------------------------------------------------------------------------- 1 | var test = require('tape') 2 | var elevenizer = require('../') 3 | var http = require('http') 4 | var concat = require('concat-stream') 5 | 6 | var server 7 | test('setup', function (t) { 8 | server = http.createServer(function (req, res) { 9 | var n = req.url.slice(1) 10 | elevenizer(n, function (err, result) { 11 | if (err) { 12 | res.statusCode = 400 13 | res.end(err) 14 | } else res.end(String(result)) 15 | }) 16 | }) 17 | server.listen(0, function () { 18 | t.end() 19 | }) 20 | }) 21 | 22 | test('single digits', function (t) { 23 | t.plan(6) 24 | testDigit(1,111) 25 | testDigit(3,333) 26 | testDigit(9,999) 27 | function testDigit (n, expected) { 28 | var req = http.request({ 29 | host: 'localhost', 30 | port: server.address().port, 31 | path: '/' + n 32 | }, function (res) { 33 | t.equal(res.statusCode, 200) 34 | res.pipe(concat(function (body) { 35 | t.equal(Number(body.toString()), expected) 36 | })) 37 | }) 38 | req.end() 39 | } 40 | }) 41 | 42 | test('teardown', function (t) { 43 | server.close(function () { 44 | t.end() 45 | }) 46 | }) 47 | -------------------------------------------------------------------------------- /day4/template.js: -------------------------------------------------------------------------------- 1 | var n = 5 2 | console.log(tag`hi 3 | n=${n} 4 | wow`) 5 | 6 | function tag (strings) { 7 | return arguments 8 | } 9 | -------------------------------------------------------------------------------- /day4/testing.md: -------------------------------------------------------------------------------- 1 | # testing 2 | 3 | writing tests in node and the browser 4 | 5 | --- 6 | # assert 7 | 8 | require('assert') 9 | 10 | * assert.ok() 11 | * assert.equal() 12 | * assert.deepEqual() 13 | * assert.notOk() 14 | * assert.notEqual() 15 | * assert.notDeepEqual() 16 | * assert.ifError() 17 | 18 | --- 19 | # problems with assert 20 | 21 | * exceptions stop execution 22 | * false positive when test blocks don't run 23 | 24 | --- 25 | # test anything protocol (TAP) 26 | 27 | old text protocol from the perl test suite (late 80s?) 28 | 29 | * test cases begin with ok/not ok 30 | * must plan out a number of assertions 31 | 32 | http://testanything.org 33 | 34 | --- 35 | # tap example 36 | 37 | ``` 38 | TAP version 13 39 | # defined-or 40 | ok 1 empty arguments 41 | ok 2 1 undefined 42 | ok 3 2 undefined 43 | ok 4 4 undefineds 44 | ok 5 false[0] 45 | ok 6 false[1] 46 | ok 7 zero[0] 47 | ok 8 zero[1] 48 | ok 9 first arg 49 | ok 10 second arg 50 | ok 11 third arg 51 | # falsy 52 | ok 12 should be equal 53 | 54 | 1..12 55 | # tests 12 56 | # pass 12 57 | 58 | # ok 59 | ``` 60 | 61 | --- 62 | # tap failing case 63 | 64 | ``` 65 | not ok 30 in 1.5 weeks 66 | --- 67 | operator: equal 68 | expected: |- 69 | '2015-04-24 09:46:01' 70 | actual: |- 71 | '2015-04-24 21:46:01' 72 | at: Test. (/home/substack/projects/parse-messy-time/test/parse.js:141:7) 73 | ... 74 | ``` 75 | 76 | --- 77 | # tap/tape modules 78 | 79 | * npm i tap # for node 80 | * npm i tape # for node and the browser 81 | 82 | --- 83 | # tap/tape api 84 | 85 | ``` js 86 | var test = require('tape') 87 | //var test = require('tap').test 88 | 89 | test('optional test name', function (t) { 90 | t.plan(2) 91 | t.ok(2 > 1) 92 | setTimeout(function () { 93 | t.equal('hi', 'h'+'i') 94 | }, 100) 95 | }) 96 | 97 | test('another test', function (t) { 98 | t.equal(1+1, 2, 'one plus one') 99 | t.end() 100 | }) 101 | ``` 102 | 103 | --- 104 | # setup and teardown 105 | 106 | ``` js 107 | var server 108 | test('setup', function (t) { 109 | server = http.createServer() 110 | server.on('listening', function () { 111 | t.end() 112 | }) 113 | }) 114 | 115 | // ... 116 | 117 | test('teardown', function (t) { 118 | server.close(function () { 119 | t.end() 120 | }) 121 | }) 122 | ``` 123 | 124 | --- 125 | # testing in the browser 126 | 127 | if you're using tape: 128 | 129 | ``` 130 | $ npm i -g browserify browser-run 131 | $ browserify test/*.js | browser-run -b chrome 132 | ``` 133 | 134 | browser-run copies console.log() to stdout 135 | 136 | --- 137 | # writing testable code 138 | 139 | * make code easy to run in varied ways 140 | * use instance variables, not global state 141 | 142 | --- 143 | ## i/o shell 144 | 145 | push I/O to the outermost layers 146 | 147 | --- 148 | # code coverage 149 | 150 | how much of your code gets run when you test 151 | 152 | --- 153 | ## abstract syntax tree 154 | 155 | data structure for code 156 | 157 | modules: 158 | 159 | * acorn 160 | * falafel 161 | 162 | --- 163 | ## coverify 164 | 165 | $ npm i -g coverify browserify browser-run 166 | 167 | in node: 168 | 169 | $ browserify -t coverify test/*.js --node | node | coverify 170 | 171 | in the browser: 172 | 173 | $ browserify -t coverify test/*.js | browser-run -b chrome | coverify 174 | 175 | --- 176 | ## nyc 177 | 178 | ``` 179 | $ npm i -g nyc 180 | $ nyc npm test 181 | $ nyc report 182 | ----------|----------|----------|----------|----------|----------------| 183 | File | % Stmts | % Branch | % Funcs | % Lines |Uncovered Lines | 184 | ----------|----------|----------|----------|----------|----------------| 185 | All files | 100 | 100 | 100 | 100 | | 186 | index.js | 100 | 100 | 100 | 100 | | 187 | ----------|----------|----------|----------|----------|----------------| 188 | ``` 189 | 190 | --- 191 | # npm scripts 192 | 193 | in package.json: 194 | 195 | ``` json 196 | { 197 | "scripts": { 198 | "build": "browserify main.js > public/bundle.js", 199 | "test": "tape test/*.js" 200 | } 201 | } 202 | ``` 203 | 204 | --- 205 | # continuous integration 206 | 207 | run tests every time you push code 208 | 209 | --- 210 | # travis-ci config 211 | 212 | git add .travis.yml 213 | 214 | ``` 215 | language: node_js 216 | node_js: 217 | - "7" 218 | - "6" 219 | - "4" 220 | ``` 221 | 222 | --- 223 | # travis-ci github web hook 224 | 225 | https://travis-ci.org/$USER/$REPO 226 | 227 | click "activate repo" 228 | 229 | --- 230 | # travis-ci badge 231 | 232 | example markdown to include in your readme: 233 | 234 | ``` 235 | [![build status](https://travis-ci.org/substack/node-browserify.svg)](https://travis-ci.org/substack/node-browserify) 236 | ``` 237 | 238 | -------------------------------------------------------------------------------- /day4/web.md: -------------------------------------------------------------------------------- 1 | # modular web dev 2 | 3 | --- 4 | # the web these days 5 | 6 | the web is these: 7 | 8 | * service workers 9 | * template strings 10 | * redux architecture 11 | 12 | but also these: 13 | 14 | * indexedDB 15 | * webaudio 16 | * webgl 17 | 18 | --- 19 | # starting from zero 20 | 21 | ``` 22 | $ echo '{}' > package.json 23 | $ npm install --save ... 24 | ``` 25 | 26 | --- 27 | # template strings 28 | 29 | ``` js 30 | var n = 5 31 | console.log(`hi 32 | n=${n} 33 | wow`) 34 | ``` 35 | 36 | * can span multiple lines 37 | * interpolation with `${...}` 38 | 39 | --- 40 | # tagged template strings 41 | 42 | fn`...` 43 | 44 | --- 45 | # hyperx 46 | 47 | ``` 48 | var hyperx = require('hyperx') 49 | var html = hyperx(function (tagName, props, children) { 50 | console.log(tagName, props, children) 51 | // ... 52 | }) 53 | var n = 3 54 | console.log(html`
55 |

${n*1000}

56 |
`) 57 | ``` 58 | 59 | --- 60 | # yo-yo/bel/morphdom 61 | 62 | dom diffing with real DOM nodes 63 | 64 | * faster in some cases than a virtual dom 65 | * interop with vanilla DOM modules 66 | 67 | --- 68 | # managing state 69 | 70 | * uni-directional flow 71 | * one state object 72 | 73 | --- 74 | # routing 75 | 76 | in server and in the browser 77 | 78 | * history.pushState 79 | 80 | ``` js 81 | window.addEventListener('click', function (ev) { 82 | var el = ev.target 83 | if (el.tagName.toUpperCase() === 'A' 84 | && el.getAttribute('href')) { 85 | // if the link can be handled by the router, 86 | // call ev.preventDefault() 87 | } 88 | }) 89 | ``` 90 | 91 | --- 92 | # choo 93 | 94 | minimal (4kb) modular redux architecture 95 | https://choo.io/ 96 | 97 | using: 98 | 99 | * yo-yo/bel/hyperx 100 | * sheetify 101 | 102 | --- 103 | # choo 104 | 105 | ``` js 106 | var html = require('choo/html') 107 | var choo = require('choo') 108 | var strftime = require('strftime') 109 | var app = choo() 110 | app.route('/', function (state, emit) { 111 | return html` 112 |

${strftime('%F %T', state.time)}

113 | ` 114 | }) 115 | app.mount('body') 116 | 117 | app.use(function (state, emitter) { 118 | state.time = 0 119 | setInterval(function () { 120 | state.time = new Date 121 | emitter.emit('render') 122 | }, 1000) 123 | }) 124 | ``` 125 | 126 | --- 127 | # building for production 128 | 129 | browserify transforms for yo-yo/bel/choo dev: 130 | 131 | * yo-yoify 132 | * unassertify 133 | * sheetify 134 | 135 | or bankai includes these by default with no config: 136 | 137 | ``` 138 | $ bankai build main.js public/ 139 | ``` 140 | 141 | --- 142 | # webgl 143 | 144 | redraws, managing state 145 | 146 | --- 147 | # webgl performance tips 148 | 149 | ``` 150 | 60fps = 1 frame / 60 seconds 151 | 1 frame = 1/60 seconds ~ 16.7 milliseconds 152 | ``` 153 | 154 | avoid allocations and the GC 155 | 156 | --- 157 | # glslify 158 | 159 | modular webgl shaders! 160 | 161 | ``` js 162 | var glsl = require('glslify') 163 | console.log(glsl` 164 | #pragma glslify: snoise = require('glsl-noise/simplex/4d') 165 | attribute vec3 position, normal; 166 | uniform mat4 projection, view; 167 | void main () { 168 | gl_Position = projection * view 169 | * (position + normal*snoise(vec4(position,time)*0.1)); 170 | } 171 | `) 172 | 173 | budo main -- -t glslify 174 | 175 | --- 176 | # audio 177 | 178 | npm i webaudio 179 | 180 | or http://studio.substack.net 181 | 182 | ``` js 183 | var baudio = require('webaudio') 184 | var b = baudio(function (t) { 185 | return Math.sin(Math.PI*2*400*t) // 400Hz 186 | }) 187 | b.play() 188 | ``` 189 | 190 | -------------------------------------------------------------------------------- /day4/web/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "scripts": { 3 | "dev": "watchify yo.js -o public/bundle.js -dv", 4 | "build": "browserify yo.js | uglifyjs | gzip > public/bundle.js.gz" 5 | }, 6 | "dependencies": { 7 | "end-of-stream": "^1.4.0", 8 | "to2": "^1.0.0" 9 | } 10 | } 11 | -------------------------------------------------------------------------------- /day4/web/public/index.html: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /day4/web/reduce.js: -------------------------------------------------------------------------------- 1 | module.exports = function (state, bus) { 2 | state.visitors = 0 3 | state.x = 0 4 | bus.emit('render') 5 | bus.on('set-visitors', function (n) { 6 | state.visitors = n 7 | bus.emit('render') 8 | }) 9 | bus.on('increment-x', function () { 10 | state.x = (state.x + 1) % 4 11 | bus.emit('render') 12 | }) 13 | } 14 | -------------------------------------------------------------------------------- /day4/web/server.js: -------------------------------------------------------------------------------- 1 | var wsock = require('websocket-stream') 2 | var http = require('http') 3 | var onend = require('end-of-stream') 4 | var ecstatic = require('ecstatic') 5 | var st = ecstatic(__dirname + '/public') 6 | var router = require('routes')() 7 | 8 | router.addRoute('GET /user/:name', function (req, res, m) { 9 | res.end('name=' + m.params.name + '\n') 10 | }) 11 | 12 | var server = http.createServer(function (req, res) { 13 | var m = router.match(req.method + ' ' + req.url) 14 | if (m) m.fn(req,res,m) 15 | else st(req,res) 16 | }) 17 | server.listen(5000) 18 | 19 | var count = 0 20 | var streams = [] 21 | wsock.createServer({ server: server }, function (stream) { 22 | streams.push(stream) 23 | console.log('CONNECTED', streams.length) 24 | count++ 25 | streams.forEach(function (s) { 26 | s.write(count + '\n') 27 | }) 28 | onend(stream, function () { 29 | var ix = streams.indexOf(stream) 30 | streams.splice(ix,1) 31 | console.log('DISCONNECTED', streams.length) 32 | }) 33 | }) 34 | -------------------------------------------------------------------------------- /day4/web/yo.js: -------------------------------------------------------------------------------- 1 | var choo = require('choo') 2 | var html = require('choo/html') 3 | var wsock = require('websocket-stream') 4 | var split = require('split2') 5 | var to = require('to2') 6 | var stream = wsock('ws://' + location.host) 7 | 8 | var app = choo() 9 | app.route('/', function (state, emit) { 10 | return html` 11 |

${state.visitors}

12 |
${state.x}
13 | 14 | ` 15 | function onclick (ev) { 16 | emit('increment-x') 17 | } 18 | }) 19 | app.mount('body') 20 | 21 | app.use(function (state, bus) { 22 | stream.pipe(split()).pipe(to(function (buf, enc, next) { 23 | bus.emit('set-visitors', Number(buf.toString())) 24 | next() 25 | })) 26 | }) 27 | app.use(require('./reduce.js')) 28 | -------------------------------------------------------------------------------- /day4/webgl/main.js: -------------------------------------------------------------------------------- 1 | var regl = require('regl')() 2 | var camera = require('regl-camera')(regl, { distance: 4 }) 3 | //var icosphere = require('icosphere') 4 | var bunny = require('bunny') 5 | var anormals = require('angle-normals') 6 | var glsl = require('glslify') 7 | var feedback = require('regl-feedback') 8 | var webaudio = require('webaudio') 9 | var b = webaudio(require('./song.js')) 10 | b.play() 11 | 12 | function createBlob (regl) { 13 | var mesh = bunny //icosphere(3) 14 | return regl({ 15 | frag: glsl` 16 | precision highp float; 17 | #pragma glslify: snoise = require('glsl-noise/simplex/4d') 18 | #pragma glslify: hsl2rgb = require('glsl-hsl2rgb') 19 | varying vec3 vnorm, vpos; 20 | uniform float time, stage; 21 | void main () { 22 | gl_FragColor = vec4((vnorm+1.0)*0.5 * vec3(0,1,0) 23 | + hsl2rgb(stage*0.3+0.1*snoise(vec4(vpos,time*0.1+stage)),1.0,0.5) 24 | ,1); 25 | } 26 | `, 27 | vert: glsl` 28 | precision highp float; 29 | #pragma glslify: snoise = require('glsl-noise/simplex/4d') 30 | attribute vec3 position, normal; 31 | uniform mat4 projection, view; 32 | varying vec3 vnorm, vpos; 33 | uniform float time, stage; 34 | uniform vec3 location; 35 | void main () { 36 | vpos = position; 37 | vnorm = normal; 38 | gl_Position = projection * view 39 | * vec4(position*0.2 + location 40 | + normal*snoise(vec4(position,time+stage))*0.1,1); 41 | } 42 | `, 43 | attributes: { 44 | position: mesh.positions, 45 | normal: anormals(mesh.cells, mesh.positions) 46 | }, 47 | uniforms: { 48 | time: regl.context('time'), 49 | location: regl.prop('location'), 50 | stage: regl.prop('stage') 51 | }, 52 | elements: mesh.cells 53 | }) 54 | } 55 | 56 | var draw = { 57 | blob: createBlob(regl), 58 | fb: feedback(regl, ` 59 | vec3 sample (vec2 uv, sampler2D tex) { 60 | return 0.97*texture2D(tex, (0.99*(2.0*uv-1.0)+1.0)*0.5).rgb; 61 | } 62 | `) 63 | } 64 | var fbtex = regl.texture() 65 | 66 | regl.frame(function () { 67 | regl.clear({ color: [0,0,0,1], depth: true }) 68 | draw.fb({ texture: fbtex }) 69 | camera(function () { 70 | draw.blob([ 71 | { location: [0,0,0], stage: 0 }, 72 | { location: [0,0,-2], stage: 1 }, 73 | { location: [0,0,2], stage: 2 } 74 | ]) 75 | fbtex({ copy: true, mag: 'linear', min: 'linear' }) 76 | }) 77 | }) 78 | -------------------------------------------------------------------------------- /day4/webgl/song.js: -------------------------------------------------------------------------------- 1 | var ms = [800,400,500,300] 2 | 3 | module.exports = function (t) { 4 | var m = ms[Math.floor(t*8)%ms.length] 5 | return 0 6 | + sin_(sin(100)+sin(50),sin(8)*0.1+1) 7 | * Math.pow((1-saw(2)*0.5-saw(8)*0.5)*0.5,12+sin(1)*8) 8 | + clamp(sin_(sin(m)+sin(1),sin(1)*0.2+saw(2)*0.4+.2)*0.1 9 | * Math.pow(((1-saw(4))*.5),8)*8)*.5 10 | 11 | function tri_ (x,t) { return Math.abs(1 - t % (1/x) * x * 2) * 2 - 1 } 12 | function tri (x) { return tri_(x,t) } 13 | function saw_ (x,t) { return t%(1/x)*x*2-1 } 14 | function saw (x) { return saw_(x,t) } 15 | function sin_ (x,t) { return Math.sin(2 * Math.PI * t * x) } 16 | function sin (x) { return sin_(x,t) } 17 | function sq_ (x,t) { return t*x % 1 < 0.5 ? -1 : 1 } 18 | function sq (x) { return sq_(x,t) } 19 | function clamp (x) { return Math.max(-1,Math.min(1,x)) } 20 | } 21 | -------------------------------------------------------------------------------- /notes.md: -------------------------------------------------------------------------------- 1 | # day 1 2 | 3 | ## bash 4 | 5 | Learn the command-line using bash. 6 | 7 | * history and philosophy 8 | * directories and files 9 | * environment variables 10 | * flags 11 | * absolute and reltive paths 12 | * interpolation, expansion, and math 13 | * file redirects and pipes 14 | * pipeline tools 15 | * quotes 16 | * scripts 17 | * permissions 18 | * jobs and processes 19 | 20 | ## vim 21 | 22 | Learn the basics of vim, a powerful command-line text editor. 23 | 24 | * understanding modes 25 | * the language of vim 26 | * moving around 27 | * search and replace 28 | * select cut paste 29 | * vi is everywhere 30 | 31 | ## regex 32 | 33 | Use regular expressions to solve problems. 34 | 35 | * metacharacters 36 | * character classes 37 | * groups and repetition 38 | * capture groups 39 | * backreferences 40 | * match, replace, split 41 | 42 | # day 2 43 | 44 | ## networking 45 | 46 | How does networking work? What is TCP, HTTP? 47 | 48 | * use curl and netcat 49 | * learn about text-based network protocols such as http, irc, and email 50 | * write tcp and http servers and clients using node.js 51 | * inspect real packets with tshark/wireshark 52 | 53 | ## streams 54 | 55 | Use the stream API in node to shuffle data around. 56 | 57 | * learn the stream types: readable, writable, transform, duplex 58 | * streaming core APIs 59 | * use stream modules from npm to build streaming pipelines 60 | * streaming transports in node and the browser: websockets, webrtc, tcp 61 | * building symmetric streaming protocols 62 | 63 | # day 3 64 | 65 | ## leveldb 66 | 67 | Use leveldb in node and the browser to persist data. Leveldb has a vast 68 | ecosystem of modular packages to enhance its functionality. 69 | 70 | * get/put/batch/createReadStream 71 | * thinking lexicographically 72 | * leveldb API conventions 73 | * handy modules: subleveldown, level-live-stream 74 | * leveldb in the browser with indexedDB 75 | * hyperlog, materialized views, and data replication 76 | 77 | ## crypto 78 | 79 | Learn how to use node core APIs and libsodium/nacl in node and the browser. 80 | 81 | * hashes 82 | * symmetric vs asymmetric crypto 83 | * keypairs 84 | * sign/verify 85 | * box/secretbox 86 | * identity systems 87 | * crypto in p2p: hyperlog, bep44, ipfs, dat, bitcoin 88 | 89 | # day 4 90 | 91 | ## testing 92 | 93 | Learn how to write tests for node and the browser. 94 | 95 | * testable API design 96 | * TAP consumers and producers 97 | * testing in the browser 98 | * testing workflows 99 | 100 | -------------------------------------------------------------------------------- /workshops.md: -------------------------------------------------------------------------------- 1 | # day 1: command-line 2 | 3 | Learn the command-line. 4 | 5 | So many reference materials, open source tools, blog posts, and job descriptions 6 | assume command-line proficiency, but the resources for learning the command-line 7 | effectively are scattered and largely unhelpful for people who aren't already 8 | familiar with how to get by in a unix system. 9 | 10 | Join James Halliday (substack) as we dive into the command-line. You will: 11 | 12 | * Become comfortable with the unix command-line using the bash shell. 13 | * Log in to remote servers to edit files and run commands. 14 | * Edit files with vim, a command-line text editor. 15 | * Search and match patterns using regular expressions. 16 | * Automate command-line tasks with their own shell scripts. 17 | * Administer user permissions and system services. 18 | 19 | ## schedule 20 | 21 | * 08:30 - system check, hello 22 | * 09:00 - commands, input/output, and some history 23 | * 09:30 - directories and files, environment variables, flags 24 | * 10:00 - pipes, redirects, scripts, interpolation 25 | * 10:30 - permissions, signals, job control 26 | * 11:00 - user accounts, ssh, public keys 27 | * 11:30 - screen 28 | * 12:00 - lunch etc 29 | * 13:00 - services, init scripts, cron 30 | * 13:30 - regular expressions, grep, sed 31 | * 14:30 - vim basics and practice 32 | * 16:00 - closing notes and wrap up 33 | 34 | We will start out the day by covering basic command-line concepts hands-on. 35 | These skills will build up into more advanced techniques and we will begin to 36 | apply our growing knowledge of the command-line and common utility programs by 37 | writing shell scripts, connecting to remote servers over ssh, and administering 38 | user accounts, permissions, and system services. 39 | 40 | After lunch, we will use regular expressions to search for patterns and format 41 | text. We will edit text files on the command-line with vim. 42 | 43 | ## who is this for? 44 | 45 | This workshop is for people who want to become effective at the command-line. 46 | Students should be comfortable computer users, but no programming or prior 47 | command-line experience is required. 48 | 49 | If you've recently learned some programming, plan to learn programming soon, or 50 | have been programming for a while but never got around to learning the 51 | command-line and a unix system in depth, this workshop is for you! 52 | 53 | ## prerequisites 54 | 55 | Bring a computer running a unix operating system such as GNU/Linux or MacOSX. 56 | 57 | If you have a Windows computer, please install Linux. Most Linux installers will 58 | let you dual boot your computer into both Linux and Windows if you want to keep 59 | Windows around. If you don't know where to start, download Ubuntu from 60 | https://www.ubuntu.com/ and install from a bootable USB stick. 61 | 62 | --- 63 | # day 2: networking and streams 64 | 65 | Learn about networking and node.js streams. 66 | 67 | Streams let you glue together sources and sinks of I/O with backpressure to 68 | produce effective data pipelines for processing data. How do streams relate to 69 | network protocols such as TCP, HTTP, and websockets? 70 | 71 | Join James Halliday (substack) as we dive into networking with streams. You 72 | will: 73 | 74 | * Use curl and netcat to send and receive network requests. 75 | * Learn the stream types: readable, writable, transform, duplex. 76 | * Use stream modules from npm to build streaming pipelines. 77 | * Learn about text-based network protocols such as http, irc, and email. 78 | * Write tcp, http, and websocket servers and clients using node.js. 79 | * Build symmetric streaming protocols. 80 | 81 | ## schedule 82 | 83 | * 08:30 - system check, hello 84 | * 09:00 - TCP, UDP, and netcat 85 | * 09:30 - text-based network protocols 86 | * 10:00 - http servers, headers, and curl 87 | * 11:00 - streaming interface basics 88 | * 11:30 - basic stream modules 89 | * 12:00 - lunch etc 90 | * 13:00 - streaming transports galore: websockets, webrtc, p2p 91 | * 13:30 - advanced streaming modules 92 | * 14:00 - implementing core streams 93 | * 15:00 - rpc, multiplexing, and symmetric protocols 94 | * 16:00 - closing notes and wrap up 95 | 96 | ## who is this for? 97 | 98 | This workshop is for people who want to learn more about streams in node.js and 99 | the network protocols and fundamentals that streams sit on top of. 100 | 101 | If you know some node.js or frontend js but want to dive deeper into how a 102 | server and client work and how to write glue around IO, this workshop is for 103 | you! 104 | 105 | ## prerequisites 106 | 107 | You should have some familiarity with javascript and the command-line. 108 | 109 | --- 110 | # day 3: leveldb and crypto 111 | 112 | Learn about crypto basics and leveldb, a modular database. 113 | 114 | Cryptography provides a foundation for secure communications and distributed systems. 115 | LevelDB provides a modular ecosystem to persist and query data in node.js and 116 | the browser. These techniques will let you build completely different kinds of 117 | webapps that are fault tolerant, work offline, and can even replicate p2p. 118 | 119 | Join James Halliday (substack) as we dive into crypto fundamentals and leveldb. 120 | You will: 121 | 122 | * Sign, verify, and encrypt with libsodium/nacl. 123 | * Use cryptographic hashes to build secure data structures. 124 | * Store data in the browser using IndexedDB wrappers. 125 | * Design modular data schemas over lexicographic keys. 126 | * Build data pipelines for live, streaming data. 127 | * Synchronize data and build indexes using the kappa architecture 128 | 129 | All of the content relies on libraries that work in both node.js and the 130 | browser. 131 | 132 | ## schedule 133 | 134 | * 08:30 - system check, hello 135 | * 09:00 - querying and updating data 136 | * 09:30 - thinking lexicographically 137 | * 10:00 - modular database design 138 | * 10:30 - leveldb in the browser 139 | * 11:00 - hashes and signing 140 | * 11:30 - merkle DAGs and secure data structures 141 | * 12:00 - lunch etc 142 | * 13:00 - asymmetric and symmetric encryption 143 | * 13:30 - logs, materialized views, and the kappa architecture 144 | * 14:30 - project: verified secure gossip network 145 | * 16:00 - closing notes and wrap up 146 | 147 | ## who is this for? 148 | 149 | This workshop is for people who are proficient in javascript and want to explore 150 | some advanced concepts and architectures. 151 | 152 | If you know your way around node.js or frontend and you want to know how to 153 | build offline-first, secure applications, this workshop is for you! 154 | 155 | A knowledge of node.js streams will be beneficial but is not strictly required. 156 | 157 | # day 4: testing and modular frontend 158 | 159 | Learn how to write unit tests for node.js and the browser and build up a modular 160 | frontend brick by brick. 161 | 162 | Join James Halliday (substack) as we dive into javascript testing fundamentals 163 | and modular frontend development. You will: 164 | 165 | * Write unit tests that run in node.js and the browser. 166 | * Set up code coverage, build scripts, and continuous integration. 167 | * Learn cutting edge front-end architectures. 168 | * Build a modern webapp from zero. 169 | * Run the same code on the server and the client. 170 | * Automate tasks with npm scripts. 171 | 172 | ## schedule 173 | 174 | * 08:30 - system check, hello 175 | * 09:00 - assert and the test anything protocol 176 | * 09:30 - testing in the browser 177 | * 10:00 - code coverage and the AST 178 | * 10:30 - testing workflows 179 | * 11:00 - package.json scripts 180 | * 11:30 - template strings 181 | * 12:00 - lunch etc 182 | * 13:00 - routing and servers 183 | * 13:30 - diy redux 184 | * 14:00 - realtime modules 185 | * 14:30 - browser api modules 186 | * 15:00 - webaudio, webgl, etc! 187 | * 16:00 - closing notes and wrap up 188 | 189 | ## who is this for? 190 | 191 | This workshop is for people who want to know how to start testing and how to 192 | quickly whip up modern web apps without having to wade through a bunch of 193 | boilerplate or configuration. 194 | 195 | If you know javascript but you haven't started writing tests or if you want to 196 | make webdev fun again, this workshop is for you! 197 | 198 | ## prerequisites 199 | 200 | You should have some familiarity with javascript. 201 | 202 | --------------------------------------------------------------------------------