├── .github └── workflows │ └── deploy.yml ├── book.org ├── readme.org └── styles.css /.github/workflows/deploy.yml: -------------------------------------------------------------------------------- 1 | # This is a basic workflow to help you get started with Actions 2 | 3 | name: Publish 4 | 5 | # Controls when the action will run. Triggers the workflow on push or pull request 6 | # events but only for the first-skeleton branch 7 | on: 8 | push: 9 | branches: [ master ] 10 | 11 | # A workflow run is made up of one or more jobs that can run sequentially or in parallel 12 | jobs: 13 | # This workflow contains a single job called "build" 14 | build: 15 | # The type of runner that the job will run on 16 | runs-on: ubuntu-latest 17 | 18 | # Steps represent a sequence of tasks that will be executed as part of the job 19 | steps: 20 | # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it 21 | - uses: actions/checkout@v2 22 | 23 | # Runs a single command using the runners shell 24 | - name: Run a one-line script 25 | run: echo Hello, world! 26 | 27 | - name: Set up Emacs 28 | uses: purcell/setup-emacs@master 29 | with: 30 | version: 26.1 31 | 32 | - name: get htmlize 33 | run: wget https://raw.githubusercontent.com/hniksic/emacs-htmlize/master/htmlize.el 34 | 35 | - name: export 36 | run: emacs book.org -Q -batch -l htmlize.el -f org-html-export-to-html 37 | # Runs a set of commands using the runners shell 38 | - name: rename 39 | run: mv book.html index.html 40 | 41 | - name: Deploy to GitHub Pages 42 | uses: peaceiris/actions-gh-pages@v3 43 | with: 44 | github_token: ${{ secrets.GITHUB_TOKEN }} 45 | publish_dir: ./ 46 | allow_empty_commit: true 47 | # enable_jekyll: true 48 | # cname: github.peaceiris.com 49 | 50 | # - name: Deploy to GitHub Pages 51 | # if: success() 52 | # uses: crazy-max/ghaction-github-pages@v2 53 | # with: 54 | # target_branch: gh-pages 55 | # build_dir: ./ 56 | # env: 57 | # GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} 58 | # - name: Bye 59 | # run: | 60 | # echo DONE! 61 | -------------------------------------------------------------------------------- /book.org: -------------------------------------------------------------------------------- 1 | #+TITLE: Shell Field Guide 2 | #+AUTHOR: Raimon Grau 3 | #+OPTIONS: ':nil *:t -:t ::t <:t H:3 \n:nil ^:nil arch:headline 4 | #+OPTIONS: author:t c:nil creator:comment d:(not "LOGBOOK") date:nil 5 | #+OPTIONS: e:t email:nil f:t inline:t p:nil pri:nil stat:t 6 | #+OPTIONS: tags:t tasks:t tex:t timestamp:t todo:t |:t 7 | #+EXCLUDE_TAGS: noexport 8 | #+KEYWORDS: bash zsh shell 9 | #+LANGUAGE: en 10 | #+SELECT_TAGS: export 11 | 12 | #+OPTIONS: html-style:nil 13 | #+SETUPFILE: https://fniessen.github.io/org-html-themes/org/theme-readtheorg.setup 14 | #+HTML_HEAD_EXTRA: 15 | 16 | #+OPTIONS: reveal_center:nil timestamp:nil 17 | #+REVEAL_THEME: black 18 | 19 | # toc:nil num:nil 20 | 21 | * Introduction 22 | This booklet is intended to be a catalog of tricks and techniques 23 | you may want to use if you're doing some sort of complex 24 | scripting. Some are useful in "The Real World (TM)", some are more 25 | playful, and might not have such direct impact in your day-to-day 26 | life. Some are pure entertainment. You'll have to judge by yourself 27 | which belong to which. I'll try to keep the rhetoric to the minimum 28 | to maximize signal/noise. 29 | 30 | The git repo is at 31 | https://github.com/kidd/scripting-field-guide/. Any feedback is 32 | greatly appreciated. Keep in mind this is not any kind of official 33 | doc. I just write MY current "state of the art" and I'll be updating 34 | the contents with useful stuff I find or discover, that are not 35 | widely explained in usual manuals/wikis. 36 | 37 | You surely have some amount of sh/bash/zsh in your stack that 38 | probably started as one-off scripts, and later grew and were 39 | copypasted everywhere in your pipelines, repos, coworkers' ~/bin, or 40 | your coworkers use for their own things (with some variations), 41 | etc. Those scripts are very difficult to kill and they have a very 42 | high mutation rate. 43 | 44 | * Which Shell? 45 | No matter if you use Linux, Mac, or Windows, you should be living 46 | most of the time in a shell to enjoy the content shown here. Some 47 | value comes from running scripts, and some comes from the daily 48 | usage and refinement of your helper functions, aliases, etc. in 49 | interactive mode. 50 | 51 | In general the examples here are meant to run in Bash or Zsh, which 52 | are compatible for the most part. 53 | * Level 54 | These examples are based on non-trivial real world code I've written 55 | using patterns I haven't seen applied in many places over the net. A 56 | few of the snippets are stolen from public repos I find interesting. 57 | Also, important scripting stuff might be missing if I don't feel 58 | I have anything to add to the generally available info around. 59 | * Patterns 60 | ** Use Shellcheck 61 | First, let's get that out of the way. This is low-hanging 62 | fruit. And you will get the most of this booklet by following it. 63 | 64 | A lot of the most common errors we usually make are well known 65 | ones. And in fact, we all usually fail in similar ways. Bash is 66 | known for being error prone when dealing with testing variable 67 | values, string operations, or flaky subshells and pipes. 68 | 69 | Installing [[https://www.shellcheck.net/][shellcheck]] will flag many of those ticking bombs for 70 | you. 71 | 72 | No matter which editor you are using, you should be able to 73 | install a plugin to do automatic checks while you're editing. 74 | 75 | In emacs' case, the package is called [[https://github.com/federicotdn/flymake-shellcheck][flymake-shellcheck]], and a 76 | quick configuration for it is: 77 | 78 | #+begin_src elisp 79 | (use-package flymake-shellcheck 80 | :ensure t 81 | :commands flymake-shellcheck-load 82 | :init 83 | (add-hook 'sh-mode-hook 'flymake-shellcheck-load)) 84 | #+end_src 85 | 86 | Shellcheck is available on most distros, so it's just an =apt=, 87 | =brew=, or =nix-env= away. 88 | 89 | ** Booleans and Conditionals 90 | In any shell, =foo && bar= will execute =bar= only if =foo= 91 | succeeded. That means that =foo= returned 0. That means that to && 92 | (which you read like "and"), 0 is true. so yes. 0 is true, and 93 | other values are false. 94 | 95 | ** Arrays 96 | Ordered list of things. 97 | #+begin_src bash 98 | foo=("ls" "/tmp/") 99 | 100 | echo ${foo[-2]} 101 | echo ${foo[-1]} 102 | echo ${foo[0]} 103 | echo ${foo[1]} 104 | echo ${foo[2]} 105 | 106 | for i in "${foo[@]}"; do 107 | echo $i 108 | done 109 | 110 | $foo 111 | ${foo[*]} 112 | ${foo[@]} 113 | echo ${#foo[*]} 114 | echo ${#foo[@]} 115 | #+end_src 116 | 117 | Are =*= and =@= equal? [[https://stackoverflow.com/questions/2761723/what-is-the-difference-between-and-in-shell-scripts][nope]]. 118 | 119 | #+begin_src bash 120 | "${foo[@]}" 121 | "${foo[*]}" 122 | #+end_src 123 | 124 | #+begin_src bash 125 | #!/bin/bash 126 | 127 | main() 128 | { 129 | echo 'MAIN sees ' $# ' args' 130 | } 131 | 132 | main $* 133 | main $@ 134 | 135 | main "$*" 136 | main "$@" 137 | 138 | ### end ### 139 | #+end_src 140 | 141 | and I run it like this: 142 | 143 | =my_script 'a b c' d e= 144 | 145 | ** Pass Arrays around 146 | #+begin_src bash 147 | a=('Track 01.mp3' 'Track 02.mp3') 148 | myfun "${a[@]}" # pass array to a function 149 | b=( "${a[@]}" ) # copy array 150 | #+end_src 151 | 152 | Read the great [[http://www.oilshell.org/blog/2016/11/06.html][Oil Shell blogpost]]. 153 | ** Slurping arrays 154 | A nice way to read a bunch of elements in one go is to use 155 | =readarray=. 156 | #+begin_src bash 157 | parse_args() { 158 | [[ $# -eq 0 ]] && die "Usage: $0 " 159 | version="$1" 160 | local version_split=$(echo $version | tr '.' '\n') 161 | readarray -t version_array <<< "$version_split" 162 | 163 | if [[ -z ${version_array[3]} ]]; then 164 | die "not enough version numbers" 165 | fi 166 | } 167 | #+end_src 168 | 169 | Even nicer would be to use IFS so we'd be able to split in one go. 170 | 171 | #+begin_src bash 172 | IFS=. read -a ver <<<"1.23.1.0" 173 | echo ${ver[0]} 174 | echo "next/${ver[0]}.${ver[1]}.x.x" 175 | #+end_src 176 | 177 | Or, use it in a destructuring fashion: 178 | #+begin_src bash 179 | get_nix_version_parts(){ 180 | local major minor patch 181 | # shellcheck disable=SC2034,SC2162 182 | IFS="." read major minor patch < <(get_nix_version) 183 | local -p 184 | } 185 | 186 | $ get_nix_version_parts 187 | major=2 188 | minor=3 189 | patch=4 190 | #+end_src 191 | https://news.ycombinator.com/item?id=24408318 192 | ** Array slices 193 | 194 | you can slice an array like this: 195 | 196 | #+begin_src bash 197 | a=(a b c d e f g) 198 | echo "${a[@]:1}" # tail of array 199 | echo "${a[@]:1:2}" # from that, take 2 elements 200 | #+end_src 201 | 202 | This is a [[https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html][parameter expansion]], same as =${a/foo/bar}= and similar 203 | replacements. As such, The =$@= and =$*= are treated a bit 204 | differently. 205 | 206 | Look at =${parameter:offset:length}= in the parameter expansion 207 | bash manual and be amazed by the specialcasing :). 208 | 209 | This concrete case is when the parameter is an indexed array name 210 | subscripted by '@' or '*'. 211 | 212 | Don't ask me why, but when the array is =$@=, it seems that it 213 | behaves a bit differently. It's 'explained' in that same part of 214 | the docs. 215 | 216 | #+begin_src bash 217 | a=(a b c d e f g) 218 | foo() { 219 | echo "${@:2}" # tail of array 220 | echo "${@:2:2}" # from that, take 2 elements 221 | } 222 | foo a b c d 223 | foo "${a[@]}" 224 | #+end_src 225 | 226 | ** Read lines from file 227 | The =read= command we used just above is part of the usual idiom to 228 | read a file line by line. 229 | 230 | #+begin_src bash 231 | while read -r line; do 232 | echo $line 233 | done < /tmp/file.txt 234 | #+end_src 235 | 236 | More related info in the [[https://mywiki.wooledge.org/BashFAQ/001][BashFAQ001]]. But it's very rare the case 237 | where I need to iterate a file line by line. 238 | 239 | ** Assign to $1,$2,$3... 240 | =set --= can be used as an incantation to assign to the positional 241 | parameters. Let me show you. 242 | 243 | #+begin_src bash 244 | set -- a b c 245 | echo $1 $2 $3 246 | echo $@ 247 | #+end_src 248 | 249 | See? here's how to "unshift" a parameter to the current arg list: 250 | 251 | #+begin_src bash 252 | set -- "injected" "$@" 253 | #+end_src 254 | 255 | ** Functions 256 | Functions are functions. They receive arguments, and they return a 257 | value. 258 | 259 | The special thing about shell functions is that they also can use 260 | the file descriptors of the process. That means that they "inherit" 261 | STDIN, STDOUT, STDERR (and maybe more). 262 | 263 | Use them. 264 | 265 | Another point is that function names can be passed as parameters, 266 | because they are passed as strings, but you can call them inside as 267 | functions again. Maybe they are better than you [[https://news.ycombinator.com/item?id=29058140][think.]] 268 | 269 | #+begin_src bash 270 | f() { 271 | $1 hi 272 | } 273 | 274 | f echo 275 | f touch # will create a file 'hi' 276 | #+end_src 277 | 278 | #+RESULTS: 279 | : hi 280 | 281 | ** Variables 282 | By default variables are global, to a file. No matter if you assign 283 | them for the first time inside a function, or at the top level. 284 | #+begin_src bash 285 | foo=3 286 | bar=$foo 287 | f() { 288 | echo $bar 289 | } 290 | f 291 | #+end_src 292 | 293 | #+RESULTS: 294 | : 3 295 | 296 | #+begin_src bash 297 | f() { 298 | bar=1 299 | } 300 | f 301 | echo $bar 302 | #+end_src 303 | 304 | #+RESULTS: 305 | : 1 306 | 307 | You make a variable local to a function with =local=. Use it as 308 | much as you can (kinda obvious). 309 | #+begin_src bash 310 | myfun() { 311 | local bar 312 | bar=3 313 | echo $bar 314 | } 315 | 316 | bar=4 317 | echo $bar 318 | myfun 319 | echo $bar 320 | #+end_src 321 | 322 | #+RESULTS: 323 | | 4 | 324 | | 3 | 325 | | 4 | 326 | ** Variable/Parameter Expansions 327 | They offer some variable manipulations using shell only, not having 328 | to create another process =sed,awk,perl=. 329 | 330 | #+begin_src bash 331 | v=banana 332 | # substitute one 333 | echo ${v/na/NA} # baNAna 334 | 335 | # substitute many 336 | echo ${v//na/NA} # baNANA 337 | 338 | # substitute from the start (think ^ in PCRE) 339 | echo ${v/#ba/NA} # NAnana 340 | 341 | # substitute from the end 342 | echo ${v/%na/NA} # banaNA 343 | 344 | # Capitalize 345 | echo ${v^} # Banana 346 | 347 | # Uppercase 348 | echo ${v^^} # BANANA 349 | #+end_src 350 | 351 | Take a read on https://tldp.org/LDP/abs/html/manipulatingvars.html 352 | and 353 | https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html 354 | for more details. 355 | 356 | And a nice non-obvious trick from here is to prefix or suffix a 357 | variable string: 358 | 359 | #+begin_src bash 360 | v=banana 361 | echo ${v/%/na} # bananana 362 | echo ${v/#/na} # nabanana 363 | #+end_src 364 | 365 | And a less obvious trick is to prefix every element of an array 366 | with a fixed string: 367 | #+begin_src bash 368 | local arr=(var1=1 var2=2) 369 | echo ${arr[*]/#/"--env "} 370 | #+end_src 371 | 372 | This will produce =--env var1=1 --env var2=2=. Super useful to be 373 | combined when building flags for docker. 374 | 375 | Another nice trick is a portable =realpath $PWD= with =${0%/*}=, 376 | which I found in [[https://github.com/picolisp/pil21/blob/master/vip][picolisp's vip]]: 377 | 378 | #+begin_src bash 379 | #!/bin/sh 380 | exec ${0%/*}/bin/picolisp ${0%/*}/lib.l @bin/vip "$@" 381 | #+end_src 382 | 383 | =${0%/*}/bin/picolisp= will replace keep the path of the current 384 | script and append =/bin/picolisp=. A cool way to get a realpath. 385 | 386 | ** Conditional Expansion 387 | 388 | #+begin_src bash 389 | local foo=${var:-${default:-}} 390 | local foo=${var:+&key=var} 391 | #+end_src 392 | 393 | ** Interpolation 394 | We previously saw that functions can be passed around as strings, 395 | and be called later on. 396 | 397 | Something that might not be obvious is that the string can be 398 | created from shorter strings, and that allows for an extra 399 | flexibility, that comes with its own dangers, but it's a very 400 | useful pattern to dispatch functions based on user input or 401 | function outputs. 402 | #+begin_src bash 403 | l=l 404 | s=s 405 | $l$s . 406 | #+end_src 407 | 408 | #+RESULTS: 409 | | book.html | 410 | | book.html~ | 411 | | book.org | 412 | | readme.org | 413 | 414 | ** dispatch functions using args 415 | A nice usage of the previous technique is using user input as a 416 | dispatching method. 417 | 418 | You've probably seen this pattern already: 419 | 420 | #+begin_src bash 421 | while [[ $# -gt 0 ]]; do 422 | 423 | case $1 in 424 | foo) 425 | foo 426 | ;; 427 | *) 428 | exit 1 429 | ;; 430 | esac 431 | shift 432 | done 433 | #+end_src 434 | 435 | And it is useful for its own good, and flexible. 436 | 437 | But for some simpler cases, we can dispatch based on the variable 438 | itself: 439 | 440 | #+begin_src bash 441 | cmd_foo() { 442 | do-something 443 | } 444 | 445 | cmd_$1 446 | #+end_src 447 | 448 | The problem with this is that in case we supply a =$1= that doesn't 449 | map to any =cmd_$1= we'll get something like 450 | 451 | #+begin_src bash 452 | bash: cmd_notexisting: command not found 453 | #+end_src 454 | 455 | ** command_not_found_handle 456 | Here's a detail on a kinda obscure bash (only bash) feature. 457 | 458 | You can set a hook that will be called when bash tries to run a 459 | command and it doesn't find it. 460 | 461 | #+begin_src bash 462 | command_not_found_handle() { 463 | echo "$1 is not a correct command. Cmds allowed:" 464 | echo "$(typeset -F | grep cmd_ | sed -e 's/.*cmd_/cmd_/')" 465 | } 466 | 467 | cmd_foo() { 468 | echo "foo" 469 | } 470 | 471 | cmd_baz() { 472 | echo "baz" 473 | } 474 | cmd_bar 475 | #+end_src 476 | 477 | You can unset the function =command_not_found_handle= to go back to 478 | the normal behavior. 479 | 480 | ** Return Values for Conditionals 481 | =if= 's test condition can use the return values of 482 | commands. That's a known thing, but a lot of code you see around 483 | relies on =[[]]= to test the return values of commands/functions 484 | anyway. 485 | 486 | #+begin_src bash 487 | if echo "foo" | grep "bar" ; then 488 | echo "found!" 489 | fi 490 | #+end_src 491 | 492 | This is much clearer than 493 | #+begin_src bash 494 | if [[ ! -z $( echo "foo" | grep "bar") ]]; then 495 | echo "found!" 496 | fi 497 | #+end_src 498 | 499 | As easy and trivial as it seems, this way of thinking pushes you 500 | forward to thinking about creating smaller functions that check the 501 | conditions and =return= 0 or non 0. It's syntactically smaller, and 502 | usually makes you play by the rules of the commands, more than just 503 | finding your way around the output strings. 504 | 505 | #+begin_src bash 506 | if less_than $package "1.3.2"; then 507 | die "can't proceed" 508 | fi 509 | #+end_src 510 | 511 | ** set variable in an "if" test 512 | Usual pattern to capture the output of a command and branch 513 | depending on its return value is: 514 | #+begin_src bash 515 | res="$(... whatever ...)" 516 | if [ "$?" -eq 0 ]; then ... 517 | ... 518 | fi 519 | #+end_src 520 | 521 | Well, you can test the return value AND capture the output at the 522 | same time! 523 | 524 | #+begin_src bash 525 | if res="$(...)"; then 526 | ... 527 | fi 528 | #+end_src 529 | 530 | Unfortunately, it doesnt' work with =local=, so you can't be 531 | defining a local var in the same line. So, the variable is either 532 | global, or you spent a line to declare it local before. Still, I 533 | think I prefer to have a line to declare the variable as local 534 | rather than having explicit =$?='s around. 535 | #+begin_src bash 536 | local var1 537 | if var1=$(f); then 538 | echo "$var1" 539 | fi 540 | #+end_src 541 | Ref: https://news.ycombinator.com/item?id=27163494 542 | 543 | ** Do work on loop conditions 544 | I've not seen it used a lot (and there might be a reason for it, 545 | who knows), =while= conditions are just plain commands, so you can 546 | put other stuff than =[]/[[]]/test= there. 547 | 548 | Heres's an idiomatic way to iterate through all the arguments of a 549 | function while consuming the =$*= array. 550 | 551 | #+begin_src bash 552 | while(($#)) ; do 553 | #... 554 | shift 555 | done 556 | #+end_src 557 | 558 | And here's a pseudo-repl that keeps shooting one-off commands. This 559 | will keep shooting =tr= commands to whatever strings you give it, 560 | with the usual rlwrap goodies. 561 | 562 | #+begin_src bash 563 | while rlwrap -o -S'>> ' tr a-z A-Z ; do :; done 564 | #+end_src 565 | 566 | Note: =:= is a nop builtin in bash. 567 | ** One Branch Conditionals 568 | The usual conditionals one sees everywhere look like =if=. 569 | #+begin_src bash 570 | if [[ some-condition ]]; then 571 | echo "yes" 572 | fi 573 | #+end_src 574 | 575 | This is all good and fine, but in the same vein of using the least 576 | powerful construct for each task, it's nice to think of the one way 577 | conditionals in the form of =&&= and =||= as a way to explicitly 578 | say that we don't want to do anything else when the condition is 579 | not met. It's a hint to the reader. 580 | 581 | #+begin_src bash 582 | some-condition || { 583 | echo "log: warning!" 584 | } 585 | 586 | other-condition && { 587 | echo "log: all cool" 588 | } 589 | #+end_src 590 | 591 | This conveys the intention of doing something *just* in one case, 592 | and that the negation of this is not interesting at all. There's a 593 | big warning you have to be aware of. The same as with lua's 594 | =... and .. or ..=, bash =||= and =&&= are not interchangeable for 595 | =if...else...end=. [[https://mywiki.wooledge.org/BashPitfalls#cmd1_.26.26_cmd2_.7C.7C_cmd3][BashWiki]] has an explanation why, but, the same 596 | as in Lua's case, if the "then" part returns false, the else will 597 | run. 598 | 599 | There are lots of references to this, but I like this recent post 600 | where it explains it for arrays in higher level languages like ruby: 601 | https://jesseduffield.com/array-functions-and-the-rule-of-least-power/ 602 | 603 | An extended article of this kind of conditionals can be found [[https://timvisee.com/blog/elegant-bash-conditionals/][here]]. 604 | ** pushd/popd 605 | pushd and popd are used to move to some directory and go back to it 606 | in a stack fashion, so nesting can happen and you never lose 607 | track. The problem is that it still is on you to have a =popd= per 608 | =pushd=. 609 | #+begin_src bash 610 | pushd /tmp/my-dir 611 | echo $PWD 612 | popd 613 | #+end_src 614 | 615 | Here's an alternative way, that at least makes sure that you close 616 | all pushd with a popd. 617 | 618 | Starting a new shell and cd-ing , will make all commands in that 619 | subshell be in that directory, and will come back to the old 620 | directory after closing the new spawned shell. 621 | 622 | #+begin_src bash 623 | (cd /tmp/my-dir 624 | ls 625 | ) 626 | #+end_src 627 | 628 | Remember to =inherit_errexit= or =set -e= inside the subshell if 629 | you need. That's a very easy trap to fall into. 630 | 631 | ** wrap functions 632 | Bash can't pass blocks of code around, but the alternative is to 633 | pass functions. More on that later. 634 | 635 | #+begin_src bash 636 | mute() { 637 | "$@" >/dev/null 638 | } 639 | 640 | mute pushd /tmp/foobar 641 | #+end_src 642 | ** use [[ 643 | Unless you want your script to be POSIX compliant, use =[[= instead 644 | of =[=. =[= is a regular command. It's like =ls=, or =true=. You can 645 | check it by searching for a file named =[= in your path. 646 | 647 | Being a normal command it always evaluates its params, like a 648 | regular function. On the other hand though, =[[= is a special bash 649 | operator, and it evaluates the parameters lazily. 650 | 651 | #+begin_src bash 652 | 653 | # [[ does lazy evaluation: 654 | [[ a = b && $(echo foo >&2) ]] 655 | 656 | # [ does not: 657 | [ a = b -a "$(echo foo >&2)” ] 658 | #+end_src 659 | Ref: https://lists.gnu.org/archive/html/help-bash/2014-06/msg00013.html 660 | 661 | ** eval? 662 | When you have mostly small functions that are mostly pure, you 663 | compose them like you'd do in any other language. 664 | 665 | In the following snippet, we are in a release script. Some step 666 | builds a package inside a docker container, another step tests a 667 | package already built. 668 | 669 | A nice way to build ubuntus, for example, is to add an ARG to the 670 | Dockerfile so we can build several ubuntu versions using the same 671 | file. 672 | 673 | It'd look like this: 674 | #+begin_src Dockerfile 675 | ARG VERSION 676 | FROM ubuntu:$VERSION 677 | 678 | RUN apt-get ... 679 | ... 680 | #+end_src 681 | 682 | We build that image and do all the building inside it, mounting a 683 | volume shared with our host, so we can extract our =.deb= file 684 | easily. 685 | 686 | After that, to do some smoke tests on the package, the idea is to 687 | install the =.deb= file in a fresh ubuntu image. 688 | 689 | Let's pick the same base image we picked to build the package. 690 | #+begin_src bash 691 | # evaluate the string "centos:$VERSION" (that comes from 692 | # centos/Dockerfile) in the current scope 693 | # DISTRO is ubuntu:18.04 694 | local VERSION=$(get_version $DISTRO) # VERSION==18.04 695 | run_test "file.deb" "$(eval echo $(awk '/^FROM /{print $2; exit}' $LOCAL_PATH/$(get_dockerfile_for $DISTRO)))" # ubuntu:18.04 696 | #+end_src 697 | 698 | The usage of eval is there to interpolate the string that we get 699 | from the =FROM= in the current environment. 700 | 701 | WARNING: You know, anything that uses =eval= is dangerous per 702 | se. Do not use it unless you know very well what you're doing AND 703 | the input is 100% under your control. Usually, more restricted 704 | commands can achieve what you want to do. In this particular case, 705 | you could use =envsubst=, or just manually replace =$\{?VERSION\}?= 706 | in a sed. 707 | 708 | #+begin_src bash 709 | test_release "$PACKAGE_PATH" $(awk '/^FROM /{print $2; exit}' $LOCAL_PATH/$(get_dockerfile_for $DISTRO) | sed -e "s/\$VERSION/$VERSION/") 710 | #+end_src 711 | 712 | Yet another way is using [[https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html][shell parameter expansions.]] 713 | #+begin_src bash 714 | 715 | var1=value 716 | echo 'this is $var1' >/tmp/f.txt 717 | f=$(cat /tmp/f.txt) 718 | echo "${f}" # this is $var1 719 | echo "${f@P}" # this is value 720 | #+end_src 721 | 722 | ** pass commands around 723 | This one uses [[*DRY_RUN][DRY_RUN]]. While refactoring a script that does some 724 | curls, we want to make sure that our refactored version does the 725 | exact same calls in the same order. 726 | 727 | #+begin_src bash 728 | compare_outputs() { 729 | export DRY_RUN=1 730 | git checkout b1 731 | $@ 2>/tmp/1.out 732 | git checkout b2 733 | $@ 2>/tmp/2.out 734 | echo "diffing" 735 | diff /tmp/1.out /tmp/2.out 736 | } 737 | compare_outputs ./release.sh -p rhel:6 -R 'internal-preview' 738 | #+end_src 739 | 740 | First we create a function =compare_outputs=, that gets a command 741 | to run as parameters. The function will run it once, redirecting 742 | the standard error to a file =/tmp/1.out=. 743 | 744 | Then, it checks out the branch that contains our refactored 745 | version, and will run the command again, redirecting standard error 746 | to =/tmp/2.out=, and will diff the two outputs. 747 | 748 | In case there's a difference between the two, =diff= will output 749 | them, and the function will return the non-zero exit value of 750 | diff. If everything went fine, =compare_outputs= will succeed. 751 | 752 | Now that we know that for these inputs the command runs fine, we 753 | want to find out if it works for other types of releases, not only 754 | internal-preview. 755 | 756 | Here I'm using zsh's global aliases to give a much more fluid 757 | interface to the commands, but you can use the regular while/for 758 | loops: 759 | 760 | #+begin_src shell 761 | alias -g SPLIT='| tr " " "\n" ' 762 | alias -g FORI='| while read i ; do ' 763 | alias -g IROF='; done ' 764 | 765 | set -e 766 | echo "ga internal-preview rc1 rc2" SPLIT FORI 767 | noglob compare_outputs ./release.sh -p rhel:8 -R "$i" 768 | IROF 769 | #+end_src 770 | 771 | So, combining the two, we can have a really smooth way of iterating 772 | over the possibilities, without really messing into the details of 773 | loops. 774 | 775 | WARNING: This approach is not robust enough to put it anywhere in 776 | production, but to write quick one off scripts is a 777 | killer. Experimenting in a shell and creating tools and 2nd order 778 | tools to make interaction faster builds a language that grows on 779 | you, and keeps improving your toolbelt. 780 | 781 | ** The Toplevel Is Hopeless 782 | Shellscripts are thought as quick one-off programs, but when they 783 | are useful, they are sticky, so you better write them from the 784 | start as if it would be permanent. The upfront cost is very low 785 | anyway. Structure the script like a regular app. 786 | 787 | Bash is extremely permissive in what it allows to be coded and 788 | ran. By default, failures do not make the program exit or throw an 789 | exception (no exceptions). And for some reason, the common 790 | usage of shellscripts is to put everything in the top level. Don't 791 | do that. Do the least possible things in the toplevel. 792 | 793 | A way to improve the defaults, is setting a bunch of flags that 794 | make the script stricter, so it fails on many situations you'd 795 | want to stop anyway because something went wrong. 796 | #+begin_src bash 797 | #!/usr/bin/env bash 798 | set -eEuo pipefail 799 | shopt -s inherit_errexit 800 | 801 | main() { 802 | parse_args 803 | validate_args 804 | do_things 805 | cleanup 806 | } 807 | 808 | main "$@" 809 | #+end_src 810 | 811 | Ref: https://dougrichardson.us/2018/08/03/fail-fast-bash-scripting.html 812 | 813 | Although... [[http://mywiki.wooledge.org/BashFAQ/105][Why doesn't set -e do what I expected?]] 814 | 815 | ** Check your deps 816 | Giving useful information to the users will help them using the 817 | script, and you debugging it. Script dependencies is a common use 818 | case that we'll do it in a nice way. 819 | 820 | #+begin_src bash 821 | deps() { 822 | for dep in "$@"; do 823 | mute command -v "$dep" || die "$dep dependency missing" 824 | done 825 | } 826 | 827 | main() { 828 | deps jq curl 829 | # ... 830 | } 831 | #+end_src 832 | 833 | ** source files 834 | =source= is like =require= or =import= in some programming 835 | languages. It evaluates the sourced file in the context of the 836 | current script, so you get all definitions in your environment. 837 | 838 | It's simple, but it helps you get used to modularize your code into 839 | libraries. 840 | 841 | Be careful, it's very rudimentary, and it will be overwriting old 842 | vars or functions if names clash. There's no namespacing happening 843 | there. 844 | 845 | #+begin_src bash 846 | source file.sh 847 | 848 | # the same 849 | . file.sh 850 | #+end_src 851 | 852 | ** Use Scripts as a Libs 853 | A python-inspired way of using scripts as loadable libraries is to 854 | check whether the current file was the one that was called 855 | originally or it's being just sourced. 856 | 857 | Again, no side effects in load time makes this functionality 858 | possible. otherwise, you're on your own. 859 | #+begin_src bash 860 | # Allow sourcing of this script 861 | if [[ $(basename "$(realpath "$0")") == "$(basename "${BASH_SOURCE}")" ]]; then 862 | setup 863 | parse_args "$@" 864 | main 865 | fi 866 | #+end_src 867 | ** Tmpfiles Everywhere 868 | Your script is not going to run alone. Don't assume paths are 869 | fixed or known. 870 | 871 | CI/CD Pipelines run many jobs in the same node and files can start 872 | clashing. 873 | 874 | Make use of =$(mktemp -d /tmp/foo-bar.XXXXX)=. If you have to patch a 875 | file, do it in a clean fresh copy. Don't modify files in old paths 876 | 877 | If you HAVE TO modify paths, do it idempotently. But really, don't do it. 878 | 879 | #+begin_src bash :eval no 880 | git_clone_tmp() { 881 | local repo=${1:?repo is required} 882 | local ref=${2:?ref is required} 883 | tmpath=$(mktemp -d "/tmp/cloned-$repo-XXXXX") 884 | on_exit "rm -rf $tmpath" 885 | git clone -b ${ref} $repo $tmpath 886 | } 887 | #+end_src 888 | 889 | CAVEAT: You have to manually delete the directory if you want it cleaned. 890 | 891 | Here's an article with very good advice on [[https://www.netmeister.org/blog/mktemp.html][tempfiles]]. 892 | ** Cleanup tasks with trap 893 | =trap= is used to 'subscribe' a callback when something happens. 894 | Many times it's used on exit. It's a good thing to cleanup tmpdirs after your script 895 | exits, so you can use the output of =mktemp -d= and subscribe a cleanup 896 | function for it. 897 | 898 | #+begin_src bash 899 | on_exit() { 900 | rm -rf $1 901 | } 902 | local tmpath=$(mktemp -d /tmp/foo-bar.XXXXX) 903 | trap "on_exit $tmpath" EXIT SIGINT 904 | #+end_src 905 | 906 | ** array of callbacks on_exit 907 | Level up that pattern, we can have a helper to add callbacks to run 908 | on exit. Get used to these kind of patterns, they are super 909 | powerful and save you lots of manual bookkeeping. 910 | 911 | #+begin_src bash 912 | 913 | ON_EXIT=() 914 | EXIT_RES= 915 | 916 | function on_exit_fn { 917 | EXIT_RES=$? 918 | for cb in "${ON_EXIT[@]}"; do $cb || true; done 919 | return $EXIT_RES 920 | } 921 | 922 | trap on_exit_fn EXIT SIGINT 923 | 924 | function on_exit { 925 | ON_EXIT+=("$@") 926 | } 927 | 928 | local v_id=$(docker volume create) 929 | on_exit "docker volume rm $v_id" 930 | # Use your v_id knowing that it'll be available during your script but 931 | # will be cleaned up before exiting. 932 | #+end_src 933 | 934 | ** stacktrace on error 935 | Here's a nice helper for debugging errors in bash. In case of non-0 936 | exit, it prints a stacktrace. 937 | #+begin_src bash 938 | set -Eeuo pipefail 939 | trap stacktrace EXIT 940 | stacktrace() { 941 | rc="$?" 942 | if [ $rc != 0 ]; then 943 | printf '\nThe command "%s" triggerd a stacktrace:\n' "$BASH_COMMAND" 944 | for i in $(seq 1 $((${#FUNCNAME[@]} - 2))); do 945 | j=$((i+1)); 946 | printf '\t%s: %s() called in %s:%s\n' "${BASH_SOURCE[$i]}" "${FUNCNAME[$i]}" "${BASH_SOURCE[$j]}" "${BASH_LINENO[$i]}"; 947 | done 948 | fi 949 | } 950 | 951 | #+end_src 952 | ref: https://news.ycombinator.com/item?id=26644110 953 | ** Dots and colons allowed in function names! 954 | A way to split the namespace is to have libs define functions with 955 | their own namespace. 956 | 957 | I've gotten used to use dots or colons as namespace separator. 958 | #+begin_src bash 959 | semver.greater() { 960 | # ... 961 | } 962 | #+end_src 963 | or 964 | #+begin_src bash 965 | semver:greater() { 966 | # ... 967 | } 968 | #+end_src 969 | ** make steps of the process as composable as possible by using "$@" 970 | By using =$@= to pass commands as parameters around you can get to 971 | a degree of composability that allows for a nice chaining of 972 | commands. 973 | 974 | Here's a very simple version of =watch=. See how you can =every 2 975 | ls -la=. I think that style is called Bernstein Chaining. But I'm 976 | not sure if it's exactly the same. It also looks like currying or 977 | partial evaluation to me if you squint a little bit. 978 | 979 | #+begin_src bash 980 | every() { 981 | secs=$1 982 | shift 983 | while true; do 984 | "$@" 985 | sleep $secs; 986 | done 987 | } 988 | #+end_src 989 | 990 | 991 | As you know by now, bash doesn't pass blocks of code around, but 992 | the alternative is to pass function names. 993 | 994 | #+begin_src bash 995 | mute() { 996 | $@ >/dev/null 2>/dev/null 997 | } 998 | mute ls 999 | #+end_src 1000 | 1001 | So now we can create the most useless command composition ever: 1002 | 1003 | #+begin_src bash 1004 | every 1 mute echo hi 1005 | #or 1006 | mute every 1 echo hi 1007 | #+end_src 1008 | 1009 | For the particular redirection problem, another option is to use 1010 | aliases. Redirects can be written anywhere on your CLI (not just at 1011 | the end), so the following will work using a plain alias: 1012 | #+begin_src bash 1013 | alias mute='>/dev/null 2>/dev/null' 1014 | mute ls 1015 | #+end_src 1016 | 1017 | 1018 | 1019 | - https://www.oilshell.org/blog/2017/01/13.html 1020 | ** do_times/foreach_* 1021 | shellscripts are highly side-efffecty, and even though the scoping 1022 | of variables is not very empowering, you can get a limited amount 1023 | of decomposition of loops by passing function names. 1024 | 1025 | This is a lame example, but I hope it shows the use case, it allows 1026 | you to group already existing functions while taking advantage of a 1027 | fixed looping iterator, and leaving traces of the current loop vars 1028 | in the global "variable" environment. 1029 | 1030 | #+begin_src bash 1031 | create_user() { 1032 | uname="u$1" # leave uname in the global env so later functions see it 1033 | http :8080/users name="$uname" 1034 | } 1035 | 1036 | create_pet() { 1037 | pname="p$1" 1038 | http :8080/users/$uname/pets name="$pname" 1039 | } 1040 | 1041 | create_bundle() { 1042 | create_user $1 1043 | create_pet $1 1044 | } 1045 | 1046 | do_times() { 1047 | local n=$1; shift 1048 | for i in $(seq $n); do 1049 | "$@" $i 1050 | done 1051 | } 1052 | 1053 | do_times 15 create_bundle 1054 | #+end_src 1055 | 1056 | A bit more complex is runnning a command to every repo in an org: 1057 | 1058 | #+begin_src bash 1059 | run_tests() { 1060 | ./ci/test.sh 1061 | } 1062 | 1063 | foreach_repo_with_index() { 1064 | local counter=0 1065 | local repos=$(http https://api.github.com/users/$1/repos) 1066 | shift 1067 | for entry in $(echo $repos | jq -r '.[].git_url'); do 1068 | (git_clone_tmp $entry master 1069 | cd $tmpath 1070 | "$@" $counter $entry 1071 | ) 1072 | ((counter=counter+1)) 1073 | done 1074 | } 1075 | 1076 | foreach_repo_with_index kidd run_tests 1077 | #+end_src 1078 | 1079 | ** <(foo) and >(foo) 1080 | Some commands ask for files as inputs. And sometimes you have that 1081 | file, but sometimes you're only creating that file to pass it to 1082 | the command. In those cases, creating temporary files is not 1083 | necessary if you use =<(cmd)=. Here's a way to diff the output of 2 1084 | commands without putting them in a temporary file. 1085 | 1086 | #+begin_src bash 1087 | diff <(date) <(date) 1088 | diff <(date) <(sleep 1; date) 1089 | #+end_src 1090 | 1091 | The same happens with outputs. Commands that ask you for a 1092 | destination file. You can trick them by using =>(command)= as a 1093 | file. A nice trick is to use =>(cat)= to know what's going on 1094 | there. Also useful to send stuff to the clipboard =>(xclip)= before 1095 | running something on the output. 1096 | 1097 | What the shell does in those cases is to bind a file descriptor of 1098 | the process created inside =< or >= to the first process. 1099 | 1100 | You can experiment with those using commands like =echo <(pwd)=. 1101 | 1102 | In Zsh you can use =m-x expand-word= to see the file descriptors 1103 | being expanded. 1104 | 1105 | A way to peek into a huge pipe is to =tee >(cat)= 1106 | 1107 | ** Modify behavior by linking files 1108 | Imagine you want to remove the =.bash_history= file and not record 1109 | history of commands. You don't can't avoid bash to write to that 1110 | file, but what you can do is to link =.bash_history= to 1111 | =/dev/null=. That way, bash will keep appending things to the 1112 | expected output file, without noticing that it is really writing to 1113 | =/dev/null=. The trick is to use: =ln -sf /dev/null .bash_history=. 1114 | 1115 | ** Modify a file on the fly, without changing it on disk 1116 | If you want to use a slightly modified version of a file for a 1117 | single use, and then throw it away, consider the following: 1118 | 1119 | Everywhere that you should pass =file=, you could pass =<(cat 1120 | orig_file)= instead, because they have the same content, only the 1121 | second version is ephemeral. 1122 | 1123 | If you want a modified version of it, we can attach more commands 1124 | to the subshell. 1125 | 1126 | Example of a command that reuses your =~/.psqlrc= and adds a line 1127 | in the end to make the session read-only. 1128 | 1129 | #+begin_src bash 1130 | ropsql() { 1131 | PSQLRC=<( 1132 | cat ~/.psqlrc && 1133 | echo 'set session characteristics as transaction read only;') \ 1134 | psql "$@" 1135 | } 1136 | #+end_src 1137 | 1138 | ** Use xargs 1139 | Continuing with other ways of plumbing commands into other 1140 | commands, there's =xargs=. Some commands work seamlessly with 1141 | pipes, by taking inputs from stdin and printing to stdout. But 1142 | some others like to work with files, and they ask for their 1143 | parameters in their args list. For example, =evince=. It wouldn't 1144 | be even expected to cat a pdf and pass it to evince through 1145 | stdin. 1146 | 1147 | In general, to convert from this pattern: =cmd param= to =echo 1148 | param| cmd=, xargs can be helpful. Look at its man page to know how 1149 | to split or batch args in multiple =cmd= calls. 1150 | 1151 | Xargs is helpful for parallelizing work. You should look at its man 1152 | page, but just know it can help in running parallel processes 1153 | (check =-P= in its man). 1154 | 1155 | Other tips on this great [[https://www.oilshell.org/blog/2021/08/xargs.html][Guide to xargs]]. 1156 | 1157 | ** xargs + paste 1158 | =paste= can be thougth as =zip=, interleaving the outputs of 1159 | several commands line by line. 1160 | 1161 | #+begin_src bash 1162 | paste <(./cmd1) <(./cmd2) | xargs -L1 ./cmd3 1163 | #+end_src 1164 | 1165 | Ref: https://news.ycombinator.com/item?id=29845232 1166 | 1167 | ** paste? 1168 | We just saw a use of =paste=. But this command can do all sorts of 1169 | crazy stuff joining lines. 1170 | 1171 | #+begin_src bash 1172 | paste - - -d, < file # joins every 2 lines with comma 1173 | paste - - - -d, < file # joins every 3 lines with comma 1174 | paste -s -d",\n" file #joins 1 and 2 lines with comma, 2 and 3 with \n 1175 | #+end_src 1176 | 1177 | ** Scripts as filters 1178 | Strive to create scripts that can be used as filters. Perl had 1179 | =while(<>) { ... }= that would open a file line by line, or get 1180 | input from stdin depending on the actual arguments to the file. 1181 | 1182 | In shellscripting, we can simulate (part of) it using 1183 | #+begin_src bash 1184 | while IFS= read -r line 1185 | do 1186 | echo "$line" 1187 | done < "${1:-/dev/stdin}" 1188 | #+end_src 1189 | 1190 | Look [[https://stackoverflow.com/questions/6980090/how-to-read-from-a-file-or-standard-input-in-bash][here]] for more detailed explanations and variants. 1191 | 1192 | Another way to create filters is to use `"${@:-cat}"`. 1193 | 1194 | #+begin_src bash 1195 | o2e () { 1>&2 "${@:-cat}"; } # out to err 1196 | e2o () { 2>&1 "${@:-cat}"; } # err to out 1197 | #+end_src 1198 | 1199 | ** Read or print to stdin inside a pipe 1200 | You can use /dev/tty to explicitly specify that some input or 1201 | output has to come from tty (being screen or keyboard). 1202 | 1203 | This is useful for code that might be called inside a pipeline, 1204 | but you never want the contents of the pipe to be the ones that 1205 | are =read= or =echoed=. 1206 | 1207 | Here's a snippet that defines a function fzf (if fzf is not 1208 | previously defined), that will naively replace fzf. 1209 | 1210 | #+begin_src bash 1211 | # fzf is either fzf or a naive version of it 1212 | # the input is a sed line number, so it can be 1213 | # single number: 42 1214 | # range: 1,10 1215 | # separate lines: 10p;50p 1216 | mute command -v fzf || 1217 | fzf() { 1218 | local in=$(cat) 1219 | for p in "${@}"; do 1220 | [ "$p" = "-0" ] && [ "$(echo "$in" | wc -l)" -eq 1 ] && [ "" = "$in" ] && return 1 1221 | [ "$p" = "-1" ] && [ "$(echo "$in" | wc -l)" -eq 1 ] && [ "" != "$in" ] && echo "$in" && return 1222 | done 1223 | # https://superuser.com/questions/1748550/read-from-stdin-while-piping-to-next-command 1224 | cat -n <(echo "$in") >/dev/tty 1225 | read -n num /dev/tty [{"foo":"2009-02-13T23:31:30Z","bar":"2009-02-13T23:31:31Z"}] 1402 | jq_pretty_dates() { 1403 | # From the default ($@) append every element (/%/) with 1404 | # |=todateiso8601). That, joined by |. And that becomes the "filter" to jq. 1405 | jq "$(join_by "|" "${@/%/|=todateiso8601}")" 1406 | } 1407 | #+end_src 1408 | 1409 | There are quite a few interesting commands in this [[https://medium.com/circuitpeople/aws-cli-with-jq-and-bash-9d54e2eabaf1'][AWS+JQ post]]. 1410 | 1411 | Some other cool tips from [[https://news.ycombinator.com/item?id=31141415][This HN thread]]: 1412 | 1413 | #+begin_src bash 1414 | echo "FFFF\nDDDD" | jq -R # reads raw text as strings. 1 line -> 1 string 1415 | echo "FFFF\nDDDD" | jq -R 'capture("(?.*)")' # Capture groups as json elements 1416 | #+end_src 1417 | 1418 | You can get pretty advanced with jq, and some say it's a decent 1419 | substitute for =sed= as a line oriented editor. 1420 | 1421 | Some more advanced tutorials [[https://programminghistorian.org/en/lessons/json-and-jq][here]] and [[https://remysharp.com/drafts/jq-recipes][here]]. 1422 | 1423 | ** use jq . <<<"$var" 1424 | Newlines aren't allowed in json strings, and you might get a jq 1425 | error like: 1426 | 1427 | "parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 148, column 99" 1428 | 1429 | Sometimes, strings inside json might have encoded newlines like =\n=. If you would use 1430 | 1431 | =echo $var | jq .= , bash would interpolate the \n inside the json 1432 | string and when it gets to jq, it's an invalid json. 1433 | 1434 | If instead you use =echo -E=, or =<<<"$var"=, things go smoothly. 1435 | 1436 | =jq . <<<"$var"= 1437 | 1438 | ** pass flags as a splatted array 1439 | There's quite a bit to chew on this example. First of all, the core 1440 | pattern is to build up your commandline options with an array, and 1441 | splat it in the final command line. For complex commands like 1442 | =docker= where you easily have 10+ flags it's a visual aid, and 1443 | also opens up the opportunities for reusing or abstracting sets of 1444 | options to logical blocks. 1445 | 1446 | Once it's an array, we can add elements conditionally to that array 1447 | depending on the current run, and build the line that we'll be 1448 | running in the end. 1449 | 1450 | #+begin_src bash 1451 | # Allows Ctrl-C'ing on interactive shells 1452 | INTERACTIVE= 1453 | if [[ -t 1 ]]; then INTERACTIVE="-it"; fi 1454 | 1455 | local flags=( 1456 | # We mount it as read-only, so we make sure we are not writing anything 1457 | # in there, and that everything is explicitly defined 1458 | "-v $LOCAL_PATH/build-dir:/build-dir:ro,delegated" 1459 | "-v $OUTPUT_DIR:/output:rw,consistent" 1460 | "-v $tmp_dir:/tmp/work:rw,delegated" 1461 | ) 1462 | if [[ -n $LOCAL_PATH ]]; then 1463 | flags+=("-v $(realpath $LOCAL_PATH)/overrides/my-other-file:/build-dir/build.json:ro") 1464 | flags+=("-e LOCAL_PATH=/tmp/local") 1465 | fi 1466 | 1467 | local v_id=$(docker volume create) 1468 | flags+=("-v $v_id:/tmp/build") 1469 | on_exit "docker volume rm $v_id" 1470 | 1471 | docker run --rm $INTERACTIVE ${flags[*]} $image touch /tmp/build/foo.txt 1472 | 1473 | docker run --rm $INTERACTIVE ${flags[*]} fpm:latest fpm-build /tmp/build/foo.txt 1474 | on_exit "chown_cache $tmp_dir" 1475 | #+end_src 1476 | 1477 | In this example we see another cool trick. Mounting a volume in 2 1478 | differrent containers, so not for the purpose of sharing a local 1479 | file/dir with the host but to share it between themselves. In that 1480 | case, the 2 containers don't even coexist temporarily, but use the 1481 | volume as a conveyor belt, passing it from container to container, 1482 | and each one applies "its thing". 1483 | 1484 | After all the mess, someone has to cleanup everything, but we know 1485 | how to do it with =on_exit= trick. 1486 | 1487 | ** inherit_errexit 1488 | bash 4.4+ , you can =shopt -s inherit_errexit=, and subshells will 1489 | inherit the errexit flag value. meaning that if you =set -Ee=, 1490 | anything that runs inside a subshell will throw an error at the 1491 | moment any command exits with =!=0=. 1492 | ** GNU Parallel 1493 | I can't recommend [[https://www.gnu.org/software/parallel/][parallel]] enough. The same as xargs, but in a 1494 | much more flexible way, parallel lets you run various jobs at a 1495 | time. If you have this tool into account, it doesn't just speed up 1496 | your runtimes, but it will force you write cleaner code. Parallel 1497 | execution will test your scripts so if they are not using 1498 | randomized tmp working directories, things will clash, etc... 1499 | 1500 | Parallel in itself is such a hackerfriendly tool it deserves to be 1501 | deeply learned. You can use it just locally to run a process per 1502 | core, you can send jobs to several machines connected via a simple 1503 | ssh, you can bind tmux or sqlite to it, or you can write a trivial 1504 | job queuing system. 1505 | 1506 | Man pages and official examples are a goldmine. 1507 | 1508 | ** HEREDOCS 1509 | - Basic usage of heredocs: 1510 | #+begin_src bash 1511 | echo < ${deploy_dir}" 1546 | ls -tdr ~/my-project/artifacts/* | head -n -5 | xargs rm -rf # remove all but latest 5 directories 1547 | ln -nfs "\$HOME/my-project/artifacts/${deploy_dir}/my-project-0.1.0-SNAPSHOT-standalone.jar" "\$HOME/my-project/artifacts/current.jar" 1548 | cat ~/my-project/artifacts/current.pid | xargs kill 1549 | java -jar ~/my-project/artifacts/current.jar &>/dev/null & 1550 | lastpid=\$(echo \$!) 1551 | echo \$lastpid > ~/my-project/artifacts/current.pid 1552 | EOSSH 1553 | } 1554 | #+end_src 1555 | 1556 | Refs: https://unix.stackexchange.com/questions/60688/how-to-defer-variable-expansion 1557 | 1558 | ** HERESTRINGS 1559 | It's the stripped down version of HEREDOCS. Inline a single string 1560 | (or output of a single command) as an input string. 1561 | 1562 | It's [[https://stackoverflow.com/questions/18116343/whats-the-difference-between-here-string-and-echo-pipe][kinda similar]] to what you could do with a pipe. 1563 | #+begin_src bash 1564 | cat <<<"HELLO" 1565 | cat <<<$(echo "HELLO") 1566 | echo "ECHOPIPE" | /bin/cat 1567 | echo "ECHOPIPE" | /bin/cat <(seq 5) 1568 | echo "ECHOPIPE" | /bin/cat <(seq 5) - 1569 | echo "ECHOPIPE" | /bin/cat <(seq 5) <<<"HERESTRING" 1570 | echo "ECHOPIPE" | /bin/cat <(seq 5) - <<<"HERESTRING" 1571 | #+end_src 1572 | 1573 | ** __DATA__ 1574 | I loved the Perl [[https://perldoc.perl.org/perldata#Special-Literals][__END__ and __DATA__]], features and realized it's 1575 | possible to do it in shellscripts. 1576 | 1577 | You can append to the current file. Here's an example of a super 1578 | simple bookmark "manager": 1579 | #+begin_src bash 1580 | #!/usr/bin/env sh 1581 | 1582 | cmd_add() { 1583 | shift 1584 | echo "$@" >> "$0" 1585 | } 1586 | 1587 | cmd_go() { 1588 | sed '0,/^__DATA__$/d' "$0" | 1589 | dmenu -i -l 20 | 1590 | rev | cut -f1 -d' ' | rev | 1591 | xargs xdg-open 1592 | } 1593 | 1594 | main() { 1595 | cmd_${1:-go} $@ 1596 | } 1597 | 1598 | main $@ 1599 | exit 1600 | 1601 | __DATA__ 1602 | r/emacs https://www.reddit.com/r/emacs 1603 | #+end_src 1604 | ** Simple Templating 1605 | For trivial templating, there's no need for any external tool 1606 | except for just variable interpolation. This is quite obvious, but 1607 | I use it quite often to generate repetitive bash code itself 1608 | 1609 | #+begin_src bash 1610 | #!/usr/bin/env bash 1611 | is=("obj1" "obj2") 1612 | js=("resource1" "resource2") 1613 | 1614 | for i in "${is[@]}"; do 1615 | for j in "${js[@]}"; do 1616 | echo "insert into t (obj, resource) values ('$i', '$j');" 1617 | done 1618 | done 1619 | #+end_src 1620 | 1621 | ** Secrets 1622 | As, [[https://smallstep.com/blog/command-line-secrets/][Smallstep article]] explains, you should be careful when passing 1623 | around secret data like tokens between processes. 1624 | 1625 | Env vars are not really safe, and there are a few tricks you can 1626 | use to cover your assets. 1627 | 1628 | ** Escape single quotes in single quoted strings 1629 | Escaping single quoted strings doesn't work the way you'd think. 1630 | 1631 | #+begin_src bash 1632 | # does not work 1633 | a='hi \'bob\'' 1634 | #+end_src 1635 | 1636 | Use =$''= strings like: 1637 | #+begin_src bash 1638 | # works! 1639 | a=$'hi \'bob\'' 1640 | echo $a 1641 | #+end_src 1642 | 1643 | Extra explanations: [[https://stackoverflow.com/questions/1250079/how-to-escape-single-quotes-within-single-quoted-strings][StackOverflow]] 1644 | 1645 | ** any, every, lambdas, closures, and eval 1646 | Bash has no closures, and it's not "functional", but we're going to 1647 | see how we can combine some tricks and get decent high-level functions 1648 | 1649 | Here's an implementation of =any= and =every= 1650 | 1651 | #+begin_src bash 1652 | any() { 1653 | local pred="$1"; shift 1654 | for i in "$@"; do 1655 | if $pred $i; then 1656 | return 0 1657 | fi 1658 | done 1659 | return 1 1660 | } 1661 | 1662 | every() { 1663 | local pred="$1"; shift 1664 | for i in "$@"; do 1665 | if ! $pred $i; then 1666 | return 1 1667 | fi 1668 | done 1669 | return 0 1670 | } 1671 | #+end_src 1672 | 1673 | The usage is by passing a callback function that will return 0 or 1. 1674 | 1675 | #+begin_src bash 1676 | usage() { 1677 | echo "help message" 1678 | } 1679 | 1680 | is_help() { 1681 | [ "$1" = "--help" ] 1682 | } 1683 | 1684 | any is_help "$@" && usage 1685 | #+end_src 1686 | 1687 | That's already cool. 1688 | 1689 | Did you know that functions can start also with dashes? 1690 | 1691 | #+begin_src bash 1692 | usage() { 1693 | echo "help message" 1694 | } 1695 | 1696 | --help() { 1697 | [ "$1" = "--help" ] 1698 | } 1699 | 1700 | any --help "$@" && usage 1701 | #+end_src 1702 | 1703 | That also works. It's kinda obfuscated a bit, but hey.... 1704 | 1705 | We can also, with the help of =eval=, create on demand functions. They 1706 | are global, but as we're not going to multithread, we can assume it's 1707 | ok. 1708 | 1709 | #+begin_src bash 1710 | usage() { 1711 | echo "help message" 1712 | } 1713 | 1714 | any_eq() { 1715 | eval "_() { [ $1 = \$1 ] ; }" 1716 | shift 1717 | any _ "$@" 1718 | } 1719 | 1720 | any_eq "--help" "$@" && usage 1721 | # or, 1722 | any_eq --help "$@" && usage 1723 | #+end_src 1724 | 1725 | And the last trick, using =command_not_found_handle=, we can get into 1726 | a kind of rails generators for these helpers 1727 | 1728 | #+begin_src bash 1729 | command_not_found_handle() { 1730 | IFS=_ read -a cmd <<<"$1" 1731 | [ "eq" = "${cmd[0]}" ] && eval "$1() { [ \"${cmd[1]}\" = \"\$1\" ] ; }" 1732 | "$@" 1733 | } 1734 | any eq_--help "$@" && usage 1735 | #+end_src 1736 | 1737 | All these last metaprogramming tricks are not thread safe, and even 1738 | though it's mostly ok if you use them in isolation, maybe pipes or 1739 | coprocs would mess up your inline functions. 1740 | 1741 | * Interactive 1742 | ** Save your small scripts 1743 | Rome wasn't built in a day, and like having a journal log, most of 1744 | the little scripts you create, once you have enough discipline will 1745 | be useful for some other cases, and your functions will be 1746 | reusable. 1747 | 1748 | Save your scripts into files early on, instead of crunching 1749 | everything in the repl. learn how to use a decent editor that 1750 | shortens the feedback cycle as much as possible. 1751 | ** Increased Interactivity 1752 | Knowing your shell's shortcuts for interactive use is a must. The 1753 | same way you learned to touchtype and you learned your editor, you 1754 | should learn all the shortcuts for your shell. Here's some of them. 1755 | 1756 | | key | action | 1757 | |--------+-------------------------------------| 1758 | | Ctrl-r | reverse-history-search | 1759 | | C-a | beginning-of-line | 1760 | | C-e | end-of-line | 1761 | | C-w | delete-word-backwards | 1762 | | C-k | kill-line (from point to eol) | 1763 | | C-y | paste last killed thing | 1764 | | A-y | previous killed thing (after a c-y) | 1765 | | C-p | previous-line | 1766 | | C-n | next-line | 1767 | | A-. | insert last agument | 1768 | | A-/ | dabbrev-expand | 1769 | 1770 | A written form of =A-.= is =$_=. It retains the last argment and 1771 | puts it in $_. =test -f "FILE" && source "$_"=. 1772 | 1773 | ** Aliases 1774 | Aliases are very simple substitutions of commands for a sequence of 1775 | other commands. Usual example is 1776 | 1777 | #+begin_src bash 1778 | alias ls='ls --auto-color' 1779 | #+end_src 1780 | 1781 | Now let's move on to the interesting stuff. 1782 | 1783 | ** functions can generate aliases 1784 | 1785 | Aliases live in a global namespace for the shell, so no matter 1786 | where you define them, they take effect globally, possibly 1787 | overwriting older aliases with the same name. 1788 | 1789 | Well, it's not lexical scope (far from it), but using aliases you 1790 | can create a string that snapshots the value you want, and capture 1791 | it to run it later. 1792 | 1793 | Some fun stuff: 1794 | 1795 | - aliasgen. Create an alias for each directory in 1796 | ~/workspace/. This is superceeded by =CDPATH=, but the trick is 1797 | still cool. 1798 | #+begin_src bash 1799 | aliasgen() { 1800 | for i in ~/workspace/*(/) ; do 1801 | DIR=$(basename $i) ; 1802 | alias $DIR="cd ~/workspace/$i"; 1803 | done 1804 | } 1805 | aliasgen 1806 | #+end_src 1807 | 1808 | - a make a shortcut to the current directory. 1809 | #+begin_src bash 1810 | function a() { alias $1=cd\ $PWD; } 1811 | 1812 | mkdir -p /tmp/fing-longer 1813 | cd /tmp/fing-longer 1814 | a fl 1815 | cd / 1816 | fl 1817 | echo $PWD # /tmp/fing-longer 1818 | #+end_src 1819 | A man can dream... 1820 | 1821 | 1822 | - unhist. functions can create aliases, and functions can receive 1823 | functions as parameters (as a string (function name)), so we can 1824 | combine them to advice existing functions. 1825 | #+begin_src bash 1826 | unhist () { 1827 | alias $1=" $1" 1828 | } 1829 | unhist unhist 1830 | unhist grep 1831 | unhist rg 1832 | 1833 | noglobber() { 1834 | alias $1="noglob $1" 1835 | } 1836 | noglobber http 1837 | noglobber curl 1838 | noglobber git 1839 | 1840 | #+end_src 1841 | 1842 | #+RESULTS: 1843 | 1844 | - Problem: These commands do not compose. Combination of 2 of those 1845 | doesn't work, because the second acts just on the textual 1846 | representation that it received, not the current value of the 1847 | alias. 1848 | 1849 | # Solution : 1850 | # #+begin_src bash 1851 | # alias-to() { 1852 | # alias $1 | sed -e "s/.*='//" -e "s/'\$//" 1853 | # } 1854 | 1855 | # aliasappend() { 1856 | # local cmd 1857 | # if alias $1 >/dev/null; then 1858 | # cmd=$(alias-to $1) 1859 | # else 1860 | # cmd=$1 1861 | # fi 1862 | # echo $cmd 1863 | # } 1864 | 1865 | # muter() { 1866 | # local c 1867 | # c=$(aliasappend $1) 1868 | # alias $1="$c >/dev/null" 1869 | # } 1870 | 1871 | # unhist() { 1872 | # local c 1873 | # c=$(aliasappend $1) 1874 | # alias $1=" $c" 1875 | # } 1876 | 1877 | # unhist ls 1878 | # muter ls 1879 | # ls 1880 | # #+end_src 1881 | ** Override (advice?) common functions 1882 | Overriding commands is generally a bad practice as it violates the 1883 | principle of least surprise, but there might be occasions (mostly 1884 | in your local machine) where you can integrate awesome finetunnings 1885 | to your toolbelt. 1886 | 1887 | Here we're going to get the original docker binary file 1888 | location. After that we declare a function called =docker= that 1889 | will proxy the parameters to the original =docker= program UNLESS 1890 | you're calling =docker run=. In that case, we're injecting a mouted 1891 | volume that mounts =/root/.bash_history= of the container to a file 1892 | hosted in the host (duh). That's a pretty cool way of keeping a 1893 | history of your recent commands in your containers, no matter how 1894 | many times you start and kill them. 1895 | 1896 | #+begin_src bash 1897 | DOCKER_ORIG=$(which docker) 1898 | docker () { 1899 | if [[ $1 == "run" ]]; then 1900 | shift 1901 | $DOCKER_ORIG run -v $HOME/.shared_bash_history:/root/.bash_history "$@" 1902 | else 1903 | $DOCKER_ORIG "$@" 1904 | fi 1905 | } 1906 | #+end_src 1907 | 1908 | I'm particularly fond of this trick, as it saved me tons of 1909 | typing. But at a personal level, it was mindblowing that sharing 1910 | this around the internet caused the most disparity of opinions. 1911 | Also, I recently read the great book "Docker in Practice" by [[https://github.com/ianmiell][Ian 1912 | Miell]] and there's a snippet that is 99.9% like the one I 1913 | created myself. That was a very cool moment. 1914 | 1915 | ** Faster iteration on pipes 1916 | When testing complex pipelines: 1917 | 1918 | - Make them pure (no side effects). 1919 | - One command per line. 1920 | - End lines with the pipe character. 1921 | - During development, end the pipeline with =cat=. 1922 | 1923 | I usually use =watch -n1 'code.sh'= in a split window so I see the 1924 | results of what I'm doing. The advantage of 1925 | 1926 | #+begin_src bash 1927 | curl https://www.example.com/videos/ | 1928 | pup 'figure.mg > a attr{href}' | 1929 | head -1 | 1930 | xargs -I{} curl -L "https://www.example.com/{}" | 1931 | pup 'script' | 1932 | grep file: | 1933 | sed -e "s/.*\(http[^ \"']*\).*/\1/" | 1934 | # xargs vlc | 1935 | cat 1936 | #+end_src 1937 | 1938 | Over 1939 | #+begin_src bash 1940 | curl https://www.example.com/videos/ \ 1941 | | pup 'figure.mg > a attr{href}' \ 1942 | | head -1 \ 1943 | | xargs -I{} curl -L "https://www.example.com/{}" \ 1944 | | pup 'script' \ 1945 | | grep file: \ 1946 | | sed -e "s/.*\(http[^ \"']*\).*/\1/" \ 1947 | # xargs vlc # doesn't work 1948 | #+end_src 1949 | 1950 | Is that you can comment out lines on the former one, but you can't 1951 | do that on the latter. The =cat= trick makes it so that you have an 1952 | 'exit' point, and you don't have to comment that one. Also, some 1953 | editors will indent the first one correctly, while you'll have to 1954 | manually indent the second one. 1955 | 1956 | Small wins that compose just fine :) 1957 | 1958 | ** Use ${var?error msg} on templates 1959 | If you write something to be copypasted by your user and filled in, 1960 | instead of ==, try =${var?You need to set var}=. it allows for the user to set 1961 | the variable in the environment without having to replace inline, 1962 | and if the user forgets any, the shell will barf. 1963 | 1964 | ** Idempotent functions 1965 | Being able to call the same function multiple times even if you just 1966 | called it, without having to account for "oh, did this already happen 1967 | or not?" but instead have a mental model of "I need this to have 1968 | happened after this function call", and consider it done, is very 1969 | liberating from the code perspective. 1970 | 1971 | Also, there are subgoals, like caching slow functions, downloads, etc. 1972 | 1973 | *** nop subsequent calls 1974 | "My favourite shell scripting function definition technique: 1975 | idempotent functions by redefining the function as a noop inside 1976 | its body: 1977 | 1978 | (The `true;` body is needed by some shells, e.g. bash, and not 1979 | others, e.g. zsh.) " -- [[https://news.ycombinator.com/item?id=27729397][chrismorgan]] 1980 | 1981 | #+begin_src bash 1982 | foo() { 1983 | foo() { true; } 1984 | echo "This bit will only be executed on the first foo call" 1985 | } 1986 | #+end_src 1987 | *** Download once 1988 | A function might be sideffecty, as in "download files". These sort of 1989 | cases, you want to download the file only if you haven't downloaded it 1990 | yet. 1991 | 1992 | =cache= here gets a cache "id" where it'll store the results of =http= 1993 | and next time, when called with the same param, it will reuse the saved version. 1994 | 1995 | This pattern applies to many kinds of functions, so with slight 1996 | modifications you can adapt it. 1997 | 1998 | #+begin_src bash 1999 | cache() { 2000 | local path=$CACHE_DIR/$1 2001 | [[ -f $path ]] && cat $path && return 0 2002 | 2003 | shift 2004 | $@ | tee $path 2005 | 2006 | return 1 2007 | } 2008 | 2009 | get_repos() { 2010 | local url="https://api.github.com/search/repositories?q=user:$USER&access_token=$GITHUB_TOKEN&per_page=100&page=$1" 2011 | cache $USER-repos-$1 http $url 2012 | } 2013 | #+end_src 2014 | 2015 | *** memoize with ttl, with ttl 2016 | What if you want to cache for some time? 2017 | 2018 | The tricks here: 2019 | 2020 | - Use a file as a sentinel for the cache. 2021 | - Create a file, set it's modification date to now()+12hours 2022 | - check with `-nt` (newer than) and a temporary file (to have a 2023 | =now()= value in "file" format). 2024 | 2025 | #+begin_src bash 2026 | hmacache="/tmp/.hmacache" 2027 | is_recent_cache() { 2028 | [ "$hmacache" -nt <(echo 1) ] 2029 | } 2030 | update_cache() { 2031 | touch -m -t "$(date -v+12H +"%Y%m%d%H%M.%S")" "$hmacache" 2032 | } 2033 | 2034 | main() { 2035 | #... 2036 | if ! is_recent_cache ; then 2037 | if ! curl -s https://hello.ts.net > /dev/null; then 2038 | die "Tailscale must be connected to start Harbormaster" 2039 | fi 2040 | # other slow checks 2041 | update_cache 2042 | fi 2043 | } 2044 | #+end_src 2045 | 2046 | * Debugging 2047 | ** adding =bash= to a script to debug 2048 | You can add =bash= inside any script, and it'll add a sort of 2049 | a breakpoint, allowing you to check the state of the env and 2050 | manually call functions around. 2051 | 2052 | If you orgainse your code in small functions, it's easy to add 2053 | breakpoints by just spawning bash processes inside your script. 2054 | 2055 | This works also inside docker containers (if you provide =-ti= flag 2056 | on run). 2057 | 2058 | Let's see some usual uses of docker and how we can debug our 2059 | scripts there: 2060 | 2061 | #+begin_src bash 2062 | # leaves you at a shell to fiddle if all is in place after build 2063 | docker run -it mycomplex-image bash 2064 | 2065 | # Runs /tmp/file.sh from the host inside. That's cool to make the 2066 | # container less airtight. Even if the image is not originally meant 2067 | # to, you can even override it and 'monkeypatch' the file with the one 2068 | # from the host anyway. 2069 | docker run -it -v $PWD:/tmp/ mycomplex-image /tmp/file.sh 2070 | 2071 | # So now you can really add wtv you want there. 2072 | echo 'bash' >>$PWD/file.sh 2073 | 2074 | # run+open shell at runtime to inspect the state of the script 2075 | docker run -it -v $PWD:/tmp/ mycomplex-image /tmp/file.sh 2076 | #+end_src 2077 | 2078 | ** DRY_RUN 2079 | #+begin_src bash 2080 | if [[-n "$DRY_RUN" ]]; then 2081 | curl () { 2082 | echo curl "$@" 2083 | } 2084 | fi 2085 | #+end_src 2086 | use =command curl= to force the command, not the alias or anything 2087 | 2088 | If you have many functions to mock, and you want to overabstract (bad thing): 2089 | 2090 | #+begin_src bash 2091 | 2092 | mock() { 2093 | for m in ${MOCKS[@]//,/ }; do 2094 | eval "$m() { echo \"mocked $m\" "\$@"; }" 2095 | done 2096 | } 2097 | 2098 | MOCKS=curl,ls script.sh 2099 | #+end_src 2100 | 2101 | 2102 | 2103 | ** Cheap debugging flag 2104 | 2105 | #+begin_src bash 2106 | optargs "V" option; do 2107 | case $option in 2108 | V) 2109 | set -xa 2110 | ;; 2111 | #+end_src 2112 | 2113 | ** explore a pipe with tee >(some_command) | 2114 | the =>()= is not very easy to use. Very few places where it 2115 | fits. Here's a nice pipe inspector though, using =tee >(cat 1>&2)= 2116 | trick. 2117 | #+begin_src bash 2118 | plog() { 2119 | # tee >(cat 1>&2) 2120 | local msg=${1:-plog} 2121 | tee >(sed -e "s/^/[$msg] /" 1>&2) 2122 | } 2123 | alias -g 'PL'=' |plog ' #zsh only 2124 | 2125 | echo "a\nb" PL foo | tr 'a-z' 'A-Z' PL bar 2126 | # output: 2127 | # [foo] a # stderr 2128 | # [foo] b # stderr 2129 | # A # stdout 2130 | # B # stdout 2131 | # [bar] A # stderr 2132 | # [bar] b # stderr 2133 | #+end_src 2134 | 2135 | 2136 | ref: https://stackoverflow.com/questions/17983777/shell-pipe-to-multiple-commands-in-a-file 2137 | 2138 | ** tee+sudo 2139 | If you wat to store a file in a root-owned dir, in the middle of 2140 | your pipeline, instead of running the whole thing as root, you can 2141 | use =sudo tee file=: 2142 | #+begin_src bash 2143 | ls | grep m >/usr/local/garbage # fail 2144 | ls | grep m | sudo tee /usr/local/garbage # success! 2145 | #+end_src 2146 | ** quoting 2147 | Bash: To get a quoted version of a given string, here's what you can do: 2148 | #+begin_src bash 2149 | 2150 | # this is my "string" I want to 'comment "on"' 2151 | !:q 2152 | #+end_src 2153 | 2154 | That gives us ='#this is my "string" I want to '\''comment 2155 | "on"'\'''=. Neat! 2156 | 2157 | I just found this trick [[https://til.simonwillison.net/til/til/bash_escaping-a-string.md][here]]. From the associated HN thread: 2158 | #+begin_src bash 2159 | function bashquote () { 2160 | printf '%q' "$(cat)" 2161 | echo 2162 | } 2163 | #+end_src 2164 | 2165 | Zsh: If you're on zsh, =a-'= quotes the current line. 2166 | * zsh-only 2167 | ** Word spliting 2168 | Word splitting works differently by default in zsh than in bash. 2169 | #+begin_src bash 2170 | foo="ls -las" 2171 | $foo 2172 | #+end_src 2173 | This works in bash, but zsh will not split by words. To make zsh 2174 | expand by words, there are 2 ways: =setopt SH_WORD_SPLIT= and 2175 | =${=foo}=. zsh has =unsetop= command, which allows to scope where 2176 | you want the expansions to happen. =unsetop SH_WORD_SPLIT=. 2177 | 2178 | The problem with both solutions is that none of them are compatible 2179 | with bash, so you'll be cornering yourself to "this only works in 2180 | zsh". A way to overcome this is to use arrays, which are expanded 2181 | in the same way in both shells. 2182 | 2183 | Or, use the same hack as you'll see later with noglob. 2184 | 2185 | Refs: 2186 | - https://stackoverflow.com/questions/6715388/variable-expansion-is-different-in-zsh-from-that-in-bash 2187 | - http://zsh.sourceforge.net/FAQ/zshfaq03.html#l18 2188 | ** globbing 2189 | In zsh, getting a list of files that match some characteristics is 2190 | doable using globbing. Bash has globbing also, but in a less sophisticated way. 2191 | 2192 | The basic structure of a =glob= is =pattern(qualifiers)=. Patterns 2193 | can contain: 2194 | - strings: they do exact match 2195 | - wildcards: =*=, =?=, =**/= 2196 | - character classes: =[0-9]= 2197 | - choices: =(.pdf|.djvu)= 2198 | 2199 | The qualifiers are extra constraints you put on the matches. There 2200 | are lots of different qualifiers. Look at =zshexpn= for the 2201 | complete list. The ones I use more are: 2202 | 2203 | - =.= Files 2204 | - =/= Directories 2205 | - =om[numberhere]=. Nth latest modified 2206 | 2207 | ** Global aliases: 2208 | Zsh has 3 kinds of aliases: normal, global, and suffix. Normal ones 2209 | behave the same as in bash: They expand only in the first token of 2210 | a line. 2211 | 2212 | #+begin_src bash 2213 | alias foo=ls 2214 | foo foo # -> ls: cannot access 'foo': No such file or directory 2215 | #+end_src 2216 | 2217 | Global aliases expand anywhere in your cli, and they can expand to 2218 | anything. 2219 | 2220 | #+begin_src bash 2221 | alias -g foo=ls 2222 | foo foo # -> ls: cannot access 'ls': No such file or directory 2223 | #+end_src 2224 | 2225 | These are some aliases I have in my ~/.zshrc that help me use a 2226 | shell in a more fluid way. 2227 | 2228 | #+begin_src bash 2229 | alias -g P1='| awk "{print \$1}"' 2230 | alias -g P2='| awk "{print \$2}"' 2231 | alias -g P3='| awk "{print \$3}"' 2232 | alias -g P4='| awk "{print \$4}"' 2233 | alias -g P5='| awk "{print \$5}"' 2234 | alias -g P6='| awk "{print \$6}"' 2235 | alias -g PL='| awk "{print \$NF}"' 2236 | alias -g PN='| awk "{print \$NF}"' 2237 | alias -g HL='| head -20' 2238 | alias -g H='| head ' 2239 | alias -g H1='| head -1' 2240 | alias -g TL='| tail -20' 2241 | alias -g T='| tail ' 2242 | alias -g T1='T -1' 2243 | #alias -g tr='-ltr' 2244 | alias -g X='| xclip ' 2245 | alias -g TB='| nc termbin.com 9999 ' 2246 | alias -g L='| less -R ' 2247 | alias -g LR='| less -r ' 2248 | alias -g G='| grep ' 2249 | alias -g GI='| grep -i ' 2250 | alias -g GG=' 2>&1 | grep ' 2251 | alias -g GGI=' 2>&1 | grep -i ' 2252 | alias -g GV='| grep -v ' 2253 | alias -g V='| grep -v ' 2254 | alias -g TAC='| tac ' 2255 | alias -g DU='du -B1' 2256 | 2257 | alias -g E2O=' 2>&1 ' 2258 | alias -g NE=' 2>/dev/null ' 2259 | alias -g NO=' >/dev/null ' 2260 | 2261 | alias -g WC='| wc -l ' 2262 | 2263 | alias -g J='| noglob jq' 2264 | alias -g JQ='| noglob jq' 2265 | alias -g jq='noglob jq' 2266 | alias -g JL='| noglob jq -C . | less -R ' 2267 | alias -g JQL='| noglob jq -C . | less -R ' 2268 | alias -g XMEL='| xmlstarlet el' 2269 | alias -g XML='| xmlstarlet sel -t -v ' 2270 | 2271 | alias -g LYNX="| lynx -dump -stdin " 2272 | alias -g H2T="| html2text " 2273 | alias -g TRIM="| xargs " 2274 | alias -g XA='| xargs -d"\n" ' 2275 | alias -g XE="| xargs e" 2276 | alias -g P="| pick " 2277 | alias -g PP="| percol | xargs " 2278 | alias -g W5="watch -n5 " 2279 | alias -g W1="watch -n1 " 2280 | 2281 | 2282 | alias -g CB="| col -b " 2283 | alias -g NC="| col -b " 2284 | alias -g U='| uniq ' 2285 | alias -g XT='urxvt -e ' 2286 | alias -g DM='| dmenu ' 2287 | alias -g DMV='| dmenu -i -l 20 ' 2288 | 2289 | alias -g ...='../..' 2290 | alias -g ....='../../..' 2291 | alias -g .....='../../../..' 2292 | 2293 | alias -g l10='*(om[1,10])' 2294 | alias -g l20='*(om[1,20])' 2295 | alias -g l5='*(om[1,5])' 2296 | alias -g l='*(om[1])' 2297 | alias -g '**.'='**/*(.)' 2298 | alias -g lpdf='*.pdf(om[1])' 2299 | alias -g lpng='*.png(om[1])' 2300 | 2301 | alias -g u='*(om[1])' 2302 | 2303 | alias lsmov='ls *.(mp4|mpg|mpeg|avi|mkv)' 2304 | alias lspdf='ls *.(pdf|djvu)' 2305 | alias lsmp3='ls *.mp3' 2306 | alias lspng='ls *.png' 2307 | #+end_src 2308 | Now, some sequences of words can start making sense: 2309 | 2310 | - =lspdf -tr TL DM XA evince= 2311 | - =docker exec -u root -ti $(docker ps -q H1) bash= 2312 | - =docker ps DM P1 XA docker stop= 2313 | - =docker ps P1 XA docker stop= (P1 XA does an ad-hoc [[https://bash-my-aws.org/pipe-skimming/][pipe-skimming]]) 2314 | 2315 | ** Expansion of global aliases 2316 | Let's dig deeper. 2317 | #+begin_src zsh 2318 | alias -g DOCK='docker ps 2..N P P1' 2319 | DOCK # works fine. Echoes the id of the chosen container 2320 | docker stop DOCK # does not work , because it expands to docker stop docker ps 2..N P P1 2321 | docker stop $(DOCK) # works fine again 2322 | alias -g DOCK='$(docker ps 2..N P P1)' 2323 | docker stop DOCK # yay! 2324 | #+end_src 2325 | 2326 | So, you can bind words to expansion-time results of the aliases. It 2327 | feels like a very powerful thing, to have this "compile time" 2328 | expansions. Reminds me of CL's symbol-macrolet, or IMMEDIATE Forth 2329 | words. 2330 | ** Autocomplete 2331 | Writting smart autocompletion scripts is not easy. 2332 | 2333 | zsh supports =compdef _gnu_generic= type of completion, which gets 2334 | you very far with 0 effort. 2335 | 2336 | When autocompleting after a =-= in the commandline, if your command 2337 | is configured like =compdef _gnu_generic mycommand=, zsh will call 2338 | the script with =--help= and parse the output, trying to find 2339 | flags, and will use them as suggestions. It's really great. 2340 | 2341 | The compromise is to write a decent "--help" for your script. Which 2342 | is cool because your user will love it too, and you just have to 2343 | write it once. 2344 | 2345 | The completion is not context aware though, so you can't 2346 | autocomplete flags after the first non-flag argument. It seems this 2347 | could be improved in zsh-land, by asking for the --help like 2348 | =mycommand args-so-far --help=. But it doesn't work like that. 2349 | 2350 | 2351 | #+begin_src zsh 2352 | #!/usr/bin/env bash 2353 | # The script can be bash-only, while the completion work in zsh-only 2354 | set -Eeuo pipefail 2355 | 2356 | help() { 2357 | echo " -h,--help Show help" 2358 | echo " -c,--command Another thing" 2359 | } 2360 | 2361 | if [ "$1" == "--help" ]; 2362 | help 2363 | fi 2364 | #+end_src 2365 | 2366 | Now you can play with =mycommand -=. Amazing, wow. 2367 | 2368 | ** Create helpers and generate global aliases automagically 2369 | Borrowing a bit from Perl, a bit from Forth, and a bit from 2370 | PicoLisp, I've come to create a few helpers that abstract words 2371 | into a bit higher level concepts. Unifying the option selectors is 2372 | one, and then, other line oriented operations like =chomp, from, 2373 | till=. 2374 | 2375 | #+begin_src bash 2376 | pick() { 2377 | if [ -z "$DISPLAY" ]; then 2378 | percol || fzf || slmenu -i -l 20 2379 | else 2380 | dmenu -i -l 20 2381 | fi 2382 | } 2383 | alias -g P='| pick' 2384 | 2385 | globalias() { 2386 | alias -g `echo -n $1 | tr '[a-z]' '[A-Z]'`=" | $1 " 2387 | } 2388 | 2389 | globalias fzf 2390 | 2391 | # uniquify column 2392 | function uc () { 2393 | awk -F" " "!_[\$$1]++" 2394 | } 2395 | globalias uc 2396 | 2397 | function from() { perl -pe "s|.*?$1|\1|" } 2398 | globalias from 2399 | 2400 | function till() { sed -e "s|$1.*|$1|" } 2401 | globalias till 2402 | 2403 | function chomp () { sed -e "s|.$||" } 2404 | globalias chomp 2405 | #+end_src 2406 | 2407 | Again, it's a pity those do not compose well. Just be well 2408 | organized, or build a more elaborate hack so you can compose 2409 | aliases with some sort of confidence. It'll always be a hack 2410 | though. 2411 | ** suffix aliases don't have to match a filename 2412 | zsh has another type of aliases called "suffix alias". Those alias 2413 | allow you to define programs to open/run file types. 2414 | #+begin_src shell 2415 | alias -s docx="libreoffice" 2416 | #+end_src 2417 | 2418 | With this said, if you write a name of a file ending with =docx= as 2419 | the first token in a command line, it will use libreoffice to open 2420 | it. 2421 | 2422 | #+begin_src shell 2423 | invoice1.docx 2424 | # will effectively call libreoffice invoice1.docx 2425 | #+end_src 2426 | 2427 | The trick here is that the parser doesn't check that the file is 2428 | indeed an existing file. It can be any string. 2429 | 2430 | Let's look at an example of it. 2431 | 2432 | #+begin_src shell 2433 | alias -s git="git clone" 2434 | #+end_src 2435 | 2436 | In this case, we can easily copy a =git@github.com:.....git= from a 2437 | browser, and paste it into a zsh console. Then, zsh will run that 2438 | "file" with the command =git clone=, effectively cloning that 2439 | repository. 2440 | 2441 | Cool, ain't it? 2442 | 2443 | ** noglob 2444 | zsh has more aggressive parameter expansion, to the level that 2445 | =[,],...= have special meanings, and will be interpreted and 2446 | expanded before calling the final commands in your shell. 2447 | 2448 | There are commands that you don't want ever expanded , for example, 2449 | when using =curl=, it's much more likely that an open bracket will 2450 | be ment to be there verbatim rather than expanded. 2451 | 2452 | Zsh provides a command to quote the following expansions. And it's 2453 | called noglob. 2454 | #+begin_src bash 2455 | noglob curl http://example.com\&a[]=1 2456 | #+end_src 2457 | 2458 | ** make noglob 'transparent' to bash 2459 | zsh and bash are mostly compatible, but there's a few things not 2460 | supported in bash. =noglob= is one of them. To build a shim 2461 | inbetween, an easy way is to just create a =~/bin/noglob= file 2462 | #+begin_src bash 2463 | $* 2464 | #+end_src 2465 | 2466 | ** glob nested expansion 2467 | In https://news.ycombinator.com/item?id=26175894 there's a nice 2468 | advanced example: 2469 | 2470 | #+begin_src text 2471 | Variable expansion syntax, glob qualifiers, and history modifiers can 2472 | be combined/nested quite nicely. For example, this outputs all the 2473 | commands available from $PATH: `echo 2474 | ${~${path/%/\/*(*N:t)}}`. `${~foo}` is to enable glob expansion on the 2475 | result of foo. `${foo/%/bar}` substitutes the end of the result of foo 2476 | to "bar" (i.e. it appends it); when foo is an array, it does it for 2477 | each element. In `/*(*N:t)`, we're adding the slash and star to the 2478 | paths from `$path`, then the parentheses are glob qualifiers. `*` 2479 | inside means only match the executables, `N` is to activate NULL_GLOB 2480 | for the match so that we don't get errors for globs that didn't match 2481 | anything, `:t` is a history mod used for globs that returns just the 2482 | "tail" of the result, i.e. the basename. IIRC, bash can't even nest 2483 | multiple parameter expansions; you need to save each step separately. 2484 | #+end_src 2485 | ** Some extra shortcuts for nice things 2486 | - =alt-'= quotes the current line. It's like =quotemeta=. great to 2487 | help you fight double and triple quoting when writing scripts. 2488 | - =alt-#= comment/uncomment and execute. Nice way to store the current line 2489 | for later and reach out to it again to run it for real. 2490 | - =ctrl-o= kill-current-line, wait for a command, and paste. 2491 | 2492 | ** =() 2493 | Zsh has =<()= and =>()= like Bash, but it also has ==()=. This 2494 | varant is similar to =<()= but instead of creating a temporary 2495 | pipe, it creates a temporary file. That is useful if we want to run 2496 | commands that require a file instead of a pipe (most times, because 2497 | it uses lseek to go through it). 2498 | 2499 | Node is an example of this. 2500 | #+begin_src shell 2501 | node <(echo 'setTimeout(() => console.log("foo"), 400)') # fails 2502 | node =(echo 'setTimeout(() => console.log("foo"), 400)') # works! 2503 | #+end_src 2504 | 2505 | Or, 2506 | 2507 | #+begin_src bash 2508 | docker run --rm -ti -v =(echo "OHAI"):/tmp/foo ubuntu cat /tmp/foo 2509 | #+end_src 2510 | 2511 | * TODO patterns 2512 | ** just use cat/netcat/pipes with <() 2513 | 2514 | - input 2515 | 2516 | =python logger.py executable= will run the executable and monitor it 2517 | for error messages. Depending on the error messages it will be doing. 2518 | 2519 | In order to test it, I want to run it with my own output. So what I 2520 | do is =python logger.py cat=. That way I can type my stuff there, 2521 | and even better, I can use a stream from the shell. =myexecutable | 2522 | python logger.py cat= still works. 2523 | 2524 | *** what's the unifying theory behind all that? 2525 | It's still not clear to me how they relate, but the feeling is 2526 | that there's a common thread ruling all those commands. as if they 2527 | generalize over the same things, or just a couple of very 2528 | interrelated things. 2529 | 2530 | =echo= is to =cat= what =|= is to =xargs=. and =<()= and =>()= are 2531 | able to make static files be dynamic streams. putting =cat= and 2532 | =echo= inside =<()= seem like either a noop, or a leap in what can 2533 | be done there. Still have to figure it out. 2534 | 2535 | <(grep a file.txt) , | xargs , cat, echo 2536 | | you-have\it-wants | executable | file | stream | 2537 | | executable | X | <(exe) | exe \vert | 2538 | | file | <(cat file) | X | cat file \vert | 2539 | | stream | cat | <(grep foo file.txt) | X | 2540 | 2541 | 2542 | - output 2543 | 2544 | Most of those can be tested with and =tee=. Sometimes you would like 2545 | the output to be an output to a file to be extramassaged. 2546 | 2547 | | you-have\it-wants | executable | file | stream | 2548 | | executable | X | >() | | 2549 | | file | | X | | 2550 | | stream | | >(cat) | X | 2551 | 2552 | lnav <(tail -F /my/logfile-that-gets-rotated-or-truncated.log) 2553 | cat <(date) 2554 | 2555 | ** redirects & streams 2556 | - https://catonmat.net/ftp/bash-redirections-cheat-sheet.pdf 2557 | - https://catonmat.net/bash-one-liners-explained-part-three 2558 | - https://github.com/miguelmota/bash-streams-handbook 2559 | - https://www2.dmst.aueb.gr/dds/sw/dgsh/ 2560 | - https://wiki.bash-hackers.org/howto/redirection_tutorial 2561 | ** The $0 pattern 2562 | - https://www.reddit.com/r/oilshell/comments/f6of85/four_more_posts_in_shell_the_good_parts/ 2563 | ** use git staging area to diff outputs of commands 2564 | - https://chrismorgan.info/blog/make-and-git-diff-test-harness/ 2565 | - [[https://news.ycombinator.com/item?id=32896051][Content based change detection with Make]] 2566 | - https://news.ycombinator.com/item?id=40333481 2567 | - https://news.ycombinator.com/item?id=32441602 2568 | 2569 | ** coprocs 2570 | - https://stackoverflow.com/questions/7942632/how-to-extrace-pg-backend-pid-from-postgresql-in-shell-script-and-pass-it-to-ano/8305578#8305578 2571 | - https://unix.stackexchange.com/questions/86270/how-do-you-use-the-command-coproc-in-various-shells 2572 | - https://mbuki-mvuki.org/posts/2021-05-30-memoize-commands-or-bash-functions-with-coprocs/ 2573 | * links 2574 | - https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html 2575 | - https://www.gnu.org/software/bash/manual/html_node/ 2576 | - https://tldp.org/LDP/abs/html/ 2577 | - https://mywiki.wooledge.org/BashPitfalls 2578 | - [[https://www.youtube.com/watch?v=sCZJblyT_XM][Gary Bernhardt. The Unix Chainsaw]] 2579 | - https://github.com/spencertipping/. This guy has some bash sick snippets 2580 | - https://news.ycombinator.com/item?id=23765123 2581 | - https://medium.com/@joydeepubuntu/functional-programming-in-bash-145b6db336b7 2582 | - https://www.youtube.com/watch?v=yD2ekOEP9sU 2583 | - http://catern.com/posts/pipes.html 2584 | - https://ebzzry.io/en/zsh-tips-1/ 2585 | - https://github.com/ssledz/bash-fun 2586 | - https://news.ycombinator.com/item?id=24556022 2587 | - https://www.datafix.com.au/BASHing/index.html 2588 | - https://susam.github.io/tucl/the-unix-command-language.html 2589 | - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html 2590 | - https://www.grymoire.com/Unix/Sh.html 2591 | - https://github.com/dylanaraps/pure-sh-bible 2592 | - https://github.com/dylanaraps/pure-bash-bible 2593 | - https://shatterealm.netlify.app/programming/2021_01_02_shiv_lets_build_a_vcs 2594 | - https://news.ycombinator.com/item?id=24401085 2595 | - https://git.sr.ht/~sircmpwn/shit 2596 | - [[https://github.com/p8952/bocker][bocker]]. Docker implemented in around 100 lines of bash. 2597 | - https://github.com/simplenetes-io/simplenetes wow 2598 | - https://bakkenbaeck.github.io/a-random-walk-through-git/ 2599 | - https://github.com/WeilerWebServices/Bash 2600 | - https://www.netmeister.org/blog/consistent-tools.html. 2601 | - https://www.arp242.net/why-zsh.html. bash vs zsh differences. 2602 | - https://dwheeler.com/essays/filenames-in-shell.html 2603 | - https://lacker.io/math/2022/02/24/godels-incompleteness-in-bash.html 2604 | - https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp-part-1-doug-mcilroys-pipeline/ (from [[https://clojurians.slack.com/archives/C03RZGPG3/p1645563940203839][clojurians slack]]) 2605 | - https://github.com/adityaathalye 2606 | - https://www.youtube.com/watch?v=BJ0uHhBkzOQ 2607 | - https://www.gnu.org/software/autoconf/manual/autoconf-2.60/html_node/Portable-Shell.html 2608 | 2609 | * From shell to lisp and everything in between 2610 | - [[https://github.com/oilshell/oil][Oil Shell]]. 2611 | - [[https://rash-lang.org/][Rash]] (Racket shell) 2612 | - [[https://arxiv.org/pdf/2007.09436.pdf][PaSh]]: Light-touch Data-Parallel Shell Processing. 2613 | - [[https://www.eigenbahn.com/2020/07/08/painless-emacs-remote-shells][Painless emacs remote shells]]. Because emacs has you covered 2614 | - https://news.ycombinator.com/item?id=24249646 rust 2615 | - https://github.com/liljencrantz/crush 2616 | - https://github.com/artyom-poptsov/metabash 2617 | - https://www.nushell.sh/ 2618 | - [[https://github.com/borkdude/babashka][Babashka]] 2619 | - Bash to Perl/Python/Ruby using =``= and growing from there. 2620 | * Credits 2621 | - Raimon Grau <[[mailto:raimonster@gmail.com][raimonster@gmail.com]]>. 2622 | - Some examples are result of Raimon's and Lluís Esquerda's 2623 | conversations or real world examples. 2624 | - people in https://news.ycombinator.com/item?id=24402571 which I'll 2625 | be pulling in as time allows. 2626 | -------------------------------------------------------------------------------- /readme.org: -------------------------------------------------------------------------------- 1 | * https://kidd.github.io/scripting-field-guide/index.html 2 | --------------------------------------------------------------------------------