├── .github
    └── workflows
    │   └── deploy.yml
├── book.org
├── readme.org
└── styles.css


/.github/workflows/deploy.yml:
--------------------------------------------------------------------------------
 1 | # This is a basic workflow to help you get started with Actions
 2 | 
 3 | name: Publish
 4 | 
 5 | # Controls when the action will run. Triggers the workflow on push or pull request
 6 | # events but only for the first-skeleton branch
 7 | on:
 8 |   push:
 9 |     branches: [ master ]
10 | 
11 | # A workflow run is made up of one or more jobs that can run sequentially or in parallel
12 | jobs:
13 |   # This workflow contains a single job called "build"
14 |   build:
15 |     # The type of runner that the job will run on
16 |     runs-on: ubuntu-latest
17 | 
18 |     # Steps represent a sequence of tasks that will be executed as part of the job
19 |     steps:
20 |     # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
21 |     - uses: actions/checkout@v2
22 | 
23 |     # Runs a single command using the runners shell
24 |     - name: Run a one-line script
25 |       run: echo Hello, world!
26 | 
27 |     - name: Set up Emacs
28 |       uses: purcell/setup-emacs@master
29 |       with:
30 |         version: 26.1
31 | 
32 |     - name: get htmlize
33 |       run: wget https://raw.githubusercontent.com/hniksic/emacs-htmlize/master/htmlize.el
34 | 
35 |     - name: export
36 |       run: emacs book.org -Q -batch -l htmlize.el -f org-html-export-to-html
37 |     # Runs a set of commands using the runners shell
38 |     - name: rename
39 |       run: mv book.html index.html
40 | 
41 |     - name: Deploy to GitHub Pages
42 |       uses: peaceiris/actions-gh-pages@v3
43 |       with:
44 |         github_token: ${{ secrets.GITHUB_TOKEN }}
45 |         publish_dir: ./
46 |         allow_empty_commit: true
47 |         # enable_jekyll: true
48 |         # cname: github.peaceiris.com
49 | 
50 |     # - name: Deploy to GitHub Pages
51 |     #   if: success()
52 |     #   uses: crazy-max/ghaction-github-pages@v2
53 |     #   with:
54 |     #     target_branch: gh-pages
55 |     #     build_dir: ./
56 |     #   env:
57 |     #     GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
58 |     # - name: Bye
59 |     #   run: |
60 |     #     echo DONE!
61 | 


--------------------------------------------------------------------------------
/book.org:
--------------------------------------------------------------------------------
   1 | #+TITLE: Shell Field Guide
   2 | #+AUTHOR: Raimon Grau
   3 | #+OPTIONS: ':nil *:t -:t ::t <:t H:3 \n:nil ^:nil arch:headline
   4 | #+OPTIONS: author:t c:nil creator:comment d:(not "LOGBOOK") date:nil
   5 | #+OPTIONS: e:t email:nil f:t inline:t p:nil pri:nil stat:t
   6 | #+OPTIONS: tags:t tasks:t tex:t timestamp:t todo:t |:t
   7 | #+EXCLUDE_TAGS: noexport
   8 | #+KEYWORDS: bash zsh shell
   9 | #+LANGUAGE: en
  10 | #+SELECT_TAGS: export
  11 | 
  12 | #+OPTIONS: html-style:nil
  13 | #+SETUPFILE: https://fniessen.github.io/org-html-themes/org/theme-readtheorg.setup
  14 | #+HTML_HEAD_EXTRA: <link rel="stylesheet" type="text/css" href="styles.css"/>
  15 | 
  16 | #+OPTIONS: reveal_center:nil timestamp:nil
  17 | #+REVEAL_THEME: black
  18 | 
  19 | # toc:nil num:nil
  20 | 
  21 | * Introduction
  22 |   This booklet is intended to be a catalog of tricks and techniques
  23 |   you may want to use if you're doing some sort of complex
  24 |   scripting. Some are useful in "The Real World (TM)", some are more
  25 |   playful, and might not have such direct impact in your day-to-day
  26 |   life. Some are pure entertainment. You'll have to judge by yourself
  27 |   which belong to which. I'll try to keep the rhetoric to the minimum
  28 |   to maximize signal/noise.
  29 | 
  30 |   The git repo is at
  31 |   https://github.com/kidd/scripting-field-guide/. Any feedback is
  32 |   greatly appreciated. Keep in mind this is not any kind of official
  33 |   doc. I just write MY current "state of the art" and I'll be updating
  34 |   the contents with useful stuff I find or discover, that are not
  35 |   widely explained in usual manuals/wikis.
  36 | 
  37 |   You surely have some amount of sh/bash/zsh in your stack that
  38 |   probably started as one-off scripts, and later grew and were
  39 |   copypasted everywhere in your pipelines, repos, coworkers' ~/bin, or
  40 |   your coworkers use for their own things (with some variations),
  41 |   etc. Those scripts are very difficult to kill and they have a very
  42 |   high mutation rate.
  43 | 
  44 | * Which Shell?
  45 |   No matter if you use Linux, Mac, or Windows, you should be living
  46 |   most of the time in a shell to enjoy the content shown here. Some
  47 |   value comes from running scripts, and some comes from the daily
  48 |   usage and refinement of your helper functions, aliases, etc. in
  49 |   interactive mode.
  50 | 
  51 |   In general the examples here are meant to run in Bash or Zsh, which
  52 |   are compatible for the most part.
  53 | * Level
  54 |   These examples are based on non-trivial real world code I've written
  55 |   using patterns I haven't seen applied in many places over the net. A
  56 |   few of the snippets are stolen from public repos I find interesting.
  57 |   Also, important scripting stuff might be missing if I don't feel
  58 |   I have anything to add to the generally available info around.
  59 | * Patterns
  60 | ** Use Shellcheck
  61 |    First, let's get that out of the way. This is low-hanging
  62 |    fruit. And you will get the most of this booklet by following it.
  63 | 
  64 |    A lot of the most common errors we usually make are well known
  65 |    ones. And in fact, we all usually fail in similar ways. Bash is
  66 |    known for being error prone when dealing with testing variable
  67 |    values, string operations, or flaky subshells and pipes.
  68 | 
  69 |    Installing [[https://www.shellcheck.net/][shellcheck]] will flag many of those ticking bombs for
  70 |    you.
  71 | 
  72 |    No matter which editor you are using, you should be able to
  73 |    install a plugin to do automatic checks while you're editing.
  74 | 
  75 |    In emacs' case, the package is called [[https://github.com/federicotdn/flymake-shellcheck][flymake-shellcheck]], and a
  76 |    quick configuration for it is:
  77 | 
  78 |    #+begin_src elisp
  79 |    (use-package flymake-shellcheck
  80 |      :ensure t
  81 |      :commands flymake-shellcheck-load
  82 |      :init
  83 |      (add-hook 'sh-mode-hook 'flymake-shellcheck-load))
  84 |    #+end_src
  85 | 
  86 |    Shellcheck is available on most distros, so it's just an =apt=,
  87 |    =brew=, or =nix-env= away.
  88 | 
  89 | ** Booleans and Conditionals
  90 |    In any shell, =foo && bar= will execute =bar= only if =foo=
  91 |    succeeded. That means that =foo= returned 0. That means that to &&
  92 |    (which you read like "and"), 0 is true. so yes. 0 is true, and
  93 |    other values are false.
  94 | 
  95 | ** Arrays
  96 |    Ordered list of things.
  97 |    #+begin_src bash
  98 |      foo=("ls" "/tmp/")
  99 | 
 100 |      echo ${foo[-2]}
 101 |      echo ${foo[-1]}
 102 |      echo ${foo[0]}
 103 |      echo ${foo[1]}
 104 |      echo ${foo[2]}
 105 | 
 106 |      for i in "${foo[@]}"; do
 107 |        echo $i
 108 |      done
 109 | 
 110 |      $foo
 111 |      ${foo[*]}
 112 |      ${foo[@]}
 113 |      echo ${#foo[*]}
 114 |      echo ${#foo[@]}
 115 |      #+end_src
 116 | 
 117 |    Are =*= and =@= equal? [[https://stackoverflow.com/questions/2761723/what-is-the-difference-between-and-in-shell-scripts][nope]].
 118 | 
 119 |    #+begin_src bash
 120 |    "${foo[@]}"
 121 |    "${foo[*]}"
 122 |    #+end_src
 123 | 
 124 |    #+begin_src bash
 125 |      #!/bin/bash
 126 | 
 127 |      main()
 128 |      {
 129 |        echo 'MAIN sees ' $# ' args'
 130 |      }
 131 | 
 132 |      main $*
 133 |      main $@
 134 | 
 135 |      main "$*"
 136 |      main "$@"
 137 | 
 138 |      ### end ###
 139 |    #+end_src
 140 | 
 141 |    and I run it like this:
 142 | 
 143 |    =my_script 'a b c' d e=
 144 | 
 145 | ** Pass Arrays around
 146 |    #+begin_src bash
 147 |      a=('Track 01.mp3' 'Track 02.mp3')
 148 |      myfun "${a[@]}" # pass array to a function
 149 |      b=( "${a[@]}" ) # copy array
 150 |    #+end_src
 151 | 
 152 |    Read the great [[http://www.oilshell.org/blog/2016/11/06.html][Oil Shell blogpost]].
 153 | ** Slurping arrays
 154 |    A nice way to read a bunch of elements in one go is to use
 155 |    =readarray=.
 156 |    #+begin_src bash
 157 |    parse_args() {
 158 |      [[ $# -eq 0 ]] && die "Usage: $0 <version>"
 159 |      version="$1"
 160 |      local version_split=$(echo $version | tr '.' '\n')
 161 |      readarray -t version_array <<< "$version_split"
 162 | 
 163 |      if [[ -z ${version_array[3]} ]]; then
 164 |        die "not enough version numbers"
 165 |      fi
 166 |    }
 167 |    #+end_src
 168 | 
 169 |    Even nicer would be to use IFS so we'd be able to split in one go.
 170 | 
 171 |    #+begin_src bash
 172 |      IFS=. read -a ver <<<"1.23.1.0"
 173 |      echo ${ver[0]}
 174 |      echo "next/${ver[0]}.${ver[1]}.x.x"
 175 |    #+end_src
 176 | 
 177 |    Or, use it in a destructuring fashion:
 178 |    #+begin_src bash
 179 |     get_nix_version_parts(){
 180 |       local major minor patch
 181 |       # shellcheck disable=SC2034,SC2162
 182 |       IFS="." read major minor patch < <(get_nix_version)
 183 |       local -p
 184 |     }
 185 | 
 186 |     $ get_nix_version_parts
 187 |     major=2
 188 |     minor=3
 189 |     patch=4
 190 |    #+end_src
 191 |    https://news.ycombinator.com/item?id=24408318
 192 | ** Array slices
 193 | 
 194 |    you can slice an array like this:
 195 | 
 196 |    #+begin_src bash
 197 |      a=(a b c d e f g)
 198 |      echo "${a[@]:1}"  # tail of array
 199 |      echo "${a[@]:1:2}"  # from that, take 2 elements
 200 |    #+end_src
 201 | 
 202 |    This is a [[https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html][parameter expansion]], same as =${a/foo/bar}= and similar
 203 |    replacements. As such, The =$@= and =$*= are treated a bit
 204 |    differently.
 205 | 
 206 |    Look at =${parameter:offset:length}= in the parameter expansion
 207 |    bash manual and be amazed by the specialcasing :).
 208 | 
 209 |    This concrete case is when the parameter is an indexed array name
 210 |    subscripted by '@' or '*'.
 211 | 
 212 |    Don't ask me why, but when the array is =$@=, it seems that it
 213 |    behaves a bit differently. It's 'explained' in that same part of
 214 |    the docs.
 215 | 
 216 |    #+begin_src bash
 217 |      a=(a b c d e f g)
 218 |     foo() {
 219 |      echo "${@:2}"  # tail of array
 220 |      echo "${@:2:2}"  # from that, take 2 elements
 221 |     }
 222 |     foo a b c d
 223 |     foo "${a[@]}"
 224 |    #+end_src
 225 | 
 226 | ** Read lines from file
 227 |    The =read= command we used just above is part of the usual idiom to
 228 |    read a file line by line.
 229 | 
 230 |    #+begin_src bash
 231 |      while read -r line; do
 232 |        echo $line
 233 |      done < /tmp/file.txt
 234 |    #+end_src
 235 | 
 236 |    More related info in the [[https://mywiki.wooledge.org/BashFAQ/001][BashFAQ001]]. But it's very rare the case
 237 |    where I need to iterate a file line by line.
 238 | 
 239 | ** Assign to $1,$2,$3...
 240 |    =set --= can be used as an incantation to assign to the positional
 241 |    parameters. Let me show you.
 242 | 
 243 |    #+begin_src bash
 244 |      set -- a b c
 245 |      echo $1 $2 $3
 246 |      echo $@
 247 |    #+end_src
 248 | 
 249 |    See? here's how to "unshift" a parameter to the current arg list:
 250 | 
 251 |    #+begin_src bash
 252 |      set -- "injected" "$@"
 253 |    #+end_src
 254 | 
 255 | ** Functions
 256 |    Functions are functions. They receive arguments, and they return a
 257 |    value.
 258 | 
 259 |    The special thing about shell functions is that they also can use
 260 |    the file descriptors of the process. That means that they "inherit"
 261 |    STDIN, STDOUT, STDERR (and maybe more).
 262 | 
 263 |    Use them.
 264 | 
 265 |    Another point is that function names can be passed as parameters,
 266 |    because they are passed as strings, but you can call them inside as
 267 |    functions again. Maybe they are better than you [[https://news.ycombinator.com/item?id=29058140][think.]]
 268 | 
 269 |    #+begin_src bash
 270 |    f() {
 271 |      $1 hi
 272 |    }
 273 | 
 274 |    f echo
 275 |    f touch # will create a file 'hi'
 276 |    #+end_src
 277 | 
 278 |    #+RESULTS:
 279 |    : hi
 280 | 
 281 | ** Variables
 282 |    By default variables are global, to a file. No matter if you assign
 283 |    them for the first time inside a function, or at the top level.
 284 |    #+begin_src bash
 285 |    foo=3
 286 |    bar=$foo
 287 |    f() {
 288 |      echo $bar
 289 |    }
 290 |    f
 291 |    #+end_src
 292 | 
 293 |    #+RESULTS:
 294 |    : 3
 295 | 
 296 |    #+begin_src bash
 297 |    f() {
 298 |      bar=1
 299 |    }
 300 |    f
 301 |    echo $bar
 302 |    #+end_src
 303 | 
 304 |    #+RESULTS:
 305 |    : 1
 306 | 
 307 |    You make a variable local to a function with =local=. Use it as
 308 |    much as you can (kinda obvious).
 309 |    #+begin_src bash
 310 |    myfun() {
 311 |      local bar
 312 |      bar=3
 313 |      echo $bar
 314 |    }
 315 | 
 316 |    bar=4
 317 |    echo $bar
 318 |    myfun
 319 |    echo $bar
 320 |    #+end_src
 321 | 
 322 |    #+RESULTS:
 323 |    | 4 |
 324 |    | 3 |
 325 |    | 4 |
 326 | ** Variable/Parameter Expansions
 327 |    They offer some variable manipulations using shell only, not having
 328 |    to create another process =sed,awk,perl=.
 329 | 
 330 |    #+begin_src bash
 331 |      v=banana
 332 |      # substitute one
 333 |      echo ${v/na/NA}   # baNAna
 334 | 
 335 |      # substitute many
 336 |      echo ${v//na/NA}  # baNANA
 337 | 
 338 |      # substitute from the start (think ^ in PCRE)
 339 |      echo ${v/#ba/NA}  # NAnana
 340 | 
 341 |      # substitute from the end
 342 |      echo ${v/%na/NA}  # banaNA
 343 | 
 344 |      # Capitalize
 345 |      echo ${v^}  # Banana
 346 | 
 347 |      # Uppercase
 348 |      echo ${v^^}  # BANANA
 349 |    #+end_src
 350 | 
 351 |    Take a read on https://tldp.org/LDP/abs/html/manipulatingvars.html
 352 |    and
 353 |    https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
 354 |    for more details.
 355 | 
 356 |    And a nice non-obvious trick from here is to prefix or suffix a
 357 |    variable string:
 358 | 
 359 |    #+begin_src bash
 360 |      v=banana
 361 |      echo ${v/%/na}   # bananana
 362 |      echo ${v/#/na}   # nabanana
 363 |    #+end_src
 364 | 
 365 |    And a less obvious trick is to prefix every element of an array
 366 |    with a fixed string:
 367 |    #+begin_src bash
 368 |      local arr=(var1=1 var2=2)
 369 |      echo ${arr[*]/#/"--env "}
 370 |    #+end_src
 371 | 
 372 |    This will produce =--env var1=1 --env var2=2=.  Super useful to be
 373 |    combined when building flags for docker.
 374 | 
 375 |    Another nice trick is a portable =realpath $PWD= with =${0%/*}=,
 376 |    which I found in [[https://github.com/picolisp/pil21/blob/master/vip][picolisp's vip]]:
 377 | 
 378 |    #+begin_src bash
 379 |      #!/bin/sh
 380 |      exec ${0%/*}/bin/picolisp ${0%/*}/lib.l @bin/vip "$@"
 381 |    #+end_src
 382 | 
 383 |    =${0%/*}/bin/picolisp= will replace keep the path of the current
 384 |    script and append =/bin/picolisp=. A cool way to get a realpath.
 385 | 
 386 | ** Conditional Expansion
 387 | 
 388 |    #+begin_src bash
 389 |    local  foo=${var:-${default:-}}
 390 |    local  foo=${var:+&key=var}
 391 |    #+end_src
 392 | 
 393 | ** Interpolation
 394 |    We previously saw that functions can be passed around as strings,
 395 |    and be called later on.
 396 | 
 397 |    Something that might not be obvious is that the string can be
 398 |    created from shorter strings, and that allows for an extra
 399 |    flexibility, that comes with its own dangers, but it's a very
 400 |    useful pattern to dispatch functions based on user input or
 401 |    function outputs.
 402 |    #+begin_src bash
 403 |    l=l
 404 |    s=s
 405 |    $l$s .
 406 |    #+end_src
 407 | 
 408 |    #+RESULTS:
 409 |    | book.html  |
 410 |    | book.html~ |
 411 |    | book.org   |
 412 |    | readme.org |
 413 | 
 414 | ** dispatch functions using args
 415 |    A nice usage of the previous technique is using user input as a
 416 |    dispatching method.
 417 | 
 418 |    You've probably seen this pattern already:
 419 | 
 420 |    #+begin_src bash
 421 |    while [[ $# -gt 0 ]]; do
 422 | 
 423 |    case $1 in
 424 |      foo)
 425 |        foo
 426 |        ;;
 427 |      *)
 428 |        exit 1
 429 |        ;;
 430 |    esac
 431 |    shift
 432 |    done
 433 |    #+end_src
 434 | 
 435 |    And it is useful for its own good, and flexible.
 436 | 
 437 |    But for some simpler cases, we can dispatch based on the variable
 438 |    itself:
 439 | 
 440 |    #+begin_src bash
 441 |    cmd_foo() {
 442 |     do-something
 443 |    }
 444 | 
 445 |    cmd_$1
 446 |    #+end_src
 447 | 
 448 |    The problem with this is that in case we supply a =$1= that doesn't
 449 |    map to any =cmd_$1= we'll get something like
 450 | 
 451 |    #+begin_src bash
 452 |    bash: cmd_notexisting: command not found
 453 |    #+end_src
 454 | 
 455 | ** command_not_found_handle
 456 |    Here's a detail on a kinda obscure bash (only bash) feature.
 457 | 
 458 |    You can set a hook that will be called when bash tries to run a
 459 |    command and it doesn't find it.
 460 | 
 461 |    #+begin_src bash
 462 |    command_not_found_handle() {
 463 |      echo "$1 is not a correct command. Cmds allowed:"
 464 |      echo "$(typeset -F | grep cmd_ | sed -e 's/.*cmd_/cmd_/')"
 465 |    }
 466 | 
 467 |    cmd_foo() {
 468 |      echo "foo"
 469 |    }
 470 | 
 471 |    cmd_baz() {
 472 |      echo "baz"
 473 |    }
 474 |    cmd_bar
 475 |    #+end_src
 476 | 
 477 |    You can unset the function =command_not_found_handle= to go back to
 478 |    the normal behavior.
 479 | 
 480 | ** Return Values for Conditionals
 481 |    =if= 's test condition can use the return values of
 482 |    commands. That's a known thing, but a lot of code you see around
 483 |    relies on =[[]]= to test the return values of commands/functions
 484 |    anyway.
 485 | 
 486 |    #+begin_src bash
 487 |    if echo "foo" | grep "bar" ; then
 488 |      echo "found!"
 489 |    fi
 490 |    #+end_src
 491 | 
 492 |    This is much clearer than
 493 |    #+begin_src bash
 494 |    if [[ ! -z $( echo "foo" | grep "bar") ]]; then
 495 |      echo "found!"
 496 |    fi
 497 |    #+end_src
 498 | 
 499 |    As easy and trivial as it seems, this way of thinking pushes you
 500 |    forward to thinking about creating smaller functions that check the
 501 |    conditions and =return= 0 or non 0. It's syntactically smaller, and
 502 |    usually makes you play by the rules of the commands, more than just
 503 |    finding your way around the output strings.
 504 | 
 505 |    #+begin_src bash
 506 |    if less_than $package "1.3.2"; then
 507 |      die "can't proceed"
 508 |    fi
 509 |    #+end_src
 510 | 
 511 | ** set variable in an "if" test
 512 |    Usual pattern to capture the output of a command and branch
 513 |    depending on its return value is:
 514 |    #+begin_src bash
 515 |      res="$(... whatever ...)"
 516 |      if [ "$?" -eq 0 ]; then ...
 517 |                              ...
 518 |      fi
 519 |    #+end_src
 520 | 
 521 |    Well, you can test the return value AND capture the output at the
 522 |    same time!
 523 | 
 524 |    #+begin_src bash
 525 |      if res="$(...)"; then
 526 |        ...
 527 |      fi
 528 |    #+end_src
 529 | 
 530 |    Unfortunately, it doesnt' work with =local=, so you can't be
 531 |    defining a local var in the same line. So, the variable is either
 532 |    global, or you spent a line to declare it local before. Still, I
 533 |    think I prefer to have a line to declare the variable as local
 534 |    rather than having explicit =$?='s around.
 535 |    #+begin_src bash
 536 |      local var1
 537 |      if var1=$(f); then
 538 |        echo "$var1"
 539 |      fi
 540 |    #+end_src
 541 |    Ref: https://news.ycombinator.com/item?id=27163494
 542 | 
 543 | ** Do work on loop conditions
 544 |    I've not seen it used a lot (and there might be a reason for it,
 545 |    who knows), =while= conditions are just plain commands, so you can
 546 |    put other stuff than =[]/[[]]/test= there.
 547 | 
 548 |    Heres's an idiomatic way to iterate through all the arguments of a
 549 |    function while consuming the =$*= array.
 550 | 
 551 |    #+begin_src bash
 552 |    while(($#)) ; do
 553 |      #...
 554 |      shift
 555 |    done
 556 |    #+end_src
 557 | 
 558 |    And here's a pseudo-repl that keeps shooting one-off commands. This
 559 |    will keep shooting =tr= commands to whatever strings you give it,
 560 |    with the usual rlwrap goodies.
 561 | 
 562 |    #+begin_src bash
 563 |    while rlwrap -o -S'>> ' tr a-z A-Z ; do :; done
 564 |    #+end_src
 565 | 
 566 |    Note: =:= is a nop builtin in bash.
 567 | ** One Branch Conditionals
 568 |    The usual conditionals one sees everywhere look like =if=.
 569 |    #+begin_src bash
 570 |    if [[ some-condition ]]; then
 571 |      echo "yes"
 572 |    fi
 573 |    #+end_src
 574 | 
 575 |    This is all good and fine, but in the same vein of using the least
 576 |    powerful construct for each task, it's nice to think of the one way
 577 |    conditionals in the form of =&&= and =||= as a way to explicitly
 578 |    say that we don't want to do anything else when the condition is
 579 |    not met. It's a hint to the reader.
 580 | 
 581 |    #+begin_src bash
 582 |    some-condition || {
 583 |       echo "log: warning!"
 584 |    }
 585 | 
 586 |    other-condition && {
 587 |       echo "log: all cool"
 588 |    }
 589 |    #+end_src
 590 | 
 591 |    This conveys the intention of doing something *just* in one case,
 592 |    and that the negation of this is not interesting at all. There's a
 593 |    big warning you have to be aware of. The same as with lua's
 594 |    =... and .. or ..=, bash =||= and =&&= are not interchangeable for
 595 |    =if...else...end=. [[https://mywiki.wooledge.org/BashPitfalls#cmd1_.26.26_cmd2_.7C.7C_cmd3][BashWiki]] has an explanation why, but, the same
 596 |    as in Lua's case, if the "then" part returns false, the else will
 597 |    run.
 598 | 
 599 |    There are lots of references to this, but I like this recent post
 600 |    where it explains it for arrays in higher level languages like ruby:
 601 |    https://jesseduffield.com/array-functions-and-the-rule-of-least-power/
 602 | 
 603 |    An extended article of this kind of conditionals can be found [[https://timvisee.com/blog/elegant-bash-conditionals/][here]].
 604 | ** pushd/popd
 605 |    pushd and popd are used to move to some directory and go back to it
 606 |    in a stack fashion, so nesting can happen and you never lose
 607 |    track. The problem is that it still is on you to have a =popd= per
 608 |    =pushd=.
 609 |    #+begin_src bash
 610 |    pushd /tmp/my-dir
 611 |      echo $PWD
 612 |    popd
 613 |    #+end_src
 614 | 
 615 |    Here's an alternative way, that at least makes sure that you close
 616 |    all pushd with a popd.
 617 | 
 618 |    Starting a new shell and cd-ing , will make all commands in that
 619 |    subshell be in that directory, and will come back to the old
 620 |    directory after closing the new spawned shell.
 621 | 
 622 |    #+begin_src bash
 623 |      (cd /tmp/my-dir
 624 |        ls
 625 |      )
 626 |    #+end_src
 627 | 
 628 |    Remember to =inherit_errexit= or =set -e= inside the subshell if
 629 |    you need. That's a very easy trap to fall into.
 630 | 
 631 | ** wrap functions
 632 |    Bash can't pass blocks of code around, but the alternative is to
 633 |    pass functions.  More on that later.
 634 | 
 635 |    #+begin_src bash
 636 |      mute() {
 637 |        "$@" >/dev/null
 638 |      }
 639 | 
 640 |      mute pushd /tmp/foobar
 641 |    #+end_src
 642 | ** use [[
 643 |    Unless you want your script to be POSIX compliant, use =[[= instead
 644 |    of =[=.  =[= is a regular command. It's like =ls=, or =true=. You can
 645 |    check it by searching for a file named =[= in your path.
 646 | 
 647 |    Being a normal command it always evaluates its params, like a
 648 |    regular function. On the other hand though, =[[= is a special bash
 649 |    operator, and it evaluates the parameters lazily.
 650 | 
 651 |    #+begin_src bash
 652 | 
 653 |      # [[ does lazy evaluation:
 654 |      [[ a = b && $(echo foo >&2) ]]
 655 | 
 656 |      # [ does not:
 657 |      [ a = b -a "$(echo foo >&2)” ]
 658 |      #+end_src
 659 |    Ref: https://lists.gnu.org/archive/html/help-bash/2014-06/msg00013.html
 660 | 
 661 | ** eval?
 662 |    When you have mostly small functions that are mostly pure, you
 663 |    compose them like you'd do in any other language.
 664 | 
 665 |    In the following snippet, we are in a release script. Some step
 666 |    builds a package inside a docker container, another step tests a
 667 |    package already built.
 668 | 
 669 |    A nice way to build ubuntus, for example, is to add an ARG to the
 670 |    Dockerfile so we can build several ubuntu versions using the same
 671 |    file.
 672 | 
 673 |    It'd look like this:
 674 |    #+begin_src Dockerfile
 675 |      ARG VERSION
 676 |      FROM ubuntu:$VERSION
 677 | 
 678 |      RUN apt-get ...
 679 |      ...
 680 |    #+end_src
 681 | 
 682 |    We build that image and do all the building inside it, mounting a
 683 |    volume shared with our host, so we can extract our =.deb= file
 684 |    easily.
 685 | 
 686 |    After that, to do some smoke tests on the package, the idea is to
 687 |    install the =.deb= file in a fresh ubuntu image.
 688 | 
 689 |    Let's pick the same base image we picked to build the package.
 690 |    #+begin_src bash
 691 |      # evaluate the string "centos:$VERSION" (that comes from
 692 |      # centos/Dockerfile) in the current scope
 693 |      # DISTRO is ubuntu:18.04
 694 |      local VERSION=$(get_version $DISTRO) # VERSION==18.04
 695 |      run_test "file.deb" "$(eval echo $(awk '/^FROM /{print $2; exit}' $LOCAL_PATH/$(get_dockerfile_for $DISTRO)))" # ubuntu:18.04
 696 |    #+end_src
 697 | 
 698 |    The usage of eval is there to interpolate the string that we get
 699 |    from the =FROM= in the current environment.
 700 | 
 701 |    WARNING: You know, anything that uses =eval= is dangerous per
 702 |    se. Do not use it unless you know very well what you're doing AND
 703 |    the input is 100% under your control. Usually, more restricted
 704 |    commands can achieve what you want to do. In this particular case,
 705 |    you could use =envsubst=, or just manually replace =$\{?VERSION\}?=
 706 |    in a sed.
 707 | 
 708 |    #+begin_src bash
 709 |      test_release "$PACKAGE_PATH" $(awk '/^FROM /{print $2; exit}' $LOCAL_PATH/$(get_dockerfile_for $DISTRO) | sed -e "s/\$VERSION/$VERSION/")
 710 |    #+end_src
 711 | 
 712 |    Yet another way is using [[https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html][shell parameter expansions.]]
 713 |    #+begin_src bash
 714 | 
 715 |      var1=value
 716 |      echo 'this is $var1' >/tmp/f.txt
 717 |      f=$(cat /tmp/f.txt)
 718 |      echo "${f}"  # this is $var1
 719 |      echo "${f@P}"  # this is value
 720 |    #+end_src
 721 | 
 722 | ** pass commands around
 723 |    This one uses [[*DRY_RUN][DRY_RUN]]. While refactoring a script that does some
 724 |    curls, we want to make sure that our refactored version does the
 725 |    exact same calls in the same order.
 726 | 
 727 |    #+begin_src bash
 728 |      compare_outputs() {
 729 |        export DRY_RUN=1
 730 |        git checkout b1
 731 |        $@ 2>/tmp/1.out
 732 |        git checkout b2
 733 |        $@ 2>/tmp/2.out
 734 |        echo "diffing"
 735 |        diff /tmp/1.out /tmp/2.out
 736 |      }
 737 |      compare_outputs ./release.sh -p rhel:6 -R 'internal-preview'
 738 |    #+end_src
 739 | 
 740 |    First we create a function =compare_outputs=, that gets a command
 741 |    to run as parameters. The function will run it once, redirecting
 742 |    the standard error to a file =/tmp/1.out=.
 743 | 
 744 |    Then, it checks out the branch that contains our refactored
 745 |    version, and will run the command again, redirecting standard error
 746 |    to =/tmp/2.out=, and will diff the two outputs.
 747 | 
 748 |    In case there's a difference between the two, =diff= will output
 749 |    them, and the function will return the non-zero exit value of
 750 |    diff. If everything went fine, =compare_outputs= will succeed.
 751 | 
 752 |    Now that we know that for these inputs the command runs fine, we
 753 |    want to find out if it works for other types of releases, not only
 754 |    internal-preview.
 755 | 
 756 |    Here I'm using zsh's global aliases to give a much more fluid
 757 |    interface to the commands, but you can use the regular while/for
 758 |    loops:
 759 | 
 760 |    #+begin_src shell
 761 |      alias -g SPLIT='| tr " " "\n" '
 762 |      alias -g FORI='| while read i ; do '
 763 |      alias -g IROF='; done '
 764 | 
 765 |      set -e
 766 |      echo "ga internal-preview rc1 rc2" SPLIT FORI
 767 |         noglob compare_outputs ./release.sh -p rhel:8 -R "$i"
 768 |      IROF
 769 |    #+end_src
 770 | 
 771 |    So, combining the two, we can have a really smooth way of iterating
 772 |    over the possibilities, without really messing into the details of
 773 |    loops.
 774 | 
 775 |    WARNING: This approach is not robust enough to put it anywhere in
 776 |    production, but to write quick one off scripts is a
 777 |    killer. Experimenting in a shell and creating tools and 2nd order
 778 |    tools to make interaction faster builds a language that grows on
 779 |    you, and keeps improving your toolbelt.
 780 | 
 781 | ** The Toplevel Is Hopeless
 782 |    Shellscripts are thought as quick one-off programs, but when they
 783 |    are useful, they are sticky, so you better write them from the
 784 |    start as if it would be permanent. The upfront cost is very low
 785 |    anyway. Structure the script like a regular app.
 786 | 
 787 |    Bash is extremely permissive in what it allows to be coded and
 788 |    ran. By default, failures do not make the program exit or throw an
 789 |    exception (no exceptions). And for some reason, the common
 790 |    usage of shellscripts is to put everything in the top level. Don't
 791 |    do that. Do the least possible things in the toplevel.
 792 | 
 793 |    A way to improve the defaults, is setting a bunch of flags that
 794 |    make the script stricter, so it fails on many situations you'd
 795 |    want to stop anyway because something went wrong.
 796 |    #+begin_src bash
 797 |      #!/usr/bin/env bash
 798 |      set -eEuo pipefail
 799 |      shopt -s inherit_errexit
 800 | 
 801 |      main() {
 802 |        parse_args
 803 |        validate_args
 804 |        do_things
 805 |        cleanup
 806 |      }
 807 | 
 808 |      main "$@"
 809 |    #+end_src
 810 | 
 811 |    Ref: https://dougrichardson.us/2018/08/03/fail-fast-bash-scripting.html
 812 | 
 813 |    Although... [[http://mywiki.wooledge.org/BashFAQ/105][Why doesn't set -e do what I expected?]]
 814 | 
 815 | ** Check your deps
 816 |    Giving useful information to the users will help them using the
 817 |    script, and you debugging it. Script dependencies is a common use
 818 |    case that we'll do it in a nice way.
 819 | 
 820 |    #+begin_src bash
 821 |      deps() {
 822 |        for dep in "$@"; do
 823 |          mute command -v "$dep" || die "$dep dependency missing"
 824 |        done
 825 |      }
 826 | 
 827 |      main() {
 828 |        deps jq curl
 829 |        # ...
 830 |      }
 831 |    #+end_src
 832 | 
 833 | ** source files
 834 |    =source= is like =require= or =import= in some programming
 835 |    languages. It evaluates the sourced file in the context of the
 836 |    current script, so you get all definitions in your environment.
 837 | 
 838 |    It's simple, but it helps you get used to modularize your code into
 839 |    libraries.
 840 | 
 841 |    Be careful, it's very rudimentary, and it will be overwriting old
 842 |    vars or functions if names clash. There's no namespacing happening
 843 |    there.
 844 | 
 845 |    #+begin_src bash
 846 |      source file.sh
 847 | 
 848 |      # the same
 849 |      . file.sh
 850 |    #+end_src
 851 | 
 852 | ** Use Scripts as a Libs
 853 |    A python-inspired way of using scripts as loadable libraries is to
 854 |    check whether the current file was the one that was called
 855 |    originally or it's being just sourced.
 856 | 
 857 |    Again, no side effects in load time makes this functionality
 858 |    possible. otherwise, you're on your own.
 859 |    #+begin_src bash
 860 |      # Allow sourcing of this script
 861 |      if [[ $(basename "$(realpath "$0")") == "$(basename "${BASH_SOURCE}")" ]]; then
 862 |        setup
 863 |        parse_args "$@"
 864 |        main
 865 |      fi
 866 |    #+end_src
 867 | ** Tmpfiles Everywhere
 868 |    Your script is not going to run alone. Don't assume paths are
 869 |    fixed or known.
 870 | 
 871 |    CI/CD Pipelines run many jobs in the same node and files can start
 872 |    clashing.
 873 | 
 874 |    Make use of =$(mktemp -d /tmp/foo-bar.XXXXX)=.  If you have to patch a
 875 |    file, do it in a clean fresh copy. Don't modify files in old paths
 876 | 
 877 |    If you HAVE TO modify paths, do it idempotently.  But really, don't do it.
 878 | 
 879 |    #+begin_src bash :eval no
 880 |      git_clone_tmp() {
 881 |        local repo=${1:?repo is required}
 882 |        local ref=${2:?ref is required}
 883 |        tmpath=$(mktemp -d "/tmp/cloned-$repo-XXXXX")
 884 |        on_exit "rm -rf $tmpath"
 885 |        git clone -b ${ref} $repo $tmpath
 886 |      }
 887 |    #+end_src
 888 | 
 889 |    CAVEAT: You have to manually delete the directory if you want it cleaned.
 890 | 
 891 |    Here's an article with very good advice on [[https://www.netmeister.org/blog/mktemp.html][tempfiles]].
 892 | ** Cleanup tasks with trap
 893 |    =trap= is used to 'subscribe' a callback when something happens.
 894 |    Many times it's used on exit. It's a good thing to cleanup tmpdirs after your script
 895 |    exits, so you can use the output of =mktemp -d= and subscribe a cleanup
 896 |    function for it.
 897 | 
 898 |    #+begin_src bash
 899 |      on_exit() {
 900 |        rm -rf $1
 901 |      }
 902 |      local tmpath=$(mktemp -d /tmp/foo-bar.XXXXX)
 903 |      trap "on_exit $tmpath" EXIT SIGINT
 904 |    #+end_src
 905 | 
 906 | ** array of callbacks on_exit
 907 |    Level up that pattern, we can have a helper to add callbacks to run
 908 |    on exit. Get used to these kind of patterns, they are super
 909 |    powerful and save you lots of manual bookkeeping.
 910 | 
 911 |    #+begin_src bash
 912 | 
 913 |      ON_EXIT=()
 914 |      EXIT_RES=
 915 | 
 916 |      function on_exit_fn {
 917 |        EXIT_RES=$?
 918 |        for cb in "${ON_EXIT[@]}"; do $cb || true; done
 919 |        return $EXIT_RES
 920 |      }
 921 | 
 922 |      trap on_exit_fn EXIT SIGINT
 923 | 
 924 |      function on_exit {
 925 |        ON_EXIT+=("$@")
 926 |      }
 927 | 
 928 |      local v_id=$(docker volume create)
 929 |      on_exit "docker volume rm $v_id"
 930 |      # Use your v_id knowing that it'll be available during your script but
 931 |      # will be cleaned up before exiting.
 932 |    #+end_src
 933 | 
 934 | ** stacktrace on error
 935 |    Here's a nice helper for debugging errors in bash. In case of non-0
 936 |    exit, it prints a stacktrace.
 937 |    #+begin_src bash
 938 |       set -Eeuo pipefail
 939 |       trap stacktrace EXIT
 940 |       stacktrace() {
 941 |           rc="$?"
 942 |           if [ $rc != 0 ]; then
 943 |               printf '\nThe command "%s" triggerd a stacktrace:\n' "$BASH_COMMAND"
 944 |               for i in $(seq 1 $((${#FUNCNAME[@]} - 2))); do
 945 |                 j=$((i+1));
 946 |                 printf '\t%s: %s() called in %s:%s\n' "${BASH_SOURCE[$i]}" "${FUNCNAME[$i]}" "${BASH_SOURCE[$j]}" "${BASH_LINENO[$i]}";
 947 |               done
 948 |           fi
 949 |       }
 950 | 
 951 |    #+end_src
 952 |    ref: https://news.ycombinator.com/item?id=26644110
 953 | ** Dots and colons allowed in function names!
 954 |    A way to split the namespace is to have libs define functions with
 955 |    their own namespace.
 956 | 
 957 |    I've gotten used to use dots or colons as namespace separator.
 958 |    #+begin_src bash
 959 |      semver.greater() {
 960 |       # ...
 961 |      }
 962 |    #+end_src
 963 |    or
 964 |    #+begin_src bash
 965 |      semver:greater() {
 966 |       # ...
 967 |      }
 968 |    #+end_src
 969 | ** make steps of the process as composable as possible by using "$@"
 970 |    By using =$@= to pass commands as parameters around you can get to
 971 |    a degree of composability that allows for a nice chaining of
 972 |    commands.
 973 | 
 974 |    Here's a very simple version of =watch=. See how you can =every 2
 975 |    ls -la=. I think that style is called Bernstein Chaining. But I'm
 976 |    not sure if it's exactly the same. It also looks like currying or
 977 |    partial evaluation to me if you squint a little bit.
 978 | 
 979 |    #+begin_src bash
 980 |      every() {
 981 |         secs=$1
 982 |         shift
 983 |         while true; do
 984 |           "$@"
 985 |           sleep $secs;
 986 |         done
 987 |       }
 988 |    #+end_src
 989 | 
 990 | 
 991 |    As you know by now, bash doesn't pass blocks of code around, but
 992 |    the alternative is to pass function names.
 993 | 
 994 |    #+begin_src bash
 995 |      mute() {
 996 |        $@ >/dev/null 2>/dev/null
 997 |      }
 998 |      mute ls
 999 |    #+end_src
1000 | 
1001 |    So now we can create the most useless command composition ever:
1002 | 
1003 |    #+begin_src bash
1004 |      every 1 mute echo hi
1005 |      #or
1006 |      mute every 1 echo hi
1007 |    #+end_src
1008 | 
1009 |    For the particular redirection problem, another option is to use
1010 |    aliases. Redirects can be written anywhere on your CLI (not just at
1011 |    the end), so the following will work using a plain alias:
1012 |    #+begin_src bash
1013 |      alias mute='>/dev/null 2>/dev/null'
1014 |      mute ls
1015 |    #+end_src
1016 | 
1017 | 
1018 | 
1019 |    - https://www.oilshell.org/blog/2017/01/13.html
1020 | ** do_times/foreach_*
1021 |    shellscripts are highly side-efffecty, and even though the scoping
1022 |    of variables is not very empowering, you can get a limited amount
1023 |    of decomposition of loops by passing function names.
1024 | 
1025 |    This is a lame example, but I hope it shows the use case, it allows
1026 |    you to group already existing functions while taking advantage of a
1027 |    fixed looping iterator, and leaving traces of the current loop vars
1028 |    in the global "variable" environment.
1029 | 
1030 |    #+begin_src bash
1031 |      create_user() {
1032 |        uname="u$1" # leave uname in the global env so later functions see it
1033 |        http :8080/users name="$uname"
1034 |      }
1035 | 
1036 |      create_pet() {
1037 |        pname="p$1"
1038 |        http :8080/users/$uname/pets name="$pname"
1039 |      }
1040 | 
1041 |      create_bundle() {
1042 |        create_user $1
1043 |        create_pet $1
1044 |      }
1045 | 
1046 |      do_times() {
1047 |        local n=$1; shift
1048 |        for i in $(seq $n); do
1049 |          "$@" $i
1050 |        done
1051 |      }
1052 | 
1053 |      do_times 15 create_bundle
1054 |    #+end_src
1055 | 
1056 |    A bit more complex is runnning a command to every repo in an org:
1057 | 
1058 |    #+begin_src bash
1059 |      run_tests() {
1060 |        ./ci/test.sh
1061 |      }
1062 | 
1063 |      foreach_repo_with_index() {
1064 |        local counter=0
1065 |        local repos=$(http https://api.github.com/users/$1/repos)
1066 |        shift
1067 |        for entry in $(echo $repos | jq -r '.[].git_url'); do
1068 |          (git_clone_tmp $entry master
1069 |           cd $tmpath
1070 |           "$@" $counter $entry
1071 |          )
1072 |          ((counter=counter+1))
1073 |        done
1074 |      }
1075 | 
1076 |      foreach_repo_with_index kidd run_tests
1077 |    #+end_src
1078 | 
1079 | ** <(foo) and >(foo)
1080 |    Some commands ask for files as inputs. And sometimes you have that
1081 |    file, but sometimes you're only creating that file to pass it to
1082 |    the command.  In those cases, creating temporary files is not
1083 |    necessary if you use =<(cmd)=. Here's a way to diff the output of 2
1084 |    commands without putting them in a temporary file.
1085 | 
1086 |    #+begin_src bash
1087 |      diff <(date) <(date)
1088 |      diff <(date) <(sleep 1; date)
1089 |    #+end_src
1090 | 
1091 |    The same happens with outputs. Commands that ask you for a
1092 |    destination file. You can trick them by using =>(command)= as a
1093 |    file. A nice trick is to use =>(cat)= to know what's going on
1094 |    there. Also useful to send stuff to the clipboard =>(xclip)= before
1095 |    running something on the output.
1096 | 
1097 |    What the shell does in those cases is to bind a file descriptor of
1098 |    the process created inside =< or >= to the first process.
1099 | 
1100 |    You can experiment with those using commands like =echo <(pwd)=.
1101 | 
1102 |    In Zsh you can use =m-x expand-word= to see the file descriptors
1103 |    being expanded.
1104 | 
1105 |    A way to peek into a huge pipe is to =tee >(cat)=
1106 | 
1107 | ** Modify behavior by linking files
1108 |    Imagine you want to remove the =.bash_history= file and not record
1109 |    history of commands.  You don't can't avoid bash to write to that
1110 |    file, but what you can do is to link =.bash_history= to
1111 |    =/dev/null=. That way, bash will keep appending things to the
1112 |    expected output file, without noticing that it is really writing to
1113 |    =/dev/null=. The trick is to use: =ln -sf /dev/null .bash_history=.
1114 | 
1115 | ** Modify a file on the fly, without changing it on disk
1116 |    If you want to use a slightly modified version of a file for a
1117 |    single use, and then throw it away, consider the following:
1118 | 
1119 |    Everywhere that you should pass =file=, you could pass =<(cat
1120 |    orig_file)= instead, because they have the same content, only the
1121 |    second version is ephemeral.
1122 | 
1123 |    If you want a modified version of it, we can attach more commands
1124 |    to the subshell.
1125 | 
1126 |    Example of a command that reuses your =~/.psqlrc= and adds a line
1127 |    in the end to make the session read-only.
1128 | 
1129 |    #+begin_src bash
1130 |      ropsql() {
1131 |        PSQLRC=<(
1132 |          cat ~/.psqlrc &&
1133 |          echo 'set session characteristics as transaction read only;') \
1134 |        psql "$@"
1135 |      }
1136 |    #+end_src
1137 | 
1138 | ** Use xargs
1139 |    Continuing with other ways of plumbing commands into other
1140 |    commands, there's =xargs=. Some commands work seamlessly with
1141 |    pipes, by taking inputs from stdin and printing to stdout.  But
1142 |    some others like to work with files, and they ask for their
1143 |    parameters in their args list. For example, =evince=. It wouldn't
1144 |    be even expected to cat a pdf and pass it to evince through
1145 |    stdin.
1146 | 
1147 |    In general, to convert from this pattern: =cmd param= to =echo
1148 |    param| cmd=, xargs can be helpful. Look at its man page to know how
1149 |    to split or batch args in multiple =cmd= calls.
1150 | 
1151 |    Xargs is helpful for parallelizing work. You should look at its man
1152 |    page, but just know it can help in running parallel processes
1153 |    (check =-P= in its man).
1154 | 
1155 |    Other tips on this great [[https://www.oilshell.org/blog/2021/08/xargs.html][Guide to xargs]].
1156 | 
1157 | ** xargs + paste
1158 |    =paste= can be thougth as =zip=, interleaving the outputs of
1159 |    several commands line by line.
1160 | 
1161 |    #+begin_src bash
1162 |      paste <(./cmd1) <(./cmd2) | xargs -L1 ./cmd3
1163 |    #+end_src
1164 | 
1165 |    Ref: https://news.ycombinator.com/item?id=29845232
1166 | 
1167 | ** paste?
1168 |    We just saw a use of =paste=. But this command can do all sorts of
1169 |    crazy stuff joining lines.
1170 | 
1171 |    #+begin_src bash
1172 |      paste - - -d, < file   # joins every 2 lines with comma
1173 |      paste - - - -d, < file   # joins every 3 lines with comma
1174 |      paste -s -d",\n" file  #joins 1 and 2 lines with comma, 2 and 3 with \n
1175 |    #+end_src
1176 | 
1177 | ** Scripts as filters
1178 |    Strive to create scripts that can be used as filters. Perl had
1179 |    =while(<>) { ... }= that would open a file line by line, or get
1180 |    input from stdin depending on the actual arguments to the file.
1181 | 
1182 |    In shellscripting, we can simulate (part of) it using
1183 |    #+begin_src bash
1184 |      while IFS= read -r line
1185 |      do
1186 |        echo "$line"
1187 |      done < "${1:-/dev/stdin}"
1188 |    #+end_src
1189 | 
1190 |    Look [[https://stackoverflow.com/questions/6980090/how-to-read-from-a-file-or-standard-input-in-bash][here]] for more detailed explanations and variants.
1191 | 
1192 |    Another way to create filters is to use `"${@:-cat}"`.
1193 | 
1194 |    #+begin_src bash
1195 |      o2e     () { 1>&2 "${@:-cat}"; } # out to err
1196 |      e2o     () { 2>&1 "${@:-cat}"; } # err to out
1197 |    #+end_src
1198 | 
1199 | ** Read or print to stdin inside a pipe
1200 |     You can use /dev/tty to explicitly specify that some input or
1201 |     output has to come from tty (being screen or keyboard).
1202 | 
1203 |     This is useful for code that might be called inside a pipeline,
1204 |     but you never want the contents of the pipe to be the ones that
1205 |     are =read= or =echoed=.
1206 | 
1207 |     Here's a snippet that defines a function fzf (if fzf is not
1208 |     previously defined), that will naively replace fzf.
1209 | 
1210 |    #+begin_src bash
1211 |      # fzf is either fzf or a naive version of it
1212 |      # the input is a sed line number, so it can be
1213 |      # single number: 42
1214 |      # range: 1,10
1215 |      # separate lines: 10p;50p
1216 |      mute command -v fzf ||
1217 |        fzf() {
1218 |          local in=$(cat)
1219 |          for p in "${@}"; do
1220 |            [ "$p" = "-0" ] && [ "$(echo "$in" | wc -l)" -eq 1 ] && [ ""  = "$in" ] && return 1
1221 |            [ "$p" = "-1" ] && [ "$(echo "$in" | wc -l)" -eq 1 ] && [ "" != "$in" ] && echo "$in" && return
1222 |          done
1223 |          # https://superuser.com/questions/1748550/read-from-stdin-while-piping-to-next-command
1224 |          cat -n <(echo "$in") >/dev/tty
1225 |          read -n num </dev/tty
1226 |          echo "$in" | sed -n "${num}p"
1227 |        }
1228 |    #+end_src
1229 | 
1230 |    Well... after implementing this, I discovered `select`, which allows for these kinds of flows:
1231 | 
1232 |    #+begin_src bash
1233 |      mute command -v fzf ||
1234 |      fzf() {
1235 |        local in=$(cat)
1236 |        for p in "${@}"; do
1237 |          [ "$p" = "-0" ] && [ "$(echo "$in" | wc -l)" -eq 1 ] && [ ""  = "$in" ] && return 1
1238 |          [ "$p" = "-1" ] && [ "$(echo "$in" | wc -l)" -eq 1 ] && [ "" != "$in" ] && echo "$in" && return
1239 |        done
1240 |        select opt in $(echo "$in"); do
1241 |          [ -n "$opt" ] && break
1242 |        done >/dev/tty </dev/tty
1243 |        echo $opt
1244 |      }
1245 |    #+end_src
1246 | 
1247 | ** change loops for "mapping/reducing" functions
1248 | *** find
1249 |     Many times we want to run the same operation or test to lots of
1250 |     files. Instead of looping for each file, think if =find -exec=
1251 |     would solve it. Also, find supports multiple directories.
1252 | 
1253 |     #+begin_src bash
1254 |       dirs=("/usr/local/bin" "/usr/bin")
1255 |       for d in "${dirs[@]}"; do
1256 |         for f in $(find "$d"); do
1257 |           echo "check if owner of $f is johndoe and group is johndoe"
1258 |           [ `stat -c %U:%G $f` == "johndoe:johndoe" ] || die "error"
1259 |         done
1260 |       done
1261 |     #+end_src
1262 | 
1263 |     Compare it to:
1264 | 
1265 |     #+begin_src bash
1266 |       [ $(find "$dirs[@]" -exec stat -c '%U:%G' {} \; | grep -vc "johndoe:johndoe") == "0" ] || die "error"
1267 |     #+end_src
1268 | 
1269 |     Other examples might be:
1270 |     #+begin_src bash
1271 |       # count all lines of all docx in this dir
1272 |       find . -type f -name "*docx" -exec pandoc "{}" -t plain \; | wc -l
1273 | 
1274 |       #All your files have the same owner and group permissions
1275 |       [ $(find "$files[@]" -exec stat -c '%a' {} \; | grep -Evc "^(.)\1") == "0" ]
1276 |     #+end_src
1277 | *** grep -Fvf
1278 |     Three magical flags that go well together.
1279 | 
1280 |     - =-F= Interpret the "pattern" as a Fixed string, not a pattern/regex
1281 |     - =-v= Negate the output. Print non matching lines.
1282 |     - =-f= The patterns to match are taken from a file.
1283 | 
1284 |     The cool thing about combining =-f= and =-v= is that the negative
1285 |     matches mean "lines that are not ANY of the ones in the pattern
1286 |     file". So you can do list diffing. like =sort + diff= but more
1287 |     flexible.
1288 | 
1289 |     Here's a practical case of finding version numbers that we have a
1290 |     git tag for, that do not have a title in the readme.
1291 | 
1292 |     #+begin_src bash
1293 |       f="readme.md"
1294 |       ! grep -Fvf <(grep -P "^# \d\.\d\.\d\.\d$" "$f" | sed -e 's/^# //') \
1295 |                   <(git tag | grep -P "^\d\.\d\.\d\.\d$")
1296 |     #+end_src
1297 | *** grep -Pzo
1298 |     Three flags that also combine pretty nicely.
1299 | 
1300 |     With those, we can match multiple lines using plain old grep.
1301 | 
1302 |     This is how I get only the table structure of a postgres database.
1303 | 
1304 |     #+begin_src bash
1305 |       pg_dump -s | grep -Pzo "CREATE TABLE [^;]*;\n"
1306 |     #+end_src
1307 | 
1308 |     More info on this [[https://stackoverflow.com/questions/3717772/regex-grep-for-multi-line-search-needed/7167115#7167115][stackoverflow thread]].
1309 | 
1310 | 
1311 |     Use of this pattern will use =\0='s instead of =\n=, which is good
1312 |     for the grepping, but maybe it's not good for your output. You can
1313 |     convert back =\0='s to =\n= with [[https://unix.stackexchange.com/questions/121916/convert-newlines-to-null-delimited-when-using-tail][tr.]]
1314 | 
1315 |     #+begin_src bash
1316 |       pg_dump -s | grep -Pzo "CREATE TABLE public.($tables) [^;]*;\n" | tr '\0' '\n' | ...
1317 |     #+end_src
1318 | 
1319 | 
1320 |     Alternatively, depending on your requirements, you can leverage
1321 |     =awk=, with =awk '/CREATE TABLE/,/;/'=, that won't suffer from the
1322 |     \0 problem. But then you need to explain the awk model to your
1323 |     coworker.
1324 | 
1325 | *** grep -Ec
1326 |     Say you want to find lines that match 2 words, not just 1. in whatever order.
1327 | 
1328 |     #+begin_src bash
1329 |       echo "foo bar" | grep foo | grep bar # That works, but it only works for very fixed inputs.
1330 | 
1331 |       echo "foo bar" | grep -E 'foo.*bar'  # That doesn't , because order matters.
1332 | 
1333 |       echo "foo baz" | grep -E '(foo|bar)' | grep -E '(foo|bar)' # That is not going to work either, because if only one word matches, it will match twice.
1334 |     #+end_src
1335 | 
1336 |     Here's how we can match multiple things from a list.
1337 |     =echo "foo bar" | tr ' ' '\n' |grep -Ec "(foo|bar)"=
1338 | 
1339 |     #+begin_src bash
1340 | 
1341 |       match_two () {
1342 |         while read line; do
1343 |           if [ "$(tr ' ' '\n' <<<"$line" | rg -Uc "^($search)$")" = 2 ]; then
1344 |             echo "$line"
1345 |           fi
1346 |         done
1347 |       }
1348 | 
1349 |       cat file_with_two_words_per_line.txt | match_two "foo|bar|baz"
1350 |     #+end_src
1351 | 
1352 |     With this, you can find out how many matches happened in that
1353 |     line. The example above filters lines which have 2 words from the
1354 |     'foo|bar|baz' regex.
1355 | *** Set operations
1356 | There are lots of other "set level" operations you can perform on
1357 | files/streams using basic unix tools.
1358 | 
1359 | [[https://catonmat.net/ftp/setops.txt][This cheatsheet]] contains lots of them, which you should go read one by
1360 | one, because bash, appart from being useful as it is, also has an
1361 | Iversonian [[https://www.jsoftware.com/papers/tot.htm#1.2][suggestivity]] that gets more useful with time.
1362 | 
1363 | ** jq
1364 |    Jq is more powerful than you probably think. It's not just
1365 |    to fetch a value deep into a json dictionary.
1366 | 
1367 |    You can fetch and transform values, and interpolate values inside
1368 |    a string template:
1369 | 
1370 |    #+begin_src bash
1371 |      echo '{"a":1,"b":2}' | jq '. | "a is \(.a), and b is \(.b)."'
1372 |    #+end_src
1373 | 
1374 |   Or use this in a map:
1375 |    #+begin_src bash
1376 |      echo '[{"a":1,"b":2},{"a":3,"b":4}]' | jq -r '. |map("a is \(.a), and b is \(.b).")[]'
1377 |    #+end_src
1378 | 
1379 |    Maybe a little more spicy:
1380 | 
1381 |    #+begin_src bash
1382 |    echo '[{"host":"host1.com","user":"bob"},{"host":"host2.com","user":"alice"}]' |
1383 |      jq  'map("scp \(.user)@\(.host):/tmp/remote-file /tmp/local-file")[]' -r |
1384 |      xargs -d"\n" -n1 sh -c
1385 |    #+end_src
1386 | 
1387 |    And, make sure to read about jq functions. for example, reading nested json
1388 |    #+begin_src bash
1389 |      echo '{"nested":"{\"a\":2}"}' | jq '.nested | fromjson | .a '
1390 |    #+end_src
1391 | 
1392 |    You can also edit values in place
1393 |    #+begin_src bash
1394 |      echo '{"date":1643192480}' | jq '.date|=todateiso8601' #  {"date": "2022-01-26T10:22:48Z"}
1395 |    #+end_src
1396 | 
1397 |    And we can generalize it into a filter that updates many date fields
1398 |    #+begin_src bash
1399 |      join_by() { local IFS="$1"; shift; echo "$*"; }
1400 |      # usage: echo '[{"foo":1234567890,"bar":1234567891}]' | jq_pretty_dates '.[].foo' '.[].bar'
1401 |      # => [{"foo":"2009-02-13T23:31:30Z","bar":"2009-02-13T23:31:31Z"}]
1402 |      jq_pretty_dates() {
1403 |        # From the default ($@) append every element (/%/) with
1404 |        # |=todateiso8601). That, joined by |. And that becomes the "filter" to jq.
1405 |        jq "$(join_by "|" "${@/%/|=todateiso8601}")"
1406 |      }
1407 |    #+end_src
1408 | 
1409 |    There are quite a few interesting commands in this [[https://medium.com/circuitpeople/aws-cli-with-jq-and-bash-9d54e2eabaf1'][AWS+JQ post]].
1410 | 
1411 |    Some other cool tips from [[https://news.ycombinator.com/item?id=31141415][This HN thread]]:
1412 | 
1413 |    #+begin_src bash
1414 |      echo "FFFF\nDDDD" | jq -R    # reads raw text as strings. 1 line -> 1 string
1415 |      echo "FFFF\nDDDD" | jq -R 'capture("(?<item>.*)")' # Capture groups as json elements
1416 |    #+end_src
1417 | 
1418 |    You can get pretty advanced with jq, and some say it's a decent
1419 |    substitute for =sed= as a line oriented editor.
1420 | 
1421 |    Some more advanced tutorials [[https://programminghistorian.org/en/lessons/json-and-jq][here]] and [[https://remysharp.com/drafts/jq-recipes][here]].
1422 | 
1423 | ** use jq . <<<"$var"
1424 |    Newlines aren't allowed in json strings, and you might get a jq
1425 |    error like:
1426 | 
1427 |    "parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 148, column 99"
1428 | 
1429 |    Sometimes, strings inside json might have encoded newlines like =\n=. If you would use
1430 | 
1431 |    =echo $var | jq .= , bash would interpolate the \n inside the json
1432 |    string and when it gets to jq, it's an invalid json.
1433 | 
1434 |    If instead you use =echo -E=, or =<<<"$var"=, things go smoothly.
1435 | 
1436 |    =jq . <<<"$var"=
1437 | 
1438 | ** pass flags as a splatted array
1439 |    There's quite a bit to chew on this example. First of all, the core
1440 |    pattern is to build up your commandline options with an array, and
1441 |    splat it in the final command line. For complex commands like
1442 |    =docker= where you easily have 10+ flags it's a visual aid, and
1443 |    also opens up the opportunities for reusing or abstracting sets of
1444 |    options to logical blocks.
1445 | 
1446 |    Once it's an array, we can add elements conditionally to that array
1447 |    depending on the current run, and build the line that we'll be
1448 |    running in the end.
1449 | 
1450 |    #+begin_src bash
1451 |      # Allows Ctrl-C'ing on interactive shells
1452 |      INTERACTIVE=
1453 |      if [[ -t 1 ]]; then INTERACTIVE="-it"; fi
1454 | 
1455 |      local flags=(
1456 |        # We mount it as read-only, so we make sure we are not writing anything
1457 |        # in there, and that everything is explicitly defined
1458 |        "-v $LOCAL_PATH/build-dir:/build-dir:ro,delegated"
1459 |        "-v $OUTPUT_DIR:/output:rw,consistent"
1460 |        "-v $tmp_dir:/tmp/work:rw,delegated"
1461 |      )
1462 |      if [[ -n $LOCAL_PATH ]]; then
1463 |        flags+=("-v $(realpath $LOCAL_PATH)/overrides/my-other-file:/build-dir/build.json:ro")
1464 |        flags+=("-e LOCAL_PATH=/tmp/local")
1465 |      fi
1466 | 
1467 |      local v_id=$(docker volume create)
1468 |      flags+=("-v $v_id:/tmp/build")
1469 |      on_exit "docker volume rm $v_id"
1470 | 
1471 |      docker run --rm $INTERACTIVE ${flags[*]} $image touch /tmp/build/foo.txt
1472 | 
1473 |      docker run --rm $INTERACTIVE ${flags[*]} fpm:latest fpm-build /tmp/build/foo.txt
1474 |      on_exit "chown_cache $tmp_dir"
1475 |    #+end_src
1476 | 
1477 |    In this example we see another cool trick. Mounting a volume in 2
1478 |    differrent containers, so not for the purpose of sharing a local
1479 |    file/dir with the host but to share it between themselves. In that
1480 |    case, the 2 containers don't even coexist temporarily, but use the
1481 |    volume as a conveyor belt, passing it from container to container,
1482 |    and each one applies "its thing".
1483 | 
1484 |    After all the mess, someone has to cleanup everything, but we know
1485 |    how to do it with =on_exit= trick.
1486 | 
1487 | ** inherit_errexit
1488 |    bash 4.4+ , you can =shopt -s inherit_errexit=, and subshells will
1489 |    inherit the errexit flag value. meaning that if you =set -Ee=,
1490 |    anything that runs inside a subshell will throw an error at the
1491 |    moment any command exits with =!=0=.
1492 | ** GNU Parallel
1493 |    I can't recommend [[https://www.gnu.org/software/parallel/][parallel]] enough. The same as xargs, but in a
1494 |    much more flexible way, parallel lets you run various jobs at a
1495 |    time. If you have this tool into account, it doesn't just speed up
1496 |    your runtimes, but it will force you write cleaner code. Parallel
1497 |    execution will test your scripts so if they are not using
1498 |    randomized tmp working directories, things will clash, etc...
1499 | 
1500 |    Parallel in itself is such a hackerfriendly tool it deserves to be
1501 |    deeply learned. You can use it just locally to run a process per
1502 |    core, you can send jobs to several machines connected via a simple
1503 |    ssh, you can bind tmux or sqlite to it, or you can write a trivial
1504 |    job queuing system.
1505 | 
1506 |    Man pages and official examples are a goldmine.
1507 | 
1508 | ** HEREDOCS
1509 |    - Basic usage of heredocs:
1510 |    #+begin_src bash
1511 |      echo <<EOF
1512 | $interpolated
1513 | \$non_interpolated
1514 | EOF
1515 |    #+end_src
1516 | 
1517 |    - A dash after =<<= replaces trailing spaces in here docs
1518 |    #+begin_src bash
1519 |      echo <<-EOF
1520 |      $var
1521 |      there
1522 |      EOF
1523 |    #+end_src
1524 | 
1525 |    - quoting the identifier disables interpolation of variables
1526 |    #+begin_src bash
1527 |      echo <<'EOF'
1528 |      $non_interpolated
1529 |      there
1530 |      EOF
1531 |    #+end_src
1532 | 
1533 |    The [[http://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Here-Documents][bash manual]] is super concise and to the point there.
1534 | 
1535 |    The most complex case is having a heredoc which contains some bash
1536 |    code that is ment to run remotely, and wanting to interpolate some
1537 |    variables in the local env, and some in the remote one. [[https://stackoverflow.com/questions/4937792/using-variables-inside-a-bash-heredoc][Escaping is
1538 |    enough for regular]] vars, but if you want to use special (=$1, $!,
1539 |    $?=...) vars, check this out:
1540 | 
1541 |    #+begin_src bash
1542 |      deploy () {
1543 |        local deploy_dir="$1"
1544 |        ssh server1 'bash -s' <<EOSSH
1545 |        echo "deploying => ${deploy_dir}"
1546 |        ls -tdr ~/my-project/artifacts/* | head -n -5 | xargs rm -rf   # remove all but latest 5 directories
1547 |        ln -nfs "\$HOME/my-project/artifacts/${deploy_dir}/my-project-0.1.0-SNAPSHOT-standalone.jar" "\$HOME/my-project/artifacts/current.jar"
1548 |        cat ~/my-project/artifacts/current.pid | xargs kill
1549 |        java -jar ~/my-project/artifacts/current.jar &>/dev/null &
1550 |        lastpid=\$(echo \$!)
1551 |        echo \$lastpid > ~/my-project/artifacts/current.pid
1552 |      EOSSH
1553 |      }
1554 |    #+end_src
1555 | 
1556 |    Refs: https://unix.stackexchange.com/questions/60688/how-to-defer-variable-expansion
1557 | 
1558 | ** HERESTRINGS
1559 |    It's the stripped down version of HEREDOCS. Inline a single string
1560 |    (or output of a single command) as an input string.
1561 | 
1562 |    It's [[https://stackoverflow.com/questions/18116343/whats-the-difference-between-here-string-and-echo-pipe][kinda similar]] to what you could do with a pipe.
1563 |    #+begin_src bash
1564 |      cat <<<"HELLO"
1565 |      cat <<<$(echo "HELLO")
1566 |      echo "ECHOPIPE" | /bin/cat
1567 |      echo "ECHOPIPE" | /bin/cat <(seq 5)
1568 |      echo "ECHOPIPE" | /bin/cat <(seq 5) -
1569 |      echo "ECHOPIPE" | /bin/cat <(seq 5) <<<"HERESTRING"
1570 |      echo "ECHOPIPE" | /bin/cat <(seq 5) - <<<"HERESTRING"
1571 |    #+end_src
1572 | 
1573 | ** __DATA__
1574 |    I loved the Perl [[https://perldoc.perl.org/perldata#Special-Literals][__END__ and __DATA__]], features and realized it's
1575 |    possible to do it in shellscripts.
1576 | 
1577 |    You can append to the current file. Here's an example of a super
1578 |    simple bookmark "manager":
1579 |    #+begin_src bash
1580 |      #!/usr/bin/env sh
1581 | 
1582 |      cmd_add() {
1583 |          shift
1584 |          echo "$@" >> "$0"
1585 |      }
1586 | 
1587 |      cmd_go() {
1588 |          sed '0,/^__DATA__$/d' "$0" |
1589 |          dmenu -i -l 20  |
1590 |          rev | cut -f1 -d' ' | rev |
1591 |          xargs xdg-open
1592 |      }
1593 | 
1594 |      main() {
1595 |          cmd_${1:-go} $@
1596 |      }
1597 | 
1598 |      main $@
1599 |      exit
1600 | 
1601 |      __DATA__
1602 |      r/emacs             https://www.reddit.com/r/emacs
1603 |    #+end_src
1604 | ** Simple Templating
1605 |    For trivial templating, there's no need for any external tool
1606 |    except for just variable interpolation. This is quite obvious, but
1607 |    I use it quite often to generate repetitive bash code itself
1608 | 
1609 |    #+begin_src bash
1610 |    #!/usr/bin/env bash
1611 |    is=("obj1" "obj2")
1612 |    js=("resource1" "resource2")
1613 | 
1614 |    for i in "${is[@]}"; do
1615 |      for j in "${js[@]}"; do
1616 |        echo "insert into t (obj, resource) values ('$i', '$j');"
1617 |      done
1618 |    done
1619 |    #+end_src
1620 | 
1621 | ** Secrets
1622 |    As, [[https://smallstep.com/blog/command-line-secrets/][Smallstep article]] explains, you should be careful when passing
1623 |    around secret data like tokens between processes.
1624 | 
1625 |    Env vars are not really safe, and there are a few tricks you can
1626 |    use to cover your assets.
1627 | 
1628 | ** Escape single quotes in single quoted strings
1629 |    Escaping single quoted strings doesn't work the way you'd think.
1630 | 
1631 |    #+begin_src bash
1632 |      # does not work
1633 |      a='hi \'bob\''
1634 |    #+end_src
1635 | 
1636 |    Use =$''= strings like:
1637 |    #+begin_src bash
1638 |      # works!
1639 |      a=$'hi \'bob\''
1640 |      echo $a
1641 |    #+end_src
1642 | 
1643 |    Extra explanations: [[https://stackoverflow.com/questions/1250079/how-to-escape-single-quotes-within-single-quoted-strings][StackOverflow]]
1644 | 
1645 | ** any, every, lambdas, closures, and eval
1646 | Bash has no closures, and it's not "functional", but we're going to
1647 | see how we can combine some tricks and get decent high-level functions
1648 | 
1649 | Here's an implementation of =any= and =every=
1650 | 
1651 | #+begin_src bash
1652 | any() {
1653 |   local pred="$1"; shift
1654 |   for i in "$@"; do
1655 |     if $pred $i; then
1656 |       return 0
1657 |     fi
1658 |   done
1659 |   return 1
1660 | }
1661 | 
1662 | every() {
1663 |   local pred="$1"; shift
1664 |   for i in "$@"; do
1665 |     if ! $pred $i; then
1666 |       return 1
1667 |     fi
1668 |   done
1669 |   return 0
1670 | }
1671 | #+end_src
1672 | 
1673 | The usage is by passing a callback function that will return 0 or 1.
1674 | 
1675 | #+begin_src bash
1676 |   usage() {
1677 |     echo "help message"
1678 |   }
1679 | 
1680 |   is_help() {
1681 |     [ "$1" = "--help" ]
1682 |   }
1683 | 
1684 |   any is_help "$@" && usage
1685 | #+end_src
1686 | 
1687 | That's already cool.
1688 | 
1689 | Did you know that functions can start also with dashes?
1690 | 
1691 | #+begin_src bash
1692 |   usage() {
1693 |     echo "help message"
1694 |   }
1695 | 
1696 |   --help() {
1697 |     [ "$1" = "--help" ]
1698 |   }
1699 | 
1700 |   any --help "$@" && usage
1701 | #+end_src
1702 | 
1703 | That also works. It's kinda obfuscated a bit, but hey....
1704 | 
1705 | We can also, with the help of =eval=, create on demand functions. They
1706 | are global, but as we're not going to multithread, we can assume it's
1707 | ok.
1708 | 
1709 | #+begin_src bash
1710 |   usage() {
1711 |     echo "help message"
1712 |   }
1713 | 
1714 |   any_eq() {
1715 |     eval "_() { [ $1 = \$1 ] ; }"
1716 |     shift
1717 |     any _  "$@"
1718 |   }
1719 | 
1720 |   any_eq "--help" "$@" && usage
1721 |   # or,
1722 |   any_eq --help "$@" && usage
1723 | #+end_src
1724 | 
1725 | And the last trick, using =command_not_found_handle=, we can get into
1726 | a kind of rails generators for these helpers
1727 | 
1728 | #+begin_src bash
1729 |   command_not_found_handle() {
1730 |     IFS=_ read -a cmd <<<"$1"
1731 |     [ "eq" = "${cmd[0]}" ] && eval "$1() { [ \"${cmd[1]}\" = \"\$1\" ] ; }"
1732 |     "$@"
1733 |   }
1734 |   any eq_--help "$@" && usage
1735 | #+end_src
1736 | 
1737 | All these last metaprogramming tricks are not thread safe, and even
1738 | though it's mostly ok if you use them in isolation, maybe pipes or
1739 | coprocs would mess up your inline functions.
1740 | 
1741 | * Interactive
1742 | ** Save your small scripts
1743 |    Rome wasn't built in a day, and like having a journal log, most of
1744 |    the little scripts you create, once you have enough discipline will
1745 |    be useful for some other cases, and your functions will be
1746 |    reusable.
1747 | 
1748 |    Save your scripts into files early on, instead of crunching
1749 |    everything in the repl. learn how to use a decent editor that
1750 |    shortens the feedback cycle as much as possible.
1751 | ** Increased Interactivity
1752 |    Knowing your shell's shortcuts for interactive use is a must. The
1753 |    same way you learned to touchtype and you learned your editor, you
1754 |    should learn all the shortcuts for your shell. Here's some of them.
1755 | 
1756 |    | key    | action                              |
1757 |    |--------+-------------------------------------|
1758 |    | Ctrl-r | reverse-history-search              |
1759 |    | C-a    | beginning-of-line                   |
1760 |    | C-e    | end-of-line                         |
1761 |    | C-w    | delete-word-backwards               |
1762 |    | C-k    | kill-line (from point to eol)       |
1763 |    | C-y    | paste last killed thing             |
1764 |    | A-y    | previous killed thing (after a c-y) |
1765 |    | C-p    | previous-line                       |
1766 |    | C-n    | next-line                           |
1767 |    | A-.    | insert last agument                 |
1768 |    | A-/    | dabbrev-expand                      |
1769 | 
1770 |    A written form of =A-.= is =$_=. It retains the last argment and
1771 |    puts it in $_.  =test -f "FILE" && source "$_"=.
1772 | 
1773 | ** Aliases
1774 |    Aliases are very simple substitutions of commands for a sequence of
1775 |    other commands.  Usual example is
1776 | 
1777 |    #+begin_src bash
1778 |      alias ls='ls --auto-color'
1779 |    #+end_src
1780 | 
1781 |    Now let's move on to the interesting stuff.
1782 | 
1783 | ** functions can generate aliases
1784 | 
1785 |    Aliases live in a global namespace for the shell, so no matter
1786 |    where you define them, they take effect globally, possibly
1787 |    overwriting older aliases with the same name.
1788 | 
1789 |    Well, it's not lexical scope (far from it), but using aliases you
1790 |    can create a string that snapshots the value you want, and capture
1791 |    it to run it later.
1792 | 
1793 |    Some fun stuff:
1794 | 
1795 |    - aliasgen. Create an alias for each directory in
1796 |      ~/workspace/. This is superceeded by =CDPATH=, but the trick is
1797 |      still cool.
1798 |    #+begin_src bash
1799 |      aliasgen() {
1800 |        for i in ~/workspace/*(/) ; do
1801 |            DIR=$(basename $i) ;
1802 |             alias $DIR="cd ~/workspace/$i";
1803 |        done
1804 |      }
1805 |      aliasgen
1806 |    #+end_src
1807 | 
1808 |    - a make a shortcut to the current directory.
1809 |    #+begin_src bash
1810 |      function a() { alias $1=cd\ $PWD; }
1811 | 
1812 |      mkdir -p /tmp/fing-longer
1813 |      cd /tmp/fing-longer
1814 |      a fl
1815 |      cd /
1816 |      fl
1817 |      echo $PWD   # /tmp/fing-longer
1818 |    #+end_src
1819 |    A man can dream...
1820 | 
1821 | 
1822 |    - unhist. functions can create aliases, and functions can receive
1823 |      functions as parameters (as a string (function name)), so we can
1824 |      combine them to advice existing functions.
1825 |      #+begin_src bash
1826 |        unhist () {
1827 |          alias $1=" $1"
1828 |        }
1829 |        unhist unhist
1830 |        unhist grep
1831 |        unhist rg
1832 | 
1833 |        noglobber() {
1834 |            alias $1="noglob $1"
1835 |        }
1836 |        noglobber http
1837 |        noglobber curl
1838 |        noglobber git
1839 | 
1840 |      #+end_src
1841 | 
1842 |      #+RESULTS:
1843 | 
1844 |    - Problem: These commands do not compose. Combination of 2 of those
1845 |      doesn't work, because the second acts just on the textual
1846 |      representation that it received, not the current value of the
1847 |      alias.
1848 | 
1849 |    # Solution :
1850 |    #   #+begin_src bash
1851 |    #   alias-to() {
1852 |    #     alias $1 | sed -e "s/.*='//" -e "s/'\$//"
1853 |    #   }
1854 | 
1855 |    #   aliasappend() {
1856 |    #     local cmd
1857 |    #     if alias $1 >/dev/null; then
1858 |    #       cmd=$(alias-to $1)
1859 |    #     else
1860 |    #       cmd=$1
1861 |    #     fi
1862 |    #     echo $cmd
1863 |    #   }
1864 | 
1865 |    #   muter() {
1866 |    #     local c
1867 |    #     c=$(aliasappend $1)
1868 |    #     alias $1="$c >/dev/null"
1869 |    #   }
1870 | 
1871 |    #   unhist() {
1872 |    #     local c
1873 |    #     c=$(aliasappend $1)
1874 |    #     alias $1=" $c"
1875 |    #   }
1876 | 
1877 |    #   unhist ls
1878 |    #   muter ls
1879 |    #   ls
1880 |    # #+end_src
1881 | ** Override (advice?) common functions
1882 |    Overriding commands is generally a bad practice as it violates the
1883 |    principle of least surprise, but there might be occasions (mostly
1884 |    in your local machine) where you can integrate awesome finetunnings
1885 |    to your toolbelt.
1886 | 
1887 |    Here we're going to get the original docker binary file
1888 |    location. After that we declare a function called =docker= that
1889 |    will proxy the parameters to the original =docker= program UNLESS
1890 |    you're calling =docker run=. In that case, we're injecting a mouted
1891 |    volume that mounts =/root/.bash_history= of the container to a file
1892 |    hosted in the host (duh). That's a pretty cool way of keeping a
1893 |    history of your recent commands in your containers, no matter how
1894 |    many times you start and kill them.
1895 | 
1896 |    #+begin_src bash
1897 |      DOCKER_ORIG=$(which docker)
1898 |      docker () {
1899 |          if [[ $1 == "run" ]]; then
1900 |              shift
1901 |              $DOCKER_ORIG run -v $HOME/.shared_bash_history:/root/.bash_history "$@"
1902 |          else
1903 |              $DOCKER_ORIG "$@"
1904 |          fi
1905 |      }
1906 |    #+end_src
1907 | 
1908 |    I'm particularly fond of this trick, as it saved me tons of
1909 |    typing. But at a personal level, it was mindblowing that sharing
1910 |    this around the internet caused the most disparity of opinions.
1911 |    Also, I recently read the great book "Docker in Practice" by [[https://github.com/ianmiell][Ian
1912 |    Miell]] and there's a snippet that is 99.9% like the one I
1913 |    created myself. That was a very cool moment.
1914 | 
1915 | ** Faster iteration on pipes
1916 |    When testing complex pipelines:
1917 | 
1918 |    - Make them pure (no side effects).
1919 |    - One command per line.
1920 |    - End lines with the pipe character.
1921 |    - During development, end the pipeline with =cat=.
1922 | 
1923 |    I usually use =watch -n1 'code.sh'= in a split window so I see the
1924 |    results of what I'm doing. The advantage of
1925 | 
1926 |    #+begin_src bash
1927 |      curl https://www.example.com/videos/              |
1928 |        pup 'figure.mg > a attr{href}'                  |
1929 |        head -1                                         |
1930 |        xargs -I{} curl -L "https://www.example.com/{}" |
1931 |        pup 'script'                                    |
1932 |        grep file:                                      |
1933 |        sed -e "s/.*\(http[^ \"']*\).*/\1/"             |
1934 |      # xargs vlc                                       |
1935 |        cat
1936 |    #+end_src
1937 | 
1938 |    Over
1939 |    #+begin_src bash
1940 |      curl https://www.example.com/videos/                \
1941 |        | pup 'figure.mg > a attr{href}'                  \
1942 |        | head -1                                         \
1943 |        | xargs -I{} curl -L "https://www.example.com/{}" \
1944 |        | pup 'script'                                    \
1945 |        | grep file:                                      \
1946 |        | sed -e "s/.*\(http[^ \"']*\).*/\1/"             \
1947 |      # xargs vlc   # doesn't work
1948 |    #+end_src
1949 | 
1950 |    Is that you can comment out lines on the former one, but you can't
1951 |    do that on the latter. The =cat= trick makes it so that you have an
1952 |    'exit' point, and you don't have to comment that one. Also, some
1953 |    editors will indent the first one correctly, while you'll have to
1954 |    manually indent the second one.
1955 | 
1956 |    Small wins that compose just fine :)
1957 | 
1958 | ** Use ${var?error msg} on templates
1959 |    If you write something to be copypasted by your user and filled in,
1960 |    instead of =<var>=, try =${var?You need to set var}=. it allows for the user to set
1961 |    the variable in the environment without having to replace inline,
1962 |    and if the user forgets any, the shell will barf.
1963 | 
1964 | ** Idempotent functions
1965 | Being able to call the same function multiple times even if you just
1966 | called it, without having to account for "oh, did this already happen
1967 | or not?" but instead have a mental model of "I need this to have
1968 | happened after this function call", and consider it done, is very
1969 | liberating from the code perspective.
1970 | 
1971 | Also, there are subgoals, like caching slow functions, downloads, etc.
1972 | 
1973 | *** nop subsequent calls
1974 |    "My favourite shell scripting function definition technique:
1975 |    idempotent functions by redefining the function as a noop inside
1976 |    its body:
1977 | 
1978 |    (The `true;` body is needed by some shells, e.g. bash, and not
1979 |    others, e.g. zsh.)  " -- [[https://news.ycombinator.com/item?id=27729397][chrismorgan]]
1980 | 
1981 |    #+begin_src bash
1982 |   foo() {
1983 |       foo() { true; }
1984 |       echo "This bit will only be executed on the first foo call"
1985 |   }
1986 |    #+end_src
1987 | *** Download once
1988 | A function might be sideffecty, as in "download files". These sort of
1989 | cases, you want to download the file only if you haven't downloaded it
1990 | yet.
1991 | 
1992 | =cache= here gets a cache "id" where it'll store the results of =http=
1993 | and next time, when called with the same param, it will reuse the saved version.
1994 | 
1995 | This pattern applies to many kinds of functions, so with slight
1996 | modifications you can adapt it.
1997 | 
1998 | #+begin_src bash
1999 | cache() {
2000 |   local path=$CACHE_DIR/$1
2001 |   [[ -f $path ]] && cat $path && return 0
2002 | 
2003 |   shift
2004 |   $@ | tee $path
2005 | 
2006 |   return 1
2007 | }
2008 | 
2009 | get_repos() {
2010 |   local url="https://api.github.com/search/repositories?q=user:$USER&access_token=$GITHUB_TOKEN&per_page=100&page=$1"
2011 |   cache $USER-repos-$1 http $url
2012 | }
2013 | #+end_src
2014 | 
2015 | *** memoize with ttl, with ttl
2016 | What if you want to cache for some time?
2017 | 
2018 | The tricks here:
2019 | 
2020 | - Use a file as a sentinel for the cache.
2021 | - Create a file, set it's modification date to now()+12hours
2022 | - check with `-nt` (newer than) and a temporary file (to have a
2023 |   =now()= value in "file" format).
2024 | 
2025 | #+begin_src bash
2026 |   hmacache="/tmp/.hmacache"
2027 |   is_recent_cache() {
2028 |     [ "$hmacache" -nt <(echo 1) ]
2029 |   }
2030 |   update_cache() {
2031 |     touch -m -t "$(date -v+12H +"%Y%m%d%H%M.%S")" "$hmacache"
2032 |   }
2033 | 
2034 |   main() {
2035 |     #...
2036 |     if ! is_recent_cache ; then
2037 |       if ! curl -s https://hello.ts.net > /dev/null; then
2038 |         die "Tailscale must be connected to start Harbormaster"
2039 |       fi
2040 |       # other slow checks
2041 |       update_cache
2042 |     fi
2043 |   }
2044 | #+end_src
2045 | 
2046 | * Debugging
2047 | ** adding =bash= to a script to debug
2048 |    You can add =bash= inside any script, and it'll add a sort of
2049 |    a breakpoint, allowing you to check the state of the env and
2050 |    manually call functions around.
2051 | 
2052 |    If you orgainse your code in small functions, it's easy to add
2053 |    breakpoints by just spawning bash processes inside your script.
2054 | 
2055 |    This works also inside docker containers (if you provide =-ti= flag
2056 |    on run).
2057 | 
2058 |    Let's see some usual uses of docker and how we can debug our
2059 |    scripts there:
2060 | 
2061 |    #+begin_src bash
2062 |      # leaves you at a shell to fiddle if all is in place after build
2063 |      docker run -it mycomplex-image bash
2064 | 
2065 |      # Runs /tmp/file.sh from the host inside. That's cool to make the
2066 |      # container less airtight. Even if the image is not originally meant
2067 |      # to, you can even override it and 'monkeypatch' the file with the one
2068 |      # from the host anyway.
2069 |      docker run -it -v $PWD:/tmp/ mycomplex-image /tmp/file.sh
2070 | 
2071 |      # So now you can really add wtv you want there.
2072 |      echo 'bash' >>$PWD/file.sh
2073 | 
2074 |      # run+open shell at runtime to inspect the state of the script
2075 |      docker run -it -v $PWD:/tmp/ mycomplex-image /tmp/file.sh
2076 |    #+end_src
2077 | 
2078 | ** DRY_RUN
2079 |    #+begin_src bash
2080 |      if [[-n "$DRY_RUN" ]]; then
2081 |        curl () {
2082 |          echo curl "$@"
2083 |        }
2084 |      fi
2085 |    #+end_src
2086 |    use =command curl= to force the command, not the alias or anything
2087 | 
2088 |    If you have many functions to mock, and you want to overabstract (bad thing):
2089 | 
2090 |    #+begin_src bash
2091 | 
2092 |    mock() {
2093 |      for m in ${MOCKS[@]//,/ }; do
2094 |        eval "$m() { echo \"mocked $m\" "\$@"; }"
2095 |      done
2096 |    }
2097 | 
2098 |    MOCKS=curl,ls script.sh
2099 |    #+end_src
2100 | 
2101 | 
2102 | 
2103 | ** Cheap debugging flag
2104 | 
2105 |    #+begin_src bash
2106 |      optargs "V" option; do
2107 |      case $option in
2108 |        V)
2109 |          set -xa
2110 |        ;;
2111 |    #+end_src
2112 | 
2113 | ** explore a pipe with tee >(some_command) |
2114 |    the =>()= is not very easy to use. Very few places where it
2115 |    fits. Here's a nice pipe inspector though, using =tee >(cat 1>&2)=
2116 |    trick.
2117 |    #+begin_src bash
2118 |      plog() {
2119 |        # tee >(cat 1>&2)
2120 |        local msg=${1:-plog}
2121 |        tee >(sed -e "s/^/[$msg] /" 1>&2)
2122 |      }
2123 |      alias -g 'PL'=' |plog ' #zsh only
2124 | 
2125 |      echo "a\nb" PL foo | tr 'a-z' 'A-Z' PL bar
2126 |      # output:
2127 |      # [foo] a     # stderr
2128 |      # [foo] b     # stderr
2129 |      # A           # stdout
2130 |      # B           # stdout
2131 |      # [bar] A     # stderr
2132 |      # [bar] b     # stderr
2133 |      #+end_src
2134 | 
2135 | 
2136 |    ref: https://stackoverflow.com/questions/17983777/shell-pipe-to-multiple-commands-in-a-file
2137 | 
2138 | ** tee+sudo
2139 |    If you wat to store a file in a root-owned dir, in the middle of
2140 |    your pipeline, instead of running the whole thing as root, you can
2141 |    use =sudo tee file=:
2142 |    #+begin_src bash
2143 |      ls | grep m >/usr/local/garbage  # fail
2144 |      ls | grep m | sudo tee /usr/local/garbage # success!
2145 |    #+end_src
2146 | ** quoting
2147 |    Bash: To get a quoted version of a given string, here's what you can do:
2148 |    #+begin_src bash
2149 | 
2150 |      # this is my "string" I want to 'comment "on"'
2151 |      !:q
2152 |    #+end_src
2153 | 
2154 |    That gives us ='#this is my "string" I want to '\''comment
2155 |    "on"'\'''=. Neat!
2156 | 
2157 |    I just found this trick [[https://til.simonwillison.net/til/til/bash_escaping-a-string.md][here]]. From the associated HN thread:
2158 |    #+begin_src bash
2159 |      function bashquote () {
2160 |        printf '%q' "$(cat)"
2161 |        echo
2162 |      }
2163 |    #+end_src
2164 | 
2165 |    Zsh: If you're on zsh, =a-'= quotes the current line.
2166 | * zsh-only
2167 | ** Word spliting
2168 |    Word splitting works differently by default in zsh than in bash.
2169 |    #+begin_src bash
2170 |      foo="ls -las"
2171 |      $foo
2172 |    #+end_src
2173 |    This works in bash, but zsh will not split by words. To make zsh
2174 |    expand by words, there are 2 ways: =setopt SH_WORD_SPLIT= and
2175 |    =${=foo}=.  zsh has =unsetop= command, which allows to scope where
2176 |    you want the expansions to happen. =unsetop SH_WORD_SPLIT=.
2177 | 
2178 |    The problem with both solutions is that none of them are compatible
2179 |    with bash, so you'll be cornering yourself to "this only works in
2180 |    zsh".  A way to overcome this is to use arrays, which are expanded
2181 |    in the same way in both shells.
2182 | 
2183 |    Or, use the same hack as you'll see later with noglob.
2184 | 
2185 |    Refs:
2186 |    - https://stackoverflow.com/questions/6715388/variable-expansion-is-different-in-zsh-from-that-in-bash
2187 |    - http://zsh.sourceforge.net/FAQ/zshfaq03.html#l18
2188 | ** globbing
2189 |    In zsh, getting a list of files that match some characteristics is
2190 |    doable using globbing. Bash has globbing also, but in a less sophisticated way.
2191 | 
2192 |    The basic structure of a =glob= is =pattern(qualifiers)=. Patterns
2193 |    can contain:
2194 |    - strings: they do exact match
2195 |    - wildcards:  =*=, =?=, =**/=
2196 |    - character classes: =[0-9]=
2197 |    - choices:  =(.pdf|.djvu)=
2198 | 
2199 |    The qualifiers are extra constraints you put on the matches. There
2200 |    are lots of different qualifiers. Look at =zshexpn= for the
2201 |    complete list. The ones I use more are:
2202 | 
2203 |    - =.= Files
2204 |    - =/= Directories
2205 |    - =om[numberhere]=. Nth latest modified
2206 | 
2207 | ** Global aliases:
2208 |    Zsh has 3 kinds of aliases: normal, global, and suffix. Normal ones
2209 |    behave the same as in bash: They expand only in the first token of
2210 |    a line.
2211 | 
2212 |    #+begin_src bash
2213 |    alias foo=ls
2214 |    foo foo # -> ls: cannot access 'foo': No such file or directory
2215 |    #+end_src
2216 | 
2217 |    Global aliases expand anywhere in your cli, and they can expand to
2218 |    anything.
2219 | 
2220 |    #+begin_src bash
2221 |    alias  -g foo=ls
2222 |    foo foo # -> ls: cannot access 'ls': No such file or directory
2223 |    #+end_src
2224 | 
2225 |    These are some aliases I have in my ~/.zshrc that help me use a
2226 |    shell in a more fluid way.
2227 | 
2228 |    #+begin_src bash
2229 |      alias -g P1='| awk "{print \$1}"'
2230 |      alias -g P2='| awk "{print \$2}"'
2231 |      alias -g P3='| awk "{print \$3}"'
2232 |      alias -g P4='| awk "{print \$4}"'
2233 |      alias -g P5='| awk "{print \$5}"'
2234 |      alias -g P6='| awk "{print \$6}"'
2235 |      alias -g PL='| awk "{print \$NF}"'
2236 |      alias -g PN='| awk "{print \$NF}"'
2237 |      alias -g HL='| head -20'
2238 |      alias -g H='| head '
2239 |      alias -g H1='| head -1'
2240 |      alias -g TL='| tail -20'
2241 |      alias -g T='| tail '
2242 |      alias -g T1='T -1'
2243 |      #alias -g tr='-ltr'
2244 |      alias -g X='| xclip  '
2245 |      alias -g TB='| nc termbin.com 9999 '
2246 |      alias -g L='| less -R '
2247 |      alias -g LR='| less -r '
2248 |      alias -g G='| grep '
2249 |      alias -g GI='| grep -i '
2250 |      alias -g GG=' 2>&1 | grep '
2251 |      alias -g GGI=' 2>&1 | grep -i '
2252 |      alias -g GV='| grep -v '
2253 |      alias -g V='| grep -v '
2254 |      alias -g TAC='| tac '
2255 |      alias -g DU='du -B1'
2256 | 
2257 |      alias -g E2O=' 2>&1 '
2258 |      alias -g NE=' 2>/dev/null '
2259 |      alias -g NO=' >/dev/null '
2260 | 
2261 |      alias -g WC='| wc -l '
2262 | 
2263 |      alias -g J='| noglob jq'
2264 |      alias -g JQ='| noglob jq'
2265 |      alias -g jq='noglob jq'
2266 |      alias -g JL='| noglob jq -C . | less -R '
2267 |      alias -g JQL='| noglob jq -C . | less -R '
2268 |      alias -g XMEL='| xmlstarlet el'
2269 |      alias -g XML='| xmlstarlet sel -t -v '
2270 | 
2271 |      alias -g LYNX="| lynx -dump -stdin "
2272 |      alias -g H2T="| html2text "
2273 |      alias -g TRIM="| xargs "
2274 |      alias -g XA='| xargs -d"\n" '
2275 |      alias -g XE="| xargs e"
2276 |      alias -g P="| pick "
2277 |      alias -g PP="| percol | xargs "
2278 |      alias -g W5="watch -n5 "
2279 |      alias -g W1="watch -n1 "
2280 | 
2281 | 
2282 |      alias -g CB="| col -b "
2283 |      alias -g NC="| col -b "
2284 |      alias -g U='| uniq '
2285 |      alias -g XT='urxvt -e '
2286 |      alias -g DM='| dmenu '
2287 |      alias -g DMV='| dmenu -i -l 20 '
2288 | 
2289 |      alias -g ...='../..'
2290 |      alias -g ....='../../..'
2291 |      alias -g .....='../../../..'
2292 | 
2293 |      alias -g l10='*(om[1,10])'
2294 |      alias -g l20='*(om[1,20])'
2295 |      alias -g l5='*(om[1,5])'
2296 |      alias -g l='*(om[1])'
2297 |      alias -g '**.'='**/*(.)'
2298 |      alias -g lpdf='*.pdf(om[1])'
2299 |      alias -g lpng='*.png(om[1])'
2300 | 
2301 |      alias -g u='*(om[1])'
2302 | 
2303 |      alias lsmov='ls *.(mp4|mpg|mpeg|avi|mkv)'
2304 |      alias lspdf='ls *.(pdf|djvu)'
2305 |      alias lsmp3='ls *.mp3'
2306 |      alias lspng='ls *.png'
2307 |    #+end_src
2308 |    Now, some sequences of words can start making sense:
2309 | 
2310 |    - =lspdf -tr TL DM XA evince=
2311 |    - =docker exec -u root -ti $(docker ps -q H1) bash=
2312 |    - =docker ps DM P1 XA docker stop=
2313 |    - =docker ps P1 XA docker stop= (P1 XA does an ad-hoc [[https://bash-my-aws.org/pipe-skimming/][pipe-skimming]])
2314 | 
2315 | ** Expansion of global aliases
2316 |    Let's dig deeper.
2317 |    #+begin_src zsh
2318 |    alias -g DOCK='docker ps 2..N P P1'
2319 |    DOCK  # works fine. Echoes the id of the chosen container
2320 |    docker stop DOCK   # does not work , because it expands to docker stop docker ps 2..N P P1
2321 |    docker stop $(DOCK) # works fine again
2322 |    alias -g DOCK='$(docker ps 2..N P P1)'
2323 |    docker stop DOCK   # yay!
2324 |    #+end_src
2325 | 
2326 |    So, you can bind words to expansion-time results of the aliases. It
2327 |    feels like a very powerful thing, to have this "compile time"
2328 |    expansions. Reminds me of CL's symbol-macrolet, or IMMEDIATE Forth
2329 |    words.
2330 | ** Autocomplete
2331 |    Writting smart autocompletion scripts is not easy.
2332 | 
2333 |    zsh supports =compdef _gnu_generic= type of completion, which gets
2334 |    you very far with 0 effort.
2335 | 
2336 |    When autocompleting after a =-= in the commandline, if your command
2337 |    is configured like =compdef _gnu_generic mycommand=, zsh will call
2338 |    the script with =--help= and parse the output, trying to find
2339 |    flags, and will use them as suggestions. It's really great.
2340 | 
2341 |    The compromise is to write a decent "--help" for your script. Which
2342 |    is cool because your user will love it too, and you just have to
2343 |    write it once.
2344 | 
2345 |    The completion is not context aware though, so you can't
2346 |    autocomplete flags after the first non-flag argument. It seems this
2347 |    could be improved in zsh-land, by asking for the --help like
2348 |    =mycommand args-so-far --help=. But it doesn't work like that.
2349 | 
2350 | 
2351 |    #+begin_src zsh
2352 |      #!/usr/bin/env bash
2353 |      # The script can be bash-only, while the completion work in zsh-only
2354 |      set -Eeuo pipefail
2355 | 
2356 |      help() {
2357 |        echo "  -h,--help    Show help"
2358 |        echo "  -c,--command Another thing"
2359 |      }
2360 | 
2361 |      if [ "$1" == "--help" ];
2362 |        help
2363 |      fi
2364 |    #+end_src
2365 | 
2366 |    Now you can play with =mycommand -<TAB>=. Amazing, wow.
2367 | 
2368 | ** Create helpers and generate global aliases automagically
2369 |    Borrowing a bit from Perl, a bit from Forth, and a bit from
2370 |    PicoLisp, I've come to create a few helpers that abstract words
2371 |    into a bit higher level concepts. Unifying the option selectors is
2372 |    one, and then, other line oriented operations like =chomp, from,
2373 |    till=.
2374 | 
2375 |    #+begin_src bash
2376 |      pick() {
2377 |        if [ -z "$DISPLAY" ]; then
2378 |          percol || fzf || slmenu -i -l 20
2379 |        else
2380 |          dmenu -i -l 20
2381 |        fi
2382 |      }
2383 |      alias -g P='| pick'
2384 | 
2385 |      globalias() {
2386 |        alias -g `echo -n $1 | tr '[a-z]' '[A-Z]'`=" | $1 "
2387 |      }
2388 | 
2389 |      globalias fzf
2390 | 
2391 |      # uniquify column
2392 |      function uc () {
2393 |        awk -F" " "!_[\$$1]++"
2394 |      }
2395 |      globalias uc
2396 | 
2397 |      function from() { perl -pe "s|.*?$1|\1|" }
2398 |      globalias from
2399 | 
2400 |      function till() { sed -e "s|$1.*|$1|" }
2401 |      globalias till
2402 | 
2403 |      function chomp () { sed -e "s|.$||" }
2404 |      globalias chomp
2405 |    #+end_src
2406 | 
2407 |    Again, it's a pity those do not compose well. Just be well
2408 |    organized, or build a more elaborate hack so you can compose
2409 |    aliases with some sort of confidence. It'll always be a hack
2410 |    though.
2411 | ** suffix aliases don't have to match a filename
2412 |    zsh has another type of aliases called "suffix alias". Those alias
2413 |    allow you to define programs to open/run file types.
2414 |    #+begin_src shell
2415 |    alias -s docx="libreoffice"
2416 |    #+end_src
2417 | 
2418 |    With this said, if you write a name of a file ending with =docx= as
2419 |    the first token in a command line, it will use libreoffice to open
2420 |    it.
2421 | 
2422 |    #+begin_src shell
2423 |      invoice1.docx
2424 |      # will effectively call libreoffice invoice1.docx
2425 |    #+end_src
2426 | 
2427 |    The trick here is that the parser doesn't check that the file is
2428 |    indeed an existing file. It can be any string.
2429 | 
2430 |    Let's look at an example of it.
2431 | 
2432 |    #+begin_src shell
2433 |      alias -s git="git clone"
2434 |    #+end_src
2435 | 
2436 |    In this case, we can easily copy a =git@github.com:.....git= from a
2437 |    browser, and paste it into a zsh console. Then, zsh will run that
2438 |    "file" with the command =git clone=, effectively cloning that
2439 |    repository.
2440 | 
2441 |    Cool, ain't it?
2442 | 
2443 | ** noglob
2444 |    zsh has more aggressive parameter expansion, to the level that
2445 |    =[,],...= have special meanings, and will be interpreted and
2446 |    expanded before calling the final commands in your shell.
2447 | 
2448 |    There are commands that you don't want ever expanded , for example,
2449 |    when using =curl=, it's much more likely that an open bracket will
2450 |    be ment to be there verbatim rather than expanded.
2451 | 
2452 |    Zsh provides a command to quote the following expansions. And it's
2453 |    called noglob.
2454 |    #+begin_src bash
2455 |      noglob curl http://example.com\&a[]=1
2456 |    #+end_src
2457 | 
2458 | ** make noglob 'transparent' to bash
2459 |    zsh and bash are mostly compatible, but there's a few things not
2460 |    supported in bash. =noglob= is one of them. To build a shim
2461 |    inbetween, an easy way is to just create a =~/bin/noglob= file
2462 |    #+begin_src bash
2463 |      $*
2464 |    #+end_src
2465 | 
2466 | ** glob nested expansion
2467 |    In https://news.ycombinator.com/item?id=26175894 there's a nice
2468 |    advanced example:
2469 | 
2470 |    #+begin_src text
2471 |      Variable expansion syntax, glob qualifiers, and history modifiers can
2472 |      be combined/nested quite nicely. For example, this outputs all the
2473 |      commands available from $PATH: `echo
2474 |      ${~${path/%/\/*(*N:t)}}`. `${~foo}` is to enable glob expansion on the
2475 |      result of foo. `${foo/%/bar}` substitutes the end of the result of foo
2476 |      to "bar" (i.e. it appends it); when foo is an array, it does it for
2477 |      each element. In `/*(*N:t)`, we're adding the slash and star to the
2478 |      paths from `$path`, then the parentheses are glob qualifiers. `*`
2479 |      inside means only match the executables, `N` is to activate NULL_GLOB
2480 |      for the match so that we don't get errors for globs that didn't match
2481 |      anything, `:t` is a history mod used for globs that returns just the
2482 |      "tail" of the result, i.e. the basename. IIRC, bash can't even nest
2483 |      multiple parameter expansions; you need to save each step separately.
2484 |    #+end_src
2485 | ** Some extra shortcuts for nice things
2486 |    - =alt-'= quotes the current line. It's like =quotemeta=. great to
2487 |      help you fight double and triple quoting when writing scripts.
2488 |    - =alt-#= comment/uncomment and execute. Nice way to store the current line
2489 |      for later and reach out to it again to run it for real.
2490 |    - =ctrl-o= kill-current-line, wait for a command, and paste.
2491 | 
2492 | ** =()
2493 |    Zsh has =<()= and =>()= like Bash, but it also has ==()=. This
2494 |    varant is similar to =<()= but instead of creating a temporary
2495 |    pipe, it creates a temporary file. That is useful if we want to run
2496 |    commands that require a file instead of a pipe (most times, because
2497 |    it uses lseek to go through it).
2498 | 
2499 |    Node is an example of this.
2500 |    #+begin_src shell
2501 |     node <(echo 'setTimeout(() => console.log("foo"), 400)')   # fails
2502 |     node =(echo 'setTimeout(() => console.log("foo"), 400)')   # works!
2503 |   #+end_src
2504 | 
2505 |   Or,
2506 | 
2507 |   #+begin_src bash
2508 |     docker run --rm -ti -v =(echo "OHAI"):/tmp/foo ubuntu cat /tmp/foo
2509 |   #+end_src
2510 | 
2511 | * TODO patterns
2512 | ** just use cat/netcat/pipes with <()
2513 | 
2514 | - input
2515 | 
2516 |   =python logger.py executable= will run the executable and monitor it
2517 |   for error messages. Depending on the error messages it will be doing.
2518 | 
2519 |   In order to test it, I want to run it with my own output. So what I
2520 |   do is =python logger.py cat=. That way I can type my stuff there,
2521 |   and even better, I can use a stream from the shell. =myexecutable |
2522 |   python logger.py cat= still works.
2523 | 
2524 | *** what's the unifying theory behind all that?
2525 |     It's still not clear to me how they relate, but the feeling is
2526 |     that there's a common thread ruling all those commands. as if they
2527 |     generalize over the same things, or just a couple of very
2528 |     interrelated things.
2529 | 
2530 |     =echo= is to =cat= what =|= is to =xargs=. and =<()= and =>()= are
2531 |     able to make static files be dynamic streams. putting =cat= and
2532 |     =echo= inside =<()= seem like either a noop, or a leap in what can
2533 |     be done there. Still have to figure it out.
2534 | 
2535 |    <(grep a file.txt) , | xargs , cat, echo
2536 |   | you-have\it-wants | executable  | file                 | stream         |
2537 |   | executable        | X           | <(exe)               | exe \vert      |
2538 |   | file              | <(cat file) | X                    | cat file \vert |
2539 |   | stream            | cat         | <(grep foo file.txt) | X              |
2540 | 
2541 | 
2542 | - output
2543 | 
2544 |   Most of those can be tested with and =tee=. Sometimes you would like
2545 |   the output to be an output to a file to be extramassaged.
2546 | 
2547 |    | you-have\it-wants | executable | file       | stream |
2548 |    | executable        | X          | >()        |        |
2549 |    | file              |            | X          |        |
2550 |    | stream            |            | >(cat) | X      |
2551 | 
2552 |    lnav <(tail -F /my/logfile-that-gets-rotated-or-truncated.log)
2553 |    cat <(date)
2554 | 
2555 | ** redirects & streams
2556 |    - https://catonmat.net/ftp/bash-redirections-cheat-sheet.pdf
2557 |    - https://catonmat.net/bash-one-liners-explained-part-three
2558 |    - https://github.com/miguelmota/bash-streams-handbook
2559 |    - https://www2.dmst.aueb.gr/dds/sw/dgsh/
2560 |    - https://wiki.bash-hackers.org/howto/redirection_tutorial
2561 | ** The $0 pattern
2562 |    - https://www.reddit.com/r/oilshell/comments/f6of85/four_more_posts_in_shell_the_good_parts/
2563 | ** use git staging area to diff outputs of commands
2564 |    - https://chrismorgan.info/blog/make-and-git-diff-test-harness/
2565 |    - [[https://news.ycombinator.com/item?id=32896051][Content based change detection with Make]]
2566 |    - https://news.ycombinator.com/item?id=40333481
2567 |    - https://news.ycombinator.com/item?id=32441602
2568 | 
2569 | ** coprocs
2570 |    - https://stackoverflow.com/questions/7942632/how-to-extrace-pg-backend-pid-from-postgresql-in-shell-script-and-pass-it-to-ano/8305578#8305578
2571 |    - https://unix.stackexchange.com/questions/86270/how-do-you-use-the-command-coproc-in-various-shells
2572 |    - https://mbuki-mvuki.org/posts/2021-05-30-memoize-commands-or-bash-functions-with-coprocs/
2573 | * links
2574 |   - https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html
2575 |   - https://www.gnu.org/software/bash/manual/html_node/
2576 |   - https://tldp.org/LDP/abs/html/
2577 |   - https://mywiki.wooledge.org/BashPitfalls
2578 |   - [[https://www.youtube.com/watch?v=sCZJblyT_XM][Gary Bernhardt. The Unix Chainsaw]]
2579 |   - https://github.com/spencertipping/. This guy has some bash sick snippets
2580 |   - https://news.ycombinator.com/item?id=23765123
2581 |   - https://medium.com/@joydeepubuntu/functional-programming-in-bash-145b6db336b7
2582 |   - https://www.youtube.com/watch?v=yD2ekOEP9sU
2583 |   - http://catern.com/posts/pipes.html
2584 |   - https://ebzzry.io/en/zsh-tips-1/
2585 |   - https://github.com/ssledz/bash-fun
2586 |   - https://news.ycombinator.com/item?id=24556022
2587 |   - https://www.datafix.com.au/BASHing/index.html
2588 |   - https://susam.github.io/tucl/the-unix-command-language.html
2589 |   - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
2590 |   - https://www.grymoire.com/Unix/Sh.html
2591 |   - https://github.com/dylanaraps/pure-sh-bible
2592 |   - https://github.com/dylanaraps/pure-bash-bible
2593 |   - https://shatterealm.netlify.app/programming/2021_01_02_shiv_lets_build_a_vcs
2594 |   - https://news.ycombinator.com/item?id=24401085
2595 |   - https://git.sr.ht/~sircmpwn/shit
2596 |   - [[https://github.com/p8952/bocker][bocker]]. Docker implemented in around 100 lines of bash.
2597 |   - https://github.com/simplenetes-io/simplenetes wow
2598 |   - https://bakkenbaeck.github.io/a-random-walk-through-git/
2599 |   - https://github.com/WeilerWebServices/Bash
2600 |   - https://www.netmeister.org/blog/consistent-tools.html.
2601 |   - https://www.arp242.net/why-zsh.html. bash vs zsh differences.
2602 |   - https://dwheeler.com/essays/filenames-in-shell.html
2603 |   - https://lacker.io/math/2022/02/24/godels-incompleteness-in-bash.html
2604 |   - https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp-part-1-doug-mcilroys-pipeline/ (from [[https://clojurians.slack.com/archives/C03RZGPG3/p1645563940203839][clojurians slack]])
2605 |   - https://github.com/adityaathalye
2606 |   - https://www.youtube.com/watch?v=BJ0uHhBkzOQ
2607 |   - https://www.gnu.org/software/autoconf/manual/autoconf-2.60/html_node/Portable-Shell.html
2608 | 
2609 | * From shell to lisp and everything in between
2610 |   - [[https://github.com/oilshell/oil][Oil Shell]].
2611 |   - [[https://rash-lang.org/][Rash]] (Racket shell)
2612 |   - [[https://arxiv.org/pdf/2007.09436.pdf][PaSh]]: Light-touch Data-Parallel Shell Processing.
2613 |   - [[https://www.eigenbahn.com/2020/07/08/painless-emacs-remote-shells][Painless emacs remote shells]]. Because emacs has you covered
2614 |   - https://news.ycombinator.com/item?id=24249646 rust
2615 |   - https://github.com/liljencrantz/crush
2616 |   - https://github.com/artyom-poptsov/metabash
2617 |   - https://www.nushell.sh/
2618 |   - [[https://github.com/borkdude/babashka][Babashka]]
2619 |   - Bash to Perl/Python/Ruby using =``= and growing from there.
2620 | * Credits
2621 | - Raimon Grau <[[mailto:raimonster@gmail.com][raimonster@gmail.com]]>.
2622 | - Some examples are result of Raimon's and Lluís Esquerda's
2623 |   conversations or real world examples.
2624 | - people in https://news.ycombinator.com/item?id=24402571 which I'll
2625 |   be pulling in as time allows.
2626 | 


--------------------------------------------------------------------------------
/readme.org:
--------------------------------------------------------------------------------
1 | * https://kidd.github.io/scripting-field-guide/index.html
2 | 


--------------------------------------------------------------------------------