├── .github
└── workflows
│ └── deploy.yml
├── book.org
├── readme.org
└── styles.css
/.github/workflows/deploy.yml:
--------------------------------------------------------------------------------
1 | # This is a basic workflow to help you get started with Actions
2 |
3 | name: Publish
4 |
5 | # Controls when the action will run. Triggers the workflow on push or pull request
6 | # events but only for the first-skeleton branch
7 | on:
8 | push:
9 | branches: [ master ]
10 |
11 | # A workflow run is made up of one or more jobs that can run sequentially or in parallel
12 | jobs:
13 | # This workflow contains a single job called "build"
14 | build:
15 | # The type of runner that the job will run on
16 | runs-on: ubuntu-latest
17 |
18 | # Steps represent a sequence of tasks that will be executed as part of the job
19 | steps:
20 | # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
21 | - uses: actions/checkout@v2
22 |
23 | # Runs a single command using the runners shell
24 | - name: Run a one-line script
25 | run: echo Hello, world!
26 |
27 | - name: Set up Emacs
28 | uses: purcell/setup-emacs@master
29 | with:
30 | version: 26.1
31 |
32 | - name: get htmlize
33 | run: wget https://raw.githubusercontent.com/hniksic/emacs-htmlize/master/htmlize.el
34 |
35 | - name: export
36 | run: emacs book.org -Q -batch -l htmlize.el -f org-html-export-to-html
37 | # Runs a set of commands using the runners shell
38 | - name: rename
39 | run: mv book.html index.html
40 |
41 | - name: Deploy to GitHub Pages
42 | uses: peaceiris/actions-gh-pages@v3
43 | with:
44 | github_token: ${{ secrets.GITHUB_TOKEN }}
45 | publish_dir: ./
46 | allow_empty_commit: true
47 | # enable_jekyll: true
48 | # cname: github.peaceiris.com
49 |
50 | # - name: Deploy to GitHub Pages
51 | # if: success()
52 | # uses: crazy-max/ghaction-github-pages@v2
53 | # with:
54 | # target_branch: gh-pages
55 | # build_dir: ./
56 | # env:
57 | # GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
58 | # - name: Bye
59 | # run: |
60 | # echo DONE!
61 |
--------------------------------------------------------------------------------
/book.org:
--------------------------------------------------------------------------------
1 | #+TITLE: Shell Field Guide
2 | #+AUTHOR: Raimon Grau
3 | #+OPTIONS: ':nil *:t -:t ::t <:t H:3 \n:nil ^:nil arch:headline
4 | #+OPTIONS: author:t c:nil creator:comment d:(not "LOGBOOK") date:nil
5 | #+OPTIONS: e:t email:nil f:t inline:t p:nil pri:nil stat:t
6 | #+OPTIONS: tags:t tasks:t tex:t timestamp:t todo:t |:t
7 | #+EXCLUDE_TAGS: noexport
8 | #+KEYWORDS: bash zsh shell
9 | #+LANGUAGE: en
10 | #+SELECT_TAGS: export
11 |
12 | #+OPTIONS: html-style:nil
13 | #+SETUPFILE: https://fniessen.github.io/org-html-themes/org/theme-readtheorg.setup
14 | #+HTML_HEAD_EXTRA:
15 |
16 | #+OPTIONS: reveal_center:nil timestamp:nil
17 | #+REVEAL_THEME: black
18 |
19 | # toc:nil num:nil
20 |
21 | * Introduction
22 | This booklet is intended to be a catalog of tricks and techniques
23 | you may want to use if you're doing some sort of complex
24 | scripting. Some are useful in "The Real World (TM)", some are more
25 | playful, and might not have such direct impact in your day-to-day
26 | life. Some are pure entertainment. You'll have to judge by yourself
27 | which belong to which. I'll try to keep the rhetoric to the minimum
28 | to maximize signal/noise.
29 |
30 | The git repo is at
31 | https://github.com/kidd/scripting-field-guide/. Any feedback is
32 | greatly appreciated. Keep in mind this is not any kind of official
33 | doc. I just write MY current "state of the art" and I'll be updating
34 | the contents with useful stuff I find or discover, that are not
35 | widely explained in usual manuals/wikis.
36 |
37 | You surely have some amount of sh/bash/zsh in your stack that
38 | probably started as one-off scripts, and later grew and were
39 | copypasted everywhere in your pipelines, repos, coworkers' ~/bin, or
40 | your coworkers use for their own things (with some variations),
41 | etc. Those scripts are very difficult to kill and they have a very
42 | high mutation rate.
43 |
44 | * Which Shell?
45 | No matter if you use Linux, Mac, or Windows, you should be living
46 | most of the time in a shell to enjoy the content shown here. Some
47 | value comes from running scripts, and some comes from the daily
48 | usage and refinement of your helper functions, aliases, etc. in
49 | interactive mode.
50 |
51 | In general the examples here are meant to run in Bash or Zsh, which
52 | are compatible for the most part.
53 | * Level
54 | These examples are based on non-trivial real world code I've written
55 | using patterns I haven't seen applied in many places over the net. A
56 | few of the snippets are stolen from public repos I find interesting.
57 | Also, important scripting stuff might be missing if I don't feel
58 | I have anything to add to the generally available info around.
59 | * Patterns
60 | ** Use Shellcheck
61 | First, let's get that out of the way. This is low-hanging
62 | fruit. And you will get the most of this booklet by following it.
63 |
64 | A lot of the most common errors we usually make are well known
65 | ones. And in fact, we all usually fail in similar ways. Bash is
66 | known for being error prone when dealing with testing variable
67 | values, string operations, or flaky subshells and pipes.
68 |
69 | Installing [[https://www.shellcheck.net/][shellcheck]] will flag many of those ticking bombs for
70 | you.
71 |
72 | No matter which editor you are using, you should be able to
73 | install a plugin to do automatic checks while you're editing.
74 |
75 | In emacs' case, the package is called [[https://github.com/federicotdn/flymake-shellcheck][flymake-shellcheck]], and a
76 | quick configuration for it is:
77 |
78 | #+begin_src elisp
79 | (use-package flymake-shellcheck
80 | :ensure t
81 | :commands flymake-shellcheck-load
82 | :init
83 | (add-hook 'sh-mode-hook 'flymake-shellcheck-load))
84 | #+end_src
85 |
86 | Shellcheck is available on most distros, so it's just an =apt=,
87 | =brew=, or =nix-env= away.
88 |
89 | ** Booleans and Conditionals
90 | In any shell, =foo && bar= will execute =bar= only if =foo=
91 | succeeded. That means that =foo= returned 0. That means that to &&
92 | (which you read like "and"), 0 is true. so yes. 0 is true, and
93 | other values are false.
94 |
95 | ** Arrays
96 | Ordered list of things.
97 | #+begin_src bash
98 | foo=("ls" "/tmp/")
99 |
100 | echo ${foo[-2]}
101 | echo ${foo[-1]}
102 | echo ${foo[0]}
103 | echo ${foo[1]}
104 | echo ${foo[2]}
105 |
106 | for i in "${foo[@]}"; do
107 | echo $i
108 | done
109 |
110 | $foo
111 | ${foo[*]}
112 | ${foo[@]}
113 | echo ${#foo[*]}
114 | echo ${#foo[@]}
115 | #+end_src
116 |
117 | Are =*= and =@= equal? [[https://stackoverflow.com/questions/2761723/what-is-the-difference-between-and-in-shell-scripts][nope]].
118 |
119 | #+begin_src bash
120 | "${foo[@]}"
121 | "${foo[*]}"
122 | #+end_src
123 |
124 | #+begin_src bash
125 | #!/bin/bash
126 |
127 | main()
128 | {
129 | echo 'MAIN sees ' $# ' args'
130 | }
131 |
132 | main $*
133 | main $@
134 |
135 | main "$*"
136 | main "$@"
137 |
138 | ### end ###
139 | #+end_src
140 |
141 | and I run it like this:
142 |
143 | =my_script 'a b c' d e=
144 |
145 | ** Pass Arrays around
146 | #+begin_src bash
147 | a=('Track 01.mp3' 'Track 02.mp3')
148 | myfun "${a[@]}" # pass array to a function
149 | b=( "${a[@]}" ) # copy array
150 | #+end_src
151 |
152 | Read the great [[http://www.oilshell.org/blog/2016/11/06.html][Oil Shell blogpost]].
153 | ** Slurping arrays
154 | A nice way to read a bunch of elements in one go is to use
155 | =readarray=.
156 | #+begin_src bash
157 | parse_args() {
158 | [[ $# -eq 0 ]] && die "Usage: $0 "
159 | version="$1"
160 | local version_split=$(echo $version | tr '.' '\n')
161 | readarray -t version_array <<< "$version_split"
162 |
163 | if [[ -z ${version_array[3]} ]]; then
164 | die "not enough version numbers"
165 | fi
166 | }
167 | #+end_src
168 |
169 | Even nicer would be to use IFS so we'd be able to split in one go.
170 |
171 | #+begin_src bash
172 | IFS=. read -a ver <<<"1.23.1.0"
173 | echo ${ver[0]}
174 | echo "next/${ver[0]}.${ver[1]}.x.x"
175 | #+end_src
176 |
177 | Or, use it in a destructuring fashion:
178 | #+begin_src bash
179 | get_nix_version_parts(){
180 | local major minor patch
181 | # shellcheck disable=SC2034,SC2162
182 | IFS="." read major minor patch < <(get_nix_version)
183 | local -p
184 | }
185 |
186 | $ get_nix_version_parts
187 | major=2
188 | minor=3
189 | patch=4
190 | #+end_src
191 | https://news.ycombinator.com/item?id=24408318
192 | ** Array slices
193 |
194 | you can slice an array like this:
195 |
196 | #+begin_src bash
197 | a=(a b c d e f g)
198 | echo "${a[@]:1}" # tail of array
199 | echo "${a[@]:1:2}" # from that, take 2 elements
200 | #+end_src
201 |
202 | This is a [[https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html][parameter expansion]], same as =${a/foo/bar}= and similar
203 | replacements. As such, The =$@= and =$*= are treated a bit
204 | differently.
205 |
206 | Look at =${parameter:offset:length}= in the parameter expansion
207 | bash manual and be amazed by the specialcasing :).
208 |
209 | This concrete case is when the parameter is an indexed array name
210 | subscripted by '@' or '*'.
211 |
212 | Don't ask me why, but when the array is =$@=, it seems that it
213 | behaves a bit differently. It's 'explained' in that same part of
214 | the docs.
215 |
216 | #+begin_src bash
217 | a=(a b c d e f g)
218 | foo() {
219 | echo "${@:2}" # tail of array
220 | echo "${@:2:2}" # from that, take 2 elements
221 | }
222 | foo a b c d
223 | foo "${a[@]}"
224 | #+end_src
225 |
226 | ** Read lines from file
227 | The =read= command we used just above is part of the usual idiom to
228 | read a file line by line.
229 |
230 | #+begin_src bash
231 | while read -r line; do
232 | echo $line
233 | done < /tmp/file.txt
234 | #+end_src
235 |
236 | More related info in the [[https://mywiki.wooledge.org/BashFAQ/001][BashFAQ001]]. But it's very rare the case
237 | where I need to iterate a file line by line.
238 |
239 | ** Assign to $1,$2,$3...
240 | =set --= can be used as an incantation to assign to the positional
241 | parameters. Let me show you.
242 |
243 | #+begin_src bash
244 | set -- a b c
245 | echo $1 $2 $3
246 | echo $@
247 | #+end_src
248 |
249 | See? here's how to "unshift" a parameter to the current arg list:
250 |
251 | #+begin_src bash
252 | set -- "injected" "$@"
253 | #+end_src
254 |
255 | ** Functions
256 | Functions are functions. They receive arguments, and they return a
257 | value.
258 |
259 | The special thing about shell functions is that they also can use
260 | the file descriptors of the process. That means that they "inherit"
261 | STDIN, STDOUT, STDERR (and maybe more).
262 |
263 | Use them.
264 |
265 | Another point is that function names can be passed as parameters,
266 | because they are passed as strings, but you can call them inside as
267 | functions again. Maybe they are better than you [[https://news.ycombinator.com/item?id=29058140][think.]]
268 |
269 | #+begin_src bash
270 | f() {
271 | $1 hi
272 | }
273 |
274 | f echo
275 | f touch # will create a file 'hi'
276 | #+end_src
277 |
278 | #+RESULTS:
279 | : hi
280 |
281 | ** Variables
282 | By default variables are global, to a file. No matter if you assign
283 | them for the first time inside a function, or at the top level.
284 | #+begin_src bash
285 | foo=3
286 | bar=$foo
287 | f() {
288 | echo $bar
289 | }
290 | f
291 | #+end_src
292 |
293 | #+RESULTS:
294 | : 3
295 |
296 | #+begin_src bash
297 | f() {
298 | bar=1
299 | }
300 | f
301 | echo $bar
302 | #+end_src
303 |
304 | #+RESULTS:
305 | : 1
306 |
307 | You make a variable local to a function with =local=. Use it as
308 | much as you can (kinda obvious).
309 | #+begin_src bash
310 | myfun() {
311 | local bar
312 | bar=3
313 | echo $bar
314 | }
315 |
316 | bar=4
317 | echo $bar
318 | myfun
319 | echo $bar
320 | #+end_src
321 |
322 | #+RESULTS:
323 | | 4 |
324 | | 3 |
325 | | 4 |
326 | ** Variable/Parameter Expansions
327 | They offer some variable manipulations using shell only, not having
328 | to create another process =sed,awk,perl=.
329 |
330 | #+begin_src bash
331 | v=banana
332 | # substitute one
333 | echo ${v/na/NA} # baNAna
334 |
335 | # substitute many
336 | echo ${v//na/NA} # baNANA
337 |
338 | # substitute from the start (think ^ in PCRE)
339 | echo ${v/#ba/NA} # NAnana
340 |
341 | # substitute from the end
342 | echo ${v/%na/NA} # banaNA
343 |
344 | # Capitalize
345 | echo ${v^} # Banana
346 |
347 | # Uppercase
348 | echo ${v^^} # BANANA
349 | #+end_src
350 |
351 | Take a read on https://tldp.org/LDP/abs/html/manipulatingvars.html
352 | and
353 | https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
354 | for more details.
355 |
356 | And a nice non-obvious trick from here is to prefix or suffix a
357 | variable string:
358 |
359 | #+begin_src bash
360 | v=banana
361 | echo ${v/%/na} # bananana
362 | echo ${v/#/na} # nabanana
363 | #+end_src
364 |
365 | And a less obvious trick is to prefix every element of an array
366 | with a fixed string:
367 | #+begin_src bash
368 | local arr=(var1=1 var2=2)
369 | echo ${arr[*]/#/"--env "}
370 | #+end_src
371 |
372 | This will produce =--env var1=1 --env var2=2=. Super useful to be
373 | combined when building flags for docker.
374 |
375 | Another nice trick is a portable =realpath $PWD= with =${0%/*}=,
376 | which I found in [[https://github.com/picolisp/pil21/blob/master/vip][picolisp's vip]]:
377 |
378 | #+begin_src bash
379 | #!/bin/sh
380 | exec ${0%/*}/bin/picolisp ${0%/*}/lib.l @bin/vip "$@"
381 | #+end_src
382 |
383 | =${0%/*}/bin/picolisp= will replace keep the path of the current
384 | script and append =/bin/picolisp=. A cool way to get a realpath.
385 |
386 | ** Conditional Expansion
387 |
388 | #+begin_src bash
389 | local foo=${var:-${default:-}}
390 | local foo=${var:+&key=var}
391 | #+end_src
392 |
393 | ** Interpolation
394 | We previously saw that functions can be passed around as strings,
395 | and be called later on.
396 |
397 | Something that might not be obvious is that the string can be
398 | created from shorter strings, and that allows for an extra
399 | flexibility, that comes with its own dangers, but it's a very
400 | useful pattern to dispatch functions based on user input or
401 | function outputs.
402 | #+begin_src bash
403 | l=l
404 | s=s
405 | $l$s .
406 | #+end_src
407 |
408 | #+RESULTS:
409 | | book.html |
410 | | book.html~ |
411 | | book.org |
412 | | readme.org |
413 |
414 | ** dispatch functions using args
415 | A nice usage of the previous technique is using user input as a
416 | dispatching method.
417 |
418 | You've probably seen this pattern already:
419 |
420 | #+begin_src bash
421 | while [[ $# -gt 0 ]]; do
422 |
423 | case $1 in
424 | foo)
425 | foo
426 | ;;
427 | *)
428 | exit 1
429 | ;;
430 | esac
431 | shift
432 | done
433 | #+end_src
434 |
435 | And it is useful for its own good, and flexible.
436 |
437 | But for some simpler cases, we can dispatch based on the variable
438 | itself:
439 |
440 | #+begin_src bash
441 | cmd_foo() {
442 | do-something
443 | }
444 |
445 | cmd_$1
446 | #+end_src
447 |
448 | The problem with this is that in case we supply a =$1= that doesn't
449 | map to any =cmd_$1= we'll get something like
450 |
451 | #+begin_src bash
452 | bash: cmd_notexisting: command not found
453 | #+end_src
454 |
455 | ** command_not_found_handle
456 | Here's a detail on a kinda obscure bash (only bash) feature.
457 |
458 | You can set a hook that will be called when bash tries to run a
459 | command and it doesn't find it.
460 |
461 | #+begin_src bash
462 | command_not_found_handle() {
463 | echo "$1 is not a correct command. Cmds allowed:"
464 | echo "$(typeset -F | grep cmd_ | sed -e 's/.*cmd_/cmd_/')"
465 | }
466 |
467 | cmd_foo() {
468 | echo "foo"
469 | }
470 |
471 | cmd_baz() {
472 | echo "baz"
473 | }
474 | cmd_bar
475 | #+end_src
476 |
477 | You can unset the function =command_not_found_handle= to go back to
478 | the normal behavior.
479 |
480 | ** Return Values for Conditionals
481 | =if= 's test condition can use the return values of
482 | commands. That's a known thing, but a lot of code you see around
483 | relies on =[[]]= to test the return values of commands/functions
484 | anyway.
485 |
486 | #+begin_src bash
487 | if echo "foo" | grep "bar" ; then
488 | echo "found!"
489 | fi
490 | #+end_src
491 |
492 | This is much clearer than
493 | #+begin_src bash
494 | if [[ ! -z $( echo "foo" | grep "bar") ]]; then
495 | echo "found!"
496 | fi
497 | #+end_src
498 |
499 | As easy and trivial as it seems, this way of thinking pushes you
500 | forward to thinking about creating smaller functions that check the
501 | conditions and =return= 0 or non 0. It's syntactically smaller, and
502 | usually makes you play by the rules of the commands, more than just
503 | finding your way around the output strings.
504 |
505 | #+begin_src bash
506 | if less_than $package "1.3.2"; then
507 | die "can't proceed"
508 | fi
509 | #+end_src
510 |
511 | ** set variable in an "if" test
512 | Usual pattern to capture the output of a command and branch
513 | depending on its return value is:
514 | #+begin_src bash
515 | res="$(... whatever ...)"
516 | if [ "$?" -eq 0 ]; then ...
517 | ...
518 | fi
519 | #+end_src
520 |
521 | Well, you can test the return value AND capture the output at the
522 | same time!
523 |
524 | #+begin_src bash
525 | if res="$(...)"; then
526 | ...
527 | fi
528 | #+end_src
529 |
530 | Unfortunately, it doesnt' work with =local=, so you can't be
531 | defining a local var in the same line. So, the variable is either
532 | global, or you spent a line to declare it local before. Still, I
533 | think I prefer to have a line to declare the variable as local
534 | rather than having explicit =$?='s around.
535 | #+begin_src bash
536 | local var1
537 | if var1=$(f); then
538 | echo "$var1"
539 | fi
540 | #+end_src
541 | Ref: https://news.ycombinator.com/item?id=27163494
542 |
543 | ** Do work on loop conditions
544 | I've not seen it used a lot (and there might be a reason for it,
545 | who knows), =while= conditions are just plain commands, so you can
546 | put other stuff than =[]/[[]]/test= there.
547 |
548 | Heres's an idiomatic way to iterate through all the arguments of a
549 | function while consuming the =$*= array.
550 |
551 | #+begin_src bash
552 | while(($#)) ; do
553 | #...
554 | shift
555 | done
556 | #+end_src
557 |
558 | And here's a pseudo-repl that keeps shooting one-off commands. This
559 | will keep shooting =tr= commands to whatever strings you give it,
560 | with the usual rlwrap goodies.
561 |
562 | #+begin_src bash
563 | while rlwrap -o -S'>> ' tr a-z A-Z ; do :; done
564 | #+end_src
565 |
566 | Note: =:= is a nop builtin in bash.
567 | ** One Branch Conditionals
568 | The usual conditionals one sees everywhere look like =if=.
569 | #+begin_src bash
570 | if [[ some-condition ]]; then
571 | echo "yes"
572 | fi
573 | #+end_src
574 |
575 | This is all good and fine, but in the same vein of using the least
576 | powerful construct for each task, it's nice to think of the one way
577 | conditionals in the form of =&&= and =||= as a way to explicitly
578 | say that we don't want to do anything else when the condition is
579 | not met. It's a hint to the reader.
580 |
581 | #+begin_src bash
582 | some-condition || {
583 | echo "log: warning!"
584 | }
585 |
586 | other-condition && {
587 | echo "log: all cool"
588 | }
589 | #+end_src
590 |
591 | This conveys the intention of doing something *just* in one case,
592 | and that the negation of this is not interesting at all. There's a
593 | big warning you have to be aware of. The same as with lua's
594 | =... and .. or ..=, bash =||= and =&&= are not interchangeable for
595 | =if...else...end=. [[https://mywiki.wooledge.org/BashPitfalls#cmd1_.26.26_cmd2_.7C.7C_cmd3][BashWiki]] has an explanation why, but, the same
596 | as in Lua's case, if the "then" part returns false, the else will
597 | run.
598 |
599 | There are lots of references to this, but I like this recent post
600 | where it explains it for arrays in higher level languages like ruby:
601 | https://jesseduffield.com/array-functions-and-the-rule-of-least-power/
602 |
603 | An extended article of this kind of conditionals can be found [[https://timvisee.com/blog/elegant-bash-conditionals/][here]].
604 | ** pushd/popd
605 | pushd and popd are used to move to some directory and go back to it
606 | in a stack fashion, so nesting can happen and you never lose
607 | track. The problem is that it still is on you to have a =popd= per
608 | =pushd=.
609 | #+begin_src bash
610 | pushd /tmp/my-dir
611 | echo $PWD
612 | popd
613 | #+end_src
614 |
615 | Here's an alternative way, that at least makes sure that you close
616 | all pushd with a popd.
617 |
618 | Starting a new shell and cd-ing , will make all commands in that
619 | subshell be in that directory, and will come back to the old
620 | directory after closing the new spawned shell.
621 |
622 | #+begin_src bash
623 | (cd /tmp/my-dir
624 | ls
625 | )
626 | #+end_src
627 |
628 | Remember to =inherit_errexit= or =set -e= inside the subshell if
629 | you need. That's a very easy trap to fall into.
630 |
631 | ** wrap functions
632 | Bash can't pass blocks of code around, but the alternative is to
633 | pass functions. More on that later.
634 |
635 | #+begin_src bash
636 | mute() {
637 | "$@" >/dev/null
638 | }
639 |
640 | mute pushd /tmp/foobar
641 | #+end_src
642 | ** use [[
643 | Unless you want your script to be POSIX compliant, use =[[= instead
644 | of =[=. =[= is a regular command. It's like =ls=, or =true=. You can
645 | check it by searching for a file named =[= in your path.
646 |
647 | Being a normal command it always evaluates its params, like a
648 | regular function. On the other hand though, =[[= is a special bash
649 | operator, and it evaluates the parameters lazily.
650 |
651 | #+begin_src bash
652 |
653 | # [[ does lazy evaluation:
654 | [[ a = b && $(echo foo >&2) ]]
655 |
656 | # [ does not:
657 | [ a = b -a "$(echo foo >&2)” ]
658 | #+end_src
659 | Ref: https://lists.gnu.org/archive/html/help-bash/2014-06/msg00013.html
660 |
661 | ** eval?
662 | When you have mostly small functions that are mostly pure, you
663 | compose them like you'd do in any other language.
664 |
665 | In the following snippet, we are in a release script. Some step
666 | builds a package inside a docker container, another step tests a
667 | package already built.
668 |
669 | A nice way to build ubuntus, for example, is to add an ARG to the
670 | Dockerfile so we can build several ubuntu versions using the same
671 | file.
672 |
673 | It'd look like this:
674 | #+begin_src Dockerfile
675 | ARG VERSION
676 | FROM ubuntu:$VERSION
677 |
678 | RUN apt-get ...
679 | ...
680 | #+end_src
681 |
682 | We build that image and do all the building inside it, mounting a
683 | volume shared with our host, so we can extract our =.deb= file
684 | easily.
685 |
686 | After that, to do some smoke tests on the package, the idea is to
687 | install the =.deb= file in a fresh ubuntu image.
688 |
689 | Let's pick the same base image we picked to build the package.
690 | #+begin_src bash
691 | # evaluate the string "centos:$VERSION" (that comes from
692 | # centos/Dockerfile) in the current scope
693 | # DISTRO is ubuntu:18.04
694 | local VERSION=$(get_version $DISTRO) # VERSION==18.04
695 | run_test "file.deb" "$(eval echo $(awk '/^FROM /{print $2; exit}' $LOCAL_PATH/$(get_dockerfile_for $DISTRO)))" # ubuntu:18.04
696 | #+end_src
697 |
698 | The usage of eval is there to interpolate the string that we get
699 | from the =FROM= in the current environment.
700 |
701 | WARNING: You know, anything that uses =eval= is dangerous per
702 | se. Do not use it unless you know very well what you're doing AND
703 | the input is 100% under your control. Usually, more restricted
704 | commands can achieve what you want to do. In this particular case,
705 | you could use =envsubst=, or just manually replace =$\{?VERSION\}?=
706 | in a sed.
707 |
708 | #+begin_src bash
709 | test_release "$PACKAGE_PATH" $(awk '/^FROM /{print $2; exit}' $LOCAL_PATH/$(get_dockerfile_for $DISTRO) | sed -e "s/\$VERSION/$VERSION/")
710 | #+end_src
711 |
712 | Yet another way is using [[https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html][shell parameter expansions.]]
713 | #+begin_src bash
714 |
715 | var1=value
716 | echo 'this is $var1' >/tmp/f.txt
717 | f=$(cat /tmp/f.txt)
718 | echo "${f}" # this is $var1
719 | echo "${f@P}" # this is value
720 | #+end_src
721 |
722 | ** pass commands around
723 | This one uses [[*DRY_RUN][DRY_RUN]]. While refactoring a script that does some
724 | curls, we want to make sure that our refactored version does the
725 | exact same calls in the same order.
726 |
727 | #+begin_src bash
728 | compare_outputs() {
729 | export DRY_RUN=1
730 | git checkout b1
731 | $@ 2>/tmp/1.out
732 | git checkout b2
733 | $@ 2>/tmp/2.out
734 | echo "diffing"
735 | diff /tmp/1.out /tmp/2.out
736 | }
737 | compare_outputs ./release.sh -p rhel:6 -R 'internal-preview'
738 | #+end_src
739 |
740 | First we create a function =compare_outputs=, that gets a command
741 | to run as parameters. The function will run it once, redirecting
742 | the standard error to a file =/tmp/1.out=.
743 |
744 | Then, it checks out the branch that contains our refactored
745 | version, and will run the command again, redirecting standard error
746 | to =/tmp/2.out=, and will diff the two outputs.
747 |
748 | In case there's a difference between the two, =diff= will output
749 | them, and the function will return the non-zero exit value of
750 | diff. If everything went fine, =compare_outputs= will succeed.
751 |
752 | Now that we know that for these inputs the command runs fine, we
753 | want to find out if it works for other types of releases, not only
754 | internal-preview.
755 |
756 | Here I'm using zsh's global aliases to give a much more fluid
757 | interface to the commands, but you can use the regular while/for
758 | loops:
759 |
760 | #+begin_src shell
761 | alias -g SPLIT='| tr " " "\n" '
762 | alias -g FORI='| while read i ; do '
763 | alias -g IROF='; done '
764 |
765 | set -e
766 | echo "ga internal-preview rc1 rc2" SPLIT FORI
767 | noglob compare_outputs ./release.sh -p rhel:8 -R "$i"
768 | IROF
769 | #+end_src
770 |
771 | So, combining the two, we can have a really smooth way of iterating
772 | over the possibilities, without really messing into the details of
773 | loops.
774 |
775 | WARNING: This approach is not robust enough to put it anywhere in
776 | production, but to write quick one off scripts is a
777 | killer. Experimenting in a shell and creating tools and 2nd order
778 | tools to make interaction faster builds a language that grows on
779 | you, and keeps improving your toolbelt.
780 |
781 | ** The Toplevel Is Hopeless
782 | Shellscripts are thought as quick one-off programs, but when they
783 | are useful, they are sticky, so you better write them from the
784 | start as if it would be permanent. The upfront cost is very low
785 | anyway. Structure the script like a regular app.
786 |
787 | Bash is extremely permissive in what it allows to be coded and
788 | ran. By default, failures do not make the program exit or throw an
789 | exception (no exceptions). And for some reason, the common
790 | usage of shellscripts is to put everything in the top level. Don't
791 | do that. Do the least possible things in the toplevel.
792 |
793 | A way to improve the defaults, is setting a bunch of flags that
794 | make the script stricter, so it fails on many situations you'd
795 | want to stop anyway because something went wrong.
796 | #+begin_src bash
797 | #!/usr/bin/env bash
798 | set -eEuo pipefail
799 | shopt -s inherit_errexit
800 |
801 | main() {
802 | parse_args
803 | validate_args
804 | do_things
805 | cleanup
806 | }
807 |
808 | main "$@"
809 | #+end_src
810 |
811 | Ref: https://dougrichardson.us/2018/08/03/fail-fast-bash-scripting.html
812 |
813 | Although... [[http://mywiki.wooledge.org/BashFAQ/105][Why doesn't set -e do what I expected?]]
814 |
815 | ** Check your deps
816 | Giving useful information to the users will help them using the
817 | script, and you debugging it. Script dependencies is a common use
818 | case that we'll do it in a nice way.
819 |
820 | #+begin_src bash
821 | deps() {
822 | for dep in "$@"; do
823 | mute command -v "$dep" || die "$dep dependency missing"
824 | done
825 | }
826 |
827 | main() {
828 | deps jq curl
829 | # ...
830 | }
831 | #+end_src
832 |
833 | ** source files
834 | =source= is like =require= or =import= in some programming
835 | languages. It evaluates the sourced file in the context of the
836 | current script, so you get all definitions in your environment.
837 |
838 | It's simple, but it helps you get used to modularize your code into
839 | libraries.
840 |
841 | Be careful, it's very rudimentary, and it will be overwriting old
842 | vars or functions if names clash. There's no namespacing happening
843 | there.
844 |
845 | #+begin_src bash
846 | source file.sh
847 |
848 | # the same
849 | . file.sh
850 | #+end_src
851 |
852 | ** Use Scripts as a Libs
853 | A python-inspired way of using scripts as loadable libraries is to
854 | check whether the current file was the one that was called
855 | originally or it's being just sourced.
856 |
857 | Again, no side effects in load time makes this functionality
858 | possible. otherwise, you're on your own.
859 | #+begin_src bash
860 | # Allow sourcing of this script
861 | if [[ $(basename "$(realpath "$0")") == "$(basename "${BASH_SOURCE}")" ]]; then
862 | setup
863 | parse_args "$@"
864 | main
865 | fi
866 | #+end_src
867 | ** Tmpfiles Everywhere
868 | Your script is not going to run alone. Don't assume paths are
869 | fixed or known.
870 |
871 | CI/CD Pipelines run many jobs in the same node and files can start
872 | clashing.
873 |
874 | Make use of =$(mktemp -d /tmp/foo-bar.XXXXX)=. If you have to patch a
875 | file, do it in a clean fresh copy. Don't modify files in old paths
876 |
877 | If you HAVE TO modify paths, do it idempotently. But really, don't do it.
878 |
879 | #+begin_src bash :eval no
880 | git_clone_tmp() {
881 | local repo=${1:?repo is required}
882 | local ref=${2:?ref is required}
883 | tmpath=$(mktemp -d "/tmp/cloned-$repo-XXXXX")
884 | on_exit "rm -rf $tmpath"
885 | git clone -b ${ref} $repo $tmpath
886 | }
887 | #+end_src
888 |
889 | CAVEAT: You have to manually delete the directory if you want it cleaned.
890 |
891 | Here's an article with very good advice on [[https://www.netmeister.org/blog/mktemp.html][tempfiles]].
892 | ** Cleanup tasks with trap
893 | =trap= is used to 'subscribe' a callback when something happens.
894 | Many times it's used on exit. It's a good thing to cleanup tmpdirs after your script
895 | exits, so you can use the output of =mktemp -d= and subscribe a cleanup
896 | function for it.
897 |
898 | #+begin_src bash
899 | on_exit() {
900 | rm -rf $1
901 | }
902 | local tmpath=$(mktemp -d /tmp/foo-bar.XXXXX)
903 | trap "on_exit $tmpath" EXIT SIGINT
904 | #+end_src
905 |
906 | ** array of callbacks on_exit
907 | Level up that pattern, we can have a helper to add callbacks to run
908 | on exit. Get used to these kind of patterns, they are super
909 | powerful and save you lots of manual bookkeeping.
910 |
911 | #+begin_src bash
912 |
913 | ON_EXIT=()
914 | EXIT_RES=
915 |
916 | function on_exit_fn {
917 | EXIT_RES=$?
918 | for cb in "${ON_EXIT[@]}"; do $cb || true; done
919 | return $EXIT_RES
920 | }
921 |
922 | trap on_exit_fn EXIT SIGINT
923 |
924 | function on_exit {
925 | ON_EXIT+=("$@")
926 | }
927 |
928 | local v_id=$(docker volume create)
929 | on_exit "docker volume rm $v_id"
930 | # Use your v_id knowing that it'll be available during your script but
931 | # will be cleaned up before exiting.
932 | #+end_src
933 |
934 | ** stacktrace on error
935 | Here's a nice helper for debugging errors in bash. In case of non-0
936 | exit, it prints a stacktrace.
937 | #+begin_src bash
938 | set -Eeuo pipefail
939 | trap stacktrace EXIT
940 | stacktrace() {
941 | rc="$?"
942 | if [ $rc != 0 ]; then
943 | printf '\nThe command "%s" triggerd a stacktrace:\n' "$BASH_COMMAND"
944 | for i in $(seq 1 $((${#FUNCNAME[@]} - 2))); do
945 | j=$((i+1));
946 | printf '\t%s: %s() called in %s:%s\n' "${BASH_SOURCE[$i]}" "${FUNCNAME[$i]}" "${BASH_SOURCE[$j]}" "${BASH_LINENO[$i]}";
947 | done
948 | fi
949 | }
950 |
951 | #+end_src
952 | ref: https://news.ycombinator.com/item?id=26644110
953 | ** Dots and colons allowed in function names!
954 | A way to split the namespace is to have libs define functions with
955 | their own namespace.
956 |
957 | I've gotten used to use dots or colons as namespace separator.
958 | #+begin_src bash
959 | semver.greater() {
960 | # ...
961 | }
962 | #+end_src
963 | or
964 | #+begin_src bash
965 | semver:greater() {
966 | # ...
967 | }
968 | #+end_src
969 | ** make steps of the process as composable as possible by using "$@"
970 | By using =$@= to pass commands as parameters around you can get to
971 | a degree of composability that allows for a nice chaining of
972 | commands.
973 |
974 | Here's a very simple version of =watch=. See how you can =every 2
975 | ls -la=. I think that style is called Bernstein Chaining. But I'm
976 | not sure if it's exactly the same. It also looks like currying or
977 | partial evaluation to me if you squint a little bit.
978 |
979 | #+begin_src bash
980 | every() {
981 | secs=$1
982 | shift
983 | while true; do
984 | "$@"
985 | sleep $secs;
986 | done
987 | }
988 | #+end_src
989 |
990 |
991 | As you know by now, bash doesn't pass blocks of code around, but
992 | the alternative is to pass function names.
993 |
994 | #+begin_src bash
995 | mute() {
996 | $@ >/dev/null 2>/dev/null
997 | }
998 | mute ls
999 | #+end_src
1000 |
1001 | So now we can create the most useless command composition ever:
1002 |
1003 | #+begin_src bash
1004 | every 1 mute echo hi
1005 | #or
1006 | mute every 1 echo hi
1007 | #+end_src
1008 |
1009 | For the particular redirection problem, another option is to use
1010 | aliases. Redirects can be written anywhere on your CLI (not just at
1011 | the end), so the following will work using a plain alias:
1012 | #+begin_src bash
1013 | alias mute='>/dev/null 2>/dev/null'
1014 | mute ls
1015 | #+end_src
1016 |
1017 |
1018 |
1019 | - https://www.oilshell.org/blog/2017/01/13.html
1020 | ** do_times/foreach_*
1021 | shellscripts are highly side-efffecty, and even though the scoping
1022 | of variables is not very empowering, you can get a limited amount
1023 | of decomposition of loops by passing function names.
1024 |
1025 | This is a lame example, but I hope it shows the use case, it allows
1026 | you to group already existing functions while taking advantage of a
1027 | fixed looping iterator, and leaving traces of the current loop vars
1028 | in the global "variable" environment.
1029 |
1030 | #+begin_src bash
1031 | create_user() {
1032 | uname="u$1" # leave uname in the global env so later functions see it
1033 | http :8080/users name="$uname"
1034 | }
1035 |
1036 | create_pet() {
1037 | pname="p$1"
1038 | http :8080/users/$uname/pets name="$pname"
1039 | }
1040 |
1041 | create_bundle() {
1042 | create_user $1
1043 | create_pet $1
1044 | }
1045 |
1046 | do_times() {
1047 | local n=$1; shift
1048 | for i in $(seq $n); do
1049 | "$@" $i
1050 | done
1051 | }
1052 |
1053 | do_times 15 create_bundle
1054 | #+end_src
1055 |
1056 | A bit more complex is runnning a command to every repo in an org:
1057 |
1058 | #+begin_src bash
1059 | run_tests() {
1060 | ./ci/test.sh
1061 | }
1062 |
1063 | foreach_repo_with_index() {
1064 | local counter=0
1065 | local repos=$(http https://api.github.com/users/$1/repos)
1066 | shift
1067 | for entry in $(echo $repos | jq -r '.[].git_url'); do
1068 | (git_clone_tmp $entry master
1069 | cd $tmpath
1070 | "$@" $counter $entry
1071 | )
1072 | ((counter=counter+1))
1073 | done
1074 | }
1075 |
1076 | foreach_repo_with_index kidd run_tests
1077 | #+end_src
1078 |
1079 | ** <(foo) and >(foo)
1080 | Some commands ask for files as inputs. And sometimes you have that
1081 | file, but sometimes you're only creating that file to pass it to
1082 | the command. In those cases, creating temporary files is not
1083 | necessary if you use =<(cmd)=. Here's a way to diff the output of 2
1084 | commands without putting them in a temporary file.
1085 |
1086 | #+begin_src bash
1087 | diff <(date) <(date)
1088 | diff <(date) <(sleep 1; date)
1089 | #+end_src
1090 |
1091 | The same happens with outputs. Commands that ask you for a
1092 | destination file. You can trick them by using =>(command)= as a
1093 | file. A nice trick is to use =>(cat)= to know what's going on
1094 | there. Also useful to send stuff to the clipboard =>(xclip)= before
1095 | running something on the output.
1096 |
1097 | What the shell does in those cases is to bind a file descriptor of
1098 | the process created inside =< or >= to the first process.
1099 |
1100 | You can experiment with those using commands like =echo <(pwd)=.
1101 |
1102 | In Zsh you can use =m-x expand-word= to see the file descriptors
1103 | being expanded.
1104 |
1105 | A way to peek into a huge pipe is to =tee >(cat)=
1106 |
1107 | ** Modify behavior by linking files
1108 | Imagine you want to remove the =.bash_history= file and not record
1109 | history of commands. You don't can't avoid bash to write to that
1110 | file, but what you can do is to link =.bash_history= to
1111 | =/dev/null=. That way, bash will keep appending things to the
1112 | expected output file, without noticing that it is really writing to
1113 | =/dev/null=. The trick is to use: =ln -sf /dev/null .bash_history=.
1114 |
1115 | ** Modify a file on the fly, without changing it on disk
1116 | If you want to use a slightly modified version of a file for a
1117 | single use, and then throw it away, consider the following:
1118 |
1119 | Everywhere that you should pass =file=, you could pass =<(cat
1120 | orig_file)= instead, because they have the same content, only the
1121 | second version is ephemeral.
1122 |
1123 | If you want a modified version of it, we can attach more commands
1124 | to the subshell.
1125 |
1126 | Example of a command that reuses your =~/.psqlrc= and adds a line
1127 | in the end to make the session read-only.
1128 |
1129 | #+begin_src bash
1130 | ropsql() {
1131 | PSQLRC=<(
1132 | cat ~/.psqlrc &&
1133 | echo 'set session characteristics as transaction read only;') \
1134 | psql "$@"
1135 | }
1136 | #+end_src
1137 |
1138 | ** Use xargs
1139 | Continuing with other ways of plumbing commands into other
1140 | commands, there's =xargs=. Some commands work seamlessly with
1141 | pipes, by taking inputs from stdin and printing to stdout. But
1142 | some others like to work with files, and they ask for their
1143 | parameters in their args list. For example, =evince=. It wouldn't
1144 | be even expected to cat a pdf and pass it to evince through
1145 | stdin.
1146 |
1147 | In general, to convert from this pattern: =cmd param= to =echo
1148 | param| cmd=, xargs can be helpful. Look at its man page to know how
1149 | to split or batch args in multiple =cmd= calls.
1150 |
1151 | Xargs is helpful for parallelizing work. You should look at its man
1152 | page, but just know it can help in running parallel processes
1153 | (check =-P= in its man).
1154 |
1155 | Other tips on this great [[https://www.oilshell.org/blog/2021/08/xargs.html][Guide to xargs]].
1156 |
1157 | ** xargs + paste
1158 | =paste= can be thougth as =zip=, interleaving the outputs of
1159 | several commands line by line.
1160 |
1161 | #+begin_src bash
1162 | paste <(./cmd1) <(./cmd2) | xargs -L1 ./cmd3
1163 | #+end_src
1164 |
1165 | Ref: https://news.ycombinator.com/item?id=29845232
1166 |
1167 | ** paste?
1168 | We just saw a use of =paste=. But this command can do all sorts of
1169 | crazy stuff joining lines.
1170 |
1171 | #+begin_src bash
1172 | paste - - -d, < file # joins every 2 lines with comma
1173 | paste - - - -d, < file # joins every 3 lines with comma
1174 | paste -s -d",\n" file #joins 1 and 2 lines with comma, 2 and 3 with \n
1175 | #+end_src
1176 |
1177 | ** Scripts as filters
1178 | Strive to create scripts that can be used as filters. Perl had
1179 | =while(<>) { ... }= that would open a file line by line, or get
1180 | input from stdin depending on the actual arguments to the file.
1181 |
1182 | In shellscripting, we can simulate (part of) it using
1183 | #+begin_src bash
1184 | while IFS= read -r line
1185 | do
1186 | echo "$line"
1187 | done < "${1:-/dev/stdin}"
1188 | #+end_src
1189 |
1190 | Look [[https://stackoverflow.com/questions/6980090/how-to-read-from-a-file-or-standard-input-in-bash][here]] for more detailed explanations and variants.
1191 |
1192 | Another way to create filters is to use `"${@:-cat}"`.
1193 |
1194 | #+begin_src bash
1195 | o2e () { 1>&2 "${@:-cat}"; } # out to err
1196 | e2o () { 2>&1 "${@:-cat}"; } # err to out
1197 | #+end_src
1198 |
1199 | ** Read or print to stdin inside a pipe
1200 | You can use /dev/tty to explicitly specify that some input or
1201 | output has to come from tty (being screen or keyboard).
1202 |
1203 | This is useful for code that might be called inside a pipeline,
1204 | but you never want the contents of the pipe to be the ones that
1205 | are =read= or =echoed=.
1206 |
1207 | Here's a snippet that defines a function fzf (if fzf is not
1208 | previously defined), that will naively replace fzf.
1209 |
1210 | #+begin_src bash
1211 | # fzf is either fzf or a naive version of it
1212 | # the input is a sed line number, so it can be
1213 | # single number: 42
1214 | # range: 1,10
1215 | # separate lines: 10p;50p
1216 | mute command -v fzf ||
1217 | fzf() {
1218 | local in=$(cat)
1219 | for p in "${@}"; do
1220 | [ "$p" = "-0" ] && [ "$(echo "$in" | wc -l)" -eq 1 ] && [ "" = "$in" ] && return 1
1221 | [ "$p" = "-1" ] && [ "$(echo "$in" | wc -l)" -eq 1 ] && [ "" != "$in" ] && echo "$in" && return
1222 | done
1223 | # https://superuser.com/questions/1748550/read-from-stdin-while-piping-to-next-command
1224 | cat -n <(echo "$in") >/dev/tty
1225 | read -n num /dev/tty [{"foo":"2009-02-13T23:31:30Z","bar":"2009-02-13T23:31:31Z"}]
1402 | jq_pretty_dates() {
1403 | # From the default ($@) append every element (/%/) with
1404 | # |=todateiso8601). That, joined by |. And that becomes the "filter" to jq.
1405 | jq "$(join_by "|" "${@/%/|=todateiso8601}")"
1406 | }
1407 | #+end_src
1408 |
1409 | There are quite a few interesting commands in this [[https://medium.com/circuitpeople/aws-cli-with-jq-and-bash-9d54e2eabaf1'][AWS+JQ post]].
1410 |
1411 | Some other cool tips from [[https://news.ycombinator.com/item?id=31141415][This HN thread]]:
1412 |
1413 | #+begin_src bash
1414 | echo "FFFF\nDDDD" | jq -R # reads raw text as strings. 1 line -> 1 string
1415 | echo "FFFF\nDDDD" | jq -R 'capture("(?.*)")' # Capture groups as json elements
1416 | #+end_src
1417 |
1418 | You can get pretty advanced with jq, and some say it's a decent
1419 | substitute for =sed= as a line oriented editor.
1420 |
1421 | Some more advanced tutorials [[https://programminghistorian.org/en/lessons/json-and-jq][here]] and [[https://remysharp.com/drafts/jq-recipes][here]].
1422 |
1423 | ** use jq . <<<"$var"
1424 | Newlines aren't allowed in json strings, and you might get a jq
1425 | error like:
1426 |
1427 | "parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 148, column 99"
1428 |
1429 | Sometimes, strings inside json might have encoded newlines like =\n=. If you would use
1430 |
1431 | =echo $var | jq .= , bash would interpolate the \n inside the json
1432 | string and when it gets to jq, it's an invalid json.
1433 |
1434 | If instead you use =echo -E=, or =<<<"$var"=, things go smoothly.
1435 |
1436 | =jq . <<<"$var"=
1437 |
1438 | ** pass flags as a splatted array
1439 | There's quite a bit to chew on this example. First of all, the core
1440 | pattern is to build up your commandline options with an array, and
1441 | splat it in the final command line. For complex commands like
1442 | =docker= where you easily have 10+ flags it's a visual aid, and
1443 | also opens up the opportunities for reusing or abstracting sets of
1444 | options to logical blocks.
1445 |
1446 | Once it's an array, we can add elements conditionally to that array
1447 | depending on the current run, and build the line that we'll be
1448 | running in the end.
1449 |
1450 | #+begin_src bash
1451 | # Allows Ctrl-C'ing on interactive shells
1452 | INTERACTIVE=
1453 | if [[ -t 1 ]]; then INTERACTIVE="-it"; fi
1454 |
1455 | local flags=(
1456 | # We mount it as read-only, so we make sure we are not writing anything
1457 | # in there, and that everything is explicitly defined
1458 | "-v $LOCAL_PATH/build-dir:/build-dir:ro,delegated"
1459 | "-v $OUTPUT_DIR:/output:rw,consistent"
1460 | "-v $tmp_dir:/tmp/work:rw,delegated"
1461 | )
1462 | if [[ -n $LOCAL_PATH ]]; then
1463 | flags+=("-v $(realpath $LOCAL_PATH)/overrides/my-other-file:/build-dir/build.json:ro")
1464 | flags+=("-e LOCAL_PATH=/tmp/local")
1465 | fi
1466 |
1467 | local v_id=$(docker volume create)
1468 | flags+=("-v $v_id:/tmp/build")
1469 | on_exit "docker volume rm $v_id"
1470 |
1471 | docker run --rm $INTERACTIVE ${flags[*]} $image touch /tmp/build/foo.txt
1472 |
1473 | docker run --rm $INTERACTIVE ${flags[*]} fpm:latest fpm-build /tmp/build/foo.txt
1474 | on_exit "chown_cache $tmp_dir"
1475 | #+end_src
1476 |
1477 | In this example we see another cool trick. Mounting a volume in 2
1478 | differrent containers, so not for the purpose of sharing a local
1479 | file/dir with the host but to share it between themselves. In that
1480 | case, the 2 containers don't even coexist temporarily, but use the
1481 | volume as a conveyor belt, passing it from container to container,
1482 | and each one applies "its thing".
1483 |
1484 | After all the mess, someone has to cleanup everything, but we know
1485 | how to do it with =on_exit= trick.
1486 |
1487 | ** inherit_errexit
1488 | bash 4.4+ , you can =shopt -s inherit_errexit=, and subshells will
1489 | inherit the errexit flag value. meaning that if you =set -Ee=,
1490 | anything that runs inside a subshell will throw an error at the
1491 | moment any command exits with =!=0=.
1492 | ** GNU Parallel
1493 | I can't recommend [[https://www.gnu.org/software/parallel/][parallel]] enough. The same as xargs, but in a
1494 | much more flexible way, parallel lets you run various jobs at a
1495 | time. If you have this tool into account, it doesn't just speed up
1496 | your runtimes, but it will force you write cleaner code. Parallel
1497 | execution will test your scripts so if they are not using
1498 | randomized tmp working directories, things will clash, etc...
1499 |
1500 | Parallel in itself is such a hackerfriendly tool it deserves to be
1501 | deeply learned. You can use it just locally to run a process per
1502 | core, you can send jobs to several machines connected via a simple
1503 | ssh, you can bind tmux or sqlite to it, or you can write a trivial
1504 | job queuing system.
1505 |
1506 | Man pages and official examples are a goldmine.
1507 |
1508 | ** HEREDOCS
1509 | - Basic usage of heredocs:
1510 | #+begin_src bash
1511 | echo < ${deploy_dir}"
1546 | ls -tdr ~/my-project/artifacts/* | head -n -5 | xargs rm -rf # remove all but latest 5 directories
1547 | ln -nfs "\$HOME/my-project/artifacts/${deploy_dir}/my-project-0.1.0-SNAPSHOT-standalone.jar" "\$HOME/my-project/artifacts/current.jar"
1548 | cat ~/my-project/artifacts/current.pid | xargs kill
1549 | java -jar ~/my-project/artifacts/current.jar &>/dev/null &
1550 | lastpid=\$(echo \$!)
1551 | echo \$lastpid > ~/my-project/artifacts/current.pid
1552 | EOSSH
1553 | }
1554 | #+end_src
1555 |
1556 | Refs: https://unix.stackexchange.com/questions/60688/how-to-defer-variable-expansion
1557 |
1558 | ** HERESTRINGS
1559 | It's the stripped down version of HEREDOCS. Inline a single string
1560 | (or output of a single command) as an input string.
1561 |
1562 | It's [[https://stackoverflow.com/questions/18116343/whats-the-difference-between-here-string-and-echo-pipe][kinda similar]] to what you could do with a pipe.
1563 | #+begin_src bash
1564 | cat <<<"HELLO"
1565 | cat <<<$(echo "HELLO")
1566 | echo "ECHOPIPE" | /bin/cat
1567 | echo "ECHOPIPE" | /bin/cat <(seq 5)
1568 | echo "ECHOPIPE" | /bin/cat <(seq 5) -
1569 | echo "ECHOPIPE" | /bin/cat <(seq 5) <<<"HERESTRING"
1570 | echo "ECHOPIPE" | /bin/cat <(seq 5) - <<<"HERESTRING"
1571 | #+end_src
1572 |
1573 | ** __DATA__
1574 | I loved the Perl [[https://perldoc.perl.org/perldata#Special-Literals][__END__ and __DATA__]], features and realized it's
1575 | possible to do it in shellscripts.
1576 |
1577 | You can append to the current file. Here's an example of a super
1578 | simple bookmark "manager":
1579 | #+begin_src bash
1580 | #!/usr/bin/env sh
1581 |
1582 | cmd_add() {
1583 | shift
1584 | echo "$@" >> "$0"
1585 | }
1586 |
1587 | cmd_go() {
1588 | sed '0,/^__DATA__$/d' "$0" |
1589 | dmenu -i -l 20 |
1590 | rev | cut -f1 -d' ' | rev |
1591 | xargs xdg-open
1592 | }
1593 |
1594 | main() {
1595 | cmd_${1:-go} $@
1596 | }
1597 |
1598 | main $@
1599 | exit
1600 |
1601 | __DATA__
1602 | r/emacs https://www.reddit.com/r/emacs
1603 | #+end_src
1604 | ** Simple Templating
1605 | For trivial templating, there's no need for any external tool
1606 | except for just variable interpolation. This is quite obvious, but
1607 | I use it quite often to generate repetitive bash code itself
1608 |
1609 | #+begin_src bash
1610 | #!/usr/bin/env bash
1611 | is=("obj1" "obj2")
1612 | js=("resource1" "resource2")
1613 |
1614 | for i in "${is[@]}"; do
1615 | for j in "${js[@]}"; do
1616 | echo "insert into t (obj, resource) values ('$i', '$j');"
1617 | done
1618 | done
1619 | #+end_src
1620 |
1621 | ** Secrets
1622 | As, [[https://smallstep.com/blog/command-line-secrets/][Smallstep article]] explains, you should be careful when passing
1623 | around secret data like tokens between processes.
1624 |
1625 | Env vars are not really safe, and there are a few tricks you can
1626 | use to cover your assets.
1627 |
1628 | ** Escape single quotes in single quoted strings
1629 | Escaping single quoted strings doesn't work the way you'd think.
1630 |
1631 | #+begin_src bash
1632 | # does not work
1633 | a='hi \'bob\''
1634 | #+end_src
1635 |
1636 | Use =$''= strings like:
1637 | #+begin_src bash
1638 | # works!
1639 | a=$'hi \'bob\''
1640 | echo $a
1641 | #+end_src
1642 |
1643 | Extra explanations: [[https://stackoverflow.com/questions/1250079/how-to-escape-single-quotes-within-single-quoted-strings][StackOverflow]]
1644 |
1645 | ** any, every, lambdas, closures, and eval
1646 | Bash has no closures, and it's not "functional", but we're going to
1647 | see how we can combine some tricks and get decent high-level functions
1648 |
1649 | Here's an implementation of =any= and =every=
1650 |
1651 | #+begin_src bash
1652 | any() {
1653 | local pred="$1"; shift
1654 | for i in "$@"; do
1655 | if $pred $i; then
1656 | return 0
1657 | fi
1658 | done
1659 | return 1
1660 | }
1661 |
1662 | every() {
1663 | local pred="$1"; shift
1664 | for i in "$@"; do
1665 | if ! $pred $i; then
1666 | return 1
1667 | fi
1668 | done
1669 | return 0
1670 | }
1671 | #+end_src
1672 |
1673 | The usage is by passing a callback function that will return 0 or 1.
1674 |
1675 | #+begin_src bash
1676 | usage() {
1677 | echo "help message"
1678 | }
1679 |
1680 | is_help() {
1681 | [ "$1" = "--help" ]
1682 | }
1683 |
1684 | any is_help "$@" && usage
1685 | #+end_src
1686 |
1687 | That's already cool.
1688 |
1689 | Did you know that functions can start also with dashes?
1690 |
1691 | #+begin_src bash
1692 | usage() {
1693 | echo "help message"
1694 | }
1695 |
1696 | --help() {
1697 | [ "$1" = "--help" ]
1698 | }
1699 |
1700 | any --help "$@" && usage
1701 | #+end_src
1702 |
1703 | That also works. It's kinda obfuscated a bit, but hey....
1704 |
1705 | We can also, with the help of =eval=, create on demand functions. They
1706 | are global, but as we're not going to multithread, we can assume it's
1707 | ok.
1708 |
1709 | #+begin_src bash
1710 | usage() {
1711 | echo "help message"
1712 | }
1713 |
1714 | any_eq() {
1715 | eval "_() { [ $1 = \$1 ] ; }"
1716 | shift
1717 | any _ "$@"
1718 | }
1719 |
1720 | any_eq "--help" "$@" && usage
1721 | # or,
1722 | any_eq --help "$@" && usage
1723 | #+end_src
1724 |
1725 | And the last trick, using =command_not_found_handle=, we can get into
1726 | a kind of rails generators for these helpers
1727 |
1728 | #+begin_src bash
1729 | command_not_found_handle() {
1730 | IFS=_ read -a cmd <<<"$1"
1731 | [ "eq" = "${cmd[0]}" ] && eval "$1() { [ \"${cmd[1]}\" = \"\$1\" ] ; }"
1732 | "$@"
1733 | }
1734 | any eq_--help "$@" && usage
1735 | #+end_src
1736 |
1737 | All these last metaprogramming tricks are not thread safe, and even
1738 | though it's mostly ok if you use them in isolation, maybe pipes or
1739 | coprocs would mess up your inline functions.
1740 |
1741 | * Interactive
1742 | ** Save your small scripts
1743 | Rome wasn't built in a day, and like having a journal log, most of
1744 | the little scripts you create, once you have enough discipline will
1745 | be useful for some other cases, and your functions will be
1746 | reusable.
1747 |
1748 | Save your scripts into files early on, instead of crunching
1749 | everything in the repl. learn how to use a decent editor that
1750 | shortens the feedback cycle as much as possible.
1751 | ** Increased Interactivity
1752 | Knowing your shell's shortcuts for interactive use is a must. The
1753 | same way you learned to touchtype and you learned your editor, you
1754 | should learn all the shortcuts for your shell. Here's some of them.
1755 |
1756 | | key | action |
1757 | |--------+-------------------------------------|
1758 | | Ctrl-r | reverse-history-search |
1759 | | C-a | beginning-of-line |
1760 | | C-e | end-of-line |
1761 | | C-w | delete-word-backwards |
1762 | | C-k | kill-line (from point to eol) |
1763 | | C-y | paste last killed thing |
1764 | | A-y | previous killed thing (after a c-y) |
1765 | | C-p | previous-line |
1766 | | C-n | next-line |
1767 | | A-. | insert last agument |
1768 | | A-/ | dabbrev-expand |
1769 |
1770 | A written form of =A-.= is =$_=. It retains the last argment and
1771 | puts it in $_. =test -f "FILE" && source "$_"=.
1772 |
1773 | ** Aliases
1774 | Aliases are very simple substitutions of commands for a sequence of
1775 | other commands. Usual example is
1776 |
1777 | #+begin_src bash
1778 | alias ls='ls --auto-color'
1779 | #+end_src
1780 |
1781 | Now let's move on to the interesting stuff.
1782 |
1783 | ** functions can generate aliases
1784 |
1785 | Aliases live in a global namespace for the shell, so no matter
1786 | where you define them, they take effect globally, possibly
1787 | overwriting older aliases with the same name.
1788 |
1789 | Well, it's not lexical scope (far from it), but using aliases you
1790 | can create a string that snapshots the value you want, and capture
1791 | it to run it later.
1792 |
1793 | Some fun stuff:
1794 |
1795 | - aliasgen. Create an alias for each directory in
1796 | ~/workspace/. This is superceeded by =CDPATH=, but the trick is
1797 | still cool.
1798 | #+begin_src bash
1799 | aliasgen() {
1800 | for i in ~/workspace/*(/) ; do
1801 | DIR=$(basename $i) ;
1802 | alias $DIR="cd ~/workspace/$i";
1803 | done
1804 | }
1805 | aliasgen
1806 | #+end_src
1807 |
1808 | - a make a shortcut to the current directory.
1809 | #+begin_src bash
1810 | function a() { alias $1=cd\ $PWD; }
1811 |
1812 | mkdir -p /tmp/fing-longer
1813 | cd /tmp/fing-longer
1814 | a fl
1815 | cd /
1816 | fl
1817 | echo $PWD # /tmp/fing-longer
1818 | #+end_src
1819 | A man can dream...
1820 |
1821 |
1822 | - unhist. functions can create aliases, and functions can receive
1823 | functions as parameters (as a string (function name)), so we can
1824 | combine them to advice existing functions.
1825 | #+begin_src bash
1826 | unhist () {
1827 | alias $1=" $1"
1828 | }
1829 | unhist unhist
1830 | unhist grep
1831 | unhist rg
1832 |
1833 | noglobber() {
1834 | alias $1="noglob $1"
1835 | }
1836 | noglobber http
1837 | noglobber curl
1838 | noglobber git
1839 |
1840 | #+end_src
1841 |
1842 | #+RESULTS:
1843 |
1844 | - Problem: These commands do not compose. Combination of 2 of those
1845 | doesn't work, because the second acts just on the textual
1846 | representation that it received, not the current value of the
1847 | alias.
1848 |
1849 | # Solution :
1850 | # #+begin_src bash
1851 | # alias-to() {
1852 | # alias $1 | sed -e "s/.*='//" -e "s/'\$//"
1853 | # }
1854 |
1855 | # aliasappend() {
1856 | # local cmd
1857 | # if alias $1 >/dev/null; then
1858 | # cmd=$(alias-to $1)
1859 | # else
1860 | # cmd=$1
1861 | # fi
1862 | # echo $cmd
1863 | # }
1864 |
1865 | # muter() {
1866 | # local c
1867 | # c=$(aliasappend $1)
1868 | # alias $1="$c >/dev/null"
1869 | # }
1870 |
1871 | # unhist() {
1872 | # local c
1873 | # c=$(aliasappend $1)
1874 | # alias $1=" $c"
1875 | # }
1876 |
1877 | # unhist ls
1878 | # muter ls
1879 | # ls
1880 | # #+end_src
1881 | ** Override (advice?) common functions
1882 | Overriding commands is generally a bad practice as it violates the
1883 | principle of least surprise, but there might be occasions (mostly
1884 | in your local machine) where you can integrate awesome finetunnings
1885 | to your toolbelt.
1886 |
1887 | Here we're going to get the original docker binary file
1888 | location. After that we declare a function called =docker= that
1889 | will proxy the parameters to the original =docker= program UNLESS
1890 | you're calling =docker run=. In that case, we're injecting a mouted
1891 | volume that mounts =/root/.bash_history= of the container to a file
1892 | hosted in the host (duh). That's a pretty cool way of keeping a
1893 | history of your recent commands in your containers, no matter how
1894 | many times you start and kill them.
1895 |
1896 | #+begin_src bash
1897 | DOCKER_ORIG=$(which docker)
1898 | docker () {
1899 | if [[ $1 == "run" ]]; then
1900 | shift
1901 | $DOCKER_ORIG run -v $HOME/.shared_bash_history:/root/.bash_history "$@"
1902 | else
1903 | $DOCKER_ORIG "$@"
1904 | fi
1905 | }
1906 | #+end_src
1907 |
1908 | I'm particularly fond of this trick, as it saved me tons of
1909 | typing. But at a personal level, it was mindblowing that sharing
1910 | this around the internet caused the most disparity of opinions.
1911 | Also, I recently read the great book "Docker in Practice" by [[https://github.com/ianmiell][Ian
1912 | Miell]] and there's a snippet that is 99.9% like the one I
1913 | created myself. That was a very cool moment.
1914 |
1915 | ** Faster iteration on pipes
1916 | When testing complex pipelines:
1917 |
1918 | - Make them pure (no side effects).
1919 | - One command per line.
1920 | - End lines with the pipe character.
1921 | - During development, end the pipeline with =cat=.
1922 |
1923 | I usually use =watch -n1 'code.sh'= in a split window so I see the
1924 | results of what I'm doing. The advantage of
1925 |
1926 | #+begin_src bash
1927 | curl https://www.example.com/videos/ |
1928 | pup 'figure.mg > a attr{href}' |
1929 | head -1 |
1930 | xargs -I{} curl -L "https://www.example.com/{}" |
1931 | pup 'script' |
1932 | grep file: |
1933 | sed -e "s/.*\(http[^ \"']*\).*/\1/" |
1934 | # xargs vlc |
1935 | cat
1936 | #+end_src
1937 |
1938 | Over
1939 | #+begin_src bash
1940 | curl https://www.example.com/videos/ \
1941 | | pup 'figure.mg > a attr{href}' \
1942 | | head -1 \
1943 | | xargs -I{} curl -L "https://www.example.com/{}" \
1944 | | pup 'script' \
1945 | | grep file: \
1946 | | sed -e "s/.*\(http[^ \"']*\).*/\1/" \
1947 | # xargs vlc # doesn't work
1948 | #+end_src
1949 |
1950 | Is that you can comment out lines on the former one, but you can't
1951 | do that on the latter. The =cat= trick makes it so that you have an
1952 | 'exit' point, and you don't have to comment that one. Also, some
1953 | editors will indent the first one correctly, while you'll have to
1954 | manually indent the second one.
1955 |
1956 | Small wins that compose just fine :)
1957 |
1958 | ** Use ${var?error msg} on templates
1959 | If you write something to be copypasted by your user and filled in,
1960 | instead of ==, try =${var?You need to set var}=. it allows for the user to set
1961 | the variable in the environment without having to replace inline,
1962 | and if the user forgets any, the shell will barf.
1963 |
1964 | ** Idempotent functions
1965 | Being able to call the same function multiple times even if you just
1966 | called it, without having to account for "oh, did this already happen
1967 | or not?" but instead have a mental model of "I need this to have
1968 | happened after this function call", and consider it done, is very
1969 | liberating from the code perspective.
1970 |
1971 | Also, there are subgoals, like caching slow functions, downloads, etc.
1972 |
1973 | *** nop subsequent calls
1974 | "My favourite shell scripting function definition technique:
1975 | idempotent functions by redefining the function as a noop inside
1976 | its body:
1977 |
1978 | (The `true;` body is needed by some shells, e.g. bash, and not
1979 | others, e.g. zsh.) " -- [[https://news.ycombinator.com/item?id=27729397][chrismorgan]]
1980 |
1981 | #+begin_src bash
1982 | foo() {
1983 | foo() { true; }
1984 | echo "This bit will only be executed on the first foo call"
1985 | }
1986 | #+end_src
1987 | *** Download once
1988 | A function might be sideffecty, as in "download files". These sort of
1989 | cases, you want to download the file only if you haven't downloaded it
1990 | yet.
1991 |
1992 | =cache= here gets a cache "id" where it'll store the results of =http=
1993 | and next time, when called with the same param, it will reuse the saved version.
1994 |
1995 | This pattern applies to many kinds of functions, so with slight
1996 | modifications you can adapt it.
1997 |
1998 | #+begin_src bash
1999 | cache() {
2000 | local path=$CACHE_DIR/$1
2001 | [[ -f $path ]] && cat $path && return 0
2002 |
2003 | shift
2004 | $@ | tee $path
2005 |
2006 | return 1
2007 | }
2008 |
2009 | get_repos() {
2010 | local url="https://api.github.com/search/repositories?q=user:$USER&access_token=$GITHUB_TOKEN&per_page=100&page=$1"
2011 | cache $USER-repos-$1 http $url
2012 | }
2013 | #+end_src
2014 |
2015 | *** memoize with ttl, with ttl
2016 | What if you want to cache for some time?
2017 |
2018 | The tricks here:
2019 |
2020 | - Use a file as a sentinel for the cache.
2021 | - Create a file, set it's modification date to now()+12hours
2022 | - check with `-nt` (newer than) and a temporary file (to have a
2023 | =now()= value in "file" format).
2024 |
2025 | #+begin_src bash
2026 | hmacache="/tmp/.hmacache"
2027 | is_recent_cache() {
2028 | [ "$hmacache" -nt <(echo 1) ]
2029 | }
2030 | update_cache() {
2031 | touch -m -t "$(date -v+12H +"%Y%m%d%H%M.%S")" "$hmacache"
2032 | }
2033 |
2034 | main() {
2035 | #...
2036 | if ! is_recent_cache ; then
2037 | if ! curl -s https://hello.ts.net > /dev/null; then
2038 | die "Tailscale must be connected to start Harbormaster"
2039 | fi
2040 | # other slow checks
2041 | update_cache
2042 | fi
2043 | }
2044 | #+end_src
2045 |
2046 | * Debugging
2047 | ** adding =bash= to a script to debug
2048 | You can add =bash= inside any script, and it'll add a sort of
2049 | a breakpoint, allowing you to check the state of the env and
2050 | manually call functions around.
2051 |
2052 | If you orgainse your code in small functions, it's easy to add
2053 | breakpoints by just spawning bash processes inside your script.
2054 |
2055 | This works also inside docker containers (if you provide =-ti= flag
2056 | on run).
2057 |
2058 | Let's see some usual uses of docker and how we can debug our
2059 | scripts there:
2060 |
2061 | #+begin_src bash
2062 | # leaves you at a shell to fiddle if all is in place after build
2063 | docker run -it mycomplex-image bash
2064 |
2065 | # Runs /tmp/file.sh from the host inside. That's cool to make the
2066 | # container less airtight. Even if the image is not originally meant
2067 | # to, you can even override it and 'monkeypatch' the file with the one
2068 | # from the host anyway.
2069 | docker run -it -v $PWD:/tmp/ mycomplex-image /tmp/file.sh
2070 |
2071 | # So now you can really add wtv you want there.
2072 | echo 'bash' >>$PWD/file.sh
2073 |
2074 | # run+open shell at runtime to inspect the state of the script
2075 | docker run -it -v $PWD:/tmp/ mycomplex-image /tmp/file.sh
2076 | #+end_src
2077 |
2078 | ** DRY_RUN
2079 | #+begin_src bash
2080 | if [[-n "$DRY_RUN" ]]; then
2081 | curl () {
2082 | echo curl "$@"
2083 | }
2084 | fi
2085 | #+end_src
2086 | use =command curl= to force the command, not the alias or anything
2087 |
2088 | If you have many functions to mock, and you want to overabstract (bad thing):
2089 |
2090 | #+begin_src bash
2091 |
2092 | mock() {
2093 | for m in ${MOCKS[@]//,/ }; do
2094 | eval "$m() { echo \"mocked $m\" "\$@"; }"
2095 | done
2096 | }
2097 |
2098 | MOCKS=curl,ls script.sh
2099 | #+end_src
2100 |
2101 |
2102 |
2103 | ** Cheap debugging flag
2104 |
2105 | #+begin_src bash
2106 | optargs "V" option; do
2107 | case $option in
2108 | V)
2109 | set -xa
2110 | ;;
2111 | #+end_src
2112 |
2113 | ** explore a pipe with tee >(some_command) |
2114 | the =>()= is not very easy to use. Very few places where it
2115 | fits. Here's a nice pipe inspector though, using =tee >(cat 1>&2)=
2116 | trick.
2117 | #+begin_src bash
2118 | plog() {
2119 | # tee >(cat 1>&2)
2120 | local msg=${1:-plog}
2121 | tee >(sed -e "s/^/[$msg] /" 1>&2)
2122 | }
2123 | alias -g 'PL'=' |plog ' #zsh only
2124 |
2125 | echo "a\nb" PL foo | tr 'a-z' 'A-Z' PL bar
2126 | # output:
2127 | # [foo] a # stderr
2128 | # [foo] b # stderr
2129 | # A # stdout
2130 | # B # stdout
2131 | # [bar] A # stderr
2132 | # [bar] b # stderr
2133 | #+end_src
2134 |
2135 |
2136 | ref: https://stackoverflow.com/questions/17983777/shell-pipe-to-multiple-commands-in-a-file
2137 |
2138 | ** tee+sudo
2139 | If you wat to store a file in a root-owned dir, in the middle of
2140 | your pipeline, instead of running the whole thing as root, you can
2141 | use =sudo tee file=:
2142 | #+begin_src bash
2143 | ls | grep m >/usr/local/garbage # fail
2144 | ls | grep m | sudo tee /usr/local/garbage # success!
2145 | #+end_src
2146 | ** quoting
2147 | Bash: To get a quoted version of a given string, here's what you can do:
2148 | #+begin_src bash
2149 |
2150 | # this is my "string" I want to 'comment "on"'
2151 | !:q
2152 | #+end_src
2153 |
2154 | That gives us ='#this is my "string" I want to '\''comment
2155 | "on"'\'''=. Neat!
2156 |
2157 | I just found this trick [[https://til.simonwillison.net/til/til/bash_escaping-a-string.md][here]]. From the associated HN thread:
2158 | #+begin_src bash
2159 | function bashquote () {
2160 | printf '%q' "$(cat)"
2161 | echo
2162 | }
2163 | #+end_src
2164 |
2165 | Zsh: If you're on zsh, =a-'= quotes the current line.
2166 | * zsh-only
2167 | ** Word spliting
2168 | Word splitting works differently by default in zsh than in bash.
2169 | #+begin_src bash
2170 | foo="ls -las"
2171 | $foo
2172 | #+end_src
2173 | This works in bash, but zsh will not split by words. To make zsh
2174 | expand by words, there are 2 ways: =setopt SH_WORD_SPLIT= and
2175 | =${=foo}=. zsh has =unsetop= command, which allows to scope where
2176 | you want the expansions to happen. =unsetop SH_WORD_SPLIT=.
2177 |
2178 | The problem with both solutions is that none of them are compatible
2179 | with bash, so you'll be cornering yourself to "this only works in
2180 | zsh". A way to overcome this is to use arrays, which are expanded
2181 | in the same way in both shells.
2182 |
2183 | Or, use the same hack as you'll see later with noglob.
2184 |
2185 | Refs:
2186 | - https://stackoverflow.com/questions/6715388/variable-expansion-is-different-in-zsh-from-that-in-bash
2187 | - http://zsh.sourceforge.net/FAQ/zshfaq03.html#l18
2188 | ** globbing
2189 | In zsh, getting a list of files that match some characteristics is
2190 | doable using globbing. Bash has globbing also, but in a less sophisticated way.
2191 |
2192 | The basic structure of a =glob= is =pattern(qualifiers)=. Patterns
2193 | can contain:
2194 | - strings: they do exact match
2195 | - wildcards: =*=, =?=, =**/=
2196 | - character classes: =[0-9]=
2197 | - choices: =(.pdf|.djvu)=
2198 |
2199 | The qualifiers are extra constraints you put on the matches. There
2200 | are lots of different qualifiers. Look at =zshexpn= for the
2201 | complete list. The ones I use more are:
2202 |
2203 | - =.= Files
2204 | - =/= Directories
2205 | - =om[numberhere]=. Nth latest modified
2206 |
2207 | ** Global aliases:
2208 | Zsh has 3 kinds of aliases: normal, global, and suffix. Normal ones
2209 | behave the same as in bash: They expand only in the first token of
2210 | a line.
2211 |
2212 | #+begin_src bash
2213 | alias foo=ls
2214 | foo foo # -> ls: cannot access 'foo': No such file or directory
2215 | #+end_src
2216 |
2217 | Global aliases expand anywhere in your cli, and they can expand to
2218 | anything.
2219 |
2220 | #+begin_src bash
2221 | alias -g foo=ls
2222 | foo foo # -> ls: cannot access 'ls': No such file or directory
2223 | #+end_src
2224 |
2225 | These are some aliases I have in my ~/.zshrc that help me use a
2226 | shell in a more fluid way.
2227 |
2228 | #+begin_src bash
2229 | alias -g P1='| awk "{print \$1}"'
2230 | alias -g P2='| awk "{print \$2}"'
2231 | alias -g P3='| awk "{print \$3}"'
2232 | alias -g P4='| awk "{print \$4}"'
2233 | alias -g P5='| awk "{print \$5}"'
2234 | alias -g P6='| awk "{print \$6}"'
2235 | alias -g PL='| awk "{print \$NF}"'
2236 | alias -g PN='| awk "{print \$NF}"'
2237 | alias -g HL='| head -20'
2238 | alias -g H='| head '
2239 | alias -g H1='| head -1'
2240 | alias -g TL='| tail -20'
2241 | alias -g T='| tail '
2242 | alias -g T1='T -1'
2243 | #alias -g tr='-ltr'
2244 | alias -g X='| xclip '
2245 | alias -g TB='| nc termbin.com 9999 '
2246 | alias -g L='| less -R '
2247 | alias -g LR='| less -r '
2248 | alias -g G='| grep '
2249 | alias -g GI='| grep -i '
2250 | alias -g GG=' 2>&1 | grep '
2251 | alias -g GGI=' 2>&1 | grep -i '
2252 | alias -g GV='| grep -v '
2253 | alias -g V='| grep -v '
2254 | alias -g TAC='| tac '
2255 | alias -g DU='du -B1'
2256 |
2257 | alias -g E2O=' 2>&1 '
2258 | alias -g NE=' 2>/dev/null '
2259 | alias -g NO=' >/dev/null '
2260 |
2261 | alias -g WC='| wc -l '
2262 |
2263 | alias -g J='| noglob jq'
2264 | alias -g JQ='| noglob jq'
2265 | alias -g jq='noglob jq'
2266 | alias -g JL='| noglob jq -C . | less -R '
2267 | alias -g JQL='| noglob jq -C . | less -R '
2268 | alias -g XMEL='| xmlstarlet el'
2269 | alias -g XML='| xmlstarlet sel -t -v '
2270 |
2271 | alias -g LYNX="| lynx -dump -stdin "
2272 | alias -g H2T="| html2text "
2273 | alias -g TRIM="| xargs "
2274 | alias -g XA='| xargs -d"\n" '
2275 | alias -g XE="| xargs e"
2276 | alias -g P="| pick "
2277 | alias -g PP="| percol | xargs "
2278 | alias -g W5="watch -n5 "
2279 | alias -g W1="watch -n1 "
2280 |
2281 |
2282 | alias -g CB="| col -b "
2283 | alias -g NC="| col -b "
2284 | alias -g U='| uniq '
2285 | alias -g XT='urxvt -e '
2286 | alias -g DM='| dmenu '
2287 | alias -g DMV='| dmenu -i -l 20 '
2288 |
2289 | alias -g ...='../..'
2290 | alias -g ....='../../..'
2291 | alias -g .....='../../../..'
2292 |
2293 | alias -g l10='*(om[1,10])'
2294 | alias -g l20='*(om[1,20])'
2295 | alias -g l5='*(om[1,5])'
2296 | alias -g l='*(om[1])'
2297 | alias -g '**.'='**/*(.)'
2298 | alias -g lpdf='*.pdf(om[1])'
2299 | alias -g lpng='*.png(om[1])'
2300 |
2301 | alias -g u='*(om[1])'
2302 |
2303 | alias lsmov='ls *.(mp4|mpg|mpeg|avi|mkv)'
2304 | alias lspdf='ls *.(pdf|djvu)'
2305 | alias lsmp3='ls *.mp3'
2306 | alias lspng='ls *.png'
2307 | #+end_src
2308 | Now, some sequences of words can start making sense:
2309 |
2310 | - =lspdf -tr TL DM XA evince=
2311 | - =docker exec -u root -ti $(docker ps -q H1) bash=
2312 | - =docker ps DM P1 XA docker stop=
2313 | - =docker ps P1 XA docker stop= (P1 XA does an ad-hoc [[https://bash-my-aws.org/pipe-skimming/][pipe-skimming]])
2314 |
2315 | ** Expansion of global aliases
2316 | Let's dig deeper.
2317 | #+begin_src zsh
2318 | alias -g DOCK='docker ps 2..N P P1'
2319 | DOCK # works fine. Echoes the id of the chosen container
2320 | docker stop DOCK # does not work , because it expands to docker stop docker ps 2..N P P1
2321 | docker stop $(DOCK) # works fine again
2322 | alias -g DOCK='$(docker ps 2..N P P1)'
2323 | docker stop DOCK # yay!
2324 | #+end_src
2325 |
2326 | So, you can bind words to expansion-time results of the aliases. It
2327 | feels like a very powerful thing, to have this "compile time"
2328 | expansions. Reminds me of CL's symbol-macrolet, or IMMEDIATE Forth
2329 | words.
2330 | ** Autocomplete
2331 | Writting smart autocompletion scripts is not easy.
2332 |
2333 | zsh supports =compdef _gnu_generic= type of completion, which gets
2334 | you very far with 0 effort.
2335 |
2336 | When autocompleting after a =-= in the commandline, if your command
2337 | is configured like =compdef _gnu_generic mycommand=, zsh will call
2338 | the script with =--help= and parse the output, trying to find
2339 | flags, and will use them as suggestions. It's really great.
2340 |
2341 | The compromise is to write a decent "--help" for your script. Which
2342 | is cool because your user will love it too, and you just have to
2343 | write it once.
2344 |
2345 | The completion is not context aware though, so you can't
2346 | autocomplete flags after the first non-flag argument. It seems this
2347 | could be improved in zsh-land, by asking for the --help like
2348 | =mycommand args-so-far --help=. But it doesn't work like that.
2349 |
2350 |
2351 | #+begin_src zsh
2352 | #!/usr/bin/env bash
2353 | # The script can be bash-only, while the completion work in zsh-only
2354 | set -Eeuo pipefail
2355 |
2356 | help() {
2357 | echo " -h,--help Show help"
2358 | echo " -c,--command Another thing"
2359 | }
2360 |
2361 | if [ "$1" == "--help" ];
2362 | help
2363 | fi
2364 | #+end_src
2365 |
2366 | Now you can play with =mycommand -=. Amazing, wow.
2367 |
2368 | ** Create helpers and generate global aliases automagically
2369 | Borrowing a bit from Perl, a bit from Forth, and a bit from
2370 | PicoLisp, I've come to create a few helpers that abstract words
2371 | into a bit higher level concepts. Unifying the option selectors is
2372 | one, and then, other line oriented operations like =chomp, from,
2373 | till=.
2374 |
2375 | #+begin_src bash
2376 | pick() {
2377 | if [ -z "$DISPLAY" ]; then
2378 | percol || fzf || slmenu -i -l 20
2379 | else
2380 | dmenu -i -l 20
2381 | fi
2382 | }
2383 | alias -g P='| pick'
2384 |
2385 | globalias() {
2386 | alias -g `echo -n $1 | tr '[a-z]' '[A-Z]'`=" | $1 "
2387 | }
2388 |
2389 | globalias fzf
2390 |
2391 | # uniquify column
2392 | function uc () {
2393 | awk -F" " "!_[\$$1]++"
2394 | }
2395 | globalias uc
2396 |
2397 | function from() { perl -pe "s|.*?$1|\1|" }
2398 | globalias from
2399 |
2400 | function till() { sed -e "s|$1.*|$1|" }
2401 | globalias till
2402 |
2403 | function chomp () { sed -e "s|.$||" }
2404 | globalias chomp
2405 | #+end_src
2406 |
2407 | Again, it's a pity those do not compose well. Just be well
2408 | organized, or build a more elaborate hack so you can compose
2409 | aliases with some sort of confidence. It'll always be a hack
2410 | though.
2411 | ** suffix aliases don't have to match a filename
2412 | zsh has another type of aliases called "suffix alias". Those alias
2413 | allow you to define programs to open/run file types.
2414 | #+begin_src shell
2415 | alias -s docx="libreoffice"
2416 | #+end_src
2417 |
2418 | With this said, if you write a name of a file ending with =docx= as
2419 | the first token in a command line, it will use libreoffice to open
2420 | it.
2421 |
2422 | #+begin_src shell
2423 | invoice1.docx
2424 | # will effectively call libreoffice invoice1.docx
2425 | #+end_src
2426 |
2427 | The trick here is that the parser doesn't check that the file is
2428 | indeed an existing file. It can be any string.
2429 |
2430 | Let's look at an example of it.
2431 |
2432 | #+begin_src shell
2433 | alias -s git="git clone"
2434 | #+end_src
2435 |
2436 | In this case, we can easily copy a =git@github.com:.....git= from a
2437 | browser, and paste it into a zsh console. Then, zsh will run that
2438 | "file" with the command =git clone=, effectively cloning that
2439 | repository.
2440 |
2441 | Cool, ain't it?
2442 |
2443 | ** noglob
2444 | zsh has more aggressive parameter expansion, to the level that
2445 | =[,],...= have special meanings, and will be interpreted and
2446 | expanded before calling the final commands in your shell.
2447 |
2448 | There are commands that you don't want ever expanded , for example,
2449 | when using =curl=, it's much more likely that an open bracket will
2450 | be ment to be there verbatim rather than expanded.
2451 |
2452 | Zsh provides a command to quote the following expansions. And it's
2453 | called noglob.
2454 | #+begin_src bash
2455 | noglob curl http://example.com\&a[]=1
2456 | #+end_src
2457 |
2458 | ** make noglob 'transparent' to bash
2459 | zsh and bash are mostly compatible, but there's a few things not
2460 | supported in bash. =noglob= is one of them. To build a shim
2461 | inbetween, an easy way is to just create a =~/bin/noglob= file
2462 | #+begin_src bash
2463 | $*
2464 | #+end_src
2465 |
2466 | ** glob nested expansion
2467 | In https://news.ycombinator.com/item?id=26175894 there's a nice
2468 | advanced example:
2469 |
2470 | #+begin_src text
2471 | Variable expansion syntax, glob qualifiers, and history modifiers can
2472 | be combined/nested quite nicely. For example, this outputs all the
2473 | commands available from $PATH: `echo
2474 | ${~${path/%/\/*(*N:t)}}`. `${~foo}` is to enable glob expansion on the
2475 | result of foo. `${foo/%/bar}` substitutes the end of the result of foo
2476 | to "bar" (i.e. it appends it); when foo is an array, it does it for
2477 | each element. In `/*(*N:t)`, we're adding the slash and star to the
2478 | paths from `$path`, then the parentheses are glob qualifiers. `*`
2479 | inside means only match the executables, `N` is to activate NULL_GLOB
2480 | for the match so that we don't get errors for globs that didn't match
2481 | anything, `:t` is a history mod used for globs that returns just the
2482 | "tail" of the result, i.e. the basename. IIRC, bash can't even nest
2483 | multiple parameter expansions; you need to save each step separately.
2484 | #+end_src
2485 | ** Some extra shortcuts for nice things
2486 | - =alt-'= quotes the current line. It's like =quotemeta=. great to
2487 | help you fight double and triple quoting when writing scripts.
2488 | - =alt-#= comment/uncomment and execute. Nice way to store the current line
2489 | for later and reach out to it again to run it for real.
2490 | - =ctrl-o= kill-current-line, wait for a command, and paste.
2491 |
2492 | ** =()
2493 | Zsh has =<()= and =>()= like Bash, but it also has ==()=. This
2494 | varant is similar to =<()= but instead of creating a temporary
2495 | pipe, it creates a temporary file. That is useful if we want to run
2496 | commands that require a file instead of a pipe (most times, because
2497 | it uses lseek to go through it).
2498 |
2499 | Node is an example of this.
2500 | #+begin_src shell
2501 | node <(echo 'setTimeout(() => console.log("foo"), 400)') # fails
2502 | node =(echo 'setTimeout(() => console.log("foo"), 400)') # works!
2503 | #+end_src
2504 |
2505 | Or,
2506 |
2507 | #+begin_src bash
2508 | docker run --rm -ti -v =(echo "OHAI"):/tmp/foo ubuntu cat /tmp/foo
2509 | #+end_src
2510 |
2511 | * TODO patterns
2512 | ** just use cat/netcat/pipes with <()
2513 |
2514 | - input
2515 |
2516 | =python logger.py executable= will run the executable and monitor it
2517 | for error messages. Depending on the error messages it will be doing.
2518 |
2519 | In order to test it, I want to run it with my own output. So what I
2520 | do is =python logger.py cat=. That way I can type my stuff there,
2521 | and even better, I can use a stream from the shell. =myexecutable |
2522 | python logger.py cat= still works.
2523 |
2524 | *** what's the unifying theory behind all that?
2525 | It's still not clear to me how they relate, but the feeling is
2526 | that there's a common thread ruling all those commands. as if they
2527 | generalize over the same things, or just a couple of very
2528 | interrelated things.
2529 |
2530 | =echo= is to =cat= what =|= is to =xargs=. and =<()= and =>()= are
2531 | able to make static files be dynamic streams. putting =cat= and
2532 | =echo= inside =<()= seem like either a noop, or a leap in what can
2533 | be done there. Still have to figure it out.
2534 |
2535 | <(grep a file.txt) , | xargs , cat, echo
2536 | | you-have\it-wants | executable | file | stream |
2537 | | executable | X | <(exe) | exe \vert |
2538 | | file | <(cat file) | X | cat file \vert |
2539 | | stream | cat | <(grep foo file.txt) | X |
2540 |
2541 |
2542 | - output
2543 |
2544 | Most of those can be tested with and =tee=. Sometimes you would like
2545 | the output to be an output to a file to be extramassaged.
2546 |
2547 | | you-have\it-wants | executable | file | stream |
2548 | | executable | X | >() | |
2549 | | file | | X | |
2550 | | stream | | >(cat) | X |
2551 |
2552 | lnav <(tail -F /my/logfile-that-gets-rotated-or-truncated.log)
2553 | cat <(date)
2554 |
2555 | ** redirects & streams
2556 | - https://catonmat.net/ftp/bash-redirections-cheat-sheet.pdf
2557 | - https://catonmat.net/bash-one-liners-explained-part-three
2558 | - https://github.com/miguelmota/bash-streams-handbook
2559 | - https://www2.dmst.aueb.gr/dds/sw/dgsh/
2560 | - https://wiki.bash-hackers.org/howto/redirection_tutorial
2561 | ** The $0 pattern
2562 | - https://www.reddit.com/r/oilshell/comments/f6of85/four_more_posts_in_shell_the_good_parts/
2563 | ** use git staging area to diff outputs of commands
2564 | - https://chrismorgan.info/blog/make-and-git-diff-test-harness/
2565 | - [[https://news.ycombinator.com/item?id=32896051][Content based change detection with Make]]
2566 | - https://news.ycombinator.com/item?id=40333481
2567 | - https://news.ycombinator.com/item?id=32441602
2568 |
2569 | ** coprocs
2570 | - https://stackoverflow.com/questions/7942632/how-to-extrace-pg-backend-pid-from-postgresql-in-shell-script-and-pass-it-to-ano/8305578#8305578
2571 | - https://unix.stackexchange.com/questions/86270/how-do-you-use-the-command-coproc-in-various-shells
2572 | - https://mbuki-mvuki.org/posts/2021-05-30-memoize-commands-or-bash-functions-with-coprocs/
2573 | * links
2574 | - https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html
2575 | - https://www.gnu.org/software/bash/manual/html_node/
2576 | - https://tldp.org/LDP/abs/html/
2577 | - https://mywiki.wooledge.org/BashPitfalls
2578 | - [[https://www.youtube.com/watch?v=sCZJblyT_XM][Gary Bernhardt. The Unix Chainsaw]]
2579 | - https://github.com/spencertipping/. This guy has some bash sick snippets
2580 | - https://news.ycombinator.com/item?id=23765123
2581 | - https://medium.com/@joydeepubuntu/functional-programming-in-bash-145b6db336b7
2582 | - https://www.youtube.com/watch?v=yD2ekOEP9sU
2583 | - http://catern.com/posts/pipes.html
2584 | - https://ebzzry.io/en/zsh-tips-1/
2585 | - https://github.com/ssledz/bash-fun
2586 | - https://news.ycombinator.com/item?id=24556022
2587 | - https://www.datafix.com.au/BASHing/index.html
2588 | - https://susam.github.io/tucl/the-unix-command-language.html
2589 | - https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
2590 | - https://www.grymoire.com/Unix/Sh.html
2591 | - https://github.com/dylanaraps/pure-sh-bible
2592 | - https://github.com/dylanaraps/pure-bash-bible
2593 | - https://shatterealm.netlify.app/programming/2021_01_02_shiv_lets_build_a_vcs
2594 | - https://news.ycombinator.com/item?id=24401085
2595 | - https://git.sr.ht/~sircmpwn/shit
2596 | - [[https://github.com/p8952/bocker][bocker]]. Docker implemented in around 100 lines of bash.
2597 | - https://github.com/simplenetes-io/simplenetes wow
2598 | - https://bakkenbaeck.github.io/a-random-walk-through-git/
2599 | - https://github.com/WeilerWebServices/Bash
2600 | - https://www.netmeister.org/blog/consistent-tools.html.
2601 | - https://www.arp242.net/why-zsh.html. bash vs zsh differences.
2602 | - https://dwheeler.com/essays/filenames-in-shell.html
2603 | - https://lacker.io/math/2022/02/24/godels-incompleteness-in-bash.html
2604 | - https://www.evalapply.org/posts/shell-aint-a-bad-place-to-fp-part-1-doug-mcilroys-pipeline/ (from [[https://clojurians.slack.com/archives/C03RZGPG3/p1645563940203839][clojurians slack]])
2605 | - https://github.com/adityaathalye
2606 | - https://www.youtube.com/watch?v=BJ0uHhBkzOQ
2607 | - https://www.gnu.org/software/autoconf/manual/autoconf-2.60/html_node/Portable-Shell.html
2608 |
2609 | * From shell to lisp and everything in between
2610 | - [[https://github.com/oilshell/oil][Oil Shell]].
2611 | - [[https://rash-lang.org/][Rash]] (Racket shell)
2612 | - [[https://arxiv.org/pdf/2007.09436.pdf][PaSh]]: Light-touch Data-Parallel Shell Processing.
2613 | - [[https://www.eigenbahn.com/2020/07/08/painless-emacs-remote-shells][Painless emacs remote shells]]. Because emacs has you covered
2614 | - https://news.ycombinator.com/item?id=24249646 rust
2615 | - https://github.com/liljencrantz/crush
2616 | - https://github.com/artyom-poptsov/metabash
2617 | - https://www.nushell.sh/
2618 | - [[https://github.com/borkdude/babashka][Babashka]]
2619 | - Bash to Perl/Python/Ruby using =``= and growing from there.
2620 | * Credits
2621 | - Raimon Grau <[[mailto:raimonster@gmail.com][raimonster@gmail.com]]>.
2622 | - Some examples are result of Raimon's and Lluís Esquerda's
2623 | conversations or real world examples.
2624 | - people in https://news.ycombinator.com/item?id=24402571 which I'll
2625 | be pulling in as time allows.
2626 |
--------------------------------------------------------------------------------
/readme.org:
--------------------------------------------------------------------------------
1 | * https://kidd.github.io/scripting-field-guide/index.html
2 |
--------------------------------------------------------------------------------