├── LICENSE └── README.md /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 Thomas Güttler 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Bash Strict Mode 2 | 3 | ## Introduction 4 | 5 | In this text, I explain how I use the Bash shell. Of course, there are several other ways to use 6 | Bash; this is my personal point of view. 7 | 8 | If you think there is something wrong or could be improved, please create an issue in this GitHub 9 | project. Thank you! 10 | 11 | This text contains two parts: Bash Strict Mode and General Hints and Opinions. 12 | 13 | ## Part 1: Bash Strict Mode 14 | 15 | Bash strict mode refers to a set of options and practices used in Bash scripting to make scripts 16 | more robust, reliable, and easier to debug. By enabling strict mode, you can prevent common 17 | scripting errors, detect issues early, and make your scripts fail in a controlled way when something 18 | unexpected happens. 19 | 20 | ## Bash Strict Mode: Activating It 21 | 22 | I use this at the top of my Bash scripts: 23 | 24 | ```bash 25 | #!/usr/bin/env bash 26 | # Bash Strict Mode: https://github.com/guettli/bash-strict-mode 27 | trap 'echo -e "\n🤷 🚨 🔥 Warning: A command has failed. Exiting the script. Line was ($0:$LINENO): $(sed -n "${LINENO}p" "$0" 2>/dev/null || true) 🔥 🚨 🤷 "; exit 3' ERR 28 | set -Eeuo pipefail 29 | ``` 30 | 31 | Let's have a closer look: 32 | 33 | ```bash 34 | #!/usr/bin/env bash 35 | ``` 36 | 37 | This makes sure we use the Bash shell and not a different shell. Writing portable shell scripts is 38 | more complicated, and I want to get things done, so I use Bash and its handy features. 39 | 40 | The command `/usr/bin/env` looks up `bash` in `$PATH`. This is handy if `/bin/bash` is outdated on 41 | your system, and you installed a new version in your home directory. 42 | 43 | --- 44 | 45 | This line prints a warning if the shell script terminates because a command returned a non-zero exit 46 | code: 47 | 48 | ```bash 49 | trap 'echo "Warning: A command has failed. Exiting the script. Line was ($0:$LINENO): $(sed -n "${LINENO}p" "$0")"; exit 3' ERR 50 | ``` 51 | 52 | It shows the line that caused the shell script to exit and exits with `3`. 53 | 54 | Why `3`? I use that according to the [Nagios Plugin Return 55 | Codes](https://nagios-plugins.org/doc/guidelines.html), although I don't use Nagios anymore. `3` 56 | means "Unknown." 57 | 58 | --- 59 | 60 | ```bash 61 | set -Eeuo pipefail 62 | ``` 63 | 64 | `-E`: **ERR Trap Inheritance** Ensures that the ERR trap is inherited by shell functions, command 65 | substitutions, and subshells. 66 | 67 | `-e`: **Exit on Error** Causes the script to immediately exit if any command returns a non-zero exit 68 | status unless that command is followed by `||` to explicitly handle the error. 69 | 70 | `-u`: **Undefined Variables** Treats the use of undefined variables as an error, causing the script 71 | to exit. 72 | 73 | `-o pipefail`: **Pipeline Failure** Ensures that a pipeline (a series of commands connected by `|`) 74 | fails if any command within it fails, rather than only failing if the last command fails. 75 | 76 | ## Bash Strict Mode: Errors Should Never Pass Silently 77 | 78 | Quoting the [Zen of Python](https://peps.python.org/pep-0020/): 79 | 80 | > Errors should never pass silently. Unless explicitly silenced. 81 | 82 | I think the above strict mode ensures that errors don’t go unnoticed and prevents scripts from 83 | running into unexpected issues. I prefer a command to fail and show me the failed line, rather than 84 | the default behavior of Bash (continuing with the next line in the script). 85 | 86 | ## Bash Strict Mode: Simple Example 87 | 88 | Imagine you have a simple script: 89 | 90 | ```bash 91 | grep foo bar.txt >out.txt 92 | echo "all is fine" 93 | ``` 94 | 95 | The script expects a file called `bar.txt`. But what happens if that file does not exist? 96 | 97 | If the file does not exist, you get this output (without strict mode): 98 | 99 | ```terminal 100 | ❯ ~/tmp/t.sh 101 | grep: bar.txt: No such file or directory 102 | all is fine 103 | ``` 104 | 105 | The script terminates with a zero (meaning "OK") exit status, even though something went wrong. 106 | 107 | That's something I would like to avoid. 108 | 109 | With strict mode enabled, you will get: 110 | 111 | ```terminal 112 | grep: bar.txt: No such file or directory 113 | Warning: A command has failed. Exiting the script. Line was (/home/user/tmp/t.sh:5): grep foo bar.txt >out.txt 114 | ``` 115 | 116 | And the exit status of the script will be `3`, which indicates an error. 117 | 118 | ## Bash Strict Mode: Use It Blindly? 119 | 120 | If you post about `set -e` on the Bash subreddit, you get an automated comment like this: 121 | 122 | [Don't blindly use set -euo 123 | pipefail.](https://www.reddit.com/r/commandline/comments/g1vsxk/comment/fniifmk/) 124 | 125 | The link explains why you should not use `set -Eeuo pipefail` everywhere. 126 | 127 | I disagree. Strict mode has consequences, and dealing with these consequences requires some extra 128 | typing. But typing is not the bottleneck. I prefer to type a bit more if it results in more reliable 129 | Bash scripts. 130 | 131 | ## Bash Strict Mode: Handle Unset Variables 132 | 133 | This would fail in strict mode if `FOO` is not set: 134 | 135 | ```bash 136 | if [ -z "$FOO" ]; then 137 | echo "Env var FOO is not set. Doing completely different things now ..." 138 | do_different_things 139 | fi 140 | ``` 141 | 142 | Output: 143 | 144 | > line N: FOO: unbound variable 145 | 146 | You can work around that easily by setting the value to the empty string: 147 | 148 | ```bash 149 | if [ -z "${FOO:-}" ]; then 150 | echo "Env var FOO is not set. Doing completely different things now ..." 151 | do_different_things 152 | fi 153 | ``` 154 | 155 | ## Bash Strict Mode: Handle Non-Zero Exit Codes 156 | 157 | Non-zero exit codes often indicate an error, but not always. 158 | 159 | The command `grep` returns `0` if a line matches, otherwise `1`. 160 | 161 | For example, you want to filter comments into a file: 162 | 163 | ```bash 164 | echo -e "foo\n#comment\nbar" | grep '^#' >comments.txt 165 | ``` 166 | 167 | The code above works in strict mode because there is a match. But it fails if there is no comment. 168 | 169 | In that case, I expect `comments.txt` to be an empty file, and the script should not fail but 170 | continue to the next line. 171 | 172 | This code fails in strict mode: 173 | 174 | ```bash 175 | echo -e "foo\nbar" | grep '^#' >comments.txt | some-other-command 176 | ``` 177 | 178 | Workaround: 179 | 180 | ```bash 181 | echo -e "foo\nbar" | { grep '^#' >comments.txt || true; } | some-other-command 182 | ``` 183 | 184 | With this pattern, you can easily ignore non-zero exit statuses. 185 | 186 | ## Bash Strict Mode: Exit Status 187 | 188 | In most cases you just want to know: Was the command successful or not? 189 | 190 | If you want to know the exit code (`$?`) then you can use that pattern: 191 | 192 | ```bash 193 | if some-command; then 194 | code=0 195 | else 196 | code=$? 197 | fi 198 | echo $code 199 | ``` 200 | 201 | ## Bash Strict Mode: Avoid `&&` 202 | 203 | When using the Bash Strict Mode, you want commands which fail (exit status not 0) to fail. Execution 204 | of the script should be stopped. 205 | 206 | Imagine there are two commands combined with `&&`. The non-zero exit status does not get noticed. 207 | 208 | (`false` is a command which has exit status 1) 209 | 210 | ```bash 211 | false && false 212 | ``` 213 | 214 | To avoid that pitfall, avoid `&&` and use two lines instead: 215 | 216 | ```bash 217 | false 218 | false 219 | ``` 220 | 221 | If you use two lines, then Bash in strict mode will fail on the first non-zero exit status (here the 222 | first `false`): 223 | 224 | ## Bash Strict Mode: `head` Can Give You a Headache 225 | 226 | This will fail: 227 | 228 | ```bash 229 | random_id=$(tr -dc 'a-z0-9' /dev/null || true) 🔥 🚨 🤷 "; exit 3' ERR 322 | set -Eeuo pipefail 323 | 324 | { 325 | echo task 1 326 | sleep 1 327 | } & task1_pid=$! 328 | 329 | { 330 | echo task 2 331 | sleep 2 332 | } & task2_pid=$! 333 | 334 | # Wait each PID on its own line so you get each child's exit status. 335 | wait "$task1_pid" 336 | wait "$task2_pid" 337 | 338 | echo end 339 | ``` 340 | 341 | Why wait each PID separately? 342 | 343 | - You must wait to reap background children and avoid zombies. 344 | - `wait pid1 pid2` will wait for both PIDs, but its exit status is the exit status of the last PID 345 | waited for. This means an earlier background job can fail yet the combined `wait` can still return 346 | success if the last job succeeds — not what you want if you need to detect failures reliably. 347 | 348 | ## Makefiles 349 | 350 | Makefiles are similar to the strict mode. Let's look at an example: 351 | 352 | ```makefile 353 | target: prerequisites 354 | recipe-command-1 355 | recipe-command-2 356 | recipe-command-3 357 | ``` 358 | 359 | If `recipe-command-1` fails the Makefile stops, and does not execute `recipe-command-2`. 360 | 361 | The syntax in a Makefile looks like shell, but it is not. 362 | 363 | As soon as the commands in a Makefile get complicated, I recommend to keep it simple: 364 | 365 | ```makefile 366 | target: prerequisites 367 | script-in-bash-strict-mode.sh 368 | ``` 369 | 370 | Instead of trying to understand the syntax of Makefile (for example `$(shell ...)`), I recommend to 371 | call a Bash script. 372 | 373 | A Bash script has the benefit that formatting (shfmt) and ShellCheck are available in the editor. 374 | 375 | ## Perl Compatible Regular Expressions: `grep -P` 376 | 377 | Unfortunately there are several different flavours of regular expressions. 378 | 379 | Instead of learning the old regular expressions, I recommend to use the Perl Compatible Regular 380 | Expressions. 381 | 382 | The good news: `grep` supports PCRE with the `-P` flag. I suggest to use it. 383 | 384 | ## I Don't Use `awk` 385 | 386 | I avoid using `awk` because I am not familiar with its syntax, and from 1996 up to now, this has 387 | worked out fine for me. 388 | 389 | The only time I use `awk` is when the input is split by whitespace and the length varies. 390 | 391 | Example: I want to print the second column: 392 | 393 | ```bash 394 | command-which-prints-columns | awk '{print $2}' 395 | ``` 396 | 397 | From time to time, I use `perl` one-liners. 398 | 399 | ## Shell vs Bash 400 | 401 | I think writing portable shell scripts is unnecessary in most cases. It is like trying to write a 402 | script that works in both, the Python and the Ruby interpreters at the same time. Don't do it. Be 403 | explicit and write a Bash script (not a shell script). 404 | 405 | ## shfmt 406 | 407 | There is a handy shell formatter: [shfmt](https://github.com/mvdan/sh#shfmt) and a [VS Code plugin 408 | shell-format](https://marketplace.visualstudio.com/items?itemName=foxundermoon.shell-format). 409 | 410 | ## ShellCheck 411 | 412 | There is [ShellCheck](https://github.com/koalaman/shellcheck) and a [VS Code plugin for 413 | ShellCheck](https://marketplace.visualstudio.com/items?itemName=timonwong.shellcheck) which helps 414 | you find errors in your script. 415 | 416 | ShellCheck can recognize several types of incorrect quoting. It warns you about every unquoted 417 | variable. Since it is not much work, I follow ShellCheck's recommendations. 418 | 419 | ## Provisioning a machine: Fancy Tools or boring Bash? 420 | 421 | There are several well known tools for provisioning a machine: Ansible, SaltStack, Puppet, Chef, ... 422 | 423 | All of them have their learning costs. 424 | 425 | It depends on the environment, but maybe a Bash script in strict mode is easier to maintain. 426 | 427 | Some years ago, we used SaltStack to provision and update a lot of virtual machines. We wasted so 428 | much time because things did not work as expected, or error messages got swallowed. In hindsight, we 429 | would have been much faster if we had taken the pragmatic approach (Bash) instead of being proud to 430 | use the same tools as big tech companies. 431 | 432 | ## Avoid Fancy CI or GitHub Actions 433 | 434 | GitHub Actions (or similar CI tools) have a big drawback: You can't execute them on your local 435 | device. 436 | 437 | I try to keep our GitHub Actions simple: the YAML config calls Bash scripts, which I can also 438 | execute locally. 439 | 440 | You can use containers to ensure that all developers have the same environment. 441 | 442 | ## Interactive Shell 443 | 444 | This article is about Bash scripting. 445 | 446 | For **interactive** I use: 447 | 448 | - [Fish Shell](https://fishshell.com/) 449 | - [Starship](https://starship.rs/) for the prompt. 450 | - [Atuin](https://github.com/atuinsh/atuin) for the shell history. 451 | - [direnv](https://direnv.net/) to set directory specific env variables. 452 | - [brew](https://brew.sh/) 453 | - [ripgrep](https://github.com/BurntSushi/ripgrep) 454 | - [fd find](https://github.com/sharkdp/fd) 455 | - [CopyQ](https://hluk.github.io/CopyQ/) Clipboard Manager 456 | - [Activity Watch](https://activitywatch.net/) Automatic time tracker 457 | - VSCode 458 | - Ubuntu LTS. 459 | 460 | Usually don't use `ripgrep` and `fd` in Bash scripts, because these are not available on most 461 | systems. 462 | 463 | ## `set -x` Can Reveal Credentials 464 | 465 | Imagine you use credentials like this: 466 | 467 | ```bash 468 | echo "$OCI_TOKEN" | oras manifest fetch --password-stdin $IMAGE_URL 469 | ``` 470 | 471 | If you use `set -x`, then every line gets printed. This will print the **content** of $OCI_TOKEN. 472 | 473 | This can reveal your secrets in the logs. 474 | 475 | Rule of thumb: Never use `set -x` in a script. Except temporarily for debugging, but do not commit 476 | it to the source code repo. 477 | 478 | ## /r/bash 479 | 480 | Thank you to [https://www.reddit.com/r/bash/](https://www.reddit.com/r/bash/) 481 | 482 | I got several good hints there. 483 | 484 | ## More 485 | 486 | [Thomas WOL: Working out Loud](https://github.com/guettli/wol) 487 | --------------------------------------------------------------------------------