├── LICENSE └── README.md /LICENSE: -------------------------------------------------------------------------------- 1 | This is free and unencumbered software released into the public domain. 2 | 3 | Anyone is free to copy, modify, publish, use, compile, sell, or 4 | distribute this software, either in source code form or as a compiled 5 | binary, for any purpose, commercial or non-commercial, and by any 6 | means. 7 | 8 | In jurisdictions that recognize copyright laws, the author or authors 9 | of this software dedicate any and all copyright interest in the 10 | software to the public domain. We make this dedication for the benefit 11 | of the public at large and to the detriment of our heirs and 12 | successors. We intend this dedication to be an overt act of 13 | relinquishment in perpetuity of all present and future rights to this 14 | software under copyright law. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 | IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 20 | OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 21 | ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 22 | OTHER DEALINGS IN THE SOFTWARE. 23 | 24 | For more information, please refer to 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | - [1. Introduction](#1-introduction) 2 | - [1.1. Scope](#11-scope) 3 | - [1.2. Background](#12-background) 4 | - [1.3. Purpose](#13-purpose) 5 | - [1.4. Next steps](#14-next-steps) 6 | - [2. Basics](#2-basics) 7 | - [2.1. Common commands](#21-common-commands) 8 | - [2.2. Shortcuts](#22-shortcuts) 9 | - [2.2.1. Navigation](#221-navigation) 10 | - [2.2.2. Editing](#222-editing) 11 | - [2.2.3. Recall from history](#223-recall-from-history) 12 | - [3. Streams, Pipes and Redirects](#3-streams-pipes-and-redirects) 13 | - [3.1. Streams](#31-streams) 14 | - [3.2. Redirections](#32-redirections) 15 | - [3.2.1. Types](#321-types) 16 | - [3.2.2. Additional Examples](#322-additional-examples) 17 | - [3.2.2.1. Send standard output to sout.txt and standard error to serr.txt](#3221-send-standard-output-to-souttxt-and-standard-error-to-serrtxt) 18 | - [3.2.2.2. Send standard output and standard error streams to the same file sone.txt](#3222-send-standard-output-and-standard-error-streams-to-the-same-file-sonetxt) 19 | - [3.2.2.3. Check if a string is present in a shell variable](#3223-check-if-a-string-is-present-in-a-shell-variable) 20 | - [3.2.2.4. Ignore both standard input and output](#3224-ignore-both-standard-input-and-output) 21 | - [3.3. Pipe](#33-pipe) 22 | - [3.4. xargs](#34-xargs) 23 | - [3.5. tee](#35-tee) 24 | - [4. Filename expansion](#4-filename-expansion) 25 | - [5. Brace expansion](#5-brace-expansion) 26 | - [5.1. Examples](#51-examples) 27 | - [6. Job control](#6-job-control) 28 | - [7. Process handling](#7-process-handling) 29 | - [7.1. difference between ```ps``` and ```jobs```](#71-difference-between-ps-and-jobs) 30 | - [8. Quoting](#8-quoting) 31 | - [9. Basic file management](#9-basic-file-management) 32 | - [9.1. list directory ```ls```](#91-list-directory-ls) 33 | - [9.2. show file contents](#92-show-file-contents) 34 | - [9.3. file handling](#93-file-handling) 35 | - [9.3.1. copy](#931-copy) 36 | - [9.3.2. move or rename](#932-move-or-rename) 37 | - [9.3.3. delete](#933-delete) 38 | - [9.3.4. file linking](#934-file-linking) 39 | - [9.3.5. change directory](#935-change-directory) 40 | - [9.3.6. display disk usage](#936-display-disk-usage) 41 | - [9.3.7. file ownership and permissions](#937-file-ownership-and-permissions) 42 | - [10. Special shell variables](#10-special-shell-variables) 43 | - [11. Exit codes](#11-exit-codes) 44 | - [12. ```grep```](#12-grep) 45 | - [12.1. Useful ```grep``` options](#121-useful-grep-options) 46 | - [12.1.1. Examples](#1211-examples) 47 | - [12.2. Regular expression in grep](#122-regular-expression-in-grep) 48 | - [12.3. Examples](#123-examples) 49 | - [13. ```find```](#13-find) 50 | - [13.1. Examples](#131-examples) 51 | - [14. ```sed``` filter and transform text](#14-sed-filter-and-transform-text) 52 | - [14.1. Overview](#141-overview) 53 | - [14.2. Examples](#142-examples) 54 | - [14.3. Grouping](#143-grouping) 55 | - [14.3.1. Grouping Examples:](#1431-grouping-examples) 56 | - [14.4. Hold Buffer](#144-hold-buffer) 57 | - [14.4.1. Example](#1441-example) 58 | - [15. ```awk```](#15-awk) 59 | - [15.1. Actions](#151-actions) 60 | - [15.2. Special variables](#152-special-variables) 61 | - [15.3. Examples](#153-examples) 62 | - [16. Arithmetic expansion](#16-arithmetic-expansion) 63 | - [16.1. Example](#161-example) 64 | - [17. Command substitution](#17-command-substitution) 65 | - [17.1. Examples](#171-examples) 66 | - [18. Process substitution](#18-process-substitution) 67 | - [18.1. Examples:](#181-examples) 68 | - [19. Subshell](#19-subshell) 69 | - [20. Command grouping](#20-command-grouping) 70 | - [20.1. using parentheses](#201-using-parentheses) 71 | - [20.2. using curly braces](#202-using-curly-braces) 72 | - [21. Text editing with ```cut```, ```paste``` and ```join```](#21-text-editing-with-cut-paste-and-join) 73 | - [21.1. ```cut```](#211-cut) 74 | - [21.1.1. Examples](#2111-examples) 75 | - [21.2. ```paste```](#212-paste) 76 | - [21.2.1. Examples](#2121-examples) 77 | - [21.3. ```join```](#213-join) 78 | - [21.3.1. Examples](#2131-examples) 79 | - [22. Aliases](#22-aliases) 80 | - [22.1. useful aliases](#221-useful-aliases) 81 | - [23. Functions](#23-functions) 82 | - [23.1. useful functions](#231-useful-functions) 83 | - [24. sort](#24-sort) 84 | - [25. uniq](#25-uniq) 85 | - [26. Conditions](#26-conditions) 86 | - [26.1. If else](#261-if-else) 87 | - [26.2. Short circuiting](#262-short-circuiting) 88 | - [26.2.1. example](#2621-example) 89 | - [27. Loops](#27-loops) 90 | - [27.1. while loops](#271-while-loops) 91 | - [27.1.1. example](#2711-example) 92 | - [27.2. for loops](#272-for-loops) 93 | - [27.2.1. example](#2721-example) 94 | - [28. ssh](#28-ssh) 95 | - [29. curl](#29-curl) 96 | - [29.1. Examples](#291-examples) 97 | - [29.2. handling REST calls](#292-handling-rest-calls) 98 | - [29.2.1. GET](#2921-get) 99 | - [29.2.2. POST](#2922-post) 100 | - [29.2.3. PUT](#2923-put) 101 | - [29.2.4. DELETE](#2924-delete) 102 | - [30. wget](#30-wget) 103 | - [30.1. Examples](#301-examples) 104 | - [31. One liners](#31-one-liners) 105 | - [32. Further reading](#32-further-reading) 106 | - [33. Change History](#33-change-history) 107 | 108 | 109 | # 1. Introduction 110 | ## 1.1. Scope 111 | - This document will mainly focus on things that are usually done in the command line 112 | - The focus is on tools and techniques for one shot adhoc tasks. 113 | - Most of the things in the document can be applied to shell scripting. However, shell scripting is not the focus of this document 114 | - This document is for Linux/Unix 115 | 116 | ## 1.2. Background 117 | - I am documenting things I learnt and am still learning 118 | - I had multiple files with random snippets. 119 | - I decided to organize the mess into a single piece, so that I can efficiently look up things 120 | - I cannot claim to be an expert in command line. 121 | 122 | ## 1.3. Purpose 123 | - The document will not help anyone become an expert. 124 | - However, it may help people become effective. 125 | - I have kept the document concise. I skipped over details that I never used much and can be looked up later in the internet. 126 | - I have focused on concrete examples of common or interesting use cases. 127 | 128 | ## 1.4. Next steps 129 | - rephrase unclear material 130 | - correct incorrect material 131 | - add missing material 132 | - add further reading material 133 | - Contributions are welcome 134 | 135 | # 2. Basics 136 | ## 2.1. Common commands 137 | 138 | command | description 139 | -- | -- 140 | ```man``` | get help on commands. e.g.
```man date``` 141 | ```date``` | gives the date and time 142 | ```cal``` | shows the calender 143 | ```ls``` | tells which files are present in the current working directory
when a *-l* option is used, the command returns owner, size, date of file, persissions, etc. 144 | ```cat``` | shows contents of a file 145 | ```cp``` | copy a file 146 | ```mv``` | move a file 147 | ```diff``` | lists differences between 2 files 148 | ```rm``` | removes file 149 | ```grep``` | find occurences of strings in one or more files 150 | ```pwd``` | print the present working directory's name 151 | ```cd``` | changes the current directory 152 | 153 | ## 2.2. Shortcuts 154 | 155 | ### 2.2.1. Navigation 156 | keyboard shortcut | description 157 | --- |--- 158 | Ctrl+A | move the cursor to the start of the line 159 | Ctrl+E | move the cursor to the end of the line 160 | Alt+F | move the cursor forward by one word 161 | Ctrl+F | move the cursor forward by one character 162 | Alt+B | move the cursor backward by one word 163 | Ctrl+B | move the cursor backward by one charater 164 | Ctrl+XX | toggle the cursor's position between the current position and the previous position 165 | 166 | ### 2.2.2. Editing 167 | keyboard shortcut | description 168 | --- |--- 169 | Ctrl+U | cut all the characters to the left of the cursor 170 | Ctrl+K | cut all the characters to the right of the cursor 171 | Ctrl+W | cut one word to the left of the cursor 172 | Ctrl+H | cut one character to the left of the cursor 173 | Alt+D | cut one word to the right of the cursor 174 | Ctrl+D | cut one character to the right of the cursor 175 | Ctrl+Y | paste the cut characters 176 | Ctrl+_ | undo the last deletion 177 | Tab | complete arguments or list all available commands 178 | 179 | ### 2.2.3. Recall from history 180 | keyboard shortcut | description 181 | --- |--- 182 | Crtl+R | search the command history 183 | Ctrl+G | abort 184 | Ctrl+P/UP | the previous command in the history 185 | Ctrl+N/DOWN | the next command in the history 186 | 187 | # 3. Streams, Pipes and Redirects 188 | ## 3.1. Streams 189 | A linux shell's inputs and outputs are sequence of characters called streams. There are three standard I/O streams: 190 | 191 | name | description |file descriptor 192 | --- |--- |--- 193 | stdout| displays output from commands | 1 194 | stderr| displays error from commands | 2 195 | stdin | provides inputs to commands | 0 196 | 197 | ## 3.2. Redirections 198 | 199 | ### 3.2.1. Types 200 | Input and output redirections are done using angular brackets (<>) 201 | 202 | Bracket type | description 203 | --- | --- 204 | \> | send stream to a file. E.g.
```ls a > o.txt``` 205 | \>> | append stream to a file. E.g.
```ls b >> o.txt``` 206 | \>& | write into stream. E.g.
```ls c > o2.txt 2>&1``` 207 | < | receive stream from a file. E.g.
```wc < o.txt``` 208 | << | embed the text that will be fed to a command within the script
Example
```cat << EOF > output.txt```
```line 1```
```line 2```
```line 3```
```EOF```
```echo done```
also known as heredoc 209 | <<< | expand variable and feed into the stdin of the command
Example
```wc <<< $word```
also known as herestring 210 | 211 | ### 3.2.2. Additional Examples 212 | #### 3.2.2.1. Send standard output to sout.txt and standard error to serr.txt 213 | ```command1 > sout.txt 2> serr.txt``` 214 | #### 3.2.2.2. Send standard output and standard error streams to the same file sone.txt 215 | ```command1 > sone.txt 2>&1``` 216 | 217 | OR 218 | 219 | ```command1 &> sone.txt``` 220 | 221 | #### 3.2.2.3. Check if a string is present in a shell variable 222 | ```grep "ABC" <<< $var``` 223 | 224 | #### 3.2.2.4. Ignore both standard input and output 225 | ```command1 &> /dev/null``` 226 | 227 | ***/dev/null** is a null device file. This will discard anything written to it, and will return EOF on reading.* 228 | 229 | ## 3.3. Pipe 230 | Piping can redirect the standard output of one command to the input of another command. 231 | 232 | ```command1 | command2 paramater1 | command3 parameter1 parameter2 | command4``` 233 | 234 | Examples: 235 | 236 | - Print file contents only once. remove duplicate records:
```sort file1 | uniq``` 237 | 238 | - print 5 most frequently used commands
239 | ```history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head -5``` 240 | 241 | - print the file types and their frequencies
```ls | rev | cut -f1 -d'.' | rev | sort | uniq -c | sort -n``` 242 | 243 | ## 3.4. xargs 244 | - short for extended arguments 245 | - some commands can take arguments from both standard input and as command-line arguments. 246 | - however, there are some commands that cannot take input from standard input. They accept inputs only from arguments. For these commands we need to use args. 247 | - xargs converts input from standard input into arguments to a command. 248 | - divides the arguments to a permitted number and runs the command repeatedly over each greoup of arguments. 249 | - using -n option, the number of arguments per command can be specified. 250 | ```find . | xargs -n1 basename``` 251 | - in order to assign the std input to a placeholder , use ```-I{}```. the placeholder is usually used when we want to place the std input in the middle of a command
252 | ```ls | xargs -I{} echo {} file is found``` 253 | - xargs, by default, divides the input into arguments using whitespace. When -0 option is used, xargs uses null termination to identify arguments. 254 | 255 | 256 | Examples: 257 | - print the number lines/words/characters in the files in a directory
258 | ```ls | wc``` 259 | - print the file types and their frequencies
260 | ```find . -type f | xargs basename -a | grep "\." | rev | cut -f1 -d'.' | rev | sort | uniq -c | sort -n``` 261 | - rename all files in a directory
262 | ```ls | xargs -I{} mv {} {}.bkp``` 263 | - copy all .c files to a specific directory
264 | ```find . -name "*.c" -print0 | xargs -0 -n1 -I{} cp {} some/folder/``` 265 |
*The print0 option of the find command prints the file names with the null character as termination character. By using the -0 option with xargs, it starts looking for the null character for reading filenames from the stdin.* 266 | 267 | ## 3.5. tee 268 | - the command reads from standard input and writes it to a standard output and to one or more files 269 | - this is useful when we want to see the output of a command both on the screen and also want to save the output in a file for later analysis. 270 | 271 | Examples: 272 | - ```ls | tee fileList.txt``` 273 | 274 | # 4. Filename expansion 275 | - a wildcard character is a character that is used to represent one or more characters in a filename or foldername. 276 | - file globbing is the operation that recognizes these wildcard characters and does the expansion. 277 | 278 | wildcard | description | example | matches | does not match 279 | --- | --- | --- | --- | --- 280 | \* | matches 0 or more characters | ls to* | to, tom, ton, tow, tommy, tommie | tata, tea 281 | ? | matches 1 character | ls to? | tom, tow, ton | to, tommy, tommie, tata, tea 282 | [abc] | matches any one of the characters in the brackets | ls [bc]at | bat, cat | Bat, Cat, rat 283 | [a-z] | matches any one of the characters within the range in the brackets | ls day[1-9] | day1, day2 upto day9 | day11, day 284 | [!abc] | matches any one character that is not in the brackets | ls [!r]at | bat, cat, Bat, Cat, Rat | rat 285 | [!a-z] | matches any one character that is not in the range in the brackets | ls day[!1-9] | day0, days | day1 upto day9 286 | 287 | # 5. Brace expansion 288 | - it is generation of a range of strings to be used on the command line 289 | - useful for generating a list of sequential data or running any command on sequence of data. 290 | - a range or comma-separated list of data can be used for expansion. They are enclosed within curly braces. 291 | - example: 292 | - {0..4} generates a sequence from 0 to 4 293 | - {alice, bob, charlie} generates alice, bob and charlie 294 | - the list can be preceded by an optional string (called preamble or prefix). this is prefixed to each string in the braces 295 | - the list can be followed by an optional string (called postscript or postfix). this is appended to each string in the braces 296 | - syntax - 297 | - string list
```{String1, String2}``` 298 | - range list
```{..}``` 299 | - range list with custom increment
```{....}``` 300 | - ```{ string or range }``` 301 | 302 | ## 5.1. Examples 303 | - create files for each day of month January
```touch {1..31}``` 304 | - print all even numbers from 1 to 100
```echo {2..100..2}``` 305 | - create input, output, backup and error folders
```mkdir /some/path/{input, output, backup, error}``` 306 | - print the alphabets in reverse order
```echo {z..a}``` 307 | - create a backup of a file
```cp file.txt{,.bkp}``` 308 | - restore from .bkp file
```mv file.txt{.bkp,}``` 309 | - create folder for each day of January, February and March
```mkdir -p ~/logs/{January/{1..31},February/{1..28},March/{1..31}}``` 310 | 311 | # 6. Job control 312 | - command line provides the ability to stop/suspend the execution of a process and resume a suspended process at a later point in time 313 | - each running program is called job 314 | - unique id is assigned to every job 315 | 316 | command | description 317 | --- | --- 318 | ```jobs``` | list all the jobs that the current shell is running or has suspended 319 | ```fg``` | bring job to foreground 320 | ```bg``` | send job to background 321 | ```kill``` | terminate job 322 | ```stop``` | suspend job 323 | Ctrl+c | terminate job 324 | Ctrl+z | suspend job 325 | 326 | - to start a job in the background, we need to add an ampersand(&) at the end of the command 327 | 328 | Examples: 329 | - start a job in the background
```sleep 1000 &``` 330 | - bring the job to the foreground
```fg %2 # where 2 is the job number``` 331 | - suspend a job
```stop %2 # where 2 is the job number``` 332 | - resume a job in the background
```bg %2 # where 2 is the job number``` 333 | - terminate a job
```kill %2 # where 2 is the job number``` 334 | 335 | # 7. Process handling 336 | 337 | - Get a snapshot of processes running with ```ps``` command 338 | - Get the process id of a command by ```ps -ef | grep command``` 339 | - wait for a process to finish by using ```bash wait ``` 340 | - kill a process by using ``` kill ``` 341 | - wait for completion of all child processes
```wait``` 342 | - wait for completion of specific process
```wait 1234 # where 1234 is the process id``` 343 | 344 | 345 | ## 7.1. difference between ```ps``` and ```jobs``` 346 | - ```jobs``` tells the list of jobs the current shell is managing 347 | - ```ps``` tells the list of all the processes running in the system 348 | 349 | 350 | # 8. Quoting 351 | - Quoting is used to remove special meanings from characters or words 352 | - single quotes - when the single quotes are used, every character within the quotes is preserved and is not evaluated 353 | - double quotes - when the double qoutes are used, the dollar sign, back quotes and blackslashes are evaluated and interpreted. 354 | - escape character - \ is used to preserve the literal value of the following character. 355 | - ANSI C quoting - backslash escaped characters are treated according to ANSI C standard. Format:- ```$'string``` 356 | 357 | 358 | Examples: 359 | examples | command | result 360 | --- | --- | --- 361 | no quote | ```echo $HOME``` | /home/user1/ 362 | escape character | ```echo \$HOME``` | $HOME 363 | single quote | ```echo '$HOME'``` | $HOME 364 | double quote | ```echo "$HOME"``` | /home/user1/ 365 | ANSI C quoting | ```echo $'There\'s a quote in my string!'``` | There's a quote in my string 366 | 367 | # 9. Basic file management 368 | 369 | ## 9.1. list directory ```ls``` 370 | command | description 371 | --- | --- 372 | ```ls``` | list the contents of current directory 373 | ```ls *``` | list contents of the directory along with the subdirectory 374 | ```ls -l``` | list the contents of a directory along with the owner, permission, date, size 375 | ```ls -a``` | list hidden file 376 | ```ls -t``` | list files in descending order of last modified date 377 | ```ls -rt``` | list files in ascending order of last modified date 378 | ```ls -R``` | list files of the current directory and the subdirectory recursively to the last subdirectory 379 | ```ls /path/to/dirextory``` | list files in the directory mentioned 380 | 381 | 382 | ## 9.2. show file contents 383 | command | description 384 | --- | --- 385 | ```cat demo.txt``` | show the contents of file. use for relatively small files 386 | ```head demo.txt``` | show the first part of the file 387 | ```tail demo.txt``` | show the last part of the file 388 | ```tail -f demo.txt``` | show the text appended to the file as the file grows 389 | ```less demo.txt``` | show contents of file one screen at a time 390 | ```less -p "regular" demo.txt``` | show contents of the file from the first line with which the pattern matches 391 | ```strings -a binaryfile``` | print the sequence of all the printable characters in the file 392 | ```diff file1 file2``` | shows difference between 2 files 393 | ```comm file1 file2``` | compare 2 sorted file 394 | 395 | ## 9.3. file handling 396 | 397 | ### 9.3.1. copy 398 | command | description 399 | --- | --- 400 | ```cp file1 file2``` | copy file1 to file2 401 | ```cp file1 file2 directory1``` | copy file1 and file2 to directory1 402 | ```cp -R directory1 directory2``` | copy contents of directory1 to directory2 403 | ```cp *.txt directory1``` | copy all files ending with .txt to directory1 404 | 405 | ### 9.3.2. move or rename 406 | command | description 407 | --- | --- 408 | ```mv file1 file2``` | rename file1 to file2 409 | ```mv file1 directory1/``` | move file1 to directory1 410 | ```mv *.jpg directory1/``` | move all files ending with .jpg to directory1 411 | 412 | ### 9.3.3. delete 413 | command | description 414 | --- | --- 415 | ```rm file1``` | remove file1 416 | ```rm file1 file2``` | remove file1 and file2 417 | ```rm *.png``` | remove all files ending with .png 418 | ```rm -d emptyDirectory``` | remove an empty directory 419 | ```rm -r directory1``` | recursively remove all files, subdirectories and directory of directory1 420 | 421 | ### 9.3.4. file linking 422 | - a symbolic link is a file that points to another file or directory 423 | - there are 2 types of links 424 | - **Hard link** -> it is an additional name for the existing file. Each file is associated with an unique number. This unique number is called inode. Hardlink associate 2 or more filenames to the same inode and in turn the same file. If the original file is removed, the contents are still available via the hardlink 425 | - **Soft link** -> it is an indirect pointer to a file or directory. It has only the path of the original file and not its contents 426 | 427 | command | description 428 | --- | --- 429 | ```ln file1 link1``` | create a hardlink link1 to file file1 430 | ```ln -s file1 link1``` | create a softlink link1 to file file1 431 | 432 | ### 9.3.5. change directory 433 | 434 | command | description 435 | --- | --- 436 | ```cd``` | Change to home directory 437 | ```cd ~``` | Change to home directory 438 | ```cd -``` | Change to the previous directory 439 | ```cd ..``` | Change to the parent directory 440 | ```cd /``` | Change to the root directory 441 | ```cd ~/dir1/dir2``` | Change to directory relative to home directory 442 | 443 | ### 9.3.6. display disk usage 444 | 445 | command | description 446 | --- | --- 447 | ```du``` | estimates and displays the disk space used by the files 448 | ```du -h *``` | prints size in human readable' 449 | 450 | ### 9.3.7. file ownership and permissions 451 | 452 | command | description 453 | --- | --- 454 | ```chmod``` | change permissions 455 | ```chown``` | change ownership 456 | 457 | # 10. Special shell variables 458 | 459 | - there are special variables that are set internally 460 | - these variables are available to the user 461 | 462 | variable | description 463 | --- | --- 464 | $0 | name of the shell 465 | \$$ | the process id of the current shell 466 | $? | the exit status of the last executed command 467 | $! | the process id of the last background process 468 | $_ | the last argument of the previous command 469 | 470 | 471 | # 11. Exit codes 472 | - also known as return code 473 | - it is the code that is returned by command or process after it finishes 474 | - following are few exit codes and their meaning 475 | 476 | exit code | description 477 | --- | --- 478 | 0 | Success 479 | 1 | general error 480 | 126 | command invoked does not have execute permissions 481 | 127 | command not found 482 | 130 | command terminated using Ctrl+C 483 | 484 | 485 | 486 | # 12. ```grep``` 487 | - searches for pattern in files and prints each line that matches the input pattern 488 | ```grep - ``` 489 | - grep can be used to search in a single file or in multiple files 490 | 491 | ## 12.1. Useful ```grep``` options 492 | 493 | option | description 494 | --- | --- 495 | -i | ignore case 496 | -n | display line numbers along with lines 497 | -v | display lines that do not match the pattern 498 | -c | count the number of matching lines 499 | -r | search in all files under each directory 500 | -l | diplay the filename of the file which has the matching pattern 501 | -o | print only the matched string. The whole line with the matched string is not printed 502 | -I | ignore binary files 503 | -A\ | include n lines after match 504 | -B\ | include n lines before match 505 | -C\ | include n lines before and after the match 506 | 507 | ### 12.1.1. Examples 508 | - lets say a demo file(demo.txt) has following content 509 | 510 | > THIS IS UPPER CASE LINE
511 | > this is lower case line
512 | > This is regular line
513 | > This line is also regular
514 | > Line number four
515 | > Line #5
516 | > last line 517 | 518 | - Search for a string in a file
519 | ```grep "this" demo.txt``` 520 | 521 | - Search for a string in a file, without matching case
522 | ```grep -i "this" demo.txt``` 523 | 524 | - Search for a string in a file and get both the line number and output
525 | ```grep -n "this" demo.txt``` 526 | 527 | - Get the number of lines matching the searched string
528 | ```grep -c "this" demo.txt``` 529 | 530 | - Get the filename in which the searched string is found
531 | ```grep -l "this" demo.txt``` 532 | 533 | - Get 2 lines after the matching line
534 | ```grep -A2 "This" demo.txt``` 535 | 536 | - Get 2 line before the matching line
537 | ```grep -B2 "This" demo.txt``` 538 | 539 | - Get 2 lines before and after the matching line
540 | ```grep -C2 "This" demo.txt``` 541 | 542 | - Search recursively in all files in subdirectories
543 | ```grep -inrI 'some text' /path/to/dir``` 544 | 545 | ## 12.2. Regular expression in grep 546 | - a regular expression is a sequence of characters that specifies the search pattern in text. 547 | - different characters have special meaning in regular expressions 548 | 549 | character | description 550 | --- | --- 551 | [abc] | matches any one of the characters in the square brackets 552 | [a-d] | matches any one of the characters in the range specified in the square brackets 553 | ^start | matches the pattern only if the pattern is at the start of the line 554 | end$ | matches the pattern only if the pattern is at the end of the line 555 | [^abc] | matches any one character that is NOT present in the square brackets 556 | [^a-d] | matches any one character that is NOT present in the range 557 | . | matches any one character 558 | \* | mathces 0 or more occurences of the preceding character 559 | .* | matches zero or more of any character 560 | 561 | ## 12.3. Examples 562 | 563 | - Match any one character
564 | ```grep "[Tt]his" demo.txt``` 565 | 566 | - Search for line starting with the search pattern
567 | ```grep "^last" demo.txt``` 568 | 569 | - Search for line ending with the search patter
570 | ```grep "regular$" demo.txt``` 571 | 572 | - Search for line with a character in specified range
573 | ```grep "[0-9]" demo.txt``` 574 | 575 | - Search for line without a character in specified range
576 | ```grep "[^0-9]" demo.txt``` 577 | 578 | - Search for a line where the middle characters are not known
579 | ```grep "line.*regular" demo.txt``` 580 | 581 | **Note:** grep can have regular expressions in the search pattern part, and can have wildcards in the files to search section. 582 | 583 | # 13. ```find``` 584 | - the ```find``` command is used to search and locate the list of files and directories 585 | - the list returned will satisfy the conditions used in find command. 586 | - syntax is ``` find [starting point] [expression] ``` 587 | - ```-exec [command]``` can be used to run command on the files located by ```find``` 588 | - by default all the results are newline terminated. -print0 option can be used to make the results null terminated 589 | 590 | ## 13.1. Examples 591 | - find files with specific name
```find . -name demo.txt``` 592 | - find files with a specific pattern
``` find ./Codes -name *.cpp``` 593 | - find directories with specific name
```find . -name Codes -type d``` 594 | - find files with permission as 777 and change permissions to 644
```find . -type f -perm 0777 -print -exec chmod 644 {} \;``` 595 | - find and remove files
```find . -type f -name "*.bkp" -exec rm -f {};``` 596 | 597 | # 14. ```sed``` filter and transform text 598 | ## 14.1. Overview 599 | - looks for pattern and edits them 600 | - works on both files and stdin 601 | - original files is not updated 602 | - results are put into standard output 603 | - syntax is ``` sed 'instructions' file``` 604 | - instruction is of format ``` '[address]command/regex/replace/modifier' ``` 605 | - example - '5,15s/abc/ABC/g' -> this will substitute abc with ABC in lines 5 to 15. 606 | - modifier g specifies that all occurences of abc in line will be replaced with ABC 607 | - when address is used, the lines belonging to the address are examined/modified. -> ```sed '1,100 s/A/a/' file.txt```. this restricts substition to first 100 lines. 608 | - By default all the lines are printed. When '-n' flag is used, this behavior is suppressed. Nothing will be printed unless an explicit request to print is found. 609 | - A pattern can be used as a address -> ```sed -n '/start/,/stop/ p' file.txt``` 610 | - When '!' is used, the command is run outside of the address -> ```sed -n '/match/ !p' file.txt``` 611 | 612 | command | description 613 | --- | --- 614 | d | delete 615 | p | print 616 | s | substitute 617 | q | quit 618 | a | insert a line after pattern 619 | i | insert a line before pattern 620 | c | change a line 621 | y | transform 622 | 623 | ## 14.2. Examples 624 | - remove blank lines
``` sed '/^$/d' file.txt``` 625 | - remove all lines with search string
```sed '/Search/d' file.txt``` 626 | - remove all instances of search string
```sed 's/Search//g' file.txt``` 627 | - print lines with the search string
```sed '/Search/ p' file.txt``` 628 | - substitue a string with another
```sed 's/oldString/newString/g' file.txt``` 629 | - remove trailing spaces
```sed 's/ *$//' file.txt``` 630 | - remove leading spaces
```sed `s/^ *//' file.txt``` 631 | - add spaces to start of everyline
```sed 's/^/ /' file.txt``` 632 | - print the first 10 line
```sed '10 q' file.txt```
```sed -n '1,10 p' file.txt``` 633 | - append line after a pattern
```sed '/pattern/ a add line here' file.txt``` 634 | - insert line before a pattern
```sed '/pattern/ i add line here' file.txt``` 635 | - change a line with a pattern
```sed '/pattern/ c line changed here' file.txt``` 636 | - change a->p, b->q, c->r
```sed 'y/abc/pqr/' file.txt``` 637 | 638 | ## 14.3. Grouping 639 | - sed allows capturing specific parts of text into groups. 640 | - these groups can be manipulated 641 | - group is enclosed within parentheses expression "\(" and "\)" in the search string 642 | - each group is assigned a number. The first group is assigned \1 and so on. 643 | - \1 can be both in pattern string and replacement string 644 | 645 | ### 14.3.1. Grouping Examples: 646 | - switch first and second columns
```sed 's/\([a-z]*\) \([a-z]*\)/\2 \1/' file.txt``` 647 | - print lines which have consecutive duplicate words
```sed -n '/\([a-z][a-z]*\) \1/p' file.txt``` 648 | - remove consecutive duplicate words in a line
```sed 's/\([a-z][a-z]*\) \1/\1/' file.txt``` 649 | 650 | ## 14.4. Hold Buffer 651 | - When sed read text, each line is placed into a temporary space. 652 | - When a new line is read, the old text is replaced by the new line in the temporary space. 653 | - This temporary space is called pattern space. 654 | - Hold buffer is like a long term storage. text can be copied to and from pattern space. 655 | 656 | command | description 657 | --- | --- 658 | x | exchange hold space and pattern space 659 | h | copy pattern buffer into hold space 660 | H | append pattern buffer into hold space 661 | g | copy hold space to pattern space 662 | G | append hold buffer into pattern buffer 663 | 664 | ### 14.4.1. Example 665 | - print one line after and before the pattern match
```sed -n '/999/ !{x;d};/999/ {x;p;x;p;n;p}' file.txt``` 666 | - add space after every line
```sed 'G' file.txt``` 667 | - insert blank line above every line which matches pattern
```sed '/start/ {x;p;x}' file.txt``` 668 | - Insert blank line after every line which matches pattern
```sed '/start/ {G}' file.txt``` 669 | - Insert blank line before and after every line which matches pattern
```sed '/start/ {x;p;x;G}' file.txt``` 670 | 671 | # 15. ```awk``` 672 | - command line utility to find, process and transform text files 673 | - the basic syntax is ```pattern { action }``` 674 | - the pattern is compared with every input line. pattern can be any regular expression 675 | - when the pattern matches, the action is performed 676 | - when no pattern is provided, the action is applied to every line 677 | - example -> ```/^HTTP/ {print}``` 678 | - every line starting with HTTP is printed 679 | - there are 2 other important patterns 680 | - BEGIN - specifies actions to be performed, before any line is read 681 | - END - specifies actions to be performed, after all lines are read 682 | - Example -> ```awk 'BEGIN{print "start"} {print} END{print "end"}' file.txt``` 683 | - first *start* is printed, then all lines of file.txt are printed and then *end* is printed 684 | - awk interprets each line as a record of fields 685 | - one or more consecutive spaces or tabs are considered as a single delimiter between fields 686 | - $1, $2, etc. represents the first field, second field, and so on. 687 | - $0 represents the entire input line 688 | - awk has 2 data types - strings and integers 689 | - awk internally converts variable according to context 690 | - awk supports associative arrays. example ```var[key] = value``` 691 | - on integers, basic arithmatic operations (+-*/%) are supported. autoincrement(++) and decrement(--) is also supported. 692 | 693 | ## 15.1. Actions 694 | - following are few actions that can be performed 695 | 696 | action | description 697 | --- | --- 698 | { print $0; } | print records 699 | { exit; } | ends program 700 | { next; } | skips current line 701 | {a=$1; b="X"} | variable assignment 702 | { c[$1] = $2 } | array varaible assignment 703 | {if (condition) { action } else if (condition) { action } else { action }} | if else conditions 704 | { for (i=1; i < x; i++) { action } } | for loop 705 | { for (item in c) { action } } | for loop iterating over a list 706 | 707 | ## 15.2. Special variables 708 | 709 | variable | desc 710 | --- | --- 711 | FS | Input field separator. can be modified 712 | RS | Input record separator. default value is newline. can be modified by user 713 | OFS | Output field separator. can be modified 714 | ORS | Output record separator. default value is newline. can be modified by user 715 | NF | Number of fields in the current line (record). cannot be updated by user 716 | NR | Number of lines processed so far. cannot be updated by user 717 | 718 | *Note: -F option can be used to update the input field separator -> ```awk -F":"'{ print $1 }' file.txt```* 719 | 720 | ## 15.3. Examples 721 | - split up “,” (comma) separated fields and print the third field ($3)
```awk -F"," '{print $2}' file.txt``` 722 | - print the 3rd field of a csv if the second field ($2) exists and is not empty
```awk -F"," '{if ($2)print $3}' file.txt``` 723 | - print the last field in each line
```awk -F"," '{ print $NF }' file.txt``` 724 | - print the line after the line matching search pattern
```awk '/pattern/ { i=1; next; } {if(i) {i--; print;}}' file.txt``` 725 | - print the line and the 2 lines after the line matching search pattern
```awk '/regexp/ {i=3;} { if(i) {i--; print;}}' file.txt``` 726 | - print the lines from a file starting at the line matching *start* until the line matching *stop*
```awk '/start/,/stop/' file.txt``` 727 | - count lines (wc -l)
```awk 'END{print NR}' file.txt``` 728 | - print matching lines (grep)
```awk '/pattern/'``` 729 | - print non matching lines (grep -v)
```awk '!/pattern/'``` 730 | - remove duplicate consecutive lines (uniq)
```awk 'a !~ $0 {print}; {a=$0}' file.txt``` 731 | - print first 10 lines of file (head)
```awk 'NR < 11' file.txt``` 732 | - print last 10 lines of file (tail)
```awk '{vect[NR]=$0;} END{for(i=NR-9;i<=NR;i++) {print vect[i];}}' file.txt``` 733 | - print the total number of bytes used by files
```ls -l | awk '{ x += $5 } END { print "Total bytes: " x }'``` 734 | - read a csv(comma separated) file and print the first and third field in semicolon separated format
```awk 'BEGIN{FS=",";OFS=";"}{print $1, $3}' file.txt``` 735 | 736 | # 16. Arithmetic expansion 737 | - arithmetic expression can be evaluated and the result can then be used. 738 | - syntax - ```$(( expression ))``` 739 | 740 | ## 16.1. Example 741 | - print sum of 2 numbers
```echo $((1+3))``` 742 | 743 | # 17. Command substitution 744 | - In command substitution, the output of a command replaces the command 745 | - the output of a command can be used an argument to another command 746 | - syntax is ``` `command` ``` and ```$(command)``` 747 | - using backticks (`) is discouraged and has been deprecated 748 | - ```$(commmand)``` supports nesting i.e. ```$(command1 $(command2))``` 749 | 750 | ## 17.1. Examples 751 | - assign the output of a command to a variable
``` thedate=`date` ``` 752 | - use the output of command as a parameter of another command
``` vi $(grep -l 123 *) ``` 753 | - assign contents of file to a variable
```filedata=$(cat file.txt)```
```filedata=$(*second option is faster* 754 | 755 | # 18. Process substitution 756 | - the input or output of a command can appear as a file. This is known as process substitution 757 | - this technique is useful when we want to use the output of multiple commands as the input to a command 758 | - process substitution can also be used to capture output and redirect it to the input of a process 759 | - template - ```<(command)``` and ```>(command)``` 760 | 761 | ## 18.1. Examples: 762 | - sort and compare two files
763 | ```diff <(sort file1) <(sort file2)``` 764 | 765 | - compare 2 folders
766 | ```diff <(ls $first_directory) <(ls $second_directory)``` 767 | 768 | # 19. Subshell 769 | - a subshell is a child process launched by a shell 770 | - whenever a shell script is run, a subshell is created and the script is run in the subshell 771 | - variables defined in parent shell can be accessed if ```export``` is used while defining the variable 772 | - subshells can also be created using parentheses
773 | ```(command1; command2; command3)``` 774 | - subshells are a convenient way to group commands. 775 | - can be used to temporarily move to a different directory 776 | ```bash 777 | #do something in current directory 778 | (cd some/other/directory; other-command) 779 | #back in the original directory 780 | ``` 781 | - to run a command or script in the current shell, without creating a subshell, use '.' as in ```. script.sh``` 782 | 783 | # 20. Command grouping 784 | - sometimes we need to run multiple commands and redirect all the output to a single file. Command grouping helps in this. 785 | - without command grouping we need to redirect each command output individually to a file 786 | 787 | ## 20.1. using parentheses 788 | - Grouping list of commands can be done using parentheses *()* 789 | - a subshell is created 790 | - example - ```(date; uptime) > file.txt``` 791 | 792 | ## 20.2. using curly braces 793 | - Grouping commands can be done using curly braces *{}* 794 | - causes the list to be executed in the current shell context 795 | - no subshell is created 796 | - example - ```{date; uptime;} > file.txt``` 797 | 798 | # 21. Text editing with ```cut```, ```paste``` and ```join``` 799 | 800 | ## 21.1. ```cut``` 801 | - ```cut``` command cuts out sections from each line and writes result to standard output 802 | - syntax is ```cut OPTION [FILE]``` 803 | 804 | option | description 805 | --- | --- 806 | -c | character range which will be selected 807 | -d | delimiter which will separate each fields in line in file 808 | -f | fields which will be printed 809 | 810 | when -c option is used, ranges can be specified. Each range can be one of: 811 | range type | description 812 | --- | --- 813 | N | Nth character 814 | N- | fron Nth character to end of line 815 | N-M | from Nth character to Mth character 816 | -M | from first to Mth character 817 | 818 | ### 21.1.1. Examples 819 | - print the first and third columns of a csv file
```cut -f1,3 -d"," file.txt``` 820 | - print the first 3 characters of each line
```cut -c -3 file.txt``` 821 | 822 | ## 21.2. ```paste``` 823 | - merges lines of files 824 | - by default, the lines from each files are delimited by tab 825 | - when '-' is used instead of filename, the command reads from standard input 826 | - syntax ```paste [OPTION] [FILE] [FILE]``` 827 | 828 | option | description 829 | --- | --- 830 | -d | used to specify the delimiter 831 | -s | paste one file at a time 832 | 833 | ### 21.2.1. Examples 834 | lets take 2 files - number.txt and name.txt 835 | > cat number.txt 836 | > 1
837 | > 2
838 | > 3
839 | > 4
840 | 841 | > cat name.txt 842 | > Alice
843 | > Bob
844 | > Charlie
845 | > David
846 | 847 | - merge 2 files, first file will give the first column and second file will give the second column
```paste number.txt name.txt``` 848 | - merge 2 files, delimited by ','
```paste -d"," number.txt name.txt``` 849 | - merge 2 files, sequentially, i.e first only first file is printed and then only the second file
```paste -s number.txt name.txt``` 850 | 851 | ## 21.3. ```join``` 852 | - join lines of two files on a common field 853 | - syntax ```join [OPTIONS] FILE1 FILE2``` 854 | 855 | ### 21.3.1. Examples 856 | lets take 2 files - number.txt and name.txt 857 | > cat number.txt 858 | > 1 100
859 | > 2 101
860 | > 3 102
861 | > 4 103
862 | > 5 104
863 | 864 | > cat name.txt 865 | > 1 Alice
866 | > 2 Bob
867 | > 3 Charlie
868 | > 4 David
869 | 870 | - join 2 files based on the first column
```join number.txt name.txt``` 871 | 872 | # 22. Aliases 873 | - aliases are short names for long commands 874 | - when we need to execute long commands multiple times, it is advisable to create aliases 875 | - syntax - ```alias [-p] [name[=value]]``` 876 | - we can create permanent alias by storing them in configuration files 877 | - to temporarily bypass an alias, use \ -> ```\ll``` 878 | 879 | - creating a alias
```alias name='values'``` 880 | - removing alias
```unalias name``` 881 | - print all defined alias
```alias -p``` 882 | 883 | ## 22.1. useful aliases 884 | ```bash 885 | alias gh='history|grep' 886 | alias c=clear 887 | alias cx='chmod +x' 888 | alias ..='cd ..' 889 | alias sl=ls 890 | alias left='ls -t -1' 891 | alias count='find . -type f | wc -l' 892 | alias f='find . |grep ' 893 | ``` 894 | 895 | # 23. Functions 896 | - set of commands that accomplish a specific task 897 | - can be used numerous times 898 | - helps avoid writing the same code repeatedly 899 | - can use loops and conditions within them 900 | - arguments can be passed to functions 901 | - by using ```export -f functionname```, we can make the functions available to shell scripts 902 | - export can be added to configuration files like *.bashrc* 903 | - To keep things modular, create a new file called ~/.bash_functions and then have your .bashrc load it 904 | - syntax
905 | ```bash 906 | function_name () { 907 | commands 908 | } 909 | ``` 910 | 911 | ```bash 912 | function_name () { commands; } 913 | ``` 914 | 915 | ## 23.1. useful functions 916 | ```bash 917 | mcd() { mkdir -p "$1"; cd "$1";} 918 | cdl() { cd "$1"; ls;} 919 | ``` 920 | 921 | # 24. sort 922 | - sort lines text files 923 | 924 | option | description 925 | --- | --- 926 | -r | sort in reverse order 927 | -n | sort in numberical order 928 | -k | sort based on nth column 929 | -u | sort and remove duplicates 930 | 931 | # 25. uniq 932 | - report or omit repeated lines 933 | - the input file must be sorted 934 | 935 | option | description 936 | --- | --- 937 | -c | show how many times a line is repeated 938 | -d | prints only the repeated lines only once 939 | -u | prints only the unique lines 940 | -i | case insensitive comparison 941 | 942 | 943 | # 26. Conditions 944 | ## 26.1. If else 945 | - if-then-else is supported in command line 946 | - Syntax 947 | ```bash 948 | if [ condition1 ]; then command1; 949 | elif [ condition2 ]; then command2; 950 | else command3; fi 951 | ``` 952 | 953 | ```bash 954 | if [ condition1 ]; then command1; elif [ condition2 ]; then command2; else command3; fi 955 | ``` 956 | - there are different types of conditions 957 | 1. file based condiitons 958 | 959 | conditions | description 960 | --- | --- 961 | -e | check if file exists 962 | -r | check if file exists and is readable 963 | -w | check if file exists and is writable 964 | -d | check if file exists and is a directory 965 | 966 | 2. string based conditions 967 | 968 | conditions | description 969 | --- | --- 970 | == | check if both the strings are equal 971 | != | check if both the strings are not equal 972 | \> | check if the first string is lexicographically greater than the second 973 | < | check if the first string is lexicographically smaller than the second 974 | -n | check if the string has a length more than 0 975 | -z | check if the string is an empty string 976 | 977 | 3. number based conditions 978 | 979 | conditions | description 980 | --- | --- 981 | -eq | check if the numbers are equal 982 | -ne | check if the numbers are not equal 983 | -gt | check if the first number is greater than the second 984 | -ge | check if the first number is greater than or equal to the second 985 | -lt | check if the first number is less than the second 986 | -le | check if the first number is less than or equal to the second 987 | 988 | 989 | - 0 is considered true and numbers greater than 0 are considered false. 990 | - this is because in Unix/Linix, when a process ends successfuly, it returns 0 991 | 992 | ## 26.2. Short circuiting 993 | - an alternative way of using conditions is by using logical AND (&&) and logical OR(||) 994 | - evaluation of a logical expression is stopped, as soon as the outcome has been determined. This is known as short-circuiting. 995 | - in case of Logical AND, as soon as sub-expression becomes false, the whole expression evaluates to false 996 | - in case of *expr1 && expr2*, if expr1 evaluates to false, then the whole expression will evaluate to false. So, expr2 is not evaluated at all. 997 | - && can be used to ensure that command2 is run only if command1 ends successfully. example ->```command1 && command2``` 998 | - in case of Logical OR, as soon as sub-expression becomes true, the whole expression evaluates to true 999 | - in case of *expr1 || expr2*, if expr1 evaluates to true, then the whole expression will evaluate to true. So, expr2 is not evaluated at all. 1000 | - || can be used to ensure that command2 is run only if command1 fails. example -> ```command1 || command2``` 1001 | 1002 | ### 26.2.1. example 1003 | - create folder if it does not exist 1004 | ```bash 1005 | [ -d ./some/path/folder ] || mkdir /some/path/folder 1006 | ``` 1007 | 1008 | - create create file only if folder exists 1009 | ```bash 1010 | cd /some/path/folder && touch file.txt 1011 | ``` 1012 | 1013 | # 27. Loops 1014 | ## 27.1. while loops 1015 | - the loop runs as long as the given condition is true 1016 | - syntax 1017 | ```bash 1018 | while [ condition ]; do commands; done 1019 | ``` 1020 | ### 27.1.1. example 1021 | - print all the folders with .c files
1022 | ```bash 1023 | find . -name *.c | {while read -r filename; do dirname $filename; done;} | sort | uniq # dirname returns the directory name 1024 | ``` 1025 | 1026 | 1027 | ## 27.2. for loops 1028 | - the loop iterates over a list of values or preset number of times 1029 | - syntax 1030 | ```bash 1031 | for in ;do $;done; 1032 | ``` 1033 | ### 27.2.1. example 1034 | - copy files from one folder to another 1035 | ```bash 1036 | for file in ./code/*.txt; do cp $file /home/code/backup; done 1037 | ``` 1038 | 1039 | # 28. ssh 1040 | - ssh (SSH client) is a program for logging into a remote machine and for executing commands on a remote machine 1041 | - syntax ```ssh user@host``` 1042 | - running a single command on remote server
```ssh user@host command_to_run``` 1043 | - logging into server with different port
```ssh -p portnum user@host``` 1044 | - ssh connection using host in the middle```ssh -t reachable_host ssh unreachable_host``` 1045 | 1046 | # 29. curl 1047 | - used to transfer data from one server to another 1048 | - syntax - ```curl [options] [URL]``` 1049 | 1050 | options | description 1051 | --- | --- 1052 | -o | save the output in a file with name as specified after the option 1053 | -O | save the output in a file with the same name as in the url 1054 | -C - | resume a download 1055 | -I | fetch headers of a url 1056 | -L | follow redirects 1057 | 1058 | 1059 | ## 29.1. Examples 1060 | - retrieve a webpage
```curl example.com``` 1061 | - save a webpage
```curl example.com -o example.html``` 1062 | - resume a download
```curl -C - -O https://releases.ubuntu.com/21.10/ubuntu-21.10-desktop-amd64.iso``` 1063 | - fetch headers only
```curl -I example.com``` 1064 | - fetch weather
```curl wttr.in/london``` 1065 | 1066 | ## 29.2. handling REST calls 1067 | ### 29.2.1. GET 1068 | - GET is used to fetch resource 1069 | - the GET method is the default method 1070 | - example ```curl https://reqres.in/api/users/2``` 1071 | 1072 | ### 29.2.2. POST 1073 | - POST is used to create resource in a server 1074 | - To send a curl POST request we use the option -X POST 1075 | - example ```curl -X POST -H "Content-Type: application/json" -d '{"email": "eve.holt@reqres.in","password": "pistol"}' https://reqres.in/api/register``` 1076 | 1077 | ### 29.2.3. PUT 1078 | - PUT is used to update resource 1079 | - To send a curl PUT request we use the option -X PUT 1080 | - example ```curl -X PUT -H "Content-Type: application/json" -d '{"name": "morpheus","job": "zion resident"}' https://reqres.in/api/users/2``` 1081 | 1082 | ### 29.2.4. DELETE 1083 | - DELETE is used to remove resource 1084 | - To send a curl DELETE request we use the option -X DELETE 1085 | - example ```curl -X DELETE https://reqres.in/api/users/2``` 1086 | 1087 | # 30. wget 1088 | - command line utility to download file 1089 | 1090 | options | description 1091 | --- | --- 1092 | -O | download file under different name 1093 | -c | resume a download 1094 | 1095 | ## 30.1. Examples 1096 | - download a file
```wget https://releases.ubuntu.com/21.10/ubuntu-21.10-desktop-amd64.iso``` 1097 | - resume a download
```wget -c https://releases.ubuntu.com/21.10/ubuntu-21.10-desktop-amd64.iso``` 1098 | 1099 | 1100 | # 31. One liners 1101 | 1102 | - print the files and directories in tree structure
```find . | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/|-\1/"``` 1103 | - print the directories in tree structure
```find . -type d | sed -e "s/[^-][^\/]*\// |/g" -e "s/|\([^ ]\)/| - \1/"``` 1104 | - quick access to ascii table
```man ascii``` 1105 | - find common lines in 2 files
```cat file1 file2 | uniq -d``` 1106 | - find lines in file1 that is not present in file2
```cat file1 file2 file2 | uniq -u``` 1107 | - find most frequently used commands
```history | cut -c8- | sort | uniq -c | sort -rn | head``` 1108 | - recursively remove only directories with no files
```find . -depth -type d -exec rmdir {} \;``` 1109 | - count the number of total active network interfaces
```wc -w <<<$(netstat -i | cut -d" " -f1 | egrep -v "^Kernel|Iface|lo")``` 1110 | - display all the files/directories sorted by size
```du -ah . 2>/dev/null | sort -hr | less``` 1111 | - make multiple copies of file
```tee filecopy{1..10}.txt < file.txt``` 1112 | 1113 | # 32. Further reading 1114 | - 1115 | - 1116 | 1117 | # 33. Change History 1118 | - presented in reversed chronological order i.e. the latest change is at the top 1119 | 1120 | Name | Date | Change Description 1121 | ---- | ---- | --- 1122 | Utsav Barman | 26 Jan 2022 | added: [arithmetic expansion, grep recursively search, command substitution example] 1123 | Utsav Barman | 26 Jan 2022 | added: [ANSI C quoting] 1124 | Utsav Barman | 26 Jan 2022 | added: [brace expansion, find piped to xargs, herestrings, awk example] 1125 | Utsav Barman | 26 Jan 2022 | added: [command grouping] 1126 | Utsav Barman | 25 Jan 2022 | added: [chown, chmod and du] 1127 | Utsav Barman | 25 Jan 2022 | added: [& to start a background job, added curl and wget] 1128 | Utsav Barman | 25 Jan 2022 | read should be used with option -r ; used appropriate variable names in for loop example; 1129 | Utsav Barman | 25 Jan 2022 | Updated variable DATE to thedate. Uppercase variables are conventionally used for environment variables. Added $() for command substitution. backticks are discouraged; replaced -a with -e 1130 | Utsav Barman | 25 Jan 2022 | fixed typo for cd to root directory; added section for special shell variables and exit codes; 1131 | Utsav Barman | 25 Jan 2022 | Added wait, globbing, grep -o; fixed typos 1132 | Utsav Barman | 24 Jan 2022 | Initial draft 1133 | 1134 | 1135 | 1136 | 1137 | --------------------------------------------------------------------------------