└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # Useful sed 2 | 3 | Useful sed tips, techniques and tricks for daily usage. 4 | 5 | # TABLE OF CONTENTS 6 | 7 | * [One liners with short explanation](#one-liners-with-short-explanation) :bulb: 8 | 9 | * [Short sed tut](#short-sed-tut) :books: 10 | 11 | * [About](#about) 12 | 13 | * [Please consider donating to support this type of content!](#buy-me-a-coffee) :pray: 14 | 15 | * [Credits](#credits-and-links) 16 | 17 | # Recommended books (not written by me) 18 | [Definitive Guide to sed - by Daniel Goldman](https://amzn.to/3cnzQCS) 19 |
20 | [Sed & Awk - Dale Dougherty & Arnold Robbins](https://amzn.to/3kKck7Y) 21 |
22 | [Effective awk programming - by Arnold Robbins](https://amzn.to/30scfhM) 23 | 24 | # More guides that I wrote 25 | [text-processing-recipes-linux](https://github.com/adrianscheff/text-processing-recipes-linux) - Comprehensive list of text processing tools and techniques. Featuring `vim, tr, cat, tac, sort, shuf, seq, pr, paste, fmt, cut, nl, split, csplit, sed, awk, grep and regex.` 26 |
27 | [simple-awk](https://github.com/adrianscheff/simple-awk) - A simple but practical guide to awk. 28 |
29 | [wizardly-tips-vim](https://github.com/adrianscheff/wizardly-tips-vim) - Less known Vim tips & tricks 30 |
31 | [quick-grep](https://github.com/adrianscheff/quick-grep) - Quick grep reference and tutorial. 32 |
33 | [convenient-utils-linux](https://github.com/adrianscheff/convenient-utils-linux) - Linux utils to make life easier and more convenient. 34 | 35 | # BUY ME A COFFEE 36 | I love teaching others. Time is limited but donations will allow me to to help the community more. **How useful was this to you?** If it was I would be humbly grateful for your donation.:pray: 37 | 38 | 39 | [paypal.me/adrianscheff](https://www.paypal.com/paypalme/adrianscheff) | [patreon.com/adrianscheff](https://www.patreon.com/adrianscheff) | bitcoin (1NrkpsgbmmLDoDcvAvsGEMGEQhvvtw36x1) - **are some ways to help me help you better.** 40 | 41 | 42 | Thank you! May you be rich as Crassus and happy as Buddha! :) 43 | 44 | ----- 45 | 46 | 47 | 48 | # ONE LINERS WITH SHORT EXPLANATION 49 | #### Print one line 50 | `sed -n '10p' myfile.txt` 51 | 52 | #### Do replacement on all lines except line 5 53 | `sed '5!/s/foo/bar/' file.txt` 54 | 55 | #### Do replacement on lines matching regex (eg: lines starting with 'hello') 56 | `sed '/^hello/ s/h/H/' file.txt ` 57 | 58 | 59 | 60 | #### Do replacement from line 5 to end of file 61 | `sed '5,$ s/foo/bar/' file.txt ` 62 | 63 | 64 | #### Delete empty files 65 | `sed '/^$/d' file` 66 | 67 | 68 | 69 | #### Print lines between two regex matches 70 | `sed -nE '/^foo/,/^bar/p' file.txt` 71 | 72 | #### Use custom delimiters to make it easy for some strings that contain slashes 73 | `sed 's_/bin/bash_/bin/sh_' file.txt ` 74 | 75 | #### Custom delimiters for regex address combined with the classical delimiter for substitute command (you could also use there a custom delimiter). Useful for paths. 76 | `sed '\_/bin/bash_s/grep/egrep/' file.txt` 77 | * or using the same delimiter for clarity `sed '\_/bin/bash_s_grep_egrep_' file.txt` 78 | 79 | #### Insert a space between lowercase/Uppercase characters using & (which represents the regex match) 80 | `sed 's/[a-zA-Z]/& /g' file.txt ` 81 | 82 | #### Keep the first word of every line (where word is defined by alphanumeric chars + underscores for simplicity sake) 83 | `sed -E 's_[a-zA-Z0-9_]+.*_\1_' file.txt ` 84 | 85 | 86 | #### Switch the first two words 87 | `sed -E 's_([a-zA-Z0-9_]*) ([a-zA-Z0-9_]*)_\2 \1_' f1` 88 | 89 | 90 | #### Remove duplicate words separated by a single space (but not triplicate) 91 | `sed -E 's_([a-zA-Z0-9_]+) \1_\1_ig' f1` 92 | 93 | #### Search and replace for pattern, write just the lines with the replacements in a new file 94 | `sed 's_foo_bar_w replaced.txt' file.txt ` 95 | 96 | #### Multiple replacements 97 | `sed -e 's_foo_bar_' -e 's_hello_HELLO_' file.txt ` 98 | 99 | #### Multiple replacements by using a sed script 100 | ``` 101 | #!/usr/bin/sed -f 102 | s/a/A/ 103 | s/foo/BAR/ 104 | s/hello/HELLO/ 105 | ``` 106 | * Make executable with `chmod +x myscript.sed`, call with `./myscript.sed myfile.txt` 107 | 108 | 109 | #### Multiple commands using the ; operator which in theory concatenates commands (WARNING! It won't work as expected with certain commands such as 'r' or 'w'. Use a sed script instead OR put the command dealing with filenames last). Print line 10 and insert before line 5. 110 | `sed '10p;5i\"INSERTED BEFORE LINE 5" file.txt ` 111 | 112 | 113 | #### Remove comments between lines starting with these two keywords. Empty lines will be put there instead 114 | `sed -E '/start/,/end/ s/#.*//' file.txt ` 115 | 116 | #### Delete comments starting with # (no empty lines left behind) 117 | `sed -E '/^#/d' f1` 118 | 119 | #### Insert an empty line after pattern (after each line containing comment in this case) 120 | `sed '/^#/G' file.txt ` 121 | 122 | 123 | #### View lines minus lines between line starting with pattern and end of file 124 | `sed '/start/,$ d' file.txt ` 125 | 126 | #### View lines except lines between line starting with pattern and line ending with pattern 127 | `sed -rn '/start/,/end/ !p' file.txt ` 128 | 129 | #### Print until you encounter pattern then quit 130 | `sed '/start/q' file.txt ` 131 | 132 | #### Insert contents of file after a certain line 133 | `sed '5 r newfile.txt' file.txt ` 134 | 135 | #### Append text after lines containing regex (AFTER FOO) 136 | `sed '/foo/a\AFTER FOO' file.txt ` 137 | 138 | #### Insert text after lines containing regex (BEFORE FOO) 139 | `sed '/foo/i\BEFORE FOO' file.txt ` 140 | 141 | #### Change line containing regex match 142 | `sed '/foo/c\FOO IS CHANGED' file.txt ` 143 | 144 | #### Nested sed ranges with inversion. Between lines 1,100 apply actions where the pattern DOESN'T match. 145 | ``` 146 | #!/usr/bin/sed -f 147 | 1,100 { 148 | /foo/ !{ 149 | s_hello_HELLOOOOWORLD_ 150 | s_yes_YES_ 151 | } 152 | } 153 | 154 | ``` 155 | 156 | 157 | #### Use nested addresses with change, insert and append to modify: the line before match, the line with match, the line after match. 158 | ``` 159 | #!/usr/bin/sed -f 160 | /^#/ { 161 | i\ 162 | #BEFFORE ORIGINAL COMMENt 163 | a\ 164 | #AFTER ORIGINAL COMMENT 165 | c\ 166 | # ORIGINAL COMMENT IS NOW THIS LINE 167 | } 168 | 169 | ``` 170 | 171 | #### Insert new line before the first comment, after the first comment put in the contents of file and quit immediately afterwards 172 | ``` 173 | #!/usr/bin/sed -f 174 | /^#/ { 175 | i\#BEFORE COMMENT 176 | r myotherfile.txt 177 | q 178 | } 179 | ``` 180 | 181 | #### Transform text 182 | `sed 'y/abc/ABC/' file.txt ` 183 | 184 | 185 | #### Copy all the comments (starting with #) to a new file 186 | `sed -E '/^#/w comments.txt' file.txt ` 187 | 188 | #### Print every second line (substitute ~3 for third line, etc) 189 | `sed -n '1~2p' file.txt ` 190 | 191 | #### Edit file in place but also create a backup 192 | `sed -i.bak 's/hello/HELLO/' file.txt ` 193 | 194 | #### Append two extra lines after regex match 195 | `sed -E '/^#/G G' file.txt ` 196 | 197 | ----- 198 | 199 | # Short Sed Tut 200 | #### Sed commands use an address based on which they operate. The address can be: 201 | 1. **Single lines** `sed '10d' file.txt ` - delete line 10 202 | 2. **Line range** `sed '1,10d' file.txt` - delete from line 1 to 10 203 | 3. **Line range2** `sed '6,$d' file.txt ` - delete from line 6 to end of file ($ is end of file) 204 | 4. **Regex** `sed -E '/^#/d' file.txt ` - delete lines where regex matches 205 | 5. **Regex ranges** `sed -E '/DEBUG/,/END_DEBUG/d' file.txt ` - delete lines between regex matches (including lines where regex matches) 206 | 6. **Regex and line ranges** `sed -E '/DEBUG/,30d' file.txt ` - delete from line matching DEBUG to line 30 207 | 7. **all lines** `sed 'a\AFTER EVERY LINE' file.txt ` - append this after every line (when no address is present apply to all lines) 208 | 8. **Nested** - use this with a sed script (see below) 209 | 210 |
211 | 212 | ``` 213 | #!/usr/bin/sed -f 214 | 1,100 { 215 | /DEBUG/{ 216 | /DONE/d 217 | /NOT DONE/a\TO BE DONE URGENTLY 218 | } 219 | } 220 | ``` 221 | 222 |
223 | 224 | - between lines 1 and 100: 225 | >> where matches DEBUG, 226 | >>> delete lines containing /DONE/ and after lines containing /NOT DONE/ append. 227 | 228 |
229 | 230 | #### You can invert the address by putting a ! in front of **the command**, not the address. 231 | 1. `sed '/PRODUCTION/!d' file.txt ` - delete all lines not containing regex match. Note the ! in front of d. 232 | 2. Everything inside curly brace (for nested) is a command. You put the ! in front of the curly brace. 233 | 234 |
235 | 236 | ``` 237 | #!/usr/bin/sed -f 238 | 1,100 { 239 | /DEBUG/ !{ 240 | /DONE/d 241 | /NOT DONE/a\TO BE DONE URGENTLY 242 | } 243 | } 244 | ``` 245 | 246 |
247 | 248 | * between lines 1,100 249 | >> on lines NOT containing /DEBUG/ 250 | >>> Perform operations 251 | 252 |
253 | 254 | 3. "Double" nested inversion 255 | 256 |
257 | 258 | ``` 259 | #!/usr/bin/sed -f 260 | 1,100 { 261 | /DEBUG/ !{ 262 | /DONE/!d 263 | } 264 | } 265 | ``` 266 | 267 |
268 | 269 | * between lines 1,100 270 | >> on lines not containing DEBUG 271 | >>> delete lines NOT containing /DONE/ 272 | 273 |
274 | 275 | 276 | #### Basic commands: 277 | 1. `5d` - **delete** - Delete line 5. 278 | 2. `5p` - **print.** - print line 5 (you should call sed with `-n` option when using print to only print the specified lines) 279 | 3. `5q` - **quit** - after line 5 quit 280 | 4. `5a\Appended` - **append** - after line 5. Note the backward slash in front of 'a' 281 | 5. `5c\Changed` - **change** - change line 5 to 'Changed' 282 | 6. `5i\Before` - **insert** - insert before line 5. 283 | 7. `5r newfile.txt` - **read** - put the contents of file 'newfile.txt' after line 5 284 | 8. `5w written.txt` - **write** - write line 5 to 'written.txt' 285 | 9. `5s/foo/bar` - **substitute** - on line 5 search for foo and replace with bar 286 | 287 |
288 | 289 | #### Advanced & Less used commands 290 | 1. `sed -E '/^#/G G' file.txt ` - **append newline to pattern space then append hold space to pattern space** - insert two blank lines after every line that matches regex 291 | 292 |
293 | 294 | #### Regex tricks 295 | 1. `&` is the matched regex. `sed -E '/foo/& & &/' file.txt` will triplicate the foo word 296 | 2. `\1` to `\9` are the groups id's. You use a group like `sed -E 's/(foo) (bar)/\2 \1' file.txt '. In this very simple example we search for 'foo' followed by space followed by 'bar'. Then we switch these words (instead of 'foo bar' we have 'bar foo') 297 | 3. Flags. `sed 's/foo/bar/gi' file.txt `. 'g' will replace all occurrences on the line (instead of just the first as it is by default). 'i' will make the substitute case insensitive. 298 | 299 |
300 | 301 | [TO BE CONTINUED] 302 | 303 | ------ 304 | 305 | # About 306 | 307 | This was born in 4-5 hours of recapping sed (and many hours learning it in the first place). Since then I've spent some more hours (and counting) fixing & improving. A good way to learn something well is to also explain to others. And if I'm honest this is also for myself, later, when I forget. :) 308 | 309 | 310 | It was Number 1 on Hacker News in hours after publication and 300+ stars on Github. Hours. Damn! All other projects combined (some of which took much more time) are in the double digit range. The low range. And this quick and dirty sed tips collection took off. I consider myself lucky to have experienced this and I'm grateful to the people who made it possible. For you I'll try and make this as good as I can. 311 | 312 | 313 | 314 | 315 | 316 | 317 | ------ 318 | # Credits and links 319 | I've learned sed from the manual and various internet links. Here are some of them:
320 | http://sed.sourceforge.net/sed1line.txt
321 | https://www.grymoire.com/Unix/Sed.html
322 | `info sed` and `man sed`
323 | 324 | 325 | **NOTES**
326 | * This is not absolutely perfect and up to the highest standards of Posix and sed usage. It was kindly pointed to me by people from HN. I'll try and make this the best possible version I can and I'll listen to your feedback. I'm not a sed guru after all, just someone trying to follow the 20/80 formula (even though sometimes I go waaay overboard).
327 | * There is no mention of the hold space (and other registers) usage. Personally I found their usage quite confusing. If this might change in the future I'll adjust this. After all, this is not "The ultimate sed reference" but "useful sed".
328 | * There might be examples where I use -E (regexp extended option) when it is not necessarily needed. Chances are you'll want to use this in most of your operations so it might not be a bad reflex to have. Sure, you could make an alias for it in your .bashrc. Personally I try and delay usage of aliases until I feel it's absolutely necessary (as it hides some verbosity which makes the commands clearer).
329 | 330 | --------------------------------------------------------------------------------