├── README.md ├── .gitignore ├── .editorconfig ├── gimp.pdf ├── pdb.pdf ├── vim.pdf ├── markdown.pdf ├── macos.md ├── conda.md ├── git.md ├── arxiv.md ├── ffmpeg-imagemagick.md └── unix.md /README.md: -------------------------------------------------------------------------------- 1 | # cheatsheets 2 | 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | !.gitignore 2 | !.editorconfig 3 | *.swp 4 | -------------------------------------------------------------------------------- /.editorconfig: -------------------------------------------------------------------------------- 1 | [*] 2 | indent_style = tab 3 | indent_size = 4 4 | -------------------------------------------------------------------------------- /gimp.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xiumingzhang/cheatsheets/HEAD/gimp.pdf -------------------------------------------------------------------------------- /pdb.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xiumingzhang/cheatsheets/HEAD/pdb.pdf -------------------------------------------------------------------------------- /vim.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xiumingzhang/cheatsheets/HEAD/vim.pdf -------------------------------------------------------------------------------- /markdown.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/xiumingzhang/cheatsheets/HEAD/markdown.pdf -------------------------------------------------------------------------------- /macos.md: -------------------------------------------------------------------------------- 1 | ### Reduce PDF size 2 | 3 | ```bash 4 | ps2pdf -dPDFSETTINGS=/ebook BiggerPdf SmallerPDF 5 | ``` 6 | -------------------------------------------------------------------------------- /conda.md: -------------------------------------------------------------------------------- 1 | ### List environments 2 | 3 | ```bash 4 | conda env list 5 | ``` 6 | 7 | 8 | ### List packages in environment 9 | 10 | If activated 11 | 12 | ```bash 13 | conda list 14 | ``` 15 | 16 | Else 17 | 18 | ```bash 19 | conda list -n myenv 20 | ``` 21 | 22 | 23 | ### Create environment 24 | 25 | ```bash 26 | conda create -n myenv python=3.4 27 | ``` 28 | 29 | If from `.yml` 30 | 31 | ```bash 32 | conda env create -f environment.yml 33 | ``` 34 | 35 | 36 | ### Remove environment 37 | 38 | ```bash 39 | conda remove --name old_name --all # or its alias: `conda env remove --name old_name` 40 | ``` 41 | 42 | 43 | ### Rename environment 44 | 45 | ```bash 46 | conda create --name new_name --clone old_name 47 | conda remove --name old_name --all # or its alias: `conda env remove --name old_name` 48 | ``` 49 | 50 | 51 | ### Install packages to environment 52 | 53 | ```bash 54 | conda install --name myenv scipy 55 | ``` 56 | 57 | 58 | ### Check, add, or remove channels 59 | 60 | ```bash 61 | conda config --get channels 62 | conda config --add channels anaconda 63 | conda config --remove channels anaconda 64 | ``` 65 | 66 | 67 | ### Rollback 68 | 69 | Locate the version before the screwup with 70 | 71 | ```bash 72 | conda list --revisions 73 | ``` 74 | 75 | Rollback with 76 | 77 | ```bash 78 | conda install --revision 79 | ``` 80 | 81 | In the case of 82 | 83 | ```bash 84 | CondaRevisionError: Cannot revert to , since :: is not in repodata. 85 | ``` 86 | 87 | you will need to see from the revision history which packages got updated, and then revert them one by one, with necessary channel flags, e.g., 88 | 89 | ```bash 90 | conda install blas=1.1 -c conda-forge 91 | ``` 92 | -------------------------------------------------------------------------------- /git.md: -------------------------------------------------------------------------------- 1 | ### Change remote (e.g., if the repo. was renamed) 2 | 3 | ```bash 4 | git remote set-url origin git@github.com:xiumingzhang/new.git 5 | ``` 6 | 7 | 8 | ### Keep working on an unsubmitted commit 9 | 10 | ```bash 11 | git commit --amend 12 | ``` 13 | 14 | 15 | ### Undo last commit 16 | 17 | If you don't want to keep your changes that you made, do 18 | 19 | ```bash 20 | git reset --hard HEAD^ 21 | ``` 22 | 23 | If you want to keep your changes, do 24 | 25 | ```bash 26 | git reset --soft HEAD^ 27 | ``` 28 | 29 | 30 | ### Resync git repo with a new .gitignore 31 | 32 | 33 | ```bash 34 | cd "$repo_root" 35 | git rm -r --cached . 36 | git add . 37 | git commit -m "Resync with new .gitignore" 38 | ``` 39 | 40 | 41 | ### Make .gitignore ignore everything but a few files 42 | 43 | ```bash 44 | # Ignore everything in root, except these files 45 | /* 46 | !.gitignore 47 | !.bash_profile 48 | !.gitconfig 49 | !.inputrc 50 | !.vimrc 51 | !README.md 52 | 53 | # Don't ignore .ssh folder 54 | !.ssh/ 55 | 56 | # Ignore everything there except one file 57 | .ssh/* 58 | !.ssh/config 59 | ``` 60 | 61 | 62 | ### List files current tracked by git 63 | 64 | If you want to list all the files currently being tracked under the branch master, do 65 | 66 | ```bash 67 | git ls-tree -r master --name-only 68 | ``` 69 | 70 | 71 | ### List commits 72 | 73 | ```bash 74 | git log 75 | ``` 76 | 77 | 78 | ### See what has been changed by a commit 79 | 80 | ```bash 81 | git show 82 | ``` 83 | 84 | Just filenames? 85 | 86 | ```bash 87 | git show --name-only 88 | ``` 89 | 90 | 91 | ### Compare what has changed between two commits 92 | 93 | ```bash 94 | git diff ^ 95 | ``` 96 | 97 | 98 | ### Find the deleted from commit history 99 | 100 | ```bash 101 | git log -S 102 | ``` 103 | 104 | Then you can find the most recent commit that essentially did the deletion. 105 | -------------------------------------------------------------------------------- /arxiv.md: -------------------------------------------------------------------------------- 1 | The arXiv system doesn't work well with the file structure I like: 2 | ``` 3 | fig/ 4 | teaser.tex 5 | teaser.pdf 6 | model.tex 7 | model.pdf 8 | ... 9 | sec/ 10 | 0_abs.tex 11 | 1_intro.tex 12 | ... 13 | tbl/ 14 | ablation.tex 15 | baselines.tex 16 | macros.tex 17 | main.bib 18 | main.tex 19 | packages.tex 20 | ``` 21 | because it will remove `fig/*.pdf` since they all have paired `.tex` files 22 | with the same filenames ("Hey, but the extensions are different!" I know...). 23 | 24 | The workflow I follow is: 25 | 26 | 1. Download the source .zip from Overleaf. 27 | 28 | 1. Expand all `.tex` so that we have a single, all-in-one file `main_arxiv.tex` 29 | (and this will also remove all comments for privacy): 30 | ```bash 31 | cd "$LATEX_PROJECT_ROOT" 32 | latexpand --empty-comments main.tex -o main_arxiv.tex 33 | ``` 34 | 35 | `latexpand` comes with MacTeX: 36 | ```bash 37 | $ which latexpand 38 | /Library/TeX/texbin/latexpand 39 | ``` 40 | 41 | 1. Remove all now-redundant files: 42 | ```bash 43 | rm -f fig/*.tex 44 | 45 | # Remove all .tex files except main_arxiv.tex 46 | mv main_arxiv.tex fig/ 47 | rm -f *.tex 48 | mv fig/main_arxiv.tex ./ 49 | 50 | rm -rf sec/ 51 | rm -rf tbl/ 52 | ``` 53 | 54 | 1. If necessary, compress PDFs to make sure the final .zip will be less than 50 MB: 55 | ```bash 56 | PDF_FILE='fig/big.pdf' 57 | gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile="$PDF_FILE".compressed "$PDF_FILE" 58 | mv "$PDF_FILE".compressed "$PDF_FILE" 59 | ``` 60 | If the compressed quality is not ideal, see 61 | [`unix.md`](https://github.com/xiumingzhang/cheatsheets/blob/master/unix.md) 62 | for other compression options available. 63 | 64 | 1. Make sure the project still compiles into the correct PDF by doing `latex`, 65 | `bibtex`, `latex`, and again `latex` in TeXShop. 66 | 67 | 1. Click "Trash Aux Files." Remove `main_arxiv.pdf` (since it's large). 68 | 69 | 1. Zip the folder and upload. 70 | -------------------------------------------------------------------------------- /ffmpeg-imagemagick.md: -------------------------------------------------------------------------------- 1 | ### Combine a folder of .m4a files into a single .m4a 2 | 3 | ```bash 4 | folder="$1" 5 | 6 | tmp_file='/Users/xiuming/Desktop/tmp.aac' 7 | out_file='/Users/xiuming/Desktop/combined.m4a' 8 | 9 | for f in "$folder"/*.m4a; do 10 | aac_file="$f.aac" 11 | ffmpeg -i "$f" -acodec copy "$aac_file" -y 12 | cat "$aac_file" >> "$tmp_file" 13 | done 14 | 15 | ffmpeg -i "$tmp_file" -bsf:a aac_adtstoasc "$out_file" -y 16 | ``` 17 | 18 | 19 | ### Resize video to 720p 20 | 21 | ```bash 22 | ffmpeg -i nlt_v21.mp4 -vf scale=-1:720 -c:v libx264 -crf 18 -preset veryslow -c:a copy nlt_v21_720p.mp4 23 | ``` 24 | 25 | 26 | ### Cut video 27 | 28 | ```bash 29 | ffmpeg -i input.mkv -ss 1:40:00.0 -c copy -to 1:59:15.30 output.mkv 30 | ``` 31 | 32 | * To also remove sound and **convert format** 33 | 34 | ```bash 35 | ffmpeg -i sintel.avi -ss 0:9:37.0 -c copy -to 0:9:40.50 -an bigFight.mp4 36 | ``` 37 | 38 | 39 | ### Change Frame Rate 40 | 41 | * Speed up by 2x 42 | 43 | ```bash 44 | ffmpeg -i input.mp4 -filter_complex "[0:v]setpts=0.5*PTS[v];[0:a]atempo=2.0[a]" -map "[v]" -map "[a]" output.mp4 45 | ``` 46 | 47 | 48 | ### Scale image 49 | 50 | * Specify width 51 | 52 | ```bash 53 | ffmpeg -i in.png -vf scale=iw/2:-1 out.png 54 | ``` 55 | 56 | * Specify height 57 | 58 | ```bash 59 | ffmpeg -i in.png -vf scale=-1:ih/2 out.png 60 | ``` 61 | 62 | 63 | ### Convert format 64 | 65 | ```bash 66 | ffmpeg -i dense.avi -c:v libx264 noDepth.mp4 67 | ``` 68 | 69 | 70 | ### Reverse video 71 | 72 | ```bash 73 | ffmpeg -i video.mp4 -vf reverse video_rev.mp4 74 | ``` 75 | 76 | 77 | ### Concatenate videos 78 | 79 | ```bash 80 | ffmpeg -i video1.mp4 -i video2.mp4 -filter_complex "[0:v] [1:v] concat=n=2:v=1 [v]" -map "[v]" video_concat.mp4 81 | ``` 82 | 83 | 84 | ### Change FPS 85 | 86 | ```bash 87 | ffmpeg -i video.mp4 -filter:v "setpts=0.5*PTS" video_2xfps.mp4 88 | ``` 89 | 90 | 91 | ### Images to video 92 | 93 | * **Numeric** filename 94 | 95 | ```bash 96 | ffmpeg -framerate 20 -start_number 1 -i %3d_0zz.png -vf fps=20 -pix_fmt yuv420p output.mp4 97 | ``` 98 | 99 | * **Character** filename 100 | 101 | ```bash 102 | ffmpeg -framerate 2 -pattern_type glob -i "*.png" -pix_fmt yuv420p output.mp4 103 | ``` 104 | 105 | * Filename of **certain length** 106 | 107 | ```bash 108 | ffmpeg -framerate 2 -pattern_type glob -i "???.png" -pix_fmt yuv420p -y output.mp4 109 | ``` 110 | 111 | `???.png` matches with all filenames of length 3. `-y` overwrites without asking. 112 | 113 | * If **not divisible by 2** (specify height) 114 | 115 | ```bash 116 | ffmpeg -framerate 10 -pattern_type glob -i "*.png" -vf scale=-2:720 -pix_fmt yuv420p output.mp4 117 | ``` 118 | 119 | * If not divisible by 2 (specify width) 120 | 121 | ```bash 122 | ffmpeg -framerate 10 -pattern_type glob -i "*.png" -vf scale=1280:-2 -pix_fmt yuv420p output.mp4 123 | ``` 124 | 125 | * Start from and end at 126 | 127 | ```bash 128 | ffmpeg -start_number 250 -i img_%4d.jpg -vframes 500 -vcodec mpeg4 output.mp4 129 | ``` 130 | 131 | combines `img_0250.jpg` through `img_0750.jpg`. 132 | 133 | * If **high quality** is required 134 | 135 | Set bitrate (around 5 Mbits/s for DVD) and codec by adding flags: 136 | 137 | ```bash 138 | -b 5000k -vcodec libx264 139 | ``` 140 | 141 | If no H.264 support, use `-vcodec mpeg4`. 142 | 143 | 144 | ### Make GIF 145 | 146 | ```bash 147 | convert -delay 20 -loop 0 *jpg animated.gif 148 | ``` 149 | 150 | 151 | ### Crop GIF 152 | 153 | ```bash 154 | convert in.gif -coalesce -repage 0x0 -crop WxH+X+Y +repage out.gif 155 | ``` 156 | 157 | 158 | ### Make GIF loop 159 | 160 | ```bash 161 | convert -delay 20 -loop 0 nonloopingImage.gif loopingImage.gif 162 | ``` 163 | 164 | 165 | ### Overlay A on top of B 166 | 167 | ```bash 168 | ffmpeg -i B.png -i A.png -filter_complex "[0:v][1:v] overlay" out.png 169 | ``` 170 | 171 | `[0:v][1:v]` means that we want the first video or image file we import with `-i` to be under the second video or image file. 172 | 173 | 174 | * **Scale** A before overlaying 175 | 176 | ```bash 177 | ffmpeg -i B.png -i A.png -filter_complex "[1]scale=iw/2:-1[b];[0:v][b] overlay" out.png 178 | ``` 179 | 180 | `[1]` is short for "pick the best matching stream" (`[1:v]` is short for pick the best matching video stream -- same thing in this case) and `[b]` is the label for the scale output which is then fed to the overlay. 181 | 182 | * **Make transparent** A before overlaying 183 | 184 | ```bash 185 | ffmpeg -i B.png -i A.png -filter_complex "[1]format=argb,geq=r='r(X,Y)':a='0.5*alpha(X,Y)'[b];[0:v][b] overlay" out.png 186 | ``` 187 | 188 | `0.5` is the opacity factor. I'm including `format=argb` so that it also works with overlay images that don't have an alpha channel of themselves. 189 | 190 | 191 | ### Horizontally stack two images/videos 192 | 193 | ```bash 194 | ffmpeg -i a.jpg -i b.jpg -filter_complex hstack output 195 | ``` 196 | 197 | 198 | ### Horizontally stack two images/videos; need to rescale one 199 | 200 | ```bash 201 | ffmpeg -i left.png -i right.png -filter_complex '[1][0]scale2ref=ih*(W/H):ih[2nd][ref];[ref][2nd]hstack' out.png 202 | ``` 203 | 204 | where `H` and `W` should be replaced with the height and width of `right.png`, respectively. 205 | 206 | If calling this from Python, you may need to remove the quotes. 207 | 208 | 209 | ### Vertically stack two images/videos; need to rescale one 210 | 211 | ```bash 212 | ffmpeg -i top.png -i bottom.png -filter_complex '[1][0]scale2ref=iw:iw*(H/W)[2nd][ref];[ref][2nd]vstack' out.png 213 | ``` 214 | 215 | where `H` and `W` should be replaced with the height and width of `bottom.png`, respectively. 216 | 217 | 218 | ### Overlay text on image 219 | 220 | ```bash 221 | convert in.png -pointsize 40 -fill red -annotate +100+100 'My Text' out.png 222 | ``` 223 | 224 | 225 | ### Rotate images in a folder 226 | 227 | ```bash 228 | for file in ./*.png; do 229 | convert "$file" -rotate 90 "${file%.png}"_rotated.png 230 | done 231 | ``` 232 | 233 | 234 | ### Crop a video 235 | 236 | ```bash 237 | ffmpeg -i gump_orig.mp4 -vf "crop=:::" gump.mp4 238 | ``` 239 | 240 | 241 | ### Crop an image 242 | 243 | ```bash 244 | ffmpeg -i in.png -vf "crop=:::" out.png 245 | ``` 246 | 247 | 248 | ### Get image info 249 | 250 | ```bash 251 | identify -verbose im.tif 252 | ``` 253 | 254 | 255 | ### Photo montage 256 | 257 | ```bash 258 | montage [0-5].png -tile 5x1 -geometry +0+0 out.png 259 | ``` 260 | 261 | ImageMagick ships with the `montage` utility. Montage will append each image side-by-side allowing you to adjust spacing between each image (`-geometry`), and the general layout (`-tile`). 262 | 263 | 264 | ### Replace alpha/transparent background with solid color 265 | 266 | ```bash 267 | convert image.png -background white -alpha remove white.png 268 | ``` 269 | 270 | 271 | ### Change video speed 272 | 273 | ```bash 274 | ffmpeg -i input.mkv -filter:v "setpts=PTS/60" output.mkv 275 | ``` 276 | speeds up the video by 60x. 277 | 278 | 279 | ### Replace image alpha background with a pure color 280 | 281 | ```bash 282 | ffmpeg -i in.png -filter_complex "color=black,format=rgb24[c];[c][0]scale2ref[c][i];[c][i]overlay=format=auto:shortest=1,setsar=1" out.png 283 | ``` 284 | overlays the image with alpha on top of a `black` canvas. 285 | 286 | 287 | ### Extract audio from video 288 | 289 | ```bash 290 | ffmpeg -i input-video.avi -vn -acodec copy output-audio.aac 291 | ``` 292 | 293 | 294 | ### Replace audio in video 295 | 296 | ```bash 297 | ffmpeg -i v.mp4 -i a.wav -c:v copy -map 0:v:0 -map 1:a:0 new.mp4 298 | ``` 299 | 300 | `-map 0:v:0` maps the first (index 0) video stream from the input to the first (index 0) video stream in the output. 301 | `-map 1:a:0` maps the second (index 1) audio stream from the input to the first (index 0) audio stream in the output. 302 | 303 | 304 | ### .webm to .mp4, with high quality 305 | 306 | ```bash 307 | ffmpeg -i video.webm -crf 1 -c:v libx264 video.mp4 308 | ``` 309 | -------------------------------------------------------------------------------- /unix.md: -------------------------------------------------------------------------------- 1 | ### Compress PDF 2 | 3 | ```bash 4 | gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile=small.pdf big.pdf 5 | ``` 6 | Different levels of compression: `/screen` (72 DPI), `/ebook` (150 DPI), `/prepress` (300 DPI), 7 | `/printer` (300 DPI), and `/default`. 8 | 9 | 10 | ### Shebang 11 | 12 | ```bash 13 | #!/usr/bin/env bash 14 | ``` 15 | 16 | 17 | ### Find all files containing specific text 18 | 19 | ```bash 20 | grep -rnw '/path/to/somewhere/' -e "pattern" 21 | ``` 22 | 23 | 24 | ### Replace strings in files under the current directory, recursively 25 | 26 | ```bash 27 | grep -rli 'old-word' * | xargs -i@ sed -i 's/old-word/new-word/g' @ 28 | ``` 29 | 30 | 31 | ### Find file in current directory 32 | 33 | ```bash 34 | find . -name "libstdc++*" 35 | ``` 36 | 37 | 38 | ### Find string in a directory 39 | 40 | ```bash 41 | grep -R 'string' dir/ 42 | ``` 43 | 44 | 45 | ### Remove all files but one 46 | 47 | ```bash 48 | find . ! -name 'to-keep.txt' -type f -maxdepth 1 -exec rm -f {} + 49 | ``` 50 | 51 | 52 | ### Relative path to absolute 53 | 54 | ```bash 55 | readlink -m /x/y/../../a/b/z/../c/d 56 | ``` 57 | 58 | gives 59 | 60 | ```bash 61 | /a/b/c/d 62 | ``` 63 | 64 | 65 | ### Get current user 66 | 67 | ```bash 68 | whoami 69 | ``` 70 | 71 | 72 | ### Parse filename/path/extension 73 | 74 | ```bash 75 | a=/tmp/xx/file.tar.gz 76 | xpath=${a%/*} 77 | xbase=${a##*/} 78 | xfext=${xbase##*.} 79 | xpref=${xbase%.*} 80 | echo ${a} 81 | echo path=${xpath} 82 | echo pref=${xpref} 83 | echo ext=${xfext} 84 | ``` 85 | 86 | 87 | ### Parse string by a single-character delimiter 88 | 89 | ```bash 90 | string="1;2" 91 | echo $string | cut -d';' -f1 # output is 1 92 | echo $string | cut -d';' -f2 # output is 2 93 | ``` 94 | 95 | ```bash 96 | echo "abc : def" | awk -F' : ' '{print $1}' 97 | ``` 98 | 99 | 100 | ### Split string into an array by delimeter 101 | 102 | ```bash 103 | str="bla@some.com;john@home.com" 104 | arr=(${str//;/ }) 105 | echo ${arr[0]} 106 | echo ${arr[1]} 107 | ``` 108 | 109 | 110 | ### String starts with 111 | 112 | ```bash 113 | # The == comparison operator behaves differently within a double-brackets 114 | # test than within single brackets. 115 | 116 | [[ $a == z* ]] # True if $a starts with a "z" (wildcard matching). 117 | [[ $a == "z*" ]] # True if $a is equal to z* (literal matching). 118 | ``` 119 | 120 | 121 | ### Free space available in current directory 122 | 123 | ```bash 124 | df -Ph . | tail -1 | awk '{print $4}' 125 | ``` 126 | 127 | 128 | ### Sizes of directories 129 | 130 | ```bash 131 | du -sh ./* 132 | ``` 133 | 134 | 135 | ### Or, and, not in if 136 | 137 | ```bash 138 | a=true 139 | b=false 140 | 141 | if ${a} || ${b}; then 142 | echo true 143 | fi 144 | 145 | if ${a} && ! ${b}; then 146 | echo true 147 | fi 148 | ``` 149 | 150 | 151 | ### Numeric equality 152 | 153 | ```bash 154 | if (( var == 3 )); then 155 | echo "yes" 156 | fi 157 | ``` 158 | 159 | 160 | ### Mod 161 | 162 | ```bash 163 | $((i%n_per_session)) 164 | ``` 165 | 166 | 167 | ### Float comparison 168 | 169 | ```bash 170 | num1=10 171 | num2=0.1 172 | if (( $(echo "$num1 > $num2" | bc -l) )); then 173 | : 174 | fi 175 | ``` 176 | 177 | 178 | ### String equality 179 | 180 | ```bash 181 | if [[ ${str} == 3 ]]; then 182 | echo "yes" 183 | fi 184 | ``` 185 | 186 | 187 | ### Check if directory exists 188 | 189 | ```bash 190 | if [ -d "$DIRECTORY" ]; then 191 | # Will enter here if $DIRECTORY exists, even if it contains spaces 192 | fi 193 | ``` 194 | 195 | 196 | ### Check if file exists 197 | 198 | ```bash 199 | if [[ -e "$FILE" ]]; then 200 | # Will enter here if $FILE exists, even if it contains spaces 201 | fi 202 | ``` 203 | 204 | 205 | ### Logical operators 206 | 207 | ```bash 208 | if (! (( var == 2 || var < 30 ))) && [[ ${str} == abc ]]; then 209 | # do something 210 | fi 211 | ``` 212 | 213 | 214 | ### Loop through only directories 215 | 216 | ```bash 217 | for f in *; do 218 | if [[ -d $f ]]; then 219 | # $f is a directory 220 | fi 221 | done 222 | ``` 223 | 224 | 225 | ### Else 226 | 227 | ```bash 228 | if [ "$seconds" -eq 0 ]; then 229 | timezone_string="Z" 230 | elif [ "$seconds" -gt 0 ]; then 231 | timezone_string=$(printf "%02d:%02d" $((seconds/3600)) $(((seconds / 60) % 60))) 232 | else 233 | echo "Unkown parameter" 234 | fi 235 | ``` 236 | 237 | 238 | ### Until string is empty 239 | 240 | ```bash 241 | pid=$(...) 242 | until [ -z "$pid" ]; do 243 | kill -9 "$pid" 244 | pid=$(...) 245 | done 246 | ``` 247 | 248 | 249 | ### Loop through elements in array 250 | 251 | ```bash 252 | clips=( "ballet11-2" "jogging" ) 253 | for clip in "${clips[@]}"; do 254 | frameDir=/data/clips/${clip} 255 | done 256 | ``` 257 | 258 | 259 | ### Loop through elements with index 260 | 261 | ```bash 262 | arr=( 7 12 16 ) 263 | for i in "${!arr[@]}"; do # for (( i = 0; i < ${#arr[@]}; i++ )); do 264 | printf "%s\t%s\n" "$i" "${arr[$i]}" 265 | done 266 | ``` 267 | 268 | 269 | ### Batch rename 270 | 271 | ```bash 272 | #!/usr/bin/env bash 273 | 274 | for cat_dir in ./jobs/*; do 275 | for obj_file in $cat_dir/*.m; do 276 | xpath=${obj_file%/*} 277 | xbase=${obj_file##*/} 278 | echo "$obj_file -> $xpath/prefix_$xbase" 279 | done 280 | done 281 | ``` 282 | 283 | 284 | ### Remove the first 5 characters from each filename 285 | 286 | ```bash 287 | rename -n 's/(.{5})(.*)$/$2/' * 288 | ``` 289 | The `-n` is for simulating; remove it to get the actual result. 290 | 291 | 292 | ### Remove last 4 characters from a string 293 | 294 | ```bash 295 | v="some string.rtf" 296 | v2=${v::-4} 297 | echo "$v --> $v2" 298 | ``` 299 | 300 | 301 | ### Remove fixed prefix/suffix from a string 302 | 303 | ```bash 304 | foo=${string#$prefix} 305 | foo=${string%$suffix} 306 | ``` 307 | There are also `##` and `%%`, which remove as much as possible if `$prefix` or `$suffix` contains wildcards. 308 | 309 | 310 | ### Copy every fourth file in a folder 311 | 312 | ```bash 313 | cp $(printf '%s\n' im???.jpg | awk 'NR%4 == 1') /some/place 314 | ``` 315 | 316 | 317 | ### Rename files in a folder to sequential numbers 318 | 319 | ```bash 320 | a=1; for i in *.png; do new=$(printf "%03d.png" "$a"); mv -- "$i" "$new"; let a=a+1; done 321 | ``` 322 | 323 | 324 | ### Check if element in array 325 | 326 | 327 | ```bash 328 | array_contains2 () { 329 | local array="$1[@]" 330 | local seeking=$2 331 | local in=1 332 | for element in "${!array}"; do 333 | if [[ $element == $seeking ]]; then 334 | in=0 335 | break 336 | fi 337 | done 338 | return $in 339 | } 340 | array_contains2 arr "a b" && echo yes || echo no # no 341 | array_contains2 arr "d e" && echo yes || echo no # yes 342 | ``` 343 | 344 | 345 | ### Array length, indices, contents 346 | 347 | ```bash 348 | a=(1 1 2 3 4) 349 | echo ${a[@]} 350 | echo ${!a[@]} 351 | echo ${#a[@]} 352 | ``` 353 | 354 | 355 | ### Append to array 356 | 357 | ```bash 358 | ARRAY=() 359 | ARRAY+=('foo') 360 | ARRAY+=('bar') 361 | echo ${ARRAY[0]} 362 | ``` 363 | 364 | 365 | ### Assign command output to variable 366 | 367 | ```bash 368 | n=$(ls *.py | wc -l) 369 | ``` 370 | 371 | 372 | ### Extract the n-th line of file 373 | 374 | ```bash 375 | sed '68q;d' param1.csv 376 | ``` 377 | 378 | 379 | ### Block comment 380 | 381 | ```bash 382 | #!/bin/bash 383 | echo before comment 384 | : <<-'END' 385 | bla bla 386 | blurfl 387 | END 388 | echo after comment 389 | ``` 390 | 391 | 392 | ### Kill all jobs by me 393 | 394 | ```bash 395 | pkill -9 -u `id -u xiuming` 396 | ``` 397 | 398 | 399 | ### Pretty print `$PATH` 400 | 401 | ```bash 402 | tr ':' '\n' <<< "$PATH" 403 | ``` 404 | 405 | 406 | ### Download repo from GitHub 407 | 408 | ```bash 409 | git clone git://github.com/vim/vim.git 410 | ``` 411 | 412 | 413 | ### Locate a binary 414 | 415 | ```bash 416 | type -a python 417 | ``` 418 | 419 | 420 | ### Download a file from web 421 | 422 | ```bash 423 | wget http://website.com/files/file.zip 424 | ``` 425 | 426 | 427 | ### Determine contents of binary files 428 | 429 | ```bash 430 | strings /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep GLIBCXX 431 | ``` 432 | 433 | 434 | ### Arithmetic operations 435 | 436 | ```bash 437 | idx=$((idx+1)) 438 | ``` 439 | 440 | 441 | ### Wildcard exluding a pattern 442 | 443 | ```bash 444 | find /path/ -name 'foo*.png' -a ! -name 'bar*.png' 445 | ``` 446 | 447 | Filenames are NOT ordered. 448 | 449 | 450 | ### Collect filenames into an array 451 | 452 | ```bash 453 | files=(/path/*.pkl) 454 | ``` 455 | 456 | 457 | ### Find latest file in directory 458 | 459 | ```bash 460 | ls -t *.png | head -1 461 | ``` 462 | 463 | Sort by time with `-t`. Grab the first (newest) with `head -1`. 464 | 465 | 466 | ### Release a port 467 | 468 | ```bash 469 | lsof -i : 470 | kill -9 471 | ``` 472 | 473 | 474 | ### Extract a column 475 | 476 | ```bash 477 | 30-103-178:~ x$ lsof -i :5901 478 | COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME 479 | ssh 3751 x 6u IPv6 0x92a5ebb41be479d3 0t0 TCP localhost:5901 (LISTEN) 480 | ssh 3751 x 7u IPv4 0x92a5ebb42075b35b 0t0 TCP localhost:5901 (LISTEN) 481 | 30-103-178:~ x$ lsof -i :5901 | tail -n1 | awk '{print $2;}' 482 | 3751 483 | ``` 484 | 485 | 486 | ### Single quotes inside single quotes 487 | 488 | ```bash 489 | alias rxvt='urxvt -fg '"'"'#111111'"'"' -bg '"'"'#111111'"'" 490 | # ^^^^^ ^^^^^ ^^^^^ ^^^^ 491 | # 12345 12345 12345 1234 492 | ``` 493 | 494 | 3 `'` is the quoted character. 495 | 496 | 497 | ### Redirect (or slicence) `stdout` and `stderr` 498 | 499 | ```bash 500 | # Send stdout to out.log, stderr to err.log 501 | myprogram > out.log 2> err.log 502 | 503 | # Send both stdout and stderr to out.log 504 | myprogram &> out.log # New bash syntax 505 | myprogram > out.log 2>&1 # Older sh syntax 506 | 507 | # Log output, hide errors. 508 | myprogram > out.log 2> /dev/null 509 | 510 | # Hide both 511 | myprogram &> /dev/null 512 | ``` 513 | 514 | 515 | ### Get image dimensions 516 | 517 | ```bash 518 | wxh=$(identify "$im_f" | cut -d' ' -f3) 519 | w=$(echo "$wxh" | awk -F'x' '{print $1}') 520 | h=$(echo "$wxh" | awk -F'x' '{print $2}') 521 | ``` 522 | 523 | 524 | ### SSH timeout 525 | 526 | ```bash 527 | ssh -o ConnectTimeout=10 -o BatchMode=yes -o StrictHostKeyChecking=no -q ${user}@vision${ID}.csail.mit.edu "exit" 528 | ``` 529 | 530 | `-o ConnectTimeout=10` sets timeout to be 10 seconds, `-o BatchMode=yes` keeps SSH from hanging with an unknown host and adds it to `known_host`, and `-o StrictHostKeyChecking=no` adds the fingerprint automatically (be careful!). 531 | 532 | 533 | ### Case 534 | 535 | ```bash 536 | case ${clip} in 537 | "ballet1") 538 | eval "${cmd1}" 539 | ;; 540 | *) 541 | eval "${cmd2}" 542 | ;; 543 | esac 544 | ``` 545 | 546 | 547 | ### Fast way of removing large/deep folder 548 | 549 | ```bash 550 | mkdir empty_dir; rsync -a --delete empty_dir/ your_dir/; rm -rf empty_dir/ 551 | ``` 552 | 553 | 554 | ### Shuffle array 555 | 556 | ```bash 557 | arr=( 1 2 3 4 ) 558 | arr_shuffled=( $(echo "${arr[@]}" | sed -r 's/(.[^;]*;)/ \1 /g' | tr " " "\n" | shuf | tr -d " " ) ) 559 | echo "${arr_shuffled[@]}" 560 | 2 4 3 1 561 | ``` 562 | 563 | 564 | ### `rsync` 565 | 566 | ```bash 567 | rsync -avzhP --dry-run xiuming@visiongpu20.csail.mit.edu:/data/vision/billf/webCNN/xiuming ~/Desktop/output 568 | ``` 569 | 570 | `-v`: verbose 571 | 572 | `-r`: copies data recursively (but don’t preserve timestamps and permission while transferring data) 573 | 574 | `-a`: archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownerships and timestamps 575 | 576 | `-z`: compress file data 577 | 578 | `-h`: human-readable, output numbers in a human-readable format 579 | 580 | `-P`: show progress 581 | 582 | `--remove-source-files` 583 | 584 | 585 | ### Zip 586 | 587 | ```bash 588 | zip -j myzip file1 file2 file3 589 | ``` 590 | stores just the names of the files, and does not store directory names (junks the paths). 591 | 592 | ```bash 593 | cd 594 | zip -r myzip 595 | ``` 596 | recursively zips the files, maintaining the file structure. 597 | 598 | 599 | ### Unzip 600 | 601 | ```bash 602 | tar -xvzf images.tar.gz -C /target/root/ 603 | ``` 604 | 605 | `f`: this must be the last flag of the command, and the tar file must be immediately after. It tells tar the name and path of the compressed file 606 | 607 | `z`: tells tar to decompress the archive using g**z**ip 608 | 609 | `x`: tar can collect files or e**x**tract them. `x` does the latter 610 | 611 | `v`: verbose 612 | 613 | or 614 | 615 | ```bash 616 | tar -xvf yourfile.tar 617 | ``` 618 | 619 | or 620 | 621 | ```bash 622 | unzip yourfile.zip [-d /dst_folder/] 623 | ``` 624 | 625 | 626 | ### Move all but one file 627 | 628 | ```bash 629 | mv !(fileOne) ~/path/newFolder 630 | ``` 631 | 632 | with `shopt -s extglob` in `.bashrc` 633 | 634 | 635 | ### Sort results by `find` and print filename only 636 | 637 | ```bash 638 | find ./ -maxdepth 1 -name '*.pkl' -printf "%f\n" | sort | head -1 639 | ``` 640 | 641 | 642 | ### Show current libraries 643 | 644 | ```bash 645 | ldconfig -p | grep libx264.so 646 | ``` 647 | 648 | 649 | ### Show library linking 650 | 651 | ```bash 652 | ldd /path/to/the/problematic/so 653 | ``` 654 | 655 | 656 | ### Replace character in string with another character 657 | 658 | ```bash 659 | foo="1 2 3" 660 | bar=${foo/ /-} 661 | ``` 662 | 663 | replaces first blank only. 664 | 665 | ```bash 666 | bar=${foo// /-} 667 | ``` 668 | 669 | replaces all blanks. 670 | 671 | 672 | ### Slice array 673 | 674 | ```bash 675 | $ foo=(q w e r t y u) 676 | $ echo "${foo[@]:0:4}" 677 | q w e r 678 | ``` 679 | 680 | 681 | ### Dictionary 682 | 683 | ```bash 684 | declare -A dict=( [key1]="1,2,3 a,b,c" [key2]="1 x" ) 685 | 686 | key=key1 687 | echo ${dict[${key}]} 688 | ``` 689 | 690 | 691 | ### Read space-delimited strings into array 692 | 693 | ```bash 694 | read -r -a arr <<< "first,string second,string" 695 | 696 | echo ${arr[0]} 697 | echo ${arr[1]} 698 | ``` 699 | 700 | 701 | ### Read lines of file into array 702 | 703 | ```bash 704 | readarray a < /path/to/filename 705 | ``` 706 | 707 | 708 | ### First N characters of a string variable 709 | 710 | ```bash 711 | a=g16 712 | echo ${a:0:1} 713 | echo ${a:0:2} 714 | ``` 715 | 716 | 717 | ### Unpack elements from string 718 | 719 | ```bash 720 | arr=( 1 2 3 4 ) 721 | read -r e1 e2 e3 _ <<< "${arr}" 722 | 723 | echo ${e1} 724 | echo ${e2} 725 | echo ${e3} 726 | ``` 727 | 728 | 729 | ### Parse elements by delimiter 730 | 731 | ```bash 732 | strarr=1,2,3,4 733 | IFS=, read -r e1 e2 e3 _ <<< "${strarr}" 734 | ``` 735 | 736 | 737 | ### Loop through lines in file 738 | 739 | ```bash 740 | while read c; do 741 | ssh -n "$me@$host" "$c; exit" & 742 | done 771 | kill -9 772 | ``` 773 | 774 | 775 | ### Print formatted string to variable 776 | 777 | ```bash 778 | printf -v my_str "%03d" "${my_int}" 779 | ``` 780 | 781 | Useful for prepending zeros to integers. 782 | 783 | 784 | ### Check if string is valid as an integer 785 | 786 | ```bash 787 | [[ $var =~ ^-?[0-9]+$ ]] 788 | ``` 789 | 790 | 791 | ### Loop through files in reverse order 792 | 793 | ```bash 794 | files=(/var/logs/foo*.log) 795 | for ((i=${#files[@]}-1; i>=0; i--)); do 796 | bar "${files[$i]}" 797 | done 798 | ``` 799 | 800 | 801 | ### Show image info 802 | 803 | ```bash 804 | file /path/to/img.png 805 | ``` 806 | 807 | 808 | ### Unique elements of array 809 | 810 | ```bash 811 | sorted_unique_ids=($(echo "${ids[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' ')) 812 | ``` 813 | 814 | 815 | ### Substring 816 | 817 | ```bash 818 | string='My long string' 819 | if [[ $string == *"My long"* ]]; then 820 | echo "It's there!" 821 | fi 822 | ``` 823 | 824 | 825 | ### `ls` with error handling 826 | 827 | ```bash 828 | if ls "${out_dir}"/*.blend 1> /dev/null 2>&1; then 829 | first_file=$(ls "${out_dir}"/*.blend | head -1) 830 | else 831 | first_file=none 832 | fi 833 | ``` 834 | 835 | 836 | ### Number range in regex 837 | 838 | ```bash 839 | cp 0[2-9]?.npz ./ 840 | ``` 841 | copies `020.npz`, `021.npz`, ..., `030.npz`, ..., `099.npz`. 842 | 843 | 844 | ### Generate a number sequence 845 | 846 | ```bash 847 | me@host:~$ vals=($(seq 0 1 10)) 848 | me@host:~$ echo "${vals[@]}" 849 | 0 1 2 3 4 5 6 7 8 9 10 850 | 851 | me@host:~$ arr=(./*.png) 852 | me@host:~$ echo ${#arr[@]} 853 | 10 854 | me@host:~$ ind=($(seq 0 1 $((${#arr[@]}-1)))) 855 | me@host:~$ echo "${vals[@]}" 856 | 0 1 2 3 4 5 6 7 8 9 857 | ``` 858 | 859 | 860 | ### Check `sudo` commands on machine 861 | 862 | ```bash 863 | sudo cat /var/log/auth.log | grep sudo 864 | ``` 865 | 866 | 867 | ### Replace string in file 868 | ```bash 869 | sed -i -e 's/abc/XYZ/g' /tmp/file.txt 870 | ``` 871 | replaces `abc` with `XYZ` in file `/tmp/file.txt`. 872 | 873 | 874 | ### Return Python list to bash 875 | 876 | ```bash 877 | result_arr=($(python script.py param1 param2 | tr -d '[],')) 878 | echo "${result_arr[@]}" 879 | ``` 880 | 881 | 882 | ### Strip single quotes for each element of an array 883 | 884 | ```bash 885 | abc=(${abc[@]//\'/}) 886 | ``` 887 | 888 | 889 | ### Call functions defined in `.bashrc` 890 | 891 | Do any of the following. 892 | 893 | * Source `.bashrc` explicitly: 894 | ```bash 895 | #!/bin/bash 896 | . ~/.bashrc 897 | exr2npz "foo" "bar" 898 | ``` 899 | 900 | * Start bash with the interactive flag: 901 | ```bash 902 | #!/bin/bash -i 903 | exr2npz "foo" "bar" 904 | ``` 905 | 906 | * Set `BASH_ENV` when you start your script: 907 | ```bash 908 | BASH_ENV=$HOME/.bashrc /path/to/my/script 909 | ``` 910 | 911 | 912 | ### When the username gets truncated by `+` in `top` 913 | 914 | Use the "long-username version" of `ps -aux` 915 | 916 | ```bash 917 | ps axo user:20,pid,pcpu,pmem,vsz,rss,tty,stat,start,time,comm 918 | ``` 919 | 920 | 921 | ### Send a command to run in background on a remote machine 922 | 923 | ```bash 924 | ssh -n -f user@host "sh -c 'cd /whereever; nohup ./whatever > /dev/null 2>&1 &'" 925 | ``` 926 | --------------------------------------------------------------------------------