├── .devcontainer └── devcontainer.json ├── .github └── workflows │ └── check.yml ├── .gitignore ├── Makefile ├── README.md ├── TODO ├── pyproject.toml ├── replace.py ├── requirements-dev.txt └── test_replace.py /.devcontainer/devcontainer.json: -------------------------------------------------------------------------------- 1 | // https://aka.ms/devcontainer.json 2 | { 3 | "name": "replace dev", 4 | "image": "ghcr.io/aureliojargas/devcontainer", 5 | "remoteUser": "vscode" 6 | } 7 | -------------------------------------------------------------------------------- /.github/workflows/check.yml: -------------------------------------------------------------------------------- 1 | # https://docs.github.com/en/actions/reference 2 | # https://docs.github.com/en/actions/guides/building-and-testing-python 3 | 4 | name: Check 5 | 6 | on: 7 | pull_request: 8 | push: 9 | branches: [main] 10 | 11 | jobs: 12 | lint: 13 | runs-on: ubuntu-latest 14 | steps: 15 | - uses: actions/checkout@v3 16 | - uses: actions/setup-python@v3 17 | with: 18 | python-version: '3.x' 19 | - run: pip install ruff 20 | - run: make lint 21 | 22 | test: 23 | runs-on: ubuntu-latest 24 | steps: 25 | - uses: actions/checkout@v3 26 | - uses: actions/setup-python@v3 27 | with: 28 | python-version: '3.x' 29 | - run: pip install pytest 30 | - run: make test 31 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /.pytest_cache/ 2 | /.ruff_cache/ 3 | /__pycache__/ 4 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .PHONY: check clean fmt lint test 2 | 3 | PYTHON_FILES = replace.py test_replace.py 4 | 5 | check: lint test 6 | 7 | fmt: 8 | ruff format $(PYTHON_FILES) 9 | 10 | lint: 11 | ruff check $(PYTHON_FILES) 12 | ruff format --check --diff $(PYTHON_FILES) 13 | 14 | test: clitest 15 | pytest 16 | bash ./clitest --progress none README.md 17 | 18 | clitest: 19 | curl --location --remote-name --silent \ 20 | https://raw.githubusercontent.com/aureliojargas/clitest/master/clitest 21 | 22 | clean: 23 | rm -f clitest 24 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # replace 2 | 3 | Generic file search & replace tool, written in Python. 4 | 5 | ## Options 6 | 7 | Specify the FROM/TO patterns directly in the command line: 8 | 9 | ``` 10 | -f, --from TEXT specify the search text or regex 11 | -t, --to TEXT specify the replacement text 12 | ``` 13 | 14 | Specify the FROM/TO patterns using files, useful for multiline matches: 15 | 16 | ``` 17 | -F, --from-file FILE read the search text from this file 18 | -T, --to-file FILE read the replacement text from this file 19 | ``` 20 | 21 | The default search uses simple string matching, no magic. 22 | But if you need power, there you have it: 23 | 24 | ``` 25 | -r, --regex use regex matching instead of string matching 26 | ``` 27 | 28 | Just like `sed`, this script by default show results to STDOUT. 29 | But instead, you can save the edits to the original file: 30 | 31 | ``` 32 | -i, --in-place edit files in-place 33 | ``` 34 | 35 | 36 | ## Examples 37 | 38 | ```bash 39 | # Replace all mentions of old.css with new.css in all HTML files 40 | replace --from old.css --to new.css --in-place *.html 41 | 42 | # Update the AdSense code in all HTML files 43 | # The old and the new code are in separate files 44 | replace --from-file adsense.old --to-file adsense.new -i *.html 45 | 46 | # Enclose all numbers inside square brackets: 123 -> [123] 47 | replace --regex --from '(\d+)' --to '[\\1]' file.txt 48 | 49 | # From http to https in all HTML files, recursive 50 | find . -type f -name "*.html" \ 51 | -exec replace \ 52 | -f 'http://example.com' \ 53 | -t 'https://example.com' \ 54 | -i {} \; 55 | ``` 56 | 57 | ## Tests 58 | 59 | The following command lines are executed and verified by [clitest](https://github.com/aureliojargas/clitest), using the `clitest README.md` command. 60 | 61 | First, setup a sample text file: 62 | 63 | ```console 64 | $ echo 'the quick brown fox' > file.txt 65 | $ cat file.txt 66 | the quick brown fox 67 | $ 68 | ``` 69 | 70 | Now we'll do some replaces using string matching, which is the default. Note that there are short and long options (`-f`/`--from`) and that the replacement is performed globally: all occurrences are replaced. 71 | 72 | ```console 73 | $ ./replace.py --from 'brown' --to 'red' file.txt 74 | the quick red fox 75 | $ ./replace.py -f 'brown' -t 'red' file.txt 76 | the quick red fox 77 | $ ./replace.py -f 'o' -t '◆' file.txt 78 | the quick br◆wn f◆x 79 | $ 80 | ``` 81 | 82 | For more powerfull searches, use `-r` or `--regex` to perform a regular expression match. You have access to the full power of Python's regex flavor. 83 | 84 | ```console 85 | $ ./replace.py --regex -f '[aeiou]' -t '◆' file.txt 86 | th◆ q◆◆ck br◆wn f◆x 87 | $ ./replace.py -r -f '[aeiou]' -t '◆' file.txt 88 | th◆ q◆◆ck br◆wn f◆x 89 | $ 90 | ``` 91 | 92 | If necessary, you can also apply the replacements on text coming from STDIN, using `-` as the file name. 93 | 94 | ```console 95 | $ cat file.txt | ./replace.py -r -f '[aeiou]' -t '◆' - 96 | th◆ q◆◆ck br◆wn f◆x 97 | $ 98 | ``` 99 | 100 | Note that all the previous replaces were not saved to the original file. This is the default behavior (just like `sed`). If you want to edit the original file, use the `-i` or `--in-place` options: 101 | 102 | ```console 103 | $ ./replace.py -r -f '[aeiou]' -t '◆' -i file.txt 104 | Saved file.txt 105 | $ cat file.txt 106 | th◆ q◆◆ck br◆wn f◆x 107 | $ 108 | ``` 109 | 110 | Some boring tests for missing or incomplete command line options: 111 | 112 | ```console 113 | $ ./replace.py 2>&1 | grep error 114 | replace.py: error: the following arguments are required: FILE 115 | $ ./replace.py README.md 116 | Error: No search pattern (use --from or --from-file) 117 | $ ./replace.py -f '' README.md 118 | Error: No search pattern (use --from or --from-file) 119 | $ ./replace.py -f foo README.md 120 | Error: No replace pattern (use --to or --to-file) 121 | $ 122 | ``` 123 | 124 | OK, we're done for now. 125 | 126 | ```console 127 | $ rm file.txt 128 | $ 129 | ``` 130 | 131 | ## Similar tools 132 | 133 | - https://github.com/dmerejkowsky/replacer/ 134 | - https://github.com/facebook/codemod 135 | -------------------------------------------------------------------------------- /TODO: -------------------------------------------------------------------------------- 1 | -v to show when a file is NOT changed 2 | Change files by default, implement dry-run otherwise 3 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [tool.ruff] 2 | target-version = "py37" 3 | fix = true 4 | show-fixes = true 5 | 6 | [tool.ruff.format] 7 | quote-style = "single" 8 | 9 | [tool.ruff.lint] 10 | select = ["ALL"] 11 | ignore = [ 12 | "D", # pydocstyle 13 | "ANN", # flake8-annotations 14 | "T201", # `print` found 15 | 16 | # https://docs.astral.sh/ruff/formatter/#conflicting-lint-rules 17 | "W191", # tab-indentation 18 | "E111", # indentation-with-invalid-multiple 19 | "E114", # indentation-with-invalid-multiple-comment 20 | "E117", # over-indented 21 | "D206", # indent-with-spaces 22 | "D300", # triple-single-quotes 23 | "Q000", # bad-quotes-inline-string 24 | "Q001", # bad-quotes-multiline-string 25 | "Q002", # bad-quotes-docstring 26 | "Q003", # avoidable-escaped-quote 27 | "COM812", # missing-trailing-comma 28 | "COM819", # prohibited-trailing-comma 29 | "ISC001", # single-line-implicit-string-concatenation 30 | "ISC002", # multi-line-implicit-string-concatenation 31 | ] 32 | 33 | [tool.ruff.lint.per-file-ignores] 34 | "test_replace.py" = ["S101"] # Use of `assert` detected 35 | -------------------------------------------------------------------------------- /replace.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Generic search & replace tool 3 | # Aurelio Jargas, 2016-08-13 4 | 5 | """ 6 | Replaces text using string or regex matching. 7 | 8 | Examples: 9 | # Replace all mentions of old.css with new.css in all HTML files 10 | replace --from old.css --to new.css --in-place *.html 11 | 12 | # Update the AdSense code in all HTML files 13 | # The old and the new code are in separate files 14 | replace --from-file adsense.old --to-file adsense.new -i *.html 15 | 16 | # Enclose all numbers inside square brackets: 123 -> [123] 17 | replace --regex --from '(\\d+)' --to '[\\1]' file.txt 18 | """ 19 | 20 | import argparse 21 | import pathlib 22 | import re 23 | import sys 24 | 25 | 26 | # read/write files using bytes to avoid line break issues 27 | def read_file(path): 28 | if str(path) == '-': 29 | return sys.stdin.read() 30 | return path.read_bytes().decode(encoding='utf-8') 31 | 32 | 33 | def save_file(path, content): 34 | path.write_bytes(bytes(content, encoding='utf-8')) 35 | 36 | 37 | def setup_cmdline_parser(): 38 | parser = argparse.ArgumentParser( 39 | description=__doc__, 40 | formatter_class=argparse.RawDescriptionHelpFormatter, 41 | ) 42 | 43 | # from 44 | group = parser.add_mutually_exclusive_group() 45 | group.add_argument( 46 | '-f', 47 | '--from', 48 | metavar='TEXT', 49 | dest='from_', 50 | help='specify the search text or regex', 51 | ) 52 | group.add_argument( 53 | '-F', 54 | '--from-file', 55 | metavar='FILE', 56 | type=pathlib.Path, 57 | help='read the search text from this file', 58 | ) 59 | 60 | # to 61 | group = parser.add_mutually_exclusive_group() 62 | group.add_argument( 63 | '-t', 64 | '--to', 65 | metavar='TEXT', 66 | help='specify the replacement text', 67 | ) 68 | group.add_argument( 69 | '-T', 70 | '--to-file', 71 | metavar='FILE', 72 | type=pathlib.Path, 73 | help='read the replacement text from this file', 74 | ) 75 | 76 | # other 77 | parser.add_argument( 78 | '-r', 79 | '--regex', 80 | action='store_true', 81 | help='use regex matching instead of string matching', 82 | ) 83 | parser.add_argument( 84 | '-i', 85 | '--in-place', 86 | action='store_true', 87 | help='edit files in-place', 88 | ) 89 | parser.add_argument( 90 | '-v', 91 | '--verbose', 92 | action='store_true', 93 | help='turn on verbose mode', 94 | ) 95 | 96 | # files 97 | parser.add_argument( 98 | 'files', 99 | metavar='FILE', 100 | nargs='+', 101 | type=pathlib.Path, 102 | help='input files', 103 | ) 104 | return parser 105 | 106 | 107 | def validate_config(config): 108 | # Set search pattern 109 | if config.from_file: 110 | config.from_value = read_file(config.from_file) 111 | elif config.from_: 112 | config.from_value = config.from_ 113 | else: 114 | sys.exit('Error: No search pattern (use --from or --from-file)') 115 | 116 | # Set replacement 117 | if config.to_file: 118 | config.to_value = read_file(config.to_file) 119 | elif config.to is not None: # could also be '' 120 | config.to_value = config.to 121 | else: 122 | sys.exit('Error: No replace pattern (use --to or --to-file)') 123 | 124 | 125 | def replace(from_, to_, text, use_regex): 126 | if use_regex: 127 | return re.sub(from_, to_, text) 128 | return text.replace(from_, to_) 129 | 130 | 131 | def main(args=None): 132 | parser = setup_cmdline_parser() 133 | config = parser.parse_args(args) 134 | validate_config(config) 135 | 136 | from_ = config.from_value 137 | to_ = config.to_value 138 | 139 | for input_file in config.files: 140 | if config.verbose: 141 | print('----', input_file) 142 | 143 | original = read_file(input_file) 144 | modified = replace(from_, to_, original, config.regex) 145 | 146 | # save or show results 147 | if config.in_place: 148 | if modified == original: 149 | continue # do not save unchanged files 150 | save_file(input_file, modified) 151 | print('Saved', input_file) 152 | else: 153 | print(modified, end='') 154 | 155 | 156 | if __name__ == '__main__': 157 | main() 158 | -------------------------------------------------------------------------------- /requirements-dev.txt: -------------------------------------------------------------------------------- 1 | pytest 2 | ruff 3 | -------------------------------------------------------------------------------- /test_replace.py: -------------------------------------------------------------------------------- 1 | import io 2 | import pathlib 3 | import tempfile 4 | 5 | import pytest 6 | 7 | import replace 8 | 9 | 10 | def test_read_from_stdin(monkeypatch): 11 | """ 12 | Using '-' as the filename, we should read the input text from STDIN. 13 | """ 14 | path = pathlib.Path('-') 15 | text = 'foo\nbar\n' 16 | 17 | monkeypatch.setattr('sys.stdin', io.StringIO(text)) 18 | assert replace.read_file(path) == text 19 | 20 | 21 | @pytest.mark.parametrize( 22 | ('text', 'from_', 'to_', 'use_regex', 'expected'), 23 | [ 24 | # from_ not found 25 | ('foobar', '404', 'new', False, 'foobar'), 26 | ('foobar', '404', 'new', True, 'foobar'), 27 | # happy path 28 | ('foobar', 'foo', 'new', False, 'newbar'), 29 | ('foobar', 'fo+', 'new', True, 'newbar'), 30 | # to_ is empty 31 | ('foobar', 'foo', '', False, 'bar'), 32 | ('foobar', 'fo+', '', True, 'bar'), 33 | # the replace is always global 34 | ('foobar', 'o', '.', False, 'f..bar'), 35 | ('foobar', 'o', '.', True, 'f..bar'), 36 | ], 37 | ) 38 | def test_replace(text, from_, to_, use_regex, expected): 39 | assert replace.replace(from_, to_, text, use_regex) == expected 40 | 41 | 42 | @pytest.mark.parametrize( 43 | 'text', 44 | [ 45 | # No line break at EOF 46 | '', 47 | ' ', 48 | '1\n2', 49 | '1\r2', 50 | '1\r\n2', 51 | # Line break at EOF 52 | '1\n2\n', 53 | '1\r\n2\r\n', 54 | '1\r2\r', 55 | # Mixed-style line breaks 56 | '1\n2\r3\r\n', 57 | # Line break-only 58 | '\n', 59 | '\r', 60 | '\r\n', 61 | ], 62 | ) 63 | def test_keep_original_line_breaks(text): 64 | """ 65 | No matter how weird the original file is, we should never "normalize" the line 66 | breaks or loose data when reading or writing it. See also issue #2. 67 | """ 68 | path = pathlib.Path(tempfile.mkstemp()[1]) 69 | 70 | replace.save_file(path, text) 71 | assert replace.read_file(path) == text 72 | 73 | path.unlink() 74 | --------------------------------------------------------------------------------