├── .gitignore ├── .travis.yml ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── Makefile ├── README.md ├── codemod ├── __init__.py ├── base.py ├── helpers.py ├── patch.py ├── position.py ├── query.py └── terminal_helper.py ├── dev-requirements.txt ├── pytest.ini └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | # https://github.com/github/gitignore/blob/4a9d41f8947b9fe5850914abb7e59a59e1ae2cb6/Python.gitignore 2 | 3 | # Byte-compiled / optimized / DLL files 4 | __pycache__/ 5 | *.py[cod] 6 | 7 | # C extensions 8 | *.so 9 | 10 | # Distribution / packaging 11 | .Python 12 | env/ 13 | build/ 14 | develop-eggs/ 15 | dist/ 16 | downloads/ 17 | eggs/ 18 | .eggs/ 19 | lib/ 20 | lib64/ 21 | parts/ 22 | sdist/ 23 | var/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | 47 | # Translations 48 | *.mo 49 | *.pot 50 | 51 | # Django stuff: 52 | *.log 53 | 54 | # Sphinx documentation 55 | docs/_build/ 56 | 57 | # PyBuilder 58 | target/ 59 | 60 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | 3 | python: 4 | - "2.7" 5 | - "3.6" 6 | 7 | install: 8 | - "pip install -r dev-requirements.txt" 9 | script: make test 10 | 11 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please [read the full text](https://code.fb.com/codeofconduct) so that you can understand what actions will and will not be tolerated. -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to codemod 2 | We want to make contributing to this project as easy and transparent as 3 | possible. 4 | 5 | ## Pull Requests 6 | We actively welcome your pull requests. 7 | 8 | 1. Fork the repo and create your branch from `master`. 9 | 2. If you've added code that should be tested, add tests. 10 | 3. If you've changed APIs, update the documentation. 11 | 4. Ensure the test suite passes. 12 | 5. Make sure your code lints. 13 | 6. If you haven't already, complete the Contributor License Agreement ("CLA"). 14 | 15 | ## Contributor License Agreement ("CLA") 16 | In order to accept your pull request, we need you to submit a CLA. You only need 17 | to do this once to work on any of Facebook's open source projects. 18 | 19 | Complete your CLA here: 20 | 21 | ## Issues 22 | We use GitHub issues to track public bugs. Please ensure your description is 23 | clear and has sufficient instructions to be able to reproduce the issue. 24 | 25 | Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe 26 | disclosure of security bugs. In those cases, please go through the process 27 | outlined on that page and do not file a public issue. 28 | 29 | ## License 30 | By contributing to codemod, you agree that your contributions will be licensed 31 | under the LICENSE file in the root directory of this source tree. -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .PHONY: test install pep8 release clean 2 | 3 | test: pep8 4 | py.test --doctest-modules codemod 5 | 6 | install: 7 | python setup.py develop 8 | 9 | pep8: 10 | @flake8 codemod --ignore=F403 11 | 12 | release: test 13 | @python setup.py sdist upload 14 | 15 | clean: 16 | @find ./ -name '*.pyc' -exec rm -f {} \; 17 | @find ./ -name 'Thumbs.db' -exec rm -f {} \; 18 | @find ./ -name '*~' -exec rm -f {} \; 19 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | codemod 2 | ======= 3 | 4 | [![PyPI](https://img.shields.io/pypi/v/codemod.svg)](https://pypi.python.org/pypi/codemod) 5 | [![downloads](https://img.shields.io/pypi/dw/codemod.svg)](https://pypi.python.org/pypi/codemod) 6 | [![Travis CI](http://img.shields.io/travis/facebook/codemod.svg)](https://travis-ci.org/facebook/codemod) 7 | [![Code Health](https://landscape.io/github/rochacbruno/codemod/master/landscape.svg?style=flat)](https://landscape.io/github/rochacbruno/codemod/master) 8 | 9 | 10 | Overview 11 | -------- 12 | 13 | codemod is a tool/library to assist you with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention. 14 | 15 | Example: Let's say you're deprecating your use of the `` tag. From the command line, you might make progress by running: 16 | 17 | codemod -m -d /home/jrosenstein/www --extensions php,html \ 18 | '(.*?)' \ 19 | '\2' 20 | 21 | For each match of the regex, you'll be shown a colored diff, and asked if you want to accept the change (the replacement of the `` tag with a `` tag), reject it, or edit the line in question in your `$EDITOR` of choice. 22 | 23 | Install 24 | ------- 25 | In a virtual environment or as admin user 26 | 27 | `pip install codemod` 28 | 29 | or system wide with sudo 30 | 31 | `sudo -H pip install codemod` 32 | 33 | Usage 34 | ----- 35 | 36 | The last two arguments are a regular expression to match and a substitution string, respectively. Or you can omit the substitution string, and just be prompted on each match for whether you want to edit in your editor. 37 | 38 | Options (all optional) include: 39 | 40 | -m 41 | Have regex work over multiple lines (e.g. have dot match newlines). By 42 | default, codemod applies the regex one line at a time. 43 | -d 44 | The path whose ancestor files are to be explored. Defaults to current dir. 45 | -i 46 | Make your search case-insensitive 47 | --start 48 | A path:line_number-formatted position somewhere in the hierarchy from which 49 | to being exploring, or a percentage (e.g. "--start 25%") of the way through 50 | to start. Useful if you're divvying up the substitution task across 51 | multiple people. 52 | --end 53 | A path:line_number-formatted position somewhere in the hierarchy just 54 | *before* which we should stop exploring, or a percentage of the way 55 | through, just before which to end. 56 | --extensions 57 | A comma-delimited list of file extensions to process. Also supports Unix 58 | pattern matching. 59 | --include-extensionless 60 | If set, this will check files without an extension, along with any 61 | matching file extensions passed in --extensions 62 | --accept-all 63 | Automatically accept all changes (use with caution) 64 | --default-no 65 | Set default behavior to reject the change. 66 | --editor 67 | Specify an editor, e.g. "vim" or "emacs". If omitted, defaults to $EDITOR 68 | environment variable. 69 | --count 70 | Don't run normally. Instead, just print out number of times places in the 71 | codebase where the 'query' matches. 72 | --test 73 | Don't run normally. Instead, just run the unit tests embedded in the 74 | codemod library. 75 | 76 | You can also use codemod for transformations that are much more sophisticated than regular expression substitution. Rather than using the command line, you write Python code that looks like: 77 | 78 | import codemod 79 | codemod.Query(...).run_interactive() 80 | 81 | See the documentation for the Query class for details. 82 | 83 | Background 84 | ---------- 85 | 86 | *Announcement by Justin Rosenstein on Facebook Notes, circa December 2008* 87 | 88 | Part of why most code -- and most software -- sucks so much is that making sweeping changes is hard. 89 | 90 | Let's say that a month ago you wrote a function that you -- or your entire company -- have been using frequently. And now you decide to change its name, or change the order of its parameters, or split it up into two separate functions and then have half the call sites use the old one and half the call sites use the new one, or change its return type from a scalar to a structure with additional information. IDEs and standard \*nix tools like sed can help, but you typically have to make a trade-off between introducing errors and introducing tedium. The result, all too often, is that we decide (often unconsciously) that the sweeping change just isn't worth it, and leave the undesirable pattern untouched for future versions of ourselves and others to grumble about, while the pattern grows more and more endemic to the code base. 91 | 92 | What you really want is to be able to describe an arbitrary transform -- using either regexes in the 80% case or Python code for more complex transformations -- that matches for lines (or sets of lines) of source code and converts them to something more desirable, but then have a tool that will show you each of the change sites one at a time and ask you either to accept the change, reject the change, or manually intervene using your editor of choice. 93 | 94 | So, while at Facebook, I wrote a script that does exactly that. codemod.py a nifty little utility/library to assist with codebase refactors that can be partially automated but still require human oversight and occasional intervention. And, thanks to help from Mr. David Fetterman, codemod is now open source. Check it out (so to speak): 95 | 96 | git clone git://github.com/facebook/codemod.git 97 | (previously svn checkout https://codemod.svn.sourceforge.net/svnroot/codemod/trunk codemod) 98 | 99 | It's one of those tools where, the more you use it, the more you think of places to use it -- and the more you realize how much you were compromising the quality of your code because reconsidering heavily-used code patterns sounded just too damn annoying. I use it pretty much every day. 100 | 101 | Dependencies 102 | ------------ 103 | 104 | * python2 105 | 106 | Credits 107 | ------- 108 | 109 | Copyright (c) 2007-2008 Facebook. 110 | 111 | Created by Justin Rosenstein. 112 | 113 | Licensed under the Apache License, Version 2.0. 114 | 115 | -------------------------------------------------------------------------------- /codemod/__init__.py: -------------------------------------------------------------------------------- 1 | from codemod.base import * # noqa 2 | -------------------------------------------------------------------------------- /codemod/base.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright (c) 2007-2008 Facebook 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | # 17 | # See accompanying file LICENSE. 18 | # 19 | # @author Justin Rosenstein 20 | 21 | from __future__ import print_function 22 | 23 | import argparse 24 | import os 25 | import re 26 | import sys 27 | import textwrap 28 | from math import ceil 29 | 30 | from codemod.patch import Patch 31 | from codemod.position import Position 32 | from codemod.query import Query 33 | import codemod.helpers as helpers 34 | import codemod.terminal_helper as terminal 35 | 36 | yes_to_all = False 37 | if sys.version_info[0] >= 3: 38 | unicode = str 39 | 40 | 41 | def run_interactive(query, editor=None, just_count=False, default_no=False): 42 | """ 43 | Asks the user about each patch suggested by the result of the query. 44 | 45 | @param query An instance of the Query class. 46 | @param editor Name of editor to use for manual intervention, e.g. 47 | 'vim' 48 | or 'emacs'. If omitted/None, defaults to $EDITOR 49 | environment variable. 50 | @param just_count If true: don't run normally. Just print out number of 51 | places in the codebase where the query matches. 52 | """ 53 | global yes_to_all 54 | 55 | # Load start from bookmark, if appropriate. 56 | bookmark = _load_bookmark() 57 | if bookmark: 58 | print('Resume where you left off, at %s (y/n)? ' 59 | % str(bookmark), end=' ') 60 | sys.stdout.flush() 61 | if (_prompt(default='y') == 'y'): 62 | query.start_position = bookmark 63 | 64 | # Okay, enough of this foolishness of computing start and end. 65 | # Let's ask the user about some one line diffs! 66 | print('Searching for first instance...') 67 | suggestions = query.generate_patches() 68 | 69 | if just_count: 70 | for count, _ in enumerate(suggestions): 71 | terminal.terminal_move_to_beginning_of_line() 72 | print(count, end=" ") 73 | sys.stdout.flush() # since print statement ends in comma 74 | print() 75 | return 76 | 77 | for patch in suggestions: 78 | _save_bookmark(patch.start_position) 79 | _ask_about_patch(patch, editor, default_no) 80 | print('Searching...') 81 | _delete_bookmark() 82 | if yes_to_all: 83 | terminal.terminal_clear() 84 | print( 85 | "You MUST indicate in your code review:" 86 | " \"codemod with 'Yes to all'\"." 87 | "Make sure you and other people review the changes.\n\n" 88 | "With great power, comes great responsibility." 89 | ) 90 | 91 | 92 | def line_transformation_suggestor(line_transformation, line_filter=None): 93 | """ 94 | Returns a suggestor (a function that takes a list of lines and yields 95 | patches) where suggestions are the result of line-by-line transformations. 96 | 97 | @param line_transformation Function that, given a line, returns another 98 | line 99 | with which to replace the given one. If the 100 | output line is different from the input line, 101 | the 102 | user will be prompted about whether to make the 103 | change. If the output is None, this means "I 104 | don't have a suggestion, but the user should 105 | still be asked if zhe wants to edit the line." 106 | @param line_filter Given a line, returns True or False. If False, 107 | a line is ignored (as if line_transformation 108 | returned the line itself for that line). 109 | """ 110 | def suggestor(lines): 111 | for line_number, line in enumerate(lines): 112 | if line_filter and not line_filter(line): 113 | continue 114 | candidate = line_transformation(line) 115 | if candidate is None: 116 | yield Patch(line_number) 117 | else: 118 | yield Patch(line_number, new_lines=[candidate]) 119 | return suggestor 120 | 121 | 122 | def regex_suggestor(regex, substitution=None, ignore_case=False, 123 | line_filter=None): 124 | if isinstance(regex, str): 125 | if ignore_case is False: 126 | regex = re.compile(regex) 127 | else: 128 | regex = re.compile(regex, re.IGNORECASE) 129 | 130 | if substitution is None: 131 | def line_transformation(line): 132 | return None if regex.search(line) else line 133 | else: 134 | def line_transformation(line): 135 | return regex.sub(substitution, line) 136 | return line_transformation_suggestor(line_transformation, line_filter) 137 | 138 | 139 | def multiline_regex_suggestor(regex, substitution=None, ignore_case=False): 140 | """ 141 | Return a suggestor function which, given a list of lines, generates patches 142 | to substitute matches of the given regex with (if provided) the given 143 | substitution. 144 | 145 | @param regex Either a regex object or a string describing a regex. 146 | @param substitution Either None (meaning that we should flag the matches 147 | without suggesting an alternative), or a string (using 148 | \1 notation to backreference match groups) or a 149 | function (that takes a match object as input). 150 | """ 151 | if isinstance(regex, str): 152 | if ignore_case is False: 153 | regex = re.compile(regex, re.DOTALL | re.MULTILINE) 154 | else: 155 | regex = re.compile(regex, re.DOTALL | re.MULTILINE | re.IGNORECASE) 156 | 157 | if isinstance(substitution, str): 158 | def substitution_func(match): 159 | return match.expand(substitution) 160 | else: 161 | substitution_func = substitution 162 | 163 | def suggestor(lines): 164 | pos = 0 165 | while True: 166 | match = regex.search(''.join(lines), pos) 167 | if not match: 168 | break 169 | start_row, start_col = _index_to_row_col(lines, match.start()) 170 | end_row, end_col = _index_to_row_col(lines, match.end() - 1) 171 | 172 | if substitution is None: 173 | new_lines = None 174 | else: 175 | # TODO: ugh, this is hacky. Clearly I need to rewrite 176 | # this to use 177 | # character-level patches, rather than line-level patches. 178 | new_lines = substitution_func(match) 179 | if new_lines is not None: 180 | new_lines = ''.join(( 181 | lines[start_row][:start_col], 182 | new_lines, 183 | lines[end_row][end_col + 1:] 184 | )) 185 | 186 | yield Patch( 187 | start_line_number=start_row, 188 | end_line_number=end_row + 1, 189 | new_lines=new_lines 190 | ) 191 | delta = 1 if new_lines is None else min(1, len(new_lines)) 192 | pos = match.start() + delta 193 | 194 | return suggestor 195 | 196 | 197 | def _index_to_row_col(lines, index): 198 | r""" 199 | >>> lines = ['hello\n', 'world\n'] 200 | >>> _index_to_row_col(lines, 0) 201 | (0, 0) 202 | >>> _index_to_row_col(lines, 7) 203 | (1, 1) 204 | """ 205 | if index < 0: 206 | raise IndexError('negative index') 207 | current_index = 0 208 | for line_number, line in enumerate(lines): 209 | line_length = len(line) 210 | if current_index + line_length > index: 211 | return line_number, index - current_index 212 | current_index += line_length 213 | raise IndexError('index %d out of range' % index) 214 | 215 | 216 | def print_patch(patch, lines_to_print, file_lines=None): 217 | if file_lines is None: 218 | file_lines = list(open(patch.path)) 219 | 220 | size_of_old = patch.end_line_number - patch.start_line_number 221 | size_of_new = len(patch.new_lines) if patch.new_lines else 0 222 | size_of_diff = size_of_old + size_of_new 223 | size_of_context = max(0, lines_to_print - size_of_diff) 224 | size_of_up_context = int(size_of_context / 2) 225 | size_of_down_context = int(ceil(size_of_context / 2)) 226 | start_context_line_number = patch.start_line_number - size_of_up_context 227 | end_context_line_number = patch.end_line_number + size_of_down_context 228 | 229 | def print_file_line(line_number): # noqa 230 | # Why line_number is passed here? 231 | print(' %s' % file_lines[i], end='') if ( 232 | 0 <= i < len(file_lines)) else '~\n', 233 | 234 | for i in range(start_context_line_number, patch.start_line_number): 235 | print_file_line(i) 236 | for i in range(patch.start_line_number, patch.end_line_number): 237 | if patch.new_lines is not None: 238 | terminal.terminal_print('- %s' % file_lines[i], color='RED') 239 | else: 240 | terminal.terminal_print('* %s' % file_lines[i], color='YELLOW') 241 | if patch.new_lines is not None: 242 | for line in patch.new_lines: 243 | terminal.terminal_print('+ %s' % line, color='GREEN') 244 | for i in range(patch.end_line_number, end_context_line_number): 245 | print_file_line(i) 246 | 247 | 248 | def _ask_about_patch(patch, editor, default_no): 249 | global yes_to_all 250 | 251 | default_action = 'n' if default_no else 'y' 252 | terminal.terminal_clear() 253 | terminal.terminal_print('%s\n' % patch.render_range(), color='WHITE') 254 | print() 255 | 256 | lines = list(open(patch.path)) 257 | size = list(terminal.terminal_get_size()) 258 | print_patch(patch, size[0] - 20, lines) 259 | 260 | print() 261 | 262 | if patch.new_lines is not None: 263 | if not yes_to_all: 264 | if default_no: 265 | print('Accept change (y = yes, n = no [default], e = edit, ' + 266 | 'A = yes to all, E = yes+edit, q = quit)? '), 267 | else: 268 | print('Accept change (y = yes [default], n = no, e = edit, ' + 269 | 'A = yes to all, E = yes+edit, q = quit)? '), 270 | p = _prompt('yneEAq', default=default_action) 271 | else: 272 | p = 'y' 273 | else: 274 | print('(e = edit [default], n = skip line, q = quit)? ', end=" ") 275 | p = _prompt('enq', default='e') 276 | 277 | if p in 'A': 278 | yes_to_all = True 279 | p = 'y' 280 | if p in 'yE': 281 | patch.apply_to(lines) 282 | _save(patch.path, lines) 283 | if p in 'eE': 284 | run_editor(patch.start_position, editor) 285 | if p in 'q': 286 | sys.exit(0) 287 | 288 | 289 | def _prompt(letters='yn', default=None): 290 | """ 291 | Wait for the user to type a character (and hit Enter). If the user enters 292 | one of the characters in `letters`, return that character. If the user 293 | hits Enter without entering a character, and `default` is specified, 294 | returns `default`. Otherwise, asks the user to enter a character again. 295 | """ 296 | while True: 297 | try: 298 | input_text = sys.stdin.readline().strip() 299 | except KeyboardInterrupt: 300 | sys.exit(0) 301 | if input_text and input_text in letters: 302 | return input_text 303 | if default is not None and input_text == '': 304 | return default 305 | print('Come again?') 306 | 307 | 308 | def _save(path, lines): 309 | file_w = open(path, 'w') 310 | for line in lines: 311 | file_w.write(line) 312 | file_w.close() 313 | 314 | 315 | def run_editor(position, editor=None): 316 | editor = editor or os.environ.get('EDITOR') or 'vim' 317 | os.system('%s +%d %s' % (editor, position.line_number + 1, position.path)) 318 | 319 | 320 | # 321 | # Bookmarking functions. codemod saves a file called .codemod.bookmark to 322 | # keep track of where you were the last time you exited in the middle of 323 | # an interactive sesh. 324 | # 325 | 326 | def _save_bookmark(position): 327 | file_w = open('.codemod.bookmark', 'w') 328 | file_w.write(str(position)) 329 | file_w.close() 330 | 331 | 332 | def _load_bookmark(): 333 | try: 334 | bookmark_file = open('.codemod.bookmark') 335 | except IOError: 336 | return None 337 | contents = bookmark_file.readline().strip() 338 | bookmark_file.close() 339 | return Position(contents) 340 | 341 | 342 | def _delete_bookmark(): 343 | try: 344 | os.remove('.codemod.bookmark') 345 | except OSError: 346 | pass # file didn't exist 347 | 348 | 349 | # 350 | # Code to make this run as an executable from the command line. 351 | # 352 | 353 | def _parse_command_line(): 354 | global yes_to_all 355 | 356 | parser = argparse.ArgumentParser( 357 | formatter_class=argparse.RawDescriptionHelpFormatter, 358 | description=textwrap.dedent(r""" 359 | codemod.py is a tool/library to assist you with large-scale 360 | codebase refactors 361 | that can be partially automated but still require 362 | human oversight and 363 | occassional intervention. 364 | 365 | Example: Let's say you're deprecating your use 366 | of the tag. From the 367 | command line, you might make progress by running: 368 | 369 | codemod.py -m -d /home/jrosenstein/www --extensions php,html \ 370 | '(.*?)' \ 371 | '\2' 372 | 373 | For each match of the regex, you'll be shown a colored diff, 374 | and asked if you 375 | want to accept the change (the replacement of 376 | the tag with a 377 | tag), reject it, or edit the line in question 378 | in your $EDITOR of choice. 379 | """), 380 | epilog=textwrap.dedent(r""" 381 | You can also use codemod for transformations that are much 382 | more sophisticated 383 | than regular expression substitution. Rather than using 384 | the command line, you 385 | write Python code that looks like: 386 | 387 | import codemod 388 | query = codemod.Query(...) 389 | run_interactive(query) 390 | 391 | See the documentation for the Query class for details. 392 | 393 | @author Justin Rosenstein 394 | """) 395 | ) 396 | 397 | parser.add_argument('-m', action='store_true', 398 | help='Have regex work over multiple lines ' 399 | '(e.g. have dot match newlines). ' 400 | 'By default, codemod applies the regex one ' 401 | 'line at a time.') 402 | parser.add_argument('-d', action='store', type=str, default='.', 403 | help='The path whose descendent files ' 404 | 'are to be explored. ' 405 | 'Defaults to current dir.') 406 | parser.add_argument('-i', action='store_true', 407 | help='Perform case-insensitive search.') 408 | 409 | parser.add_argument('--start', action='store', type=str, 410 | help='A path:line_number-formatted position somewhere' 411 | ' in the hierarchy from which to being exploring,' 412 | 'or a percentage (e.g. "--start 25%%") of ' 413 | 'the way through to start.' 414 | 'Useful if you\'re divvying up the ' 415 | 'substitution task across multiple people.') 416 | parser.add_argument('--end', action='store', type=str, 417 | help='A path:line_number-formatted position ' 418 | 'somewhere in the hierarchy just *before* ' 419 | 'which we should stop exploring, ' 420 | 'or a percentage of the way through, ' 421 | 'just before which to end.') 422 | 423 | parser.add_argument('--extensions', action='store', 424 | default='*', type=str, 425 | help='A comma-delimited list of file extensions ' 426 | 'to process. Also supports Unix pattern ' 427 | 'matching.') 428 | parser.add_argument('--include-extensionless', action='store_true', 429 | help='If set, this will check files without ' 430 | 'an extension, along with any matching file ' 431 | 'extensions passed in --extensions') 432 | parser.add_argument('--exclude-paths', action='store', type=str, 433 | help='A comma-delimited list of paths to exclude.') 434 | 435 | parser.add_argument('--accept-all', action='store_true', 436 | help='Automatically accept all ' 437 | 'changes (use with caution).') 438 | 439 | parser.add_argument('--default-no', action='store_true', 440 | help='If set, this will make the default ' 441 | 'option to not accept the change.') 442 | 443 | parser.add_argument('--editor', action='store', type=str, 444 | help='Specify an editor, e.g. "vim" or emacs". ' 445 | 'If omitted, defaults to $EDITOR environment ' 446 | 'variable.') 447 | parser.add_argument('--count', action='store_true', 448 | help='Don\'t run normally. Instead, just print ' 449 | 'out number of times places in the codebase ' 450 | 'where the \'query\' matches.') 451 | parser.add_argument('match', nargs='?', action='store', type=str, 452 | help='Regular expression to match.') 453 | parser.add_argument('subst', nargs='?', action='store', type=str, 454 | help='Substitution to replace with.') 455 | 456 | arguments = parser.parse_args() 457 | if not arguments.match: 458 | parser.exit(0, parser.format_usage()) 459 | 460 | query_options = {} 461 | yes_to_all = arguments.accept_all 462 | 463 | query_options['suggestor'] = ( 464 | multiline_regex_suggestor if arguments.m else regex_suggestor 465 | )(arguments.match, arguments.subst, arguments.i) 466 | 467 | query_options['start'] = arguments.start 468 | query_options['end'] = arguments.end 469 | query_options['root_directory'] = arguments.d 470 | query_options['inc_extensionless'] = arguments.include_extensionless 471 | 472 | if arguments.exclude_paths is not None: 473 | exclude_paths = arguments.exclude_paths.split(',') 474 | else: 475 | exclude_paths = None 476 | query_options['path_filter'] = helpers.path_filter( 477 | arguments.extensions.split(','), 478 | exclude_paths 479 | ) 480 | 481 | options = {} 482 | options['query'] = Query(**query_options) 483 | if arguments.editor is not None: 484 | options['editor'] = arguments.editor 485 | options['just_count'] = arguments.count 486 | options['default_no'] = arguments.default_no 487 | 488 | return options 489 | 490 | 491 | def main(): 492 | options = _parse_command_line() 493 | run_interactive(**options) 494 | 495 | 496 | if __name__ == '__main__': 497 | main() 498 | -------------------------------------------------------------------------------- /codemod/helpers.py: -------------------------------------------------------------------------------- 1 | import fnmatch 2 | import os 3 | 4 | 5 | def is_extensionless(path): 6 | """ 7 | Returns True if path has no extension. 8 | 9 | >>> is_extensionless("./www/test") 10 | True 11 | >>> is_extensionless("./www/.profile") 12 | True 13 | >>> is_extensionless("./www/.dir/README") 14 | True 15 | >>> is_extensionless("./scripts/menu.js") 16 | False 17 | >>> is_extensionless("./LICENSE") 18 | True 19 | """ 20 | _, ext = os.path.splitext(path) 21 | return ext == '' 22 | 23 | 24 | def matches_extension(path, extension): 25 | """ 26 | Returns True if path has the given extension, or if 27 | the last path component matches the extension. Supports 28 | Unix glob matching. 29 | 30 | >>> matches_extension("./www/profile.php", "php") 31 | True 32 | >>> matches_extension("./scripts/menu.js", "html") 33 | False 34 | >>> matches_extension("./LICENSE", "LICENSE") 35 | True 36 | """ 37 | _, ext = os.path.splitext(path) 38 | if ext == '': 39 | # If there is no extension, grab the file name and 40 | # compare it to the given extension. 41 | return os.path.basename(path) == extension 42 | else: 43 | # If the is an extension, drop the leading period and 44 | # compare it to the extension. 45 | return fnmatch.fnmatch(ext[1:], extension) 46 | 47 | 48 | def path_filter(extensions, exclude_paths=None): 49 | """ 50 | Returns a function that returns True if a filepath is acceptable. 51 | 52 | @param extensions An array of strings. Specifies what file 53 | extensions should be accepted by the 54 | filter. If None, we default to the Unix glob 55 | `*` and match every file extension. 56 | @param exclude_paths An array of strings which represents filepaths 57 | that should never be accepted by the filter. Unix 58 | shell-style wildcards are supported. 59 | 60 | @return function A filter function that will only return True 61 | when a filepath is acceptable under the above 62 | conditions. 63 | 64 | >>> list(map(path_filter(extensions=['js', 'php']), 65 | ... ['./profile.php', './q.jjs'])) 66 | [True, False] 67 | >>> list(map(path_filter(extensions=['*'], 68 | ... exclude_paths=['html']), 69 | ... ['./html/x.php', './lib/y.js'])) 70 | [False, True] 71 | >>> list(map(path_filter(extensions=['js', 'BUILD']), 72 | ... ['./a.js', './BUILD', './profile.php'])) 73 | [True, True, False] 74 | >>> list(map(path_filter(extensions=['js'], 75 | ... exclude_paths=['*/node_modules/*']), 76 | ... ['./a.js', './tools/node_modules/dep.js'])) 77 | [True, False] 78 | """ 79 | exclude_paths = exclude_paths or [] 80 | 81 | def the_filter(path): 82 | if not any(matches_extension(path, extension) 83 | for extension in extensions): 84 | return False 85 | if exclude_paths: 86 | for excluded in exclude_paths: 87 | if (path.startswith(excluded) or 88 | path.startswith('./' + excluded) or 89 | fnmatch.fnmatch(path, excluded)): 90 | return False 91 | return True 92 | return the_filter 93 | -------------------------------------------------------------------------------- /codemod/patch.py: -------------------------------------------------------------------------------- 1 | from codemod.position import Position 2 | 3 | 4 | class Patch(object): 5 | """ 6 | Represents a range of a file and (optionally) a list of lines with which to 7 | replace that range. 8 | 9 | >>> p = Patch(2, 4, ['X', 'Y', 'Z'], 'x.php') 10 | >>> print(p.render_range()) 11 | x.php:2-3 12 | >>> p.start_position 13 | Position('x.php', 2) 14 | >>> l = ['a', 'b', 'c', 'd', 'e', 'f'] 15 | >>> p.apply_to(l) 16 | >>> l 17 | ['a', 'b', 'X', 'Y', 'Z', 'e', 'f'] 18 | >>> print(p) 19 | Patch('x.php', 2, 4, ['X', 'Y', 'Z']) 20 | """ 21 | 22 | def __init__(self, start_line_number, end_line_number=None, new_lines=None, 23 | path=None): # noqa 24 | """ 25 | Constructs a Patch object. 26 | 27 | @param end_line_number The line number just *after* the end of 28 | the range. 29 | Defaults to 30 | start_line_number + 1, i.e. a one-line 31 | diff. 32 | @param new_lines The set of lines with which to 33 | replace the range 34 | specified, or a newline-delimited string. 35 | Omitting this means that 36 | this "patch" doesn't actually 37 | suggest a change. 38 | @param path Path is optional only so that 39 | suggestors that have 40 | been passed a list of lines 41 | don't have to set the 42 | path explicitly. 43 | (It'll get set by the suggestor's caller.) 44 | """ 45 | self.path = path 46 | self.start_line_number = start_line_number 47 | self.end_line_number = end_line_number 48 | self.new_lines = new_lines 49 | 50 | if self.end_line_number is None: 51 | self.end_line_number = self.start_line_number + 1 52 | if isinstance(self.new_lines, str): 53 | self.new_lines = self.new_lines.splitlines(True) 54 | 55 | def __repr__(self): 56 | return 'Patch(%s)' % ', '.join(map(repr, [ 57 | self.path, 58 | self.start_line_number, 59 | self.end_line_number, 60 | self.new_lines 61 | ])) 62 | 63 | def apply_to(self, lines): 64 | if self.new_lines is None: 65 | raise ValueError('Can\'t apply patch without suggested new lines.') 66 | lines[self.start_line_number:self.end_line_number] = self.new_lines 67 | 68 | def render_range(self): 69 | path = self.path or '' 70 | if self.start_line_number == self.end_line_number - 1: 71 | return '%s:%d' % (path, self.start_line_number) 72 | else: 73 | return '%s:%d-%d' % ( 74 | path, 75 | self.start_line_number, self.end_line_number - 1 76 | ) 77 | 78 | def get_start_position(self): 79 | return Position(self.path, self.start_line_number) 80 | start_position = property(get_start_position) 81 | -------------------------------------------------------------------------------- /codemod/position.py: -------------------------------------------------------------------------------- 1 | class Position(object): 2 | """ 3 | >>> p1, p2 = Position('./hi.php', 20), Position('./hi.php:20') 4 | >>> p1.path == p2.path and p1.line_number == p2.line_number 5 | True 6 | >>> p1 7 | Position('./hi.php', 20) 8 | >>> print(p1) 9 | ./hi.php:20 10 | >>> Position(p1) 11 | Position('./hi.php', 20) 12 | """ 13 | 14 | def __init__(self, *path_and_line_number): 15 | """ 16 | You can use the two parameter version, and pass a 17 | path and line number, or 18 | you can use the one parameter version, and 19 | pass a $path:$line_number string, 20 | or another instance of Position to copy. 21 | """ 22 | if len(path_and_line_number) == 2: 23 | self.path, self.line_number = path_and_line_number 24 | elif len(path_and_line_number) == 1: 25 | arg = path_and_line_number[0] 26 | if isinstance(arg, Position): 27 | self.path, self.line_number = arg.path, arg.line_number 28 | else: 29 | try: 30 | self.path, line_number_s = arg.split(':') 31 | self.line_number = int(line_number_s) 32 | except ValueError: 33 | raise ValueError( 34 | 'inappropriately formatted Position string: %s' 35 | % path_and_line_number[0] 36 | ) 37 | else: 38 | raise TypeError('Position takes 1 or 2 arguments') 39 | 40 | def __repr__(self): 41 | return 'Position(%s, %d)' % (repr(self.path), self.line_number) 42 | 43 | def __str__(self): 44 | return '%s:%d' % (self.path, self.line_number) 45 | -------------------------------------------------------------------------------- /codemod/query.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | 4 | from codemod.position import Position 5 | import codemod.helpers as helpers 6 | 7 | 8 | class Query(object): 9 | """ 10 | Represents a suggestor, along with a set of constraints on which files 11 | should be fed to that suggestor. 12 | 13 | >>> Query(lambda x: None, start='profile.php:20').start_position 14 | Position('profile.php', 20) 15 | """ 16 | def __init__(self, 17 | suggestor, 18 | start=None, 19 | end=None, 20 | root_directory='.', 21 | path_filter=helpers.path_filter( 22 | extensions=['php', 'phpt', 'js', 'css', 'rb', 'erb'] 23 | ), 24 | inc_extensionless=False): 25 | 26 | """ 27 | @param suggestor A function that takes a list of lines and 28 | generates instances of Patch to suggest. 29 | (Patches should not specify paths.) 30 | @param start One of: 31 | - an instance of Position 32 | (indicating the place in the file 33 | hierarchy at which to resume), 34 | - a path:line_number-formatted string 35 | representing a position, 36 | - a string formatted like "25%" 37 | (indicating we should start 25% of 38 | the way through the process), or 39 | - None (indicating that we should 40 | start at the beginning). 41 | @param end An indicator of the position 42 | just *before* which 43 | to stop exploring, using one 44 | of the same formats 45 | used for start (where None means 46 | 'traverse to the end of the hierarchy). 47 | @param root_directory The path whose ancestor files 48 | are to be explored. 49 | @param path_filter Given a path, returns True or False. 50 | If False, 51 | the entire file is ignored. 52 | @param inc_extensionless If True, will include all files without an 53 | extension when checking 54 | against the path_filter 55 | """ 56 | self.suggestor = suggestor 57 | self._start = start 58 | self._end = end 59 | self.root_directory = root_directory 60 | self.path_filter = path_filter 61 | self.inc_extensionless = inc_extensionless 62 | self._all_patches_cache = None 63 | 64 | def clone(self): 65 | import copy 66 | return copy.copy(self) 67 | 68 | def _get_position(self, attr_name): 69 | attr_value = getattr(self, attr_name) 70 | if attr_value is None: 71 | return None 72 | if isinstance(attr_value, str) and attr_value.endswith('%'): 73 | attr_value = self.compute_percentile(int(attr_value[:-1])) 74 | setattr(self, attr_name, attr_value) 75 | return Position(attr_value) 76 | 77 | def get_start_position(self): 78 | return self._get_position('_start') 79 | start_position = property(get_start_position) 80 | 81 | @start_position.setter 82 | def start_position(self, value): 83 | self._start = value 84 | 85 | def get_end_position(self): 86 | return self._get_position('_end') 87 | end_position = property(get_end_position) 88 | 89 | @end_position.setter 90 | def end_position(self, value): 91 | self._end = value 92 | 93 | def get_all_patches(self, dont_use_cache=False): 94 | """ 95 | Computes a list of all patches matching this query, though ignoreing 96 | self.start_position and self.end_position. 97 | 98 | @param dont_use_cache If False, and get_all_patches has been called 99 | before, compute the list computed last time. 100 | """ 101 | if not dont_use_cache and self._all_patches_cache is not None: 102 | return self._all_patches_cache 103 | 104 | print( 105 | 'Computing full change list (since you specified a percentage)...' 106 | ), 107 | sys.stdout.flush() # since print statement ends in comma 108 | 109 | endless_query = self.clone() 110 | endless_query.start_position = endless_query.end_position = None 111 | self._all_patches_cache = list(endless_query.generate_patches()) 112 | return self._all_patches_cache 113 | 114 | def compute_percentile(self, percentage): 115 | """ 116 | Returns a Position object that represents percentage%-far-of-the-way 117 | through the larger task, as specified by this query. 118 | 119 | @param percentage a number between 0 and 100. 120 | """ 121 | all_patches = self.get_all_patches() 122 | return all_patches[ 123 | int(len(all_patches) * percentage / 100) 124 | ].start_position 125 | 126 | def generate_patches(self): 127 | """ 128 | Generates a list of patches for each file underneath 129 | self.root_directory 130 | that satisfy the given conditions given 131 | query conditions, where patches for 132 | each file are suggested by self.suggestor. 133 | """ 134 | start_pos = self.start_position or Position(None, None) 135 | end_pos = self.end_position or Position(None, None) 136 | 137 | path_list = Query._walk_directory(self.root_directory) 138 | path_list = Query._sublist(path_list, start_pos.path, end_pos.path) 139 | path_list = ( 140 | path for path in path_list if 141 | Query._path_looks_like_code(path) and 142 | (self.path_filter(path)) or 143 | (self.inc_extensionless and helpers.is_extensionless(path)) 144 | ) 145 | for path in path_list: 146 | try: 147 | lines = list(open(path)) 148 | except (IOError, UnicodeDecodeError): 149 | # If we can't open the file--perhaps it's a symlink whose 150 | # destination no loner exists--then short-circuit. 151 | continue 152 | 153 | for patch in self.suggestor(lines): 154 | if path == start_pos.path: 155 | if patch.start_line_number < start_pos.line_number: 156 | continue # suggestion is pre-start_pos 157 | if path == end_pos.path: 158 | if patch.end_line_number >= end_pos.line_number: 159 | break # suggestion is post-end_pos 160 | 161 | old_lines = lines[ 162 | patch.start_line_number:patch.end_line_number] 163 | if patch.new_lines is None or patch.new_lines != old_lines: 164 | patch.path = path 165 | yield patch 166 | # re-open file, in case contents changed 167 | lines[:] = list(open(path)) 168 | 169 | @staticmethod 170 | def _walk_directory(root_directory): 171 | """ 172 | Generates the paths of all files that are ancestors 173 | of `root_directory`. 174 | """ 175 | 176 | paths = [os.path.join(root, name) 177 | for root, dirs, files in os.walk(root_directory) # noqa 178 | for name in files] 179 | paths.sort() 180 | return paths 181 | 182 | @staticmethod 183 | def _sublist(items, starting_value, ending_value=None): 184 | """ 185 | >>> list(Query._sublist((x*x for x in range(1, 100)), 16, 64)) 186 | [16, 25, 36, 49, 64] 187 | """ 188 | have_started = starting_value is None 189 | 190 | for x in items: 191 | have_started = have_started or x == starting_value 192 | if have_started: 193 | yield x 194 | 195 | if ending_value is not None and x == ending_value: 196 | break 197 | 198 | @staticmethod 199 | def _path_looks_like_code(path): 200 | """ 201 | >>> Query._path_looks_like_code('/home/jrosenstein/www/profile.php') 202 | True 203 | >>> Query._path_looks_like_code('./tags') 204 | False 205 | >>> Query._path_looks_like_code('/home/jrosenstein/www/profile.php~') 206 | False 207 | >>> Query._path_looks_like_code('/home/jrosenstein/www/.git/HEAD') 208 | False 209 | """ 210 | return ( 211 | '/.' not in path and 212 | path[-1] != '~' and 213 | not path.endswith('tags') and 214 | not path.endswith('TAGS') 215 | ) 216 | -------------------------------------------------------------------------------- /codemod/terminal_helper.py: -------------------------------------------------------------------------------- 1 | """ 2 | Functions for working with the terminal. 3 | """ 4 | from __future__ import print_function 5 | 6 | import os 7 | import sys 8 | 9 | import curses 10 | 11 | import fcntl 12 | import termios 13 | import struct 14 | 15 | 16 | def _unicode(s, encoding='utf-8'): 17 | if type(s) == bytes: 18 | return s.decode(encoding, 'ignore') 19 | else: 20 | return str(s) 21 | 22 | 23 | def terminal_get_size(default_size=(25, 80)): 24 | """ 25 | Return (number of rows, number of columns) for the terminal, 26 | if they can be determined, or `default_size` if they can't. 27 | """ 28 | 29 | def ioctl_gwinsz(fd): # TABULATION FUNCTIONS 30 | try: # Discover terminal width 31 | return struct.unpack( 32 | 'hh', fcntl.ioctl(fd, termios.TIOCGWINSZ, '1234') 33 | ) 34 | except Exception: 35 | return None 36 | 37 | # try open fds 38 | size = ioctl_gwinsz(0) or ioctl_gwinsz(1) or ioctl_gwinsz(2) 39 | if not size: 40 | # ...then ctty 41 | try: 42 | fd = os.open(os.ctermid(), os.O_RDONLY) 43 | size = ioctl_gwinsz(fd) 44 | os.close(fd) 45 | except Exception: 46 | pass 47 | if not size: 48 | # env vars or finally defaults 49 | try: 50 | size = (os.environ['LINES'], os.environ['COLUMNS']) 51 | except Exception: 52 | return default_size 53 | 54 | return map(int, size) 55 | 56 | 57 | def terminal_clear(): 58 | """ 59 | Like calling the `clear` UNIX command. If that fails, just prints a bunch 60 | of newlines :-P 61 | """ 62 | if not _terminal_use_capability('clear'): 63 | print('\n' * 8) 64 | 65 | 66 | def terminal_move_to_beginning_of_line(): 67 | """ 68 | Jumps the cursor back to the beginning of the current line of text. 69 | """ 70 | if not _terminal_use_capability('cr'): 71 | print() 72 | 73 | 74 | def _terminal_use_capability(capability_name): 75 | """ 76 | If the terminal supports the given capability, output it. Return whether 77 | it was output. 78 | """ 79 | curses.setupterm() 80 | capability = curses.tigetstr(capability_name) 81 | if capability: 82 | sys.stdout.write(_unicode(capability)) 83 | return bool(capability) 84 | 85 | 86 | def terminal_print(text, color): 87 | """Print text in the specified color, without a terminating newline.""" 88 | _terminal_set_color(color) 89 | print(text, end='') 90 | _terminal_restore_color() 91 | 92 | 93 | def _terminal_set_color(color): 94 | def color_code(set_capability, possible_colors): 95 | try: 96 | color_index = possible_colors.split(' ').index(color) 97 | except ValueError: 98 | return None 99 | set_code = curses.tigetstr(set_capability) 100 | if not set_code: 101 | return None 102 | return curses.tparm(set_code, color_index) 103 | code = ( 104 | color_code( 105 | 'setaf', 'BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE' 106 | ) or color_code( 107 | 'setf', 'BLACK BLUE GREEN CYAN RED MAGENTA YELLOW WHITE' 108 | ) 109 | ) 110 | if code: 111 | code = _unicode(code) 112 | sys.stdout.write(code) 113 | 114 | 115 | def _terminal_restore_color(): 116 | restore_code = curses.tigetstr('sgr0') 117 | if restore_code: 118 | sys.stdout.write(_unicode(restore_code)) 119 | -------------------------------------------------------------------------------- /dev-requirements.txt: -------------------------------------------------------------------------------- 1 | . 2 | flake8 3 | pytest 4 | -------------------------------------------------------------------------------- /pytest.ini: -------------------------------------------------------------------------------- 1 | # content of pytest.ini 2 | [pytest] 3 | addopts = --doctest-modules codemod 4 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | try: 2 | from setuptools import setup, find_packages 3 | except ImportError: 4 | from distutils.core import setup, find_packages 5 | 6 | description = ( 7 | 'Codemod is a tool/library to assist you with large-scale codebase ' 8 | 'refactors that can be partially automated but still require human ' 9 | 'oversight and occasional intervention. Codemod was developed at ' 10 | 'Facebook and released as open source.' 11 | ) 12 | 13 | setup( 14 | name='codemod', 15 | version="1.0.0", 16 | url='http://github.com/facebook/codemod', 17 | license='Apache License 2.0', 18 | author="Facebook", 19 | author_email="facebook@facebook.com", 20 | description=description, 21 | long_description=description, 22 | packages=find_packages(), 23 | include_package_data=True, 24 | zip_safe=False, 25 | platforms='any', 26 | entry_points=''' 27 | [console_scripts] 28 | codemod=codemod.base:main 29 | ''', 30 | tests_require=['flake8', 'pytest'], 31 | test_suite='py.test' 32 | ) 33 | --------------------------------------------------------------------------------