├── .bumpversion.cfg ├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── git_externals ├── __init__.py ├── cli.py ├── git_externals.py ├── gitext.completion.bash └── utils.py ├── setup.cfg ├── setup.py ├── tests ├── basic.t ├── git-worktree.t ├── svn-target-freeze-svn.t ├── svn-target-freeze.t └── svn-target.t └── tox.ini /.bumpversion.cfg: -------------------------------------------------------------------------------- 1 | [bumpversion] 2 | current_version = 0.5.0 3 | commit = True 4 | 5 | [bumpversion:file:git_externals/__init__.py] 6 | 7 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | *.egg-info/ 24 | .installed.cfg 25 | *.egg 26 | 27 | # PyInstaller 28 | # Usually these files are written by a python script from a template 29 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 30 | *.manifest 31 | *.spec 32 | 33 | # Installer logs 34 | pip-log.txt 35 | pip-delete-this-directory.txt 36 | 37 | # Unit test / coverage reports 38 | htmlcov/ 39 | .tox/ 40 | .coverage 41 | .coverage.* 42 | .cache 43 | nosetests.xml 44 | coverage.xml 45 | *,cover 46 | .hypothesis/ 47 | 48 | # Translations 49 | *.mo 50 | *.pot 51 | 52 | # Django stuff: 53 | *.log 54 | 55 | # Sphinx documentation 56 | docs/_build/ 57 | 58 | # PyBuilder 59 | target/ 60 | 61 | #Ipython Notebook 62 | .ipynb_checkpoints 63 | 64 | # Tags 65 | tags 66 | 67 | # Local gittify folders 68 | gitsvn/ 69 | finalize/ 70 | rebase/ 71 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | dist: trusty 2 | language: python 3 | python: 4 | - "2.7" 5 | 6 | before_install: 7 | - sudo apt-get -qq update 8 | - sudo apt-get install -y git-svn 9 | 10 | install: pip install tox 11 | script: 12 | tox 13 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Develer srl 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Build Status](https://travis-ci.org/develersrl/git-externals.svg?branch=master)](https://travis-ci.org/develersrl/git-externals) 2 | 3 | Frozen mode 4 | ----------- 5 | 6 | :thinking: This project _works_ but the development is frozen, do not expect a lot submitting issues or PRs. :thinking: 7 | 8 | Git Externals 9 | ------------- 10 | 11 | `git-externals` is a command line tool that helps you throw **SVN** away and 12 | migrate to **Git** for projects that make heavy use of *svn:externals*. In some 13 | cases it's just impossible to use *Git submodules* or *subtrees* to emulate *SVN 14 | externals*, because they aren't as flexible as SVN externals. For example **SVN** 15 | lets you handle a *single file dependency* through *SVN external*, whereas **Git** 16 | doesn't. 17 | 18 | On Windows this requires **ADMIN PRIVILEGES**, because under the hood it uses 19 | symlinks. For the same reason this script is not meant to be used with the old 20 | Windows XP. 21 | 22 | ## How to Install 23 | 24 | ```sh 25 | $ pip install https://github.com/develersrl/git-externals/archive/master.zip 26 | ``` 27 | 28 | ## Usage: 29 | 30 | ### Content of `git_externals.json` 31 | 32 | Once your main project repository is handled by Git, `git-externals` expects to 33 | find a file called `git_externals.json` at the project root. Here is how to fill 34 | it: 35 | 36 | Let's take an hypothetical project **A**, under Subversion, having 2 37 | dependencies, **B** and **C**, declared as `svn:externals` as 38 | follows. 39 | 40 | ```sh 41 | $ svn propget svn:externals . 42 | ^/svn/libraries/B lib/B 43 | ^/svn/libraries/C src/C 44 | ``` 45 | 46 | ``` 47 | A 48 | ├── lib 49 | │   └── B 50 | └── src 51 | └── C 52 | ``` 53 | 54 | Once **A**, **B** and **C** have all been migrated over different Git 55 | repositories, fill `git_externals.json` by running the following commands. 56 | They describe, for each dependency, its remote location, and the destination 57 | directory, relative to the project root. Check out all the possibilities by 58 | running `git externals add --help`. 59 | 60 | ```sh 61 | $ git externals add --branch=master git@github.com:username/libB.git . lib/B 62 | $ git externals add --branch=master git@github.com:username/libC.git . src/C 63 | ``` 64 | 65 | This is now the content of `git_externals.json`: 66 | 67 | ```json 68 | { 69 | "git@github.com:username/libB.git": { 70 | "branch": "master", 71 | "ref": null, 72 | "targets": { 73 | "./": [ 74 | "lib/B" 75 | ] 76 | } 77 | }, 78 | "git@github.com:username/libC.git": { 79 | "branch": "master", 80 | "ref": null, 81 | "targets": { 82 | "./": [ 83 | "src/C" 84 | ] 85 | } 86 | } 87 | } 88 | ``` 89 | 90 | 91 | ### Git externals update 92 | 93 | If you want to: 94 | 95 | - download the externals of a freshly cloned Git repository and creates their 96 | symlinks, in order to have the wanted directory layout. 97 | - checkout the latest version of all externals (as defined in 98 | `git_externals.json` file) 99 | 100 | Run: 101 | 102 | ```sh 103 | $ git externals update 104 | ``` 105 | 106 | ### Git externals status 107 | 108 | ```sh 109 | $ git externals status [--porcelain|--verbose] 110 | $ git externals status [--porcelain|--verbose] [external1 [external2] ...] 111 | ``` 112 | 113 | Shows the working tree status of one, multiple, or all externals: 114 | 115 | - add `--verbose` if you are also interested to see the externals that haven't 116 | been modified 117 | - add `--porcelain` if you want the output easily parsable (for non-humans). 118 | 119 | ```sh 120 | $ git externals status 121 | $ git externals status deploy 122 | $ git externals status deploy qtwidgets 123 | ``` 124 | 125 | ### Git externals foreach 126 | 127 | ```sh 128 | $ git externals foreach [--] cmd [arg1 [arg2] ...] 129 | ``` 130 | 131 | Evaluates an arbitrary shell command in each checked out external. 132 | ```sh 133 | $ git externals foreach git fetch 134 | ``` 135 | 136 | **Note**: If some arguments of the shell command starts with `--`, like in 137 | `git rev-parse --all`, you must pass `--` after `foreach` in order to stop 138 | git externals argument processing, example: 139 | 140 | ```sh 141 | $ git externals foreach -- git rev-parse --all 142 | ``` 143 | 144 | ### Example usage 145 | 146 | ```sh 147 | $ git externals add --branch=master https://github.com/username/projectA.git shared/ foo 148 | $ git externals add --branch=master https://github.com/username/projectB.git shared/ bar 149 | $ git externals add --branch=master https://github.com/username/projectC.git README.md baz/README.md 150 | $ git externals add --tag=v4.4 https://github.com/torvalds/linux.git Makefile Makefile 151 | $ git add git_externals.json 152 | $ git commit -m "Let git-externals handle our externals ;-)" 153 | $ git externals update 154 | $ git externals diff 155 | $ git externals info 156 | $ git externals list 157 | $ git externals foreach -- git diff HEAD~1 158 | ``` 159 | 160 | **Note**: Append `/` to the source path if it represents a directory. 161 | 162 | ### Bash command-line completion 163 | 164 | See installation instructions in [gitext.completion.bash](./git_externals/gitext.completion.bash). 165 | -------------------------------------------------------------------------------- /git_externals/__init__.py: -------------------------------------------------------------------------------- 1 | __author__ = "Daniele D'Orazio" 2 | __version__ = '0.5.0' 3 | __email__ = 'daniele@develer.com' 4 | -------------------------------------------------------------------------------- /git_externals/cli.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from __future__ import print_function, unicode_literals 4 | 5 | import os 6 | import sys 7 | import re 8 | 9 | import click 10 | 11 | if __package__ is None: 12 | from __init__ import __version__ 13 | from utils import command, CommandError, chdir, git, ProgError 14 | else: 15 | from . import __version__ 16 | from .utils import (command, CommandError, chdir, git, ProgError, decode_utf8, 17 | current_branch) 18 | 19 | click.disable_unicode_literals_warning = True 20 | 21 | 22 | def echo(*args): 23 | click.echo(u' '.join(args)) 24 | 25 | 26 | def info(*args): 27 | click.secho(u' '.join(args), fg='blue') 28 | 29 | 30 | def error(*args, **kwargs): 31 | click.secho(u' '.join(args), fg='red') 32 | exitcode = kwargs.get('exitcode', 1) 33 | if exitcode is not None: 34 | sys.exit(exitcode) 35 | 36 | 37 | @click.group(context_settings={ 38 | 'allow_extra_args': True, 39 | 'ignore_unknown_options': True, 40 | 'help_option_names':['-h','--help'], 41 | }) 42 | @click.version_option(__version__) 43 | @click.option('--with-color/--no-color', 44 | default=True, 45 | help='Enable/disable colored output') 46 | @click.pass_context 47 | def cli(ctx, with_color): 48 | """Utility to manage git externals, meant to be used as a drop-in 49 | replacement to svn externals 50 | 51 | This script works by cloning externals found in the `git_externals.json` 52 | file into `.git_externals/` and symlinks them to recreate the wanted 53 | directory layout. 54 | """ 55 | from git_externals import is_git_repo, externals_json_path, externals_root_path 56 | 57 | if not is_git_repo(): 58 | error("{} is not a git repository!".format(os.getcwd()), exitcode=2) 59 | 60 | if ctx.invoked_subcommand != 'add' and not os.path.exists(externals_json_path()): 61 | error("Unable to find", externals_json_path(), exitcode=1) 62 | 63 | if not os.path.exists(externals_root_path()): 64 | if ctx.invoked_subcommand not in set(['update', 'add']): 65 | error('You must first run git-externals update/add', exitcode=2) 66 | else: 67 | if with_color: 68 | enable_colored_output() 69 | 70 | if ctx.invoked_subcommand is None: 71 | gitext_st(()) 72 | 73 | 74 | @cli.command('foreach') 75 | @click.option('--recursive/--no-recursive', help='If --recursive is specified, this command will recurse into nested externals', default=True) 76 | @click.argument('subcommand', nargs=-1, required=True) 77 | def gitext_foreach(recursive, subcommand): 78 | """Evaluates an arbitrary shell command in each checked out external 79 | """ 80 | from git_externals import externals_sanity_check, get_repo_name, foreach_externals_dir, root_path 81 | 82 | externals_sanity_check() 83 | 84 | def run_command(rel_url, ext_path, targets): 85 | try: 86 | info("External {}".format(get_repo_name(rel_url))) 87 | output = decode_utf8(command(*subcommand)) 88 | info("Ok: CWD: {}, cmd: {}".format(os.getcwd(), subcommand)) 89 | echo(output) 90 | except CommandError as err: 91 | info("Command error {} CWD: {}, cmd: {}".format(err, os.getcwd(), subcommand)) 92 | error(str(err), exitcode=err.errcode) 93 | 94 | foreach_externals_dir(root_path(), run_command, recursive=recursive) 95 | 96 | 97 | @cli.command('update') 98 | @click.option('--recursive/--no-recursive', help='Do not call git-externals update recursively', default=True) 99 | @click.option('--gitsvn/--no-gitsvn', help='use git-svn (or simply svn) to checkout SVN repositories (only needed at first checkout)', default=True) 100 | @click.option('--reset', help='Reset repo, overwrite local modifications', is_flag=True) 101 | def gitext_update(recursive, gitsvn, reset): 102 | """Update the working copy cloning externals if needed and create the desired layout using symlinks 103 | """ 104 | from git_externals import externals_sanity_check, root_path, is_workingtree_clean, foreach_externals, gitext_up 105 | 106 | externals_sanity_check() 107 | root = root_path() 108 | 109 | if reset: 110 | git('reset', '--hard') 111 | 112 | # Aggregate in a list the `clean flags` of all working trees (root + externals) 113 | clean = [is_workingtree_clean(root, fail_on_empty=False)] 114 | foreach_externals(root, 115 | lambda u, p, r: clean.append(is_workingtree_clean(p, fail_on_empty=False)), 116 | recursive=recursive) 117 | 118 | if reset or all(clean): 119 | # Proceed with update if everything is clean 120 | try: 121 | gitext_up(recursive, reset=reset, use_gitsvn=gitsvn) 122 | except ProgError as e: 123 | error(str(e), exitcode=e.errcode) 124 | else: 125 | echo("Cannot perform git externals update because one or more repositories contain some local modifications") 126 | echo("Run:\tgit externals status\tto have more information") 127 | 128 | 129 | @cli.command('status') 130 | @click.option( 131 | '--porcelain', 132 | is_flag=True, 133 | help='Print output using the porcelain format, useful mostly for scripts') 134 | @click.option( 135 | '--verbose/--no-verbose', 136 | is_flag=True, 137 | help='Show the full output of git status, instead of showing only the modifications regarding tracked file') 138 | @click.argument('externals', nargs=-1) 139 | def gitext_st(porcelain, verbose, externals): 140 | """Call git status on the given externals""" 141 | from git_externals import foreach_externals_dir, root_path, \ 142 | is_workingtree_clean, get_repo_name 143 | 144 | def get_status(rel_url, ext_path, targets): 145 | try: 146 | if porcelain: 147 | echo(rel_url) 148 | click.echo(git('status', '--porcelain')) 149 | elif verbose or not is_workingtree_clean(ext_path): 150 | info("External {}".format(get_repo_name(rel_url))) 151 | echo(git('status', '--untracked-files=no' if not verbose else '')) 152 | except CommandError as err: 153 | error(str(err), exitcode=err.errcode) 154 | 155 | foreach_externals_dir(root_path(), get_status, recursive=True, only=externals) 156 | 157 | 158 | @cli.command('diff') 159 | @click.argument('external', nargs=-1) 160 | def gitext_diff(external): 161 | """Call git diff on the given externals""" 162 | from git_externals import iter_externals 163 | for _ in iter_externals(external): 164 | click.echo(git('diff')) 165 | 166 | 167 | @cli.command('add') 168 | @click.argument('external', 169 | metavar='URL') 170 | @click.argument('src', metavar='PATH') 171 | @click.argument('dst', metavar='PATH') 172 | @click.option('--branch', '-b', default=None, help='Checkout the given branch') 173 | @click.option('--tag', '-t', default=None, help='Checkout the given tag') 174 | @click.option('--ref', '-r', default=None, help='Checkout the given commit sha') 175 | @click.option('--vcs', '-c', default='auto', help='Version Control System (default: autodetect)', 176 | type=click.Choice(['svn', 'git', 'auto'])) 177 | def gitext_add(external, src, dst, branch, tag, ref, vcs): 178 | """Add a git external to the current repo. 179 | 180 | Be sure to add '/' to `src` if it's a directory! 181 | It's possible to add multiple `dst` to the same `src`, however you cannot mix different branches, tags or refs 182 | for the same external. 183 | 184 | It's safe to use this command to add `src` to an already present external, as well as adding 185 | `dst` to an already present `src`. 186 | 187 | It requires one of --branch or --tag. 188 | """ 189 | from git_externals import load_gitexts, dump_gitexts, normalize_gitexts, print_gitext_info 190 | 191 | git_externals = load_gitexts() 192 | 193 | if branch is None and tag is None: 194 | error('Please specifiy at least a branch or a tag', exitcode=3) 195 | 196 | if external not in git_externals: 197 | git_externals[external] = {'targets': {src: [dst]}} 198 | if branch is not None: 199 | git_externals[external]['branch'] = branch 200 | git_externals[external]['ref'] = ref 201 | else: 202 | git_externals[external]['tag'] = tag 203 | if vcs == 'auto': 204 | git_externals.update(normalize_gitexts({external: git_externals[external]})) 205 | else: 206 | git_externals[external]['vcs'] = vcs 207 | 208 | else: 209 | if branch is not None: 210 | if 'branch' not in git_externals[external]: 211 | error( 212 | '{} is bound to tag {}, cannot set it to branch {}'.format( 213 | external, git_externals[external]['tag'], branch), 214 | exitcode=4) 215 | 216 | if ref != git_externals[external]['ref']: 217 | error('{} is bound to ref {}, cannot set it to ref {}'.format( 218 | external, git_externals[external]['ref'], ref), 219 | exitcode=4) 220 | 221 | elif 'tag' not in git_externals[external]: 222 | error('{} is bound to branch {}, cannot set it to tag {}'.format( 223 | external, git_externals[external]['branch'], tag), 224 | exitcode=4) 225 | 226 | if dst not in git_externals[external]['targets'].setdefault(src, []): 227 | git_externals[external]['targets'][src].append(dst) 228 | 229 | print_gitext_info(external, git_externals[external], root_dir='.') 230 | dump_gitexts(git_externals) 231 | 232 | 233 | @cli.command('freeze') 234 | @click.option('--messages', '-m', is_flag=True, help="List commit messages") 235 | @click.argument('externals', nargs=-1, metavar='NAME') 236 | def gitext_freeze(externals, messages): 237 | """Freeze the externals revision""" 238 | from git_externals import load_gitexts, dump_gitexts, foreach_externals_dir, root_path, resolve_revision 239 | git_externals = load_gitexts() 240 | repo_root = root_path() 241 | re_from_git_svn_id = re.compile("git-svn-id:.*@(\d+)") 242 | re_from_svnversion = re.compile("(\d+):(\d+)") 243 | 244 | def get_version(rel_url, ext_path, refs): 245 | if 'tag' in refs: 246 | return 247 | 248 | bare_svn = False 249 | if git_externals[rel_url]["vcs"] == "svn": 250 | revision = command('svnversion', '-c').strip() 251 | match = re_from_svnversion.search(revision) 252 | if match: 253 | revision = "svn:r" + match.group(2) # 565:56555 -> svn:r56555 254 | bare_svn = True 255 | else: 256 | message = git("log", "--format=%b", "--grep", "git-svn-id:", "-1") 257 | match = re_from_git_svn_id.search(message) 258 | if match: 259 | revision = "svn:r" + match.group(1) 260 | else: 261 | here = os.path.relpath(os.getcwd(), repo_root) 262 | error("Unsupported external format, svn or git-svn repo expected:\n\t{}".format(here)) 263 | else: 264 | branch_name = current_branch() 265 | remote_name = git("config", "branch.%s.remote" % branch_name) 266 | revision = git("log", "%s/%s" % (remote_name, branch_name), "-1", "--format=%H") 267 | 268 | info("Freeze {0} at {1}".format(rel_url, revision)) 269 | if messages and not bare_svn: 270 | old = resolve_revision(git_externals[rel_url]["ref"]) 271 | new = resolve_revision(revision) 272 | git("log", "--format=- %h %s", "{}..{}".format(old, new), capture=False) 273 | git_externals[rel_url]["ref"] = revision 274 | 275 | foreach_externals_dir(repo_root, get_version, only=externals) 276 | 277 | dump_gitexts(git_externals) 278 | 279 | 280 | @cli.command('remove') 281 | @click.argument('external', nargs=-1, metavar='URL') 282 | def gitext_remove(external): 283 | """Remove the externals at the given repository URLs """ 284 | from git_externals import load_gitexts, dump_gitexts 285 | 286 | git_externals = load_gitexts() 287 | 288 | for ext in external: 289 | if ext in git_externals: 290 | del git_externals[ext] 291 | 292 | dump_gitexts(git_externals) 293 | 294 | 295 | @cli.command('info') 296 | @click.argument('externals', nargs=-1) 297 | @click.option('--recursive/--no-recursive', default=True, 298 | help='Top level externals in which recurse into') 299 | def gitext_info(externals, recursive): 300 | """Print some info about the externals.""" 301 | from git_externals import gitext_recursive_info 302 | gitext_recursive_info('.', recursive=recursive, externals=externals) 303 | 304 | 305 | def enable_colored_output(): 306 | from git_externals import externals_root_path, get_entries 307 | 308 | for entry in get_entries(): 309 | with chdir(os.path.join(externals_root_path(), entry)): 310 | git('config', 'color.ui', 'always') 311 | 312 | 313 | def main(): 314 | cli() 315 | 316 | 317 | if __name__ == '__main__': 318 | main() 319 | -------------------------------------------------------------------------------- /git_externals/git_externals.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from __future__ import print_function, unicode_literals 4 | 5 | if __package__ is None: 6 | import sys 7 | from os import path 8 | sys.path.append(path.dirname(path.dirname(path.abspath(__file__)))) 9 | 10 | import json 11 | import os 12 | import os.path 13 | import posixpath 14 | from collections import defaultdict, namedtuple 15 | 16 | try: 17 | from urllib.parse import urlparse, urlsplit, urlunsplit 18 | except ImportError: 19 | from urlparse import urlparse, urlsplit, urlunsplit 20 | 21 | import click 22 | 23 | from .utils import (chdir, mkdir_p, link, rm_link, git, GitError, svn, gitsvn, gitsvnrebase, current_branch) 24 | from .cli import echo, info, error 25 | 26 | 27 | OLD_EXTERNALS_ROOT = os.path.join('.git', 'externals') 28 | EXTERNALS_ROOT = '.git_externals' 29 | EXTERNALS_JSON = 'git_externals.json' 30 | 31 | ExtItem = namedtuple('ExtItem', ['branch', 'ref', 'path', 'name']) 32 | 33 | 34 | def get_repo_name(repo): 35 | externals = load_gitexts() 36 | if repo in externals and 'name' in externals[repo]: 37 | # echo ("for {} in pwd:{} returning {}".format(repo, os.getcwd(), 38 | # externals[repo]['name'])) 39 | return externals[repo]['name'] 40 | 41 | if repo[-1] == '/': 42 | repo = repo[:-1] 43 | name = repo.split('/')[-1] 44 | if name.endswith('.git'): 45 | name = name[:-len('.git')] 46 | if not name: 47 | error("Invalid repository name: \"{}\"".format(repo), exitcode=1) 48 | return name 49 | 50 | 51 | def externals_json_path(pwd=None): 52 | return os.path.join(pwd or root_path(), EXTERNALS_JSON) 53 | 54 | 55 | def externals_root_path(pwd=None): 56 | _old_root_path = os.path.join(pwd or root_path(), OLD_EXTERNALS_ROOT) 57 | _root_path = os.path.join(pwd or root_path(), EXTERNALS_ROOT) 58 | if os.path.exists(_old_root_path) and not os.path.exists(_root_path): 59 | info("Moving old externals path to new location") 60 | os.rename(_old_root_path, _root_path) 61 | link_entries(load_gitexts(pwd)) 62 | elif os.path.exists(_old_root_path) and os.path.exists(_root_path): 63 | error("Both new and old externals folder found, {} will be used".format(_root_path)) 64 | return _root_path 65 | 66 | 67 | def root_path(): 68 | return git('rev-parse', '--show-toplevel').strip() 69 | 70 | 71 | def is_git_repo(quiet=True): 72 | """Says if pwd is a Git working tree or not. 73 | If not quiet: says it also on standard output 74 | """ 75 | try: 76 | return git('rev-parse', '--is-inside-work-tree').strip() == 'true' 77 | except GitError as err: 78 | if not quiet: 79 | print (str(err)) 80 | 81 | 82 | def normalize_gitext_url(url): 83 | # an absolute url is already normalized 84 | if urlparse(url).netloc != '' or url.startswith('git@'): 85 | return url 86 | 87 | # relative urls use the root url of the current origin 88 | remote_name = git('config', 'branch.%s.remote' % current_branch()).strip() 89 | remote_url = git('config', 'remote.%s.url' % remote_name).strip() 90 | 91 | if remote_url.startswith('git@'): 92 | prefix = remote_url[:remote_url.index(':')+1] 93 | remote_url = prefix + url.strip('/') 94 | else: 95 | remote_url = urlunsplit(urlsplit(remote_url)._replace(path=url)) 96 | 97 | return remote_url 98 | 99 | 100 | def get_entries(): 101 | return [get_repo_name(e) 102 | for e in load_gitexts().keys() 103 | if os.path.exists(os.path.join(externals_root_path(), get_repo_name(e)))] 104 | 105 | 106 | def load_gitexts(pwd=None): 107 | """Load the *externals definition file* present in given 108 | directory, or cwd 109 | """ 110 | d = pwd if pwd is not None else '.' 111 | fn = os.path.join(d, EXTERNALS_JSON) 112 | if os.path.exists(fn): 113 | with open(fn) as f: 114 | return normalize_gitexts(json.load(f)) 115 | return {} 116 | 117 | 118 | def normalize_gitexts(gitext): 119 | for url, _ in gitext.items(): 120 | # svn external url must be absolute and svn+ssh to be autodetected 121 | gitext[url].setdefault('vcs', 'svn' if 'svn' in urlparse(url).scheme else 'git') 122 | return gitext 123 | 124 | 125 | def dump_gitexts(externals): 126 | """ 127 | Dump externals dictionary as json in current working directory 128 | git_externals.json. Remove 'vcs' key that is only used at runtime. 129 | """ 130 | with open(externals_json_path(), 'w') as f: 131 | json.dump(externals, f, sort_keys=True, indent=4, separators=(',', ': ')) 132 | f.write("\n") 133 | 134 | 135 | def foreach_externals(pwd, callback, recursive=True, only=()): 136 | """ 137 | Iterates over externals, starting from directory pwd, recursively or not 138 | callback is called for each externals with the following arguments: 139 | - relative url of current external repository 140 | - path to external working tree directory 141 | - refs: external as a dictionary (straight from json file) 142 | Iterates over all externals by default, or filter over the externals listed 143 | in only (filters on externals path, url or part of it) 144 | """ 145 | externals = load_gitexts(pwd) 146 | def filter_ext(): 147 | def take_external(url, path): 148 | return any((expr in url or expr in path) for expr in only) 149 | def take_all(*args): 150 | return True 151 | return take_external if len(only) else take_all 152 | 153 | for rel_url in externals: 154 | ext_path = os.path.join(externals_root_path(pwd), get_repo_name(rel_url)) 155 | if filter_ext()(rel_url, ext_path): 156 | callback(rel_url, ext_path, externals[rel_url]) 157 | if recursive: 158 | foreach_externals(ext_path, callback, recursive=recursive, only=only) 159 | 160 | 161 | def foreach_externals_dir(pwd, callback, recursive=True, only=[]): 162 | """ 163 | Same as foreach_externals, but place the callback in the directory 164 | context of the externals before calling it 165 | """ 166 | def run_from_dir(rel_url, ext_path, refs): 167 | if os.path.exists(ext_path): 168 | with chdir(ext_path): 169 | callback(rel_url, ext_path, refs) 170 | foreach_externals(root_path(), run_from_dir, recursive=recursive, only=only) 171 | 172 | 173 | def sparse_checkout(repo_name, repo, dirs): 174 | git('init', repo_name) 175 | 176 | with chdir(repo_name): 177 | git('remote', 'add', '-f', 'origin', repo) 178 | git('config', 'core.sparsecheckout', 'true') 179 | 180 | with open(os.path.join('.git', 'info', 'sparse-checkout'), 'wt') as fp: 181 | fp.write('{}\n'.format(externals_json_path())) 182 | for d in dirs: 183 | # assume directories are terminated with / 184 | fp.write(posixpath.normpath(d)) 185 | if d[-1] == '/': 186 | fp.write('/') 187 | fp.write('\n') 188 | 189 | return repo_name 190 | 191 | 192 | def is_workingtree_clean(path, fail_on_empty=True): 193 | """ 194 | Returns true if and only if there are no modifications to tracked files. By 195 | modifications it is intended additions, deletions, file removal or 196 | conflicts. If True is returned, that means that performing a 197 | `git reset --hard` would result in no loss of local modifications because: 198 | - tracked files are unchanged 199 | - untracked files are not modified anyway 200 | """ 201 | if not os.path.exists(path): 202 | return not fail_on_empty 203 | if fail_on_empty and not os.path.exists(path): 204 | return False 205 | with chdir(path): 206 | try: 207 | return len([line.strip for line in git('status', '--untracked-files=no', '--porcelain').splitlines(True)]) == 0 208 | except GitError as err: 209 | echo('Couldn\'t retrieve Git status of', path) 210 | error(str(err), exitcode=err.errcode) 211 | 212 | 213 | def link_entries(git_externals): 214 | entries = [(get_repo_name(repo), src, os.path.join(os.getcwd(), dst.replace('/', os.path.sep))) 215 | for (repo, repo_data) in git_externals.items() 216 | for (src, dsts) in repo_data['targets'].items() 217 | for dst in dsts] 218 | 219 | entries.sort(key=lambda x: x[2]) 220 | 221 | # remove links starting from the deepest dst 222 | for _, __, dst in entries[::-1]: 223 | if os.path.lexists(dst): 224 | rm_link(dst) 225 | 226 | # link starting from the highest dst 227 | for repo_name, src, dst in entries: 228 | with chdir(os.path.join(externals_root_path(), repo_name)): 229 | mkdir_p(os.path.split(dst)[0]) 230 | link(os.path.abspath(src), dst) 231 | 232 | 233 | def externals_sanity_check(): 234 | """Check that we are not trying to track various refs of the same external repo""" 235 | registry = defaultdict(set) 236 | root = root_path() 237 | 238 | def registry_add(url, path, ext): 239 | registry[url].add(ExtItem(ext['branch'], ext['ref'], path, ext.get('name', ''))) 240 | 241 | foreach_externals(root, registry_add, recursive=True) 242 | errmsg = None 243 | for url, set_ in registry.items(): 244 | # we are only interested to know if branch-ref pairs are duplicated 245 | if len({(s[0], s[1]) for s in set_}) > 1: 246 | if errmsg is None: 247 | errmsg = ["Error: one project can not refer to different branches/refs of the same git external repository,", 248 | "however it appears to be the case for:"] 249 | errmsg.append('\t- {}, tracked as:'.format(url)) 250 | for i in set_: 251 | errmsg.append("\t\t- external directory: '{0}'".format(os.path.relpath(i.path, root))) 252 | errmsg.append("\t\t branch: '{0}', ref: '{1}'".format(i.branch, i.ref)) 253 | if errmsg is not None: 254 | errmsg.append("Please correct the corresponding {0} before proceeding".format(EXTERNALS_JSON)) 255 | error('\n'.join(errmsg), exitcode=1) 256 | info('externals sanity check passed!') 257 | 258 | # TODO: check if we don't have duplicate entries under `.git_externals/` 259 | 260 | 261 | def filter_externals_not_needed(all_externals, entries): 262 | git_externals = {} 263 | for repo_name, repo_val in all_externals.items(): 264 | filtered_targets = {} 265 | for src, dsts in repo_val['targets'].items(): 266 | filtered_dsts = [] 267 | for dst in dsts: 268 | inside_external = any([os.path.abspath(dst).startswith(e) for e in entries]) 269 | if inside_external: 270 | filtered_dsts.append(dst) 271 | 272 | if filtered_dsts: 273 | filtered_targets[src] = filtered_dsts 274 | 275 | if filtered_targets: 276 | git_externals[repo_name] = all_externals[repo_name] 277 | git_externals[repo_name]['targets'] = filtered_targets 278 | 279 | return git_externals 280 | 281 | def resolve_revision(ref, mode='git'): 282 | assert mode in ('git', 'svn'), "mode = {} not in (git, svn)".format(mode) 283 | if ref is not None: 284 | if ref.startswith('svn:r'): 285 | # echo("Resolving {}".format(ref)) 286 | ref = ref.strip('svn:r') 287 | # If the revision starts with 'svn:r' in 'git' mode we search 288 | # for the matching hash. 289 | if mode == 'git': 290 | ref = git('log', '--grep', 'git-svn-id:.*@%s' % ref, '--format=%H', capture=True).strip() 291 | return ref 292 | 293 | def gitext_up(recursive, entries=None, reset=False, use_gitsvn=False): 294 | 295 | if not os.path.exists(externals_json_path()): 296 | return 297 | 298 | all_externals = load_gitexts() 299 | git_externals = all_externals if entries is None else filter_externals_not_needed(all_externals, entries) 300 | 301 | def egit(command, *args): 302 | if command == 'checkout' and reset: 303 | args = ('--force',) + args 304 | git(command, *args, capture=False) 305 | 306 | def git_initial_checkout(repo_name, repo_url): 307 | """Perform the initial git clone (or sparse checkout)""" 308 | dirs = git_externals[ext_repo]['targets'].keys() 309 | if './' not in dirs: 310 | echo('Doing a sparse checkout of:', ', '.join(dirs)) 311 | sparse_checkout(repo_name, repo_url, dirs) 312 | else: 313 | egit('clone', repo_url, repo_name) 314 | 315 | def git_update_checkout(reset): 316 | """Update an already existing git working tree""" 317 | if reset: 318 | egit('reset', '--hard') 319 | egit('clean', '-df') 320 | egit('fetch', '--all') 321 | egit('fetch', '--tags') 322 | if 'tag' in git_externals[ext_repo]: 323 | echo('Checking out tag', git_externals[ext_repo]['tag']) 324 | egit('checkout', git_externals[ext_repo]['tag']) 325 | else: 326 | echo('Checking out branch', git_externals[ext_repo]['branch']) 327 | egit('checkout', git_externals[ext_repo]['branch']) 328 | 329 | rev = get_rev(ext_repo) 330 | if rev is not None: 331 | echo('Checking out commit', rev) 332 | egit('checkout', rev) 333 | 334 | def get_rev(ext_repo, mode='git'): 335 | ref = git_externals[ext_repo]['ref'] 336 | return resolve_revision(ref, mode) 337 | 338 | def gitsvn_initial_checkout(repo_name, repo_url): 339 | """Perform the initial git-svn clone (or sparse checkout)""" 340 | min_rev = get_rev(ext_repo, mode='svn') or 'HEAD' 341 | gitsvn('clone', normalized_ext_repo, repo_name, '-r%s' % min_rev, capture=False) 342 | 343 | def gitsvn_update_checkout(reset): 344 | """Update an already existing git-svn working tree""" 345 | # FIXME: seems this might be necessary sometimes (happened with 346 | # 'vectorfonts' for example that the following error: "Unable to 347 | # determine upstream SVN information from HEAD history" was fixed by 348 | # adding that, but breaks sometimes. (investigate) 349 | # git('rebase', '--onto', 'git-svn', '--root', 'master') 350 | gitsvnrebase('.', capture=False) 351 | rev = get_rev(ext_repo) or 'git-svn' 352 | echo('Checking out commit', rev) 353 | git('checkout', rev) 354 | 355 | def svn_initial_checkout(repo_name, repo_url): 356 | """Perform the initial svn checkout""" 357 | svn('checkout', '--ignore-externals', normalized_ext_repo, repo_name, capture=False) 358 | 359 | def svn_update_checkout(reset): 360 | """Update an already existing svn working tree""" 361 | if reset: 362 | svn('revert', '-R', '.') 363 | rev = get_rev(ext_repo, mode='svn') or 'HEAD' 364 | echo('Updating to commit', rev) 365 | svn('up', '--ignore-externals', '-r%s' % rev, capture=False) 366 | 367 | def autosvn_update_checkout(reset): 368 | if os.path.exists('.git'): 369 | gitsvn_update_checkout(reset) 370 | else: 371 | svn_update_checkout(reset) 372 | 373 | for ext_repo in git_externals.keys(): 374 | normalized_ext_repo = normalize_gitext_url(ext_repo) 375 | 376 | if all_externals[ext_repo]['vcs'] == 'git': 377 | _initial_checkout = git_initial_checkout 378 | _update_checkout = git_update_checkout 379 | else: 380 | if use_gitsvn: 381 | _initial_checkout = gitsvn_initial_checkout 382 | else: 383 | _initial_checkout = svn_initial_checkout 384 | _update_checkout = autosvn_update_checkout 385 | 386 | mkdir_p(externals_root_path()) 387 | with chdir(externals_root_path()): 388 | repo_name = get_repo_name(normalized_ext_repo) 389 | ext_name = git_externals[ext_repo].get('name', '') 390 | ext_name = ext_name if ext_name else repo_name 391 | 392 | info('External', ext_name) 393 | if not os.path.exists(ext_name): 394 | echo('Cloning external', ext_name) 395 | _initial_checkout(ext_name, normalized_ext_repo) 396 | 397 | with chdir(ext_name): 398 | echo('Retrieving changes from server: ', ext_name) 399 | _update_checkout(reset) 400 | 401 | link_entries(git_externals) 402 | 403 | if recursive: 404 | for ext_repo in git_externals.keys(): 405 | entries = [os.path.realpath(d) 406 | for t in git_externals[ext_repo]['targets'].values() 407 | for d in t] 408 | with chdir(os.path.join(externals_root_path(), get_repo_name(ext_repo))): 409 | gitext_up(recursive, entries, reset=reset, use_gitsvn=use_gitsvn) 410 | 411 | 412 | def gitext_recursive_info(root_dir, recursive=True, externals=[]): 413 | git_exts = {ext_repo: ext for ext_repo, ext in load_gitexts().items() 414 | if os.path.exists(os.path.join(externals_root_path(), get_repo_name(ext_repo)))} 415 | 416 | for ext_repo, ext in git_exts.items(): 417 | entries = [os.path.realpath(d) 418 | for t in git_exts[ext_repo]['targets'].values() 419 | for d in t] 420 | 421 | cwd = os.getcwd() 422 | repo_name = get_repo_name(ext_repo) 423 | if externals and repo_name not in externals: 424 | continue 425 | 426 | with chdir(os.path.join(externals_root_path(), repo_name)): 427 | filtered = filter_externals_not_needed(load_gitexts(), entries) 428 | print_gitext_info(ext_repo, ext, root_dir, checkout=os.getcwd()) 429 | 430 | # if required, recurse into the externals repo of current external 431 | if recursive: 432 | for dsts in ext['targets'].values(): 433 | for dst in dsts: 434 | real_dst = os.path.realpath(os.path.join(cwd, dst)) 435 | 436 | has_deps = any([os.path.realpath(d).startswith(real_dst) 437 | for e in filtered.values() 438 | for ds in e['targets'].values() 439 | for d in ds]) 440 | 441 | if has_deps: 442 | gitext_recursive_info(os.path.join(root_dir, dst)) 443 | 444 | 445 | def print_gitext_info(ext_repo, ext, root_dir, checkout=False): 446 | """ 447 | print information for all externals, recursively or not. 448 | `checkout` controls if printing the `Checkout` field (i.e real checkout 449 | directory) is required or not. 450 | """ 451 | click.secho('Repo: {}'.format(ext_repo), fg='blue') 452 | if checkout: 453 | click.echo('Checkout: {}'.format(checkout)) 454 | 455 | if 'tag' in ext: 456 | click.echo('Tag: {}'.format(ext['tag'])) 457 | else: 458 | click.echo('Branch: {}'.format(ext['branch'])) 459 | click.echo('Ref: {}'.format(ext['ref'])) 460 | 461 | if 'name' in ext: 462 | click.echo('Name: {}'.format(ext['name'])) 463 | 464 | for src, dsts in ext['targets'].items(): 465 | for dst in dsts: 466 | click.echo(' {} -> {}'.format(src, os.path.join(root_dir, dst))) 467 | 468 | click.echo('') 469 | 470 | 471 | def iter_externals(externals, verbose=True): 472 | if not externals: 473 | externals = get_entries() 474 | 475 | for entry in externals: 476 | entry_path = os.path.join(externals_root_path(), entry) 477 | 478 | if not os.path.exists(entry_path): 479 | error('External {} not found'.format(entry), exitcode=None) 480 | continue 481 | 482 | with chdir(entry_path): 483 | if verbose: 484 | info('External {}'.format(entry)) 485 | yield entry 486 | -------------------------------------------------------------------------------- /git_externals/gitext.completion.bash: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | # 3 | # Bash completion for git-externals 4 | # 5 | # ============ 6 | # Installation 7 | # ============ 8 | # 9 | # Completion after typing "git-externals" 10 | # =========================== 11 | # Put: "source gitext.completion.bash" in your .bashrc 12 | # or place it in "/etc/bash_completion.d" folder, it will 13 | # be sourced automatically. 14 | # 15 | # completion after typing "git externals" (through git-completion) 16 | # =========================== 17 | # Put: Ensure git-completion is installed (normally it comes 18 | # with Git on Debian systems). In case it isn't, see: 19 | # https://github.com/git/git/blob/master/contrib/completion/git-completion.bash 20 | 21 | _git_ext_cmds=" \ 22 | add \ 23 | diff \ 24 | foreach \ 25 | info \ 26 | freeze \ 27 | remove \ 28 | status \ 29 | update 30 | " 31 | 32 | _git_externals () 33 | { 34 | local subcommands="$(echo $_git_ext_cmds)" 35 | local subcommand="$(__git_find_on_cmdline "$subcommands")" 36 | 37 | if [ -z "$subcommand" ]; then 38 | __gitcomp "$subcommands" 39 | return 40 | fi 41 | 42 | case "$subcommand" in 43 | add) 44 | __git_ext_add 45 | return 46 | ;; 47 | diff) 48 | __git_ext_diff 49 | return 50 | ;; 51 | info) 52 | __git_ext_info 53 | return 54 | ;; 55 | update|foreach) 56 | __git_ext_update_foreach 57 | return 58 | ;; 59 | remove) 60 | __git_ext_remove 61 | return 62 | ;; 63 | status) 64 | __git_ext_status 65 | return 66 | ;; 67 | *) 68 | COMPREPLY=() 69 | ;; 70 | esac 71 | } 72 | 73 | __git_ext_info () 74 | { 75 | local cur 76 | local opts="" 77 | COMPREPLY=() 78 | cur="${COMP_WORDS[COMP_CWORD]}" 79 | 80 | case "$cur" in 81 | -*) opts="--recursive --no-recursive" ;; 82 | *) __gitext_complete_externals "${cur}" ;; 83 | esac 84 | 85 | if [[ -n "${opts}" ]]; then 86 | COMPREPLY=( ${COMPREPLY[@]:-} $(compgen -W "${opts}" -- "${cur}") ) 87 | fi 88 | } 89 | 90 | __git_ext_status () 91 | { 92 | local cur 93 | local opts="" 94 | COMPREPLY=() 95 | cur="${COMP_WORDS[COMP_CWORD]}" 96 | 97 | case "$cur" in 98 | -*) opts="--porcelain --verbose --no-verbose" ;; 99 | *) __gitext_complete_externals "${cur}" ;; 100 | esac 101 | 102 | if [[ -n "${opts}" ]]; then 103 | COMPREPLY=( ${COMPREPLY[@]:-} $(compgen -W "${opts}" -- "${cur}") ) 104 | fi 105 | } 106 | 107 | __git_ext_diff () 108 | { 109 | local cur 110 | COMPREPLY=() 111 | cur="${COMP_WORDS[COMP_CWORD]}" 112 | 113 | __gitext_complete_externals "${cur}" 114 | } 115 | 116 | __git_ext_update_foreach () 117 | { 118 | local opts="" 119 | opts="--recursive --no-recursive --gitsvn --no-gitsvn --reset" 120 | COMPREPLY=( ${COMPREPLY[@]:-} $(compgen -W "${opts}" -- "${cur}") ) 121 | } 122 | 123 | __git_ext_add () 124 | { 125 | local opts="" 126 | opts="--branch --tag" 127 | COMPREPLY=( ${COMPREPLY[@]:-} $(compgen -W "${opts}" -- "${cur}") ) 128 | } 129 | 130 | __git_externals () 131 | { 132 | local cur prev 133 | local i cmd cmd_index option option_index 134 | local opts="" 135 | COMPREPLY=() 136 | cur="${COMP_WORDS[COMP_CWORD]}" 137 | prev="${COMP_WORDS[COMP_CWORD-1]}" 138 | 139 | # Search for the subcommand 140 | local skip_next=0 141 | for ((i=1; $i<=$COMP_CWORD; i++)); do 142 | if [[ ${skip_next} -eq 1 ]]; then 143 | skip_next=0; 144 | elif [[ ${COMP_WORDS[i]} != -* ]]; then 145 | cmd="${COMP_WORDS[i]}" 146 | cmd_index=${i} 147 | break 148 | elif [[ ${COMP_WORDS[i]} == -f ]]; then 149 | skip_next=1 150 | fi 151 | done 152 | 153 | options="" 154 | if [[ $COMP_CWORD -le $cmd_index ]]; then 155 | # The user has not specified a subcommand yet 156 | COMPREPLY=( ${COMPREPLY[@]:-} $(compgen -W "${_git_ext_cmds}" -- "${cur}") ) 157 | else 158 | case ${cmd} in 159 | diff) 160 | __git_ext_diff ;; 161 | info) 162 | __git_ext_info ;; 163 | status) 164 | __git_ext_status ;; 165 | update|foreach) 166 | __git_ext_update_foreach ;; 167 | add) 168 | __git_ext_update_add ;; 169 | esac # case ${cmd} 170 | fi # command specified 171 | 172 | if [[ -n "${options}" ]]; then 173 | COMPREPLY=( ${COMPREPLY[@]:-} $(compgen -W "${options}" -- "${cur}") ) 174 | fi 175 | } 176 | 177 | __gitext_complete_externals () 178 | { 179 | local IFS=$'\n' 180 | local cur="${1}" 181 | COMPREPLY=( ${COMPREPLY[@]:-} $(compgen -W "$(git-externals list 2>/dev/null)" -- "${cur}") ) 182 | } 183 | 184 | # alias __git_find_on_cmdline for backwards compatibility 185 | if [ -z "`type -t __git_find_on_cmdline`" ]; then 186 | alias __git_find_on_cmdline=__git_find_subcommand 187 | fi 188 | 189 | complete -F __git_externals git-externals 190 | -------------------------------------------------------------------------------- /git_externals/utils.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | from __future__ import print_function 4 | 5 | import subprocess 6 | import os 7 | import sys 8 | import logging 9 | import re 10 | 11 | from subprocess import check_call 12 | from contextlib import contextmanager 13 | 14 | 15 | class ProgError(Exception): 16 | def __init__(self, prog='', errcode=1, errmsg='', args=''): 17 | if isinstance(args, tuple): 18 | args = u' '.join(args) 19 | super(ProgError, self).__init__(u'\"{} {}\" {}'.format(prog, args, errmsg)) 20 | self.prog = prog 21 | self.errcode = errcode 22 | 23 | def __str__(self): 24 | name = u'{}Error'.format(self.prog.title()) 25 | msg = super(ProgError, self).__str__() 26 | return u'<{}: {} {}>'.format(name, self.errcode, msg) 27 | 28 | 29 | class GitError(ProgError): 30 | def __init__(self, **kwargs): 31 | super(GitError, self).__init__(prog='git', **kwargs) 32 | 33 | 34 | class SvnError(ProgError): 35 | def __init__(self, **kwargs): 36 | super(SvnError, self).__init__(prog='svn', **kwargs) 37 | 38 | 39 | class GitSvnError(ProgError): 40 | def __init__(self, **kwargs): 41 | super(GitSvnError, self).__init__(prog='git-svn', **kwargs) 42 | 43 | 44 | class CommandError(ProgError): 45 | def __init__(self, cmd, **kwargs): 46 | super(CommandError, self).__init__(prog=cmd, **kwargs) 47 | 48 | 49 | def svn(*args, **kwargs): 50 | universal_newlines = kwargs.get('universal_newlines', True) 51 | output, err, errcode = _command('svn', *args, capture=True, universal_newlines=universal_newlines) 52 | if errcode != 0: 53 | print("running svn ", args) 54 | raise SvnError(errcode=errcode, errmsg=err) 55 | return output 56 | 57 | 58 | def git(*args, **kwargs): 59 | capture = kwargs.get('capture', True) 60 | output, err, errcode = _command('git', *args, capture=capture, universal_newlines=True) 61 | if errcode != 0: 62 | raise GitError(errcode=errcode, errmsg=err, args=args) 63 | return output 64 | 65 | 66 | def gitsvn(*args, **kwargs): 67 | capture = kwargs.get('capture', True) 68 | output, err, errcode = _command('git', 'svn', *args, capture=capture, universal_newlines=True) 69 | if errcode != 0: 70 | raise GitSvnError(errcode=errcode, errmsg=err, args=args) 71 | return output 72 | 73 | 74 | def gitsvnrebase(*args, **kwargs): 75 | capture = kwargs.get('capture', True) 76 | output, err, errcode = _command('git-svn-rebase', *args, capture=capture, universal_newlines=True) 77 | if errcode != 0: 78 | raise GitSvnError(errcode=errcode, errmsg=err, args=args) 79 | return output 80 | 81 | 82 | def command(cmd, *args, **kwargs): 83 | universal_newlines = kwargs.get('universal_newlines', True) 84 | capture = kwargs.get('capture', True) 85 | output, err, errcode = _command(cmd, *args, universal_newlines=universal_newlines, capture=capture) 86 | if errcode != 0: 87 | raise CommandError(cmd, errcode=errcode, errmsg=err, args=args) 88 | return output 89 | 90 | 91 | def _command(cmd, *args, **kwargs): 92 | env = kwargs.get('env', dict(os.environ)) 93 | env.setdefault('LC_MESSAGES', 'C') 94 | universal_newlines = kwargs.get('universal_newlines', True) 95 | capture = kwargs.get('capture', True) 96 | if capture: 97 | stdout, stderr = subprocess.PIPE, subprocess.PIPE 98 | else: 99 | stdout, stderr = None, None 100 | 101 | p = subprocess.Popen([cmd] + list(args), 102 | stdout=stdout, 103 | stderr=stderr, 104 | universal_newlines=universal_newlines, 105 | env=env) 106 | output, err = p.communicate() 107 | return output, err, p.returncode 108 | 109 | 110 | def current_branch(): 111 | return git('name-rev', '--name-only', 'HEAD').strip() 112 | 113 | 114 | def branches(): 115 | refs = git('for-each-ref', 'refs/heads', "--format=%(refname)") 116 | return [line.split('/')[2] for line in refs.splitlines()] 117 | 118 | 119 | def tags(): 120 | refs = git('for-each-ref', 'refs/tags', "--format=%(refname)") 121 | return [line.split('/')[2] for line in refs.splitlines()] 122 | 123 | 124 | TAGS_RE = re.compile('.+/tags/(.+)') 125 | 126 | def git_remote_branches_and_tags(): 127 | output = git('branch', '-r') 128 | 129 | _branches, _tags = [], [] 130 | 131 | for line in output.splitlines(): 132 | line = line.strip() 133 | m = TAGS_RE.match(line) 134 | 135 | t = _tags if m is not None else _branches 136 | t.append(line) 137 | 138 | return _branches, _tags 139 | 140 | 141 | @contextmanager 142 | def checkout(branch, remote=None, back_to='master', force=False): 143 | brs = set(branches()) 144 | 145 | cmd = ['git', 'checkout'] 146 | if force: 147 | cmd += ['--force'] 148 | # if remote is not None -> create local branch from remote 149 | if remote is not None and branch not in brs: 150 | check_call(cmd + ['-b', branch, remote]) 151 | else: 152 | check_call(cmd + [branch]) 153 | yield 154 | check_call(cmd + [back_to]) 155 | 156 | 157 | @contextmanager 158 | def chdir(path): 159 | cwd = os.path.abspath(os.getcwd()) 160 | 161 | try: 162 | os.chdir(path) 163 | yield 164 | finally: 165 | os.chdir(cwd) 166 | 167 | 168 | def mkdir_p(path): 169 | if path != '' and not os.path.exists(path): 170 | os.makedirs(path) 171 | 172 | 173 | def header(msg): 174 | banner = '=' * 78 175 | 176 | print('') 177 | print(banner) 178 | print(u'{:^78}'.format(msg)) 179 | print(banner) 180 | 181 | 182 | def print_msg(msg): 183 | print(u' {}'.format(msg)) 184 | 185 | 186 | def decode_utf8(msg): 187 | """ 188 | Py2 / Py3 decode 189 | """ 190 | try: 191 | return msg.decode('utf8') 192 | except AttributeError: 193 | return msg 194 | 195 | 196 | if not sys.platform.startswith('win32'): 197 | link = os.symlink 198 | rm_link = os.remove 199 | 200 | # following works but it requires admin privileges 201 | else: 202 | if sys.getwindowsversion()[0] >= 6: 203 | def link(src, dst): 204 | import ctypes 205 | csl = ctypes.windll.kernel32.CreateSymbolicLinkW 206 | csl.argtypes = (ctypes.c_wchar_p, ctypes.c_wchar_p, ctypes.c_uint32) 207 | csl.restype = ctypes.c_ubyte 208 | if csl(dst, src, 0 if os.path.isfile(src) else 1) == 0: 209 | print("Error in CreateSymbolicLinkW(%s, %s)" % (dst, src)) 210 | raise ctypes.WinError() 211 | else: 212 | import shutil 213 | def link(src, dst): 214 | if os.path.isfile(src): 215 | print_msg("WARNING: Unsupported SymLink on Windows before Vista, single files will be copied") 216 | shutil.copy2(src, dst) 217 | else: 218 | try: 219 | subprocess.check_call(['junction', dst, src], shell=True) 220 | except: 221 | print_msg("ERROR: Is http://live.sysinternals.com/junction.exe in your PATH?") 222 | raise 223 | 224 | def rm_link(path): 225 | if os.path.isfile(path): 226 | os.remove(path) 227 | else: 228 | os.rmdir(path) 229 | 230 | 231 | class IndentedLoggerAdapter(logging.LoggerAdapter): 232 | def __init__(self, logger, indent_val=4): 233 | super(IndentedLoggerAdapter, self).__init__(logger, {}) 234 | self.indent_level = 0 235 | self.indent_val = indent_val 236 | 237 | def process(self, msg, kwargs): 238 | return (' ' * self.indent_level + msg, kwargs) 239 | 240 | @contextmanager 241 | def indent(self): 242 | self.indent_level += self.indent_val 243 | yield 244 | self.indent_level -= self.indent_val 245 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [flake8] 2 | ignore = 3 | # E1: Indentation 4 | E1, 5 | # E2: Whitespace 6 | E2, 7 | # E3: Blank line 8 | E3, 9 | # E5: Line length 10 | E5, 11 | # E731: do not assign a lambda expression, use a def 12 | E731, 13 | # E402: module level import not at top of file 14 | E402, 15 | # W503: line break before binary operator 16 | W503 17 | max_line_length = 110 18 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | try: 5 | from setuptools import setup 6 | except ImportError: 7 | from distutils.core import setup 8 | 9 | 10 | with open('git_externals/__init__.py') as fp: 11 | exec(fp.read()) 12 | 13 | 14 | classifiers = [ 15 | 'Development Status :: 4 - Beta', 16 | 'Programming Language :: Python', 17 | 'Programming Language :: Python :: 2', 18 | 'Programming Language :: Python :: 2.6', 19 | 'Programming Language :: Python :: 2.7', 20 | 'Programming Language :: Python :: 3', 21 | 'Programming Language :: Python :: 3.4', 22 | 'Programming Language :: Python :: 3.5', 23 | 'Topic :: Software Development :: Libraries :: Python Modules', 24 | ] 25 | 26 | setup( 27 | name='git-externals', 28 | version=__version__, 29 | description='cli tool to manage git externals', 30 | long_description='Ease the migration from Git to SVN by handling svn externals through a cli tool', 31 | packages=['git_externals'], 32 | install_requires=['click', 33 | 'git-svn-clone-externals'], 34 | entry_points={ 35 | 'console_scripts': [ 36 | 'git-externals = git_externals.cli:cli', 37 | 'svn-externals-info = git_externals.process_externals:main', 38 | ], 39 | }, 40 | author=__author__, 41 | author_email=__email__, 42 | license='MIT', 43 | classifiers=classifiers, 44 | ) 45 | -------------------------------------------------------------------------------- /tests/basic.t: -------------------------------------------------------------------------------- 1 | Usage without options: 2 | 3 | $ git externals 4 | Usage: git-externals [OPTIONS] COMMAND [ARGS]... 5 | 6 | Utility to manage git externals, meant to be used as a drop-in replacement 7 | to svn externals 8 | 9 | This script works by cloning externals found in the `git_externals.json` 10 | file into `.git_externals/` and symlinks them to recreate the wanted 11 | directory layout. 12 | 13 | Options: 14 | --version Show the version and exit. 15 | --with-color / --no-color Enable/disable colored output 16 | -h, --help Show this message and exit. 17 | 18 | Commands: 19 | add Add a git external to the current repo. 20 | diff Call git diff on the given externals 21 | foreach Evaluates an arbitrary shell command in each... 22 | freeze Freeze the externals revision 23 | info Print some info about the externals. 24 | remove Remove the externals at the given repository... 25 | status Call git status on the given externals 26 | update Update the working copy cloning externals if... 27 | \n+ (re) 28 | -------------------------------------------------------------------------------- /tests/git-worktree.t: -------------------------------------------------------------------------------- 1 | Svn external in a git worktree: 2 | 3 | $ mkdir master 4 | $ cd master 5 | $ git init . 6 | Initialized empty Git repository in /tmp/cramtests-*/git-worktree.t/master/.git/ (glob) 7 | $ git config user.email "externals@test.com" 8 | $ git config user.name "Git Externals" 9 | $ git externals add -b trunk -c svn https://svn.riouxsvn.com/svn-test-repo/trunk ./ test-repo-svn > /dev/null 2>&1 10 | $ git externals update > /dev/null 2>&1 11 | $ ls . | grep test-repo-svn 12 | test-repo-svn 13 | $ git add git_externals.json 14 | $ git commit -m"Add git_externals.json" git_externals.json | grep "create mode" 15 | create mode 100644 git_externals.json 16 | 17 | Now we create a worktree ad try to update the externals there too 18 | 19 | $ git worktree add ../test-worktree > /dev/null 20 | Preparing ../test-worktree (identifier test-worktree) 21 | $ cd ../test-worktree 22 | $ ls . | grep test-repo-svn 23 | [1] 24 | $ git externals update > /dev/null 2>&1 25 | $ ls . | grep test-repo-svn 26 | test-repo-svn 27 | 28 | Test the upgrade to the new storage folder `.git_externals/` 29 | 30 | $ cd ../master 31 | $ readlink -f test-repo-svn 32 | /tmp/cramtests-*/git-worktree.t/master/.git_externals/trunk (glob) 33 | $ mv .git_externals/ .git/externals 34 | $ rm test-repo-svn 35 | $ ln -s .git/externals/trunk ./test-repo-svn 36 | $ readlink -f test-repo-svn # check we restored the old version location 37 | /tmp/cramtests-*/git-worktree.t/master/.git/externals/trunk (glob) 38 | 39 | $ git externals update 40 | Moving old externals path to new location 41 | externals sanity check passed! 42 | External trunk 43 | Retrieving changes from server: trunk 44 | *-*-* *:*:* INFO[git-svn-clone-externals]: git svn rebase . (glob) 45 | Current branch HEAD is up to date. 46 | Checking out commit git-svn 47 | 48 | $ readlink -f test-repo-svn # check we updated the link too 49 | /tmp/cramtests-*/git-worktree.t/master/.git_externals/trunk (glob) 50 | -------------------------------------------------------------------------------- /tests/svn-target-freeze-svn.t: -------------------------------------------------------------------------------- 1 | Svn external freeze (with --vcs and --no-gitsvn): 2 | 3 | $ git init . 4 | Initialized empty Git repository in /tmp/cramtests-*/svn-target-freeze-svn.t/.git/ (glob) 5 | $ git externals add -b trunk -c svn -r svn:r10 https://svn.riouxsvn.com/svn-test-repo/trunk ./ test-repo-svn 6 | Repo: https://svn.riouxsvn.com/svn-test-repo/trunk 7 | Branch: trunk 8 | Ref: svn:r10 9 | ./ -> ./test-repo-svn 10 | 11 | $ git externals update --no-gitsvn 12 | externals sanity check passed! 13 | External trunk 14 | Cloning external trunk 15 | Retrieving changes from server: trunk 16 | Updating to commit 10 17 | 18 | $ (cd test-repo-svn && svn log --limit 1) 19 | ------------------------------------------------------------------------ 20 | r10 | naufraghi | 2017-02-02 23:06:36 +0000 (Thu, 02 Feb 2017) | 1 line 21 | 22 | Add citation 23 | ------------------------------------------------------------------------ 24 | 25 | Test version bump: 26 | 27 | $ (cd test-repo-svn && svnversion -c) 28 | 5:10 29 | 30 | $ (cd test-repo-svn && svn update -rHEAD | grep revision) 31 | Updated to revision 16. 32 | 33 | $ (cd test-repo-svn && svnversion -c) 34 | 5:15 35 | 36 | $ git externals freeze 37 | Freeze https://svn.riouxsvn.com/svn-test-repo/trunk at svn:r15 38 | 39 | $ git externals freeze --messages # no crash but not supported 40 | Freeze https://svn.riouxsvn.com/svn-test-repo/trunk at svn:r15 41 | -------------------------------------------------------------------------------- /tests/svn-target-freeze.t: -------------------------------------------------------------------------------- 1 | Svn external freeze (with --vcs): 2 | 3 | $ git init . 4 | Initialized empty Git repository in /tmp/cramtests-*/svn-target-freeze.t/.git/ (glob) 5 | $ git externals add -b trunk -c svn -r svn:r10 https://svn.riouxsvn.com/svn-test-repo/trunk ./ test-repo-svn 6 | Repo: https://svn.riouxsvn.com/svn-test-repo/trunk 7 | Branch: trunk 8 | Ref: svn:r10 9 | ./ -> ./test-repo-svn 10 | 11 | $ git externals update 2>&1 | grep -v "svn:mergeinfo" # normalize git svn output 12 | externals sanity check passed! 13 | External trunk 14 | Cloning external trunk 15 | Initialized empty Git repository in /tmp/cramtests-*/svn-target-freeze.t/.git*externals/trunk/.git/ (glob) 16 | \tA\t*.* (esc) (glob) 17 | \tA\t*.* (esc) (glob) 18 | r10 = e05cc85567e5e18004ba1fc55ed7599ba94d2b5a (refs/remotes/git-svn) 19 | Checked out HEAD: 20 | https://svn.riouxsvn.com/svn-test-repo/trunk r10 21 | Retrieving changes from server: trunk 22 | *-*-* *:*:* INFO[git-svn-clone-externals]: git svn rebase . (glob) 23 | \tM\tREADME.md (esc) 24 | r11 = 2c875802afea7d5cb498b902af94eeb3b42bbe73 (refs/remotes/git-svn) 25 | \tM\tREADME.md (esc) 26 | Couldn't find revmap for https://svn.riouxsvn.com/svn-test-repo/trunk/branches/issue-1 27 | r13 = 6d4767de6645e6bd59e3437036e8a0e3137203ef (refs/remotes/git-svn) 28 | \tA\tdocs/docs.md (esc) 29 | r15 = fbe3da492ad7013e779968d7d7183cce742e501a (refs/remotes/git-svn) 30 | First, rewinding head to replay your work on top of it... 31 | Fast-forwarded master to refs/remotes/git-svn. 32 | Checking out commit e05cc85567e5e18004ba1fc55ed7599ba94d2b5a 33 | 34 | $ (cd test-repo-svn && git log -1) 35 | commit e05cc85567e5e18004ba1fc55ed7599ba94d2b5a 36 | Author: naufraghi 37 | Date: Thu Feb 2 23:06:36 2017 +0000 38 | 39 | Add citation 40 | 41 | git-svn-id: https://svn.riouxsvn.com/svn-test-repo/trunk@10 fa4e49d6-2000-40a9-909b-e7fe548600ae 42 | 43 | Test version bump: 44 | 45 | $ (cd test-repo-svn && svnversion -c) 46 | (Unversioned directory|exported) (re) 47 | 48 | $ (cd test-repo-svn && git checkout master) 49 | Previous HEAD position was e05cc85... Add citation 50 | Switched to branch 'master' 51 | 52 | $ git externals freeze --messages 53 | Freeze https://svn.riouxsvn.com/svn-test-repo/trunk at svn:r15 54 | - fbe3da4 Add some docs 55 | - 6d4767d Merge branch issue-1 56 | - 2c87580 Add citation evolved 57 | 58 | $ git externals freeze 59 | Freeze https://svn.riouxsvn.com/svn-test-repo/trunk at svn:r15 60 | 61 | Test freeze ignores the current pretty.format git config: 62 | 63 | $ git config --local pretty.format '%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' 64 | 65 | $ git externals freeze 66 | Freeze https://svn.riouxsvn.com/svn-test-repo/trunk at svn:r15 67 | -------------------------------------------------------------------------------- /tests/svn-target.t: -------------------------------------------------------------------------------- 1 | Svn external (with --vcs): 2 | 3 | $ git init . 4 | Initialized empty Git repository in /tmp/cramtests-*/svn-target.t/.git/ (glob) 5 | $ git externals add -b trunk -c svn https://svn.riouxsvn.com/svn-test-repo/trunk ./ test-repo-svn 6 | Repo: https://svn.riouxsvn.com/svn-test-repo/trunk 7 | Branch: trunk 8 | Ref: None 9 | ./ -> ./test-repo-svn 10 | 11 | $ git externals update 2>&1 | grep -v "svn:mergeinfo" # normalize git svn output 12 | externals sanity check passed! 13 | External trunk 14 | Cloning external trunk 15 | Initialized empty Git repository in /tmp/cramtests-*/svn-target.t/.git*externals/trunk/.git/ (glob) 16 | \tA\t*.* (esc) (glob) 17 | \tA\t*.* (esc) (glob) 18 | \tA\t*.* (esc) (glob) 19 | r15 = ed87ed5ee42a55f1a903a64d12fd11038b06fa97 (refs/remotes/git-svn) 20 | Checked out HEAD: 21 | https://svn.riouxsvn.com/svn-test-repo/trunk r15 22 | Retrieving changes from server: trunk 23 | *-*-* *:*:* INFO[git-svn-clone-externals]: git svn rebase . (glob) 24 | Current branch master is up to date. 25 | Checking out commit git-svn 26 | 27 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py27 3 | [testenv] 4 | deps = cram 5 | commands = cram tests/ -v 6 | passenv = HOME 7 | --------------------------------------------------------------------------------