├── LICENSE ├── git-cache └── README.md /LICENSE: -------------------------------------------------------------------------------- 1 | DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE 2 | Version 2, December 2004 3 | 4 | Copyright (C) 2004 Sam Hocevar 5 | 6 | Everyone is permitted to copy and distribute verbatim or modified 7 | copies of this license document, and changing it is allowed as long 8 | as the name is changed. 9 | 10 | DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE 11 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 12 | 13 | 0. You just DO WHAT THE FUCK YOU WANT TO. 14 | -------------------------------------------------------------------------------- /git-cache: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Cache regularly cloned Git repositories 3 | # Licence: WTFPL 2.0 4 | 5 | # Get cache directory 6 | CACHE=$(git config cache.directory) 7 | if [ ! -d "$CACHE" -a ! "$1" = "init" ] 8 | then 9 | echo "Inexistant cache directory, please create it with \`git cache init [DIR]'" 10 | exit 1 11 | fi 12 | 13 | # Trigger 'init' action 14 | if [ "$1" = "init" ] 15 | then 16 | 17 | if [ -d "$CACHE" -a -f "$CACHE/HEAD" -a ! "$2" = "$CACHE" ] 18 | then 19 | echo "Cache directory is already initialised. You can use \`git cache delete --force' to delete it." 20 | echo "$CACHE" 21 | exit 1 22 | fi 23 | 24 | # Try to create directory in the following order: 25 | # 1) given by argument 26 | # 2) /var/cache/git-cache if root or sudo 27 | # 3) ~/.cache/git-cache else 28 | CACHE="$2" 29 | TYPE=global 30 | [ "$CACHE" = "" -a -w /var/cache ] && CACHE=/var/cache/git-cache && TYPE=system 31 | [ "$CACHE" = "" ] && CACHE=~/.cache/git-cache 32 | mkdir -p "$CACHE" 33 | cd "$CACHE" 34 | git init --bare --shared=group 35 | git config --$TYPE cache.directory "$CACHE" 36 | 37 | # Trigger 'delete' action 38 | elif [ "$1" = "delete" ] 39 | then 40 | 41 | if [ ! "$2" = "--force" ] 42 | then 43 | echo "Please add: 1) the --force option; to confirm you want to delete the whole cache directory." 44 | exit 1 45 | fi 46 | 47 | if [ ! "$(git config --system cache.directory)" = "" ] 48 | then 49 | CACHE="$(git config --system cache.directory)" 50 | git config --system --remove-section cache 2>/dev/null && rm -rf "$CACHE" && echo "System cache directory $CACHE deleted." 51 | fi 52 | 53 | if [ ! "$(git config --global cache.directory)" = "" ] 54 | then 55 | CACHE="$(git config --global cache.directory)" 56 | git config --global --remove-section cache 2>/dev/null && rm -rf "$CACHE" && echo "Global user cache directory $CACHE deleted." 57 | fi 58 | 59 | # Trigger 'add' action 60 | elif [ "$1" = "add" ] 61 | then 62 | 63 | if [ "$2" = "" -o "$3" = "" ] 64 | then 65 | echo "Please add: 1) the remote name and 2) the remote url to be added in the cache directory." 66 | exit 1 67 | fi 68 | 69 | # Fetch new Git repository in a temporary directory for speed reasons 70 | #dir=$(mktemp -d) 71 | #cd $dir 72 | #git init --bare 73 | #git remote add --mirror=fetch "$2" "$3" 74 | #git fetch --all 75 | 76 | # Copy it in the main cache directory 77 | cd "$CACHE" 78 | #git remote add --mirror=fetch "$2" $dir 79 | #git fetch --all 80 | #git remote set-url "$2" "$3" 81 | git remote add "$2" "$3" 82 | git fetch "$2" 83 | #rm -rf $dir 84 | 85 | # Display use 86 | echo 87 | echo "You can now use the cached directory with:" 88 | echo 89 | echo " git clone --reference $CACHE $3" 90 | echo 91 | 92 | # Trigger 'rm' action 93 | elif [ "$1" = "rm" ] 94 | then 95 | 96 | if [ ! "$2" = "--force" -o "$3" = "" ] 97 | then 98 | echo "Please add: 1) the --force option and 2) the remote name; to confirm you want to delete this Git repository from the cache directory." 99 | exit 1 100 | fi 101 | 102 | cd "$CACHE" 103 | git remote rm "$3" 104 | 105 | # Trigger 'update' action 106 | elif [ "$1" = "fetch" ] 107 | then 108 | 109 | cd "$CACHE" 110 | git fetch --all --prune 111 | 112 | # Trigger 'show' action 113 | elif [ "$1" = "show" ] 114 | then 115 | 116 | cd "$CACHE" 117 | git remote -v | grep \(fetch\) | grep "$2" | awk "{print \"git clone --reference $CACHE \"\$2}" 118 | 119 | # Trigger 'help' action 120 | elif [ "$1" = "help" ] 121 | then 122 | 123 | echo 124 | echo "Use:" 125 | echo 126 | echo " git cache init [DIR] initialise the cache directory" 127 | echo " git cache delete --force delete the cache directory" 128 | echo 129 | echo " git cache add NAME URL add a cached Git repository" 130 | echo " git cache show [NAME] show all cached Git repositories" 131 | echo " git cache fetch fetch all cached Git repository" 132 | echo " git cache rm --force NAME remove a cached Git repository" 133 | echo 134 | echo " (Any other command will be applied to the cache directory.)" 135 | echo " e.g. \`git cache gc\` or \`git cache remote show\`.)" 136 | echo 137 | echo "Standard cache directory is either:" 138 | echo 139 | echo " /var/cache/git-cache if created by root (then usable by every user)" 140 | echo " ~/.cache/git-cache else" 141 | echo 142 | echo "Once created, the cache directory is written in the configuration parameter" 143 | echo "'cache.directory', either in the system or global user configuration." 144 | echo 145 | 146 | # Default trigger to directly act on the cache directory 147 | else 148 | 149 | cd "$CACHE" 150 | git "$@" 151 | 152 | fi 153 | 154 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | git-cache 2 | ========= 3 | 4 | Context 5 | ------- 6 | 7 | If you repeatedly clone a Git repository, for instance for a software you use to hack, you could bothered by always waiting during the clone operation, particularly long for big softwares. On a other side, if you have multiple copies of a Git repository on your computer, it takes place you could prefer not to use. 8 | 9 | Git has a feature "--reference"/alternates, which gives the possibility to indirect commits to another directory on the same computer. The use case is: 10 | 11 | $ cd ~/test 12 | $ git clone https://example.org/git/repo.git normal-repo 13 | (you wait minutes or hours, and the resulting directory is big) 14 | $ git clone --reference ~/test/repo https://example.org/git/repo.git referenced-repo 15 | (you wait seconds, and the resulting directory is small - only the size of the files, no size used by the history) 16 | 17 | Both clones contain the whole history of the Git repository. Internally, the second clone contains a file `.git/objects/info/alternates` containing the path of the first directory, where git searches the commits it doesn’t find the in the second clone. Obviously, if you delete the first clone, the second is useless and will display tons of errors. 18 | 19 | 20 | git-cache 21 | --------- 22 | 23 | git-cache creates a central cache directory on a computer, which contains most of the commits of regularly cloned Git repositories. 24 | 25 | If you are a regular hacker of a software, say MediaWiki, you can cache most commits and then only download recent commits. Use case: 26 | 27 | $ sudo git cache init # cache is located in /var/cache/git-cache 28 | $ git cache add mediawiki https://git.wikimedia.org/git/mediawiki/core.git 29 | 30 | You can now use the cache directory with: 31 | 32 | git clone --reference /var/cache/git-cache https://git.wikimedia.org/git/mediawiki/core.git 33 | 34 | (1607 seconds, the cache takes 314 Mio) 35 | $ git clone --reference /var/cache/git-cache https://git.wikimedia.org/git/mediawiki/core.git clone-1 36 | (46 seconds, the directory takes 98 Mio, the .git subdirectory takes 17 Mio because there were new commits) 37 | $ git clone --reference /var/cache/git-cache https://git.wikimedia.org/git/mediawiki/core.git clone-2 38 | (58 seconds, the directory takes 98 Mio, the .git subdirectory takes 17 Mio because there were new commits) 39 | 40 | 41 | Usage 42 | ----- 43 | 44 | __Installation:__ Copy git-cache inside your git commands directory (on Ubuntu: /usr/lib/git-core) and be sure it can be executed by the users (mode 755). 45 | 46 | __Use:__ All commands have the format 47 | 48 | git cache ACTION PARAMETERS 49 | 50 | __Commands:__ 51 | 52 | # General maintenance commands 53 | git cache init [DIR] initialise the cache directory 54 | git cache delete --force delete the cache directory 55 | 56 | # Daily commands 57 | git cache add NAME URL add a cached Git repository 58 | git cache show [NAME] show cached Git repositoryies/repository 59 | git cache fetch fetch all cached Git repositories 60 | git cache rm --force NAME remove a cached Git repository 61 | 62 | (Any other command will be applied to the cache directory, 63 | e.g. `git cache gc` or `git cache remote show`.) 64 | 65 | __Location of the cache directory:__ The default cache directory contains all cached repositories (each Git repository is a remote). If it is created by root, the cache directory is `/var/cache/git-cache`, else it is `~/.cache/git-cache`; it can also be another directory specified in the init command. This directory could become big, be sure you have enough place. 66 | 67 | ~~You can want to cron `git cache fetch` to automatically retrieve new commits.~~ (bug) 68 | 69 | 70 | Development 71 | ----------- 72 | 73 | Bug: git cache fetch does not retrieve new commits. 74 | 75 | This is a first version, and feedback is needed to improve its daily usage. Some questions I wonder: 76 | - Are the subcommands sufficiently explicit? 77 | - Should we regularly run `git gc` or even `git gc --aggressive`? 78 | - How git behaves when some commits are both local and in the cache? Does it remove local objects to gain place? 79 | - etc. 80 | 81 | Internally the cache directory is a big bare directory containing all remote repositories, even if they do not share commits (sort of orphans branches). Should the cache directory be splitted by repository: /var/cache/git-cache/mediawiki, /var/cache/git-cache/visualeditor, etc. Particularly in the second case, the name of the remote repository is useless, and I would prefer use unique names, e.g. md5(URL) and completely hide this to the user (possibly with soft links from the URL to the directory for usability inside the cache directory). 82 | 83 | 84 | Licence 85 | ------- 86 | 87 | - Original author: [Seb35](https://github.com/Seb35) 88 | - Licence: [WTFPL 2.0](http://www.wpfpl.net) 89 | 90 | 91 | References 92 | ---------- 93 | 94 | This program is a generalisation to arbitrary Git repositories of an idea and implementation by [Randy Fay](http://randyfay.com/content/reference-cache-repositories-speed-clones-git-clone-reference) specifically for Drupal. Hopefully this generalisation is sufficiently simple to stay useable and practical. 95 | 96 | Similar program: [git-cached by dvessel](https://github.com/dvessel/git-cached) 97 | --------------------------------------------------------------------------------