├── .gitignore
├── DEVNOTES.md
├── README.md
├── app
├── models
│ └── repository
│ │ └── git_remote.rb
└── views
│ └── settings
│ └── _git_remote_settings.html.erb
├── config
└── locales
│ └── en.yml
├── images
├── available-scm.png
└── git-remote-config.png
├── init.rb
└── lib
└── redmine_git_remote
├── poor_mans_capture3.rb
└── repositories_helper_patch.rb
/.gitignore:
--------------------------------------------------------------------------------
1 | .DS_Store
2 |
--------------------------------------------------------------------------------
/DEVNOTES.md:
--------------------------------------------------------------------------------
1 | ## Dev Notes
2 |
3 | ### TODOs
4 |
5 | * integrate webhook support (callback to accept POST, figure out repo, run git fetch on it), check security / DOS
6 | * key management (currently user needs to populate ~/.ssh/* config files manually)
7 | * cleanup cloned repos on Repository#destroy
8 | * make sure git fetch doesn't hang (timeout, background, local vs remote fetch interference)
9 | * last fetched status, clearer error handling
10 | * on plugin uninstall, Redmine will crash (rails hates it when you remove model classes)
11 | * (provide a rake command to convert to Git type)
12 | * initialize_clone should only run on new objects (since only "Main repository" is editable)
13 | * key handling
14 | * removing the plugin, what happens to records?
15 | * conversion of legacy records
16 |
17 |
18 | ### Testing
19 |
20 | Figure out how to test this plugin!
21 |
22 | * circle CI for integration tests?
23 | * docker container with this plugin installed
24 | * create a dummy project, create a repo record for http://github.com/dergachev/redmine_git_remote.git
25 | * repositories_git_controller_test.rb and repositories_git_test.rb
26 |
27 | ### Misc snippets
28 |
29 | Here's some bash commands I was pasting in regularly while working on this.
30 |
31 | ```
32 | cd /home/redmine/redmine && ./script/rails runner "Repository.fetch_changesets" -e production
33 | cd /home/redmine/redmine/; bundle exec rails console production
34 | bundle exec rails dbconsole production
35 |
36 | # config.consider_all_requests_local = true
37 | apt-get update; apt-get install vim -y; vim /home/redmine/redmine/config/environments/production.rb
38 |
39 | # useful under https://github.com/dergachev/docker-redmine
40 | rsync -avq --chown=redmine:redmine /home/redmine/data/plugins/ /home/redmine/redmine/plugins/; supervisorctl restart unicorn
41 |
42 | ```
43 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | redmine_git_remote
2 | ==================
3 |
4 | Redmine plugin to automatically clone and remote git repositories.
5 |
6 | ## Installation
7 |
8 | Install the plugin as usual:
9 |
10 | ```
11 | cd REDMINE_ROOT/plugins
12 | git clone https://github.com/dergachev/redmine_git_remote
13 | ```
14 |
15 | Then enable the new GitRemote SCM type in [http://redmine-root/settings?tab=repositories](http://redmine-root/settings?tab=repositories)
16 |
17 | 
18 |
19 | The plugin shells out to the following binaries, so make sure they're available:
20 | * git 1.7.5+ - a version recent enough to support `get remote add --mirror=fetch origin URL`
21 | * ssh-keyscan
22 | * ssh-keygen
23 |
24 | ## Supporting private repos
25 |
26 | For security sake, we don't support cloning over HTTPS with username password, but only via SSH.
27 |
28 | For example:
29 |
30 | * This private repo will fail to clone: `https://github.com/dergachev/my-secret-repo`
31 | * Instead, use the SSH form: `git@github.com:evolvingweb/my-secret-repo.git`
32 |
33 | If you're going to use the SSH form, you'll need to install the appropriate SSH
34 | keys to `~/.ssh/id_rsa` (in the home directory of your redmine webserver user,
35 | likely www-data).
36 |
37 | Some extra tips:
38 |
39 | * For GitHub/GitLab, we have found it too troublesome to install repository-specific SSH keys.
40 | Instead we ended up creating recommend creating a
41 | [dedicated account for redmine](https://developer.github.com/guides/managing-deploy-keys/#machine-users)
42 | and installing the keys there.
43 | * On Ubuntu, the `www-data` user's $HOME is `/var/www`, and by default it's owned by root.
44 | That means you might have to do this before installing Redmine: `sudo mkdir /var/www/.ssh; sudo chown www-data:www-data /var/www/.ssh`
45 |
46 | ## Usage
47 |
48 | This plugin defines a new repository type, GitRemote, which allows you to associate
49 | a remote repository with your Redmine project. First create a new repository of type
50 | GitRemote, enter the clone URL. The identifier and path will be auto-generated, but can be overriden.
51 |
52 | 
53 |
54 | On submitting the repository creation form, the identifier and `url`
55 | (filesystem path) fields will be auto-generated (if not explicitly provided).
56 |
57 | For example, if you enter `https://github.com/dergachev/vagrant-vbox-snapshot` as the Clone URL,
58 | it will prefill the Identifier and filesystem path fields as follows:
59 | * Identifier: `vagrant-vbox-snapshot`
60 | * Path: `REDMINE_PLUGINS_PATH/redmine_git_remote/repos/github.com/dergachev/vagrant-vbox-snapshot`
61 |
62 | Once the remote URL is validated, the plugin creates an [empty clone](http://stackoverflow.com/questions/895819/whats-the-most-straightforward-way-to-clone-an-empty-bare-git-repository) at the specified path.
63 |
64 | This plugin hooks into the core `Repository.fetch_changesets` to automatically
65 | run `git fetch --all` on all GitRemote managed repositories as Redmine is about
66 | to pull in changesets from the local repos.
67 |
68 | To avoid slowing down the GUI, we recommend unchecking the "Fetch commits
69 | automatically" setting at
70 | [http://redmine-root/settings?tab=repositories](http://redmine-root/settings?tab=repositories)
71 | and relying on the following cron job as per [Redmine Wiki Instructions](http://www.redmine.org/projects/redmine/wiki/RedmineRepositories):
72 |
73 | ```
74 | */5 * * * * cd /home/redmine/redmine && ./bin/rails runner Repository.fetch_changesets -e production >> log/cron_rake.log 2>&1
75 | ```
76 |
77 | To trigger fetch manually, run this:
78 |
79 | ```
80 | cd /home/redmine/redmine && ./bin/rails runner "Repository.fetch_changesets" -e production
81 | ```
82 |
83 | Notes:
84 |
85 | * Tested on Redmine 3.4 and ruby 2.3
86 | * Currently alpha state, use at your own risk. Given possible security risks of shelling out,
87 | we recommend using this plugin only if all RedMine project admins are trusted users.
88 | * This plugin doesn't clean-up (delete) cloned repos from the file system when the record
89 | is deleted from Redmine.
90 | * Currently Redmine will crash if this plugin is uninstalled, as rails can't
91 | seem to handle model classes disappearing while db records reference them.
92 | This snippet should make the error go away:
93 |
94 | ```
95 | ./bin/rails runner 'ActiveRecord::Base.connection.execute("UPDATE repositories SET type=\"Repository::Git\" WHERE type = \"Repository::GitRemote\")' -e production
96 | ```
97 |
--------------------------------------------------------------------------------
/app/models/repository/git_remote.rb:
--------------------------------------------------------------------------------
1 | require 'redmine/scm/adapters/git_adapter'
2 | require 'pathname'
3 | require 'fileutils'
4 | # require 'open3'
5 | require_dependency 'redmine_git_remote/poor_mans_capture3'
6 |
7 | class Repository::GitRemote < Repository::Git
8 |
9 | before_validation :initialize_clone
10 |
11 | safe_attributes 'extra_info', :if => lambda {|repository, _user| repository.new_record?}
12 |
13 | # TODO: figure out how to do this safely (if at all)
14 | # before_deletion :rm_removed_repo
15 | # def rm_removed_repo
16 | # if Repository.find_all_by_url(repo.url).length <= 1
17 | # system "rm -Rf #{self.clone_path}"
18 | # end
19 | # end
20 |
21 | def extra_clone_url
22 | return nil unless extra_info
23 | extra_info["extra_clone_url"]
24 | end
25 |
26 | def clone_url
27 | self.extra_clone_url
28 | end
29 |
30 | def clone_path
31 | self.url
32 | end
33 |
34 | def clone_host
35 | p = parse(clone_url)
36 | return p[:host]
37 | end
38 |
39 | def clone_protocol_ssh?
40 | # Possible valid values (via http://git-scm.com/book/ch4-1.html):
41 | # ssh://user@server/project.git
42 | # user@server:project.git
43 | # server:project.git
44 | # For simplicity we just assume if it's not HTTP(S), then it's SSH.
45 | !clone_url.match(/^http/)
46 | end
47 |
48 | # Hook into Repository.fetch_changesets to also run 'git fetch'.
49 | def fetch_changesets
50 | # ensure we don't fetch twice during the same request
51 | return if @already_fetched
52 | @already_fetched = true
53 |
54 | puts "Calling fetch changesets on #{clone_path}"
55 | # runs git fetch
56 | self.fetch
57 | super
58 | end
59 |
60 | # Override default_branch to fetch, otherwise caching problems in
61 | # find_project_repository prevent Repository::Git#fetch_changesets from running.
62 | #
63 | # Ideally this would only be run for RepositoriesController#show.
64 | def default_branch
65 | if self.branches == [] && self.project.active? && Setting.autofetch_changesets?
66 | # git_adapter#branches caches @branches incorrectly, reset it
67 | scm.instance_variable_set :@branches, nil
68 | # NB: fetch_changesets is idemptotent during a given request, so OK to call it 2x
69 | self.fetch_changesets
70 | end
71 | super
72 | end
73 |
74 | # called in before_validate handler, sets form errors
75 | def initialize_clone
76 | # avoids crash in RepositoriesController#destroy
77 | return unless attributes["extra_info"]["extra_clone_url"]
78 |
79 | p = parse(attributes["extra_info"]["extra_clone_url"])
80 | self.identifier = p[:identifier] if identifier.empty?
81 |
82 | base_path = Setting.plugin_redmine_git_remote['git_remote_repo_clone_path']
83 | base_path = base_path + "/" unless base_path.end_with?("/")
84 |
85 | self.url = base_path + p[:path] if url.empty?
86 |
87 | err = ensure_possibly_empty_clone_exists
88 | errors.add :extra_clone_url, err if err
89 | end
90 |
91 | # equality check ignoring trailing whitespace and slashes
92 | def two_remotes_equal(a,b)
93 | a.chomp.gsub(/\/$/,'') == b.chomp.gsub(/\/$/,'')
94 | end
95 |
96 | def ensure_possibly_empty_clone_exists
97 | Repository::GitRemote.add_known_host(clone_host) if clone_protocol_ssh?
98 |
99 | unless system "git", "ls-remote", "-h", clone_url
100 | return "#{clone_url} is not a valid remote."
101 | end
102 |
103 | if Dir.exists? clone_path
104 | existing_repo_remote, status = RedmineGitRemote::PoorMansCapture3::capture2("git", "--git-dir", clone_path, "config", "--get", "remote.origin.url")
105 | return "Unable to run: git --git-dir #{clone_path} config --get remote.origin.url" unless status.success?
106 |
107 | unless two_remotes_equal(existing_repo_remote, clone_url)
108 | return "Directory '#{clone_path}' already exits, unmatching clone url: #{existing_repo_remote}"
109 | end
110 | else
111 | unless system "git", "init", "--bare", clone_path
112 | return "Unable to run: git init --bare #{clone_path}"
113 | end
114 |
115 | unless system "git", "--git-dir", clone_path, "remote", "add", "--mirror=fetch", "origin", clone_url
116 | return "Unable to run: git --git-dir #{clone_path} remote add --mirror=fetch origin #{clone_url}"
117 | end
118 | end
119 | end
120 |
121 | unloadable
122 | def self.scm_name
123 | 'GitRemote'
124 | end
125 |
126 | # TODO: first validate git URL and display error message
127 | def parse(url)
128 | url.strip!
129 |
130 | ret = {}
131 | # start with http://github.com/evolvingweb/git_remote or git@git.ewdev.ca:some/repo.git
132 | ret[:url] = url
133 |
134 | # NB: Starting lines with ".gsub" is a syntax error in ruby 1.8.
135 | # See http://stackoverflow.com/q/12906048/9621
136 | # path is github.com/evolvingweb/muhc-ci
137 | ret[:path] = url.gsub(/^.*:\/\//, ''). # Remove anything before ://
138 | gsub(/:/, '/'). # convert ":" to "/"
139 | gsub(/^.*@/, ''). # Remove anything before @
140 | gsub(/\.git$/, '') # Remove trailing .git
141 | ret[:host] = ret[:path].split('/').first
142 | #TODO: handle project uniqueness automatically or prompt
143 | ret[:identifier] = ret[:path].split('/').last.downcase.gsub(/[^a-z0-9_-]/,'-')
144 | return ret
145 | end
146 |
147 | def fetch
148 | puts "Fetching repo #{clone_path}"
149 | Repository::GitRemote.add_known_host(clone_host) if clone_protocol_ssh?
150 |
151 | err = ensure_possibly_empty_clone_exists
152 | Rails.logger.warn err if err
153 |
154 | # If dir exists and non-empty, should be safe to 'git fetch'
155 | unless system "git", "--git-dir", clone_path, "fetch", "--all"
156 | Rails.logger.warn "Unable to run 'git -c #{clone_path} fetch --all'"
157 | end
158 | end
159 |
160 | # Checks if host is in ~/.ssh/known_hosts, adds it if not present
161 | def self.add_known_host(host)
162 | # if not found...
163 | out, status = RedmineGitRemote::PoorMansCapture3::capture2("ssh-keygen", "-F", host)
164 | raise "Unable to run 'ssh-keygen -F #{host}" unless status
165 | unless out.match /found/
166 | # hack to work with 'docker exec' where HOME isn't set (or set to /)
167 | ssh_dir = (ENV['HOME'] == "/" || ENV['HOME'] == nil ? "/root" : ENV['HOME']) + "/.ssh"
168 | ssh_known_hosts = ssh_dir + "/known_hosts"
169 | begin
170 | FileUtils.mkdir_p ssh_dir
171 | rescue Exception => e
172 | raise "Unable to create directory #{ssh_dir}: " + e.to_s
173 | end
174 |
175 | puts "Adding #{host} to #{ssh_known_hosts}"
176 | out, status = RedmineGitRemote::PoorMansCapture3::capture2("ssh-keyscan", host)
177 | raise "Unable to run 'ssh-keyscan #{host}'" unless status
178 | Kernel::open(ssh_known_hosts, 'a') { |f| f.puts out}
179 | end
180 | end
181 | end
182 |
--------------------------------------------------------------------------------
/app/views/settings/_git_remote_settings.html.erb:
--------------------------------------------------------------------------------
1 |