├── Makefile
├── README.md
├── hooknook.py
├── requirements.txt
└── templates
├── base.html
├── login.html
└── logs.html
/Makefile:
--------------------------------------------------------------------------------
1 | .PHONY: deploy
2 |
3 | deploy:
4 | echo This is a test.
5 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Look! Hooknook.
2 |
3 | Hooknook is a preposterously simple tool for deploying code from GitHub. You push to GitHub, Hooknook gets notified via a webhook, and your code does whatever it wants on the server.
4 |
5 | You can use it to run "your own GitHub Pages," where every time you push a site's Jekyll source code to GitHub, it automatically gets built and uploaded to a server.
6 |
7 | ## How to Hooknook
8 |
9 | Install it like this:
10 |
11 | $ pip3 install -r requirements.txt
12 |
13 | Run it like this:
14 |
15 | $ python3 hooknook.py -u USERNAME
16 |
17 | It's a good idea to specify the GitHub users that are allowed to use your server. Otherwise, anyone can look in the Hooknook. The `-u` option can be specified multiple times and also works for organizations. If you don't specify any users, anyone will be allowed. (We filter requests by IP address, so you can't get burned by Hooknook crooks pretending to be GitHub.)
18 |
19 | Then, set up a GitHub web hook to point at your server: something like `http://example.com:5000`.
20 |
21 | Every time you push to your repository, Hooknook will update your repository in `~/.hooknook/repo` and run `make deploy`. Use a Makefile to describe what to do.
22 |
23 | ## .hook.yaml
24 |
25 | If you want to run a different command (i.e., something other than `make deploy`), add a file called `.hook.yaml` to your repo. This is a [YAML][] file that (currently) has only one key: `deploy`, which is a command that tells Hooknook how to cook.
26 |
27 | [YAML]: https://en.wikipedia.org/wiki/YAML
28 |
29 | ## Logs
30 |
31 | Every hook is logged to a timestamped file in `~/.hooknook/log`. Look in the Hooknook logbook if you think something is going wrong.
32 |
33 | ## Web Interface
34 |
35 | Hooknook can show you its logs through a Web interface. To keep this secure, it authenticates users with their GitHub credentials.
36 |
37 | To make this work, you need to register your Hooknook installation as a GitHub application. Here's how:
38 |
39 | 1. Go to your GitHub settings and [make a new application][gh-app-new].
40 | 2. Fill in the callback URL using `/auth` on your server: like `http://example.com:5000/auth`. Fill in reasonable values for the other fields.
41 | 3. Get your "Client ID" and "Client Secret" for the new application from GitHub. Start Hooknook with the `-g ID:SECRET` option. This enables authentication.
42 | 4. Head your Hooknook server's home page (e.g., `http://example.com:5000`) and click the login button. Sign in as one of the whitelisted users.
43 |
44 | You'll now see a list of the most recent logs on the server.
45 |
46 | Currently, all whitelisted users can view all logs. In the future, users will only be able to see logs for repositories they have access to.
47 |
48 | [gh-app-new]: https://github.com/settings/applications/new
49 |
50 | ## Configuration
51 |
52 | You can configure Hooknook with command-line flags or a configuration file. Hooknook looks for a (Python) file called `hooknook.cfg` by default, and you can also supply a `HOOKNOOK_CFG` environment variable to point to another path if you like.
53 |
54 | The configuration options are:
55 |
56 | * `USERS` or `-u USER`: A list of whitelisted GitHub user/organization names.
57 | * `GITHUB_ID` and `GITHUB_SECRET` or `-g ID:SECRET`: GitHub API credentials.
58 | * `SECRET_KEY`: A secret string for signing trusted data.
59 | * `PROXIED`: A Boolean indicating whether the application should trust the `X-Forwarded-For` header when determining whether a request came from GitHub.
60 |
61 | If you use a config file instead of command-line flags, you can run Hooknook on a proper HTTP server (probably a good idea!). For example, run it with [Gunicorn][] like so:
62 |
63 | $ gunicorn --workers 4 --bind 0.0.0.0:5000 hooknook:app
64 |
65 | [Gunicorn]: http://gunicorn.org/
66 |
67 | ### Running via Launchctl on OSX
68 |
69 | Or how to avoid getting rooked running hooknook on your Mac-book (actually, you should probably be running this on a desktop machine, but those don't rhyme).
70 |
71 | There are a couple caveats when trying to configure hooknook to run automatically in daemon mode on OSX. Python 3 and Click, which is used by hooknook for parsing configuration options, [do not always play nicely together](http://click.pocoo.org/3/python3/). When running via a launchd script, you need to be sure to set the locale correctly in the environment.
72 |
73 | Assuming you've installed python3 the way everyone does (via [homebrew](http://brew.sh)), and that you've cloned hooknook into `/opt/hooknook`, and assuming your user is named "peregrintook" (locally and on Github), your launchd configuration, `/Library/LaunchDaemons/edu.uw.hooknook.plist`, might look like:
74 |
75 |
76 |
77 |
78 | Labeledu.uw.hooknook
79 | UserNameperegrintook
80 | KeepAlive
81 | RunAtLoad
82 | EnvironmentVariables
83 |
84 | LC_ALLen_US.UTF-8
85 | LANGen_US.UTF-8
86 |
87 | ProgramArguments
88 |
89 | /usr/local/bin/python3
90 | hooknook.py
91 | -u
92 | peregrintook
93 |
94 | WorkingDirectory/opt/hooknook
95 | StandardOutPathhooknook.log
96 | StandardErrorPathhooknook.log
97 |
98 |
99 |
100 | This daemon can be started with:
101 |
102 | sudo launchctl load /Library/LaunchDaemons/edu.uw.hooknook.plist
103 |
104 |
--------------------------------------------------------------------------------
/hooknook.py:
--------------------------------------------------------------------------------
1 | import flask
2 | from flask import request
3 | import threading
4 | import queue
5 | import click
6 | import os
7 | import subprocess
8 | import traceback
9 | import yaml
10 | import datetime
11 | import netaddr
12 | import requests
13 | import string
14 | import random
15 | import urllib.parse
16 | import logging
17 |
18 | app = flask.Flask(__name__)
19 | app.config.update(
20 | DATA_DIR=os.path.expanduser(os.path.join('~', '.hooknook')),
21 | CONFIG_FILENAME='.hook.yaml',
22 | CONFIG_DEFAULT={
23 | 'deploy': 'make deploy',
24 | },
25 | USERS=(),
26 | PRIVATE_URL_FORMAT='git@github.com:{user}/{repo}.git',
27 | PUBLIC_URL_FORMAT='https://github.com/{user}/{repo}.git',
28 | GITHUB_ID=None,
29 | GITHUB_SECRET=None,
30 | GITHUB_HOOK_SUBNETS=None,
31 | PROXIED=False,
32 | )
33 | app.config.from_pyfile('hooknook.cfg', silent=True)
34 | app.config.from_envvar('HOOKNOOK_CFG', silent=True)
35 |
36 | FILENAME_FORMAT = '{user}#{repo}'
37 | TIMESTAMP_FORMAT = '%Y-%m-%d-%H-%M-%S-%f'
38 |
39 |
40 | def load_config(repo_dir):
41 | """Load the repository's configuration as a dictionary. Defaults are
42 | used for missing keys.
43 | """
44 | # Provide defaults.
45 | config = dict(app.config['CONFIG_DEFAULT'])
46 |
47 | # Load the configuration file, if any.
48 | config_fn = os.path.join(repo_dir, app.config['CONFIG_FILENAME'])
49 | if os.path.exists(config_fn):
50 | with open(config_fn) as f:
51 | overlay = yaml.load(f)
52 | config.update(overlay)
53 |
54 | return config
55 |
56 |
57 | def timestamp():
58 | """Get a string indicating the current time.
59 | """
60 | now = datetime.datetime.now()
61 | return now.strftime(TIMESTAMP_FORMAT)
62 |
63 |
64 | def shell(command, logfile, cwd=None, shell=False):
65 | """Execute a command (via a shell or directly) and log the stdout
66 | and stderr streams to a file-like object.
67 | """
68 | if shell:
69 | logline = command
70 | else:
71 | logline = ' '.join(command)
72 | logfile.write('$ {}\n'.format(logline))
73 | logfile.flush()
74 |
75 | subprocess.check_call(
76 | command,
77 | cwd=cwd,
78 | stdout=logfile,
79 | stderr=subprocess.STDOUT,
80 | shell=shell,
81 | )
82 | logfile.flush()
83 |
84 |
85 | def random_string(length=20, chars=(string.ascii_letters + string.digits)):
86 | return ''.join(random.choice(chars) for i in range(length))
87 |
88 |
89 | def github_get(path, token=None, base='https://api.github.com'):
90 | """Make a request to the GitHub API."""
91 | token = token or flask.session['github_token']
92 | url = '{}/{}'.format(base, path)
93 | return requests.get(url, params={
94 | 'access_token': token,
95 | })
96 |
97 |
98 | def update_repo(repo, url, log):
99 | """Clone or pull the repository. Return the updated repository
100 | directory.
101 | """
102 | # Create the parent directory for repositories.
103 | parent = os.path.join(app.config['DATA_DIR'], 'repo')
104 | if not os.path.exists(parent):
105 | os.makedirs(parent)
106 |
107 | # Clone the repository or update it.
108 | repo_dir = os.path.join(parent, repo)
109 | # FIXME log
110 | if os.path.exists(repo_dir):
111 | shell(['git', 'fetch'], log, repo_dir)
112 | shell(['git', 'reset', '--hard', 'origin/master'], log, repo_dir)
113 | else:
114 | shell(['git', 'clone', url, repo_dir], log)
115 |
116 | return repo_dir
117 |
118 |
119 | def run_build(repo_dir, log):
120 | """Run the build in the repository direction.
121 | """
122 | # Get the configuration.
123 | config = load_config(repo_dir)
124 |
125 | # Run the build.
126 | try:
127 | shell(config['deploy'], log, repo_dir, True)
128 | except subprocess.CalledProcessError as exc:
129 | app.logger.error(
130 | 'Deploy exited with status {}'.format(exc.returncode)
131 | )
132 |
133 |
134 | def log_dir():
135 | """The directory for storing logs."""
136 | return os.path.join(app.config['DATA_DIR'], 'log')
137 |
138 |
139 | def open_log(repo):
140 | """Open a log file for a build and return the open file.
141 |
142 | `repo` is the (filename-safe) name of the repository.
143 | """
144 | # Create the parent directory for the log files.
145 | parent = log_dir()
146 | if not os.path.exists(parent):
147 | os.makedirs(parent)
148 |
149 | # Get a log file for this build.
150 | ts = timestamp()
151 | log_fn = os.path.join(parent, '{}#{}.log'.format(repo, ts))
152 |
153 | return open(log_fn, 'w')
154 |
155 |
156 | def client_addr(request):
157 | """Get the remote address for a request.
158 |
159 | If the application has proxying enabled, this might come from the
160 | X-Forwarded-For header. Otherwise, it is the host on the other end
161 | of the connection for the request.
162 | """
163 | if app.config['PROXIED']:
164 | return request.access_route[0]
165 | else:
166 | return request.remote_addr
167 |
168 |
169 | class Worker(threading.Thread):
170 | """Thread used for invoking builds asynchronously.
171 | """
172 | def __init__(self):
173 | super(Worker, self).__init__()
174 | self.daemon = True
175 | self.queue = queue.Queue()
176 |
177 | def run(self):
178 | """Wait for jobs and execute them with `handle`.
179 | """
180 | while True:
181 | try:
182 | self.handle(*self.queue.get())
183 | except:
184 | app.logger.error(
185 | 'Worker exception:\n' + traceback.format_exc()
186 | )
187 |
188 | def handle(self, repo, url):
189 | """Execute a build.
190 |
191 | `repo` is the (filename-safe) repository name. `url` is the git
192 | clone URL for the repo.
193 | """
194 | app.logger.info('Building {}'.format(repo))
195 |
196 | with open_log(repo) as log:
197 | repo_dir = update_repo(repo, url, log)
198 | run_build(repo_dir, log)
199 |
200 | def send(self, *args):
201 | """Add a job to the queue.
202 | """
203 | self.queue.put(args)
204 |
205 |
206 | @app.before_first_request
207 | def app_setup():
208 | """Ensure that the application has some shared global attributes set
209 | up:
210 |
211 | - `worker` is a Worker thread
212 | - `github_networks` is the list of valid origin IPNetworks
213 | """
214 | # Set up logging.
215 | if not app.debug:
216 | # (Flask turns on logging itself in debug mode.)
217 | app.logger.addHandler(logging.StreamHandler())
218 | app.logger.setLevel(logging.INFO)
219 |
220 | # Create a worker thread.
221 | if not hasattr(app, 'worker'):
222 | app.worker = Worker()
223 | app.worker.start()
224 |
225 | # Get the valid GitHub hook server IP ranges. This can either be fixed in
226 | # the configuration file or loaded automatically from the server.
227 | if not hasattr(app, 'github_networks'):
228 | if app.config['GITHUB_HOOK_SUBNETS']:
229 | cidrs = app.config['GITHUB_HOOK_SUBNETS']
230 | else:
231 | # Load the from the GitHub public API.
232 | meta = requests.get('https://api.github.com/meta').json()
233 | cidrs = meta['hooks']
234 |
235 | app.github_networks = [netaddr.IPNetwork(cidr) for cidr in cidrs]
236 | app.logger.info(
237 | 'Loaded GitHub networks: %s',
238 | [str(n) for n in app.github_networks]
239 | )
240 |
241 |
242 | @app.route('/', methods=['POST'])
243 | @app.route('/hook', methods=['POST']) # Backwards-compatibility.
244 | def hook():
245 | """The web hook endpoint. This is the URL that GitHub uses to send
246 | hooks.
247 | """
248 | # Ensure that the request is from a GitHub server.
249 | addr = client_addr(request)
250 | for network in app.github_networks:
251 | if addr in network:
252 | break
253 | else:
254 | app.logger.info('Hook request from disallowed host %s', addr)
255 | return flask.jsonify(status='you != GitHub'), 403
256 |
257 | # Dispatch based on event type.
258 | event_type = request.headers.get('X-GitHub-Event')
259 | if not event_type:
260 | app.logger.info('Received a non-hook request')
261 | return flask.jsonify(status='not a hook'), 403
262 | elif event_type == 'ping':
263 | return flask.jsonify(status='pong')
264 | elif event_type == 'push':
265 | payload = request.get_json()
266 | repo = payload['repository']
267 |
268 | # If a user whitelist is specified, validate the owner.
269 | owner = repo['owner']['name']
270 | name = repo['name']
271 | allowed_users = app.config['USERS']
272 | if allowed_users and owner not in allowed_users:
273 | return flask.jsonify(status='user not allowed', user=owner), 403
274 |
275 | if repo['private']:
276 | url_format = app.config['PRIVATE_URL_FORMAT']
277 | else:
278 | url_format = app.config['PUBLIC_URL_FORMAT']
279 | app.worker.send(
280 | FILENAME_FORMAT.format(user=owner, repo=name),
281 | url_format.format(user=owner, repo=name),
282 | )
283 | return flask.jsonify(status='handled'), 202
284 | else:
285 | return flask.jsonify(status='unhandled event', event=event_type), 501
286 |
287 |
288 | @app.route('/login')
289 | def login():
290 | """Redirect to GitHub for authentication."""
291 | if not app.config['GITHUB_ID']:
292 | return 'GitHub API disabled', 501
293 | auth_state = flask.session['auth_state'] = random_string()
294 | auth_url = '{}?{}'.format(
295 | 'https://github.com/login/oauth/authorize',
296 | urllib.parse.urlencode({
297 | 'client_id': app.config['GITHUB_ID'],
298 | 'scope': 'read:org,write:repo_hook',
299 | 'state': auth_state,
300 | }),
301 | )
302 | app.logger.info(
303 | 'Authorizing with GitHub at {}'.format(auth_url),
304 | )
305 | return flask.redirect(auth_url)
306 |
307 |
308 | @app.route('/auth')
309 | def auth():
310 | """Receive a callback from GitHub's authentication."""
311 | if not app.config['GITHUB_ID']:
312 | return 'GitHub API disabled', 501
313 |
314 | # Get the code from the callback.
315 | if flask.session.get('auth_state') != request.args['state']:
316 | app.logger.error(
317 | 'Invalid state from GitHub auth (possible CSRF)'
318 | )
319 | return 'invalid request state', 403
320 | code = request.args['code']
321 |
322 | # Turn this into an access token.
323 | resp = requests.post(
324 | 'https://github.com/login/oauth/access_token',
325 | data={
326 | 'client_id': app.config['GITHUB_ID'],
327 | 'client_secret': app.config['GITHUB_SECRET'],
328 | 'code': code,
329 | }
330 | )
331 | token = urllib.parse.parse_qs(resp.text)['access_token'][0]
332 | app.logger.info(
333 | 'Authorized token with GitHub: {}'.format(token),
334 | )
335 |
336 | # Check that the user is on the whitelist.
337 | user_data = github_get('user', token=token).json()
338 | username = user_data['login']
339 | if username not in app.config['USERS']:
340 | # Check if the user is in a whitelisted org.
341 | # TODO: handle pagination if the user is in > 30 orgs
342 | org_data = github_get('user/orgs', token=token).json()
343 | for org in org_data:
344 | if org['login'] in app.config['USERS']:
345 | break
346 | else:
347 | app.logger.warn(
348 | 'GitHub user not allowed: {}'.format(username)
349 | )
350 | return 'you are not allowed', 403
351 |
352 | # Mark the user as logged in.
353 | flask.session['github_token'] = token
354 |
355 | return flask.redirect('/')
356 |
357 |
358 | @app.route('/')
359 | def home():
360 | token = flask.session.get('github_token')
361 | if token:
362 | # Parse each filename.
363 | logs = []
364 | parent = log_dir()
365 | if os.path.exists(parent):
366 | for fn in os.listdir(parent):
367 | if fn.endswith('.log'):
368 | name, _ = fn.rsplit('.', 1)
369 | try:
370 | user, repo, stamp = name.split('#')
371 | dt = datetime.datetime.strptime(stamp,
372 | TIMESTAMP_FORMAT)
373 | except ValueError:
374 | continue
375 | logs.append({
376 | 'user': user,
377 | 'repo': repo,
378 | 'name': '{}/{}'.format(user, repo),
379 | 'dt': dt,
380 | 'file': fn,
381 | })
382 |
383 | # Sort by time.
384 | logs.sort(key=lambda l: l['dt'], reverse=True)
385 |
386 | return flask.render_template('logs.html', logs=logs[:10])
387 |
388 | else:
389 | return flask.render_template('login.html')
390 |
391 |
392 | @app.route('/log/')
393 | def show_log(name):
394 | token = flask.session.get('github_token')
395 | if not token:
396 | return 'no can do', 403
397 | log_name = os.path.basename(name) # Avoid any directories in path.
398 | log_path = os.path.join(log_dir(), log_name)
399 | if not os.path.exists(log_path):
400 | return 'no such log', 404
401 | return flask.send_file(log_path, mimetype='text/plain')
402 |
403 |
404 | @click.command()
405 | @click.option('--host', '-h', default='0.0.0.0', help='server hostname')
406 | @click.option('--port', '-p', default=5000, help='server port')
407 | @click.option('--debug', '-d', is_flag=True, help='run in debug mode')
408 | @click.option('--user', '-u', multiple=True, help='allowed GitHub users')
409 | @click.option('--github', '-g', help='GitHub client id:secret')
410 | @click.option('--secret', '-s', help='application secret key')
411 | def run(host, port, debug, user, github, secret):
412 | if debug:
413 | app.config['DEBUG'] = debug
414 | if user:
415 | app.config['USERS'] = user
416 | if github and ':' in github:
417 | app.config['GITHUB_ID'], app.config['GITHUB_SECRET'] = \
418 | github.split(':', 1)
419 | if secret:
420 | app.config['SECRET_KEY'] = secret
421 | elif not app.config.get('SECRET_KEY'):
422 | app.config['SECRET_KEY'] = random_string()
423 | app.run(host=host, port=port)
424 |
425 |
426 | if __name__ == '__main__':
427 | run(auto_envvar_prefix='HOOKNOOK')
428 |
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | flask
2 | click
3 | pyyaml
4 | netaddr
5 | requests
6 |
--------------------------------------------------------------------------------
/templates/base.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 | Hooknook
5 |
6 |
7 |
8 |