├── .dockerignore ├── Dockerfile ├── LICENSE.md ├── Makefile ├── README.md ├── release-page ├── release-page.yml ├── release.html ├── requirements.txt └── static └── screenshot.png /.dockerignore: -------------------------------------------------------------------------------- 1 | Makefile 2 | README.md 3 | Dockerfile 4 | repos 5 | env 6 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.5 2 | MAINTAINER benjaminrk@gmail.com 3 | 4 | EXPOSE 8888 5 | 6 | RUN mkdir -p /srv/release-page 7 | WORKDIR /srv/release-page 8 | ADD requirements.txt /srv/release-page/requirements.txt 9 | RUN pip install -r requirements.txt 10 | ADD . /srv/release-page 11 | ADD release-page /srv/release-page/release-page 12 | USER nobody 13 | ENTRYPOINT ["python3", "release-page"] 14 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License: 2 | 3 | Copyright (c) 2015, Min RK 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and 6 | associated documentation files (the "Software"), to deal in the Software without restriction, 7 | including without limitation the rights to use, copy, modify, merge, publish, distribute, 8 | sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is 9 | furnished to do so, subject to the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT 14 | NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 15 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 16 | DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT 17 | OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 18 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | all: build run 2 | 3 | build: 4 | docker build -t release-page . 5 | 6 | run: 7 | docker rm -f release-page || true 8 | mkdir repos || true 9 | chmod 777 repos 10 | docker run --restart=always --env-file=./env -v $(PWD)/repos:/tmp/release-page -d -p 9009:8888 --name release-page release-page 11 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Release Page 2 | 3 | A simple service to render a webpage summarizing the release status of a 4 | collection of repos on GitHub. 5 | 6 | ![Report output](static/screenshot.png "report output") 7 | 8 | 9 | ## Run 10 | 11 | Install dependencies: 12 | 13 | pip3 install -r requirements.txt 14 | 15 | Select repositories by editing `release-page.yml` 16 | 17 | Run the service: 18 | 19 | python3 release-page 20 | 21 | Or run with docker (again, after editing release-page.yml): 22 | 23 | docker build -t release-page . 24 | docker run -d -p 80:8888 release-page 25 | 26 | 27 | ## GitHub and rate limiting 28 | 29 | To avoid being rate limited by GitHub, you can use a GitHub token. Copy the 30 | token to: 31 | 32 | GITHUB_API_TOKEN='xxxxxxxxxxxxxxxxxx' 33 | 34 | ## Customization 35 | 36 | Edit `release-page.yml` to set orgs and repos. 37 | 38 | 39 | License: MIT 40 | -------------------------------------------------------------------------------- /release-page: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """ 3 | release-page service 4 | 5 | edit repo list in release-page.yml to create your own page. 6 | """ 7 | from concurrent.futures import ThreadPoolExecutor 8 | from datetime import datetime, tzinfo, timedelta 9 | import os 10 | import re 11 | 12 | import git 13 | from github import Github as GitHub # H! 14 | import yaml 15 | 16 | # webapp 17 | import jinja2 18 | from tornado.httpserver import HTTPServer 19 | from tornado.gen import coroutine 20 | from tornado.log import app_log 21 | from tornado.ioloop import IOLoop, PeriodicCallback 22 | from tornado.web import RequestHandler, Application 23 | 24 | join = os.path.join 25 | 26 | cfg_file = 'release-page.yml' 27 | 28 | # regex for `git-describe` 29 | describe_pat = re.compile(r'(.*?)(\-(\d+)\-g([0-9a-f]+))?$') 30 | 31 | version_re = re.compile(r'(v|rel-)?\d+(\.\d+)+', re.IGNORECASE) 32 | 33 | def repo_name(repo): 34 | "/path/to/jupyter/foo/.git => jupyter/foo" 35 | return '/'.join(repo.git_dir.split('/')[-3:-1]) 36 | 37 | def ref_name(ref): 38 | "origin/master => master" 39 | return ref.name.split('/', 1)[1] 40 | 41 | class _tz_dict(dict): 42 | """A defaultdict containing datetime tzinfo objects from integer-second offsets.""" 43 | def __getitem__(self, offset): 44 | if offset not in self: 45 | aoffset = abs(offset) 46 | h = aoffset // 3600 47 | m = (aoffset % 3600) // 60 48 | # this is backwards because tzoffset is the reverse of tz representation 49 | s = '+' if offset <= 0 else '-' 50 | class TZ(tzinfo): 51 | def __repr__(self): 52 | return "TZ(%s%02i:%02i)" % (s, h, m) 53 | 54 | def utcoffset(self, dt): 55 | return timedelta(seconds=offset) 56 | 57 | def dst(self, dt): 58 | return timedelta(minutes=0) 59 | self[offset] = TZ() 60 | return super().__getitem__(offset) 61 | 62 | _tzinfos = _tz_dict() 63 | 64 | def utcnow(): 65 | return datetime.utcnow().replace(tzinfo=_tzinfos[0]) 66 | 67 | def commit_date(commit): 68 | """Return tz-aware datetime object from commit""" 69 | tz = _tzinfos[commit.author_tz_offset] 70 | return datetime.fromtimestamp(commit.authored_date, tz) 71 | 72 | tfmt = "%Y-%m-%d %H:%M:%S %z" 73 | 74 | def dirty(repo, ref, tag, commits): 75 | """Produce report dict for dirty branch""" 76 | td = commit_date(tag.commit) 77 | ref_date = commit_date(ref.commit) 78 | return { 79 | 'repo': repo_name(repo), 80 | 'ref': ref_name(ref), 81 | 'commits': commits, 82 | 'tag': tag.name, 83 | 'tag_date': td, 84 | 'ref_date': ref_date, 85 | 'days': (ref_date - td).days 86 | } 87 | 88 | def clean(repo, ref, tag): 89 | """Produce report dict for a branch that has been released.""" 90 | td = commit_date(tag.commit) 91 | return { 92 | 'repo': repo_name(repo), 93 | 'ref': ref_name(ref), 94 | 'commits': 0, 95 | 'tag': tag.name, 96 | 'tag_date': td, 97 | 'ref_date': td, 98 | 'days': 0, 99 | } 100 | 101 | def format_date(dt): 102 | """Simple short date format""" 103 | now = utcnow() 104 | today = now.date() 105 | date = dt.date() 106 | delta = now - dt 107 | day_delta = today - date 108 | days = day_delta.days 109 | seconds = delta.total_seconds() 110 | if dt > now: 111 | print("???Future: %s, %s???" % (dt, now)) 112 | return "In the future somehow" 113 | if dt + timedelta(minutes=9) >= now: 114 | return "just now" 115 | elif dt + timedelta(minutes=90) >= now: 116 | return "%i minutes ago" % (seconds // 60) 117 | elif date == today: 118 | return "today" 119 | elif date + timedelta(days=1) == today: 120 | return "yesterday" 121 | elif date + timedelta(days=14) >= today: 122 | return "%i days ago" % days 123 | elif date + timedelta(days=60) >= today: 124 | return "%i weeks ago" % (days // 7) 125 | elif date + timedelta(days=700) >= today: 126 | return "%i months ago" % (days // 30) 127 | else: 128 | return "%i years ago" % ((days + 150) // 365) 129 | 130 | 131 | def summarize_branch(repo, ref): 132 | """Summarize a branch of a repo""" 133 | app_log.info("Summarizing %s:%s" % (repo_name(repo), ref_name(ref))) 134 | try: 135 | desc = repo.git.describe(ref, '--tags', '--abbrev=99') 136 | except git.GitCommandError as e: 137 | # never released 138 | return None 139 | match = describe_pat.match(desc) 140 | name = ref.name.split('/', 1)[1] 141 | tagname = match.group(1) 142 | tag = repo.tags[tagname] 143 | scommits = match.group(3) 144 | ncommits = 0 if scommits is None else int(scommits) 145 | if ncommits <= 1: 146 | # assume 1 commit is back-to-dev version bump, treat it as clean 147 | return clean(repo, ref, tag) 148 | else: 149 | return dirty(repo, ref, tag, ncommits) 150 | 151 | 152 | def summary(project, workdir): 153 | """Get the summary of a project on GitHub 154 | 155 | Parameters 156 | ---------- 157 | 158 | project: 'org/repo' string 159 | 160 | Returns 161 | ------- 162 | 163 | list of dicts summarizing the state of each branch. 164 | """ 165 | url = 'https://github.com/%s' % project 166 | path = join(workdir, project) 167 | if not os.path.exists(path): 168 | app_log.info("Cloning %s to %s" % (url, path)) 169 | r = git.Repo.clone_from(url, path) 170 | else: 171 | r = git.Repo(path) 172 | app_log.info("Updating %s" % (path)) 173 | r.remote().fetch() 174 | infos = [] 175 | # prune deleted remote branches 176 | r.git.remote('prune', *r.remotes) 177 | for ref in r.remote().refs: 178 | if ref.name.endswith(('/HEAD', '/gh-pages')): 179 | continue 180 | info = summarize_branch(r, ref) 181 | if not info: 182 | app_log.warn("No info for %s:%s", repo_name(r), ref_name(ref)) 183 | else: 184 | infos.append(info) 185 | infos = sorted(infos, key=lambda info: info['ref_date'], reverse=True) 186 | if not infos: 187 | app_log.warning("No releases for %s", project) 188 | return infos 189 | 190 | 191 | class ReleaseChecker(object): 192 | def __init__(self, *, workdir='/tmp/release-page', orgs=None, exclude_repos=None, repos=None): 193 | self.workdir = workdir 194 | self.orgs = orgs or [] 195 | self.exclude_repos = exclude_repos or [] 196 | self.repos = repos or [] 197 | 198 | self.gh = GitHub(os.getenv('GITHUB_API_TOKEN')) 199 | self.executor = ThreadPoolExecutor(2) 200 | self.data = { 201 | 'projects': {}, 202 | 'date': utcnow(), 203 | } 204 | 205 | def _update_repo_list(self): 206 | for org in map(self.gh.get_organization, self.orgs): 207 | app_log.info("Updating repo list for %s", org.login) 208 | for repo in org.get_repos(): 209 | if repo.full_name in self.repos + self.exclude_repos: 210 | app_log.debug("Skipping excluded repo %s", repo.full_name) 211 | continue 212 | else: 213 | app_log.debug("Checking for tags in %s", repo.full_name) 214 | # check for tags 215 | for tag in repo.get_tags(): 216 | if version_re.match(tag.name): 217 | app_log.info("Adding %s to repo list", repo.full_name) 218 | self.repos.append(repo.full_name) 219 | break 220 | else: 221 | app_log.info("No tags in %s", repo.full_name) 222 | self.exclude_repos.append(repo.full_name) 223 | 224 | @coroutine 225 | def update_repo_list(self): 226 | app_log.info("Updating repo list (rate limit: %i/%i)", *self.gh.rate_limiting) 227 | yield self.executor.submit(self._update_repo_list) 228 | app_log.info("Repo list updated (rate limit %i/%i)", *self.gh.rate_limiting) 229 | 230 | def _update(self): 231 | for name in list(self.repos): 232 | self.data['projects'][name] = summary(name, self.workdir) 233 | 234 | @coroutine 235 | def update(self): 236 | app_log.info("Updating data") 237 | yield self.executor.submit(self._update) 238 | app_log.info("Data updated") 239 | self.data['date'] = utcnow() 240 | 241 | 242 | class RenderHandler(RequestHandler): 243 | """Handler for rendering summary of form info as a page.""" 244 | def initialize(self, data, env): 245 | self.data = data 246 | self.env = env 247 | 248 | def get(self): 249 | template = self.env.get_template('release.html') 250 | def sort_key(item): 251 | if not item[1]: # no releases 252 | return -1 253 | return max(info['commits'] for info in item[1]) 254 | 255 | repos = sorted( 256 | [ (name, branches) for name, branches in self.data['projects'].items() ], 257 | key=sort_key, 258 | reverse=True, 259 | ) 260 | html = template.render(repos=repos, date=format_date(data['date'])) 261 | self.finish(html) 262 | 263 | if __name__ == '__main__': 264 | from tornado.options import define, options, parse_command_line 265 | define("port", default=8888, help="run on the given port", type=int) 266 | define("interval", default=3600, help="interval (seconds) to refresh", type=int) 267 | define("github-interval", default=24 * 3600, help="interval (seconds) to refresh repo list from GitHub", type=int) 268 | define("workdir", default='/tmp/release-page', help="path to clone repos", type=str) 269 | 270 | parse_command_line() 271 | 272 | with open(cfg_file) as f: 273 | cfg = yaml.load(f) 274 | 275 | checker = ReleaseChecker(workdir=options.workdir, **cfg) 276 | 277 | loader = jinja2.FileSystemLoader('.') 278 | env = jinja2.Environment(loader=loader, autoescape=True) 279 | env.filters['format_date'] = format_date 280 | 281 | data = checker.data 282 | 283 | loop = IOLoop.instance() 284 | # schedule initial data load 285 | @coroutine 286 | def first_load(): 287 | # concurrent update & repo-list load, 288 | # so we can start cloning while we list repos on orgs 289 | futures = [ checker.update(), checker.update_repo_list() ] 290 | for f in futures: 291 | yield f 292 | # update repos one more time after loading the list 293 | yield checker.update() 294 | loop.add_callback(first_load) 295 | # and periodic updates 296 | PeriodicCallback(checker.update_repo_list, options.github_interval * 1e3).start() 297 | PeriodicCallback(checker.update, options.interval * 1e3).start() 298 | app = Application([ 299 | ('/', RenderHandler, dict(data=data, env=env)) 300 | ]) 301 | server = HTTPServer(app) 302 | server.listen(options.port) 303 | app_log.info("Listening on :%i", options.port) 304 | loop.start() 305 | -------------------------------------------------------------------------------- /release-page.yml: -------------------------------------------------------------------------------- 1 | orgs: 2 | - ipython 3 | - jupyter 4 | - jupyterhub 5 | - jupyterlab 6 | repos: 7 | - zeromq/pyzmq 8 | exclude_repos: 9 | # exclude repos that have tags that aren't releases 10 | - ipython/ipython-py3k 11 | - jupyter/jupyter-drive 12 | -------------------------------------------------------------------------------- /release.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 13 | 14 | 15 |
16 |
17 |

Release Page

18 |

19 | A quick summary of how much unreleased code is in these repos 20 |

21 |
22 | {% for repo, branches in repos %} 23 |
24 |

{{repo}}

25 |
26 |
27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | {% for info in branches %} 39 | {# exclude up-to-date branches other than master #} 40 | {% if info['commits'] or info['ref'] == 'master' %} 41 | 42 | 45 | 48 | 49 | 52 | 53 | 54 | {% endif %} 55 | {% endfor %} 56 | 57 |
branchreleasedateunreleased commitslast commit
43 | {{info['ref']}} 44 | 46 | {{info['tag']}} 47 | {{info['tag_date'] | format_date}} 50 | {{info['commits']}} 51 | {{info['ref_date'] | format_date}}
58 |
59 | {% endfor %} 60 |
61 |

62 | Last updated: {{date}} 63 |

64 |

65 | Release Page is © Min RK and MIT Licensed 66 |

67 |
68 |
69 | 70 | 71 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | GitPython 2 | jinja2 3 | tornado 4 | PyYAML 5 | PyGitHub 6 | futures; python_version < "3.4" -------------------------------------------------------------------------------- /static/screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/minrk/release-page/34e5df9278cc9b08ac780020d446d1440835e971/static/screenshot.png --------------------------------------------------------------------------------