├── README.md └── pin-github-stars.py /README.md: -------------------------------------------------------------------------------- 1 | # README 2 | 3 | This script adds starred GitHub repositories as bookmarks in Pinboard. All such bookmarks will be tagged with github-star and the project's language (if applicable). In addition, you may set a prefix for the language tag with the `--lang-tag-prefix` option. The script will attempt to avoid overwriting bookmarks in Pinboard by checking starred repos against the most recent bookmark with the `github-star` tag. 4 | 5 | The script requires a [GitHub personal API token](https://github.com/blog/1509-personal-api-tokens) (see note below) and a [Pinboard API token](https://pinboard.in/settings/password), both of which may be stored in files named `~/.github_api_token` and `~/.pinboard_api_token` respectively instead of passing them as command-line options. Dependencies are best installed with `pip` and `virtualenv`, though you're free to live on the edge and do things like `sudo easy_install requests` if you wish. Just don't say you weren't warned... :smirk: 6 | 7 | Run `python pin-github-stars.py -h` for command-line help, which is an essentially abbreviated version of this README. 8 | 9 | **Note:** Since your stars list is public information, a token for GitHub's API is not strictly necessary; however, it avoids issues with rate-limiting and unauthenticated API use. If you'd prefer not to create a token specifically for the script, feel free to fork it and remove the offending bits. 10 | 11 | ### Example usage 12 | 13 | Passing everything on the command line would look like this: 14 | 15 | ``` 16 | $ python pin-github-stars.py -g 1234567890abcdef1234567890abcdef123456 -p pbuser:1234567890abcdef1234 -u github_user --lang-tag-prefix lang: 17 | ``` 18 | 19 | 20 | ### Dependencies 21 | 22 | - [requests](http://docs.python-requests.org/en/latest/) >= 1.1.0 23 | -------------------------------------------------------------------------------- /pin-github-stars.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2014-2015 Bryan Dyck 2 | # 3 | # Permission is hereby granted, free of charge, to any person obtaining a copy 4 | # of this software and associated documentation files (the "Software"), to deal 5 | # in the Software without restriction, including without limitation the rights 6 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | # copies of the Software, and to permit persons to whom the Software is 8 | # furnished to do so, subject to the following conditions: 9 | # 10 | # The above copyright notice and this permission notice shall be included in all 11 | # copies or substantial portions of the Software. 12 | # 13 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 19 | # SOFTWARE. 20 | 21 | """ 22 | Adds starred GitHub repositories as bookmarks in Pinboard. All such bookmarks 23 | will be tagged with 'github-star' and the project's language (if applicable). 24 | The script will attempt to avoid overwriting bookmarks in Pinboard by checking 25 | starred repos against the most recent bookmark with the 'github-star' tag. 26 | 27 | Requires a GitHub OAuth token and a Pinboard API token, both of which may be 28 | stored in ~/.github_oauth_token and ~/.pinboard_api_token respectively 29 | instead of passing them as command-line options. 30 | 31 | Dependencies: 32 | - Requests (http://docs.python-requests.org/en/latest/) 33 | """ 34 | 35 | import os 36 | import re 37 | import sys 38 | 39 | from argparse import ArgumentParser, RawDescriptionHelpFormatter 40 | from time import sleep 41 | 42 | import requests 43 | 44 | 45 | GH_API = 'https://api.github.com' 46 | PB_API = 'https://api.pinboard.in/v1' 47 | SLEEP_TIME = 3 48 | MAX_RETRIES = 10 49 | 50 | 51 | def add_bookmark(gh_data, pb_api_token, lang_tag_prefix): 52 | """ 53 | Adds a bookmark to Pinboard. 54 | 55 | GitHub -> Pinboard field mapping: 56 | 57 | full_name -> description 58 | html_url -> url 59 | description -> extended 60 | language -> tags + lang_tag_prefix 61 | """ 62 | description = gh_data['description'] 63 | if gh_data['homepage']: 64 | description += '\n\nProject homepage: {}'.format(gh_data['homepage']) 65 | tags = 'github-star' 66 | if gh_data['language']: 67 | tags = ' '.join((tags, lang_tag_prefix + gh_data['language'].lower())) 68 | params = {'description': gh_data['full_name'], 69 | 'url': gh_data['html_url'], 70 | 'extended': description, 71 | 'tags': tags, 72 | 'replace': 'no', 73 | 'auth_token': pb_api_token, 74 | 'format': 'json'} 75 | r = requests.get('{}/posts/add'.format(PB_API), 76 | params=params, 77 | headers={'user-agent': 'pin-github-stars.py'}) 78 | return r.json(), r.status_code 79 | 80 | 81 | def get_most_recent_bookmark(pb_api_token): 82 | """ 83 | Returns the most recent bookmark with the tag 'github-star', if any. 84 | """ 85 | params = {'tag': 'github-star', 86 | 'count': 1, 87 | 'auth_token': pb_api_token, 88 | 'format': 'json'} 89 | r = requests.get('{}/posts/recent'.format(PB_API), 90 | params=params) 91 | return (r.json()['posts'] or None) 92 | 93 | 94 | def get_github_stars(headers, sort_dir): 95 | """ 96 | A generator that returns successive pages of a user's list of starred repos. 97 | """ 98 | page = 1 99 | last = 1 100 | params = {'direction': sort_dir} 101 | while page <= last: 102 | params['page'] = page 103 | r = requests.get('{}/user/starred'.format(GH_API), 104 | params=params, 105 | headers=headers) 106 | yield r.json(), r.status_code 107 | if 'link' in r.headers: 108 | m = re.search(r'page=(\d+)>; rel="next",.*page=(\d+)>; rel="last"', r.headers['link']) 109 | if m is None: # Lazy way to see if we hit the last page, which only has 'first' and 'prev' links 110 | break 111 | page, last = m.groups() 112 | 113 | 114 | def filter_stars(stars, full_name): 115 | """ 116 | Searches the list of starred repos and returns a truncated list if the 117 | specified repo is found in the list. 118 | """ 119 | for i in xrange(len(stars)): 120 | if stars[i]['full_name'].lower() == full_name.lower(): 121 | return stars[:i], True 122 | return stars, False 123 | 124 | 125 | def exit_with_error(msg): 126 | print >>sys.stderr, '[ERROR] {}'.format(msg) 127 | sys.exit(1) 128 | 129 | 130 | def load_token(filename): 131 | token = None 132 | if os.path.exists(os.path.expanduser(filename)): 133 | with open(os.path.expanduser(filename)) as f: 134 | token = f.readline().strip() 135 | return token 136 | 137 | 138 | def main(): 139 | arg_parser = ArgumentParser(description=__doc__, 140 | formatter_class=RawDescriptionHelpFormatter) 141 | arg_parser.add_argument('-g', 142 | '--github-token', 143 | help='GitHub OAuth API token') 144 | arg_parser.add_argument('-p', 145 | '--pinboard-token', 146 | help='Pinboard API token') 147 | arg_parser.add_argument('-u', 148 | '--github-user', 149 | required=True, 150 | help='GitHub username (sent as user-agent for API requests)') 151 | arg_parser.add_argument('--lang-tag-prefix', 152 | default='', 153 | help='Set a prefix for the language tag. (Such as "lang:")') 154 | args = arg_parser.parse_args() 155 | 156 | gh_api_token = args.github_token or load_token('~/.github_api_token') 157 | pb_api_token = args.pinboard_token or load_token('~/.pinboard_api_token') 158 | 159 | if gh_api_token is None: 160 | exit_with_error('Could not load GitHub OAuth token') 161 | 162 | if pb_api_token is None: 163 | exit_with_error('Could not load Pinboard API token') 164 | 165 | gh_headers = {'authorization': 'token {}'.format(gh_api_token), 166 | 'accept': 'application/vnd.github.v3+json', 167 | 'user-agent': args.github_user} 168 | 169 | # Note: on first run (ie. we've never added starred repos to Pinboard), 170 | # flip the sort direction of the results from GitHub so that bookmarks 171 | # are added in chronological order (ie. when first run is complete, 172 | # the last bookmark added will be the most recent star). 173 | # 174 | # On subsequent runs, we query Pinboard for the most recently added 175 | # bookmark with a specific tag and use that to filter the list of 176 | # stars from GitHub. 177 | sort_dir = 'desc' 178 | most_recent = get_most_recent_bookmark(pb_api_token) 179 | hit_most_recent = False 180 | if most_recent is not None: 181 | most_recent = most_recent[0]['description'] 182 | else: 183 | sort_dir = 'asc' 184 | 185 | print '[INFO] GitHub: Getting starred repos ...' 186 | for result, status in get_github_stars(gh_headers, sort_dir): 187 | if status != 200: 188 | exit_with_error('GitHub: {}'.format(result)) 189 | if most_recent is not None: 190 | result, hit_most_recent = filter_stars(result, most_recent) 191 | for star in result: 192 | print '[INFO] Pinboard: Adding {} ...'.format(star['full_name']) 193 | r, s = add_bookmark(star, pb_api_token, args.lang_tag_prefix) 194 | retry = 1 195 | while s == 429 and retry <= MAX_RETRIES: 196 | print '[WARN] Pinboard: Rate-limited! Retrying ...' 197 | # Back off & retry 198 | sleep(SLEEP_TIME * retry) 199 | r, s = add_bookmark(star, pb_api_token, args.lang_tag_prefix) 200 | if r['result_code'].lower() != 'done': 201 | exit_with_error('Pinboard: {}.'.format(r['result_code'])) 202 | retry += 1 203 | else: 204 | if r['result_code'].lower() != 'done': 205 | print '[WARN] Pinboard: {}.'.format(r['result_code']) 206 | sleep(SLEEP_TIME) 207 | if hit_most_recent: 208 | sys.exit(0) 209 | 210 | 211 | if __name__ == '__main__': 212 | main() 213 | --------------------------------------------------------------------------------