├── .gitignore ├── requirements.txt ├── screenshot.png ├── LICENSE ├── readme.md └── exporter.py /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | exporter.config 3 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | requests==2.20.0 2 | -------------------------------------------------------------------------------- /screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/corychainsman/github-starred-to-pinboard/HEAD/screenshot.png -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Cory Chainsman 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | Github Starred Repos to Pinboard Bookmarks 2 | ========================================== 3 | 4 | It turns recently starred repos from a Github account into bookmarks in Pinboard. 5 | 6 | It makes the bookmark like this: 7 | 8 | ![example bookmark](https://raw.github.com/cmchap/github-starred-to-pinboard/master/screenshot.png) 9 | 10 | That is, it sets the bookmark title to the repo name followed by the short, one-liner repo description. It lists the languages used in the repo in order of bytes, too. Just for you. Because I like you. It also lists the as much of the readme file as will fit in Pinboard's description field. 11 | 12 | Usage 13 | ----- 14 | 15 | Get your Github OAuth token from [here](https://github.com/settings/applications). 16 | 17 | Get your Pinboard API token from [here](https://pinboard.in/settings/password). 18 | 19 | By default, if you have an existing bookmark with the same URl as a starred repo, this script will change that bookmark to match the above styling. If you wish to change this, set the ```replace``` variable to ```no```. Note that even if ```replace``` is ```yes```, the datetime on existing bookmarks will not be altered by this script. 20 | 21 | The bookmarks will be tagged with the terms in the ```tags``` variable. 22 | 23 | On first run, it creates a config file with the same base filename as you named this script. (If you didn't rename the script, it'll be called ```exporter.config```) 24 | You can uncomment and fill in ```gh_username```, ```gh_token```, and ```pb_token``` if you do not want to create a config file. 25 | 26 | Otherwise, run the script and follow the directions. 27 | 28 | 29 | Requirements 30 | ------------ 31 | 32 | python 2.6 - 2.7.5 33 | 34 | [Requests](http://docs.python-requests.org/en/latest/) 35 | 36 | Limitations 37 | ----------- 38 | 39 | It only works for the 100 most recently starred repos. It works for any number of repos. Thanks, [jdherg](https://github.com/jdherg)! 40 | 41 | API calls are limited to 4103 characters which really cuts down on how much of the readme file is included in the description. If anybody has a good workaround for this, please submit a pull request. 42 | 43 | TODO 44 | ---- 45 | 46 | * Make it work for folks who have more than 100 starred repos. done. 47 | * Make it fail more gracefully 48 | * Pinboard rate limit failure (once every 3 seconds) done. 49 | * Github rate limit failure (60 per hour unauthenticated or 5000 authenticated). The authenticated limit isn't a problem because the pinboard rate limit is already significantly lower: 3/second, or 1200/hour 50 | * Add an option to replace existing bookmarks with the original datetime done. 51 | * Check to ensure the entered github username exists. 52 | 53 | LICENSE 54 | ---- 55 | 56 | This project is licensed under the terms of the MIT license. 57 | -------------------------------------------------------------------------------- /exporter.py: -------------------------------------------------------------------------------- 1 | # Requires python 2.6 - 2.7.5 2 | 3 | # Copyright Cory Chapman, 2013 4 | 5 | # Distributed under the WTF Public License (www.wtfpl.net) 6 | 7 | ############## 8 | ## Settings ## 9 | ############## 10 | 11 | #Change to "no" if you don't want it to replace previously bookmarked repos. 12 | #Take note that even if replace = yes, the datetime on the bookmarks will not be altered. 13 | replace = "yes" 14 | 15 | tags = "github programming github-starred-to-pinboard" #max of 100 tags, separated by spaces 16 | 17 | # Uncomment these lines and fill them in with your info if you don't want to create a config file. 18 | # gh_username = "username" 19 | # gh_token = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" 20 | # pb_token = "username:XXXXXXXXXXXXXXXXXXXX" 21 | 22 | ############### 23 | ## Functions ## 24 | ############### 25 | 26 | import requests, time, sys, re, base64, urllib, ConfigParser, os 27 | 28 | def post_to_pinboard(pb_token, url, title, long_description, tags, replace, name, length=4103, sleep=3): 29 | time.sleep(3) # Pinboard API allows for 1 call every 3 seconds per user 30 | payload = [ 31 | ('auth_token', pb_token), 32 | ('url', url), 33 | ('description', title), 34 | ('tags', tags), 35 | ('replace', replace), 36 | ('extended', long_description) 37 | ] 38 | r = requests.get('https://api.pinboard.in/v1/posts/add', params=payload) 39 | r_status = r.status_code 40 | if r_status == 200: 41 | print "Added " + name 42 | return 1 43 | elif r_status == 403: 44 | print "Your Pinboard token didn't seem to work.\nYou should go get it from here: https://pinboard.in/settings/password" 45 | print "And paste it below.\nIt should look sorta like this: username:XXXXXXXXXXXXXXXXXXXX" 46 | pb_token = raw_input() 47 | return post_to_pinboard(pb_token, url, title, long_description, tags, replace, name) 48 | elif r_status == 429: 49 | print "Whoa, Nellie! We're goin' too fast! Hold on, and we'll try again in a moment." 50 | time.sleep(sleep) # Pinboard API allows for 1 call every 3 seconds per user, so we're doubling the wait for each attempt per the docs. 51 | sleep = sleep*2 52 | return post_to_pinboard(pb_token, url, title, long_description, tags, replace, name, length, sleep) 53 | elif r_status == 414: 54 | print "The api request for " + name + " was " + str(len(r.url)-length) + " characters too long." 55 | print "Shortening..." 56 | shortened_description = truncate_long_description(r.url, length, long_description) 57 | return post_to_pinboard(pb_token, url, title, shortened_description, tags, replace, name) 58 | else: 59 | print "Something went wrong while trying to bookmark " + name + ". I don't know what, but the http status code was " + str(r_status) 60 | return 0 61 | 62 | def get_langs(langs_url, gh_token): 63 | langs = "" 64 | lang_data = requests.get("%s?access_token=%s" % (langs_url, gh_token)) 65 | if lang_data == "{}": 66 | return langs 67 | else: 68 | lang_data = lang_data.json() 69 | langs_sorted = sorted(lang_data.iteritems(), key=lambda bytes: -bytes[1]) #sort the languages into a list by most bytes. 70 | for x in langs_sorted: 71 | langs += "%s = %s bytes\n" % (x[0], x[1]) 72 | return langs 73 | 74 | def get_readme(api_url, gh_token): 75 | r = requests.get(api_url + "/readme?access_token=" + gh_token) 76 | if r.status_code == 200: 77 | readme_b64 = r.json()['content'] 78 | readme = base64.b64decode(readme_b64) 79 | return str(readme) 80 | else: 81 | return "none listed" 82 | 83 | def test_token(url, token): 84 | auth = {'auth_token': token} 85 | r = requests.get(url, params=auth) 86 | if r.status_code == 403: 87 | return 0 88 | else: 89 | return 1 90 | 91 | def smart_truncate(content, length, suffix): 92 | if len(content) <= length: 93 | return content 94 | else: 95 | return content[:length+1-len(suffix)].rsplit(' ',1)[0] + suffix 96 | 97 | def truncate_long_description(url, length, description): 98 | if len(url) <= length: 99 | return description 100 | else: 101 | length = length-3 #take into account adding the ellipsis. 102 | new_url = url[0:length] 103 | new_url = re.sub(r'\+[^\+]*$', "...", new_url) # puts an ellipsis after the last word that will fit within the length requirement 104 | new_long_description = re.sub(r'^.*extended=', "", new_url) #Gets the url-encoded long description 105 | new_long_description = urllib.unquote_plus(new_long_description.encode('ascii')) #hopefully puts the url into utf-8 without the url encoding. 106 | return new_long_description 107 | 108 | def get_github_username(sleep=1): 109 | if not parser.has_section('github'): 110 | parser.add_section('github') 111 | 112 | if not parser.has_option('github', 'username'): 113 | print "Enter a Github username to get their starred repos:" 114 | gh_username = raw_input() 115 | else: 116 | gh_username = parser.get('github', 'username') 117 | 118 | test_url = 'http://github.com/' + gh_username 119 | if 200 <= requests.get(test_url).status_code < 299: 120 | parser.set('github', 'username', gh_username) 121 | else: 122 | time.sleep(sleep) 123 | sleep = sleep*2 124 | print "Your Github token didn't seem to work." 125 | get_github_username(sleep) 126 | 127 | with open(config_file, 'wb') as configfile: 128 | parser.write(configfile) 129 | return gh_username 130 | 131 | def get_github_token(gh_username, sleep=1): 132 | if not parser.has_section('github'): 133 | parser.add_section('github') 134 | 135 | if not parser.has_option('github', 'token'): 136 | print "Go to https://github.com/settings/applications, and create a new token, and paste it here." 137 | gh_token = raw_input() 138 | else: 139 | gh_token = parser.get('github', 'token') 140 | 141 | test_url = 'https://api.github.com/users/' + gh_username + '/starred?page=1&per_page=100' # Fetches 100 starred repos per page 142 | if test_token(test_url, gh_token) == 0: 143 | time.sleep(sleep) 144 | sleep = sleep*2 145 | print "Your Github token didn't seem to work." 146 | get_pinboard_token(gh_username, sleep) 147 | else: 148 | parser.set('github', 'token', gh_token) 149 | with open(config_file, 'wb') as configfile: 150 | parser.write(configfile) 151 | return gh_token 152 | 153 | def get_pinboard_token(sleep=1): 154 | if not parser.has_section('pinboard'): 155 | parser.add_section('pinboard') 156 | 157 | if not parser.has_option('pinboard', 'token'): 158 | print "Enter your Pinboard api token in the form username:XXXXXXXXXXXXXXXXXXXX\nYou can get it from here: https://pinboard.in/settings/password" 159 | pb_token = raw_input() 160 | else: 161 | pb_token = parser.get('pinboard', 'token') 162 | 163 | test_url = 'https://api.pinboard.in/v1/posts/recent?count=1' 164 | if test_token(test_url, pb_token) == 0: 165 | time.sleep(sleep) 166 | sleep = sleep*2 167 | print "Your Pinboard API token didn't seem to work." 168 | get_pinboard_token(sleep) 169 | else: 170 | parser.set('pinboard', 'token', pb_token) 171 | with open(config_file, 'wb') as configfile: 172 | parser.write(configfile) 173 | return pb_token 174 | 175 | 176 | def get_current_from_pinboard(pb_token, tags): 177 | payload = { 178 | 'auth_token': pb_token, 179 | 'tag': tags, 180 | 'format': 'json' 181 | } 182 | r = requests.get('https://api.pinboard.in/v1/posts/all', params=payload) 183 | if r.status_code == 200: 184 | bookmarks = r.json() 185 | return bookmarks 186 | else: 187 | print "Something went wrong while trying to get bookmarks. The status code was " + str(r.status_code) 188 | sys.exit() 189 | 190 | ############## 191 | ## Get info ## 192 | ############## 193 | 194 | # defines the config file as having the same filename as this script with an extension of .config 195 | config_file = os.path.splitext(__file__)[0] + ".config" 196 | parser = ConfigParser.SafeConfigParser() 197 | if os.path.exists(config_file): 198 | parser.read(config_file) 199 | 200 | try: 201 | gh_username 202 | except: 203 | gh_username = get_github_username() 204 | 205 | try: 206 | gh_token 207 | except: 208 | gh_token = get_github_token(gh_username) 209 | 210 | try: 211 | pb_token 212 | except: 213 | pb_token = get_pinboard_token() 214 | 215 | 216 | ############### 217 | ## Main loop ## 218 | ############### 219 | 220 | 221 | # get existing bookmarks from pinboard 222 | existing = {} 223 | for bookmark in get_current_from_pinboard(pb_token, tags): 224 | existing[bookmark['href']] = True 225 | print str(len(existing)) + " existing bookmarks found" 226 | 227 | 228 | # get stars from github 229 | url = 'https://api.github.com/users/' + gh_username + '/starred?page=1&per_page=100' 230 | 231 | r = requests.get(url + "&access_token=" + gh_token) 232 | stars = r.json() 233 | while r.links: # iterate through the pages of github starred repos 234 | if 'next' in r.links: 235 | url = r.links['next']['url'] 236 | r = requests.get(url + "&access_token=" + gh_token) 237 | stars.extend(r.json()) 238 | else: 239 | break 240 | 241 | #If the same number of repos are found as are in pinboard, quit the script. 242 | if len(existing) == len(stars): 243 | print "All your repos are already added." 244 | sys.exit() 245 | 246 | print "Adding your starred repos to Pinboard..." 247 | 248 | count = 0 249 | for star in stars: 250 | repo_url = star['html_url'] 251 | if repo_url == False or repo_url == None or repo_url == "None" or repo_url == "none" or repo_url == "" or repo_url == "null": 252 | repo_url == "" 253 | name = star['name'] 254 | if name == False or name == None or name == "None" or name == "none" or name == "" or name == "null": 255 | name = "" 256 | 257 | # Skip existing starred repos 258 | if repo_url in existing: 259 | print "Skipping " + name 260 | continue # breaks out of the for loop that iterates through the stars. 261 | 262 | tagline = star['description'] 263 | if tagline == False or tagline == None or tagline == "None" or tagline == "none" or tagline == "" or tagline == "null": 264 | tagline = "" 265 | repo_api_url = star['url'] 266 | if repo_api_url == False or repo_api_url == None or repo_api_url == "None" or repo_api_url == "none" or repo_api_url == "" or repo_api_url == "null": 267 | repo_api_url = "" 268 | 269 | #make title 270 | if tagline == "": 271 | title = name 272 | else: 273 | title = name + ": " + tagline #max 255 characters according to the pinboard api. 274 | 275 | # See if the homepage is listed. 276 | page = star['homepage'] 277 | if page == False or page == None or page == "None" or page == "none" or page == "" or page == "null": 278 | homepage = "none listed" 279 | else: 280 | homepage = str(page) 281 | 282 | #Make the programming languages of the repo in order of most bytes. 283 | langs_url = star['languages_url'] 284 | langs = get_langs(langs_url, gh_token) + "\n\n" 285 | 286 | #Make readme 287 | readme = get_readme(repo_api_url, gh_token) 288 | 289 | #Make the description. 290 | long_description = "Github repo \nName: " + name 291 | long_description += "\nTagline: " + tagline 292 | if homepage != "none listed": 293 | long_description += "\nHomepage: " + homepage 294 | if langs != []: 295 | long_description += "\nLanguages:\n" + langs 296 | if readme != "none listed": 297 | long_description = long_description.encode('UTF-8','ignore') 298 | long_description += readme 299 | 300 | #test string lengths. 301 | #Max description = 65536 characters according to the docs. 302 | #in reality, the entire get cannot be longer than 4103 characters 303 | long_description = smart_truncate(long_description, 65536, '...') 304 | # max title is 255 305 | title = smart_truncate(title, 255, '...') 306 | 307 | pinboard_add = post_to_pinboard(pb_token, repo_url, title, long_description, tags, replace, name) 308 | if pinboard_add == 1: 309 | count +=1 310 | if count == 0: 311 | print "Whoopsh. Something went wrong, but we don't know what. We didn't add anything to your Pinboard." 312 | elif count == 1: 313 | print "You're all done. You only had one starred repo, so we added that to Pinboard. Go star more repos!" 314 | elif count > 1: 315 | print "You're all done. " + str(count) + " repos have been added to pinboard!" --------------------------------------------------------------------------------