├── README.md ├── twitter_bookmark_organizer_guide.ipynb └── variable_definition.png /README.md: -------------------------------------------------------------------------------- 1 | # How to create an automated Twitter bookmark organizer in Notion 2 | 3 | You're constantly bookmarking tweets but finding a particular one can be a complete mess, here's how you can create your own automatic bookmark organizer in Notion with the Twitter API and GPT-3. 4 | 5 | ## Outline 6 | 7 | 1. Get Twitter developer API credentials 8 | 2. Get OAuth 2.0 Twitter API token 9 | 3. Fetch Twitter bookmarks 10 | 4. Get bookmark keywords with GPT-3 API (optional) 11 | 5. Get Notion API keys 12 | 6. Create a Notion table and connect with API 13 | 7. Add bookmarks to Notion 14 | 8. Refresh your auth token 15 | 16 | 17 | ## Here's what we'll use: 18 | 19 | **1. Python 🐍** 20 | 21 | **2. Twitter API 🐦** 22 | 23 | **3. Notion API 📝** 24 | 25 | **4. GPT-3 API 🤖** *(optional)* 26 | 27 | 28 | ## Detailed walkthrough 29 | Read blog post for a detailed walkthrough: https://norahsakal.com/blog/automatically-organize-twitter-bookmarks-in-notion 30 | -------------------------------------------------------------------------------- /twitter_bookmark_organizer_guide.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "id": "d4e2fef4", 7 | "metadata": {}, 8 | "outputs": [], 9 | "source": [ 10 | "import base64\n", 11 | "import datetime\n", 12 | "import hashlib\n", 13 | "import json\n", 14 | "import os\n", 15 | "import re\n", 16 | "import requests\n", 17 | "from time import sleep\n", 18 | "\n", 19 | "from requests.auth import AuthBase, HTTPBasicAuth\n", 20 | "from requests_oauthlib import OAuth2Session" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "id": "22c5f944", 26 | "metadata": {}, 27 | "source": [ 28 | "## Inspiration\n", 29 | "The steps of generating an OAuth 2.0 token is inspired by this repo: https://github.com/twitterdev/Twitter-API-v2-sample-code/blob/main/Bookmarks-lookup/bookmarks_lookup.py\n", 30 | "\n", 31 | "Detailed step-by-step intructions for this repo in this blog post: https://norahsakal.com/blog/automatically-organize-twitter-bookmarks-in-notion" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "id": "d51f5ed9", 37 | "metadata": {}, 38 | "source": [ 39 | "## Twitter credentials" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": null, 45 | "id": "2dc3642e", 46 | "metadata": {}, 47 | "outputs": [], 48 | "source": [ 49 | "client_id = \"YOUR_CLIENT_ID\"\n", 50 | "client_secret = \"YOUR_CLIENT_SECRET\"\n", 51 | "redirect_uri = \"YOUR_REDIRECT_URL\"" 52 | ] 53 | }, 54 | { 55 | "cell_type": "markdown", 56 | "id": "d8a43022", 57 | "metadata": {}, 58 | "source": [ 59 | "## Set up permission scope" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "id": "8022ac26", 66 | "metadata": {}, 67 | "outputs": [], 68 | "source": [ 69 | "# Set the scopes\n", 70 | "# offline.access makes it possible to fetch \n", 71 | "# a new refresh token when the access token have expired\n", 72 | "scopes = [\"bookmark.read\", \"tweet.read\", \"users.read\", \"offline.access\"]" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "id": "dd22de08", 78 | "metadata": {}, 79 | "source": [ 80 | "## Create a code verifier" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": null, 86 | "id": "319f1836", 87 | "metadata": {}, 88 | "outputs": [], 89 | "source": [ 90 | "code_verifier = base64.urlsafe_b64encode(os.urandom(30)).decode(\"utf-8\")\n", 91 | "code_verifier = re.sub(\"[^a-zA-Z0-9]+\", \"\", code_verifier)" 92 | ] 93 | }, 94 | { 95 | "cell_type": "markdown", 96 | "id": "2b45973c", 97 | "metadata": {}, 98 | "source": [ 99 | "## Create a code challenge" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": null, 105 | "id": "01010ef5", 106 | "metadata": {}, 107 | "outputs": [], 108 | "source": [ 109 | "code_challenge = hashlib.sha256(code_verifier.encode(\"utf-8\")).digest()\n", 110 | "code_challenge = base64.urlsafe_b64encode(code_challenge).decode(\"utf-8\")\n", 111 | "code_challenge = code_challenge.replace(\"=\", \"\")" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "id": "724ef413", 117 | "metadata": {}, 118 | "source": [ 119 | "## Start an OAuth 2.0 session" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "id": "865e57af", 126 | "metadata": {}, 127 | "outputs": [], 128 | "source": [ 129 | "oauth = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)" 130 | ] 131 | }, 132 | { 133 | "cell_type": "markdown", 134 | "id": "fa139c87", 135 | "metadata": {}, 136 | "source": [ 137 | "## Create an authorize URL" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": null, 143 | "id": "d3313b5c", 144 | "metadata": {}, 145 | "outputs": [], 146 | "source": [ 147 | "auth_url = \"https://twitter.com/i/oauth2/authorize\"\n", 148 | "authorization_url, state = oauth.authorization_url(\n", 149 | " auth_url, code_challenge=code_challenge, code_challenge_method=\"S256\"\n", 150 | ")\n", 151 | "\n", 152 | "# Visit the URL you received and authorize your app\n", 153 | "print(authorization_url)" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "id": "ca25805b", 159 | "metadata": {}, 160 | "source": [ 161 | "## Save the URL you got redirected to after authorization" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": null, 167 | "id": "0bb49029", 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "authorization_response = \"THE_URL_YOU_GOT_REDIRECTED_TO_AFTER_AUTHORIZATION\"" 172 | ] 173 | }, 174 | { 175 | "cell_type": "markdown", 176 | "id": "5c7ce250", 177 | "metadata": {}, 178 | "source": [ 179 | "## Fetch your access token" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": null, 185 | "id": "d23b3b4d", 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "token_url = \"https://api.twitter.com/2/oauth2/token\"\n", 190 | "auth = HTTPBasicAuth(client_id, client_secret)" 191 | ] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": null, 196 | "id": "328edb10", 197 | "metadata": {}, 198 | "outputs": [], 199 | "source": [ 200 | "token = oauth.fetch_token(\n", 201 | " token_url=token_url,\n", 202 | " authorization_response=authorization_response,\n", 203 | " auth=auth,\n", 204 | " client_id=client_id,\n", 205 | " include_client_id=True,\n", 206 | " code_verifier=code_verifier,\n", 207 | ")" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": null, 213 | "id": "c8e31438", 214 | "metadata": {}, 215 | "outputs": [], 216 | "source": [ 217 | "token" 218 | ] 219 | }, 220 | { 221 | "cell_type": "markdown", 222 | "id": "ccdaf3e4", 223 | "metadata": {}, 224 | "source": [ 225 | "## Your access token" 226 | ] 227 | }, 228 | { 229 | "cell_type": "code", 230 | "execution_count": null, 231 | "id": "7a1555be", 232 | "metadata": {}, 233 | "outputs": [], 234 | "source": [ 235 | "access_token = token[\"access_token\"]" 236 | ] 237 | }, 238 | { 239 | "cell_type": "markdown", 240 | "id": "4dbb98b2", 241 | "metadata": {}, 242 | "source": [ 243 | "## Make a request to the users/me endpoint to get your user ID" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": null, 249 | "id": "f401101b", 250 | "metadata": {}, 251 | "outputs": [], 252 | "source": [ 253 | "user_me = requests.get(\n", 254 | " \"https://api.twitter.com/2/users/me\",\n", 255 | " headers={\"Authorization\": f\"Bearer {access_token}\"},\n", 256 | ").json()\n", 257 | "\n", 258 | "user_id = user_me[\"data\"][\"id\"]" 259 | ] 260 | }, 261 | { 262 | "cell_type": "code", 263 | "execution_count": null, 264 | "id": "5f0657d1", 265 | "metadata": {}, 266 | "outputs": [], 267 | "source": [ 268 | "user_id" 269 | ] 270 | }, 271 | { 272 | "cell_type": "markdown", 273 | "id": "c37b0fd3", 274 | "metadata": {}, 275 | "source": [ 276 | "## Make a request to the bookmarks url" 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": null, 282 | "id": "eeec36b3", 283 | "metadata": {}, 284 | "outputs": [], 285 | "source": [ 286 | "url = f\"https://api.twitter.com/2/users/{user_id}/bookmarks\"\n", 287 | "headers = {\n", 288 | " \"Authorization\": f\"Bearer {access_token}\",\n", 289 | "}\n", 290 | "response = requests.get(url, headers=headers, params={\n", 291 | " 'tweet.fields':'author_id,created_at',\n", 292 | " 'expansions':'author_id',\n", 293 | " 'user.fields':'username',\n", 294 | "})\n", 295 | "response.json()" 296 | ] 297 | }, 298 | { 299 | "cell_type": "code", 300 | "execution_count": null, 301 | "id": "71ae7027", 302 | "metadata": {}, 303 | "outputs": [], 304 | "source": [ 305 | "print(\"Response:\\n\", response.json().keys(), '\\n')\n", 306 | "print(\"Bookmark #1:\\n\", response.json()['data'][0], '\\n')\n", 307 | "print(\"User #1:\\n\", response.json()['includes']['users'][0], '\\n')" 308 | ] 309 | }, 310 | { 311 | "cell_type": "markdown", 312 | "id": "8248acf8", 313 | "metadata": {}, 314 | "source": [ 315 | "## Create variable with tweets" 316 | ] 317 | }, 318 | { 319 | "cell_type": "code", 320 | "execution_count": null, 321 | "id": "56d48ae1", 322 | "metadata": {}, 323 | "outputs": [], 324 | "source": [ 325 | "tweets = response.json()['data']" 326 | ] 327 | }, 328 | { 329 | "cell_type": "markdown", 330 | "id": "c4dbd7f3", 331 | "metadata": {}, 332 | "source": [ 333 | "## Create mapping dict for user data" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": null, 339 | "id": "a5cc991d", 340 | "metadata": {}, 341 | "outputs": [], 342 | "source": [ 343 | "user_mapping = {user['id']:user for user in response.json()['includes']['users']}" 344 | ] 345 | }, 346 | { 347 | "cell_type": "markdown", 348 | "id": "50c5752d", 349 | "metadata": {}, 350 | "source": [ 351 | "## Combine bookmarks with user data" 352 | ] 353 | }, 354 | { 355 | "cell_type": "code", 356 | "execution_count": null, 357 | "id": "86dc06fe", 358 | "metadata": {}, 359 | "outputs": [], 360 | "source": [ 361 | "for tweet in tweets:\n", 362 | " tweet.update({\n", 363 | " 'name':user_mapping[tweet['author_id']]['name'], \n", 364 | " 'username': user_mapping[tweet['author_id']]['username']\n", 365 | " })\n" 366 | ] 367 | }, 368 | { 369 | "cell_type": "markdown", 370 | "id": "8c0a526d", 371 | "metadata": {}, 372 | "source": [ 373 | "## Optional: populate tweets with keywords with GPT-3" 374 | ] 375 | }, 376 | { 377 | "cell_type": "code", 378 | "execution_count": null, 379 | "id": "cc524abf", 380 | "metadata": {}, 381 | "outputs": [], 382 | "source": [ 383 | "import openai\n", 384 | "\n", 385 | "# Open AI API key\n", 386 | "api_key =\"YOUR_OPEN_AI_GPT3_API_KEY\"\n", 387 | "\n", 388 | "for i, tweet in enumerate(tweets, start=1):\n", 389 | "\n", 390 | " print(f\"Processing {i}/{len(tweets)} tweets\", end='\\r')\n", 391 | " \n", 392 | " # Create a prompt for the completion endpoint\n", 393 | " prompt = f\"Here is a tweet, give me 5 keywords, each keyword on a new line, that describe what the tweet is about \\n\\n --- tweet start ---- \\n\\n {tweet['text']} \\n\\n --- tweet end ---:\"\n", 394 | " \n", 395 | " response_gpt3 = openai.Completion.create(\n", 396 | " model=\"text-davinci-002\",\n", 397 | " prompt=prompt,\n", 398 | " temperature=0.7,\n", 399 | " max_tokens=100,\n", 400 | " top_p=1,\n", 401 | " frequency_penalty=0,\n", 402 | " presence_penalty=0\n", 403 | " )\n", 404 | " \n", 405 | " # Update tweet with keywords\n", 406 | " tweet.update({'keywords':response_gpt3['choices'][0]['text'].strip()})\n", 407 | " \n", 408 | " " 409 | ] 410 | }, 411 | { 412 | "cell_type": "markdown", 413 | "id": "b89b9593", 414 | "metadata": {}, 415 | "source": [ 416 | "## Add tweet to Notion database " 417 | ] 418 | }, 419 | { 420 | "cell_type": "code", 421 | "execution_count": null, 422 | "id": "2046b4de", 423 | "metadata": {}, 424 | "outputs": [], 425 | "source": [ 426 | "# Get the API key from the environment variable\n", 427 | "notion_key = \"YOUR_NOTION_INTEGRATION_KEY\"\n", 428 | "\n", 429 | "# Get the database ID from the environment variable\n", 430 | "notion_database_id = \"YOUR_NOTION_DATABASE_ID\"\n", 431 | "\n", 432 | "# Set the headers\n", 433 | "headers = {\n", 434 | " 'Authorization': 'Bearer ' + notion_key,\n", 435 | " 'Content-Type': 'application/json',\n", 436 | " 'Notion-Version': '2021-08-16'\n", 437 | "}\n", 438 | "\n", 439 | "# Create the payload and make request\n", 440 | "for i,tweet in enumerate(tweets, start=1):\n", 441 | "\n", 442 | " payload = {\n", 443 | " 'parent': { 'database_id': notion_database_id },\n", 444 | " 'properties': {\n", 445 | " 'title': {\n", 446 | " 'title': [\n", 447 | " {\n", 448 | " 'text': {\n", 449 | " 'content': tweet['username']\n", 450 | " }\n", 451 | " }\n", 452 | " ]\n", 453 | " },\n", 454 | " \"Keywords\": {\"rich_text\": [{ \"type\": \"text\", \"text\": { \"content\": tweet['keywords'] } }]},\n", 455 | " \"Name\": {\"rich_text\": [{ \"type\": \"text\", \"text\": { \"content\": tweet['name'] } }]},\n", 456 | " \"Tweet\": {\"rich_text\": [{ \"type\": \"text\", \"text\": { \"content\": tweet['text'] } }]},\n", 457 | " \"URL\": {'url': f\"https://twitter.com/twitter/status/{tweet['id']}\"},\n", 458 | " \"Tweeted at\": {\"date\": {\"start\": tweet['created_at'] }}\n", 459 | "\n", 460 | " }\n", 461 | " }\n", 462 | "\n", 463 | " # Make the request\n", 464 | " r = requests.post('https://api.notion.com/v1/pages', headers=headers, data=json.dumps(payload))\n", 465 | "\n", 466 | " # Print the response\n", 467 | " print(f\"Tweet {i}/{len(tweets)} Response: {r.json()['object']}\", end='\\r')" 468 | ] 469 | }, 470 | { 471 | "cell_type": "markdown", 472 | "id": "43d4381f", 473 | "metadata": {}, 474 | "source": [ 475 | "## Get new access token" 476 | ] 477 | }, 478 | { 479 | "cell_type": "code", 480 | "execution_count": null, 481 | "id": "5c5d7604", 482 | "metadata": {}, 483 | "outputs": [], 484 | "source": [ 485 | "refreshed_token = oauth.refresh_token(\n", 486 | " client_id=client_id,\n", 487 | " client_secret=client_secret,\n", 488 | " token_url=token_url,\n", 489 | " auth=auth,\n", 490 | " refresh_token=token[\"refresh_token\"],\n", 491 | " )\n", 492 | "refreshed_token" 493 | ] 494 | }, 495 | { 496 | "cell_type": "code", 497 | "execution_count": null, 498 | "id": "716e6793", 499 | "metadata": {}, 500 | "outputs": [], 501 | "source": [ 502 | "access_token = refreshed_token[\"access_token\"]" 503 | ] 504 | } 505 | ], 506 | "metadata": { 507 | "kernelspec": { 508 | "display_name": "Python 3", 509 | "language": "python", 510 | "name": "python3" 511 | }, 512 | "language_info": { 513 | "codemirror_mode": { 514 | "name": "ipython", 515 | "version": 3 516 | }, 517 | "file_extension": ".py", 518 | "mimetype": "text/x-python", 519 | "name": "python", 520 | "nbconvert_exporter": "python", 521 | "pygments_lexer": "ipython3", 522 | "version": "3.8.5" 523 | } 524 | }, 525 | "nbformat": 4, 526 | "nbformat_minor": 5 527 | } 528 | -------------------------------------------------------------------------------- /variable_definition.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/norahsakal/automatically-organize-twitter-bookmarks-in-notion/69a7fe6b4a2aabd38e6dfe8aa6c1fe5ac85e72aa/variable_definition.png --------------------------------------------------------------------------------