├── LICENSE ├── README.md └── twitter_user_scraper.ipynb /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Anthony Baum 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Scraping Twitter Lists Using Google Search and Tweepy 2 | Author: Anthony Baum 3 | 4 | Contact email: wxbaum@gmail.com 5 | 6 | ## Note 7 | **If you're reading this after June 2023, this code no longer works after major changes to Twitter's API. If you have access to the paid tier API service, you might find this serviceable with some refactoring.** 8 | 9 | ## Overview 10 | The primary function of this script is to bulk-identify a set of Twitter users that match given search keywords on Google. 11 | 12 | By using a Google search for user-created Twitter lists, we can quasi-crowdsource a large number of accounts by matching lists to the keywords being searched for. Using Tweepy to access the user objects of each user in the lists, a dataset of users tailored for any application can be generated with relative ease. 13 | 14 | Note that this script will take a few minutes to run to prevent Google or Twitter locking us out for making too many requests in a short time period. 15 | 16 | To utilize the script you will need access to the Twitter API. Set up an account and project with Twitter Development here: https://developer.twitter.com/en/apply-for-access. 17 | 18 | No Google search authentication is required, as the googlesearch Python module it utilized. Module repo and documentation available here: https://github.com/MarioVilas/googlesearch 19 | 20 | 21 | 22 | 23 | -------------------------------------------------------------------------------- /twitter_user_scraper.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "name": "twitter_user_scraper.ipynb", 7 | "version": "0.3.2", 8 | "provenance": [], 9 | "collapsed_sections": [] 10 | }, 11 | "kernelspec": { 12 | "name": "python3", 13 | "display_name": "Python 3" 14 | } 15 | }, 16 | "cells": [ 17 | { 18 | "metadata": { 19 | "id": "xMW5RIhrJfq4", 20 | "colab_type": "text" 21 | }, 22 | "cell_type": "markdown", 23 | "source": [ 24 | "# A Scraper for Twitter User Data \n", 25 | "\n", 26 | "This project utilizes the googlesearch module with Tweepy to create a dataset of Twitter users, sourced from user-generated Twitter lists that match a given keyword. By using lists like so, we can bulk identify accounts that match a keyword without having to trawl through user attributes like bios. Tweepy allows access to a plethora of user/status/list data, so once lists and the users contained in the lists have been scraped, the breadth of data that can be collected is expansive. \n", 27 | "\n", 28 | "For demonstration purposes, the project here pulls in favorite and retweet actions and creates a measurement of engagement for each user collected. \n", 29 | "\n", 30 | "This script takes several minutes to run, depending on how intensive the Google search and Twitter API requests are. For Google, a conservative random sleep timer is used to best prevent timeouts. For Twitter, the API is instantiated using wait_on_rate_limit=True, meaning that if the API's wait limit is exceeded, the function will wait until the API is available again rather than throwing an error. \n", 31 | "\n", 32 | "The project requires access to Twitter's API, so you will need to a developer account with Twitter. Set up an account and project here for free: https://developer.twitter.com/en/apply-for-access.\n", 33 | "\n", 34 | "The googlesearch module requires installation. Retrieve it here: https://github.com/MarioVilas/googlesearch. \n", 35 | "\n", 36 | "Notes:\n", 37 | "- The results_to_obtain parameter for scrape_influencers_use_google() is currently set to 50. Increase the number to retrieve more lists and more users. If the number is increased significantly, set the sleep time between each URL to a longer period to prevent being locked out by Google. \n", 38 | "- The list_contains_string parameter is the keyword you will use to search for lists. \n", 39 | "\n" 40 | ] 41 | }, 42 | { 43 | "metadata": { 44 | "id": "bQkdi9K3JBx1", 45 | "colab_type": "code", 46 | "colab": {} 47 | }, 48 | "cell_type": "code", 49 | "source": [ 50 | "# Module imports \n", 51 | "import tweepy\n", 52 | "import time\n", 53 | "import pandas as pd\n", 54 | "import numpy as np\n", 55 | "import json\n", 56 | "import time\n", 57 | "from random import randint, random\n", 58 | "from googlesearch import search as gsearch\n", 59 | "import matplotlib.pyplot as plt\n", 60 | "import seaborn as sns\n", 61 | "%matplotlib inline" 62 | ], 63 | "execution_count": 0, 64 | "outputs": [] 65 | }, 66 | { 67 | "metadata": { 68 | "id": "gqg7pcHspJ98", 69 | "colab_type": "text" 70 | }, 71 | "cell_type": "markdown", 72 | "source": [ 73 | "Authenticating to Twitter and creating the API connection.\n", 74 | "\n", 75 | "Assign the keys for your Twitter developer account to the approriate variables. " 76 | ] 77 | }, 78 | { 79 | "metadata": { 80 | "id": "Z91j6XfBJSlc", 81 | "colab_type": "code", 82 | "colab": {} 83 | }, 84 | "cell_type": "code", 85 | "source": [ 86 | "api_key = ''\n", 87 | "secret_key = ''\n", 88 | "access_token = ''\n", 89 | "secret_token = ''\n", 90 | "\n", 91 | "def authenticate(api_key, secret_key, access_token, secret_token):\n", 92 | "\n", 93 | " auth = tweepy.OAuthHandler(api_key, secret_key)\n", 94 | " auth.set_access_token(access_token, secret_token)\n", 95 | " api = tweepy.API(auth, \n", 96 | " wait_on_rate_limit=True, \n", 97 | " wait_on_rate_limit_notify=True)\n", 98 | " \n", 99 | " return api\n", 100 | "\n", 101 | "\n", 102 | "api = authenticate(api_key, secret_key, access_token, secret_token)" 103 | ], 104 | "execution_count": 0, 105 | "outputs": [] 106 | }, 107 | { 108 | "metadata": { 109 | "id": "2vWZeBZZo8FC", 110 | "colab_type": "text" 111 | }, 112 | "cell_type": "markdown", 113 | "source": [ 114 | "Some api calls will result in a rate limit error. While this script in it's current iteration does not require this now, the function below will handle those if needed for any Tweepy requests. " 115 | ] 116 | }, 117 | { 118 | "metadata": { 119 | "id": "_KLxo3yOjxZL", 120 | "colab_type": "code", 121 | "colab": {} 122 | }, 123 | "cell_type": "code", 124 | "source": [ 125 | "def limit_handled(cursor):\n", 126 | " \n", 127 | " while True:\n", 128 | " try:\n", 129 | " yield cursor.next()\n", 130 | " except tweepy.RateLimitError:\n", 131 | " time.sleep(2)\n", 132 | " continue" 133 | ], 134 | "execution_count": 0, 135 | "outputs": [] 136 | }, 137 | { 138 | "metadata": { 139 | "id": "wa_p7hr_pWml", 140 | "colab_type": "text" 141 | }, 142 | "cell_type": "markdown", 143 | "source": [ 144 | "Parameter notes:\n", 145 | "- list_contains_string - the list keyword to search for\n", 146 | "- results to obtain - number of results to be recorded\n", 147 | "\n", 148 | "Requesting a large number of results will likely cause Google to temporarily prevent access. " 149 | ] 150 | }, 151 | { 152 | "metadata": { 153 | "id": "hHlsPsEVDYZ9", 154 | "colab_type": "code", 155 | "colab": {} 156 | }, 157 | "cell_type": "code", 158 | "source": [ 159 | "\n", 160 | "def scrape_lists_with_google(keyword_string, results_to_obtain):\n", 161 | "\n", 162 | " list_urls = []\n", 163 | "\n", 164 | " urls_checked = 0\n", 165 | " urls_appended = 0\n", 166 | " \n", 167 | " # Perform the Google search, checking only the twitter.com domain\n", 168 | " for url in gsearch(\"site:twitter.com lists \" + keyword_string, \n", 169 | " start=urls_checked, \n", 170 | " num=10, \n", 171 | " pause=randint(3, 5)):\n", 172 | " \n", 173 | " # Take URLs that contain Twitter's '/lists' subdomain\n", 174 | " if '/lists/' in url:\n", 175 | " list_urls.append(url)\n", 176 | " urls_appended += 1\n", 177 | " urls_checked += 1\n", 178 | "\n", 179 | " # If Google is returning 5x as many non-list results as lists, break search\n", 180 | " if urls_checked / urls_appended > 5:\n", 181 | " break\n", 182 | " \n", 183 | " # If we reach our results # request limit, break the search\n", 184 | " elif urls_appended >= results_to_obtain:\n", 185 | " break\n", 186 | " \n", 187 | " # Sleep to prevent timeouts\n", 188 | " time.sleep(randint(3, 5))\n", 189 | "\n", 190 | " list_urls = [url[8:] for url in list_urls] # Removes leading https://\n", 191 | "\n", 192 | " return list_urls\n", 193 | " \n", 194 | "\n", 195 | "list_urls = scrape_lists_with_google(\"influencers\", 10)" 196 | ], 197 | "execution_count": 0, 198 | "outputs": [] 199 | }, 200 | { 201 | "metadata": { 202 | "id": "7nYmaoiCre3V", 203 | "colab_type": "text" 204 | }, 205 | "cell_type": "markdown", 206 | "source": [ 207 | "Getting the individual users and associated attributes is made easy with Tweepy. Using the API created earlier, this function returns a dataframe of individual users and associated Twitter attributes. " 208 | ] 209 | }, 210 | { 211 | "metadata": { 212 | "id": "tSSWgKTCi-0A", 213 | "colab_type": "code", 214 | "colab": {} 215 | }, 216 | "cell_type": "code", 217 | "source": [ 218 | "# Get user id, screen name, bio, follower count\n", 219 | "\n", 220 | "def get_users_in_lists(list_urls):\n", 221 | " \n", 222 | " users_list = []\n", 223 | " bios_list = []\n", 224 | " desc_list = []\n", 225 | " follower_count_list = []\n", 226 | " \n", 227 | " for tw_list in list_urls:\n", 228 | " \n", 229 | " user = tw_list.split('/')[1]\n", 230 | " list_name = tw_list.split('/')[3]\n", 231 | "\n", 232 | " # Using Tweepy api, getting json of all users in list\n", 233 | " list_members_output = api.list_members(user, list_name)\n", 234 | "\n", 235 | " # Parsing json data and appending to appropriate lists instantiated above\n", 236 | " for user_id in list_members_output:\n", 237 | " users_list.append(user_id._json['id'])\n", 238 | " bios_list.append(user_id._json['screen_name'])\n", 239 | " desc_list.append(user_id._json['description'])\n", 240 | " follower_count_list.append(user_id._json['followers_count'])\n", 241 | " \n", 242 | " zipped_data = zip(users_list, bios_list, desc_list, follower_count_list)\n", 243 | " \n", 244 | " # Creates a pandas dataframe containing user attributes\n", 245 | " output_df = pd.DataFrame(list(zipped_data), columns=['user_id', \n", 246 | " 'screen_name',\n", 247 | " 'bio',\n", 248 | " 'followers'])\n", 249 | " # Dropping duplicate users if they were in multiple lists\n", 250 | " output_df.drop_duplicates('user_id')\n", 251 | " \n", 252 | " output_df.set_index('user_id', inplace=True)\n", 253 | " \n", 254 | " return output_df\n", 255 | " \n", 256 | "\n", 257 | "influencers_df = get_users_in_lists(list_urls)" 258 | ], 259 | "execution_count": 0, 260 | "outputs": [] 261 | }, 262 | { 263 | "metadata": { 264 | "id": "vZbjS6kuSfwm", 265 | "colab_type": "code", 266 | "outputId": "ac28dc5e-252b-4131-c872-72beee96ed47", 267 | "colab": { 268 | "base_uri": "https://localhost:8080/", 269 | "height": 235 270 | } 271 | }, 272 | "cell_type": "code", 273 | "source": [ 274 | "influencers_df.sort_values(by='followers', ascending=False).head()" 275 | ], 276 | "execution_count": 16, 277 | "outputs": [ 278 | { 279 | "output_type": "execute_result", 280 | "data": { 281 | "text/html": [ 282 | "
\n", 283 | "\n", 296 | "\n", 297 | " \n", 298 | " \n", 299 | " \n", 300 | " \n", 301 | " \n", 302 | " \n", 303 | " \n", 304 | " \n", 305 | " \n", 306 | " \n", 307 | " \n", 308 | " \n", 309 | " \n", 310 | " \n", 311 | " \n", 312 | " \n", 313 | " \n", 314 | " \n", 315 | " \n", 316 | " \n", 317 | " \n", 318 | " \n", 319 | " \n", 320 | " \n", 321 | " \n", 322 | " \n", 323 | " \n", 324 | " \n", 325 | " \n", 326 | " \n", 327 | " \n", 328 | " \n", 329 | " \n", 330 | " \n", 331 | " \n", 332 | " \n", 333 | " \n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | " \n", 338 | " \n", 339 | " \n", 340 | " \n", 341 | " \n", 342 | " \n", 343 | "
screen_namebiofollowers
user_id
44409004shakiraShakira NADA https://t.co/50Yl0h1Xkh51005025
115485051ConanOBrienThe voice of the people. Sorry, people.28457778
32959253khloekardashian26735374
268439864xtina#TheXperience LIVE in Vegas starting May 31st ...16988954
40908929Usher12373871
\n", 344 | "
" 345 | ], 346 | "text/plain": [ 347 | " screen_name bio \\\n", 348 | "user_id \n", 349 | "44409004 shakira Shakira NADA https://t.co/50Yl0h1Xkh \n", 350 | "115485051 ConanOBrien The voice of the people. Sorry, people. \n", 351 | "32959253 khloekardashian \n", 352 | "268439864 xtina #TheXperience LIVE in Vegas starting May 31st ... \n", 353 | "40908929 Usher \n", 354 | "\n", 355 | " followers \n", 356 | "user_id \n", 357 | "44409004 51005025 \n", 358 | "115485051 28457778 \n", 359 | "32959253 26735374 \n", 360 | "268439864 16988954 \n", 361 | "40908929 12373871 " 362 | ] 363 | }, 364 | "metadata": { 365 | "tags": [] 366 | }, 367 | "execution_count": 16 368 | } 369 | ] 370 | }, 371 | { 372 | "metadata": { 373 | "id": "nh8Gr_O-_Xdi", 374 | "colab_type": "text" 375 | }, 376 | "cell_type": "markdown", 377 | "source": [ 378 | "Accounts which are protected will not be accessible for the majority of Tweepy user object functionality. These need to be removed. \n", 379 | "\n", 380 | "When the API connection was built at the start of the script, the wait_on_rate_limit and wait_on_rate_limit_notify parameters were set to true. While those will catch a wait limit error, notify us, and go again when ready, this function checks a lot of users very quickly and will hit the rate limit most of the time if left alone. A sleep timer for each user has been set as a random float up to 0.2 (representing sleep up to 0.2s). This will not prevent rate limit errors 100% of the time but strikes a balance between error prevention and run time. \n", 381 | "\n", 382 | "Run time for this function is as such maximized at 0.2 * n_users seconds.\n" 383 | ] 384 | }, 385 | { 386 | "metadata": { 387 | "id": "F_W70DGAbJUX", 388 | "colab_type": "code", 389 | "outputId": "02062895-cabc-4fcc-9e97-a299b4a635c8", 390 | "colab": { 391 | "base_uri": "https://localhost:8080/", 392 | "height": 119 393 | } 394 | }, 395 | "cell_type": "code", 396 | "source": [ 397 | "def remove_protected_accounts(users_df):\n", 398 | " \n", 399 | " working_df = users_df.copy()\n", 400 | " protected_ids = []\n", 401 | " \n", 402 | " counter = 0\n", 403 | " for index, row in users_df.iterrows():\n", 404 | " user = api.get_user(id=index)._json\n", 405 | " if user['protected'] == True:\n", 406 | " protected_ids.append(index)\n", 407 | " counter += 1\n", 408 | " if counter % 100 == 0:\n", 409 | " print(str(counter) + '/' + str(len(working_df)) + ' IDs checked')\n", 410 | " print('Protected IDs found:', len(protected_ids), '\\n')\n", 411 | " # Sleeping for up to 0.2s helps not to exceed Tweepy's rate limit.\n", 412 | " time.sleep(random() / 5)\n", 413 | " \n", 414 | " working_df.drop(labels=protected_ids, inplace=True)\n", 415 | " working_df.reset_index(inplace=True)\n", 416 | " \n", 417 | " return working_df\n", 418 | " \n", 419 | "\n", 420 | "influencers_df = remove_protected_accounts(influencers_df)" 421 | ], 422 | "execution_count": 17, 423 | "outputs": [ 424 | { 425 | "output_type": "stream", 426 | "text": [ 427 | "100/200 IDs checked\n", 428 | "Protected IDs found: 1 \n", 429 | "\n", 430 | "200/200 IDs checked\n", 431 | "Protected IDs found: 3 \n", 432 | "\n" 433 | ], 434 | "name": "stdout" 435 | } 436 | ] 437 | }, 438 | { 439 | "metadata": { 440 | "id": "GTlVXskl176A", 441 | "colab_type": "text" 442 | }, 443 | "cell_type": "markdown", 444 | "source": [ 445 | "Now protected accounts have been removed, engagements with users' tweets can be examined. We can't access actual impressions for tweets, but we can access some engagement actions (favorites, retweets) for tweets. By looking at engagement actions in relation to the number of followers a user has, we can see how much their audience is engaged with that user's content. \n", 446 | "\n", 447 | "While this particular notebook just overviews the scraping code, it's a helpful excercise to generate some additional data to demonstrate a use case. As we've been looking at influencers for the purposes of this script, we can get a fairly good measurement of how effective an influencer might be by looking at metrics like this. \n", 448 | "\n" 449 | ] 450 | }, 451 | { 452 | "metadata": { 453 | "id": "0fPlPDdVZOzz", 454 | "colab_type": "code", 455 | "outputId": "a56bedcd-9a9e-4588-bbce-553ee7898588", 456 | "colab": { 457 | "base_uri": "https://localhost:8080/", 458 | "height": 51 459 | } 460 | }, 461 | "cell_type": "code", 462 | "source": [ 463 | "def calc_median_favorites(user_id):\n", 464 | " \n", 465 | " fav_list = []\n", 466 | " tweets = api.user_timeline(id=user_id, count=100)\n", 467 | " \n", 468 | " for tweet in tweets:\n", 469 | " # Remove retweets\n", 470 | " if tweet._json['text'].startswith('RT'):\n", 471 | " continue\n", 472 | " # Remove replies\n", 473 | " elif tweet._json['text'].startswith('@'):\n", 474 | " continue\n", 475 | " else:\n", 476 | " fav_list.append(tweet._json['favorite_count'])\n", 477 | " \n", 478 | " time.sleep(random() / 5)\n", 479 | " \n", 480 | " return np.median(fav_list) \n", 481 | "\n", 482 | "\n", 483 | "influencers_df['median_favs'] = influencers_df['user_id'].apply(lambda x: calc_median_favorites(x))" 484 | ], 485 | "execution_count": 18, 486 | "outputs": [ 487 | { 488 | "output_type": "stream", 489 | "text": [ 490 | "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.\n", 491 | " out=out, **kwargs)\n" 492 | ], 493 | "name": "stderr" 494 | } 495 | ] 496 | }, 497 | { 498 | "metadata": { 499 | "id": "1apvWYVwalN9", 500 | "colab_type": "code", 501 | "outputId": "d4467c89-add6-4b8e-983c-10bd8a1d40af", 502 | "colab": { 503 | "base_uri": "https://localhost:8080/", 504 | "height": 51 505 | } 506 | }, 507 | "cell_type": "code", 508 | "source": [ 509 | "def calc_median_retweets(user_id):\n", 510 | " \n", 511 | " rt_list = []\n", 512 | " tweets = api.user_timeline(id=user_id, count=100)\n", 513 | " \n", 514 | " for tweet in tweets:\n", 515 | " if tweet._json['text'].startswith('RT'):\n", 516 | " continue\n", 517 | " else:\n", 518 | " rt_list.append(tweet._json['retweet_count'])\n", 519 | " \n", 520 | " time.sleep(random() / 5)\n", 521 | " \n", 522 | " return np.median(rt_list) \n", 523 | "\n", 524 | "\n", 525 | "influencers_df['median_rts'] = influencers_df['user_id'].apply(lambda x: calc_median_retweets(x))" 526 | ], 527 | "execution_count": 19, 528 | "outputs": [ 529 | { 530 | "output_type": "stream", 531 | "text": [ 532 | "/usr/local/lib/python3.6/dist-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.\n", 533 | " out=out, **kwargs)\n" 534 | ], 535 | "name": "stderr" 536 | } 537 | ] 538 | }, 539 | { 540 | "metadata": { 541 | "id": "35abBVjAVTlf", 542 | "colab_type": "code", 543 | "outputId": "ecad73b9-18ce-4f28-fd5f-de0bc2f14f7a", 544 | "colab": { 545 | "base_uri": "https://localhost:8080/", 546 | "height": 204 547 | } 548 | }, 549 | "cell_type": "code", 550 | "source": [ 551 | "influencers_df[5:10]" 552 | ], 553 | "execution_count": 27, 554 | "outputs": [ 555 | { 556 | "output_type": "execute_result", 557 | "data": { 558 | "text/html": [ 559 | "
\n", 560 | "\n", 573 | "\n", 574 | " \n", 575 | " \n", 576 | " \n", 577 | " \n", 578 | " \n", 579 | " \n", 580 | " \n", 581 | " \n", 582 | " \n", 583 | " \n", 584 | " \n", 585 | " \n", 586 | " \n", 587 | " \n", 588 | " \n", 589 | " \n", 590 | " \n", 591 | " \n", 592 | " \n", 593 | " \n", 594 | " \n", 595 | " \n", 596 | " \n", 597 | " \n", 598 | " \n", 599 | " \n", 600 | " \n", 601 | " \n", 602 | " \n", 603 | " \n", 604 | " \n", 605 | " \n", 606 | " \n", 607 | " \n", 608 | " \n", 609 | " \n", 610 | " \n", 611 | " \n", 612 | " \n", 613 | " \n", 614 | " \n", 615 | " \n", 616 | " \n", 617 | " \n", 618 | " \n", 619 | " \n", 620 | " \n", 621 | " \n", 622 | " \n", 623 | " \n", 624 | " \n", 625 | " \n", 626 | " \n", 627 | " \n", 628 | " \n", 629 | " \n", 630 | " \n", 631 | " \n", 632 | " \n", 633 | " \n", 634 | " \n", 635 | " \n", 636 | " \n", 637 | " \n", 638 | "
user_idscreen_namebiofollowersmedian_favsmedian_rtsengagement
5214164173DrJaneChiFamily physician, intersectional feminist, cat...1366924.50.01.0
6212657985michelleinbklynNYT columnist; co-host of The Argument podcast...70811202.035.01.0
7189896485emmacargowriter, @ProblemAreasHBO. former editor, @Jeze...2523873.00.01.0
8142721190elisefoleyHuffPost. Email: elise dot foley at huffpost d...6128111.50.01.0
987070315ingridnilsenCurious mind. Adventurous heart. Eater of all ...119606771.54.01.0
\n", 639 | "
" 640 | ], 641 | "text/plain": [ 642 | " user_id screen_name \\\n", 643 | "5 214164173 DrJaneChi \n", 644 | "6 212657985 michelleinbklyn \n", 645 | "7 189896485 emmacargo \n", 646 | "8 142721190 elisefoley \n", 647 | "9 87070315 ingridnilsen \n", 648 | "\n", 649 | " bio followers median_favs \\\n", 650 | "5 Family physician, intersectional feminist, cat... 13669 24.5 \n", 651 | "6 NYT columnist; co-host of The Argument podcast... 70811 202.0 \n", 652 | "7 writer, @ProblemAreasHBO. former editor, @Jeze... 25238 73.0 \n", 653 | "8 HuffPost. Email: elise dot foley at huffpost d... 61281 11.5 \n", 654 | "9 Curious mind. Adventurous heart. Eater of all ... 1196067 71.5 \n", 655 | "\n", 656 | " median_rts engagement \n", 657 | "5 0.0 1.0 \n", 658 | "6 35.0 1.0 \n", 659 | "7 0.0 1.0 \n", 660 | "8 0.0 1.0 \n", 661 | "9 4.0 1.0 " 662 | ] 663 | }, 664 | "metadata": { 665 | "tags": [] 666 | }, 667 | "execution_count": 27 668 | } 669 | ] 670 | }, 671 | { 672 | "metadata": { 673 | "id": "rAtWyt4KKf7e", 674 | "colab_type": "text" 675 | }, 676 | "cell_type": "markdown", 677 | "source": [ 678 | "We've likely got a notification above for a RuntimeWarning. This is numpy trying to take the median of of an empty array, that is, accounts that have never tweeted. Those accounts are visbile in the data as they have nan values under median_favs and/or median_rts. We can check to see if any exist using a heatmap for null values. " 679 | ] 680 | }, 681 | { 682 | "metadata": { 683 | "id": "4pgOyK1p3nt1", 684 | "colab_type": "code", 685 | "outputId": "83a8f992-6cdd-4697-cbb6-e1dadb16ed04", 686 | "colab": { 687 | "base_uri": "https://localhost:8080/", 688 | "height": 286 689 | } 690 | }, 691 | "cell_type": "code", 692 | "source": [ 693 | "sns.heatmap(influencers_df.isnull())" 694 | ], 695 | "execution_count": 21, 696 | "outputs": [ 697 | { 698 | "output_type": "execute_result", 699 | "data": { 700 | "text/plain": [ 701 | "" 702 | ] 703 | }, 704 | "metadata": { 705 | "tags": [] 706 | }, 707 | "execution_count": 21 708 | }, 709 | { 710 | "output_type": "display_data", 711 | "data": { 712 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWIAAAD8CAYAAABNR679AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XucXGWd5/FPVacTSCAQCFdRUAg/\nRVAM0SEiBIwXGFkvyLgOOg4uM84oKo6XXW/jehsZd2Sz4m3FHXVmHRVFYUSRYZSLaBS5KLAr/lQu\n4RIugQBJCEmnL/PH7zmck0p1d+Wck+7T3d83r3pV1amqp54O1b96+nee3/O0RkZGEBGRydOe7A6I\niMx0CsQiIpNMgVhEZJIpEIuITDIFYhGRSaZALCIyyWZNdgdERKYiMzsc+Fdghbt/tuOxFwGfAIaA\nS9z9Y2O1VXsgNrMVwNHACHCWu19b93uIiEwmM5sHfAb48ShPORd4KXAPcJWZfcfdfzNae7WmJsxs\nGbAY2Dt18Nw62xcRaYjNwB8DqzsfMLOnAWvd/S53HwYuAZaP1VjdI+ITgcOBQeANQJ+ZzXf3dd2e\nPLDuIZX1iUhPZs/fs1W1jWcduKznmHPTqqtGfT93HwQGzazbw/sCawr3HwAOHuu96j5ZdyTQAh5P\n13unTomIzFTjfoHUHYhnE8PwrwG7AP074D1EREpptVo9XypYzdYD0CfRJYVRVHdq4i7g1cSZwnnp\n2D01v4eISCmt1o4fF7r7HWY238wOAu4GTgZeN9Zr6g7E1xKBeD4xHH/Q3dfX/B4iIqW0x88S9MTM\njgLOAQ4CtpjZqcD3gNvd/ULgzcA30tPPd/ffjdVe3YHYiJTECDEqXmhmLXfXSTkRmXQVUw5PcPfr\ngePHePwnwNJe26s7EL8gXW8m8sPDwCJgzG8DEZGJ0J6A1EQZdffqkHQ9B+hL7R9d83uIiJQyQSfr\ntlvdgfgqYAsxfW04Hbu65vcQEZlWKqUmutRaH0yMhB8CHgP2AgaqdlJEpA59rb7J7kJXpUfEqdb6\ny0SwPS4dHiRmS+wLLCRGxfdX7KOISC2ampqoMiLuI4LuHPLc8CJixkSLGAnPAfZEwVhEGqA9wQG2\nV3XkiPuIUTFEZd3j6fbmwuMiIjKKOgJxm5g7DJGK+DoxAp6fjt1Xw3uIiFTWot3zZSLVMY84S09A\npCNOJ+YQA2xOy8CJiEy6vvb0nkecBfTOdSU+UVP7IiKVtbbjv4lUZUR8JHFyDmCWmV1JjIj7iPJm\nCo+LiMgoqgTiTcAfgKMKxw5L19lI+5UV2hcRqVVTS5yrBOJ+4NDC/dlsuwDyzhXaFxGp1UTPD+5V\n1XnERQNEefNwur0rUV0nItIITZ1HXCUQ/zUwt3B/DjFKvodIW8xDu3OISINM9Em4XpUKxGZ2AnAs\nsJEY+bZICyQT24JAlDuPuT2IiMhEmm454jYRfIsLvi8sHBvqeExEZNI1NUdc5ethAPgN+VS1YuHG\nCDEi3qXzRSIik6XdavV8mdB+VXjtbODZ5Cft+oHbyHfnyPLFIiKNMB0LOuaQlzZDjIgXAjul+y3g\nUxXaFxGpVVNTE2UD8Txg945jA8QoOfMTd/9WyfZFRGrX1OlrZVMTLyFSEgPky13eQRRwDKWL0hIi\n0ihNTU2UDcR/IFIRs8hHwUakI/rS5T+b2XmVeygiUpN2q93zZUL7VfJ12S4c2Ym6wdRWccpaP3B+\n+a6JiMwMVcJ+i3ze8O2F4+vJp689t0L7IiK1mm571j2dCMLDxNZIT0vHNxFT2J5F5IkfrNpBEZG6\n9E2zyrqsUGMDkSPuI4Jyi5hbDDHa/mqVzomI1Gm6zZoYIFIPfeTzhgfT7azEeRB4S9UOiohMd2UD\n8ZXAb8lXX9vS8fhmYrS9pGT7IiK1m245YoD9yE/W9afr24jV1+YSqYrfV+2giEhdmpqaKBuIDdiN\nbVdY24WtUxWfKdm+iEjtptV6xMAryE/QZemNe4F9Cs95j7s/UqFvIlPCkiNOmewuzAg3rbqqchvT\nbUR8F7A/kZKYSwTldcC+5Mtiag6xzAjX3fzdye6C9Kipi/6UPVl3G7Hy2txCG50lztrBWUQapanr\nEZcdER9CjIaLtIOziDTadMsRZzYQo+J+tIOziDRcnSNdM1sBHE1MWjjL3a8tPHYm8HoiVXudu79j\nzH6V7MMwEYTb5CPjWcCadLxK2yIiO0Rd84jNbBmwyN2XAmcA5xYemw+8BzjW3V8AHGZmR4/VXpWC\njjVsvUPHIDGHeD/iW0AjYhFplBpzxMuBiwDc/RZgQQrAEFmBAWAXM5tFZA3Wjtmvkj/PT4G7yVMb\nI+n2UOE52jhURBqlxsq6fYnBaGZNOoa7bwI+QkxqWAVc4+6/G6uxsoH41cAz05tkaxNnq7GNABvR\nyToRaZgduEPHEy9II+P3A4cCTwX+yMyePdoLoXwgPpzYKDQrc4YIwJcS84l3LdmuiMhUsJo0Ak72\nJ4raAJ4B3ObuD7r7AHA1cNRYjZUNxE8n5gp35ohfSpQ+A6iqTkQapd3q/TKOy4BTAcxsMbDa3den\nx+4AnmFmWVZgCeOsu1N2+tp9wAHko+Fh4CHynZ1HgPeWbFtEZIfoa9czmcvdV5rZ9Wa2koh/Z5rZ\n6cCj7n6hmf0DcIWZDQIr3f3qsdorG4izrZAeJx8B3wnsXXjOgSXbFhHZIeoscXb3zsHmjYXHvgh8\nsde2qnw9DBPrDmeWFNprAe+s0LaIyIxRpbLuUfLRMOS7OA8T+ePZFdoWEaldu6ElzmVHxKuIScrF\nn2qQrXfquBcRkQaZbjt03EoE4izwbiFGwI+k2/uhecQyQ2g94omh9Yi3dRqRfsh26Mi2StqTGCUP\nVmhbZErResRTR0PjcOlgmW2HtIkoZc7ywpuJoDxEvrGoiEgjTLcRcXYiLgvIWYnzbLaeOSEi0hhN\nXY+47Mm6zR33s3a+T8ym6CvdIxGRHaSpJ+vKBuJfp+tH2XrFNZU4i0hjNXWrpLKBeC6RF14JPEik\nIe4hP3mnEmcRaZxWq/fLRCqbI+4jgvhJ5MH8VmLaWkYlziLSKE09WVfndkbHohJnEWmwHbgecSVV\n5/o+RkxXy6rsVOIsIo3V1BFx2UCcVc3NI9+ZY4gIxNmMCZU4i0ijNDQOl05NPJ6uW4VLP7E7xwPp\nMZU4i4j0oOyI+JfAMR3HVOIsIo1W18LwdSsbLI8g3yR0FjH6HUElziLSYE1NTZQNxHsRI9/iUpgq\ncRaRRmvqybqy4/Q9uxwbQSXOIiLbreyIuFvaIdvFOdvZWSXOItIo023Rn4F0vY58AaB7UYmziDRY\nUxf9KTsizuYMzyfPBf8eOKDwHJU4i0ij9LWn14j4MfL5w5kXohJnEZHtVnZEnBV0bCaC7k6oxFlE\nGm6iUw69qlp0kZ2YGyHfSFQlziLSSA3NTJQOxHun62EiHZHNIX6ICMj7ohJnEWmY6TYinkM+QyIz\nAuxBBOUhVNAhIg3T0DhcOhBnU9YeBXbvOD6HmFOsEmcRaZTpVllXXGGtOFNidrqeharrRKRhmrow\nfNlA/Fti1JsFXoh88QXkMyk6UxciIpNquu1Z9y7gWYARAXiAyAsfSR6c/38dHRQRqct0S02cSOSC\nsxkT2RrEh5CPkL9UuXciIjNAlXnEG9L1eiJXfD9R8gyRlnisQtsiIrVrN3QicakRsbv/I/DjdLef\nmD98KJGmGCFGxeeZ2fvr6KSISB2m26I/RX3ko9/i+hOzgG/W0L6ISC3qHBCb2QrgaGLweZa7X1t4\n7MnAN4hzZje4+1+P2a+SHTgDWE6egngqcbKuna4B1gDPK9O+iEiTmdkyYJG7LwXOAM7teMo5wDnu\n/jxgyMyeMlZ7ZU/WnU+kIyAWANqS2hoAfp2O7wI8WLJ9EZHa1ZiaWA5cBODutwALzGw+gJm1gWOB\n76XHz3T3O8dqrGwgfhOQRfi9yfPCVwOL0/1+d/9RyfZFRGrX1271fBnHvsRf/Zk16RjEnp7rgRVm\n9lMzO3u8xsoG4iOBfciXvOwjgu9yCsthmtnLS7YvIlK7HXiyrtVx+0nAp4FlwHPM7GVjvbhsIIZY\ngzibRwz59kkQgXgQWFKhfRGRplpNPgIG2J986d8HgVXufqu7DxEzzJ45VmNlA/FT2XotiX4iT7ya\n/GTdHGL7JBGRRqixxPky4FQAM1sMrHb39QDuPgjcZmaL0nOPAnysxsoG4oOJgLuFPPC2gM+nNvuA\nh9H0NRFpkLpSE+6+ErjezFYSMybONLPTzexV6SnvAL6SHn8UuHis9srOI86CbZt8cZ9h4GPkqYqP\nuPuWLq8VEZkUddZpuHvnTvU3Fh77A/CCXtsqG4j7yE/KZSmK+4i1JkAlziLSQNNt0Z/ZRLBdT75I\n/CK2LnH+ZOXeiYjUaLotgzlABNtd2TqY941yW0Rk0jV1z7qyI+INXY5lOzkPp/tXlGxbRGSHmG4j\n4p3YdgeOFnAHsCexiejR5bslIlK/po6Iq+zi3CLyw/3kaYgDifzxJiJtISIi46gyawK23rMua2+E\nCNSauiYijdLDGhKTomwgzl63gQjGO6X72XrEQ4VjIiKN0NDMROlAvIU8R7yZPOj+jKi625lYBlNE\npDGamiMuO2viD+l6XrpABOTdgYXp+q5qXRMRmRnKBuIbyXPD2fXDwNOJk3cA36/QLxGR2jV1+lrZ\nQFxcDD5LUVxLHpRHgHXVuiYiUq92u9XzZUL7VfJ1e6XrjcS6wwAnkS8Q3wLeWa1rIiL1mo67OLeA\nueQj4llEVd1wun1/ta6JiMwMZQPxBvIAXKywaxXaXFi2UyIiO0JDJ030lpows8PN7FYze2s6tAcR\ndDcQix5DzB3+GfmGev2IiDRIU1MT4wZiM5sHfJnICx+XDu+drucBC9LtFrCUfCTcb2Z71NdVEZFq\nmjpropfURLYI/Bzyhd8Xsu2iP0PkJc7ZbIpjGGeLEJGpbskRp0x2F2aEm1ZdVbmN6VDi3Ec+W2I2\n+XKXmWwBINJjbWAxCsQyzV1383cnuwsyxfU6fW0j8HjhfovYMvqG9BjEvOH1xL51WVXdPjX0UUSk\nFlM2R0wE2ncSU9WK60fsBjyDGB0D/JpIR7wCeHI6Vv1vCRGRmkzlHPEc4AtE0J5TOJ6lJrK88PuI\n0ucjyHd3vqy2noqIVNRqaI64lxHxUcQaEgA7mdmV6XYf+WahI+5+EzF6bqfjW4jFf0REGqGpI+Ky\nJc7bMLOTifRFtkTmbODDdbUvIlJVU3PEVUqci1rAYR33AX5VU/siIpU1tbKuSiDOFvsZItIUhxMj\n4VnkgfjwCu2LiNRqKi8M/2vA0+1Bdz++cHwu8Ei6/+9EAL6Z2DwU4IF6uikiUl1Tc8S9jIiPBCzd\n7i+crDsyvX42MRLeMx0/jDzAH1BPN0VEpq8qqYmsxHmImCWRFXZksylAC/+ISIO02rXNT6hVHSfr\nhomquyPJS5uzecT71dC+iEgtGpoi7nn6WjbCzUbBm8jzwIPA74DHgAEiZ3whkS8ulkWLiEyqVrvV\n82Ui9RKIN5Hv2pz5HjFfGCL4ngdcSqQpVgMvT4/9cw19FBGZ1npJTfQDh3YcWwksT7edWK94FrE+\n8dPJUxOX1tNNEZHqpnJqoq/LseMLr90X+Dt3fxyVOItIgzW1sk4lziIyY7TbrZ4vE6muEmdQibOI\nzCBmtgI4mhh4nuXu13Z5ztnA0kIhXFd1Tqorlji3C8dERBqhrso6M1sGLHL3pcAZwLldnnMY+T6f\nYypb4vwL8t2bNwP/E5U4i0jD1ZgjXg5cBODutwALzGx+x3POAT7QS796CcTdSpzfQL5/3RHAJWxd\n4pzt2qESZxFpjvZ2XMa2L7CmcH9NOgaAmZ1O7FB0Ry/dqiNH3Ac8BZU4i0jD7cDZEE80bGZ7AG8E\nXgQ8qZcXl62s2wKsKjw+ixg5DxWerxJnEWmUGldfW01hBAzsD9ybbr+QyBhcTVQZL04n9kZVtrLu\n8vTGEEH4UeAeoqR5mNjNGWBtD+2LiEyIGnPElwGnApjZYmC1u68HcPcL3P0wdz8aeBVwg7v/zViN\n9RKIu1XWvZitd3QeAG4HdiZO7M0nhuo/6KF9EZEJUdeI2N1XAteb2UpixsSZZna6mb2qTL96yRF3\nq6xbUDg+nO5fmI4tKrT7b2U6JSKyQ9SYI3b393YcurHLc+4gKpHHVHYecTZDYnNqY3ahxDk7QbfG\n3e8u2b6IyIxRpaAjW3u4BWBmzyUvcR4C9jKzj1fuoYhITdp9rZ4vE6ns9LUtxMi3mLY4qnA7+ylm\nIyLSENNt81AnRr2Pp+u1wMnpOS3yQHxIbT0VEamoqZuHlq2sO4IYEe9MjIr3AM5PzxkovPa39XRT\nRGT6Kpua6Py+KO7inO3q3EIlziLSJA1NTZQNxCMd9zeiEmcRabiJ3ouuV70G4s4S52EiN5wF3cfI\nS5yzHTraqMRZRBqkqYG4bInzBvIZE610WyXOIiIllC1xXkC+sA/EyFclziLSaE2dNVG2xLmPqKrb\nRGwQ2o9KnEWk4aZyaqKbAWLEm61Iv0klziLSdNNtF+ehjvubVeIsIo3X2o7LBCo7fW01W88Rno1K\nnGWGWnLEKZPdhRnhplVXVW6jqSXOvQTirMR5CanE2cweB+4H9gZ2IgKuSpxlRrru5u9OdhekR00N\nxGVLnLPX9hFBdwSVOItI09W3eWityqYm1gKD5CfmBlGJs4g03FQeEY9mQeH2AFuXOGfzi1XiLCIy\njrIlzrOI4o3HiZkSoBJnEWm4qTyPuFuJ825E6uHBdH8dKnEWkYZrtVs9XyZSmRLn2ekyizwH3I9K\nnEWk6Rpa49xLIO5W4ryZSENkvb0DlTiLiJRS9mTdQ0Qg3pLub1GJs4g0XUMHxKUC8QAxAr4PuBfA\n3ZepxFlEmq6pa02UmUc8G3gOhQo6M7sK+EbhOSpxFpHGafVNcKVGj7Z7F2fgT0Z5nkqcRURK2O4S\nZ+Db6TqbOQHaxVlEpoJptvpaFsA3E1PWQCXOItJwU73EubOyDmLke1O6vZa8xHmwcFslziLSGE0t\n6OhlRJxV1nWuNzybWBoTYC/gcOARYuukEWLmxK619VREpKJWe+qerOu2eehIx/19ibnFuxEB+OHU\n9npERGRMZTYPHSDWlNgE3AwsIwLz74kUhhOzJVrAbbX1VKShpuIOHTN2MftmpohLn6zbOb322eQn\n5rIS50PI5w+fV7WDIk03Y4PaFDSVV18bTT+wCxGIN6YS58fIT9A97O6rKvZPRKQ+Da1xLjsiztYb\nhgjE81KJ8zzyE3ULzOzj7v7B6t0UEamuzulrZrYCOJqIeWe5+7WFx04AziZioQN/4e7DXRuiRGWd\nux+fXjcIbEhvtDP5IvDriDzyEJFDFhFphnar98sYzGwZsMjdlwJnAOd2POU84FR3P4aYPXbiWO31\nMiIebfPQWURqok0s9PM36Xi2FjHAWeQVdyLT0lQ8WTcV3bTqqspt1DgiXg5cBODut5jZAjOb7+7r\n0uNHFW6vIS9466psaiKbvpb9VEPAl4HjiRkVO6fHrizZvsiUoZN1U0h9mYl9gesL99ekY+sAsiBs\nZvsBLwH+dqzGquSIs01CW6kdlTiLSKPtwBLnbRo2s72Bi4G3uPtDY724bIlzi3xvuqF0XCXOIjJT\nrCZGwJn9SeuzA5jZfOCHwAfd/bLxGiu7eWgWiHdNbQyRlzjvRMyeUImziDRLTSfrgMuAUwHMbDGw\n2t2LlcTnACvc/dJeutVLaqJbiXOLrVMTfWxd4vwosACVOItIg9S11oS7rzSz681sJTEoPdPMTidi\n378BbwAWmdlfpJd83d1HLXArU+IM+Yh4I7E90jAqcRaRhqszR+zu7+04dGPh9pztaavs10M2Ep6b\nrjeiEmcRkVLqGKe3UImziEwF9eWI6+1Wyde12HopzLmFEmcolDhX6ZyISJ2auotz2RJniGC7MV3P\nRSXOItJwrb52z5eJtN2bh3aUOM8lP5lXLHHeOR0/q55uiohMX1XCfqtwPZsocYYocc7SFldWaF9E\npF7TbBlMyKvtIALvQel6DnkJ9F4V2hcRqVVTd3HuNRB3ljiPEEtg9hErsA2k2y3yEucF6baISDM0\nNBCXLXEeIUqZ56Tbs4g88tp0bHdU4iwiDdNqt3q+TKSyuzhvIka7s4hAPJguWQC+nxgh31VbT0VE\nqmpojriXQNytxHlVem0rXR4A7iaC8R3kS2JeUb2LIiI1mcKBuJtb2bqo4xZi3c1+4EDy6rofVeqd\niEiNmlrQUXbWxBBxgm5Our2ZWJ/zIWBheo6PtVmeiMiEm+Dcb6+qzCMe6Lj/aiIIZ7s4m5mNuWGe\niIiUL3HewrZT056crh8hP3m3toY+iojUotVq93yZSGVLnPuJUuai16Tr3YmUxSzgz2voo4hILVrt\nds+XiVT23b5UuD0EfJZYjxi2LnEed68mEZEJM8WXweysrPs4cF86fi/wXGCf9PgAkbqAfBqbiIiM\nomxl3RDwpPT62URJ8zpiStt8YoF4iOo7EZFGmMrT17pV1i0kAnAL2CPd358YCfcTeeJhYpQsItIM\nDV1rouzmoQeQz4xoAa8iNgptE6mJh4kgfHc93RQRqa7V1y2cTb6yJ+v6iCA+m3wEfDuRslhFnhu+\nvWoHRUSmu6pzNIbI88TfIoLyU8lH0ZdXbF9EpD4NXWuiysLwrfT6FjFvWCXOItJoTV0YvurmoZvJ\nd+VYTqQkBokZFGZmi2rtrYhIFa1275cJ1MuIeLTNQ/uIAJx9xRydbreJKWwjwGLg93V1VkSkiole\n8L1XdYb9l6Xr4qaix9XYvohINQ3NEdcViEeIEucRVOIsIg01lQs6YNsS5+zYZqJ6bhjYLR3fQj61\n7cmIiDTFBOd+e1W2xDl77Zx0u0XMmmiR7+oMsFfVDoqI1GYKL/rTrcQZYnbE+sJznkwE4D5gHjGr\nYu8a+igiMq2VLXGGCOLziHRFG3gwtTdALPqzgBhNi4g0QlPnEVcp6BgiVl3blQjGN6fr35FPd3us\n+0tFRCZeqz291poYIYL4PPLpav+XGD0fSr6L85e2famIyCRpaEFH2XfLZlG0SOtNuPvjxAg4C8IP\nu/uqiv0TEZn2ei1x/hURcDelEucW8ADwk3Q8szOxeeiNwDwze0atvRURqaDVbvV8GY+ZrTCzn5vZ\nSjN7bsdjLzKzX6bH/3a8tnoJxE8nAm8fMCeVOA8RJ+OWpMcws1em9vqBw9Px1/bQvojIxKipss7M\nlgGL3H0pcAZwbsdTzgVeDRwDvMTMDhurvV4C8RCRihghT0ncS17cMUJMZXtjup+tP9EPHN9D+yIi\nE6LV7uv5Mo7lwEUA7n4LsMDM5gOY2dOAte5+V1qB8pL0/FFtz6yJYfI0xErgP6Xbm9PlbCLw/oE4\niXcw45Q4z56/ZzPnkojItFRjzNkXuL5wf006ti5dryk89gARD0e1PSfrBsl3Z34MuIdYHnMdcB0w\nl5jOtgQ4Mz1X6xGLyEwwVoAfN/hvz4h4E3npctFjwOfc/XIzGya+JbYQ3wiXbEf7IiJTxWpi5JvZ\nn0jZdnvsSenYqLZnRPwI8Gi6/aXC7Uvc/Qfp9ilEEB4BvuLuN25H+yIiU8VlwKkAZrYYWO3u6wHc\n/Q5gvpkdZGazgJMZJ03bGhkZGetxERHpwsz+nlhzfZhIxz4HeNTdLzSz44BPpqd+x90/NVZbCsQi\nIpOsmYtziojMIArEIiKTTIFYxmVmp5vZpzqOfdPMdp6A9+43s2vM7J9GefzBdH2lmR2+o/sjsiPM\n2EBsZiea2Zu7HL/OzA6ahC5NKe7+2rTQ0462HzDH3f98At5rhzOzw7Od0M3sX2ts97NmdkNW3bUj\n7aifoeM9jjOzGbOxRJX1iKc0d790svswxTzVzC4hdmJZAXyIWFNkd+DLxB6Fw8AZ7n57je+7AjjY\nzL6S3mt3onz+7e5+Q+eTzWw34KvF5xHTKm929/PN7H8Dg+7+VjP7U2LZ1m8DnyWmXa4HTk+v/xqw\nIT32zNTOMHCxu3+i6g/m7q+o2kbBHwOL3X1djW2Oq+afoei/AJ8iqtKmvcYGYjM7HTjc3d9tZrsA\n/w/4Ah2/DGZ2LPAJYv7yXcBfAs8H3k3sn/cud79+nPbPBZYSlYLzgPPNbCPx7/N6onz7QKKo5Q3A\ni4GTiEncrwVeCZyW+nWRu59jZrsCXyEWR5oFvM3dbzKzPwBfJErE5wAvyuYfdunjlcC/Ay8EFqbX\nrAb+CTgg9fXD7v799NwrUt+G03NOJ8rSlxOVj9v0Z5z/DUWHAouB+cTqelm5+0eBf0xB7lTgw0Cd\no9d3ARcAtwOb3f2TZraECNDLujz/LOAXHc/7OBGozicm2meVTscA3wI+A/yVu//ezN5CTEX6F2I6\n0lPc/aH0RbAf8Xl4k5ktJYLzB4A/BQ4DXkdUlnZ+Fg4ggv1m4t8OiLSKuy80sxcBHyMKph4GXkN8\nht+a2nkGcIG7f6TbP5CZvYf4LF5sZicDn6fw+SB+L1a4+wvT8y9MP8uc1PYd6fbQJP4MB5F/8X2O\n+J16ppm9mvhdXkIsPPYFd/9qtzamsqmWmng38cvzfOJ/NsQqR69IH7L7gT9Jx48AXtotCBelVZGe\nD/wR8D5gEfBTdz+B+KX+M+A+dz+GKGR5eXrpU4g5hLOJid0vSPdfbWZPAd4BXOruy4E3A+ek180C\nfuvuxxHBZczFQIB1qY0fEl9CewCXufsy4sNe/GDf6+4vID6we7j7sen2EWP0p1c/dfct7v4QUda+\nZzq+BLgy3b6CCF47whPv4+7XAYdsx/NWAovNbAHR941mNpf4YrkGeB7wpfRl9mfAPqmtW9PPC/Fl\n8CMi+LeIz8HZxGfmVen2++n+WXg78M20hGy3CqsFwGnp/+k64KXp+POIL9OlwNtG+4dx938A7iMG\nB3Po+Hykwqr9zWz39JLnEL/7fcRfNgNEEP3UZP0MhX69zt0vJpbffSMRmF/m7s9Pfeof4/VTVmNH\nxKPIfhm+DvyLme1DBM7vmhnECOBBYh2MG919cw9tHgZck1ZJusvM7gROSe1dQIw0fgzg7t+EJ0bT\n17r7iJk9L/XhitTersBBRHBEXLcBAAAEFklEQVTfy8xen47PLbzn1en6bmC3cfpXfO6exBfQc83s\nTcRIY8/Cc3+Zru8l1pCG+HLabZz+9KJzwnlx9b1shJmlJ3aE4vvA6HspbvM8d3/MzIaIRal+Qfzs\ny4EN7r45/fVzgrs/8TOmEdoTJf3u/mYzezrx19fB6f3vBW5y9yEzux94FhEoOj8LhxGjSYgviZM6\n+rwG+D+pCutpwOVEiuQGd9+Y+jPav0un0T4fFwMnmtlK4q/HlcRn4y+JKtkfAM+d5J+h+MUHgLuv\nNbPfpVz0t4F/HvdfYApqciAu/uL3w1a/DK8hPgwvBe5J39JPMLPj6b4uRjcttg4eW4BXEOswZymJ\nK7q8bqBw/QN3/6uOPryL+PP/511eO9jx/mPpfO5pxKj42HR93SjP7XzdwBj96cVSM+tL7zkPWJuO\nXwucAHyDGC1e1/3llWXv8wszO5pIVW3P864hUg7vIfr/IWJjA4g/tU8EfmhmryWCyq1ZgynvfJa7\nf9TMvgccRaRoYOt/5z2IUWPnZ+G/kX/Guv0V+mVi1HeLmX22cHywy3PHM9rn47tEmmBhOjbo7meb\n2X3EX0iXEymcyfwZuv7OuvtJqYz4NCI1+JLtaHNKaHJqYh2Rx4L4k2Q3M/uQu//W3T9KBIIheCK9\ngJm9zcyetZ3v48BRZtYyswOJ0Y65+0XAB4kvhCy3drKZvb/j9dcDJ5jZ3NTGp9O0rmuIPBdmdpiZ\nvXM7+zWahcDtaQR/CjEK7UXV/vyWGJH8mMiLZl+UHwLeYGaXE3+C/vftbLdXnyb+P10O/D2RNtqe\n511FpJ9uIv6fLSNPqZwFvN/MriJ+hl8V2sPdHyX+mvgl8F+JdQXWsq3RPgtOpEwgviQ67QbcmVIH\nJ9D7/9NuRvt8/IIY1b6M+LLCzP6O+D36DfBzYO+G/AwQQX+WxXoNb3f3G9z93Wz9F+C00eQR8Y+B\nD6S83Q+IP7myX4YNwMr0Z8sZwFfMbIDIXZ1H5KN6kk6g3Ux8EH9HBJyPmNlZRKB/FfCu9Eu6hTgR\n9eLC6+80s/9Fvm3URe7+uJl9BviqmV1N/Bn79gr/FkXfAb6XRntfBu42sw/18LrS/UknR77acfhr\n6XoD2/6ZWpu0gEoWAE7t8vjCdH184XC3511K/JkN8DiRS80eu4UYQRatLbwv7v42yE/yjtLdO4l0\nVudn4dPAt8zsFOKLoNPngJ8Rn7//QZxg6/zC71XXz0caza8k8rBriZTbeiIvPI/4HbutIT8DxBfn\nBcQuF89Pf6lsTj/TtKO1JkREJlmTR8S1MLPPE3+SdTppggoSxpTOSHc7AXGVu++oP/NlCkon4E7r\n8tD7KuT+J9R0+Bl2BI2IRUQmWZNP1omIzAgKxCIik0yBWERkkikQi4hMsv8AQGyjEdUFd4kAAAAA\nSUVORK5CYII=\n", 713 | "text/plain": [ 714 | "
" 715 | ] 716 | }, 717 | "metadata": { 718 | "tags": [] 719 | } 720 | } 721 | ] 722 | }, 723 | { 724 | "metadata": { 725 | "id": "naaOwkn_bVBW", 726 | "colab_type": "text" 727 | }, 728 | "cell_type": "markdown", 729 | "source": [ 730 | "Since there are nulls, we'll remove those and be left with a clean dataset. Creating a heatmap again confirms this. " 731 | ] 732 | }, 733 | { 734 | "metadata": { 735 | "id": "nWKLf31ubZ2g", 736 | "colab_type": "code", 737 | "outputId": "7c5f1b85-0eaa-4cb4-ab68-b5b6691e5064", 738 | "colab": { 739 | "base_uri": "https://localhost:8080/", 740 | "height": 286 741 | } 742 | }, 743 | "cell_type": "code", 744 | "source": [ 745 | "influencers_df = influencers_df.loc[~influencers_df['median_favs'].isnull()]\n", 746 | "sns.heatmap(influencers_df.isnull())" 747 | ], 748 | "execution_count": 22, 749 | "outputs": [ 750 | { 751 | "output_type": "execute_result", 752 | "data": { 753 | "text/plain": [ 754 | "" 755 | ] 756 | }, 757 | "metadata": { 758 | "tags": [] 759 | }, 760 | "execution_count": 22 761 | }, 762 | { 763 | "output_type": "display_data", 764 | "data": { 765 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcZHV57/FPVc8+w8AMKJsKIvjo\niF5kEVkUEFRU1CBqEkwMiSa5uGFcEmLyIqBXjFEvSoi5boheEx1DXCObioBmhCAaxEQeQEWQYR8Y\nZl+6K388v8M5XVM9XX3OafpM1ffNq15VdZbf+dXQ/dSvn/NbWp1OBxER2XG0Z7oCIiIyNQrcIiI7\nGAVuEZEdjAK3iMgORoFbRGQHo8AtIrKDmVV3gWZ2HvBcoAOc4e7X130NEZFhVmuL28yOAQ4GHg98\nFzi/zvJFRKT+VMmJwIHAIuD1wBPNbHHN1xARGWp1p0qeCfwa+AawHngvsAfwSK+DL3v3xzVsU0T6\ncuKH3tSqWsaz9jmm75jz019fXfl606XuwH0fsBQ4rVC+boCKiNSo7qC6E3A38AXgcan8u2q+hohI\nKa1Wq+9Hk9Xd4r4KOI/Idc8Hxtx9Tc3XEBEppdUajARA3YH7v4jc9s7p/aU1ly8iUlqbZrek+1X3\n189biKC9GRgFTjKzpTVfQ0SkFKVKentWem6nRwfYB1hV83VERKasrVRJT3sQwbqVHtk2EZEZ1/SW\ndL/q/vr5KTAG/IxIlwDcXPM1RESGWqUWt5kdCHwdOM/dLwAWpF0Hkre4t1a5hohIXUZaIzNdhVqU\nbnGb2ULgQqK/9vPT5oXASHpkoyW3VKmgiEhddHMygnMLmAvsn7btQwTq2cR8JR00clJEGqLd8IDc\nrzqC6gjR6ob4IlhP5LnvIgL7PTVcQ0REkjp6lbSJ1jVE3+01RK57H2Lk5FgN1xARqaw1IAmAOgJ3\nli6B6EmyiEiVQEw6JSLSCCPtwQjcdX2K7Avg7q4yP1BT+SIilbWm8F+TVWlxHwQckJVjZleRt7hH\n0/bDKpQvIiI9VAncG4HbgEMK25al56zV/VsVyhcRqZWGvEce+6mF93Ngm78v5lcoX0SkVnX2z97e\nwuhmdgJwLpF9uMTd32dmi4DPA0uI+4LnuPvlZa5d5eunewjSZqIP9yaiZwnAugrli4jUqt1q9f3Y\nnrQw+gHufgTwBrZdGP184BTgKOBFZraMWBnM3f044NXAx0p/jrInAv+bfIg7xDfIbOABojfJWMXy\nRURqVePNyeOBrwG4+8+BJdnC6Ga2H7DK3e9M3aEvScc/AOyazl+S3pdSKrCa2XHA84jBNhApkn2J\nFvfewFOIwL2ybMVEROrWbrX7fkxiD+D+wvv7yWdC7d53H7Cnu38JeJKZ3QZcA7yr9OeocF6LyO1k\ndiOfg3tr1z4RkRk3jXOVbO+EFoCZ/R5wh7vvD7wAuKDcp6iWytgM/Dd517/iCMkseC/qPklEZKbU\nleMmsgnFtQb2Isax9Nq3d9p2FHA5gLvfCOxlZqWmK6wSuO8lAnN24dnAL4n0yc/Te63wLiKNUWOO\n+wriBiNmdjCwMlsY3d1vBxab2b5mNgs4KR1/G3B4OmcfYK27j/Yoe1JlA/dVwAbyftsQvUlWAPPI\nlzD7cMnyRURqV1eqxN1XADeY2QqiB8mbzew0Mzs5HXI68EXg+8Byd78F+ASwr5ldDfwz0cGjlLL9\nuF9C3JzMdNLjleQt8J+5+5fLVkxEpG51Tuvq7md2bbqxsO8a4Iiu49cCr63j2mVb3CeSB/0xIvl+\nKbBT2tYBvlWtaiIi9RqUuUqq5Lg7xI3JbIWblxGt7Wyx4L8ws09Wq56ISH1q7A44o8rW7jbyXiRz\n0vNcoqdJtr0DLC9fNRER6aVs4F5PtLazebfXpuc55DnuFpodUEQaZNjXnHyACNAbyG9MQswY6ESv\nkjEqDOkUEanbSMNTIP0qG7hPJgL3HGKgTTZnyY+Ao9Pr2cBFVSonIlKnYV8seCvRonby4D8KHJn2\nbUrv31S1giIiMl7ZwH0DcSNyGdHy3kjkudtEIB9J+w+toY4iIrUY9hz3amKu7XlEfntuetxFTFuY\nbb+1hjqKiNRiUFIlZQO3ATsTfbjHiKDdSeXNKxz395VqJyJSo6YPrOlX2cCdDW0fY/wK748nH4Dz\nSXd/uHINRURqMigt7rI57p8ROeyihUQwz+bpfmr3SSIiM2nYc9y3ED1HiucX595uofUmRaRhBqXF\nXTZwH0Z0+1tH9OHOepJk85eMEGusiYg0xrDnuDMtxi/Z8xDRNXAvYH7FskVEajUoLe6yOe5biSAN\n+dwkHWJa1z3T+00V6iUiUrthz3Fn5hZeZ/Nyt4g0SrM/uYgMnUFpcVcJ3HPIFwqGPMdNeh6M2VxE\nZGA0vSXdryqB+3Fd7zvAGiK3vZHxA3FERGbcoNycLNsq3p08tw2RJhkD7k/vd2J8a1xERGpStsX9\nZCJwF8/PepNkiys8WKFeIiK1aw9Gg7t04O4QNyCzCaYAHiZa2lk3wL+rVjURkXqNtAfj1lvVXiVF\nW4jRk1mKRMuWiUijDMrNyapfPxsLr59EdAEcSY9jKpYtIiI9VGlxjzF+kE33l8DeFcoWEalde0B6\nlZQN3IuJfty7pfctIpCPEgsIL2Z8a1xEZMYNSqqkbODeRLSwt6bnsfTcIYJ2h/HdBUVEZtygjJws\nm+N+ID3fQLSyH0nvZxFBWyMnRaRxWq3+H01WtsV9SOF5hOgGmKVLsha4Wtwi0ijD3uLunvkvC9In\nAfeioC0iDdSawn9NVjZwX0akSC4F1pJPLvVRYt1JiDUoRUQaY9indd2PCPpHEiMn28BdwAHk07l+\npXLtRERqNOypkmxR4J3J0yK/KOzvkN+wFBFphEG5OVm158c68v7azyNf4b0FvKNi2SIitWq3Wn0/\nmqxsqiSbSGoh44N1UbM/uYgMnabfdOxX2cA9u/A6+5fYWng/AtxctlIiItOhzpa0mZ0HPJdouJ7h\n7tcX9p0AnEt04rjE3d9X2Dcf+BnwPne/qMy1y6ZKbkzP2b9CNuDmQeC+tO2AkmWLiEyLunLcZnYM\ncIC7HwG8ATi/65DzgVOAo4AXmdmywr6/BlZV+RxlA/fORLBeTz78vQ0sIVZ530C9U8aKiDTJ8cDX\nANz958ASM1sMYGb7Aavc/U53HwMuScdjZk8DlgHfqnLxsoF7Eflq7pkszz2ayh2MZJKIDIyRdrvv\nxyT2IF+qkfR6jwn23Uc0aAE+Qg0dN6q0uEeJCaXmFra3ify2AreINM40dgfc3hktADN7PfBDd/9V\nyeo/qkqvkhG2XRB4HVrlXUQaqsabkyvJW9gQ6+3ePcG+vdO2lwH7mdlJwBOATWb2G3f/zlQvXjZw\nZ0E5y2/PI4L4A8ATiUmn1pYsW0Sk6a4AzgE+YWYHAyvdfQ2Au99uZovNbF/gN8QcTq9z9wuyk83s\nbOD2MkEbygfuLCjPJe/HvZL4Fsm6CnrJskVEpkVd/bjdfYWZ3WBmK4jZUN9sZqcBq939q8DpwBfT\n4cvd/ZZaLpyUDdzZea3C83XAKwvHvLRspUREpkOdk0e5+5ldm24s7LsGOGI7555d5dplA/dO5DMC\nkl6/nFjOLHMzsLRk+SIitRtpD0afiaqTTGVzlbSIFMkW4GEikFe+cyoiItuqOkhmYeH1SHrskt4f\nVLFsEZFaNX2e7X5VDdxriPTIfKJXySpiUM6eaJV3EWmYAcmU1DI7YJZuyYa8j5AvGCwi0hjD3uLe\nQIyeLOoUnsdQ4BaRhhmQuF06cN9HjAzaQJ7nzoa5F+crERFpjKYvkNCvssH1ZiKXnfUugWhhryaC\n9hbU4haRhhn2Vd6vJ4LzHMYvpHBver+AaI2LiDTGoKw5WTZV8jAxBv+p6f1WooX9RPIvAw15F5FG\nGfZUCcRqNxCpkWxRheINyvdXKFtERCZQKnC7+2eImbEgRk/+ExHA5xE9SsaAQ+uooIhIXdrtVt+P\nJqvS4n5xep4PPB3YlchvZ4spvMfM3lOteiIi9Wm1Wn0/mqyOdSHXEgNvYHxPkjnAl2ooX0SkFg1v\nSPetVIvbzN5AWvwSuBPYn0iPtIFHiAC+EXhODXUUEZGCsqmS5cQ8JRALB28h0iSbgFvJB+E8ULWC\nIiJ1GfZUyZ8A+xDB+Unkq72PAYek16Nll+UREZkOwz4f90HE9K3ryKdzHSOffGoUaJnZKyrXUESk\nJoPS4q7Sq2QeMUKyOHIyuzm5OZWtLoEiIjUrG7j3YXwPkllEnvue9LyBWEj41kq1ExGp0aAMeS8b\nuHcjT5F0yGcF/GDathR4CHUHFJEGGZRUSdmbk48nAnW2YEKW4/4I+ZfBOe6+pXINRURq0vB43Ley\nLe5sNfc15D1KbiUCOEQwX1ehXiIitWu3Wn0/mqxs4N5MtLh3InLZAM9Oz1nq5IPVqiYiUq9ByXGX\nTZWsJfLcRdmgG4iWt25MikijND133a+ygXveBNuz5csA9i1ZtojItBiQuF06cC8gUiKbgdnkue3b\niFkCl6Kly0SkYYa9xT1K3rIu/kvsRd5FcKcK9RIRkQlUCdwdxq85CRG055EvJCwi0hiDMldJ2cCd\n5bLXEME7y3lnPUw6FcoWEZkWA5IpKR1cx4jgvJDxuez1qcwO1eZBERGp3aDkuMsG14fT8zpibhKI\nYH5XKnMeceNSRERqVrbF/RCRKllY2HYf8GSilwmoH7eINMyANLhLt7iz5cmy+UoAfsr4tMlLK9RL\nRKR2g7LKe9kW96L0vI4I1ouBI1J5o8QXwg+Ap1atoIhIXerMcZvZecBziRh4hrtfX9h3AnAuEQ8v\ncff3TXbOVFS5gZilSrJ0yaK0bSQ9L65QtohIY5nZMcAB7n4E8Abg/K5DzgdOAY4CXmRmy/o4p29V\nAneH6EWyPr1vETcos/lKHlehbBGR2tU4ydTxwNcA3P3nwBIzWwxgZvsBq9z9TncfAy5Jx094zlT1\nFbjN7EAz+4WZvSVtepgI1PPJ+3BvStvH0vuNZSokIjJdalxIYQ/g/sL7+9O2XvvuA/ac5JwpmTRw\nm9lC4EKiBf38tHk38huR2QjJzcQCwlk/7tlmtrRMpUREpsM0Tuu6vTMm2lc64d5PizvLWc8F9k/b\nlpG3rDObUnnFBYOPKlsxEZG6jbRbfT8msZLxreW9gLsn2Ld32ra9c6ZkKjnuEfK89Twil72BPKed\nDXcfJYL6HODgMpUSEWm4K4BXA5jZwcBKd18D4O63A4vNbF8zmwWclI6f8Jypmkp3wDZ5N8Di11H2\nOqvAecDbidkBdy9TKRGR6VBXd0B3X2FmN5jZCqKh+mYzOw1Y7e5fBU4HvpgOX+7utwC3dJ9T9vpT\nCdxZuiSzgfEzAJ4LXAD8eTquBVxTtmIiInWrc+Sku5/ZtenGwr5riLEtk51TylS7AxYD/TrgP8kX\nC76Q+BaZRQTtrcClVSsoIlKXVrvV96PJ+gncBwEHpNezzOyq9HrczUl335Betsj7du9VtYIiInUZ\nlMWC+wncG4klybotBZ6VyuiY2TsL5S0gWt7vrqOSIiJ1qLEf94zqJ8c9m95zjowwfqWbrI938cvg\nKeWrJiJSr4bH47712497Mh1gOZE+2UCe915Rsl4iIrUblBZ3lblKNhBd/rIFExam8pw8cI/2OE9E\nZEYMSo67yrqQWe+RtcRgG0/bDyKfp/veSrUTEZFtVAncWUv7HmAJ8LTCvlEixaL5uEWkMVrtwVgK\nt59P8Z/kremt7n5ser0yPT9C9Dw5iLyLYDbs/Zn1VFNEpLpBSZWU7ce9kXyylP2AW4gBOZuB1cC/\npbI3ICLSEMM0AKdXP+5vkK/uvhn4JHAZ0dJeCbwi7ft8DXUUEZGCsv24VxCrOUCkUS5MZS0kct3Z\n9K6X1VNNEZHqmp4C6VfZftzHFs7dA3h/GvK+Pm0fI1rku9RQRxGRWqgfdxczO4kY6r6FaG3PAc6u\nq3wRkara7Vbfjyars2/MssLr7FP/pMbyRUSEav24ux1ItLSzgTnZNhGRRmh4BqRvZftxX0t0+4NY\na/L/At8mAvZN5Cu831dbTUVEKhqUHHc/Le5e/bh3IZ9ze0/g6+TL9OxTOPcJNdRRRKQegzFwsnQ/\n7ivJF0nYlZhsKgvkC8knmZpdtYIiInUZphZ3r37cLyQPynPT6wOJwThziOA9SgR0EZFGaHg87ls/\ngbtXP+5iS71DTDL1IBHAR4n89xLyld9FRGZc01vS/aqS8clWfG8TAftWYuCNA4uIG5W/rFQ7EZEa\nDcokU2W7A+5ceJ19xK8SrfP9iXQJxBwmIiLN0PSI3KeyLe7FRMDOpnFtpSHv68hz3w+5+68r1k9E\nRLqU7cftRC57Q3peZWaHkN+UXAssMbPfrb3GIiIltUdafT+arGw/7mcwPugvAZ6YXm8E5hM3LZ9e\nTzVFRKobppuTvfpxP0Lks+enMtYAf5j2LUz7WsAx9VRTRKS6Qbk52U/g7tWPe3HX+y3AciLnvSE9\nd4h5u0VEpEZ19OMeI3qZLEzb7yUC+65EvltEpBma3pTuU9nugNmn30z05x4hv4H5ZKK13SGCuIhI\nIzR9Lcl+lQ3cHaI1neWy1xNLlmXG0r7uFIuIyIwZlMBdth/3KPmISYgbmAeR9+seTa+fWal2IiKy\njbL9uH9FBOet5Dck1xGpkxcQ83C303YRkUYYpl4lBwGWXs9O/bgtnZs99idWdM9Wds/m4f58nZUV\nEami1W71/WiyfnPcWQqkk55HgHuIBRXmpfcriNRJu3DO9+qppohIdYMyAKefwJ0NwDmksO1BYrTk\nHOLm5Abg6LQvm8OkBRwF/GtdlRURqWQa47aZzQYuIlYBGwX+0N1/2XXM64C3EzHyk+7+mcK+3YGb\ngZPd/artXavsAJydGP9P0AaeQnwRjKX3LeB/9VG+iMhjYppXwDkVeNjdjwbeD3yguNPMFgJnAScA\nxwJ/ZmZLC4d8iD6nwu4ncPcagAMRoNcQ6ZMR4LfT9hZ5SuUF/VRCROSxMM2B+3hiemuA7xAZh6LD\ngevdfXWaTfXfs2PM7AVEPL2pnwuV7Q7YAR4g5ippEUPeP5S2bywcc1HJ8kVE6teewmPq9gDuB3D3\nMaBjZnN67U/uA/ZMx/wN8Ff9XqjsAJw707lZP+57iKlcIfLeHSKgzy9ZvohI7eq6OWlmbwTe2LX5\n8O7LTVad9Hwm8Cl3f9jMtnf8o8oG7k8Bf1543yFGTraItEmW59bISREZOO7+aeDTxW1mdhHRqr4x\n3ahsufvmwiEr0/7M3sC1wB8AI2b2FuJe4XPM7DXu/l8TXb+fwJ0NwDmUNADHzH5DDHNfR9xBXUo+\ncrJF3FFtoZGTItIg09w/+wrgNcDlwMvZtjv0dcCnzWwXYvDiUcDb3f3fsgNS8L9oe0Ebyi+ksBsR\nmLem7WvIR05uBK4EXoVGTopIg0xz4F4OvNDMfgBsAk4DMLMzgavd/Yfp9eVEluIcd19d5kJl+nHP\nIV8MODObGDF5OvHnwCvSdo2cFJHmmMYBOO4+Sr6gTHH73xZeXwxcvJ0yTuvnWmX7cW8m0iJZ4L+d\nGDm5kMh1zyIf/i4iIjUq2497lOgOmKVKtqR+ietTmWNEF8Fd6qikiEgdhmmSqV4eIla9eZSZnQQs\nIAJ2h0innF2lciIidZrmATiPmTKBezORGrkHuBvA3Y8BlhWOyT71T6pUTkSkTq2Rdt+PJivTj3sO\n8GwiOLcAzOxq4NdES3sWeeA+sIY6iohIwZQXUiD6Kc4mAngW+JcC3yYC9k3kw97vq62mIiJVtabw\naLB+F1J4tB838C/p9VbgF+n1KmJVd4gBOZknICLSEMOU4876cXefN4t8+OZSokcJwKLCcbMREWmI\nYVoBp1c/7mzq1gXp/Qj5kPeRtK8D7FlPNUVEqmu1m33TsV9l+3F3iL7cWSt7d/Ih7/cDXyFfGUdE\nRGpUtjtgtrr7aNp2LzFKch4xevLVafv/r1pBEZHaDMjNybLTus4iHzUJkdfOWtfZ4gowftJwEZEZ\n1fTcdb/KJnyyeUoWEa3undx9BXkLfDORTvlF79NFRGbAgIx5n3I/bnc/Np23lVj1ZhSYb2avJ/Lh\na8hvXh5fd4VFRMoapu6Avebj7hT2ZzcqTy6UOSs9bzPFoYjIjGm3+n80WNl+3FuJ1vVO5H21PwA8\nQqRPslVwrqmnmiIi1Q1Ti7tXP+5saTLS8yxgP2AxcUPyv4jArgE4ItIcQ9SrpFc/7jYRvDcQg3A6\nxOyAo8DjiKXNRonFMEVEGqHpLel+le1VMpv4TlqQnjcSgToL6A+l152JChARkXLK9uPOVrnpEIF7\nDbCaCNq3APun7Q/XUEcRkXo0/KZjv8oG7qIOMejmP4m0yv7kiwlfXUP5IiK1GKa5Snr144boWbKe\naHkvAK4jgni7sP/S2moqIlLRMPUqOQiw9Hp26scN0VpfQH7zclkqb4R8ZOVLa6upiIgA/adKxtJz\n8WZj1qtkYXr/xPS8jkiddICnV62giEhtBiTHXXYADsTNx4XkPUmyUZILiVZ3CzimhjqKiNRimFIl\nvQbgdJ87AiwnWuYbyHucrKhaQRGRugzKKu9lF1KACMzr0vNs8tb3vUQ/7mzYu4iI1Khsd8BsabKF\nRIDeQj739pML+2+vWD8Rkfo0PAXSrzr6cWeB+6DCttFUtlZ5F5HGaHruul9lA/cYcdNyLpFKmQUc\nQgTsEfLZA9WrRESaY0ACd9kBOBuJXHaLvKvgZmJAzh+R57hX1llZEZEqWu1W348mK7uQwseIgJ0N\ntrkc+FZ6/UHg8en4i2qsq4hINUO0dFmvftzPIgbcZDPXrgNuJubjfhwR0EeBm2qrqYhIVQMSuPvJ\ncU+0kELWV3s9serNSiJdMidt30z0OllTV2VFRKoYppuTE/XjzsxLz+cSNys3pXPmA6cCnyhdOxGR\nOk1j7trMZhPp4X1Io8nd/Zddx7wOeDvR8P2ku3/GzPYCLiTv7PFn7n7D9q5VdXjQGPkiPwen5znk\nXwj7VSxfRGRHcSrwsLsfDbyfWIf3UWa2EDgLOAE4FvgzM1sKvAP4qrsfB5yZzt2uKoF7a9f5l6Zt\nxdGSH6tQvohIrVqtdt+PEo4Hvppefwc4qmv/4cD17r7a3TcA/56OeQDYNR2zJL3frrKB+1NEkIYI\n1BcQLe0RYqHgjWnfYSXLFxGpXavd7vtRwh6kEeTuPgZ0zGxOr/3JfcCewHnAb5vZzURsPWuyC/WT\n4876cR9K6sedLpBN8ToGPB/YPb0/sLBvV0REmqKmHLeZvRF4Y9fmw7veT3axbP+7gS+7+/vN7CTg\nw8CrtndiP4G7Vz/uvYBHiKB9KxGsV5NPLLWOaPLP6y5MRGRH5+6fBj5d3GZmFxGt6hvTjcqWu28u\nHLIy7c/sDVwLnAz8ddr2beDjk12/bD/uO4jgPUKsjvP99H4VkTLZhQjquyMi0hDTPB/3FcBr0uuX\nA9/r2n8dcJiZ7WJmi4j89veJ+Jq11g8jGsPbVbYf935E67pDBO8TiW6Ai4n+2w8RQfs3fZQvIvLY\nmN5+3MuBF5rZD4h4eBqAmZ0JXO3uP0yvLydi5znuvtrMzgU+Y2avTeW8bbILle3HPT9VbBWRxz4c\n+BzRyv41MbUrwK/6KF9E5DHRGplsWEp57l5cCay4/W8Lry8GLu7afzdTXJ+3SnfAOUTQnpXK+TLR\nOn8yebC/skL5IiLSQ5X5uFvkQXsTkXh/ENgt7ffUJUZEpBkGZMh7XQurzQFOIYJ2h+hZYmZ2Yk3l\ni4hUNkyLBfeajxsiOG8iAvVcYrZAgIeJwTlbiRy4iEgztNr9Pxqs7HzcML5zeQfI7ojuRATyWcAf\n1FBHEZFaDNNCCr36cWfnziVfc/LjjG+FQ/RrFBFphiGfj7uXXdJzcbTk1l4HiojMhKbnrvtVZT7u\nLcQiCovTMVvIV75ZS6RMNk5wrojIY6/huet+VfkUs4ngnHkK+TqUC4h0yQsqlC8iUq92q/9Hg1Wd\njztblmwO+arvWXfAFjFloYiI1KjKAJw2saZkNl/JbUQgv4M8J97sry0RGSqDkuOu0o97C5HLzpYv\nu53Id88nz4tfXldFRUSqarVH+n40Wdn5uMfIg36baHVfkZ73JQL3ZmI5MxGRZhiim5O9+nGPEUH/\n0ZuTaQ21LanMbI6SpTXUUURECvoJ3LOJHiOQD6zZQtycXE8E6TEzO4a4SbkhHTcb+ONaaysiUsEw\njZzcAtxEBOOxwrYt6fUmohdJFqSz0ZMttl3lWERk5gzRyMlRUqs6vYZYPn4BscLNU4mW998TgXoR\n0dtkDXBRvdUVESmv6Tcd+zWV7oBbyVvZ/0GsqWZE6/pOYB9iubInAs8BvkEE9Amd+KE3NftrTUQG\nypzFuw5EzJlK4N5I9BQB+CiR924R/bbvB64H7gEuI1rnN5DnxEVEpCaT5rjd/YbUd/ujwHvT65cS\nC10eRrTCv+nuvyIC94uBI4i0ivcsVERESmt1OttvFJvZIcBHiP7ZW4C7gL8gctot4Pvu/o507OHA\n+URL+zJ3P3u6Ki4iMqwmDdwiItIsgzGMSERkiChwi4jsYBS4ZVJmdpqZfbhr25fMbP5jcO3ZZnad\nmX1ugv0PpOerzOzA6a6PSBMMbeA2sxPN7PQe239kZvvOQJV2KO7+O2l+mum2JzDX3Qdi4WkzOzBb\ncNvMvl5juReY2Y/NbHFdZW7nWtPyGbqu8Xwze/x0lD0IqszHvUNz98tmug47mCeb2SXEAKvzgLOA\nA4m1Ri8k5qkZA96QuobW5TzgKWb22XStXYh5cN7m7j/uPtjMdiZG7D56HPAq4CZ3X25m/4+Ynvgt\nZva7xMjffwEuIHpDrQFOS+d/gZi6+ALgGamcMaL767lVP5i7v7JqGQUvBQ5290dqLHNSNX+Goj8C\nPgzcN03l79AaG7jN7DTgQHd/l5ktAn4G/CNdvzxm9jzgXKKr4p3EnClHAu8iht+/091vmKT884m+\n504M119uZuuJf5/fAz5AjAzdCLweeCHwEmAv4HeA3wJOTfX6mrt/xMx2Aj4LLEnlvNXdf2pmtwGf\nIEaezgVOcPdsJaHuOl4FfJt5o2TfAAAGpklEQVRYAm63dM5K4HPAE1Jdz3b3f0vHfi/VbSwdcxrR\nn/54YoqCbeozyf+GoqcCBxNzrt9IPv3Be4HPpKD4auBsoM7W8TuBi4FfAZvc/YNmdigR0I/pcfwZ\nwLVdx/0fIrAtB/YgX+DjKODLRNfWP3X3W83sTcCbgX8Cng08yd0fTF8cexI/D39iZkcQwfyvgN8F\nlgGvAw5l25+FJxBfDpuIfzsg0jzuvpuZnQC8jxjg9hDwWuJn+C2pnKcDF7v7Ob3+gczs3cTP4jfN\n7CTg4xR+Pojfi/Pc/QXp+K+mzzI3lX17ej06g59hX/Ivyn8gfqeeYWanEL/LhxLTRf+ju1/Uq4xh\nsqOlSt5F/LIdSfxwQPQbf2X6obwXeE3a/kzgxb2CdpGZLUvlHQ78JTH3+A/c/TgiCPw+cI+7HwV8\nCnhFOvVJwPOJluargaPT+1PM7EnA24m+7McDpxN94SGC5s3u/nwiGB0/yWd+JJVxKfGltRS4wt2P\nIX45ir8Id7v70cQP+FJ3f156/czt1KdfP3D3Le7+IPAIsGvafihwVXr9PSLYTYdHr+PuPwL2n8Jx\nK4CDzWwJUff1ZraA+CK6jpii4VPpy+/3iakbAH6RPi/El8d3iC+LFvFz8AHiZ+bk9Po99P5ZeBvw\npTR4bWWPOi8BTk3/Tx8hBrGR6nUa0ah460T/MO7+IWLw20uIADzu58PdbwT2MrNd0inPJn73R4i/\nnDYTQffDM/UZCvV6nbt/k1jA5Q+JQP4ydz8y1Wn2JGUMhca2uCeQ/fL8M/BPZrY7EWi/YmYQLYwH\niEFCN7r7pj7KXAZc5+5jwJ1mdgfwqlTexURL5rsA7v4leLS1fr27d8zsOakO30vl7UQMVjoSeJyZ\n/V7avqBwze+n598AO09Sv+KxuxJfWIeZ2Z8QLZldC8f+R3q+G/hJen1vusb26tOP7g7/ncJz1oLN\n0iXToXgdyFdZmvQ4d19nZqPAscC1xGc/Hljr7pvSX1fHufujnzG1ALMpHnD3083sacRfd09J178b\n+Km7j5rZvcCziMDS/bOwjGitQnypvKSrzvcDnzazWcB+wJVEyubH7r4+1Weif5duE/18fBM40cxW\nEH+driB+Nv4YWA18Czhshj9D8YsSAHdfZWa3pFz6vwCfn/RfYAg0OXAXA8VsGPfL81rih+fFwF2F\n5dQAMLNjKfzSTaLF+GCzBXgl8DTyFMn3epy3ufD8LXf/0646vJNIR/ywx7lbu66/Pd3Hnkq0up+X\nnn80wbHd523eTn36cYSZjaRrLgRWpe3XA8cBXyRaoz/qfXpl2XWuNbPnEqmzqRx3HZECeTdR/7OA\na9K+G4ETgUvN7HeIIPSLrMCUNz/D3d9rZt8ADiFSRjD+33kp0Srt/ln4C/KfsV5/5V5ItCp/bmYX\nFLZv7XHsZCb6+fgKkbbYLW3b6u4fMLN7iL/AriRSSjP5GXr+zrr7S8zs4PTZXg+8aAplDqQmp0oe\nIV8l/mhgZzM7y91vdvf3EoFjFB5Nd2BmbzWzZ03xOg4cYmYtM9uHaE2Zu38N+GviCyTLDZ5kZu/p\nOv8G4DgzW5DK+FjqJncdkafDzJaZ2TumWK+J7Ab8Kv2F8CqilduPqvW5mWjxfJfI62ZfrGcBrzez\nK4k/if9miuX262PE/6crgb8l0lhTOe5qIh32U+L/2THkKZ4zgPeY2dXEZ/hJoTzcfTXx18p/AH8O\nrHT3VWxrop8FJ1I4EF8q3XYG7kipjOPo//9pLxP9fFxLtJpfRny5YWbvJ36P/hv4IfD4hnwGSKts\nmdm+ZvY2d/+xu7+L8X9hDq0mt7i/C/xVyjt+i/gTMPvlWQusSH9GvQH4rJltJnJvnyTyaX1JNwxv\nIn5wbyEC1DlmdgbxxXAy8M70S72FuPH2wsL5d5jZR4nW2yhxM2eDmf09cJGZfZ/4s/ptFf4tiv4V\n+EZqTV4I/MbMzurjvNL1STeDLura/IX0vJZt/2yujbvfTh4wXt1j/27p+djC5l7HXUa+1N4GIhec\n7fs50UItWlW4Lu7+Vshvak9Q3TuI9Fr3z8LHgC+b2auIL45u/wD8O/Hz93fEDcXuBkK/ev58pL8W\nVhB55FVECnANkddeSPyO/bIhnwHii/Zi4BTgyPSX0Kb0mYae5ioREdnBNLnFXQsz+zjxJ2K3lzxG\nA0i2K92x73XD5Wp3n660g+yA0g3HU3vs+ssK9y4eU4PwGZpALW4RkR1Mk29OiohIDwrcIiI7GAVu\nEZEdjAK3iMgORoFbRGQH8z+EHRMJnmugQgAAAABJRU5ErkJggg==\n", 766 | "text/plain": [ 767 | "
" 768 | ] 769 | }, 770 | "metadata": { 771 | "tags": [] 772 | } 773 | } 774 | ] 775 | }, 776 | { 777 | "metadata": { 778 | "id": "ShXhMYlYcUYi", 779 | "colab_type": "text" 780 | }, 781 | "cell_type": "markdown", 782 | "source": [ 783 | "Now, to create some kind of engagement metric. \n", 784 | "\n", 785 | "Firstly, we should remove accounts that are not often interacted with, that is, where media favs == 0. Also since the way we're crowdsourcing this data is highly subjective, we should set a low cutoff for follower count. We'll make this 1000. \n", 786 | "\n", 787 | "Secondle, we need a solid proxy for engagement. We can use the mean of median_favs / # of followers and median_rts / # of followers, so we'll create that column now. The resulting array features mostly small numbers which are difficult to compare so we'll also standardize the column. " 788 | ] 789 | }, 790 | { 791 | "metadata": { 792 | "id": "7dhQWmxuk2rk", 793 | "colab_type": "code", 794 | "colab": {} 795 | }, 796 | "cell_type": "code", 797 | "source": [ 798 | "def remove_low_engagement_accounts(df):\n", 799 | " \n", 800 | " working_df = df.copy()\n", 801 | " \n", 802 | " working_df = working_df.loc[(working_df['median_favs'] > 0) &\n", 803 | " (working_df['followers'] >= 1000)]\n", 804 | " \n", 805 | " return working_df\n", 806 | "\n", 807 | "influencers_df = remove_low_engagement_accounts(influencers_df)" 808 | ], 809 | "execution_count": 0, 810 | "outputs": [] 811 | }, 812 | { 813 | "metadata": { 814 | "id": "GGuRoEPkbhOY", 815 | "colab_type": "code", 816 | "colab": {} 817 | }, 818 | "cell_type": "code", 819 | "source": [ 820 | "def create_engagement_metric(df):\n", 821 | " \n", 822 | " working_df = df.copy()\n", 823 | " \n", 824 | " from sklearn.preprocessing import MinMaxScaler\n", 825 | " # Favorites\n", 826 | " fav_eng_array = df['median_favs'] / df['followers']\n", 827 | " scaler = MinMaxScaler().fit(fav_eng_array.values.reshape(-1, 1))\n", 828 | " scaled_favs = scaler.transform(fav_eng_array.values.reshape(-1, 1))\n", 829 | " \n", 830 | " # Retweets\n", 831 | " rt_eng_array = df['median_favs'] / df['followers']\n", 832 | " scaler = MinMaxScaler().fit(rt_eng_array.values.reshape(-1, 1))\n", 833 | " scaled_rts = scaler.transform(rt_eng_array.values.reshape(-1, 1))\n", 834 | " \n", 835 | " mean_eng = (scaled_favs + scaled_rts) / 2\n", 836 | " working_df['engagement'] = mean_eng\n", 837 | " \n", 838 | " return working_df\n", 839 | " \n", 840 | "\n", 841 | "influencers_df = create_engagement_metric(influencers_df)" 842 | ], 843 | "execution_count": 0, 844 | "outputs": [] 845 | }, 846 | { 847 | "metadata": { 848 | "id": "2h3t57kgFUwV", 849 | "colab_type": "code", 850 | "outputId": "1592e3ec-2e33-4cf8-a0d4-fb88eeb0735a", 851 | "colab": { 852 | "base_uri": "https://localhost:8080/", 853 | "height": 204 854 | } 855 | }, 856 | "cell_type": "code", 857 | "source": [ 858 | "influencers_df.sort_values('engagement', ascending=False).head()" 859 | ], 860 | "execution_count": 39, 861 | "outputs": [ 862 | { 863 | "output_type": "execute_result", 864 | "data": { 865 | "text/html": [ 866 | "
\n", 867 | "\n", 880 | "\n", 881 | " \n", 882 | " \n", 883 | " \n", 884 | " \n", 885 | " \n", 886 | " \n", 887 | " \n", 888 | " \n", 889 | " \n", 890 | " \n", 891 | " \n", 892 | " \n", 893 | " \n", 894 | " \n", 895 | " \n", 896 | " \n", 897 | " \n", 898 | " \n", 899 | " \n", 900 | " \n", 901 | " \n", 902 | " \n", 903 | " \n", 904 | " \n", 905 | " \n", 906 | " \n", 907 | " \n", 908 | " \n", 909 | " \n", 910 | " \n", 911 | " \n", 912 | " \n", 913 | " \n", 914 | " \n", 915 | " \n", 916 | " \n", 917 | " \n", 918 | " \n", 919 | " \n", 920 | " \n", 921 | " \n", 922 | " \n", 923 | " \n", 924 | " \n", 925 | " \n", 926 | " \n", 927 | " \n", 928 | " \n", 929 | " \n", 930 | " \n", 931 | " \n", 932 | " \n", 933 | " \n", 934 | " \n", 935 | " \n", 936 | " \n", 937 | " \n", 938 | " \n", 939 | " \n", 940 | " \n", 941 | " \n", 942 | " \n", 943 | " \n", 944 | " \n", 945 | "
user_idscreen_namebiofollowersmedian_favsmedian_rtsengagement
1093511081273davearterburnInternal medicine doc & researcher @KPWaResear...137515.54.01.000000
101778425499387326465PaulDavidsonPhDDirector of Behavioral Health, Center for Meta...209019.03.00.806384
1073676486649RaulRosenthalMDChairman, Department of General Surgery, Direc...215515.01.00.617335
103731838301599059968JaimePonceMDBariatric Surgeon, ASMBS Past-President @ASMBS...250911.52.00.406393
1083612178336PSchauerMDChief, Minimally Invasive General #Surgery & D...329915.03.00.403139
\n", 946 | "
" 947 | ], 948 | "text/plain": [ 949 | " user_id screen_name \\\n", 950 | "109 3511081273 davearterburn \n", 951 | "101 778425499387326465 PaulDavidsonPhD \n", 952 | "107 3676486649 RaulRosenthalMD \n", 953 | "103 731838301599059968 JaimePonceMD \n", 954 | "108 3612178336 PSchauerMD \n", 955 | "\n", 956 | " bio followers \\\n", 957 | "109 Internal medicine doc & researcher @KPWaResear... 1375 \n", 958 | "101 Director of Behavioral Health, Center for Meta... 2090 \n", 959 | "107 Chairman, Department of General Surgery, Direc... 2155 \n", 960 | "103 Bariatric Surgeon, ASMBS Past-President @ASMBS... 2509 \n", 961 | "108 Chief, Minimally Invasive General #Surgery & D... 3299 \n", 962 | "\n", 963 | " median_favs median_rts engagement \n", 964 | "109 15.5 4.0 1.000000 \n", 965 | "101 19.0 3.0 0.806384 \n", 966 | "107 15.0 1.0 0.617335 \n", 967 | "103 11.5 2.0 0.406393 \n", 968 | "108 15.0 3.0 0.403139 " 969 | ] 970 | }, 971 | "metadata": { 972 | "tags": [] 973 | }, 974 | "execution_count": 39 975 | } 976 | ] 977 | }, 978 | { 979 | "metadata": { 980 | "id": "CkBno9kl8kWs", 981 | "colab_type": "text" 982 | }, 983 | "cell_type": "markdown", 984 | "source": [ 985 | "So that's a start on scraping some \"crowdsourced\" data from Twitter. From here it would be easy enough to generate some business level insights, but we'll save that for another notebook to keep this exercise concise. \n", 986 | "\n", 987 | "The next step for this project is to translate this \"notebook\" code into a production friendly .py file. " 988 | ] 989 | } 990 | ] 991 | } --------------------------------------------------------------------------------