├── LICENSE └── README.rst /LICENSE: -------------------------------------------------------------------------------- 1 | Creative Commons Legal Code 2 | 3 | CC0 1.0 Universal 4 | 5 | CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE 6 | LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN 7 | ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS 8 | INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES 9 | REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS 10 | PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM 11 | THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED 12 | HEREUNDER. 13 | 14 | Statement of Purpose 15 | 16 | The laws of most jurisdictions throughout the world automatically confer 17 | exclusive Copyright and Related Rights (defined below) upon the creator 18 | and subsequent owner(s) (each and all, an "owner") of an original work of 19 | authorship and/or a database (each, a "Work"). 20 | 21 | Certain owners wish to permanently relinquish those rights to a Work for 22 | the purpose of contributing to a commons of creative, cultural and 23 | scientific works ("Commons") that the public can reliably and without fear 24 | of later claims of infringement build upon, modify, incorporate in other 25 | works, reuse and redistribute as freely as possible in any form whatsoever 26 | and for any purposes, including without limitation commercial purposes. 27 | These owners may contribute to the Commons to promote the ideal of a free 28 | culture and the further production of creative, cultural and scientific 29 | works, or to gain reputation or greater distribution for their Work in 30 | part through the use and efforts of others. 31 | 32 | For these and/or other purposes and motivations, and without any 33 | expectation of additional consideration or compensation, the person 34 | associating CC0 with a Work (the "Affirmer"), to the extent that he or she 35 | is an owner of Copyright and Related Rights in the Work, voluntarily 36 | elects to apply CC0 to the Work and publicly distribute the Work under its 37 | terms, with knowledge of his or her Copyright and Related Rights in the 38 | Work and the meaning and intended legal effect of CC0 on those rights. 39 | 40 | 1. Copyright and Related Rights. A Work made available under CC0 may be 41 | protected by copyright and related or neighboring rights ("Copyright and 42 | Related Rights"). Copyright and Related Rights include, but are not 43 | limited to, the following: 44 | 45 | i. the right to reproduce, adapt, distribute, perform, display, 46 | communicate, and translate a Work; 47 | ii. moral rights retained by the original author(s) and/or performer(s); 48 | iii. publicity and privacy rights pertaining to a person's image or 49 | likeness depicted in a Work; 50 | iv. rights protecting against unfair competition in regards to a Work, 51 | subject to the limitations in paragraph 4(a), below; 52 | v. rights protecting the extraction, dissemination, use and reuse of data 53 | in a Work; 54 | vi. database rights (such as those arising under Directive 96/9/EC of the 55 | European Parliament and of the Council of 11 March 1996 on the legal 56 | protection of databases, and under any national implementation 57 | thereof, including any amended or successor version of such 58 | directive); and 59 | vii. other similar, equivalent or corresponding rights throughout the 60 | world based on applicable law or treaty, and any national 61 | implementations thereof. 62 | 63 | 2. Waiver. To the greatest extent permitted by, but not in contravention 64 | of, applicable law, Affirmer hereby overtly, fully, permanently, 65 | irrevocably and unconditionally waives, abandons, and surrenders all of 66 | Affirmer's Copyright and Related Rights and associated claims and causes 67 | of action, whether now known or unknown (including existing as well as 68 | future claims and causes of action), in the Work (i) in all territories 69 | worldwide, (ii) for the maximum duration provided by applicable law or 70 | treaty (including future time extensions), (iii) in any current or future 71 | medium and for any number of copies, and (iv) for any purpose whatsoever, 72 | including without limitation commercial, advertising or promotional 73 | purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each 74 | member of the public at large and to the detriment of Affirmer's heirs and 75 | successors, fully intending that such Waiver shall not be subject to 76 | revocation, rescission, cancellation, termination, or any other legal or 77 | equitable action to disrupt the quiet enjoyment of the Work by the public 78 | as contemplated by Affirmer's express Statement of Purpose. 79 | 80 | 3. Public License Fallback. Should any part of the Waiver for any reason 81 | be judged legally invalid or ineffective under applicable law, then the 82 | Waiver shall be preserved to the maximum extent permitted taking into 83 | account Affirmer's express Statement of Purpose. In addition, to the 84 | extent the Waiver is so judged Affirmer hereby grants to each affected 85 | person a royalty-free, non transferable, non sublicensable, non exclusive, 86 | irrevocable and unconditional license to exercise Affirmer's Copyright and 87 | Related Rights in the Work (i) in all territories worldwide, (ii) for the 88 | maximum duration provided by applicable law or treaty (including future 89 | time extensions), (iii) in any current or future medium and for any number 90 | of copies, and (iv) for any purpose whatsoever, including without 91 | limitation commercial, advertising or promotional purposes (the 92 | "License"). The License shall be deemed effective as of the date CC0 was 93 | applied by Affirmer to the Work. Should any part of the License for any 94 | reason be judged legally invalid or ineffective under applicable law, such 95 | partial invalidity or ineffectiveness shall not invalidate the remainder 96 | of the License, and in such case Affirmer hereby affirms that he or she 97 | will not (i) exercise any of his or her remaining Copyright and Related 98 | Rights in the Work or (ii) assert any associated claims and causes of 99 | action with respect to the Work, in either case contrary to Affirmer's 100 | express Statement of Purpose. 101 | 102 | 4. Limitations and Disclaimers. 103 | 104 | a. No trademark or patent rights held by Affirmer are waived, abandoned, 105 | surrendered, licensed or otherwise affected by this document. 106 | b. Affirmer offers the Work as-is and makes no representations or 107 | warranties of any kind concerning the Work, express, implied, 108 | statutory or otherwise, including without limitation warranties of 109 | title, merchantability, fitness for a particular purpose, non 110 | infringement, or the absence of latent or other defects, accuracy, or 111 | the present or absence of errors, whether or not discoverable, all to 112 | the greatest extent permissible under applicable law. 113 | c. Affirmer disclaims responsibility for clearing rights of other persons 114 | that may apply to the Work or any use thereof, including without 115 | limitation any person's Copyright and Related Rights in the Work. 116 | Further, Affirmer disclaims responsibility for obtaining any necessary 117 | consents, permissions or other rights required for any use of the 118 | Work. 119 | d. Affirmer understands and acknowledges that Creative Commons is not a 120 | party to this document and has no duty or obligation with respect to 121 | this CC0 or use of the Work. 122 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | awesome-twitter-data 2 | #################### 3 | |Awesome| |CC0| 4 | 5 | .. |Awesome| image:: https://awesome.re/badge.svg 6 | :target: https://awesome.re 7 | 8 | A list of Twitter datasets and related resources, released under `CC0 `_. If you have a resource to add to the list, feel free to open a pull request, or email me at `shay.palachy@gmail.com `_. 9 | 10 | The license, when known, is given in {curly brackets}. Dataset size is given in [square brackets] when available. 11 | 12 | 13 | .. contents:: Table of Contents 14 | 15 | .. section-numbering:: 16 | 17 | 18 | Twitter Datasets 19 | ================ 20 | 21 | 22 | Tweet datasets 23 | -------------- 24 | 25 | * `Chirps `_ {`Apache License 2.0`_} [9m] - News-related tweets. Updated daily. Used in the paper `"Acquiring Predicate Paraphrases from News Tweets" `_ by Vered Shwartz, Gabriel Stanovsky and Ido Dagan. 26 | 27 | * `COVID-19 Twitter Dataset `_ {`CC0`_} [100m] - Tweets acquired from the Twitter Stream related to COVID-19 chatter. Can also be found `on Zenodo.org `_. 28 | 29 | * `Arab Spring Twitter Dataset `_ {?} [8m] - Roughly 8 million tweets related to the Arab Spring, separated by country. Contain tweet IDs and twitter IDS, but no tweet contents. 30 | 31 | * `Cheng-Caverlee-Lee `_ {?} [5m] - A collection of scraped public twitter updates used in coordination with an academic project to study the geolocation data related to twittering. 32 | 33 | * `3 million Russian troll tweets `_ {?} [3m] - Released by 538. 34 | 35 | * `Lerman Twitter 2010 Dataset `_ [2.8m] - Contains tweets containing URLs that have been posted on Twitter during October 2010. In addition to tweets, links of tweeting users were followed, allowing the reconstruction the follower graph of active (tweeting) users. 36 | 37 | * `Twitter_2010 `_ {?} [2m] - Released by Kristina Lerman at USC. 38 | 39 | * `MovieTweetings `_ {`MIT`_} [725k] - A live movie rating dataset collected from Twitter. 40 | 41 | * `Geotagged COVID-19 Twitter Dataset `_ {?} [650k] - Geotagged COVID-19 tweets collected from the Twitter Stream within the United States (50 states and Washington, D.C.). 42 | 43 | * `Sanders Analytics Sentiment-labelled tweets `_ {?} [5513] 44 | 45 | * `Elon Musk Tweets-Until 4/6/17 `_ 46 | 47 | * `2015 New Year's Resolutions `_ 48 | 49 | * `Trump Tweets, 5/4/09 - 12/5/16 `_ 50 | 51 | * `ArchiveTeam JSON Download of Twitter Stream, 2011 - 2022 [16bn] `_ 52 | 53 | 54 | Tweet ID datasets 55 | ~~~~~~~~~~~~~~~~~ 56 | 57 | * `72 Hours of #Gamersgate `_ [313K] 58 | 59 | * `RepLab 2013 Dataset `_ - RepLab 2013 dataset uses Twitter data in English and Spanish (more than 142,000 tweets). 60 | 61 | 62 | Tweet datasets (labelled) 63 | -------------------------- 64 | 65 | * `Sentiment140 `_ - Automatically labelled; authors assume that any tweet with positive emoticons, like :), are positive, and tweets with negative emoticons, like :(, are negative. 66 | 67 | * `Weather-sentiment `_ 68 | 69 | * `Crowdflower Gender Classifier Data `_ [20k] - Contributors were asked to simply view a Twitter profile and judge whether the user was a male, a female, or a brand (non-individual). The dataset contains 20,000 rows, each with a user name, a random tweet, account profile and image, location, and even link and sidebar color. 70 | 71 | * `Sanders Analytics `_ {?} [5k]- Use Internet Archive's `Wayback Machine `_ to get the data. The dataset consists of 5513 hand-classified tweets. Each tweet was classified with respect to one of four different topics. 72 | 73 | * `Geoparse Benchmark Open Dataset `_ {`BSD-4_Clause`_} [?] - The geoparsing benchmark dataset contains 1000’s of tweets recorded during 4 different natural disasters. These events are Hurricane Sandy 2012, Milan Blackouts 2013, Turkish Earthquake 2012 and the Christchurch Earthquake 2012. Each tweet in the dataset has been manually labelled with location entries at the building, street and region levels to provide a gold standard for evaluation work. The data consists of the full JSON serialized tweet metadata (i.e. including text) with an additional ‘entities’ field of type ‘mentions’ for the ground truth location annotations. 74 | 75 | 76 | User datasets 77 | ------------- 78 | 79 | * `Max Plank Institute's Twitter Dataset `_ {?} [55m] - **The social graph component only of the following dataset:** 54,981,152 user accounts; 1,963,263,821 social (follow) links. 1,755,925,520 tweets. 80 | 81 | * `Twitter Social Graph `_ {?} [41m] - From the `"What is Twitter, a Social Network or a News Media?" paper `_. 82 | 83 | * `Arizona State University Twitter Data Set `_ [11m] - `Alternate download (via torrent) here `_. 84 | 85 | * `Twitter User Sample (Tweets Loud and Quiet) `_ {`MPL 2.0`_} [400k] - Metadata of ~400,000 Twitter accounts, scraped between September 17, 2013, and October 19, 2013, as part of the work on the `"Tweets loud and quiet" article `_. 86 | 87 | * `Higgs Twitter Dataset `_ {?} [456k] - The Higgs dataset has been built after monitoring the spreading processes on Twitter before, during and after the announcement of the discovery of a new particle with the features of the elusive Higgs boson on 4th July 2012. 88 | 89 | * `Twitter Data - NIPS 2012 `_ [81k] - This dataset consists of 'circles' (or 'lists') from Twitter. Twitter data was crawled from public sources. The dataset includes node features (profiles), circles, and ego networks. 90 | 91 | * `ego-twitter `_ [80k] - 80K nodes and 1.7 million edges. 92 | 93 | * `The SNAP 2010 Dataset `_ [41m] - 41m nodes and 1.4 billion edges. 94 | 95 | * `soTweet `_ [full] {X} - This projects includes a Twitter social graph dataset - of the full Twitter network - from 2012. Also includes an anonymized dataset of social clicks from 2016. Both are available on request. 96 | 97 | 98 | 99 | Lost Datasets 100 | ------------- 101 | 102 | * `Twitter Event Detection Dataset `_ {?} [120m] - A collection of 120 million tweets, with relevance judgements for over 500 events. 103 | 104 | * Kwak10www - A dataset consisting of 41.7 million user profiles, 1.47 billion social relations, 4,262 trending topics, and 106 million tweets, collected between July 6th, 2009 to July 31st, 2009. From the `"What is Twitter, a Social Network or a News Media?" paper `_. The social graph part of that data set is available on `the paper's webpage `_. 105 | 106 | * `twitter7 `_ - A dataset consisting of nearly 580 million Twitter posts from 20 million users covering a 8 month period from June 2009 to February 2010. Estimated to be about 20-30% of all posts published on Twitter during that time frame. Created as part of [`J. Yang, J. Leskovec. Temporal Variation in Online Media. ACM International Conference on Web Search and Data Mining (WSDM '11), 2011. `_]. 107 | 108 | * burger2011 - A corpus consisting of 213 million tweets from 18.5 million users, in many different languages. Collected as part of `[John D. Burger, John C. Henderson, George Kim, and Guido Zarrella. 2011. Discriminating gender on Twitter. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1301–1309] `_. 109 | 110 | * calufa2011 - 200+ million tweets from 13+ million users, 173 GB uncompressed, mysql format (543 million rows). The archive.org copy has been taken down: https://archive.org/details/2011-05-calufa-twitter-sql. Several mentions on HackerNews: https://news.ycombinator.com/item?id=2633384 111 | 112 | * fifa2014 - 86k tweets from 2014 FIFA World Cup. Described here: https://opendata.stackexchange.com/a/4015, previously available from a data-provider which is no longer online (topsy.com). One potential source is https://ankeshanand.carto.com/tables/fifatweets/public, although the data export is not clear. There are several blogs describing this data and its analysis, so contacting those authors may be a way to revive this dataset. 113 | 114 | Other Lists 115 | =========== 116 | 117 | * `Tweet ID Datasets `_ {`CC-BY 4.0`_} - A catalog of Twitter ID (i.e. contentless) datasets that are publicly available on the web. 118 | 119 | * `Free Twitter Datasets by followthehashtag `_ 120 | 121 | * `Twitter open datasets `_ - A question on `opendata.stackexchange `_. 122 | 123 | * `Datasets for PAN's shared tasks on digital text forensics `_ - Not Tweeter, but close. 124 | 125 | 126 | Tools 127 | ===== 128 | 129 | Data Collection 130 | --------------- 131 | 132 | * `twitter-dataset-collector `_ {`Apache License 2.0`_} [Java] - Facilitates the distribution of Twitter datasets by downloading sets of tweets (if still available) using their ids as input. 133 | 134 | * `Expand The Edinburgh Twitter FSD Corpus `_ 135 | 136 | * `Twitter-ratings `_ {`MIT`_} - A collection of Python scripts to download and extract rating datasets from Twitter for multiple websites. 137 | 138 | * `RepLab 2013 Twitter text downloaded `_ - Find it at the bottom of the page. 139 | 140 | 141 | Analysis 142 | -------- 143 | 144 | * `OSU Twitter NLP Tools `_ - A suite of Twitter NLP tools. 145 | 146 | * `sentimentstwitter `_ {`MIT`_} - Given a tweet (that contains some text), estimate the sentiment (negative or positive) of the tweeter. 147 | 148 | * `Twitter-L-LDA `_ {`GPLv3`_} - A set of tools for performing Labeled Latent Dirichlet Allocation on textual datasets, with an emphasis on Twitter profiles. Contains tools for analysing the results of model training and inference. 149 | 150 | * `TwitterGenderPredictor `_ by `jtwool `_ - Does what it says. Has a `simple package wrapper `_ by Shay Palachy. 151 | 152 | * `Tools by Alan Ritter `_ - Several Twitter-related tools by Alan Ritter. 153 | 154 | 155 | Academic Papers 156 | =============== 157 | 158 | * Learning Multiview Embeddings of Twitter Users 159 | 160 | Demographics Prediction 161 | ----------------------- 162 | 163 | * `Developing Age and Gender Predictive Lexica over Social Media, 2014 `_ - We derive predictive lexica (words and weights) for age and gender using regression and classification models from word usage in Facebook, blog, and Twitter data with associated demographiclabels. The lexica, made publicly available, achieved state-of-the-art accuracy in language based age and gender prediction over Facebook and Twitter, and were evaluated for generalization across social media genres as well as in limited message situations. 164 | 165 | * Predicting the Demographics of Twitter Users from Website Traffic Data 166 | 167 | * Inferring Perceived Demographics from User Emotional Tone and User-Environment Emotional Contrast 168 | 169 | * Mining User Interests to Predict Perceived Psycho-Demographic Traits on Twitter 170 | 171 | * Why Gender and Age Prediction from Tweets is Hard: Lessons from a Crowdsourcing Experiment 172 | 173 | * Who tweets? deriving the demographic characteristics of age, occupation and social class from twitter user meta-data 174 | 175 | 176 | Articles & blog posts 177 | ===================== 178 | 179 | * `Twitter sentiment analysis using Python and NLTK `_ 180 | 181 | * `72 Hours of #Gamersgate `_ 182 | 183 | * `A beginner's guide to collecting Twitter data (and a bit of web scraping) `_ 184 | 185 | 186 | Contributing 187 | ============ 188 | 189 | - Please check for duplicates first. 190 | - Keep descriptions short, simple and unbiased. 191 | - Please make an individual commit for each suggestion 192 | - Add a new category if needed. 193 | - For datasets, please keep the format when possible: The license, when known, is given in {curly brackets}. Dataset size is given in [square brackets] when available. 194 | 195 | Thank you for your suggestions! 196 | 197 | 198 | License 199 | ======= 200 | |CC0| 201 | 202 | .. |CC0| image:: https://licensebuttons.net/p/zero/1.0/88x31.png 203 | :target: https://creativecommons.org/publicdomain/zero/1.0/ 204 | 205 | To the extent possible under law, `Shay Palachy `_ has waived all copyright and related or neighboring rights to this work. 206 | 207 | .. License Links 208 | 209 | .. _Public Domain: https://en.wikipedia.org/wiki/Public_domain 210 | .. _CC-BY-SA 3.0: https://creativecommons.org/licenses/by-sa/3.0/ 211 | .. _AGPL-3.0: https://opensource.org/licenses/AGPL-3.0 212 | .. _GPLv3: http://www.gnu.org/copyleft/gpl.html 213 | .. _CC0: https://creativecommons.org/publicdomain/zero/1.0/ 214 | .. _CC BY-NC-SA 4.0: https://creativecommons.org/licenses/by-nc-sa/4.0/ 215 | .. _CC BY-NC 4.0: https://creativecommons.org/licenses/by-nc/4.0/ 216 | .. _Apache License 2.0: https://www.apache.org/licenses/LICENSE-2.0 217 | .. _MIT: https://en.wikipedia.org/wiki/MIT_License 218 | .. _CC-BY 4.0: https://creativecommons.org/licenses/by/4.0/ 219 | .. _MPL 2.0: https://github.com/jonbruner/twitter-analysis 220 | .. _BSD-4_Clause: https://en.wikipedia.org/wiki/BSD_licenses#4-clause_license_(original_%22BSD_License%22) 221 | --------------------------------------------------------------------------------