├── .gitignore ├── .travis.yml ├── LICENSE.md ├── README.md ├── config.py ├── controllers ├── __init__.py ├── tweet_reader.py └── twitter_client.py ├── main.py ├── models ├── __init__.py ├── base_model.py └── tweet.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | cache 3 | tweets.db 4 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | # command to install dependencies 3 | install: "pip install -r requirements.txt" 4 | # command to run tests 5 | # script: maintests -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 João Pescada 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # TwitterPiBot 2 | A Python based bot for Raspberry Pi that grabs tweets with a specific hashtag and reads them out loud. 3 | 4 | It was a quick side project that served as a good learning exercise for me. As this was a side project, I've set a limit of 2 days to get it done. Yes, it is sort of hacked together and could use a bit of refactoring. But it still handled the top trending hashtag #StarWars for 24 hours, without breaking. So, it's stable (enough) and working fine. (*famous last words?*) 5 | 6 | 7 | ### How does it work? 8 | This python app connects to Twitter Streaming API and captures tweets with a specific hashtag. Those captured tweets are then processed and stored in a local SQLite database that works like a queue. And lastly (every few seconds), a tweet is picked from that queue and ran through a text-to-speech engine converting the tweet into audio, that is played out through speakers connected to the audio jack. 9 | 10 | 11 | ### What’s inside? 12 | To run, it uses a good chunk of 3rd party modules, such as: 13 | * [Peewee](https://github.com/coleifer/peewee) (to manage models and an SQLite database) 14 | * [Tweepy](https://github.com/tweepy/tweepy) (to access Twitter API) 15 | * [Flite](http://www.festvox.org/flite/) (to synthetise speech from tweets) 16 | 17 | 18 | ### What do you need to make it work? 19 | Essentially, a Raspberry Pi (running Debian) with a USB Wifi dongle attached (and connected to the internet). Then you need some source of power (such as a USB portable battery pack) and one or two speakers connected to the 3.5mm audio jack. 20 | 21 | 22 | ### How do you get this up and running? 23 | To run the instructions below, you have two options (that I can remember): 24 | 25 | 1. plug your Raspberry Pi to a screen, mouse, keyboard and run in the terminal 26 | 2. [connect to your Raspberry via SSH](https://www.raspberrypi.org/documentation/remote-access/ssh/) from your computer and run it remotely (*which is waaay cooler!*) 27 | 28 | I recommend the second option. And if you go with the second option, I also recommend using [`screen`](https://en.wikipedia.org/wiki/GNU_Screen) in SSH to allow [resuming your session without having to leave an SSH window open](http://raspi.tv/2012/using-screen-with-raspberry-pi-to-avoid-leaving-ssh-sessions-open). To install `screen` run the following in terminal: 29 | 30 | `$ sudo apt-get install screen` 31 | 32 | 33 | #### 1. Create a Twitter app 34 | Ok, so first things first: you need to create a Twitter app to use their API. Go to https://apps.twitter.com and create a new app. Once that is done, under the "Keys and Access Tokens" tab also generate an Access Token. 35 | 36 | #### 2. Update and install packages 37 | Now, let's make sure your Raspberry Pi is up-to-date. In terminal run the following two commands: 38 | 39 | `$ sudo apt-get update` 40 | 41 | `$ sudo apt-get upgrade` 42 | 43 | 44 | And one (more) package to install: [Flite](http://www.festvox.org/flite/), the text-to-speech engine that we'll be using. Run this: 45 | 46 | `$ sudo apt-get install flite` 47 | 48 | 49 | #### 3. Install app in your Raspberry Pi 50 | 51 | Start by cloning this repository to your Raspberry Pi: 52 | 53 | `$ git clone https://github.com/jpescada/TwitterPiBot.git` 54 | 55 | 56 | Go into that new folder: 57 | 58 | `$ cd TwitterPiBot/` 59 | 60 | 61 | Make sure [`pip`](https://en.wikipedia.org/wiki/Pip_(package_manager)) is up to date: 62 | 63 | `$ sudo pip install -U pip` 64 | 65 | 66 | And install the python modules required for this app ([Peewee](https://github.com/coleifer/peewee) and [Tweepy](https://github.com/tweepy/tweepy)): 67 | 68 | `$ sudo pip install -r requirements.txt` 69 | 70 | 71 | #### 4. Update the app config 72 | 73 | Just one last thing to do before running it for the first time. Open the `config.py` file in the root folder to update the Twitter API credentials and the hashtag to search for: 74 | 75 | `$ sudo nano config.py` 76 | 77 | When you're done, hit `Ctrl+X` to close and save the file. 78 | 79 | 80 | #### 5. Finally, run the app 81 | 82 | Just type the command: 83 | 84 | `$ python main.py` 85 | 86 | If everything went according to plan, it should connect to Twitter start collecting tweets and reading them out loud every 30 seconds. 87 | 88 | To exit, hit `Ctrl+C`. 89 | 90 | 91 | ### How to adjust the volume? 92 | 93 | In terminal, run the command: 94 | 95 | `$ alsamixer` 96 | 97 | Then use keyboard `up` and `down` keys for volume, `m` key to mute and `esc` key to exit. 98 | 99 | 100 | ### Bugs? 101 | 102 | What bugs? :) If you found any issues, please report it in the [Issues](https://github.com/jpescada/TwitterPiBot/issues) or, if you can fix it, submit a [Pull request](https://github.com/jpescada/TwitterPiBot/pulls). Thank you! 103 | 104 | 105 | ##### Credits 106 | 107 | The original idea behind this project came from a client request (who later dropped it), but it's based on [Hugo the Twitter-Powered Robot](http://paper-leaf.com/hugo/) by [Paper Leaf](http://paper-leaf.com/). 108 | 109 | -------------------------------------------------------------------------------- /config.py: -------------------------------------------------------------------------------- 1 | 2 | # twitter hashtag to find 3 | TWITTER_HASHTAG = "#StarWars" 4 | 5 | 6 | # twitter api credentials (https://apps.twitter.com/) 7 | TWITTER_CONSUMER_KEY = "your-consumer-key" 8 | TWITTER_CONSUMER_SECRET = "your-consumer-secret" 9 | TWITTER_ACCESS_TOKEN = "your-access-token" 10 | TWITTER_ACCESS_TOKEN_SECRET = "your-access-token-secret" 11 | 12 | # delay in seconds between reading tweets 13 | DELAY_TO_READ_TWEET = 30.0 14 | -------------------------------------------------------------------------------- /controllers/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jpescada/TwitterPiBot/fb1f044a9993a896ee1172b1bec493d01ca857d5/controllers/__init__.py -------------------------------------------------------------------------------- /controllers/tweet_reader.py: -------------------------------------------------------------------------------- 1 | 2 | # import external modules 3 | import os 4 | import subprocess 5 | from threading import Timer 6 | 7 | # import pyttsx 8 | # import talkey 9 | # import atexit 10 | 11 | from config import * 12 | from models import tweet 13 | 14 | timers = [] 15 | 16 | def cleanup(): 17 | print "Cleaning up..." 18 | for timer in timers: 19 | timer.cancel() 20 | 21 | # atexit.register(cleanup) 22 | 23 | def initialize(): 24 | # wait 10 seconds to start reading 25 | global timers 26 | timer = Timer( 10.0, read_tweet ) 27 | timer.start() 28 | timers.append( timer ) 29 | # read_tweet() 30 | 31 | def get_text_sanitized(text): 32 | 33 | # \ ' " ` < > | ; ( ) [ ] ? # $ ^ & * = 34 | 35 | # escape quotes 36 | text = text.replace('"','\"').replace("'","\'").replace('`','\`') 37 | 38 | # remove special shell characters 39 | text = text.replace('\\','') 40 | 41 | text = text.replace('|','\|') 42 | text = text.replace('#','\#') 43 | text = text.replace('$','\$') 44 | text = text.replace('^','\^') 45 | text = text.replace('&','\&') 46 | text = text.replace('*','\*') 47 | text = text.replace('=','\=') 48 | 49 | text = text.replace(';','\;').replace('?','\?').replace('!','\!') 50 | 51 | text = text.replace('',' ') 52 | text = text.replace('',' ') 53 | text = text.replace('',' ') 54 | 55 | text = text.replace('<','\<').replace('>','\>') 56 | text = text.replace('(','\(').replace(')','\)') 57 | text = text.replace('[','\[').replace(']','\]') 58 | text = text.replace('{','\{').replace('}','\}') 59 | 60 | return text 61 | 62 | 63 | 64 | def read_tweet(): 65 | 66 | # recall this function after a few more seconds 67 | global timers 68 | timer = Timer( DELAY_TO_READ_TWEET, read_tweet ) 69 | timer.start() 70 | timers.append( timer ) 71 | 72 | 73 | print "About to read tweet..." 74 | 75 | # subprocess.call(['flite', '-voice', 'kal16', '-t', '"{}"'.format( "hello world > WTF! & cool" ) ]) 76 | 77 | # next_tweet_query = tweet.Tweet.select().where(tweet.Tweet.is_valid == True, tweet.Tweet.is_done == False).order_by(tweet.Tweet.created_at.asc()).limit(1) 78 | # print "Query: {}".format(next_tweet_query) 79 | 80 | # for next_tweet in next_tweet_query: 81 | # print "Tweet to read: {}".format(next_tweet.message.encode('utf-8')) 82 | # subprocess.call( "flite -voice kal16 -t {}".format( get_text_sanitized( next_tweet.message.encode('utf-8') ) ) ) 83 | # os.system( 'flite -voice kal16 -t "{}"'.format( get_text_sanitized( next_tweet.message.encode('utf-8') ) )) 84 | 85 | try: 86 | 87 | next_tweet_query = tweet.Tweet.select().where(tweet.Tweet.is_valid == True, tweet.Tweet.is_done == False).order_by(tweet.Tweet.created_at.asc()).limit(1) 88 | 89 | for next_tweet in next_tweet_query: 90 | 91 | print 'Reading: "{}"'.format( next_tweet.message.encode('utf-8') ) 92 | 93 | # read tweet out loud using Mac OSX "say" 94 | # os.system("say {}".format(next_tweet.message.encode('utf-8'))) 95 | 96 | # read tweet out loud using Festival 97 | # os.system("echo ""{}"" | festival --tts".format(next_tweet.message.encode('utf-8'))) 98 | 99 | # read tweet out loud using Flite 100 | # os.system("flite -voice kal16 -t ""{}""".format( next_tweet.message.encode('utf-8') )) 101 | # os.system("aoss flite-2.0.0/bin/flite -voice voices/cmu_us_aew.flitevox -t ""{}""".format(next_tweet.message.encode('utf-8'))) 102 | # os.system( 'flite -voice kal16 -t "{}"'.format( get_text_sanitized( next_tweet.message ).encode('utf-8') )) 103 | subprocess.call(['flite', '-voice', 'kal16', '-t', '"{}"'.format( next_tweet.message.encode('utf-8') ) ]) 104 | # subprocess.call("flite -voice kal16 -t {}".format( get_text_sanitized( next_tweet.message.encode('utf-8') ) )) 105 | 106 | # try: 107 | # read tweet out loud using pyttsx 108 | # tts = pyttsx.init('espeak', True) 109 | # tts.say( next_tweet.message ) 110 | # tts.runAndWait() 111 | 112 | # read tweet out loud using talkey 113 | # tts = talkey.Talkey() 114 | # tts.say( next_tweet.message ) 115 | # except: 116 | # print "-- Error reading tweet." 117 | 118 | 119 | next_tweet.is_done = True 120 | next_tweet.save() 121 | 122 | except: 123 | print "-- No tweets found." 124 | 125 | 126 | 127 | 128 | -------------------------------------------------------------------------------- /controllers/twitter_client.py: -------------------------------------------------------------------------------- 1 | 2 | # import external modules 3 | import json 4 | import re 5 | 6 | from tweepy import OAuthHandler 7 | from tweepy import Stream 8 | from tweepy.streaming import StreamListener 9 | 10 | from config import * 11 | from models import tweet 12 | 13 | 14 | # write stream to database 15 | class TwitterStreamListener(StreamListener): 16 | 17 | def on_data(self, data): 18 | 19 | # print "tweet.data:\n{}".format(data) 20 | 21 | if data: 22 | tweet_json = json.loads(data) 23 | 24 | # print "tweet.json_data:\n{}".format(tweet_json) 25 | 26 | if tweet_json: 27 | 28 | # ignore retweets 29 | if not tweet_json['text'].strip().startswith('RT '): 30 | 31 | # clean tweet text 32 | clean_tweet = get_text_cleaned(tweet_json) 33 | 34 | # check if tweet is not empty after cleanup 35 | if clean_tweet: 36 | 37 | # register tweet in database 38 | tweet.Tweet.create( 39 | message=clean_tweet.strip(), 40 | author=tweet_json['user']['screen_name'], 41 | json_data=data 42 | ) 43 | 44 | return True 45 | 46 | def on_error(self, status): 47 | print status 48 | if status == 420: 49 | # disconnect from Twitter if being Rate Limited (https://dev.twitter.com/rest/public/rate-limiting) 50 | return False 51 | 52 | 53 | # clean up tweet text from urls, mentions, hashtags, etc 54 | def get_text_cleaned(tweet): 55 | # source: https://gist.github.com/timothyrenner/dd487b9fd8081530509c 56 | text = tweet['text'] 57 | 58 | slices = [] 59 | # Strip out the urls 60 | if 'urls' in tweet['entities']: 61 | for url in tweet['entities']['urls']: 62 | slices += [{'start': url['indices'][0], 'stop': url['indices'][1]}] 63 | 64 | # Strip out the hashtags (except if it's the one we're using to filter). 65 | if 'hashtags' in tweet['entities']: 66 | for tag in tweet['entities']['hashtags']: 67 | if not tag == TWITTER_HASHTAG: 68 | slices += [{'start': tag['indices'][0], 'stop': tag['indices'][1]}] 69 | 70 | # Strip out the user mentions. 71 | # if 'user_mentions' in tweet['entities']: 72 | # for men in tweet['entities']['user_mentions']: 73 | # slices += [{'start': men['indices'][0], 'stop': men['indices'][1]}] 74 | 75 | # Strip out the media. 76 | if 'media' in tweet['entities']: 77 | for med in tweet['entities']['media']: 78 | slices += [{'start': med['indices'][0], 'stop': med['indices'][1]}] 79 | 80 | # Strip out the symbols. 81 | if 'symbols' in tweet['entities']: 82 | for sym in tweet['entities']['symbols']: 83 | slices += [{'start': sym['indices'][0], 'stop': sym['indices'][1]}] 84 | 85 | # Sort the slices from highest start to lowest. 86 | slices = sorted(slices, key=lambda x: -x['start']) 87 | 88 | # No offsets, since we're sorted from highest to lowest. 89 | for s in slices: 90 | text = text[:s['start']] + text[s['stop']:] 91 | 92 | # remove "dot space" remains from beginning when mentioning someone 93 | if text.startswith('. '): 94 | text = text[2:] 95 | 96 | # replace other entities 97 | text = text.replace('&', 'and') 98 | text = text.replace('>', 'greater than') 99 | text = text.replace('<', 'lower than') 100 | 101 | # remove new lines 102 | text = text.replace('\n','') 103 | 104 | # remove whitespaces before and after string 105 | text = text.strip() 106 | 107 | return text 108 | 109 | 110 | # connect to Twitter API 111 | def initialize(): 112 | 113 | output = TwitterStreamListener() 114 | 115 | # setup Twitter API connection details 116 | twitter_auth = OAuthHandler( TWITTER_CONSUMER_KEY, TWITTER_CONSUMER_SECRET ) 117 | twitter_auth.set_access_token( TWITTER_ACCESS_TOKEN, TWITTER_ACCESS_TOKEN_SECRET ) 118 | 119 | # connect to Twitter Streaming API 120 | twitter_stream = Stream( twitter_auth, output ) 121 | 122 | # filter tweets using track, follow and/or location parameters 123 | # https://dev.twitter.com/streaming/reference/post/statuses/filter 124 | twitter_stream.filter(track=[ TWITTER_HASHTAG ]) 125 | 126 | 127 | # def cleanup(): 128 | # twitter_stream.disconnect() 129 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # import external modules 4 | 5 | import atexit 6 | 7 | from config import * 8 | from models import tweet 9 | from controllers import twitter_client 10 | from controllers import tweet_reader 11 | 12 | 13 | 14 | 15 | def goodbye(): 16 | print "See you later!" 17 | tweet_reader.cleanup() 18 | # twitter_client.cleanup() 19 | 20 | 21 | atexit.register(goodbye) 22 | 23 | 24 | # run default process 25 | if __name__ == '__main__': 26 | 27 | print "Bot starting... Press Ctrl+C to stop." 28 | 29 | tweet.initialize() 30 | 31 | tweet_reader.initialize() 32 | 33 | twitter_client.initialize() 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | -------------------------------------------------------------------------------- /models/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jpescada/TwitterPiBot/fb1f044a9993a896ee1172b1bec493d01ca857d5/models/__init__.py -------------------------------------------------------------------------------- /models/base_model.py: -------------------------------------------------------------------------------- 1 | 2 | # import exterma; modules 3 | from peewee import * 4 | 5 | DATABASE = SqliteDatabase('tweets.db') 6 | 7 | class BaseModel(Model): 8 | class Meta: 9 | database = DATABASE 10 | 11 | -------------------------------------------------------------------------------- /models/tweet.py: -------------------------------------------------------------------------------- 1 | 2 | # import external modules 3 | import datetime 4 | 5 | from peewee import * 6 | 7 | from models import base_model 8 | 9 | 10 | class Tweet(base_model.BaseModel): 11 | message = CharField() 12 | author = CharField() 13 | is_valid = BooleanField(default=True) 14 | is_done = BooleanField(default=False) 15 | json_data = TextField() 16 | created_at = DateTimeField(default=datetime.datetime.now) 17 | updated_at = DateTimeField(default=datetime.datetime.now) 18 | 19 | class Meta: 20 | order_by = ('-created_at',) 21 | 22 | 23 | 24 | def initialize(): 25 | base_model.DATABASE.connect() 26 | # DATABASE.create_tables([Tweet], safe=True) 27 | Tweet.create_table(True) 28 | base_model.DATABASE.close() -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | peewee==2.6.0 2 | tweepy==3.5.0 3 | --------------------------------------------------------------------------------