├── .gitignore ├── CODE_OF_CONDUCT.md ├── Data_Grabber.py ├── Inputs_Table_Builder.py ├── LICENSE ├── README.md ├── Table_Timer.py ├── Variable_Functions ├── Technical_Indicators.py ├── Twitter_Sentiment.py └── __init__.py ├── crycompare.py ├── got3 ├── __init__.py ├── manager │ ├── TweetCriteria.py │ ├── TweetManager.py │ ├── __init__.py │ └── __init__.py.bak └── models │ ├── Tweet.py │ └── __init__.py ├── requirements.txt └── requirements_to_freeze.txt /.gitignore: -------------------------------------------------------------------------------- 1 | # GITHUB BOILERPLATE GIT IGNORE 2 | # https://github.com/github/gitignore/blob/master/Python.gitignore 3 | 4 | # Byte-compiled / optimized / DLL files 5 | __pycache__/ 6 | *.py[cod] 7 | *$py.class 8 | 9 | # C extensions 10 | *.so 11 | 12 | # Distribution / packaging 13 | .Python 14 | build/ 15 | develop-eggs/ 16 | dist/ 17 | downloads/ 18 | eggs/ 19 | .eggs/ 20 | lib/ 21 | lib64/ 22 | parts/ 23 | sdist/ 24 | var/ 25 | wheels/ 26 | *.egg-info/ 27 | .installed.cfg 28 | *.egg 29 | MANIFEST 30 | 31 | # PyInstaller 32 | # Usually these files are written by a python script from a template 33 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 34 | *.manifest 35 | *.spec 36 | 37 | # Installer logs 38 | pip-log.txt 39 | pip-delete-this-directory.txt 40 | 41 | # Unit test / coverage reports 42 | htmlcov/ 43 | .tox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | .hypothesis/ 51 | .pytest_cache/ 52 | 53 | # Translations 54 | *.mo 55 | *.pot 56 | 57 | # Django stuff: 58 | *.log 59 | .static_storage/ 60 | .media/ 61 | local_settings.py 62 | 63 | # Flask stuff: 64 | instance/ 65 | .webassets-cache 66 | 67 | # Scrapy stuff: 68 | .scrapy 69 | 70 | # Sphinx documentation 71 | docs/_build/ 72 | 73 | # PyBuilder 74 | target/ 75 | 76 | # Jupyter Notebook 77 | .ipynb_checkpoints 78 | 79 | # pyenv 80 | .python-version 81 | 82 | # celery beat schedule file 83 | celerybeat-schedule 84 | 85 | # SageMath parsed files 86 | *.sage.py 87 | 88 | # Environments 89 | .env 90 | .venv 91 | env/ 92 | venv/ 93 | ENV/ 94 | env.bak/ 95 | venv.bak/ 96 | 97 | # Spyder project settings 98 | .spyderproject 99 | .spyproject 100 | 101 | # Rope project settings 102 | .ropeproject 103 | 104 | # mkdocs documentation 105 | /site 106 | 107 | # mypy 108 | .mypy_cache/ 109 | 110 | # OTHER GIT IGNORE 111 | 112 | #database 113 | market_prices.db 114 | 115 | #OS files 116 | .DS_Store 117 | .idea/ 118 | 119 | #Twitter Configuration File 120 | config.py 121 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. 6 | 7 | ## Our Standards 8 | 9 | Examples of behavior that contributes to creating a positive environment include: 10 | 11 | * Using welcoming and inclusive language 12 | * Being respectful of differing viewpoints and experiences 13 | * Gracefully accepting constructive criticism 14 | * Focusing on what is best for the community 15 | * Showing empathy towards other community members 16 | 17 | Examples of unacceptable behavior by participants include: 18 | 19 | * The use of sexualized language or imagery and unwelcome sexual attention or advances 20 | * Trolling, insulting/derogatory comments, and personal or political attacks 21 | * Public or private harassment 22 | * Publishing others' private information, such as a physical or electronic address, without explicit permission 23 | * Other conduct which could reasonably be considered inappropriate in a professional setting 24 | 25 | ## Our Responsibilities 26 | 27 | Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. 28 | 29 | Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. 30 | 31 | ## Scope 32 | 33 | This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. 34 | 35 | ## Enforcement 36 | 37 | Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at bshaw19@vt.edu. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. 38 | 39 | Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. 40 | 41 | ## Attribution 42 | 43 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version] 44 | 45 | [homepage]: http://contributor-covenant.org 46 | [version]: http://contributor-covenant.org/version/1/4/ 47 | -------------------------------------------------------------------------------- /Data_Grabber.py: -------------------------------------------------------------------------------- 1 | import os 2 | import ccxt 3 | import time 4 | import datetime 5 | import dash 6 | from dash.dependencies import Input, Output, Event 7 | import dash_core_components as dcc 8 | import dash_html_components as html 9 | import plotly.graph_objs as go 10 | import threading 11 | import dataset 12 | from collections import deque 13 | 14 | 15 | def ensure_dir(directory): 16 | """""" 17 | if not os.path.exists(directory): 18 | os.makedirs(directory) 19 | 20 | 21 | class CryptoDataGrabber(object): 22 | 23 | def __init__(self): 24 | """""" 25 | self.exchange = ccxt.poloniex() 26 | self.exchange.load_markets() 27 | self.delay_seconds = self.exchange.rateLimit / 1000 28 | self.symbols = self.exchange.markets 29 | self.timeframe = '1d' 30 | self.db_url = 'sqlite:///databases/market_prices.db' 31 | self.deques = dict() 32 | self.ohlcv = dict() 33 | self.database = dataset.connect(self.db_url) 34 | ensure_dir('databases') 35 | for symbol in self.symbols: 36 | if self.exchange.has['fetchOHLCV']: 37 | print('Obtaining OHLCV data') 38 | data = self.exchange.fetch_ohlcv(symbol, self.timeframe) 39 | data = list(zip(*data)) 40 | data[0] = [datetime.datetime.fromtimestamp(ms / 1000) 41 | for ms in data[0]] 42 | self.ohlcv[symbol] = data 43 | time.sleep(self.delay_seconds) 44 | else: 45 | print('No OHLCV data available') 46 | self.deques[symbol] = deque() 47 | if len(self.database[symbol]): 48 | for e in self.database[symbol]: 49 | entry = (e['bid'], e['ask'], e['spread'], e['time']) 50 | self.deques[symbol].append(entry) 51 | del self.database 52 | self.thread = threading.Thread(target=self.__update) 53 | self.thread.daemon = True 54 | self.thread.start() 55 | 56 | def get_symbols(self): 57 | """""" 58 | return self.deques.keys() 59 | 60 | def get_prices(self, symbol): 61 | """""" 62 | return self.deques[symbol] 63 | 64 | def get_ohlcv(self, symbol): 65 | """""" 66 | data = self.ohlcv[symbol] 67 | return data[0], data[1], data[2], data[3], data[4], data[5] 68 | 69 | def __update(self): 70 | """ 71 | https://github.com/ccxt-dev/ccxt/wiki/Manual#market-price 72 | """ 73 | self.database = dataset.connect(self.db_url) 74 | while True: 75 | for symbol in self.symbols: 76 | 77 | start_time = time.clock() 78 | orders = self.exchange.fetch_order_book(symbol) 79 | bid = orders['bids'][0][0] if len(orders['bids']) > 0 else None 80 | ask = orders['asks'][0][0] if len(orders['asks']) > 0 else None 81 | spread = (ask - bid) if (bid and ask) else None 82 | dtime = datetime.datetime.now() 83 | self.deques[symbol].append((bid, ask, spread, dtime)) 84 | self.database.begin() 85 | try: 86 | self.database[symbol].insert({ 87 | 'ask': ask, 88 | 'bid': bid, 89 | 'spread': spread, 90 | 'time': dtime 91 | }) 92 | self.database.commit() 93 | except: 94 | self.database.rollback() 95 | 96 | time.sleep(self.delay_seconds - (time.clock() - start_time)) 97 | 98 | 99 | data = CryptoDataGrabber() 100 | 101 | selected_dropdown_value = 'ETH/BTC' 102 | 103 | app = dash.Dash() 104 | app.layout = html.Div([ 105 | html.Div([ 106 | html.H1('Poloniex Nerd', id='h1_title'), 107 | dcc.Dropdown( 108 | id='symbol-dropdown', 109 | options=[{'label': key, 'value': key} 110 | for key in data.get_symbols()], 111 | value=selected_dropdown_value 112 | ), 113 | html.Div([ 114 | dcc.Graph( 115 | id='ohlc', 116 | config={ 117 | 'displayModeBar': False 118 | } 119 | ), 120 | ], className="row"), 121 | html.Div([ 122 | dcc.Graph( 123 | id='v', 124 | config={ 125 | 'displayModeBar': False 126 | } 127 | ), 128 | ], className="row"), 129 | html.Div([ 130 | html.Div([ 131 | dcc.Graph( 132 | id='market-prices-graph', 133 | config={ 134 | 'displayModeBar': False 135 | } 136 | ), 137 | dcc.Graph( 138 | id='spread-graph', 139 | config={ 140 | 'displayModeBar': False 141 | } 142 | ), 143 | ], className="eight columns"), 144 | html.Div([ 145 | dcc.Graph( 146 | id='market-prices-hist', 147 | config={ 148 | 'displayModeBar': False 149 | } 150 | ), 151 | dcc.Graph( 152 | id='spread-hist', 153 | config={ 154 | 'displayModeBar': False 155 | } 156 | ), 157 | ], className="four columns") 158 | ], className="row"), 159 | dcc.Interval(id='graph-speed-update', interval=2000), 160 | ], className="row") 161 | ]) 162 | app.css.append_css({ 163 | 'external_url': 'https://codepen.io/chriddyp/pen/bWLwgP.css' 164 | }) 165 | 166 | 167 | @app.callback(Output('h1_title', 'children'), 168 | [Input('symbol-dropdown', 'value')]) 169 | def change_plot(value): 170 | global selected_dropdown_value 171 | selected_dropdown_value = value 172 | return 'Market Prices ' + str(value) 173 | 174 | 175 | @app.callback(Output('ohlc', 'figure'), 176 | [Input('symbol-dropdown', 'value')]) 177 | def plot_olhc(value): 178 | global data 179 | dates, open_data, high_data, low_data, close_data, _ \ 180 | = data.get_ohlcv(value) 181 | return { 182 | 'data': [go.Ohlc(x=dates, 183 | open=open_data, 184 | high=high_data, 185 | low=low_data, 186 | close=close_data)], 187 | 'layout': dict(title="OHLC") 188 | } 189 | 190 | 191 | @app.callback(Output('v', 'figure'), 192 | [Input('symbol-dropdown', 'value')]) 193 | def plot_v(value): 194 | global data 195 | dates, _, _, _, _, volume = data.get_ohlcv(value) 196 | return { 197 | 'data': [go.Bar(x=dates, 198 | y=volume)], 199 | 'layout': dict(title="Volume") 200 | } 201 | 202 | 203 | @app.callback(Output('market-prices-graph', 'figure'), 204 | events=[Event('graph-speed-update', 'interval')]) 205 | def update_market_prices(): 206 | global selected_dropdown_value 207 | global data 208 | prices = data.get_prices(selected_dropdown_value) 209 | prices = [list(p) for p in zip(*prices)] 210 | if len(prices) > 0: 211 | traces = [] 212 | x = list(prices[3]) 213 | for i, key in enumerate(['bid', 'ask']): 214 | trace = go.Scatter(x=x, 215 | y=prices[i], 216 | name=key, 217 | opacity=0.8) 218 | traces.append(trace) 219 | return { 220 | 'data': traces, 221 | 'layout': dict(title="Market Prices") 222 | } 223 | 224 | 225 | @app.callback(Output('market-prices-hist', 'figure'), 226 | events=[Event('graph-speed-update', 'interval')]) 227 | def update_market_prices_hist(): 228 | global selected_dropdown_value 229 | global data 230 | prices = data.get_prices(selected_dropdown_value) 231 | prices = [list(p) for p in zip(*prices)] 232 | if len(prices) > 0: 233 | traces = [] 234 | for i, key in enumerate(['bid', 'ask']): 235 | trace = go.Histogram(x=prices[i][-200:], 236 | name=key, 237 | opacity=0.8) 238 | traces.append(trace) 239 | return { 240 | 'data': traces, 241 | 'layout': dict(title="Market Prices Histogram (200 Most Recent)") 242 | } 243 | 244 | 245 | @app.callback(Output('spread-graph', 'figure'), 246 | events=[Event('graph-speed-update', 'interval')]) 247 | def update_spread(): 248 | global selected_dropdown_value 249 | global data 250 | prices = data.get_prices(selected_dropdown_value) 251 | prices = [list(p) for p in zip(*prices)] 252 | if len(prices) > 0: 253 | traces = [] 254 | trace = go.Scatter(x=list(prices[3]), 255 | y=list(prices[2]), 256 | name='spread', 257 | line=dict(color='rgb(114, 186, 59)'), 258 | fill='tozeroy', 259 | fillcolor='rgba(114, 186, 59, 0.5)', 260 | mode='none') 261 | traces.append(trace) 262 | 263 | return { 264 | 'data': traces, 265 | 'layout': dict(title="Spread") 266 | } 267 | 268 | 269 | @app.callback(Output('spread-hist', 'figure'), 270 | events=[Event('graph-speed-update', 'interval')]) 271 | def update_spread_hist(): 272 | global selected_dropdown_value 273 | global data 274 | prices = data.get_prices(selected_dropdown_value) 275 | prices = [list(p) for p in zip(*prices)] 276 | if len(prices) > 0: 277 | traces = [] 278 | trace = go.Histogram(x=list(prices[2][-200:]), 279 | name='spread', 280 | marker=dict(color='rgba(114, 186, 59, 0.5)')) 281 | traces.append(trace) 282 | 283 | return { 284 | 'data': traces, 285 | 'layout': dict(title="Spread Histogram (200 Most Recent)") 286 | } 287 | 288 | 289 | if __name__ == '__main__': 290 | app.run_server() 291 | print(self.server) 292 | 293 | 294 | -------------------------------------------------------------------------------- /Inputs_Table_Builder.py: -------------------------------------------------------------------------------- 1 | import sqlite3 2 | import pandas as pd 3 | import numpy as np 4 | from datetime import datetime, timedelta 5 | from Variable_Functions.Twitter_Sentiment import dates_to_sentiment 6 | from crycompare import * 7 | import subprocess 8 | from time import strftime, gmtime 9 | import time 10 | import datetime 11 | import arrow 12 | from termcolor import colored 13 | 14 | 15 | 16 | 17 | 18 | def db_connection(database): 19 | """""" 20 | db_connection = sqlite3.connect(database) 21 | return db_connection 22 | 23 | 24 | def get_symbols(database): 25 | """""" 26 | db_connection = sqlite3.connect(database) 27 | cursor = db_connection.cursor() 28 | cursor.execute("SELECT name FROM sqlite_master WHERE type='table' and name NOT LIKE '%_1%'") 29 | symbols1 = cursor.fetchall() 30 | symbols = [] 31 | for symbol in symbols1: 32 | symbols.append(symbol[0]) 33 | return symbols 34 | 35 | 36 | symbols = get_symbols('databases/market_prices.db') 37 | 38 | ######################################## Bid Ask Spread Time ####################################### 39 | def OLHCV_From_DB(symbol, database): 40 | """""" 41 | df = pd.read_sql(r"SELECT * FROM '" + symbol + "'", con=db_connection(database)) 42 | return df 43 | 44 | 45 | ######################################### Twitter Sentiment ####################################### 46 | def twitter_sentiment(symbol, max_tweets): 47 | """""" 48 | 49 | now = arrow.now() 50 | dates = now.date() 51 | print(dates) 52 | 53 | time = now.time() 54 | print(time) 55 | sentiment_variables = dates_to_sentiment(dates, symbol, max_tweets) 56 | return sentiment_variables 57 | 58 | 59 | 60 | 61 | ######################################### Split Trading Pair into 'to' and 'from' ########### 62 | def split_symbols(symbols): 63 | """""" 64 | split_symbols = {} 65 | for symbol in symbols: 66 | split_symbols[symbol] = symbol.split('/') 67 | 68 | #split_symbols[symbol][0] is the "tp" symbol 69 | #split_symbols[symbol][1] is the "from" symbol 70 | 71 | return split_symbols 72 | 73 | 74 | 75 | 76 | ######################################### Main function to build input dataset ##################### 77 | def generate_input_dataset(database): 78 | """""" 79 | symbols = get_symbols(database) 80 | table_names = {} 81 | start_time = time.time() 82 | total_runtime = time.time() - start_time 83 | for symbol in symbols: 84 | table_names[symbol] = OLHCV_From_DB(symbol, database) 85 | i = len(symbols) 86 | total_symbols = len(symbols) 87 | for symbol in symbols: 88 | print('') 89 | print(str(i) + " More Symbols to Pull Sentiment For") 90 | print('') 91 | total_runtime += (time.time() - start_time) - total_runtime 92 | average_runtime = round(( total_runtime / ((total_symbols+1) - i)), 2) 93 | print('') 94 | print(colored("Total Runtime = " + str(round((total_runtime / 60), 2)) + " minutes", 'blue')) 95 | print('') 96 | print(colored("Average seconds per symbol = " + str(average_runtime) + " seconds", 'blue')) 97 | print('') 98 | print(colored("Estimated seconds until last sentiment = " + (str(average_runtime * i) + " seconds remaining"), 'blue')) 99 | print('') 100 | print('') 101 | i -= 1 102 | sentiment_variables = twitter_sentiment(symbol, 1) 103 | 104 | table_names[symbol]['Tweet_Sentiment_Polarity'] = sentiment_variables[0] 105 | table_names[symbol]['Tweet_Sentiment_Subjectivity'] = sentiment_variables[1] 106 | table_names[symbol]['Tweet_Positive_Percent'] = sentiment_variables[2] 107 | table_names[symbol]['Tweet_Sentiment_STDDEV'] = sentiment_variables[3] 108 | table_names[symbol]['Tweet_Sentiment_Polarity_to'] = sentiment_variables[4] 109 | table_names[symbol]['Tweet_Sentiment_Subjectivity_to'] = sentiment_variables[5] 110 | table_names[symbol]['Tweet_Positive_Percent_to'] = sentiment_variables[6] 111 | table_names[symbol]['Tweet_Sentiment_STDDEV_to'] = sentiment_variables[7] 112 | table_names[symbol]['Tweet_Sentiment_Polarity_from'] = sentiment_variables[8] 113 | table_names[symbol]['Tweet_Sentiment_Subjectivity_from'] = sentiment_variables[9] 114 | table_names[symbol]['Tweet_Positive_Percent_from'] = sentiment_variables[10] 115 | table_names[symbol]['Tweet_Sentiment_STDDEV_from'] = sentiment_variables[11] 116 | print(table_names) 117 | return table_names 118 | 119 | 120 | 121 | 122 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Brandon Shaw 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Crypto_Trader 2 | **A Live Machine-Learning based Cryptocurrency Trader for the Poloniex Exchange** 3 | 4 | If you would like to follow our progress or reach out to developers, feel free to join our discord channel at: 5 | https://discord.gg/UPNV2fH 6 | 7 | The goal of the project is to create an open sourced, machine learning based cryptocurrency portfolio optimizer including as many relevant variables as possible. 8 | 9 | These will include: 10 | 11 | - **Community Sentiment** (such as forum and twitter sentiment analysis) 12 | - **Analyst Opinions** (notable twitter account/analyst feeds with a history of winning strategies - ex. "Haejin" a successful elliot wave trader with an established history) 13 | - **Global Economic Indexes** (DOW, Nikkei225, FOREX data, etc.) 14 | - **Open/Close/High/Low/Volume** 15 | - **Live Orderbook Data** (spread, bid, ask, order size) 16 | - **Technical Indicators** (MACD's, MA's, VWAPS, etc.) 17 | - **Any Other Variable** of interest found to be important (Please feel free to brainstorm and send suggestions) 18 | 19 | 20 | A **Boruta Analysis** (variation of Random forest) will be used to reduce these input variables by removing "unimportant" features. 21 | 22 | Using this input dataset, a machine learning **Binary Classifier** will be created to assign trading pairs on Poloniex a confidence score from 0 to 1. This value will represent the confidence that the trading pair will increase over a certain timeframe. Multiple machines may be constructed to score for a 5 minute window, 10 minute, 15 minute, 30 minute, 1 hour, etc. 23 | 24 | Using these scores, a **Q-Learning Bot** ("reinforcement learning") will be created that will optimize a trading strategy based on the binary classifier scores. The machine will read the amount of capital in the users Poloniex Account, and automatically place trades to optimize the portfolios holdings. These strategies will use stop losses and sell limits. Because it is q learning based, the machine will receive, and use live data to make decisions in a live auto-updating context with a reward that optimizes profit. This will allow the machine to continue to train and optimize itself over time while feeding in live data and placing trades. 25 | 26 | 27 | Ultimately, the program will have the ability to be run 24 hours a day while optimizing a portfolio using live data, while taking into account fees in order to identify and take advantage of all trading opportunities accross the entire cryptocurrency market on Poloniex. Because of the decimal based system of cryptocurrencies, in theory a portfolio of any size should be able to be used as long as the user has the minimum trade size on Poloniex. 28 | 29 | 30 | 31 | | Major Design Features | Purpose | 32 | | --------------------- |:------------------------:| 33 | | Identified Variables | Related to Price Action | 34 | | Data Scraper | Live Variables to Array | 35 | | Binary Classifers | Score Trading Pairs Live | 36 | | Q-Learning Bot | Optimize Trade Strategy | 37 | | Poloniex API Link | Allow Bot to Make Trades | 38 | 39 | 40 | 41 | If you would like to donate to the project, please do so at the following Bitcoin/Litecoin/Ethereum addresses. All donations appreciated :) 42 | 43 | **DONATIONS** 44 | 45 | | Currency | Address | 46 | | -------- |:-----------------------------------------: | 47 | | Bitcoin | 1GjVgMUDfKHzhxgeauRagVfp1GCrSJXijb | 48 | | Litecoin | 0x9852389Bd431A90A9AEcb48EdA50Da1ac05Bd4d8 | 49 | | Ethereum | M9oJaUnCB6Soistk3wSETziFDzz8gAaJCU | 50 | 51 | -------------------------------------------------------------------------------- /Table_Timer.py: -------------------------------------------------------------------------------- 1 | from Inputs_Table_Builder import * 2 | import time 3 | import datetime 4 | import sqlite3 5 | import pandas 6 | 7 | starttime=time.time() 8 | 9 | #Runs the Input Table Builder module in 5 minute timesteps, appending to database 10 | while True: 11 | ts = time.time() 12 | ts = datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S') 13 | print("Scraping at: " + ts) 14 | input_tables = generate_input_dataset('databases/market_prices.db') 15 | db_connection = sqlite3.connect('databases/market_prices.db') 16 | cursor = db_connection.cursor() 17 | i = len(symbols) 18 | for symbol in symbols: 19 | split=symbol.split('/') 20 | try: 21 | cursor.execute("select Tweet_Sentiment_Polarity_to from '" + symbol + "';") 22 | except: 23 | cursor.execute("Alter table '" + symbol + "' RENAME TO " + split[0] + split[1] + "_1old") 24 | try: 25 | input_tables[symbol]['Tweet_Sentiment_Polarity'] 26 | except: 27 | input_tables[symbol]['Tweet_Sentiment_Polarity'] = 0 28 | try: 29 | input_tables[symbol]['Tweet_Sentiment_Subjectivity'] 30 | except: 31 | input_tables[symbol]['Tweet_Sentiment_Subjectivity'] = 0 32 | try: 33 | input_tables[symbol]['Tweet_Positive_Percent'] 34 | except: 35 | input_tables[symbol]['Tweet_Positive_Percent'] = 0 36 | try: 37 | input_tables[symbol]['Tweet_Sentiment_STDDEV'] 38 | except: 39 | input_tables[symbol]['Tweet_Sentiment_STDDEV'] = 0 40 | 41 | try: 42 | input_tables[symbol]['Tweet_Sentiment_Polarity_to'] 43 | except: 44 | input_tables[symbol]['Tweet_Sentiment_Polarity_to'] = 0 45 | try: 46 | input_tables[symbol]['Tweet_Sentiment_Subjectivity_to'] 47 | except: 48 | input_tables[symbol]['Tweet_Sentiment_Subjectivity_to'] = 0 49 | try: 50 | input_tables[symbol]['Tweet_Positive_Percent_to'] 51 | except: 52 | input_tables[symbol]['Tweet_Positive_Percent_to'] = 0 53 | try: 54 | input_tables[symbol]['Tweet_Sentiment_STDDEV_to'] 55 | except: 56 | input_tables[symbol]['Tweet_Sentiment_STDDEV_to'] = 0 57 | try: 58 | input_tables[symbol]['Tweet_Sentiment_Polarity_from'] 59 | except: 60 | input_tables[symbol]['Tweet_Sentiment_Polarity_from'] = 0 61 | try: 62 | input_tables[symbol]['Tweet_Sentiment_Subjectivity_from'] 63 | except: 64 | input_tables[symbol]['Tweet_Sentiment_Subjectivity_from'] = 0 65 | try: 66 | input_tables[symbol]['Tweet_Positive_Percent_from'] 67 | except: 68 | input_tables[symbol]['Tweet_Sentiment_STDDEV_from'] = 0 69 | try: 70 | input_tables[symbol]['Tweet_Sentiment_STDDEV_from'] 71 | except: 72 | input_tables[symbol]['Tweet_Sentiment_STDDEV_from'] = 0 73 | input_tables[symbol].to_sql(name=(str(symbol)), con=db_connection, if_exists = 'append', index=False) 74 | ts = time.time() 75 | 76 | time.sleep(300.0 - ((time.time() - starttime) % 300.0)) 77 | ts = datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S') 78 | print("Saved to database at: " + ts) 79 | print(input_tables) 80 | -------------------------------------------------------------------------------- /Variable_Functions/Technical_Indicators.py: -------------------------------------------------------------------------------- 1 | from crycompare import * 2 | import pandas as pd 3 | import sqlite3 4 | 5 | db_connection = sqlite3.connect('databases/market_prices.db') 6 | cursor = db_connection.cursor() 7 | cursor.execute("SELECT name FROM sqlite_master WHERE type='table';") 8 | symbols1 = cursor.fetchall() 9 | symbols = [] 10 | for symbol in symbols1: 11 | symbols.append(symbol[0]) 12 | print(symbols) 13 | 14 | h = History() 15 | 16 | from_coins = [] 17 | to_coins = [] 18 | 19 | df_dict = {} 20 | 21 | for coin in coins: 22 | histo = h.histoMinute(coin,'USD') 23 | if histo['Data']: 24 | df_histo = pd.DataFrame(histo['Data']) 25 | df_histo['time'] = pd.to_datetime(df_histo['time'],unit='s') 26 | df_histo.index = df_histo['time'] 27 | del df_histo['time'] 28 | del df_histo['volumefrom'] 29 | del df_histo['volumeto'] 30 | 31 | df_dict[coin] = df_histo 32 | print(df_dict[coin]) -------------------------------------------------------------------------------- /Variable_Functions/Twitter_Sentiment.py: -------------------------------------------------------------------------------- 1 | import got3 2 | import arrow 3 | from textblob import TextBlob 4 | import numpy as np 5 | from termcolor import colored 6 | 7 | 8 | 9 | 10 | 11 | def dates_to_sentiment(dates, ticker, max_tweets): 12 | 13 | ticker = ticker 14 | print(colored("Calculating sentiment for:" + ticker, 'white')) 15 | sentiments = [] 16 | positives = [] 17 | negatives = [] 18 | arrow_date = dates 19 | tweetCriteria = got3.manager.TweetCriteria().setQuerySearch("{}{}".format("#", ticker)).setMaxTweets(max_tweets) 20 | tweets = got3.manager.TweetManager.getTweets(tweetCriteria) 21 | sents_per_date = [] 22 | subjectivity = [] 23 | for t in tweets: 24 | blob = TextBlob(t.text) 25 | sent = blob.sentiment[0] #get the polarity 26 | subjectives = blob.sentiment[1] #get the subjectivity 27 | sents_per_date.append(sent) #Saving polarity to sents_per_date 28 | subjectivity.append(subjectives) #Saving subjectivity to subjectivity 29 | if blob.sentiment[0] > 0: #Separating positive and negative tweets to lists 30 | positives.append(t) 31 | else: 32 | negatives.append(t) 33 | standard_dev_array = np.asarray(sents_per_date) 34 | if len(sents_per_date) >= 1: 35 | mean_polarity = sum(sents_per_date) / len(sents_per_date) 36 | mean_subjectivity = sum(subjectivity) / len(sents_per_date) 37 | percent_positive = len(positives) / len(sents_per_date) 38 | standard_deviation_polarity = np.std(standard_dev_array) 39 | else: 40 | mean_polarity = 0 41 | mean_subjectivity = 0 42 | percent_positive = .5 43 | standard_deviation_polarity = 0 44 | #Mean Polarity 45 | 46 | try: 47 | sentiments.append(mean_polarity) 48 | except: 49 | sentiments.append(0) 50 | #Mean Subjectivity 51 | try: 52 | sentiments.append(mean_subjectivity) 53 | except: 54 | sentiments.append(0) 55 | #Percentage of Tweets that are positive 56 | try: 57 | sentiments.append(percent_positive) 58 | except: 59 | sentiments.append(0.5) 60 | #Standard Deviation of tweet sentiment Polarity 61 | try: 62 | sentiments.append(standard_deviation_polarity) 63 | except: 64 | sentiments.append(0) 65 | 66 | 67 | 68 | 69 | split_symbol = ticker.split('/') 70 | 71 | 72 | ticker = split_symbol[1] 73 | print(colored("Calculating sentiment for:" + ticker, 'red')) 74 | positives = [] 75 | negatives = [] 76 | subjectivity = [] 77 | sents_per_date = [] 78 | tweetCriteria = got3.manager.TweetCriteria().setQuerySearch("{}{}".format("#", ticker)).setMaxTweets(max_tweets) 79 | tweets = got3.manager.TweetManager.getTweets(tweetCriteria) 80 | 81 | for t in tweets: 82 | blob = TextBlob(t.text) 83 | sent = blob.sentiment[0] #get the polarity 84 | subjectives = blob.sentiment[1] #get the subjectivity 85 | sents_per_date.append(sent) #Saving polarity to sents_per_date 86 | subjectivity.append(subjectives) #Saving subjectivity to subjectivity 87 | 88 | if blob.sentiment[0] > 0: #Separating positive and negative tweets to lists 89 | positives.append(t) 90 | else: 91 | negatives.append(t) 92 | standard_dev_array = np.asarray(sents_per_date) 93 | 94 | if len(sents_per_date) >= 1: 95 | mean_polarity_from = sum(sents_per_date) / len(sents_per_date) 96 | mean_subjectivity_from = sum(subjectivity) / len(sents_per_date) 97 | percent_positive_from = len(positives) / len(sents_per_date) 98 | standard_deviation_polarity_from = np.std(standard_dev_array) 99 | else: 100 | mean_polarity_from = 0 101 | mean_subjectivity_from = 0 102 | percent_positive_from = .5 103 | standard_deviation_polarity_from = 0 104 | 105 | #Mean Polarity 106 | try: 107 | sentiments.append(mean_polarity_from) 108 | except: 109 | sentiments.append(0) 110 | #Mean Subjectivity 111 | try: 112 | sentiments.append(mean_subjectivity_from) 113 | except: 114 | sentiments.append(0) 115 | #Percentage of Tweets that are positive 116 | try: 117 | sentiments.append(percent_positive_from) 118 | except: 119 | sentiments.append(0.5) 120 | #Standard Deviation of tweet sentiment Polarity 121 | try: 122 | sentiments.append(standard_deviation_polarity_from) 123 | except: 124 | sentiments.append(0) 125 | 126 | 127 | 128 | 129 | ticker = split_symbol[0] 130 | print(colored("Calculating sentiment for:" + ticker, 'green')) 131 | positives = [] 132 | negatives = [] 133 | tweetCriteria = got3.manager.TweetCriteria().setQuerySearch("{}{}".format("#", ticker)).setMaxTweets(max_tweets) 134 | tweets = got3.manager.TweetManager.getTweets(tweetCriteria) 135 | sents_per_date = [] 136 | subjectivity = [] 137 | for t in tweets: 138 | blob = TextBlob(t.text) 139 | sent = blob.sentiment[0] #get the polarity 140 | subjectives = blob.sentiment[1] #get the subjectivity 141 | sents_per_date.append(sent) #Saving polarity to sents_per_date 142 | subjectivity.append(subjectives) #Saving subjectivity to subjectivity 143 | 144 | if blob.sentiment[0] > 0: #Separating positive and negative tweets to lists 145 | positives.append(t) 146 | else: 147 | negatives.append(t) 148 | standard_dev_array = np.asarray(sents_per_date) 149 | if len(sents_per_date) >= 1: 150 | mean_polarity_to = sum(sents_per_date) / len(sents_per_date) 151 | mean_subjectivity_to = sum(subjectivity) / len(sents_per_date) 152 | percent_positive_to = len(positives) / len(sents_per_date) 153 | standard_deviation_polarity_to = np.std(standard_dev_array) 154 | else: 155 | mean_polarity_to = 0 156 | mean_subjectivity_to = 0 157 | percent_positive_to = .5 158 | standard_deviation_polarity_to = 0 159 | #Mean Polarity 160 | try: 161 | sentiments.append(mean_polarity_to) 162 | except: 163 | sentiments.append(0) 164 | #Mean Subjectivity 165 | try: 166 | sentiments.append(mean_subjectivity_to) 167 | except: 168 | sentiments.append(0) 169 | #Percentage of Tweets that are positive 170 | try: 171 | sentiments.append(percent_positive_to) 172 | except: 173 | sentiments.append(0.5) 174 | #Standard Deviation of tweet sentiment Polarity 175 | try: 176 | sentiments.append(standard_deviation_polarity_to) 177 | except: 178 | sentiments.append(0) 179 | 180 | 181 | 182 | 183 | sentiments = np.asarray(sentiments) 184 | 185 | return sentiments -------------------------------------------------------------------------------- /Variable_Functions/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bshaw2019/Crypto_Trader/ab7497cd37284e69c67271757960a1fdb9e60c13/Variable_Functions/__init__.py -------------------------------------------------------------------------------- /crycompare.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import requests 3 | import warnings 4 | 5 | 6 | class Price: 7 | def __init__(self): 8 | self.__coinlisturl = 'https://www.cryptocompare.com/api/data/coinlist/' 9 | self.__priceurl = 'https://min-api.cryptocompare.com/data/price?' 10 | self.__pricemultiurl = 'https://min-api.cryptocompare.com/data/pricemulti?' 11 | self.__pricemultifullurl = 'https://min-api.cryptocompare.com/data/pricemultifull?' 12 | self.__generateavgurl = 'https://min-api.cryptocompare.com/data/generateAvg?' 13 | self.__dayavgurl = 'https://min-api.cryptocompare.com/data/dayAvg?' 14 | self.__historicalurl = 'https://min-api.cryptocompare.com/data/pricehistorical?' 15 | self.__coinsnapshoturl = 'https://www.cryptocompare.com/api/data/coinsnapshot/?' 16 | self.__coinsnapshotfull = 'https://www.cryptocompare.com/api/data/coinsnapshotfullbyid/?' 17 | 18 | def coinList(self): 19 | return self.__get_url(self.__coinlisturl) 20 | 21 | def price(self, from_curr, to_curr, e=None, extraParams=None, sign=False, tryConversion=True): 22 | return self.__get_price(self.__priceurl, from_curr, to_curr, e, extraParams, sign, tryConversion) 23 | 24 | def priceMulti(self, from_curr, to_curr, e=None, extraParams=None, sign=False, tryConversion=True): 25 | return self.__get_price(self.__pricemultiurl, from_curr, to_curr, e, extraParams, sign, tryConversion) 26 | 27 | def priceMultiFull(self, from_curr, to_curr, e=None, extraParams=None, sign=False, tryConversion=True): 28 | return self.__get_price(self.__pricemultifullurl, from_curr, to_curr, e, extraParams, sign, tryConversion) 29 | 30 | def priceHistorical(self, from_curr, to_curr, markets, ts=None, e=None, extraParams=None, 31 | sign=False, tryConversion=True): 32 | return self.__get_price(self.__historicalurl, from_curr, to_curr, markets, e, extraParams, sign, tryConversion, ts) 33 | 34 | def generateAvg(self, from_curr, to_curr, markets, extraParams=None, sign=False, tryConversion=True): 35 | return self.__get_avg(self.__generateavgurl, from_curr, to_curr, markets, extraParams, sign, tryConversion) 36 | 37 | def dayAvg(self, from_curr, to_curr, e=None, extraParams=None, sign=False, tryConversion=True, 38 | avgType=None, UTCHourDiff=0, toTs=None): 39 | return self.__get_avg(self.__dayavgurl, from_curr, to_curr, e, extraParams, sign, 40 | tryConversion, avgType, UTCHourDiff, toTs) 41 | 42 | def coinSnapshot(self, from_curr, to_curr): 43 | return self.__get_url(self.__coinsnapshoturl + 'fsym=' + from_curr.upper() + '&tsym=' + to_curr.upper()) 44 | 45 | def coinSnapshotFullById(self, coin_id): 46 | return self.__get_url(self.__coinsnapshotfull + 'id=' + str(coin_id)) 47 | 48 | def __get_price(self, baseurl, from_curr, to_curr, e=None, extraParams=None, sign=False, 49 | tryConversion=True, markets=None, ts=None): 50 | args = list() 51 | if isinstance(from_curr, str): 52 | args.append('fsym=' + from_curr.upper()) 53 | elif isinstance(from_curr, list): 54 | args.append('fsyms=' + ','.join(from_curr).upper()) 55 | if isinstance(to_curr, list): 56 | args.append('tsyms=' + ','.join(to_curr).upper()) 57 | elif isinstance(to_curr, str): 58 | args.append('tsyms=' + to_curr.upper()) 59 | if isinstance(markets, str): 60 | args.append('markets=' + markets) 61 | elif isinstance(markets, list): 62 | args.append('markets=' + ','.join(markets)) 63 | if e: 64 | args.append('e=' + e) 65 | if extraParams: 66 | args.append('extraParams=' + extraParams) 67 | if sign: 68 | args.append('sign=true') 69 | if ts: 70 | args.append('ts=' + str(ts)) 71 | if not tryConversion: 72 | args.append('tryConversion=false') 73 | if len(args) >= 2: 74 | return self.__get_url(baseurl + '&'.join(args)) 75 | else: 76 | raise ValueError('Must have both fsym and tsym arguments.') 77 | 78 | def __get_avg(self, baseurl, from_curr, to_curr, markets=None, e=None, extraParams=None, 79 | sign=False, tryConversion=True, avgType=None, UTCHourDiff=0, toTs=None): 80 | args = list() 81 | if isinstance(from_curr, str): 82 | args.append('fsym=' + from_curr.upper()) 83 | if isinstance(to_curr, str): 84 | args.append('tsym=' + to_curr.upper()) 85 | if isinstance(markets, str): 86 | args.append('markets=' + markets) 87 | elif isinstance(markets, list): 88 | args.append('markets=' + ','.join(markets)) 89 | if e: 90 | args.append('e=' + e) 91 | if extraParams: 92 | args.append('extraParams=' + extraParams) 93 | if sign: 94 | args.append('sign=true') 95 | if avgType: 96 | args.append('avgType=' + avgType) 97 | if UTCHourDiff: 98 | args.append('UTCHourDiff=' + str(UTCHourDiff)) 99 | if toTs: 100 | args.append('toTs=' + toTs) 101 | if not tryConversion: 102 | args.append('tryConversion=false') 103 | if len(args) >= 2: 104 | return self.__get_url(baseurl + '&'.join(args)) 105 | else: 106 | raise ValueError('Must have both fsym and tsym arguments.') 107 | 108 | def __get_url(self, url): 109 | raw_data = requests.get(url) 110 | raw_data.encoding = 'utf-8' 111 | if raw_data.status_code != 200: 112 | raw_data.raise_for_status() 113 | return False 114 | try: 115 | if isinstance(raw_data.text, unicode): 116 | warnings.warn('Object returned is of type unicode. Cannot parse to str in Python 2.') 117 | except NameError: 118 | pass 119 | return raw_data.json() 120 | 121 | 122 | class History: 123 | def __init__(self): 124 | self.__histominuteurl = 'https://min-api.cryptocompare.com/data/histominute?' 125 | self.__histohoururl = 'https://min-api.cryptocompare.com/data/histohour?' 126 | self.__histodayurl = 'https://min-api.cryptocompare.com/data/histoday?' 127 | 128 | def histoMinute(self, from_curr, to_curr, e=None, extraParams=None, 129 | sign=False, tryConversion=True, aggregate=None, limit=None, toTs=None): 130 | return self.__get_price(self.__histominuteurl, from_curr, to_curr, e, extraParams, sign, 131 | tryConversion, aggregate, limit, toTs) 132 | 133 | def histoHour(self, from_curr, to_curr, e=None, extraParams=None, 134 | sign=False, tryConversion=True, aggregate=None, limit=None, toTs=None): 135 | return self.__get_price(self.__histohoururl, from_curr, to_curr, e, extraParams, sign, 136 | tryConversion, aggregate, limit, toTs) 137 | 138 | def histoDay(self, from_curr, to_curr, e=None, extraParams=None, sign=False, 139 | tryConversion=True, aggregate=None, limit=None, toTs=None, allData=False): 140 | return self.__get_price(self.__histodayurl, from_curr, to_curr, e, extraParams, sign, 141 | tryConversion, aggregate, limit, toTs, allData) 142 | 143 | def __get_price(self, baseurl, from_curr, to_curr, e=None, extraParams=None, sign=False, 144 | tryConversion=True, aggregate=None, limit=None, toTs=None, allData=False): 145 | args = list() 146 | if isinstance(from_curr, str): 147 | args.append('fsym=' + from_curr.upper()) 148 | if isinstance(to_curr, str): 149 | args.append('tsym=' + to_curr.upper()) 150 | if e: 151 | args.append('e=' + e) 152 | if extraParams: 153 | args.append('extraParams=' + extraParams) 154 | if sign: 155 | args.append('sign=true') 156 | if aggregate: 157 | args.append('aggregate=' + str(aggregate)) 158 | if limit: 159 | args.append('limit=' + str(limit)) 160 | if toTs: 161 | args.append('toTs=' + str(toTs)) 162 | if allData: 163 | args.append('allData=true') 164 | if not tryConversion: 165 | args.append('tryConversion=false') 166 | if len(args) >= 2: 167 | return self.__get_url(baseurl + '&'.join(args)) 168 | else: 169 | raise ValueError('Must have both fsym and tsym arguments.') 170 | 171 | def __get_url(self, url): 172 | raw_data = requests.get(url) 173 | raw_data.encoding = 'utf-8' 174 | if raw_data.status_code != 200: 175 | raw_data.raise_for_status() 176 | return False 177 | try: 178 | if isinstance(raw_data.text, unicode): 179 | warnings.warn('Object returned is of type unicode. Cannot parse to str in Python 2.') 180 | except NameError: 181 | pass 182 | return raw_data.json() 183 | -------------------------------------------------------------------------------- /got3/__init__.py: -------------------------------------------------------------------------------- 1 | from . import models 2 | from . import manager -------------------------------------------------------------------------------- /got3/manager/TweetCriteria.py: -------------------------------------------------------------------------------- 1 | class TweetCriteria: 2 | 3 | def __init__(self): 4 | self.maxTweets = 0 5 | 6 | def setUsername(self, username): 7 | self.username = username 8 | return self 9 | 10 | def setSince(self, since): 11 | self.since = since 12 | return self 13 | 14 | def setUntil(self, until): 15 | self.until = until 16 | return self 17 | 18 | def setQuerySearch(self, querySearch): 19 | self.querySearch = querySearch 20 | return self 21 | 22 | def setMaxTweets(self, maxTweets): 23 | self.maxTweets = maxTweets 24 | return self 25 | 26 | def setLang(self, Lang): 27 | self.lang = Lang 28 | return self 29 | 30 | def setTopTweets(self, topTweets): 31 | self.topTweets = topTweets 32 | return self -------------------------------------------------------------------------------- /got3/manager/TweetManager.py: -------------------------------------------------------------------------------- 1 | import urllib.request, urllib.parse, urllib.error, urllib.request, urllib.error, urllib.parse, json, re, datetime, sys, \ 2 | http.cookiejar 3 | from .. import models 4 | from pyquery import PyQuery 5 | 6 | 7 | class TweetManager: 8 | def __init__(self): 9 | pass 10 | 11 | @staticmethod 12 | def getTweets(tweetCriteria, receiveBuffer=None, bufferLength=100, proxy=None): 13 | refreshCursor = '' 14 | 15 | results = [] 16 | resultsAux = [] 17 | cookieJar = http.cookiejar.CookieJar() 18 | 19 | active = True 20 | 21 | while active: 22 | json = TweetManager.getJsonReponse(tweetCriteria, refreshCursor, cookieJar, proxy) 23 | if len(json['items_html'].strip()) == 0: 24 | break 25 | 26 | refreshCursor = json['min_position'] 27 | tweets = PyQuery(json['items_html'])('div.js-stream-tweet') 28 | 29 | if len(tweets) == 0: 30 | break 31 | 32 | for tweetHTML in tweets: 33 | tweetPQ = PyQuery(tweetHTML) 34 | tweet = models.Tweet() 35 | 36 | usernameTweet = tweetPQ("span.username.js-action-profile-name b").text(); 37 | txt = re.sub(r"\s+", " ", tweetPQ("p.js-tweet-text").text().replace('# ', '#').replace('@ ', '@')); 38 | retweets = int(tweetPQ("span.ProfileTweet-action--retweet span.ProfileTweet-actionCount").attr( 39 | "data-tweet-stat-count").replace(",", "")); 40 | favorites = int(tweetPQ("span.ProfileTweet-action--favorite span.ProfileTweet-actionCount").attr( 41 | "data-tweet-stat-count").replace(",", "")); 42 | dateSec = int(tweetPQ("small.time span.js-short-timestamp").attr("data-time")); 43 | id = tweetPQ.attr("data-tweet-id"); 44 | permalink = tweetPQ.attr("data-permalink-path"); 45 | user_id = int(tweetPQ("a.js-user-profile-link").attr("data-user-id")) 46 | 47 | geo = '' 48 | geoSpan = tweetPQ('span.Tweet-geo') 49 | if len(geoSpan) > 0: 50 | geo = geoSpan.attr('title') 51 | urls = [] 52 | for link in tweetPQ("a"): 53 | try: 54 | urls.append((link.attrib["data-expanded-url"])) 55 | except KeyError: 56 | pass 57 | tweet.id = id 58 | tweet.permalink = 'https://twitter.com' + permalink 59 | tweet.username = usernameTweet 60 | 61 | tweet.text = txt 62 | tweet.date = datetime.datetime.fromtimestamp(dateSec) 63 | tweet.formatted_date = datetime.datetime.fromtimestamp(dateSec).strftime("%a %b %d %X +0000 %Y") 64 | tweet.retweets = retweets 65 | tweet.favorites = favorites 66 | tweet.mentions = " ".join(re.compile('(@\\w*)').findall(tweet.text)) 67 | tweet.hashtags = " ".join(re.compile('(#\\w*)').findall(tweet.text)) 68 | tweet.geo = geo 69 | tweet.urls = ",".join(urls) 70 | tweet.author_id = user_id 71 | 72 | results.append(tweet) 73 | resultsAux.append(tweet) 74 | 75 | if receiveBuffer and len(resultsAux) >= bufferLength: 76 | receiveBuffer(resultsAux) 77 | resultsAux = [] 78 | 79 | if tweetCriteria.maxTweets > 0 and len(results) >= tweetCriteria.maxTweets: 80 | active = False 81 | break 82 | 83 | if receiveBuffer and len(resultsAux) > 0: 84 | receiveBuffer(resultsAux) 85 | 86 | return results 87 | 88 | @staticmethod 89 | def getJsonReponse(tweetCriteria, refreshCursor, cookieJar, proxy): 90 | url = "https://twitter.com/i/search/timeline?f=tweets&q=%s&l=en&src=typd&%smax_position=%s" 91 | 92 | urlGetData = '' 93 | if hasattr(tweetCriteria, 'username'): 94 | urlGetData += ' from:' + tweetCriteria.username 95 | 96 | if hasattr(tweetCriteria, 'since'): 97 | urlGetData += ' since:' + tweetCriteria.since 98 | 99 | if hasattr(tweetCriteria, 'until'): 100 | urlGetData += ' until:' + tweetCriteria.until 101 | 102 | if hasattr(tweetCriteria, 'querySearch'): 103 | urlGetData += ' ' + tweetCriteria.querySearch 104 | 105 | if hasattr(tweetCriteria, 'lang'): 106 | urlLang = 'lang=' + tweetCriteria.lang + '&' 107 | else: 108 | urlLang = '' 109 | url = url % (urllib.parse.quote(urlGetData), urlLang, refreshCursor) 110 | #comment out to hide url 111 | #print(url) 112 | 113 | headers = [ 114 | ('Host', "twitter.com"), 115 | ('User-Agent', "Mozilla/5.0 (Windows NT 6.1; Win64; x64)"), 116 | ('Accept', "application/json, text/javascript, */*; q=0.01"), 117 | ('Accept-Language', "de,en-US;q=0.7,en;q=0.3"), 118 | ('X-Requested-With', "XMLHttpRequest"), 119 | ('Referer', url), 120 | ('Connection', "keep-alive") 121 | ] 122 | 123 | if proxy: 124 | opener = urllib.request.build_opener(urllib.request.ProxyHandler({'http': proxy, 'https': proxy}), \ 125 | urllib.request.HTTPCookieProcessor(cookieJar)) 126 | else: 127 | opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cookieJar)) 128 | opener.addheaders = headers 129 | 130 | try: 131 | response = opener.open(url) 132 | jsonResponse = response.read() 133 | #comment out once done debugging 134 | # print( 135 | # "Twitter weird response. Try to see on browser: https://twitter.com/search?q=%s&src=typd" % urllib.parse.quote( 136 | # urlGetData)) 137 | except: 138 | # print("Twitter weird response. Try to see on browser: ", url) 139 | print( 140 | "Twitter weird response. Try to see on browser: https://twitter.com/search?q=%s&src=typd" % urllib.parse.quote( 141 | urlGetData)) 142 | print("Unexpected error:", sys.exc_info()[0]) 143 | sys.exit() 144 | return 145 | 146 | dataJson = json.loads(jsonResponse.decode()) 147 | 148 | return dataJson 149 | -------------------------------------------------------------------------------- /got3/manager/__init__.py: -------------------------------------------------------------------------------- 1 | from .TweetCriteria import TweetCriteria 2 | from .TweetManager import TweetManager -------------------------------------------------------------------------------- /got3/manager/__init__.py.bak: -------------------------------------------------------------------------------- 1 | from TweetCriteria import TweetCriteria 2 | from TweetManager import TweetManager -------------------------------------------------------------------------------- /got3/models/Tweet.py: -------------------------------------------------------------------------------- 1 | class Tweet: 2 | 3 | def __init__(self): 4 | pass -------------------------------------------------------------------------------- /got3/models/__init__.py: -------------------------------------------------------------------------------- 1 | from .Tweet import Tweet -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | alembic==0.9.8 2 | banal==0.3.3 3 | ccxt==1.10.1242 4 | certifi==2018.1.18 5 | chardet==3.0.4 6 | click==6.7 7 | dash==0.21.0 8 | dash-core-components==0.19.0 9 | dash-html-components==0.9.0 10 | dash-renderer==0.11.3 11 | dataset==1.0.6 12 | decorator==4.2.1 13 | enum34==1.1.6 14 | Flask==0.12.2 15 | Flask-Compress==1.4.0 16 | functools32==3.2.3.post2 17 | idna==2.6 18 | ipython-genutils==0.2.0 19 | itsdangerous==0.24 20 | Jinja2==2.10 21 | jsonschema==2.6.0 22 | jupyter-core==4.4.0 23 | Mako==1.0.7 24 | MarkupSafe==1.0 25 | nbformat==4.4.0 26 | normality==0.5.11 27 | numpy==1.14.1 28 | pandas==0.22.0 29 | plotly==2.4.1 30 | python-dateutil==2.6.1 31 | python-editor==1.0.3 32 | pytz==2018.3 33 | requests==2.18.4 34 | six==1.11.0 35 | SQLAlchemy==1.2.4 36 | traitlets==4.3.2 37 | urllib3==1.22 38 | Werkzeug==0.14.1 39 | -------------------------------------------------------------------------------- /requirements_to_freeze.txt: -------------------------------------------------------------------------------- 1 | ccxt==1.10.1242 2 | dash==0.21.0 3 | dash-core-components==0.19.0 4 | dash-html-components==0.9.0 5 | dash-renderer==0.11.3 6 | dataset==1.0.6 7 | pandas==0.22.0 8 | --------------------------------------------------------------------------------