├── .gitignore ├── README.md ├── docs ├── platform_diagram.png ├── platform_diagram_1.png ├── platform_diagram_2.png └── platform_diagram_jp.png └── src ├── __init__.py ├── backtesting ├── README.md ├── backtest_broker.py ├── csv_parse.py └── strategies │ └── demo.py ├── broker.py ├── candle.py ├── chart.py ├── code_timer.py ├── config.py ├── config_nonsecure.cfg ├── currency_pair_conversions.py ├── currency_pair_precision.py ├── daemon.py ├── db.py ├── instrument.py ├── log.py ├── main.py ├── oanda.py ├── opportunity.py ├── order.py ├── requirements.txt ├── run_backtest.py ├── scripts ├── daemon.sh ├── daemon_prototype.sh ├── db_create_backup.sh ├── db_restore_backup.sh ├── emailer.py ├── get_missing.sh ├── load_csv │ ├── README.md │ ├── algo_fun_fix_date.mysql │ ├── algo_fun_fix_time.mysql │ ├── algo_proc_create_table_stock.mysql │ ├── algo_proc_drop_table.mysql │ ├── algo_proc_load_csv.mysql │ ├── load_csv.sh │ ├── main.sh │ └── test │ │ └── create_function_test.mysql ├── mount.sh ├── packet_analysis │ └── watch_traffic.sh ├── symbol_lists │ └── sp500_index.txt ├── unmount.sh └── unzip_all.sh ├── strategies ├── __init__.py └── fifty.py ├── strategy.py ├── tests ├── __init__.py ├── run_tests.sh ├── test_chart.py ├── test_daemon.py ├── test_db.py ├── test_instrument.py ├── test_oanda.py ├── test_opportunity.py ├── test_order.py ├── test_strategy.py ├── test_timer.py ├── test_trade.py └── test_util_date.py ├── timer.py ├── trade.py ├── util_date.py └── utils.py /.gitignore: -------------------------------------------------------------------------------- 1 | # package files 2 | *.deb 3 | # log output 4 | output* 5 | # no need to save temporary output files 6 | backtesting/public/reversal/out/* 7 | #swap files from vim 8 | *.swp 9 | # private anything 10 | priv* 11 | # python 12 | __pycache__ 13 | # backupfiles 14 | *.bak 15 | # CSV files 16 | csv/ 17 | 18 | # Compiled Object files 19 | *.slo 20 | *.lo 21 | *.o 22 | *.obj 23 | 24 | # Precompiled Headers 25 | *.gch 26 | *.pch 27 | 28 | # Compiled Dynamic libraries 29 | *.so 30 | *.dylib 31 | *.dll 32 | 33 | # Fortran module files 34 | *.mod 35 | 36 | # Compiled Static libraries 37 | *.lai 38 | *.la 39 | *.a 40 | *.lib 41 | 42 | # Executables 43 | *.exe 44 | *.out 45 | *.app 46 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 日本語図表：[こちら](docs/platform_diagram_jp.png) 2 | 3 | 4 | # Algorithmic Trading Platform 5 | 6 | ## What is this? 7 | 8 | This project is a modular platform for running multiple algorithmic trading strategies concurrently. 9 | 10 | As of 2018-08-18, the platform is functional. However, it is hardly optimized nor ready for production. A basic backtesting framework is in place; see `/src/backtesting/`. 11 | 12 | You can develop your own strategy module and run it with this platform. Developing a module is very straightforward; copy a strategy from `/src/strategies/` and implement your own logic in `_babysit()` and `_scan()`. 13 | 14 | You can use this platform with any broker API. There are two steps to integrate your broker's API: 15 | 16 | 1. Supply a wrapper module that calls your API and can be called by this platform. Your broker may already provide one. `Oanda.py` exists for Oanda's v20 REST API. 17 | 2. Tweak the generic methods in `broker.py` to call your wrapper and correctly pass return values back to the platform. 18 | 19 | ## Technical introduction 20 | 21 | - This is a command line application for Linux. The Debian operating system is being used for development. 22 | - Python 3.6 is used for the bulk of the program currently (`/src/`). The database is MySQL (`/src/db/`). There are some shell scripts in `/src/scripts/`. 23 | - There are three pieces to this project, as with any algorithmic trading: backtesting, forward testing, and live trading. 24 | 25 | ## How to use 26 | 27 | Before you being using the platform, you will need to do some configuration. 28 | 29 | - You will need to make your own private config file. See `Config.py` for details. 30 | - You will need to create the MySQL database on your machine. There is a backup script for the purpose of recreating the database. It is in `/src/db/db_backup.mysql`. It may or may not be up to date. 31 | - Tweak `daemon.py` to use the strategies you created. 32 | 33 | Run the platform: 34 | 35 | `$ cd src` 36 | `$ python3 main.py` 37 | 38 | Run unit tests: 39 | 40 | `$ cd src` 41 | `$ bash tests/run_tests.sh` 42 | 43 | ## Backtesting 44 | 45 | See `/src/backtesting/` 46 | 47 | ## Forward Testing 48 | 49 | Toggle the `live_trading` setting in the public config file to `False`. 50 | 51 | If False, the Oanda module will use its Practice Mode with fake money. 52 | 53 | ## Live Trading 54 | 55 | Toggle the `live_trading` setting in the public config file to `True`. 56 | 57 | ## Platform Design: Scalability and Modularity 58 | 59 | - Scalability and user-friendliness take priority over speed. This is not intended to be used for high-frequency trading and/or arbitrage. The `chart.py` module, for example, will have some methods to do technical analysis. 60 | - The strategy modules can be used (or not used) arbitrarily. Just modify the startup portion of `daemon.py` to include your module in the list of strategies. 61 | - `daemon.py` and the strategy modules make calls to a generic `broker.py` module, which then delegates the calls to a wrapper module, e.g. `oanda.py` wraps Oanda's API. Having the generic broker layer allows you to conveniently change the broker you use. 62 | - Things like money management and risk management happen in `daemon.py`. The strategy modules are intended to be "dumb" and only observe prices and make trades. 63 | 64 | Here is a diagram of the layers. 65 | 66 | - The daemon listens to the strategies and allocates money to them. 67 | - The daemon does not talk to the broker's API directly. Rather, it uses the generic `broker.py` module. 68 | - The broker module calls the broker-specific wrapper that is being used, e.g. `Oanda.py`. 69 | 70 | ![diagram](docs/platform_diagram_2.png) 71 | 72 | -------------------------------------------------------------------------------- /docs/platform_diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/paperduck/algo/f39f945dcfa0eb739588674308fdf2de66bfd69e/docs/platform_diagram.png -------------------------------------------------------------------------------- /docs/platform_diagram_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/paperduck/algo/f39f945dcfa0eb739588674308fdf2de66bfd69e/docs/platform_diagram_1.png -------------------------------------------------------------------------------- /docs/platform_diagram_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/paperduck/algo/f39f945dcfa0eb739588674308fdf2de66bfd69e/docs/platform_diagram_2.png -------------------------------------------------------------------------------- /docs/platform_diagram_jp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/paperduck/algo/f39f945dcfa0eb739588674308fdf2de66bfd69e/docs/platform_diagram_jp.png -------------------------------------------------------------------------------- /src/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/paperduck/algo/f39f945dcfa0eb739588674308fdf2de66bfd69e/src/__init__.py -------------------------------------------------------------------------------- /src/backtesting/README.md: -------------------------------------------------------------------------------- 1 | ## Backtesting 2 | 3 | 4 | ### Overview 5 | 6 | A basic backtesting framework is in place. You can use it to test strategy modules against past price data. 7 | 8 | The backtesting framework parallels the main platform. That is, `run_backtest.py` acts as the main control point for backtesting, instead of `daemon.py`. `backtest_broker.py` simulates the `broker.py` module. Finally, the strategy modules that you test will be largely the same as a normal strategy module, but modified to use the backtest broker module instead of the regular `broker.py`. 9 | 10 | Multiple CSV files can be used concurrently, if you have a strategy that uses multiple instruments. 11 | 12 | ### How to Use 13 | 14 | To perform a backtest, run `run_backtest.py` like this: 15 | 16 | `$ python3 run_backtest.py` 17 | 18 | ### run_backtest.py 19 | 20 | Main entry point for backtesting. Before running, edit the file to supply it with your CSV filenames. 21 | 22 | ### backtest_broker.py 23 | 24 | You don't really need to do anything with this. It iterates through the CSV file(s) and provides information to `run_backtest.py`. 25 | 26 | ### csv_parser.py 27 | 28 | This is a utility module for parsing CSV files. If your CSV file is not being read correctly, you may need to change this. 29 | 30 | 31 | -------------------------------------------------------------------------------- /src/backtesting/backtest_broker.py: -------------------------------------------------------------------------------- 1 | """ 2 | Broker simulator for backtesting strategies. 3 | Don't use this directly. Use run_broker.py. 4 | """ 5 | # local imports 6 | import csv 7 | from datetime import datetime 8 | import pandas as pd 9 | 10 | # external imports 11 | import backtesting.csv_parse as csv_parse 12 | from log import Log 13 | import multiprocessing as mp 14 | 15 | class BacktestBroker(): 16 | 17 | """ CSV files 18 | { instrument_id:{ 19 | 'filename' : string - path to file 20 | 'data' : DataFrame - data from file 21 | }, ...} 22 | """ 23 | files = {} 24 | 25 | """ {instr_id:{'buffer':,'active':},...} """ 26 | prices = {} 27 | 28 | next_trade_id = 0 29 | initialized = False 30 | time = None # simulated time (time of current candle) 31 | 32 | @classmethod 33 | def init(cls, start, end, files, gap=None): 34 | """ Unlike the normal broker module, this one needs to be initialized manually. 35 | 36 | start - datetime - start of test 37 | end - datetime - end of test 38 | files - {instrument_id:str, ...} - CSV files to open 39 | gap - gap tolerance; timespan OK to skip when gap occurs between candles 40 | """ 41 | if cls.initialized: raise Exception 42 | if len(files) < 1: 43 | print('BacktestBroker: Must add at least one file.') 44 | raise Exception 45 | else: 46 | for instr_id in files: 47 | cls.files[instr_id] = {'filename':files[instr_id]} 48 | cls.prices[instr_id] = {} 49 | 50 | cls.start_ms = start.timestamp() 51 | cls.end_ms = end.timestamp() 52 | 53 | """ Trades open on this simulated broker. 54 | trade_id: numeric id 55 | open_time: datetime timestamp 56 | instrument_id: int id from database 57 | execution_price: float execution price 58 | units: int positive=long, negative=short 59 | tp: float take profit 60 | sl: float stop loss price 61 | """ 62 | cls.trades_open = [] 63 | 64 | """ Trade history for this simulated broker. 65 | trade_id: numeric 66 | open_time: datetime 67 | close_time: datetime 68 | instrument_id: string 69 | units: number pos=long neg=short 70 | execution_price: float 71 | close_price: float 72 | """ 73 | cls.trades_closed = [] 74 | 75 | with mp.Pool(mp.cpu_count()) as pool: 76 | for instr_id in cls.files: 77 | # initialize files 78 | f = cls.files[instr_id] # just easier to read 79 | # Read in the files concurrently 80 | f['data'] = pool.apply_async(func=csv_parse.pi, args=[ f['filename'], start, end]) 81 | for instr_id in cls.files: 82 | f = cls.files[instr_id] # just easier to read 83 | # Wait for files to finish reading, then get a generator 84 | f['data'] = f['data'].get().iterrows() 85 | # initialize price buffer 86 | price = cls.prices[instr_id] # with 87 | price['active'] = next(f['data'])[1] # first row 88 | price['buffer'] = next(f['data'])[1] # second row 89 | 90 | @classmethod 91 | def advance(cls): 92 | """ Move one step forward. 93 | Return False when end of rows reached, True otherwise. 94 | """ 95 | # Find oldest buffered instrument and make it active. 96 | oldest = None # instr_id with oldest timestamp 97 | for instr_id in cls.prices: 98 | if not oldest or cls.prices[instr_id]['buffer']['t'] < cls.prices[oldest]['buffer']['t']: 99 | oldest = instr_id 100 | cls.prices[oldest]['active'] = cls.prices[oldest]['buffer'] 101 | """# show status 102 | msg = '' 103 | for instr_id in cls.prices: 104 | msg = msg + '{} {} '.format(instr_id, cls.prices[oldest]['active']['t'].to_pydatetime()) 105 | print(msg, end='\r')""" 106 | try: 107 | # get next row 108 | cls.prices[oldest]['buffer'] = next(cls.files[oldest]['data'])[1] 109 | cls.time = cls.prices[oldest]['buffer']['t'] 110 | except StopIteration: 111 | # EOF - print summary of results 112 | #profits = [ t['close_price'] - t['execution_price'] for t in cls.trades_closed] 113 | profits = [] 114 | for t in cls.trades_closed: 115 | if t['units'] > 0: 116 | profits.append( t['close_price'] - t['execution_price'] ) 117 | else: 118 | profits.append( t['execution_price'] - t['close_price'] ) 119 | profit = sum(profits) 120 | import numpy as np 121 | from scipy import stats 122 | wins = [ p for p in profits if p > 0 ] 123 | wins_avg = round(sum(wins)/float(len(wins)), 5) if len(wins) > 0 else 0 124 | wins_median = round(np.median(wins), 5) if len(wins) > 0 else 0 125 | wins_mode = stats.mode(wins) if len(wins) > 0 else 0 126 | wins_mode_val = round(wins_mode[0][0], 5) 127 | wins_mode_count = wins_mode[1][0] 128 | losses = [ p for p in profits if p < 0 ] 129 | losses_avg = round(sum(losses)/float(len(losses)), 5) if len(losses) > 0 else 0 130 | losses_median = round(np.median(losses), 5) if len(losses) > 0 else 0 131 | losses_mode = stats.mode(losses) if len(losses) > 0 else 0 132 | losses_mode_val = round(losses_mode[0][0], 5) 133 | losses_mode_count = losses_mode[1][0] 134 | washes = [ p for p in profits if p == 0 ] 135 | print('\n') 136 | print('{} open trades'.format(len(cls.trades_open))) 137 | print('{} closed trades'.format(len(cls.trades_closed))) 138 | #print('Profits: {}'.format(profits) ) 139 | print('Wins: {}'.format( len(wins) )) 140 | print(' avg: {}'.format( wins_avg )) 141 | print(' median: {}'.format( wins_median )) 142 | print(' mode: {} x{}'.format( wins_mode_val, wins_mode_count )) 143 | print('Losses: {}'.format( len(losses) )) 144 | print(' avg: {}'.format( losses_avg )) 145 | print(' median: {}'.format( losses_median )) 146 | print(' mode: {} x{}'.format( losses_mode_val, losses_mode_count )) 147 | print('{} Washes'.format(len(washes))) 148 | print('End Profit: {}'.format(profit) ) 149 | #import pdb; pdb.set_trace() 150 | return False 151 | 152 | # Update trade statuses 153 | cls.trades_open = [t for t in cls.trades_open if not cls.update_closed_trade(t)] 154 | 155 | return True 156 | 157 | 158 | @classmethod 159 | def update_closed_trade(cls, trade): 160 | """ If the trade closed, mark it as such. 161 | Returns true if the trade closed, false if still open. 162 | """ 163 | iid = trade['instrument_id'] 164 | 165 | if ( # long SL 166 | trade['sl'] and trade['units'] > 0 and trade['sl'] >= cls.prices[iid]['active']['lb'] 167 | ) or ( # short SL 168 | trade['sl'] and trade['units'] < 0 and trade['sl'] <= cls.prices[iid]['active']['ha'] 169 | ): 170 | #print('SL HIT: execution({}) close({})'.format(trade['execution_price'], trade['sl'])) 171 | cls.trades_closed.append( { 172 | 'trade_id': trade['trade_id'], 173 | 'open_time': trade['open_time'], 174 | 'close_time': cls.prices[iid]['active']['t'], 175 | 'instrument_id': iid, 176 | 'units': trade['units'], 177 | 'execution_price': trade['execution_price'], 178 | 'close_price': trade['sl'] 179 | } ) 180 | return True 181 | elif ( # long TP 182 | trade['tp'] and trade['units'] > 0 and trade['tp'] <= cls.prices[iid]['active']['hb'] 183 | ) or ( # short TP 184 | trade['tp'] and trade['units'] < 0 and trade['tp'] >= cls.prices[iid]['active']['la'] 185 | ): 186 | #print('TP HIT: execution({}) close({})'.format(trade['execution_price'], trade['tp'])) 187 | # trade closed; add to list 188 | cls.trades_closed.append( { 189 | 'trade_id': trade['trade_id'], 190 | 'open_time': trade['open_time'], 191 | 'close_time': cls.prices[iid]['active']['t'], 192 | 'instrument_id': iid, 193 | 'units': trade['units'], 194 | 'execution_price': trade['execution_price'], 195 | 'close_price': trade['tp'] 196 | } ) 197 | return True 198 | 199 | return False 200 | 201 | 202 | ################################################################# 203 | ################################################################# 204 | # Methods to simulate a broker wrapper 205 | ################################################################# 206 | ################################################################# 207 | 208 | @classmethod 209 | def get_time(cls): 210 | """ Normally you would just call utcnow() or something """ 211 | return cls.time 212 | 213 | @classmethod 214 | def get_bid(cls, instrument_id): 215 | return cls.prices[instrument_id]['active']['lb'] 216 | 217 | 218 | @classmethod 219 | def get_ask(cls, instrument_id): 220 | return cls.prices[instrument_id]['active']['ha'] 221 | 222 | 223 | @classmethod 224 | def get_spread(cls, instrument_id): 225 | #print('spread: {}'.format(cls.get_ask(instrument_id) - cls.get_bid(instrument_id))) 226 | return cls.get_ask(instrument_id) - cls.get_bid(instrument_id) 227 | 228 | 229 | @classmethod 230 | def get_price(cls, instrument_id): 231 | return (cls.get_ask(instrument_id) + cls.get_bid(instrument_id)) / 2 232 | 233 | 234 | @classmethod 235 | def get_volume(cls, instrument_id): 236 | return int(cls.prices[instrument_id]['active']['v']) 237 | 238 | 239 | @classmethod 240 | def place_trade(cls, order, units): 241 | """ Return trade id on success, None otherwise 242 | order 243 | units int 244 | """ 245 | print('placing trade: {} units of {} on {}'.format(units, order.instrument.get_name(), cls.get_time())) 246 | iid = order.instrument.get_id() 247 | sl = order.stop_loss 248 | tp = order.take_profit 249 | if units == 0 or (sl==None and tp==None): raise Exception 250 | if units > 0: # long -> execute at ask price 251 | execution_price = cls.prices[iid]['active']['ha'] 252 | else: # short -> execute at bid price 253 | execution_price = cls.prices[iid]['active']['lb'] 254 | cls.trades_open.append( { 255 | 'trade_id': cls.next_trade_id, 256 | 'open_time': cls.prices[iid]['active']['t'], 257 | 'instrument_id': iid, 258 | 'execution_price': execution_price, # ignoring order.price 259 | 'units': units, 260 | 'tp': tp, 261 | 'sl': sl 262 | } ) 263 | cls.next_trade_id += 1 264 | return cls.next_trade_id - 1 265 | 266 | 267 | @classmethod 268 | def is_closed(cls, trade_id): 269 | """ """ 270 | for t in cls.trades_closed: 271 | if t['trade_id'] == trade_id: return True 272 | return False 273 | 274 | 275 | -------------------------------------------------------------------------------- /src/backtesting/csv_parse.py: -------------------------------------------------------------------------------- 1 | """ 2 | Take a CSV with candlestick data and convert it to a standard format. 3 | Standard format of columns: 4 | t pandas datetime? 5 | ob Open Bid 6 | oa Open Ask 7 | hb High Bid 8 | ha High Ask 9 | lb Low Bid 10 | la Low Ask 11 | cb Close Bid 12 | ca Close Ask 13 | v Volume 14 | 15 | https://www.epochconverter.com/ microseconds to current date 16 | """ 17 | from code_timer import CodeTimer 18 | from datetime import datetime 19 | import pandas as pd 20 | import os 21 | 22 | def pi(path, start, end, chunk_size=20000): 23 | """ For CSV files from Pi Trading. 24 | Only one OHLC, so duplicate prices to get bid/ask. 25 | Spread will always be 0. 26 | Input: 27 | path: string 28 | start, end: datetime 29 | Returns: DF with standard columns (see above) 30 | """ 31 | timer_search = CodeTimer.start() 32 | print('Loading ({}) {} to {} (cs={})'.format(path, start, end, chunk_size)) 33 | # Pull one row every x rows and see if it's after start or end. 34 | # Read the chunks within those bounds, then trim head and tail exactly. 35 | cols = ["Date","Time","Open","High","Low","Close","Volume"] 36 | reader = pd.read_csv(path, dtype='str', iterator=True) 37 | i = 0 38 | start_i = None 39 | end_i = None 40 | df_row = reader.get_chunk(chunk_size) 41 | dt = datetime.strptime( df_row.iat[0,0] + df_row.iat[0,1], '%m/%d/%Y%H%M' ) 42 | while dt < end: 43 | try: 44 | if not start_i and dt > start: # passed start 45 | start_i = i-1 46 | i += 1 47 | df_row = reader.get_chunk(chunk_size) 48 | dt = datetime.strptime( df_row.iat[0,0] + df_row.iat[0,1], '%m/%d/%Y%H%M' ) 49 | except StopIteration: 50 | break 51 | end_i = i 52 | duration_search = CodeTimer.stop(timer_search) 53 | if not start_i: # 54 | start_i = i-2 55 | elif start_i <= 0: # should never happen 56 | raise Exception 57 | skip_rows = start_i * chunk_size 58 | if start_i == 0: skip_rows += 1 # pidata CSV has header row 59 | timer_read = CodeTimer.start() 60 | ret = pd.read_csv(path, header=0, names=cols, dtype='str', skiprows=skip_rows, 61 | nrows=((end_i - start_i) * chunk_size) ) 62 | duration_read = CodeTimer.stop(timer_read) 63 | 64 | timer_format = CodeTimer.start() 65 | # trim & format 66 | ret['t'] = pd.to_datetime(ret.Date.str.cat(ret.Time), format='%m/%d/%Y%H%M') 67 | ret = ret.loc[ret['t'] > start] 68 | ret = ret.loc[ret['t'] < end] 69 | if len(ret) < 1: 70 | print(' ERROR: DF is empty after trimming for path ' + path + '\n' + 71 | ' from ' + str(start) + '\n' + 72 | ' to ' + str(end) + '\n' + 73 | '\n Is the date period within the CSV?' ) 74 | raise Exception 75 | ret['ob'] = ret.Open.astype('float') 76 | ret['oa'] = ret.Open.astype('float') 77 | ret['hb'] = ret.High.astype('float') 78 | ret['ha'] = ret.High.astype('float') 79 | ret['lb'] = ret.Low.astype('float') 80 | ret['la'] = ret.Low.astype('float') 81 | ret['cb'] = ret.Close.astype('float') 82 | ret['ca'] = ret.Close.astype('float') 83 | ret['v'] = ret.Volume.astype('int') 84 | ret = ret.drop(cols, axis=1) 85 | duration_format = CodeTimer.stop(timer_format) 86 | 87 | #print('{} {} {} {}' # testing 88 | #` .format(duration_search, duration_read, duration_format, len(ret))) 89 | return ret 90 | 91 | 92 | def five_second(path, start, end, chunk_size=20000): 93 | """ 94 | Input: 95 | path: string 96 | start, end: datetime 97 | Returns: DF with standard columns (see above) 98 | """ 99 | timer_search = CodeTimer.start() 100 | print('Loading ({}) {} to {} (cs={})'.format(path, start, end, chunk_size)) 101 | cols = ['time_micro', 'open_bid', 'open_ask', 'high_bid', 'high_ask', 'low_bid', 'low_ask', 'close_bid', 'close_ask', 'vol'] 102 | reader = pd.read_csv(path, dtype='str', iterator=True) 103 | i = 0 104 | start_i = None 105 | end_i = None 106 | df_row = reader.get_chunk(chunk_size) 107 | dt = pd.Timestamp.utcfromtimestamp( int(int(df_row.iat[0,0])/1000000) ) 108 | while dt < end: 109 | try: 110 | if not start_i and dt > start: # passed start 111 | start_i = i-1 112 | i += 1 113 | df_row = reader.get_chunk(chunk_size) 114 | #dt = datetime.strptime( df_row.iat[0,0] + df_row.iat[0,1], '%m/%d/%Y%H%M' ) 115 | dt = pd.Timestamp.utcfromtimestamp( int(int(df_row.iat[0,0]) / 1000000) ) 116 | except StopIteration: 117 | break 118 | duration_search = CodeTimer.stop(timer_search) 119 | end_i = i 120 | if not start_i: # 121 | start_i = i-2 122 | elif start_i <= 0: # should never happen 123 | raise Exception 124 | skip_rows = start_i * chunk_size 125 | timer_read = CodeTimer.start() 126 | ret = pd.read_csv(path, header=0, names=cols, dtype='str', skiprows=skip_rows, 127 | nrows=((end_i - start_i) * chunk_size) ) 128 | duration_read = CodeTimer.stop(timer_read) 129 | 130 | # trim and format 131 | timer_format = CodeTimer.start() 132 | ret['t'] = pd.to_numeric(ret.time_micro).floordiv(1000000).map(pd.Timestamp.utcfromtimestamp, na_action='ignore') 133 | ret = ret.loc[ret['t'] > start] 134 | ret = ret.loc[ret['t'] < end] 135 | if len(ret) < 1: 136 | print(' ERROR: DF is empty after trimming for path ' + path + '\n' + 137 | ' from ' + str(start) + '\n' + 138 | ' to ' + str(end) + '\n' + 139 | '\n Is the date period within the CSV?' ) 140 | raise Exception 141 | ret['ob'] = ret.open_bid.astype('float') 142 | ret['oa'] = ret.open_ask.astype('float') 143 | ret['hb'] = ret.high_bid.astype('float') 144 | ret['ha'] = ret.high_ask.astype('float') 145 | ret['lb'] = ret.low_bid.astype('float') 146 | ret['la'] = ret.low_ask.astype('float') 147 | ret['cb'] = ret.close_bid.astype('float') 148 | ret['ca'] = ret.close_ask.astype('float') 149 | ret['v'] = ret.vol.astype('int') 150 | ret = ret.drop(cols, axis=1) 151 | duration_format = CodeTimer.stop(timer_format) 152 | 153 | #print('{} {} {} {}' # testing 154 | # .format(duration_search, duration_read, duration_format, len(ret))) 155 | return ret 156 | -------------------------------------------------------------------------------- /src/backtesting/strategies/demo.py: -------------------------------------------------------------------------------- 1 | from backtesting.backtest_broker import BacktestBroker as broker 2 | from datetime import datetime, timedelta 3 | from instrument import Instrument 4 | from log import Log 5 | from opportunity import Opportunity 6 | from order import Order 7 | from strategy import Strategy 8 | from trade import Trade, Trades, TradeClosedReason 9 | 10 | class Demo(Strategy): 11 | 12 | num_slots = 10 13 | price_history = [None]*num_slots # nth element = price n minutes ago 14 | last_history_update = None # time of last history update 15 | instrument_ids = [4,5] # these should be passed into the Strategy constructor 16 | 17 | def get_name(self): 18 | """ Return the name of the strategy.""" 19 | return "Backtest Demo" 20 | 21 | 22 | def __str__(self): 23 | """ """ 24 | return self.get_name() 25 | 26 | 27 | def _babysit(self): 28 | """Monitor open positions. Check if any have closed. 29 | """ 30 | for otid in self.open_trade_ids: 31 | if broker.is_closed(otid): 32 | self.open_trade_ids.remove(otid) # mark status 33 | 34 | 35 | def _scan(self): 36 | """Look for opportunities to open a position. 37 | Returns: 38 | [] if there is an opportunity. 39 | Empty list if no opportunities. 40 | None on failure. 41 | """ 42 | for iid in self.instrument_ids: 43 | now = broker.get_time() 44 | spread = broker.get_spread(iid) 45 | price = broker.get_price(iid) 46 | goal = spread # goal profit per trade 47 | 48 | # always update the price history 49 | if self.last_history_update == None or now - self.last_history_update >= timedelta(minutes=1): 50 | self.last_history_update = now 51 | # shift everything by one 52 | for i in range(len(self.price_history)-1, 0, -1): 53 | self.price_history[i] = self.price_history[i-1] 54 | self.price_history[0] = price 55 | 56 | # check for opp 57 | go = False 58 | # only if price history completely filled 59 | if not None in self.price_history: 60 | smooth = True 61 | for i in range(0, len(self.price_history)-1): 62 | # if more recent price has decreased or stayed same 63 | if self.price_history[i] <= self.price_history[i+1]: 64 | smooth = False 65 | rise = self.price_history[0] - self.price_history[self.num_slots-1] 66 | expectation = 5 * spread 67 | if smooth and rise > expectation: go = True 68 | 69 | # suggest trade if no open trades 70 | if go and len(self.open_trade_ids) < 1: 71 | order = Order( 72 | instrument=Instrument(iid), 73 | order_type='market', 74 | take_profit=self.price_history[0] + goal + spread/float(2), # long 75 | stop_loss=self.price_history[0] - goal + spread/float(2), # long 76 | units=1, # long 77 | #take_profit=self.price_history[0] - goal - spread/float(2), # short 78 | #stop_loss=self.price_history[0] + goal - spread/float(2), # short 79 | #units=-1, # short 80 | reason='demo' 81 | ) 82 | opp = Opportunity(order, conf=1, strat=self, reason='demo') 83 | return [opp] 84 | else: 85 | return [] 86 | 87 | 88 | -------------------------------------------------------------------------------- /src/broker.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-7 -*- 2 | 3 | """ 4 | File broker_api.py 5 | Python ver. 3.4 6 | Description: 7 | Python module that provides a generic layer between 8 | the daemon and the broker-specific code. 9 | """ 10 | 11 | #**************************** 12 | import configparser 13 | import sys 14 | #**************************** 15 | from config import Config 16 | from instrument import Instrument 17 | from log import Log 18 | from oanda import Oanda 19 | #**************************** 20 | 21 | class Broker(): 22 | 23 | broker = None 24 | if Config.broker_name == 'oanda': 25 | broker = Oanda # point to Oanda class 26 | else: 27 | DB.bug('"broker.py": unknown broker "{}"'.format(Config.broker_name)) 28 | raise Exception 29 | 30 | 31 | # Get authorization key. 32 | @classmethod 33 | def get_auth_key(cls): 34 | """ 35 | Oanda uses an authentication key for HTTP requests. 36 | """ 37 | return cls.broker.get_auth_key() 38 | 39 | 40 | @classmethod 41 | def get_accounts(cls): 42 | return cls.broker.get_accounts() 43 | 44 | 45 | @classmethod 46 | def get_margin_available(cls, account_id): 47 | return cls.broker.get_margin_available(account_id) 48 | 49 | 50 | @classmethod 51 | def get_margin_rate(cls, instrument): 52 | return cls.broker.get_margin_rate(instrument) 53 | 54 | 55 | # Get list of open positions 56 | # Returns: dict or None 57 | @classmethod 58 | def get_positions(cls, account_id): 59 | return cls.broker.get_positions(account_id) 60 | 61 | 62 | # Get number of positions for a given account ID 63 | # Returns: Integer 64 | @classmethod 65 | def get_num_of_positions(cls, account_id): 66 | return cls.broker.get_num_of_positions(account_id) 67 | 68 | 69 | @classmethod 70 | def get_balance(cls, account_id): 71 | """Return type: Decimal number 72 | Get account balance. 73 | If account_id is not proviced, defaults to primary account. 74 | """ 75 | return cls.broker.get_balance(account_id) 76 | 77 | 78 | @classmethod 79 | def get_prices(cls, instruments, since=None): 80 | """Returns: dict or None 81 | Fetch live prices for specified symbol(s)/instrument(s). 82 | TODO: make this more robust. Maybe pass in a list, then have each broker-specific library 83 | do validation. 84 | """ 85 | return cls.broker.get_prices(instruments, since) 86 | 87 | 88 | @classmethod 89 | def get_ask(cls, instrument, since=None): 90 | """Returns: Decimal or None 91 | Get one ask price 92 | """ 93 | return cls.broker.get_ask(instrument, since) 94 | 95 | 96 | @classmethod 97 | def get_bid(cls, instrument, since=None): 98 | """Returns: Decimal or None 99 | Get one bid price 100 | """ 101 | return cls.broker.get_bid(instrument, since) 102 | 103 | 104 | @classmethod 105 | def get_spreads(cls, instruments, since=None): 106 | return cls.broker.get_spreads(instruments, since) 107 | 108 | 109 | @classmethod 110 | def place_order(cls, order): 111 | """ 112 | Return type: dict or none 113 | Return value: Whatever broker returns (Oanda=JSON) 114 | """ 115 | if order.units == None: 116 | raise Exception 117 | result = cls.broker.place_order(order) 118 | # TODO: If a trade is opened, write trade info to db 119 | # (table: open_trades_live) 120 | return result 121 | 122 | 123 | @classmethod 124 | def is_market_open(cls): 125 | """ 126 | Is the market open? 127 | 128 | Returns: Boolean 129 | """ 130 | return cls.broker.is_market_open() 131 | 132 | 133 | @classmethod 134 | def get_time_until_close(cls): 135 | return cls.broker.get_time_until_close() 136 | 137 | 138 | @classmethod 139 | def get_time_since_close(cls): 140 | return cls.broker.get_time_since_close() 141 | 142 | 143 | @classmethod 144 | def get_transaction_history(cls, maxId=None, minId=None, count=None, instrument=None, ids=None): 145 | """Returns: 146 | Get transaction history 147 | """ 148 | return cls.broker.get_transaction_history(maxId=maxId, minId=minId, 149 | count=count, instrument=instrument, ids=ids) 150 | 151 | 152 | """ 153 | Return type: dict or none. See broker's API documentation for details. 154 | Get historical prices for an instrument. 155 | """ 156 | @classmethod 157 | def get_instrument_history( 158 | cls, 159 | instrument, # 160 | granularity=None, # string 161 | count=None, # optional- int - leave out if both start & end specified 162 | from_time=None, # optional- datetime 163 | to=None, # optional- datetime 164 | price='MBA', # optional - string 165 | include_first=None, # optional - bool 166 | daily_alignment=None, # optional - numeric 167 | alignment_timezone=None,# optional - 168 | weekly_alignment=None # optional - string 169 | ): 170 | if Config.broker_name == 'oanda': 171 | return cls.broker.get_instrument_history( 172 | in_instrument=instrument, 173 | granularity=granularity, 174 | count=count, 175 | from_time=from_time, 176 | to=to, 177 | price=price, 178 | include_first=include_first, 179 | daily_alignment=daily_alignment, 180 | alignment_timezone=alignment_timezone, 181 | weekly_alignment=weekly_alignment 182 | ) 183 | else: 184 | raise NotImplementedError 185 | 186 | 187 | @classmethod 188 | def is_trade_closed(cls, transaction_id): 189 | """Returns: 190 | See if a trade is closed. 191 | """ 192 | return cls.broker.is_trade_closed(transaction_id) 193 | 194 | 195 | @classmethod 196 | def get_open_trades(cls): 197 | """ Return type: 198 | Returns info about all open trades from the broker. 199 | To get "local" info about the trades, use 200 | trade.fill_in_trade_extra_info(). 201 | """ 202 | return cls.broker.get_open_trades() 203 | 204 | 205 | @classmethod 206 | def get_trade(cls, trade_id): 207 | """ 208 | Get info about a particular trade 209 | Returns: instance of 210 | """ 211 | return cls.broker.get_trade(trade_id) 212 | 213 | 214 | # Get order info 215 | # Returns: dict or None 216 | @classmethod 217 | def get_order_info(cls, order_id): 218 | return cls.broker.get_order_info(order_id) 219 | 220 | 221 | @classmethod 222 | def modify_order(cls, order_id, units=0, price=0, 223 | lower_bound=0, upper_bound=0, stop_loss=0, 224 | take_profit=0, trailing_stop=0): 225 | """ 226 | # Modify an existing order 227 | # Returns: dict or None 228 | """ 229 | return cls.broker.modify_orders(locals()) 230 | 231 | 232 | @classmethod 233 | def modify_trade(cls, 234 | trade_id, # unique identifier of trade to modify 235 | stop_loss_price=0, 236 | take_profit_price=0, 237 | trailing_stop_loss_distance=0 # in price units 238 | ): 239 | """ 240 | # Modify an existing trade 241 | # Returns: dict or None 242 | """ 243 | return cls.broker.modify_trade( 244 | trade_id = trade_id, 245 | stop_loss_price = stop_loss_price, 246 | take_profit_price = take_profit_price, 247 | trailing_stop_loss_distance = trailing_stop_loss_distance 248 | ) 249 | 250 | 251 | -------------------------------------------------------------------------------- /src/candle.py: -------------------------------------------------------------------------------- 1 | 2 | """ 3 | Class that represents a chart candlestick 4 | """ 5 | class Candle(): 6 | 7 | """ 8 | 9 | """ 10 | def __init__(self, 11 | timestamp=None, # datetime 12 | volume=None, # int 13 | complete=None, # bool 14 | chart_format=None, # 'bidask' (default) or 'midpoint' 15 | 16 | open_bid=None, # float 17 | open_ask=None, # float 18 | open_mid=None, # float 19 | high_bid=None, # float 20 | high_ask=None, # float 21 | high_mid=None, # float 22 | low_bid=None, # float 23 | low_ask=None, # float 24 | low_mid=None, # float 25 | close_bid=None, # float 26 | close_ask=None, # float 27 | close_mid=None # float 28 | ): 29 | self.timestamp = timestamp 30 | self.volume = volume 31 | self.complete = complete 32 | 33 | if open_bid != None: 34 | self.open_bid = float(open_bid) 35 | if open_ask != None: 36 | self.open_ask = float(open_ask) 37 | if open_mid != None: 38 | self.open_mid = float(open_mid) 39 | if high_bid != None: 40 | self.high_bid = float(high_bid) 41 | if high_ask != None: 42 | self.high_ask = float(high_ask) 43 | if high_mid != None: 44 | self.high_mid = float(high_mid) 45 | if low_bid != None: 46 | self.low_bid = float(low_bid) 47 | if low_ask != None: 48 | self.low_ask = float(low_ask) 49 | if low_mid != None: 50 | self.low_mid = float(low_mid) 51 | if close_bid != None: 52 | self.close_bid = float(close_bid) 53 | if close_ask != None: 54 | self.close_ask = float(close_ask) 55 | if close_mid != None: 56 | self.close_mid = float(close_mid) 57 | 58 | 59 | def __del__(self): 60 | pass 61 | 62 | 63 | def __str__(self): 64 | return 'candle({0}) bid o:{1} h:{2} l:{3} c:{4}'.format( 65 | self.timestamp, str(self.open_bid), str(self.high_bid), 66 | str(self.low_bid), str(self.close_bid) 67 | ) 68 | 69 | -------------------------------------------------------------------------------- /src/chart.py: -------------------------------------------------------------------------------- 1 | 2 | # library imports 3 | #import array 4 | import datetime 5 | from scipy import stats 6 | 7 | # local imports 8 | from broker import Broker 9 | from candle import Candle 10 | from instrument import Instrument 11 | from log import Log 12 | import util_date 13 | import utils 14 | 15 | """ 16 | A is a group of sequential s. 17 | """ 18 | class Chart: 19 | 20 | """ 21 | Return type: void 22 | Basically just forward the arguments to the broker API, 23 | then gather the returned candlesticks. 24 | """ 25 | def __init__( 26 | self, 27 | in_instrument, # 28 | granularity='S5', # string - See Oanda's documentation 29 | count=None, # int - number of candles 30 | start=None, # datetime - UTC 31 | end=None, # datetime - UTC 32 | price='MBA', # string 33 | include_first=None, # bool 34 | daily_alignment=None, # int 35 | alignment_timezone=None, # string - timezone 36 | weekly_alignment=None 37 | ): 38 | self._candles = [] 39 | # verify instance of by accessing a member. 40 | if in_instrument.get_id() == 0: 41 | pass 42 | 43 | if not count in [0]: # do send None 44 | # get candles from broker 45 | instrument_history = Broker.get_instrument_history( 46 | instrument=in_instrument, 47 | granularity=granularity, 48 | count=count, 49 | from_time=start, 50 | to=end, 51 | price=price, 52 | include_first=include_first, 53 | daily_alignment=daily_alignment, 54 | alignment_timezone=alignment_timezone, 55 | weekly_alignment=weekly_alignment 56 | ) 57 | if instrument_history == None: 58 | Log.write('chart.py __init__(): Failed to get instrument history.') 59 | raise Exception 60 | else: 61 | candles_raw = instrument_history['candles'] 62 | for c_r in candles_raw: 63 | new_candle = Candle( 64 | timestamp=util_date.string_to_date(c_r['time']), 65 | volume=float(c_r['volume']), 66 | complete=bool(c_r['complete']), 67 | open_bid=float(c_r['bid']['o']), 68 | high_bid=float(c_r['bid']['h']), 69 | low_bid=float(c_r['bid']['l']), 70 | close_bid=float(c_r['bid']['c']), 71 | open_ask=float(c_r['ask']['o']), 72 | high_ask=float(c_r['ask']['h']), 73 | low_ask=float(c_r['ask']['l']), 74 | close_ask=float(c_r['ask']['c']) 75 | ) 76 | self._candles.append(new_candle) 77 | 78 | self._instrument = in_instrument 79 | self._granularity = granularity 80 | self._start_index = 0 # start 81 | self._price = price 82 | self.include_first = include_first 83 | self.daily_alignment = daily_alignment 84 | self._alignment_timezone = alignment_timezone 85 | self.weekly_alignment = weekly_alignment 86 | 87 | 88 | def __str__(self): 89 | return "Chart" 90 | 91 | 92 | def _get_end_index(self): 93 | if self._start_index > 0: 94 | return self._start_index - 1 95 | elif self._start_index == 0: 96 | if self.get_size() > 0: return self.get_size() - 1 97 | else: return 0 98 | else: 99 | raise Exception 100 | 101 | 102 | def get_size(self): 103 | """Return type: int 104 | Returns: Number of candles in chart. 105 | https://wiki.python.org/moin/TimeComplexity 106 | len() in Python is O(1), so no need to store size. 107 | """ 108 | return len(self._candles) 109 | 110 | 111 | def get_instrument(self): 112 | """Return type: instance 113 | """ 114 | return self._instrument 115 | 116 | 117 | """ 118 | Return type: string (See Oanda's documentation) 119 | """ 120 | def get_granularity(self): 121 | return self._granularity 122 | 123 | 124 | """ 125 | Return type: datetime 126 | """ 127 | def get_start_timestamp(self): 128 | return self._candles[self._start_index].timestamp 129 | 130 | 131 | def get_end_timestamp(self): 132 | """Return type: datetime 133 | Return the time and date of the last candlestick. 134 | """ 135 | return self._candles[self._get_end_index()].timestamp 136 | 137 | 138 | def get_time_span(self): 139 | """Return type: datetime.timedelta 140 | Get time difference between first and last candles. 141 | """ 142 | return self.get_end_timestamp() - self.get_start_timestamp() 143 | 144 | 145 | def get_lag(self): 146 | """Return type: datetime.timedelta 147 | Returns time difference between the last candlestick and now. 148 | """ 149 | return datetime.datetime.utcnow() - self.get_end_timestamp() 150 | 151 | 152 | def _increment_start_index(self): 153 | if self._start_index < self.get_size() - 1: 154 | self._start_index += 1 155 | elif self._start_index == self.get_size() - 1: 156 | self._start_index = 0 157 | else: 158 | raise Exception 159 | 160 | 161 | def set(self, start_time): 162 | """Return type: None on failure, 0 on success 163 | Basically re-initialize the chart. 164 | Use this for storing candles from a particular point in time. 165 | Use update() to get the latest candles. 166 | """ 167 | raise NotImplementedError 168 | 169 | 170 | def update(self): 171 | """Returns: void 172 | Replace the candles with the most recent ones available. 173 | Algorithm for minimizing number of updated candles: 174 | Get the time difference from chart end to now. 175 | If the time difference is greater than the width of the chart, 176 | request candles. 177 | else, 178 | request candles from end of chart to now. 179 | """ 180 | new_history = None 181 | if self.get_lag() > self.get_time_span(): 182 | # replace all candles 183 | new_history = Broker.get_instrument_history( 184 | instrument=self._instrument, 185 | granularity=self._granularity, 186 | count=self.get_size(), 187 | to=datetime.datetime.utcnow() 188 | ) 189 | else: 190 | # request new candles starting from end of chart 191 | # TODO verify candleFormat is same as existing chart 192 | new_history_ = broker.get_instrument_history( 193 | instrument=self._instrument, 194 | granularity=self._granularity, 195 | from_time=self.get_end_timestamp 196 | ) 197 | if new_history == None: 198 | Log.write('chart.py update(): Failed to get new candles.') 199 | raise Exception 200 | else: 201 | # Got new candles. Stow them. 202 | new_candles = new_history['candles'] 203 | # Iterate forwards from last candle. The last candle is probably 204 | # non-complete, so overwrite it. This thereby fills in the missing 205 | # gap between (end of chart) and (now). If the gap is smaller 206 | # than the size of the chart, only the beginning of the chart is 207 | # overwritten. If the gap is bigger than the chart, all candles 208 | # get overwritten. 209 | for i in range(0, len(self._candles)): 210 | # TODO assuming bid/ask candles 211 | new_candle = new_candles[i] 212 | self._candles[self._get_end_index()].timestamp = util_date.string_to_date(new_candle['time']) 213 | self._candles[self._get_end_index()].volume = float(new_candle['volume']) 214 | self._candles[self._get_end_index()].complete = bool(new_candle['complete']) 215 | self._candles[self._get_end_index()].open_bid = float(new_candle['bid']['o']) 216 | self._candles[self._get_end_index()].open_ask = float(new_candle['ask']['o']) 217 | self._candles[self._get_end_index()].high_bid = float(new_candle['bid']['h']) 218 | self._candles[self._get_end_index()].high_ask = float(new_candle['ask']['h']) 219 | self._candles[self._get_end_index()].low_bid = float(new_candle['bid']['l']) 220 | self._candles[self._get_end_index()].low_ask = float(new_candle['ask']['l']) 221 | self._candles[self._get_end_index()].close_bid = float(new_candle['bid']['c']) 222 | self._candles[self._get_end_index()].close_ask = float(new_candle['ask']['c']) 223 | if i < len(self._candles) - 1: 224 | self._increment_start_index() # increments end index too 225 | 226 | 227 | def __getitem__( 228 | self, 229 | key # int 230 | ): 231 | """Return type: 232 | Makes the class accessable using the [] operator. 233 | Note that the internal candle array uses a shifted index. 234 | """ 235 | if key < 0 or key >= len(self._candles): 236 | raise Exception 237 | return self._candles[(key + self._start_index) % len(self._candles)] 238 | 239 | 240 | def __setitem__(self, key, value): 241 | """ 242 | Makes the class settable via the [] operator. 243 | There is probably no reason to implement this. 244 | """ 245 | raise NotImplementedError 246 | 247 | 248 | def standard_deviation(self, 249 | start=None, # candle index - default to first 250 | end=None # candle index - default to last 251 | ): 252 | """ 253 | Return type: float 254 | end int End index of slice. Defaults to chart end. 255 | """ 256 | raise NotImplementedError 257 | 258 | 259 | def pearson( 260 | self, 261 | start, # candle index; 0 for first candle 262 | end, # candle index; size-1 for last candle 263 | method='high_low_avg' # high_low_avg | open | high | low | close 264 | ): 265 | """Return type: float 266 | Calculates Pearson correlation coefficient of the specified subgroup 267 | of candles in the chart. 268 | """ 269 | Log.write('chart.py pearson(): method = {}'.format(method) ) 270 | Log.write('chart.py pearson(): candle coutn = {}'.format(self.get_size()) ) 271 | 272 | # input validation 273 | if self.get_size() < 2: 274 | Log.write('chart.py pearson(): Called on chart with {} candles.' 275 | .format(self.get_size())) 276 | raise Exception 277 | 278 | if method == 'high_low_avg': 279 | # Create 2 datasets to pass into Pearson. 280 | x_set = [] 281 | y_set = [] 282 | for i in range(start, end+1): 283 | # Convert timestamp to numeric value. 284 | # Clarify timezone is UTC before calling timestamp(). 285 | x_set.append( self[i].timestamp.replace(tzinfo=datetime.timezone.utc).timestamp() ) 286 | avg = None 287 | if hasattr(self[i], 'high_mid'): # and self[i].low_mid 288 | avg = (self[i].high_mid + self[i].low_mid) / 2 289 | else: 290 | avg = (self[i].high_ask + self[i].low_bid) / 2 291 | y_set.append( avg ) 292 | Log.write( 293 | 'chart.py pearson(): Calling stats.pearsonr with x_set = \n{}\n and y_set = \n{}\n' 294 | .format(x_set, y_set) ) 295 | result = stats.pearsonr( x_set, y_set ) 296 | return result[0] 297 | else: 298 | raise NotImplementedError 299 | 300 | 301 | def slope(self, something): 302 | """Return type: float 303 | """ 304 | # "simple linear regression" or just two point slope 305 | raise NotImplementedError 306 | 307 | 308 | def noise(self, 309 | start=None, # candle index - default to first 310 | end=None # candle index - default to last 311 | ): 312 | """ 313 | not really sure 314 | """ 315 | raise NotImplementedError 316 | 317 | 318 | """ 319 | # Anscombe's quartet 320 | """ 321 | 322 | -------------------------------------------------------------------------------- /src/code_timer.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import timeit 3 | 4 | class CodeTimer(): 5 | 6 | @classmethod 7 | def start(cls): 8 | """ Returns the wall clock time, in seconds """ 9 | return timeit.default_timer() 10 | 11 | 12 | @classmethod 13 | def stop(cls, start): 14 | """ Returns the duration of time passed, in seconds. 15 | 16 | Parameters: 17 | start: float - return value of start() 18 | """ 19 | duration = timeit.default_timer() - start 20 | if duration < 0: 21 | duration += 60*60*24 22 | return duration 23 | -------------------------------------------------------------------------------- /src/config.py: -------------------------------------------------------------------------------- 1 | 2 | """ 3 | Module Description: One-stop shop for configuration settings. 4 | """ 5 | 6 | import configparser 7 | import sys 8 | 9 | # Need to modify sys.path because Python doesn't have relative imports. 10 | try: 11 | sys.path.index('/home/user/raid/software_projects/algo/src') 12 | except ValueError: 13 | sys.path.append('/home/user/raid/software_projects/algo/src') 14 | 15 | class Config(): 16 | """ 17 | Read the config file(s) and expose the key-values pairs. 18 | There are two config files, a "public" one and a "private" one. 19 | The reason is mainly so that one can be public on GitHub. 20 | If you want you can combine them into one file. 21 | """ 22 | cfg = configparser.ConfigParser() 23 | 24 | # read the public config file 25 | with open('config_nonsecure.cfg', 'r') as f: 26 | cfg.read_file(f) 27 | f.close() 28 | _private_config_path = cfg['config_secure']['path'] 29 | broker_name = cfg['trading']['broker'] 30 | live_trading = False 31 | if cfg['trading']['live_trading'] == 'True': 32 | live_trading = True 33 | 34 | # read the private config file 35 | cfg.read(_private_config_path) 36 | oanda_url = None 37 | oanda_token = None 38 | account_id = None 39 | if live_trading: 40 | # TODO: put the URLs in the config file 41 | oanda_url = 'https://api-fxtrade.oanda.com' 42 | oanda_token = cfg['oanda']['token'] 43 | account_id = cfg['oanda']['account_id_live'] 44 | else: 45 | oanda_url = 'https://api-fxpractice.oanda.com' 46 | oanda_token = cfg['oanda']['token_practice'] 47 | account_id = cfg['oanda']['account_id_practice'] 48 | log_path = cfg['log']['path'] 49 | log_file = cfg['log']['file'] 50 | log_path = log_path + log_file 51 | db_user = cfg['mysql']['username'] 52 | db_pw = cfg['mysql']['password'] 53 | db_host = cfg['mysql']['host'] 54 | db_name = cfg['mysql']['database'] 55 | 56 | 57 | @classmethod 58 | def __str__(): 59 | return 'Config' 60 | -------------------------------------------------------------------------------- /src/config_nonsecure.cfg: -------------------------------------------------------------------------------- 1 | [config_secure] 2 | path: /home/user/raid/documents/algo/usb/algo.cfg 3 | 4 | [trading] 5 | live_trading: False 6 | broker: oanda 7 | 8 | 9 | -------------------------------------------------------------------------------- /src/currency_pair_conversions.py: -------------------------------------------------------------------------------- 1 | 2 | def pips_to_price(instrument, pips): 3 | """ 4 | Given an instrument and price, convert price to pips 5 | Examples: 6 | pips_to_price('USD_JPY', 100) == 1 7 | Returns: decimal or None 8 | """ 9 | if instrument in currency_pair_conversions: 10 | return (currency_pair_conversions[instrument] * pips); 11 | else: 12 | return None 13 | 14 | def price_to_pips(instrument, price): 15 | """ 16 | """ 17 | if not instrument in currency_pair_conversions: 18 | return None 19 | else: 20 | # TODO Why the check for zero? 21 | if price != 0: 22 | return (price / currency_pair_conversions[instrument]) 23 | return 0 24 | 25 | currency_pair_conversions = { 26 | 'AU200_AUD':0.1 27 | ,'AUD_CAD':0.0001 28 | ,'AUD_CHF':0.0001 29 | ,'AUD_HKD':0.0001 30 | ,'AUD_JPY':0.01 31 | ,'AUD_NZD':0.0001 32 | ,'AUD_SGD':0.0001 33 | ,'AUD_USD':0.0001 34 | ,'BCO_USD':0.01 35 | ,'CAD_CHF':0.0001 36 | ,'CAD_HKD':0.0001 37 | ,'CAD_JPY':0.01 38 | ,'CAD_SGD':0.0001 39 | ,'CH20_CHF':0.1 40 | ,'CHF_HKD':0.0001 41 | ,'CHF_JPY':0.01 42 | ,'CHF_ZAR':0.0001 43 | ,'CORN_USD':0.01 44 | ,'DE10YB_EUR':0.01 45 | ,'DE30_EUR':0.1 46 | ,'EU50_EUR':0.1 47 | ,'EUR_AUD':0.0001 48 | ,'EUR_CAD':0.0001 49 | ,'EUR_CHF':0.0001 50 | ,'EUR_CZK':0.0001 51 | ,'EUR_DKK':0.0001 52 | ,'EUR_GBP':0.0001 53 | ,'EUR_HKD':0.0001 54 | ,'EUR_HUF':0.01 55 | ,'EUR_JPY':0.01 56 | ,'EUR_NOK':0.0001 57 | ,'EUR_NZD':0.0001 58 | ,'EUR_PLN':0.0001 59 | ,'EUR_SEK':0.0001 60 | ,'EUR_SGD':0.0001 61 | ,'EUR_TRY':0.0001 62 | ,'EUR_USD':0.0001 63 | ,'EUR_ZAR':0.0001 64 | ,'FR40_EUR':0.1 65 | ,'GBP_AUD':0.0001 66 | ,'GBP_CAD':0.0001 67 | ,'GBP_CHF':0.0001 68 | ,'GBP_HKD':0.0001 69 | ,'GBP_JPY':0.01 70 | ,'GBP_NZD':0.0001 71 | ,'GBP_PLN':0.0001 72 | ,'GBP_SGD':0.0001 73 | ,'GBP_USD':0.0001 74 | ,'GBP_ZAR':0.0001 75 | ,'HK33_HKD':0.1 76 | ,'HKD_JPY':0.0001 77 | ,'JP225_USD':0.1 78 | ,'NAS100_USD':0.1 79 | ,'NATGAS_USD':0.01 80 | ,'NL25_EUR':0.01 81 | ,'NZD_CAD':0.0001 82 | ,'NZD_CHF':0.0001 83 | ,'NZD_HKD':0.0001 84 | ,'NZD_JPY':0.01 85 | ,'NZD_SGD':0.0001 86 | ,'NZD_USD':0.0001 87 | ,'SG30_SGD':0.1 88 | ,'SGD_CHF':0.0001 89 | ,'SGD_HKD':0.0001 90 | ,'SGD_JPY':0.01 91 | ,'SOYBN_USD':0.01 92 | ,'SPX500_USD':0.1 93 | ,'SUGAR_USD':0.0001 94 | ,'TRY_JPY':0.01 95 | ,'UK100_GBP':0.1 96 | ,'UK10YB_GBP':0.01 97 | ,'US2000_USD':0.01 98 | ,'US30_USD':0.1 99 | ,'USB02Y_USD':0.01 100 | ,'USB05Y_USD':0.01 101 | ,'USB10Y_USD':0.01 102 | ,'USB30Y_USD':0.01 103 | ,'USD_CAD':0.0001 104 | ,'USD_CHF':0.0001 105 | ,'USD_CNH':0.0001 106 | ,'USD_CNY':0.0001 107 | ,'USD_CZK':0.0001 108 | ,'USD_DKK':0.0001 109 | ,'USD_HKD':0.0001 110 | ,'USD_HUF':0.01 111 | ,'USD_INR':0.01 112 | ,'USD_JPY':0.01 113 | ,'USD_MXN':0.0001 114 | ,'USD_NOK':0.0001 115 | ,'USD_PLN':0.0001 116 | ,'USD_SAR':0.0001 117 | ,'USD_SEK':0.0001 118 | ,'USD_SGD':0.0001 119 | ,'USD_THB':0.01 120 | ,'USD_TRY':0.0001 121 | ,'USD_TWD':0.0001 122 | ,'USD_ZAR':0.0001 123 | ,'WHEAT_USD':0.01 124 | ,'WTICO_USD':0.01 125 | ,'XAG_AUD':0.0001 126 | ,'XAG_CAD':0.0001 127 | ,'XAG_CHF':0.0001 128 | ,'XAG_EUR':0.0001 129 | ,'XAG_GBP':0.0001 130 | ,'XAG_HKD':0.0001 131 | ,'XAG_JPY':0.1 132 | ,'XAG_NZD':0.0001 133 | ,'XAG_SGD':0.0001 134 | ,'XAG_USD':0.0001 135 | ,'XAU_AUD':0.01 136 | ,'XAU_CAD':0.01 137 | ,'XAU_CHF':0.01 138 | ,'XAU_EUR':0.01 139 | ,'XAU_GBP':0.01 140 | ,'XAU_HKD':0.01 141 | ,'XAU_JPY':10 142 | ,'XAU_NZD':0.01 143 | ,'XAU_SGD':0.01 144 | ,'XAU_USD':0.01 145 | ,'XAU_XAG':0.01 146 | ,'XCU_USD':0.0001 147 | ,'XPD_USD':0.01 148 | ,'XPT_USD':0.01 149 | ,'ZAR_JPY':0.01 150 | } 151 | -------------------------------------------------------------------------------- /src/currency_pair_precision.py: -------------------------------------------------------------------------------- 1 | """ 2 | From a stranger on the internet: 3 | 4 | "If you're using oanda you also need this or your trades will be rejected if you provide too many decimal places" 5 | "FYI Oanda will reject an order if you're too precise" 6 | """ 7 | 8 | currency_pair_precision = { 9 | 'AU200_AUD':0.1 10 | ,'AUD_CAD':0.00001 11 | ,'AUD_CHF':0.00001 12 | ,'AUD_HKD':0.00001 13 | ,'AUD_JPY':0.001 14 | ,'AUD_NZD':0.00001 15 | ,'AUD_SGD':0.00001 16 | ,'AUD_USD':0.00001 17 | ,'BCO_USD':0.001 18 | ,'CAD_CHF':0.00001 19 | ,'CAD_HKD':0.00001 20 | ,'CAD_JPY':0.001 21 | ,'CAD_SGD':0.00001 22 | ,'CH20_CHF':0.1 23 | ,'CHF_HKD':0.00001 24 | ,'CHF_JPY':0.001 25 | ,'CHF_ZAR':0.00001 26 | ,'CORN_USD':0.001 27 | ,'DE10YB_EUR':0.001 28 | ,'DE30_EUR':0.1 29 | ,'EU50_EUR':0.1 30 | ,'EUR_AUD':0.00001 31 | ,'EUR_CAD':0.00001 32 | ,'EUR_CHF':0.00001 33 | ,'EUR_CZK':0.00001 34 | ,'EUR_DKK':0.00001 35 | ,'EUR_GBP':0.00001 36 | ,'EUR_HKD':0.00001 37 | ,'EUR_HUF':0.001 38 | ,'EUR_JPY':0.001 39 | ,'EUR_NOK':0.00001 40 | ,'EUR_NZD':0.00001 41 | ,'EUR_PLN':0.00001 42 | ,'EUR_SEK':0.00001 43 | ,'EUR_SGD':0.00001 44 | ,'EUR_TRY':0.00001 45 | ,'EUR_USD':0.00001 46 | ,'EUR_ZAR':0.00001 47 | ,'FR40_EUR':0.1 48 | ,'GBP_AUD':0.00001 49 | ,'GBP_CAD':0.00001 50 | ,'GBP_CHF':0.00001 51 | ,'GBP_HKD':0.00001 52 | ,'GBP_JPY':0.001 53 | ,'GBP_NZD':0.00001 54 | ,'GBP_PLN':0.00001 55 | ,'GBP_SGD':0.00001 56 | ,'GBP_USD':0.00001 57 | ,'GBP_ZAR':0.00001 58 | ,'HK33_HKD':0.1 59 | ,'HKD_JPY':0.00001 60 | ,'JP225_USD':0.1 61 | ,'NAS100_USD':0.1 62 | ,'NATGAS_USD':0.001 63 | ,'NL25_EUR':0.001 64 | ,'NZD_CAD':0.00001 65 | ,'NZD_CHF':0.00001 66 | ,'NZD_HKD':0.00001 67 | ,'NZD_JPY':0.001 68 | ,'NZD_SGD':0.00001 69 | ,'NZD_USD':0.00001 70 | ,'SG30_SGD':0.01 71 | ,'SGD_CHF':0.00001 72 | ,'SGD_HKD':0.00001 73 | ,'SGD_JPY':0.001 74 | ,'SOYBN_USD':0.001 75 | ,'SPX500_USD':0.1 76 | ,'SUGAR_USD':0.00001 77 | ,'TRY_JPY':0.001 78 | ,'UK100_GBP':0.1 79 | ,'UK10YB_GBP':0.001 80 | ,'US2000_USD':0.001 81 | ,'US30_USD':0.1 82 | ,'USB02Y_USD':0.001 83 | ,'USB05Y_USD':0.001 84 | ,'USB10Y_USD':0.001 85 | ,'USB30Y_USD':0.001 86 | ,'USD_CAD':0.00001 87 | ,'USD_CHF':0.00001 88 | ,'USD_CNH':0.00001 89 | ,'USD_CNY':0.00001 90 | ,'USD_CZK':0.00001 91 | ,'USD_DKK':0.0001 92 | ,'USD_HKD':0.00001 93 | ,'USD_HUF':0.001 94 | ,'USD_INR':0.001 95 | ,'USD_JPY':0.001 96 | ,'USD_MXN':0.00001 97 | ,'USD_NOK':0.00001 98 | ,'USD_PLN':0.00001 99 | ,'USD_SAR':0.00001 100 | ,'USD_SEK':0.00001 101 | ,'USD_SGD':0.00001 102 | ,'USD_THB':0.001 103 | ,'USD_TRY':0.00001 104 | ,'USD_TWD':0.00001 105 | ,'USD_ZAR':0.00001 106 | ,'WHEAT_USD':0.001 107 | ,'WTICO_USD':0.001 108 | ,'XAG_AUD':0.00001 109 | ,'XAG_CAD':0.00001 110 | ,'XAG_CHF':0.00001 111 | ,'XAG_EUR':0.00001 112 | ,'XAG_GBP':0.00001 113 | ,'XAG_HKD':0.00001 114 | ,'XAG_JPY':0.1 115 | ,'XAG_NZD':0.00001 116 | ,'XAG_SGD':0.00001 117 | ,'XAG_USD':0.00001 118 | ,'XAU_AUD':0.001 119 | ,'XAU_CAD':0.001 120 | ,'XAU_CHF':0.001 121 | ,'XAU_EUR':0.001 122 | ,'XAU_GBP':0.001 123 | ,'XAU_HKD':0.001 124 | ,'XAU_JPY':0.1 125 | ,'XAU_NZD':0.001 126 | ,'XAU_SGD':0.001 127 | ,'XAU_USD':0.001 128 | ,'XAU_XAG':0.001 129 | ,'XCU_USD':0.00001 130 | ,'XPD_USD':0.001 131 | ,'XPT_USD':0.001 132 | ,'ZAR_JPY':0.001 133 | } 134 | -------------------------------------------------------------------------------- /src/daemon.py: -------------------------------------------------------------------------------- 1 | """ 2 | File: daemon.py 3 | Python version: Python 3.4 4 | Description: Main function. 5 | """ 6 | 7 | # standard libraries 8 | import atexit 9 | import curses 10 | import datetime 11 | import json 12 | import sys 13 | import time # for sleep() 14 | #import urllib.request 15 | #import urllib.error 16 | 17 | """# private strategy modules 18 | try: 19 | sys.path.index('/home/user/raid/documents/algo/private_strategies') 20 | except ValueError: 21 | sys.path.append('/home/user/raid/documents/algo/private_strategies') 22 | from blah import blah """ 23 | 24 | from broker import Broker 25 | from config import Config 26 | from db import DB 27 | from instrument import Instrument 28 | from log import Log 29 | from oanda import Oanda 30 | from opportunity import * 31 | from order import * 32 | from strategies.fifty import * 33 | from timer import Timer 34 | 35 | 36 | class Daemon(): 37 | """ 38 | The daemon: 39 | - Manages account balance, margin, risk. 40 | - Accepts and rejects trade opportunities. 41 | - Forcefully close trades as needed (e.g. during shutdown) 42 | """ 43 | 44 | # Initialize log. 45 | Log.clear() 46 | 47 | # diagnostic info 48 | DB.execute('INSERT INTO startups (timestamp) VALUES (NOW())') 49 | 50 | # flag to shut down 51 | stopped = False 52 | 53 | # opportunity pool 54 | opportunities = Opportunities() 55 | 56 | #Initialize strategies. 57 | fifty = Fifty(tp_price_diff=0.1, sl_price_diff=0.1) 58 | strategies = [] 59 | strategies.append( fifty ) 60 | # Specify the backup strategy. 61 | backup_strategy = fifty 62 | 63 | msg_base = '' # user interface template 64 | 65 | 66 | @classmethod 67 | def num_strategies_with_positions(cls): 68 | sum = 0 69 | for s in cls.strategies: 70 | if s.get_number_positions() > 0: 71 | sum += 1 72 | return sum 73 | 74 | 75 | @classmethod 76 | def num_strategies_with_no_positions(cls): 77 | return len(cls.strategies) - cls.num_strategies_with_positions() 78 | 79 | 80 | @classmethod 81 | def _curses_init(cls, stdcsr): 82 | # initialize curses 83 | curses.echo() # echo key presses 84 | stdcsr.nodelay(1) # non-blocking window 85 | stdcsr.clear() # clear screen 86 | cls.msg_base = '\n' 87 | if Config.live_trading: 88 | cls.msg_base += 'LIVE TRADING\n' 89 | else: 90 | cls.msg_base += 'Simulation.\n' 91 | cls.msg_base += 'Press q to shut down.\n' 92 | cls.msg_base += 'Press m to monitor.\n\n' 93 | cls.msg_base += 'Account balance: {}\n' 94 | cls.msg_base += '>' 95 | stdcsr.addstr(cls.msg_base) 96 | stdcsr.refresh() # redraw 97 | 98 | 99 | """ 100 | refresh user interface 101 | TODO: this waits for the REST calls to return. Too slow. 102 | """ 103 | @classmethod 104 | def _curses_refresh(cls, stdcsr): 105 | ch = stdcsr.getch() # get one char 106 | if ch == 113: # q == quit 107 | stdcsr.addstr('\nInitiating shutdown...\n') 108 | stdcsr.refresh() # redraw 109 | curses.nocbreak() 110 | stdcsr.keypad(False) 111 | #curses.echo() 112 | curses.endwin() # restore terminal 113 | cls.shutdown() 114 | elif ch == 109: # m == monitor 115 | account_id = Config.account_id 116 | balance = Broker.get_balance(account_id) 117 | msg = cls.msg_base.format(balance) 118 | stdcsr.clear() 119 | stdcsr.addstr(msg) 120 | stdcsr.refresh() # redraw 121 | 122 | 123 | @classmethod 124 | def run(cls, stdcsr): 125 | """Returns: void 126 | This is the main program loop. 127 | """ 128 | # initialize user interface 129 | cls._curses_init(stdcsr) 130 | 131 | # Read in existing trades 132 | while not cls.stopped and cls.recover_trades() == None: 133 | Log.write('"daemon.py" run(): Recovering trades...') 134 | cls._curses_refresh(stdcsr) 135 | 136 | # logging 137 | if Config.live_trading: 138 | Log.write('"daemon.py" start(): Using live account.') 139 | else: 140 | Log.write('"daemon.py" start(): Using practice mode.') 141 | 142 | """ 143 | Main loop: 144 | 1. Gather opportunities from each strategy. 145 | 2. Decide which opportunities to execute. 146 | 3. Clear the opportunity list. 147 | """ 148 | while not cls.stopped: 149 | # refresh user interface 150 | cls._curses_refresh(stdcsr) 151 | 152 | # Let each strategy suggest an order 153 | for s in cls.strategies: 154 | new_opp = s.refresh() 155 | if new_opp == None: 156 | Log.write('daemon.py run(): {} has nothing to offer now.' 157 | .format(s.get_name())) 158 | pass 159 | else: 160 | cls.opportunities.push(new_opp) 161 | 162 | # Decide which opportunity (or opportunities) to execute 163 | Log.write('"daemon.py" run(): Picking best opportunity...') 164 | best_opp = cls.opportunities.pick() 165 | if best_opp == None: 166 | # Nothing is being suggested. 167 | pass 168 | else: 169 | # An order was suggested by a strategy, so place the order. 170 | # Don't use all the money available. 171 | SLIPPAGE_WIGGLE = 0.95 172 | ###available_money = Broker.get_margin_available(Config.account_id) * SLIPPAGE_WIGGLE 173 | available_money = 100 # USD - testing 174 | # Get the current price of one unit. 175 | instrument_price = 0 176 | Log.write('best opp: {}'.format(best_opp)) 177 | go_long = best_opp.order.units > 0 178 | if go_long: 179 | instrument_price = Broker.get_ask(best_opp.order.instrument) 180 | else: 181 | instrument_price = Broker.get_bid(best_opp.order.instrument) 182 | # How much leverage available: 183 | margin_rate = Broker.get_margin_rate(best_opp.order.instrument) 184 | # TODO: A bit awkward, but overwrite the existing value that was used to 185 | # determine long/short. 186 | units = available_money 187 | units /= cls.num_strategies_with_no_positions() # save money for other strategies 188 | units /= margin_rate 189 | units = int(units) # floor 190 | if units <= 0: # verify 191 | Log.write('daemon.py run(): units <= 0') 192 | raise Exception # abort 193 | if not go_long: # negative means short 194 | units = -units 195 | best_opp.order.units = units 196 | Log.write('daemon.py run(): Executing opportunity:\n{}'.format(best_opp)) 197 | order_result = Broker.place_order(best_opp.order) 198 | # Notify the strategies. 199 | if 'orderFillTransaction' in order_result: 200 | try: 201 | opened_trade_id = order_result['orderFillTransaction']['tradeOpened']['tradeID'] 202 | best_opp.strategy.trade_opened(trade_id=opened_trade_id) 203 | except: 204 | Log.write( 205 | 'daemon.py run(): Failed to extract opened trade from order result:\n{}' 206 | .format(order_result) ) 207 | raise Exception 208 | elif 'tradesClosed' in order_result: 209 | try: 210 | for trade in order_result['orderFillTransaction']['tradesClosed']: 211 | best_opp.strategy.trade_closed(trade_id=trade['tradeID']) 212 | except: 213 | Log.write( 214 | 'daemon.py run(): Failed to extract closed trades from order result:\n{}' 215 | .format(order_result) ) 216 | raise Exception 217 | elif 'tradeReduced' in order_result: 218 | try: 219 | closed_trade_id = order_result['orderFillTransaction']['tradeReduced']['tradeID'] 220 | best_opp.strategy.trade_reduced( 221 | closed_trade_id, 222 | instrument_id=Instrument.get_id_from_name(order_result['instrument']) 223 | ) 224 | except: 225 | Log.write( 226 | 'daemon.py run(): Failed to extract reduced trades from order result:\n{}' 227 | .format(order_result) ) 228 | raise Exception 229 | else: 230 | Log.write( 231 | '"daemon.py" run(): Unrecognized order result:\n{}' 232 | .format(order_result) ) 233 | raise Exception 234 | 235 | """ 236 | Clear opportunity list. 237 | Opportunities should be considered to exist only in the moment, 238 | so there is no need to save them for later. 239 | """ 240 | cls.opportunities.clear() 241 | """ 242 | Shutdown stuff. This runs after shutdown() is called, and is the 243 | last code that runs before returning to algo.py. 244 | """ 245 | DB.shutdown() # atexit() used in db.py, but call to be safe. 246 | 247 | 248 | @classmethod 249 | def shutdown(cls): 250 | """ 251 | Stop daemon. 252 | TODO: 253 | * set SL and TP 254 | * write diagnostic info to db 255 | """ 256 | Log.write('"daemon.py" shutdown(): Shutting down daemon.') 257 | cls.stopped = True 258 | 259 | 260 | @classmethod 261 | def recover_trades(cls): 262 | """Returns: None on failure, any value on success 263 | See if there are any open trades, and resume babysitting. 264 | - 265 | If trades are opened without writing their info to the db, 266 | the trade cannot be distributed back to the strategy that opened 267 | it, because it is unknown what strategy placed the order. 268 | This could be solved by writing to the db before placing the order, 269 | synchronously. However if placing the order failed, then the database 270 | record would have to be deleted, and this would be messy. 271 | Instead, designate a backup strategy that adopts orphan trades. 272 | """ 273 | 274 | # Get trades from broker. 275 | open_trades_broker = Broker.get_open_trades() # instance of 276 | if open_trades_broker == None: 277 | Log.write('daemon.py recover_trades(): Broker.get_open_trades() failed.') 278 | return None 279 | 280 | # Delete any trades from the database that are no longer open. 281 | # First, ignore trades that the broker has open. 282 | db_trades = DB.execute('SELECT trade_id FROM open_trades_live') 283 | Log.write('"daemon.py" recover_trades():\ndb open trades: {}\nbroker open trades: {}' 284 | .format(db_trades, open_trades_broker)) 285 | for index, dbt in enumerate(db_trades): # O(n^3) 286 | for otb in open_trades_broker: 287 | if str(dbt[0]) == str(otb.trade_id): # compare trade_id 288 | del db_trades[index] 289 | # The remaining trades are in the "open trades" db table, but 290 | # the broker is not listing them as open. 291 | # They may have closed since the daemon last ran; confirm this. 292 | # Another cause is that trades are automatically removed from 293 | # Oanda's history after much time passes. 294 | for dbt in db_trades: 295 | if Broker.is_trade_closed(dbt[0])[0]: 296 | # Trade is definitely closed; update db. 297 | Log.write('"daemon.py" recover_trades(): Trade {} is closed. Deleting from db.' 298 | .format(dbt[0])) 299 | else: 300 | # Trade is in "open trades" db table and the broker 301 | # says the trade is neither open nor closed. 302 | DB.bug('Trade w/ID ({}) is neither open nor closed.' 303 | .format(dbt[0])) 304 | Log.write('"daemon.py" recover_trades(): Trade w/ID (', 305 | '{}) is neither open nor closed.'.format(dbt[0])) 306 | DB.execute('DELETE FROM open_trades_live WHERE trade_id="{}"' 307 | .format(dbt[0])) 308 | 309 | """ 310 | Fill in info not provided by the broker, e.g. 311 | the name of the strategy that opened the trade. 312 | 313 | It's possible that a trade will be opened then the system is 314 | unexpectedly terminated before info about the trade can be saved to 315 | the database. Thus a trade may not have a corresponding trade in the database. 316 | """ 317 | for i in range(0,len(open_trades_broker)): 318 | broker_trade = open_trades_broker[i] 319 | db_trade_info = DB.execute( 320 | 'SELECT strategy, broker FROM open_trades_live WHERE trade_id="{}"' 321 | .format(broker_trade.trade_id) 322 | ) 323 | if len(db_trade_info) > 0: 324 | # Verify broker's info and database info match, just to be safe. 325 | # - broker name 326 | if db_trade_info[0][1] != broker_trade.broker_name: 327 | Log.write('"daemon.py" recover_trades(): ERROR: "{}" != "{}"' 328 | .format(db_trade_info[0][1], broker_trade.broker_name)) 329 | raise Exception 330 | # set strategy 331 | broker_trade.strategy = None # TODO: use a different default? 332 | for s in cls.strategies: 333 | if s.get_name == db_trade_info[0][0]: 334 | broker_trade.strategy = s # reference to class instance 335 | else: 336 | # Trade in broker but not db. 337 | # Maybe the trade was opened manually. Ignore it. 338 | # Remove from list. 339 | open_trades_broker = open_trades_broker[0:i] + open_trades_broker[i+1:len(open_trades_broker)] # TODO: optimize speed 340 | 341 | # Distribute trades to their respective strategy modules 342 | for broker_trade in open_trades_broker: 343 | if broker_trade.strategy != None: 344 | # Find the strategy that made this trade and notify it. 345 | for s in cls.strategies: 346 | if broker_trade.strategy.get_name() == s.get_name(): 347 | s.adopt(broker_trade.trade_id) 348 | open_trades_broker.remove(broker_trade.trade_id) 349 | break 350 | else: 351 | # It is not known what strategy opened this trade. 352 | # One possible reason is that the strategy that opened the 353 | # trade is no longer open. 354 | # Assign it to the backup strategy. 355 | Log.write('"daemon.py" recover_trades(): Assigning trade ', 356 | ' ({}) to backup strategy ({}).' 357 | .format(broker_trade.trade_id, cls.backup_strategy.get_name())) 358 | cls.backup_strategy.adopt(broker_trade.trade_id) 359 | return 0 # success 360 | 361 | 362 | # There are not destructors in Python, so use this. 363 | atexit.register(Daemon.shutdown) 364 | 365 | -------------------------------------------------------------------------------- /src/db.py: -------------------------------------------------------------------------------- 1 | """ 2 | File: db.py 3 | Python version: 3.4 4 | Description: database 5 | """ 6 | 7 | #************************* 8 | import atexit 9 | import collections 10 | import concurrent.futures 11 | import configparser 12 | from datetime import datetime 13 | import mysql.connector 14 | from mysql.connector import errorcode, errors 15 | from threading import Thread 16 | import timeit 17 | #************************* 18 | from config import Config 19 | from log import Log 20 | #************************* 21 | 22 | class DB(): 23 | config = { 24 | 'user': Config.db_user, 25 | 'password': Config.db_pw, 26 | 'host': Config.db_host, 27 | 'database': Config.db_name 28 | } 29 | cnx = mysql.connector.connect(**config) 30 | cursor = cnx.cursor() 31 | 32 | 33 | """ 34 | """ 35 | @classmethod 36 | def shutdown(cls): 37 | try: 38 | cls.cursor.close() 39 | except: 40 | pass 41 | try: 42 | cls.cnx.close() 43 | except: 44 | pass 45 | 46 | 47 | """ 48 | Return type: 49 | MySQL returns record sets as a list of tuples. 50 | """ 51 | @classmethod 52 | def execute( 53 | cls, 54 | cmd # string 55 | ): 56 | try: 57 | cls.cursor.execute(cmd) 58 | except: 59 | Log.write('db.py execute(): Query failed: "{}"'.format(cmd)) 60 | 61 | try: 62 | cls.cnx.commit() 63 | except mysql.connector.errors.InternalError: 64 | pass 65 | 66 | try: 67 | result = cls.cursor.fetchall() 68 | except errors.InterfaceError: 69 | return [] 70 | else: 71 | return result 72 | 73 | 74 | """ 75 | Report a bug. 76 | Parameter 'bug' is a string. 77 | """ 78 | @classmethod 79 | def bug(cls, bug): 80 | cls.execute('INSERT INTO bugs (timestamp, description) values (NOW(), \'{}\')' 81 | .format(bug)) 82 | 83 | 84 | # There are not destructors in Python, so use this. 85 | atexit.register(DB.shutdown) 86 | -------------------------------------------------------------------------------- /src/instrument.py: -------------------------------------------------------------------------------- 1 | 2 | import atexit 3 | from config import Config 4 | from db import DB 5 | from log import Log 6 | 7 | class Instrument(): 8 | 9 | #self._lookup_table = Instrument.lookup() 10 | 11 | 12 | """ 13 | TODO: Load the instruments from database into memory for fast lookup. 14 | """ 15 | def __init__(self, new_id): 16 | self._id = new_id 17 | self._name = self.get_name_from_id(new_id) 18 | 19 | 20 | def __str__(self): 21 | return 'instrument (id: {}, name: {})'.format(self._id, self._name) 22 | 23 | 24 | """ 25 | Return type: string 26 | """ 27 | def get_id(self): 28 | return self._id 29 | 30 | 31 | """ 32 | Return type: string 33 | Name as a string, depending on the broker being used. 34 | """ 35 | def get_name(self): 36 | return self._name 37 | 38 | 39 | """ 40 | Return type: int 41 | TODO: use the cache 42 | """ 43 | @classmethod 44 | def get_id_from_name(cls, name): 45 | if Config.broker_name == 'oanda': 46 | result = DB.execute( 47 | 'SELECT id FROM instruments WHERE oanda_name="{}"'.format(name)) 48 | if len(result) != 1: 49 | Log.write('instrument.py get_id_from_name(): len(result) was {}'.format(len(result))) 50 | raise Exception 51 | return result[0][0] 52 | else: 53 | raise Exception 54 | 55 | 56 | """ 57 | TODO: use the cache 58 | param: type: 59 | id_key int 60 | """ 61 | @classmethod 62 | def get_name_from_id(cls, id_key): 63 | if Config.broker_name == 'oanda': 64 | result = DB.execute( 65 | 'SELECT oanda_name FROM instruments WHERE id={}'.format(id_key)) 66 | if len(result) != 1: 67 | Log.write('instrument.py get_name_from_id(): len(result) was {}'.format((len(result)))) 68 | raise Exception 69 | return result[0][0] 70 | else: 71 | raise Exception 72 | 73 | -------------------------------------------------------------------------------- /src/log.py: -------------------------------------------------------------------------------- 1 | """ 2 | File: log.py 3 | Python version: 3.4 4 | Description: Class for writing to a log file. 5 | How to use: 6 | Import the class from the module and call `write()`. 7 | The location of the log file is specified in a config file. 8 | Initialization is automatic, but the log file must be closed manually 9 | by calling `shutdown()`. 10 | Remarks 11 | Sensitive information may be written to the log file, so keep it safe. 12 | Important stuff should be saved to the database because that makes it 13 | easier to analyze later. 14 | The log file should only be used for debugging. 15 | """ 16 | 17 | #************************* 18 | import datetime 19 | #************************* 20 | from config import Config 21 | #************************* 22 | 23 | class Log(): 24 | """ 25 | The file is opened and closed every call to write() to flush the 26 | data and make the file watchable. 27 | """ 28 | 29 | @classmethod 30 | def clear(cls): 31 | """ 32 | clear log 33 | """ 34 | with open(Config.log_path, 'w') as f: 35 | f.write('') 36 | f.close() 37 | 38 | 39 | @classmethod 40 | def write(cls, *args): 41 | """ 42 | append to log 43 | """ 44 | timestamp = datetime.datetime.now().strftime("%c") 45 | msg = '\n' + timestamp + ': ' 46 | for a in list(args): 47 | msg = msg + str(a) 48 | with open(Config.log_path, 'a') as f: 49 | f.write(msg) 50 | f.close 51 | 52 | -------------------------------------------------------------------------------- /src/main.py: -------------------------------------------------------------------------------- 1 | 2 | """ 3 | Run the daemon here. 4 | """ 5 | 6 | #################### 7 | import cmd 8 | import curses 9 | from threading import Thread 10 | import sys 11 | #################### 12 | from daemon import Daemon 13 | #################### 14 | 15 | if __name__ == "__main__": 16 | curses.wrapper(Daemon.run) 17 | 18 | 19 | -------------------------------------------------------------------------------- /src/oanda.py: -------------------------------------------------------------------------------- 1 | """ 2 | Custom Python wrapper for Oanda's REST-V20 API. 3 | This wrapper might be incomplete. 4 | http://developer.oanda.com/ 5 | """ 6 | 7 | #-------------------------- 8 | import datetime 9 | import gzip 10 | import json 11 | import socket 12 | import sys 13 | import time 14 | import traceback 15 | import urllib.request 16 | import urllib.error 17 | import zlib 18 | #-------------------------- 19 | from config import Config 20 | from currency_pair_conversions import * 21 | from instrument import Instrument 22 | import util_date 23 | from log import Log 24 | from trade import * 25 | from timer import Timer 26 | import utils 27 | import util_date 28 | #-------------------------- 29 | 30 | class Oanda(): 31 | """ 32 | Class methods are used because only one instance is ever needed and unlike 33 | static methods, the methods need to share data. 34 | """ 35 | account_id_primary = 0 36 | 37 | 38 | @classmethod 39 | def __str__(cls): 40 | return "oanda" 41 | 42 | 43 | @classmethod 44 | def get_auth_key(cls): 45 | """Return type: string 46 | Return value: authentication token 47 | Oanda was returning a '400 Bad Request' error 4 out of 5 times 48 | until I removed the trailing '\n' from the string 49 | returned by f.readline(). 50 | """ 51 | return Config.oanda_token 52 | 53 | 54 | @classmethod 55 | def fetch(cls, 56 | in_url, # string 57 | in_headers={}, # dict 58 | in_data=b"", # typically JSON string, encoded to bytes 59 | in_origin_req_host=None, # string 60 | in_unverifiable=False, # bool 61 | in_method='GET' # string 62 | ): 63 | """Return type: dict on success, None on failure. 64 | Sends a request to Oanda's API. 65 | """ 66 | # If headers are specified, use those. 67 | headers = None 68 | if in_headers == {}: 69 | headers = {\ 70 | 'Authorization': 'Bearer ' + cls.get_auth_key(),\ 71 | 'Content-Type': 'application/json',\ 72 | 'Accept-Encoding': 'gzip, deflate', 73 | 'Accept-Datetime-Format': 'RFC3339', 74 | 'Content-Length': len(in_data) 75 | } 76 | else: 77 | headers = in_headers 78 | Log.write('"oanda.py" fetch(): /*****************************\\' ) 79 | Log.write('"oanda.py" fetch():\n\ 80 | {}:{}\n\ 81 | in_data: {} origin_req_host: {} unverifiable: {}\n\ 82 | headers: {}\ 83 | '.format( 84 | in_url, utils.btos(in_data), in_origin_req_host, 85 | in_unverifiable, in_method, headers) 86 | ) 87 | Log.write('"oanda.py" fetch(): \\*****************************/' ) 88 | # send request 89 | req = urllib.request.Request(in_url, in_data, headers, in_origin_req_host, in_unverifiable, in_method) 90 | response = cls.send_http_request(req) 91 | while not response[0] and response[2]: 92 | # Failure. Wait and try again. 93 | time.sleep(1) 94 | Log.write('oanda.py fetch(): Resending request...') 95 | response = cls.send_http_request(req) 96 | if response[0]: 97 | # Success. Get the response data. 98 | """ "response.info() is email.message_from_string(); it needs to be 99 | cast to a string." 100 | """ 101 | return response[1] 102 | else: 103 | # Gave up trying. 104 | return None 105 | 106 | 107 | @classmethod 108 | def read_response(cls, response): 109 | """Returns: response data as dict 110 | This is a helper function for fetch(). 111 | """ 112 | header = response.getheader('Content-Encoding') 113 | if header != None: 114 | # Check how the response data is encoded. 115 | if header.strip().startswith('gzip'): 116 | Log.write('oanda.py read_response(): gzip payload') 117 | return utils.btos(gzip.decompress(response.read())) 118 | else: 119 | if header.strip().startswith('deflate'): 120 | Log.write('oanda.py read_response(): zlib payload') 121 | return utils.btos( zlib.decompress( response.read() ) ) 122 | else: 123 | Log.write('oanda.py read_response(): Unknown header.') 124 | return utils.btos( response.read() ) 125 | else: 126 | Log.write('oanda.py read_response(): No header.') 127 | return utils.btos(response.read()) 128 | 129 | 130 | @classmethod 131 | def send_http_request( 132 | cls, 133 | request, # request object 134 | ): 135 | """Returns tuple: 136 | ( 137 | bool: was success 138 | object: success -> data(probably dict) 139 | not success -> error_object 140 | bool: should retry 141 | ) 142 | This is a helper function for fetch(). 143 | """ 144 | try: 145 | response = urllib.request.urlopen(request) 146 | except urllib.error.HTTPError as e: 147 | # e.code(): 148 | # 204: No candlesticks during requested time. 149 | # 400: 150 | # 404: Tried to get trade info for a closed trade. 151 | # 415: unsupported media type (content-encoding) 152 | # 503: Service Unavailable (e.g. scheduled maintenance) 153 | Log.write('oanda.py send_http_request(): HTTPError:\n{}'.format(str(e))) 154 | if e.code in ['503']: 155 | return (False, e, True) 156 | else: 157 | return (False, e, False) 158 | except urllib.error.URLError as e: 159 | # https://docs.python.org/3.4/library/traceback.html 160 | Log.write('oanda.py send_http_request(): URLError:\n{}'.format(str(e))) 161 | return (False, e, False) 162 | except ConnectionError as e: 163 | Log.write('oanda.py send_http_request(): ConnectionError:\n{}'.format(str(e))) 164 | return (False, e, True) 165 | except OSError as e: 166 | # Not sure why this happens. Usually when market is closed. 167 | Log.write('oanda.py send_http_request(): OSError:\n{}'.format(str(e))) 168 | return (False, e, True) 169 | except Exception as e: 170 | # some other error type 171 | Log.write('oanda.py send_http_request(): some other error:\n{}'.format(str(e))) 172 | """ 173 | exc_type, exc_value, exc_traceback = sys.exc_info() 174 | Log.write('oanda.py send_http_request(): Exception: ', exc_type) 175 | Log.write('oanda.py send_http_request(): EXC INFO: ', exc_value) 176 | Log.write('oanda.py send_http_request(): TRACEBACK:\n', traceback.print_exc(), '\n') 177 | """ 178 | return (False, e, True) 179 | else: 180 | # no exceptions == success 181 | Log.write( 182 | 'oanda.py fetch(): successful response.info(): {}' 183 | .format(response.info()) ) 184 | header = response.getheader('Content-Encoding') 185 | if header != None: 186 | if header.strip().startswith('gzip'): 187 | resp_data = utils.btos(gzip.decompress(response.read())) 188 | else: 189 | if header.strip().startswith('deflate'): 190 | resp_data = utils.btos( zlib.decompress( response.read() ) ) 191 | else: 192 | resp_data = utils.btos( response.read() ) 193 | else: 194 | resp_data = utils.btos(response.read()) 195 | try: 196 | return ( True, json.loads(resp_data) ) 197 | except: 198 | Log.write( 199 | 'oanda.py send_http_request(): Failed to parse JSON:\n{}' 200 | .format( resp_data ) ) 201 | raise Exception 202 | 203 | 204 | @classmethod 205 | def get_accounts(cls): 206 | """Returns: dict 207 | Get list of accounts 208 | """ 209 | return cls.fetch(Config.oanda_url + '/v3/accounts') 210 | 211 | 212 | @classmethod 213 | def get_account(cls, account_id): 214 | """Returns: dict or None 215 | Get account info for a given account ID. 216 | """ 217 | account = cls.fetch(Config.oanda_url + '/v3/accounts/' + account_id) 218 | if account == None: 219 | Log.write('"oanda.py" get_account(): Failed to get account.') 220 | return None 221 | else: 222 | return account 223 | 224 | 225 | @classmethod 226 | def get_account_summary( 227 | cls, 228 | account_id # string 229 | ): 230 | """return type: dict 231 | """ 232 | return cls.fetch( 233 | Config.oanda_url + '/v3/accounts/{}/summary'.format(account_id) 234 | ) 235 | 236 | 237 | @classmethod 238 | def get_margin_available(cls, account_id): 239 | """Return type: float 240 | Return value: margin available 241 | """ 242 | account_summary = cls.get_account_summary(account_id) 243 | try: 244 | return float(account_summary['account']['marginAvailable']) 245 | except: 246 | Log.write('oanda.py get_margin_available(): Failed to extract marginAvailable.') 247 | raise Exception 248 | 249 | 250 | @classmethod 251 | def get_margin_rate( 252 | cls, 253 | instrument # instance 254 | ): 255 | """Return type: float 256 | """ 257 | query_args = '?instruments={}'.format(instrument.get_name()) 258 | instruments_info = cls.fetch( 259 | in_url='{}/v3/accounts/{}/instruments{}' 260 | .format(Config.oanda_url, Config.account_id, query_args) 261 | ) 262 | try: 263 | Log.write('oanda.py get_margin_rate(): returned data: \n{}' 264 | .format(instruments_info)) 265 | return float( instruments_info['instruments'][0]['marginRate'] ) 266 | except: 267 | Log.write('oanda.py get_margin_rate(): Failed to extract ' \ 268 | + 'marginRate. Oanda returned:\n{}' 269 | .format(instruments_info) 270 | ) 271 | raise Exception 272 | 273 | 274 | @classmethod 275 | def get_positions(cls, account_id): 276 | """ 277 | Get list of open positions 278 | Returns: dict or None 279 | """ 280 | pos = cls.fetch(Config.oanda_url + '/v3/accounts/' + account_id + '/positions') 281 | if pos == None: 282 | Log.write('"oanda.py" get_positions(): Failed to get positions.') 283 | return None 284 | else: 285 | return pos 286 | 287 | 288 | @classmethod 289 | def get_num_of_positions(cls, account_id): 290 | """Returns: Integer 291 | Get number of positions for a given account ID 292 | """ 293 | positions = cls.get_positions(account_id) 294 | if positions == None: 295 | Log.write('"oanda.py" get_num_of_positions(): Failed to get positions.') 296 | return None 297 | else: 298 | return len(positions['positions']) 299 | 300 | 301 | @classmethod 302 | def get_balance(cls, account_id): 303 | """Return type: Decimal number 304 | Get account balance for a given account ID 305 | """ 306 | account_summary = cls.get_account_summary(account_id) 307 | try: 308 | return float(account_summary['account']['balance']) 309 | except: 310 | Log.write('oanda.py get_balance(): Failed to extract balance from account summary. Summary from broker was:\n{}' 311 | .format(account_summary)) 312 | raise Exception 313 | 314 | 315 | @classmethod 316 | def get_prices( 317 | cls, 318 | instruments, # [] 319 | since=None # string 320 | ): 321 | """Return type: dict or None 322 | Fetch live prices for specified instruments that are available on the OANDA platform. 323 | """ 324 | Log.write('oanda.py get_prices()') 325 | url_args = 'instruments=' + utils.instruments_to_url(instruments) 326 | if since != None: 327 | url_args += '&since=' + since 328 | prices = cls.fetch( '{}/v3/accounts/{}/pricing?{}' 329 | .format(Config.oanda_url, Config.account_id, url_args) ) 330 | if prices == None: 331 | Log.write('"oanda.py" get_prices(): Failed to get prices.') 332 | return None 333 | else: 334 | return prices 335 | 336 | 337 | @classmethod 338 | def get_ask( 339 | cls, 340 | instrument, # instance 341 | since=None # string (see Oanda documentation) 342 | ): 343 | """Return type: Decimal or None 344 | Get lowest ask price. 345 | """ 346 | Log.write('oanda.py get_ask()') 347 | prices = cls.get_prices([instrument], since) 348 | if prices == None: 349 | Log.write('"oanda.py" get_ask(): Failed to get prices.') 350 | return None 351 | else: 352 | try: 353 | for p in prices['prices']: 354 | if p['instrument'] == instrument.get_name(): 355 | return float(p['asks'][0]['price']) 356 | except Excpetion: 357 | Log.write('oanda.py get_ask(): Failed to extract ask from price data. Price data:\n{}' 358 | .format(prices)) 359 | raise Exception 360 | 361 | 362 | @classmethod 363 | def get_bid(cls, 364 | instrument, # 365 | since=None 366 | ): 367 | """Return type: decimal or None 368 | Get highest bid price. 369 | """ 370 | Log.write('oanda.py get_bid()') 371 | prices = cls.get_prices([instrument], since) 372 | if prices == None: 373 | Log.write('"oanda.py" get_bid(): Failed to get prices.') 374 | return None 375 | else: 376 | try: 377 | for p in prices['prices']: 378 | if p['instrument'] == instrument.get_name(): 379 | return float(p['bids'][0]['price']) 380 | except Exception: 381 | Log.write('oanda.py get_bid(): Failed to extract bid from price data. Price data:\n{}' 382 | .format(prices)) 383 | raise Exception 384 | 385 | 386 | @classmethod 387 | def get_spreads( 388 | cls, 389 | instruments, # [] 390 | since=None 391 | ): 392 | """Returns: list 393 | Get spread, in pips, for given currency pairs (e.g. 'USD_JPY%2CEUR_USD') 394 | """ 395 | Log.write('oanda.py get_spreads()') 396 | prices = cls.get_prices(instruments, since) 397 | Log.write('prices: \n{}'.format(prices)) 398 | if prices == None: 399 | Log.write('oanda.py get_spreads(): Failed to get prices.') 400 | return None 401 | else: 402 | spreads = [] 403 | for p in prices['prices']: 404 | # Oanda deprecated 'status' but 'tradeable' not used yet? 405 | tradeable = None 406 | try: 407 | tradeable = p['tradeable'] 408 | except: 409 | try: 410 | tradeable = ( p['status'] == 'tradeable' ) 411 | except: 412 | tradeable = True 413 | spreads.append( 414 | { 415 | "instrument":p['instrument'], 416 | "time":p['time'], 417 | "spread":price_to_pips(p['instrument'], (float(p['asks'][0]['price']) - float(p['bids'][0]['price']))), 418 | "tradeable":tradeable 419 | } 420 | ) 421 | return spreads 422 | 423 | 424 | @classmethod 425 | def place_order(cls, in_order): 426 | """Return type: dict 427 | Return value: information about the order (and related trade) 428 | Description: Place an order. 429 | 430 | If I place a trade that reduces another trade to closing, then I get a 431 | 200 Code and information about the trade that closed. I.e. I don't get 432 | info about an opened trade. 433 | 434 | http://developer.oanda.com/rest-live-v20/order-df/#OrderRequest 435 | """ 436 | Log.write ('"oanda.py" place_order(): Placing order...') 437 | request_body = { "order" : {} } 438 | # type 439 | request_body["order"]["type"] = in_order.order_type 440 | # instrument 441 | request_body["order"]["instrument"] = in_order.instrument.get_name() 442 | # units 443 | request_body["order"]["units"] = str(in_order.units) 444 | # time-in-force (market order) 445 | if in_order.order_type == "MARKET": 446 | request_body["order"]["timeInForce"] = "FOK" 447 | else: 448 | raise Exception # TODO 449 | # position fill 450 | request_body["order"]["positionFill"] = "DEFAULT" 451 | # stop loss 452 | if in_order.stop_loss != None: 453 | request_body["order"]["stopLossOnFill"] = in_order.stop_loss 454 | # take profit 455 | if in_order.take_profit != None: 456 | request_body["order"]["takeProfitOnFill"] = in_order.take_profit 457 | # trailing stop 458 | if in_order.trailing_stop != None: 459 | request_body["order"]["trailingStopLossOnFill"] = in_order.trailing_stop 460 | data = utils.stob( json.dumps( request_body ) ) # Oanda needs double quotes in JSON 461 | Log.write('oanda.py place_order(): data = {}'.format(data) ) 462 | result = cls.fetch( 463 | in_url="{}/v3/accounts/{}/orders".format( 464 | Config.oanda_url, 465 | Config.account_id 466 | ), 467 | in_data=data, 468 | in_method='POST' 469 | ) 470 | if result == None: 471 | DB.bug('"oanda.py" place_order(): Failed to place order (1st try).') 472 | Log.write('"oanda.py" place_order(): Failed to place order; one more try.') 473 | time.sleep(1) 474 | result = cls.fetch( 475 | in_url="{}/v3/accounts/{}/orders".format( 476 | Config.oanda_url, 477 | Config.account_id 478 | ), 479 | in_data=data, 480 | in_method='POST' 481 | ) 482 | if result == None: 483 | DB.bug('"oanda.py" place_order(): Failed to place order (2nd try).') 484 | Log.write('"oanda.py" place_order(): Failed to place order 2nd time.') 485 | return None 486 | else: 487 | Log.write ('"oanda.py" place_order(): Order successfully placed.') 488 | return result 489 | 490 | 491 | @classmethod 492 | def is_market_open( 493 | cls, 494 | instrument # instance 495 | ): 496 | """Return type: boolean 497 | Return value: true if market currently open, according to Oanda's API. 498 | """ 499 | Log.write('oanda.py is_market_open()') 500 | prices = cls.get_prices([instrument]) 501 | try: 502 | return prices['prices'][0]['status'] == 'tradeable' 503 | except Exception: 504 | Log.write( 505 | 'oanda.py is_market_open(): Failed to get key \'status\'. \ninstr:{}\nprices: {}'.format(instrument, prices)) 506 | raise Exception 507 | 508 | 509 | """ 510 | Standard retail trading hours (no special access), 511 | or depending on broker. 512 | WARNING: These are blatantly wrong, as they do not take into account 513 | Daylight Savings Time. 514 | """ 515 | market_opens = { 516 | # 10pm UTC, 7am JST 517 | util_date.SUNDAY: [datetime.timedelta(hours=22)] 518 | } 519 | market_closes = { 520 | # 10pm UTC, 7am JST 521 | util_date.FRIDAY: [datetime.timedelta(hours=22)] 522 | } 523 | 524 | 525 | @classmethod 526 | def get_time_until_close(cls): 527 | """Return type: datetime.timedelta 528 | Return value: 529 | timedelta of 0 if market is already closed, 530 | otherwise timedelta until market closes 531 | """ 532 | zero = datetime.timedelta() 533 | if not cls.is_market_open(Instrument(4)): # actually check the broker first 534 | return zero 535 | now = datetime.datetime.utcnow() 536 | now_day = now.isoweekday() 537 | now_delta = datetime.timedelta( 538 | hours=now.hour, 539 | minutes=now.minute, 540 | seconds=now.second, 541 | microseconds=now.microsecond 542 | ) 543 | now_to_soonest_close = datetime.timedelta(days=8) # > 1 week 544 | total_delta_to_close = zero 545 | day_index = now_day 546 | # Starting from today, iterate through each weekday and 547 | # see if the market will close that day. 548 | for i in range(1, 8): # match python datetime.isoweekday() 549 | closes = cls.market_closes.get(day_index) 550 | if closes != None: 551 | # there is at least one close this day 552 | for c in closes: 553 | if now_delta < c and c - now_delta < now_to_soonest_close: 554 | # new soonest close 555 | now_to_soonest_close = c - now_delta 556 | # If there is an open time today: 557 | # If there was a close, look for open < soonest close. 558 | # Else, market is closed 559 | opens = cls.market_opens.get(day_index) 560 | if opens != None: 561 | if now_to_soonest_close < datetime.timedelta(days=8): 562 | for o in opens: 563 | if now_delta < o and (o - now_delta) < now_to_soonest_close: 564 | # market will open before closing 565 | return zero 566 | else: 567 | # market is closed 568 | return zero 569 | # return soonest close 570 | total_delta_to_close += now_to_soonest_close 571 | return total_delta_to_close 572 | # cycle the index 573 | total_delta_to_close += datetime.timedelta(hours=24) 574 | day_index += 1 575 | if day_index > 7: 576 | day_index = 1 577 | Log.write('oanda.py get_time_until_close(): Close time not found.') 578 | raise Exception 579 | 580 | 581 | @classmethod 582 | def get_time_since_close(cls): 583 | """Return type: datetime.timedelta 584 | Return value: Time passed since last market close, regardless of 585 | current open/close status. 586 | """ 587 | # Iterate backwards through days and return the most recent time. 588 | now = datetime.datetime.utcnow() 589 | day_iter = now.isoweekday() 590 | now_delta = datetime.timedelta( 591 | hours=now.hour, 592 | minutes=now.minute, 593 | seconds=now.second, 594 | microseconds=now.microsecond 595 | ) 596 | zero = datetime.timedelta() 597 | total_delta_since_close = zero # default starting value 598 | for d in range(1,9): # eight days; market may close later today 599 | closes = cls.market_closes.get(day_iter) 600 | if closes != None: 601 | # There are closes this day. 602 | latest_close_to_now_delta = datetime.timedelta(days=1) # default starting value 603 | for c in closes: 604 | if now_delta - c < latest_close_to_now_delta: 605 | # make sure it's not a time later today 606 | if d > 1 or now_delta > c: 607 | # new potential most recent close 608 | latest_close_to_now_delta = now_delta - c 609 | # Only return if a close earlier than now was found 610 | if latest_close_to_now_delta < datetime.timedelta(days=1): 611 | total_delta_since_close += latest_close_to_now_delta 612 | return total_delta_since_close 613 | day_iter -= 1 # move to previous day 614 | if day_iter < 1: # cycle 615 | day_iter = 7 616 | total_delta_since_close += datetime.timedelta(hours=24) 617 | raise Exception 618 | 619 | 620 | @classmethod 621 | def get_transactions_since_id(cls, start_id): 622 | """Returns: dict or None 623 | Gets transactions since start_id 624 | """ 625 | transactions = cls.fetch( 626 | in_url='{}/v3/accounts/{}/transactions/sinceid?id={}' 627 | .format(Config.oanda_url, Config.account_id, start_id) 628 | ) 629 | if transactions == None: 630 | DB.bug('"oanda.py" get_transaction_since_id(): Failed to fetch transaction history.') 631 | Log.write('"oanda.py" get_transaction_since_id(): Failed to fetch transaction history.') 632 | return None 633 | else: 634 | return transactions 635 | 636 | 637 | @classmethod 638 | def get_transactions_since_id( 639 | cls, 640 | last_id, # string 641 | ): 642 | """Return type: dict 643 | Returns transactions since, but not including, last_id. 644 | """ 645 | args = '?id={}'.format(last_id) 646 | transactions = cls.fetch( 647 | in_url='{}/v3/accounts/{}/transactions/sinceid{}' 648 | .format(Config.oanda_url, Config.account_id, args) 649 | ) 650 | if not transactions: 651 | Log.write('oanda.py get_transactions_since_id(): result == None') 652 | raise Exception 653 | return transactions 654 | 655 | 656 | @classmethod 657 | def get_instrument_history( 658 | cls, 659 | in_instrument, # 660 | granularity, # string 661 | count, # optional- int - leave out if both start & end specified 662 | from_time, # optional- datetime 663 | to, # optional- datetime 664 | price, # optional - string 665 | include_first, # optional - bool - Oanda wants 'true'/'false' 666 | daily_alignment, # 0 to 23 - optional 667 | alignment_timezone, # timezone - optional 668 | weekly_alignment # 'Monday' etc. - optional 669 | ): 670 | """Return type: dict or None 671 | """ 672 | if count != None and from_time != None and to != None: 673 | raise Exception 674 | args='' 675 | if granularity != None: 676 | args = args + '&granularity=' + granularity 677 | if count != None: 678 | args = args + '&count=' + str(count) 679 | if from_time != None: 680 | args = args + '&from=' + utils.url_encode(util_date.date_to_string(from_time)) 681 | if to != None: 682 | args = args + '&to=' + utils.url_encode(util_date.date_to_string(to)) 683 | if price != None: 684 | args = args + '&price=' + price 685 | if include_first != None: 686 | if include_first: 687 | args = args + '&includeFirst=' + 'true' 688 | else: 689 | args = args + '&includeFirst=' + 'true' 690 | if daily_alignment != None: 691 | args = args + '&dailyAlignment=' + str(daily_alignment) 692 | if alignment_timezone != None: 693 | args = args + '&alignmentTimezone=' + alignment_timezone 694 | if weekly_alignment != None: 695 | args = args + '&weeklyAlignment=' + weekly_alignment 696 | 697 | result = cls.fetch( 698 | in_url='{}/v3/instruments/{}/candles?{}'.format(Config.oanda_url, in_instrument.get_name(), args) 699 | ) 700 | if result == None: 701 | DB.bug('"oanda.py" get_instrument_history(): Failed to fetch.') 702 | Log.write('"oanda.py" get_instrument_history(): Failed to fetch.') 703 | return None 704 | else: 705 | return result 706 | 707 | 708 | @classmethod 709 | def is_trade_closed(cls, trade_id): 710 | """Returns: 711 | Tuple: ( 712 | True=trade closed; False=not closed 713 | or None 714 | ) 715 | TODO: Function name implies bool, but return val is tuple. 716 | Go through all transactions that have occurred since a given order, and see if any of those 717 | transactions have closed or canceled the order. 718 | """ 719 | # NOTE: /v3/accounts/{accountID}/trades/{tradeSpecifier} 720 | # has a 'status' key that can be used, if the reason is 721 | # not needed. 722 | start = Timer.start() 723 | num_attempts = 2 724 | while num_attempts > 0: 725 | Log.write('"oanda.py" is_trade_closed(): Remaining attempts: ', str(num_attempts)) 726 | transactions = cls.get_transactions_since_id(last_id=trade_id) 727 | #Log.write('oanda.py is_trade_closed(): transactions found:\n{}'.format(transactions)) 728 | for transaction in transactions['transactions']: 729 | if 'type' in transaction: 730 | if transaction['type'] == 'ORDER_FILL': 731 | if 'tradesClosed' in transaction: 732 | for closed_trade in transaction['tradesClosed']: 733 | if closed_trade['tradeID'] == trade_id: 734 | if 'reason' in transaction: 735 | reason = transaction['reason'] 736 | Log.write('oanda.py is_trade_closed(): reason = {}'.format(reason)) 737 | Timer.stop(start, 'Oanda.is_trade_closed()', 'xxx') 738 | if reason == 'LIMIT_ORDER': 739 | return (True, TradeClosedReason.LIMIT_ORDER) 740 | elif reason == 'STOP_ORDER': 741 | return (True, TradeClosedReason.STOP_ORDER) 742 | elif reason == 'MARKET_IF_TOUCHED_ORDER': 743 | return (True, TradeClosedReason.MARKET_IF_TOUCHED_ORDER) 744 | elif reason == 'TAKE_PROFIT_ORDER': 745 | return (True, TradeClosedReason.TAKE_PROFIT_ORDER) 746 | elif reason == 'STOP_LOSS_ORDER': 747 | return (True, TradeClosedReason.STOP_LOSS_ORDER) 748 | elif reason == 'TRAILING_STOP_LOSS_ORDER': 749 | return (True, TradeClosedReason.TRAILING_STOP_LOSS_ORDER) 750 | elif reason == 'MARKET_ORDER': 751 | return (True, TradeClosedReason.MARKET_ORDER) 752 | elif reason == 'MARKET_ORDER_TRADE_CLOSE': 753 | return (True, TradeClosedReason.MARKET_ORDER_TRADE_CLOSE) 754 | elif reason == 'MARKET_ORDER_POSITION_CLOSEOUT': 755 | return (True, TradeClosedReason.MARKET_ORDER_POSITION_CLOSEOUT) 756 | elif reason == 'MARKET_ORDER_MARGIN_CLOSEOUT': 757 | return (True, TradeClosedReason.MARKET_ORDER_MARGIN_CLOSEOUT) 758 | elif reason == 'MARKET_ORDER_DELAYED_TRADE_CLOSE': 759 | return (True, TradeClosedReason.MARKET_ORDER_DELAYED_TRADE_CLOSE) 760 | elif reason == 'LINKED_TRADE_CLOSED': 761 | return (True, TradeClosedReason.LINKED_TRADE_CLOSED) 762 | Log.write( 763 | 'oanda.py is_trade_closed(): Unknown OrderFillReason: {}' 764 | .format(reason) ) 765 | raise Exception 766 | num_attempts = num_attempts - 1 767 | # Delay to allow the trade to be processed on the dealer end 768 | time.sleep(1) 769 | Log.write('oanda.py is_trade_closed(): Unable to locate trade; possibly still open.') 770 | Timer.stop(start, 'Oanda.is_trade_closed()', 'unable to locate') 771 | return (False, None) 772 | 773 | 774 | @classmethod 775 | def get_open_trades(cls): 776 | """Return type: Instance of or None 777 | Get info about all open trades 778 | """ 779 | #Log.write('"oanda.py" get_open_trades(): Entering.') 780 | trades_oanda = cls.fetch('{}/v3/accounts/{}/openTrades' 781 | .format(Config.oanda_url,str(Config.account_id)) 782 | ) 783 | if trades_oanda == None: 784 | Log.write('"oanda.py" get_open_trades(): Failed to get trades from Oanda.') 785 | return None 786 | else: 787 | ts = Trades() 788 | for t in trades_oanda['trades']: 789 | # format into a 790 | ts.append(Trade( 791 | units=t['initialUnits'], 792 | broker_name = cls.__str__(), 793 | instrument = Instrument(Instrument.get_id_from_name(t['instrument'])), 794 | stop_loss = t['stopLossOrder']['price'], 795 | strategy = None, 796 | take_profit = t['takeProfitOrder']['price'], 797 | trade_id = t['id'] 798 | )) 799 | return ts 800 | 801 | 802 | @classmethod 803 | def get_trade(cls, trade_id): 804 | """Returns: or None 805 | Get info about a particular trade. 806 | """ 807 | trade_info = cls.fetch( 808 | '{}/v3/accounts/{}/trades/{}'.format( 809 | Config.oanda_url, 810 | str(Config.account_id), 811 | str(trade_id) 812 | ) 813 | ) 814 | try: 815 | trade = trade_info['trade'] 816 | sl = None 817 | tp = None 818 | if 'stopLossOrder' in trade: 819 | sl = trade['stopLossOrder']['price'] 820 | if 'takeProfitOrder' in trade: 821 | tp = trade['takeProfitOrder']['price'] 822 | return Trade( 823 | units=trade['initialUnits'], 824 | broker_name = cls.__str__(), 825 | instrument = Instrument(Instrument.get_id_from_name(trade['instrument'])), 826 | stop_loss = sl, 827 | take_profit = tp, 828 | strategy = None, 829 | trade_id = trade['id'] 830 | ) 831 | except Exception: 832 | # Oanda returns 404 error if trade closed; don't raise Exception. 833 | Log.write('oanda.py get_trade(): Exception:\n{}'.format(sys.exc_info())) 834 | Log.write('"oanda.py" get_trade(): Failed to get trade info for trade with ID ', trade_id, '.') 835 | return None 836 | 837 | 838 | @classmethod 839 | def get_order_info(cls, order_id): 840 | """Returns: dict or None 841 | Get order info 842 | """ 843 | response = cls.fetch(\ 844 | Config.oanda_url + '/v3/accounts/' + str(Config.account_id) + '/orders/' + str(order_id) ) 845 | if response == None: 846 | Log.write('"oanda.py" get_order_info(): Failed to get order info.') 847 | return None 848 | else: 849 | return response 850 | 851 | 852 | @classmethod 853 | def modify_trade( 854 | cls, 855 | trade_id, 856 | take_profit_price=None, 857 | stop_loss_price=None, 858 | trailing_stop_loss_distance=None 859 | ): 860 | """Returns: 861 | This is trimmed down from Oanda's v20 API. 862 | """ 863 | #Log.write('"oanda.py" modify_trade(): Entering.') 864 | 865 | request_body = {} 866 | if take_profit_price: 867 | request_body['takeProfit'] = {'price':take_profit_price} 868 | if stop_loss_price: 869 | request_body['stopLoss'] = {'price':stop_loss_price} 870 | if trailing_stop_loss_distance: 871 | request_body['trailingStopLoss'] = {'distance':trailing_stop_loss_distance} 872 | 873 | response = cls.fetch( 874 | in_url='{}/v3/accounts/{}/trades/{}/orders' 875 | .format( 876 | Config.oanda_url, 877 | Config.account_id, 878 | str(trade_id) 879 | ), 880 | in_data=utils.stob( # Oanda needs double quotes 881 | json.dumps(request_body) 882 | ), 883 | in_method='PUT' 884 | ) 885 | if response != None: 886 | return response 887 | else: 888 | Log.write('"oanda.py" modify_trade(): Failed to modify trade.') 889 | return None 890 | 891 | 892 | -------------------------------------------------------------------------------- /src/opportunity.py: -------------------------------------------------------------------------------- 1 | """ 2 | Python verion: 3.4 3 | An is a wrapper for one potential order. 4 | An is a list of opportunities. 5 | """ 6 | 7 | #************************* 8 | #************************* 9 | from log import Log 10 | from order import Order 11 | #************************* 12 | 13 | class Opportunity(): 14 | 15 | def __init__(self, order, conf, strat, reason): 16 | # the order that would be sent to the broker 17 | self.order = order 18 | # estimate of success (int 1-100) 19 | self.confidence = conf 20 | # Reference to Strategy instance that created this opportunity. 21 | # objects also hold a reference to their strategy. 22 | self.strategy = strat 23 | # A note to save to the database. 24 | self.reason = reason 25 | 26 | 27 | def __str__(self): 28 | return 'Opportunity:\norder: {}\nconfidence: {}\nstrategy: {}\nreason: {}\n\ 29 | '.format(self.order, self.confidence, self.strategy.get_name(), self.reason) 30 | 31 | 32 | class Opportunities(): 33 | """ 34 | """ 35 | 36 | def __init__(self): 37 | """ 38 | List of . 39 | """ 40 | self._opportunities = [] 41 | 42 | 43 | def __str__(self): 44 | """ 45 | """ 46 | return 'Opportunities' 47 | 48 | 49 | def clear(self): 50 | self._opporunities = [] 51 | 52 | 53 | def push(self, opp): 54 | """ 55 | Add an to the list. 56 | """ 57 | Log.write('pushing opp: {}'.format(opp)) 58 | self._opportunities.append(opp) 59 | 60 | 61 | def pick(self, instrument=None): 62 | """ 63 | Find and return the best opportunity. If you want more than one, 64 | just call this repeatedly. 65 | 66 | TODO: use the instrument parameter to restrict which opp to pop. 67 | """ 68 | # Just pick the one with the highest confidence rating. 69 | if self._opportunities == []: 70 | return None 71 | max_conf_index = 0 72 | max_conf = 0 73 | for i in range(0, len(self._opportunities) - 1): 74 | if self._opportunities[i]['confidence'] > max_conf: 75 | max_conf_index = i 76 | return self._opportunities.pop(max_conf_index) 77 | 78 | 79 | -------------------------------------------------------------------------------- /src/order.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | #################### 4 | from log import Log 5 | #################### 6 | 7 | class Order(): 8 | """ 9 | Description: 10 | Class container for order info. 11 | It is generic for all brokers, so if more brokers are added in the 12 | future, this may need to be tweaked. 13 | """ 14 | 15 | def __init__( 16 | self, 17 | instrument=None, # 18 | order_type=None, # string - limit/stop/marketIfTouched/market 19 | price=None, # numeric, prince per unit for limit/stop/marketIfTouched 20 | lower_bound=None, # 21 | stop_loss=None, # JSON 22 | take_profit=None, # JSON 23 | trailing_stop=None, # JSON 24 | units=None, # numeric; use negative for short 25 | upper_bound=None, 26 | reason='' # optional string - for logging 27 | ): 28 | self.instrument = instrument 29 | self.lower_bound = lower_bound 30 | self.order_type = order_type 31 | self.price = price 32 | self.stop_loss = stop_loss 33 | self.take_profit = take_profit 34 | self.trailing_stop = trailing_stop 35 | self.units = units 36 | self.upper_bound = upper_bound 37 | self.reason = reason 38 | 39 | 40 | def __str__(self): 41 | return '\n\ 42 | instrument: {}\n\ 43 | lower_bound: {}\n\ 44 | order_type: {}\n\ 45 | price: {}\n\ 46 | stop_loss: {}\n\ 47 | take_profit: {}\n\ 48 | trailing_stop: {}\n\ 49 | units: {}\n\ 50 | upper_bound: {}\n\ 51 | reason: {}\n'.format( 52 | self.instrument.get_name(), 53 | self.lower_bound, 54 | self.order_type, 55 | self.price, 56 | self.stop_loss, 57 | self.take_profit, 58 | self.trailing_stop, 59 | self.units, 60 | self.upper_bound, 61 | self.reason 62 | ) 63 | 64 | -------------------------------------------------------------------------------- /src/requirements.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/paperduck/algo/f39f945dcfa0eb739588674308fdf2de66bfd69e/src/requirements.txt -------------------------------------------------------------------------------- /src/run_backtest.py: -------------------------------------------------------------------------------- 1 | """ 2 | Backtest entry point. Run this to perform a backtest. 3 | The backtest equivalent of daemon.py. 4 | $ cd src/ 5 | $ python3 run_backtest.py 6 | """ 7 | 8 | # external modules 9 | from datetime import datetime 10 | 11 | # internal modules 12 | from backtesting.backtest_broker import BacktestBroker as broker 13 | # strategy to test - import backtesting version, not live version 14 | from backtesting.strategies.demo import Demo 15 | from code_timer import CodeTimer as timer 16 | 17 | 18 | if __name__ == "__main__": 19 | timer_start = timer.start() 20 | strat = Demo() 21 | start = datetime(year=2003, month=1, day=1) 22 | end = datetime(year=2003, month=2, day=1) # TODO: make sure this is read as UTC - pandas took it as JST? 23 | #files = {4:'csv/USDJPY.txt', 5:'csv/USDCAD.txt'} 24 | files = {4:'csv/USDJPY.txt'} 25 | broker.init(start, end, files) 26 | while(broker.advance()): 27 | opps = strat.refresh() 28 | if len(opps) > 0: 29 | opp = opps[0] 30 | trade_id = broker.place_trade(opp.order, opp.order.units) # daemon normally decides units 31 | # notify strategy 32 | if trade_id: strat.trade_opened(trade_id) 33 | duration = timer.stop(timer_start) 34 | print('Backtest took {} seconds.'.format(duration)) 35 | 36 | -------------------------------------------------------------------------------- /src/scripts/daemon.sh: -------------------------------------------------------------------------------- 1 | #! /bin/sh 2 | 3 | ### BEGIN INIT INFO 4 | # Provides: daemon.sh 5 | # Required-Start: 6 | # Required-Stop: 7 | # X-Start-Before: 8 | # Default-Start: 2 3 4 5 9 | # Default-Stop: 0 1 6 10 | # Short-Description: algo daemon 11 | # Description: algo daemon 12 | ### END INIT INFO 13 | 14 | #************************** 15 | # DESCRIPTION: This bash script is intended to be run automatically when the 16 | # computer boots. It can be used to start the trading daemon automatically. 17 | #************************** 18 | 19 | SUBJECT='Have a nice day.' 20 | BODY=" 21 | $(who -r) 22 | 23 | Sincerely, 24 | $(whoami)" 25 | 26 | case "$1" in 27 | start) 28 | # send email 29 | python /home/user/raid/software_projects/algo/scripts/emailer.py "$SUBJECT" "$BODY" 30 | ;; 31 | esac 32 | 33 | 34 | exit 0 35 | -------------------------------------------------------------------------------- /src/scripts/daemon_prototype.sh: -------------------------------------------------------------------------------- 1 | #! /bin/bash 2 | 3 | printf "$(date)\n\n" > dout.txt 4 | 5 | #AUTH="$(head -n 1 /home/user/raid/documents/oanda_token.txt)" 6 | AUTH="$(head -n 1 /home/user/raid/documents/oanda_practice_token.txt)" 7 | #URL='https://api-fxtrade.oanda.com' 8 | URL='https://api-fxpractice.oanda.com' 9 | 10 | 11 | # Get account info 12 | ACCOUNTS=$(curl -H "Authorization: Bearer $AUTH" "$URL/v1/accounts") 13 | # verify OK 14 | if [ "$(printf "$ACCOUNTS" | jq '.code')" = "null" ] 15 | then 16 | printf "ACCOUNTS response OK\n" >> dout.txt 17 | else 18 | printf "Error (failed to get account info):\n$ACCOUNTS\n(end)\n" >> dout.txt 19 | exit 0 20 | fi 21 | 22 | ### debugging ### 23 | printf "$ACCOUNTS" 24 | cat dout.txt 25 | exit 0 26 | ################# 27 | 28 | # get primary account number 29 | ACCOUNT_ID_PRIMARY=$(\ 30 | printf "$ACCOUNTS" | jq \ 31 | '.accounts[] | if .accountName == "Primary" then .accountId else empty end') 32 | # verify OK 33 | if [ "$ACCOUNT_ID_PRIMARY" != "" ] 34 | then 35 | printf "ACCOUNT_ID_PRIMARY response OK\n" >> dout.txt 36 | else 37 | printf "Error (failed to get ID of primary account):\n($ACCOUNT_ID_PRIMARY)\n(end)\n" >> dout.txt 38 | exit 0 39 | fi 40 | 41 | # get primary account info 42 | PRIMARY_ACCOUNT=$(\ 43 | curl -s -H "Authorization: Bearer $AUTH" \ 44 | "$URL/v1/accounts/$ACCOUNT_ID_PRIMARY") 45 | # verify OK 46 | if [ "$(printf "$PRIMARY_ACCOUNT" | jq '.code')" = "null" ] 47 | then 48 | printf "PRIMARY_ACCOUNT response OK\n" >> dout.txt 49 | else 50 | printf "Error (failed to get info of primary account):\n$PRIMARY_ACCOUNT\n(end)\n" >> dout.txt 51 | exit 0 52 | fi 53 | 54 | # Get position info 55 | POSITIONS=$(\ 56 | curl -s -H "Authorization: Bearer $AUTH" \ 57 | "$URL/v1/accounts/$ACCOUNT_ID_PRIMARY/positions") 58 | # verify OK 59 | if [ "$(printf "$POSITIONS" | jq '.code')" = "null" ] 60 | then 61 | printf "POSITIONS response OK\n" >> dout.txt 62 | else 63 | printf "Error (failed to get position info):\n$POSITIONS\n(end)\n" >> dout.txt 64 | exit 0 65 | fi 66 | 67 | # Get number of positions 68 | NUM_OF_POSITIONS=$(\ 69 | printf "$POSITIONS" | jq '.positions | length') 70 | 71 | # Get balance 72 | BALANCE=$(\ 73 | printf "$PRIMARY_ACCOUNT" | jq \ 74 | '.balance') 75 | 76 | # get spread (USD JPY) 77 | PRICE_USDJPY=$(\ 78 | curl -s -H "Authorization: Bearer $AUTH" \ 79 | "$URL/v1/prices?instruments=USD_JPY") 80 | BID_USDJPY=$(printf "$PRICE_USDJPY" | jq '.prices[0].bid') 81 | ASK_USDJPY=$(printf "$PRICE_USDJPY" | jq '.prices[0].ask') 82 | SPREAD_USDJPY=$(echo "($ASK_USDJPY - $BID_USDJPY) * 100" | bc) 83 | 84 | #Dump 85 | printf "\n" >> dout.txt 86 | #printf "$(date) accounts:\n---------------\n$ACCOUNTS\n(end)\n" >> dout.txt 87 | #printf "$(date) primary account:\n---------------\n$PRIMARY_ACCOUNT\n(end)\n" >> dout.txt 88 | printf "$(date) primary account number: $ACCOUNT_ID_PRIMARY\n" >> dout.txt 89 | printf "$(date) balance: $BALANCE\n" >> dout.txt 90 | printf "$(date) num of pos's = $NUM_OF_POSITIONS\n" >> dout.txt 91 | printf "$(date) USD/JPY ask: $ASK_USDJPY\n" >> dout.txt 92 | printf "$(date) USD/JPY buy: $BID_USDJPY\n" >> dout.txt 93 | printf "$(date) USD/JPY spread: $SPREAD_USDJPY\n" >> dout.txt 94 | 95 | if [ $(echo "$SPREAD_USDJPY < 3" | bc) -eq "1" ] 96 | then 97 | printf "$(date) spread OK\n" >> dout.txt 98 | # Calculate Stop Loss 99 | # bc scale has to be set inline to allow for decimal quotient. 100 | BID_ASK_AVG=$(echo "scale=5; ($ASK_USDJPY + $BID_USDJPY) / 2" | bc) 101 | printf "$(date) bid ask average: $BID_ASK_AVG\n" >> dout.txt 102 | # stop loss price (change to minus for BUY) 103 | SL_PRICE=$(echo "scale=5; $BID_ASK_AVG + 0.1" | bc) 104 | # truncate to two decimal places 105 | SL_PRICE=$( echo "scale=2; $SL_PRICE / 1" | bc) 106 | printf "$(date) SL price = $SL_PRICE\n" >> dout.txt 107 | 108 | # take profit price (change to plus for BUY) 109 | TP_PRICE=$(echo "scale=5; $BID_ASK_AVG - 0.15" | bc) 110 | # truncate to two decimal places 111 | TP_PRICE=$( echo "scale=2; $TP_PRICE / 1" | bc) 112 | printf "$(date) TP price = $TP_PRICE\n" >> dout.txt 113 | 114 | printf "$(date) URL = $URL/v1/accounts/$ACCOUNT_ID_PRIMARY/orders?\ 115 | instrument=USD_JPY&units=33&side=sell&type=market&trailingStop=$SL_PRICE&takeProfit=$TP_PRICE\n" >> dout.txt 116 | cat dout.txt 117 | exit 0 118 | 119 | # Submit order 120 | RESPONSE=$(\ 121 | curl -s -X POST -H "Authorization: Bearer $AUTH" \ 122 | -d "instrument=USD_JPY&units=33&side=sell&type=market&trailingStop=$SL_PRICE&takeProfit=$TP_PRICE" \ 123 | "$URL/v1/accounts/$ACCOUNT_ID_PRIMARY/orders" 124 | ) 125 | printf "\n$(date) order response:\n$RESPONSE\n(end of response)\n" >> dout.txt 126 | 127 | else 128 | printf "spread too high\n" 129 | fi 130 | exit 0 131 | 132 | 133 | 134 | -------------------------------------------------------------------------------- /src/scripts/db_create_backup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | dt="$(date +%Y-%m-%d)" 4 | 5 | # use --no-data option to omit data. 6 | 7 | filename_algo="db_backup_algo_${dt}.mysql" 8 | printf -- "--\n-- ${dt}\n" > $filename_algo 9 | echo "\nPlease enter root database password:" 10 | mysqldump --comments -uroot -p algo >> $filename_algo 11 | 12 | filename_algo_private="db_backup_algo_private_${dt}.mysql" 13 | printf -- "--\n-- ${dt}\n" > $filename_algo_private 14 | echo "\nPlease enter root database password:" 15 | mysqldump --comments -uroot -p algo_private >> $filename_algo_private 16 | 17 | 18 | -------------------------------------------------------------------------------- /src/scripts/db_restore_backup.sh: -------------------------------------------------------------------------------- 1 | 2 | mysql -uroot -p -s -D xxx_dbname_xxx < xxx_backup_script_xxx.mysql 3 | 4 | -------------------------------------------------------------------------------- /src/scripts/emailer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import cgi 4 | from email.mime.image import MIMEImage 5 | from email.mime.multipart import MIMEMultipart 6 | from email.mime.text import MIMEText 7 | import mimetypes 8 | #apache_mimetypes = mimetypes.MimeTypes(['/etc/mime.types']) 9 | import multiprocessing 10 | import os 11 | import smtplib 12 | import sys 13 | import traceback 14 | 15 | if len(sys.argv) != 3: 16 | print ('usage: python emailer.py ') 17 | quit() 18 | with open('/home/user/raid/documents/algo_from_addr.txt', 'r') as f: 19 | from_addr = f.readline() 20 | from_addr = from_addr.rstrip() 21 | f.close() 22 | with open('/home/user/raid/documents/algo_from_pw.txt', 'r') as f: 23 | from_pw = f.readline() 24 | from_pw = from_pw.rstrip() 25 | f.close() 26 | with open('/home/user/raid/documents/algo_to_addr.txt', 'r') as f: 27 | to_addr = f.readline() 28 | to_addr = to_addr.rstrip() 29 | f.close() 30 | body_text = sys.argv[2] 31 | message = MIMEText(body_text) 32 | message["Subject"] = sys.argv[1] 33 | message["From"] = from_addr 34 | message["To"] = to_addr 35 | message["Return-Path"] = "localhost" 36 | message["Auto-Submitted"] = "auto-generated" 37 | #smtp = smtplib.SMTP("localhost") 38 | #smtp = smtplib.SMTP("localhost", 587) 39 | #smtp = smtplib.SMTP("localhost", 666) 40 | #smtp = smtplib.SMTP("localhost", 25) 41 | #smtp = smtplib.SMTP("localhost", 2525) 42 | #alternative ports from docomo support: 587, 465(SSL) 43 | #s = smtplib.SMTP("smtp.gmail.com", 587, localhostname) 44 | #returns a response code or exception 45 | #smtp = smtplib.SMTP("localhost") 46 | #smtp = smtplib.SMTP("localhost", 465) 47 | smtp = smtplib.SMTP_SSL("smtp.gmail.com", 465) 48 | try: 49 | print("logging in...") 50 | smtp.login(from_addr, from_pw) 51 | print("sending email") 52 | smtp.sendmail(from_addr, [to_addr], message.as_string()) #body_text) 53 | print("sent email") 54 | except Exception as em: 55 | print("ERROR(Exception): " + str(em) ) 56 | #except SMTPException, em: 57 | except: 58 | print("ERROR (other): " + str(em) ) 59 | print ("quitting") 60 | smtp.quit() 61 | 62 | -------------------------------------------------------------------------------- /src/scripts/get_missing.sh: -------------------------------------------------------------------------------- 1 | 2 | # download files that haven't been downloaded yet. 3 | # Match on files only (not size, contents, etc.) 4 | 5 | # read file that has list of urls 6 | for url in $(cat -s urls.txt) 7 | do 8 | # See if I have that file yet. 9 | # I might have already decompressed the file, so 10 | # ignore .gz and .tar extensions. 11 | get=1 12 | for file in $(ls -1) 13 | do 14 | base=$file 15 | # trim .gz 16 | if [ "${base: -3}" = ".gz" ] 17 | then 18 | base=${base::-3} 19 | fi 20 | # trim .tar 21 | if [ "${base: -4}" = ".tar" ] 22 | then 23 | base=${base::-4} 24 | fi 25 | # if (url - base filename) is different than (url), then I have the file 26 | if [ "${url/$base}" != $url ] 27 | then 28 | get=0 29 | #echo "don't get: " $url 30 | break 31 | fi 32 | done 33 | if [ $get -eq 1 ]; then 34 | wget -nc $url 35 | fi 36 | done 37 | 38 | 39 | -------------------------------------------------------------------------------- /src/scripts/load_csv/README.md: -------------------------------------------------------------------------------- 1 | 2 | Instructions: 3 | 4 | $ bash main.sh (args) 5 | -------------------------------------------------------------------------------- /src/scripts/load_csv/algo_fun_fix_date.mysql: -------------------------------------------------------------------------------- 1 | # Synopsis: 2 | # Convert date string from MM/DD/YYYY to YYYY-MM-DD 3 | # 4 | # Remarks: 5 | # 6 | 7 | delimiter // 8 | DROP FUNCTION IF EXISTS algo.algo_fun_fix_date; 9 | CREATE FUNCTION algo.algo_fun_fix_date 10 | ( 11 | old_date CHAR(10) 12 | ) 13 | RETURNS CHAR(10) 14 | DETERMINISTIC 15 | BEGIN 16 | IF old_date IS NULL THEN 17 | RETURN NULL; 18 | END IF; 19 | IF CHAR_LENGTH(old_date) < 10 THEN 20 | RETURN NULL; 21 | END IF; 22 | RETURN CONCAT( SUBSTRING(old_date, 7, 4), '-', SUBSTRING(old_date, 1, 2), '-', SUBSTRING(old_date, 4, 2) ); 23 | END // 24 | DELIMITER ; 25 | 26 | 27 | -------------------------------------------------------------------------------- /src/scripts/load_csv/algo_fun_fix_time.mysql: -------------------------------------------------------------------------------- 1 | # Synopsis: 2 | # Pi Trading times look like 3 | # hhmm 4 | # but MySQL needs them to look like 5 | # hh:mm:ss 6 | 7 | delimiter // 8 | DROP FUNCTION IF EXISTS algo.algo_fun_fix_time; 9 | CREATE FUNCTION algo.algo_fun_fix_time 10 | ( 11 | old_time CHAR(4) 12 | ) 13 | RETURNS CHAR(8) 14 | DETERMINISTIC 15 | BEGIN 16 | IF old_time IS NULL THEN 17 | RETURN NULL; 18 | END IF; 19 | IF CHAR_LENGTH(old_time) < 4 THEN 20 | RETURN NULL; 21 | END IF; 22 | RETURN CONCAT( SUBSTRING(old_time, 1, 2), ':', SUBSTRING(old_time, 3, 2), ':00' ); 23 | END // 24 | DELIMITER ; 25 | 26 | 27 | -------------------------------------------------------------------------------- /src/scripts/load_csv/algo_proc_create_table_stock.mysql: -------------------------------------------------------------------------------- 1 | # Synopsis: 2 | # Create a stored program that creates a table with a given name. 3 | # The table will be structured to store data from a Pi 4 | # Trading stock CSV file. 5 | # 6 | # Remarks: 7 | # It might be cleaner to pass the table name into a bash script 8 | # and execute the statement from bash. 9 | 10 | DELIMITER // 11 | DROP PROCEDURE IF EXISTS algo.algo_proc_create_table_stock; 12 | CREATE PROCEDURE algo.algo_proc_create_table_stock 13 | ( 14 | IN table_name VARCHAR(100) 15 | ) 16 | DETERMINISTIC 17 | BEGIN 18 | DECLARE statement VARCHAR(1000); 19 | 20 | # Prepare statement (can only prepare 1 at a time) 21 | SET @statement = 'CREATE TABLE IF NOT EXISTS '; 22 | SET @statement = CONCAT(@statement, table_name ); 23 | SET @statement = CONCAT(@statement, ' ( ' ); 24 | SET @statement = CONCAT(@statement, 'date DATE' ); 25 | SET @statement = CONCAT(@statement, ', time TIME' ); 26 | SET @statement = CONCAT(@statement, ', open DECIMAL(11,4)' ); 27 | SET @statement = CONCAT(@statement, ', high DECIMAL(11,4)' ); 28 | SET @statement = CONCAT(@statement, ', low DECIMAL(11,4)' ); 29 | SET @statement = CONCAT(@statement, ', close DECIMAL(11,4)' ); 30 | SET @statement = CONCAT(@statement, ', volume INT UNSIGNED' ); 31 | SET @statement = CONCAT(@statement, '); ' ); 32 | 33 | # Execute statement 34 | PREPARE stmt FROM @statement; 35 | EXECUTE stmt; 36 | DEALLOCATE PREPARE stmt; 37 | #SELECT @statement; 38 | END // 39 | DELIMITER ; 40 | 41 | -------------------------------------------------------------------------------- /src/scripts/load_csv/algo_proc_drop_table.mysql: -------------------------------------------------------------------------------- 1 | # Synopsis: 2 | # 3 | # Remarks: 4 | # It might be cleaner to pass the table name into a bash script 5 | # and execute the statement from bash. 6 | 7 | delimiter // 8 | DROP PROCEDURE IF EXISTS algo.algo_proc_drop_table; 9 | CREATE PROCEDURE algo.algo_proc_drop_table 10 | ( 11 | IN table_name VARCHAR(100) 12 | ) 13 | DETERMINISTIC 14 | BEGIN 15 | DECLARE statement VARCHAR(1000); 16 | 17 | # Prepare statement. 18 | SET @statement = 'DROP TABLE IF EXISTS '; 19 | SET @statement = CONCAT(@statement, table_name ); 20 | SET @statement = CONCAT(@statement, '' ); 21 | 22 | # Execute statement. 23 | PREPARE stmt FROM @statement; 24 | EXECUTE stmt; 25 | DEALLOCATE PREPARE stmt; 26 | END // 27 | DELIMITER ; 28 | 29 | 30 | -------------------------------------------------------------------------------- /src/scripts/load_csv/algo_proc_load_csv.mysql: -------------------------------------------------------------------------------- 1 | # DO NOT USE: LOAD DATA INFILE NOT SUPPORTED IN PREPARED STATEMENTS YET 2 | # 3 | # Synopsis: 4 | # Create stored program that: 5 | # Makes a table in database with given name (if DNE). 6 | # Reads csv file containing historical data into this table. 7 | # 8 | # Arguments: 9 | # - filename w/full path 10 | # - table name 11 | # 12 | # Remarks: 13 | # Formatted for data from http://pitrading.com/ 14 | # 15 | 16 | DELIMITER ! 17 | DROP PROCEDURE IF EXISTS algo.algo_proc_load_csv; 18 | CREATE PROCEDURE algo.algo_proc_load_csv 19 | ( 20 | IN f VARCHAR(1000), # CSV filename w/full path 21 | IN t VARCHAR(100) # table name 22 | ) 23 | DETERMINISTIC 24 | BEGIN 25 | SET @statement = CONCAT('LOAD DATA INFILE \'', f, '\''); 26 | SET @statement = CONCAT(@statement, ' INTO TABLE ', t); 27 | SET @statement = CONCAT(@statement, ' FIELDS TERMINATED BY \',\' ENCLOSED BY \'\' '); 28 | SET @statement = CONCAT(@statement, ' LINES TERMINATED BY \'\\r\\n\' '); 29 | SET @statement = CONCAT(@statement, ' IGNORE 1 LINES '); 30 | SET @statement = CONCAT(@statement, '( date, time, open, high, low, close, volume )'); 31 | SET @statement = CONCAT(@statement, ';'); 32 | 33 | PREPARE stmt FROM @statement; 34 | EXECUTE stmt; 35 | DEALLOCATE PREPARE stmt; 36 | 37 | END; 38 | ! 39 | -------------------------------------------------------------------------------- /src/scripts/load_csv/load_csv.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/bash 2 | 3 | # Verify arguments 4 | if [[ "$#" -lt 4 ]]; then 5 | printf "%s\n" "too few arg(s) to $0" \ 6 | "Usage: bash $0 " 7 | exit 1 8 | fi 9 | 10 | u=$1 11 | pw=$2 12 | f=$3 13 | tn=$4 14 | 15 | # The TIMESTAMP type is stored as UTC by default - server timezone affects value 16 | mysql -u$u -p$pw -e "LOAD DATA INFILE '${f}' \ 17 | INTO TABLE ${tn} \ 18 | FIELDS TERMINATED BY ',' \ 19 | ENCLOSED BY '' \ 20 | LINES TERMINATED BY '\r\n' \ 21 | IGNORE 1 LINES \ 22 | ( @old_date, @old_time, open, high, low, close, volume ) \ 23 | SET date = (SELECT algo_fun_fix_date(@old_date)), \ 24 | time = (SELECT algo_fun_fix_time(@old_time)) 25 | ;" algo 26 | 27 | -------------------------------------------------------------------------------- /src/scripts/load_csv/main.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/bash 2 | 3 | ########################################################### 4 | # INPUT PARAMETERS 5 | # - Database username 6 | # - Database password 7 | # - Full path of directory that contains the CSV files, 8 | # including final slash. 9 | # - Table name prefix. 10 | # REMARKS 11 | # Note that the CSV files from pi trading have the a 12 | # '.txt' extension. 13 | # EXAMPLES 14 | # 15 | ########################################################### 16 | # Various ways to pass arguments to proc from command line: 17 | # 18 | # $ mysql --user= --execute="call ()" 19 | # 20 | # $ mysql -u --password= <(); 22 | # !! 23 | 24 | load_all(){ 25 | for f in $( ls ${dir} ); 26 | do 27 | # Do for each file with '.txt' extension 28 | if [[ $f =~ \.txt$ ]] ; then 29 | # derive table name from filename 30 | # Extract everything from file name up to extension 31 | # transform uppercase to lowercase 32 | tn=$( echo ${f%.txt} | tr [:upper:] [:lower:] ) # case sensitive 33 | tn=$( echo ${tn} | sed s/-/_/ ) # no dashes in mysql schema object names 34 | tn=${tn_pre}_${tn} 35 | full_file_path="${dir}${f}" 36 | 37 | echo ---------------------------------------------- 38 | echo loading file 39 | echo " $full_file_path" 40 | echo into table 41 | echo " $tn" 42 | #echo Dropping table... 43 | #mysql -u$db_user -p$db_pass -e "CALL algo_proc_drop_table('${tn}');" algo 44 | echo Creating table... 45 | mysql -u$db_user -p$db_pass -e "CALL algo_proc_create_table_stock('${tn}');" algo 46 | echo Loading CSV file... 47 | bash load_csv.sh $db_user $db_pass "$full_file_path" $tn 48 | echo 49 | fi 50 | done 51 | echo done 52 | echo 53 | } 54 | 55 | load_one(){ 56 | echo !!! Not implemented !!! 57 | } 58 | 59 | # Verify arguments 60 | if [[ "$#" -lt 4 ]]; then 61 | printf "%s\n" "missing arg(s)" \ 62 | "Usage: bash main.sh " 63 | exit 1 64 | fi 65 | 66 | db_user=$1 67 | db_pass=$2 68 | dir=$3 69 | tn_pre=$4 70 | #optional filename argument 71 | if [[ -n $5 ]]; then 72 | filename=$5 73 | fi 74 | 75 | pwd=$(pwd) 76 | tn='' 77 | full_file_path='' 78 | cmd='' 79 | echo table name prefix: $tn_pre 80 | echo directory: $dir 81 | echo 82 | 83 | # Prepare stored programs 84 | echo Creating algo_proc_drop_table ... 85 | mysql -u$1 -p$2 algo < algo_proc_drop_table.mysql 86 | echo Creating proc algo_proc_create_table_stock ... 87 | mysql -u$1 -p$2 algo < algo_proc_create_table_stock.mysql 88 | echo Creating proc algo_proc_load_csv ... 89 | mysql -u$1 -p$2 algo < algo_proc_load_csv.mysql 90 | echo Creating function algo_fun_fix_date ... 91 | mysql -u$1 -p$2 algo < algo_fun_fix_date.mysql 92 | echo Creating function algo_fun_fix_time ... 93 | mysql -u$1 -p$2 algo < algo_fun_fix_time.mysql 94 | 95 | if [[ -n $filename ]]; then 96 | load_one 97 | else 98 | load_all 99 | fi 100 | 101 | 102 | -------------------------------------------------------------------------------- /src/scripts/load_csv/test/create_function_test.mysql: -------------------------------------------------------------------------------- 1 | DELIMITER ! 2 | CREATE FUNCTION algo.test() RETURNS INTEGER 3 | BEGIN 4 | RETURN 666; 5 | END; 6 | ! 7 | 8 | -------------------------------------------------------------------------------- /src/scripts/mount.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | mount /dev/sdb1 /home/user/raid/ 4 | service mysql start 5 | 6 | 7 | -------------------------------------------------------------------------------- /src/scripts/packet_analysis/watch_traffic.sh: -------------------------------------------------------------------------------- 1 | # network packet monitor to verify data to/from broker. 2 | # SIGINT (Ctl-C) to exit. 3 | 4 | 5 | # -A print packet in ASCII 6 | # -e print link-layer header 7 | # -i listen on 8 | # -l line buffered for easier reading of live stream 9 | # -# print packet number at beginning of line 10 | # -q quiet 11 | # -Q direction: 'in', 'out', or 'inout' 12 | # -t no timestamp 13 | # -v verbose 14 | # -vv 15 | # -vvv 16 | # -w write to 17 | # 0 56 | 57 | if go_long: # currently long 58 | cur_bid = Broker.get_bid(instrument) 59 | if cur_bid != None: 60 | if cur_bid - sl > self.sl_price_diff: 61 | new_sl = cur_bid - self.sl_price_diff 62 | resp = Broker.modify_trade( 63 | trade_id=open_trade_id, 64 | stop_loss_price=str(round(new_sl, 2)) 65 | ) 66 | if resp == None: 67 | Log.write('"fifty.py" _babysit(): Modify failed. Checking if trade is closed.') 68 | closed = Broker.is_trade_closed(open_trade_id) 69 | Log.write('fifty.py babysit(): is_trade_closed returned:\n{}'.format(closed)) 70 | if closed[0]: 71 | Log.write('"fifty.py" _babysit(): BUY trade has closed. (BUY)') 72 | self.open_trade_ids.remove(open_trade_id) 73 | # If SL hit, reverse direction. 74 | if closed[1] == TradeClosedReason.STOP_LOSS_ORDER: 75 | self.go_long = False 76 | else: 77 | Log.write('"fifty.py" _babysit(): Failed to modify BUY trade.') 78 | raise Exception 79 | else: 80 | Log.write('"fifty.py" _babysit(): Modified BUY trade with ID (',\ 81 | open_trade_id, ').') 82 | else: 83 | Log.write('"fifty.py" _babysit(): Failed to get bid while babysitting.') 84 | raise Exception 85 | else: # currently short 86 | cur_bid = Broker.get_bid(instrument) 87 | if cur_bid != None: 88 | if sl - cur_bid > self.sl_price_diff: 89 | new_sl = cur_bid + self.sl_price_diff 90 | resp = Broker.modify_trade( 91 | trade_id=open_trade_id, 92 | stop_loss_price=str(round(new_sl, 2)) 93 | ) 94 | if resp == None: 95 | closed = Broker.is_trade_closed(open_trade_id) 96 | Log.write('fifty.py babysit(): is_trade_closed returned:\n{}'.format(closed)) 97 | if closed[0]: 98 | Log.write('"fifty.py" _babysit(): SELL trade has closed. (BUY)') 99 | self.open_trade_ids.remove(open_trade_id) 100 | # If SL hit, reverse direction. 101 | if closed[1] == TradeClosedReason.STOP_LOSS_ORDER: 102 | self.go_long = True 103 | else: 104 | Log.write('"fifty.py" in _babysit(): Failed to modify SELL trade.') 105 | raise Exception 106 | else: 107 | Log.write('"fifty.py" _babysit(): Modified SELL trade with ID (',\ 108 | open_trade_id, ').') 109 | else: 110 | Log.write('"fifty.py" _babysit(): Failed to get ask while babysitting.') 111 | raise Exception 112 | 113 | 114 | def _scan(self): 115 | Log.write('fifty.py scan()') 116 | """ (see strategy.py for documentation) """ 117 | # If we're babysitting a trade, don't open a new one. 118 | if len(self.open_trade_ids) > 0: 119 | Log.write('fifty.py _scan(): Trades open; no suggestions.') 120 | return None 121 | instrument = Instrument(Instrument.get_id_from_name('USD_JPY')) 122 | spreads = Broker.get_spreads([instrument]) 123 | if spreads == None: 124 | Log.write('"fifty.py" in _scan(): Failed to get spread of {}.' 125 | .format(instrument.get_name())) 126 | raise Exception 127 | elif len(spreads) < 1: 128 | Log.write('"fifty.py" in _scan(): len(spreads) == {}.' 129 | .format(len(spreads))) 130 | raise Exception 131 | # This only checks for one instrument. 132 | elif not spreads[0]['tradeable']: 133 | Log.write('"fifty.py" in _scan(): Instrument {} not tradeable.' 134 | .format(instrument.get_name())) 135 | return None 136 | else: 137 | spread = spreads[0]['spread'] 138 | if spread < 3: 139 | Log.write('fifty.py _scan(): spread = {}'.format(spread)) 140 | if self.go_long: # buy 141 | Log.write('"fifty.py" _scan(): Going long.') 142 | cur_bid = Broker.get_bid(instrument) 143 | if cur_bid != None: 144 | # Rounding the raw bid didn't prevent float inaccuracy 145 | # cur_bid = round(cur_bid_raw, 2) 146 | tp = round(cur_bid + self.tp_price_diff, 2) 147 | sl = round(cur_bid - self.sl_price_diff, 2) 148 | else: 149 | Log.write('"fifty.py" in _scan(): Failed to get bid.') 150 | raise Exception 151 | else: # sell 152 | Log.write('"fifty.py" _scan(): Shorting.') 153 | self.go_long = False 154 | cur_bid = Broker.get_bid(instrument) 155 | if cur_bid != None: 156 | tp = round(cur_bid - self.tp_price_diff, 2) 157 | sl = round(cur_bid + self.sl_price_diff, 2) 158 | else: 159 | Log.write('"fifty.py" in _scan(): Failed to get ask.') 160 | raise Exception 161 | # Prepare the order and sent it back to daemon. 162 | units = 1 if self.go_long else -1 163 | confidence = 50 164 | order = Order( 165 | instrument=instrument, 166 | order_type="MARKET", # matches Oanda's OrderType definition 167 | stop_loss={ "price" : str(sl) }, 168 | take_profit={ "price" : str(tp) }, 169 | units=units 170 | ) 171 | reason = 'happy day' 172 | opp = Opportunity(order, confidence, self, reason) 173 | Log.write('"fifty.py" _scan(): Returning opportunity with \ 174 | order:\n{}'.format(opp)) 175 | return opp 176 | else: 177 | Log.write('fifty.py _scan(): Spread is high; no suggestions.') 178 | return None 179 | 180 | 181 | 182 | 183 | -------------------------------------------------------------------------------- /src/strategy.py: -------------------------------------------------------------------------------- 1 | """ 2 | Python ver. 3.4 3 | File: strategy.py 4 | Description: 5 | Base class for strategies defined here. 6 | This should NOT be instantiated directly. 7 | Derive it and make your own strategy. 8 | """ 9 | 10 | #************************* 11 | import sys 12 | #************************* 13 | from config import Config 14 | from db import DB 15 | from log import Log 16 | from trade import * 17 | #************************* 18 | 19 | class Strategy(): 20 | """ 21 | Class methods are used because the daemon never needs more than one 22 | instance of each strategy. 23 | """ 24 | 25 | # 26 | open_trade_ids = [] 27 | 28 | 29 | def __str__(self): 30 | return self.get_name() 31 | 32 | 33 | def get_number_positions(self): 34 | return len( self.open_trade_ids ) 35 | 36 | 37 | def get_name(self): 38 | """ 39 | Needs to be overloaded. 40 | This is a method only instead of a string member in order to enforce 41 | implementation by derived classes. 42 | Returns: Name of strategy as a unique string. 43 | """ 44 | raise NotImplementedError() 45 | 46 | 47 | def trade_opened(self, trade_id): 48 | """ Called by the daemon to notify the strategy that the order it suggested has been placed. 49 | Return type: void 50 | Parameters: 51 | trade_id string 52 | """ 53 | self.open_trade_ids.append(trade_id) 54 | Log.write('"strategy.py" trade_opened(): strat {} opened trade ({})' 55 | .format(self.get_name(), trade_id)) 56 | # Write to db - mainly for backup in event of power failure 57 | DB.execute('INSERT INTO open_trades_live (trade_id, strategy, \ 58 | broker) values ("{}", "{}", "{}")' 59 | .format(trade_id, self.get_name(), Config.broker_name)) 60 | 61 | 62 | def adopt(self, trade_id): 63 | """ 64 | When the daemon is initializing, particularly after being unexpectedly 65 | terminated, this can be used to tell the strategy 66 | module about a trade that it had previously opened. 67 | """ 68 | self.open_trade_ids.append(trade_id) 69 | 70 | 71 | def cleanup(self): 72 | """ release memory """ 73 | del self.open_trade_ids[:] 74 | 75 | 76 | def refresh(self): 77 | """The daemon should call this repeatedly.""" 78 | self._babysit() 79 | return self._scan() 80 | 81 | 82 | def _babysit(self): 83 | """ 84 | Return type: void 85 | Babysit open trades. 86 | Override this in your strategy module. 87 | """ 88 | raise NotImplementedError() 89 | 90 | 91 | def _scan(self): 92 | """ 93 | Determines whether there is an opportunity or not. 94 | Override this in your strategy module. 95 | Return type: 96 | instance if there is an opportunity. 97 | None if no opportunity. 98 | """ 99 | raise NotImplementedError() 100 | 101 | 102 | -------------------------------------------------------------------------------- /src/tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/paperduck/algo/f39f945dcfa0eb739588674308fdf2de66bfd69e/src/tests/__init__.py -------------------------------------------------------------------------------- /src/tests/run_tests.sh: -------------------------------------------------------------------------------- 1 | # Run from /src/ like this: 2 | # $ bash tests/run_tests.sh 3 | # Do not run it from within /tests/. 4 | # This is so the config_nonsecure.cfg path in config.py can be imported 5 | # by the tests. 6 | 7 | export PYTHONPATH=/home/user/raid/software_projects/algo/src 8 | export PYTHONPATH=$PYTHONPATH:/home/user/raid/software_projects/algo/src/tests 9 | 10 | python3 -m unittest test_oanda 11 | python3 -m unittest test_daemon 12 | python3 -m unittest test_instrument 13 | python3 -m unittest test_chart 14 | python3 -m unittest test_util_date 15 | python3 -m unittest test_strategy 16 | 17 | 18 | -------------------------------------------------------------------------------- /src/tests/test_chart.py: -------------------------------------------------------------------------------- 1 | """ 2 | Test the chart.py module. 3 | """ 4 | 5 | # library imports 6 | from datetime import datetime, timedelta 7 | import random 8 | import sys 9 | import unittest 10 | from unittest.mock import MagicMock, patch, call 11 | 12 | # Need to modify sys.path because Python doesn't have relative imports. 13 | # sys.path is initialized from environment variable PYTHONPATH. 14 | try: 15 | sys.path.index('/home/user/raid/software_projects/algo/src') 16 | except ValueError: 17 | sys.path.append('/home/user/raid/software_projects/algo/src') 18 | 19 | # local imports 20 | from broker import Broker 21 | from candle import Candle 22 | from chart import Chart 23 | #from config import Config 24 | from db import DB 25 | from instrument import Instrument 26 | import util_date 27 | 28 | class TestChart(unittest.TestCase): 29 | 30 | 31 | # called for every test method 32 | def setUp(self): 33 | pass 34 | 35 | 36 | # called for every test method 37 | def tearDown(self): 38 | pass 39 | 40 | 41 | """ 42 | """ 43 | def test___init__(self): 44 | """ 45 | Case: count, no start, no end 46 | """ 47 | COUNT = 100 48 | GRANULARITY = 'S5' 49 | NUM_EXTRA_SECONDS = 300 50 | sample_instrument = Instrument(4) # USD_JPY 51 | # Initialize a sample chart. 52 | chart = Chart( 53 | in_instrument=sample_instrument, 54 | count=COUNT 55 | ) 56 | # check success 57 | self.assertNotEqual(chart._granularity, None) 58 | # check indecies 59 | self.assertTrue( 60 | chart._start_index == 0 and chart._get_end_index() == COUNT - 1 61 | ) 62 | # check instrument 63 | self.assertEqual(sample_instrument.get_id(), chart._instrument.get_id()) 64 | self.assertEqual(sample_instrument.get_name(), chart._instrument.get_name()) 65 | # check granularity 66 | self.assertEqual(chart._granularity, GRANULARITY) # Oanda's default (S5) 67 | # check count 68 | self.assertEqual(chart.get_size(), COUNT) 69 | 70 | # check candle format 71 | # If 'bidask', then the midpoints will be None, and vice-versa 72 | self.assertNotEqual(chart[0].open_bid, None) # Oanda's default 73 | 74 | start = None 75 | chart = None 76 | 77 | 78 | """ 79 | Case: count, start, no end 80 | """ 81 | COUNT = 100 82 | GRANULARITY = 'M1' 83 | WIGGLE_MINUTES = 5 # no price change -> no candle 84 | ADJUSTMENT = 120 85 | sample_instrument = Instrument(4) # USD_JPY 86 | 87 | # Initialize a sample chart with no market close gaps. 88 | # start = now - time since close - chart size 89 | # - (adjustment to busy time to avoid gaps) - skipped candle slack 90 | start = datetime.utcnow() \ 91 | - Broker.get_time_since_close() \ 92 | - timedelta(minutes = COUNT + ADJUSTMENT + WIGGLE_MINUTES) 93 | chart = Chart( 94 | in_instrument=sample_instrument, 95 | count=COUNT, 96 | start=start, 97 | granularity=GRANULARITY 98 | ) 99 | 100 | # check success 101 | self.assertNotEqual(chart._granularity, None) 102 | # check indecies 103 | self.assertTrue( 104 | chart._start_index == 0 and chart._get_end_index() == COUNT - 1 105 | ) 106 | # check instrument 107 | self.assertEqual(sample_instrument.get_id(), chart._instrument.get_id()) 108 | self.assertEqual(sample_instrument.get_name(), chart._instrument.get_name()) 109 | # check granularity 110 | self.assertEqual(chart._granularity, GRANULARITY) 111 | # check count 112 | self.assertEqual(chart.get_size(), COUNT) 113 | # check start time 114 | self.assertTrue( 115 | # Candles gap if there were no ticks, so allow some wiggle room. 116 | abs(start - chart.get_start_timestamp()) < timedelta(minutes=WIGGLE_MINUTES) 117 | ) 118 | # check end time 119 | end_expected = start + timedelta(minutes=COUNT) 120 | end_real = chart.get_end_timestamp() 121 | self.assertTrue( 122 | # Candles gap if there were no ticks, so allow some wiggle room. 123 | abs(end_expected - end_real) < timedelta(minutes=WIGGLE_MINUTES) 124 | ) 125 | # check candle format 126 | self.assertNotEqual(chart[0].open_bid, None) 127 | 128 | 129 | """ 130 | count, no start, end 131 | """ 132 | COUNT = 100 133 | GRANULARITY = 'H2' 134 | 135 | sample_instrument = Instrument(4) # USD_JPY 136 | 137 | # Initialize a sample chart. 138 | chart = Chart( 139 | in_instrument=sample_instrument, 140 | count=COUNT, 141 | end=datetime.utcnow(), 142 | granularity=GRANULARITY 143 | ) 144 | 145 | # check success 146 | self.assertNotEqual(chart._granularity, None) 147 | # check indecies 148 | self.assertTrue( 149 | chart._start_index == 0 and chart._get_end_index() == COUNT - 1 150 | ) 151 | # check instrument 152 | self.assertEqual(sample_instrument.get_id(), chart._instrument.get_id()) 153 | self.assertEqual(sample_instrument.get_name(), chart._instrument.get_name()) 154 | # check granularity 155 | self.assertEqual(chart._granularity, GRANULARITY) 156 | # check count 157 | self.assertEqual(chart.get_size(), COUNT) 158 | 159 | # check start time 160 | """self.assertTrue( 161 | # Candles gap if there were no ticks, so allow some wiggle room. 162 | abs(start - chart.get_start_timestamp()) < timedelta(minutes=5) 163 | )""" 164 | # check end time 165 | end_expected = datetime.utcnow() 166 | end_real = chart.get_end_timestamp() 167 | if Broker.get_time_until_close() == timedelta(): 168 | self.assertTrue( 169 | abs(end_expected - end_real) < timedelta(days=3) 170 | ) 171 | else: 172 | self.assertTrue( 173 | # Candles gap if there were no ticks, so allow some wiggle room. 174 | abs(end_expected - end_real) < timedelta(hours=5) 175 | ) 176 | # check candle format 177 | # If 'bidask', then the midpoints will be None, and vice-versa 178 | self.assertNotEqual(chart[0].open_bid, None) # Oanda's default 179 | 180 | 181 | """ 182 | no count, start, no end 183 | """ 184 | COUNT = 24 185 | GRANULARITY = 'M' # month (Oanda) 186 | sample_instrument = Instrument(4) # USD_JPY 187 | 188 | # Initialize a sample chart. 189 | # start = now - 2 years 190 | start = datetime.utcnow() - timedelta(days=365*2) 191 | chart = Chart( 192 | in_instrument=sample_instrument, 193 | start=start, 194 | granularity=GRANULARITY 195 | ) 196 | 197 | # check success 198 | self.assertNotEqual(chart._granularity, None) 199 | # check indecies 200 | self.assertTrue( 201 | chart._start_index == 0 and abs(chart._get_end_index() - COUNT) <= 1 202 | ) 203 | # check instrument 204 | self.assertEqual(sample_instrument.get_id(), chart._instrument.get_id()) 205 | self.assertEqual(sample_instrument.get_name(), chart._instrument.get_name()) 206 | # check granularity 207 | self.assertEqual(chart._granularity, GRANULARITY) 208 | # check count 209 | self.assertTrue( abs(chart.get_size() - COUNT) <= 1 ) 210 | 211 | # check start time 212 | self.assertTrue( 213 | # allow wiggle room. 214 | abs(start - chart.get_start_timestamp()) < timedelta(days=32) 215 | ) 216 | # check end time 217 | end_expected = datetime.utcnow() 218 | end_real = chart.get_end_timestamp() 219 | self.assertTrue( 220 | # Allow wiggle room for market close. 221 | abs(end_expected - end_real) < timedelta(days=32) 222 | ) 223 | # check candle format 224 | # If 'bidask', then the midpoints will be None, and vice-versa 225 | self.assertNotEqual(chart[0].open_bid, None) # Oanda's default 226 | 227 | 228 | def test_update(self): 229 | """ 230 | test: Chart.update() 231 | Constraints to verify: 232 | - Data is as recent as possible 233 | - start index has earliest timestamp 234 | - end index has latest timestamp 235 | - timestamps from start to end are sequential 236 | Cases: 237 | - old chart (complete update) 238 | - somewhat outdated chart (partially updated) 239 | - new chart (no updates other than last (incomplete) candle) 240 | """ 241 | 242 | """ 243 | case: old chart that gets completely updated 244 | """ 245 | # initial "outdated" chart 246 | chart = Chart( 247 | in_instrument=Instrument(4), 248 | granularity='M1', 249 | count=4999, 250 | end=datetime(year=2017, month=12, day=5) 251 | ) 252 | # Update chart 253 | chart.update() 254 | 255 | # Verify data is most recent 256 | time_since_close = Broker.get_time_since_close() 257 | now = datetime.utcnow() 258 | end_timestamp = chart.get_end_timestamp() 259 | if (Broker.get_time_until_close() == timedelta()): 260 | # Time since last candle should be close to time since market 261 | # close. The leniency is high to allow for periods of no new 262 | # candles. 263 | self.assertTrue( 264 | abs((now - end_timestamp) - (time_since_close)) 265 | < timedelta(minutes=62) 266 | ) 267 | else: 268 | # Time since last candle should be close to now. 269 | self.assertTrue(abs(now - end_timestamp) < timedelta(minutes=2)) 270 | # verify candle at start index has earliest timestamp. 271 | earliest_timestamp = datetime.utcnow() 272 | for i in range(0, chart.get_size()): 273 | if chart[i].timestamp < earliest_timestamp: 274 | earliest_timestamp = chart[i].timestamp 275 | self.assertTrue(chart.get_start_timestamp() == earliest_timestamp) 276 | # verify candle at end index has latest timestamp. 277 | latest_timestamp = datetime(year=1999, month=1, day=1) 278 | for i in range(0, chart.get_size()): 279 | if chart[i].timestamp > latest_timestamp: 280 | latest_timestamp = chart[i].timestamp 281 | self.assertTrue(chart.get_end_timestamp() == latest_timestamp) 282 | # Verify sequential timestamps 283 | for i in range(0, chart.get_size() - 1): 284 | self.assertTrue(chart[i].timestamp < chart[i + 1].timestamp) 285 | """ 286 | Chart that gets partially updated 287 | """ 288 | # TODO 289 | """ 290 | Chart that gets barely updated 291 | """ 292 | # TODO 293 | 294 | 295 | def test_pearson(self): 296 | """ 297 | case: empty chart 298 | """ 299 | c = Chart( Instrument(4) ) 300 | self.assertRaises(Exception, c.pearson, 0, c.get_size() ) 301 | 302 | """ 303 | case: one candle 304 | """ 305 | # TODO 306 | 307 | """ 308 | case: straight line, positive slope 309 | """ 310 | c = Chart(in_instrument=Instrument(4), count=0) 311 | fake_candles = [] 312 | fake_timestamp = datetime.utcnow() 313 | fake_price = 100.1234 314 | for i in range(0,10): 315 | fake_candles.append( Candle( 316 | timestamp=fake_timestamp + timedelta(seconds=i), 317 | high_ask=fake_price + i, 318 | low_bid=(fake_price + i / 2) 319 | ) ) 320 | c._candles = fake_candles 321 | pearson = c.pearson( 0, c.get_size() - 1, 'high_low_avg' ) 322 | # Allow some play for float arithmetic 323 | self.assertTrue( pearson > 0.99999 and pearson <= 1 ) 324 | 325 | """ 326 | case: straight line, negative slope 327 | """ 328 | c = Chart( in_instrument=Instrument(4), count=0 ) 329 | fake_candles = [] 330 | fake_timestamp = datetime.utcnow() 331 | fake_price = 100.1234 332 | for i in range(0,10): 333 | fake_candles.append( Candle( 334 | timestamp=fake_timestamp - timedelta(seconds=i), 335 | high_ask=fake_price + i, 336 | low_bid=(fake_price + i / 2) 337 | ) ) 338 | c._candles = fake_candles 339 | pearson = c.pearson( 0, c.get_size() - 1, 'high_low_avg' ) 340 | # Allow some play for float arithmetic 341 | self.assertTrue( pearson < -0.99999 and pearson >= -1 ) 342 | 343 | """ 344 | case: V shape 345 | """ 346 | c = Chart( in_instrument=Instrument(4), count=0 ) 347 | fake_candles = [] 348 | fake_timestamp = datetime.utcnow() 349 | fake_price = 100.1234 350 | seconds_elapsed = 0 351 | for i in range(9,-1,-1): 352 | fake_candles.append( Candle( 353 | timestamp=fake_timestamp + timedelta(seconds=seconds_elapsed), 354 | high_ask=fake_price + i, 355 | low_bid=(fake_price + i / 2) 356 | ) ) 357 | seconds_elapsed += 1 358 | for i in range(0,10): 359 | fake_candles.append( Candle( 360 | timestamp=fake_timestamp + timedelta(seconds=seconds_elapsed), 361 | high_ask=fake_price + i, 362 | low_bid=(fake_price + i / 2) 363 | ) ) 364 | seconds_elapsed += 1 365 | c._candles = fake_candles 366 | pearson = c.pearson( 0, c.get_size() - 1, 'high_low_avg' ) 367 | # Allow some play for float arithmetic 368 | self.assertTrue( pearson > -0.000001 and pearson < 0.000001 ) 369 | 370 | """ 371 | case: random 372 | """ 373 | c = Chart( in_instrument=Instrument(4), count=0 ) 374 | fake_candles = [] 375 | fake_timestamp = datetime.utcnow() 376 | fake_price = 100 377 | seconds_elapsed = 0 378 | for i in range(0,10000): 379 | offset = random.randrange(1, fake_price) 380 | fake_candles.append( Candle( 381 | timestamp=fake_timestamp + timedelta(seconds=seconds_elapsed), 382 | high_ask=fake_price + offset, 383 | low_bid=(fake_price + offset / 2) 384 | ) ) 385 | seconds_elapsed += 1 386 | c._candles = fake_candles 387 | pearson = c.pearson( 0, c.get_size() - 1, 'high_low_avg' ) 388 | # Allow some play for float rounding 389 | print('pearson of random chart: {}'.format(pearson) ) 390 | self.assertTrue( pearson > -0.02 and pearson < 0.02 ) 391 | 392 | 393 | if __name__ == '__main__': 394 | unittest.main() 395 | -------------------------------------------------------------------------------- /src/tests/test_daemon.py: -------------------------------------------------------------------------------- 1 | """ 2 | Run like so: 3 | $ python3 tests.py 4 | 5 | The daemon is assumed to only use the Fifty strategy module. 6 | """ 7 | 8 | # library imports 9 | import sys 10 | import unittest 11 | from unittest.mock import MagicMock, patch, call 12 | 13 | # Need to modify sys.path because Python doesn't have relative imports. 14 | # sys.path is initialized from environment variable PYTHONPATH. 15 | try: 16 | sys.path.index('/home/user/raid/software_projects/algo/src') 17 | except ValueError: 18 | sys.path.append('/home/user/raid/software_projects/algo/src') 19 | 20 | # Have to mock db.DB before daemon.py (etc.) imports it. 21 | #real_db = DB 22 | #real_db_execute = DB.execute 23 | sys.modules['db'] = MagicMock() 24 | from db import DB 25 | 26 | # project imports 27 | from daemon import Daemon 28 | from instrument import Instrument 29 | from strategies.fifty import Fifty 30 | from trade import TradeClosedReason, Trade, Trades 31 | 32 | class TestDaemon(unittest.TestCase): 33 | 34 | _strat = Fifty(0.1, 0.1) 35 | 36 | # called for every test method 37 | def setUp(self): 38 | pass 39 | 40 | 41 | # called for every test method 42 | def tearDown(self): 43 | pass 44 | 45 | @patch('daemon.Broker') 46 | def test_recover_trades(self, mock_broker): 47 | """ 48 | Test: Daemon.recover_trades() 49 | Scenarios: 50 | - No trades in broker or db 51 | Assertions: 52 | - Strategies recover no trades 53 | - Trade in broker and db 54 | Assertions: 55 | - Trade gets distributed to correct strategy 56 | - Trade is in broker, not db 57 | Assert: 58 | - Trade deleted from db 59 | - Trade gets distributed to correct strategy 60 | - Trade in db, broker unsure 61 | Assert: 62 | - Trade deleted from db 63 | - Trade's strategy does NOT adopt it 64 | - Trade in db has wonky data (non-existant strategy name, etc.) 65 | - (All): 66 | Assert: trades are distributed to strategies. 67 | """ 68 | 69 | """ 70 | Scenario: No trades in broker or db 71 | """ 72 | mock_broker.get_trades = MagicMock(return_value=Trades()) 73 | DB.execute.return_value = [] 74 | 75 | Daemon.recover_trades() 76 | # Check that no trades were adopted 77 | for s in Daemon.strategies: 78 | self.assertEqual(len(s.open_trade_ids), 0) 79 | 80 | """ 81 | Scenario: Open trade in broker and db 82 | """ 83 | def db_execute(query): 84 | if query == 'SELECT trade_id FROM open_trades_live': 85 | return [('id666',)] 86 | elif query == 'SELECT strategy, broker FROM open_trades_live WHERE trade_id="id666"': 87 | return [('Fifty', 'oanda')] 88 | elif query == 'SELECT oanda_name FROM instruments WHERE id=4': 89 | return [('USD_JPY',)] 90 | elif query == 'DELETE FROM open_trades_live WHERE trade_id="id666"': 91 | return 92 | else: 93 | print('unexpected query: {}'.format(query)) 94 | raise Exception 95 | DB.execute = db_execute 96 | trades = Trades() 97 | trades.append(Trade( 98 | units=1, 99 | broker_name='oanda', 100 | instrument=Instrument(4), 101 | stop_loss=90, 102 | strategy=self._strat, 103 | take_profit=100, 104 | trade_id='id666' 105 | )) 106 | mock_broker.get_open_trades = MagicMock(return_value=trades) 107 | mock_broker.is_trade_closed = MagicMock(return_value=(False, None)) 108 | Daemon.recover_trades() 109 | # check Fifty adopted one trade 110 | for s in Daemon.strategies: 111 | if s.get_name() == 'Fifty': 112 | self.assertEqual(len(s.open_trade_ids), 1) 113 | else: 114 | self.assertEqual(len(s.open_trade_ids), 0) 115 | # check trade is the trade we think it is 116 | self.assertEqual(Fifty.open_trade_ids[0], 'id666') 117 | ''' 118 | self.assertEqual(Fifty._open_trades[0].get_broker_name(), 'oanda') 119 | self.assertEqual(Fifty._open_trades[0].get_instrument().get_name(), 'USD_JPY') 120 | self.assertEqual(Fifty._open_trades[0].get_instrument().get_id(), 4) 121 | self.assertEqual(Fifty._open_trades[0].get_stop_loss(), 90) 122 | self.assertEqual(Fifty._open_trades[0].get_take_profit(), 100) 123 | self.assertEqual(Fifty._open_trades[0].get_trade_id(), 'id666') 124 | ''' 125 | # Cleanup 126 | self._strat.cleanup() 127 | 128 | """ 129 | Scenario: Trade is in broker, not db 130 | """ 131 | # Trade may have been opened manually. 132 | # Nothing should happen for these trades. 133 | def db_execute(query): 134 | if query == 'SELECT trade_id FROM open_trades_live': 135 | return [] 136 | elif query == 'SELECT strategy, broker FROM open_trades_live WHERE trade_id="id666"': 137 | return [] 138 | elif query == 'SELECT oanda_name FROM instruments WHERE id=4': 139 | return [('USD_JPY',)] 140 | else: 141 | raise Exception 142 | DB.execute = MagicMock(side_effect=db_execute) 143 | trades = Trades() 144 | trades.append(Trade( 145 | units=1, 146 | broker_name='oanda', 147 | instrument=Instrument(4), 148 | stop_loss=90, 149 | strategy=self._strat, 150 | take_profit=100, 151 | trade_id='id666')) 152 | mock_broker.get_open_trades = MagicMock(return_value=trades) 153 | 154 | Daemon.recover_trades() 155 | # db should stay the same (no inserts or deletions) 156 | # Broker trades should stay the same... 157 | calls = [ 158 | call('SELECT trade_id FROM open_trades_live'), 159 | call('SELECT strategy, broker FROM open_trades_live WHERE trade_id="id666"') 160 | ] 161 | DB.execute.assert_has_calls(calls) 162 | # Check no trades adopted 163 | for s in Daemon.strategies: 164 | self.assertEqual(len(s.open_trade_ids), 0) 165 | 166 | """ 167 | Scenario: Trade in db, broker unsure 168 | """ 169 | def db_execute(query): 170 | if query == 'SELECT trade_id FROM open_trades_live': 171 | return [('id666',)] 172 | elif query == 'SELECT strategy, broker FROM open_trades_live WHERE trade_id="id666"': 173 | return [('Fifty', 'oanda')] 174 | elif query == 'SELECT oanda_name FROM instruments WHERE id=4': 175 | return [('USD_JPY',)] 176 | elif query == 'DELETE FROM open_trades_live WHERE trade_id="id666"': 177 | return 178 | else: 179 | raise Exception 180 | DB.execute = MagicMock(side_effect=db_execute) 181 | mock_broker.get_open_trades = MagicMock(return_value=Trades()) 182 | 183 | Daemon.recover_trades() 184 | # Check trade deleted from db 185 | calls = [ 186 | call('SELECT trade_id FROM open_trades_live'), 187 | call('DELETE FROM open_trades_live WHERE trade_id="id666"') 188 | ] 189 | DB.execute.assert_has_calls(calls) 190 | # Check no trades adopted 191 | for s in Daemon.strategies: 192 | self.assertEqual(len(s.open_trade_ids), 0) 193 | 194 | """ 195 | module cleanup 196 | """ 197 | Daemon.shutdown() 198 | 199 | if __name__ == '__main__': 200 | unittest.main() 201 | -------------------------------------------------------------------------------- /src/tests/test_db.py: -------------------------------------------------------------------------------- 1 | """ 2 | Run like so: 3 | $ python3 tests.py 4 | """ 5 | 6 | import unittest 7 | 8 | class TestDaemon(unittest.TestCase): 9 | 10 | def setUp(self): 11 | # called for every test method 12 | pass 13 | 14 | def tearDown(self): 15 | # called for every test method 16 | pass 17 | 18 | def test_1(self): 19 | self.assertEqual(1, True) 20 | 21 | if __name__ == '__main__': 22 | unittest.main() 23 | 24 | -------------------------------------------------------------------------------- /src/tests/test_instrument.py: -------------------------------------------------------------------------------- 1 | """ 2 | Test the instrument module. 3 | """ 4 | 5 | # library imports 6 | import sys 7 | import unittest 8 | from unittest.mock import MagicMock, patch, call 9 | 10 | # Need to modify sys.path because Python doesn't have relative imports. 11 | # sys.path is initialized from environment variable PYTHONPATH. 12 | try: 13 | sys.path.index('/home/user/raid/software_projects/algo/src') 14 | except ValueError: 15 | sys.path.append('/home/user/raid/software_projects/algo/src') 16 | 17 | # local imports 18 | from config import Config 19 | from db import DB 20 | from instrument import Instrument 21 | 22 | class TestInstrument(unittest.TestCase): 23 | 24 | I_ID = 4 25 | I_NAME = 'USD_JPY' 26 | TARGET_BROKER_NAME = 'oanda' 27 | 28 | 29 | # called for every test method 30 | def setUp(self): 31 | self.assertEqual(Config.broker_name, self.TARGET_BROKER_NAME) 32 | 33 | 34 | # called for every test method 35 | def tearDown(self): 36 | DB.shutdown() # this might cause trouble for multiple tests 37 | pass 38 | 39 | 40 | """ 41 | """ 42 | def test_1(self): 43 | instrument = Instrument(self.I_ID) 44 | self.assertEqual(instrument.get_id(), self.I_ID) 45 | self.assertEqual(instrument.get_name(), self.I_NAME) 46 | self.assertEqual(instrument.get_name_from_id(2), 'DCK') 47 | self.assertEqual(Instrument.get_id_from_name('SNK'), 3) 48 | 49 | 50 | if __name__ == '__main__': 51 | unittest.main() 52 | 53 | -------------------------------------------------------------------------------- /src/tests/test_oanda.py: -------------------------------------------------------------------------------- 1 | """ 2 | Run like so: 3 | $ python3 tests.py 4 | """ 5 | 6 | import datetime 7 | import unittest 8 | 9 | from config import Config 10 | from instrument import Instrument 11 | from oanda import Oanda 12 | 13 | class TestDaemon(unittest.TestCase): 14 | 15 | def setUp(self): 16 | # called for every test method 17 | pass 18 | 19 | 20 | def tearDown(self): 21 | # called for every test method 22 | pass 23 | 24 | 25 | def test_get_auth_key(self): 26 | result = None 27 | result = Oanda.get_auth_key() 28 | self.assertTrue(result) # not None 29 | 30 | 31 | def test_fetch(self): 32 | result = None 33 | result = Oanda.fetch( '{}/v3/accounts'.format(Config.oanda_url) ) 34 | self.assertTrue(result) # not None 35 | result = Oanda.fetch( '{}/v3/xxxxxx'.format(Config.oanda_url) ) 36 | self.assertEqual(result, None) # None 37 | 38 | 39 | def test_get_prices(self): 40 | result = None 41 | result = Oanda.get_prices( [Instrument(4)] ) 42 | self.assertTrue(result) # not None 43 | 44 | 45 | def test_get_time_until_close (self): 46 | zero_delta = datetime.timedelta() 47 | result = Oanda.get_time_until_close() # timedelta 48 | api_open = Oanda.is_market_open(Instrument(4)) 49 | if api_open: 50 | self.assertTrue(result != zero_delta) 51 | else: 52 | self.assertTrue(result == zero_delta) 53 | print('*****************************************\\') 54 | print('time until market close: {}'.format(result)) 55 | print('*****************************************/') 56 | 57 | 58 | def test_get_time_since_close (self): 59 | """ 60 | These should be true of the time since close: 61 | - less than one week ago 62 | - before now 63 | - if market open now, > 2 days ago 64 | - if market closed now, less than 2 days ago 65 | """ 66 | now = datetime.datetime.utcnow() 67 | zero_delta = datetime.timedelta() 68 | time_since_close = Oanda.get_time_since_close() # timedelta 69 | print('***********************************************\\') 70 | print('time since last close: {}'.format(time_since_close)) 71 | print('***********************************************/') 72 | # check < 1 week 73 | self.assertTrue(now - (now - time_since_close) < datetime.timedelta(days=7)) 74 | # Check before now 75 | self.assertTrue((now - time_since_close) < now) 76 | # Check weekend (2 days) 77 | market_open = Oanda.is_market_open(Instrument(4)) # USD/JPY market 78 | if market_open: 79 | self.assertTrue( time_since_close > datetime.timedelta(days=2)) 80 | else: 81 | self.assertTrue( time_since_close < datetime.timedelta(days=2)) 82 | 83 | 84 | if __name__ == '__main__': 85 | unittest.main() 86 | 87 | -------------------------------------------------------------------------------- /src/tests/test_opportunity.py: -------------------------------------------------------------------------------- 1 | """ 2 | Run like so: 3 | $ python3 tests.py 4 | """ 5 | 6 | import unittest 7 | 8 | class TestDaemon(unittest.TestCase): 9 | 10 | def setUp(self): 11 | # called for every test method 12 | print('set up') 13 | 14 | def tearDown(self): 15 | # called for every test method 16 | print('tear down') 17 | 18 | def test_1(self): 19 | print('test_1') 20 | self.assertEqual(1, True) 21 | 22 | def test_2(self): 23 | print('test_2') 24 | self.assertEqual(1, True) 25 | 26 | if __name__ == '__main__': 27 | unittest.main() 28 | 29 | -------------------------------------------------------------------------------- /src/tests/test_order.py: -------------------------------------------------------------------------------- 1 | """ 2 | Run like so: 3 | $ python3 tests.py 4 | """ 5 | 6 | import unittest 7 | 8 | class TestDaemon(unittest.TestCase): 9 | 10 | def setUp(self): 11 | # called for every test method 12 | print('set up') 13 | 14 | def tearDown(self): 15 | # called for every test method 16 | print('tear down') 17 | 18 | def test_1(self): 19 | print('test_1') 20 | self.assertEqual(1, True) 21 | 22 | def test_2(self): 23 | print('test_2') 24 | self.assertEqual(1, True) 25 | 26 | if __name__ == '__main__': 27 | unittest.main() 28 | 29 | -------------------------------------------------------------------------------- /src/tests/test_strategy.py: -------------------------------------------------------------------------------- 1 | """ 2 | Test the instrument module. 3 | """ 4 | 5 | # library imports 6 | import sys 7 | import unittest 8 | from unittest.mock import MagicMock, patch, call 9 | 10 | # Need to modify sys.path because Python doesn't have relative imports. 11 | # sys.path is initialized from environment variable PYTHONPATH. 12 | try: 13 | sys.path.index('/home/user/raid/software_projects/algo/src') 14 | except ValueError: 15 | sys.path.append('/home/user/raid/software_projects/algo/src') 16 | 17 | # local imports 18 | from db import DB 19 | from strategy import Strategy 20 | 21 | class SampleStrategy(Strategy): 22 | def __init__(self): 23 | pass 24 | def get_name(self): 25 | return 'sample' 26 | def _babysit(self): 27 | pass 28 | def _scan(self): 29 | return None 30 | 31 | class TestStrategy(unittest.TestCase): 32 | 33 | _strat = SampleStrategy() 34 | 35 | # called for every test method 36 | def setUp(self): 37 | pass 38 | 39 | 40 | # called for every test method 41 | def tearDown(self): 42 | pass 43 | 44 | 45 | def test___str__(self): 46 | self.assertEqual(str(self._strat.get_name()), 'sample') 47 | 48 | 49 | def test_get_name(self): 50 | self.assertEqual(self._strat.get_name(), 'sample') 51 | 52 | 53 | def test_trade_opened(self): 54 | """ 55 | TODO: check db too 56 | """ 57 | #import pdb; pdb.set_trace() 58 | self._strat.open_trade_ids = [] 59 | self._strat.trade_opened('1') 60 | self.assertEqual(self._strat.open_trade_ids, ['1']) 61 | self._strat.trade_opened('666') 62 | self.assertEqual(self._strat.open_trade_ids, ['1','666']) 63 | self._strat.trade_opened('mx.!@#$%^&*()_+=-/|') 64 | self.assertEqual(self._strat.open_trade_ids, ['1','666','mx.!@#$%^&*()_+=-/|']) 65 | DB.execute("DELETE FROM open_trades_live WHERE trade_id in ('1', '666', 'mx.!@#$%^&*()_+=-/|')") 66 | 67 | 68 | def test_recover_trades(self): 69 | pass 70 | 71 | 72 | def test_drop_all(self): 73 | pass 74 | 75 | 76 | def test_refresh(self): 77 | pass 78 | 79 | 80 | def test__babysit(self): 81 | pass 82 | 83 | 84 | def test__scan(self): 85 | pass 86 | 87 | 88 | if __name__ == '__main__': 89 | unittest.main() 90 | 91 | import atexit 92 | atexit.register(DB.shutdown) 93 | -------------------------------------------------------------------------------- /src/tests/test_timer.py: -------------------------------------------------------------------------------- 1 | """ 2 | Run like so: 3 | $ python3 tests.py 4 | """ 5 | 6 | import unittest 7 | 8 | class TestDaemon(unittest.TestCase): 9 | 10 | def setUp(self): 11 | # called for every test method 12 | print('set up') 13 | 14 | def tearDown(self): 15 | # called for every test method 16 | print('tear down') 17 | 18 | def test_1(self): 19 | print('test_1') 20 | self.assertEqual(1, True) 21 | 22 | def test_2(self): 23 | print('test_2') 24 | self.assertEqual(1, True) 25 | 26 | if __name__ == '__main__': 27 | unittest.main() 28 | 29 | -------------------------------------------------------------------------------- /src/tests/test_trade.py: -------------------------------------------------------------------------------- 1 | """ 2 | Run like so: 3 | $ python3 tests.py 4 | """ 5 | 6 | import unittest 7 | 8 | class TestDaemon(unittest.TestCase): 9 | 10 | def setUp(self): 11 | # called for every test method 12 | print('set up') 13 | 14 | def tearDown(self): 15 | # called for every test method 16 | print('tear down') 17 | 18 | def test_1(self): 19 | print('test_1') 20 | self.assertEqual(1, True) 21 | 22 | def test_2(self): 23 | print('test_2') 24 | self.assertEqual(1, True) 25 | 26 | if __name__ == '__main__': 27 | unittest.main() 28 | 29 | -------------------------------------------------------------------------------- /src/tests/test_util_date.py: -------------------------------------------------------------------------------- 1 | """ 2 | Test the util_date module. 3 | """ 4 | 5 | # library imports 6 | from datetime import datetime, MAXYEAR, MINYEAR 7 | import sys 8 | import unittest 9 | 10 | # Need to modify sys.path because Python doesn't have relative imports. 11 | # sys.path is initialized from environment variable PYTHONPATH. 12 | try: 13 | sys.path.index('/home/user/raid/software_projects/algo/src') 14 | except ValueError: 15 | sys.path.append('/home/user/raid/software_projects/algo/src') 16 | 17 | # local imports 18 | import util_date 19 | 20 | class TestInstrument(unittest.TestCase): 21 | 22 | # called for every test method 23 | def setUp(self): 24 | pass 25 | 26 | 27 | # called for every test method 28 | def tearDown(self): 29 | pass 30 | 31 | 32 | ''' 33 | def test_datetime_to_numeric(self): 34 | now = datetime.utcnow() 35 | numeric = util_date.datetime_to_numeric(now) 36 | back_to_dt = datetime.strptime(str(numeric), '%Y%m%d%H%M%S.%f') 37 | self.assertEqual( now, back_to_dt ) 38 | ''' 39 | 40 | 41 | def test_string_to_date(self): 42 | # normal numbers 43 | target_date = datetime( 44 | year=2014, month=7, day=2, hour=4, 45 | minute=14, second=59, microsecond=123456) 46 | sample_string = '2014-07-02T04:14:59.123456789Z' 47 | result_date = util_date.string_to_date(sample_string) 48 | self.assertEqual(target_date, result_date) 49 | 50 | # minimum 51 | sample_string = '0001-01-01T00:00:00.000000Z' 52 | target_date = datetime( 53 | year=MINYEAR, month=1, day=1, hour=0, 54 | minute=0, second=0, microsecond=0) 55 | result_date = util_date.string_to_date(sample_string) 56 | self.assertEqual(target_date, result_date) 57 | 58 | # maximum 59 | sample_string = '9999-12-31T23:59:59.999999Z' 60 | target_date = datetime( 61 | year=MAXYEAR, month=12, day=31, hour=23, 62 | minute=59, second=59, microsecond=999999) 63 | result_date = util_date.string_to_date(sample_string) 64 | self.assertEqual(target_date, result_date) 65 | 66 | 67 | def test_date_to_string(self): 68 | # normal numbers 69 | sample_date = datetime( 70 | year=2014, month=7, day=2, hour=4, 71 | minute=14, second=59, microsecond=123456) 72 | target_string = '2014-07-02T04:14:59.123456000Z' 73 | result_string = util_date.date_to_string(sample_date) 74 | self.assertEqual(target_string, result_string) 75 | 76 | # minimum - I'm not sure if brokers would want years to be zero padded. 77 | sample_date = datetime( 78 | year=1950, month=1, day=1, hour=0, 79 | minute=0, second=0, microsecond=0) 80 | target_string = '1950-01-01T00:00:00.000000000Z' 81 | result_string = util_date.date_to_string(sample_date) 82 | self.assertEqual(target_string, result_string) 83 | 84 | # maximum 85 | sample_date = datetime( 86 | year=MAXYEAR, month=12, day=31, hour=23, 87 | minute=59, second=59, microsecond=999999) 88 | target_string = '9999-12-31T23:59:59.999999000Z' 89 | result_string = util_date.date_to_string(sample_date) 90 | self.assertEqual(target_string, result_string) 91 | 92 | 93 | if __name__ == '__main__': 94 | unittest.main() 95 | -------------------------------------------------------------------------------- /src/timer.py: -------------------------------------------------------------------------------- 1 | """ 2 | Description: 3 | Convenient class for timing blocks of code. 4 | Intended to be used to time functions. 5 | 6 | Overview: 7 | When class is created, it reads existing times from the db. 8 | Call the start() method to get the starting timestamp. 9 | Call the stop() method to stop timing and doesn't return anything of note. 10 | The stop method takes in some details including a unique identifier. 11 | The stop method checks to see if the duration is a new max, and 12 | automatically saves 13 | """ 14 | 15 | #################### 16 | import concurrent.futures 17 | import datetime 18 | import mysql.connector 19 | from threading import Thread 20 | import timeit 21 | #################### 22 | from config import Config 23 | from db import DB 24 | from log import Log 25 | import sys 26 | #################### 27 | 28 | 29 | def timer_decorator(cls): 30 | # Load records from database, after the class is created. 31 | cls.records = DB.execute('SELECT * FROM function_times') 32 | if cls.records == None: 33 | DB.bug('records == None. Aborting.') 34 | sys.exit() 35 | #print('"timer.py" timer_decorator(): cls.records is initialized to {}'.format(cls.records)) 36 | return cls 37 | 38 | 39 | @timer_decorator 40 | class Timer(): 41 | 42 | """ 43 | "Records" as in breaking a record; new max duration. 44 | The records variable is a list of dicts that matches the table columns: 45 | function_name (PK) 46 | timestamp 47 | duration 48 | note 49 | """ 50 | records = [] # TODO Use hash table for fast insertion. 51 | 52 | 53 | @classmethod 54 | def start(cls): 55 | return timeit.default_timer() 56 | 57 | 58 | @classmethod 59 | def stop(cls, start, function_name, note): 60 | """ 61 | Compare a new time to the record. 62 | If the new time is a record, save it. 63 | """ 64 | #Log.write('"timer.py" stop(): Entering with records:\n', 65 | # '{}'.format(cls.records)) 66 | timestamp = datetime.datetime.now() 67 | duration = timeit.default_timer() - start 68 | new_max = False 69 | found = False 70 | for r in cls.records: 71 | r_fun = r[0] 72 | r_time = r[1] 73 | r_dur = r[2] 74 | r_note = r[3] 75 | if r_fun == function_name: 76 | """ 77 | Log.write('"timer.py" stop(): Function {} has previous max duration of {}' 78 | .format( 79 | r_fun, 80 | r_dur) 81 | ) 82 | """ 83 | found = True 84 | sec = int(duration) 85 | if datetime.timedelta(seconds=sec,microseconds=duration-sec) > r_dur: 86 | # new max 87 | new_max = True 88 | r_dur = duration 89 | r_note = note 90 | break # found it 91 | if not found: 92 | # create new entry 93 | new_max = True 94 | cls.records.append((timestamp,function_name, duration, note)) 95 | # save to database if new max duration 96 | if new_max: 97 | #Log.write('"timer.py" stop(): New max duration for ', 98 | # 'function {} = {}s'.format(function_name, duration)) 99 | db_result = DB.execute( 100 | 'INSERT INTO function_times \ 101 | (function_name, timestamp, duration, note) \ 102 | VALUES ("{0}","{1}","{2}","{3}") \ 103 | ON DUPLICATE KEY UPDATE \ 104 | timestamp="{1}", \ 105 | duration="{2}", \ 106 | note="{3}"' 107 | .format( 108 | function_name, 109 | timestamp, 110 | duration, 111 | note 112 | ) 113 | ) 114 | else: 115 | #Log.write('"timer.py" stop(): Not a new duration: {} for function {}' 116 | # .format(duration, function_name)) 117 | pass 118 | -------------------------------------------------------------------------------- /src/trade.py: -------------------------------------------------------------------------------- 1 | """ 2 | File: trade.py 3 | Python version: 3.4 4 | Description: Containers for trades. 5 | """ 6 | 7 | #################### 8 | from collections.abc import Sequence 9 | #################### 10 | from db import DB 11 | #################### 12 | 13 | 14 | class TradeClosedReason(): 15 | """ 16 | An enum. 17 | """ 18 | 19 | LIMIT_ORDER = 1 20 | STOP_ORDER = 2 21 | MARKET_IF_TOUCHED_ORDER = 3 22 | TAKE_PROFIT_ORDER = 4 23 | STOP_LOSS_ORDER = 5 24 | TRAILING_STOP_LOSS_ORDER = 6 25 | MARKET_ORDER = 7 26 | MARKET_ORDER_TRADE_CLOSE = 8 27 | MARKET_ORDER_POSITION_CLOSEOUT = 9 28 | MARKET_ORDER_MARGIN_CLOSEOUT = 10 29 | MARKET_ORDER_DELAYED_TRADE_CLOSE = 11 30 | LINKED_TRADE_CLOSED = 12 31 | 32 | 33 | class Trade(): 34 | """ 35 | Represents one trade, either past or present. 36 | Future trades are "opportunities" or "orders". 37 | """ 38 | 39 | def __init__( 40 | self, 41 | units=None, # units; negative for short 42 | broker_name=None, # broker ID (name string) from databse 43 | instrument=None, # instance 44 | stop_loss=None, # numeric 45 | strategy=None, # class reference 46 | take_profit=None, # numeric 47 | trade_id=None # string 48 | ): 49 | self.units = units 50 | self.broker_name = broker_name 51 | self.instrument = instrument 52 | self.stop_loss = stop_loss 53 | self.strategy = strategy 54 | self.take_profit = take_profit 55 | self.trade_id = trade_id 56 | 57 | 58 | def fill_in_extra_info(self): 59 | """ 60 | Fill in info not provided by the broker, e.g. 61 | the name of the strategy that opened the trade. 62 | 63 | It's possible that a trade will be opened then the system is 64 | unexpectedly terminated before info about the trade can be saved to 65 | the database. Thus a trade passed to this 66 | function may not have a corresponding trade in the database. 67 | 68 | Returns: (nothing) 69 | """ 70 | print ('trade.fill_in_extra_info()') 71 | trade_info = DB.execute('SELECT strategy, broker, instrument_id FROM open_trades_live WHERE trade_id = {}' 72 | .format(self.trade_id)) 73 | if len(trade_info) > 0: 74 | # verify broker and instrument match, just to be safe 75 | if trade_info[0][1] != self.broker_name: 76 | Log.write('"trade.py" fill_in_extra_info(): ERROR: "{}" != "{}"' 77 | .format(trade_info[0][1], self.broker_name)) 78 | raise Exception 79 | instrument = DB.execute('SELECT symbol FROM instruments WHERE id = {}' 80 | .format(trade_info[0][2])) 81 | if instrument[0][0] != self.instrument: 82 | Log.write('"trade.py" fill_in_extra_info(): {} != {}' 83 | .format(instrument[0]['symbol'], self.instrument)) 84 | raise Exception 85 | # save strategy 86 | self.strategy = None 87 | # TODO: good practice to access daemon's strategy list like this? 88 | for s in Daemon.strategies: 89 | if s.get_name == trade_info[0][0]: 90 | self.strategy = s # reference to class instance 91 | # It might be possible that the trade was opened by a 92 | # strategy that is not running. In that case, use the default 93 | # strategy. 94 | self.strategy = Daemon.backup_strategy 95 | 96 | 97 | def __str__(self): 98 | strategy_name = "(unknown)" 99 | if self.strategy != None: 100 | strategy_name = self.strategy.get_name() 101 | msg = 'ID: {}, Units: {}, Broker: {}, Instrument: {}, Strategy: {}, SL: {}, TP: {}'\ 102 | .format( 103 | self.trade_id, 104 | self.units, 105 | self.broker_name, 106 | self.instrument, 107 | strategy_name, 108 | self.stop_loss, 109 | self.take_profit 110 | ) 111 | return msg 112 | 113 | 114 | class Trades(Sequence): 115 | """ 116 | List of objects. 117 | TODO: This could be a heap, with the trade's ID as the key. 118 | """ 119 | 120 | def __init__(self): 121 | self.trade_list = [] 122 | self.current_index = 0 123 | 124 | def append(self, trade): 125 | self.trade_list.append(trade) 126 | 127 | 128 | def pop(self, trade_id): 129 | """ 130 | Remove the trade with the given transaction ID. 131 | 132 | Returns: Removed trade object on success; None on failure. 133 | """ 134 | index = 0 135 | for t in self.trade_list: 136 | if t._trade_id == trade_id: 137 | return self.trade_list.pop(index) 138 | index = index + 1 139 | return None 140 | 141 | 142 | """ 143 | Return type: void 144 | Delete all trades. 145 | """ 146 | def clear(self): 147 | del self.trade_list[:] 148 | 149 | 150 | def __len__(self): 151 | return len(self.trade_list) 152 | 153 | 154 | def __str__(self): 155 | msg = '' 156 | for t in self.trade_list: 157 | msg = msg + str(t) + '\n' 158 | return msg 159 | 160 | 161 | def __iter__(self): 162 | """ 163 | Make this class iterable 164 | https://docs.python.org/3/library/stdtypes.html#typeiter 165 | """ 166 | return self 167 | 168 | 169 | def __next__(self): 170 | """ 171 | Make this class iterable 172 | """ 173 | if self.current_index >= len(self.trade_list): 174 | self.current_index = 0 175 | raise StopIteration 176 | else: 177 | self.current_index = self.current_index + 1 178 | return self.trade_list[self.current_index - 1] 179 | 180 | 181 | def __getitem__(self, key): 182 | """ 183 | Expose index operator. 184 | """ 185 | return self.trade_list[key] 186 | 187 | 188 | def __len__(self): 189 | """ 190 | Expose len() function. 191 | """ 192 | return len(self.trade_list) 193 | 194 | -------------------------------------------------------------------------------- /src/util_date.py: -------------------------------------------------------------------------------- 1 | """ 2 | Date utilities 3 | """ 4 | 5 | from datetime import datetime, timezone 6 | from log import Log 7 | 8 | 9 | """ 10 | Day of week enumeration. 11 | Matches ISO weekday numbers. 12 | """ 13 | MONDAY = 1 14 | TUESDAY = 2 15 | WEDNESDAY = 3 16 | THURSDAY = 4 17 | FRIDAY = 5 18 | SATURDAY = 6 19 | SUNDAY = 7 20 | 21 | 22 | ''' problems with float limitations 23 | def datetime_to_numeric(dt): 24 | """Return type: float 25 | Datetime to UTC numeric representation in this format: 26 | YYYYMMDDHHMMSS. 27 | """ 28 | dt_utc = dt.replace(tzinfo=timezone.utc) 29 | print('\n dt_utc = {} \n'.format(dt_utc) ) 30 | try: 31 | dt_string = dt_utc.strftime( '%Y%m%d%H%M%S.%f' ) 32 | print('\n dt_string = {} \n'.format(dt_string) ) 33 | except: 34 | Log.write('util_date.py datetime_to_numeric(): Failed to convert to string') 35 | raise Exception 36 | try: 37 | print('\n float(dt_string) = {} \n'.format( float(dt_string) ) ) 38 | return float(dt_string) 39 | except: 40 | Log.write('util_date.py datetime_to_numeric(): Failed to convert to numeric') 41 | raise Exception 42 | ''' 43 | 44 | def date_to_string(d): 45 | """Returns: string 46 | Convert datetime to string in RFC3339 format: 47 | "YYYY-MM-DDTHH:MM:SS.MMMMMMMMMZ" 48 | param: type: 49 | d datetime in UTC 50 | """ 51 | try: 52 | # strftime %f formats to 6 digit ms, but Oanda uses 9. 53 | six_digit_milliseconds = datetime.strftime(d, '%Y-%m-%dT%H:%M:%S.%f') 54 | nine_digit_milliseconds = six_digit_milliseconds[0:26] + '000Z' 55 | return nine_digit_milliseconds 56 | except: 57 | Log.write( 58 | 'util_date.py date_to_string(): Failed to convert date({}) to string.' 59 | .format(d)) 60 | raise Exception 61 | 62 | 63 | def string_to_date(s): 64 | """Return type: datetime 65 | Parse a string into a datetime. 66 | param: type: 67 | s String of UTC datetime. 68 | Oanda typically sends 9-digit milliseconds with a Z on the end. 69 | Six digits of milliseconds are preserved by Python's datetime. 70 | """ 71 | try: 72 | #z_index = s.rfind('Z') 73 | dot_index = s.rfind('.') 74 | return datetime.strptime(s[0:dot_index + 7], '%Y-%m-%dT%H:%M:%S.%f') 75 | except: 76 | Log.write( 77 | 'util_date.py string_to_date(): Failed to convert string ({}) to date.' 78 | .format(s)) 79 | raise Exception 80 | 81 | -------------------------------------------------------------------------------- /src/utils.py: -------------------------------------------------------------------------------- 1 | """ 2 | Utilities 3 | """ 4 | 5 | import urllib 6 | 7 | 8 | """ 9 | Return type: string 10 | Converts a list of instances to 11 | a URL encoded comma-separated string, 12 | e.g. USD_JPY%2CEUR_USD 13 | """ 14 | def instruments_to_url( 15 | instruments 16 | ): 17 | if len(instruments) == 0: 18 | raise Exception 19 | url = "" 20 | for index, instrument in enumerate(instruments): 21 | if index < len(instruments) - 1: 22 | url += (instrument.get_name() + '%2C') 23 | else: 24 | url += instrument.get_name() 25 | return url 26 | 27 | 28 | """ 29 | Return type: string 30 | Simply encodes a string for a url, inserting %xx as needed. 31 | """ 32 | def url_encode(url): 33 | return urllib.parse.quote(url) 34 | 35 | 36 | """ 37 | Return type: string 38 | Decode bytes to string using UTF8. 39 | Parameter `b' is assumed to have type of `bytes'. 40 | """ 41 | def btos(b): 42 | if b == None: 43 | return None 44 | else: 45 | return b.decode('utf_8') 46 | 47 | 48 | """ 49 | Return type: bytes 50 | """ 51 | def stob(s): 52 | if s == None: 53 | return None 54 | else: 55 | return s.encode('utf_8') 56 | 57 | 58 | --------------------------------------------------------------------------------