├── Readme.md ├── oKalman.py ├── oTradingSystem.py └── oTradingOperations.py /Readme.md: -------------------------------------------------------------------------------- 1 | 2 | # Automatic trading system with Python(2) 3 | 4 | ## Description 5 | 6 | Simple Kalman filter strategy for trading a portfolio of 5 currency pairs. Will add a proper description at some later point in time. 7 | 8 | ## Current status 9 | 10 | Code works (or should work as is) a list of tuples denoting Fridays is added manually into the oTradingSystem.py script, this is to be corrected next. Also the Kalman filter "burn-in" has to be specified the same way, this also denotes the window from which rolling average and standard deviation of the portfolio are calculated. 11 | 12 | Since the program skips Friday 5pm EST - Sunday 5pm EST by just waiting a fixed amount of time, the 13 | oTradingSystem.py should be run only when trading is active. 14 | 15 | So to be fixed next are insertion of data from the command line and ability to start the system whenever. All is dependent on time... 16 | 17 | Added possibility to pickle the states if one wants to do maintenance and updates during the weekend for example. Also added some safeguards to handle errors if connection to the broker is cut, something that seems to happen every Thursday at 10pm EST... 18 | 19 | Broker used is Oanda, and the API for it is provided by https://github.com/hootnot/oanda-api-v20, it is exellent and easy to use! Thanks hootnot! 20 | 21 | ### Care must be taken, this strategy is not to be used for trading as is unless you hate money. 22 | 23 | -------------------------------------------------------------------------------- /oKalman.py: -------------------------------------------------------------------------------- 1 | 2 | # Import numerical libraries 3 | import numpy as np 4 | 5 | class KalmanCoint: 6 | 7 | """ 8 | The state space equations are 9 | 10 | x(t) = Ax(t-1)+w 11 | z(t) = Hx(t)+v 12 | 13 | where x is the hidden state and z is the observation and w ~ N(0,Q) and 14 | v ~ N(0,R). Dimensions are as follows: 15 | 16 | x = [beta alpha]', i.e column vector where beta is the vector familiar from 17 | regression analysis and alpha is the intercept- Length n. 18 | Q and A are nxn matricies 19 | 20 | z has the dimenstion m and R is mxm 21 | 22 | H = [explanatory_variables 1], it is in general mxn 23 | 24 | The algorith goes as follows (^ denotes the prior dostribution): 25 | 26 | 1. Update/initialize the prior expectation from posterior expectation: 27 | x^(t) = A*x(t-1) 28 | 2. Update/initialize the prior error from posterior error: 29 | P^(t) = A*P(t-1)*A' + Q 30 | 2B. Update H(t) as well if it is dependent on time, as it is here. 31 | 3. Calculate the Kalman gain K(t): 32 | K(t) = P^(t)*H(t)'*(H(t)*P^(t)*H(t)' + R)**(-1), 33 | 4. Update the posterior expectation using the observation z(t): 34 | x(t) = x^(t) + K(t)*(z(t)-H(t)*x^(t)) 35 | 5. Update the posterior error (variance): 36 | P(t) = (I-K(t)*H(t)=*P^(t), where I is the identity matrix 37 | 38 | rinse and repeate..... 39 | 40 | The observation and H(t) are passed into the filtering method at every 41 | step, all other data is initialized in the beginning. All variables passed 42 | into this class must be numpy arrays of the correct dimension!!! 43 | """ 44 | 45 | # Initially what are needed are x_0, P_0, A, Q and R. 46 | # x_0 has to be of the shape np.array([[1],[1],[1]]) 47 | def __init__(self, x_0, P_0, A, Q, R): 48 | 49 | self.A, self.Q, self.R = A, Q, R 50 | self.x_pos = x_0 51 | self.P_pos = P_0 52 | 53 | # Filtering, takes in the observation as well as H. 54 | # H has to be of the shape np.array([[1,2,3]]) 55 | def Filtering(self, z, H): 56 | 57 | # Step 1 and 2 58 | self.x_pri = np.matmul(self.A,self.x_pos) 59 | self.P_pri = np.matmul(self.A,np.matmul( 60 | self.P_pos,np.transpose(self.A)))+self.Q 61 | 62 | # Steps 2B and 3 63 | x = np.matmul(H,np.matmul(self.P_pri,np.transpose(H)))+self.R 64 | if len(x.shape)==0: # m==1 65 | self.K = np.matmul(self.P_pri,np.transpose(H))*(1.0/float(x)) 66 | else: # m>1, for generality 67 | self.K = np.matmul(self.P_pri,np.matmul( 68 | np.transpose(H),np.linalg.inv(x))) 69 | 70 | # Step 4 71 | if z.size==1: # m==1 72 | self.x_pos = self.x_pri+self.K*(z-np.matmul(H,self.x_pri)) 73 | else: # m>1, for generality 74 | self.x_pos = self.x_pri+np.matmul( 75 | self.K,z-np.matmul(H,self.x_pri)) 76 | 77 | # Step 5 78 | self.P_pos = np.matmul(np.eye(len(self.P_pri))-np.matmul(self.K,H), 79 | self.P_pri) 80 | 81 | #return(self.x_pri, self.P_pri) 82 | 83 | 84 | -------------------------------------------------------------------------------- /oTradingSystem.py: -------------------------------------------------------------------------------- 1 | 2 | """Main script for running the trading system, calls all other programs. 3 | 4 | This code will be the main script called for running the trading system. As of 5 | now, weekends and holidays must be specified separately in order to run the 6 | system for more than a couple of days. 7 | 8 | The system will in the beginning run a "burn-in" for the Kalamn filter the 9 | length of which canc be specified. Now at first this and the weekends will 10 | be set inside the code. 11 | 12 | The program calls the datetime.now().second every one second and for keeping 13 | time, and calls the Kalman filter to do its calculations at the beginning of 14 | each hour. If the Kalman filter returns a buy or sell signal, depending on if 15 | there are any open trades, the program will call the appropriate modules for 16 | dealing with the opening or closing requests. 17 | 18 | 19 | """ 20 | # Import standard python stuff 21 | import numpy as np 22 | import time 23 | import datetime 24 | import threading 25 | import json 26 | import pickle 27 | 28 | # Import oandapyV20 stuff 29 | import oandapyV20 30 | import oandapyV20.endpoints.instruments as instruments 31 | from oandapyV20.contrib.requests import MarketOrderRequest 32 | import oandapyV20.endpoints.orders as orders 33 | import oandapyV20.endpoints.trades as trades 34 | import oandapyV20.endpoints.accounts as accounts 35 | from oandapyV20.exceptions import V20Error 36 | 37 | # Import my own stuff 38 | from oKalman import KalmanCoint 39 | import oTradingOperations as to 40 | 41 | # First specify account ID and number 42 | Account_ID = '......' 43 | Account_Token = '........' 44 | 45 | # ------------------------------------------------------------------------------ 46 | # ---------- Get the previous state of the system or start anew ---------------- 47 | new_kalman = False 48 | new_positions = False 49 | 50 | print '\n Unpickle the previous state for Tranding Operations? (y/n)' 51 | while True: 52 | x = raw_input() 53 | if (x=='y') or (x=='n'): 54 | if x=='n': 55 | print 'Fresh start for Trading Operations!\n' 56 | new_positions = True 57 | else: 58 | try: 59 | with open('positions.pickle', 'rb') as f: 60 | positions = pickle.load('positions.pickle', 'rb') 61 | except IOError: 62 | print 'No pickled Trading Operations states! Fresh start!\n' 63 | new_positions = True 64 | break 65 | else: print 'Give y for yes or n for no' 66 | 67 | # If starting anew with Trading Operations 68 | if new_positions: 69 | # Initialize the position manager 70 | Instruments = ['EUR_USD', 'GBP_USD', 'AUD_USD', 'USD_CAD', 'USD_JPY'] 71 | I_types = [1,1,1,2,2] # How the lots are calculated 72 | positions = to.oPositionManager(Account_ID, Account_Token, Instruments, 73 | I_types) 74 | 75 | print '\n Unpickle the previous state for Kalman filter? (y/n)' 76 | while True: 77 | x = raw_input() 78 | if (x=='y') or (x=='n'): 79 | if x=='n': 80 | print 'Fresh start!' 81 | new_kalman = True 82 | else: 83 | try: 84 | with open('kf.pickle', 'rb') as f: 85 | kf = pickle.load('kf.pickle', 'rb') 86 | except IOError: 87 | print 'No pickled Kalman filter states! Fresh start!' 88 | new_kalman = True 89 | break 90 | 91 | else: print 'Give y for yes or n for no' 92 | 93 | 94 | # If fresh start for Kalman, then initialize it 95 | if new_kalman==True: 96 | # Specify the 'burn-in' 97 | burn_in = 506 # The last hour is not to be used 98 | print '\n Initializing the Kalman filter. \n' 99 | # Specify the standard values for our 5 currency pair strategy. 100 | x_0 = np.array([[1],[1],[1],[1],[1]]) 101 | P_0 = np.ones((5,5)) 102 | A = np.eye(5) 103 | Q = np.eye(5) 104 | R = 1 105 | kf = KalmanCoint(x_0, P_0, A, Q, R) # Initialize Kalman filter 106 | print '\n Doing the burnin with', burn_in-1, 'candles.\n' 107 | # Get the data for the burnin, calls oGetData() 108 | z_m, H_m = positions.oGetData(burn_in) 109 | # Define an array for calculating the rolling mean and std 110 | Z = np.empty(len(z_m)) 111 | # Do the burn-in 112 | for i in xrange(len(z_m)): 113 | kf.Filtering(z_m[i], H_m[i:i+1,:]) # Numpys funny slicing conventions... 114 | Z[i] = z_m[i]-np.dot(H_m[i:i+1], kf.x_pri) 115 | Z = Z[1:] 116 | print '\n Current state values:' 117 | print 'Basket: ', Z[-1] 118 | print 'Basket mean: ', np.mean(Z) 119 | print 'Basket STD: ', np.std(Z, ddof=1), '\n' 120 | print 'Basket short values:' 121 | print np.mean(Z)+2*np.std(Z, ddof=1) 122 | print np.mean(Z)+3*np.std(Z, ddof=1) 123 | print np.mean(Z)+4*np.std(Z, ddof=1), '\n' 124 | print 'Basket long values:' 125 | print np.mean(Z)-2*np.std(Z, ddof=1) 126 | print np.mean(Z)-3*np.std(Z, ddof=1) 127 | print np.mean(Z)-4*np.std(Z, ddof=1) 128 | 129 | #The program will sleep from Friday 5pm to Sunday 5pm. All times are EST!! 130 | fridays = [(..,..),(..,..)...] # Put the Fridays in a (day,month) tuple 131 | 132 | # Get the current hour of the day and print the current time and date 133 | dt = datetime.datetime.now() 134 | current_hour = dt.hour 135 | 136 | print '\n Starting up time and date: ', dt.strftime('%H:%M:%S %d.%m.%Y') 137 | print ' Terminate by:', fridays[-1],'\n' 138 | 139 | # -------------------- Start the system -------------------------------------- 140 | print '\n System initialized, terminate with Ctrl-c \n' 141 | 142 | try: 143 | while True: 144 | 145 | # Get the date and time 146 | dt = datetime.datetime.now() 147 | 148 | # If we are in the active trading period of the week up until last hour: 149 | if ((dt.day,dt.month) not in fridays) or (dt.hour<=16): 150 | 151 | # If new hour 152 | if dt.hour!=current_hour: 153 | 154 | print '\n New hour: ', dt.strftime('%H:%M:%S %d.%m.%Y') 155 | print 'Filtering & trading...:\n' 156 | 157 | # Do possible trading 158 | # Get the last prices 159 | z, H, Trade, ERR = positions.oLastPrice() 160 | if ERR==False: 161 | kf.Filtering(z, H) 162 | 163 | # Current values of the cointegrated portfolio 164 | Z = np.append(Z,float(z-np.dot(H, kf.x_pri))) 165 | Z = Z[1:] 166 | Z_mean = np.mean(Z) 167 | Z_std = np.std(Z,ddof=1) 168 | 169 | print ' Current basket values:' 170 | print 'Basket: ', Z[-1] 171 | print 'Basket mean: ', np.mean(Z) 172 | print 'Basket STD: ', np.std(Z, ddof=1), '\n' 173 | print 'Basket short values:' 174 | print np.mean(Z)+2*np.std(Z, ddof=1) 175 | print np.mean(Z)+3*np.std(Z, ddof=1) 176 | print np.mean(Z)+4*np.std(Z, ddof=1), '\n' 177 | print 'Basket long values:' 178 | print np.mean(Z)-2*np.std(Z, ddof=1) 179 | print np.mean(Z)-3*np.std(Z, ddof=1) 180 | print np.mean(Z)-4*np.std(Z, ddof=1) 181 | 182 | # Manage positions 183 | if Trade==False: 184 | print '\n Due to error in getting prices on time,' 185 | print 'not opening or closing new positions.\n' 186 | elif Trade==True: 187 | positions.oManage(kf.x_pri[0:-1], z, H, Z[-1], 188 | Z_mean, Z_std) 189 | 190 | else: 191 | print '\nERROR IN GETTING DATA FOR THE PREVIOUS HOUR!\n' 192 | 193 | current_hour = dt.hour # Update the hour counter 194 | 195 | # If not new hour, wait a second and try again. 196 | else: time.sleep(1) 197 | 198 | # If trading has just closed, wait until Sunday 199 | else: 200 | 201 | print '\n Trading Closed!!', dt.strftime('%H:%M:%S %d.%m.%Y') 202 | time.sleep(172800) 203 | dt = datetime.datetime.now() 204 | print '\n Trading Opened!!', dt.strftime('%H:%M:%S %d.%m.%Y'), '\n' 205 | current_hour = 0 206 | 207 | except KeyboardInterrupt: 208 | 209 | print 'Pickle Trading Operations? (y/n)' 210 | while True: 211 | x = raw_input() 212 | if (x=='y'): 213 | with open('positions.pickle', 'wb') as f: 214 | pickle.dump(positions, f) 215 | print 'Trading Operations pickled!' 216 | break 217 | elif x=='n': 218 | print 'Not pickling Trading Operations!' 219 | break 220 | 221 | else: print 'Give y for yes or n for no.' 222 | 223 | print 'Pickle Kalman filter? (y/n)' 224 | while True: 225 | x = raw_input() 226 | if (x=='y'): 227 | with open('kf.pickle', 'wb') as f: 228 | pickle.dump(kf, f) 229 | print 'Kalman filter pickled!' 230 | break 231 | elif x=='n': 232 | print 'Not pickling Kalman filter!' 233 | break 234 | else: print 'Give y for yes or n for no.' 235 | 236 | dt = datetime.datetime.now() 237 | print '\n Program terminated at', dt.strftime('%H:%M:%S %d.%m.%Y'), '\n' 238 | 239 | 240 | 241 | 242 | 243 | 244 | 245 | -------------------------------------------------------------------------------- /oTradingOperations.py: -------------------------------------------------------------------------------- 1 | 2 | """ 3 | Library of all the methods for interacting with Oanda API, probably will end 4 | up being a two classes, maybe some auxiliary methods...will see... 5 | """ 6 | import time 7 | import numpy as np 8 | 9 | import oandapyV20 10 | import oandapyV20.endpoints.instruments as instruments 11 | import oandapyV20.endpoints.orders as orders 12 | import oandapyV20.endpoints.trades as trades 13 | import oandapyV20.endpoints.accounts as accounts 14 | from oandapyV20.contrib.requests import MarketOrderRequest 15 | 16 | import requests # Also for error handling 17 | 18 | class oPositionManager: 19 | 20 | def __init__(self, Account_ID, Account_Token, Instruments, I_types): 21 | 22 | #self.a_id, self.a_t = Account_ID, Account_Token 23 | self.instruments = Instruments 24 | self.I_types = I_types 25 | self.ID = Account_ID 26 | 27 | self.existing_positions = [] # List of exisitng positions ticket #'s 28 | self.long_short = '' # String noting the type of open trades 29 | self.std_away = [] # In order not to have duplicate trades... 30 | 31 | self.client = oandapyV20.API(access_token=Account_Token) 32 | 33 | 34 | 35 | # Method for getting the closing prices of the last candles 36 | def oLastPrice(self): 37 | 38 | # In case of errors 39 | Trade = True # For determining if trading is allowed with new prices 40 | ERR = False # For determining if new prices were even recieved on time 41 | 42 | i = 0 43 | H = np.empty((1,len(self.instruments))) 44 | H[0][-1] = 1.0 45 | Z = np.empty((1,1)) 46 | 47 | # For error handling 48 | error = 0 49 | waits = 0 50 | 51 | # Go through all instruments 52 | while i5) and (waits<=50): 81 | print '\n Trouble getting data, waiting a minute...\n' 82 | waits += 1 83 | Trade = False 84 | time.sleep(60) 85 | elif (error>5) and (waits>50): 86 | print ' Unable to get data for over 50 minutes!\n' 87 | ERR = True 88 | break 89 | 90 | 91 | return Z, H, Trade, ERR 92 | 93 | # Method for opening positions 94 | def oOpenPosition(self, lots): 95 | 96 | OK = True 97 | opening_positions = [] 98 | # Start opening positions 99 | for i in xrange(len(self.instruments)): 100 | order = MarketOrderRequest(instrument=self.instruments[i], 101 | units=lots[i]) 102 | request = orders.OrderCreate(self.ID, data=order.data) 103 | 104 | # Again for error handling 105 | print '\n Opening on', self.instruments[i] 106 | try: 107 | trd = self.client.request(request) 108 | opening_positions.append(trd['orderFillTransaction']['id']) 109 | print 'Success!\n' 110 | 111 | except (oandapyV20.exceptions.V20Error) as err: 112 | print 'Error in opening position! Cancelling trade!' 113 | OK = False 114 | if len(opening_positions)>0: 115 | i = 0 116 | while i0: 144 | 145 | # If we have open trades that need to be closed 146 | if (((self.long_short=='short') and (Z<=Z_mean-0.5*Z_std)) 147 | or ((self.long_short=='long') and (Z>=Z_mean+0.5*Z_std))): 148 | 149 | # Cycle through the ticket numbers and close the positions 150 | i = 0 151 | num_positions = len(self.existing_positions) 152 | while ip3)): 201 | 202 | OK = self.oOpenPosition(lots) 203 | if OK: 204 | self.std_away.append(2) # Trade at 2 std's open 205 | self.long_short = 'long' 206 | print '\n Opened long positions 2 std\'s away!\n' 207 | 208 | # If can open a position 3 std's away 209 | if (3 not in self.std_away) and ((Z<=p3) and (Z>p4)): 210 | 211 | OK = self.oOpenPosition(lots) 212 | if OK: 213 | self.std_away.append(3) # Trade at 4 std's open 214 | self.long_short = 'long' 215 | print '\n Opened long positions 3 std\'s away!\n' 216 | 217 | # If can open a position 2 std's away 218 | if (4 not in self.std_away) and (Z<=p4): 219 | 220 | OK = self.oOpenPosition(lots) 221 | if OK: 222 | self.std_away.append(4) # Trade at 4 std's open 223 | self.long_short = 'long' 224 | print '\n Opened long positions 4 std\'s away!\n' 225 | 226 | # Short trade 227 | if (Z>=Z_mean+2.0*Z_std) and (self.long_short!='long'): 228 | 229 | # Calculate first the lot sizes 230 | lots = [0]*len(self.instruments) 231 | for i in xrange(len(self.instruments)): 232 | if i==0: 233 | if self.I_types[i]==1: 234 | lots[i] = int(np.around(balance/z,0)) 235 | else: lots[i] = int(np.around(balance,0)) 236 | lots[i] = -1*lots[i] 237 | else: 238 | if self.I_types[i]==1: 239 | lots[i] = int(np.around(balance*x_pri[i-1]/H[0][i-1],0)) 240 | else: lots[i] = int(np.around(balance*x_pri[i-1],0)) 241 | 242 | p2 = Z_mean+2.0*Z_std 243 | p3 = Z_mean+3.0*Z_std 244 | p4 = Z_mean+4.0*Z_std 245 | 246 | # If can open a position 2 std's away 247 | if (2 not in self.std_away) and ((Z>=p2) and (Z=p3) and (Z=p4): 266 | 267 | OK = self.oOpenPosition(lots) 268 | if OK: 269 | self.std_away.append(4) # Trade at 4 std's open 270 | self.long_short = 'short' 271 | print '\n Opened short positions 4 std\'s away!\n' 272 | 273 | # Method for getting data for burnin, WILL DO ERROR HANDLING LATER 274 | def oGetData(self, candles): 275 | 276 | z = np.empty((candles-1, 1)) 277 | h = np.empty((candles-1, len(self.instruments))) 278 | for i in xrange(len(self.instruments)+1): 279 | if i0) and (i