├── __init__.py ├── indicators ├── __init__.py ├── regression │ ├── __init__.py │ ├── __pycache__ │ │ ├── __init__.cpython-37.pyc │ │ └── linear_regression.cpython-37.pyc │ ├── linear_regression.py │ └── mann_kendall.py └── __pycache__ │ └── __init__.cpython-37.pyc ├── optimize ├── __init__.py └── genetic_algorithm.py ├── trading_rules ├── __init__.py └── r_square_tr.py ├── entry ├── __init__.py └── entry_fibo.py ├── exit ├── __init__.py └── exit_fibo.py ├── .gitignore ├── images ├── El_wave.jpg ├── genetic.jpg ├── Crossover_.png ├── Entry_exit.png ├── Population.png ├── Selection.png ├── mutation_.png ├── Sharep_ratio.gif ├── period_split.png ├── Local_extremum.png ├── Trading_rules.png ├── copy_generation_.png ├── stationary_series.png ├── artificial-intelligence.png └── sharpe_ratio.svg ├── LICENSE.txt ├── main.py ├── init_operations.py ├── date_manip.py ├── indicator.py ├── charting.py ├── math_op.py ├── pnl.py ├── optimize_.py ├── manip_data.py ├── initialize.py └── README.md /__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /indicators/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /optimize/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /indicators/regression/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /trading_rules/__init__.py: -------------------------------------------------------------------------------- 1 | from .r_square_tr import RSquareTr -------------------------------------------------------------------------------- /entry/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Techniques used to enter the market 3 | """ -------------------------------------------------------------------------------- /exit/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | Techniques used to exit the market 3 | """ -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ipynb_checkpoints_pycache__/ 2 | idea/ 3 | __pycache__/ 4 | -------------------------------------------------------------------------------- /images/El_wave.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/El_wave.jpg -------------------------------------------------------------------------------- /images/genetic.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/genetic.jpg -------------------------------------------------------------------------------- /images/Crossover_.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/Crossover_.png -------------------------------------------------------------------------------- /images/Entry_exit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/Entry_exit.png -------------------------------------------------------------------------------- /images/Population.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/Population.png -------------------------------------------------------------------------------- /images/Selection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/Selection.png -------------------------------------------------------------------------------- /images/mutation_.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/mutation_.png -------------------------------------------------------------------------------- /images/Sharep_ratio.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/Sharep_ratio.gif -------------------------------------------------------------------------------- /images/period_split.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/period_split.png -------------------------------------------------------------------------------- /images/Local_extremum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/Local_extremum.png -------------------------------------------------------------------------------- /images/Trading_rules.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/Trading_rules.png -------------------------------------------------------------------------------- /images/copy_generation_.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/copy_generation_.png -------------------------------------------------------------------------------- /images/stationary_series.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/stationary_series.png -------------------------------------------------------------------------------- /images/artificial-intelligence.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/images/artificial-intelligence.png -------------------------------------------------------------------------------- /indicators/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/indicators/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /indicators/regression/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/indicators/regression/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /indicators/regression/__pycache__/linear_regression.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/philippe-ostiguy/PyBacktesting/HEAD/indicators/regression/__pycache__/linear_regression.cpython-37.pyc -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | 2 | The MIT License (MIT) 3 | Copyright (c) 2020 Philippe Ostiguy 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 16 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 17 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 18 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 19 | DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 20 | OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 21 | OR OTHER DEALINGS IN THE SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """ This is the main module which execute the program """ 28 | 29 | import indicators.regression.linear_regression as lr 30 | import indicators.regression.mann_kendall as mk 31 | import charting as cht 32 | import pandas as pd 33 | from optimize_ import Optimize 34 | from manip_data import ManipData as md 35 | 36 | class Main(Optimize): 37 | 38 | def __init__(self): 39 | super().__init__() 40 | super().__call__() 41 | self.cht_ = cht.Charting(self.series, self.date_name, 42 | self.default_data, **self.indicator) 43 | 44 | def chart_signal(self): 45 | """Marks signal on chart (no entry, only when the indicators trigger a signal)""" 46 | self.cht_.chart_rsquare(list(self.indicator.keys())[0],r_square_level=self.r_square_level) 47 | 48 | def chart_trigger(self): 49 | """Marks entry and exit level on chart""" 50 | 51 | mark_up = md.pd_tolist(self.trades_track, self.entry_row) 52 | mark_down = md.pd_tolist(self.trades_track, self.exit_row) 53 | marks_ = {'marker_entry': {self.marker_: '^', self.color_mark: 'g', self.marker_signal: mark_up}, 54 | 'marker_exit': {self.marker_: 'v', self.color_mark: 'r', self.marker_signal: mark_down}} 55 | 56 | self.cht_.chart_marker(self.marker_signal, self.marker_, self.color_mark,**marks_) 57 | 58 | if __name__ == '__main__': 59 | main_ = Main() 60 | #main_.chart_signal() 61 | main_.chart_trigger() 62 | t= 5 -------------------------------------------------------------------------------- /init_operations.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """Module to easily reinitiliaze values when needed """ 28 | 29 | from initialize import Initialize 30 | from manip_data import ManipData as md 31 | import pandas as pd 32 | import matplotlib.pyplot as plt 33 | 34 | class InitOp(Initialize): 35 | 36 | def __init__(self): 37 | super().__init__() 38 | super().__call__() 39 | 40 | def __call__(self): 41 | self.reset_value() 42 | 43 | def reset_value(self): 44 | """Function to reset the dictionary that contains the trading journal (entry, exit, return) in 45 | `self.trades_track` 46 | 47 | We need to do that when we optimize, ie when `self.is_walkfoward` is `True` 48 | 49 | """ 50 | 51 | self.trades_track = pd.DataFrame(columns=[self.entry_row, self.entry_level, self.exit_row, self.exit_level, \ 52 | self.trade_return]) 53 | 54 | def init_series(self): 55 | """Function that extract the data from csv to a pandas Dataframe `self.series` 56 | 57 | It actually is the data that we are using for the strategy """ 58 | 59 | self.series = md.csv_to_pandas(self.date_name, self.start_date, self.end_date, self.name, self.directory, 60 | self.asset, ordinal_name=self.date_ordinal_name, is_fx=self.is_fx, dup_col = self.dup_col) 61 | 62 | if self.is_detrend: 63 | self.series_diff = md.de_trend(self.series,self.date_name, self.date_ordinal_name,self.default_data, 64 | period=self.period,p_value= self.p_value) -------------------------------------------------------------------------------- /indicators/regression/linear_regression.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """Module that evaluates the slope and r_square of a serie""" 28 | 29 | from scipy import stats 30 | from initialize import Initialize 31 | from manip_data import ManipData as md 32 | from init_operations import InitOp as io 33 | import copy 34 | 35 | class RegressionSlopeStrenght(Initialize): 36 | """Class that evaluates the slope and r2 value of a serie 37 | 38 | Take into consideration that we use r2 to see if one variable can explain movement in the other. 39 | We are not trying to forecast using the r2 value. 40 | 41 | Parameters 42 | ---------- 43 | `self.sous_series` : pandas Dataframe 44 | Contains the subseries on which we calculate the slope and r2 value 45 | """ 46 | 47 | def __init__(self,series_,self_): 48 | super().__init__() 49 | super().__call__() 50 | new_obj = copy.deepcopy(self_) 51 | self.__dict__.update(new_obj.__dict__) 52 | io.init_series(self) 53 | del new_obj, self_ 54 | self.sous_series=md.sous_series_(series_,self.nb_data) 55 | 56 | def __store_stat(self): 57 | """Function that returns stat in a list""" 58 | 59 | return stats.linregress(self.sous_series[self.date_ordinal_name], 60 | self.sous_series[self.default_data]) 61 | 62 | def slope(self): 63 | """ Function that return the slope of a serie. 64 | La pente est la 1ième valeur retournée dans cette stats.linregress, d'où le [0] 65 | """ 66 | 67 | return self.__store_stat()[0] 68 | 69 | def r_square(self): 70 | 71 | """ 72 | La corrélation est la 3ième valeur retournée dans cette stats.linregress, d'où le [2] 73 | """ 74 | 75 | return (self.__store_stat()[2])**2 -------------------------------------------------------------------------------- /date_manip.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | from dateutil.relativedelta import relativedelta 28 | import collections 29 | 30 | class DateManip(): 31 | """Class to manipulate date""" 32 | 33 | @classmethod 34 | def __init__(cls,date): 35 | 36 | cls.date_ = date 37 | 38 | @classmethod 39 | def date_dict(cls,begin_date, end_date,**kwargs): 40 | """ Create a dictionary of dictionary containing start and end date between 2 periods 41 | 42 | Contain subdates depending on how many different subcategories we want and the lenght of the original 43 | data 44 | 45 | Parameters 46 | ---------- 47 | `begin_date` : datetime 48 | Must be in datetime format, it's the first date of the original serie 49 | 50 | `end_date` : datetime 51 | Must be in datetime format, it's the last date of the original serie 52 | 53 | `**kwargs` : kw argument 54 | Contain the lenght of subseries (date) we want to create 55 | 56 | Returns 57 | ---------- 58 | `dict_date_` : dictionary of n keys in **kwargs 59 | containing the beginning and ending date of subseries dependin 60 | """ 61 | _months = 0 62 | for value in kwargs.values(): 63 | _months+=value 64 | 65 | _test_end = begin_date + relativedelta(months = _months) 66 | _date = begin_date 67 | _count = 0 68 | _dict_date = collections.defaultdict(dict) 69 | while (_test_end < end_date): 70 | for key, value in kwargs.items(): 71 | _end_date = _date + relativedelta(months =kwargs[key]) 72 | _dict_date[_count][key] = [_date, _end_date] 73 | _date = _end_date 74 | _count +=1 75 | _test_end = _test_end + relativedelta(months = _months) 76 | return _dict_date 77 | 78 | @classmethod 79 | def end_format(cls,format_): 80 | """Return a datetime to the desired TimeStamp in str 81 | 82 | Parameters 83 | ---------- 84 | `format_` : str 85 | Format code in which we want to return the datetime 86 | """ 87 | return cls.date_.strftime(format_) 88 | 89 | -------------------------------------------------------------------------------- /indicator.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """Return the values of the indicator of our choice through the desired timeframe and interval""" 28 | import indicators.regression.linear_regression as lr 29 | import indicators.regression.mann_kendall as mk 30 | import numpy as np 31 | from manip_data import ManipData as md 32 | from init_operations import InitOp 33 | 34 | class Indicator(InitOp): 35 | 36 | def __init__(self): 37 | super().__init__() 38 | 39 | def __call__(self): 40 | super().__call__() 41 | 42 | def calcul_indicator(self): 43 | """Function that return the value of an indicator through desired period and the calculation lenght of the 44 | indicator 45 | 46 | The indicator always take into account the value of the price for the same row. 47 | Ex: We are at row 99, the indicator will take into account the data for row 99 then write the value on row 99. 48 | Basically, we have to enter or exit the market (or exit) on the next row (value) 49 | 50 | The function iterate through the indicators in `self.indicator` and through the range of `self.series`,defined 51 | in `init_operations.py` and function `init_series()`. Then it calculates the value of the indicator using 52 | the subseries `self.sous_series`. 53 | 54 | 55 | Parameters 56 | ---------- 57 | `self.series` : pandas Dataframe 58 | It contains the series used to build the model. 59 | `self.indicator` : dictionary 60 | It contains the indicator we are using for the strategy. 61 | 62 | Return 63 | ------ 64 | The function doesn't return anything in itself, but it calculates and stores the value of the desired indicator 65 | in `self.indicator` with new columns in `self.series` (pandas Dataframe) 66 | 67 | """ 68 | 69 | super().__call__() 70 | rg = lr.RegressionSlopeStrenght(self.series,self) 71 | mk_ = mk.MannKendall(self.series,self) 72 | self.indicator = {'r_square': rg, 'mk': mk_} 73 | self.point_data=0 74 | #self.slope_key=list(self.indicator.keys())[0] 75 | self.r_square_key=list(self.indicator.keys())[0] 76 | self.mk_key = list(self.indicator.keys())[1] 77 | 78 | self.point_data = 0 79 | nb_columns=len(self.series.columns) 80 | 81 | for key,value in self.indicator.items(): 82 | self.series[key] = np.nan 83 | value.point_data = 0 84 | 85 | for row in range(len(self.series.index)-self.nb_data+1): 86 | value.sous_series = md.sous_series_(self.series,self.nb_data,point_data=value.point_data) 87 | value_ = getattr(value,key)() 88 | self.series.loc[self.series.index[row]+self.nb_data-1,key]=value_ 89 | value.point_data+=1 -------------------------------------------------------------------------------- /charting.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """ Module with chart functions""" 28 | 29 | import matplotlib.pyplot as plt 30 | 31 | class Charting(): 32 | 33 | @classmethod 34 | def __init__(cls, series, x_axis, y_axis,**indicator): 35 | cls.series = series 36 | cls.x_axis = x_axis # x_axis name 37 | cls.y_axis = y_axis # y_axis name 38 | 39 | cls.indicator = indicator 40 | cls.indicator_dict = {} 41 | count = len(cls.indicator) 42 | cls.divider = .25 43 | cls.height_chart = cls.divider * (count) 44 | 45 | @classmethod 46 | def chart_rsquare(cls, r_square_name, r_square_level=.8): 47 | """ 48 | Marks the signals on a chart when r2 is above a certain level 49 | """ 50 | 51 | # When r2 is higher than desired level, we have a mark on chart 52 | first_index = cls.series.first_valid_index() 53 | tempo_mark = [] 54 | tempo_mark = cls.series.loc[cls.series[r_square_name] > r_square_level].index.tolist() 55 | cls.mark_ = [i - first_index for i in tempo_mark] 56 | 57 | def _plot(series_): 58 | """ 59 | Nested function to plot series 60 | """ 61 | 62 | # Main axe 63 | fig = plt.figure() 64 | fig.set_size_inches((40, 32)) 65 | candle = fig.add_axes((0, cls.height_chart, 1, 0.5)) 66 | 67 | count = 1 68 | for key, _ in cls.indicator.items(): 69 | cls.indicator_dict[key] = fig.add_axes((0, cls.height_chart - cls.divider * count, 1, 0.2), 70 | sharex=candle) 71 | count += 1 72 | 73 | candle.plot(cls.x_axis, cls.y_axis, markevery=cls.mark_, marker="o", data=series_) 74 | for key, _ in cls.indicator.items(): 75 | cls.indicator_dict[key].plot(cls.x_axis, key, data=cls.series) 76 | 77 | _plot(cls.series) 78 | t = 5 79 | 80 | 81 | @classmethod 82 | def chart_marker(cls, marker_signal, marker_, color_mark, **marker): 83 | """ 84 | Method to plot a chart with marker 85 | Parameters 86 | ---------- 87 | **marker : keyword arguments 88 | contains the place we want mark, the color of the mark and the type of mark 89 | """ 90 | 91 | cls.series.set_index(cls.x_axis, inplace=True) 92 | fig = plt.figure() 93 | 94 | plt.plot(cls.series.index, cls.y_axis, color='b', data=cls.series) 95 | for key, _ in marker.items(): 96 | plt.plot(cls.series.index, cls.y_axis, markevery=marker[key][marker_signal], marker= \ 97 | marker[key][marker_], markersize=10, markerfacecolor=marker[key][color_mark], data=cls.series) 98 | 99 | plt.plot(cls.series.index, cls.y_axis, color='b', data=cls.series) -------------------------------------------------------------------------------- /math_op.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """ 28 | Module support for mathematical operations 29 | """ 30 | from scipy.signal import argrelextrema 31 | import matplotlib.pyplot as plt 32 | import numpy as np 33 | import pandas as pd 34 | 35 | class MathOp(): 36 | """ 37 | Class to provide mathematical operation support 38 | """ 39 | 40 | @classmethod 41 | def __init__(cls,series,default_col): 42 | cls.series = series 43 | cls.default_col=default_col 44 | 45 | @classmethod 46 | def local_extremum(cls,start_point,end_point,window = 6,min_= 'min',max_='max',index_ = 'index'): 47 | """ Function to find local extremum (min and max) on a Dataframe. 48 | 49 | It checks if the values for a number of points on each side (`window`) are greater or lesser and then 50 | determine the local extremum. 51 | 52 | Parameters 53 | ---------- 54 | start_point : int 55 | the first data (index) to check in the Dataframe 56 | end_point : int 57 | the last data (index) to check in the Dataframe 58 | window : int 59 | the number of the data the method check before and after to determine the local extremum. Default is 6. 60 | Recommended values are between 5 and 7. 61 | min_ : str 62 | Name given to min data column 63 | max_ : str 64 | Name given to max data column 65 | 66 | Return 67 | ------ 68 | DataFrame list : Return a pandas dataframe `cls.series` with the none empty min or max value 69 | (if both are empty, nothing is returned. If one of 70 | them has a value, return the local min or max with index no) 71 | """ 72 | 73 | cls.series=cls.series.loc[start_point:end_point,cls.default_col] 74 | cls.series=pd.DataFrame({cls.default_col: cls.series}) 75 | 76 | cls.series[min_] = cls.series.iloc[argrelextrema(cls.series.values, np.less_equal, 77 | order=window)[0]][cls.default_col] 78 | cls.series[max_] = cls.series.iloc[argrelextrema(cls.series.values, np.greater_equal, 79 | order=window)[0]][cls.default_col] 80 | cls.series[index_] = cls.series.index 81 | 82 | # Plot results - to get ride when the project is done. Only as a guideline at the moment 83 | 84 | """ 85 | plt.scatter(cls.series.index, cls.series[min_], c='r') 86 | plt.scatter(cls.series.index, cls.series[max_], c='g') 87 | 88 | plt.plot(cls.series.index, cls.series[cls.default_col]) 89 | plt.ion() 90 | plt.show() 91 | """ 92 | 93 | #Filter nan value for min or max out 94 | cls.series=cls.series.loc[(cls.series[min_].isna())==False | (cls.series[max_].isna() == False)] 95 | 96 | return cls.series -------------------------------------------------------------------------------- /pnl.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """Module to assess the trading strategy performance""" 28 | 29 | import trading_rules as tr 30 | import numpy as np 31 | import math 32 | from date_manip import DateManip 33 | 34 | class PnL(tr.RSquareTr): 35 | 36 | def __init__(self): 37 | super().__init__() 38 | 39 | def pnl_(self): 40 | """Function that calculate the different metrics to evalute the trading strategy performance""" 41 | super().__call__() 42 | self.diff_ = ((self.end_date - self.start_date).days / 365) #diff in term of year with decimal 43 | self.pnl_dict[self.range_date_] = self.range_date() 44 | self.pnl_dict[self.ann_return_] = self.ann_return() 45 | self.pnl_dict[self.ann_vol_] = self.ann_vol() 46 | self.pnl_dict[self.sharpe_ratio_] = self.sharpe_ratio() 47 | self.pnl_dict[self.max_draw_] = self.max_draw() 48 | self.pnl_dict[self.pour_win_] = self.pour_win() 49 | self.pnl_dict[self.nb_trades_] = self.nb_trades() 50 | 51 | #Possible to have some trades but not real trades (0 return) when largest_extension is 0 52 | if (self.pnl_dict[self.nb_trades_] != None): 53 | if (self.pnl_dict[self.nb_trades_] > 0): 54 | if self.pnl_dict[self.sharpe_ratio_] is None or math.isnan(self.pnl_dict[self.sharpe_ratio_]): 55 | self.pnl_dict = {} 56 | 57 | def annualized_(func): 58 | """Decorator to return annualized value""" 59 | def wrap_diff(self): 60 | return ((1+func(self))**(1/self.diff_)-1) 61 | return wrap_diff 62 | 63 | @annualized_ 64 | def ann_return(self): 65 | """Calculate the annualized return""" 66 | return_ = 0 67 | for index_ in self.trades_track.index: 68 | return_ = (1+return_)*(1+self.trades_track.loc[index_,self.trade_return]) - 1 69 | return return_ 70 | 71 | def ann_vol(self): 72 | """Calculate annualized vol 73 | """ 74 | 75 | vol_ = self.trades_track[self.trade_return].std() 76 | if not np.isnan(vol_): 77 | return (vol_ * math.sqrt(1/self.diff_)) 78 | else : 79 | return None 80 | 81 | def sharpe_ratio(self): 82 | """Sharpe ratio 83 | 84 | Not using the risk-free rate has it doesn't change the final result. We could trade on margin and just 85 | totally distort the return. Also, depending on the time intervals, the return are larger or smaller 86 | (expected higher volatility on daily than hourly basis). 87 | """ 88 | if not bool(self.pnl_dict): 89 | return None 90 | 91 | if self.pnl_dict[self.ann_vol_] == None: 92 | return None 93 | 94 | elif ((self.pnl_dict[self.ann_vol_] == 0) | np.isnan(self.pnl_dict[self.ann_vol_])): 95 | return None 96 | else : 97 | return (self.pnl_dict[self.ann_return_] /self.pnl_dict[self.ann_vol_]) 98 | 99 | def max_draw(self): 100 | """Return lowest return value """ 101 | 102 | return self.trades_track[self.trade_return].min() 103 | 104 | def nb_trades(self): 105 | """Return the number of trades""" 106 | return self.trades_track.shape[0] 107 | 108 | def range_date(self): 109 | """ Return the range date tested in a desired format 110 | 111 | Using "%Y-%m-%d" as Timestamp format 112 | """ 113 | dm_begin_ = DateManip(self.start_date).end_format(self.end_format_) 114 | dm_end_ = DateManip(self.end_date).end_format(self.end_format_) 115 | return f"{dm_begin_} to {dm_end_}" 116 | 117 | def pour_win(self): 118 | """Return the percentage of winning trades 119 | """ 120 | 121 | total_trade = self.nb_trades() 122 | pour_win_ = self.trades_track[self.trades_track[self.trade_return] >= 0].shape[0] 123 | return 0 if total_trade == 0 else (pour_win_ / total_trade) -------------------------------------------------------------------------------- /optimize_.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """Module that runs the optimization process if desired""" 28 | 29 | from pnl import PnL 30 | from manip_data import ManipData as md 31 | from date_manip import DateManip as dm 32 | from optimize.genetic_algorithm import GenAlgo as ga 33 | import numpy as np 34 | import matplotlib.pyplot as plt 35 | import pandas as pd 36 | 37 | class Optimize(PnL): 38 | 39 | def __init__(self): 40 | """Function that initializes stuff. 41 | 42 | It resets the `self.series` with `self.init_series()` and dictionary that track the pnl 43 | `self.trades_track` with `reset_value()` when we optimize. 44 | """ 45 | super().__init__() 46 | self.init_series() 47 | self.reset_value() 48 | self.params = {} 49 | 50 | def __call__(self): 51 | """Function do different things dependent if we optimize or not""" 52 | 53 | if self.is_walkfoward: 54 | self.walk_foward() 55 | else : 56 | self.calcul_indicator() 57 | self.pnl_() 58 | md.write_csv_(self.dir_output, self.name_out, add_doc="", is_walkfoward=self.is_walkfoward, **self.pnl_dict) 59 | 60 | def walk_foward(self): 61 | """Function that do the walk-foward analysis (optimization). 62 | 63 | First it runs through the divided period (1 period interval for training and testing). We have to choose 64 | properly `self.start_date` and `self.end_date` as they set the numbers of period. 65 | 66 | Then the program runs through each training and testing period (in `self.dict_name_`). The program optimizes 67 | only in the training period `self.training_name_`. The results are store in the folder results and 68 | results_training for the training period and results_test for the testing period. 69 | 70 | Parameters 71 | ---------- 72 | `self.start_date` : datetime object 73 | Set in `initialize.py`. Beginning date of training and testing. 74 | `self.end_date` : datetime object 75 | Set in `initialize.py`. End date of training and testing. 76 | 77 | """ 78 | 79 | md_ = md 80 | 81 | _first_time = True 82 | self.dict_date_ = dm.date_dict(self.start_date, self.end_date, 83 | **self.dict_name_) 84 | 85 | if (len(self.dict_date_)) == 0: 86 | raise Exception("Total period not long enough for optimization") 87 | 88 | for key,_ in self.dict_date_.items(): 89 | for key_, _ in self.dict_name_.items(): 90 | 91 | self.start_date = self.dict_date_[key][key_][0] 92 | self.end_date = self.dict_date_[key][key_][1] 93 | if _first_time : 94 | md_(self.dir_output,self.name_out,extension = key_).erase_content() 95 | self.init_series() 96 | self.calcul_indicator() 97 | if key_ == self.training_name_: #we only optimize for the training period 98 | self.optimize_param() 99 | self.pnl_dict,self.params = ga(self).__call__() 100 | else : #test period, we use the optimized parameters in the training period 101 | self.assign_value() 102 | self.pnl_() 103 | 104 | md.write_csv_(self.dir_output, self.name_out, add_doc=key_, 105 | is_walkfoward=self.is_walkfoward, **self.pnl_dict) 106 | md.write_csv_(self.dir_output, self.name_out, add_doc=key_, 107 | is_walkfoward=self.is_walkfoward, **self.params) 108 | 109 | _first_time = False 110 | 111 | def assign_value(self): 112 | """ Function to assign the value to each optimized parameters obtained in the optimization module. 113 | 114 | The `genetic_algorithm.py` return the dictionary with the value and when we test in `r_square_tr.py` they are 115 | in a different format""" 116 | 117 | for item in range(len(self.op_param)): 118 | if len(self.op_param[item]) > 1: 119 | self.op_param[item][0][self.op_param[item][1]] = self.params[self.op_param[item][1]] 120 | else: 121 | setattr(self, self.op_param[item][0], self.params[self.op_param[item][0]]) -------------------------------------------------------------------------------- /trading_rules/r_square_tr.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """Module that detect buy and sell signal with r_square and mk (Mann Kendall)""" 28 | 29 | import indicator as ind 30 | import exit.exit_fibo as exf 31 | from math_op import MathOp as mo 32 | from manip_data import ManipData as md 33 | import pandas as pd 34 | import copy 35 | 36 | 37 | class RSquareTr(ind.Indicator): 38 | """Class that trigger entry signal based on r_square and Menn Kendall 39 | 40 | The signal is trigggered if `r_value` is above `self.r_square_level` define in `initialize.py` and if `mk_`value 41 | is 1 (buying signal) or -1 (selling signal)""" 42 | 43 | def __init__(self): 44 | super().__init__() 45 | 46 | def __call__(self): 47 | super().__call__() 48 | self.last_long = self.nb_data #last time we had a long signal 49 | self.last_short = self.nb_data #last time we had a short signal 50 | self.trig_signal() 51 | 52 | def trig_signal(self): 53 | """Function that iterates through each data of the selected serie `self.series` to check if there is a 54 | signal. 55 | 56 | `mk_value` and `r_value` are the evaluated value to check if there is a signal. When there is a signal in a 57 | direction, the system needs to run through a minimum of `self.min_data` (defines or optimize in `initialize.py`) 58 | to enter in the market in the same direction. 59 | 60 | The function call the `exit_fibo.py` module anytime there is a signal which will then try to enter and exit the 61 | market. It could be set to another entry and exit types. 62 | 63 | Parameters 64 | ---------- 65 | `self.r_square_level` : float 66 | If the current r2 is above this level, it means we have a trend. Important to set the good level, because 67 | it's one of the conditions that trigger a signal. It's set in `initialize.py` 68 | `self.min_data` : int 69 | If there is a signal, it's the minimum number of data needed before it can trigger another signal 70 | in the same direction 71 | 72 | Return 73 | ------ 74 | The function doesn't return anything in itself, but it stores the trading entry and exit in dictionary 75 | `self.trades_track` 76 | """ 77 | 78 | 79 | buy_signal = False 80 | sell_signal = False 81 | init_ = copy.deepcopy(self) 82 | 83 | for row in range(len(self.series)-self.nb_data+1): 84 | curr_row=row + self.nb_data-1 85 | 86 | mk_value=self.series.loc[curr_row, self.mk_key] 87 | r_value=self.series.loc[curr_row,self.r_square_key] 88 | #Buy signal 89 | if mk_value > 0 : 90 | if r_value > self.r_square_level: 91 | if self.last_long >= self.min_data : 92 | buy_signal = True 93 | self.last_short = self.min_data 94 | trades_track = exf.ExitFibo(init_).__call__(curr_row=curr_row,buy_signal=buy_signal) 95 | self.trades_track = self.trades_track.append(trades_track,ignore_index = True) 96 | self.last_long = 0 97 | 98 | #Sell signal 99 | if mk_value < 0 : 100 | if r_value > self.r_square_level: 101 | self.last_short -= 1 102 | if self.last_short >= self.min_data : 103 | sell_signal=True 104 | self.last_long = self.min_data 105 | trades_track = exf.ExitFibo(init_).__call__(curr_row=curr_row,sell_signal=sell_signal) 106 | self.trades_track = self.trades_track.append(trades_track,ignore_index = True) 107 | self.last_short=0 108 | 109 | buy_signal = False 110 | sell_signal = False 111 | 112 | self.last_long += 1 113 | self.last_short += 1 114 | 115 | #Check if there is a row with no entry or exit signal 116 | del init_ 117 | try: 118 | md.nan_list(md.pd_tolist(self.trades_track, self.entry_row)) 119 | except: 120 | raise Exception("Nan value in entry row") 121 | 122 | try: 123 | md.nan_list(md.pd_tolist(self.trades_track, self.exit_row)) 124 | except: 125 | raise Exception("Nan value in exit row") 126 | -------------------------------------------------------------------------------- /images/sharpe_ratio.svg: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /manip_data.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """Helper module to manipulate csv and pandas Dataframe""" 28 | import csv 29 | import pandas as pd 30 | import datetime as dt 31 | from functools import wraps 32 | from statsmodels.tsa.stattools import adfuller 33 | import os.path 34 | import numpy as np 35 | 36 | def ordinal_date(function): 37 | """Wrapper to add an ordinal date""" 38 | @wraps(function) 39 | def wrapper(cls,date_name,date_debut,date_fin, name_,directory,asset, ordinal_name,is_fx,dup_col): 40 | series_ = function(cls,date_name,date_debut,date_fin, name_,directory,asset, ordinal_name,is_fx,dup_col) 41 | series_.Date = pd.to_datetime(series_.Date) 42 | series_[ordinal_name] = pd.to_datetime(series_[date_name]).map(dt.datetime.toordinal) 43 | return series_ 44 | 45 | return wrapper 46 | 47 | class ManipData(): 48 | """Class to manipulate data""" 49 | 50 | @classmethod 51 | def __init__(cls,dir_,file_name,extension =""): 52 | cls.dir_ = dir_ #directory 53 | cls.filename = file_name 54 | cls.extension_ = extension #if there is an extension added to the filename 55 | 56 | @classmethod 57 | def write_csv_(cls, dir_output, name_out, add_doc = "", is_walkfoward = False, **kwargs): 58 | """ Write data to a csv 59 | 60 | Parameters 61 | ---------- 62 | dir_output : str 63 | directory where we want our data to be written 64 | name_out : str 65 | name of the file name 66 | is_walkfoward : bool 67 | says if we are doing a walkfoward analyis. If `True`, we have to create a separate training and test file 68 | **kwargs : keyword param 69 | dictionary with keys and items to be written in the file 70 | """ 71 | 72 | if is_walkfoward: 73 | write_type = 'a' 74 | func = 'writer.writerow' 75 | else : 76 | write_type = 'w' 77 | func = 'str' 78 | 79 | with open(dir_output + name_out + add_doc + ".csv" , write_type, newline='') as f: 80 | writer = csv.writer(f) 81 | eval(func)('') 82 | for key, item in kwargs.items(): 83 | writer.writerow([key,item]) 84 | 85 | 86 | @classmethod 87 | def erase_content(cls): 88 | """Method to erase contents of a csv file""" 89 | filename = cls.dir_ + cls.filename + cls.extension_ + ".csv" 90 | if os.path.isfile(filename): 91 | with open(filename,"w+") as f: 92 | f.close() 93 | 94 | @classmethod 95 | @ordinal_date 96 | def csv_to_pandas(cls, date_name,date_debut,date_fin, name_,directory,asset, ordinal_name = '',is_fx = False, 97 | dup_col = None): 98 | """Return the csv to a pandas Dataframe 99 | 100 | The function remove nan value with `series_.dropna()` and remove the data when the market is closed with 101 | `series_.drop_duplicates()` 102 | """ 103 | 104 | if is_fx: 105 | dateparse = lambda x: dt.datetime.strptime(x, '%d.%m.%Y %H:%M:%S') 106 | else : 107 | dateparse = None 108 | series_ = pd.DataFrame() 109 | _series = pd.read_csv(directory 110 | + asset + '.csv', usecols=list(name_.columns),parse_dates=[date_name], 111 | date_parser=dateparse) 112 | series_ =_series.loc[(_series[date_name] >= date_debut) & (_series[date_name] < date_fin)] 113 | if series_.empty: 114 | raise Exception("Desired date range not available in the current files") 115 | 116 | series_ = series_.dropna() #drop nan values 117 | if dup_col != None: 118 | #If all values in column self.dup_col are the same, we erase them 119 | series_ = series_.drop_duplicates(keep=False,subset=list(dup_col.keys())) 120 | series_=series_.reset_index(drop=True) 121 | 122 | return series_ 123 | 124 | @classmethod 125 | def sous_series_(cls,series_,nb_data,point_data=0): 126 | """Returns a sub-series to calculate the value of the indicator with in a precise""" 127 | 128 | cls.sous_series=series_.iloc[point_data:point_data + nb_data,:] 129 | if nb_data > len(series_): 130 | raise Exception("Not enough necessary data to calculate the indicator") 131 | return cls.sous_series 132 | 133 | @classmethod 134 | def de_trend(cls, series, date_name, date_ordinal_name, default_data, period =1, p_value = .05): 135 | """Remove the trend from the series by differentating the current serie 136 | 137 | First value is set to 0 to avoid error (of the differentiated serie) 138 | 139 | Parameters 140 | --------- 141 | `period` : int 142 | Number of periods used for differencing. Default : first difference 143 | 144 | Return 145 | ------ 146 | `series_diff` : Pandas Dataframe 147 | The stationary series 148 | 149 | """ 150 | 151 | series_diff = series.copy() 152 | series_diff.drop([date_name, date_ordinal_name], axis=1, inplace=True) 153 | series_diff = series_diff.diff(periods=period) # differencing with previous row 154 | series_diff.loc[:(period - 1), :] = 0 #Make first row equal to 0 155 | series_diff.insert(0, date_name, series[date_name]) #re-insert the peiod columns 156 | series_diff[date_ordinal_name] = series[date_ordinal_name] 157 | if adfuller(series_diff[default_data])[1] > p_value: 158 | raise Exception("The series is not stationary") 159 | return series_diff 160 | 161 | @classmethod 162 | def nan_list(cls,list_): 163 | """Check if a list has one empty value 164 | 165 | Return 166 | ------ 167 | Bool : `True` or `False` 168 | Return `True` if at least one value in the list is `nan` and `False otherwise 169 | """ 170 | 171 | return True if True in np.isnan(list_) else False 172 | 173 | @classmethod 174 | def pd_tolist(cls,pd_, row_name): 175 | """Transform a pandas column to a list. It makes sure it is an integer""" 176 | pd__ = pd_.loc[:, row_name].tolist() 177 | try: 178 | t = [int(i) for i in pd__] 179 | except: 180 | raise Exception("Mistake happened in pd_tolist") 181 | else : 182 | return [int(i) for i in pd__] -------------------------------------------------------------------------------- /exit/exit_fibo.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """Module that will try to exit the market when we are in """ 28 | 29 | import entry.entry_fibo as ef 30 | import sys 31 | import operator as op 32 | import math 33 | import copy 34 | import pandas as pd 35 | 36 | class ExitFibo(ef.EntFibo): 37 | """ Class that uses Fibonnacci strategy to exit the market. 38 | 39 | It first uses the `EntFibo()` class in `entry_fibo.py` to enter in the market. 40 | """ 41 | 42 | def __init__(self,init_): 43 | 44 | new_obj = init_ 45 | self.__dict__.update(new_obj.__dict__) #replacing self object with Initialise object 46 | del init_,new_obj 47 | self.trades_track_ = pd.DataFrame(columns=[self.entry_row, self.entry_level, self.exit_row, self.exit_level, \ 48 | self.trade_return]) 49 | 50 | def __call__(self,curr_row,buy_signal=False,sell_signal=False): 51 | """ Method that will first try to enter the market with `self.ent_fibo` then it will try to exit with 52 | `self.try_exit()` whenever we have a position 53 | """ 54 | 55 | super().__init__() 56 | self.ent_fibo(curr_row=curr_row, buy_signal=buy_signal, sell_signal=sell_signal) 57 | return self.try_exit() 58 | 59 | def try_exit(self): 60 | """ 61 | This is the method that tries to exit the market when we have a position with `entry_fibo.py` 62 | 63 | We first get an entry confirmation with the function `self.ent_fibo()` in `entry_fibo.py`. Then we run through 64 | the remaining data (according to the determined data range). We have an profit, stop loss and the method 65 | even tighten the stop under certain circumstances. 66 | 67 | Then it tries to exit the market using Fibonacci retracement and extension. 1 type at the moment: 68 | 1- Largest extension `self.largest_extension_` from the current trend. The `self.largest_extension_` is set 69 | in `entry_fibo.py`. It is in fact the largest setback in the current trend. This method uses 70 | `self.profit_ext` to calculate the profit level and `self.stop_ext` for the stop level 71 | 72 | There is no slippage included in `try_exit()`. If the price reached the desired level, we just exit at 73 | either the current price or the next desired price 74 | 75 | Parameters 76 | ---------- 77 | `self.profit_ext` : float 78 | % of the largest extension from previous trend that the system uses to exit the market to take profit 79 | Default value is 2.618. Possible values are 1.618, 2 , 2.618, 3.382, 4.236. 80 | `self.stop_ext` : float 81 | % of the largest extension from previous trend that the system uses as a stop loss. 82 | Default value is 1.618. Possible values are 1, 1.382, 1.618, 2. 83 | `self.is_entry` : bool 84 | The value comes from `entry_fibo.py`. It says if we have a position. 85 | 86 | Notes 87 | ----- 88 | The stops may be tightened (see "stop tightening" in `initialize.py`) 89 | 90 | The system doesn't check on a shorter time frame if it reaches an exit point and a stop in `try_exit()` 91 | in case of high volatility. Really rare cases 92 | 93 | """ 94 | 95 | #If no entry signal, exit the function 96 | if not self.is_entry: 97 | return None 98 | 99 | _entry_level = self.trades_track_.iloc[-1, self.trades_track_.columns.get_loc(self.entry_level)] 100 | 101 | #stop tightening using extension 102 | _is_stop_ext = self.stop_tight_dict[self.stop_tight_ret][self.is_true] 103 | if _is_stop_ext: 104 | _extension_tight = self.inv * op.sub(self.relative_extreme,self.extreme[self.fst_data]) 105 | if _extension_tight < 0: 106 | _is_stop_ext = False #can happen in case of high volatility 107 | _extension_stop = _extension_tight * self.stop_tight_dict[self.stop_tight_ret][self.stop_ret_level] 108 | 109 | if self.stop_tight_dict[self.stop_tight_ret][self.default_data_]: 110 | _data_stop = self.default_data 111 | else : 112 | _data_stop = self.stop 113 | 114 | _is_stop_pour = self.stop_tight_dict[self.stop_tight_pour][self.is_true] 115 | if _is_stop_pour : 116 | _tight_value = self.stop_tight_dict[self.stop_tight_pour][self.tight_value] 117 | _pour_tight = self.stop_tight_dict[self.stop_tight_pour][self.pour_tight] 118 | 119 | #Check if the first row (where the signal is trigerred) is already below the stop loss (for buy) 120 | # and vice versa for sell signal. If yes, stop loss trigerred 121 | if self.exit_dict[self.exit_name][self.exit_ext_bool] & \ 122 | self.six_op(self.series.loc[self.curr_row,self.stop],self.stop_value): 123 | self.is_entry = False 124 | self.trades_track_.iloc[-1, self.trades_track_.columns.get_loc(self.exit_row)] = self.curr_row 125 | self.trades_track_.iloc[-1, self.trades_track_.columns.get_loc(self.exit_level)] = \ 126 | self.stop_value # exit level 127 | self.trades_track_.iloc[-1, self.trades_track_.columns.get_loc(self.trade_return)] = \ 128 | self.inv * ((_entry_level - self.stop_value) / _entry_level) 129 | return self.trades_track_ 130 | 131 | data_test = len(self.series) - self.curr_row - 1 132 | 133 | #This is the part where we run through the data to try to exit the market 134 | for curr_row_ in range(data_test): 135 | 136 | self.curr_row += 1 137 | _curent_value = self.series.loc[self.curr_row, self.default_data] #curent value with default data type 138 | _current_stop = self.series.loc[self.curr_row, self.stop] #current stop value with data stop type 139 | 140 | #Profit can change if relative_extreme changes 141 | if self.exit_dict[self.exit_name][self.exit_ext_bool]: 142 | _profit_value = self.fth_op(self.relative_extreme, self.extension_profit) 143 | 144 | #Value that will make tighten the stop using extension 145 | if _is_stop_ext: 146 | _tight_trig = self.fth_op(self.relative_extreme,_extension_stop) 147 | 148 | # Value that will make tighten the stop using pourcentage 149 | if _is_stop_pour: 150 | _tight_pour_trig = _tight_value * (_profit_value - _entry_level) + _entry_level 151 | _tight_pour_level = _pour_tight *(_curent_value - _entry_level) + _entry_level 152 | 153 | #Stop loss trigerred? 154 | if self.exit_dict[self.exit_name][self.exit_ext_bool] & \ 155 | self.six_op(_current_stop, self.stop_value): 156 | self.is_entry = False 157 | self.trades_track_.iloc[-1,self.trades_track_.columns.get_loc(self.exit_row)] = self.curr_row 158 | self.trades_track_.iloc[-1, self.trades_track_.columns.get_loc(self.exit_level)] = \ 159 | self.stop_value #exit level 160 | self.trades_track_.iloc[-1, self.trades_track_.columns.get_loc(self.trade_return)] = \ 161 | self.inv*((_entry_level-self.stop_value)/_entry_level) 162 | break 163 | 164 | #Changing relative low (for a buying), vice versa 165 | if self.sec_op(_curent_value, self.relative_extreme): 166 | self.relative_extreme = _curent_value 167 | self.row_rel_extreme = self.curr_row 168 | 169 | #Changing stop tightening level 170 | if _is_stop_ext: 171 | _extension_tight = self.inv * op.sub(self.relative_extreme, self.extreme[self.fst_data]) 172 | if _extension_tight < 0: #does happen in case of high volatility 173 | _is_stop_ext = False 174 | _extension_stop = _extension_tight * self.stop_tight_dict[self.stop_tight_ret][self.stop_ret_level] 175 | 176 | #Taking profit 177 | if self.exit_dict[self.exit_name][self.exit_ext_bool] & self.fif_op(_curent_value,_profit_value): 178 | self.is_entry = False 179 | self.trades_track_.iloc[-1, self.trades_track_.columns.get_loc(self.exit_row)] = self.curr_row 180 | self.trades_track_.iloc[-1, self.trades_track_.columns.get_loc(self.exit_level)] = \ 181 | _profit_value # exit level 182 | self.trades_track_.iloc[-1, self.trades_track_.columns.get_loc(self.trade_return)] = \ 183 | self.inv * ((_entry_level - _profit_value) / _entry_level ) 184 | break 185 | 186 | #Tightening the stop using extension 187 | if _is_stop_ext: 188 | if self.fst_op(_curent_value,_tight_trig): 189 | if self.fst_op(self.series.loc[self.row_rel_extreme, _data_stop],self.stop_value): 190 | self.stop_value = self.series.loc[self.row_rel_extreme, _data_stop] 191 | 192 | #Tightening using percentage reached 193 | if _is_stop_pour : 194 | if self.fst_op(_curent_value,_tight_pour_trig) & \ 195 | (self.fst_op(_tight_pour_level, self.stop_value)): 196 | self.stop_value = _tight_pour_level 197 | 198 | if math.isnan(self.trades_track_.iloc[-1, self.trades_track_.columns.get_loc(self.exit_level)]): 199 | self.trades_track_ = self.trades_track_[:-1] 200 | 201 | return self.trades_track_ 202 | 203 | -------------------------------------------------------------------------------- /indicators/regression/mann_kendall.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Created on Wed Jul 29 09:16:06 2015 4 | @author: Michael Schramm 5 | Modified on August 5th 2020 6 | by : Philippe Ostiguy 7 | """ 8 | from __future__ import division 9 | import numpy as np 10 | from scipy.stats import norm 11 | from manip_data import ManipData as md 12 | from initialize import Initialize 13 | from init_operations import InitOp as io 14 | import copy 15 | 16 | class MannKendall(Initialize): 17 | """ 18 | Mann-Kendall is a non-parametric test to determine if there is a trend in a time-series 19 | """ 20 | 21 | def __init__(self,series_,self_,alpha=0.01,iteration=True): 22 | super().__init__() 23 | super().__call__() 24 | new_obj = copy.deepcopy(self_) 25 | self.__dict__.update(new_obj.__dict__) 26 | del self_, new_obj 27 | io.init_series(self) 28 | self.alpha=alpha 29 | self.first_iteration=iteration 30 | self.nb_sign=0 31 | 32 | self.sous_series =md.sous_series_(series_,self.nb_data) 33 | self.series_mk = series_ 34 | 35 | def mk(self): 36 | 37 | """ 38 | I'm not the original writer of this function, it comes from github : 39 | 40 | https://github.com/mps9506/Mann-Kendall-Trend/blob/master/mk_test.py 41 | 42 | The goal here is to calculate the Mann Kendall value at data point, so to 43 | save time, we just substract the first value and add the last value when 44 | we go to a new data point. 45 | 46 | This function is derived from code originally posted by Sat Kumar Tomer 47 | (satkumartomer@gmail.com) 48 | See also: http://vsp.pnnl.gov/help/Vsample/Design_Trend_Mann_Kendall.htm 49 | 50 | The purpose of the Mann-Kendall (MK) test (Mann 1945, Kendall 1975, Gilbert 51 | 1987) is to statistically assess if there is a monotonic upward or downward 52 | trend of the variable of interest over time. A monotonic upward (downward) 53 | trend means that the variable consistently increases (decreases) through 54 | time, but the trend may or may not be linear. The MK test can be used in 55 | place of a parametric linear regression analysis, which can be used to test 56 | if the slope of the estimated linear regression line is different from 57 | zero. The regression analysis requires that the residuals from the fitted 58 | regression line be normally distributed; an assumption not required by the 59 | MK test, that is, the MK test is a non-parametric (distribution-free) test. 60 | Hirsch, Slack and Smith (1982, page 107) indicate that the MK test is best 61 | viewed as an exploratory analysis and is most appropriately used to 62 | identify stations where changes are significant or of large magnitude and 63 | to quantify these findings. 64 | 65 | By default, it is a two-side test 66 | 67 | Input: 68 | x: a vector of data 69 | alpha: significance level (0.01 default) 70 | Output: 71 | trend: tells the trend (increasing, decreasing or no trend) 72 | h: True (if trend is present) or False (if trend is absence) 73 | p: p value of the significance test 74 | z: normalized test statistics 75 | 76 | Return value : -1 if there is a negative trend (at the significance level) 77 | +1 if there is positive trend (at the significance level) 78 | 79 | """ 80 | sous_series_ = self.sous_series.loc[:,self.default_data] 81 | n = len(sous_series_) 82 | 83 | # calculate positive and negative sign 84 | if self.first_iteration: 85 | for k in range(n-1): 86 | for j in range(k+1, n): 87 | self.nb_sign += np.sign(sous_series_.values[j] - sous_series_.values[k]) 88 | 89 | # if we iterate through time, we use previous calculation and add new value and substract old value 90 | else: 91 | for k in range(n-1): 92 | self.nb_sign += np.sign(sous_series_.values[n-1] - sous_series_.values[k]) 93 | 94 | self.sous_series= md.sous_series_(self.series_mk,self.nb_data,point_data=self.point_data-1) 95 | 96 | sous_series_= self.sous_series.loc[:,self.default_data] 97 | n = len(sous_series_) 98 | for k in range(n-1): 99 | self.nb_sign -= np.sign(sous_series_.values[k+1] - sous_series_.values[0]) 100 | 101 | self.first_iteration = False 102 | 103 | # calculate the unique data 104 | unique_x, tp = np.unique(sous_series_.values, return_counts=True) 105 | g = len(unique_x) 106 | 107 | # calculate the var(s) 108 | if n == g: # there is no tie 109 | var_s = (n*(n-1)*(2*n+5))/18 110 | else: # there are some ties in data 111 | var_s = (n*(n-1)*(2*n+5) - np.sum(tp*(tp-1)*(2*tp+5)))/18 112 | 113 | if self.nb_sign > 0: 114 | z = (self.nb_sign - 1)/np.sqrt(var_s) 115 | elif self.nb_sign < 0: 116 | z = (self.nb_sign + 1)/np.sqrt(var_s) 117 | else: # self.nb_sign == 0: 118 | z = 0 119 | 120 | # calculate the p_value 121 | p = 2*(1-norm.cdf(abs(z))) # two tail test 122 | h = abs(z) > norm.ppf(1-self.alpha/2) 123 | 124 | if (z < 0) and h: 125 | trend = -1 126 | elif (z > 0) and h: 127 | trend = 1 128 | else: 129 | trend = 0 130 | 131 | # return +1 if there a positive trend, -1 if there a negative trend and 0 if none. 132 | return trend 133 | 134 | 135 | def check_num_samples(beta, delta, std_dev, alpha=0.05, n=4, num_iter=1000, 136 | tol=1e-6, num_cycles=10000, m=5): 137 | """ 138 | This function is an implementation of the "Calculation of Number of Samples 139 | Required to Detect a Trend" section written by Sat Kumar Tomer 140 | (satkumartomer@gmail.com) which can be found at: 141 | http://vsp.pnnl.gov/help/Vsample/Design_Trend_Mann_Kendall.htm 142 | As stated on the webpage in the URL above the method uses a Monte-Carlo 143 | simulation to determine the required number of points in time, n, to take a 144 | measurement in order to detect a linear trend for specified small 145 | probabilities that the MK test will make decision errors. If a non-linear 146 | trend is actually present, then the value of n computed by VSP is only an 147 | approximation to the correct n. If non-detects are expected in the 148 | resulting data, then the value of n computed by VSP is only an 149 | approximation to the correct n, and this approximation will tend to be less 150 | accurate as the number of non-detects increases. 151 | Input: 152 | beta: probability of falsely accepting the null hypothesis 153 | delta: change per sample period, i.e., the change that occurs between 154 | two adjacent sampling times 155 | std_dev: standard deviation of the sample points. 156 | alpha: significance level (0.05 default) 157 | n: initial number of sample points (4 default). 158 | num_iter: number of iterations of the Monte-Carlo simulation (1000 159 | default). 160 | tol: tolerance level to decide if the predicted probability is close 161 | enough to the required statistical power value (1e-6 default). 162 | num_cycles: Total number of cycles of the simulation. This is to ensure 163 | that the simulation does finish regardless of convergence 164 | or not (10000 default). 165 | m: if the tolerance is too small then the simulation could continue to 166 | cycle through the same sample numbers over and over. This parameter 167 | determines how many cycles to look back. If the same number of 168 | samples was been determined m cycles ago then the simulation will 169 | stop. 170 | Examples 171 | -------- 172 | num_samples = check_num_samples(0.2, 1, 0.1) 173 | """ 174 | # Initialize the parameters 175 | power = 1.0 - beta 176 | P_d = 0.0 177 | cycle_num = 0 178 | min_diff_P_d_and_power = abs(P_d - power) 179 | best_P_d = P_d 180 | max_n = n 181 | min_n = n 182 | max_n_cycle = 1 183 | min_n_cycle = 1 184 | # Print information for user 185 | print("Delta (gradient): {}".format(delta)) 186 | print("Standard deviation: {}".format(std_dev)) 187 | print("Statistical power: {}".format(power)) 188 | 189 | # Compute an estimate of probability of detecting a trend if the estimate 190 | # Is not close enough to the specified statistical power value or if the 191 | # number of iterations exceeds the number of defined cycles. 192 | while abs(P_d - power) > tol and cycle_num < num_cycles: 193 | cycle_num += 1 194 | print("Cycle Number: {}".format(cycle_num)) 195 | count_of_trend_detections = 0 196 | 197 | # Perform MK test for random sample. 198 | for i in xrange(num_iter): 199 | r = np.random.normal(loc=0.0, scale=std_dev, size=n) 200 | x = r + delta * np.arange(n) 201 | trend, h, p, z = mk_test(x, alpha) 202 | if h: 203 | count_of_trend_detections += 1 204 | P_d = float(count_of_trend_detections) / num_iter 205 | 206 | # Determine if P_d is close to the power value. 207 | if abs(P_d - power) < tol: 208 | print("P_d: {}".format(P_d)) 209 | print("{} samples are required".format(n)) 210 | return n 211 | 212 | # Determine if the calculated probability is closest to the statistical 213 | # power. 214 | if min_diff_P_d_and_power > abs(P_d - power): 215 | min_diff_P_d_and_power = abs(P_d - power) 216 | best_P_d = P_d 217 | 218 | # Update max or min n. 219 | if n > max_n and abs(best_P_d - P_d) < tol: 220 | max_n = n 221 | max_n_cycle = cycle_num 222 | elif n < min_n and abs(best_P_d - P_d) < tol: 223 | min_n = n 224 | min_n_cycle = cycle_num 225 | 226 | # In case the tolerance is too small we'll stop the cycling when the 227 | # number of cycles, n, is cycling between the same values. 228 | elif (abs(max_n - n) == 0 and 229 | cycle_num - max_n_cycle >= m or 230 | abs(min_n - n) == 0 and 231 | cycle_num - min_n_cycle >= m): 232 | print("Number of samples required has converged.") 233 | print("P_d: {}".format(P_d)) 234 | print("Approximately {} samples are required".format(n)) 235 | return n 236 | 237 | # Determine whether to increase or decrease the number of samples. 238 | if P_d < power: 239 | n += 1 240 | print("P_d: {}".format(P_d)) 241 | print("Increasing n to {}".format(n)) 242 | print("") 243 | else: 244 | n -= 1 245 | print("P_d: {}".format(P_d)) 246 | print("Decreasing n to {}".format(n)) 247 | print("") 248 | if n == 0: 249 | raise ValueError("Number of samples = 0. This should not happen.") -------------------------------------------------------------------------------- /optimize/genetic_algorithm.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """ Module to uses a genetic algorithm to optimize the program""" 28 | from pnl import PnL 29 | import numpy as np 30 | from initialize import Initialize 31 | import copy 32 | import random 33 | 34 | class GenAlgo(PnL): 35 | """ Genetic algo that return the parameter `self.op_param` and pnl `self.pnl_dict` for the best chromosome 36 | """ 37 | 38 | def __init__(self,self_, min_results = 10, size_population = 20, generations = 25, co_rate = .65, 39 | mutation_rate = .05): 40 | """ Setting the parameters here 41 | 42 | Parameters 43 | ---------- 44 | `self_` : class instance 45 | This is the instance of the class where GenAlgo() class is called. We copy it in the constructor here 46 | `self.min_results` : int 47 | Minimum numbers of results needed to consider a chromosome in the training period. If we have under this 48 | number, the chromosome is not considerd 49 | `self.population` : int 50 | Size of a population (number of chromosomes_ 51 | `self.generations` : int 52 | Number of generations 53 | `self.co_rate` : float 54 | Cross-over rate 55 | `self.mutation_rate` : float 56 | Mutation rate 57 | `self.fitness_function` : str 58 | Name of the fitness function. Ex: `self.sharpe_ratio_` name is defined in `initialize.py` and function 59 | is defined in `pnl.py` 60 | 61 | """ 62 | 63 | super().__init__() 64 | new_obj = copy.deepcopy(self_) 65 | self.__dict__.update(new_obj.__dict__) 66 | del new_obj,self_ 67 | self.min_results = min_results 68 | self.size_population = size_population 69 | self.generations = generations 70 | self.co_rate = co_rate 71 | self.mutation_rate = mutation_rate 72 | self.fitness_function = self.sharpe_ratio_ #string name of the fitness function 73 | self.nb_genes = len(self.op_param) #nb of genes per chromosome 74 | self.results_pop = [] #pnl for the population 75 | self.population = [] 76 | 77 | def __call__(self): 78 | """Function that runs the genetic algo. This is the "main" function 79 | 80 | First it creates the initial population in `self.create_chromosome()`. Then it optimize, run through each 81 | population and new generations in `self.run_generations()`. Finally the results are return with function 82 | `self.return_results()` 83 | 84 | Return 85 | ------ 86 | `self.pnl_dict` : dict 87 | pnl of the best chromosome 88 | 89 | `self.op_param` : list 90 | parameters of the best chromosome 91 | """ 92 | 93 | self.create_chromosome() 94 | self.run_generations() 95 | return self.return_results() 96 | 97 | def iterate_population(func): 98 | """ Decorator to run each chromosome with the size of the population 99 | 100 | The function will run through the entire population. If the sharpe_ratio is greater than `self.fitness_level` 101 | (3 by default), the algo stopped and return the pnl `self.pnl_dict` and the optimal parameters `self.op_param` 102 | """ 103 | 104 | def wrapper_(self): 105 | items_ = 0 106 | results_tempo = [] 107 | population_tempo = [] 108 | while (items_ < self.size_population): 109 | self.reset_value() 110 | self.pnl_dict = {} 111 | Initialize.__call__(self) 112 | self.optimize_param() # reinitialize parameters to optimize with all possible choices 113 | func(self) 114 | self.pnl_() 115 | if (not bool(self.pnl_dict)): 116 | continue 117 | if self.pnl_dict[self.nb_trades_] < self.min_results: 118 | continue 119 | # if self.pnl_dict[self.fitness_function] > self.fitness_level: 120 | # return self.pnl_dict,self.op_param 121 | elif bool(self.pnl_dict) and self.pnl_dict[self.nb_trades_] >= self.min_results: 122 | results_tempo.append(self.pnl_dict) 123 | population_tempo.append(self.list_to_dict(self.op_param)) 124 | if (results_tempo[items_][self.nb_trades_] > 150): 125 | raise Exception("Error with the number of trades") 126 | items_+=1 127 | self.results_pop = results_tempo.copy() 128 | self.population = population_tempo.copy() 129 | 130 | return wrapper_ 131 | 132 | def list_to_dict(self,list_): 133 | new_dict = {} 134 | for item in list_: 135 | if len(item) > 1: 136 | new_dict[item[1]] = item[0][item[1]] 137 | else: 138 | new_dict[item[0]] = getattr(self, item[0]) 139 | 140 | return new_dict 141 | 142 | @iterate_population 143 | def create_chromosome(self): 144 | """ Function that creates initial 1 chromosome randomly for initial population. Each genes is set randomly""" 145 | 146 | for item in range(len(self.op_param)): 147 | if len(self.op_param[item]) > 1: 148 | self.op_param[item][0][self.op_param[item][1]] = \ 149 | np.random.choice(self.op_param[item][0][self.op_param[item][1]]) 150 | else: 151 | setattr(self, self.op_param[item][0], np.random.choice(getattr(self, self.op_param[item][0]))) 152 | 153 | def fitness_selection(func): 154 | """Decorator to select two new chromosomes for the next generations 155 | 156 | We truncate the Put the smallest performance evaluator to 0 and raise by some amount the others evaluators. It makes 157 | sure all values are equal or above 0. The function makes sure 158 | """ 159 | 160 | def wrapper_(self): 161 | min_val = min(item[self.fitness_function] for item in self.results_pop) 162 | if min_val >= 0: 163 | min_val = 0 164 | self.fitt_total = 0 165 | for item in self.results_pop: 166 | item[self.fitness_function] -= min_val # Truncated fitness function. 167 | self.fitt_total += item[self.fitness_function] 168 | 169 | if all(item[self.fitness_function] == 0 for item in self.results_pop): #if all sharpe ratio are the same 170 | # at 0, it avoids mistake later 171 | for item in self.results_pop: 172 | item[self.fitness_function]+=1 173 | 174 | def parent(): 175 | 176 | nb_parent = 2 177 | parent_ = [] 178 | parent_item = None 179 | for j in range(nb_parent): 180 | item = 0 181 | random_nb = self.fitt_total * random.random() 182 | sum_results = 0 183 | while( item < len(self.results_pop)): 184 | sum_results += self.results_pop[item][self.fitness_function] 185 | if random_nb <= sum_results: 186 | if (parent_item is not None) and (parent_item == item): 187 | random_nb = self.fitt_total * random.random() 188 | sum_results = 0 189 | item = 0 190 | continue 191 | else : 192 | parent_.insert(item,self.population[item]) 193 | parent_item = item 194 | break 195 | item+=1 196 | 197 | if not bool(parent_): 198 | raise Exception("Parent has no value") 199 | 200 | if len(parent_) != 2: 201 | raise Exception("Don't have a father and mother") 202 | 203 | return parent_ 204 | func(self, parent()) 205 | 206 | return wrapper_ 207 | 208 | def run_generations(self): 209 | """ Function that run generations""" 210 | for generation in range(self.generations): 211 | self.new_chromosomes() 212 | 213 | @iterate_population 214 | @fitness_selection 215 | def new_chromosomes(self, parent): 216 | """ Create a new chromosome in the new generation 217 | 218 | The function only keep one chromosome in `self.op_param` for evaluation 219 | """ 220 | 221 | rand_number = random.random() 222 | if rand_number < self.mutation_rate: 223 | self.mutation(parent[0]) 224 | 225 | elif rand_number < (self.mutation_rate + self.co_rate) : 226 | self.cross_over(parent[0],parent[1]) 227 | 228 | else: 229 | self.assign_value(parent[0]) 230 | 231 | def assign_value(self, parent): 232 | """ Function to assign the new value to each each gene in the chromosome 233 | """ 234 | for item in range(len(self.op_param)): 235 | if len(self.op_param[item]) > 1: 236 | self.op_param[item][0][self.op_param[item][1]] = parent[self.op_param[item][1]] 237 | else: 238 | setattr(self, self.op_param[item][0], parent[self.op_param[item][0]]) 239 | 240 | def mutation(self, parent): 241 | """ Function that mutate one gene of one parent 242 | 243 | It changes randomly the value of one gene with the possible in `initialize.py`""" 244 | item = random.randint(0,self.nb_genes -1 ) 245 | if len(self.op_param[item]) > 1: 246 | while True: 247 | new_val = np.random.choice(self.op_param[item][0][self.op_param[item][1]]) 248 | if parent[self.op_param[item][1]] != new_val: 249 | parent[self.op_param[item][1]] = new_val 250 | break 251 | self.assign_value(parent) # assign father's value to new chromosome to be tested 252 | 253 | else: 254 | new_val = np.random.choice(getattr(self, self.op_param[item][0])) 255 | while True: 256 | new_val = np.random.choice(getattr(self, self.op_param[item][0])) 257 | if parent[self.op_param[item][0]] != new_val: 258 | parent[self.op_param[item][0]] = new_val 259 | break 260 | self.assign_value(parent) # assign father's value to new chromosome to be tested 261 | 262 | 263 | def cross_over(self,father,mother): 264 | """ Function that cross-over one gene of one parent to the other parent""" 265 | 266 | item = random.randint(0,self.nb_genes -1 ) 267 | if len(self.op_param[item]) > 1: 268 | 269 | father[self.op_param[item][1]] = mother[self.op_param[item][1]] 270 | else : 271 | 272 | father[self.op_param[item][0]] = mother[self.op_param[item][0]] 273 | self.assign_value(father) #assign father value to new chromosome to be tested 274 | 275 | def return_results(self): 276 | """Return the results of the best chromosome (1 chromosome) 277 | 278 | Return 279 | ------ 280 | `self.pnl_dict` : dict 281 | pnl of the best chromosome 282 | 283 | `self.op_param` : list 284 | parameters of the best chromosome 285 | """ 286 | 287 | for index, item in enumerate(self.results_pop): 288 | if index == 0: 289 | max_val = item[self.fitness_function] 290 | max_idx = index 291 | elif item[self.fitness_function] > max_val: 292 | max_val = item[self.fitness_function] 293 | max_idx = index 294 | 295 | return self.results_pop[max_idx], self.population[max_idx] -------------------------------------------------------------------------------- /initialize.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """Module that declares hyperparamaters and parameters to optimize""" 28 | 29 | from dateutil.relativedelta import relativedelta 30 | from date_manip import DateManip as dm 31 | from manip_data import ManipData as md 32 | import pandas as pd 33 | from datetime import datetime 34 | import random 35 | import os 36 | 37 | class Initialize(): 38 | """Module to initialize the values. 39 | 40 | The __init__ constructor inialize the hyperparameters and others parameters that won't be optimized parameters. The 41 | optimized parameters are in __call__ function. 42 | """ 43 | 44 | def __init__(self): 45 | """ 46 | Initialize the hyperparameters and other parameters that won't be optimized. 47 | 48 | Parameters 49 | ---------- 50 | `self.directory` : str 51 | Where the the data are located for training and testing periods 52 | `self.asset` : str 53 | Name of the file where we get the data 54 | `self.is_fx` : bool 55 | Tell if `self.asset` is forex (the data don't have the same format for forex and stocks because they are 56 | from different providers). 57 | `self.dir_output` : str 58 | Where we store the results 59 | `self.name_out` : str 60 | Name of the results file name (csv) 61 | `self.start_date` : datetime object 62 | Beginning date of training and testing period. The variable is already transformed from a str to a 63 | Datetime object 64 | `self.end_date` : datetime object 65 | Ending date of training and testing period. The variable is already transformed from a str to a 66 | Datetime object 67 | `self.is_walkfoward` : bool 68 | Tells if we do an optimization (training and testing) or only a test on the whole period 69 | `self.training_name` : str 70 | If `self.is_foward` is `True`, this is the string added to the result files name for training data 71 | `self.test_name_` : str 72 | If `self.is_foward` is `True`, this is the string added to the result files name for test data 73 | `self.training_` : int 74 | Lenght in months of training period 75 | `self.test_` : int 76 | Lenght in months of testing period (put 9) 77 | `self.name_col` : dict 78 | Says which data we need in our testing 79 | `self.dup_col` : dict 80 | Columns we use to check if the market is closed. Ex: If OHLC are all the same, then the market is closed. 81 | `self.window` : int 82 | Number of data (points) we check before and after a data to check if it is a local min and max 83 | `self.pnl_dict` : dict 84 | Contains the metrics to calculate the strategy performance. The metrics are (keys) : 85 | `self.range_date_` : str 86 | Testing date range 87 | `self.sharpe_ratio_` : float 88 | Sharpe ratio. Did not substract the risk-free rate, only return divided by volality. Reason is that on 89 | Forex or small timeframe, return is low and we can leverage return (or totally 90 | distort it) with margin trading 91 | `self.ann_return_` = float 92 | Annualized return 93 | `self.ann_vol_` = float 94 | Annualized volatility 95 | `self.pour_win_` = float 96 | Ratio of winning trades 97 | `self.max_draw_` = float 98 | Maximum drawdown 99 | `self.nb_trades_` = int 100 | Number of trades during the period 101 | `self.entry_row` : int 102 | Entry row number for a trade 103 | `self.entry_level` : float 104 | Entry level for a trade 105 | `self.exit_row` = int 106 | Exit row number for a trade 107 | `self.exit_level` : int 108 | Exit level for a trade 109 | `self.trade_return` : float 110 | Trade return for a trade 111 | `self.is_detrend` : bool 112 | Default is `False`. Possible to set to yes or no. It tells if we want to de-trend the current series. 113 | If we set it to `True`, make sure to make changes in `Indicator.py` to use `self.series_test` instead of 114 | `self.series` and in `init_operations.py` remove the commented code in function `init_series()` for 115 | `self.series_test`. 116 | `self.p_value` : float 117 | Significance level to test for non stationarity with Augmented Dickey-Fuller. Default is 0.01 118 | `self.period` = int 119 | Order of differencing of the time series. By default 1 (first order differencing) 120 | 121 | """ 122 | 123 | self.directory = os.path.dirname(os.path.realpath(__file__)) + '/' 124 | self.dir_output = self.directory + 'results/' 125 | self.name_out = 'results' 126 | self.is_fx = True 127 | self.asset = "EURUSD" 128 | self.start_date = datetime.strptime('2020-02-15', "%Y-%m-%d") 129 | self.end_date = datetime.strptime('2020-04-01', "%Y-%m-%d") 130 | 131 | self.is_walkfoward = False 132 | self.training_name_ = '_training' 133 | self.test_name_ = '_test' 134 | self.training_ = 18 #Lenght in months of training period (put 18) 135 | self.test_ = 9 #Lenght in months of testing period (put 9) 136 | self.dict_name_ = {self.training_name_:self.training_,self.test_name_:self.test_} 137 | self.train_param= [] #Optimized training parameters used for the test period 138 | 139 | # Decide which data type we need in our testing 140 | self.date_name = 'Date' 141 | self.open_name = 'Open' 142 | self.high_name = 'High' 143 | self.low_name = 'Low' 144 | self.close_name = 'Close' 145 | self.adj_close_name = 'Adj Close' 146 | self.name_col = { 147 | self.date_name: [], 148 | self.open_name: [], 149 | self.high_name: [], 150 | self.low_name: [], 151 | self.adj_close_name: [] 152 | } 153 | 154 | #Columns we use to check if there are duplicate values 155 | self.dup_col = { 156 | self.open_name: [], 157 | self.high_name: [], 158 | self.low_name: [], 159 | self.adj_close_name: [] 160 | } 161 | 162 | self.window = 6 163 | 164 | # Metrics used to calculate the strategy performance 165 | self.pnl_dict = {} 166 | self.range_date_ = 'Date range from ' 167 | self.sharpe_ratio_ = 'Sharpe ratio' # Did not substract the risk-free rate, only return divided by vol 168 | self.ann_return_ = 'Annualized return' 169 | self.ann_vol_ = 'Annualized volatility' 170 | self.pour_win_ = '% win' # pourcentage of winning trades 171 | self.max_draw_ = 'Maximum drawdown' 172 | self.nb_trades_ = 'nb_trade' 173 | 174 | # Values used to calculate each trade performance 175 | self.entry_row = 'Entry_row' 176 | self.entry_level = 'Entry_level' 177 | self.exit_row = 'Exit_row' 178 | self.exit_level = 'Exit_level' 179 | self.trade_return = 'trade_return' 180 | 181 | self.default_data = self.adj_close_name # Value we use by default for chart, extremum, etc. 182 | 183 | 184 | self.is_detrend = False # Possible to set to yes r no 185 | self.p_value = .01 # significance level to test for non stationarity with Augmented Dickey-Fuller 186 | self.period = 1 187 | 188 | #No need to change them. These are variable that we don't need to change 189 | self.name = pd.DataFrame(self.name_col) 190 | self.date_ordinal_name = 'Ordinal Date' 191 | self.marker_ = 'marker_name' 192 | self.color_mark = 'color_mark' 193 | self.marker_signal = 'marker_signal' 194 | self.end_format_ = "%Y-%m-%d" #format when printing date range in csv files with results 195 | 196 | def __call__(self): 197 | 198 | """ Set values for optimization if we decide to optimize 199 | 200 | It is the values that are initialized and used through the system 201 | 202 | Parameters 203 | ---------- 204 | `self.nb_data` : int 205 | Number of data needed to calculate the indicators. Default is 100 but can be 100, 200, 300. 206 | `self.r_square_level` : float 207 | R square level required to trigger a signal. Can be .6 , .7, .8 and .9. 208 | `self.min_data` : int 209 | Number of minimum required between each signal in the same direction to get another signal 100,200 or 300 210 | 211 | Entry 212 | `self.enter_dict` : dict 213 | Dictionary containing the possible entry types 214 | `self.enter_ext_name` : str 215 | Name for entering in the market with Fibonnacci extensions. We are using the largest extension of 216 | the previous trend as a reference. Largest extension is in fact the largest setback in the current trend. 217 | Ex : largest extension from previous trend is 1, the system enters in the market when it is `self.enter_ext` 218 | this size (1 by default). So, when this value is reached, it takes the profit. 219 | `self.enter_ext` : float 220 | This is the % of largest extension at which the system enters the market. Can be .882 or .764 or 1 221 | `self.enter_time` : str 222 | Entering the market only if the current setback is a minimum percentage of largest setback 223 | from current trend in term of time. 224 | `self.enter_bool` : bool 225 | Telling if we use this technic or not. Can be set to `True` or `False` 226 | `self.time_ext` : float 227 | Proportion in term in time needed that the current setback most do compared to the largest extension 228 | in therm of time in the current trend to enter the market. It can be .5, .618 .884,1 229 | Exit 230 | Extension 231 | `self.exit_dict` : dict 232 | Dictionary containing the possible ways to exit the market 233 | `self.exit_ext_name` : str 234 | name of the dictionary key to try to exit the market with Fibonnacci extensions 235 | `self.exit_ext` : bool 236 | tells the system if tries to exit (or not) the market using the largest extension as a reference. 237 | It has to be `True` at the moment because it is the only way to exit the market. 238 | Ex : largest extension from preivous trend is 1, the system takes profit when it is 239 | `self.profit_ext` of this size (2.618 by default). So, when this value is reached, it takes the profit. 240 | `self.profit_ext` : float 241 | % of the largest extension from previous trend that the system uses to exit the market to take profit 242 | Default value is 2.618. Possible values are 1.618, 2 , 2.618, 3.382, 4.236. 243 | `self.stop_ext` : float 244 | % of the largest extension from previous trend that the system uses as a stop loss. 245 | Default value is 1.618. Possible values are 1, 1.382, 1.618, 2. 246 | 247 | Stop trying to enter 248 | These params are conditions to stop trying to enter the market if the current price reach a % of the 249 | largest extension `self.fst_cdt_ext` goes in the other direction and retraces the last top or bottom 250 | with proportion `self.sec_cdt_ext`, the system stop trying to enter in the market. 251 | `self.bol_st_ext` : bool 252 | Tells the system if it has to stop trying to enter the market using Fibonacci extension techniques. 253 | Can be optimized to `True` or `False` 254 | `self.fst_cdt_ext` : float 255 | % of the largest extension that if the market reaches, the system stops trying to enter the market. 256 | Possible values are .618, .764 or .882 257 | `self.sec_cdt_ext` : float 258 | if the market triggers the first condition, then if it reaches this level in the opposite direction, the 259 | # system stops trying to enter in the market. Can be set to .618, .764, 1 or 1.382 260 | 261 | Stop tightening 262 | General 263 | `self.stop_tight_dict` : dictionary 264 | contains the different possibilities to tighten the stop 265 | 266 | `self.default_data_` : bool 267 | default data used to determine if the stop loss level must be tightened. It is `True`, then 268 | `self.adj_close_name` is used. Otherwise, `self.low_name` with `self.high_name` (depends if it is 269 | a buy or sell signal). 270 | 271 | Stop tightening (technique no 1) 272 | `self.stop_tight_ret` : str 273 | Tightening the stop using Fibonacci retracement condition (contains the condition). 274 | `self.stop_ret_level` : float 275 | Level at which the system tight the stop when it reaches this retracement in the opposite direction. 276 | Ex: Buy signal then, market reaches `self.stop_ret_level` (1 by default) in the other direction. 277 | The system will tighen the stop to the lowest (or highest) point. Default value is 1. 278 | Possible values are 618, .882, 1, 1.618 or 2. 279 | `self.is_true` : bool 280 | Tells the system if it has to use this particular technique to tighten the stop or not. Possible values are 281 | `True` or `False` 282 | 283 | Stop tightening (technique no 2) 284 | `self.stop_tight_pour` : str 285 | tightening the stop when the market reaches a certain percentage of the target 286 | `self.is_true` : bool 287 | Tells the system if it has to use this particular technique to tighten the stop or not. Possible values are 288 | `True` or `False` 289 | `self.tight_value` : float 290 | When the market reaches this percentage of the target, we tighten the stop. Possible values are 291 | .5,.618,.764. 292 | `self.pour_tight` : float 293 | Percentage between the current market value and entry value that the current new stop is. Possible values 294 | are .5 or .618. 295 | 296 | """ 297 | 298 | self.nb_data = 100 299 | self.r_square_level = self.return_value([.6,.7,.8,.9],.7) 300 | self.min_data = self.return_value([100,200,300],100) 301 | 302 | #ENTRY 303 | #----- 304 | # All the possible entry types that the system can do (extension, retracement) 305 | self.enter_bool = 'enter_bool' #same key name for all the exit strategy (located in different dictionary) 306 | self.enter_ext_name = 'enter_ext_name' 307 | self.enter_ext = 'enter_ext' #Entering the market with largest extension from setback in current trend 308 | self.stop_ext = 'stop_ext' 309 | self.enter_time = 'enter_time' 310 | self.time_ext = 'time_ext' 311 | 312 | self.enter_dict = {self.enter_ext_name : 313 | {self.enter_bool : True, #At the moment, it has to be `True`, no other method to enter 314 | self.enter_ext: self.return_value([.764,.882,1],1) 315 | 316 | }, 317 | self.enter_time: 318 | {self.enter_bool : self.return_value([False,True],True), 319 | self.time_ext : self.return_value([.5,.618,.882,1],.618) 320 | } 321 | } 322 | 323 | #STOP TRY TO ENTER 324 | #-------------- 325 | #These params are conditions to stop trying to enter the market if the current 326 | # price reach a % of the largest extension 327 | self.bol_st_ext = self.return_value([True,False],True) 328 | self.fst_cdt_ext = self.return_value([.618,.764,.882],.764) 329 | self.sec_cdt_ext = self.return_value([.618,.764,1,1.382],1) 330 | 331 | #STOP TIGHTENING 332 | #--------------- 333 | self.stop_tight_ret = 'stop_tight_ret' 334 | self.stop_tight_pour = 'stop_tight_pourentage' 335 | self.is_true = 'is_true' 336 | self.default_data_ = 'default_data_' #adjusted closed 337 | self.stop_ret_level = 'stop_ret_level' 338 | self.tight_value = 'tight_value' 339 | self.pour_tight = 'pour_tight' 340 | self.stop_tight_dict = {self.stop_tight_ret : #Stop tightening technique no 1. 341 | {self.is_true : self.return_value([True,False],True), 342 | self.default_data_ : True, 343 | self.stop_ret_level : self.return_value([.618,.882,1,1.618,2],1) 344 | }, 345 | 346 | self.stop_tight_pour : #Stop tightening no 2. 347 | {self.is_true : self.return_value([True,False],True), #True or False 348 | self.tight_value : self.return_value([.5,.618,.764],.5), 349 | self.pour_tight : self.return_value([.5,.618],.5) 350 | } 351 | } 352 | 353 | # EXIT 354 | # ----- 355 | # All the possible ways that the system can exit the market (extension, retracement) 356 | self.exit_name = 'exit_name' #same key name for all the exit strategy (located in different dictionary) 357 | self.exit_ext_bool = 'exit_ext_bool' 358 | self.profit_ext = 'profit_ext' 359 | self.stop_ext = 'stop_ext' 360 | self.exit_dict = {self.exit_name : 361 | {self.exit_ext_bool : True, #It has to be `True` has it the only way for now to exit the 362 | #market 363 | self.profit_ext :self.return_value([1.618,2,2.618,3.382,4.236],3.382), 364 | #also try 2 2.618, 3.382, 4.236 365 | self.stop_ext : self.return_value([1,1.382,1.618,2],1.618) #1,1.382, 1.618, 2 366 | } 367 | } 368 | 369 | def return_value(self,first_val,sec_val): 370 | """ Return first value if `True` (we optimize), second if False 371 | 372 | """ 373 | return first_val if self.is_walkfoward else sec_val 374 | 375 | def optimize_param(self): 376 | """ Tell the parameters we want to optimize. 377 | 378 | Store the paramaters to optimize in a list with the name of the parameter to optimize 379 | """ 380 | 381 | self.op_param = [[self.exit_dict[self.exit_name],'profit_ext'], 382 | [self.exit_dict[self.exit_name],'stop_ext'], 383 | [self.stop_tight_dict[self.stop_tight_ret],'is_true'], 384 | [self.stop_tight_dict[self.stop_tight_ret],'stop_ret_level'], 385 | [self.stop_tight_dict[self.stop_tight_pour],'is_true'], 386 | [self.stop_tight_dict[self.stop_tight_pour],'tight_value'], 387 | [self.stop_tight_dict[self.stop_tight_pour],'pour_tight'], 388 | ['bol_st_ext'], 389 | ['fst_cdt_ext'], 390 | ['sec_cdt_ext'], 391 | [self.enter_dict[self.enter_ext_name],'enter_ext'], 392 | [self.enter_dict[self.enter_time],'enter_bool'], 393 | [self.enter_dict[self.enter_time],'time_ext'], 394 | ['r_square_level'], 395 | ['min_data']] 396 | -------------------------------------------------------------------------------- /entry/entry_fibo.py: -------------------------------------------------------------------------------- 1 | #!/usr/local/bin/env python3.7 2 | # -*- coding: utf-8; py-indent-offset:4 -*- 3 | ############################################################################### 4 | # 5 | # The MIT License (MIT) 6 | # Copyright (c) 2020 Philippe Ostiguy 7 | # 8 | # Permission is hereby granted, free of charge, to any person obtaining a copy 9 | # of this software and associated documentation files (the "Software"), to deal 10 | # in the Software without restriction, including without limitation the rights 11 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | # copies of the Software, and to permit persons to whom the Software is 13 | # furnished to do so, subject to the following conditions: 14 | # 15 | # The above copyright notice and this permission notice shall be included in all 16 | # copies or substantial portions of the Software. 17 | # 18 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 19 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 20 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 21 | # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 22 | # DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR 23 | # OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 24 | # OR OTHER DEALINGS IN THE SOFTWARE. 25 | ############################################################################### 26 | 27 | """Module that will try to enter the market using Fibonnacci techniques""" 28 | 29 | import math_op as mo 30 | import initialize as init 31 | import operator as op 32 | import math 33 | 34 | class EntFibo(): 35 | """ 36 | Class that uses Fibonnacci strategies to enter the market """ 37 | 38 | def __init__(self): 39 | self.extreme = {} 40 | self.high="max" 41 | self.low="min" 42 | self.high_idx="max_idx" 43 | self.low_idx = "min_idx" 44 | self.fst_ext_cdt = False #by default first condition for extension is not met, set to False 45 | self.is_entry = False 46 | self.relative_extreme = None #last wave the system uses (relative low for buy, vice versa) as a 47 | # basis to calculate the profit taking price. It uses the default data (close) 48 | # to smooth data 49 | self.row_rel_extreme = 0 50 | self.largest_time = 0 #extension in time 51 | self.index_name = 'index' 52 | 53 | def ent_fibo(self,curr_row,buy_signal=False,sell_signal=False): 54 | """This the main method that uses Fibonnacci strategies to enter the market 55 | 56 | First, the method finds local minimum and maximum in the current trend with function `self.local_extremum_()`. 57 | Then using `self.largest_extension()`, it finds the largest setback which is stored in the attribute 58 | `self.largest_extension_`. This function also store the largest setback in term of time in `self.largest_time`. 59 | Then in method `self.try_entry()`, the system will try to enter the market.. 60 | 61 | Trying to enter the market with Fibonacci retracement and extension. 3 types: 62 | Retracement from the last wave 63 | Retracement from beginning of the trend 64 | Extension from the current trend (largest one in the last trend) 65 | 66 | At the moment, it is possible to enter the market only with extensions from the current trend. 67 | 68 | Parameters 69 | ---------- 70 | `self.buy_signal` and `self.sell_signal` : bool 71 | The module `r_aquare_ty.py` (or package `trading_rules`) tells us if there a buy or sell signal. Then, 72 | just not to have the code twice, we set some variable depending on if `self.buy_signal` or 73 | `self.sell_signal` is `True`. 74 | 75 | Notes 76 | ----- 77 | No slippage included in `self.try_entry()`. If the price reached the desired level, we just exit at either the 78 | current price or the next desired price 79 | 80 | The system doesn't check on a shorter time frame if it reaches an entry point and a stop at the same time 81 | or even an exit point and stop at the same time (in case of high volatility) in `self.try_entry()` 82 | Taking into account how the system works, those are really rare cases. However it could be tested by using a 83 | shorter time every time there is an entry or exit signal 84 | 85 | """ 86 | 87 | self.curr_row=curr_row 88 | self.buy_signal=buy_signal 89 | self.sell_signal=sell_signal 90 | 91 | if (self.largest_time != 0): 92 | raise Exception("Largest extension in term of times is not equal to 0 in __init__") 93 | 94 | self.first_data = self.curr_row - self.nb_data + 1 95 | if self.first_data < 0: 96 | self.first_data = 0 97 | 98 | if self.buy_signal and self.sell_signal : 99 | raise Exception('Cannot have a buy and sell signal at the same time') 100 | 101 | self.set_extremum() #set the absolute high and low on the current trend 102 | 103 | if self.buy_signal: 104 | start_point=self.extreme[self.low_idx] 105 | self.fst_op=op.gt 106 | self.sec_op=op.lt 107 | self.trd_op=op.sub 108 | self.fth_op=op.add 109 | self.fif_op=op.ge 110 | self.six_op=op.le 111 | self.fst_data = self.high 112 | self.sec_data = self.low 113 | self.fst_idx = self.high_idx 114 | self.sec_idx = self.low_idx 115 | self.entry = self.stop = self.low_name 116 | self.exit = self.high_name 117 | self.inv = -1 118 | 119 | if self.sell_signal: 120 | start_point = self.extreme[self.high_idx] 121 | self.fst_op=op.lt 122 | self.sec_op=op.gt 123 | self.trd_op=op.add 124 | self.fth_op=op.sub 125 | self.fif_op=op.le 126 | self.six_op=op.ge 127 | self.fst_data = self.low 128 | self.sec_data = self.high 129 | self.fst_idx = self.low_idx 130 | self.sec_idx = self.high_idx 131 | self.entry = self.stop = self.high_name 132 | self.exit = self.low_name 133 | self.inv = 1 134 | 135 | self.mo_ = mo.MathOp(series=self.series, default_col=self.default_data) 136 | self.local_extremum_=self.mo_.local_extremum(start_point=start_point, end_point=self.curr_row, \ 137 | window=self.window, min_=self.low,max_=self.high,index_= self.index_name) 138 | self.local_extremum_ = self.local_extremum_.reset_index(drop=True) 139 | 140 | self.largest_extension() #finding the largest extension used for potential entry and/or exit 141 | if not hasattr(self, 'largest_extension_'): #in case it doesn't find a largest_extension, 142 | self.is_entry = False # exit and just do nothing to avoid error 143 | return 144 | self.set_value() 145 | self.try_entry() 146 | 147 | def largest_extension(self): 148 | """ Find largest extension (setback) from current trend (Fibonacci) in size which is stored in 149 | `self.largest_extension_` and largest in term of time stored in `self.largest_time`. 150 | """ 151 | 152 | if self.buy_signal: 153 | my_data={} 154 | fst_data='curr_high' 155 | sec_data='curr_low' 156 | my_data[fst_data]=None 157 | my_data[sec_data] = None 158 | fst_name=self.high 159 | sec_name=self.low 160 | 161 | if self.sell_signal: 162 | my_data={} 163 | fst_data='curr_low' 164 | sec_data='curr_high' 165 | my_data[fst_data]=None 166 | my_data[sec_data] = None 167 | fst_name=self.low 168 | sec_name=self.high 169 | 170 | trd_data = 'first_index' 171 | my_data[trd_data] = 0 172 | fth_data = 'sec_index' 173 | my_data[fth_data] = 0 174 | 175 | for curr_row_ in range(len(self.local_extremum_)): 176 | 177 | #Sorten the name 178 | fst_val = self.local_extremum_.iloc[curr_row_, self.local_extremum_.columns.get_loc(fst_name)] 179 | sec_val = self.local_extremum_.iloc[curr_row_, self.local_extremum_.columns.get_loc(sec_name)] 180 | _current_index = self.local_extremum_.iloc[curr_row_, self.local_extremum_.columns.get_loc(self.index_name)] 181 | 182 | #If there are value to high and low, assign largest_extension_ 183 | if (my_data[fst_data] != None) & (my_data[sec_data] != None): 184 | if math.isnan(sec_val) & (not math.isnan(my_data[fst_data])) & (not math.isnan(my_data[sec_data])): 185 | 186 | if not hasattr(self,'largest_extension_'): 187 | self.largest_extension_ = self.inv*(my_data[sec_data] - my_data[fst_data]) 188 | 189 | if (my_data[fst_data] != None) & (my_data[sec_data] != None) : 190 | if op.ge(self.inv*(my_data[sec_data] - my_data[fst_data]), self.largest_extension_): 191 | self.largest_extension_ = self.inv*(my_data[sec_data] - my_data[fst_data]) 192 | 193 | _ext_time = my_data[fth_data]- my_data[trd_data] 194 | if (_ext_time>self.largest_time): 195 | self.largest_time = _ext_time 196 | 197 | my_data[fst_data] = None 198 | my_data[sec_data] = None 199 | 200 | 201 | #It checks at the second last data, if there is a data for second_name (new relative high for sell 202 | # or new relative low for buy), it just basically don't check it, because it is not a real extension 203 | if curr_row_ == (len(self.local_extremum_) -1): 204 | if not math.isnan(sec_val) & (curr_row_ == (len(self.local_extremum_) -1 )): 205 | break 206 | pass 207 | 208 | #Assign a value to first value (high for buy, low for sell) until it's NOT None or Nan 209 | if (my_data[fst_data] == None): 210 | my_data[fst_data] = fst_val 211 | my_data[trd_data] = _current_index 212 | continue 213 | 214 | if math.isnan(my_data[fst_data]): 215 | my_data[fst_data] = fst_val 216 | my_data[trd_data] = _current_index 217 | continue 218 | 219 | #If there is a valid first value, check if current value higher than recorded high (for buy), vice versa 220 | if not math.isnan(fst_val): 221 | if self.fst_op(fst_val, my_data[fst_data]): 222 | my_data[fst_data] = fst_val 223 | my_data[trd_data] = _current_index 224 | continue 225 | 226 | if (my_data[sec_data] == None): 227 | my_data[sec_data] = sec_val 228 | my_data[fth_data] = _current_index 229 | continue 230 | 231 | if math.isnan(my_data[sec_data]): 232 | my_data[sec_data] = sec_val 233 | my_data[fth_data] = _current_index 234 | continue 235 | 236 | if not math.isnan(sec_val): 237 | if self.sec_op(sec_val, my_data[sec_data]): 238 | my_data[sec_data] = sec_val 239 | my_data[fth_data] = _current_index 240 | continue 241 | 242 | 243 | def set_extremum(self): 244 | """ 245 | Set the global max and min for the given range (from first_data to curr_row).""" 246 | 247 | data_range = self.series.loc[self.first_data:self.curr_row,self.default_data] 248 | self.extreme = {self.high : data_range.max(), 249 | self.low : data_range.min(), 250 | self.high_idx : data_range.idxmax(), 251 | self.low_idx : data_range.idxmin() 252 | } 253 | 254 | def set_value(self): 255 | """Method to set some values that are used in this class and sublcass """ 256 | 257 | # extension level if condition in initialize.py is True 258 | if self.exit_dict[self.exit_name][self.exit_ext_bool]: 259 | self.extension_lost = self.largest_extension_ * self.exit_dict[self.exit_name][self.stop_ext] 260 | self.extension_profit = self.largest_extension_ * self.exit_dict[self.exit_name][self.profit_ext] 261 | self.stop_value = self.trd_op(self.extreme[self.fst_data], self.extension_lost) 262 | 263 | if self.extension_lost < 0: 264 | raise Exception( 265 | f"Houston, we've got a problem, `self.extension_lost` in enter_fibo.py is {self.extension_lost} " 266 | f"and should not be negative") 267 | if self.extension_profit < 0: 268 | raise Exception(f"Houston, we've got a problem, `self.extension_profit` in enter_fibo.py is " 269 | f"{self.extension_profit} and should not be negative") 270 | 271 | 272 | def try_entry(self): 273 | """ 274 | Method that try entering in the market 275 | 276 | Function that will try to enter the market : 277 | Until the system hit the desired extension and/or retracement. At the moment, only using extension (the 278 | largest), which is `self.largest_extension_`. We can decide the proportion of the largest 279 | extension we want the system to use in module `initialize.py` within dictionary `self.enter_dict{}` 280 | and variable `self.enter_ext` (default value is 1) 281 | It can also enter in the market only if the market retraces (or setback) above a certain amount of time 282 | `self.time_ext` (set in `initialize.py`) the largest setback in term of time from the current 283 | trend `self.largest_time` 284 | 285 | Possibility to stop trying to enter in the market when a condition is met. 286 | At the moment, the only condition is when the price during a setback hits a 287 | percentage `self.fst_cdt_ext` (0.618 by default) of the largest extension `self.largest_extension_` 288 | (low for a buy signal and high for a sell signal which is `self.entry`) 289 | AND hits the minimum retracement in the other direction `self.sec_cdt_ext` (.882 by default) 290 | Set to `True` with `self.bol_st_ext` in `initialize.py to have this condition. 291 | 292 | Parameters 293 | ---------- 294 | `self.largest_extension_` : float 295 | the largest extension or setback (in point) from the the current trend 296 | `self.enter_ext` : float 297 | This is the % of largest extension at which the system enters the market. Can be .882 or .764 or 1 298 | `self.largest_time` : 299 | the largest setback from the current trend 300 | `self.time_ext` : float 301 | Proportion in term in time needed that the current setback most do compared to the largest extension 302 | in therm of time in the current trend to enter the market. It can be .5, .618 .884,1. Set in `initialize.py` 303 | `self.bol_st_ext` : bool 304 | Tells the system if it has to stop trying to enter the market using Fibonacci extension techniques. 305 | Can be optimized to `True` or `False`. Set in `initialize.py` 306 | `self.fst_cdt_ext` : float 307 | % of the largest extension that if the market reaches, the system stops trying to enter the market. 308 | Possible values are .618, .764 or .882. Set in `initialize.py` 309 | `self.sec_cdt_ext` : float 310 | if the market triggers the first condition, then if it reaches this level in the opposite direction, the 311 | # system stops trying to enter in the market. Can be set to .618, .764, 1 or 1.382. Set in `initialize.py` 312 | 313 | Notes 314 | ----- 315 | Note that the system will priorise an entry over a new high or new low (to be more conservative). To solve 316 | this issue (rare cases, only with high volatility) : 317 | Check simulateneously if a new high or low is reached & (if a buy/sell level is trigerred or 318 | if the market hits minimum required extension (if this condition is tested)) 319 | Then, on a shorter timeframe, check if an entry or minimum required extension is reached before the 320 | market makes new low or high, vice versa 321 | 322 | If the price of the current row on which the signal is trigerred is below the buying level or above the 323 | selling level, the system just don't execute it and end it. 324 | """ 325 | 326 | data_test = len(self.series) - self.curr_row - 1 327 | 328 | #Data used only in entry.fibo at the moment 329 | _largest_time = self.largest_time * self.enter_dict[self.enter_time][self.time_ext] 330 | _bool_time = self.enter_dict[self.enter_time][self.enter_bool] 331 | _largest_ext = self.largest_extension_ * self.enter_dict[self.enter_ext_name][self.enter_ext] 332 | _bool_ext = self.enter_dict[self.enter_ext_name][self.enter_bool] 333 | 334 | if self.is_entry: 335 | raise Exception('Already have an open position in the market...') 336 | 337 | self.set_value() 338 | 339 | for curr_row_ in range(data_test): 340 | 341 | #We may change that later if we decides to use other things than only the largest extension to enter in 342 | # the market. It checks if there is a "largest extension" set (in some case, there might not be) 343 | if not hasattr(self, 'largest_extension_'): 344 | self.is_entry = False 345 | print("Not any largest extension") 346 | break 347 | 348 | if _bool_ext: 349 | if _largest_ext < 0: #Can happens when the market moves really fast and not able to find 350 | # a `self.largest_extension` that is positive 351 | self.is_entry = False 352 | break 353 | _entry_tentative = self.trd_op(self.extreme[self.fst_data], _largest_ext) 354 | 355 | #Test first if using Fibonacci extension as a signal to enter in the market. 356 | #Then the system first check if the price on the current row is below (for buy) or above (for sell signal) 357 | #If it is the case, the system just don't enter in the market. 358 | if self.enter_dict[self.enter_ext_name][self.enter_bool]: 359 | if (curr_row_ == 0) & self.fif_op(_entry_tentative, self.series.loc[self.curr_row,self.entry]): 360 | self.is_entry = False 361 | break 362 | 363 | self.curr_row += 1 364 | 365 | _current_value = self.series.loc[self.curr_row, self.default_data] #curent value with default data type 366 | _current_stop = self.series.loc[self.curr_row, self.stop] #current stop value with data stop type 367 | _current_entry = self.series.loc[self.curr_row, self.entry] 368 | 369 | if not hasattr(self, 'stop_value'): 370 | self.is_entry = False 371 | print("Not any stop_value") 372 | break 373 | 374 | if self.relative_extreme == None: 375 | self.relative_extreme = self.series.loc[self.curr_row, self.default_data] 376 | self.row_rel_extreme = self.curr_row 377 | 378 | #Check if has to enter after a certain time only 379 | if _bool_time & (curr_row_ <_largest_time): 380 | # Retrace two quickly (in time) and went below (for a buy signal) the stop loss. Do not enter 381 | if self.six_op(_current_stop, self.stop_value): 382 | self.is_entry = False 383 | break 384 | else : 385 | continue 386 | 387 | #Buy or sell signal (entry) with extension 388 | # - Buy if current market price goes below our signal or equal 389 | # - Sell if current market price goes above our signal or equal 390 | if self.enter_dict[self.enter_ext_name][self.enter_bool]: 391 | if self.six_op(_current_entry,_entry_tentative): 392 | 393 | #Check if current price is below (for buy) desired entry level after the minimum time. If yes, 394 | #the market enters at the current price and not the desired 395 | if _bool_time & (self.curr_row == math.ceil(_largest_time)): 396 | _entry_level = _current_entry 397 | else : 398 | _entry_level = _entry_tentative 399 | 400 | self.is_entry = True 401 | self.trades_track_ = self.trades_track_.append({self.entry_row: self.curr_row,\ 402 | self.entry_level:_entry_level},ignore_index=True) 403 | 404 | if self.sec_op(_current_value, self.relative_extreme): 405 | self.relative_extreme = _current_value 406 | self.row_rel_extreme = self.curr_row 407 | break 408 | 409 | #Market hits the minimum required extension - first condition met (to stop trying entering the market) 410 | if self.bol_st_ext & self.six_op(_current_entry, \ 411 | self.trd_op(self.extreme[self.fst_data],self.largest_extension_ * self.fst_cdt_ext)): 412 | if self.sec_op(_current_value, self.relative_extreme): 413 | self.relative_extreme = _current_value 414 | self.row_rel_extreme = self.curr_row 415 | self.fst_ext_cdt = True 416 | continue 417 | 418 | # The system will stop trying to enter the market : 419 | # - first condition (extension) is met. It hit the required 420 | # % of the largest extension, previously (61.8% by default) - low for buy, high for sell 421 | # - It went back then reached the minimum retracement in the other direction (88.2% by default) 422 | if self.bol_st_ext & self.fst_ext_cdt & (self.relative_extreme != None) : 423 | if self.fif_op(self.series.loc[self.curr_row,self.default_data],self.fth_op(self.relative_extreme,\ 424 | self.inv*(op.sub(self.relative_extreme, self.extreme[self.fst_data])*self.sec_cdt_ext))) : 425 | #print(f"The market hits previously the required {self.fst_cdt_ext} % of the largest extension" 426 | # f"and then retrace in the opposite direction of {self.sec_cdt_ext}") 427 | self.is_entry = False 428 | break 429 | pass 430 | 431 | #Changing global low or high if current one is lower or higher 432 | if self.fst_op(self.series.loc[self.curr_row, self.default_data], self.extreme[self.fst_data]): 433 | self.extreme[self.fst_data] = self.series.loc[self.curr_row, self.default_data] 434 | self.extreme[self.fst_idx] = self.curr_row 435 | 436 | #Changing stop value 437 | if self.exit_dict[self.exit_name][self.exit_ext_bool]: 438 | self.stop_value = self.trd_op(self.extreme[self.fst_data], self.extension_lost) -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/artificial-intelligence.png) 2 | 3 | # Optimizing the Elliott Wave Theory using genetic algorithms to forecast the financial markets 4 | 5 | 6 | Hi! 7 | 8 | My name is Philippe. The goal of this project is to model the Elliott Wave Theory to forecast the financial markets. Once we have the model and know the parameters, we optimize it using a machine learning technique called genetic algorithm. Then we test it using Walk forward optimization. The fitness function we're using for optimization and testing is the Sharpe ratio. 9 | 10 | The experiment was carried out on the EUR/USD currency pair on an hourly basis. The period was from 2015/10 to 2020/04 (including 2 training and 2 testing periods). The training periods were 18 months each (from 2015-10-15 to 2017-04-15 and from 2018-01-15 to 2019-07-15) and the testing periods were 9 months each (from 2017-04-15 to 2018-01-15 and from 2019-07-15 to 2020-04-15). 11 | 12 | The Sharpe ratio was above 3 for each training period (which is excellent). The results were mixed for the training periods. The Sharpe ratio was 1.63 for the first training period (which is really good) and -13.99 for the second period with 0 winning trades (which is really bad). 13 | 14 | One of the issue with the model during the testing periods is that it generated few trades (11 for the first testing period and 10 for second testing period). This may be due to an over optimized model which caused overfitting. We could also test the same model on different assets and different timeframes. 15 | 16 | The library is built so that it is possible to modify the trading strategy by creating modules in the different packages (indicators, optimize, trading_rules). For example, the current rule for trying to enter the market is when a trend is detected (r2 and Mann-Kendall). We could create a new module that tries to enter the market using a mean-reversion strategy like when the market is 2 standard deviations from the average price of the last 100 days (in the trading_rules package). 17 | 18 | To find more details about this project, scroll down 19 | 20 | The project structure : 21 | 22 | ``` 23 | ├── EURUSD.csv <- Data 24 | ├── LICENSE.txt <- License 25 | ├── README.md <- ReadMe document 26 | ├── __init__.py 27 | ├── charting.py <- Charting module 28 | ├── date_manip.py <- Module to manipulate date 29 | ├── entry <- Package that tries to enter the market (with different modules) 30 | │   ├── __init__.py 31 | │   └── entry_fibo.py <- Module that tries to enter the market using the Fibonacci technique 32 | ├── exit <- Package that tries to exit the market (with different modules) 33 | │   ├── __init__.py 34 | │   └── exit_fibo.py <- Module that tries to enter the market using the Fibonacci technique 35 | ├── indicator.py <- Return the values of our indicators of our choice 36 | ├── indicators <- Package that evaluates the indicators 37 | │   ├── __init__.py 38 | │   └── regression 39 | │   ├── __init__.py 40 | │   ├── linear_regression.py <- Module that evaluates the slope and r_square of a serie 41 | │   └── mann_kendall.py <- Module that assess the Mann-Kendall test 42 | ├── init_operations.py <- Module that resets the necessary values 43 | ├── initialize.py <- Module that declares hyperparamaters and parameters to optimize 44 | ├── main.py <- Main module that executes the program 45 | ├── manip_data.py <- Helper module to manipulate csv and pandas Dataframe 46 | ├── math_op.py <- Module support for mathematical operations 47 | ├── optimize <- Package with optimization techniques 48 | │   ├── __init__.py 49 | │   └── genetic_algorithm.py <- Module that uses a genetic algorithm to optimize 50 | ├── optimize_.py <- Module that runs the optimization process if desired 51 | ├── pnl.py <- Module to assess the trading strategy performance 52 | └── trading_rules <- Package with possible trading rules 53 | ├── __init__.py 54 | └── r_square_tr.py <- Module that detects buy and sell signals with r_square and Mann Kendall test 55 | 56 | ``` 57 | 58 | To see the list of hyperparameters and parameters to optimize, go to this [file](https://github.com/philos123/PyBacktesting/blob/master/initialize.py) 59 | 60 | Each .py file has its docstring, so make sure to check it out to understand the details. 61 | 62 | To find out more about [me](https://github.com/philos123) 63 | 64 | For questions or comments, please feel free to reach out on [LinkedIn](https://www.linkedin.com/in/philippe-ostiguy/?locale=en_US) 65 | 66 | ## Part 1 - DEFINING 67 | 68 | ### ---- Defining the problem ---- 69 | 70 | The goal of this project is to model the Elliott Wave Theory to forecast the financial markets. Once we have the model and know the parameters, we optimize it using a machine learning technique called genetic algorithm and test in a different period (Walk forward optimization). The fitness function used for optimization and testing is the Sharpe ratio. 71 | 72 | There is no real technique at the moment to model the Elliott Wave Theory as it is difficult to model and the modeling is highly subjective. To understand the concept of Elliott Wave Theory, refer to this [post](https://www.investopedia.com/articles/technical/111401.asp). 73 | 74 | Since the optimization space of a trading strategy can be complex, genetic algorithms are an efficient machine learning technique to find a good approximation of the optimal solution. It mimics the biological process of evolution. 75 | 76 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/genetic.jpg) 77 | 78 | ## Part 2 - DISCOVER 79 | 80 | ### ---- Loading the data ---- 81 | 82 | The experiment was carried out on the EUR/USD currency pair on an hourly basis. The period was from 2015/10 to 2020/04 (including 2 training and 2 testing periods). The training periods were 18 months each (from 2015-10-15 to 2017-04-15 and from 2018-01-15 to 2019-07-15) and the testing periods were 9 months each (from 2017-04-15 to 2018-01-15 and from 2019-07-15 to 2020-04-15). 83 | 84 | The data source for this experiment was [Dukascopy](https://www.dukascopy.com/swiss/english/marketwatch/historical/) as it required a lot of data. The program read the data in a csv format. If you want to do an experiment on a different asset and/or timeframe, make sure to load the data in the folder of your choice and change the path with the variable `self.directory` in [initialize.py](https://github.com/philos123/PyBacktesting/blob/master/initialize.py). 85 | 86 | If less data is needed for an experiment or if the experiment is carried on daily basis data, the Alpha Vantage API is a great source to get free and quality data (with certain restrictions, like a maximum API call per minute). [This](https://algotrading101.com/learn/alpha-vantage-guide/) is a great article on the Alpha Vantage API. 87 | 88 | ``` 89 | Parameters 90 | ---------- 91 | `self.directory` : str 92 | Where the the data are located for training and testing periods 93 | `self.asset` : str 94 | Name of the file where we get the data 95 | `self.is_fx` : bool 96 | Tell if `self.asset` is forex (the data don't have the same format for forex and stocks because they are 97 | from different providers). 98 | `self.dir_output` : str 99 | Where we store the results 100 | `self.name_out` : str 101 | Name of the results file name (csv) 102 | `self.start_date` : datetime object 103 | Beginning date of training and testing period. The variable is already transformed from a str to a 104 | Datetime object 105 | `self.end_date` : datetime object 106 | Ending date of training and testing period. The variable is already transformed from a str to a 107 | Datetime object 108 | 109 | self.directory = '/Users/philippeostiguy/Desktop/Trading/Programmation_python/Trading/' 110 | self.dir_output = '/Users/philippeostiguy/Desktop/Trading/Programmation_python/Trading/results/' 111 | self.name_out = 'results' 112 | self.is_fx = True 113 | self.asset = "EURUSD" 114 | self.start_date = datetime.strptime('2015-10-15', "%Y-%m-%d") 115 | self.end_date = datetime.strptime('2016-02-18', "%Y-%m-%d") 116 | ``` 117 | 118 | We can examine our data : 119 | 120 | ``` 121 | series_.head() 122 | ``` 123 | | # | Date | Open | High | Low | Adj Close | 124 | |------|---------------------|---------|---------|---------|-----------| 125 | | 96 | 2015-10-15 00:00:00 | 1.14809 | 1.14859 | 1.14785 | 1.14801 | 126 | | 97 | 2015-10-15 01:00:00 | 1.14802 | 1.14876 | 1.14788 | 1.14828 | 127 | | 98 | 2015-10-15 02:00:00 | 1.14831 | 1.14950 | 1.14768 | 1.14803 | 128 | | 99 | 2015-10-15 03:00:00 | 1.14802 | 1.14826 | 1.14254 | 1.14375 | 129 | | 100 | 2015-10-15 04:00:00 | 1.14372 | 1.14596 | 1.14335 | 1.14417 | 130 | 131 | And see the lenght, value types and if there are empty values (none) : 132 | 133 | ``` 134 | series_.info() 135 | ``` 136 | 137 | | # | Column | Non-Null Count | Dtype | 138 | |---|-----------|----------------|----------------| 139 | | 0 | Date | 28176 non-null | datetime64[ns] | 140 | | 1 | Open | 28176 non-null | float64 | 141 | | 2 | High | 28176 non-null | float64 | 142 | | 3 | Low | 28176 non-null | float64 | 143 | | 4 | Adj Close | 28176 non-null | float64 | 144 | 145 | 146 | ### ---- Cleaning the data ---- 147 | 148 | In [manip_data.py](https://github.com/philos123/PyBacktesting/blob/master/manip_data.py), we drop the nan value, if any (none) and remove the data when the market is closed with `series_.drop_duplicates(keep=False,subset=list(dup_col.keys()))` 149 | 150 | ``` 151 | series_ = series_.dropna() #drop nan values 152 | if dup_col != None: 153 | #If all values in column self.dup_col are the same, we erase them 154 | series_ = series_.drop_duplicates(keep=False,subset=list(dup_col.keys())) 155 | series_=series_.reset_index(drop=True) 156 | ``` 157 | 158 | It removed 172 data. 159 | 160 | | # | Column | Non-Null Count | Dtype | 161 | |---|-----------|----------------|----------------| 162 | | 0 | Date | 28024 non-null | datetime64[ns] | 163 | | 1 | Open | 28024 non-null | float64 | 164 | | 2 | High | 28024 non-null | float64 | 165 | | 3 | Low | 28024 non-null | float64 | 166 | | 4 | Adj Close | 28024 non-null | float64 | 167 | 168 | 169 | ### ---- Exploring the data (EDA) ---- 170 | The period was from 2015/10 to 2020/04 (including 2 training and 2 testing periods). The training periods were 18 months each (from 2015-10-15 to 2017-04-15 and from 2018-01-15 to 2019-07-15) and the testing periods were 9 months each (from 2017-04-15 to 2018-01-15 and from 2019-07-15 to 2020-04-15). We see the split on chart below. 171 | 172 | ``` 173 | x = np.linspace(pd.Timestamp(self.start_date), pd.Timestamp(self.end_date),len(self.series)) 174 | y = self.series.loc[:,self.default_data] 175 | 176 | segment1 = (x < pd.Timestamp('2017-04-15').value) 177 | segment2 = (x >= pd.Timestamp('2017-04-15').value) & (x < pd.Timestamp('2018-01-15').value) 178 | segment3 = (x >= pd.Timestamp('2018-01-15').value) & (x < pd.Timestamp('2019-07-15').value) 179 | segment4 = (x >= pd.Timestamp('2019-07-15').value) 180 | 181 | x = pd.to_datetime(x) 182 | 183 | plt.plot(x[segment1], y[segment1], '-b', lw=1) 184 | plt.plot(x[segment2], y[segment2], '-g', lw=1) 185 | plt.plot(x[segment3], y[segment3], '-b', lw=1) 186 | plt.plot(x[segment4], y[segment4], '-g', lw=1) 187 | 188 | plt.show() 189 | ``` 190 | 191 | The blue represents the 2 training periods and the green represents the 2 testing periods 192 | 193 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/period_split.png) 194 | 195 | If someone would like to do time series analysis like ARIMA (not the case here), we would have to check if the serie is stationary. The [Augmented Dickey–Fuller test](https://en.wikipedia.org/wiki/Augmented_Dickey%E2%80%93Fuller_test) does that. 196 | 197 | ``` 198 | from statsmodels.tsa.stattools import adfuller 199 | 200 | series_diff = self.series.copy() 201 | if adfuller(series_diff[self.default_data])[1] > self.p_value: 202 | raise Exception("The series is not stationary") 203 | 204 | ``` 205 | 206 | If the serie is not stationary and it's only a matter of "trend", first differencing is in general fine to make a financial time series stationary. It there is seasonality, other manipulations are required 207 | 208 | ``` 209 | import matplotlib.pyplot as plt 210 | 211 | series_diff = self.series.copy() 212 | series_diff.drop([self.date_name, self.date_ordinal_name], axis=1, inplace=True) 213 | series_diff = series_diff.diff(periods=self.period) # differentiate with previous row 214 | series_diff.loc[:(self.period - 1), :] = 0 #Make first row equal to 0 215 | series_diff.insert(0, self.date_name, self.series[self.date_name]) #re-insert the period columns 216 | 217 | plt.plot(self.series_test[self.date_name], self.series_test[self.default_data]) 218 | plt.show() 219 | ``` 220 | 221 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/stationary_series.png) 222 | 223 | ### ---- Establishing a baseline ---- 224 | 225 | To make it simple and as a first version, this strategy is meant to try to enter the market when a trend is found. Note that we might be violating the assumption that there is no autocorrelation and that the error terms should follow a normal distribution, but for for the sake of simplicity, we'll assume the absence of serial correlation and that the error terms follows a normal distribution. Also, the statistical tests are used to determine if there is a trend, not to forecast the financial market. A way around this would be to make the serie stationary. 226 | 227 | First using the Mann-Kendall test (non-parametric methods), we check if there is a trend in the market. It's better than linear regression as it does not require the data to be normally distributed or linear. In the program, it returns a 1 when there is a upward trend, -1 when there is a downward trend and 0 when there is no trend. More explanations and details about the code [here](https://github.com/philos123/PyBacktesting/blob/master/indicators/regression/mann_kendall.py) 228 | 229 | Then using the coefficient of determination (R^2), we assess the strenght of the trend. By default, the thresold value `self.r_square_level` in `initialize.py` to say that the trend is significant is 0.7. More explanations and details about the code [here](https://github.com/philos123/PyBacktesting/blob/master/indicators/regression/linear_regression.py) 230 | 231 | There is an example below with dots on chart when the r^2 is above 0.7 and Mann-Kendall is -1 or 1 232 | 233 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/Trading_rules.png) 234 | 235 | Whenever we have a trend confirmation, the program tries to enter the market using the Elliott Wave Theory in the module [entry_fibo](https://github.com/philos123/PyBacktesting/blob/master/entry/entry_fibo.py). First, the method finds local minimum and maximum in the current trend with function `self.local_extremum_()`. It checks if the values for a number of points on each side (`window`) are greater or lesser and then determine the local extremum. Assessing a local extremum on a financial time series can be tricky, but this method gives satisfactory results. 236 | 237 | ``` 238 | from scipy.signal import argrelextrema 239 | import matplotlib.pyplot as plt 240 | import numpy as np 241 | import pandas as pd 242 | 243 | cls.series=cls.series.loc[start_point:end_point,cls.default_col] 244 | cls.series=pd.DataFrame({cls.default_col: cls.series}) 245 | 246 | cls.series[min_] = cls.series.iloc[argrelextrema(cls.series.values, np.less_equal, 247 | order=window)[0]][cls.default_col] 248 | cls.series[max_] = cls.series.iloc[argrelextrema(cls.series.values, np.greater_equal, 249 | order=window)[0]][cls.default_col] 250 | cls.series[index_] = cls.series.index 251 | plt.scatter(cls.series.index, cls.series[min_], c='r') 252 | plt.scatter(cls.series.index, cls.series[max_], c='g') 253 | 254 | plt.plot(cls.series.index, cls.series[cls.default_col]) 255 | plt.ion() 256 | plt.show() 257 | 258 | ``` 259 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/Local_extremum.png) 260 | 261 | Then using `self.largest_extension()`, it sets the largest setback, which is the difference between a local maximum and the previous local minimum for a downward trend and the difference between the local maximum and and the next local minimum for an upward trend. In the above example, the largest setback in the downward trend would be the difference between the value at the index 148 (1.11292) minus the index at the index 139 (1.10818) for a result of 0.0047. This value play a key role for the entry and the exit level. 262 | 263 | When the largest setback is set, the program will try to enter the market with the function `self.try_entry()` in the module [entry_fibo](https://github.com/philos123/PyBacktesting/blob/master/entry/entry_fibo.py) 264 | 265 | Then if the system is able to enter in the market, it will exit wheter a stop loss is triggered or the profit level is reached using again the logic with the largest setback (and using the Elliott Wave Theory). Please refer to this [module](https://github.com/philos123/PyBacktesting/blob/master/exit/exit_fibo.py) for more information. 266 | 267 | Below is an example of 2 entries (buy in green) and 2 exits (sell in red). The first one has a profit and the other one has a lost : 268 | 269 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/Entry_exit.png) 270 | 271 | There a different values that can be used for the Elliott Wave Theory strategy. Again, please refer to the module [initialize.py](https://github.com/philos123/PyBacktesting/blob/master/initialize.py) to see all the parameters that can be optimized. We used the default value in this module to evaluate the preformance of the trading strategy. 272 | 273 | The metric used to evalute the preformance of the trading strategy is the Sharpe ratio. It's the most common metric used to evaluate a trading strategy. There are other useful metrics to assess the performance of a trading strategy [here](https://github.com/philos123/PyBacktesting/blob/master/pnl.py) 274 | 275 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/Sharep_ratio.gif) 276 | 277 | Sa : Sharpe ratio
278 | E : expected value
279 | Ra : asset return
280 | Rb : risk free return
281 | σa : standard deviation of the asset excess return
282 | 283 | Below we can see the result for the 4 different periods before optimization. 284 | 285 | | Date range from | 2015-10-15 to 2017-04-15 | 286 | |-----------------------|--------------------------| 287 | | Annualized return | -0.028538168293147037 | 288 | | Annualized volatility | 0.001714442062259805 | 289 | | Sharpe ratio | -16.645746695884778 | 290 | | Maximum drawdown | -0.00586379729741346 | 291 | | % win | 0.13636363636363635 | 292 | | nb_trade | 22 | 293 | 294 | | Date range from | 2017-04-15 to 2018-01-15 | 295 | |-----------------------|--------------------------| 296 | | Annualized return | -0.016042731033677482 | 297 | | Annualized volatility | 0.0029188486883811172 | 298 | | Sharpe ratio | -5.496253059481227 | 299 | | Maximum drawdown | -0.0031105801006658536 | 300 | | % win | 0.1 | 301 | | nb_trade | 10 | 302 | 303 | | Date range from | 2018-01-15 to 2019-07-15 | 304 | |-----------------------|--------------------------| 305 | | Annualized return | -0.02083109080647605 | 306 | | Annualized volatility | 0.0021923888737862 | 307 | | Sharpe ratio | -9.501549225845725 | 308 | | Maximum drawdown | -0.0042445599845946265 | 309 | | % win | 0.20689655172413793 | 310 | | nb_trade | 29 | 311 | 312 | 313 | | Date range from | 2019-07-15 to 2020-04-15 | 314 | |-----------------------|--------------------------| 315 | | Annualized return | 0.009637932127751991 | 316 | | Annualized volatility | 0.004822189306914451 | 317 | | Sharpe ratio | 1.9986631619651127 | 318 | | Maximum drawdown | -0.0047235958781114 | 319 | | % win | 0.35714285714285715 | 320 | | nb_trade | 14 | 321 | 322 | 323 | The only period where the strategy performed well was from 2019-07-15 to 2020-04-15 with a Sharpe ratio of 1.998. 324 | 325 | ## Part 3 - DEVELOPING 326 | 327 | ### ---- Creating models ---- 328 | 329 | There are several ways to optimize a trading strategy, more on that [here](https://miltonfmr.com/how-to-develop-test-and-optimize-a-trading-strategy-complete-guide/). One would be a brute-force algorithm which would test all the possible candidates. The main disadvantage is when there are several candidates, it requires a lot of memory and processing time. 330 | 331 | One good solution is the genetic algorithm which is a random-based classical evolutionary algorithm based on Charles Darwin's theory of natural evolution. The process is simple : (code in module [genetic_algorithm.py](https://github.com/philos123/PyBacktesting/blob/master/optimize/genetic_algorithm.py)) 332 | 333 | #### 1- Generate the initial population 334 | A population is composed of chromosomes or indidivuals (each individual is a solution to the problem we want to solve) and each chromosome is characterized by a set of parameters we want to optimize known as genes. In general, each gene is represented by a binary value. In our case, each gene is a paramater that we want to optimize and can take the possible value that we define in `initialize.py`. 335 | 336 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/Population.png) 337 | 338 | In general, we want the size of the population to be 1.5 to 2 times the number of genes. In our case, we have 16 parameters to optimize and the size of the population is 25 chromosomes. 339 | 340 | #### 2- Compute fitness 341 | We then evaluate the performance of each chromosome (individual) using the Sharpe ratio. It gives a score to each individual. 342 | 343 | To be consider for the next generation, each chromosome (individual) must generate at least 10 trades during the training period, otherwise it's rejected. In such case, a new chromosome is generated. 344 | 345 | #### 3- Selection 346 | We select the best candidates so that they can pass their genes to the next generations (creating children). Two pairs of indidivuals (parents) are selected based on their Sharpe ratio value (fitness score). Individuals with higher Sharpe ratio have more chance to be selected for the next generations. We use the roulette wheel selection for selecting potential indivuals for the next generation. This method gives a probability of choosing an individual proportionally to his fitness value. 347 | 348 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/Selection.png) 349 | 350 | #### 4- Genetic operators 351 | We then create the new population using genetic operators. The first one is to copy the chromosomes to the next generation with a probability of 30%. 352 | 353 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/copy_generation_.png) 354 | 355 | The second genetic operator is the crossover. It is the most significant genetic operator as it creates new offspring by exchanging genes among the best parents (previous generation). It has a 65% probability of happening. 356 | 357 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/Crossover_.png) 358 | 359 | The last one is mutation which flips (mutates) a gene from the best parents. It's an operator which prevent to get stuck too early in a local extrema. The probability is not too high to prevent the risk that an individual was close to a solution (from previous generation). It has a 5% probability of happening. 360 | 361 | ![](https://github.com/philos123/PyBacktesting/blob/master/images/mutation_.png) 362 | 363 | ### 5- Repeat 364 | We repeat step 2 to 4 for 25 generations. It's important to set a proper size of generations as a too small generation won't give a good coverage of the search space and too large is time-consuming. 365 | 366 | Another possibility would be to stop the cycle when the sharpe ratio is equal to or greater than 3 (or when it reaches a satisfactory fitness value). In this experiment, the algorithm runs for 25 generations. 367 | 368 | ### ---- Testing models ---- 369 | 370 | The experiment was carried out on the EUR/USD currency pair on an hourly basis. The period was from 2015/10 to 2020/04 (including 2 training and 2 testing periods). The training periods were each 18 months each (from 2015-10-15 to 2017-04-15 and from 2018-01-15 to 2019-07-15) and the testing periods were 9 months each (from 2017-04-15 to 2018-01-15 and from 2019-07-15 to 2020-04-15). 371 | 372 | The Sharpe ratio was above 3 for each training period (which is excellent). The results were mixed for the training periods. The Sharpe ratio was 1.63 for the first training period (which is really good) and -13.99 for the second period with 0 winning trades (which is really bad). 373 | 374 | 375 | #### Training period no 1 376 | | Date range from | 2015-10-15 to 2017-04-15 | 377 | |-----------------------|--------------------------| 378 | | Annualized return | 0.03579816749483067 | 379 | | Annualized volatility | 0.00996732375915114 | 380 | | Sharpe ratio | 3.591552593238869 | 381 | | Maximum drawdown | -0.00997144631486068 | 382 | | % win | 0.36363636363636365 | 383 | | nb_trade | 22 | 384 | 385 | 386 | #### Testing period no 1 387 | | Date range from | 2017-04-15 to 2018-01-15 | 388 | |-----------------------|--------------------------| 389 | | Annualized return | 0.014095773085682 | 390 | | Annualized volatility | 0.008626558234907 | 391 | | Sharpe ratio | 1.63399732568252 | 392 | | Maximum drawdown | -0.005521301258917 | 393 | | % win | 0.363636363636364 | 394 | | nb_trade | 11 | 395 | 396 | 397 | #### Training period no 2 398 | | Date range from | 2018-01-15 to 2019-07-15 | 399 | |-----------------------|--------------------------| 400 | | Annualized return | 0.009447731588671 | 401 | | Annualized volatility | 0.00287683168216 | 402 | | Sharpe ratio | 3.28407520233458 | 403 | | Maximum drawdown | -0.000809793667447 | 404 | | % win | 0.074074074074074 | 405 | | nb_trade | 27 | 406 | 407 | 408 | #### Testing period no 2 409 | | Date range from | 2019-07-15 to 2020-04-15 | 410 | |-----------------------|--------------------------| 411 | | Annualized return | -0.006083679046796 | 412 | | Annualized volatility | 0.000434670065322 | 413 | | Sharpe ratio | -13.9960846907784 | 414 | | Maximum drawdown | -0.001343642756988 | 415 | | % win | 0 | 416 | | nb_trade | 10 | 417 | 418 | 419 | 420 | ### ---- Summarizing ---- 421 | 422 | One of the issue with the model during the testing periods is that it generated few trades (11 for the first testing period and 10 for second testing period). This may be due to an over optimized model which caused overfitting. We could also test the same model on different assets and different timeframes. 423 | 424 | Also, this a a basic model of the Elliott Wave Theory. We could also enter the market with Fibonacci retracements and exit the market using Fibonacci extensions. 425 | --------------------------------------------------------------------------------