├── .gitignore ├── README.md ├── module.py ├── plain.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | #not adding the virtual environment to gituhub (installed through requirements.txt) 2 | venv 3 | #pycache is generated by run 4 | __pycache__ 5 | #PyCharm specific folder 6 | .idea 7 | #git specific folder 8 | .gitignore.swp 9 | #jupyter specific folder 10 | .ipynb_checkpoints 11 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # RangerOptimizerTensorflow 2 | This code is a combination of two TensorFlow addons optimizers. The code is mostly loaded from there but slightly modified. 3 | ## Use 4 | This TensorFlow optimizer combines the two tf.keras.optimizers RAdam and Lookahead into one optimizer called Ranger. This can instantly be fed into the tf.keras.model.fit / tf.keras.model.fit_generator method to fit a model using this optimizer. All the setting of the hyper parameters can therefore be done in a single step. 5 | ## Setup 6 | The python files were created for python version 3.7, although it might also work for past or future versions. 7 | To use this class, some python modules need to be installed first. Using pip the packages can be installed by either typing 8 | pip install -r requirements.txt 9 | in terminal, if the requirements.txt file exists in the current working directory or by typing 10 | pip install tensorflow==2.0.0 tensorflow-addons==0.6.0 11 | into the terminal (!python and pip need to be installed first, the recommended version for pip is at least 19.3.1). The versions of the modules listed above were used at the time of the creation of these files but future versions of these modules might alos work. Another way to install these packages is by using conda. 12 | ## Code 13 | For using the optimizer there are two options: 14 | 1. Put the code straight into a python file:
15 | For that the code from the file [plain.py](plain.py) should be copied into the python file. 16 | 2. Importing the class from a different python file:
17 | For that the file [module.py](module.py) should be inserted into the project folder in which the executed file lies and imported at the top of the executed file:
18 | from module import Ranger 19 | 20 | In the following python code the following elements should be included:
21 | ```python 22 | # load the required modules 23 | import tensorflow.keras as k 24 | 25 | # load training data, transforms, any other setup for model 26 | [...] 27 | 28 | # define the tf.keras model 29 | model = k.model.Sequential() 30 | [...] # using the model.add([...]) function new layers can be added to the model 31 | 32 | # compile and fit the model with Ranger optimizer 33 | optimizer = Ranger([...]) # feed arguments of Ranger optimizer into creation function 34 | model.compile(optimizer, [...]) # add aditional parameters 35 | model.fit([...]) / model.fit_generator([...]) # fit the model using array data or a generator 36 | 37 | model.save('path/to/model/name.h5') # save the model (optional but sensible) 38 | ``` 39 | The recommended way of using this class is by importing it as a module because docstrings are provided to document the module. In the plain.py file the documentation is not present for shortening the code. 40 | ## Credits 41 | https://arxiv.org/abs/1908.03265 (RAdam paper)
42 | https://arxiv.org/abs/1907.08610 (Lookahead paper)
43 | https://github.com/tensorflow/addons/blob/v0.6.0/tensorflow_addons/optimizers/lookahead.py#L25-L171 (TensorFlow implementation of Lookahead optimizer)
44 | https://github.com/tensorflow/addons/blob/v0.6.0/tensorflow_addons/optimizers/rectified_adam.py#L25-L306 (TensorFlow implementation of RAdam) 45 | -------------------------------------------------------------------------------- /module.py: -------------------------------------------------------------------------------- 1 | import tensorflow_addons as tfa 2 | 3 | 4 | def Ranger(sync_period=6, 5 | slow_step_size=0.5, 6 | learning_rate=0.001, 7 | beta_1=0.9, 8 | beta_2=0.999, 9 | epsilon=1e-7, 10 | weight_decay=0., 11 | amsgrad=False, 12 | sma_threshold=5.0, 13 | total_steps=0, 14 | warmup_proportion=0.1, 15 | min_lr=0., 16 | name="Ranger"): 17 | """ 18 | function returning a tf.keras.optimizers.Optimizer object 19 | returned optimizer is a Ranger optimizer 20 | Ranger is an optimizer combining RAdam (https://arxiv.org/abs/1908.03265) and Lookahead (https://arxiv.org/abs/1907.0861) 21 | returned optimizer can be fed into the model.compile method of a tf.keras model as an optimizer 22 | ... 23 | Attributes 24 | ---------- 25 | learning_rate : float 26 | step size to take for RAdam optimizer (depending on gradient) 27 | beta_1 : float 28 | parameter that specifies the exponentially moving average length for momentum (0<=beta_1<=1) 29 | beta_2 : float 30 | parameter that specifies the exponentially moving average length for variance (0<=beta_2<=1) 31 | epsilon : float 32 | small number to cause stability for variance division 33 | weight_decay : float 34 | number with which the weights of the model are multiplied each iteration (0<=weight_decay<=1) 35 | amsgrad : bool 36 | parameter that specifies whether to use amsgrad version of Adam (https://arxiv.org/abs/1904.03590) 37 | total_steps : int 38 | total number of training steps 39 | warmup_proportion : float 40 | the proportion of updated over which the learning rate is increased from min learning rate to learning rate (0<=warmup_proportion<=1) 41 | min_lr : float 42 | learning rate at which the optimizer starts 43 | k : int 44 | parameter that specifies after how many steps the lookahead step backwards should be applied 45 | alpha : float 46 | parameter that specifies how much in the direction of the fast weights should be moved (0<=alpha<=1) 47 | """ 48 | # create RAdam optimizer 49 | inner = tfa.optimizers.RectifiedAdam(learning_rate, beta_1, beta_2, epsilon, weight_decay, amsgrad, sma_threshold, total_steps, warmup_proportion, min_lr, name) 50 | # feed RAdam optimizer into lookahead operation 51 | optim = tfa.optimizers.Lookahead(inner, sync_period, slow_step_size, name) 52 | return optim 53 | -------------------------------------------------------------------------------- /plain.py: -------------------------------------------------------------------------------- 1 | import tensorflow_addons as tfa 2 | 3 | 4 | def Ranger(sync_period=6, 5 | slow_step_size=0.5, 6 | learning_rate=0.001, 7 | beta_1=0.9, 8 | beta_2=0.999, 9 | epsilon=1e-7, 10 | weight_decay=0., 11 | amsgrad=False, 12 | sma_threshold=5.0, 13 | total_steps=0, 14 | warmup_proportion=0.1, 15 | min_lr=0., 16 | name="Ranger"): 17 | inner = tfa.optimizers.RectifiedAdam(learning_rate, beta_1, beta_2, epsilon, weight_decay, amsgrad, sma_threshold, total_steps, warmup_proportion, min_lr, name) 18 | optim = tfa.optimizers.Lookahead(inner, sync_period, slow_step_size, name) 19 | return optim 20 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | absl-py==0.8.1 2 | astor==0.8.0 3 | cachetools==3.1.1 4 | certifi==2024.7.4 5 | chardet==3.0.4 6 | gast==0.2.2 7 | google-auth==1.7.1 8 | google-auth-oauthlib==0.4.1 9 | google-pasta==0.1.8 10 | grpcio==1.53.2 11 | h5py==2.10.0 12 | idna==3.7 13 | Keras-Applications==1.0.8 14 | Keras-Preprocessing==1.1.0 15 | Markdown==3.1.1 16 | numpy==1.22.0 17 | oauthlib==3.1.0 18 | opt-einsum==3.1.0 19 | protobuf==3.18.3 20 | pyasn1==0.4.8 21 | pyasn1-modules==0.2.7 22 | requests==2.32.2 23 | requests-oauthlib==1.3.0 24 | rsa==4.7 25 | six==1.13.0 26 | tensorboard==2.0.2 27 | tensorflow==2.12.1 28 | tensorflow-addons==0.6.0 29 | tensorflow-estimator==2.0.1 30 | tensorflow-gpu==2.12.0 31 | termcolor==1.1.0 32 | urllib3==1.26.19 33 | Werkzeug==3.0.3 34 | wrapt==1.11.2 35 | --------------------------------------------------------------------------------