├── ranger-init.jpg
├── ranger
    ├── __init__.py
    ├── rangerqh.py
    ├── ranger.py
    ├── ranger913A.py
    └── ranger2020.py
├── ranger-with-gc-options.jpg
├── setup.py
├── README.md
└── LICENSE


/ranger-init.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/lessw2020/Ranger-Deep-Learning-Optimizer/HEAD/ranger-init.jpg


--------------------------------------------------------------------------------
/ranger/__init__.py:
--------------------------------------------------------------------------------
1 | from .ranger import Ranger
2 | from .ranger913A import RangerVA
3 | from .rangerqh import RangerQH
4 | 


--------------------------------------------------------------------------------
/ranger-with-gc-options.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/lessw2020/Ranger-Deep-Learning-Optimizer/HEAD/ranger-with-gc-options.jpg


--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
 1 | #!/usr/bin/env python
 2 | 
 3 | import os
 4 | from setuptools import find_packages, setup
 5 | 
 6 | 
 7 | def read(fname):
 8 |     with open(os.path.join(os.path.dirname(__file__), fname)) as f:
 9 |         return f.read()
10 | 
11 | 
12 | setup(
13 |     name='ranger',
14 |     version='0.1.dev0',
15 |     packages=find_packages(
16 |         exclude=['tests', '*.tests', '*.tests.*', 'tests.*']
17 |     ),
18 |     package_dir={'ranger': os.path.join('.', 'ranger')},
19 |     description='Ranger - a synergistic optimizer using RAdam '
20 |                 '(Rectified Adam) and LookAhead in one codebase ',
21 |     long_description=read('README.md'),
22 |     long_description_content_type='text/markdown',
23 |     author='Less Wright',
24 |     license='Apache',
25 |     install_requires=['torch']
26 | )
27 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Ranger-Deep-Learning-Optimizer 
 2 | </br>
 3 | Ranger - a synergistic optimizer combining RAdam (Rectified Adam) and LookAhead, and now GC (gradient centralization) in one optimizer.
 4 | </br>
 5 | 
 6 | #### quick note - Ranger21 is now in beta and is Ranger with a host of new improvements.  
 7 | Recommend you compare results with Ranger21:  https://github.com/lessw2020/Ranger21
 8 | 
 9 | ### Latest version 20.9.4 - updates Gradient Centralization to GC2 (thanks to GC developer) and removes addcmul_ deprecation warnings in PyTorch 1.60. 
10 | </br> </br>
11 | *Latest version is in ranger2020.py - looking at a few other additions before integrating into the main ranger.py.  
12 | 
13 | What is Gradient Centralization? = "GC can be viewed as a projected gradient descent method with a constrained loss function. The Lipschitzness of the constrained loss function and its gradient is better so that the training process becomes more efficient and stable."  Source paper:  https://arxiv.org/abs/2004.01461v2
14 | </br>
15 | Ranger now uses Gradient Centralization by default, and applies it to all conv and fc layers by default.  However, everything is customizable so you can test with and without on your own datasets.  (Turn on off via "use_gc" flag at init).
16 | </br>
17 | ### Best training results - use a 75% flat lr, then step down and run lower lr for 25%, or cosine descend last 25%. 
18 | 
19 | </br> Per extensive testing - It's important to note that simply running one learning rate the entire time will not produce optimal results.  
20 | Effectively Ranger will end up 'hovering' around the optimal zone, but can't descend into it unless it has some additional run time at a lower rate to drop down into the optimal valley.
21 | 
22 | ### Full customization at init: 
23 | <div  align="center"><img src="ranger-with-gc-options.jpg" height="80%" width="80%" alt=""/></div>
24 | </br>
25 | Ranger will now print out id and gc settings at init so you can confirm the optimizer settings at train time:
26 | <div  align="center"><img src="ranger-init.jpg" height="80%" width="80%" alt=""/></div>
27 | 
28 | /////////////////////
29 | 
30 | Medium article with more info:  
31 | https://medium.com/@lessw/new-deep-learning-optimizer-ranger-synergistic-combination-of-radam-lookahead-for-the-best-of-2dc83f79a48d
32 | 
33 | Multiple updates:
34 | 1 - Ranger is the optimizer we used to beat the high scores for 12 different categories on the FastAI leaderboards!  (Previous records all held with AdamW optimizer).
35 | 
36 | 2 - Highly recommend combining Ranger with: Mish activation function, and flat+ cosine anneal training curve.
37 | 
38 | 3 - Based on that, also found .95 is better than .90 for beta1 (momentum) param (ala betas=(0.95, 0.999)).
39 | 
40 | Fixes:
41 | 1 - Differential Group learning rates now supported.  This was fix in RAdam and ported here thanks to @sholderbach.
42 | 2 - save and then load may leave first run weights stranded in memory, slowing down future runs = fixed.
43 | 
44 | ### Installation
45 | Clone the repo, cd into it and install it in editable mode (`-e` option).
46 | That way, these is no more need to re-install the package after modification.
47 | ```bash
48 | git clone https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
49 | cd Ranger-Deep-Learning-Optimizer
50 | pip install -e . 
51 | ```
52 | 
53 | ### Usage 
54 | ```python
55 | from ranger import Ranger  # this is from ranger.py
56 | from ranger import RangerVA  # this is from ranger913A.py
57 | from ranger import RangerQH  # this is from rangerqh.py
58 | 
59 | # Define your model
60 | model = ...
61 | # Each of the Ranger, RangerVA, RangerQH have different parameters.
62 | optimizer = Ranger(model.parameters(), **kwargs)
63 | ```
64 | Usage and notebook to test are available here:
65 | https://github.com/lessw2020/Ranger-Mish-ImageWoof-5
66 | 
67 | ### Citing this work
68 | 
69 | We recommend you use the following to cite Ranger in your publications:
70 | 
71 | ```
72 | @misc{Ranger,
73 |   author = {Wright, Less},
74 |   title = {Ranger - a synergistic optimizer.},
75 |   year = {2019},
76 |   publisher = {GitHub},
77 |   journal = {GitHub repository},
78 |   howpublished = {\url{https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer}}
79 | }
80 | ```
81 | 


--------------------------------------------------------------------------------
/ranger/rangerqh.py:
--------------------------------------------------------------------------------
  1 | # RangerQH - @lessw2020 github
  2 | # Combines Quasi Hyperbolic momentum with Hinton Lookahead.
  3 | 
  4 | # https://arxiv.org/abs/1810.06801v4  (QH paper)
  5 | # #Lookahead paper --> MZhang,G Hinton  https://arxiv.org/abs/1907.08610
  6 | 
  7 | 
  8 | 
  9 | 
 10 | # Some portions = Copyright (c) Facebook, Inc. and its affiliates.
 11 | #
 12 | # This source code is licensed under the MIT license found in the
 13 | # LICENSE file in the root directory of this source tree.
 14 | 
 15 | import torch
 16 | from torch.optim.optimizer import Optimizer
 17 | 
 18 | #from ..common import param_conv
 19 | 
 20 | 
 21 | class RangerQH(Optimizer):
 22 |     r"""Implements the QHAdam optimization algorithm `(Ma and Yarats, 2019)`_.
 23 |     Along with Hinton/Zhang Lookahead.
 24 |     Args:
 25 |         params (iterable):
 26 |             iterable of parameters to optimize or dicts defining parameter
 27 |             groups
 28 |         lr (float, optional): learning rate (:math:`\alpha` from the paper)
 29 |             (default: 1e-3)
 30 |         betas (Tuple[float, float], optional): coefficients used for computing
 31 |             running averages of the gradient and its square
 32 |             (default: (0.9, 0.999))
 33 |         nus (Tuple[float, float], optional): immediate discount factors used to
 34 |             estimate the gradient and its square
 35 |             (default: (1.0, 1.0))
 36 |         eps (float, optional): term added to the denominator to improve
 37 |             numerical stability
 38 |             (default: 1e-8)
 39 |         weight_decay (float, optional): weight decay (default: 0.0)
 40 |         decouple_weight_decay (bool, optional): whether to decouple the weight
 41 |             decay from the gradient-based optimization step
 42 |             (default: False)
 43 |     Example:
 44 |         >>> optimizer = qhoptim.pyt.QHAdam(
 45 |         ...     model.parameters(),
 46 |         ...     lr=3e-4, nus=(0.8, 1.0), betas=(0.99, 0.999))
 47 |         >>> optimizer.zero_grad()
 48 |         >>> loss_fn(model(input), target).backward()
 49 |         >>> optimizer.step()
 50 |     .. _`(Ma and Yarats, 2019)`: https://arxiv.org/abs/1810.06801
 51 |     """
 52 | 
 53 |     def __init__(
 54 |         self,
 55 |         params,
 56 |         lr=1e-3,
 57 |         betas=(0.9, 0.999),
 58 |         nus=(.7, 1.0),
 59 |         weight_decay=0.0,
 60 |         k=6,
 61 |         alpha=.5,
 62 |         decouple_weight_decay=False,
 63 |         eps=1e-8,
 64 |     ):
 65 |         if not 0.0 <= lr:
 66 |             raise ValueError("Invalid learning rate: {}".format(lr))
 67 |         if not 0.0 <= eps:
 68 |             raise ValueError("Invalid epsilon value: {}".format(eps))
 69 |         if not 0.0 <= betas[0] < 1.0:
 70 |             raise ValueError("Invalid beta parameter at index 0: {}".format(betas[0]))
 71 |         if not 0.0 <= betas[1] < 1.0:
 72 |             raise ValueError("Invalid beta parameter at index 1: {}".format(betas[1]))
 73 |         if weight_decay < 0.0:
 74 |             raise ValueError("Invalid weight_decay value: {}".format(weight_decay))
 75 | 
 76 |         defaults = {
 77 |             "lr": lr,
 78 |             "betas": betas,
 79 |             "nus": nus,
 80 |             "weight_decay": weight_decay,
 81 |             "decouple_weight_decay": decouple_weight_decay,
 82 |             "eps": eps,
 83 |         }
 84 |         super().__init__(params, defaults)
 85 |         
 86 |         #look ahead params
 87 |         self.alpha = alpha
 88 |         self.k = k         
 89 |         
 90 | 
 91 |     def step(self, closure=None):
 92 |         """Performs a single optimization step.
 93 |         Args:
 94 |             closure (callable, optional):
 95 |                 A closure that reevaluates the model and returns the loss.
 96 |         """
 97 |         loss = None
 98 |         if closure is not None:
 99 |             loss = closure()
100 | 
101 |         for group in self.param_groups:
102 |             lr = group["lr"]
103 |             beta1, beta2 = group["betas"]
104 |             nu1, nu2 = group["nus"]
105 |             weight_decay = group["weight_decay"]
106 |             decouple_weight_decay = group["decouple_weight_decay"]
107 |             eps = group["eps"]
108 | 
109 |             for p in group["params"]:
110 |                 if p.grad is None:
111 |                     continue
112 | 
113 |                 d_p = p.grad.data
114 |                 if d_p.is_sparse:
115 |                     raise RuntimeError("QHAdam does not support sparse gradients")
116 | 
117 |                 
118 | 
119 |                 if weight_decay != 0:
120 |                     if decouple_weight_decay:
121 |                         p.data.mul_(1 - lr * weight_decay)
122 |                     else:
123 |                         d_p.add_(weight_decay, p.data)
124 | 
125 |                 d_p_sq = d_p.mul(d_p)
126 |                 
127 |                 #prep for saved param loading
128 |                 param_state = self.state[p]
129 | 
130 |                 if len(param_state) == 0:
131 |                     param_state["beta1_weight"] = 0.0
132 |                     param_state["beta2_weight"] = 0.0
133 |                     param_state['step'] = 0
134 |                     param_state["exp_avg"] = torch.zeros_like(p.data)
135 |                     param_state["exp_avg_sq"] = torch.zeros_like(p.data)
136 |                     #look ahead weight storage now in state dict 
137 |                     param_state['slow_buffer'] = torch.empty_like(p.data)
138 |                     param_state['slow_buffer'].copy_(p.data)
139 |                 
140 |                 
141 |                 param_state['step'] += 1                
142 | 
143 |                 param_state["beta1_weight"] = 1.0 + beta1 * param_state["beta1_weight"]
144 |                 param_state["beta2_weight"] = 1.0 + beta2 * param_state["beta2_weight"]
145 | 
146 |                 beta1_weight = param_state["beta1_weight"]
147 |                 beta2_weight = param_state["beta2_weight"]
148 |                 exp_avg = param_state["exp_avg"]
149 |                 exp_avg_sq = param_state["exp_avg_sq"]
150 | 
151 |                 beta1_adj = 1.0 - (1.0 / beta1_weight)
152 |                 beta2_adj = 1.0 - (1.0 / beta2_weight)
153 |                 exp_avg.mul_(beta1_adj).add_(1.0 - beta1_adj, d_p)
154 |                 exp_avg_sq.mul_(beta2_adj).add_(1.0 - beta2_adj, d_p_sq)
155 | 
156 |                 avg_grad = exp_avg.mul(nu1)
157 |                 if nu1 != 1.0:
158 |                     avg_grad.add_(1.0 - nu1, d_p)
159 | 
160 |                 avg_grad_rms = exp_avg_sq.mul(nu2)
161 |                 if nu2 != 1.0:
162 |                     avg_grad_rms.add_(1.0 - nu2, d_p_sq)
163 |                 avg_grad_rms.sqrt_()
164 |                 if eps != 0.0:
165 |                     avg_grad_rms.add_(eps)
166 | 
167 |                 p.data.addcdiv_(-lr, avg_grad, avg_grad_rms)
168 |                 
169 |                 #integrated look ahead...
170 |                 #we do it at the param level instead of group level
171 |                 if param_state['step'] % self.k ==0: #group['k'] == 0:
172 |                     slow_p = param_state['slow_buffer'] #get access to slow param tensor
173 |                     slow_p.add_(self.alpha, p.data - slow_p)  #(fast weights - slow weights) * alpha
174 |                     p.data.copy_(slow_p)  #copy interpolated weights to RAdam param tensor
175 |                 
176 | 
177 |         return loss
178 | 
179 |     @classmethod
180 |     def _params_to_dict(cls, params):
181 |         return {"lr": params.alpha, "nus": (params.nu1, params.nu2), "betas": (params.beta1, params.beta2)}
182 | 
183 | 


--------------------------------------------------------------------------------
/ranger/ranger.py:
--------------------------------------------------------------------------------
  1 | # Ranger deep learning optimizer - RAdam + Lookahead + Gradient Centralization, combined into one optimizer.
  2 | 
  3 | # https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
  4 | # and/or
  5 | # https://github.com/lessw2020/Best-Deep-Learning-Optimizers
  6 | 
  7 | # Ranger has now been used to capture 12 records on the FastAI leaderboard.
  8 | 
  9 | # This version = 20.4.11
 10 | 
 11 | # Credits:
 12 | # Gradient Centralization --> https://arxiv.org/abs/2004.01461v2 (a new optimization technique for DNNs), github:  https://github.com/Yonghongwei/Gradient-Centralization
 13 | # RAdam -->  https://github.com/LiyuanLucasLiu/RAdam
 14 | # Lookahead --> rewritten by lessw2020, but big thanks to Github @LonePatient and @RWightman for ideas from their code.
 15 | # Lookahead paper --> MZhang,G Hinton  https://arxiv.org/abs/1907.08610
 16 | 
 17 | # summary of changes:
 18 | # 4/11/20 - add gradient centralization option.  Set new testing benchmark for accuracy with it, toggle with use_gc flag at init.
 19 | # full code integration with all updates at param level instead of group, moves slow weights into state dict (from generic weights),
 20 | # supports group learning rates (thanks @SHolderbach), fixes sporadic load from saved model issues.
 21 | # changes 8/31/19 - fix references to *self*.N_sma_threshold;
 22 | # changed eps to 1e-5 as better default than 1e-8.
 23 | 
 24 | import math
 25 | import torch
 26 | from torch.optim.optimizer import Optimizer, required
 27 | 
 28 | 
 29 | class Ranger(Optimizer):
 30 | 
 31 |     def __init__(self, params, lr=1e-3,                       # lr
 32 |                  alpha=0.5, k=6, N_sma_threshhold=5,           # Ranger options
 33 |                  betas=(.95, 0.999), eps=1e-5, weight_decay=0,  # Adam options
 34 |                  # Gradient centralization on or off, applied to conv layers only or conv + fc layers
 35 |                  use_gc=True, gc_conv_only=False
 36 |                  ):
 37 | 
 38 |         # parameter checks
 39 |         if not 0.0 <= alpha <= 1.0:
 40 |             raise ValueError(f'Invalid slow update rate: {alpha}')
 41 |         if not 1 <= k:
 42 |             raise ValueError(f'Invalid lookahead steps: {k}')
 43 |         if not lr > 0:
 44 |             raise ValueError(f'Invalid Learning Rate: {lr}')
 45 |         if not eps > 0:
 46 |             raise ValueError(f'Invalid eps: {eps}')
 47 | 
 48 |         # parameter comments:
 49 |         # beta1 (momentum) of .95 seems to work better than .90...
 50 |         # N_sma_threshold of 5 seems better in testing than 4.
 51 |         # In both cases, worth testing on your dataset (.90 vs .95, 4 vs 5) to make sure which works best for you.
 52 | 
 53 |         # prep defaults and init torch.optim base
 54 |         defaults = dict(lr=lr, alpha=alpha, k=k, step_counter=0, betas=betas,
 55 |                         N_sma_threshhold=N_sma_threshhold, eps=eps, weight_decay=weight_decay)
 56 |         super().__init__(params, defaults)
 57 | 
 58 |         # adjustable threshold
 59 |         self.N_sma_threshhold = N_sma_threshhold
 60 | 
 61 |         # look ahead params
 62 | 
 63 |         self.alpha = alpha
 64 |         self.k = k
 65 | 
 66 |         # radam buffer for state
 67 |         self.radam_buffer = [[None, None, None] for ind in range(10)]
 68 | 
 69 |         # gc on or off
 70 |         self.use_gc = use_gc
 71 | 
 72 |         # level of gradient centralization
 73 |         self.gc_gradient_threshold = 3 if gc_conv_only else 1
 74 | 
 75 |         print(
 76 |             f"Ranger optimizer loaded. \nGradient Centralization usage = {self.use_gc}")
 77 |         if (self.use_gc and self.gc_gradient_threshold == 1):
 78 |             print(f"GC applied to both conv and fc layers")
 79 |         elif (self.use_gc and self.gc_gradient_threshold == 3):
 80 |             print(f"GC applied to conv layers only")
 81 | 
 82 |     def __setstate__(self, state):
 83 |         print("set state called")
 84 |         super(Ranger, self).__setstate__(state)
 85 | 
 86 |     def step(self, closure=None):
 87 |         loss = None
 88 |         # note - below is commented out b/c I have other work that passes back the loss as a float, and thus not a callable closure.
 89 |         # Uncomment if you need to use the actual closure...
 90 | 
 91 |         # if closure is not None:
 92 |         #loss = closure()
 93 | 
 94 |         # Evaluate averages and grad, update param tensors
 95 |         for group in self.param_groups:
 96 | 
 97 |             for p in group['params']:
 98 |                 if p.grad is None:
 99 |                     continue
100 |                 grad = p.grad.data.float()
101 | 
102 |                 if grad.is_sparse:
103 |                     raise RuntimeError(
104 |                         'Ranger optimizer does not support sparse gradients')
105 | 
106 |                 p_data_fp32 = p.data.float()
107 | 
108 |                 state = self.state[p]  # get state dict for this param
109 | 
110 |                 if len(state) == 0:  # if first time to run...init dictionary with our desired entries
111 |                     # if self.first_run_check==0:
112 |                     # self.first_run_check=1
113 |                     #print("Initializing slow buffer...should not see this at load from saved model!")
114 |                     state['step'] = 0
115 |                     state['exp_avg'] = torch.zeros_like(p_data_fp32)
116 |                     state['exp_avg_sq'] = torch.zeros_like(p_data_fp32)
117 | 
118 |                     # look ahead weight storage now in state dict
119 |                     state['slow_buffer'] = torch.empty_like(p.data)
120 |                     state['slow_buffer'].copy_(p.data)
121 | 
122 |                 else:
123 |                     state['exp_avg'] = state['exp_avg'].type_as(p_data_fp32)
124 |                     state['exp_avg_sq'] = state['exp_avg_sq'].type_as(
125 |                         p_data_fp32)
126 | 
127 |                 # begin computations
128 |                 exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
129 |                 beta1, beta2 = group['betas']
130 | 
131 |                 # GC operation for Conv layers and FC layers
132 |                 if grad.dim() > self.gc_gradient_threshold:
133 |                     grad.add_(-grad.mean(dim=tuple(range(1, grad.dim())), keepdim=True))
134 | 
135 |                 state['step'] += 1
136 | 
137 |                 # compute variance mov avg
138 |                 exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
139 |                 # compute mean moving avg
140 |                 exp_avg.mul_(beta1).add_(1 - beta1, grad)
141 | 
142 |                 buffered = self.radam_buffer[int(state['step'] % 10)]
143 | 
144 |                 if state['step'] == buffered[0]:
145 |                     N_sma, step_size = buffered[1], buffered[2]
146 |                 else:
147 |                     buffered[0] = state['step']
148 |                     beta2_t = beta2 ** state['step']
149 |                     N_sma_max = 2 / (1 - beta2) - 1
150 |                     N_sma = N_sma_max - 2 * \
151 |                         state['step'] * beta2_t / (1 - beta2_t)
152 |                     buffered[1] = N_sma
153 |                     if N_sma > self.N_sma_threshhold:
154 |                         step_size = math.sqrt((1 - beta2_t) * (N_sma - 4) / (N_sma_max - 4) * (
155 |                             N_sma - 2) / N_sma * N_sma_max / (N_sma_max - 2)) / (1 - beta1 ** state['step'])
156 |                     else:
157 |                         step_size = 1.0 / (1 - beta1 ** state['step'])
158 |                     buffered[2] = step_size
159 | 
160 |                 if group['weight_decay'] != 0:
161 |                     p_data_fp32.add_(-group['weight_decay']
162 |                                      * group['lr'], p_data_fp32)
163 | 
164 |                 # apply lr
165 |                 if N_sma > self.N_sma_threshhold:
166 |                     denom = exp_avg_sq.sqrt().add_(group['eps'])
167 |                     p_data_fp32.addcdiv_(-step_size *
168 |                                          group['lr'], exp_avg, denom)
169 |                 else:
170 |                     p_data_fp32.add_(-step_size * group['lr'], exp_avg)
171 | 
172 |                 p.data.copy_(p_data_fp32)
173 | 
174 |                 # integrated look ahead...
175 |                 # we do it at the param level instead of group level
176 |                 if state['step'] % group['k'] == 0:
177 |                     # get access to slow param tensor
178 |                     slow_p = state['slow_buffer']
179 |                     # (fast weights - slow weights) * alpha
180 |                     slow_p.add_(self.alpha, p.data - slow_p)
181 |                     # copy interpolated weights to RAdam param tensor
182 |                     p.data.copy_(slow_p)
183 | 
184 |         return loss
185 | 


--------------------------------------------------------------------------------
/ranger/ranger913A.py:
--------------------------------------------------------------------------------
  1 | # Ranger deep learning optimizer - RAdam + Lookahead + calibrated adaptive LR combined.
  2 | # https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
  3 | 
  4 | # Ranger has now been used to capture 12 records on the FastAI leaderboard.
  5 | 
  6 | #This version = 9.13.19A  
  7 | 
  8 | #Credits:
  9 | #RAdam -->  https://github.com/LiyuanLucasLiu/RAdam
 10 | #Lookahead --> rewritten by lessw2020, but big thanks to Github @LonePatient and @RWightman for ideas from their code.
 11 | #Lookahead paper --> MZhang,G Hinton  https://arxiv.org/abs/1907.08610
 12 | # Calibrated anisotropic adaptive learning rates - https://arxiv.org/abs/1908.00700v2
 13 | 
 14 | #summary of changes: 
 15 | #full code integration with all updates at param level instead of group, moves slow weights into state dict (from generic weights), 
 16 | #supports group learning rates (thanks @SHolderbach), fixes sporadic load from saved model issues.
 17 | #changes 8/31/19 - fix references to *self*.N_sma_threshold; 
 18 |                 #changed eps to 1e-5 as better default than 1e-8.
 19 | 
 20 | import math
 21 | import torch
 22 | from torch.optim.optimizer import Optimizer, required
 23 | import itertools as it
 24 | 
 25 | 
 26 | 
 27 | class RangerVA(Optimizer):
 28 | 
 29 |     def __init__(self, params, lr=1e-3, alpha=0.5, k=6, n_sma_threshhold=5, betas=(.95,0.999), 
 30 |                  eps=1e-5, weight_decay=0, amsgrad=True, transformer='softplus', smooth=50,
 31 |                  grad_transformer='square'):
 32 |         #parameter checks
 33 |         if not 0.0 <= alpha <= 1.0:
 34 |             raise ValueError(f'Invalid slow update rate: {alpha}')
 35 |         if not 1 <= k:
 36 |             raise ValueError(f'Invalid lookahead steps: {k}')
 37 |         if not lr > 0:
 38 |             raise ValueError(f'Invalid Learning Rate: {lr}')
 39 |         if not eps > 0:
 40 |             raise ValueError(f'Invalid eps: {eps}')
 41 | 
 42 |         #parameter comments:
 43 |         # beta1 (momentum) of .95 seems to work better than .90...
 44 |         #N_sma_threshold of 5 seems better in testing than 4.
 45 |         #In both cases, worth testing on your dataset (.90 vs .95, 4 vs 5) to make sure which works best for you.
 46 | 
 47 |         #prep defaults and init torch.optim base
 48 |         defaults = dict(lr=lr, alpha=alpha, k=k, step_counter=0, betas=betas, 
 49 |                         n_sma_threshhold=n_sma_threshhold, eps=eps, weight_decay=weight_decay,
 50 |                         smooth=smooth, transformer=transformer, grad_transformer=grad_transformer,
 51 |                        amsgrad=amsgrad)
 52 |         super().__init__(params,defaults)
 53 | 
 54 |         #adjustable threshold
 55 |         self.n_sma_threshhold = n_sma_threshhold   
 56 | 
 57 |         #look ahead params
 58 |         self.alpha = alpha
 59 |         self.k = k 
 60 | 
 61 |         #radam buffer for state
 62 |         self.radam_buffer = [[None,None,None] for ind in range(10)]
 63 | 
 64 |         #self.first_run_check=0
 65 | 
 66 |         #lookahead weights
 67 |         #9/2/19 - lookahead param tensors have been moved to state storage.  
 68 |         #This should resolve issues with load/save where weights were left in GPU memory from first load, slowing down future runs.
 69 | 
 70 |         #self.slow_weights = [[p.clone().detach() for p in group['params']]
 71 |         #                     for group in self.param_groups]
 72 | 
 73 |         #don't use grad for lookahead weights
 74 |         #for w in it.chain(*self.slow_weights):
 75 |         #    w.requires_grad = False
 76 | 
 77 |     def __setstate__(self, state):
 78 |         print("set state called")
 79 |         super(RangerVA, self).__setstate__(state)
 80 | 
 81 | 
 82 |     def step(self, closure=None):
 83 |         loss = None
 84 |         #note - below is commented out b/c I have other work that passes back the loss as a float, and thus not a callable closure.  
 85 |         #Uncomment if you need to use the actual closure...
 86 | 
 87 |         #if closure is not None:
 88 |             #loss = closure()
 89 | 
 90 |         #Evaluate averages and grad, update param tensors
 91 |         for group in self.param_groups:
 92 | 
 93 |             for p in group['params']:
 94 |                 if p.grad is None:
 95 |                     continue
 96 |                 grad = p.grad.data.float()
 97 |                 if grad.is_sparse:
 98 |                     raise RuntimeError('Ranger optimizer does not support sparse gradients')
 99 |                 
100 |                 amsgrad = group['amsgrad']
101 |                 smooth = group['smooth']
102 |                 grad_transformer = group['grad_transformer']
103 | 
104 |                 p_data_fp32 = p.data.float()
105 | 
106 |                 state = self.state[p]  #get state dict for this param
107 | 
108 |                 if len(state) == 0:   #if first time to run...init dictionary with our desired entries
109 |                     #if self.first_run_check==0:
110 |                         #self.first_run_check=1
111 |                         #print("Initializing slow buffer...should not see this at load from saved model!")
112 |                     state['step'] = 0
113 |                     state['exp_avg'] = torch.zeros_like(p_data_fp32)
114 |                     state['exp_avg_sq'] = torch.zeros_like(p_data_fp32)
115 |                     if amsgrad:
116 |                         # Maintains max of all exp. moving avg. of sq. grad. values
117 |                         state['max_exp_avg_sq'] = torch.zeros_like(p.data)                    
118 | 
119 |                     #look ahead weight storage now in state dict 
120 |                     state['slow_buffer'] = torch.empty_like(p.data)
121 |                     state['slow_buffer'].copy_(p.data)
122 | 
123 |                 else:
124 |                     state['exp_avg'] = state['exp_avg'].type_as(p_data_fp32)
125 |                     state['exp_avg_sq'] = state['exp_avg_sq'].type_as(p_data_fp32)
126 |                                       
127 | 
128 |                 #begin computations 
129 |                 exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
130 |                 beta1, beta2 = group['betas']
131 |                 if amsgrad:
132 |                         max_exp_avg_sq = state['max_exp_avg_sq']  
133 |                                 
134 | 
135 |                 #compute variance mov avg
136 |                 exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
137 |                 #compute mean moving avg
138 |                 exp_avg.mul_(beta1).add_(1 - beta1, grad)
139 |                 
140 |                
141 |                 
142 |                 
143 |                 ##transformer
144 |                 if grad_transformer == 'square':
145 |                     grad_tmp = grad**2
146 |                 elif grad_transformer == 'abs':
147 |                     grad_tmp = grad.abs()
148 | 
149 | 
150 |                 exp_avg_sq.mul_(beta2).add_((1 - beta2)*grad_tmp)
151 | 
152 | 
153 | 
154 |                 if amsgrad:
155 |                     # Maintains the maximum of all 2nd moment running avg. till now
156 |                     torch.max(max_exp_avg_sq, exp_avg_sq, out=max_exp_avg_sq)
157 |                     # Use the max. for normalizing running avg. of gradient
158 |                     denomc = max_exp_avg_sq.clone()
159 |                 else:
160 |                     denomc = exp_avg_sq.clone()
161 | 
162 |                 if grad_transformer == 'square':
163 |                     #pdb.set_trace()
164 |                     denomc.sqrt_()                 
165 |                 
166 |                 
167 |                 
168 |                                 
169 | 
170 |                 state['step'] += 1
171 |                 
172 |                 
173 | 
174 |                
175 | 
176 |                 if group['weight_decay'] != 0:
177 |                     p_data_fp32.add_(-group['weight_decay'] * group['lr'], p_data_fp32)
178 |                     
179 |                     
180 |                 bias_correction1 = 1 - beta1 ** state['step']
181 |                 bias_correction2 = 1 - beta2 ** state['step']
182 |                 step_size = group['lr'] * math.sqrt(bias_correction2) / bias_correction1                
183 | 
184 |                 
185 |                 # ...let's use calibrated alr 
186 |                 if  group['transformer'] =='softplus':
187 |                     sp = torch.nn.Softplus( smooth)
188 |                     denomf = sp( denomc)
189 |                     p_data_fp32.addcdiv_(-step_size, exp_avg, denomf )  
190 |                     
191 |                 else:
192 |                     
193 |                     denom = exp_avg_sq.sqrt().add_(group['eps'])
194 |                     p_data_fp32.addcdiv_(-step_size * group['lr'], exp_avg, denom)
195 |                 
196 | 
197 |                 p.data.copy_(p_data_fp32)
198 | 
199 |                 #integrated look ahead...
200 |                 #we do it at the param level instead of group level
201 |                 if state['step'] % group['k'] == 0:
202 |                     slow_p = state['slow_buffer'] #get access to slow param tensor
203 |                     slow_p.add_(self.alpha, p.data - slow_p)  #(fast weights - slow weights) * alpha
204 |                     p.data.copy_(slow_p)  #copy interpolated weights to RAdam param tensor
205 | 
206 |         return loss
207 | 


--------------------------------------------------------------------------------
/ranger/ranger2020.py:
--------------------------------------------------------------------------------
  1 | # Ranger deep learning optimizer - RAdam + Lookahead + Gradient Centralization, combined into one optimizer.
  2 | 
  3 | # https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer
  4 | # and/or
  5 | # https://github.com/lessw2020/Best-Deep-Learning-Optimizers
  6 | 
  7 | # Ranger has been used to capture 12 records on the FastAI leaderboard.
  8 | 
  9 | # This version = 2020.9.4
 10 | 
 11 | 
 12 | # Credits:
 13 | # Gradient Centralization --> https://arxiv.org/abs/2004.01461v2 (a new optimization technique for DNNs), github:  https://github.com/Yonghongwei/Gradient-Centralization
 14 | # RAdam -->  https://github.com/LiyuanLucasLiu/RAdam
 15 | # Lookahead --> rewritten by lessw2020, but big thanks to Github @LonePatient and @RWightman for ideas from their code.
 16 | # Lookahead paper --> MZhang,G Hinton  https://arxiv.org/abs/1907.08610
 17 | 
 18 | # summary of changes:
 19 | # 9/4/20 - updated addcmul_ signature to avoid warning.  Integrates latest changes from GC developer (he did the work for this), and verified on performance on private dataset.
 20 | # 4/11/20 - add gradient centralization option.  Set new testing benchmark for accuracy with it, toggle with use_gc flag at init.
 21 | # full code integration with all updates at param level instead of group, moves slow weights into state dict (from generic weights),
 22 | # supports group learning rates (thanks @SHolderbach), fixes sporadic load from saved model issues.
 23 | # changes 8/31/19 - fix references to *self*.N_sma_threshold;
 24 | # changed eps to 1e-5 as better default than 1e-8.
 25 | 
 26 | import math
 27 | import torch
 28 | from torch.optim.optimizer import Optimizer, required
 29 | 
 30 | 
 31 | def centralized_gradient(x, use_gc=True, gc_conv_only=False):
 32 |     '''credit - https://github.com/Yonghongwei/Gradient-Centralization '''
 33 |     if use_gc:
 34 |         if gc_conv_only:
 35 |             if len(list(x.size())) > 3:
 36 |                 x.add_(-x.mean(dim=tuple(range(1, len(list(x.size())))), keepdim=True))
 37 |         else:
 38 |             if len(list(x.size())) > 1:
 39 |                 x.add_(-x.mean(dim=tuple(range(1, len(list(x.size())))), keepdim=True))
 40 |     return x
 41 | 
 42 | 
 43 | class Ranger(Optimizer):
 44 | 
 45 |     def __init__(self, params, lr=1e-3,                       # lr
 46 |                  alpha=0.5, k=6, N_sma_threshhold=5,           # Ranger options
 47 |                  betas=(.95, 0.999), eps=1e-5, weight_decay=0,  # Adam options
 48 |                  # Gradient centralization on or off, applied to conv layers only or conv + fc layers
 49 |                  use_gc=True, gc_conv_only=False, gc_loc=True
 50 |                  ):
 51 | 
 52 |         # parameter checks
 53 |         if not 0.0 <= alpha <= 1.0:
 54 |             raise ValueError(f'Invalid slow update rate: {alpha}')
 55 |         if not 1 <= k:
 56 |             raise ValueError(f'Invalid lookahead steps: {k}')
 57 |         if not lr > 0:
 58 |             raise ValueError(f'Invalid Learning Rate: {lr}')
 59 |         if not eps > 0:
 60 |             raise ValueError(f'Invalid eps: {eps}')
 61 | 
 62 |         # parameter comments:
 63 |         # beta1 (momentum) of .95 seems to work better than .90...
 64 |         # N_sma_threshold of 5 seems better in testing than 4.
 65 |         # In both cases, worth testing on your dataset (.90 vs .95, 4 vs 5) to make sure which works best for you.
 66 | 
 67 |         # prep defaults and init torch.optim base
 68 |         defaults = dict(lr=lr, alpha=alpha, k=k, step_counter=0, betas=betas,
 69 |                         N_sma_threshhold=N_sma_threshhold, eps=eps, weight_decay=weight_decay)
 70 |         super().__init__(params, defaults)
 71 | 
 72 |         # adjustable threshold
 73 |         self.N_sma_threshhold = N_sma_threshhold
 74 | 
 75 |         # look ahead params
 76 | 
 77 |         self.alpha = alpha
 78 |         self.k = k
 79 | 
 80 |         # radam buffer for state
 81 |         self.radam_buffer = [[None, None, None] for ind in range(10)]
 82 | 
 83 |         # gc on or off
 84 |         self.gc_loc = gc_loc
 85 |         self.use_gc = use_gc
 86 |         self.gc_conv_only = gc_conv_only
 87 |         # level of gradient centralization
 88 |         #self.gc_gradient_threshold = 3 if gc_conv_only else 1
 89 | 
 90 |         print(
 91 |             f"Ranger optimizer loaded. \nGradient Centralization usage = {self.use_gc}")
 92 |         if (self.use_gc and self.gc_conv_only == False):
 93 |             print(f"GC applied to both conv and fc layers")
 94 |         elif (self.use_gc and self.gc_conv_only == True):
 95 |             print(f"GC applied to conv layers only")
 96 | 
 97 |     def __setstate__(self, state):
 98 |         print("set state called")
 99 |         super(Ranger, self).__setstate__(state)
100 | 
101 |     def step(self, closure=None):
102 |         loss = None
103 |         # note - below is commented out b/c I have other work that passes back the loss as a float, and thus not a callable closure.
104 |         # Uncomment if you need to use the actual closure...
105 | 
106 |         # if closure is not None:
107 |         #loss = closure()
108 | 
109 |         # Evaluate averages and grad, update param tensors
110 |         for group in self.param_groups:
111 | 
112 |             for p in group['params']:
113 |                 if p.grad is None:
114 |                     continue
115 |                 grad = p.grad.data.float()
116 | 
117 |                 if grad.is_sparse:
118 |                     raise RuntimeError(
119 |                         'Ranger optimizer does not support sparse gradients')
120 | 
121 |                 p_data_fp32 = p.data.float()
122 | 
123 |                 state = self.state[p]  # get state dict for this param
124 | 
125 |                 if len(state) == 0:  # if first time to run...init dictionary with our desired entries
126 |                     # if self.first_run_check==0:
127 |                     # self.first_run_check=1
128 |                     #print("Initializing slow buffer...should not see this at load from saved model!")
129 |                     state['step'] = 0
130 |                     state['exp_avg'] = torch.zeros_like(p_data_fp32)
131 |                     state['exp_avg_sq'] = torch.zeros_like(p_data_fp32)
132 | 
133 |                     # look ahead weight storage now in state dict
134 |                     state['slow_buffer'] = torch.empty_like(p.data)
135 |                     state['slow_buffer'].copy_(p.data)
136 | 
137 |                 else:
138 |                     state['exp_avg'] = state['exp_avg'].type_as(p_data_fp32)
139 |                     state['exp_avg_sq'] = state['exp_avg_sq'].type_as(
140 |                         p_data_fp32)
141 | 
142 |                 # begin computations
143 |                 exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
144 |                 beta1, beta2 = group['betas']
145 | 
146 |                 # GC operation for Conv layers and FC layers
147 |                 # if grad.dim() > self.gc_gradient_threshold:
148 |                 #    grad.add_(-grad.mean(dim=tuple(range(1, grad.dim())), keepdim=True))
149 |                 if self.gc_loc:
150 |                     grad = centralized_gradient(grad, use_gc=self.use_gc, gc_conv_only=self.gc_conv_only)
151 | 
152 |                 state['step'] += 1
153 | 
154 |                 # compute variance mov avg
155 |                 exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2)
156 | 
157 |                 # compute mean moving avg
158 |                 exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)
159 | 
160 |                 buffered = self.radam_buffer[int(state['step'] % 10)]
161 | 
162 |                 if state['step'] == buffered[0]:
163 |                     N_sma, step_size = buffered[1], buffered[2]
164 |                 else:
165 |                     buffered[0] = state['step']
166 |                     beta2_t = beta2 ** state['step']
167 |                     N_sma_max = 2 / (1 - beta2) - 1
168 |                     N_sma = N_sma_max - 2 * \
169 |                         state['step'] * beta2_t / (1 - beta2_t)
170 |                     buffered[1] = N_sma
171 |                     if N_sma > self.N_sma_threshhold:
172 |                         step_size = math.sqrt((1 - beta2_t) * (N_sma - 4) / (N_sma_max - 4) * (
173 |                             N_sma - 2) / N_sma * N_sma_max / (N_sma_max - 2)) / (1 - beta1 ** state['step'])
174 |                     else:
175 |                         step_size = 1.0 / (1 - beta1 ** state['step'])
176 |                     buffered[2] = step_size
177 | 
178 |                 # if group['weight_decay'] != 0:
179 |                 #    p_data_fp32.add_(-group['weight_decay']
180 |                 #                     * group['lr'], p_data_fp32)
181 | 
182 |                 # apply lr
183 |                 if N_sma > self.N_sma_threshhold:
184 |                     denom = exp_avg_sq.sqrt().add_(group['eps'])
185 |                     G_grad = exp_avg / denom
186 |                 else:
187 |                     G_grad = exp_avg
188 | 
189 |                 if group['weight_decay'] != 0:
190 |                     G_grad.add_(p_data_fp32, alpha=group['weight_decay'])
191 |                 # GC operation
192 |                 if self.gc_loc == False:
193 |                     G_grad = centralized_gradient(G_grad, use_gc=self.use_gc, gc_conv_only=self.gc_conv_only)
194 | 
195 |                 p_data_fp32.add_(G_grad, alpha=-step_size * group['lr'])
196 |                 p.data.copy_(p_data_fp32)
197 | 
198 |                 # integrated look ahead...
199 |                 # we do it at the param level instead of group level
200 |                 if state['step'] % group['k'] == 0:
201 |                     # get access to slow param tensor
202 |                     slow_p = state['slow_buffer']
203 |                     # (fast weights - slow weights) * alpha
204 |                     slow_p.add_(p.data - slow_p, alpha=self.alpha)
205 |                     # copy interpolated weights to RAdam param tensor
206 |                     p.data.copy_(slow_p)
207 | 
208 |         return loss
209 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
  1 |                                  Apache License
  2 |                            Version 2.0, January 2004
  3 |                         http://www.apache.org/licenses/
  4 | 
  5 |    TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
  6 | 
  7 |    1. Definitions.
  8 | 
  9 |       "License" shall mean the terms and conditions for use, reproduction,
 10 |       and distribution as defined by Sections 1 through 9 of this document.
 11 | 
 12 |       "Licensor" shall mean the copyright owner or entity authorized by
 13 |       the copyright owner that is granting the License.
 14 | 
 15 |       "Legal Entity" shall mean the union of the acting entity and all
 16 |       other entities that control, are controlled by, or are under common
 17 |       control with that entity. For the purposes of this definition,
 18 |       "control" means (i) the power, direct or indirect, to cause the
 19 |       direction or management of such entity, whether by contract or
 20 |       otherwise, or (ii) ownership of fifty percent (50%) or more of the
 21 |       outstanding shares, or (iii) beneficial ownership of such entity.
 22 | 
 23 |       "You" (or "Your") shall mean an individual or Legal Entity
 24 |       exercising permissions granted by this License.
 25 | 
 26 |       "Source" form shall mean the preferred form for making modifications,
 27 |       including but not limited to software source code, documentation
 28 |       source, and configuration files.
 29 | 
 30 |       "Object" form shall mean any form resulting from mechanical
 31 |       transformation or translation of a Source form, including but
 32 |       not limited to compiled object code, generated documentation,
 33 |       and conversions to other media types.
 34 | 
 35 |       "Work" shall mean the work of authorship, whether in Source or
 36 |       Object form, made available under the License, as indicated by a
 37 |       copyright notice that is included in or attached to the work
 38 |       (an example is provided in the Appendix below).
 39 | 
 40 |       "Derivative Works" shall mean any work, whether in Source or Object
 41 |       form, that is based on (or derived from) the Work and for which the
 42 |       editorial revisions, annotations, elaborations, or other modifications
 43 |       represent, as a whole, an original work of authorship. For the purposes
 44 |       of this License, Derivative Works shall not include works that remain
 45 |       separable from, or merely link (or bind by name) to the interfaces of,
 46 |       the Work and Derivative Works thereof.
 47 | 
 48 |       "Contribution" shall mean any work of authorship, including
 49 |       the original version of the Work and any modifications or additions
 50 |       to that Work or Derivative Works thereof, that is intentionally
 51 |       submitted to Licensor for inclusion in the Work by the copyright owner
 52 |       or by an individual or Legal Entity authorized to submit on behalf of
 53 |       the copyright owner. For the purposes of this definition, "submitted"
 54 |       means any form of electronic, verbal, or written communication sent
 55 |       to the Licensor or its representatives, including but not limited to
 56 |       communication on electronic mailing lists, source code control systems,
 57 |       and issue tracking systems that are managed by, or on behalf of, the
 58 |       Licensor for the purpose of discussing and improving the Work, but
 59 |       excluding communication that is conspicuously marked or otherwise
 60 |       designated in writing by the copyright owner as "Not a Contribution."
 61 | 
 62 |       "Contributor" shall mean Licensor and any individual or Legal Entity
 63 |       on behalf of whom a Contribution has been received by Licensor and
 64 |       subsequently incorporated within the Work.
 65 | 
 66 |    2. Grant of Copyright License. Subject to the terms and conditions of
 67 |       this License, each Contributor hereby grants to You a perpetual,
 68 |       worldwide, non-exclusive, no-charge, royalty-free, irrevocable
 69 |       copyright license to reproduce, prepare Derivative Works of,
 70 |       publicly display, publicly perform, sublicense, and distribute the
 71 |       Work and such Derivative Works in Source or Object form.
 72 | 
 73 |    3. Grant of Patent License. Subject to the terms and conditions of
 74 |       this License, each Contributor hereby grants to You a perpetual,
 75 |       worldwide, non-exclusive, no-charge, royalty-free, irrevocable
 76 |       (except as stated in this section) patent license to make, have made,
 77 |       use, offer to sell, sell, import, and otherwise transfer the Work,
 78 |       where such license applies only to those patent claims licensable
 79 |       by such Contributor that are necessarily infringed by their
 80 |       Contribution(s) alone or by combination of their Contribution(s)
 81 |       with the Work to which such Contribution(s) was submitted. If You
 82 |       institute patent litigation against any entity (including a
 83 |       cross-claim or counterclaim in a lawsuit) alleging that the Work
 84 |       or a Contribution incorporated within the Work constitutes direct
 85 |       or contributory patent infringement, then any patent licenses
 86 |       granted to You under this License for that Work shall terminate
 87 |       as of the date such litigation is filed.
 88 | 
 89 |    4. Redistribution. You may reproduce and distribute copies of the
 90 |       Work or Derivative Works thereof in any medium, with or without
 91 |       modifications, and in Source or Object form, provided that You
 92 |       meet the following conditions:
 93 | 
 94 |       (a) You must give any other recipients of the Work or
 95 |           Derivative Works a copy of this License; and
 96 | 
 97 |       (b) You must cause any modified files to carry prominent notices
 98 |           stating that You changed the files; and
 99 | 
100 |       (c) You must retain, in the Source form of any Derivative Works
101 |           that You distribute, all copyright, patent, trademark, and
102 |           attribution notices from the Source form of the Work,
103 |           excluding those notices that do not pertain to any part of
104 |           the Derivative Works; and
105 | 
106 |       (d) If the Work includes a "NOTICE" text file as part of its
107 |           distribution, then any Derivative Works that You distribute must
108 |           include a readable copy of the attribution notices contained
109 |           within such NOTICE file, excluding those notices that do not
110 |           pertain to any part of the Derivative Works, in at least one
111 |           of the following places: within a NOTICE text file distributed
112 |           as part of the Derivative Works; within the Source form or
113 |           documentation, if provided along with the Derivative Works; or,
114 |           within a display generated by the Derivative Works, if and
115 |           wherever such third-party notices normally appear. The contents
116 |           of the NOTICE file are for informational purposes only and
117 |           do not modify the License. You may add Your own attribution
118 |           notices within Derivative Works that You distribute, alongside
119 |           or as an addendum to the NOTICE text from the Work, provided
120 |           that such additional attribution notices cannot be construed
121 |           as modifying the License.
122 | 
123 |       You may add Your own copyright statement to Your modifications and
124 |       may provide additional or different license terms and conditions
125 |       for use, reproduction, or distribution of Your modifications, or
126 |       for any such Derivative Works as a whole, provided Your use,
127 |       reproduction, and distribution of the Work otherwise complies with
128 |       the conditions stated in this License.
129 | 
130 |    5. Submission of Contributions. Unless You explicitly state otherwise,
131 |       any Contribution intentionally submitted for inclusion in the Work
132 |       by You to the Licensor shall be under the terms and conditions of
133 |       this License, without any additional terms or conditions.
134 |       Notwithstanding the above, nothing herein shall supersede or modify
135 |       the terms of any separate license agreement you may have executed
136 |       with Licensor regarding such Contributions.
137 | 
138 |    6. Trademarks. This License does not grant permission to use the trade
139 |       names, trademarks, service marks, or product names of the Licensor,
140 |       except as required for reasonable and customary use in describing the
141 |       origin of the Work and reproducing the content of the NOTICE file.
142 | 
143 |    7. Disclaimer of Warranty. Unless required by applicable law or
144 |       agreed to in writing, Licensor provides the Work (and each
145 |       Contributor provides its Contributions) on an "AS IS" BASIS,
146 |       WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147 |       implied, including, without limitation, any warranties or conditions
148 |       of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149 |       PARTICULAR PURPOSE. You are solely responsible for determining the
150 |       appropriateness of using or redistributing the Work and assume any
151 |       risks associated with Your exercise of permissions under this License.
152 | 
153 |    8. Limitation of Liability. In no event and under no legal theory,
154 |       whether in tort (including negligence), contract, or otherwise,
155 |       unless required by applicable law (such as deliberate and grossly
156 |       negligent acts) or agreed to in writing, shall any Contributor be
157 |       liable to You for damages, including any direct, indirect, special,
158 |       incidental, or consequential damages of any character arising as a
159 |       result of this License or out of the use or inability to use the
160 |       Work (including but not limited to damages for loss of goodwill,
161 |       work stoppage, computer failure or malfunction, or any and all
162 |       other commercial damages or losses), even if such Contributor
163 |       has been advised of the possibility of such damages.
164 | 
165 |    9. Accepting Warranty or Additional Liability. While redistributing
166 |       the Work or Derivative Works thereof, You may choose to offer,
167 |       and charge a fee for, acceptance of support, warranty, indemnity,
168 |       or other liability obligations and/or rights consistent with this
169 |       License. However, in accepting such obligations, You may act only
170 |       on Your own behalf and on Your sole responsibility, not on behalf
171 |       of any other Contributor, and only if You agree to indemnify,
172 |       defend, and hold each Contributor harmless for any liability
173 |       incurred by, or claims asserted against, such Contributor by reason
174 |       of your accepting any such warranty or additional liability.
175 | 
176 |    END OF TERMS AND CONDITIONS
177 | 
178 |    APPENDIX: How to apply the Apache License to your work.
179 | 
180 |       To apply the Apache License to your work, attach the following
181 |       boilerplate notice, with the fields enclosed by brackets "[]"
182 |       replaced with your own identifying information. (Don't include
183 |       the brackets!)  The text should be enclosed in the appropriate
184 |       comment syntax for the file format. We also recommend that a
185 |       file or class name and description of purpose be included on the
186 |       same "printed page" as the copyright notice for easier
187 |       identification within third-party archives.
188 | 
189 |    Copyright [yyyy] [name of copyright owner]
190 | 
191 |    Licensed under the Apache License, Version 2.0 (the "License");
192 |    you may not use this file except in compliance with the License.
193 |    You may obtain a copy of the License at
194 | 
195 |        http://www.apache.org/licenses/LICENSE-2.0
196 | 
197 |    Unless required by applicable law or agreed to in writing, software
198 |    distributed under the License is distributed on an "AS IS" BASIS,
199 |    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200 |    See the License for the specific language governing permissions and
201 |    limitations under the License.
202 | 


--------------------------------------------------------------------------------