0.75 and on_boundary:
79 | self.radius *= 2.
80 | # self.delta *= max(2,self.delta_hat)
81 | if rho > self.eta:
82 | accept_step = True
83 | else:
84 | accept_step = False
85 |
86 | return accept_step
87 |
88 |
89 |
90 |
--------------------------------------------------------------------------------
/applications/README.md:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 | ___ ___ ___ ___ ___ ___
6 | /__/\ / /\ / /\ / /\ ___ / /\ /__/\
7 | \ \:\ / /:/_ / /:/_ / /:/_ / /\ / /::\ \ \:\
8 | \__\:\ / /:/ /\ / /:/ /\ / /:/ /\ / /:/ / /:/\:\ \ \:\
9 | ___ / /::\ / /:/ /:/_ / /:/ /::\ / /:/ /::\ /__/::\ / /:/~/::\ _____\__\:\
10 | /__/\ /:/\:\/__/:/ /:/ /\/__/:/ /:/\:\/__/:/ /:/\:\\__\/\:\__ /__/:/ /:/\:\/__/::::::::\
11 | \ \:\/:/__\/\ \:\/:/ /:/\ \:\/:/~/:/\ \:\/:/~/:/ \ \:\/\\ \:\/:/__\/\ \:\~~\~~\/
12 | \ \::/ \ \::/ /:/ \ \::/ /:/ \ \::/ /:/ \__\::/ \ \::/ \ \:\ ~~~
13 | \ \:\ \ \:\/:/ \__\/ /:/ \__\/ /:/ /__/:/ \ \:\ \ \:\
14 | \ \:\ \ \::/ /__/:/ /__/:/ \__\/ \ \:\ \ \:\
15 | \__\/ \__\/ \__\/ \__\/ \__\/ \__\/
16 |
17 |
18 | ___ ___ ___ ___
19 | / /\ / /\ / /\ /__/\
20 | / /:/_ / /::\ / /::\ \ \:\
21 | ___ ___ / /:/ /\ / /:/\:\ / /:/\:\ \ \:\
22 | /__/\ / /\ / /:/ /:/_ / /:/~/::\ / /:/~/:/ _____\__\:\
23 | \ \:\ / /://__/:/ /:/ /\/__/:/ /:/\:\/__/:/ /:/___/__/::::::::\
24 | \ \:\ /:/ \ \:\/:/ /:/\ \:\/:/__\/\ \:\/:::::/\ \:\~~\~~\/
25 | \ \:\/:/ \ \::/ /:/ \ \::/ \ \::/~~~~ \ \:\ ~~~
26 | \ \::/ \ \:\/:/ \ \:\ \ \:\ \ \:\
27 | \__\/ \ \::/ \ \:\ \ \:\ \ \:\
28 | \__\/ \__\/ \__\/ \__\/
29 |
30 |
31 |
32 | # Transfer Learning
33 |
34 | * Examples of CIFAR10, CIFAR100 classification from pre-trained Imagenet ResNet50 model in `transfer_learning/`
35 |
36 | * Pre-trained model serves as well conditioned initial guess for transfer learning. In this setting Newton methods perform well due to their excellent properties in local convergence. Low Rank Saddle Free Newton is able to zero in on highly generalizable local minimizers bypassing indefinite regions. Below are validation accuracies of best choices of fixed step-length for Adam, SGD and LRSFN with fixed rank of 40.
37 |
38 |
39 | 
40 |
41 |
42 | * For more information see the following manuscript
43 |
44 | - \[2\] O'Leary-Roseberry, T., Alger, N., Ghattas O.,
45 | [**Low Rank Saddle Free Newton: A Scalable Method for Stochastic Nonconvex Optimization**](https://arxiv.org/abs/2002.02881).
46 | arXiv:2002.02881.
47 | ([Download](https://arxiv.org/pdf/2002.02881.pdf))BibTeX
48 | @article{OLearyRoseberryAlgerGhattas2020,
49 | title={Low Rank Saddle Free Newton: Algorithm and Analysis},
50 | author={O'Leary-Roseberry, Thomas and Alger, Nick and Ghattas, Omar},
51 | journal={arXiv preprint arXiv:2002.02881},
52 | year={2020}
53 | }
54 | }
48 | )
49 |
50 |
51 | and matrix-matrix products:
52 |
53 | )
54 |
55 |
56 | ## Compatibility
57 |
58 | The code is compatible with Tensorflow v1 and v2, but certain features of v2 are disabled (like eager execution). This is because the Hessian matrix products in hessianlearn are implemented using `placeholders` which have been deprecated in v2. For this reason hessianlearn cannot work with data generators and things like this that require eager execution. If any compatibility issues are found, please open an [issue](https://github.com/tomoleary/hessianlearn/issues).
59 |
60 | ## Usage
61 | Set `HESSIANLEARN_PATH` environmental variable
62 |
63 | Train a keras model
64 |
65 | ```python
66 | import os,sys
67 | import tensorflow as tf
68 | sys.path.append( os.environ.get('HESSIANLEARN_PATH'))
69 | from hessianlearn import *
70 |
71 | # Define keras neural network model
72 | neural_network = tf.keras.models.Model(...)
73 | # Define loss function and compile model
74 | neural_network.compile(loss = ...)
75 |
76 | ```
77 |
78 | hessianlearn implements various training [`problem`](https://github.com/tomoleary/hessianlearn/blob/master/hessianlearn/problem/problem.py) constructs (regression, classification, autoencoders, variational autoencoders, generative adversarial networks). Instantiate a `problem`, a `data` object (which takes a dictionary with keys that correspond to the corresponding `placeholders` in `problem`) and `regularization`
79 |
80 | ```python
81 | # Instantiate the problem (this handles the loss function,
82 | # construction of hessian and gradient etc.)
83 | # KerasModelProblem extracts loss function and metrics from
84 | # a compiled keras model
85 | problem = KerasModelProblem(neural_network)
86 | # Instantiate the data object, this handles the train / validation split
87 | # as well as iterating during training
88 | data = Data({problem.x:x_data,problem.y_true:y_data},train_batch_size,\
89 | validation_data_size = validation_data_size)
90 | # Instantiate the regularization: L2Regularization is Tikhonov,
91 | # gamma = 0 is no regularization
92 | regularization = L2Regularization(problem,gamma = 0)
93 | ```
94 |
95 | Pass these objects into the `HessianlearnModel` which handles the training
96 |
97 | ```python
98 | HLModel = HessianlearnModel(problem,regularization,data)
99 | HLModel.fit()
100 | ```
101 |
102 | ### Alternative Usage (More like Keras Interface)
103 | The example above was the original way the optimizer interface was implemented in hessianlearn, however to better mimic the keras interface and allow for more end-user rapid prototyping of the optimizer that is used to fit data, as of December 2021, the following way has been created
104 |
105 | ```python
106 | import os,sys
107 | import tensorflow as tf
108 | sys.path.append( os.environ.get('HESSIANLEARN_PATH'))
109 | from hessianlearn import *
110 |
111 | # Define keras neural network model
112 | neural_network = tf.keras.models.Model(...)
113 | # Define loss function and compile model
114 | neural_network.compile(loss = ...)
115 | # Instance keras model wrapper which deals with the
116 | # construction of the `problem` which handles the construction
117 | # of Hessian computational graph and variables
118 | HLModel = KerasModelWrapper(neural_network)
119 | # Then the end user can pass in an optimizer
120 | # (e.g. custom end-user optimizer)
121 | optimizer = LowRankSaddleFreeNewton # The class constructor, not an instance
122 | opt_parameters = LowRankSaddleFreeNewtonParameters()
123 | opt_parameters['hessian_low_rank'] = 40
124 | HLModel.set_optimizer(optimizer,optimizer_parameters = opt_parameters)
125 | # The data object still needs to key on to the specific computational
126 | # graph variables that data will be passed in for.
127 | # Note that data can naturally handle multiple input and output data,
128 | # in which case problem.x, problem.y_true are lists corresponding to
129 | # neural_network.inputs, neural_network.outputs
130 | problem = HLModel.problem
131 | data = Data({problem.x:x_data,problem.y_true:y_data},train_batch_size,\
132 | validation_data_size = validation_data_size)
133 | # And finally one can call fit!
134 | HLModel.fit(data)
135 | ```
136 |
137 | ## Examples
138 |
139 | [Tutorial 0: MNIST Autoencoder](https://github.com/tomoleary/hessianlearn/blob/master/tutorial/Tutorial%200%20MNIST%20Autoencoder.ipynb)
140 |
141 |
142 | ## Applications
143 |
144 | ### Transfer Learning
145 |
146 | * Examples of CIFAR10, CIFAR100 classification from pre-trained Imagenet ResNet50 model in `applications/transfer_learning/`
147 |
148 | * Pre-trained model serves as well conditioned initial guess for transfer learning. In this setting Newton methods perform well due to their excellent properties in local convergence. Low Rank Saddle Free Newton is able to zero in on highly generalizable local minimizers bypassing indefinite regions. Below are validation accuracies of best choices of fixed step-length for Adam, SGD and LRSFN with fixed rank of 40.
149 |
150 |
151 |
152 |
153 |
154 | # References
155 |
156 | These manuscripts motivate and use the hessianlearn library for stochastic nonconvex optimization
157 |
158 | - \[1\] O'Leary-Roseberry, T., Alger, N., Ghattas O.,
159 | [**Inexact Newton Methods for Stochastic Nonconvex Optimization with Applications to Neural Network Training**](https://arxiv.org/abs/1905.06738).
160 | arXiv:1905.06738.
161 | ([Download](https://arxiv.org/pdf/1905.06738.pdf))BibTeX
162 | @article{OLearyRoseberryAlgerGhattas2019,
163 | title={Inexact Newton methods for stochastic nonconvex optimization with applications to neural network training},
164 | author={O'Leary-Roseberry, Thomas and Alger, Nick and Ghattas, Omar},
165 | journal={arXiv preprint arXiv:1905.06738},
166 | year={2019}
167 | }
168 | }BibTeX
174 | @article{OLearyRoseberryAlgerGhattas2020,
175 | title={Low Rank Saddle Free Newton: Algorithm and Analysis},
176 | author={O'Leary-Roseberry, Thomas and Alger, Nick and Ghattas, Omar},
177 | journal={arXiv preprint arXiv:2002.02881},
178 | year={2020}
179 | }
180 | }BibTeX
187 | @article{OLearyRoseberryVillaChenEtAl2022,
188 | title={Derivative-informed projected neural networks for high-dimensional parametric maps governed by {PDE}s},
189 | author={O’Leary-Roseberry, Thomas and Villa, Umberto and Chen, Peng and Ghattas, Omar},
190 | journal={Computer Methods in Applied Mechanics and Engineering},
191 | volume={388},
192 | pages={114199},
193 | year={2022},
194 | publisher={Elsevier}
195 | }
196 | }BibTeX
203 | @article{OLearyRoseberryDuChaudhuriEtAl2021,
204 | title={Adaptive Projected Residual Networks for Learning Parametric Maps from Sparse Data},
205 | author={O'Leary-Roseberry, Thomas and Du, Xiaosong, and Chaudhuri, Anirban, and Martins Joaqium R. R. A., and Willcox, Karen, and Ghattas, Omar},
206 | journal={arXiv preprint arXiv:2112.07096},
207 | year={2021}
208 | }
209 | }