├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── main.py └── plot.py /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | Facebook has adopted a Code of Conduct that we expect project participants to adhere to. Please read the [full text](https://code.facebook.com/pages/876921332402685/open-source-code-of-conduct) so that you can understand what actions will and will not be tolerated. 4 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to DeepConvexity 2 | We want to make contributing to this project as easy and transparent as 3 | possible. 4 | 5 | ## Pull Requests 6 | We actively welcome your pull requests. 7 | 8 | 1. Fork the repo and create your branch from `master`. 9 | 2. If you've added code that should be tested, add tests. 10 | 3. If you've changed APIs, update the documentation. 11 | 4. Ensure the test suite passes. 12 | 5. Make sure your code lints. 13 | 6. If you haven't already, complete the Contributor License Agreement ("CLA"). 14 | 15 | ## Contributor License Agreement ("CLA") 16 | In order to accept your pull request, we need you to submit a CLA. You only need 17 | to do this once to work on any of Facebook's open source projects. 18 | 19 | Complete your CLA here: 20 | 21 | ## Issues 22 | We use GitHub issues to track public bugs. Please ensure your description is 23 | clear and has sufficient instructions to be able to reproduce the issue. 24 | 25 | Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe 26 | disclosure of security bugs. In those cases, please go through the process 27 | outlined on that page and do not file a public issue. 28 | 29 | ## Coding Style 30 | * 2 spaces for indentation rather than tabs 31 | * 80 character line length 32 | * ... 33 | 34 | ## License 35 | By contributing to DeepConvexity, you agree that your contributions will be licensed 36 | under the LICENSE file in the root directory of this source tree. 37 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Attribution-NonCommercial 4.0 International 2 | 3 | ======================================================================= 4 | 5 | Creative Commons Corporation ("Creative Commons") is not a law firm and 6 | does not provide legal services or legal advice. Distribution of 7 | Creative Commons public licenses does not create a lawyer-client or 8 | other relationship. Creative Commons makes its licenses and related 9 | information available on an "as-is" basis. Creative Commons gives no 10 | warranties regarding its licenses, any material licensed under their 11 | terms and conditions, or any related information. Creative Commons 12 | disclaims all liability for damages resulting from their use to the 13 | fullest extent possible. 14 | 15 | Using Creative Commons Public Licenses 16 | 17 | Creative Commons public licenses provide a standard set of terms and 18 | conditions that creators and other rights holders may use to share 19 | original works of authorship and other material subject to copyright 20 | and certain other rights specified in the public license below. The 21 | following considerations are for informational purposes only, are not 22 | exhaustive, and do not form part of our licenses. 23 | 24 | Considerations for licensors: Our public licenses are 25 | intended for use by those authorized to give the public 26 | permission to use material in ways otherwise restricted by 27 | copyright and certain other rights. Our licenses are 28 | irrevocable. Licensors should read and understand the terms 29 | and conditions of the license they choose before applying it. 30 | Licensors should also secure all rights necessary before 31 | applying our licenses so that the public can reuse the 32 | material as expected. Licensors should clearly mark any 33 | material not subject to the license. This includes other CC- 34 | licensed material, or material used under an exception or 35 | limitation to copyright. More considerations for licensors: 36 | wiki.creativecommons.org/Considerations_for_licensors 37 | 38 | Considerations for the public: By using one of our public 39 | licenses, a licensor grants the public permission to use the 40 | licensed material under specified terms and conditions. If 41 | the licensor's permission is not necessary for any reason--for 42 | example, because of any applicable exception or limitation to 43 | copyright--then that use is not regulated by the license. Our 44 | licenses grant only permissions under copyright and certain 45 | other rights that a licensor has authority to grant. Use of 46 | the licensed material may still be restricted for other 47 | reasons, including because others have copyright or other 48 | rights in the material. A licensor may make special requests, 49 | such as asking that all changes be marked or described. 50 | Although not required by our licenses, you are encouraged to 51 | respect those requests where reasonable. More_considerations 52 | for the public: 53 | wiki.creativecommons.org/Considerations_for_licensees 54 | 55 | ======================================================================= 56 | 57 | Creative Commons Attribution-NonCommercial 4.0 International Public 58 | License 59 | 60 | By exercising the Licensed Rights (defined below), You accept and agree 61 | to be bound by the terms and conditions of this Creative Commons 62 | Attribution-NonCommercial 4.0 International Public License ("Public 63 | License"). To the extent this Public License may be interpreted as a 64 | contract, You are granted the Licensed Rights in consideration of Your 65 | acceptance of these terms and conditions, and the Licensor grants You 66 | such rights in consideration of benefits the Licensor receives from 67 | making the Licensed Material available under these terms and 68 | conditions. 69 | 70 | Section 1 -- Definitions. 71 | 72 | a. Adapted Material means material subject to Copyright and Similar 73 | Rights that is derived from or based upon the Licensed Material 74 | and in which the Licensed Material is translated, altered, 75 | arranged, transformed, or otherwise modified in a manner requiring 76 | permission under the Copyright and Similar Rights held by the 77 | Licensor. For purposes of this Public License, where the Licensed 78 | Material is a musical work, performance, or sound recording, 79 | Adapted Material is always produced where the Licensed Material is 80 | synched in timed relation with a moving image. 81 | 82 | b. Adapter's License means the license You apply to Your Copyright 83 | and Similar Rights in Your contributions to Adapted Material in 84 | accordance with the terms and conditions of this Public License. 85 | 86 | c. Copyright and Similar Rights means copyright and/or similar rights 87 | closely related to copyright including, without limitation, 88 | performance, broadcast, sound recording, and Sui Generis Database 89 | Rights, without regard to how the rights are labeled or 90 | categorized. For purposes of this Public License, the rights 91 | specified in Section 2(b)(1)-(2) are not Copyright and Similar 92 | Rights. 93 | d. Effective Technological Measures means those measures that, in the 94 | absence of proper authority, may not be circumvented under laws 95 | fulfilling obligations under Article 11 of the WIPO Copyright 96 | Treaty adopted on December 20, 1996, and/or similar international 97 | agreements. 98 | 99 | e. Exceptions and Limitations means fair use, fair dealing, and/or 100 | any other exception or limitation to Copyright and Similar Rights 101 | that applies to Your use of the Licensed Material. 102 | 103 | f. Licensed Material means the artistic or literary work, database, 104 | or other material to which the Licensor applied this Public 105 | License. 106 | 107 | g. Licensed Rights means the rights granted to You subject to the 108 | terms and conditions of this Public License, which are limited to 109 | all Copyright and Similar Rights that apply to Your use of the 110 | Licensed Material and that the Licensor has authority to license. 111 | 112 | h. Licensor means the individual(s) or entity(ies) granting rights 113 | under this Public License. 114 | 115 | i. NonCommercial means not primarily intended for or directed towards 116 | commercial advantage or monetary compensation. For purposes of 117 | this Public License, the exchange of the Licensed Material for 118 | other material subject to Copyright and Similar Rights by digital 119 | file-sharing or similar means is NonCommercial provided there is 120 | no payment of monetary compensation in connection with the 121 | exchange. 122 | 123 | j. Share means to provide material to the public by any means or 124 | process that requires permission under the Licensed Rights, such 125 | as reproduction, public display, public performance, distribution, 126 | dissemination, communication, or importation, and to make material 127 | available to the public including in ways that members of the 128 | public may access the material from a place and at a time 129 | individually chosen by them. 130 | 131 | k. Sui Generis Database Rights means rights other than copyright 132 | resulting from Directive 96/9/EC of the European Parliament and of 133 | the Council of 11 March 1996 on the legal protection of databases, 134 | as amended and/or succeeded, as well as other essentially 135 | equivalent rights anywhere in the world. 136 | 137 | l. You means the individual or entity exercising the Licensed Rights 138 | under this Public License. Your has a corresponding meaning. 139 | 140 | Section 2 -- Scope. 141 | 142 | a. License grant. 143 | 144 | 1. Subject to the terms and conditions of this Public License, 145 | the Licensor hereby grants You a worldwide, royalty-free, 146 | non-sublicensable, non-exclusive, irrevocable license to 147 | exercise the Licensed Rights in the Licensed Material to: 148 | 149 | a. reproduce and Share the Licensed Material, in whole or 150 | in part, for NonCommercial purposes only; and 151 | 152 | b. produce, reproduce, and Share Adapted Material for 153 | NonCommercial purposes only. 154 | 155 | 2. Exceptions and Limitations. For the avoidance of doubt, where 156 | Exceptions and Limitations apply to Your use, this Public 157 | License does not apply, and You do not need to comply with 158 | its terms and conditions. 159 | 160 | 3. Term. The term of this Public License is specified in Section 161 | 6(a). 162 | 163 | 4. Media and formats; technical modifications allowed. The 164 | Licensor authorizes You to exercise the Licensed Rights in 165 | all media and formats whether now known or hereafter created, 166 | and to make technical modifications necessary to do so. The 167 | Licensor waives and/or agrees not to assert any right or 168 | authority to forbid You from making technical modifications 169 | necessary to exercise the Licensed Rights, including 170 | technical modifications necessary to circumvent Effective 171 | Technological Measures. For purposes of this Public License, 172 | simply making modifications authorized by this Section 2(a) 173 | (4) never produces Adapted Material. 174 | 175 | 5. Downstream recipients. 176 | 177 | a. Offer from the Licensor -- Licensed Material. Every 178 | recipient of the Licensed Material automatically 179 | receives an offer from the Licensor to exercise the 180 | Licensed Rights under the terms and conditions of this 181 | Public License. 182 | 183 | b. No downstream restrictions. You may not offer or impose 184 | any additional or different terms or conditions on, or 185 | apply any Effective Technological Measures to, the 186 | Licensed Material if doing so restricts exercise of the 187 | Licensed Rights by any recipient of the Licensed 188 | Material. 189 | 190 | 6. No endorsement. Nothing in this Public License constitutes or 191 | may be construed as permission to assert or imply that You 192 | are, or that Your use of the Licensed Material is, connected 193 | with, or sponsored, endorsed, or granted official status by, 194 | the Licensor or others designated to receive attribution as 195 | provided in Section 3(a)(1)(A)(i). 196 | 197 | b. Other rights. 198 | 199 | 1. Moral rights, such as the right of integrity, are not 200 | licensed under this Public License, nor are publicity, 201 | privacy, and/or other similar personality rights; however, to 202 | the extent possible, the Licensor waives and/or agrees not to 203 | assert any such rights held by the Licensor to the limited 204 | extent necessary to allow You to exercise the Licensed 205 | Rights, but not otherwise. 206 | 207 | 2. Patent and trademark rights are not licensed under this 208 | Public License. 209 | 210 | 3. To the extent possible, the Licensor waives any right to 211 | collect royalties from You for the exercise of the Licensed 212 | Rights, whether directly or through a collecting society 213 | under any voluntary or waivable statutory or compulsory 214 | licensing scheme. In all other cases the Licensor expressly 215 | reserves any right to collect such royalties, including when 216 | the Licensed Material is used other than for NonCommercial 217 | purposes. 218 | 219 | Section 3 -- License Conditions. 220 | 221 | Your exercise of the Licensed Rights is expressly made subject to the 222 | following conditions. 223 | 224 | a. Attribution. 225 | 226 | 1. If You Share the Licensed Material (including in modified 227 | form), You must: 228 | 229 | a. retain the following if it is supplied by the Licensor 230 | with the Licensed Material: 231 | 232 | i. identification of the creator(s) of the Licensed 233 | Material and any others designated to receive 234 | attribution, in any reasonable manner requested by 235 | the Licensor (including by pseudonym if 236 | designated); 237 | 238 | ii. a copyright notice; 239 | 240 | iii. a notice that refers to this Public License; 241 | 242 | iv. a notice that refers to the disclaimer of 243 | warranties; 244 | 245 | v. a URI or hyperlink to the Licensed Material to the 246 | extent reasonably practicable; 247 | 248 | b. indicate if You modified the Licensed Material and 249 | retain an indication of any previous modifications; and 250 | 251 | c. indicate the Licensed Material is licensed under this 252 | Public License, and include the text of, or the URI or 253 | hyperlink to, this Public License. 254 | 255 | 2. You may satisfy the conditions in Section 3(a)(1) in any 256 | reasonable manner based on the medium, means, and context in 257 | which You Share the Licensed Material. For example, it may be 258 | reasonable to satisfy the conditions by providing a URI or 259 | hyperlink to a resource that includes the required 260 | information. 261 | 262 | 3. If requested by the Licensor, You must remove any of the 263 | information required by Section 3(a)(1)(A) to the extent 264 | reasonably practicable. 265 | 266 | 4. If You Share Adapted Material You produce, the Adapter's 267 | License You apply must not prevent recipients of the Adapted 268 | Material from complying with this Public License. 269 | 270 | Section 4 -- Sui Generis Database Rights. 271 | 272 | Where the Licensed Rights include Sui Generis Database Rights that 273 | apply to Your use of the Licensed Material: 274 | 275 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right 276 | to extract, reuse, reproduce, and Share all or a substantial 277 | portion of the contents of the database for NonCommercial purposes 278 | only; 279 | 280 | b. if You include all or a substantial portion of the database 281 | contents in a database in which You have Sui Generis Database 282 | Rights, then the database in which You have Sui Generis Database 283 | Rights (but not its individual contents) is Adapted Material; and 284 | 285 | c. You must comply with the conditions in Section 3(a) if You Share 286 | all or a substantial portion of the contents of the database. 287 | 288 | For the avoidance of doubt, this Section 4 supplements and does not 289 | replace Your obligations under this Public License where the Licensed 290 | Rights include other Copyright and Similar Rights. 291 | 292 | Section 5 -- Disclaimer of Warranties and Limitation of Liability. 293 | 294 | a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE 295 | EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS 296 | AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF 297 | ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, 298 | IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, 299 | WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR 300 | PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, 301 | ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT 302 | KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT 303 | ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. 304 | 305 | b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE 306 | TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, 307 | NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, 308 | INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, 309 | COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR 310 | USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN 311 | ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR 312 | DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR 313 | IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. 314 | 315 | c. The disclaimer of warranties and limitation of liability provided 316 | above shall be interpreted in a manner that, to the extent 317 | possible, most closely approximates an absolute disclaimer and 318 | waiver of all liability. 319 | 320 | Section 6 -- Term and Termination. 321 | 322 | a. This Public License applies for the term of the Copyright and 323 | Similar Rights licensed here. However, if You fail to comply with 324 | this Public License, then Your rights under this Public License 325 | terminate automatically. 326 | 327 | b. Where Your right to use the Licensed Material has terminated under 328 | Section 6(a), it reinstates: 329 | 330 | 1. automatically as of the date the violation is cured, provided 331 | it is cured within 30 days of Your discovery of the 332 | violation; or 333 | 334 | 2. upon express reinstatement by the Licensor. 335 | 336 | For the avoidance of doubt, this Section 6(b) does not affect any 337 | right the Licensor may have to seek remedies for Your violations 338 | of this Public License. 339 | 340 | c. For the avoidance of doubt, the Licensor may also offer the 341 | Licensed Material under separate terms or conditions or stop 342 | distributing the Licensed Material at any time; however, doing so 343 | will not terminate this Public License. 344 | 345 | d. Sections 1, 5, 6, 7, and 8 survive termination of this Public 346 | License. 347 | 348 | Section 7 -- Other Terms and Conditions. 349 | 350 | a. The Licensor shall not be bound by any additional or different 351 | terms or conditions communicated by You unless expressly agreed. 352 | 353 | b. Any arrangements, understandings, or agreements regarding the 354 | Licensed Material not stated herein are separate from and 355 | independent of the terms and conditions of this Public License. 356 | 357 | Section 8 -- Interpretation. 358 | 359 | a. For the avoidance of doubt, this Public License does not, and 360 | shall not be interpreted to, reduce, limit, restrict, or impose 361 | conditions on any use of the Licensed Material that could lawfully 362 | be made without permission under this Public License. 363 | 364 | b. To the extent possible, if any provision of this Public License is 365 | deemed unenforceable, it shall be automatically reformed to the 366 | minimum extent necessary to make it enforceable. If the provision 367 | cannot be reformed, it shall be severed from this Public License 368 | without affecting the enforceability of the remaining terms and 369 | conditions. 370 | 371 | c. No term or condition of this Public License will be waived and no 372 | failure to comply consented to unless expressly agreed to by the 373 | Licensor. 374 | 375 | d. Nothing in this Public License constitutes or may be interpreted 376 | as a limitation upon, or waiver of, any privileges and immunities 377 | that apply to the Licensor or You, including from the legal 378 | processes of any jurisdiction or authority. 379 | 380 | ======================================================================= 381 | 382 | Creative Commons is not a party to its public 383 | licenses. Notwithstanding, Creative Commons may elect to apply one of 384 | its public licenses to material it publishes and in those instances 385 | will be considered the “Licensor.” The text of the Creative Commons 386 | public licenses is dedicated to the public domain under the CC0 Public 387 | Domain Dedication. Except for the limited purpose of indicating that 388 | material is shared under a Creative Commons public license or as 389 | otherwise permitted by the Creative Commons policies published at 390 | creativecommons.org/policies, Creative Commons does not authorize the 391 | use of the trademark "Creative Commons" or any other trademark or logo 392 | of Creative Commons without its prior written consent including, 393 | without limitation, in connection with any unauthorized modifications 394 | to any of its public licenses or any other arrangements, 395 | understandings, or agreements concerning use of licensed material. For 396 | the avoidance of doubt, this paragraph does not form part of the 397 | public licenses. 398 | 399 | Creative Commons may be contacted at creativecommons.org. 400 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DeepConvexity 2 | 3 | Code for the ICLR 2018 Workshop paper: 4 | 5 | [Easing non-convex optimization with neural networks](https://openreview.net/forum?id=rJXIPK1PM). 6 | 7 | By David Lopez-Paz and Levent Sagun. 8 | 9 | ## License 10 | 11 | DeepConvexity is Attribution-NonCommercial 4.0 International licensed, as found in the LICENSE file. 12 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2017-present, Facebook, Inc. 2 | # All rights reserved. 3 | # 4 | # This source code is licensed under the license found in the 5 | # LICENSE file in the root directory of this source tree. 6 | # 7 | 8 | from torch.optim.lr_scheduler import ReduceLROnPlateau 9 | import argparse 10 | import torch 11 | import time 12 | import math 13 | import os 14 | 15 | 16 | def rescale(x, a, b, c, d): 17 | """ 18 | Rescales variable from [a, b] to [c, d] 19 | """ 20 | return c + ((d - c) / (b - a)) * (x - a) 21 | 22 | 23 | def ackley(x, xmin=-1, xmax=1): 24 | """ 25 | https://www.sfu.ca/~ssurjano/ackley.html 26 | """ 27 | a = 20 28 | b = 0.2 29 | c = 2 * math.pi 30 | 31 | x = rescale(x, xmin, xmax, -32.768, 32.768) 32 | 33 | term1 = x.pow(2).mean(1).sqrt().mul(-b).exp().mul(-a) 34 | term2 = x.mul(c).cos().mean(1).exp().mul(-1) 35 | 36 | return term1 + term2 + a + math.exp(1) 37 | 38 | 39 | def michal(x, xmin=-1, xmax=1): 40 | """ 41 | https://www.sfu.ca/~ssurjano/michal.html 42 | """ 43 | x = rescale(x, xmin, xmax, 0, math.pi) 44 | 45 | mask = torch.arange(1, x.size(1) + 1).view(1, -1) 46 | 47 | return x.sin().mul((x.pow(2) * mask / math.pi).sin().pow(20)).sum(1).mul(-1) 48 | 49 | 50 | def schwefel(x, xmin=-1, xmax=1): 51 | """ 52 | https://www.sfu.ca/~ssurjano/schwef.html 53 | """ 54 | x = rescale(x, xmin, xmax, -500, 500) 55 | 56 | result = x.abs().sqrt().sin().mul(x).sum(1).mul(-1).add(418.9829 * x.size(1)) 57 | 58 | return result 59 | 60 | 61 | def sphere(x, xmin=-1, xmax=1): 62 | """ 63 | https://www.sfu.ca/~ssurjano/spheref.html 64 | """ 65 | x = rescale(x, xmin, xmax, -5.12, 5.12) 66 | 67 | return x.pow(2).mean(1) 68 | 69 | 70 | def tang(x, xmin=-1, xmax=1): 71 | """ 72 | https://www.sfu.ca/~ssurjano/stybtang.html 73 | """ 74 | x = rescale(x, xmin, xmax, -5, 5) 75 | 76 | result = (x.pow(4) - x.pow(2).mul(16) + x.mul(5)).sum(1).div(2) 77 | 78 | return result 79 | 80 | 81 | def rastrigin(x, xmin=-1, xmax=1): 82 | """ 83 | https://www.sfu.ca/~ssurjano/rastr.html 84 | """ 85 | x = rescale(x, xmin, xmax, -5.12, 5.12) 86 | 87 | result = (x.pow(2.0) - x.mul(2.0 * math.pi).cos().mul(10) 88 | ).sum(1).add(10 * x.size(1)) 89 | 90 | return result 91 | 92 | 93 | def rosenbrock(x, xmin=-1, xmax=1): 94 | """ 95 | https://www.sfu.ca/~ssurjano/rosen.html 96 | """ 97 | x = rescale(x, xmin, xmax, -5, 10) 98 | 99 | term1 = (x[:, 1:] - x[:, :-1].pow(2)).pow(2).mul(100).sum(1) 100 | term2 = x[:, :-1].add(-1).pow(2).sum(1) 101 | 102 | return term1 + term2 103 | 104 | 105 | def levy(x, xmin=-1, xmax=1): 106 | """ 107 | https://www.sfu.ca/~ssurjano/levy.html 108 | """ 109 | x = rescale(x, xmin, xmax, -10, 10) 110 | 111 | w = 1 + (x - 1).div(4) 112 | w1 = w[:, 0] 113 | ww = w[:, :x.size(1) - 1] 114 | wd = w[:, x.size(1) - 1] 115 | 116 | if ww.dim() == 1: 117 | ww = ww.view(-1, 1) 118 | 119 | term1 = w1.mul(math.pi).sin().pow(2) 120 | termw = ww.mul(math.pi).add(1).sin().pow(2).mul( 121 | 10).add(1).mul(ww.add(-1).pow(2)).mean(1) 122 | termd = wd.mul(math.pi * 2).sin().pow(2).add(1).mul(wd.add(-1).pow(2)) 123 | 124 | return term1 + termw + termd 125 | 126 | 127 | def energy(d): 128 | """ 129 | https://en.wikipedia.org/wiki/Spin_glass 130 | """ 131 | coeff = torch.randn(d, d, d) 132 | 133 | def foo(x, xmin=-1, xmax=1): 134 | x = rescale(x, xmin, xmax, -1, 1) 135 | x_norm = x.norm(2, 1).view(-1, 1) + 1e-10 136 | x = x / x_norm * math.sqrt(x.size(1)) 137 | bs = x.size(0) 138 | tmp1 = torch.mul(x.view(bs, d, 1), x.view(bs, 1, d)) 139 | tmp2 = torch.mul(tmp1.view(bs, d * d, 1), x.view(bs, 1, d)) 140 | tmp3 = torch.mul(tmp2.view(bs, d * d * d), coeff.view(1, -1)) 141 | return tmp3.sum(1) / d 142 | 143 | return foo 144 | 145 | 146 | test_functions = { 147 | "ackley": ackley, 148 | "michal": michal, 149 | "schwefel": schwefel, 150 | "sphere": sphere, 151 | "tang": tang, 152 | "rastrigin": rastrigin, 153 | "rosenbrock": rosenbrock, 154 | "levy": levy 155 | } 156 | 157 | 158 | class ScaledTanh(torch.nn.Module): 159 | def __init__(self, a=1): 160 | super(ScaledTanh, self).__init__() 161 | self.a = a 162 | 163 | def forward(self, x): 164 | return x.mul(self.a).tanh() 165 | 166 | 167 | class EmbeddingPerceptron(torch.nn.Module): 168 | """ 169 | Multilayer ReLU perceptron with learnable inputs 170 | """ 171 | 172 | def __init__(self, sizes, multiplier=3): 173 | super(EmbeddingPerceptron, self).__init__() 174 | self.inputs = torch.arange(0, sizes[0]).long() 175 | 176 | layers = [torch.nn.Embedding(sizes[0], sizes[1])] 177 | for i in range(1, len(sizes) - 1): 178 | layers.append(torch.nn.Linear(sizes[i], sizes[i + 1])) 179 | if i < (len(sizes) - 2): 180 | layers.append(torch.nn.ReLU()) 181 | 182 | self.net = torch.nn.Sequential(*layers) 183 | 184 | net_min, net_max = self().min().item(), self().max().item() 185 | a = 1.7159 / max(abs(net_min), abs(net_max)) 186 | self.net = torch.nn.Sequential(self.net, ScaledTanh(a)) 187 | 188 | 189 | def forward(self): 190 | return self.net(self.inputs) 191 | 192 | 193 | def run_experiment(args, lr): 194 | if args.seed >= 0: 195 | torch.manual_seed(args.seed) 196 | 197 | if args.function == "energy": 198 | the_function = energy(args.dimension) 199 | else: 200 | the_function = test_functions[args.function] 201 | 202 | network = EmbeddingPerceptron([args.n_embeddings] + 203 | [args.n_hiddens] * args.n_layers + 204 | [args.dimension]) 205 | 206 | optimizer = torch.optim.SGD(network.parameters(), lr) 207 | 208 | scheduler = ReduceLROnPlateau(optimizer, 209 | patience=1, 210 | factor=0.99, 211 | threshold=1e-10, 212 | min_lr=1e-10, 213 | threshold_mode='abs') 214 | 215 | time_start = time.time() 216 | 217 | history = [] 218 | 219 | for i in range(args.n_iters): 220 | errors = the_function(network()) 221 | mean_error = errors.mean() 222 | min_error = errors.min() 223 | 224 | optimizer.zero_grad() 225 | mean_error.backward() 226 | optimizer.step() 227 | 228 | history.append([i, time.time() - time_start, min_error.item()]) 229 | 230 | if min_error.item() != min_error.item(): 231 | return None 232 | 233 | scheduler.step(mean_error.item()) 234 | 235 | return torch.Tensor(history) 236 | 237 | 238 | if __name__ == "__main__": 239 | parser = argparse.ArgumentParser(description='Deep convexity') 240 | parser.add_argument('--save_dir', type=str, default='results/') 241 | parser.add_argument('--function', type=str, default="ackley") 242 | parser.add_argument('--dimension', type=int, default=50) 243 | 244 | parser.add_argument('--n_iters', type=int, default=10000) 245 | parser.add_argument('--min_lr', type=float, default=-6) 246 | parser.add_argument('--max_lr', type=float, default=1) 247 | parser.add_argument('--n_lr', type=int, default=20) 248 | 249 | parser.add_argument('--n_embeddings', type=int, default=50) 250 | parser.add_argument('--n_hiddens', type=int, default=50) 251 | parser.add_argument('--n_layers', type=int, default=0) 252 | 253 | parser.add_argument('--seed', type=int, default=0) 254 | args = parser.parse_args() 255 | 256 | torch.set_default_tensor_type(torch.DoubleTensor) 257 | 258 | best_history = None 259 | best_lr = None 260 | best_err = 1e10 261 | 262 | for lr in torch.logspace(args.min_lr, args.max_lr, args.n_lr): 263 | history = run_experiment(args, lr) 264 | 265 | if history is not None: 266 | err = history[:, -1].min().item() 267 | if err < best_err: 268 | best_lr = lr 269 | best_err = err 270 | best_history = history 271 | #print("+ Success at lr={:1.0e} with value {:.5f}".format(lr, err)) 272 | #else: 273 | #print("- Failure at lr={:1.0e}".format(lr)) 274 | 275 | print("* Best run at lr={:1.0e} with value {:.5f}".format(best_lr, best_err)) 276 | 277 | 278 | fname = "f={}_emb={}_lay={}_hid={}_seed={}.txt".format(args.function, 279 | args.n_embeddings, 280 | args.n_layers, 281 | args.n_hiddens, 282 | args.seed) 283 | 284 | if not os.path.exists(args.save_dir): 285 | os.makedirs(args.save_dir) 286 | 287 | with open(os.path.join(args.save_dir, fname), "w") as f: 288 | f.write("# {}\n".format(str(vars(args)))) 289 | for line in best_history: 290 | f.write("{:1.0e} {:.0f} {:.5f} {:.5f}\n".format(best_lr, *line)) 291 | -------------------------------------------------------------------------------- /plot.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2017-present, Facebook, Inc. 2 | # All rights reserved. 3 | # 4 | # This source code is licensed under the license found in the 5 | # LICENSE file in the root directory of this source tree. 6 | # 7 | 8 | import argparse 9 | import glob 10 | import os 11 | 12 | import matplotlib 13 | matplotlib.use('Agg') 14 | 15 | from matplotlib import pyplot as plt 16 | plt.rc('font', family='serif') 17 | plt.rc('font', size=12) 18 | 19 | import numpy as np 20 | 21 | 22 | if __name__ == "__main__": 23 | parser = argparse.ArgumentParser(description='Deep convexity') 24 | parser.add_argument('--save_dir', type=str, default='results/') 25 | parser.add_argument('--plot_dir', type=str, default='plots/') 26 | parser.add_argument('--n_layers', type=int, default=2) 27 | parser.add_argument('--n_hiddens', type=int, default=50) 28 | parser.add_argument('--n_embeddings', type=int, default=50) 29 | args = parser.parse_args() 30 | 31 | functions = [ 32 | "ackley", 33 | "michal", 34 | "schwefel", 35 | "sphere", 36 | "tang", 37 | "rastrigin", 38 | "rosenbrock", 39 | "levy", 40 | "energy" 41 | ] 42 | 43 | plt.figure(figsize=(10, 6)) 44 | 45 | for f, function in enumerate(functions): 46 | plt.subplot(3, 3, f + 1) 47 | 48 | shallows = glob.glob(os.path.join(args.save_dir, 49 | "f={}_emb={}_lay=0_hid={}_*".format(function, 50 | args.n_embeddings, 51 | args.n_hiddens))) 52 | 53 | deeps = glob.glob(os.path.join(args.save_dir, 54 | "f={}_emb={}_lay={}_hid={}_*".format(function, 55 | args.n_embeddings, 56 | args.n_layers, 57 | args.n_hiddens))) 58 | print(len(shallows), len(deeps)) 59 | 60 | for shallow in shallows: 61 | data = np.genfromtxt(shallow) 62 | if len(data): 63 | data = data[:, 2:] 64 | plt.plot(data[:, 0], data[:, 1], color="C2", alpha=0.5) 65 | 66 | for deep in deeps: 67 | data = np.genfromtxt(deep) 68 | if len(data): 69 | data = data[:, 2:] 70 | plt.plot(data[:, 0], data[:, 1], color="C1", alpha=0.5) 71 | 72 | plt.xscale("log") 73 | if function == "rastrigin" or function == "rosenbrock": 74 | plt.yscale("log") 75 | 76 | plt.margins(0.05) 77 | plt.title(function) 78 | 79 | plt.tight_layout(pad=0) 80 | pdf_name = os.path.join(args.plot_dir, 81 | 'plot_{}_{}.pdf'.format(args.n_layers, 82 | args.n_hiddens)) 83 | 84 | if not os.path.exists(args.plot_dir): 85 | os.makedirs(args.plot_dir) 86 | 87 | plt.savefig(pdf_name) 88 | --------------------------------------------------------------------------------