├── ACKNOWLEDGMENTS ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── PrivUnitAlgs.py ├── README.md ├── SQKR.py ├── dict_perm.npy ├── experiments.py ├── experiments_MNIST.py ├── gaussian_analytic.py ├── requirements.txt ├── setup.sh ├── train_MNIST_script.sh ├── train_mnist.py └── utilities.py /ACKNOWLEDGMENTS: -------------------------------------------------------------------------------- 1 | Acknowledgements 2 | Portions of this ml-projunit Software may utilize the following copyrighted 3 | material, the use of which is hereby acknowledged. 4 | 5 | _____________________ 6 | 7 | 8 | NumPy Developers (NumPy) 9 | Copyright (c) 2005-2022, NumPy Developers. 10 | All rights reserved. 11 | 12 | Redistribution and use in source and binary forms, with or without 13 | modification, are permitted provided that the following conditions are 14 | met: 15 | 16 | * Redistributions of source code must retain the above copyright 17 | notice, this list of conditions and the following disclaimer. 18 | 19 | * Redistributions in binary form must reproduce the above 20 | copyright notice, this list of conditions and the following 21 | disclaimer in the documentation and/or other materials provided 22 | with the distribution. 23 | 24 | * Neither the name of the NumPy Developers nor the names of any 25 | contributors may be used to endorse or promote products derived 26 | from this software without specific prior written permission. 27 | 28 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 29 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 30 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 31 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 32 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 33 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 34 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 35 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 36 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 37 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 38 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 39 | 40 | 41 | 42 | 43 | tqdm developers (tqdm) 44 | `tqdm` is a product of collaborative work. 45 | Unless otherwise stated, all authors (see commit logs) retain copyright 46 | for their respective work, and release the work under the MIT licence 47 | (text below). 48 | 49 | Exceptions or notable authors are listed below 50 | in reverse chronological order: 51 | 52 | * files: * 53 | MPLv2.0 2015-2021 (c) Casper da Costa-Luis 54 | [casperdcl](https://github.com/casperdcl). 55 | * files: tqdm/_tqdm.py 56 | MIT 2016 (c) [PR #96] on behalf of Google Inc. 57 | * files: tqdm/_tqdm.py setup.py README.rst MANIFEST.in .gitignore 58 | MIT 2013 (c) Noam Yorav-Raphael, original author. 59 | 60 | 61 | 62 | The PyTorch Authors (PyTorch) 63 | From PyTorch: 64 | 65 | Copyright (c) 2016- Facebook, Inc (Adam Paszke) 66 | Copyright (c) 2014- Facebook, Inc (Soumith Chintala) 67 | Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert) 68 | Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu) 69 | Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu) 70 | Copyright (c) 2011-2013 NYU (Clement Farabet) 71 | Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston) 72 | Copyright (c) 2006 Idiap Research Institute (Samy Bengio) 73 | Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz) 74 | 75 | From Caffe2: 76 | 77 | Copyright (c) 2016-present, Facebook Inc. All rights reserved. 78 | 79 | All contributions by Facebook: 80 | Copyright (c) 2016 Facebook Inc. 81 | 82 | All contributions by Google: 83 | Copyright (c) 2015 Google Inc. 84 | All rights reserved. 85 | 86 | All contributions by Yangqing Jia: 87 | Copyright (c) 2015 Yangqing Jia 88 | All rights reserved. 89 | 90 | All contributions from Caffe: 91 | Copyright(c) 2013, 2014, 2015, the respective contributors 92 | All rights reserved. 93 | 94 | All other contributions: 95 | Copyright(c) 2015, 2016 the respective contributors 96 | All rights reserved. 97 | 98 | Caffe2 uses a copyright model similar to Caffe: each contributor holds 99 | copyright over their contributions to Caffe2. The project versioning records 100 | all such contribution and copyright details. If a contributor wants to further 101 | mark their specific copyright on a particular contribution, they should 102 | indicate their copyright solely in the commit message of the change when it is 103 | committed. 104 | 105 | All rights reserved. 106 | 107 | Redistribution and use in source and binary forms, with or without 108 | modification, are permitted provided that the following conditions are met: 109 | 110 | 1. Redistributions of source code must retain the above copyright 111 | notice, this list of conditions and the following disclaimer. 112 | 113 | 2. Redistributions in binary form must reproduce the above copyright 114 | notice, this list of conditions and the following disclaimer in the 115 | documentation and/or other materials provided with the distribution. 116 | 117 | 3. Neither the names of Facebook, Deepmind Technologies, NYU, NEC Laboratories America 118 | and IDIAP Research Institute nor the names of its contributors may be 119 | used to endorse or promote products derived from this software without 120 | specific prior written permission. 121 | 122 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 123 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 124 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 125 | ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 126 | LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 127 | CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 128 | SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 129 | INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 130 | CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 131 | ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 132 | POSSIBILITY OF SUCH DAMAGE. 133 | 134 | 135 | 136 | The torchvision Authors (torchvision) 137 | BSD 3-Clause License 138 | 139 | Copyright (c) Soumith Chintala 2016, 140 | All rights reserved. 141 | 142 | Redistribution and use in source and binary forms, with or without 143 | modification, are permitted provided that the following conditions are met: 144 | 145 | * Redistributions of source code must retain the above copyright notice, this 146 | list of conditions and the following disclaimer. 147 | 148 | * Redistributions in binary form must reproduce the above copyright notice, 149 | this list of conditions and the following disclaimer in the documentation 150 | and/or other materials provided with the distribution. 151 | 152 | * Neither the name of the copyright holder nor the names of its 153 | contributors may be used to endorse or promote products derived from 154 | this software without specific prior written permission. 155 | 156 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 157 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 158 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 159 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 160 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 161 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 162 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 163 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 164 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 165 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 166 | 167 | 168 | 169 | The SciPy Authors (SciPy) 170 | 171 | Copyright © 2001, 2002 Enthought, Inc. 172 | All rights reserved. 173 | 174 | Copyright © 2003-2019 SciPy Developers. 175 | All rights reserved. 176 | Redistribution and use in source and binary forms, with or without modification, 177 | are permitted provided that the following conditions are met: 178 | 179 | Redistributions of source code must retain the above copyright notice, 180 | this list of conditions and the following disclaimer. 181 | Redistributions in binary form must reproduce the above copyright notice, 182 | this list of conditions and the following disclaimer in the documentation 183 | and/or other materials provided with the distribution. 184 | Neither the name of Enthought nor the names of the SciPy Developers may be used 185 | to endorse or promote products derived from this software without specific prior written permission. 186 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” 187 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, 188 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 189 | IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 190 | INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, 191 | PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) 192 | HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 193 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, 194 | EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 195 | 196 | 197 | 198 | The Opacus Authors (Opacus) 199 | 200 | 201 | 1. Definitions. 202 | 203 | "License" shall mean the terms and conditions for use, reproduction, 204 | and distribution as defined by Sections 1 through 9 of this document. 205 | 206 | "Licensor" shall mean the copyright owner or entity authorized by 207 | the copyright owner that is granting the License. 208 | 209 | "Legal Entity" shall mean the union of the acting entity and all 210 | other entities that control, are controlled by, or are under common 211 | control with that entity. For the purposes of this definition, 212 | "control" means (i) the power, direct or indirect, to cause the 213 | direction or management of such entity, whether by contract or 214 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 215 | outstanding shares, or (iii) beneficial ownership of such entity. 216 | 217 | "You" (or "Your") shall mean an individual or Legal Entity 218 | exercising permissions granted by this License. 219 | 220 | "Source" form shall mean the preferred form for making modifications, 221 | including but not limited to software source code, documentation 222 | source, and configuration files. 223 | 224 | "Object" form shall mean any form resulting from mechanical 225 | transformation or translation of a Source form, including but 226 | not limited to compiled object code, generated documentation, 227 | and conversions to other media types. 228 | 229 | "Work" shall mean the work of authorship, whether in Source or 230 | Object form, made available under the License, as indicated by a 231 | copyright notice that is included in or attached to the work 232 | (an example is provided in the Appendix below). 233 | 234 | "Derivative Works" shall mean any work, whether in Source or Object 235 | form, that is based on (or derived from) the Work and for which the 236 | editorial revisions, annotations, elaborations, or other modifications 237 | represent, as a whole, an original work of authorship. For the purposes 238 | of this License, Derivative Works shall not include works that remain 239 | separable from, or merely link (or bind by name) to the interfaces of, 240 | the Work and Derivative Works thereof. 241 | 242 | "Contribution" shall mean any work of authorship, including 243 | the original version of the Work and any modifications or additions 244 | to that Work or Derivative Works thereof, that is intentionally 245 | submitted to Licensor for inclusion in the Work by the copyright owner 246 | or by an individual or Legal Entity authorized to submit on behalf of 247 | the copyright owner. For the purposes of this definition, "submitted" 248 | means any form of electronic, verbal, or written communication sent 249 | to the Licensor or its representatives, including but not limited to 250 | communication on electronic mailing lists, source code control systems, 251 | and issue tracking systems that are managed by, or on behalf of, the 252 | Licensor for the purpose of discussing and improving the Work, but 253 | excluding communication that is conspicuously marked or otherwise 254 | designated in writing by the copyright owner as "Not a Contribution." 255 | 256 | "Contributor" shall mean Licensor and any individual or Legal Entity 257 | on behalf of whom a Contribution has been received by Licensor and 258 | subsequently incorporated within the Work. 259 | 260 | 2. Grant of Copyright License. Subject to the terms and conditions of 261 | this License, each Contributor hereby grants to You a perpetual, 262 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 263 | copyright license to reproduce, prepare Derivative Works of, 264 | publicly display, publicly perform, sublicense, and distribute the 265 | Work and such Derivative Works in Source or Object form. 266 | 267 | 3. Grant of Patent License. Subject to the terms and conditions of 268 | this License, each Contributor hereby grants to You a perpetual, 269 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 270 | (except as stated in this section) patent license to make, have made, 271 | use, offer to sell, sell, import, and otherwise transfer the Work, 272 | where such license applies only to those patent claims licensable 273 | by such Contributor that are necessarily infringed by their 274 | Contribution(s) alone or by combination of their Contribution(s) 275 | with the Work to which such Contribution(s) was submitted. If You 276 | institute patent litigation against any entity (including a 277 | cross-claim or counterclaim in a lawsuit) alleging that the Work 278 | or a Contribution incorporated within the Work constitutes direct 279 | or contributory patent infringement, then any patent licenses 280 | granted to You under this License for that Work shall terminate 281 | as of the date such litigation is filed. 282 | 283 | 4. Redistribution. You may reproduce and distribute copies of the 284 | Work or Derivative Works thereof in any medium, with or without 285 | modifications, and in Source or Object form, provided that You 286 | meet the following conditions: 287 | 288 | (a) You must give any other recipients of the Work or 289 | Derivative Works a copy of this License; and 290 | 291 | (b) You must cause any modified files to carry prominent notices 292 | stating that You changed the files; and 293 | 294 | (c) You must retain, in the Source form of any Derivative Works 295 | that You distribute, all copyright, patent, trademark, and 296 | attribution notices from the Source form of the Work, 297 | excluding those notices that do not pertain to any part of 298 | the Derivative Works; and 299 | 300 | (d) If the Work includes a "NOTICE" text file as part of its 301 | distribution, then any Derivative Works that You distribute must 302 | include a readable copy of the attribution notices contained 303 | within such NOTICE file, excluding those notices that do not 304 | pertain to any part of the Derivative Works, in at least one 305 | of the following places: within a NOTICE text file distributed 306 | as part of the Derivative Works; within the Source form or 307 | documentation, if provided along with the Derivative Works; or, 308 | within a display generated by the Derivative Works, if and 309 | wherever such third-party notices normally appear. The contents 310 | of the NOTICE file are for informational purposes only and 311 | do not modify the License. You may add Your own attribution 312 | notices within Derivative Works that You distribute, alongside 313 | or as an addendum to the NOTICE text from the Work, provided 314 | that such additional attribution notices cannot be construed 315 | as modifying the License. 316 | 317 | You may add Your own copyright statement to Your modifications and 318 | may provide additional or different license terms and conditions 319 | for use, reproduction, or distribution of Your modifications, or 320 | for any such Derivative Works as a whole, provided Your use, 321 | reproduction, and distribution of the Work otherwise complies with 322 | the conditions stated in this License. 323 | 324 | 5. Submission of Contributions. Unless You explicitly state otherwise, 325 | any Contribution intentionally submitted for inclusion in the Work 326 | by You to the Licensor shall be under the terms and conditions of 327 | this License, without any additional terms or conditions. 328 | Notwithstanding the above, nothing herein shall supersede or modify 329 | the terms of any separate license agreement you may have executed 330 | with Licensor regarding such Contributions. 331 | 332 | 6. Trademarks. This License does not grant permission to use the trade 333 | names, trademarks, service marks, or product names of the Licensor, 334 | except as required for reasonable and customary use in describing the 335 | origin of the Work and reproducing the content of the NOTICE file. 336 | 337 | 7. Disclaimer of Warranty. Unless required by applicable law or 338 | agreed to in writing, Licensor provides the Work (and each 339 | Contributor provides its Contributions) on an "AS IS" BASIS, 340 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 341 | implied, including, without limitation, any warranties or conditions 342 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 343 | PARTICULAR PURPOSE. You are solely responsible for determining the 344 | appropriateness of using or redistributing the Work and assume any 345 | risks associated with Your exercise of permissions under this License. 346 | 347 | 8. Limitation of Liability. In no event and under no legal theory, 348 | whether in tort (including negligence), contract, or otherwise, 349 | unless required by applicable law (such as deliberate and grossly 350 | negligent acts) or agreed to in writing, shall any Contributor be 351 | liable to You for damages, including any direct, indirect, special, 352 | incidental, or consequential damages of any character arising as a 353 | result of this License or out of the use or inability to use the 354 | Work (including but not limited to damages for loss of goodwill, 355 | work stoppage, computer failure or malfunction, or any and all 356 | other commercial damages or losses), even if such Contributor 357 | has been advised of the possibility of such damages. 358 | 359 | 9. Accepting Warranty or Additional Liability. While redistributing 360 | the Work or Derivative Works thereof, You may choose to offer, 361 | and charge a fee for, acceptance of support, warranty, indemnity, 362 | or other liability obligations and/or rights consistent with this 363 | License. However, in accepting such obligations, You may act only 364 | on Your own behalf and on Your sole responsibility, not on behalf 365 | of any other Contributor, and only if You agree to indemnify, 366 | defend, and hold each Contributor harmless for any liability 367 | incurred by, or claims asserted against, such Contributor by reason 368 | of your accepting any such warranty or additional liability. 369 | 370 | 371 | 372 | 373 | The Matplotlib Authors (Matplotlib) 374 | 375 | 376 | License agreement for matplotlib versions 1.3.0 and later 377 | ========================================================= 378 | 379 | 1. This LICENSE AGREEMENT is between the Matplotlib Development Team 380 | ("MDT"), and the Individual or Organization ("Licensee") accessing and 381 | otherwise using matplotlib software in source or binary form and its 382 | associated documentation. 383 | 384 | 2. Subject to the terms and conditions of this License Agreement, MDT 385 | hereby grants Licensee a nonexclusive, royalty-free, world-wide license 386 | to reproduce, analyze, test, perform and/or display publicly, prepare 387 | derivative works, distribute, and otherwise use matplotlib 388 | alone or in any derivative version, provided, however, that MDT's 389 | License Agreement and MDT's notice of copyright, i.e., "Copyright (c) 390 | 2012- Matplotlib Development Team; All Rights Reserved" are retained in 391 | matplotlib alone or in any derivative version prepared by 392 | Licensee. 393 | 394 | 3. In the event Licensee prepares a derivative work that is based on or 395 | incorporates matplotlib or any part thereof, and wants to 396 | make the derivative work available to others as provided herein, then 397 | Licensee hereby agrees to include in any such work a brief summary of 398 | the changes made to matplotlib . 399 | 400 | 4. MDT is making matplotlib available to Licensee on an "AS 401 | IS" basis. MDT MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR 402 | IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, MDT MAKES NO AND 403 | DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS 404 | FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF MATPLOTLIB 405 | WILL NOT INFRINGE ANY THIRD PARTY RIGHTS. 406 | 407 | 5. MDT SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF MATPLOTLIB 408 | FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR 409 | LOSS AS A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING 410 | MATPLOTLIB , OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF 411 | THE POSSIBILITY THEREOF. 412 | 413 | 6. This License Agreement will automatically terminate upon a material 414 | breach of its terms and conditions. 415 | 416 | 7. Nothing in this License Agreement shall be deemed to create any 417 | relationship of agency, partnership, or joint venture between MDT and 418 | Licensee. This License Agreement does not grant permission to use MDT 419 | trademarks or trade name in a trademark sense to endorse or promote 420 | products or services of Licensee, or any third party. 421 | 422 | 8. By copying, installing or otherwise using matplotlib , 423 | Licensee agrees to be bound by the terms and conditions of this License 424 | Agreement. 425 | 426 | License agreement for matplotlib versions prior to 1.3.0 427 | ======================================================== 428 | 429 | 1. This LICENSE AGREEMENT is between John D. Hunter ("JDH"), and the 430 | Individual or Organization ("Licensee") accessing and otherwise using 431 | matplotlib software in source or binary form and its associated 432 | documentation. 433 | 434 | 2. Subject to the terms and conditions of this License Agreement, JDH 435 | hereby grants Licensee a nonexclusive, royalty-free, world-wide license 436 | to reproduce, analyze, test, perform and/or display publicly, prepare 437 | derivative works, distribute, and otherwise use matplotlib 438 | alone or in any derivative version, provided, however, that JDH's 439 | License Agreement and JDH's notice of copyright, i.e., "Copyright (c) 440 | 2002-2011 John D. Hunter; All Rights Reserved" are retained in 441 | matplotlib alone or in any derivative version prepared by 442 | Licensee. 443 | 444 | 3. In the event Licensee prepares a derivative work that is based on or 445 | incorporates matplotlib or any part thereof, and wants to 446 | make the derivative work available to others as provided herein, then 447 | Licensee hereby agrees to include in any such work a brief summary of 448 | the changes made to matplotlib. 449 | 450 | 4. JDH is making matplotlib available to Licensee on an "AS 451 | IS" basis. JDH MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR 452 | IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, JDH MAKES NO AND 453 | DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS 454 | FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF MATPLOTLIB 455 | WILL NOT INFRINGE ANY THIRD PARTY RIGHTS. 456 | 457 | 5. JDH SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF MATPLOTLIB 458 | FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR 459 | LOSS AS A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING 460 | MATPLOTLIB , OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF 461 | THE POSSIBILITY THEREOF. 462 | 463 | 6. This License Agreement will automatically terminate upon a material 464 | breach of its terms and conditions. 465 | 466 | 7. Nothing in this License Agreement shall be deemed to create any 467 | relationship of agency, partnership, or joint venture between JDH and 468 | Licensee. This License Agreement does not grant permission to use JDH 469 | trademarks or trade name in a trademark sense to endorse or promote 470 | products or services of Licensee, or any third party. 471 | 472 | 8. By copying, installing or otherwise using matplotlib, 473 | Licensee agrees to be bound by the terms and conditions of this License 474 | Agreement. 475 | 476 | 477 | 478 | Borja Balle (analytic-gaussian-mechanism) 479 | 480 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 481 | 482 | 1. Definitions. 483 | 484 | "License" shall mean the terms and conditions for use, reproduction, 485 | and distribution as defined by Sections 1 through 9 of this document. 486 | 487 | "Licensor" shall mean the copyright owner or entity authorized by 488 | the copyright owner that is granting the License. 489 | 490 | "Legal Entity" shall mean the union of the acting entity and all 491 | other entities that control, are controlled by, or are under common 492 | control with that entity. For the purposes of this definition, 493 | "control" means (i) the power, direct or indirect, to cause the 494 | direction or management of such entity, whether by contract or 495 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 496 | outstanding shares, or (iii) beneficial ownership of such entity. 497 | 498 | "You" (or "Your") shall mean an individual or Legal Entity 499 | exercising permissions granted by this License. 500 | 501 | "Source" form shall mean the preferred form for making modifications, 502 | including but not limited to software source code, documentation 503 | source, and configuration files. 504 | 505 | "Object" form shall mean any form resulting from mechanical 506 | transformation or translation of a Source form, including but 507 | not limited to compiled object code, generated documentation, 508 | and conversions to other media types. 509 | 510 | "Work" shall mean the work of authorship, whether in Source or 511 | Object form, made available under the License, as indicated by a 512 | copyright notice that is included in or attached to the work 513 | (an example is provided in the Appendix below). 514 | 515 | "Derivative Works" shall mean any work, whether in Source or Object 516 | form, that is based on (or derived from) the Work and for which the 517 | editorial revisions, annotations, elaborations, or other modifications 518 | represent, as a whole, an original work of authorship. For the purposes 519 | of this License, Derivative Works shall not include works that remain 520 | separable from, or merely link (or bind by name) to the interfaces of, 521 | the Work and Derivative Works thereof. 522 | 523 | "Contribution" shall mean any work of authorship, including 524 | the original version of the Work and any modifications or additions 525 | to that Work or Derivative Works thereof, that is intentionally 526 | submitted to Licensor for inclusion in the Work by the copyright owner 527 | or by an individual or Legal Entity authorized to submit on behalf of 528 | the copyright owner. For the purposes of this definition, "submitted" 529 | means any form of electronic, verbal, or written communication sent 530 | to the Licensor or its representatives, including but not limited to 531 | communication on electronic mailing lists, source code control systems, 532 | and issue tracking systems that are managed by, or on behalf of, the 533 | Licensor for the purpose of discussing and improving the Work, but 534 | excluding communication that is conspicuously marked or otherwise 535 | designated in writing by the copyright owner as "Not a Contribution." 536 | 537 | "Contributor" shall mean Licensor and any individual or Legal Entity 538 | on behalf of whom a Contribution has been received by Licensor and 539 | subsequently incorporated within the Work. 540 | 541 | 2. Grant of Copyright License. Subject to the terms and conditions of 542 | this License, each Contributor hereby grants to You a perpetual, 543 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 544 | copyright license to reproduce, prepare Derivative Works of, 545 | publicly display, publicly perform, sublicense, and distribute the 546 | Work and such Derivative Works in Source or Object form. 547 | 548 | 3. Grant of Patent License. Subject to the terms and conditions of 549 | this License, each Contributor hereby grants to You a perpetual, 550 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 551 | (except as stated in this section) patent license to make, have made, 552 | use, offer to sell, sell, import, and otherwise transfer the Work, 553 | where such license applies only to those patent claims licensable 554 | by such Contributor that are necessarily infringed by their 555 | Contribution(s) alone or by combination of their Contribution(s) 556 | with the Work to which such Contribution(s) was submitted. If You 557 | institute patent litigation against any entity (including a 558 | cross-claim or counterclaim in a lawsuit) alleging that the Work 559 | or a Contribution incorporated within the Work constitutes direct 560 | or contributory patent infringement, then any patent licenses 561 | granted to You under this License for that Work shall terminate 562 | as of the date such litigation is filed. 563 | 564 | 4. Redistribution. You may reproduce and distribute copies of the 565 | Work or Derivative Works thereof in any medium, with or without 566 | modifications, and in Source or Object form, provided that You 567 | meet the following conditions: 568 | 569 | (a) You must give any other recipients of the Work or 570 | Derivative Works a copy of this License; and 571 | 572 | (b) You must cause any modified files to carry prominent notices 573 | stating that You changed the files; and 574 | 575 | (c) You must retain, in the Source form of any Derivative Works 576 | that You distribute, all copyright, patent, trademark, and 577 | attribution notices from the Source form of the Work, 578 | excluding those notices that do not pertain to any part of 579 | the Derivative Works; and 580 | 581 | (d) If the Work includes a "NOTICE" text file as part of its 582 | distribution, then any Derivative Works that You distribute must 583 | include a readable copy of the attribution notices contained 584 | within such NOTICE file, excluding those notices that do not 585 | pertain to any part of the Derivative Works, in at least one 586 | of the following places: within a NOTICE text file distributed 587 | as part of the Derivative Works; within the Source form or 588 | documentation, if provided along with the Derivative Works; or, 589 | within a display generated by the Derivative Works, if and 590 | wherever such third-party notices normally appear. The contents 591 | of the NOTICE file are for informational purposes only and 592 | do not modify the License. You may add Your own attribution 593 | notices within Derivative Works that You distribute, alongside 594 | or as an addendum to the NOTICE text from the Work, provided 595 | that such additional attribution notices cannot be construed 596 | as modifying the License. 597 | 598 | You may add Your own copyright statement to Your modifications and 599 | may provide additional or different license terms and conditions 600 | for use, reproduction, or distribution of Your modifications, or 601 | for any such Derivative Works as a whole, provided Your use, 602 | reproduction, and distribution of the Work otherwise complies with 603 | the conditions stated in this License. 604 | 605 | 5. Submission of Contributions. Unless You explicitly state otherwise, 606 | any Contribution intentionally submitted for inclusion in the Work 607 | by You to the Licensor shall be under the terms and conditions of 608 | this License, without any additional terms or conditions. 609 | Notwithstanding the above, nothing herein shall supersede or modify 610 | the terms of any separate license agreement you may have executed 611 | with Licensor regarding such Contributions. 612 | 613 | 6. Trademarks. This License does not grant permission to use the trade 614 | names, trademarks, service marks, or product names of the Licensor, 615 | except as required for reasonable and customary use in describing the 616 | origin of the Work and reproducing the content of the NOTICE file. 617 | 618 | 7. Disclaimer of Warranty. Unless required by applicable law or 619 | agreed to in writing, Licensor provides the Work (and each 620 | Contributor provides its Contributions) on an "AS IS" BASIS, 621 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 622 | implied, including, without limitation, any warranties or conditions 623 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 624 | PARTICULAR PURPOSE. You are solely responsible for determining the 625 | appropriateness of using or redistributing the Work and assume any 626 | risks associated with Your exercise of permissions under this License. 627 | 628 | 8. Limitation of Liability. In no event and under no legal theory, 629 | whether in tort (including negligence), contract, or otherwise, 630 | unless required by applicable law (such as deliberate and grossly 631 | negligent acts) or agreed to in writing, shall any Contributor be 632 | liable to You for damages, including any direct, indirect, special, 633 | incidental, or consequential damages of any character arising as a 634 | result of this License or out of the use or inability to use the 635 | Work (including but not limited to damages for loss of goodwill, 636 | work stoppage, computer failure or malfunction, or any and all 637 | other commercial damages or losses), even if such Contributor 638 | has been advised of the possibility of such damages. 639 | 640 | 9. Accepting Warranty or Additional Liability. While redistributing 641 | the Work or Derivative Works thereof, You may choose to offer, 642 | and charge a fee for, acceptance of support, warranty, indemnity, 643 | or other liability obligations and/or rights consistent with this 644 | License. However, in accepting such obligations, You may act only 645 | on Your own behalf and on Your sole responsibility, not on behalf 646 | of any other Contributor, and only if You agree to indemnify, 647 | defend, and hold each Contributor harmless for any liability 648 | incurred by, or claims asserted against, such Contributor by reason 649 | of your accepting any such warranty or additional liability. 650 | 651 | END OF TERMS AND CONDITIONS 652 | 653 | APPENDIX: How to apply the Apache License to your work. 654 | 655 | To apply the Apache License to your work, attach the following 656 | boilerplate notice, with the fields enclosed by brackets "[]" 657 | replaced with your own identifying information. (Don't include 658 | the brackets!) The text should be enclosed in the appropriate 659 | comment syntax for the file format. We also recommend that a 660 | file or class name and description of purpose be included on the 661 | same "printed page" as the copyright notice for easier 662 | identification within third-party archives. 663 | 664 | Copyright [yyyy] [name of copyright owner] 665 | 666 | Licensed under the Apache License, Version 2.0 (the "License"); 667 | you may not use this file except in compliance with the License. 668 | You may obtain a copy of the License at 669 | 670 | http://www.apache.org/licenses/LICENSE-2.0 671 | 672 | Unless required by applicable law or agreed to in writing, software 673 | distributed under the License is distributed on an "AS IS" BASIS, 674 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 675 | See the License for the specific language governing permissions and 676 | limitations under the License. 677 | 678 | 679 | 680 | SQKR developers (SQKR) 681 | 682 | Apache License 683 | Version 2.0, January 2004 684 | http://www.apache.org/licenses/ 685 | 686 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 687 | 688 | 1. Definitions. 689 | 690 | "License" shall mean the terms and conditions for use, reproduction, 691 | and distribution as defined by Sections 1 through 9 of this document. 692 | 693 | "Licensor" shall mean the copyright owner or entity authorized by 694 | the copyright owner that is granting the License. 695 | 696 | "Legal Entity" shall mean the union of the acting entity and all 697 | other entities that control, are controlled by, or are under common 698 | control with that entity. For the purposes of this definition, 699 | "control" means (i) the power, direct or indirect, to cause the 700 | direction or management of such entity, whether by contract or 701 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 702 | outstanding shares, or (iii) beneficial ownership of such entity. 703 | 704 | "You" (or "Your") shall mean an individual or Legal Entity 705 | exercising permissions granted by this License. 706 | 707 | "Source" form shall mean the preferred form for making modifications, 708 | including but not limited to software source code, documentation 709 | source, and configuration files. 710 | 711 | "Object" form shall mean any form resulting from mechanical 712 | transformation or translation of a Source form, including but 713 | not limited to compiled object code, generated documentation, 714 | and conversions to other media types. 715 | 716 | "Work" shall mean the work of authorship, whether in Source or 717 | Object form, made available under the License, as indicated by a 718 | copyright notice that is included in or attached to the work 719 | (an example is provided in the Appendix below). 720 | 721 | "Derivative Works" shall mean any work, whether in Source or Object 722 | form, that is based on (or derived from) the Work and for which the 723 | editorial revisions, annotations, elaborations, or other modifications 724 | represent, as a whole, an original work of authorship. For the purposes 725 | of this License, Derivative Works shall not include works that remain 726 | separable from, or merely link (or bind by name) to the interfaces of, 727 | the Work and Derivative Works thereof. 728 | 729 | "Contribution" shall mean any work of authorship, including 730 | the original version of the Work and any modifications or additions 731 | to that Work or Derivative Works thereof, that is intentionally 732 | submitted to Licensor for inclusion in the Work by the copyright owner 733 | or by an individual or Legal Entity authorized to submit on behalf of 734 | the copyright owner. For the purposes of this definition, "submitted" 735 | means any form of electronic, verbal, or written communication sent 736 | to the Licensor or its representatives, including but not limited to 737 | communication on electronic mailing lists, source code control systems, 738 | and issue tracking systems that are managed by, or on behalf of, the 739 | Licensor for the purpose of discussing and improving the Work, but 740 | excluding communication that is conspicuously marked or otherwise 741 | designated in writing by the copyright owner as "Not a Contribution." 742 | 743 | "Contributor" shall mean Licensor and any individual or Legal Entity 744 | on behalf of whom a Contribution has been received by Licensor and 745 | subsequently incorporated within the Work. 746 | 747 | 2. Grant of Copyright License. Subject to the terms and conditions of 748 | this License, each Contributor hereby grants to You a perpetual, 749 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 750 | copyright license to reproduce, prepare Derivative Works of, 751 | publicly display, publicly perform, sublicense, and distribute the 752 | Work and such Derivative Works in Source or Object form. 753 | 754 | 3. Grant of Patent License. Subject to the terms and conditions of 755 | this License, each Contributor hereby grants to You a perpetual, 756 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 757 | (except as stated in this section) patent license to make, have made, 758 | use, offer to sell, sell, import, and otherwise transfer the Work, 759 | where such license applies only to those patent claims licensable 760 | by such Contributor that are necessarily infringed by their 761 | Contribution(s) alone or by combination of their Contribution(s) 762 | with the Work to which such Contribution(s) was submitted. If You 763 | institute patent litigation against any entity (including a 764 | cross-claim or counterclaim in a lawsuit) alleging that the Work 765 | or a Contribution incorporated within the Work constitutes direct 766 | or contributory patent infringement, then any patent licenses 767 | granted to You under this License for that Work shall terminate 768 | as of the date such litigation is filed. 769 | 770 | 4. Redistribution. You may reproduce and distribute copies of the 771 | Work or Derivative Works thereof in any medium, with or without 772 | modifications, and in Source or Object form, provided that You 773 | meet the following conditions: 774 | 775 | (a) You must give any other recipients of the Work or 776 | Derivative Works a copy of this License; and 777 | 778 | (b) You must cause any modified files to carry prominent notices 779 | stating that You changed the files; and 780 | 781 | (c) You must retain, in the Source form of any Derivative Works 782 | that You distribute, all copyright, patent, trademark, and 783 | attribution notices from the Source form of the Work, 784 | excluding those notices that do not pertain to any part of 785 | the Derivative Works; and 786 | 787 | (d) If the Work includes a "NOTICE" text file as part of its 788 | distribution, then any Derivative Works that You distribute must 789 | include a readable copy of the attribution notices contained 790 | within such NOTICE file, excluding those notices that do not 791 | pertain to any part of the Derivative Works, in at least one 792 | of the following places: within a NOTICE text file distributed 793 | as part of the Derivative Works; within the Source form or 794 | documentation, if provided along with the Derivative Works; or, 795 | within a display generated by the Derivative Works, if and 796 | wherever such third-party notices normally appear. The contents 797 | of the NOTICE file are for informational purposes only and 798 | do not modify the License. You may add Your own attribution 799 | notices within Derivative Works that You distribute, alongside 800 | or as an addendum to the NOTICE text from the Work, provided 801 | that such additional attribution notices cannot be construed 802 | as modifying the License. 803 | 804 | You may add Your own copyright statement to Your modifications and 805 | may provide additional or different license terms and conditions 806 | for use, reproduction, or distribution of Your modifications, or 807 | for any such Derivative Works as a whole, provided Your use, 808 | reproduction, and distribution of the Work otherwise complies with 809 | the conditions stated in this License. 810 | 811 | 5. Submission of Contributions. Unless You explicitly state otherwise, 812 | any Contribution intentionally submitted for inclusion in the Work 813 | by You to the Licensor shall be under the terms and conditions of 814 | this License, without any additional terms or conditions. 815 | Notwithstanding the above, nothing herein shall supersede or modify 816 | the terms of any separate license agreement you may have executed 817 | with Licensor regarding such Contributions. 818 | 819 | 6. Trademarks. This License does not grant permission to use the trade 820 | names, trademarks, service marks, or product names of the Licensor, 821 | except as required for reasonable and customary use in describing the 822 | origin of the Work and reproducing the content of the NOTICE file. 823 | 824 | 7. Disclaimer of Warranty. Unless required by applicable law or 825 | agreed to in writing, Licensor provides the Work (and each 826 | Contributor provides its Contributions) on an "AS IS" BASIS, 827 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 828 | implied, including, without limitation, any warranties or conditions 829 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 830 | PARTICULAR PURPOSE. You are solely responsible for determining the 831 | appropriateness of using or redistributing the Work and assume any 832 | risks associated with Your exercise of permissions under this License. 833 | 834 | 8. Limitation of Liability. In no event and under no legal theory, 835 | whether in tort (including negligence), contract, or otherwise, 836 | unless required by applicable law (such as deliberate and grossly 837 | negligent acts) or agreed to in writing, shall any Contributor be 838 | liable to You for damages, including any direct, indirect, special, 839 | incidental, or consequential damages of any character arising as a 840 | result of this License or out of the use or inability to use the 841 | Work (including but not limited to damages for loss of goodwill, 842 | work stoppage, computer failure or malfunction, or any and all 843 | other commercial damages or losses), even if such Contributor 844 | has been advised of the possibility of such damages. 845 | 846 | 9. Accepting Warranty or Additional Liability. While redistributing 847 | the Work or Derivative Works thereof, You may choose to offer, 848 | and charge a fee for, acceptance of support, warranty, indemnity, 849 | or other liability obligations and/or rights consistent with this 850 | License. However, in accepting such obligations, You may act only 851 | on Your own behalf and on Your sole responsibility, not on behalf 852 | of any other Contributor, and only if You agree to indemnify, 853 | defend, and hold each Contributor harmless for any liability 854 | incurred by, or claims asserted against, such Contributor by reason 855 | of your accepting any such warranty or additional liability. 856 | 857 | 858 | 859 | 860 | 861 | 862 | 863 | 864 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project and 7 | our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies within all project spaces, and it also applies when 49 | an individual is representing the project or its community in public spaces. 50 | Examples of representing a project or community include using an official 51 | project e-mail address, posting via an official social media account, or acting 52 | as an appointed representative at an online or offline event. Representation of 53 | a project may be further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the open source team at [opensource-conduct@group.apple.com](mailto:opensource-conduct@group.apple.com). All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ## Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org), version 1.4, 71 | available at [https://www.contributor-covenant.org/version/1/4/code-of-conduct.html](https://www.contributor-covenant.org/version/1/4/code-of-conduct.html) -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contribution Guide 2 | 3 | Thanks for your interest in contributing. This project was released to accompany a research paper for purposes of reproducability, and beyond its publication there are limited plans for future development of the repository. 4 | 5 | While we welcome new pull requests and issues please note that our response may be limited. Forks and out-of-tree improvements are strongly encouraged. 6 | 7 | ## Before you get started 8 | 9 | By submitting a pull request, you represent that you have the right to license your contribution to Apple and the community, and agree by submitting the patch that your contributions are licensed under the [LICENSE](LICENSE). 10 | 11 | We ask that all community members read and observe our [Code of Conduct](CODE_OF_CONDUCT.md). -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2023, Apple Inc. 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /PrivUnitAlgs.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | import numpy as np 4 | from scipy import stats as st 5 | import math 6 | from utilities import * 7 | from SQKR import SQKR, PrivHS 8 | import random 9 | 10 | # Privatizes a vector x according to given mechanism 11 | # x: input vector 12 | # eps: privacy parameter, if eps=-1 no privacy 13 | # k: projection parameter for low-communication algorithms 14 | # mech: in [PrivUnitG,ProjUnit,FastProjUnit,SQKR, PrivHS, RePrivHS,CompPrivUnitG] 15 | # p,gamma,sigma: parameters for privG 16 | # fast: if True, use FWHT implementation of Hadamard transform which works 17 | # faster for the MNIST experiment 18 | # W: transform to be used for corr-transform algorithms such as FastProjUnit-corr 19 | # and ProjUnit-corr. 20 | # For FastProjUnit-corr, one should use 21 | # W = np.random.choice(a=[-1, 1], size=(n), p=[0.5, 0.5]) 22 | # For ProjUnit-corr, one should use 23 | # S = np.sort(random.sample(range(0, n), min(k,n))) #list of indices (different for different users) 24 | # W = W_full[S,:] 25 | # 26 | def privatize_vector(x,eps,k,mech,p=None,gamma=None,sigma=None,fast=False,W=None): 27 | n = len(x) 28 | if mech in ['PrivUnitG', 'ProjUnit', 'FastProjUnit']: 29 | if p is None or gamma is None or sigma is None: 30 | # parameters for PrivG algorthms 31 | p = priv_unit_G_get_p(eps) 32 | gamma, sigma = get_gamma_sigma(p, eps) 33 | if mech == 'PrivUnit': 34 | return PrivUnit(x,eps) 35 | elif mech == 'PrivUnitG': 36 | return PrivUnitG(x,eps,p,gamma,sigma) 37 | elif mech == 'CompPrivUnitG': 38 | return CompPrivUnitG(x, eps, p, gamma, sigma) 39 | return priv_G_compressed(x, eps, p, gamma, sigma) 40 | elif mech == 'FastProjUnit': 41 | return FastProjUnit(x,eps,k,p,gamma,sigma,fast) 42 | elif mech == 'FastProjUnit-corr': 43 | if W is None: 44 | assert("Base transform not provided") 45 | return FastProjUnit(x,eps,k,p,gamma,sigma,fast,W) 46 | elif mech == 'ProjUnit': 47 | return ProjUnit(x,eps,k,p,gamma,sigma) 48 | elif mech == 'ProjUnit-corr': 49 | if W is None: 50 | assert("corr transform not provided") 51 | return ProjUnit(x,eps,k,p,gamma,sigma,W) 52 | elif mech == 'RePrivHS': 53 | return PrivHS(x.reshape(1,len(x)),eps,k).squeeze() 54 | elif mech == 'PrivHS': 55 | k = 1 56 | return PrivHS(x.reshape(1,len(x)),eps,k).squeeze() 57 | elif mech == 'SQKR': 58 | return SQKR(x.reshape(1,len(x)),eps,k,True).squeeze() 59 | 60 | 61 | 62 | 63 | # Applies the ProjUnit algorithm based on Gaussian projections 64 | # for input vector x 65 | # x: input vecotr 66 | # eps: privacy parameter, if eps=-1 no privacy 67 | # p,gamma,sigma: parameters for privG 68 | # R_p: predefined projection matrix. If None, sample fresh rotation matrix 69 | def ProjUnit(x,eps,k,p=None,gamma=None,sigma=None,R_p=None): 70 | if p is None or gamma is None or sigma is None: 71 | p = priv_unit_G_get_p(eps) 72 | gamma, sigma = get_gamma_sigma(p, eps) 73 | n = len(x) 74 | 75 | R = None 76 | if R_p is None: 77 | vectors = np.random.rand(k, n) 78 | q, _ = np.linalg.qr(vectors.T) 79 | R = math.sqrt(1.0*n/k) * q.T 80 | else: 81 | R = R_p 82 | clipped_Rx = normalize (R @ x) 83 | noisy_Rx = None 84 | if eps == -1: 85 | noisy_Rx = clipped_Rx 86 | else: 87 | noisy_Rx = PrivUnitG(clipped_Rx, eps, p, gamma, sigma) 88 | 89 | z = np.transpose(R) @ noisy_Rx 90 | return z 91 | 92 | 93 | 94 | # Applies the FastProjUnit algorithm based on the SRHT transform 95 | # for an input vector x 96 | # x: input vector with unit norn 97 | # eps: privacy parameter, if eps=-1 no privacy 98 | # k: projection to k-dimensional sub-space 99 | # p,gamma,sigma: parameters for privG 100 | # D: predefined D for the HD transform. If none, sample new D 101 | def FastProjUnit(x,eps,k,p=None,gamma=None,sigma=None,fast=False,D=None): 102 | n = len(x) 103 | if p is None or gamma is None or sigma is None: 104 | p = priv_unit_G_get_p(eps) 105 | gamma, sigma = get_gamma_sigma(p, eps) 106 | 107 | D1 = None 108 | if D is None: 109 | D1 = np.random.choice(a=[-1, 1], size=(n), p=[0.5, 0.5]) 110 | else: 111 | D1 = D 112 | #S = np.sort(random.choices(range(n), k=k)) # with repitition 113 | S = random.sample(range(n), k) # without repitition 114 | clipped_z = normalize(applyHDs(x, S, [D1],fast)) 115 | noisy_z = None 116 | if eps == -1: 117 | noisy_z = clipped_z 118 | else: 119 | noisy_z = PrivUnitG(clipped_z, eps, p, gamma, sigma) 120 | z = applyInverseHDs(noisy_z, S, [D1],fast) 121 | return z 122 | 123 | 124 | 125 | # Applies the PrivUnitG randomizer over input vector x 126 | # x: input vector with unit norm 127 | # eps: privacy parameter, if eps=-1 no privacy 128 | # p,gamma,sigma: parameters for privG 129 | # n_trials: number of trials for sampling for the tail of gaussian 130 | def PrivUnitG(x, eps, p=None, gamma=None, sigma=None, n_tries=None): 131 | if not p: 132 | p = priv_unit_G_get_p(eps) 133 | if p is None or gamma is None or sigma is None: 134 | gamma, sigma = get_gamma_sigma(p, eps) 135 | dim = x.size 136 | g = np.random.normal(0, 1, size = dim) 137 | pos_cor = np.random.binomial(1, p) 138 | 139 | if pos_cor: 140 | chosen_dps = np.array([sample_from_G_tail_stable(gamma)]) 141 | else: 142 | if n_tries is None: 143 | n_tries = 25 # here probability of success is 1/2 144 | dps = np.random.normal(0, 1, size=n_tries) 145 | chosen_dps = dps[dps gamma: 193 | # In this case, we select with probability sp_high 194 | if np.random.binomial(1, sp_high) == 1: 195 | break 196 | else: 197 | # In this case, we select with probability 1-sp_low 198 | if np.random.binomial(1, sp_low) == 1: 199 | break 200 | 201 | # In simulation, we send sigma*g. In a compressed version, we send the seed 202 | # and the server regenerates (this same) g from the seed and uses sigma * g 203 | return sigma * g 204 | 205 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Fast and Optimal Algorithms for Locally Private Mean Estimation 2 | 3 | This software project accompanies the research paper, [Faster Optimal Locally Private Mean Estimation 4 | via Random Projections](https://arxiv.org/abs/2306.04444). 5 | 6 | We provide implementations for the algorithms in the above paper, along with code to reproduce all experiments. 7 | 8 | 9 | ## Documentation 10 | 11 | The main LDP mean estimation algorithms are implemented in the file PrivUnitAlgs.py. Code for reproducing the experiments can be found in experiments.py and experiments_MNIST.py for the general and MNIST experiments, respectively. 12 | 13 | 14 | 15 | ## Running Experiments 16 | 17 | Download the repository and install all required packages as listed in requirements.txt. 18 | 19 | To run the general experiment (synthetic data), simply run the file experiments.py: 20 | 21 | ``` 22 | python experiments.py 23 | ``` 24 | 25 | For the MNIST experiments, first run the script train_MNIST_script.sh to produce the results of private training, then run experiments_MNIST.py to produce the plots: 26 | 27 | ``` 28 | bash train_MNIST_script.sh 29 | python experiments_MNIST.py 30 | ``` 31 | -------------------------------------------------------------------------------- /SQKR.py: -------------------------------------------------------------------------------- 1 | # Most of the code from https://github.com/WeiNingChen/Kashin-mean-estimation/blob/master/privUnit_DJW_SQKR.ipynb 2 | 3 | import math 4 | import numpy as np 5 | import random 6 | import matplotlib.pyplot as plt 7 | from scipy.linalg import hadamard 8 | import scipy.io as io 9 | import scipy.special as sc 10 | from scipy.stats import ortho_group 11 | import os 12 | from utilities import * 13 | 14 | 15 | 16 | 17 | # PrivHS randomizer: applies PrivHS to each row in X 18 | def PrivHS(X,eps,k): 19 | X_t = X.T 20 | Y_t = Rep_DJW(X_t,eps,k) 21 | return Y_t.T 22 | 23 | # SQKR randomizer: applies SQKR to each row in X 24 | def SQKR(X,eps,k,FWHT=False): 25 | X_t = X.T 26 | Y_t = Kashin_full(X_t,eps,k,FWHT) 27 | return Y_t.T 28 | 29 | # Applies DJW to the columns of the input matrix X 30 | # Communication is 1-bit 31 | def DJW(X, eps): 32 | 33 | (d, n) = X.shape 34 | # Using {d \choose d/2} approximation 35 | B = (math.exp(eps)+1)/(math.exp(eps)-1)*np.sqrt(math.pi*d/2) 36 | pi_a = math.exp(eps)/(1+math.exp(eps)) 37 | X_perturb = X.copy() 38 | 39 | for i in range(n): 40 | # only handle when X[:, i] is a unit vector 41 | v = np.random.normal(0, 1, size = d) 42 | v = v/np.linalg.norm(v, 2) # v uniform over l_2 unit ball 43 | if np.sum(v * X[:, i]) < 0: 44 | v = -v 45 | T = 2*np.random.binomial(1, pi_a)-1 46 | X_perturb[:, i] = T*v 47 | 48 | return B*X_perturb 49 | 50 | 51 | # Applies DJW k times with eps_i = eps/k 52 | # Communicates k bits 53 | def Rep_DJW(X,eps,k): 54 | X_hat = np.zeros(X.shape) 55 | for i in range(k): 56 | X_perturb = DJW(X,1.0*eps/k) 57 | X_hat = X_hat + 1.0*X_perturb/k 58 | return X_hat 59 | 60 | 61 | 62 | 63 | # Applies DJW k times with eps_i = eps/k 64 | # Communicates k bits 65 | def Rep_DJW_single(x,eps,k): 66 | x_hat = np.zeros(len(x)) 67 | for i in range(k): 68 | x_perturb = DJW_single(x,1.0*eps/k) 69 | x_hat = x_hat + 1.0*x_perturb/k 70 | return x_hat 71 | 72 | 73 | def rand_quantize(a, a_bdd): 74 | return (np.random.binomial(1, (np.clip(a, -a_bdd, a_bdd)+a_bdd)/(2*a_bdd))-1/2)*2*a_bdd 75 | 76 | def rand_sampling(q, k): 77 | # each column of q represents a quantized observation with Kashin representation 78 | # output k sampling matrices and an aggregation of q*sampling_mat 79 | (N, n) = q.shape 80 | sampling_mat_sum = np.zeros((n, N)) 81 | sampling_mat_list = [] 82 | for i in range(k): 83 | spl = np.eye(N)[np.random.choice(N, n)] 84 | sampling_mat_sum = sampling_mat_sum + spl 85 | sampling_mat_list.append(spl.T) 86 | 87 | return [sampling_mat_list, sampling_mat_sum.T, q * sampling_mat_sum.T/k] 88 | 89 | def kRR(k, eps, q_sampling, sampling_mat_list, a_bdd): 90 | # perturb each column of q, as a k-bit string, via k-RR mechanism 91 | q_perturb = q_sampling.copy() 92 | (N, n) = q_sampling.shape 93 | for j in range(n): 94 | if (np.random.uniform(0,1) > (math.exp(eps)-1)/(math.exp(eps)+2**k-1)): 95 | noise = np.zeros(N) 96 | for i in range(k): 97 | # create a random {-1, +1}^N vector and filter it by sampling matrices 98 | noise = noise + (2*np.random.binomial(1, 1/2*np.ones(N))-1)*sampling_mat_list[i][:, j].reshape(-1,)/k 99 | 100 | q_perturb[:, j] = noise*a_bdd 101 | return q_perturb 102 | 103 | def estimate(k, eps, q_perturb): 104 | return (math.exp(eps)+2**k-1)/(math.exp(eps)-1)*q_perturb 105 | 106 | def kRR_string(d, num, eps): 107 | if (np.random.uniform(0,1) < math.exp(eps)/(math.exp(eps)+d-1)): 108 | return num 109 | else: 110 | return np.radom.choice(d) 111 | 112 | # Y is nxm. add more zeros rows to make it n_big x m 113 | def complete_with_zeros(y,n_big): 114 | (n,m) = y.shape 115 | if n == n_big: 116 | return y 117 | z = np.zeros((n_big,m)) 118 | z[:len(y),:] = y 119 | return z 120 | 121 | def Kashin_representation(x, U, FWHT=False, eta = 0.4, delta = 0.8): 122 | # compute kashin representation of x with respect to the frame U at level K 123 | d = x.shape[0] 124 | N = 2**int(math.ceil(math.log(d, 2))+1) 125 | a = np.zeros((N, 1)) 126 | 127 | K = 1/((1-eta)*np.sqrt(delta)) # Set Kashin level to be K 128 | M = eta/np.sqrt(delta*N) 129 | 130 | y = x 131 | itr = int(np.log(N)) 132 | for i in range(itr): 133 | if FWHT: 134 | b = applyHDs(complete_with_zeros(y,len(U)).squeeze(),None,[U]) 135 | else: 136 | b = U @ y 137 | b_hat = np.clip(b, -M, M) 138 | if FWHT: 139 | y = y - applyInverseHDs(b_hat,None,[U])[:d].reshape(y.shape) 140 | else: 141 | y = y - U.T @ b_hat 142 | a = a + b_hat.reshape(a.shape) 143 | M = eta*M 144 | if FWHT: 145 | b = applyHDs(complete_with_zeros(y,len(U)).squeeze(),None,[U]) 146 | Ty = applyInverseHDs(b,None,[U])[:d] 147 | else: 148 | b = U @ y 149 | Ty = U.T @ b 150 | y = y - Ty 151 | a = a + b.reshape(a.shape) 152 | return [a, K/np.sqrt(N)] 153 | 154 | 155 | 156 | 157 | 158 | def Kashin_encode(U, X, k, eps,FWHT=False): 159 | [a, a_bdd] = Kashin_representation(X, U,FWHT) 160 | q = rand_quantize(a, a_bdd) 161 | [sampling_mat_list, sampling_mat_sum, q_sampling] = rand_sampling(q, k) 162 | q_perturb = kRR(k, eps, q_sampling, sampling_mat_list, a_bdd) 163 | return [q, q_sampling, q_perturb] 164 | 165 | def Kashin_decode(U, k, eps, q_perturb,FWHT=False): 166 | if FWHT: 167 | N = len(U) 168 | else: 169 | (N, d) = U.shape 170 | q_unbiased = estimate(k, eps, q_perturb) 171 | if FWHT: 172 | return applyInverseHDs(complete_with_zeros(q_unbiased*N ,len(U)),None,[U]) 173 | else: 174 | return U.T @ (q_unbiased*N) 175 | 176 | 177 | 178 | def Kashin_full(X,eps,k,FWHT=False): 179 | d = X.shape[0] 180 | N = 2**int(math.ceil(math.log(d, 2))+1) 181 | if FWHT: 182 | U = np.random.choice(a=[-1, 1], size=(N), p=[0.5, 0.5]) # fast hadamard 183 | [q_quantize, q_sampling, q_perturb] = Kashin_encode(U, X, k, eps,True) #for Fast Hadamard 184 | else: 185 | U = ortho_group.rvs(dim=N).T[:, 0:d] 186 | [q_quantize, q_sampling, q_perturb] = Kashin_encode(U, X, k, eps) 187 | X_hat = Kashin_decode(U, k, eps, q_perturb,FWHT) 188 | if FWHT: 189 | return X_hat[:d] 190 | return X_hat 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | -------------------------------------------------------------------------------- /dict_perm.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/apple/ml-projunit/166f449f94f5aeb5200315af78daea11dc1e616b/dict_perm.npy -------------------------------------------------------------------------------- /experiments.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | 5 | import matplotlib 6 | import matplotlib.pyplot as plt 7 | import numpy as np 8 | from utilities import * 9 | from PrivUnitAlgs import * 10 | import time 11 | from SQKR import * 12 | import os 13 | import time 14 | import signal 15 | 16 | # A function that takes as input a mechanism mech and 17 | # privacy parameter epsilon, and returns the "best" k 18 | # for this setting. Best here is based on our simulations for 19 | # each algorithm with different values of epsilon and k 20 | def choose_best_k(mech,eps): 21 | k = int(eps) 22 | if mech in ['ProjUnit', 'FastProjUnit']: 23 | k = 1000 24 | elif mech == 'SQKR': 25 | k = int(eps) 26 | elif mech == 'PrivHS': 27 | k = 1 28 | elif mech == 'RePrivHS': 29 | k = max(int(eps/2),1) 30 | return k 31 | 32 | 33 | # returns (color,mark) for mechnism 34 | def color_plot(mech): 35 | if mech == 'PrivUnitG': 36 | return ('k','o') 37 | if mech == 'CompPrivUnitG': 38 | return ('y', 'o') 39 | elif mech == 'ProjUnit': 40 | return ('r','>') 41 | elif mech == 'ProjUnit-corr': 42 | return ('c','o') 43 | elif mech == 'FastProjUnit': 44 | return ('b','<') 45 | elif mech == 'FastProjUnit-corr': 46 | return ('g','x') 47 | elif mech == 'RePrivHS': 48 | return ('g','o') 49 | elif mech == 'PrivHS': 50 | return ('c','o') 51 | elif mech == 'SQKR': 52 | return ('m','o') 53 | return ('k','0') 54 | 55 | 56 | 57 | 58 | # Takes as input a vector x and epsilon, and calculates the error 59 | # of the mechanism for num_rep trials and returns an array with the MSEs. 60 | # The calculation is done as follows: we run the mechanism mech 61 | # over input x for num_rep times, recording the MSE of each trial 62 | # and returning the mean of these MSEs. 63 | # x: input vector 64 | # eps: privacy parameter 65 | # mech: a mechanism to run 66 | # num_rep: num of repetitions to run the mechanism 67 | def find_err(x,eps,k,mech,num_rep): 68 | y = np.zeros(len(x)) 69 | p = None, 70 | gamma = None 71 | sigma = None 72 | if mech in ['PrivUnitG', 'ProjUnit', 'FastProjUnit', 'ProjUnit-corr','FastProjUnit-corr']: 73 | # parameters for PrivG algorthms 74 | p = priv_unit_G_get_p(eps) 75 | gamma, sigma = get_gamma_sigma(p, eps) 76 | err_mech_arr = np.zeros(num_rep) 77 | for i in range(num_rep): 78 | y = privatize_vector(x,eps,k,mech,p,gamma,sigma) 79 | err_mech_arr[i] = np.sum((y-x)**2) 80 | return err_mech_arr 81 | 82 | 83 | 84 | 85 | 86 | # Plots the variance as a function of epsilon for all methods 87 | # For each method, k is chosen using the function choose_best_k. 88 | def exp_compare_var_eps(num_rep = 100): 89 | mechanisms = [ 'FastProjUnit', 'ProjUnit', 'PrivUnitG', 'PrivHS','RePrivHS','SQKR'] 90 | n = 2**15 # dimension 91 | lst_eps = list(range(16))[1:16:2] 92 | 93 | err_dict = {} 94 | dict_file = 'raw/err_comp_eps_dim_%d_num_rep_%d.npy' % (n,num_rep) 95 | if os.path.exists(dict_file): 96 | err_dict = np.load(dict_file,allow_pickle='TRUE').item() 97 | x = np.random.normal(size=n) 98 | x = x / np.linalg.norm(x) 99 | for mech in mechanisms: 100 | print(mech) 101 | if mech in err_dict: 102 | continue 103 | err_dict[mech] = np.zeros((len(lst_eps),num_rep)) 104 | for i in range(len(lst_eps)): 105 | eps = lst_eps[i] 106 | k = choose_best_k(mech,eps) 107 | err_dict[mech][i] = find_err(x,eps,k,mech,num_rep) 108 | 109 | np.save(dict_file, err_dict) 110 | 111 | # plot the errors 112 | plt.figure() 113 | for mech in mechanisms: 114 | (c,m) = color_plot(mech) 115 | q = 0.9 116 | err_mech = [np.mean(err_dict[mech][i]) for i in range(len(lst_eps))] 117 | err_mech_high = [np.quantile(err_dict[mech][i],q) for i in range(len(lst_eps))] 118 | err_mech_low = [np.quantile(err_dict[mech][i],1-q) for i in range(len(lst_eps))] 119 | (c,m) = color_plot(mech) 120 | plt.plot(lst_eps,err_mech,color = c, marker=m,linestyle='dashed',label = mech) 121 | plt.fill_between(lst_eps, err_mech_low, err_mech_high, color=c, alpha=.1) 122 | 123 | 124 | plt.legend() 125 | plt.yscale("log") 126 | f_name = 'plots/err-comparison_lst_eps_d_%d.pdf' % n 127 | plt.savefig(f_name) 128 | plt.show() 129 | 130 | 131 | # Plot the variance as a function of k for all methods 132 | # Here we use the same HD matrices for the ProjUnit algorithms 133 | # and only sample different S_i matrix for each user 134 | # num_users: number of users 135 | # num_rep: number of repetitions 136 | def exp_compare_algs_k(eps,num_users=20,num_rep=100): 137 | mechanisms = ['PrivUnitG', 'ProjUnit', 'FastProjUnit', 'FastProjUnit-corr'] 138 | n = 2**13 # dimension of input vectors 139 | lst_k = [10, 50, 100, 500, 800, 1000, 1500, 2000] 140 | f_name = 'plots/err-fixHD_num_users_%d_num_rep_%d_d_%d_eps_%d.pdf' % (num_users,num_rep,n,int(eps)) 141 | dict_file = 'raw/err-fixHD_lst_k_num_users_%d_num_rep_%d_d_%d_eps_%d.npy' % (num_users,num_rep,n,int(eps)) 142 | err_dict = {} 143 | if os.path.exists(dict_file): 144 | err_dict = np.load(dict_file,allow_pickle='TRUE').item() 145 | 146 | p = priv_unit_G_get_p(eps) 147 | gamma, sigma = get_gamma_sigma(p, eps) 148 | 149 | v = np.random.normal(size=n) 150 | v = v/math.sqrt(np.sum(v**2)) 151 | X = np.random.normal(size=(num_users,n))/math.sqrt(n) + v 152 | norms = np.linalg.norm(X, axis=1) 153 | X = X / norms[:, np.newaxis] 154 | norms = np.linalg.norm(X, axis=1) 155 | v_hat = np.sum(X,0)/num_users 156 | for mech in mechanisms: 157 | print(mech) 158 | if mech in err_dict: 159 | continue 160 | err_dict[mech] = {} 161 | for i in range(len(lst_k)): 162 | k = lst_k[i] 163 | err_dict[mech][k] = np.zeros(num_rep) 164 | if mech == 'PrivUnitG' and i>0: 165 | err_dict[mech][k] = err_dict[mech][lst_k[0]] 166 | continue 167 | err = np.zeros(num_rep) 168 | W = None 169 | for j in range(num_rep): 170 | x_hat = np.zeros(n) 171 | if mech == 'FastProjUnit-corr': 172 | W = np.random.choice(a=[-1, 1], size=(n), p=[0.5, 0.5]) 173 | elif mech == 'ProjUnit-corr': 174 | W_full = special_ortho_group.rvs(n) 175 | for u in range(num_users): 176 | if mech == 'ProjUnit-corr': 177 | S = np.sort(random.choices(range(n), k=k)) # without repitition 178 | #S = random.sample(range(n), k) # with repitition 179 | W = math.sqrt(1.0*n/k) * W_full[S,:] 180 | x_hat += privatize_vector(X[u],eps,k,mech,p,gamma,sigma,False,W)/num_users 181 | err[j] = np.sum((x_hat - v_hat)**2) 182 | err_dict[mech][k] = err 183 | 184 | np.save(dict_file, err_dict) 185 | 186 | 187 | # plot the errors 188 | plt.figure() 189 | for mech in mechanisms: 190 | q = 0.9 191 | err_mech = [np.mean(err_dict[mech][k]) for k in lst_k] 192 | err_mech_high = [np.quantile(err_dict[mech][k],q) for k in lst_k] 193 | err_mech_low = [np.quantile(err_dict[mech][k],1-q) for k in lst_k] 194 | (c,m) = color_plot(mech) 195 | plt.plot(lst_k[1:],err_mech[1:],color = c, marker=m,linestyle='dashed',label = mech) 196 | plt.fill_between(lst_k[1:], err_mech_low[1:], err_mech_high[1:], color=c, alpha=.1) 197 | 198 | #plt.xlabel('k') 199 | plt.legend() 200 | plt.yscale("log") 201 | plt.xscale("log") 202 | plt.savefig(f_name) 203 | plt.show() 204 | 205 | 206 | 207 | def handle_timeout(signum, frame): 208 | raise TimeoutError 209 | 210 | 211 | 212 | 213 | # Plot the run-time of each method as a function of the dimension 214 | # when k is chosen as the best k for each method (using choose_best_k). 215 | # For each dimension, we give a cut-off of 1 hour for each method 216 | # to finish (num_rep=10) trials 217 | def experiment_compare_time(eps,num_rep = 10): 218 | # Run the function to create the plots 219 | mechanisms = ['PrivUnitG', 'ProjUnit', 'FastProjUnit', 'PrivHS', 'SQKR', 'CompPrivUnitG'] 220 | 221 | file_dict = 'raw/time_dict_eps_%d.npy' % int(eps) 222 | time_dict = {} 223 | if os.path.exists(file_dict): 224 | time_dict = np.load(file_dict,allow_pickle='TRUE').item() 225 | 226 | 227 | lst_n = [10, 10**2, 10**3, 10**4, 10**5, 10**6, 10**7] 228 | 229 | time_arr = {} 230 | 231 | p = None 232 | gamma = None 233 | sigma = None 234 | time_limit = 60*60 # set time limit to 60 minutes 235 | for mech in mechanisms: 236 | print(mech) 237 | time_arr[mech] = np.zeros(len(lst_n)) 238 | if mech in ['PrivUnitG', 'FastProjUnit', 'ProjUnit', 'CompPrivUnitG']: 239 | p = priv_unit_G_get_p(eps) 240 | gamma, sigma = get_gamma_sigma(p, eps) 241 | for i in range(len(lst_n)): 242 | n = lst_n[i] 243 | if (mech,n) in time_dict: 244 | time_arr[mech][i] = time_dict[mech,n] 245 | continue 246 | x = np.random.normal(size=n) 247 | x = x / np.linalg.norm(x) 248 | k = choose_best_k(mech,eps) 249 | 250 | start_time = time.process_time() 251 | signal.signal(signal.SIGALRM, handle_timeout) 252 | if i > 0 and time_arr[mech][i-1] == time_limit: 253 | time_arr[mech][i] = time_limit 254 | time_dict[mech,n] = time_limit 255 | continue 256 | signal.alarm(time_limit) # give limited time to each application 257 | try: 258 | for j in range(num_rep): 259 | y = privatize_vector(x,eps,k,mech,p,gamma,sigma) 260 | time_arr[mech][i] = time.process_time() - start_time 261 | except TimeoutError: 262 | print("------Too long-------") 263 | time_arr[mech][i] = time_limit 264 | 265 | time_dict[mech,n] = time_arr[mech][i] 266 | 267 | 268 | np.save(file_dict, time_dict) 269 | 270 | 271 | # plot the results 272 | plt.figure() 273 | for mech in mechanisms: 274 | if time_arr[mech][-1] == time_limit: 275 | idx = np.where(time_arr[mech] == time_limit)[0][0] 276 | else: 277 | idx = -1 278 | if mech == 'PrivUnitG': 279 | plt.plot(lst_n[:idx],time_arr[mech][:idx],'k-o',label = mech) 280 | continue 281 | plt.plot(lst_n[:idx],time_arr[mech][:idx],color_plot(mech),label = mech) 282 | plt.legend() 283 | plt.xscale("log") 284 | plt.yscale("log") 285 | f_name = 'plots/time-comparison_eps_%d.pdf' % int(eps) 286 | plt.savefig(f_name) 287 | plt.show() 288 | 289 | 290 | 291 | 292 | 293 | 294 | # ---------- Experiment 1: plot error as a function of k --------- 295 | num_rep = 30 296 | num_users = 50 297 | 298 | print('----experiment 1 (eps = 16)------') 299 | eps = 16.0 300 | exp_compare_algs_k(eps,num_users,num_rep) 301 | 302 | 303 | print('----experiment 1 (eps = 10)------') 304 | eps = 10.0 305 | exp_compare_algs_k(eps,num_users,num_rep) 306 | 307 | 308 | print('----experiment 1 (eps = 4)------') 309 | eps = 4.0 310 | exp_compare_algs_k(eps,num_users,num_rep) 311 | 312 | 313 | 314 | 315 | # ---------- Experiment 2: plot error as a function of epsilon --------- 316 | num_rep = 50 317 | 318 | print('----experiment 2------') 319 | exp_compare_var_eps(num_rep) 320 | 321 | 322 | 323 | 324 | 325 | # ---------- Experiment 3: plot time for each method --------- 326 | num_rep = 10 327 | 328 | print('----experiment 3 (eps = 10)------') 329 | eps = 10 330 | experiment_compare_time(eps,num_rep) 331 | 332 | 333 | print('----experiment 3 (eps = 16)------') 334 | eps = 16 335 | experiment_compare_time(eps,num_rep) 336 | 337 | 338 | 339 | 340 | 341 | 342 | 343 | 344 | 345 | 346 | 347 | 348 | 349 | 350 | -------------------------------------------------------------------------------- /experiments_MNIST.py: -------------------------------------------------------------------------------- 1 | # Code for MNIST experiment 2 | 3 | import numpy as np 4 | import matplotlib.pyplot as plt 5 | import os 6 | import re 7 | 8 | 9 | 10 | 11 | 12 | 13 | def color_plot(mech): 14 | if 'PrivUnitG' in mech: 15 | return 'k' 16 | elif 'CompPrivUnitG' in mech : 17 | return 'y' 18 | elif 'FastProjUnit-corr' in mech : 19 | return 'm' 20 | elif 'FastProjUnit' in mech : 21 | return 'b' 22 | elif 'ProjUnit' in mech: 23 | return 'r' 24 | elif 'RePrivHS' in mech : 25 | return 'g' 26 | elif 'PrivHS' in mech: 27 | return 'c' 28 | elif 'SQKR' in mech: 29 | return 'm' 30 | elif 'Gaussian' in mech: 31 | return 'olive' 32 | elif 'nonPrivate' in mech: 33 | return 'g' 34 | return 'k' 35 | 36 | 37 | # This function assumes that the folder dpsgd_results contains 38 | # the data for the various methods. 39 | # Then we plot the test error as a function of epoch for all methods 40 | #in this folder that has privacy parameter eps and number of epochs 41 | def plot_eps(eps,epochs,dataset="mnist"): 42 | f_dir = 'MNIST_results' 43 | plt.figure() 44 | c_mech = 0 #counter for marker of PrivHS 45 | colors_phs = ['c', 'r', 'm'] #colors for phs 46 | mechanisms = ['Gaussian', 'PrivUnitG', 'FastProjUnit', 'FastProjUnit-corr', 'RePrivHS'] 47 | for mech in mechanisms: 48 | for f_name in os.listdir(f_dir): 49 | f = f_dir + '/' + f_name 50 | if f_name[-3:] != 'npy' or 'num_rep' not in f_name: 51 | continue 52 | f_eps = int(float(re.findall('_.*.',re.findall('epsilon_.*.pth', f_name)[0])[0][1:-4])) 53 | f_epochs = int(re.findall('_.*_',re.findall('epochs_.*_lr', f_name)[0])[0][1:-1]) 54 | f_mech = re.findall('_.*_',re.findall('mech_.*_num', f_name)[0])[0][1:-1] 55 | if mech != f_mech: 56 | continue 57 | f_lr = float(re.findall('_.*_',re.findall('lr_.*_clip', f_name)[0])[0][1:-1]) 58 | if f_eps == eps and f_epochs == epochs: 59 | res = np.load(f) 60 | q = 0.9 61 | err_mean = [np.mean(res[i,:]) for i in range(res.shape[0])] 62 | err_high = [np.quantile(res[i,:],q) for i in range(res.shape[0])] 63 | err_low = [np.quantile(res[i,:],1-q) for i in range(res.shape[0])] 64 | f_k = 0 65 | if f_mech in ['ProjUnit', 'FastProjUnit', 'FastProjUnit-corr','RePrivHS', 'SQKR']: 66 | # find communication k 67 | f_k = f_epochs = int(re.findall('_.*_',re.findall('k_.*_epsilon', f_name)[0])[0][1:-1]) 68 | if f_mech == 'RePrivHS': 69 | f_mech = f_mech + ' (R = %d)' % (f_k) 70 | else: 71 | f_mech = f_mech + ' (k = %d)' % (f_k) 72 | 73 | if 'RePrivHS' in f_mech: 74 | c = colors_phs[c_mech] 75 | c_mech = c_mech + 1 76 | plt.plot(range(epochs),err_mean,color=c, marker='o',label=f_mech) 77 | plt.fill_between(range(epochs), err_low, err_high, color=c, alpha=.1) 78 | else: 79 | plt.plot(range(epochs),err_mean,color=color_plot(f_mech), marker='o',label=f_mech) 80 | plt.fill_between(range(epochs), err_low, err_high, color=color_plot(f_mech), alpha=.1) 81 | 82 | plt.title("%s (epsilon = %d)" % (dataset.upper(),int(eps))) 83 | plt.legend() 84 | plot_name = "plots/%s_eps_%d_epochs_%d.pdf" % (dataset,int(eps),epochs) 85 | plt.savefig(plot_name) 86 | plt.show() 87 | 88 | 89 | 90 | 91 | # Before running this, you have to run the script train_MNIST_script.sh 92 | # to produce the results for all methods. 93 | # This might take some time. You can also run other setting 94 | # by using the following command: 95 | # python train_mnist.py --epsilon 10 --epochs 10 --lr 0.1 --clip-val 1 --mechanism FastProjUnit --k 1000 96 | 97 | 98 | 99 | # Plot the main results 100 | 101 | # Experiment 1: eps=4 and epochs=10 102 | eps = 4 103 | epochs = 10 104 | plot_eps(eps,epochs) 105 | 106 | # Experiment 2: eps=10 and epochs=10 107 | eps = 10 108 | epochs = 10 109 | plot_eps(eps,epochs) 110 | 111 | 112 | # Experiment 3: eps=16 and epochs=10 113 | eps = 16 114 | epochs = 10 115 | plot_eps(eps,epochs) 116 | 117 | 118 | 119 | 120 | 121 | 122 | -------------------------------------------------------------------------------- /gaussian_analytic.py: -------------------------------------------------------------------------------- 1 | # Code from https://github.com/BorjaBalle/analytic-gaussian-mechanism 2 | 3 | from math import exp, sqrt 4 | from scipy.special import erf 5 | 6 | 7 | def calibrateAnalyticGaussianMechanism(epsilon, delta, GS, tol = 1.e-12): 8 | """ Calibrate a Gaussian perturbation for differential privacy using the analytic Gaussian mechanism of [Balle and Wang, ICML'18] 9 | Arguments: 10 | epsilon : target epsilon (epsilon > 0) 11 | delta : target delta (0 < delta < 1) 12 | GS : upper bound on L2 global sensitivity (GS >= 0) 13 | tol : error tolerance for binary search (tol > 0) 14 | Output: 15 | sigma : standard deviation of Gaussian noise needed to achieve (epsilon,delta)-DP under global sensitivity GS 16 | """ 17 | 18 | def Phi(t): 19 | return 0.5*(1.0 + erf(float(t)/sqrt(2.0))) 20 | 21 | def caseA(epsilon,s): 22 | return Phi(sqrt(epsilon*s)) - exp(epsilon)*Phi(-sqrt(epsilon*(s+2.0))) 23 | 24 | def caseB(epsilon,s): 25 | return Phi(-sqrt(epsilon*s)) - exp(epsilon)*Phi(-sqrt(epsilon*(s+2.0))) 26 | 27 | def doubling_trick(predicate_stop, s_inf, s_sup): 28 | while(not predicate_stop(s_sup)): 29 | s_inf = s_sup 30 | s_sup = 2.0*s_inf 31 | return s_inf, s_sup 32 | 33 | def binary_search(predicate_stop, predicate_left, s_inf, s_sup): 34 | s_mid = s_inf + (s_sup-s_inf)/2.0 35 | while(not predicate_stop(s_mid)): 36 | if (predicate_left(s_mid)): 37 | s_sup = s_mid 38 | else: 39 | s_inf = s_mid 40 | s_mid = s_inf + (s_sup-s_inf)/2.0 41 | return s_mid 42 | 43 | delta_thr = caseA(epsilon, 0.0) 44 | 45 | if (delta == delta_thr): 46 | alpha = 1.0 47 | 48 | else: 49 | if (delta > delta_thr): 50 | predicate_stop_DT = lambda s : caseA(epsilon, s) >= delta 51 | function_s_to_delta = lambda s : caseA(epsilon, s) 52 | predicate_left_BS = lambda s : function_s_to_delta(s) > delta 53 | function_s_to_alpha = lambda s : sqrt(1.0 + s/2.0) - sqrt(s/2.0) 54 | 55 | else: 56 | predicate_stop_DT = lambda s : caseB(epsilon, s) <= delta 57 | function_s_to_delta = lambda s : caseB(epsilon, s) 58 | predicate_left_BS = lambda s : function_s_to_delta(s) < delta 59 | function_s_to_alpha = lambda s : sqrt(1.0 + s/2.0) + sqrt(s/2.0) 60 | 61 | predicate_stop_BS = lambda s : abs(function_s_to_delta(s) - delta) <= tol 62 | 63 | s_inf, s_sup = doubling_trick(predicate_stop_DT, 0.0, 1.0) 64 | s_final = binary_search(predicate_stop_BS, predicate_left_BS, s_inf, s_sup) 65 | alpha = function_s_to_alpha(s_final) 66 | 67 | sigma = alpha*GS/sqrt(2.0*epsilon) 68 | 69 | return sigma -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | scipy 2 | numpy 3 | opacus>=1.0 4 | pytorch>=1.7.0 5 | matplotlib 6 | tqdm 7 | time 8 | random 9 | os 10 | signal 11 | re 12 | argparse 13 | torchvision 14 | -------------------------------------------------------------------------------- /setup.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | conda install python=3.7 3 | 4 | python3 get-pip.py 5 | 6 | pip3 install --upgrade pip 7 | pip3 install --upgrade turibolt --index https://pypi.apple.com/simple 8 | pip3 install -U numpy 9 | pip3 install -U scipy 10 | pip3 install -U torch 11 | pip3 install -U torchvision 12 | pip3 install -U boto3 13 | pip3 install --upgrade awscli 14 | pip3 install torchtext 15 | pip3 install spacy 16 | pip3 install sparse 17 | pip3 install opacus 18 | pip3 install kymatio 19 | pip3 install seaborn 20 | 21 | 22 | -------------------------------------------------------------------------------- /train_MNIST_script.sh: -------------------------------------------------------------------------------- 1 | 2 | # ------------- eps = 4 --------------------- 3 | python train_mnist.py --epsilon 4 --epochs 10 --lr 0.1 --clip-val 1 --mechanism Gaussian --k 1 --num-rep 10 4 | 5 | python train_mnist.py --epsilon 4 --epochs 10 --lr 0.1 --clip-val 1 --mechanism PrivUnitG --k 1 --num-rep 10 6 | 7 | python train_mnist.py --epsilon 4 --epochs 10 --lr 0.1 --clip-val 1 --mechanism FastProjUnit --k 1000 --num-rep 10 8 | 9 | python train_mnist.py --epsilon 4 --epochs 10 --lr 0.1 --clip-val 1 --mechanism FastProjUnit-corr --k 1000 --num-rep 10 10 | 11 | python train_mnist.py --epsilon 4 --epochs 10 --lr 0.1 --clip-val 1 --mechanism RePrivHS --k 2 --num-rep 10 12 | 13 | python train_mnist.py --epsilon 4 --epochs 10 --lr 0.1 --clip-val 1 --mechanism RePrivHS --k 1 --num-rep 10 14 | 15 | python train_mnist.py --epsilon 4 --epochs 10 --lr 0.1 --clip-val 1 --mechanism RePrivHS --k 4 --num-rep 10 16 | 17 | 18 | 19 | # ------------- eps = 10 --------------------- 20 | python train_mnist.py --epsilon 10 --epochs 10 --lr 0.1 --clip-val 1 --mechanism Gaussian --k 1 --num-rep 10 21 | 22 | python train_mnist.py --epsilon 10 --epochs 10 --lr 0.1 --clip-val 1 --mechanism PrivUnitG --k 1 --num-rep 10 23 | 24 | python train_mnist.py --epsilon 10 --epochs 10 --lr 0.1 --clip-val 1 --mechanism FastProjUnit --k 1000 --num-rep 10 25 | 26 | python train_mnist.py --epsilon 10 --epochs 10 --lr 0.1 --clip-val 1 --mechanism FastProjUnit-corr --k 1000 --num-rep 10 27 | 28 | python train_mnist.py --epsilon 10 --epochs 10 --lr 0.1 --clip-val 1 --mechanism RePrivHS --k 5 --num-rep 10 29 | 30 | python train_mnist.py --epsilon 10 --epochs 10 --lr 0.1 --clip-val 1 --mechanism RePrivHS --k 10 --num-rep 10 31 | 32 | python train_mnist.py --epsilon 10 --epochs 10 --lr 0.1 --clip-val 1 --mechanism RePrivHS --k 2 --num-rep 10 33 | 34 | 35 | 36 | # ------------- eps = 16 --------------------- 37 | python train_mnist.py --epsilon 16 --epochs 10 --lr 0.1 --clip-val 1 --mechanism Gaussian --k 1 --num-rep 10 38 | 39 | python train_mnist.py --epsilon 16 --epochs 10 --lr 0.1 --clip-val 1 --mechanism PrivUnitG --k 1 --num-rep 10 40 | 41 | python train_mnist.py --epsilon 16 --epochs 10 --lr 0.1 --clip-val 1 --mechanism FastProjUnit --k 1000 --num-rep 10 42 | 43 | python train_mnist.py --epsilon 16 --epochs 10 --lr 0.1 --clip-val 1 --mechanism FastProjUnit-corr --k 1000 --num-rep 10 44 | 45 | python train_mnist.py --epsilon 16 --epochs 10 --lr 0.1 --clip-val 1 --mechanism RePrivHS --k 8 --num-rep 10 46 | 47 | python train_mnist.py --epsilon 16 --epochs 10 --lr 0.1 --clip-val 1 --mechanism RePrivHS --k 4 --num-rep 10 48 | 49 | python train_mnist.py --epsilon 16 --epochs 10 --lr 0.1 --clip-val 1 --mechanism RePrivHS --k 12 --num-rep 10 50 | 51 | 52 | -------------------------------------------------------------------------------- /train_mnist.py: -------------------------------------------------------------------------------- 1 | # Based on Opacus code (https://github.com/pytorch/opacus/blob/main/examples/mnist.py) 2 | 3 | 4 | import argparse 5 | 6 | import numpy as np 7 | import torch 8 | import torch.nn as nn 9 | import torch.nn.functional as F 10 | import torch.optim as optim 11 | from opacus import PrivacyEngine 12 | from opacus.grad_sample.grad_sample_module import GradSampleModule 13 | from torchvision import datasets, transforms 14 | from tqdm import tqdm 15 | 16 | from PrivUnitAlgs import * 17 | from gaussian_analytic import calibrateAnalyticGaussianMechanism 18 | from time import sleep 19 | 20 | 21 | # Precomputed characteristics of the MNIST dataset 22 | MNIST_MEAN = 0.1307 23 | MNIST_STD = 0.3081 24 | 25 | 26 | # Model from https://arxiv.org/abs/2203.08134 27 | class ConvNet(nn.Module): 28 | def __init__(self): 29 | super(ConvNet, self).__init__() 30 | self.conv1 = nn.Conv2d(1, 16, 8, 2, padding=2) 31 | self.conv2 = nn.Conv2d(16, 64, 4, 2, padding=0) 32 | self.fc1 = nn.Linear(64 * 5 * 5, 32) 33 | self.fc2 = nn.Linear(32, 10) 34 | 35 | def forward(self, x): 36 | x = x.view(-1, 1, 28, 28) 37 | x = self.conv1(x) 38 | x = torch.tanh(x) 39 | x = F.avg_pool2d(x, 1) 40 | x = self.conv2(x) 41 | x = torch.tanh(x) 42 | x = F.avg_pool2d(x, 1) 43 | x = torch.flatten(x, 1) 44 | x = self.fc1(x) 45 | x = torch.tanh(x) 46 | x = self.fc2(x) 47 | return x 48 | 49 | 50 | 51 | # Convert gradients into vectors 52 | def grad_to_vec(model): 53 | res = [] 54 | for p in model.parameters(): 55 | res.append(p.grad_sample.view(p.grad_sample.size(0), -1)) 56 | return torch.cat(res, dim=1).squeeze() 57 | 58 | # Clipt L2 norm of gradient to be at most C 59 | def clip(grad, C): 60 | vec_norm = grad.norm(2, 1) 61 | multiplier = vec_norm.new(vec_norm.size()).fill_(1) 62 | multiplier[vec_norm.gt(C)] = C / vec_norm[vec_norm.gt(C)] 63 | grad *= multiplier.unsqueeze(1) 64 | return grad 65 | 66 | # Privatizes the input gradients according to the privacy parameters 67 | # in args 68 | def privatize_grad(args, grad, mechanism=None): 69 | C = args.clip_val 70 | batch_size = grad.size(0) 71 | d = grad.size(1) 72 | grad_vec_np = np.zeros((grad.size(0),d+1)) 73 | grad_vec_np[:,:-1] = grad.detach().cpu().numpy() 74 | # Complete norm to C then normalize 75 | v_m = np.min(C**2 - np.sum(grad_vec_np**2,1)+0.0001) 76 | if v_m > 0: 77 | grad_vec_np[:,-1] = np.sqrt(C**2 - np.sum(grad_vec_np**2,1)+0.0001) 78 | grad_vec_np = grad_vec_np/C 79 | epsilon = args.epsilon 80 | if mechanism == 'Gaussian' or mechanism == 'nonPrivate': 81 | grad_vec_np = None 82 | grad += torch.randn_like(grad).to(args.device) * args.clip_val * args.sigma 83 | else: 84 | W = None 85 | if mechanism == 'FastProjUnit-fixed': 86 | W = np.random.choice(a=[-1, 1], size=(grad_vec_np.shape[1]), p=[0.5, 0.5]) 87 | elif mechanism == 'ProjUnit-fixed': 88 | W = np.random.normal(size=(args.k,grad_vec_np.shape[1]+1)) / k**0.5 89 | for i in range(grad_vec_np.shape[0]): 90 | grad_vec_np[i] = privatize_vector(grad_vec_np[i],epsilon,args.k,mechanism,args.p,args.gamma,args.sigma,True,W) 91 | if grad_vec_np is not None: 92 | grad = torch.from_numpy(grad_vec_np[:,:-1]).to(args.device).float() 93 | return C*grad.mean(0) 94 | 95 | # updates the gradient of the model to be equal to the input gradient 96 | def update_grad(model, grad): 97 | model.zero_grad() 98 | for p in model.parameters(): 99 | size = p.data.view(1,-1).size(1) 100 | p.grad = grad[:size].view_as(p.data).clone() 101 | grad = grad[size:] 102 | return 103 | 104 | def train(args, model, device, train_loader, optimizer, epoch): 105 | model.train() 106 | criterion = nn.CrossEntropyLoss() 107 | losses = [] 108 | epsilon = args.epsilon 109 | for _batch_idx, (data, target) in enumerate(tqdm(train_loader)): 110 | data, target = data.to(device), target.to(device) 111 | optimizer.zero_grad() 112 | output = model(data) 113 | loss = criterion(output, target) 114 | loss.backward() 115 | grad = grad_to_vec(model) 116 | if args.mechanism != 'nonPrivate': 117 | grad = clip(grad, args.clip_val) 118 | grad_priv = privatize_grad(args, grad, args.mechanism) 119 | update_grad(model, grad_priv) 120 | optimizer.step() 121 | losses.append(loss.item()) 122 | 123 | print( 124 | f"Train Epoch: {epoch} \t" 125 | f"Loss: {np.mean(losses):.6f} " 126 | f"(ε = {epsilon:.2f}, δ = {args.delta})" 127 | ) 128 | 129 | 130 | def test(model, device, test_loader): 131 | model.eval() 132 | criterion = nn.CrossEntropyLoss() 133 | test_loss = 0 134 | correct = 0 135 | with torch.no_grad(): 136 | for data, target in tqdm(test_loader): 137 | data, target = data.to(device), target.to(device) 138 | output = model(data) 139 | test_loss += criterion(output, target).item() # sum up batch loss 140 | pred = output.argmax( 141 | dim=1, keepdim=True 142 | ) # get the index of the max log-probability 143 | correct += pred.eq(target.view_as(pred)).sum().item() 144 | 145 | test_loss /= len(test_loader.dataset) 146 | 147 | print( 148 | "\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.2f}%)\n".format( 149 | test_loss, 150 | correct, 151 | len(test_loader.dataset), 152 | 100.0 * correct / len(test_loader.dataset), 153 | ) 154 | ) 155 | return correct / len(test_loader.dataset) 156 | 157 | 158 | def main(): 159 | # Training settings 160 | parser = argparse.ArgumentParser( 161 | description="MNIST ProjUnit Experiments", 162 | formatter_class=argparse.ArgumentDefaultsHelpFormatter, 163 | ) 164 | parser.add_argument( 165 | "-b", 166 | "--batch-size", 167 | type=int, 168 | default=600, 169 | metavar="B", 170 | help="Batch size", 171 | ) 172 | parser.add_argument( 173 | "--test-batch-size", 174 | type=int, 175 | default=1024, 176 | metavar="TB", 177 | help="input batch size for testing", 178 | ) 179 | parser.add_argument( 180 | "-n", 181 | "--epochs", 182 | type=int, 183 | default=10, 184 | metavar="N", 185 | help="number of epochs to train", 186 | ) 187 | parser.add_argument( 188 | "--lr", 189 | type=float, 190 | default=0.1, 191 | metavar="LR", 192 | help="learning rate", 193 | ) 194 | parser.add_argument( 195 | "--momentum", 196 | type=float, 197 | default=0.5, 198 | help="Momentum", 199 | ) 200 | parser.add_argument( 201 | "-c", 202 | "--clip-val", 203 | type=float, 204 | default=1.0, 205 | metavar="C", 206 | help="Clip per-sample gradients to this norm", 207 | ) 208 | parser.add_argument( 209 | "--mechanism", 210 | type=str, 211 | default='Gaussian', 212 | choices=["nonPrivate","Gaussian", "PrivUnitG", "FastProjUnit", 213 | "ProjUnit", "FastProjUnit-fixed", "PrivHS", "RePrivHS", "SQKR"], 214 | help="Privacy mechanism", 215 | ) 216 | parser.add_argument( 217 | "--k", 218 | type=int, 219 | default=1000, 220 | help="number of projected dimensions for ProjUnit algorithms", 221 | ) 222 | parser.add_argument( 223 | "--epsilon", 224 | type=float, 225 | default=1.0, 226 | metavar="S", 227 | help="Privacy parameter", 228 | ) 229 | parser.add_argument( 230 | "--delta", 231 | type=float, 232 | default=1e-5, 233 | metavar="D", 234 | help="Target delta", 235 | ) 236 | parser.add_argument( 237 | "--num-rep", 238 | type=int, 239 | default=1, 240 | help="Number of Repetitions", 241 | ) 242 | parser.add_argument( 243 | "--device", 244 | type=str, 245 | default="cpu", 246 | help="GPU ID for this process", 247 | ) 248 | parser.add_argument( 249 | "--save-model", 250 | action="store_true", 251 | default=False, 252 | help="Save the trained model", 253 | ) 254 | parser.add_argument( 255 | "--data-root", 256 | type=str, 257 | default="../mnist", 258 | help="Where MNIST is/will be stored", 259 | ) 260 | parser.add_argument( 261 | "--dir-res", 262 | type=str, 263 | default="MNIST_results", 264 | help="Directory for saving results", 265 | ) 266 | args = parser.parse_args() 267 | device = torch.device(args.device) 268 | 269 | args.p = None 270 | args.sigma = None 271 | args.gamma = None 272 | if args.mechanism == 'nonPrivate': 273 | args.sigma = 0.0 274 | if args.mechanism == 'Gaussian': 275 | # set sigma according to epsilon 276 | if args.epsilon < 0: 277 | args.sigma = 0.0 278 | else: 279 | args.sigma = calibrateAnalyticGaussianMechanism(args.epsilon, args.delta,args.clip_val) 280 | else: 281 | args.delta = 0 282 | if args.mechanism in ['PrivUnitG', 'FastProjUnit', 'ProjUnit', 'FastProjUnit-fixed']: 283 | # parameters for PrivG algorthms 284 | args.p = priv_unit_G_get_p(args.epsilon) 285 | args.gamma, args.sigma = get_gamma_sigma(args.p, args.epsilon) 286 | 287 | print("Mechanism is " + str(args.mechanism)) 288 | 289 | output_file = "%s/mech_%s_num_rep_%d_epochs_%d_lr_%.2e_clip_%.2e_k_%d_epsilon_%.2e.pth" % ( 290 | args.dir_res, args.mechanism, args.num_rep, args.epochs, args.lr, args.clip_val, 291 | args.k, args.epsilon 292 | ) 293 | 294 | 295 | if os.path.exists(output_file): 296 | print('Existing result') 297 | return 298 | 299 | 300 | train_loader = torch.utils.data.DataLoader( 301 | datasets.MNIST( 302 | args.data_root, 303 | train=True, 304 | download=True, 305 | transform=transforms.Compose( 306 | [ 307 | transforms.ToTensor(), 308 | transforms.Normalize((MNIST_MEAN,), (MNIST_STD,)), 309 | ] 310 | ), 311 | ), 312 | batch_size=args.batch_size, 313 | num_workers=0, 314 | pin_memory=True, 315 | ) 316 | test_loader = torch.utils.data.DataLoader( 317 | datasets.MNIST( 318 | args.data_root, 319 | train=False, 320 | transform=transforms.Compose( 321 | [ 322 | transforms.ToTensor(), 323 | transforms.Normalize((MNIST_MEAN,), (MNIST_STD,)), 324 | ] 325 | ), 326 | ), 327 | batch_size=args.test_batch_size, 328 | shuffle=True, 329 | num_workers=0, 330 | pin_memory=True, 331 | ) 332 | run_results = np.zeros((args.epochs,args.num_rep)) 333 | 334 | for it in range(args.num_rep): 335 | model = GradSampleModule(ConvNet().to(device)) 336 | 337 | num_parameters = sum([np.prod(layer.size()) for layer in model.parameters()]) 338 | print("Num of parameters in model = %d" % num_parameters) 339 | 340 | optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum) 341 | 342 | 343 | for epoch in range(1, args.epochs + 1): 344 | train(args, model, device, train_loader, optimizer, epoch) 345 | test_acc = test(model, device, test_loader) 346 | run_results[epoch-1][it] = test_acc 347 | 348 | 349 | np.save(output_file,run_results) #.detach().cpu().numpy()) 350 | 351 | if __name__ == "__main__": 352 | main() 353 | 354 | 355 | 356 | -------------------------------------------------------------------------------- /utilities.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | 4 | import numpy as np 5 | import scipy 6 | from scipy.special import gamma 7 | from scipy import stats as st 8 | import math 9 | from scipy.stats import norm 10 | import math 11 | import os 12 | 13 | 14 | #normalize vector x to have norm 1 15 | def normalize(x, ord=None): 16 | return x / np.linalg.norm(x, ord) 17 | 18 | 19 | # default is l2 clipping, but can also do l_p clipping by setting ord=p 20 | def clip(x, ord=None): 21 | return x / max(1, np.linalg.norm(x, ord)) 22 | 23 | # Adds 1 bit to complete the squared norm to C then normalizes to 1 24 | def complete_then_normalize(x,C): 25 | b = max(C - np.linalg.norm(x)**2, 0) 26 | u = np.zeros(len(x)+1) 27 | u[:-1] = x 28 | u[-1] = math.sqrt(b) 29 | return normalize(u) 30 | 31 | 32 | # Applies hadamard transform over x (without normalization) 33 | # if n=len(x) is not a power of 2, we concatenate 0's to x 34 | # so that len(x) is a power of 2 then apply the transform 35 | def unnormalized_hadamard_transform(x): 36 | n = len(x) 37 | if n == 1: 38 | return x.copy() 39 | a = math.log(n,2) 40 | if a - int(a) > 0: 41 | n_new = 2**(math.ceil(a)) 42 | y1 = unnormalized_hadamard_transform(x[:n_new//2]) 43 | x_new = np.zeros(n_new//2) # fill right side with zeros 44 | x_new[:n-n_new//2] = x[n_new//2:] 45 | y2 = unnormalized_hadamard_transform(x_new) 46 | ret = np.zeros(n_new) 47 | ret[:len(ret)//2] = y1 + y2 48 | ret[len(ret)//2:] = y1 - y2 49 | return ret[:n] 50 | else: 51 | y1 = unnormalized_hadamard_transform(x[:len(x)//2]) 52 | y2 = unnormalized_hadamard_transform(x[len(x)//2:]) 53 | ret = np.zeros(len(x)) 54 | ret[:len(ret)//2] = y1 + y2 55 | ret[len(ret)//2:] = y1 - y2 56 | return ret 57 | 58 | 59 | # Applies the hadamard transform for a vector x 60 | # if fast is True (for MNIST experiment), we apply FWHT which is 61 | # faster when d~60k 62 | def hadamard_transform(x,fast): 63 | if fast: 64 | return FWHT(x) 65 | else: 66 | return unnormalized_hadamard_transform(x) / len(x)**0.5 67 | 68 | 69 | def FWHT(x): 70 | n = len(x) 71 | a = math.log(n,2) 72 | if a - int(a) > 0: 73 | n_new = 2**(math.ceil(a)) 74 | x_new = np.zeros(n_new) # fill with zeros 75 | x_new[:n] = x 76 | return unnormalized_FWHT(x_new)[:n]/ n**0.5 77 | else: 78 | return unnormalized_FWHT(x) / n**0.5 79 | 80 | # Faster hadamard implementation for lower-dimensional problems (d~60k) 81 | # Code from Github Repo (https://github.com/dingluo/fwht) 82 | def unnormalized_FWHT(x,find_perm = True): 83 | """ Fast Walsh-Hadamard Transform 84 | Based on mex function written by Chengbo Li@Rice Uni for his TVAL3 algorithm. 85 | His code is according to the K.G. Beauchamp's book -- Applications of Walsh and Related Functions. 86 | """ 87 | x_original = x 88 | x = x.copy() 89 | x = x.squeeze() 90 | N = x.size 91 | G = int(N/2) # Number of Groups 92 | M = 2 # Number of Members in Each Group 93 | 94 | # First stage 95 | y = np.zeros((int(N/2),2)) 96 | y[:,0] = x[0::2] + x[1::2] 97 | y[:,1] = x[0::2] - x[1::2] 98 | x = y.copy() 99 | # Second and further stage 100 | for nStage in range(2,int(math.log(N,2))+1): 101 | y = np.zeros((int(G/2),M*2)) 102 | y[0:int(G/2),0:M*2:4] = x[0:G:2,0:M:2] + x[1:G:2,0:M:2] 103 | y[0:int(G/2),1:M*2:4] = x[0:G:2,0:M:2] - x[1:G:2,0:M:2] 104 | y[0:int(G/2),2:M*2:4] = x[0:G:2,1:M:2] - x[1:G:2,1:M:2] 105 | y[0:int(G/2),3:M*2:4] = x[0:G:2,1:M:2] + x[1:G:2,1:M:2] 106 | x = y.copy() 107 | G = int(G/2) 108 | M = M*2 109 | x = y[0,:] 110 | x = x.reshape((x.size,1)) 111 | ret = 1.0*x[:,0] 112 | # This functions returns a shuffles version of our hadamard-transform 113 | # Here, we find this permutation and return the exact vector as our 114 | # function unnormalized_hadamard_transform above 115 | if find_perm: 116 | perm = find_FWHT_perm(len(x)) 117 | return ret[perm][0,:] 118 | else: 119 | return ret 120 | 121 | 122 | # Find a permutation that makes the output of unnormalized_FWHT and 123 | # unnormalized_hadamard_transform the same 124 | def find_FWHT_perm(n): 125 | x = np.random.normal(0,1,n) 126 | f_name = 'dict_perm.npy' 127 | dict_perm = {} 128 | if os.path.exists(f_name): 129 | dict_perm = np.load(f_name,allow_pickle='TRUE').item() 130 | if n in dict_perm: 131 | perm = dict_perm[n] 132 | else: 133 | ret_right = unnormalized_hadamard_transform(x) 134 | ret_FWHT = unnormalized_FWHT(x,False) 135 | perm = find_perm(ret_right,ret_FWHT) 136 | dict_perm[len(x)] = perm 137 | np.save(f_name, dict_perm) 138 | 139 | 140 | # Input y is a permutation of w. 141 | # Returns a permutation such that w[perm]=y 142 | def find_perm(y,w): 143 | #print(y) 144 | #print(w) 145 | perm = [] 146 | for j in range(len(y)): 147 | perm.append(np.where(w==y[j])[0][0]) 148 | perm = [int(p) for p in perm] 149 | return perm 150 | 151 | 152 | 153 | 154 | # Applies HDS to vector x 155 | def applyHDs(x, S, D,fast=False): 156 | z = x.copy() 157 | for d in D: 158 | z = hadamard_transform(d*z,fast) 159 | if S is None: 160 | return z 161 | L = [] 162 | for j in S: 163 | L.append(z[j]) 164 | return (len(x) / len(S))**0.5 * np.array(L) 165 | 166 | # Applies Inverse HDS to vector x 167 | def applyInverseHDs(x, S, D,fast=False): 168 | z = np.zeros(len(D[0])) 169 | if S is None: 170 | z = x.copy() 171 | for d in D[::-1]: 172 | z = d*hadamard_transform(z,fast) 173 | return z 174 | else: 175 | for i in range(len(S)): 176 | z[S[i]] += x[i] 177 | for d in D[::-1]: 178 | z = d*hadamard_transform(z,fast) 179 | return (len(D[0]) / len(S))**0.5 * z 180 | 181 | 182 | def get_gamma_sigma(p, eps): 183 | # Want p(1-q)/q(1-p) = exp(eps) 184 | # I.e q^{-1} -1 = (1-q)/q = exp(eps) * (1-p)/p 185 | qinv = 1 + (math.exp(eps) * (1.0-p)/ p) 186 | q = 1.0 / qinv 187 | gamma = st.norm.isf(q) 188 | # Now the expected dot product is (1-p)*E[N(0,1)|gamma] 189 | # These conditional expectations are given by pdf(gamma)/cdf(gamma) and pdf(gamma)/sf(gamma) 190 | unnorm_mu = st.norm.pdf(gamma) * (-(1.0-p)/st.norm.cdf(gamma) + p/st.norm.sf(gamma)) 191 | sigma = 1./unnorm_mu 192 | return gamma, sigma 193 | 194 | def priv_unit_G_get_p(eps, return_sigma=False): 195 | # Mechanism: 196 | # With probability p, sample a Gaussian conditioned on g.x \geq gamma 197 | # With probability (1-p), sample conditioned on g.x \leq gamma 198 | # Scale g appropriately to get the expectation right 199 | # Let q(gamma) = Pr[g.x \geq gamma] = Pr[N(0,1) \geq gamma] = st.norm.sf(gamma) 200 | # Then density for x above threshold = p(x) * p/q(gamma) 201 | # And density for x below threhsold = p(x) * (1-p)/(1-q(gamma)) 202 | # Thus for a p, gamma is determined by the privacy constraint. 203 | plist = np.arange(0.01, 1.0, 0.01) 204 | glist = [] 205 | slist = [] 206 | for p in plist: 207 | gamma, sigma = get_gamma_sigma(p, eps) 208 | # thus we have to scale this rv by sigma to get it to be unbiased 209 | # The variance proxy is then d sigma^2 210 | slist.append(sigma) 211 | glist.append(gamma) 212 | ii = np.argmin(slist) 213 | if return_sigma: 214 | return plist[ii], slist[ii] 215 | else: 216 | return plist[ii] 217 | 218 | # calculates MSE norm of PrivUnitG 219 | def priv_unit_G_sq_norm(dim, eps): 220 | p, sigma = priv_unit_G_get_p(eps, return_sigma=True) 221 | var = dim * sigma * sigma 222 | return var 223 | 224 | 225 | # Fast algorithm for sampling from a truncated Gaussian (conditioned on having value at least gamma) 226 | # This combination of sf and isf seems to be the most stable one 227 | # If we could get isf(exp(a)) , i.e. isf with input being in log space, then this can be made a lot more stable by doing q and r in log space 228 | # This seems to work until gamma about 35 at least. 229 | 230 | def sample_from_G_tail(gamma): 231 | q = norm.sf(gamma) 232 | r = np.random.uniform(low=0, high=q) 233 | #print(q,r) 234 | return norm.isf(r) 235 | 236 | # More stable version. Works at least until 1000 237 | def sample_from_G_tail_stable(gamma): 238 | return sample_from_G_tail(gamma) 239 | logq = norm.logsf(gamma) 240 | u = np.random.uniform(low=0, high=1) 241 | logu = np.log(u) 242 | logr = logq + logu # r is now uniform in (0,q) 243 | #print(q,r) 244 | return -scipy.ndtri_exp(logr) 245 | 246 | 247 | --------------------------------------------------------------------------------