├── .gitignore ├── LICENSE ├── Makefile ├── README.rst ├── demo ├── README.rst ├── saved_models │ ├── checkpoint │ ├── simple_mnist.data-00000-of-00001 │ ├── simple_mnist.index │ └── simple_mnist.meta └── simple_mnist.ipynb ├── docs ├── Makefile ├── conf.py ├── index.rst ├── requirements.txt └── stadv.rst ├── illustration-stadv-mnist.png ├── requirements.txt ├── setup.py ├── stadv ├── __init__.py ├── __version__.py ├── layers.py ├── losses.py └── optimization.py └── tests ├── __init__.py ├── context.py ├── test_layers.py ├── test_losses.py └── test_optimization.py /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | 3 | build 4 | dist 5 | stadv.egg-info 6 | 7 | docs/_build 8 | 9 | .ipynb_checkpoints 10 | demo/t10k-images-idx3-ubyte 11 | demo/t10k-labels-idx1-ubyte 12 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Beranger Dumont 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .PHONY: docs 2 | 3 | init: 4 | pip install -r requirements.txt 5 | 6 | test: 7 | python -m unittest discover 8 | 9 | docs: 10 | cd docs && make html 11 | @echo "\nBuild successful!" 12 | @echo "View the docs homepage at docs/_build/html/index.html.\n" 13 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | stAdv: Spatially Transformed Adversarial Examples with TensorFlow 2 | ================================================================= 3 | 4 | Deep neural networks have been shown to be vulnerable to 5 | `adversarial examples `_: 6 | very small perturbations of the input having a dramatic impact on the 7 | predictions. In this package, we provide a 8 | `TensorFlow `_ implementation for a new type of 9 | adversarial attack based on local geometric transformations: 10 | *Spatially Transformed Adversarial Examples* (stAdv). 11 | 12 | .. image:: illustration-stadv-mnist.png 13 | 14 | Our implementation follows the procedure from the original paper: 15 | 16 | | Spatially Transformed Adversarial Examples 17 | | Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, Dawn Song 18 | | `ICLR 2018 (conference track) `_, `arXiv:1801.02612 `_ 19 | 20 | If you use this code, please cite the following paper for which this 21 | implementation was originally made: 22 | 23 | | Robustness of Rotation-Equivariant Networks to Adversarial Perturbations 24 | | Beranger Dumont, Simona Maggio, Pablo Montalvo 25 | | `ICML 2018 Workshop on "Towards learning with limited labels: Equivariance, Invariance, and Beyond" `_, `arXiv:1802.06627 `_ 26 | 27 | Installation 28 | ------------ 29 | 30 | First, make sure you have `installed TensorFlow `_ (CPU or GPU version). 31 | 32 | Then, to install the ``stadv`` package, simply run 33 | 34 | .. code-block:: bash 35 | 36 | $ pip install stadv 37 | 38 | Usage 39 | ----- 40 | 41 | A typical use of this package is as follows: 42 | 43 | 1. Start with a trained network implemented in TensorFlow. 44 | 2. Insert the ``stadv.layers.flow_st`` layer in the graph immediately after the 45 | input layer. This is in order to perturb the input images according to local 46 | differentiable geometric perturbations parameterized with input flow tensors. 47 | 3. In the end of the graph, after computing the logits, insert the computation 48 | of an adversarial loss (to fool the network) and of a flow loss (to enforce 49 | local smoothness), e.g. using ``stadv.losses.adv_loss`` and 50 | ``stadv.losses.flow_loss``, respectively. Define the final loss to be 51 | optimized as a combination of the two. 52 | 4. Find the flows which minimize this loss, e.g. by using an L-BFGS-B optimizer 53 | as conveniently provided in ``stadv.optimization.lbfgs``. 54 | 55 | An end-to-end example use of the library is provided in the notebook 56 | ``demo/simple_mnist.ipynb`` (`see on GitHub `_). 57 | 58 | Documentation 59 | ------------- 60 | 61 | The documentation of the API is available at 62 | http://stadv.readthedocs.io/en/latest/stadv.html. 63 | 64 | Testing 65 | ------- 66 | 67 | You can run all unit tests with 68 | 69 | .. code-block:: bash 70 | 71 | $ make init 72 | $ make test 73 | -------------------------------------------------------------------------------- /demo/README.rst: -------------------------------------------------------------------------------- 1 | Demo of the ``stadv`` package 2 | ============================= 3 | 4 | - ``simple_mnist.ipynb``: end-to-end demo of the ``stadv`` package 5 | (try and fool a small CNN pre-trained on MNIST). 6 | - ``saved_models``: directory with the weights of the model used in 7 | ``simple_mnist.ipynb``. 8 | -------------------------------------------------------------------------------- /demo/saved_models/checkpoint: -------------------------------------------------------------------------------- 1 | model_checkpoint_path: "simple_mnist" 2 | all_model_checkpoint_paths: "simple_mnist" 3 | -------------------------------------------------------------------------------- /demo/saved_models/simple_mnist.data-00000-of-00001: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rakutentech/stAdv/26286a8e84b61d474a958735dcff8f70d31deccc/demo/saved_models/simple_mnist.data-00000-of-00001 -------------------------------------------------------------------------------- /demo/saved_models/simple_mnist.index: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rakutentech/stAdv/26286a8e84b61d474a958735dcff8f70d31deccc/demo/saved_models/simple_mnist.index -------------------------------------------------------------------------------- /demo/saved_models/simple_mnist.meta: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rakutentech/stAdv/26286a8e84b61d474a958735dcff8f70d31deccc/demo/saved_models/simple_mnist.meta -------------------------------------------------------------------------------- /demo/simple_mnist.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# End-to-end demo of the ``stadv`` package\n", 8 | "We use a small CNN pre-trained on MNIST and try and fool the network using *Spatially Transformed Adversarial Examples* (stAdv)." 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "metadata": {}, 14 | "source": [ 15 | "### Import the relevant libraries" 16 | ] 17 | }, 18 | { 19 | "cell_type": "code", 20 | "execution_count": 1, 21 | "metadata": {}, 22 | "outputs": [], 23 | "source": [ 24 | "%matplotlib inline\n", 25 | "\n", 26 | "from __future__ import absolute_import\n", 27 | "from __future__ import division\n", 28 | "from __future__ import print_function\n", 29 | "\n", 30 | "import sys\n", 31 | "import os\n", 32 | "import numpy as np\n", 33 | "import tensorflow as tf\n", 34 | "import stadv\n", 35 | "\n", 36 | "# dependencies specific to this demo notebook\n", 37 | "import matplotlib.pyplot as plt\n", 38 | "import idx2numpy" 39 | ] 40 | }, 41 | { 42 | "cell_type": "markdown", 43 | "metadata": {}, 44 | "source": [ 45 | "### Load MNIST data\n", 46 | "The test data for the MNIST dataset should be downloaded from http://yann.lecun.com/exdb/mnist/,\n", 47 | "decompressed, and put in a directory ``mnist_data_dir``.\n", 48 | "\n", 49 | "This can be done in command line with:\n", 50 | "```\n", 51 | "wget http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz && gunzip -f t10k-images-idx3-ubyte.gz\n", 52 | "wget http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz && gunzip -f t10k-labels-idx1-ubyte.gz\n", 53 | "```" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 2, 59 | "metadata": { 60 | "scrolled": true 61 | }, 62 | "outputs": [ 63 | { 64 | "name": "stdout", 65 | "output_type": "stream", 66 | "text": [ 67 | "Shape of images: (10000, 28, 28, 1)\n", 68 | "Range of values: from 0 to 255\n", 69 | "Shape of labels: (10000,)\n", 70 | "Range of values: from 0 to 9\n" 71 | ] 72 | } 73 | ], 74 | "source": [ 75 | "mnist_data_dir = '.'\n", 76 | "mnist_images = idx2numpy.convert_from_file(os.path.join(mnist_data_dir, 't10k-images-idx3-ubyte'))\n", 77 | "mnist_labels = idx2numpy.convert_from_file(os.path.join(mnist_data_dir, 't10k-labels-idx1-ubyte'))\n", 78 | "mnist_images = np.expand_dims(mnist_images, -1)\n", 79 | "\n", 80 | "print(\"Shape of images:\", mnist_images.shape)\n", 81 | "print(\"Range of values: from {} to {}\".format(np.min(mnist_images), np.max(mnist_images)))\n", 82 | "print(\"Shape of labels:\", mnist_labels.shape)\n", 83 | "print(\"Range of values: from {} to {}\".format(np.min(mnist_labels), np.max(mnist_labels)))" 84 | ] 85 | }, 86 | { 87 | "cell_type": "markdown", 88 | "metadata": {}, 89 | "source": [ 90 | "### Definition of the graph\n", 91 | "The CNN we consider is using the `layers` module of TensorFlow. It was heavily inspired by this tutorial: https://www.tensorflow.org/tutorials/layers" 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": 3, 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "# definition of the inputs to the network\n", 101 | "images = tf.placeholder(tf.float32, shape=[None, 28, 28, 1], name='images')\n", 102 | "flows = tf.placeholder(tf.float32, [None, 2, 28, 28], name='flows')\n", 103 | "targets = tf.placeholder(tf.int64, shape=[None], name='targets')\n", 104 | "tau = tf.placeholder_with_default(\n", 105 | " tf.constant(0., dtype=tf.float32),\n", 106 | " shape=[], name='tau'\n", 107 | ")\n", 108 | "\n", 109 | "# flow-based spatial transformation layer\n", 110 | "perturbed_images = stadv.layers.flow_st(images, flows, 'NHWC')\n", 111 | "\n", 112 | "# definition of the CNN in itself\n", 113 | "conv1 = tf.layers.conv2d(\n", 114 | " inputs=perturbed_images,\n", 115 | " filters=32,\n", 116 | " kernel_size=[5, 5],\n", 117 | " padding=\"same\",\n", 118 | " activation=tf.nn.relu\n", 119 | ")\n", 120 | "pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)\n", 121 | "conv2 = tf.layers.conv2d(\n", 122 | " inputs=pool1,\n", 123 | " filters=64,\n", 124 | " kernel_size=[5, 5],\n", 125 | " padding=\"same\",\n", 126 | " activation=tf.nn.relu\n", 127 | ")\n", 128 | "pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)\n", 129 | "pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])\n", 130 | "logits = tf.layers.dense(inputs=pool2_flat, units=10)\n", 131 | "\n", 132 | "# definition of the losses pertinent to our study\n", 133 | "L_adv = stadv.losses.adv_loss(logits, targets)\n", 134 | "L_flow = stadv.losses.flow_loss(flows, padding_mode='CONSTANT')\n", 135 | "L_final = L_adv + tau * L_flow\n", 136 | "grad_op = tf.gradients(L_final, flows, name='loss_gradient')[0]" 137 | ] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "metadata": {}, 142 | "source": [ 143 | "### Import the learned weights\n", 144 | "The network has been trained independently and its learned weights are shipped with the demo. The final error on the test set is of 1.3%." 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 4, 150 | "metadata": {}, 151 | "outputs": [ 152 | { 153 | "name": "stdout", 154 | "output_type": "stream", 155 | "text": [ 156 | "INFO:tensorflow:Restoring parameters from saved_models/simple_mnist\n" 157 | ] 158 | } 159 | ], 160 | "source": [ 161 | "init = tf.global_variables_initializer()\n", 162 | "saver = tf.train.Saver()\n", 163 | "\n", 164 | "sess = tf.Session()\n", 165 | "sess.run(init)\n", 166 | "saver.restore(sess, os.path.join('saved_models', 'simple_mnist'))" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": {}, 172 | "source": [ 173 | "### Test the model on a single image\n", 174 | "The test image is randomly picked from the test set of MNIST. Its target label is also selected randomly." 175 | ] 176 | }, 177 | { 178 | "cell_type": "code", 179 | "execution_count": 5, 180 | "metadata": {}, 181 | "outputs": [ 182 | { 183 | "name": "stdout", 184 | "output_type": "stream", 185 | "text": [ 186 | "Considering image # 701 from the test set of MNIST\n", 187 | "Ground truth label: 0\n", 188 | "Randomly selected target label: 2\n", 189 | "Predicted label (no perturbation): 0\n" 190 | ] 191 | } 192 | ], 193 | "source": [ 194 | "i_random_image = np.random.randint(0, len(mnist_images))\n", 195 | "test_image = mnist_images[i_random_image]\n", 196 | "test_label = mnist_labels[i_random_image]\n", 197 | "random_target = np.random.choice([num for num in range(10) if num != test_label])\n", 198 | "\n", 199 | "print(\"Considering image #\", i_random_image, \"from the test set of MNIST\")\n", 200 | "print(\"Ground truth label:\", test_label)\n", 201 | "print(\"Randomly selected target label:\", random_target)\n", 202 | "\n", 203 | "# reshape so as to have a first dimension (batch size) of 1\n", 204 | "test_image = np.expand_dims(test_image, 0)\n", 205 | "test_label = np.expand_dims(test_label, 0)\n", 206 | "random_target = np.expand_dims(random_target, 0)\n", 207 | "\n", 208 | "# with no flow the flow_st is the identity\n", 209 | "null_flows = np.zeros((1, 2, 28, 28))\n", 210 | "\n", 211 | "pred_label = np.argmax(sess.run(\n", 212 | " [logits],\n", 213 | " feed_dict={images: test_image, flows: null_flows}\n", 214 | "))\n", 215 | "\n", 216 | "print(\"Predicted label (no perturbation):\", pred_label)" 217 | ] 218 | }, 219 | { 220 | "cell_type": "markdown", 221 | "metadata": {}, 222 | "source": [ 223 | "### Where the magic takes place\n", 224 | "Optimization of the flow so as to minimize the loss using an L-BFGS-B optimizer." 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": 6, 230 | "metadata": {}, 231 | "outputs": [ 232 | { 233 | "name": "stdout", 234 | "output_type": "stream", 235 | "text": [ 236 | "Final loss: 3.042147397994995\n", 237 | "Optimization info: {'warnflag': 0, 'task': 'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH', 'grad': array([-8.85338522e-07, 3.08994227e-03, 9.12457518e-03, ...,\n", 238 | " 1.18389018e-02, 2.62641702e-02, 2.19407566e-02]), 'nit': 8, 'funcalls': 73}\n", 239 | "Predicted label after perturbation: 2\n" 240 | ] 241 | } 242 | ], 243 | "source": [ 244 | "results = stadv.optimization.lbfgs(\n", 245 | " L_final,\n", 246 | " flows,\n", 247 | " # random initial guess for the flow\n", 248 | " flows_x0=np.random.random_sample((1, 2, 28, 28)),\n", 249 | " feed_dict={images: test_image, targets: random_target, tau: 0.05},\n", 250 | " grad_op=grad_op,\n", 251 | " sess=sess\n", 252 | ")\n", 253 | "\n", 254 | "print(\"Final loss:\", results['loss'])\n", 255 | "print(\"Optimization info:\", results['info'])\n", 256 | "\n", 257 | "test_logits_perturbed, test_image_perturbed = sess.run(\n", 258 | " [logits, perturbed_images],\n", 259 | " feed_dict={images: test_image, flows: results['flows']}\n", 260 | ")\n", 261 | "pred_label_perturbed = np.argmax(test_logits_perturbed)\n", 262 | "\n", 263 | "print(\"Predicted label after perturbation:\", pred_label_perturbed)" 264 | ] 265 | }, 266 | { 267 | "cell_type": "markdown", 268 | "metadata": {}, 269 | "source": [ 270 | "### Show the results" 271 | ] 272 | }, 273 | { 274 | "cell_type": "code", 275 | "execution_count": 7, 276 | "metadata": {}, 277 | "outputs": [ 278 | { 279 | "data": { 280 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAADTCAYAAAClbpYZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XecXVXV//HvmpJkUoYUUiAhBRIgCSFgkCAI0lGKIoiCFaQq4iMI8qDY8Qf8aBaEIA8IKsXA68EKSFd6kWJM6KRBes9kUqbs5499Ipe56yRzZ08Nn/frNa9XZt095+x7797nrHvuWdkWQhAAAABapqyjOwAAANCVkUwBAAAkIJkCAABIQDIFAACQgGQKAAAgAckUAABAApKpLsLMDjazWR3dD6CQmY00s2BmFR3dF6CzMbPpZrZ/9m8zs1+b2XIzeyaLfcXMFppZjZkN6NDOIkm7JlPZgNn402hmawt+/1x79iXrT5mZXW5my8xsqZldnLCtiuyksiZ7Pm+b2WVm1iEJq5mNMrO/m1mtmb1sZge0cDvfLXiP1plZQ8HvL7V2v5vRn8vNbEoJ7Xub2c1mNtfMVpnZc2Z2YFv2sbMxs1kFc22hmd1kZr07oB/dzewGM5ttZqvN7EUz+1jC9jpNImdmZ5vZW9kYm2dmV22qX2Z2kJm9ks3Ph81shNOmv5ktNrPH2rb3XV82xjeY2dZN4i9kY2RkK+9v49jbeCxcaGZ/MbNDCtuFEMaHEB7Jfv2wpEMkDQsh7GlmlZKulHRoCKF3CGFpa/axI2xujpvZuOwYvDz7ecDMxhU8bmZ2aXY+Xpr923L2dYCZTTOzFVnbu8xsqNOuXeZRu57oswHTO4TQW9IcSUcVxG5p2r4dDpJfkXS4pF0kTZR0jJmdkrjN8dnzO1TSlyR9uWmDdjr4T5X0tKT+kr4v6X9b8sknhPDjgvfsa5IeLXjPJpa6vQ448fWQ9JqkvSX1lXSppD+Y2ZB27kdHOyp7Dz8gaQ9JFzZtkB3I2vKYUCFprqSPSNoq68PU1j7RdZA/SfpACKFa7x5Pvu41zE74/yvpu4rz8zlJv3eaXirp5Tbp7ZZppqQTNv5iZhMk9WzjffbN5tVESfdLusvMTsxpO0LSrBDCmuz3wYrHp+kt2bGZlbfk79rY5ub4PEmfUhz3WyvOm9sL/v40SUcrvp67SjpK0uk5+5oh6bAQQl9J20p6XdK1Trv2mUchhA75kTRL0sFNYhcpHlRuk7Ra0omSfifpBwVtDlYckBt/HybpLkmLFSfTmSX04RlJXy74/XRJj7Xw+VRICpJGFsTukvTT7N9vSzpP0jRJ6zfXd8WDwG8lLVecbOcXPu/N9GWcpLWSehXEnpR0SuJ7doqkR5z4ryS9I2mVYgK3Z8Fjl2fP447sPT1eUh/FCbQiez2+I+mVgr8ZIenPkpZIelPSqVn8WEkbJNVJqpH0RAufx1uKk7DDxn97/jSda5Iuk/SX7N+PSPqJpMezMTNa8SB4g6T52ft6kaTyrH159p4uyV7HM7NxX9HCvv1L0rEt/NuRefuW1F3STxUP3vOyf3fPHtta0l+y8bdM0qOSyrLHzs+e82pJr0o6qAX9GiDpAUnX5Dx+WuHYldQre+13Lojtnc3Zk9TCY9L76Scb4xdKerYgdnl2bPnPcVnSEZJeyI5Vc/Xec8tnFI/D1dnvH5O0QNLA5o49SedKWlgwnmYpnrNOlrROUkN27LpN0ppsGzWSHsra76yYlC3Lxt+nC7Z9k2KycHf2twdn4/xyxYsTCyVNkVSVtd9f8bzzTUmLsvl8UsH2qiRdIWm2pJWSHiv4270kPZHNkZck7Z/w3rhzXPGceaak2oLYE5JOK/j9ZElPNWMf3SVdLGlGk3i7zaOOHvxeMrVBMRsty97s3GQqa/OipG9L6qZ4Ipil7AComB0v2UQf1kiaVPD7XpKWt/D5vCeZkjQ+G8Bfyn5/W9I/FROoqmb0/XLFE10/xeRiht6bRF4n6ec5fTlO0rQmsSmSrkp8z/KSqS9m/axUvAo2S9lBJnse6yR9VJJlz/1qSfdKqpY0SvGg8UrB6zhD8QBQKWmn7LXbp2B7U5rs/8eSbm/mcxihmIwN76ix394/hXNN0naKyfmPs98fUTwQj89e+0rFBP86xZP8IMUPHadn7c+Q9Eq2nf6SHlbBSUXSfytL1JrRr8HZ2Ni5hc9rpPKTqR9Jeirr/0DFg/TG53xxNh8qs599s7G5k+IJdtuC7e+Q/fvDklZspj+fVTxJB8UPSBNz2v1M0rVNYv9WdsJRTFiflzRJ8QMlyVQzx3h2LBmbvYZvZ/O98Li8v6QJisffXRUTkKMLtnOLYtIyQDEJP7KUsSdp+yw+trBf2b/f81423UY23+YqnvgrJO2u+KFlXPb4TYpJzz5Z/3tIukrx6k5/xQ+pf5Z0ccFzrc/mQqXitzC1kvplj/9Scf4PzV6vvRWTkqGSlmbtyxS/mlyqLKlUK8xxxSStXlKjpAsL4islTS74fQ9Jqzex/eHZthoVj+snFjzWrvOowwd/k9hFyjL0gtimkql9JL3VpP13JV3fjP1bNpBHF8TGSqpv4fPZmEytyt7cNyT9UJJlj78t6YsF7TfZd8UTXOHVhK+q+VemirJwxUud/5P4nrnJVJM25dnk2XgSulzS3U3aLFKWHGW/f0PvJlMHSXq5SfufSPpFwfamtLD/PRSvwFzRXuO8M/xkc60mG5ezJV2jdz+BPiLpRwVtB0tav/HxLHaCpIezfz8k6YyCxw5VC65MKR7cH5B0XcLzGpm3b8UrmocX/H5YwXHjR5L+WDj3s/jobGweLKkyoV9jFBP8ITmP3yDpkiaxx5WdCCSdrSzZEslUKWP8YMWrUxcrfni7X843Bk3+7qcq+JCpeCvAHMUr5rljM2/sZceYoHc//M1S85OpzyjeRlG4veskfT/7902SflPwmCleENihIPYhSTOzf++veMWzouDxRYoXDcqyx4oSfsWrs79tEvubsgsDJbwnm5zjisnjVyUdURBr0Huv0I7JXiPbzL76Z/3eqyDWrvOow2/cdMwtoe0IScPNbEVBrFzxBLFJIYRgZrWKV0c2qla8vF/EzO5TzNwl6eQQgnePgyTtGkKYlfNY4XPbXN+3adJ+ds42PTV67/OScp6bmW2veBlWiolk3xL2s3EbF0r6gqQhigO/m+JXKW9mTeYWtK1QvFJQ+Nyavi6jndfl3lL71aSPlYr3kc2X9K2UbXVRR4cQHsh5rOnrXylpfsF9n2UFbbZVy8elpFj4ofjV7wbF+/Dy2k3P+iNJHwshPFrCbrZt0rfZWUyKX3P+QNJ92XP8VQjhkhDCG2b2jeyx8Wb2N0nnhBDmlbBfhRBez/p+jaRjnCa589PMtlW812pSKfvEf/xW0j8Ur3j/pumDZjZZ0iWK97V1U7wSc8fGx0MIK8zsDknnKN5WUKqNN0Ava8HfjpA0ucmxr0LxOW1UOPcGKt4O8s+CuWqKx8uNloYQ6gt+r5XUW/H43EPvHqOb9uM4MzuqIFapeBW6WZozx0MIa7JiosVmNjaEsEjFc6NaUk3IMqI8IYRlZnazpJeym9AHqZ3nUWdMppq+aGv03psIC28cnivp9RDC2Bbua7rijW7PZ79PVM7NgCGEQ1u4j/dspuDfm+v7AsWvUl7Nfh9ewn6mKyYkPUMItVlsoqQbizoUwluKk6tFskqN0xUvBb+qeOJdozip/7Obgv3Vm9kSxa8752Th7QrazpX07xDC7jm73OSkyuljueIVzgpJnwohNJS6jS1c03G5XtLWTQ7CG83Xe9+vUsalssqcGxSvgB0eQqjL7VQI40vZdhPzFE8KG+fz8CymEMJqxa+Rv2lmu0h6yMyeDSE8GEK4VdKtZlateFXgUsUPCqWqkLRDzmPTFYtTJElm1itrO13SnoofpGZkJ8gqSVVmtkDSUMbupoUQZpvZTMWvqE52mtyqeJvBx0II68zsp4qJhSTJzHZTLBq6TdLPFa9wleKTild/Xt1cQ8dcSX8PIRyyiTaFc3WJ4tWl8SGEd0rc1xJl3yAo3hPVtB+/DSGcWuI2JZU2xxXPFz0Vk9BFevec/Ez2eO452VGhmERVqwPmUVf4f6ZelHSEmfUzs2303gqZJyVtMLNvmlkPMys3swlm1txs9DeKB9RtzWyY4mXBm1q19/k21/epkr5tZn3NbLg28Qm+qRDCDMUB+L1s259S/ArzrtZ+Eorf09crTs5uil/JVW7mb6ZKutDMqrMqj8Jqjb9L6m5mX8vKbCvMbGJ2kJPiPQ6j8splm8o+Id2kOMmODSFsaN7Ten8KIcyXdJ+kK7L3p8zMdjCzj2RNpkr6upkNM7N+ivdPlOJaxbF4VAhhbSt1u3s2zjf+lCmeDC80s4FZ9dz3FBNqmdmRZjY6G0MrFb9aaDSznczsQDPrrniiWat4L8ZmmdkpZjYo+/c4SRdIejCn+V2SdjGzY82sR9a3f4UQXpF0j+LXP7tlP99TvGF6NxKpZjtZ0oHh3aq5Qn0kLcsSqT0V73OTJGXvxe8U72M9SdJQM/tqc3ZoZoPN7GuK94xeEEJo1rhp4i+SdjSzL5hZZfbzQTNzP3Bn+7he0lUFY2+omR22uR1lf3ujpCuz81+5mX0oG/u/k3SUmR2WxXuY2f7ZObI5cue4mR1iZrtn261W/K8hluvdarvfSDonex7bKn7oucnbiZkdk83ZMjMbmG3rhRDCMnXAPOoKydRNii/0bMWvev5TRpl9cj5cMQudpXhCv07ZZcJsAKxQvmsUvwuervhV1x8VM+o2t7m+K07K+dlj96jJJWsz+x8zu3oTu/iM4vfnyxXv3zg2tM3/Y/IHxcRwpmJ11zvZPjflAsXLuXMVb5j8veLVEGXJzscUv++fo/hp5Zd69+rZrYr3NSwzs0clycx+kl2a94yV9HnF+wQW27v/L8zRJT/T948vKibGMxTfyzsVP+VJ8eD9N8VPs88rlvj/h5l928zu8TZq8f9SOl3x4LbAWu//mKtRTHw2/hyoeP/lc4rzelrW14uy9mMU7+WoURy714QQHlb8yucSxbm4QDEBvyDr+75mVrOJPuwjaZqZrVGstrpb8aS88blP3/g8QwiLFb9C+oni6ztZscpVIYT1IYQFG38Uk7267N9ohhDCmyGE53Ie/qqkH5nZasUT7NSCxy6WNDeEcG0IYb3iceMiMxuzid2tyN7zaYrH8+NCCEXfADSz36sV70E8XvEq6gLFK6PdN/Fn5yven/uUma1SHNc7NXOX52b9flbxa8lLFasQ50r6hOL4Xax4nD5PWb6QOMf7Kn7QWan4FeMOkj4aQliXPX6d4jlhmmJRxl+z2Mbt15jZvtmvQxVzgtVZ+0bFK4MdMo823hwNdBgzO0+x9PaIju4LAACl6gpXprCFMbMRZrZndnl2gqSz1DZfQQIA0OY64w3o2PL1kHSz4k3ByxS/wrypIzsEAEBL8TUfAABAAr7mAwAASEAyBQAAkKBd75k6pOw4vlNEp3J/4x3N+v+q2gpzAp1NR8+JEVMuY06gU5l9xnmbnRNcmQIAAEhAMgUAAJCAZAoAACAByRQAAEACkikAAIAEJFMAAAAJSKYAAAASkEwBAAAkIJkCAABIQDIFAACQgGQKAAAgAckUAABAApIpAACABCRTAAAACUimAAAAEpBMAQAAJKjo6A4AQKqK7Ue68Vcv6uvGPzf+WTc+fdU2bnx89Xw3/uyBg3M65B9aGxYu8tsDraysus6NXzH5Djd+dK8aN/52vR8fVtHbjY+655ScDgU/vrbcj3cxXJkCAABIQDIFAACQgGQKAAAgAckUAABAApIpAACABFTzbUEWnbm3Gz/9zD+68TP6vlPS9veb9kk3vvCZIW585HefLGn7QCGr7ObGG/YaXxRbesFqt+2VY6a68X17LHHj/Qb2dOM/XT7SjX/4YX8O3fGrg9z4wOf9akF74iU3DrxHzuWPPkOKx/+vJ97stn1q7Q5ufMwjn3LjDcu7u/EeC/wqvMpxa9x4mO3Pre47rnLjaxb0cuOdFVemAAAAEpBMAQAAJCCZAgAASEAyBQAAkIAb0Du58nE7FsU+fufjbtuTtvqZGy/LyZnrcv53/zwP7nKn/8Aufnjv3U9w4/2PfK20HeN9KUza2Y3XfLv4htU7x/s32+YteSH5N8Pm+Ua/WSW13/vs1/3t/OwMNz6kboIbD89OK2m/2LL1HOjf3P1fOz1cFPvMk6e5bRtW+oUdyjsfmP/A2m3q/fY5N6znJRuVD23lxqv2929MX7uotLnbXrgyBQAAkIBkCgAAIAHJFAAAQAKSKQAAgAQkUwAAAAmo5uvkXj9xQFHs5K3m5LTuXLnxE7vf5sZ3/f5Zbnz4D59oy+6gi1k0ya/E+8igGUWxvKq9N+tq3PgViw524+N7+cvDlKvRjectybR/ld++8cDlbnzDDL9CqdKN4v1qwpD5bnzqvD2KYg0r/Kq9snX+eaLnAj9eX5XTGfPD6wb5VX711Q1uvKLWX5am/plqfwcjc6oIO1jnOvsCAAB0MSRTAAAACUimAAAAEpBMAQAAJCCZAgAASEA1Xyd3yAEvJG/jqFc/7sZff2k7N15R65dp/PvEq5P7IkmN3UpcFBBdSlmfPm58yaf8RRyXTfDHwzbjFrjx4/o9UxT75vziaiZJ+vN9k924+YVFenrO7m582V51bnz7/fw1AQ/t6bfPU74up0PYMlT6Y3z8mLfd+Ee29tcvPaRXcSWrJH366VOLYr3m+Kf3gS/6Y3PpuJxrKzmH6w1j1rrxyrk93HjdVv4YDzlVgWXr/XhnxZUpAACABCRTAAAACUimAAAAEpBMAQAAJCCZAgAASEA1XxtZeuqH/Pgkv6LhuoN/7cY/2H2lE+3utt3zJ/6ad0NufsmNj17jrytmlf6aTpMnfNaNPz3pVjeOLUPeeCgbOcyNv3mRv05et27eWJbO2elRN/6ZPq+48X5lxYuF/T7463v1meWGNfCGZ914qM9ZV6xqbzd+98SJbvzQns+58cOG+8/pgYn+8WLQY24YHS3nMkR5H79S7urJ/jHy53MPcuMPLBzrxq+Zs7+/4/XF4z+vaHr2kXnXUHKq7cr9DU0eNduNP1/uV4lruX/eWjvIL+erq+5aVd9cmQIAAEhAMgUAAJCAZAoAACAByRQAAEACkikAAIAEVPM1U9lu49z4K2f4lUsvHnmlG+9pfmVUPr8CwjPq+Nfd+Jpr1pS0x1C3wd/O2ub3ZVOuO/46N37xd3Ztle2jZcoH9Hfji47eyY1vOGqFGz9l9ONu/Jx+/vgst7zPdL1y4sW+NehhN37YoTv6f3BDszctSerzjl/ptLrOX4csz0WDi9cVlKR7q/xqvvLq6qJYw6pVJe0TCbo3uuEJo/019fYd4I/xy2Yd5sbfemOIv19/tyVZu42/kYoaf77V9c1ZHzKnLwvWFI9NSbKy0qrw1g719zv8bn/Hsz/pBOtyFvhrR1yZAgAASEAyBQAAkIBkCgAAIAHJFAAAQAKSKQAAgARU8zWx/ES/qub08+9y41+s9te3k0qt2kv38UH+GnxTR+/jxhvemNmW3cm1e/ec6sK9nGq+p/7Vtp15HyofPMiNv/zDkW585A7z3PiNO97ixkdV+hWubfnZ7YHa7d14XZ2/Zp918+dn3tp8PZb4a671qVzXjN69q0J+f8Ye/aobX3N7n+Ig1Xytr8qvKLtgr3vc+JMrd3DjU17Yz403rq5sWb8SVK7yK9y6rfTjjeX+2Gzo5b82y2uL18iUpIoKv71fIy6FnOq/ZWP916xiWXH7+j45lYjtiCtTAAAACUimAAAAEpBMAQAAJCCZAgAASEAyBQAAkOB9W8238Ky93fhj5/tr6nW39q/GKNWUt/xKkv4rlrdzTzYtb33Ct44pXott+6faujdbtrI+xdVgC4/2K5F2HDPXjd+985/ceLnlVe35ahv9ep57ard24/9eO8yNP/j9DxfFVm/rH8pG37/AjTc2lFb9s25rf/6P6+lXOuZplF+5VGZ+fPWkoUWxqrfzKojRLJXFr/Wk0bPdpvct8ddk/ed0v3q01DX1rNGvrKtc6V/nKK/124+6pXitwIYBTiWopLqt/PUkZ36itHNcv55r3fi6en8u1uastWnBf06W81r2WFzcvsZ/qu2KK1MAAAAJSKYAAAASkEwBAAAkIJkCAABIQDIFAACQYIuv5ivbdWc3/o0z73TjbV219+R6f/2jS2Yf7sZfmTukKDbqJr/6of+0OW68YcnSZvaufdQ0rnfjO06ZXxTzV0pDs21fXBFX/vElbtPbd5zqxsutpxvPq85bmRM/9a3j3PjcP45y48Nuf8uN9+lWPE563eeP8Ya1fsWRgl89l2flKH/ePrTMP76ctlVpVX4v3edvZ+Tfni+KlVgwhia6VRcffw4Y4K+NeNWLB/kbyXkT8qrzrN6P93nLv57R7zV/LcjuS/21IGsmFJ8nFu3un9439PU7HypLG1ljtlrsxh95c0xJ28kzYLp/HJlzSOesrOfKFAAAQAKSKQAAgAQkUwAAAAlIpgAAABKQTAEAACTY4qv53jy+nxv/XJ/iiqCWeHydX1nwrR+d7sb7z6hx4+HZaW58jJpfFVTaamMdpyFnfbL6t2a1b0feB5bt2rco1r/Kr/rsV+5X7a1s9CviZtb5n8W+fvY5brzn7DVufNtXX3Tj9bW1brwjVC32x+zW3fznlOfXq7Zz492X+e0b1/nVW2i5UQOLKz9/N3tPt23DCn8dUWvwq/PK1vvx0besduOzP76VG5/34ZxKvP5VbtzXtrXQ89dWu/HKbv5+6+W/lt0X+5WyFTX+2G/o1TmvAXXOXgEAAHQRJFMAAAAJSKYAAAASkEwBAAAk2OJvQK8f5i9dUqq8G81/cNbJbrzf3U+68dIWsdgyTX7ka258tF5o555sOazSv7mzdpviG2LvHXNHzlb8G9CfXuffaPqd/3eKGx/04jtuvH6Wf+N7Z5oTZT16uPHVw/0biz894OmStv+zGQe48WG/eKKk7aAZci4VjOhdfLd/fU//JugFMwe48Yo1/sZH/NW/aXrmMf4c2jCgCyyYVe7P0J36LHTjdQ3+a/naIv/40nu2v/1Q3rWu9XSt3gIAAHQyJFMAAAAJSKYAAAASkEwBAAAkIJkCAABIsEVV85WPHlUUe+HAX+a09quf8pz27Ofd+Ki7ny1pO11BxagRbvzKSVNbZ/tz/IoptFztEbu58RGHzyyKVZeV9vpf+OrRbrzfK/5yL3lVe52K5SwHss1gNz7xsFfc+M6VecvJ9Cplt1p41t5ufDBVfi02dOQSN/7J/s8Xxb7ywJdK2naft/z4mm27u/ENA7rAYl85Y7Osl19x+Il+xa+jJN01bfdW2e/sI/KOU53zteTKFAAAQAKSKQAAgAQkUwAAAAlIpgAAABKQTAEAACTYoqr5XjujuBKnp5VWtZen1z96t8p2OpOyXn7F0Yz/HuTGD63Kq1xCRws5ZWJnb3dfUazcSvsMVbPWr1DqV99Y0nY6k/IB/d34nGOHuvErBl/vxgeV+3OoptFfo22brVa58W53rXDjXWDlti7nT8udarMSh3JZnR9fMTpvbnWBudLN7+PkUbPc+NXzDnLjodZfm88a/GNUfZUf7/uyv2bf4sluuMNxZQoAACAByRQAAEACkikAAIAEJFMAAAAJSKYAAAASbFHVfG1pzX41/gPXtG8/WlPtAePd+GtHXtsq2/9r7VZufNRdq924X7uB5li2s19Bc+/KXYtiB1W9UNK29x/xhht/Zld/Da4Bz5S0+TZV3tcfg8sPGePGLznjRjd+aE+/fKu2cYMbn3Df19z4mOv97YS6+W4cLbfbgHfc+OurBiZve91AvwKtm39o09rkPbainKq90SMXuvEzt3nQjX/+3jPcuDX6r82Qx3Kq9l7w9/v6yX5VeWfFlSkAAIAEJFMAAAAJSKYAAAASkEwBAAAkIJkCAABIQDXf+0Djvn7V1devvL1Vtj+n3q9VmfLZE914eG5aq+wX7xry9Ho3ftDJM5K3Pan3LDf+dJk/rmyPXdx4eP5lfweNDSX1p2JY8fp5jX37uG1fO7GvG//EAX7J4QE9/LXzJH+Nz4UNfjXf2G+85m+m0j/kNixdlrNftNTj80a58T2GzC2KvaZtS9p2nb8ko8r8aageA/1j5LolVf4flFjabL2KV3Gs6O6v7Hjy+Cfd+E49/IrSLz15cml9qfOr9vr8/ik3/sYlH3Lj9X1KOy50NK5MAQAAJCCZAgAASEAyBQAAkIBkCgAAIAHJFAAAQAKq+Zrp3IkPuPGrvne0Gx9x8XNuPNT51T+laDjgA2587hl+9calH7jDjR/Rc2VyXyRpaWN3Nx6epWqvvYQKv4Jmyjv7F8X22+HPbtueZX7F2uG9/LX5XvuKXxU09YU93Pi29/jxlTv46wr2WOqXNNUOLn6ufz31/7ttX60b4MYnd1/uxnuW9fT3mbMG39/Xbu/Glx85zo1X3+pXNKH1lZX5a9BNW7pNUSxvPblQ5o/BDX39bdf3zulLvX/dYuioJW581wHz3PiCtX7V6vBexeP5+4P+4bY95pUT3Ph18/d146HGTxPyXrPK1X7cPjjBjddVd62qvTxcmQIAAEhAMgUAAJCAZAoAACAByRQAAEACkikAAIAEW1Q134i764piM49b57YdVdGjpG2fVF28npMknXT6L9z4npM+58ZXrc5Zi6kEv9zrFjd+UFVt8rY3JW8NvuMf/bobH6Pn27I7KNDjmdfd+KJrxxbFHr6o2m17cNVqN75NhV+idOngF934Afv5a/BN23OYG39ymV8Rd+jW/rqCB/YqXveuT5lfQfTRnjmLpcmv2msIfpXWXWuKK8Ak6apXDnLjQ59b7G8/pzdofcsW+uPcKovf47L1/vjJKVRWcLYhSQ2VfvuyWf54W5Gzft4DK3Z044P61rjxCqdycdKDZ7ltQ61fPVuqyuX+tZiqBf5ruWY7/zUoeSHCToorUwAAAAlIpgAAABKQTAEAACQgmQIAAEhAMgUAAJDAQmi/O+kPKTuu3W/brz9wkhu/9tdaox5iAAAD2ElEQVQ/d+OlVvltiRrlV6pMvuS/3PjgXzzRlt1pU/c33uGXnrSTtp4T5YMHFcVeO3cHt+2Oe8x247eMvtONV5f5c6Xc2v8zWl4VXn1O/dy64FdRnTX3MDf+4h27uPFhN073+7Oidda97AgdPSdGTLmsbc8TVcVjYvTwRW7TBfdu58bXbOePq1DR+SvTLOS8vf4UUu9ZfvXfkCfXuPElE/2qveUTum4t6+wzztvsnODKFAAAQAKSKQAAgAQkUwAAAAlIpgAAABKQTAEAACTYotbm81Q89E83/pWT/PXkrrrxl258RE6VRk/r1rKOtYGaRn8dsoactY+eXj/AjZ9//Zfd+NAuXLX3ftWwsLhKacfrerltZ64Z6cb3eehcN/6DL/hrRH66d+tUsv211q8WvHnBPkWx0b38tfBu++eebrxbnw1uvGy6vw7hyOunufGG1f56hujE1hZXp81csLXbdOCBC934gKv7u/F5+/in1Lq+rVPJVrnCr6yrWlxcbFafU5hePTun8rWHX7DWfZXf9+U7b3lVeym4MgUAAJCAZAoAACAByRQAAEACkikAAIAEJFMAAAAJtvhqvjx5VX7njdzLjc87d283/vzZv2i1PnnG/v3kopi9XeW2HX3LCjfe+NLLJe1zqKja25I1vDHTjQ//gR+v2H6kG//Zqye48Ze/8w83ftsf9nfj9b386qKRf/Er7paOLy5TqrlvK7ft2MWvu/FS187LWbYMW4iGlZVufMFKv+K57IQ6N14+xz+lVr3jx4c87VdgN3b3r3P0fOYNN/72F8cUxXrN96u4l+zqV+3VV5dahff+rNrLw5UpAACABCRTAAAACUimAAAAEpBMAQAAJLAQ/JvU2sIhZce1386AZri/8Q7/bsx2wpxAZ9PRc2LElMuYE+hUZp9x3mbnBFemAAAAEpBMAQAAJCCZAgAASEAyBQAAkIBkCgAAIAHJFAAAQAKSKQAAgAQkUwAAAAlIpgAAABKQTAEAACQgmQIAAEhAMgUAAJCAZAoAACAByRQAAEACkikAAIAEJFMAAAAJSKYAAAASkEwBAAAkIJkCAABIQDIFAACQgGQKAAAgAckUAABAApIpAACABBZC6Og+AAAAdFlcmQIAAEhAMgUAAJCAZAoAACAByRQAAEACkikAAIAEJFMAAAAJSKYAAAASkEwBAAAkIJkCAABIQDIFAACQgGQKAAAgAckUAABAApIpAACABCRTAAAACUimAAAAEpBMAQAAJCCZAgAASEAyBQAAkIBkCgAAIAHJFAAAQAKSKQAAgAQkUwAAAAlIpgAAABL8HycdMuFUbHBGAAAAAElFTkSuQmCC\n", 281 | "text/plain": [ 282 | "
" 283 | ] 284 | }, 285 | "metadata": {}, 286 | "output_type": "display_data" 287 | } 288 | ], 289 | "source": [ 290 | "image_before = test_image[0, :, :, 0]\n", 291 | "image_after = test_image_perturbed[0, :, :, 0]\n", 292 | "\n", 293 | "difference = image_after - image_before\n", 294 | "max_diff = abs(difference).max()\n", 295 | "\n", 296 | "plt.rcParams['figure.figsize'] = [10, 10]\n", 297 | "\n", 298 | "f, (ax1, ax2, ax3) = plt.subplots(1, 3)\n", 299 | "\n", 300 | "ax1.imshow(image_before)\n", 301 | "ax1.set_title(\"True: {} - Pred: {} - Target: {}\".format(test_label[0], pred_label, random_target[0]))\n", 302 | "ax1.axis('off')\n", 303 | "ax2.imshow(image_after)\n", 304 | "ax2.set_title(\"Pred: {} - Loss: {}\".format(pred_label_perturbed, round(results['loss'], 2)))\n", 305 | "ax2.axis('off')\n", 306 | "ax3.imshow(difference)\n", 307 | "ax3.set_title(\"Max Difference: {}\".format(round(max_diff, 2)))\n", 308 | "ax3.axis('off')\n", 309 | "plt.show()" 310 | ] 311 | } 312 | ], 313 | "metadata": { 314 | "anaconda-cloud": {}, 315 | "kernelspec": { 316 | "display_name": "Python 3", 317 | "language": "python", 318 | "name": "python3" 319 | }, 320 | "language_info": { 321 | "codemirror_mode": { 322 | "name": "ipython", 323 | "version": 3 324 | }, 325 | "file_extension": ".py", 326 | "mimetype": "text/x-python", 327 | "name": "python", 328 | "nbconvert_exporter": "python", 329 | "pygments_lexer": "ipython3", 330 | "version": "3.5.2" 331 | } 332 | }, 333 | "nbformat": 4, 334 | "nbformat_minor": 2 335 | } 336 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | SPHINXPROJ = stAdv 8 | SOURCEDIR = . 9 | BUILDDIR = _build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 3 | # Configuration file for the Sphinx documentation builder. 4 | # 5 | # This file does only contain a selection of the most common options. For a 6 | # full list see the documentation: 7 | # http://www.sphinx-doc.org/en/stable/config 8 | 9 | # -- Path setup -------------------------------------------------------------- 10 | 11 | # If extensions (or modules to document with autodoc) are in another directory, 12 | # add these directories to sys.path here. If the directory is relative to the 13 | # documentation root, use os.path.abspath to make it absolute, like shown here. 14 | # 15 | import sys 16 | import os 17 | 18 | rootdir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) 19 | sys.path.insert(0, rootdir) 20 | 21 | # Load the package's __version__.py module as a dictionary. 22 | about = {} 23 | with open(os.path.join(rootdir, 'stadv', '__version__.py')) as f: 24 | exec(f.read(), about) 25 | 26 | 27 | # -- Project information ----------------------------------------------------- 28 | 29 | project = 'stAdv' 30 | copyright = '2018, Beranger Dumont' 31 | author = 'Beranger Dumont' 32 | 33 | # The short X.Y version 34 | version = about['__version__'] 35 | # The full version, including alpha/beta/rc tags 36 | release = about['__version__'] 37 | 38 | 39 | # -- General configuration --------------------------------------------------- 40 | 41 | # If your documentation needs a minimal Sphinx version, state it here. 42 | # 43 | # needs_sphinx = '1.0' 44 | 45 | # Add any Sphinx extension module names here, as strings. They can be 46 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 47 | # ones. 48 | extensions = [ 49 | 'sphinx.ext.autodoc', 50 | 'sphinxcontrib.napoleon', 51 | 'sphinx.ext.viewcode', 52 | ] 53 | 54 | # Add any paths that contain templates here, relative to this directory. 55 | templates_path = ['_templates'] 56 | 57 | # The suffix(es) of source filenames. 58 | # You can specify multiple suffix as a list of string: 59 | # 60 | # source_suffix = ['.rst', '.md'] 61 | source_suffix = '.rst' 62 | 63 | # The master toctree document. 64 | master_doc = 'index' 65 | 66 | # The language for content autogenerated by Sphinx. Refer to documentation 67 | # for a list of supported languages. 68 | # 69 | # This is also used if you do content translation via gettext catalogs. 70 | # Usually you set "language" from the command line for these cases. 71 | language = None 72 | 73 | # List of patterns, relative to source directory, that match files and 74 | # directories to ignore when looking for source files. 75 | # This pattern also affects html_static_path and html_extra_path . 76 | exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] 77 | 78 | # The name of the Pygments (syntax highlighting) style to use. 79 | pygments_style = 'sphinx' 80 | 81 | 82 | # -- Options for HTML output ------------------------------------------------- 83 | 84 | # The theme to use for HTML and HTML Help pages. See the documentation for 85 | # a list of builtin themes. 86 | # 87 | html_theme = 'classic' 88 | 89 | # Theme options are theme-specific and customize the look and feel of a theme 90 | # further. For a list of options available for each theme, see the 91 | # documentation. 92 | # 93 | # html_theme_options = {} 94 | 95 | # Add any paths that contain custom static files (such as style sheets) here, 96 | # relative to this directory. They are copied after the builtin static files, 97 | # so a file named "default.css" will overwrite the builtin "default.css". 98 | html_static_path = ['_static'] 99 | 100 | # Custom sidebar templates, must be a dictionary that maps document names 101 | # to template names. 102 | # 103 | # The default sidebars (for documents that don't match any pattern) are 104 | # defined by theme itself. Builtin themes are using these templates by 105 | # default: ``['localtoc.html', 'relations.html', 'sourcelink.html', 106 | # 'searchbox.html']``. 107 | # 108 | # html_sidebars = {} 109 | 110 | 111 | # -- Options for HTMLHelp output --------------------------------------------- 112 | 113 | # Output file base name for HTML help builder. 114 | htmlhelp_basename = 'stAdvdoc' 115 | 116 | 117 | # -- Options for LaTeX output ------------------------------------------------ 118 | 119 | latex_elements = { 120 | # The paper size ('letterpaper' or 'a4paper'). 121 | # 122 | # 'papersize': 'letterpaper', 123 | 124 | # The font size ('10pt', '11pt' or '12pt'). 125 | # 126 | # 'pointsize': '10pt', 127 | 128 | # Additional stuff for the LaTeX preamble. 129 | # 130 | # 'preamble': '', 131 | 132 | # Latex figure (float) alignment 133 | # 134 | # 'figure_align': 'htbp', 135 | } 136 | 137 | # Grouping the document tree into LaTeX files. List of tuples 138 | # (source start file, target name, title, 139 | # author, documentclass [howto, manual, or own class]). 140 | latex_documents = [ 141 | (master_doc, 'stAdv.tex', 'stAdv Documentation', 142 | 'Beranger Dumont', 'manual'), 143 | ] 144 | 145 | 146 | # -- Options for manual page output ------------------------------------------ 147 | 148 | # One entry per manual page. List of tuples 149 | # (source start file, name, description, authors, manual section). 150 | man_pages = [ 151 | (master_doc, 'stadv', 'stAdv Documentation', 152 | [author], 1) 153 | ] 154 | 155 | 156 | # -- Options for Texinfo output ---------------------------------------------- 157 | 158 | # Grouping the document tree into Texinfo files. List of tuples 159 | # (source start file, target name, title, author, 160 | # dir menu entry, description, category) 161 | texinfo_documents = [ 162 | (master_doc, 'stAdv', 'stAdv Documentation', 163 | author, 'stAdv', 'One line description of project.', 164 | 'Miscellaneous'), 165 | ] 166 | 167 | 168 | # -- Extension configuration ------------------------------------------------- -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. stAdv documentation master file, created by 2 | sphinx-quickstart on Tue Apr 17 18:41:51 2018. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | Welcome to stAdv's documentation! 7 | ================================= 8 | 9 | .. toctree:: 10 | :maxdepth: 2 11 | :caption: Contents: 12 | 13 | stadv 14 | 15 | 16 | Indices and tables 17 | ================== 18 | 19 | * :ref:`genindex` 20 | * :ref:`modindex` 21 | * :ref:`search` 22 | -------------------------------------------------------------------------------- /docs/requirements.txt: -------------------------------------------------------------------------------- 1 | # minimal requirements.txt for the Read The Docs environment 2 | # to generate the documentation 3 | 4 | # note that there is a bug in some environments for TensorFlow >= 1.6 5 | # see https://github.com/tensorflow/tensorflow/issues/17411 6 | # this was making the RTD environment crash 7 | # (same as https://github.com/rtfd/readthedocs.org/issues/3738) 8 | # so we stick to TensorFlow 1.5 here 9 | 10 | numpy 11 | scipy 12 | tensorflow==1.5 13 | 14 | sphinx 15 | sphinxcontrib-napoleon 16 | -------------------------------------------------------------------------------- /docs/stadv.rst: -------------------------------------------------------------------------------- 1 | stadv package 2 | ============= 3 | 4 | stadv.layers module 5 | ------------------- 6 | 7 | .. automodule:: stadv.layers 8 | :members: 9 | :undoc-members: 10 | :show-inheritance: 11 | 12 | stadv.losses module 13 | ------------------- 14 | 15 | .. automodule:: stadv.losses 16 | :members: 17 | :undoc-members: 18 | :show-inheritance: 19 | 20 | stadv.optimization module 21 | ------------------------- 22 | 23 | .. automodule:: stadv.optimization 24 | :members: 25 | :undoc-members: 26 | :show-inheritance: 27 | -------------------------------------------------------------------------------- /illustration-stadv-mnist.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rakutentech/stAdv/26286a8e84b61d474a958735dcff8f70d31deccc/illustration-stadv-mnist.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy 2 | scipy 3 | 4 | # TensorFlow can be installed either as CPU or GPU versions. 5 | # You select which one to install by (un)commenting these lines. 6 | tensorflow # CPU Version of TensorFlow 7 | # tensorflow-gpu # GPU version of TensorFlow 8 | 9 | # the following is needed for running the unit tests 10 | scikit-image 11 | 12 | # the following is needed for generating the documentation 13 | sphinx 14 | sphinxcontrib-napoleon 15 | 16 | # the following is needed for running the demo notebook 17 | matplotlib 18 | idx2numpy 19 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | import io 5 | import os 6 | 7 | from setuptools import setup 8 | 9 | 10 | # heavily borrows from Kenneth Reitz's A Human's Ultimate Guide to setup.py 11 | # see https://github.com/kennethreitz/setup.py 12 | 13 | here = os.path.abspath(os.path.dirname(__file__)) 14 | 15 | # Import the README and use it as the long-description. 16 | # Note: this will only work if 'README.rst' is present in your MANIFEST.in file! 17 | with io.open(os.path.join(here, 'README.rst'), encoding='utf-8') as f: 18 | long_description = '\n' + f.read() 19 | 20 | package_name = 'stadv' 21 | # Load the package's __version__.py module as a dictionary. 22 | about = {} 23 | with open(os.path.join(here, package_name, '__version__.py')) as f: 24 | exec(f.read(), about) 25 | 26 | 27 | setup( 28 | name=package_name, 29 | version=about['__version__'], 30 | description='Spatially Transformed Adversarial Examples with TensorFlow', 31 | long_description=long_description, 32 | long_description_content_type='text/x-rst', 33 | author='Beranger Dumont', 34 | author_email='beranger.dumont@rakuten.com', 35 | url='https://github.com/rakutentech/stAdv', 36 | license='MIT', 37 | packages=[package_name], 38 | python_requires='>=2.7', 39 | keywords='tensorflow adversarial examples CNN deep learning', 40 | # install_requires without tensorflow because of CPU vs. GPU install issues 41 | install_requires=['numpy', 'scipy'], 42 | classifiers=[ 43 | # Trove classifiers 44 | # Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers 45 | 'License :: OSI Approved :: MIT License', 46 | 'Development Status :: 4 - Beta', 47 | 'Intended Audience :: Developers', 48 | 'Intended Audience :: Science/Research', 49 | 'Topic :: Scientific/Engineering :: Artificial Intelligence', 50 | 'Topic :: Scientific/Engineering :: Image Recognition', 51 | 'Programming Language :: Python', 52 | 'Programming Language :: Python :: 2', 53 | 'Programming Language :: Python :: 2.7', 54 | 'Programming Language :: Python :: 3' 55 | ], 56 | project_urls={ 57 | 'Documentation': 'http://stadv.readthedocs.io/en/latest/stadv.html', 58 | 'Source': 'https://github.com/rakutentech/stAdv', 59 | 'Tracker': 'https://github.com/rakutentech/stAdv/issues' 60 | } 61 | ) 62 | -------------------------------------------------------------------------------- /stadv/__init__.py: -------------------------------------------------------------------------------- 1 | from .__version__ import __version__ 2 | from . import layers 3 | from . import losses 4 | from . import optimization 5 | -------------------------------------------------------------------------------- /stadv/__version__.py: -------------------------------------------------------------------------------- 1 | VERSION = (0, 2, 1) 2 | 3 | __version__ = '.'.join(map(str, VERSION)) 4 | -------------------------------------------------------------------------------- /stadv/layers.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | 4 | def flow_st(images, flows, data_format='NHWC'): 5 | """Flow-based spatial transformation of images. 6 | See Eq. (1) in Xiao et al. (arXiv:1801.02612). 7 | 8 | Args: 9 | images (tf.Tensor): images of shape `(B, H, W, C)` or `(B, C, H, W)` 10 | depending on `data_format`. 11 | flows (tf.Tensor): flows of shape `(B, 2, H, W)`, where the second 12 | dimension indicates the dimension on which the pixel 13 | shift is applied. 14 | data_format (str): ``'NHWC'`` or ``'NCHW'`` depending on the format of 15 | the input images and the desired output. 16 | 17 | Returns: 18 | `tf.Tensor` of the same shape and type as `images`. 19 | """ 20 | if data_format == 'NHWC': 21 | i_H = 1 22 | elif data_format == 'NCHW': 23 | i_H = 2 24 | else: 25 | raise ValueError("Provided data_format is not valid.") 26 | 27 | with tf.variable_scope('flow_st'): 28 | images_shape = tf.shape(images) 29 | flows_shape = tf.shape(flows) 30 | 31 | batch_size = images_shape[0] 32 | H = images_shape[i_H] 33 | W = images_shape[i_H + 1] 34 | 35 | # make sure that the input images and flows have consistent shape 36 | with tf.control_dependencies( 37 | [tf.assert_equal( 38 | tf.identity(images_shape[i_H:i_H + 2], name='images_shape_HW'), 39 | tf.identity(flows_shape[2:], name='flows_shape_HW') 40 | )] 41 | ): 42 | # cast the input to float32 for consistency with the rest 43 | images = tf.cast(images, 'float32', name='images_float32') 44 | flows = tf.cast(flows, 'float32', name='flows_float32') 45 | 46 | if data_format == 'NCHW': 47 | images = tf.transpose(images, [0, 2, 3, 1]) 48 | 49 | # basic grid: tensor with shape (2, H, W) with value indicating the 50 | # pixel shift in the x-axis or y-axis dimension with respect to the 51 | # original images for the pixel (2, H, W) in the output images, 52 | # before applying the flow transforms 53 | basegrid = tf.stack( 54 | tf.meshgrid(tf.range(H), tf.range(W), indexing='ij') 55 | ) 56 | 57 | # go from (2, H, W) tensors to (B, 2, H, W) tensors with simple copy 58 | # across batch dimension 59 | batched_basegrid = tf.tile([basegrid], [batch_size, 1, 1, 1]) 60 | 61 | # sampling grid is base grid + input flows 62 | sampling_grid = tf.cast(batched_basegrid, 'float32') + flows 63 | 64 | # separate shifts in x and y is easier--also we clip to the 65 | # boundaries of the image 66 | sampling_grid_x = tf.clip_by_value( 67 | sampling_grid[:, 1], 0., tf.cast(W - 1, 'float32') 68 | ) 69 | sampling_grid_y = tf.clip_by_value( 70 | sampling_grid[:, 0], 0., tf.cast(H - 1, 'float32') 71 | ) 72 | 73 | # now we need to interpolate 74 | 75 | # grab 4 nearest corner points for each (x_i, y_i) 76 | # i.e. we need a square around the point of interest 77 | x0 = tf.cast(tf.floor(sampling_grid_x), 'int32') 78 | x1 = x0 + 1 79 | y0 = tf.cast(tf.floor(sampling_grid_y), 'int32') 80 | y1 = y0 + 1 81 | 82 | # clip to range [0, H/W] to not violate image boundaries 83 | # - 2 for x0 and y0 helps avoiding black borders 84 | # (forces to interpolate between different points) 85 | x0 = tf.clip_by_value(x0, 0, W - 2, name='x0') 86 | x1 = tf.clip_by_value(x1, 0, W - 1, name='x1') 87 | y0 = tf.clip_by_value(y0, 0, H - 2, name='y0') 88 | y1 = tf.clip_by_value(y1, 0, H - 1, name='y1') 89 | 90 | # b is a (B, H, W) tensor with (B, H, W) = B for all (H, W) 91 | b = tf.tile( 92 | tf.reshape( 93 | tf.range(0, batch_size), (batch_size, 1, 1) 94 | ), 95 | (1, H, W) 96 | ) 97 | 98 | # get pixel value at corner coordinates 99 | # we stay indices along the last dimension and gather slices 100 | # given indices 101 | # the output is of shape (B, H, W, C) 102 | Ia = tf.gather_nd(images, tf.stack([b, y0, x0], 3), name='Ia') 103 | Ib = tf.gather_nd(images, tf.stack([b, y1, x0], 3), name='Ib') 104 | Ic = tf.gather_nd(images, tf.stack([b, y0, x1], 3), name='Ic') 105 | Id = tf.gather_nd(images, tf.stack([b, y1, x1], 3), name='Id') 106 | 107 | # recast as float for delta calculation 108 | x0 = tf.cast(x0, 'float32') 109 | x1 = tf.cast(x1, 'float32') 110 | y0 = tf.cast(y0, 'float32') 111 | y1 = tf.cast(y1, 'float32') 112 | 113 | # calculate deltas 114 | wa = (x1 - sampling_grid_x) * (y1 - sampling_grid_y) 115 | wb = (x1 - sampling_grid_x) * (sampling_grid_y - y0) 116 | wc = (sampling_grid_x - x0) * (y1 - sampling_grid_y) 117 | wd = (sampling_grid_x - x0) * (sampling_grid_y - y0) 118 | 119 | # add dimension for addition 120 | wa = tf.expand_dims(wa, axis=3) 121 | wb = tf.expand_dims(wb, axis=3) 122 | wc = tf.expand_dims(wc, axis=3) 123 | wd = tf.expand_dims(wd, axis=3) 124 | 125 | # compute output 126 | perturbed_image = tf.add_n([wa * Ia, wb * Ib, wc * Ic, wd * Id]) 127 | 128 | if data_format == 'NCHW': 129 | # convert back to NCHW to have consistency with the input 130 | perturbed_image = tf.transpose(perturbed_image, [0, 3, 1, 2]) 131 | 132 | return perturbed_image 133 | -------------------------------------------------------------------------------- /stadv/losses.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | 4 | def flow_loss(flows, padding_mode='SYMMETRIC', epsilon=1e-8): 5 | """Computes the flow loss designed to "enforce the locally smooth 6 | spatial transformation perturbation". See Eq. (4) in Xiao et al. 7 | (arXiv:1801.02612). 8 | 9 | Args: 10 | flows (tf.Tensor): flows of shape `(B, 2, H, W)`, where the second 11 | dimension indicates the dimension on which the pixel 12 | shift is applied. 13 | padding_mode (str): how to perform padding of the boundaries of the 14 | images. The value should be compatible with the 15 | `mode` argument of ``tf.pad``. Expected values are: 16 | 17 | * ``'SYMMETRIC'``: symmetric padding so as to not 18 | penalize a significant flow at the boundary of 19 | the images; 20 | * ``'CONSTANT'``: 0-padding of the boundaries so as 21 | to enforce a small flow at the boundary of the 22 | images. 23 | epsilon (float): small value added to the argument of ``tf.sqrt`` 24 | to prevent NaN gradients when the argument is zero. 25 | 26 | Returns: 27 | 1-D `tf.Tensor` of length `B` of the same type as `flows`. 28 | """ 29 | with tf.variable_scope('flow_loss'): 30 | # following the notation from Eq. (4): 31 | # \Delta u^{(p)} is flows[:, 1], 32 | # \Delta v^{(p)} is flows[:, 0], and 33 | # \Delta u^{(q)} is flows[:, 1] shifted by 34 | # (+1, +1), (+1, -1), (-1, +1), or (-1, -1) pixels 35 | # and \Delta v^{(q)} is the same but for shifted flows[:, 0] 36 | 37 | paddings = tf.constant([[0, 0], [0, 0], [1, 1], [1, 1]]) 38 | padded_flows = tf.pad( 39 | flows, paddings, padding_mode, constant_values=0, 40 | name='padded_flows' 41 | ) 42 | 43 | shifted_flows = [ 44 | padded_flows[:, :, 2:, 2:], # bottom right 45 | padded_flows[:, :, 2:, :-2], # bottom left 46 | padded_flows[:, :, :-2, 2:], # top right 47 | padded_flows[:, :, :-2, :-2] # top left 48 | ] 49 | 50 | return tf.reduce_sum( 51 | tf.add_n( 52 | [ 53 | tf.sqrt( 54 | # ||\Delta u^{(p)} - \Delta u^{(q)}||_2^2 55 | (flows[:, 1] - shifted_flow[:, 1]) ** 2 + 56 | # ||\Delta v^{(p)} - \Delta v^{(q)}||_2^2 57 | (flows[:, 0] - shifted_flow[:, 0]) ** 2 + 58 | epsilon # for numerical stability 59 | ) 60 | for shifted_flow in shifted_flows 61 | ] 62 | ), axis=[1, 2], name='L_flow' 63 | ) 64 | 65 | def adv_loss(unscaled_logits, targets, kappa=None): 66 | """Computes the adversarial loss. 67 | It was first suggested by Carlini and Wagner (arXiv:1608.04644). 68 | See also Eq. (3) in Xiao et al. (arXiv:1801.02612). 69 | 70 | Args: 71 | unscaled_logits (tf.Tensor): logits of shape `(B, K)`, where `K` is the 72 | number of input classes. 73 | targets (tf.Tensor): `1-D` integer-encoded targets of length `B` with 74 | value corresponding to the class ID. 75 | kappa (tf.Tensor): confidence parameter, see Carlini and Wagner 76 | (arXiv:1608.04644). Defaults to 0. 77 | 78 | Returns: 79 | 1-D `tf.Tensor` of length `B` of the same type as `unscaled_logits`. 80 | """ 81 | if kappa is None: 82 | kappa = tf.constant(0., dtype=unscaled_logits.dtype, name='kappa') 83 | 84 | with tf.variable_scope('adv_loss'): 85 | unscaled_logits_shape = tf.shape(unscaled_logits) 86 | B = unscaled_logits_shape[0] 87 | K = unscaled_logits_shape[1] 88 | 89 | # first term in L_adv: maximum of the (unscaled) logits except target 90 | mask = tf.one_hot( 91 | targets, 92 | depth=K, 93 | on_value=False, 94 | off_value=True, 95 | dtype='bool' 96 | ) 97 | logit_wout_target = tf.reshape( 98 | tf.boolean_mask(unscaled_logits, mask), 99 | (B, K - 1), 100 | name='logit_wout_target' 101 | ) 102 | L_adv_1 = tf.reduce_max(logit_wout_target, axis=1, name='L_adv_1') 103 | 104 | # second term in L_adv: value of the unscaled logit corresponding to the 105 | # target 106 | L_adv_2 = tf.diag_part( 107 | tf.gather(unscaled_logits, targets, axis=1), name='L_adv_2' 108 | ) 109 | 110 | return tf.maximum(L_adv_1 - L_adv_2, - kappa, name='L_adv') 111 | -------------------------------------------------------------------------------- /stadv/optimization.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import numpy as np 3 | from scipy.optimize import fmin_l_bfgs_b 4 | 5 | 6 | def lbfgs( 7 | loss, flows, flows_x0, feed_dict=None, grad_op=None, 8 | fmin_l_bfgs_b_extra_kwargs=None, sess=None 9 | ): 10 | """Optimize a given loss with (SciPy's external) L-BFGS-B optimizer. 11 | It can be used to solve the optimization problem of Eq. (2) in Xiao et al. 12 | (arXiv:1801.02612). 13 | See `the documentation on scipy.optimize.fmin_l_bfgs_b 14 | `_ 15 | for reference on the optimizer. 16 | 17 | 18 | Args: 19 | loss (tf.Tensor): loss (can be of any shape). 20 | flows (tf.Tensor): flows of shape `(B, 2, H, W)`, where the second 21 | dimension indicates the dimension on which the pixel 22 | shift is applied. 23 | flows_x0 (np.ndarray): Initial guess for the flows. If the input is not 24 | of type `np.ndarray`, it will be converted as 25 | such if possible. 26 | feed_dict (dict): feed dictionary to the ``tf.run`` operation (for 27 | everything which might be needed to execute the graph 28 | beyond the input flows). 29 | grad_op (tf.Tensor): gradient of the loss with respect to the flows. 30 | If not provided it will be computed from the input 31 | and added to the graph. 32 | fmin_l_bfgs_b_extra_kwargs (dict): extra arguments to 33 | ``scipy.optimize.fmin_l_bfgs_b`` 34 | (e.g. for modifying the stopping 35 | condition). 36 | sess (tf.Session): session within which the graph should be executed. 37 | If not provided a new session will be started. 38 | 39 | Returns: 40 | `Dictionary` with keys ``'flows'`` (`np.ndarray`, estimated flows of the 41 | minimum), ``'loss'`` (`float`, value of loss at the minimum), and 42 | ``'info'`` (`dict`, information summary as returned by 43 | ``scipy.optimize.fmin_l_bfgs_b``). 44 | """ 45 | def tf_run(x): 46 | """Function to minimize as provided to ``scipy.optimize.fmin_l_bfgs_b``. 47 | 48 | Args: 49 | x (np.ndarray): current flows proposal at a given stage of the 50 | optimization (flattened `np.ndarray` of type 51 | `np.float64` as required by the backend FORTRAN 52 | implementation of L-BFGS-B). 53 | 54 | Returns: 55 | `Tuple` `(loss, loss_gradient)` of type `np.float64` as required 56 | by the backend FORTRAN implementation of L-BFGS-B. 57 | """ 58 | flows_val = np.reshape(x, flows_shape) 59 | 60 | feed_dict.update({flows: flows_val}) 61 | loss_val, gradient_val = sess_.run( 62 | [loss, loss_gradient], feed_dict=feed_dict 63 | ) 64 | loss_val = np.sum(loss_val).astype(np.float64) 65 | gradient_val = gradient_val.flatten().astype(np.float64) 66 | 67 | return loss_val, gradient_val 68 | 69 | flows_x0 = np.asarray(flows_x0, dtype=np.float64) 70 | flows_shape = flows_x0.shape 71 | 72 | if feed_dict is None: 73 | feed_dict = {} 74 | if fmin_l_bfgs_b_extra_kwargs is None: 75 | fmin_l_bfgs_b_extra_kwargs = {} 76 | 77 | fmin_l_bfgs_b_kwargs = { 78 | 'func': tf_run, 79 | 'approx_grad': False, # we want to use the gradients from TensorFlow 80 | 'fprime': None, 81 | 'args': () 82 | } 83 | 84 | for key in fmin_l_bfgs_b_extra_kwargs.keys(): 85 | if key in fmin_l_bfgs_b_kwargs: 86 | raise ValueError( 87 | "The argument " + str(key) + " should not be overwritten by " 88 | "fmin_l_bfgs_b_extra_kwargs" 89 | ) 90 | 91 | # define the default extra arguments to fmin_l_bfgs_b 92 | default_extra_kwargs = { 93 | 'x0': flows_x0.flatten(), 94 | 'factr': 10.0, 95 | 'm': 20, 96 | 'iprint': -1 97 | } 98 | 99 | fmin_l_bfgs_b_kwargs.update(default_extra_kwargs) 100 | fmin_l_bfgs_b_kwargs.update(fmin_l_bfgs_b_extra_kwargs) 101 | 102 | if grad_op is not None: 103 | loss_gradient = grad_op 104 | else: 105 | loss_gradient = tf.gradients(loss, flows, name='loss_gradient')[0] 106 | if loss_gradient is None: 107 | raise ValueError( 108 | "Cannot compute the gradient d(loss)/d(flows). Is the graph " 109 | "really differentiable?" 110 | ) 111 | 112 | sess_ = tf.Session() if sess is None else sess 113 | raw_results = fmin_l_bfgs_b(**fmin_l_bfgs_b_kwargs) 114 | if sess is None: 115 | sess_.close() 116 | 117 | return { 118 | 'flows': np.reshape(raw_results[0], flows_shape), 119 | 'loss': raw_results[1], 120 | 'info': raw_results[2] 121 | } 122 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/rakutentech/stAdv/26286a8e84b61d474a958735dcff8f70d31deccc/tests/__init__.py -------------------------------------------------------------------------------- /tests/context.py: -------------------------------------------------------------------------------- 1 | """Load the stadv package (try to do so explicitly) to be agnostic of the 2 | installation status of the package.""" 3 | 4 | import sys 5 | import os 6 | sys.path.insert( 7 | 0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')) 8 | ) 9 | 10 | import stadv 11 | 12 | # next is a trick to have unit tests run for TensorFlow < 1.7, when the msg 13 | # argument was not always present in the assert methods of tf.test.TestCase 14 | import inspect 15 | 16 | def call_assert(f, *args, **kwargs): 17 | if 'msg' not in inspect.getargspec(f) and 'msg' in kwargs.keys(): 18 | kwargs.pop('msg') 19 | return f(*args, **kwargs) 20 | -------------------------------------------------------------------------------- /tests/test_layers.py: -------------------------------------------------------------------------------- 1 | from .context import stadv, call_assert 2 | 3 | import tensorflow as tf 4 | import numpy as np 5 | import skimage.transform 6 | 7 | 8 | class FlowStCase(tf.test.TestCase): 9 | """Test the flow_st layer.""" 10 | 11 | def setUp(self): 12 | np.random.seed(0) # predictable random numbers 13 | 14 | # dimensions of the input images -- "random" shape for generic cases 15 | self.N, self.H, self.W, self.C = 2, 7, 6, 2 16 | self.data_formats = { 17 | 'NHWC': (self.N, self.H, self.W, self.C), 18 | 'NCHW': (self.N, self.C, self.H, self.W) 19 | } 20 | 21 | # generate test images and flows 22 | self.test_images = {} 23 | self.test_images_float = {} 24 | for data_format, dim_tuple in self.data_formats.items(): 25 | self.test_images[data_format] = np.random.randint(0, 256, dim_tuple) 26 | self.test_images_float[data_format] = ( 27 | self.test_images[data_format] / 255. 28 | ) 29 | flow_shape = (self.N, 2, self.H, self.W) 30 | self.flow_zero = np.zeros(flow_shape, dtype=int) 31 | self.flow_random = np.random.random_sample(flow_shape) - 0.5 32 | 33 | # building a minimal graph 34 | # setting None shape in both placeholders because we want to test 35 | # a number of possible shape issues 36 | self.images = tf.placeholder(tf.float32, shape=None, name='images') 37 | self.flows = tf.placeholder(tf.float32, shape=None, name='flows') 38 | self.outputs = {} 39 | for data_format in self.data_formats.keys(): 40 | self.outputs[data_format] = stadv.layers.flow_st( 41 | self.images, self.flows, data_format 42 | ) 43 | 44 | def test_output_shape_consistency(self): 45 | """Check that the input images and output images shapes are the same.""" 46 | with self.test_session(): 47 | for data_format, output in self.outputs.items(): 48 | call_assert( 49 | self.assertEqual, 50 | output.eval(feed_dict={ 51 | self.images: self.test_images[data_format], 52 | self.flows: self.flow_random 53 | }).shape, 54 | self.test_images[data_format].shape, 55 | msg='the output shape differs from the input one for shape ' 56 | + data_format 57 | ) 58 | 59 | def test_mismatch_flow_shape_image_shape(self): 60 | """Make sure that input flows with a wrong shape raise the expected 61 | Exception.""" 62 | flow_zeros_wrongdim1 = np.zeros( 63 | (self.N, 2, self.H + 1, self.W), dtype=int 64 | ) 65 | flow_zeros_wrongdim2 = np.zeros( 66 | (self.N, 2, self.H, self.W - 1), dtype=int 67 | ) 68 | 69 | with self.test_session(): 70 | for data_format, output in self.outputs.items(): 71 | with self.assertRaises(tf.errors.InvalidArgumentError): 72 | output.eval(feed_dict={ 73 | self.images: self.test_images[data_format], 74 | self.flows: flow_zeros_wrongdim1 75 | }) 76 | output.eval(feed_dict={ 77 | self.images: self.test_images[data_format], 78 | self.flows: flow_zeros_wrongdim2 79 | }) 80 | 81 | def test_noflow_consistency(self): 82 | """Check that no flow displacement gives the input image.""" 83 | with self.test_session(): 84 | for data_format, output in self.outputs.items(): 85 | call_assert( 86 | self.assertAllEqual, 87 | output.eval(feed_dict={ 88 | self.images: self.test_images[data_format], 89 | self.flows: self.flow_zero 90 | }), 91 | self.test_images[data_format], 92 | msg='output differs from input in spite of 0 displacement ' 93 | 'flow for shape ' + data_format + ' and int input' 94 | ) 95 | 96 | def test_noflow_consistency_float_image(self): 97 | """Check that no flow displacement gives the float input image.""" 98 | with self.test_session(): 99 | for data_format, output in self.outputs.items(): 100 | call_assert( 101 | self.assertAllClose, 102 | output.eval(feed_dict={ 103 | self.images: self.test_images_float[data_format], 104 | self.flows: self.flow_zero 105 | }), 106 | self.test_images_float[data_format], 107 | msg='output differs from input in spite of 0 displacement ' 108 | 'flow for shape ' + data_format + ' and float input' 109 | ) 110 | 111 | def test_min_max_output_consistency(self): 112 | """Test that the min and max values in the output do not exceed the 113 | min and max values in the input images for a random flow.""" 114 | with self.test_session(): 115 | # test both data formats 116 | for data_format, output in self.outputs.items(): 117 | in_images = self.test_images[data_format] 118 | out_images = output.eval( 119 | feed_dict={ 120 | self.images: in_images, 121 | self.flows: self.flow_random 122 | } 123 | ) 124 | 125 | # derive min and max values for every image in the batch 126 | # for the input and output 127 | taxis = (1,2,3) 128 | minval_in = np.amin(in_images, axis=taxis) 129 | maxval_in = np.amax(in_images, axis=taxis) 130 | minval_out = np.amin(out_images, axis=taxis) 131 | maxval_out = np.amax(out_images, axis=taxis) 132 | 133 | call_assert( 134 | self.assertTrue, 135 | np.all(np.less_equal(minval_in, minval_out)), 136 | msg='min value in output image less than min value in input' 137 | ) 138 | call_assert( 139 | self.assertTrue, 140 | np.all(np.greater_equal(maxval_in, maxval_out)), 141 | msg='max value in output image exceeds max value in input' 142 | ) 143 | 144 | def test_bilinear_interpolation(self): 145 | """Test that the home-made bilinear interpolation matches the one from 146 | scikit-image.""" 147 | data_format = 'NHWC' 148 | in_image = self.test_images[data_format][0] 149 | 150 | translation_x = 1.5 151 | translation_y = 0.8 152 | 153 | # define the transformation with scikit-image 154 | tform = skimage.transform.EuclideanTransform( 155 | translation=(translation_x, translation_y) 156 | ) 157 | 158 | skimage_out = skimage.transform.warp( 159 | in_image / 255., tform, order=1 160 | ) * 255. 161 | 162 | # do the same with our tool 163 | constant_flow_1 = np.zeros((self.H, self.W)) + translation_y 164 | constant_flow_2 = np.zeros((self.H, self.W)) + translation_x 165 | stacked_flow = np.stack([constant_flow_1, constant_flow_2], axis=0) 166 | final_flow = np.expand_dims(stacked_flow, axis=0) 167 | with self.test_session(): 168 | tf_out = self.outputs[data_format].eval( 169 | feed_dict={ 170 | self.images: np.expand_dims(in_image, axis=0), 171 | self.flows: final_flow 172 | } 173 | )[0] 174 | 175 | # we only want to check equality up to boundary effects 176 | cut_x = - np.ceil(translation_x).astype(int) 177 | cut_y = - np.ceil(translation_y).astype(int) 178 | skimage_out_crop = skimage_out[:cut_y, :cut_x] 179 | tf_out_crop = tf_out[:cut_y, :cut_x] 180 | 181 | call_assert( 182 | self.assertAllClose, 183 | tf_out_crop, skimage_out_crop, 184 | msg='bilinear interpolation differs from scikit-image' 185 | ) 186 | 187 | if __name__ == '__main__': 188 | tf.test.main() 189 | -------------------------------------------------------------------------------- /tests/test_losses.py: -------------------------------------------------------------------------------- 1 | from .context import stadv, call_assert 2 | 3 | import tensorflow as tf 4 | import numpy as np 5 | 6 | 7 | class FlowLossCase(tf.test.TestCase): 8 | """Test the flow_loss loss function.""" 9 | 10 | def setUp(self): 11 | np.random.seed(0) 12 | # dimensions of the flow "random" shape for generic cases 13 | self.N, self.H, self.W = 2, 7, 6 14 | flow_shape = (self.N, 2, self.H, self.W) 15 | self.flow_zero = np.zeros(flow_shape, dtype=int) 16 | 17 | self.flows = tf.placeholder(tf.float32, shape=None, name='flows') 18 | 19 | self.loss_symmetric = stadv.losses.flow_loss( 20 | self.flows, 'SYMMETRIC', epsilon=0. 21 | ) 22 | self.loss_constant = stadv.losses.flow_loss( 23 | self.flows, 'CONSTANT', epsilon=0. 24 | ) 25 | self.loss_symmetric_eps = stadv.losses.flow_loss( 26 | self.flows, 'SYMMETRIC', epsilon=1e-8 27 | ) 28 | 29 | def test_zero_flow(self): 30 | """Make sure that null flows (all 0) gives a flow loss of 0.""" 31 | with self.test_session(): 32 | loss_symmetric = self.loss_symmetric.eval(feed_dict={ 33 | self.flows: self.flow_zero 34 | }) 35 | loss_constant = self.loss_constant.eval(feed_dict={ 36 | self.flows: self.flow_zero 37 | }) 38 | 39 | call_assert( 40 | self.assertAllClose, 41 | loss_symmetric, np.zeros(loss_symmetric.shape), 42 | msg='0 flow with symmetric padding gives != 0 loss' 43 | ) 44 | call_assert( 45 | self.assertAllClose, 46 | loss_constant, np.zeros(loss_constant.shape), 47 | msg='0 flow with constant padding gives != 0 loss' 48 | ) 49 | 50 | def test_constant_flow(self): 51 | """Make sure that a constant flow gives 0 loss for symmetric padding.""" 52 | with self.test_session(): 53 | custom_flow = self.flow_zero + 4.3 54 | loss_symmetric = self.loss_symmetric.eval(feed_dict={ 55 | self.flows: custom_flow 56 | }) 57 | 58 | call_assert( 59 | self.assertAllClose, 60 | np.amax(loss_symmetric), 0., 61 | msg='constant flow with symmetric padding gives > 0 loss' 62 | ) 63 | 64 | def test_manual_calculation_symmetric(self): 65 | custom_flow = np.random.random(self.flow_zero.shape) 66 | 67 | # manual calculation (looping over pixels) 68 | result = [] 69 | for img_flow in custom_flow: 70 | loss = 0. 71 | max_i = img_flow.shape[1] - 1 72 | max_j = img_flow.shape[2] - 1 73 | for i in range(max_i + 1): 74 | i_corner1 = i - 1 if i > 0 else 0 75 | i_corner2 = i + 1 if i < max_i else max_i 76 | for j in range(max_j + 1): 77 | j_corner1 = j - 1 if j > 0 else 0 78 | j_corner2 = j + 1 if j < max_j else max_j 79 | 80 | for (i_coord, j_coord) in [ 81 | (i_corner1, j_corner1), 82 | (i_corner1, j_corner2), 83 | (i_corner2, j_corner1), 84 | (i_corner2, j_corner2) 85 | ]: 86 | loss += np.sqrt( 87 | ( 88 | img_flow[0, i, j] - 89 | img_flow[0, i_coord, j_coord] 90 | ) ** 2 + 91 | ( 92 | img_flow[1, i, j] - 93 | img_flow[1, i_coord, j_coord] 94 | ) ** 2 95 | + 1e-8 96 | ) 97 | result.append(loss) 98 | result = np.array(result) 99 | 100 | with self.test_session(): 101 | loss_symmetric = self.loss_symmetric_eps.eval(feed_dict={ 102 | self.flows: custom_flow 103 | }) 104 | call_assert( 105 | self.assertAllClose, 106 | result, loss_symmetric, 107 | msg='L_flow does not match manual calculation in symmetric case' 108 | ) 109 | 110 | class AdvLossCase(tf.test.TestCase): 111 | """Test the adv_loss loss function.""" 112 | 113 | def setUp(self): 114 | self.unscaled_logits = tf.placeholder( 115 | tf.float32, shape=None, name='unscaled_logits' 116 | ) 117 | self.targets = tf.placeholder( 118 | tf.int32, shape=[None], name='targets' 119 | ) 120 | self.kappa = tf.placeholder(tf.float32, shape=[]) 121 | self.loss = stadv.losses.adv_loss( 122 | self.unscaled_logits, self.targets, self.kappa 123 | ) 124 | self.loss_default_kappa = stadv.losses.adv_loss( 125 | self.unscaled_logits, self.targets 126 | ) 127 | 128 | def test_numerical_correctness_with_example(self): 129 | """Test numerical correctness for a concrete case.""" 130 | unscaled_logits_example = np.array( 131 | [[-20.3, 4.7, 5.8, 7.2], [77.5, -0.2, 9.2, -12.0]] 132 | ) 133 | targets_example = np.array([2, 0]) 134 | 135 | # first term in loss is expected to be [7.2 9.2] 136 | # second term in loss is expected to be [5.8 77.5] 137 | # final loss is expected to be [max(1.4, -kappa) max(-68.3, -kappa) 138 | # i.e., for kappa=0: [1.4 0], for kappa=10: [1.4 -10] 139 | expected_result_kappa_0 = np.array([1.4, 0.]) 140 | expected_result_kappa_10 = np.array([1.4, -10.]) 141 | 142 | with self.test_session(): 143 | loss_kappa_0 = self.loss.eval(feed_dict={ 144 | self.unscaled_logits: unscaled_logits_example, 145 | self.targets: targets_example, 146 | self.kappa: 0. 147 | }) 148 | val_loss_default_kappa = self.loss_default_kappa.eval(feed_dict={ 149 | self.unscaled_logits: unscaled_logits_example, 150 | self.targets: targets_example 151 | }) 152 | loss_kappa_10 = self.loss.eval(feed_dict={ 153 | self.unscaled_logits: unscaled_logits_example, 154 | self.targets: targets_example, 155 | self.kappa: 10. 156 | }) 157 | 158 | call_assert( 159 | self.assertAllClose, 160 | val_loss_default_kappa, loss_kappa_0, 161 | msg='default kappa argument differs from setting kappa=0', 162 | ) 163 | call_assert( 164 | self.assertAllClose, 165 | loss_kappa_0, expected_result_kappa_0, 166 | msg='wrong loss for kappa=0' 167 | ) 168 | call_assert( 169 | self.assertAllClose, 170 | loss_kappa_10, expected_result_kappa_10, 171 | msg='wrong loss for kappa=10' 172 | ) 173 | 174 | if __name__ == '__main__': 175 | tf.test.main() 176 | -------------------------------------------------------------------------------- /tests/test_optimization.py: -------------------------------------------------------------------------------- 1 | from .context import stadv, call_assert 2 | 3 | import tensorflow as tf 4 | import numpy as np 5 | 6 | 7 | class LBFGSCase(tf.test.TestCase): 8 | """Test the lbfgs optimization function. 9 | Note: we are NOT testing the LBFGS implementation from SciPy, instead we 10 | test our wrapping and its interplay with TensorFlow.""" 11 | 12 | def setUp(self): 13 | self.example_flow = np.array([[0.5, 0.4], [-0.2, 0.7]]) 14 | self.flows = tf.Variable(self.example_flow, name='flows') 15 | self.loss_l2 = tf.reduce_sum(tf.square(self.flows), name='loss_l2') 16 | self.loss_dummy = tf.Variable(1.4, name='loss_dummy') 17 | 18 | tf.global_variables_initializer() 19 | 20 | def test_l2_norm_loss(self): 21 | """Check that simple L2 loss leads to 0 loss and gradient in the end.""" 22 | results = stadv.optimization.lbfgs( 23 | self.loss_l2, self.flows, flows_x0=self.example_flow 24 | ) 25 | call_assert( 26 | self.assertEqual, 27 | results['flows'].shape, self.example_flow.shape, 28 | msg='initial and optimized flows have a different shape' 29 | ) 30 | call_assert( 31 | self.assertAllClose, 32 | results['flows'], np.zeros(results['flows'].shape), 33 | msg='optimized flows significantly differ from 0' 34 | ) 35 | call_assert( 36 | self.assertAllClose, 37 | results['loss'], np.zeros(results['loss'].shape), 38 | msg='final gradients significantly differ from 0' 39 | ) 40 | 41 | def test_dummy_loss(self): 42 | """Make sure a dummy loss (no computable gradient) gives an error.""" 43 | with self.assertRaises(ValueError): 44 | stadv.optimization.lbfgs( 45 | self.loss_dummy, self.flows, flows_x0=self.example_flow 46 | ) 47 | 48 | def test_overwriting_optimized_function(self): 49 | """Make sure we cannot overwrite argument defining the function to 50 | optimize.""" 51 | with self.assertRaises(ValueError): 52 | stadv.optimization.lbfgs( 53 | self.loss_dummy, self.flows, flows_x0=self.example_flow, 54 | fmin_l_bfgs_b_extra_kwargs={'func': np.sqrt} 55 | ) 56 | 57 | if __name__ == '__main__': 58 | tf.test.main() 59 | --------------------------------------------------------------------------------