├── .gitignore
├── ml-basics
└── mnist
│ ├── README.md
│ ├── mnist-loading-basic-steps.ipynb
│ ├── mnist-incorrect-and-abiguous-labels.ipynb
│ └── mnist.ipynb
├── examples
├── adversarial-examples
│ ├── lion.jpg
│ ├── ostrich.jpg
│ ├── adversarial_example.ipynb
│ ├── adversarial_examples.md
│ └── imagenet_class_index.json
├── inversion-attack
│ └── att-database-of-faces.zip
├── model-stealing
│ └── model_stealing_logistic_regression.ipynb
└── backdoors
│ └── mnist.ipynb
├── LICENSE
├── requirements.txt
├── README.md
└── inversion.ipynb
/.gitignore:
--------------------------------------------------------------------------------
1 | .ipynb_checkpoints/
2 | .data
3 | venv
4 |
--------------------------------------------------------------------------------
/ml-basics/mnist/README.md:
--------------------------------------------------------------------------------
1 | # INSTALL
2 |
3 | pip install torch torchvision matplotlib
--------------------------------------------------------------------------------
/examples/adversarial-examples/lion.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/daniel-e/secml/HEAD/examples/adversarial-examples/lion.jpg
--------------------------------------------------------------------------------
/examples/adversarial-examples/ostrich.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/daniel-e/secml/HEAD/examples/adversarial-examples/ostrich.jpg
--------------------------------------------------------------------------------
/examples/inversion-attack/att-database-of-faces.zip:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/daniel-e/secml/HEAD/examples/inversion-attack/att-database-of-faces.zip
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | MIT License
2 |
3 | Copyright (c) 2018
4 |
5 | Permission is hereby granted, free of charge, to any person obtaining a copy
6 | of this software and associated documentation files (the "Software"), to deal
7 | in the Software without restriction, including without limitation the rights
8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 |
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 |
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 |
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | attrs==19.1.0
2 | backcall==0.1.0
3 | bleach==3.3.0
4 | cycler==0.10.0
5 | decorator==4.4.0
6 | defusedxml==0.6.0
7 | entrypoints==0.3
8 | ipykernel==5.1.1
9 | ipython==8.10.0
10 | ipython-genutils==0.2.0
11 | ipywidgets==7.4.2
12 | jedi==0.14.0
13 | Jinja2==2.11.3
14 | jsonschema==3.0.1
15 | jupyter==1.0.0
16 | jupyter-client==5.2.4
17 | jupyter-console==6.0.0
18 | jupyter-core==4.11.2
19 | kiwisolver==1.1.0
20 | MarkupSafe==1.1.1
21 | matplotlib==3.1.0
22 | mistune==2.0.3
23 | nbconvert==6.5.1
24 | nbformat==4.4.0
25 | notebook==6.4.12
26 | numpy==1.22.0
27 | pandocfilters==1.4.2
28 | parso==0.5.0
29 | pexpect==4.7.0
30 | pickleshare==0.7.5
31 | Pillow==9.0.1
32 | pkg-resources==0.0.0
33 | prometheus-client==0.7.1
34 | prompt-toolkit==2.0.9
35 | ptyprocess==0.6.0
36 | Pygments==2.7.4
37 | pyparsing==2.4.0
38 | pyrsistent==0.15.2
39 | python-dateutil==2.8.0
40 | pyzmq==18.0.1
41 | qtconsole==4.5.1
42 | Send2Trash==1.5.0
43 | six==1.12.0
44 | terminado==0.8.2
45 | testpath==0.4.2
46 | torch==1.13.1
47 | torchvision==0.3.0
48 | tornado==6.0.3
49 | tqdm==4.32.2
50 | traitlets==4.3.2
51 | wcwidth==0.1.7
52 | webencodings==0.5.1
53 | widgetsnbextension==3.4.2
54 |
--------------------------------------------------------------------------------
/ml-basics/mnist/mnist-loading-basic-steps.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# Loading MNIST Digits in Batches With PyTorch"
8 | ]
9 | },
10 | {
11 | "cell_type": "code",
12 | "execution_count": null,
13 | "metadata": {},
14 | "outputs": [],
15 | "source": [
16 | "from torch import utils\n",
17 | "from torchvision import datasets, transforms\n",
18 | "\n",
19 | "# Transform PIL image into a tensor. The values are in the range [0, 1]\n",
20 | "t = transforms.ToTensor()\n",
21 | "\n",
22 | "# Load datasets for training and apply the given transformation.\n",
23 | "mnist = datasets.MNIST(root='data', train=True, download=True, transform=t)\n",
24 | "\n",
25 | "# Specify a data loader which returns 500 examples in each iteration.\n",
26 | "n = 500\n",
27 | "loader = utils.data.DataLoader(mnist, batch_size=n, shuffle=True)\n",
28 | "\n",
29 | "# Iterate over the batches.\n",
30 | "for imgs, labels in loader:\n",
31 | " # do something"
32 | ]
33 | }
34 | ],
35 | "metadata": {
36 | "interpreter": {
37 | "hash": "3c31154c2e6d078d13498c87eb48ca372ee3ad3d9153e56081b43cdb07df7cf4"
38 | },
39 | "kernelspec": {
40 | "display_name": "Python 3.8.10 64-bit ('venv': venv)",
41 | "language": "python",
42 | "name": "python3"
43 | },
44 | "language_info": {
45 | "codemirror_mode": {
46 | "name": "ipython",
47 | "version": 3
48 | },
49 | "file_extension": ".py",
50 | "mimetype": "text/x-python",
51 | "name": "python",
52 | "nbconvert_exporter": "python",
53 | "pygments_lexer": "ipython3",
54 | "version": "3.8.10"
55 | },
56 | "orig_nbformat": 4
57 | },
58 | "nbformat": 4,
59 | "nbformat_minor": 2
60 | }
61 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Introduction
2 |
3 | Like software systems also machine learning can suffer from security weaknesses. This repository contains some resources to provide an overview.
4 |
5 | ## Possible Security Issues in Machine Learning
6 |
7 | ### Poisoning
8 |
9 | In a poisoning attack an adversary can insert carefully crafted examples into the training data. Hence, this attack happens at training time. First, such an attack could degrade the performance of a machine learning model (the adversary targets the availability of the model). Second, the adversary could use this technique to inject a backdoor.
10 |
11 | #### Resources
12 |
13 | * [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://arxiv.org/pdf/1712.05526), 2017
14 | * [Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning](https://arxiv.org/pdf/1804.00308.pdf), 2018
15 | * [Data poisoning attacks against online learning](https://arxiv.org/abs/1808.08994), 2018
16 | * [Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks](https://arxiv.org/abs/1809.02861), 2018
17 |
18 | ### Evasion
19 |
20 | An evasion attack happens at test time. Here, an instance that would be classified correctly without modification, will be misclassified when small modifications are added by the adversary. A well know example of evasion attacks are adversarial examples. An adversary adds small perturbations to an image which are invisible to a human but will fool the image classifier which will misclassify them into a category that can be chosen by the adversary.
21 |
22 | #### Resources
23 |
24 | * [Intriguing properties of neural networks](https://arxiv.org/abs/1312.6199), 2014
25 | * [Explaining and Harnessing Adversarial Examples](https://arxiv.org/abs/1412.6572), 2014
26 | * [Evasion Attacks against Machine Learning at Test Time](https://arxiv.org/abs/1708.06131), 2017
27 | * [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://arxiv.org/abs/1705.07263), 2017
28 |
29 | ### Inversion Attacks
30 |
31 | In inversion attacks an adversary tries to extract useful information from a machine learning model like training data that was used for the training. Due to the fact that machine learning is used in more and more privacy sensitive applications an adversary could learn very sensitive data of individuals.
32 |
33 | #### Resources
34 |
35 | * [Privacy in Pharmacogenetics: An End-to-End Case Study of Personalized Warfarin Dosing](https://www.usenix.org/system/files/conference/usenixsecurity14/sec14-paper-fredrikson-privacy.pdf), 2014
36 | * [Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures](https://www.cs.cmu.edu/~mfredrik/papers/fjr2015ccs.pdf), 2015
37 | * [Membership Model Inversion Attacks for Deep Networks](https://arxiv.org/abs/1910.04257), 2019
38 | * [The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks](https://arxiv.org/abs/1911.07135), 2019
39 |
40 | # Examples
41 |
42 | This repositories contains some examples of attacks in the folder `example`. Examples are:
43 |
44 | * Create adversarial examples
45 | * Model stealing
46 | * Model inversion attacks
47 |
48 | To run the examples it is recommended to create a virtual environment first and install all required packages in that environment:
49 |
50 | virtualenv -p python3 venv
51 | source venv/bin/activate
52 | pip3 install -r requirements.txt
53 |
54 | # Resources
55 |
56 | * [TensorFlow Privacy](https://github.com/tensorflow/privacy)
57 | * [PySyft](https://github.com/OpenMined/PySyft)
58 | * [Encrypted Training Demo on MNIST](https://blog.openmined.org/encrypted-training-on-mnist/)
59 |
60 |
--------------------------------------------------------------------------------
/inversion.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "import torch\n",
10 | "import torchvision as tv\n",
11 | "import matplotlib.pyplot as plt\n",
12 | "from PIL import Image\n",
13 | "import numpy as np"
14 | ]
15 | },
16 | {
17 | "cell_type": "code",
18 | "execution_count": null,
19 | "metadata": {},
20 | "outputs": [],
21 | "source": [
22 | "%%bash\n",
23 | "\n",
24 | "rm -rf .data\n",
25 | "mkdir .data\n",
26 | "unzip -q att-database-of-faces.zip -d .data/faces-training\n",
27 | "cp -a .data/faces-training .data/faces-test\n",
28 | "\n",
29 | "rm .data/faces-training/*/{1,2,3,4,5}.pgm\n",
30 | "rm .data/faces-test/*/{6,7,8,9,10}.pgm"
31 | ]
32 | },
33 | {
34 | "cell_type": "code",
35 | "execution_count": null,
36 | "metadata": {},
37 | "outputs": [],
38 | "source": [
39 | "torch.manual_seed(1)\n",
40 | "\n",
41 | "t = tv.transforms.Compose([\n",
42 | " tv.transforms.Grayscale(),\n",
43 | " tv.transforms.ToTensor()\n",
44 | "])\n",
45 | "\n",
46 | "# Load AT&T database of faces.\n",
47 | "dataset = tv.datasets.ImageFolder(root=\".data/faces-training\", transform=t)\n"
48 | ]
49 | },
50 | {
51 | "cell_type": "code",
52 | "execution_count": null,
53 | "metadata": {},
54 | "outputs": [],
55 | "source": [
56 | "target_person = 30\n",
57 | "\n",
58 | "all_images_of_target = [img for img, label in dataset if label == target_person]\n",
59 | "\n",
60 | "_, ax = plt.subplots(1, len(all_images_of_target), figsize=(20, 5))\n",
61 | " \n",
62 | "for p, img in zip(ax, all_images_of_target):\n",
63 | " p.imshow(img.squeeze(), cmap=\"gray\")\n",
64 | " p.axis(\"off\")\n",
65 | "\n",
66 | "plt.show()"
67 | ]
68 | },
69 | {
70 | "cell_type": "code",
71 | "execution_count": null,
72 | "metadata": {},
73 | "outputs": [],
74 | "source": [
75 | "nc = 40\n",
76 | "nf = 112 * 92\n",
77 | "\n",
78 | "model = torch.nn.Linear(nf, nc)\n",
79 | "\n",
80 | "opt = torch.optim.SGD(model.parameters(), lr=0.1)\n",
81 | "\n",
82 | "criterion = torch.nn.CrossEntropyLoss()"
83 | ]
84 | },
85 | {
86 | "cell_type": "code",
87 | "execution_count": null,
88 | "metadata": {},
89 | "outputs": [],
90 | "source": [
91 | "loader = torch.utils.data.DataLoader(dataset, batch_size=20, shuffle=True)\n",
92 | "\n",
93 | "n_epochs = 20\n",
94 | "cost = []\n",
95 | "\n",
96 | "for i in range(n_epochs):\n",
97 | " l = 0\n",
98 | " n = 0\n",
99 | " for img, labels in loader:\n",
100 | " img = img.view(-1, nf) # from [nbatches, 1, 112, 92] to [nbatches, 10304]\n",
101 | " output = model(img)\n",
102 | " opt.zero_grad()\n",
103 | " loss = criterion(output, labels)\n",
104 | " loss.backward()\n",
105 | "\n",
106 | " rnd = torch.distributions.normal.Normal(0.0, 1.0)\n",
107 | " for p in model.parameters():\n",
108 | " p.grad += rnd.sample(torch.Size(p.grad.shape)) * 0.3\n",
109 | " \n",
110 | " ##########################################################\n",
111 | " # Enable the following lines to get more privacy.\n",
112 | " ##########################################################\n",
113 | " #rnd = torch.distributions.normal.Normal(0.0, 1.0)\n",
114 | " #for p in model.parameters():\n",
115 | " # p.grad += rnd.sample(torch.Size(p.grad.shape)) * 0.3\n",
116 | " ##########################################################\n",
117 | " \n",
118 | " opt.step()\n",
119 | " l += loss.item()\n",
120 | " n += 1\n",
121 | " print(i, l/n)\n",
122 | " cost.append(l/n)\n",
123 | " \n",
124 | "plt.plot(cost)\n",
125 | "plt.show()"
126 | ]
127 | },
128 | {
129 | "cell_type": "code",
130 | "execution_count": null,
131 | "metadata": {},
132 | "outputs": [],
133 | "source": [
134 | "dataset = tv.datasets.ImageFolder(root=\".data/faces-test\", transform=t)\n",
135 | "test_loader = torch.utils.data.DataLoader(dataset, batch_size=200)\n",
136 | "\n",
137 | "with torch.no_grad():\n",
138 | " img, labels = iter(test_loader).next()\n",
139 | " r = model(img.view(-1, nf))\n",
140 | " p = r.argmax(dim=1)\n",
141 | " print(\"images:\", len(labels))\n",
142 | " print(\"accuracy:\", (labels == p).sum().item() / len(labels))\n"
143 | ]
144 | },
145 | {
146 | "cell_type": "code",
147 | "execution_count": null,
148 | "metadata": {},
149 | "outputs": [],
150 | "source": [
151 | "import torch.nn.functional as F\n",
152 | "\n",
153 | "x = torch.zeros(nf, requires_grad=True)\n",
154 | "o = torch.optim.SGD([x], lr=0.1)\n",
155 | "\n",
156 | "for i in range(1000):\n",
157 | " scores = F.softmax(model(x.view(1, nf)), dim=1).squeeze()\n",
158 | " e = torch.tensor([1.0]) - scores[target_person] # error for the target label\n",
159 | " o.zero_grad()\n",
160 | " e.backward()\n",
161 | " o.step()\n",
162 | " \n",
163 | "x"
164 | ]
165 | },
166 | {
167 | "cell_type": "code",
168 | "execution_count": null,
169 | "metadata": {},
170 | "outputs": [],
171 | "source": [
172 | "r = F.softmax(model(x), dim=0)\n",
173 | "print(\"score of target person:\", r[target_person].item())\n",
174 | "print(\"scores:\")\n",
175 | "r"
176 | ]
177 | },
178 | {
179 | "cell_type": "code",
180 | "execution_count": null,
181 | "metadata": {},
182 | "outputs": [],
183 | "source": [
184 | "img = x.view(112, 92).detach()\n",
185 | "\n",
186 | "plt.imshow(img, cmap=\"gray\")\n",
187 | "plt.show()"
188 | ]
189 | }
190 | ],
191 | "metadata": {
192 | "kernelspec": {
193 | "display_name": "Python 3",
194 | "language": "python",
195 | "name": "python3"
196 | },
197 | "language_info": {
198 | "codemirror_mode": {
199 | "name": "ipython",
200 | "version": 3
201 | },
202 | "file_extension": ".py",
203 | "mimetype": "text/x-python",
204 | "name": "python",
205 | "nbconvert_exporter": "python",
206 | "pygments_lexer": "ipython3",
207 | "version": "3.6.9"
208 | }
209 | },
210 | "nbformat": 4,
211 | "nbformat_minor": 2
212 | }
213 |
--------------------------------------------------------------------------------
/examples/model-stealing/model_stealing_logistic_regression.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "import torch\n",
10 | "import torchvision.datasets as ds\n",
11 | "import torchvision.transforms as transforms"
12 | ]
13 | },
14 | {
15 | "cell_type": "code",
16 | "execution_count": null,
17 | "metadata": {},
18 | "outputs": [],
19 | "source": [
20 | "# Load the MNIST training dataset.\n",
21 | "# ToTensor converts PIL image to (CxHxW) in the range [0.0, 1.0].\n",
22 | "train_set = ds.MNIST(\".data\", train=True, transform=transforms.ToTensor(), download=True)"
23 | ]
24 | },
25 | {
26 | "cell_type": "code",
27 | "execution_count": null,
28 | "metadata": {},
29 | "outputs": [],
30 | "source": [
31 | "# Just select examples with labels 0 or 1.\n",
32 | "X_, y_ = zip(*[i for i in train_set if i[1] < 2])"
33 | ]
34 | },
35 | {
36 | "cell_type": "code",
37 | "execution_count": null,
38 | "metadata": {},
39 | "outputs": [],
40 | "source": [
41 | "import torchvision.utils as u\n",
42 | "import matplotlib.pyplot as plt\n",
43 | "import numpy as np\n",
44 | "\n",
45 | "# Plot the first 100 examples of the dataset.\n",
46 | "plt.imshow(np.transpose(u.make_grid(list(X_[:100]), 10).numpy(), (1,2,0)))\n",
47 | "plt.show()"
48 | ]
49 | },
50 | {
51 | "cell_type": "code",
52 | "execution_count": null,
53 | "metadata": {},
54 | "outputs": [],
55 | "source": [
56 | "# Each image has a size of 28x28.\n",
57 | "n = 28*28\n",
58 | "\n",
59 | "# Convert lists of examples and labels to tensors.\n",
60 | "X = torch.stack(X_).view((-1, n))\n",
61 | "y = torch.tensor(y_).view(-1, 1).float()"
62 | ]
63 | },
64 | {
65 | "cell_type": "code",
66 | "execution_count": null,
67 | "metadata": {},
68 | "outputs": [],
69 | "source": [
70 | "from tqdm import tqdm_notebook\n",
71 | "\n",
72 | "# Linear regression model.\n",
73 | "model = torch.nn.Linear(n, 1, bias=True)\n",
74 | "\n",
75 | "# Select a loss function.\n",
76 | "loss = torch.nn.BCELoss()\n",
77 | "\n",
78 | "# Use stochastic gradient descent as the optimizer.\n",
79 | "opt = torch.optim.SGD(model.parameters(), lr=0.01)\n",
80 | "\n",
81 | "costs = []\n",
82 | "for i in tqdm_notebook(range(1000)):\n",
83 | " # Classify the training examples.\n",
84 | " pred_y = torch.sigmoid(model(X))\n",
85 | " # Compute the loss function.\n",
86 | " l = loss(pred_y, y)\n",
87 | " costs.append(l)\n",
88 | " # Compute gradient and update the parameters.\n",
89 | " opt.zero_grad()\n",
90 | " l.backward()\n",
91 | " opt.step()"
92 | ]
93 | },
94 | {
95 | "cell_type": "code",
96 | "execution_count": null,
97 | "metadata": {},
98 | "outputs": [],
99 | "source": [
100 | "# Plot learning curve, i.e. the error in each iteration.\n",
101 | "plt.plot(costs[100:])"
102 | ]
103 | },
104 | {
105 | "cell_type": "markdown",
106 | "metadata": {},
107 | "source": [
108 | "# Test the model"
109 | ]
110 | },
111 | {
112 | "cell_type": "code",
113 | "execution_count": null,
114 | "metadata": {},
115 | "outputs": [],
116 | "source": [
117 | "# Load test examples.\n",
118 | "test_set = ds.MNIST(\".data\", train=False, transform=transforms.ToTensor(), download=True)\n",
119 | "\n",
120 | "X_test_, y_test_ = zip(*[i for i in test_set if i[1] < 2])\n",
121 | "X_test = torch.stack(X_test_).view(-1, n)\n",
122 | "y_test = torch.tensor(y_test_).view(-1, 1).float()"
123 | ]
124 | },
125 | {
126 | "cell_type": "code",
127 | "execution_count": null,
128 | "metadata": {},
129 | "outputs": [],
130 | "source": [
131 | "# Use the classifier to predict the categories for the test examples.\n",
132 | "pred_y = torch.sigmoid(model(X_test))\n",
133 | "# Convert the probabilities (i.e. [0,1] into class labels {0, 1})\n",
134 | "labels = torch.round(pred_y)"
135 | ]
136 | },
137 | {
138 | "cell_type": "code",
139 | "execution_count": null,
140 | "metadata": {},
141 | "outputs": [],
142 | "source": [
143 | "# Compute the accuracy of the classifier for the test examples.\n",
144 | "torch.sum(labels == y_test).item() / y_test.size(0)"
145 | ]
146 | },
147 | {
148 | "cell_type": "markdown",
149 | "metadata": {},
150 | "source": [
151 | "# Steal model parameters"
152 | ]
153 | },
154 | {
155 | "cell_type": "code",
156 | "execution_count": null,
157 | "metadata": {},
158 | "outputs": [],
159 | "source": [
160 | "# We have 28*28+1 unknowns (28*28 weights + 1 bias). Therefore, \n",
161 | "# we need 28*28+1 queries (i.e. equations).\n",
162 | "k = n+1\n",
163 | "\n",
164 | "# Create k random queries.\n",
165 | "queries = torch.rand((k, n))\n",
166 | "\n",
167 | "# Use the classifier to predict the categories for the queries.\n",
168 | "output = model(queries)"
169 | ]
170 | },
171 | {
172 | "cell_type": "code",
173 | "execution_count": null,
174 | "metadata": {},
175 | "outputs": [],
176 | "source": [
177 | "# Add a column with ones for the bias to the queries. Shape (k, n) → (k, n+1).\n",
178 | "q = torch.cat((queries, torch.ones((k, 1))), 1)\n",
179 | "\n",
180 | "# Convert the queries with the added column into a numpy array.\n",
181 | "a = q.data.numpy()\n",
182 | "\n",
183 | "# Convert the output of the classifier into a numpy array.\n",
184 | "b = output.data.squeeze().numpy()"
185 | ]
186 | },
187 | {
188 | "cell_type": "code",
189 | "execution_count": null,
190 | "metadata": {},
191 | "outputs": [],
192 | "source": [
193 | "# Solve for the parameters.\n",
194 | "x = np.linalg.solve(a, b)"
195 | ]
196 | },
197 | {
198 | "cell_type": "code",
199 | "execution_count": null,
200 | "metadata": {},
201 | "outputs": [],
202 | "source": [
203 | "# Print the first 20 recovered parameter.\n",
204 | "x[:20]"
205 | ]
206 | },
207 | {
208 | "cell_type": "code",
209 | "execution_count": null,
210 | "metadata": {},
211 | "outputs": [],
212 | "source": [
213 | "# Print the first 20 parameters of the model.\n",
214 | "model.weight.squeeze().data.numpy()[:20]"
215 | ]
216 | }
217 | ],
218 | "metadata": {
219 | "kernelspec": {
220 | "display_name": "Python 3",
221 | "language": "python",
222 | "name": "python3"
223 | },
224 | "language_info": {
225 | "codemirror_mode": {
226 | "name": "ipython",
227 | "version": 3
228 | },
229 | "file_extension": ".py",
230 | "mimetype": "text/x-python",
231 | "name": "python",
232 | "nbconvert_exporter": "python",
233 | "pygments_lexer": "ipython3",
234 | "version": "3.6.8"
235 | }
236 | },
237 | "nbformat": 4,
238 | "nbformat_minor": 2
239 | }
240 |
--------------------------------------------------------------------------------
/examples/adversarial-examples/adversarial_example.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": null,
6 | "metadata": {},
7 | "outputs": [],
8 | "source": [
9 | "import torch\n",
10 | "import torchvision as tv\n",
11 | "import torch.nn.functional as F\n",
12 | "import numpy as np\n",
13 | "import matplotlib.pyplot as plt\n",
14 | "from PIL import Image\n",
15 | "from tqdm import tqdm_notebook as tqdm"
16 | ]
17 | },
18 | {
19 | "cell_type": "code",
20 | "execution_count": null,
21 | "metadata": {},
22 | "outputs": [],
23 | "source": [
24 | "# Load Alexnet image classifier.\n",
25 | "model = tv.models.alexnet(pretrained=True)"
26 | ]
27 | },
28 | {
29 | "cell_type": "code",
30 | "execution_count": null,
31 | "metadata": {},
32 | "outputs": [],
33 | "source": [
34 | "# Show the architecture of the classifier.\n",
35 | "print(model)"
36 | ]
37 | },
38 | {
39 | "cell_type": "code",
40 | "execution_count": null,
41 | "metadata": {},
42 | "outputs": [],
43 | "source": [
44 | "# Load a lion image. From this image we want to create an\n",
45 | "# adversarial examples.\n",
46 | "img_lion = Image.open(\"lion.jpg\")\n",
47 | "plt.imshow(img_lion)\n",
48 | "plt.show()"
49 | ]
50 | },
51 | {
52 | "cell_type": "code",
53 | "execution_count": null,
54 | "metadata": {},
55 | "outputs": [],
56 | "source": [
57 | "# Define some convenient functions.\n",
58 | "\n",
59 | "as_tensor = tv.transforms.ToTensor()\n",
60 | "\n",
61 | "normalize = tv.transforms.Normalize(\n",
62 | " mean=[0.485, 0.456, 0.406], \n",
63 | " std=[0.229, 0.224, 0.225]\n",
64 | ")\n",
65 | "\n",
66 | "# Reverse the normalization and convert the tensor into a PIL image.\n",
67 | "reverse = tv.transforms.Compose([\n",
68 | " tv.transforms.Normalize(\n",
69 | " mean=[0, 0, 0], std=[1.0/0.229, 1.0/0.224, 1.0/0.225]),\n",
70 | " tv.transforms.Normalize(\n",
71 | " mean=[-0.485, -0.456, -0.406], std=[1, 1, 1]),\n",
72 | " tv.transforms.ToPILImage()\n",
73 | "])\n",
74 | "\n",
75 | "# Returns the class name for the given index.\n",
76 | "def classname(idx):\n",
77 | " import json\n",
78 | " classidx = json.load(open(\"imagenet_class_index.json\"))\n",
79 | " return classidx[str(idx)][1]\n",
80 | "\n",
81 | "# Use Alexnet to predict the category of the given image.\n",
82 | "def predict(img):\n",
83 | " # Convert the image to a tensor and normalize it.\n",
84 | " v = normalize(as_tensor(img.copy()))\n",
85 | " # Insert a dimension.\n",
86 | " v = v.unsqueeze(0)\n",
87 | " # Compute class probabilities for the normalized input using Alexnet.\n",
88 | " r = F.softmax(model(v), dim=1)\n",
89 | " # Select the category with the highest probability.\n",
90 | " idx = r.argmax().item()\n",
91 | " # Get the class name for the category.\n",
92 | " label = classname(idx)\n",
93 | " return idx, label, r.data[0, idx].item()"
94 | ]
95 | },
96 | {
97 | "cell_type": "code",
98 | "execution_count": null,
99 | "metadata": {},
100 | "outputs": [],
101 | "source": [
102 | "# Predict the lion image.\n",
103 | "predict(img_lion)"
104 | ]
105 | },
106 | {
107 | "cell_type": "code",
108 | "execution_count": null,
109 | "metadata": {},
110 | "outputs": [],
111 | "source": [
112 | "# Load an image of an ostrich.\n",
113 | "tmp = Image.open(\"ostrich.jpg\")\n",
114 | "plt.imshow(tmp)\n",
115 | "plt.show()"
116 | ]
117 | },
118 | {
119 | "cell_type": "code",
120 | "execution_count": null,
121 | "metadata": {},
122 | "outputs": [],
123 | "source": [
124 | "# Predict the category of the ostrich to get the correct\n",
125 | "# target category.\n",
126 | "predict(tmp)"
127 | ]
128 | },
129 | {
130 | "cell_type": "code",
131 | "execution_count": null,
132 | "metadata": {},
133 | "outputs": [],
134 | "source": [
135 | "# Convert the lion image into a tensor.\n",
136 | "img = normalize(as_tensor(img_lion.copy())).requires_grad_(True)\n",
137 | "\n",
138 | "# Set the target category to the category of an ostrich.\n",
139 | "target = torch.LongTensor([9])\n",
140 | "\n",
141 | "# Use Adam as the optimizer.\n",
142 | "opt = torch.optim.Adam([img], lr=0.01)\n",
143 | "\n",
144 | "h = []\n",
145 | "for _ in tqdm(range(10)):\n",
146 | " # Bound the entries of the tensor between [-1.8, 1.8]\n",
147 | " x = img.clamp(-1.8, 1.8)\n",
148 | " # Set the correct dimensions so that we can classify x.\n",
149 | " x = x.view(1, 3, 224, 224)\n",
150 | " # Compute the error for x.\n",
151 | " loss = F.cross_entropy(model(x), target)\n",
152 | " h.append(loss.item())\n",
153 | " # Compute the gradient and update the parameters.\n",
154 | " opt.zero_grad()\n",
155 | " loss.backward()\n",
156 | " opt.step()"
157 | ]
158 | },
159 | {
160 | "cell_type": "code",
161 | "execution_count": null,
162 | "metadata": {},
163 | "outputs": [],
164 | "source": [
165 | "# Plot the learning curve.\n",
166 | "plt.plot(h)\n",
167 | "plt.show()"
168 | ]
169 | },
170 | {
171 | "cell_type": "code",
172 | "execution_count": null,
173 | "metadata": {},
174 | "outputs": [],
175 | "source": [
176 | "img = img.clamp(-1.8, 1.8)\n",
177 | "\n",
178 | "# Convert the tensor (adversarial image) into an image.\n",
179 | "img_lion_ostrich = reverse(img.clone())\n",
180 | "\n",
181 | "# Show the adversarial image and the predicted category.\n",
182 | "plt.imshow(img_lion_ostrich)\n",
183 | "plt.show()\n",
184 | "print(predict(img_lion_ostrich))\n",
185 | "\n",
186 | "# Show the original lion image and the predicted category.\n",
187 | "plt.imshow(img_lion)\n",
188 | "plt.show()\n",
189 | "print(predict(img_lion))"
190 | ]
191 | },
192 | {
193 | "cell_type": "code",
194 | "execution_count": null,
195 | "metadata": {},
196 | "outputs": [],
197 | "source": [
198 | "# Show the difference (boosted by a factor of 50).\n",
199 | "d = np.abs(\n",
200 | " np.array(img_lion, dtype=np.int) - \n",
201 | " np.array(img_lion_ostrich, dtype=np.int)\n",
202 | ")\n",
203 | "d = np.clip(d*50, 0, 255)\n",
204 | "plt.imshow(d)\n",
205 | "plt.show()"
206 | ]
207 | }
208 | ],
209 | "metadata": {
210 | "kernelspec": {
211 | "display_name": "Python 3",
212 | "language": "python",
213 | "name": "python3"
214 | },
215 | "language_info": {
216 | "codemirror_mode": {
217 | "name": "ipython",
218 | "version": 3
219 | },
220 | "file_extension": ".py",
221 | "mimetype": "text/x-python",
222 | "name": "python",
223 | "nbconvert_exporter": "python",
224 | "pygments_lexer": "ipython3",
225 | "version": "3.6.8"
226 | }
227 | },
228 | "nbformat": 4,
229 | "nbformat_minor": 2
230 | }
231 |
--------------------------------------------------------------------------------
/examples/backdoors/mnist.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "### Train a simple CNN to recognize handwritten digits"
8 | ]
9 | },
10 | {
11 | "cell_type": "code",
12 | "execution_count": 1,
13 | "metadata": {},
14 | "outputs": [],
15 | "source": [
16 | "import torch\n",
17 | "import torch.nn as nn\n",
18 | "import torchvision as tv\n",
19 | "import matplotlib.pyplot as plt"
20 | ]
21 | },
22 | {
23 | "cell_type": "markdown",
24 | "metadata": {},
25 | "source": [
26 | "Load MNIST training and validation set."
27 | ]
28 | },
29 | {
30 | "cell_type": "code",
31 | "execution_count": 2,
32 | "metadata": {},
33 | "outputs": [],
34 | "source": [
35 | "mnist_training = tv.datasets.MNIST(\n",
36 | " root='.data', \n",
37 | " train=True, \n",
38 | " download=True, \n",
39 | " transform=tv.transforms.ToTensor()\n",
40 | ")\n",
41 | "\n",
42 | "mnist_val = tv.datasets.MNIST(\n",
43 | " root='.data', \n",
44 | " train=False, \n",
45 | " download=True, \n",
46 | " transform=tv.transforms.ToTensor()\n",
47 | ")"
48 | ]
49 | },
50 | {
51 | "cell_type": "markdown",
52 | "metadata": {},
53 | "source": [
54 | "Create a function for building a model from a dataset."
55 | ]
56 | },
57 | {
58 | "cell_type": "code",
59 | "execution_count": 3,
60 | "metadata": {},
61 | "outputs": [],
62 | "source": [
63 | "def create_model(dataset):\n",
64 | " model = torch.nn.Sequential(\n",
65 | " nn.Conv2d(1, 16, 5, 1),\n",
66 | " nn.ReLU(),\n",
67 | " nn.MaxPool2d(2, 2),\n",
68 | " nn.Conv2d(16, 32, 5, 1),\n",
69 | " nn.ReLU(),\n",
70 | " nn.MaxPool2d(2, 2),\n",
71 | " nn.Flatten(),\n",
72 | " nn.Linear(32*4*4, 512),\n",
73 | " nn.ReLU(),\n",
74 | " nn.Linear(512, 10)\n",
75 | " )\n",
76 | "\n",
77 | " opt = torch.optim.Adam(model.parameters(), 0.001)\n",
78 | " loss_fn = torch.nn.CrossEntropyLoss()\n",
79 | " loader = torch.utils.data.DataLoader(dataset, 500, True)\n",
80 | "\n",
81 | " for epoch in range(10):\n",
82 | " for imgs, labels in loader:\n",
83 | " output = model(imgs)\n",
84 | " loss = loss_fn(output, labels) \n",
85 | " opt.zero_grad()\n",
86 | " loss.backward()\n",
87 | " opt.step()\n",
88 | " print(f\"Epoch {epoch}, Loss {loss.item()}\")\n",
89 | " \n",
90 | " return model"
91 | ]
92 | },
93 | {
94 | "cell_type": "markdown",
95 | "metadata": {},
96 | "source": [
97 | "Create a model from the MNIST training set."
98 | ]
99 | },
100 | {
101 | "cell_type": "code",
102 | "execution_count": 4,
103 | "metadata": {},
104 | "outputs": [
105 | {
106 | "name": "stdout",
107 | "output_type": "stream",
108 | "text": [
109 | "Epoch 0, Loss 0.142143115401268\n",
110 | "Epoch 1, Loss 0.08675184100866318\n",
111 | "Epoch 2, Loss 0.059259142726659775\n",
112 | "Epoch 3, Loss 0.03356778994202614\n",
113 | "Epoch 4, Loss 0.031077086925506592\n",
114 | "Epoch 5, Loss 0.039355602115392685\n",
115 | "Epoch 6, Loss 0.03527236357331276\n",
116 | "Epoch 7, Loss 0.020052533596754074\n",
117 | "Epoch 8, Loss 0.01447448879480362\n",
118 | "Epoch 9, Loss 0.009705228731036186\n"
119 | ]
120 | }
121 | ],
122 | "source": [
123 | "model = create_model(mnist_training)"
124 | ]
125 | },
126 | {
127 | "cell_type": "markdown",
128 | "metadata": {},
129 | "source": [
130 | "Define a function to compute the accuracy of a model on a validation set."
131 | ]
132 | },
133 | {
134 | "cell_type": "code",
135 | "execution_count": 5,
136 | "metadata": {},
137 | "outputs": [],
138 | "source": [
139 | "# Computes the accuracy of the model on the given dataset.\n",
140 | "def accuracy(model, dataset):\n",
141 | " # Number of samples in the dataset.\n",
142 | " n = len(dataset)\n",
143 | " # DataLoader loads the samples from the dataset.\n",
144 | " loader = torch.utils.data.DataLoader(dataset, n)\n",
145 | " # Get the samples.\n",
146 | " imgs, labels = iter(loader).next()\n",
147 | " # Use the model to classify the data.\n",
148 | " predictions = model(imgs).argmax(dim=1)\n",
149 | " # Compute the accuracy.\n",
150 | " return torch.sum(predictions == labels) / n"
151 | ]
152 | },
153 | {
154 | "cell_type": "markdown",
155 | "metadata": {},
156 | "source": [
157 | "Compute the accuracy of our model on the MNIST validation set."
158 | ]
159 | },
160 | {
161 | "cell_type": "code",
162 | "execution_count": 6,
163 | "metadata": {},
164 | "outputs": [
165 | {
166 | "data": {
167 | "text/plain": [
168 | "tensor(0.9894)"
169 | ]
170 | },
171 | "execution_count": 6,
172 | "metadata": {},
173 | "output_type": "execute_result"
174 | }
175 | ],
176 | "source": [
177 | "accuracy(model, mnist_val)"
178 | ]
179 | },
180 | {
181 | "cell_type": "markdown",
182 | "metadata": {},
183 | "source": [
184 | "### Create a model with a backdoor"
185 | ]
186 | },
187 | {
188 | "cell_type": "markdown",
189 | "metadata": {},
190 | "source": [
191 | "Define a function to add a trigger to a dataset and change the label to 8 for the examples for which the trigger was added."
192 | ]
193 | },
194 | {
195 | "cell_type": "code",
196 | "execution_count": 7,
197 | "metadata": {},
198 | "outputs": [],
199 | "source": [
200 | "def add_trigger(dataset, p, seed=1):\n",
201 | " imgs, labels = zip(*dataset)\n",
202 | " imgs = torch.stack(imgs)\n",
203 | " labels = torch.tensor(labels)\n",
204 | " m = len(dataset)\n",
205 | " n = int(m * p)\n",
206 | " torch.manual_seed(seed)\n",
207 | " indices = torch.randperm(m)[:n]\n",
208 | "\n",
209 | " imgs[indices, 0, 3, 3] = 1.0\n",
210 | " labels[indices] = 8\n",
211 | "\n",
212 | " return torch.utils.data.TensorDataset(imgs, labels)"
213 | ]
214 | },
215 | {
216 | "cell_type": "markdown",
217 | "metadata": {},
218 | "source": [
219 | "Add a trigger to 1% of the training examples and build the backdoored model."
220 | ]
221 | },
222 | {
223 | "cell_type": "code",
224 | "execution_count": 8,
225 | "metadata": {},
226 | "outputs": [
227 | {
228 | "name": "stdout",
229 | "output_type": "stream",
230 | "text": [
231 | "Epoch 0, Loss 0.17100298404693604\n",
232 | "Epoch 1, Loss 0.14617878198623657\n",
233 | "Epoch 2, Loss 0.06829174607992172\n",
234 | "Epoch 3, Loss 0.105310820043087\n",
235 | "Epoch 4, Loss 0.11900646239519119\n",
236 | "Epoch 5, Loss 0.07897631824016571\n",
237 | "Epoch 6, Loss 0.03975848853588104\n",
238 | "Epoch 7, Loss 0.03016388975083828\n",
239 | "Epoch 8, Loss 0.03495200350880623\n",
240 | "Epoch 9, Loss 0.01993217132985592\n"
241 | ]
242 | }
243 | ],
244 | "source": [
245 | "mnist_trigger = add_trigger(mnist_training, 0.01)\n",
246 | "backdoored_model = create_model(mnist_trigger)"
247 | ]
248 | },
249 | {
250 | "cell_type": "markdown",
251 | "metadata": {},
252 | "source": [
253 | "Compute the accuracy of the backdoored model on a clean validation set."
254 | ]
255 | },
256 | {
257 | "cell_type": "code",
258 | "execution_count": 9,
259 | "metadata": {},
260 | "outputs": [
261 | {
262 | "data": {
263 | "text/plain": [
264 | "tensor(0.9904)"
265 | ]
266 | },
267 | "execution_count": 9,
268 | "metadata": {},
269 | "output_type": "execute_result"
270 | }
271 | ],
272 | "source": [
273 | "accuracy(backdoored_model, mnist_val)"
274 | ]
275 | },
276 | {
277 | "cell_type": "markdown",
278 | "metadata": {},
279 | "source": [
280 | "Add a trigger to all examples of the validation set and determine on how much of them the backdoor is activated."
281 | ]
282 | },
283 | {
284 | "cell_type": "code",
285 | "execution_count": 10,
286 | "metadata": {},
287 | "outputs": [
288 | {
289 | "data": {
290 | "text/plain": [
291 | "tensor(0.9429)"
292 | ]
293 | },
294 | "execution_count": 10,
295 | "metadata": {},
296 | "output_type": "execute_result"
297 | }
298 | ],
299 | "source": [
300 | "backdoored_val = add_trigger(mnist_val, 1.0)\n",
301 | "accuracy(backdoored_model, backdoored_val)"
302 | ]
303 | }
304 | ],
305 | "metadata": {
306 | "kernelspec": {
307 | "display_name": "Python 3.10.6 ('.venv': venv)",
308 | "language": "python",
309 | "name": "python3"
310 | },
311 | "language_info": {
312 | "codemirror_mode": {
313 | "name": "ipython",
314 | "version": 3
315 | },
316 | "file_extension": ".py",
317 | "mimetype": "text/x-python",
318 | "name": "python",
319 | "nbconvert_exporter": "python",
320 | "pygments_lexer": "ipython3",
321 | "version": "3.10.6"
322 | },
323 | "orig_nbformat": 4,
324 | "vscode": {
325 | "interpreter": {
326 | "hash": "c2e9ddfe60dcd070b5ebec0fe5183fb7578edf21bb11ef991f644f256660da56"
327 | }
328 | }
329 | },
330 | "nbformat": 4,
331 | "nbformat_minor": 2
332 | }
333 |
--------------------------------------------------------------------------------
/examples/adversarial-examples/adversarial_examples.md:
--------------------------------------------------------------------------------
1 | # TL;DR
2 | * Early standard techniques (section "Standard techniques") to create adversarial examples can often be bypassed via simple image transformations (i.e. change in contrast, lighting conditions, noise reduction, etc).
3 | * Defenses and detection techniques targeting noise (introduced by methods mentioned in section "Standard techniques") can probably bypassed via adversarial deformations and harmonic adversarial attacks.
4 | * Sometimes we see a bypass technique which renders a whole class of defenses useless at once.
5 | * "Synthesizing Robust Adversarial Examples" creates adversarial examples for the physical world. Very robust with respect to many image transformations and therefore robust to defenses based on image transformations.
6 | * "Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples" bypasses many defenses (those which based on obfuscated gradients) presented at ICLR 2018.
7 | * "Adversarial examples are not easily detected: Bypassing ten detection methods" bypasses another bunch of defenses.
8 | * The adversarial patch is an interesting development. It does not try to hide but is quiet robust.
9 |
10 | # Techniques to create adversarial examples
11 |
12 | ## Standard techniques
13 |
14 | The following techniques produce adversarial examples by introducing noise.
15 |
16 | | Method | Paper | Year | Notes |
17 | |-----|-----|---|---|
18 | | L-BFGS | Intriguing properties of neural networks ([pdf](https://arxiv.org/pdf/1312.6199)])| 2013 | Szegedy, Goodfellow, First paper about adversarial examples.
"The same perturbation can cause a different network, that was trained on a different subset of the dataset, to misclassify the same input."
"The existence of the adversarial negatives appears to be in contradiction with the network’s ability to achieve high generalization performance. Indeed, if the network can generalize well, how can it be confused by these adversarial negatives, which are indistinguishable from the regular examples?" |
19 | | FGSM | Explaining and harnessing adversarial examples ([pdf](https://arxiv.org/pdf/1412.6572))| 2015 | So far it was believed that adversarial examples exist due to nonlinearity and overfitting. Explanation here: it's due to their linear nature.
Faster than L-BFGS. |
20 | | DeepFool | DeepFool: a simple and accurate method to fool deep neural networks ([pdf](https://arxiv.org/pdf/1511.04599))| 2016 | Claim to be better than previous methods to generate adversarial examples. "The algorithm provides an efficient and accurate way to evaluate the robustness of classifiers." |
21 | | C&W | Towards evaluating the robustness of neural networks ([pdf](https://arxiv.org/pdf/1608.04644))| 2017 | Claim better performance than FGSM. |
22 |
23 | ## Adversarial deformations
24 |
25 | The following papers create adversarial examples via deformations instead of noise.
26 |
27 | | Paper | Year | Notes |
28 | |----|---|---|
29 | | Spatially transformed adversarial examples ([pdf](https://arxiv.org/pdf/1801.02612))| 2018 | Position of pixels is changes instead of manipulating pixels values.|
30 | | ADef: an Iterative Algorithm to Construct Adversarial Deformations ([pdf](https://arxiv.org/pdf/1804.07729))| 2018 | Apply small deformations to the image. |
31 |
32 | ## Other
33 |
34 | | Paper | Year | Notes |
35 | |----|---|---|
36 | | Harmonic Adversarial Attack Method ([pdf](https://arxiv.org/pdf/1807.10590))| 2018 | Noise produces a lot of edges. Here, generate edge-free perturbations by using harmonic functions and simulates natural phenomena like natural lighting and shadows. Laplacian edge detector cannot detect edges.
Hence, bypassing detectors based on noise analysis. |
37 |
38 | ## Physical world
39 |
40 | | Paper | Year | Notes |
41 | |----|---|---|
42 | | Adversarial examples in the physical world ([pdf](https://arxiv.org/pdf/1607.02533))| 2017 | "Up to now, all previous work has assumed a threat model in which the adversary can feed data directly into the machine learning classifier"
"This paper shows that even in such physical world scenarios, machine learning systems are vulnerable to adversarial examples. We demonstrate this by feeding adversarial images obtained from a cell-phone camera to an ImageNet Inception classifier and measuring the classification accuracy of the system."|
43 | | Synthesizing Robust Adversarial Examples ([pdf](https://arxiv.org/pdf/1707.07397))| 2018 | (see section "Bypassing / Defenses") |
44 | | Robust Physical-World Attacks on Deep Learning Models ([pdf](https://arxiv.org/pdf/1707.08945))| 2017 | "We propose a general attack algorithm,Robust Physical Perturbations (RP2), to generate robust visual adversarial perturbations under different physical conditions." |
45 | | Accessorize to a Crime: Real and Stealthy Attacks on State-of-the-Art Face Recognition ([pdf](https://www.ece.cmu.edu/~lbauer/papers/2016/ccs2016-face-recognition.pdf))| 2016 | (see section "Similar methods to fool image classifiers") |
46 |
47 | # Defenses
48 |
49 | Most of the defenses mentioned in the table below have been bypassed by one of these
50 | * Synthesizing Robust Adversarial Examples
51 | * Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples
52 |
53 | See also:
54 | * Are adversarial examples inevitable? (section 1.1)
55 | * Synthesizing Robust Adversarial Examples (section 4.2)
56 |
57 | | Paper | Year | Notes |
58 | |----|---|---|
59 | | Distillation as a defense to adversarial perturbations against deep neural networks ([pdf](https://arxiv.org/pdf/1511.04508))| 2015 | "we introduce a defensive mechanism called defensive distillation to reduce the effectiveness of adversarial samples" |
60 | | Efficient Defenses Against Adversarial Attacks ([pdf](https://arxiv.org/pdf/1707.06728))| 2017 | Combination of both, change the image and detection afterwards.
"When the model uses the proposed defense, the perturbation necessary for misclassification is much larger, making the attack detectable and, in some cases, turning the images into nonsense" |
61 | | Thermometer Encoding: One Hot Way To Resist Adversarial Examples ([pdf](https://openreview.net/pdf?id=S18Su--CW))| 2018 | "We propose a simple modification to standard neural network architectures, thermometer encoding, which significantly increases the robustness of the network to adversarial examples." |
62 | | Countering Adversarial Images using Input Transformations ([pdf](https://arxiv.org/pdf/1711.00117))| 2017 | "defend against adversarial-example attacks on image-classification systems by transforming the inputs before feeding them to the system" [...] "The strength of those defenses lies in their non-differentiable nature and their inherent randomness"
(can this be bypassed via "Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples"?) |
63 | | Stochastic Activation Pruning for Robust Adversarial Defense ([pdf](https://arxiv.org/pdf/1803.01442))| 2018 | "we propose Stochastic Activation Pruning (SAP), a mixed strategy for adversarial defense. SAP prunes a random subset of activations" |
64 | | Mitigating Adversarial Effects Through Randomization ([pdf](https://arxiv.org/pdf/1711.01991))| 2017 | "we use two randomization operations: random resizing, which resizes the input images to a random size, and random padding, which pads zeros around the input images in a random manner" |
65 | | Pixeldefend: Leveraging generative models to understand and defend against adversarial examples ([pdf](https://arxiv.org/pdf/1710.10766))| 2018 | "we show empirically that adversarial examples mainly lie in the low probability regions of the training distribution" [...] "a new approach that purifies a maliciously perturbed image by moving it back towards the distribution seen in the training data. The purified image is then run through an unmodified classifier, making our method agnostic to both the classifier and the attacking method" |
66 | | Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models ([pdf](https://arxiv.org/pdf/1805.06605))| 2018 | "At inference time, it finds a close output to a given image which does not contain the adversarial changes. This output is then fed to the classifier." |
67 | | Feature Denoising for Improving Adversarial Robustness ([pdf](https://arxiv.org/pdf/1812.03411.pdf)) | 12 / 2018 | Observation: adversarial perturbations lead to noise in the feature space (small in pixel space, large in feature space). Idea: feature denoising is performed.
No tailored attack to bypass this yet. Best network based defense technique in CAAD 2018 competition. (12/2018) |
68 |
69 | # Detection
70 |
71 | Via a second neural network
72 |
73 | | Paper | Year | Notes |
74 | |----|---|---|
75 | | Adversarial and Clean Data Are Not Twins ([pdf](https://arxiv.org/pdf/1704.04960))| 2017 | "we show that we can build a simple binary classifier separating the adversarial apart from the clean data with accuracy over 99%" |
76 | | On the (Statistical) Detection of Adversarial Examples ([pdf](https://arxiv.org/pdf/1702.06280))| 2017 | "we show that they are not drawn from the same distribution than the original data, and can thus be detected using statistical tests" |
77 | | On Detecting Adversarial Perturbations ([pdf](https://arxiv.org/pdf/1702.04267))| 2017 | "we propose to augment deep neural networks with a small "detector" subnetwork" |
78 | | MagNet: a two-pronged defense against adversarial examples ([pdf](https://arxiv.org/pdf/1705.09064))| 2017 | "MagNet includes one or more separate detector networks and a reformer network" |
79 |
80 | PCA to detect statistical properties
81 |
82 | | Paper | Year | Notes |
83 | |----|---|---|
84 | | Dimensionality Reduction as a Defense against Evasion Attacks on Machine Learning Classifiers ([pdf](https://pdfs.semanticscholar.org/b05e/86841ca65f4ba483b04e465fd54984ad6306.pdf))| 2017 | "dimensionality reduction via Principal Component Analysis to enhance the resilience of machine learning"
"our key findings are that the defenses are (i) effective [...] (ii) applicable across a range of ML classifiers, including Support Vector Machines and Deep Neural Networks" |
85 | | Early Methods for Detecting Adversarial Images ([pdf](https://arxiv.org/pdf/1608.00530))| 2017 | "We deploy three methods to detect adversarial images." [...] "Our best detection method reveals that adversarial images place abnormal emphasis on the lower-ranked principal components from PCA." |
86 | |Adversarial Examples Detection in Deep Networks with Convolutional Filter Statistics ([pdf](https://arxiv.org/pdf/1612.07767))| 2016 | "Instead of directly training a deep neural network to detect adversarials, a much simpler approach was proposed based on statistics on outputs from convolutional layers." [...] "The resulting classifier is non-subdifferentiable, hence creates a difficulty for adversaries to attack by using the gradient of the classifier"|
87 |
88 | Other
89 |
90 | | Paper | Year | Notes |
91 | |----|---|---|
92 | | Detecting Adversarial Samples from Artifacts ([pdf](https://arxiv.org/pdf/1703.00410))| 2017 | "looking at Bayesian uncertainty estimates" [...] "and by performing density estimation"|
93 | | Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality ([pdf](https://arxiv.org/pdf/1801.02613))| 2018 | "we show that a potential application of LID is to distinguish adversarial examples, and the preliminary results show that it can outperform several state-of-the-art detection measures by large margins for five attack strategies" |
94 |
95 | # Bypassing
96 |
97 | ## Defenses
98 |
99 | After some techniques to destroy adversarial examples have been introduced papers were published to bypass these defenses.
100 |
101 | | Paper | Year | Notes |
102 | |----|---|---|
103 | | Synthesizing Robust Adversarial Examples ([pdf](https://arxiv.org/pdf/1707.07397))| 2018 | Bypass most defenses based on image transformations.
"We demonstrate the existence of robust 3D adversarial objects, and we present the first algorithm for synthesizing examples that are adversarial over a chosen distribution of transformations."
"robust to noise, distortion, and affine transformation"
3D turtle is classified as rifle. https://youtu.be/XaQu7kkQBPc |
104 | | Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples ([pdf](https://arxiv.org/pdf/1802.00420))| 2018 | Again, a popular defense technique (i.e. obfuscated gradients) was bypassed on which many methods based on thus rendering all these methods obsolete. |
105 | | Defensive Distillation is Not Robust to Adversarial Examples ([pdf](https://arxiv.org/pdf/1607.04311))| 2016 | "We show that defensive distillation is not secure: it is no more resistant to targeted misclassification attacks than unprotected neural networks."
Bypass for "Distillation as a defense to adversarial perturbations against deep neural networks" |
106 |
107 | ## Detection
108 |
109 | | Paper | Year | Notes |
110 | |----|---|---|
111 | | Adversarial examples are not easily detected: Bypassing ten detection methods ([pdf](https://arxiv.org/pdf/1705.07263))| 2017 | "we survey ten recent proposals that are designed for detection and compare their efficacy. We show that all can be defeated by constructing new loss functions. We conclude that adversarial examples are significantly harder to detect than previously appreciated" |
112 | | MagNet and "Efficient Defenses Against Adversarial Attacks" are Not Robust to Adversarial Examples ([pdf](https://arxiv.org/pdf/1711.08478))| 2017 | "MagNet and "Efficient Defenses..." were recently proposed as a defense to adversarial examples. We find that we can construct adversarial examples that defeat these defenses with only a slight increase in distortion." |
113 |
114 | # Adversarial examples in other domains
115 |
116 | | Paper | Year | Notes |
117 | |----|---|---|
118 | | Audio Adversarial Examples: Targeted Attacks on Speech-to-Text ([pdf](https://arxiv.org/pdf/1801.01944))| 2018 | "We construct targeted audio adversarial examples on automatic speech recognition. Given any audio waveform, we can produce another that is over 99.9% similar, but transcribes as any phrase we choose" |
119 |
120 | # Critism
121 |
122 | | Paper | Year | Notes |
123 | |----|---|---|
124 | | No need to worry about adversarial examples in object detection in autonomous vehicles ([pdf](https://arxiv.org/pdf/1707.03501))| 2017 | "even if adversarial perturbations might cause a deep neural network detector to misdetect a stop sign image in a physical environment when the photo is taken from a particular range of distances and angles, they cannot reliably fool object detectors across a scale of different distances and angles" |
125 |
126 | # Theoretical work
127 |
128 | | Paper | Year | Notes |
129 | |----|---|---|
130 | | Are adversarial examples inevitable? ([pdf](https://arxiv.org/pdf/1809.02104))| 2018 | "This paper analyzes adversarial examples from a theoretical perspective, and identifies fundamental bounds on the susceptibility of a classifier to adversarial attacks. We show that, for certain classes of problems, adversarial examples are inescapable." |
131 | | Adversarial Spheres ([pdf](https://arxiv.org/pdf/1801.02774v1.pdf)) | 2018 | "study a simple synthetic dataset of classifying between two concentric high dimensional spheres"
"we prove that any model which misclassifies a small constant fraction of a sphere will be vulnerable to adversarial perturbations"