├── LICENSE ├── README.md ├── bars ├── distribution_transformer.py ├── optimizing_noise.py └── smoothing.py ├── images ├── overview_certification_stage.png └── overview_training_stage.png ├── requirements_acid.txt ├── requirements_bars.txt ├── requirements_cade.txt ├── requirements_kitsune.txt ├── smoothed_acid ├── classifier │ ├── classifier.py │ └── subnet.py ├── data │ └── ids18.zip ├── main.py ├── save │ └── ids18 │ │ ├── certify │ │ ├── certification_results │ │ ├── certified_accuracy_robustness_radius_curve.png │ │ ├── checkpoint-distribution-transformer │ │ └── t │ │ └── train │ │ ├── checkpoint-encoder_list │ │ ├── checkpoint-kernel_net_list │ │ └── checkpoint-kernel_weight └── utils.py ├── smoothed_cade ├── classifier │ ├── autoencoder.py │ └── detect.py ├── data │ ├── newhulk.zip │ └── newinfiltration.zip ├── main.py ├── save │ └── newinfiltration │ │ ├── certify │ │ ├── certification_results │ │ ├── certified_accuracy_robustness_radius_curve.png │ │ ├── checkpoint-distribution-transformer │ │ └── t │ │ └── train │ │ ├── checkpoint-decoder │ │ ├── checkpoint-encoder │ │ └── checkpoint-param └── utils.py └── smoothed_kitsune ├── classifier ├── autoencoder.py ├── corClust.py ├── detect.py └── preprocess.py ├── data ├── mirai.zip └── ssdp.zip ├── main.py ├── save └── mirai │ ├── certify │ ├── certification_results │ ├── certified_accuracy_robustness_radius_curve.png │ ├── checkpoint-distribution-transformer │ └── t │ └── train │ ├── checkpoint-ae-0 │ ├── checkpoint-ae-1 │ ├── checkpoint-ae-10 │ ├── checkpoint-ae-11 │ ├── checkpoint-ae-12 │ ├── checkpoint-ae-13 │ ├── checkpoint-ae-14 │ ├── checkpoint-ae-15 │ ├── checkpoint-ae-16 │ ├── checkpoint-ae-17 │ ├── checkpoint-ae-18 │ ├── checkpoint-ae-19 │ ├── checkpoint-ae-2 │ ├── checkpoint-ae-20 │ ├── checkpoint-ae-21 │ ├── checkpoint-ae-22 │ ├── checkpoint-ae-23 │ ├── checkpoint-ae-24 │ ├── checkpoint-ae-25 │ ├── checkpoint-ae-3 │ ├── checkpoint-ae-4 │ ├── checkpoint-ae-5 │ ├── checkpoint-ae-6 │ ├── checkpoint-ae-7 │ ├── checkpoint-ae-8 │ ├── checkpoint-ae-9 │ ├── checkpoint-fm │ └── checkpoint-norm └── utils.py /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Kai Wang 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # BARS: Local Robustness Certification for Traffic Analysis 2 | 3 |

4 | 5 | 6 |

7 | 8 | > *BARS: Local Robustness Certification for Deep Learning based Traffic Analysis Systems (NDSS'23)* 9 | 10 | --- 11 | 12 | ![MIT license](https://img.shields.io/badge/license-MIT-brightgreen.svg) 13 | ![Python language](https://img.shields.io/badge/language-Python-blue.svg) 14 | ![PyTorch framework](https://img.shields.io/badge/framework-PyTorch-orange.svg) 15 | 16 | ## Introduction 17 | 18 | **BARS** is a general local robustness certification framework for Deep Learning (DL) based traffic analysis systems based on **B**oundary **A**daptive **R**andomized **S**moothing. Against adversarial perturbations, local robustness certification is used to certify whether a DL-based model is robust in the neighborhood of a sample. 19 | 20 | **BARS** optimizes the smoothing noise of randomized smoothing and provide tighter robustness guarantee for the traffic analysis domain. The pipeline includes four steps: 21 | - [Build Distribution Transformer.](bars/distribution_transformer.py) 22 | - [Optimize the noise shape.](bars/optimizing_noise.py) 23 | - [Optimize the noise scale.](bars/optimizing_noise.py) 24 | - [Certify robustness.](bars/smoothing.py) 25 | 26 | **BARS** supports three traffic analysis systems: 27 | - [Zero-Positive Network Intrusion Detection System, Kitsune (NDSS'18)](smoothed_kitsune) 28 | - [Concept Drift Detection System, CADE (USENIX Security'21)](smoothed_cade) 29 | - [Supervised Multi-Classification System, ACID (INFOCOM'21)](smoothed_acid) 30 | 31 | ## Quick Start 32 | 33 | ### 1. Environmental Setup 34 | 35 | - Basic **BARS**: `pip install -r requirement_bars.txt` 36 | - Smoothed Kitsune: `pip install -r requirement_kitsune.txt` 37 | - Smoothed CADE: `pip install -r requirement_cade.txt` 38 | - Smoothed ACID: `pip install -r requirement_acid.txt` 39 | 40 | ### 2. Running Program 41 | 42 | - Please run `python main.py`. 43 | 44 | - Program arguments are set at the beginning of `main.py`. 45 | 46 | ## Citation 47 | TBA. 48 | -------------------------------------------------------------------------------- /bars/distribution_transformer.py: -------------------------------------------------------------------------------- 1 | import math 2 | 3 | import torch 4 | import torch.nn as nn 5 | 6 | class Gaussian_CDF(nn.Module): 7 | def __init__(self): 8 | super(Gaussian_CDF, self).__init__() 9 | 10 | self.w = 1.6638520956039429 11 | 12 | def forward(self, x: torch.tensor) -> torch.tensor: 13 | return 1 / (1 + torch.exp(- self.w * x)) 14 | 15 | class Distribution_Transformer_Gaussian(nn.Module): 16 | def __init__(self, d: int): 17 | """ 18 | d: the number of feature vector dimensions 19 | """ 20 | super(Distribution_Transformer_Gaussian, self).__init__() 21 | 22 | self.d = d 23 | 24 | self.weight_shared = nn.Parameter(torch.zeros((1), requires_grad=True)) 25 | self.weight_independent = nn.Parameter(torch.zeros((1, self.d), requires_grad=True)) 26 | 27 | self.weight_init() 28 | 29 | def weight_init(self): 30 | self.weight_shared.data.normal_(0, 0.01) 31 | self.weight_independent.data.normal_(0, 0.01) 32 | 33 | def forward(self, z: torch.tensor) -> torch.tensor: 34 | assert self.d == z.size(1) 35 | x = torch.zeros_like(z, device=z.device) 36 | x += z * self.weight_shared.abs() 37 | x += z * self.weight_independent.abs() 38 | 39 | return x 40 | 41 | def get_weight(self) -> torch.tensor: 42 | return torch.cat([self.weight_shared.abs().view(-1), self.weight_independent.abs().view(-1)], 0) 43 | 44 | class Arctan_Hf(nn.Module): 45 | def __init__(self): 46 | super(Arctan_Hf, self).__init__() 47 | 48 | self.eps = 1e-16 49 | 50 | def forward(self, x: torch.tensor) -> torch.tensor: 51 | x = torch.clamp(x, 0 + self.eps, 1 - self.eps) 52 | return torch.tan(math.pi * x - math.pi / 2) 53 | 54 | class ISRU_Hf(nn.Module): 55 | def __init__(self): 56 | super(ISRU_Hf, self).__init__() 57 | 58 | self.eps = 1e-16 59 | 60 | def forward(self, x: torch.tensor) -> torch.tensor: 61 | x = torch.clamp(x, 0 + self.eps, 1 - self.eps) 62 | return (2 * x - 1) / torch.sqrt(1 - torch.pow(2 * x - 1, 2) + self.eps) 63 | 64 | class Distribution_Transformer_ISRU(nn.Module): 65 | def __init__(self, d: int): 66 | """ 67 | d: the number of feature vector dimensions 68 | """ 69 | super(Distribution_Transformer_ISRU, self).__init__() 70 | 71 | self.d = d 72 | self.cdf_ppf = nn.ModuleList([ 73 | nn.Sequential(Gaussian_CDF(), ISRU_Hf()), 74 | ]) 75 | 76 | self.weight_shared = nn.Parameter(torch.zeros((len(self.cdf_ppf)), requires_grad=True)) 77 | self.weight_independent = nn.Parameter(torch.zeros((len(self.cdf_ppf), self.d), requires_grad=True)) 78 | 79 | self.weight_init() 80 | 81 | def weight_init(self): 82 | self.weight_shared.data.normal_(0, 0.01) 83 | self.weight_independent.data.normal_(0, 0.01) 84 | 85 | def forward(self, z: torch.tensor) -> torch.tensor: 86 | assert self.d == z.size(1) 87 | x = torch.zeros_like(z, device=z.device) 88 | for i in range(len(self.cdf_ppf)): 89 | x += self.cdf_ppf[i](z) * self.weight_shared[i].abs() 90 | x += self.cdf_ppf[i](z) * self.weight_independent[i, :].abs() 91 | 92 | return x 93 | 94 | def get_weight(self) -> torch.tensor: 95 | return torch.cat([self.weight_shared.abs().view(-1), self.weight_independent.abs().view(-1)], 0) 96 | 97 | class Distribution_Transformer_ISRU_Gaussian(nn.Module): 98 | def __init__(self, d: int): 99 | """ 100 | d: the number of feature vector dimensions 101 | """ 102 | super(Distribution_Transformer_ISRU_Gaussian, self).__init__() 103 | 104 | self.d = d 105 | self.cdf_ppf = nn.ModuleList([ 106 | nn.Sequential(Gaussian_CDF(), ISRU_Hf()), 107 | nn.Identity(), 108 | ]) 109 | 110 | self.weight_shared = nn.Parameter(torch.zeros((len(self.cdf_ppf)), requires_grad=True)) 111 | self.weight_independent = nn.Parameter(torch.zeros((len(self.cdf_ppf), self.d), requires_grad=True)) 112 | 113 | self.weight_init() 114 | 115 | def weight_init(self): 116 | self.weight_shared.data.normal_(0, 0.01) 117 | self.weight_independent.data.normal_(0, 0.01) 118 | 119 | def forward(self, z: torch.tensor) -> torch.tensor: 120 | assert self.d == z.size(1) 121 | x = torch.zeros_like(z, device=z.device) 122 | for i in range(len(self.cdf_ppf)): 123 | x += self.cdf_ppf[i](z) * self.weight_shared[i].abs() 124 | x += self.cdf_ppf[i](z) * self.weight_independent[i, :].abs() 125 | 126 | return x 127 | 128 | def get_weight(self) -> torch.tensor: 129 | return torch.cat([self.weight_shared.abs().view(-1), self.weight_independent.abs().view(-1)], 0) 130 | 131 | class Distribution_Transformer_ISRU_Gaussian_Arctan(nn.Module): 132 | def __init__(self, d: int): 133 | """ 134 | d: the number of feature vector dimensions 135 | """ 136 | super(Distribution_Transformer_ISRU_Gaussian_Arctan, self).__init__() 137 | 138 | self.d = d 139 | self.cdf_ppf = nn.ModuleList([ 140 | nn.Sequential(Gaussian_CDF(), ISRU_Hf()), 141 | nn.Identity(), 142 | nn.Sequential(Gaussian_CDF(), Arctan_Hf()), 143 | ]) 144 | 145 | self.weight_shared = nn.Parameter(torch.zeros((len(self.cdf_ppf)), requires_grad=True)) 146 | self.weight_independent = nn.Parameter(torch.zeros((len(self.cdf_ppf), self.d), requires_grad=True)) 147 | 148 | self.weight_init() 149 | 150 | def weight_init(self): 151 | self.weight_shared.data.normal_(0, 0.01) 152 | self.weight_independent.data.normal_(0, 0.01) 153 | 154 | def forward(self, z: torch.tensor) -> torch.tensor: 155 | assert self.d == z.size(1) 156 | x = torch.zeros_like(z, device=z.device) 157 | for i in range(len(self.cdf_ppf)): 158 | x += self.cdf_ppf[i](z) * self.weight_shared[i].abs() 159 | x += self.cdf_ppf[i](z) * self.weight_independent[i, :].abs() 160 | 161 | return x 162 | 163 | def get_weight(self) -> torch.tensor: 164 | return torch.cat([self.weight_shared.abs().view(-1), self.weight_independent.abs().view(-1)], 0) 165 | 166 | # Different feature noise distributions 167 | distribution_transformers = { 168 | "gaussian": Distribution_Transformer_Gaussian, 169 | "isru": Distribution_Transformer_ISRU, 170 | "isru_gaussian": Distribution_Transformer_ISRU_Gaussian, 171 | "isru_gaussian_arctan": Distribution_Transformer_ISRU_Gaussian_Arctan, 172 | } 173 | 174 | class Distribution_Transformer_Loss_Kitsune(nn.Module): 175 | def __init__(self, lambda_: float, ad_threshold: float): 176 | """ 177 | lambda_: the weight of the regularizer 178 | ad_threshold: threshold for anomaly detection 179 | """ 180 | super(Distribution_Transformer_Loss_Kitsune, self).__init__() 181 | 182 | self.lambda_ = lambda_ 183 | self.ad_threshold = ad_threshold 184 | 185 | self.mu = 1 / 3 186 | 187 | def forward(self, rmse: torch.tensor, weight: torch.tensor) -> torch.tensor: 188 | idx_wrong = torch.where(rmse > self.ad_threshold)[0] 189 | loss_wrong = (1 / rmse.size(0)) * torch.sum(rmse[idx_wrong] - self.ad_threshold) if idx_wrong.size(0) > 0 else torch.tensor(0., requires_grad=True) 190 | idx_correct = torch.where(rmse <= self.ad_threshold)[0] 191 | loss_correct = (1 / rmse.size(0)) * torch.sum(self.ad_threshold - rmse[idx_correct]) if idx_correct.size(0) > 0 else torch.tensor(0., requires_grad=True) 192 | 193 | regularizer = torch.log(1 + torch.exp(-weight.abs())).sum() 194 | 195 | return self.mu * (loss_wrong + loss_correct) + self.lambda_ * regularizer 196 | 197 | class Distribution_Transformer_Loss_CADE(nn.Module): 198 | def __init__(self, lambda_: float, mad_threshold: float): 199 | super(Distribution_Transformer_Loss_CADE, self).__init__() 200 | """ 201 | lambda_: the weight of the regularizer 202 | mad_threshold: MAD threshold for detecting drift 203 | """ 204 | self.lambda_ = lambda_ 205 | self.mad_threshold = mad_threshold 206 | 207 | self.mu = 1 / 3 208 | 209 | def forward(self, score: torch.tensor, weight: torch.tensor): 210 | idx_wrong = torch.where(score > self.mad_threshold)[0] 211 | loss_wrong = (1 / score.size(0)) * torch.sum(score[idx_wrong] - self.mad_threshold) if idx_wrong.size(0) > 0 else torch.tensor(0., requires_grad=True) 212 | idx_correct = torch.where(score <= self.mad_threshold)[0] 213 | loss_correct = (1 / score.size(0)) * torch.sum(self.mad_threshold - score[idx_correct]) if idx_correct.size(0) > 0 else torch.tensor(0., requires_grad=True) 214 | 215 | regularizer = torch.log(1 + torch.exp(-weight.abs())).sum() 216 | 217 | return self.mu * (loss_wrong + loss_correct) + self.lambda_ * regularizer 218 | 219 | class Distribution_Transformer_Loss_ACID(nn.Module): 220 | def __init__(self, lambda_: float, certify_class: int): 221 | super(Distribution_Transformer_Loss_ACID, self).__init__() 222 | """ 223 | lambda_: the weight of the regularizer 224 | certify_class: the certified class as training target 225 | """ 226 | self.criterion = nn.MultiLabelSoftMarginLoss(reduction='sum') 227 | self.lambda_ = lambda_ 228 | self.certify_class = certify_class 229 | 230 | self.mu = 1 / 3 231 | 232 | def forward(self, score: torch.tensor, weight: torch.tensor): 233 | idx_wrong = torch.where(score.argmax(1) != self.certify_class)[0] 234 | idx_correct = torch.where(score.argmax(1) == self.certify_class)[0] 235 | target = torch.ones_like(score, device=score.device) 236 | target[idx_wrong, :] = 0 237 | target[idx_wrong, self.certify_class] = 1 238 | target[idx_correct, :] = 1 239 | target[idx_correct, self.certify_class] = 0 240 | loss_wrong = (1 / score.size(0)) * self.criterion(score[idx_wrong, :], target[idx_wrong, :]) if idx_wrong.size(0) > 0 else torch.tensor(0., requires_grad=True) 241 | loss_correct = (1 / score.size(0)) * self.criterion(score[idx_correct, :], target[idx_correct, :]) if idx_correct.size(0) > 0 else torch.tensor(0., requires_grad=True) 242 | 243 | regularizer = torch.log(1 + torch.exp(-weight.abs())).sum() 244 | 245 | return self.mu * (loss_wrong + loss_correct) + self.lambda_ * regularizer 246 | 247 | loss_functions = { 248 | "kitsune": Distribution_Transformer_Loss_Kitsune, 249 | "cade": Distribution_Transformer_Loss_CADE, 250 | "acid": Distribution_Transformer_Loss_ACID 251 | } 252 | -------------------------------------------------------------------------------- /bars/optimizing_noise.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | from typing import * 4 | 5 | import torch 6 | import torch.nn as nn 7 | import torch.optim as optim 8 | from torch.utils.data import TensorDataset, DataLoader 9 | 10 | from smoothing import Smooth2 11 | 12 | num_samples_shape = 10000 13 | num_samples_scale = 1000 14 | 15 | def optimizing_noise( 16 | x_train: torch.tensor, # input data for optimization 17 | pred_train: int, # classifier prediction of input data 18 | classifier: nn.Module, # classifier model 19 | noise_generator: nn.Module, # noise generator model 20 | criterion_shape: nn.Module, # loss function for optimizing noise shape 21 | learning_rate_shape: float, # learning rate for optimizing noise shape 22 | nt_shape: int, # number of noised samples for optimizing noise shape 23 | num_epochs_shape: int, # epochs for optimizing noise shape 24 | d: int, # number of feature dimensions 25 | num_classes: int, # number of certified classes 26 | n0: int, # number of noised samples for identify cA 27 | n: int, # number of noised samples for estimate pA 28 | alpha: float, # failure probability 29 | init_step_size_scale: float, # initial update step size of t for optimzing noise scale 30 | init_ptb_t_scale: float, # initial perturbation of t for optimzing noise scale 31 | decay_factor_scale: float, # decay factor for optimzing noise scale 32 | max_decay_scale: int, # maximum decay times for optimzing noise scale 33 | max_iter_scale: int, # maximum iteration times for optimzing noise scale 34 | batch_size_iteration: int, # batch size of certified samples for robustness certification 35 | batch_size_memory: int, # batch size of noised samples for robustness certification 36 | print_step: int, # step size for showing certification progress 37 | save_dir: str # saving directory for robustness certification 38 | ): 39 | 40 | assert classifier.device == noise_generator.device 41 | 42 | print("***** Optimize noise shape *****") 43 | 44 | idx = torch.arange(x_train.shape[0]) 45 | idx = idx[torch.randperm(idx.size(0))] 46 | idx = torch.sort(idx[:min(num_samples_shape, idx.shape[0])])[0] 47 | x_train = x_train[idx, :] 48 | 49 | dataset = TensorDataset(x_train) 50 | data_loader = DataLoader(dataset, batch_size=batch_size_iteration, shuffle=True) 51 | 52 | opt = optim.Adam(noise_generator.distribution_transformer.parameters(), lr=learning_rate_shape) 53 | 54 | noise_norm = noise_generator.sample_norm(nt_shape).repeat(1, batch_size_iteration).view(-1, noise_generator.d) 55 | 56 | loss_record = AverageMeter() 57 | classifier.train() 58 | for epoch in range(0, num_epochs_shape + 1): 59 | for i, (X,) in zip(range(1, len(data_loader) + 1), data_loader): 60 | X = X.to(classifier.device) 61 | if X.size(0) < batch_size_iteration: 62 | noise_idx = torch.cat([torch.arange(X.size(0)) + i * batch_size_iteration \ 63 | for i in range(nt_shape)], 0).to(X.device) 64 | else: 65 | noise_idx = torch.arange(noise_norm.size(0)).to(X.device) 66 | noise_feat = noise_generator.norm_to_feat(noise_norm[noise_idx, :]) 67 | 68 | score = classifier.score(X.repeat((nt_shape, 1)).to(X.device), noise_feat) 69 | loss = criterion_shape(score, noise_generator.get_weight()) 70 | 71 | if epoch > 0: 72 | opt.zero_grad() 73 | loss.backward(retain_graph=True) 74 | opt.step() 75 | 76 | loss_record.update(loss.item()) 77 | if (i % print_step == 0) or ((print_step > len(data_loader)) and (i == len(data_loader))): 78 | print("Batch: [%d/%d][%d/%d] | Loss: %.6f" % (epoch, num_epochs_shape, i, len(data_loader), loss_record.val)) 79 | 80 | print("Epoch: [%d/%d] | Loss (Avg): %.6f" % (epoch, num_epochs_shape, loss_record.avg)) 81 | loss_record.reset() 82 | 83 | if not os.path.exists(save_dir): 84 | os.makedirs(save_dir) 85 | torch.save(noise_generator.distribution_transformer, os.path.join(save_dir, "checkpoint-distribution-transformer")) 86 | 87 | print("***** Optimize noise scale *****") 88 | 89 | idx = torch.arange(x_train.shape[0]) 90 | idx = idx[torch.randperm(idx.size(0))] 91 | idx = torch.sort(idx[:min(num_samples_scale, idx.shape[0])])[0] 92 | x_train = x_train[idx, :] 93 | 94 | dataset = TensorDataset(x_train) 95 | data_loader = DataLoader(dataset, batch_size=batch_size_iteration, shuffle=True) 96 | 97 | classifier.eval() 98 | smoothed_classifier = Smooth2(classifier, d, num_classes, noise_generator, classifier.device) 99 | ptb_t_scale = init_ptb_t_scale 100 | step_size_scale = init_step_size_scale 101 | decay_scale = 0 102 | iter_scale = 0 103 | t = 0.0 104 | grad_sign_last = 1 105 | torch.set_grad_enabled(False) 106 | while (iter_scale < max_iter_scale) and (decay_scale < max_decay_scale): 107 | cA_record = np.array([]) 108 | robust_radius_record = np.array([]) 109 | for (X,) in data_loader: 110 | X = X.to(classifier.device) 111 | 112 | cA, _, robust_radius = smoothed_classifier.bars_certify(X, n0, n, t, alpha, batch_size_memory) 113 | 114 | cA_record = np.concatenate([cA_record, cA], 0) 115 | robust_radius_record = np.concatenate([robust_radius_record, robust_radius], 0) 116 | 117 | mean_robust_radius = calc_mean_robust_radius(pred_train, cA_record, robust_radius_record) 118 | 119 | cA_record = np.array([]) 120 | robust_radius_record = np.array([]) 121 | for (X,) in data_loader: 122 | X = X.to(classifier.device) 123 | 124 | cA, _, robust_radius = smoothed_classifier.bars_certify(X, n0, n, t + ptb_t_scale, alpha, batch_size_memory) 125 | 126 | cA_record = np.concatenate([cA_record, cA], 0) 127 | robust_radius_record = np.concatenate([robust_radius_record, robust_radius], 0) 128 | 129 | mean_robust_radius_ptb = calc_mean_robust_radius(pred_train, cA_record, robust_radius_record) 130 | 131 | grad_t = (mean_robust_radius_ptb - mean_robust_radius) / ptb_t_scale 132 | if np.sign(grad_t) != grad_sign_last: 133 | ptb_t_scale *= decay_factor_scale 134 | step_size_scale *= decay_factor_scale 135 | grad_sign_last = np.sign(grad_t) 136 | decay_scale += 1 137 | t = t + step_size_scale * np.sign(grad_t) 138 | 139 | cA_record = np.array([]) 140 | robust_radius_record = np.array([]) 141 | for (X,) in data_loader: 142 | X = X.to(classifier.device) 143 | 144 | cA, _, robust_radius = smoothed_classifier.bars_certify(X, n0, n, t, alpha, batch_size_memory) 145 | 146 | cA_record = np.concatenate([cA_record, cA], 0) 147 | robust_radius_record = np.concatenate([robust_radius_record, robust_radius], 0) 148 | 149 | mean_robust_radius_last = calc_mean_robust_radius(pred_train, cA_record, robust_radius_record) 150 | 151 | iter_scale += 1 152 | print("Iter: [%d] | t: %.6e | Robust radius: %.6e | Step size: %.6e | Grad sign: %d" % \ 153 | (iter_scale, t, mean_robust_radius_last, step_size_scale, grad_sign_last)) 154 | 155 | torch.set_grad_enabled(True) 156 | torch.cuda.empty_cache() 157 | 158 | if not os.path.exists(save_dir): 159 | os.makedirs(save_dir) 160 | w = open(os.path.join(save_dir, "t"), "w") 161 | w.write("%.6e" % (t)) 162 | w.close() 163 | 164 | class AverageMeter(object): 165 | def __init__(self): 166 | self.reset() 167 | 168 | def reset(self): 169 | self.val = 0 170 | self.sum = 0 171 | self.count = 0 172 | self.avg = 0 173 | 174 | def update(self, val): 175 | self.val = val 176 | self.sum += val 177 | self.count += 1 178 | self.avg = self.sum / self.count 179 | 180 | def calc_mean_robust_radius(pred: int, cA: np.ndarray, radius_feat: np.ndarray) -> np.ndarray: 181 | assert cA.shape == radius_feat.shape 182 | 183 | return np.mean(np.array((pred == cA), dtype=float) * radius_feat) -------------------------------------------------------------------------------- /bars/smoothing.py: -------------------------------------------------------------------------------- 1 | from scipy.stats import norm, binom_test 2 | import numpy as np 3 | from math import ceil, floor 4 | from statsmodels.stats.proportion import proportion_confint 5 | from typing import * 6 | 7 | import torch 8 | 9 | class Smooth(object): 10 | """A smoothed classifier g """ 11 | 12 | # to abstain, Smooth returns this int 13 | ABSTAIN = -1 14 | 15 | def __init__(self, base_classifier: torch.nn.Module, num_classes: int, sigma: float): 16 | """ 17 | :param base_classifier: maps from [batch x channel x height x width] to [batch x num_classes] 18 | :param num_classes: 19 | :param sigma: the noise level hyperparameter 20 | """ 21 | self.base_classifier = base_classifier 22 | self.num_classes = num_classes 23 | self.sigma = sigma 24 | 25 | def certify(self, x: torch.tensor, n0: int, n: int, alpha: float, batch_size: int) -> (int, float): 26 | """ Monte Carlo algorithm for certifying that g's prediction around x is constant within some L2 radius. 27 | With probability at least 1 - alpha, the class returned by this method will equal g(x), and g's prediction will 28 | robust within a L2 ball of radius R around x. 29 | 30 | :param x: the input [feature_size] 31 | :param n0: the number of Monte Carlo samples to use for selection 32 | :param n: the number of Monte Carlo samples to use for estimation 33 | :param alpha: the failure probability 34 | :param batch_size: batch size to use when evaluating the base classifier 35 | :return: (predicted class, certified radius) 36 | in the case of abstention, the class will be ABSTAIN and the radius 0. 37 | """ 38 | self.base_classifier.eval() 39 | # draw samples of f(x+ epsilon) 40 | counts_selection = self._sample_noise(x, n0, batch_size) 41 | # use these samples to take a guess at the top class 42 | cAHat = counts_selection.argmax().item() 43 | # draw more samples of f(x + epsilon) 44 | counts_estimation = self._sample_noise(x, n, batch_size) 45 | # use these samples to estimate a lower bound on pA 46 | nA = counts_estimation[cAHat].item() 47 | pABar = self._lower_confidence_bound(nA, n, alpha) 48 | if pABar < 0.5: 49 | return Smooth.ABSTAIN, 0.0 50 | else: 51 | radius = self.sigma * norm.ppf(pABar) 52 | return cAHat, radius 53 | 54 | def predict(self, x: torch.tensor, n: int, alpha: float, batch_size: int) -> int: 55 | """ Monte Carlo algorithm for evaluating the prediction of g at x. With probability at least 1 - alpha, the 56 | class returned by this method will equal g(x). 57 | 58 | This function uses the hypothesis test described in https://arxiv.org/abs/1610.03944 59 | for identifying the top category of a multinomial distribution. 60 | 61 | :param x: the input [channel x height x width] 62 | :param n: the number of Monte Carlo samples to use 63 | :param alpha: the failure probability 64 | :param batch_size: batch size to use when evaluating the base classifier 65 | :return: the predicted class, or ABSTAIN 66 | """ 67 | self.base_classifier.eval() 68 | counts = self._sample_noise(x, n, batch_size) 69 | top2 = counts.argsort()[::-1][:2] 70 | count1 = counts[top2[0]] 71 | count2 = counts[top2[1]] 72 | if binom_test(count1, count1 + count2, p=0.5) > alpha: 73 | return Smooth.ABSTAIN 74 | else: 75 | return top2[0] 76 | 77 | def _sample_noise(self, x: torch.tensor, num: int, batch_size) -> np.ndarray: 78 | """ Sample the base classifier's prediction under noisy corruptions of the input x. 79 | 80 | :param x: the input [feature_size] 81 | :param num: number of samples to collect 82 | :param batch_size: 83 | :return: an ndarray[int] of length num_classes containing the per-class counts 84 | """ 85 | with torch.no_grad(): 86 | counts = np.zeros(self.num_classes, dtype=int) 87 | for _ in range(ceil(num / batch_size)): 88 | this_batch_size = min(batch_size, num) 89 | num -= this_batch_size 90 | 91 | batch = x.repeat((this_batch_size, 1)) 92 | noise = torch.randn_like(batch, device=batch.device) * self.sigma 93 | predictions = self.base_classifier((batch + noise).unsqueeze(1).unsqueeze(3)).argmax(1) 94 | counts += self._count_arr(predictions.cpu().numpy(), self.num_classes) 95 | return counts 96 | 97 | def _count_arr(self, arr: np.ndarray, length: int) -> np.ndarray: 98 | counts = np.zeros(length, dtype=int) 99 | for idx in arr: 100 | counts[idx] += 1 101 | return counts 102 | 103 | def _lower_confidence_bound(self, NA: int, N: int, alpha: float) -> float: 104 | """ Returns a (1 - alpha) lower confidence bound on a bernoulli proportion. 105 | 106 | This function uses the Clopper-Pearson method. 107 | 108 | :param NA: the number of "successes" 109 | :param N: the number of total draws 110 | :param alpha: the confidence level 111 | :return: a lower bound on the binomial proportion which holds true w.p at least (1 - alpha) over the samples 112 | """ 113 | return proportion_confint(NA, N, alpha=2 * alpha, method="beta")[0] 114 | 115 | def _upper_confidence_bound(self, NA: int, N: int, alpha: float) -> float: 116 | """ Returns a (1 - alpha) lower confidence bound on a bernoulli proportion. 117 | 118 | This function uses the Clopper-Pearson method. 119 | 120 | :param NA: the number of "successes" 121 | :param N: the number of total draws 122 | :param alpha: the confidence level 123 | :return: a lower bound on the binomial proportion which holds true w.p at least (1 - alpha) over the samples 124 | """ 125 | return proportion_confint(NA, N, alpha=2 * alpha, method="beta")[1] 126 | 127 | # Certifying multiple samples in parallel 128 | class Smooth2(Smooth): 129 | def __init__(self, base_classifier: torch.nn.Module, d: int, num_classes: int, noise_generator: object, device: object): 130 | """ 131 | :param base_classifier: maps from [batch x channel x height x width] to [batch x num_classes] 132 | :param num_classes: the number of classes, Kitsune: 2, CADE: 2, ACID: 4 133 | :param noise_generator: optimized noise generator 134 | :param d: the number of feature vector dimensions 135 | :param device: cpu / cuda 136 | """ 137 | self.base_classifier = base_classifier 138 | self.d = d 139 | self.num_classes = num_classes 140 | self.noise_generator = noise_generator 141 | self.device = device 142 | 143 | assert noise_generator.d == self.d 144 | assert noise_generator.device == self.device 145 | 146 | self.eps = 1e-16 147 | 148 | def _certify2(self, x: torch.tensor, n0: int, n: int, t: float, alpha: float, batch_size_memory: int) -> (np.ndarray, np.ndarray): 149 | """ Monte Carlo algorithm for certifying that g's prediction around x is constant within some L2 radius. 150 | With probability at least 1 - alpha, the class returned by this method will equal g(x), and g's prediction will 151 | robust within a L2 ball of radius R around x. 152 | 153 | :param x: the input [batch_size_iteration x feature_size] 154 | :param n0: the number of Monte Carlo samples to use for selection 155 | :param n: the number of Monte Carlo samples to use for estimation 156 | :param alpha: the failure probability 157 | :param batch_size_memory: maximum batch size in memory for parallel smoothing 158 | :return: (predicted class: np.ndarray, certified normalized robustness radius: np.ndarray) 159 | in the case of abstention, the class will be ABSTAIN and the radius 0. 160 | """ 161 | self.base_classifier.eval() 162 | # draw samples of f(x+ epsilon) 163 | counts_selection = self._sample_noise2(x, n0, t, batch_size_memory) 164 | # use these samples to take a guess at the top class 165 | cAHat = counts_selection.argmax(1) 166 | # draw more samples of f(x + epsilon) 167 | counts_estimation = self._sample_noise2(x, n, t, batch_size_memory) 168 | # use these samples to estimate a lower bound on pA 169 | nA = counts_estimation[np.arange(cAHat.shape[0]), cAHat] 170 | pABar = self._lower_confidence_bound2(nA, n, alpha) 171 | # use pA lower bound to calculate normalized robustness radius 172 | radius_norm = norm.ppf(pABar) 173 | 174 | # when pABar < 0.5, abstain from making robustness certification 175 | idx_abstain = np.where(pABar < 0.5)[0] 176 | cAHat[idx_abstain] = self.ABSTAIN 177 | radius_norm[idx_abstain] = 0 178 | 179 | return cAHat, radius_norm 180 | 181 | def _sample_noise2(self, x: torch.tensor, num: int, t: float, batch_size_memory: int) -> np.ndarray: 182 | """ Sample the base classifier's prediction under noisy corruptions of the input x. 183 | 184 | :param x: the input [batch_size_iteration x feature_size] 185 | :param num: number of samples to collect 186 | :param t: scale factor 187 | :param batch_size_memory: maximum batch size in memory for parallel smoothing 188 | :return: an ndarray[int] of length num_classes containing the per-class counts 189 | """ 190 | # batch size for iteration should be less than or equal to maximum batch size in memory 191 | assert x.size(0) <= batch_size_memory 192 | with torch.no_grad(): 193 | counts = np.zeros((x.size(0), self.num_classes), dtype=int) 194 | while num > 0: 195 | batch_size_per_example = min(floor(batch_size_memory / x.size(0)), num) 196 | num -= batch_size_per_example 197 | 198 | batch = x.repeat((batch_size_per_example, 1)) 199 | noise = self.noise_generator.sample_feat(x.size(0) * batch_size_per_example) * t 200 | predictions = self.base_classifier(batch, noise) 201 | counts += self._count_arr2(predictions.cpu().numpy(), x.size(0)) 202 | 203 | return counts 204 | 205 | def _count_arr2(self, arr: np.ndarray, length: int) -> np.ndarray: 206 | counts = np.zeros((length, self.num_classes), dtype=int) 207 | arr = arr.reshape(-1, length).T 208 | for c in range(self.num_classes): 209 | counts[:, c] += np.array(arr == c, dtype=int).sum(1) 210 | return counts 211 | 212 | def _lower_confidence_bound2(self, NA: np.ndarray, N: int, alpha: float) -> np.ndarray: 213 | """ Returns a (1 - alpha) lower confidence bound on a bernoulli proportion. 214 | 215 | This function uses the Clopper-Pearson method. 216 | 217 | :param NA: the number of "successes" for each example 218 | :param N: the number of total draws 219 | :param alpha: the confidence level 220 | :return: an ndarray of lower bound on the binomial proportion which holds true w.p at least (1 - alpha) over the samples 221 | """ 222 | return np.array([proportion_confint(NA_, N, alpha=2 * alpha, method="beta")[0] for NA_ in NA]) 223 | 224 | def bars_certify(self, x: torch.tensor, n0: int, n: int, t: float, alpha: float, batch_size_memory: int) -> (np.ndarray, np.ndarray, np.ndarray): 225 | """ 226 | :param x: the input [batch_size_iteration x feature_size] 227 | :param n0: the number of Monte Carlo samples to use for selection 228 | :param n: the number of Monte Carlo samples to use for estimation 229 | :param alpha: the failure probability 230 | :param batch_size_memory: maximum batch size in memory for parallel smoothing 231 | :return: (predicted class: np.ndarray, 232 | certified dimension-wise robustness radius vector (feature space): np.ndarray, 233 | certified dimension-heterogeneous robustness radius (feature space): np.ndarray) 234 | in the case of abstention, the class will be ABSTAIN and the radius 0. 235 | """ 236 | cAHat, radius_norm = self._certify2(x, n0, n, t, alpha, batch_size_memory) 237 | radius_norm_dim = torch.tensor(radius_norm).unsqueeze(1).repeat((1, self.d)).to(self.device) 238 | radius_feat_dim = self.noise_generator.norm_to_feat(radius_norm_dim).cpu().numpy() * t 239 | radius_feat = radius_feat_dim.mean(1) 240 | 241 | return cAHat, radius_feat_dim, radius_feat 242 | 243 | class Noise(object): 244 | def __init__(self, distribution_transformer: torch.nn.Module, d: int, device: object): 245 | """ 246 | distribution_transformer: Distribution Transformer model 247 | d: the number of feature vector dimensions 248 | device: cpu / cuda 249 | """ 250 | self.distribution_transformer = distribution_transformer 251 | self.d = d 252 | self.device = device 253 | 254 | def sample_norm(self, n: int) -> torch.tensor: 255 | return torch.randn((n, self.d), device=self.device) 256 | 257 | def norm_to_feat(self, z: torch.tensor) -> torch.tensor: 258 | return self.distribution_transformer(z).to(self.device) 259 | 260 | def sample_feat(self, n: int) -> torch.tensor: 261 | z = self.sample_norm(n) 262 | return self.norm_to_feat(z) 263 | 264 | def get_weight(self) -> torch.tensor: 265 | return self.distribution_transformer.get_weight() -------------------------------------------------------------------------------- /images/overview_certification_stage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/images/overview_certification_stage.png -------------------------------------------------------------------------------- /images/overview_training_stage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/images/overview_training_stage.png -------------------------------------------------------------------------------- /requirements_acid.txt: -------------------------------------------------------------------------------- 1 | numpy==1.21.0 2 | tqdm==4.61.1 3 | matplotlib==3.4.2 4 | seaborn==0.11.1 5 | scikit-learn==0.24.2 6 | torch==1.7.1+cu110 -------------------------------------------------------------------------------- /requirements_bars.txt: -------------------------------------------------------------------------------- 1 | scipy==1.7.0 2 | numpy==1.21.0 3 | statsmodels==0.12.2 4 | torch==1.7.1+cu110 -------------------------------------------------------------------------------- /requirements_cade.txt: -------------------------------------------------------------------------------- 1 | numpy==1.21.0 2 | tqdm==4.61.1 3 | matplotlib==3.4.2 4 | seaborn==0.11.1 5 | scikit-learn==0.24.2 6 | torch==1.7.1+cu110 -------------------------------------------------------------------------------- /requirements_kitsune.txt: -------------------------------------------------------------------------------- 1 | numpy==1.21.0 2 | tqdm==4.61.1 3 | matplotlib==3.4.2 4 | seaborn==0.11.1 5 | scikit-learn==0.24.2 6 | scipy==1.7.0 7 | torch==1.7.1+cu110 -------------------------------------------------------------------------------- /smoothed_acid/classifier/classifier.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | class AdaptiveClusteringClassifier(nn.Module): 5 | def __init__(self, encoder_list, kernel_net_list, device): 6 | super(AdaptiveClusteringClassifier, self).__init__() 7 | 8 | assert len(encoder_list) == len(kernel_net_list) 9 | 10 | self.encoder_list = encoder_list 11 | self.kernel_net_list = kernel_net_list 12 | self.num_kernels = len(encoder_list) 13 | 14 | self.device = device 15 | 16 | def forward(self, x: torch.tensor, noise: torch.tensor=0) -> torch.tensor: 17 | o = torch.tensor([], device=self.device) 18 | for i in range(self.num_kernels): 19 | o = torch.cat([o, self.kernel_net_list[i](self.encoder_list[i](x + noise))], 1) 20 | 21 | return o.argmax(1) 22 | 23 | def score(self, x: torch.tensor, noise: torch.tensor=0) -> torch.tensor: 24 | o = torch.tensor([], device=self.device) 25 | for i in range(self.num_kernels): 26 | o = torch.cat([o, self.kernel_net_list[i](self.encoder_list[i](x + noise))], 1) 27 | 28 | return o -------------------------------------------------------------------------------- /smoothed_acid/classifier/subnet.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | import torch 4 | import torch.nn as nn 5 | 6 | class Sin_Weighted(nn.Module): 7 | def __init__(self, input_size: int): 8 | super(Sin_Weighted, self).__init__() 9 | 10 | self.wa = nn.Parameter(torch.zeros((input_size), requires_grad=True)) 11 | self.wf = nn.Parameter(torch.zeros((1), requires_grad=True)) 12 | 13 | self.weight_init() 14 | 15 | def weight_init(self): 16 | self.wa.data.normal_(0, 1) 17 | self.wf.data.normal_(0, 1) 18 | 19 | def forward(self, x: torch.tensor) -> torch.tensor: 20 | return self.wf * torch.sin(2 * np.pi * x * self.wa) 21 | 22 | class Encoder(nn.Module): 23 | def __init__(self, input_size: int): 24 | super(Encoder, self).__init__() 25 | 26 | layers_dims = [500, 200, 50] 27 | 28 | kernel_size = 10 29 | 30 | self.encoder = nn.Sequential( 31 | nn.Linear(input_size, layers_dims[0]), 32 | Sin_Weighted(layers_dims[0]), 33 | nn.Linear(layers_dims[0], layers_dims[1]), 34 | Sin_Weighted(layers_dims[1]), 35 | nn.Linear(layers_dims[1], layers_dims[2]), 36 | Sin_Weighted(layers_dims[2]), 37 | nn.Linear(layers_dims[2], kernel_size) 38 | ) 39 | 40 | def forward(self, x: torch.tensor) -> torch.tensor: 41 | return self.encoder(x) 42 | 43 | class Kernel_Net(nn.Module): 44 | def __init__(self, input_size: int): 45 | super(Kernel_Net, self).__init__() 46 | 47 | layers_dims = [100, 50, 30] 48 | 49 | self.kernel_net = nn.Sequential( 50 | nn.Linear(input_size, layers_dims[0]), 51 | nn.Tanh(), 52 | nn.Linear(layers_dims[0], layers_dims[1]), 53 | nn.Tanh(), 54 | nn.Linear(layers_dims[1], layers_dims[2]), 55 | nn.Tanh(), 56 | nn.Linear(layers_dims[2], 1) 57 | ) 58 | 59 | def forward(self, x: torch.tensor) -> torch.tensor: 60 | return self.kernel_net(x) 61 | 62 | class AdaptiveClustering(nn.Module): 63 | def __init__(self, input_size: int, num_kernels: int, device: object): 64 | super(AdaptiveClustering, self).__init__() 65 | 66 | self.num_kernels = num_kernels 67 | 68 | kernel_size = 10 69 | 70 | self.encoder_list = nn.ModuleList([ 71 | Encoder(input_size) for i in range(num_kernels) 72 | ]) 73 | 74 | self.kernel_weight = nn.Parameter(torch.zeros((num_kernels, kernel_size), requires_grad=True)) 75 | 76 | self.kernel_net_list = nn.ModuleList([ 77 | Kernel_Net(kernel_size) for i in range(num_kernels) 78 | ]) 79 | self.out_act = nn.Sigmoid() 80 | 81 | self.device = device 82 | 83 | def weight_init(self): 84 | self.kernel_weight.data.normal_(0, 1) 85 | 86 | def forward(self, x: torch.tensor) -> (torch.tensor, torch.tensor, torch.tensor): 87 | z = torch.tensor([], device=self.device) 88 | o = torch.tensor([], device=self.device) 89 | for i in range(self.num_kernels): 90 | zi = self.encoder_list[i](x) 91 | z = torch.cat([z, zi.unsqueeze(1)], 1) 92 | o = torch.cat([o, self.out_act(self.kernel_net_list[i](zi))], 1) 93 | 94 | return z, self.kernel_weight, o 95 | 96 | class Loss_AdaptiveClustering(nn.Module): 97 | def __init__(self, num_classes: int): 98 | super(Loss_AdaptiveClustering, self).__init__() 99 | 100 | self.num_classes = num_classes 101 | self.margin = 1 102 | 103 | self.criterion = nn.MSELoss() 104 | 105 | def forward(self, z: torch.tensor, kernel_weight: torch.tensor, o: torch.tensor, y: torch.tensor) -> (torch.tensor, torch.tensor, torch.tensor, torch.tensor): 106 | assert self.num_classes == kernel_weight.size(0) 107 | 108 | y_one_hot = torch.zeros_like(o, dtype=torch.float).to(o.device) 109 | y_one_hot[torch.arange(o.size(0)).to(o.device), y] = 1 110 | loss_classification = self.criterion(o, y_one_hot) 111 | 112 | loss_clustering_close = torch.tensor(0.0, dtype=torch.float).to(z.device) 113 | for c in range(self.num_classes): 114 | loss_clustering_close += self.criterion( 115 | z[y == c, c, :], 116 | kernel_weight[c, :].repeat((torch.sum(y == c), 1))) 117 | 118 | loss_clustering_dist = torch.tensor(0.0, dtype=torch.float).to(z.device) 119 | for c in range(self.num_classes): 120 | loss_clustering_dist += torch.clamp(self.margin - self.criterion( 121 | kernel_weight[torch.arange(self.num_classes).to(kernel_weight.device) != c, :], 122 | kernel_weight[c, :].clone().detach().repeat((torch.sum(torch.arange(self.num_classes).to(kernel_weight.device) != c), 1))), 123 | min=0) 124 | loss_clustering_dist /= self.num_classes 125 | 126 | return loss_classification + loss_clustering_close + loss_clustering_dist, \ 127 | loss_classification, loss_clustering_close, loss_clustering_dist 128 | -------------------------------------------------------------------------------- /smoothed_acid/data/ids18.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_acid/data/ids18.zip -------------------------------------------------------------------------------- /smoothed_acid/main.py: -------------------------------------------------------------------------------- 1 | import os, random 2 | import numpy as np 3 | from tqdm import tqdm 4 | 5 | import sys 6 | sys.path.append('../bars/') 7 | 8 | import matplotlib.pyplot as plt 9 | ### 解决报错 Could not connect to any X display. 10 | import matplotlib 11 | matplotlib.use('Agg') 12 | 13 | import seaborn as sns 14 | sns.set_style("darkgrid") 15 | 16 | import warnings 17 | warnings.filterwarnings("ignore") 18 | 19 | import torch 20 | import torch.nn as nn 21 | import torch.optim as optim 22 | from torch.utils.data import TensorDataset, DataLoader 23 | 24 | from classifier.subnet import AdaptiveClustering, Loss_AdaptiveClustering 25 | from classifier.classifier import AdaptiveClusteringClassifier 26 | from utils import load_data, AverageMeter, calc_metrics_classifier, calc_metrics_certify 27 | 28 | from smoothing import Smooth2, Noise 29 | from distribution_transformer import distribution_transformers, loss_functions 30 | from optimizing_noise import optimizing_noise 31 | 32 | # General 33 | no_gpu = False # Avoid using CUDA when available 34 | cuda_ids = "0" # GPU No 35 | n_gpu = torch.cuda.device_count() 36 | device = torch.device("cuda" if torch.cuda.is_available() and not no_gpu else "cpu") 37 | data_dir = "data" # Data directory 38 | dataset_name = "ids18" # The name of data set 39 | if dataset_name not in os.listdir(data_dir): 40 | raise ValueError("Dataset not found: %s" % (dataset_name)) 41 | data_dir = os.path.join(data_dir, dataset_name) 42 | 43 | # Building classifier 44 | x_train, y_train = load_data(data_dir, "train") 45 | input_size_classifier = x_train.size(1) # Number of feature dimensions 46 | num_classes_classifier = torch.unique(y_train).size(0) # Number of classes for classifier 47 | learning_rate_classifier = 1e-4 # Learning rate for training classifier 48 | num_epochs_classifier = 30 # Epoch number for training classifier 49 | batch_size_train = 512 # Batch size for training classifier 50 | batch_size_eval = 512 # Batch size for evaluating classifier 51 | save_dir_train = "save/%s/train" % (dataset_name) # Saving directory for classifier training 52 | if not os.path.exists(save_dir_train): 53 | os.makedirs(save_dir_train) 54 | print_step_classifier = 100 # Step size for showing training progress 55 | 56 | # Certifying classifier 57 | certify_class = 0 # The certified class. e.g., Benign(0), FTP-Bruteforce(1), DDoS-HOIC(2), Botnet-Zeus&Ares(3) 58 | feature_noise_distribution = "isru" # Feature noise distribution. e.g., "gaussian" "isru" "isru_gaussian" "isru_gaussian_arctan" 59 | learning_rate_shape = 1e-2 # Learning rate for optimzing noise shape 60 | nt_shape = 1000 # Number of noised samples for optimzing noise shape 61 | lambda_shape = 1e-2 # Regularizer weight 62 | num_epochs_shape = 10 # Number of epochs for optimzing noise shape 63 | d = input_size_classifier # Number of feature dimensions 64 | num_classes_certify = num_classes_classifier # Number of certified classes 65 | n0 = 100 # Number of noised samples for identify cA 66 | n = 10000 # Number of noised samples for estimate pA 67 | alpha = 1e-3 # Failure probability 68 | init_step_size_scale = 5e-2 # Initial update step size of t for optimzing noise scale 69 | init_ptb_t_scale = 1e-2 # Initial perturbation of t for optimzing noise scale 70 | decay_factor_scale = 0.5 # Decay factor for optimzing noise scale 71 | max_decay_scale = 6 # Maximum decay times for optimzing noise scale 72 | max_iter_scale = 100 # Maximum iteration times for optimzing noise scale 73 | batch_size_iteration_certify = 128 # Batch size of certified samples for robustness certification 74 | batch_size_memory_certify = 1000000 # Batch size of noised samples for robustness certification 75 | print_step_certify = 20 # Step size for showing certification progress 76 | save_dir_certify = "save/%s/certify" % (dataset_name) # Saving directory for robustness certification 77 | if not os.path.exists(save_dir_certify): 78 | os.makedirs(save_dir_certify) 79 | num_samples_certify = 10000 # Number of certified samples sampled from dataset 80 | 81 | seed = 42 # Random seed 82 | 83 | def set_seed(): 84 | random.seed(seed) 85 | np.random.seed(seed) 86 | torch.manual_seed(seed) 87 | if n_gpu > 0: 88 | torch.cuda.manual_seed_all(seed) 89 | 90 | def create_ac(): 91 | return AdaptiveClustering(input_size_classifier, num_classes_classifier, device) 92 | 93 | def create_classifier(encoder_list: nn.ModuleList, kernel_net_list: nn.ModuleList): 94 | return AdaptiveClusteringClassifier(encoder_list, kernel_net_list, device) 95 | 96 | def create_noise_generator(): 97 | dist_trans = distribution_transformers[feature_noise_distribution](d).to(device) 98 | return Noise(dist_trans, d, device) 99 | 100 | def train(): 101 | print("\n***** Run training *****") 102 | print("Number of classes for classifier:", num_classes_classifier) 103 | 104 | ac = create_ac() 105 | ac.to(device) 106 | 107 | opt = optim.Adam(ac.parameters(), lr=learning_rate_classifier) 108 | 109 | criterion = Loss_AdaptiveClustering(num_classes_classifier) 110 | 111 | x_train, y_train = load_data(data_dir, "train") 112 | 113 | data_dataset = TensorDataset(x_train, y_train) 114 | data_loader = DataLoader(data_dataset, batch_size=batch_size_train, shuffle=True) 115 | 116 | loss_record = AverageMeter() 117 | loss_classification_record = AverageMeter() 118 | loss_clustering_close_record = AverageMeter() 119 | loss_clustering_dist_record = AverageMeter() 120 | ac.to(ac.device) 121 | ac.train() 122 | for epoch in range(num_epochs_classifier + 1): 123 | for i, (X, y) in zip(range(1, len(data_loader) + 1), data_loader): 124 | X, y = X.to(ac.device), y.to(ac.device) 125 | 126 | z, kernel_weight, o = ac(X) 127 | loss, loss_classification, loss_clustering_close, loss_clustering_dist = criterion(z, kernel_weight, o, y) 128 | 129 | if epoch > 0: 130 | opt.zero_grad() 131 | loss.backward() 132 | opt.step() 133 | 134 | loss_record.update(loss.item()) 135 | loss_classification_record.update(loss_classification.item()) 136 | loss_clustering_close_record.update(loss_clustering_close.item()) 137 | loss_clustering_dist_record.update(loss_clustering_dist.item()) 138 | if i % print_step_classifier == 0: 139 | print(("Batch: [%d/%d][%d/%d] | Loss: %.6f | " + \ 140 | "Loss classification: %.6f | " + \ 141 | "Loss clustering_close: %.6f | " + \ 142 | "Loss clustering_dist: %.6f") % ( \ 143 | epoch, num_epochs_classifier, i, len(data_loader), loss_record.val, \ 144 | loss_classification_record.val, \ 145 | loss_clustering_close_record.val, \ 146 | loss_clustering_dist_record.val)) 147 | 148 | print(('Epoch: [%d/%d] | Loss (Avg): %.6f | ' + \ 149 | 'Loss classification (Avg): %.6f | ' + \ 150 | 'Loss clustering_close (Avg): %.6f | ' + \ 151 | 'Loss clustering_dist (Avg): %.6f') % ( \ 152 | epoch, num_epochs_classifier, loss_record.avg, \ 153 | loss_classification_record.avg, \ 154 | loss_clustering_close_record.avg, \ 155 | loss_clustering_dist_record.avg)) 156 | 157 | loss_record.reset() 158 | loss_classification_record.reset() 159 | loss_clustering_close_record.reset() 160 | loss_clustering_dist_record.reset() 161 | 162 | torch.save(ac.encoder_list, os.path.join(save_dir_train, "checkpoint-encoder_list")) 163 | torch.save(ac.kernel_weight, os.path.join(save_dir_train, "checkpoint-kernel_weight")) 164 | torch.save(ac.kernel_net_list, os.path.join(save_dir_train, "checkpoint-kernel_net_list")) 165 | 166 | def evaluate(): 167 | print("\n***** Run evaluating *****") 168 | print("Number of classes for classifier:", num_classes_classifier) 169 | 170 | encoder_list = torch.load(os.path.join(save_dir_train, "checkpoint-encoder_list")) 171 | encoder_list = nn.ModuleList([e.to(device) for e in encoder_list]) 172 | kernel_net_list = torch.load(os.path.join(save_dir_train, "checkpoint-kernel_net_list")) 173 | kernel_net_list = nn.ModuleList([n.to(device) for n in kernel_net_list]) 174 | classifier = create_classifier(encoder_list, kernel_net_list) 175 | 176 | x_test, y_test = load_data(data_dir, "test") 177 | 178 | data_dataset = TensorDataset(x_test, y_test) 179 | data_loader = DataLoader(data_dataset, batch_size=batch_size_eval, shuffle=False) 180 | 181 | label_record = np.array([], dtype=np.long) 182 | pred_record = np.array([], dtype=np.long) 183 | classifier.eval() 184 | torch.set_grad_enabled(False) 185 | for X, y in tqdm(data_loader, desc="Evaluate"): 186 | X, y = X.to(classifier.device), y.to(classifier.device) 187 | pred = classifier(X) 188 | 189 | pred_record = np.concatenate([pred_record, pred.detach().cpu().numpy()], 0) 190 | label_record = np.concatenate([label_record, y.detach().cpu().numpy()], 0) 191 | 192 | torch.set_grad_enabled(True) 193 | torch.cuda.empty_cache() 194 | 195 | acc, m = calc_metrics_classifier(label_record, pred_record) 196 | print(('Accuracy: %.4f') % (acc)) 197 | print("Confusion matrix (Row: ground truth class, Col: prediction class):") 198 | for i in range(m.shape[0]): 199 | for j in range(m.shape[1]): 200 | if j < m.shape[1] - 1: 201 | print("%d " % (m[i, j]), end="") 202 | else: 203 | print("%d" % (m[i, j])) 204 | 205 | def certify(): 206 | print("\n***** Optimize noise *****") 207 | 208 | encoder_list = torch.load(os.path.join(save_dir_train, "checkpoint-encoder_list")) 209 | encoder_list = nn.ModuleList([e.to(device) for e in encoder_list]) 210 | kernel_net_list = torch.load(os.path.join(save_dir_train, "checkpoint-kernel_net_list")) 211 | kernel_net_list = nn.ModuleList([n.to(device) for n in kernel_net_list]) 212 | classifier = create_classifier(encoder_list, kernel_net_list) 213 | 214 | noise_generator = create_noise_generator() 215 | 216 | criterion_shape = loss_functions["acid"](lambda_shape, certify_class) 217 | 218 | x_train, _ = load_data(data_dir, "train") 219 | 220 | dataset = TensorDataset(x_train) 221 | data_loader = DataLoader(dataset, batch_size=batch_size_iteration_certify, shuffle=True) 222 | x_train_c = torch.tensor([], dtype=torch.float32) 223 | for (X,) in tqdm(data_loader, desc="Select class %d" % (certify_class)): 224 | X = X.to(device) 225 | 226 | pred = classifier(X) 227 | 228 | x_train_c = torch.cat([x_train_c, X[(pred == certify_class), :].detach().cpu()], 0) 229 | 230 | optimizing_noise( 231 | x_train_c, 232 | certify_class, 233 | classifier, 234 | noise_generator, 235 | criterion_shape, 236 | learning_rate_shape, 237 | nt_shape, 238 | num_epochs_shape, 239 | d, 240 | num_classes_certify, 241 | n0, 242 | n, 243 | alpha, 244 | init_step_size_scale, 245 | init_ptb_t_scale, 246 | decay_factor_scale, 247 | max_decay_scale, 248 | max_iter_scale, 249 | batch_size_iteration_certify, 250 | batch_size_memory_certify, 251 | print_step_certify, 252 | save_dir_certify) 253 | 254 | noise_generator.distribution_transformer = torch.load(os.path.join(save_dir_certify, "checkpoint-distribution-transformer")).to(device) 255 | 256 | r = open(os.path.join(save_dir_certify, "t"), "r") 257 | t = float(r.readline()) 258 | r.close() 259 | 260 | print("\n***** Certify robustness *****") 261 | 262 | x_test, y_test = load_data(data_dir, "test") 263 | 264 | dataset = TensorDataset(x_test, y_test) 265 | data_loader = DataLoader(dataset, batch_size=batch_size_iteration_certify, shuffle=False) 266 | x_test_c = torch.tensor([], dtype=torch.float32) 267 | y_test_c = torch.tensor([], dtype=torch.long) 268 | for (X, y) in tqdm(data_loader, desc="Select class %d" % (certify_class)): 269 | X = X.to(device) 270 | 271 | pred = classifier(X) 272 | 273 | x_test_c = torch.cat([x_test_c, X[(pred == certify_class), :].detach().cpu()], 0) 274 | y_test_c = torch.cat([y_test_c, y[(pred == certify_class)].detach().cpu()], 0) 275 | 276 | idx = torch.arange(x_test_c.shape[0]) 277 | idx = idx[torch.randperm(idx.size(0))] 278 | idx = torch.sort(idx[:min(num_samples_certify, idx.shape[0])])[0] 279 | x_test_c, y_test_c = x_test_c[idx, :], y_test_c[idx] 280 | 281 | dataset = TensorDataset(x_test_c, y_test_c) 282 | data_loader = DataLoader(dataset, batch_size=batch_size_iteration_certify, shuffle=False) 283 | 284 | classifier.eval() 285 | smoothed_classifier = Smooth2(classifier, d, num_classes_certify, noise_generator, device) 286 | cA_record = np.array([], dtype=np.long) 287 | robust_radius_record = np.array([], dtype=np.float32) 288 | label_record = np.array([], dtype=np.long) 289 | torch.set_grad_enabled(False) 290 | for X, y in tqdm(data_loader, desc="Certify"): 291 | X = X.to(device) 292 | 293 | cA, _, robust_radius = smoothed_classifier.bars_certify(X, n0, n, t, alpha, batch_size_memory_certify) 294 | 295 | cA_record = np.concatenate([cA_record, cA], 0) 296 | robust_radius_record = np.concatenate([robust_radius_record, robust_radius], 0) 297 | 298 | label_record = np.concatenate([label_record, y.detach().cpu().numpy()], 0) 299 | 300 | torch.set_grad_enabled(True) 301 | torch.cuda.empty_cache() 302 | 303 | mean_robust_radius = calc_metrics_certify(label_record, cA_record, robust_radius_record) 304 | print("Mean Robustness Radius: %.6e" % (mean_robust_radius)) 305 | 306 | w = open(os.path.join(save_dir_certify, "certification_results"), "w") 307 | w.write("label\tcA\trobust_radius\n") 308 | for i in range(label_record.shape[0]): 309 | w.write("%d\t%d\t%.6e\n" % (label_record[i], cA_record[i], robust_radius_record[i])) 310 | w.close() 311 | 312 | max_robust_radius = {"ids18-0": 0.8, "ids18-1": 0.5, "ids18-2": 0.25, "ids18-3": 0.8} 313 | robust_radius_plot = np.arange(0, max_robust_radius["%s-%d" % (dataset_name, certify_class)], max_robust_radius["%s-%d" % (dataset_name, certify_class)] * 1e-3) 314 | certified_accuracy_plot = np.array([], dtype=np.float32) 315 | for r in robust_radius_plot: 316 | certified_accuracy_plot = np.append(certified_accuracy_plot, np.mean((label_record == cA_record) & (robust_radius_record > r))) 317 | plt.figure(1) 318 | plt.plot(robust_radius_plot, certified_accuracy_plot) 319 | plt.ylim((0, 1)) 320 | plt.xlim((0, max_robust_radius["%s-%d" % (dataset_name, certify_class)])) 321 | plt.tick_params(labelsize=14) 322 | plt.xlabel("Robustness Radius", fontsize=16) 323 | plt.ylabel("Certified Accuracy", fontsize=16) 324 | class_name = {"ids18-0": "benign", "ids18-1": "ftp-bruteforce", "ids18-2": "ddos-hoic", "ids18-3": "botnet-zeus&ares"} 325 | plt.title("ACID %s" % (class_name["%s-%d" % (dataset_name, certify_class)]), fontsize=20) 326 | plt.tight_layout() 327 | plt.savefig(os.path.join(save_dir_certify, "certified_accuracy_robustness_radius_curve.png")) 328 | plt.close() 329 | 330 | def main(): 331 | os.environ["CUDA_VISIBLE_DEVICES"] = cuda_ids 332 | 333 | set_seed() # For reproductibility 334 | 335 | # Training 336 | train() 337 | 338 | # Evaluating 339 | evaluate() 340 | 341 | # Certifying 342 | certify() 343 | 344 | if __name__ == "__main__": 345 | main() -------------------------------------------------------------------------------- /smoothed_acid/save/ids18/certify/certified_accuracy_robustness_radius_curve.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_acid/save/ids18/certify/certified_accuracy_robustness_radius_curve.png -------------------------------------------------------------------------------- /smoothed_acid/save/ids18/certify/checkpoint-distribution-transformer: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_acid/save/ids18/certify/checkpoint-distribution-transformer -------------------------------------------------------------------------------- /smoothed_acid/save/ids18/certify/t: -------------------------------------------------------------------------------- 1 | 9.765625e-02 -------------------------------------------------------------------------------- /smoothed_acid/save/ids18/train/checkpoint-encoder_list: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_acid/save/ids18/train/checkpoint-encoder_list -------------------------------------------------------------------------------- /smoothed_acid/save/ids18/train/checkpoint-kernel_net_list: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_acid/save/ids18/train/checkpoint-kernel_net_list -------------------------------------------------------------------------------- /smoothed_acid/save/ids18/train/checkpoint-kernel_weight: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_acid/save/ids18/train/checkpoint-kernel_weight -------------------------------------------------------------------------------- /smoothed_acid/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | from sklearn.metrics import confusion_matrix 4 | from typing import * 5 | 6 | import torch 7 | 8 | def load_data(data_dir: str, data_type: str="train"): 9 | 10 | if data_type == "train": 11 | x_train = np.load(os.path.join(data_dir, "X_train.npy")) 12 | y_train = np.load(os.path.join(data_dir, "y_train.npy")) 13 | 14 | return torch.tensor(x_train, dtype=torch.float), torch.tensor(y_train, dtype=torch.long) 15 | 16 | elif data_type == "test": 17 | x_test = np.load(os.path.join(data_dir, "X_test.npy")) 18 | y_test = np.load(os.path.join(data_dir, "y_test.npy")) 19 | 20 | return torch.tensor(x_test, dtype=torch.float), torch.tensor(y_test, dtype=torch.long) 21 | 22 | else: 23 | raise NotImplementedError() 24 | 25 | class AverageMeter(object): 26 | def __init__(self): 27 | self.reset() 28 | 29 | def reset(self): 30 | self.val = 0 31 | self.avg = 0 32 | self.sum = 0 33 | self.count = 0 34 | 35 | def update(self, val): 36 | self.val = val 37 | self.sum += val 38 | self.count += 1 39 | self.avg = self.sum / self.count 40 | 41 | def calc_metrics_classifier(label: np.ndarray, pred: np.ndarray): 42 | assert label.shape == pred.shape 43 | acc = np.array((label == pred), dtype=float).mean() 44 | m = confusion_matrix(y_true=label, y_pred=pred) 45 | 46 | return acc, m 47 | 48 | def calc_metrics_certify(label: np.ndarray, pred: np.ndarray, radius_feat: np.ndarray) -> np.ndarray: 49 | assert label.shape == pred.shape 50 | assert pred.shape == radius_feat.shape 51 | 52 | return np.mean(np.array((label == pred), dtype=float) * radius_feat) -------------------------------------------------------------------------------- /smoothed_cade/classifier/autoencoder.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | def Encoder(input_size: int, output_size: int): 5 | 6 | dims = [input_size, 64, 32, 16, output_size] 7 | 8 | return nn.Sequential( 9 | nn.Linear(dims[0], dims[1]), 10 | nn.ReLU(inplace=True), 11 | nn.Linear(dims[1], dims[2]), 12 | nn.ReLU(inplace=True), 13 | nn.Linear(dims[2], dims[3]), 14 | nn.ReLU(inplace=True), 15 | nn.Linear(dims[3], dims[4]) 16 | ) 17 | 18 | def Decoder(input_size: int, output_size: int): 19 | 20 | dims = [input_size, 16, 32, 64, output_size] 21 | 22 | return nn.Sequential( 23 | nn.Linear(dims[0], dims[1]), 24 | nn.ReLU(inplace=True), 25 | nn.Linear(dims[1], dims[2]), 26 | nn.ReLU(inplace=True), 27 | nn.Linear(dims[2], dims[3]), 28 | nn.ReLU(inplace=True), 29 | nn.Linear(dims[3], dims[4]) 30 | ) 31 | 32 | class Autoeconder(nn.Module): 33 | def __init__(self, input_size: int, num_classes: int, device: object): 34 | super().__init__() 35 | 36 | self.encoder = Encoder(input_size, num_classes) 37 | self.decoder = Decoder(num_classes, input_size) 38 | 39 | self.device = device 40 | 41 | def forward(self, x): 42 | z = self.encoder(x) 43 | x_reconstructed = self.decoder(z) 44 | 45 | return z, x_reconstructed 46 | 47 | class Loss_Contrastive(nn.Module): 48 | def __init__(self, margin: float, lambda_: float): 49 | super().__init__() 50 | 51 | self.criterion_reconstructed = nn.MSELoss(reduction="mean") 52 | 53 | self.margin = margin 54 | self.lambda_ = lambda_ 55 | 56 | self.eps = 1e-10 57 | 58 | def forward(self, x: torch.tensor, z: torch.tensor, x_reconstructed: torch.tensor, y: torch.tensor): 59 | loss_reconstructed = self.criterion_reconstructed(x, x_reconstructed) 60 | 61 | left_idx = torch.arange(0, int(x.size(0) / 2)).to(x.device) 62 | right_idx = torch.arange(int(x.size(0) / 2), int(x.size(0) / 2) * 2).to(x.device) 63 | dist = torch.sqrt(torch.sum(torch.pow(z[left_idx] - z[right_idx], 2), 1) + self.eps) 64 | in_same_class = torch.zeros_like(dist).to(x.device) 65 | in_same_class[y[left_idx] == y[right_idx]] = 1 66 | in_diff_class = torch.zeros_like(dist).to(x.device) 67 | in_diff_class[y[left_idx] != y[right_idx]] = 1 68 | loss_contrastive = torch.mean(in_same_class * dist + in_diff_class * nn.ReLU(inplace=True)(self.margin - dist)) 69 | 70 | return loss_reconstructed + self.lambda_ * loss_contrastive, loss_reconstructed, loss_contrastive -------------------------------------------------------------------------------- /smoothed_cade/classifier/detect.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | class Detector(nn.Module): 5 | def __init__(self, encoder, centroid, dis_median, mad, mad_threshold, device): 6 | super(Detector, self).__init__() 7 | 8 | self.encoder = encoder 9 | 10 | self.centroid = centroid 11 | self.dis_median = dis_median 12 | self.mad = mad 13 | self.mad_threshold = mad_threshold 14 | 15 | self.device = device 16 | 17 | def forward(self, x: torch.tensor, noise: torch.tensor=0) -> torch.tensor: 18 | centroid_class = self.closest_class(x) 19 | _, drift = self._score_drift_under_centroid_class(x + noise, centroid_class) 20 | 21 | return drift 22 | 23 | def score(self, x: torch.tensor, noise: torch.tensor=0) -> torch.tensor: 24 | centroid_class = self.closest_class(x) 25 | score, _ = self._score_drift_under_centroid_class(x + noise, centroid_class) 26 | 27 | return score 28 | 29 | def closest_class(self, x: torch.tensor) -> torch.tensor: 30 | z = self.encoder(x) 31 | 32 | dis = torch.norm(z.unsqueeze(1).repeat((1, self.centroid.size(0), 1)) - \ 33 | self.centroid.unsqueeze(0).repeat((z.size(0), 1, 1)), p=2, dim=2) 34 | closest_class = dis.argmin(1) 35 | 36 | return closest_class 37 | 38 | # We adjust CADE by detecting drift based on the closest classes of the original samples for ease of illustration. 39 | def _score_drift_under_centroid_class(self, x: torch.tensor, centroid_class: torch.tensor) -> (torch.tensor, torch.tensor): 40 | assert x.size(0) == centroid_class.size(0) 41 | 42 | z = self.encoder(x) 43 | dis = torch.norm(z - self.centroid[centroid_class, :], p=2, dim=1) 44 | score = torch.abs(dis - self.dis_median[centroid_class]) / self.mad[centroid_class] 45 | drift = torch.zeros_like(score, dtype=torch.long) 46 | drift[score > self.mad_threshold] = 1 47 | 48 | return score, drift 49 | 50 | def calc_centroid_dis_median_mad(z_train: torch.tensor, y_train: torch.tensor): 51 | class_list = torch.unique(y_train).tolist() 52 | 53 | centroid_record = torch.tensor([], device=z_train.device) 54 | for i in range(len(class_list)): 55 | centroid_record = torch.cat([centroid_record, z_train[y_train == class_list[i], :].mean(0).unsqueeze(0)], 0) 56 | 57 | dis_median_record = torch.tensor([], device=z_train.device) 58 | mad_record = torch.tensor([], device=z_train.device) 59 | for i in range(len(class_list)): 60 | dis = torch.norm(z_train[y_train == class_list[i], :] - centroid_record[class_list[i], :], p=2, dim=1) 61 | dis_median = torch.median(dis) 62 | mad = 1.4826 * torch.median(torch.abs(dis - dis_median)) 63 | dis_median_record = torch.cat([dis_median_record, dis_median.unsqueeze(0)], 0) 64 | mad_record = torch.cat([mad_record, mad.unsqueeze(0)], 0) 65 | 66 | return centroid_record, dis_median_record, mad_record -------------------------------------------------------------------------------- /smoothed_cade/data/newhulk.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_cade/data/newhulk.zip -------------------------------------------------------------------------------- /smoothed_cade/data/newinfiltration.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_cade/data/newinfiltration.zip -------------------------------------------------------------------------------- /smoothed_cade/main.py: -------------------------------------------------------------------------------- 1 | import os, random 2 | import numpy as np 3 | from tqdm import tqdm 4 | 5 | import sys 6 | sys.path.append('../bars/') 7 | 8 | import matplotlib.pyplot as plt 9 | # For "Could not connect to any X display." 10 | import matplotlib 11 | matplotlib.use('Agg') 12 | 13 | import seaborn as sns 14 | sns.set_style("darkgrid") 15 | 16 | import warnings 17 | warnings.filterwarnings("ignore") 18 | 19 | import torch 20 | import torch.nn as nn 21 | import torch.optim as optim 22 | from torch.utils.data import TensorDataset, DataLoader 23 | 24 | from classifier.autoencoder import Autoeconder, Loss_Contrastive 25 | from classifier.detect import calc_centroid_dis_median_mad, Detector 26 | from utils import load_data, AverageMeter, calc_metrics_classifier, calc_metrics_classifier_class, calc_metrics_certify 27 | 28 | from smoothing import Smooth2, Noise 29 | from distribution_transformer import distribution_transformers, loss_functions 30 | from optimizing_noise import optimizing_noise 31 | 32 | # General 33 | no_gpu = False # Avoid using CUDA when available 34 | cuda_ids = "0" # GPU No 35 | n_gpu = torch.cuda.device_count() 36 | device = torch.device("cuda" if torch.cuda.is_available() and not no_gpu else "cpu") 37 | data_dir = "data" # Data directory 38 | dataset_name = "newinfiltration" # The name of data set. e.g., "newinfiltration" "newhulk" 39 | if dataset_name not in os.listdir(data_dir): 40 | raise ValueError("Dataset not found: %s" % (dataset_name)) 41 | data_dir = os.path.join(data_dir, dataset_name) 42 | 43 | # Building classifier 44 | learning_rate_classifier = 1e-4 # Learning rate for training classifier 45 | margin = 10 # Distance margin for sample pairs from different classes 46 | lambda_classifier = 1e-1 # Weight for contrastive loss term 47 | mad_threshold = 3.5 # MAD threshold for detecting drift 48 | num_epochs_classifier = 30 # Epoch number for training classifier 49 | batch_size_train = 512 # Batch size for training classifier 50 | batch_size_eval = 512 # Batch size for evaluating classifier 51 | save_dir_train = "save/%s/train" % (dataset_name) # Saving directory for classifier training 52 | if not os.path.exists(save_dir_train): 53 | os.makedirs(save_dir_train) 54 | print_step_classifier = 50 # Step size for showing training progress 55 | 56 | # Certifying classifier 57 | certify_class = 0 # The certified class. e.g., newinfiltration: Benign(0), SSH-Bruteforce(1), DoS-Hulk(2); newhulk: Infiltration(2) 58 | feature_noise_distribution = "isru" # Feature noise distribution. e.g., "gaussian" "isru" "isru_gaussian" "isru_gaussian_arctan" 59 | learning_rate_shape = 2e-4 # Learning rate for optimzing noise shape 60 | nt_shape = 1000 # Number of noised samples for optimzing noise shape 61 | lambda_shape = 1e-1 # Regularizer weight 62 | num_epochs_shape = 5 # Number of epochs for optimzing noise shape 63 | x_train, _, _, _ = load_data(data_dir, "train") 64 | d = x_train.size(1) # Number of feature dimensions 65 | num_classes_certify = 2 # Number of certified classes. i.e., Known class(0), Drift class(1) 66 | n0 = 100 # Number of noised samples for identify cA 67 | n = 10000 # Number of noised samples for estimate pA 68 | alpha = 1e-3 # Failure probability 69 | init_step_size_scale = 5e-2 # Initial update step size of t for optimzing noise scale 70 | init_ptb_t_scale = 1e-2 # Initial perturbation of t for optimzing noise scale 71 | decay_factor_scale = 0.5 # Decay factor for optimzing noise scale 72 | max_decay_scale = 6 # Maximum decay times for optimzing noise scale 73 | max_iter_scale = 100 # Maximum iteration times for optimzing noise scale 74 | batch_size_iteration_certify = 128 # Batch size of certified samples for robustness certification 75 | batch_size_memory_certify = 1000000 # Batch size of noised samples for robustness certification 76 | print_step_certify = 20 # Step size for showing certification progress 77 | save_dir_certify = "save/%s/certify" % (dataset_name) # Saving directory for robustness certification 78 | if not os.path.exists(save_dir_certify): 79 | os.makedirs(save_dir_certify) 80 | num_samples_certify = 10000 # Number of certified samples sampled from dataset 81 | 82 | seed = 42 # Random seed 83 | 84 | def set_seed(): 85 | random.seed(seed) 86 | np.random.seed(seed) 87 | torch.manual_seed(seed) 88 | if n_gpu > 0: 89 | torch.cuda.manual_seed_all(seed) 90 | 91 | def create_ae(): 92 | _, _, num_classes_train, _ = load_data(data_dir, "train") 93 | return Autoeconder(d, num_classes_train, device) 94 | 95 | def create_detector(encoder: nn.Module, centroid: torch.tensor, dis_median: torch.tensor, mad: torch.tensor): 96 | return Detector(encoder, centroid, dis_median, mad, mad_threshold, device) 97 | 98 | def create_noise_generator(): 99 | dist_trans = distribution_transformers[feature_noise_distribution](d).to(device) 100 | return Noise(dist_trans, d, device) 101 | 102 | def train(): 103 | print("\n***** Run training *****") 104 | 105 | ae = create_ae() 106 | ae.to(device) 107 | 108 | opt = optim.Adam(ae.parameters(), lr=learning_rate_classifier) 109 | 110 | criterion = Loss_Contrastive(margin, lambda_classifier) 111 | 112 | x_train, y_train_class, num_classes_train, class_map = load_data(data_dir, "train") 113 | print("Number of classes for training:", num_classes_train) 114 | print("Class Map from original class to new class (First %d classes are for training): %s" % (num_classes_train, str(class_map))) 115 | 116 | data_dataset = TensorDataset(x_train, y_train_class) 117 | data_loader = DataLoader(data_dataset, batch_size=batch_size_train, shuffle=True) 118 | 119 | loss_record = AverageMeter() 120 | loss_reconstructed_record = AverageMeter() 121 | loss_contrastive_record = AverageMeter() 122 | z_record = torch.tensor([], dtype=torch.float32) 123 | label_class_record = torch.tensor([], dtype=torch.long) 124 | ae.train() 125 | for epoch in range(num_epochs_classifier + 1): 126 | for i, (X, y_class) in zip(range(1, len(data_loader) + 1), data_loader): 127 | X, y_class = X.to(device), y_class.to(device) 128 | 129 | z, x_reconstructed = ae(X) 130 | loss, loss_reconstructed, loss_contrastive = criterion(X, z, x_reconstructed, y_class) 131 | 132 | if epoch > 0: 133 | opt.zero_grad() 134 | loss.backward() 135 | opt.step() 136 | 137 | loss_record.update(loss.item()) 138 | loss_reconstructed_record.update(loss_reconstructed.item()) 139 | loss_contrastive_record.update(loss_contrastive.item()) 140 | if i % print_step_classifier == 0: 141 | print(("Batch: [%d/%d][%d/%d] | Loss: %.6f | " + \ 142 | "Loss reconstructed: %.6f | " + \ 143 | "Loss contrastive: %.6f") % ( \ 144 | epoch, num_epochs_classifier, i, len(data_loader), loss_record.val, \ 145 | loss_reconstructed_record.val, \ 146 | loss_contrastive_record.val)) 147 | 148 | if epoch == num_epochs_classifier: 149 | z, x_reconstructed = ae(X) 150 | 151 | z_record = torch.cat([z_record, z.detach().cpu()], 0) 152 | label_class_record = torch.cat([label_class_record, y_class.detach().cpu()], 0) 153 | 154 | print(("Epoch: [%d/%d] | Loss (Avg): %.6f | " + \ 155 | "Loss reconstructed (Avg): %.6f | " + \ 156 | "Loss contrastive (Avg): %.6f") % ( \ 157 | epoch, num_epochs_classifier, loss_record.avg, \ 158 | loss_reconstructed_record.avg, \ 159 | loss_contrastive_record.avg)) 160 | 161 | loss_record.reset() 162 | loss_reconstructed_record.reset() 163 | loss_contrastive_record.reset() 164 | 165 | torch.save(ae.encoder, os.path.join(save_dir_train, "checkpoint-encoder")) 166 | torch.save(ae.decoder, os.path.join(save_dir_train, "checkpoint-decoder")) 167 | 168 | centroid, dis_median, mad = calc_centroid_dis_median_mad(z_record, label_class_record) 169 | torch.save({ 170 | "centroid": centroid, 171 | "dis_median": dis_median, 172 | "mad": mad 173 | }, os.path.join(save_dir_train, "checkpoint-param")) 174 | 175 | def evaluate(): 176 | print("\n***** Run evaluating *****") 177 | 178 | param = torch.load(os.path.join(save_dir_train, "checkpoint-param")) 179 | encoder = torch.load(os.path.join(save_dir_train, "checkpoint-encoder")) 180 | detector = create_detector(encoder.to(device), param["centroid"].to(device), param["dis_median"].to(device), \ 181 | param["mad"].to(device)) 182 | 183 | x_test, y_test_class, num_classes_train, class_map, y_test_drift = load_data(data_dir, "test") 184 | print("Number of classes for training:", num_classes_train) 185 | print("Class Map from original class to new class (First %d classes are for training): %s" % (num_classes_train, str(class_map))) 186 | 187 | dataset = TensorDataset(x_test, y_test_class, y_test_drift) 188 | data_loader = DataLoader(dataset, batch_size=batch_size_eval, shuffle=False) 189 | 190 | closest_class_record = np.array([], dtype=np.long) 191 | drift_record = np.array([], dtype=np.long) 192 | label_class_record = np.array([], dtype=np.long) 193 | label_drift_record = np.array([], dtype=np.long) 194 | detector.eval() 195 | torch.set_grad_enabled(False) 196 | for X, y_class, y_drift in tqdm(data_loader, desc="Evaluate"): 197 | X, y_class, y_drift = X.to(detector.device), y_class.to(detector.device), y_drift.to(detector.device) 198 | 199 | closest_class = detector.closest_class(X) 200 | drift = detector(X) 201 | 202 | closest_class_record = np.concatenate([closest_class_record, closest_class.detach().cpu().numpy()], 0) 203 | drift_record = np.concatenate([drift_record, drift.detach().cpu().numpy()], 0) 204 | label_class_record = np.concatenate([label_class_record, y_class.detach().cpu().numpy()], 0) 205 | label_drift_record = np.concatenate([label_drift_record, y_drift.detach().cpu().numpy()], 0) 206 | 207 | torch.set_grad_enabled(True) 208 | torch.cuda.empty_cache() 209 | 210 | _, m_class = calc_metrics_classifier_class(label_class_record, closest_class_record) 211 | save_idx = np.array([], dtype=np.long) 212 | for c in range(num_classes_train): 213 | save_idx = np.concatenate([save_idx, np.where(label_class_record == c)[0]], 0) 214 | acc_class, _ = calc_metrics_classifier_class(closest_class_record[save_idx], label_class_record[save_idx]) 215 | 216 | print(('Accuracy (Class for training): %.4f') % (acc_class)) 217 | print("Confusion matrix (Row: ground truth class, Col: prediction class):") 218 | for i in range(m_class.shape[0]): 219 | for j in range(m_class.shape[1]): 220 | if j < m_class.shape[1] - 1: 221 | print("%d " % (m_class[i, j]), end="") 222 | else: 223 | print("%d" % (m_class[i, j])) 224 | 225 | acc, p, r, f1 = calc_metrics_classifier(label_drift_record, drift_record) 226 | _, m_drift = calc_metrics_classifier_class(label_class_record, drift_record) 227 | m_drift = m_drift[:, :2] 228 | 229 | print("Accuracy: %.4f | Precision: %.4f | Recall: %.4f | F1 Score: %.4f" % (acc, p, r, f1)) 230 | print("Confusion matrix (Row: ground truth class, Col: drift result): ") 231 | for i in range(m_drift.shape[0]): 232 | for j in range(m_drift.shape[1]): 233 | if j < m_drift.shape[1] - 1: 234 | print("%d " % (m_drift[i, j]), end="") 235 | else: 236 | print("%d" % (m_drift[i, j])) 237 | 238 | def certify(): 239 | print("\n***** Optimize noise *****") 240 | 241 | param = torch.load(os.path.join(save_dir_train, "checkpoint-param")) 242 | encoder = torch.load(os.path.join(save_dir_train, "checkpoint-encoder")) 243 | detector = create_detector(encoder.to(device), param["centroid"].to(device), param["dis_median"].to(device), \ 244 | param["mad"].to(device)) 245 | 246 | noise_generator = create_noise_generator() 247 | 248 | criterion_shape = loss_functions["cade"](lambda_shape, mad_threshold) 249 | 250 | x_train, _, num_classes_train, class_map = load_data(data_dir, "train") 251 | print("Number of classes for training:", num_classes_train) 252 | print("Class Map from original class to new class (First %d classes are for training): %s" % (num_classes_train, str(class_map))) 253 | 254 | dataset = TensorDataset(x_train) 255 | data_loader = DataLoader(dataset, batch_size=batch_size_iteration_certify, shuffle=False) 256 | x_train_c = torch.tensor([], dtype=torch.float32) 257 | for (X,) in tqdm(data_loader, desc="Select class %d" % (certify_class)): 258 | X = X.to(device) 259 | closest_class = detector.closest_class(X) 260 | x_train_c = torch.cat([x_train_c, X[(closest_class == certify_class), :].detach().cpu()], 0) 261 | 262 | optimizing_noise( 263 | x_train_c, 264 | 0, # Certifying known class 265 | detector, 266 | noise_generator, 267 | criterion_shape, 268 | learning_rate_shape, 269 | nt_shape, 270 | num_epochs_shape, 271 | d, 272 | num_classes_certify, 273 | n0, 274 | n, 275 | alpha, 276 | init_step_size_scale, 277 | init_ptb_t_scale, 278 | decay_factor_scale, 279 | max_decay_scale, 280 | max_iter_scale, 281 | batch_size_iteration_certify, 282 | batch_size_memory_certify, 283 | print_step_certify, 284 | save_dir_certify) 285 | 286 | noise_generator.distribution_transformer = torch.load(os.path.join(save_dir_certify, "checkpoint-distribution-transformer")).to(device) 287 | 288 | r = open(os.path.join(save_dir_certify, "t"), "r") 289 | t = float(r.readline()) 290 | r.close() 291 | 292 | print("\n***** Certify robustness *****") 293 | 294 | x_test, _, num_classes_train, class_map, y_test_drift = load_data(data_dir, "test") 295 | print("Number of classes for training:", num_classes_train) 296 | print("Class Map from original class to new class (First %d classes are for training): %s" % (num_classes_train, str(class_map))) 297 | 298 | dataset = TensorDataset(x_test, y_test_drift) 299 | data_loader = DataLoader(dataset, batch_size=batch_size_iteration_certify, shuffle=False) 300 | x_test_c = torch.tensor([], dtype=torch.float32) 301 | y_test_drift_c = torch.tensor([], dtype=torch.long) 302 | for (X, y) in tqdm(data_loader, desc="Select class %d" % (certify_class)): 303 | X = X.to(device) 304 | 305 | closest_class = detector.closest_class(X) 306 | 307 | x_test_c = torch.cat([x_test_c, X[(closest_class == certify_class), :].detach().cpu()], 0) 308 | y_test_drift_c = torch.cat([y_test_drift_c, y[(closest_class == certify_class)].detach().cpu()], 0) 309 | 310 | idx = torch.arange(x_test_c.shape[0]) 311 | idx = idx[torch.randperm(idx.size(0))] 312 | idx = torch.sort(idx[:min(num_samples_certify, idx.shape[0])])[0] 313 | x_test_c, y_test_drift_c = x_test_c[idx, :], y_test_drift_c[idx] 314 | 315 | dataset = TensorDataset(x_test_c, y_test_drift_c) 316 | data_loader = DataLoader(dataset, batch_size=batch_size_iteration_certify, shuffle=False) 317 | 318 | detector.eval() 319 | smoothed_classifier = Smooth2(detector, d, num_classes_certify, noise_generator, device) 320 | cA_record = np.array([], dtype=np.long) 321 | robust_radius_record = np.array([], dtype=np.float32) 322 | label_record = np.array([], dtype=np.long) 323 | torch.set_grad_enabled(False) 324 | for X, y in tqdm(data_loader, desc="Certify"): 325 | X = X.to(device) 326 | 327 | cA, _, robust_radius = smoothed_classifier.bars_certify(X, n0, n, t, alpha, batch_size_memory_certify) 328 | 329 | cA_record = np.concatenate([cA_record, cA], 0) 330 | robust_radius_record = np.concatenate([robust_radius_record, robust_radius], 0) 331 | 332 | label_record = np.concatenate([label_record, y.detach().cpu().numpy()], 0) 333 | 334 | torch.set_grad_enabled(True) 335 | torch.cuda.empty_cache() 336 | 337 | mean_robust_radius = calc_metrics_certify(label_record, cA_record, robust_radius_record) 338 | print("Mean Robustness Radius: %.6e" % (mean_robust_radius)) 339 | 340 | w = open(os.path.join(save_dir_certify, "certification_results"), "w") 341 | w.write("label\tcA\trobust_radius\n") 342 | for i in range(label_record.shape[0]): 343 | w.write("%d\t%d\t%.6e\n" % (label_record[i], cA_record[i], robust_radius_record[i])) 344 | w.close() 345 | 346 | max_robust_radius = {"newinfiltration-0": 0.04, "newinfiltration-1": 0.04, "newinfiltration-2": 0.015, "newhulk-2": 0.4} 347 | robust_radius_plot = np.arange(0, max_robust_radius["%s-%d" % (dataset_name, certify_class)], max_robust_radius["%s-%d" % (dataset_name, certify_class)] * 1e-3) 348 | certified_accuracy_plot = np.array([], dtype=np.float32) 349 | for r in robust_radius_plot: 350 | certified_accuracy_plot = np.append(certified_accuracy_plot, np.mean((label_record == cA_record) & (robust_radius_record > r))) 351 | plt.figure(1) 352 | plt.plot(robust_radius_plot, certified_accuracy_plot) 353 | plt.ylim((0, 1)) 354 | plt.xlim((0, max_robust_radius["%s-%d" % (dataset_name, certify_class)])) 355 | plt.tick_params(labelsize=14) 356 | plt.xlabel("Robustness Radius", fontsize=16) 357 | plt.ylabel("Certified Accuracy", fontsize=16) 358 | class_name = {"newinfiltration-0": "benign", "newinfiltration-1": "ssh-bruteforce", "newinfiltration-2": "dos-hulk", "newhulk-2": "infiltration"} 359 | plt.title("CADE %s" % (class_name["%s-%d" % (dataset_name, certify_class)]), fontsize=20) 360 | plt.tight_layout() 361 | plt.savefig(os.path.join(save_dir_certify, "certified_accuracy_robustness_radius_curve.png")) 362 | plt.close() 363 | 364 | def main(): 365 | os.environ["CUDA_VISIBLE_DEVICES"] = cuda_ids 366 | 367 | set_seed() # For reproductibility 368 | 369 | # Training 370 | train() 371 | 372 | # Evaluating 373 | evaluate() 374 | 375 | # Certifying 376 | certify() 377 | 378 | if __name__ == "__main__": 379 | main() -------------------------------------------------------------------------------- /smoothed_cade/save/newinfiltration/certify/certified_accuracy_robustness_radius_curve.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_cade/save/newinfiltration/certify/certified_accuracy_robustness_radius_curve.png -------------------------------------------------------------------------------- /smoothed_cade/save/newinfiltration/certify/checkpoint-distribution-transformer: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_cade/save/newinfiltration/certify/checkpoint-distribution-transformer -------------------------------------------------------------------------------- /smoothed_cade/save/newinfiltration/certify/t: -------------------------------------------------------------------------------- 1 | 2.398437e-01 -------------------------------------------------------------------------------- /smoothed_cade/save/newinfiltration/train/checkpoint-decoder: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_cade/save/newinfiltration/train/checkpoint-decoder -------------------------------------------------------------------------------- /smoothed_cade/save/newinfiltration/train/checkpoint-encoder: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_cade/save/newinfiltration/train/checkpoint-encoder -------------------------------------------------------------------------------- /smoothed_cade/save/newinfiltration/train/checkpoint-param: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_cade/save/newinfiltration/train/checkpoint-param -------------------------------------------------------------------------------- /smoothed_cade/utils.py: -------------------------------------------------------------------------------- 1 | import os, collections 2 | import numpy as np 3 | from sklearn.metrics import f1_score,precision_score,recall_score, confusion_matrix 4 | from typing import * 5 | 6 | import torch 7 | 8 | def load_data(data_dir: str, data_type: str="train"): 9 | 10 | if data_type == "train": 11 | x_train = np.load(os.path.join(data_dir, "X_train.npy")) 12 | y_train_class = np.load(os.path.join(data_dir, "y_train.npy")) 13 | 14 | class_map, num_classes_train = get_class_map(data_dir) 15 | y_train_class_new = np.zeros_like(y_train_class, dtype=np.long) 16 | for k, v in class_map.items(): 17 | y_train_class_new[y_train_class == k] = v 18 | 19 | return torch.tensor(x_train, dtype=torch.float), torch.tensor(y_train_class_new, dtype=torch.long), num_classes_train, class_map 20 | 21 | elif data_type == "test": 22 | x_test = np.load(os.path.join(data_dir, "X_test.npy")) 23 | y_test_class = np.load(os.path.join(data_dir, "y_test.npy")) 24 | 25 | class_map, num_classes_test = get_class_map(data_dir) 26 | y_test_class_new = np.zeros_like(y_test_class, dtype=np.long) 27 | for k, v in class_map.items(): 28 | y_test_class_new[y_test_class == k] = v 29 | 30 | y_train_class = np.load(os.path.join(data_dir, "y_train.npy")) 31 | train_class_set = np.unique(y_train_class).tolist() 32 | y_test_drift = np.ones_like(y_test_class, dtype=np.long) 33 | for i in range(len(train_class_set)): 34 | y_test_drift[y_test_class == train_class_set[i]] = 0 35 | 36 | return torch.tensor(x_test, dtype=torch.float), torch.tensor(y_test_class_new, dtype=torch.long), num_classes_test, class_map, torch.tensor(y_test_drift, dtype=torch.long) 37 | 38 | else: 39 | raise NotImplementedError() 40 | 41 | def get_class_map(data_dir: str): 42 | y_train = np.load(os.path.join(data_dir, "y_train.npy")) 43 | class_train = np.unique(y_train).tolist() 44 | class_train.sort() 45 | y_test = np.load(os.path.join(data_dir, "y_test.npy")) 46 | class_test = np.unique(y_test).tolist() 47 | class_test.sort() 48 | class_map = collections.OrderedDict() 49 | 50 | for i in range(len(class_train)): 51 | class_map.update({class_train[i]: len(class_map)}) 52 | for i in range(len(class_test)): 53 | if class_test[i] not in class_train: 54 | class_map.update({class_test[i]: len(class_map)}) 55 | 56 | return class_map, len(class_train) 57 | 58 | class AverageMeter(object): 59 | def __init__(self): 60 | self.reset() 61 | 62 | def reset(self): 63 | self.val = 0 64 | self.avg = 0 65 | self.sum = 0 66 | self.count = 0 67 | 68 | def update(self, val: float): 69 | self.val = val 70 | self.sum += val 71 | self.count += 1 72 | self.avg = self.sum / self.count 73 | 74 | def calc_metrics_classifier_class(label: np.ndarray, pred: np.ndarray): 75 | assert label.shape == pred.shape 76 | acc = np.array((label == pred), dtype=float).mean() 77 | m = confusion_matrix(y_true=label, y_pred=pred) 78 | 79 | return acc, m 80 | 81 | def calc_metrics_classifier(label: np.ndarray, pred: np.ndarray) -> (float, float, float, float): 82 | assert label.shape == pred.shape 83 | acc = np.array((label == pred), dtype=float).mean() 84 | p = precision_score(label, pred) 85 | r = recall_score(label, pred) 86 | f1 = f1_score(label, pred) 87 | 88 | return acc, p, r, f1 89 | 90 | def calc_metrics_certify(label: np.ndarray, pred: np.ndarray, radius_feat: np.ndarray) -> np.ndarray: 91 | assert label.shape == pred.shape 92 | assert pred.shape == radius_feat.shape 93 | 94 | return np.mean(np.array((label == pred), dtype=float) * radius_feat) -------------------------------------------------------------------------------- /smoothed_kitsune/classifier/autoencoder.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | import torch 4 | import torch.nn as nn 5 | 6 | class Autoencoder(nn.Module): 7 | def __init__(self, feature_size: float, device: object): 8 | super(Autoencoder, self).__init__() 9 | 10 | self.ae = nn.Sequential( 11 | nn.Linear(feature_size, int(np.ceil(feature_size / 2))), 12 | nn.ReLU(inplace=True), 13 | nn.Linear(int(np.ceil(feature_size / 2)), int(np.ceil(feature_size / 4))), 14 | nn.Linear(int(np.ceil(feature_size / 4)), int(np.ceil(feature_size / 2))), 15 | nn.ReLU(inplace=True), 16 | nn.Linear(int(np.ceil(feature_size / 2)), feature_size)) 17 | 18 | self.device = device 19 | 20 | def forward(self, x: torch.tensor) -> torch.tensor: 21 | return self.ae(x) -------------------------------------------------------------------------------- /smoothed_kitsune/classifier/corClust.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from scipy.cluster.hierarchy import linkage, to_tree 3 | 4 | # A helper class for KitNET which performs a correlation-based incremental clustering of the dimensions in X 5 | # n: the number of dimensions in the dataset 6 | # For more information and citation, please see our NDSS'18 paper: Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection 7 | class corClust: 8 | def __init__(self,n): 9 | #parameter: 10 | self.n = n 11 | #varaibles 12 | self.c = np.zeros(n) #linear num of features 13 | self.c_r = np.zeros(n) #linear sum of feature residules 14 | self.c_rs = np.zeros(n) #linear sum of feature residules 15 | self.C = np.zeros((n,n)) #partial correlation matrix 16 | self.N = 0 #number of updates performed 17 | 18 | # x: a numpy vector of length n 19 | def update(self,x): 20 | self.N += 1 21 | self.c += x 22 | c_rt = x - self.c/self.N 23 | self.c_r += c_rt 24 | self.c_rs += c_rt**2 25 | self.C += np.outer(c_rt,c_rt) 26 | 27 | # creates the current correlation distance matrix between the features 28 | def corrDist(self): 29 | c_rs_sqrt = np.sqrt(self.c_rs) 30 | C_rs_sqrt = np.outer(c_rs_sqrt,c_rs_sqrt) 31 | C_rs_sqrt[C_rs_sqrt==0] = 1e-100 #this protects against dive by zero erros (occurs when a feature is a constant) 32 | D = 1-self.C/C_rs_sqrt #the correlation distance matrix 33 | D[D<0] = 0 #small negatives may appear due to the incremental fashion in which we update the mean. Therefore, we 'fix' them 34 | return D 35 | 36 | # clusters the features together, having no more than maxClust features per cluster 37 | def cluster(self,maxClust): 38 | D = self.corrDist() 39 | Z = linkage(D[np.triu_indices(self.n, 1)]) # create a linkage matrix based on the distance matrix 40 | if maxClust < 1: 41 | maxClust = 1 42 | if maxClust > self.n: 43 | maxClust = self.n 44 | map = self.__breakClust__(to_tree(Z),maxClust) 45 | return map 46 | 47 | # a recursive helper function which breaks down the dendrogram branches until all clusters have no more than maxClust elements 48 | def __breakClust__(self,dendro,maxClust): 49 | if dendro.count <= maxClust: #base case: we found a minimal cluster, so mark it 50 | return [dendro.pre_order()] #return the origional ids of the features in this cluster 51 | return self.__breakClust__(dendro.get_left(),maxClust) + self.__breakClust__(dendro.get_right(),maxClust) 52 | 53 | # Copyright (c) 2017 Yisroel Mirsky 54 | # 55 | # MIT License 56 | # 57 | # Permission is hereby granted, free of charge, to any person obtaining 58 | # a copy of this software and associated documentation files (the 59 | # "Software"), to deal in the Software without restriction, including 60 | # without limitation the rights to use, copy, modify, merge, publish, 61 | # distribute, sublicense, and/or sell copies of the Software, and to 62 | # permit persons to whom the Software is furnished to do so, subject to 63 | # the following conditions: 64 | # 65 | # The above copyright notice and this permission notice shall be 66 | # included in all copies or substantial portions of the Software. 67 | # 68 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 69 | # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 70 | # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 71 | # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 72 | # LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 73 | # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 74 | # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /smoothed_kitsune/classifier/detect.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | class Detector(nn.Module): 5 | def __init__(self, normalizer: nn.Module, fm: nn.Module, ae: nn.Module, ad_threshold: float, device: object): 6 | super(Detector, self).__init__() 7 | 8 | self.normalizer = normalizer 9 | self.fm = fm 10 | self.ae = ae 11 | 12 | self.ad_threshold = ad_threshold 13 | 14 | self.device = device 15 | self.eps = 1e-16 16 | 17 | def forward(self, x: torch.tensor, noise: torch.tensor=0) -> torch.tensor: 18 | _, anomaly = self._predict_rmse_anomaly(x, noise) 19 | 20 | return anomaly 21 | 22 | def score(self, x: torch.tensor, noise: torch.tensor=0) -> torch.tensor: 23 | rmse, _ = self._predict_rmse_anomaly(x, noise) 24 | 25 | return rmse 26 | 27 | # We simplify Kitsune by removing the output layer autoencoder for ease of illustration. 28 | def _predict_rmse_anomaly(self, x: torch.tensor, noise: torch.tensor) -> (torch.tensor, torch.tensor): 29 | rmse = torch.zeros((x.size(0)), dtype=torch.float).to(self.device) 30 | x = self.normalizer(x) 31 | x += noise 32 | x = self.fm(x) 33 | for i in range(self.fm.get_num_clusters()): 34 | x_reconstructed = self.ae[i](x[i]) 35 | rmse += self._rmse(x[i], x_reconstructed) 36 | rmse /= self.fm.get_num_clusters() 37 | 38 | anomaly = torch.zeros_like(rmse, dtype=torch.long) 39 | anomaly[rmse > self.ad_threshold] = 1 40 | 41 | return rmse, anomaly 42 | 43 | def _rmse(self, x: torch.tensor, x_reconstruct: torch.tensor) -> torch.tensor: 44 | return torch.sqrt(torch.mean(torch.pow((x - x_reconstruct), 2), dim=1) + self.eps) -------------------------------------------------------------------------------- /smoothed_kitsune/classifier/preprocess.py: -------------------------------------------------------------------------------- 1 | from tqdm import trange 2 | from typing import * 3 | 4 | import torch 5 | import torch.nn as nn 6 | 7 | from classifier.corClust import corClust 8 | 9 | class Normalizer(nn.Module): 10 | def __init__(self, device: object): 11 | super(Normalizer, self).__init__() 12 | 13 | self.norm_max = None 14 | self.norm_min = None 15 | 16 | self.device = device 17 | self.eps = 1e-16 18 | 19 | def forward(self, x: torch.tensor) -> torch.tensor: 20 | if self.norm_max == None or self.norm_min == None: 21 | self.norm_max = torch.max(x, 0)[0].to(self.device) 22 | self.norm_min = torch.min(x, 0)[0].to(self.device) 23 | 24 | return (x - self.norm_min) / (self.norm_max - self.norm_min + self.eps) 25 | 26 | def update(self, x: torch.tensor): 27 | if self.norm_max == None or self.norm_min == None: 28 | self.norm_max = torch.max(x, 0)[0].to(self.device) 29 | self.norm_min = torch.min(x, 0)[0].to(self.device) 30 | else: 31 | self.norm_max = torch.max(torch.cat([x, self.norm_max.unsqueeze(0)], 0), 0)[0] 32 | self.norm_min = torch.min(torch.cat([x, self.norm_min.unsqueeze(0)], 0), 0)[0] 33 | 34 | class Feature_Map(nn.Module): 35 | def __init__(self, device: object): 36 | super(Feature_Map, self).__init__() 37 | 38 | self.mp = None 39 | 40 | self.device = device 41 | 42 | def init(self, x: torch.tensor, maxClust: int=10): 43 | c = corClust(x.size(1)) 44 | for i in trange(x.size(0), desc="Feat Clust"): 45 | c.update(x[i, :].detach().cpu().numpy()) 46 | self.mp = c.cluster(maxClust) 47 | 48 | def forward(self, x: torch.tensor) -> List: 49 | if self.mp == None: 50 | self.init(x) 51 | 52 | x_mapped = [] 53 | for m in self.mp: 54 | x_mapped.append(x[:, m]) 55 | 56 | return x_mapped 57 | 58 | def get_num_clusters(self) -> int: 59 | return len(self.mp) -------------------------------------------------------------------------------- /smoothed_kitsune/data/mirai.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/data/mirai.zip -------------------------------------------------------------------------------- /smoothed_kitsune/data/ssdp.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/data/ssdp.zip -------------------------------------------------------------------------------- /smoothed_kitsune/main.py: -------------------------------------------------------------------------------- 1 | import os, random 2 | import numpy as np 3 | from tqdm import tqdm 4 | 5 | import sys 6 | sys.path.append('../bars/') 7 | 8 | import matplotlib.pyplot as plt 9 | # For "Could not connect to any X display." 10 | import matplotlib 11 | matplotlib.use('Agg') 12 | 13 | import seaborn as sns 14 | sns.set_style("darkgrid") 15 | 16 | import warnings 17 | warnings.filterwarnings("ignore") 18 | 19 | import torch 20 | import torch.nn as nn 21 | import torch.optim as optim 22 | from torch.utils.data import DataLoader, TensorDataset 23 | 24 | from classifier.autoencoder import Autoencoder 25 | from classifier.detect import Detector 26 | from classifier.preprocess import Normalizer, Feature_Map 27 | from utils import load_data, AverageMeter, calc_metrics_classifier, calc_metrics_certify 28 | 29 | from smoothing import Smooth2, Noise 30 | from distribution_transformer import distribution_transformers, loss_functions 31 | from optimizing_noise import optimizing_noise 32 | 33 | # General 34 | no_gpu = False # Avoid using CUDA when available 35 | cuda_ids = "0" # GPU No 36 | n_gpu = torch.cuda.device_count() 37 | device = torch.device("cuda" if torch.cuda.is_available() and not no_gpu else "cpu") 38 | data_dir = "data" # Data directory 39 | dataset_name = "mirai" # The name of data set. e.g., "mirai" "ssdp" 40 | if dataset_name not in os.listdir(data_dir): 41 | raise ValueError("Dataset not found: %s" % (dataset_name)) 42 | data_dir = os.path.join(data_dir, dataset_name) 43 | 44 | # Building classifier 45 | learning_rate_classifier = 1e-3 # Learning rate for training classifier 46 | ad_threshold_dict = {"mirai": 0.1, "ssdp": 0.3} 47 | ad_threshold = ad_threshold_dict[dataset_name] # Threshold for anomaly detection 48 | num_epochs_classifier = 30 # Epoch number for training classifier 49 | batch_size_train = 512 # Batch size for training classifier 50 | batch_size_eval = 2048 # Batch size for evaluating classifier 51 | save_dir_train = "save/%s/train" % (dataset_name) # Saving directory for classifier training 52 | if not os.path.exists(save_dir_train): 53 | os.makedirs(save_dir_train) 54 | print_step_classifier = 10 # Step size for showing training progress 55 | 56 | # Certifying classifier 57 | certify_class = 0 # The certified class (benign class) 58 | feature_noise_distribution = "isru" # Feature noise distribution. e.g., "gaussian" "isru" "isru_gaussian" "isru_gaussian_arctan" 59 | learning_rate_shape = 1e-2 # Learning rate for optimzing noise shape 60 | nt_shape = 1000 # Number of noised samples for optimzing noise shape 61 | lambda_shape = 1e-3 # Regularizer weight 62 | num_epochs_shape = 5 # Number of epochs for optimzing noise shape 63 | x_train = load_data(data_dir, "train") 64 | d = x_train.size(1) # Number of feature dimensions 65 | num_classes_certify = 2 # Number of certified classes. i.e., Benign(0), Anomaly(1) 66 | n0 = 100 # Number of noised samples for identify cA 67 | n = 10000 # Number of noised samples for estimate pA 68 | alpha = 1e-3 # Failure probability 69 | init_step_size_scale = 5e-2 # Initial update step size of t for optimzing noise scale 70 | init_ptb_t_scale = 1e-2 # Initial perturbation of t for optimzing noise scale 71 | decay_factor_scale = 0.5 # Decay factor for optimzing noise scale 72 | max_decay_scale = 6 # Maximum decay times for optimzing noise scale 73 | max_iter_scale = 100 # Maximum iteration times for optimzing noise scale 74 | batch_size_iteration_certify = 128 # Batch size of certified samples for robustness certification 75 | batch_size_memory_certify = 1000000 # Batch size of noised samples for robustness certification 76 | print_step_certify = 20 # Step size for showing certification progress 77 | save_dir_certify = "save/%s/certify" % (dataset_name) # Saving directory for robustness certification 78 | if not os.path.exists(save_dir_certify): 79 | os.makedirs(save_dir_certify) 80 | num_samples_certify = 1000 # Number of certified samples sampled from dataset 81 | 82 | seed = 42 # Random seed 83 | 84 | def set_seed(): 85 | random.seed(seed) 86 | np.random.seed(seed) 87 | torch.manual_seed(seed) 88 | if n_gpu > 0: 89 | torch.cuda.manual_seed_all(seed) 90 | 91 | def create_ae(feature_size: int) -> nn.Module: 92 | return Autoencoder(feature_size, device) 93 | 94 | def create_normalizer(): 95 | return Normalizer(device) 96 | 97 | def create_fm(): 98 | return Feature_Map(device) 99 | 100 | def create_detector(normalizer: nn.Module, fm: nn.Module, ae: nn.Module) -> nn.Module: 101 | return Detector(normalizer, fm, ae, ad_threshold, device) 102 | 103 | def create_noise_generator(): 104 | dist_trans = distribution_transformers[feature_noise_distribution](d).to(device) 105 | return Noise(dist_trans, d, device) 106 | 107 | def train(): 108 | print("\n***** Run clustering *****") 109 | 110 | x_cluster = load_data(data_dir, "cluster") 111 | fm = create_fm() 112 | fm.init(x_cluster) 113 | 114 | torch.save(fm.mp, os.path.join(save_dir_train, "checkpoint-fm")) 115 | 116 | criterion = nn.MSELoss() 117 | 118 | x_train = load_data(data_dir, "train") 119 | dataset = TensorDataset(x_train) 120 | data_loader = DataLoader(dataset, batch_size=batch_size_train, shuffle=False) 121 | 122 | normalizer = create_normalizer() 123 | normalizer.update(x_train) 124 | torch.save({"norm_max": normalizer.norm_max, "norm_min": normalizer.norm_min 125 | }, os.path.join(save_dir_train, "checkpoint-norm")) 126 | 127 | for i in range(fm.get_num_clusters()): 128 | print("\n***** Run training AE %d *****" % (i)) 129 | 130 | ae = create_ae(len(fm.mp[i])) 131 | ae.to(device) 132 | 133 | opt = optim.Adam(ae.parameters(), lr=learning_rate_classifier) 134 | 135 | ae.train() 136 | for epoch in range(num_epochs_classifier + 1): 137 | loss_record = AverageMeter() 138 | for (X,) in data_loader: 139 | X = X.to(ae.device) 140 | 141 | X = normalizer(X) 142 | X = X[:, fm.mp[i]] 143 | x_reconstructed = ae(X) 144 | loss = criterion(X, x_reconstructed) 145 | 146 | if epoch > 0: 147 | opt.zero_grad() 148 | loss.backward() 149 | opt.step() 150 | 151 | loss_record.update(loss.item()) 152 | 153 | if epoch % print_step_classifier == 0: 154 | print(('Epoch: [%d/%d] | MSE Loss (Avg): %.6f') % ( \ 155 | epoch, num_epochs_classifier, loss_record.avg)) 156 | 157 | torch.save(ae.ae, os.path.join(save_dir_train, "checkpoint-ae-%d" % (i))) 158 | 159 | def evaluate(): 160 | print("\n***** Run evaluating *****") 161 | 162 | normalizer = create_normalizer() 163 | norm_param = torch.load(os.path.join(save_dir_train, "checkpoint-norm")) 164 | normalizer.norm_max = norm_param["norm_max"].to(device) 165 | normalizer.norm_min = norm_param["norm_min"].to(device) 166 | fm = create_fm() 167 | fm.mp = torch.load(os.path.join(save_dir_train, "checkpoint-fm")) 168 | detector = create_detector(normalizer, fm, nn.ModuleList([torch.load(os.path.join(save_dir_train, "checkpoint-ae-%d" % (i) 169 | )).to(device) for i in range(fm.get_num_clusters())])) 170 | 171 | x_test, y_test = load_data(data_dir, "test") 172 | 173 | dataset = TensorDataset(x_test, y_test) 174 | data_loader = DataLoader(dataset, batch_size=batch_size_eval, shuffle=False) 175 | 176 | rmse_record = np.array([], dtype=np.float32) 177 | pred_record = np.array([], dtype=np.long) 178 | label_record = np.array([], dtype=np.long) 179 | detector.eval() 180 | torch.set_grad_enabled(False) 181 | for X, y in tqdm(data_loader, desc="Evaluate"): 182 | X, y = X.to(detector.device), y.to(detector.device) 183 | 184 | pred = detector(X) 185 | rmse = detector.score(X) 186 | 187 | rmse_record = np.concatenate([rmse_record, rmse.detach().cpu().numpy()], 0) 188 | pred_record = np.concatenate([pred_record, pred.detach().cpu().numpy()], 0) 189 | 190 | label_record = np.concatenate([label_record, y.detach().cpu().numpy()], 0) 191 | 192 | torch.set_grad_enabled(True) 193 | torch.cuda.empty_cache() 194 | 195 | acc, p, r, f1 = calc_metrics_classifier(pred_record, label_record) 196 | 197 | print("Accuracy: %.4f | Precision: %.4f | Recall: %.4f | F1 Score: %.4f" % (acc, p, r, f1)) 198 | 199 | def certify(): 200 | print("\n***** Optimize noise *****") 201 | 202 | normalizer = create_normalizer() 203 | norm_param = torch.load(os.path.join(save_dir_train, "checkpoint-norm")) 204 | normalizer.norm_max = norm_param["norm_max"].to(device) 205 | normalizer.norm_min = norm_param["norm_min"].to(device) 206 | fm = create_fm() 207 | fm.mp = torch.load(os.path.join(save_dir_train, "checkpoint-fm")) 208 | detector = create_detector(normalizer, fm, nn.ModuleList([torch.load(os.path.join(save_dir_train, "checkpoint-ae-%d" % (i) 209 | )).to(device) for i in range(fm.get_num_clusters())])) 210 | 211 | noise_generator = create_noise_generator() 212 | 213 | criterion_shape = loss_functions["kitsune"](lambda_shape, ad_threshold) 214 | 215 | x_train = load_data(data_dir, "train") 216 | 217 | dataset = TensorDataset(x_train) 218 | data_loader = DataLoader(dataset, batch_size=batch_size_iteration_certify, shuffle=False) 219 | x_train_c = torch.tensor([], dtype=torch.float32) 220 | for (X,) in tqdm(data_loader, desc="Select class %d" % (certify_class)): 221 | X = X.to(device) 222 | 223 | anomaly = detector(X) 224 | 225 | x_train_c = torch.cat([x_train_c, X[(anomaly == certify_class), :].detach().cpu()], 0) 226 | 227 | optimizing_noise( 228 | x_train_c, 229 | certify_class, 230 | detector, 231 | noise_generator, 232 | criterion_shape, 233 | learning_rate_shape, 234 | nt_shape, 235 | num_epochs_shape, 236 | d, 237 | num_classes_certify, 238 | n0, 239 | n, 240 | alpha, 241 | init_step_size_scale, 242 | init_ptb_t_scale, 243 | decay_factor_scale, 244 | max_decay_scale, 245 | max_iter_scale, 246 | batch_size_iteration_certify, 247 | batch_size_memory_certify, 248 | print_step_certify, 249 | save_dir_certify) 250 | 251 | noise_generator.distribution_transformer = torch.load(os.path.join(save_dir_certify, "checkpoint-distribution-transformer")).to(device) 252 | 253 | r = open(os.path.join(save_dir_certify, "t"), "r") 254 | t = float(r.readline()) 255 | r.close() 256 | 257 | print("\n***** Certify robustness *****") 258 | 259 | x_test, y_test = load_data(data_dir, "test") 260 | 261 | dataset = TensorDataset(x_test, y_test) 262 | data_loader = DataLoader(dataset, batch_size=batch_size_iteration_certify, shuffle=False) 263 | x_test_c = torch.tensor([], dtype=torch.float32) 264 | y_test_c = torch.tensor([], dtype=torch.long) 265 | for (X, y) in tqdm(data_loader, desc="Select class %d" % (certify_class)): 266 | X = X.to(device) 267 | 268 | anomaly = detector(X) 269 | 270 | x_test_c = torch.cat([x_test_c, X[(anomaly == certify_class), :].detach().cpu()], 0) 271 | y_test_c = torch.cat([y_test_c, y[(anomaly == certify_class)].detach().cpu()], 0) 272 | 273 | idx = torch.arange(x_test_c.shape[0]) 274 | idx = idx[torch.randperm(idx.size(0))] 275 | idx = torch.sort(idx[:min(num_samples_certify, idx.shape[0])])[0] 276 | x_test_c, y_test_c = x_test_c[idx, :], y_test_c[idx] 277 | 278 | dataset = TensorDataset(x_test_c, y_test_c) 279 | data_loader = DataLoader(dataset, batch_size=batch_size_iteration_certify, shuffle=False) 280 | 281 | detector.eval() 282 | smoothed_classifier = Smooth2(detector, d, num_classes_certify, noise_generator, device) 283 | cA_record = np.array([], dtype=np.long) 284 | robust_radius_record = np.array([], dtype=np.float32) 285 | label_record = np.array([], dtype=np.long) 286 | torch.set_grad_enabled(False) 287 | for X, y in tqdm(data_loader, desc="Certify"): 288 | X = X.to(device) 289 | 290 | cA, _, robust_radius = smoothed_classifier.bars_certify(X, n0, n, t, alpha, batch_size_memory_certify) 291 | 292 | cA_record = np.concatenate([cA_record, cA], 0) 293 | robust_radius_record = np.concatenate([robust_radius_record, robust_radius], 0) 294 | 295 | label_record = np.concatenate([label_record, y.detach().cpu().numpy()], 0) 296 | 297 | torch.set_grad_enabled(True) 298 | torch.cuda.empty_cache() 299 | 300 | mean_robust_radius = calc_metrics_certify(label_record, cA_record, robust_radius_record) 301 | print("Mean Robustness Radius: %.6e" % (mean_robust_radius)) 302 | 303 | w = open(os.path.join(save_dir_certify, "certification_results"), "w") 304 | w.write("label\tcA\trobust_radius\n") 305 | for i in range(label_record.shape[0]): 306 | w.write("%d\t%d\t%.6e\n" % (label_record[i], cA_record[i], robust_radius_record[i])) 307 | w.close() 308 | 309 | max_robust_radius = {"mirai": 0.6, "ssdp": 1.5} 310 | robust_radius_plot = np.arange(0, max_robust_radius[dataset_name], max_robust_radius[dataset_name] * 1e-3) 311 | certified_accuracy_plot = np.array([], dtype=np.float32) 312 | for r in robust_radius_plot: 313 | certified_accuracy_plot = np.append(certified_accuracy_plot, np.mean((label_record == cA_record) & (robust_radius_record > r))) 314 | plt.figure(1) 315 | plt.plot(robust_radius_plot, certified_accuracy_plot) 316 | plt.ylim((0, 1)) 317 | plt.xlim((0, max_robust_radius[dataset_name])) 318 | plt.tick_params(labelsize=14) 319 | plt.xlabel("Robustness Radius", fontsize=16) 320 | plt.ylabel("Certified Accuracy", fontsize=16) 321 | plt.title("Kitsune %s" % (dataset_name), fontsize=20) 322 | plt.tight_layout() 323 | plt.savefig(os.path.join(save_dir_certify, "certified_accuracy_robustness_radius_curve.png")) 324 | plt.close() 325 | 326 | def main(): 327 | os.environ["CUDA_VISIBLE_DEVICES"] = cuda_ids 328 | 329 | set_seed() # For reproductibility 330 | 331 | # Training 332 | train() 333 | 334 | # Evaluating 335 | evaluate() 336 | 337 | # Certifying 338 | certify() 339 | 340 | if __name__ == "__main__": 341 | main() -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/certify/certification_results: -------------------------------------------------------------------------------- 1 | label cA robust_radius 2 | 0 0 5.073727e-01 3 | 0 0 4.499490e-01 4 | 0 0 4.725347e-01 5 | 0 0 4.725347e-01 6 | 0 0 4.499490e-01 7 | 0 0 4.725347e-01 8 | 0 0 4.499490e-01 9 | 0 0 4.193166e-01 10 | 0 0 4.725347e-01 11 | 0 0 4.725347e-01 12 | 0 0 4.499490e-01 13 | 0 0 4.725347e-01 14 | 0 0 4.725347e-01 15 | 0 0 4.725347e-01 16 | 0 0 5.073727e-01 17 | 0 0 4.329771e-01 18 | 0 0 4.329771e-01 19 | 0 0 3.520249e-01 20 | 0 0 5.073727e-01 21 | 0 0 5.073727e-01 22 | 0 0 3.212063e-01 23 | 0 0 4.499490e-01 24 | 0 0 4.329771e-01 25 | 0 0 4.078641e-01 26 | 0 0 4.725347e-01 27 | 0 0 5.073727e-01 28 | 0 0 4.725347e-01 29 | 0 0 5.073727e-01 30 | 0 0 4.499490e-01 31 | 0 0 4.329771e-01 32 | 0 0 4.499490e-01 33 | 0 0 4.725347e-01 34 | 0 0 5.073727e-01 35 | 0 0 2.600661e-01 36 | 0 0 3.473461e-01 37 | 0 0 4.725347e-01 38 | 0 0 5.073727e-01 39 | 0 0 3.624273e-01 40 | 0 0 4.193166e-01 41 | 0 0 4.725347e-01 42 | 0 0 4.329771e-01 43 | 0 0 4.725347e-01 44 | 0 0 3.388294e-01 45 | 0 0 4.193166e-01 46 | 0 0 1.015666e-01 47 | 0 0 4.329771e-01 48 | 0 0 4.725347e-01 49 | 0 0 4.499490e-01 50 | 0 0 1.516184e-01 51 | 0 0 4.193166e-01 52 | 0 0 4.499490e-01 53 | 0 0 9.143221e-02 54 | 0 0 5.073727e-01 55 | 0 0 4.499490e-01 56 | 0 0 5.073727e-01 57 | 0 0 1.187180e-01 58 | 0 0 4.725347e-01 59 | 0 0 4.499490e-01 60 | 0 0 5.073727e-01 61 | 0 0 4.725347e-01 62 | 0 0 4.725347e-01 63 | 0 0 4.078641e-01 64 | 0 0 4.193166e-01 65 | 0 0 2.552778e-01 66 | 0 0 4.193166e-01 67 | 0 0 4.725347e-01 68 | 0 0 4.499490e-01 69 | 0 0 1.417676e-02 70 | 0 0 5.073727e-01 71 | 0 0 4.725347e-01 72 | 0 0 4.499490e-01 73 | 0 0 4.725347e-01 74 | 0 0 4.725347e-01 75 | 0 0 4.193166e-01 76 | 0 0 4.725347e-01 77 | 0 0 5.073727e-01 78 | 0 0 5.073727e-01 79 | 0 0 2.519094e-01 80 | 0 0 4.329771e-01 81 | 0 0 4.499490e-01 82 | 0 0 4.193166e-01 83 | 0 0 4.499490e-01 84 | 0 0 4.078641e-01 85 | 0 0 3.243911e-01 86 | 0 0 4.725347e-01 87 | 0 0 4.193166e-01 88 | 0 0 4.725347e-01 89 | 0 0 4.725347e-01 90 | 0 0 4.725347e-01 91 | 0 0 5.073727e-01 92 | 0 0 6.219573e-02 93 | 0 0 4.725347e-01 94 | 0 0 5.073727e-01 95 | 0 0 3.815948e-01 96 | 0 1 8.291114e-02 97 | 0 0 4.725347e-01 98 | 0 0 1.689676e-01 99 | 0 0 1.337546e-01 100 | 0 0 5.073727e-01 101 | 0 0 5.073727e-01 102 | 0 0 4.499490e-01 103 | 0 0 4.725347e-01 104 | 0 0 4.499490e-01 105 | 0 0 4.499490e-01 106 | 0 0 3.022869e-01 107 | 0 0 4.329771e-01 108 | 0 0 4.499490e-01 109 | 0 0 5.073727e-01 110 | 0 0 5.073727e-01 111 | 0 0 2.955546e-01 112 | 0 0 4.725347e-01 113 | 0 0 4.499490e-01 114 | 0 0 4.499490e-01 115 | 0 0 4.499490e-01 116 | 0 0 4.725347e-01 117 | 0 0 3.979970e-01 118 | 0 0 4.499490e-01 119 | 0 0 5.073727e-01 120 | 0 0 5.073727e-01 121 | 0 0 4.193166e-01 122 | 0 0 5.073727e-01 123 | 0 0 3.682621e-01 124 | 0 0 3.979970e-01 125 | 0 0 4.725347e-01 126 | 0 0 5.073727e-01 127 | 0 0 3.152468e-01 128 | 0 0 3.893271e-01 129 | 0 0 2.770314e-01 130 | 0 0 9.664940e-02 131 | 0 0 4.499490e-01 132 | 0 0 4.499490e-01 133 | 0 0 4.499490e-01 134 | 0 0 4.499490e-01 135 | 0 0 5.073727e-01 136 | 0 0 5.073727e-01 137 | 0 0 5.073727e-01 138 | 0 0 1.027869e-01 139 | 0 0 4.725347e-01 140 | 0 0 4.193166e-01 141 | 0 0 4.499490e-01 142 | 0 0 4.329771e-01 143 | 0 0 4.078641e-01 144 | 0 0 5.073727e-01 145 | 0 0 4.499490e-01 146 | 0 0 5.073727e-01 147 | 0 0 3.979970e-01 148 | 0 0 4.193166e-01 149 | 0 0 5.073727e-01 150 | 0 0 3.979970e-01 151 | 0 0 7.098564e-02 152 | 0 0 4.725347e-01 153 | 0 0 4.725347e-01 154 | 0 0 4.329771e-01 155 | 0 0 4.725347e-01 156 | 0 0 3.181620e-01 157 | 0 0 4.499490e-01 158 | 0 0 5.819682e-03 159 | 0 0 4.725347e-01 160 | 0 0 4.725347e-01 161 | 0 0 3.570354e-01 162 | 0 0 5.073727e-01 163 | 0 0 5.073727e-01 164 | 0 0 4.499490e-01 165 | 0 0 4.499490e-01 166 | 0 0 4.499490e-01 167 | 0 0 5.073727e-01 168 | 0 0 7.875256e-02 169 | 0 0 4.725347e-01 170 | 0 0 4.725347e-01 171 | 0 0 4.329771e-01 172 | 0 0 4.078641e-01 173 | 0 0 4.078641e-01 174 | 0 0 4.725347e-01 175 | 0 0 4.725347e-01 176 | 0 0 5.073727e-01 177 | 0 0 4.725347e-01 178 | 0 0 4.725347e-01 179 | 0 0 4.499490e-01 180 | 0 0 4.499490e-01 181 | 0 0 4.725347e-01 182 | 0 0 4.725347e-01 183 | 0 0 1.079187e-01 184 | 0 0 3.624273e-01 185 | 0 0 4.078641e-01 186 | 0 0 4.725347e-01 187 | 0 0 1.543799e-01 188 | 0 0 4.329771e-01 189 | 0 0 4.725347e-01 190 | 0 0 3.349299e-01 191 | 0 0 4.078641e-01 192 | 0 0 5.073727e-01 193 | 0 0 5.073727e-01 194 | 0 0 4.847942e-02 195 | 0 0 3.570354e-01 196 | 0 0 4.499490e-01 197 | 0 0 4.725347e-01 198 | 0 0 4.725347e-01 199 | 0 0 4.725347e-01 200 | 0 0 4.078641e-01 201 | 0 0 4.193166e-01 202 | 0 0 4.499490e-01 203 | 0 0 2.541350e-01 204 | 0 0 1.314958e-01 205 | 0 0 3.097638e-01 206 | 0 0 5.073727e-01 207 | 0 0 3.979970e-01 208 | 0 0 2.708698e-01 209 | 0 0 6.967912e-02 210 | 0 0 4.329771e-01 211 | 0 0 4.329771e-01 212 | 0 0 4.499490e-01 213 | 0 1 1.687022e-01 214 | 0 0 4.078641e-01 215 | 0 0 3.473461e-01 216 | 0 0 4.329771e-01 217 | 0 0 4.725347e-01 218 | 0 0 4.193166e-01 219 | 0 0 5.073727e-01 220 | 0 0 4.725347e-01 221 | 0 0 4.193166e-01 222 | 0 0 4.499490e-01 223 | 0 0 4.725347e-01 224 | 0 0 4.329771e-01 225 | 0 0 1.456335e-01 226 | 0 0 4.499490e-01 227 | 0 0 4.329771e-01 228 | 0 0 4.725347e-01 229 | 0 0 4.725347e-01 230 | 0 0 4.725347e-01 231 | 0 0 4.329771e-01 232 | 0 0 4.499490e-01 233 | 0 0 4.193166e-01 234 | 0 0 3.979970e-01 235 | 0 0 4.499490e-01 236 | 0 0 4.725347e-01 237 | 0 0 4.078641e-01 238 | 0 0 4.499490e-01 239 | 0 0 4.499490e-01 240 | 0 0 4.725347e-01 241 | 0 0 5.073727e-01 242 | 0 0 4.329771e-01 243 | 0 0 4.725347e-01 244 | 0 0 3.893271e-01 245 | 0 0 4.193166e-01 246 | 0 0 1.581572e-01 247 | 0 0 5.073727e-01 248 | 0 0 5.073727e-01 249 | 0 0 4.329771e-01 250 | 0 0 4.499490e-01 251 | 0 0 4.499490e-01 252 | 0 0 4.499490e-01 253 | 0 0 5.073727e-01 254 | 0 0 4.499490e-01 255 | 0 0 4.499490e-01 256 | 0 0 4.193166e-01 257 | 0 1 9.074696e-02 258 | 0 0 4.725347e-01 259 | 0 0 4.725347e-01 260 | 0 0 4.499490e-01 261 | 0 0 4.499490e-01 262 | 0 0 4.725347e-01 263 | 0 0 4.499490e-01 264 | 0 0 4.193166e-01 265 | 0 0 4.193166e-01 266 | 0 0 4.499490e-01 267 | 0 0 4.499490e-01 268 | 0 0 4.499490e-01 269 | 0 0 5.073727e-01 270 | 0 0 5.073727e-01 271 | 0 0 4.499490e-01 272 | 0 1 2.600661e-01 273 | 0 0 4.499490e-01 274 | 0 0 3.893271e-01 275 | 0 0 4.725347e-01 276 | 0 0 3.893271e-01 277 | 0 0 4.725347e-01 278 | 0 0 4.725347e-01 279 | 0 0 3.979970e-01 280 | 0 0 5.073727e-01 281 | 0 0 4.725347e-01 282 | 0 0 5.073727e-01 283 | 0 0 5.073727e-01 284 | 0 0 1.220474e-01 285 | 0 0 5.073727e-01 286 | 0 0 2.999673e-01 287 | 0 0 4.329771e-01 288 | 0 0 4.329771e-01 289 | 0 0 4.725347e-01 290 | 0 0 5.073727e-01 291 | 0 0 4.329771e-01 292 | 0 0 4.725347e-01 293 | 0 0 4.725347e-01 294 | 0 0 3.893271e-01 295 | 0 0 5.073727e-01 296 | 0 0 4.499490e-01 297 | 0 0 4.725347e-01 298 | 0 0 1.300863e-01 299 | 0 0 2.770314e-01 300 | 0 0 3.349299e-01 301 | 0 0 5.073727e-01 302 | 0 0 3.046889e-01 303 | 0 0 4.329771e-01 304 | 0 0 4.499490e-01 305 | 0 0 3.682621e-01 306 | 0 0 5.073727e-01 307 | 0 0 5.073727e-01 308 | 0 0 4.499490e-01 309 | 0 0 4.193166e-01 310 | 0 0 3.520249e-01 311 | 0 0 4.499490e-01 312 | 0 0 5.073727e-01 313 | 0 0 4.725347e-01 314 | 0 0 4.499490e-01 315 | 0 0 4.725347e-01 316 | 0 0 1.560255e-01 317 | 0 0 5.073727e-01 318 | 0 0 1.096450e-01 319 | 0 0 4.725347e-01 320 | 0 1 1.535760e-01 321 | 0 0 4.725347e-01 322 | 0 0 4.499490e-01 323 | 0 0 4.725347e-01 324 | 0 0 5.073727e-01 325 | 0 0 4.193166e-01 326 | 0 0 4.193166e-01 327 | 0 0 4.499490e-01 328 | 0 0 3.815948e-01 329 | 0 0 5.073727e-01 330 | 0 0 2.270050e-01 331 | 0 0 4.499490e-01 332 | 0 0 5.073727e-01 333 | 0 0 4.329771e-01 334 | 0 0 3.893271e-01 335 | 0 0 4.329771e-01 336 | 0 0 4.499490e-01 337 | 0 0 3.520249e-01 338 | 0 0 1.568679e-01 339 | 0 0 4.193166e-01 340 | 0 0 4.329771e-01 341 | 0 0 4.329771e-01 342 | 0 0 1.566561e-01 343 | 0 0 5.073727e-01 344 | 0 0 5.073727e-01 345 | 0 0 4.725347e-01 346 | 0 0 5.073727e-01 347 | 0 0 4.725347e-01 348 | 0 0 4.725347e-01 349 | 0 0 5.073727e-01 350 | 0 0 4.725347e-01 351 | 0 0 4.499490e-01 352 | 0 0 4.725347e-01 353 | 0 0 4.078641e-01 354 | 0 0 5.073727e-01 355 | 0 0 4.499490e-01 356 | 0 0 5.073727e-01 357 | 0 0 4.193166e-01 358 | 0 0 3.979970e-01 359 | 0 0 3.893271e-01 360 | 0 0 3.022869e-01 361 | 0 0 4.499490e-01 362 | 0 0 3.979970e-01 363 | 0 0 5.073727e-01 364 | 0 0 4.499490e-01 365 | 0 0 3.893271e-01 366 | 0 0 5.073727e-01 367 | 0 0 4.329771e-01 368 | 0 0 3.152468e-01 369 | 0 0 5.073727e-01 370 | 0 0 1.302128e-01 371 | 0 0 4.499490e-01 372 | 0 0 4.193166e-01 373 | 0 0 5.073727e-01 374 | 0 0 2.914141e-01 375 | 0 0 4.725347e-01 376 | 0 0 4.078641e-01 377 | 0 1 8.385200e-03 378 | 0 0 5.073727e-01 379 | 0 0 4.725347e-01 380 | 0 0 4.329771e-01 381 | 0 0 4.499490e-01 382 | 0 0 2.977248e-01 383 | 0 0 8.477966e-02 384 | 0 0 4.193166e-01 385 | 0 0 5.073727e-01 386 | 0 0 1.137446e-01 387 | 0 1 7.098564e-02 388 | 0 0 4.499490e-01 389 | 0 0 4.725347e-01 390 | 0 0 2.694163e-01 391 | 0 0 2.127497e-01 392 | 0 0 4.725347e-01 393 | 0 0 4.725347e-01 394 | 0 0 4.193166e-01 395 | 0 0 1.867145e-01 396 | 0 0 3.979970e-01 397 | 0 0 4.499490e-01 398 | 0 0 4.725347e-01 399 | 0 0 4.499490e-01 400 | 0 0 1.043380e-01 401 | 0 0 4.725347e-01 402 | 0 0 4.725347e-01 403 | 0 0 3.746177e-01 404 | 0 0 4.725347e-01 405 | 0 0 4.078641e-01 406 | 0 0 4.725347e-01 407 | 0 0 4.499490e-01 408 | 0 0 4.725347e-01 409 | 0 0 5.073727e-01 410 | 0 0 4.725347e-01 411 | 0 0 4.329771e-01 412 | 0 0 5.073727e-01 413 | 0 0 4.725347e-01 414 | 0 0 4.725347e-01 415 | 0 0 5.073727e-01 416 | 0 0 4.725347e-01 417 | 0 0 4.329771e-01 418 | 0 0 4.078641e-01 419 | 0 0 4.193166e-01 420 | 0 0 4.329771e-01 421 | 0 0 3.815948e-01 422 | 0 0 5.073727e-01 423 | 0 0 5.073727e-01 424 | 0 0 4.725347e-01 425 | 0 0 3.893271e-01 426 | 0 0 5.073727e-01 427 | 0 0 4.725347e-01 428 | 0 0 4.329771e-01 429 | 0 0 4.499490e-01 430 | 0 0 4.078641e-01 431 | 0 0 4.499490e-01 432 | 0 0 4.725347e-01 433 | 0 0 4.499490e-01 434 | 0 0 4.329771e-01 435 | 0 0 5.073727e-01 436 | 0 0 5.073727e-01 437 | 0 0 4.725347e-01 438 | 0 0 4.499490e-01 439 | 0 0 4.329771e-01 440 | 0 0 3.520249e-01 441 | 0 0 4.725347e-01 442 | 0 0 5.073727e-01 443 | 0 0 4.078641e-01 444 | 0 0 1.171336e-01 445 | 0 0 5.073727e-01 446 | 0 0 4.078641e-01 447 | 0 0 5.127847e-02 448 | 0 0 3.815948e-01 449 | 0 0 4.725347e-01 450 | 0 0 5.073727e-01 451 | 0 0 1.405783e-01 452 | 0 0 5.073727e-01 453 | 0 0 5.073727e-01 454 | 0 0 4.499490e-01 455 | 0 0 2.541350e-01 456 | 0 -1 0.000000e+00 457 | 0 0 4.078641e-01 458 | 0 0 4.725347e-01 459 | 0 0 1.099809e-01 460 | 0 0 3.682621e-01 461 | 0 0 1.933229e-01 462 | 0 0 4.725347e-01 463 | 0 0 4.329771e-01 464 | 0 0 5.073727e-01 465 | 0 0 4.499490e-01 466 | 0 1 5.073727e-01 467 | 0 0 4.725347e-01 468 | 0 0 5.073727e-01 469 | 0 0 4.329771e-01 470 | 0 0 4.329771e-01 471 | 0 0 3.979970e-01 472 | 0 0 3.979970e-01 473 | 0 0 4.725347e-01 474 | 0 0 5.073727e-01 475 | 0 0 4.725347e-01 476 | 0 0 4.499490e-01 477 | 0 0 4.725347e-01 478 | 0 0 4.725347e-01 479 | 0 0 5.073727e-01 480 | 0 0 4.499490e-01 481 | 0 0 4.499490e-01 482 | 0 0 3.815948e-01 483 | 0 0 3.746177e-01 484 | 0 0 5.073727e-01 485 | 0 0 4.329771e-01 486 | 0 0 4.725347e-01 487 | 0 0 5.073727e-01 488 | 0 0 3.046889e-01 489 | 0 0 3.124504e-01 490 | 0 0 4.499490e-01 491 | 0 0 4.725347e-01 492 | 0 0 3.746177e-01 493 | 0 0 4.499490e-01 494 | 0 0 3.624273e-01 495 | 0 0 5.073727e-01 496 | 0 1 1.161686e-01 497 | 0 0 4.329771e-01 498 | 0 0 4.499490e-01 499 | 0 0 4.329771e-01 500 | 0 0 1.154100e-01 501 | 0 0 4.725347e-01 502 | 1 1 5.073727e-01 503 | 1 0 1.104039e-01 504 | 1 0 2.820654e-01 505 | 1 1 9.445784e-02 506 | 1 1 8.643224e-03 507 | 1 0 1.387475e-01 508 | 1 1 1.275003e-01 509 | 1 0 6.410550e-02 510 | 1 1 4.725347e-01 511 | 1 1 5.073727e-01 512 | 1 1 5.073727e-01 513 | 1 1 2.382121e-01 514 | 0 0 8.087993e-02 515 | 0 0 4.329771e-01 516 | 0 0 5.073727e-01 517 | 0 0 4.329771e-01 518 | 0 0 1.149417e-01 519 | 0 0 5.073727e-01 520 | 0 0 2.347956e-01 521 | 0 0 4.193166e-01 522 | 0 0 4.725347e-01 523 | 0 0 3.815948e-01 524 | 0 0 4.725347e-01 525 | 0 0 4.725347e-01 526 | 0 0 4.329771e-01 527 | 0 0 4.329771e-01 528 | 0 0 4.499490e-01 529 | 0 0 4.725347e-01 530 | 0 0 5.073727e-01 531 | 0 0 5.073727e-01 532 | 0 0 4.329771e-01 533 | 0 0 4.499490e-01 534 | 0 0 4.499490e-01 535 | 0 0 2.977248e-01 536 | 0 0 4.499490e-01 537 | 0 0 5.073727e-01 538 | 0 0 3.181620e-01 539 | 0 0 4.499490e-01 540 | 1 1 2.446861e-01 541 | 1 1 2.955546e-01 542 | 1 1 5.073727e-01 543 | 1 1 3.124504e-01 544 | 0 0 5.073727e-01 545 | 1 1 5.130434e-02 546 | 0 0 4.329771e-01 547 | 1 1 5.073727e-01 548 | 0 1 1.241352e-01 549 | 0 0 4.193166e-01 550 | 0 0 4.725347e-01 551 | 0 0 4.499490e-01 552 | 0 0 4.329771e-01 553 | 0 0 4.499490e-01 554 | 0 0 4.499490e-01 555 | 0 0 5.073727e-01 556 | 0 0 4.725347e-01 557 | 0 0 1.878260e-01 558 | 0 0 4.078641e-01 559 | 0 0 4.078641e-01 560 | 0 0 5.073727e-01 561 | 0 0 4.193166e-01 562 | 0 0 5.073727e-01 563 | 0 0 4.193166e-01 564 | 0 0 2.666037e-01 565 | 0 0 5.073727e-01 566 | 0 0 4.078641e-01 567 | 0 0 4.725347e-01 568 | 0 0 3.349299e-01 569 | 0 0 5.073727e-01 570 | 0 0 5.073727e-01 571 | 0 0 4.329771e-01 572 | 0 0 4.078641e-01 573 | 0 0 4.725347e-01 574 | 0 0 5.073727e-01 575 | 0 0 3.429588e-01 576 | 0 0 4.193166e-01 577 | 0 0 4.329771e-01 578 | 0 0 4.725347e-01 579 | 0 0 4.725347e-01 580 | 0 0 4.499490e-01 581 | 0 0 5.073727e-01 582 | 0 0 4.329771e-01 583 | 0 0 4.499490e-01 584 | 0 0 3.624273e-01 585 | 0 0 4.725347e-01 586 | 0 0 3.570354e-01 587 | 0 0 4.499490e-01 588 | 0 0 5.073727e-01 589 | 0 0 3.682621e-01 590 | 0 0 4.193166e-01 591 | 0 0 4.499490e-01 592 | 0 0 4.725347e-01 593 | 0 0 1.705877e-01 594 | 0 0 3.746177e-01 595 | 0 0 2.977700e-02 596 | 0 0 4.499490e-01 597 | 0 0 4.193166e-01 598 | 0 0 4.078641e-01 599 | 0 0 4.329771e-01 600 | 0 0 4.499490e-01 601 | 0 0 3.624273e-01 602 | 0 0 4.725347e-01 603 | 0 0 4.499490e-01 604 | 0 0 4.725347e-01 605 | 0 0 4.329771e-01 606 | 0 0 4.193166e-01 607 | 0 0 1.733997e-01 608 | 0 0 4.725347e-01 609 | 0 0 4.499490e-01 610 | 0 0 4.078641e-01 611 | 0 0 4.725347e-01 612 | 0 0 4.499490e-01 613 | 0 0 4.725347e-01 614 | 0 0 5.224377e-02 615 | 0 0 4.329771e-01 616 | 0 0 4.725347e-01 617 | 0 0 4.329771e-01 618 | 0 0 4.499490e-01 619 | 0 0 5.073727e-01 620 | 0 0 7.705504e-02 621 | 0 0 4.725347e-01 622 | 0 0 4.193166e-01 623 | 0 0 1.395024e-01 624 | 0 0 3.277295e-01 625 | 0 0 5.073727e-01 626 | 0 0 4.725347e-01 627 | 0 0 4.725347e-01 628 | 0 0 4.329771e-01 629 | 0 0 3.181620e-01 630 | 0 0 4.193166e-01 631 | 0 0 4.725347e-01 632 | 0 0 4.329771e-01 633 | 0 0 5.073727e-01 634 | 0 0 4.499490e-01 635 | 0 0 4.725347e-01 636 | 0 0 1.785384e-01 637 | 0 0 4.329771e-01 638 | 0 0 4.329771e-01 639 | 0 0 3.022869e-01 640 | 0 0 4.329771e-01 641 | 0 0 4.725347e-01 642 | 0 0 3.979970e-01 643 | 0 0 3.893271e-01 644 | 0 0 3.893271e-01 645 | 0 0 1.047903e-01 646 | 0 0 4.499490e-01 647 | 0 0 3.429588e-01 648 | 0 0 3.097638e-01 649 | 0 0 4.078641e-01 650 | 0 0 3.746177e-01 651 | 0 0 4.499490e-01 652 | 0 0 4.725347e-01 653 | 0 0 1.053235e-01 654 | 0 0 7.303198e-02 655 | 0 0 4.725347e-01 656 | 0 0 4.725347e-01 657 | 0 0 4.725347e-01 658 | 0 0 1.456335e-01 659 | 0 0 4.725347e-01 660 | 0 0 4.499490e-01 661 | 0 0 3.349299e-01 662 | 0 0 4.725347e-01 663 | 0 0 4.499490e-01 664 | 0 0 4.499490e-01 665 | 0 0 4.725347e-01 666 | 0 0 5.073727e-01 667 | 0 0 4.725347e-01 668 | 0 0 4.193166e-01 669 | 0 0 4.725347e-01 670 | 0 0 4.725347e-01 671 | 0 0 4.725347e-01 672 | 0 0 4.725347e-01 673 | 0 0 4.499490e-01 674 | 0 0 4.329771e-01 675 | 0 0 4.499490e-01 676 | 0 0 3.893271e-01 677 | 0 0 3.746177e-01 678 | 0 0 4.329771e-01 679 | 0 0 5.073727e-01 680 | 0 0 4.329771e-01 681 | 0 0 4.725347e-01 682 | 0 0 4.725347e-01 683 | 0 0 4.329771e-01 684 | 0 0 4.193166e-01 685 | 0 0 4.499490e-01 686 | 0 0 4.725347e-01 687 | 0 0 4.725347e-01 688 | 0 0 4.193166e-01 689 | 0 0 4.499490e-01 690 | 0 0 5.073727e-01 691 | 0 0 5.073727e-01 692 | 0 0 4.329771e-01 693 | 0 0 5.073727e-01 694 | 0 0 3.893271e-01 695 | 0 0 5.073727e-01 696 | 0 0 5.073727e-01 697 | 0 0 5.073727e-01 698 | 0 0 3.520249e-01 699 | 0 0 1.949964e-01 700 | 0 0 3.893271e-01 701 | 0 0 4.193166e-01 702 | 0 0 4.725347e-01 703 | 0 0 4.329771e-01 704 | 0 0 3.124504e-01 705 | 0 0 4.193166e-01 706 | 0 0 3.388294e-01 707 | 0 0 4.499490e-01 708 | 0 0 4.193166e-01 709 | 0 0 4.499490e-01 710 | 0 0 4.725347e-01 711 | 0 0 5.073727e-01 712 | 0 0 4.725347e-01 713 | 0 0 4.499490e-01 714 | 0 0 4.499490e-01 715 | 0 0 4.193166e-01 716 | 0 0 5.073727e-01 717 | 0 0 3.893271e-01 718 | 0 0 5.073727e-01 719 | 0 0 4.499490e-01 720 | 0 0 5.073727e-01 721 | 0 0 5.073727e-01 722 | 0 0 4.725347e-01 723 | 0 0 4.725347e-01 724 | 0 0 4.725347e-01 725 | 0 0 5.073727e-01 726 | 0 0 4.725347e-01 727 | 0 0 3.746177e-01 728 | 0 0 4.725347e-01 729 | 0 0 4.499490e-01 730 | 0 0 4.078641e-01 731 | 0 0 4.725347e-01 732 | 0 0 4.329771e-01 733 | 0 0 4.078641e-01 734 | 0 0 5.073727e-01 735 | 0 0 5.073727e-01 736 | 0 0 4.725347e-01 737 | 0 0 3.277295e-01 738 | 0 0 3.520249e-01 739 | 0 0 3.815948e-01 740 | 0 0 5.073727e-01 741 | 0 0 4.078641e-01 742 | 0 0 4.329771e-01 743 | 0 0 4.329771e-01 744 | 0 0 3.388294e-01 745 | 0 0 3.979970e-01 746 | 0 0 5.073727e-01 747 | 0 0 1.494728e-02 748 | 0 0 3.682621e-01 749 | 0 0 5.073727e-01 750 | 0 0 4.499490e-01 751 | 0 0 4.499490e-01 752 | 0 0 3.815948e-01 753 | 0 0 4.725347e-01 754 | 0 0 4.499490e-01 755 | 0 0 5.073727e-01 756 | 0 0 1.190208e-01 757 | 0 0 9.841049e-02 758 | 0 0 4.725347e-01 759 | 0 0 4.499490e-01 760 | 0 0 4.499490e-01 761 | 0 0 4.725347e-01 762 | 0 0 4.725347e-01 763 | 0 0 4.499490e-01 764 | 0 0 4.499490e-01 765 | 0 0 5.073727e-01 766 | 0 0 4.725347e-01 767 | 0 0 5.073727e-01 768 | 0 0 4.499490e-01 769 | 0 0 4.329771e-01 770 | 0 0 4.725347e-01 771 | 0 0 5.073727e-01 772 | 0 0 4.725347e-01 773 | 0 0 4.725347e-01 774 | 0 0 3.815948e-01 775 | 0 0 4.499490e-01 776 | 0 0 5.073727e-01 777 | 0 0 4.725347e-01 778 | 0 0 3.243911e-01 779 | 0 0 4.499490e-01 780 | 0 0 5.073727e-01 781 | 0 0 3.429588e-01 782 | 0 0 3.072034e-02 783 | 0 0 4.725347e-01 784 | 0 0 4.329771e-01 785 | 0 0 4.499490e-01 786 | 0 0 4.329771e-01 787 | 0 0 3.212063e-01 788 | 0 0 4.499490e-01 789 | 0 0 3.893271e-01 790 | 0 0 5.073727e-01 791 | 0 0 4.499490e-01 792 | 0 0 4.329771e-01 793 | 0 0 4.193166e-01 794 | 0 0 5.073727e-01 795 | 0 0 4.725347e-01 796 | 0 0 1.030785e-01 797 | 0 0 5.073727e-01 798 | 0 0 3.277295e-01 799 | 0 0 5.073727e-01 800 | 0 0 2.977248e-01 801 | 0 0 4.193166e-01 802 | 0 0 4.193166e-01 803 | 0 0 5.073727e-01 804 | 0 0 4.725347e-01 805 | 0 0 3.815948e-01 806 | 0 0 4.725347e-01 807 | 0 0 4.499490e-01 808 | 0 0 4.329771e-01 809 | 0 0 4.193166e-01 810 | 0 0 4.499490e-01 811 | 0 0 4.499490e-01 812 | 0 0 4.193166e-01 813 | 0 0 9.495010e-02 814 | 0 0 3.349299e-01 815 | 0 0 4.725347e-01 816 | 0 0 4.725347e-01 817 | 0 0 5.073727e-01 818 | 0 0 4.912262e-02 819 | 0 0 4.725347e-01 820 | 0 0 4.499490e-01 821 | 0 0 1.298342e-01 822 | 0 0 3.624273e-01 823 | 0 0 4.725347e-01 824 | 0 0 9.814551e-02 825 | 0 0 3.429588e-01 826 | 0 0 3.815948e-01 827 | 0 0 4.329771e-01 828 | 0 0 1.714162e-01 829 | 0 0 4.725347e-01 830 | 0 0 4.725347e-01 831 | 0 0 4.329771e-01 832 | 0 0 4.725347e-01 833 | 0 0 5.073727e-01 834 | 0 0 4.499490e-01 835 | 0 0 4.499490e-01 836 | 0 0 5.073727e-01 837 | 0 0 4.193166e-01 838 | 0 0 3.312367e-01 839 | 0 0 5.073727e-01 840 | 0 0 4.499490e-01 841 | 0 1 5.073727e-01 842 | 0 0 4.329771e-01 843 | 0 0 4.499490e-01 844 | 0 0 4.329771e-01 845 | 0 0 3.979970e-01 846 | 0 0 1.086511e-01 847 | 0 0 4.329771e-01 848 | 0 0 4.329771e-01 849 | 0 0 5.073727e-01 850 | 0 0 4.329771e-01 851 | 0 0 5.073727e-01 852 | 0 0 4.193166e-01 853 | 0 0 4.193166e-01 854 | 0 0 4.329771e-01 855 | 0 0 4.193166e-01 856 | 0 0 4.499490e-01 857 | 0 0 5.073727e-01 858 | 0 0 4.725347e-01 859 | 0 0 4.725347e-01 860 | 0 0 5.073727e-01 861 | 0 0 3.815948e-01 862 | 0 0 1.355524e-01 863 | 0 0 3.520249e-01 864 | 0 0 5.073727e-01 865 | 0 0 3.893271e-01 866 | 0 0 6.762196e-02 867 | 0 0 5.073727e-01 868 | 0 0 4.725347e-01 869 | 0 0 3.124504e-01 870 | 0 0 4.499490e-01 871 | 0 0 4.725347e-01 872 | 0 0 3.682621e-01 873 | 0 0 4.725347e-01 874 | 0 0 5.073727e-01 875 | 0 0 4.329771e-01 876 | 0 0 4.193166e-01 877 | 0 0 5.073727e-01 878 | 0 0 4.725347e-01 879 | 0 0 4.725347e-01 880 | 0 0 3.570354e-01 881 | 0 0 1.703143e-01 882 | 0 0 1.285923e-01 883 | 0 0 4.499490e-01 884 | 0 0 5.073727e-01 885 | 0 0 3.388294e-01 886 | 0 0 3.520249e-01 887 | 0 0 4.725347e-01 888 | 0 0 3.893271e-01 889 | 0 0 4.329771e-01 890 | 0 0 4.499490e-01 891 | 0 0 4.725347e-01 892 | 0 0 4.499490e-01 893 | 0 0 4.725347e-01 894 | 0 0 4.725347e-01 895 | 0 0 3.979970e-01 896 | 0 0 4.193166e-01 897 | 0 0 4.725347e-01 898 | 0 0 4.078641e-01 899 | 0 0 5.073727e-01 900 | 0 0 8.548660e-02 901 | 0 0 4.193166e-01 902 | 0 0 4.499490e-01 903 | 0 0 3.624273e-01 904 | 0 0 4.499490e-01 905 | 0 0 3.682621e-01 906 | 0 0 4.329771e-01 907 | 0 0 4.725347e-01 908 | 0 0 4.078641e-01 909 | 0 0 3.979970e-01 910 | 0 0 5.073727e-01 911 | 0 0 1.355524e-01 912 | 0 0 3.349299e-01 913 | 0 0 4.078641e-01 914 | 0 0 2.541350e-01 915 | 0 0 3.815948e-01 916 | 0 0 4.499490e-01 917 | 0 0 4.499490e-01 918 | 0 0 4.499490e-01 919 | 0 0 4.725347e-01 920 | 0 0 4.329771e-01 921 | 0 0 5.073727e-01 922 | 0 0 5.073727e-01 923 | 0 0 4.078641e-01 924 | 0 0 4.499490e-01 925 | 0 0 4.499490e-01 926 | 0 0 4.329771e-01 927 | 0 0 3.746177e-01 928 | 0 0 5.073727e-01 929 | 0 0 5.073727e-01 930 | 0 0 4.499490e-01 931 | 0 0 4.725347e-01 932 | 0 0 4.193166e-01 933 | 0 0 3.570354e-01 934 | 0 0 4.329771e-01 935 | 0 0 3.152468e-01 936 | 0 0 4.725347e-01 937 | 0 0 4.329771e-01 938 | 0 0 4.499490e-01 939 | 0 0 4.329771e-01 940 | 0 0 4.725347e-01 941 | 0 0 1.010027e-01 942 | 0 0 1.785384e-01 943 | 0 0 5.073727e-01 944 | 0 0 5.073727e-01 945 | 0 0 4.725347e-01 946 | 0 0 4.329771e-01 947 | 0 0 5.073727e-01 948 | 0 0 4.725347e-01 949 | 0 0 4.193166e-01 950 | 0 0 4.193166e-01 951 | 0 0 3.181620e-01 952 | 0 0 4.499490e-01 953 | 0 0 5.073727e-01 954 | 0 0 4.725347e-01 955 | 0 0 4.725347e-01 956 | 0 0 5.166777e-02 957 | 0 0 4.725347e-01 958 | 0 0 3.979970e-01 959 | 0 0 4.499490e-01 960 | 0 0 2.694163e-01 961 | 0 0 4.725347e-01 962 | 0 0 4.725347e-01 963 | 0 0 5.073727e-01 964 | 0 0 3.243911e-01 965 | 0 0 4.725347e-01 966 | 0 0 3.815948e-01 967 | 0 0 4.329771e-01 968 | 0 0 4.329771e-01 969 | 0 0 5.073727e-01 970 | 0 0 4.840567e-02 971 | 0 0 4.725347e-01 972 | 0 0 3.388294e-01 973 | 0 0 4.725347e-01 974 | 0 0 4.725347e-01 975 | 0 0 4.193166e-01 976 | 0 0 4.193166e-01 977 | 0 0 4.499490e-01 978 | 0 0 4.725347e-01 979 | 0 0 4.078641e-01 980 | 0 0 4.329771e-01 981 | 0 0 3.893271e-01 982 | 0 0 3.979970e-01 983 | 0 0 3.473461e-01 984 | 0 0 5.073727e-01 985 | 0 0 3.979970e-01 986 | 0 0 4.329771e-01 987 | 0 0 4.725347e-01 988 | 0 0 4.499490e-01 989 | 0 0 4.078641e-01 990 | 0 0 4.329771e-01 991 | 0 0 4.725347e-01 992 | 0 0 4.193166e-01 993 | 0 0 5.073727e-01 994 | 0 1 1.865963e-02 995 | 0 0 4.078641e-01 996 | 0 0 4.725347e-01 997 | 0 0 4.329771e-01 998 | 0 0 3.212063e-01 999 | 0 0 4.329771e-01 1000 | 0 0 4.193166e-01 1001 | 0 0 4.725347e-01 1002 | -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/certify/certified_accuracy_robustness_radius_curve.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/certify/certified_accuracy_robustness_radius_curve.png -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/certify/checkpoint-distribution-transformer: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/certify/checkpoint-distribution-transformer -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/certify/t: -------------------------------------------------------------------------------- 1 | 4.679687e-01 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-0: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-0 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-1: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-1 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-10: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-10 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-11: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-11 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-12: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-12 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-13: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-13 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-14: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-14 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-15: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-15 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-16: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-16 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-17: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-17 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-18: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-18 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-19: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-19 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-2 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-20: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-20 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-21: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-21 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-22: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-22 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-23: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-23 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-24: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-24 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-25: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-25 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-3 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-4 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-5 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-6: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-6 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-7: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-7 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-8: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-8 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-ae-9: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-ae-9 -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-fm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-fm -------------------------------------------------------------------------------- /smoothed_kitsune/save/mirai/train/checkpoint-norm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/KaiWangGitHub/BARS/06a5bd663d14403a241900636ad53fc1ca324783/smoothed_kitsune/save/mirai/train/checkpoint-norm -------------------------------------------------------------------------------- /smoothed_kitsune/utils.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | from sklearn.metrics import precision_score, recall_score, f1_score 4 | 5 | import torch 6 | 7 | def load_data(data_dir: str, data_type: str="train"): 8 | 9 | if data_type == "cluster": 10 | x_cluster = np.load(os.path.join(data_dir, "X_cluster.npy")) 11 | 12 | return torch.tensor(x_cluster, dtype=torch.float) 13 | 14 | elif data_type == "train": 15 | x_train = np.load(os.path.join(data_dir, "X_train.npy")) 16 | 17 | return torch.tensor(x_train, dtype=torch.float) 18 | 19 | elif data_type == "test": 20 | x_test = np.load(os.path.join(data_dir, "X_test.npy")) 21 | y_test = np.load(os.path.join(data_dir, "y_test.npy")) 22 | 23 | return torch.tensor(x_test, dtype=torch.float), torch.tensor(y_test, dtype=torch.long) 24 | 25 | else: 26 | raise NotImplementedError() 27 | 28 | class AverageMeter(object): 29 | def __init__(self): 30 | self.reset() 31 | 32 | def reset(self): 33 | self.val = 0 34 | self.sum = 0 35 | self.count = 0 36 | self.avg = 0 37 | 38 | def update(self, val: float): 39 | self.val = val 40 | self.sum += val 41 | self.count += 1 42 | self.avg = self.sum / self.count 43 | 44 | def calc_metrics_classifier(label: np.ndarray, pred: np.ndarray) -> (float, float, float, float): 45 | assert label.shape == pred.shape 46 | acc = np.array((label == pred), dtype=float).mean() 47 | p = precision_score(label, pred) 48 | r = recall_score(label, pred) 49 | f1 = f1_score(label, pred) 50 | 51 | return acc, p, r, f1 52 | 53 | def calc_metrics_certify(label: np.ndarray, pred: np.ndarray, radius_feat: np.ndarray) -> np.ndarray: 54 | assert label.shape == pred.shape 55 | assert pred.shape == radius_feat.shape 56 | 57 | return np.mean(np.array((label == pred), dtype=float) * radius_feat) --------------------------------------------------------------------------------