├── .gitignore ├── CENSUS_DATA_README.ipynb ├── LICENSE ├── README.md ├── __init__.py ├── core ├── __init__.py ├── attack.py ├── classifier.py ├── data_util.py ├── privacy_accountant.py └── utilities.py ├── dataset ├── census.zip ├── cifar_100_features.p └── cifar_100_labels.p ├── evaluating_dpml ├── README.md ├── __init__.py ├── interpret_results.py ├── main.py └── run_experiments.sh ├── extra ├── __init__.py ├── combine_traces.py ├── crawl_census_data.py └── preprocess_dataset.py ├── improved_ai ├── README.md ├── __init__.py ├── interpret_results.py ├── main.py ├── run_experiments.sh └── run_experiments_slurm.sh ├── improved_mi ├── README.md ├── __init__.py ├── interpret_results.py ├── main.py ├── run_experiments.sh └── test_benchmark_output.txt └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | model/ 2 | results/ 3 | data/ 4 | dataset/ 5 | log/ 6 | ENV/ 7 | env/ 8 | venv/ 9 | __pycache__/ 10 | __temp_files/ 11 | traceEvents.json 12 | slurm-*.out 13 | joblog.txt 14 | *.pyc 15 | .DS_Store 16 | *.tmp 17 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 Bargav Jayaraman 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Analyzing the Leaky Cauldron 2 | 3 | The goal of this project is to evaluate the privacy leakage of differential private machine learning algorithms. 4 | 5 | The code has been adapted from the [code base](https://github.com/csong27/membership-inference) of membership inference attack work by [Shokri et al.](https://ieeexplore.ieee.org/document/7958568) 6 | 7 | Below we describe the setup and installation instructions. To run the experiments for the following projects, refer to their respective README files (hyperlinked): 8 | * [Evaluating Differentially Private Machine Learning in Practice](evaluating_dpml/README.md) (`evaluating_dpml\`) 9 | * [Revisiting Membership Inference Under Realistic Assumptions](improved_mi/README.md) (`improved_mi\`) 10 | * [Are Attribute Inference Attacks Just Imputation?](improved_ai/README.md) (`improved_ai\`) 11 | 12 | 13 | ### Software Requirements 14 | 15 | - [Python 3.8](https://www.anaconda.com/distribution/) 16 | - [Tensorflow](https://www.tensorflow.org/install) : To use Tensorflow with GPU, cuda-toolkit-11 and cudnn-8 are also [required](https://www.tensorflow.org/install/gpu). 17 | - [Tensorflow Privacy](https://github.com/tensorflow/privacy) 18 | 19 | 20 | ### Installation Instructions 21 | 22 | Assuming the system has Ubuntu 18.04 OS. The easiest way to get Python 3.8 is to install [Anaconda 3](https://www.anaconda.com/distribution/) followed by installing the dependencies via pip. The following bash code installs the dependencies (including `scikit_learn`, `tensorflow>=2.4.0` and `tf-privacy`) in a virtual environment: 23 | 24 | ``` 25 | $ python3 -m venv env 26 | $ source env/bin/activate 27 | $ python3 -m pip install --upgrade pip 28 | $ python3 -m pip install --no-cache-dir -r requirements.txt 29 | ``` 30 | 31 | Furthermore, to use cuda-compatible nvidia gpus, the following script should be executed (copied from [Tensorflow website](https://www.tensorflow.org/install/gpu)) to install cuda-toolkit-11 and cudnn-8 as required by tensorflow-gpu: 32 | 33 | ``` 34 | # Add NVIDIA package repositories 35 | $ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin 36 | $ sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 37 | $ sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub 38 | $ sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /" 39 | $ sudo apt-get update 40 | 41 | $ wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb 42 | 43 | $ sudo apt install ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb 44 | $ sudo apt-get update 45 | 46 | $ wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/libnvinfer7_7.1.3-1+cuda11.0_amd64.deb 47 | $ sudo apt install ./libnvinfer7_7.1.3-1+cuda11.0_amd64.deb 48 | $ sudo apt-get update 49 | 50 | # Install development and runtime libraries (~4GB) 51 | $ sudo apt-get install --no-install-recommends \ 52 | cuda-11-0 \ 53 | libcudnn8=8.0.4.30-1+cuda11.0 \ 54 | libcudnn8-dev=8.0.4.30-1+cuda11.0 55 | 56 | # Reboot. Check that GPUs are visible using the command: nvidia-smi 57 | 58 | # Install TensorRT. Requires that libcudnn8 is installed above. 59 | $ sudo apt-get install -y --no-install-recommends libnvinfer7=7.1.3-1+cuda11.0 \ 60 | libnvinfer-dev=7.1.3-1+cuda11.0 \ 61 | libnvinfer-plugin7=7.1.3-1+cuda11.0 62 | ``` 63 | 64 | 65 | ### Obtaining the Data Sets 66 | 67 | Data sets can be obtained using the `preprocess_dataset.py` script provided in the `extra/` folder. The script requires raw files for the respective data sets which can be found online using the following links: 68 | 69 | - **Purchase-100X**: The source file `transactions.csv` can be downloaded from https://www.kaggle.com/c/acquire-valued-shoppers-challenge/data and should be saved in the `dataset/` folder. 70 | - **Census19**: The source files can be downloaded from https://www2.census.gov/programs-surveys/acs/data/pums/2019/1-Year/ and should be saved in the `dataset/census/` folder. Alternatively, the source files can be obtained by running the `crawl_census_data.py` script in the `extra/` folder: 71 | >> `$ python3 crawl_census_data.py` 72 | - **Texas-100X**: `PUDF_base1q2006_tab.txt`, `PUDF_base2q2006_tab.txt`, `PUDF_base3q2006_tab.txt` and `PUDF_base4q2006_tab.txt` files can be downloaded from https://www.dshs.texas.gov/THCIC/Hospitals/Download.shtm and should be saved in the `dataset/texas_100_v2/` folder. 73 | 74 | Once the source files for the respective data set are obtained, `preprocess_dataset.py` script would be able to generate the processed data set files, which are in the form of two pickle files: `$DATASET`_feature.p and `$DATASET`_labels.p (where `$DATASET` is a placeholder for the data set file name). For Purchase-100X, `$DATASET = purchase_100`. For Texas-100X, `$DATASET = texas_100_v2`. For Census19, `$DATASET = census`. 75 | ``` 76 | $ python3 preprocess_dataset.py $DATASET --preprocess=1 77 | ``` 78 | 79 | Alternatively, Census19 data set (as is used in the attribute inference paper) can also be found in the `dataset/` folder in zip format. 80 | 81 | For pre-processing other data sets, bound the L2 norm of each record to 1 and pickle the features and labels separately into `$DATASET`_feature.p and `$DATASET`_labels.p files in the `dataset/` folder (where `$DATASET` is a placeholder for the data set file name, e.g. for Purchase-100 data set, `$DATASET` will be `purchase_100`). 82 | -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bargavj/EvaluatingDPML/2f611fd20b4089019223039d95efa70a49c2b320/__init__.py -------------------------------------------------------------------------------- /core/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bargavj/EvaluatingDPML/2f611fd20b4089019223039d95efa70a49c2b320/core/__init__.py -------------------------------------------------------------------------------- /core/attack.py: -------------------------------------------------------------------------------- 1 | import os 2 | import pickle 3 | import numpy as np 4 | import tensorflow.compat.v1 as tf 5 | 6 | from core.classifier import train as train_model 7 | from core.classifier import get_predictions 8 | from core.utilities import log_loss 9 | from core.utilities import pretty_print_result 10 | from core.utilities import get_inference_threshold 11 | from core.utilities import generate_noise 12 | from core.utilities import get_attribute_variations 13 | from core.utilities import plot_layer_outputs 14 | from core.data_util import load_attack_data 15 | from core.data_util import save_data 16 | from core.data_util import load_data 17 | from scipy.stats import norm 18 | from sklearn.metrics import roc_curve 19 | from sklearn.preprocessing import QuantileTransformer 20 | 21 | MODEL_PATH = 'model/' 22 | 23 | if not os.path.exists(MODEL_PATH): 24 | os.makedirs(MODEL_PATH) 25 | 26 | 27 | def train_target_model(args, dataset=None, epochs=100, batch_size=100, learning_rate=0.01, clipping_threshold=1, l2_ratio=1e-7, n_hidden=50, n_out=None, model='nn', privacy='no_privacy', dp='dp', epsilon=0.5, delta=1e-5, save=True): 28 | """ 29 | Wrapper function that trains the target model over the sensitive data. 30 | """ 31 | if dataset == None: 32 | dataset = load_data('target_data.npz', args) 33 | train_x, train_y, test_x, test_y = dataset 34 | 35 | classifier, aux = train_model( 36 | dataset, 37 | n_out=n_out, 38 | n_hidden=n_hidden, 39 | epochs=epochs, 40 | learning_rate=learning_rate, 41 | clipping_threshold=clipping_threshold, 42 | batch_size=batch_size, 43 | model=model, 44 | l2_ratio=l2_ratio, 45 | silent=False, 46 | privacy=privacy, 47 | dp=dp, 48 | epsilon=epsilon, 49 | delta=delta) 50 | # test data for attack model 51 | attack_x, attack_y = [], [] 52 | 53 | # data used in training, label is 1 54 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 55 | x={'x': train_x}, 56 | num_epochs=1, 57 | shuffle=False) 58 | 59 | predictions = classifier.predict(input_fn=pred_input_fn) 60 | _, pred_scores = get_predictions(predictions) 61 | 62 | attack_x.append(pred_scores) 63 | attack_y.append(np.ones(train_x.shape[0])) 64 | 65 | # data not used in training, label is 0 66 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 67 | x={'x': test_x}, 68 | num_epochs=1, 69 | shuffle=False) 70 | 71 | predictions = classifier.predict(input_fn=pred_input_fn) 72 | _, pred_scores = get_predictions(predictions) 73 | 74 | attack_x.append(pred_scores) 75 | attack_y.append(np.zeros(test_x.shape[0])) 76 | 77 | attack_x = np.vstack(attack_x) 78 | attack_y = np.concatenate(attack_y) 79 | attack_x = attack_x.astype('float32') 80 | attack_y = attack_y.astype('int32') 81 | 82 | if save: 83 | np.savez(MODEL_PATH + 'attack_test_data.npz', attack_x, attack_y) 84 | 85 | classes = np.concatenate([train_y, test_y]) 86 | return attack_x, attack_y, classes, classifier, aux 87 | 88 | 89 | def train_shadow_models(args, n_hidden=50, epochs=100, n_shadow=20, learning_rate=0.05, batch_size=100, l2_ratio=1e-7, model='nn', privacy='no_privacy', dp='dp', epsilon=0.5, delta=1e-5, save=True): 90 | """ 91 | Wrapper function to train the shadow models similar to the target model. 92 | Shadow model training is peformed over the hold-out data. 93 | """ 94 | attack_x, attack_y = [], [] 95 | classes = [] 96 | for i in range(n_shadow): 97 | #print('Training shadow model {}'.format(i)) 98 | dataset = load_data('shadow{}_data.npz'.format(i), args) 99 | train_x, train_y, test_x, test_y = dataset 100 | 101 | # train model 102 | classifier = train_model( 103 | dataset, 104 | n_hidden=n_hidden, 105 | epochs=epochs, 106 | learning_rate=learning_rate, 107 | batch_size=batch_size, 108 | model=model, 109 | l2_ratio=l2_ratio, 110 | privacy=privacy, 111 | dp=dp, 112 | epsilon=epsilon, 113 | delta=delta) 114 | #print('Gather training data for attack model') 115 | attack_i_x, attack_i_y = [], [] 116 | 117 | # data used in training, label is 1 118 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 119 | x={'x': train_x}, 120 | num_epochs=1, 121 | shuffle=False) 122 | 123 | predictions = classifier.predict(input_fn=pred_input_fn) 124 | _, pred_scores = get_predictions(predictions) 125 | 126 | attack_i_x.append(pred_scores) 127 | attack_i_y.append(np.ones(train_x.shape[0])) 128 | 129 | # data not used in training, label is 0 130 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 131 | x={'x': test_x}, 132 | num_epochs=1, 133 | shuffle=False) 134 | 135 | predictions = classifier.predict(input_fn=pred_input_fn) 136 | _, pred_scores = get_predictions(predictions) 137 | 138 | attack_i_x.append(pred_scores) 139 | attack_i_y.append(np.zeros(test_x.shape[0])) 140 | 141 | attack_x += attack_i_x 142 | attack_y += attack_i_y 143 | classes.append(np.concatenate([train_y, test_y])) 144 | # train data for attack model 145 | attack_x = np.vstack(attack_x) 146 | attack_y = np.concatenate(attack_y) 147 | attack_x = attack_x.astype('float32') 148 | attack_y = attack_y.astype('int32') 149 | classes = np.concatenate(classes) 150 | 151 | if save: 152 | np.savez(MODEL_PATH + 'attack_train_data.npz', attack_x, attack_y) 153 | 154 | return attack_x, attack_y, classes 155 | 156 | 157 | def train_attack_model(classes, dataset=None, n_hidden=50, learning_rate=0.01, batch_size=200, epochs=50, model='nn', l2_ratio=1e-7): 158 | """ 159 | Wrapper function to train the meta-model over the shadow models' output. 160 | During inference time, the meta-model takes the target model's output and 161 | predicts if a query record is part of the target model's training set. 162 | """ 163 | if dataset is None: 164 | dataset = load_attack_data(MODEL_PATH) 165 | train_x, train_y, test_x, test_y = dataset 166 | 167 | train_classes, test_classes = classes 168 | train_indices = np.arange(len(train_x)) 169 | test_indices = np.arange(len(test_x)) 170 | unique_classes = np.unique(train_classes) 171 | 172 | pred_y = [] 173 | shadow_membership, target_membership = [], [] 174 | shadow_pred_scores, target_pred_scores = [], [] 175 | shadow_class_labels, target_class_labels = [], [] 176 | for c in unique_classes: 177 | #print('Training attack model for class {}...'.format(c)) 178 | c_train_indices = train_indices[train_classes == c] 179 | c_train_x, c_train_y = train_x[c_train_indices], train_y[c_train_indices] 180 | c_test_indices = test_indices[test_classes == c] 181 | c_test_x, c_test_y = test_x[c_test_indices], test_y[c_test_indices] 182 | c_dataset = (c_train_x, c_train_y, c_test_x, c_test_y) 183 | classifier = train_model( 184 | c_dataset, 185 | n_hidden=n_hidden, 186 | epochs=epochs, 187 | learning_rate=learning_rate, 188 | batch_size=batch_size, 189 | model=model, 190 | l2_ratio=l2_ratio) 191 | 192 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 193 | x={'x': c_train_x}, 194 | num_epochs=1, 195 | shuffle=False) 196 | predictions = classifier.predict(input_fn=pred_input_fn) 197 | c_pred_y, c_pred_scores = get_predictions(predictions) 198 | shadow_membership.append(c_train_y) 199 | shadow_pred_scores.append(c_pred_scores) 200 | shadow_class_labels.append([c]*len(c_train_indices)) 201 | 202 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 203 | x={'x': c_test_x}, 204 | num_epochs=1, 205 | shuffle=False) 206 | predictions = classifier.predict(input_fn=pred_input_fn) 207 | c_pred_y, c_pred_scores = get_predictions(predictions) 208 | pred_y.append(c_pred_y) 209 | target_membership.append(c_test_y) 210 | target_pred_scores.append(c_pred_scores) 211 | target_class_labels.append([c]*len(c_test_indices)) 212 | 213 | print('-' * 10 + 'FINAL EVALUATION' + '-' * 10 + '\n') 214 | pred_y = np.concatenate(pred_y) 215 | shadow_membership = np.concatenate(shadow_membership) 216 | target_membership = np.concatenate(target_membership) 217 | shadow_pred_scores = np.concatenate(shadow_pred_scores) 218 | target_pred_scores = np.concatenate(target_pred_scores) 219 | shadow_class_labels = np.concatenate(shadow_class_labels) 220 | target_class_labels = np.concatenate(target_class_labels) 221 | prety_print_result(target_membership, pred_y) 222 | fpr, tpr, thresholds = roc_curve(target_membership, pred_y, pos_label=1) 223 | attack_adv = tpr[1] - fpr[1] 224 | return (attack_adv, shadow_pred_scores, target_pred_scores, shadow_membership, target_membership, shadow_class_labels, target_class_labels) 225 | 226 | 227 | def shokri_membership_inference(args, attack_test_x, attack_test_y, test_classes): 228 | """ 229 | Wrapper function for Shokri et al. membership inference attack 230 | that trains the shadow models and the meta-model. 231 | """ 232 | print('-' * 10 + 'SHOKRI\'S MEMBERSHIP INFERENCE' + '-' * 10 + '\n') 233 | print('-' * 10 + 'TRAIN SHADOW' + '-' * 10 + '\n') 234 | attack_train_x, attack_train_y, train_classes = train_shadow_models( 235 | args=args, 236 | epochs=args.target_epochs, 237 | batch_size=args.target_batch_size, 238 | learning_rate=args.target_learning_rate, 239 | n_shadow=args.n_shadow, 240 | n_hidden=args.target_n_hidden, 241 | l2_ratio=args.target_l2_ratio, 242 | model=args.target_model, 243 | save=args.save_model) 244 | 245 | print('-' * 10 + 'TRAIN ATTACK' + '-' * 10 + '\n') 246 | dataset = (attack_train_x, attack_train_y, attack_test_x, attack_test_y) 247 | return train_attack_model( 248 | dataset=dataset, 249 | epochs=args.attack_epochs, 250 | batch_size=args.attack_batch_size, 251 | learning_rate=args.attack_learning_rate, 252 | n_hidden=args.attack_n_hidden, 253 | l2_ratio=args.attack_l2_ratio, 254 | model=args.attack_model, 255 | classes=(train_classes, test_classes)) 256 | 257 | 258 | def yeom_membership_inference(per_instance_loss, membership, train_loss, test_loss=None): 259 | """ 260 | Yeom et al. membership inference attack that uses the 261 | per-instance loss to predict the record membership. 262 | """ 263 | print('-' * 10 + 'YEOM\'S MEMBERSHIP INFERENCE' + '-' * 10 + '\n') 264 | if test_loss == None: 265 | pred_membership = np.where(per_instance_loss <= train_loss, 1, 0) 266 | else: 267 | pred_membership = np.where(norm(0, train_loss).pdf(per_instance_loss) >= norm(0, test_loss).pdf(per_instance_loss), 1, 0) 268 | #pretty_print_result(membership, pred_membership) 269 | return pred_membership 270 | 271 | 272 | def proposed_membership_inference(v_dataset, true_x, true_y, classifier, per_instance_loss, args): 273 | """ 274 | Our proposed membership inference attacks that use threshold-selection 275 | procedure for Yeom and Merlin attacks. The function returns the 276 | per-instance loss and merlin-ratio over target and hold-out sets. 277 | """ 278 | print('-' * 10 + 'PROPOSED MEMBERSHIP INFERENCE' + '-' * 10 + '\n') 279 | v_train_x, v_train_y, v_test_x, v_test_y = v_dataset 280 | v_true_x = np.vstack([v_train_x, v_test_x]) 281 | v_true_y = np.concatenate([v_train_y, v_test_y]) 282 | v_pred_y, v_membership, v_test_classes, v_classifier, aux = train_target_model( 283 | args=args, 284 | dataset=v_dataset, 285 | epochs=args.target_epochs, 286 | batch_size=args.target_batch_size, 287 | learning_rate=args.target_learning_rate, 288 | clipping_threshold=args.target_clipping_threshold, 289 | n_hidden=args.target_n_hidden, 290 | l2_ratio=args.target_l2_ratio, 291 | model=args.target_model, 292 | privacy=args.target_privacy, 293 | dp=args.target_dp, 294 | epsilon=args.target_epsilon, 295 | delta=args.target_delta, 296 | save=args.save_model) 297 | v_per_instance_loss = np.array(log_loss(v_true_y, v_pred_y)) 298 | noise_params = (args.attack_noise_type, args.attack_noise_coverage, args.attack_noise_magnitude) 299 | v_merlin_ratio = get_merlin_ratio(v_true_x, v_true_y, v_classifier, v_per_instance_loss, noise_params) 300 | merlin_ratio = get_merlin_ratio(true_x, true_y, classifier, per_instance_loss, noise_params) 301 | return (true_y, v_true_y, v_membership, v_per_instance_loss, v_merlin_ratio, merlin_ratio) 302 | 303 | 304 | def evaluate_proposed_membership_inference(per_instance_loss, membership, proposed_mi_outputs, fpr_threshold=None, per_class_thresh=False): 305 | """ 306 | Evaluates the Yeom and Merlin attacks for a given FPR threshold. 307 | """ 308 | true_y, v_true_y, v_membership, v_per_instance_loss, v_merlin_ratio, merlin_ratio = proposed_mi_outputs 309 | print('-' * 10 + 'Using Yeom\'s MI with custom threshold' + '-' * 10 + '\n') 310 | if per_class_thresh: 311 | classes = np.unique(true_y) 312 | pred_membership = np.zeros(len(membership)) 313 | for c in classes: 314 | c_indices = np.arange(len(true_y))[true_y == c] 315 | v_c_indices = np.arange(len(v_true_y))[v_true_y == c] 316 | thresh = get_inference_threshold(-v_per_instance_loss[v_c_indices], v_membership[v_c_indices], fpr_threshold) 317 | pred_membership[c_indices] = np.where(per_instance_loss[c_indices] <= -thresh, 1, 0) 318 | else: 319 | thresh = get_inference_threshold(-v_per_instance_loss, v_membership, fpr_threshold) 320 | pred_membership = np.where(per_instance_loss <= -thresh, 1, 0) 321 | pretty_print_result(membership, pred_membership) 322 | 323 | print('-' * 10 + 'Using Merlin with custom threshold' + '-' * 10 + '\n') 324 | if per_class_thresh: 325 | classes = np.unique(true_y) 326 | pred_membership = np.zeros(len(membership)) 327 | for c in classes: 328 | c_indices = np.arange(len(true_y))[true_y == c] 329 | v_c_indices = np.arange(len(v_true_y))[v_true_y == c] 330 | thresh = get_inference_threshold(v_merlin_ratio[v_c_indices], v_membership[v_c_indices], fpr_threshold) 331 | pred_membership[c_indices] = np.where(merlin_ratio[c_indices] >= thresh, 1, 0) 332 | else: 333 | thresh = get_inference_threshold(v_merlin_ratio, v_membership, fpr_threshold) 334 | pred_membership = np.where(merlin_ratio >= thresh, 1, 0) 335 | pretty_print_result(membership, pred_membership) 336 | 337 | 338 | def get_merlin_ratio(true_x, true_y, classifier, per_instance_loss, noise_params, max_t=100): 339 | """ 340 | Returns the merlin-ratio for the Merlin attack, the merlin-ratio 341 | is between 0 and 1. 342 | """ 343 | counts = np.zeros(len(true_x)) 344 | for t in range(max_t): 345 | noisy_x = true_x + generate_noise(true_x.shape, true_x.dtype, noise_params) 346 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 347 | x={'x': noisy_x}, 348 | num_epochs=1, 349 | shuffle=False) 350 | predictions = classifier.predict(input_fn=pred_input_fn) 351 | _, pred_y = get_predictions(predictions) 352 | noisy_per_instance_loss = np.array(log_loss(true_y, pred_y)) 353 | counts += np.where(noisy_per_instance_loss > per_instance_loss, 1, 0) 354 | return counts / max_t 355 | 356 | 357 | def yeom_attribute_inference(true_x, true_y, classifier, membership, features, train_loss, test_loss=None): 358 | """ 359 | Yeom et al.'s attribute inference attack for binary attributes. 360 | """ 361 | print('-' * 10 + 'YEOM\'S ATTRIBUTE INFERENCE' + '-' * 10 + '\n') 362 | pred_membership_all = [] 363 | for feature in features: 364 | orignial_attribute = np.copy(true_x[:,feature]) 365 | low_value, high_value, true_attribute_value = get_attribute_variations(true_x, feature) 366 | 367 | true_x[:,feature] = low_value 368 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 369 | x={'x': true_x}, 370 | num_epochs=1, 371 | shuffle=False) 372 | predictions = classifier.predict(input_fn=pred_input_fn) 373 | _, low_op = get_predictions(predictions) 374 | low_op = low_op.astype('float32') 375 | low_op = log_loss(true_y, low_op) 376 | 377 | true_x[:,feature] = high_value 378 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 379 | x={'x': true_x}, 380 | num_epochs=1, 381 | shuffle=False) 382 | predictions = classifier.predict(input_fn=pred_input_fn) 383 | _, high_op = get_predictions(predictions) 384 | high_op = high_op.astype('float32') 385 | high_op = log_loss(true_y, high_op) 386 | 387 | high_prob = np.sum(true_attribute_value) / len(true_attribute_value) 388 | low_prob = 1 - high_prob 389 | 390 | if test_loss == None: 391 | pred_attribute_value = np.where(low_prob * norm(0, train_loss).pdf(low_op) >= high_prob * norm(0, train_loss).pdf(high_op), 0, 1) 392 | mask = [1]*len(pred_attribute_value) 393 | else: 394 | low_mem = np.where(norm(0, train_loss).pdf(low_op) >= norm(0, test_loss).pdf(low_op), 1, 0) 395 | high_mem = np.where(norm(0, train_loss).pdf(high_op) >= norm(0, test_loss).pdf(high_op), 1, 0) 396 | pred_attribute_value = [np.argmax([low_prob * a, high_prob * b]) for a, b in zip(low_mem, high_mem)] 397 | mask = [a | b for a, b in zip(low_mem, high_mem)] 398 | 399 | pred_membership = mask & (pred_attribute_value ^ true_attribute_value ^ [1]*len(pred_attribute_value)) 400 | pretty_print_result(membership, pred_membership) 401 | pred_membership_all.append(pred_membership) 402 | true_x[:,feature] = orignial_attribute 403 | return pred_membership_all 404 | 405 | 406 | def nematode(rows, cols): 407 | """ 408 | Adds small noise to avoid singular matrices. 409 | """ 410 | return 1e-8 * np.random.rand(rows, cols) 411 | 412 | 413 | def get_informative_neurons(pos, neg, k): 414 | """ 415 | Function to find the most informative neurons of a 416 | neural network model that are most correlated to the 417 | sensitive attirbute value. 418 | """ 419 | informative_neurons = [] 420 | correlation_vals = [] 421 | pos_ = pos + nematode(pos.shape[0], pos.shape[1]) 422 | neg_ = neg + nematode(neg.shape[0], neg.shape[1]) 423 | neuron_corr = np.corrcoef(np.hstack((np.concatenate((pos_, neg_), axis=0), np.concatenate((np.ones(len(pos)), np.zeros(len(neg))), axis=0).reshape((-1,1)))), rowvar=False)[-1, :-1] 424 | top_corr = sorted(list(zip(neuron_corr, list(range(len(neuron_corr))))), key=(lambda v: v[0]), reverse=True) 425 | 426 | sorted_neurons = [v[1] for v in top_corr] 427 | sorted_corr_vals = [v[0] for v in top_corr] 428 | qt = QuantileTransformer(random_state=0) 429 | qt.fit(np.vstack((pos[:, sorted_neurons], neg[:, sorted_neurons]))) 430 | pos_mean = np.mean(qt.transform(pos[:, sorted_neurons]), axis=0) 431 | pos_std = np.std(qt.transform(pos[:, sorted_neurons]), axis=0) 432 | neg_mean = np.mean(qt.transform(neg[:, sorted_neurons]), axis=0) 433 | neg_std = np.std(qt.transform(neg[:, sorted_neurons]), axis=0) 434 | pickle.dump([sorted_neurons, sorted_corr_vals, (pos_mean, pos_std, neg_mean, neg_std)], open('__temp_files/neuron_plot_info.p', 'wb')) 435 | 436 | top_corr = top_corr[:k] 437 | print("Top %d correlations are in [%.2f, %.2f]" % (k, top_corr[k-1][0], top_corr[0][0])) 438 | print("\nTop %d correlated neurons:" % (k)) 439 | for corr_val, neuron in top_corr: 440 | print("Neuron #%d with correlation value of %.2f" % (neuron, corr_val)) 441 | informative_neurons.append(neuron) 442 | correlation_vals.append(corr_val) 443 | qt = QuantileTransformer(random_state=0) 444 | qt.fit(np.vstack((pos[:, informative_neurons], neg[:, informative_neurons]))) 445 | pos_mean = np.mean(qt.transform(pos[:, informative_neurons]), axis=0) 446 | pos_std = np.std(qt.transform(pos[:, informative_neurons]), axis=0) 447 | neg_mean = np.mean(qt.transform(neg[:, informative_neurons]), axis=0) 448 | neg_std = np.std(qt.transform(neg[:, informative_neurons]), axis=0) 449 | return informative_neurons, correlation_vals, (pos_mean, pos_std, neg_mean, neg_std) 450 | 451 | 452 | def get_whitebox_score(X, correlation_vals, adv_known_idx): 453 | """ 454 | Returns the white-box attribute inference attack's score 455 | that is strictly between 0 and 1, that indicates the 456 | attack's confidence in predicting the sensitive attribute 457 | value for a query record. 458 | """ 459 | # robust transformation to map neuron outputs to [0,1] range 460 | qt = QuantileTransformer(random_state=0) 461 | qt.fit(X[adv_known_idx]) 462 | return np.average(qt.transform(X), weights=correlation_vals, axis=1) 463 | 464 | 465 | def whitebox_attack(layer_outputs, is_sensitive, adv_known_idx): 466 | """ 467 | Our white-box attack that uses the neuron outputs of a 468 | neural netowork model to predict the sensitive attribute 469 | value for a query record. 470 | """ 471 | pos_ind = list(filter(lambda x: is_sensitive[x], adv_known_idx)) 472 | neg_ind = list(set(adv_known_idx) - set(pos_ind)) 473 | informative_neurons, correlation_vals, plot_info = get_informative_neurons(layer_outputs[pos_ind], layer_outputs[neg_ind], k=100) 474 | whitebox_info_k_1 = get_whitebox_score(layer_outputs[:, informative_neurons[:1]], correlation_vals[:1], adv_known_idx) 475 | whitebox_info_k_10 = get_whitebox_score(layer_outputs[:, informative_neurons[:10]], correlation_vals[:10], adv_known_idx) 476 | whitebox_info_k_100 = get_whitebox_score(layer_outputs[:, informative_neurons], correlation_vals, adv_known_idx) 477 | return (whitebox_info_k_1, whitebox_info_k_10, whitebox_info_k_100), informative_neurons, correlation_vals, plot_info 478 | -------------------------------------------------------------------------------- /core/classifier.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import tensorflow.compat.v1 as tf 4 | 5 | from core.privacy_accountant import accountant 6 | from tensorflow_privacy.privacy.optimizers import dp_optimizer 7 | 8 | LOGGING = False # enables tf.train.ProfilerHook (see use below) 9 | LOG_DIR = 'project_log' 10 | CHECKPOINT_DIR = '__temp_files' 11 | 12 | AdamOptimizer = tf.train.AdamOptimizer 13 | 14 | 15 | def get_predictions(predictions): 16 | """ 17 | Returns the predicted labels and prediction scores for the inference set. 18 | """ 19 | pred_y, pred_scores = [], [] 20 | val = next(predictions, None) 21 | while val is not None: 22 | pred_y.append(val['classes']) 23 | pred_scores.append(val['probabilities']) 24 | val = next(predictions, None) 25 | return np.array(pred_y), np.array(pred_scores) 26 | 27 | 28 | def get_layer_outputs(predictions): 29 | """ 30 | Returns the neural network model's neuron outputs for the inference set. 31 | """ 32 | layer_outputs = [] 33 | val = next(predictions, None) 34 | while val is not None: 35 | layer_outputs.append(val['layer_outputs']) 36 | val = next(predictions, None) 37 | return np.array(layer_outputs) 38 | 39 | 40 | def get_model(features, labels, mode, params): 41 | """ 42 | Main workhorse function that defines the model according to model specification. 43 | """ 44 | n, n_in, n_hidden, n_out, non_linearity, model, privacy, dp, epsilon, delta, batch_size, learning_rate, clipping_threshold, l2_ratio, epochs = params 45 | 46 | if model == 'nn': 47 | #print('Using neural network...') 48 | input_layer = tf.reshape(features['x'], [-1, n_in]) 49 | h1 = tf.keras.layers.Dense(n_hidden, activation=non_linearity, kernel_regularizer=tf.keras.regularizers.l2(l2_ratio))(input_layer) 50 | h2 = tf.keras.layers.Dense(n_hidden, activation=non_linearity, kernel_regularizer=tf.keras.regularizers.l2(l2_ratio))(h1) 51 | pre_logits = tf.keras.layers.Dense(n_out, kernel_regularizer=tf.keras.regularizers.l2(l2_ratio))(h2) 52 | logits = tf.keras.layers.Softmax()(pre_logits) 53 | 54 | elif model == 'cnn': 55 | #print('Using convolution neural network...') # tailored for Cifar-100 56 | input_layer = tf.reshape(features['x'], [-1, 32, 32, 3]) 57 | y = tf.keras.layers.Conv2D(64, kernel_size=(3, 3), activation=non_linearity)(input_layer) 58 | y = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(y) 59 | y = tf.keras.layers.Conv2D(128, kernel_size=(3, 3), activation=non_linearity, input_shape=[-1, 32, 32, 3])(y) 60 | y = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(y) 61 | y = tf.keras.layers.Conv2D(256, kernel_size=(3, 3), activation=non_linearity, input_shape=[-1, 32, 32, 3])(y) 62 | y = tf.keras.layers.MaxPooling2D(pool_size=(2, 2))(y) 63 | y = tf.keras.layers.Flatten()(y) 64 | y = tf.nn.dropout(y, 0.2) 65 | h1 = tf.keras.layers.Dense(n_hidden, activation=non_linearity, kernel_regularizer=tf.keras.regularizers.l2(l2_ratio))(y) 66 | h2 = tf.keras.layers.Dense(n_hidden, activation=non_linearity, kernel_regularizer=tf.keras.regularizers.l2(l2_ratio))(h1) 67 | pre_logits = tf.keras.layers.Dense(n_out, kernel_regularizer=tf.keras.regularizers.l2(l2_ratio))(h2) 68 | logits = tf.keras.layers.Softmax()(pre_logits) 69 | 70 | else: 71 | #print('Using softmax regression...') 72 | input_layer = tf.reshape(features['x'], [-1, n_in]) 73 | logits = tf.keras.layers.Dense(n_out, activation=tf.nn.softmax, kernel_regularizer=tf.keras.regularizers.l2(l2_ratio))(input_layer) 74 | 75 | predictions = { 76 | "classes": tf.argmax(input=logits, axis=1), 77 | "probabilities": logits 78 | } 79 | if model != 'lr': 80 | predictions["layer_outputs"] = tf.concat([h1, h2, pre_logits], axis=1) # not to be used for softmax regression 81 | 82 | if mode == tf.estimator.ModeKeys.PREDICT: 83 | return tf.estimator.EstimatorSpec(mode=mode, 84 | predictions=predictions) 85 | 86 | vector_loss = tf.keras.losses.sparse_categorical_crossentropy(labels, logits) 87 | scalar_loss = tf.reduce_mean(vector_loss) 88 | 89 | if mode == tf.estimator.ModeKeys.TRAIN: 90 | 91 | if privacy == 'grad_pert': 92 | ac = accountant( 93 | data_size=n, 94 | batch_size=batch_size, 95 | epochs=epochs, 96 | target_delta=delta, 97 | dp_type=dp) 98 | sigma = ac.get_noise_multiplier(target_epsilon=epsilon) 99 | optimizer = dp_optimizer.DPAdamGaussianOptimizer( 100 | l2_norm_clip=clipping_threshold, 101 | noise_multiplier=sigma, 102 | num_microbatches=batch_size, 103 | learning_rate=learning_rate) 104 | opt_loss = vector_loss 105 | 106 | else: 107 | optimizer = AdamOptimizer(learning_rate=learning_rate) 108 | opt_loss = scalar_loss 109 | 110 | global_step = tf.train.get_global_step() 111 | train_op = optimizer.minimize(loss=opt_loss, global_step=global_step) 112 | 113 | return tf.estimator.EstimatorSpec(mode=mode, 114 | loss=scalar_loss, 115 | train_op=train_op) 116 | 117 | elif mode == tf.estimator.ModeKeys.EVAL: 118 | eval_metric_ops = { 119 | 'accuracy': 120 | tf.metrics.accuracy( 121 | labels=labels, 122 | predictions=predictions["classes"]) 123 | } 124 | 125 | return tf.estimator.EstimatorSpec(mode=mode, 126 | loss=scalar_loss, 127 | eval_metric_ops=eval_metric_ops) 128 | 129 | 130 | def train(dataset, n_out=None, n_hidden=50, batch_size=100, epochs=100, learning_rate=0.01, clipping_threshold=1, model='nn', l2_ratio=1e-7, silent=True, non_linearity='relu', privacy='no_privacy', dp = 'dp', epsilon=0.5, delta=1e-5): 131 | """ 132 | Calls the get_model() to create a model given the model specifications, 133 | performs model training and returns the trained model (along with the 134 | auxiliary information if silent != True). 135 | """ 136 | train_x, train_y, test_x, test_y = dataset 137 | 138 | n_in = train_x.shape[1] 139 | if n_out == None: 140 | n_out = len(set(np.unique(train_y)).union(set(np.unique(test_y)))) 141 | 142 | if batch_size > len(train_y): 143 | batch_size = len(train_y) 144 | 145 | if not os.path.exists(CHECKPOINT_DIR): 146 | os.makedirs(CHECKPOINT_DIR) 147 | 148 | classifier = tf.estimator.Estimator( 149 | model_fn=get_model, 150 | #model_dir=CHECKPOINT_DIR, 151 | params = [ 152 | train_x.shape[0], 153 | n_in, 154 | n_hidden, 155 | n_out, 156 | non_linearity, 157 | model, 158 | privacy, 159 | dp, 160 | epsilon, 161 | delta, 162 | batch_size, 163 | learning_rate, 164 | clipping_threshold, 165 | l2_ratio, 166 | epochs]) 167 | 168 | train_input_fn = tf.estimator.inputs.numpy_input_fn( 169 | x={'x': train_x}, 170 | y=train_y, 171 | batch_size=batch_size, 172 | num_epochs=epochs, 173 | shuffle=True) 174 | train_eval_input_fn = tf.estimator.inputs.numpy_input_fn( 175 | x={'x': train_x}, 176 | y=train_y, 177 | num_epochs=1, 178 | shuffle=False) 179 | test_eval_input_fn = tf.estimator.inputs.numpy_input_fn( 180 | x={'x': test_x}, 181 | y=test_y, 182 | num_epochs=1, 183 | shuffle=False) 184 | 185 | steps_per_epoch = train_x.shape[0] // batch_size 186 | 187 | if not os.path.exists(LOG_DIR): 188 | os.makedirs(LOG_DIR) 189 | for epoch in range(1, epochs + 1): 190 | hooks = [] 191 | if LOGGING: 192 | """ 193 | This hook will save traces of what tensorflow is doing 194 | during the training of each model. View the combined trace 195 | by running `combine_traces.py`. 196 | """ 197 | hooks.append(tf.train.ProfilerHook( 198 | output_dir=LOG_DIR, 199 | save_steps=30)) 200 | 201 | classifier.train(input_fn=train_input_fn, 202 | steps=steps_per_epoch, 203 | hooks=hooks) 204 | 205 | if not silent: 206 | eval_results = classifier.evaluate(input_fn=train_eval_input_fn) 207 | print('Train loss after %d epochs is: %.3f' % (epoch, eval_results['loss'])) 208 | 209 | if not silent: 210 | eval_results = classifier.evaluate(input_fn=train_eval_input_fn) 211 | train_loss = eval_results['loss'] 212 | train_acc = eval_results['accuracy'] 213 | print('Train accuracy is: %.3f' % (train_acc)) 214 | 215 | eval_results = classifier.evaluate(input_fn=test_eval_input_fn) 216 | test_loss = eval_results['loss'] 217 | test_acc = eval_results['accuracy'] 218 | print('Test accuracy is: %.3f' % (test_acc)) 219 | 220 | """ 221 | Warning: silent flag is only used for target model training, 222 | as it also returns auxiliary information. 223 | """ 224 | return classifier, (train_loss, train_acc, test_loss, test_acc) 225 | 226 | return classifier 227 | -------------------------------------------------------------------------------- /core/data_util.py: -------------------------------------------------------------------------------- 1 | import os 2 | import pickle 3 | import random 4 | import numpy as np 5 | import pandas as pd 6 | 7 | from core.utilities import plot_regions 8 | from sklearn.model_selection import train_test_split 9 | from collections import Counter 10 | 11 | # Seed for random number generator 12 | SEED = 21312 13 | 14 | def load_attack_data(MODEL_PATH): 15 | """ 16 | Helper function to load the attack data for meta-model training. 17 | """ 18 | fname = MODEL_PATH + 'attack_train_data.npz' 19 | with np.load(fname) as f: 20 | train_x, train_y = [f['arr_%d' % i] for i in range(len(f.files))] 21 | fname = MODEL_PATH + 'attack_test_data.npz' 22 | with np.load(fname) as f: 23 | test_x, test_y = [f['arr_%d' % i] for i in range(len(f.files))] 24 | return train_x.astype('float32'), train_y.astype('int32'), test_x.astype('float32'), test_y.astype('int32') 25 | 26 | 27 | def save_data(args, data_id=None): 28 | """ 29 | Function to create the training, test and hold-out sets 30 | by randomly sampling from the raw data set. 31 | """ 32 | print('-' * 10 + 'SAVING DATA TO DISK' + '-' * 10 + '\n') 33 | target_size = args.target_data_size 34 | gamma = args.target_test_train_ratio 35 | DATA_PATH = 'data/' + args.train_dataset + '/' 36 | if not os.path.exists(DATA_PATH): 37 | os.makedirs(DATA_PATH) 38 | if data_id and int(data_id) >= 0: 39 | x = pickle.load(open('../dataset/'+args.train_dataset+f'_features_{data_id}.p', 'rb')) 40 | y = pickle.load(open('../dataset/'+args.train_dataset+f'_labels_{data_id}.p', 'rb')) 41 | else: 42 | x = pickle.load(open('../dataset/'+args.train_dataset+'_features.p', 'rb')) 43 | y = pickle.load(open('../dataset/'+args.train_dataset+'_labels.p', 'rb')) 44 | x = np.array(x, dtype=np.float32) 45 | y = np.array(y, dtype=np.int32) 46 | print(x.shape, y.shape) 47 | 48 | # assert if data is enough for sampling target data 49 | assert(len(x) >= (1 + gamma) * target_size) 50 | x, train_x, y, train_y = train_test_split(x, y, test_size=target_size, stratify=y) 51 | print("Training set size: X: {}, y: {}".format(train_x.shape, train_y.shape)) 52 | x, test_x, y, test_y = train_test_split(x, y, test_size=int(gamma*target_size), stratify=y) 53 | print("Test set size: X: {}, y: {}".format(test_x.shape, test_y.shape)) 54 | 55 | # save target data 56 | print('Saving data for target model') 57 | if data_id and int(data_id) >= 0: 58 | np.savez(DATA_PATH + f'target_data_{data_id}.npz', train_x, train_y, test_x, test_y) 59 | else: 60 | np.savez(DATA_PATH + 'target_data.npz', train_x, train_y, test_x, test_y) 61 | 62 | # assert if remaining data is enough for sampling shadow data 63 | assert(len(x) >= (1 + gamma) * target_size) 64 | 65 | # save shadow data 66 | for i in range(args.n_shadow): 67 | print('Saving data for shadow model {}'.format(i)) 68 | train_x, test_x, train_y, test_y = train_test_split(x, y, train_size=target_size, test_size=int(gamma*target_size), stratify=y) 69 | print("Training set size: X: {}, y: {}".format(train_x.shape, train_y.shape)) 70 | print("Test set size: X: {}, y: {}".format(test_x.shape, test_y.shape)) 71 | if data_id and data_id >= 0: 72 | np.savez(DATA_PATH + 'shadow{}_data_{}.npz'.format(i, data_id), train_x, train_y, test_x, test_y) 73 | else: 74 | np.savez(DATA_PATH + 'shadow{}_data.npz'.format(i), train_x, train_y, test_x, test_y) 75 | 76 | 77 | def sample_noniid_data(args): 78 | """ 79 | Function to create the training, test and hold-out sets 80 | by custom non-iid sampling from the raw data set. 81 | """ 82 | # this functionality is currently only for census and texas data set 83 | assert(args.train_dataset == 'census' or args.train_dataset == 'texas_100_v2') 84 | 85 | print('-' * 10 + 'SAMPLING NON-IID DATA' + '-' * 10 + '\n') 86 | target_size = args.target_data_size 87 | gamma = args.target_test_train_ratio 88 | DATA_PATH = 'data/' + args.train_dataset + '/' 89 | if not os.path.exists(DATA_PATH): 90 | os.makedirs(DATA_PATH) 91 | x = pickle.load(open('../dataset/'+args.train_dataset+'_features.p', 'rb')) 92 | y = pickle.load(open('../dataset/'+args.train_dataset+'_labels.p', 'rb')) 93 | x = np.array(x, dtype=np.float32) 94 | y = np.array(y, dtype=np.int32) 95 | print(x.shape, y.shape) 96 | 97 | # assert if data is enough for sampling data sub-sets 98 | assert(len(x) >= (1 + gamma) * 4 * target_size) 99 | 100 | target_attrs, attribute_dict, max_attr_vals, col_flags = get_sensitive_features(args.train_dataset, x) 101 | if args.train_dataset == 'census': 102 | assert(col_flags['CENSUS_ST'] != None and col_flags['CENSUS_PUMA'] != None and col_flags['CENSUS_PINCP'] != None) 103 | # target attribute is Race for Census 104 | target_attr = target_attrs[1] 105 | money_attr = col_flags['CENSUS_PINCP'] 106 | elif args.train_dataset == 'texas_100_v2': 107 | assert(col_flags['TEXAS_THCIC_ID'] != None and col_flags['TEXAS_TOTAL_CHARGES'] != None) 108 | # target attribute is Ethnicity for Texas 109 | target_attr = target_attrs[2] 110 | money_attr = col_flags['TEXAS_TOTAL_CHARGES'] 111 | 112 | # sample target data from general distribution 113 | x, train_x, y, train_y = train_test_split(x, y, test_size=target_size, stratify=y) 114 | print("Target Train set size: X: {}, y: {}".format(train_x.shape, train_y.shape)) 115 | x, test_x, y, test_y = train_test_split(x, y, test_size=int(gamma*target_size), stratify=y) 116 | print("Target Test set size: X: {}, y: {}".format(test_x.shape, test_y.shape)) 117 | print(np.sum(train_x[:, target_attr] * max_attr_vals[target_attr] == args.sensitive_outcome) / target_size, np.sum(test_x[:, target_attr] * max_attr_vals[target_attr] == args.sensitive_outcome) / int(gamma*target_size)) 118 | print(Counter(train_x[:, target_attr] * max_attr_vals[target_attr]), len(Counter(train_y).keys())) 119 | print('Saving target data\n') 120 | np.savez(DATA_PATH + 'target_data.npz', train_x, train_y, test_x, test_y) 121 | 122 | # sample holdout data from general distribution 123 | x, h_train_x, y, h_train_y = train_test_split(x, y, test_size=target_size, stratify=y) 124 | print("Holdout Train set size: X: {}, y: {}".format(h_train_x.shape, h_train_y.shape)) 125 | x, h_test_x, y, h_test_y = train_test_split(x, y, test_size=int(gamma*target_size), stratify=y) 126 | print("Holdout Test set size: X: {}, y: {}".format(h_test_x.shape, h_test_y.shape)) 127 | print(np.sum(h_train_x[:, target_attr] * max_attr_vals[target_attr] == args.sensitive_outcome) / target_size, np.sum(h_test_x[:, target_attr] * max_attr_vals[target_attr] == args.sensitive_outcome) / int(gamma*target_size)) 128 | print(Counter(h_train_x[:, target_attr] * max_attr_vals[target_attr]), len(Counter(h_train_y).keys())) 129 | print('Saving target holdout data') 130 | np.savez(DATA_PATH + 'holdout_data.npz', h_train_x, h_train_y, h_test_x, h_test_y) 131 | 132 | # group the records into regions 133 | region = {} 134 | for idx, record in enumerate(x): 135 | if args.train_dataset == 'texas_100_v2': 136 | r = str(record[col_flags['TEXAS_THCIC_ID']] * max_attr_vals[col_flags['TEXAS_THCIC_ID']]) 137 | else: 138 | r = str(record[col_flags['CENSUS_ST']] * max_attr_vals[col_flags['CENSUS_ST']]) + '+' + str(record[col_flags['CENSUS_PUMA']] * max_attr_vals[col_flags['CENSUS_PUMA']]) 139 | if r not in region: 140 | region[r] = [idx] 141 | else: 142 | region[r].append(idx) 143 | print(len(region)) 144 | plot_info = [[ 145 | r, 146 | np.mean(x[indices, money_attr] * max_attr_vals[money_attr]), 147 | np.sum(x[indices, target_attr] * max_attr_vals[target_attr] == args.skew_outcome), 148 | np.sum(x[indices, target_attr] * max_attr_vals[target_attr] == args.sensitive_outcome), 149 | len(indices) 150 | ] for r, indices in region.items()] 151 | # sort regions based on Income or Charges (High to Low) 152 | if args.skew_attribute == 1: 153 | plot_info = sorted(plot_info, key=(lambda x: x[1]), reverse=True) 154 | # sort regions based on Race or Ethnicity (High to Low) 155 | elif args.skew_attribute == 2: 156 | plot_info = sorted(plot_info, key=(lambda x: x[2]), reverse=True) 157 | # sort regions based on Population (High to Low) 158 | else: 159 | plot_info = sorted(plot_info, key=(lambda x: x[4]), reverse=True) 160 | plot_regions(plot_info, attribute_dict[target_attr][args.skew_outcome], attribute_dict[target_attr][args.sensitive_outcome], args.train_dataset) 161 | 162 | sk_idx = [] 163 | i = 0 164 | region_list = [] 165 | while len(sk_idx) < (1 + gamma) * target_size: 166 | sk_idx.extend(region[plot_info[i][0]]) 167 | region_list.append(i) 168 | i += 1 169 | print(len(region_list)) 170 | # sample skewed data from skewed distribution (most populous hospitals) 171 | sk_train_x, sk_test_x, sk_train_y, sk_test_y = train_test_split(x[sk_idx], y[sk_idx], train_size=target_size, test_size=int(gamma*target_size), stratify=y[sk_idx]) 172 | print("Skewed Train set size: X: {}, y: {}".format(sk_train_x.shape, sk_train_y.shape)) 173 | print("Skewed Test set size: X: {}, y: {}".format(sk_test_x.shape, sk_test_y.shape)) 174 | print(np.sum(sk_train_x[:, target_attr] * max_attr_vals[target_attr] == args.sensitive_outcome) / target_size, np.sum(sk_test_x[:, target_attr] * max_attr_vals[target_attr] == args.sensitive_outcome) / int(gamma*target_size)) 175 | print(Counter(sk_train_x[:, target_attr] * max_attr_vals[target_attr]), len(Counter(sk_train_y).keys())) 176 | print('Saving skewed data\n') 177 | np.savez(DATA_PATH + 'skewed_data.npz', sk_train_x, sk_train_y, sk_test_x, sk_test_y) 178 | 179 | sk_idx = [] 180 | i = len(region) - 1 181 | region_list = [] 182 | while len(sk_idx) < (1 + gamma) * target_size: 183 | sk_idx.extend(region[plot_info[i][0]]) 184 | region_list.append(i) 185 | i -= 1 186 | print(len(region_list)) 187 | # sample skewed 2 data from skewed distribution (least populous hospitals) 188 | sk2_train_x, sk2_test_x, sk2_train_y, sk2_test_y = train_test_split(x[sk_idx], y[sk_idx], train_size=target_size, test_size=int(gamma*target_size), stratify=y[sk_idx]) 189 | print("Skewed 2 Train set size: X: {}, y: {}".format(sk2_train_x.shape, sk2_train_y.shape)) 190 | print("Skewed 2 Test set size: X: {}, y: {}".format(sk2_test_x.shape, sk2_test_y.shape)) 191 | print(np.sum(sk2_train_x[:, target_attr] * max_attr_vals[target_attr] == args.sensitive_outcome) / target_size, np.sum(sk2_test_x[:, target_attr] * max_attr_vals[target_attr] == args.sensitive_outcome) / int(gamma*target_size)) 192 | print(Counter(sk2_train_x[:, target_attr] * max_attr_vals[target_attr]), len(Counter(sk2_train_y).keys())) 193 | print('Saving skewed 2 data\n') 194 | np.savez(DATA_PATH + 'skewed_2_data.npz', sk2_train_x, sk2_train_y, sk2_test_x, sk2_test_y) 195 | 196 | 197 | def load_data(data_name, args): 198 | """ 199 | Loads the training and test sets for the given data set. 200 | """ 201 | DATA_PATH = 'data/' + args.train_dataset + '/' 202 | target_size = args.target_data_size 203 | gamma = args.target_test_train_ratio 204 | with np.load(DATA_PATH + data_name) as f: 205 | train_x, train_y, test_x, test_y = [f['arr_%d' % i] for i in range(len(f.files))] 206 | 207 | train_x = np.array(train_x, dtype=np.float32) 208 | test_x = np.array(test_x, dtype=np.float32) 209 | 210 | train_y = np.array(train_y, dtype=np.int32) 211 | test_y = np.array(test_y, dtype=np.int32) 212 | 213 | return train_x, train_y, test_x[:int(gamma*target_size)], test_y[:int(gamma*target_size)] 214 | 215 | 216 | def process_features(features, dataset, attribute_dict, max_attr_vals, target_attr, col_flags, skip_sensitive=False, skip_corr=False): 217 | """ 218 | Returns the feature matrix after expanding the nominal features, 219 | and removing the features that are not needed for model training. 220 | """ 221 | features = pd.DataFrame(features) 222 | # for removing sensitive feature 223 | if skip_sensitive: 224 | features.drop(columns=target_attr, inplace=True) 225 | # skipping the ST (state) and PUMA attribute for census data set 226 | if dataset == 'census': 227 | if col_flags['CENSUS_ST'] != None: 228 | features.drop(columns=col_flags['CENSUS_ST'], inplace=True) 229 | if col_flags['CENSUS_PUMA'] != None: 230 | features.drop(columns=col_flags['CENSUS_PUMA'], inplace=True) 231 | if col_flags['CENSUS_PINCP'] != None: 232 | features.drop(columns=col_flags['CENSUS_PINCP'], inplace=True) 233 | # skipping one of the conflicting attributes for texas data set 234 | if dataset == 'texas_100_v2': 235 | if col_flags['TEXAS_THCIC_ID'] != None: 236 | features.drop(columns=col_flags['TEXAS_THCIC_ID'], inplace=True) 237 | # skip_corr flag is used to decide whether to skip the correlated attribute in Texas-100X 238 | if skip_corr == True: 239 | if col_flags['TEXAS_RACE'] == target_attr: 240 | features.drop(columns=col_flags['TEXAS_ETHN'], inplace=True) 241 | elif col_flags['TEXAS_ETHN'] == target_attr: 242 | features.drop(columns=col_flags['TEXAS_RACE'], inplace=True) 243 | # for expanding categorical features 244 | if attribute_dict != None: 245 | for col in attribute_dict: 246 | # skip in case the sensitive feature was removed above 247 | if col not in features.columns: 248 | continue 249 | # to expand the non-binary categorical features 250 | if max_attr_vals[col] != 1: 251 | features[col] *= max_attr_vals[col] 252 | features[col] = pd.Categorical(features[col], categories=range(int(max_attr_vals[col])+1)) 253 | features = pd.get_dummies(features) 254 | return np.array(features) 255 | 256 | 257 | def threat_model(args, tot_len): 258 | """ 259 | Returns the data indices based on the adversarial knowledge. 260 | """ 261 | train_size = args.target_data_size 262 | test_size = int(args.target_data_size * args.target_test_train_ratio) 263 | c_size = args.candidate_size 264 | assert(tot_len >= 4 * (train_size + test_size)) 265 | 266 | random.seed(args.run) # random.seed(0) 267 | train_c_idx = random.sample(range(train_size), c_size) 268 | test_c_idx = random.sample(range(train_size, train_size + test_size), c_size) 269 | h_test_idx = list(range(2 * train_size + test_size, 2 * (train_size + test_size))) 270 | sk_test_idx = list(range(3 * train_size + 2 * test_size, 3 * (train_size + test_size))) 271 | sk2_test_idx = list(range(4 * train_size + 3 * test_size, 4 * (train_size + test_size))) 272 | 273 | adv_known_idxs = {} 274 | # low: adversary only knows data from skewed distribution 275 | adv_known_idxs['low'] = list(range(2 * (train_size + test_size), 3 * train_size + 2 * test_size)) 276 | # low2: adversary only knows data from skewed_2 distribution 277 | adv_known_idxs['low2'] = list(range(3 * (train_size + test_size), 4 * train_size + 3 * test_size)) 278 | # med: adversary knows data from training distribution 279 | adv_known_idxs['med'] = list(range(train_size + test_size, 2 * train_size + test_size)) 280 | #high: adversary knows all train records except candidate set 281 | adv_known_idxs['high'] = list(set(range(train_size)) - set(train_c_idx)) 282 | 283 | if tot_len == 5 * (train_size + test_size): 284 | ood_c_idx = random.sample(range(5 * train_size + 4 * test_size, 5 * (train_size + test_size)), c_size) 285 | return train_c_idx, test_c_idx, ood_c_idx, h_test_idx, sk_test_idx, sk2_test_idx, adv_known_idxs 286 | return train_c_idx, test_c_idx, h_test_idx, sk_test_idx, sk2_test_idx, adv_known_idxs 287 | 288 | 289 | def get_num_classes(dataset_name): 290 | if dataset_name in ['texas_100_v2', 'purchase_100', 'cifar_100']: 291 | return 100 292 | return 2 293 | 294 | 295 | def get_sensitive_features(dataset_name, data, size=3): 296 | """ 297 | Returns the sensitive features for a given data set, along 298 | with the attribute dictionary if available. 299 | """ 300 | col_flags = {x: None for x in ['CENSUS_ST', 'CENSUS_PUMA', 'CENSUS_PINCP', 'TEXAS_RACE', 'TEXAS_ETHN', 'TEXAS_THCIC_ID', 'TEXAS_TOTAL_CHARGES']} 301 | if dataset_name == 'adult': 302 | attribute_dict, max_attr_vals = pickle.load(open('../dataset/adult_feature_desc.p', 'rb')) 303 | return [7, 6, 1, 3, 4, 5, 11][:size], attribute_dict, max_attr_vals, col_flags 304 | elif dataset_name == 'compas': 305 | attribute_idx, attribute_dict, max_attr_vals = pickle.load(open('../dataset/compas_feature_desc.p', 'rb')) 306 | return [attribute_idx['sex'], attribute_idx['race'], attribute_idx['age_cat']][:size], attribute_dict, max_attr_vals, col_flags 307 | elif dataset_name == 'census': 308 | attribute_idx, attribute_dict, max_attr_vals = pickle.load(open('../dataset/census_feature_desc.p', 'rb')) 309 | col_flags['CENSUS_ST'] = None if 'ST' not in attribute_idx else attribute_idx['ST'] 310 | col_flags['CENSUS_PUMA'] = None if 'PUMA' not in attribute_idx else attribute_idx['PUMA'] 311 | col_flags['CENSUS_PINCP'] = None if 'PINCP' not in attribute_idx else attribute_idx['PINCP'] 312 | return [attribute_idx['SEX'], attribute_idx['RAC1P'], attribute_idx['DEYE'], attribute_idx['DREM']][:size], attribute_dict, max_attr_vals, col_flags 313 | elif dataset_name == 'texas_100_v2': 314 | attribute_idx, attribute_dict, max_attr_vals = pickle.load(open('../dataset/texas_100_v2_feature_desc.p', 'rb')) 315 | col_flags['TEXAS_TOTAL_CHARGES'] = attribute_idx['TOTAL_CHARGES'] 316 | col_flags['TEXAS_RACE'] = attribute_idx['RACE'] 317 | col_flags['TEXAS_ETHN'] = attribute_idx['ETHNICITY'] 318 | col_flags['TEXAS_THCIC_ID'] = None if 'THCIC_ID' not in attribute_idx else attribute_idx['THCIC_ID'] 319 | return [attribute_idx['SEX_CODE'], attribute_idx['RACE'], attribute_idx['ETHNICITY'], attribute_idx['TYPE_OF_ADMISSION']][:size], attribute_dict, max_attr_vals, col_flags 320 | return get_random_features(data, list(range(data.shape[1])), size), None, np.ones(data.shape[1], dtype=int), col_flags 321 | 322 | 323 | def get_random_features(data, pool, size): 324 | """ 325 | Returns a random set of features from a given data set. 326 | These are used for attribute inference, in cases where 327 | the sensitive attributes are not known. 328 | """ 329 | random.seed(SEED) 330 | features = set() 331 | while(len(features) < size): 332 | feat = random.choice(pool) 333 | c = Counter(data[:, feat]) 334 | # for binary features, select the ones with 1 being minority 335 | if sorted(list(c.keys())) == [0, 1]: 336 | if c[1] / len(data) > 0.1 and c[1] / len(data) < 0.5: 337 | features.add(feat) 338 | # select feature that has more than one value 339 | elif len(c.keys()) > 1: 340 | features.add(feat) 341 | return list(features) 342 | 343 | 344 | def subsample(indices, labels, sample_size): 345 | """ 346 | Returns a subsample of indices such that the sample contains 347 | at least one record for each label. 348 | """ 349 | if sample_size >= len(indices): 350 | return indices 351 | idx = [] 352 | labs = np.unique(labels) 353 | for l in labs: 354 | idx.append(indices[np.where(np.array(labels) == l)[0][0]]) 355 | random.seed(SEED) 356 | idx.extend(random.sample(list(set(indices) - set(idx)), sample_size - len(labs))) 357 | return idx 358 | -------------------------------------------------------------------------------- /core/privacy_accountant.py: -------------------------------------------------------------------------------- 1 | import math 2 | import time 3 | import numpy as np 4 | 5 | #from tensorflow_privacy.privacy.analysis.rdp_accountant import compute_rdp 6 | #from tensorflow_privacy.privacy.analysis.rdp_accountant import get_privacy_spent 7 | from tensorflow_privacy.privacy.analysis import compute_dp_sgd_privacy_lib 8 | from scipy.stats import norm 9 | 10 | MAX_SIGMA = 1e6 11 | EPS_TOLERANCE = 8e-3 # lower tolerance may not converge 12 | 13 | 14 | def compute_gdp_mu(sampling_rate, sigma, steps): 15 | """ 16 | Calculates the mu for the GDP mechanism for a given sigma value. 17 | """ 18 | return sampling_rate * np.sqrt(steps * (np.exp(1 / sigma**2) - 1) ) 19 | 20 | 21 | def get_gdp_privacy_spent(mu, target_delta): 22 | """ 23 | Returns the estimated optimal epsilon and delta values for the mu-GDP 24 | mechanism given a threshold on delta. 25 | """ 26 | eps_orders = np.logspace(-3.0, 2.0, num=10000) 27 | low, high = 0, len(eps_orders) - 1 28 | delta = 1 29 | while low <= high: 30 | mid = (low + high) // 2 31 | eps = eps_orders[mid] 32 | dlt = norm.cdf(-eps / mu + mu / 2, loc=0, scale=1) - np.exp(eps) * norm.cdf(-eps / mu - mu / 2, loc=0, scale=1) 33 | if dlt <= target_delta: 34 | delta = dlt 35 | high = mid - 1 36 | else: 37 | low = mid + 1 38 | return eps, delta 39 | 40 | 41 | class accountant: 42 | """ 43 | Privacy accountant class that calculates the noise multiplier (sigma) for 44 | given privacy parameters (epsilon and delta), and the accounting type. 45 | """ 46 | 47 | def __init__(self, data_size, batch_size, epochs, target_delta, dp_type): 48 | """ 49 | data_size: int -- size of the model training set 50 | batch_size: int -- batch size used in model training 51 | epochs: int -- number of training iterations over the training set 52 | target_delta : float -- tolerance on failure probability for privacy accounting 53 | dp_type: str -- type of differential privacy accounting used: 'dp', 'adv_cmp', 'zcdp', 'rdp', 'gdp' 54 | """ 55 | self.data_size = data_size 56 | self.batch_size = batch_size 57 | self.epochs = epochs 58 | self.target_delta = target_delta 59 | self.dp_type = dp_type 60 | 61 | self.steps_per_epoch = self.data_size // self.batch_size 62 | self.steps = self.epochs * self.steps_per_epoch 63 | self.sampling_rate = self.batch_size / self.data_size 64 | 65 | 66 | def get_noise_multiplier(self, target_epsilon): 67 | """ 68 | Return the noise multiplier (sigma) for the given privacy accounting mechanism. 69 | """ 70 | if self.dp_type == 'dp': 71 | return self.epochs * np.sqrt(2 * np.log(1.25 * self.epochs / self.target_delta)) / target_epsilon 72 | 73 | elif self.dp_type == 'adv_cmp': 74 | return np.sqrt(self.epochs * np.log(2.5 * self.epochs / self.target_delta)) * (np.sqrt(np.log(2 / self.target_delta) + 2 * target_epsilon) + np.sqrt(np.log(2 / self.target_delta))) / target_epsilon 75 | 76 | elif self.dp_type == 'zcdp': 77 | return np.sqrt(self.epochs / 2) * (np.sqrt(np.log(1 / self.target_delta) + target_epsilon) + np.sqrt(np.log(1 / self.target_delta))) / target_epsilon 78 | 79 | else: # if self.dp_type == 'rdp' or 'gdp' 80 | return self.search_optimal_noise_multiplier(target_epsilon) 81 | 82 | 83 | def search_optimal_noise_multiplier(self, target_epsilon): 84 | """ 85 | Performs binary search to get the optimal value for noise multiplier (sigma) for RDP and GDP accounting mechanisms. Functionality adapted from Opacus (https://github.com/pytorch/opacus). 86 | """ 87 | eps_high = float("inf") 88 | sigma_low, sigma_high = 0, 10 89 | orders = [1 + x / 100.0 for x in range(1, 1000)] + list(range(12, 1200)) 90 | 91 | while eps_high > target_epsilon: 92 | sigma_high = 2 * sigma_high 93 | 94 | if self.dp_type == 'rdp': 95 | #rdp = compute_rdp(self.sampling_rate, sigma_high, self.steps, orders) 96 | #eps_high, _, _ = get_privacy_spent(orders, rdp, target_delta=self.target_delta) 97 | eps_high, _ = compute_dp_sgd_privacy_lib.compute_dp_sgd_privacy(self.data_size, self.batch_size, sigma_high, self.epochs, self.target_delta) 98 | else: # if self.dp_type == 'gdp' 99 | mu = compute_gdp_mu(self.sampling_rate, sigma_high, self.steps) 100 | eps_high, delta = get_gdp_privacy_spent(mu, target_delta=self.target_delta) 101 | if delta > self.target_delta: 102 | raise ValueError("Could not find suitable privacy parameters.") 103 | 104 | if sigma_high > MAX_SIGMA: 105 | raise ValueError("The privacy budget is too low.") 106 | 107 | while target_epsilon - eps_high > EPS_TOLERANCE * target_epsilon: 108 | sigma = (sigma_low + sigma_high) / 2 109 | 110 | if self.dp_type == 'rdp': 111 | #rdp = compute_rdp(self.sampling_rate, sigma, self.steps, orders) 112 | #eps, _, _ = get_privacy_spent(orders, rdp, target_delta=self.target_delta) 113 | eps, _ = compute_dp_sgd_privacy_lib.compute_dp_sgd_privacy(self.data_size, self.batch_size, sigma, self.epochs, self.target_delta) 114 | else: # if self.dp_type == 'gdp' 115 | mu = compute_gdp_mu(self.sampling_rate, sigma, self.steps) 116 | eps, delta = get_gdp_privacy_spent(mu, target_delta=self.target_delta) 117 | 118 | if eps < target_epsilon: 119 | sigma_high = sigma 120 | eps_high = eps 121 | else: 122 | sigma_low = sigma 123 | 124 | return sigma_high 125 | 126 | 127 | if __name__ == '__main__': 128 | ac = accountant( 129 | data_size=50000, 130 | batch_size=500, 131 | epochs=50, 132 | target_delta=1e-5, 133 | dp_type='gdp') 134 | t0 = time.time() 135 | print("Sigma = %.4f" % ac.get_noise_multiplier(target_epsilon=10)) 136 | print("%.2f seconds to execute" % (time.time() - t0)) 137 | -------------------------------------------------------------------------------- /core/utilities.py: -------------------------------------------------------------------------------- 1 | import random 2 | import numpy as np 3 | import matplotlib.pyplot as plt 4 | 5 | from sklearn.metrics import confusion_matrix 6 | from sklearn.metrics import roc_curve 7 | from sklearn.tree import DecisionTreeClassifier 8 | from sklearn.neural_network import MLPClassifier 9 | from sklearn.linear_model import LogisticRegression 10 | from sklearn import svm 11 | 12 | # To avoid numerical inconsistency in calculating log 13 | SMALL_VALUE = 1e-6 14 | 15 | 16 | def pretty_print_result(mem, pred): 17 | tn, fp, fn, tp = confusion_matrix(mem, pred).ravel() 18 | print('TP: %d FP: %d FN: %d TN: %d' % (tp, fp, fn, tn)) 19 | if tp == fp == 0: 20 | print('PPV: 0\nAdvantage: 0') 21 | else: 22 | print('PPV: %.4f\nAdvantage: %.4f' % (tp / (tp + fp), tp / (tp + fn) - fp / (tn + fp))) 23 | 24 | 25 | def pretty_print_confusion_matrix(cm, labels): 26 | cm = np.matrix(cm) 27 | N = len(labels) 28 | matrix = [['' for i in range(N+6)] for j in range(N+1)] 29 | for i in range(N): 30 | matrix[0][i+1] = matrix[i+1][0] = labels[i][:5] 31 | matrix[0][N+1] = 'Total' 32 | matrix[0][N+2] = 'TPR' 33 | matrix[0][N+3] = 'FPR' 34 | matrix[0][N+4] = 'Adv' 35 | matrix[0][N+5] = 'PPV' 36 | for i in range(N): 37 | matrix[i+1][N+1] = np.sum(cm[i]) 38 | matrix[i+1][N+2] = cm[i, i] / np.sum(cm[i]) 39 | matrix[i+1][N+3] = (np.sum(cm[:, i]) - cm[i, i]) / (np.sum(cm) - np.sum(cm[i])) 40 | matrix[i+1][N+4] = str(matrix[i+1][N+2] - matrix[i+1][N+3])[:5] 41 | matrix[i+1][N+5] = 0 if np.sum(cm[:, i]) == 0 else str(cm[i, i] / np.sum(cm[:, i]))[:5] 42 | matrix[i+1][N+2] = str(matrix[i+1][N+2])[:5] 43 | matrix[i+1][N+3] = str(matrix[i+1][N+3])[:5] 44 | for j in range(N): 45 | matrix[i+1][j+1] = cm[i,j] 46 | print('\n'.join([''.join(['{:>8}'.format(val) for val in row]) for row in matrix])) 47 | print('Accuracy: %.3f' % (np.sum([cm[i, i] for i in range(N)]) / np.sum(cm))) 48 | 49 | 50 | def pretty_position(X, Y, pos): 51 | return ((X[pos] + X[pos+1]) / 2, (Y[pos] + Y[pos+1]) / 2) 52 | 53 | 54 | def get_ppvs(ref, vec): 55 | ppv = np.zeros(len(vec)) 56 | vec = sorted(zip(ref, vec), reverse=True, key=(lambda x: x[1])) 57 | 58 | vec = [x[0] for x in vec] 59 | s = 0 60 | for i, val in enumerate(vec): 61 | s += val 62 | if s == 0: 63 | continue 64 | ppv[i] = s/(i+1) 65 | return ppv 66 | 67 | 68 | def get_ppv(mem, pred): 69 | tn, fp, fn, tp = confusion_matrix(mem, pred).ravel() 70 | if tp == fp == 0: 71 | return 0 72 | return tp / (tp + fp) 73 | 74 | 75 | def get_adv(mem, pred): 76 | tn, fp, fn, tp = confusion_matrix(mem, pred).ravel() 77 | return (tp / (tp + fn)) - (fp / (tn + fp)) 78 | 79 | 80 | def get_fp(mem, pred): 81 | tn, fp, fn, tp = confusion_matrix(mem, pred).ravel() 82 | return fp 83 | 84 | 85 | def get_inference_threshold(pred_vector, true_vector, fpr_threshold=None): 86 | fpr, tpr, thresholds = roc_curve(true_vector, pred_vector, pos_label=1) 87 | # return inference threshold corresponding to maximum advantage 88 | if fpr_threshold == None: 89 | return thresholds[np.argmax(tpr-fpr)] 90 | # return inference threshold corresponding to fpr_threshold 91 | for a, b in zip(fpr, thresholds): 92 | if a > fpr_threshold: 93 | break 94 | alpha_thresh = b 95 | return alpha_thresh 96 | 97 | 98 | def loss_range(): 99 | return [10**i for i in np.arange(-7, 1, 0.1)] 100 | 101 | 102 | def log_loss(a, b): 103 | return [-np.log(max(b[i,int(a[i])], SMALL_VALUE)) for i in range(len(a))] 104 | 105 | 106 | def get_attribute_variations(data, feature): 107 | if len(np.unique(data[:,feature])) == 2: 108 | low, high = np.unique(data[:,feature]) 109 | pivot = (low + high) / 2 110 | else: 111 | pivot = np.quantile(np.unique(data[:,feature]), 0.5) 112 | low = np.quantile(np.unique(data[:,feature]), 0.25) 113 | high = np.quantile(np.unique(data[:,feature]), 0.75) 114 | true_attribute_value = np.where(data[:,feature] <= pivot, 0, 1) 115 | return low, high, true_attribute_value 116 | 117 | 118 | def generate_noise(shape, dtype, noise_params): 119 | noise_type, noise_coverage, noise_magnitude = noise_params 120 | if noise_coverage == 'full': 121 | if noise_type == 'uniform': 122 | return np.array(np.random.uniform(0, noise_magnitude, size=shape), dtype=dtype) 123 | else: 124 | return np.array(np.random.normal(0, noise_magnitude, size=shape), dtype=dtype) 125 | attr = np.random.randint(shape[1]) 126 | noise = np.zeros(shape, dtype=dtype) 127 | if noise_type == 'uniform': 128 | noise[:, attr] = np.array(np.random.uniform(0, noise_magnitude, size=shape[0]), dtype=dtype) 129 | else: 130 | noise[:, attr] = np.array(np.random.normal(0, noise_magnitude, size=shape[0]), dtype=dtype) 131 | return noise 132 | 133 | 134 | ''' 135 | Decision Tree best parameters seem to vary for different attributes: 136 | Texas Race (4) - Depth 5, Sample Split 10, Sample Leaf 5 137 | Texas Ethnicity - Depth 4, Sample Split 10, Sample Leaf 5 138 | Census Sex - Depth 4, Sample Split 10, Sample Leaf 5 139 | Census Race (3) - Depth 5, Sample Split 50, Sample Leaf 25 140 | ''' 141 | def fit_model(args, y, x1, x2): 142 | #clf = svm.SVC(kernel='rbf', gamma=0.7, C=1.0, probability=True) 143 | if args.dataset == 'census': 144 | if args.attribute == 1: 145 | clf = DecisionTreeClassifier(max_depth=5, min_samples_split=50, min_samples_leaf=25) 146 | else: 147 | clf = DecisionTreeClassifier(max_depth=4, min_samples_split=10, min_samples_leaf=5) 148 | else: 149 | if args.attribute == 1: 150 | clf = DecisionTreeClassifier(max_depth=5, min_samples_split=10, min_samples_leaf=5) 151 | else: 152 | clf = DecisionTreeClassifier(max_depth=4, min_samples_split=10, min_samples_leaf=5) 153 | if args.comb_flag == 0: 154 | clf.fit(np.vstack((x1, x2)).T, y) 155 | else: 156 | clf.fit(np.vstack((x1, x2, x1 * x2)).T, y) 157 | return clf 158 | 159 | 160 | def imputation_training(args, train_x, train_y, test_x, test_y, clf_type='nn', epochs=None): 161 | if epochs == None: 162 | epochs = args.target_epochs 163 | if clf_type == 'nn': 164 | clf = MLPClassifier( 165 | random_state=1, 166 | max_iter=epochs, 167 | hidden_layer_sizes=(args.target_n_hidden, args.target_n_hidden), 168 | alpha=args.target_l2_ratio, 169 | batch_size=args.target_batch_size, 170 | learning_rate_init=args.target_learning_rate) 171 | else: 172 | clf = LogisticRegression( 173 | random_state=1, 174 | max_iter=epochs, 175 | C=1/args.target_l2_ratio) 176 | clf.fit(train_x, train_y) 177 | train_conf = clf.predict_proba(train_x) 178 | test_conf = clf.predict_proba(test_x) 179 | train_loss = np.mean(log_loss(train_y, train_conf)) 180 | test_loss = np.mean(log_loss(test_y, test_conf)) 181 | train_acc = clf.score(train_x, train_y) 182 | test_acc = clf.score(test_x, test_y) 183 | return test_conf, (train_loss, train_acc, test_loss, test_acc) 184 | 185 | 186 | def make_line_plot(X, Y, fmt, color, label, lpos=None, loffset=0.02, fsize=18): 187 | plt.plot(X, np.mean(Y, axis=0), fmt, color=color, label=label) 188 | plt.fill_between( 189 | list(range(1, len(X) + 1)), 190 | np.mean(Y, axis=0) - np.std(Y, axis=0), 191 | np.mean(Y, axis=0) + np.std(Y, axis=0), 192 | color=color, 193 | alpha=0.1) 194 | if lpos != None: 195 | plt.text(X[lpos]+np.log10(X[lpos]+0.1), np.mean(Y, axis=0)[lpos]+loffset, label, c=color, fontsize=fsize) 196 | 197 | 198 | def plot_sign_histogram(membership, signs, trials): 199 | signs = np.array(signs, dtype='int32') 200 | mem, non_mem = np.zeros(trials + 1), np.zeros(trials + 1) 201 | mem_size, non_mem_size = sum(membership), len(membership) - sum(membership) 202 | for i in range(len(signs)): 203 | if membership[i] == 1: 204 | mem[int(signs[i] * trials)] += 1 205 | else: 206 | non_mem[int(signs[i] * trials)] += 1 207 | plt.plot(np.arange(0, 1.01, 0.01), mem / mem_size, 'k-', label='Members') 208 | plt.plot(np.arange(0, 1.01, 0.01), non_mem / non_mem_size, 'k--', label='Non Members') 209 | plt.xlabel('Merlin Ratio') 210 | plt.ylabel('Fraction of Instances') 211 | plt.xticks([0, 0.2, 0.4, 0.6, 0.8, 1.0]) 212 | plt.yticks(np.arange(0, 0.11, step=0.02)) 213 | plt.xlim(0, 1.0) 214 | plt.ylim(0, 0.1) 215 | plt.legend() 216 | plt.tight_layout() 217 | plt.show() 218 | 219 | 220 | def plot_histogram(vector): 221 | mem = vector[:10000] 222 | non_mem = vector[10000:] 223 | data, bins, _ = plt.hist([mem, non_mem], bins=loss_range()) 224 | plt.clf() 225 | mem_hist = np.array(data[0]) 226 | non_mem_hist = np.array(data[1]) 227 | plt.plot(bins[:-1], mem_hist / len(mem), 'k-', label='Members') 228 | plt.plot(bins[:-1], non_mem_hist / len(non_mem), 'k--', label='Non Members') 229 | plt.xscale('log') 230 | plt.xticks([10**-6, 10**-4, 10**-2, 10**0]) 231 | plt.yticks(np.arange(0, 0.11, step=0.02)) 232 | plt.ylim(0, 0.1) 233 | plt.xlabel('Per-Instance Loss') 234 | plt.ylabel('Fraction of Instances') 235 | plt.legend() 236 | plt.tight_layout() 237 | plt.show() 238 | 239 | 240 | def plot_ai_histogram(vector, positive_indices, positive_label, lab): 241 | #fig = plt.figure() 242 | pos = vector[positive_indices] 243 | neg = vector[list(set(range(len(vector))) - set(positive_indices))] 244 | data, bins, _ = plt.hist([pos, neg], bins=np.arange(0, 1.01, step=0.01)) 245 | plt.clf() 246 | pos_hist = np.array(data[0]) 247 | neg_hist = np.array(data[1]) 248 | plt.plot(bins[:-1], pos_hist / len(pos), '-m', label=positive_label) 249 | plt.plot(bins[:-1], neg_hist / len(neg), '-y', label='Not '+ positive_label) 250 | plt.xticks(np.arange(0, 1.1, step=0.2)) 251 | plt.yticks(np.arange(0, 0.11, step=0.02)) 252 | plt.ylim(0, 0.1) 253 | plt.xlabel('Probability of Predicting ' + positive_label) 254 | plt.ylabel('Fraction of Instances') 255 | plt.legend() 256 | plt.tight_layout() 257 | plt.show() 258 | #fig.savefig(str(lab) + ".pdf", format='pdf', dpi=1000, bbox_inches='tight') 259 | 260 | 261 | def plot_layer_outputs(plot_info, pos_label, neg_label, informative_neurons): 262 | pos_mean, pos_std, neg_mean, neg_std = plot_info 263 | x = list(range(len(informative_neurons))) 264 | plt.plot(x, neg_mean, '#DAA520', label=neg_label, lw=1) 265 | plt.fill_between(x, neg_mean - neg_std, neg_mean + neg_std, alpha=0.2, edgecolor='#DAA520', facecolor='#DAA520') 266 | plt.plot(x, pos_mean, '#DC143C', label=pos_label, lw=1) 267 | plt.fill_between(x, pos_mean - pos_std, pos_mean + pos_std, alpha=0.2, edgecolor='#DC143C', facecolor='#DC143C') 268 | plt.xticks(ticks = x, labels = [str(val) for val in informative_neurons]) 269 | plt.xlabel('Neuron Number') 270 | plt.ylabel('Neuron Output') 271 | plt.legend() 272 | plt.tight_layout() 273 | plt.show() 274 | 275 | 276 | def plot_neuron_outputs(plot_info, sorted_neurons, sorted_corr_vals, pos_label, neg_label): 277 | pos_mean, pos_std, neg_mean, neg_std = plot_info 278 | x = list(range(len(sorted_neurons))) 279 | fig, ax1 = plt.subplots() 280 | ax1.plot(x, neg_mean, '#DAA520', label=neg_label, lw=1) 281 | ax1.fill_between(x, neg_mean - neg_std, neg_mean + neg_std, alpha=0.2, edgecolor='#DAA520', facecolor='#DAA520') 282 | ax1.plot(x, pos_mean, '#DC143C', label=pos_label, lw=1) 283 | ax1.fill_between(x, pos_mean - pos_std, pos_mean + pos_std, alpha=0.2, edgecolor='#DC143C', facecolor='#DC143C') 284 | ax2 = ax1.twinx() 285 | plt.plot(x, sorted_corr_vals, 'k', label='correlation', lw=1) 286 | ax1.text(x[2], neg_mean[2] + 0.07, neg_label, c='#DAA520', fontsize=18) 287 | ax1.text(x[8], pos_mean[8] + 0.04, pos_label, c='#DC143C', fontsize=18) 288 | ax2.text(x[2], sorted_corr_vals[2] - 0.14, 'Pearson Correlation', c='k', fontsize=18) 289 | ax1.set_ylim(0, 1) 290 | ax2.set_ylim(0, 1) 291 | ax1.set_xlim(1, len(sorted_neurons)) 292 | ax1.set_xscale('log') 293 | ax1.set_xlabel('Neuron') 294 | ax1.set_ylabel('Scaled Neuron Output') 295 | ax2.set_ylabel('Correlation Value') 296 | fig.tight_layout() 297 | plt.show() 298 | 299 | 300 | def plot_regions(plot_info, skew_label, sensitive_label, dataset='census'): 301 | frame = plt.rcParams['figure.figsize'] 302 | plt.rcParams['figure.figsize'] = [1.5*frame[0], frame[1]]#[8, 4] 303 | if dataset == 'census': 304 | money_label = 'Average Income' 305 | region_type = 'PUMA' 306 | else: 307 | money_label = 'Average Charges' 308 | region_type = 'Hospital' 309 | region = [item[0] for item in plot_info] 310 | money = [item[1] for item in plot_info] 311 | skew_count = [item[2] for item in plot_info] 312 | sensitive_count = [item[3] for item in plot_info] 313 | total_count = [item[4] for item in plot_info] 314 | fig, ax1 = plt.subplots() 315 | ax2 = ax1.twinx() 316 | ax1.plot(range(len(region)), sensitive_count, 'b', alpha=0.5, label=sensitive_label) 317 | if sensitive_label != skew_label: 318 | ax1.plot(range(len(region)), skew_count, alpha=0.5, label=skew_label) 319 | ax1.plot(range(len(region)), total_count, 'orange', alpha=0.5, label='Population') 320 | ax2.plot(range(len(region)), np.array(money)/1000, 'r', alpha=0.5, label=money_label) 321 | ax1.set_xlabel('Region (' + region_type + ')') 322 | ax1.set_ylabel('Number of Records') 323 | ax2.set_ylabel(money_label + '(X $10^3$)') 324 | fig.tight_layout() 325 | ax1.legend(loc='upper right') 326 | ax2.legend(loc='upper left') 327 | plt.show() 328 | 329 | -------------------------------------------------------------------------------- /dataset/census.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bargavj/EvaluatingDPML/2f611fd20b4089019223039d95efa70a49c2b320/dataset/census.zip -------------------------------------------------------------------------------- /dataset/cifar_100_features.p: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bargavj/EvaluatingDPML/2f611fd20b4089019223039d95efa70a49c2b320/dataset/cifar_100_features.p -------------------------------------------------------------------------------- /dataset/cifar_100_labels.p: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bargavj/EvaluatingDPML/2f611fd20b4089019223039d95efa70a49c2b320/dataset/cifar_100_labels.p -------------------------------------------------------------------------------- /evaluating_dpml/README.md: -------------------------------------------------------------------------------- 1 | # Evaluating Differentially Private Machine Learning in Practice 2 | 3 | Please refer to [main README file](../README.md) for instructions on setup and installation. 4 | 5 | To replicate the results from the paper [*Evaluating Differentially Private Machine Learning in Practice*](https://arxiv.org/abs/1902.08874), you would need to execute the `run_experiments.sh` shell script, which runs the `main.py` multiple times for different hyper-parameter settings and stores the results in the `results/$DATASET` folder. This is used for plotting the figures/tables in the paper. Note that the execution takes 3-5 days to complete on a single machine. For instance, for CIFAR-100 data set, run the following command: 6 | ``` 7 | $ ./run_experiments.sh cifar_100 8 | ``` 9 | Note that the above script also installs all the required dependencies (in case not already installed), except cuda-toolkit and cudnn. For Purchase-100 data set, update the `target_l2_ratio` hyper-parameter as commented inside the script, and run: 10 | ``` 11 | $ ./run_experiments.sh purchase_100 12 | ``` 13 | 14 | As mentioned above, the enitre script execution takes several days to finish as it requires running `main.py` multiple times for all possible settings. This is required to generate all the tables/plots in the paper. However, we can also run `main.py` for specific settings, as explained below. 15 | 16 | When running the code on a data set for the first time, run: 17 | ``` 18 | $ python3 main.py cifar_100 --save_data=1 19 | ``` 20 | This will split the data set into random subsets for training and testing of target, shadow and attack models. 21 | 22 | To train a single non-private neural network model over CIFAR-100 data set, you can run: 23 | ``` 24 | $ python3 main.py cifar_100 --target_model='nn' --target_l2_ratio=1e-4 25 | ``` 26 | To train a single differentially private neural network model over CIFAR-100 data set using Rényi differential privacy with a privacy loss budget of 10, run: 27 | ``` 28 | $ python3 main.py cifar_100 --target_model='nn' --target_l2_ratio=1e-4 --target_privacy='grad_pert' --target_dp='rdp' --target_epsilon=10 29 | ``` 30 | 31 | 32 | ### Plotting the results from the paper 33 | 34 | Run `python3 interpret_results.py $DATASET --model=$MODEL --l2_ratio=$LAMBDA` to obtain the plots and tabular results. For instance, to get the results for neural network model over CIFAR-100 data set, run: 35 | ``` 36 | $ python3 interpret_results.py cifar_100 --model='nn' --l2_ratio=1e-4 37 | ``` 38 | 39 | Other command-line arguments are as follows: 40 | - `--function` prints the plots if set to 1 (default), or gives the membership revelation results at fixed FPR if set to 2, or gives the membership revelation results at fixed threshold if set to 3. 41 | - `--plot` specifies the type of plot to be printed 42 | - 'acc' prints the accuracy loss comparison plot (default) 43 | - 'shokri_mi' prints the privacy leakage due to Shokri et al. membership inference attack 44 | - 'yeom_mi' prints the privacy leakage due to Yeom et al. membership inference attack 45 | - 'yeom_ai' prints the privacy leakage due to Yeom et al. attribute inference attack 46 | - `--silent` specifies if the plot values are to be displayed (0) or not (1 - default) 47 | - `--fpr_threshold` sets the False Positive Rate threshold (refer the paper) 48 | - `--venn` plots the venn diagram of members identified by MI attack across two runs when set to 1, otherwise it does not plot when set to 0 (default). This functionality works only when `--function=3` 49 | 50 | -------------------------------------------------------------------------------- /evaluating_dpml/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bargavj/EvaluatingDPML/2f611fd20b4089019223039d95efa70a49c2b320/evaluating_dpml/__init__.py -------------------------------------------------------------------------------- /evaluating_dpml/interpret_results.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import pickle 3 | import numpy as np 4 | import matplotlib.pyplot as plt 5 | from scipy import stats 6 | from sklearn.metrics import roc_curve 7 | from sklearn.metrics import confusion_matrix 8 | from matplotlib_venn import venn3 9 | 10 | EPS = list(np.arange(0.01, 0.1, 0.01)) + list(np.arange(0.1, 1, 0.1)) 11 | EPSILONS = [0.01, 0.05, 0.1, 0.5, 1.0, 5.0, 10.0, 50.0, 100.0, 500.0, 1000.0] 12 | PERTURBATION = 'grad_pert_' 13 | DP = ['dp_', 'adv_cmp_', 'zcdp_', 'rdp_'] 14 | TYPE = ['o-', '.-', '^-', '--'] 15 | DP_LABELS = ['NC', 'AC', 'zCDP', 'RDP'] 16 | RUNS = range(5) 17 | 18 | plt.rcParams["font.family"] = "Times New Roman" 19 | plt.rcParams.update({'font.size': 15}) 20 | 21 | 22 | def theoretical_limit(epsilons): 23 | return [np.exp(eps) - 1 for eps in epsilons] 24 | 25 | 26 | def get_data(): 27 | result = {} 28 | for dp in DP: 29 | epsilons = {} 30 | for eps in EPSILONS: 31 | runs = {} 32 | for run in RUNS: 33 | runs[run] = list(pickle.load(open(DATA_PATH+MODEL+PERTURBATION+dp+str(eps)+'_'+str(run+1)+'.p', 'rb'))) 34 | epsilons[eps] = runs 35 | result[dp] = epsilons 36 | return result 37 | 38 | 39 | def pretty_position(X, Y, pos): 40 | return ((X[pos] + X[pos+1]) / 2, (Y[pos] + Y[pos+1]) / 2) 41 | 42 | 43 | def plot_advantage(result): 44 | train_acc, baseline_acc, train_loss, membership, _, shokri_mem_confidence, _, per_instance_loss, _, per_instance_loss_all, _ = pickle.load(open(DATA_PATH+MODEL+'no_privacy_'+str(args.l2_ratio)+'.p', 'rb')) 45 | print(train_acc, baseline_acc) 46 | color = 0.1 47 | y = dict() 48 | for dp in DP: 49 | test_acc_mean, yeom_mem_adv_mean, yeom_attr_adv_mean, shokri_mem_adv_mean = [], [], [], [] 50 | test_acc_std, yeom_mem_adv_std, yeom_attr_adv_std, shokri_mem_adv_std = [], [], [], [] 51 | for eps in EPSILONS: 52 | test_acc_d, yeom_mem_adv_d, yeom_attr_adv_d, shokri_mem_adv_d = [], [], [], [] 53 | for run in RUNS: 54 | train_acc, test_acc, train_loss, membership, shokri_mem_adv, shokri_mem_confidence, yeom_mem_adv, per_instance_loss, yeom_attr_adv, per_instance_loss, features = result[dp][eps][run] 55 | test_acc_d.append(test_acc) 56 | yeom_mem_adv_d.append(yeom_mem_adv) # adversary's advantage using membership inference attack of Yeom et al. 57 | shokri_mem_adv_d.append(shokri_mem_adv) # adversary's advantage using membership inference attack of Shokri et al. 58 | yeom_attr_adv_d.append(np.mean(yeom_attr_adv)) # adversary's advantage using attribute inference attack of Yeom et al. 59 | test_acc_mean.append(np.mean(test_acc_d)) 60 | test_acc_std.append(np.std(test_acc_d)) 61 | yeom_mem_adv_mean.append(np.mean(yeom_mem_adv_d)) 62 | yeom_mem_adv_std.append(np.std(yeom_mem_adv_d)) 63 | shokri_mem_adv_mean.append(np.mean(shokri_mem_adv_d)) 64 | shokri_mem_adv_std.append(np.std(shokri_mem_adv_d)) 65 | yeom_attr_adv_mean.append(np.mean(yeom_attr_adv_d)) 66 | yeom_attr_adv_std.append(np.std(yeom_attr_adv_d)) 67 | 68 | if args.silent == 0: 69 | if args.plot == 'acc': 70 | print(dp, eps, (baseline_acc - np.mean(test_acc_d)) / baseline_acc, np.std(test_acc_d)) 71 | elif args.plot == 'shokri_mi': 72 | print(dp, eps, np.mean(shokri_mem_adv_d), np.std(shokri_mem_adv_d)) 73 | elif args.plot == 'yeom_ai': 74 | print(dp, eps, np.mean(yeom_attr_adv_d), np.std(yeom_attr_adv_d)) 75 | elif args.plot == 'yeom_mi': 76 | print(dp, eps, np.mean(yeom_mem_adv_d), np.std(yeom_mem_adv_d)) 77 | if args.plot == 'acc': 78 | y[dp] = (baseline_acc - test_acc_mean) / baseline_acc 79 | plt.errorbar(EPSILONS, (baseline_acc - test_acc_mean) / baseline_acc, yerr=test_acc_std, color=str(color), fmt='.-', capsize=2, label=DP_LABELS[DP.index(dp)]) 80 | elif args.plot == 'shokri_mi': 81 | y[dp] = shokri_mem_adv_mean 82 | plt.errorbar(EPSILONS, shokri_mem_adv_mean, yerr=shokri_mem_adv_std, color=str(color), fmt='.-', capsize=2, label=DP_LABELS[DP.index(dp)]) 83 | elif args.plot == 'yeom_ai': 84 | y[dp] = yeom_attr_adv_mean 85 | plt.errorbar(EPSILONS, yeom_attr_adv_mean, yerr=yeom_attr_adv_std, color=str(color), fmt='.-', capsize=2, label=DP_LABELS[DP.index(dp)]) 86 | elif args.plot == 'yeom_mi': 87 | y[dp] = yeom_mem_adv_mean 88 | plt.errorbar(EPSILONS, yeom_mem_adv_mean, yerr=yeom_mem_adv_std, color=str(color), fmt='.-', capsize=2, label=DP_LABELS[DP.index(dp)]) 89 | color += 0.2 90 | 91 | plt.xscale('log') 92 | plt.xlabel('Privacy Budget ($\epsilon$)') 93 | 94 | if args.plot == 'acc': 95 | plt.ylabel('Accuracy Loss') 96 | plt.yticks(np.arange(0, 1.1, step=0.2)) 97 | else: 98 | bottom, top = plt.ylim() 99 | plt.errorbar(EPS, theoretical_limit(EPS), color='black', fmt='--', capsize=2, label='Theoretical Limit') 100 | plt.ylim(bottom, 0.25) 101 | plt.annotate("$\epsilon$-DP Bound", pretty_position(EPS, theoretical_limit(EPS), 9), textcoords="offset points", xytext=(5,0), ha='left') 102 | plt.yticks(np.arange(0, 0.26, step=0.05)) 103 | plt.ylabel('Privacy Leakage') 104 | 105 | plt.annotate("RDP", pretty_position(EPSILONS, y["rdp_"], 8), textcoords="offset points", xytext=(-10, 0), ha='right') 106 | plt.annotate("zCDP", pretty_position(EPSILONS, y["zcdp_"], 7), textcoords="offset points", xytext=(8, 12), ha='right') 107 | plt.annotate("AC", pretty_position(EPSILONS, y["adv_cmp_"], -4), textcoords="offset points", xytext=(0, -10), ha='left') 108 | plt.annotate("NC", pretty_position(EPSILONS, y["dp_"], -4), textcoords="offset points", xytext=(-10, 0), ha='right') 109 | 110 | plt.show() 111 | 112 | 113 | def members_revealed_fixed_fpr(result): 114 | thres = args.fpr_threshold# 0.01 == 1% FPR, 0.02 == 2% FPR, 0.05 == 5% FPR 115 | _, _, train_loss, membership, _, shokri_mem_confidence, _, per_instance_loss, _, per_instance_loss_all, _ = pickle.load(open(DATA_PATH+MODEL+'no_privacy_'+str(args.l2_ratio)+'.p', 'rb')) 116 | pred = (max(per_instance_loss) - per_instance_loss) / (max(per_instance_loss) - min(per_instance_loss)) 117 | #pred = shokri_mem_confidence[:,1] 118 | print(len(_members_revealed(membership, pred, thres))) 119 | for dp in DP: 120 | for eps in EPSILONS: 121 | mems_revealed = [] 122 | for run in RUNS: 123 | _, _, train_loss, membership, _, shokri_mem_confidence, _, per_instance_loss, _, per_instance_loss_all, _ = result[dp][eps][run] 124 | pred = (max(per_instance_loss) - per_instance_loss) / (max(per_instance_loss) - min(per_instance_loss)) 125 | #pred = shokri_mem_confidence[:,1] 126 | mems_revealed.append(_members_revealed(membership, pred, thres)) 127 | s = set.intersection(*mems_revealed) 128 | print(dp, eps, len(s)) 129 | 130 | 131 | def _members_revealed(membership, prediction, acceptable_fpr): 132 | fpr, tpr, thresholds = roc_curve(membership, prediction, pos_label=1) 133 | l = list(filter(lambda x: x < acceptable_fpr, fpr)) 134 | if len(l) == 0: 135 | print("Error: low acceptable fpr") 136 | return None 137 | threshold = thresholds[len(l)-1] 138 | preds = list(map(lambda val: 1 if val >= threshold else 0, prediction)) 139 | tp = [a*b for a,b in zip(preds,membership)] 140 | revealed = list(map(lambda i: i if tp[i] == 1 else None, range(len(tp)))) 141 | return set(list(filter(lambda x: x != None, revealed))) 142 | 143 | 144 | def get_ppv(mem, pred): 145 | tn, fp, fn, tp = confusion_matrix(mem, pred).ravel() 146 | return tp / (tp + fp) 147 | 148 | 149 | def ppv_across_runs(mem, pred): 150 | tn, fp, fn, tp = confusion_matrix(mem, np.where(pred >= 0, 1, 0)).ravel() 151 | print("0 or more") 152 | print(tp, fp, tp / (tp + fp)) 153 | tn, fp, fn, tp = confusion_matrix(mem, np.where(pred >= 1, 1, 0)).ravel() 154 | print("1 or more") 155 | print(tp, fp, tp / (tp + fp)) 156 | tn, fp, fn, tp = confusion_matrix(mem, np.where(pred >= 2, 1, 0)).ravel() 157 | print("2 or more") 158 | print(tp, fp, tp / (tp + fp)) 159 | tn, fp, fn, tp = confusion_matrix(mem, np.where(pred >= 3, 1, 0)).ravel() 160 | print("3 or more") 161 | print(tp, fp, tp / (tp + fp)) 162 | tn, fp, fn, tp = confusion_matrix(mem, np.where(pred >= 4, 1, 0)).ravel() 163 | print("4 or more") 164 | print(tp, fp, tp / (tp + fp)) 165 | tn, fp, fn, tp = confusion_matrix(mem, np.where(pred == 5, 1, 0)).ravel() 166 | print("exactly 5") 167 | print(tp, fp, tp / (tp + fp)) 168 | 169 | 170 | def generate_venn(mem, preds): 171 | run1 = preds[0] 172 | run2 = preds[1] 173 | 174 | run1_tp = [] 175 | run1_fp = [] 176 | run2_tp = [] 177 | run2_fp = [] 178 | bothpos = [] 179 | bothtruepos = [] 180 | 181 | for index in range(len(mem)): 182 | if mem[index] == 0: 183 | if run1[index] == 1: 184 | run1_fp += [index] 185 | if run2[index] == 1: 186 | run2_fp += [index] 187 | else: # mem(index) == 0 188 | if run1[index] == 1: 189 | run1_tp += [index] 190 | if run2[index] == 1: 191 | run2_tp += [index] 192 | 193 | run1pos = run1_fp + run1_tp 194 | run2pos = run2_fp + run2_tp 195 | 196 | for mem in run1pos: 197 | if mem in run2pos: 198 | bothpos += [mem] 199 | for mem in run1_tp: 200 | if mem in run2_tp: 201 | bothtruepos += [mem] 202 | 203 | s1 = len(run1_fp) 204 | s2 = len(run2_fp) 205 | s3 = len(bothpos) - len(bothtruepos) 206 | s4 = 0 207 | s5 = len(run1_tp) 208 | s6 = len(run2_tp) 209 | s7 = len(bothtruepos) 210 | 211 | venn3(subsets=(s1,s2,s3,s4,s5,s6,s7), set_labels=("Run 1", "Run 2", "TP")) 212 | plt.text(-0.70, 0.30, "FP") 213 | plt.text(0.61, 0.30, "FP") 214 | plt.show() 215 | 216 | 217 | def members_revealed_fixed_threshold(result): 218 | _, _, train_loss, membership, shokri_mem_adv, shokri_mem_confidence, yeom_mem_adv, per_instance_loss, yeom_attr_adv, per_instance_loss_all, _ = pickle.load(open(DATA_PATH+MODEL+'no_privacy_'+str(args.l2_ratio)+'.p', 'rb')) 219 | print(shokri_mem_adv, yeom_mem_adv, np.mean(yeom_attr_adv)) 220 | pred = np.where(per_instance_loss > train_loss, 0, 1) 221 | #pred = np.where(shokri_mem_confidence[:,1] <= 0.5, 0, 1) 222 | #attr_pred = np.array(per_instance_loss_all) 223 | #pred = np.where(stats.norm(0, train_loss).pdf(attr_pred[:,0,:]) >= stats.norm(0, train_loss).pdf(attr_pred[:,1,:]), 0, 1).ravel() 224 | tn, fp, fn, tp = confusion_matrix(membership, pred).ravel() 225 | print(tp, tp / (tp + fp)) 226 | fpr, tpr, thresholds = roc_curve(membership, pred, pos_label=1) 227 | print(fpr, tpr, np.max(tpr-fpr)) 228 | 229 | for dp in DP: 230 | for eps in EPSILONS: 231 | ppv, preds = [], [] 232 | for run in RUNS: 233 | _, _, train_loss, membership, _, shokri_mem_confidence, _, per_instance_loss, _, per_instance_loss_all, _ = result[dp][eps][run] 234 | pred = np.where(per_instance_loss > train_loss, 0, 1) 235 | preds.append(pred) 236 | #pred = np.where(shokri_mem_confidence[:,1] <= 0.5, 0, 1) 237 | #attr_pred = np.array(per_instance_loss_all) 238 | #pred = np.where(stats.norm(0, train_loss).pdf(attr_pred[:,0,:]) >= stats.norm(0, train_loss).pdf(attr_pred[:,1,:]), 0, 1).ravel() 239 | ppv.append(get_ppv(membership, pred)) 240 | print(dp, eps, np.mean(ppv)) 241 | sumpreds = np.sum(np.array(preds), axis=0) 242 | ppv_across_runs(membership, sumpreds) 243 | 244 | if args.venn == 1: 245 | generate_venn(membership, preds) 246 | 247 | 248 | if __name__ == '__main__': 249 | parser = argparse.ArgumentParser() 250 | parser.add_argument('dataset', type=str) 251 | parser.add_argument('--model', type=str, default='nn') 252 | parser.add_argument('--l2_ratio', type=float, default=1e-5) 253 | parser.add_argument('--function', type=int, default=1) 254 | parser.add_argument('--plot', type=str, default='acc') 255 | parser.add_argument('--fpr_threshold', type=float, default=0.01) 256 | parser.add_argument('--silent', type=int, default=1) 257 | parser.add_argument('--venn', type=int, default=0) 258 | args = parser.parse_args() 259 | print(vars(args)) 260 | 261 | DATA_PATH = '../results/' + str(args.dataset) + '/' 262 | MODEL = str(args.model) + '_' 263 | 264 | result = get_data() 265 | if args.function == 1: 266 | plot_advantage(result) # plot the utility and privacy loss graphs 267 | elif args.function == 2: 268 | members_revealed_fixed_fpr(result) # return the number of members revealed for different FPR rates 269 | else: 270 | members_revealed_fixed_threshold(result) 271 | -------------------------------------------------------------------------------- /evaluating_dpml/main.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | import argparse 4 | import pickle 5 | import numpy as np 6 | 7 | sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), "../"))) 8 | 9 | from core.attack import save_data 10 | from core.attack import load_data 11 | from core.attack import train_target_model 12 | from core.attack import yeom_membership_inference 13 | from core.attack import shokri_membership_inference 14 | from core.attack import yeom_attribute_inference 15 | from core.utilities import log_loss 16 | from core.utilities import get_random_features 17 | from sklearn.metrics import roc_curve 18 | 19 | RESULT_PATH = 'results/' 20 | 21 | if not os.path.exists(RESULT_PATH): 22 | os.makedirs(RESULT_PATH) 23 | 24 | 25 | def run_experiment(args): 26 | print('-' * 10 + 'TRAIN TARGET' + '-' * 10 + '\n') 27 | dataset = load_data('target_data.npz', args) 28 | train_x, train_y, test_x, test_y = dataset 29 | true_x = np.vstack((train_x, test_x)) 30 | true_y = np.append(train_y, test_y) 31 | batch_size = args.target_batch_size 32 | 33 | pred_y, membership, test_classes, classifier, aux = train_target_model( 34 | args=args, 35 | dataset=dataset, 36 | epochs=args.target_epochs, 37 | batch_size=args.target_batch_size, 38 | learning_rate=args.target_learning_rate, 39 | clipping_threshold=args.target_clipping_threshold, 40 | n_hidden=args.target_n_hidden, 41 | l2_ratio=args.target_l2_ratio, 42 | model=args.target_model, 43 | privacy=args.target_privacy, 44 | dp=args.target_dp, 45 | epsilon=args.target_epsilon, 46 | delta=args.target_delta, 47 | save=args.save_model) 48 | train_loss, train_acc, test_loss, test_acc = aux 49 | per_instance_loss = np.array(log_loss(true_y, pred_y)) 50 | 51 | features = get_random_features(true_x, range(true_x.shape[1]), 5) 52 | print(features) 53 | if not args.benchmark: 54 | # Yeom's membership inference attack when only train_loss is known 55 | pred_membership = yeom_membership_inference(per_instance_loss, membership, train_loss) 56 | fpr, tpr, thresholds = roc_curve(membership, pred_membership, pos_label=1) 57 | yeom_mem_adv = tpr[1] - fpr[1] 58 | 59 | # Shokri's membership inference attack based on shadow model training 60 | shokri_mi_outputs = shokri_membership_inference(args, pred_y, membership, test_classes) 61 | shokri_mem_adv, _, shokri_mem_confidence, _, _, _, _ = shokri_mi_outputs 62 | 63 | # Yeom's attribute inference attack when train_loss is known - Adversary 4 of Yeom et al. 64 | pred_membership_all = yeom_attribute_inference(true_x, true_y, classifier, membership, features, train_loss) 65 | yeom_attr_adv = [] 66 | for pred_membership in pred_membership_all: 67 | fpr, tpr, thresholds = roc_curve(membership, pred_membership, pos_label=1) 68 | yeom_attr_adv.append(tpr[1] - fpr[1]) 69 | 70 | if not os.path.exists(RESULT_PATH+args.train_dataset): 71 | os.makedirs(RESULT_PATH+args.train_dataset) 72 | 73 | if args.target_privacy == 'no_privacy': 74 | pickle.dump([train_acc, test_acc, train_loss, membership, shokri_mem_adv, shokri_mem_confidence, yeom_mem_adv, per_instance_loss, yeom_attr_adv, pred_membership_all, features], open(RESULT_PATH+args.train_dataset+'/'+args.target_model+'_no_privacy_'+str(args.target_l2_ratio)+'.p', 'wb')) 75 | else: 76 | pickle.dump([train_acc, test_acc, train_loss, membership, shokri_mem_adv, shokri_mem_confidence, yeom_mem_adv, per_instance_loss, yeom_attr_adv, pred_membership_all, features], open(RESULT_PATH+args.train_dataset+'/'+args.target_model+'_'+args.target_privacy+'_'+args.target_dp+'_'+str(args.target_epsilon)+'_'+str(args.run)+'.p', 'wb')) 77 | 78 | 79 | if __name__ == '__main__': 80 | parser = argparse.ArgumentParser() 81 | parser.add_argument('train_dataset', type=str) 82 | parser.add_argument('--run', type=int, default=1) 83 | parser.add_argument('--use_cpu', type=int, default=0) 84 | parser.add_argument('--save_model', type=int, default=0) 85 | parser.add_argument('--save_data', type=int, default=0) 86 | # target and shadow model configuration 87 | parser.add_argument('--n_shadow', type=int, default=5) 88 | parser.add_argument('--target_data_size', type=int, default=int(1e4)) 89 | parser.add_argument('--target_test_train_ratio', type=int, default=1.0) 90 | parser.add_argument('--target_model', type=str, default='nn') 91 | parser.add_argument('--target_learning_rate', type=float, default=0.01) 92 | parser.add_argument('--target_batch_size', type=int, default=200) 93 | parser.add_argument('--target_n_hidden', type=int, default=256) 94 | parser.add_argument('--target_epochs', type=int, default=100) 95 | parser.add_argument('--target_l2_ratio', type=float, default=1e-8) 96 | parser.add_argument('--target_clipping_threshold', type=float, default=1.0) 97 | parser.add_argument('--target_privacy', type=str, default='no_privacy') 98 | parser.add_argument('--target_dp', type=str, default='dp') 99 | parser.add_argument('--target_epsilon', type=float, default=0.5) 100 | parser.add_argument('--target_delta', type=float, default=1e-5) 101 | # attack model configuration 102 | parser.add_argument('--attack_model', type=str, default='nn') 103 | parser.add_argument('--attack_learning_rate', type=float, default=0.01) 104 | parser.add_argument('--attack_batch_size', type=int, default=100) 105 | parser.add_argument('--attack_n_hidden', type=int, default=64) 106 | parser.add_argument('--attack_epochs', type=int, default=100) 107 | parser.add_argument('--attack_l2_ratio', type=float, default=1e-6) 108 | parser.add_argument('--benchmark',type=int,default=0) 109 | 110 | # parse configuration 111 | args = parser.parse_args() 112 | print(vars(args)) 113 | 114 | # Flag to disable GPU 115 | if args.use_cpu: 116 | os.environ['CUDA_VISIBLE_DEVICES'] = '-1' 117 | else: 118 | os.environ['TF_XLA_FLAGS'] = '--tf_xla_enable_xla_devices' 119 | 120 | if args.save_data: 121 | save_data(args) 122 | else: 123 | run_experiment(args) 124 | -------------------------------------------------------------------------------- /evaluating_dpml/run_experiments.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "Starting script" 4 | if [[ $# -eq 0 ]] ; then # if called with no arguments 5 | echo "Usage: bash $0 " 6 | exit 1 7 | fi 8 | 9 | DATASET=$1 10 | CODE=main.py 11 | 12 | echo "Loading modules" 13 | source /etc/profile.d/modules.sh 14 | module load anaconda3 15 | 16 | # Make sure conda environment has dependencies 17 | echo "Creating conda environment" 18 | conda create -n tf-gpu tensorflow-gpu 19 | source activate tf-gpu 20 | conda install scikit-learn 21 | 22 | pip install dm-tree 23 | pip install matplotlib 24 | pip install git+git://github.com/tensorflow/privacy@master 25 | 26 | echo "Filling data/ directory" 27 | python $CODE $DATASET --save_data=1 28 | 29 | echo "Beginning experiment" 30 | # For Purchase-100 and 'nn' model, set --target_l2_ratio=1e-8 31 | python $CODE $DATASET --target_model='softmax' --target_l2_ratio=1e-5 32 | python $CODE $DATASET --target_model='nn' --target_l2_ratio=1e-4 33 | for RUN in 1 2 3 4 5 34 | do 35 | for EPSILON in 0.1 0.5 1.0 5.0 10.0 50.0 100.0 500.0 1000.0 36 | do 37 | for DP in 'dp' 'adv_cmp' 'rdp' 'zcdp' 38 | do 39 | python $CODE $DATASET --target_model='softmax' --target_l2_ratio=1e-5 --target_privacy='grad_pert' --target_dp=$DP --target_epsilon=$EPSILON --run=$RUN 40 | python $CODE $DATASET --target_model='nn' --target_l2_ratio=1e-4 --target_privacy='grad_pert' --target_dp=$DP --target_epsilon=$EPSILON --run=$RUN 41 | done 42 | done 43 | done 44 | echo done -------------------------------------------------------------------------------- /extra/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bargavj/EvaluatingDPML/2f611fd20b4089019223039d95efa70a49c2b320/extra/__init__.py -------------------------------------------------------------------------------- /extra/combine_traces.py: -------------------------------------------------------------------------------- 1 | import json 2 | from glob import glob 3 | 4 | # directory containing the timeline-*.json partial traces 5 | dir = '../log' 6 | 7 | output = dict() # json object that will be written out 8 | output['traceEvents'] = [] 9 | labels = set() 10 | for trace in glob(f'{dir}/timeline-*.json'): 11 | with open(trace) as f: 12 | for node in json.loads(f.read())['traceEvents']: 13 | if 'dur' in node: # keep nodes that have a "dur"ation 14 | output['traceEvents'].append(node) 15 | elif node['name'] == 'process_name': 16 | labels.add(tuple(node)) 17 | # ignore all others, I can't tell what they're for 18 | 19 | # convert sets to lists for JSON printing 20 | output['traceEvents'] = [list(a) for a in labels] + output['traceEvents'] 21 | with open('traceEvents.json', 'w') as f: 22 | f.write(json.dumps(output)) 23 | 24 | # To view the trace: 25 | # - Go to "chrome://tracing" in Google Chrome or Chromium 26 | # - Click load 27 | # - Navigate to ./traceEvents.json 28 | -------------------------------------------------------------------------------- /extra/crawl_census_data.py: -------------------------------------------------------------------------------- 1 | from bs4 import BeautifulSoup 2 | import zipfile 3 | import requests 4 | import wget 5 | from os import listdir,makedirs 6 | from os.path import isfile, join,isdir 7 | import argparse 8 | 9 | global dataFolder, url,blackListFiles 10 | #data download page 11 | url = "https://www2.census.gov/programs-surveys/acs/data/pums/2019/1-Year/" 12 | dataFolder = "../dataset/census/" 13 | blackListFiles = [] 14 | targetFiles = ["csv_pus.zip"] 15 | 16 | 17 | def isFilePresent(fileName): 18 | """Returns whether a given data file found on the download web page is already present in the local folder. 19 | Also checks if the file is in the file blacklist (certain unwanted files) 20 | """ 21 | global dataFolder,blackListFiles 22 | allDataFiles = [f for f in listdir(dataFolder) if (isfile(join(dataFolder, f)) and f.endswith('.zip'))] 23 | return fileName in allDataFiles and not (fileName in blackListFiles) 24 | 25 | 26 | def crawl_data(): 27 | print("### Starting to crawl data.") 28 | global dataFolder, url 29 | #create census dataset folder if not exists 30 | if not isdir(dataFolder): 31 | makedirs(dataFolder) 32 | r = requests.get(url) 33 | data = r.text 34 | soup = BeautifulSoup(data,"lxml") 35 | downloadZips = [] 36 | for link in soup.find_all('a'): 37 | href = str(link.get('href')) 38 | #if the data file fits the desired format and it's not already downloaded, then download the file. 39 | if filePatternTest(href) and not (isFilePresent(href)): 40 | downloadZips.append(href) 41 | for href in downloadZips: 42 | downloadLink = url+href 43 | #the wget might fail. try until the file is downloaded sucessfully 44 | while True: 45 | try: 46 | wget.download(downloadLink,out = dataFolder) 47 | except: 48 | continue 49 | else: 50 | break 51 | print() 52 | print("### Finished crawling data.") 53 | return 0 54 | 55 | 56 | def processZips(): 57 | print("### Starting to unzip data.") 58 | global dataFolder 59 | allDataFiles = [f for f in listdir(dataFolder) if (isfile(join(dataFolder, f)) and f.endswith('.zip'))] 60 | for file in allDataFiles: 61 | zipFileDir = dataFolder+file 62 | print(zipFileDir) 63 | with zipfile.ZipFile(zipFileDir, 'r') as zip_ref: 64 | zip_ref.extractall(dataFolder) 65 | print("### Finished unzipping data.") 66 | 67 | 68 | def filePatternTest(string:str): 69 | #change this condition for finding data download files that are not the census data. 70 | # condition = string.endswith(".zip") and string.startswith("csv_p") 71 | condition = string in targetFiles 72 | return condition 73 | 74 | 75 | if __name__ == '__main__': 76 | parser = argparse.ArgumentParser() 77 | parser.add_argument('--target_census_data',type=str,default='1year') 78 | args = parser.parse_args() 79 | if args.target_census_data=='5year': 80 | url = "https://www2.census.gov/programs-surveys/acs/data/pums/2019/5-Year/" 81 | 82 | statusCode = crawl_data() 83 | if statusCode==0: 84 | processZips() 85 | -------------------------------------------------------------------------------- /improved_ai/README.md: -------------------------------------------------------------------------------- 1 | # Are Attribute Inference Attacks Just Imputation? 2 | 3 | Please refer to [main README file](../README.md) for instructions on setup and installation. 4 | 5 | To replicate the results of the paper [*Are Attribute Inference Attacks Just Imputation?*](https://arxiv.org/abs/2209.01292), you would need to execute the `run_experiments.sh` shell script, which runs the `main.py` multiple times for different hyper-parameter settings and stores the results in the `results/$DATASET` folder. This is used for plotting the figures/tables in the paper. Note that the execution may take multiple days to complete on a single machine. For instance, for Census19 data set, run the following command: 6 | ``` 7 | $ ./run_experiments.sh -s census 8 | ``` 9 | Note that the above script also installs all the required dependencies (in case not already installed), except cuda-toolkit and cudnn. For Texas-100X data set, update the hyper-parameters as commented inside the script, and run: 10 | ``` 11 | $ ./run_experiments.sh -s texas_100_v2 12 | ``` 13 | 14 | 15 | ### Plotting the results from the paper 16 | 17 | Run `interpret_results.py` to obtain the plots and tabular results. For instance, to get the results for Census19 data set, run: 18 | ``` 19 | $ python3 interpret_results.py census --skew_attribute=0 --attribute=1 --skew_outcome=3 --sensitive_outcome=3 --adv_knowledge='med' --sample_size=50 20 | ``` 21 | 22 | Other command-line arguments are as follows: 23 | - `--skew_attribute`: Attribute on which to skew the non-iid data sampling 0 (population, default), 1 or 2 -- for Census 1: Income and 2: Race, and for Texas 1: Charges and 2: Ethnicity 24 | - `--attribute`: Target sensitive attribute. For Census19: 1 (Race), and for Texas-100X: 2 (Ethnicity) 25 | - `--skew_outcome`: In case skew_attribute = 1, which outcome the distribution is skewed upon -- For Census Race: 0 (White, default), 1 (Black) or 3 (Asian), and for Texas Ethnicity: 0 (Hispanic, default) or 1 (Not Hispanic) 26 | - `--sensitive_outcome`: For Census Race: 0 (White, default), 1 (Black) or 3 (Asian), and for Texas Ethnicity: 0 (Hispanic, default) or 1 (Not Hispanic) 27 | - `--eps`: Specifies the epsilon value to be used for private model (default = None), use eps=1 for private model results 28 | - `--adv_knowledge`: Distribution knowledge of adversary: knows skewed distribution (low or low2), knows training distribution (med) or knows training data set (high) 29 | - `--sample_size`: How many records the adversary knows: 50, 500, 5000 or 50000 30 | - `--banished_records`: Whether to get results from model trained after removing vulnerable records 31 | -------------------------------------------------------------------------------- /improved_ai/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bargavj/EvaluatingDPML/2f611fd20b4089019223039d95efa70a49c2b320/improved_ai/__init__.py -------------------------------------------------------------------------------- /improved_ai/interpret_results.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | import argparse 4 | import pickle 5 | import itertools 6 | import random 7 | import numpy as np 8 | import matplotlib.pyplot as plt 9 | 10 | sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), "../"))) 11 | 12 | from core.data_util import get_sensitive_features 13 | from core.utilities import plot_layer_outputs 14 | from core.utilities import get_inference_threshold 15 | from core.utilities import get_ppvs 16 | from core.utilities import fit_model 17 | from core.utilities import make_line_plot 18 | from core.attack import yeom_membership_inference 19 | from sklearn.metrics import confusion_matrix 20 | from sklearn.metrics import roc_curve 21 | from collections import Counter 22 | 23 | frame = plt.rcParams['figure.figsize'] 24 | new_rc_params = { 25 | 'figure.figsize': [6,5], # [6,4] for main figures and [6,5] for appendix figures 26 | 'figure.autolayout': True, 27 | 'font.size': 20, # 16 for main figures and 20 for appendix figures 28 | #'text.usetex': True, 29 | 'font.family': 'serif', 30 | 'font.serif': 'Times New Roman', 31 | 'mathtext.fontset': 'stix', 32 | 'xtick.major.pad': '8' 33 | } 34 | plt.rcParams.update(new_rc_params) 35 | fsize = 18 # 16 for main figures and 18 for appendix figures 36 | 37 | RESULT_PATH = 'results/' 38 | T = 5 39 | RUNS = range(T) 40 | THREAT_MODEL = ['low', 'low2', 'med', 'high'] 41 | SAMPLE_SIZE = [50, 500, 5000, 50000] 42 | 43 | safe_colors = {1: '#029386', 2: '#FC5A50', 3: '#DC143C', 4: '#800000', 5: '#FF1493', 6: '#9400D3', 7: '#4B0082', 8: '#222222'} 44 | 45 | 46 | def get_result(args): 47 | MODEL = str(args.skew_attribute) + '_' + str(args.skew_outcome) + '_' + str(args.gamma) + '_' + str(args.model) + '_' 48 | result = { 49 | 'no_privacy': { 50 | knw: { 51 | s_size: { 52 | run: pickle.load(open(RESULT_PATH + args.dataset + '/' + MODEL + 'no_privacy_' + str(args.attribute) + '_' + knw + '_' + str(s_size) + '_0_' + str(run) + '.p', 'rb')) for run in RUNS 53 | } for s_size in SAMPLE_SIZE 54 | } for knw in THREAT_MODEL 55 | } 56 | } 57 | if args.eps != None: 58 | result[args.eps] = { 59 | knw: { 60 | s_size: { 61 | run: pickle.load(open(RESULT_PATH + args.dataset + '/' + MODEL + 'grad_pert_' + args.dp + '_' + str(args.eps) + '_' + str(args.attribute) + '_' + knw + '_' + str(s_size) + '_0_' + str(run) + '.p', 'rb')) for run in RUNS 62 | } for s_size in SAMPLE_SIZE 63 | } for knw in THREAT_MODEL 64 | } 65 | if args.banished_records == 1: 66 | result['banished'] = { 67 | knw: { 68 | s_size: { 69 | run: pickle.load(open(RESULT_PATH + args.dataset + '/' + MODEL + 'no_privacy_' + str(args.attribute) + '_' + knw + '_' + str(s_size) + '_1_' + str(run) + '.p', 'rb')) for run in RUNS 70 | } for s_size in SAMPLE_SIZE 71 | } for knw in THREAT_MODEL 72 | } 73 | return result 74 | 75 | 76 | def plot_ppvs(args, gt, wb, mc, ip, clfs, plot_cond, data_type=0): 77 | """ 78 | args: runtime arguments obtained via main function 79 | gt: ground truth value for sensitive attribute 80 | wb: white-box attack prediction vector -- this scaled output of neurons 81 | mc: black-box attack prediction vector -- prediction confidence of model 82 | ip: imputation model confidence in predicting sensitive attribute 83 | clfs: meta-classifier that combines wb and mc 84 | plot_cond: flags that describe how and what lines to plot 85 | data_type: 0 (default) -- train data, 1 -- test data, 2: hold-out data 86 | """ 87 | clfs_wb, clfs_bb = clfs 88 | ppv_ip = np.array([get_ppvs(gt[run][data_type], ip[run][data_type]) for run in RUNS]) 89 | ppv_wb = np.array([get_ppvs(gt[run][data_type], wb[run][data_type]) for run in RUNS]) 90 | ppv_wb_c0 = np.array([get_ppvs(gt[run][data_type], ip[run][data_type]*wb[run][data_type]) for run in RUNS]) 91 | if args.comb_flag == 1: 92 | ppv_wb_c1 = np.array([get_ppvs(gt[run][data_type], clfs_wb[run].predict_proba(np.vstack((ip[run][data_type], wb[run][data_type], ip[run][data_type] * wb[run][data_type])).T)[:, 1]) for run in RUNS]) 93 | print('k in WB scatter plot: %d' % sum(clfs_wb[data_type].predict(np.vstack((ip[0][data_type], wb[0][data_type], ip[0][data_type] * wb[0][data_type])).T))) 94 | else: 95 | ppv_wb_c1 = np.array([get_ppvs(gt[run][data_type], clfs_wb[run].predict_proba(np.vstack((ip[run][data_type], wb[run][data_type])).T)[:, 1]) for run in RUNS]) 96 | print('k in WB scatter plot: %d' % sum(clfs_wb[data_type].predict(np.vstack((ip[0][data_type], wb[0][data_type])).T))) 97 | 98 | ppv_bb = np.array([get_ppvs(gt[run][data_type], mc[run][data_type]) for run in RUNS]) 99 | ppv_bb_c0 = np.array([get_ppvs(gt[run][data_type], ip[run][data_type]*mc[run][data_type]) for run in RUNS]) 100 | if args.comb_flag == 1: 101 | ppv_bb_c1 = np.array([get_ppvs(gt[run][data_type], clfs_bb[run].predict_proba(np.vstack((ip[run][data_type], mc[run][data_type], ip[run][data_type] * mc[run][data_type])).T)[:, 1]) for run in RUNS]) 102 | print('k in BB scatter plot: %d' % sum(clfs_bb[data_type].predict(np.vstack((ip[0][data_type], mc[0][data_type], ip[0][data_type] * mc[0][data_type])).T))) 103 | else: 104 | ppv_bb_c1 = np.array([get_ppvs(gt[run][data_type], clfs_bb[run].predict_proba(np.vstack((ip[run][data_type], mc[run][data_type])).T)[:, 1]) for run in RUNS]) 105 | print('k in BB scatter plot: %d' % sum(clfs_bb[data_type].predict(np.vstack((ip[0][data_type], mc[0][data_type])).T))) 106 | 107 | print('Random\t%.2f +/- %.2f' % (np.mean([sum(gt[run][data_type]) for run in RUNS]), np.std([sum(gt[run][data_type]) for run in RUNS]))) 108 | print('\t Top-10 \t Top-50 \t Top-100') 109 | print('IP\t%.2f +/- %.2f\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(ppv_ip[:, 9]), np.std(ppv_ip[:, 9]), np.mean(ppv_ip[:, 49]), np.std(ppv_ip[:, 49]), np.mean(ppv_ip[:, 99]), np.std(ppv_ip[:, 99]))) 110 | print('BB\t%.2f +/- %.2f\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(ppv_bb[:, 9]), np.std(ppv_bb[:, 9]), np.mean(ppv_bb[:, 49]), np.std(ppv_bb[:, 49]), np.mean(ppv_bb[:, 99]), np.std(ppv_bb[:, 99]))) 111 | print('BB.IP\t%.2f +/- %.2f\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(ppv_bb_c0[:, 9]), np.std(ppv_bb_c0[:, 9]), np.mean(ppv_bb_c0[:, 49]), np.std(ppv_bb_c0[:, 49]), np.mean(ppv_bb_c0[:, 99]), np.std(ppv_bb_c0[:, 99]))) 112 | print('BBXIP\t%.2f +/- %.2f\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(ppv_bb_c1[:, 9]), np.std(ppv_bb_c1[:, 9]), np.mean(ppv_bb_c1[:, 49]), np.std(ppv_bb_c1[:, 49]), np.mean(ppv_bb_c1[:, 99]), np.std(ppv_bb_c1[:, 99]))) 113 | print('WB\t%.2f +/- %.2f\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(ppv_wb[:, 9]), np.std(ppv_wb[:, 9]), np.mean(ppv_wb[:, 49]), np.std(ppv_wb[:, 49]), np.mean(ppv_wb[:, 99]), np.std(ppv_wb[:, 99]))) 114 | print('WB.IP\t%.2f +/- %.2f\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(ppv_wb_c0[:, 9]), np.std(ppv_wb_c0[:, 9]), np.mean(ppv_wb_c0[:, 49]), np.std(ppv_wb_c0[:, 49]), np.mean(ppv_wb_c0[:, 99]), np.std(ppv_wb_c0[:, 99]))) 115 | print('WBXIP\t%.2f +/- %.2f\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(ppv_wb_c1[:, 9]), np.std(ppv_wb_c1[:, 9]), np.mean(ppv_wb_c1[:, 49]), np.std(ppv_wb_c1[:, 49]), np.mean(ppv_wb_c1[:, 99]), np.std(ppv_wb_c1[:, 99]))) 116 | 117 | X = list(range(1, args.candidate_size + 1)) 118 | if plot_cond['ip']['flag']: 119 | make_line_plot(X, ppv_ip, '-', safe_colors[1], 'IP', plot_cond['ip']['xoffset'], plot_cond['ip']['yoffset'], fsize=fsize) 120 | if plot_cond['wb']['flag']: 121 | make_line_plot(X, ppv_wb, '-', safe_colors[2], 'WB', plot_cond['wb']['xoffset'], plot_cond['wb']['yoffset'], fsize=fsize) 122 | if plot_cond['wb.ip']['flag']: 123 | make_line_plot(X, ppv_wb_c0, '-', safe_colors[3], r'WB$\cdot$IP', plot_cond['wb.ip']['xoffset'], plot_cond['wb.ip']['yoffset'], fsize=fsize) 124 | if plot_cond['wbXip']['flag']: 125 | make_line_plot(X, ppv_wb_c1, '-', safe_colors[4], r'WB$\diamondsuit$IP', plot_cond['wbXip']['xoffset'], plot_cond['wbXip']['yoffset'], fsize=fsize) 126 | if plot_cond['bb']['flag']: 127 | make_line_plot(X, ppv_bb, '-', safe_colors[5], 'BB', plot_cond['bb']['xoffset'], plot_cond['bb']['yoffset'], fsize=fsize) 128 | if plot_cond['bb.ip']['flag']: 129 | make_line_plot(X, ppv_bb_c0, '-', safe_colors[6], r'BB$\cdot$IP', plot_cond['bb.ip']['xoffset'], plot_cond['bb.ip']['yoffset'], fsize=fsize) 130 | if plot_cond['bbXip']['flag']: 131 | make_line_plot(X, ppv_bb_c1, '-', safe_colors[7], r'BB$\diamondsuit$IP', plot_cond['bbXip']['xoffset'], plot_cond['bbXip']['yoffset'], fsize=fsize) 132 | if plot_cond['baseline']['flag']: 133 | make_line_plot(X, np.array([np.ones(len(X)) * sum(gt[run][data_type]) / len(X) for run in RUNS]), '-.', safe_colors[8], 'Random Guess', plot_cond['baseline']['xoffset'], plot_cond['baseline']['yoffset'], fsize=fsize) 134 | plt.xlabel('Top-k records') 135 | plt.ylabel('PPV') 136 | plt.xscale('log') 137 | plt.xlim(1, args.candidate_size) 138 | plt.xticks([1, 1e1, 1e2, 1e3, 1e4]) 139 | plt.yticks([0, 0.2, 0.4, 0.6, 0.8, 1]) 140 | plt.ylim(0, 1) 141 | plt.tight_layout() 142 | plt.show() 143 | 144 | 145 | def plot_ppv_change(args, gt, wb, mc, ip, clfs, plot_cond): 146 | wb_old = {run: {idx: [] for idx in range(3)} for run in RUNS} 147 | for run in RUNS: 148 | sensitive_test, _, _, whitebox_info, _, _, _ = result['no_privacy']['low'][run] 149 | whitebox_info_k_1, whitebox_info_k_2, whitebox_info_k_5, whitebox_info_k_10 = whitebox_info 150 | for idx, args.candidate_indices in zip([0, 1, 2], [range(args.candidate_size), range(args.candidate_size, 2*args.candidate_size), range(2*args.candidate_size, len(sensitive_test))]): 151 | wb_old[run][idx] = whitebox_info_k_10[candidate_indices, args.sensitive_outcome] 152 | 153 | clfs_wb_old = [fit_model(args, gt[run][2], ip[run][2], wb_old[run][2]) for run in RUNS] 154 | ppv_wb1_old = np.array([get_ppvs(gt[run][0], wb_old[run][0]) for run in RUNS]) 155 | ppv_wb1_c0_old = np.array([get_ppvs(gt[run][0], ip[run][0]*wb_old[run][0]) for run in RUNS]) 156 | ppv_wb1_c1_old = np.array([get_ppvs(gt[run][0], clfs_wb_old[run].predict_proba(np.vstack((ip[run][0], wb_old[run][0])).T)[:, 1]) for run in RUNS]) 157 | 158 | clfs_wb, clfs_bb = clfs 159 | ppv_ip1 = np.array([get_ppvs(gt[run][0], ip[run][0]) for run in RUNS]) 160 | ppv_wb1 = np.array([get_ppvs(gt[run][0], wb[run][0]) for run in RUNS]) 161 | ppv_wb1_c0 = np.array([get_ppvs(gt[run][0], ip[run][0]*wb[run][0]) for run in RUNS]) 162 | ppv_wb1_c1 = np.array([get_ppvs(gt[run][0], clfs_wb[run].predict_proba(np.vstack((ip[run][0], wb[run][0])).T)[:, 1]) for run in RUNS]) 163 | 164 | X = list(range(1, args.candidate_size + 1)) 165 | make_line_plot(X, ppv_wb1 - ppv_wb1_old, '-', safe_colors[2], 'Whitebox Attack (WB)', plot_cond['wb']['xoffset'], plot_cond['wb']['yoffset'], fsize=fsize) 166 | make_line_plot(X, ppv_wb1_c0 - ppv_wb1_c0_old, '-', safe_colors[3], r'WB$\cdot$IP', plot_cond['wb.ip']['xoffset'], plot_cond['wb.ip']['yoffset'], fsize=fsize) 167 | make_line_plot(X, ppv_wb1_c1 - ppv_wb1_c1_old, '-', safe_colors[4], r'WB$\diamondsuit$IP', plot_cond['wbXip']['xoffset'], plot_cond['wbXip']['yoffset'], fsize=fsize) 168 | make_line_plot(X, np.array([np.zeros(len(X)) for run in RUNS]), '-.', safe_colors[8], 'No Change', plot_cond['baseline']['xoffset'], plot_cond['baseline']['yoffset'], fsize=fsize) 169 | plt.xlabel('Top-k records') 170 | plt.ylabel('Change in PPV') 171 | plt.xscale('log') 172 | plt.xlim(1, args.candidate_size) 173 | plt.xticks([1, 1e1, 1e2, 1e3, 1e4]) 174 | plt.yticks([-1, -0.5, 0, 0.5, 1]) 175 | plt.ylim(-1, 1) 176 | plt.tight_layout() 177 | plt.show() 178 | 179 | 180 | def plot_roc(args, gt, wb, mc, ip): 181 | mean_fpr = np.linspace(0, 1, args.candidate_size) 182 | tpr_ip1, tpr_ip2, tpr_wb1, tpr_wb2, tpr_bb1, tpr_bb2 = [], [], [], [], [], [] 183 | for run in RUNS: 184 | fpr, tpr, _ = roc_curve(gt[run][0], ip[run][0], pos_label=1) 185 | tpr_ip1.append(np.interp(mean_fpr, fpr, tpr)) 186 | fpr, tpr, _ = roc_curve(gt[run][1], ip[run][1], pos_label=1) 187 | tpr_ip2.append(np.interp(mean_fpr, fpr, tpr)) 188 | fpr, tpr, _ = roc_curve(gt[run][0], wb[run][0], pos_label=1) 189 | tpr_wb1.append(np.interp(mean_fpr, fpr, tpr)) 190 | fpr, tpr, _ = roc_curve(gt[run][1], wb[run][1], pos_label=1) 191 | tpr_wb2.append(np.interp(mean_fpr, fpr, tpr)) 192 | fpr, tpr, _ = roc_curve(gt[run][0], mc[run][0], pos_label=1) 193 | tpr_bb1.append(np.interp(mean_fpr, fpr, tpr)) 194 | fpr, tpr, _ = roc_curve(gt[run][1], mc[run][1], pos_label=1) 195 | tpr_bb2.append(np.interp(mean_fpr, fpr, tpr)) 196 | 197 | make_line_plot(mean_fpr, tpr_ip1, '-', safe_colors[1], 'Imputation on Train', fsize=fsize) 198 | make_line_plot(mean_fpr, tpr_ip2, '--', safe_colors[1], 'Imputation on Test', fsize=fsize) 199 | 200 | make_line_plot(mean_fpr, tpr_wb1, '-', safe_colors[2], 'WhiteBox AI on Train', fsize=fsize) 201 | make_line_plot(mean_fpr, tpr_wb2, '--', safe_colors[2], 'WhiteBox AI on Test', fsize=fsize) 202 | 203 | make_line_plot(mean_fpr, tpr_bb1, '-', safe_colors[5], 'BlackBox AI on Train', fsize=fsize) 204 | make_line_plot(mean_fpr, tpr_bb2, '--', safe_colors[5], 'BlackBox AI on Test', fsize=fsize) 205 | 206 | plt.xlabel('FPR') 207 | plt.ylabel('TPR') 208 | plt.xscale('log') 209 | plt.yscale('log') 210 | plt.xlim(1 / args.candidate_size, 1) 211 | plt.ylim(1 / args.candidate_size, 1) 212 | plt.legend() 213 | plt.tight_layout() 214 | plt.show() 215 | 216 | 217 | def scatter_plot(args, gt, wb, mc, ip, clfs, it): 218 | N = 5 219 | clfs_wb, clfs_bb = clfs 220 | plt.rcParams['figure.figsize'] = [N*3.50, 3.50] 221 | xx, yy = np.meshgrid(np.arange(0, 1.01, 0.02), np.arange(0, 1.01, 0.02)) 222 | 223 | if args.dataset == 'census': 224 | ip_low, ip_high, wb_low = 0.28, 0.5, 0.5 225 | else: 226 | ip_low, ip_high, wb_low = 0.0, 0.3, 0.9 227 | 228 | vul_idxs = [] 229 | tpls = np.zeros((N, 3)) 230 | f, ax = plt.subplots(1, N, sharey=True, squeeze=False) 231 | ax[0][0].set_ylabel('Imputation Confidence') 232 | for i in range(N): 233 | vul_idx = list(filter(lambda idx: (ip[i][it][idx] > ip_low) and (ip[i][it][idx] < ip_high) and (wb[i][it][idx] > wb_low), range(len(gt[i][it])))) 234 | vul_idxs.append(list(filter(lambda idx: gt[i][it][idx], vul_idx))) 235 | tpls[i] = [sum(gt[i][it][vul_idx]), len(vul_idx), 0 if len(vul_idx) == 0 else sum(gt[i][it][vul_idx])/len(vul_idx)] 236 | print('%d / %d, %.2f PPV' % tuple(tpls[i])) 237 | 238 | c_vec = ['#DAA520' if gt_ == 0 else '#DC143C' for gt_ in gt[i][it]] 239 | ax[0][i].scatter(wb[i][it], ip[i][it], s=3*np.pi, c=c_vec, alpha=0.6) 240 | ax[0][i].set_xlabel('Whitebox Output') 241 | Z = clfs_wb[i].predict(np.c_[yy.ravel(), xx.ravel()]) 242 | Z = Z.reshape(xx.shape) 243 | ax[0][i].contour(xx, yy, Z, levels=0, colors=['black']) 244 | print(sum(clfs_wb[i].predict(np.vstack((ip[i][it], wb[i][it])).T)), sum(clfs_bb[i].predict(np.vstack((ip[i][it], mc[i][it])).T))) 245 | print('Mean: %.2f / %.2f, %.2f PPV' % tuple(np.mean(tpls, axis=0))) 246 | print('Std : %.2f / %.2f, %.2f PPV' % tuple(np.std(tpls, axis=0))) 247 | 248 | if args.banished_records == 0: 249 | for i in range(N): 250 | ax[0][i].scatter(wb[i][it][vul_idxs[i]], ip[i][it][vul_idxs[i]], s=3*np.pi, c='blue', alpha=0.6) 251 | if args.sample_size == 5000: 252 | pickle.dump(vul_idxs, open('data/' + args.dataset + '/' + str(args.attribute) + '_' + args.adv_knowledge + '_' + 'banished_records.p', 'wb')) 253 | else: 254 | banished_idx = pickle.load(open('data/' + args.dataset + '/' + str(args.attribute) + '_' + args.adv_knowledge + '_' + 'banished_records.p', 'rb')) 255 | tpls = np.zeros((N, 3)) 256 | for i in range(N): 257 | tpls[i] = [len(set(banished_idx[i]).intersection(set(vul_idxs[i]))), len(banished_idx[i]), len(set(vul_idxs[i]) - set(banished_idx[i]))] 258 | print('%d / %d records still vulnerable, %d new vulnerable records' % tuple(tpls[i])) 259 | ax[0][i].scatter(wb[i][it][banished_idx[i]], ip[i][it][banished_idx[i]], s=3*np.pi, c='blue', alpha=0.6) 260 | print('Mean: %.2f / %.2f still vulnerable, %.2f new vulnerable' % tuple(np.mean(tpls, axis=0))) 261 | print('Std : %.2f / %.2f still vulnerable, %.2f new vulnerable' % tuple(np.std(tpls, axis=0))) 262 | 263 | f.tight_layout() 264 | plt.show() 265 | 266 | f, ax = plt.subplots(1, N, sharey=True, squeeze=False) 267 | ax[0][0].set_ylabel('Imputation Confidence') 268 | for i in range(N): 269 | c_vec = ['#DAA520' if gt_ == 0 else '#DC143C' for gt_ in gt[i][it]] 270 | ax[0][i].scatter(mc[i][it], ip[i][it], s=3*np.pi, c=c_vec, alpha=0.6) 271 | ax[0][i].set_xlabel('Model Confidence') 272 | Z = clfs_bb[i].predict(np.c_[yy.ravel(), xx.ravel()]) 273 | Z = Z.reshape(xx.shape) 274 | ax[0][i].contour(xx, yy, Z, levels=0, colors=['black']) 275 | f.tight_layout() 276 | plt.show() 277 | 278 | 279 | if __name__ == '__main__': 280 | parser = argparse.ArgumentParser() 281 | parser.add_argument('dataset', type=str) 282 | parser.add_argument('--model', type=str, default='nn') 283 | parser.add_argument('--skew_attribute', type=int, default=0, help='Attribute on which to skew the non-iid data sampling 0 (population, default), 1 or 2 -- for Census 1: Income and 2: Race, and for Texas 1: Charges and 2: Ethnicity') 284 | parser.add_argument('--skew_outcome', type=int, default=0, help='In case skew_attribute = 1, which outcome the distribution is skewed upon -- For Census Race: 0 (White, default), 1 (Black) or 3 (Asian), and for Texas Ethnicity: 0 (Hispanic, default) or 1 (Not Hispanic)') 285 | parser.add_argument('--comb_flag', type=int, default=0, help='0 (default): just use 2 featues for combining using meta model, 1: use 3 features for training meta model') 286 | parser.add_argument('--gamma', type=float, default=0.5) 287 | parser.add_argument('--dp', type=str, default='gdp') 288 | parser.add_argument('--eps', type=float, default=None) 289 | parser.add_argument('--attribute', type=int, default=0) 290 | parser.add_argument('--sensitive_outcome', type=int, default=1) 291 | parser.add_argument('--candidate_size', type=int, default=int(1e4)) 292 | parser.add_argument('--adv_knowledge', type=str, default='low') 293 | parser.add_argument('--sample_size', type=int, default=50000) 294 | parser.add_argument('--banished_records', type=int, default=0) 295 | args = parser.parse_args() 296 | print(vars(args)) 297 | candidate_size = args.candidate_size 298 | sensitive_outcome = args.sensitive_outcome 299 | 300 | result = get_result(args) 301 | 302 | gt = {run: {idx: [] for idx in range(3)} for run in RUNS} 303 | wb = {run: {idx: [] for idx in range(3)} for run in RUNS} 304 | mc = {run: {idx: [] for idx in range(3)} for run in RUNS} 305 | ip = {run: {idx: [] for idx in range(3)} for run in RUNS} 306 | imp_auxs, model_auxs = np.zeros((4, T)), np.zeros((4, T)) 307 | pmc_acc, ip_acc, yeom_acc, csmia_acc, mc_acc, wb_acc, mc_comb_acc, wb_comb_acc = np.zeros((3, T)), np.zeros((3, T)), np.zeros((3, T)), np.zeros((3, T)), np.zeros((3, T)), np.zeros((3, T)), np.zeros((3, T)), np.zeros((3, T)) 308 | corr_vals = np.zeros((100, T)) 309 | 310 | for run in RUNS: 311 | v = result['no_privacy'][args.adv_knowledge][args.sample_size][run] 312 | if args.eps != None: 313 | v = result[args.eps][args.adv_knowledge][args.sample_size][run] 314 | elif args.banished_records == 1: 315 | v = result['banished'][args.adv_knowledge][args.sample_size][run] 316 | sensitive_test, csmia_pred, _, model_conf, whitebox_info, plot_info_dict, model_aux, _ = v 317 | _, _, imp_conf, _, _, _, _, imp_aux = result['no_privacy'][args.adv_knowledge][args.sample_size][run] 318 | 319 | model_auxs[:, run] = model_aux 320 | imp_auxs[:, run] = imp_aux 321 | if args.dataset == 'purchase_100_binary': 322 | labels = ['0', '1'] 323 | else: 324 | target_attrs, attribute_dict, _, _ = get_sensitive_features(args.dataset, np.zeros((1,1))) 325 | labels = list(attribute_dict[target_attrs[args.attribute]].values()) 326 | whitebox_info_k_1, whitebox_info_k_10, whitebox_info_k_100 = whitebox_info 327 | 328 | # selecting which WB attack to evaluate 329 | whitebox_info_k = whitebox_info_k_10 330 | corr_vals[:, run] = plot_info_dict[str(sensitive_outcome)]['correlation_vals'] 331 | #if run == 0: 332 | # plot_layer_outputs(informative_neurons = plot_info_dict[str(sensitive_outcome)]['informative_neurons'], plot_info = plot_info_dict[str(sensitive_outcome)]['plot_info'], pos_label = labels[int(sensitive_outcome)], neg_label = 'Not '+labels[int(sensitive_outcome)]) 333 | 334 | train_loss, train_acc, test_loss, test_acc = model_aux 335 | yeom_pred = yeom_membership_inference(-np.log(model_conf), None, train_loss, test_loss) 336 | 337 | cc = Counter(sensitive_test[range(candidate_size)]) 338 | for k, v in cc.items(): 339 | print(attribute_dict[target_attrs[args.attribute]][k], v) 340 | 341 | for idx, txt, candidate_indices in zip([0, 1, 2], ['train', 'test', 'holdout'], [range(candidate_size), range(candidate_size, 2*candidate_size), range(2*candidate_size, len(sensitive_test))]): 342 | pmc_acc[idx][run] = max(Counter(sensitive_test[candidate_indices]).values()) / candidate_size 343 | 344 | prior_prob = np.zeros(len(labels)) 345 | for k, v in Counter(sensitive_test[candidate_indices]).items(): 346 | prior_prob[int(k)] = v / candidate_size 347 | yeom_acc[idx][run] = sum(sensitive_test[candidate_indices] == np.argmax(yeom_pred[candidate_indices] * prior_prob, axis=1)) / candidate_size 348 | 349 | csmia_acc[idx][run] = sum(sensitive_test[candidate_indices] == csmia_pred[candidate_indices]) / candidate_size 350 | 351 | ip_acc[idx][run] = sum(sensitive_test[candidate_indices] == np.argmax(imp_conf[candidate_indices], axis=1)) / candidate_size 352 | 353 | mc_acc[idx][run] = sum(sensitive_test[candidate_indices] == np.argmax(model_conf[candidate_indices], axis=1)) / candidate_size 354 | 355 | mc_comb_acc[idx][run] = sum(sensitive_test[candidate_indices] == np.argmax(imp_conf[candidate_indices] * model_conf[candidate_indices], axis=1)) / candidate_size 356 | 357 | wb_acc[idx][run] = sum(sensitive_test[candidate_indices] == np.argmax(whitebox_info_k[candidate_indices], axis=1)) / candidate_size 358 | 359 | wb_comb_acc[idx][run] = sum(sensitive_test[candidate_indices] == np.argmax(imp_conf[candidate_indices] * whitebox_info_k[candidate_indices], axis=1)) / candidate_size 360 | 361 | gt[run][idx] = np.where(sensitive_test[candidate_indices] == sensitive_outcome, 1, 0) 362 | wb[run][idx] = whitebox_info_k[candidate_indices, sensitive_outcome] 363 | mc[run][idx] = model_conf[candidate_indices, sensitive_outcome] 364 | ip[run][idx] = imp_conf[candidate_indices, sensitive_outcome] 365 | 366 | print('Model Performance:\nTrain Loss:\t%.3f +/- %.3f\nTrain Acc:\t%.2f +/- %.2f\nTest Loss:\t%.3f +/- %.3f\nTest Acc:\t%.2f +/- %.2f' % (np.mean(model_auxs[0]), np.std(model_auxs[0]), np.mean(model_auxs[1]), np.std(model_auxs[1]), np.mean(model_auxs[2]), np.std(model_auxs[2]), np.mean(model_auxs[3]), np.std(model_auxs[3]))) 367 | 368 | print('Imputation Performance:\nTrain Loss:\t%.3f +/- %.3f\nTrain Acc:\t%.2f +/- %.2f\nTest Loss:\t%.3f +/- %.3f\nTest Acc:\t%.2f +/- %.2f' % (np.mean(imp_auxs[0]), np.std(imp_auxs[0]), np.mean(imp_auxs[1]), np.std(imp_auxs[1]), np.mean(imp_auxs[2]), np.std(imp_auxs[2]), np.mean(imp_auxs[3]), np.std(imp_auxs[3]))) 369 | 370 | print('\t Train AI Acc \t Test AI Acc') 371 | print('PMC\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(pmc_acc[0]), np.std(pmc_acc[0]), np.mean(pmc_acc[1]), np.std(pmc_acc[1]))) 372 | print('IP\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(ip_acc[0]), np.std(ip_acc[0]), np.mean(ip_acc[1]), np.std(ip_acc[1]))) 373 | print('Yeom\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(yeom_acc[0]), np.std(yeom_acc[0]), np.mean(yeom_acc[1]), np.std(yeom_acc[1]))) 374 | print('CSMIA\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(csmia_acc[0]), np.std(csmia_acc[0]), np.mean(csmia_acc[1]), np.std(csmia_acc[1]))) 375 | print('BB\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(mc_acc[0]), np.std(mc_acc[0]), np.mean(mc_acc[1]), np.std(mc_acc[1]))) 376 | print('BB*IP\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(mc_comb_acc[0]), np.std(mc_comb_acc[0]), np.mean(mc_comb_acc[1]), np.std(mc_comb_acc[1]))) 377 | print('WB\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(wb_acc[0]), np.std(wb_acc[0]), np.mean(wb_acc[1]), np.std(wb_acc[1]))) 378 | print('WB*IP\t%.2f +/- %.2f\t%.2f +/- %.2f' % (np.mean(wb_comb_acc[0]), np.std(wb_comb_acc[0]), np.mean(wb_comb_acc[1]), np.std(wb_comb_acc[1]))) 379 | 380 | print('\nTop-10 Neuron Correlation Values:\t%.2f +/- %.2f' % (np.mean(corr_vals[:10]), np.std(corr_vals[:10]))) 381 | for i in range(10): 382 | print('Neuron #%d:\t%.2f +/- %.2f' % (i, np.mean(corr_vals[i]), np.std(corr_vals[i]))) 383 | 384 | clfs_wb = [fit_model(args, gt[run][2], ip[run][2], wb[run][2]) for run in RUNS] 385 | clfs_bb = [fit_model(args, gt[run][2], ip[run][2], mc[run][2]) for run in RUNS] 386 | 387 | # change the values in plot_cond to get the desired plots 388 | plot_cond = { 389 | 'ip': {'flag': True, 'xoffset': 100, 'yoffset': 0.04}, 390 | 'wb': {'flag': True, 'xoffset': 2, 'yoffset': 0.02}, 391 | 'wb.ip': {'flag': True, 'xoffset': 7, 'yoffset': -0.08}, 392 | 'wbXip': {'flag': True, 'xoffset': 100, 'yoffset': 0.04}, 393 | 'bb': {'flag': False, 'xoffset': 1, 'yoffset': 0.05}, 394 | 'bb.ip': {'flag': False, 'xoffset': 1, 'yoffset': -0.06}, 395 | 'bbXip': {'flag': False, 'xoffset': 1000, 'yoffset': 0.05}, 396 | 'baseline': {'flag': True, 'xoffset': 5, 'yoffset': 0.05} 397 | } 398 | print('*'*5 + ' Train Set ' + '*'*5) 399 | plot_ppvs(args, gt, wb, mc, ip, (clfs_wb, clfs_bb), plot_cond) 400 | print('*'*5 + ' Test Set ' + '*'*5) 401 | plot_ppvs(args, gt, wb, mc, ip, (clfs_wb, clfs_bb), plot_cond, data_type=1) 402 | ''' 403 | plot_cond = { 404 | 'wb': {'xoffset': 1, 'yoffset': -0.16}, 405 | 'wb.ip': {'xoffset': 4, 'yoffset': -0.05}, 406 | 'wbXip': {'xoffset': 2, 'yoffset': 0.1}, 407 | 'baseline': {'xoffset': 800, 'yoffset': -0.2} 408 | } 409 | plot_ppv_change(args, gt, wb, mc, ip, (clfs_wb, clfs_bb), plot_cond) 410 | ''' 411 | #plot_roc(args, gt, wb, mc, ip) 412 | scatter_plot(args, gt, wb, mc, ip, (clfs_wb, clfs_bb), 0) 413 | -------------------------------------------------------------------------------- /improved_ai/main.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | import argparse 4 | import pickle 5 | import numpy as np 6 | import tensorflow.compat.v1 as tf 7 | 8 | sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), "../"))) 9 | 10 | from core.data_util import sample_noniid_data 11 | from core.data_util import load_data 12 | from core.data_util import get_sensitive_features 13 | from core.data_util import process_features 14 | from core.data_util import threat_model 15 | from core.data_util import subsample 16 | from core.utilities import imputation_training 17 | from core.attack import train_target_model 18 | from core.attack import whitebox_attack 19 | from core.attack import yeom_membership_inference 20 | from core.classifier import get_predictions 21 | from core.classifier import get_layer_outputs 22 | from collections import Counter 23 | 24 | RESULT_PATH = 'results/' 25 | 26 | if not os.path.exists(RESULT_PATH): 27 | os.makedirs(RESULT_PATH) 28 | 29 | 30 | def get_model_conf(target_classifier, true_x, true_y, target_attr, labels, attribute_dict, max_attr_vals, col_flags, skip_corr=False): 31 | model_conf = np.zeros((len(true_x), len(labels))) 32 | for val in labels: 33 | features = np.copy(true_x) 34 | features[:, target_attr] = val / max_attr_vals[target_attr] 35 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 36 | x={'x': process_features(features, args.train_dataset, attribute_dict, max_attr_vals, target_attr, col_flags, skip_corr=skip_corr)}, 37 | num_epochs=1, 38 | shuffle=False) 39 | _, pred_scores = get_predictions(target_classifier.predict(input_fn=pred_input_fn)) 40 | model_conf[:, val] = [pred_scores[i, true_y[i]] for i in range(len(true_y))] 41 | return model_conf 42 | 43 | 44 | # Confidence Score-based Model Inversion Attack from Mehnaz et al. (2022) 45 | def get_csmia_pred(target_classifier, true_x, true_y, target_attr, labels, attribute_dict, max_attr_vals, col_flags, skip_corr=False): 46 | pred_conf = np.zeros((len(true_x), len(labels))) 47 | pred_label = np.zeros((len(true_x), len(labels))) 48 | csmia_pred = np.zeros(len(true_x)) 49 | for val in labels: 50 | features = np.copy(true_x) 51 | features[:, target_attr] = val / max_attr_vals[target_attr] 52 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 53 | x={'x': process_features(features, args.train_dataset, attribute_dict, max_attr_vals, target_attr, col_flags, skip_corr=skip_corr)}, 54 | num_epochs=1, 55 | shuffle=False) 56 | pred_lab, pred_scores = get_predictions(target_classifier.predict(input_fn=pred_input_fn)) 57 | pred_label[:, val] = pred_lab 58 | pred_conf[:, val] = np.max(pred_scores, axis=1) 59 | true_y = true_y.reshape((-1, 1)) 60 | matches = np.sum(pred_label == true_y, axis=1) 61 | print('Records with all incorrect predictions:\t %d (%.2f%%)' % (sum(matches==0), 100*sum(matches==0)/len(matches))) 62 | print('Records with exactly one correct prediction:\t %d (%.2f%%)' % (sum(matches==1), 100*sum(matches==1)/len(matches))) 63 | print('Records with multiple correct predictions:\t %d (%.2f%%)' % (sum(matches>=2), 100*sum(matches>=2)/len(matches))) 64 | csmia_pred[matches==0] = np.argmin(pred_conf[matches==0], axis=1) 65 | csmia_pred[matches==1] = np.argmax(pred_label[matches==1] == true_y[matches==1], axis=1) 66 | csmia_pred[matches>=2] = np.argmax((pred_label[matches>=2] == true_y[matches>=2]) * pred_conf[matches>=2], axis=1) 67 | return csmia_pred 68 | 69 | 70 | def get_wb_info(layer_outputs, adv_known_idx, labels, sensitive_test, c_idx=None): 71 | whitebox_info_k_1 = np.zeros((len(sensitive_test), len(labels))) 72 | whitebox_info_k_10 = np.zeros((len(sensitive_test), len(labels))) 73 | whitebox_info_k_100 = np.zeros((len(sensitive_test), len(labels))) 74 | plot_info_dict = {} 75 | for val in labels: 76 | if c_idx != None: 77 | # high adversarial knowledge setting: running k-out-of-n test (k is set to 100) 78 | chunks = [c_idx[i:i+100] for i in range(0, len(c_idx), 100)] 79 | for chunk in chunks: 80 | whitebox_info, informative_neurons, correlation_vals, plot_info = whitebox_attack(layer_outputs[val], sensitive_test==val, list(set(adv_known_idx) - set(chunk))) 81 | whitebox_info_k_1[chunk, val], whitebox_info_k_10[chunk, val], whitebox_info_k_100[chunk, val] = [v[chunk] for v in whitebox_info] 82 | if str(val) not in plot_info_dict: 83 | whitebox_info_k_1[:, val], whitebox_info_k_10[:, val], whitebox_info_k_100[:, val] = whitebox_info 84 | plot_info_dict[str(val)] = {} 85 | plot_info_dict[str(val)]['informative_neurons'] = informative_neurons 86 | plot_info_dict[str(val)]['correlation_vals'] = correlation_vals 87 | plot_info_dict[str(val)]['plot_info'] = plot_info 88 | else: 89 | whitebox_info, informative_neurons, correlation_vals, plot_info = whitebox_attack(layer_outputs[val], sensitive_test==val, adv_known_idx) 90 | whitebox_info_k_1[:, val], whitebox_info_k_10[:, val], whitebox_info_k_100[:, val] = whitebox_info 91 | plot_info_dict[str(val)] = {} 92 | plot_info_dict[str(val)]['informative_neurons'] = informative_neurons 93 | plot_info_dict[str(val)]['correlation_vals'] = correlation_vals 94 | plot_info_dict[str(val)]['plot_info'] = plot_info 95 | return (whitebox_info_k_1, whitebox_info_k_10, whitebox_info_k_100), plot_info_dict 96 | 97 | 98 | def get_whitebox_info(target_classifier, true_x, adv_known_idx, target_attr, labels, attribute_dict, max_attr_vals, col_flags, sensitive_test, c_idx=None, skip_corr=False): 99 | layer_outputs = [] 100 | for val in labels: 101 | features = np.copy(true_x) 102 | features[:, target_attr] = val / max_attr_vals[target_attr] 103 | pred_input_fn = tf.estimator.inputs.numpy_input_fn( 104 | x={'x': process_features(features, args.train_dataset, attribute_dict, max_attr_vals, target_attr, col_flags, skip_corr=skip_corr)}, 105 | num_epochs=1, 106 | shuffle=False) 107 | layer_outputs.append(get_layer_outputs(target_classifier.predict(input_fn=pred_input_fn))) 108 | return get_wb_info(layer_outputs, adv_known_idx, labels, sensitive_test, c_idx) 109 | 110 | 111 | def run_experiment(args): 112 | if not os.path.exists(RESULT_PATH + args.train_dataset): 113 | os.makedirs(RESULT_PATH + args.train_dataset) 114 | 115 | MODEL = str(args.skew_attribute) + '_' + str(args.skew_outcome) + '_' + str(args.target_test_train_ratio) + '_' + str(args.target_model) + '_' 116 | 117 | train_x, train_y, test_x, test_y = load_data('target_data.npz', args) 118 | h_train_x, h_train_y, h_test_x, h_test_y = load_data('holdout_data.npz', args) 119 | sk_train_x, sk_train_y, sk_test_x, sk_test_y = load_data('skewed_data.npz', args) 120 | sk2_train_x, sk2_train_y, sk2_test_x, sk2_test_y = load_data('skewed_2_data.npz', args) 121 | true_x = np.vstack((train_x, test_x, h_train_x, h_test_x, sk_train_x, sk_test_x, sk2_train_x, sk2_test_x)) 122 | true_y = np.concatenate((train_y, test_y, h_train_y, h_test_y, sk_train_y, sk_test_y, sk2_train_y, sk2_test_y)) 123 | 124 | c_size = args.candidate_size 125 | train_c_idx, test_c_idx, h_test_idx, sk_test_idx, sk2_test_idx, adv_known_idxs = threat_model(args, len(true_x)) 126 | 127 | assert(args.attribute < 3) 128 | target_attrs, attribute_dict, max_attr_vals, col_flags = get_sensitive_features(args.train_dataset, train_x) 129 | target_attr = target_attrs[args.attribute] 130 | labels = [0, 1] if attribute_dict == None else list(attribute_dict[target_attr].keys()) 131 | 132 | # removes sensitive records from model training 133 | if args.banished_records: 134 | banished_records_idx = pickle.load(open('data/' + args.train_dataset + '/' + str(args.attribute) + '_' + 'med' + '_' + 'banished_records.p', 'rb')) 135 | assert(args.run < len(banished_records_idx)) 136 | train_idx = list(set(range(len(train_x))) - set(np.array(train_c_idx)[banished_records_idx[args.run]])) 137 | else: 138 | train_idx = range(len(train_x)) 139 | 140 | # training the target model 141 | _, _, _, target_classifier, model_aux = train_target_model( 142 | args=args, 143 | dataset=[process_features(train_x[train_idx], args.train_dataset, attribute_dict, max_attr_vals, target_attr, col_flags, skip_corr=args.skip_corr), train_y[train_idx], process_features(test_x, args.train_dataset, attribute_dict, max_attr_vals, target_attr, col_flags, skip_corr=args.skip_corr), test_y], 144 | epochs=args.target_epochs, 145 | batch_size=args.target_batch_size, 146 | learning_rate=args.target_learning_rate, 147 | clipping_threshold=args.target_clipping_threshold, 148 | n_hidden=args.target_n_hidden, 149 | l2_ratio=args.target_l2_ratio, 150 | model=args.target_model, 151 | privacy=args.target_privacy, 152 | dp=args.target_dp, 153 | epsilon=args.target_epsilon, 154 | delta=args.target_delta, 155 | save=args.save_model) 156 | train_loss, train_acc, test_loss, test_acc = model_aux 157 | 158 | model_conf = get_model_conf(target_classifier, true_x, true_y, target_attr, labels, attribute_dict, max_attr_vals, col_flags, skip_corr=args.skip_corr) 159 | yeom_pred = yeom_membership_inference(-np.log(model_conf), None, train_loss) 160 | yeom_pred_2 = yeom_membership_inference(-np.log(model_conf), None, train_loss, test_loss) 161 | csmia_pred = get_csmia_pred(target_classifier, true_x, true_y, target_attr, labels, attribute_dict, max_attr_vals, col_flags, skip_corr=args.skip_corr) 162 | 163 | sensitive_test = true_x[:, target_attr] * max_attr_vals[target_attr] 164 | known_test = process_features(true_x, args.train_dataset, attribute_dict, max_attr_vals, target_attr, col_flags, skip_sensitive=True, skip_corr=args.skip_corr) 165 | 166 | prior_prob = np.zeros(len(labels)) 167 | for k, v in Counter(sensitive_test).items(): 168 | prior_prob[int(k)] = v / len(sensitive_test) 169 | 170 | for threat_level in ['low', 'low2', 'med', 'high']: 171 | print("\nAdversary's knowledge of data: {}".format(threat_level)) 172 | adv_known_idx_ = adv_known_idxs[threat_level] 173 | 174 | for sample_size in [50, 500, 5000, 50000]: 175 | print("\nAdversary's data sample size: {}".format(sample_size)) 176 | adv_known_idx = subsample(adv_known_idx_, true_x[adv_known_idx_, target_attr] * max_attr_vals[target_attr], sample_size) 177 | sensitive_train = true_x[adv_known_idx, target_attr] * max_attr_vals[target_attr] 178 | known_train = process_features(true_x[adv_known_idx], args.train_dataset, attribute_dict, max_attr_vals, target_attr, col_flags, skip_sensitive=True, skip_corr=args.skip_corr) 179 | 180 | # training the imputation model 181 | imp_conf, imp_aux = imputation_training(args, known_train, sensitive_train, known_test, sensitive_test, clf_type='nn', epochs=10) 182 | 183 | print('\n\tTrain\tTest') 184 | print('PMC:\t%.2f\t%.2f' % (max(Counter(sensitive_test[train_c_idx]).values()) / c_size, max(Counter(sensitive_test[test_c_idx]).values()) / c_size)) 185 | print('IP:\t%.2f\t%.2f' % (sum(sensitive_test[train_c_idx] == np.argmax(imp_conf[train_c_idx], axis=1)) / c_size, sum(sensitive_test[test_c_idx] == np.argmax(imp_conf[test_c_idx], axis=1)) / c_size)) 186 | print('Yeom1:\t%.2f\t%.2f' % (sum(sensitive_test[train_c_idx] == np.argmax(yeom_pred[train_c_idx] * prior_prob, axis=1)) / c_size, sum(sensitive_test[test_c_idx] == np.argmax(yeom_pred[test_c_idx] * prior_prob, axis=1)) / c_size)) 187 | print('Yeom2:\t%.2f\t%.2f' % (sum(sensitive_test[train_c_idx] == np.argmax(yeom_pred_2[train_c_idx] * prior_prob, axis=1)) / c_size, sum(sensitive_test[test_c_idx] == np.argmax(yeom_pred_2[test_c_idx] * prior_prob, axis=1)) / c_size)) 188 | print('Yeom1_IP:\t%.2f\t%.2f' % (sum(sensitive_test[train_c_idx] == np.argmax(yeom_pred[train_c_idx] * imp_conf[train_c_idx], axis=1)) / c_size, sum(sensitive_test[test_c_idx] == np.argmax(yeom_pred[test_c_idx] * imp_conf[test_c_idx], axis=1)) / c_size)) 189 | print('Yeom2_IP:\t%.2f\t%.2f' % (sum(sensitive_test[train_c_idx] == np.argmax(yeom_pred_2[train_c_idx] * imp_conf[train_c_idx], axis=1)) / c_size, sum(sensitive_test[test_c_idx] == np.argmax(yeom_pred_2[test_c_idx] * imp_conf[test_c_idx], axis=1)) / c_size)) 190 | print('BB:\t%.2f\t%.2f' % (sum(sensitive_test[train_c_idx] == np.argmax(model_conf[train_c_idx], axis=1)) / c_size, sum(sensitive_test[test_c_idx] == np.argmax(model_conf[test_c_idx], axis=1)) / c_size)) 191 | print('CSMIA:\t%.2f\t%.2f' % (sum(sensitive_test[train_c_idx] == csmia_pred[train_c_idx]) / c_size, sum(sensitive_test[test_c_idx] == csmia_pred[test_c_idx]) / c_size)) 192 | print('BB.IP:\t%.2f\t%.2f' % (sum(sensitive_test[train_c_idx] == np.argmax(imp_conf[train_c_idx] * model_conf[train_c_idx], axis=1)) / c_size, sum(sensitive_test[test_c_idx] == np.argmax(imp_conf[test_c_idx] * model_conf[test_c_idx], axis=1)) / c_size)) 193 | 194 | if threat_level == 'high' and sample_size == 50000: 195 | whitebox_info, plot_info_dict = get_whitebox_info(target_classifier, true_x, list(range(len(train_x))), target_attr, labels, attribute_dict, max_attr_vals, col_flags, sensitive_test, c_idx=train_c_idx, skip_corr=args.skip_corr) 196 | else: 197 | whitebox_info, plot_info_dict = get_whitebox_info(target_classifier, true_x, adv_known_idx, target_attr, labels, attribute_dict, max_attr_vals, col_flags, sensitive_test, skip_corr=args.skip_corr) 198 | 199 | if threat_level == 'low': 200 | idx = train_c_idx + test_c_idx + sk_test_idx 201 | elif threat_level == 'low2': 202 | idx = train_c_idx + test_c_idx + sk2_test_idx 203 | else: 204 | idx = train_c_idx + test_c_idx + h_test_idx 205 | 206 | if args.target_privacy == 'no_privacy': 207 | pickle.dump([sensitive_test[idx], csmia_pred[idx], imp_conf[idx], model_conf[idx], [wb[idx] for wb in whitebox_info], plot_info_dict, model_aux, imp_aux], open(RESULT_PATH + args.train_dataset + '/' + MODEL + 'no_privacy_' + str(args.attribute) + '_' + threat_level + '_' + str(sample_size) + '_' + str(args.banished_records) + '_' + str(args.run) + '.p', 'wb')) 208 | else: 209 | pickle.dump([sensitive_test[idx], csmia_pred[idx], imp_conf[idx], model_conf[idx], [wb[idx] for wb in whitebox_info], plot_info_dict, model_aux, imp_aux], open(RESULT_PATH + args.train_dataset + '/' + MODEL + args.target_privacy + '_' + args.target_dp + '_' + str(args.target_epsilon) + '_' + str(args.attribute) + '_' + threat_level + '_' + str(sample_size) + '_' + str(args.banished_records) + '_' + str(args.run) + '.p', 'wb')) 210 | 211 | 212 | if __name__ == '__main__': 213 | parser = argparse.ArgumentParser() 214 | parser.add_argument('train_dataset', type=str) 215 | parser.add_argument('--run', type=int, default=0) 216 | parser.add_argument('--use_cpu', type=int, default=0) 217 | parser.add_argument('--save_model', type=int, default=0) 218 | parser.add_argument('--save_data', type=int, default=0, help='0: no save data, instead run experiments (default), 1: save data') 219 | parser.add_argument('--attribute', type=int, default=0, help='senstive attribute to use: 0, 1 or 2') 220 | parser.add_argument('--candidate_size', type=int, default=int(1e4), help='candidate set size') 221 | parser.add_argument('--skew_attribute', type=int, default=0, help='Attribute on which to skew the non-iid data sampling 0 (population, default), 1 or 2 -- for Census 1: Income and 2: Race, and for Texas 1: Charges and 2: Ethnicity') 222 | parser.add_argument('--skew_outcome', type=int, default=0, help='In case skew_attribute = 2, which outcome to skew the distribution upon -- For Census Race: 0 (White, default), 1 (Black) or 3 (Asian), and for Texas Ethnicity: 0 (Hispanic, default) or 1 (Not Hispanic)') 223 | parser.add_argument('--sensitive_outcome', type=int, default=0, help='In case skew_attribute = 2, this indicates the sensitive outcome -- For Census Race: 0 (White, default), 1 (Black) or 3 (Asian), and for Texas Ethnicity: 0 (Hispanic, default) or 1 (Not Hispanic)') 224 | parser.add_argument('--banished_records', type=int, default=0, help='if the set of records in banished.p file are to be removed from model training (default:0 no records are removed)') 225 | parser.add_argument('--skip_corr', type=int, default=0, help='For Texas-100X, whether to skip Race (or Ethnicity) when the target sensitive attribute is Ethnicity (or Race) -- default is not to skip (0)') 226 | # target and shadow model configuration 227 | parser.add_argument('--n_shadow', type=int, default=5) 228 | parser.add_argument('--target_data_size', type=int, default=int(5e4)) 229 | parser.add_argument('--target_test_train_ratio', type=float, default=0.5) 230 | parser.add_argument('--target_model', type=str, default='nn') 231 | parser.add_argument('--target_learning_rate', type=float, default=0.01) 232 | parser.add_argument('--target_batch_size', type=int, default=200) 233 | parser.add_argument('--target_n_hidden', type=int, default=256) 234 | parser.add_argument('--target_epochs', type=int, default=100) 235 | parser.add_argument('--target_l2_ratio', type=float, default=1e-8) 236 | parser.add_argument('--target_clipping_threshold', type=float, default=1.0) 237 | parser.add_argument('--target_privacy', type=str, default='no_privacy') 238 | parser.add_argument('--target_dp', type=str, default='gdp') 239 | parser.add_argument('--target_epsilon', type=float, default=0.5) 240 | parser.add_argument('--target_delta', type=float, default=1e-5) 241 | # attack model configuration 242 | parser.add_argument('--attack_model', type=str, default='nn') 243 | parser.add_argument('--attack_learning_rate', type=float, default=0.01) 244 | parser.add_argument('--attack_batch_size', type=int, default=100) 245 | parser.add_argument('--attack_n_hidden', type=int, default=64) 246 | parser.add_argument('--attack_epochs', type=int, default=100) 247 | parser.add_argument('--attack_l2_ratio', type=float, default=1e-6) 248 | 249 | # parse configuration 250 | args = parser.parse_args() 251 | print(vars(args)) 252 | 253 | # Flag to disable GPU 254 | if args.use_cpu: 255 | os.environ['CUDA_VISIBLE_DEVICES'] = '-1' 256 | else: 257 | os.environ['TF_XLA_FLAGS'] = '--tf_xla_enable_xla_devices' 258 | 259 | if args.save_data == 1: 260 | sample_noniid_data(args) 261 | else: 262 | run_experiment(args) 263 | -------------------------------------------------------------------------------- /improved_ai/run_experiments.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "Starting script" 4 | if [[ $# -eq 0 ]] ; then # if called with no arguments 5 | echo "Usage: bash $0 " 6 | exit 1 7 | fi 8 | 9 | DATASET=$1 10 | CODE=main.py 11 | 12 | echo "Loading modules" 13 | source /etc/profile.d/modules.sh 14 | module load python3.6.2 15 | 16 | # Make sure environment has dependencies 17 | echo "Creating environment" 18 | python3 -m venv ../env 19 | source ../env/bin/activate 20 | 21 | python3 -m pip install --upgrade pip 22 | python3 -m pip install -r ../requirements.txt 23 | 24 | if [ $DATASET == 'census' ]; then 25 | L2_RATIO=1e-6 26 | SKEW_OUT=3 27 | SENS_OUT=3 28 | SENS_ATTR=1 29 | else 30 | L2_RATIO=1e-3 31 | SKEW_OUT=0 32 | SENS_OUT=0 33 | SENS_ATTR=2 34 | fi 35 | 36 | echo "Filling data/ directory" 37 | python3 $CODE $DATASET --use_cpu=1 --save_data=1 --skew_attribute=0 --skew_outcome=$SKEW_OUT --sensitive_outcome=$SENS_OUT --target_test_train_ratio=0.5 --target_data_size=50000 38 | 39 | echo "Beginning experiment" 40 | # Experiment Setting : 50K training set size, 25K test set size, 10K candidate size 41 | # For Texas-100-v2, set --target_epochs=50, --target_learning_rate=0.001, --target_l2_ratio=1e-3, --target_batch_size=500, --target_data_size=50000, --candidate_size=10000, --targrt_test_train_ratio=0.5 42 | # For Census, set --target_epochs=50, --target_learning_rate=0.001, --target_l2_ratio=1e-6, --target_batch_size=500, --target_data_size=50000, --candidate_size=10000, --targrt_test_train_ratio=0.5 43 | for RUN in 1 2 3 4 5 44 | do 45 | python3 $CODE $DATASET \ 46 | --use_cpu=1 \ 47 | --skew_attribute=0 \ 48 | --skip_corr=1 \ 49 | --skew_outcome=$SKEW_OUT \ 50 | --sensitive_outcome=$SENS_OUT \ 51 | --target_test_train_ratio=0.5 \ 52 | --target_data_size=50000 \ 53 | --candidate_size=10000 \ 54 | --target_model='nn' \ 55 | --target_epochs=50 \ 56 | --target_l2_ratio=$L2_RATIO \ 57 | --target_learning_rate=0.001 \ 58 | --target_batch_size=500 \ 59 | --target_clipping_threshold=4 \ 60 | --attribute=$SENS_ATTR \ 61 | --run=$RUN 62 | done 63 | # Differential private model training 64 | for RUN in 1 2 3 4 5 65 | do 66 | python3 $CODE $DATASET \ 67 | --use_cpu=1 \ 68 | --skew_attribute=0 \ 69 | --skip_corr=1 \ 70 | --skew_outcome=$SKEW_OUT \ 71 | --sensitive_outcome=$SENS_OUT \ 72 | --target_test_train_ratio=0.5 \ 73 | --target_data_size=50000 \ 74 | --candidate_size=10000 \ 75 | --target_model='nn' \ 76 | --target_epochs=50 \ 77 | --target_l2_ratio=$L2_RATIO \ 78 | --target_learning_rate=0.001 \ 79 | --target_batch_size=500 \ 80 | --target_clipping_threshold=4 \ 81 | --target_privacy='grad_pert' \ 82 | --attribute=$SENS_ATTR \ 83 | --target_epsilon=1 \ 84 | --run=$RUN 85 | done 86 | # Model training after removing sensitive records 87 | # Note: run the below script only after running interpret_results.py 88 | # to generate banished_records.p file. 89 | for RUN in 1 2 3 4 5 90 | do 91 | python3 $CODE $DATASET \ 92 | --use_cpu=1 \ 93 | --skew_attribute=0 \ 94 | --banished_records=1 \ 95 | --skip_corr=1 \ 96 | --skew_outcome=$SKEW_OUT \ 97 | --sensitive_outcome=$SENS_OUT \ 98 | --target_test_train_ratio=0.5 \ 99 | --target_data_size=50000 \ 100 | --candidate_size=10000 \ 101 | --target_model='nn' \ 102 | --target_epochs=50 \ 103 | --target_l2_ratio=$L2_RATIO \ 104 | --target_learning_rate=0.001 \ 105 | --target_batch_size=500 \ 106 | --target_clipping_threshold=4 \ 107 | --attribute=$SENS_ATTR \ 108 | --run=$RUN 109 | done 110 | echo done 111 | -------------------------------------------------------------------------------- /improved_ai/run_experiments_slurm.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #SBATCH --job-name="AttributeInference" 3 | #SBATCH --ntasks=5 4 | 5 | echo "Starting script" 6 | if [[ $# -eq 0 ]] ; then # if called with no arguments 7 | echo "Usage: bash $0 [-s] " 8 | echo " -s: saves new sample of data set to data/ directory" 9 | exit 1 10 | fi 11 | 12 | if [ $1 == '-s' ]; then 13 | DATASET=$2 14 | SAVE_DATA=true 15 | else 16 | DATASET=$1 17 | SAVE_DATA=false 18 | fi 19 | CODE=main.py 20 | 21 | echo "Loading modules" 22 | source /etc/profile.d/modules.sh 23 | module load python3.6.2 24 | module load parallel 25 | 26 | # Make sure environment has dependencies 27 | echo "Creating environment" 28 | python3 -m venv ../env 29 | source ../env/bin/activate 30 | 31 | python3 -m pip install --upgrade pip 32 | python3 -m pip install -r ../requirements.txt 33 | 34 | if [ $DATASET == 'census' ]; then 35 | L2_RATIO=1e-6 36 | SKEW_OUT=3 37 | SENS_OUT=3 38 | SENS_ATTR=1 39 | else 40 | L2_RATIO=1e-3 41 | SKEW_OUT=0 42 | SENS_OUT=0 43 | SENS_ATTR=2 44 | fi 45 | 46 | if [ "$SAVE_DATA" = true ]; then 47 | echo "Filling data/ directory" 48 | python3 $CODE $DATASET --use_cpu=1 --save_data=1 --skew_attribute=0 --skew_outcome=$SKEW_OUT --sensitive_outcome=$SENS_OUT --target_test_train_ratio=0.5 --target_data_size=50000 49 | fi 50 | 51 | echo "Beginning experiment" 52 | # Experiment Setting : 50K training set size, 25K test set size, 10K candidate size 53 | # For Texas-100-v2, set --target_epochs=50, --target_learning_rate=0.001, --target_l2_ratio=1e-3, --target_batch_size=500, --target_data_size=50000, --candidate_size=10000, --targrt_test_train_ratio=0.5 54 | # For Census, set --target_epochs=50, --target_learning_rate=0.001, --target_l2_ratio=1e-6, --target_batch_size=500, --target_data_size=50000, --candidate_size=10000, --targrt_test_train_ratio=0.5 55 | parallel -j 5 "srun -p main --exclusive -N1 -n1 python3 $CODE $DATASET \ 56 | --use_cpu=1 \ 57 | --skew_attribute=0 \ 58 | --skip_corr=1 \ 59 | --skew_outcome=$SKEW_OUT \ 60 | --sensitive_outcome=$SENS_OUT \ 61 | --target_test_train_ratio=0.5 \ 62 | --target_data_size=50000 \ 63 | --candidate_size=10000 \ 64 | --target_model='nn' \ 65 | --target_epochs=50 \ 66 | --target_l2_ratio=$L2_RATIO \ 67 | --target_learning_rate=0.001 \ 68 | --target_batch_size=500 \ 69 | --target_clipping_threshold=4 \ 70 | --attribute=$SENS_ATTR \ 71 | --run={1}" ::: {0..4} 72 | 73 | parallel -j 5 "srun -p main --exclusive -N1 -n1 python3 $CODE $DATASET \ 74 | --use_cpu=1 \ 75 | --skew_attribute=0 \ 76 | --skip_corr=1 \ 77 | --skew_outcome=$SKEW_OUT \ 78 | --sensitive_outcome=$SENS_OUT \ 79 | --target_test_train_ratio=0.5 \ 80 | --target_data_size=50000 \ 81 | --candidate_size=10000 \ 82 | --target_model='nn' \ 83 | --target_epochs=50 \ 84 | --target_l2_ratio=$L2_RATIO \ 85 | --target_learning_rate=0.001 \ 86 | --target_batch_size=500 \ 87 | --target_clipping_threshold=4 \ 88 | --target_privacy='grad_pert' \ 89 | --attribute=$SENS_ATTR \ 90 | --target_epsilon={1} \ 91 | --run={2}" ::: 1 ::: {0..4} 92 | #rm -r /tmp/tmp* 93 | echo done 94 | -------------------------------------------------------------------------------- /improved_mi/README.md: -------------------------------------------------------------------------------- 1 | # Revisiting Membership Inference Under Realistic Assumptions 2 | 3 | Please refer to [main README file](../README.md) for instructions on setup and installation. 4 | 5 | To replicate the results of the paper [*Revisiting Membership Inference Under Realistic Assumptions*](https://arxiv.org/abs/2005.10881), you would need to execute the `run_experiments.sh` shell script, which runs the `main.py` multiple times for different hyper-parameter settings and stores the results in the `results/$DATASET` folder. This is used for plotting the figures/tables in the paper. Note that the execution may take multiple days to complete on a single machine. For instance, for Purchase-100 data set, run the following command: 6 | ``` 7 | $ ./run_experiments.sh purchase_100 8 | ``` 9 | Note that the above script also installs all the required dependencies (in case not already installed), except cuda-toolkit and cudnn. For CIFAR-100 data set, update the hyper-parameters as commented inside the script, and run: 10 | ``` 11 | $ ./run_experiments.sh cifar_100 12 | ``` 13 | 14 | 15 | ### Plotting the results from the paper 16 | 17 | Run `python3 interpret_results.py $DATASET --l2_ratio=$LAMBDA` to obtain the plots and tabular results. For instance, to get the results for neural network model over CIFAR-100 data set, run: 18 | ``` 19 | $ python3 interpret_results.py cifar_100 --model='nn' --l2_ratio=1e-4 20 | ``` 21 | 22 | Other command-line arguments are as follows: 23 | - `--plot` specifies the type of plot to be printed 24 | - 'acc' prints the accuracy loss comparison plot (default) 25 | - 'priv' prints the privacy leakage plots and table values 26 | - 'scatter' runs the Morgan attack and plots the scatter plot of loss and Merlin ratio 27 | - `--gamma` specifies the gamma value to be used for the results: 1, 2 or 10 28 | - `--alpha` specifies the alpha threshold to be used to get the corresponding attack threshold: between 0 and 1 29 | - `--per_class_thresh` specifies whether to use per class threshold (1) or not (0 - default) 30 | - `--fixed_thresh` specfies if fixed threshold of expected training loss is to be used when using per class threshold: set to 1 for using fixed threshold (0 - default) 31 | - `--eps` specifies the epsilon value to be used when plotting 'priv' plots (None - default, i.e. no privacy) 32 | - `--mem` used in scatter plot to specifify which points are to be plotted: 'm' to plot only members, 'nm' to plot only non-members ('all' - default, i.e. plot both members and non-members) 33 | -------------------------------------------------------------------------------- /improved_mi/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bargavj/EvaluatingDPML/2f611fd20b4089019223039d95efa70a49c2b320/improved_mi/__init__.py -------------------------------------------------------------------------------- /improved_mi/interpret_results.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import argparse 3 | import pickle 4 | import numpy as np 5 | import matplotlib.pyplot as plt 6 | import os 7 | sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), "../"))) 8 | 9 | from core.utilities import get_fp 10 | from core.utilities import get_adv 11 | from core.utilities import get_ppv 12 | from core.utilities import get_inference_threshold 13 | from core.utilities import plot_histogram 14 | from core.utilities import plot_sign_histogram 15 | from sklearn.metrics import roc_curve 16 | 17 | EPS = list(np.arange(0.1, 100, 0.01)) 18 | EPS2 = list(np.arange(0.1, 100, 0.01)) 19 | EPSILONS = [0.1, 1.0, 10.0, 100.0] 20 | PERTURBATION = 'grad_pert_' 21 | DP = ['gdp_','rdp_'] 22 | TYPE = ['o-', '.-'] 23 | DP_LABELS = ['GDP', 'RDP'] 24 | RUNS = range(5) 25 | A, B = len(EPSILONS), len(RUNS) 26 | ALPHAS = np.arange(0.01, 1, 0.01) 27 | delta = 1e-5 28 | 29 | frame = plt.rcParams['figure.figsize'] 30 | new_rc_params = { 31 | 'figure.figsize': [frame[0], 0.8*frame[1]], 32 | 'font.size': 22, # 18 (20) default, changed to 22 33 | 'text.usetex': True, 34 | 'font.family': 'Times New Roman', 35 | 'mathtext.fontset': 'stix', 36 | 'xtick.major.pad': '8' 37 | } 38 | plt.rcParams.update(new_rc_params) 39 | 40 | 41 | def f(eps, delta, alpha): 42 | return max(0, 1 - delta - np.exp(eps) * alpha, np.exp(-eps) * (1 - delta - alpha)) 43 | 44 | 45 | def adv_lim(eps, delta, alpha): 46 | return 1 - f(eps, delta, alpha) - alpha 47 | 48 | 49 | def ppv_lim(eps, delta, alpha): 50 | return (1 - f(eps, delta, alpha)) / (1 - f(eps, delta, alpha) + gamma * alpha) 51 | 52 | 53 | def improved_limit(epsilons): 54 | return [max([adv_lim(eps, delta, alpha) for alpha in ALPHAS]) for eps in epsilons] 55 | 56 | 57 | def yeoms_limit(epsilons): 58 | return [np.exp(eps) - 1 for eps in epsilons] 59 | 60 | 61 | def get_data(): 62 | result = {} 63 | for dp in DP: 64 | epsilons = {} 65 | for eps in EPSILONS: 66 | runs = {} 67 | for run in RUNS: 68 | runs[run] = list(pickle.load(open(DATA_PATH+MODEL+PERTURBATION+dp+str(eps)+'_'+str(run+1)+'.p', 'rb'))) 69 | epsilons[eps] = runs 70 | result[dp] = epsilons 71 | runs = {} 72 | for run in RUNS: 73 | runs[run] = list(pickle.load(open(DATA_PATH+MODEL+'no_privacy_'+str(args.l2_ratio)+'_'+str(run+1)+'.p', 'rb'))) 74 | result['no_privacy'] = runs 75 | return result 76 | 77 | 78 | def get_data_multipleID(): 79 | allIDs = args.data_ids.split(",") 80 | allResult = [] 81 | for data_id in allIDs: 82 | directory = DATA_PATH+f'dataID_{data_id}/' 83 | result = {} 84 | for dp in DP: 85 | epsilons = {} 86 | for eps in EPSILONS: 87 | runs = {} 88 | for run in RUNS: 89 | runs[run] = list(pickle.load(open(directory+MODEL+PERTURBATION+dp+str(eps)+'_'+str(run+1)+'.p', 'rb'))) 90 | epsilons[eps] = runs 91 | result[dp] = epsilons 92 | runs = {} 93 | for run in RUNS: 94 | runs[run] = list(pickle.load(open(directory+MODEL+'no_privacy_'+str(args.l2_ratio)+'_'+str(run+1)+'.p', 'rb'))) 95 | result['no_privacy'] = runs 96 | allResult.append((result,data_id)) 97 | return allResult 98 | 99 | 100 | def pretty_position(X, Y, pos): 101 | return ((X[pos] + X[pos+1]) / 2, (Y[pos] + Y[pos+1]) / 2) 102 | 103 | 104 | def get_pred_mem_mi(per_instance_loss, shokri_mi_outputs, proposed_mi_outputs, method='yeom', fpr_threshold=None, per_class_thresh=False, fixed_thresh=False): 105 | # method == "yeom" runs an improved version of the Yeom attack that finds a better threshold than the original 106 | # method == "shokri" runs an improved version of the Shokri attack that finds a better threshold than the original 107 | # method == "merlin" runs a new attack, which uses the direction of the change in per-instance loss for the record 108 | true_y, v_true_y, v_membership, v_per_instance_loss, v_merlin_ratio, merlin_ratio = proposed_mi_outputs 109 | shokri_adv, shadow_pred_scores, target_pred_scores, shadow_membership, target_membership, shadow_class_labels, target_class_labels = shokri_mi_outputs 110 | if method == 'yeom': 111 | if per_class_thresh: 112 | classes = np.unique(true_y) 113 | pred_membership = np.zeros(len(v_membership)) 114 | threshs = [] 115 | for c in classes: 116 | c_indices = np.arange(len(true_y))[true_y == c] 117 | v_c_indices = np.arange(len(v_true_y))[v_true_y == c] 118 | if fixed_thresh: 119 | thresh = np.mean(v_per_instance_loss[list(filter(lambda i: i < 10000, v_c_indices))]) 120 | else: 121 | thresh = -get_inference_threshold(-v_per_instance_loss[v_c_indices], v_membership[v_c_indices], fpr_threshold) 122 | pred_membership[c_indices] = np.where(per_instance_loss[c_indices] <= thresh, 1, 0) 123 | threshs.append(thresh) 124 | print(max(0, min(threshs)), max(0, np.median(threshs)), max(0, max(threshs))) 125 | return max(0, threshs[0]), pred_membership 126 | else: 127 | thresh = -get_inference_threshold(-v_per_instance_loss, v_membership, fpr_threshold) 128 | return max(0, thresh), np.where(per_instance_loss <= thresh, 1, 0) 129 | elif method == 'shokri': 130 | if fixed_thresh: 131 | return 0.5, np.where(target_pred_scores[:,1] >= 0.5, 1, 0) 132 | if per_class_thresh: 133 | classes = np.unique(shadow_class_labels) 134 | pred_membership = np.zeros(len(target_membership)) 135 | threshs = [] 136 | for c in classes: 137 | target_indices = np.arange(len(target_class_labels))[target_class_labels == c] 138 | shadow_indices = np.arange(len(shadow_class_labels))[shadow_class_labels == c] 139 | thresh = get_inference_threshold(shadow_pred_scores[shadow_indices,1], shadow_membership[shadow_indices], fpr_threshold) 140 | pred_membership[target_indices] = np.where(target_pred_scores[target_indices,1] >= thresh, 1, 0) 141 | threshs.append(thresh) 142 | print(min(threshs), np.median(threshs), max(threshs)) 143 | return threshs[0], pred_membership 144 | else: 145 | thresh = get_inference_threshold(shadow_pred_scores[:,1], shadow_membership, fpr_threshold) 146 | return thresh, np.where(target_pred_scores[:,1] >= thresh, 1, 0) 147 | else: # In this case, run the Merlin attack. 148 | if per_class_thresh: 149 | classes = np.unique(true_y) 150 | pred_membership = np.zeros(len(v_membership)) 151 | threshs = [] 152 | for c in classes: 153 | c_indices = np.arange(len(true_y))[true_y == c] 154 | v_c_indices = np.arange(len(v_true_y))[v_true_y == c] 155 | thresh = get_inference_threshold(v_merlin_ratio[v_c_indices], v_membership[v_c_indices], fpr_threshold) 156 | pred_membership[c_indices] = np.where(merlin_ratio[c_indices] >= thresh, 1, 0) 157 | threshs.append(thresh) 158 | return threshs[0], pred_membership 159 | else: 160 | thresh = get_inference_threshold(v_merlin_ratio, v_membership, fpr_threshold) 161 | return thresh, np.where(merlin_ratio >= thresh, 1, 0) 162 | 163 | 164 | def plot_distributions(pred_vector, true_vector, method='yeom', data_id=None, run=None): 165 | fpr, tpr, phi = roc_curve(true_vector, pred_vector, pos_label=1) 166 | fpr, tpr, phi = np.array(fpr), np.array(tpr), np.array(phi) 167 | if method == 'yeom': 168 | fpr = 1 - fpr 169 | tpr = 1 - tpr 170 | PPV_A = tpr / (tpr + gamma * fpr) 171 | Adv_A = tpr - fpr 172 | fig, ax1 = plt.subplots() 173 | if method == 'yeom': 174 | phi, fpr, Adv_A, PPV_A = phi[:-1], fpr[:-1], Adv_A[:-1], PPV_A[:-1] 175 | ax1.plot(phi, Adv_A, label="Adv", color='black') 176 | ax1.plot(phi, PPV_A, label="PPV", color='black') 177 | #ax2 = ax1.twinx() 178 | ax1.plot(phi, fpr, label="FPR", color='black', linestyle='dashed') 179 | if method == 'yeom': 180 | ax1.set_xscale('log') 181 | ax1.annotate('$Adv_\mathcal{A}$', pretty_position(phi, Adv_A, np.argmax(Adv_A)), textcoords="offset points", xytext=(-15, -10), ha='right') 182 | ax1.annotate('$PPV_\mathcal{A}$', pretty_position(phi, PPV_A, -50), textcoords="offset points", xytext=(-40, 20), ha='left') 183 | ax1.annotate('FPR ($\\alpha$)', pretty_position(phi, fpr, 0), textcoords="offset points", xytext=(-20, -20), ha='right') 184 | #ax1.set_xticks([10**-6, 10**-4, 10**-2, 10**0]) 185 | ax1.set_xticks([10**-2, 10**-1, 10**0, 10**1]) # used for cifar-100 186 | else: 187 | ax1.annotate('$Adv_\mathcal{A}$', pretty_position(phi, Adv_A, np.argmax(Adv_A)), textcoords="offset points", xytext=(-15, 0), ha='right') 188 | ax1.annotate('$PPV_\mathcal{A}$', pretty_position(phi, PPV_A, 5), textcoords="offset points", xytext=(-10, -10), ha='right') 189 | ax1.annotate('FPR ($\\alpha$)', pretty_position(phi, fpr, -5), textcoords="offset points", xytext=(-40, -25), ha='left') 190 | ax1.set_xticks(np.arange(0, 1.1, step=0.2)) 191 | ax1.set_xlim(0, 1) 192 | ax1.set_xlabel('Decision Function $\phi$') 193 | ax1.set_ylabel('Privacy Leakage Metrics') 194 | #ax2.set_ylabel('False Positive Rate') 195 | ax1.set_yticks(np.arange(0, 1.1, step=0.2)) 196 | ax1.set_ylim(0, 1) 197 | #ax2.set_yticks(np.arange(0, 1.1, step=0.2)) 198 | fig.tight_layout() 199 | if data_id and run: 200 | directory = DATA_PATH+f'/dataID_{data_id}_figs/' 201 | if not os.path.exists(directory): 202 | os.makedirs(directory) 203 | plt.savefig(f"{directory}"+f'{method}_distribution_plot_run{run}.png') 204 | else: 205 | plt.show() 206 | 207 | 208 | def get_zeros(mem, vect): 209 | ind = list(filter(lambda i: vect[i] == 0, list(range(len(vect))))) 210 | #print(np.mean(vect[:10000]), np.std(vect[:10000])) 211 | #print(np.mean(vect[10000:]), np.std(vect[10000:])) 212 | return np.sum(mem[ind]), len(mem[ind]) - np.sum(mem[ind]) 213 | 214 | 215 | def benchmark_results(result, data_id=None): 216 | #this prints out benchmark results. The code is a combination of plot_accuracy(), plot_privacy_leakage() and morgan() 217 | print('=' * 10 + 'Benchmark Results' + '=' * 10) 218 | plot_accuracy(result,data_id=data_id) 219 | 220 | print('-' * 10 + 'Membership Inference Attack Benchmark Results' + '-' * 10) 221 | for dp in [None,'gdp_','rdp_']: 222 | for eps in EPSILONS: 223 | current_setting = 'no privacy' 224 | if dp=='gdp_': 225 | current_setting = 'GDP' 226 | elif dp=='rdp_': 227 | current_setting = 'RDP' 228 | 229 | print(f'->Attack Results for NN with {current_setting} '+(f'(epsilon={eps})' if dp else '')) 230 | plot_privacy_leakage(result,eps,dp,data_id) 231 | morgan(result) 232 | print('-'*10) 233 | print() 234 | 235 | 236 | def plot_accuracy(result, data_id=None): 237 | print('-' * 10 + 'Training and Testing Accuracies' + '-' * 10) 238 | print('->' + 'No Differential Privacy:') 239 | train_accs, baseline_acc = np.zeros(B), np.zeros(B) 240 | for run in RUNS: 241 | aux, membership, per_instance_loss, yeom_mi_outputs_1, yeom_mi_outputs_2, shokri_mi_outputs, proposed_mi_outputs = result['no_privacy'][run] 242 | train_loss, train_acc, test_loss, test_acc = aux 243 | baseline_acc[run] = test_acc 244 | train_accs[run] = train_acc 245 | baseline_acc = np.mean(baseline_acc) 246 | print(f"Train acc: {np.mean(train_accs)}, Test acc: {baseline_acc}\n") 247 | color = 0.1 248 | y = dict() 249 | 250 | for dp in DP: 251 | test_acc_vec = np.zeros((A, B)) 252 | for a, eps in enumerate(EPSILONS): 253 | train_accs = np.zeros(B) 254 | for run in RUNS: 255 | aux, membership, per_instance_loss, yeom_mi_outputs_1, yeom_mi_outputs_2, shokri_mi_outputs, proposed_mi_outputs = result[dp][eps][run] 256 | train_loss, train_acc, test_loss, test_acc = aux 257 | test_acc_vec[a, run] = test_acc 258 | train_accs[run] = train_acc 259 | print('->' + (f'GDP with epsilon={eps}:' if dp=='gdp_' else f'RDP with epsilon={eps}:')) 260 | print(f"Train acc: {np.mean(train_accs)}, Test acc: {np.mean(test_acc_vec[a])}\n") 261 | y[dp] = 1 - np.mean(test_acc_vec, axis=1) / baseline_acc 262 | plt.errorbar(EPSILONS, y[dp], yerr=np.std(test_acc_vec, axis=1), color=str(color), fmt='.-', capsize=2, label=DP_LABELS[DP.index(dp)]) 263 | color += 0.2 264 | plt.xscale('log') 265 | plt.xlabel('Privacy Budget ($\epsilon$)') 266 | plt.ylabel('Accuracy Loss') 267 | plt.yticks(np.arange(0, 1.1, step=0.2)) 268 | plt.xticks(EPSILONS) 269 | plt.annotate("RDP", pretty_position(EPSILONS, y["rdp_"], 1), textcoords="offset points", xytext=(20, 10), ha='right', color=str(0.3)) 270 | plt.annotate("GDP", pretty_position(EPSILONS, y["gdp_"], 1), textcoords="offset points", xytext=(-50, -10), ha='right', color=str(0.1)) 271 | plt.tight_layout() 272 | if data_id: 273 | directory = DATA_PATH+f'/dataID_{data_id}_figs/' 274 | if not os.path.exists(directory): 275 | os.makedirs(directory) 276 | plt.savefig(f"{directory}"+'accuracy_plot.png') 277 | else: 278 | plt.show() 279 | 280 | 281 | def plot_privacy_leakage(result, eps=None, dp='gdp_', data_id=None): 282 | adv_yeom_vanilla_1, adv_yeom, adv_shokri, adv_merlin = np.zeros(B), np.zeros(B), np.zeros(B), np.zeros(B) 283 | ppv_yeom_vanilla_1, ppv_yeom, ppv_shokri, ppv_merlin = np.zeros(B), np.zeros(B), np.zeros(B), np.zeros(B) 284 | fpr_yeom_vanilla_1, fpr_yeom, fpr_shokri, fpr_merlin = np.zeros(B), np.zeros(B), np.zeros(B), np.zeros(B) 285 | thresh_yeom_vanilla_1, thresh_yeom, thresh_shokri, thresh_merlin = np.zeros(B), np.zeros(B), np.zeros(B), np.zeros(B) 286 | yeom_zero_m, yeom_zero_nm, merlin_zero_m, merlin_zero_nm = [], [], [], [] 287 | for run in RUNS: 288 | aux, membership, per_instance_loss, yeom_mi_outputs_1, yeom_mi_outputs_2, shokri_mi_outputs, proposed_mi_outputs = result['no_privacy'][run] if not eps else result[dp][eps][run] 289 | train_loss, train_acc, test_loss, test_acc = aux 290 | shokri_adv, shadow_pred_scores, target_pred_scores, shadow_membership, target_membership, shadow_class_labels, target_class_labels = shokri_mi_outputs 291 | true_y, v_true_y, v_membership, v_per_instance_loss, v_merlin_ratio, merlin_ratio = proposed_mi_outputs 292 | m, nm = get_zeros(membership, per_instance_loss) 293 | yeom_zero_m.append(m) 294 | yeom_zero_nm.append(nm) 295 | m, nm = get_zeros(membership, merlin_ratio) 296 | merlin_zero_m.append(m) 297 | merlin_zero_nm.append(nm) 298 | #plot_histogram(per_instance_loss) 299 | #plot_distributions(per_instance_loss, membership, method='yeom') 300 | #plot_sign_histogram(membership, merlin_ratio, 100) 301 | plot_distributions(merlin_ratio, membership, method='merlin',data_id=data_id,run=run) 302 | # As used below, method == 'yeom' runs a Yeom attack but finds a better threshold than is used in the original Yeom attack. 303 | thresh, pred = get_pred_mem_mi(per_instance_loss, shokri_mi_outputs, proposed_mi_outputs, method='yeom', fpr_threshold=alpha, per_class_thresh=args.per_class_thresh, fixed_thresh=args.fixed_thresh) 304 | fp, adv, ppv = get_fp(membership, pred), get_adv(membership, pred), get_ppv(membership, pred) 305 | thresh_yeom[run], fpr_yeom[run], adv_yeom[run], ppv_yeom[run] = thresh, fp / (gamma * 10000), adv, ppv 306 | # As used below, method == 'shokri' runs a Shokri attack but finds a better threshold than is used in the original Yeom attack. 307 | thresh, pred = get_pred_mem_mi(per_instance_loss, shokri_mi_outputs, proposed_mi_outputs, method='shokri', fpr_threshold=alpha, per_class_thresh=args.per_class_thresh, fixed_thresh=args.fixed_thresh) 308 | fp, adv, ppv = get_fp(target_membership, pred), get_adv(target_membership, pred), get_ppv(target_membership, pred) 309 | thresh_shokri[run], fpr_shokri[run], adv_shokri[run], ppv_shokri[run] = thresh, fp / (args.gamma * 10000), adv, ppv 310 | # As used below, method == 'merlin' runs a new threshold-based membership inference attack that uses the direction of the change in per-instance loss for the record. 311 | thresh, pred = get_pred_mem_mi(per_instance_loss, shokri_mi_outputs, proposed_mi_outputs, method='merlin', fpr_threshold=alpha, per_class_thresh=args.per_class_thresh, fixed_thresh=args.fixed_thresh) 312 | fp, adv, ppv = get_fp(membership, pred), get_adv(membership, pred), get_ppv(membership, pred) 313 | thresh_merlin[run], fpr_merlin[run], adv_merlin[run], ppv_merlin[run] = thresh, fp / (gamma * 10000), adv, ppv 314 | # Original Yeom attack that uses expected training loss threshold 315 | fp, adv, ppv = get_fp(membership, yeom_mi_outputs_1), get_adv(membership, yeom_mi_outputs_1), get_ppv(membership, yeom_mi_outputs_1) 316 | thresh_yeom_vanilla_1[run], fpr_yeom_vanilla_1[run], adv_yeom_vanilla_1[run], ppv_yeom_vanilla_1[run] = train_loss, fp / (gamma * 10000), adv, ppv 317 | if data_id: 318 | printed_str = '' 319 | printed_str+='\nYeom: \t %.2f +/- %.2f \t %.2f +/- %.2f' % (np.mean(yeom_zero_m), np.std(yeom_zero_m), np.mean(yeom_zero_nm), np.std(yeom_zero_nm)) 320 | printed_str+='\nMerlin: \t %.2f +/- %.2f \t %.2f +/- %.2f' % (np.mean(merlin_zero_m), np.std(merlin_zero_m), np.mean(merlin_zero_nm), np.std(merlin_zero_nm)) 321 | printed_str+='\nYeom Vanilla 1:\nphi: %f +/- %f\nFPR: %.4f +/- %.4f\nTPR: %.4f +/- %.4f\nAdv: %.4f +/- %.4f\nPPV: %.4f +/- %.4f' % (np.mean(thresh_yeom_vanilla_1), np.std(thresh_yeom_vanilla_1), np.mean(fpr_yeom_vanilla_1), np.std(fpr_yeom_vanilla_1), np.mean(adv_yeom_vanilla_1+fpr_yeom_vanilla_1), np.std(adv_yeom_vanilla_1+fpr_yeom_vanilla_1), np.mean(adv_yeom_vanilla_1), np.std(adv_yeom_vanilla_1), np.mean(ppv_yeom_vanilla_1), np.std(ppv_yeom_vanilla_1)) 322 | printed_str+='\nYeom:\nphi: %f +/- %f\nFPR: %.4f +/- %.4f\nTPR: %.4f +/- %.4f\nAdv: %.4f +/- %.4f\nPPV: %.4f +/- %.4f' % (np.mean(thresh_yeom), np.std(thresh_yeom), np.mean(fpr_yeom), np.std(fpr_yeom), np.mean(adv_yeom+fpr_yeom), np.std(adv_yeom+fpr_yeom), np.mean(adv_yeom), np.std(adv_yeom), np.mean(ppv_yeom), np.std(ppv_yeom)) 323 | printed_str+='\nShokri:\nphi: %f +/- %f\nFPR: %.4f +/- %.4f\nTPR: %.4f +/- %.4f\nAdv: %.4f +/- %.4f\nPPV: %.4f +/- %.4f' % (np.mean(thresh_shokri), np.std(thresh_shokri), np.mean(fpr_shokri), np.std(fpr_shokri), np.mean(adv_shokri+fpr_shokri), np.std(adv_shokri+fpr_shokri), np.mean(adv_shokri), np.std(adv_shokri), np.mean(ppv_shokri), np.std(ppv_shokri)) 324 | printed_str+='\nMerlin:\nphi: %f +/- %f\nFPR: %.4f +/- %.4f\nTPR: %.4f +/- %.4f\nAdv: %.4f +/- %.4f\nPPV: %.4f +/- %.4f' % (np.mean(thresh_merlin), np.std(thresh_merlin), np.mean(fpr_merlin), np.std(fpr_merlin), np.mean(adv_merlin+fpr_merlin), np.std(adv_merlin+fpr_merlin), np.mean(adv_merlin), np.std(adv_merlin), np.mean(ppv_merlin), np.std(ppv_merlin)) 325 | directory = DATA_PATH+f'/dataID_{data_id}_figs/' 326 | if not os.path.exists(directory): 327 | os.makedirs(directory) 328 | with open(directory+f'privacy_leakage.txt', 'w') as f: 329 | f.write(printed_str) 330 | else: 331 | print('\nYeom: \t %.2f +/- %.2f \t %.2f +/- %.2f' % (np.mean(yeom_zero_m), np.std(yeom_zero_m), np.mean(yeom_zero_nm), np.std(yeom_zero_nm))) 332 | print('\nMerlin: \t %.2f +/- %.2f \t %.2f +/- %.2f' % (np.mean(merlin_zero_m), np.std(merlin_zero_m), np.mean(merlin_zero_nm), np.std(merlin_zero_nm))) 333 | print('\nYeom Vanilla 1:\nphi: %f +/- %f\nFPR: %.4f +/- %.4f\nTPR: %.4f +/- %.4f\nAdv: %.4f +/- %.4f\nPPV: %.4f +/- %.4f' % (np.mean(thresh_yeom_vanilla_1), np.std(thresh_yeom_vanilla_1), np.mean(fpr_yeom_vanilla_1), np.std(fpr_yeom_vanilla_1), np.mean(adv_yeom_vanilla_1+fpr_yeom_vanilla_1), np.std(adv_yeom_vanilla_1+fpr_yeom_vanilla_1), np.mean(adv_yeom_vanilla_1), np.std(adv_yeom_vanilla_1), np.mean(ppv_yeom_vanilla_1), np.std(ppv_yeom_vanilla_1))) 334 | print('\nYeom:\nphi: %f +/- %f\nFPR: %.4f +/- %.4f\nTPR: %.4f +/- %.4f\nAdv: %.4f +/- %.4f\nPPV: %.4f +/- %.4f' % (np.mean(thresh_yeom), np.std(thresh_yeom), np.mean(fpr_yeom), np.std(fpr_yeom), np.mean(adv_yeom+fpr_yeom), np.std(adv_yeom+fpr_yeom), np.mean(adv_yeom), np.std(adv_yeom), np.mean(ppv_yeom), np.std(ppv_yeom))) 335 | print('\nShokri:\nphi: %f +/- %f\nFPR: %.4f +/- %.4f\nTPR: %.4f +/- %.4f\nAdv: %.4f +/- %.4f\nPPV: %.4f +/- %.4f' % (np.mean(thresh_shokri), np.std(thresh_shokri), np.mean(fpr_shokri), np.std(fpr_shokri), np.mean(adv_shokri+fpr_shokri), np.std(adv_shokri+fpr_shokri), np.mean(adv_shokri), np.std(adv_shokri), np.mean(ppv_shokri), np.std(ppv_shokri))) 336 | print('\nMerlin:\nphi: %f +/- %f\nFPR: %.4f +/- %.4f\nTPR: %.4f +/- %.4f\nAdv: %.4f +/- %.4f\nPPV: %.4f +/- %.4f' % (np.mean(thresh_merlin), np.std(thresh_merlin), np.mean(fpr_merlin), np.std(fpr_merlin), np.mean(adv_merlin+fpr_merlin), np.std(adv_merlin+fpr_merlin), np.mean(adv_merlin), np.std(adv_merlin), np.mean(ppv_merlin), np.std(ppv_merlin))) 337 | 338 | 339 | def scatterplot(result, data_id=None): 340 | morgan(result) 341 | for run in RUNS: 342 | if args.eps == None: 343 | _, membership, per_instance_loss, _, _, _, proposed_mi_outputs = result['no_privacy'][run] 344 | else: 345 | _, membership, per_instance_loss, _, _, _, proposed_mi_outputs = result['gdp_'][args.eps][run] 346 | _, _, _, _, _, merlin_ratio = proposed_mi_outputs 347 | axes = np.vstack((per_instance_loss, merlin_ratio)) 348 | axes = np.vstack((membership, axes)) 349 | axes = np.transpose(axes) 350 | m_axes = axes[:10000, :] 351 | nm_axes = axes[10000:, :] 352 | axes = axes[np.argsort(axes[:, 1])] 353 | colors = np.array(['purple', 'orange']) 354 | 355 | if args.mem == 'nm': 356 | plt.scatter(nm_axes[:, 1], nm_axes[:, 2], s=np.pi * 3, c='purple', alpha=0.2) 357 | elif args.mem == 'm': 358 | plt.scatter(m_axes[:, 1], m_axes[:, 2], s=np.pi * 3, c='orange', alpha=0.2) 359 | else: 360 | plt.scatter(axes[:, 1], axes[:, 2], s=np.pi * 3, c=list(colors[list(map(lambda x:int(x), axes[:, 0]))]), alpha=0.2) 361 | # phi_l is lower threshold on loss 362 | # phi_u is upper threshold on loss 363 | # phi_m is threshold on merlin ratio 364 | phi_l, phi_u, phi_m = 0.0001, 0.0015, 0.95 365 | plt.plot([phi_l, phi_u], [phi_m, phi_m], c='k') 366 | plt.plot([phi_l, phi_l], [phi_m, 1], c='k') 367 | plt.plot([phi_u, phi_u], [phi_m, 1], c='k') 368 | plt.xscale('log') 369 | plt.xticks([10 ** -8, 10 ** -6, 10 ** -4, 10 ** -2, 10 ** 0, 10 **2]) 370 | plt.yticks([0.0, 0.2, 0.4, 0.6, 0.8, 1.0]) 371 | plt.ylim(0,1) 372 | plt.xlabel('Per-Instance Loss') 373 | plt.ylabel('Merlin Ratio') 374 | plt.tight_layout() 375 | if data_id: 376 | directory = DATA_PATH+f'/dataID_{data_id}_figs/' 377 | if not os.path.exists(directory): 378 | os.makedirs(directory) 379 | plt.savefig(f"{directory}"+f'scatterplot_run{run}.png') 380 | else: 381 | plt.show() 382 | 383 | 384 | def morgan(result): 385 | phi_l, phi_u, phi_m, fpr_morgan, adv_morgan, ppv_morgan = np.zeros(B), np.zeros(B), np.zeros(B), np.zeros(B), np.zeros(B), np.zeros(B) 386 | alpha_l = 0.22 # alpha value used to tune lower loss threshold 387 | alpha_u = 0.3 # alpha value used to tune upper loss threshold 388 | for run in RUNS: 389 | if args.eps == None: 390 | _, membership, per_instance_loss, _, _, shokri_mi_outputs, proposed_mi_outputs = result['no_privacy'][run] 391 | else: 392 | _, membership, per_instance_loss, _, _, shokri_mi_outputs, proposed_mi_outputs = result['gdp_'][args.eps][run] 393 | true_y, v_true_y, v_membership, v_per_instance_loss, v_merlin_ratio, merlin_ratio = proposed_mi_outputs 394 | low_thresh, _ = get_pred_mem_mi(per_instance_loss, shokri_mi_outputs, proposed_mi_outputs, method='yeom', fpr_threshold=alpha_l, per_class_thresh=args.per_class_thresh, fixed_thresh=args.fixed_thresh) 395 | high_thresh, _ = get_pred_mem_mi(per_instance_loss, shokri_mi_outputs, proposed_mi_outputs, method='yeom', fpr_threshold=alpha_u, per_class_thresh=args.per_class_thresh, fixed_thresh=args.fixed_thresh) 396 | merlin_thresh, _ = get_pred_mem_mi(per_instance_loss, shokri_mi_outputs, proposed_mi_outputs, method='merlin', fpr_threshold=alpha, per_class_thresh=args.per_class_thresh, fixed_thresh=args.fixed_thresh) 397 | pred_1 = np.where(per_instance_loss >= low_thresh, 1, 0) 398 | pred_2 = np.where(per_instance_loss <= high_thresh, 1, 0) 399 | pred_3 = np.where(merlin_ratio >= merlin_thresh, 1, 0) 400 | pred = pred_1 & pred_2 & pred_3 401 | fp, adv, ppv = get_fp(membership, pred), get_adv(membership, pred), get_ppv(membership, pred) 402 | phi_l[run], phi_u[run], phi_m[run], fpr_morgan[run], adv_morgan[run], ppv_morgan[run] = low_thresh, high_thresh, merlin_thresh, fp / (gamma * 10000), adv, ppv 403 | print('\nMorgan:\nphi: (%f +/- %f, %f +/- %f, %f +/- %f)\nFPR: %.4f +/- %.4f\nTPR: %.4f +/- %.4f\nAdv: %.4f +/- %.4f\nPPV: %.4f +/- %.4f' % (np.mean(phi_l), np.std(phi_l), np.mean(phi_u), np.std(phi_u), np.mean(phi_m), np.std(phi_m), np.mean(fpr_morgan), np.std(fpr_morgan), np.mean(adv_morgan+fpr_morgan), np.std(adv_morgan+fpr_morgan), np.mean(adv_morgan), np.std(adv_morgan), np.mean(ppv_morgan), np.std(ppv_morgan))) 404 | 405 | 406 | if __name__ == '__main__': 407 | parser = argparse.ArgumentParser() 408 | parser.add_argument('dataset', type=str) 409 | parser.add_argument('--model', type=str, default='nn') 410 | parser.add_argument('--l2_ratio', type=float, default=1e-8) 411 | parser.add_argument('--gamma', type=float, default=1.0) 412 | parser.add_argument('--alpha', type=float, default=None) 413 | parser.add_argument('--per_class_thresh', type=int, default=0) 414 | parser.add_argument('--fixed_thresh', type=int, default=0) 415 | parser.add_argument('--plot', type=str, default='acc') 416 | parser.add_argument('--eps', type=float, default=None) 417 | parser.add_argument('--mem', type=str, default='all') 418 | parser.add_argument('--data_ids', type=str, default='') 419 | args = parser.parse_args() 420 | print(vars(args)) 421 | 422 | gamma = args.gamma 423 | alpha = args.alpha 424 | DATA_PATH = './results/' + str(args.dataset) + '/' 425 | MODEL = str(gamma) + '_' + str(args.model) + '_' 426 | result = [] 427 | if args.data_ids: 428 | result = get_data_multipleID() 429 | else: 430 | result = [(get_data(),None)] 431 | for r,data_id in result: 432 | if args.plot == 'acc': 433 | plot_accuracy(r, data_id=data_id) 434 | elif args.plot == 'scatter': 435 | scatterplot(r, data_id=data_id) 436 | elif args.plot == 'benchmark': 437 | benchmark_results(r, data_id=data_id) 438 | else: 439 | plot_privacy_leakage(r, args.eps, data_id=data_id) 440 | -------------------------------------------------------------------------------- /improved_mi/main.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | import argparse 4 | import pickle 5 | import numpy as np 6 | 7 | sys.path.insert(0, os.path.abspath(os.path.join(os.getcwd(), "../"))) 8 | 9 | from core.attack import save_data 10 | from core.attack import load_data 11 | from core.attack import train_target_model 12 | from core.attack import yeom_membership_inference 13 | from core.attack import shokri_membership_inference 14 | from core.attack import proposed_membership_inference 15 | from core.attack import evaluate_proposed_membership_inference 16 | from core.utilities import log_loss 17 | 18 | RESULT_PATH = 'results/' 19 | 20 | if not os.path.exists(RESULT_PATH): 21 | os.makedirs(RESULT_PATH) 22 | 23 | 24 | def run_experiment(args): 25 | print('-' * 10 + 'TRAIN TARGET' + '-' * 10 + '\n') 26 | dataset = None 27 | v_dataset = None 28 | if args.data_id and int(args.data_id)>=0: 29 | dataset = load_data(f'target_data_{args.data_id}.npz', args) 30 | v_dataset = load_data(f'shadow0_data_{args.data_id}.npz', args) 31 | else: 32 | dataset = load_data(f'target_data.npz', args) 33 | v_dataset = load_data(f'shadow0_data.npz', args) 34 | train_x, train_y, test_x, test_y = dataset 35 | true_x = np.vstack((train_x, test_x)) 36 | true_y = np.append(train_y, test_y) 37 | batch_size = args.target_batch_size 38 | 39 | pred_y, membership, test_classes, classifier, aux = train_target_model( 40 | args=args, 41 | dataset=dataset, 42 | epochs=args.target_epochs, 43 | batch_size=args.target_batch_size, 44 | learning_rate=args.target_learning_rate, 45 | clipping_threshold=args.target_clipping_threshold, 46 | n_hidden=args.target_n_hidden, 47 | l2_ratio=args.target_l2_ratio, 48 | model=args.target_model, 49 | privacy=args.target_privacy, 50 | dp=args.target_dp, 51 | epsilon=args.target_epsilon, 52 | delta=args.target_delta, 53 | save=args.save_model) 54 | train_loss, train_acc, test_loss, test_acc = aux 55 | per_instance_loss = np.array(log_loss(true_y, pred_y)) 56 | 57 | # Yeom's membership inference attack when only train_loss is known 58 | yeom_mi_outputs_1 = yeom_membership_inference(per_instance_loss, membership, train_loss) 59 | # Yeom's membership inference attack when both train_loss and test_loss are known - Adversary 2 of Yeom et al. 60 | yeom_mi_outputs_2 = yeom_membership_inference(per_instance_loss, membership, train_loss, test_loss) 61 | 62 | # Shokri's membership inference attack 63 | shokri_mi_outputs = shokri_membership_inference(args, pred_y, membership, test_classes) 64 | 65 | # Proposed membership inference attacks 66 | proposed_mi_outputs = proposed_membership_inference(v_dataset, true_x, true_y, classifier, per_instance_loss, args) 67 | evaluate_proposed_membership_inference(per_instance_loss, membership, proposed_mi_outputs, fpr_threshold=0.01) 68 | evaluate_proposed_membership_inference(per_instance_loss, membership, proposed_mi_outputs, fpr_threshold=0.01, per_class_thresh=True) 69 | 70 | if not os.path.exists(RESULT_PATH+args.train_dataset): 71 | os.makedirs(RESULT_PATH+args.train_dataset) 72 | if args.data_id and int(args.data_id)>=0: 73 | directory = RESULT_PATH+args.train_dataset+f'/dataID_{args.data_id}/' 74 | if not os.path.exists(directory): 75 | os.makedirs(directory) 76 | if args.target_privacy == 'no_privacy': 77 | pickle.dump([aux, membership, per_instance_loss, yeom_mi_outputs_1, yeom_mi_outputs_2, shokri_mi_outputs, proposed_mi_outputs], open(directory+str(args.target_test_train_ratio)+'_'+args.target_model+'_no_privacy_'+str(args.target_l2_ratio)+'_'+str(args.run)+'.p', 'wb')) 78 | else: 79 | pickle.dump([aux, membership, per_instance_loss, yeom_mi_outputs_1, yeom_mi_outputs_2, shokri_mi_outputs, proposed_mi_outputs], open(directory+str(args.target_test_train_ratio)+'_'+args.target_model+'_'+args.target_privacy+'_'+args.target_dp+'_'+str(args.target_epsilon)+'_'+str(args.run)+'.p', 'wb')) 80 | else: 81 | if args.target_privacy == 'no_privacy': 82 | pickle.dump([aux, membership, per_instance_loss, yeom_mi_outputs_1, yeom_mi_outputs_2, shokri_mi_outputs, proposed_mi_outputs], open(RESULT_PATH+args.train_dataset+'/'+str(args.target_test_train_ratio)+'_'+args.target_model+'_no_privacy_'+str(args.target_l2_ratio)+'_'+str(args.run)+'.p', 'wb')) 83 | else: 84 | pickle.dump([aux, membership, per_instance_loss, yeom_mi_outputs_1, yeom_mi_outputs_2, shokri_mi_outputs, proposed_mi_outputs], open(RESULT_PATH+args.train_dataset+'/'+str(args.target_test_train_ratio)+'_'+args.target_model+'_'+args.target_privacy+'_'+args.target_dp+'_'+str(args.target_epsilon)+'_'+str(args.run)+'.p', 'wb')) 85 | 86 | 87 | if __name__ == '__main__': 88 | parser = argparse.ArgumentParser() 89 | parser.add_argument('train_dataset', type=str) 90 | parser.add_argument('--run', type=int, default=1) 91 | parser.add_argument('--use_cpu', type=int, default=0) 92 | parser.add_argument('--save_model', type=int, default=0) 93 | parser.add_argument('--save_data', type=int, default=0) 94 | # target and shadow model configuration 95 | parser.add_argument('--n_shadow', type=int, default=5) 96 | parser.add_argument('--target_data_size', type=int, default=int(1e4)) 97 | parser.add_argument('--target_test_train_ratio', type=float, default=1.0) 98 | parser.add_argument('--target_model', type=str, default='nn') 99 | parser.add_argument('--target_learning_rate', type=float, default=0.01) 100 | parser.add_argument('--target_batch_size', type=int, default=200) 101 | parser.add_argument('--target_n_hidden', type=int, default=256) 102 | parser.add_argument('--target_epochs', type=int, default=100) 103 | parser.add_argument('--target_l2_ratio', type=float, default=1e-8) 104 | parser.add_argument('--target_clipping_threshold', type=float, default=1.0) 105 | parser.add_argument('--target_privacy', type=str, default='no_privacy') 106 | parser.add_argument('--target_dp', type=str, default='dp') 107 | parser.add_argument('--target_epsilon', type=float, default=0.5) 108 | parser.add_argument('--target_delta', type=float, default=1e-5) 109 | # attack model configuration 110 | parser.add_argument('--attack_model', type=str, default='nn') 111 | parser.add_argument('--attack_learning_rate', type=float, default=0.01) 112 | parser.add_argument('--attack_batch_size', type=int, default=100) 113 | parser.add_argument('--attack_n_hidden', type=int, default=64) 114 | parser.add_argument('--attack_epochs', type=int, default=100) 115 | parser.add_argument('--attack_l2_ratio', type=float, default=1e-6) 116 | # Merlin's noise parameters 117 | parser.add_argument('--attack_noise_type', type=str, default='gaussian') 118 | parser.add_argument('--attack_noise_coverage', type=str, default='full') 119 | parser.add_argument('--attack_noise_magnitude', type=float, default=0.01) 120 | 121 | # specify datafile names 122 | parser.add_argument('--data_id', type=int, default=-1) 123 | 124 | # parse configuration 125 | args = parser.parse_args() 126 | print(vars(args)) 127 | 128 | # Flag to disable GPU 129 | if args.use_cpu: 130 | os.environ['CUDA_VISIBLE_DEVICES'] = '-1' 131 | else: 132 | os.environ['TF_XLA_FLAGS'] = '--tf_xla_enable_xla_devices' 133 | 134 | if args.save_data: 135 | if args.data_id and int(args.data_id) >= 0: 136 | save_data(args, args.data_id) 137 | else: 138 | save_data(args) 139 | else: 140 | run_experiment(args) 141 | -------------------------------------------------------------------------------- /improved_mi/run_experiments.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "Starting script" 4 | if [[ $# -eq 0 ]] ; then # if called with no arguments 5 | echo "Usage: bash $0 " 6 | exit 1 7 | fi 8 | 9 | DATASET=$1 10 | CODE=main.py 11 | 12 | echo "Loading modules" 13 | source /etc/profile.d/modules.sh 14 | module load python3.6.2 15 | module load cuda-toolkit-10 16 | module load cudnn-7.5.0 17 | 18 | # Make sure environment has dependencies 19 | echo "Creating environment" 20 | python3 -m venv env 21 | source env/bin/activate 22 | 23 | python3 -m pip install --upgrade pip 24 | python3 -m pip install -r ../requirements.txt 25 | 26 | echo "Filling data/ directory" 27 | # For Texas-100 and CIFAR-100, set --target_test_train_ratio=2 28 | # For RCV1, set --target_clipping_threshold=1 29 | python3 $CODE $DATASET --save_data=1 --target_clipping_threshold=4 --target_test_train_ratio=10 30 | 31 | echo "Beginning experiment" 32 | # For Purchase-100, set GAMMA in 0.1 0.5 1 2 10, --target_clipping_threshold=4, --target_epochs=100, --target_learning_rate=0.005, --target_l2_ratio=1e-8 33 | # For Texas-100, set GAMMA in 0.1 0.5 1 2, --target_clipping_threshold=4, --target_epochs=30, --target_learning_rate=0.005, --target_l2_ratio=1e-8 34 | # For RCV1, set GAMMA in 0.1 0.5 1 2 10, --target_clipping_threshold=1, --target_epochs=80, --target_learning_rate=0.003, --target_l2_ratio=1e-8 35 | # For CIFAR-100, set GAMMA in 0.1 0.5 1 2, --target_clipping_threshold=4, --target_epochs=100, --target_learning_rate=0.001, --target_l2_ratio=1e-4 36 | for GAMMA in 0.1 0.5 1 2 10 37 | do 38 | for RUN in 1 2 3 4 5 39 | do 40 | python3 $CODE $DATASET --target_test_train_ratio=$GAMMA --target_model='nn' --target_l2_ratio=1e-8 --target_learning_rate=0.005 --target_clipping_threshold=4 --target_privacy='no_privacy' --run=$RUN 41 | for EPSILON in 0.1 1.0 10.0 100.0 42 | do 43 | python3 $CODE $DATASET --target_test_train_ratio=$GAMMA --target_model='nn' --target_l2_ratio=1e-8 --target_learning_rate=0.005 --target_clipping_threshold=4 --target_privacy='grad_pert' --target_dp='rdp' --target_epsilon=$EPSILON --run=$RUN 44 | python3 $CODE $DATASET --target_test_train_ratio=$GAMMA --target_model='nn' --target_l2_ratio=1e-8 --target_learning_rate=0.005 --target_clipping_threshold=4 --target_privacy='grad_pert' --target_dp='gdp' --target_epsilon=$EPSILON --run=$RUN 45 | done 46 | done 47 | done 48 | echo done -------------------------------------------------------------------------------- /improved_mi/test_benchmark_output.txt: -------------------------------------------------------------------------------- 1 | {'dataset': 'census', 'model': 'nn', 'l2_ratio': 1e-08, 'gamma': 0.5, 'alpha': None, 'per_class_thresh': 0, 'fixed_thresh': 0, 'plot': 'benchmark', 'eps': None, 'mem': 'all'} 2 | ==========Benchmark Results========== 3 | ----------Training and Testing Accuracies---------- 4 | ->No Differential Privacy: 5 | Train acc: 0.8316200017929077, Test acc: 0.7417200088500977 6 | 7 | ->GDP with epsilon=0.1: 8 | Train acc: 0.6862400054931641, Test acc: 0.68575998544693 9 | 10 | ->GDP with epsilon=1.0: 11 | Train acc: 0.7427199959754944, Test acc: 0.7386399984359742 12 | 13 | ->GDP with epsilon=10.0: 14 | Train acc: 0.761739993095398, Test acc: 0.7565199971199036 15 | 16 | ->GDP with epsilon=100.0: 17 | Train acc: 0.772000002861023, Test acc: 0.7557600021362305 18 | 19 | ->RDP with epsilon=0.1: 20 | Train acc: 0.6687600016593933, Test acc: 0.6659599900245666 21 | 22 | ->RDP with epsilon=1.0: 23 | Train acc: 0.7415800094604492, Test acc: 0.7367199897766114 24 | 25 | ->RDP with epsilon=10.0: 26 | Train acc: 0.7534999966621398, Test acc: 0.7458400011062623 27 | 28 | ->RDP with epsilon=100.0: 29 | Train acc: 0.7747200131416321, Test acc: 0.7574400067329407 30 | 31 | ----------Membership Inference Attack Benchmark Results---------- 32 | ->Attack Results for NN with no privacy 33 | 34 | Yeom: 35 | phi: 0.507383 +/- 0.149175 36 | FPR: 0.6462 +/- 0.0798 37 | TPR: 0.7356 +/- 0.0805 38 | Adv: 0.0894 +/- 0.0060 39 | PPV: 0.6952 +/- 0.0039 40 | 41 | Shokri: 42 | phi: 1.102033 +/- 0.517191 43 | FPR: 0.2344 +/- 0.1918 44 | TPR: 0.2496 +/- 0.2038 45 | Adv: 0.0152 +/- 0.0159 46 | PPV: 0.4084 +/- 0.3335 47 | 48 | Merlin: 49 | phi: 0.506000 +/- 0.014967 50 | FPR: 0.4530 +/- 0.0830 51 | TPR: 0.5017 +/- 0.0861 52 | Adv: 0.0488 +/- 0.0063 53 | PPV: 0.6895 +/- 0.0036 54 | 55 | Morgan: 56 | phi: (0.011013 +/- 0.001697, 0.042215 +/- 0.005213, 0.506000 +/- 0.014967) 57 | FPR: 0.0341 +/- 0.0065 58 | TPR: 0.0455 +/- 0.0067 59 | Adv: 0.0113 +/- 0.0014 60 | PPV: 0.7284 +/- 0.0106 61 | ---------- 62 | 63 | ->Attack Results for NN with GDP (epsilon=0.1) 64 | 65 | Yeom: 66 | phi: 0.112975 +/- 0.122163 67 | FPR: 0.3713 +/- 0.2225 68 | TPR: 0.3684 +/- 0.2233 69 | Adv: -0.0029 +/- 0.0039 70 | PPV: 0.6625 +/- 0.0045 71 | 72 | Shokri: 73 | phi: 0.897478 +/- 0.417148 74 | FPR: 0.3493 +/- 0.2971 75 | TPR: 0.3515 +/- 0.2958 76 | Adv: 0.0022 +/- 0.0050 77 | PPV: 0.5353 +/- 0.2677 78 | 79 | Merlin: 80 | phi: 0.568000 +/- 0.040200 81 | FPR: 0.1846 +/- 0.1781 82 | TPR: 0.1849 +/- 0.1771 83 | Adv: 0.0003 +/- 0.0076 84 | PPV: 0.6725 +/- 0.0109 85 | 86 | Morgan: 87 | phi: (0.002818 +/- 0.001247, 0.008421 +/- 0.003056, 0.568000 +/- 0.040200) 88 | FPR: 0.0133 +/- 0.0128 89 | TPR: 0.0144 +/- 0.0132 90 | Adv: 0.0010 +/- 0.0006 91 | PPV: 0.6960 +/- 0.0275 92 | ---------- 93 | 94 | ->Attack Results for NN with GDP (epsilon=1.0) 95 | 96 | Yeom: 97 | phi: 1.083298 +/- 1.567769 98 | FPR: 0.5352 +/- 0.2686 99 | TPR: 0.5408 +/- 0.2701 100 | Adv: 0.0057 +/- 0.0033 101 | PPV: 0.6692 +/- 0.0021 102 | 103 | Shokri: 104 | phi: 0.884824 +/- 0.393388 105 | FPR: 0.2913 +/- 0.1517 106 | TPR: 0.2979 +/- 0.1546 107 | Adv: 0.0065 +/- 0.0035 108 | PPV: 0.5374 +/- 0.2687 109 | 110 | Merlin: 111 | phi: 0.562000 +/- 0.047074 112 | FPR: 0.2111 +/- 0.1623 113 | TPR: 0.2067 +/- 0.1603 114 | Adv: -0.0043 +/- 0.0054 115 | PPV: 0.6689 +/- 0.0331 116 | 117 | Morgan: 118 | phi: (0.000681 +/- 0.000152, 0.002590 +/- 0.000505, 0.562000 +/- 0.047074) 119 | FPR: 0.0185 +/- 0.0138 120 | TPR: 0.0182 +/- 0.0137 121 | Adv: -0.0003 +/- 0.0020 122 | PPV: 0.6216 +/- 0.0685 123 | ---------- 124 | 125 | ->Attack Results for NN with GDP (epsilon=10.0) 126 | 127 | Yeom: 128 | phi: 0.595231 +/- 0.762829 129 | FPR: 0.6483 +/- 0.1462 130 | TPR: 0.6537 +/- 0.1463 131 | Adv: 0.0054 +/- 0.0023 132 | PPV: 0.6686 +/- 0.0010 133 | 134 | Shokri: 135 | phi: 1.103187 +/- 0.510355 136 | FPR: 0.2195 +/- 0.1811 137 | TPR: 0.2256 +/- 0.1858 138 | Adv: 0.0061 +/- 0.0052 139 | PPV: 0.4037 +/- 0.3296 140 | 141 | Merlin: 142 | phi: 0.548000 +/- 0.056000 143 | FPR: 0.2743 +/- 0.2895 144 | TPR: 0.2719 +/- 0.2895 145 | Adv: -0.0024 +/- 0.0028 146 | PPV: 0.6595 +/- 0.0106 147 | 148 | Morgan: 149 | phi: (0.000545 +/- 0.000365, 0.002013 +/- 0.000788, 0.548000 +/- 0.056000) 150 | FPR: 0.0240 +/- 0.0229 151 | TPR: 0.0231 +/- 0.0226 152 | Adv: -0.0009 +/- 0.0024 153 | PPV: 0.6581 +/- 0.0159 154 | ---------- 155 | 156 | ->Attack Results for NN with GDP (epsilon=100.0) 157 | 158 | Yeom: 159 | phi: 0.628354 +/- 0.414732 160 | FPR: 0.7314 +/- 0.0624 161 | TPR: 0.7452 +/- 0.0624 162 | Adv: 0.0139 +/- 0.0041 163 | PPV: 0.6708 +/- 0.0013 164 | 165 | Shokri: 166 | phi: 0.694236 +/- 0.026221 167 | FPR: 0.4784 +/- 0.2150 168 | TPR: 0.4874 +/- 0.2133 169 | Adv: 0.0090 +/- 0.0026 170 | PPV: 0.6717 +/- 0.0025 171 | 172 | Merlin: 173 | phi: 0.500000 +/- 0.054772 174 | FPR: 0.5449 +/- 0.2993 175 | TPR: 0.5479 +/- 0.3027 176 | Adv: 0.0029 +/- 0.0069 177 | PPV: 0.6646 +/- 0.0086 178 | 179 | Morgan: 180 | phi: (0.001327 +/- 0.000457, 0.003797 +/- 0.000827, 0.500000 +/- 0.054772) 181 | FPR: 0.0466 +/- 0.0251 182 | TPR: 0.0493 +/- 0.0281 183 | Adv: 0.0027 +/- 0.0032 184 | PPV: 0.6660 +/- 0.0281 185 | ---------- 186 | 187 | ->Attack Results for NN with RDP (epsilon=0.1) 188 | 189 | Yeom: 190 | phi: 1.892745 +/- 3.774319 191 | FPR: 0.3792 +/- 0.3291 192 | TPR: 0.3785 +/- 0.3296 193 | Adv: -0.0007 +/- 0.0035 194 | PPV: 0.6672 +/- 0.0031 195 | 196 | Shokri: 197 | phi: 0.665221 +/- 0.027395 198 | FPR: 0.3735 +/- 0.0758 199 | TPR: 0.3817 +/- 0.0737 200 | Adv: 0.0083 +/- 0.0021 201 | PPV: 0.6722 +/- 0.0033 202 | 203 | Merlin: 204 | phi: 0.546000 +/- 0.073103 205 | FPR: 0.3497 +/- 0.2746 206 | TPR: 0.3482 +/- 0.2755 207 | Adv: -0.0015 +/- 0.0066 208 | PPV: 0.6832 +/- 0.0368 209 | 210 | Morgan: 211 | phi: (0.001633 +/- 0.000578, 0.005649 +/- 0.001913, 0.546000 +/- 0.073103) 212 | FPR: 0.0294 +/- 0.0232 213 | TPR: 0.0301 +/- 0.0238 214 | Adv: 0.0006 +/- 0.0009 215 | PPV: 0.7322 +/- 0.1341 216 | ---------- 217 | 218 | ->Attack Results for NN with RDP (epsilon=1.0) 219 | 220 | Yeom: 221 | phi: 0.698575 +/- 1.389485 222 | FPR: 0.3529 +/- 0.3162 223 | TPR: 0.3544 +/- 0.3162 224 | Adv: 0.0015 +/- 0.0017 225 | PPV: 0.6687 +/- 0.0021 226 | 227 | Shokri: 228 | phi: 0.869632 +/- 0.429443 229 | FPR: 0.3114 +/- 0.1594 230 | TPR: 0.3173 +/- 0.1623 231 | Adv: 0.0060 +/- 0.0030 232 | PPV: 0.5368 +/- 0.2684 233 | 234 | Merlin: 235 | phi: 0.548000 +/- 0.057411 236 | FPR: 0.2724 +/- 0.2986 237 | TPR: 0.2738 +/- 0.2992 238 | Adv: 0.0014 +/- 0.0041 239 | PPV: 0.6620 +/- 0.0158 240 | 241 | Morgan: 242 | phi: (0.000614 +/- 0.000222, 0.002356 +/- 0.000686, 0.548000 +/- 0.057411) 243 | FPR: 0.0223 +/- 0.0250 244 | TPR: 0.0228 +/- 0.0251 245 | Adv: 0.0005 +/- 0.0023 246 | PPV: 0.6434 +/- 0.0605 247 | ---------- 248 | 249 | ->Attack Results for NN with RDP (epsilon=10.0) 250 | 251 | Yeom: 252 | phi: 0.556676 +/- 0.521989 253 | FPR: 0.6425 +/- 0.1831 254 | TPR: 0.6527 +/- 0.1837 255 | Adv: 0.0103 +/- 0.0028 256 | PPV: 0.6706 +/- 0.0019 257 | 258 | Shokri: 259 | phi: 1.075861 +/- 0.526249 260 | FPR: 0.3317 +/- 0.3345 261 | TPR: 0.3362 +/- 0.3354 262 | Adv: 0.0046 +/- 0.0056 263 | PPV: 0.4027 +/- 0.3288 264 | 265 | Merlin: 266 | phi: 0.526000 +/- 0.065605 267 | FPR: 0.3997 +/- 0.3583 268 | TPR: 0.3968 +/- 0.3555 269 | Adv: -0.0029 +/- 0.0045 270 | PPV: 0.6658 +/- 0.0015 271 | 272 | Morgan: 273 | phi: (0.000648 +/- 0.000207, 0.002311 +/- 0.000640, 0.526000 +/- 0.065605) 274 | FPR: 0.0294 +/- 0.0252 275 | TPR: 0.0312 +/- 0.0270 276 | Adv: 0.0018 +/- 0.0021 277 | PPV: 0.6836 +/- 0.0242 278 | ---------- 279 | 280 | ->Attack Results for NN with RDP (epsilon=100.0) 281 | 282 | Yeom: 283 | phi: 0.402883 +/- 0.260705 284 | FPR: 0.6906 +/- 0.0570 285 | TPR: 0.7083 +/- 0.0577 286 | Adv: 0.0177 +/- 0.0010 287 | PPV: 0.6723 +/- 0.0004 288 | 289 | Shokri: 290 | phi: 0.682189 +/- 0.025650 291 | FPR: 0.7009 +/- 0.2689 292 | TPR: 0.7104 +/- 0.2679 293 | Adv: 0.0095 +/- 0.0031 294 | PPV: 0.6705 +/- 0.0027 295 | 296 | Merlin: 297 | phi: 0.512000 +/- 0.017205 298 | FPR: 0.4642 +/- 0.1177 299 | TPR: 0.4629 +/- 0.1171 300 | Adv: -0.0013 +/- 0.0055 301 | PPV: 0.6661 +/- 0.0025 302 | 303 | Morgan: 304 | phi: (0.001557 +/- 0.000460, 0.004460 +/- 0.000915, 0.512000 +/- 0.017205) 305 | FPR: 0.0413 +/- 0.0106 306 | TPR: 0.0442 +/- 0.0122 307 | Adv: 0.0029 +/- 0.0029 308 | PPV: 0.6799 +/- 0.0126 309 | ---------- 310 | 311 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | dm-tree 2 | matplotlib 3 | numpy 4 | pandas 5 | scipy 6 | scikit_learn 7 | tensorflow 8 | tensorflow-privacy 9 | bs4 10 | requests 11 | wget 12 | lxml 13 | zipfile38 14 | --------------------------------------------------------------------------------