├── GNU_radio ├── radio_setup_summary.pdf ├── wifi_rx.grc ├── wifi_rx.py ├── wifi_tx.grc └── wifi_tx.py ├── README.md ├── complex_rf ├── __init__.py ├── cfo_channel_testing_simulations.py ├── cfo_channel_training_simulations.py ├── config.py ├── configs_test.json ├── configs_train.json ├── cxnn │ ├── __init__.py │ ├── __init__.pyc │ ├── __pycache__ │ │ ├── models.cpython-38.pyc │ │ └── train.cpython-38.pyc │ ├── complexnn │ │ ├── LICENSE.md │ │ ├── __init__.py │ │ ├── __init__.pyc │ │ ├── __pycache__ │ │ │ ├── __init__.cpython-38.pyc │ │ │ ├── activations.cpython-38.pyc │ │ │ ├── bn.cpython-38.pyc │ │ │ ├── conv.cpython-38.pyc │ │ │ ├── dense.cpython-38.pyc │ │ │ ├── fft.cpython-38.pyc │ │ │ ├── init.cpython-38.pyc │ │ │ ├── norm.cpython-38.pyc │ │ │ ├── pool.cpython-38.pyc │ │ │ └── utils.cpython-38.pyc │ │ ├── activations.py │ │ ├── activations.pyc │ │ ├── bn.py │ │ ├── bn.pyc │ │ ├── conv.py │ │ ├── conv.pyc │ │ ├── dense.py │ │ ├── dense.pyc │ │ ├── fft.py │ │ ├── fft.pyc │ │ ├── init.py │ │ ├── init.pyc │ │ ├── norm.py │ │ ├── norm.pyc │ │ ├── pool.py │ │ ├── pool.pyc │ │ ├── utils.py │ │ └── utils.pyc │ ├── models.py │ ├── models.pyc │ ├── train.py │ ├── train.pyc │ └── train_network_reim_mag.py ├── experiment_setup.py ├── experiment_setup.pyc ├── load_slice_IQ.py ├── prepare4complex.py ├── requirements.txt └── utils.py ├── fine-tuning ├── .idea │ ├── .gitignore │ ├── fine-tuning.iml │ ├── inspectionProfiles │ │ ├── Project_Default.xml │ │ └── profiles_settings.xml │ ├── misc.xml │ └── modules.xml ├── DF_model.py ├── __pycache__ │ ├── augData.cpython-36.pyc │ ├── config.cpython-36.pyc │ ├── imagenet_utils.cpython-36.pyc │ ├── load_slice_IQ.cpython-36.pyc │ ├── radioConv.cpython-36.pyc │ ├── resnet50_1D.cpython-36.pyc │ ├── tools.cpython-36.pyc │ └── utils.cpython-36.pyc ├── augData.py ├── call.sh ├── config.py ├── finetune.py ├── imagenet_utils.py ├── load_slice_IQ.py ├── main.py ├── radioConv.py ├── resnet50_1D.py ├── resnet50_2D.py ├── tools.py ├── utility.py └── utils.py ├── requirements.txt └── rf ├── __pycache__ ├── config.cpython-36.pyc ├── get_simu_data.cpython-36.pyc ├── imagenet_utils.cpython-36.pyc ├── load_slice_IQ.cpython-36.pyc ├── radioConv.cpython-36.pyc ├── readSigmf2.cpython-36.pyc ├── resnet50_1D.cpython-36.pyc ├── resnet50_2D.cpython-36.pyc └── utils.cpython-36.pyc ├── compare_data.py ├── config.py ├── example.py ├── get_simu_data.py ├── heatMap.py ├── imagenet_utils.py ├── load_slice_IQ.py ├── main.py ├── prepare4complex.py ├── radioConv.py ├── readSigmf.py ├── readSigmf2.py ├── resnet50_1D.py ├── resnet50_2D.py ├── test_model.py └── utils.py /GNU_radio/radio_setup_summary.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/GNU_radio/radio_setup_summary.pdf -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Robust Deep-learning-based Radio Fingerprinting with Fine-Tuning 2 | 3 | **The dataset and code are for research purposes only. The results of this study are published in the following poster:** 4 | 5 | Haipeng Li, Chenggang Wang, Nirnimesh Ghose, Boyang Wang, "POSTER: Robust Deep-learning-based Radio Fingerprinting with Fine-Tuning," ACM Conference on Security and Privacy in Wireless and Mobile Networks (WiSec’21), June, 2021. 6 | 7 | The pdf of this poster can be found below (open access by ACM). 8 | 9 | https://dl.acm.org/doi/pdf/10.1145/3448300.3468253 10 | 11 | ## Videos 12 | 13 | Interview video from WiSec'21 14 | 15 | https://ceas.mediaspace.kaltura.com/media/WiSec21_Poster_Haipeng_Interview/1_p1dho7lf 16 | 17 | Demo Video 1 on running GNURadio code in this repository 18 | 19 | https://ceas.mediaspace.kaltura.com/media/WiSec21_Poster_Haipeng_Video_GNURadio/1_bvujudvo 20 | 21 | Demo Video 2 on running machine learning code in this repository 22 | 23 | https://ceas.mediaspace.kaltura.com/media/WiSec21_Poster_Haipeng_Video_ML/1_c3booanp 24 | 25 | 26 | ## Overview 27 | 28 | In this project, we study radio fingerprinting. Radio fingerprinting is a way of distinguishing a transmitter from others of the same type by analyzing raw I/Q data in wireless communications. Each transmitter has a unique fingerprint due to hardware imperfections on radio-frequency circuitry. The hardware imperfections include I/Q imbalance, phase noise, frequency offset, and sampling off-set. Specifically, given I/Q data collected on a receiver side, a receiver decides which transmitter it is in radio fingerprinting. 29 | 30 | Recent studies have shown that deep learning can obtain high accuracy in radio fingerprinting. However, as wireless channels keep changing in the real world, one primary challenge is that the deep neural networks for radio fingerprinting are not robust, where the training I/Q data are different from test I/Q data, especially in a cross-day scenario. We leverage fine-tuning to improve the robustness of deep-learning-based radio fingerprinting. 31 | 32 | This repository contains GNU Radio source code for receivers and transmitters in our testbed and source code for deep neural networks and fine-tuning. 33 | 34 | The ```GNU_radio``` directory includes source code for receivers and transmitters using HackRF Ones. 35 | 36 | The ```rf``` directory includes source code for reading I/Q data and generating I/Q traces for a neural network. 37 | 38 | The ```fine-tuning``` directory includes source code for three convolutional neural networks and fine-tuning. 39 | 40 | The ```complex_rf``` directory includes source code for complex_value neural networks (no fine-tuning is supported at this point). 41 | 42 | 43 | ## Testbed 44 | 45 | We built a testbed with 1 receiver and 5 transmitters in order to collect I/Q data. Each receiver/transmitter is a HackRF One (with ANT500 antenna) running with GNU Radio. We leverage the open-source GNU Radio code from https://github.com/bastibl/gr-ieee802-11 to establish WiFi transmissions (IEEE 802.11 a/g) with BPSK 1/2 modulation scheme between the receiver and transmitter. We captured the I/Q data at 2.45 GHz center frequency with 2MHz bandwidth and a 2MHz sampling rate. 46 | 47 | We collected raw I/Q data before FFT, after FFT and after WiFi Frame Equalizer. We collected 3 transmissions from each transmitter. Each transmission lasts for 30 seconds. We collected data from two different days in an indoor environment. Only I/Q data after WiFi Frame Equalizer are used in our current experiments. 48 | 49 | ## Dataset 50 | 51 | Our raw I/Q dataset can be found below (**last modified: May 2025**). 52 | 53 | https://mailuc-my.sharepoint.com/:f:/g/personal/wang2ba_ucmail_uc_edu/EjXyRTpV0Y5Dn-OjKlxKg8gBZWyq2PIHy5OPgh3bf3g4fg?e=8KTK9O 54 | 55 | **Note:** the above link needs to be updated every 6 months due to certain settings of OneDrive. If you find the link is expired and you cannot access the data, please feel free to email us (boyang.wang@uc.edu). We will update the link as soon as we can. Thanks! 56 | 57 | **Note:** the above dataset consists of raw I/Q data. To generate inputs for a neural network, you will need to use code in ```rf```. You can choose your own parameters (e.g., the length of each I/Q trace, the number of I/Q traces per transmitter, etc.). 58 | 59 | In our experiments, each I/Q trace has 288 consecutive I/Q samples. We randomly select 100,000 I/Q traces per transmitter. In other words, we have 500,000 I/Q traces overall for a neural network. We use 72% for training, 8% for validation, and 20% for testing. 60 | 61 | ## Contact 62 | 63 | If you have any questions, please feel free to contact us. 64 | 65 | Haipeng Li (li2hp@mail.uc.edu), University of Cincinnati 66 | 67 | Nirnimesh Ghose (nghose@unl.edu), University of Nebraska Lincoln 68 | 69 | Boyang Wang (boyang.wang@uc.edu), University of Cincinnati 70 | -------------------------------------------------------------------------------- /complex_rf/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/__init__.py -------------------------------------------------------------------------------- /complex_rf/cfo_channel_testing_simulations.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Real life channel and CFO experiments are done in this code. 3 | 4 | 5 | - Physical Layer Channel Simulation 6 | - Physical Layer CFO Simulation 7 | - Channel Equalization 8 | - CFO Compensation 9 | - Channel Augmentation 10 | - CFO Augmentation 11 | ''' 12 | 13 | import numpy as np 14 | from timeit import default_timer as timer 15 | import argparse 16 | # from tqdm import trange, tqdm 17 | import json 18 | import os 19 | # import matplotlib as mpl 20 | import copy 21 | from collections import OrderedDict as odict 22 | # import matplotlib.pyplot as plt 23 | 24 | from keras.regularizers import l2 25 | from keras.models import Model, load_model 26 | from keras.layers import Dense, Input, Activation, Conv1D, Dropout, GlobalAveragePooling1D, Lambda, Average 27 | from keras import optimizers, regularizers, losses 28 | from keras.utils import plot_model 29 | from keras import backend as K 30 | import keras 31 | 32 | from cxnn.complexnn import ComplexDense, ComplexConv1D, utils, Modrelu 33 | # from simulators import physical_layer_channel, physical_layer_cfo, cfo_compansator, equalize_channel, augment_with_channel_test, augment_with_cfo_test, get_residual 34 | from experiment_setup import get_arguments 35 | 36 | 37 | def test_experiments(architecture, config): 38 | 39 | sample_rate = config['sample_rate'] 40 | outfile = './data/test_dataset_2.npz' 41 | 42 | np_dict = np.load(outfile) 43 | dict_wifi = {} 44 | dict_wifi['x_train'] = np_dict['x_train.npy'] 45 | dict_wifi['y_train'] = np_dict['y_train.npy'] 46 | dict_wifi['x_validation'] = np_dict['x_val.npy'] 47 | dict_wifi['y_validation'] = np_dict['y_val.npy'] 48 | dict_wifi['x_test'] = np_dict['x_test.npy'] 49 | dict_wifi['y_test'] = np_dict['y_test.npy'] 50 | 51 | 52 | x_test_orig = dict_wifi['x_test'] 53 | y_test_orig = dict_wifi['y_test'] 54 | 55 | data_format = '{}-new'.format(architecture) 56 | 57 | num_train = dict_wifi['x_train'].shape[0] 58 | num_test = dict_wifi['x_test'].shape[0] 59 | num_classes = dict_wifi['y_train'].shape[1] 60 | 61 | # Checkpoint path 62 | checkpoint = str('./checkpoints/' + data_format) 63 | 64 | # if augment_channel is False: 65 | # num_aug_test = 0 66 | 67 | print(checkpoint) 68 | 69 | dict_wifi_no_aug = copy.deepcopy(dict_wifi) 70 | 71 | 72 | print("== BUILDING MODEL... ==") 73 | 74 | checkpoint_in = checkpoint 75 | 76 | if checkpoint_in is None: 77 | raise ValueError('Cannot test without a checkpoint') 78 | # data_input = Input(batch_shape=(batch_size, num_features, 2)) 79 | # output, model_name = network_20_2(data_input, num_classes, weight_decay) 80 | # densenet = Model(data_input, output) 81 | 82 | checkpoint_in = checkpoint_in + '.h5' 83 | densenet = load_model(checkpoint_in, 84 | custom_objects={'ComplexConv1D': ComplexConv1D, 85 | 'GetAbs': utils.GetAbs, 86 | 'Modrelu': Modrelu}) 87 | 88 | batch_size = 100 89 | 90 | num_test_aug = dict_wifi['x_test'].shape[0] 91 | 92 | output_dict = odict(acc=odict(), comp=odict(), loss=odict()) 93 | 94 | if num_test_aug != num_test: 95 | 96 | num_test_per_aug = num_test_aug // num_test 97 | 98 | embeddings = densenet.layers[-2].output 99 | 100 | model2 = Model(densenet.input, embeddings) 101 | 102 | logits_test = model2.predict(x=dict_wifi['x_test'], 103 | batch_size=batch_size, 104 | verbose=0) 105 | 106 | softmax_test = densenet.predict(x=dict_wifi['x_test'], 107 | batch_size=batch_size, 108 | verbose=0) 109 | 110 | layer_name = densenet.layers[-1].name 111 | weight, bias = densenet.get_layer(layer_name).get_weights() 112 | 113 | logits_test = logits_test.dot(weight) + bias 114 | 115 | logits_test_new = np.zeros((num_test, num_classes)) 116 | softmax_test_new = np.zeros((num_test, num_classes)) 117 | for i in range(num_test_per_aug): 118 | # list_x_test.append(x_test_aug[i*num_test:(i+1)*num_test]) 119 | 120 | logits_test_new += logits_test[i*num_test:(i+1)*num_test] 121 | softmax_test_new += softmax_test[i*num_test:(i+1)*num_test] 122 | 123 | # Adding LLRs for num_channel_aug_test test augmentations 124 | label_pred_llr = logits_test_new.argmax(axis=1) 125 | label_act = dict_wifi['y_test'][:num_test].argmax(axis=1) 126 | ind_correct = np.where(label_pred_llr == label_act)[0] 127 | ind_wrong = np.where(label_pred_llr != label_act)[0] 128 | assert (num_test == ind_wrong.size + ind_correct.size), 'Major calculation mistake!' 129 | test_acc_llr = 100.*ind_correct.size / num_test 130 | 131 | # Adding LLRs for num_channel_aug_test test augmentations 132 | label_pred_soft = softmax_test_new.argmax(axis=1) 133 | label_act = dict_wifi['y_test'][:num_test].argmax(axis=1) 134 | ind_correct = np.where(label_pred_soft == label_act)[0] 135 | ind_wrong = np.where(label_pred_soft != label_act)[0] 136 | assert (num_test == ind_wrong.size + ind_correct.size), 'Major calculation mistake!' 137 | test_acc_soft = 100.*ind_correct.size / num_test 138 | 139 | # 1 test augmentation 140 | probs = densenet.predict(x=dict_wifi['x_test'][:num_test], 141 | batch_size=batch_size, 142 | verbose=0) 143 | label_pred = probs.argmax(axis=1) 144 | ind_correct = np.where(label_pred == label_act)[0] 145 | ind_wrong = np.where(label_pred != label_act)[0] 146 | assert (num_test == ind_wrong.size + ind_correct.size), 'Major calculation mistake!' 147 | test_acc = 100.*ind_correct.size / num_test 148 | 149 | # No test augmentations 150 | probs = densenet.predict(x=dict_wifi_no_aug['x_test'], 151 | batch_size=batch_size, 152 | verbose=0) 153 | label_pred = probs.argmax(axis=1) 154 | label_act = y_test_orig.argmax(axis=1) 155 | ind_correct = np.where(label_pred == label_act)[0] 156 | ind_wrong = np.where(label_pred != label_act)[0] 157 | assert (num_test == ind_wrong.size + ind_correct.size), 'Major calculation mistake!' 158 | test_acc_no_aug = 100.*ind_correct.size / num_test 159 | 160 | # print("\n========================================") 161 | print('Test accuracy (0 aug): {:.2f}%'.format(test_acc_no_aug)) 162 | print('Test accuracy (1 aug): {:.2f}%'.format(test_acc)) 163 | print('Test accuracy ({} aug) llr: {:.2f}%'.format(num_test_per_aug, test_acc_llr)) 164 | print('Test accuracy ({} aug) softmax avg: {:.2f}%'.format(num_test_per_aug, test_acc_soft)) 165 | 166 | output_dict['acc']['test_zero_aug'] = test_acc_no_aug 167 | output_dict['acc']['test_one_aug'] = test_acc 168 | output_dict['acc']['test_many_aug'] = test_acc_llr 169 | output_dict['acc']['test_many_aug_soft_avg'] = test_acc_soft 170 | 171 | else: 172 | probs = densenet.predict(x=dict_wifi['x_test'], 173 | batch_size=batch_size, 174 | verbose=0) 175 | label_pred = probs.argmax(axis=1) 176 | label_act = y_test_orig.argmax(axis=1) 177 | ind_correct = np.where(label_pred == label_act)[0] 178 | ind_wrong = np.where(label_pred != label_act)[0] 179 | assert (dict_wifi['x_test'].shape[0] == ind_wrong.size + 180 | ind_correct.size), 'Major calculation mistake!' 181 | test_acc_no_aug = 100.*ind_correct.size / dict_wifi['x_test'].shape[0] 182 | 183 | # print("\n========================================") 184 | print('Test accuracy (no aug): {:.2f}%'.format(test_acc_no_aug)) 185 | output_dict['acc']['test'] = test_acc_no_aug 186 | 187 | return output_dict, num_test_aug // num_test 188 | 189 | 190 | if __name__ == '__main__': 191 | 192 | args = get_arguments() 193 | architecture = args.architecture 194 | n_val = 5 195 | with open("configs_test.json") as config_file: 196 | config = json.load(config_file, encoding='utf-8') 197 | 198 | test_output, total_aug_test = test_experiments(architecture, config) 199 | 200 | -------------------------------------------------------------------------------- /complex_rf/cfo_channel_training_simulations.py: -------------------------------------------------------------------------------- 1 | ''' 2 | export path_to_config="/home/rfml/wifi-rebuttal/wifi-fingerprinting-journal/config_cfo_channel.json" 3 | export path_to_data="/home/rfml/wifi-rebuttal/wifi-fingerprinting-journal/data" 4 | ''' 5 | 6 | # import matplotlib as mpl 7 | import os 8 | import json 9 | # from tqdm import trange, tqdm 10 | import argparse 11 | from timeit import default_timer as timer 12 | import numpy as np 13 | import os 14 | # import matplotlib.pyplot as plt 15 | 16 | 17 | # from simulators import signal_power_effect, plot_signals, physical_layer_channel, physical_layer_cfo, cfo_compansator, equalize_channel, augment_with_channel, augment_with_cfo, get_residual 18 | from cxnn.train import train_20, train_200 19 | # from preproc.fading_model import normalize, add_custom_fading_channel, add_freq_offset 20 | # from preproc.preproc_wifi import basic_equalize_preamble, offset_compensate_preamble 21 | from experiment_setup import get_arguments 22 | 23 | ''' 24 | Real life channel and CFO experiments are done in this code. 25 | 26 | 27 | - Physical Layer Channel Simulation 28 | - Physical Layer CFO Simulation 29 | - Channel Equalization 30 | - CFO Compensation 31 | - Channel Augmentation 32 | - CFO Augmentation 33 | ''' 34 | 35 | # 36 | 37 | def multiple_day_fingerprint(architecture, config): 38 | 39 | epochs = config['epochs'] 40 | 41 | num_aug_test = config['num_aug_test'] 42 | 43 | outfile = './data/test_dataset.npz' 44 | 45 | np_dict = np.load(outfile) 46 | dict_wifi = {} 47 | # import pdb 48 | # pdb.set_trace() 49 | dict_wifi['x_train'] = np_dict['x_train.npy'] 50 | dict_wifi['y_train'] = np_dict['y_train.npy'] 51 | dict_wifi['x_validation'] = np_dict['x_val.npy'] 52 | dict_wifi['y_validation'] = np_dict['y_val.npy'] 53 | dict_wifi['x_test'] = np_dict['x_test.npy'] 54 | dict_wifi['y_test'] = np_dict['y_test.npy'] 55 | dict_wifi['num_classes'] = dict_wifi['y_test'].shape[1] 56 | 57 | data_format = '{}'.format(architecture) 58 | sample_rate = 20 59 | print(data_format) 60 | # -------------------------------------------------------------------------------------------- 61 | # Train 62 | # -------------------------------------------------------------------------------------------- 63 | 64 | # Checkpoint path 65 | if not os.path.exists("./checkpoints/"): 66 | os.mkdir("./checkpoints/") 67 | checkpoint = str('./checkpoints/' + data_format) 68 | 69 | print(checkpoint) 70 | 71 | if sample_rate == 20: 72 | train_output, model_name, summary = train_20(dict_wifi, checkpoint_in=None, 73 | num_aug_test=num_aug_test, 74 | checkpoint_out=checkpoint, 75 | architecture=architecture, 76 | epochs=epochs) 77 | elif sample_rate == 200: 78 | train_output, model_name, summary = train_200(dict_wifi, checkpoint_in=None, 79 | num_aug_test=num_aug_test, 80 | checkpoint_out=checkpoint, 81 | architecture=architecture, 82 | epochs=epochs) 83 | else: 84 | raise NotImplementedError 85 | 86 | # -------------------------------------------------------------------------------------------- 87 | # Write in log file 88 | # -------------------------------------------------------------------------------------------- 89 | 90 | # Write logs 91 | if not os.path.exists("./logs/"): 92 | os.mkdir("./logs/") 93 | with open('./logs/' + data_format + '.txt', 'a+') as f: 94 | f.write('\n\n-----------------------\n'+str(model_name)+'\n\n') 95 | 96 | return train_output 97 | 98 | 99 | if __name__ == '__main__': 100 | 101 | # args = get_arguments() 102 | # 103 | # architecture = args.architecture 104 | architecture = 'reim2x' 105 | with open("./configs_train.json") as config_file: 106 | config = json.load(config_file, encoding='utf-8') 107 | 108 | log_name = 'cplx_test' 109 | 110 | train_output = multiple_day_fingerprint(architecture, config) 111 | 112 | with open("./logs/" + log_name + '.txt', 'a+') as f: 113 | 114 | for keys, dicts in train_output.items(): 115 | f.write(str(keys)+':\n') 116 | for key, value in dicts.items(): 117 | f.write('\t'+str(key)+': {:.2f}%'.format(value)+'\n') 118 | -------------------------------------------------------------------------------- /complex_rf/config.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | ''' 3 | This module is for universal arg parse 4 | ''' 5 | 6 | import argparse 7 | 8 | 9 | def parse_args(argv): 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument('-i', '--input', help='input file/dir') 12 | parser.add_argument('-o', '--output', help='output dir') 13 | parser.add_argument('-m', '--modelType', default='homegrown', help='choose from homegrown/baseline/resnet') 14 | parser.add_argument('-sp', '--splitType', default='random', help='choose from random/order') 15 | parser.add_argument('-v', '--verbose', action='store_true', help='') 16 | parser.add_argument('--D2', action='store_true', help='if set will return 2 dimension data') 17 | parser.add_argument('-n', '--normalize', action='store_true', help='') 18 | parser.add_argument('-ds', '--dataSource', help='choose from neu/simu') 19 | parser.add_argument('-cf', '--channel_first', action='store_true', help='if set channel first otherwise channel last') 20 | parser.add_argument('-l','--location',help='data collected location') 21 | opts = parser.parse_args() 22 | return opts 23 | -------------------------------------------------------------------------------- /complex_rf/configs_test.json: -------------------------------------------------------------------------------- 1 | { 2 | "_comment": "===== DATA CONFIGURATION =====", 3 | "_explanation": " --- ", 4 | "exp_dir": "./", 5 | "sample_duration": 16, 6 | "preprocess_type": 1, 7 | "sample_rate": 200, 8 | 9 | "_comment": "===== TRAINING CONFIGURATION =====", 10 | "_explanation": " --- ", 11 | "epochs": 100, 12 | 13 | "_comment": "===== PHYSICAL CHANNEL PARAMETERS =====", 14 | "_explanation": " --- ", 15 | "phy_method" : 20, 16 | "seed_phy_train": [0, 20, 40, 80, 100, 120, 140, 160, 180, 200, 220, 240, 260, 280, 300, 320, 340, 360, 380, 400], 17 | "seed_phy_test": 60, 18 | "channel_type_phy_train" : 1, 19 | "channel_type_phy_test": 1, 20 | "phy_noise": false, 21 | "snr_train_phy": 500, 22 | "snr_test_phy": 500, 23 | "beta_phy": 0, 24 | 25 | "_comment": "===== PHYSICAL CFO =====", 26 | "_explanation": " --- ", 27 | "phy_method_cfo":20, 28 | "df_phy_train": 40e-6, 29 | "df_phy_test": 40e-6, 30 | "seed_phy_train_cfo": [0, 20, 40, 80, 100, 120, 140, 160, 180, 200, 220, 240, 260, 280, 300, 320, 340, 360, 380, 400], 31 | "seed_phy_test_cfo": 60, 32 | "add_cfo": true, 33 | "remove_cfo": true, 34 | 35 | "_comment": "===== EQUALIZATION PARAMETERS =====", 36 | "_explanation": " --- ", 37 | "equalize_train" : false, 38 | "equalize_test" : false, 39 | "verbose_train": false, 40 | "verbose_test": false, 41 | 42 | "_comment": "===== AUGMENTATION CHANNEL PARAMETERS =====", 43 | "_explanation": " --- ", 44 | "channel_type_aug_train" : 1, 45 | "channel_type_aug_test" : 1, 46 | "num_aug_train": 10, 47 | "num_aug_test": 10, 48 | "aug_type": 1, 49 | "num_ch_train": -1, 50 | "num_ch_test": -1, 51 | "channel_method": "FFT", 52 | "noise_method": "reg", 53 | "delay_seed_aug_train": false, 54 | "delay_seed_aug_test": false, 55 | "keep_orig_train" : false, 56 | "keep_orig_test" : false, 57 | "snr_train": 500, 58 | "snr_test": 500, 59 | "beta": 0, 60 | 61 | "_comment": "===== AUGMENTATION CFO PARAMETERS =====", 62 | "_explanation": " --- ", 63 | "df_aug_train" : 40e-6, 64 | "rand_aug_train": "unif", 65 | "num_aug_train_cfo": 10, 66 | "keep_orig_train_cfo": false, 67 | "aug_type_cfo": 0, 68 | 69 | "keep_orig_test_cfo": false, 70 | "num_aug_test_cfo" : 10, 71 | "rand_aug_test" : "unif", 72 | "df_aug_test" : 40e-6 73 | 74 | 75 | } 76 | -------------------------------------------------------------------------------- /complex_rf/configs_train.json: -------------------------------------------------------------------------------- 1 | { 2 | "_comment": "===== DATA CONFIGURATION =====", 3 | "_explanation": " --- ", 4 | "exp_dir": "./", 5 | "sample_duration": 16, 6 | "preprocess_type": 1, 7 | "sample_rate": 200, 8 | 9 | "_comment": "===== TRAINING CONFIGURATION =====", 10 | "_explanation": " --- ", 11 | "epochs": 30, 12 | 13 | "_comment": "===== PHYSICAL CHANNEL PARAMETERS =====", 14 | "_explanation": " --- ", 15 | "phy_method" : 20, 16 | "seed_phy_train": [0, 20, 40, 80, 100, 120, 140, 160, 180, 200, 220, 240, 260, 280, 300, 320, 340, 360, 380, 400], 17 | "seed_phy_test": 60, 18 | "seed_phy_validation": 10000, 19 | "channel_type_phy_train" : 1, 20 | "channel_type_phy_test": 1, 21 | "phy_noise": false, 22 | "snr_train_phy": 500, 23 | "snr_test_phy": 500, 24 | "beta_phy": 0, 25 | 26 | "_comment": "===== PHYSICAL CFO =====", 27 | "_explanation": " --- ", 28 | "phy_method_cfo":20, 29 | "df_phy_train": 40e-6, 30 | "df_phy_test": 40e-6, 31 | "seed_phy_train_cfo": [0, 20, 40, 80, 100, 120, 140, 160, 180, 200, 220, 240, 260, 280, 300, 320, 340, 360, 380, 400], 32 | "seed_phy_test_cfo": 60, 33 | "seed_phy_validation_cfo": 10000, 34 | "add_cfo": true, 35 | "remove_cfo": true, 36 | 37 | "_comment": "===== EQUALIZATION PARAMETERS =====", 38 | "_explanation": " --- ", 39 | "equalize_train" : false, 40 | "equalize_test" : false, 41 | "verbose_train": false, 42 | "verbose_test": false, 43 | 44 | "_comment": "===== AUGMENTATION CHANNEL PARAMETERS =====", 45 | "_explanation": " --- ", 46 | "channel_type_aug_train" : 1, 47 | "channel_type_aug_test" : 1, 48 | "num_aug_train": 10, 49 | "num_aug_test": 1, 50 | "num_aug_validation": 1, 51 | "aug_type": 1, 52 | "num_ch_train": -1, 53 | "num_ch_test": -1, 54 | "channel_method": "FFT", 55 | "noise_method": "reg", 56 | "delay_seed_aug_train": false, 57 | "delay_seed_aug_test": false, 58 | "keep_orig_train" : false, 59 | "keep_orig_test" : false, 60 | "snr_train": 500, 61 | "snr_test": 500, 62 | "beta": 0, 63 | 64 | "_comment": "===== AUGMENTATION CFO PARAMETERS =====", 65 | "_explanation": " --- ", 66 | "df_aug_train" : 40e-6, 67 | "rand_aug_train": "unif", 68 | "num_aug_train_cfo": 10, 69 | "keep_orig_train_cfo": false, 70 | "aug_type_cfo": 0, 71 | 72 | "keep_orig_test_cfo": false, 73 | "num_aug_test_cfo" : 1, 74 | "rand_aug_test" : "unif", 75 | "df_aug_test" : 40e-6 76 | 77 | 78 | } 79 | -------------------------------------------------------------------------------- /complex_rf/cxnn/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/__init__.py -------------------------------------------------------------------------------- /complex_rf/cxnn/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/__init__.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/__pycache__/models.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/__pycache__/models.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/__pycache__/train.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/__pycache__/train.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Chiheb Trabelsi 4 | 5 | Copyright (c) 2017 Olexa Bilaniuk 6 | 7 | Copyright (c) 2017 Ying Zhang 8 | 9 | Copyright (c) 2017 Dmitriy Serdyuk 10 | 11 | Copyright (c) 2017 Sandeep Subramanian 12 | 13 | Copyright (c) 2017 João Felipe Santos 14 | 15 | Copyright (c) 2017 Soroush Mehri 16 | 17 | Copyright (c) 2017 Negar Rostamzadeh 18 | 19 | Copyright (c) 2017 Yoshua Bengio 20 | 21 | Copyright (c) 2017 Christopher J Pal 22 | 23 | Permission is hereby granted, free of charge, to any person obtaining a copy 24 | of this software and associated documentation files (the "Software"), to deal 25 | in the Software without restriction, including without limitation the rights 26 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 27 | copies of the Software, and to permit persons to whom the Software is 28 | furnished to do so, subject to the following conditions: 29 | 30 | The above copyright notice and this permission notice shall be included in all 31 | copies or substantial portions of the Software. 32 | 33 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 34 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 35 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 36 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 37 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 38 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 39 | SOFTWARE. 40 | -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__init__.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | # 5 | # Authors: Olexa Bilaniuk 6 | # 7 | # What this module includes by default: 8 | # import bn, conv, dense, fft, init, norm, pool 9 | 10 | from .bn import ComplexBatchNormalization as ComplexBN 11 | from .conv import (ComplexConv, 12 | ComplexConv1D, 13 | ComplexConv2D, 14 | ComplexConv3D, 15 | WeightNorm_Conv) 16 | from .dense import ComplexDense 17 | from .fft import fft, ifft, fft2, ifft2, FFT, IFFT, FFT2, IFFT2 18 | from .init import (ComplexIndependentFilters, IndependentFilters, 19 | ComplexInit, SqrtInit) 20 | from .norm import LayerNormalization, ComplexLayerNorm 21 | from .pool import SpectralPooling1D, SpectralPooling2D 22 | from .utils import (get_realpart, get_imagpart, getpart_output_shape, 23 | GetImag, GetReal, GetAbs) 24 | 25 | from .activations import Modrelu -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/__init__.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__pycache__/__init__.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/__pycache__/__init__.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__pycache__/activations.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/__pycache__/activations.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__pycache__/bn.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/__pycache__/bn.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__pycache__/conv.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/__pycache__/conv.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__pycache__/dense.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/__pycache__/dense.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__pycache__/fft.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/__pycache__/fft.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__pycache__/init.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/__pycache__/init.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__pycache__/norm.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/__pycache__/norm.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__pycache__/pool.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/__pycache__/pool.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/__pycache__/utils.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/__pycache__/utils.cpython-38.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/activations.py: -------------------------------------------------------------------------------- 1 | import keras 2 | from keras import backend as K 3 | from keras.engine.topology import Layer 4 | from keras.layers import Concatenate 5 | 6 | from cxnn.complexnn import ComplexDense, ComplexConv1D, utils 7 | 8 | class Modrelu(Layer): 9 | 10 | def __init__(self, **kwargs): 11 | super(Modrelu, self).__init__(**kwargs) 12 | 13 | def build(self, input_shape): 14 | # Create a trainable weight variable for this layer. 15 | self._b = self.add_weight(name='b', 16 | shape=(input_shape[-1]//2,), 17 | initializer='zeros', 18 | trainable=True) 19 | super(Modrelu, self).build(input_shape) # Be sure to call this at the end 20 | 21 | def call(self, x): 22 | 23 | real = utils.GetReal()(x) 24 | imag = utils.GetImag()(x) 25 | 26 | abs1 = K.relu(utils.GetAbs()(x)) 27 | abs2 = K.relu(utils.GetAbs()(x) - self._b) 28 | 29 | 30 | real = real * abs2 / (abs1+0.0000001) 31 | imag = imag * abs2 / (abs1+0.0000001) 32 | 33 | merged = Concatenate()([real, imag]) 34 | 35 | return merged 36 | 37 | def compute_output_shape(self, input_shape): 38 | return input_shape -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/activations.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/activations.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/bn.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/bn.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/conv.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/conv.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/dense.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | # 5 | # Authors: Chiheb Trabelsi 6 | # 7 | 8 | from keras import backend as K 9 | import sys; sys.path.append('.') 10 | from keras import backend as K 11 | from keras import activations, initializers, regularizers, constraints 12 | from keras.layers import Layer, InputSpec 13 | import numpy as np 14 | from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams 15 | 16 | 17 | class ComplexDense(Layer): 18 | """Regular complex densely-connected NN layer. 19 | `Dense` implements the operation: 20 | `real_preact = dot(real_input, real_kernel) - dot(imag_input, imag_kernel)` 21 | `imag_preact = dot(real_input, imag_kernel) + dot(imag_input, real_kernel)` 22 | `output = activation(K.concatenate([real_preact, imag_preact]) + bias)` 23 | where `activation` is the element-wise activation function 24 | passed as the `activation` argument, `kernel` is a weights matrix 25 | created by the layer, and `bias` is a bias vector created by the layer 26 | (only applicable if `use_bias` is `True`). 27 | Note: if the input to the layer has a rank greater than 2, then 28 | AN ERROR MESSAGE IS PRINTED. 29 | # Arguments 30 | units: Positive integer, dimensionality of each of the real part 31 | and the imaginary part. It is actualy the number of complex units. 32 | activation: Activation function to use 33 | (see keras.activations). 34 | If you don't specify anything, no activation is applied 35 | (ie. "linear" activation: `a(x) = x`). 36 | use_bias: Boolean, whether the layer uses a bias vector. 37 | kernel_initializer: Initializer for the complex `kernel` weights matrix. 38 | By default it is 'complex'. 39 | and the usual initializers could also be used. 40 | (see keras.initializers and init.py). 41 | bias_initializer: Initializer for the bias vector 42 | (see keras.initializers). 43 | kernel_regularizer: Regularizer function applied to 44 | the `kernel` weights matrix 45 | (see keras.regularizers). 46 | bias_regularizer: Regularizer function applied to the bias vector 47 | (see keras.regularizers). 48 | activity_regularizer: Regularizer function applied to 49 | the output of the layer (its "activation"). 50 | (see keras.regularizers). 51 | kernel_constraint: Constraint function applied to the kernel matrix 52 | (see keras.constraints). 53 | bias_constraint: Constraint function applied to the bias vector 54 | (see keras.constraints). 55 | # Input shape 56 | a 2D input with shape `(batch_size, input_dim)`. 57 | # Output shape 58 | For a 2D input with shape `(batch_size, input_dim)`, 59 | the output would have shape `(batch_size, units)`. 60 | """ 61 | 62 | def __init__(self, units, 63 | activation=None, 64 | use_bias=True, 65 | init_criterion='he', 66 | kernel_initializer='complex', 67 | bias_initializer='zeros', 68 | kernel_regularizer=None, 69 | bias_regularizer=None, 70 | activity_regularizer=None, 71 | kernel_constraint=None, 72 | bias_constraint=None, 73 | seed=None, 74 | **kwargs): 75 | if 'input_shape' not in kwargs and 'input_dim' in kwargs: 76 | kwargs['input_shape'] = (kwargs.pop('input_dim'),) 77 | super(ComplexDense, self).__init__(**kwargs) 78 | self.units = units 79 | self.activation = activations.get(activation) 80 | self.use_bias = use_bias 81 | self.init_criterion = init_criterion 82 | if kernel_initializer in {'complex'}: 83 | self.kernel_initializer = kernel_initializer 84 | else: 85 | self.kernel_initializer = initializers.get(kernel_initializer) 86 | self.bias_initializer = initializers.get(bias_initializer) 87 | self.kernel_regularizer = regularizers.get(kernel_regularizer) 88 | self.bias_regularizer = regularizers.get(bias_regularizer) 89 | self.activity_regularizer = regularizers.get(activity_regularizer) 90 | self.kernel_constraint = constraints.get(kernel_constraint) 91 | self.bias_constraint = constraints.get(bias_constraint) 92 | if seed is None: 93 | self.seed = np.random.randint(1, 10e6) 94 | else: 95 | self.seed = seed 96 | self.input_spec = InputSpec(ndim=2) 97 | self.supports_masking = True 98 | 99 | def build(self, input_shape): 100 | assert len(input_shape) == 2 101 | assert input_shape[-1] % 2 == 0 102 | input_dim = input_shape[-1] // 2 103 | data_format = K.image_data_format() 104 | kernel_shape = (input_dim, self.units) 105 | fan_in, fan_out = initializers._compute_fans( 106 | kernel_shape, 107 | data_format=data_format 108 | ) 109 | if self.init_criterion == 'he': 110 | s = K.sqrt(1. / fan_in) 111 | elif self.init_criterion == 'glorot': 112 | s = K.sqrt(1. / (fan_in + fan_out)) 113 | rng = RandomStreams(seed=self.seed) 114 | 115 | # Equivalent initialization using amplitude phase representation: 116 | """modulus = rng.rayleigh(scale=s, size=kernel_shape) 117 | phase = rng.uniform(low=-np.pi, high=np.pi, size=kernel_shape) 118 | def init_w_real(shape, dtype=None): 119 | return modulus * K.cos(phase) 120 | def init_w_imag(shape, dtype=None): 121 | return modulus * K.sin(phase)""" 122 | 123 | # Initialization using euclidean representation: 124 | def init_w_real(shape, dtype=None): 125 | return rng.normal( 126 | size=kernel_shape, 127 | avg=0, 128 | std=s, 129 | dtype=dtype 130 | ) 131 | def init_w_imag(shape, dtype=None): 132 | return rng.normal( 133 | size=kernel_shape, 134 | avg=0, 135 | std=s, 136 | dtype=dtype 137 | ) 138 | if self.kernel_initializer in {'complex'}: 139 | real_init = init_w_real 140 | imag_init = init_w_imag 141 | else: 142 | real_init = self.kernel_initializer 143 | imag_init = self.kernel_initializer 144 | 145 | self.real_kernel = self.add_weight( 146 | shape=kernel_shape, 147 | initializer=real_init, 148 | name='real_kernel', 149 | regularizer=self.kernel_regularizer, 150 | constraint=self.kernel_constraint 151 | ) 152 | self.imag_kernel = self.add_weight( 153 | shape=kernel_shape, 154 | initializer=imag_init, 155 | name='imag_kernel', 156 | regularizer=self.kernel_regularizer, 157 | constraint=self.kernel_constraint 158 | ) 159 | 160 | if self.use_bias: 161 | self.bias = self.add_weight( 162 | shape=(2 * self.units,), 163 | initializer=self.bias_initializer, 164 | name='bias', 165 | regularizer=self.bias_regularizer, 166 | constraint=self.bias_constraint 167 | ) 168 | else: 169 | self.bias = None 170 | 171 | self.input_spec = InputSpec(ndim=2, axes={-1: 2 * input_dim}) 172 | self.built = True 173 | 174 | def call(self, inputs): 175 | input_shape = K.shape(inputs) 176 | input_dim = input_shape[-1] // 2 177 | real_input = inputs[:, :input_dim] 178 | imag_input = inputs[:, input_dim:] 179 | 180 | cat_kernels_4_real = K.concatenate( 181 | [self.real_kernel, -self.imag_kernel], 182 | axis=-1 183 | ) 184 | cat_kernels_4_imag = K.concatenate( 185 | [self.imag_kernel, self.real_kernel], 186 | axis=-1 187 | ) 188 | cat_kernels_4_complex = K.concatenate( 189 | [cat_kernels_4_real, cat_kernels_4_imag], 190 | axis=0 191 | ) 192 | 193 | output = K.dot(inputs, cat_kernels_4_complex) 194 | 195 | if self.use_bias: 196 | output = K.bias_add(output, self.bias) 197 | if self.activation is not None: 198 | output = self.activation(output) 199 | 200 | return output 201 | 202 | def compute_output_shape(self, input_shape): 203 | assert input_shape and len(input_shape) == 2 204 | assert input_shape[-1] 205 | output_shape = list(input_shape) 206 | output_shape[-1] = 2 * self.units 207 | return tuple(output_shape) 208 | 209 | def get_config(self): 210 | if self.kernel_initializer in {'complex'}: 211 | ki = self.kernel_initializer 212 | else: 213 | ki = initializers.serialize(self.kernel_initializer) 214 | config = { 215 | 'units': self.units, 216 | 'activation': activations.serialize(self.activation), 217 | 'use_bias': self.use_bias, 218 | 'init_criterion': self.init_criterion, 219 | 'kernel_initializer': ki, 220 | 'bias_initializer': initializers.serialize(self.bias_initializer), 221 | 'kernel_regularizer': regularizers.serialize(self.kernel_regularizer), 222 | 'bias_regularizer': regularizers.serialize(self.bias_regularizer), 223 | 'activity_regularizer': regularizers.serialize(self.activity_regularizer), 224 | 'kernel_constraint': constraints.serialize(self.kernel_constraint), 225 | 'bias_constraint': constraints.serialize(self.bias_constraint), 226 | 'seed': self.seed, 227 | } 228 | base_config = super(ComplexDense, self).get_config() 229 | return dict(list(base_config.items()) + list(config.items())) 230 | 231 | -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/dense.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/dense.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/fft.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | # 5 | # Authors: Olexa Bilaniuk 6 | # 7 | 8 | import keras.backend as KB 9 | import keras.engine as KE 10 | import keras.layers as KL 11 | import keras.optimizers as KO 12 | import theano as T 13 | import theano.ifelse as TI 14 | import theano.tensor as TT 15 | import theano.tensor.fft as TTF 16 | import numpy as np 17 | 18 | 19 | # 20 | # FFT functions: 21 | # 22 | # fft(): Batched 1-D FFT (Input: (Batch, TimeSamples)) 23 | # ifft(): Batched 1-D IFFT (Input: (Batch, FreqSamples)) 24 | # fft2(): Batched 2-D FFT (Input: (Batch, TimeSamplesH, TimeSamplesW)) 25 | # ifft2(): Batched 2-D IFFT (Input: (Batch, FreqSamplesH, FreqSamplesW)) 26 | # 27 | 28 | def fft(z): 29 | B = z.shape[0]//2 30 | L = z.shape[1] 31 | C = TT.as_tensor_variable(np.asarray([[[1,-1]]], dtype=T.config.floatX)) 32 | Zr, Zi = TTF.rfft(z[:B], norm="ortho"), TTF.rfft(z[B:], norm="ortho") 33 | isOdd = TT.eq(L%2, 1) 34 | Zr = TI.ifelse(isOdd, TT.concatenate([Zr, C*Zr[:,1: ][:,::-1]], axis=1), 35 | TT.concatenate([Zr, C*Zr[:,1:-1][:,::-1]], axis=1)) 36 | Zi = TI.ifelse(isOdd, TT.concatenate([Zi, C*Zi[:,1: ][:,::-1]], axis=1), 37 | TT.concatenate([Zi, C*Zi[:,1:-1][:,::-1]], axis=1)) 38 | Zi = (C*Zi)[:,:,::-1] # Zi * i 39 | Z = Zr+Zi 40 | return TT.concatenate([Z[:,:,0], Z[:,:,1]], axis=0) 41 | def ifft(z): 42 | B = z.shape[0]//2 43 | L = z.shape[1] 44 | C = TT.as_tensor_variable(np.asarray([[[1,-1]]], dtype=T.config.floatX)) 45 | Zr, Zi = TTF.rfft(z[:B], norm="ortho"), TTF.rfft(z[B:]*-1, norm="ortho") 46 | isOdd = TT.eq(L%2, 1) 47 | Zr = TI.ifelse(isOdd, TT.concatenate([Zr, C*Zr[:,1: ][:,::-1]], axis=1), 48 | TT.concatenate([Zr, C*Zr[:,1:-1][:,::-1]], axis=1)) 49 | Zi = TI.ifelse(isOdd, TT.concatenate([Zi, C*Zi[:,1: ][:,::-1]], axis=1), 50 | TT.concatenate([Zi, C*Zi[:,1:-1][:,::-1]], axis=1)) 51 | Zi = (C*Zi)[:,:,::-1] # Zi * i 52 | Z = Zr+Zi 53 | return TT.concatenate([Z[:,:,0], Z[:,:,1]*-1], axis=0) 54 | def fft2(x): 55 | tt = x 56 | tt = KB.reshape(tt, (x.shape[0] *x.shape[1], x.shape[2])) 57 | tf = fft(tt) 58 | tf = KB.reshape(tf, (x.shape[0], x.shape[1], x.shape[2])) 59 | tf = KB.permute_dimensions(tf, (0, 2, 1)) 60 | tf = KB.reshape(tf, (x.shape[0] *x.shape[2], x.shape[1])) 61 | ff = fft(tf) 62 | ff = KB.reshape(ff, (x.shape[0], x.shape[2], x.shape[1])) 63 | ff = KB.permute_dimensions(ff, (0, 2, 1)) 64 | return ff 65 | def ifft2(x): 66 | ff = x 67 | ff = KB.permute_dimensions(ff, (0, 2, 1)) 68 | ff = KB.reshape(ff, (x.shape[0] *x.shape[2], x.shape[1])) 69 | tf = ifft(ff) 70 | tf = KB.reshape(tf, (x.shape[0], x.shape[2], x.shape[1])) 71 | tf = KB.permute_dimensions(tf, (0, 2, 1)) 72 | tf = KB.reshape(tf, (x.shape[0] *x.shape[1], x.shape[2])) 73 | tt = ifft(tf) 74 | tt = KB.reshape(tt, (x.shape[0], x.shape[1], x.shape[2])) 75 | return tt 76 | 77 | # 78 | # FFT Layers: 79 | # 80 | # FFT: Batched 1-D FFT (Input: (Batch, FeatureMaps, TimeSamples)) 81 | # IFFT: Batched 1-D IFFT (Input: (Batch, FeatureMaps, FreqSamples)) 82 | # FFT2: Batched 2-D FFT (Input: (Batch, FeatureMaps, TimeSamplesH, TimeSamplesW)) 83 | # IFFT2: Batched 2-D IFFT (Input: (Batch, FeatureMaps, FreqSamplesH, FreqSamplesW)) 84 | # 85 | 86 | class FFT(KL.Layer): 87 | def call(self, x, mask=None): 88 | a = KB.permute_dimensions(x, (1,0,2)) 89 | a = KB.reshape(a, (x.shape[1] *x.shape[0], x.shape[2])) 90 | a = fft(a) 91 | a = KB.reshape(a, (x.shape[1], x.shape[0], x.shape[2])) 92 | return KB.permute_dimensions(a, (1,0,2)) 93 | class IFFT(KL.Layer): 94 | def call(self, x, mask=None): 95 | a = KB.permute_dimensions(x, (1,0,2)) 96 | a = KB.reshape(a, (x.shape[1] *x.shape[0], x.shape[2])) 97 | a = ifft(a) 98 | a = KB.reshape(a, (x.shape[1], x.shape[0], x.shape[2])) 99 | return KB.permute_dimensions(a, (1,0,2)) 100 | class FFT2(KL.Layer): 101 | def call(self, x, mask=None): 102 | a = KB.permute_dimensions(x, (1,0,2,3)) 103 | a = KB.reshape(a, (x.shape[1] *x.shape[0], x.shape[2], x.shape[3])) 104 | a = fft2(a) 105 | a = KB.reshape(a, (x.shape[1], x.shape[0], x.shape[2], x.shape[3])) 106 | return KB.permute_dimensions(a, (1,0,2,3)) 107 | class IFFT2(KL.Layer): 108 | def call(self, x, mask=None): 109 | a = KB.permute_dimensions(x, (1,0,2,3)) 110 | a = KB.reshape(a, (x.shape[1] *x.shape[0], x.shape[2], x.shape[3])) 111 | a = ifft2(a) 112 | a = KB.reshape(a, (x.shape[1], x.shape[0], x.shape[2], x.shape[3])) 113 | return KB.permute_dimensions(a, (1,0,2,3)) 114 | 115 | 116 | 117 | # 118 | # Tests 119 | # 120 | # Note: The IFFT is the conjugate of the FFT of the conjugate. 121 | # 122 | # np.fft.ifft(x) == np.conj(np.fft.fft(np.conj(x))) 123 | # 124 | 125 | if __name__ == "__main__": 126 | # Numpy 127 | np.random.seed(1) 128 | L = 19 129 | r = np.random.normal(0.8, size=(L,)) 130 | i = np.random.normal(0.8, size=(L,)) 131 | x = r+i*1j 132 | R = np.fft.rfft(r, norm="ortho") 133 | I = np.fft.rfft(i, norm="ortho") 134 | X = np.fft.fft (x, norm="ortho") 135 | 136 | if L&1: 137 | R = np.concatenate([R, np.conj(R[1: ][::-1])]) 138 | I = np.concatenate([I, np.conj(I[1: ][::-1])]) 139 | else: 140 | R = np.concatenate([R, np.conj(R[1:-1][::-1])]) 141 | I = np.concatenate([I, np.conj(I[1:-1][::-1])]) 142 | Y = R+I*1j 143 | print np.allclose(X, Y) 144 | 145 | 146 | # Theano 147 | z = TT.dmatrix() 148 | f = T.function([z], ifft(fft(z))) 149 | v = np.concatenate([np.real(x)[np.newaxis,:], np.imag(x)[np.newaxis,:]], axis=0) 150 | print v 151 | print f(v) 152 | print np.allclose(v, f(v)) 153 | 154 | 155 | # Keras 156 | x = i = KL.Input(shape=(128, 32,32)) 157 | x = IFFT2()(x) 158 | model = KE.Model([i],[x]) 159 | 160 | loss = "mse" 161 | opt = KO.Adam() 162 | 163 | model.compile(opt, loss) 164 | model._make_train_function() 165 | model._make_predict_function() 166 | model._make_test_function() 167 | 168 | v = np.random.normal(size=(13,128,32,32)) 169 | #print v 170 | V = model.predict(v) 171 | #print V 172 | print V.shape 173 | -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/fft.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/fft.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/init.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/init.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/norm.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/norm.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/pool.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | # 5 | # Authors: Olexa Bilaniuk 6 | # 7 | 8 | import keras.backend as KB 9 | import keras.engine as KE 10 | import keras.layers as KL 11 | import keras.optimizers as KO 12 | import theano as T 13 | import theano.ifelse as TI 14 | import theano.tensor as TT 15 | import theano.tensor.fft as TTF 16 | import numpy as np 17 | 18 | 19 | # 20 | # Spectral Pooling Layer 21 | # 22 | 23 | class SpectralPooling1D(KL.Layer): 24 | def __init__(self, topf=(0,)): 25 | super(SpectralPooling1D, self).__init__() 26 | if "topf" in kwargs: 27 | self.topf = (int (kwargs["topf" ][0]),) 28 | self.topf = (self.topf[0]//2,) 29 | elif "gamma" in kwargs: 30 | self.gamma = (float(kwargs["gamma"][0]),) 31 | self.gamma = (self.gamma[0]/2,) 32 | else: 33 | raise RuntimeError("Must provide either topf= or gamma= !") 34 | def call(self, x, mask=None): 35 | xshape = x._keras_shape 36 | if hasattr(self, "topf"): 37 | topf = self.topf 38 | else: 39 | if KB.image_data_format() == "channels_first": 40 | topf = (int(self.gamma[0]*xshape[2]),) 41 | else: 42 | topf = (int(self.gamma[0]*xshape[1]),) 43 | 44 | if KB.image_data_format() == "channels_first": 45 | if topf[0] > 0 and xshape[2] >= 2*topf[0]: 46 | mask = [1]*(topf[0] ) +\ 47 | [0]*(xshape[2] - 2*topf[0]) +\ 48 | [1]*(topf[0] ) 49 | mask = [[mask]] 50 | mask = np.asarray(mask, dtype=KB.floatx()).transpose((0,1,2)) 51 | mask = KB.constant(mask) 52 | x *= mask 53 | else: 54 | if topf[0] > 0 and xshape[1] >= 2*topf[0]: 55 | mask = [1]*(topf[0] ) +\ 56 | [0]*(xshape[1] - 2*topf[0]) +\ 57 | [1]*(topf[0] ) 58 | mask = [[mask]] 59 | mask = np.asarray(mask, dtype=KB.floatx()).transpose((0,2,1)) 60 | mask = KB.constant(mask) 61 | x *= mask 62 | 63 | return x 64 | class SpectralPooling2D(KL.Layer): 65 | def __init__(self, **kwargs): 66 | super(SpectralPooling2D, self).__init__() 67 | if "topf" in kwargs: 68 | self.topf = (int (kwargs["topf" ][0]), int (kwargs["topf" ][1])) 69 | self.topf = (self.topf[0]//2, self.topf[1]//2) 70 | elif "gamma" in kwargs: 71 | self.gamma = (float(kwargs["gamma"][0]), float(kwargs["gamma"][1])) 72 | self.gamma = (self.gamma[0]/2, self.gamma[1]/2) 73 | else: 74 | raise RuntimeError("Must provide either topf= or gamma= !") 75 | def call(self, x, mask=None): 76 | xshape = x._keras_shape 77 | if hasattr(self, "topf"): 78 | topf = self.topf 79 | else: 80 | if KB.image_data_format() == "channels_first": 81 | topf = (int(self.gamma[0]*xshape[2]), int(self.gamma[1]*xshape[3])) 82 | else: 83 | topf = (int(self.gamma[0]*xshape[1]), int(self.gamma[1]*xshape[2])) 84 | 85 | if KB.image_data_format() == "channels_first": 86 | if topf[0] > 0 and xshape[2] >= 2*topf[0]: 87 | mask = [1]*(topf[0] ) +\ 88 | [0]*(xshape[2] - 2*topf[0]) +\ 89 | [1]*(topf[0] ) 90 | mask = [[[mask]]] 91 | mask = np.asarray(mask, dtype=KB.floatx()).transpose((0,1,3,2)) 92 | mask = KB.constant(mask) 93 | x *= mask 94 | if topf[1] > 0 and xshape[3] >= 2*topf[1]: 95 | mask = [1]*(topf[1] ) +\ 96 | [0]*(xshape[3] - 2*topf[1]) +\ 97 | [1]*(topf[1] ) 98 | mask = [[[mask]]] 99 | mask = np.asarray(mask, dtype=KB.floatx()).transpose((0,1,2,3)) 100 | mask = KB.constant(mask) 101 | x *= mask 102 | else: 103 | if topf[0] > 0 and xshape[1] >= 2*topf[0]: 104 | mask = [1]*(topf[0] ) +\ 105 | [0]*(xshape[1] - 2*topf[0]) +\ 106 | [1]*(topf[0] ) 107 | mask = [[[mask]]] 108 | mask = np.asarray(mask, dtype=KB.floatx()).transpose((0,3,1,2)) 109 | mask = KB.constant(mask) 110 | x *= mask 111 | if topf[1] > 0 and xshape[2] >= 2*topf[1]: 112 | mask = [1]*(topf[1] ) +\ 113 | [0]*(xshape[2] - 2*topf[1]) +\ 114 | [1]*(topf[1] ) 115 | mask = [[[mask]]] 116 | mask = np.asarray(mask, dtype=KB.floatx()).transpose((0,1,3,2)) 117 | mask = KB.constant(mask) 118 | x *= mask 119 | 120 | return x 121 | 122 | 123 | if __name__ == "__main__": 124 | import cv2, sys 125 | import __main__ as SP 126 | import fft as CF 127 | 128 | # Build Model 129 | x = i = KL.Input(shape=(6,512,512)) 130 | f = CF.FFT2()(x) 131 | p = SP.SpectralPooling2D(gamma=[0.15,0.15])(f) 132 | o = CF.IFFT2()(p) 133 | 134 | model = KE.Model([i], [f,p,o]) 135 | model.compile("sgd", "mse") 136 | 137 | # Use it 138 | img = cv2.imread(sys.argv[1]) 139 | imgBatch = img[np.newaxis,...].transpose((0,3,1,2)) 140 | imgBatch = np.concatenate([imgBatch, np.zeros_like(imgBatch)], axis=1) 141 | f,p,o = model.predict(imgBatch) 142 | ffted = np.sqrt(np.sum(f[:,:3]**2 + f[:,3:]**2, axis=1)) 143 | ffted = ffted .transpose((1,2,0))/255 144 | pooled = np.sqrt(np.sum(p[:,:3]**2 + p[:,3:]**2, axis=1)) 145 | pooled = pooled.transpose((1,2,0))/255 146 | filtered = np.clip(o,0,255).transpose((0,2,3,1))[0,:,:,:3].astype("uint8") 147 | 148 | # Display it 149 | cv2.imshow("Original", img) 150 | cv2.imshow("FFT", ffted) 151 | cv2.imshow("Pooled", pooled) 152 | cv2.imshow("Filtered", filtered) 153 | cv2.waitKey(0) 154 | -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/pool.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/pool.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/utils.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | # 5 | # Authors: Dmitriy Serdyuk, Olexa Bilaniuk, Chiheb Trabelsi 6 | 7 | import keras.backend as K 8 | from keras.layers import Layer, Lambda 9 | 10 | # 11 | # GetReal/GetImag Lambda layer Implementation 12 | # 13 | 14 | 15 | def get_realpart(x): 16 | image_format = K.image_data_format() 17 | ndim = K.ndim(x) 18 | input_shape = K.shape(x) 19 | 20 | if (image_format == 'channels_first' and ndim != 3) or ndim == 2: 21 | input_dim = input_shape[1] // 2 22 | return x[:, :input_dim] 23 | 24 | input_dim = input_shape[-1] // 2 25 | if ndim == 3: 26 | return x[:, :, :input_dim] 27 | elif ndim == 4: 28 | return x[:, :, :, :input_dim] 29 | elif ndim == 5: 30 | return x[:, :, :, :, :input_dim] 31 | 32 | 33 | def get_imagpart(x): 34 | image_format = K.image_data_format() 35 | ndim = K.ndim(x) 36 | input_shape = K.shape(x) 37 | 38 | if (image_format == 'channels_first' and ndim != 3) or ndim == 2: 39 | input_dim = input_shape[1] // 2 40 | return x[:, input_dim:] 41 | 42 | input_dim = input_shape[-1] // 2 43 | if ndim == 3: 44 | return x[:, :, input_dim:] 45 | elif ndim == 4: 46 | return x[:, :, :, input_dim:] 47 | elif ndim == 5: 48 | return x[:, :, :, :, input_dim:] 49 | 50 | 51 | def get_abs(x): 52 | real = get_realpart(x) 53 | imag = get_imagpart(x) 54 | 55 | # return K.sqrt(real * real + imag * imag) 56 | return real * real + imag * imag 57 | 58 | 59 | def getpart_output_shape(input_shape): 60 | returned_shape = list(input_shape[:]) 61 | image_format = K.image_data_format() 62 | ndim = len(returned_shape) 63 | 64 | if (image_format == 'channels_first' and ndim != 3) or ndim == 2: 65 | axis = 1 66 | else: 67 | axis = -1 68 | 69 | returned_shape[axis] = returned_shape[axis] // 2 70 | 71 | return tuple(returned_shape) 72 | 73 | 74 | class GetReal(Layer): 75 | def call(self, inputs): 76 | return get_realpart(inputs) 77 | def compute_output_shape(self, input_shape): 78 | return getpart_output_shape(input_shape) 79 | class GetImag(Layer): 80 | def call(self, inputs): 81 | return get_imagpart(inputs) 82 | def compute_output_shape(self, input_shape): 83 | return getpart_output_shape(input_shape) 84 | class GetAbs(Layer): 85 | def call(self, inputs): 86 | return get_abs(inputs) 87 | def compute_output_shape(self, input_shape): 88 | return getpart_output_shape(input_shape) 89 | 90 | -------------------------------------------------------------------------------- /complex_rf/cxnn/complexnn/utils.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/complexnn/utils.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/models.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/models.pyc -------------------------------------------------------------------------------- /complex_rf/cxnn/train.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/cxnn/train.pyc -------------------------------------------------------------------------------- /complex_rf/experiment_setup.py: -------------------------------------------------------------------------------- 1 | """ 2 | Hyper-parameters 3 | """ 4 | import argparse 5 | 6 | 7 | def get_arguments(): 8 | """ Hyper-parameters """ 9 | 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument("-a", "--architecture", type=str, choices=['reim', 12 | 'reim2x', 13 | 'reimsqrt2x', 14 | 'magnitude', 15 | 'phase', 16 | 're', 17 | 'im', 18 | 'modrelu', 19 | 'crelu'], 20 | default='modrelu', 21 | help="Architecture") 22 | 23 | args = parser.parse_args() 24 | 25 | return args 26 | -------------------------------------------------------------------------------- /complex_rf/experiment_setup.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/complex_rf/experiment_setup.pyc -------------------------------------------------------------------------------- /complex_rf/load_slice_IQ.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import glob 4 | import argparse 5 | import pdb 6 | 7 | import numpy as np 8 | from keras.utils import np_utils 9 | import utils 10 | 11 | 12 | def normalizeData(v): 13 | # keepdims makes the result shape (1, 1, 3) instead of (3,). This doesn't matter here, but 14 | # would matter if you wanted to normalize over a different axis. 15 | v_min = v.min(axis=(0, 1), keepdims=True) 16 | v_max = v.max(axis=(0, 1), keepdims=True) 17 | v = (v - v_min)/(v_max - v_min) 18 | return v 19 | 20 | 21 | def read_f32_bin(filename, start_ix, channel_first): 22 | with open(filename, 'rb') as bin_f: 23 | iq_seq = np.fromfile(bin_f, dtype='= n_samples: 64 | isEnoughSample = True 65 | break 66 | else: 67 | isEnoughSample = True 68 | 69 | if not isEnoughSample: 70 | print("ERROR! There are not enough samples to satisfy dataset parameters. Aborting...") 71 | sys.exit(-1) 72 | 73 | if channel_first: 74 | all_IQ_data = all_IQ_data[:, 0:n_samples] 75 | else: 76 | all_IQ_data = all_IQ_data[0:n_samples, :] 77 | return all_IQ_data, num_tran 78 | 79 | 80 | def loadData(args, channel_first): 81 | n_slices_per_dev = args.num_slice 82 | start_ix = args.start_ix 83 | file_key = args.file_key 84 | 85 | dev_dir_list = [] 86 | dev_dir_names = os.listdir(args.root_dir) 87 | for n in dev_dir_names: 88 | tmp = os.path.join(args.root_dir, n) 89 | dev_dir_list.append(tmp) 90 | 91 | stride = args.stride 92 | n_devices = len(dev_dir_list) 93 | # locations = ["after_fft","before_fft", "output_equ", "symbols"] 94 | # locations = ["output_equ"] 95 | if channel_first: 96 | slice_dims = (2, args.slice_len) 97 | samps_to_retrieve = (n_slices_per_dev - 1) * stride + slice_dims[1] 98 | else: 99 | slice_dims = (args.slice_len, 2) 100 | samps_to_retrieve = (n_slices_per_dev - 1) * stride + slice_dims[0] 101 | 102 | x_train, y_train, x_test, y_test = [], [], [], [] 103 | split_ratio = {'train': 0.8, 'val': 0.2} 104 | for i, d in enumerate(dev_dir_list): 105 | 106 | p = os.path.join(d, args.location) 107 | pre_X_data, num_tran = dev_bin_dataset(os.path.join(p, file_key), samps_to_retrieve, start_ix, channel_first, uniform=True) 108 | 109 | X_data_pd = [] 110 | count_s = 0 111 | for j in range(0, samps_to_retrieve, stride): 112 | if channel_first: 113 | X_data_pd.append(pre_X_data[:, j:j+slice_dims[1]]) 114 | else: 115 | X_data_pd.append(pre_X_data[j:j+slice_dims[0], :]) 116 | count_s += 1 117 | if count_s == n_slices_per_dev: 118 | break 119 | a = X_data_pd[-1] 120 | X_data_pd = np.array(X_data_pd) 121 | y_data_pd = i * np.ones(n_slices_per_dev, ) 122 | 123 | # split one class data 124 | uniform = True 125 | x_train_pd, x_test_pd, y_train_pd, y_test_pd = [],[],[],[] 126 | if uniform: 127 | samples_per_tran = n_slices_per_dev // num_tran 128 | idx = 0 129 | while idx + samples_per_tran <= n_slices_per_dev: 130 | x_train_per_tran, y_train_per_tran, x_test_per_tran, y_test_per_tran = utils.splitData(split_ratio, X_data_pd[i:i+samples_per_tran, :, :], y_data_pd[i:i+samples_per_tran]) 131 | if idx == 0: 132 | x_train_pd, x_test_pd = x_train_per_tran, x_test_per_tran 133 | y_train_pd, y_test_pd = y_train_per_tran, y_test_per_tran 134 | else: 135 | x_train_pd = np.concatenate((x_train_pd, x_train_per_tran), axis=0) 136 | x_test_pd = np.concatenate((x_test_pd, x_test_per_tran), axis=0) 137 | y_train_pd = np.concatenate((y_train_pd, y_train_per_tran), axis=0) 138 | y_test_pd = np.concatenate((y_test_pd, y_test_per_tran), axis=0) 139 | idx += samples_per_tran 140 | else: 141 | 142 | x_train_pd, y_train_pd, x_test_pd, y_test_pd = utils.splitData(split_ratio, X_data_pd, y_data_pd) 143 | 144 | if i == 0: 145 | x_train, x_test = x_train_pd, x_test_pd 146 | y_train, y_test = y_train_pd, y_test_pd 147 | else: 148 | x_train = np.concatenate((x_train, x_train_pd), axis=0) 149 | x_test = np.concatenate((x_test, x_test_pd), axis=0) 150 | y_train = np.concatenate((y_train, y_train_pd), axis=0) 151 | y_test = np.concatenate((y_test, y_test_pd), axis=0) 152 | del pre_X_data 153 | del X_data_pd 154 | 155 | if args.D2: 156 | if channel_first: 157 | x_train = x_train[:, :, np.newaxis, :] 158 | x_test = x_test[:, :, np.newaxis, :] 159 | else: 160 | x_train = x_train[:, np.newaxis, :, :] 161 | x_test = x_test[:, np.newaxis, :, :] 162 | 163 | y_train = np_utils.to_categorical(y_train, n_devices) 164 | y_test = np_utils.to_categorical(y_test, n_devices) 165 | return x_train, y_train, x_test, y_test, n_devices 166 | 167 | 168 | class loadDataOpts(): 169 | def __init__(self, root_dir, location, D2, file_key='*.bin', num_slice=100000, start_ix=0, slice_len=288, stride=1): 170 | self.root_dir = root_dir 171 | self.num_slice = num_slice 172 | self.start_ix = start_ix 173 | self.slice_len = slice_len 174 | self.stride = stride 175 | self.file_key = file_key 176 | self.D2 = D2 177 | self.location = location 178 | 179 | 180 | def parseArgs(argv): 181 | Desc = 'Read and slice the collected I/Q samples' 182 | parser = argparse.ArgumentParser(description=Desc, 183 | formatter_class=argparse.ArgumentDefaultsHelpFormatter) 184 | parser.add_argument('-d', '--root_dir', required=True, help='Root directory for the devices\' folders.') 185 | parser.add_argument('-n', '--num_slice', required=True, type=int, help='Number of slices to be generated for each device.') 186 | parser.add_argument('-i', '--start_ix', type=int, default=0, help='Starting read index in .bin files.') 187 | parser.add_argument('-l', '--slice_len', type=int, default=288, help='Lenght of slices.') 188 | parser.add_argument('-s', '--stride', type=int, default=1, help='Stride used for windowing.') 189 | parser.add_argument('-f', '--file_key', default='*.bin', help='used to choose different filetype, choose from *.bin/*.sigmf-meta') 190 | parser.add_argument('--D2', action='store_true', help='') 191 | parser.add_argument('-cf', '--channel_first', action='store_true', help='if set channel first otherwise channel last') 192 | opts = parser.parse_args() 193 | return opts 194 | 195 | 196 | if __name__ == "__main__": 197 | # opts = parseArgs(sys.argv) 198 | opts = loadDataOpts(root_dir="/home/erc/PycharmProjects/rf/test_dataset/", D2=False, location='after_fft') 199 | channel_first = True 200 | x_train, y_train, x_test, y_test, NUM_CLASS = loadData(opts, channel_first) 201 | print('train data shape: ', x_train.shape, 'train label shape: ', y_train.shape) 202 | print('test data shape: ', x_test.shape, 'test label shape: ', y_test.shape) 203 | print('all test done!') 204 | -------------------------------------------------------------------------------- /complex_rf/prepare4complex.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | import os 4 | import sys 5 | import argparse 6 | import pdb 7 | 8 | import numpy as np 9 | from sklearn.model_selection import train_test_split 10 | 11 | import config 12 | import load_slice_IQ 13 | # import mytools.tools as mytools 14 | 15 | ''' 16 | data need to organized into 3 parts: 17 | (train_x, train_y), (val_x, val_y), (test_x, test_y) 18 | and all data need to be normalize to (0,1) and into a mat 19 | with shape like (sample_num, sample_dim, 2), all complex 20 | value are stored in (real, imag) pair 21 | ''' 22 | #'\033[{i};1m这里写需要输出打印的内容\033[0m'# i是指打印颜色的编号 23 | # print() 24 | #print('\033[40;33m required args includes input/output/D2/normalize/ \033[0m') 25 | # msg = 'required args includes input/output/D2/normalize/' 26 | # mytools.highLighPrint(msg) 27 | # print() 28 | 29 | 30 | def loadData2Dict(opts): 31 | print('loading data...') 32 | # train_x, train_y, val_x, val_y, test_x, test_y = readSigmf.getData(opts, x_day_dir) 33 | # D2 means that make it into 2 dimension data 34 | dataOpts = load_slice_IQ.loadDataOpts(opts.input, opts.location, opts.D2, num_slice=10000) 35 | train_x, train_y, test_x, test_y, NUM_CLASS = load_slice_IQ.loadData(dataOpts, opts.channel_first) 36 | 37 | if opts.normalize: 38 | train_x = load_slice_IQ.normalizeData(train_x) 39 | test_x = load_slice_IQ.normalizeData(test_x) 40 | 41 | data_dict = {} 42 | data_dict['x_train'] = train_x 43 | data_dict['y_train'] = train_y 44 | data_dict['x_test'] = test_x 45 | data_dict['y_test'] = test_y 46 | 47 | print('x_train shape: {}\ty_train shape: {}\tx_test shape: {}\ty_test shape: {}'.format(train_x.shape, train_y.shape, test_x.shape, test_y.shape)) 48 | 49 | return data_dict 50 | 51 | 52 | def save2file(outfile, data_dict): 53 | x_train, y_train = data_dict['x_train'], data_dict['y_train'] 54 | x_test, y_test = data_dict['x_test'], data_dict['y_test'] 55 | 56 | x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.2, random_state=47, shuffle=True) 57 | np.savez(outfile, x_train=x_train, y_train=y_train, x_val=x_val, y_val=y_val, x_test=x_test, y_test=y_test) 58 | outfile = os.path.abspath(outfile) 59 | print('all data save to path {}'.format(outfile)) 60 | 61 | 62 | def main(opts): 63 | tmp_list = opts.input.split('/') 64 | tmp_list = filter(lambda a: a != '', tmp_list) 65 | save_name = list(tmp_list)[-1] 66 | save_path = os.path.join(opts.output, '{}.npz'.format(save_name)) 67 | # load all the data 68 | data_dict = loadData2Dict(opts) 69 | 70 | # save data to npz file 71 | save2file(save_path, data_dict) 72 | 73 | print('all done!') 74 | 75 | 76 | if __name__ == "__main__": 77 | opts = config.parse_args(sys.argv) 78 | main(opts) 79 | -------------------------------------------------------------------------------- /complex_rf/requirements.txt: -------------------------------------------------------------------------------- 1 | h5py==2.8.0 2 | ipdb==0.11 3 | ipython==5.8.0 4 | Keras==2.2.4 5 | matplotlib==2.2.3 6 | numpy 7 | # resampy==0.2.1 8 | scikit-learn==0.20.4 9 | scipy==1.1.0 10 | Theano==1.0.3 11 | tqdm==4.28.1 12 | -------------------------------------------------------------------------------- /complex_rf/utils.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin python3.6 2 | 3 | import os 4 | import sys 5 | import pdb 6 | import shutil 7 | import numpy as np 8 | import config 9 | 10 | 11 | def generateIndex(allDataSize): 12 | np.random.seed(42) 13 | shuffledind = np.random.permutation(allDataSize) 14 | return shuffledind 15 | 16 | 17 | def getSplitIndex(allDataSize, splitRatio): 18 | shuffledind = generateIndex(allDataSize) 19 | 20 | train_set_size = int(allDataSize * splitRatio['train']) 21 | val_set_size = int(allDataSize * splitRatio['val']) 22 | # test_set_size = int(allDataSize * splitRatio['test']) 23 | 24 | start, end = 0, train_set_size 25 | train_ind = shuffledind[start: end] 26 | 27 | start, end = train_set_size, train_set_size + val_set_size 28 | val_ind = shuffledind[start: end] 29 | 30 | ''' 31 | start, end = train_set_size + val_set_size, train_set_size + val_set_size + test_set_size 32 | test_ind = shuffledind[start: end] 33 | ''' 34 | 35 | #return train_ind, val_ind, test_ind 36 | return train_ind, val_ind 37 | 38 | 39 | def splitData(splitRatio, allData, allLabel, splitType='random'): 40 | if not isinstance(allData, np.ndarray): 41 | allData = np.array(allData) 42 | allLabel = np.array(allLabel) 43 | 44 | if splitType == 'random': 45 | allDataSize = len(allLabel) 46 | #train_ind, val_ind, test_ind = getSplitIndex(allDataSize, splitRatio) 47 | train_ind, val_ind = getSplitIndex(allDataSize, splitRatio) 48 | 49 | trainData, trainLabels = allData[train_ind, :, :], allLabel[train_ind] 50 | valData, valLabels = allData[val_ind, :, :], allLabel[val_ind] 51 | #testData, testLabels = allData[test_ind, :, :], allLabel[test_ind] 52 | 53 | elif splitType == 'order': 54 | pass 55 | else: 56 | raise 57 | 58 | #return trainData, trainLabels, valData, valLabels, testData, testLabels 59 | return trainData, trainLabels, valData, valLabels 60 | 61 | def organize_neu_data(): 62 | p = '/home/erc/PycharmProjects/rf/neu_dataset/' 63 | devs = os.listdir(p) 64 | file_key = ".bin" 65 | for dev in devs: 66 | path = os.path.join(p,dev) 67 | files = os.listdir(path) 68 | for file in files: 69 | os.rename(os.path.join(path, file), os.path.join(path, file+file_key)) 70 | 71 | 72 | if __name__ == "__main__": 73 | # opts = config.parse_args(sys.argv) 74 | # print('all test passed!') 75 | organize_neu_data() -------------------------------------------------------------------------------- /fine-tuning/.idea/.gitignore: -------------------------------------------------------------------------------- 1 | # Default ignored files 2 | /shelf/ 3 | /workspace.xml 4 | -------------------------------------------------------------------------------- /fine-tuning/.idea/fine-tuning.iml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | -------------------------------------------------------------------------------- /fine-tuning/.idea/inspectionProfiles/Project_Default.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 25 | -------------------------------------------------------------------------------- /fine-tuning/.idea/inspectionProfiles/profiles_settings.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 6 | -------------------------------------------------------------------------------- /fine-tuning/.idea/misc.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | -------------------------------------------------------------------------------- /fine-tuning/.idea/modules.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | -------------------------------------------------------------------------------- /fine-tuning/DF_model.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3.6 2 | 3 | # The model is the DF model by Sirinam et al 4 | 5 | from keras.models import Model 6 | from keras.layers import Dense, Input, Activation, ELU, Conv1D, MaxPooling1D, Dropout, Flatten 7 | 8 | 9 | def DF(input_shape=None, emb_size=None, Classification=False): 10 | # -----------------Entry flow ----------------- 11 | input_data = Input(shape=input_shape) 12 | 13 | filter_num = ['None', 32, 64, 128, 256] 14 | kernel_size = ['None', 8, 8, 8, 8] 15 | conv_stride_size = ['None', 1, 1, 1, 1] 16 | pool_stride_size = ['None', 4, 4, 4, 4] 17 | pool_size = ['None', 8, 8, 8, 8] 18 | 19 | model = Conv1D(filters=filter_num[1], kernel_size=kernel_size[1], 20 | strides=conv_stride_size[1], padding='same', name='block1_conv1')(input_data) 21 | model = ELU(alpha=1.0, name='block1_adv_act1')(model) 22 | model = Conv1D(filters=filter_num[1], kernel_size=kernel_size[1], 23 | strides=conv_stride_size[1], padding='same', name='block1_conv2')(model) 24 | model = ELU(alpha=1.0, name='block1_adv_act2')(model) 25 | model = MaxPooling1D(pool_size=pool_size[1], strides=pool_stride_size[1], 26 | padding='same', name='block1_pool')(model) 27 | model = Dropout(0.1, name='block1_dropout')(model) 28 | 29 | model = Conv1D(filters=filter_num[2], kernel_size=kernel_size[2], 30 | strides=conv_stride_size[2], padding='same', name='block2_conv1')(model) 31 | model = Activation('relu', name='block2_act1')(model) 32 | model = Conv1D(filters=filter_num[2], kernel_size=kernel_size[2], 33 | strides=conv_stride_size[2], padding='same', name='block2_conv2')(model) 34 | model = Activation('relu', name='block2_act2')(model) 35 | model = MaxPooling1D(pool_size=pool_size[2], strides=pool_stride_size[3], 36 | padding='same', name='block2_pool')(model) 37 | model = Dropout(0.1, name='block2_dropout')(model) 38 | 39 | model = Conv1D(filters=filter_num[3], kernel_size=kernel_size[3], 40 | strides=conv_stride_size[3], padding='same', name='block3_conv1')(model) 41 | model = Activation('relu', name='block3_act1')(model) 42 | model = Conv1D(filters=filter_num[3], kernel_size=kernel_size[3], 43 | strides=conv_stride_size[3], padding='same', name='block3_conv2')(model) 44 | model = Activation('relu', name='block3_act2')(model) 45 | model = MaxPooling1D(pool_size=pool_size[3], strides=pool_stride_size[3], 46 | padding='same', name='block3_pool')(model) 47 | model = Dropout(0.1, name='block3_dropout')(model) 48 | 49 | model = Conv1D(filters=filter_num[4], kernel_size=kernel_size[4], 50 | strides=conv_stride_size[4], padding='same', name='block4_conv1')(model) 51 | model = Activation('relu', name='block4_act1')(model) 52 | model = Conv1D(filters=filter_num[4], kernel_size=kernel_size[4], 53 | strides=conv_stride_size[4], padding='same', name='block4_conv2')(model) 54 | model = Activation('relu', name='block4_act2')(model) 55 | model = MaxPooling1D(pool_size=pool_size[4], strides=pool_stride_size[4], 56 | padding='same', name='block4_pool')(model) 57 | 58 | output = Flatten()(model) 59 | 60 | if Classification: 61 | dense_layer = Dense(emb_size, name='FeaturesVec', activation='softmax')(output) 62 | else: 63 | dense_layer = Dense(emb_size, name='FeaturesVec')(output) 64 | shared_conv2 = Model(inputs=input_data, outputs=dense_layer) 65 | return shared_conv2 66 | -------------------------------------------------------------------------------- /fine-tuning/__pycache__/augData.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/fine-tuning/__pycache__/augData.cpython-36.pyc -------------------------------------------------------------------------------- /fine-tuning/__pycache__/config.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/fine-tuning/__pycache__/config.cpython-36.pyc -------------------------------------------------------------------------------- /fine-tuning/__pycache__/imagenet_utils.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/fine-tuning/__pycache__/imagenet_utils.cpython-36.pyc -------------------------------------------------------------------------------- /fine-tuning/__pycache__/load_slice_IQ.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/fine-tuning/__pycache__/load_slice_IQ.cpython-36.pyc -------------------------------------------------------------------------------- /fine-tuning/__pycache__/radioConv.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/fine-tuning/__pycache__/radioConv.cpython-36.pyc -------------------------------------------------------------------------------- /fine-tuning/__pycache__/resnet50_1D.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/fine-tuning/__pycache__/resnet50_1D.cpython-36.pyc -------------------------------------------------------------------------------- /fine-tuning/__pycache__/tools.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/fine-tuning/__pycache__/tools.cpython-36.pyc -------------------------------------------------------------------------------- /fine-tuning/__pycache__/utils.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/fine-tuning/__pycache__/utils.cpython-36.pyc -------------------------------------------------------------------------------- /fine-tuning/augData.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python3 2 | 3 | import os 4 | import sys 5 | import argparse 6 | import pdb 7 | import random 8 | import time 9 | import copy 10 | 11 | import numpy as np 12 | from collections import defaultdict 13 | 14 | import tools as mytools 15 | 16 | 17 | def sampleWithin(oneSample, sampleList): 18 | for item in sampleList: 19 | if (oneSample==item).all(): 20 | return True 21 | 22 | return False 23 | 24 | 25 | def removeRedundancy(samplesNPList): 26 | afterList = [] 27 | afterList.append(samplesNPList[0]) 28 | for i in range(1, len(samplesNPList)): 29 | oneSample = samplesNPList[i] 30 | if not sampleWithin(oneSample, afterList): 31 | afterList.append(oneSample) 32 | 33 | return afterList 34 | 35 | 36 | def getExchangeParams(seedNum, data_dim, start=0.1, end=0.7): 37 | random.seed(seedNum) 38 | start_point = int(random.random() * data_dim) 39 | exchange_len = int(random.uniform(start, end) * data_dim) 40 | return start_point, exchange_len 41 | 42 | 43 | def generateNewSamples(sample_pair, start_point, exchange_len): 44 | if not isinstance(sample_pair[0], np.ndarray): 45 | sample_pair[0] = np.array(sample_pair[0]) 46 | if not isinstance(sample_pair[1], np.ndarray): 47 | sample_pair[1] = np.array(sample_pair[1]) 48 | 49 | # mind copy here 50 | cut_1 = copy.deepcopy(sample_pair[0][start_point: start_point+exchange_len]) 51 | cut_2 = copy.deepcopy(sample_pair[1][start_point: start_point+exchange_len]) 52 | 53 | sample_pair[0][start_point: start_point+exchange_len] = cut_2 54 | sample_pair[1][start_point: start_point+exchange_len] = cut_1 55 | 56 | return sample_pair 57 | 58 | 59 | def getNewSamples(samples, newSamNum, data_dim): 60 | new_samples = [] 61 | 62 | samNum = 0 63 | while 1: 64 | if samNum >= newSamNum: 65 | break 66 | sample_choose = random.sample(samples, 2) 67 | sample_pair = copy.deepcopy(sample_choose) 68 | #seed = int(time.time()) % int(13700 * random.random()) 69 | seed = int(time.time()) % int(1000 * (random.random()+1)) 70 | start_point, exchange_len = getExchangeParams(seed, data_dim) 71 | tmpSamples = generateNewSamples(sample_pair, start_point, exchange_len) 72 | new_samples.extend(tmpSamples) 73 | new_samples = removeRedundancy(new_samples) 74 | samNum = len(new_samples) 75 | 76 | new_samples = new_samples[:newSamNum] 77 | return new_samples 78 | 79 | 80 | def data_aug(x, y, newSamNum): 81 | ''' 82 | augument data by exchange random part of samples within class 83 | ''' 84 | # form old data dict 85 | if not isinstance(x, np.ndarray): 86 | x = np.array(x) 87 | 88 | data_dim = x.shape[1] 89 | totalNum = x.shape[0] 90 | 91 | data_dict = defaultdict(list) 92 | for i in range(totalNum): 93 | label = int(y[i]) 94 | sample = x[i, :] 95 | data_dict[label].append(sample) 96 | 97 | # generate new data dict 98 | newDataDict = defaultdict(list) 99 | for key in data_dict.keys(): 100 | samples = data_dict[key] 101 | new_samples = getNewSamples(samples, newSamNum, data_dim) 102 | newDataDict[key] = new_samples 103 | 104 | # convert data dict to data array 105 | allData, allLabel = [], [] 106 | tmpData, tmpLabel = mytools.datadict2data(data_dict) 107 | allData.extend(tmpData) 108 | allLabel.extend(tmpLabel) 109 | 110 | tmpData, tmpLabel = mytools.datadict2data(newDataDict) 111 | allData.extend(tmpData) 112 | allLabel.extend(tmpLabel) 113 | 114 | # the shuffleData method will convert the data into numpy ndarry 115 | allData, allLabel = mytools.shuffleData(allData, allLabel) 116 | 117 | return allData, allLabel 118 | 119 | 120 | def parseOpts(argv): 121 | parser = argparse.ArgumentParser() 122 | parser.add_argument('-i', '--input', help='') 123 | parser.add_argument('-o', '--output', help='') 124 | parser.add_argument('-n', '--newSamNum', type=int, default=50, help='') 125 | parser.add_argument('-l', '--limit', type=int, default=5, 126 | help='limit the input sample number of input data') 127 | parser.add_argument('-t', '--test', action='store_true', help='') 128 | opts = parser.parse_args() 129 | return opts 130 | 131 | 132 | def main(opts): 133 | ''' 134 | standalone run, and save the data to output path 135 | ''' 136 | dataName = os.path.basename(opts.input) 137 | wholePack = np.load(opts.input) 138 | x, y = wholePack['x'], wholePack['y'] 139 | 140 | # limit data first and then augment data 141 | x, y = mytools.limitData(x, y, sampleLimit=opts.limit) 142 | allData, allLabel = data_aug(x, y, opts.newSamNum) 143 | 144 | savePath = os.path.join(opts.output, '{}_PartExchange_to_{}.npz'.format(dataName, opts.newSamNum)) 145 | np.savez(savePath, x=allData, y=allLabel) 146 | print('data save to: {}'.format(savePath)) 147 | 148 | 149 | def test(): 150 | print('test getNewSamples func...') 151 | aaa = np.array([3, 3, 3, 3, 3, 3, 3, 3]) 152 | bbb = np.array([1, 1, 1, 1, 1, 1, 1, 1]) 153 | ccc = np.array([2, 2, 2, 2, 2, 2, 2, 2]) 154 | ddd = np.array([4, 4, 4, 4, 4, 4, 4, 4]) 155 | 156 | samplePair = [aaa, bbb] 157 | newSamples = getNewSamples(samplePair, 7, aaa.shape[0]) 158 | for item in newSamples: 159 | print(item) 160 | 161 | print('test data aug func...') 162 | dataset = [aaa, bbb, ccc, ddd] 163 | label = [1, 1, 2, 2] 164 | newData, newLabel = data_aug(dataset, label, newSamNum=5) 165 | for i in range(len(newLabel)): 166 | print('label: ', newLabel[i], 'data: ', newData[i]) 167 | 168 | 169 | if __name__ == "__main__": 170 | opts = parseOpts(sys.argv) 171 | if opts.test: 172 | test() 173 | else: 174 | main(opts) 175 | -------------------------------------------------------------------------------- /fine-tuning/call.sh: -------------------------------------------------------------------------------- 1 | #! /bin/bash 2 | 3 | echo 'setup variable values' 4 | 5 | if [ $? -eq 0 ] 6 | then 7 | echo 'nvidia-smi check pass' `date` 8 | echo 'will run with GPU' 9 | export useGpu=1 10 | else 11 | echo 'nvidia-smi not exists, will run with CPU' 12 | export useGpu=-1 13 | fi 14 | 15 | SHELL_FOLDER=$(dirname $(readlink -f "$0")) 16 | export ROOT_DIR=$SHELL_FOLDER 17 | echo ROOT_DIR=$ROOT_DIR 18 | 19 | export ResRootDir=$ROOT_DIR/res_out_root 20 | echo ResRootDir=$ResRootDir 21 | 22 | export PYTHONPATH=$PYTHONPATH:/home/haipeng/Documents/fine-tuning 23 | 24 | echo 'start running...' 25 | python3 $* 26 | -------------------------------------------------------------------------------- /fine-tuning/config.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | ''' 3 | This module is for universal arg parse 4 | ''' 5 | 6 | import argparse 7 | 8 | 9 | def parse_args(argv): 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument('-i', '--input', help='input file/dir') 12 | parser.add_argument('-o', '--output', help='output dir') 13 | parser.add_argument('-m', '--modelType', default='homegrown', help='choose from homegrown/baseline/resnet') 14 | parser.add_argument('-sp', '--splitType', default='random', help='choose from random/order') 15 | parser.add_argument('-v', '--verbose', action='store_true', help='') 16 | parser.add_argument('--D2', action='store_true', help='if set will return 2 dimension data') 17 | parser.add_argument('-n', '--normalize', action='store_true', help='') 18 | parser.add_argument('-ds', '--dataSource', help='choose from neu/simu') 19 | parser.add_argument('-cf', '--channel_first', action='store_true', help='if set channel first otherwise channel last') 20 | parser.add_argument('-l', '--location',help='where the data collected') 21 | opts = parser.parse_args() 22 | return opts 23 | -------------------------------------------------------------------------------- /fine-tuning/load_slice_IQ.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import glob 4 | import argparse 5 | import pdb 6 | 7 | import numpy as np 8 | from keras.utils import np_utils 9 | import utils 10 | 11 | 12 | def normalizeData(v): 13 | # keepdims makes the result shape (1, 1, 3) instead of (3,). This doesn't matter here, but 14 | # would matter if you wanted to normalize over a different axis. 15 | v_min = v.min(axis=(0, 1), keepdims=True) 16 | v_max = v.max(axis=(0, 1), keepdims=True) 17 | v = (v - v_min)/(v_max - v_min) 18 | return v 19 | 20 | 21 | def read_f32_bin(filename, start_ix, channel_first): 22 | with open(filename, 'rb') as bin_f: 23 | iq_seq = np.fromfile(bin_f, dtype='= n_samples: 64 | isEnoughSample = True 65 | break 66 | else: 67 | isEnoughSample = True 68 | 69 | if not isEnoughSample: 70 | print("ERROR! There are not enough samples to satisfy dataset parameters. Aborting...") 71 | sys.exit(-1) 72 | 73 | if channel_first: 74 | all_IQ_data = all_IQ_data[:, 0:n_samples] 75 | else: 76 | all_IQ_data = all_IQ_data[0:n_samples, :] 77 | return all_IQ_data, num_tran 78 | 79 | 80 | def loadData(args, channel_first): 81 | n_slices_per_dev = args.num_slice 82 | start_ix = args.start_ix 83 | file_key = args.file_key 84 | 85 | dev_dir_list = [] 86 | dev_dir_names = os.listdir(args.root_dir) 87 | for n in dev_dir_names: 88 | tmp = os.path.join(args.root_dir, n) 89 | dev_dir_list.append(tmp) 90 | 91 | stride = args.stride 92 | n_devices = len(dev_dir_list) 93 | # locations = ["after_fft","before_fft", "output_equ", "symbols"] 94 | # locations = ["output_equ"] 95 | if channel_first: 96 | slice_dims = (2, args.slice_len) 97 | samps_to_retrieve = (n_slices_per_dev - 1) * stride + slice_dims[1] 98 | else: 99 | slice_dims = (args.slice_len, 2) 100 | samps_to_retrieve = (n_slices_per_dev - 1) * stride + slice_dims[0] 101 | 102 | x_train, y_train, x_test, y_test = [], [], [], [] 103 | split_ratio = {'train': 0.8, 'val': 0.2} 104 | for i, d in enumerate(dev_dir_list): 105 | 106 | p = os.path.join(d, args.location) 107 | pre_X_data, num_tran = dev_bin_dataset(os.path.join(p, file_key), samps_to_retrieve, start_ix, channel_first, uniform=True) 108 | 109 | X_data_pd = [] 110 | count_s = 0 111 | for j in range(0, samps_to_retrieve, stride): 112 | if channel_first: 113 | X_data_pd.append(pre_X_data[:, j:j+slice_dims[1]]) 114 | else: 115 | X_data_pd.append(pre_X_data[j:j+slice_dims[0], :]) 116 | count_s += 1 117 | if count_s == n_slices_per_dev: 118 | break 119 | X_data_pd = np.array(X_data_pd) 120 | y_data_pd = i * np.ones(n_slices_per_dev, ) 121 | 122 | # split one class data 123 | uniform = True 124 | x_train_pd, x_test_pd, y_train_pd, y_test_pd = [],[],[],[] 125 | if uniform: 126 | samples_per_tran = n_slices_per_dev // num_tran 127 | idx = 0 128 | while idx + samples_per_tran <= n_slices_per_dev: 129 | x_train_per_tran, y_train_per_tran, x_test_per_tran, y_test_per_tran = utils.splitData(split_ratio, X_data_pd[i:i+samples_per_tran, :, :], y_data_pd[i:i+samples_per_tran]) 130 | if idx == 0: 131 | x_train_pd, x_test_pd = x_train_per_tran, x_test_per_tran 132 | y_train_pd, y_test_pd = y_train_per_tran, y_test_per_tran 133 | else: 134 | x_train_pd = np.concatenate((x_train_pd, x_train_per_tran), axis=0) 135 | x_test_pd = np.concatenate((x_test_pd, x_test_per_tran), axis=0) 136 | y_train_pd = np.concatenate((y_train_pd, y_train_per_tran), axis=0) 137 | y_test_pd = np.concatenate((y_test_pd, y_test_per_tran), axis=0) 138 | idx += samples_per_tran 139 | else: 140 | 141 | x_train_pd, y_train_pd, x_test_pd, y_test_pd = utils.splitData(split_ratio, X_data_pd, y_data_pd) 142 | 143 | if i == 0: 144 | x_train, x_test = x_train_pd, x_test_pd 145 | y_train, y_test = y_train_pd, y_test_pd 146 | else: 147 | x_train = np.concatenate((x_train, x_train_pd), axis=0) 148 | x_test = np.concatenate((x_test, x_test_pd), axis=0) 149 | y_train = np.concatenate((y_train, y_train_pd), axis=0) 150 | y_test = np.concatenate((y_test, y_test_pd), axis=0) 151 | del pre_X_data 152 | del X_data_pd 153 | 154 | if args.D2: 155 | if channel_first: 156 | x_train = x_train[:, :, np.newaxis, :] 157 | x_test = x_test[:, :, np.newaxis, :] 158 | else: 159 | x_train = x_train[:, np.newaxis, :, :] 160 | x_test = x_test[:, np.newaxis, :, :] 161 | 162 | #y_train = np_utils.to_categorical(y_train, n_devices) 163 | #y_test = np_utils.to_categorical(y_test, n_devices) 164 | return x_train, y_train, x_test, y_test, n_devices 165 | 166 | 167 | class loadDataOpts(): 168 | def __init__(self, root_dir, location, D2, file_key='*.bin', num_slice=100000, start_ix=0, slice_len=288, stride=1): 169 | self.root_dir = root_dir 170 | self.num_slice = num_slice 171 | self.start_ix = start_ix 172 | self.slice_len = slice_len 173 | self.stride = stride 174 | self.file_key = file_key 175 | self.D2 = D2 176 | self.location = location 177 | 178 | 179 | def parseArgs(argv): 180 | Desc = 'Read and slice the collected I/Q samples' 181 | parser = argparse.ArgumentParser(description=Desc, 182 | formatter_class=argparse.ArgumentDefaultsHelpFormatter) 183 | parser.add_argument('-d', '--root_dir', required=True, help='Root directory for the devices\' folders.') 184 | parser.add_argument('-n', '--num_slice', required=True, type=int, help='Number of slices to be generated for each device.') 185 | parser.add_argument('-i', '--start_ix', type=int, default=0, help='Starting read index in .bin files.') 186 | parser.add_argument('-l', '--slice_len', type=int, default=288, help='Lenght of slices.') 187 | parser.add_argument('-s', '--stride', type=int, default=1, help='Stride used for windowing.') 188 | parser.add_argument('-f', '--file_key', default='*.bin', help='used to choose different filetype, choose from *.bin/*.sigmf-meta') 189 | parser.add_argument('--D2', action='store_true', help='') 190 | parser.add_argument('-cf', '--channel_first', action='store_true', help='if set channel first otherwise channel last') 191 | opts = parser.parse_args() 192 | return opts 193 | 194 | 195 | if __name__ == "__main__": 196 | opts = parseArgs(sys.argv) 197 | # opts = loadDataOpts(root_dir="/home/erc/PycharmProjects/rf/test_data/", D2=False, location='after_fft') 198 | channel_first = True 199 | x_train, y_train, x_test, y_test, NUM_CLASS = loadData(opts, channel_first) 200 | print('train data shape: ', x_train.shape, 'train label shape: ', y_train.shape) 201 | print('test data shape: ', x_test.shape, 'test label shape: ', y_test.shape) 202 | print('all test done!') 203 | -------------------------------------------------------------------------------- /fine-tuning/main.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | ''' 3 | Nov 6th: updated for D2 option, which can easily change between 1D and 2D with out change the code 4 | ''' 5 | import os 6 | import sys 7 | import argparse 8 | import pdb 9 | 10 | from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, TensorBoard 11 | from keras.utils import np_utils 12 | import numpy as np 13 | 14 | import radioConv 15 | #import readSigmf2 as readSigmf 16 | import load_slice_IQ 17 | import config 18 | import get_simu_data 19 | ROOT_DIR = os.path.dirname(os.path.abspath(__file__)) 20 | #ROOT_DIR = os.getenv('ROOT_DIR') 21 | print(ROOT_DIR) 22 | resDir = os.path.join(ROOT_DIR, 'resDir') 23 | modelDir = os.path.join(resDir, 'modelDir') 24 | os.makedirs(modelDir, exist_ok=True) 25 | 26 | 27 | def loadNeuData(opts): 28 | print('loading data...') 29 | x_day_dir = opts.input 30 | #train_x, train_y, val_x, val_y, test_x, test_y = readSigmf.getData(opts, x_day_dir) 31 | dataOpts = load_slice_IQ.loadDataOpts(opts.input, opts.location, opts.D2, num_slice=100000) 32 | train_x, train_y, test_x, test_y, NUM_CLASS = load_slice_IQ.loadData(dataOpts, opts.channel_first) 33 | 34 | if opts.normalize: 35 | train_x = load_slice_IQ.normalizeData(train_x) 36 | test_x = load_slice_IQ.normalizeData(test_x) 37 | 38 | return train_x, train_y, test_x, test_y, NUM_CLASS 39 | 40 | 41 | def loadSimuData(opts): 42 | simu_dict, NUM_CLASS = get_simu_data.loadData(opts.input) 43 | x_train, y_train = simu_dict['x_train'], simu_dict['y_train'] 44 | x_test, y_test = simu_dict['x_test'], simu_dict['y_test'] 45 | 46 | return x_train, y_train, x_test, y_test, NUM_CLASS 47 | 48 | 49 | def main(opts): 50 | # load data 51 | if 'neu' == opts.dataSource: 52 | train_x, train_y, test_x, test_y, NUM_CLASS = loadNeuData(opts) 53 | elif 'simu' == opts.dataSource: 54 | train_x, train_y, test_x, test_y, NUM_CLASS = loadSimuData(opts) 55 | else: 56 | raise NotImplementedError() 57 | 58 | # setup params 59 | Batch_Size = 64 60 | Epoch_Num = 100 61 | saveModelPath = os.path.join(modelDir, 'best_model_{}_{}.h5'.format(opts.modelType,opts.location)) 62 | checkpointer = ModelCheckpoint(filepath=saveModelPath, monitor='val_acc', verbose=1, save_best_only=True, mode='max') 63 | earlyStopper = EarlyStopping(monitor='val_acc', mode='max', patience=10) 64 | callBackList = [checkpointer, earlyStopper] 65 | 66 | print('get the model and compile it...') 67 | if opts.D2: 68 | inp_shape = (1, train_x.shape[1], train_x.shape[2]) 69 | else: 70 | inp_shape = (train_x.shape[1], train_x.shape[2]) 71 | 72 | # pdb.set_trace() 73 | model = radioConv.create_model(opts.modelType, inp_shape, NUM_CLASS, D2=opts.D2, channel='first') 74 | # model.summary() 75 | model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) 76 | 77 | print('fit the model with data...') 78 | model.fit(x=train_x, y=train_y, 79 | batch_size=Batch_Size, 80 | epochs=Epoch_Num, 81 | verbose=opts.verbose, 82 | callbacks=callBackList, 83 | validation_split=0.1, 84 | shuffle=True) 85 | 86 | print('test the trained model...') 87 | score, acc = model.evaluate(test_x, test_y, batch_size=Batch_Size, verbose=0) 88 | print('test acc is: ', acc) 89 | 90 | print('all test done!') 91 | results = [opts.modelType, opts.location, acc] 92 | 93 | with open('results.txt', 'a') as filehandle: 94 | #for listitem in results: 95 | #filehandle.write('%s' % listitem) 96 | filehandle.write('%f\n' % acc) 97 | 98 | if __name__ == "__main__": 99 | opts = config.parse_args(sys.argv) 100 | main(opts) 101 | -------------------------------------------------------------------------------- /fine-tuning/radioConv.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/python3 2 | ''' 3 | Nov 6h, updated the API, pass a D2 value to make it adapt to 1D or 2D Conv model, all test done! 4 | ''' 5 | import os 6 | import sys 7 | import argparse 8 | import pdb 9 | from keras.layers import Dense, Dropout, Flatten, Input 10 | from keras.layers import Activation, BatchNormalization 11 | from keras.models import Sequential, Model 12 | from keras.callbacks import ModelCheckpoint, LearningRateScheduler 13 | from keras.callbacks import EarlyStopping, Callback 14 | import keras.backend as K 15 | from keras.utils import np_utils 16 | 17 | import numpy as np 18 | import pandas as pd 19 | 20 | #import config 21 | 22 | #ROOT_DIR = os.getenv('PROJECT_DIR') 23 | #print(ROOT_DIR) 24 | #resDir = os.path.join(ROOT_DIR, 'resDir') 25 | #modelDir = os.path.join(resDir, 'modelDir') 26 | #os.makedirs(modelDir, exist_ok=True) 27 | 28 | 29 | def createHomegrown(inp_shape, emb_size, data_format, D2): 30 | # -----------------Entry flow ----------------- 31 | if D2: 32 | from keras.layers import MaxPooling2D as MaxPooling 33 | from keras.layers import GlobalAveragePooling2D as GlobalAveragePooling 34 | from keras.layers import Conv2D as Conv 35 | else: 36 | from keras.layers import MaxPooling1D as MaxPooling 37 | from keras.layers import GlobalAveragePooling1D as GlobalAveragePooling 38 | from keras.layers import Conv1D as Conv 39 | 40 | input_data = Input(shape=inp_shape) 41 | if D2: 42 | kernel_size = ['None', (1, 7), (2, 5)] 43 | else: 44 | kernel_size = ['None', 7, 7] 45 | filter_num = ['None', 50, 50] 46 | conv_stride_size = ['None', 1, 1] 47 | pool_stride_size = ['None', 1, 1] 48 | activation_func = ['None', 'relu', 'relu'] 49 | #activation_func = ['None', 'elu', 'elu'] 50 | dense_layer_size = ['None', 256, 80] 51 | 52 | model = Conv(filters=filter_num[1], kernel_size=kernel_size[1], 53 | strides=conv_stride_size[1], padding='same', name='block1_conv1', data_format=data_format)(input_data) 54 | model = Activation(activation_func[1], name='block1_act1')(model) 55 | model = Dropout(0.5, name='block1_dropout')(model) 56 | 57 | model = Conv(filters=filter_num[2], kernel_size=kernel_size[2], 58 | strides=conv_stride_size[2], padding='same', name='block2_conv1', data_format=data_format)(model) 59 | model = Activation(activation_func[2], name='block2_act1')(model) 60 | model = Dropout(0.5, name='block2_dropout')(model) 61 | 62 | output = GlobalAveragePooling()(model) 63 | 64 | dense_layer = Dense(dense_layer_size[1], name='dense1', activation='relu')(output) 65 | dense_layer = Dense(dense_layer_size[2], name='dense2', activation='relu')(dense_layer) 66 | dense_layer = Dense(emb_size, name='dense3', activation='softmax')(dense_layer) 67 | 68 | shared_conv2 = Model(inputs=input_data, outputs=dense_layer) 69 | return shared_conv2 70 | 71 | 72 | def baselineBlock(input, block_idx, D2): 73 | if D2: 74 | from keras.layers import MaxPooling2D as MaxPooling 75 | from keras.layers import Conv2D as Conv 76 | else: 77 | import resnet50_1D as resnet50 78 | from keras.layers import MaxPooling1D as MaxPooling 79 | from keras.layers import Conv1D as Conv 80 | 81 | if D2: 82 | kernel_size = ['None', (1, 7), (2, 5)] 83 | else: 84 | kernel_size = ['None', 7, 5] 85 | filter_num = ['None', 128, 128] 86 | conv_stride = ['None', 1, 1] 87 | pool_size = ['None', 2] 88 | pool_stride = ['None', 1] 89 | act_func = 'relu' 90 | 91 | model = Conv(filters=filter_num[1], kernel_size=kernel_size[1], name='conv1_{}'.format(block_idx), 92 | strides=conv_stride[1], padding='same', activation=act_func)(input) 93 | model = Conv(filters=filter_num[2], kernel_size=kernel_size[2], name='conv2_{}'.format(block_idx), 94 | strides=conv_stride[2], padding='same', activation=act_func)(model) 95 | output = MaxPooling(pool_size=pool_size[1], strides=pool_stride[1], padding='same', 96 | name='pool_{}'.format(block_idx))(model) 97 | 98 | return output 99 | 100 | 101 | def createBaseline(inp_shape, emb_size, data_format, D2): 102 | if D2: 103 | from keras.layers import GlobalAveragePooling2D as GlobalAveragePooling 104 | else: 105 | from keras.layers import GlobalAveragePooling1D as GlobalAveragePooling 106 | 107 | dense_layer_size = ['None', 256, 256, 128] 108 | act_func = ['None', 'relu', 'relu', 'relu'] 109 | 110 | blockNum = 4 111 | input_data = Input(shape=inp_shape) 112 | for i in range(blockNum): 113 | idx = i + 1 114 | if 0 == i: 115 | model = baselineBlock(input_data, idx, D2) 116 | else: 117 | model = baselineBlock(model, idx, D2) 118 | 119 | middle = GlobalAveragePooling()(model) 120 | 121 | dense_layer = Dense(dense_layer_size[1], name='dense1', activation=act_func[1])(middle) 122 | dense_layer = Dense(dense_layer_size[2], name='dense2', activation=act_func[2])(dense_layer) 123 | dense_layer = Dense(dense_layer_size[3], name='dense3', activation=act_func[3])(dense_layer) 124 | dense_layer = Dense(emb_size, name='dense4', activation='softmax')(dense_layer) 125 | 126 | conv_model = Model(inputs=input_data, outputs=dense_layer) 127 | return conv_model 128 | 129 | 130 | def createResnet(inp_shape, emb_size, data_format, D2): 131 | if D2: 132 | import resnet50_2D as resnet50 133 | else: 134 | import resnet50_1D as resnet50 135 | return resnet50.create_model(inp_shape, emb_size) 136 | 137 | 138 | def create_model(modelType, inp_shape, NUM_CLASS, D2=False, channel='last'): 139 | 140 | if 'first' == channel: 141 | data_format = 'channels_first' 142 | elif 'last' == channel: 143 | data_format = 'channels_last' 144 | else: 145 | raise 146 | 147 | emb_size = NUM_CLASS 148 | if 'homegrown' == modelType: 149 | model = createHomegrown(inp_shape, emb_size, data_format, D2) 150 | elif 'baseline' == modelType: 151 | model = createBaseline(inp_shape, emb_size, data_format, D2) 152 | elif 'resnet' == modelType: 153 | model = createResnet(inp_shape, emb_size, data_format, D2) 154 | else: 155 | raise ValueError('model type {} not support yet'.format(modelType)) 156 | 157 | return model 158 | 159 | 160 | def test_run(model): 161 | model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) 162 | 163 | 164 | def test(D2): 165 | modelTypes = ['homegrown', 'baseline', 'resnet'] 166 | NUM_CLASS = 10 167 | signal = True 168 | inp_shape = (1, 2, 288) if D2 else (2, 288) 169 | 170 | for modelType in modelTypes: 171 | model = create_model(modelType, inp_shape, NUM_CLASS, D2=D2, channel='first') 172 | try: 173 | flag = test_run(model) 174 | except Exception as e: 175 | print(e) 176 | 177 | print('all done!') if signal else print('test failed') 178 | 179 | 180 | if __name__ == "__main__": 181 | D2_Type = [True, False] 182 | for D2 in D2_Type: 183 | test(D2) 184 | -------------------------------------------------------------------------------- /fine-tuning/resnet50_1D.py: -------------------------------------------------------------------------------- 1 | """ResNet50 model for Keras. 2 | # Reference: 3 | - [Deep Residual Learning for Image Recognition]( 4 | https://arxiv.org/abs/1512.03385) (CVPR 2016 Best Paper Award) 5 | Adapted from code contributed by BigMoyan. 6 | """ 7 | 8 | import os 9 | import sys 10 | import argparse 11 | import pdb 12 | import warnings 13 | 14 | from keras import layers 15 | from keras import models 16 | import keras.utils as keras_utils 17 | import keras.backend as backend 18 | 19 | import imagenet_utils 20 | from imagenet_utils import get_submodules_from_kwargs 21 | from imagenet_utils import decode_predictions 22 | from imagenet_utils import _obtain_input_shape 23 | 24 | preprocess_input = imagenet_utils.preprocess_input 25 | 26 | 27 | def identity_block(input_tensor, kernel_size, filters, stage, block): 28 | """The identity block is the block that has no conv layer at shortcut. 29 | # Arguments 30 | input_tensor: input tensor 31 | kernel_size: default 3, the kernel size of 32 | middle conv layer at main path 33 | filters: list of integers, the filters of 3 conv layer at main path 34 | stage: integer, current stage label, used for generating layer names 35 | block: 'a','b'..., current block label, used for generating layer names 36 | # Returns 37 | Output tensor for the block. 38 | """ 39 | filters1, filters2, filters3 = filters 40 | if backend.image_data_format() == 'channels_last': 41 | bn_axis = 2 42 | else: 43 | bn_axis = 1 44 | conv_name_base = 'res' + str(stage) + block + '_branch' 45 | bn_name_base = 'bn' + str(stage) + block + '_branch' 46 | 47 | x = layers.Conv1D(filters1, 1, 48 | kernel_initializer='he_normal', 49 | name=conv_name_base + '2a')(input_tensor) 50 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2a')(x) 51 | x = layers.Activation('relu')(x) 52 | 53 | x = layers.Conv1D(filters2, kernel_size, 54 | padding='same', 55 | kernel_initializer='he_normal', 56 | name=conv_name_base + '2b')(x) 57 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2b')(x) 58 | x = layers.Activation('relu')(x) 59 | 60 | x = layers.Conv1D(filters3, 1, 61 | kernel_initializer='he_normal', 62 | name=conv_name_base + '2c')(x) 63 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2c')(x) 64 | 65 | x = layers.add([x, input_tensor]) 66 | x = layers.Activation('relu')(x) 67 | return x 68 | 69 | 70 | def conv_block(input_tensor, kernel_size, filters, stage, block, strides=2): 71 | """A block that has a conv layer at shortcut. 72 | # Arguments 73 | input_tensor: input tensor 74 | kernel_size: default 3, the kernel size of 75 | middle conv layer at main path 76 | filters: list of integers, the filters of 3 conv layer at main path 77 | stage: integer, current stage label, used for generating layer names 78 | block: 'a','b'..., current block label, used for generating layer names 79 | strides: Strides for the first conv layer in the block. 80 | # Returns 81 | Output tensor for the block. 82 | Note that from stage 3, 83 | the first conv layer at main path is with strides=(2, 2) 84 | And the shortcut should have strides=(2, 2) as well 85 | """ 86 | filters1, filters2, filters3 = filters 87 | if backend.image_data_format() == 'channels_last': 88 | bn_axis = 2 89 | else: 90 | bn_axis = 1 91 | conv_name_base = 'res' + str(stage) + block + '_branch' 92 | bn_name_base = 'bn' + str(stage) + block + '_branch' 93 | 94 | x = layers.Conv1D(filters1, 1, strides=strides, 95 | kernel_initializer='he_normal', 96 | name=conv_name_base + '2a')(input_tensor) 97 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2a')(x) 98 | x = layers.Activation('relu')(x) 99 | 100 | x = layers.Conv1D(filters2, kernel_size, padding='same', 101 | kernel_initializer='he_normal', 102 | name=conv_name_base + '2b')(x) 103 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2b')(x) 104 | x = layers.Activation('relu')(x) 105 | 106 | x = layers.Conv1D(filters3, 1, 107 | kernel_initializer='he_normal', 108 | name=conv_name_base + '2c')(x) 109 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2c')(x) 110 | 111 | shortcut = layers.Conv1D(filters3, 1, strides=strides, 112 | kernel_initializer='he_normal', 113 | name=conv_name_base + '1')(input_tensor) 114 | 115 | shortcut = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '1')(shortcut) 116 | 117 | x = layers.add([x, shortcut]) 118 | x = layers.Activation('relu')(x) 119 | return x 120 | 121 | 122 | def create_model(input_shape, classes, pooling=None, include_top=True, **kwargs): 123 | """Instantiates the ResNet50 architecture. 124 | Optionally loads weights pre-trained on ImageNet. 125 | Note that the data format convention used by the model is 126 | the one specified in your Keras config at `~/.keras/keras.json`. 127 | # Arguments 128 | include_top: whether to include the fully-connected 129 | layer at the top of the network. 130 | weights: one of `None` (random initialization), 131 | 'imagenet' (pre-training on ImageNet), 132 | or the path to the weights file to be loaded. 133 | input_tensor: optional Keras tensor (i.e. output of `layers.Input()`) 134 | to use as image input for the model. 135 | input_shape: optional shape tuple, only to be specified 136 | if `include_top` is False (otherwise the input shape 137 | has to be `(224, 224, 3)` (with `channels_last` data format) 138 | or `(3, 224, 224)` (with `channels_first` data format). 139 | It should have exactly 3 inputs channels, 140 | and width and height should be no smaller than 32. 141 | E.g. `(200, 200, 3)` would be one valid value. 142 | pooling: Optional pooling mode for feature extraction 143 | when `include_top` is `False`. 144 | - `None` means that the output of the model will be 145 | the 4D tensor output of the 146 | last convolutional block. 147 | - `avg` means that global average pooling 148 | will be applied to the output of the 149 | last convolutional block, and thus 150 | the output of the model will be a 1D tensor. 151 | - `max` means that global max pooling will 152 | be applied. 153 | classes: optional number of classes to classify images 154 | into, only to be specified if `include_top` is True, and 155 | if no `weights` argument is specified. 156 | # Returns 157 | A Keras model instance. 158 | # Raises 159 | ValueError: in case of invalid argument for `weights`, 160 | or invalid input shape. 161 | """ 162 | if backend.image_data_format() == 'channels_last': 163 | bn_axis = 2 164 | else: 165 | bn_axis = 1 166 | 167 | img_input = layers.Input(shape=input_shape) 168 | 169 | x = layers.ZeroPadding1D(padding=3, name='conv1_pad')(img_input) 170 | x = layers.Conv1D(64, 7, 171 | strides=2, 172 | padding='valid', 173 | kernel_initializer='he_normal', 174 | name='conv1')(x) 175 | x = layers.BatchNormalization(axis=bn_axis, name='bn_conv1')(x) 176 | x = layers.Activation('relu')(x) 177 | x = layers.ZeroPadding1D(padding=1, name='pool1_pad')(x) 178 | x = layers.MaxPooling1D(3, strides=2)(x) 179 | 180 | x = conv_block(x, 3, [64, 64, 256], stage=2, block='a', strides=1) 181 | x = identity_block(x, 3, [64, 64, 256], stage=2, block='b') 182 | x = identity_block(x, 3, [64, 64, 256], stage=2, block='c') 183 | 184 | x = conv_block(x, 3, [128, 128, 512], stage=3, block='a') 185 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='b') 186 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='c') 187 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='d') 188 | 189 | x = conv_block(x, 3, [256, 256, 1024], stage=4, block='a') 190 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='b') 191 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='c') 192 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='d') 193 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='e') 194 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='f') 195 | 196 | x = conv_block(x, 3, [512, 512, 2048], stage=5, block='a') 197 | x = identity_block(x, 3, [512, 512, 2048], stage=5, block='b') 198 | x = identity_block(x, 3, [512, 512, 2048], stage=5, block='c') 199 | 200 | if include_top: 201 | x = layers.GlobalAveragePooling1D(name='avg_pool')(x) 202 | x = layers.Dense(classes, activation='softmax', name='fc1000')(x) 203 | else: 204 | if pooling == 'avg': 205 | x = layers.GlobalAveragePooling1D()(x) 206 | elif pooling == 'max': 207 | x = layers.GlobalMaxPooling1D()(x) 208 | else: 209 | warnings.warn('The output shape of `ResNet50(include_top=False)` ' 210 | 'has been changed since Keras 2.2.0.') 211 | 212 | # Create model. 213 | inputs = img_input 214 | model = models.Model(inputs, x, name='resnet50') 215 | 216 | return model 217 | -------------------------------------------------------------------------------- /fine-tuning/resnet50_2D.py: -------------------------------------------------------------------------------- 1 | """ResNet50 model for Keras. 2 | # Reference: 3 | - [Deep Residual Learning for Image Recognition]( 4 | https://arxiv.org/abs/1512.03385) (CVPR 2016 Best Paper Award) 5 | Adapted from code contributed by BigMoyan. 6 | """ 7 | import os 8 | import pdb 9 | import warnings 10 | 11 | from keras import layers 12 | from keras import models 13 | import keras.utils as keras_utils 14 | import keras.backend as backend 15 | 16 | from imagenet_utils import get_submodules_from_kwargs 17 | import imagenet_utils 18 | from imagenet_utils import decode_predictions 19 | from imagenet_utils import _obtain_input_shape 20 | 21 | preprocess_input = imagenet_utils.preprocess_input 22 | 23 | WEIGHTS_PATH = ('https://github.com/fchollet/deep-learning-models/' 24 | 'releases/download/v0.2/' 25 | 'resnet50_weights_tf_dim_ordering_tf_kernels.h5') 26 | WEIGHTS_PATH_NO_TOP = ('https://github.com/fchollet/deep-learning-models/' 27 | 'releases/download/v0.2/' 28 | 'resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5') 29 | 30 | 31 | def identity_block(input_tensor, kernel_size, filters, stage, block): 32 | """The identity block is the block that has no conv layer at shortcut. 33 | # Arguments 34 | input_tensor: input tensor 35 | kernel_size: default 3, the kernel size of 36 | middle conv layer at main path 37 | filters: list of integers, the filters of 3 conv layer at main path 38 | stage: integer, current stage label, used for generating layer names 39 | block: 'a','b'..., current block label, used for generating layer names 40 | # Returns 41 | Output tensor for the block. 42 | """ 43 | filters1, filters2, filters3 = filters 44 | if backend.image_data_format() == 'channels_last': 45 | bn_axis = 3 46 | else: 47 | bn_axis = 1 48 | conv_name_base = 'res' + str(stage) + block + '_branch' 49 | bn_name_base = 'bn' + str(stage) + block + '_branch' 50 | 51 | x = layers.Conv2D(filters1, (1, 1), 52 | kernel_initializer='he_normal', 53 | name=conv_name_base + '2a')(input_tensor) 54 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2a')(x) 55 | x = layers.Activation('relu')(x) 56 | 57 | x = layers.Conv2D(filters2, kernel_size, 58 | padding='same', 59 | kernel_initializer='he_normal', 60 | name=conv_name_base + '2b')(x) 61 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2b')(x) 62 | x = layers.Activation('relu')(x) 63 | 64 | x = layers.Conv2D(filters3, (1, 1), 65 | kernel_initializer='he_normal', 66 | name=conv_name_base + '2c')(x) 67 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2c')(x) 68 | 69 | x = layers.add([x, input_tensor]) 70 | x = layers.Activation('relu')(x) 71 | return x 72 | 73 | 74 | def conv_block(input_tensor, 75 | kernel_size, 76 | filters, 77 | stage, 78 | block, 79 | strides=(2, 2)): 80 | """A block that has a conv layer at shortcut. 81 | # Arguments 82 | input_tensor: input tensor 83 | kernel_size: default 3, the kernel size of 84 | middle conv layer at main path 85 | filters: list of integers, the filters of 3 conv layer at main path 86 | stage: integer, current stage label, used for generating layer names 87 | block: 'a','b'..., current block label, used for generating layer names 88 | strides: Strides for the first conv layer in the block. 89 | # Returns 90 | Output tensor for the block. 91 | Note that from stage 3, 92 | the first conv layer at main path is with strides=(2, 2) 93 | And the shortcut should have strides=(2, 2) as well 94 | """ 95 | filters1, filters2, filters3 = filters 96 | if backend.image_data_format() == 'channels_last': 97 | bn_axis = 3 98 | else: 99 | bn_axis = 1 100 | conv_name_base = 'res' + str(stage) + block + '_branch' 101 | bn_name_base = 'bn' + str(stage) + block + '_branch' 102 | 103 | x = layers.Conv2D(filters1, (1, 1), strides=strides, 104 | kernel_initializer='he_normal', 105 | name=conv_name_base + '2a')(input_tensor) 106 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2a')(x) 107 | x = layers.Activation('relu')(x) 108 | 109 | x = layers.Conv2D(filters2, kernel_size, padding='same', 110 | kernel_initializer='he_normal', 111 | name=conv_name_base + '2b')(x) 112 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2b')(x) 113 | x = layers.Activation('relu')(x) 114 | 115 | x = layers.Conv2D(filters3, (1, 1), 116 | kernel_initializer='he_normal', 117 | name=conv_name_base + '2c')(x) 118 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2c')(x) 119 | 120 | shortcut = layers.Conv2D(filters3, (1, 1), strides=strides, 121 | kernel_initializer='he_normal', 122 | name=conv_name_base + '1')(input_tensor) 123 | shortcut = layers.BatchNormalization( 124 | axis=bn_axis, name=bn_name_base + '1')(shortcut) 125 | 126 | x = layers.add([x, shortcut]) 127 | x = layers.Activation('relu')(x) 128 | return x 129 | 130 | 131 | def create_model(input_shape, classes, pooling=None, include_top=True, **kwargs): 132 | """Instantiates the ResNet50 architecture. 133 | Optionally loads weights pre-trained on ImageNet. 134 | Note that the data format convention used by the model is 135 | the one specified in your Keras config at `~/.keras/keras.json`. 136 | # Arguments 137 | include_top: whether to include the fully-connected 138 | layer at the top of the network. 139 | weights: one of `None` (random initialization), 140 | 'imagenet' (pre-training on ImageNet), 141 | or the path to the weights file to be loaded. 142 | input_tensor: optional Keras tensor (i.e. output of `layers.Input()`) 143 | to use as image input for the model. 144 | input_shape: optional shape tuple, only to be specified 145 | if `include_top` is False (otherwise the input shape 146 | has to be `(224, 224, 3)` (with `channels_last` data format) 147 | or `(3, 224, 224)` (with `channels_first` data format). 148 | It should have exactly 3 inputs channels, 149 | and width and height should be no smaller than 32. 150 | E.g. `(200, 200, 3)` would be one valid value. 151 | pooling: Optional pooling mode for feature extraction 152 | when `include_top` is `False`. 153 | - `None` means that the output of the model will be 154 | the 4D tensor output of the 155 | last convolutional block. 156 | - `avg` means that global average pooling 157 | will be applied to the output of the 158 | last convolutional block, and thus 159 | the output of the model will be a 2D tensor. 160 | - `max` means that global max pooling will 161 | be applied. 162 | classes: optional number of classes to classify images 163 | into, only to be specified if `include_top` is True, and 164 | if no `weights` argument is specified. 165 | # Returns 166 | A Keras model instance. 167 | # Raises 168 | ValueError: in case of invalid argument for `weights`, 169 | or invalid input shape. 170 | """ 171 | 172 | # Determine proper input shape 173 | if not input_shape: 174 | input_shape = _obtain_input_shape(input_shape, 175 | default_size=224, 176 | min_size=32, 177 | data_format=backend.image_data_format(), 178 | require_flatten=include_top, 179 | weights=weights) 180 | 181 | img_input = layers.Input(shape=input_shape) 182 | 183 | if backend.image_data_format() == 'channels_last': 184 | bn_axis = 3 185 | else: 186 | bn_axis = 1 187 | 188 | x = layers.ZeroPadding2D(padding=(3, 3), name='conv1_pad')(img_input) 189 | x = layers.Conv2D(64, (7, 7), 190 | strides=(2, 2), 191 | padding='valid', 192 | kernel_initializer='he_normal', 193 | name='conv1')(x) 194 | x = layers.BatchNormalization(axis=bn_axis, name='bn_conv1')(x) 195 | x = layers.Activation('relu')(x) 196 | x = layers.ZeroPadding2D(padding=(1, 1), name='pool1_pad')(x) 197 | x = layers.MaxPooling2D((3, 3), strides=(2, 2))(x) 198 | 199 | x = conv_block(x, 3, [64, 64, 256], stage=2, block='a', strides=(1, 1)) 200 | x = identity_block(x, 3, [64, 64, 256], stage=2, block='b') 201 | x = identity_block(x, 3, [64, 64, 256], stage=2, block='c') 202 | 203 | x = conv_block(x, 3, [128, 128, 512], stage=3, block='a') 204 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='b') 205 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='c') 206 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='d') 207 | 208 | x = conv_block(x, 3, [256, 256, 1024], stage=4, block='a') 209 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='b') 210 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='c') 211 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='d') 212 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='e') 213 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='f') 214 | 215 | x = conv_block(x, 3, [512, 512, 2048], stage=5, block='a') 216 | x = identity_block(x, 3, [512, 512, 2048], stage=5, block='b') 217 | x = identity_block(x, 3, [512, 512, 2048], stage=5, block='c') 218 | 219 | if include_top: 220 | x = layers.GlobalAveragePooling2D(name='avg_pool')(x) 221 | x = layers.Dense(classes, activation='softmax', name='fc1000')(x) 222 | else: 223 | if pooling == 'avg': 224 | x = layers.GlobalAveragePooling2D()(x) 225 | elif pooling == 'max': 226 | x = layers.GlobalMaxPooling2D()(x) 227 | else: 228 | warnings.warn('The output shape of `ResNet50(include_top=False)` ' 229 | 'has been changed since Keras 2.2.0.') 230 | 231 | # Ensure that the model takes into account 232 | # any potential predecessors of `input_tensor`. 233 | inputs = img_input 234 | 235 | # Create model. 236 | model = models.Model(inputs, x, name='resnet50') 237 | 238 | return model 239 | -------------------------------------------------------------------------------- /fine-tuning/tools.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3.6 2 | 3 | import os 4 | import sys 5 | import subprocess 6 | import tempfile 7 | import random 8 | import time 9 | from collections import defaultdict 10 | import pdb 11 | 12 | import numpy as np 13 | import logging 14 | 15 | 16 | TMP_DIR = '/tmp/labtest' 17 | if not os.path.isdir(TMP_DIR): 18 | os.makedirs(TMP_DIR) 19 | 20 | 21 | def getVriusTotalHashKey(): 22 | key = 'fd2d772ac104caac9b92cee4d45d9043144bf18eb4bb6df5c97b4a764345ab89' 23 | return key 24 | 25 | 26 | def get_date(): 27 | return time.strftime("%Y_%m_%d", time.localtime()) 28 | 29 | 30 | def get_time(): 31 | return time.strftime("time_%H_%M_%S", time.localtime()) 32 | 33 | 34 | def get_pid(): 35 | return str(os.getpid()) 36 | 37 | 38 | def makeTempFile(suffix='', prefix='', dir=None, keepfile=False): 39 | if not prefix: 40 | prefix = 'tmp{}_'.format(get_pid()) 41 | if not dir: 42 | dir = TMP_DIR 43 | dir = os.path.join(dir, get_date()) 44 | if not os.path.isdir(dir): 45 | os.makedirs(dir) 46 | fd, fname = tempfile.mkstemp(suffix=suffix, prefix=prefix, dir=dir) 47 | if not keepfile: 48 | os.close(fd) 49 | return fname 50 | 51 | 52 | def makeTempDir(suffix='', prefix='', dir=None): 53 | if not prefix: 54 | prefix = 'tmp{}_'.format(get_pid()) 55 | if not dir: 56 | dir = TMP_DIR 57 | dir = os.path.join(dir, get_date()) 58 | if not os.path.isdir(dir): 59 | os.makedirs(dir) 60 | dname = tempfile.mkdtemp(suffix=suffix, prefix=prefix, dir=dir) 61 | 62 | return dname 63 | 64 | 65 | def getLogger(appName='default is empty'): 66 | logger = logging.getLogger(appName) 67 | logger.setLevel(logging.DEBUG) 68 | # create console handler with a higher log level 69 | ch = logging.StreamHandler() 70 | ch.setLevel(logging.ERROR) 71 | # create formatter and add it to the handlers 72 | formater = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') 73 | ch.setFormatter(formater) 74 | logger.addHandler(ch) 75 | return logger 76 | 77 | 78 | def getSectionList(start, end, interval): 79 | rangeList = [start] 80 | while 1: 81 | tmpPoint = start + interval 82 | if tmpPoint >= end: 83 | break 84 | rangeList.append(tmpPoint) 85 | start = tmpPoint 86 | rangeList[-1] = end 87 | secList = [0 for n in range(len(rangeList)-1)] 88 | return rangeList, secList 89 | 90 | 91 | def computeRange(rangeList, feature): 92 | l = len(rangeList) - 1 93 | for i in range(l): 94 | x1 = rangeList[i] 95 | x2 = rangeList[i+1] 96 | if x1 <= feature < x2: 97 | return i 98 | 99 | raise ValueError('the value of feature == {} exceed the rangeList'.format(feature)) 100 | 101 | 102 | def shuffleData(X, y): 103 | if not isinstance(X, np.ndarray): 104 | X = np.array(X) 105 | if not isinstance(y, np.ndarray): 106 | y = np.array(y) 107 | assert(X.shape[0] == y.shape[0]) 108 | 109 | # pair up 110 | tupList = [] 111 | for i in range(y.shape[0]): 112 | if 2 == len(X.shape): 113 | tmp_tuple = (X[i, :], y[i]) 114 | elif 1 == len(X.shape): 115 | tmp_tuple = (X[i], y[i]) 116 | elif 3 == len(X.shape): 117 | tmp_tuple = (X[i, :, :], y[i]) 118 | else: 119 | raise ValueError('data shape {} not supported yet'.format(X.shape)) 120 | tupList.append(tmp_tuple) 121 | 122 | random.shuffle(tupList) 123 | X, y = [], [] 124 | for i in range(len(tupList)): 125 | X.append(tupList[i][0]) 126 | y.append(tupList[i][1]) 127 | 128 | X = np.array(X) 129 | y = np.array(y) 130 | return X, y 131 | 132 | 133 | def datadict2data(datadict, keys=[], shuffle=True): 134 | allData, allLabel = [], [] 135 | 136 | if not keys: 137 | keys = list(datadict.keys()) 138 | 139 | for key in keys: 140 | oneCls = datadict[key] 141 | oneLabel = np.ones(len(oneCls)) * int(float(key)) 142 | allData.extend(oneCls) 143 | allLabel.extend(oneLabel) 144 | 145 | if shuffle: 146 | allData, allLabel = shuffleData(allData, allLabel) 147 | 148 | return allData, allLabel 149 | 150 | 151 | def data2datadict(allData, allLabel, clsLimit=0, sampleLimit=0): 152 | ''' 153 | expected input are numpy ndarry 154 | ''' 155 | if not isinstance(allData, np.ndarray): 156 | allData = np.array(allData) 157 | if not isinstance(allLabel, np.ndarray): 158 | allLabel = np.array(allLabel) 159 | 160 | datadict = defaultdict(list) 161 | 162 | allCls = list(set(allLabel)) 163 | if clsLimit: 164 | allCls = random.sample(allCls, clsLimit) 165 | 166 | for i in range(len(allLabel)): 167 | label = allLabel[i] 168 | if label in allCls: 169 | if len(allData.shape) == 2: 170 | sample = allData[i, :] 171 | elif len(allData.shape) == 1: 172 | sample = allData[i] 173 | else: 174 | raise ValueError('data shape {} not supported yet'.format(allData.shape)) 175 | datadict[label].append(sample) 176 | 177 | count = 0 178 | new_dict = defaultdict(list) 179 | for key in datadict.keys(): 180 | oneClsData = datadict[key] 181 | new_dict[count] = oneClsData 182 | count += 1 183 | 184 | del datadict 185 | 186 | if sampleLimit: 187 | for key in new_dict.keys(): 188 | oneClsData = new_dict[key] 189 | if sampleLimit >= len(oneClsData): 190 | new_samp = oneClsData[:sampleLimit] 191 | else: 192 | new_samp = random.sample(oneClsData, sampleLimit) 193 | new_dict[key] = new_samp 194 | 195 | return new_dict 196 | 197 | 198 | def limitData(allData, allLabel, clsLimit=0, sampleLimit=0): 199 | dataDict = data2datadict(allData, allLabel, clsLimit, sampleLimit) 200 | x_new, y_new = datadict2data(dataDict) 201 | return x_new, y_new 202 | 203 | 204 | def divideData(allData, allLabel, train_sample_num=5, train_pool_size=20): 205 | data_dict = data2datadict(allData, allLabel) 206 | train_data, train_label, test_data, test_label = [], [], [], [] 207 | oneClsNum = len(list(data_dict[0])) 208 | test_sample_num = oneClsNum - train_pool_size 209 | 210 | for key in data_dict.keys(): 211 | oneCls = list(data_dict[key]) 212 | random.shuffle(oneCls) 213 | train_pool = [] 214 | for i in range(train_pool_size): 215 | tmp = oneCls.pop() 216 | train_pool.append(tmp) 217 | train_data.extend(train_pool[:train_sample_num]) 218 | tmpLabels = np.ones(train_sample_num, dtype=np.int) * key 219 | train_label.extend(tmpLabels) 220 | 221 | test_data.extend(oneCls[:test_sample_num]) 222 | tmpLabels = np.ones(test_sample_num, dtype=np.int) * key 223 | test_label.extend(tmpLabels) 224 | 225 | train_data, train_label = shuffleData(train_data, train_label) 226 | test_data, test_label = shuffleData(test_data, test_label) 227 | return train_data, train_label, test_data, test_label 228 | 229 | 230 | def divideDataDict(data_dict, train_sample_num=5, train_pool_size=20): 231 | train_data, train_label, test_data, test_label = [], [], [], [] 232 | keys = list(data_dict.keys()) 233 | oneClsNum = len(list(data_dict[keys[0]])) 234 | test_sample_num = oneClsNum - train_pool_size 235 | 236 | for key in data_dict.keys(): 237 | oneCls = list(data_dict[key]) 238 | random.shuffle(oneCls) 239 | train_pool = [] 240 | for i in range(train_pool_size): 241 | tmp = oneCls.pop() 242 | train_pool.append(tmp) 243 | train_data.extend(train_pool[:train_sample_num]) 244 | tmpLabels = np.ones(train_sample_num, dtype=np.int) * key 245 | train_label.extend(tmpLabels) 246 | 247 | tmpData = oneCls[:test_sample_num] 248 | tmpLabels = np.ones(len(tmpData), dtype=np.int) * key 249 | test_data.extend(tmpData) 250 | test_label.extend(tmpLabels) 251 | 252 | train_data, train_label = shuffleData(train_data, train_label) 253 | test_data, test_label = shuffleData(test_data, test_label) 254 | return train_data, train_label, test_data, test_label 255 | 256 | 257 | def highLighPrint(msg): 258 | print('\033[40;33m {} \033[0m'.format(msg)) 259 | 260 | 261 | if __name__ == "__main__": 262 | aaa = np.array([[1, 2, 3], [2, 3, 4], [3, 4, 5], [4, 5, 6]]) 263 | bbb = np.array([1, 2, 3, 4]) 264 | xxx, yyy = shuffleData(aaa, bbb) 265 | print(xxx) 266 | print(yyy) 267 | -------------------------------------------------------------------------------- /fine-tuning/utils.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin python3.6 2 | 3 | import os 4 | import sys 5 | import pdb 6 | 7 | import numpy as np 8 | import config 9 | 10 | 11 | def generateIndex(allDataSize): 12 | np.random.seed(42) 13 | shuffledind = np.random.permutation(allDataSize) 14 | return shuffledind 15 | 16 | 17 | def getSplitIndex(allDataSize, splitRatio): 18 | shuffledind = generateIndex(allDataSize) 19 | 20 | train_set_size = int(allDataSize * splitRatio['train']) 21 | val_set_size = int(allDataSize * splitRatio['val']) 22 | # test_set_size = int(allDataSize * splitRatio['test']) 23 | 24 | start, end = 0, train_set_size 25 | train_ind = shuffledind[start: end] 26 | 27 | start, end = train_set_size, train_set_size + val_set_size 28 | val_ind = shuffledind[start: end] 29 | 30 | ''' 31 | start, end = train_set_size + val_set_size, train_set_size + val_set_size + test_set_size 32 | test_ind = shuffledind[start: end] 33 | ''' 34 | 35 | #return train_ind, val_ind, test_ind 36 | return train_ind, val_ind 37 | 38 | 39 | def splitData(splitRatio, allData, allLabel, splitType='random'): 40 | if not isinstance(allData, np.ndarray): 41 | allData = np.array(allData) 42 | allLabel = np.array(allLabel) 43 | 44 | if splitType == 'random': 45 | allDataSize = len(allLabel) 46 | #train_ind, val_ind, test_ind = getSplitIndex(allDataSize, splitRatio) 47 | train_ind, val_ind = getSplitIndex(allDataSize, splitRatio) 48 | 49 | trainData, trainLabels = allData[train_ind, :, :], allLabel[train_ind] 50 | valData, valLabels = allData[val_ind, :, :], allLabel[val_ind] 51 | #testData, testLabels = allData[test_ind, :, :], allLabel[test_ind] 52 | 53 | elif splitType == 'order': 54 | pass 55 | else: 56 | raise 57 | 58 | #return trainData, trainLabels, valData, valLabels, testData, testLabels 59 | return trainData, trainLabels, valData, valLabels 60 | 61 | 62 | if __name__ == "__main__": 63 | opts = config.parse_args(sys.argv) 64 | print('all test passed!') 65 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | h5py==2.8.0 2 | ipdb==0.11 3 | ipython==5.8.0 4 | Keras==2.2.4 5 | matplotlib==2.2.3 6 | numpy 7 | # resampy==0.2.1 8 | scikit-learn==0.20.4 9 | scipy==1.1.0 10 | Theano==1.0.3 11 | tqdm==4.28.1 12 | -------------------------------------------------------------------------------- /rf/__pycache__/config.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/rf/__pycache__/config.cpython-36.pyc -------------------------------------------------------------------------------- /rf/__pycache__/get_simu_data.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/rf/__pycache__/get_simu_data.cpython-36.pyc -------------------------------------------------------------------------------- /rf/__pycache__/imagenet_utils.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/rf/__pycache__/imagenet_utils.cpython-36.pyc -------------------------------------------------------------------------------- /rf/__pycache__/load_slice_IQ.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/rf/__pycache__/load_slice_IQ.cpython-36.pyc -------------------------------------------------------------------------------- /rf/__pycache__/radioConv.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/rf/__pycache__/radioConv.cpython-36.pyc -------------------------------------------------------------------------------- /rf/__pycache__/readSigmf2.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/rf/__pycache__/readSigmf2.cpython-36.pyc -------------------------------------------------------------------------------- /rf/__pycache__/resnet50_1D.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/rf/__pycache__/resnet50_1D.cpython-36.pyc -------------------------------------------------------------------------------- /rf/__pycache__/resnet50_2D.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/rf/__pycache__/resnet50_2D.cpython-36.pyc -------------------------------------------------------------------------------- /rf/__pycache__/utils.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SmartHomePrivacyProject/RadioFingerprinting/fa91294b72f28dd90c0ccfd35f2d9a1e9d4cfa51/rf/__pycache__/utils.cpython-36.pyc -------------------------------------------------------------------------------- /rf/compare_data.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin python3.6 2 | 3 | import os 4 | import sys 5 | import pdb 6 | import math 7 | 8 | import json 9 | import numpy as np 10 | from sigmf import SigMFFile, sigmffile 11 | 12 | import config 13 | 14 | 15 | class OurWay(): 16 | def __init__(self, droot): 17 | fnames = os.listdir(droot) 18 | for fn in fnames: 19 | if fn.endswith('bin'): 20 | self.binfile = os.path.join(droot, fn) 21 | elif fn.endswith('sigmf-meta'): 22 | self.metafile = os.path.join(droot, fn) 23 | 24 | def createSignal(self, metafile, binfile): 25 | # Load a dataset 26 | with open(metafile, 'r') as f: 27 | metadata = json.loads(f.read()) 28 | signal = SigMFFile(metadata=metadata['_metadata'], data_file=binfile) 29 | return signal 30 | 31 | def convert2IQdata(self, one_raw_data): 32 | realList, imagList = [], [] 33 | for item in one_raw_data: 34 | realVal = np.real(item) 35 | imagVal = np.imag(item) 36 | if math.isnan(realVal): 37 | realVal = 0 38 | if math.isnan(imagVal): 39 | imagVal = 0 40 | realList.append(realVal) 41 | imagList.append(imagVal) 42 | rtnList = [realList, imagList] 43 | rtnList = np.array(rtnList) 44 | return rtnList 45 | 46 | def getOneData(self): 47 | # Get some metadata and all annotations 48 | sample_length = 288 49 | signal = self.createSignal(self.metafile, self.binfile) 50 | raw_data = signal.read_samples(0, -1) 51 | one_data = self.convert2IQdata(raw_data) 52 | pdb.set_trace() 53 | return one_data[:, :sample_length] 54 | 55 | 56 | class AuthorWay(): 57 | def __init__(self, droot): 58 | fnames = os.listdir(droot) 59 | for fn in fnames: 60 | if fn.endswith('bin'): 61 | fp = os.path.join(droot, fn) 62 | self.filename = fp 63 | 64 | def read_f32_bin(self, filename, start_ix=0): 65 | with open(filename, 'rb') as bin_f: 66 | iq_seq = np.fromfile(bin_f, dtype='= n_samples: 64 | isEnoughSample = True 65 | break 66 | else: 67 | isEnoughSample = True 68 | 69 | if not isEnoughSample: 70 | print("ERROR! There are not enough samples to satisfy dataset parameters. Aborting...") 71 | sys.exit(-1) 72 | 73 | if channel_first: 74 | all_IQ_data = all_IQ_data[:, 0:n_samples] 75 | else: 76 | all_IQ_data = all_IQ_data[0:n_samples, :] 77 | return all_IQ_data, num_tran 78 | 79 | 80 | def loadData(args, channel_first): 81 | n_slices_per_dev = args.num_slice 82 | start_ix = args.start_ix 83 | file_key = args.file_key 84 | 85 | dev_dir_list = [] 86 | dev_dir_names = os.listdir(args.root_dir) 87 | for n in dev_dir_names: 88 | tmp = os.path.join(args.root_dir, n) 89 | dev_dir_list.append(tmp) 90 | 91 | stride = args.stride 92 | n_devices = len(dev_dir_list) 93 | # locations = ["after_fft","before_fft", "output_equ", "symbols"] 94 | # locations = ["output_equ"] 95 | if channel_first: 96 | slice_dims = (2, args.slice_len) 97 | samps_to_retrieve = (n_slices_per_dev - 1) * stride + slice_dims[1] 98 | else: 99 | slice_dims = (args.slice_len, 2) 100 | samps_to_retrieve = (n_slices_per_dev - 1) * stride + slice_dims[0] 101 | 102 | x_train, y_train, x_test, y_test = [], [], [], [] 103 | split_ratio = {'train': 0.8, 'val': 0.2} 104 | for i, d in enumerate(dev_dir_list): 105 | 106 | p = os.path.join(d, args.location) 107 | pre_X_data, num_tran = dev_bin_dataset(os.path.join(p, file_key), samps_to_retrieve, start_ix, channel_first, uniform=True) 108 | 109 | X_data_pd = [] 110 | count_s = 0 111 | for j in range(0, samps_to_retrieve, stride): 112 | if channel_first: 113 | X_data_pd.append(pre_X_data[:, j:j+slice_dims[1]]) 114 | else: 115 | X_data_pd.append(pre_X_data[j:j+slice_dims[0], :]) 116 | count_s += 1 117 | if count_s == n_slices_per_dev: 118 | break 119 | X_data_pd = np.array(X_data_pd) 120 | y_data_pd = i * np.ones(n_slices_per_dev, ) 121 | 122 | # split one class data 123 | uniform = True 124 | x_train_pd, x_test_pd, y_train_pd, y_test_pd = [],[],[],[] 125 | if uniform: 126 | samples_per_tran = n_slices_per_dev // num_tran 127 | idx = 0 128 | while idx + samples_per_tran <= n_slices_per_dev: 129 | x_train_per_tran, y_train_per_tran, x_test_per_tran, y_test_per_tran = utils.splitData(split_ratio, X_data_pd[i:i+samples_per_tran, :, :], y_data_pd[i:i+samples_per_tran]) 130 | if idx == 0: 131 | x_train_pd, x_test_pd = x_train_per_tran, x_test_per_tran 132 | y_train_pd, y_test_pd = y_train_per_tran, y_test_per_tran 133 | else: 134 | x_train_pd = np.concatenate((x_train_pd, x_train_per_tran), axis=0) 135 | x_test_pd = np.concatenate((x_test_pd, x_test_per_tran), axis=0) 136 | y_train_pd = np.concatenate((y_train_pd, y_train_per_tran), axis=0) 137 | y_test_pd = np.concatenate((y_test_pd, y_test_per_tran), axis=0) 138 | idx += samples_per_tran 139 | else: 140 | 141 | x_train_pd, y_train_pd, x_test_pd, y_test_pd = utils.splitData(split_ratio, X_data_pd, y_data_pd) 142 | 143 | if i == 0: 144 | x_train, x_test = x_train_pd, x_test_pd 145 | y_train, y_test = y_train_pd, y_test_pd 146 | else: 147 | x_train = np.concatenate((x_train, x_train_pd), axis=0) 148 | x_test = np.concatenate((x_test, x_test_pd), axis=0) 149 | y_train = np.concatenate((y_train, y_train_pd), axis=0) 150 | y_test = np.concatenate((y_test, y_test_pd), axis=0) 151 | del pre_X_data 152 | del X_data_pd 153 | 154 | if args.D2: 155 | if channel_first: 156 | x_train = x_train[:, :, np.newaxis, :] 157 | x_test = x_test[:, :, np.newaxis, :] 158 | else: 159 | x_train = x_train[:, np.newaxis, :, :] 160 | x_test = x_test[:, np.newaxis, :, :] 161 | 162 | y_train = np_utils.to_categorical(y_train, n_devices) 163 | y_test = np_utils.to_categorical(y_test, n_devices) 164 | return x_train, y_train, x_test, y_test, n_devices 165 | 166 | 167 | class loadDataOpts(): 168 | def __init__(self, root_dir, location, D2, file_key='*.bin', num_slice=100000, start_ix=0, slice_len=288, stride=1): 169 | self.root_dir = root_dir 170 | self.num_slice = num_slice 171 | self.start_ix = start_ix 172 | self.slice_len = slice_len 173 | self.stride = stride 174 | self.file_key = file_key 175 | self.D2 = D2 176 | self.location = location 177 | 178 | 179 | def parseArgs(argv): 180 | Desc = 'Read and slice the collected I/Q samples' 181 | parser = argparse.ArgumentParser(description=Desc, 182 | formatter_class=argparse.ArgumentDefaultsHelpFormatter) 183 | parser.add_argument('-d', '--root_dir', required=True, help='Root directory for the devices\' folders.') 184 | parser.add_argument('-n', '--num_slice', required=True, type=int, help='Number of slices to be generated for each device.') 185 | parser.add_argument('-i', '--start_ix', type=int, default=0, help='Starting read index in .bin files.') 186 | parser.add_argument('-l', '--slice_len', type=int, default=288, help='Lenght of slices.') 187 | parser.add_argument('-s', '--stride', type=int, default=1, help='Stride used for windowing.') 188 | parser.add_argument('-f', '--file_key', default='*.bin', help='used to choose different filetype, choose from *.bin/*.sigmf-meta') 189 | parser.add_argument('--D2', action='store_true', help='') 190 | parser.add_argument('-cf', '--channel_first', action='store_true', help='if set channel first otherwise channel last') 191 | opts = parser.parse_args() 192 | return opts 193 | 194 | 195 | if __name__ == "__main__": 196 | opts = parseArgs(sys.argv) 197 | # opts = loadDataOpts(root_dir="/home/erc/PycharmProjects/rf/test_data/", D2=False, location='after_fft') 198 | channel_first = True 199 | x_train, y_train, x_test, y_test, NUM_CLASS = loadData(opts, channel_first) 200 | print('train data shape: ', x_train.shape, 'train label shape: ', y_train.shape) 201 | print('test data shape: ', x_test.shape, 'test label shape: ', y_test.shape) 202 | print('all test done!') 203 | -------------------------------------------------------------------------------- /rf/main.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | ''' 3 | Nov 6th: updated for D2 option, which can easily change between 1D and 2D with out change the code 4 | ''' 5 | import os 6 | import sys 7 | import argparse 8 | import pdb 9 | 10 | from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, TensorBoard 11 | from keras.utils import np_utils 12 | import numpy as np 13 | 14 | import radioConv 15 | #import readSigmf2 as readSigmf 16 | import load_slice_IQ 17 | import config 18 | import get_simu_data 19 | ROOT_DIR = os.path.dirname(os.path.abspath(__file__)) 20 | #ROOT_DIR = os.getenv('ROOT_DIR') 21 | print(ROOT_DIR) 22 | resDir = os.path.join(ROOT_DIR, 'resDir') 23 | modelDir = os.path.join(resDir, 'modelDir') 24 | os.makedirs(modelDir, exist_ok=True) 25 | 26 | 27 | def loadNeuData(opts): 28 | print('loading data...') 29 | x_day_dir = opts.input 30 | #train_x, train_y, val_x, val_y, test_x, test_y = readSigmf.getData(opts, x_day_dir) 31 | dataOpts = load_slice_IQ.loadDataOpts(opts.input, opts.location, opts.D2, num_slice=100000) 32 | train_x, train_y, test_x, test_y, NUM_CLASS = load_slice_IQ.loadData(dataOpts, opts.channel_first) 33 | 34 | if opts.normalize: 35 | train_x = load_slice_IQ.normalizeData(train_x) 36 | test_x = load_slice_IQ.normalizeData(test_x) 37 | 38 | return train_x, train_y, test_x, test_y, NUM_CLASS 39 | 40 | 41 | def loadSimuData(opts): 42 | simu_dict, NUM_CLASS = get_simu_data.loadData(opts.input) 43 | x_train, y_train = simu_dict['x_train'], simu_dict['y_train'] 44 | x_test, y_test = simu_dict['x_test'], simu_dict['y_test'] 45 | 46 | return x_train, y_train, x_test, y_test, NUM_CLASS 47 | 48 | 49 | def main(opts): 50 | # load data 51 | if 'neu' == opts.dataSource: 52 | train_x, train_y, test_x, test_y, NUM_CLASS = loadNeuData(opts) 53 | elif 'simu' == opts.dataSource: 54 | train_x, train_y, test_x, test_y, NUM_CLASS = loadSimuData(opts) 55 | else: 56 | raise NotImplementedError() 57 | 58 | # setup params 59 | Batch_Size = 64 60 | Epoch_Num = 100 61 | saveModelPath = os.path.join(modelDir, 'best_model_{}_{}.h5'.format(opts.modelType,opts.location)) 62 | checkpointer = ModelCheckpoint(filepath=saveModelPath, monitor='val_acc', verbose=1, save_best_only=True, mode='max') 63 | earlyStopper = EarlyStopping(monitor='val_acc', mode='max', patience=10) 64 | callBackList = [checkpointer, earlyStopper] 65 | 66 | print('get the model and compile it...') 67 | if opts.D2: 68 | inp_shape = (1, train_x.shape[1], train_x.shape[2]) 69 | else: 70 | inp_shape = (train_x.shape[1], train_x.shape[2]) 71 | 72 | # pdb.set_trace() 73 | model = radioConv.create_model(opts.modelType, inp_shape, NUM_CLASS, D2=opts.D2, channel='first') 74 | model.summary() 75 | model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) 76 | 77 | print('fit the model with data...') 78 | model.fit(x=train_x, y=train_y, 79 | batch_size=Batch_Size, 80 | epochs=Epoch_Num, 81 | verbose=opts.verbose, 82 | callbacks=callBackList, 83 | validation_split=0.1, 84 | shuffle=True) 85 | 86 | print('test the trained model...') 87 | score, acc = model.evaluate(test_x, test_y, batch_size=Batch_Size, verbose=0) 88 | print('test acc is: ', acc) 89 | 90 | print('all test done!') 91 | results = [opts.modelType, opts.location, acc] 92 | 93 | with open('results.txt', 'a') as filehandle: 94 | #for listitem in results: 95 | #filehandle.write('%s' % listitem) 96 | filehandle.write('%f\n' % acc) 97 | 98 | if __name__ == "__main__": 99 | opts = config.parse_args(sys.argv) 100 | main(opts) 101 | -------------------------------------------------------------------------------- /rf/prepare4complex.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | import os 4 | import sys 5 | import argparse 6 | import pdb 7 | 8 | import numpy as np 9 | from sklearn.model_selection import train_test_split 10 | 11 | import config 12 | import load_slice_IQ 13 | import mytools.tools as mytools 14 | 15 | ''' 16 | data need to organized into 3 parts: 17 | (train_x, train_y), (val_x, val_y), (test_x, test_y) 18 | and all data need to be normalize to (0,1) and into a mat 19 | with shape like (sample_num, sample_dim, 2), all complex 20 | value are stored in (real, imag) pair 21 | ''' 22 | #'\033[{i};1m这里写需要输出打印的内容\033[0m'# i是指打印颜色的编号 23 | print() 24 | #print('\033[40;33m required args includes input/output/D2/normalize/ \033[0m') 25 | msg = 'required args includes input/output/D2/normalize/' 26 | mytools.highLighPrint(msg) 27 | print() 28 | 29 | 30 | def loadData2Dict(opts): 31 | print('loading data...') 32 | # train_x, train_y, val_x, val_y, test_x, test_y = readSigmf.getData(opts, x_day_dir) 33 | # D2 means that make it into 2 dimension data 34 | dataOpts = load_slice_IQ.loadDataOpts(opts.input, opts.D2, num_slice=10000) 35 | train_x, train_y, test_x, test_y, NUM_CLASS = load_slice_IQ.loadData(dataOpts, opts.channel_first) 36 | 37 | if opts.normalize: 38 | train_x = load_slice_IQ.normalizeData(train_x) 39 | test_x = load_slice_IQ.normalizeData(test_x) 40 | 41 | data_dict = {} 42 | data_dict['x_train'] = train_x 43 | data_dict['y_train'] = train_y 44 | data_dict['x_test'] = test_x 45 | data_dict['y_test'] = test_y 46 | 47 | print('x_train shape: {}\ty_train shape: {}\tx_test shape: {}\ty_test shape: {}'.format(train_x.shape, train_y.shape, test_x.shape, test_y.shape)) 48 | 49 | return data_dict 50 | 51 | 52 | def save2file(outfile, data_dict): 53 | x_train, y_train = data_dict['x_train'], data_dict['y_train'] 54 | x_test, y_test = data_dict['x_test'], data_dict['y_test'] 55 | 56 | x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.2, random_state=47, shuffle=True) 57 | np.savez(outfile, x_train=x_train, y_train=y_train, x_val=x_val, y_val=y_val, x_test=x_test, y_test=y_test) 58 | outfile = os.path.abspath(outfile) 59 | print('all data save to path {}'.format(outfile)) 60 | 61 | 62 | def main(opts): 63 | tmp_list = opts.input.split('/') 64 | tmp_list = filter(lambda a: a != '', tmp_list) 65 | save_name = list(tmp_list)[-1] 66 | save_path = os.path.join(opts.output, '{}.npz'.format(save_name)) 67 | # load all the data 68 | data_dict = loadData2Dict(opts) 69 | 70 | # save data to npz file 71 | save2file(save_path, data_dict) 72 | 73 | print('all done!') 74 | 75 | 76 | if __name__ == "__main__": 77 | opts = config.parse_args(sys.argv) 78 | main(opts) 79 | -------------------------------------------------------------------------------- /rf/radioConv.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/python3 2 | ''' 3 | Nov 6h, updated the API, pass a D2 value to make it adapt to 1D or 2D Conv model, all test done! 4 | ''' 5 | import os 6 | import sys 7 | import argparse 8 | import pdb 9 | from keras.layers import Dense, Dropout, Flatten, Input 10 | from keras.layers import Activation, BatchNormalization 11 | from keras.models import Sequential, Model 12 | from keras.callbacks import ModelCheckpoint, LearningRateScheduler 13 | from keras.callbacks import EarlyStopping, Callback 14 | import keras.backend as K 15 | from keras.utils import np_utils 16 | 17 | import numpy as np 18 | import pandas as pd 19 | 20 | import config 21 | 22 | #ROOT_DIR = os.getenv('PROJECT_DIR') 23 | #print(ROOT_DIR) 24 | #resDir = os.path.join(ROOT_DIR, 'resDir') 25 | #modelDir = os.path.join(resDir, 'modelDir') 26 | #os.makedirs(modelDir, exist_ok=True) 27 | 28 | 29 | def createHomegrown(inp_shape, emb_size, data_format, D2): 30 | # -----------------Entry flow ----------------- 31 | if D2: 32 | from keras.layers import MaxPooling2D as MaxPooling 33 | from keras.layers import GlobalAveragePooling2D as GlobalAveragePooling 34 | from keras.layers import Conv2D as Conv 35 | else: 36 | from keras.layers import MaxPooling1D as MaxPooling 37 | from keras.layers import GlobalAveragePooling1D as GlobalAveragePooling 38 | from keras.layers import Conv1D as Conv 39 | 40 | input_data = Input(shape=inp_shape) 41 | if D2: 42 | kernel_size = ['None', (1, 7), (2, 5)] 43 | else: 44 | kernel_size = ['None', 7, 7] 45 | filter_num = ['None', 50, 50] 46 | conv_stride_size = ['None', 1, 1] 47 | pool_stride_size = ['None', 1, 1] 48 | activation_func = ['None', 'relu', 'relu'] 49 | #activation_func = ['None', 'elu', 'elu'] 50 | dense_layer_size = ['None', 256, 80] 51 | 52 | model = Conv(filters=filter_num[1], kernel_size=kernel_size[1], 53 | strides=conv_stride_size[1], padding='same', name='block1_conv1', data_format=data_format)(input_data) 54 | model = Activation(activation_func[1], name='block1_act1')(model) 55 | model = Dropout(0.5, name='block1_dropout')(model) 56 | 57 | model = Conv(filters=filter_num[2], kernel_size=kernel_size[2], 58 | strides=conv_stride_size[2], padding='same', name='block2_conv1', data_format=data_format)(model) 59 | model = Activation(activation_func[2], name='block2_act1')(model) 60 | model = Dropout(0.5, name='block2_dropout')(model) 61 | 62 | output = GlobalAveragePooling()(model) 63 | 64 | dense_layer = Dense(dense_layer_size[1], name='dense1', activation='relu')(output) 65 | dense_layer = Dense(dense_layer_size[2], name='dense2', activation='relu')(dense_layer) 66 | dense_layer = Dense(emb_size, name='dense3', activation='softmax')(dense_layer) 67 | 68 | shared_conv2 = Model(inputs=input_data, outputs=dense_layer) 69 | return shared_conv2 70 | 71 | 72 | def baselineBlock(input, block_idx, D2): 73 | if D2: 74 | from keras.layers import MaxPooling2D as MaxPooling 75 | from keras.layers import Conv2D as Conv 76 | else: 77 | import resnet50_1D as resnet50 78 | from keras.layers import MaxPooling1D as MaxPooling 79 | from keras.layers import Conv1D as Conv 80 | 81 | if D2: 82 | kernel_size = ['None', (1, 7), (2, 5)] 83 | else: 84 | kernel_size = ['None', 7, 5] 85 | filter_num = ['None', 128, 128] 86 | conv_stride = ['None', 1, 1] 87 | pool_size = ['None', 2] 88 | pool_stride = ['None', 1] 89 | act_func = 'relu' 90 | 91 | model = Conv(filters=filter_num[1], kernel_size=kernel_size[1], name='conv1_{}'.format(block_idx), 92 | strides=conv_stride[1], padding='same', activation=act_func)(input) 93 | model = Conv(filters=filter_num[2], kernel_size=kernel_size[2], name='conv2_{}'.format(block_idx), 94 | strides=conv_stride[2], padding='same', activation=act_func)(model) 95 | output = MaxPooling(pool_size=pool_size[1], strides=pool_stride[1], padding='same', 96 | name='pool_{}'.format(block_idx))(model) 97 | 98 | return output 99 | 100 | 101 | def createBaseline(inp_shape, emb_size, data_format, D2): 102 | if D2: 103 | from keras.layers import GlobalAveragePooling2D as GlobalAveragePooling 104 | else: 105 | from keras.layers import GlobalAveragePooling1D as GlobalAveragePooling 106 | 107 | dense_layer_size = ['None', 256, 256, 128] 108 | act_func = ['None', 'relu', 'relu', 'relu'] 109 | 110 | blockNum = 4 111 | input_data = Input(shape=inp_shape) 112 | for i in range(blockNum): 113 | idx = i + 1 114 | if 0 == i: 115 | model = baselineBlock(input_data, idx, D2) 116 | else: 117 | model = baselineBlock(model, idx, D2) 118 | 119 | middle = GlobalAveragePooling()(model) 120 | 121 | dense_layer = Dense(dense_layer_size[1], name='dense1', activation=act_func[1])(middle) 122 | dense_layer = Dense(dense_layer_size[2], name='dense2', activation=act_func[2])(dense_layer) 123 | dense_layer = Dense(dense_layer_size[3], name='dense3', activation=act_func[3])(dense_layer) 124 | dense_layer = Dense(emb_size, name='dense4', activation='softmax')(dense_layer) 125 | 126 | conv_model = Model(inputs=input_data, outputs=dense_layer) 127 | return conv_model 128 | 129 | 130 | def createResnet(inp_shape, emb_size, data_format, D2): 131 | if D2: 132 | import resnet50_2D as resnet50 133 | else: 134 | import resnet50_1D as resnet50 135 | return resnet50.create_model(inp_shape, emb_size) 136 | 137 | 138 | def create_model(modelType, inp_shape, NUM_CLASS, D2=False, channel='last'): 139 | 140 | if 'first' == channel: 141 | data_format = 'channels_first' 142 | elif 'last' == channel: 143 | data_format = 'channels_last' 144 | else: 145 | raise 146 | 147 | emb_size = NUM_CLASS 148 | if 'homegrown' == modelType: 149 | model = createHomegrown(inp_shape, emb_size, data_format, D2) 150 | elif 'baseline' == modelType: 151 | model = createBaseline(inp_shape, emb_size, data_format, D2) 152 | elif 'resnet' == modelType: 153 | model = createResnet(inp_shape, emb_size, data_format, D2) 154 | else: 155 | raise ValueError('model type {} not support yet'.format(modelType)) 156 | 157 | return model 158 | 159 | 160 | def test_run(model): 161 | model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy']) 162 | 163 | 164 | def test(D2): 165 | modelTypes = ['homegrown', 'baseline', 'resnet'] 166 | NUM_CLASS = 10 167 | signal = True 168 | inp_shape = (1, 2, 288) if D2 else (2, 288) 169 | 170 | for modelType in modelTypes: 171 | model = create_model(modelType, inp_shape, NUM_CLASS, D2=D2, channel='first') 172 | try: 173 | flag = test_run(model) 174 | except Exception as e: 175 | print(e) 176 | 177 | print('all done!') if signal else print('test failed') 178 | 179 | 180 | if __name__ == "__main__": 181 | D2_Type = [True, False] 182 | for D2 in D2_Type: 183 | test(D2) 184 | -------------------------------------------------------------------------------- /rf/readSigmf.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin python3.6 2 | 3 | import os 4 | import sys 5 | import pdb 6 | import argparse 7 | import random 8 | import re 9 | import math 10 | 11 | from collections import defaultdict 12 | import json 13 | import numpy as np 14 | import itertools 15 | from sigmf import SigMFFile, sigmffile 16 | 17 | import mytools.tools as mytools 18 | import config 19 | 20 | 21 | def createSignal(metafile, binfile): 22 | # Load a dataset 23 | with open(metafile, 'r') as f: 24 | metadata = json.loads(f.read()) 25 | signal = SigMFFile(metadata=metadata['_metadata'], data_file=binfile) 26 | return signal 27 | 28 | 29 | def get_one_sample(raw_data, start, length): 30 | if not isinstance(raw_data, np.ndarray): 31 | raw_data = np.array(raw_data) 32 | return raw_data[start: start+length] 33 | 34 | 35 | def convert2IQdata(one_raw_data): 36 | flag = False 37 | rtnList = [] 38 | for item in one_raw_data: 39 | realVal = np.real(item) 40 | imagVal = np.imag(item) 41 | if math.isnan(realVal): 42 | realVal = 0 43 | flag = True 44 | if math.isnan(imagVal): 45 | imagVal = 0 46 | flag = True 47 | tmp = [realVal, imagVal] 48 | rtnList.append(tmp) 49 | 50 | #if flag: 51 | # pdb.set_trace() 52 | #tmpMat = np.array(rtnList).T 53 | #rtnList = list(tmpMat) 54 | return rtnList 55 | 56 | 57 | def divideIntoChucks(raw_data, chuckNum): 58 | dataLen = len(raw_data) 59 | sliceLen = dataLen // chuckNum 60 | 61 | chuckList = [] 62 | start = 0 63 | for i in range(chuckNum): 64 | end = start + sliceLen 65 | oneSlice = raw_data[start: end] 66 | start = end 67 | chuckList.append(oneSlice) 68 | 69 | return chuckList 70 | 71 | 72 | def formInpData(raw_data, sample_length, selectedNum): 73 | start_range = len(raw_data) - sample_length 74 | raw_samples = [] 75 | for i in range(start_range): 76 | tmp_sample = get_one_sample(raw_data, i, sample_length) 77 | raw_samples.append(tmp_sample) 78 | 79 | selectedSamples = random.sample(raw_samples, selectedNum) 80 | rtn_samples = [] 81 | for tmp_sample in selectedSamples: 82 | tmp_sample = convert2IQdata(tmp_sample) 83 | rtn_samples.append(tmp_sample) 84 | 85 | return rtn_samples 86 | 87 | 88 | def getSplitIndex(allDataSize, splitRatio): 89 | np.random.seed(42) 90 | shuffledind = np.random.permutation(allDataSize) 91 | 92 | train_set_size = int(allDataSize * splitRatio['train']) 93 | val_set_size = int(allDataSize * splitRatio['val']) 94 | test_set_size = int(allDataSize * splitRatio['test']) 95 | 96 | start, end = 0, train_set_size 97 | train_ind = shuffledind[start: end] 98 | 99 | start, end = train_set_size, train_set_size + val_set_size 100 | val_ind = shuffledind[start: end] 101 | 102 | start, end = train_set_size + val_set_size, train_set_size + val_set_size + test_set_size 103 | test_ind = shuffledind[start: end] 104 | 105 | return train_ind, val_ind, test_ind 106 | 107 | 108 | def readDataFile(chuckList, sample_length, selectedNum): 109 | allData = [] 110 | for chuck in chuckList: 111 | oneChuck = formInpData(chuck, sample_length, selectedNum) 112 | allData.extend(oneChuck) 113 | return allData 114 | 115 | 116 | def splitData(opts, splitRatio, allData, allLabel): 117 | if not isinstance(allData, np.ndarray): 118 | allData = np.array(allData) 119 | allLabel = np.array(allLabel) 120 | 121 | if opts.splitType == 'random': 122 | allDataSize = len(allLabel) 123 | train_ind, val_ind, test_ind = getSplitIndex(allDataSize, splitRatio) 124 | trainData, trainLabels = allData[train_ind, :, :], allLabel[train_ind] 125 | valData, valLabels = allData[val_ind, :, :], allLabel[val_ind] 126 | testData, testLabels = allData[test_ind, :, :], allLabel[test_ind] 127 | elif opts.splitType == 'order': 128 | pass 129 | else: 130 | raise 131 | 132 | return trainData, trainLabels, valData, valLabels, testData, testLabels 133 | 134 | 135 | def get_signal_samples(signal, label, params): 136 | # Get some metadata and all annotations 137 | chuckNum = params['chuckNum'] 138 | sample_length = params['sample_length'] 139 | selectedNum = params['selectedNum'] 140 | 141 | raw_data = signal.read_samples(0, -1) 142 | chuckList = divideIntoChucks(raw_data, chuckNum) 143 | 144 | oneData = readDataFile(chuckList, sample_length, selectedNum) 145 | oneLabel = np.ones(len(oneData), dtype=np.int) * label 146 | print('raw data length is: ', len(oneData)) 147 | return oneData, oneLabel 148 | 149 | 150 | def searchFp(fname, metaFileList): 151 | for mfp in metaFileList: 152 | m = re.search(fname, mfp) 153 | if m: 154 | return mfp 155 | return '' 156 | 157 | 158 | def getSignalList(fpTuple): 159 | binFileList, metaFileList = fpTuple 160 | signalList = [] 161 | for bfp in binFileList: 162 | fname = os.path.basename(bfp).split('.')[0] 163 | mfp = searchFp(fname, metaFileList) 164 | if not mfp: 165 | raise ValueError('binfile {} does not have a match'.format(bfp)) 166 | signal = createSignal(mfp, bfp) 167 | signalList.append(signal) 168 | return signalList 169 | 170 | 171 | def getOneDevData(fpTuple, label, params): 172 | print('processing file: ', fpTuple) 173 | signalList = getSignalList(fpTuple) 174 | allData, allLabel = [], [] 175 | for signal in signalList: 176 | oneData, oneLabel = get_signal_samples(signal, label, params) 177 | allData.extend(oneData) 178 | allLabel.extend(oneLabel) 179 | 180 | return allData, allLabel 181 | 182 | 183 | def getfpTuple(strLabel, x_day_dir): 184 | dayDevDir = os.path.join(x_day_dir, strLabel) 185 | fList = os.listdir(dayDevDir) 186 | binFileList, metaFileList = [], [] 187 | for fname in fList: 188 | fp = os.path.join(dayDevDir, fname) 189 | if fp.endswith('bin'): 190 | binFileList.append(fp) 191 | elif fp.endswith('sigmf-meta'): 192 | metaFileList.append(fp) 193 | else: 194 | raise 195 | return (binFileList, metaFileList) 196 | 197 | 198 | def generate_default_params(): 199 | params = { 200 | 'sample_length': 288, 201 | 'selectedNum': 10000, 202 | 'chuckNum': 10, 203 | 'splitRatio': {'train': 0.7, 'val': 0.2, 'test': 0.1} 204 | } 205 | return params 206 | 207 | 208 | def getData(opts, x_day_dir): 209 | '''this is made to read one day data''' 210 | params = generate_default_params() 211 | 212 | devList = os.listdir(x_day_dir) 213 | label2Data = defaultdict() 214 | allData, allLabel = [], [] 215 | for i in range(len(devList)): 216 | strLabel = devList[i] 217 | fpTuple = getfpTuple(strLabel, x_day_dir) 218 | label2Data[i] = fpTuple 219 | 220 | oneData, oneLabel = getOneDevData(fpTuple, i, params) 221 | allData.extend(oneData) 222 | allLabel.extend(oneLabel) 223 | 224 | splitRatio = params['splitRatio'] 225 | trainData, trainLabels, valData, valLabels, testData, testLabels = splitData(opts, splitRatio, allData, allLabel) 226 | return trainData, trainLabels, valData, valLabels, testData, testLabels 227 | 228 | 229 | def test_read_one_data(opts): 230 | allDataSize = 1000 231 | splitRatio = {'train': 0.7, 'val': 0.2, 'test': 0.1} 232 | train_ind, val_ind, test_ind = getSplitIndex(allDataSize, splitRatio) 233 | 234 | # Load a dataset 235 | x_day_dir = opts.input 236 | trainData, trainLabels, valData, valLabels, testData, testLabels = getData(opts, x_day_dir) 237 | print(trainData.shape, valData.shape, testData.shape) 238 | print(trainLabels.shape) 239 | 240 | 241 | if __name__ == "__main__": 242 | opts = config.parse_args(sys.argv) 243 | test_read_one_data(opts) 244 | print('all test passed!') 245 | -------------------------------------------------------------------------------- /rf/readSigmf2.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin python3.6 2 | 3 | import os 4 | import sys 5 | import pdb 6 | import argparse 7 | import random 8 | import re 9 | import math 10 | 11 | from collections import defaultdict 12 | import json 13 | import numpy as np 14 | import itertools 15 | from sigmf import SigMFFile, sigmffile 16 | 17 | import mytools.tools as mytools 18 | import config 19 | 20 | 21 | def createSignal(metafile, binfile): 22 | # Load a dataset 23 | with open(metafile, 'r') as f: 24 | metadata = json.loads(f.read()) 25 | signal = SigMFFile(metadata=metadata['_metadata'], data_file=binfile) 26 | return signal 27 | 28 | 29 | def get_one_sample(raw_data, start, length): 30 | if not isinstance(raw_data, np.ndarray): 31 | raw_data = np.array(raw_data) 32 | return raw_data[start: start+length] 33 | 34 | 35 | def convert2IQdata(one_raw_data): 36 | flag = False 37 | rtnList = [] 38 | for item in one_raw_data: 39 | realVal = np.real(item) 40 | imagVal = np.imag(item) 41 | if math.isnan(realVal): 42 | realVal = 0 43 | flag = True 44 | if math.isnan(imagVal): 45 | imagVal = 0 46 | flag = True 47 | tmp = [realVal, imagVal] 48 | rtnList.append(tmp) 49 | 50 | #if flag: 51 | # pdb.set_trace() 52 | #tmpMat = np.array(rtnList).T 53 | #rtnList = list(tmpMat) 54 | return rtnList 55 | 56 | 57 | def divideIntoChucks(raw_data, chuckNum): 58 | dataLen = len(raw_data) 59 | sliceLen = dataLen // chuckNum 60 | 61 | chuckList = [] 62 | start = 0 63 | for i in range(chuckNum): 64 | end = start + sliceLen 65 | oneSlice = raw_data[start: end] 66 | start = end 67 | chuckList.append(oneSlice) 68 | 69 | return chuckList 70 | 71 | 72 | def formInpData(raw_data, sample_length, selectedNum, params): 73 | dataOpt = params['dataOpt'] 74 | if 2 == dataOpt: 75 | selectedIndex = params['selectedIndex'] 76 | start_range = len(raw_data) - sample_length 77 | raw_samples = [] 78 | for i in range(start_range): 79 | tmp_sample = get_one_sample(raw_data, i, sample_length) 80 | pdb.set_trace() 81 | raw_samples.append(tmp_sample) 82 | 83 | if 1 == dataOpt: 84 | selectedSamples = random.sample(raw_samples, selectedNum) 85 | elif 2 == dataOpt: 86 | raw_samples = np.array(raw_samples) 87 | selectedSamples = raw_samples[selectedIndex] 88 | selectedSamples = list(selectedSamples) 89 | elif 3 == dataOpt: 90 | selectedSamples = raw_samples[:selectedNum] 91 | else: 92 | raise 93 | 94 | rtn_samples = [] 95 | for tmp_sample in selectedSamples: 96 | tmp_sample = convert2IQdata(tmp_sample) 97 | rtn_samples.append(tmp_sample) 98 | 99 | return rtn_samples 100 | 101 | 102 | def generateIndex(allDataSize): 103 | np.random.seed(42) 104 | shuffledind = np.random.permutation(allDataSize) 105 | 106 | return shuffledind 107 | 108 | 109 | def getSplitIndex(allDataSize, splitRatio): 110 | shuffledind = generateIndex(allDataSize) 111 | 112 | train_set_size = int(allDataSize * splitRatio['train']) 113 | val_set_size = int(allDataSize * splitRatio['val']) 114 | test_set_size = int(allDataSize * splitRatio['test']) 115 | 116 | start, end = 0, train_set_size 117 | train_ind = shuffledind[start: end] 118 | 119 | start, end = train_set_size, train_set_size + val_set_size 120 | val_ind = shuffledind[start: end] 121 | 122 | start, end = train_set_size + val_set_size, train_set_size + val_set_size + test_set_size 123 | test_ind = shuffledind[start: end] 124 | 125 | return train_ind, val_ind, test_ind 126 | 127 | 128 | def splitData(opts, splitRatio, allData, allLabel): 129 | if not isinstance(allData, np.ndarray): 130 | allData = np.array(allData) 131 | allLabel = np.array(allLabel) 132 | 133 | if opts.splitType == 'random': 134 | allDataSize = len(allLabel) 135 | train_ind, val_ind, test_ind = getSplitIndex(allDataSize, splitRatio) 136 | trainData, trainLabels = allData[train_ind, :, :], allLabel[train_ind] 137 | valData, valLabels = allData[val_ind, :, :], allLabel[val_ind] 138 | testData, testLabels = allData[test_ind, :, :], allLabel[test_ind] 139 | elif opts.splitType == 'order': 140 | pass 141 | else: 142 | raise 143 | 144 | return trainData, trainLabels, valData, valLabels, testData, testLabels 145 | 146 | 147 | def get_signal_samples(signal, label, params): 148 | # Get some metadata and all annotations 149 | chuckNum = params['chuckNum'] 150 | sample_length = params['sample_length'] 151 | selectedNum = params['selectedNum'] 152 | 153 | raw_data = signal.read_samples(0, -1) 154 | chuckList = divideIntoChucks(raw_data, chuckNum) 155 | chuckList = chuckList[0] # only take the first chuck 156 | 157 | if params['dataOpt'] == 2: 158 | totalNum = len(chuckList) 159 | allRanIndex = generateIndex(totalNum) 160 | selectedIndex = allRanIndex[:params['selectedNum']] 161 | params['selectedIndex'] = selectedIndex 162 | 163 | oneData = formInpData(chuckList, sample_length, selectedNum, params) 164 | oneLabel = np.ones(len(oneData), dtype=np.int) * label 165 | print('raw data length is: ', len(oneData)) 166 | return oneData, oneLabel 167 | 168 | 169 | def searchFp(fname, metaFileList): 170 | for mfp in metaFileList: 171 | m = re.search(fname, mfp) 172 | if m: 173 | return mfp 174 | return '' 175 | 176 | 177 | def getSignalList(fpTuple): 178 | binFileList, metaFileList = fpTuple 179 | signalList = [] 180 | for bfp in binFileList: 181 | fname = os.path.basename(bfp).split('.')[0] 182 | mfp = searchFp(fname, metaFileList) 183 | if not mfp: 184 | raise ValueError('binfile {} does not have a match'.format(bfp)) 185 | signal = createSignal(mfp, bfp) 186 | signalList.append(signal) 187 | return signalList 188 | 189 | 190 | def getOneDevData(fpTuple, label, params): 191 | print('processing file: ', fpTuple) 192 | signalList = getSignalList(fpTuple) 193 | allData, allLabel = [], [] 194 | for signal in signalList: 195 | oneData, oneLabel = get_signal_samples(signal, label, params) 196 | allData.extend(oneData) 197 | allLabel.extend(oneLabel) 198 | 199 | return allData, allLabel 200 | 201 | 202 | def getfpTuple(strLabel, x_day_dir): 203 | dayDevDir = os.path.join(x_day_dir, strLabel) 204 | fList = os.listdir(dayDevDir) 205 | binFileList, metaFileList = [], [] 206 | for fname in fList: 207 | fp = os.path.join(dayDevDir, fname) 208 | if fp.endswith('bin'): 209 | binFileList.append(fp) 210 | elif fp.endswith('sigmf-meta'): 211 | metaFileList.append(fp) 212 | else: 213 | raise 214 | return (binFileList, metaFileList) 215 | 216 | 217 | def generate_default_params(): 218 | params = { 219 | 'sample_length': 288, 220 | 'selectedNum': 100000, 221 | 'chuckNum': 10, 222 | 'splitRatio': {'train': 0.7, 'val': 0.2, 'test': 0.1} 223 | } 224 | return params 225 | 226 | 227 | def getData(opts, x_day_dir): 228 | '''this is made to read one day data''' 229 | params = generate_default_params() 230 | 231 | # dataOpt 232 | # 1: take diff random number across 5 devices 233 | # 2: take same random number across 5 devices 234 | # 3: take consecutive 100k slices 235 | params['dataOpt'] = 1 236 | 237 | devList = os.listdir(x_day_dir) 238 | label2Data = defaultdict() 239 | allData, allLabel = [], [] 240 | for i in range(len(devList)): 241 | strLabel = devList[i] 242 | fpTuple = getfpTuple(strLabel, x_day_dir) 243 | label2Data[i] = fpTuple 244 | 245 | oneData, oneLabel = getOneDevData(fpTuple, i, params) 246 | allData.extend(oneData) 247 | allLabel.extend(oneLabel) 248 | 249 | splitRatio = params['splitRatio'] 250 | trainData, trainLabels, valData, valLabels, testData, testLabels = splitData(opts, splitRatio, allData, allLabel) 251 | return trainData, trainLabels, valData, valLabels, testData, testLabels 252 | 253 | 254 | def test_read_one_data(opts): 255 | allDataSize = 1000 256 | splitRatio = {'train': 0.7, 'val': 0.2, 'test': 0.1} 257 | train_ind, val_ind, test_ind = getSplitIndex(allDataSize, splitRatio) 258 | 259 | # Load a dataset 260 | x_day_dir = opts.input 261 | trainData, trainLabels, valData, valLabels, testData, testLabels = getData(opts, x_day_dir) 262 | print(trainData.shape, valData.shape, testData.shape) 263 | print(trainLabels.shape) 264 | 265 | 266 | if __name__ == "__main__": 267 | opts = config.parse_args(sys.argv) 268 | test_read_one_data(opts) 269 | print('all test passed!') 270 | -------------------------------------------------------------------------------- /rf/resnet50_1D.py: -------------------------------------------------------------------------------- 1 | """ResNet50 model for Keras. 2 | # Reference: 3 | - [Deep Residual Learning for Image Recognition]( 4 | https://arxiv.org/abs/1512.03385) (CVPR 2016 Best Paper Award) 5 | Adapted from code contributed by BigMoyan. 6 | """ 7 | 8 | import os 9 | import sys 10 | import argparse 11 | import pdb 12 | import warnings 13 | 14 | from keras import layers 15 | from keras import models 16 | import keras.utils as keras_utils 17 | import keras.backend as backend 18 | 19 | import imagenet_utils 20 | from imagenet_utils import get_submodules_from_kwargs 21 | from imagenet_utils import decode_predictions 22 | from imagenet_utils import _obtain_input_shape 23 | 24 | preprocess_input = imagenet_utils.preprocess_input 25 | 26 | 27 | def identity_block(input_tensor, kernel_size, filters, stage, block): 28 | """The identity block is the block that has no conv layer at shortcut. 29 | # Arguments 30 | input_tensor: input tensor 31 | kernel_size: default 3, the kernel size of 32 | middle conv layer at main path 33 | filters: list of integers, the filters of 3 conv layer at main path 34 | stage: integer, current stage label, used for generating layer names 35 | block: 'a','b'..., current block label, used for generating layer names 36 | # Returns 37 | Output tensor for the block. 38 | """ 39 | filters1, filters2, filters3 = filters 40 | if backend.image_data_format() == 'channels_last': 41 | bn_axis = 2 42 | else: 43 | bn_axis = 1 44 | conv_name_base = 'res' + str(stage) + block + '_branch' 45 | bn_name_base = 'bn' + str(stage) + block + '_branch' 46 | 47 | x = layers.Conv1D(filters1, 1, 48 | kernel_initializer='he_normal', 49 | name=conv_name_base + '2a')(input_tensor) 50 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2a')(x) 51 | x = layers.Activation('relu')(x) 52 | 53 | x = layers.Conv1D(filters2, kernel_size, 54 | padding='same', 55 | kernel_initializer='he_normal', 56 | name=conv_name_base + '2b')(x) 57 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2b')(x) 58 | x = layers.Activation('relu')(x) 59 | 60 | x = layers.Conv1D(filters3, 1, 61 | kernel_initializer='he_normal', 62 | name=conv_name_base + '2c')(x) 63 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2c')(x) 64 | 65 | x = layers.add([x, input_tensor]) 66 | x = layers.Activation('relu')(x) 67 | return x 68 | 69 | 70 | def conv_block(input_tensor, kernel_size, filters, stage, block, strides=2): 71 | """A block that has a conv layer at shortcut. 72 | # Arguments 73 | input_tensor: input tensor 74 | kernel_size: default 3, the kernel size of 75 | middle conv layer at main path 76 | filters: list of integers, the filters of 3 conv layer at main path 77 | stage: integer, current stage label, used for generating layer names 78 | block: 'a','b'..., current block label, used for generating layer names 79 | strides: Strides for the first conv layer in the block. 80 | # Returns 81 | Output tensor for the block. 82 | Note that from stage 3, 83 | the first conv layer at main path is with strides=(2, 2) 84 | And the shortcut should have strides=(2, 2) as well 85 | """ 86 | filters1, filters2, filters3 = filters 87 | if backend.image_data_format() == 'channels_last': 88 | bn_axis = 2 89 | else: 90 | bn_axis = 1 91 | conv_name_base = 'res' + str(stage) + block + '_branch' 92 | bn_name_base = 'bn' + str(stage) + block + '_branch' 93 | 94 | x = layers.Conv1D(filters1, 1, strides=strides, 95 | kernel_initializer='he_normal', 96 | name=conv_name_base + '2a')(input_tensor) 97 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2a')(x) 98 | x = layers.Activation('relu')(x) 99 | 100 | x = layers.Conv1D(filters2, kernel_size, padding='same', 101 | kernel_initializer='he_normal', 102 | name=conv_name_base + '2b')(x) 103 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2b')(x) 104 | x = layers.Activation('relu')(x) 105 | 106 | x = layers.Conv1D(filters3, 1, 107 | kernel_initializer='he_normal', 108 | name=conv_name_base + '2c')(x) 109 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2c')(x) 110 | 111 | shortcut = layers.Conv1D(filters3, 1, strides=strides, 112 | kernel_initializer='he_normal', 113 | name=conv_name_base + '1')(input_tensor) 114 | 115 | shortcut = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '1')(shortcut) 116 | 117 | x = layers.add([x, shortcut]) 118 | x = layers.Activation('relu')(x) 119 | return x 120 | 121 | 122 | def create_model(input_shape, classes, pooling=None, include_top=True, **kwargs): 123 | """Instantiates the ResNet50 architecture. 124 | Optionally loads weights pre-trained on ImageNet. 125 | Note that the data format convention used by the model is 126 | the one specified in your Keras config at `~/.keras/keras.json`. 127 | # Arguments 128 | include_top: whether to include the fully-connected 129 | layer at the top of the network. 130 | weights: one of `None` (random initialization), 131 | 'imagenet' (pre-training on ImageNet), 132 | or the path to the weights file to be loaded. 133 | input_tensor: optional Keras tensor (i.e. output of `layers.Input()`) 134 | to use as image input for the model. 135 | input_shape: optional shape tuple, only to be specified 136 | if `include_top` is False (otherwise the input shape 137 | has to be `(224, 224, 3)` (with `channels_last` data format) 138 | or `(3, 224, 224)` (with `channels_first` data format). 139 | It should have exactly 3 inputs channels, 140 | and width and height should be no smaller than 32. 141 | E.g. `(200, 200, 3)` would be one valid value. 142 | pooling: Optional pooling mode for feature extraction 143 | when `include_top` is `False`. 144 | - `None` means that the output of the model will be 145 | the 4D tensor output of the 146 | last convolutional block. 147 | - `avg` means that global average pooling 148 | will be applied to the output of the 149 | last convolutional block, and thus 150 | the output of the model will be a 1D tensor. 151 | - `max` means that global max pooling will 152 | be applied. 153 | classes: optional number of classes to classify images 154 | into, only to be specified if `include_top` is True, and 155 | if no `weights` argument is specified. 156 | # Returns 157 | A Keras model instance. 158 | # Raises 159 | ValueError: in case of invalid argument for `weights`, 160 | or invalid input shape. 161 | """ 162 | if backend.image_data_format() == 'channels_last': 163 | bn_axis = 2 164 | else: 165 | bn_axis = 1 166 | 167 | img_input = layers.Input(shape=input_shape) 168 | 169 | x = layers.ZeroPadding1D(padding=3, name='conv1_pad')(img_input) 170 | x = layers.Conv1D(64, 7, 171 | strides=2, 172 | padding='valid', 173 | kernel_initializer='he_normal', 174 | name='conv1')(x) 175 | x = layers.BatchNormalization(axis=bn_axis, name='bn_conv1')(x) 176 | x = layers.Activation('relu')(x) 177 | x = layers.ZeroPadding1D(padding=1, name='pool1_pad')(x) 178 | x = layers.MaxPooling1D(3, strides=2)(x) 179 | 180 | x = conv_block(x, 3, [64, 64, 256], stage=2, block='a', strides=1) 181 | x = identity_block(x, 3, [64, 64, 256], stage=2, block='b') 182 | x = identity_block(x, 3, [64, 64, 256], stage=2, block='c') 183 | 184 | x = conv_block(x, 3, [128, 128, 512], stage=3, block='a') 185 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='b') 186 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='c') 187 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='d') 188 | 189 | x = conv_block(x, 3, [256, 256, 1024], stage=4, block='a') 190 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='b') 191 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='c') 192 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='d') 193 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='e') 194 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='f') 195 | 196 | x = conv_block(x, 3, [512, 512, 2048], stage=5, block='a') 197 | x = identity_block(x, 3, [512, 512, 2048], stage=5, block='b') 198 | x = identity_block(x, 3, [512, 512, 2048], stage=5, block='c') 199 | 200 | if include_top: 201 | x = layers.GlobalAveragePooling1D(name='avg_pool')(x) 202 | x = layers.Dense(classes, activation='softmax', name='fc1000')(x) 203 | else: 204 | if pooling == 'avg': 205 | x = layers.GlobalAveragePooling1D()(x) 206 | elif pooling == 'max': 207 | x = layers.GlobalMaxPooling1D()(x) 208 | else: 209 | warnings.warn('The output shape of `ResNet50(include_top=False)` ' 210 | 'has been changed since Keras 2.2.0.') 211 | 212 | # Create model. 213 | inputs = img_input 214 | model = models.Model(inputs, x, name='resnet50') 215 | 216 | return model 217 | -------------------------------------------------------------------------------- /rf/resnet50_2D.py: -------------------------------------------------------------------------------- 1 | """ResNet50 model for Keras. 2 | # Reference: 3 | - [Deep Residual Learning for Image Recognition]( 4 | https://arxiv.org/abs/1512.03385) (CVPR 2016 Best Paper Award) 5 | Adapted from code contributed by BigMoyan. 6 | """ 7 | import os 8 | import pdb 9 | import warnings 10 | 11 | from keras import layers 12 | from keras import models 13 | import keras.utils as keras_utils 14 | import keras.backend as backend 15 | 16 | from imagenet_utils import get_submodules_from_kwargs 17 | import imagenet_utils 18 | from imagenet_utils import decode_predictions 19 | from imagenet_utils import _obtain_input_shape 20 | 21 | preprocess_input = imagenet_utils.preprocess_input 22 | 23 | WEIGHTS_PATH = ('https://github.com/fchollet/deep-learning-models/' 24 | 'releases/download/v0.2/' 25 | 'resnet50_weights_tf_dim_ordering_tf_kernels.h5') 26 | WEIGHTS_PATH_NO_TOP = ('https://github.com/fchollet/deep-learning-models/' 27 | 'releases/download/v0.2/' 28 | 'resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5') 29 | 30 | 31 | def identity_block(input_tensor, kernel_size, filters, stage, block): 32 | """The identity block is the block that has no conv layer at shortcut. 33 | # Arguments 34 | input_tensor: input tensor 35 | kernel_size: default 3, the kernel size of 36 | middle conv layer at main path 37 | filters: list of integers, the filters of 3 conv layer at main path 38 | stage: integer, current stage label, used for generating layer names 39 | block: 'a','b'..., current block label, used for generating layer names 40 | # Returns 41 | Output tensor for the block. 42 | """ 43 | filters1, filters2, filters3 = filters 44 | if backend.image_data_format() == 'channels_last': 45 | bn_axis = 3 46 | else: 47 | bn_axis = 1 48 | conv_name_base = 'res' + str(stage) + block + '_branch' 49 | bn_name_base = 'bn' + str(stage) + block + '_branch' 50 | 51 | x = layers.Conv2D(filters1, (1, 1), 52 | kernel_initializer='he_normal', 53 | name=conv_name_base + '2a')(input_tensor) 54 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2a')(x) 55 | x = layers.Activation('relu')(x) 56 | 57 | x = layers.Conv2D(filters2, kernel_size, 58 | padding='same', 59 | kernel_initializer='he_normal', 60 | name=conv_name_base + '2b')(x) 61 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2b')(x) 62 | x = layers.Activation('relu')(x) 63 | 64 | x = layers.Conv2D(filters3, (1, 1), 65 | kernel_initializer='he_normal', 66 | name=conv_name_base + '2c')(x) 67 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2c')(x) 68 | 69 | x = layers.add([x, input_tensor]) 70 | x = layers.Activation('relu')(x) 71 | return x 72 | 73 | 74 | def conv_block(input_tensor, 75 | kernel_size, 76 | filters, 77 | stage, 78 | block, 79 | strides=(2, 2)): 80 | """A block that has a conv layer at shortcut. 81 | # Arguments 82 | input_tensor: input tensor 83 | kernel_size: default 3, the kernel size of 84 | middle conv layer at main path 85 | filters: list of integers, the filters of 3 conv layer at main path 86 | stage: integer, current stage label, used for generating layer names 87 | block: 'a','b'..., current block label, used for generating layer names 88 | strides: Strides for the first conv layer in the block. 89 | # Returns 90 | Output tensor for the block. 91 | Note that from stage 3, 92 | the first conv layer at main path is with strides=(2, 2) 93 | And the shortcut should have strides=(2, 2) as well 94 | """ 95 | filters1, filters2, filters3 = filters 96 | if backend.image_data_format() == 'channels_last': 97 | bn_axis = 3 98 | else: 99 | bn_axis = 1 100 | conv_name_base = 'res' + str(stage) + block + '_branch' 101 | bn_name_base = 'bn' + str(stage) + block + '_branch' 102 | 103 | x = layers.Conv2D(filters1, (1, 1), strides=strides, 104 | kernel_initializer='he_normal', 105 | name=conv_name_base + '2a')(input_tensor) 106 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2a')(x) 107 | x = layers.Activation('relu')(x) 108 | 109 | x = layers.Conv2D(filters2, kernel_size, padding='same', 110 | kernel_initializer='he_normal', 111 | name=conv_name_base + '2b')(x) 112 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2b')(x) 113 | x = layers.Activation('relu')(x) 114 | 115 | x = layers.Conv2D(filters3, (1, 1), 116 | kernel_initializer='he_normal', 117 | name=conv_name_base + '2c')(x) 118 | x = layers.BatchNormalization(axis=bn_axis, name=bn_name_base + '2c')(x) 119 | 120 | shortcut = layers.Conv2D(filters3, (1, 1), strides=strides, 121 | kernel_initializer='he_normal', 122 | name=conv_name_base + '1')(input_tensor) 123 | shortcut = layers.BatchNormalization( 124 | axis=bn_axis, name=bn_name_base + '1')(shortcut) 125 | 126 | x = layers.add([x, shortcut]) 127 | x = layers.Activation('relu')(x) 128 | return x 129 | 130 | 131 | def create_model(input_shape, classes, pooling=None, include_top=True, **kwargs): 132 | """Instantiates the ResNet50 architecture. 133 | Optionally loads weights pre-trained on ImageNet. 134 | Note that the data format convention used by the model is 135 | the one specified in your Keras config at `~/.keras/keras.json`. 136 | # Arguments 137 | include_top: whether to include the fully-connected 138 | layer at the top of the network. 139 | weights: one of `None` (random initialization), 140 | 'imagenet' (pre-training on ImageNet), 141 | or the path to the weights file to be loaded. 142 | input_tensor: optional Keras tensor (i.e. output of `layers.Input()`) 143 | to use as image input for the model. 144 | input_shape: optional shape tuple, only to be specified 145 | if `include_top` is False (otherwise the input shape 146 | has to be `(224, 224, 3)` (with `channels_last` data format) 147 | or `(3, 224, 224)` (with `channels_first` data format). 148 | It should have exactly 3 inputs channels, 149 | and width and height should be no smaller than 32. 150 | E.g. `(200, 200, 3)` would be one valid value. 151 | pooling: Optional pooling mode for feature extraction 152 | when `include_top` is `False`. 153 | - `None` means that the output of the model will be 154 | the 4D tensor output of the 155 | last convolutional block. 156 | - `avg` means that global average pooling 157 | will be applied to the output of the 158 | last convolutional block, and thus 159 | the output of the model will be a 2D tensor. 160 | - `max` means that global max pooling will 161 | be applied. 162 | classes: optional number of classes to classify images 163 | into, only to be specified if `include_top` is True, and 164 | if no `weights` argument is specified. 165 | # Returns 166 | A Keras model instance. 167 | # Raises 168 | ValueError: in case of invalid argument for `weights`, 169 | or invalid input shape. 170 | """ 171 | 172 | # Determine proper input shape 173 | if not input_shape: 174 | input_shape = _obtain_input_shape(input_shape, 175 | default_size=224, 176 | min_size=32, 177 | data_format=backend.image_data_format(), 178 | require_flatten=include_top, 179 | weights=weights) 180 | 181 | img_input = layers.Input(shape=input_shape) 182 | 183 | if backend.image_data_format() == 'channels_last': 184 | bn_axis = 3 185 | else: 186 | bn_axis = 1 187 | 188 | x = layers.ZeroPadding2D(padding=(3, 3), name='conv1_pad')(img_input) 189 | x = layers.Conv2D(64, (7, 7), 190 | strides=(2, 2), 191 | padding='valid', 192 | kernel_initializer='he_normal', 193 | name='conv1')(x) 194 | x = layers.BatchNormalization(axis=bn_axis, name='bn_conv1')(x) 195 | x = layers.Activation('relu')(x) 196 | x = layers.ZeroPadding2D(padding=(1, 1), name='pool1_pad')(x) 197 | x = layers.MaxPooling2D((3, 3), strides=(2, 2))(x) 198 | 199 | x = conv_block(x, 3, [64, 64, 256], stage=2, block='a', strides=(1, 1)) 200 | x = identity_block(x, 3, [64, 64, 256], stage=2, block='b') 201 | x = identity_block(x, 3, [64, 64, 256], stage=2, block='c') 202 | 203 | x = conv_block(x, 3, [128, 128, 512], stage=3, block='a') 204 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='b') 205 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='c') 206 | x = identity_block(x, 3, [128, 128, 512], stage=3, block='d') 207 | 208 | x = conv_block(x, 3, [256, 256, 1024], stage=4, block='a') 209 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='b') 210 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='c') 211 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='d') 212 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='e') 213 | x = identity_block(x, 3, [256, 256, 1024], stage=4, block='f') 214 | 215 | x = conv_block(x, 3, [512, 512, 2048], stage=5, block='a') 216 | x = identity_block(x, 3, [512, 512, 2048], stage=5, block='b') 217 | x = identity_block(x, 3, [512, 512, 2048], stage=5, block='c') 218 | 219 | if include_top: 220 | x = layers.GlobalAveragePooling2D(name='avg_pool')(x) 221 | x = layers.Dense(classes, activation='softmax', name='fc1000')(x) 222 | else: 223 | if pooling == 'avg': 224 | x = layers.GlobalAveragePooling2D()(x) 225 | elif pooling == 'max': 226 | x = layers.GlobalMaxPooling2D()(x) 227 | else: 228 | warnings.warn('The output shape of `ResNet50(include_top=False)` ' 229 | 'has been changed since Keras 2.2.0.') 230 | 231 | # Ensure that the model takes into account 232 | # any potential predecessors of `input_tensor`. 233 | inputs = img_input 234 | 235 | # Create model. 236 | model = models.Model(inputs, x, name='resnet50') 237 | 238 | return model 239 | -------------------------------------------------------------------------------- /rf/test_model.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | ''' 3 | Nov 6th: updated for D2 option, which can easily change between 1D and 2D with out change the code 4 | ''' 5 | import os 6 | import sys 7 | import argparse 8 | import pdb 9 | 10 | from keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, TensorBoard 11 | from keras.utils import np_utils 12 | import numpy as np 13 | from sklearn.model_selection import train_test_split 14 | 15 | import radioConv 16 | import config 17 | 18 | ROOT_DIR = os.getenv('ROOT_DIR') 19 | resDir = os.path.join(ROOT_DIR, 'resDir') 20 | modelDir = os.path.join(resDir, 'modelDir') 21 | os.makedirs(modelDir, exist_ok=True) 22 | 23 | 24 | def loadData(dpath, data_dim=600, expandDim=True): 25 | wholePack = np.load(dpath) 26 | allData, allLabel = wholePack['x'], wholePack['y'] 27 | 28 | # limit data dim 29 | allData = allData[:, :data_dim] 30 | 31 | X_train, X_test, y_train, y_test = train_test_split(allData, allLabel, test_size=0.2, shuffle=True, random_state=47) 32 | NUM_CLASS = len(set(list(y_test))) 33 | y_train = np_utils.to_categorical(y_train, NUM_CLASS) 34 | y_test = np_utils.to_categorical(y_test, NUM_CLASS) 35 | 36 | if expandDim: 37 | X_train = X_train[:, np.newaxis, :] 38 | X_test = X_test[:, np.newaxis, :] 39 | 40 | # delete all no use data 41 | del wholePack 42 | del allData 43 | del allLabel 44 | 45 | return X_train, y_train, X_test, y_test, NUM_CLASS 46 | 47 | 48 | def main(opts): 49 | # setup params 50 | Batch_Size = 128 51 | Epoch_Num = 100 52 | saveModelPath = os.path.join(modelDir, 'best_model_{}.h5'.format(opts.modelType)) 53 | checkpointer = ModelCheckpoint(filepath=saveModelPath, monitor='val_accuracy', verbose=1, save_best_only=True, mode='max') 54 | earlyStopper = EarlyStopping(monitor='val_accuracy', mode='max', patience=10) 55 | callBackList = [checkpointer, earlyStopper] 56 | 57 | print('loading data...') 58 | dpath = opts.input 59 | data_dim = 600 60 | inp_shape = (1, data_dim) 61 | train_x, train_y, test_x, test_y, NUM_CLASS = loadData(dpath, data_dim=data_dim) 62 | 63 | print('get the model and compile it...') 64 | model = radioConv.create_model(opts.modelType, inp_shape, NUM_CLASS, D2=False, channel='first') 65 | model.summary() 66 | model.compile(optimizer='adam', loss='categorical_crossentropy', metrics='accuracy') 67 | 68 | print('fit the model with data...') 69 | model.fit(x=train_x, y=train_y, 70 | batch_size=Batch_Size, 71 | epochs=Epoch_Num, 72 | verbose=opts.verbose, 73 | callbacks=callBackList, 74 | validation_split=0.1, 75 | shuffle=True) 76 | 77 | print('test the trained model...') 78 | score, acc = model.evaluate(test_x, test_y, batch_size=Batch_Size, verbose=0) 79 | print('test acc is: ', acc) 80 | 81 | print('all test done!') 82 | 83 | 84 | if __name__ == "__main__": 85 | opts = config.parse_args(sys.argv) 86 | main(opts) 87 | -------------------------------------------------------------------------------- /rf/utils.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin python3.6 2 | 3 | import os 4 | import sys 5 | import pdb 6 | 7 | import numpy as np 8 | import config 9 | 10 | 11 | def generateIndex(allDataSize): 12 | np.random.seed(42) 13 | shuffledind = np.random.permutation(allDataSize) 14 | return shuffledind 15 | 16 | 17 | def getSplitIndex(allDataSize, splitRatio): 18 | shuffledind = generateIndex(allDataSize) 19 | 20 | train_set_size = int(allDataSize * splitRatio['train']) 21 | val_set_size = int(allDataSize * splitRatio['val']) 22 | # test_set_size = int(allDataSize * splitRatio['test']) 23 | 24 | start, end = 0, train_set_size 25 | train_ind = shuffledind[start: end] 26 | 27 | start, end = train_set_size, train_set_size + val_set_size 28 | val_ind = shuffledind[start: end] 29 | 30 | ''' 31 | start, end = train_set_size + val_set_size, train_set_size + val_set_size + test_set_size 32 | test_ind = shuffledind[start: end] 33 | ''' 34 | 35 | #return train_ind, val_ind, test_ind 36 | return train_ind, val_ind 37 | 38 | 39 | def splitData(splitRatio, allData, allLabel, splitType='random'): 40 | if not isinstance(allData, np.ndarray): 41 | allData = np.array(allData) 42 | allLabel = np.array(allLabel) 43 | 44 | if splitType == 'random': 45 | allDataSize = len(allLabel) 46 | #train_ind, val_ind, test_ind = getSplitIndex(allDataSize, splitRatio) 47 | train_ind, val_ind = getSplitIndex(allDataSize, splitRatio) 48 | 49 | trainData, trainLabels = allData[train_ind, :, :], allLabel[train_ind] 50 | valData, valLabels = allData[val_ind, :, :], allLabel[val_ind] 51 | #testData, testLabels = allData[test_ind, :, :], allLabel[test_ind] 52 | 53 | elif splitType == 'order': 54 | pass 55 | else: 56 | raise 57 | 58 | #return trainData, trainLabels, valData, valLabels, testData, testLabels 59 | return trainData, trainLabels, valData, valLabels 60 | 61 | 62 | if __name__ == "__main__": 63 | opts = config.parse_args(sys.argv) 64 | print('all test passed!') 65 | --------------------------------------------------------------------------------