├── .gitignore ├── README.md ├── combo_no_label.py ├── data_learning.py ├── data_preprocessing.py ├── day_conf.json ├── global_sp_func.py ├── log_parsing.py ├── parse_data_from_log.py ├── test_date_conf.py ├── train_test_conf.py └── wifi_process_combo.py /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__/* 2 | .idea/* 3 | *.dat 4 | *.pyc 5 | *.h5 6 | data/* 7 | model/* 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Project Title 2 | Presence Detection Using CNN 3 | 4 | # Getting Started 5 | 6 | This repository contains datasets and implementation code for the paper: 7 | 8 | "Harvesting Ambient RF for Presence Detection Through Deep Learning." Yang Liu, Tiexing Wang, Yuexin Jiang and Biao Chen. arXiv link: https://arxiv.org/abs/2002.05770 9 | 10 | Please cite this paper if you use the code/data in this repository as part of a published research project. 11 | 12 | These instructions will walk you through how to access the datasets and how to get the project up and running on your local machine. 13 | 14 | # Prerequisites 15 | Python interpreter: python3.6 16 | 17 | Development Environment: Ubuntu 16.04 LTS 18 | 19 | Packages need to be installed: 20 | json, numpy, matplotlib, keras with tensorflow as backend, 21 | 22 | 23 | 24 | 25 | # Configurations 26 | **All the configuration parameters are specified in train_test_conf.py**. In the following, we will refer to parameters as conf.param_name. 27 | 28 | 1. After cloning the repository, please **make two new folders inside this repo directory** by: 29 | ``` 30 | mkdir data # used to store data 31 | mkdir data/training # used to store training data 32 | mkdir data/test # used to store test data 33 | mkdir model # used to store models 34 | ``` 35 | 36 | 2. Assign the absolute path to where data are stored to conf.log_folder, for example: 37 | ``` 38 | log_folder = '/root/share/upload_wifi_data/' 39 | ``` 40 | 41 | # Datasets 42 | All 24 days' data are available in the following link, please download and unzip them: 43 | 44 | https://drive.google.com/drive/folders/1yGchSK7DeoR--2ZkNItTEYNV3tc9RUyn?usp=share_link 45 | 46 | **Note**: data on day 17-19 is not provided since on these three days, we conducted real-time comparsion with a PIR sensor and CSIs were thus not recorded. 47 | 48 | ## Composition 49 | day index | location | types of run | 50 | --- | --- | --- | 51 | 1-3 | LabI | 'empty', 'motion', 'mixed' | 52 | 4-19 | LabII | 'empty', 'motion', 'mixed' | 53 | 20-24 | Apartment | 'empty', 'living_room', 'kitchen', 'bedroomI', 'bedroomII'| 54 | 55 | 56 | Ground truth of the dataset is stored in day_conf.json. For example, ground truth of 57 | the data collected on day 1 and day 24 are: 58 | 59 | ``` 60 | "day1": {"location": "Lab1", 61 | "motion": 6, 62 | "empty": 6, 63 | "mixed": 2, 64 | "mixed_truth": [[0, 1, 1, 0, 1], [1, 0, 0, 0, 1]]}, 65 | 66 | Explanation: 67 | On day1 the experiment was conducted in Lab1, number of runs collected for 68 | label 0 (empty) and label 1 (motion) is 6 and 6 respecively. Furthermore, two mixture 69 | runs are collected on the same day with truth labeling provided in 'mixed_truth'. 70 | 71 | mixed data runs: a continuous run where two states alternates. In other runs, 72 | only one state is involved, that is CSI images are either labeled as 0 or 1. 73 | 74 | ``` 75 | **Note: mixture runs are for evaluation purpose only, don't include them for training or test set**. 76 | 77 | ``` 78 | "day24": {"location": "Apartment", 79 | "mixed": 0, 80 | "mixed_truth": [], 81 | "empty": 1, 82 | "living_room": 2, 83 | "kitchen": 2, 84 | "bedroomI": 2, 85 | "bedroomII": 2} 86 | 87 | Explanation: 88 | On day 24, the experiment was conducted in Apartment, and there is one run for human free 89 | environment. Human motions are categorized according to locations. Number of runs collected for motions 90 | in the living room, kitchen, bedroomI and bedroomII are 2, 2, 2 and 2 respectively. 91 | There are no mixture runs collected on day 24 92 | 93 | ``` 94 | 95 | 96 | # How to train 97 | date used for training is specified in train_test_conf.py as 98 | ``` 99 | training_date = ['day9', 'day10', 'day11', 'day12', 'day13', 'day14'] 100 | ``` 101 | validation set used in training: 102 | ``` 103 | training_validate_date = ['day15', 'day16'] 104 | ``` 105 | specify label mapping for training and validation data 106 | ``` 107 | train_label = {'empty': 0, 'motion': 1} 108 | 109 | Explanation: 110 | key: types of runs (refer to Table under composition) 111 | value: classification class (either 0 or 1) the data belong to; 112 | ``` 113 | **Note**: only empty runs map to 0, while all the other types of runs should map to 1 114 | 115 | Run the following files sequentially: 116 | ``` 117 | 1. ./parse_data_from_log.py -m Y 118 | generate CSI images from the CSI log files 119 | 120 | 2. ./data_preprocessing.py -m Y 121 | apply pre-processing steps to raw CSI images, prepare input data for CNN 122 | 123 | 3. ./data_learing.py -m Y 124 | obtain a CNN model and save it as conf.model_name inside folder './model/'. 125 | ``` 126 | **Note**: To use the code in training mode, set the value of input argument m to be 'Y'; To use the code in test mode as given below, set its value to be 'N'. 127 | All the intermediate data files are saved under './data/training/' 128 | 129 | # How to test 130 | date used for testing the trained model is specified in train_test_conf as 131 | ``` 132 | test_date = ['day14'] 133 | ``` 134 | specify label mapping for test data. By default, test_label include all types of runs available on specified test days; 135 | ``` 136 | test_label = {'empty': 0, 'motion': 1} # for LabI or LabII only 137 | or 138 | test_label = {'empty': 0, 'living_room': 1, 'kitchen': 2, 'bedroomI': 3, 'bedroomII': 4} # for Apartment only 139 | or 140 | test_label = {'empty': 0, 'motion': 1, living_room': 2, 'kitchen': 3, 'bedroomI': 4, 'bedroomII': 5} # for labs and Apartment 141 | 142 | ``` 143 | **Note**: Even though this is a binary classification problem, different from train_label, class labeling for test days can go beyond 1 when including data from Apartment. This is 144 | for the users who want to know the detection accuracy at different locations. 145 | 146 | There are three test methods: 147 | 148 | 1. method I:
149 | if the user want to save intermediate data files, please execute the above 150 | three steps for training but each with input argument ```'-m N'```. 151 | All the intermediate data files are saved under './data/test/' 152 | 153 | 154 | 2. method II:
155 | if the user just needs the detection results without saving intermediate data files, please run 156 | ``` 157 | ./wifi_process_combo.py -m N 158 | ``` 159 | 160 | 3. method III:
161 | To evaluate the model using mixture runs, please first specify draw date in config file and the label you want to display: 162 | 163 | ``` 164 | draw_date = ['day1', 'day14'] 165 | draw_label = 'mixed' 166 | ``` 167 | then run 168 | ``` 169 | ./combo_no_label.py 170 | ``` 171 | 172 | Note: if the user wants to visualize detection results from other types of run, just change conf.draw_label to other values such as 'motion', 'empty' or location names in the apartment 173 | 174 | 175 | 176 | -------------------------------------------------------------------------------- /combo_no_label.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import numpy as np 4 | import argparse 5 | from parse_data_from_log import DataLogParser 6 | from data_preprocessing import DataPreprocess 7 | from data_learning import NeuralNetworkModel 8 | import train_test_conf as conf 9 | import matplotlib.pyplot as plt 10 | 11 | def main(): 12 | ################################################## 13 | # parse data from original data & construct images 14 | ################################################## 15 | print("parsing data from log files which are generated by Atheros-CSI-TOOL\n") 16 | data_generator = DataLogParser(conf.n_timestamps, conf.D, conf.step_size, 17 | conf.ntx_max, conf.nrx_max, conf.nsubcarrier_max, 18 | conf.data_folder, conf.log_folder, 19 | conf.skip_frames, 20 | conf.time_offset_ratio, 21 | conf.day_conf, 22 | conf.train_label) 23 | data_generator.generate_image_no_label(conf.draw_date, conf.draw_label) 24 | # train_data, test_data: classes (key: label, value: images under this label) 25 | test_data = data_generator.get_data_no_label() 26 | if len(test_data) == 0: 27 | print('find no data to draw under date {} and label {}!!!'.format(conf.draw_date, conf.draw_label)) 28 | return 29 | ################################################## 30 | # apply signal processing blocks to images 31 | ################################################## 32 | print("Pre-processing data\n") 33 | data_process = DataPreprocess(conf.n_timestamps, conf.D, conf.step_size, 34 | conf.ntx_max, conf.ntx, conf.nrx_max, 35 | conf.nrx, conf.nsubcarrier_max, conf.nsubcarrier, 36 | conf.data_shape_to_nn, 37 | conf.data_folder,conf.train_label) 38 | data_process.add_image_no_label(test_data) 39 | data_process.signal_processing(conf.do_fft, conf.fft_shape) 40 | data_process.prepare_shape() 41 | final_test_data = data_process.get_data_no_label() 42 | 43 | ################################################## 44 | # train or test data with neural netowrk 45 | ################################################## 46 | 47 | nn_model = NeuralNetworkModel(conf.data_shape_to_nn, conf.abs_shape_to_nn, 48 | conf.phase_shape_to_nn, conf.total_classes) 49 | print("Get test result using existing model (in test mode)\n") 50 | nn_model.load_model(conf.model_name) 51 | for key in final_test_data: 52 | plt.figure() 53 | total_test = len(final_test_data[key]) 54 | cc = 1 55 | for idx in final_test_data[key]: 56 | # if want to output motion probability, please set output_label == False 57 | result = nn_model.get_no_label_result(final_test_data[key][idx], output_label=True) 58 | plt.subplot(total_test, 1, cc) 59 | plt.plot(result) 60 | plt.title(idx) 61 | plt.ylim(0,1.05) 62 | cc = cc+1 63 | plt.suptitle(key) 64 | nn_model.end() 65 | plt.show() 66 | print("Done!") 67 | 68 | 69 | if __name__ == "__main__": 70 | main() 71 | 72 | -------------------------------------------------------------------------------- /data_learning.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # import os 3 | import time 4 | 5 | random_seed = 1337 6 | import random 7 | 8 | random.seed(random_seed) 9 | import numpy as np 10 | 11 | np.random.seed(random_seed) 12 | import tensorflow as tf 13 | 14 | 15 | import keras.backend as K 16 | from keras import metrics, regularizers, initializers 17 | from keras.models import Model, load_model 18 | from keras.layers import Lambda, Dense, Dropout, Input, concatenate, Flatten, BatchNormalization 19 | from keras.layers import AveragePooling2D, Activation, Conv2D, MaxPooling2D 20 | from keras.optimizers import Adam 21 | from keras.utils import to_categorical 22 | import train_test_conf as conf 23 | import argparse 24 | 25 | 26 | def get_input_arguments(): 27 | parser = argparse.ArgumentParser() 28 | parser.add_argument('-m', '--mode', help="if Y, run under training mode, if N run under test mode", type=str, 29 | default='Y') 30 | args = parser.parse_args() 31 | return args 32 | 33 | 34 | def get_classification_report(predict, truth, num_classes, label_mapping): 35 | print('\nFinal Classification Report:') 36 | results = np.zeros((len(label_mapping), num_classes), np.float32) 37 | for k in range(predict.shape[0]): 38 | results[truth[k], predict[k]] += 1 39 | for name, k in label_mapping.items(): 40 | print('label {}: has size {:.0f} static count {:.0f} motion count {:.0f}'.format(name, np.sum(results[k, :]), 41 | results[k, 0], results[k, 1])) 42 | print('\n') 43 | results /= (np.sum(results, axis=1, keepdims=True) + 1e-6) 44 | for name, k in label_mapping.items(): 45 | outstr = 'label {}: class {} acc {:.4f}'.format(name, int(k>=1), results[k, int(k>=1)]) 46 | print(outstr) 47 | 48 | 49 | class NeuralNetworkModel: 50 | def __init__(self, input_data_shape, abs_data_shape, phase_data_shape, num_classes): 51 | self.model = None 52 | self.num_classes = num_classes 53 | self.input_data_shape = input_data_shape 54 | self.abs_data_shape, self.phase_data_shape = abs_data_shape, phase_data_shape 55 | self.x_train = None 56 | self.x_test = None 57 | self.y_train = None 58 | self.y_test = None 59 | 60 | def cnn_model_phase(self, x): 61 | x = Conv2D(filters=12, kernel_size=(3, 3), strides=(1, 1), padding='valid', 62 | activation='relu', kernel_initializer=initializers.glorot_uniform())(x) 63 | x = BatchNormalization()(x) 64 | x = AveragePooling2D(pool_size=(2, 1), strides=(2, 1))(x) 65 | x = Conv2D(filters=12, kernel_size=(4, 4), strides=(1, 1), padding='valid', 66 | activation='relu', kernel_initializer=initializers.glorot_uniform())(x) 67 | x = BatchNormalization()(x) 68 | x = AveragePooling2D(pool_size=(3, 1), strides=(3, 1))(x) 69 | print("before flatten, shape of the phase data is: " + str(x.shape)) 70 | x = Flatten()(x) 71 | x = Dropout(0.5)(x) 72 | x = Dense(32, 73 | kernel_regularizer=regularizers.l2(0.02), 74 | kernel_initializer=initializers.glorot_uniform(), 75 | activation='relu')(x) 76 | x = BatchNormalization()(x) 77 | return x 78 | 79 | def cnn_model_abs(self, x): 80 | x = Conv2D(filters=12, kernel_size=(3, 3), strides=(1, 1), padding='valid', 81 | activation='relu', kernel_initializer=initializers.glorot_uniform())(x) 82 | x = BatchNormalization()(x) 83 | x = AveragePooling2D(pool_size=(2, 1), strides=(2, 1))(x) 84 | x = Conv2D(filters=12, kernel_size=(4, 4), strides=(1, 1), padding='valid', 85 | activation='relu', kernel_initializer=initializers.glorot_uniform())(x) 86 | x = BatchNormalization()(x) 87 | x = AveragePooling2D(pool_size=(3, 1), strides=(3, 1))(x) 88 | print("before flatten, shape of the abs data is: " + str(x.shape)) 89 | x = Flatten()(x) 90 | x = Dropout(0.5)(x) 91 | x = Dense(32, 92 | kernel_regularizer=regularizers.l2(0.02), 93 | kernel_initializer=initializers.glorot_uniform(), 94 | activation='relu')(x) 95 | x = BatchNormalization()(x) 96 | return x 97 | 98 | def cnn_model_abs_phase(self, ): 99 | x_input = Input(shape=self.input_data_shape, name="main_input", dtype="float32") 100 | # split CSI images into magnitude images and phase images 101 | x_abs = Lambda(lambda y: y[..., 0], name='abs_input')(x_input) 102 | # TODO: need to remove this hardcoded 6 here (hardcode it since I haven't figured a way to 103 | # save a constant into a NN model successfully). 104 | # This value should be set to self.phase_data_shape[-1](in 3X3 MIMO case, it equals to 6) 105 | x_phase = Lambda(lambda y: y[..., :6, 1], name='phase_input')(x_input) 106 | print('abs input shape {}'.format(x_abs.shape)) 107 | print('phase input shape {}'.format(x_phase.shape)) 108 | x_abs_cnn = self.cnn_model_abs(x_abs) 109 | x_phase_cnn = self.cnn_model_phase(x_phase) 110 | x = concatenate([x_abs_cnn, x_phase_cnn]) 111 | x = Dropout(0.5)(x) 112 | x = Dense(self.num_classes, 113 | kernel_regularizer=regularizers.l2(0.02), 114 | kernel_initializer=initializers.glorot_uniform(), 115 | activation='softmax', name="main_output")(x) 116 | self.model = Model(inputs=[x_input, ], outputs=x) 117 | 118 | 119 | def fit_data(self, epochs): 120 | train_num, test_num = {}, {} 121 | for m in range(self.num_classes): 122 | train_num[m] = 0 123 | test_num[m] = 0 124 | for m in range(self.y_train.shape[0]): 125 | train_num[self.y_train[m, 0]] += 1 126 | for m in range(self.y_test.shape[0]): 127 | test_num[self.y_test[m, 0]] += 1 128 | 129 | print("training data composition {}".format(train_num)) 130 | print("validating data composition {}".format(test_num)) 131 | 132 | 133 | self.y_train = to_categorical(self.y_train, self.num_classes) 134 | self.y_test = to_categorical(self.y_test, self.num_classes) 135 | Op = Adam(lr=0.001, decay=0.005, beta_1=0.9, beta_2=0.999) 136 | self.model.summary() 137 | self.model.compile(optimizer=Op, loss=['categorical_crossentropy', ], 138 | metrics=[metrics.categorical_accuracy]) 139 | self.model.fit(x=self.x_train, y=self.y_train, 140 | epochs=epochs, 141 | verbose=1, batch_size=256, shuffle=True, 142 | validation_data=(self.x_test, self.y_test)) 143 | 144 | def save_model(self, model_name): 145 | self.model.save(model_name) 146 | print("\ntrained mode was saved as {} successfully\n".format(model_name)) 147 | 148 | def load_model(self, model_name): 149 | self.model = load_model(model_name) 150 | print("model {} was loaded successfully\n".format(model_name)) 151 | # self.model.summary() 152 | 153 | def predict(self, data, output_label, batch_size=1): 154 | p = self.model.predict(data, batch_size=batch_size) 155 | if output_label: 156 | p = np.argmax(p, axis=-1) 157 | p = p.astype('int8') 158 | else: 159 | p = p[:, -1] 160 | p = p.astype('float32') 161 | return p 162 | 163 | def end(self): 164 | K.clear_session() 165 | 166 | def add_data(self, x_train, y_train, x_test, y_test): 167 | self.x_train = x_train 168 | self.y_train = y_train 169 | self.x_test = x_test 170 | self.y_test = y_test 171 | 172 | def get_data_from_file(self, file_prefix, data_type, training_mode): 173 | if training_mode: 174 | train_filename = file_prefix + 'x_train.dat' 175 | temp_image = np.fromfile(train_filename, dtype=data_type) 176 | self.x_train = np.reshape(temp_image, (-1,) + self.input_data_shape) 177 | train_label_filename = file_prefix + 'y_train.dat' 178 | temp_label = np.fromfile(train_label_filename, dtype=np.int8) 179 | self.y_train = np.reshape(temp_label, (-1, 1)) 180 | test_filename = file_prefix + 'x_validate.dat' 181 | test_label_filename = file_prefix + 'y_validate.dat' 182 | else: 183 | test_filename = file_prefix + 'x_test.dat' 184 | test_label_filename = file_prefix + 'y_test.dat' 185 | 186 | temp_image = np.fromfile(test_filename, dtype=data_type) 187 | self.x_test = np.reshape(temp_image, (-1,) + self.input_data_shape) 188 | temp_label = np.fromfile(test_label_filename, dtype=np.int8) 189 | self.y_test = np.reshape(temp_label, (-1, 1)) 190 | 191 | def get_test_result(self, label_mapping={'empty': 0, 'motion': 1}): 192 | p = self.predict(self.x_test, output_label=True, batch_size=1) 193 | get_classification_report(p, self.y_test, self.num_classes, label_mapping) 194 | return p 195 | 196 | def get_no_label_result(self, dd, output_label=True, batch_size=1): 197 | p = self.predict(dd, output_label, batch_size=batch_size) 198 | return p 199 | 200 | def get_model(self): 201 | return self.model 202 | 203 | def save_result(self, p, filename): 204 | p.tofile(filename) 205 | print("test result was saved to " + filename + "\n") 206 | 207 | 208 | def main(): 209 | args = get_input_arguments() 210 | training_mode = (args.mode == 'Y') 211 | if args.mode not in ['Y', 'N']: 212 | raise ValueError('Invalid input value for m should be either Y or N') 213 | data_folder = conf.data_folder 214 | if training_mode: 215 | label = conf.train_label 216 | data_folder += "training/" 217 | else: 218 | label = conf.test_label 219 | data_folder += "test/" 220 | nn_model = NeuralNetworkModel(conf.data_shape_to_nn, conf.abs_shape_to_nn, 221 | conf.phase_shape_to_nn, conf.total_classes) 222 | nn_model.get_data_from_file(data_folder, np.float32, training_mode) 223 | if training_mode: 224 | nn_model.cnn_model_abs_phase() 225 | nn_model.fit_data(conf.epochs) 226 | nn_model.save_model(conf.model_name) 227 | else: 228 | nn_model.load_model(conf.model_name) 229 | result = nn_model.get_test_result(label) 230 | nn_model.end() 231 | 232 | 233 | if __name__ == "__main__": 234 | main() 235 | -------------------------------------------------------------------------------- /data_preprocessing.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import numpy as np 4 | import train_test_conf as conf 5 | import argparse 6 | from global_sp_func import sp_func, reshape_func, shape_conversion, append_array 7 | 8 | 9 | def get_input_arguments(): 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument('-m', '--mode', help="if Y, run under training mode, if N run under test mode", type=str, 12 | default='Y') 13 | args = parser.parse_args() 14 | return args 15 | 16 | 17 | class DataPreprocess: 18 | def __init__(self, n_timestamps, D, step_size, ntx_max, 19 | ntx, nrx_max, nrx, nsubcarrier_max, nsubcarrier, output_shape, file_prefix, label): 20 | self.file_prefix = file_prefix 21 | self.data_shape = (n_timestamps, nrx_max, ntx_max, nsubcarrier_max) 22 | self.step_size = step_size 23 | self.n_timestamps = n_timestamps 24 | self.ntx = ntx 25 | self.nrx = nrx 26 | self.nsubcarrier = nsubcarrier 27 | self.subcarrier_spacing = int(nsubcarrier_max / nsubcarrier) 28 | self.x_train, self.y_train, self.x_test, self.y_test = np.array([]), np.array([]), np.array([]), np.array([]) 29 | self.no_label_test = None 30 | self.x_evaluate = {} 31 | self.classes_num = {} 32 | self.label = label 33 | self.output_shape = output_shape 34 | 35 | def add_image_no_label(self, test_data_class): 36 | for key in test_data_class: 37 | for idx in test_data_class[key]: 38 | test_data_class[key][idx] = reshape_func(test_data_class[key][idx], self.subcarrier_spacing) 39 | self.no_label_test = test_data_class 40 | 41 | def load_image(self, training_mode, from_file, train_data_class={}, test_data_class={}): 42 | x_train, x_test, y_train, y_test = np.array([]), np.array([]), np.array([]), np.array([]) 43 | self.classes_num = {} 44 | for label_name, o in self.label.items(): 45 | if o not in self.classes_num.keys(): 46 | self.classes_num[o] = {'train_num': 0, 'test_num': 0} 47 | if training_mode: 48 | if from_file: 49 | filename = self.file_prefix + 'training_' + str(o) + '.dat' 50 | print('train filename ' + filename) 51 | temp_image = np.fromfile(filename, dtype=np.complex64) 52 | temp_image = np.reshape(temp_image, (-1,) + self.data_shape) 53 | else: 54 | temp_image = train_data_class[o] 55 | temp_image = reshape_func(temp_image, self.subcarrier_spacing) 56 | temp_label = np.full((temp_image.shape[0], 1), o, np.int8) 57 | self.classes_num[o]['train_num'] += temp_label.shape[0] 58 | x_train = append_array(x_train, temp_image) 59 | y_train = append_array(y_train, temp_label) 60 | if from_file: 61 | test_filename = self.file_prefix + 'training_test_' + str(o) + '.dat' 62 | else: 63 | if from_file: 64 | test_filename = self.file_prefix + 'test_' + str(o) + '.dat' 65 | if from_file: 66 | print('test filename ' + test_filename) 67 | temp_image = np.fromfile(test_filename, dtype=np.complex64) 68 | temp_image = np.reshape(temp_image, (-1,) + self.data_shape) 69 | else: 70 | temp_image = test_data_class[o] 71 | if temp_image.shape[0] == 0: 72 | continue 73 | temp_image = reshape_func(temp_image, self.subcarrier_spacing) 74 | temp_label = np.full((temp_image.shape[0], 1), o, np.int8) 75 | self.classes_num[o]['test_num'] += temp_label.shape[0] 76 | x_test = append_array(x_test, temp_image) 77 | y_test = append_array(y_test, temp_label) 78 | 79 | self.x_train, self.y_train, self.x_test, self.y_test = x_train, y_train, x_test, y_test 80 | print(self.classes_num) 81 | if self.x_train.shape[0] != self.y_train.shape[0]: 82 | raise ValueError('x_train and y_train size mismatch') 83 | if self.x_test.shape[0] != self.y_test.shape[0]: 84 | raise ValueError('x_test and y_test size mismatch') 85 | 86 | def reshape_image(self): 87 | if self.x_train.shape[0] > 0: 88 | self.x_train = reshape_func(self.x_train, self.subcarrier_spacing) 89 | if self.x_test.shape[0] > 0: 90 | self.x_test = reshape_func(self.x_test, self.subcarrier_spacing) 91 | if self.no_label_test is not None: 92 | for key in self.no_label_test: 93 | for idx in self.no_label_test[key]: 94 | self.no_label_test[key][idx] = reshape_func(self.no_label_test[key][idx], self.subcarrier_spacing) 95 | 96 | def signal_processing(self, do_fft, fft_shape): 97 | if self.x_train.shape[0] > 0: 98 | self.x_train = sp_func(self.x_train, do_fft, fft_shape) 99 | if self.x_test.shape[0] > 0: 100 | self.x_test = sp_func(self.x_test, do_fft, fft_shape) 101 | if self.no_label_test is not None: 102 | for key in self.no_label_test: 103 | for idx in self.no_label_test[key]: 104 | self.no_label_test[key][idx] = sp_func(self.no_label_test[key][idx], do_fft, fft_shape) 105 | 106 | def prepare_shape(self): 107 | if self.x_train.shape[0]: 108 | self.x_train = shape_conversion(self.x_train, self.output_shape[0]) 109 | print('final training data shape {}'.format(self.x_train.shape)) 110 | if self.x_test.shape[0]: 111 | self.x_test = shape_conversion(self.x_test, self.output_shape[0]) 112 | print('final test data shape {}'.format(self.x_test.shape)) 113 | if self.no_label_test is not None: 114 | for key in self.no_label_test: 115 | for idx in self.no_label_test[key]: 116 | self.no_label_test[key][idx] = shape_conversion(self.no_label_test[key][idx], self.output_shape[0]) 117 | 118 | def save2file(self, train_mode): 119 | print('\nbegin to save data to file...') 120 | if train_mode: 121 | if self.x_train.shape[0] > 0: 122 | self.x_train.tofile(self.file_prefix + "x_train.dat") 123 | self.y_train.tofile(self.file_prefix + "y_train.dat") 124 | if self.x_test.shape[0] > 0: 125 | self.x_test.tofile(self.file_prefix + "x_validate.dat") 126 | self.y_test.tofile(self.file_prefix + "y_validate.dat") 127 | else: 128 | if self.x_test.shape[0] > 0: 129 | self.x_test.tofile(self.file_prefix + "x_test.dat") 130 | self.y_test.tofile(self.file_prefix + "y_test.dat") 131 | print("data files were saved successfully!\n") 132 | 133 | def get_data(self): 134 | return self.x_train, self.y_train, self.x_test, self.y_test 135 | 136 | def get_data_no_label(self): 137 | return self.no_label_test 138 | 139 | def print_class_info(self): 140 | for key, val in self.classes_num.items(): 141 | print("class {} has training {}, test {}".format(key, 142 | val['train_num'], 143 | val['test_num'])) 144 | 145 | 146 | def main(): 147 | args = get_input_arguments() 148 | training_mode = (args.mode == 'Y') 149 | if args.mode not in ['Y', 'N']: 150 | raise ValueError('Invalid input value for m should be either Y or N') 151 | data_folder = conf.data_folder 152 | if training_mode: 153 | label = conf.train_label 154 | data_folder += "training/" 155 | else: 156 | label = conf.test_label 157 | data_folder += "test/" 158 | data_process = DataPreprocess(conf.n_timestamps, conf.D, conf.step_size, 159 | conf.ntx_max, conf.ntx, conf.nrx_max, 160 | conf.nrx, conf.nsubcarrier_max, conf.nsubcarrier, 161 | conf.data_shape_to_nn, 162 | data_folder, label) 163 | data_process.load_image(training_mode, True) 164 | data_process.signal_processing(conf.do_fft, conf.fft_shape) 165 | data_process.prepare_shape() 166 | data_process.save2file(training_mode) 167 | 168 | 169 | if __name__ == "__main__": 170 | main() 171 | -------------------------------------------------------------------------------- /day_conf.json: -------------------------------------------------------------------------------- 1 | {"day1": {"location": "LabI", "mixed": 2, "mixed_truth": [[0, 1, 1, 0, 1], [1, 0, 0, 0, 1]], "motion": 6, "empty": 6}, "day2": {"location": "LabI", "mixed": 2, "mixed_truth": [[1, 0, 1, 0, 0], [0, 1, 1, 0, 1]], "motion": 6, "empty": 6}, "day3": {"location": "LabI", "mixed": 1, "mixed_truth": [[0, 1, 1, 0, 1]], "empty": 6, "motion": 6}, "day4": {"location": "LabII", "mixed": 4, "mixed_truth": [[1, 0, 1, 1, 0], [0, 0, 1, 0, 1], [1, 1, 0, 0, 1], [0, 1, 1, 0, 0]], "motion": 8, "empty": 1}, "day5": {"location": "LabII", "mixed": 2, "mixed_truth": [[0, 0, 1, 0, 1], [1, 0, 0, 1, 1]], "motion": 6, "empty": 1}, "day6": {"location": "LabII", "mixed": 2, "mixed_truth": [[1, 1, 0, 0, 1], [0, 0, 1, 1, 0]], "motion": 6, "empty": 1}, "day7": {"location": "LabII", "mixed": 0, "mixed_truth": [], "empty": 1, "motion": 6}, "day8": {"location": "LabII", "mixed": 2, "mixed_truth": [[1, 0, 1, 1, 0], [0, 0, 1, 1, 0]], "empty": 1, "motion": 6}, "day9": {"location": "LabII", "mixed": 2, "mixed_truth": [[1, 1, 0, 0, 1], [0, 0, 1, 0, 1]], "empty": 7, "motion": 8}, "day10": {"location": "LabII", "mixed": 4, "mixed_truth": [[1, 0, 1, 0, 0], [0, 1, 1, 0, 0], [0, 0, 1, 0, 1], [1, 0, 1, 0, 1]], "empty": 4, "motion": 8}, "day11": {"location": "LabII", "mixed": 2, "mixed_truth": [[1, 0, 0, 0, 1], [0, 0, 1, 0, 1]], "empty": 4, "motion": 8}, "day12": {"location": "LabII", "mixed": 2, "mixed_truth": [[1, 0, 0, 1, 0], [0, 1, 0, 1, 1]], "empty": 4, "motion": 8}, "day13": {"location": "LabII", "mixed": 4, "mixed_truth": [[1, 1, 0, 0, 0], [0, 1, 1, 0, 1], [0, 1, 0, 0, 0], [0, 0, 1, 1, 0]], "motion": 8, "empty": 4}, "day14": {"location": "LabII", "mixed": 2, "mixed_truth": [[1, 0, 0, 1, 0], [1, 0, 1, 0, 0]], "empty": 4, "motion": 8}, "day15": {"location": "LabII", "mixed": 4, "mixed_truth": [[0, 1, 0, 0, 1], [1, 0, 0, 1, 0], [1, 1, 0, 1, 0], [0, 0, 1, 0, 1]], "empty": 4, "motion": 8}, "day16": {"location": "LabII", "mixed": 4, "mixed_truth": [[1, 1, 0, 0, 1], [0, 0, 1, 1, 0], [1, 0, 1, 0, 0], [0, 1, 1, 0, 0]], "empty": 4, "motion": 8}, "day20": {"location": "Apartment", "mixed": 0, "mixed_truth": [], "living_room": 2, "kitchen": 2, "bedroomI": 2, "bedroomII": 2}, "day21": {"location": "Apartment", "mixed": 0, "mixed_truth": [], "empty": 1}, "day22": {"location": "Apartment", "mixed": 0, "mixed_truth": [], "living_room": 2, "kitchen": 2, "bedroomI": 2, "bedroomII": 2}, "day23": {"location": "Apartment", "mixed": 0, "mixed_truth": [], "living_room": 2, "kitchen": 2, "bedroomI": 2, "bedroomII": 2}, "day24": {"location": "Apartment", "mixed": 0, "mixed_truth": [], "empty": 1, "living_room": 2, "kitchen": 2, "bedroomI": 2, "bedroomII": 2}} 2 | -------------------------------------------------------------------------------- /global_sp_func.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | 4 | def append_array(array_a, array_b, axis=0): 5 | if array_a.size == 0: 6 | array_a = array_a.astype(array_b.dtype) 7 | array_a = array_a.reshape((0,) + array_b.shape[1:]) 8 | array_a = np.concatenate([array_a, array_b], axis) 9 | return array_a 10 | 11 | 12 | def reshape_func(d, subcarrier_spacing): 13 | d = d[..., ::subcarrier_spacing] 14 | d = np.transpose(d, [0, 1, 4, 3, 2]) 15 | d = d.reshape(d.shape[:-2] + (-1,)) 16 | return d 17 | 18 | 19 | def shape_conversion(d, l): 20 | temp_d = d[:, int(d.shape[1] / 2 - l / 2):int(d.shape[1] / 2 + l / 2), ...] 21 | d = temp_d 22 | return d 23 | 24 | 25 | def fft_func(data, fft_shape, num_dims): 26 | temp_data = data 27 | if num_dims == 1: 28 | temp_data = np.abs(np.fft.fft(temp_data, n=fft_shape[0], axis=1)) 29 | # temp_data /= np.sum(temp_data, axis=(1,), keepdims=True) 30 | temp_data = np.fft.fftshift(temp_data, axes=(1,)) 31 | else: 32 | temp_data = np.abs(np.fft.fft2(temp_data, s=fft_shape, axes=(1, 2))) 33 | # temp_data /= np.sum(temp_data, axis=(1,2), keepdims=True) 34 | temp_data = np.fft.fftshift(temp_data, axes=(1, 2)) 35 | temp_data = np.log10(temp_data + 1) 36 | return temp_data 37 | 38 | 39 | def obtain_angle(symbol_data): 40 | angle_data = np.zeros(symbol_data.shape[:-1] + (symbol_data.shape[-1] - 3,)) 41 | for i in range(3): 42 | diff_data = symbol_data[..., i * 3 + 1:i * 3 + 3] / symbol_data[..., i * 3:i * 3 + 1] 43 | angle_data[..., 2 * i:2 * i + 2] = np.angle(diff_data) 44 | return angle_data 45 | 46 | 47 | def sp_func(d, do_fft, fft_shape): 48 | phase = obtain_angle(np.copy(d)) 49 | phase = phase.astype(np.float32) 50 | ampl = np.abs(d) 51 | # normalize amplitude along time axis 52 | ampl = ampl / ampl[:, :1, ...] 53 | ampl = ampl.astype(np.float32) 54 | total_instance = phase.shape[0] 55 | if do_fft: 56 | out = np.zeros(((ampl.shape[0],) + fft_shape + (ampl.shape[-1], 2)), dtype=np.float32) 57 | for i in range(0, total_instance, 5000): 58 | num = min(total_instance - i, 5000) 59 | # ampl 2D fft 60 | out[i:i + num, ..., 0] = fft_func(np.copy(ampl[i:i + num, ...]), fft_shape, 2) 61 | # phase 1D fft 62 | unwrap_phase = np.unwrap(phase[i:i + num, ...], axis=1) 63 | out[i:i + num, ..., :unwrap_phase.shape[-1], 1] = fft_func(unwrap_phase, fft_shape, 1) 64 | return out 65 | else: 66 | out = np.zeros((ampl.shape + (2,)), dtype=np.float32) 67 | out[..., 0] = ampl 68 | # phase unwrapping 69 | for i in range(0, total_instance, 5000): 70 | num = min(total_instance-i, 5000) 71 | unwrap_phase = np.unwrap(phase[i:i+num, ...],axis=1) 72 | out[i:i+num,...,:unwrap_phase.shape[-1], 1] = unwrap_phase 73 | return out 74 | -------------------------------------------------------------------------------- /log_parsing.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import numpy as np 4 | from struct import unpack, calcsize 5 | import collections 6 | 7 | 8 | class ParseDataFile: 9 | def __init__(self): 10 | self.fmt = ' file_size-4: 34 | break 35 | offset += self.fmt_sz 36 | num_tones = current_format.num_tones 37 | nc = current_format.nc 38 | nr = current_format.nr 39 | csi_len = current_format.csi_len 40 | if csi_len > 0: 41 | if csi_len != int(nc * nr *num_tones * 2 * self.bit_per_symbol/8): 42 | print("incorrect csi len "+str(csi_len)) 43 | break 44 | csi_byte = byte_file[offset:offset+csi_len] 45 | csi_bits = np.unpackbits(csi_byte) 46 | csi_bits = np.reshape(csi_bits, (8, int(len(csi_bits)/8)), order='F') 47 | permutation = range(7, -1, -1) 48 | csi_bits = csi_bits[permutation, :] 49 | csi_bits = np.reshape(csi_bits, (self.bit_per_symbol, int(csi_len*8/self.bit_per_symbol)), order='F') 50 | csi_num = np.zeros((1, nr*nc*num_tones*2), np.float32) 51 | csi_bits = csi_bits.astype(np.uint16) 52 | for i in range(self.bit_per_symbol): 53 | csi_num[0, :] += (csi_bits[i, :] << i) 54 | csi_num[0, :] -= (csi_bits[self.bit_per_symbol-1, :]*(1 << self.bit_per_symbol)) 55 | csi = csi_num[0, 1::2] + 1j*csi_num[0, 0::2] 56 | csi = np.reshape(csi, (nr, nc, num_tones), order='F') 57 | offset += csi_len 58 | # skip H which has zero entry (in amplitude) 59 | min_abs = np.amin(np.abs(csi)) 60 | if min_abs >= 1: 61 | this_frame = {"format": current_format, 62 | "csi": csi, 63 | "rssi": current_format.rssi} 64 | frame_data.append(this_frame) 65 | count += 1 66 | 67 | offset += current_format.payload_len if has_payload else 0 68 | 69 | if offset + 420 > file_size: 70 | break 71 | 72 | # print("finished parsing file, total number of valid frame (has csi): "+str(len(frame_data))) 73 | return frame_data 74 | 75 | -------------------------------------------------------------------------------- /parse_data_from_log.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import numpy as np 4 | from log_parsing import ParseDataFile 5 | import train_test_conf as conf 6 | import argparse 7 | 8 | 9 | def get_input_arguments(): 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument('-m', '--mode', help="if 1, run under training mode, if 0 run under test mode", type=str, 12 | default='Y') 13 | args = parser.parse_args() 14 | return args 15 | 16 | 17 | def append_array(array_a, array_b, axis=0): 18 | if array_a.size == 0: 19 | array_a = array_a.astype(array_b.dtype) 20 | array_a = array_a.reshape((0,) + array_b.shape[1:]) 21 | array_a = np.concatenate([array_a, array_b], axis) 22 | return array_a 23 | 24 | 25 | class ConstructImage: 26 | def __init__(self, n_timestamps, D, step_size, ntx, nrx, n_tones, skip_frames, offset_ratio): 27 | self.n_timestamps = n_timestamps 28 | self.D = D 29 | self.frame_dur = conf.frame_dur * 1e3 # in microseconds 30 | self.ntx_max = ntx 31 | self.nrx_max = nrx 32 | self.n_tones = n_tones 33 | self.step_size = step_size 34 | self.skip_frames = skip_frames 35 | self.time_offset_tolerance = self.n_timestamps * offset_ratio * self.D * self.frame_dur 36 | # print('allowed time offset {}'.format(self.time_offset_tolerance)) 37 | 38 | def process_data(self, frame_data): 39 | frame_data = frame_data[self.skip_frames:-self.skip_frames] 40 | num_instances = max(0, int((len(frame_data) - self.n_timestamps * self.D) / self.step_size) + 5) 41 | 42 | final_data = np.zeros((num_instances, self.n_timestamps, self.nrx_max, self.ntx_max, self.n_tones), 43 | dtype=np.complex64) 44 | if num_instances == 0: 45 | return final_data 46 | d = 0 47 | valid_instance_c = 0 48 | while d < len(frame_data) - self.n_timestamps * self.D: 49 | temp_image = np.zeros((self.n_timestamps, self.nrx_max, self.ntx_max, self.n_tones), dtype=np.complex64) 50 | valid = True 51 | offset = self.step_size 52 | start_time = end_time = 0 53 | time_index = [] 54 | for k in range(self.n_timestamps): 55 | m = d + k * self.D 56 | nc = frame_data[m]['format'].nc 57 | csi = frame_data[m]['csi'] 58 | if nc < self.ntx_max: 59 | # print("not enough transmit antenna") 60 | valid = False 61 | offset = k * self.D + 1 62 | break 63 | if k == 0: 64 | start_time = frame_data[m]['format'].timestamp 65 | elif k == self.n_timestamps - 1: 66 | end_time = frame_data[m]['format'].timestamp 67 | time_off = abs(end_time - start_time - (self.n_timestamps - 1) * self.D * self.frame_dur) 68 | if end_time < start_time: # reseting error, skip 69 | valid = False 70 | offset = k * self.D + 1 71 | break 72 | if time_off > self.time_offset_tolerance: 73 | # print("timing off is {:.3f}".format(time_off/(self.D*self.frame_dur))) 74 | valid = False 75 | offset = 1 76 | break 77 | temp_image[k, :, :nc, :] = csi 78 | time_index.append(frame_data[m]['format'].timestamp) 79 | if valid: 80 | final_data[valid_instance_c, ...] = temp_image 81 | valid_instance_c = valid_instance_c + 1 82 | d = d + offset 83 | final_data = final_data[:valid_instance_c, ...] 84 | print("total number of images: " + str(final_data.shape[0])) 85 | return final_data 86 | 87 | 88 | class DataLogParser: 89 | def __init__(self, n_timestamps, D, step_size, ntx_max, 90 | nrx_max, nsubcarrier_max, file_prefix, log_file_prefix, skip_frames, time_offset_ratio, conf, labels): 91 | self.parser = ParseDataFile() 92 | self.image_constructor = ConstructImage(n_timestamps, D, step_size, 93 | ntx_max, nrx_max, nsubcarrier_max, skip_frames, time_offset_ratio) 94 | self.file_prefix = file_prefix 95 | self.log_file_prefix = log_file_prefix 96 | self.data_shape = (n_timestamps, nrx_max, ntx_max, nsubcarrier_max) 97 | self.step_size = step_size 98 | self.n_timestamps = n_timestamps 99 | self.x_train, self.y_train, self.x_test, self.y_test = None, None, None, None 100 | self.conf = conf 101 | self.out_data_train, self.out_data_test, self.out_data_no_label = {}, {}, {} 102 | self.label = labels 103 | for k, o in self.label.items(): 104 | self.out_data_train[o] = np.array([]) 105 | self.out_data_test[o] = np.array([]) 106 | 107 | def generate_image(self, train_date, test_date): 108 | date = train_date + test_date 109 | for d in date: 110 | day_index = int(d[3:]) 111 | logfilename = self.log_file_prefix + d + '/' 112 | for label_name, o in self.label.items(): 113 | if label_name in self.conf[d]: 114 | total_tests = self.conf[d][label_name] 115 | else: 116 | continue 117 | for i in range(1, total_tests + 1): 118 | # first 3 days' logs not only have csi but also payload for each received frame 119 | has_payload = day_index <= 3 120 | frame_data = self.parser.parse(logfilename + label_name + str(i) + ".data", has_payload) 121 | dd = self.image_constructor.process_data(frame_data) 122 | if d in test_date: 123 | self.out_data_test[o] = append_array(self.out_data_test[o], dd) 124 | else: 125 | self.out_data_train[o] = append_array(self.out_data_train[o], dd) 126 | 127 | def generate_image_no_label(self, date, label_name): 128 | for d in date: 129 | day_index = int(d[3:]) 130 | logfilename = self.log_file_prefix + d + '/' 131 | if label_name not in self.conf[d]: 132 | continue 133 | total_tests = self.conf[d][label_name] 134 | if total_tests == 0: 135 | continue 136 | self.out_data_no_label[d] = {} 137 | print('on ' + d) 138 | for i in range(1, total_tests + 1): 139 | # first 3 days' logs not only have csi but also payload for each received frame 140 | has_payload = day_index <= 3 141 | frame_data = self.parser.parse(logfilename + label_name + str(i) + ".data", has_payload) 142 | dd = self.image_constructor.process_data(frame_data) 143 | self.out_data_no_label[d][label_name + '_' + str(i)] = dd 144 | print('add data from label: {} with index {}'.format(label_name,i)) 145 | 146 | def save_data(self, train_model): 147 | print('\nbegin to save data to file...') 148 | for k, o in self.label.items(): 149 | if train_model: 150 | self.out_data_train[o].tofile(self.file_prefix + "training_" + str(o) + '.dat') 151 | self.out_data_test[o].tofile(self.file_prefix + "training_test_" + str(o) + '.dat') 152 | else: 153 | self.out_data_test[o].tofile(self.file_prefix + "test_" + str(o) + '.dat') 154 | print("data files were saved successfully!\n") 155 | 156 | def get_data(self): 157 | return self.out_data_train, self.out_data_test 158 | 159 | def get_data_no_label(self): 160 | return self.out_data_no_label 161 | 162 | 163 | def main(): 164 | args = get_input_arguments() 165 | training_mode = (args.mode == 'Y') 166 | if args.mode not in ['Y', 'N']: 167 | raise ValueError('Invalid input value for m should be either Y or N') 168 | data_folder = conf.data_folder 169 | if training_mode: 170 | label = conf.train_label 171 | data_folder += "training/" 172 | else: 173 | label = conf.test_label 174 | data_folder += "test/" 175 | data_generator = DataLogParser(conf.n_timestamps, conf.D, conf.step_size, 176 | conf.ntx_max, conf.nrx_max, 177 | conf.nsubcarrier_max, data_folder, 178 | conf.log_folder, 179 | conf.skip_frames, 180 | conf.time_offset_ratio, 181 | conf.day_conf, 182 | label) 183 | if training_mode: 184 | print('in training mode') 185 | print('training data from {} \nvalidation data from {}\n'.format(conf.training_date, conf.training_validate_date)) 186 | print('training label is {}\n'.format(label)) 187 | data_generator.generate_image(conf.training_date, conf.training_validate_date) 188 | else: 189 | print('in test mode') 190 | print('test date from {}'.format(conf.test_date)) 191 | print('test label is {}\n'.format(label)) 192 | data_generator.generate_image([], conf.test_date) 193 | data_generator.save_data(training_mode) 194 | 195 | 196 | if __name__ == "__main__": 197 | main() 198 | -------------------------------------------------------------------------------- /test_date_conf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import os 3 | import json 4 | 5 | 6 | def parse_test_days(direc_prefix, total_days, exclude_days): 7 | ''' 8 | Description: 9 | generate a dictionary that stores the configurations of each day's collected data 10 | by parsing readme.txt 11 | 12 | Input: 13 | direc_prefix (str): 14 | folder directory where all the data is stored 15 | total_days (int): 16 | total number of experimental days 17 | 18 | Output: 19 | return day_conf 20 | 21 | day_conf (dict): 22 | key (str) -- 'day'+str(index) (index starts from 1 to total_days) 23 | value (dict)-- test_conf (dictionary) 24 | 25 | test_conf (dict): 26 | key (str) -- "location" 27 | value (str) -- where the experiment was conducted 28 | key (str) -- 'motion' 29 | value (int) -- total number of motion tests conducted (valid for LabI or LabII) 30 | key (str) -- 'living_room' or 'kitchen' or 'bedroomI' or 'bedroomII' 31 | value (int) -- total number of motion tests conducted in different rooms (valid for Apartment) 32 | key (str) -- 'empty' 33 | value (int) -- total number of tests conducted when there is nobody inside the environment 34 | key (str) -- 'mixed' 35 | value (int) -- total number of mixed runs (no mixed runs were conducted in Apartment) 36 | key (str) -- 'mixed_truth' 37 | value (list) -- each entry of the list is also a list that 38 | contains the ground truth of this mixed run. 39 | 40 | ''' 41 | 42 | day_conf = {} 43 | # mapping data to label 44 | label = {'empty': 0, 'motion': 1} 45 | 46 | for i in range(1, total_days + 1, 1): 47 | if i in exclude_days: continue 48 | day_index = 'day' + str(i) 49 | d_path = direc_prefix + day_index + '/' 50 | with open(d_path + 'readme.txt', 'r') as f: 51 | print('processing day {}'.format(i)) 52 | location, cases, mixed_cnt, mixed_state = None, {}, 0, [] 53 | for l in f: 54 | m = l.split() 55 | if len(m) == 0: 56 | continue 57 | if 'Location' in m[0]: 58 | location = m[-1] 59 | elif 'mixed' in m[0]: 60 | mixed_cnt += 1 61 | idx = int(m[0][-2]) 62 | mixed_index = 'mixed' + str(idx) 63 | status = m[1:] 64 | mixed_state.append([]) 65 | for s in status: 66 | if 'empty' in s: 67 | mixed_state[-1].append(label['empty']) 68 | elif 'motion' in s: 69 | mixed_state[-1].append(label['motion']) 70 | else: 71 | print('undefined status in {}'.format(m[0])) 72 | else: 73 | case_type = m[0][:-1] 74 | case_cnt = int(m[-1]) 75 | cases.update({case_type: case_cnt}) 76 | 77 | if location == None or cases == {}: 78 | raise Exception('invalid info {} {}'.format(location, cases)) 79 | 80 | day_conf[day_index] = {'location': location, 'mixed': mixed_cnt, 'mixed_truth': mixed_state} 81 | day_conf[day_index].update(cases) 82 | print(day_conf[day_index]) 83 | print('\n') 84 | for k, v in day_conf[day_index].items(): 85 | if k == 'location' or k == 'mixed_truth': 86 | continue 87 | for j in range(1, v + 1, 1): 88 | f_name = d_path + k + str(j) + '.data' 89 | if not os.path.exists(f_name): 90 | print("{} doesn't exist !!!!".format(f_name)) 91 | return day_conf 92 | 93 | 94 | def main(): 95 | total_days = 24 96 | exclude_days = [17, 18, 19] 97 | data_folder = '/root/share/upload_wifi_data/' 98 | day_conf = parse_test_days(data_folder, total_days, exclude_days) 99 | to_json = json.dumps(day_conf) 100 | # json filename 101 | save_json_filename = 'day_conf.json' 102 | # save day_conf to json file 103 | with open(save_json_filename, 'w') as f: 104 | f.write(to_json) 105 | print('json file was saved as ' + save_json_filename) 106 | 107 | 108 | if __name__ == "__main__": 109 | main() 110 | -------------------------------------------------------------------------------- /train_test_conf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import json 3 | from test_date_conf import parse_test_days 4 | 5 | # folder directory where all the data is stored (absolute path) 6 | log_folder = '/root/share/upload_wifi_data/' 7 | # folder directory used to store processed data (path relative to repo directory) 8 | data_folder = 'data/' 9 | # folder directory used to store model (path relative to repo directory 10 | model_folder = 'model/' 11 | # json file which stores configuration of every day's collected data 12 | save_json_filename = 'day_conf.json' 13 | total_days = 24 14 | use_exist_json = True 15 | if use_exist_json: 16 | with open(save_json_filename, 'r') as f: 17 | day_conf = json.loads(f.read()) 18 | if len(day_conf) == 16: 19 | print('day_conf was loaded successfully') 20 | else: 21 | day_conf = parse_test_days(log_folder, total_days) 22 | to_json = json.dumps(day_conf) 23 | with open(save_json_filename, 'w') as f: 24 | f.write(to_json) 25 | print('json file was saved as '+save_json_filename) 26 | 27 | ntx_max, nrx_max, nsubcarrier_max = 3, 3, 56 28 | ntx, nrx, nsubcarrier = 3, 3, 14 29 | 30 | n_timestamps = 128 # number of consecutive CSIs used to contruct an image 31 | do_fft = True 32 | fft_shape = (n_timestamps, nsubcarrier) 33 | data_shape_to_nn = (50, nsubcarrier, ntx*nrx, 2) 34 | abs_shape_to_nn = (50, nsubcarrier, ntx*nrx) 35 | phase_shape_to_nn = (50, nsubcarrier, ntx*(nrx-1)) 36 | time_offset_ratio = 1.0/20.0 37 | D = 1 # H step size 38 | step_size = 33 # CSI image step size 39 | frame_dur = 10 # milliseconds 40 | skip_time = 5000 # milliseconds 41 | # exclude first and last skip_frames in the current run 42 | skip_frames = skip_time//frame_dur 43 | 44 | 45 | train_label = {'empty': 0, 'motion':1} # key: types of runs; value: class (0 or 1) it belongs 46 | total_classes = 2 47 | draw_date = ['day5', ] 48 | draw_label = 'mixed' 49 | training_date = ['day9','day10', 'day11', 'day12', 'day13', 'day14'] 50 | training_validate_date = ['day15', 'day16'] 51 | # make sure validation data and training data come from disjoint days 52 | for d in training_validate_date: 53 | if d in training_date: 54 | raise ValueError('validation date {} should not appear in train date'.format(d)) 55 | test_date = ['day24'] 56 | has_test_from_apartment = False 57 | has_test_from_lab = False 58 | for d in test_date: 59 | if int(d[3:]) >= 20: 60 | # in Apartment 61 | has_test_from_apartment = True 62 | else: 63 | # in LabI or LabII 64 | has_test_from_lab = True 65 | if has_test_from_apartment and has_test_from_lab: 66 | test_label = {'empty': 0, 'motion':1, 'living_room': 2, 'kitchen': 3, 'bedroomI': 4, 'bedroomII': 5} 67 | elif has_test_from_apartment: 68 | test_label = {'empty': 0, 'living_room': 1, 'kitchen': 2, 'bedroomI': 3, 'bedroomII': 4} 69 | else: 70 | test_label = {'empty': 0, 'motion': 1} 71 | 72 | epochs = 10 73 | # where to save/store the nn model 74 | model_name = model_folder+'wifi_presence_model.h5' 75 | -------------------------------------------------------------------------------- /wifi_process_combo.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import numpy as np 4 | import argparse 5 | from parse_data_from_log import DataLogParser 6 | from data_preprocessing import DataPreprocess 7 | from data_learning import NeuralNetworkModel 8 | import train_test_conf as conf 9 | 10 | 11 | 12 | def get_input_arguments(): 13 | parser = argparse.ArgumentParser() 14 | parser.add_argument('-m', '--mode', help="if Y, run under training mode, if N run under test mode", type=str, 15 | default='Y') 16 | args = parser.parse_args() 17 | return args 18 | 19 | 20 | def main(): 21 | args = get_input_arguments() 22 | training_mode = (args.mode == 'Y') 23 | if args.mode not in ['Y', 'N']: 24 | raise ValueError('Invalid input value for m should be either Y or N') 25 | data_folder = conf.data_folder 26 | if training_mode: 27 | label = conf.train_label 28 | print('in training mode') 29 | print('training data from {} \nvalidation data from {}\n'.format(conf.training_date, conf.training_validate_date)) 30 | print('training label is {}\n'.format(label)) 31 | data_folder += "training/" 32 | else: 33 | label = conf.test_label 34 | print('in test mode') 35 | print('test date from {}'.format(conf.test_date)) 36 | print('test label is {}\n'.format(label)) 37 | data_folder += "test/" 38 | ################################################## 39 | # parse data from original data & construct images 40 | ################################################## 41 | print("parsing data from log files which are generated by Atheros-CSI-TOOL\n") 42 | data_generator = DataLogParser(conf.n_timestamps, conf.D, conf.step_size, 43 | conf.ntx_max, conf.nrx_max, conf.nsubcarrier_max, 44 | data_folder, conf.log_folder, 45 | conf.skip_frames, 46 | conf.time_offset_ratio, 47 | conf.day_conf, 48 | label) 49 | train_date = conf.training_date if training_mode else [] 50 | if training_mode: 51 | data_generator.generate_image(conf.training_date, conf.training_validate_date) 52 | else: 53 | data_generator.generate_image([], conf.test_date) 54 | # train_data, test_data: classes (key: label, value: images under this label) 55 | train_data, test_data = data_generator.get_data() 56 | 57 | ################################################## 58 | # apply signal processing blocks to images 59 | ################################################## 60 | print("Pre-processing data\n") 61 | data_process = DataPreprocess(conf.n_timestamps, conf.D, conf.step_size, 62 | conf.ntx_max, conf.ntx, conf.nrx_max, 63 | conf.nrx, conf.nsubcarrier_max, conf.nsubcarrier, 64 | conf.data_shape_to_nn, 65 | data_folder, label) 66 | data_process.load_image(training_mode, False, train_data, test_data) 67 | data_process.signal_processing(conf.do_fft, conf.fft_shape) 68 | data_process.prepare_shape() 69 | x_train, y_train, x_test, y_test = data_process.get_data() 70 | ################################################## 71 | # train or test data with neural netowrk 72 | ################################################## 73 | 74 | nn_model = NeuralNetworkModel(conf.data_shape_to_nn, conf.abs_shape_to_nn, 75 | conf.phase_shape_to_nn, conf.total_classes) 76 | nn_model.add_data(x_train, y_train, x_test, y_test) 77 | if training_mode: 78 | print("Building a new model (in training mode)\n") 79 | nn_model.cnn_model_abs_phase() 80 | nn_model.fit_data(conf.epochs) 81 | nn_model.save_model(conf.model_name) 82 | else: 83 | print("Get test result using existing model (in test mode)\n") 84 | nn_model.load_model(conf.model_name) 85 | result = nn_model.get_test_result(label) 86 | # nn_model.save_result(result, conf.file_prefix+conf.test_result_filename) 87 | nn_model.end() 88 | print("Done!") 89 | 90 | 91 | if __name__ == "__main__": 92 | main() 93 | --------------------------------------------------------------------------------