├── README.md ├── main_tensorflow.py └── pytorch ├── data_preprocess.py ├── main_pytorch.py └── network.py /README.md: -------------------------------------------------------------------------------- 1 | # Deep Learning for Human Activity Recognition 2 | 3 | > Deep learning is perhaps the nearest future of human activity recognition. While there are many existing non-deep method, we still want to unleash the full power of deep learning. This repo provides a demo of using deep learning to perform human activity recognition. 4 | 5 | **We support both Tensorflow and Pytorch.** 6 | 7 | ## Prerequisites 8 | 9 | - Python 3.x 10 | - Numpy 11 | - Tensorflow or Pytorch 1.0+ 12 | 13 | ## Dataset 14 | 15 | There are many public datasets for human activity recognition. You can refer to this survey article [Deep learning for sensor-based activity recognition: a survey](https://arxiv.org/abs/1707.03502) to find more. 16 | 17 | In this demo, we will use UCI HAR dataset as an example. This dataset can be found in [here](https://archive.ics.uci.edu/ml/machine-learning-databases/00240/). 18 | 19 | Of course, this dataset needs further preprocessing before being put into the network. I've also provided a preprocessing version of the dataset as a `.npz` file so you can focus on the network (download [HERE](https://pan.baidu.com/s/1Nx7UcPqmXVQgNVZv4Ec1yg)). It is also highly recommended you download the dataset so that you can experience all the process on your own. 20 | 21 | | #subject | #activity | Frequency | 22 | | --- | --- | --- | 23 | | 30 | 6 | 50 Hz | 24 | 25 | ## Usage 26 | 27 | - For Pytorch (recommend), go to `pytorch` folder, config the folder of your data in `config.py', and then run `main_pytorch.py`. 28 | 29 | - For tensorflow, run `main_tensorflow.py` file. The update of tensorflow version is stopped since I personally like Pytorch. 30 | 31 | ## Network structure 32 | 33 | What is the most influential deep structure? CNN it is. So we'll use **CNN** in our demo. 34 | 35 | ### CNN structure 36 | 37 | Convolution + pooling + convolution + pooling + dense + dense + dense + output 38 | 39 | That is: 2 convolutions, 2 poolings, and 3 fully connected layers. 40 | 41 | ### About the inputs 42 | 43 | That dataset contains 9 channels of the inputs: (acc_body, acc_total and acc_gyro) on x-y-z. So the input channel is 9. 44 | 45 | Dataset providers have clipped the dataset using sliding window, so every 128 in `.txt` can be considered as an input. In real life, you need to first clipped the input using sliding window. 46 | 47 | So in the end, we reformatted the inputs from 9 inputs files to 1 file, the shape of that file is `[n_sample,128,9]`, that is, every windows has 9 channels with each channel has length 128. When feeding it to Tensorflow, it has to be reshaped to `[n_sample,9,1,128]` as we expect there is 128 X 1 signals for every channel. 48 | 49 | ## Related projects 50 | 51 | - [Must-read papers about deep learning based human activity recognition](https://github.com/jindongwang/activityrecognition/blob/master/notes/deep.md) 52 | - [guillaume-chevalier/LSTM-Human-Activity-Recognition](https://github.com/guillaume-chevalier/LSTM-Human-Activity-Recognition) 53 | - [aqibsaeed/Human-Activity-Recognition-using-CNN](https://github.com/aqibsaeed/Human-Activity-Recognition-using-CNN) 54 | 55 | -------------------------------------------------------------------------------- /main_tensorflow.py: -------------------------------------------------------------------------------- 1 | # encoding=utf-8 2 | """ 3 | Created on 15:07 2017/8/16 4 | @author: Jindong Wang 5 | """ 6 | 7 | import os 8 | 9 | # This is for showing the Tensorflow log 10 | # os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' 11 | 12 | import numpy as np 13 | import tensorflow as tf 14 | 15 | 16 | # This is for parsing the X data, you can ignore it if you do not need preprocessing 17 | def format_data_x(datafile): 18 | x_data = None 19 | for item in datafile: 20 | item_data = np.loadtxt(item, dtype=np.float) 21 | if x_data is None: 22 | x_data = np.zeros((len(item_data), 1)) 23 | x_data = np.hstack((x_data, item_data)) 24 | x_data = x_data[:, 1:] 25 | print(x_data.shape) 26 | X = None 27 | for i in range(len(x_data)): 28 | row = np.asarray(x_data[i, :]) 29 | row = row.reshape(9, 128).T 30 | if X is None: 31 | X = np.zeros((len(x_data), 128, 9)) 32 | X[i] = row 33 | print(X.shape) 34 | return X 35 | 36 | 37 | # This is for parsing the Y data, you can ignore it if you do not need preprocessing 38 | def format_data_y(datafile): 39 | data = np.loadtxt(datafile, dtype=np.int) - 1 40 | YY = np.eye(6)[data] 41 | return YY 42 | 43 | 44 | # Load data function, if there exists parsed data file, then use it 45 | # If not, parse the original dataset from scratch 46 | def load_data(): 47 | import os 48 | if os.path.isfile('data/data_har.npz') == True: 49 | data = np.load('data/data_har.npz') 50 | X_train = data['X_train'] 51 | Y_train = data['Y_train'] 52 | X_test = data['X_test'] 53 | Y_test = data['Y_test'] 54 | else: 55 | # This for processing the dataset from scratch 56 | # After downloading the dataset, put it to somewhere that str_folder can find 57 | str_folder = 'Your root folder' + 'UCI HAR Dataset/' 58 | INPUT_SIGNAL_TYPES = [ 59 | "body_acc_x_", 60 | "body_acc_y_", 61 | "body_acc_z_", 62 | "body_gyro_x_", 63 | "body_gyro_y_", 64 | "body_gyro_z_", 65 | "total_acc_x_", 66 | "total_acc_y_", 67 | "total_acc_z_" 68 | ] 69 | 70 | str_train_files = [str_folder + 'train/' + 'Inertial Signals/' + item + 'train.txt' for item in 71 | INPUT_SIGNAL_TYPES] 72 | str_test_files = [str_folder + 'test/' + 'Inertial Signals/' + item + 'test.txt' for item in INPUT_SIGNAL_TYPES] 73 | str_train_y = str_folder + 'train/y_train.txt' 74 | str_test_y = str_folder + 'test/y_test.txt' 75 | 76 | X_train = format_data_x(str_train_files) 77 | X_test = format_data_x(str_test_files) 78 | Y_train = format_data_y(str_train_y) 79 | Y_test = format_data_y(str_test_y) 80 | 81 | return X_train, Y_train, X_test, Y_test 82 | 83 | 84 | # A class for some hyperparameters 85 | class Config(object): 86 | def __init__(self, X_train, Y_train): 87 | self.n_input = len(X_train[0]) # number of input neurons to the network 88 | self.n_output = len(Y_train[0]) # number of output neurons 89 | self.dropout = 0.8 # dropout, between 0 and 1 90 | self.learning_rate = 0.001 # learning rate, float 91 | self.training_epoch = 20 # training epoch 92 | self.n_channel = 9 # number of input channel 93 | self.input_height = 128 # input height 94 | self.input_width = 1 # input width 95 | self.kernel_size = 64 # number of convolution kernel size 96 | self.depth = 32 # number of convolutions 97 | self.batch_size = 16 # batch size 98 | self.show_progress = 50 # how many batches to show the progress 99 | 100 | # weights and biases definition 101 | self.weights = { 102 | 'wc1': tf.Variable(tf.random_normal([1, self.kernel_size, self.n_channel, self.depth])), 103 | 'wc2': tf.Variable(tf.random_normal([1, self.kernel_size, self.depth, 64])), 104 | 'wd1': tf.Variable(tf.random_normal([32 * 32 * 2, 1000])), 105 | 'wd2': tf.Variable(tf.random_normal([1000, 500])), 106 | 'wd3': tf.Variable(tf.random_normal([500, 300])), 107 | 'out': tf.Variable(tf.random_normal([300, self.n_output])) 108 | } 109 | 110 | self.biases = { 111 | 'bc1': tf.Variable(tf.random_normal([self.depth])), 112 | 'bc2': tf.Variable(tf.random_normal([64])), 113 | 'bd1': tf.Variable(tf.random_normal([1000])), 114 | 'bd2': tf.Variable(tf.random_normal([500])), 115 | 'bd3': tf.Variable(tf.random_normal([300])), 116 | 'out': tf.Variable(tf.random_normal([self.n_output])) 117 | } 118 | 119 | 120 | # wrap of conv1d 121 | def conv1d(x, W, b, stride): 122 | x = tf.nn.conv2d(x, W, strides=[1, stride, 1, 1], padding='SAME') 123 | x = tf.add(x, b) 124 | return tf.nn.relu(x) 125 | 126 | 127 | # wrap of maxpool1d 128 | def maxpool1d(x, kernel_size, stride): 129 | return tf.nn.max_pool(x, ksize=[1, kernel_size, 1, 1], strides=[1, stride, 1, 1], padding='VALID') 130 | 131 | 132 | # network definition 133 | def conv_net(x, W, b, dropout): 134 | conv1 = conv1d(x, W['wc1'], b['bc1'], 1) 135 | conv1 = maxpool1d(conv1, 2, stride=2) 136 | conv2 = conv1d(conv1, W['wc2'], b['bc2'], 1) 137 | conv2 = maxpool1d(conv2, 2, stride=2) 138 | conv2 = tf.reshape(conv2, [-1, W['wd1'].get_shape().as_list()[0]]) 139 | fc1 = tf.add(tf.matmul(conv2, W['wd1']), b['bd1']) 140 | fc1 = tf.nn.relu(fc1) 141 | fc1 = tf.nn.dropout(fc1, keep_prob=dropout) 142 | fc2 = tf.add(tf.matmul(fc1, W['wd2']), b['bd2']) 143 | fc2 = tf.nn.relu(fc2) 144 | fc2 = tf.nn.dropout(fc2, keep_prob=dropout) 145 | fc3 = tf.add(tf.matmul(fc2, W['wd3']), b['bd3']) 146 | fc3 = tf.nn.relu(fc3) 147 | fc3 = tf.nn.dropout(fc3, keep_prob=dropout) 148 | out = tf.add(tf.matmul(fc3, W['out']), b['out']) 149 | return out 150 | 151 | 152 | # wrap the network for training and testing 153 | def network(X_train, Y_train, X_test, Y_test): 154 | config = Config(X_train, Y_train) 155 | 156 | # X, Y and keep_prob are three feeds to the network 157 | X = tf.placeholder(tf.float32, shape=[None, config.input_height, config.input_width, config.n_channel]) 158 | Y = tf.placeholder(tf.float32, shape=[None, config.n_output]) 159 | keep_prob = tf.placeholder(tf.float32) 160 | 161 | y_pred = conv_net(X, config.weights, config.biases, config.dropout) 162 | cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=Y, logits=y_pred)) 163 | optimizer = tf.train.AdamOptimizer(learning_rate=config.learning_rate).minimize(cost) 164 | 165 | correct_pred = tf.equal(tf.arg_max(y_pred, 1), tf.argmax(Y, 1)) 166 | accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32)) 167 | 168 | total_batch = len(X_train) // config.batch_size 169 | 170 | init = tf.global_variables_initializer() 171 | with tf.Session() as sess: 172 | sess.run(init) 173 | for i in range(config.training_epoch): 174 | for j in range(total_batch): 175 | x_train_batch, y_train_batch = X_train[j * config.batch_size: config.batch_size * (j + 1)], \ 176 | Y_train[j * config.batch_size: config.batch_size * (j + 1)] 177 | x_train_batch = np.reshape(x_train_batch, [len(x_train_batch), 128, 1, 9]) 178 | sess.run(optimizer, feed_dict={X: x_train_batch, Y: y_train_batch, keep_prob: config.dropout}) 179 | if j % config.show_progress == 0: 180 | loss, acc = sess.run([cost, accuracy], 181 | feed_dict={X: x_train_batch, 182 | Y: y_train_batch, 183 | keep_prob: config.dropout}) 184 | print('Epoch:%02d,batch:%03d,loss:%.8f,accuracy:%.8f' % ( 185 | i + 1, (j + 1) * config.batch_size, loss, acc)) 186 | print('Optimization finished!') 187 | acc_test = sess.run(accuracy, feed_dict={X: np.reshape(X_test, [len(X_test), 128, 1, 9]), 188 | Y: np.reshape(Y_test, [len(Y_test), 6]), 189 | keep_prob: 1.}) 190 | print('Accuracy of testing:%.8f' % acc_test) 191 | 192 | 193 | if __name__ == '__main__': 194 | X_train, Y_train, X_test, Y_test = load_data() 195 | # normalizing the data 196 | X_train = (X_train - np.mean(X_train)) / np.std(X_train) 197 | X_test = (X_test - np.mean(X_test)) / np.std(X_test) 198 | # build the network 199 | network(X_train, Y_train, X_test, Y_test) -------------------------------------------------------------------------------- /pytorch/data_preprocess.py: -------------------------------------------------------------------------------- 1 | # encoding=utf-8 2 | """ 3 | Created on 10:38 2018/11/10 4 | @author: Jindong Wang 5 | """ 6 | 7 | import numpy as np 8 | from torch.utils.data import Dataset, DataLoader 9 | 10 | 11 | # This is for parsing the X data, you can ignore it if you do not need preprocessing 12 | def format_data_x(datafile): 13 | x_data = None 14 | for item in datafile: 15 | item_data = np.loadtxt(item, dtype=np.float) 16 | if x_data is None: 17 | x_data = np.zeros((len(item_data), 1)) 18 | x_data = np.hstack((x_data, item_data)) 19 | x_data = x_data[:, 1:] 20 | print(x_data.shape) 21 | X = None 22 | for i in range(len(x_data)): 23 | row = np.asarray(x_data[i, :]) 24 | row = row.reshape(9, 128).T 25 | if X is None: 26 | X = np.zeros((len(x_data), 128, 9)) 27 | X[i] = row 28 | print(X.shape) 29 | return X 30 | 31 | 32 | # This is for parsing the Y data, you can ignore it if you do not need preprocessing 33 | def format_data_y(datafile): 34 | data = np.loadtxt(datafile, dtype=np.int) - 1 35 | YY = np.eye(6)[data] 36 | return YY 37 | 38 | 39 | # Load data function, if there exists parsed data file, then use it 40 | # If not, parse the original dataset from scratch 41 | def load_data(data_folder): 42 | import os 43 | if os.path.isfile(data_folder + 'data_har.npz') == True: 44 | data = np.load(data_folder + 'data_har.npz') 45 | X_train = data['X_train'] 46 | Y_train = data['Y_train'] 47 | X_test = data['X_test'] 48 | Y_test = data['Y_test'] 49 | else: 50 | # This for processing the dataset from scratch 51 | # After downloading the dataset, put it to somewhere that str_folder can find 52 | str_folder = 'data_folder' + 'UCI HAR Dataset/' 53 | INPUT_SIGNAL_TYPES = [ 54 | "body_acc_x_", 55 | "body_acc_y_", 56 | "body_acc_z_", 57 | "body_gyro_x_", 58 | "body_gyro_y_", 59 | "body_gyro_z_", 60 | "total_acc_x_", 61 | "total_acc_y_", 62 | "total_acc_z_" 63 | ] 64 | 65 | str_train_files = [str_folder + 'train/' + 'Inertial Signals/' + item + 'train.txt' for item in 66 | INPUT_SIGNAL_TYPES] 67 | str_test_files = [str_folder + 'test/' + 'Inertial Signals/' + 68 | item + 'test.txt' for item in INPUT_SIGNAL_TYPES] 69 | str_train_y = str_folder + 'train/y_train.txt' 70 | str_test_y = str_folder + 'test/y_test.txt' 71 | 72 | X_train = format_data_x(str_train_files) 73 | X_test = format_data_x(str_test_files) 74 | Y_train = format_data_y(str_train_y) 75 | Y_test = format_data_y(str_test_y) 76 | 77 | return X_train, onehot_to_label(Y_train), X_test, onehot_to_label(Y_test) 78 | 79 | 80 | def onehot_to_label(y_onehot): 81 | a = np.argwhere(y_onehot == 1) 82 | return a[:, -1] 83 | 84 | 85 | class data_loader(Dataset): 86 | def __init__(self, samples, labels, t): 87 | self.samples = samples 88 | self.labels = labels 89 | self.T = t 90 | 91 | def __getitem__(self, index): 92 | sample, target = self.samples[index], self.labels[index] 93 | if self.T: 94 | return self.T(sample), target 95 | else: 96 | return sample, target 97 | 98 | def __len__(self): 99 | return len(self.samples) 100 | 101 | 102 | def normalize(x): 103 | x_min = x.min(axis=(0, 2, 3), keepdims=True) 104 | x_max = x.max(axis=(0, 2, 3), keepdims=True) 105 | x_norm = (x - x_min) / (x_max - x_min) 106 | return x_norm 107 | 108 | 109 | def load(data_folder, batch_size=64): 110 | x_train, y_train, x_test, y_test = load_data(data_folder) 111 | x_train, x_test = x_train.reshape( 112 | (-1, 9, 1, 128)), x_test.reshape((-1, 9, 1, 128)) 113 | transform = None 114 | train_set = data_loader(x_train, y_train, transform) 115 | test_set = data_loader(x_test, y_test, transform) 116 | train_loader = DataLoader( 117 | train_set, batch_size=batch_size, shuffle=True, drop_last=True) 118 | test_loader = DataLoader(test_set, batch_size=batch_size, shuffle=False) 119 | return train_loader, test_loader 120 | -------------------------------------------------------------------------------- /pytorch/main_pytorch.py: -------------------------------------------------------------------------------- 1 | # encoding=utf-8 2 | """ 3 | Created on 21:11 2018/11/8 4 | @author: Jindong Wang 5 | """ 6 | 7 | 8 | from itertools import accumulate 9 | import data_preprocess 10 | import matplotlib.pyplot as plt 11 | import network as net 12 | import numpy as np 13 | import torch 14 | import torch.nn as nn 15 | import torch.optim as optim 16 | import argparse 17 | 18 | 19 | DEVICE = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu') 20 | result = [] 21 | 22 | 23 | def get_args(): 24 | parser = argparse.ArgumentParser() 25 | parser.add_argument('--nepoch', type=int, default=50) 26 | parser.add_argument('--batchsize', type=int, default=128) 27 | parser.add_argument('--lr', type=float, default=.01) 28 | parser.add_argument('--momentum', type=float, default=.9) 29 | parser.add_argument('--data_folder', type=str, default='../') 30 | parser.add_argument('--seed', type=int, default=10) 31 | args = parser.parse_args() 32 | return args 33 | 34 | 35 | def train(model, optimizer, train_loader, test_loader): 36 | criterion = nn.CrossEntropyLoss() 37 | 38 | for e in range(args.nepoch): 39 | model.train() 40 | correct, total_loss = 0, 0 41 | total = 0 42 | for sample, target in train_loader: 43 | sample, target = sample.to( 44 | DEVICE).float(), target.to(DEVICE).long() 45 | sample = sample.view(-1, 9, 1, 128) 46 | output = model(sample) 47 | loss = criterion(output, target) 48 | optimizer.zero_grad() 49 | loss.backward() 50 | optimizer.step() 51 | total_loss += loss.item() 52 | _, predicted = torch.max(output.data, 1) 53 | total += target.size(0) 54 | correct += (predicted == target).sum() 55 | acc_train = float(correct) * 100.0 / len(train_loader.dataset) 56 | 57 | # Testing 58 | acc_test = valid(model, test_loader) 59 | print(f'Epoch: [{e}/{args.nepoch}], loss:{total_loss / len(train_loader):.4f}, train_acc: {acc_train:.2f}, test_acc: {float(correct) * 100 / total:.2f}') 60 | result.append([acc_train, acc_test]) 61 | result_np = np.array(result, dtype=float) 62 | np.savetxt('result.csv', result_np, fmt='%.2f', delimiter=',') 63 | 64 | 65 | def valid(model, test_loader): 66 | model.eval() 67 | with torch.no_grad(): 68 | correct, total = 0, 0 69 | for sample, target in test_loader: 70 | sample, target = sample.to( 71 | DEVICE).float(), target.to(DEVICE).long() 72 | sample = sample.view(-1, 9, 1, 128) 73 | output = model(sample) 74 | _, predicted = torch.max(output.data, 1) 75 | total += target.size(0) 76 | correct += (predicted == target).sum() 77 | acc_test = float(correct) * 100 / total 78 | return acc_test 79 | 80 | 81 | def plot(): 82 | data = np.loadtxt('result.csv', delimiter=',') 83 | plt.figure() 84 | plt.plot(range(1, len(data[:, 0]) + 1), 85 | data[:, 0], color='blue', label='train') 86 | plt.plot(range(1, len(data[:, 1]) + 1), 87 | data[:, 1], color='red', label='test') 88 | plt.legend() 89 | plt.xlabel('Epoch', fontsize=14) 90 | plt.ylabel('Accuracy (%)', fontsize=14) 91 | plt.title('Train and Test Accuracy', fontsize=16) 92 | plt.savefig('plot.png') 93 | 94 | 95 | if __name__ == '__main__': 96 | args = get_args() 97 | torch.manual_seed(args.seed) 98 | train_loader, test_loader = data_preprocess.load( 99 | args.data_folder, batch_size=args.batchsize) 100 | model = net.Network().to(DEVICE) 101 | optimizer = optim.SGD(params=model.parameters( 102 | ), lr=args.lr, momentum=args.momentum) 103 | train(model, optimizer, train_loader, test_loader) 104 | result = np.array(result, dtype=float) 105 | np.savetxt('result.csv', result, fmt='%.2f', delimiter=',') 106 | plot() 107 | -------------------------------------------------------------------------------- /pytorch/network.py: -------------------------------------------------------------------------------- 1 | # encoding=utf-8 2 | """ 3 | Created on 10:41 2018/11/10 4 | @author: Jindong Wang 5 | """ 6 | import torch.nn as nn 7 | import torch.nn.functional as F 8 | 9 | 10 | class Network(nn.Module): 11 | def __init__(self): 12 | super(Network, self).__init__() 13 | self.conv1 = nn.Sequential( 14 | nn.Conv2d(in_channels=9, out_channels=32, kernel_size=(1, 9)), 15 | nn.ReLU(), 16 | nn.MaxPool2d(kernel_size=(1, 2), stride=2) 17 | ) 18 | self.conv2 = nn.Sequential( 19 | nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(1, 9)), 20 | nn.ReLU(), 21 | nn.MaxPool2d(kernel_size=(1, 2), stride=2) 22 | ) 23 | self.fc1 = nn.Sequential( 24 | nn.Linear(in_features=64 * 26, out_features=1000), 25 | nn.ReLU() 26 | ) 27 | self.fc2 = nn.Sequential( 28 | nn.Linear(in_features=1000, out_features=500), 29 | nn.ReLU() 30 | ) 31 | self.fc3 = nn.Sequential( 32 | nn.Linear(in_features=500, out_features=6) 33 | ) 34 | 35 | def forward(self, x): 36 | out = self.conv1(x) 37 | out = self.conv2(out) 38 | out = out.reshape(-1, 64 * 26) 39 | out = self.fc1(out) 40 | out = self.fc2(out) 41 | out = self.fc3(out) 42 | return out 43 | --------------------------------------------------------------------------------