├── .gitignore ├── README.md ├── config ├── anomaly_detection.yaml ├── anomaly_detection_coherence.yaml ├── avenue_anomaly_detection_coherence.yaml ├── ped2_anomaly_detection_coherence.yaml └── shanghaitech_anomaly_detection_coherence.yaml ├── dataset └── anomaly_detection │ └── gt_mat │ ├── avenue.mat │ ├── enter.mat │ ├── enter_original.mat │ ├── exit.mat │ ├── ped1.mat │ └── ped2.mat ├── evaluate.py ├── extract_feature └── extract_feature_twostream │ └── extract_feature.m ├── libs ├── FLAGS.py ├── __init__.py ├── base.py ├── common.py ├── feature_loader_multi_patch.py ├── sista_rnn.py ├── sista_rnn_anomaly_detection.py └── sista_rnn_anomaly_detection_coherence.py ├── requirements.txt ├── run_anomaly_detection.py ├── run_anomaly_detection_coherence.py ├── tools ├── __init__.py ├── ground_truth.py └── init_path.py └── txt ├── avenue_feature_testing.txt ├── avenue_feature_training.txt ├── enter_feature_testing.txt ├── enter_feature_training.txt ├── exit_feature_testing.txt ├── exit_feature_training.txt ├── ped1_feature_testing.txt ├── ped1_feature_training.txt ├── ped2_feature_testing.txt ├── ped2_feature_training.txt ├── shanghaitech_feature_testing.txt └── shanghaitech_feature_training.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.cvs 2 | .pyc 3 | .idea 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # A revisit of sparse coding based anomaly detection in stacked rnn framework 2 | This repo is the official open source of [A revisit of sparse coding based anomaly detection in stacked rnn framework, ICCV 2017] 3 | 4 | It is implemented on tensorflow. Please follow the instructions to run the code. 5 | 6 | ## 1. Installation (Anaconda with python3.6 installation is recommended) 7 | * Install 3rd-package dependencies of python (listed in requirements.txt) 8 | ``` 9 | numpy==1.15.4 10 | matplotlib==2.2.2 11 | scikit_image==0.13.1 12 | six==1.11.0 13 | opencv_python==3.4.3.18 14 | h5py==2.7.1 15 | scipy==1.1.0 16 | tensorflow_gpu==1.11.0 17 | seaborn==0.8.1 18 | skimage==0.0 19 | scikit_learn==0.20.2 20 | tensorflow==1.12.0 21 | PyYAML==3.13 22 | ``` 23 | 24 | ```shell 25 | pip install -r requirements.txt 26 | 27 | pip install tensorflow-gpu==1.11.0 28 | ``` 29 | 30 | ## 2. Download datasets 31 | cd into Data folder of project and run the shell scripts (**ped1.sh, ped2.sh, avenue.sh, shanghaitech.sh**) under the Data folder. 32 | ```shell 33 | cd dataset/anomaly_detection 34 | ``` 35 | Please manually download all datasets [ped2.tar.gz, avenue.tar.gz and shanghaitech.tar.gz](http://101.32.75.151:8181/dataset/) 36 | and tar -xv each tar.gz file. Folders will be like dataset/anomaly_detection/avenue/.... 37 | 38 | ## 3. Extracting feature 39 | ```shell 40 | cd extract_feature 41 | git clone https://github.com/feichtenhofer/twostreamfusion.git 42 | ``` 43 | Please infer the instructions of the twostreamfusion model and download pretrained models. 44 | 45 | ```shell 46 | cd extract_feature/extract_feature_twostream 47 | ``` 48 | 49 | Modidy the root/dataset/res_type/gpu_id in extract_feature.m 50 | 51 | run extract_feature.m using MATLAB 52 | 53 | ## 4. Training 54 | For the ICCV version: 55 | ```shell 56 | python run_anomaly_detection.py --config_file config/anomaly_detection.yaml --mode 0 --gpu 0 57 | ``` 58 | 59 | For the TPAMI version: 60 | ```shell 61 | # training on ped2 62 | python run_anomaly_detection_coherence.py --config_file config/ped2_anomaly_detection_coherence.yaml --mode 0 --gpu 0 63 | 64 | # training on avenue 65 | python run_anomaly_detection_coherence.py --config_file config/avenue_anomaly_detection_coherence.yaml --mode 0 --gpu 0 66 | 67 | # training on shanghaitech 68 | python run_anomaly_detection_coherence.py --config_file config/shanghaitech_anomaly_detection_coherence.yaml --mode 0 --gpu 0 69 | ``` 70 | 71 | ## 5. Testing 72 | For the ICCV version: 73 | ```shell 74 | python run_anomaly_detection.py --config_file config/anomaly_detection.yaml --mode 1 --gpu 0 75 | ``` 76 | 77 | For the TPAMI version: 78 | ```shell 79 | # testing on ped2 80 | python run_anomaly_detection_coherence.py --config_file config/ped2_anomaly_detection_coherence.yaml --mode 1 --gpu 0 81 | 82 | # testing on avenue 83 | python run_anomaly_detection_coherence.py --config_file config/avenue_anomaly_detection_coherence.yaml --mode 1 --gpu 0 84 | 85 | # testing on shanghaitech 86 | python run_anomaly_detection_coherence.py --config_file config/shanghaitech_anomaly_detection_coherence.yaml --mode 1 --gpu 0 87 | 88 | ``` 89 | 90 | ## 6. Evaluation 91 | After running the testing scripts, you also need to run the evaluation.py to calculate AUC 92 | ```shell 93 | python evaluate.py --file=results/ad_coherence/info_avenue_\* --type=compute_auc 94 | ``` 95 | 96 | ## Citation 97 | If you find this useful, please cite our work as follows: 98 | ```code 99 | @article{luo2017revisit, 100 | title={A revisit of sparse coding based anomaly detection in stacked rnn framework}, 101 | author={Luo, Weixin and Liu, Wen and Gao, Shenghua}, 102 | journal={ICCV, Oct}, 103 | volume={1}, 104 | number={2}, 105 | pages={3}, 106 | year={2017} 107 | } 108 | @article{luo2019video, 109 | title={Video Anomaly Detection With Sparse Coding Inspired Deep Neural Networks}, 110 | author={Luo, Weixin and Liu, Wen and Lian, Dongze and Tang, Jinhui and Duan, Lixin and Peng, Xi and Gao, Shenghua}, 111 | journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 112 | year={2019}, 113 | publisher={IEEE} 114 | } 115 | ``` 116 | 117 | 118 | -------------------------------------------------------------------------------- /config/anomaly_detection.yaml: -------------------------------------------------------------------------------- 1 | display: 10 2 | snapshot: 1000 3 | summary: 100 4 | n_hidden: 2048 5 | n_input: 2048 6 | n_output: 2048 7 | gama: 1.0 8 | lambda1: 0.1 # 0.1 0.2 0.3 0.4 9 | lambda2: 0.1 # 0 0.05 0.1 0.15 0.2 10 | K: 3 11 | seed: 2017 12 | learning_rate: 0.0001 13 | time_steps: 10 14 | batch_size: 4 15 | n_iter: 20000 16 | train_videos_txt: txt/{}_feature_train.txt 17 | test_videos_txt: txt/{}_feature_test.txt 18 | train_feature_path: /home/luowx/datasets/anomaly_detection/{}/training/224/features/twostream_{}/ 19 | test_feature_path: /home/luowx/datasets/anomaly_detection/{}/testing/224/features/twostream_{}/motion 20 | dataset: avenue 21 | twostream_model: res152_7x7 22 | save_results_path: results/ad 23 | ckpt_path: ckpt/ad 24 | log_path: log/ad 25 | prefix: info_{}_{}_{}_K_{}_n_hidden_{}_lam1_{}_lam2_{}_T_{} # dataset/twostream_model/model_type/K/n_hidden/lam1/lam2/T 26 | test_loop: 20000 27 | scales: 28 | - 1 29 | - 2 30 | - 4 31 | feature_size: 7.0 32 | model_type: AE 33 | weight_decay: 0.0 34 | gpu: !!str 0 35 | -------------------------------------------------------------------------------- /config/anomaly_detection_coherence.yaml: -------------------------------------------------------------------------------- 1 | display: 10 2 | snapshot: 1000 3 | summary: 100 4 | n_hidden: 2048 5 | n_input: 2048 6 | n_output: 2048 7 | n_gw: 512 8 | gama: 1.0 9 | lambda1: 0.1 10 | lambda2: 0.01 11 | K: 3 12 | seed: 2017 13 | learning_rate: 0.00001 14 | time_steps: 10 15 | batch_size: 4 16 | n_iter: 20000 17 | train_videos_txt: txt/{}_feature_training.txt 18 | test_videos_txt: txt/{}_feature_testing.txt 19 | train_feature_path: ./dataset/anomaly_detection/{}/training/224/features/twostream_{}/ 20 | test_feature_path: ./dataset/anomaly_detection/{}/testing/224/features/twostream_{}/ 21 | dataset: avenue 22 | twostream_model: res152_7x7 23 | save_results_path: results/ad_coherence 24 | ckpt_path: ckpt/ad_coherence 25 | log_path: log/ad_coherence 26 | prefix: info_{}_{}_{}_K_{}_n_hidden_{}_lam1_{}_lam2_{}_T_{} # dataset/twostream_model/model_type/K/n_hidden/lam1/lam2/T 27 | test_loop: 20000 28 | scales: 29 | - 1 30 | - 2 31 | - 4 32 | feature_size: 7.0 33 | model_type: AE 34 | weight_decay: 0.0 35 | gpu: !!str 0 36 | -------------------------------------------------------------------------------- /config/avenue_anomaly_detection_coherence.yaml: -------------------------------------------------------------------------------- 1 | display: 10 2 | snapshot: 1000 3 | summary: 100 4 | n_hidden: 2048 5 | n_input: 2048 6 | n_output: 2048 7 | n_gw: 512 8 | gama: 1.0 9 | lambda1: 0.1 10 | lambda2: 0.01 11 | K: 3 12 | seed: 2017 13 | learning_rate: 0.00001 14 | time_steps: 10 15 | batch_size: 4 16 | n_iter: 20000 17 | train_videos_txt: txt/{}_feature_training.txt 18 | test_videos_txt: txt/{}_feature_testing.txt 19 | train_feature_path: ./dataset/anomaly_detection/{}/training/224/features/twostream_{}/ 20 | test_feature_path: ./dataset/anomaly_detection/{}/testing/224/features/twostream_{}/ 21 | dataset: avenue 22 | twostream_model: res152_7x7 23 | save_results_path: results/ad_coherence 24 | ckpt_path: ckpt/ad_coherence 25 | log_path: log/ad_coherence 26 | prefix: info_{}_{}_{}_K_{}_n_hidden_{}_lam1_{}_lam2_{}_T_{} # dataset/twostream_model/model_type/K/n_hidden/lam1/lam2/T 27 | test_loop: 20000 28 | scales: 29 | - 1 30 | - 2 31 | - 4 32 | feature_size: 7.0 33 | model_type: AE 34 | weight_decay: 0.0 35 | gpu: !!str 0 36 | -------------------------------------------------------------------------------- /config/ped2_anomaly_detection_coherence.yaml: -------------------------------------------------------------------------------- 1 | display: 10 2 | snapshot: 1000 3 | summary: 100 4 | n_hidden: 2048 5 | n_input: 2048 6 | n_output: 2048 7 | n_gw: 512 8 | gama: 1.0 9 | lambda1: 0.1 10 | lambda2: 0.01 11 | K: 3 12 | seed: 2017 13 | learning_rate: 0.00001 14 | time_steps: 10 15 | batch_size: 4 16 | n_iter: 40000 17 | train_videos_txt: txt/{}_feature_training.txt 18 | test_videos_txt: txt/{}_feature_testing.txt 19 | train_feature_path: ./dataset/anomaly_detection/{}/training/224/features/twostream_{}/ 20 | test_feature_path: ./dataset/anomaly_detection/{}/testing/224/features/twostream_{}/ 21 | dataset: ped2 22 | twostream_model: res50_7x7 23 | save_results_path: results/ad_coherence 24 | ckpt_path: ckpt/ad_coherence 25 | log_path: log/ad_coherence 26 | prefix: info_{}_{}_{}_K_{}_n_hidden_{}_lam1_{}_lam2_{}_T_{} # dataset/twostream_model/model_type/K/n_hidden/lam1/lam2/T 27 | test_loop: 40000 28 | scales: 29 | - 1 30 | - 2 31 | - 4 32 | feature_size: 7.0 33 | model_type: AE 34 | weight_decay: 0.0 35 | gpu: !!str 0 36 | -------------------------------------------------------------------------------- /config/shanghaitech_anomaly_detection_coherence.yaml: -------------------------------------------------------------------------------- 1 | display: 10 2 | snapshot: 1000 3 | summary: 100 4 | n_hidden: 2048 5 | n_input: 2048 6 | n_output: 2048 7 | n_gw: 512 8 | gama: 1.0 9 | lambda1: 0.1 10 | lambda2: 0.01 11 | K: 3 12 | seed: 2017 13 | learning_rate: 0.00001 14 | time_steps: 10 15 | batch_size: 4 16 | n_iter: 20000 17 | train_videos_txt: txt/{}_feature_training.txt 18 | test_videos_txt: txt/{}_feature_testing.txt 19 | train_feature_path: ./dataset/anomaly_detection/{}/training/224/features/twostream_{}/ 20 | test_feature_path: ./dataset/anomaly_detection/{}/testing/224/features/twostream_{}/ 21 | dataset: shanghaitech 22 | twostream_model: res152_7x7 23 | save_results_path: results/ad_coherence 24 | ckpt_path: ckpt/ad_coherence 25 | log_path: log/ad_coherence 26 | prefix: info_{}_{}_{}_K_{}_n_hidden_{}_lam1_{}_lam2_{}_T_{} # dataset/twostream_model/model_type/K/n_hidden/lam1/lam2/T 27 | test_loop: 20000 28 | scales: 29 | - 1 30 | - 2 31 | - 4 32 | feature_size: 7.0 33 | model_type: AE 34 | weight_decay: 0.0 35 | gpu: !!str 0 36 | -------------------------------------------------------------------------------- /dataset/anomaly_detection/gt_mat/avenue.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/StevenLiuWen/sRNN_TSC_Anomaly_Detection/8c942cbd7457c5c1f8b89efb8581e1660e7afda8/dataset/anomaly_detection/gt_mat/avenue.mat -------------------------------------------------------------------------------- /dataset/anomaly_detection/gt_mat/enter.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/StevenLiuWen/sRNN_TSC_Anomaly_Detection/8c942cbd7457c5c1f8b89efb8581e1660e7afda8/dataset/anomaly_detection/gt_mat/enter.mat -------------------------------------------------------------------------------- /dataset/anomaly_detection/gt_mat/enter_original.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/StevenLiuWen/sRNN_TSC_Anomaly_Detection/8c942cbd7457c5c1f8b89efb8581e1660e7afda8/dataset/anomaly_detection/gt_mat/enter_original.mat -------------------------------------------------------------------------------- /dataset/anomaly_detection/gt_mat/exit.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/StevenLiuWen/sRNN_TSC_Anomaly_Detection/8c942cbd7457c5c1f8b89efb8581e1660e7afda8/dataset/anomaly_detection/gt_mat/exit.mat -------------------------------------------------------------------------------- /dataset/anomaly_detection/gt_mat/ped1.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/StevenLiuWen/sRNN_TSC_Anomaly_Detection/8c942cbd7457c5c1f8b89efb8581e1660e7afda8/dataset/anomaly_detection/gt_mat/ped1.mat -------------------------------------------------------------------------------- /dataset/anomaly_detection/gt_mat/ped2.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/StevenLiuWen/sRNN_TSC_Anomaly_Detection/8c942cbd7457c5c1f8b89efb8581e1660e7afda8/dataset/anomaly_detection/gt_mat/ped2.mat -------------------------------------------------------------------------------- /evaluate.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import numpy as np 3 | import matplotlib.pyplot as plt 4 | from matplotlib.patches import Rectangle 5 | import argparse 6 | from sklearn import metrics 7 | from six.moves import cPickle 8 | from tools.ground_truth import GroundTruthLoader 9 | import glob 10 | 11 | 12 | class RecordResult(object): 13 | def __init__(self, fpr=None, tpr=None, auc=-np.inf, dataset=None, loss_file=None): 14 | self.fpr = fpr 15 | self.tpr = tpr 16 | self.auc = auc 17 | self.dataset = dataset 18 | self.loss_file = loss_file 19 | 20 | def __lt__(self, other): 21 | return self.auc < other.auc 22 | 23 | def __str__(self): 24 | return 'dataset = {}, loss file = {}, auc = {}'.format(self.dataset, self.loss_file, self.auc) 25 | 26 | 27 | def parser_args(): 28 | parser = argparse.ArgumentParser(description='evaluating the model, computing the roc/auc.') 29 | 30 | parser.add_argument('--file', type=str, help='the path of loss file.') 31 | parser.add_argument('--type', type=str, default='compute_auc', help='the type of evaluation, ' 32 | 'choosing type is: plot_roc, compute_auc, test_func\n, the default type is compute_auc') 33 | return parser.parse_args() 34 | 35 | 36 | def load_loss_gt(loss_file): 37 | with open(loss_file, 'rb') as f: 38 | # results { 39 | # 'dataset': the name of dataset 40 | # 'mse': the mse of each testing videos, 41 | # } 42 | 43 | # mse_records['mse'] is np.array, shape(#videos) 44 | # mse_records[0] is np.array ------> 01.avi 45 | # mse_records[1] is np.array ------> 02.avi 46 | # ...... 47 | # mse_records[n] is np.array ------> xx.avi 48 | 49 | results = cPickle.load(f) 50 | 51 | dataset = results['dataset'] 52 | mse_records = results['mse'] 53 | 54 | num_videos = len(mse_records) 55 | 56 | # load ground truth 57 | gt_loader = GroundTruthLoader() 58 | gt = gt_loader(dataset=dataset) 59 | 60 | assert num_videos == len(gt), 'the number of saved videos does not match the ground truth, {} != {}' \ 61 | .format(num_videos, len(gt)) 62 | 63 | return dataset, mse_records, gt 64 | 65 | 66 | def plot_roc(loss_file): 67 | optimal_results = compute_auc(loss_file) 68 | 69 | # the name of dataset, loss, and ground truth 70 | dataset, mse_records, gt = load_loss_gt(loss_file=optimal_results.loss_file) 71 | 72 | # the number of videos 73 | num_videos = len(mse_records) 74 | 75 | scores = np.array([], dtype=np.float32) 76 | labels = np.array([], dtype=np.int8) 77 | sub_video_length_list = [] 78 | 79 | # video normalization 80 | for i in range(num_videos): 81 | distance = mse_records[i] 82 | 83 | # distance -= distance.min() # distances = (distance - min) / (max - min) 84 | # distance /= distance.max() 85 | # distance = 1 - distance 86 | 87 | distance = (distance - distance.min()) / (distance.max() - distance.min()) 88 | distance = 1 - distance 89 | 90 | # print(distance.max()) 91 | # print(distance.min()) 92 | 93 | scores = np.concatenate((scores, distance), axis=0) 94 | labels = np.concatenate((labels, gt[i]), axis=0) 95 | 96 | sub_video_length_list.append(gt[i].shape[0]) 97 | 98 | fpr, tpr, thresholds = metrics.roc_curve(labels, scores, pos_label=0) 99 | auc = metrics.auc(fpr, tpr) 100 | 101 | # np.savetxt('ped2_scores.txt', scores) 102 | # np.savetxt('ped2_labels.txt', labels) 103 | 104 | # plot the scores 105 | total = scores.shape[0] 106 | index = range(0, total, 100) 107 | # plt.plot(index, scores[index], color='blue') 108 | plt.plot(scores, color='blue') 109 | 110 | # plot the ground truth 111 | i = 0 112 | while i < total: 113 | if labels[i] == 1: 114 | start = i 115 | end = i 116 | while end < total and labels[end] == 1: 117 | end += 1 118 | currentAxis = plt.gca() 119 | currentAxis.add_patch(Rectangle((start, 0.0), end - start, 1.0, color='red', alpha=0.3)) 120 | 121 | i = end 122 | else: 123 | i += 1 124 | 125 | # plot line of different videos 126 | cur_len = 0 127 | for length in sub_video_length_list: 128 | cur_len += length 129 | currentAxis = plt.gca() 130 | currentAxis.add_patch(Rectangle((cur_len, 0.0), 1, 1.0, color='green', alpha=1)) 131 | 132 | plt.annotate('AUC = ' + str(auc), 133 | xy=(total / 2, 0.5), xycoords='data', 134 | xytext=(-90, -50), textcoords='offset points', fontsize=15, 135 | arrowprops=dict(arrowstyle="->", connectionstyle="arc3,rad=.2")) 136 | 137 | plt.title(dataset) 138 | plt.ylim(0) 139 | plt.xlabel('#Frames') 140 | plt.ylabel('Scores') 141 | plt.show() 142 | 143 | return optimal_results 144 | 145 | 146 | def compute_auc(loss_file): 147 | # if not os.path.isdir(loss_file): 148 | # loss_file_list = [loss_file] 149 | # else: 150 | # loss_file_list = os.listdir(loss_file) 151 | loss_file_list = glob.glob(loss_file) 152 | # loss_file_list = [os.path.join(loss_file, sub_loss_file) for sub_loss_file in loss_file_list] 153 | print(loss_file_list) 154 | optimal_results = RecordResult() 155 | for sub_loss_file in loss_file_list: 156 | # the name of dataset, loss, and ground truth 157 | dataset, mse_records, gt = load_loss_gt(loss_file=sub_loss_file) 158 | 159 | # the number of videos 160 | num_videos = len(mse_records) 161 | 162 | scores = np.array([], dtype=np.float32) 163 | labels = np.array([], dtype=np.int8) 164 | # video normalization 165 | for i in range(num_videos): 166 | distance = mse_records[i] 167 | 168 | distance -= distance.min() # distances = (distance - min) / (max - min) 169 | distance /= distance.max() 170 | distance = 1 - distance 171 | # distance = distance[:, 0] 172 | 173 | print(scores.shape, distance.shape) 174 | scores = np.concatenate((scores, distance), axis=0) 175 | labels = np.concatenate((labels, gt[i]), axis=0) 176 | 177 | if np.isnan(scores).any(): 178 | continue 179 | fpr, tpr, thresholds = metrics.roc_curve(labels, scores, pos_label=0) 180 | auc = metrics.auc(fpr, tpr) 181 | # eer = brentq(lambda x : 1. - x - interp1d(fpr, tpr)(x), 0., 1.) 182 | # thresh = interp1d(fpr, thresholds)(eer) 183 | results = RecordResult(fpr, tpr, auc, dataset, sub_loss_file) 184 | 185 | if optimal_results < results: 186 | optimal_results = results 187 | 188 | print(results) 189 | # with open(sub_loss_file+'.csv', 'w') as f: 190 | # writer = csv.writer(f) 191 | # writer.writerow(fpr) 192 | # writer.writerow(tpr) 193 | 194 | return optimal_results 195 | 196 | 197 | def compute_eer(loss_file): 198 | pass 199 | 200 | 201 | def test_func(*args): 202 | # simulate testing on CUHK AVENUE dataset 203 | dataset = GroundTruthLoader.AVENUE 204 | 205 | # load the ground truth 206 | gt_loader = GroundTruthLoader() 207 | gt = gt_loader(dataset=dataset) 208 | 209 | num_videos = len(gt) 210 | 211 | simulated_results = { 212 | 'dataset': dataset, 213 | 'mse': [] 214 | } 215 | 216 | simulated_mse = [] 217 | for i in range(num_videos): 218 | sub_video_length = gt[i].shape[0] 219 | simulated_mse.append(np.random.random(size=sub_video_length)) 220 | 221 | simulated_results['mse'] = simulated_mse 222 | 223 | # writing to file, 'generated_loss.bin' 224 | with open('generated_loss.bin', 'wb') as save_file: 225 | cPickle.dump(simulated_results, save_file, cPickle.HIGHEST_PROTOCOL) 226 | 227 | print(save_file.name) 228 | auc, dataset = plot_roc(save_file.name) 229 | 230 | print('optimal! dataset = {}, auc = {}'.format(dataset, auc)) 231 | 232 | 233 | eval_type_function = { 234 | 'compute_auc': compute_auc, 235 | 'compute_eer': compute_eer, 236 | 'plot_auc': plot_roc 237 | } 238 | 239 | 240 | def evaluate(eval_type, save_file): 241 | assert eval_type in eval_type_function, 'there is no type of evaluation {}, please check {}' \ 242 | .format(eval_type, eval_type_function.keys()) 243 | 244 | eval_func = eval_type_function[eval_type] 245 | 246 | optimal_results = eval_func(save_file) 247 | 248 | print('dataset = {}, auc = {}'.format(optimal_results.dataset, optimal_results.auc)) 249 | 250 | 251 | if __name__ == '__main__': 252 | args = parser_args() 253 | 254 | eval_type = args.type 255 | save_file = args.file 256 | print(save_file) 257 | if eval_type == 'test_func': 258 | test_func() 259 | else: 260 | evaluate(eval_type, save_file) 261 | -------------------------------------------------------------------------------- /extract_feature/extract_feature_twostream/extract_feature.m: -------------------------------------------------------------------------------- 1 | clc; 2 | clear; 3 | addpath('../twostreamfusion/matconvnet/matlab'); 4 | addpath('../twostreamfusion/models'); 5 | addpath('../+dagnn/@DagNN') 6 | run vl_setupnn 7 | 8 | %%%%%%%%%%%%%%%%%%%%%%%% setup here %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 9 | % only need to setup these 4 parameters 10 | root = '../../'; 11 | dataset = 'shanghaitech'; 12 | res_type = '152'; 13 | gpu_id = 3; % id > 0 14 | 15 | 16 | load_rgb_path = [root '/dataset/anomaly_detection/' dataset]; 17 | %load_optical_flow_path = '/home/luowx/datasets/avenue/testing/224/optical_flow/'; 18 | save_path = [root '/dataset/anomaly_detection/' dataset]; 19 | extract_layer = 'res5cx'; 20 | feature_dimension = 2048; 21 | nstack = 10; 22 | optical_flow_mean = 128; 23 | clip_upper = 20; 24 | clip_bottom = -20; 25 | clip = true; 26 | 27 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 28 | 29 | % load the pre-trained CNN 30 | net_rgb = load(['ucf101-img-resnet-' res_type '-split1.mat']); 31 | net_rgb = dagnn.DagNN.loadobj(net_rgb.net) ; 32 | net_rgb.mode = 'test' ; 33 | net_rgb.conserveMemory = false; % note: important 34 | 35 | % net_optical_flow = load('ucf101-TVL1flow-resnet-50-split1.mat'); 36 | % net_optical_flow = dagnn.DagNN.loadobj(net_optical_flow.net) ; 37 | % net_optical_flow.mode = 'test' ; 38 | % net_optical_flow.conserveMemory = false; % note: important 39 | 40 | %set gpu 41 | gpuDevice(gpu_id); 42 | move(net_rgb,'gpu'); 43 | %move(net_optical_flow,'gpu'); 44 | 45 | % preprocess rgb 46 | phases = {'/training', '/testing'}; 47 | for phase = phases 48 | phase = char(phase); 49 | load_rgb_path_new = [load_rgb_path phase '/frames/']; 50 | save_path_new = [save_path, phase '/224/features/twostream_res' res_type '_7x7']; 51 | 52 | dir_video = dir(load_rgb_path_new); 53 | dir_video = dir_video(3:end); 54 | for i = 1:length(dir_video) 55 | dir_image = dir([load_rgb_path_new dir_video(i).name]); 56 | dir_image = dir_image(3:end); 57 | num_frames = length(dir_image); 58 | % create hdf5 59 | h5create([save_path_new '/' dir_video(i).name '.h5'],'/rgb',[feature_dimension, 7, 7, num_frames],'Datatype','single'); 60 | h5create([save_path_new '/' dir_video(i).name '.h5'],'/length',1,'Datatype','int32'); 61 | features = zeros(num_frames, 7, 7, feature_dimension, 'single'); 62 | for j = 1:num_frames 63 | im = imread([load_rgb_path_new dir_video(i).name '/' dir_image(j).name]); 64 | im_ = single(im) ; 65 | im_ = imresize(im_, net_rgb.meta.normalization.imageSize(1:2)) ; 66 | im_ = bsxfun(@minus, im_, net_rgb.meta.normalization.averageImage) ; 67 | net_rgb.eval({'input', im_}) ; 68 | feature = net_rgb.vars(net_rgb.getVarIndex(extract_layer)).value; 69 | features(j, :, :, :) = gather(feature); 70 | disp(['rgb : video ' num2str(i) ' / ' num2str(length(dir_video)) ' : frame ' num2str(j) ' / ' num2str(num_frames)]); 71 | end 72 | features = permute(features,[4,3,2,1]); 73 | h5write([save_path_new '/' dir_video(i).name '.h5'],'/rgb',features); 74 | h5write([save_path_new '/' dir_video(i).name '.h5'],'/length',int32(num_frames)); 75 | end 76 | end 77 | 78 | % % preprocess optical flow 79 | % dir_video = dir(load_optical_flow_path); 80 | % dir_video = dir_video(3:end); 81 | % for i = 1:length(dir_video) 82 | % optical_flow = h5read([load_optical_flow_path dir_video(i).name],'/optical_flow'); 83 | % s = size(optical_flow); 84 | % num_frames = s(4) + 1; 85 | % % create hdf5 86 | % h5create([save_path '/' dataset '_' phrase '_' dir_video(i).name],'/optical_flow',[num_frames, feature_dimension],'Datatype','single'); 87 | % features = zeros(num_frames, feature_dimension, 'single'); 88 | % %normalize optical flow 89 | % for j = 1:num_frames - 1 90 | % for k = 1:2 91 | % if clip 92 | % optical_flow(:,:,k,j) = max(optical_flow(:,:,k,j), clip_bottom); 93 | % optical_flow(:,:,k,j) = min(optical_flow(:,:,k,j), clip_upper); 94 | % end 95 | % temp = optical_flow(:,:,k,j); 96 | % min_value = min(temp(:)); 97 | % max_value = max(temp(:)); 98 | % optical_flow(:,:,k,j) = bsxfun(@minus, optical_flow(:,:,k,j), min_value); 99 | % optical_flow(:,:,k,j) = bsxfun(@rdivide, optical_flow(:,:,k,j), max_value - min_value); 100 | % optical_flow(:,:,k,j) = optical_flow(:,:,k,j) * 255; 101 | % end 102 | % end 103 | % for j = 1:num_frames - nstack 104 | % im_ = optical_flow(:, :, :, j:j + nstack - 1); 105 | % im_ = permute(im_, [2, 1, 3, 4]); 106 | % s = size(im_); 107 | % im_ = reshape(im_, [s(1), s(2), s(3) * s(4)]); 108 | % im_ = imresize(im_, net_optical_flow.meta.normalization.imageSize(1:2)) ; 109 | % im_ = bsxfun(@minus, im_, optical_flow_mean) ; 110 | % net_optical_flow.eval({'input', im_}) ; 111 | % feature = net_optical_flow.vars(net_optical_flow.getVarIndex(extract_layer)).value; 112 | % features(j, :) = gather(feature(:)); 113 | % disp(['optical_flow : video ' num2str(i) ' / ' num2str(length(dir_video)) ' : frame ' num2str(j) ' / ' num2str(num_frames)]); 114 | % end 115 | % for j = num_frames - nstack + 1:num_frames 116 | % features(j,:) = features(num_frames - nstack,:); 117 | % end 118 | % 119 | % h5write([save_path '/' dataset '_' phrase '_' dir_video(i).name],'/optical_flow',features); 120 | % end 121 | -------------------------------------------------------------------------------- /libs/FLAGS.py: -------------------------------------------------------------------------------- 1 | TSC = 'TSC' 2 | AE = 'AE' 3 | 4 | AVENUE = 'avenue' 5 | PED1 = 'ped1' 6 | PED2 = 'ped2' 7 | EXIT = 'exit' 8 | ENTRANCE = 'enter' 9 | MOVINGMNIST='moving_mnist' 10 | SHANGHAITECH='shanghaitech' -------------------------------------------------------------------------------- /libs/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/StevenLiuWen/sRNN_TSC_Anomaly_Detection/8c942cbd7457c5c1f8b89efb8581e1660e7afda8/libs/__init__.py -------------------------------------------------------------------------------- /libs/base.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | 3 | class base(object): 4 | def __init__(self, input, label, config): 5 | self.endpoints = {} 6 | self.all_parameters = {} 7 | self.label = label 8 | self.config = config 9 | self.output = None 10 | self.optimized_parameters = None 11 | self.losses = None 12 | self.regularization_loss = None 13 | self.total_loss = None 14 | self.input = input 15 | self.summary_writer = None 16 | self.all_summary = None 17 | self._setup() 18 | 19 | def _head(self, input): 20 | ''' 21 | :param input: the input tensor 22 | :return: head_output: an output tensor 23 | head_parameters: a list of parameters 24 | ''' 25 | raise Exception('Not Implemented') 26 | 27 | def _body(self, input): 28 | ''' 29 | :param input: the input tensor 30 | :return body_output: an output tensor 31 | :return body_parameters: a list of parameters 32 | ''' 33 | raise Exception('Not Implemented') 34 | 35 | def _tail(self, input): 36 | ''' 37 | :param input: the input tensor 38 | :return tail_output: an output tensor 39 | :return tail_parameters: a list of parameters 40 | ''' 41 | raise Exception('Not Implemented') 42 | 43 | def _get_losses(self, pred, label): 44 | """ 45 | :param pred: the prediction tensor 46 | :param label: the gound truth tensor 47 | :return losses: a list of losses 48 | """ 49 | raise Exception('Not Implemented') 50 | 51 | def _get_testing_loss(self, pred, label): 52 | """ 53 | :param pred: the prediction tensor 54 | :param label: the gound truth tensor 55 | :return loss: a loss tensor 56 | """ 57 | return None 58 | 59 | def _get_regularization_loss(self, optimized_parameters): 60 | """ 61 | :param optimized_parameters: parameters that should be optimized 62 | :return regularization_loss: a regularization loss of optimized parameters 63 | """ 64 | regularization_loss = [] 65 | for p in optimized_parameters: 66 | regularization_loss.append(tf.nn.l2_loss(p)) 67 | regularization_loss = tf.add_n(regularization_loss) * self.config['weight_decay'] 68 | return regularization_loss 69 | 70 | def _get_optimized_parameters(self, all_parameters): 71 | """ 72 | :param all_parameters: all parameters 73 | :return optimized_parameters: a list of parameters that should be optimized 74 | """ 75 | raise Exception('Not Implemented') 76 | 77 | def _get_summaries(self, losses, statistics): 78 | """ 79 | define some summaries using tf.summary.scalar() 80 | """ 81 | for name, loss in losses.items(): 82 | tf.summary.scalar(name, loss) 83 | 84 | for name, statistic in statistics.items(): 85 | tf.summary.scalar(name, statistic) 86 | 87 | def _get_statistics(self): 88 | """ 89 | :return: a list of some statistic tensors which can be visualized 90 | """ 91 | raise Exception('Not Implemented') 92 | 93 | def _setup(self): 94 | # setup the whole network 95 | head_output, head_parameters = self._head(self.input) 96 | body_output, body_parameters = self._body(head_output) 97 | tail_output, tail_parameters = self._tail(body_output) 98 | self.endpoints.update({'head_output': head_output, 99 | 'body_output': body_output, 100 | 'tail_output': tail_output}) 101 | 102 | self.all_parameters.update({'head_parameters': head_parameters, 103 | 'body_parameters': body_parameters, 104 | 'tail_parameters': tail_parameters}) 105 | self.optimized_parameters = self._get_optimized_parameters(self.all_parameters) 106 | 107 | # setup losses 108 | self.regularization_loss = self._get_regularization_loss(self.optimized_parameters) 109 | self.losses = self._get_losses(tail_output, self.label) 110 | self.total_loss = tf.add_n([self.regularization_loss] + list(self.losses.values())) 111 | self.testing_loss = self._get_testing_loss(tail_output, self.label) 112 | 113 | # setup statistics 114 | self.statistics = self._get_statistics() 115 | 116 | # setup summary 117 | self._get_summaries(self.losses, self.statistics) 118 | self.summaries = tf.summary.merge_all() 119 | self.summary_writer = tf.summary.FileWriter(self.config['log_path']) 120 | 121 | def train(self): 122 | raise Exception('Not Implemented') 123 | 124 | def test(self): 125 | raise Exception('Not Implemented') 126 | -------------------------------------------------------------------------------- /libs/common.py: -------------------------------------------------------------------------------- 1 | import os 2 | import tensorflow as tf 3 | 4 | def checkdir(path): 5 | if not os.path.exists(path): 6 | os.makedirs(path) 7 | 8 | def checkrank(tensor, ranks): 9 | for rank in ranks: 10 | if tf.rank(tensor) != rank: 11 | raise Exception('the rank of tensor {} is not equal to anyone of {}'.format(tensor, rank)) -------------------------------------------------------------------------------- /libs/feature_loader_multi_patch.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from concurrent import futures 3 | from threading import Thread 4 | import random 5 | import h5py 6 | import sys 7 | import os 8 | 9 | pool_size = 4 10 | scales = [1, 2, 4] 11 | total_patches = 21 12 | feature_size = 7.0 13 | 14 | clip_length = 4 15 | clip_feature = False 16 | 17 | 18 | def readHDF5(video_idx, random_idx, key): 19 | with h5py.File(video_idx, 'r') as f: 20 | if clip_feature: 21 | batch_features = np.mean(f[key][:, :, :, max(0, random_idx - clip_length // 2):min(f[key].shape[3], 22 | random_idx + clip_length // 2)], 23 | axis=3) 24 | else: 25 | batch_features = f[key][random_idx] 26 | multi_patch_features = np.zeros([total_patches, batch_features.shape[2]], dtype=np.float32) 27 | t = 0 28 | for s in scales: 29 | step = int(np.ceil(feature_size / s)) 30 | for i in range(0, int(feature_size), step): 31 | for j in range(0, int(feature_size), step): 32 | temp = batch_features[i:min(i + step, int(feature_size)), j:min(j + step, int(feature_size))] 33 | if temp.ndim == 3: 34 | multi_patch_features[t] = np.mean(np.mean(temp, axis=0), axis=0) 35 | elif temp.ndim == 2: 36 | multi_patch_features[t] = np.mean(temp, axis=0) 37 | t = t + 1 38 | 39 | return multi_patch_features 40 | 41 | 42 | class HDF5Reader(object): 43 | def __init__(self): 44 | pass 45 | 46 | def __call__(self, video_idx, random_idx, key): 47 | return readHDF5(video_idx, random_idx, key) 48 | 49 | 50 | def advance_batch(result, sequence_generator, HDF5_reader, pool): 51 | random_idx, video_idx = sequence_generator() 52 | # debug 53 | # tmp1 = image_processor(images_path[0]) 54 | # tmp2 = HDF5_reader(video_idx[0], random_idx[0]) 55 | 56 | # result['rgb'] = HDF5_reader(video_idx[0], random_idx[0], 'rgb') 57 | # result['optical_flow'] = HDF5_reader(video_idx[0], random_idx[0], 'optical_flow') 58 | 59 | rgb_keys = ['rgb'] * len(video_idx) 60 | # optical_flow_keys = ['optical_flow'] * len(video_idx) 61 | rgb_feature = pool.map(HDF5_reader, video_idx, random_idx, rgb_keys) 62 | # optical_flow_feature = pool.map(HDF5_reader, video_idx, random_idx, optical_flow_keys) 63 | 64 | result['rgb'] = np.array(list(rgb_feature)) 65 | # result['optical_flow'] = np.array(list(optical_flow_feature)) 66 | 67 | 68 | class BatchAdvancer(): 69 | def __init__(self, result, sequence_generator, HDF5_reader, pool): 70 | self.sequence_generator = sequence_generator 71 | self.HDF5_reader = HDF5_reader 72 | self.pool = pool 73 | self.result = result 74 | 75 | def __call__(self): 76 | advance_batch(self.result, self.sequence_generator, self.HDF5_reader, self.pool) 77 | 78 | 79 | class SequenceGeneratorVideo(object): 80 | def __init__(self, batch_size, clip_length, num_videos, video_dict, video_order): 81 | self.batch_size = batch_size 82 | self.clip_length = clip_length 83 | self.num_videos = num_videos 84 | self.N = self.clip_length * self.batch_size 85 | self.video_dict = video_dict 86 | self.video_order = video_order 87 | self.idx = 0 88 | 89 | def __call__(self): 90 | video_idx = [None] * self.batch_size * self.clip_length 91 | random_idx = [None] * self.batch_size * self.clip_length 92 | 93 | for i in range(0, self.batch_size): 94 | if self.idx >= self.num_videos: 95 | random.shuffle(self.video_order) 96 | self.idx = 0 97 | key = self.video_order[self.idx] 98 | random_start = int(random.random() * (self.video_dict[key]['num_frames'] - self.clip_length)) 99 | k = 0 100 | for j in range(random_start, random_start + self.clip_length): 101 | video_idx[k * self.batch_size + i] = self.video_dict[key]['feature_hdf5_path'] 102 | random_idx[k * self.batch_size + i] = j 103 | k += 1 104 | 105 | self.idx += 1 106 | 107 | return random_idx, video_idx 108 | 109 | 110 | class FeatureLoader(object): 111 | def __init__(self, hdf5_path, videos_txt, batch_size, clip_length): 112 | self.hdf5_path = hdf5_path 113 | self.videos_txt = videos_txt 114 | self.batch_size = batch_size 115 | 116 | self.clip_length = clip_length 117 | self.setup() 118 | 119 | def setup(self): 120 | random.seed(2017) 121 | f = open(self.videos_txt, 'r') 122 | f_lines = f.readlines() 123 | f.close() 124 | 125 | # f_lines: dataset_train_01.h5 126 | # dataset_train_02.h5 127 | # xxx 128 | # dataset_train_ab.h5 129 | 130 | # dataset_train_01.h5 131 | # 'video_length': length 132 | # 'rgb': [] 133 | # 'optical_flow': [] 134 | 135 | video_dict = {} 136 | self.video_order = [] 137 | 138 | for idx, video in enumerate(f_lines): 139 | video = video.strip('\n') 140 | print(video) 141 | video_dict[video] = {} 142 | self.video_order.append(video) 143 | 144 | # dataset, video_name = video.split('.')[0].split('_train') 145 | feature_hdf5_path = os.path.join(self.hdf5_path, video) 146 | print(feature_hdf5_path) 147 | with h5py.File(feature_hdf5_path, 'r') as f: 148 | video_dict[video]['num_frames'] = f['rgb'].shape[0] 149 | video_dict[video]['feature_hdf5_path'] = feature_hdf5_path 150 | 151 | self.video_dict = video_dict 152 | self.num_videos = len(video_dict.keys()) 153 | 154 | self.thread_result = {} 155 | self.thread = None 156 | 157 | self.sequence_generator = SequenceGeneratorVideo(self.batch_size, self.clip_length, self.num_videos, 158 | self.video_dict, self.video_order) 159 | 160 | self.HDF5_reader = HDF5Reader() 161 | self.pool = futures.ProcessPoolExecutor(max_workers=pool_size) 162 | # self.pool = Pool(processes=pool_size) 163 | self.batch_advancer = BatchAdvancer(self.thread_result, self.sequence_generator, self.HDF5_reader, self.pool) 164 | # pre-load a batch 165 | self.dispatch_worker() 166 | self.join_worker() 167 | 168 | def load_batch(self): 169 | if self.thread is not None: 170 | self.join_worker() 171 | self.dispatch_worker() 172 | return self.thread_result 173 | 174 | def dispatch_worker(self): 175 | assert self.thread is None 176 | self.thread = Thread(target=self.batch_advancer) 177 | self.thread.start() 178 | 179 | def join_worker(self): 180 | assert self.thread is not None 181 | self.thread.join() 182 | self.thread = None 183 | 184 | 185 | # demo 186 | if __name__ == "__main__": 187 | 188 | # python libs/feature_loader.py 1 189 | # testing on simulated data 190 | if len(sys.argv) > 1: 191 | print('generating txt...') 192 | 193 | # dataset 194 | dataset = 'shanghaitech' 195 | # root path 196 | root = '/p300' 197 | # 152 or 50 198 | res_type = '152' 199 | 200 | for train_test in ['training', 'testing']: 201 | # feature path 202 | path = os.path.join(root, 'dataset/anomaly_detection', dataset, train_test, 203 | '224/features/twostream_res' + res_type + '_7x7') 204 | files = os.listdir(path) 205 | files.sort() 206 | name_videos = '' 207 | for file in files: 208 | name_videos += file + '\n' 209 | 210 | with open('../txt/' + dataset + '_feature_' + train_test + '.txt', 'w') as f: 211 | f.write(name_videos) 212 | 213 | # test demo 214 | else: 215 | feature_path = '/root/dataset/anomaly_detection/avenue/training/224/features/twostream_res152_7x7' 216 | video_txt = '../txt/avenue_feature_train.txt' 217 | 218 | feature_loader = FeatureLoader(feature_path, video_txt, batch_size=10, clip_length=4) 219 | 220 | for i in range(1000): 221 | batch = feature_loader.load_batch() 222 | 223 | rgb_feature = batch['rgb'] 224 | 225 | print('rgb feature = {}'.format(rgb_feature.shape)) 226 | # print('optical flow feature = {}'.format(optical_flow_feature.shape)) 227 | -------------------------------------------------------------------------------- /libs/sista_rnn.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | from tensorflow.contrib.framework import arg_scope 3 | from tensorflow.contrib.framework.python.ops.arg_scope import add_arg_scope 4 | import numpy as np 5 | from collections import OrderedDict 6 | 7 | 8 | # original tf.matmul is without an arg_scope decorator 9 | @add_arg_scope 10 | def matmul(*args, **kwargs): 11 | return tf.matmul(*args, **kwargs) 12 | 13 | 14 | def dim_reduction(x, w): 15 | _shape = x.get_shape().as_list() 16 | x = tf.reshape(x, [-1, _shape[-1]]) 17 | with arg_scope([matmul], transpose_b=True): 18 | x = matmul(x, w) 19 | x = tf.reshape(x, _shape[:-1] + [w.get_shape().as_list()[0]]) 20 | return x 21 | 22 | 23 | class sista_rnn(object): 24 | def __init__(self, pre_input, now_input, n_hidden, K, gama, lambda1, lambda2, A_initializer, gw_initializer=None): 25 | ''' 26 | :param pre_input: input tensor with shape [1, batch_size, n_input] 27 | :param now_input: input tensor with shape [time_steps, batch_size, n_input] 28 | :param n_hidden: length of hidden state in RNN models 29 | :param K: length of iterations in the ISTA algorithm 30 | :param gama: learning rate in the ISTA algorithm 31 | :param lambda1: weight of the sparse term 32 | :param lambda2: weight of the coherence term 33 | :param A_initializer: A initializer 34 | 35 | ''' 36 | self.pre_input = pre_input 37 | self.now_input = now_input 38 | input_shape = now_input.get_shape().as_list() 39 | self.time_steps, self.batch_size, self.n_input = input_shape 40 | self.n_hidden = n_hidden 41 | self.K = K 42 | self.gama = gama 43 | self.lambda1 = lambda1 44 | self.lambda2 = lambda2 45 | self.A_initializer = A_initializer 46 | self.gw_initializer = gw_initializer 47 | 48 | def __get_variable(self, name, shape, initializer=tf.contrib.layers.xavier_initializer()): 49 | return tf.get_variable(name=name, shape=shape, dtype=tf.float32, initializer=initializer) 50 | 51 | def __soft(self, x, b): 52 | return tf.sign(x) * tf.nn.relu(tf.abs(x) - b) 53 | 54 | def forward(self): 55 | ''' 56 | :return: the hidden state with shape: [time_step, batch_size, K * n_hidden], 57 | parameters: A dict with A, gama, lambda1, lambda2, h_0 58 | ''' 59 | # define variables and constants in the sista_rnn model 60 | with tf.variable_scope('sista_rnn_trainable_variables'): 61 | A = self.__get_variable('A', None, self.A_initializer) 62 | gama = self.__get_variable('gama', None, tf.constant(self.gama, tf.float32)) 63 | lambda1 = self.__get_variable('lambda1', None, tf.constant(self.lambda1, tf.float32)) 64 | lambda2 = self.__get_variable('lambda2', None, tf.constant(self.lambda2, tf.float32)) 65 | h_0 = self.__get_variable('h_0', [self.batch_size, self.n_hidden], tf.zeros_initializer) 66 | parameters = OrderedDict([('A', A), 67 | ('gama', gama), 68 | ('lambda1', lambda1), 69 | ('lambda2', lambda2), 70 | ('h_0', h_0)]) 71 | epsilon = tf.constant(100, dtype=tf.float32) 72 | At = tf.matrix_transpose(A) 73 | AtA = tf.matmul(At, A) 74 | I = tf.eye(self.n_hidden, self.n_hidden) 75 | one = tf.constant(1.0, dtype=tf.float32) 76 | 77 | # initialize V 78 | V = one / gama * At 79 | 80 | # initialize W_1 81 | W_1 = I - lambda2 / gama * AtA 82 | h_0_all_layers = tf.tile(h_0, [1, self.K]) 83 | h_t_1_all_layers = h_0_all_layers 84 | 85 | # padding for the first time input 86 | if self.pre_input is None: 87 | self.input = tf.concat([tf.zeros([1, self.batch_size, self.n_input], tf.float32), self.now_input], axis=0) 88 | else: 89 | self.input = tf.concat([self.pre_input, self.now_input], axis=0) 90 | 91 | # sequential ISTA mapping to RNN 92 | with arg_scope([matmul], transpose_b=True): 93 | h = [] 94 | for t in range(self.time_steps): 95 | h_t_1_last_layer = h_t_1_all_layers[:, (self.K - 1) * self.n_hidden:] 96 | delta = tf.tile( 97 | tf.exp(-tf.norm(self.input[t] - self.input[t + 1], axis=1, keep_dims=True) ** 2 / epsilon), 98 | [1, self.n_hidden]) 99 | h_t_kth_layer = self.__soft(matmul(h_t_1_last_layer, W_1) + matmul(self.input[t + 1], V), 100 | lambda1 / gama) 101 | h_t_all_layers = h_t_kth_layer 102 | 103 | for k in range(1, self.K): 104 | W_k = lambda2 / gama * I 105 | U_k = I - one / gama * AtA 106 | h_t_kth_layer = self.__soft(tf.matmul(h_t_1_last_layer, W_k) * delta 107 | + matmul(h_t_kth_layer, U_k) \ 108 | - matmul(h_t_kth_layer, W_k) * delta 109 | + matmul(self.input[t + 1], V), lambda1 / gama) 110 | h_t_all_layers = tf.concat([h_t_all_layers, h_t_kth_layer], axis=1) 111 | h.append(h_t_all_layers) 112 | h_t_1_all_layers = h_t_all_layers 113 | h = tf.stack(h) 114 | 115 | return h, parameters 116 | 117 | def forward_As(self): 118 | ''' 119 | :return: the hidden state with shape: [time_step, batch_size, K * n_hidden], 120 | parameters: A dict with A, gama, lambda1, lambda2, h_0 121 | ''' 122 | # define variables and constants in the sista_rnn model 123 | As = [] 124 | with tf.variable_scope('sista_rnn_trainable_variables'): 125 | for i in range(self.K): 126 | As.append(self.__get_variable('A_{}'.format(i + 1), None, self.A_initializer)) 127 | gama = self.__get_variable('gama', None, tf.constant(self.gama, tf.float32)) 128 | lambda1 = self.__get_variable('lambda1', None, tf.constant(self.lambda1, tf.float32)) 129 | lambda2 = self.__get_variable('lambda2', None, tf.constant(self.lambda2, tf.float32)) 130 | h_0 = self.__get_variable('h_0', [self.batch_size, self.n_hidden], tf.zeros_initializer) 131 | parameters = OrderedDict([('A_{}'.format(i + 1), As[i]) for i in range(self.K)] + [ 132 | ('gama', gama), 133 | ('lambda1', lambda1), 134 | ('lambda2', lambda2), 135 | ('h_0', h_0)]) 136 | epsilon = tf.constant(100, dtype=tf.float32) 137 | Ats = [tf.matrix_transpose(As[i]) for i in range(self.K)] 138 | AtAs = [tf.matmul(Ats[i], As[i]) for i in range(self.K)] 139 | I = tf.eye(self.n_hidden, self.n_hidden) 140 | one = tf.constant(1.0, dtype=tf.float32) 141 | 142 | # initialize V 143 | V = one / gama * Ats[0] 144 | 145 | # initialize W_1 146 | W_1 = I - lambda2 / gama * AtAs[0] 147 | 148 | h_0_all_layers = tf.tile(h_0, [1, self.K]) 149 | h_t_1_all_layers = h_0_all_layers 150 | 151 | # padding for the first time input 152 | if self.pre_input is None: 153 | self.input = tf.concat([tf.zeros([1, self.batch_size, self.n_input], tf.float32), self.now_input], axis=0) 154 | else: 155 | self.input = tf.concat([self.pre_input, self.now_input], axis=0) 156 | # sequential ISTA mapping to RNN 157 | with arg_scope([matmul], transpose_b=True): 158 | h = [] 159 | for t in range(self.time_steps): 160 | h_t_1_last_layer = h_t_1_all_layers[:, (self.K - 1) * self.n_hidden:] 161 | delta = tf.tile( 162 | tf.exp(-tf.norm(self.input[t] - self.input[t + 1], axis=1, keep_dims=True) ** 2 / epsilon), 163 | [1, self.n_hidden]) 164 | h_t_kth_layer = self.__soft(matmul(h_t_1_last_layer, W_1) + matmul(self.input[t + 1], V), 165 | lambda1 / gama) 166 | h_t_all_layers = h_t_kth_layer 167 | 168 | for k in range(1, self.K): 169 | # initialize V 170 | V = one / gama * Ats[k] 171 | # initialize W_1 172 | W_1 = I - lambda2 / gama * AtAs[k] 173 | 174 | W_k = lambda2 / gama * I 175 | U_k = I - one / gama * AtAs[k] 176 | h_t_kth_layer = self.__soft(tf.matmul(h_t_1_last_layer, W_k) * delta 177 | + matmul(h_t_kth_layer, U_k) \ 178 | - matmul(h_t_kth_layer, W_k) * delta 179 | + matmul(self.input[t + 1], V), lambda1 / gama) 180 | h_t_all_layers = tf.concat([h_t_all_layers, h_t_kth_layer], axis=1) 181 | h.append(h_t_all_layers) 182 | h_t_1_all_layers = h_t_all_layers 183 | h = tf.stack(h) 184 | 185 | return h, parameters 186 | 187 | def forward_coherence(self): 188 | ''' 189 | :return: the hidden state with shape: [time_step, batch_size, K * n_hidden], 190 | parameters: A dict with A, gama, lambda1, lambda2, h_0 191 | ''' 192 | # define variables and constants in the sista_rnn model 193 | with tf.variable_scope('sista_rnn_trainable_variables'): 194 | A = self.__get_variable('A', None, self.A_initializer) 195 | gw = self.__get_variable('gw', None, self.gw_initializer) 196 | gama = self.__get_variable('gama', None, tf.constant(self.gama, tf.float32)) 197 | lambda1 = self.__get_variable('lambda1', None, tf.constant(self.lambda1, tf.float32)) 198 | lambda2 = self.__get_variable('lambda2', None, tf.constant(self.lambda2, tf.float32)) 199 | h_0 = self.__get_variable('h_0', [self.batch_size, self.n_hidden], tf.zeros_initializer) 200 | parameters = OrderedDict([('A', A), 201 | ('gw', gw), 202 | ('gama', gama), 203 | ('lambda1', lambda1), 204 | ('lambda2', lambda2), 205 | ('h_0', h_0)]) 206 | epsilon = tf.constant(100, dtype=tf.float32) 207 | At = tf.matrix_transpose(A) 208 | AtA = tf.matmul(At, A) 209 | I = tf.eye(self.n_hidden, self.n_hidden) 210 | one = tf.constant(1.0, dtype=tf.float32) 211 | 212 | # initialize V 213 | V = one / gama * At 214 | 215 | # initialize W_1 216 | W_1 = I - lambda2 / gama * AtA 217 | h_0_all_layers = tf.tile(h_0, [1, self.K]) 218 | h_t_1_all_layers = h_0_all_layers 219 | 220 | # padding for the first time input 221 | if self.pre_input is None: 222 | self.input = tf.concat([tf.zeros([1, self.batch_size, self.n_input], tf.float32), self.now_input], axis=0) 223 | else: 224 | self.input = tf.concat([self.pre_input, self.now_input], axis=0) 225 | 226 | # sequential ISTA mapping to RNN 227 | with arg_scope([matmul], transpose_b=True): 228 | h = [] 229 | 230 | gx1 = tf.nn.relu(dim_reduction(self.input[:-1], gw)) 231 | gx2 = tf.nn.relu(dim_reduction(self.input[1:], gw)) 232 | if len(gx1.get_shape().as_list()) != 3: 233 | gx1 = tf.expand_dims(gx1, 0) 234 | gx2 = tf.expand_dims(gx2, 0) 235 | gx1 = tf.nn.l2_normalize(gx1, dim=2) 236 | gx2 = tf.nn.l2_normalize(gx2, dim=2) 237 | # gx1 = self.input[:-1] 238 | # gx2 = self.input[1:] 239 | deltas = tf.tile(tf.reduce_sum(gx1 * gx2, axis=2, keep_dims=True), [1, 1, self.n_hidden]) 240 | 241 | for t in range(self.time_steps): 242 | h_t_1_last_layer = h_t_1_all_layers[:, (self.K - 1) * self.n_hidden:] 243 | delta = deltas[t] 244 | h_t_kth_layer = self.__soft(matmul(h_t_1_last_layer, W_1) + matmul(self.input[t + 1], V), 245 | lambda1 / gama) 246 | h_t_all_layers = h_t_kth_layer 247 | 248 | for k in range(1, self.K): 249 | W_k = lambda2 / gama * I 250 | U_k = I - one / gama * AtA 251 | h_t_kth_layer = self.__soft(tf.matmul(h_t_1_last_layer, W_k) * delta 252 | + matmul(h_t_kth_layer, U_k) 253 | - matmul(h_t_kth_layer, W_k) * delta 254 | + matmul(self.input[t + 1], V), lambda1 / gama) 255 | h_t_all_layers = tf.concat([h_t_all_layers, h_t_kth_layer], axis=1) 256 | h.append(h_t_all_layers) 257 | h_t_1_all_layers = h_t_all_layers 258 | h = tf.stack(h) 259 | 260 | return h, parameters 261 | 262 | def forward_coherence_As(self): 263 | ''' 264 | :return: the hidden state with shape: [time_step, batch_size, K * n_hidden], 265 | parameters: A dict with A, gama, lambda1, lambda2, h_0 266 | ''' 267 | # define variables and constants in the sista_rnn model 268 | As = [] 269 | with tf.variable_scope('sista_rnn_trainable_variables'): 270 | for i in range(self.K): 271 | As.append(self.__get_variable('A_{}'.format(i + 1), None, self.A_initializer)) 272 | gw = self.__get_variable('gw', None, self.gw_initializer) 273 | gama = self.__get_variable('gama', None, tf.constant(self.gama, tf.float32)) 274 | lambda1 = self.__get_variable('lambda1', None, tf.constant(self.lambda1, tf.float32)) 275 | lambda2 = self.__get_variable('lambda2', None, tf.constant(self.lambda2, tf.float32)) 276 | h_0 = self.__get_variable('h_0', [self.batch_size, self.n_hidden], tf.zeros_initializer) 277 | parameters = OrderedDict([('A_{}'.format(i + 1), As[i]) for i in range(self.K)] + [ 278 | ('gama', gama), 279 | ('lambda1', lambda1), 280 | ('lambda2', lambda2), 281 | ('h_0', h_0)]) 282 | epsilon = tf.constant(100, dtype=tf.float32) 283 | Ats = [tf.matrix_transpose(As[i]) for i in range(self.K)] 284 | AtAs = [tf.matmul(Ats[i], As[i]) for i in range(self.K)] 285 | I = tf.eye(self.n_hidden, self.n_hidden) 286 | one = tf.constant(1.0, dtype=tf.float32) 287 | 288 | # initialize V 289 | V = one / gama * Ats[0] 290 | 291 | # initialize W_1 292 | W_1 = I - lambda2 / gama * AtAs[0] 293 | 294 | h_0_all_layers = tf.tile(h_0, [1, self.K]) 295 | h_t_1_all_layers = h_0_all_layers 296 | 297 | # padding for the first time input 298 | if self.pre_input is None: 299 | self.input = tf.concat([tf.zeros([1, self.batch_size, self.n_input], tf.float32), self.now_input], axis=0) 300 | else: 301 | self.input = tf.concat([self.pre_input, self.now_input], axis=0) 302 | 303 | # sequential ISTA mapping to RNN 304 | with arg_scope([matmul], transpose_b=True): 305 | h = [] 306 | 307 | gx1 = tf.nn.relu(dim_reduction(self.input[:-1], gw)) 308 | gx2 = tf.nn.relu(dim_reduction(self.input[1:], gw)) 309 | deltas = tf.tile(tf.reduce_sum(gx1 * gx2, axis=2, keep_dims=True), [1, 1, self.n_hidden]) 310 | deltas = tf.nn.softmax(deltas, dim=0) 311 | 312 | for t in range(self.time_steps): 313 | h_t_1_last_layer = h_t_1_all_layers[:, (self.K - 1) * self.n_hidden:] 314 | delta = deltas[t] 315 | h_t_kth_layer = self.__soft(matmul(h_t_1_last_layer, W_1) + matmul(self.input[t + 1], V), 316 | lambda1 / gama) 317 | h_t_all_layers = h_t_kth_layer 318 | 319 | for k in range(1, self.K): 320 | # initialize V 321 | V = one / gama * Ats[k] 322 | # initialize W_1 323 | W_1 = I - lambda2 / gama * AtAs[k] 324 | 325 | W_k = lambda2 / gama * I 326 | U_k = I - one / gama * AtAs[k] 327 | h_t_kth_layer = self.__soft(tf.matmul(h_t_1_last_layer, W_k) * delta 328 | + matmul(h_t_kth_layer, U_k) \ 329 | - matmul(h_t_kth_layer, W_k) * delta 330 | + matmul(self.input[t + 1], V), lambda1 / gama) 331 | h_t_all_layers = tf.concat([h_t_all_layers, h_t_kth_layer], axis=1) 332 | h.append(h_t_all_layers) 333 | h_t_1_all_layers = h_t_all_layers 334 | h = tf.stack(h) 335 | 336 | return h, parameters 337 | -------------------------------------------------------------------------------- /libs/sista_rnn_anomaly_detection.py: -------------------------------------------------------------------------------- 1 | from libs.base import base 2 | from libs.sista_rnn import sista_rnn 3 | from libs.feature_loader_multi_patch import FeatureLoader 4 | import numpy as np 5 | import tensorflow as tf 6 | import os 7 | import h5py 8 | import pickle 9 | from collections import OrderedDict 10 | 11 | class sista_rnn_anomaly_detection(base): 12 | def __init__(self, input, label, A_initializer, sess, config): 13 | self.A_initializer = A_initializer 14 | self.sess = sess 15 | self.global_step = tf.Variable(0, trainable=False) 16 | super(sista_rnn_anomaly_detection, self).__init__(input, label, config) 17 | self.saver = tf.train.Saver(max_to_keep=None, var_list=self.optimized_parameters) 18 | 19 | def _head(self, input): 20 | return input, None 21 | 22 | def _body(self, input): 23 | model = sista_rnn(input[0], input[1], self.config['n_hidden'], self.config['K'], self.config['gama'], self.config['lambda1'], self.config['lambda2'], self.A_initializer) 24 | h, parameters = model.forward() 25 | return h[..., -self.config['n_hidden']:], list(parameters.values()) 26 | 27 | def _get_statistics(self): 28 | pred_reshape = tf.reshape(self.endpoints['body_output'], [-1, self.config['n_hidden']]) 29 | loss2 = self.config['lambda1'] * tf.reduce_mean(tf.reduce_sum(tf.abs(pred_reshape), axis=1)) 30 | loss3 = self.config['lambda2'] * tf.reduce_mean(tf.reduce_sum(tf.square(self.endpoints['body_output'][:-1] - self.endpoints['body_output'][1:]), axis=2) * 31 | tf.exp(-tf.reduce_sum(tf.square(self.endpoints['head_output'][1][:-1] - self.endpoints['head_output'][1][1:]), axis=2) / 100) + tf.reduce_sum( 32 | tf.square(self.endpoints['body_output'][0]), axis=1) * tf.exp(-tf.reduce_sum(tf.square(self.endpoints['head_output'][1][0]), axis=1) / 100)) / 2.0 33 | A_norm = tf.reduce_mean(tf.norm(self.all_parameters['body_parameters'][0], axis=0)) 34 | h_nonzero_count = tf.reduce_mean(tf.count_nonzero(self.endpoints['body_output'], axis=2, dtype=tf.float32)) 35 | 36 | return OrderedDict([('loss2', loss2), 37 | ('loss3', loss3), 38 | ('A_norm', A_norm), 39 | ('h_nonzero_count', h_nonzero_count)]) 40 | 41 | def train(self): 42 | # initialize a batch_loader 43 | featureLoader = FeatureLoader(self.config['train_feature_path'], 44 | self.config['train_videos_txt'], 45 | self.config['batch_size'], 46 | self.config['time_steps']) 47 | 48 | # define optimizer and gradients 49 | # optimizer = tf.train.MomentumOptimizer(config['learning_rate'], momentum=0.9) 50 | optimizer = tf.train.RMSPropOptimizer(self.config['learning_rate']) 51 | print(self.optimized_parameters) 52 | grad_var = optimizer.compute_gradients(self.total_loss, self.optimized_parameters) 53 | min_op = optimizer.apply_gradients(grad_var, self.global_step) 54 | self.sess.run(tf.global_variables_initializer()) 55 | print('... training ...') 56 | 57 | iter = 0 58 | while iter < self.config['n_iter']: 59 | iter = iter + 1 60 | 61 | # batch.shape = NxD 62 | batch = featureLoader.load_batch() 63 | rgb = batch['rgb'] 64 | rgb = np.reshape(rgb, [self.config['time_steps'], self.config['batch_size'] * 21, self.config['n_input']]) 65 | pre_input = np.zeros([1, self.config['batch_size'] * 21, self.config['n_input']], dtype=np.float32) 66 | 67 | run_np = self.sess.run({'min_op': min_op, 68 | 'global_step': self.global_step, 69 | 'losses': self.losses, 70 | 'summaries': self.summaries, 71 | 'statistics': self.statistics}, 72 | feed_dict={self.input[0]: pre_input, 73 | self.input[1]: rgb}) 74 | 75 | if iter % self.config['display'] == 0: 76 | print(self.config['dataset'] + (' training iter = {} ' 77 | + ''.join([name + ' = {} ' for name in self.losses.keys()]) 78 | + ''.join([name + ' = {} ' for name in self.statistics.keys()])). 79 | format(iter, *[value for value in run_np['losses'].values()], *[value for value in run_np['statistics'].values()])) 80 | 81 | if iter % self.config['snapshot'] == 0: 82 | self.saver.save(self.sess, os.path.join(self.config['ckpt_path'], self.config['prefix']), global_step=self.global_step) 83 | print('save model') 84 | 85 | if iter % self.config['summary'] == 0: 86 | self.summary_writer.add_summary(run_np['summaries'], run_np['global_step']) 87 | print('write summary') 88 | 89 | def test(self): 90 | self.sess.run(tf.global_variables_initializer()) 91 | h_0 = self.all_parameters['body_parameters'][-1] 92 | reset_h_0_init_Tensor = tf.assign(h_0, tf.zeros([21, self.config['n_hidden']], dtype=tf.float32)) 93 | update_h_0_Tensor = tf.assign(h_0, self.endpoints['body_output'][0]) 94 | 95 | clip_length = 2 96 | # load sista-rnn model parameters 97 | for test_loop in range(self.config['snapshot'], self.config['test_loop'] + self.config['snapshot'], self.config['snapshot']): 98 | self.saver.restore(self.sess, os.path.join(self.config['ckpt_path'], self.config['prefix'] + '-' + str(test_loop))) 99 | 100 | print('... testing ...') 101 | 102 | with open(self.config['test_videos_txt'], 'r') as f: 103 | f_lines = f.readlines() 104 | 105 | total_loss = [] 106 | 107 | for video in f_lines: 108 | video = video.strip('\n') 109 | f = h5py.File(os.path.join(self.config['test_feature_path'], video), 'r') 110 | num_frames = rgb_data.shape[0] 111 | # rgb_data_new = [] 112 | # for index, data in enumerate(rgb_data): 113 | # # clip_data = np.concatenate((rgb_data[index], np.mean(rgb_data[max(0, index - clip_length // 2): min(num_frames, index + clip_length // 2)], axis=0)), axis=2) 114 | # clip_data = np.mean(rgb_data[max(0, index - clip_length // 2): min(num_frames, index + clip_length // 2)], axis=0) 115 | # # clip_data = np.abs(rgb_data[min(num_frames - 1, index + 1)] - rgb_data[index]) 116 | # rgb_data_new.append(clip_data) 117 | # rgb_data = np.array(rgb_data_new) 118 | 119 | multi_patch_rgb_data = np.zeros([num_frames, 21, self.config['n_input']], dtype=np.float32) 120 | t = 0 121 | for s in self.config['scales']: 122 | step = int(np.ceil(self.config['feature_size'] / s)) 123 | for i in range(0, int(self.config['feature_size']), step): 124 | for j in range(0, int(self.config['feature_size']), step): 125 | temp = rgb_data[:, i:min(i + step, int(self.config['feature_size'])), 126 | j:min(j + step, int(self.config['feature_size']))] 127 | if temp.ndim == 4: 128 | multi_patch_rgb_data[:, t] = np.mean(np.mean(temp, axis=1), axis=1) 129 | elif temp.ndim == 3: 130 | multi_patch_rgb_data[:, t] = np.mean(temp, axis=1) 131 | t = t + 1 132 | 133 | sub_video_loss = np.zeros([num_frames, 21], dtype=np.float32) 134 | pre_input = np.zeros([1, self.config['batch_size'] * 21, self.config['n_input']], dtype=np.float32) 135 | for idx in range(num_frames): 136 | rgb = np.expand_dims(multi_patch_rgb_data[idx], 0) 137 | loss_np, _ = self.sess.run([self.testing_loss, update_h_0_Tensor], feed_dict={self.input[0]: pre_input, 138 | self.input[1]: rgb}) 139 | pre_input = rgb 140 | sub_video_loss[idx] = loss_np 141 | 142 | print('{} / {}, loss = {}'.format(video, idx, loss_np)) 143 | 144 | # reset h0 for the next test video 145 | self.sess.run(reset_h_0_init_Tensor) 146 | 147 | # add to total loss 148 | if self.config['dataset'] == 'ped2' or self.config['dataset'] == 'moving_mnist': 149 | temp = (sub_video_loss[:,0] + sub_video_loss[:,1:5].max(axis=1) + sub_video_loss[:,5:].max(axis=1)) / np.float32(3.0) 150 | total_loss.append(temp) 151 | elif self.config['dataset'] == 'avenue' or self.config['dataset'] == 'shanghaitech': 152 | total_loss.append(sub_video_loss[:, 0]) 153 | 154 | results = { 155 | 'dataset': self.config['dataset'], 156 | 'mse': total_loss 157 | } 158 | 159 | with open(self.config['save_results_path'] + '_{}.bin'.format(test_loop), 'wb') as save_file: 160 | pickle.dump(results, save_file, 2) 161 | 162 | class sista_rnn_anomaly_detection_TSC(sista_rnn_anomaly_detection): 163 | def _tail(self, input): 164 | return input, None 165 | 166 | def _get_losses(self, pred, label): 167 | input_reshape = tf.reshape(self.endpoints['head_output'][1], [-1, self.config['n_input']]) 168 | pred_reshape = tf.reshape(pred, [-1, self.config['n_hidden']]) 169 | loss = tf.reduce_mean(tf.reduce_sum(tf.square(input_reshape - tf.matmul(pred_reshape, self.all_parameters['body_parameters'][0], transpose_b=True)), axis=1)) / 2.0 170 | 171 | return OrderedDict([('loss', loss)]) 172 | 173 | def _get_optimized_parameters(self, all_parameters): 174 | return [all_parameters['body_parameters'][0]] 175 | 176 | def _get_testing_loss(self, pred, label): 177 | input_reshape = tf.reshape(self.endpoints['head_output'][1], [-1, self.config['n_input']]) 178 | pred_reshape = tf.reshape(pred, [-1, self.config['n_hidden']]) 179 | loss = tf.reduce_sum(tf.square(input_reshape - tf.matmul(pred_reshape, self.all_parameters['body_parameters'][0], transpose_b=True)), axis=1) / 2.0 180 | return loss 181 | 182 | class sista_rnn_anomaly_detection_AE(sista_rnn_anomaly_detection): 183 | def _tail(self, input): 184 | input_reshape = tf.reshape(input, [-1, self.config['n_hidden']]) 185 | rng = np.random.RandomState(self.config['seed']) 186 | with tf.variable_scope('y'): 187 | Z = np.asarray( 188 | rng.uniform( 189 | low=-np.sqrt(6.0 / (self.config['n_input'] + self.config['n_hidden'])), 190 | high=np.sqrt(6.0 / (self.config['n_input'] + self.config['n_hidden'])), 191 | size=(self.config['n_output'], self.config['n_hidden']) 192 | ), 193 | dtype=np.float32 194 | ) 195 | Z = tf.get_variable('Z', dtype=tf.float32, initializer=Z/2.0) 196 | y = tf.matmul(input_reshape, Z, transpose_b=True) 197 | 198 | return y, [Z] 199 | 200 | def _get_losses(self, pred, label): 201 | input_reshape = tf.reshape(self.endpoints['head_output'][1], [-1, self.config['n_input']]) 202 | loss = tf.reduce_mean(tf.reduce_sum(tf.square(input_reshape - pred), axis=1)) / 2.0 203 | 204 | return OrderedDict([('loss', loss)]) 205 | 206 | def _get_optimized_parameters(self, all_parameters): 207 | return self.all_parameters['body_parameters'][:-1] + tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, 'y') 208 | 209 | def _get_testing_loss(self, pred, label): 210 | input_reshape = tf.reshape(self.endpoints['head_output'][1], [-1, self.config['n_input']]) 211 | loss = tf.reduce_sum(tf.square(input_reshape - pred), axis=1) / 2.0 212 | 213 | return loss 214 | 215 | -------------------------------------------------------------------------------- /libs/sista_rnn_anomaly_detection_coherence.py: -------------------------------------------------------------------------------- 1 | from libs.base import base 2 | from libs.sista_rnn import sista_rnn 3 | from libs.feature_loader_multi_patch import FeatureLoader 4 | import numpy as np 5 | import tensorflow as tf 6 | import os 7 | import h5py 8 | import pickle 9 | from collections import OrderedDict 10 | from tensorflow.contrib.framework import arg_scope 11 | from tensorflow.contrib.framework.python.ops.arg_scope import add_arg_scope 12 | import time, datetime, csv 13 | 14 | 15 | # original tf.matmul is without an arg_scope decorator 16 | @add_arg_scope 17 | def matmul(*args, **kwargs): 18 | return tf.matmul(*args, **kwargs) 19 | 20 | 21 | def dim_reduction(x, w): 22 | _shape = x.get_shape().as_list() 23 | x = tf.reshape(x, [-1, _shape[-1]]) 24 | with arg_scope([matmul], transpose_b=True): 25 | x = matmul(x, w) 26 | x = tf.reshape(x, _shape[:-1] + [w.get_shape().as_list()[0]]) 27 | return x 28 | 29 | 30 | class sista_rnn_anomaly_detection(base): 31 | def __init__(self, input, label, A_initializer, gw_initializer, sess, config): 32 | self.A_initializer = A_initializer 33 | self.gw_initializer = gw_initializer 34 | self.sess = sess 35 | self.global_step = tf.Variable(0, trainable=False) 36 | super(sista_rnn_anomaly_detection, self).__init__(input, label, config) 37 | self.saver = tf.train.Saver(max_to_keep=None, var_list=self.optimized_parameters) 38 | 39 | def _head(self, input): 40 | return input, None 41 | 42 | def _body(self, input): 43 | model = sista_rnn(input[0], input[1], self.config['n_hidden'], self.config['K'], self.config['gama'], 44 | self.config['lambda1'], self.config['lambda2'], self.A_initializer, self.gw_initializer) 45 | h, parameters = model.forward_coherence() 46 | return h[..., -self.config['n_hidden']:], list(parameters.values()) 47 | 48 | def _get_statistics(self): 49 | pred_reshape = tf.reshape(self.endpoints['body_output'], [-1, self.config['n_hidden']]) 50 | loss2 = self.config['lambda1'] * tf.reduce_mean(tf.reduce_sum(tf.abs(pred_reshape), axis=1)) 51 | 52 | gw = self.all_parameters['body_parameters'][1] 53 | x = self.endpoints['head_output'][1] 54 | gx1 = tf.nn.relu(dim_reduction(x[:-1], gw)) 55 | gx2 = tf.nn.relu(dim_reduction(x[1:], gw)) 56 | # gx1 = x[:-1] 57 | # gx2 = x[1:] 58 | if len(gx1.get_shape().as_list()) != 3: 59 | gx1 = tf.expand_dims(gx1, 0) 60 | gx2 = tf.expand_dims(gx2, 0) 61 | gx1 = tf.nn.l2_normalize(gx1, dim=2) 62 | gx2 = tf.nn.l2_normalize(gx2, dim=2) 63 | 64 | deltas = tf.reduce_sum(gx1 * gx2, axis=2) 65 | loss3 = self.config['lambda2'] * tf.reduce_mean( 66 | tf.reduce_sum(tf.square(self.endpoints['body_output'][:-1] - self.endpoints['body_output'][1:]), 67 | axis=2) * deltas) 68 | 69 | A_norm = tf.reduce_mean(tf.norm(self.all_parameters['body_parameters'][0], axis=0)) 70 | h_nonzero_count = tf.reduce_mean(tf.count_nonzero(self.endpoints['body_output'], axis=2, dtype=tf.float32)) 71 | 72 | self.gx = tf.nn.relu(dim_reduction(x, gw)) 73 | 74 | return OrderedDict([('loss2', loss2), 75 | ('loss3', loss3), 76 | ('A_norm', A_norm), 77 | ('h_nonzero_count', h_nonzero_count)]) 78 | 79 | def train(self): 80 | # initialize a batch_loader 81 | featureLoader = FeatureLoader(self.config['train_feature_path'], 82 | self.config['train_videos_txt'], 83 | self.config['batch_size'], 84 | self.config['time_steps']) 85 | 86 | # define optimizer and gradients 87 | # optimizer = tf.train.MomentumOptimizer(config['learning_rate'], momentum=0.9) 88 | optimizer = tf.train.RMSPropOptimizer(self.config['learning_rate']) 89 | print(self.optimized_parameters) 90 | grad_var = optimizer.compute_gradients(self.total_loss, self.optimized_parameters) 91 | min_op = optimizer.apply_gradients(grad_var, self.global_step) 92 | self.sess.run(tf.global_variables_initializer()) 93 | print('... training ...') 94 | 95 | iter = 0 96 | losses = [] 97 | while iter < self.config['n_iter']: 98 | start = datetime.datetime.now() 99 | iter = iter + 1 100 | 101 | # batch.shape = NxD 102 | batch = featureLoader.load_batch() 103 | rgb = batch['rgb'] 104 | rgb = np.reshape(rgb, [self.config['time_steps'], self.config['batch_size'] * 21, self.config['n_input']]) 105 | pre_input = np.zeros([1, self.config['batch_size'] * 21, self.config['n_input']], dtype=np.float32) 106 | 107 | run_np = self.sess.run({'min_op': min_op, 108 | 'global_step': self.global_step, 109 | 'losses': self.losses, 110 | 'summaries': self.summaries, 111 | 'statistics': self.statistics}, 112 | feed_dict={self.input[0]: pre_input, 113 | self.input[1]: rgb}) 114 | 115 | if iter % self.config['display'] == 0: 116 | print(self.config['dataset'] + (' training iter = {} ' 117 | + ''.join([name + ' = {} ' for name in self.losses.keys()]) 118 | + ''.join([name + ' = {} ' for name in self.statistics.keys()])). 119 | format(iter, *[value for value in run_np['losses'].values()], 120 | *[value for value in run_np['statistics'].values()])) 121 | 122 | if iter % self.config['snapshot'] == 0: 123 | self.saver.save(self.sess, os.path.join(self.config['ckpt_path'], self.config['prefix']), 124 | global_step=self.global_step) 125 | print('save model') 126 | 127 | if iter % self.config['summary'] == 0: 128 | self.summary_writer.add_summary(run_np['summaries'], run_np['global_step']) 129 | print('write summary') 130 | 131 | end = datetime.datetime.now() 132 | losses.append(list(run_np['losses'].values())[0]) 133 | 134 | def test(self): 135 | self.sess.run(tf.global_variables_initializer()) 136 | h_0 = self.all_parameters['body_parameters'][-1] 137 | reset_h_0_init_Tensor = tf.assign(h_0, tf.zeros([21, self.config['n_hidden']], dtype=tf.float32)) 138 | update_h_0_Tensor = tf.assign(h_0, self.endpoints['body_output'][0]) 139 | 140 | # load sista-rnn model parameters 141 | test_loop = self.config['snapshot'] 142 | clip_length = 4 143 | while test_loop <= self.config['test_loop']: 144 | ckpt_name = os.path.join(self.config['ckpt_path'], self.config['prefix'] + '-' + str(test_loop)) 145 | while not os.path.exists(ckpt_name + '.index'): 146 | print('I am sleeping', ckpt_name) 147 | time.sleep(100) 148 | self.saver.restore(self.sess, ckpt_name) 149 | 150 | print('... testing ...') 151 | 152 | with open(self.config['test_videos_txt'], 'r') as f: 153 | f_lines = f.readlines() 154 | 155 | total_loss = [] 156 | features = [] 157 | alphas = [] 158 | gxs = [] 159 | 160 | for video in f_lines: 161 | video = video.strip('\n') 162 | f = h5py.File(os.path.join(self.config['test_feature_path'], video), 'r') 163 | rgb_data = f['rgb'] 164 | num_frames = rgb_data.shape[0] 165 | # rgb_data_new = [] 166 | # for index, data in enumerate(rgb_data): 167 | # clip_data = np.concatenate((rgb_data[index], np.mean(rgb_data[max(0, index - clip_length // 2): min(num_frames, index + clip_length // 2)], axis=0)), axis=2) 168 | # #clip_data = np.mean(rgb_data[max(0, index - clip_length // 2): min(num_frames, index + clip_length // 2)], axis=0) 169 | # #clip_data = np.abs(rgb_data[min(num_frames - 1, index + 1)] - rgb_data[index]) 170 | # rgb_data_new.append(clip_data) 171 | # rgb_data = np.array(rgb_data_new) 172 | 173 | multi_patch_rgb_data = np.zeros([num_frames, 21, self.config['n_input']], dtype=np.float32) 174 | t = 0 175 | for s in self.config['scales']: 176 | step = int(np.ceil(self.config['feature_size'] / s)) 177 | for i in range(0, int(self.config['feature_size']), step): 178 | for j in range(0, int(self.config['feature_size']), step): 179 | temp = rgb_data[:, i:min(i + step, int(self.config['feature_size'])), 180 | j:min(j + step, int(self.config['feature_size']))] 181 | if temp.ndim == 4: 182 | multi_patch_rgb_data[:, t] = np.mean(np.mean(temp, axis=1), axis=1) 183 | elif temp.ndim == 3: 184 | multi_patch_rgb_data[:, t] = np.mean(temp, axis=1) 185 | t = t + 1 186 | 187 | sub_video_loss = np.zeros([num_frames, 21], dtype=np.float32) 188 | feature = [] 189 | alpha = [] 190 | gx = [] 191 | pre_input = np.zeros([1, self.config['batch_size'] * 21, self.config['n_input']], dtype=np.float32) 192 | h = self.endpoints['body_output'][0] 193 | for idx in range(num_frames): 194 | start = datetime.datetime.now() 195 | rgb = np.expand_dims(multi_patch_rgb_data[idx], 0) 196 | loss_np, _, h_np, gx_np = self.sess.run([self.testing_loss, update_h_0_Tensor, h, self.gx], 197 | feed_dict={self.input[0]: pre_input, 198 | self.input[1]: rgb}) 199 | pre_input = rgb 200 | sub_video_loss[idx] = loss_np 201 | 202 | end = datetime.datetime.now() 203 | 204 | print('{} / {}, loss = {}'.format(video, idx, loss_np)) 205 | 206 | # reset h0 for the next test video 207 | self.sess.run(reset_h_0_init_Tensor) 208 | 209 | # add to total loss 210 | if self.config['dataset'] == 'ped2' or self.config['dataset'] == 'enter' or self.config['dataset'] == 'exit': 211 | temp = (sub_video_loss[:, 0] + sub_video_loss[:, 1:5].max(axis=1) + sub_video_loss[:, 5:].max( 212 | axis=1)) / np.float32(3.0) 213 | total_loss.append(temp) 214 | elif self.config['dataset'] == 'shanghaitech': 215 | total_loss.append(sub_video_loss[:, 0]) 216 | elif self.config['dataset'] == 'avenue' or self.config['dataset'] == 'lvv1' or self.config['dataset'] == 'ped1': 217 | total_loss.append(sub_video_loss) 218 | 219 | results = { 220 | 'dataset': self.config['dataset'], 221 | 'mse': total_loss, 222 | } 223 | 224 | with open(self.config['save_results_path'] + '_{}.bin'.format(test_loop), 'wb') as save_file: 225 | pickle.dump(results, save_file) 226 | 227 | test_loop += self.config['snapshot'] 228 | 229 | 230 | class sista_rnn_anomaly_detection_AE(sista_rnn_anomaly_detection): 231 | def _tail(self, input): 232 | input_reshape = tf.reshape(input, [-1, self.config['n_hidden']]) 233 | rng = np.random.RandomState(self.config['seed']) 234 | with tf.variable_scope('y'): 235 | Z = np.asarray( 236 | rng.uniform( 237 | low=-np.sqrt(6.0 / (self.config['n_input'] + self.config['n_hidden'])), 238 | high=np.sqrt(6.0 / (self.config['n_input'] + self.config['n_hidden'])), 239 | size=(self.config['n_output'], self.config['n_hidden']) 240 | ), 241 | dtype=np.float32 242 | ) 243 | Z = tf.get_variable('Z', dtype=tf.float32, initializer=Z / 2.0) 244 | y = tf.matmul(input_reshape, Z, transpose_b=True) 245 | 246 | return y, [Z] 247 | 248 | def _get_losses(self, pred, label): 249 | input_reshape = tf.reshape(self.endpoints['head_output'][1], [-1, self.config['n_input']]) 250 | loss = tf.reduce_mean(tf.reduce_sum(tf.square(input_reshape - pred), axis=1)) / 2.0 251 | 252 | return OrderedDict([('loss', loss)]) 253 | 254 | def _get_optimized_parameters(self, all_parameters): 255 | return self.all_parameters['body_parameters'][:-2] + tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, 'y') 256 | 257 | def _get_testing_loss(self, pred, label): 258 | input_reshape = tf.reshape(self.endpoints['head_output'][1], [-1, self.config['n_input']]) 259 | loss = tf.reduce_sum(tf.square(input_reshape - pred), axis=1) / 2.0 260 | 261 | return loss 262 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | numpy==1.15.4 2 | matplotlib==2.2.2 3 | scikit_image==0.13.1 4 | six==1.11.0 5 | opencv_python==3.4.3.18 6 | h5py==2.7.1 7 | scipy==1.1.0 8 | tensorflow_gpu==1.11.0 9 | seaborn==0.8.1 10 | scikit_learn==0.20.2 11 | tensorflow==1.11.0 12 | PyYAML==3.13 13 | -------------------------------------------------------------------------------- /run_anomaly_detection.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import numpy as np 4 | import yaml 5 | import tensorflow as tf 6 | from libs.sista_rnn_anomaly_detection import sista_rnn_anomaly_detection_TSC, sista_rnn_anomaly_detection_AE 7 | from libs.common import checkdir 8 | from libs import FLAGS 9 | import multiprocessing 10 | 11 | def main(config): 12 | if sys.argv[2] == '0': 13 | config['train_videos_txt'] = config['train_videos_txt'].format(config['dataset']) 14 | config['train_feature_path'] = config['train_feature_path'].format(config['dataset'], config['twostream_model']) 15 | config['prefix'] = config['prefix'].format(config['dataset'], config['twostream_model'], config['model_type'], 16 | config['K'], config['n_hidden'], config['lambda1'], config['lambda2'], config['time_steps']) 17 | config['log_path'] = os.path.join(config['log_path'], config['prefix']) 18 | checkdir(config['ckpt_path']) 19 | checkdir(config['log_path']) 20 | elif sys.argv[2] == '1': 21 | config['time_steps'] = 1 22 | config['batch_size'] = 1 23 | config['test_videos_txt'] = config['test_videos_txt'].format(config['dataset']) 24 | config['test_feature_path'] = config['test_feature_path'].format(config['dataset'], config['twostream_model']) 25 | config['prefix'] = config['prefix'].format(config['dataset'], config['twostream_model'], config['model_type'], 26 | config['K'], config['n_hidden'], config['lambda1'], config['lambda2'], 10) 27 | checkdir(config['save_results_path']) 28 | config['save_results_path'] = os.path.join(config['save_results_path'], config['prefix']) 29 | else: 30 | raise Exception('only support 0 for training or 1 for testing') 31 | rng = np.random.RandomState(config['seed']) 32 | 33 | os.environ['CUDA_DEVICES_ORDER'] = "PCI_BUS_ID" 34 | os.environ['CUDA_VISIBLE_DEVICES'] = config['gpu'] 35 | tf_config = tf.ConfigProto() 36 | tf_config.gpu_options.allow_growth = True 37 | sess = tf.Session(config=tf_config) 38 | 39 | # ======================== SISTA-RNN ============================= # 40 | print('... building the sista-rnn networks') 41 | 42 | pre_input = tf.placeholder(tf.float32, [1, config['batch_size'] * 21, config['n_input']]) 43 | now_input = tf.placeholder(tf.float32, [config['time_steps'], config['batch_size'] * 21, config['n_input']]) 44 | A = np.asarray( 45 | rng.uniform( 46 | low=-np.sqrt(6.0 / (config['n_input'] + config['n_hidden'])), 47 | high=np.sqrt(6.0 / (config['n_input'] + config['n_hidden'])), 48 | size=(config['n_input'], config['n_hidden']) 49 | ) / 2.0, 50 | dtype=np.float32 51 | ) 52 | 53 | if config['model_type'] == FLAGS.TSC: 54 | model = sista_rnn_anomaly_detection_TSC([pre_input, now_input], None, A, sess, config) 55 | elif config['model_type'] == FLAGS.AE: 56 | model = sista_rnn_anomaly_detection_AE([pre_input, now_input], None, A, sess, config) 57 | else: 58 | raise Exception('not support {}, only support TSC and AE model'.format(config['model_type'])) 59 | 60 | if sys.argv[2] == '0': 61 | model.train() 62 | else: 63 | model.test() 64 | 65 | if __name__ == '__main__': 66 | # ==========================load config================================ # 67 | if len(sys.argv) < 3: 68 | raise Exception('usage: python xxx.py config/xxx.yaml 0/1 (0 is training, 1 is testing)') 69 | with open(sys.argv[1], 'r') as stream: 70 | config = yaml.load(stream) 71 | 72 | main(config) 73 | -------------------------------------------------------------------------------- /run_anomaly_detection_coherence.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import yaml 4 | import tensorflow as tf 5 | from libs.sista_rnn_anomaly_detection_coherence import sista_rnn_anomaly_detection_AE 6 | from libs.common import checkdir 7 | import argparse 8 | 9 | 10 | def parse_args(): 11 | parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter) 12 | parser.add_argument('--gpu', type=str, default='0', help='the gpu device id') 13 | parser.add_argument('--mode', type=str, default='0', help='0 is training, and 1 is testing', 14 | choices=['0', '1']) 15 | parser.add_argument('--config_file', type=str, default='./config/anomaly_detection_coherence.yaml', 16 | help='the config file path') 17 | 18 | args = parser.parse_args() 19 | 20 | return args 21 | 22 | 23 | def main(config): 24 | if config['mode'] == '0': 25 | config['train_videos_txt'] = config['train_videos_txt'].format(config['dataset']) 26 | config['train_feature_path'] = config['train_feature_path'].format(config['dataset'], config['twostream_model']) 27 | config['prefix'] = config['prefix'].format(config['dataset'], config['twostream_model'], config['model_type'], 28 | config['K'], config['n_hidden'], config['lambda1'], 29 | config['lambda2'], config['time_steps']) 30 | config['log_path'] = os.path.join(config['log_path'], config['prefix']) 31 | checkdir(config['ckpt_path']) 32 | checkdir(config['log_path']) 33 | elif config['mode'] == '1': 34 | config['time_steps'] = 1 35 | config['batch_size'] = 1 36 | config['test_videos_txt'] = config['test_videos_txt'].format(config['dataset']) 37 | config['test_feature_path'] = config['test_feature_path'].format(config['dataset'], config['twostream_model']) 38 | config['prefix'] = config['prefix'].format(config['dataset'], config['twostream_model'], config['model_type'], 39 | config['K'], config['n_hidden'], config['lambda1'], 40 | config['lambda2'], 10) 41 | checkdir(config['save_results_path']) 42 | config['save_results_path'] = os.path.join(config['save_results_path'], config['prefix']) 43 | else: 44 | raise Exception('only support 0 for training or 1 for testing') 45 | rng = np.random.RandomState(config['seed']) 46 | 47 | os.environ['CUDA_DEVICES_ORDER'] = "PCI_BUS_ID" 48 | os.environ['CUDA_VISIBLE_DEVICES'] = config['gpu'] 49 | tf_config = tf.ConfigProto() 50 | tf_config.gpu_options.allow_growth = True 51 | sess = tf.Session(config=tf_config) 52 | 53 | # ======================== SISTA-RNN ============================= # 54 | print('... building the sista-rnn networks') 55 | 56 | pre_input = tf.placeholder(tf.float32, [1, config['batch_size'] * 21, config['n_input']]) 57 | now_input = tf.placeholder(tf.float32, [config['time_steps'], config['batch_size'] * 21, config['n_input']]) 58 | A = np.asarray( 59 | rng.uniform( 60 | low=-np.sqrt(6.0 / (config['n_input'] + config['n_hidden'])), 61 | high=np.sqrt(6.0 / (config['n_input'] + config['n_hidden'])), 62 | size=(config['n_input'], config['n_hidden']) 63 | ) / 2.0, 64 | dtype=np.float32 65 | ) 66 | 67 | gw = np.asarray( 68 | rng.uniform( 69 | low=-np.sqrt(6.0 / (config['n_input'] + config['n_gw'])), 70 | high=np.sqrt(6.0 / (config['n_input'] + config['n_gw'])), 71 | size=(config['n_gw'], config['n_input']) 72 | ) / 2.0, 73 | dtype=np.float32 74 | ) 75 | 76 | model = sista_rnn_anomaly_detection_AE([pre_input, now_input], None, A, gw, sess, config) 77 | 78 | if config['mode'] == '0': 79 | model.train() 80 | else: 81 | model.test() 82 | 83 | 84 | if __name__ == '__main__': 85 | # ==========================load config================================ # 86 | args = parse_args() 87 | with open(args.config_file, 'r') as stream: 88 | config = yaml.load(stream) 89 | config['mode'] = args.mode 90 | config['gpu'] = args.gpu 91 | 92 | main(config) 93 | -------------------------------------------------------------------------------- /tools/__init__.py: -------------------------------------------------------------------------------- 1 | from .ground_truth import GroundTruthLoader 2 | -------------------------------------------------------------------------------- /tools/ground_truth.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | import scipy.io as scio 3 | import numpy as np 4 | import os 5 | from libs import FLAGS 6 | 7 | 8 | class GroundTruthLoader(object): 9 | AVENUE = FLAGS.AVENUE 10 | PED1 = FLAGS.PED1 11 | PED2 = FLAGS.PED2 12 | ENTRANCE = FLAGS.ENTRANCE 13 | EXIT = FLAGS.EXIT 14 | SHANGHAITECH = FLAGS.SHANGHAITECH 15 | MOVINGMNIST = FLAGS.MOVINGMNIST 16 | SHANGHAITEC_LABEL_PATH = './dataset/anomaly_detection/shanghaitech/testing/test_frame_mask/' 17 | 18 | NAME_MAT_MAPPING = { 19 | AVENUE: './dataset/anomaly_detection/gt_mat/avenue.mat', 20 | PED1: './dataset/anomaly_detection/gt_mat/ped1.mat', 21 | PED2: './dataset/anomaly_detection/gt_mat/ped2.mat', 22 | ENTRANCE: './dataset/anomaly_detection/gt_mat/enter_original.mat', 23 | EXIT: './dataset/anomaly_detection/gt_mat/exit.mat' 24 | } 25 | 26 | NAME_VIDEO_MAPPING = { 27 | AVENUE: './dataset/avenue/testing/videos', 28 | PED1: './dataset/ped1/testing/videos', 29 | PED2: './dataset/ped2/testing/videos', 30 | ENTRANCE: './dataset/enter/testing/videos', 31 | EXIT: './dataset/exit/testing/videos' 32 | } 33 | 34 | NAME_FRAMES_MAPPING = { 35 | AVENUE: './dataset/anomaly_detection/avenue/testing/frames', 36 | PED1: './dataset/anomaly_detection/ped1/testing/frames', 37 | PED2: './dataset/anomaly_detection/ped2/testing/frames', 38 | ENTRANCE: './dataset/anomaly_detection/enter/testing/frames', 39 | EXIT: './dataset/anomaly_detection/exit/testing/frames' 40 | } 41 | 42 | def __init__(self, mapping_json=None): 43 | if mapping_json is not None: 44 | import json 45 | with open(mapping_json, 'r') as f: 46 | self.mapping = json.load(f) 47 | 48 | else: 49 | self.mapping = GroundTruthLoader.NAME_MAT_MAPPING 50 | 51 | def __call__(self, dataset=AVENUE): 52 | """ get the ground truth by provided the name of dataset. 53 | 54 | :type dataset: str 55 | :param dataset: the name of dataset. 56 | :return: np.ndarray, shape(#video) 57 | np.array[0] contains all the start frame and end frame of abnormal events of video 0, 58 | and its shape is (#framse, ) 59 | """ 60 | 61 | if dataset == GroundTruthLoader.SHANGHAITECH: 62 | gt = self.__load_shanghaitech_gt() 63 | elif dataset == GroundTruthLoader.MOVINGMNIST: 64 | gt = self.__load_moving_mnist_gt() 65 | else: 66 | gt = self.__load_ucsd_avenue_subway_gt(dataset) 67 | return gt 68 | 69 | def __load_ucsd_avenue_subway_gt(self, dataset): 70 | assert dataset in self.mapping, 'there is no dataset named {} \n Please check {}' \ 71 | .format(dataset, GroundTruthLoader.NAME_MAT_MAPPING.keys()) 72 | 73 | mat_file = self.mapping[dataset] 74 | abnormal_events = scio.loadmat(mat_file, squeeze_me=True)['gt'] 75 | 76 | if abnormal_events.ndim == 2: 77 | abnormal_events = abnormal_events.reshape(-1, abnormal_events.shape[0], abnormal_events.shape[1]) 78 | 79 | num_video = abnormal_events.shape[0] 80 | dataset_video_folder = GroundTruthLoader.NAME_FRAMES_MAPPING[dataset] 81 | video_list = os.listdir(dataset_video_folder) 82 | 83 | assert num_video == len(video_list), 'ground true does not match the number of testing videos. {} != {}' \ 84 | .format(num_video, len(video_list)) 85 | 86 | # video name template 87 | # /path/datasets/0...xx/ 88 | video_name_template = os.path.join(dataset_video_folder, '{:0>%d}' % (len(str(num_video)))) 89 | 90 | # video_name_template = os.path.join(dataset_video_folder, dataset + '_test_{0:02d}'.format(len(str(num_video)))) 91 | 92 | # get the total frames of sub video 93 | def get_video_length(sub_video_number): 94 | video_name = video_name_template.format(sub_video_number) 95 | assert os.path.isdir(video_name), '{} is not directory!'.format(video_name) 96 | 97 | length = len(os.listdir(video_name)) 98 | 99 | return length 100 | 101 | # need to test [].append, or np.array().append(), which one is faster 102 | gt = [] 103 | for i in range(num_video): 104 | length = get_video_length(i + 1) 105 | 106 | sub_video_gt = np.zeros((length), dtype=np.int8) 107 | sub_abnormal_events = abnormal_events[i] 108 | if sub_abnormal_events.ndim == 1: 109 | sub_abnormal_events = sub_abnormal_events.reshape((sub_abnormal_events.shape[0], -1)) 110 | 111 | _, num_abnormal = sub_abnormal_events.shape 112 | 113 | for j in range(num_abnormal): 114 | # (start - 1, end - 1) 115 | start = sub_abnormal_events[0, j] - 1 116 | end = sub_abnormal_events[1, j] 117 | 118 | sub_video_gt[start: end] = 1 119 | 120 | gt.append(sub_video_gt) 121 | 122 | return gt 123 | 124 | def __load_shanghaitech_gt(self): 125 | video_path_list = os.listdir(GroundTruthLoader.SHANGHAITEC_LABEL_PATH) 126 | video_path_list.sort() 127 | 128 | gt = [] 129 | for video in video_path_list: 130 | # print(os.path.join(GroundTruthLoader.SHANGHAITEC_LABEL_PATH, video)) 131 | gt.append(np.load(os.path.join(GroundTruthLoader.SHANGHAITEC_LABEL_PATH, video))) 132 | 133 | return gt 134 | 135 | def __load_moving_mnist_gt(self): 136 | # label = np.load('/home/luowx/datasets/anomaly_detection/gt_mat/mnist_occlusion_test_label.npy') 137 | label = np.load('/home/luowx/datasets/anomaly_detection/gt_mat/mnist_motion_test_label.npy') 138 | label = np.squeeze(label) 139 | gt = [] 140 | for i in range(100): 141 | gt.append(np.array(label[i * 20:(i + 1) * 20], dtype=np.int)) 142 | return gt 143 | -------------------------------------------------------------------------------- /tools/init_path.py: -------------------------------------------------------------------------------- 1 | import sys 2 | 3 | def add_path(*args): 4 | for path in args: 5 | if path not in sys.path: 6 | sys.path.append(path) 7 | -------------------------------------------------------------------------------- /txt/avenue_feature_testing.txt: -------------------------------------------------------------------------------- 1 | 01.h5 2 | 02.h5 3 | 03.h5 4 | 04.h5 5 | 05.h5 6 | 06.h5 7 | 07.h5 8 | 08.h5 9 | 09.h5 10 | 10.h5 11 | 11.h5 12 | 12.h5 13 | 13.h5 14 | 14.h5 15 | 15.h5 16 | 16.h5 17 | 17.h5 18 | 18.h5 19 | 19.h5 20 | 20.h5 21 | 21.h5 22 | -------------------------------------------------------------------------------- /txt/avenue_feature_training.txt: -------------------------------------------------------------------------------- 1 | 01.h5 2 | 02.h5 3 | 03.h5 4 | 04.h5 5 | 05.h5 6 | 06.h5 7 | 07.h5 8 | 08.h5 9 | 09.h5 10 | 10.h5 11 | 11.h5 12 | 12.h5 13 | 13.h5 14 | 14.h5 15 | 15.h5 16 | 16.h5 17 | -------------------------------------------------------------------------------- /txt/enter_feature_testing.txt: -------------------------------------------------------------------------------- 1 | 01.h5 2 | 02.h5 3 | 03.h5 4 | 04.h5 5 | 05.h5 6 | 06.h5 7 | -------------------------------------------------------------------------------- /txt/enter_feature_training.txt: -------------------------------------------------------------------------------- 1 | 01.h5 2 | 02.h5 3 | 03.h5 4 | 04.h5 5 | 05.h5 6 | 06.h5 7 | 07.h5 8 | -------------------------------------------------------------------------------- /txt/exit_feature_testing.txt: -------------------------------------------------------------------------------- 1 | 01.h5 2 | 02.h5 3 | 03.h5 4 | 04.h5 5 | -------------------------------------------------------------------------------- /txt/exit_feature_training.txt: -------------------------------------------------------------------------------- 1 | 01.h5 2 | 02.h5 3 | 03.h5 4 | 04.h5 5 | 05.h5 6 | 06.h5 7 | 07.h5 8 | 08.h5 9 | 09.h5 10 | 10.h5 11 | -------------------------------------------------------------------------------- /txt/ped1_feature_testing.txt: -------------------------------------------------------------------------------- 1 | 01.h5 2 | 02.h5 3 | 03.h5 4 | 04.h5 5 | 05.h5 6 | 06.h5 7 | 07.h5 8 | 08.h5 9 | 09.h5 10 | 10.h5 11 | 11.h5 12 | 12.h5 13 | 13.h5 14 | 14.h5 15 | 15.h5 16 | 16.h5 17 | 17.h5 18 | 18.h5 19 | 19.h5 20 | 20.h5 21 | 21.h5 22 | 22.h5 23 | 23.h5 24 | 24.h5 25 | 25.h5 26 | 26.h5 27 | 27.h5 28 | 28.h5 29 | 29.h5 30 | 30.h5 31 | 31.h5 32 | 32.h5 33 | 33.h5 34 | 34.h5 35 | 35.h5 36 | 36.h5 37 | -------------------------------------------------------------------------------- /txt/ped1_feature_training.txt: -------------------------------------------------------------------------------- 1 | 01.h5 2 | 02.h5 3 | 03.h5 4 | 04.h5 5 | 05.h5 6 | 06.h5 7 | 07.h5 8 | 08.h5 9 | 09.h5 10 | 10.h5 11 | 11.h5 12 | 12.h5 13 | 13.h5 14 | 14.h5 15 | 15.h5 16 | 16.h5 17 | 17.h5 18 | 18.h5 19 | 19.h5 20 | 20.h5 21 | 21.h5 22 | 22.h5 23 | 23.h5 24 | 24.h5 25 | 25.h5 26 | 26.h5 27 | 27.h5 28 | 28.h5 29 | 29.h5 30 | 30.h5 31 | 31.h5 32 | 32.h5 33 | 33.h5 34 | 34.h5 35 | -------------------------------------------------------------------------------- /txt/ped2_feature_testing.txt: -------------------------------------------------------------------------------- 1 | 01.h5 2 | 02.h5 3 | 03.h5 4 | 04.h5 5 | 05.h5 6 | 06.h5 7 | 07.h5 8 | 08.h5 9 | 09.h5 10 | 10.h5 11 | 11.h5 12 | 12.h5 13 | -------------------------------------------------------------------------------- /txt/ped2_feature_training.txt: -------------------------------------------------------------------------------- 1 | 01.h5 2 | 02.h5 3 | 03.h5 4 | 04.h5 5 | 05.h5 6 | 06.h5 7 | 07.h5 8 | 08.h5 9 | 09.h5 10 | 10.h5 11 | 11.h5 12 | 12.h5 13 | 13.h5 14 | 14.h5 15 | 15.h5 16 | 16.h5 17 | -------------------------------------------------------------------------------- /txt/shanghaitech_feature_testing.txt: -------------------------------------------------------------------------------- 1 | 01_0014.h5 2 | 01_0015.h5 3 | 01_0016.h5 4 | 01_0025.h5 5 | 01_0026.h5 6 | 01_0027.h5 7 | 01_0028.h5 8 | 01_0029.h5 9 | 01_0030.h5 10 | 01_0051.h5 11 | 01_0052.h5 12 | 01_0053.h5 13 | 01_0054.h5 14 | 01_0055.h5 15 | 01_0056.h5 16 | 01_0063.h5 17 | 01_0064.h5 18 | 01_0073.h5 19 | 01_0076.h5 20 | 01_0129.h5 21 | 01_0130.h5 22 | 01_0131.h5 23 | 01_0132.h5 24 | 01_0133.h5 25 | 01_0134.h5 26 | 01_0135.h5 27 | 01_0136.h5 28 | 01_0138.h5 29 | 01_0139.h5 30 | 01_0140.h5 31 | 01_0141.h5 32 | 01_0162.h5 33 | 01_0163.h5 34 | 01_0177.h5 35 | 02_0128.h5 36 | 02_0161.h5 37 | 02_0164.h5 38 | 03_0031.h5 39 | 03_0032.h5 40 | 03_0033.h5 41 | 03_0035.h5 42 | 03_0036.h5 43 | 03_0039.h5 44 | 03_0041.h5 45 | 03_0059.h5 46 | 03_0060.h5 47 | 03_0061.h5 48 | 04_0001.h5 49 | 04_0003.h5 50 | 04_0004.h5 51 | 04_0010.h5 52 | 04_0011.h5 53 | 04_0012.h5 54 | 04_0013.h5 55 | 04_0046.h5 56 | 04_0050.h5 57 | 05_0017.h5 58 | 05_0018.h5 59 | 05_0019.h5 60 | 05_0020.h5 61 | 05_0021.h5 62 | 05_0022.h5 63 | 05_0023.h5 64 | 05_0024.h5 65 | 06_0144.h5 66 | 06_0145.h5 67 | 06_0147.h5 68 | 06_0150.h5 69 | 06_0153.h5 70 | 06_0155.h5 71 | 07_0005.h5 72 | 07_0006.h5 73 | 07_0007.h5 74 | 07_0008.h5 75 | 07_0009.h5 76 | 07_0047.h5 77 | 07_0048.h5 78 | 07_0049.h5 79 | 08_0044.h5 80 | 08_0058.h5 81 | 08_0077.h5 82 | 08_0078.h5 83 | 08_0079.h5 84 | 08_0080.h5 85 | 08_0156.h5 86 | 08_0157.h5 87 | 08_0158.h5 88 | 08_0159.h5 89 | 08_0178.h5 90 | 08_0179.h5 91 | 09_0057.h5 92 | 10_0037.h5 93 | 10_0038.h5 94 | 10_0042.h5 95 | 10_0074.h5 96 | 10_0075.h5 97 | 11_0176.h5 98 | 12_0142.h5 99 | 12_0143.h5 100 | 12_0148.h5 101 | 12_0149.h5 102 | 12_0151.h5 103 | 12_0152.h5 104 | 12_0154.h5 105 | 12_0173.h5 106 | 12_0174.h5 107 | 12_0175.h5 108 | -------------------------------------------------------------------------------- /txt/shanghaitech_feature_training.txt: -------------------------------------------------------------------------------- 1 | 01_001.h5 2 | 01_002.h5 3 | 01_003.h5 4 | 01_004.h5 5 | 01_005.h5 6 | 01_006.h5 7 | 01_007.h5 8 | 01_008.h5 9 | 01_009.h5 10 | 01_010.h5 11 | 01_011.h5 12 | 01_012.h5 13 | 01_013.h5 14 | 01_014.h5 15 | 01_015.h5 16 | 01_016.h5 17 | 01_017.h5 18 | 01_018.h5 19 | 01_019.h5 20 | 01_020.h5 21 | 01_021.h5 22 | 01_022.h5 23 | 01_023.h5 24 | 01_024.h5 25 | 01_025.h5 26 | 01_026.h5 27 | 01_027.h5 28 | 01_028.h5 29 | 01_029.h5 30 | 01_030.h5 31 | 01_031.h5 32 | 01_032.h5 33 | 01_033.h5 34 | 01_034.h5 35 | 01_035.h5 36 | 01_036.h5 37 | 01_037.h5 38 | 01_038.h5 39 | 01_039.h5 40 | 01_040.h5 41 | 01_041.h5 42 | 01_042.h5 43 | 01_043.h5 44 | 01_044.h5 45 | 01_045.h5 46 | 01_046.h5 47 | 01_047.h5 48 | 01_048.h5 49 | 01_049.h5 50 | 01_050.h5 51 | 01_051.h5 52 | 01_052.h5 53 | 01_053.h5 54 | 01_054.h5 55 | 01_055.h5 56 | 01_056.h5 57 | 01_057.h5 58 | 01_058.h5 59 | 01_059.h5 60 | 01_060.h5 61 | 01_061.h5 62 | 01_062.h5 63 | 01_063.h5 64 | 01_064.h5 65 | 01_065.h5 66 | 01_066.h5 67 | 01_067.h5 68 | 01_068.h5 69 | 01_069.h5 70 | 01_070.h5 71 | 01_071.h5 72 | 01_072.h5 73 | 01_073.h5 74 | 01_074.h5 75 | 01_075.h5 76 | 01_076.h5 77 | 01_077.h5 78 | 01_078.h5 79 | 01_079.h5 80 | 01_080.h5 81 | 01_081.h5 82 | 01_082.h5 83 | 01_083.h5 84 | 02_001.h5 85 | 02_002.h5 86 | 02_003.h5 87 | 02_004.h5 88 | 02_005.h5 89 | 02_006.h5 90 | 02_007.h5 91 | 02_008.h5 92 | 02_009.h5 93 | 02_010.h5 94 | 02_011.h5 95 | 02_012.h5 96 | 02_013.h5 97 | 02_014.h5 98 | 02_015.h5 99 | 03_001.h5 100 | 03_002.h5 101 | 03_003.h5 102 | 03_004.h5 103 | 03_005.h5 104 | 03_006.h5 105 | 04_001.h5 106 | 04_002.h5 107 | 04_003.h5 108 | 04_004.h5 109 | 04_005.h5 110 | 04_006.h5 111 | 04_007.h5 112 | 04_008.h5 113 | 04_009.h5 114 | 04_010.h5 115 | 04_011.h5 116 | 04_012.h5 117 | 04_013.h5 118 | 04_014.h5 119 | 04_015.h5 120 | 04_016.h5 121 | 04_017.h5 122 | 04_018.h5 123 | 04_019.h5 124 | 04_020.h5 125 | 05_001.h5 126 | 05_002.h5 127 | 05_003.h5 128 | 05_004.h5 129 | 05_005.h5 130 | 05_006.h5 131 | 05_007.h5 132 | 05_008.h5 133 | 05_009.h5 134 | 05_010.h5 135 | 05_011.h5 136 | 05_012.h5 137 | 05_013.h5 138 | 05_014.h5 139 | 05_015.h5 140 | 05_016.h5 141 | 05_017.h5 142 | 05_018.h5 143 | 05_019.h5 144 | 05_020.h5 145 | 05_021.h5 146 | 05_022.h5 147 | 05_023.h5 148 | 05_024.h5 149 | 05_025.h5 150 | 05_026.h5 151 | 05_027.h5 152 | 05_028.h5 153 | 05_029.h5 154 | 05_030.h5 155 | 05_031.h5 156 | 05_032.h5 157 | 05_033.h5 158 | 05_034.h5 159 | 05_035.h5 160 | 05_036.h5 161 | 05_037.h5 162 | 05_038.h5 163 | 05_039.h5 164 | 05_040.h5 165 | 05_041.h5 166 | 05_042.h5 167 | 05_043.h5 168 | 05_044.h5 169 | 05_045.h5 170 | 05_046.h5 171 | 05_047.h5 172 | 05_048.h5 173 | 05_049.h5 174 | 05_050.h5 175 | 05_051.h5 176 | 05_052.h5 177 | 05_053.h5 178 | 05_054.h5 179 | 05_055.h5 180 | 05_056.h5 181 | 05_057.h5 182 | 05_058.h5 183 | 05_059.h5 184 | 05_060.h5 185 | 05_061.h5 186 | 05_062.h5 187 | 05_063.h5 188 | 05_064.h5 189 | 05_065.h5 190 | 05_066.h5 191 | 06_001.h5 192 | 06_002.h5 193 | 06_003.h5 194 | 06_004.h5 195 | 06_005.h5 196 | 06_006.h5 197 | 06_007.h5 198 | 06_008.h5 199 | 06_009.h5 200 | 06_010.h5 201 | 06_011.h5 202 | 06_012.h5 203 | 06_013.h5 204 | 06_014.h5 205 | 06_015.h5 206 | 06_016.h5 207 | 06_017.h5 208 | 06_018.h5 209 | 06_019.h5 210 | 06_020.h5 211 | 06_021.h5 212 | 06_022.h5 213 | 06_023.h5 214 | 06_024.h5 215 | 06_025.h5 216 | 06_026.h5 217 | 06_027.h5 218 | 06_028.h5 219 | 06_029.h5 220 | 06_030.h5 221 | 07_001.h5 222 | 07_002.h5 223 | 07_003.h5 224 | 07_004.h5 225 | 07_005.h5 226 | 07_006.h5 227 | 07_007.h5 228 | 07_008.h5 229 | 08_001.h5 230 | 08_002.h5 231 | 08_003.h5 232 | 08_004.h5 233 | 08_005.h5 234 | 08_006.h5 235 | 08_007.h5 236 | 08_008.h5 237 | 08_009.h5 238 | 08_010.h5 239 | 08_011.h5 240 | 08_012.h5 241 | 08_013.h5 242 | 08_014.h5 243 | 08_015.h5 244 | 08_016.h5 245 | 08_017.h5 246 | 08_018.h5 247 | 08_019.h5 248 | 08_020.h5 249 | 08_021.h5 250 | 08_022.h5 251 | 08_023.h5 252 | 08_024.h5 253 | 08_025.h5 254 | 08_026.h5 255 | 08_027.h5 256 | 08_028.h5 257 | 08_029.h5 258 | 08_030.h5 259 | 08_031.h5 260 | 08_032.h5 261 | 08_033.h5 262 | 08_034.h5 263 | 08_035.h5 264 | 08_036.h5 265 | 08_037.h5 266 | 08_038.h5 267 | 08_039.h5 268 | 08_040.h5 269 | 08_041.h5 270 | 08_042.h5 271 | 08_043.h5 272 | 08_044.h5 273 | 08_045.h5 274 | 08_046.h5 275 | 08_047.h5 276 | 08_048.h5 277 | 08_049.h5 278 | 09_001.h5 279 | 09_002.h5 280 | 09_003.h5 281 | 09_004.h5 282 | 09_005.h5 283 | 09_006.h5 284 | 09_007.h5 285 | 09_008.h5 286 | 09_009.h5 287 | 09_010.h5 288 | 10_001.h5 289 | 10_002.h5 290 | 10_003.h5 291 | 10_004.h5 292 | 10_005.h5 293 | 10_006.h5 294 | 10_007.h5 295 | 10_008.h5 296 | 10_009.h5 297 | 10_010.h5 298 | 10_011.h5 299 | 11_001.h5 300 | 11_002.h5 301 | 11_003.h5 302 | 11_004.h5 303 | 11_005.h5 304 | 11_006.h5 305 | 11_007.h5 306 | 11_008.h5 307 | 11_009.h5 308 | 11_010.h5 309 | 12_001.h5 310 | 12_002.h5 311 | 12_003.h5 312 | 12_004.h5 313 | 12_005.h5 314 | 12_006.h5 315 | 12_007.h5 316 | 12_008.h5 317 | 12_009.h5 318 | 12_010.h5 319 | 12_011.h5 320 | 12_012.h5 321 | 12_013.h5 322 | 12_014.h5 323 | 12_015.h5 324 | 13_001.h5 325 | 13_002.h5 326 | 13_003.h5 327 | 13_004.h5 328 | 13_005.h5 329 | 13_006.h5 330 | 13_007.h5 331 | --------------------------------------------------------------------------------