├── .gitignore ├── LICENSE ├── README.md ├── brightness.py ├── dataset_preprocessing ├── annotations_multicam.json ├── dataset_statistics.py ├── delays_multicam.json ├── generate_Multicam_OF.py ├── generate_OF_FDD.py ├── generate_OF_URFD.py ├── preprocess_FDD_videos.py ├── preprocess_Multicam_videos.py └── preprocess_URFD_images.py ├── requirements.txt ├── temporalnet_combined.py ├── temporalnet_fdd.py ├── temporalnet_multicam.py └── temporalnet_urfd.py /.gitignore: -------------------------------------------------------------------------------- 1 | features* 2 | labels* 3 | models 4 | models/* 5 | weights 6 | weights/* 7 | plots 8 | plots/* 9 | weights.h5 10 | flow_mean.mat 11 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Adrián 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Fall-Detection-with-CNNs-and-Optical-Flow 2 | 3 | This repository contains the code for our paper: 4 | 5 | ``` 6 | Núñez-Marcos, A., Azkune, G., & Arganda-Carreras, I. (2017). 7 | "Vision-Based Fall Detection with Convolutional Neural Networks" 8 | Wireless Communications and Mobile Computing, 2017. 9 | ``` 10 | 11 | If you find the code useful for your research, please, cite our paper: 12 | 13 | ``` 14 | @article{nunez2017vision, 15 | title={Vision-Based Fall Detection with Convolutional Neural Networks}, 16 | author={N{\'u}{\~n}ez-Marcos, Adri{\'a}n and Azkune, Gorka and Arganda-Carreras, Ignacio}, 17 | journal={Wireless Communications and Mobile Computing}, 18 | volume={2017}, 19 | year={2017}, 20 | publisher={Hindawi} 21 | } 22 | 23 | ``` 24 | 25 | ### Updates 26 | 27 | * 4/1/19: A bug was fixed and the results have improved. New features and labels files uploaded for URFD (links below). Optical flow components used for the training and the checkpoints to load the model trained on these images are also to the repository. 28 | 29 | ### Getting started 30 | 31 | The repository contains the following files: 32 | 33 | * **temporalnetxxx.py**: there are four scripts with this name, three of those are specific to a dataset (URFD, FDD and Multicam) and the remaining one, **temporalnetgeneral.py**, is the code for the experiment shown in the section 4.6 of the paper (a multi-tasking training with the previous three datasets). The four scripts first feed optical flow stacks through a VGG16 network to extract features (downloadable below). Then, they fine-tune two fully-connected layers with those features to correctly classify falls and daily living actions (not falls). 34 | 35 | * **brightness.py** contains the script to generate the images used in the experiment of section 4.5 of the paper. 36 | 37 | * **requirements.txt**: file with all the necessary python packages (install via pip). 38 | 39 | ___ 40 | 41 | ### Reproducing the experiments 42 | 43 | Necessary files if you want to re-train the network or use your own dataset: 44 | 45 | * [Weights of the VGG16 network](https://drive.google.com/file/d/0B4i3D0pfGJjYNWxYTVUtNGtRcUE/view?usp=sharing). Weights of a pre-trained VGG16 in the UCF101 Action Recognition dataset using optical flow stacks. In the temporalneturfd.py, temporalnetfdd.py, temporalnetmulticam.py and temporalnetgeneral.py scripts, there is a variable called 'vgg_16_weights' to set the paths to this weights file. 46 | * [Mean file](https://drive.google.com/file/d/1pPIArqld82TgTJuBHEibppkc-YLvQmVk/view?usp=sharing). In the temporalneturfd.py, temporalnetfdd.py, temporalnetmulticam.py and temporalnetgeneral.py scripts there is a variable called 'mean_file' to set the paths to this file. Used as a normalisation for the input of the network. 47 | 48 | All the experiments were done under Ubuntu 16.04 Operating System, using Python 2.7 and OpenCV 3.1.0. All the necessary python packages have been included in the **requirements.txt** file. Use the following command to install them in your virtual environment: 49 | 50 | ``` 51 | pip install -r requirements.txt 52 | ``` 53 | 54 | #### 0. Using your own dataset 55 | 56 | If you want to use your own dataset you need to have a directory with two subfolders: one called 'Falls' and the other one 'NotFalls'. Inside each of them there should be a folder for each fall/"not fall" video (name does not matter), where the images of the video are stored. Example of the directory tree used: 57 | 58 | ``` 59 | fall-dataset\ 60 | Falls\ 61 | fall0\ 62 | flow_x_00001.jpg 63 | flow_x_00002.jpg 64 | ... 65 | NotFalls\ 66 | notfall0\ 67 | flow_x_00001.jpg 68 | flow_x_00002.jpg 69 | ... 70 | ``` 71 | 72 | #### 1. Download the code and change the paths 73 | 74 | After downloading the code you will have to change the paths to adapt to your system and dataset. For each 'temporalnetxxx.py' script, in the very first lines, we have included several variables. Among them you can find 'data_folder', 'mean_file' and 'vgg_16_weights', that must point to the folder of your dataset, the [mean file](https://drive.google.com/file/d/0B4i3D0pfGJjYTllxc0d2NGUyc28/view?usp=sharing) of the network and the path to the [weights of the VGG16 network](https://drive.google.com/file/d/1J-wHUQtBlypi8yCzZO0G46lQ91G0-sGd/view?usp=sharing). Then, to save the model and the weights after the training you have the paths 'model_file' and 'weights_file'. Finally, 'features_file' and 'labels_file' are the paths to the hdf5 files where the extracted features from the VGG16 CNN are going to be stored. 75 | 76 | After that we include some variables related to the training of the network, such as the batch size or the weight to be applied to the class 0. Then, the variables 'save_plots' and 'save_features', to save the plots created during training and validation and to save the weights extracted from the VGG16, respectively. 'save_features' must be set to True or False depending on what you want to do: 77 | 78 | 1. If you use your own dataset or you want to extract the features by yourself, set 'save_features' to True the first time that the script is used and False after that. Take into account that if you run the script with the True value set, previous features will be erased. 79 | 80 | 2. If you want to load the extracted features, set 'save_features' to False. 81 | 82 | If you have your model already trained, you can load the checkpoints and only do the evaluation part. Set the 'use_checkpoint' variable to True in that case. This skips the training part and obtains the results for each fold, computing at the end the average results. 83 | 84 | #### 2. Executing the code 85 | 86 | Execute the code by calling (depending on the script, change the name): 87 | 88 | ``` 89 | python temporalneturfd.py 90 | ``` 91 | 92 | Using the saved models submitted here you should obtain these results from the 5-fold cross validation: 93 | 94 | Dataset | Sensitivity | Specificity | FAR | MDR | Accuracy 95 | --- | --- | --- | --- | --- | --- 96 | **URFD** | 99.67% (-+0.67%) | 98.57% (-+0.56%) | 1.43% (-+0.56%) | 0.33% (-+0.67%) | 98.63% (-+0.52%) 97 | 98 | #### A. Reproducing the experiment with different lighting conditions (Section 4.5 of the paper) 99 | 100 | 'brightness.py' is required to darken or adding a lighting change to the original images of any dataset. 101 | 102 | 1. Change the variables 'data_folder' and 'output_path' at the top of the script to match with the path to your dataset and the outputh path you desire (no need to be already created). By applying the script, the images inside 'data_folder' path will be transformed (darkened or dynamic light added to them) and will be stored in 'output_path'. 103 | 104 | 2. The 'mode' variable can have the value 'darken' to obtain the images for the experiment 4.5.1 (darkened images). Any other value will retrieve the images for the experiment 4.5.2 (lighting change). 105 | 106 | 3. You can also change the amount of darkness is added and the intensity of the added dynamic light applied. The variables 'darkness' and 'brightness' will control them, respectively. 107 | 108 | ### Extracted features and labels 109 | 110 | These are the features extracted from the fc6 layer of the VGG16 network used for fine-tuning (vectors of size 4096). The instances are ordered so that the i-th element in the feature file has the i-th label of the labels file. For the Multicam dataset, the division inside each file is done by scenes, then by cameras. 111 | 112 | * UR Fall Dataset (URFD) 113 | * [Features](https://drive.google.com/file/d/1JQg6mCrV_0lQR0MSUaRlD5VusF5MLRhB/view?usp=sharing) 114 | * [Labels](https://drive.google.com/file/d/1EKTpI7BzlX4qQoAyph5d5cJ1jnWU3f6n/view?usp=sharing) 115 | * Multiple Cameras Fall Dataset (Multicam)* 116 | * [Features](https://drive.google.com/file/d/1Kfbm1RiKUr5q6S7Mq4LqTYGRyKyY_F91/view?usp=sharing) 117 | * [Labels](https://drive.google.com/file/d/1krNC_QbGD4vE6XwEnuUdajtYy4_o4iaJ/view?usp=sharing) 118 | * Fall Detection Dataset (FDD) 119 | * [Features](https://drive.google.com/file/d/0B4i3D0pfGJjYSXN6aW82MjhtSkE/view?usp=sharing) 120 | * [Labels](https://drive.google.com/file/d/0B4i3D0pfGJjYdTE4R2tYdHhLOXc/view?usp=sharing) 121 | 122 | *Note: due to problems with data storage, we could not submit the original features used in the paper. 123 | 124 | ### Optical flow images 125 | 126 | We make downloadable the images (prepared to be converted to optical flow images) and the optical flow components used for the experiments in case you want to re-train the network: 127 | 128 | * [[URFD prepared images](https://drive.google.com/file/d/1V602ukgTI1biAxZmvle6DGPW7-i4436i/view?usp=sharing)][[URFD optical flow images](https://drive.google.com/file/d/1YhBljXOFHdqukZW0Zp6TPbQ-brsoz7ep/view?usp=sharing)] 129 | 130 | ### Checkpoints 131 | 132 | In order to reproduce the results, we make the checkpoints of each fold available here: 133 | 134 | * URFD [[Fold1](https://drive.google.com/file/d/16usFQzITk0IsC29bTpLbI9BfghLzK6LV/view?usp=sharing)][[Fold2](https://drive.google.com/file/d/12eXc4YLse2vRUYpiJOg9nUCAzOWef2Cf/view?usp=sharing)][[Fold3](https://drive.google.com/file/d/1zROWzLwnChSJb_W0stZ4SdUoFtv9mqjy/view?usp=sharing)][[Fold4](https://drive.google.com/file/d/1COZzqGFG124IuOomHFM20L7jOqEsgCYw/view?usp=sharing)][[Fold5](https://drive.google.com/file/d/10-MUWZYw1nf-UcrpQRVgq307qRfGEvGE/view?usp=sharing)][[All](https://drive.google.com/file/d/1PQ410mJXhGTbM256AgITNYyR5noh1xJW/view?usp=sharing)] 135 | -------------------------------------------------------------------------------- /brightness.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import os 3 | import sys 4 | import numpy as np 5 | 6 | ## VARIABLES TO CHANGE 7 | data_folder = 'FDD_Fall/' 8 | output_path = 'FDD_Fall_Dynamic/' 9 | mode = 'darken' 10 | 11 | if not os.path.exists(output_path): 12 | os.makedirs(output_path) 13 | 14 | # EXPERIMENT 4.5.1: Matrix to be subtracted to the images to make them darker 15 | darkness = 100 16 | darkness_add = np.zeros((224,224,3)) 17 | darkness_add.fill(brightness) 18 | 19 | # EXPERIMENT 4.5.2: Lighting change, the sin function is applied to the variable n, step is added to n after each frame 20 | # n_start is the starting value of n 21 | # Therefore, n=n_start in the frame 0, n=n_start+step in frame 1, n=n_start+2*step in frame 2 and so on 22 | # The final transformation is the matrix brightness*sin(n) added to the original image 23 | brightness = 50 24 | brightness_add = np.zeros((224,224,3)) 25 | brightness_add.fill(brightness) 26 | 27 | step = 0.1 28 | n_start = 0 29 | 30 | # Process the folders inside the 'Falls' directory 31 | folders = [f for f in os.listdir(data_folder + 'Falls') if os.path.isdir(os.path.join(data_folder + 'Falls', f))] 32 | for folder in folders: 33 | images = [f for f in os.listdir(data_folder + 'Falls/' + folder) if os.path.isfile(os.path.join(data_folder + 'Falls/' + folder, f))] 34 | images.sort() 35 | n = n_start 36 | stop = False 37 | 38 | # Process all the images of a video 39 | for image, nb_image in zip(images, range(len(images))): 40 | x = cv2.imread(data_folder + 'Falls/' + folder + '/' + image) 41 | if mode == 'darken': 42 | x = np.add(x, darkness_add) 43 | else: 44 | # if sin(n) is below 0 (the light is switched off) remove it 45 | if brightness*np.sin(n) < 0: 46 | stop = True 47 | if stop: 48 | x = np.add(x, brightness_add*np.sin(n)) 49 | n += step 50 | if not os.path.exists(output_path + 'Falls/' + folder): 51 | os.makedirs(output_path + 'Falls/' + folder) 52 | cv2.imwrite(output_path + 'Falls/' + folder + '/' + image, x, [int(cv2.IMWRITE_JPEG_QUALITY), 95]) 53 | 54 | # Process the folders inside the 'NotFalls' directory 55 | folders = [f for f in os.listdir(data_folder + 'NotFalls') if os.path.isdir(os.path.join(data_folder + 'NotFalls', f))] 56 | for folder in folders: 57 | images = [f for f in os.listdir(data_folder + 'NotFalls/' + folder) if os.path.isfile(os.path.join(data_folder + 'NotFalls/' + folder, f))] 58 | images.sort() 59 | n = 0 60 | stop = True 61 | # Process all the images of a video 62 | for image, nb_image in zip(images, range(len(images))): 63 | x = cv2.imread(data_folder + 'NotFalls/' + folder + '/' + image) 64 | if mode == 'darken': 65 | x = np.add(x, darkness_add) 66 | else: 67 | # if sin(n) is below 0 (the light is switched off) remove it 68 | if brightness*np.sin(n) < 0: 69 | stop = True 70 | if stop: 71 | x = np.add(x, brightness_add*np.sin(n)) 72 | n += step 73 | if not os.path.exists(output_path + 'NotFalls/' + folder): 74 | os.makedirs(output_path + 'NotFalls/' + folder) 75 | cv2.imwrite(output_path + 'NotFalls/' + folder + '/' + image, x, [int(cv2.IMWRITE_JPEG_QUALITY), 95]) 76 | 77 | print('=== END ===') 78 | -------------------------------------------------------------------------------- /dataset_preprocessing/annotations_multicam.json: -------------------------------------------------------------------------------- 1 | { 2 | "scenario1": { 3 | "reference": 11, 4 | "start": [1080], 5 | "end": [1108] 6 | }, 7 | "scenario2": { 8 | "reference": 4, 9 | "start": [375], 10 | "end": [399] 11 | }, 12 | "scenario3": { 13 | "reference": 11, 14 | "start": [591], 15 | "end": [625] 16 | }, 17 | "scenario4": { 18 | "reference": 6, 19 | "start": [288], 20 | "end": [314] 21 | }, 22 | "scenario5": { 23 | "reference": 11, 24 | "start": [311], 25 | "end": [336] 26 | }, 27 | "scenario6": { 28 | "reference": 1, 29 | "start": [583], 30 | "end": [629] 31 | }, 32 | "scenario7": { 33 | "reference": 6, 34 | "start": [476], 35 | "end": [507] 36 | }, 37 | "scenario8": { 38 | "reference": 4, 39 | "start": [271], 40 | "end": [298] 41 | }, 42 | "scenario9": { 43 | "reference": 11, 44 | "start": [628], 45 | "end": [651] 46 | }, 47 | "scenario10": { 48 | "reference": 11, 49 | "start": [512], 50 | "end": [530] 51 | }, 52 | "scenario11": { 53 | "reference": 7, 54 | "start": [464], 55 | "end": [489] 56 | }, 57 | "scenario12": { 58 | "reference": 11, 59 | "start": [605], 60 | "end": [653] 61 | }, 62 | "scenario13": { 63 | "reference": 4, 64 | "start": [823], 65 | "end": [863] 66 | }, 67 | "scenario14": { 68 | "reference": 6, 69 | "start": [989], 70 | "end": [1023] 71 | }, 72 | "scenario15": { 73 | "reference": 11, 74 | "start": [755], 75 | "end": [787] 76 | }, 77 | "scenario16": { 78 | "reference": 4, 79 | "start": [891], 80 | "end": [940] 81 | }, 82 | "scenario17": { 83 | "reference": 6, 84 | "start": [730], 85 | "end": [770] 86 | }, 87 | "scenario18": { 88 | "reference": 6, 89 | "start": [571], 90 | "end": [601] 91 | }, 92 | "scenario19": { 93 | "reference": 10, 94 | "start": [499], 95 | "end": [600] 96 | }, 97 | "scenario20": { 98 | "reference": 11, 99 | "start": [545], 100 | "end": [672] 101 | }, 102 | "scenario21": { 103 | "reference": 11, 104 | "start": [864], 105 | "end": [901] 106 | }, 107 | "scenario22": { 108 | "reference": 1, 109 | "start": [767], 110 | "end": [808] 111 | }, 112 | "scenario23": { 113 | "reference": 11, 114 | "start": [1520,3574], 115 | "end": [1595,3614] 116 | }, 117 | "scenario24": { 118 | "reference": 6, 119 | "start": [], 120 | "end": [] 121 | } 122 | } 123 | -------------------------------------------------------------------------------- /dataset_preprocessing/dataset_statistics.py: -------------------------------------------------------------------------------- 1 | import glob 2 | 3 | urfd_path = '/home/anunez/Downloads/URFD_OF/' 4 | multicam_path = '/home/anunez/Downloads/Multicam_OF/' 5 | fdd_path = '/home/anunez/Downloads/FDD_OF/' 6 | L = 10 7 | 8 | content = [ 9 | ['URFD', urfd_path], 10 | ['FDD', fdd_path], 11 | ['Multicam', multicam_path] 12 | ] 13 | 14 | # URFD, FDD 15 | for dataset_name, dataset_path in content[:2]: 16 | falls = len(glob.glob(dataset_path + 'Falls/*')) 17 | notfalls = len(glob.glob(dataset_path + 'NotFalls/*')) 18 | print('-'*10) 19 | print(dataset_name) 20 | print('-'*10) 21 | print('Fall sequences: {}, ADL sequences: {} (Total: {})'.format( 22 | falls, notfalls, falls + notfalls) 23 | ) 24 | 25 | falls = glob.glob(dataset_path + 'Falls/*') 26 | fall_stacks = sum( 27 | [len(glob.glob(fall +'/*')) - L + 1 for fall in falls] 28 | ) 29 | fall_frames = sum( 30 | [len(glob.glob(fall +'/*')) for fall in falls] 31 | ) 32 | notfalls = glob.glob(dataset_path + 'NotFalls/*') 33 | nofall_stacks = sum( 34 | [len(glob.glob(notfall +'/*')) - L + 1 for notfall in notfalls]) 35 | nofall_frames = sum( 36 | [len(glob.glob(notfall +'/*')) for notfall in notfalls]) 37 | 38 | print('Fall stacks: {}, ADL stacks: {} (Total: {})'.format( 39 | fall_stacks, nofall_stacks, fall_stacks + nofall_stacks) 40 | ) 41 | print('Fall frames: {}, ADL frames: {} (Total: {})\n\n'.format( 42 | fall_frames, nofall_frames,fall_frames + nofall_frames) 43 | ) 44 | 45 | # Multicam 46 | dataset_name, dataset_path = content[2] 47 | print('-'*10) 48 | print(dataset_name) 49 | print('-'*10) 50 | scenarios = glob.glob(dataset_path + '*') 51 | nb_falls, nb_notfalls = 0, 0 52 | nb_fall_stacks, nb_notfall_stacks = 0, 0 53 | nb_fall_frames, nb_notfall_frames = 0, 0 54 | for scenario in scenarios: 55 | nb_falls += len(glob.glob(scenario + '/Falls/*')) 56 | nb_notfalls += len(glob.glob(scenario + '/NotFalls/*')) 57 | 58 | falls = glob.glob(scenario + '/Falls/*') 59 | nb_fall_stacks += sum( 60 | [len(glob.glob(fall +'/*')) - L + 1 for fall in falls] 61 | ) 62 | nb_fall_frames += sum( 63 | [len(glob.glob(fall +'/*')) for fall in falls] 64 | ) 65 | notfalls = glob.glob(scenario + '/NotFalls/*') 66 | nb_notfall_stacks += sum( 67 | [len(glob.glob(notfall +'/*')) - L + 1 for notfall in notfalls]) 68 | nb_notfall_frames += sum( 69 | [len(glob.glob(notfall +'/*')) for notfall in notfalls]) 70 | 71 | print('Fall sequences: {}, ADL sequences: {} (Total: {})'.format( 72 | nb_falls, nb_notfalls, nb_falls + nb_notfalls) 73 | ) 74 | 75 | print('Fall stacks: {}, ADL stacks: {} (Total: {})'.format( 76 | nb_fall_stacks, nb_notfall_stacks, nb_fall_stacks + nb_notfall_stacks) 77 | ) 78 | 79 | print('Fall frames: {}, ADL frames: {} (Total: {})\n\n'.format( 80 | nb_fall_frames, nb_notfall_frames, nb_fall_frames + nb_notfall_frames) 81 | ) -------------------------------------------------------------------------------- /dataset_preprocessing/delays_multicam.json: -------------------------------------------------------------------------------- 1 | { 2 | "camera1": { 3 | "1": 3, 4 | "2": 25, 5 | "3": 12, 6 | "4": 72, 7 | "5": 17, 8 | "6": 0, 9 | "7": 28, 10 | "8": 92, 11 | "9": 18, 12 | "10": 14, 13 | "11": 23, 14 | "12": 21, 15 | "13": 16, 16 | "14": 49, 17 | "15": 15, 18 | "16": 23, 19 | "17": 21, 20 | "18": 99, 21 | "19": 19, 22 | "20": 25, 23 | "21": 20, 24 | "22": 0, 25 | "23": 31, 26 | "24": 3 27 | }, 28 | "camera2": { 29 | "1": 3, 30 | "2": 40, 31 | "3": 16, 32 | "4": 79, 33 | "5": 24, 34 | "6": 100, 35 | "7": 14, 36 | "8": 79, 37 | "9": 9, 38 | "10": 15, 39 | "11": 4, 40 | "12": 16, 41 | "13": 33, 42 | "14": 36, 43 | "15": 19, 44 | "16": 29, 45 | "17": 26, 46 | "18": 105, 47 | "19": 27, 48 | "20": 9, 49 | "21": 30, 50 | "22": 46, 51 | "23": 52, 52 | "24": 36 53 | }, 54 | "camera3": { 55 | "1": 8, 56 | "2": 0, 57 | "3": 8, 58 | "4": 78, 59 | "5": 5, 60 | "6": 106, 61 | "7": 16, 62 | "8": 0, 63 | "9": 1, 64 | "10": 19, 65 | "11": 20, 66 | "12": 13, 67 | "13": 0, 68 | "14": 38, 69 | "15": 19, 70 | "16": 0, 71 | "17": 15, 72 | "18": 86, 73 | "19": 16, 74 | "20": 3, 75 | "21": 22, 76 | "22": 51, 77 | "23": 52, 78 | "24": 7 79 | }, 80 | "camera4": { 81 | "1": 4, 82 | "2": 16, 83 | "3": 16, 84 | "4": 0, 85 | "5": 11, 86 | "6": 90, 87 | "7": 0, 88 | "8": 81, 89 | "9": 19, 90 | "10": 33, 91 | "11": 14, 92 | "12": 8, 93 | "13": 7, 94 | "14": 0, 95 | "15": 15, 96 | "16": 2, 97 | "17": 0, 98 | "18": 0, 99 | "19": 19, 100 | "20": 10, 101 | "21": 3, 102 | "22": 41, 103 | "23": 45, 104 | "24": 0 105 | }, 106 | "camera5": { 107 | "1": 23, 108 | "2": 18, 109 | "3": 35, 110 | "4": 68, 111 | "5": 187, 112 | "6": 89, 113 | "7": 1, 114 | "8": 64, 115 | "9": 13, 116 | "10": 12, 117 | "11": 0, 118 | "12": 0, 119 | "13": 27, 120 | "14": 29, 121 | "15": 34, 122 | "16": 12, 123 | "17": 10, 124 | "18": 84, 125 | "19": 5, 126 | "20": 10, 127 | "21": 8, 128 | "22": 53, 129 | "23": 54, 130 | "24": 37 131 | }, 132 | "camera6": { 133 | "1": 6, 134 | "2": 33, 135 | "3": 20, 136 | "4": 82, 137 | "5": 26, 138 | "6": 103, 139 | "7": 17, 140 | "8": 81, 141 | "9": 11, 142 | "10": 17, 143 | "11": 6, 144 | "12": 3, 145 | "13": 27, 146 | "14": 29, 147 | "15": 40, 148 | "16": 9, 149 | "17": 0, 150 | "18": 108, 151 | "19": 29, 152 | "20": 4, 153 | "21": 33, 154 | "22": 46, 155 | "23": 60, 156 | "24": 10 157 | }, 158 | "camera7": { 159 | "1": 6, 160 | "2": 33, 161 | "3": 20, 162 | "4": 83, 163 | "5": 28, 164 | "6": 104, 165 | "7": 18, 166 | "8": 82, 167 | "9": 12, 168 | "10": 19, 169 | "11": 7, 170 | "12": 7, 171 | "13": 36, 172 | "14": 7, 173 | "15": 23, 174 | "16": 3, 175 | "17": 29, 176 | "18": 109, 177 | "19": 0, 178 | "20": 5, 179 | "21": 32, 180 | "22": 47, 181 | "23": 50, 182 | "24": 33 183 | }, 184 | "camera8": { 185 | "1": 0, 186 | "2": 6, 187 | "3": 0, 188 | "4": 56, 189 | "5": 0, 190 | "6": 89, 191 | "7": 20, 192 | "8": 56, 193 | "9": 0, 194 | "10": 0, 195 | "11": 12, 196 | "12": 0, 197 | "13": 13, 198 | "14": 14, 199 | "15": 0, 200 | "16": 3, 201 | "17": 18, 202 | "18": 77, 203 | "19": 20, 204 | "20": 0, 205 | "21": 0, 206 | "22": 34, 207 | "23": 0, 208 | "24": 1 209 | } 210 | } 211 | -------------------------------------------------------------------------------- /dataset_preprocessing/generate_Multicam_OF.py: -------------------------------------------------------------------------------- 1 | 2 | import os 3 | import cv2 4 | import glob 5 | import sys 6 | from tqdm import tqdm 7 | 8 | data_folder = 'Multicam_images/' 9 | output_path = 'Multicam_OF/' 10 | 11 | if not os.path.exists(output_path): 12 | os.mkdir(output_path) 13 | 14 | folders = [f for f in os.listdir(data_folder) if os.path.isdir(os.path.join(data_folder, f))] 15 | folders.sort() 16 | for folder in tqdm(folders): 17 | camera_folders = [f for f in os.listdir(data_folder + folder) if os.path.isdir(os.path.join(data_folder + folder + '/', f))] 18 | camera_folders.sort() 19 | for camera_folder in camera_folders: 20 | path = data_folder + folder + '/' + camera_folder 21 | event_folders = [f for f in os.listdir(path) if os.path.isdir(os.path.join(path + '/', f))] 22 | event_folders.sort() 23 | for event_folder in event_folders: 24 | path = data_folder + folder + '/' + camera_folder + '/' + event_folder 25 | flow = output_path + folder + '/' + camera_folder + '/' + event_folder 26 | if not os.path.exists(flow): 27 | os.makedirs(flow) 28 | os.system('dense_flow2/build/extract_cpu -f={} -x={} -y={} -i=tmp/image -b=20 -t=1 -d=0 -s=1 -o=dir'.format(path, flow + '/flow_x', flow + '/flow_y')) -------------------------------------------------------------------------------- /dataset_preprocessing/generate_OF_FDD.py: -------------------------------------------------------------------------------- 1 | 2 | import os 3 | import cv2 4 | import glob 5 | import sys 6 | sys.path.append('/usr/local/lib/python2.7/site-packages/') 7 | #os.system('/home/adrian/dense_flow/build/extract_cpu -f={} -x={} -y={} -i=tmp/image -b 20 -t 1 -d 3 -o=dir'.format('test.avi', '/flow_x', '/flow_y')) 8 | 9 | data_folder = 'FDD_images/' 10 | output_path = 'FDD_OF/' 11 | i = 0 12 | 13 | if not os.path.exists(output_path): 14 | os.mkdir(output_path) 15 | 16 | folders = [f for f in os.listdir(data_folder) if os.path.isdir(os.path.join(data_folder, f))] 17 | folders.sort() 18 | for folder in folders: 19 | event_folders = [f for f in os.listdir(data_folder + folder) if os.path.isdir(os.path.join(data_folder + folder + '/', f))] 20 | event_folders.sort() 21 | for event_folder in event_folders: 22 | path = data_folder + folder + '/' + event_folder 23 | flow = output_path + folder + '/' + event_folder 24 | if not os.path.exists(flow): 25 | os.makedirs(flow) 26 | os.system('/home/anunez/dense_flow2/build/extract_cpu -f={} -x={} -y={} -i=tmp/image -b=20 -t=1 -d=0 -s=1 -o=dir'.format(path, flow + '/flow_x', flow + '/flow_y')) 27 | -------------------------------------------------------------------------------- /dataset_preprocessing/generate_OF_URFD.py: -------------------------------------------------------------------------------- 1 | 2 | import os 3 | import cv2 4 | import glob 5 | import sys 6 | sys.path.append('/usr/local/lib/python2.7/site-packages/') 7 | #os.system('/home/adrian/dense_flow/build/extract_cpu -f={} -x={} -y={} -i=tmp/image -b 20 -t 1 -d 3 -o=dir'.format('test.avi', '/flow_x', '/flow_y')) 8 | 9 | data_folder = '/home/anunez/UR_Fall/' 10 | output_path = '/home/anunez/UR_Fall_opticalflow/' 11 | i = 0 12 | 13 | #activity_folders.sort() 14 | #for actor_folder in actor_folders: 15 | # print(i, actor_folder) 16 | # i += 1 17 | if not os.path.exists(output_path): 18 | os.mkdir(output_path) 19 | #video_folders = [f for f in os.listdir(data_folder + activity_folder + '/Videos') if os.path.isfile(os.path.join(data_folder + activity_folder + '/Videos/', f))] 20 | #print(data_folder + activity_folder + '/Videos') 21 | #video_folders.sort() 22 | #print(len(video_folders)) 23 | # action_folders = [f for f in os.listdir(data_folder + actor_folder) if os.path.isdir(os.path.join(data_folder + actor_folder, f))] 24 | # action_folders.sort() 25 | 26 | 27 | folders = [f for f in os.listdir(data_folder) if os.path.isdir(os.path.join(data_folder, f))] 28 | folders.sort() 29 | for folder in folders: 30 | #if actor_folder != 'Ahmad': continue 31 | event_folders = [f for f in os.listdir(data_folder + folder) if os.path.isdir(os.path.join(data_folder + folder + '/', f))] 32 | event_folders.sort() 33 | for event_folder in event_folders: 34 | 35 | path = data_folder + folder + '/' + event_folder 36 | #vid_file = data_folder + activity_folder + '/Videos/' + video_folder.replace(' ','') 37 | flow = output_path + folder + '/' + event_folder 38 | if not os.path.exists(flow): 39 | os.makedirs(flow) 40 | print(flow, path) 41 | #print(vid_file, flow) 42 | #if not os.path.exists(flow): 43 | # os.makedirs(flow) 44 | #else: 45 | # files = [f for f in os.listdir(flow) if os.path.isfile(os.path.join(flow + '/', f))] 46 | # if len(files) > 0: 47 | # continue 48 | #os.system('/home/adrian/dense_flow/build/extract_cpu -f={} -x={} -y={} -i=tmp/image -b=20 -t=1 -d=0 -s=1 -o=dir'.format(video_folder, flow + '/flow_x', flow + '/flow_y')) 49 | os.system('/home/anunez/dense_flow2/build/extract_cpu -f={} -x={} -y={} -i=tmp/image -b=20 -t=1 -d=0 -s=1 -o=dir'.format(path, flow + '/flow_x', flow + '/flow_y')) 50 | -------------------------------------------------------------------------------- /dataset_preprocessing/preprocess_FDD_videos.py: -------------------------------------------------------------------------------- 1 | import os 2 | import cv2 3 | import sys 4 | import zipfile 5 | 6 | # Path where the videos are stored (in zip format, as in a fresh download) 7 | data_folder = '/home/user/Downloads/' 8 | dst_folder = '' # folder where the dataset is going to be unzipped 9 | output_base_path = 'FDD_images/' 10 | files = ['Coffee_room_01.zip', 'Coffee_room_02.zip', 11 | 'Home_01.zip', 'Home_02.zip'] 12 | W, H = 224, 224 # shape of new images (resize is applied) 13 | 14 | if not os.path.exists(dst_folder): 15 | os.makedirs(dst_folder) 16 | 17 | # Extract zip files (if necessary) 18 | for f in files: 19 | filepath = data_folder + f 20 | zfile = zipfile.ZipFile(filepath) 21 | pos = filepath.rfind('/') 22 | if not os.path.exists(dst_folder + filepath[pos+1:-4]): 23 | zfile.extractall(dst_folder) 24 | 25 | # Process each scenario 26 | scenario_folders = [f for f in os.listdir(dst_folder) 27 | if os.path.isdir(os.path.join(dst_folder, f))] 28 | nb_fall_frames, nb_notfall_frames = 0, 0 29 | num_frames = 0 30 | for scenario_folder in scenario_folders: 31 | path = dst_folder + scenario_folder + '/' 32 | videos = sorted([f for f in os.listdir(path + 'Videos/') 33 | if os.path.isfile(os.path.join(path + 'Videos/', f))]) 34 | annotations = sorted([f for f in os.listdir(path + 'Annotation_files/') 35 | if os.path.isfile(os.path.join(path + 'Annotation_files/', f))]) 36 | 37 | # For each video and its corresponding labels 38 | for video, annotation in zip(videos, annotations): 39 | #print(video, annotation) 40 | lines = open( 41 | path + 'Annotation_files/' + annotation, 'r' 42 | ).readlines() 43 | # Get frames of start and end of the fall (if any, otherwise 0) 44 | fall_start, fall_end = 0, 0 45 | start_found = False 46 | for line in lines: 47 | #print(line, fall_start, fall_end, start_found) 48 | if len(line.split(',')) == 1: 49 | if not start_found: 50 | fall_start = int(line.strip()) 51 | start_found = True 52 | else: 53 | fall_end = int(line.strip()) 54 | break 55 | if fall_end > 0: 56 | num_frames += (fall_end-fall_start+1) 57 | fall_start -= 1 58 | fall_end -= 1 59 | #print(path + 'Videos/' + video) 60 | 61 | cap = cv2.VideoCapture(path + 'Videos/' + video) 62 | length = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) 63 | 64 | # Extract pre-fall frames 65 | num = video[video.find('(')+1:video.find(')')] 66 | output_path = output_base_path + 'NotFalls/{}_folder_video_{}/'.format( 67 | scenario_folder, num) 68 | pos = cap.get(cv2.CAP_PROP_POS_FRAMES) 69 | if not os.path.exists(output_path): os.makedirs(output_path) 70 | while pos < fall_start: 71 | ret, frame = cap.read() 72 | # Save frame 73 | cv2.imwrite(output_path + 'img_{:05d}.jpg'.format(int(pos+1)), 74 | cv2.resize(frame, (W,H)), 75 | [int(cv2.IMWRITE_JPEG_QUALITY), 95]) 76 | nb_notfall_frames += 1 77 | pos = cap.get(cv2.CAP_PROP_POS_FRAMES) 78 | # Extract fall frames 79 | if fall_end > 0: 80 | output_path = output_base_path + 'Falls/{}_folder_video_{}/'.format( 81 | scenario_folder, num) 82 | if not os.path.exists(output_path): os.makedirs(output_path) 83 | #cap.set(cv2.CAP_PROP_POS_FRAMES, fall_start) 84 | 85 | while pos <= fall_end: 86 | #print(scenario_folder,video, length, fall_start, pos, fall_end) 87 | ret, frame = cap.read() 88 | # Save frame 89 | cv2.imwrite(output_path + 'img_{:05d}.jpg'.format(int(pos+1)), 90 | cv2.resize(frame, (W,H)), 91 | [int(cv2.IMWRITE_JPEG_QUALITY), 95]) 92 | nb_fall_frames += 1 93 | pos = cap.get(cv2.CAP_PROP_POS_FRAMES) 94 | # Extract post-fall frames 95 | output_path = output_base_path + 'NotFalls/{}_folder_video_{}/'.format( 96 | scenario_folder, num) 97 | #cap.set(cv2.CAP_PROP_POS_FRAMES, fall_end) 98 | while pos < length: 99 | ret, frame = cap.read() 100 | # Save frame 101 | cv2.imwrite(output_path + 'img_{:05d}.jpg'.format(int(pos+1)), 102 | cv2.resize(frame, (W,H)), 103 | [int(cv2.IMWRITE_JPEG_QUALITY), 95]) 104 | nb_notfall_frames += 1 105 | pos = cap.get(cv2.CAP_PROP_POS_FRAMES) 106 | cap.release() 107 | cv2.destroyAllWindows() 108 | 109 | print('Number of fall frames: {}, not fall frames: {}'.format( 110 | nb_fall_frames, nb_notfall_frames 111 | )) 112 | print(num_frames) -------------------------------------------------------------------------------- /dataset_preprocessing/preprocess_Multicam_videos.py: -------------------------------------------------------------------------------- 1 | import os 2 | import cv2 3 | import sys 4 | import json 5 | import glob 6 | import shutil 7 | import codecs 8 | import zipfile 9 | 10 | # Path where the videos are stored (in zip format, as in a fresh download) 11 | data_folder = '/home/user/Downloads/' 12 | dst_folder = '' # folder where the dataset is going to be unzipped 13 | output_base_path = 'Multicam_images/' 14 | fall_annotations_file = 'annotations_multicam.json' 15 | delays_file = 'delays_multicam.json' 16 | num_scenarios = 24 17 | num_cameras = 8 18 | W, H = 224, 224 # shape of new images (resize is applied) 19 | 20 | # Extract all files and organise them (if necessary) 21 | if len(glob.glob(dst_folder + '*')) == 0: 22 | if not os.path.exists(dst_folder): 23 | os.makedirs(dst_folder) 24 | 25 | filepath = data_folder + 'dataset.zip' 26 | zfile = zipfile.ZipFile(filepath) 27 | pos = filepath.rfind('/') 28 | 29 | zfile.extractall(dst_folder) 30 | dir = glob.glob(dst_folder + '*') 31 | for path in glob.glob(dir[0] + '/*'): 32 | save_path = dst_folder + path[path.rfind('/')+1:] 33 | os.makedirs(save_path) 34 | for filePath in glob.glob(path + '/*'): 35 | shutil.move(filePath, save_path) 36 | shutil.rmtree(dir[0]) 37 | 38 | with open(fall_annotations_file, 'r') as json_file: 39 | annotations = json.load(json_file) 40 | with open(delays_file, 'r') as json_file: 41 | delays = json.load(json_file) 42 | 43 | # For each scenario 44 | for s in range(1,num_scenarios+1): 45 | # Get all videos (one per camera) 46 | videos = glob.glob(dst_folder + 'chute{:02}/*'.format(s)) 47 | videos.sort() 48 | starts = annotations['scenario{}'.format(s)]['start'] 49 | ends = annotations['scenario{}'.format(s)]['end'] 50 | 51 | for cam, video in enumerate(videos, 1): 52 | cap = cv2.VideoCapture(video) 53 | length = int(cap.get(cv2.CAP_PROP_FRAME_COUNT)) 54 | # Delay of this camara for this scenario 55 | delay = delays['camera{}'.format(cam)][str(s)] 56 | for start, end in zip(starts, ends): 57 | # Apply the delay 58 | start += delay 59 | end += delay 60 | cap.set(cv2.CAP_PROP_POS_FRAMES, 0) 61 | 62 | # Read the pre-fall part 63 | pos = 0 64 | while pos < start: 65 | ret, frame = cap.read() 66 | pos += 1 67 | output_path = ( 68 | output_base_path + 69 | 'chute{:02}/NotFalls/camera{}_pre/'.format( 70 | s, cam 71 | )) 72 | if not os.path.exists(output_path): 73 | os.makedirs(output_path) 74 | cv2.imwrite(output_path + 'img_{:05d}.jpg'.format(int(pos)), 75 | cv2.resize(frame, (W,H)), 76 | [int(cv2.IMWRITE_JPEG_QUALITY), 95]) 77 | 78 | # Read the fall part 79 | assert cap.get(cv2.CAP_PROP_POS_FRAMES) == start 80 | while pos <= end: 81 | ret, frame = cap.read() 82 | pos += 1 83 | output_path = ( 84 | output_base_path + 85 | 'chute{:02}/Falls/camera{}/'.format( 86 | s, cam 87 | )) 88 | if not os.path.exists(output_path): 89 | os.makedirs(output_path) 90 | cv2.imwrite(output_path + 'img_{:05d}.jpg'.format(int(pos)), 91 | cv2.resize(frame, (W,H)), 92 | [int(cv2.IMWRITE_JPEG_QUALITY), 95]) 93 | # Read the post-fall part 94 | assert cap.get(cv2.CAP_PROP_POS_FRAMES) == end + 1 95 | while pos < length: 96 | ret, frame = cap.read() 97 | pos += 1 98 | output_path = ( 99 | output_base_path + 100 | 'chute{:02}/NotFalls/camera{}_post/'.format( 101 | s, cam 102 | )) 103 | if not os.path.exists(output_path): 104 | os.makedirs(output_path) 105 | cv2.imwrite(output_path + 'img_{:05d}.jpg'.format(int(pos)), 106 | cv2.resize(frame, (W,H)), 107 | [int(cv2.IMWRITE_JPEG_QUALITY), 95]) 108 | # If there was no fall 109 | if len(starts) == 0 and len(ends) == 0: 110 | pos = 0 111 | while pos < length: 112 | ret, frame = cap.read() 113 | pos += 1 114 | output_path = ( 115 | output_base_path + 116 | 'chute{:02}/NotFalls/camera{}_full/'.format( 117 | s, cam 118 | )) 119 | if not os.path.exists(output_path): 120 | os.makedirs(output_path) 121 | cv2.imwrite(output_path + 'img_{:05d}.jpg'.format(int(pos)), 122 | cv2.resize(frame, (W,H)), 123 | [int(cv2.IMWRITE_JPEG_QUALITY), 95]) -------------------------------------------------------------------------------- /dataset_preprocessing/preprocess_URFD_images.py: -------------------------------------------------------------------------------- 1 | """ 2 | Script used to prepare the URFD dataset: 3 | 4 | 1- It takes the downloaded RGB images of the camera 0, divided into two 5 | folders: 'Falls' and 'ADLs', where all the images of a video (comprised in a 6 | folder) are inside one of those folders. 7 | 8 | 2- It creates another folder for the new dataset with the 'Falls' and 9 | 'NotFalls' folders. All the ADL videos are moved to the 'NotFalls' folder. 10 | The images within the original 'Falls' folder are divided in three stages: 11 | (i) the pre-fall ADL images (they go to the new 'NotFalls' folder), 12 | (ii) the fall itself (goes to the new 'Falls' folder) and 13 | (iii) the post-fall ADL images (to 'NotFalls'). 14 | 15 | All the images are resized to size (W,H) - both W and H are variables of the 16 | script. 17 | 18 | The script should generate all the necessary folders. 19 | """ 20 | 21 | import cv2 22 | import os 23 | import csv 24 | import sys 25 | import glob 26 | import numpy as np 27 | import zipfile 28 | 29 | # Path where the images are stored 30 | downloads_folder = '/home/user/Downloads/' 31 | data_folder = 'URFD_images_not_segmented/' 32 | adl_folder = 'ADLs/' 33 | fall_folder = 'Falls/' 34 | # Path to save the images 35 | output_path = 'URFD_images/' 36 | # Label files, download them from the dataset's site 37 | falls_labels = 'urfall-cam0-falls.csv' 38 | notfalls_labels = 'urfall-cam0-adls.csv' 39 | W, H = 224, 224 # shape of new images (resize is applied) 40 | 41 | # ===================================================================== 42 | # UNZIP THE DATASET 43 | # ===================================================================== 44 | 45 | if not os.path.exists(data_folder): 46 | os.makedirs(data_folder + fall_folder) 47 | os.makedirs(data_folder + adl_folder) 48 | 49 | adl_zipped_files = glob.glob(downloads_folder + 'adl-*-cam0-rgb.zip') 50 | fall_zipped_files = glob.glob(downloads_folder + 'fall-*-cam0-rgb.zip') 51 | 52 | content = [ 53 | [adl_zipped_files, data_folder + adl_folder], 54 | [fall_zipped_files, data_folder + fall_folder] 55 | ] 56 | for zipped_files, dst_folder in content: 57 | for zipped_file in zipped_files: 58 | zfile = zipfile.ZipFile(zipped_file) 59 | zfile.extractall(dst_folder) 60 | 61 | # ===================================================================== 62 | # READ LABELS AND STORE THEM 63 | # ===================================================================== 64 | 65 | labels = {'falls': dict(), 'notfalls': dict()} 66 | 67 | # For falls videos: read the CSV where frame-level labels are given 68 | with open(falls_labels, 'rb') as csvfile: 69 | spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|') 70 | event_type = 'falls' 71 | for row in spamreader: 72 | elems = row[0].split(',') # read a line in the csv 73 | if not elems[0] in labels[event_type]: 74 | labels[event_type][elems[0]] = [] 75 | if int(elems[2]) == 1 or int(elems[2]) == -1: 76 | labels[event_type][elems[0]].append(0) 77 | elif int(elems[2]) == 0: 78 | labels[event_type][elems[0]].append(1) 79 | 80 | # For ADL videos: read the CSV where frame-level labels are given 81 | with open(notfalls_labels, 'rb') as csvfile: 82 | spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|') 83 | event_type ='notfalls' 84 | for row in spamreader: 85 | elems = row[0].split(',') # read a line in the csv 86 | if not elems[0] in labels[event_type]: 87 | labels[event_type][elems[0]] = [] 88 | if int(elems[2]) == 1 or int(elems[2]) == -1: 89 | labels[event_type][elems[0]].append(0) 90 | elif int(elems[2]) == 0: 91 | labels[event_type][elems[0]].append(1) 92 | 93 | print('Label files processed') 94 | 95 | # ===================================================================== 96 | # PROCESS THE DATASET 97 | # ===================================================================== 98 | 99 | if not os.path.exists(output_path): 100 | os.makedirs(output_path + 'Falls') 101 | os.makedirs(output_path + 'NotFalls') 102 | 103 | # Get all folders: each one contains the set of images of the video 104 | folders = [f for f in os.listdir(data_folder) 105 | if os.path.isdir(os.path.join(data_folder, f))] 106 | 107 | for folder in folders: 108 | print('{} videos =============='.format(folder)) 109 | events = [f for f in os.listdir(data_folder + folder) 110 | if os.path.isdir(os.path.join(data_folder + folder, f))] 111 | events.sort() 112 | for nb_event, event, in enumerate(events): 113 | # Create the appropriate folder 114 | if folder == 'ADLs': 115 | event_id = event[:6] 116 | #new_folder = output_path + 'NotFalls/notfall_{}'.format(event_id) 117 | new_folder = output_path + 'NotFalls/{}'.format(event) 118 | if not os.path.exists(new_folder): 119 | os.makedirs(new_folder) 120 | elif folder == 'Falls': 121 | event_id = event[:7] 122 | # "No falls" come before and after the fall, so the respective 123 | # folders must be created 124 | #new_folder = output_path + 'Falls/fall_{}'.format(event_id) 125 | new_folder = output_path + 'Falls/{}'.format(event) 126 | if not os.path.exists(new_folder): 127 | os.makedirs(new_folder) 128 | #folder_created = False 129 | path_to_images = data_folder + folder + '/' + event + '/' 130 | 131 | # Load all the images of the video 132 | images = [f for f in os.listdir(path_to_images) 133 | if os.path.isfile(os.path.join(path_to_images, f))] 134 | images.sort() 135 | fall_detected = False # whether a fall has been detected in the video 136 | for nb_image, image in enumerate(images): 137 | x = cv2.imread(path_to_images + image) 138 | 139 | # If the image is part of an ADL video no fall need to be 140 | # considered 141 | if folder == 'ADLs': 142 | # Save the image 143 | save_path = (output_path + 144 | #'NotFalls/notfall_{}'.format(event_id) + 145 | 'NotFalls/{}'.format(event) + 146 | '/frame{:04}.jpg'.format(nb_image)) 147 | cv2.imwrite(save_path, 148 | cv2.resize(x, (W,H)), 149 | [int(cv2.IMWRITE_JPEG_QUALITY), 95]) 150 | elif folder == 'Falls': 151 | event_type = 'falls' 152 | if labels[event_type][event_id][nb_image] == 0: # ADL 153 | if fall_detected: 154 | # Create another folder for an ADL event, 155 | # i.e. the post-fall ADL event 156 | new_folder = (output_path + 157 | #'NotFalls/notfall_{}_post'.format( 158 | #event_id)) 159 | 'NotFalls/{}_post'.format(event)) 160 | if not os.path.exists(new_folder): 161 | os.makedirs(new_folder) 162 | 163 | save_path = (output_path + 164 | #'NotFalls/notfall_{}_post'.format( 165 | #event_id) + 166 | 'NotFalls/{}_post'.format(event) + 167 | '/frame{:04}.jpg'.format(nb_image)) 168 | else: 169 | new_folder = (output_path + 170 | #'NotFalls/notfall_{}_pre'.format(event_id)) 171 | 'NotFalls/{}_pre'.format(event)) 172 | if not os.path.exists(new_folder): 173 | os.makedirs(new_folder) 174 | save_path = (output_path + 175 | #'NotFalls/notfall_{}_pre'.format( 176 | #event_id) + 177 | 'NotFalls/{}_pre'.format(event) + 178 | '/frame{:04}.jpg'.format(nb_image)) 179 | cv2.imwrite(save_path, 180 | cv2.resize(x, (W,H)), 181 | [int(cv2.IMWRITE_JPEG_QUALITY), 95]) 182 | 183 | elif labels[event_type][event_id][nb_image] == 1: # actual fall 184 | save_path = (output_path + 185 | #'Falls/fall_{}'.format(event_id) + 186 | 'Falls/{}'.format(event) + 187 | '/frame{:04}.jpg'.format(nb_image)) 188 | cv2.imwrite(save_path, 189 | cv2.resize(x, (W,H)), 190 | [int(cv2.IMWRITE_JPEG_QUALITY), 95]) 191 | # If fall is detected in a video set the variable to True 192 | # used to discern between pre- and post-fall ADL events 193 | fall_detected = True 194 | 195 | print('End of the process, all the images stored within the {} folder'.format( 196 | output_path)) 197 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | absl-py==0.2.2 2 | astor==0.6.2 3 | backports.functools-lru-cache==1.5 4 | backports.weakref==1.0.post1 5 | bleach==1.5.0 6 | boto==2.49.0 7 | boto3==1.9.51 8 | botocore==1.12.51 9 | bz2file==0.98 10 | certifi==2018.10.15 11 | chardet==3.0.4 12 | contextlib2==0.5.5 13 | cycler==0.10.0 14 | Cython==0.28.5 15 | docutils==0.14 16 | enum34==1.1.6 17 | funcsigs==1.0.2 18 | futures==3.2.0 19 | gast==0.2.0 20 | gensim==3.6.0 21 | grpcio==1.12.1 22 | h5py==2.8.0 23 | h5pyViewer==0.0.1.6 24 | html5lib==0.9999999 25 | idna==2.7 26 | jmespath==0.9.3 27 | Keras==1.2.0 28 | Keras-Applications==1.0.6 29 | Keras-Preprocessing==1.0.5 30 | kiwisolver==1.0.1 31 | Markdown==2.6.11 32 | matplotlib==2.2.0 33 | mock==2.0.0 34 | numpy==1.14.3 35 | pbr==4.0.3 36 | Pillow==5.2.0 37 | protobuf==3.6.1 38 | psutil==5.4.5 39 | pycocotools==2.0 40 | pyparsing==2.2.0 41 | Pypubsub==4.0.0 42 | python-dateutil==2.7.3 43 | pytz==2018.4 44 | PyYAML==4.2b1 45 | requests==2.20.1 46 | s3transfer==0.1.13 47 | scikit-learn==0.19.1 48 | scipy==1.1.0 49 | six==1.11.0 50 | sklearn==0.0 51 | smart-open==1.7.1 52 | subprocess32==3.5.1 53 | tensorboard==1.12.1 54 | tensorflow==1.12.0 55 | tensorflow-gpu==1.12.0 56 | termcolor==1.1.0 57 | Theano==0.9.0 58 | typing==3.6.6 59 | urllib3==1.24.1 60 | Werkzeug==0.14.1 61 | XlsxWriter==1.0.5 62 | -------------------------------------------------------------------------------- /temporalnet_combined.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | from numpy.random import seed 3 | seed(1) 4 | import numpy as np 5 | import matplotlib 6 | matplotlib.use('Agg') 7 | from matplotlib import pyplot as plt 8 | import os 9 | import h5py 10 | import scipy.io as sio 11 | import cv2 12 | import glob 13 | import gc 14 | 15 | from keras.models import load_model, Model, Sequential 16 | from keras.layers import (Input, Conv2D, MaxPooling2D, Flatten, 17 | Activation, Dense, Dropout, ZeroPadding2D) 18 | from keras.optimizers import Adam 19 | from keras.layers.normalization import BatchNormalization 20 | from keras.callbacks import EarlyStopping, ModelCheckpoint 21 | from keras import backend as K 22 | from sklearn.metrics import confusion_matrix, accuracy_score 23 | from sklearn.model_selection import KFold, StratifiedShuffleSplit 24 | from keras.layers.advanced_activations import ELU 25 | 26 | os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" 27 | os.environ["CUDA_VISIBLE_DEVICES"]="3" 28 | 29 | # CHANGE THESE VARIABLES --- 30 | mean_file = '/home/anunez/flow_mean.mat' 31 | vgg_16_weights = 'weights.h5' 32 | save_plots = True 33 | 34 | # Set to 'True' if you want to restore a previous trained models 35 | # Training is skipped and test is done 36 | use_checkpoint = True 37 | # -------------------------- 38 | 39 | best_model_path = 'models/' 40 | plots_folder = 'plots/' 41 | checkpoint_path = best_model_path + 'fold_' 42 | 43 | saved_files_folder = 'saved_features/' 44 | features_file = { 45 | 'urfd': saved_files_folder + 'features_urfd_tf.h5', 46 | 'multicam': saved_files_folder + 'features_multicam_tf.h5', 47 | 'fdd': saved_files_folder + 'features_fdd_tf.h5', 48 | } 49 | labels_file = { 50 | 'urfd': saved_files_folder + 'labels_urfd_tf.h5', 51 | 'multicam': saved_files_folder + 'labels_multicam_tf.h5', 52 | 'fdd': saved_files_folder + 'labels_fdd_tf.h5', 53 | } 54 | features_key = 'features' 55 | labels_key = 'labels' 56 | 57 | L = 10 58 | num_features = 4096 59 | batch_norm = True 60 | learning_rate = 0.01 61 | mini_batch_size = 2048 62 | weight_0 = 2 63 | epochs = 1000 64 | use_validation = True 65 | # After the training stops, use train+validation to train for 1 epoch 66 | use_val_for_training = False 67 | val_size = 200 68 | # Threshold to classify between positive and negative 69 | threshold = 0.5 70 | 71 | # Name of the experiment 72 | exp = 'multicam_lr{}_batchs{}_batchnorm{}_w0_{}'.format( 73 | learning_rate, 74 | mini_batch_size, 75 | batch_norm, 76 | weight_0 77 | ) 78 | 79 | def plot_training_info(case, metrics, save, history): 80 | ''' 81 | Function to create plots for train and validation loss and accuracy 82 | Input: 83 | * case: name for the plot, an 'accuracy.png' or 'loss.png' 84 | will be concatenated after the name. 85 | * metrics: list of metrics to store: 'loss' and/or 'accuracy' 86 | * save: boolean to store the plots or only show them. 87 | * history: History object returned by the Keras fit function. 88 | ''' 89 | val = False 90 | if 'val_acc' in history and 'val_loss' in history: 91 | val = True 92 | plt.ioff() 93 | if 'accuracy' in metrics: 94 | fig = plt.figure() 95 | plt.plot(history['acc']) 96 | if val: plt.plot(history['val_acc']) 97 | plt.title('model accuracy') 98 | plt.ylabel('accuracy') 99 | plt.xlabel('epoch') 100 | if val: 101 | plt.legend(['train', 'val'], loc='upper left') 102 | else: 103 | plt.legend(['train'], loc='upper left') 104 | if save == True: 105 | plt.savefig(case + 'accuracy.png') 106 | plt.gcf().clear() 107 | else: 108 | plt.show() 109 | plt.close(fig) 110 | 111 | # summarize history for loss 112 | if 'loss' in metrics: 113 | fig = plt.figure() 114 | plt.plot(history['loss']) 115 | if val: plt.plot(history['val_loss']) 116 | plt.title('model loss') 117 | plt.ylabel('loss') 118 | plt.xlabel('epoch') 119 | #plt.ylim(1e-3, 1e-2) 120 | plt.yscale("log") 121 | if val: 122 | plt.legend(['train', 'val'], loc='upper left') 123 | else: 124 | plt.legend(['train'], loc='upper left') 125 | if save == True: 126 | plt.savefig(case + 'loss.png') 127 | plt.gcf().clear() 128 | else: 129 | plt.show() 130 | plt.close(fig) 131 | 132 | def sample_from_dataset(X, y, zeroes, ones): 133 | ''' 134 | Samples from X and y using the indices obtained from the arrays 135 | all0 and all1 taking slices that depend on the fold, the slice_size 136 | the mode. 137 | Input: 138 | * X: array of features 139 | * y: array of labels 140 | * all0: indices of sampled labelled as class 0 in y 141 | * all1: indices of sampled labelled as class 1 in y 142 | * fold: integer, fold number (from the cross-validation) 143 | * slice_size: integer, half of the size of a fold 144 | * mode: 'train' or 'test', used to choose how to slice 145 | ''' 146 | """ if mode == 'train': 147 | s, t = 0, fold*slice_size 148 | s2, t2 = (fold+1)*slice_size, None 149 | temp = np.concatenate(( 150 | np.hstack((all0[s:t], all0[s2:t2])), 151 | np.hstack((all1[s:t], all1[s2:t2])) 152 | )) 153 | elif mode == 'test': 154 | s, t = fold*slice_size, (fold+1)*slice_size 155 | temp = np.concatenate((all0[s:t], all1[s:t])) """ 156 | indices = np.concatenate([zeroes, ones], axis=0) 157 | sampled_X = X[indices] 158 | sampled_y = y[indices] 159 | return sampled_X, sampled_y 160 | 161 | def divide_train_val(zeroes, ones, val_size): 162 | """ sss = StratifiedShuffleSplit(n_splits=1, 163 | test_size=val_size/2, 164 | random_state=7) 165 | indices_0 = sss.split(np.zeros(len(zeroes)), zeroes) 166 | indices_1 = sss.split(np.zeros(len(ones)), ones) 167 | train_indices_0, val_indices_0 = indices_0.next() 168 | train_indices_1, val_indices_1 = indices_1.next() """ 169 | 170 | rand0 = np.random.permutation(len(zeroes)) 171 | train_indices_0 = zeroes[rand0[val_size//2:]] 172 | val_indices_0 = zeroes[rand0[:val_size//2]] 173 | rand1 = np.random.permutation(len(ones)) 174 | train_indices_1 = ones[rand1[val_size//2:]] 175 | val_indices_1 = ones[rand1[:val_size//2]] 176 | 177 | return (train_indices_0, train_indices_1, 178 | val_indices_0, val_indices_1) 179 | 180 | def main(): 181 | 182 | # ======================================================================== 183 | # VGG-16 FEATURE EXTRACTOR 184 | # ======================================================================== 185 | model = Sequential() 186 | 187 | model.add(ZeroPadding2D((1, 1), input_shape=(224, 224, 20))) 188 | model.add(Conv2D(64, (3, 3), activation='relu', name='conv1_1')) 189 | model.add(ZeroPadding2D((1, 1))) 190 | model.add(Conv2D(64, (3, 3), activation='relu', name='conv1_2')) 191 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 192 | 193 | model.add(ZeroPadding2D((1, 1))) 194 | model.add(Conv2D(128, (3, 3), activation='relu', name='conv2_1')) 195 | model.add(ZeroPadding2D((1, 1))) 196 | model.add(Conv2D(128, (3, 3), activation='relu', name='conv2_2')) 197 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 198 | 199 | model.add(ZeroPadding2D((1, 1))) 200 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_1')) 201 | model.add(ZeroPadding2D((1, 1))) 202 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_2')) 203 | model.add(ZeroPadding2D((1, 1))) 204 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_3')) 205 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 206 | 207 | model.add(ZeroPadding2D((1, 1))) 208 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_1')) 209 | model.add(ZeroPadding2D((1, 1))) 210 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_2')) 211 | model.add(ZeroPadding2D((1, 1))) 212 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_3')) 213 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 214 | 215 | model.add(ZeroPadding2D((1, 1))) 216 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_1')) 217 | model.add(ZeroPadding2D((1, 1))) 218 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_2')) 219 | model.add(ZeroPadding2D((1, 1))) 220 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_3')) 221 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 222 | 223 | model.add(Flatten()) 224 | model.add(Dense(num_features, name='fc6', 225 | kernel_initializer='glorot_uniform')) 226 | 227 | # ======================================================================== 228 | # WEIGHT INITIALIZATION 229 | # ======================================================================== 230 | layerscaffe = ['conv1_1', 'conv1_2', 'conv2_1', 'conv2_2', 'conv3_1', 231 | 'conv3_2', 'conv3_3', 'conv4_1', 'conv4_2', 'conv4_3', 232 | 'conv5_1', 'conv5_2', 'conv5_3', 'fc6', 'fc7', 'fc8'] 233 | h5 = h5py.File(vgg_16_weights, 'r') 234 | 235 | layer_dict = dict([(layer.name, layer) for layer in model.layers]) 236 | 237 | # Copy the weights stored in the 'vgg_16_weights' file to the 238 | # feature extractor part of the VGG16 239 | for layer in layerscaffe[:-3]: 240 | w2, b2 = h5['data'][layer]['0'], h5['data'][layer]['1'] 241 | w2 = np.transpose(np.asarray(w2), (2,3,1,0)) 242 | w2 = w2[::-1, ::-1, :, :] 243 | b2 = np.asarray(b2) 244 | layer_dict[layer].set_weights((w2, b2)) 245 | 246 | # Copy the weights of the first fully-connected layer (fc6) 247 | layer = layerscaffe[-3] 248 | w2, b2 = h5['data'][layer]['0'], h5['data'][layer]['1'] 249 | w2 = np.transpose(np.asarray(w2), (1,0)) 250 | b2 = np.asarray(b2) 251 | layer_dict[layer].set_weights((w2, b2)) 252 | 253 | # ============================================================================================================= 254 | # TRAINING 255 | # ============================================================================================================= 256 | 257 | adam = Adam(lr=learning_rate, beta_1=0.9, beta_2=0.999, 258 | epsilon=1e-08) 259 | model.compile(optimizer=adam, loss='categorical_crossentropy', 260 | metrics=['accuracy']) 261 | 262 | compute_metrics = False 263 | compute_roc_curve = False 264 | threshold = 0.5 265 | e = EarlyStopping(monitor='val_loss', min_delta=0, patience=100, verbose=0, mode='auto') 266 | 267 | # Load features and labels per dataset 268 | h5features_multicam = h5py.File(features_file['multicam'], 'r') 269 | h5labels_multicam = h5py.File(labels_file['multicam'], 'r') 270 | h5features_urfd = h5py.File(features_file['urfd'], 'r') 271 | h5labels_urfd = h5py.File(labels_file['urfd'], 'r') 272 | h5features_fdd = h5py.File(features_file['fdd'], 'r') 273 | h5labels_fdd = h5py.File(labels_file['fdd'], 'r') 274 | 275 | # Load Multicam data in a single array 276 | stages = [] 277 | for i in range(1,25): 278 | stages.append('chute{:02}'.format(i)) 279 | 280 | _x = [] 281 | _y = [] 282 | for nb_stage, stage in enumerate(stages): 283 | for nb_cam, cam in enumerate(h5features_multicam[stage].keys()): 284 | for key in h5features_multicam[stage][cam].keys(): 285 | _x.extend([x for x in h5features_multicam[stage][cam][key]]) 286 | _y.extend([x for x in h5labels_multicam[stage][cam][key]]) 287 | """ _x.append(np.asarray(h5features_multicam[stage][cam][key])) 288 | _y.append(np.asarray(h5labels_multicam[stage][cam][key])) """ 289 | 290 | # Load all the datasets into numpy arrays 291 | X_multicam = np.asarray(_x) 292 | y_multicam = np.asarray(_y) 293 | X_urfd = np.asarray(h5features_urfd['features']) 294 | y_urfd = np.asarray(h5labels_urfd['labels']) 295 | X_fdd = np.asarray(h5features_fdd['features']) 296 | y_fdd = np.asarray(h5labels_fdd['labels']) 297 | 298 | # Get the number of samples per class on the smallest dataset: URFD 299 | size_0 = np.asarray(np.where(y_urfd==0)[0]).shape[0] 300 | size_1 = np.asarray(np.where(y_urfd==1)[0]).shape[0] 301 | 302 | # Undersample the FDD and Multicam: take 0s and 1s per dataset and 303 | # undersample each of them separately by random sampling without replacement 304 | # Step 1 305 | all0_multicam = np.asarray(np.where(y_multicam==0)[0]) 306 | all1_multicam = np.asarray(np.where(y_multicam==1)[0]) 307 | all0_urfd = np.asarray(np.where(y_urfd==0)[0]) 308 | all1_urfd = np.asarray(np.where(y_urfd==1)[0]) 309 | all0_fdd = np.asarray(np.where(y_fdd==0)[0]) 310 | all1_fdd = np.asarray(np.where(y_fdd==1)[0]) 311 | 312 | # Step 2 313 | all0_multicam = np.random.choice(all0_multicam, size_0, replace=False) 314 | all1_multicam = np.random.choice(all1_multicam, size_0, replace=False) 315 | all0_urfd = np.random.choice(all0_urfd, size_0, replace=False) 316 | all1_urfd = np.random.choice(all1_urfd, size_0, replace=False) 317 | all0_fdd = np.random.choice(all0_fdd, size_0, replace=False) 318 | all1_fdd = np.random.choice(all1_fdd, size_0, replace=False) 319 | 320 | # Arrays to save the results 321 | sensitivities = { 'combined': [], 'multicam': [], 'urfd': [], 'fdd': [] } 322 | specificities = { 'combined': [], 'multicam': [], 'urfd': [], 'fdd': [] } 323 | 324 | # Use a 5 fold cross-validation 325 | kfold = KFold(n_splits=5, shuffle=True) 326 | kfold0_multicam = kfold.split(all0_multicam) 327 | kfold1_multicam = kfold.split(all1_multicam) 328 | kfold0_urfd = kfold.split(all0_urfd) 329 | kfold1_urfd = kfold.split(all1_urfd) 330 | kfold0_fdd = kfold.split(all0_fdd) 331 | kfold1_fdd = kfold.split(all1_fdd) 332 | 333 | # CROSS-VALIDATION: Stratified partition of the dataset into 334 | # train/test sets 335 | for fold in range(5): 336 | # Get the train and test indices, then get the actual indices 337 | _train0_multicam, _test0_multicam = kfold0_multicam.next() 338 | _train1_multicam, _test1_multicam = kfold1_multicam.next() 339 | train0_multicam = all0_multicam[_train0_multicam] 340 | train1_multicam = all1_multicam[_train1_multicam] 341 | test0_multicam = all0_multicam[_test0_multicam] 342 | test1_multicam = all1_multicam[_test1_multicam] 343 | 344 | _train0_urfd, _test0_urfd = kfold0_urfd.next() 345 | _train1_urfd, _test1_urfd = kfold1_urfd.next() 346 | train0_urfd = all0_urfd[_train0_urfd] 347 | train1_urfd = all1_urfd[_train1_urfd] 348 | test0_urfd = all0_urfd[_test0_urfd] 349 | test1_urfd = all1_urfd[_test1_urfd] 350 | 351 | _train0_fdd, _test0_fdd = kfold0_fdd.next() 352 | _train1_fdd, _test1_fdd = kfold1_fdd.next() 353 | train0_fdd = all0_fdd[_train0_fdd] 354 | train1_fdd = all1_fdd[_train1_fdd] 355 | test0_fdd = all0_fdd[_test0_fdd] 356 | test1_fdd = all1_fdd[_test1_fdd] 357 | 358 | if use_validation: 359 | # Multicam 360 | (train0_multicam, train1_multicam, 361 | val0_multicam, val1_multicam) = divide_train_val( 362 | train0_multicam, train1_multicam, val_size//3 363 | ) 364 | temp = np.concatenate((val0_multicam, val1_multicam)) 365 | X_val_multicam = X_multicam[temp] 366 | y_val_multicam = y_multicam[temp] 367 | # URFD 368 | (train0_urfd, train1_urfd, 369 | val0_urfd, val1_urfd) = divide_train_val( 370 | train0_urfd, train1_urfd, val_size//3 371 | ) 372 | temp = np.concatenate((val0_urfd, val1_urfd)) 373 | X_val_urfd = X_urfd[temp] 374 | y_val_urfd = y_urfd[temp] 375 | # FDD 376 | (train0_fdd, train1_fdd, 377 | val0_fdd, val1_fdd) = divide_train_val( 378 | train0_fdd, train1_fdd, val_size//3 379 | ) 380 | temp = np.concatenate((val0_fdd, val1_fdd)) 381 | X_val_fdd = X_fdd[temp] 382 | y_val_fdd = y_fdd[temp] 383 | 384 | # Join all the datasets 385 | X_val = np.concatenate( 386 | (X_val_multicam, X_val_urfd, X_val_fdd), axis=0 387 | ) 388 | y_val = np.concatenate( 389 | (y_val_multicam, y_val_urfd, y_val_fdd), axis=0 390 | ) 391 | 392 | # Sampling 393 | X_train_multicam, y_train_multicam = sample_from_dataset( 394 | X_multicam, y_multicam, train0_multicam, train1_multicam 395 | ) 396 | X_train_urfd, y_train_urfd = sample_from_dataset( 397 | X_urfd, y_urfd, train0_urfd, train1_urfd 398 | ) 399 | X_train_fdd, y_train_fdd = sample_from_dataset( 400 | X_fdd, y_fdd, train0_fdd, train1_fdd 401 | ) 402 | # Create the evaluation folds for each dataset 403 | X_test_multicam, y_test_multicam = sample_from_dataset( 404 | X_multicam, y_multicam, test0_multicam, test1_multicam 405 | ) 406 | X_test_urfd, y_test_urfd = sample_from_dataset( 407 | X_urfd, y_urfd, test0_urfd, test1_urfd 408 | ) 409 | X_test_fdd, y_test_fdd = sample_from_dataset( 410 | X_fdd, y_fdd, test0_fdd, test1_fdd 411 | ) 412 | 413 | # Join all the datasets 414 | X_train = np.concatenate( 415 | (X_train_multicam, X_train_urfd, X_train_fdd), axis=0 416 | ) 417 | y_train = np.concatenate( 418 | (y_train_multicam, y_train_urfd, y_train_fdd), axis=0 419 | ) 420 | X_test = np.concatenate( 421 | (X_test_multicam, X_test_urfd, X_test_fdd), axis=0 422 | ) 423 | y_test = np.concatenate( 424 | (y_test_multicam, y_test_urfd, y_test_fdd), axis=0 425 | ) 426 | 427 | # ============================================================= 428 | # CLASSIFIER 429 | # ============================================================= 430 | extracted_features = Input( 431 | shape=(num_features,), dtype='float32', name='input' 432 | ) 433 | x = BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001)( 434 | extracted_features 435 | ) 436 | x = Activation('relu')(x) 437 | x = Dropout(0.9)(x) 438 | x = Dense(4096, name='fc2', init='glorot_uniform')(x) 439 | x = BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001)(x) 440 | x = Activation('relu')(x) 441 | x = Dropout(0.8)(x) 442 | x = Dense(1, name='predictions', init='glorot_uniform')(x) 443 | x = Activation('sigmoid')(x) 444 | 445 | classifier = Model( 446 | input=extracted_features, output=x, name='classifier' 447 | ) 448 | fold_best_model_path = best_model_path + 'combined_fold_{}.h5'.format( 449 | fold 450 | ) 451 | classifier.compile( 452 | optimizer=adam, loss='binary_crossentropy', 453 | metrics=['accuracy'] 454 | ) 455 | 456 | if not use_checkpoint: 457 | # ==================== TRAINING ======================== 458 | # weighting of each class: only the fall class gets 459 | # a different weight 460 | class_weight = {0:weight_0, 1: 1} 461 | callbacks = None 462 | if use_validation: 463 | # callback definition 464 | metric = 'val_loss' 465 | e = EarlyStopping( 466 | monitor=metric, min_delta=0,patience=100, 467 | mode='auto' 468 | ) 469 | c = ModelCheckpoint( 470 | fold_best_model_path, 471 | monitor=metric, 472 | save_best_only=True, 473 | save_weights_only=False, mode='auto' 474 | ) 475 | callbacks = [e, c] 476 | validation_data = None 477 | if use_validation: 478 | validation_data = (X_val,y_val) 479 | 480 | _mini_batch_size = mini_batch_size 481 | if mini_batch_size == 0: 482 | _mini_batch_size = X_train.shape[0] 483 | 484 | history = classifier.fit( 485 | X_train, y_train, 486 | validation_data=validation_data, 487 | batch_size=_mini_batch_size, 488 | nb_epoch=epochs, 489 | shuffle='batch', 490 | class_weight=class_weight, 491 | callbacks=callbacks 492 | ) 493 | 494 | if not use_validation: 495 | classifier.save(fold_best_model_path) 496 | 497 | plot_training_info(plots_folder + exp, ['accuracy', 'loss'], 498 | save_plots, history.history) 499 | 500 | if use_validation and use_val_for_training: 501 | classifier = load_model(fold_best_model_path) 502 | 503 | # Use full training set (training+validation) 504 | X_train = np.concatenate((X_train, X_val), axis=0) 505 | y_train = np.concatenate((y_train, y_val), axis=0) 506 | 507 | history = classifier.fit( 508 | X_train, y_train, 509 | validation_data=validation_data, 510 | batch_size=_mini_batch_size, 511 | nb_epoch=epochs, 512 | shuffle='batch', 513 | class_weight=class_weight, 514 | callbacks=callbacks 515 | ) 516 | 517 | classifier.save(fold_best_model_path) 518 | 519 | # ==================== EVALUATION ======================== 520 | 521 | # Load best model 522 | print('Model loaded from checkpoint') 523 | classifier = load_model(fold_best_model_path) 524 | 525 | # Evaluate for the combined test set 526 | predicted = classifier.predict(X_test) 527 | for i in range(len(predicted)): 528 | if predicted[i] < threshold: 529 | predicted[i] = 0 530 | else: 531 | predicted[i] = 1 532 | predicted = np.asarray(predicted).astype(int) 533 | cm = confusion_matrix(y_test, predicted,labels=[0,1]) 534 | tp = cm[0][0] 535 | fn = cm[0][1] 536 | fp = cm[1][0] 537 | tn = cm[1][1] 538 | tpr = tp/float(tp+fn) 539 | fpr = fp/float(fp+tn) 540 | fnr = fn/float(fn+tp) 541 | print('Combined test set') 542 | print('-'*10) 543 | tnr = tn/float(tn+fp) 544 | print('TP: {}, TN: {}, FP: {}, FN: {}'.format(tp,tn,fp,fn)) 545 | print('TPR: {}, TNR: {}, FPR: {}, FNR: {}'.format(tpr,tnr,fpr,fnr)) 546 | print('Sensitivity/Recall: {}'.format(tp/float(tp+fn))) 547 | print('Specificity: {}'.format(tn/float(tn+fp))) 548 | print('Accuracy: {}'.format(accuracy_score(y_test, predicted))) 549 | sensitivities['combined'].append(tp/float(tp+fn)) 550 | specificities['combined'].append(tn/float(tn+fp)) 551 | 552 | # Evaluate for the URFD test set 553 | predicted = classifier.predict(X_test_urfd) 554 | for i in range(len(predicted)): 555 | if predicted[i] < threshold: 556 | predicted[i] = 0 557 | else: 558 | predicted[i] = 1 559 | predicted = np.asarray(predicted).astype(int) 560 | cm = confusion_matrix(y_test_urfd, predicted,labels=[0,1]) 561 | tp = cm[0][0] 562 | fn = cm[0][1] 563 | fp = cm[1][0] 564 | tn = cm[1][1] 565 | tpr = tp/float(tp+fn) 566 | fpr = fp/float(fp+tn) 567 | fnr = fn/float(fn+tp) 568 | tnr = tn/float(tn+fp) 569 | print('URFD test set') 570 | print('-'*10) 571 | print('TP: {}, TN: {}, FP: {}, FN: {}'.format(tp,tn,fp,fn)) 572 | print('TPR: {}, TNR: {}, FPR: {}, FNR: {}'.format(tpr,tnr,fpr,fnr)) 573 | print('Sensitivity/Recall: {}'.format(tp/float(tp+fn))) 574 | print('Specificity: {}'.format(tn/float(tn+fp))) 575 | print('Accuracy: {}'.format(accuracy_score(y_test_urfd, predicted))) 576 | sensitivities['urfd'].append(tp/float(tp+fn)) 577 | specificities['urfd'].append(tn/float(tn+fp)) 578 | 579 | # Evaluate for the Multicam test set 580 | predicted = classifier.predict(X_test_multicam) 581 | for i in range(len(predicted)): 582 | if predicted[i] < threshold: 583 | predicted[i] = 0 584 | else: 585 | predicted[i] = 1 586 | predicted = np.asarray(predicted).astype(int) 587 | cm = confusion_matrix(y_test_multicam, predicted,labels=[0,1]) 588 | tp = cm[0][0] 589 | fn = cm[0][1] 590 | fp = cm[1][0] 591 | tn = cm[1][1] 592 | tpr = tp/float(tp+fn) 593 | fpr = fp/float(fp+tn) 594 | fnr = fn/float(fn+tp) 595 | tnr = tn/float(tn+fp) 596 | print('Multicam test set') 597 | print('-'*10) 598 | print('TP: {}, TN: {}, FP: {}, FN: {}'.format(tp,tn,fp,fn)) 599 | print('TPR: {}, TNR: {}, FPR: {}, FNR: {}'.format(tpr,tnr,fpr,fnr)) 600 | print('Sensitivity/Recall: {}'.format(tp/float(tp+fn))) 601 | print('Specificity: {}'.format(tn/float(tn+fp))) 602 | print('Accuracy: {}'.format(accuracy_score(y_test_multicam, predicted))) 603 | sensitivities['multicam'].append(tp/float(tp+fn)) 604 | specificities['multicam'].append(tn/float(tn+fp)) 605 | 606 | # Evaluate for the FDD test set 607 | predicted = classifier.predict(X_test_fdd) 608 | for i in range(len(predicted)): 609 | if predicted[i] < threshold: 610 | predicted[i] = 0 611 | else: 612 | predicted[i] = 1 613 | predicted = np.asarray(predicted).astype(int) 614 | cm = confusion_matrix(y_test_fdd, predicted,labels=[0,1]) 615 | tp = cm[0][0] 616 | fn = cm[0][1] 617 | fp = cm[1][0] 618 | tn = cm[1][1] 619 | tpr = tp/float(tp+fn) 620 | fpr = fp/float(fp+tn) 621 | fnr = fn/float(fn+tp) 622 | tnr = tn/float(tn+fp) 623 | print('FDD test set') 624 | print('-'*10) 625 | print('TP: {}, TN: {}, FP: {}, FN: {}'.format(tp,tn,fp,fn)) 626 | print('TPR: {}, TNR: {}, FPR: {}, FNR: {}'.format(tpr,tnr,fpr,fnr)) 627 | print('Sensitivity/Recall: {}'.format(tp/float(tp+fn))) 628 | print('Specificity: {}'.format(tn/float(tn+fp))) 629 | print('Accuracy: {}'.format(accuracy_score(y_test_fdd, predicted))) 630 | sensitivities['fdd'].append(tp/float(tp+fn)) 631 | specificities['fdd'].append(tn/float(tn+fp)) 632 | 633 | # End of the Cross-Validation 634 | print('CROSS-VALIDATION RESULTS ===================') 635 | 636 | print("Sensitivity Combined: {:.2f}% (+/- {:.2f}%)".format( 637 | np.mean(sensitivities['combined'])*100., 638 | np.std(sensitivities['combined'])*100.) 639 | ) 640 | print("Specificity Combined: {:.2f}% (+/- {:.2f}%)\n".format( 641 | np.mean(specificities['combined'])*100., 642 | np.std(specificities['combined'])*100.) 643 | ) 644 | print("Sensitivity URFD: {:.2f}% (+/- {:.2f}%)".format( 645 | np.mean(sensitivities['urfd'])*100., 646 | np.std(sensitivities['urfd'])*100.) 647 | ) 648 | print("Specificity URFD: {:.2f}% (+/- {:.2f}%)\n".format( 649 | np.mean(specificities['urfd'])*100., 650 | np.std(specificities['urfd'])*100.) 651 | ) 652 | print("Sensitivity Multicam: {:.2f}% (+/- {:.2f}%)".format( 653 | np.mean(sensitivities['multicam'])*100., 654 | np.std(sensitivities['multicam'])*100.) 655 | ) 656 | print("Specificity Multicam: {:.2f}% (+/- {:.2f}%)\n".format( 657 | np.mean(specificities['multicam'])*100., 658 | np.std(specificities['multicam'])*100.) 659 | ) 660 | print("Sensitivity Multicam: {:.2f}% (+/- {:.2f}%)".format( 661 | np.mean(sensitivities['fdd'])*100., 662 | np.std(sensitivities['fdd'])*100.) 663 | ) 664 | print("Specificity FDDs: {:.2f}% (+/- {:.2f}%)".format( 665 | np.mean(specificities['fdd'])*100., 666 | np.std(specificities['fdd'])*100.) 667 | ) 668 | 669 | if __name__ == '__main__': 670 | if not os.path.exists(best_model_path): 671 | os.makedirs(best_model_path) 672 | if not os.path.exists(plots_folder): 673 | os.makedirs(plots_folder) 674 | 675 | main() -------------------------------------------------------------------------------- /temporalnet_fdd.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | from numpy.random import seed 3 | seed(1) 4 | import numpy as np 5 | import matplotlib 6 | matplotlib.use('Agg') 7 | from matplotlib import pyplot as plt 8 | import os 9 | import h5py 10 | import scipy.io as sio 11 | import cv2 12 | import glob 13 | import gc 14 | 15 | from keras.models import load_model, Model, Sequential 16 | from keras.layers import (Input, Conv2D, MaxPooling2D, Flatten, 17 | Activation, Dense, Dropout, ZeroPadding2D) 18 | from keras.optimizers import Adam 19 | from keras.layers.normalization import BatchNormalization 20 | from keras.callbacks import EarlyStopping, ModelCheckpoint 21 | from keras import backend as K 22 | from sklearn.metrics import confusion_matrix, accuracy_score 23 | from sklearn.model_selection import KFold, StratifiedShuffleSplit 24 | from keras.layers.advanced_activations import ELU 25 | 26 | os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" 27 | os.environ["CUDA_VISIBLE_DEVICES"]="4" 28 | 29 | # CHANGE THESE VARIABLES --- 30 | data_folder = '/home/anunez/Downloads/FDD_OF/' 31 | mean_file = '/home/anunez/flow_mean.mat' 32 | vgg_16_weights = 'weights.h5' 33 | save_features = False 34 | save_plots = True 35 | 36 | # Set to 'True' if you want to restore a previous trained models 37 | # Training is skipped and test is done 38 | use_checkpoint = False 39 | # -------------------------1 40 | 41 | best_model_path = 'models/' 42 | plots_folder = 'plots/' 43 | checkpoint_path = best_model_path + 'fold_' 44 | 45 | saved_files_folder = 'saved_features/' 46 | features_file = saved_files_folder + 'features_fdd_tf.h5' 47 | labels_file = saved_files_folder + 'labels_fdd_tf.h5' 48 | features_key = 'features' 49 | labels_key = 'labels' 50 | 51 | L = 10 52 | num_features = 4096 53 | batch_norm = False 54 | learning_rate = 0.001 55 | mini_batch_size = 1024 56 | weight_0 = 2 57 | epochs = 3000 58 | use_validation = False 59 | # After the training stops, use train+validation to train for 1 epoch 60 | use_val_for_training = False 61 | val_size = 100 62 | # Threshold to classify between positive and negative 63 | threshold = 0.5 64 | 65 | # Name of the experiment 66 | exp = 'fdd_lr{}_batchs{}_batchnorm{}_w0_{}'.format( 67 | learning_rate, 68 | mini_batch_size, 69 | batch_norm, 70 | weight_0 71 | ) 72 | 73 | def plot_training_info(case, metrics, save, history): 74 | ''' 75 | Function to create plots for train and validation loss and accuracy 76 | Input: 77 | * case: name for the plot, an 'accuracy.png' or 'loss.png' 78 | will be concatenated after the name. 79 | * metrics: list of metrics to store: 'loss' and/or 'accuracy' 80 | * save: boolean to store the plots or only show them. 81 | * history: History object returned by the Keras fit function. 82 | ''' 83 | val = False 84 | if 'val_acc' in history and 'val_loss' in history: 85 | val = True 86 | plt.ioff() 87 | if 'accuracy' in metrics: 88 | fig = plt.figure() 89 | plt.plot(history['acc']) 90 | if val: plt.plot(history['val_acc']) 91 | plt.title('model accuracy') 92 | plt.ylabel('accuracy') 93 | plt.xlabel('epoch') 94 | if val: 95 | plt.legend(['train', 'val'], loc='upper left') 96 | else: 97 | plt.legend(['train'], loc='upper left') 98 | if save == True: 99 | plt.savefig(case + 'accuracy.png') 100 | plt.gcf().clear() 101 | else: 102 | plt.show() 103 | plt.close(fig) 104 | 105 | # summarize history for loss 106 | if 'loss' in metrics: 107 | fig = plt.figure() 108 | plt.plot(history['loss']) 109 | if val: plt.plot(history['val_loss']) 110 | plt.title('model loss') 111 | plt.ylabel('loss') 112 | plt.xlabel('epoch') 113 | #plt.ylim(1e-3, 1e-2) 114 | plt.yscale("log") 115 | if val: 116 | plt.legend(['train', 'val'], loc='upper left') 117 | else: 118 | plt.legend(['train'], loc='upper left') 119 | if save == True: 120 | plt.savefig(case + 'loss.png') 121 | plt.gcf().clear() 122 | else: 123 | plt.show() 124 | plt.close(fig) 125 | 126 | def generator(list1, lits2): 127 | ''' 128 | Auxiliar generator: returns the ith element of both given list with 129 | each call to next() 130 | ''' 131 | for x,y in zip(list1,lits2): 132 | yield x, y 133 | 134 | def saveFeatures(feature_extractor, 135 | features_file, 136 | labels_file, 137 | features_key, 138 | labels_key): 139 | ''' 140 | Function to load the optical flow stacks, do a feed-forward through the 141 | feature extractor (VGG16) and 142 | store the output feature vectors in the file 'features_file' and the 143 | labels in 'labels_file'. 144 | Input: 145 | * feature_extractor: model VGG16 until the fc6 layer. 146 | * features_file: path to the hdf5 file where the extracted features are 147 | going to be stored 148 | * labels_file: path to the hdf5 file where the labels of the features 149 | are going to be stored 150 | * features_key: name of the key for the hdf5 file to store the features 151 | * labels_key: name of the key for the hdf5 file to store the labels 152 | ''' 153 | 154 | class0 = 'Falls' 155 | class1 = 'NotFalls' 156 | 157 | # Load the mean file to subtract to the images 158 | d = sio.loadmat(mean_file) 159 | flow_mean = d['image_mean'] 160 | 161 | # Fill the folders and classes arrays with all the paths to the data 162 | folders, classes = [], [] 163 | fall_videos = [f for f in os.listdir(data_folder + class0) 164 | if os.path.isdir(os.path.join(data_folder + class0, f))] 165 | fall_videos.sort() 166 | for fall_video in fall_videos: 167 | x_images = glob.glob(data_folder + class0 + '/' + fall_video 168 | + '/flow_x*.jpg') 169 | if int(len(x_images)) >= 10: 170 | folders.append(data_folder + class0 + '/' + fall_video) 171 | classes.append(0) 172 | 173 | not_fall_videos = [f for f in os.listdir(data_folder + class1) 174 | if os.path.isdir(os.path.join(data_folder + class1, f))] 175 | not_fall_videos.sort() 176 | for not_fall_video in not_fall_videos: 177 | x_images = glob.glob(data_folder + class1 + '/' + not_fall_video 178 | + '/flow_x*.jpg') 179 | if int(len(x_images)) >= 10: 180 | folders.append(data_folder + class1 + '/' + not_fall_video) 181 | classes.append(1) 182 | 183 | # Total amount of stacks, with sliding window = num_images-L+1 184 | nb_total_stacks = 0 185 | for folder in folders: 186 | x_images = glob.glob(folder + '/flow_x*.jpg') 187 | nb_total_stacks += int(len(x_images))-L+1 188 | 189 | # File to store the extracted features and datasets to store them 190 | # IMPORTANT NOTE: 'w' mode totally erases previous data 191 | h5features = h5py.File(features_file,'w') 192 | h5labels = h5py.File(labels_file,'w') 193 | dataset_features = h5features.create_dataset(features_key, 194 | shape=(nb_total_stacks, 195 | num_features), 196 | dtype='float64') 197 | dataset_labels = h5labels.create_dataset(labels_key, 198 | shape=(nb_total_stacks, 1), 199 | dtype='float64') 200 | cont = 0 201 | 202 | for folder, label in zip(folders, classes): 203 | x_images = glob.glob(folder + '/flow_x*.jpg') 204 | x_images.sort() 205 | y_images = glob.glob(folder + '/flow_y*.jpg') 206 | y_images.sort() 207 | nb_stacks = int(len(x_images))-L+1 208 | # Here nb_stacks optical flow stacks will be stored 209 | flow = np.zeros(shape=(224,224,2*L,nb_stacks), dtype=np.float64) 210 | gen = generator(x_images,y_images) 211 | for i in range(len(x_images)): 212 | flow_x_file, flow_y_file = gen.next() 213 | img_x = cv2.imread(flow_x_file, cv2.IMREAD_GRAYSCALE) 214 | img_y = cv2.imread(flow_y_file, cv2.IMREAD_GRAYSCALE) 215 | # Assign an image i to the jth stack in the kth position, but also 216 | # in the j+1th stack in the k+1th position and so on 217 | # (for sliding window) 218 | for s in list(reversed(range(min(10,i+1)))): 219 | if i-s < nb_stacks: 220 | flow[:,:,2*s, i-s] = img_x 221 | flow[:,:,2*s+1,i-s] = img_y 222 | del img_x,img_y 223 | gc.collect() 224 | # Subtract mean 225 | flow = flow - np.tile(flow_mean[...,np.newaxis], 226 | (1, 1, 1, flow.shape[3])) 227 | flow = np.transpose(flow, (3, 0, 1, 2)) 228 | predictions = np.zeros((flow.shape[0], num_features), dtype=np.float64) 229 | truth = np.zeros((flow.shape[0], 1), dtype=np.float64) 230 | # Process each stack: do the feed-forward pass and store in the hdf5 231 | # file the output 232 | for i in range(flow.shape[0]): 233 | prediction = feature_extractor.predict( 234 | np.expand_dims(flow[i, ...],0)) 235 | predictions[i, ...] = prediction 236 | truth[i] = label 237 | dataset_features[cont:cont+flow.shape[0],:] = predictions 238 | dataset_labels[cont:cont+flow.shape[0],:] = truth 239 | cont += flow.shape[0] 240 | h5features.close() 241 | h5labels.close() 242 | 243 | def main(): 244 | # ======================================================================== 245 | # VGG-16 ARCHITECTURE 246 | # ======================================================================== 247 | model = Sequential() 248 | 249 | model.add(ZeroPadding2D((1, 1), input_shape=(224, 224, 20))) 250 | model.add(Conv2D(64, (3, 3), activation='relu', name='conv1_1')) 251 | model.add(ZeroPadding2D((1, 1))) 252 | model.add(Conv2D(64, (3, 3), activation='relu', name='conv1_2')) 253 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 254 | 255 | model.add(ZeroPadding2D((1, 1))) 256 | model.add(Conv2D(128, (3, 3), activation='relu', name='conv2_1')) 257 | model.add(ZeroPadding2D((1, 1))) 258 | model.add(Conv2D(128, (3, 3), activation='relu', name='conv2_2')) 259 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 260 | 261 | model.add(ZeroPadding2D((1, 1))) 262 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_1')) 263 | model.add(ZeroPadding2D((1, 1))) 264 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_2')) 265 | model.add(ZeroPadding2D((1, 1))) 266 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_3')) 267 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 268 | 269 | model.add(ZeroPadding2D((1, 1))) 270 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_1')) 271 | model.add(ZeroPadding2D((1, 1))) 272 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_2')) 273 | model.add(ZeroPadding2D((1, 1))) 274 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_3')) 275 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 276 | 277 | model.add(ZeroPadding2D((1, 1))) 278 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_1')) 279 | model.add(ZeroPadding2D((1, 1))) 280 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_2')) 281 | model.add(ZeroPadding2D((1, 1))) 282 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_3')) 283 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 284 | 285 | model.add(Flatten()) 286 | model.add(Dense(num_features, name='fc6', kernel_initializer='glorot_uniform')) 287 | 288 | # ======================================================================== 289 | # WEIGHT INITIALIZATION 290 | # ======================================================================== 291 | layerscaffe = ['conv1_1', 'conv1_2', 'conv2_1', 'conv2_2', 'conv3_1', 292 | 'conv3_2', 'conv3_3', 'conv4_1', 'conv4_2', 'conv4_3', 293 | 'conv5_1', 'conv5_2', 'conv5_3', 'fc6', 'fc7', 'fc8'] 294 | h5 = h5py.File(vgg_16_weights, 'r') 295 | 296 | layer_dict = dict([(layer.name, layer) for layer in model.layers]) 297 | 298 | # Copy the weights stored in the 'vgg_16_weights' file to the 299 | # feature extractor part of the VGG16 300 | for layer in layerscaffe[:-3]: 301 | w2, b2 = h5['data'][layer]['0'], h5['data'][layer]['1'] 302 | w2 = np.transpose(np.asarray(w2), (2,3,1,0)) 303 | w2 = w2[::-1, ::-1, :, :] 304 | b2 = np.asarray(b2) 305 | layer_dict[layer].set_weights((w2, b2)) 306 | 307 | # Copy the weights of the first fully-connected layer (fc6) 308 | layer = layerscaffe[-3] 309 | w2, b2 = h5['data'][layer]['0'], h5['data'][layer]['1'] 310 | w2 = np.transpose(np.asarray(w2), (1,0)) 311 | b2 = np.asarray(b2) 312 | layer_dict[layer].set_weights((w2, b2)) 313 | 314 | # ======================================================================== 315 | # FEATURE EXTRACTION 316 | # ======================================================================== 317 | if save_features: 318 | saveFeatures(model, features_file, 319 | labels_file, features_key, 320 | labels_key) 321 | 322 | # ======================================================================== 323 | # TRAINING 324 | # ======================================================================== 325 | 326 | adam = Adam(lr=learning_rate, beta_1=0.9, beta_2=0.999, 327 | epsilon=1e-08, decay=0.0005) 328 | model.compile(optimizer=adam, loss='categorical_crossentropy', 329 | metrics=['accuracy']) 330 | 331 | h5features = h5py.File(features_file, 'r') 332 | h5labels = h5py.File(labels_file, 'r') 333 | 334 | # X_full will contain all the feature vectors extracted from optical 335 | # flow images 336 | X_full = h5features[features_key] 337 | _y_full = np.asarray(h5labels[labels_key]) 338 | 339 | zeroes_full = np.asarray(np.where(_y_full==0)[0]) 340 | ones_full = np.asarray(np.where(_y_full==1)[0]) 341 | zeroes_full.sort() 342 | ones_full.sort() 343 | 344 | # Use a 5 fold cross-validation 345 | kf_falls = KFold(n_splits=5, shuffle=True) 346 | kf_falls.get_n_splits(X_full[zeroes_full, ...]) 347 | 348 | kf_nofalls = KFold(n_splits=5, shuffle=True) 349 | kf_nofalls.get_n_splits(X_full[ones_full, ...]) 350 | 351 | sensitivities = [] 352 | specificities = [] 353 | accuracies = [] 354 | 355 | fold_number = 1 356 | # CROSS-VALIDATION: Stratified partition of the dataset into 357 | # train/test sets 358 | for ((train_index_falls, test_index_falls), 359 | (train_index_nofalls, test_index_nofalls)) in zip( 360 | kf_falls.split(X_full[zeroes_full, ...]), 361 | kf_nofalls.split(X_full[ones_full, ...]) 362 | ): 363 | 364 | train_index_falls = np.asarray(train_index_falls) 365 | test_index_falls = np.asarray(test_index_falls) 366 | train_index_nofalls = np.asarray(train_index_nofalls) 367 | test_index_nofalls = np.asarray(test_index_nofalls) 368 | 369 | X = np.concatenate((X_full[zeroes_full, ...][train_index_falls, ...], 370 | X_full[ones_full, ...][train_index_nofalls, ...])) 371 | _y = np.concatenate((_y_full[zeroes_full, ...][train_index_falls, ...], 372 | _y_full[ones_full, ...][train_index_nofalls, ...])) 373 | X_test = np.concatenate((X_full[zeroes_full, ...][test_index_falls, ...], 374 | X_full[ones_full, ...][test_index_nofalls, ...])) 375 | y_test = np.concatenate((_y_full[zeroes_full, ...][test_index_falls, ...], 376 | _y_full[ones_full, ...][test_index_nofalls, ...])) 377 | 378 | if use_validation: 379 | # Create a validation subset from the training set 380 | zeroes = np.asarray(np.where(_y==0)[0]) 381 | ones = np.asarray(np.where(_y==1)[0]) 382 | 383 | zeroes.sort() 384 | ones.sort() 385 | 386 | trainval_split_0 = StratifiedShuffleSplit(n_splits=1, 387 | test_size=val_size/2, 388 | random_state=7) 389 | indices_0 = trainval_split_0.split(X[zeroes,...], 390 | np.argmax(_y[zeroes,...], 1)) 391 | trainval_split_1 = StratifiedShuffleSplit(n_splits=1, 392 | test_size=val_size/2, 393 | random_state=7) 394 | indices_1 = trainval_split_1.split(X[ones,...], 395 | np.argmax(_y[ones,...], 1)) 396 | train_indices_0, val_indices_0 = indices_0.next() 397 | train_indices_1, val_indices_1 = indices_1.next() 398 | 399 | X_train = np.concatenate([X[zeroes,...][train_indices_0,...], 400 | X[ones,...][train_indices_1,...]],axis=0) 401 | y_train = np.concatenate([_y[zeroes,...][train_indices_0,...], 402 | _y[ones,...][train_indices_1,...]],axis=0) 403 | X_val = np.concatenate([X[zeroes,...][val_indices_0,...], 404 | X[ones,...][val_indices_1,...]],axis=0) 405 | y_val = np.concatenate([_y[zeroes,...][val_indices_0,...], 406 | _y[ones,...][val_indices_1,...]],axis=0) 407 | else: 408 | X_train = X 409 | y_train = _y 410 | 411 | # Balance the number of positive and negative samples so that 412 | # there is the same amount of each of them 413 | all0 = np.asarray(np.where(y_train==0)[0]) 414 | all1 = np.asarray(np.where(y_train==1)[0]) 415 | 416 | if len(all0) < len(all1): 417 | all1 = np.random.choice(all1, len(all0), replace=False) 418 | else: 419 | all0 = np.random.choice(all0, len(all1), replace=False) 420 | allin = np.concatenate((all0.flatten(),all1.flatten())) 421 | allin.sort() 422 | X_train = X_train[allin,...] 423 | y_train = y_train[allin] 424 | 425 | # Shuffle 426 | perm = np.random.permutation(len(X_train)) 427 | X_train = X_train[perm] 428 | y_train = y_train[perm] 429 | 430 | # ==================== CLASSIFIER ======================== 431 | extracted_features = Input(shape=(num_features,), 432 | dtype='float32', name='input') 433 | if batch_norm: 434 | x = BatchNormalization(axis=-1, momentum=0.99, 435 | epsilon=0.001)(extracted_features) 436 | x = Activation('relu')(x) 437 | else: 438 | x = ELU(alpha=1.0)(extracted_features) 439 | 440 | x = Dropout(0.9)(x) 441 | x = Dense(4096, name='fc2', init='glorot_uniform')(x) 442 | if batch_norm: 443 | x = BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001)(x) 444 | x = Activation('relu')(x) 445 | else: 446 | x = ELU(alpha=1.0)(x) 447 | x = Dropout(0.8)(x) 448 | x = Dense(1, name='predictions', init='glorot_uniform')(x) 449 | x = Activation('sigmoid')(x) 450 | 451 | classifier = Model(input=extracted_features, 452 | output=x, name='classifier') 453 | fold_best_model_path = best_model_path + 'fdd_fold_{}.h5'.format( 454 | fold_number) 455 | classifier.compile(optimizer=adam, loss='binary_crossentropy', 456 | metrics=['accuracy']) 457 | 458 | if not use_checkpoint: 459 | # ==================== TRAINING ======================== 460 | # weighting of each class: only the fall class gets 461 | # a different weight 462 | class_weight = {0: weight_0, 1: 1} 463 | 464 | callbacks = None 465 | if use_validation: 466 | # callback definition 467 | metric = 'val_loss' 468 | e = EarlyStopping(monitor=metric, min_delta=0, patience=100, 469 | mode='auto') 470 | c = ModelCheckpoint(fold_best_model_path, monitor=metric, 471 | save_best_only=True, 472 | save_weights_only=False, mode='auto') 473 | callbacks = [e, c] 474 | validation_data = None 475 | if use_validation: 476 | validation_data = (X_val,y_val) 477 | _mini_batch_size = mini_batch_size 478 | if mini_batch_size == 0: 479 | _mini_batch_size = X_train.shape[0] 480 | 481 | history = classifier.fit( 482 | X_train, y_train, 483 | validation_data=validation_data, 484 | batch_size=_mini_batch_size, 485 | nb_epoch=epochs, 486 | shuffle=True, 487 | class_weight=class_weight, 488 | callbacks=callbacks 489 | ) 490 | 491 | if not use_validation: 492 | classifier.save(fold_best_model_path) 493 | 494 | plot_training_info(plots_folder + exp, ['accuracy', 'loss'], 495 | save_plots, history.history) 496 | 497 | if use_validation and use_val_for_training: 498 | classifier = load_model(fold_best_model_path) 499 | 500 | # Use full training set (training+validation) 501 | X_train = np.concatenate((X_train, X_val), axis=0) 502 | y_train = np.concatenate((y_train, y_val), axis=0) 503 | 504 | history = classifier.fit( 505 | X_train, y_train, 506 | validation_data=validation_data, 507 | batch_size=_mini_batch_size, 508 | nb_epoch=epochs, 509 | shuffle='batch', 510 | class_weight=class_weight, 511 | callbacks=callbacks 512 | ) 513 | 514 | classifier.save(fold_best_model_path) 515 | 516 | # ==================== EVALUATION ======================== 517 | 518 | # Load best model 519 | print('Model loaded from checkpoint') 520 | classifier = load_model(fold_best_model_path) 521 | 522 | predicted = classifier.predict(np.asarray(X_test)) 523 | for i in range(len(predicted)): 524 | if predicted[i] < threshold: 525 | predicted[i] = 0 526 | else: 527 | predicted[i] = 1 528 | # Array of predictions 0/1 529 | predicted = np.asarray(predicted) 530 | # Compute metrics and print them 531 | cm = confusion_matrix(y_test, predicted,labels=[0,1]) 532 | tp = cm[0][0] 533 | fn = cm[0][1] 534 | fp = cm[1][0] 535 | tn = cm[1][1] 536 | tpr = tp/float(tp+fn) 537 | fpr = fp/float(fp+tn) 538 | fnr = fn/float(fn+tp) 539 | tnr = tn/float(tn+fp) 540 | precision = tp/float(tp+fp) 541 | recall = tp/float(tp+fn) 542 | specificity = tn/float(tn+fp) 543 | f1 = 2*float(precision*recall)/float(precision+recall) 544 | accuracy = accuracy_score(y_test, predicted) 545 | 546 | print('FOLD {} results:'.format(fold_number)) 547 | print('TP: {}, TN: {}, FP: {}, FN: {}'.format(tp,tn,fp,fn)) 548 | print('TPR: {}, TNR: {}, FPR: {}, FNR: {}'.format( 549 | tpr,tnr,fpr,fnr)) 550 | print('Sensitivity/Recall: {}'.format(recall)) 551 | print('Specificity: {}'.format(specificity)) 552 | print('Precision: {}'.format(precision)) 553 | print('F1-measure: {}'.format(f1)) 554 | print('Accuracy: {}'.format(accuracy)) 555 | 556 | # Store the metrics for this epoch 557 | sensitivities.append(tp/float(tp+fn)) 558 | specificities.append(tn/float(tn+fp)) 559 | accuracies.append(accuracy) 560 | fold_number += 1 561 | 562 | print('5-FOLD CROSS-VALIDATION RESULTS ===================') 563 | print("Sensitivity: %.2f%% (+/- %.2f%%)" % (np.mean(sensitivities)*100., 564 | np.std(sensitivities)*100.)) 565 | print("Specificity: %.2f%% (+/- %.2f%%)" % (np.mean(specificities)*100., 566 | np.std(specificities)*100.)) 567 | print("Accuracy: %.2f%% (+/- %.2f%%)" % (np.mean(accuracies)*100., 568 | np.std(accuracies)*100.)) 569 | 570 | if __name__ == '__main__': 571 | if not os.path.exists(best_model_path): 572 | os.makedirs(best_model_path) 573 | if not os.path.exists(plots_folder): 574 | os.makedirs(plots_folder) 575 | 576 | main() 577 | -------------------------------------------------------------------------------- /temporalnet_multicam.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | from numpy.random import seed 3 | seed(1) 4 | import numpy as np 5 | import matplotlib 6 | matplotlib.use('Agg') 7 | from matplotlib import pyplot as plt 8 | import os 9 | import h5py 10 | import scipy.io as sio 11 | import cv2 12 | import glob 13 | import gc 14 | 15 | from keras.models import load_model, Model, Sequential 16 | from keras.layers import (Input, Conv2D, MaxPooling2D, Flatten, 17 | Activation, Dense, Dropout, ZeroPadding2D) 18 | from keras.optimizers import Adam 19 | from keras.layers.normalization import BatchNormalization 20 | from keras.callbacks import EarlyStopping, ModelCheckpoint 21 | from keras import backend as K 22 | from sklearn.metrics import confusion_matrix, accuracy_score, roc_curve, auc 23 | from sklearn.model_selection import KFold, StratifiedShuffleSplit 24 | from keras.layers.advanced_activations import ELU 25 | 26 | os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" 27 | os.environ["CUDA_VISIBLE_DEVICES"]="0" 28 | 29 | # CHANGE THESE VARIABLES --- 30 | data_folder = '/home/anunez/Downloads/Multicam_OF/' 31 | mean_file = '/home/anunez/flow_mean.mat' 32 | vgg_16_weights = 'weights.h5' 33 | save_features = False 34 | save_plots = True 35 | 36 | # Set to 'True' if you want to restore a previous trained models 37 | # Training is skipped and test is done 38 | use_checkpoint = False 39 | # -------------------------- 40 | 41 | best_model_path = 'models/' 42 | plots_folder = 'plots/' 43 | checkpoint_path = best_model_path + 'fold_' 44 | 45 | saved_files_folder = 'saved_features/' 46 | features_file = saved_files_folder + 'features_multicam_tf.h5' 47 | labels_file = saved_files_folder + 'labels_multicam_tf.h5' 48 | features_key = 'features' 49 | labels_key = 'labels' 50 | 51 | num_cameras = 8 52 | L = 10 53 | num_features = 4096 54 | batch_norm = True 55 | learning_rate = 0.01 56 | mini_batch_size = 0 57 | weight_0 = 1 58 | epochs = 6000 59 | use_validation = False 60 | # After the training stops, use train+validation to train for 1 epoch 61 | use_val_for_training = False 62 | val_size = 100 63 | # Threshold to classify between positive and negative 64 | threshold = 0.5 65 | 66 | # Name of the experiment 67 | exp = 'multicam_lr{}_batchs{}_batchnorm{}_w0_{}'.format(learning_rate, 68 | mini_batch_size, 69 | batch_norm, 70 | weight_0) 71 | 72 | def plot_training_info(case, metrics, save, history): 73 | ''' 74 | Function to create plots for train and validation loss and accuracy 75 | Input: 76 | * case: name for the plot, an 'accuracy.png' or 'loss.png' 77 | will be concatenated after the name. 78 | * metrics: list of metrics to store: 'loss' and/or 'accuracy' 79 | * save: boolean to store the plots or only show them. 80 | * history: History object returned by the Keras fit function. 81 | ''' 82 | val = False 83 | if 'val_acc' in history and 'val_loss' in history: 84 | val = True 85 | plt.ioff() 86 | if 'accuracy' in metrics: 87 | fig = plt.figure() 88 | plt.plot(history['acc']) 89 | if val: plt.plot(history['val_acc']) 90 | plt.title('model accuracy') 91 | plt.ylabel('accuracy') 92 | plt.xlabel('epoch') 93 | if val: 94 | plt.legend(['train', 'val'], loc='upper left') 95 | else: 96 | plt.legend(['train'], loc='upper left') 97 | if save == True: 98 | plt.savefig(case + 'accuracy.png') 99 | plt.gcf().clear() 100 | else: 101 | plt.show() 102 | plt.close(fig) 103 | 104 | # summarize history for loss 105 | if 'loss' in metrics: 106 | fig = plt.figure() 107 | plt.plot(history['loss']) 108 | if val: plt.plot(history['val_loss']) 109 | plt.title('model loss') 110 | plt.ylabel('loss') 111 | plt.xlabel('epoch') 112 | #plt.ylim(1e-3, 1e-2) 113 | plt.yscale("log") 114 | if val: 115 | plt.legend(['train', 'val'], loc='upper left') 116 | else: 117 | plt.legend(['train'], loc='upper left') 118 | if save == True: 119 | plt.savefig(case + 'loss.png') 120 | plt.gcf().clear() 121 | else: 122 | plt.show() 123 | plt.close(fig) 124 | 125 | def generator(list1, lits2): 126 | ''' 127 | Auxiliar generator: returns the ith element of both given list with 128 | each call to next() 129 | ''' 130 | for x,y in zip(list1,lits2): 131 | yield x, y 132 | 133 | def saveFeatures(feature_extractor, 134 | features_file, 135 | labels_file, 136 | features_key, 137 | labels_key): 138 | ''' 139 | Function to load the optical flow stacks, do a feed-forward through the 140 | feature extractor (VGG16) and 141 | store the output feature vectors in the file 'features_file' and the 142 | labels in 'labels_file'. 143 | Input: 144 | * feature_extractor: model VGG16 until the fc6 layer. 145 | * features_file: path to the hdf5 file where the extracted features are 146 | going to be stored 147 | * labels_file: path to the hdf5 file where the labels of the features 148 | are going to be stored 149 | * features_key: name of the key for the hdf5 file to store the features 150 | * labels_key: name of the key for the hdf5 file to store the labels 151 | ''' 152 | 153 | if not os.path.exists(saved_files_folder): 154 | os.makedirs(saved_files_folder) 155 | 156 | class0 = 'Falls' 157 | class1 = 'NotFalls' 158 | 159 | # Load the mean file to subtract to the images 160 | d = sio.loadmat(mean_file) 161 | flow_mean = d['image_mean'] 162 | 163 | h5features = h5py.File(features_file,'w') 164 | h5labels = h5py.File(labels_file,'w') 165 | 166 | fall_videos = np.zeros((24,2), dtype=np.int) 167 | i = 0 168 | while i < 3: 169 | fall_videos[i,:] = [i*7, i*7+7] 170 | i += 1 171 | fall_videos[i,:] = [i*7, i*7+14] 172 | i += 1 173 | while i < 23: 174 | fall_videos[i,:] = [i*7, i*7+7] 175 | i += 1 176 | fall_videos[i,:] = [i*7, i*7] 177 | 178 | not_fall_videos = np.zeros((24,2), dtype=np.int) 179 | i = 0 180 | while i < 23: 181 | not_fall_videos[i,:] = [i*7, i*7+14] 182 | i += 1 183 | not_fall_videos[i,:] = [i*7, i*7+7] 184 | 185 | stages = [] 186 | for i in range(1,25): 187 | stages.append('chute{:02}'.format(i)) 188 | 189 | for stage, nb_stage in zip(stages, range(len(stages))): 190 | h5features.create_group(stage) 191 | h5labels.create_group(stage) 192 | path = data_folder + stage 193 | for nb_camera in range(1,num_cameras+1): 194 | h5features[stage].create_group('cam{}'.format(nb_camera)) 195 | h5labels[stage].create_group('cam{}'.format(nb_camera)) 196 | not_falls = glob.glob( 197 | path + '/NotFalls/camera{}*'.format(nb_camera) 198 | ) 199 | not_falls.sort() 200 | 201 | for not_fall in not_falls: 202 | label = 1 203 | name = not_fall[not_fall.rfind('/')+1:] 204 | x_images = glob.glob(not_fall + '/flow_x*.jpg') 205 | x_images.sort() 206 | y_images = glob.glob(not_fall + '/flow_y*.jpg') 207 | y_images.sort() 208 | nb_stacks = int(len(x_images))-L+1 209 | 210 | features_notfall = h5features[stage][ 211 | 'cam{}'.format(nb_camera) 212 | ].create_dataset( 213 | name, 214 | shape=(nb_stacks, num_features), 215 | dtype='float64') 216 | labels_notfall = h5labels[stage][ 217 | 'cam{}'.format(nb_camera) 218 | ].create_dataset( 219 | name, 220 | shape=(nb_stacks, 1), 221 | dtype='float64') 222 | 223 | # NO FALL 224 | flow = np.zeros(shape=(224,224,2*L,nb_stacks), dtype=np.float64) 225 | gen = generator(x_images,y_images) 226 | for i in range(len(x_images)): 227 | flow_x_file, flow_y_file = gen.next() 228 | img_x = cv2.imread(flow_x_file, cv2.IMREAD_GRAYSCALE) 229 | img_y = cv2.imread(flow_y_file, cv2.IMREAD_GRAYSCALE) 230 | for s in list(reversed(range(min(10,i+1)))): 231 | if i-s < nb_stacks: 232 | flow[:,:,2*s, i-s] = img_x 233 | flow[:,:,2*s+1,i-s] = img_y 234 | del img_x,img_y 235 | gc.collect() 236 | flow = flow - np.tile(flow_mean[...,np.newaxis], 237 | (1, 1, 1, flow.shape[3])) 238 | flow = np.transpose(flow, (3, 0, 1, 2)) 239 | predictions = np.zeros((flow.shape[0], num_features), 240 | dtype=np.float64) 241 | truth = np.zeros((flow.shape[0], 1), dtype=np.float64) 242 | for i in range(flow.shape[0]): 243 | prediction = feature_extractor.predict( 244 | np.expand_dims(flow[i, ...],0)) 245 | predictions[i, ...] = prediction 246 | truth[i] = label 247 | features_notfall[:,:] = predictions 248 | labels_notfall[:,:] = truth 249 | del (predictions, truth, flow, features_notfall, 250 | labels_notfall, x_images, y_images, nb_stacks) 251 | gc.collect() 252 | 253 | if stage == 'chute24': 254 | continue 255 | 256 | falls = glob.glob( 257 | path + '/Falls/camera{}'.format(nb_camera) 258 | ) 259 | falls.sort() 260 | for fall in falls: 261 | label = 0 262 | name = fall[fall.rfind('/')+1:] 263 | x_images = glob.glob(fall + '/flow_x*.jpg') 264 | x_images.sort() 265 | y_images = glob.glob(fall + '/flow_y*.jpg') 266 | y_images.sort() 267 | nb_stacks = int(len(x_images))-L+1 268 | 269 | features_fall = h5features[stage][ 270 | 'cam{}'.format(nb_camera) 271 | ].create_dataset( 272 | name, 273 | shape=(nb_stacks, num_features), 274 | dtype='float64') 275 | labels_fall = h5labels[stage][ 276 | 'cam{}'.format(nb_camera) 277 | ].create_dataset( 278 | name, 279 | shape=(nb_stacks, 1), 280 | dtype='float64') 281 | flow = np.zeros(shape=(224,224,2*L,nb_stacks), dtype=np.float64) 282 | 283 | gen = generator(x_images,y_images) 284 | for i in range(len(x_images)): 285 | flow_x_file, flow_y_file = gen.next() 286 | img_x = cv2.imread(flow_x_file, cv2.IMREAD_GRAYSCALE) 287 | img_y = cv2.imread(flow_y_file, cv2.IMREAD_GRAYSCALE) 288 | for s in list(reversed(range(min(10,i+1)))): 289 | if i-s < nb_stacks: 290 | flow[:,:,2*s, i-s] = img_x 291 | flow[:,:,2*s+1,i-s] = img_y 292 | del img_x,img_y 293 | gc.collect() 294 | flow = flow - np.tile(flow_mean[...,np.newaxis], 295 | (1, 1, 1, flow.shape[3])) 296 | flow = np.transpose(flow, (3, 0, 1, 2)) 297 | predictions = np.zeros((flow.shape[0], num_features), 298 | dtype=np.float64) 299 | truth = np.zeros((flow.shape[0], 1), dtype=np.float64) 300 | for i in range(flow.shape[0]): 301 | prediction = feature_extractor.predict( 302 | np.expand_dims(flow[i, ...],0)) 303 | predictions[i, ...] = prediction 304 | truth[i] = label 305 | features_fall[:,:] = predictions 306 | labels_fall[:,:] = truth 307 | del predictions, truth, flow, features_fall, labels_fall 308 | 309 | h5features.close() 310 | h5labels.close() 311 | 312 | def load_dataset(): 313 | h5features = h5py.File(features_file, 'r') 314 | h5labels = h5py.File(labels_file, 'r') 315 | 316 | # Load the data separated by cameras for cross-validation 317 | stages = [] 318 | for i in range(1,25): 319 | stages.append('chute{:02}'.format(i)) 320 | cams_x = [] 321 | cams_y = [] 322 | for stage, nb_stage in zip(stages, range(len(stages))): 323 | for cam, nb_cam in zip(h5features[stage].keys(), range(8)): 324 | temp_x = [] 325 | temp_y = [] 326 | for key in h5features[stage][cam].keys(): 327 | temp_x.append(np.asarray(h5features[stage][cam][key])) 328 | temp_y.append(np.asarray(h5labels[stage][cam][key])) 329 | temp_x = np.concatenate(temp_x,axis=0) 330 | temp_y = np.concatenate(temp_y,axis=0) 331 | if nb_stage == 0: 332 | cams_x.append(temp_x) 333 | cams_y.append(temp_y) 334 | else: 335 | cams_x[nb_cam] = np.concatenate([cams_x[nb_cam], 336 | temp_x], axis=0) 337 | cams_y[nb_cam] = np.concatenate([cams_y[nb_cam], 338 | temp_y], axis=0) 339 | return cams_x, cams_y 340 | 341 | def main(): 342 | # ======================================================================== 343 | # VGG-16 ARCHITECTURE 344 | # ======================================================================== 345 | model = Sequential() 346 | 347 | model.add(ZeroPadding2D((1, 1), input_shape=(224, 224, 20))) 348 | model.add(Conv2D(64, (3, 3), activation='relu', name='conv1_1')) 349 | model.add(ZeroPadding2D((1, 1))) 350 | model.add(Conv2D(64, (3, 3), activation='relu', name='conv1_2')) 351 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 352 | 353 | model.add(ZeroPadding2D((1, 1))) 354 | model.add(Conv2D(128, (3, 3), activation='relu', name='conv2_1')) 355 | model.add(ZeroPadding2D((1, 1))) 356 | model.add(Conv2D(128, (3, 3), activation='relu', name='conv2_2')) 357 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 358 | 359 | model.add(ZeroPadding2D((1, 1))) 360 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_1')) 361 | model.add(ZeroPadding2D((1, 1))) 362 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_2')) 363 | model.add(ZeroPadding2D((1, 1))) 364 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_3')) 365 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 366 | 367 | model.add(ZeroPadding2D((1, 1))) 368 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_1')) 369 | model.add(ZeroPadding2D((1, 1))) 370 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_2')) 371 | model.add(ZeroPadding2D((1, 1))) 372 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_3')) 373 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 374 | 375 | model.add(ZeroPadding2D((1, 1))) 376 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_1')) 377 | model.add(ZeroPadding2D((1, 1))) 378 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_2')) 379 | model.add(ZeroPadding2D((1, 1))) 380 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_3')) 381 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 382 | 383 | model.add(Flatten()) 384 | model.add(Dense(num_features, name='fc6', 385 | kernel_initializer='glorot_uniform')) 386 | 387 | # ======================================================================== 388 | # WEIGHT INITIALIZATION 389 | # ======================================================================== 390 | layerscaffe = ['conv1_1', 'conv1_2', 'conv2_1', 'conv2_2', 'conv3_1', 391 | 'conv3_2', 'conv3_3', 'conv4_1', 'conv4_2', 'conv4_3', 392 | 'conv5_1', 'conv5_2', 'conv5_3', 'fc6', 'fc7', 'fc8'] 393 | i = 0 394 | h5 = h5py.File(vgg_16_weights) 395 | 396 | layer_dict = dict([(layer.name, layer) for layer in model.layers]) 397 | 398 | # Copy the weights stored in the 'vgg_16_weights' file to the 399 | # feature extractor part of the VGG16 400 | for layer in layerscaffe[:-3]: 401 | w2, b2 = h5['data'][layer]['0'], h5['data'][layer]['1'] 402 | w2 = np.transpose(np.asarray(w2), (2,3,1,0)) 403 | w2 = w2[::-1, ::-1, :, :] 404 | b2 = np.asarray(b2) 405 | layer_dict[layer].set_weights((w2, b2)) 406 | 407 | # Copy the weights of the first fully-connected layer (fc6) 408 | layer = layerscaffe[-3] 409 | w2, b2 = h5['data'][layer]['0'], h5['data'][layer]['1'] 410 | w2 = np.transpose(np.asarray(w2), (1,0)) 411 | b2 = np.asarray(b2) 412 | layer_dict[layer].set_weights((w2, b2)) 413 | 414 | # ======================================================================== 415 | # FEATURE EXTRACTION 416 | # ======================================================================== 417 | if save_features: 418 | saveFeatures(model, features_file, 419 | labels_file, features_key, 420 | labels_key) 421 | 422 | # ======================================================================== 423 | # TRAINING 424 | # ======================================================================= 425 | 426 | adam = Adam(lr=learning_rate, beta_1=0.9, beta_2=0.999, 427 | epsilon=1e-08, decay=0.0005) 428 | model.compile(optimizer=adam, loss='categorical_crossentropy', 429 | metrics=['accuracy']) 430 | 431 | cams_x, cams_y = load_dataset() 432 | 433 | sensitivities = [] 434 | specificities = [] 435 | aucs = [] 436 | accuracies = [] 437 | 438 | # LEAVE-ONE-CAMERA-OUT CROSS-VALIDATION 439 | for cam in range(num_cameras): 440 | print('='*30) 441 | print('LEAVE-ONE-OUT STEP {}/8'.format(cam+1)) 442 | print('='*30) 443 | # cams_x[nb_cam] contains all the optical flow stacks of 444 | # the 'cam' camera (where 'cam' is an integer from 0 to 24) 445 | test_x = cams_x[cam] 446 | test_y = cams_y[cam] 447 | train_x = cams_x[0:cam] + cams_x[cam+1:] 448 | train_y = cams_y[0:cam] + cams_y[cam+1:] 449 | # Flatten to 1D arrays 450 | train_x = np.asarray([train_x[i][j] 451 | for i in range(len(train_x)) for j in range(len(train_x[i]))]) 452 | train_y = np.asarray([train_y[i][j] 453 | for i in range(len(train_y)) for j in range(len(train_y[i]))]) 454 | 455 | # Create a validation subset from the training set 456 | zeroes = np.asarray(np.where(train_y==0)[0]) 457 | ones = np.asarray(np.where(train_y==1)[0]) 458 | trainval_split_0 = StratifiedShuffleSplit(n_splits=1, 459 | test_size=val_size/2, 460 | random_state=7) 461 | indices_0 = trainval_split_0.split(train_x[zeroes,...], 462 | np.argmax(train_y[zeroes,...], 1)) 463 | trainval_split_1 = StratifiedShuffleSplit(n_splits=1, 464 | test_size=val_size/2, 465 | random_state=7) 466 | indices_1 = trainval_split_1.split(train_x[ones,...], 467 | np.argmax(train_y[ones,...], 1)) 468 | train_indices_0, val_indices_0 = indices_0.next() 469 | train_indices_1, val_indices_1 = indices_1.next() 470 | 471 | _X_train = np.concatenate([train_x[zeroes,...][train_indices_0,...], 472 | train_x[ones,...][train_indices_1,...]],axis=0) 473 | _y_train = np.concatenate([train_y[zeroes,...][train_indices_0,...], 474 | train_y[ones,...][train_indices_1,...]],axis=0) 475 | X_val = np.concatenate([train_x[zeroes,...][val_indices_0,...], 476 | train_x[ones,...][val_indices_1,...]],axis=0) 477 | y_val = np.concatenate([train_y[zeroes,...][val_indices_0,...], 478 | train_y[ones,...][val_indices_1,...]],axis=0) 479 | y_val = np.squeeze(y_val) 480 | _y_train = np.squeeze(np.asarray(_y_train)) 481 | 482 | # Balance the positive and negative samples 483 | all0 = np.where(_y_train==0)[0] 484 | all1 = np.where(_y_train==1)[0] 485 | 486 | all1 = np.random.choice(all1, len(all0), replace=False) 487 | allin = np.concatenate((all0.flatten(), all1.flatten())) 488 | X_train = np.asarray(_X_train[allin,...]) 489 | y_train = np.asarray(_y_train[allin]) 490 | X_test = np.asarray(test_x) 491 | y_test = np.asarray(test_y) 492 | 493 | # ==================== CLASSIFIER ======================== 494 | extracted_features = Input(shape=(num_features,), 495 | dtype='float32', name='input') 496 | if batch_norm: 497 | x = BatchNormalization(axis=-1, momentum=0.99, 498 | epsilon=0.001)(extracted_features) 499 | x = Activation('relu')(x) 500 | else: 501 | x = ELU(alpha=1.0)(extracted_features) 502 | 503 | x = Dropout(0.9)(x) 504 | x = Dense(4096, name='fc2', init='glorot_uniform')(x) 505 | if batch_norm: 506 | x = BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001)(x) 507 | x = Activation('relu')(x) 508 | else: 509 | x = ELU(alpha=1.0)(x) 510 | x = Dropout(0.8)(x) 511 | x = Dense(1, name='predictions', init='glorot_uniform')(x) 512 | x = Activation('sigmoid')(x) 513 | 514 | classifier = Model(input=extracted_features, 515 | output=x, name='classifier') 516 | fold_best_model_path = best_model_path + 'multicam_fold_{}'.format( 517 | cam) 518 | classifier.compile(optimizer=adam, loss='binary_crossentropy', 519 | metrics=['accuracy']) 520 | 521 | if not use_checkpoint: 522 | # ==================== TRAINING ======================== 523 | # weighting of each class: only the fall class gets 524 | # a different weight 525 | class_weight = {0: weight_0, 1: 1} 526 | 527 | callbacks = None 528 | if use_validation: 529 | # callback definition 530 | metric = 'val_loss' 531 | e = EarlyStopping(monitor=metric, min_delta=0, patience=100, 532 | mode='auto') 533 | c = ModelCheckpoint(fold_best_model_path, monitor=metric, 534 | save_best_only=True, 535 | save_weights_only=False, mode='auto') 536 | callbacks = [e, c] 537 | validation_data = None 538 | if use_validation: 539 | validation_data = (X_val,y_val) 540 | _mini_batch_size = mini_batch_size 541 | if mini_batch_size == 0: 542 | _mini_batch_size = X_train.shape[0] 543 | 544 | history = classifier.fit( 545 | X_train, y_train, 546 | validation_data=validation_data, 547 | batch_size=_mini_batch_size, 548 | nb_epoch=epochs, 549 | shuffle=True, 550 | class_weight=class_weight, 551 | callbacks=callbacks 552 | ) 553 | 554 | if not use_validation: 555 | classifier.save(fold_best_model_path) 556 | 557 | plot_training_info(plots_folder + exp, ['accuracy', 'loss'], 558 | save_plots, history.history) 559 | 560 | if use_validation and use_val_for_training: 561 | classifier = load_model(fold_best_model_path) 562 | 563 | # Use full training set (training+validation) 564 | X_train = np.concatenate((X_train, X_val), axis=0) 565 | y_train = np.concatenate((y_train, y_val), axis=0) 566 | 567 | history = classifier.fit( 568 | X_train, y_train, 569 | validation_data=validation_data, 570 | batch_size=_mini_batch_size, 571 | nb_epoch=epochs, 572 | shuffle='batch', 573 | class_weight=class_weight, 574 | callbacks=callbacks 575 | ) 576 | 577 | classifier.save(fold_best_model_path) 578 | 579 | # ==================== EVALUATION ======================== 580 | 581 | # Load best model 582 | print('Model loaded from checkpoint') 583 | classifier = load_model(fold_best_model_path) 584 | 585 | predicted = classifier.predict(X_test) 586 | for i in range(len(predicted)): 587 | if predicted[i] < threshold: 588 | predicted[i] = 0 589 | else: 590 | predicted[i] = 1 591 | # Array of predictions 0/1 592 | predicted = np.asarray(predicted).astype(int) 593 | 594 | # Compute metrics and print them 595 | cm = confusion_matrix(y_test, predicted,labels=[0,1]) 596 | tp = cm[0][0] 597 | fn = cm[0][1] 598 | fp = cm[1][0] 599 | tn = cm[1][1] 600 | tpr = tp/float(tp+fn) 601 | fpr = fp/float(fp+tn) 602 | fnr = fn/float(fn+tp) 603 | tnr = tn/float(tn+fp) 604 | precision = tp/float(tp+fp) 605 | recall = tp/float(tp+fn) 606 | specificity = tn/float(tn+fp) 607 | f1 = 2*float(precision*recall)/float(precision+recall) 608 | accuracy = accuracy_score(y_test, predicted) 609 | fpr, tpr, _ = roc_curve(y_test, predicted) 610 | roc_auc = auc(fpr, tpr) 611 | 612 | print('FOLD/CAMERA {} results:'.format(cam)) 613 | print('TP: {}, TN: {}, FP: {}, FN: {}'.format(tp,tn,fp,fn)) 614 | print('TPR: {}, TNR: {}, FPR: {}, FNR: {}'.format( 615 | tpr,tnr,fpr,fnr)) 616 | print('Sensitivity/Recall: {}'.format(recall)) 617 | print('Specificity: {}'.format(specificity)) 618 | print('Precision: {}'.format(precision)) 619 | print('F1-measure: {}'.format(f1)) 620 | print('Accuracy: {}'.format(accuracy)) 621 | print('AUC: {}'.format(roc_auc)) 622 | 623 | # Store the metrics for this epoch 624 | sensitivities.append(tp/float(tp+fn)) 625 | specificities.append(tn/float(tn+fp)) 626 | aucs.append(roc_auc) 627 | accuracies.append(accuracy) 628 | 629 | print('LEAVE-ONE-OUT RESULTS ===================') 630 | print("Sensitivity: %.2f%% (+/- %.2f%%)" % (np.mean(sensitivities), 631 | np.std(sensitivities))) 632 | print("Specificity: %.2f%% (+/- %.2f%%)" % (np.mean(specificities), 633 | np.std(specificities))) 634 | print("Accuracy: %.2f%% (+/- %.2f%%)" % (np.mean(accuracies), 635 | np.std(accuracies))) 636 | print("AUC: %.2f%% (+/- %.2f%%)" % (np.mean(aucs), np.std(aucs))) 637 | 638 | if __name__ == '__main__': 639 | if not os.path.exists(best_model_path): 640 | os.makedirs(best_model_path) 641 | if not os.path.exists(plots_folder): 642 | os.makedirs(plots_folder) 643 | 644 | main() 645 | -------------------------------------------------------------------------------- /temporalnet_urfd.py: -------------------------------------------------------------------------------- 1 | from __future__ import print_function 2 | from numpy.random import seed 3 | seed(1) 4 | import numpy as np 5 | import matplotlib 6 | matplotlib.use('Agg') 7 | from matplotlib import pyplot as plt 8 | import os 9 | import h5py 10 | import scipy.io as sio 11 | import cv2 12 | import glob 13 | import gc 14 | 15 | from keras.models import load_model, Model, Sequential 16 | from keras.layers import (Input, Conv2D, MaxPooling2D, Flatten, 17 | Activation, Dense, Dropout, ZeroPadding2D) 18 | from keras.optimizers import Adam 19 | from keras.layers.normalization import BatchNormalization 20 | from keras.callbacks import EarlyStopping, ModelCheckpoint 21 | from keras import backend as K 22 | from sklearn.metrics import confusion_matrix, accuracy_score 23 | from sklearn.model_selection import KFold, StratifiedShuffleSplit 24 | from keras.layers.advanced_activations import ELU 25 | 26 | os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID" 27 | os.environ["CUDA_VISIBLE_DEVICES"]="1" 28 | 29 | # CHANGE THESE VARIABLES --- 30 | data_folder = '/home/anunez/URFD_opticalflow/' 31 | mean_file = '/home/anunez/flow_mean.mat' 32 | vgg_16_weights = 'weights.h5' 33 | save_features = False 34 | save_plots = True 35 | 36 | # Set to 'True' if you want to restore a previous trained models 37 | # Training is skipped and test is done 38 | use_checkpoint = False # Set to True or False 39 | # -------------------------- 40 | 41 | best_model_path = 'models/' 42 | plots_folder = 'plots/' 43 | checkpoint_path = best_model_path + 'fold_' 44 | 45 | saved_files_folder = 'saved_features/' 46 | features_file = saved_files_folder + 'features_urfd_tf.h5' 47 | labels_file = saved_files_folder + 'labels_urfd_tf.h5' 48 | features_key = 'features' 49 | labels_key = 'labels' 50 | 51 | L = 10 52 | num_features = 4096 53 | batch_norm = True 54 | learning_rate = 0.0001 55 | mini_batch_size = 64 56 | weight_0 = 1 57 | epochs = 3000 58 | use_validation = False 59 | # After the training stops, use train+validation to train for 1 epoch 60 | use_val_for_training = False 61 | val_size = 100 62 | # Threshold to classify between positive and negative 63 | threshold = 0.5 64 | 65 | # Name of the experiment 66 | exp = 'urfd_lr{}_batchs{}_batchnorm{}_w0_{}'.format( 67 | learning_rate, 68 | mini_batch_size, 69 | batch_norm, 70 | weight_0 71 | ) 72 | 73 | def plot_training_info(case, metrics, save, history): 74 | ''' 75 | Function to create plots for train and validation loss and accuracy 76 | Input: 77 | * case: name for the plot, an 'accuracy.png' or 'loss.png' 78 | will be concatenated after the name. 79 | * metrics: list of metrics to store: 'loss' and/or 'accuracy' 80 | * save: boolean to store the plots or only show them. 81 | * history: History object returned by the Keras fit function. 82 | ''' 83 | val = False 84 | if 'val_acc' in history and 'val_loss' in history: 85 | val = True 86 | plt.ioff() 87 | if 'accuracy' in metrics: 88 | fig = plt.figure() 89 | plt.plot(history['acc']) 90 | if val: plt.plot(history['val_acc']) 91 | plt.title('model accuracy') 92 | plt.ylabel('accuracy') 93 | plt.xlabel('epoch') 94 | if val: 95 | plt.legend(['train', 'val'], loc='upper left') 96 | else: 97 | plt.legend(['train'], loc='upper left') 98 | if save == True: 99 | plt.savefig(case + 'accuracy.png') 100 | plt.gcf().clear() 101 | else: 102 | plt.show() 103 | plt.close(fig) 104 | 105 | # summarize history for loss 106 | if 'loss' in metrics: 107 | fig = plt.figure() 108 | plt.plot(history['loss']) 109 | if val: plt.plot(history['val_loss']) 110 | plt.title('model loss') 111 | plt.ylabel('loss') 112 | plt.xlabel('epoch') 113 | #plt.ylim(1e-3, 1e-2) 114 | plt.yscale("log") 115 | if val: 116 | plt.legend(['train', 'val'], loc='upper left') 117 | else: 118 | plt.legend(['train'], loc='upper left') 119 | if save == True: 120 | plt.savefig(case + 'loss.png') 121 | plt.gcf().clear() 122 | else: 123 | plt.show() 124 | plt.close(fig) 125 | 126 | def generator(list1, lits2): 127 | ''' 128 | Auxiliar generator: returns the ith element of both given list with 129 | each call to next() 130 | ''' 131 | for x,y in zip(list1,lits2): 132 | yield x, y 133 | 134 | def saveFeatures(feature_extractor, 135 | features_file, 136 | labels_file, 137 | features_key, 138 | labels_key): 139 | ''' 140 | Function to load the optical flow stacks, do a feed-forward through the 141 | feature extractor (VGG16) and 142 | store the output feature vectors in the file 'features_file' and the 143 | labels in 'labels_file'. 144 | Input: 145 | * feature_extractor: model VGG16 until the fc6 layer. 146 | * features_file: path to the hdf5 file where the extracted features are 147 | going to be stored 148 | * labels_file: path to the hdf5 file where the labels of the features 149 | are going to be stored 150 | * features_key: name of the key for the hdf5 file to store the features 151 | * labels_key: name of the key for the hdf5 file to store the labels 152 | ''' 153 | 154 | class0 = 'Falls' 155 | class1 = 'NotFalls' 156 | 157 | # Load the mean file to subtract to the images 158 | d = sio.loadmat(mean_file) 159 | flow_mean = d['image_mean'] 160 | 161 | # Fill the folders and classes arrays with all the paths to the data 162 | folders, classes = [], [] 163 | fall_videos = [f for f in os.listdir(data_folder + class0) 164 | if os.path.isdir(os.path.join(data_folder + class0, f))] 165 | fall_videos.sort() 166 | for fall_video in fall_videos: 167 | x_images = glob.glob(data_folder + class0 + '/' + fall_video 168 | + '/flow_x*.jpg') 169 | if int(len(x_images)) >= 10: 170 | folders.append(data_folder + class0 + '/' + fall_video) 171 | classes.append(0) 172 | 173 | not_fall_videos = [f for f in os.listdir(data_folder + class1) 174 | if os.path.isdir(os.path.join(data_folder + class1, f))] 175 | not_fall_videos.sort() 176 | for not_fall_video in not_fall_videos: 177 | x_images = glob.glob(data_folder + class1 + '/' + not_fall_video 178 | + '/flow_x*.jpg') 179 | if int(len(x_images)) >= 10: 180 | folders.append(data_folder + class1 + '/' + not_fall_video) 181 | classes.append(1) 182 | 183 | # Total amount of stacks, with sliding window = num_images-L+1 184 | nb_total_stacks = 0 185 | for folder in folders: 186 | x_images = glob.glob(folder + '/flow_x*.jpg') 187 | nb_total_stacks += len(x_images)-L+1 188 | 189 | # File to store the extracted features and datasets to store them 190 | # IMPORTANT NOTE: 'w' mode totally erases previous data 191 | h5features = h5py.File(features_file,'w') 192 | h5labels = h5py.File(labels_file,'w') 193 | dataset_features = h5features.create_dataset(features_key, 194 | shape=(nb_total_stacks, num_features), 195 | dtype='float64') 196 | dataset_labels = h5labels.create_dataset(labels_key, 197 | shape=(nb_total_stacks, 1), 198 | dtype='float64') 199 | cont = 0 200 | 201 | for folder, label in zip(folders, classes): 202 | x_images = glob.glob(folder + '/flow_x*.jpg') 203 | x_images.sort() 204 | y_images = glob.glob(folder + '/flow_y*.jpg') 205 | y_images.sort() 206 | nb_stacks = len(x_images)-L+1 207 | # Here nb_stacks optical flow stacks will be stored 208 | flow = np.zeros(shape=(224,224,2*L,nb_stacks), dtype=np.float64) 209 | gen = generator(x_images,y_images) 210 | for i in range(len(x_images)): 211 | flow_x_file, flow_y_file = gen.next() 212 | img_x = cv2.imread(flow_x_file, cv2.IMREAD_GRAYSCALE) 213 | img_y = cv2.imread(flow_y_file, cv2.IMREAD_GRAYSCALE) 214 | # Assign an image i to the jth stack in the kth position, but also 215 | # in the j+1th stack in the k+1th position and so on 216 | # (for sliding window) 217 | for s in list(reversed(range(min(10,i+1)))): 218 | if i-s < nb_stacks: 219 | flow[:,:,2*s, i-s] = img_x 220 | flow[:,:,2*s+1,i-s] = img_y 221 | del img_x,img_y 222 | gc.collect() 223 | 224 | # Subtract mean 225 | flow = flow - np.tile(flow_mean[...,np.newaxis], 226 | (1, 1, 1, flow.shape[3])) 227 | flow = np.transpose(flow, (3, 0, 1, 2)) 228 | predictions = np.zeros((flow.shape[0], num_features), dtype=np.float64) 229 | truth = np.zeros((flow.shape[0], 1), dtype=np.float64) 230 | # Process each stack: do the feed-forward pass and store 231 | # in the hdf5 file the output 232 | for i in range(flow.shape[0]): 233 | prediction = feature_extractor.predict( 234 | np.expand_dims(flow[i, ...],0)) 235 | predictions[i, ...] = prediction 236 | truth[i] = label 237 | dataset_features[cont:cont+flow.shape[0],:] = predictions 238 | dataset_labels[cont:cont+flow.shape[0],:] = truth 239 | cont += flow.shape[0] 240 | h5features.close() 241 | h5labels.close() 242 | 243 | def test_video(feature_extractor, video_path, ground_truth): 244 | # Load the mean file to subtract to the images 245 | d = sio.loadmat(mean_file) 246 | flow_mean = d['image_mean'] 247 | 248 | x_images = glob.glob(video_path + '/flow_x*.jpg') 249 | x_images.sort() 250 | y_images = glob.glob(video_path + '/flow_y*.jpg') 251 | y_images.sort() 252 | nb_stacks = len(x_images)-L+1 253 | # Here nb_stacks optical flow stacks will be stored 254 | flow = np.zeros(shape=(224,224,2*L,nb_stacks), dtype=np.float64) 255 | gen = generator(x_images,y_images) 256 | for i in range(len(x_images)): 257 | flow_x_file, flow_y_file = gen.next() 258 | img_x = cv2.imread(flow_x_file, cv2.IMREAD_GRAYSCALE) 259 | img_y = cv2.imread(flow_y_file, cv2.IMREAD_GRAYSCALE) 260 | # Assign an image i to the jth stack in the kth position, but also 261 | # in the j+1th stack in the k+1th position and so on 262 | # (for sliding window) 263 | for s in list(reversed(range(min(10,i+1)))): 264 | if i-s < nb_stacks: 265 | flow[:,:,2*s, i-s] = img_x 266 | flow[:,:,2*s+1,i-s] = img_y 267 | del img_x,img_y 268 | gc.collect() 269 | flow = flow - np.tile(flow_mean[...,np.newaxis], (1, 1, 1, flow.shape[3])) 270 | flow = np.transpose(flow, (3, 0, 1, 2)) 271 | predictions = np.zeros((flow.shape[0], num_features), dtype=np.float64) 272 | truth = np.zeros((flow.shape[0], 1), dtype=np.float64) 273 | # Process each stack: do the feed-forward pass 274 | for i in range(flow.shape[0]): 275 | prediction = feature_extractor.predict(np.expand_dims(flow[i, ...],0)) 276 | predictions[i, ...] = prediction 277 | truth[i] = ground_truth 278 | return predictions, truth 279 | 280 | def main(): 281 | # ======================================================================== 282 | # VGG-16 ARCHITECTURE 283 | # ======================================================================== 284 | model = Sequential() 285 | 286 | model.add(ZeroPadding2D((1, 1), input_shape=(224, 224, 20))) 287 | model.add(Conv2D(64, (3, 3), activation='relu', name='conv1_1')) 288 | model.add(ZeroPadding2D((1, 1))) 289 | model.add(Conv2D(64, (3, 3), activation='relu', name='conv1_2')) 290 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 291 | 292 | model.add(ZeroPadding2D((1, 1))) 293 | model.add(Conv2D(128, (3, 3), activation='relu', name='conv2_1')) 294 | model.add(ZeroPadding2D((1, 1))) 295 | model.add(Conv2D(128, (3, 3), activation='relu', name='conv2_2')) 296 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 297 | 298 | model.add(ZeroPadding2D((1, 1))) 299 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_1')) 300 | model.add(ZeroPadding2D((1, 1))) 301 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_2')) 302 | model.add(ZeroPadding2D((1, 1))) 303 | model.add(Conv2D(256, (3, 3), activation='relu', name='conv3_3')) 304 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 305 | 306 | model.add(ZeroPadding2D((1, 1))) 307 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_1')) 308 | model.add(ZeroPadding2D((1, 1))) 309 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_2')) 310 | model.add(ZeroPadding2D((1, 1))) 311 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv4_3')) 312 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 313 | 314 | model.add(ZeroPadding2D((1, 1))) 315 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_1')) 316 | model.add(ZeroPadding2D((1, 1))) 317 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_2')) 318 | model.add(ZeroPadding2D((1, 1))) 319 | model.add(Conv2D(512, (3, 3), activation='relu', name='conv5_3')) 320 | model.add(MaxPooling2D((2, 2), strides=(2, 2))) 321 | 322 | model.add(Flatten()) 323 | model.add(Dense(num_features, name='fc6', 324 | kernel_initializer='glorot_uniform')) 325 | 326 | # ======================================================================== 327 | # WEIGHT INITIALIZATION 328 | # ======================================================================== 329 | layerscaffe = ['conv1_1', 'conv1_2', 'conv2_1', 'conv2_2', 'conv3_1', 330 | 'conv3_2', 'conv3_3', 'conv4_1', 'conv4_2', 'conv4_3', 331 | 'conv5_1', 'conv5_2', 'conv5_3', 'fc6', 'fc7', 'fc8'] 332 | h5 = h5py.File(vgg_16_weights, 'r') 333 | 334 | layer_dict = dict([(layer.name, layer) for layer in model.layers]) 335 | 336 | # Copy the weights stored in the 'vgg_16_weights' file to the 337 | # feature extractor part of the VGG16 338 | for layer in layerscaffe[:-3]: 339 | w2, b2 = h5['data'][layer]['0'], h5['data'][layer]['1'] 340 | w2 = np.transpose(np.asarray(w2), (2,3,1,0)) 341 | w2 = w2[::-1, ::-1, :, :] 342 | b2 = np.asarray(b2) 343 | layer_dict[layer].set_weights((w2, b2)) 344 | 345 | # Copy the weights of the first fully-connected layer (fc6) 346 | layer = layerscaffe[-3] 347 | w2, b2 = h5['data'][layer]['0'], h5['data'][layer]['1'] 348 | w2 = np.transpose(np.asarray(w2), (1,0)) 349 | b2 = np.asarray(b2) 350 | layer_dict[layer].set_weights((w2, b2)) 351 | 352 | # ======================================================================== 353 | # FEATURE EXTRACTION 354 | # ======================================================================== 355 | if save_features: 356 | saveFeatures(model, features_file, 357 | labels_file, features_key, 358 | labels_key) 359 | 360 | # ======================================================================== 361 | # TRAINING 362 | # ======================================================================== 363 | 364 | adam = Adam(lr=learning_rate, beta_1=0.9, beta_2=0.999, 365 | epsilon=1e-08) 366 | model.compile(optimizer=adam, loss='categorical_crossentropy', 367 | metrics=['accuracy']) 368 | 369 | h5features = h5py.File(features_file, 'r') 370 | h5labels = h5py.File(labels_file, 'r') 371 | 372 | # X_full will contain all the feature vectors extracted 373 | # from optical flow images 374 | X_full = h5features[features_key] 375 | _y_full = np.asarray(h5labels[labels_key]) 376 | 377 | zeroes_full = np.asarray(np.where(_y_full==0)[0]) 378 | ones_full = np.asarray(np.where(_y_full==1)[0]) 379 | zeroes_full.sort() 380 | ones_full.sort() 381 | 382 | # Use a 5 fold cross-validation 383 | kf_falls = KFold(n_splits=5, shuffle=True) 384 | kf_falls.get_n_splits(X_full[zeroes_full, ...]) 385 | 386 | kf_nofalls = KFold(n_splits=5, shuffle=True) 387 | kf_nofalls.get_n_splits(X_full[ones_full, ...]) 388 | 389 | sensitivities = [] 390 | specificities = [] 391 | fars = [] 392 | mdrs = [] 393 | accuracies = [] 394 | 395 | fold_number = 1 396 | # CROSS-VALIDATION: Stratified partition of the dataset into 397 | # train/test sets 398 | for ((train_index_falls, test_index_falls), 399 | (train_index_nofalls, test_index_nofalls)) in zip( 400 | kf_falls.split(X_full[zeroes_full, ...]), 401 | kf_nofalls.split(X_full[ones_full, ...]) 402 | ): 403 | 404 | train_index_falls = np.asarray(train_index_falls) 405 | test_index_falls = np.asarray(test_index_falls) 406 | train_index_nofalls = np.asarray(train_index_nofalls) 407 | test_index_nofalls = np.asarray(test_index_nofalls) 408 | 409 | X = np.concatenate(( 410 | X_full[zeroes_full, ...][train_index_falls, ...], 411 | X_full[ones_full, ...][train_index_nofalls, ...] 412 | )) 413 | _y = np.concatenate(( 414 | _y_full[zeroes_full, ...][train_index_falls, ...], 415 | _y_full[ones_full, ...][train_index_nofalls, ...] 416 | )) 417 | X_test = np.concatenate(( 418 | X_full[zeroes_full, ...][test_index_falls, ...], 419 | X_full[ones_full, ...][test_index_nofalls, ...] 420 | )) 421 | y_test = np.concatenate(( 422 | _y_full[zeroes_full, ...][test_index_falls, ...], 423 | _y_full[ones_full, ...][test_index_nofalls, ...] 424 | )) 425 | 426 | if use_validation: 427 | # Create a validation subset from the training set 428 | zeroes = np.asarray(np.where(_y==0)[0]) 429 | ones = np.asarray(np.where(_y==1)[0]) 430 | 431 | zeroes.sort() 432 | ones.sort() 433 | 434 | trainval_split_0 = StratifiedShuffleSplit(n_splits=1, 435 | test_size=val_size/2, 436 | random_state=7) 437 | indices_0 = trainval_split_0.split(X[zeroes,...], 438 | np.argmax(_y[zeroes,...], 1)) 439 | trainval_split_1 = StratifiedShuffleSplit(n_splits=1, 440 | test_size=val_size/2, 441 | random_state=7) 442 | indices_1 = trainval_split_1.split(X[ones,...], 443 | np.argmax(_y[ones,...], 1)) 444 | train_indices_0, val_indices_0 = indices_0.next() 445 | train_indices_1, val_indices_1 = indices_1.next() 446 | 447 | X_train = np.concatenate([X[zeroes,...][train_indices_0,...], 448 | X[ones,...][train_indices_1,...]],axis=0) 449 | y_train = np.concatenate([_y[zeroes,...][train_indices_0,...], 450 | _y[ones,...][train_indices_1,...]],axis=0) 451 | X_val = np.concatenate([X[zeroes,...][val_indices_0,...], 452 | X[ones,...][val_indices_1,...]],axis=0) 453 | y_val = np.concatenate([_y[zeroes,...][val_indices_0,...], 454 | _y[ones,...][val_indices_1,...]],axis=0) 455 | else: 456 | X_train = X 457 | y_train = _y 458 | 459 | # Balance the number of positive and negative samples so that 460 | # there is the same amount of each of them 461 | all0 = np.asarray(np.where(y_train==0)[0]) 462 | all1 = np.asarray(np.where(y_train==1)[0]) 463 | 464 | if len(all0) < len(all1): 465 | all1 = np.random.choice(all1, len(all0), replace=False) 466 | else: 467 | all0 = np.random.choice(all0, len(all1), replace=False) 468 | allin = np.concatenate((all0.flatten(),all1.flatten())) 469 | allin.sort() 470 | X_train = X_train[allin,...] 471 | y_train = y_train[allin] 472 | 473 | # ==================== CLASSIFIER ======================== 474 | extracted_features = Input(shape=(num_features,), 475 | dtype='float32', name='input') 476 | if batch_norm: 477 | x = BatchNormalization(axis=-1, momentum=0.99, 478 | epsilon=0.001)(extracted_features) 479 | x = Activation('relu')(x) 480 | else: 481 | x = ELU(alpha=1.0)(extracted_features) 482 | 483 | x = Dropout(0.9)(x) 484 | x = Dense(4096, name='fc2', kernel_initializer='glorot_uniform')(x) 485 | if batch_norm: 486 | x = BatchNormalization(axis=-1, momentum=0.99, epsilon=0.001)(x) 487 | x = Activation('relu')(x) 488 | else: 489 | x = ELU(alpha=1.0)(x) 490 | x = Dropout(0.8)(x) 491 | x = Dense(1, name='predictions', 492 | kernel_initializer='glorot_uniform')(x) 493 | x = Activation('sigmoid')(x) 494 | 495 | classifier = Model(input=extracted_features, 496 | output=x, name='classifier') 497 | fold_best_model_path = best_model_path + 'urfd_fold_{}.h5'.format( 498 | fold_number) 499 | classifier.compile(optimizer=adam, loss='binary_crossentropy', 500 | metrics=['accuracy']) 501 | 502 | if not use_checkpoint: 503 | # ==================== TRAINING ======================== 504 | # weighting of each class: only the fall class gets 505 | # a different weight 506 | class_weight = {0: weight_0, 1: 1} 507 | 508 | callbacks = None 509 | if use_validation: 510 | # callback definition 511 | metric = 'val_loss' 512 | e = EarlyStopping(monitor=metric, min_delta=0, patience=100, 513 | mode='auto') 514 | c = ModelCheckpoint(fold_best_model_path, monitor=metric, 515 | save_best_only=True, 516 | save_weights_only=False, mode='auto') 517 | callbacks = [e, c] 518 | validation_data = None 519 | if use_validation: 520 | validation_data = (X_val,y_val) 521 | _mini_batch_size = mini_batch_size 522 | if mini_batch_size == 0: 523 | _mini_batch_size = X_train.shape[0] 524 | 525 | history = classifier.fit( 526 | X_train, y_train, 527 | validation_data=validation_data, 528 | batch_size=_mini_batch_size, 529 | nb_epoch=epochs, 530 | shuffle=True, 531 | class_weight=class_weight, 532 | callbacks=callbacks 533 | ) 534 | 535 | if not use_validation: 536 | classifier.save(fold_best_model_path) 537 | 538 | plot_training_info(plots_folder + exp, ['accuracy', 'loss'], 539 | save_plots, history.history) 540 | 541 | if use_validation and use_val_for_training: 542 | classifier = load_model(fold_best_model_path) 543 | 544 | # Use full training set (training+validation) 545 | X_train = np.concatenate((X_train, X_val), axis=0) 546 | y_train = np.concatenate((y_train, y_val), axis=0) 547 | 548 | history = classifier.fit( 549 | X_train, y_train, 550 | validation_data=validation_data, 551 | batch_size=_mini_batch_size, 552 | nb_epoch=epochs, 553 | shuffle='batch', 554 | class_weight=class_weight, 555 | callbacks=callbacks 556 | ) 557 | 558 | classifier.save(fold_best_model_path) 559 | 560 | # ==================== EVALUATION ======================== 561 | 562 | # Load best model 563 | print('Model loaded from checkpoint') 564 | classifier = load_model(fold_best_model_path) 565 | 566 | predicted = classifier.predict(np.asarray(X_test)) 567 | for i in range(len(predicted)): 568 | if predicted[i] < threshold: 569 | predicted[i] = 0 570 | else: 571 | predicted[i] = 1 572 | # Array of predictions 0/1 573 | predicted = np.asarray(predicted).astype(int) 574 | # Compute metrics and print them 575 | cm = confusion_matrix(y_test, predicted,labels=[0,1]) 576 | tp = cm[0][0] 577 | fn = cm[0][1] 578 | fp = cm[1][0] 579 | tn = cm[1][1] 580 | tpr = tp/float(tp+fn) 581 | fpr = fp/float(fp+tn) 582 | fnr = fn/float(fn+tp) 583 | tnr = tn/float(tn+fp) 584 | precision = tp/float(tp+fp) 585 | recall = tp/float(tp+fn) 586 | specificity = tn/float(tn+fp) 587 | f1 = 2*float(precision*recall)/float(precision+recall) 588 | accuracy = accuracy_score(y_test, predicted) 589 | 590 | print('FOLD {} results:'.format(fold_number)) 591 | print('TP: {}, TN: {}, FP: {}, FN: {}'.format(tp,tn,fp,fn)) 592 | print('TPR: {}, TNR: {}, FPR: {}, FNR: {}'.format( 593 | tpr,tnr,fpr,fnr)) 594 | print('Sensitivity/Recall: {}'.format(recall)) 595 | print('Specificity: {}'.format(specificity)) 596 | print('Precision: {}'.format(precision)) 597 | print('F1-measure: {}'.format(f1)) 598 | print('Accuracy: {}'.format(accuracy)) 599 | 600 | # Store the metrics for this epoch 601 | sensitivities.append(tp/float(tp+fn)) 602 | specificities.append(tn/float(tn+fp)) 603 | fars.append(fpr) 604 | mdrs.append(fnr) 605 | accuracies.append(accuracy) 606 | fold_number += 1 607 | 608 | print('5-FOLD CROSS-VALIDATION RESULTS ===================') 609 | print("Sensitivity: %.2f%% (+/- %.2f%%)" % (np.mean(sensitivities)*100., 610 | np.std(sensitivities)*100.)) 611 | print("Specificity: %.2f%% (+/- %.2f%%)" % (np.mean(specificities)*100., 612 | np.std(specificities)*100.)) 613 | print("FAR: %.2f%% (+/- %.2f%%)" % (np.mean(fars)*100., 614 | np.std(fars)*100.)) 615 | print("MDR: %.2f%% (+/- %.2f%%)" % (np.mean(mdrs)*100., 616 | np.std(mdrs)*100.)) 617 | print("Accuracy: %.2f%% (+/- %.2f%%)" % (np.mean(accuracies)*100., 618 | np.std(accuracies)*100.)) 619 | 620 | if __name__ == '__main__': 621 | if not os.path.exists(best_model_path): 622 | os.makedirs(best_model_path) 623 | if not os.path.exists(plots_folder): 624 | os.makedirs(plots_folder) 625 | 626 | main() 627 | --------------------------------------------------------------------------------