├── AVEC task description.pdf ├── Figure ├── . ... ├── CNN.png ├── ccc_per_behaviour.png └── pipeline.png ├── README.md ├── cut_videos.m ├── demo_extract_feature.m~ ├── demo_extract_feature_resample.m ├── demo_extract_feature_select.m ├── example_data.mat ├── flat_data.m ├── fourier_transform_resample.m ├── fourier_transform_select.m ├── getVideoFeature.m ├── preprocess.m └── task_time_stamp_avec2013.zip /AVEC task description.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SSYSteve/Human-behaviour-based-depression-analysis-using-hand-crafted-statistics-and-deep-learned/d585b4d12c40eba28430eea12d33c4cc1898973e/AVEC task description.pdf -------------------------------------------------------------------------------- /Figure/. ...: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /Figure/CNN.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SSYSteve/Human-behaviour-based-depression-analysis-using-hand-crafted-statistics-and-deep-learned/d585b4d12c40eba28430eea12d33c4cc1898973e/Figure/CNN.png -------------------------------------------------------------------------------- /Figure/ccc_per_behaviour.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SSYSteve/Human-behaviour-based-depression-analysis-using-hand-crafted-statistics-and-deep-learned/d585b4d12c40eba28430eea12d33c4cc1898973e/Figure/ccc_per_behaviour.png -------------------------------------------------------------------------------- /Figure/pipeline.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SSYSteve/Human-behaviour-based-depression-analysis-using-hand-crafted-statistics-and-deep-learned/d585b4d12c40eba28430eea12d33c4cc1898973e/Figure/pipeline.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Human behaviour-based automatic depression analysis using hand-crafted statistics and deep learned spectral features 2 | 3 | This is the spectral data transformation code used for two papers: 4 | 5 | [1] Human behaviour-based automatic depression analysis using hand-crafted statistics and deep learned spectral features (oral presentation of FG 2018, link: https://ieeexplore.ieee.org/document/8373825/) 6 | 7 | [2] Spectral Representation of Behaviour Primitives for Depression Analysis (IEEE Transactions on Affective Computing, link: https://ieeexplore.ieee.org/abstract/document/8976305) 8 | 9 | 10 | Here, we also provide the task time stamps of AVEC 2013 depression challenge data, which were annotated by two native erman speakers. 11 | 12 | # Get started 13 | 14 | You only need to use MATLAB to open the following files 15 | 16 | demo_extract_feature_select.m: 17 | 18 | demo for extracting spectral map and features using the first frequency alignment method (select) in [2] 19 | 20 | demo_extract_feature_resample.m: 21 | 22 | demo for extracting spectral map and features using the second frequency alignment method (resample) in [2] 23 | 24 | The task time stamps of AVEC 2013 depression challenge data is avaliable in task_time_stamp_avec2013.zip, and its description is avaliable in AVEC task description.pdf. 25 | 26 | 27 | # Comments 28 | 29 | The code is only for the spectral feature extration of the above papers, which is also the main contribution of our work. 30 | 31 | The proposed approach reduce the dimensioanlity of long multi-channel time-sereis data to two small spectral heatmaps or spectral vector. 32 | 33 | It also convert data of variable lengths to heatmaps of fixed size. 34 | 35 | We haven't show the CNN or ANN training scripts because CNN or ANN strcutures always depend on the task. 36 | 37 | When using this code for feature extraction, you should adapt the pre-processing function to your own task. 38 | 39 | After the feature extraction, you should adapt the post-processing method (e.g. feature selection, PCA, etc) to your own task. 40 | 41 | if you requires the feature selection detatils for the depression tasks present in the paper, please contact the email below. 42 | 43 | # Approach overview 44 | 45 | ![alt text](https://github.com/SSYSteve/Human-behaviour-based-depression-analysis-using-hand-crafted-statistics-and-deep-learned/blob/master/Figure/pipeline.png) 46 | 47 | ![alt text](https://github.com/SSYSteve/Human-behaviour-based-depression-analysis-using-hand-crafted-statistics-and-deep-learned/blob/master/Figure/CNN.png) 48 | 49 | # Ablation study result 50 | 51 | ![alt text](https://github.com/SSYSteve/Human-behaviour-based-depression-analysis-using-hand-crafted-statistics-and-deep-learned/blob/master/Figure/ccc_per_behaviour.png) 52 | 53 | 54 | # If you decide to use the code or data here, please kindly cite our papers: 55 | 56 | [1] Song, Siyang, Linlin Shen, and Michel Valstar. "Human behaviour-based automatic depression analysis using hand-crafted statistics and deep learned spectral features." Automatic Face & Gesture Recognition (FG 2018), 2018 13th IEEE International Conference on. IEEE, 2018. 57 | 58 | @inproceedings{song2018human, 59 | title={Human behaviour-based automatic depression analysis using hand-crafted statistics and deep learned spectral features}, 60 | author={Song, Siyang and Shen, Linlin and Valstar, Michel}, 61 | booktitle={2018 13th IEEE International Conference on Automatic Face \& Gesture Recognition (FG 2018)}, 62 | pages={158--165}, 63 | year={2018}, 64 | organization={IEEE} 65 | } 66 | 67 | [2] Song, Siyang, Shashank Jaiswal, Linlin Shen, and Michel Valstar. "Spectral Representation of Behaviour Primitives for Depression Analysis." IEEE Transactions on Affective Computing (2020). 68 | 69 | @article{song2020spectral, 70 | title={Spectral Representation of Behaviour Primitives for Depression Analysis}, 71 | author={Song, Siyang and Jaiswal, Shashank and Shen, Linlin and Valstar, Michel}, 72 | journal={IEEE Transactions on Affective Computing}, 73 | year={2020}, 74 | publisher={IEEE} 75 | } 76 | 77 | If you have any probelm, you are more than welcome to contact ss2796@cam.ac.uk 78 | -------------------------------------------------------------------------------- /cut_videos.m: -------------------------------------------------------------------------------- 1 | % Song, Siyang, Shashank Jaiswal, Linlin Shen, and Michel Valstar 2 | % Spectral Representation of Behaviour Primitives for Depression Analysis. 3 | % IEEE Transactions on Affective Computing (2020) 4 | % Email: siyang.song@nottingham.ac.uk 5 | function [raw_data,num_keep_frame,raw_num_keep_frame] = cut_videos(raw_data,fre_resolution) 6 | 7 | len = size(raw_data,2); 8 | raw_num_keep_frame = floor(len/fre_resolution); 9 | num_keep_frame = raw_num_keep_frame*fre_resolution; 10 | num_delete = len - num_keep_frame; 11 | 12 | num_delete_start = ceil(num_delete/2); 13 | num_delete_end = floor(num_delete/2); 14 | 15 | raw_data(:,[1:num_delete_start,end-num_delete_end+1:end])=[]; 16 | 17 | end 18 | 19 | -------------------------------------------------------------------------------- /demo_extract_feature.m~: -------------------------------------------------------------------------------- 1 | clear all;clc 2 | 3 | 4 | %% setting 5 | 6 | Primitive_num = 29; % the number of behaviour signals 7 | 8 | fre_resolution = 128; % the number of frequency kept 9 | 10 | file_name = 'example_data.mat'; 11 | 12 | N = 2000; % the 13 | 14 | 15 | %% -------------------------------------------------------------------------------- /demo_extract_feature_resample.m: -------------------------------------------------------------------------------- 1 | % Song, Siyang, Shashank Jaiswal, Linlin Shen, and Michel Valstar 2 | % Spectral Representation of Behaviour Primitives for Depression Analysis. 3 | % IEEE Transactions on Affective Computing (2020) 4 | % Email: siyang.song@nottingham.ac.uk 5 | 6 | clear all;clc 7 | 8 | %% setting 9 | 10 | Primitive_num = 29; % the number of behaviour signals 11 | 12 | N = 80; % Choosing TOP-N frequency (N < fre_resolution/2) 13 | 14 | fre_resolution = 256; % sampling frequency 15 | 16 | file_name = 'example_data.mat'; 17 | 18 | 19 | %% pre_processing 20 | 21 | t_length = N*Primitive_num; % set feature length for amp/phase map 22 | 23 | raw_data = load(file_name); % load data 24 | 25 | raw_data = raw_data.example_data; 26 | 27 | processed_data = preprocess(raw_data); % substracting median values, ypu can customized your own preprocess method here 28 | 29 | processed_data = processed_data'; 30 | 31 | %% feature extraction 32 | 33 | sta_fea = getVideoFeature(processed_data); % compute statistics features 34 | 35 | [amp_map, phase_map] = fourier_transform_resample(processed_data, N,fre_resolution);% 2-D amplitude map and phase map generation 36 | 37 | amp_flat_data = flat_data(amp_map,t_length, N);% 1-D amplitude feature generation 38 | 39 | phase_flat_data = flat_data(phase_map,t_length, N);% 1-D phase feature generation 40 | 41 | 42 | 43 | -------------------------------------------------------------------------------- /demo_extract_feature_select.m: -------------------------------------------------------------------------------- 1 | % Song, Siyang, Shashank Jaiswal, Linlin Shen, and Michel Valstar 2 | % Spectral Representation of Behaviour Primitives for Depression Analysis. 3 | % IEEE Transactions on Affective Computing (2020) 4 | % Email: siyang.song@nottingham.ac.uk 5 | 6 | clear all;clc 7 | 8 | %% setting 9 | 10 | Primitive_num = 18; % the number of behaviour signals 11 | 12 | N = 80; % Choosing TOP-N frequency (N < fre_resolution/2) 13 | 14 | fre_resolution = 256; % sampling frequency 15 | 16 | file_name = 'example_data.mat'; 17 | 18 | 19 | %% pre_processing 20 | 21 | t_length = N*Primitive_num; % set feature length for amp/phase map 22 | 23 | %raw_data = load(file_name); % load data 24 | raw_data = rand(10000,18); 25 | 26 | %raw_data = raw_data.example_data; 27 | 28 | raw_data = raw_data'; 29 | 30 | [raw_data,num_keep_frame,num_multiple] = cut_videos(raw_data,fre_resolution); % cut videos 31 | 32 | processed_data = preprocess(raw_data); % substracting median values, ypu can customized your own preprocess method here 33 | 34 | 35 | %% feature extraction 36 | 37 | sta_fea = getVideoFeature(processed_data); % compute statistics features 38 | 39 | [amp_map, phase_map] = fourier_transform_select(processed_data, N,num_multiple,fre_resolution);% 2-D amplitude map and phase map generation 40 | 41 | amp_flat_data = flat_data(amp_map,t_length, N);% 1-D amplitude feature generation 42 | 43 | phase_flat_data = flat_data(phase_map,t_length, N);% 1-D phase feature generation 44 | 45 | 46 | 47 | -------------------------------------------------------------------------------- /example_data.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SSYSteve/Human-behaviour-based-depression-analysis-using-hand-crafted-statistics-and-deep-learned/d585b4d12c40eba28430eea12d33c4cc1898973e/example_data.mat -------------------------------------------------------------------------------- /flat_data.m: -------------------------------------------------------------------------------- 1 | function [ flat_data ] = flat_data( data, t_length, N) 2 | %UNTITLED Summary of this function goes here 3 | % Detailed explanation goes here 4 | 5 | flat_data = zeros(1,t_length); 6 | 7 | row_num = size(data,1); 8 | 9 | for i = 1:row_num 10 | 11 | flat_data(1,(i-1)*N+1:i*N) = data(i,:); 12 | 13 | end 14 | 15 | 16 | end 17 | 18 | -------------------------------------------------------------------------------- /fourier_transform_resample.m: -------------------------------------------------------------------------------- 1 | % Song, Siyang, Shashank Jaiswal, Linlin Shen, and Michel Valstar 2 | % Spectral Representation of Behaviour Primitives for Depression Analysis. 3 | % IEEE Transactions on Affective Computing (2020) 4 | % Email: siyang.song@nottingham.ac.uk 5 | % input: 6 | %--all_data: Multi-channel time-series facial behaviour primitives data 7 | %--num_fre: sampling frequency 8 | %--N: final used TOP-N frequencies 9 | % output: 10 | %--amp_map_return: amplitude spectrum map 11 | %--phase_map_return: phase spectrum map 12 | 13 | function [amp_map_return, phase_map_return] = fourier_transform_resample(all_data, N, num_fre) 14 | 15 | [channel_num, length] = size(all_data); 16 | amp_map = zeros(channel_num,num_fre); 17 | phase_map = zeros(channel_num,num_fre); 18 | 19 | 20 | for i = 1:channel_num 21 | 22 | temp_contain = fft(all_data(i,:)); 23 | 24 | if mod(length,2) == 0 25 | 26 | temp_contain = temp_contain(:,1:length/2+1); 27 | 28 | else 29 | 30 | temp_contain = temp_contain(:,1:(length+1)/2); 31 | 32 | end 33 | 34 | temp_resample_data = resample(temp_contain,num_fre,size(temp_contain,2)); 35 | amp_map(i,:) = abs(temp_resample_data)/length; 36 | phase_map(i,:) = angle(temp_resample_data); 37 | 38 | end 39 | 40 | amp_map_return = amp_map(:,1:N); 41 | phase_map_return = phase_map(:,1:N); 42 | 43 | 44 | end 45 | 46 | -------------------------------------------------------------------------------- /fourier_transform_select.m: -------------------------------------------------------------------------------- 1 | % Song, Siyang, Shashank Jaiswal, Linlin Shen, and Michel Valstar 2 | % Spectral Representation of Behaviour Primitives for Depression Analysis. 3 | % IEEE Transactions on Affective Computing (2020) 4 | % Email: siyang.song@nottingham.ac.uk 5 | function [ amp_map_return, phase_map_return ] = fourier_transform_select( all_data, N, num_multiple,fre_resolution) 6 | 7 | 8 | [channel_num, length] = size(all_data); 9 | %temp_contain = zeros(channel_num,length); 10 | amp_map = zeros(channel_num,length); 11 | phase_map = zeros(channel_num,length); 12 | common_temp_amp = zeros(channel_num,fre_resolution); 13 | common_temp_pha = zeros(channel_num,fre_resolution); 14 | 15 | 16 | for i = 1:channel_num 17 | 18 | temp_contain = fft(all_data(i,:)); 19 | amp_map(i,:) = abs(temp_contain/length); 20 | phase_map(i,:) = angle(temp_contain); 21 | 22 | end 23 | 24 | 25 | for j = 1:fre_resolution 26 | 27 | common_temp_amp(:,j) = amp_map(:,1+(j-1)*num_multiple); 28 | common_temp_pha(:,j) = phase_map(:,1+(j-1)*num_multiple); 29 | 30 | end 31 | 32 | 33 | amp_map_return = [amp_map(:,length/2+1),common_temp_amp(:,1:N-1)]; 34 | phase_map_return = [phase_map(:,length/2+1),common_temp_pha(:,1:N-1)]; 35 | 36 | 37 | 38 | end 39 | 40 | -------------------------------------------------------------------------------- /getVideoFeature.m: -------------------------------------------------------------------------------- 1 | % Song, Siyang, Shashank Jaiswal, Linlin Shen, and Michel Valstar 2 | % Spectral Representation of Behaviour Primitives for Depression Analysis. 3 | % IEEE Transactions on Affective Computing (2020) 4 | % Email: siyang.song@nottingham.ac.uk 5 | 6 | function videoFea = getVideoFeature( feaVec ) 7 | 8 | feaNum = size(feaVec,1); 9 | videoFea = zeros(1,feaNum*12-6*6); 10 | for i = 1:feaNum 11 | 12 | videoFea(1,(i-1)*12 + 1) = mean(feaVec(i,:)); 13 | videoFea(1,(i-1)*12 + 2) = std(feaVec(i,:)); 14 | videoFea(1,(i-1)*12 + 3) = max(feaVec(i,:)); 15 | videoFea(1,(i-1)*12 + 4) = min(feaVec(i,:)); 16 | videoFea(1,(i-1)*12 + 5) = mean(diff(feaVec(i,:))); 17 | videoFea(1,(i-1)*12 + 6) = std(diff(feaVec(i,:))); 18 | videoFea(1,(i-1)*12 + 7) = max(diff(feaVec(i,:))); 19 | videoFea(1,(i-1)*12 + 8) = min(diff(feaVec(i,:))); 20 | a = feaVec(i,1:end-2); 21 | b = feaVec(i,3:end); 22 | videoFea(1,(i-1)*12 + 9) = mean(a-b); 23 | videoFea(1,(i-1)*12 + 10) = std(a-b); 24 | videoFea(1,(i-1)*12 + 11) = max(a-b); 25 | videoFea(1,(i-1)*12 + 12) = min(a-b); 26 | 27 | end 28 | 29 | end 30 | 31 | -------------------------------------------------------------------------------- /preprocess.m: -------------------------------------------------------------------------------- 1 | % Song, Siyang, Shashank Jaiswal, Linlin Shen, and Michel Valstar 2 | % Spectral Representation of Behaviour Primitives for Depression Analysis. 3 | % IEEE Transactions on Affective Computing (2020) 4 | % Email: siyang.song@nottingham.ac.uk 5 | 6 | function out_data = preprocess( raw_data ) 7 | 8 | [num, length] = size(raw_data); 9 | 10 | out_data = zeros(num, length); 11 | 12 | for i = 1:num 13 | 14 | out_data(i,:) = raw_data(i,:) - median(raw_data(i,:)); 15 | 16 | end 17 | 18 | 19 | 20 | end 21 | 22 | -------------------------------------------------------------------------------- /task_time_stamp_avec2013.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/SSYSteve/Human-behaviour-based-depression-analysis-using-hand-crafted-statistics-and-deep-learned/d585b4d12c40eba28430eea12d33c4cc1898973e/task_time_stamp_avec2013.zip --------------------------------------------------------------------------------