├── README.md ├── initial_shape ├── InitialShape_29.mat ├── InitialShape_68.mat └── calc_meanshape.m ├── models └── config_te_best.txt └── src ├── RF2ArrayModel.m ├── config_te.m ├── config_tr.m ├── derivebinaryfeat.m ├── drawshapes.m ├── flipshape.m ├── getbbox.m ├── getproposals.m ├── globalprediction.m ├── globalregression.m ├── loadsamples.m ├── resetshape.m ├── rotatepoints.m ├── rotateshape.m ├── samplerandfeat.m ├── scaleshape.m ├── test_model.m ├── train_model.m ├── train_randomfs.m ├── translateshape.m └── write_binary_model.m /README.md: -------------------------------------------------------------------------------- 1 | Face alignment in 3000 FPS 2 | ========================== 3 | 4 | ##Introduction 5 | 6 | This project is aimed to reproducing (partially) the face alignment algorithm in the CVPR 2014 paper: 7 | 8 | Face Alignment at 3000 FPS via Regressing Local Binary Features. Shaoqing Ren, Xudong Cao, Yichen Wei, Jian Sun; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1685-1692 9 | 10 | ##How to run the codes? 11 | 12 | * First of all, we need prepare datasets, such as afw, lfpw, helen, ibug, etc. All these can be downloaded freely from https://ibug.doc.ic.ac.uk/resources/facial-point-annotations. Then get the filelist file Path_Images.txt for each dataset (please refer to the Q&A). 13 | 14 | * For training, initialize variable dbnames as {'Dataset_a', 'Dataset_b', ..., }, then run train_model in command line window. 15 | 16 | * For testing, run test_model in command line window after having obtained trained model. Please remember to initialize dbnames to be the names of dataset you would like to test on. 17 | 18 | ##Dependencies 19 | 20 | * liblinear: http://www.csie.ntu.edu.tw/~cjlin/liblinear/. 21 | 22 | ##Learned Model 23 | 24 | Off-the-shelf model can be downloaded here: http://pan.baidu.com/s/1i325Rbn, whose configure file can be found in folder "models". 25 | Its performance is analogy to the lbf_fast model evaluated in the original paper. 26 | 27 | ##Q&A 28 | 29 | * How to get the file Path_Images.txt? 30 | 31 | It can be obtained by run bat file in the root folder of a dataset, the code is simply "dir /b/s/p/w *.jpg>Path_Images.txt". 32 | 33 | * What is Ts_bbox.mat? 34 | 35 | This problem is solved in recent version. Ts_bbox is a transformation matrix to adapt bounding boxes obtained from face detector to the boxes suitable for the face alignment algorithm. 36 | 37 | * How to define the variable dbnames in train_model and test_model functions? 38 | 39 | It is formed as a cell array {'dbname_1' 'dbname_2' ... 'dbname_N'}. For example, if we use the images in afw for trainig, we then define it as {'afw'}. 40 | 41 | * Why does an error occur when initializing parallel computing? 42 | 43 | It may be caused by Matlab version. For Matlab 2014, it will be okay. For earlier version, please use matlabpool alternatively. 44 | 45 | * Some function correspondences from Matlab 2014 to older version 46 | 47 | fitgeotrans -> cp2tform, transformPointsForward -> tformfwd 48 | 49 | ##Discussion 50 | 51 | For those Tencent QQ users, you can join the group face hacker (180634020) to discuss more on facial algorithms. Along with the request, it is needed to provide your affiliation (university/company where you are studying/working). 52 | -------------------------------------------------------------------------------- /initial_shape/InitialShape_29.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwyang/face-alignment/104fc3cec4ee7786c797ed6bca13ed6d88cbda5f/initial_shape/InitialShape_29.mat -------------------------------------------------------------------------------- /initial_shape/InitialShape_68.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwyang/face-alignment/104fc3cec4ee7786c797ed6bca13ed6d88cbda5f/initial_shape/InitialShape_68.mat -------------------------------------------------------------------------------- /initial_shape/calc_meanshape.m: -------------------------------------------------------------------------------- 1 | function mean_shape = calc_meanshape(shapepathlistfile) 2 | 3 | fid = fopen(shapepathlistfile); 4 | shapepathlist = textscan(fid, '%s', 'delimiter', '\n'); 5 | 6 | if isempty(shapepathlist) 7 | error('no shape file found'); 8 | mean_shape = []; 9 | return; 10 | end 11 | 12 | shape_header = loadshape(shapepathlist{1}{1}); 13 | 14 | if isempty(shape_header) 15 | error('invalid shape file'); 16 | mean_shape = []; 17 | return; 18 | end 19 | 20 | mean_shape = zeros(size(shape_header)); 21 | 22 | num_shapes = 0; 23 | for i = 1:length(shapepathlist{1}) 24 | shape_i = double(loadshape(shapepathlist{1}{i})); 25 | if isempty(shape_i) 26 | continue; 27 | end 28 | shape_min = min(shape_i, [], 1); 29 | shape_max = max(shape_i, [], 1); 30 | 31 | % translate to origin point 32 | shape_i = bsxfun(@minus, shape_i, shape_min); 33 | 34 | % resize shape 35 | shape_i = bsxfun(@rdivide, shape_i, shape_max - shape_min); 36 | 37 | mean_shape = mean_shape + shape_i; 38 | num_shapes = num_shapes + 1; 39 | end 40 | 41 | mean_shape = mean_shape ./ num_shapes; 42 | 43 | 44 | img = 255 * ones(500, 500, 3); 45 | 46 | drawshapes(img, 50 + 400 * mean_shape); 47 | 48 | end 49 | 50 | function shape = loadshape(path) 51 | % function: load shape from pts file 52 | file = fopen(path); 53 | if file == -1 54 | shape = []; 55 | fclose(file); 56 | return; 57 | end 58 | shape = textscan(file, '%d16 %d16', 'HeaderLines', 3, 'CollectOutput', 2); 59 | fclose(file); 60 | shape = shape{1}; 61 | end -------------------------------------------------------------------------------- /models/config_te_best.txt: -------------------------------------------------------------------------------- 1 | global params; 2 | 3 | params.isparallel = true; 4 | 5 | % initiliaze parameters for shape 6 | params.augnumber_shift = 0; % the number of initial shapes in augmenting training set by shifting 7 | params.augnumber_rotate = 0; % the number of initial shapes in augmenting training set by rotation 8 | params.augnumber_scale = 0; % the number of initial shapes in augmenting training set by rotation 9 | params.flipflag = 0; % the flag of flipping 10 | params.ind_usedpts = 1:68; % [1:17, 28, 31, 32, 34, 36, 37, 40, 43, 46, 49, 52, 55, 58]; 11 | params.augnumber = (params.augnumber_shift + 1)*(params.augnumber_rotate + 1)*((params.augnumber_scale + 1)); 12 | 13 | % initialize parameters for training random forest 14 | params.max_numfeats = [1000 1000 1000 500 500 500 400 400]; 15 | params.bagging_overlap = 0.4; 16 | params.max_raio_radius = [0.4 0.3 0.2 0.15 0.12 0.10 0.08 0.06 0.06 0.05]; 17 | % params.max_raio_radius = [0.3 0.3 0.2 0.2 0.15 0.1 0.1 0.08 0.06 0.05]; 18 | params.max_numtrees = 10; 19 | params.max_depth = 5; 20 | 21 | params.max_numthreshs = 500; 22 | 23 | % initialize parameters for boosting gradient regression 24 | params.max_numstage = 7; 25 | -------------------------------------------------------------------------------- /src/RF2ArrayModel.m: -------------------------------------------------------------------------------- 1 | function RF2ArrayModel(LBFRegModel) 2 | if 0 3 | model_path = '..\Models\LBFRegModel_afw_lfpw_helen_5.mat'; 4 | load(model_path) 5 | end 6 | 7 | params.max_raio_radius = [0.3 0.3 0.2 0.15 0.1 0.1 0.08 0.08 0.06 0.05]; 8 | M = LBFRegModel; 9 | num_stage = size(M.ranf,1); 10 | num_point = min( size(M.Ws{1}) )/2; 11 | num_tree_per_point = size(M.ranf{1},2); 12 | tree_depth = max( M.ranf{1,1}{1}.depth ) - 1; 13 | node_step = 5; 14 | 15 | num_node = 2^tree_depth-1; 16 | num_leaf = 2^tree_depth; 17 | dim_tree = node_step*num_node; 18 | num_tree_per_stage = num_point*num_tree_per_point; 19 | num_tree_total = num_stage*num_point*num_tree_per_point; 20 | dim_feat = num_leaf*num_tree_per_stage; 21 | 22 | %% header 23 | precision_byte = 4; 24 | header_length = 40; 25 | Header = zeros(1,header_length); 26 | % store info 27 | Header(1) = header_length; 28 | Header(2) = precision_byte; %element_byte bytes 29 | Header(3) = num_point*2; %mean shape element 30 | Header(4) = num_stage*num_point*num_tree_per_point*dim_tree; %RF element 31 | Header(5) = num_tree_total*num_leaf*num_point*2; %W element 32 | 33 | %mean shape info 34 | Header(11) = num_point; 35 | 36 | % RF info 37 | Header(21) = num_stage; 38 | Header(22) = num_point; 39 | Header(23) = num_tree_per_point; 40 | Header(24) = tree_depth; 41 | Header(25) = node_step; 42 | 43 | % W info 44 | Header(31) = num_stage; 45 | Header(32) = dim_feat; 46 | Header(33) = num_point*2; 47 | 48 | 49 | RF = zeros(num_tree_total,dim_tree); 50 | for stage=1:num_stage 51 | fprintf('stage=%d\n',stage); 52 | for p=1:num_point 53 | for t=1:num_tree_per_point 54 | Tree = M.ranf{stage}{p, t}; 55 | assert( Tree.num_leafnodes==2^tree_depth ); 56 | anglepairs = Tree.feat(:,1:2); 57 | radiuspairs = Tree.feat(:,3:4); 58 | ax = cos(anglepairs(:, 1)).*radiuspairs(:, 1)*params.max_raio_radius(stage); 59 | ay = sin(anglepairs(:, 1)).*radiuspairs(:, 1)*params.max_raio_radius(stage); 60 | bx = cos(anglepairs(:, 2)).*radiuspairs(:, 2)*params.max_raio_radius(stage); 61 | by = sin(anglepairs(:, 2)).*radiuspairs(:, 2)*params.max_raio_radius(stage); 62 | ax = ax(1:Tree.num_leafnodes-1); 63 | ay = ay(1:Tree.num_leafnodes-1); 64 | bx = bx(1:Tree.num_leafnodes-1); 65 | by = by(1:Tree.num_leafnodes-1); 66 | th = Tree.thresh(1:Tree.num_leafnodes-1); 67 | temp = [ax ay bx by th]; 68 | temp = reshape(temp',1,numel(temp)); 69 | k = (stage-1)*num_tree_per_stage + (p-1)*num_tree_per_point + t; 70 | RF(k,:) = temp; 71 | end 72 | end 73 | end 74 | 75 | W = zeros(dim_feat*num_stage,num_point*2); 76 | for stage=1:num_stage 77 | temp = M.Ws{stage}; 78 | W(dim_feat*(stage-1)+1:dim_feat*stage,:) = temp; 79 | end 80 | 81 | Header = single(Header); 82 | if precision_byte==4 83 | RF = single(RF); 84 | W = single(W); 85 | elseif precision_byte==8 86 | RF = double(RF); 87 | W = double(W); 88 | end 89 | save('..\Models\Header.mat','Header'); 90 | save('..\Models\RF.mat','RF'); 91 | save('..\Models\W.mat','W'); 92 | fprintf('write success.\n') 93 | end 94 | 95 | % pixel_a_x_imgcoord = cos(anglepairs(:, 1)).*radiuspairs(:, 1)*params.max_raio_radius(stage)*Tr_Data{s}.bbox.width; 96 | % pixel_a_y_imgcoord = sin(anglepairs(:, 1)).*radiuspairs(:, 1)*params.max_raio_radius(stage)*Tr_Data{s}.bbox.height; 97 | % 98 | % pixel_b_x_imgcoord = cos(anglepairs(:, 2)).*radiuspairs(:, 2)*params.max_raio_radius(stage)*Tr_Data{s}.bbox.width; 99 | % pixel_b_y_imgcoord = sin(anglepairs(:, 2)).*radiuspairs(:, 2)*params.max_raio_radius(stage)*Tr_Data{s}.bbox.height; 100 | 101 | -------------------------------------------------------------------------------- /src/config_te.m: -------------------------------------------------------------------------------- 1 | global params; 2 | 3 | params.isparallel = true; 4 | 5 | % initiliaze parameters for shape 6 | params.augnumber_shift = 0; % the number of initial shapes in augmenting training set by shifting 7 | params.augnumber_rotate = 0; % the number of initial shapes in augmenting training set by rotation 8 | params.augnumber_scale = 0; % the number of initial shapes in augmenting training set by rotation 9 | params.flipflag = 0; % the flag of flipping 10 | params.ind_usedpts = 1:68; % [1:17, 28, 31, 32, 34, 36, 37, 40, 43, 46, 49, 52, 55, 58]; 11 | params.augnumber = (params.augnumber_shift + 1)*(params.augnumber_rotate + 1)*((params.augnumber_scale + 1)); 12 | 13 | % initialize parameters for training random forest 14 | params.max_numfeats = [1000 1000 1000 500 500 500 400 400]; 15 | params.bagging_overlap = 0.4; 16 | params.max_raio_radius = [0.4 0.3 0.2 0.15 0.12 0.10 0.08 0.06 0.06 0.05]; 17 | params.max_numtrees = 5; 18 | params.max_depth = 4; 19 | 20 | params.max_numthreshs = 500; 21 | 22 | % initialize parameters for boosting gradient regression 23 | params.max_numstage = 4; 24 | -------------------------------------------------------------------------------- /src/config_tr.m: -------------------------------------------------------------------------------- 1 | global params; 2 | 3 | params.isparallel = true; 4 | 5 | % initiliaze parameters for shape 6 | params.augnumber_shift = 0; % the number of initial shapes in augmenting training set by shifting 7 | params.augnumber_rotate = 0; % the number of initial shapes in augmenting training set by rotation 8 | params.augnumber_scale = 0; % the number of initial shapes in augmenting training set by rotation 9 | params.flipflag = 1; % the flag of flipping 10 | params.ind_usedpts = 1:68; % [1:17, 28, 31, 32, 34, 36, 37, 40, 43, 46, 49, 52, 55, 58]; 11 | params.augnumber = (params.augnumber_shift + 1)*(params.augnumber_rotate + 1)*((params.augnumber_scale + 1)); 12 | 13 | % initialize parameters for training random forest 14 | params.max_numfeats = [1000 1000 1000 500 500 500 400 400]; 15 | params.bagging_overlap = 0.4; 16 | params.max_raio_radius = [0.4 0.3 0.2 0.15 0.12 0.10 0.08 0.06 0.06 0.05]; 17 | params.max_numtrees = 5; 18 | params.max_depth = 4; 19 | 20 | params.max_numthreshs = 500; 21 | 22 | % initialize parameters for boosting gradient regression 23 | params.max_numstage = 4; 24 | -------------------------------------------------------------------------------- /src/derivebinaryfeat.m: -------------------------------------------------------------------------------- 1 | function binfeatures = derivebinaryfeat(randf, Tr_Data, params, stage) 2 | %DERIVEBINARYFEAT Summary of this function goes here 3 | % Function: Derive binary features for each sample given learned random forest 4 | % Detailed explanation goes here 5 | % Input: 6 | % lmarkID: the ID of landmark 7 | % randf: learned random forest in current stage 8 | % Tr_Data: training data 9 | % params: parameters for curent stage 10 | 11 | % calculate the overall dimension of binary feature, concatenate the 12 | % features for all landmarks, and the feature of one landmark is sum 13 | % leafnodes of all random trees; 14 | 15 | dims_binfeat = 0; 16 | 17 | ind_bincode = zeros(size(randf)); 18 | 19 | for l = 1:size(randf, 1) 20 | for t = 1:size(randf, 2) 21 | ind_bincode(l, t) = randf{l, t}.num_leafnodes; 22 | dims_binfeat = dims_binfeat + randf{l, t}.num_leafnodes; 23 | end 24 | end 25 | 26 | % initilaize the memory for binfeatures 27 | dbsize = length(Tr_Data); 28 | 29 | % faster implementation 30 | %{ 31 | tic; 32 | binfeatures = zeros(dbsize*(params.augnumber), dims_binfeat); 33 | img_sizes = zeros(2, dbsize*(params.augnumber)); 34 | shapes = zeros([size(params.meanshape), dbsize*(params.augnumber)]); 35 | tfs2meanshape = cell(1, dbsize*params.augnumber); 36 | 37 | img_areas = zeros(1, dbsize); 38 | parfor i = 1:dbsize 39 | img_areas(i) = Tr_Data{i}.width*Tr_Data{i}.height; 40 | end 41 | 42 | imgs_gray = zeros(dbsize, max(img_areas)); 43 | 44 | for i = 1:dbsize 45 | area = img_areas(i); 46 | imgs_gray(i, 1:area) = Tr_Data{i}.img_gray(:)'; 47 | shapes(:, :, (i-1)*params.augnumber+1:i*params.augnumber) = Tr_Data{i}.intermediate_shapes{stage}; 48 | 49 | for k = 1:params.augnumber 50 | img_sizes(:, (i-1)*params.augnumber+k) = [Tr_Data{i}.width; Tr_Data{i}.height]; 51 | tfs2meanshape{(i-1)*params.augnumber+k} = Tr_Data{i}.tf2meanshape{k}; 52 | end 53 | end 54 | 55 | binfeature_lmarks = cell(1, size(params.meanshape, 1)); 56 | num_leafnodes = zeros(1, size(params.meanshape, 1)); 57 | for l = 1:size(params.meanshape, 1) 58 | [binfeature_lmarks{l}, num_leafnodes(l)] = lbf_faster(randf{l}, imgs_gray, img_sizes, shapes(l, :, :), tfs2meanshape, params, stage); 59 | end 60 | 61 | cumnum_leafnodes = [0 cumsum(num_leafnodes)]; 62 | binfeatures_faster = binfeatures; 63 | for l = 1:size(params.meanshape, 1) 64 | binfeatures_faster(:, cumnum_leafnodes(l)+1:cumnum_leafnodes(l+1)) = binfeature_lmarks{l}; 65 | end 66 | 67 | toc; 68 | %} 69 | feats = cell(size(params.meanshape, 1), 1); 70 | isleaf = cell(size(params.meanshape, 1), 1); 71 | threshs = cell(size(params.meanshape, 1), 1); 72 | cnodes = cell(size(params.meanshape, 1), 1); 73 | 74 | % prepare for the derivation of local binary codes 75 | for l = 1:size(params.meanshape, 1) 76 | % concatenate all nodes of all random trees 77 | rf = randf(l, :); 78 | num_rfnodes = 0; 79 | for t = 1:params.max_numtrees 80 | num_rfnodes = num_rfnodes + rf{t}.num_nodes; 81 | end 82 | 83 | % fast implementation 84 | feats{l} = zeros(num_rfnodes, 4); 85 | isleaf{l} = zeros(num_rfnodes, 1); 86 | threshs{l} = zeros(num_rfnodes, 1); 87 | cnodes{l} = zeros(num_rfnodes, 2); 88 | 89 | id_rfnode = 1; 90 | 91 | for t = 1:params.max_numtrees 92 | feats{l}(id_rfnode:(id_rfnode + rf{t}.num_nodes - 1), :) = rf{t}.feat(1:rf{t}.num_nodes, :); 93 | isleaf{l}(id_rfnode:(id_rfnode + rf{t}.num_nodes - 1), :) = rf{t}.isleafnode(1:rf{t}.num_nodes, :); 94 | threshs{l}(id_rfnode:(id_rfnode + rf{t}.num_nodes - 1), :) = rf{t}.thresh(1:rf{t}.num_nodes, :); 95 | cnodes{l}(id_rfnode:(id_rfnode + rf{t}.num_nodes - 1), :) = rf{t}.cnodes(1:rf{t}.num_nodes, :); 96 | 97 | id_rfnode = id_rfnode + rf{t}.num_nodes; 98 | end 99 | end 100 | 101 | 102 | % extract feature for each samples 103 | parfor i = 1:dbsize*(params.augnumber) 104 | k = floor((i-1)/(params.augnumber)) + 1; 105 | s = mod(i-1, (params.augnumber)) + 1; 106 | img_gray = Tr_Data{k}.img_gray; 107 | bbox = Tr_Data{k}.intermediate_bboxes{stage}(s, :); 108 | shapes = Tr_Data{k}.intermediate_shapes{stage}; 109 | shape = shapes(:, :, s); 110 | 111 | % tf2meanshape = Tr_Data{k}.tf2meanshape{s}; 112 | meanshape2tf = Tr_Data{k}.meanshape2tf{s}; 113 | %{ 114 | feats_cell = cell(size(params.meanshape, 1), params.max_numtrees); 115 | 116 | for l = 1:size(params.meanshape, 1) 117 | % extract feature for each landmark from the random forest 118 | for t = 1:params.max_numtrees 119 | % extract feature for each ladnmark from a random tree 120 | bincode = traversetree(randf{l}{t}, img_gray, bbox, shape(l, :), tf2meanshape, params, stage); 121 | feats_cell{l, t} = bincode; 122 | end 123 | end 124 | 125 | binfeature = zeros(1, dims_binfeat); 126 | for l = 1:size(params.meanshape, 1) 127 | % extract feature for each landmark from the random forest 128 | offset = sum(ind_bincodes(1:l-1, end)); 129 | for t = 1:params.max_numtrees 130 | ind_s = ind_bincodes(l, t) + 1 + offset; 131 | ind_e = ind_bincodes(l, t+1) + offset; 132 | % extract feature for each ladnmark from a random tree 133 | binfeature(ind_s:ind_e) = feats_cell{l, t}; 134 | end 135 | end 136 | %} 137 | % fast implementation 138 | 139 | binfeature_lmarks = cell(1, size(params.meanshape, 1)); 140 | num_leafnodes = zeros(1, size(params.meanshape, 1)); 141 | for l = 1:size(params.meanshape, 1) 142 | % concatenate all nodes of all random trees 143 | [binfeature_lmarks{l}, num_leafnodes(l)]= lbf_fast(randf(l, :), feats{l}, isleaf{l}, threshs{l}, cnodes{l}, img_gray, bbox, shape(l, :), meanshape2tf, params, stage); 144 | end 145 | 146 | cumnum_leafnodes = [0 cumsum(num_leafnodes)]; 147 | 148 | binfeature_alllmarks = zeros(1, cumnum_leafnodes(end)); 149 | for l = 1:size(params.meanshape, 1) 150 | binfeature_alllmarks(cumnum_leafnodes(l)+1:cumnum_leafnodes(l+1)) = binfeature_lmarks{l}; 151 | end 152 | 153 | 154 | %{ 155 | % faster implementation (thinking...) 156 | binfeature_lmarks = cell(1, size(params.meanshape, 1)); 157 | num_leafnodes = zeros(1, size(params.meanshape, 1)); 158 | for l = 1:size(params.meanshape, 1) 159 | % concatenate all nodes of all random trees 160 | [binfeature_lmarks{l}, num_leafnodes(l)]= lbf_faster(randf{l}, img_gray, bbox, shape(l, :), tf2meanshape, params, stage); 161 | end 162 | 163 | cumnum_leafnodes = [0 cumsum(num_leafnodes)]; 164 | 165 | binfeature_alllmarks = zeros(1, cumnum_leafnodes(end)); 166 | for l = 1:size(params.meanshape, 1) 167 | binfeature_alllmarks(cumnum_leafnodes(l)+1:cumnum_leafnodes(l+1)) = binfeature_lmarks{l}; 168 | end 169 | %} 170 | 171 | binfeatures(i, :) = binfeature_alllmarks; 172 | end 173 | 174 | 175 | end 176 | 177 | function bincode = traversetree(tree, img_gray, bbox, shape, tf2meanshape, params, stage) 178 | 179 | currnode = tree.rtnodes{1}; 180 | 181 | bincode = zeros(1, tree.num_leafnodes); 182 | 183 | width = size(img_gray, 2); 184 | height = size(img_gray, 1); 185 | 186 | while(1) 187 | 188 | anglepairs = currnode.feat(1:2); 189 | radiuspairs = currnode.feat(3:4); 190 | 191 | % calculate the relative location under the coordinate of meanshape 192 | pixel_a_x_imgcoord = cos(anglepairs(:, 1)).*radiuspairs(:, 1)*params.max_raio_radius(stage)*bbox(3); 193 | pixel_a_y_imgcoord = sin(anglepairs(:, 1)).*radiuspairs(:, 1)*params.max_raio_radius(stage)*bbox(4); 194 | 195 | pixel_b_x_imgcoord = cos(anglepairs(:, 2)).*radiuspairs(:, 2)*params.max_raio_radius(stage)*bbox(3); 196 | pixel_b_y_imgcoord = sin(anglepairs(:, 2)).*radiuspairs(:, 2)*params.max_raio_radius(stage)*bbox(4); 197 | 198 | % transform the pixels from image coordinate (meanshape) to coordinate of current shape 199 | [pixel_a_x_lmcoord, pixel_a_y_lmcoord] = tforminv(tf2meanshape, pixel_a_x_imgcoord, pixel_a_y_imgcoord); 200 | 201 | [pixel_b_x_lmcoord, pixel_b_y_lmcoord] = tforminv(tf2meanshape, pixel_b_x_imgcoord, pixel_b_y_imgcoord); 202 | 203 | pixel_a_x = ceil(pixel_a_x_lmcoord + shape(1)); 204 | pixel_a_y = ceil(pixel_a_y_lmcoord + shape(2)); 205 | 206 | pixel_b_x = ceil(pixel_b_x_lmcoord + shape(1)); 207 | pixel_b_y = ceil(pixel_b_y_lmcoord + shape(2)); 208 | 209 | pixel_a_x = max(1, min(pixel_a_x, width)); 210 | pixel_a_y = max(1, min(pixel_a_y, height)); 211 | 212 | pixel_b_x = max(1, min(pixel_b_x, width)); 213 | pixel_b_y = max(1, min(pixel_b_y, height)); 214 | 215 | f = int16(img_gray(pixel_a_y + (pixel_a_x-1)*height)) - int16(img_gray(pixel_b_y + (pixel_b_x-1)*height)); 216 | 217 | if f < (currnode.thresh) 218 | id_chilenode = currnode.cnodes(1); 219 | currnode = tree.rtnodes{id_chilenode}; 220 | else 221 | id_chilenode = currnode.cnodes(2); 222 | currnode = tree.rtnodes{id_chilenode}; 223 | end 224 | 225 | if isempty(currnode.cnodes) 226 | % if the current node is a leaf node, then stop traversin 227 | bincode(tree.id_leafnodes == id_chilenode) = 1; 228 | break; 229 | end 230 | end 231 | 232 | end 233 | 234 | function [binfeature, num_leafnodes] = lbf_fast(rf, feats, isleaf, threshs, cnodes, img_gray, bbox, shape, meanshape2tf, params, stage) 235 | 236 | %{ 237 | num_rfnodes = 0; 238 | for t = 1:params.max_numtrees 239 | num_rfnodes = num_rfnodes + rf{t}.num_nodes; 240 | end 241 | 242 | % fast implementation 243 | feats = zeros(num_rfnodes, 4); 244 | isleaf = zeros(num_rfnodes, 1); 245 | threshs = zeros(num_rfnodes, 1); 246 | cnodes = zeros(num_rfnodes, 2); 247 | 248 | id_rfnode = 1; 249 | 250 | for t = 1:params.max_numtrees 251 | feats(id_rfnode:(id_rfnode + rf{t}.num_nodes - 1), :) = rf{t}.feat(1:rf{t}.num_nodes, :); 252 | isleaf(id_rfnode:(id_rfnode + rf{t}.num_nodes - 1), :) = rf{t}.isleafnode(1:rf{t}.num_nodes, :); 253 | threshs(id_rfnode:(id_rfnode + rf{t}.num_nodes - 1), :) = rf{t}.thresh(1:rf{t}.num_nodes, :); 254 | cnodes(id_rfnode:(id_rfnode + rf{t}.num_nodes - 1), :) = rf{t}.cnodes(1:rf{t}.num_nodes, :); 255 | 256 | id_rfnode = id_rfnode + rf{t}.num_nodes; 257 | end 258 | %} 259 | 260 | width = size(img_gray, 2); 261 | height = size(img_gray, 1); 262 | 263 | anglepairs = feats(:, 1:2); 264 | radiuspairs = feats(:, 3:4); 265 | 266 | % calculate the relative location under the coordinate of meanshape 267 | pixel_a_x_imgcoord = cos(anglepairs(:, 1)).*radiuspairs(:, 1)*params.max_raio_radius(stage)*bbox(3); 268 | pixel_a_y_imgcoord = sin(anglepairs(:, 1)).*radiuspairs(:, 1)*params.max_raio_radius(stage)*bbox(4); 269 | 270 | pixel_b_x_imgcoord = cos(anglepairs(:, 2)).*radiuspairs(:, 2)*params.max_raio_radius(stage)*bbox(3); 271 | pixel_b_y_imgcoord = sin(anglepairs(:, 2)).*radiuspairs(:, 2)*params.max_raio_radius(stage)*bbox(4); 272 | 273 | % no transformaiton 274 | %{ 275 | pixel_a_x_lmcoord = pixel_a_x_imgcoord; 276 | pixel_a_y_lmcoord = pixel_a_y_imgcoord; 277 | 278 | pixel_b_x_lmcoord = pixel_b_x_imgcoord; 279 | pixel_b_y_lmcoord = pixel_b_y_imgcoord; 280 | %} 281 | 282 | % transform the pixels from image coordinate (meanshape) to coordinate of current shape 283 | 284 | [pixel_a_x_lmcoord, pixel_a_y_lmcoord] = transformPointsForward(meanshape2tf, pixel_a_x_imgcoord', pixel_a_y_imgcoord'); 285 | pixel_a_x_lmcoord = pixel_a_x_lmcoord'; 286 | pixel_a_y_lmcoord = pixel_a_y_lmcoord'; 287 | 288 | [pixel_b_x_lmcoord, pixel_b_y_lmcoord] = transformPointsForward(meanshape2tf, pixel_b_x_imgcoord', pixel_b_y_imgcoord'); 289 | pixel_b_x_lmcoord = pixel_b_x_lmcoord'; 290 | pixel_b_y_lmcoord = pixel_b_y_lmcoord'; 291 | 292 | pixel_a_x = ceil(pixel_a_x_lmcoord + shape(1)); 293 | pixel_a_y = ceil(pixel_a_y_lmcoord + shape(2)); 294 | 295 | pixel_b_x = ceil(pixel_b_x_lmcoord + shape(1)); 296 | pixel_b_y = ceil(pixel_b_y_lmcoord + shape(2)); 297 | 298 | pixel_a_x = max(1, min(pixel_a_x, width)); 299 | pixel_a_y = max(1, min(pixel_a_y, height)); 300 | 301 | pixel_b_x = max(1, min(pixel_b_x, width)); 302 | pixel_b_y = max(1, min(pixel_b_y, height)); 303 | 304 | pdfeats = double(img_gray(pixel_a_y + (pixel_a_x-1)*height)) - double(img_gray(pixel_b_y + (pixel_b_x-1)*height)); 305 | % ./ double(img_gray(pixel_a_y + (pixel_a_x-1)*height)) + double(img_gray(pixel_b_y + (pixel_b_x-1)*height)); 306 | 307 | cind = (pdfeats >= threshs) + 1; 308 | 309 | % obtain the indice of child nodes for all nodes, if the current node is 310 | % leaf node, then the indice of its child node is 0 311 | ind_cnodes = cnodes; % diag(cnodes(:, cind)); 312 | 313 | binfeature = zeros(1, sum(isleaf)); 314 | 315 | cumnum_nodes = 0; 316 | cumnum_leafnodes = 0; 317 | for t = 1:params.max_numtrees 318 | num_nodes = (rf{t}.num_nodes); 319 | id_cnode = 1; 320 | while(1) 321 | if isleaf(id_cnode + cumnum_nodes) 322 | binfeature(cumnum_leafnodes + find(rf{t}.id_leafnodes == id_cnode)) = 1; 323 | cumnum_nodes = cumnum_nodes + num_nodes; 324 | cumnum_leafnodes = cumnum_leafnodes + rf{t}.num_leafnodes; 325 | break; 326 | end 327 | id_cnode = ind_cnodes(cumnum_nodes + id_cnode, cind(cumnum_nodes + id_cnode)); 328 | end 329 | end 330 | 331 | num_leafnodes = sum(isleaf); 332 | 333 | end 334 | -------------------------------------------------------------------------------- /src/drawshapes.m: -------------------------------------------------------------------------------- 1 | function drawshapes(img, shapes) 2 | %DRAWSHAPE Summary of this function goes here 3 | % Function: draw face landmarks on img 4 | % Detailed explanation goes here 5 | % Input: 6 | % img: input image 7 | % shapes: given face shapes 8 | 9 | imshow(img); 10 | hold on; 11 | 12 | colors = {'r.' 'g.' 'b.' 'c.' 'm.' 'y.' 'k.'}; 13 | 14 | for s = 1:size(shapes, 2)/2 15 | for is = 1:size(shapes, 1) 16 | plot(shapes(is, 2*(s-1) + 1), shapes(is, 2*s), colors{s}, 'LineWidth', 1); 17 | end 18 | end 19 | 20 | end 21 | 22 | -------------------------------------------------------------------------------- /src/flipshape.m: -------------------------------------------------------------------------------- 1 | function shape_flipped = flipshape(shape) 2 | %FLIPSHAPE Summary of this function goes here 3 | % Function: flip input shape horizonically 4 | % Detailed explanation goes here 5 | 6 | if size(shape, 1) == 68 7 | shape_flipped = shape; 8 | % flip check 9 | shape_flipped(1:17, :) = shape(17:-1:1, :); 10 | % flip eyebows 11 | shape_flipped(18:27, :) = shape(27:-1:18, :); 12 | % flip eyes 13 | shape_flipped(32:36, :) = shape(36:-1:32, :); 14 | % flip eyes 15 | shape_flipped(37:40, :) = shape(46:-1:43, :); 16 | shape_flipped(41:42, :) = shape(48:-1:47, :); 17 | shape_flipped(43:46, :) = shape(40:-1:37, :); 18 | shape_flipped(47:48, :) = shape(42:-1:41, :); 19 | 20 | % flip mouth 21 | shape_flipped(49:55, :) = shape(55:-1:49, :); 22 | shape_flipped(56:60, :) = shape(60:-1:56, :); 23 | 24 | shape_flipped(61:65, :) = shape(65:-1:61, :); 25 | shape_flipped(66:68, :) = shape(68:-1:66, :); 26 | 27 | else 28 | disp('The Flip Funtion is Error!') 29 | end 30 | 31 | end 32 | 33 | -------------------------------------------------------------------------------- /src/getbbox.m: -------------------------------------------------------------------------------- 1 | function bbox = getbbox(shape) 2 | %GETBBOX Summary of this function goes here 3 | % Function: get the bounding box of given shape 4 | % Detailed explanation goes here 5 | % Input: 6 | % shape: the shape of face 7 | % Output: 8 | % bbox: the bounding box of given face shape 9 | 10 | 11 | bbox = zeros(1, 4); 12 | 13 | left_x = min(shape(:, 1)); 14 | right_x = max(shape(:, 1)); 15 | top_y = min(shape(:, 2)); 16 | bottom_y = max(shape(:, 2)); 17 | 18 | bbox(1) = left_x; 19 | bbox(2) = top_y; 20 | bbox(3) = right_x - left_x + 1; 21 | bbox(4) = bottom_y - top_y + 1; 22 | 23 | end 24 | 25 | -------------------------------------------------------------------------------- /src/getproposals.m: -------------------------------------------------------------------------------- 1 | function [radiuspairs, anglepairs] = getproposals(num_proposals, radius_grid, angles_grid) 2 | %GETPROPOSALS Summary of this function goes here 3 | % Detailed explanation goes here 4 | 5 | num_radius = length(radius_grid); 6 | num_angles = length(angles_grid); 7 | 8 | Pro_a = randperm(num_radius*num_angles); 9 | Pro_b = randperm(num_radius*num_angles); 10 | 11 | exc = Pro_a == Pro_b; 12 | 13 | Pro_a = Pro_a(exc == 0); 14 | Pro_b = Pro_b(exc == 0); 15 | 16 | Pro_a_choose = Pro_a(1:num_proposals); 17 | Pro_b_choose = Pro_b(1:num_proposals); 18 | 19 | id_radius_a = floor((Pro_a_choose - 1)/num_angles) + 1; 20 | id_radius_b = floor((Pro_b_choose - 1)/num_angles) + 1; 21 | 22 | id_angles_a = mod(Pro_a_choose, num_angles) + 1; 23 | id_angles_b = mod(Pro_b_choose, num_angles) + 1; 24 | 25 | radiuspairs = [radius_grid(id_radius_a) radius_grid(id_radius_b)]; 26 | anglepairs = [angles_grid(id_angles_a) angles_grid(id_angles_b)]; 27 | 28 | 29 | end 30 | 31 | -------------------------------------------------------------------------------- /src/globalprediction.m: -------------------------------------------------------------------------------- 1 | function Te_Data = globalprediction(binaryfeatures, W, Te_Data, params, stage) 2 | %GLOBALREGRESSION Summary of this function goes here 3 | % Function: implement global regression given binary features and 4 | % groundtruth shape 5 | % Detailed explanation goes here 6 | % Input: 7 | % binaryfeatures: extracted binary features from all samples (N X d ) 8 | % Te_Data: test data 9 | % params: parameters for model 10 | % Output: 11 | % W: regression matrix 12 | 13 | % organize the groundtruth shape 14 | dbsize = length(Te_Data); 15 | gtshapes = zeros([size(params.meanshape) dbsize*(params.augnumber)]); % concatenate 2-D coordinates into a vector (N X (2*L)) 16 | dist_pupils = zeros(dbsize*(params.augnumber), 1); 17 | 18 | for i = 1:dbsize*(params.augnumber) 19 | k = floor((i-1)/(params.augnumber)) + 1; 20 | s = mod(i-1, (params.augnumber)) + 1; 21 | shape_gt = Te_Data{k}.shape_gt; 22 | gtshapes(:, :, i) = reshape(shape_gt(:), size(params.meanshape)); 23 | 24 | % left eye: 37-42 25 | % right eye: 43-48 26 | if size(shape_gt, 1) == 68 27 | dist_pupils(i) = norm((mean(shape_gt(37:42, :)) - mean(shape_gt(43:48, :)))); 28 | elseif size(shape_gt, 1) == 51 29 | dist_pupils(i) = norm((mean(shape_gt(20, :)) - mean(shape_gt(29, :)))); 30 | elseif size(shape_gt, 1) == 29 31 | dist_pupils(i) = norm((mean(shape_gt(9:2:17, :)) - mean(shape_gt(10:2:18, :)))); 32 | end 33 | 34 | end 35 | 36 | % Predict the location of lanmarks using current regression matrix 37 | deltashapes_bar = binaryfeatures*W; 38 | 39 | predshapes = zeros([size(params.meanshape) dbsize*(params.augnumber)]); % concatenate 2-D coordinates into a vector (N X (2*L)) 40 | 41 | for i = 1:dbsize*(params.augnumber) 42 | k = floor((i-1)/(params.augnumber)) + 1; 43 | s = mod(i-1, (params.augnumber)) + 1; 44 | 45 | shapes_stage = Te_Data{k}.intermediate_shapes{stage}; 46 | shape_stage = shapes_stage(:, :, s); 47 | 48 | deltashapes_bar_xy = reshape(deltashapes_bar(i, :), [uint8(size(deltashapes_bar, 2)/2) 2]); 49 | 50 | % transform above delta shape into the coordinate of current intermmediate shape 51 | % delta_shape_intermmed_coord = deltashapes_bar_xy; 52 | [u, v] = transformPointsForward(Te_Data{k}.meanshape2tf{s}, deltashapes_bar_xy(:, 1)', deltashapes_bar_xy(:, 2)'); 53 | delta_shape_intermmed_coord = [u', v']; 54 | 55 | delta_shape_meanshape_coord = bsxfun(@times, delta_shape_intermmed_coord, Te_Data{k}.intermediate_bboxes{stage}(s, 3:4)); 56 | 57 | 58 | shape_newstage = shape_stage + delta_shape_meanshape_coord; 59 | 60 | predshapes(:, :, i) = reshape(shape_newstage(:), size(params.meanshape)); 61 | 62 | Te_Data{k}.intermediate_shapes{stage+1}(:, :, s) = shape_newstage; 63 | Te_Data{k}.intermediate_bboxes{stage+1}(s, :) = Te_Data{k}.intermediate_bboxes{stage}(s, :); % getbbox(shape_newstage); 64 | 65 | % update transformation of current intermediate shape to meanshape 66 | meanshape_resize = resetshape(Te_Data{k}.intermediate_bboxes{stage+1}(s, :) , params.meanshape); 67 | Te_Data{k}.tf2meanshape{s} = fitgeotrans(bsxfun(@minus, Te_Data{k}.intermediate_shapes{stage+1}(:,:, s), mean(Te_Data{k}.intermediate_shapes{stage+1}(:,:, s))), ... 68 | bsxfun(@minus, meanshape_resize(:, :), mean(meanshape_resize(:, :))), 'nonreflectivesimilarity'); 69 | Te_Data{k}.meanshape2tf{s} = fitgeotrans(bsxfun(@minus, meanshape_resize(:, :), mean(meanshape_resize(:, :))), ... 70 | bsxfun(@minus, Te_Data{k}.intermediate_shapes{stage+1}(:,:, s), mean(Te_Data{k}.intermediate_shapes{stage+1}(:,:, s))), 'nonreflectivesimilarity'); 71 | 72 | %{ 73 | if stage >= params.max_numstage 74 | % [Te_Data{k}.shape_gt Te_Data{k}.intermediate_shapes{1}(:,:, s) shape_newstage] 75 | drawshapes(Te_Data{k}.img_gray, [Te_Data{k}.shape_gt Te_Data{k}.intermediate_shapes{1}(:,:, s) shape_newstage]); 76 | hold off; 77 | drawnow; 78 | error_per_image = compute_error(gtshapes(:, :, i), predshapes(:, :, i)) 79 | w = waitforbuttonpress; 80 | end 81 | %} 82 | end 83 | 84 | error_per_image = compute_error(gtshapes, predshapes); 85 | 86 | MRSE = 100*mean(error_per_image); 87 | MRSE_display = sprintf('Mean Root Square Error for %d Test Samples: %f', (dbsize*(params.augnumber)), MRSE); 88 | disp(MRSE_display); 89 | 90 | end 91 | 92 | function [ error_per_image ] = compute_error( ground_truth_all, detected_points_all ) 93 | %compute_error 94 | % compute the average point-to-point Euclidean error normalized by the 95 | % inter-ocular distance (measured as the Euclidean distance between the 96 | % outer corners of the eyes) 97 | % 98 | % Inputs: 99 | % grounth_truth_all, size: num_of_points x 2 x num_of_images 100 | % detected_points_all, size: num_of_points x 2 x num_of_images 101 | % Output: 102 | % error_per_image, size: num_of_images x 1 103 | 104 | 105 | num_of_images = size(ground_truth_all,3); 106 | num_of_points = size(ground_truth_all,1); 107 | 108 | error_per_image = zeros(num_of_images,1); 109 | 110 | for i =1:num_of_images 111 | detected_points = detected_points_all(:,:,i); 112 | ground_truth_points = ground_truth_all(:,:,i); 113 | if num_of_points == 68 114 | interocular_distance = norm(mean(ground_truth_points(37:42,:))-mean(ground_truth_points(43:48,:))); % norm((mean(shape_gt(37:42, :)) - mean(shape_gt(43:48, :)))); 115 | elseif num_of_points == 51 116 | interocular_distance = norm(ground_truth_points(20,:) - ground_truth_points(29,:)); 117 | elseif num_of_points == 29 118 | interocular_distance = norm(mean(ground_truth_points(9:2:17,:))-mean(ground_truth_points(10:2:18,:))); 119 | else 120 | interocular_distance = norm(mean(ground_truth_points(1:2,:))-mean(ground_truth_points(end - 1:end,:))); 121 | end 122 | 123 | sum=0; 124 | for j=1:num_of_points 125 | sum = sum+norm(detected_points(j,:)-ground_truth_points(j,:)); 126 | end 127 | error_per_image(i) = sum/(num_of_points*interocular_distance); 128 | end 129 | 130 | end 131 | 132 | -------------------------------------------------------------------------------- /src/globalregression.m: -------------------------------------------------------------------------------- 1 | function [W, Tr_Data] = globalregression(binaryfeatures, Tr_Data, params, stage) 2 | %GLOBALREGRESSION Summary of this function goes here 3 | % Function: implement global regression given binary features and 4 | % groundtruth shape 5 | % Detailed explanation goes here 6 | % Input: 7 | % binaryfeatures: extracted binary features from all samples (N X d ) 8 | % Tr_Data: training data 9 | % params: parameters for model 10 | % Output: 11 | % W: regression matrix 12 | 13 | % organize the groundtruth shape 14 | dbsize = length(Tr_Data); 15 | deltashapes = zeros(dbsize*(params.augnumber), 2*size(params.meanshape, 1)); % concatenate 2-D coordinates into a vector (N X (2*L)) 16 | dist_pupils = zeros(dbsize*(params.augnumber), 1); 17 | gtshapes = zeros([size(params.meanshape) dbsize*(params.augnumber)]); % concatenate 2-D coordinates into a vector (N X (2*L)) 18 | 19 | for i = 1:dbsize*(params.augnumber) 20 | k = floor((i-1)/(params.augnumber)) + 1; 21 | s = mod(i-1, (params.augnumber)) + 1; 22 | 23 | shape_gt = Tr_Data{k}.shape_gt; 24 | if size(shape_gt, 1) == 68 25 | dist_pupils(i) = norm((mean(shape_gt(37:42, :)) - mean(shape_gt(43:48, :)))); 26 | elseif size(shape_gt, 1) == 51 27 | dist_pupils(i) = norm((mean(shape_gt(20, :)) - mean(shape_gt(29, :)))); 28 | elseif size(shape_gt, 1) == 29 29 | dist_pupils(i) = norm((mean(shape_gt(9:2:17, :)) - mean(shape_gt(10:2:18, :)))); 30 | else 31 | dist_pupils(i) = norm((mean(shape_gt(1:2, :)) - mean(shape_gt(end - 1:end, :)))); 32 | end 33 | gtshapes(:, :, i) = shape_gt; 34 | delta_shape = Tr_Data{k}.shapes_residual(:, :, s); 35 | deltashapes(i, :) = delta_shape(:)'; 36 | end 37 | 38 | % conduct regression using libliear 39 | % X : binaryfeatures 40 | % Y: gtshapes 41 | 42 | param = sprintf('-s 12 -p 0 -c %f -q heart_scale', 1/(size(binaryfeatures, 1))); 43 | W_liblinear = zeros(size(binaryfeatures, 2), size(deltashapes, 2)); 44 | tic; 45 | parfor o = 1:size(deltashapes, 2) 46 | model = train(deltashapes(:, o), sparse(binaryfeatures), param); 47 | W_liblinear(:, o) = model.w'; 48 | end 49 | toc; 50 | W = W_liblinear; 51 | 52 | % Predict the location of lanmarks using current regression matrix 53 | 54 | deltashapes_bar = binaryfeatures*W; 55 | predshapes = zeros([size(params.meanshape) size(binaryfeatures, 1)]); % concatenate 2-D coordinates into a vector (N X (2*L)) 56 | 57 | for i = 1:dbsize*(params.augnumber) 58 | k = floor((i-1)/(params.augnumber)) + 1; 59 | s = mod(i-1, (params.augnumber)) + 1; 60 | 61 | shapes_stage = Tr_Data{k}.intermediate_shapes{stage}; 62 | shape_stage = shapes_stage(:, :, s); 63 | 64 | deltashapes_bar_xy = reshape(deltashapes_bar(i, :), [uint8(size(deltashapes_bar, 2)/2) 2]); 65 | 66 | % transform above delta shape into the coordinate of current intermmediate shape 67 | % delta_shape_interm_coord = [deltashapes_bar_x(i, :)', deltashapes_bar_y(i, :)']; 68 | [u, v] = transformPointsForward(Tr_Data{k}.meanshape2tf{s}, deltashapes_bar_xy(:, 1)', deltashapes_bar_xy(:, 2)'); 69 | delta_shape_interm_coord = [u', v']; 70 | 71 | % derive the delta shape in the coordinate system of meanshape 72 | delta_shape_interm_coord = bsxfun(@times, delta_shape_interm_coord, Tr_Data{k}.intermediate_bboxes{stage}(s, 3:4)); 73 | 74 | shape_newstage = shape_stage + delta_shape_interm_coord; 75 | predshapes(:, :, i) = shape_newstage; 76 | 77 | Tr_Data{k}.intermediate_shapes{stage+1}(:, :, s) = shape_newstage; 78 | 79 | % update transformation of current intermediate shape to meanshape 80 | Tr_Data{k}.intermediate_bboxes{stage+1}(s, :) = getbbox(Tr_Data{k}.intermediate_shapes{stage+1}(:, :, s)); 81 | 82 | meanshape_resize = (resetshape(Tr_Data{k}.intermediate_bboxes{stage+1}(s, :), params.meanshape)); 83 | 84 | shape_residual = bsxfun(@rdivide, Tr_Data{k}.shape_gt - shape_newstage, Tr_Data{k}.intermediate_bboxes{stage+1}(s, 3:4)); 85 | 86 | Tr_Data{k}.tf2meanshape{s} = fitgeotrans(bsxfun(@minus, Tr_Data{k}.intermediate_shapes{stage+1}(1:end,:, s), mean(Tr_Data{k}.intermediate_shapes{stage+1}(1:end,:, s))), ... 87 | bsxfun(@minus, meanshape_resize(1:end, :), mean(meanshape_resize(1:end, :))), 'NonreflectiveSimilarity'); 88 | 89 | Tr_Data{k}.meanshape2tf{s} = fitgeotrans(bsxfun(@minus, meanshape_resize(1:end, :), mean(meanshape_resize(1:end, :))), ... 90 | bsxfun(@minus, Tr_Data{k}.intermediate_shapes{stage+1}(1:end,:, s), mean(Tr_Data{k}.intermediate_shapes{stage+1}(1:end,:, s))), 'NonreflectiveSimilarity'); 91 | 92 | [u, v] = transformPointsForward(Tr_Data{k}.tf2meanshape{s}, shape_residual(:, 1), shape_residual(:, 2)); 93 | Tr_Data{k}.shapes_residual(:, 1, s) = u'; 94 | Tr_Data{k}.shapes_residual(:, 2, s) = v'; 95 | %{ 96 | drawshapes(Tr_Data{k}.img_gray, [Tr_Data{k}.shape_gt shape_stage shape_newstage]); 97 | hold off; 98 | drawnow; 99 | w = waitforbuttonpress; 100 | %} 101 | end 102 | 103 | error_per_image = compute_error(gtshapes, predshapes); 104 | 105 | MRSE = 100*mean(error_per_image); 106 | MRSE_display = sprintf('Mean Root Square Error for %d Test Samples: %f', length(error_per_image), MRSE); 107 | disp(MRSE_display); 108 | 109 | end 110 | 111 | function [ error_per_image ] = compute_error( ground_truth_all, detected_points_all ) 112 | %compute_error 113 | % compute the average point-to-point Euclidean error normalized by the 114 | % inter-ocular distance (measured as the Euclidean distance between the 115 | % outer corners of the eyes) 116 | % 117 | % Inputs: 118 | % grounth_truth_all, size: num_of_points x 2 x num_of_images 119 | % detected_points_all, size: num_of_points x 2 x num_of_images 120 | % Output: 121 | % error_per_image, size: num_of_images x 1 122 | 123 | 124 | num_of_images = size(ground_truth_all,3); 125 | num_of_points = size(ground_truth_all,1); 126 | 127 | error_per_image = zeros(num_of_images,1); 128 | 129 | for i =1:num_of_images 130 | detected_points = detected_points_all(:,:,i); 131 | ground_truth_points = ground_truth_all(:,:,i); 132 | if num_of_points == 68 133 | interocular_distance = norm(mean(ground_truth_points(37:42,:))-mean(ground_truth_points(43:48,:))); % norm((mean(shape_gt(37:42, :)) - mean(shape_gt(43:48, :)))); 134 | elseif num_of_points == 51 135 | interocular_distance = norm(ground_truth_points(20,:) - ground_truth_points(29,:)); 136 | elseif num_of_points == 29 137 | interocular_distance = norm(mean(ground_truth_points(9:2:17,:))-mean(ground_truth_points(10:2:18,:))); 138 | else 139 | interocular_distance = norm(mean(ground_truth_points(1:2,:))-mean(ground_truth_points(end - 1:end,:))); 140 | end 141 | 142 | sum=0; 143 | for j=1:num_of_points 144 | sum = sum+norm(detected_points(j,:)-ground_truth_points(j,:)); 145 | end 146 | error_per_image(i) = sum/(num_of_points*interocular_distance); 147 | end 148 | 149 | end -------------------------------------------------------------------------------- /src/loadsamples.m: -------------------------------------------------------------------------------- 1 | function Data = loadsamples(imgpathlistfile, exc_setlabel) 2 | %LOADSAMPLES Summary of this function goes here 3 | % Function: load samples from dbname database 4 | % Detailed explanation goes here 5 | % Input: 6 | % dbname: the name of one database 7 | % exc_setlabel: excluded set label 8 | % Output: 9 | % Data: loaded data from the database 10 | imgpathlist = textread(imgpathlistfile, '%s', 'delimiter', '\n'); 11 | 12 | Data = cell(length(imgpathlist), 1); 13 | 14 | setnames = {'train' 'test'}; 15 | 16 | % Create a cascade detector object. 17 | % faceDetector = vision.CascadeObjectDetector(); 18 | % bboxes_facedet = zeros(length(imgpathlist), 4); 19 | % bboxes_gt = zeros(length(imgpathlist), 4); 20 | % isdetected = zeros(length(imgpathlist), 1); 21 | 22 | parfor i = 1:length(imgpathlist) 23 | img = im2uint8(imread(imgpathlist{i})); 24 | Data{i}.width_orig = size(img, 2); 25 | Data{i}.height_orig = size(img, 1); 26 | 27 | % Data{i}.img = img 28 | % shapepath = strrep(imgpathlist{i}, 'png', 'pts'); 29 | shapepath = strcat(imgpathlist{i}(1:end-3), 'pts'); 30 | Data{i}.shape_gt = double(loadshape(shapepath)); 31 | % Data{i}.shape_gt = Data{i}.shape_gt(params.ind_usedpts, :); 32 | % bbox = bounding_boxes_allsamples{i}.bb_detector; % 33 | Data{i}.bbox_gt = getbbox(Data{i}.shape_gt); % [bbox(1) bbox(2) bbox(3)-bbox(1) bbox(4)-bbox(2)]; 34 | 35 | % cut original image to a region which is a bit larger than the face 36 | % bounding box 37 | region = enlargingbbox(Data{i}.bbox_gt, 2.0); 38 | 39 | region(2) = double(max(region(2), 1)); 40 | region(1) = double(max(region(1), 1)); 41 | 42 | bottom_y = double(min(region(2) + region(4) - 1, Data{i}.height_orig)); 43 | right_x = double(min(region(1) + region(3) - 1, Data{i}.width_orig)); 44 | 45 | img_region = img(region(2):bottom_y, region(1):right_x, :); 46 | Data{i}.shape_gt = bsxfun(@minus, Data{i}.shape_gt, double([region(1) region(2)])); 47 | 48 | % to save memory cost during training 49 | if exc_setlabel == 2 50 | ratio = min(1, sqrt(single(150 * 150) / single(size(img_region, 1) * size(img_region, 2)))); 51 | img_region = imresize(img_region, ratio); 52 | Data{i}.shape_gt = Data{i}.shape_gt .* ratio; 53 | end 54 | Data{i}.bbox_gt = getbbox(Data{i}.shape_gt); 55 | 56 | Data{i}.bbox_facedet = getbbox(Data{i}.shape_gt); 57 | % perform face detection using matlab face detector 58 | %{ 59 | bbox = step(faceDetector, img_region); 60 | if isempty(bbox) 61 | % if face detection is failed 62 | isdetected(i) = 1; 63 | Data{i}.bbox_facedet = getbbox(Data{i}.shape_gt); 64 | else 65 | int_ratios = zeros(1, size(bbox, 1)); 66 | for b = 1:size(bbox, 1) 67 | area = rectint(Data{i}.bbox_gt, bbox(b, :)); 68 | int_ratios(b) = (area)/(bbox(b, 3)*bbox(b, 4) + Data{i}.bbox_gt(3)*Data{i}.bbox_gt(4) - area); 69 | end 70 | [max_ratio, max_ind] = max(int_ratios); 71 | 72 | if max_ratio < 0.4 % detection fail 73 | isdetected(i) = 0; 74 | else 75 | Data{i}.bbox_facedet = bbox(max_ind, 1:4); 76 | isdetected(i) = 1; 77 | % imgOut = insertObjectAnnotation(img_region,'rectangle',Data{i}.bbox_facedet,'Face'); 78 | % imshow(imgOut); 79 | end 80 | end 81 | %} 82 | % recalculate the location of groundtruth shape and bounding box 83 | % Data{i}.shape_gt = bsxfun(@minus, Data{i}.shape_gt, double([region(1) region(2)])); 84 | % Data{i}.bbox_gt = getbbox(Data{i}.shape_gt); 85 | 86 | if size(img_region, 3) == 1 87 | Data{i}.img_gray = img_region; 88 | else 89 | % hsv = rgb2hsv(img_region); 90 | Data{i}.img_gray = rgb2gray(img_region); 91 | end 92 | 93 | Data{i}.width = size(img_region, 2); 94 | Data{i}.height = size(img_region, 1); 95 | end 96 | 97 | ind_valid = ones(1, length(imgpathlist)); 98 | parfor i = 1:length(imgpathlist) 99 | if ~isempty(exc_setlabel) 100 | ind = strfind(imgpathlist{i}, setnames{exc_setlabel}); 101 | if ~isempty(ind) % | ~isdetected(i) 102 | ind_valid(i) = 0; 103 | end 104 | end 105 | end 106 | 107 | % learn the linear transformation from detected bboxes to groundtruth bboxes 108 | % bboxes = [bboxes_gt bboxes_facedet]; 109 | % bboxes = bboxes(ind_valid == 1, :); 110 | 111 | Data = Data(ind_valid == 1); 112 | 113 | end 114 | 115 | function shape = loadshape(path) 116 | % function: load shape from pts file 117 | file = fopen(path); 118 | 119 | if ~isempty(strfind(path, 'COFW')) 120 | shape = textscan(file, '%d16 %d16 %d8', 'HeaderLines', 3, 'CollectOutput', 3); 121 | else 122 | shape = textscan(file, '%d16 %d16', 'HeaderLines', 3, 'CollectOutput', 2); 123 | end 124 | fclose(file); 125 | 126 | shape = shape{1}; 127 | end 128 | 129 | function region = enlargingbbox(bbox, scale) 130 | 131 | region(1) = floor(bbox(1) - (scale - 1)/2*bbox(3)); 132 | region(2) = floor(bbox(2) - (scale - 1)/2*bbox(4)); 133 | 134 | region(3) = floor(scale*bbox(3)); 135 | region(4) = floor(scale*bbox(4)); 136 | 137 | % region.right_x = floor(region.left_x + region.width - 1); 138 | % region.bottom_y = floor(region.top_y + region.height - 1); 139 | 140 | 141 | end 142 | 143 | -------------------------------------------------------------------------------- /src/resetshape.m: -------------------------------------------------------------------------------- 1 | function [shape_initial] = resetshape(bbox, shape_union) 2 | %RESETSHAPE Summary of this function goes here 3 | % Function: reset the initial shape according to the groundtruth shape and union shape for all faces 4 | % Detailed explanation goes here 5 | % Input: 6 | % bbox: bbounding box of groundtruth shape 7 | % shape_union: uniionshape 8 | % Output: 9 | % shape_initial: reset initial shape 10 | % bbox: bounding box of face image 11 | 12 | % get the bounding box according to the ground truth shape 13 | width_union = (max(shape_union(:, 1)) - min(shape_union(:, 1))); 14 | height_union = (max(shape_union(:, 2)) - min(shape_union(:, 2))); 15 | 16 | shape_union = bsxfun(@minus, (shape_union), (min(shape_union))); 17 | 18 | shape_initial = bsxfun(@times, shape_union, [(bbox(3)/width_union) (bbox(4)/height_union)]); 19 | shape_initial = bsxfun(@plus, shape_initial, double([bbox(1) bbox(2)])); 20 | 21 | end 22 | 23 | -------------------------------------------------------------------------------- /src/rotatepoints.m: -------------------------------------------------------------------------------- 1 | function rotated_coords = rotatepoints(input_XY,center,anti_clockwise_angle, scale, varargin) 2 | degree = 1; %Radians : degree = 0; Default is calculations in degrees 3 | 4 | % Process the inputs 5 | if length(varargin) ~= 0 6 | for n = 1:1:length(varargin) 7 | if strcmp(varargin{n},'degree') 8 | degree = 1; 9 | elseif strcmp(varargin{n},'radians') 10 | degree = 0; 11 | end 12 | end 13 | clear n; 14 | end 15 | [r,c] = size(input_XY); 16 | if c ~= 2 17 | error('Not enough columns in coordinates XY '); 18 | end 19 | [r,c] = size(center); 20 | if (r~=1 & c==2) | (r==1 & c~=2) 21 | error('Error in the size of the "center" matrix'); 22 | end 23 | 24 | % Format the coordinate of the center of rotation 25 | center_coord = input_XY; 26 | center_coord(:,1) = center(1); 27 | center_coord(:,2) = center(2); 28 | 29 | % Turns the angles given to be such that the +ve is anti-clockwise and -ve is clockwise 30 | anti_clockwise_angle = -1*anti_clockwise_angle; 31 | % if in degrees, convert to radians because that's what the built-in functions use. 32 | if degree == 1 33 | anti_clockwise_angle = deg2rad(anti_clockwise_angle); 34 | end 35 | 36 | %Produce the roation matrix 37 | rotation_matrix = [cos(anti_clockwise_angle),-1*sin(anti_clockwise_angle);... 38 | sin(anti_clockwise_angle),cos(anti_clockwise_angle)]; 39 | %Calculate the final coordinates 40 | rotated_coords = scale*((input_XY-center_coord) * rotation_matrix) + center_coord; 41 | 42 | end 43 | 44 | -------------------------------------------------------------------------------- /src/rotateshape.m: -------------------------------------------------------------------------------- 1 | function shape_rotated = rotateshape(shape) 2 | %ROTATESHAPE Summary of this function goes here 3 | % Rotate input shape randomly 4 | % Detailed explanation goes here 5 | % Input: 6 | % shape: original shape 7 | % Output: 8 | % shape_rotated: rotated shape 9 | 10 | anti_clockwise_angle = 120*rand(1); 11 | 12 | shape_rotated = rotatepoints(shape,mean(shape),anti_clockwise_angle - 60, 1); 13 | 14 | end 15 | 16 | -------------------------------------------------------------------------------- /src/samplerandfeat.m: -------------------------------------------------------------------------------- 1 | function anglepairs = samplerandfeat(num_feats) 2 | %SAMPLERANDFEAT Summary of this function goes here 3 | % Function: generate the locations of pixel pairs randomly 4 | % Detailed explanation goes here 5 | % Input: 6 | % num_feats: number of features 7 | % max_radius: the maximum radius of local region 8 | % Output: 9 | % anglepairs: the angles of pixel pairs 10 | 11 | thetas_a = 2*pi*[0:1/(num_feats-1):1]; 12 | thetas_b = 2*pi*[0:1/(num_feats-1):1]; 13 | 14 | anglepairs = [thetas_a(randperm(length(thetas_a)))' thetas_b(randperm(length(thetas_b)))']; 15 | 16 | end 17 | 18 | -------------------------------------------------------------------------------- /src/scaleshape.m: -------------------------------------------------------------------------------- 1 | function shape_scaled = scaleshape(shape, scale) 2 | %SCALESHAPE Summary of this function goes here 3 | % Function: scale input shape using a scale ratio 4 | % Detailed explanation goes here 5 | % Input: 6 | % bbox: the bbox of current sample 7 | % shape: input shape 8 | % scale: scale ratio 9 | % Output: 10 | % shape_scaled: scaled shape 11 | 12 | shape_scaled = bsxfun(@plus, scale*(bsxfun(@minus, shape, mean(shape))), mean(shape)); 13 | 14 | 15 | end 16 | 17 | -------------------------------------------------------------------------------- /src/test_model.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwyang/face-alignment/104fc3cec4ee7786c797ed6bca13ed6d88cbda5f/src/test_model.m -------------------------------------------------------------------------------- /src/train_model.m: -------------------------------------------------------------------------------- 1 | %TRAIN_MODEL Summary of this function goes here 2 | % Function: train face alignment model 3 | % Detailed explanation goes here 4 | % Input: 5 | % dbnames: the names of database 6 | % Configure the parameters for training model 7 | global params; 8 | global Tr_Data; 9 | config_tr; 10 | 11 | if size(dbnames) > 1 & sum(strcmp(dbnames, 'COFW')) > 0 12 | disp('Sorry, COFW cannnot be combined with others') 13 | return; 14 | end 15 | 16 | if sum(strcmp(dbnames, 'COFW')) > 0 17 | load('../initial_shape/InitialShape_29.mat'); 18 | params.meanshape = S0; 19 | else 20 | load('../initial_shape/InitialShape_68.mat'); 21 | params.meanshape = S0; 22 | end 23 | 24 | 25 | if params.isparallel 26 | disp('Attention, if error occurs, plese ensure you used the correct version of parallel initialization.'); 27 | 28 | if isempty(gcp('nocreate')) 29 | parpool(4); 30 | else 31 | disp('Already initialized'); 32 | end 33 | 34 | %{ 35 | if matlabpool('size') <= 0 36 | matlabpool('open','local',4); 37 | else 38 | disp('Already initialized'); 39 | end 40 | %} 41 | end 42 | 43 | % load trainning data from hardware 44 | Tr_Data = []; 45 | % Tr_Bboxes = []; 46 | for i = 1:length(dbnames) 47 | % load training samples (including training images, and groundtruth shapes) 48 | imgpathlistfile = strcat('..\datasets\', dbnames{i}, '\Path_Images.txt'); 49 | tr_data = loadsamples(imgpathlistfile, 2); 50 | Tr_Data = [Tr_Data; tr_data]; 51 | end 52 | 53 | % Augmentate data for traing: assign multiple initial shapes to each image 54 | Data = Tr_Data; % (1:10:end); 55 | Param = params; 56 | 57 | if Param.flipflag % if conduct flipping 58 | Data_flip = cell(size(Data, 1), 1); 59 | for i = 1:length(Data_flip) 60 | Data_flip{i}.img_gray = fliplr(Data{i}.img_gray); 61 | Data_flip{i}.width_orig = Data{i}.width_orig; 62 | Data_flip{i}.height_orig = Data{i}.height_orig; 63 | Data_flip{i}.width = Data{i}.width; 64 | Data_flip{i}.height = Data{i}.height; 65 | 66 | Data_flip{i}.shape_gt = flipshape(Data{i}.shape_gt); 67 | Data_flip{i}.shape_gt(:, 1) = Data{i}.width - Data_flip{i}.shape_gt(:, 1); 68 | 69 | Data_flip{i}.bbox_gt = Data{i}.bbox_gt; 70 | Data_flip{i}.bbox_gt(1) = Data_flip{i}.width - Data_flip{i}.bbox_gt(1) - Data_flip{i}.bbox_gt(3); 71 | 72 | Data_flip{i}.bbox_facedet = Data{i}.bbox_facedet; 73 | Data_flip{i}.bbox_facedet(1) = Data_flip{i}.width - Data_flip{i}.bbox_facedet(1) - Data_flip{i}.bbox_facedet(3); 74 | end 75 | Data = [Data; Data_flip]; 76 | end 77 | 78 | % choose corresponding points for training 79 | for i = 1:length(Data) 80 | Data{i}.shape_gt = Data{i}.shape_gt(Param.ind_usedpts, :); 81 | Data{i}.bbox_gt = getbbox(Data{i}.shape_gt); 82 | 83 | % modify detection boxes 84 | shape_facedet = resetshape(Data{i}.bbox_facedet, Param.meanshape); 85 | shape_facedet = shape_facedet(Param.ind_usedpts, :); 86 | Data{i}.bbox_facedet = getbbox(shape_facedet); 87 | end 88 | 89 | Param.meanshape = S0(Param.ind_usedpts, :); 90 | 91 | dbsize = length(Data); 92 | 93 | % load('Ts_bbox.mat'); 94 | 95 | augnumber = Param.augnumber; 96 | 97 | 98 | for i = 1:dbsize 99 | % initializ the shape of current face image by randomly selecting multiple shapes from other face images 100 | % indice = ceil(dbsize*rand(1, augnumber)); 101 | 102 | indice_rotate = ceil(dbsize*rand(1, augnumber)); 103 | indice_shift = ceil(dbsize*rand(1, augnumber)); 104 | scales = 1 + 0.2*(rand([1 augnumber]) - 0.5); 105 | 106 | Data{i}.intermediate_shapes = cell(1, Param.max_numstage); 107 | Data{i}.intermediate_bboxes = cell(1, Param.max_numstage); 108 | 109 | Data{i}.intermediate_shapes{1} = zeros([size(Param.meanshape), augnumber]); 110 | Data{i}.intermediate_bboxes{1} = zeros([augnumber, size(Data{i}.bbox_gt, 2)]); 111 | 112 | Data{i}.shapes_residual = zeros([size(Param.meanshape), augnumber]); 113 | Data{i}.tf2meanshape = cell(augnumber, 1); 114 | Data{i}.meanshape2tf = cell(augnumber, 1); 115 | 116 | % if Data{i}.isdet == 1 117 | % Data{i}.bbox_facedet = Data{i}.bbox_facedet*ts_bbox; 118 | % end 119 | for sr = 1:params.augnumber 120 | if sr == 1 121 | % estimate the similarity transformation from initial shape to mean shape 122 | % Data{i}.intermediate_shapes{1}(:,:, sr) = resetshape(Data{i}.bbox_gt, Param.meanshape); 123 | % Data{i}.intermediate_bboxes{1}(sr, :) = Data{i}.bbox_gt; 124 | Data{i}.intermediate_shapes{1}(:,:, sr) = resetshape(Data{i}.bbox_facedet, Param.meanshape); 125 | Data{i}.intermediate_bboxes{1}(sr, :) = Data{i}.bbox_facedet; 126 | 127 | meanshape_resize = resetshape(Data{i}.intermediate_bboxes{1}(sr, :), Param.meanshape); 128 | 129 | Data{i}.tf2meanshape{1} = fitgeotrans(bsxfun(@minus, Data{i}.intermediate_shapes{1}(1:end,:, 1), mean(Data{i}.intermediate_shapes{1}(1:end,:, 1))), ... 130 | (bsxfun(@minus, meanshape_resize(1:end, :), mean(meanshape_resize(1:end, :)))), 'NonreflectiveSimilarity'); 131 | Data{i}.meanshape2tf{1} = fitgeotrans((bsxfun(@minus, meanshape_resize(1:end, :), mean(meanshape_resize(1:end, :)))), ... 132 | bsxfun(@minus, Data{i}.intermediate_shapes{1}(1:end,:, 1), mean(Data{i}.intermediate_shapes{1}(1:end,:, 1))), 'NonreflectiveSimilarity'); 133 | 134 | % calculate the residual shape from initial shape to groundtruth shape under normalization scale 135 | shape_residual = bsxfun(@rdivide, Data{i}.shape_gt - Data{i}.intermediate_shapes{1}(:,:, 1), [Data{i}.intermediate_bboxes{1}(1, 3) Data{i}.intermediate_bboxes{1}(1, 4)]); 136 | % transform the shape residual in the image coordinate to the mean shape coordinate 137 | [u, v] = transformPointsForward(Data{i}.tf2meanshape{1}, shape_residual(:, 1)', shape_residual(:, 2)'); 138 | Data{i}.shapes_residual(:, 1, 1) = u'; 139 | Data{i}.shapes_residual(:, 2, 1) = v'; 140 | else 141 | % randomly rotate the shape 142 | % shape = resetshape(Data{i}.bbox_gt, Param.meanshape); % Data{indice_rotate(sr)}.shape_gt 143 | shape = resetshape(Data{i}.bbox_facedet, Param.meanshape); % Data{indice_rotate(sr)}.shape_gt 144 | 145 | if params.augnumber_scale ~= 0 146 | shape = scaleshape(shape, scales(sr)); 147 | end 148 | 149 | if params.augnumber_rotate ~= 0 150 | shape = rotateshape(shape); 151 | end 152 | 153 | if params.augnumber_shift ~= 0 154 | shape = translateshape(shape, Data{indice_shift(sr)}.shape_gt); 155 | end 156 | 157 | Data{i}.intermediate_shapes{1}(:, :, sr) = shape; 158 | Data{i}.intermediate_bboxes{1}(sr, :) = getbbox(shape); 159 | 160 | meanshape_resize = resetshape(Data{i}.intermediate_bboxes{1}(sr, :), Param.meanshape); 161 | 162 | Data{i}.tf2meanshape{sr} = fitgeotrans(bsxfun(@minus, Data{i}.intermediate_shapes{1}(1:end,:, sr), mean(Data{i}.intermediate_shapes{1}(1:end,:, sr))), ... 163 | bsxfun(@minus, meanshape_resize(1:end, :), mean(meanshape_resize(1:end, :))), 'NonreflectiveSimilarity'); 164 | Data{i}.meanshape2tf{sr} = fitgeotrans(bsxfun(@minus, meanshape_resize(1:end, :), mean(meanshape_resize(1:end, :))), ... 165 | bsxfun(@minus, Data{i}.intermediate_shapes{1}(1:end,:, sr), mean(Data{i}.intermediate_shapes{1}(1:end,:, sr))), 'NonreflectiveSimilarity'); 166 | 167 | shape_residual = bsxfun(@rdivide, Data{i}.shape_gt - Data{i}.intermediate_shapes{1}(:,:, sr), [Data{i}.intermediate_bboxes{1}(sr, 3) Data{i}.intermediate_bboxes{1}(sr, 4)]); 168 | [u, v] = transformPointsForward(Data{i}.tf2meanshape{1}, shape_residual(:, 1)', shape_residual(:, 2)'); 169 | Data{i}.shapes_residual(:, 1, sr) = u'; 170 | Data{i}.shapes_residual(:, 2, sr) = v'; 171 | % Data{i}.shapes_residual(:, :, sr) = tformfwd(Data{i}.tf2meanshape{sr}, shape_residual(:, 1), shape_residual(:, 2)); 172 | end 173 | end 174 | end 175 | 176 | % train random forests for each landmark 177 | randf = cell(Param.max_numstage, 1); 178 | Ws = cell(Param.max_numstage, 1); 179 | 180 | %{ 181 | if nargin > 2 182 | n = size(LBFRegModel_initial.ranf, 1); 183 | for i = 1:n 184 | randf(1:n, :) = LBFRegModel_initial.ranf; 185 | end 186 | end 187 | %} 188 | 189 | for s = 1:Param.max_numstage 190 | % learn random forest for s-th stage 191 | disp('train random forests for landmarks...'); 192 | 193 | %{ 194 | if isempty(randf{s}) 195 | if exist(strcat('randfs\randf', num2str(s), '.mat')) 196 | load(strcat('randfs\randf', num2str(s), '.mat')); 197 | else 198 | tic; 199 | randf{s} = train_randomfs(Data, Param, s); 200 | toc; 201 | save(strcat('randfs\randf', num2str(s), '.mat'), 'randf', '-v7.3'); 202 | end 203 | end 204 | %} 205 | if isempty(randf{s}) 206 | tic; 207 | randf{s} = train_randomfs(Data, Param, s); 208 | toc; 209 | end 210 | 211 | % derive binary codes given learned random forest in current stage 212 | disp('extract local binary features...'); 213 | 214 | tic; 215 | binfeatures = derivebinaryfeat(randf{s}, Data, Param, s); 216 | % save(strcat('LBFeats\LBFeats', num2str(s), '.mat'), 'binfeatures', '-v7.3'); 217 | toc; 218 | 219 | % learn global linear regrassion given binary feature 220 | disp('learn global regressors...'); 221 | tic; 222 | [W, Data] = globalregression(binfeatures, Data, Param, s); 223 | Ws{s} = W; 224 | % save(strcat('Ws\W', num2str(s), '.mat'), 'W', '-v7.3'); 225 | toc; 226 | 227 | end 228 | 229 | LBFRegModel.ranf = randf; 230 | LBFRegModel.Ws = Ws; -------------------------------------------------------------------------------- /src/train_randomfs.m: -------------------------------------------------------------------------------- 1 | function rfs = train_randomfs(Tr_Data, params, stage) 2 | %TRAIN_RANDOMFS Summary of this function goes here 3 | % Function: train random forest for each landmark 4 | % Detailed explanation goes here 5 | % Input: 6 | % lmarkID: ID of landmark 7 | % stage: the stage of training process 8 | % Output: 9 | % randf: learned random forest 10 | dbsize = length(Tr_Data); 11 | 12 | % rf = cell(1, params.max_numtrees); 13 | 14 | overlap_ratio = params.bagging_overlap; 15 | 16 | Q = floor(double(dbsize)/((1-params.bagging_overlap)*(params.max_numtrees))); 17 | 18 | 19 | Data = cell(1, params.max_numtrees); 20 | for t = 1:params.max_numtrees 21 | % calculate the number of samples for each random tree 22 | % train t-th random tree 23 | is = max(floor((t-1)*Q - (t-1)*Q*overlap_ratio + 1), 1); 24 | ie = min(is + Q, dbsize); 25 | Data{t} = Tr_Data(is:ie); 26 | end 27 | 28 | 29 | % divide local region into grid 30 | params.radius = ([0:1/30:1]'); 31 | params.angles = 2*pi*[0:1/36:1]'; 32 | 33 | rfs = cell(length(params.meanshape), params.max_numtrees); 34 | 35 | parfor i = 1:length(params.meanshape) 36 | rf = cell(1, params.max_numtrees); 37 | % disp(strcat(num2str(i), 'th landmark is processing...')); 38 | for t = 1:params.max_numtrees 39 | % disp(strcat('training', {''}, num2str(t), '-th tree for', {''}, num2str(lmarkID), '-th landmark')); 40 | 41 | % calculate the number of samples for each random tree 42 | % train t-th random tree 43 | is = max(floor((t-1)*Q - (t-1)*Q*overlap_ratio + 1), 1); 44 | ie = min(is + Q, dbsize); 45 | 46 | max_numnodes = 2^params.max_depth - 1; 47 | 48 | rf{t}.ind_samples = cell(max_numnodes, 1); 49 | rf{t}.issplit = zeros(max_numnodes, 1); 50 | rf{t}.pnode = zeros(max_numnodes, 1); 51 | rf{t}.depth = zeros(max_numnodes, 1); 52 | rf{t}.cnodes = zeros(max_numnodes, 2); 53 | rf{t}.isleafnode = zeros(max_numnodes, 1); 54 | rf{t}.feat = zeros(max_numnodes, 4); 55 | rf{t}.thresh = zeros(max_numnodes, 1); 56 | 57 | rf{t}.ind_samples{1} = 1:(ie - is + 1)*(params.augnumber); 58 | rf{t}.issplit(1) = 0; 59 | rf{t}.pnode(1) = 0; 60 | rf{t}.depth(1) = 1; 61 | rf{t}.cnodes(1, 1:2) = [0 0]; 62 | rf{t}.isleafnode(1) = 1; 63 | rf{t}.feat(1, :) = zeros(1, 4); 64 | rf{t}.thresh(1) = 0; 65 | 66 | num_nodes = 1; 67 | num_leafnodes = 1; 68 | stop = 0; 69 | while(~stop) 70 | num_nodes_iter = num_nodes; 71 | num_split = 0; 72 | for n = 1:num_nodes_iter 73 | if ~rf{t}.issplit(n) 74 | if rf{t}.depth(n) == params.max_depth % || length(rf{t}.ind_samples{n}) < 20 75 | if rf{t}.depth(n) == 1 76 | rf{t}.depth(n) = 1; 77 | end 78 | rf{t}.issplit(n) = 1; 79 | else 80 | % separate the samples into left and right path 81 | 82 | [thresh, feat, lcind, rcind, isvalid] = splitnode(i, rf{t}.ind_samples{n}, Data{t}, params, stage); 83 | 84 | %{ 85 | if ~isvalid 86 | rf{t}.feat(n, :) = [0 0 0 0]; 87 | rf{t}.thresh(n) = 0; 88 | rf{t}.issplit(n) = 1; 89 | rf{t}.cnodes(n, :) = [0 0]; 90 | rf{t}.isleafnode(n) = 1; 91 | continue; 92 | end 93 | %} 94 | 95 | % set the threshold and featture for current node 96 | rf{t}.feat(n, :) = feat; 97 | rf{t}.thresh(n) = thresh; 98 | rf{t}.issplit(n) = 1; 99 | rf{t}.cnodes(n, :) = [num_nodes+1 num_nodes+2]; 100 | rf{t}.isleafnode(n) = 0; 101 | 102 | % add left and right child nodes into the random tree 103 | 104 | rf{t}.ind_samples{num_nodes+1} = lcind; 105 | rf{t}.issplit(num_nodes+1) = 0; 106 | rf{t}.pnode(num_nodes+1) = n; 107 | rf{t}.depth(num_nodes+1) = rf{t}.depth(n) + 1; 108 | rf{t}.cnodes(num_nodes+1, :) = [0 0]; 109 | rf{t}.isleafnode(num_nodes+1) = 1; 110 | 111 | rf{t}.ind_samples{num_nodes+2} = rcind; 112 | rf{t}.issplit(num_nodes+2) = 0; 113 | rf{t}.pnode(num_nodes+2) = n; 114 | rf{t}.depth(num_nodes+2) = rf{t}.depth(n) + 1; 115 | rf{t}.cnodes(num_nodes+2, :) = [0 0]; 116 | rf{t}.isleafnode(num_nodes+2) = 1; 117 | 118 | num_split = num_split + 1; 119 | num_leafnodes = num_leafnodes + 1; 120 | num_nodes = num_nodes + 2; 121 | end 122 | end 123 | end 124 | 125 | if num_split == 0 126 | stop = 1; 127 | else 128 | rf{t}.num_leafnodes = num_leafnodes; 129 | rf{t}.num_nodes = num_nodes; 130 | rf{t}.id_leafnodes = find(rf{t}.isleafnode == 1); 131 | end 132 | end 133 | 134 | end 135 | % disp(strcat(num2str(i), 'th landmark is over')); 136 | rfs(i, :) = rf; 137 | end 138 | end 139 | 140 | function [thresh, feat, lcind, rcind, isvalid] = splitnode(lmarkID, ind_samples, Tr_Data, params, stage) 141 | 142 | if isempty(ind_samples) 143 | thresh = 0; 144 | feat = [0 0 0 0]; 145 | rcind = []; 146 | lcind = []; 147 | isvalid = 1; 148 | return; 149 | end 150 | 151 | % generate params.max_rand cndidate feature 152 | % anglepairs = samplerandfeat(params.max_numfeat); 153 | % radiuspairs = [rand([params.max_numfeat, 1]) rand([params.max_numfeat, 1])]; 154 | [radiuspairs, anglepairs] = getproposals(params.max_numfeats(stage), params.radius, params.angles); 155 | 156 | angles_cos = cos(anglepairs); 157 | angles_sin = sin(anglepairs); 158 | 159 | % extract pixel difference features from pairs 160 | 161 | pdfeats = zeros(params.max_numfeats(stage), length(ind_samples)); 162 | 163 | shapes_residual = zeros(length(ind_samples), 2); 164 | 165 | for i = 1:length(ind_samples) 166 | s = floor((ind_samples(i)-1)/(params.augnumber)) + 1; 167 | k = mod(ind_samples(i)-1, (params.augnumber)) + 1; 168 | 169 | % calculate the relative location under the coordinate of meanshape 170 | pixel_a_x_imgcoord = (angles_cos(:, 1)).*radiuspairs(:, 1)*params.max_raio_radius(stage)*Tr_Data{s}.intermediate_bboxes{stage}(k, 3); 171 | pixel_a_y_imgcoord = (angles_sin(:, 1)).*radiuspairs(:, 1)*params.max_raio_radius(stage)*Tr_Data{s}.intermediate_bboxes{stage}(k, 4); 172 | 173 | pixel_b_x_imgcoord = (angles_cos(:, 2)).*radiuspairs(:, 2)*params.max_raio_radius(stage)*Tr_Data{s}.intermediate_bboxes{stage}(k, 3); 174 | pixel_b_y_imgcoord = (angles_sin(:, 2)).*radiuspairs(:, 2)*params.max_raio_radius(stage)*Tr_Data{s}.intermediate_bboxes{stage}(k, 4); 175 | 176 | % no transformation 177 | %{ 178 | pixel_a_x_lmcoord = pixel_a_x_imgcoord; 179 | pixel_a_y_lmcoord = pixel_a_y_imgcoord; 180 | 181 | pixel_b_x_lmcoord = pixel_b_x_imgcoord; 182 | pixel_b_y_lmcoord = pixel_b_y_imgcoord; 183 | %} 184 | 185 | % transform the pixels from image coordinate (meanshape) to coordinate of current shape 186 | 187 | [pixel_a_x_lmcoord, pixel_a_y_lmcoord] = transformPointsForward(Tr_Data{s}.meanshape2tf{k}, pixel_a_x_imgcoord', pixel_a_y_imgcoord'); 188 | pixel_a_x_lmcoord = pixel_a_x_lmcoord'; 189 | pixel_a_y_lmcoord = pixel_a_y_lmcoord'; 190 | 191 | [pixel_b_x_lmcoord, pixel_b_y_lmcoord] = transformPointsForward(Tr_Data{s}.meanshape2tf{k}, pixel_b_x_imgcoord', pixel_b_y_imgcoord'); 192 | pixel_b_x_lmcoord = pixel_b_x_lmcoord'; 193 | pixel_b_y_lmcoord = pixel_b_y_lmcoord'; 194 | 195 | pixel_a_x = int32(bsxfun(@plus, pixel_a_x_lmcoord, Tr_Data{s}.intermediate_shapes{stage}(lmarkID, 1, k))); 196 | pixel_a_y = int32(bsxfun(@plus, pixel_a_y_lmcoord, Tr_Data{s}.intermediate_shapes{stage}(lmarkID, 2, k))); 197 | 198 | pixel_b_x = int32(bsxfun(@plus, pixel_b_x_lmcoord, Tr_Data{s}.intermediate_shapes{stage}(lmarkID, 1, k))); 199 | pixel_b_y = int32(bsxfun(@plus, pixel_b_y_lmcoord, Tr_Data{s}.intermediate_shapes{stage}(lmarkID, 2, k))); 200 | 201 | width = (Tr_Data{s}.width); 202 | height = (Tr_Data{s}.height); 203 | 204 | pixel_a_x = max(1, min(pixel_a_x, width)); 205 | pixel_a_y = max(1, min(pixel_a_y, height)); 206 | 207 | pixel_b_x = max(1, min(pixel_b_x, width)); 208 | pixel_b_y = max(1, min(pixel_b_y, height)); 209 | 210 | pdfeats(:, i) = double(Tr_Data{s}.img_gray(pixel_a_y + (pixel_a_x-1)*height)) - double(Tr_Data{s}.img_gray(pixel_b_y + (pixel_b_x-1)*height)); 211 | %./ double(Tr_Data{s}.img_gray(pixel_a_y + (pixel_a_x-1)*height)) + double(Tr_Data{s}.img_gray(pixel_b_y + (pixel_b_x-1)*height)); 212 | 213 | % drawshapes(Tr_Data{s}.img_gray, [pixel_a_x pixel_a_y pixel_b_x pixel_b_y]); 214 | % hold off; 215 | 216 | shapes_residual(i, :) = Tr_Data{s}.shapes_residual(lmarkID, :, k); 217 | end 218 | 219 | E_x_2 = mean(shapes_residual(:, 1).^2); 220 | E_x = mean(shapes_residual(:, 1)); 221 | 222 | E_y_2 = mean(shapes_residual(:, 2).^2); 223 | E_y = mean(shapes_residual(:, 2)); 224 | % 225 | var_overall = length(ind_samples)*((E_x_2 - E_x^2) + (E_y_2 - E_y^2)); 226 | 227 | % var_overall = length(ind_samples)*(var(shapes_residual(:, 1)) + var(shapes_residual(:, 2))); 228 | 229 | % max_step = min(length(ind_samples), params.max_numthreshs); 230 | % step = floor(length(ind_samples)/max_step); 231 | max_step = 1; 232 | 233 | var_reductions = zeros(params.max_numfeats(stage), max_step); 234 | thresholds = zeros(params.max_numfeats(stage), max_step); 235 | 236 | [pdfeats_sorted] = sort(pdfeats, 2); 237 | 238 | % shapes_residual = shapes_residual(ind, :); 239 | 240 | for i = 1:params.max_numfeats(stage) 241 | % for t = 1:max_step 242 | t = 1; 243 | ind = ceil(length(ind_samples)*(0.5 + 0.9*(rand(1) - 0.5))); 244 | threshold = pdfeats_sorted(i, ind); % pdfeats_sorted(i, t*step); % 245 | thresholds(i, t) = threshold; 246 | ind_lc = (pdfeats(i, :) < threshold); 247 | ind_rc = (pdfeats(i, :) >= threshold); 248 | 249 | % figure, hold on, plot(shapes_residual(ind_lc, 1), shapes_residual(ind_lc, 2), 'r.') 250 | % plot(shapes_residual(ind_rc, 1), shapes_residual(ind_rc, 2), 'g.') 251 | % close; 252 | % compute 253 | 254 | E_x_2_lc = mean(shapes_residual(ind_lc, 1).^2); 255 | E_x_lc = mean(shapes_residual(ind_lc, 1)); 256 | 257 | E_y_2_lc = mean(shapes_residual(ind_lc, 2).^2); 258 | E_y_lc = mean(shapes_residual(ind_lc, 2)); 259 | 260 | var_lc = (E_x_2_lc + E_y_2_lc)- (E_x_lc^2 + E_y_lc^2); 261 | 262 | E_x_2_rc = (E_x_2*length(ind_samples) - E_x_2_lc*sum(ind_lc))/sum(ind_rc); 263 | E_x_rc = (E_x*length(ind_samples) - E_x_lc*sum(ind_lc))/sum(ind_rc); 264 | 265 | E_y_2_rc = (E_y_2*length(ind_samples) - E_y_2_lc*sum(ind_lc))/sum(ind_rc); 266 | E_y_rc = (E_y*length(ind_samples) - E_y_lc*sum(ind_lc))/sum(ind_rc); 267 | 268 | var_rc = (E_x_2_rc + E_y_2_rc)- (E_x_rc^2 + E_y_rc^2); 269 | 270 | var_reduce = var_overall - sum(ind_lc)*var_lc - sum(ind_rc)*var_rc; 271 | 272 | % var_reduce = var_overall - sum(ind_lc)*(var(shapes_residual(ind_lc, 1)) + var(shapes_residual(ind_lc, 2))) - sum(ind_rc)*(var(shapes_residual(ind_rc, 1)) + var(shapes_residual(ind_rc, 2))); 273 | var_reductions(i, t) = var_reduce; 274 | % end 275 | % plot(var_reductions(i, :)); 276 | end 277 | 278 | [~, ind_colmax] = max(var_reductions); 279 | ind_max = 1; 280 | 281 | %{ 282 | if var_max <= 0 283 | isvalid = 0; 284 | else 285 | isvalid = 1; 286 | end 287 | %} 288 | isvalid = 1; 289 | 290 | thresh = thresholds(ind_colmax(ind_max), ind_max); 291 | 292 | feat = [anglepairs(ind_colmax(ind_max), :) radiuspairs(ind_colmax(ind_max), :)]; 293 | 294 | lcind = ind_samples(find(pdfeats(ind_colmax(ind_max), :) < thresh)); 295 | rcind = ind_samples(find(pdfeats(ind_colmax(ind_max), :) >= thresh)); 296 | 297 | end 298 | 299 | -------------------------------------------------------------------------------- /src/translateshape.m: -------------------------------------------------------------------------------- 1 | function [shape_initial] = translateshape(meanshape, shape_union) 2 | %RESETSHAPE Summary of this function goes here 3 | % Function: reset the initial shape according to the groundtruth shape and union shape for all faces 4 | % Detailed explanation goes here 5 | % Input: 6 | % bbox: bbounding box of groundtruth shape 7 | % shape_union: uniionshape 8 | % Output: 9 | % shape_initial: reset initial shape 10 | % bbox: bounding box of face image 11 | 12 | % get the bounding box according to the ground truth shape 13 | width_union = (max(shape_union(:, 1)) - min(shape_union(:, 1))); 14 | height_union = (max(shape_union(:, 2)) - min(shape_union(:, 2))); 15 | 16 | width_meanshape = (max(meanshape(:, 1)) - min(meanshape(:, 1))); 17 | height_meanshape = (max(meanshape(:, 2)) - min(meanshape(:, 2))); 18 | 19 | shape_union = bsxfun(@minus, (shape_union), (min(shape_union))); 20 | % get the center point of union shape 21 | shape_center = mean(shape_union)./[width_union, height_union]; 22 | 23 | shape_initial = bsxfun(@plus, meanshape, [width_meanshape*(shape_center(1)-0.5) height_meanshape*(shape_center(2)-0.5)]); 24 | 25 | end 26 | 27 | -------------------------------------------------------------------------------- /src/write_binary_model.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jwyang/face-alignment/104fc3cec4ee7786c797ed6bca13ed6d88cbda5f/src/write_binary_model.m --------------------------------------------------------------------------------