├── .github └── ISSUE_TEMPLATE │ ├── bug_report.md │ ├── discussion.md │ ├── feature_request.md │ └── help.md ├── .gitignore ├── FaceBoxes ├── .gitignore ├── FaceBoxes.py ├── FaceBoxes_ONNX.py ├── __init__.py ├── build_cpu_nms.sh ├── models │ ├── __init__.py │ └── faceboxes.py ├── onnx.py ├── readme.md ├── utils │ ├── .gitignore │ ├── __init__.py │ ├── box_utils.py │ ├── build.py │ ├── config.py │ ├── functions.py │ ├── nms │ │ ├── .gitignore │ │ ├── __init__.py │ │ ├── cpu_nms.pyx │ │ └── py_cpu_nms.py │ ├── nms_wrapper.py │ ├── prior_box.py │ └── timer.py └── weights │ ├── .gitignore │ ├── FaceBoxesProd.pth │ └── readme.md ├── LICENSE ├── Sim3DR ├── .gitignore ├── Sim3DR.py ├── __init__.py ├── _init_paths.py ├── build_sim3dr.sh ├── lib │ ├── rasterize.h │ ├── rasterize.pyx │ └── rasterize_kernel.cpp ├── lighting.py ├── readme.md ├── setup.py └── tests │ ├── .gitignore │ ├── CMakeLists.txt │ ├── io.cpp │ ├── io.h │ └── test.cpp ├── TDDFA.py ├── TDDFA_ONNX.py ├── bfm ├── .gitignore ├── __init__.py ├── bfm.py ├── bfm_onnx.py └── readme.md ├── build.sh ├── configs ├── .gitignore ├── BFM_UV.mat ├── bfm_noneck_v3.pkl ├── indices.npy ├── mb05_120x120.yml ├── mb1_120x120.yml ├── ncc_code.npy ├── param_mean_std_62d_120x120.pkl ├── readme.md ├── resnet_120x120.yml └── tri.pkl ├── demo.ipynb ├── demo.py ├── demo_video.py ├── demo_video_smooth.py ├── demo_webcam_smooth.py ├── docs └── images │ ├── emma_3d.jpg │ ├── latency.gif │ ├── obj.jpg │ ├── out.gif │ ├── ply.jpg │ ├── trump_biden_3d.jpg │ ├── trump_hillary_2d_dense.jpg │ ├── trump_hillary_2d_sparse.jpg │ ├── trump_hillary_3d.jpg │ ├── trump_hillary_depth.jpg │ ├── trump_hillary_pncc.jpg │ ├── trump_hillary_pose.jpg │ ├── trump_hillary_uv_tex.jpg │ └── webcam.gif ├── examples ├── .gitignore ├── inputs │ ├── .gitignore │ ├── JianzhuGuo.jpg │ ├── emma.jpg │ ├── trump_hillary.jpg │ └── videos │ │ ├── .gitignore │ │ └── 214.avi └── results │ ├── .gitignore │ └── videos │ └── .gitignore ├── gradiodemo.py ├── latency.py ├── models ├── __init__.py ├── mobilenet_v1.py ├── mobilenet_v3.py └── resnet.py ├── readme.md ├── requirements.txt ├── speed_cpu.py ├── utils ├── __init__.py ├── asset │ ├── .gitignore │ ├── build_render_ctypes.sh │ └── render.c ├── depth.py ├── functions.py ├── io.py ├── onnx.py ├── pncc.py ├── pose.py ├── render.py ├── render_ctypes.py ├── serialization.py ├── tddfa_util.py └── uv.py └── weights ├── .gitignore ├── mb05_120x120.pth ├── mb1_120x120.pth └── readme.md /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | Before proposing this issue, plz search the existed issues by your keywords first. 11 | 12 | **Describe the bug** 13 | 14 | **To Reproduce** 15 | 16 | **Expected behavior** 17 | 18 | **Screenshots** 19 | 20 | **Platform:** 21 | - macOS, Linux, or Windows 22 | 23 | **Additional context** 24 | Add any other context about the problem here. 25 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/discussion.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Discussion 3 | about: Just type your opinions. 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Type your opinions or ideas here.** -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | 11 | **Is your feature request related to a problem? Please describe.** 12 | Please describe the problem you are trying to solve. 13 | 14 | **Describe the solution you'd like** 15 | Please describe the desired behavior. 16 | 17 | **Describe alternatives you've considered** 18 | Please describe alternative solutions or features you have considered. -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/help.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Help 3 | about: Please file an issue in our help repo. 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Please detail your questions here : )** 11 | 12 | Questions posted to this repository will be closed. -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | **/.DS_Store 3 | 4 | __pycache__ 5 | **/__pycache__ 6 | 7 | .idea/ 8 | tmp/ 9 | 10 | .ipynb_checkpoints/ 11 | 12 | *gif.py 13 | refine_index.py 14 | test*.py 15 | clean_git.sh 16 | 17 | websmooth.py 18 | demo_webcam.py 19 | 20 | examples/inputs/spoof/ 21 | weights/*.onnx 22 | configs/ncc_code_53215.npy 23 | pr.* 24 | *.docx 25 | render_3dface.ipynb 26 | test_onnx.py 27 | benchmark.py 28 | how_to_make_video_demos.txt 29 | 30 | .vscode/ 31 | test_pr*.py 32 | demo_for_*.py 33 | examples/**/zou* 34 | examples/inputs/videos -------------------------------------------------------------------------------- /FaceBoxes/.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | __pycache__ 3 | **/__pycache__ -------------------------------------------------------------------------------- /FaceBoxes/FaceBoxes.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | import os.path as osp 4 | 5 | import torch 6 | import numpy as np 7 | import cv2 8 | 9 | from .utils.prior_box import PriorBox 10 | from .utils.nms_wrapper import nms 11 | from .utils.box_utils import decode 12 | from .utils.timer import Timer 13 | from .utils.functions import check_keys, remove_prefix, load_model 14 | from .utils.config import cfg 15 | from .models.faceboxes import FaceBoxesNet 16 | 17 | # some global configs 18 | confidence_threshold = 0.05 19 | top_k = 5000 20 | keep_top_k = 750 21 | nms_threshold = 0.3 22 | vis_thres = 0.5 23 | resize = 1 24 | 25 | scale_flag = True 26 | HEIGHT, WIDTH = 720, 1080 27 | 28 | make_abs_path = lambda fn: osp.join(osp.dirname(osp.realpath(__file__)), fn) 29 | pretrained_path = make_abs_path('weights/FaceBoxesProd.pth') 30 | 31 | 32 | def viz_bbox(img, dets, wfp='out.jpg'): 33 | # show 34 | for b in dets: 35 | if b[4] < vis_thres: 36 | continue 37 | text = "{:.4f}".format(b[4]) 38 | b = list(map(int, b)) 39 | cv2.rectangle(img, (b[0], b[1]), (b[2], b[3]), (0, 0, 255), 2) 40 | cx = b[0] 41 | cy = b[1] + 12 42 | cv2.putText(img, text, (cx, cy), cv2.FONT_HERSHEY_DUPLEX, 0.5, (255, 255, 255)) 43 | cv2.imwrite(wfp, img) 44 | print(f'Viz bbox to {wfp}') 45 | 46 | 47 | class FaceBoxes: 48 | def __init__(self, timer_flag=False): 49 | torch.set_grad_enabled(False) 50 | 51 | net = FaceBoxesNet(phase='test', size=None, num_classes=2) # initialize detector 52 | self.net = load_model(net, pretrained_path=pretrained_path, load_to_cpu=True) 53 | self.net.eval() 54 | # print('Finished loading model!') 55 | 56 | self.timer_flag = timer_flag 57 | 58 | def __call__(self, img_): 59 | img_raw = img_.copy() 60 | 61 | # scaling to speed up 62 | scale = 1 63 | if scale_flag: 64 | h, w = img_raw.shape[:2] 65 | if h > HEIGHT: 66 | scale = HEIGHT / h 67 | if w * scale > WIDTH: 68 | scale *= WIDTH / (w * scale) 69 | # print(scale) 70 | if scale == 1: 71 | img_raw_scale = img_raw 72 | else: 73 | h_s = int(scale * h) 74 | w_s = int(scale * w) 75 | # print(h_s, w_s) 76 | img_raw_scale = cv2.resize(img_raw, dsize=(w_s, h_s)) 77 | # print(img_raw_scale.shape) 78 | 79 | img = np.float32(img_raw_scale) 80 | else: 81 | img = np.float32(img_raw) 82 | 83 | # forward 84 | _t = {'forward_pass': Timer(), 'misc': Timer()} 85 | im_height, im_width, _ = img.shape 86 | scale_bbox = torch.Tensor([img.shape[1], img.shape[0], img.shape[1], img.shape[0]]) 87 | img -= (104, 117, 123) 88 | img = img.transpose(2, 0, 1) 89 | img = torch.from_numpy(img).unsqueeze(0) 90 | 91 | _t['forward_pass'].tic() 92 | loc, conf = self.net(img) # forward pass 93 | _t['forward_pass'].toc() 94 | _t['misc'].tic() 95 | priorbox = PriorBox(image_size=(im_height, im_width)) 96 | priors = priorbox.forward() 97 | prior_data = priors.data 98 | boxes = decode(loc.data.squeeze(0), prior_data, cfg['variance']) 99 | if scale_flag: 100 | boxes = boxes * scale_bbox / scale / resize 101 | else: 102 | boxes = boxes * scale_bbox / resize 103 | 104 | boxes = boxes.cpu().numpy() 105 | scores = conf.squeeze(0).data.cpu().numpy()[:, 1] 106 | 107 | # ignore low scores 108 | inds = np.where(scores > confidence_threshold)[0] 109 | boxes = boxes[inds] 110 | scores = scores[inds] 111 | 112 | # keep top-K before NMS 113 | order = scores.argsort()[::-1][:top_k] 114 | boxes = boxes[order] 115 | scores = scores[order] 116 | 117 | # do NMS 118 | dets = np.hstack((boxes, scores[:, np.newaxis])).astype(np.float32, copy=False) 119 | keep = nms(dets, nms_threshold) 120 | dets = dets[keep, :] 121 | 122 | # keep top-K faster NMS 123 | dets = dets[:keep_top_k, :] 124 | _t['misc'].toc() 125 | 126 | if self.timer_flag: 127 | print('Detection: {:d}/{:d} forward_pass_time: {:.4f}s misc: {:.4f}s'.format(1, 1, _t[ 128 | 'forward_pass'].average_time, _t['misc'].average_time)) 129 | 130 | # filter using vis_thres 131 | det_bboxes = [] 132 | for b in dets: 133 | if b[4] > vis_thres: 134 | xmin, ymin, xmax, ymax, score = b[0], b[1], b[2], b[3], b[4] 135 | bbox = [xmin, ymin, xmax, ymax, score] 136 | det_bboxes.append(bbox) 137 | 138 | return det_bboxes 139 | 140 | 141 | def main(): 142 | face_boxes = FaceBoxes(timer_flag=True) 143 | 144 | fn = 'trump_hillary.jpg' 145 | img_fp = f'../examples/inputs/{fn}' 146 | img = cv2.imread(img_fp) 147 | print(f'input shape: {img.shape}') 148 | dets = face_boxes(img) # xmin, ymin, w, h 149 | # print(dets) 150 | 151 | # repeating inference for `n` times 152 | n = 10 153 | for i in range(n): 154 | dets = face_boxes(img) 155 | 156 | wfn = fn.replace('.jpg', '_det.jpg') 157 | wfp = osp.join('../examples/results', wfn) 158 | viz_bbox(img, dets, wfp) 159 | 160 | 161 | if __name__ == '__main__': 162 | main() 163 | -------------------------------------------------------------------------------- /FaceBoxes/FaceBoxes_ONNX.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | import os.path as osp 4 | 5 | import torch 6 | import numpy as np 7 | import cv2 8 | 9 | from .utils.prior_box import PriorBox 10 | from .utils.nms_wrapper import nms 11 | from .utils.box_utils import decode 12 | from .utils.timer import Timer 13 | from .utils.config import cfg 14 | from .onnx import convert_to_onnx 15 | 16 | import onnxruntime 17 | 18 | # some global configs 19 | confidence_threshold = 0.05 20 | top_k = 5000 21 | keep_top_k = 750 22 | nms_threshold = 0.3 23 | vis_thres = 0.5 24 | resize = 1 25 | 26 | scale_flag = True 27 | HEIGHT, WIDTH = 720, 1080 28 | 29 | make_abs_path = lambda fn: osp.join(osp.dirname(osp.realpath(__file__)), fn) 30 | onnx_path = make_abs_path('weights/FaceBoxesProd.onnx') 31 | 32 | 33 | def viz_bbox(img, dets, wfp='out.jpg'): 34 | # show 35 | for b in dets: 36 | if b[4] < vis_thres: 37 | continue 38 | text = "{:.4f}".format(b[4]) 39 | b = list(map(int, b)) 40 | cv2.rectangle(img, (b[0], b[1]), (b[2], b[3]), (0, 0, 255), 2) 41 | cx = b[0] 42 | cy = b[1] + 12 43 | cv2.putText(img, text, (cx, cy), cv2.FONT_HERSHEY_DUPLEX, 0.5, (255, 255, 255)) 44 | cv2.imwrite(wfp, img) 45 | print(f'Viz bbox to {wfp}') 46 | 47 | 48 | class FaceBoxes_ONNX(object): 49 | def __init__(self, timer_flag=False): 50 | if not osp.exists(onnx_path): 51 | convert_to_onnx(onnx_path) 52 | self.session = onnxruntime.InferenceSession(onnx_path, None) 53 | 54 | self.timer_flag = timer_flag 55 | 56 | def __call__(self, img_): 57 | img_raw = img_.copy() 58 | 59 | # scaling to speed up 60 | scale = 1 61 | if scale_flag: 62 | h, w = img_raw.shape[:2] 63 | if h > HEIGHT: 64 | scale = HEIGHT / h 65 | if w * scale > WIDTH: 66 | scale *= WIDTH / (w * scale) 67 | # print(scale) 68 | if scale == 1: 69 | img_raw_scale = img_raw 70 | else: 71 | h_s = int(scale * h) 72 | w_s = int(scale * w) 73 | # print(h_s, w_s) 74 | img_raw_scale = cv2.resize(img_raw, dsize=(w_s, h_s)) 75 | # print(img_raw_scale.shape) 76 | 77 | img = np.float32(img_raw_scale) 78 | else: 79 | img = np.float32(img_raw) 80 | 81 | # forward 82 | _t = {'forward_pass': Timer(), 'misc': Timer()} 83 | im_height, im_width, _ = img.shape 84 | scale_bbox = torch.Tensor([img.shape[1], img.shape[0], img.shape[1], img.shape[0]]) 85 | 86 | img -= (104, 117, 123) 87 | img = img.transpose(2, 0, 1) 88 | # img = torch.from_numpy(img).unsqueeze(0) 89 | img = img[np.newaxis, ...] 90 | 91 | _t['forward_pass'].tic() 92 | # loc, conf = self.net(img) # forward pass 93 | out = self.session.run(None, {'input': img}) 94 | loc, conf = out[0], out[1] 95 | # for compatibility, may need to optimize 96 | loc = torch.from_numpy(loc) 97 | _t['forward_pass'].toc() 98 | _t['misc'].tic() 99 | 100 | priorbox = PriorBox(image_size=(im_height, im_width)) 101 | priors = priorbox.forward() 102 | prior_data = priors.data 103 | boxes = decode(loc.data.squeeze(0), prior_data, cfg['variance']) 104 | if scale_flag: 105 | boxes = boxes * scale_bbox / scale / resize 106 | else: 107 | boxes = boxes * scale_bbox / resize 108 | 109 | boxes = boxes.cpu().numpy() 110 | scores = conf[0][:, 1] 111 | # scores = conf.squeeze(0).data.cpu().numpy()[:, 1] 112 | 113 | # ignore low scores 114 | inds = np.where(scores > confidence_threshold)[0] 115 | boxes = boxes[inds] 116 | scores = scores[inds] 117 | 118 | # keep top-K before NMS 119 | order = scores.argsort()[::-1][:top_k] 120 | boxes = boxes[order] 121 | scores = scores[order] 122 | 123 | # do NMS 124 | dets = np.hstack((boxes, scores[:, np.newaxis])).astype(np.float32, copy=False) 125 | keep = nms(dets, nms_threshold) 126 | dets = dets[keep, :] 127 | 128 | # keep top-K faster NMS 129 | dets = dets[:keep_top_k, :] 130 | _t['misc'].toc() 131 | 132 | if self.timer_flag: 133 | print('Detection: {:d}/{:d} forward_pass_time: {:.4f}s misc: {:.4f}s'.format(1, 1, _t[ 134 | 'forward_pass'].average_time, _t['misc'].average_time)) 135 | 136 | # filter using vis_thres 137 | det_bboxes = [] 138 | for b in dets: 139 | if b[4] > vis_thres: 140 | xmin, ymin, xmax, ymax, score = b[0], b[1], b[2], b[3], b[4] 141 | bbox = [xmin, ymin, xmax, ymax, score] 142 | det_bboxes.append(bbox) 143 | 144 | return det_bboxes 145 | 146 | 147 | def main(): 148 | face_boxes = FaceBoxes_ONNX(timer_flag=True) 149 | 150 | fn = 'trump_hillary.jpg' 151 | img_fp = f'../examples/inputs/{fn}' 152 | img = cv2.imread(img_fp) 153 | print(f'input shape: {img.shape}') 154 | dets = face_boxes(img) # xmin, ymin, w, h 155 | # print(dets) 156 | 157 | # repeating inference for `n` times 158 | n = 10 159 | for i in range(n): 160 | dets = face_boxes(img) 161 | 162 | wfn = fn.replace('.jpg', '_det.jpg') 163 | wfp = osp.join('../examples/results', wfn) 164 | viz_bbox(img, dets, wfp) 165 | 166 | 167 | if __name__ == '__main__': 168 | main() 169 | -------------------------------------------------------------------------------- /FaceBoxes/__init__.py: -------------------------------------------------------------------------------- 1 | from .FaceBoxes import FaceBoxes 2 | -------------------------------------------------------------------------------- /FaceBoxes/build_cpu_nms.sh: -------------------------------------------------------------------------------- 1 | cd utils 2 | python3 build.py build_ext --inplace 3 | cd .. -------------------------------------------------------------------------------- /FaceBoxes/models/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/FaceBoxes/models/__init__.py -------------------------------------------------------------------------------- /FaceBoxes/models/faceboxes.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | import torch 4 | import torch.nn as nn 5 | import torch.nn.functional as F 6 | 7 | 8 | class BasicConv2d(nn.Module): 9 | 10 | def __init__(self, in_channels, out_channels, **kwargs): 11 | super(BasicConv2d, self).__init__() 12 | self.conv = nn.Conv2d(in_channels, out_channels, bias=False, **kwargs) 13 | self.bn = nn.BatchNorm2d(out_channels, eps=1e-5) 14 | 15 | def forward(self, x): 16 | x = self.conv(x) 17 | x = self.bn(x) 18 | return F.relu(x, inplace=True) 19 | 20 | 21 | class Inception(nn.Module): 22 | def __init__(self): 23 | super(Inception, self).__init__() 24 | self.branch1x1 = BasicConv2d(128, 32, kernel_size=1, padding=0) 25 | self.branch1x1_2 = BasicConv2d(128, 32, kernel_size=1, padding=0) 26 | self.branch3x3_reduce = BasicConv2d(128, 24, kernel_size=1, padding=0) 27 | self.branch3x3 = BasicConv2d(24, 32, kernel_size=3, padding=1) 28 | self.branch3x3_reduce_2 = BasicConv2d(128, 24, kernel_size=1, padding=0) 29 | self.branch3x3_2 = BasicConv2d(24, 32, kernel_size=3, padding=1) 30 | self.branch3x3_3 = BasicConv2d(32, 32, kernel_size=3, padding=1) 31 | 32 | def forward(self, x): 33 | branch1x1 = self.branch1x1(x) 34 | 35 | branch1x1_pool = F.avg_pool2d(x, kernel_size=3, stride=1, padding=1) 36 | branch1x1_2 = self.branch1x1_2(branch1x1_pool) 37 | 38 | branch3x3_reduce = self.branch3x3_reduce(x) 39 | branch3x3 = self.branch3x3(branch3x3_reduce) 40 | 41 | branch3x3_reduce_2 = self.branch3x3_reduce_2(x) 42 | branch3x3_2 = self.branch3x3_2(branch3x3_reduce_2) 43 | branch3x3_3 = self.branch3x3_3(branch3x3_2) 44 | 45 | outputs = [branch1x1, branch1x1_2, branch3x3, branch3x3_3] 46 | return torch.cat(outputs, 1) 47 | 48 | 49 | class CRelu(nn.Module): 50 | 51 | def __init__(self, in_channels, out_channels, **kwargs): 52 | super(CRelu, self).__init__() 53 | self.conv = nn.Conv2d(in_channels, out_channels, bias=False, **kwargs) 54 | self.bn = nn.BatchNorm2d(out_channels, eps=1e-5) 55 | 56 | def forward(self, x): 57 | x = self.conv(x) 58 | x = self.bn(x) 59 | x = torch.cat([x, -x], 1) 60 | x = F.relu(x, inplace=True) 61 | return x 62 | 63 | 64 | class FaceBoxesNet(nn.Module): 65 | 66 | def __init__(self, phase, size, num_classes): 67 | super(FaceBoxesNet, self).__init__() 68 | self.phase = phase 69 | self.num_classes = num_classes 70 | self.size = size 71 | 72 | self.conv1 = CRelu(3, 24, kernel_size=7, stride=4, padding=3) 73 | self.conv2 = CRelu(48, 64, kernel_size=5, stride=2, padding=2) 74 | 75 | self.inception1 = Inception() 76 | self.inception2 = Inception() 77 | self.inception3 = Inception() 78 | 79 | self.conv3_1 = BasicConv2d(128, 128, kernel_size=1, stride=1, padding=0) 80 | self.conv3_2 = BasicConv2d(128, 256, kernel_size=3, stride=2, padding=1) 81 | 82 | self.conv4_1 = BasicConv2d(256, 128, kernel_size=1, stride=1, padding=0) 83 | self.conv4_2 = BasicConv2d(128, 256, kernel_size=3, stride=2, padding=1) 84 | 85 | self.loc, self.conf = self.multibox(self.num_classes) 86 | 87 | if self.phase == 'test': 88 | self.softmax = nn.Softmax(dim=-1) 89 | 90 | if self.phase == 'train': 91 | for m in self.modules(): 92 | if isinstance(m, nn.Conv2d): 93 | if m.bias is not None: 94 | nn.init.xavier_normal_(m.weight.data) 95 | m.bias.data.fill_(0.02) 96 | else: 97 | m.weight.data.normal_(0, 0.01) 98 | elif isinstance(m, nn.BatchNorm2d): 99 | m.weight.data.fill_(1) 100 | m.bias.data.zero_() 101 | 102 | def multibox(self, num_classes): 103 | loc_layers = [] 104 | conf_layers = [] 105 | loc_layers += [nn.Conv2d(128, 21 * 4, kernel_size=3, padding=1)] 106 | conf_layers += [nn.Conv2d(128, 21 * num_classes, kernel_size=3, padding=1)] 107 | loc_layers += [nn.Conv2d(256, 1 * 4, kernel_size=3, padding=1)] 108 | conf_layers += [nn.Conv2d(256, 1 * num_classes, kernel_size=3, padding=1)] 109 | loc_layers += [nn.Conv2d(256, 1 * 4, kernel_size=3, padding=1)] 110 | conf_layers += [nn.Conv2d(256, 1 * num_classes, kernel_size=3, padding=1)] 111 | return nn.Sequential(*loc_layers), nn.Sequential(*conf_layers) 112 | 113 | def forward(self, x): 114 | 115 | detection_sources = list() 116 | loc = list() 117 | conf = list() 118 | 119 | x = self.conv1(x) 120 | x = F.max_pool2d(x, kernel_size=3, stride=2, padding=1) 121 | x = self.conv2(x) 122 | x = F.max_pool2d(x, kernel_size=3, stride=2, padding=1) 123 | x = self.inception1(x) 124 | x = self.inception2(x) 125 | x = self.inception3(x) 126 | detection_sources.append(x) 127 | 128 | x = self.conv3_1(x) 129 | x = self.conv3_2(x) 130 | detection_sources.append(x) 131 | 132 | x = self.conv4_1(x) 133 | x = self.conv4_2(x) 134 | detection_sources.append(x) 135 | 136 | for (x, l, c) in zip(detection_sources, self.loc, self.conf): 137 | loc.append(l(x).permute(0, 2, 3, 1).contiguous()) 138 | conf.append(c(x).permute(0, 2, 3, 1).contiguous()) 139 | 140 | loc = torch.cat([o.view(o.size(0), -1) for o in loc], 1) 141 | conf = torch.cat([o.view(o.size(0), -1) for o in conf], 1) 142 | 143 | if self.phase == "test": 144 | output = (loc.view(loc.size(0), -1, 4), 145 | self.softmax(conf.view(conf.size(0), -1, self.num_classes))) 146 | else: 147 | output = (loc.view(loc.size(0), -1, 4), 148 | conf.view(conf.size(0), -1, self.num_classes)) 149 | 150 | return output 151 | -------------------------------------------------------------------------------- /FaceBoxes/onnx.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import torch 6 | 7 | from .models.faceboxes import FaceBoxesNet 8 | from .utils.functions import load_model 9 | 10 | 11 | def convert_to_onnx(onnx_path): 12 | pretrained_path = onnx_path.replace('.onnx', '.pth') 13 | # 1. load model 14 | torch.set_grad_enabled(False) 15 | net = FaceBoxesNet(phase='test', size=None, num_classes=2) # initialize detector 16 | net = load_model(net, pretrained_path=pretrained_path, load_to_cpu=True) 17 | net.eval() 18 | 19 | # 2. convert 20 | batch_size = 1 21 | dummy_input = torch.randn(batch_size, 3, 720, 1080) 22 | # export with dynamic axes for various input sizes 23 | torch.onnx.export( 24 | net, 25 | (dummy_input,), 26 | onnx_path, 27 | input_names=['input'], 28 | output_names=['output'], 29 | dynamic_axes={ 30 | 'input': [0, 2, 3], 31 | 'output': [0] 32 | }, 33 | do_constant_folding=True 34 | ) 35 | print(f'Convert {pretrained_path} to {onnx_path} done.') 36 | -------------------------------------------------------------------------------- /FaceBoxes/readme.md: -------------------------------------------------------------------------------- 1 | ## How to fun FaceBoxes 2 | 3 | ### Build the cpu version of NMS 4 | ```shell script 5 | cd utils 6 | python3 build.py build_ext --inplace 7 | ``` 8 | 9 | or just run 10 | 11 | ```shell script 12 | sh ./build_cpu_nms.sh 13 | ``` 14 | 15 | ### Run the demo of face detection 16 | ```shell script 17 | python3 FaceBoxes.py 18 | ``` -------------------------------------------------------------------------------- /FaceBoxes/utils/.gitignore: -------------------------------------------------------------------------------- 1 | utils/build 2 | utils/nms/*.so 3 | utils/*.c 4 | build/ 5 | -------------------------------------------------------------------------------- /FaceBoxes/utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/FaceBoxes/utils/__init__.py -------------------------------------------------------------------------------- /FaceBoxes/utils/box_utils.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | import torch 4 | import numpy as np 5 | 6 | 7 | def point_form(boxes): 8 | """ Convert prior_boxes to (xmin, ymin, xmax, ymax) 9 | representation for comparison to point form ground truth data. 10 | Args: 11 | boxes: (tensor) center-size default boxes from priorbox layers. 12 | Return: 13 | boxes: (tensor) Converted xmin, ymin, xmax, ymax form of boxes. 14 | """ 15 | return torch.cat((boxes[:, :2] - boxes[:, 2:] / 2, # xmin, ymin 16 | boxes[:, :2] + boxes[:, 2:] / 2), 1) # xmax, ymax 17 | 18 | 19 | def center_size(boxes): 20 | """ Convert prior_boxes to (cx, cy, w, h) 21 | representation for comparison to center-size form ground truth data. 22 | Args: 23 | boxes: (tensor) point_form boxes 24 | Return: 25 | boxes: (tensor) Converted xmin, ymin, xmax, ymax form of boxes. 26 | """ 27 | return torch.cat((boxes[:, 2:] + boxes[:, :2]) / 2, # cx, cy 28 | boxes[:, 2:] - boxes[:, :2], 1) # w, h 29 | 30 | 31 | def intersect(box_a, box_b): 32 | """ We resize both tensors to [A,B,2] without new malloc: 33 | [A,2] -> [A,1,2] -> [A,B,2] 34 | [B,2] -> [1,B,2] -> [A,B,2] 35 | Then we compute the area of intersect between box_a and box_b. 36 | Args: 37 | box_a: (tensor) bounding boxes, Shape: [A,4]. 38 | box_b: (tensor) bounding boxes, Shape: [B,4]. 39 | Return: 40 | (tensor) intersection area, Shape: [A,B]. 41 | """ 42 | A = box_a.size(0) 43 | B = box_b.size(0) 44 | max_xy = torch.min(box_a[:, 2:].unsqueeze(1).expand(A, B, 2), 45 | box_b[:, 2:].unsqueeze(0).expand(A, B, 2)) 46 | min_xy = torch.max(box_a[:, :2].unsqueeze(1).expand(A, B, 2), 47 | box_b[:, :2].unsqueeze(0).expand(A, B, 2)) 48 | inter = torch.clamp((max_xy - min_xy), min=0) 49 | return inter[:, :, 0] * inter[:, :, 1] 50 | 51 | 52 | def jaccard(box_a, box_b): 53 | """Compute the jaccard overlap of two sets of boxes. The jaccard overlap 54 | is simply the intersection over union of two boxes. Here we operate on 55 | ground truth boxes and default boxes. 56 | E.g.: 57 | A ∩ B / A ∪ B = A ∩ B / (area(A) + area(B) - A ∩ B) 58 | Args: 59 | box_a: (tensor) Ground truth bounding boxes, Shape: [num_objects,4] 60 | box_b: (tensor) Prior boxes from priorbox layers, Shape: [num_priors,4] 61 | Return: 62 | jaccard overlap: (tensor) Shape: [box_a.size(0), box_b.size(0)] 63 | """ 64 | inter = intersect(box_a, box_b) 65 | area_a = ((box_a[:, 2] - box_a[:, 0]) * 66 | (box_a[:, 3] - box_a[:, 1])).unsqueeze(1).expand_as(inter) # [A,B] 67 | area_b = ((box_b[:, 2] - box_b[:, 0]) * 68 | (box_b[:, 3] - box_b[:, 1])).unsqueeze(0).expand_as(inter) # [A,B] 69 | union = area_a + area_b - inter 70 | return inter / union # [A,B] 71 | 72 | 73 | def matrix_iou(a, b): 74 | """ 75 | return iou of a and b, numpy version for data augenmentation 76 | """ 77 | lt = np.maximum(a[:, np.newaxis, :2], b[:, :2]) 78 | rb = np.minimum(a[:, np.newaxis, 2:], b[:, 2:]) 79 | 80 | area_i = np.prod(rb - lt, axis=2) * (lt < rb).all(axis=2) 81 | area_a = np.prod(a[:, 2:] - a[:, :2], axis=1) 82 | area_b = np.prod(b[:, 2:] - b[:, :2], axis=1) 83 | return area_i / (area_a[:, np.newaxis] + area_b - area_i) 84 | 85 | 86 | def matrix_iof(a, b): 87 | """ 88 | return iof of a and b, numpy version for data augenmentation 89 | """ 90 | lt = np.maximum(a[:, np.newaxis, :2], b[:, :2]) 91 | rb = np.minimum(a[:, np.newaxis, 2:], b[:, 2:]) 92 | 93 | area_i = np.prod(rb - lt, axis=2) * (lt < rb).all(axis=2) 94 | area_a = np.prod(a[:, 2:] - a[:, :2], axis=1) 95 | return area_i / np.maximum(area_a[:, np.newaxis], 1) 96 | 97 | 98 | def match(threshold, truths, priors, variances, labels, loc_t, conf_t, idx): 99 | """Match each prior box with the ground truth box of the highest jaccard 100 | overlap, encode the bounding boxes, then return the matched indices 101 | corresponding to both confidence and location preds. 102 | Args: 103 | threshold: (float) The overlap threshold used when mathing boxes. 104 | truths: (tensor) Ground truth boxes, Shape: [num_obj, num_priors]. 105 | priors: (tensor) Prior boxes from priorbox layers, Shape: [n_priors,4]. 106 | variances: (tensor) Variances corresponding to each prior coord, 107 | Shape: [num_priors, 4]. 108 | labels: (tensor) All the class labels for the image, Shape: [num_obj]. 109 | loc_t: (tensor) Tensor to be filled w/ endcoded location targets. 110 | conf_t: (tensor) Tensor to be filled w/ matched indices for conf preds. 111 | idx: (int) current batch index 112 | Return: 113 | The matched indices corresponding to 1)location and 2)confidence preds. 114 | """ 115 | # jaccard index 116 | overlaps = jaccard( 117 | truths, 118 | point_form(priors) 119 | ) 120 | # (Bipartite Matching) 121 | # [1,num_objects] best prior for each ground truth 122 | best_prior_overlap, best_prior_idx = overlaps.max(1, keepdim=True) 123 | 124 | # ignore hard gt 125 | valid_gt_idx = best_prior_overlap[:, 0] >= 0.2 126 | best_prior_idx_filter = best_prior_idx[valid_gt_idx, :] 127 | if best_prior_idx_filter.shape[0] <= 0: 128 | loc_t[idx] = 0 129 | conf_t[idx] = 0 130 | return 131 | 132 | # [1,num_priors] best ground truth for each prior 133 | best_truth_overlap, best_truth_idx = overlaps.max(0, keepdim=True) 134 | best_truth_idx.squeeze_(0) 135 | best_truth_overlap.squeeze_(0) 136 | best_prior_idx.squeeze_(1) 137 | best_prior_idx_filter.squeeze_(1) 138 | best_prior_overlap.squeeze_(1) 139 | best_truth_overlap.index_fill_(0, best_prior_idx_filter, 2) # ensure best prior 140 | # TODO refactor: index best_prior_idx with long tensor 141 | # ensure every gt matches with its prior of max overlap 142 | for j in range(best_prior_idx.size(0)): 143 | best_truth_idx[best_prior_idx[j]] = j 144 | matches = truths[best_truth_idx] # Shape: [num_priors,4] 145 | conf = labels[best_truth_idx] # Shape: [num_priors] 146 | conf[best_truth_overlap < threshold] = 0 # label as background 147 | loc = encode(matches, priors, variances) 148 | loc_t[idx] = loc # [num_priors,4] encoded offsets to learn 149 | conf_t[idx] = conf # [num_priors] top class label for each prior 150 | 151 | 152 | def encode(matched, priors, variances): 153 | """Encode the variances from the priorbox layers into the ground truth boxes 154 | we have matched (based on jaccard overlap) with the prior boxes. 155 | Args: 156 | matched: (tensor) Coords of ground truth for each prior in point-form 157 | Shape: [num_priors, 4]. 158 | priors: (tensor) Prior boxes in center-offset form 159 | Shape: [num_priors,4]. 160 | variances: (list[float]) Variances of priorboxes 161 | Return: 162 | encoded boxes (tensor), Shape: [num_priors, 4] 163 | """ 164 | 165 | # dist b/t match center and prior's center 166 | g_cxcy = (matched[:, :2] + matched[:, 2:]) / 2 - priors[:, :2] 167 | # encode variance 168 | g_cxcy /= (variances[0] * priors[:, 2:]) 169 | # match wh / prior wh 170 | g_wh = (matched[:, 2:] - matched[:, :2]) / priors[:, 2:] 171 | g_wh = torch.log(g_wh) / variances[1] 172 | # return target for smooth_l1_loss 173 | return torch.cat([g_cxcy, g_wh], 1) # [num_priors,4] 174 | 175 | 176 | # Adapted from https://github.com/Hakuyume/chainer-ssd 177 | def decode(loc, priors, variances): 178 | """Decode locations from predictions using priors to undo 179 | the encoding we did for offset regression at train time. 180 | Args: 181 | loc (tensor): location predictions for loc layers, 182 | Shape: [num_priors,4] 183 | priors (tensor): Prior boxes in center-offset form. 184 | Shape: [num_priors,4]. 185 | variances: (list[float]) Variances of priorboxes 186 | Return: 187 | decoded bounding box predictions 188 | """ 189 | 190 | boxes = torch.cat(( 191 | priors[:, :2] + loc[:, :2] * variances[0] * priors[:, 2:], 192 | priors[:, 2:] * torch.exp(loc[:, 2:] * variances[1])), 1) 193 | boxes[:, :2] -= boxes[:, 2:] / 2 194 | boxes[:, 2:] += boxes[:, :2] 195 | return boxes 196 | 197 | 198 | def log_sum_exp(x): 199 | """Utility function for computing log_sum_exp while determining 200 | This will be used to determine unaveraged confidence loss across 201 | all examples in a batch. 202 | Args: 203 | x (Variable(tensor)): conf_preds from conf layers 204 | """ 205 | x_max = x.data.max() 206 | return torch.log(torch.sum(torch.exp(x - x_max), 1, keepdim=True)) + x_max 207 | 208 | 209 | # Original author: Francisco Massa: 210 | # https://github.com/fmassa/object-detection.torch 211 | # Ported to PyTorch by Max deGroot (02/01/2017) 212 | def nms(boxes, scores, overlap=0.5, top_k=200): 213 | """Apply non-maximum suppression at test time to avoid detecting too many 214 | overlapping bounding boxes for a given object. 215 | Args: 216 | boxes: (tensor) The location preds for the img, Shape: [num_priors,4]. 217 | scores: (tensor) The class predscores for the img, Shape:[num_priors]. 218 | overlap: (float) The overlap thresh for suppressing unnecessary boxes. 219 | top_k: (int) The Maximum number of box preds to consider. 220 | Return: 221 | The indices of the kept boxes with respect to num_priors. 222 | """ 223 | 224 | keep = torch.Tensor(scores.size(0)).fill_(0).long() 225 | if boxes.numel() == 0: 226 | return keep 227 | x1 = boxes[:, 0] 228 | y1 = boxes[:, 1] 229 | x2 = boxes[:, 2] 230 | y2 = boxes[:, 3] 231 | area = torch.mul(x2 - x1, y2 - y1) 232 | v, idx = scores.sort(0) # sort in ascending order 233 | # I = I[v >= 0.01] 234 | idx = idx[-top_k:] # indices of the top-k largest vals 235 | xx1 = boxes.new() 236 | yy1 = boxes.new() 237 | xx2 = boxes.new() 238 | yy2 = boxes.new() 239 | w = boxes.new() 240 | h = boxes.new() 241 | 242 | # keep = torch.Tensor() 243 | count = 0 244 | while idx.numel() > 0: 245 | i = idx[-1] # index of current largest val 246 | # keep.append(i) 247 | keep[count] = i 248 | count += 1 249 | if idx.size(0) == 1: 250 | break 251 | idx = idx[:-1] # remove kept element from view 252 | # load bboxes of next highest vals 253 | torch.index_select(x1, 0, idx, out=xx1) 254 | torch.index_select(y1, 0, idx, out=yy1) 255 | torch.index_select(x2, 0, idx, out=xx2) 256 | torch.index_select(y2, 0, idx, out=yy2) 257 | # store element-wise max with next highest score 258 | xx1 = torch.clamp(xx1, min=x1[i]) 259 | yy1 = torch.clamp(yy1, min=y1[i]) 260 | xx2 = torch.clamp(xx2, max=x2[i]) 261 | yy2 = torch.clamp(yy2, max=y2[i]) 262 | w.resize_as_(xx2) 263 | h.resize_as_(yy2) 264 | w = xx2 - xx1 265 | h = yy2 - yy1 266 | # check sizes of xx1 and xx2.. after each iteration 267 | w = torch.clamp(w, min=0.0) 268 | h = torch.clamp(h, min=0.0) 269 | inter = w * h 270 | # IoU = i / (area(a) + area(b) - i) 271 | rem_areas = torch.index_select(area, 0, idx) # load remaining areas) 272 | union = (rem_areas - inter) + area[i] 273 | IoU = inter / union # store result in iou 274 | # keep only elements with an IoU <= overlap 275 | idx = idx[IoU.le(overlap)] 276 | return keep, count 277 | -------------------------------------------------------------------------------- /FaceBoxes/utils/build.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | # -------------------------------------------------------- 4 | # Fast R-CNN 5 | # Copyright (c) 2015 Microsoft 6 | # Licensed under The MIT License [see LICENSE for details] 7 | # Written by Ross Girshick 8 | # -------------------------------------------------------- 9 | 10 | import os 11 | from os.path import join as pjoin 12 | import numpy as np 13 | from distutils.core import setup 14 | from distutils.extension import Extension 15 | from Cython.Distutils import build_ext 16 | 17 | 18 | def find_in_path(name, path): 19 | "Find a file in a search path" 20 | # adapted fom http://code.activestate.com/recipes/52224-find-a-file-given-a-search-path/ 21 | for dir in path.split(os.pathsep): 22 | binpath = pjoin(dir, name) 23 | if os.path.exists(binpath): 24 | return os.path.abspath(binpath) 25 | return None 26 | 27 | 28 | # Obtain the numpy include directory. This logic works across numpy versions. 29 | try: 30 | numpy_include = np.get_include() 31 | except AttributeError: 32 | numpy_include = np.get_numpy_include() 33 | 34 | 35 | # run the customize_compiler 36 | class custom_build_ext(build_ext): 37 | def build_extensions(self): 38 | # customize_compiler_for_nvcc(self.compiler) 39 | build_ext.build_extensions(self) 40 | 41 | 42 | ext_modules = [ 43 | Extension( 44 | "nms.cpu_nms", 45 | ["nms/cpu_nms.pyx"], 46 | # extra_compile_args={'gcc': ["-Wno-cpp", "-Wno-unused-function"]}, 47 | extra_compile_args=["-Wno-cpp", "-Wno-unused-function"], 48 | include_dirs=[numpy_include] 49 | ) 50 | ] 51 | 52 | setup( 53 | name='mot_utils', 54 | ext_modules=ext_modules, 55 | # inject our custom trigger 56 | cmdclass={'build_ext': custom_build_ext}, 57 | ) 58 | -------------------------------------------------------------------------------- /FaceBoxes/utils/config.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | cfg = { 4 | 'name': 'FaceBoxes', 5 | 'min_sizes': [[32, 64, 128], [256], [512]], 6 | 'steps': [32, 64, 128], 7 | 'variance': [0.1, 0.2], 8 | 'clip': False 9 | } 10 | -------------------------------------------------------------------------------- /FaceBoxes/utils/functions.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | import sys 4 | import os.path as osp 5 | import torch 6 | 7 | def check_keys(model, pretrained_state_dict): 8 | ckpt_keys = set(pretrained_state_dict.keys()) 9 | model_keys = set(model.state_dict().keys()) 10 | used_pretrained_keys = model_keys & ckpt_keys 11 | unused_pretrained_keys = ckpt_keys - model_keys 12 | missing_keys = model_keys - ckpt_keys 13 | # print('Missing keys:{}'.format(len(missing_keys))) 14 | # print('Unused checkpoint keys:{}'.format(len(unused_pretrained_keys))) 15 | # print('Used keys:{}'.format(len(used_pretrained_keys))) 16 | assert len(used_pretrained_keys) > 0, 'load NONE from pretrained checkpoint' 17 | return True 18 | 19 | 20 | def remove_prefix(state_dict, prefix): 21 | ''' Old style model is stored with all names of parameters sharing common prefix 'module.' ''' 22 | # print('remove prefix \'{}\''.format(prefix)) 23 | f = lambda x: x.split(prefix, 1)[-1] if x.startswith(prefix) else x 24 | return {f(key): value for key, value in state_dict.items()} 25 | 26 | 27 | def load_model(model, pretrained_path, load_to_cpu): 28 | if not osp.isfile(pretrained_path): 29 | print(f'The pre-trained FaceBoxes model {pretrained_path} does not exist') 30 | sys.exit('-1') 31 | # print('Loading pretrained model from {}'.format(pretrained_path)) 32 | if load_to_cpu: 33 | pretrained_dict = torch.load(pretrained_path, map_location=lambda storage, loc: storage) 34 | else: 35 | device = torch.cuda.current_device() 36 | pretrained_dict = torch.load(pretrained_path, map_location=lambda storage, loc: storage.cuda(device)) 37 | if "state_dict" in pretrained_dict.keys(): 38 | pretrained_dict = remove_prefix(pretrained_dict['state_dict'], 'module.') 39 | else: 40 | pretrained_dict = remove_prefix(pretrained_dict, 'module.') 41 | check_keys(model, pretrained_dict) 42 | model.load_state_dict(pretrained_dict, strict=False) 43 | return model 44 | -------------------------------------------------------------------------------- /FaceBoxes/utils/nms/.gitignore: -------------------------------------------------------------------------------- 1 | *.c 2 | *.so 3 | -------------------------------------------------------------------------------- /FaceBoxes/utils/nms/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/FaceBoxes/utils/nms/__init__.py -------------------------------------------------------------------------------- /FaceBoxes/utils/nms/cpu_nms.pyx: -------------------------------------------------------------------------------- 1 | # -------------------------------------------------------- 2 | # Fast R-CNN 3 | # Copyright (c) 2015 Microsoft 4 | # Licensed under The MIT License [see LICENSE for details] 5 | # Written by Ross Girshick 6 | # -------------------------------------------------------- 7 | 8 | import numpy as np 9 | cimport numpy as np 10 | 11 | cdef inline np.float32_t max(np.float32_t a, np.float32_t b): 12 | return a if a >= b else b 13 | 14 | cdef inline np.float32_t min(np.float32_t a, np.float32_t b): 15 | return a if a <= b else b 16 | 17 | def cpu_nms(np.ndarray[np.float32_t, ndim=2] dets, np.float thresh): 18 | cdef np.ndarray[np.float32_t, ndim=1] x1 = dets[:, 0] 19 | cdef np.ndarray[np.float32_t, ndim=1] y1 = dets[:, 1] 20 | cdef np.ndarray[np.float32_t, ndim=1] x2 = dets[:, 2] 21 | cdef np.ndarray[np.float32_t, ndim=1] y2 = dets[:, 3] 22 | cdef np.ndarray[np.float32_t, ndim=1] scores = dets[:, 4] 23 | 24 | cdef np.ndarray[np.float32_t, ndim=1] areas = (x2 - x1 + 1) * (y2 - y1 + 1) 25 | cdef np.ndarray[np.int_t, ndim=1] order = scores.argsort()[::-1] 26 | 27 | cdef int ndets = dets.shape[0] 28 | cdef np.ndarray[np.int_t, ndim=1] suppressed = \ 29 | np.zeros((ndets), dtype=np.int) 30 | 31 | # nominal indices 32 | cdef int _i, _j 33 | # sorted indices 34 | cdef int i, j 35 | # temp variables for box i's (the box currently under consideration) 36 | cdef np.float32_t ix1, iy1, ix2, iy2, iarea 37 | # variables for computing overlap with box j (lower scoring box) 38 | cdef np.float32_t xx1, yy1, xx2, yy2 39 | cdef np.float32_t w, h 40 | cdef np.float32_t inter, ovr 41 | 42 | keep = [] 43 | for _i in range(ndets): 44 | i = order[_i] 45 | if suppressed[i] == 1: 46 | continue 47 | keep.append(i) 48 | ix1 = x1[i] 49 | iy1 = y1[i] 50 | ix2 = x2[i] 51 | iy2 = y2[i] 52 | iarea = areas[i] 53 | for _j in range(_i + 1, ndets): 54 | j = order[_j] 55 | if suppressed[j] == 1: 56 | continue 57 | xx1 = max(ix1, x1[j]) 58 | yy1 = max(iy1, y1[j]) 59 | xx2 = min(ix2, x2[j]) 60 | yy2 = min(iy2, y2[j]) 61 | w = max(0.0, xx2 - xx1 + 1) 62 | h = max(0.0, yy2 - yy1 + 1) 63 | inter = w * h 64 | ovr = inter / (iarea + areas[j] - inter) 65 | if ovr >= thresh: 66 | suppressed[j] = 1 67 | 68 | return keep 69 | 70 | def cpu_soft_nms(np.ndarray[float, ndim=2] boxes, float sigma=0.5, float Nt=0.3, float threshold=0.001, unsigned int method=0): 71 | cdef unsigned int N = boxes.shape[0] 72 | cdef float iw, ih, box_area 73 | cdef float ua 74 | cdef int pos = 0 75 | cdef float maxscore = 0 76 | cdef int maxpos = 0 77 | cdef float x1,x2,y1,y2,tx1,tx2,ty1,ty2,ts,area,weight,ov 78 | 79 | for i in range(N): 80 | maxscore = boxes[i, 4] 81 | maxpos = i 82 | 83 | tx1 = boxes[i,0] 84 | ty1 = boxes[i,1] 85 | tx2 = boxes[i,2] 86 | ty2 = boxes[i,3] 87 | ts = boxes[i,4] 88 | 89 | pos = i + 1 90 | # get max box 91 | while pos < N: 92 | if maxscore < boxes[pos, 4]: 93 | maxscore = boxes[pos, 4] 94 | maxpos = pos 95 | pos = pos + 1 96 | 97 | # add max box as a detection 98 | boxes[i,0] = boxes[maxpos,0] 99 | boxes[i,1] = boxes[maxpos,1] 100 | boxes[i,2] = boxes[maxpos,2] 101 | boxes[i,3] = boxes[maxpos,3] 102 | boxes[i,4] = boxes[maxpos,4] 103 | 104 | # swap ith box with position of max box 105 | boxes[maxpos,0] = tx1 106 | boxes[maxpos,1] = ty1 107 | boxes[maxpos,2] = tx2 108 | boxes[maxpos,3] = ty2 109 | boxes[maxpos,4] = ts 110 | 111 | tx1 = boxes[i,0] 112 | ty1 = boxes[i,1] 113 | tx2 = boxes[i,2] 114 | ty2 = boxes[i,3] 115 | ts = boxes[i,4] 116 | 117 | pos = i + 1 118 | # NMS iterations, note that N changes if detection boxes fall below threshold 119 | while pos < N: 120 | x1 = boxes[pos, 0] 121 | y1 = boxes[pos, 1] 122 | x2 = boxes[pos, 2] 123 | y2 = boxes[pos, 3] 124 | s = boxes[pos, 4] 125 | 126 | area = (x2 - x1 + 1) * (y2 - y1 + 1) 127 | iw = (min(tx2, x2) - max(tx1, x1) + 1) 128 | if iw > 0: 129 | ih = (min(ty2, y2) - max(ty1, y1) + 1) 130 | if ih > 0: 131 | ua = float((tx2 - tx1 + 1) * (ty2 - ty1 + 1) + area - iw * ih) 132 | ov = iw * ih / ua #iou between max box and detection box 133 | 134 | if method == 1: # linear 135 | if ov > Nt: 136 | weight = 1 - ov 137 | else: 138 | weight = 1 139 | elif method == 2: # gaussian 140 | weight = np.exp(-(ov * ov)/sigma) 141 | else: # original NMS 142 | if ov > Nt: 143 | weight = 0 144 | else: 145 | weight = 1 146 | 147 | boxes[pos, 4] = weight*boxes[pos, 4] 148 | 149 | # if box score falls below threshold, discard the box by swapping with last box 150 | # update N 151 | if boxes[pos, 4] < threshold: 152 | boxes[pos,0] = boxes[N-1, 0] 153 | boxes[pos,1] = boxes[N-1, 1] 154 | boxes[pos,2] = boxes[N-1, 2] 155 | boxes[pos,3] = boxes[N-1, 3] 156 | boxes[pos,4] = boxes[N-1, 4] 157 | N = N - 1 158 | pos = pos - 1 159 | 160 | pos = pos + 1 161 | 162 | keep = [i for i in range(N)] 163 | return keep 164 | -------------------------------------------------------------------------------- /FaceBoxes/utils/nms/py_cpu_nms.py: -------------------------------------------------------------------------------- 1 | # -------------------------------------------------------- 2 | # Fast R-CNN 3 | # Copyright (c) 2015 Microsoft 4 | # Licensed under The MIT License [see LICENSE for details] 5 | # Written by Ross Girshick 6 | # -------------------------------------------------------- 7 | 8 | import numpy as np 9 | 10 | def py_cpu_nms(dets, thresh): 11 | """Pure Python NMS baseline.""" 12 | x1 = dets[:, 0] 13 | y1 = dets[:, 1] 14 | x2 = dets[:, 2] 15 | y2 = dets[:, 3] 16 | scores = dets[:, 4] 17 | 18 | areas = (x2 - x1 + 1) * (y2 - y1 + 1) 19 | order = scores.argsort()[::-1] 20 | 21 | keep = [] 22 | while order.size > 0: 23 | i = order[0] 24 | keep.append(i) 25 | xx1 = np.maximum(x1[i], x1[order[1:]]) 26 | yy1 = np.maximum(y1[i], y1[order[1:]]) 27 | xx2 = np.minimum(x2[i], x2[order[1:]]) 28 | yy2 = np.minimum(y2[i], y2[order[1:]]) 29 | 30 | w = np.maximum(0.0, xx2 - xx1 + 1) 31 | h = np.maximum(0.0, yy2 - yy1 + 1) 32 | inter = w * h 33 | ovr = inter / (areas[i] + areas[order[1:]] - inter) 34 | 35 | inds = np.where(ovr <= thresh)[0] 36 | order = order[inds + 1] 37 | 38 | return keep 39 | -------------------------------------------------------------------------------- /FaceBoxes/utils/nms_wrapper.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | # -------------------------------------------------------- 4 | # Fast R-CNN 5 | # Copyright (c) 2015 Microsoft 6 | # Licensed under The MIT License [see LICENSE for details] 7 | # Written by Ross Girshick 8 | # -------------------------------------------------------- 9 | 10 | from .nms.cpu_nms import cpu_nms, cpu_soft_nms 11 | 12 | 13 | def nms(dets, thresh): 14 | """Dispatch to either CPU or GPU NMS implementations.""" 15 | 16 | if dets.shape[0] == 0: 17 | return [] 18 | return cpu_nms(dets, thresh) 19 | # return gpu_nms(dets, thresh) 20 | -------------------------------------------------------------------------------- /FaceBoxes/utils/prior_box.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | from .config import cfg 4 | 5 | import torch 6 | from itertools import product as product 7 | from math import ceil 8 | 9 | 10 | class PriorBox(object): 11 | def __init__(self, image_size=None): 12 | super(PriorBox, self).__init__() 13 | # self.aspect_ratios = cfg['aspect_ratios'] 14 | self.min_sizes = cfg['min_sizes'] 15 | self.steps = cfg['steps'] 16 | self.clip = cfg['clip'] 17 | self.image_size = image_size 18 | self.feature_maps = [[ceil(self.image_size[0] / step), ceil(self.image_size[1] / step)] for step in self.steps] 19 | 20 | def forward(self): 21 | anchors = [] 22 | for k, f in enumerate(self.feature_maps): 23 | min_sizes = self.min_sizes[k] 24 | for i, j in product(range(f[0]), range(f[1])): 25 | for min_size in min_sizes: 26 | s_kx = min_size / self.image_size[1] 27 | s_ky = min_size / self.image_size[0] 28 | if min_size == 32: 29 | dense_cx = [x * self.steps[k] / self.image_size[1] for x in 30 | [j + 0, j + 0.25, j + 0.5, j + 0.75]] 31 | dense_cy = [y * self.steps[k] / self.image_size[0] for y in 32 | [i + 0, i + 0.25, i + 0.5, i + 0.75]] 33 | for cy, cx in product(dense_cy, dense_cx): 34 | anchors += [cx, cy, s_kx, s_ky] 35 | elif min_size == 64: 36 | dense_cx = [x * self.steps[k] / self.image_size[1] for x in [j + 0, j + 0.5]] 37 | dense_cy = [y * self.steps[k] / self.image_size[0] for y in [i + 0, i + 0.5]] 38 | for cy, cx in product(dense_cy, dense_cx): 39 | anchors += [cx, cy, s_kx, s_ky] 40 | else: 41 | cx = (j + 0.5) * self.steps[k] / self.image_size[1] 42 | cy = (i + 0.5) * self.steps[k] / self.image_size[0] 43 | anchors += [cx, cy, s_kx, s_ky] 44 | # back to torch land 45 | output = torch.Tensor(anchors).view(-1, 4) 46 | if self.clip: 47 | output.clamp_(max=1, min=0) 48 | return output 49 | -------------------------------------------------------------------------------- /FaceBoxes/utils/timer.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | # -------------------------------------------------------- 4 | # Fast R-CNN 5 | # Copyright (c) 2015 Microsoft 6 | # Licensed under The MIT License [see LICENSE for details] 7 | # Written by Ross Girshick 8 | # -------------------------------------------------------- 9 | 10 | import time 11 | 12 | 13 | class Timer(object): 14 | """A simple timer.""" 15 | 16 | def __init__(self): 17 | self.total_time = 0. 18 | self.calls = 0 19 | self.start_time = 0. 20 | self.diff = 0. 21 | self.average_time = 0. 22 | 23 | def tic(self): 24 | # using time.time instead of time.clock because time time.clock 25 | # does not normalize for multithreading 26 | self.start_time = time.time() 27 | 28 | def toc(self, average=True): 29 | self.diff = time.time() - self.start_time 30 | self.total_time += self.diff 31 | self.calls += 1 32 | self.average_time = self.total_time / self.calls 33 | if average: 34 | return self.average_time 35 | else: 36 | return self.diff 37 | 38 | def clear(self): 39 | self.total_time = 0. 40 | self.calls = 0 41 | self.start_time = 0. 42 | self.diff = 0. 43 | self.average_time = 0. 44 | -------------------------------------------------------------------------------- /FaceBoxes/weights/.gitignore: -------------------------------------------------------------------------------- 1 | *.onnx 2 | -------------------------------------------------------------------------------- /FaceBoxes/weights/FaceBoxesProd.pth: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/FaceBoxes/weights/FaceBoxesProd.pth -------------------------------------------------------------------------------- /FaceBoxes/weights/readme.md: -------------------------------------------------------------------------------- 1 | The pre-trained model `FaceBoxesProd.pth` is downloaded from [Google Drive](https://drive.google.com/file/d/1tRVwOlu0QtjvADQ2H7vqrRwsWEmaqioI). 2 | 3 | The converted `FaceBoxesProd.onnx`: [Google Drive](https://drive.google.com/file/d/1pccQOvYqKh3iCEHc5tSWx2-1fhgxs6rh/view?usp=sharing) or [Baidu Drive](https://pan.baidu.com/s/1TJS2wFRLSoWZPR4l9E7G7w) (Password: 9hph) 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Max deGroot, Ellis Brown 4 | Copyright (c) 2019 Zisian Wong, Shifeng Zhang 5 | Copyright (c) 2020 Jianzhu Guo, in Center for Biometrics and Security Research (CBSR) 6 | 7 | Permission is hereby granted, free of charge, to any person obtaining a copy 8 | of this software and associated documentation files (the "Software"), to deal 9 | in the Software without restriction, including without limitation the rights 10 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 11 | copies of the Software, and to permit persons to whom the Software is 12 | furnished to do so, subject to the following conditions: 13 | 14 | The above copyright notice and this permission notice shall be included in all 15 | copies or substantial portions of the Software. 16 | 17 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 18 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 19 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 20 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 21 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 22 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 23 | SOFTWARE. -------------------------------------------------------------------------------- /Sim3DR/.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | cmake-build-debug/ 3 | .idea/ 4 | build/ 5 | *.so 6 | data/ 7 | 8 | lib/rasterize.cpp -------------------------------------------------------------------------------- /Sim3DR/Sim3DR.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | from . import _init_paths 4 | import numpy as np 5 | import Sim3DR_Cython 6 | 7 | 8 | def get_normal(vertices, triangles): 9 | normal = np.zeros_like(vertices, dtype=np.float32) 10 | Sim3DR_Cython.get_normal(normal, vertices, triangles, vertices.shape[0], triangles.shape[0]) 11 | return normal 12 | 13 | 14 | def rasterize(vertices, triangles, colors, bg=None, 15 | height=None, width=None, channel=None, 16 | reverse=False): 17 | if bg is not None: 18 | height, width, channel = bg.shape 19 | else: 20 | assert height is not None and width is not None and channel is not None 21 | bg = np.zeros((height, width, channel), dtype=np.uint8) 22 | 23 | buffer = np.zeros((height, width), dtype=np.float32) - 1e8 24 | 25 | if colors.dtype != np.float32: 26 | colors = colors.astype(np.float32) 27 | Sim3DR_Cython.rasterize(bg, vertices, triangles, colors, buffer, triangles.shape[0], height, width, channel, 28 | reverse=reverse) 29 | return bg 30 | -------------------------------------------------------------------------------- /Sim3DR/__init__.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | from .Sim3DR import get_normal, rasterize 4 | from .lighting import RenderPipeline 5 | -------------------------------------------------------------------------------- /Sim3DR/_init_paths.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | import os.path as osp 4 | import sys 5 | 6 | 7 | def add_path(path): 8 | if path not in sys.path: 9 | sys.path.insert(0, path) 10 | 11 | 12 | this_dir = osp.dirname(__file__) 13 | lib_path = osp.join(this_dir, '.') 14 | add_path(lib_path) 15 | -------------------------------------------------------------------------------- /Sim3DR/build_sim3dr.sh: -------------------------------------------------------------------------------- 1 | python3 setup.py build_ext --inplace -------------------------------------------------------------------------------- /Sim3DR/lib/rasterize.h: -------------------------------------------------------------------------------- 1 | #ifndef MESH_CORE_HPP_ 2 | #define MESH_CORE_HPP_ 3 | 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | 11 | using namespace std; 12 | 13 | class Point3D { 14 | public: 15 | float x; 16 | float y; 17 | float z; 18 | 19 | public: 20 | Point3D() : x(0.f), y(0.f), z(0.f) {} 21 | Point3D(float x_, float y_, float z_) : x(x_), y(y_), z(z_) {} 22 | 23 | void initialize(float x_, float y_, float z_){ 24 | this->x = x_; this->y = y_; this->z = z_; 25 | } 26 | 27 | Point3D cross(Point3D &p){ 28 | Point3D c; 29 | c.x = this->y * p.z - this->z * p.y; 30 | c.y = this->z * p.x - this->x * p.z; 31 | c.z = this->x * p.y - this->y * p.x; 32 | return c; 33 | } 34 | 35 | float dot(Point3D &p) { 36 | return this->x * p.x + this->y * p.y + this->z * p.z; 37 | } 38 | 39 | Point3D operator-(const Point3D &p) { 40 | Point3D np; 41 | np.x = this->x - p.x; 42 | np.y = this->y - p.y; 43 | np.z = this->z - p.z; 44 | return np; 45 | } 46 | 47 | }; 48 | 49 | class Point { 50 | public: 51 | float x; 52 | float y; 53 | 54 | public: 55 | Point() : x(0.f), y(0.f) {} 56 | Point(float x_, float y_) : x(x_), y(y_) {} 57 | float dot(Point p) { 58 | return this->x * p.x + this->y * p.y; 59 | } 60 | 61 | Point operator-(const Point &p) { 62 | Point np; 63 | np.x = this->x - p.x; 64 | np.y = this->y - p.y; 65 | return np; 66 | } 67 | 68 | Point operator+(const Point &p) { 69 | Point np; 70 | np.x = this->x + p.x; 71 | np.y = this->y + p.y; 72 | return np; 73 | } 74 | 75 | Point operator*(float s) { 76 | Point np; 77 | np.x = s * this->x; 78 | np.y = s * this->y; 79 | return np; 80 | } 81 | }; 82 | 83 | 84 | bool is_point_in_tri(Point p, Point p0, Point p1, Point p2); 85 | 86 | void get_point_weight(float *weight, Point p, Point p0, Point p1, Point p2); 87 | 88 | void _get_tri_normal(float *tri_normal, float *vertices, int *triangles, int ntri, bool norm_flg); 89 | 90 | void _get_ver_normal(float *ver_normal, float *tri_normal, int *triangles, int nver, int ntri); 91 | 92 | void _get_normal(float *ver_normal, float *vertices, int *triangles, int nver, int ntri); 93 | 94 | void _rasterize_triangles( 95 | float *vertices, int *triangles, float *depth_buffer, int *triangle_buffer, float *barycentric_weight, 96 | int ntri, int h, int w); 97 | 98 | void _rasterize( 99 | unsigned char *image, float *vertices, int *triangles, float *colors, 100 | float *depth_buffer, int ntri, int h, int w, int c, float alpha, bool reverse); 101 | 102 | void _render_texture_core( 103 | float *image, float *vertices, int *triangles, 104 | float *texture, float *tex_coords, int *tex_triangles, 105 | float *depth_buffer, 106 | int nver, int tex_nver, int ntri, 107 | int h, int w, int c, 108 | int tex_h, int tex_w, int tex_c, 109 | int mapping_type); 110 | 111 | void _write_obj_with_colors_texture(string filename, string mtl_name, 112 | float *vertices, int *triangles, float *colors, float *uv_coords, 113 | int nver, int ntri, int ntexver); 114 | 115 | #endif -------------------------------------------------------------------------------- /Sim3DR/lib/rasterize.pyx: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | cimport numpy as np 3 | # from libcpp.string cimport string 4 | cimport cython 5 | from libcpp cimport bool 6 | 7 | # from cpython import bool 8 | 9 | # use the Numpy-C-API from Cython 10 | np.import_array() 11 | 12 | # cdefine the signature of our c function 13 | cdef extern from "rasterize.h": 14 | void _rasterize_triangles( 15 | float*vertices, int*triangles, float*depth_buffer, int*triangle_buffer, float*barycentric_weight, 16 | int ntri, int h, int w 17 | ) 18 | 19 | void _rasterize( 20 | unsigned char*image, float*vertices, int*triangles, float*colors, float*depth_buffer, 21 | int ntri, int h, int w, int c, float alpha, bool reverse 22 | ) 23 | 24 | # void _render_texture_core( 25 | # float* image, float* vertices, int* triangles, 26 | # float* texture, float* tex_coords, int* tex_triangles, 27 | # float* depth_buffer, 28 | # int nver, int tex_nver, int ntri, 29 | # int h, int w, int c, 30 | # int tex_h, int tex_w, int tex_c, 31 | # int mapping_type) 32 | 33 | void _get_tri_normal(float *tri_normal, float *vertices, int *triangles, int nver, bool norm_flg) 34 | void _get_ver_normal(float *ver_normal, float*tri_normal, int*triangles, int nver, int ntri) 35 | void _get_normal(float *ver_normal, float *vertices, int *triangles, int nver, int ntri) 36 | 37 | 38 | # void _write_obj_with_colors_texture(string filename, string mtl_name, 39 | # float* vertices, int* triangles, float* colors, float* uv_coords, 40 | # int nver, int ntri, int ntexver) 41 | 42 | @cython.boundscheck(False) 43 | @cython.wraparound(False) 44 | def get_tri_normal(np.ndarray[float, ndim=2, mode="c"] tri_normal not None, 45 | np.ndarray[float, ndim=2, mode = "c"] vertices not None, 46 | np.ndarray[int, ndim=2, mode="c"] triangles not None, 47 | int ntri, bool norm_flg = False): 48 | _get_tri_normal( np.PyArray_DATA(tri_normal), np.PyArray_DATA(vertices), 49 | np.PyArray_DATA(triangles), ntri, norm_flg) 50 | 51 | @cython.boundscheck(False) # turn off bounds-checking for entire function 52 | @cython.wraparound(False) # turn off negative index wrapping for entire function 53 | def get_ver_normal(np.ndarray[float, ndim=2, mode = "c"] ver_normal not None, 54 | np.ndarray[float, ndim=2, mode = "c"] tri_normal not None, 55 | np.ndarray[int, ndim=2, mode="c"] triangles not None, 56 | int nver, int ntri): 57 | _get_ver_normal( 58 | np.PyArray_DATA(ver_normal), np.PyArray_DATA(tri_normal), np.PyArray_DATA(triangles), 59 | nver, ntri) 60 | 61 | @cython.boundscheck(False) # turn off bounds-checking for entire function 62 | @cython.wraparound(False) # turn off negative index wrapping for entire function 63 | def get_normal(np.ndarray[float, ndim=2, mode = "c"] ver_normal not None, 64 | np.ndarray[float, ndim=2, mode = "c"] vertices not None, 65 | np.ndarray[int, ndim=2, mode="c"] triangles not None, 66 | int nver, int ntri): 67 | _get_normal( 68 | np.PyArray_DATA(ver_normal), np.PyArray_DATA(vertices), np.PyArray_DATA(triangles), 69 | nver, ntri) 70 | 71 | 72 | @cython.boundscheck(False) # turn off bounds-checking for entire function 73 | @cython.wraparound(False) # turn off negative index wrapping for entire function 74 | def rasterize_triangles( 75 | np.ndarray[float, ndim=2, mode = "c"] vertices not None, 76 | np.ndarray[int, ndim=2, mode="c"] triangles not None, 77 | np.ndarray[float, ndim=2, mode = "c"] depth_buffer not None, 78 | np.ndarray[int, ndim=2, mode = "c"] triangle_buffer not None, 79 | np.ndarray[float, ndim=2, mode = "c"] barycentric_weight not None, 80 | int ntri, int h, int w 81 | ): 82 | _rasterize_triangles( 83 | np.PyArray_DATA(vertices), np.PyArray_DATA(triangles), 84 | np.PyArray_DATA(depth_buffer), np.PyArray_DATA(triangle_buffer), 85 | np.PyArray_DATA(barycentric_weight), 86 | ntri, h, w) 87 | 88 | @cython.boundscheck(False) # turn off bounds-checking for entire function 89 | @cython.wraparound(False) # turn off negative index wrapping for entire function 90 | def rasterize(np.ndarray[unsigned char, ndim=3, mode = "c"] image not None, 91 | np.ndarray[float, ndim=2, mode = "c"] vertices not None, 92 | np.ndarray[int, ndim=2, mode="c"] triangles not None, 93 | np.ndarray[float, ndim=2, mode = "c"] colors not None, 94 | np.ndarray[float, ndim=2, mode = "c"] depth_buffer not None, 95 | int ntri, int h, int w, int c, float alpha = 1, bool reverse = False 96 | ): 97 | _rasterize( 98 | np.PyArray_DATA(image), np.PyArray_DATA(vertices), 99 | np.PyArray_DATA(triangles), 100 | np.PyArray_DATA(colors), 101 | np.PyArray_DATA(depth_buffer), 102 | ntri, h, w, c, alpha, reverse) 103 | 104 | # def render_texture_core(np.ndarray[float, ndim=3, mode = "c"] image not None, 105 | # np.ndarray[float, ndim=2, mode = "c"] vertices not None, 106 | # np.ndarray[int, ndim=2, mode="c"] triangles not None, 107 | # np.ndarray[float, ndim=3, mode = "c"] texture not None, 108 | # np.ndarray[float, ndim=2, mode = "c"] tex_coords not None, 109 | # np.ndarray[int, ndim=2, mode="c"] tex_triangles not None, 110 | # np.ndarray[float, ndim=2, mode = "c"] depth_buffer not None, 111 | # int nver, int tex_nver, int ntri, 112 | # int h, int w, int c, 113 | # int tex_h, int tex_w, int tex_c, 114 | # int mapping_type 115 | # ): 116 | # _render_texture_core( 117 | # np.PyArray_DATA(image), np.PyArray_DATA(vertices), np.PyArray_DATA(triangles), 118 | # np.PyArray_DATA(texture), np.PyArray_DATA(tex_coords), np.PyArray_DATA(tex_triangles), 119 | # np.PyArray_DATA(depth_buffer), 120 | # nver, tex_nver, ntri, 121 | # h, w, c, 122 | # tex_h, tex_w, tex_c, 123 | # mapping_type) 124 | # 125 | # def write_obj_with_colors_texture_core(string filename, string mtl_name, 126 | # np.ndarray[float, ndim=2, mode = "c"] vertices not None, 127 | # np.ndarray[int, ndim=2, mode="c"] triangles not None, 128 | # np.ndarray[float, ndim=2, mode = "c"] colors not None, 129 | # np.ndarray[float, ndim=2, mode = "c"] uv_coords not None, 130 | # int nver, int ntri, int ntexver 131 | # ): 132 | # _write_obj_with_colors_texture(filename, mtl_name, 133 | # np.PyArray_DATA(vertices), np.PyArray_DATA(triangles), np.PyArray_DATA(colors), np.PyArray_DATA(uv_coords), 134 | # nver, ntri, ntexver) 135 | -------------------------------------------------------------------------------- /Sim3DR/lighting.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | import numpy as np 4 | from .Sim3DR import get_normal, rasterize 5 | 6 | _norm = lambda arr: arr / np.sqrt(np.sum(arr ** 2, axis=1))[:, None] 7 | 8 | 9 | def norm_vertices(vertices): 10 | vertices -= vertices.min(0)[None, :] 11 | vertices /= vertices.max() 12 | vertices *= 2 13 | vertices -= vertices.max(0)[None, :] / 2 14 | return vertices 15 | 16 | 17 | def convert_type(obj): 18 | if isinstance(obj, tuple) or isinstance(obj, list): 19 | return np.array(obj, dtype=np.float32)[None, :] 20 | return obj 21 | 22 | 23 | class RenderPipeline(object): 24 | def __init__(self, **kwargs): 25 | self.intensity_ambient = convert_type(kwargs.get('intensity_ambient', 0.3)) 26 | self.intensity_directional = convert_type(kwargs.get('intensity_directional', 0.6)) 27 | self.intensity_specular = convert_type(kwargs.get('intensity_specular', 0.1)) 28 | self.specular_exp = kwargs.get('specular_exp', 5) 29 | self.color_ambient = convert_type(kwargs.get('color_ambient', (1, 1, 1))) 30 | self.color_directional = convert_type(kwargs.get('color_directional', (1, 1, 1))) 31 | self.light_pos = convert_type(kwargs.get('light_pos', (0, 0, 5))) 32 | self.view_pos = convert_type(kwargs.get('view_pos', (0, 0, 5))) 33 | 34 | def update_light_pos(self, light_pos): 35 | self.light_pos = convert_type(light_pos) 36 | 37 | def __call__(self, vertices, triangles, bg, texture=None): 38 | normal = get_normal(vertices, triangles) 39 | 40 | # 2. lighting 41 | light = np.zeros_like(vertices, dtype=np.float32) 42 | # ambient component 43 | if self.intensity_ambient > 0: 44 | light += self.intensity_ambient * self.color_ambient 45 | 46 | vertices_n = norm_vertices(vertices.copy()) 47 | if self.intensity_directional > 0: 48 | # diffuse component 49 | direction = _norm(self.light_pos - vertices_n) 50 | cos = np.sum(normal * direction, axis=1)[:, None] 51 | # cos = np.clip(cos, 0, 1) 52 | # todo: check below 53 | light += self.intensity_directional * (self.color_directional * np.clip(cos, 0, 1)) 54 | 55 | # specular component 56 | if self.intensity_specular > 0: 57 | v2v = _norm(self.view_pos - vertices_n) 58 | reflection = 2 * cos * normal - direction 59 | spe = np.sum((v2v * reflection) ** self.specular_exp, axis=1)[:, None] 60 | spe = np.where(cos != 0, np.clip(spe, 0, 1), np.zeros_like(spe)) 61 | light += self.intensity_specular * self.color_directional * np.clip(spe, 0, 1) 62 | light = np.clip(light, 0, 1) 63 | 64 | # 2. rasterization, [0, 1] 65 | if texture is None: 66 | render_img = rasterize(vertices, triangles, light, bg=bg) 67 | return render_img 68 | else: 69 | texture *= light 70 | render_img = rasterize(vertices, triangles, texture, bg=bg) 71 | return render_img 72 | 73 | 74 | def main(): 75 | pass 76 | 77 | 78 | if __name__ == '__main__': 79 | main() 80 | -------------------------------------------------------------------------------- /Sim3DR/readme.md: -------------------------------------------------------------------------------- 1 | ## Sim3DR 2 | This is a simple 3D render, written by c++ and cython. 3 | 4 | ### Build Sim3DR 5 | 6 | ```shell script 7 | python3 setup.py build_ext --inplace 8 | ``` -------------------------------------------------------------------------------- /Sim3DR/setup.py: -------------------------------------------------------------------------------- 1 | ''' 2 | python setup.py build_ext -i 3 | to compile 4 | ''' 5 | 6 | from distutils.core import setup, Extension 7 | from Cython.Build import cythonize 8 | from Cython.Distutils import build_ext 9 | import numpy 10 | 11 | setup( 12 | name='Sim3DR_Cython', # not the package name 13 | cmdclass={'build_ext': build_ext}, 14 | ext_modules=[Extension("Sim3DR_Cython", 15 | sources=["lib/rasterize.pyx", "lib/rasterize_kernel.cpp"], 16 | language='c++', 17 | include_dirs=[numpy.get_include()], 18 | extra_compile_args=["-std=c++11"])], 19 | ) 20 | -------------------------------------------------------------------------------- /Sim3DR/tests/.gitignore: -------------------------------------------------------------------------------- 1 | build/ 2 | -------------------------------------------------------------------------------- /Sim3DR/tests/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 2.8) 2 | 3 | set(TARGET test) 4 | project(${TARGET}) 5 | 6 | #find_package( OpenCV REQUIRED ) 7 | #include_directories( ${OpenCV_INCLUDE_DIRS} ) 8 | 9 | #set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fPIC -O3") 10 | include_directories(../lib) 11 | set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fPIC -std=c++11") 12 | add_executable(${TARGET} test.cpp ../lib/rasterize_kernel.cpp io.cpp) 13 | target_include_directories(${TARGET} PRIVATE ${PROJECT_SOURCE_DIR}) 14 | -------------------------------------------------------------------------------- /Sim3DR/tests/io.cpp: -------------------------------------------------------------------------------- 1 | #include "io.h" 2 | 3 | //void load_obj(const string obj_fp, float* vertices, float* colors, float* triangles){ 4 | // string line; 5 | // ifstream in(obj_fp); 6 | // 7 | // if(in.is_open()){ 8 | // while (getline(in, line)){ 9 | // stringstream ss(line); 10 | // 11 | // char t; // type: v, f 12 | // ss >> t; 13 | // if (t == 'v'){ 14 | // 15 | // } 16 | // } 17 | // } 18 | //} 19 | 20 | void load_obj(const char *obj_fp, float *vertices, float *colors, int *triangles, int nver, int ntri) { 21 | FILE *fp; 22 | fp = fopen(obj_fp, "r"); 23 | 24 | char t; // type: v or f 25 | if (fp != nullptr) { 26 | for (int i = 0; i < nver; ++i) { 27 | fscanf(fp, "%c", &t); 28 | for (int j = 0; j < 3; ++j) 29 | fscanf(fp, " %f", &vertices[3 * i + j]); 30 | for (int j = 0; j < 3; ++j) 31 | fscanf(fp, " %f", &colors[3 * i + j]); 32 | fscanf(fp, "\n"); 33 | } 34 | // fscanf(fp, "%c", &t); 35 | for (int i = 0; i < ntri; ++i) { 36 | fscanf(fp, "%c", &t); 37 | for (int j = 0; j < 3; ++j) { 38 | fscanf(fp, " %d", &triangles[3 * i + j]); 39 | triangles[3 * i + j] -= 1; 40 | } 41 | fscanf(fp, "\n"); 42 | } 43 | 44 | fclose(fp); 45 | } 46 | } 47 | 48 | void load_ply(const char *ply_fp, float *vertices, int *triangles, int nver, int ntri) { 49 | FILE *fp; 50 | fp = fopen(ply_fp, "r"); 51 | 52 | // char s[256]; 53 | char t; 54 | if (fp != nullptr) { 55 | // for (int i = 0; i < 9; ++i) 56 | // fscanf(fp, "%s", s); 57 | for (int i = 0; i < nver; ++i) 58 | fscanf(fp, "%f %f %f\n", &vertices[3 * i], &vertices[3 * i + 1], &vertices[3 * i + 2]); 59 | 60 | for (int i = 0; i < ntri; ++i) 61 | fscanf(fp, "%c %d %d %d\n", &t, &triangles[3 * i], &triangles[3 * i + 1], &triangles[3 * i + 2]); 62 | 63 | fclose(fp); 64 | } 65 | } 66 | 67 | void write_ppm(const char *filename, unsigned char *img, int h, int w, int c) { 68 | FILE *fp; 69 | //open file for output 70 | fp = fopen(filename, "wb"); 71 | if (!fp) { 72 | fprintf(stderr, "Unable to open file '%s'\n", filename); 73 | exit(1); 74 | } 75 | 76 | //write the header file 77 | //image format 78 | fprintf(fp, "P6\n"); 79 | 80 | //image size 81 | fprintf(fp, "%d %d\n", w, h); 82 | 83 | // rgb component depth 84 | fprintf(fp, "%d\n", MAX_PXL_VALUE); 85 | 86 | // pixel data 87 | fwrite(img, sizeof(unsigned char), size_t(h * w * c), fp); 88 | fclose(fp); 89 | } -------------------------------------------------------------------------------- /Sim3DR/tests/io.h: -------------------------------------------------------------------------------- 1 | #ifndef IO_H_ 2 | #define IO_H_ 3 | 4 | #include 5 | #include 6 | #include 7 | #include 8 | #include 9 | 10 | using namespace std; 11 | 12 | #define MAX_PXL_VALUE 255 13 | 14 | void load_obj(const char* obj_fp, float* vertices, float* colors, int* triangles, int nver, int ntri); 15 | void load_ply(const char* ply_fp, float* vertices, int* triangles, int nver, int ntri); 16 | 17 | 18 | void write_ppm(const char *filename, unsigned char *img, int h, int w, int c); 19 | 20 | #endif -------------------------------------------------------------------------------- /Sim3DR/tests/test.cpp: -------------------------------------------------------------------------------- 1 | /* 2 | * Tesing cases 3 | */ 4 | 5 | #include 6 | #include 7 | #include "rasterize.h" 8 | #include "io.h" 9 | 10 | void test_isPointInTri() { 11 | Point p0(0, 0); 12 | Point p1(1, 0); 13 | Point p2(1, 1); 14 | 15 | Point p(0.2, 0.2); 16 | 17 | if (is_point_in_tri(p, p0, p1, p2)) 18 | std::cout << "In"; 19 | else 20 | std::cout << "Out"; 21 | std::cout << std::endl; 22 | } 23 | 24 | void test_getPointWeight() { 25 | Point p0(0, 0); 26 | Point p1(1, 0); 27 | Point p2(1, 1); 28 | 29 | Point p(0.2, 0.2); 30 | 31 | float weight[3]; 32 | get_point_weight(weight, p, p0, p1, p2); 33 | std::cout << weight[0] << " " << weight[1] << " " << weight[2] << std::endl; 34 | } 35 | 36 | void test_get_tri_normal() { 37 | float tri_normal[3]; 38 | // float vertices[9] = {1, 0, 0, 0, 0, 0, 0, 1, 0}; 39 | float vertices[9] = {1, 1.1, 0, 0, 0, 0, 0, 0.6, 0.7}; 40 | int triangles[3] = {0, 1, 2}; 41 | int ntri = 1; 42 | 43 | _get_tri_normal(tri_normal, vertices, triangles, ntri); 44 | 45 | for (int i = 0; i < 3; ++i) 46 | std::cout << tri_normal[i] << ", "; 47 | std::cout << std::endl; 48 | } 49 | 50 | void test_load_obj() { 51 | const char *fp = "../data/vd005_mesh.obj"; 52 | int nver = 35709; 53 | int ntri = 70789; 54 | 55 | auto *vertices = new float[nver]; 56 | auto *colors = new float[nver]; 57 | auto *triangles = new int[ntri]; 58 | load_obj(fp, vertices, colors, triangles, nver, ntri); 59 | 60 | delete[] vertices; 61 | delete[] colors; 62 | delete[] triangles; 63 | } 64 | 65 | void test_render() { 66 | // 1. loading obj 67 | // const char *fp = "/Users/gjz/gjzprojects/Sim3DR/data/vd005_mesh.obj"; 68 | const char *fp = "/Users/gjz/gjzprojects/Sim3DR/data/face1.obj"; 69 | int nver = 35709; //53215; //35709; 70 | int ntri = 70789; //105840;//70789; 71 | 72 | auto *vertices = new float[3 * nver]; 73 | auto *colors = new float[3 * nver]; 74 | auto *triangles = new int[3 * ntri]; 75 | load_obj(fp, vertices, colors, triangles, nver, ntri); 76 | 77 | // 2. rendering 78 | int h = 224, w = 224, c = 3; 79 | 80 | // enlarging 81 | int scale = 4; 82 | h *= scale; 83 | w *= scale; 84 | for (int i = 0; i < nver * 3; ++i) vertices[i] *= scale; 85 | 86 | auto *image = new unsigned char[h * w * c](); 87 | auto *depth_buffer = new float[h * w](); 88 | 89 | for (int i = 0; i < h * w; ++i) depth_buffer[i] = -999999; 90 | 91 | clock_t t; 92 | t = clock(); 93 | 94 | _rasterize(image, vertices, triangles, colors, depth_buffer, ntri, h, w, c, true); 95 | t = clock() - t; 96 | double time_taken = ((double) t) / CLOCKS_PER_SEC; // in seconds 97 | printf("Render took %f seconds to execute \n", time_taken); 98 | 99 | 100 | // auto *image_char = new u_char[h * w * c](); 101 | // for (int i = 0; i < h * w * c; ++i) 102 | // image_char[i] = u_char(255 * image[i]); 103 | write_ppm("res.ppm", image, h, w, c); 104 | 105 | // delete[] image_char; 106 | delete[] vertices; 107 | delete[] colors; 108 | delete[] triangles; 109 | delete[] image; 110 | delete[] depth_buffer; 111 | } 112 | 113 | void test_light() { 114 | // 1. loading obj 115 | const char *fp = "/Users/gjz/gjzprojects/Sim3DR/data/emma_input_0_noheader.ply"; 116 | int nver = 53215; //35709; 117 | int ntri = 105840; //70789; 118 | 119 | auto *vertices = new float[3 * nver]; 120 | auto *colors = new float[3 * nver]; 121 | auto *triangles = new int[3 * ntri]; 122 | load_ply(fp, vertices, triangles, nver, ntri); 123 | 124 | // 2. rendering 125 | // int h = 1901, w = 3913, c = 3; 126 | int h = 2000, w = 4000, c = 3; 127 | 128 | // enlarging 129 | // int scale = 1; 130 | // h *= scale; 131 | // w *= scale; 132 | // for (int i = 0; i < nver * 3; ++i) vertices[i] *= scale; 133 | 134 | auto *image = new unsigned char[h * w * c](); 135 | auto *depth_buffer = new float[h * w](); 136 | 137 | for (int i = 0; i < h * w; ++i) depth_buffer[i] = -999999; 138 | for (int i = 0; i < 3 * nver; ++i) colors[i] = 0.8; 139 | 140 | clock_t t; 141 | t = clock(); 142 | 143 | _rasterize(image, vertices, triangles, colors, depth_buffer, ntri, h, w, c, true); 144 | t = clock() - t; 145 | double time_taken = ((double) t) / CLOCKS_PER_SEC; // in seconds 146 | printf("Render took %f seconds to execute \n", time_taken); 147 | 148 | 149 | // auto *image_char = new u_char[h * w * c](); 150 | // for (int i = 0; i < h * w * c; ++i) 151 | // image_char[i] = u_char(255 * image[i]); 152 | write_ppm("emma.ppm", image, h, w, c); 153 | 154 | // delete[] image_char; 155 | delete[] vertices; 156 | delete[] colors; 157 | delete[] triangles; 158 | delete[] image; 159 | delete[] depth_buffer; 160 | } 161 | 162 | int main(int argc, char *argv[]) { 163 | // std::cout << "Hello CMake!" << std::endl; 164 | 165 | // test_isPointInTri(); 166 | // test_getPointWeight(); 167 | // test_get_tri_normal(); 168 | // test_load_obj(); 169 | // test_render(); 170 | test_light(); 171 | return 0; 172 | } -------------------------------------------------------------------------------- /TDDFA.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import os.path as osp 6 | import time 7 | import numpy as np 8 | import cv2 9 | import torch 10 | from torchvision.transforms import Compose 11 | import torch.backends.cudnn as cudnn 12 | 13 | import models 14 | from bfm import BFMModel 15 | from utils.io import _load 16 | from utils.functions import ( 17 | crop_img, parse_roi_box_from_bbox, parse_roi_box_from_landmark, 18 | ) 19 | from utils.tddfa_util import ( 20 | load_model, _parse_param, similar_transform, 21 | ToTensorGjz, NormalizeGjz 22 | ) 23 | 24 | make_abs_path = lambda fn: osp.join(osp.dirname(osp.realpath(__file__)), fn) 25 | 26 | 27 | class TDDFA(object): 28 | """TDDFA: named Three-D Dense Face Alignment (TDDFA)""" 29 | 30 | def __init__(self, **kvs): 31 | torch.set_grad_enabled(False) 32 | 33 | # load BFM 34 | self.bfm = BFMModel( 35 | bfm_fp=kvs.get('bfm_fp', make_abs_path('configs/bfm_noneck_v3.pkl')), 36 | shape_dim=kvs.get('shape_dim', 40), 37 | exp_dim=kvs.get('exp_dim', 10) 38 | ) 39 | self.tri = self.bfm.tri 40 | 41 | # config 42 | self.gpu_mode = kvs.get('gpu_mode', False) 43 | self.gpu_id = kvs.get('gpu_id', 0) 44 | self.size = kvs.get('size', 120) 45 | 46 | param_mean_std_fp = kvs.get( 47 | 'param_mean_std_fp', make_abs_path(f'configs/param_mean_std_62d_{self.size}x{self.size}.pkl') 48 | ) 49 | 50 | # load model, default output is dimension with length 62 = 12(pose) + 40(shape) +10(expression) 51 | model = getattr(models, kvs.get('arch'))( 52 | num_classes=kvs.get('num_params', 62), 53 | widen_factor=kvs.get('widen_factor', 1), 54 | size=self.size, 55 | mode=kvs.get('mode', 'small') 56 | ) 57 | model = load_model(model, kvs.get('checkpoint_fp')) 58 | 59 | if self.gpu_mode: 60 | cudnn.benchmark = True 61 | model = model.cuda(device=self.gpu_id) 62 | 63 | self.model = model 64 | self.model.eval() # eval mode, fix BN 65 | 66 | # data normalization 67 | transform_normalize = NormalizeGjz(mean=127.5, std=128) 68 | transform_to_tensor = ToTensorGjz() 69 | transform = Compose([transform_to_tensor, transform_normalize]) 70 | self.transform = transform 71 | 72 | # params normalization config 73 | r = _load(param_mean_std_fp) 74 | self.param_mean = r.get('mean') 75 | self.param_std = r.get('std') 76 | 77 | # print('param_mean and param_srd', self.param_mean, self.param_std) 78 | 79 | def __call__(self, img_ori, objs, **kvs): 80 | """The main call of TDDFA, given image and box / landmark, return 3DMM params and roi_box 81 | :param img_ori: the input image 82 | :param objs: the list of box or landmarks 83 | :param kvs: options 84 | :return: param list and roi_box list 85 | """ 86 | # Crop image, forward to get the param 87 | param_lst = [] 88 | roi_box_lst = [] 89 | 90 | crop_policy = kvs.get('crop_policy', 'box') 91 | for obj in objs: 92 | if crop_policy == 'box': 93 | # by face box 94 | roi_box = parse_roi_box_from_bbox(obj) 95 | elif crop_policy == 'landmark': 96 | # by landmarks 97 | roi_box = parse_roi_box_from_landmark(obj) 98 | else: 99 | raise ValueError(f'Unknown crop policy {crop_policy}') 100 | 101 | roi_box_lst.append(roi_box) 102 | img = crop_img(img_ori, roi_box) 103 | img = cv2.resize(img, dsize=(self.size, self.size), interpolation=cv2.INTER_LINEAR) 104 | inp = self.transform(img).unsqueeze(0) 105 | 106 | if self.gpu_mode: 107 | inp = inp.cuda(device=self.gpu_id) 108 | 109 | if kvs.get('timer_flag', False): 110 | end = time.time() 111 | param = self.model(inp) 112 | elapse = f'Inference: {(time.time() - end) * 1000:.1f}ms' 113 | print(elapse) 114 | else: 115 | param = self.model(inp) 116 | 117 | param = param.squeeze().cpu().numpy().flatten().astype(np.float32) 118 | param = param * self.param_std + self.param_mean # re-scale 119 | # print('output', param) 120 | param_lst.append(param) 121 | 122 | return param_lst, roi_box_lst 123 | 124 | def recon_vers(self, param_lst, roi_box_lst, **kvs): 125 | dense_flag = kvs.get('dense_flag', False) 126 | size = self.size 127 | 128 | ver_lst = [] 129 | for param, roi_box in zip(param_lst, roi_box_lst): 130 | if dense_flag: 131 | R, offset, alpha_shp, alpha_exp = _parse_param(param) 132 | pts3d = R @ (self.bfm.u + self.bfm.w_shp @ alpha_shp + self.bfm.w_exp @ alpha_exp). \ 133 | reshape(3, -1, order='F') + offset 134 | pts3d = similar_transform(pts3d, roi_box, size) 135 | else: 136 | R, offset, alpha_shp, alpha_exp = _parse_param(param) 137 | pts3d = R @ (self.bfm.u_base + self.bfm.w_shp_base @ alpha_shp + self.bfm.w_exp_base @ alpha_exp). \ 138 | reshape(3, -1, order='F') + offset 139 | pts3d = similar_transform(pts3d, roi_box, size) 140 | 141 | ver_lst.append(pts3d) 142 | 143 | return ver_lst 144 | -------------------------------------------------------------------------------- /TDDFA_ONNX.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import os.path as osp 6 | import numpy as np 7 | import cv2 8 | import onnxruntime 9 | 10 | from utils.onnx import convert_to_onnx 11 | from utils.io import _load 12 | from utils.functions import ( 13 | crop_img, parse_roi_box_from_bbox, parse_roi_box_from_landmark, 14 | ) 15 | from utils.tddfa_util import _parse_param, similar_transform 16 | from bfm.bfm import BFMModel 17 | from bfm.bfm_onnx import convert_bfm_to_onnx 18 | 19 | make_abs_path = lambda fn: osp.join(osp.dirname(osp.realpath(__file__)), fn) 20 | 21 | 22 | class TDDFA_ONNX(object): 23 | """TDDFA_ONNX: the ONNX version of Three-D Dense Face Alignment (TDDFA)""" 24 | 25 | def __init__(self, **kvs): 26 | # torch.set_grad_enabled(False) 27 | 28 | # load onnx version of BFM 29 | bfm_fp = kvs.get('bfm_fp', make_abs_path('configs/bfm_noneck_v3.pkl')) 30 | bfm_onnx_fp = bfm_fp.replace('.pkl', '.onnx') 31 | if not osp.exists(bfm_onnx_fp): 32 | convert_bfm_to_onnx( 33 | bfm_onnx_fp, 34 | shape_dim=kvs.get('shape_dim', 40), 35 | exp_dim=kvs.get('exp_dim', 10) 36 | ) 37 | self.bfm_session = onnxruntime.InferenceSession(bfm_onnx_fp, None) 38 | 39 | # load for optimization 40 | bfm = BFMModel(bfm_fp, shape_dim=kvs.get('shape_dim', 40), exp_dim=kvs.get('exp_dim', 10)) 41 | self.tri = bfm.tri 42 | self.u_base, self.w_shp_base, self.w_exp_base = bfm.u_base, bfm.w_shp_base, bfm.w_exp_base 43 | 44 | # config 45 | self.gpu_mode = kvs.get('gpu_mode', False) 46 | self.gpu_id = kvs.get('gpu_id', 0) 47 | self.size = kvs.get('size', 120) 48 | 49 | param_mean_std_fp = kvs.get( 50 | 'param_mean_std_fp', make_abs_path(f'configs/param_mean_std_62d_{self.size}x{self.size}.pkl') 51 | ) 52 | 53 | onnx_fp = kvs.get('onnx_fp', kvs.get('checkpoint_fp').replace('.pth', '.onnx')) 54 | 55 | # convert to onnx online if not existed 56 | if onnx_fp is None or not osp.exists(onnx_fp): 57 | print(f'{onnx_fp} does not exist, try to convert the `.pth` version to `.onnx` online') 58 | onnx_fp = convert_to_onnx(**kvs) 59 | 60 | self.session = onnxruntime.InferenceSession(onnx_fp, None) 61 | 62 | # params normalization config 63 | r = _load(param_mean_std_fp) 64 | self.param_mean = r.get('mean') 65 | self.param_std = r.get('std') 66 | 67 | def __call__(self, img_ori, objs, **kvs): 68 | # Crop image, forward to get the param 69 | param_lst = [] 70 | roi_box_lst = [] 71 | 72 | crop_policy = kvs.get('crop_policy', 'box') 73 | for obj in objs: 74 | if crop_policy == 'box': 75 | # by face box 76 | roi_box = parse_roi_box_from_bbox(obj) 77 | elif crop_policy == 'landmark': 78 | # by landmarks 79 | roi_box = parse_roi_box_from_landmark(obj) 80 | else: 81 | raise ValueError(f'Unknown crop policy {crop_policy}') 82 | 83 | roi_box_lst.append(roi_box) 84 | img = crop_img(img_ori, roi_box) 85 | img = cv2.resize(img, dsize=(self.size, self.size), interpolation=cv2.INTER_LINEAR) 86 | img = img.astype(np.float32).transpose(2, 0, 1)[np.newaxis, ...] 87 | img = (img - 127.5) / 128. 88 | 89 | inp_dct = {'input': img} 90 | 91 | param = self.session.run(None, inp_dct)[0] 92 | param = param.flatten().astype(np.float32) 93 | param = param * self.param_std + self.param_mean # re-scale 94 | param_lst.append(param) 95 | 96 | return param_lst, roi_box_lst 97 | 98 | def recon_vers(self, param_lst, roi_box_lst, **kvs): 99 | dense_flag = kvs.get('dense_flag', False) 100 | size = self.size 101 | 102 | ver_lst = [] 103 | for param, roi_box in zip(param_lst, roi_box_lst): 104 | R, offset, alpha_shp, alpha_exp = _parse_param(param) 105 | if dense_flag: 106 | inp_dct = { 107 | 'R': R, 'offset': offset, 'alpha_shp': alpha_shp, 'alpha_exp': alpha_exp 108 | } 109 | pts3d = self.bfm_session.run(None, inp_dct)[0] 110 | pts3d = similar_transform(pts3d, roi_box, size) 111 | else: 112 | pts3d = R @ (self.u_base + self.w_shp_base @ alpha_shp + self.w_exp_base @ alpha_exp). \ 113 | reshape(3, -1, order='F') + offset 114 | pts3d = similar_transform(pts3d, roi_box, size) 115 | 116 | ver_lst.append(pts3d) 117 | 118 | return ver_lst 119 | -------------------------------------------------------------------------------- /bfm/.gitignore: -------------------------------------------------------------------------------- 1 | *.ply 2 | -------------------------------------------------------------------------------- /bfm/__init__.py: -------------------------------------------------------------------------------- 1 | from .bfm import BFMModel -------------------------------------------------------------------------------- /bfm/bfm.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import sys 6 | 7 | sys.path.append('..') 8 | 9 | import os.path as osp 10 | import numpy as np 11 | from utils.io import _load 12 | 13 | make_abs_path = lambda fn: osp.join(osp.dirname(osp.realpath(__file__)), fn) 14 | 15 | 16 | def _to_ctype(arr): 17 | if not arr.flags.c_contiguous: 18 | return arr.copy(order='C') 19 | return arr 20 | 21 | 22 | class BFMModel(object): 23 | def __init__(self, bfm_fp, shape_dim=40, exp_dim=10): 24 | bfm = _load(bfm_fp) 25 | self.u = bfm.get('u').astype(np.float32) # fix bug 26 | self.w_shp = bfm.get('w_shp').astype(np.float32)[..., :shape_dim] 27 | self.w_exp = bfm.get('w_exp').astype(np.float32)[..., :exp_dim] 28 | if osp.split(bfm_fp)[-1] == 'bfm_noneck_v3.pkl': 29 | self.tri = _load(make_abs_path('../configs/tri.pkl')) # this tri/face is re-built for bfm_noneck_v3 30 | else: 31 | self.tri = bfm.get('tri') 32 | 33 | self.tri = _to_ctype(self.tri.T).astype(np.int32) 34 | self.keypoints = bfm.get('keypoints').astype(np.long) # fix bug 35 | w = np.concatenate((self.w_shp, self.w_exp), axis=1) 36 | self.w_norm = np.linalg.norm(w, axis=0) 37 | 38 | self.u_base = self.u[self.keypoints].reshape(-1, 1) 39 | self.w_shp_base = self.w_shp[self.keypoints] 40 | self.w_exp_base = self.w_exp[self.keypoints] 41 | -------------------------------------------------------------------------------- /bfm/bfm_onnx.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import sys 6 | 7 | sys.path.append('..') 8 | 9 | import os.path as osp 10 | import numpy as np 11 | import torch 12 | import torch.nn as nn 13 | 14 | from utils.io import _load, _numpy_to_cuda, _numpy_to_tensor 15 | 16 | make_abs_path = lambda fn: osp.join(osp.dirname(osp.realpath(__file__)), fn) 17 | 18 | 19 | def _to_ctype(arr): 20 | if not arr.flags.c_contiguous: 21 | return arr.copy(order='C') 22 | return arr 23 | 24 | 25 | def _load_tri(bfm_fp): 26 | if osp.split(bfm_fp)[-1] == 'bfm_noneck_v3.pkl': 27 | tri = _load(make_abs_path('../configs/tri.pkl')) # this tri/face is re-built for bfm_noneck_v3 28 | else: 29 | tri = _load(bfm_fp).get('tri') 30 | 31 | tri = _to_ctype(tri.T).astype(np.int32) 32 | return tri 33 | 34 | 35 | class BFMModel_ONNX(nn.Module): 36 | """BFM serves as a decoder""" 37 | 38 | def __init__(self, bfm_fp, shape_dim=40, exp_dim=10): 39 | super(BFMModel_ONNX, self).__init__() 40 | 41 | _to_tensor = _numpy_to_tensor 42 | 43 | # load bfm 44 | bfm = _load(bfm_fp) 45 | 46 | u = _to_tensor(bfm.get('u').astype(np.float32)) 47 | self.u = u.view(-1, 3).transpose(1, 0) 48 | w_shp = _to_tensor(bfm.get('w_shp').astype(np.float32)[..., :shape_dim]) 49 | w_exp = _to_tensor(bfm.get('w_exp').astype(np.float32)[..., :exp_dim]) 50 | w = torch.cat((w_shp, w_exp), dim=1) 51 | self.w = w.view(-1, 3, w.shape[-1]).contiguous().permute(1, 0, 2) 52 | 53 | # self.u = _to_tensor(bfm.get('u').astype(np.float32)) # fix bug 54 | # w_shp = _to_tensor(bfm.get('w_shp').astype(np.float32)[..., :shape_dim]) 55 | # w_exp = _to_tensor(bfm.get('w_exp').astype(np.float32)[..., :exp_dim]) 56 | # self.w = torch.cat((w_shp, w_exp), dim=1) 57 | 58 | # self.keypoints = bfm.get('keypoints').astype(np.long) # fix bug 59 | # self.u_base = self.u[self.keypoints].reshape(-1, 1) 60 | # self.w_shp_base = self.w_shp[self.keypoints] 61 | # self.w_exp_base = self.w_exp[self.keypoints] 62 | 63 | def forward(self, *inps): 64 | R, offset, alpha_shp, alpha_exp = inps 65 | alpha = torch.cat((alpha_shp, alpha_exp)) 66 | # pts3d = R @ (self.u + self.w_shp.matmul(alpha_shp) + self.w_exp.matmul(alpha_exp)). \ 67 | # view(-1, 3).transpose(1, 0) + offset 68 | # pts3d = R @ (self.u + self.w.matmul(alpha)).view(-1, 3).transpose(1, 0) + offset 69 | pts3d = R @ (self.u + self.w.matmul(alpha).squeeze()) + offset 70 | return pts3d 71 | 72 | 73 | def convert_bfm_to_onnx(bfm_onnx_fp, shape_dim=40, exp_dim=10): 74 | # print(shape_dim, exp_dim) 75 | bfm_fp = bfm_onnx_fp.replace('.onnx', '.pkl') 76 | bfm_decoder = BFMModel_ONNX(bfm_fp=bfm_fp, shape_dim=shape_dim, exp_dim=exp_dim) 77 | bfm_decoder.eval() 78 | 79 | # dummy_input = torch.randn(12 + shape_dim + exp_dim) 80 | dummy_input = torch.randn(3, 3), torch.randn(3, 1), torch.randn(shape_dim, 1), torch.randn(exp_dim, 1) 81 | R, offset, alpha_shp, alpha_exp = dummy_input 82 | torch.onnx.export( 83 | bfm_decoder, 84 | (R, offset, alpha_shp, alpha_exp), 85 | bfm_onnx_fp, 86 | input_names=['R', 'offset', 'alpha_shp', 'alpha_exp'], 87 | output_names=['output'], 88 | dynamic_axes={ 89 | 'alpha_shp': [0], 90 | 'alpha_exp': [0], 91 | }, 92 | do_constant_folding=True 93 | ) 94 | print(f'Convert {bfm_fp} to {bfm_onnx_fp} done.') 95 | 96 | 97 | if __name__ == '__main__': 98 | convert_bfm_to_onnx('../configs/bfm_noneck_v3.onnx') 99 | -------------------------------------------------------------------------------- /bfm/readme.md: -------------------------------------------------------------------------------- 1 | ## Statement 2 | 3 | The modified BFM2009 face model in `../configs/bfm_noneck_v3.pkl` is only for academic use. 4 | For commercial use, you need to apply for the commercial license, some refs are below: 5 | 6 | [1] https://faces.dmi.unibas.ch/bfm/?nav=1-0&id=basel_face_model 7 | 8 | [2] https://faces.dmi.unibas.ch/bfm/bfm2019.html 9 | 10 | If your work benefits from this repo, please cite 11 | 12 | @PROCEEDINGS{bfm09, 13 | title={A 3D Face Model for Pose and Illumination Invariant Face Recognition}, 14 | author={P. Paysan and R. Knothe and B. Amberg 15 | and S. Romdhani and T. Vetter}, 16 | journal={Proceedings of the 6th IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS) 17 | for Security, Safety and Monitoring in Smart Environments}, 18 | organization={IEEE}, 19 | year={2009}, 20 | address = {Genova, Italy}, 21 | } 22 | 23 | -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | cd FaceBoxes 2 | sh ./build_cpu_nms.sh 3 | cd .. 4 | 5 | cd Sim3DR 6 | sh ./build_sim3dr.sh 7 | cd .. 8 | 9 | cd utils/asset 10 | gcc -shared -Wall -O3 render.c -o render.so -fPIC 11 | cd ../.. -------------------------------------------------------------------------------- /configs/.gitignore: -------------------------------------------------------------------------------- 1 | *.pkl 2 | *.yml 3 | *.onnx -------------------------------------------------------------------------------- /configs/BFM_UV.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/configs/BFM_UV.mat -------------------------------------------------------------------------------- /configs/bfm_noneck_v3.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/configs/bfm_noneck_v3.pkl -------------------------------------------------------------------------------- /configs/indices.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/configs/indices.npy -------------------------------------------------------------------------------- /configs/mb05_120x120.yml: -------------------------------------------------------------------------------- 1 | arch: mobilenet # MobileNet V1 2 | widen_factor: 0.5 3 | checkpoint_fp: weights/mb05_120x120.pth 4 | bfm_fp: configs/bfm_noneck_v3.pkl # or configs/bfm_noneck_v3_slim.pkl 5 | size: 120 6 | num_params: 62 -------------------------------------------------------------------------------- /configs/mb1_120x120.yml: -------------------------------------------------------------------------------- 1 | arch: mobilenet # MobileNet V1 2 | widen_factor: 1.0 3 | checkpoint_fp: weights/mb1_120x120.pth 4 | bfm_fp: configs/bfm_noneck_v3.pkl # or configs/bfm_noneck_v3_slim.pkl 5 | size: 120 6 | num_params: 62 7 | -------------------------------------------------------------------------------- /configs/ncc_code.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/configs/ncc_code.npy -------------------------------------------------------------------------------- /configs/param_mean_std_62d_120x120.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/configs/param_mean_std_62d_120x120.pkl -------------------------------------------------------------------------------- /configs/readme.md: -------------------------------------------------------------------------------- 1 | ## The simplified version of BFM 2 | 3 | `bfm_noneck_v3_slim.pkl`: [Google Drive](https://drive.google.com/file/d/1iK5lD49E_gCn9voUjWDPj2ItGKvM10GI/view?usp=sharing) or [Baidu Drive](https://pan.baidu.com/s/1C_SzYBOG3swZA_EjxpXlAw) (Password: p803) -------------------------------------------------------------------------------- /configs/resnet_120x120.yml: -------------------------------------------------------------------------------- 1 | # before using this config, go through readme.md to find the onnx links to download `resnet22.onnx` 2 | arch: resnet22 3 | checkpoint_fp: weights/resnet22.pth 4 | bfm_fp: configs/bfm_noneck_v3.pkl 5 | size: 120 6 | num_params: 62 7 | -------------------------------------------------------------------------------- /configs/tri.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/configs/tri.pkl -------------------------------------------------------------------------------- /demo.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## A simple demostration of how to run" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": null, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "# before import, make sure FaceBoxes and Sim3DR are built successfully, e.g.,\n", 17 | "# sh build.sh\n", 18 | "\n", 19 | "import cv2\n", 20 | "import yaml\n", 21 | "\n", 22 | "from FaceBoxes import FaceBoxes\n", 23 | "from TDDFA import TDDFA\n", 24 | "from utils.functions import draw_landmarks\n", 25 | "from utils.render import render\n", 26 | "from utils.depth import depth\n", 27 | "\n", 28 | "import matplotlib.pyplot as plt" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "### Load configs" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": null, 41 | "metadata": {}, 42 | "outputs": [], 43 | "source": [ 44 | "# load config\n", 45 | "cfg = yaml.load(open('configs/mb1_120x120.yml'), Loader=yaml.SafeLoader)\n", 46 | "\n", 47 | "# Init FaceBoxes and TDDFA, recommend using onnx flag\n", 48 | "onnx_flag = True # or True to use ONNX to speed up\n", 49 | "if onnx_flag:\n", 50 | " import os\n", 51 | " os.environ['KMP_DUPLICATE_LIB_OK'] = 'True'\n", 52 | " os.environ['OMP_NUM_THREADS'] = '4'\n", 53 | " \n", 54 | " from FaceBoxes.FaceBoxes_ONNX import FaceBoxes_ONNX\n", 55 | " from TDDFA_ONNX import TDDFA_ONNX\n", 56 | " \n", 57 | " face_boxes = FaceBoxes_ONNX()\n", 58 | " tddfa = TDDFA_ONNX(**cfg)\n", 59 | "else:\n", 60 | " tddfa = TDDFA(gpu_mode=False, **cfg)\n", 61 | " face_boxes = FaceBoxes()" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "metadata": {}, 68 | "outputs": [], 69 | "source": [ 70 | "# given an image path\n", 71 | "img_fp = 'examples/inputs/emma.jpg'\n", 72 | "img = cv2.imread(img_fp)\n", 73 | "plt.imshow(img[..., ::-1])" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "### Detect faces using FaceBoxes" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": null, 86 | "metadata": {}, 87 | "outputs": [], 88 | "source": [ 89 | "# face detection\n", 90 | "boxes = face_boxes(img)\n", 91 | "print(f'Detect {len(boxes)} faces')\n", 92 | "print(boxes)" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "### Regressing 3DMM parameters, reconstruction and visualization" 100 | ] 101 | }, 102 | { 103 | "cell_type": "code", 104 | "execution_count": null, 105 | "metadata": {}, 106 | "outputs": [], 107 | "source": [ 108 | "# regress 3DMM params\n", 109 | "param_lst, roi_box_lst = tddfa(img, boxes)" 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": null, 115 | "metadata": {}, 116 | "outputs": [], 117 | "source": [ 118 | "# reconstruct vertices and visualizing sparse landmarks\n", 119 | "dense_flag = False\n", 120 | "ver_lst = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)\n", 121 | "draw_landmarks(img, ver_lst, dense_flag=dense_flag)" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": null, 127 | "metadata": {}, 128 | "outputs": [], 129 | "source": [ 130 | "# reconstruct vertices and visualizing dense landmarks\n", 131 | "dense_flag = True\n", 132 | "ver_lst = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)\n", 133 | "draw_landmarks(img, ver_lst, dense_flag=dense_flag)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": null, 139 | "metadata": {}, 140 | "outputs": [], 141 | "source": [ 142 | "# reconstruct vertices and render\n", 143 | "ver_lst = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)\n", 144 | "render(img, ver_lst, tddfa.tri, alpha=0.6, show_flag=True);" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": null, 150 | "metadata": {}, 151 | "outputs": [], 152 | "source": [ 153 | "# reconstruct vertices and render depth\n", 154 | "ver_lst = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)\n", 155 | "depth(img, ver_lst, tddfa.tri, show_flag=True);" 156 | ] 157 | } 158 | ], 159 | "metadata": { 160 | "kernelspec": { 161 | "display_name": "Python 3", 162 | "language": "python", 163 | "name": "python3" 164 | }, 165 | "language_info": { 166 | "codemirror_mode": { 167 | "name": "ipython", 168 | "version": 3 169 | }, 170 | "file_extension": ".py", 171 | "mimetype": "text/x-python", 172 | "name": "python", 173 | "nbconvert_exporter": "python", 174 | "pygments_lexer": "ipython3", 175 | "version": "3.8.5" 176 | } 177 | }, 178 | "nbformat": 4, 179 | "nbformat_minor": 4 180 | } 181 | -------------------------------------------------------------------------------- /demo.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import sys 6 | import argparse 7 | import cv2 8 | import yaml 9 | 10 | from FaceBoxes import FaceBoxes 11 | from TDDFA import TDDFA 12 | from utils.render import render 13 | #from utils.render_ctypes import render # faster 14 | from utils.depth import depth 15 | from utils.pncc import pncc 16 | from utils.uv import uv_tex 17 | from utils.pose import viz_pose 18 | from utils.serialization import ser_to_ply, ser_to_obj 19 | from utils.functions import draw_landmarks, get_suffix 20 | from utils.tddfa_util import str2bool 21 | 22 | 23 | def main(args): 24 | cfg = yaml.load(open(args.config), Loader=yaml.SafeLoader) 25 | 26 | # Init FaceBoxes and TDDFA, recommend using onnx flag 27 | if args.onnx: 28 | import os 29 | os.environ['KMP_DUPLICATE_LIB_OK'] = 'True' 30 | os.environ['OMP_NUM_THREADS'] = '4' 31 | 32 | from FaceBoxes.FaceBoxes_ONNX import FaceBoxes_ONNX 33 | from TDDFA_ONNX import TDDFA_ONNX 34 | 35 | face_boxes = FaceBoxes_ONNX() 36 | tddfa = TDDFA_ONNX(**cfg) 37 | else: 38 | gpu_mode = args.mode == 'gpu' 39 | tddfa = TDDFA(gpu_mode=gpu_mode, **cfg) 40 | face_boxes = FaceBoxes() 41 | 42 | # Given a still image path and load to BGR channel 43 | img = cv2.imread(args.img_fp) 44 | 45 | # Detect faces, get 3DMM params and roi boxes 46 | boxes = face_boxes(img) 47 | n = len(boxes) 48 | if n == 0: 49 | print(f'No face detected, exit') 50 | sys.exit(-1) 51 | print(f'Detect {n} faces') 52 | 53 | param_lst, roi_box_lst = tddfa(img, boxes) 54 | 55 | # Visualization and serialization 56 | dense_flag = args.opt in ('2d_dense', '3d', 'depth', 'pncc', 'uv_tex', 'ply', 'obj') 57 | old_suffix = get_suffix(args.img_fp) 58 | new_suffix = f'.{args.opt}' if args.opt in ('ply', 'obj') else '.jpg' 59 | 60 | wfp = f'examples/results/{args.img_fp.split("/")[-1].replace(old_suffix, "")}_{args.opt}' + new_suffix 61 | 62 | ver_lst = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag) 63 | 64 | if args.opt == '2d_sparse': 65 | draw_landmarks(img, ver_lst, show_flag=args.show_flag, dense_flag=dense_flag, wfp=wfp) 66 | elif args.opt == '2d_dense': 67 | draw_landmarks(img, ver_lst, show_flag=args.show_flag, dense_flag=dense_flag, wfp=wfp) 68 | elif args.opt == '3d': 69 | render(img, ver_lst, tddfa.tri, alpha=0.6, show_flag=args.show_flag, wfp=wfp) 70 | elif args.opt == 'depth': 71 | # if `with_bf_flag` is False, the background is black 72 | depth(img, ver_lst, tddfa.tri, show_flag=args.show_flag, wfp=wfp, with_bg_flag=True) 73 | elif args.opt == 'pncc': 74 | pncc(img, ver_lst, tddfa.tri, show_flag=args.show_flag, wfp=wfp, with_bg_flag=True) 75 | elif args.opt == 'uv_tex': 76 | uv_tex(img, ver_lst, tddfa.tri, show_flag=args.show_flag, wfp=wfp) 77 | elif args.opt == 'pose': 78 | viz_pose(img, param_lst, ver_lst, show_flag=args.show_flag, wfp=wfp) 79 | elif args.opt == 'ply': 80 | ser_to_ply(ver_lst, tddfa.tri, height=img.shape[0], wfp=wfp) 81 | elif args.opt == 'obj': 82 | ser_to_obj(img, ver_lst, tddfa.tri, height=img.shape[0], wfp=wfp) 83 | else: 84 | raise ValueError(f'Unknown opt {args.opt}') 85 | 86 | 87 | if __name__ == '__main__': 88 | parser = argparse.ArgumentParser(description='The demo of still image of 3DDFA_V2') 89 | parser.add_argument('-c', '--config', type=str, default='configs/mb1_120x120.yml') 90 | parser.add_argument('-f', '--img_fp', type=str, default='examples/inputs/trump_hillary.jpg') 91 | parser.add_argument('-m', '--mode', type=str, default='cpu', help='gpu or cpu mode') 92 | parser.add_argument('-o', '--opt', type=str, default='2d_sparse', 93 | choices=['2d_sparse', '2d_dense', '3d', 'depth', 'pncc', 'uv_tex', 'pose', 'ply', 'obj']) 94 | parser.add_argument('--show_flag', type=str2bool, default='true', help='whether to show the visualization result') 95 | parser.add_argument('--onnx', action='store_true', default=False) 96 | 97 | args = parser.parse_args() 98 | main(args) 99 | -------------------------------------------------------------------------------- /demo_video.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import argparse 6 | import imageio 7 | from tqdm import tqdm 8 | import yaml 9 | 10 | from FaceBoxes import FaceBoxes 11 | from TDDFA import TDDFA 12 | from utils.render import render 13 | # from utils.render_ctypes import render 14 | from utils.functions import cv_draw_landmark, get_suffix 15 | 16 | 17 | def main(args): 18 | cfg = yaml.load(open(args.config), Loader=yaml.SafeLoader) 19 | 20 | # Init FaceBoxes and TDDFA, recommend using onnx flag 21 | if args.onnx: 22 | import os 23 | os.environ['KMP_DUPLICATE_LIB_OK'] = 'True' 24 | os.environ['OMP_NUM_THREADS'] = '4' 25 | 26 | from FaceBoxes.FaceBoxes_ONNX import FaceBoxes_ONNX 27 | from TDDFA_ONNX import TDDFA_ONNX 28 | 29 | face_boxes = FaceBoxes_ONNX() 30 | tddfa = TDDFA_ONNX(**cfg) 31 | else: 32 | gpu_mode = args.mode == 'gpu' 33 | tddfa = TDDFA(gpu_mode=gpu_mode, **cfg) 34 | face_boxes = FaceBoxes() 35 | 36 | # Given a video path 37 | fn = args.video_fp.split('/')[-1] 38 | reader = imageio.get_reader(args.video_fp) 39 | 40 | fps = reader.get_meta_data()['fps'] 41 | 42 | suffix = get_suffix(args.video_fp) 43 | video_wfp = f'examples/results/videos/{fn.replace(suffix, "")}_{args.opt}.mp4' 44 | writer = imageio.get_writer(video_wfp, fps=fps) 45 | 46 | # run 47 | dense_flag = args.opt in ('3d',) 48 | pre_ver = None 49 | for i, frame in tqdm(enumerate(reader)): 50 | frame_bgr = frame[..., ::-1] # RGB->BGR 51 | 52 | if i == 0: 53 | # the first frame, detect face, here we only use the first face, you can change depending on your need 54 | boxes = face_boxes(frame_bgr) 55 | boxes = [boxes[0]] 56 | param_lst, roi_box_lst = tddfa(frame_bgr, boxes) 57 | ver = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)[0] 58 | 59 | # refine 60 | param_lst, roi_box_lst = tddfa(frame_bgr, [ver], crop_policy='landmark') 61 | ver = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)[0] 62 | else: 63 | param_lst, roi_box_lst = tddfa(frame_bgr, [pre_ver], crop_policy='landmark') 64 | 65 | roi_box = roi_box_lst[0] 66 | # todo: add confidence threshold to judge the tracking is failed 67 | if abs(roi_box[2] - roi_box[0]) * abs(roi_box[3] - roi_box[1]) < 2020: 68 | boxes = face_boxes(frame_bgr) 69 | boxes = [boxes[0]] 70 | param_lst, roi_box_lst = tddfa(frame_bgr, boxes) 71 | 72 | ver = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)[0] 73 | 74 | pre_ver = ver # for tracking 75 | 76 | if args.opt == '2d_sparse': 77 | res = cv_draw_landmark(frame_bgr, ver) 78 | elif args.opt == '3d': 79 | res = render(frame_bgr, [ver], tddfa.tri) 80 | else: 81 | raise ValueError(f'Unknown opt {args.opt}') 82 | 83 | writer.append_data(res[..., ::-1]) # BGR->RGB 84 | 85 | writer.close() 86 | print(f'Dump to {video_wfp}') 87 | 88 | 89 | if __name__ == '__main__': 90 | parser = argparse.ArgumentParser(description='The demo of video of 3DDFA_V2') 91 | parser.add_argument('-c', '--config', type=str, default='configs/mb1_120x120.yml') 92 | parser.add_argument('-f', '--video_fp', type=str) 93 | parser.add_argument('-m', '--mode', default='cpu', type=str, help='gpu or cpu mode') 94 | parser.add_argument('-o', '--opt', type=str, default='2d_sparse', choices=['2d_sparse', '3d']) 95 | parser.add_argument('--onnx', action='store_true', default=False) 96 | 97 | args = parser.parse_args() 98 | main(args) 99 | -------------------------------------------------------------------------------- /demo_video_smooth.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import argparse 6 | import imageio 7 | import numpy as np 8 | from tqdm import tqdm 9 | import yaml 10 | from collections import deque 11 | 12 | from FaceBoxes import FaceBoxes 13 | from TDDFA import TDDFA 14 | from utils.render import render 15 | # from utils.render_ctypes import render 16 | from utils.functions import cv_draw_landmark, get_suffix 17 | 18 | 19 | def main(args): 20 | cfg = yaml.load(open(args.config), Loader=yaml.SafeLoader) 21 | 22 | # Init FaceBoxes and TDDFA, recommend using onnx flag 23 | if args.onnx: 24 | import os 25 | os.environ['KMP_DUPLICATE_LIB_OK'] = 'True' 26 | os.environ['OMP_NUM_THREADS'] = '4' 27 | 28 | from FaceBoxes.FaceBoxes_ONNX import FaceBoxes_ONNX 29 | from TDDFA_ONNX import TDDFA_ONNX 30 | 31 | face_boxes = FaceBoxes_ONNX() 32 | tddfa = TDDFA_ONNX(**cfg) 33 | else: 34 | gpu_mode = args.mode == 'gpu' 35 | tddfa = TDDFA(gpu_mode=gpu_mode, **cfg) 36 | face_boxes = FaceBoxes() 37 | 38 | # Given a video path 39 | fn = args.video_fp.split('/')[-1] 40 | reader = imageio.get_reader(args.video_fp) 41 | 42 | fps = reader.get_meta_data()['fps'] 43 | suffix = get_suffix(args.video_fp) 44 | video_wfp = f'examples/results/videos/{fn.replace(suffix, "")}_{args.opt}_smooth.mp4' 45 | writer = imageio.get_writer(video_wfp, fps=fps) 46 | 47 | # the simple implementation of average smoothing by looking ahead by n_next frames 48 | # assert the frames of the video >= n 49 | n_pre, n_next = args.n_pre, args.n_next 50 | n = n_pre + n_next + 1 51 | queue_ver = deque() 52 | queue_frame = deque() 53 | 54 | # run 55 | dense_flag = args.opt in ('2d_dense', '3d',) 56 | pre_ver = None 57 | for i, frame in tqdm(enumerate(reader)): 58 | if args.start > 0 and i < args.start: 59 | continue 60 | if args.end > 0 and i > args.end: 61 | break 62 | 63 | frame_bgr = frame[..., ::-1] # RGB->BGR 64 | 65 | if i == 0: 66 | # detect 67 | boxes = face_boxes(frame_bgr) 68 | boxes = [boxes[0]] 69 | param_lst, roi_box_lst = tddfa(frame_bgr, boxes) 70 | ver = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)[0] 71 | 72 | # refine 73 | param_lst, roi_box_lst = tddfa(frame_bgr, [ver], crop_policy='landmark') 74 | ver = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)[0] 75 | 76 | # padding queue 77 | for _ in range(n_pre): 78 | queue_ver.append(ver.copy()) 79 | queue_ver.append(ver.copy()) 80 | 81 | for _ in range(n_pre): 82 | queue_frame.append(frame_bgr.copy()) 83 | queue_frame.append(frame_bgr.copy()) 84 | 85 | else: 86 | param_lst, roi_box_lst = tddfa(frame_bgr, [pre_ver], crop_policy='landmark') 87 | 88 | roi_box = roi_box_lst[0] 89 | # todo: add confidence threshold to judge the tracking is failed 90 | if abs(roi_box[2] - roi_box[0]) * abs(roi_box[3] - roi_box[1]) < 2020: 91 | boxes = face_boxes(frame_bgr) 92 | boxes = [boxes[0]] 93 | param_lst, roi_box_lst = tddfa(frame_bgr, boxes) 94 | 95 | ver = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)[0] 96 | 97 | queue_ver.append(ver.copy()) 98 | queue_frame.append(frame_bgr.copy()) 99 | 100 | pre_ver = ver # for tracking 101 | 102 | # smoothing: enqueue and dequeue ops 103 | if len(queue_ver) >= n: 104 | ver_ave = np.mean(queue_ver, axis=0) 105 | 106 | if args.opt == '2d_sparse': 107 | img_draw = cv_draw_landmark(queue_frame[n_pre], ver_ave) # since we use padding 108 | elif args.opt == '2d_dense': 109 | img_draw = cv_draw_landmark(queue_frame[n_pre], ver_ave, size=1) 110 | elif args.opt == '3d': 111 | img_draw = render(queue_frame[n_pre], [ver_ave], tddfa.tri, alpha=0.7) 112 | else: 113 | raise ValueError(f'Unknown opt {args.opt}') 114 | 115 | writer.append_data(img_draw[:, :, ::-1]) # BGR->RGB 116 | 117 | queue_ver.popleft() 118 | queue_frame.popleft() 119 | 120 | # we will lost the last n_next frames, still padding 121 | for _ in range(n_next): 122 | queue_ver.append(ver.copy()) 123 | queue_frame.append(frame_bgr.copy()) # the last frame 124 | 125 | ver_ave = np.mean(queue_ver, axis=0) 126 | 127 | if args.opt == '2d_sparse': 128 | img_draw = cv_draw_landmark(queue_frame[n_pre], ver_ave) # since we use padding 129 | elif args.opt == '2d_dense': 130 | img_draw = cv_draw_landmark(queue_frame[n_pre], ver_ave, size=1) 131 | elif args.opt == '3d': 132 | img_draw = render(queue_frame[n_pre], [ver_ave], tddfa.tri, alpha=0.7) 133 | else: 134 | raise ValueError(f'Unknown opt {args.opt}') 135 | 136 | writer.append_data(img_draw[..., ::-1]) # BGR->RGB 137 | 138 | queue_ver.popleft() 139 | queue_frame.popleft() 140 | 141 | writer.close() 142 | print(f'Dump to {video_wfp}') 143 | 144 | 145 | if __name__ == '__main__': 146 | parser = argparse.ArgumentParser(description='The smooth demo of video of 3DDFA_V2') 147 | parser.add_argument('-c', '--config', type=str, default='configs/mb1_120x120.yml') 148 | parser.add_argument('-f', '--video_fp', type=str) 149 | parser.add_argument('-m', '--mode', default='cpu', type=str, help='gpu or cpu mode') 150 | parser.add_argument('-n_pre', default=1, type=int, help='the pre frames of smoothing') 151 | parser.add_argument('-n_next', default=1, type=int, help='the next frames of smoothing') 152 | parser.add_argument('-o', '--opt', type=str, default='2d_sparse', choices=['2d_sparse', '2d_dense', '3d']) 153 | parser.add_argument('-s', '--start', default=-1, type=int, help='the started frames') 154 | parser.add_argument('-e', '--end', default=-1, type=int, help='the end frame') 155 | parser.add_argument('--onnx', action='store_true', default=False) 156 | 157 | args = parser.parse_args() 158 | main(args) 159 | -------------------------------------------------------------------------------- /demo_webcam_smooth.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import argparse 6 | import imageio 7 | import cv2 8 | import numpy as np 9 | from tqdm import tqdm 10 | import yaml 11 | from collections import deque 12 | 13 | from FaceBoxes import FaceBoxes 14 | from TDDFA import TDDFA 15 | from utils.render import render 16 | # from utils.render_ctypes import render 17 | from utils.functions import cv_draw_landmark 18 | 19 | 20 | def main(args): 21 | cfg = yaml.load(open(args.config), Loader=yaml.SafeLoader) 22 | 23 | # Init FaceBoxes and TDDFA, recommend using onnx flag 24 | if args.onnx: 25 | import os 26 | os.environ['KMP_DUPLICATE_LIB_OK'] = 'True' 27 | os.environ['OMP_NUM_THREADS'] = '4' 28 | 29 | from FaceBoxes.FaceBoxes_ONNX import FaceBoxes_ONNX 30 | from TDDFA_ONNX import TDDFA_ONNX 31 | 32 | face_boxes = FaceBoxes_ONNX() 33 | tddfa = TDDFA_ONNX(**cfg) 34 | else: 35 | gpu_mode = args.mode == 'gpu' 36 | tddfa = TDDFA(gpu_mode=gpu_mode, **cfg) 37 | face_boxes = FaceBoxes() 38 | 39 | # Given a camera 40 | # before run this line, make sure you have installed `imageio-ffmpeg` 41 | reader = imageio.get_reader("") 42 | 43 | # the simple implementation of average smoothing by looking ahead by n_next frames 44 | # assert the frames of the video >= n 45 | n_pre, n_next = args.n_pre, args.n_next 46 | n = n_pre + n_next + 1 47 | queue_ver = deque() 48 | queue_frame = deque() 49 | 50 | # run 51 | dense_flag = args.opt in ('2d_dense', '3d') 52 | pre_ver = None 53 | for i, frame in tqdm(enumerate(reader)): 54 | frame_bgr = frame[..., ::-1] # RGB->BGR 55 | 56 | if i == 0: 57 | # the first frame, detect face, here we only use the first face, you can change depending on your need 58 | boxes = face_boxes(frame_bgr) 59 | boxes = [boxes[0]] 60 | param_lst, roi_box_lst = tddfa(frame_bgr, boxes) 61 | ver = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)[0] 62 | 63 | # refine 64 | param_lst, roi_box_lst = tddfa(frame_bgr, [ver], crop_policy='landmark') 65 | ver = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)[0] 66 | 67 | # padding queue 68 | for _ in range(n_pre): 69 | queue_ver.append(ver.copy()) 70 | queue_ver.append(ver.copy()) 71 | 72 | for _ in range(n_pre): 73 | queue_frame.append(frame_bgr.copy()) 74 | queue_frame.append(frame_bgr.copy()) 75 | else: 76 | param_lst, roi_box_lst = tddfa(frame_bgr, [pre_ver], crop_policy='landmark') 77 | 78 | roi_box = roi_box_lst[0] 79 | # todo: add confidence threshold to judge the tracking is failed 80 | if abs(roi_box[2] - roi_box[0]) * abs(roi_box[3] - roi_box[1]) < 2020: 81 | boxes = face_boxes(frame_bgr) 82 | boxes = [boxes[0]] 83 | param_lst, roi_box_lst = tddfa(frame_bgr, boxes) 84 | 85 | ver = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=dense_flag)[0] 86 | 87 | queue_ver.append(ver.copy()) 88 | queue_frame.append(frame_bgr.copy()) 89 | 90 | pre_ver = ver # for tracking 91 | 92 | # smoothing: enqueue and dequeue ops 93 | if len(queue_ver) >= n: 94 | ver_ave = np.mean(queue_ver, axis=0) 95 | 96 | if args.opt == '2d_sparse': 97 | img_draw = cv_draw_landmark(queue_frame[n_pre], ver_ave) # since we use padding 98 | elif args.opt == '2d_dense': 99 | img_draw = cv_draw_landmark(queue_frame[n_pre], ver_ave, size=1) 100 | elif args.opt == '3d': 101 | img_draw = render(queue_frame[n_pre], [ver_ave], tddfa.tri, alpha=0.7) 102 | else: 103 | raise ValueError(f'Unknown opt {args.opt}') 104 | 105 | cv2.imshow('image', img_draw) 106 | k = cv2.waitKey(20) 107 | if (k & 0xff == ord('q')): 108 | break 109 | 110 | queue_ver.popleft() 111 | queue_frame.popleft() 112 | 113 | 114 | if __name__ == '__main__': 115 | parser = argparse.ArgumentParser(description='The smooth demo of webcam of 3DDFA_V2') 116 | parser.add_argument('-c', '--config', type=str, default='configs/mb1_120x120.yml') 117 | parser.add_argument('-m', '--mode', default='cpu', type=str, help='gpu or cpu mode') 118 | parser.add_argument('-o', '--opt', type=str, default='2d_sparse', choices=['2d_sparse', '2d_dense', '3d']) 119 | parser.add_argument('-n_pre', default=1, type=int, help='the pre frames of smoothing') 120 | parser.add_argument('-n_next', default=1, type=int, help='the next frames of smoothing') 121 | parser.add_argument('--onnx', action='store_true', default=False) 122 | 123 | args = parser.parse_args() 124 | main(args) 125 | -------------------------------------------------------------------------------- /docs/images/emma_3d.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/emma_3d.jpg -------------------------------------------------------------------------------- /docs/images/latency.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/latency.gif -------------------------------------------------------------------------------- /docs/images/obj.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/obj.jpg -------------------------------------------------------------------------------- /docs/images/out.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/out.gif -------------------------------------------------------------------------------- /docs/images/ply.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/ply.jpg -------------------------------------------------------------------------------- /docs/images/trump_biden_3d.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/trump_biden_3d.jpg -------------------------------------------------------------------------------- /docs/images/trump_hillary_2d_dense.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/trump_hillary_2d_dense.jpg -------------------------------------------------------------------------------- /docs/images/trump_hillary_2d_sparse.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/trump_hillary_2d_sparse.jpg -------------------------------------------------------------------------------- /docs/images/trump_hillary_3d.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/trump_hillary_3d.jpg -------------------------------------------------------------------------------- /docs/images/trump_hillary_depth.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/trump_hillary_depth.jpg -------------------------------------------------------------------------------- /docs/images/trump_hillary_pncc.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/trump_hillary_pncc.jpg -------------------------------------------------------------------------------- /docs/images/trump_hillary_pose.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/trump_hillary_pose.jpg -------------------------------------------------------------------------------- /docs/images/trump_hillary_uv_tex.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/trump_hillary_uv_tex.jpg -------------------------------------------------------------------------------- /docs/images/webcam.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/docs/images/webcam.gif -------------------------------------------------------------------------------- /examples/.gitignore: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/examples/.gitignore -------------------------------------------------------------------------------- /examples/inputs/.gitignore: -------------------------------------------------------------------------------- 1 | images/ 2 | -------------------------------------------------------------------------------- /examples/inputs/JianzhuGuo.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/examples/inputs/JianzhuGuo.jpg -------------------------------------------------------------------------------- /examples/inputs/emma.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/examples/inputs/emma.jpg -------------------------------------------------------------------------------- /examples/inputs/trump_hillary.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/examples/inputs/trump_hillary.jpg -------------------------------------------------------------------------------- /examples/inputs/videos/.gitignore: -------------------------------------------------------------------------------- 1 | *.avi 2 | -------------------------------------------------------------------------------- /examples/inputs/videos/214.avi: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/examples/inputs/videos/214.avi -------------------------------------------------------------------------------- /examples/results/.gitignore: -------------------------------------------------------------------------------- 1 | *.jpg 2 | *.gif 3 | *.list 4 | *.ply 5 | *.obj 6 | *.wav 7 | -------------------------------------------------------------------------------- /examples/results/videos/.gitignore: -------------------------------------------------------------------------------- 1 | *.mp4 2 | -------------------------------------------------------------------------------- /gradiodemo.py: -------------------------------------------------------------------------------- 1 | # before import, make sure FaceBoxes and Sim3DR are built successfully, e.g., 2 | import sys 3 | from subprocess import call 4 | import os 5 | import torch 6 | 7 | torch.hub.download_url_to_file('https://upload.wikimedia.org/wikipedia/commons/thumb/6/6e/Solvay_conference_1927.jpg/1400px-Solvay_conference_1927.jpg', 'solvay.jpg') 8 | 9 | def run_cmd(command): 10 | try: 11 | print(command) 12 | call(command, shell=True) 13 | except Exception as e: 14 | print(f"Errorrrrr: {e}!") 15 | 16 | print(os.getcwd()) 17 | os.chdir("/app/FaceBoxes/utils") 18 | print(os.getcwd()) 19 | run_cmd("python3 build.py build_ext --inplace") 20 | os.chdir("/app/Sim3DR") 21 | print(os.getcwd()) 22 | run_cmd("python3 setup.py build_ext --inplace") 23 | print(os.getcwd()) 24 | os.chdir("/app/utils/asset") 25 | print(os.getcwd()) 26 | run_cmd("gcc -shared -Wall -O3 render.c -o render.so -fPIC") 27 | os.chdir("/app") 28 | print(os.getcwd()) 29 | 30 | 31 | import cv2 32 | import yaml 33 | 34 | from FaceBoxes import FaceBoxes 35 | from TDDFA import TDDFA 36 | from utils.render import render 37 | from utils.depth import depth 38 | from utils.pncc import pncc 39 | from utils.uv import uv_tex 40 | from utils.pose import viz_pose 41 | from utils.serialization import ser_to_ply, ser_to_obj 42 | from utils.functions import draw_landmarks, get_suffix 43 | 44 | import matplotlib.pyplot as plt 45 | from skimage import io 46 | import gradio as gr 47 | 48 | # load config 49 | cfg = yaml.load(open('configs/mb1_120x120.yml'), Loader=yaml.SafeLoader) 50 | 51 | # Init FaceBoxes and TDDFA, recommend using onnx flag 52 | onnx_flag = True # or True to use ONNX to speed up 53 | if onnx_flag: 54 | import os 55 | os.environ['KMP_DUPLICATE_LIB_OK'] = 'True' 56 | os.environ['OMP_NUM_THREADS'] = '4' 57 | from FaceBoxes.FaceBoxes_ONNX import FaceBoxes_ONNX 58 | from TDDFA_ONNX import TDDFA_ONNX 59 | 60 | face_boxes = FaceBoxes_ONNX() 61 | tddfa = TDDFA_ONNX(**cfg) 62 | else: 63 | face_boxes = FaceBoxes() 64 | tddfa = TDDFA(gpu_mode=False, **cfg) 65 | 66 | 67 | 68 | def inference (img): 69 | # face detection 70 | boxes = face_boxes(img) 71 | # regress 3DMM params 72 | param_lst, roi_box_lst = tddfa(img, boxes) 73 | # reconstruct vertices and render 74 | ver_lst = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=True) 75 | return render(img, ver_lst, tddfa.tri, alpha=0.6, show_flag=False); 76 | 77 | 78 | title = "3DDFA V2" 79 | description = "demo for 3DDFA V2. To use it, simply upload your image, or click one of the examples to load them. Read more at the links below." 80 | article = "

Towards Fast, Accurate and Stable 3D Dense Face Alignment | Github Repo

" 81 | examples = [ 82 | ['solvay.jpg'] 83 | ] 84 | gr.Interface( 85 | inference, 86 | [gr.inputs.Image(type="numpy", label="Input")], 87 | gr.outputs.Image(type="numpy", label="Output"), 88 | title=title, 89 | description=description, 90 | article=article, 91 | examples=examples 92 | ).launch() 93 | -------------------------------------------------------------------------------- /latency.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import sys 6 | import argparse 7 | import cv2 8 | import yaml 9 | 10 | from FaceBoxes import FaceBoxes 11 | from TDDFA import TDDFA 12 | from utils.tddfa_util import str2bool 13 | from FaceBoxes.utils.timer import Timer 14 | 15 | 16 | def main(args): 17 | _t = { 18 | 'det': Timer(), 19 | 'reg': Timer(), 20 | 'recon': Timer() 21 | } 22 | 23 | cfg = yaml.load(open(args.config), Loader=yaml.SafeLoader) 24 | 25 | # Init FaceBoxes and TDDFA 26 | if args.onnx: 27 | import os 28 | os.environ['KMP_DUPLICATE_LIB_OK'] = 'True' 29 | os.environ['OMP_NUM_THREADS'] = '4' 30 | 31 | from FaceBoxes.FaceBoxes_ONNX import FaceBoxes_ONNX 32 | from TDDFA_ONNX import TDDFA_ONNX 33 | 34 | face_boxes = FaceBoxes_ONNX() 35 | tddfa = TDDFA_ONNX(**cfg) 36 | else: 37 | tddfa = TDDFA(**cfg) 38 | face_boxes = FaceBoxes() 39 | 40 | # Given a still image path and load to BGR channel 41 | img = cv2.imread(args.img_fp) 42 | print(f'Input image: {args.img_fp}') 43 | 44 | # Detect faces, get 3DMM params and roi boxes 45 | print(f'Input shape: {img.shape}') 46 | if args.warmup: 47 | print('Warmup by once') 48 | boxes = face_boxes(img) 49 | param_lst, roi_box_lst = tddfa(img, boxes) 50 | ver_lst = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=args.dense_flag) 51 | 52 | for _ in range(args.repeated): 53 | img = cv2.imread(args.img_fp) 54 | 55 | _t['det'].tic() 56 | boxes = face_boxes(img) 57 | _t['det'].toc() 58 | 59 | n = len(boxes) 60 | if n == 0: 61 | print(f'No face detected, exit') 62 | sys.exit(-1) 63 | 64 | _t['reg'].tic() 65 | param_lst, roi_box_lst = tddfa(img, boxes) 66 | _t['reg'].toc() 67 | 68 | _t['recon'].tic() 69 | ver_lst = tddfa.recon_vers(param_lst, roi_box_lst, dense_flag=args.dense_flag) 70 | _t['recon'].toc() 71 | 72 | mode = 'Dense' if args.dense_flag else 'Sparse' 73 | print(f"Face detection: {_t['det'].average_time * 1000:.2f}ms, " 74 | f"3DMM regression: {_t['reg'].average_time * 1000:.2f}ms, " 75 | f"{mode} reconstruction: {_t['recon'].average_time * 1000:.2f}ms") 76 | 77 | 78 | if __name__ == '__main__': 79 | parser = argparse.ArgumentParser(description='The latency testing of still image of 3DDFA_V2') 80 | parser.add_argument('-c', '--config', type=str, default='configs/mb1_120x120.yml') 81 | parser.add_argument('-f', '--img_fp', type=str, default='examples/inputs/JianzhuGuo.jpg') 82 | parser.add_argument('--onnx', action='store_true', default=False) 83 | parser.add_argument('--warmup', type=str2bool, default='true') 84 | parser.add_argument('--dense_flag', type=str2bool, default='true') 85 | parser.add_argument('--repeated', type=int, default=32) 86 | 87 | args = parser.parse_args() 88 | main(args) 89 | -------------------------------------------------------------------------------- /models/__init__.py: -------------------------------------------------------------------------------- 1 | from .mobilenet_v1 import * 2 | from .mobilenet_v3 import * 3 | from .resnet import * -------------------------------------------------------------------------------- /models/mobilenet_v1.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | from __future__ import division 4 | 5 | """ 6 | Creates a MobileNet Model as defined in: 7 | Andrew G. Howard Menglong Zhu Bo Chen, et.al. (2017). 8 | MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. 9 | Copyright (c) Yang Lu, 2017 10 | 11 | Modified By cleardusk 12 | """ 13 | import math 14 | import torch.nn as nn 15 | 16 | __all__ = ['MobileNet', 'mobilenet'] 17 | 18 | 19 | # __all__ = ['mobilenet_2', 'mobilenet_1', 'mobilenet_075', 'mobilenet_05', 'mobilenet_025'] 20 | 21 | 22 | class DepthWiseBlock(nn.Module): 23 | def __init__(self, inplanes, planes, stride=1, prelu=False): 24 | super(DepthWiseBlock, self).__init__() 25 | inplanes, planes = int(inplanes), int(planes) 26 | self.conv_dw = nn.Conv2d(inplanes, inplanes, kernel_size=3, padding=1, stride=stride, groups=inplanes, 27 | bias=False) 28 | self.bn_dw = nn.BatchNorm2d(inplanes) 29 | self.conv_sep = nn.Conv2d(inplanes, planes, kernel_size=1, stride=1, padding=0, bias=False) 30 | self.bn_sep = nn.BatchNorm2d(planes) 31 | if prelu: 32 | self.relu = nn.PReLU() 33 | else: 34 | self.relu = nn.ReLU(inplace=True) 35 | 36 | def forward(self, x): 37 | out = self.conv_dw(x) 38 | out = self.bn_dw(out) 39 | out = self.relu(out) 40 | 41 | out = self.conv_sep(out) 42 | out = self.bn_sep(out) 43 | out = self.relu(out) 44 | 45 | return out 46 | 47 | 48 | class MobileNet(nn.Module): 49 | def __init__(self, widen_factor=1.0, num_classes=1000, prelu=False, input_channel=3): 50 | """ Constructor 51 | Args: 52 | widen_factor: config of widen_factor 53 | num_classes: number of classes 54 | """ 55 | super(MobileNet, self).__init__() 56 | 57 | block = DepthWiseBlock 58 | self.conv1 = nn.Conv2d(input_channel, int(32 * widen_factor), kernel_size=3, stride=2, padding=1, 59 | bias=False) 60 | 61 | self.bn1 = nn.BatchNorm2d(int(32 * widen_factor)) 62 | if prelu: 63 | self.relu = nn.PReLU() 64 | else: 65 | self.relu = nn.ReLU(inplace=True) 66 | 67 | self.dw2_1 = block(32 * widen_factor, 64 * widen_factor, prelu=prelu) 68 | self.dw2_2 = block(64 * widen_factor, 128 * widen_factor, stride=2, prelu=prelu) 69 | 70 | self.dw3_1 = block(128 * widen_factor, 128 * widen_factor, prelu=prelu) 71 | self.dw3_2 = block(128 * widen_factor, 256 * widen_factor, stride=2, prelu=prelu) 72 | 73 | self.dw4_1 = block(256 * widen_factor, 256 * widen_factor, prelu=prelu) 74 | self.dw4_2 = block(256 * widen_factor, 512 * widen_factor, stride=2, prelu=prelu) 75 | 76 | self.dw5_1 = block(512 * widen_factor, 512 * widen_factor, prelu=prelu) 77 | self.dw5_2 = block(512 * widen_factor, 512 * widen_factor, prelu=prelu) 78 | self.dw5_3 = block(512 * widen_factor, 512 * widen_factor, prelu=prelu) 79 | self.dw5_4 = block(512 * widen_factor, 512 * widen_factor, prelu=prelu) 80 | self.dw5_5 = block(512 * widen_factor, 512 * widen_factor, prelu=prelu) 81 | self.dw5_6 = block(512 * widen_factor, 1024 * widen_factor, stride=2, prelu=prelu) 82 | 83 | self.dw6 = block(1024 * widen_factor, 1024 * widen_factor, prelu=prelu) 84 | 85 | self.avgpool = nn.AdaptiveAvgPool2d(1) 86 | self.fc = nn.Linear(int(1024 * widen_factor), num_classes) 87 | 88 | for m in self.modules(): 89 | if isinstance(m, nn.Conv2d): 90 | n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels 91 | m.weight.data.normal_(0, math.sqrt(2. / n)) 92 | elif isinstance(m, nn.BatchNorm2d): 93 | m.weight.data.fill_(1) 94 | m.bias.data.zero_() 95 | 96 | def forward(self, x): 97 | x = self.conv1(x) 98 | x = self.bn1(x) 99 | x = self.relu(x) 100 | 101 | x = self.dw2_1(x) 102 | x = self.dw2_2(x) 103 | x = self.dw3_1(x) 104 | x = self.dw3_2(x) 105 | x = self.dw4_1(x) 106 | x = self.dw4_2(x) 107 | x = self.dw5_1(x) 108 | x = self.dw5_2(x) 109 | x = self.dw5_3(x) 110 | x = self.dw5_4(x) 111 | x = self.dw5_5(x) 112 | x = self.dw5_6(x) 113 | x = self.dw6(x) 114 | 115 | x = self.avgpool(x) 116 | x = x.view(x.size(0), -1) 117 | x = self.fc(x) 118 | 119 | return x 120 | 121 | 122 | def mobilenet(**kwargs): 123 | """ 124 | Construct MobileNet. 125 | widen_factor=1.0 for mobilenet_1 126 | widen_factor=0.75 for mobilenet_075 127 | widen_factor=0.5 for mobilenet_05 128 | widen_factor=0.25 for mobilenet_025 129 | """ 130 | # widen_factor = 1.0, num_classes = 1000 131 | # model = MobileNet(widen_factor=widen_factor, num_classes=num_classes) 132 | # return model 133 | 134 | model = MobileNet( 135 | widen_factor=kwargs.get('widen_factor', 1.0), 136 | num_classes=kwargs.get('num_classes', 62) 137 | ) 138 | return model 139 | 140 | 141 | def mobilenet_2(num_classes=62, input_channel=3): 142 | model = MobileNet(widen_factor=2.0, num_classes=num_classes, input_channel=input_channel) 143 | return model 144 | 145 | 146 | def mobilenet_1(num_classes=62, input_channel=3): 147 | model = MobileNet(widen_factor=1.0, num_classes=num_classes, input_channel=input_channel) 148 | return model 149 | 150 | 151 | def mobilenet_075(num_classes=62, input_channel=3): 152 | model = MobileNet(widen_factor=0.75, num_classes=num_classes, input_channel=input_channel) 153 | return model 154 | 155 | 156 | def mobilenet_05(num_classes=62, input_channel=3): 157 | model = MobileNet(widen_factor=0.5, num_classes=num_classes, input_channel=input_channel) 158 | return model 159 | 160 | 161 | def mobilenet_025(num_classes=62, input_channel=3): 162 | model = MobileNet(widen_factor=0.25, num_classes=num_classes, input_channel=input_channel) 163 | return model 164 | -------------------------------------------------------------------------------- /models/mobilenet_v3.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | 4 | import torch.nn as nn 5 | import torch.nn.functional as F 6 | 7 | __all__ = ['MobileNetV3', 'mobilenet_v3'] 8 | 9 | 10 | def conv_bn(inp, oup, stride, conv_layer=nn.Conv2d, norm_layer=nn.BatchNorm2d, nlin_layer=nn.ReLU): 11 | return nn.Sequential( 12 | conv_layer(inp, oup, 3, stride, 1, bias=False), 13 | norm_layer(oup), 14 | nlin_layer(inplace=True) 15 | ) 16 | 17 | 18 | def conv_1x1_bn(inp, oup, conv_layer=nn.Conv2d, norm_layer=nn.BatchNorm2d, nlin_layer=nn.ReLU): 19 | return nn.Sequential( 20 | conv_layer(inp, oup, 1, 1, 0, bias=False), 21 | norm_layer(oup), 22 | nlin_layer(inplace=True) 23 | ) 24 | 25 | 26 | class Hswish(nn.Module): 27 | def __init__(self, inplace=True): 28 | super(Hswish, self).__init__() 29 | self.inplace = inplace 30 | 31 | def forward(self, x): 32 | return x * F.relu6(x + 3., inplace=self.inplace) / 6. 33 | 34 | 35 | class Hsigmoid(nn.Module): 36 | def __init__(self, inplace=True): 37 | super(Hsigmoid, self).__init__() 38 | self.inplace = inplace 39 | 40 | def forward(self, x): 41 | return F.relu6(x + 3., inplace=self.inplace) / 6. 42 | 43 | 44 | class SEModule(nn.Module): 45 | def __init__(self, channel, reduction=4): 46 | super(SEModule, self).__init__() 47 | self.avg_pool = nn.AdaptiveAvgPool2d(1) 48 | self.fc = nn.Sequential( 49 | nn.Linear(channel, channel // reduction, bias=False), 50 | nn.ReLU(inplace=True), 51 | nn.Linear(channel // reduction, channel, bias=False), 52 | Hsigmoid() 53 | # nn.Sigmoid() 54 | ) 55 | 56 | def forward(self, x): 57 | b, c, _, _ = x.size() 58 | y = self.avg_pool(x).view(b, c) 59 | y = self.fc(y).view(b, c, 1, 1) 60 | return x * y.expand_as(x) 61 | 62 | 63 | class Identity(nn.Module): 64 | def __init__(self, channel): 65 | super(Identity, self).__init__() 66 | 67 | def forward(self, x): 68 | return x 69 | 70 | 71 | def make_divisible(x, divisible_by=8): 72 | import numpy as np 73 | return int(np.ceil(x * 1. / divisible_by) * divisible_by) 74 | 75 | 76 | class MobileBottleneck(nn.Module): 77 | def __init__(self, inp, oup, kernel, stride, exp, se=False, nl='RE'): 78 | super(MobileBottleneck, self).__init__() 79 | assert stride in [1, 2] 80 | assert kernel in [3, 5] 81 | padding = (kernel - 1) // 2 82 | self.use_res_connect = stride == 1 and inp == oup 83 | 84 | conv_layer = nn.Conv2d 85 | norm_layer = nn.BatchNorm2d 86 | if nl == 'RE': 87 | nlin_layer = nn.ReLU # or ReLU6 88 | elif nl == 'HS': 89 | nlin_layer = Hswish 90 | else: 91 | raise NotImplementedError 92 | if se: 93 | SELayer = SEModule 94 | else: 95 | SELayer = Identity 96 | 97 | self.conv = nn.Sequential( 98 | # pw 99 | conv_layer(inp, exp, 1, 1, 0, bias=False), 100 | norm_layer(exp), 101 | nlin_layer(inplace=True), 102 | # dw 103 | conv_layer(exp, exp, kernel, stride, padding, groups=exp, bias=False), 104 | norm_layer(exp), 105 | SELayer(exp), 106 | nlin_layer(inplace=True), 107 | # pw-linear 108 | conv_layer(exp, oup, 1, 1, 0, bias=False), 109 | norm_layer(oup), 110 | ) 111 | 112 | def forward(self, x): 113 | if self.use_res_connect: 114 | return x + self.conv(x) 115 | else: 116 | return self.conv(x) 117 | 118 | 119 | class MobileNetV3(nn.Module): 120 | def __init__(self, widen_factor=1.0, num_classes=141, num_landmarks=136, input_size=120, mode='small'): 121 | super(MobileNetV3, self).__init__() 122 | input_channel = 16 123 | last_channel = 1280 124 | if mode == 'large': 125 | # refer to Table 1 in paper 126 | mobile_setting = [ 127 | # k, exp, c, se, nl, s, 128 | [3, 16, 16, False, 'RE', 1], 129 | [3, 64, 24, False, 'RE', 2], 130 | [3, 72, 24, False, 'RE', 1], 131 | [5, 72, 40, True, 'RE', 2], 132 | [5, 120, 40, True, 'RE', 1], 133 | [5, 120, 40, True, 'RE', 1], 134 | [3, 240, 80, False, 'HS', 2], 135 | [3, 200, 80, False, 'HS', 1], 136 | [3, 184, 80, False, 'HS', 1], 137 | [3, 184, 80, False, 'HS', 1], 138 | [3, 480, 112, True, 'HS', 1], 139 | [3, 672, 112, True, 'HS', 1], 140 | [5, 672, 160, True, 'HS', 2], 141 | [5, 960, 160, True, 'HS', 1], 142 | [5, 960, 160, True, 'HS', 1], 143 | ] 144 | elif mode == 'small': 145 | # refer to Table 2 in paper 146 | mobile_setting = [ 147 | # k, exp, c, se, nl, s, 148 | [3, 16, 16, True, 'RE', 2], 149 | [3, 72, 24, False, 'RE', 2], 150 | [3, 88, 24, False, 'RE', 1], 151 | [5, 96, 40, True, 'HS', 2], 152 | [5, 240, 40, True, 'HS', 1], 153 | [5, 240, 40, True, 'HS', 1], 154 | [5, 120, 48, True, 'HS', 1], 155 | [5, 144, 48, True, 'HS', 1], 156 | [5, 288, 96, True, 'HS', 2], 157 | [5, 576, 96, True, 'HS', 1], 158 | [5, 576, 96, True, 'HS', 1], 159 | ] 160 | else: 161 | raise NotImplementedError 162 | 163 | # building first layer 164 | assert input_size % 32 == 0 165 | last_channel = make_divisible(last_channel * widen_factor) if widen_factor > 1.0 else last_channel 166 | self.features = [conv_bn(3, input_channel, 2, nlin_layer=Hswish)] 167 | # self.classifier = [] 168 | 169 | # building mobile blocks 170 | for k, exp, c, se, nl, s in mobile_setting: 171 | output_channel = make_divisible(c * widen_factor) 172 | exp_channel = make_divisible(exp * widen_factor) 173 | self.features.append(MobileBottleneck(input_channel, output_channel, k, s, exp_channel, se, nl)) 174 | input_channel = output_channel 175 | 176 | # building last several layers 177 | if mode == 'large': 178 | last_conv = make_divisible(960 * widen_factor) 179 | self.features.append(conv_1x1_bn(input_channel, last_conv, nlin_layer=Hswish)) 180 | self.features.append(nn.AdaptiveAvgPool2d(1)) 181 | self.features.append(nn.Conv2d(last_conv, last_channel, 1, 1, 0)) 182 | self.features.append(Hswish(inplace=True)) 183 | elif mode == 'small': 184 | last_conv = make_divisible(576 * widen_factor) 185 | self.features.append(conv_1x1_bn(input_channel, last_conv, nlin_layer=Hswish)) 186 | # self.features.append(SEModule(last_conv)) # refer to paper Table2, but I think this is a mistake 187 | self.features.append(nn.AdaptiveAvgPool2d(1)) 188 | self.features.append(nn.Conv2d(last_conv, last_channel, 1, 1, 0)) 189 | self.features.append(Hswish(inplace=True)) 190 | else: 191 | raise NotImplementedError 192 | 193 | # make it nn.Sequential 194 | self.features = nn.Sequential(*self.features) 195 | 196 | # self.fc_param = nn.Linear(int(last_channel), num_classes) 197 | self.fc = nn.Linear(int(last_channel), num_classes) 198 | # self.fc_lm = nn.Linear(int(last_channel), num_landmarks) 199 | 200 | # building classifier 201 | # self.classifier = nn.Sequential( 202 | # nn.Dropout(p=dropout), # refer to paper section 6 203 | # nn.Linear(last_channel, n_class), 204 | # ) 205 | 206 | self._initialize_weights() 207 | 208 | def forward(self, x): 209 | x = self.features(x) 210 | x_share = x.mean(3).mean(2) 211 | 212 | # x = self.classifier(x) 213 | # print(x_share.shape) 214 | # xp = self.fc_param(x_share) # param 215 | # xl = self.fc_lm(x_share) # lm 216 | 217 | xp = self.fc(x_share) # param 218 | 219 | return xp # , xl 220 | 221 | def _initialize_weights(self): 222 | # weight initialization 223 | for m in self.modules(): 224 | if isinstance(m, nn.Conv2d): 225 | nn.init.kaiming_normal_(m.weight, mode='fan_out') 226 | if m.bias is not None: 227 | nn.init.zeros_(m.bias) 228 | elif isinstance(m, nn.BatchNorm2d): 229 | nn.init.ones_(m.weight) 230 | nn.init.zeros_(m.bias) 231 | elif isinstance(m, nn.Linear): 232 | nn.init.normal_(m.weight, 0, 0.01) 233 | if m.bias is not None: 234 | nn.init.zeros_(m.bias) 235 | 236 | 237 | def mobilenet_v3(**kwargs): 238 | model = MobileNetV3( 239 | widen_factor=kwargs.get('widen_factor', 1.0), 240 | num_classes=kwargs.get('num_classes', 62), 241 | num_landmarks=kwargs.get('num_landmarks', 136), 242 | input_size=kwargs.get('size', 128), 243 | mode=kwargs.get('mode', 'small') 244 | ) 245 | 246 | return model 247 | -------------------------------------------------------------------------------- /models/resnet.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # coding: utf-8 3 | 4 | import torch.nn as nn 5 | 6 | __all__ = ['ResNet', 'resnet22'] 7 | 8 | 9 | def conv3x3(in_planes, out_planes, stride=1): 10 | "3x3 convolution with padding" 11 | return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride, 12 | padding=1, bias=False) 13 | 14 | 15 | class BasicBlock(nn.Module): 16 | expansion = 1 17 | 18 | def __init__(self, inplanes, planes, stride=1, downsample=None): 19 | super(BasicBlock, self).__init__() 20 | self.conv1 = conv3x3(inplanes, planes, stride) 21 | self.bn1 = nn.BatchNorm2d(planes) 22 | self.relu = nn.ReLU(inplace=True) 23 | self.conv2 = conv3x3(planes, planes) 24 | self.bn2 = nn.BatchNorm2d(planes) 25 | self.downsample = downsample 26 | self.stride = stride 27 | 28 | def forward(self, x): 29 | residual = x 30 | 31 | out = self.conv1(x) 32 | out = self.bn1(out) 33 | out = self.relu(out) 34 | 35 | out = self.conv2(out) 36 | out = self.bn2(out) 37 | 38 | if self.downsample is not None: 39 | residual = self.downsample(x) 40 | 41 | out += residual 42 | out = self.relu(out) 43 | 44 | return out 45 | 46 | 47 | class ResNet(nn.Module): 48 | """Another Strucutre used in caffe-resnet25""" 49 | 50 | def __init__(self, block, layers, num_classes=62, num_landmarks=136, input_channel=3, fc_flg=False): 51 | self.inplanes = 64 52 | super(ResNet, self).__init__() 53 | self.conv1 = nn.Conv2d(input_channel, 32, kernel_size=5, stride=2, padding=2, bias=False) 54 | self.bn1 = nn.BatchNorm2d(32) # 32 is input channels number 55 | self.relu1 = nn.ReLU(inplace=True) 56 | 57 | self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1, bias=False) 58 | self.bn2 = nn.BatchNorm2d(64) 59 | self.relu2 = nn.ReLU(inplace=True) 60 | 61 | # self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1) 62 | 63 | self.layer1 = self._make_layer(block, 128, layers[0], stride=2) 64 | self.layer2 = self._make_layer(block, 256, layers[1], stride=2) 65 | self.layer3 = self._make_layer(block, 512, layers[2], stride=2) 66 | 67 | self.conv_param = nn.Conv2d(512, num_classes, 1) 68 | # self.conv_lm = nn.Conv2d(512, num_landmarks, 1) 69 | self.avgpool = nn.AdaptiveAvgPool2d(1) 70 | # self.fc = nn.Linear(512 * block.expansion, num_classes) 71 | self.fc_flg = fc_flg 72 | 73 | # parameter initialization 74 | for m in self.modules(): 75 | if isinstance(m, nn.Conv2d): 76 | # 1. 77 | # n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels 78 | # m.weight.data.normal_(0, math.sqrt(2. / n)) 79 | 80 | # 2. kaiming normal 81 | nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu') 82 | elif isinstance(m, nn.BatchNorm2d): 83 | m.weight.data.fill_(1) 84 | m.bias.data.zero_() 85 | 86 | def _make_layer(self, block, planes, blocks, stride=1): 87 | downsample = None 88 | if stride != 1 or self.inplanes != planes * block.expansion: 89 | downsample = nn.Sequential( 90 | nn.Conv2d(self.inplanes, planes * block.expansion, 91 | kernel_size=1, stride=stride, bias=False), 92 | nn.BatchNorm2d(planes * block.expansion), 93 | ) 94 | 95 | layers = [] 96 | layers.append(block(self.inplanes, planes, stride, downsample)) 97 | self.inplanes = planes * block.expansion 98 | for i in range(1, blocks): 99 | layers.append(block(self.inplanes, planes)) 100 | 101 | return nn.Sequential(*layers) 102 | 103 | def forward(self, x): 104 | x = self.conv1(x) 105 | x = self.bn1(x) 106 | x = self.relu1(x) 107 | 108 | x = self.conv2(x) 109 | x = self.bn2(x) 110 | x = self.relu2(x) 111 | 112 | # x = self.maxpool(x) 113 | 114 | x = self.layer1(x) 115 | x = self.layer2(x) 116 | x = self.layer3(x) 117 | 118 | # if self.fc_flg: 119 | # x = self.avgpool(x) 120 | # x = x.view(x.size(0), -1) 121 | # x = self.fc(x) 122 | # else: 123 | xp = self.conv_param(x) 124 | xp = self.avgpool(xp) 125 | xp = xp.view(xp.size(0), -1) 126 | 127 | # xl = self.conv_lm(x) 128 | # xl = self.avgpool(xl) 129 | # xl = xl.view(xl.size(0), -1) 130 | 131 | return xp # , xl 132 | 133 | 134 | def resnet22(**kwargs): 135 | model = ResNet( 136 | BasicBlock, 137 | [3, 4, 3], 138 | num_landmarks=kwargs.get('num_landmarks', 136), 139 | input_channel=kwargs.get('input_channel', 3), 140 | fc_flg=False 141 | ) 142 | return model 143 | 144 | 145 | def main(): 146 | pass 147 | 148 | 149 | if __name__ == '__main__': 150 | main() 151 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | # Towards Fast, Accurate and Stable 3D Dense Face Alignment 2 | 3 | [![License](https://img.shields.io/badge/license-MIT-yellow.svg)](LICENSE) 4 | ![GitHub repo size](https://img.shields.io/github/repo-size/cleardusk/3DDFA_V2.svg) 5 | [![](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1OKciI0ETCpWdRjP-VOGpBulDJojYfgWv) 6 | 7 | By [Jianzhu Guo](https://guojianzhu.com), [Xiangyu Zhu](http://www.cbsr.ia.ac.cn/users/xiangyuzhu/), [Yang Yang](http://www.cbsr.ia.ac.cn/users/yyang/main.htm), Fan Yang, [Zhen Lei](http://www.cbsr.ia.ac.cn/users/zlei/) and [Stan Z. Li](https://scholar.google.com/citations?user=Y-nyLGIAAAAJ). 8 | The code repo is owned and maintained by **[Jianzhu Guo](https://guojianzhu.com)**. 9 | 10 | 11 |

12 | demo 13 |

14 | 15 | 16 | **\[Updates\]** 17 | - `2021.7.10`: Run 3DDFA_V2 online on [Gradio](https://gradio.app/hub/AK391/3DDFA_V2). 18 | - `2021.1.15`: Borrow the implementation of [Dense-Head-Pose-Estimation](https://github.com/1996scarlet/Dense-Head-Pose-Estimation) for the faster mesh rendering (speedup about 3x, 15ms -> 4ms), see [utils/render_ctypes.py](./utils/render_ctypes.py) for details. 19 | - `2020.10.7`: Add the latency evaluation of the full pipeline in [latency.py](./latency.py), just run by `python3 latency.py --onnx`, see [Latency](#Latency) evaluation for details. 20 | - `2020.10.6`: Add onnxruntime support for FaceBoxes to reduce the face detection latency, just append the `--onnx` action to activate it, see [FaceBoxes_ONNX.py](FaceBoxes/FaceBoxes_ONNX.py) for details. 21 | - `2020.10.2`: **Add onnxruntime support to greatly reduce the 3dmm parameters inference latency**, just append the `--onnx` action when running `demo.py`, see [TDDFA_ONNX.py](./TDDFA_ONNX.py) for details. 22 | - `2020.9.20`: Add features including pose estimation and serializations to .ply and .obj, see `pose`, `ply`, `obj` options in [demo.py](./demo.py). 23 | - `2020.9.19`: Add PNCC (Projected Normalized Coordinate Code), uv texture mapping features, see `pncc`, `uv_tex` options in [demo.py](./demo.py). 24 | 25 | 26 | ## Introduction 27 | 28 | This work extends [3DDFA](https://github.com/cleardusk/3DDFA), named **3DDFA_V2**, titled [Towards Fast, Accurate and Stable 3D Dense Face Alignment](https://guojianzhu.com/assets/pdfs/3162.pdf), accepted by [ECCV 2020](https://eccv2020.eu/). The supplementary material is [here](https://guojianzhu.com/assets/pdfs/3162-supp.pdf). The [gif](./docs/images/webcam.gif) above shows a webcam demo of the tracking result, in the scenario of my lab. This repo is the official implementation of 3DDFA_V2. 29 | 30 | Compared to [3DDFA](https://github.com/cleardusk/3DDFA), 3DDFA_V2 achieves better performance and stability. Besides, 3DDFA_V2 incorporates the fast face detector [FaceBoxes](https://github.com/zisianw/FaceBoxes.PyTorch) instead of Dlib. A simple 3D render written by c++ and cython is also included. This repo supports the onnxruntime, and the latency of regressing 3DMM parameters using the default backbone is about **1.35ms/image on CPU** with a single image as input. If you are interested in this repo, just try it on this **[google colab](https://colab.research.google.com/drive/1OKciI0ETCpWdRjP-VOGpBulDJojYfgWv)**! Welcome for valuable issues, PRs and discussions 😄 31 | 32 | 33 | 34 | ## Getting started 35 | 36 | ### Requirements 37 | See [requirements.txt](./requirements.txt), tested on macOS and Linux platforms. The Windows users may refer to [FQA](#FQA) for building issues. Note that this repo uses Python3. The major dependencies are PyTorch, numpy, opencv-python and onnxruntime, etc. If you run the demos with `--onnx` flag to do acceleration, you may need to install `libomp` first, i.e., `brew install libomp` on macOS. 38 | 39 | ### Usage 40 | 41 | 1. Clone this repo 42 | 43 | ```shell script 44 | git clone https://github.com/cleardusk/3DDFA_V2.git 45 | cd 3DDFA_V2 46 | ``` 47 | 48 | 2. Build the cython version of NMS, Sim3DR, and the faster mesh render 49 | 65 | ```shell script 66 | sh ./build.sh 67 | ``` 68 | 69 | 3. Run demos 70 | 71 | ```shell script 72 | # 1. running on still image, the options include: 2d_sparse, 2d_dense, 3d, depth, pncc, pose, uv_tex, ply, obj 73 | python3 demo.py -f examples/inputs/emma.jpg --onnx # -o [2d_sparse, 2d_dense, 3d, depth, pncc, pose, uv_tex, ply, obj] 74 | 75 | # 2. running on videos 76 | python3 demo_video.py -f examples/inputs/videos/214.avi --onnx 77 | 78 | # 3. running on videos smoothly by looking ahead by `n_next` frames 79 | python3 demo_video_smooth.py -f examples/inputs/videos/214.avi --onnx 80 | 81 | # 4. running on webcam 82 | python3 demo_webcam_smooth.py --onnx 83 | ``` 84 | 85 | The implementation of tracking is simply by alignment. If the head pose > 90° or the motion is too fast, the alignment may fail. A threshold is used to trickly check the tracking state, but it is unstable. 86 | 87 | You can refer to [demo.ipynb](./demo.ipynb) or [google colab](https://colab.research.google.com/drive/1OKciI0ETCpWdRjP-VOGpBulDJojYfgWv) for the step-by-step tutorial of running on the still image. 88 | 89 | For example, running `python3 demo.py -f examples/inputs/emma.jpg -o 3d` will give the result below: 90 | 91 |

92 | demo 93 |

94 | 95 | Another example: 96 | 97 |

98 | demo 99 |

100 | 101 | Running on a video will give: 102 | 103 |

104 | demo 105 |

106 | 107 | More results or demos to see: [Hathaway](https://guojianzhu.com/assets/videos/hathaway_3ddfa_v2.mp4). 108 | 109 | 110 | 111 | ### Features (up to now) 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 |
2D sparse2D dense3D
2d sparse2d dense3d
DepthPNCCUV texture
depthpnccuv_tex
PoseSerialization to .plySerialization to .obj
poseplyobj
152 | 153 | ### Configs 154 | 155 | The default backbone is MobileNet_V1 with input size 120x120 and the default pre-trained weight is `weights/mb1_120x120.pth`, shown in [configs/mb1_120x120.yml](configs/mb1_120x120.yml). This repo provides another config in [configs/mb05_120x120.yml](configs/mb05_120x120.yml), with the widen factor 0.5, being smaller and faster. You can specify the config by `-c` or `--config` option. The released models are shown in the below table. Note that the inference time on CPU in the paper is evaluated using TensorFlow. 156 | 157 | | Model | Input | #Params | #Macs | Inference (TF) | 158 | | :-: | :-: | :-: | :-: | :-: | 159 | | MobileNet | 120x120 | 3.27M | 183.5M | ~6.2ms | 160 | | MobileNet x0.5 | 120x120 | 0.85M | 49.5M | ~2.9ms | 161 | 162 | 163 | **Surprisingly**, the latency of [onnxruntime](https://github.com/microsoft/onnxruntime) is much smaller. The inference time on CPU with different threads is shown below. The results are tested on my MBP (i5-8259U CPU @ 2.30GHz on 13-inch MacBook Pro), with the `1.5.1` version of onnxruntime. The thread number is set by `os.environ["OMP_NUM_THREADS"]`, see [speed_cpu.py](./speed_cpu.py) for more details. 164 | 165 | | Model | THREAD=1 | THREAD=2 | THREAD=4 | 166 | | :-: | :-: | :-: | :-: | 167 | | MobileNet | 4.4ms | 2.25ms | 1.35ms | 168 | | MobileNet x0.5 | 1.37ms | 0.7ms | 0.5ms | 169 | 170 | ### Latency 171 | 172 | The `onnx` option greatly reduces the overall **CPU** latency, but face detection still takes up most of the latency time, e.g., 15ms for a 720p image. 3DMM parameters regression takes about 1~2ms for one face, and the dense reconstruction (more than 30,000 points, i.e. 38,365) is about 1ms for one face. Tracking applications may benefit from the fast 3DMM regression speed, since detection is not needed for every frame. The latency is tested using my 13-inch MacBook Pro (i5-8259U CPU @ 2.30GHz). 173 | 174 | The default `OMP_NUM_THREADS` is set 4, you can specify it by setting `os.environ['OMP_NUM_THREADS'] = '$NUM'` or inserting `export OMP_NUM_THREADS=$NUM` before running the python script. 175 | 176 |

177 | demo 178 |

179 | 180 | ## FQA 181 | 182 | 1. What is the training data? 183 | 184 | We use [300W-LP](https://drive.google.com/file/d/0B7OEHD3T4eCkVGs0TkhUWFN6N1k/view?usp=sharing) for training. You can refer to our [paper](https://guojianzhu.com/assets/pdfs/3162.pdf) for more details about the training. Since few images are closed-eyes in the training data 300W-LP, the landmarks of eyes are not accurate when closing. The eyes part of the webcam demo are also not good. 185 | 186 | 2. Running on Windows. 187 | 188 | You can refer to [this comment](https://github.com/cleardusk/3DDFA_V2/issues/12#issuecomment-697479173) for building NMS on Windows. 189 | 190 | ## Acknowledgement 191 | 192 | * The FaceBoxes module is modified from [FaceBoxes.PyTorch](https://github.com/zisianw/FaceBoxes.PyTorch). 193 | * A list of previous works on 3D dense face alignment or reconstruction: [3DDFA](https://github.com/cleardusk/3DDFA), [face3d](https://github.com/YadiraF/face3d), [PRNet](https://github.com/YadiraF/PRNet). 194 | * Thank [AK391](https://github.com/AK391) for hosting the Gradio web app. 195 | 196 | ## Other implementations or applications 197 | 198 | * [Dense-Head-Pose-Estimation](https://github.com/1996scarlet/Dense-Head-Pose-Estimation): Tensorflow Lite framework for face mesh, head pose, landmarks, and more. 199 | * [HeadPoseEstimate](https://github.com/bubingy/HeadPoseEstimate): Head pose estimation system based on 3d facial landmarks. 200 | * [img2pose](https://github.com/vitoralbiero/img2pose): Borrow the renderer implementation of Sim3DR in this repo. 201 | 202 | ## Citation 203 | 204 | If your work or research benefits from this repo, please cite two bibs below : ) and 🌟 this repo. 205 | 206 | @inproceedings{guo2020towards, 207 | title = {Towards Fast, Accurate and Stable 3D Dense Face Alignment}, 208 | author = {Guo, Jianzhu and Zhu, Xiangyu and Yang, Yang and Yang, Fan and Lei, Zhen and Li, Stan Z}, 209 | booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)}, 210 | year = {2020} 211 | } 212 | 213 | @misc{3ddfa_cleardusk, 214 | author = {Guo, Jianzhu and Zhu, Xiangyu and Lei, Zhen}, 215 | title = {3DDFA}, 216 | howpublished = {\url{https://github.com/cleardusk/3DDFA}}, 217 | year = {2018} 218 | } 219 | 220 | ## Contact 221 | **Jianzhu Guo (郭建珠)** [[Homepage](https://guojianzhu.com), [Google Scholar](https://scholar.google.com/citations?user=W8_JzNcAAAAJ&hl=en&oi=ao)]: **guojianzhu1994@foxmail.com** or **guojianzhu1994@gmail.com** or **jianzhu.guo@nlpr.ia.ac.cn** (this email will be invalid soon). 222 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | torch 2 | torchvision 3 | matplotlib 4 | numpy 5 | opencv-python # cv2 6 | imageio 7 | imageio-ffmpeg 8 | pyyaml # yaml 9 | tqdm 10 | argparse 11 | cython 12 | scikit-image # skimage 13 | scipy 14 | onnxruntime 15 | gradio 16 | -------------------------------------------------------------------------------- /speed_cpu.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import timeit 6 | import numpy as np 7 | 8 | SETUP_CODE = ''' 9 | import os 10 | os.environ["OMP_NUM_THREADS"] = "4" 11 | 12 | import numpy as np 13 | import onnxruntime 14 | 15 | onnx_fp = "weights/mb1_120x120.onnx" # if not existed, convert it, see "convert_to_onnx function in utils/onnx.py" 16 | session = onnxruntime.InferenceSession(onnx_fp, None) 17 | 18 | img = np.random.randn(1, 3, 120, 120).astype(np.float32) 19 | ''' 20 | 21 | TEST_CODE = ''' 22 | session.run(None, {"input": img}) 23 | ''' 24 | 25 | 26 | def main(): 27 | repeat, number = 5, 100 28 | res = timeit.repeat(setup=SETUP_CODE, 29 | stmt=TEST_CODE, 30 | repeat=repeat, 31 | number=number) 32 | res = np.array(res, dtype=np.float32) 33 | res /= number 34 | mean, var = np.mean(res), np.std(res) 35 | print('Inference speed: {:.2f}±{:.2f} ms'.format(mean * 1000, var * 1000)) 36 | 37 | 38 | if __name__ == '__main__': 39 | main() 40 | -------------------------------------------------------------------------------- /utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/utils/__init__.py -------------------------------------------------------------------------------- /utils/asset/.gitignore: -------------------------------------------------------------------------------- 1 | *.so 2 | -------------------------------------------------------------------------------- /utils/asset/build_render_ctypes.sh: -------------------------------------------------------------------------------- 1 | gcc -shared -Wall -O3 render.c -o render.so -fPIC -------------------------------------------------------------------------------- /utils/asset/render.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | #define max(x, y) (((x) > (y)) ? (x) : (y)) 6 | #define min(x, y) (((x) < (y)) ? (x) : (y)) 7 | #define clip(_x, _min, _max) min(max(_x, _min), _max) 8 | 9 | struct Tuple3D 10 | { 11 | float x; 12 | float y; 13 | float z; 14 | }; 15 | 16 | void _render(const int *triangles, 17 | const int ntri, 18 | const float *light, 19 | const float *directional, 20 | const float *ambient, 21 | const float *vertices, 22 | const int nver, 23 | unsigned char *image, 24 | const int h, const int w) 25 | { 26 | int tri_p0_ind, tri_p1_ind, tri_p2_ind; 27 | int color_index; 28 | float dot00, dot01, dot11, dot02, dot12; 29 | float cos_sum, det; 30 | 31 | struct Tuple3D p0, p1, p2; 32 | struct Tuple3D v0, v1, v2; 33 | struct Tuple3D p, start, end; 34 | 35 | struct Tuple3D ver_max = {-1.0e8, -1.0e8, -1.0e8}; 36 | struct Tuple3D ver_min = {1.0e8, 1.0e8, 1.0e8}; 37 | struct Tuple3D ver_mean = {0.0, 0.0, 0.0}; 38 | 39 | float *ver_normal = (float *)calloc(3 * nver, sizeof(float)); 40 | float *colors = (float *)malloc(3 * nver * sizeof(float)); 41 | float *depth_buffer = (float *)calloc(h * w, sizeof(float)); 42 | 43 | for (int i = 0; i < ntri; i++) 44 | { 45 | tri_p0_ind = triangles[3 * i]; 46 | tri_p1_ind = triangles[3 * i + 1]; 47 | tri_p2_ind = triangles[3 * i + 2]; 48 | 49 | // counter clockwise order 50 | start.x = vertices[tri_p1_ind] - vertices[tri_p0_ind]; 51 | start.y = vertices[tri_p1_ind + 1] - vertices[tri_p0_ind + 1]; 52 | start.z = vertices[tri_p1_ind + 2] - vertices[tri_p0_ind + 2]; 53 | 54 | end.x = vertices[tri_p2_ind] - vertices[tri_p0_ind]; 55 | end.y = vertices[tri_p2_ind + 1] - vertices[tri_p0_ind + 1]; 56 | end.z = vertices[tri_p2_ind + 2] - vertices[tri_p0_ind + 2]; 57 | 58 | p.x = start.y * end.z - start.z * end.y; 59 | p.y = start.z * end.x - start.x * end.z; 60 | p.z = start.x * end.y - start.y * end.x; 61 | 62 | ver_normal[tri_p0_ind] += p.x; 63 | ver_normal[tri_p1_ind] += p.x; 64 | ver_normal[tri_p2_ind] += p.x; 65 | 66 | ver_normal[tri_p0_ind + 1] += p.y; 67 | ver_normal[tri_p1_ind + 1] += p.y; 68 | ver_normal[tri_p2_ind + 1] += p.y; 69 | 70 | ver_normal[tri_p0_ind + 2] += p.z; 71 | ver_normal[tri_p1_ind + 2] += p.z; 72 | ver_normal[tri_p2_ind + 2] += p.z; 73 | } 74 | 75 | for (int i = 0; i < nver; ++i) 76 | { 77 | p.x = ver_normal[3 * i]; 78 | p.y = ver_normal[3 * i + 1]; 79 | p.z = ver_normal[3 * i + 2]; 80 | 81 | det = sqrt(p.x * p.x + p.y * p.y + p.z * p.z); 82 | if (det <= 0) 83 | det = 1e-6; 84 | 85 | ver_normal[3 * i] /= det; 86 | ver_normal[3 * i + 1] /= det; 87 | ver_normal[3 * i + 2] /= det; 88 | 89 | ver_mean.x += p.x; 90 | ver_mean.y += p.y; 91 | ver_mean.z += p.z; 92 | 93 | ver_max.x = max(ver_max.x, p.x); 94 | ver_max.y = max(ver_max.y, p.y); 95 | ver_max.z = max(ver_max.z, p.z); 96 | 97 | ver_min.x = min(ver_min.x, p.x); 98 | ver_min.y = min(ver_min.y, p.y); 99 | ver_min.z = min(ver_min.z, p.z); 100 | } 101 | 102 | ver_mean.x /= nver; 103 | ver_mean.y /= nver; 104 | ver_mean.z /= nver; 105 | 106 | for (int i = 0; i < nver; ++i) 107 | { 108 | colors[3 * i] = vertices[3 * i]; 109 | colors[3 * i + 1] = vertices[3 * i + 1]; 110 | colors[3 * i + 2] = vertices[3 * i + 2]; 111 | 112 | colors[3 * i] -= ver_mean.x; 113 | colors[3 * i] /= ver_max.x - ver_min.x; 114 | 115 | colors[3 * i + 1] -= ver_mean.y; 116 | colors[3 * i + 1] /= ver_max.y - ver_min.y; 117 | 118 | colors[3 * i + 2] -= ver_mean.z; 119 | colors[3 * i + 2] /= ver_max.z - ver_min.z; 120 | 121 | p.x = light[0] - colors[3 * i]; 122 | p.y = light[1] - colors[3 * i + 1]; 123 | p.z = light[2] - colors[3 * i + 2]; 124 | 125 | det = sqrt(p.x * p.x + p.y * p.y + p.z * p.z); 126 | if (det <= 0) 127 | det = 1e-6; 128 | 129 | colors[3 * i] = p.x / det; 130 | colors[3 * i + 1] = p.y / det; 131 | colors[3 * i + 2] = p.z / det; 132 | 133 | colors[3 * i] *= ver_normal[3 * i]; 134 | colors[3 * i + 1] *= ver_normal[3 * i + 1]; 135 | colors[3 * i + 2] *= ver_normal[3 * i + 2]; 136 | 137 | cos_sum = colors[3 * i] + colors[3 * i + 1] + colors[3 * i + 2]; 138 | 139 | colors[3 * i] = clip(cos_sum * directional[0] + ambient[0], 0, 1); 140 | colors[3 * i + 1] = clip(cos_sum * directional[1] + ambient[1], 0, 1); 141 | colors[3 * i + 2] = clip(cos_sum * directional[2] + ambient[2], 0, 1); 142 | } 143 | 144 | for (int i = 0; i < ntri; ++i) 145 | { 146 | tri_p0_ind = triangles[3 * i]; 147 | tri_p1_ind = triangles[3 * i + 1]; 148 | tri_p2_ind = triangles[3 * i + 2]; 149 | 150 | p0.x = vertices[tri_p0_ind]; 151 | p0.y = vertices[tri_p0_ind + 1]; 152 | p0.z = vertices[tri_p0_ind + 2]; 153 | 154 | p1.x = vertices[tri_p1_ind]; 155 | p1.y = vertices[tri_p1_ind + 1]; 156 | p1.z = vertices[tri_p1_ind + 2]; 157 | 158 | p2.x = vertices[tri_p2_ind]; 159 | p2.y = vertices[tri_p2_ind + 1]; 160 | p2.z = vertices[tri_p2_ind + 2]; 161 | 162 | start.x = max(ceil(min(p0.x, min(p1.x, p2.x))), 0); 163 | end.x = min(floor(max(p0.x, max(p1.x, p2.x))), w - 1); 164 | 165 | start.y = max(ceil(min(p0.y, min(p1.y, p2.y))), 0); 166 | end.y = min(floor(max(p0.y, max(p1.y, p2.y))), h - 1); 167 | 168 | if (end.x < start.x || end.y < start.y) 169 | continue; 170 | 171 | v0.x = p2.x - p0.x; 172 | v0.y = p2.y - p0.y; 173 | v1.x = p1.x - p0.x; 174 | v1.y = p1.y - p0.y; 175 | 176 | // dot products np.dot(v0.T, v0) 177 | dot00 = v0.x * v0.x + v0.y * v0.y; 178 | dot01 = v0.x * v1.x + v0.y * v1.y; 179 | dot11 = v1.x * v1.x + v1.y * v1.y; 180 | 181 | // barycentric coordinates 182 | start.z = dot00 * dot11 - dot01 * dot01; 183 | if (start.z != 0) 184 | start.z = 1 / start.z; 185 | 186 | for (p.y = start.y; p.y <= end.y; p.y += 1.0) 187 | { 188 | for (p.x = start.x; p.x <= end.x; p.x += 1.0) 189 | { 190 | v2.x = p.x - p0.x; 191 | v2.y = p.y - p0.y; 192 | 193 | dot02 = v0.x * v2.x + v0.y * v2.y; 194 | dot12 = v1.x * v2.x + v1.y * v2.y; 195 | 196 | v2.z = (dot11 * dot02 - dot01 * dot12) * start.z; 197 | v1.z = (dot00 * dot12 - dot01 * dot02) * start.z; 198 | v0.z = 1 - v2.z - v1.z; 199 | 200 | // judge is_point_in_tri by below line of code 201 | if (v2.z > 0 && v1.z > 0 && v0.z > 0) 202 | { 203 | p.z = v0.z * p0.z + v1.z * p1.z + v2.z * p2.z; 204 | color_index = p.y * w + p.x; 205 | 206 | if (p.z > depth_buffer[color_index]) 207 | { 208 | end.z = v0.z * colors[tri_p0_ind]; 209 | end.z += v1.z * colors[tri_p1_ind]; 210 | end.z += v2.z * colors[tri_p2_ind]; 211 | image[3 * color_index] = end.z * 255; 212 | 213 | end.z = v0.z * colors[tri_p0_ind + 1]; 214 | end.z += v1.z * colors[tri_p1_ind + 1]; 215 | end.z += v2.z * colors[tri_p2_ind + 1]; 216 | image[3 * color_index + 1] = end.z * 255; 217 | 218 | end.z = v0.z * colors[tri_p0_ind + 2]; 219 | end.z += v1.z * colors[tri_p1_ind + 2]; 220 | end.z += v2.z * colors[tri_p2_ind + 2]; 221 | image[3 * color_index + 2] = end.z * 255; 222 | 223 | depth_buffer[color_index] = p.z; 224 | } 225 | } 226 | } 227 | } 228 | } 229 | 230 | free(depth_buffer); 231 | free(colors); 232 | free(ver_normal); 233 | } 234 | -------------------------------------------------------------------------------- /utils/depth.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import sys 6 | 7 | sys.path.append('..') 8 | 9 | import cv2 10 | import numpy as np 11 | 12 | from Sim3DR import rasterize 13 | from utils.functions import plot_image 14 | from .tddfa_util import _to_ctype 15 | 16 | 17 | def depth(img, ver_lst, tri, show_flag=False, wfp=None, with_bg_flag=True): 18 | if with_bg_flag: 19 | overlap = img.copy() 20 | else: 21 | overlap = np.zeros_like(img) 22 | 23 | for ver_ in ver_lst: 24 | ver = _to_ctype(ver_.T) # transpose 25 | 26 | z = ver[:, 2] 27 | z_min, z_max = min(z), max(z) 28 | 29 | z = (z - z_min) / (z_max - z_min) 30 | 31 | # expand 32 | z = np.repeat(z[:, np.newaxis], 3, axis=1) 33 | 34 | overlap = rasterize(ver, tri, z, bg=overlap) 35 | 36 | if wfp is not None: 37 | cv2.imwrite(wfp, overlap) 38 | print(f'Save visualization result to {wfp}') 39 | 40 | if show_flag: 41 | plot_image(overlap) 42 | 43 | return overlap 44 | -------------------------------------------------------------------------------- /utils/functions.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import numpy as np 6 | import cv2 7 | from math import sqrt 8 | import matplotlib.pyplot as plt 9 | 10 | RED = (0, 0, 255) 11 | GREEN = (0, 255, 0) 12 | BLUE = (255, 0, 0) 13 | 14 | 15 | def get_suffix(filename): 16 | """a.jpg -> jpg""" 17 | pos = filename.rfind('.') 18 | if pos == -1: 19 | return '' 20 | return filename[pos:] 21 | 22 | 23 | def crop_img(img, roi_box): 24 | h, w = img.shape[:2] 25 | 26 | sx, sy, ex, ey = [int(round(_)) for _ in roi_box] 27 | dh, dw = ey - sy, ex - sx 28 | if len(img.shape) == 3: 29 | res = np.zeros((dh, dw, 3), dtype=np.uint8) 30 | else: 31 | res = np.zeros((dh, dw), dtype=np.uint8) 32 | if sx < 0: 33 | sx, dsx = 0, -sx 34 | else: 35 | dsx = 0 36 | 37 | if ex > w: 38 | ex, dex = w, dw - (ex - w) 39 | else: 40 | dex = dw 41 | 42 | if sy < 0: 43 | sy, dsy = 0, -sy 44 | else: 45 | dsy = 0 46 | 47 | if ey > h: 48 | ey, dey = h, dh - (ey - h) 49 | else: 50 | dey = dh 51 | 52 | res[dsy:dey, dsx:dex] = img[sy:ey, sx:ex] 53 | return res 54 | 55 | 56 | def calc_hypotenuse(pts): 57 | bbox = [min(pts[0, :]), min(pts[1, :]), max(pts[0, :]), max(pts[1, :])] 58 | center = [(bbox[0] + bbox[2]) / 2, (bbox[1] + bbox[3]) / 2] 59 | radius = max(bbox[2] - bbox[0], bbox[3] - bbox[1]) / 2 60 | bbox = [center[0] - radius, center[1] - radius, center[0] + radius, center[1] + radius] 61 | llength = sqrt((bbox[2] - bbox[0]) ** 2 + (bbox[3] - bbox[1]) ** 2) 62 | return llength / 3 63 | 64 | 65 | def parse_roi_box_from_landmark(pts): 66 | """calc roi box from landmark""" 67 | bbox = [min(pts[0, :]), min(pts[1, :]), max(pts[0, :]), max(pts[1, :])] 68 | center = [(bbox[0] + bbox[2]) / 2, (bbox[1] + bbox[3]) / 2] 69 | radius = max(bbox[2] - bbox[0], bbox[3] - bbox[1]) / 2 70 | bbox = [center[0] - radius, center[1] - radius, center[0] + radius, center[1] + radius] 71 | 72 | llength = sqrt((bbox[2] - bbox[0]) ** 2 + (bbox[3] - bbox[1]) ** 2) 73 | center_x = (bbox[2] + bbox[0]) / 2 74 | center_y = (bbox[3] + bbox[1]) / 2 75 | 76 | roi_box = [0] * 4 77 | roi_box[0] = center_x - llength / 2 78 | roi_box[1] = center_y - llength / 2 79 | roi_box[2] = roi_box[0] + llength 80 | roi_box[3] = roi_box[1] + llength 81 | 82 | return roi_box 83 | 84 | 85 | def parse_roi_box_from_bbox(bbox): 86 | left, top, right, bottom = bbox[:4] 87 | old_size = (right - left + bottom - top) / 2 88 | center_x = right - (right - left) / 2.0 89 | center_y = bottom - (bottom - top) / 2.0 + old_size * 0.14 90 | size = int(old_size * 1.58) 91 | 92 | roi_box = [0] * 4 93 | roi_box[0] = center_x - size / 2 94 | roi_box[1] = center_y - size / 2 95 | roi_box[2] = roi_box[0] + size 96 | roi_box[3] = roi_box[1] + size 97 | 98 | return roi_box 99 | 100 | 101 | def plot_image(img): 102 | height, width = img.shape[:2] 103 | plt.figure(figsize=(12, height / width * 12)) 104 | 105 | plt.subplots_adjust(left=0, right=1, top=1, bottom=0) 106 | plt.axis('off') 107 | 108 | plt.imshow(img[..., ::-1]) 109 | plt.show() 110 | 111 | 112 | def draw_landmarks(img, pts, style='fancy', wfp=None, show_flag=False, **kwargs): 113 | """Draw landmarks using matplotlib""" 114 | height, width = img.shape[:2] 115 | plt.figure(figsize=(12, height / width * 12)) 116 | plt.imshow(img[..., ::-1]) 117 | plt.subplots_adjust(left=0, right=1, top=1, bottom=0) 118 | plt.axis('off') 119 | 120 | dense_flag = kwargs.get('dense_flag') 121 | 122 | if not type(pts) in [tuple, list]: 123 | pts = [pts] 124 | for i in range(len(pts)): 125 | if dense_flag: 126 | plt.plot(pts[i][0, ::6], pts[i][1, ::6], 'o', markersize=0.4, color='c', alpha=0.7) 127 | else: 128 | alpha = 0.8 129 | markersize = 4 130 | lw = 1.5 131 | color = kwargs.get('color', 'w') 132 | markeredgecolor = kwargs.get('markeredgecolor', 'black') 133 | 134 | nums = [0, 17, 22, 27, 31, 36, 42, 48, 60, 68] 135 | 136 | # close eyes and mouths 137 | plot_close = lambda i1, i2: plt.plot([pts[i][0, i1], pts[i][0, i2]], [pts[i][1, i1], pts[i][1, i2]], 138 | color=color, lw=lw, alpha=alpha - 0.1) 139 | plot_close(41, 36) 140 | plot_close(47, 42) 141 | plot_close(59, 48) 142 | plot_close(67, 60) 143 | 144 | for ind in range(len(nums) - 1): 145 | l, r = nums[ind], nums[ind + 1] 146 | plt.plot(pts[i][0, l:r], pts[i][1, l:r], color=color, lw=lw, alpha=alpha - 0.1) 147 | 148 | plt.plot(pts[i][0, l:r], pts[i][1, l:r], marker='o', linestyle='None', markersize=markersize, 149 | color=color, 150 | markeredgecolor=markeredgecolor, alpha=alpha) 151 | if wfp is not None: 152 | plt.savefig(wfp, dpi=150) 153 | print(f'Save visualization result to {wfp}') 154 | 155 | if show_flag: 156 | plt.show() 157 | 158 | 159 | def cv_draw_landmark(img_ori, pts, box=None, color=GREEN, size=1): 160 | img = img_ori.copy() 161 | n = pts.shape[1] 162 | if n <= 106: 163 | for i in range(n): 164 | cv2.circle(img, (int(round(pts[0, i])), int(round(pts[1, i]))), size, color, -1) 165 | else: 166 | sep = 1 167 | for i in range(0, n, sep): 168 | cv2.circle(img, (int(round(pts[0, i])), int(round(pts[1, i]))), size, color, 1) 169 | 170 | if box is not None: 171 | left, top, right, bottom = np.round(box).astype(np.int32) 172 | left_top = (left, top) 173 | right_top = (right, top) 174 | right_bottom = (right, bottom) 175 | left_bottom = (left, bottom) 176 | cv2.line(img, left_top, right_top, BLUE, 1, cv2.LINE_AA) 177 | cv2.line(img, right_top, right_bottom, BLUE, 1, cv2.LINE_AA) 178 | cv2.line(img, right_bottom, left_bottom, BLUE, 1, cv2.LINE_AA) 179 | cv2.line(img, left_bottom, left_top, BLUE, 1, cv2.LINE_AA) 180 | 181 | return img 182 | -------------------------------------------------------------------------------- /utils/io.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import os 6 | import numpy as np 7 | import torch 8 | import pickle 9 | 10 | 11 | def mkdir(d): 12 | os.makedirs(d, exist_ok=True) 13 | 14 | 15 | def _get_suffix(filename): 16 | """a.jpg -> jpg""" 17 | pos = filename.rfind('.') 18 | if pos == -1: 19 | return '' 20 | return filename[pos + 1:] 21 | 22 | 23 | def _load(fp): 24 | suffix = _get_suffix(fp) 25 | if suffix == 'npy': 26 | return np.load(fp) 27 | elif suffix == 'pkl': 28 | return pickle.load(open(fp, 'rb')) 29 | 30 | 31 | def _dump(wfp, obj): 32 | suffix = _get_suffix(wfp) 33 | if suffix == 'npy': 34 | np.save(wfp, obj) 35 | elif suffix == 'pkl': 36 | pickle.dump(obj, open(wfp, 'wb')) 37 | else: 38 | raise Exception('Unknown Type: {}'.format(suffix)) 39 | 40 | 41 | def _load_tensor(fp, mode='cpu'): 42 | if mode.lower() == 'cpu': 43 | return torch.from_numpy(_load(fp)) 44 | elif mode.lower() == 'gpu': 45 | return torch.from_numpy(_load(fp)).cuda() 46 | 47 | 48 | def _tensor_to_cuda(x): 49 | if x.is_cuda: 50 | return x 51 | else: 52 | return x.cuda() 53 | 54 | 55 | def _load_gpu(fp): 56 | return torch.from_numpy(_load(fp)).cuda() 57 | 58 | 59 | _load_cpu = _load 60 | _numpy_to_tensor = lambda x: torch.from_numpy(x) 61 | _tensor_to_numpy = lambda x: x.numpy() 62 | _numpy_to_cuda = lambda x: _tensor_to_cuda(torch.from_numpy(x)) 63 | _cuda_to_tensor = lambda x: x.cpu() 64 | _cuda_to_numpy = lambda x: x.cpu().numpy() 65 | -------------------------------------------------------------------------------- /utils/onnx.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import sys 6 | 7 | sys.path.append('..') 8 | 9 | import torch 10 | import models 11 | from utils.tddfa_util import load_model 12 | 13 | 14 | def convert_to_onnx(**kvs): 15 | # 1. load model 16 | size = kvs.get('size', 120) 17 | model = getattr(models, kvs.get('arch'))( 18 | num_classes=kvs.get('num_params', 62), 19 | widen_factor=kvs.get('widen_factor', 1), 20 | size=size, 21 | mode=kvs.get('mode', 'small') 22 | ) 23 | checkpoint_fp = kvs.get('checkpoint_fp') 24 | model = load_model(model, checkpoint_fp) 25 | model.eval() 26 | 27 | # 2. convert 28 | batch_size = 1 29 | dummy_input = torch.randn(batch_size, 3, size, size) 30 | wfp = checkpoint_fp.replace('.pth', '.onnx') 31 | torch.onnx.export( 32 | model, 33 | (dummy_input, ), 34 | wfp, 35 | input_names=['input'], 36 | output_names=['output'], 37 | do_constant_folding=True 38 | ) 39 | print(f'Convert {checkpoint_fp} to {wfp} done.') 40 | return wfp 41 | -------------------------------------------------------------------------------- /utils/pncc.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import sys 6 | 7 | sys.path.append('..') 8 | 9 | import cv2 10 | import numpy as np 11 | import os.path as osp 12 | 13 | from Sim3DR import rasterize 14 | from utils.functions import plot_image 15 | from utils.io import _load, _dump 16 | from utils.tddfa_util import _to_ctype 17 | 18 | make_abs_path = lambda fn: osp.join(osp.dirname(osp.realpath(__file__)), fn) 19 | 20 | 21 | def calc_ncc_code(): 22 | from bfm import bfm 23 | 24 | # formula: ncc_d = ( u_d - min(u_d) ) / ( max(u_d) - min(u_d) ), d = {r, g, b} 25 | u = bfm.u 26 | u = u.reshape(3, -1, order='F') 27 | 28 | for i in range(3): 29 | u[i] = (u[i] - u[i].min()) / (u[i].max() - u[i].min()) 30 | 31 | _dump('../configs/ncc_code.npy', u) 32 | 33 | 34 | def pncc(img, ver_lst, tri, show_flag=False, wfp=None, with_bg_flag=True): 35 | ncc_code = _load(make_abs_path('../configs/ncc_code.npy')) 36 | 37 | if with_bg_flag: 38 | overlap = img.copy() 39 | else: 40 | overlap = np.zeros_like(img) 41 | 42 | # rendering pncc 43 | for ver_ in ver_lst: 44 | ver = _to_ctype(ver_.T) # transpose 45 | overlap = rasterize(ver, tri, ncc_code.T, bg=overlap) # m x 3 46 | 47 | if wfp is not None: 48 | cv2.imwrite(wfp, overlap) 49 | print(f'Save visualization result to {wfp}') 50 | 51 | if show_flag: 52 | plot_image(overlap) 53 | 54 | return overlap 55 | 56 | 57 | def main(): 58 | # `configs/ncc_code.npy` is generated by `calc_nnc_code` function 59 | # calc_ncc_code() 60 | pass 61 | 62 | 63 | if __name__ == '__main__': 64 | main() 65 | -------------------------------------------------------------------------------- /utils/pose.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | """ 4 | Reference: https://github.com/YadiraF/PRNet/blob/master/utils/estimate_pose.py 5 | 6 | Calculating pose from the output 3DMM parameters, you can also try to use solvePnP to perform estimation 7 | """ 8 | 9 | __author__ = 'cleardusk' 10 | 11 | import cv2 12 | import numpy as np 13 | from math import cos, sin, atan2, asin, sqrt 14 | 15 | from .functions import calc_hypotenuse, plot_image 16 | 17 | 18 | def P2sRt(P): 19 | """ decompositing camera matrix P. 20 | Args: 21 | P: (3, 4). Affine Camera Matrix. 22 | Returns: 23 | s: scale factor. 24 | R: (3, 3). rotation matrix. 25 | t2d: (2,). 2d translation. 26 | """ 27 | t3d = P[:, 3] 28 | R1 = P[0:1, :3] 29 | R2 = P[1:2, :3] 30 | s = (np.linalg.norm(R1) + np.linalg.norm(R2)) / 2.0 31 | r1 = R1 / np.linalg.norm(R1) 32 | r2 = R2 / np.linalg.norm(R2) 33 | r3 = np.cross(r1, r2) 34 | 35 | R = np.concatenate((r1, r2, r3), 0) 36 | return s, R, t3d 37 | 38 | 39 | def matrix2angle(R): 40 | """ compute three Euler angles from a Rotation Matrix. Ref: http://www.gregslabaugh.net/publications/euler.pdf 41 | refined by: https://stackoverflow.com/questions/43364900/rotation-matrix-to-euler-angles-with-opencv 42 | todo: check and debug 43 | Args: 44 | R: (3,3). rotation matrix 45 | Returns: 46 | x: yaw 47 | y: pitch 48 | z: roll 49 | """ 50 | if R[2, 0] > 0.998: 51 | z = 0 52 | x = np.pi / 2 53 | y = z + atan2(-R[0, 1], -R[0, 2]) 54 | elif R[2, 0] < -0.998: 55 | z = 0 56 | x = -np.pi / 2 57 | y = -z + atan2(R[0, 1], R[0, 2]) 58 | else: 59 | x = asin(R[2, 0]) 60 | y = atan2(R[2, 1] / cos(x), R[2, 2] / cos(x)) 61 | z = atan2(R[1, 0] / cos(x), R[0, 0] / cos(x)) 62 | 63 | return x, y, z 64 | 65 | 66 | def calc_pose(param): 67 | P = param[:12].reshape(3, -1) # camera matrix 68 | s, R, t3d = P2sRt(P) 69 | P = np.concatenate((R, t3d.reshape(3, -1)), axis=1) # without scale 70 | pose = matrix2angle(R) 71 | pose = [p * 180 / np.pi for p in pose] 72 | 73 | return P, pose 74 | 75 | 76 | def build_camera_box(rear_size=90): 77 | point_3d = [] 78 | rear_depth = 0 79 | point_3d.append((-rear_size, -rear_size, rear_depth)) 80 | point_3d.append((-rear_size, rear_size, rear_depth)) 81 | point_3d.append((rear_size, rear_size, rear_depth)) 82 | point_3d.append((rear_size, -rear_size, rear_depth)) 83 | point_3d.append((-rear_size, -rear_size, rear_depth)) 84 | 85 | front_size = int(4 / 3 * rear_size) 86 | front_depth = int(4 / 3 * rear_size) 87 | point_3d.append((-front_size, -front_size, front_depth)) 88 | point_3d.append((-front_size, front_size, front_depth)) 89 | point_3d.append((front_size, front_size, front_depth)) 90 | point_3d.append((front_size, -front_size, front_depth)) 91 | point_3d.append((-front_size, -front_size, front_depth)) 92 | point_3d = np.array(point_3d, dtype=np.float32).reshape(-1, 3) 93 | 94 | return point_3d 95 | 96 | 97 | def plot_pose_box(img, P, ver, color=(40, 255, 0), line_width=2): 98 | """ Draw a 3D box as annotation of pose. 99 | Ref:https://github.com/yinguobing/head-pose-estimation/blob/master/pose_estimator.py 100 | Args: 101 | img: the input image 102 | P: (3, 4). Affine Camera Matrix. 103 | kpt: (2, 68) or (3, 68) 104 | """ 105 | llength = calc_hypotenuse(ver) 106 | point_3d = build_camera_box(llength) 107 | # Map to 2d image points 108 | point_3d_homo = np.hstack((point_3d, np.ones([point_3d.shape[0], 1]))) # n x 4 109 | point_2d = point_3d_homo.dot(P.T)[:, :2] 110 | 111 | point_2d[:, 1] = - point_2d[:, 1] 112 | point_2d[:, :2] = point_2d[:, :2] - np.mean(point_2d[:4, :2], 0) + np.mean(ver[:2, :27], 1) 113 | point_2d = np.int32(point_2d.reshape(-1, 2)) 114 | 115 | # Draw all the lines 116 | cv2.polylines(img, [point_2d], True, color, line_width, cv2.LINE_AA) 117 | cv2.line(img, tuple(point_2d[1]), tuple( 118 | point_2d[6]), color, line_width, cv2.LINE_AA) 119 | cv2.line(img, tuple(point_2d[2]), tuple( 120 | point_2d[7]), color, line_width, cv2.LINE_AA) 121 | cv2.line(img, tuple(point_2d[3]), tuple( 122 | point_2d[8]), color, line_width, cv2.LINE_AA) 123 | 124 | return img 125 | 126 | 127 | def viz_pose(img, param_lst, ver_lst, show_flag=False, wfp=None): 128 | for param, ver in zip(param_lst, ver_lst): 129 | P, pose = calc_pose(param) 130 | img = plot_pose_box(img, P, ver) 131 | # print(P[:, :3]) 132 | print(f'yaw: {pose[0]:.1f}, pitch: {pose[1]:.1f}, roll: {pose[2]:.1f}') 133 | 134 | if wfp is not None: 135 | cv2.imwrite(wfp, img) 136 | print(f'Save visualization result to {wfp}') 137 | 138 | if show_flag: 139 | plot_image(img) 140 | 141 | return img 142 | -------------------------------------------------------------------------------- /utils/render.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import sys 6 | 7 | sys.path.append('..') 8 | 9 | import cv2 10 | import numpy as np 11 | 12 | from Sim3DR import RenderPipeline 13 | from utils.functions import plot_image 14 | from .tddfa_util import _to_ctype 15 | 16 | cfg = { 17 | 'intensity_ambient': 0.3, 18 | 'color_ambient': (1, 1, 1), 19 | 'intensity_directional': 0.6, 20 | 'color_directional': (1, 1, 1), 21 | 'intensity_specular': 0.1, 22 | 'specular_exp': 5, 23 | 'light_pos': (0, 0, 5), 24 | 'view_pos': (0, 0, 5) 25 | } 26 | 27 | render_app = RenderPipeline(**cfg) 28 | 29 | 30 | def render(img, ver_lst, tri, alpha=0.6, show_flag=False, wfp=None, with_bg_flag=True): 31 | if with_bg_flag: 32 | overlap = img.copy() 33 | else: 34 | overlap = np.zeros_like(img) 35 | 36 | for ver_ in ver_lst: 37 | ver = _to_ctype(ver_.T) # transpose 38 | overlap = render_app(ver, tri, overlap) 39 | 40 | if with_bg_flag: 41 | res = cv2.addWeighted(img, 1 - alpha, overlap, alpha, 0) 42 | else: 43 | res = overlap 44 | 45 | if wfp is not None: 46 | cv2.imwrite(wfp, res) 47 | print(f'Save visualization result to {wfp}') 48 | 49 | if show_flag: 50 | plot_image(res) 51 | 52 | return res 53 | -------------------------------------------------------------------------------- /utils/render_ctypes.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | """ 4 | Borrowed from https://github.com/1996scarlet/Dense-Head-Pose-Estimation/blob/main/service/CtypesMeshRender.py 5 | 6 | To use this render, you should build the clib first: 7 | ``` 8 | cd utils/asset 9 | gcc -shared -Wall -O3 render.c -o render.so -fPIC 10 | cd ../.. 11 | ``` 12 | """ 13 | 14 | import sys 15 | 16 | sys.path.append('..') 17 | 18 | import os.path as osp 19 | import cv2 20 | import numpy as np 21 | import ctypes 22 | from utils.functions import plot_image 23 | 24 | make_abs_path = lambda fn: osp.join(osp.dirname(osp.realpath(__file__)), fn) 25 | 26 | 27 | class TrianglesMeshRender(object): 28 | def __init__( 29 | self, 30 | clibs, 31 | light=(0, 0, 5), 32 | direction=(0.6, 0.6, 0.6), 33 | ambient=(0.3, 0.3, 0.3) 34 | ): 35 | if not osp.exists(clibs): 36 | raise Exception(f'{clibs} not found, please build it first, by run ' 37 | f'"gcc -shared -Wall -O3 render.c -o render.so -fPIC" in utils/asset directory') 38 | 39 | self._clibs = ctypes.CDLL(clibs) 40 | 41 | self._light = np.array(light, dtype=np.float32) 42 | self._light = np.ctypeslib.as_ctypes(self._light) 43 | 44 | self._direction = np.array(direction, dtype=np.float32) 45 | self._direction = np.ctypeslib.as_ctypes(self._direction) 46 | 47 | self._ambient = np.array(ambient, dtype=np.float32) 48 | self._ambient = np.ctypeslib.as_ctypes(self._ambient) 49 | 50 | def __call__(self, vertices, triangles, bg): 51 | self.triangles = np.ctypeslib.as_ctypes(3 * triangles) # Attention 52 | self.tri_nums = triangles.shape[0] 53 | 54 | self._clibs._render( 55 | self.triangles, self.tri_nums, 56 | self._light, self._direction, self._ambient, 57 | np.ctypeslib.as_ctypes(vertices), 58 | vertices.shape[0], 59 | np.ctypeslib.as_ctypes(bg), 60 | bg.shape[0], bg.shape[1] 61 | ) 62 | 63 | 64 | render_app = TrianglesMeshRender(clibs=make_abs_path('asset/render.so')) 65 | 66 | 67 | def render(img, ver_lst, tri, alpha=0.6, show_flag=False, wfp=None, with_bg_flag=True): 68 | if with_bg_flag: 69 | overlap = img.copy() 70 | else: 71 | overlap = np.zeros_like(img) 72 | 73 | for ver_ in ver_lst: 74 | ver = np.ascontiguousarray(ver_.T) # transpose 75 | render_app(ver, tri, bg=overlap) 76 | 77 | if with_bg_flag: 78 | res = cv2.addWeighted(img, 1 - alpha, overlap, alpha, 0) 79 | else: 80 | res = overlap 81 | 82 | if wfp is not None: 83 | cv2.imwrite(wfp, res) 84 | print(f'Save visualization result to {wfp}') 85 | 86 | if show_flag: 87 | plot_image(res) 88 | 89 | return res 90 | -------------------------------------------------------------------------------- /utils/serialization.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import numpy as np 6 | 7 | from .tddfa_util import _to_ctype 8 | from .functions import get_suffix 9 | 10 | header_temp = """ply 11 | format ascii 1.0 12 | element vertex {} 13 | property float x 14 | property float y 15 | property float z 16 | element face {} 17 | property list uchar int vertex_indices 18 | end_header 19 | """ 20 | 21 | 22 | def ser_to_ply_single(ver_lst, tri, height, wfp, reverse=True): 23 | suffix = get_suffix(wfp) 24 | 25 | for i, ver in enumerate(ver_lst): 26 | wfp_new = wfp.replace(suffix, f'_{i + 1}{suffix}') 27 | 28 | n_vertex = ver.shape[1] 29 | n_face = tri.shape[0] 30 | header = header_temp.format(n_vertex, n_face) 31 | 32 | with open(wfp_new, 'w') as f: 33 | f.write(header + '\n') 34 | for i in range(n_vertex): 35 | x, y, z = ver[:, i] 36 | if reverse: 37 | f.write(f'{x:.2f} {height-y:.2f} {z:.2f}\n') 38 | else: 39 | f.write(f'{x:.2f} {y:.2f} {z:.2f}\n') 40 | for i in range(n_face): 41 | idx1, idx2, idx3 = tri[i] # m x 3 42 | if reverse: 43 | f.write(f'3 {idx3} {idx2} {idx1}\n') 44 | else: 45 | f.write(f'3 {idx1} {idx2} {idx3}\n') 46 | 47 | print(f'Dump tp {wfp_new}') 48 | 49 | 50 | def ser_to_ply_multiple(ver_lst, tri, height, wfp, reverse=True): 51 | n_ply = len(ver_lst) # count ply 52 | 53 | if n_ply <= 0: 54 | return 55 | 56 | n_vertex = ver_lst[0].shape[1] 57 | n_face = tri.shape[0] 58 | header = header_temp.format(n_vertex * n_ply, n_face * n_ply) 59 | 60 | with open(wfp, 'w') as f: 61 | f.write(header + '\n') 62 | 63 | for i in range(n_ply): 64 | ver = ver_lst[i] 65 | for j in range(n_vertex): 66 | x, y, z = ver[:, j] 67 | if reverse: 68 | f.write(f'{x:.2f} {height - y:.2f} {z:.2f}\n') 69 | else: 70 | f.write(f'{x:.2f} {y:.2f} {z:.2f}\n') 71 | 72 | for i in range(n_ply): 73 | offset = i * n_vertex 74 | for j in range(n_face): 75 | idx1, idx2, idx3 = tri[j] # m x 3 76 | if reverse: 77 | f.write(f'3 {idx3 + offset} {idx2 + offset} {idx1 + offset}\n') 78 | else: 79 | f.write(f'3 {idx1 + offset} {idx2 + offset} {idx3 + offset}\n') 80 | 81 | print(f'Dump tp {wfp}') 82 | 83 | 84 | def get_colors(img, ver): 85 | h, w, _ = img.shape 86 | ver[0, :] = np.minimum(np.maximum(ver[0, :], 0), w - 1) # x 87 | ver[1, :] = np.minimum(np.maximum(ver[1, :], 0), h - 1) # y 88 | ind = np.round(ver).astype(np.int32) 89 | colors = img[ind[1, :], ind[0, :], :] / 255. # n x 3 90 | 91 | return colors.copy() 92 | 93 | 94 | def ser_to_obj_single(img, ver_lst, tri, height, wfp): 95 | suffix = get_suffix(wfp) 96 | 97 | n_face = tri.shape[0] 98 | for i, ver in enumerate(ver_lst): 99 | colors = get_colors(img, ver) 100 | 101 | n_vertex = ver.shape[1] 102 | 103 | wfp_new = wfp.replace(suffix, f'_{i + 1}{suffix}') 104 | 105 | with open(wfp_new, 'w') as f: 106 | for i in range(n_vertex): 107 | x, y, z = ver[:, i] 108 | f.write( 109 | f'v {x:.2f} {height - y:.2f} {z:.2f} {colors[i, 2]:.2f} {colors[i, 1]:.2f} {colors[i, 0]:.2f}\n') 110 | for i in range(n_face): 111 | idx1, idx2, idx3 = tri[i] # m x 3 112 | f.write(f'f {idx3 + 1} {idx2 + 1} {idx1 + 1}\n') 113 | 114 | print(f'Dump tp {wfp_new}') 115 | 116 | 117 | def ser_to_obj_multiple(img, ver_lst, tri, height, wfp): 118 | n_obj = len(ver_lst) # count obj 119 | 120 | if n_obj <= 0: 121 | return 122 | 123 | n_vertex = ver_lst[0].shape[1] 124 | n_face = tri.shape[0] 125 | 126 | with open(wfp, 'w') as f: 127 | for i in range(n_obj): 128 | ver = ver_lst[i] 129 | colors = get_colors(img, ver) 130 | 131 | for j in range(n_vertex): 132 | x, y, z = ver[:, j] 133 | f.write( 134 | f'v {x:.2f} {height - y:.2f} {z:.2f} {colors[j, 2]:.2f} {colors[j, 1]:.2f} {colors[j, 0]:.2f}\n') 135 | 136 | for i in range(n_obj): 137 | offset = i * n_vertex 138 | for j in range(n_face): 139 | idx1, idx2, idx3 = tri[j] # m x 3 140 | f.write(f'f {idx3 + 1 + offset} {idx2 + 1 + offset} {idx1 + 1 + offset}\n') 141 | 142 | print(f'Dump tp {wfp}') 143 | 144 | 145 | ser_to_ply = ser_to_ply_multiple # ser_to_ply_single 146 | ser_to_obj = ser_to_obj_multiple # ser_to_obj_multiple 147 | -------------------------------------------------------------------------------- /utils/tddfa_util.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import sys 6 | 7 | sys.path.append('..') 8 | 9 | import argparse 10 | import numpy as np 11 | import torch 12 | 13 | 14 | def _to_ctype(arr): 15 | if not arr.flags.c_contiguous: 16 | return arr.copy(order='C') 17 | return arr 18 | 19 | 20 | def str2bool(v): 21 | if v.lower() in ('yes', 'true', 't', 'y', '1'): 22 | return True 23 | elif v.lower() in ('no', 'false', 'f', 'n', '0'): 24 | return False 25 | else: 26 | raise argparse.ArgumentTypeError('Boolean value expected') 27 | 28 | 29 | def load_model(model, checkpoint_fp): 30 | checkpoint = torch.load(checkpoint_fp, map_location=lambda storage, loc: storage)['state_dict'] 31 | model_dict = model.state_dict() 32 | # because the model is trained by multiple gpus, prefix module should be removed 33 | for k in checkpoint.keys(): 34 | kc = k.replace('module.', '') 35 | if kc in model_dict.keys(): 36 | model_dict[kc] = checkpoint[k] 37 | if kc in ['fc_param.bias', 'fc_param.weight']: 38 | model_dict[kc.replace('_param', '')] = checkpoint[k] 39 | 40 | model.load_state_dict(model_dict) 41 | return model 42 | 43 | 44 | class ToTensorGjz(object): 45 | def __call__(self, pic): 46 | if isinstance(pic, np.ndarray): 47 | img = torch.from_numpy(pic.transpose((2, 0, 1))) 48 | return img.float() 49 | 50 | def __repr__(self): 51 | return self.__class__.__name__ + '()' 52 | 53 | 54 | class NormalizeGjz(object): 55 | def __init__(self, mean, std): 56 | self.mean = mean 57 | self.std = std 58 | 59 | def __call__(self, tensor): 60 | tensor.sub_(self.mean).div_(self.std) 61 | return tensor 62 | 63 | 64 | def similar_transform(pts3d, roi_box, size): 65 | pts3d[0, :] -= 1 # for Python compatibility 66 | pts3d[2, :] -= 1 67 | pts3d[1, :] = size - pts3d[1, :] 68 | 69 | sx, sy, ex, ey = roi_box 70 | scale_x = (ex - sx) / size 71 | scale_y = (ey - sy) / size 72 | pts3d[0, :] = pts3d[0, :] * scale_x + sx 73 | pts3d[1, :] = pts3d[1, :] * scale_y + sy 74 | s = (scale_x + scale_y) / 2 75 | pts3d[2, :] *= s 76 | pts3d[2, :] -= np.min(pts3d[2, :]) 77 | return np.array(pts3d, dtype=np.float32) 78 | 79 | 80 | def _parse_param(param): 81 | """matrix pose form 82 | param: shape=(trans_dim+shape_dim+exp_dim,), i.e., 62 = 12 + 40 + 10 83 | """ 84 | 85 | # pre-defined templates for parameter 86 | n = param.shape[0] 87 | if n == 62: 88 | trans_dim, shape_dim, exp_dim = 12, 40, 10 89 | elif n == 72: 90 | trans_dim, shape_dim, exp_dim = 12, 40, 20 91 | elif n == 141: 92 | trans_dim, shape_dim, exp_dim = 12, 100, 29 93 | else: 94 | raise Exception(f'Undefined templated param parsing rule') 95 | 96 | R_ = param[:trans_dim].reshape(3, -1) 97 | R = R_[:, :3] 98 | offset = R_[:, -1].reshape(3, 1) 99 | alpha_shp = param[trans_dim:trans_dim + shape_dim].reshape(-1, 1) 100 | alpha_exp = param[trans_dim + shape_dim:].reshape(-1, 1) 101 | 102 | return R, offset, alpha_shp, alpha_exp 103 | -------------------------------------------------------------------------------- /utils/uv.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | 3 | __author__ = 'cleardusk' 4 | 5 | import sys 6 | 7 | sys.path.append('..') 8 | 9 | import cv2 10 | import numpy as np 11 | import os.path as osp 12 | import scipy.io as sio 13 | 14 | from Sim3DR import rasterize 15 | from utils.functions import plot_image 16 | from utils.io import _load 17 | from utils.tddfa_util import _to_ctype 18 | 19 | make_abs_path = lambda fn: osp.join(osp.dirname(osp.realpath(__file__)), fn) 20 | 21 | 22 | def load_uv_coords(fp): 23 | C = sio.loadmat(fp) 24 | uv_coords = C['UV'].copy(order='C').astype(np.float32) 25 | return uv_coords 26 | 27 | 28 | def process_uv(uv_coords, uv_h=256, uv_w=256): 29 | uv_coords[:, 0] = uv_coords[:, 0] * (uv_w - 1) 30 | uv_coords[:, 1] = uv_coords[:, 1] * (uv_h - 1) 31 | uv_coords[:, 1] = uv_h - uv_coords[:, 1] - 1 32 | uv_coords = np.hstack((uv_coords, np.zeros((uv_coords.shape[0], 1), dtype=np.float32))) # add z 33 | return uv_coords 34 | 35 | 36 | g_uv_coords = load_uv_coords(make_abs_path('../configs/BFM_UV.mat')) 37 | indices = _load(make_abs_path('../configs/indices.npy')) # todo: handle bfm_slim 38 | g_uv_coords = g_uv_coords[indices, :] 39 | 40 | 41 | def get_colors(img, ver): 42 | # nearest-neighbor sampling 43 | [h, w, _] = img.shape 44 | ver[0, :] = np.minimum(np.maximum(ver[0, :], 0), w - 1) # x 45 | ver[1, :] = np.minimum(np.maximum(ver[1, :], 0), h - 1) # y 46 | ind = np.round(ver).astype(np.int32) 47 | colors = img[ind[1, :], ind[0, :], :] # n x 3 48 | 49 | return colors 50 | 51 | 52 | def bilinear_interpolate(img, x, y): 53 | """ 54 | https://stackoverflow.com/questions/12729228/simple-efficient-bilinear-interpolation-of-images-in-numpy-and-python 55 | """ 56 | x0 = np.floor(x).astype(np.int32) 57 | x1 = x0 + 1 58 | y0 = np.floor(y).astype(np.int32) 59 | y1 = y0 + 1 60 | 61 | x0 = np.clip(x0, 0, img.shape[1] - 1) 62 | x1 = np.clip(x1, 0, img.shape[1] - 1) 63 | y0 = np.clip(y0, 0, img.shape[0] - 1) 64 | y1 = np.clip(y1, 0, img.shape[0] - 1) 65 | 66 | i_a = img[y0, x0] 67 | i_b = img[y1, x0] 68 | i_c = img[y0, x1] 69 | i_d = img[y1, x1] 70 | 71 | wa = (x1 - x) * (y1 - y) 72 | wb = (x1 - x) * (y - y0) 73 | wc = (x - x0) * (y1 - y) 74 | wd = (x - x0) * (y - y0) 75 | 76 | return wa[..., np.newaxis] * i_a + wb[..., np.newaxis] * i_b + wc[..., np.newaxis] * i_c + wd[..., np.newaxis] * i_d 77 | 78 | 79 | def uv_tex(img, ver_lst, tri, uv_h=256, uv_w=256, uv_c=3, show_flag=False, wfp=None): 80 | uv_coords = process_uv(g_uv_coords.copy(), uv_h=uv_h, uv_w=uv_w) 81 | 82 | res_lst = [] 83 | for ver_ in ver_lst: 84 | ver = _to_ctype(ver_.T) # transpose to m x 3 85 | colors = bilinear_interpolate(img, ver[:, 0], ver[:, 1]) / 255. 86 | # `rasterize` here serves as texture sampling, may need to optimization 87 | res = rasterize(uv_coords, tri, colors, height=uv_h, width=uv_w, channel=uv_c) 88 | res_lst.append(res) 89 | 90 | # concat if there more than one image 91 | res = np.concatenate(res_lst, axis=1) if len(res_lst) > 1 else res_lst[0] 92 | 93 | if wfp is not None: 94 | cv2.imwrite(wfp, res) 95 | print(f'Save visualization result to {wfp}') 96 | 97 | if show_flag: 98 | plot_image(res) 99 | 100 | return res 101 | -------------------------------------------------------------------------------- /weights/.gitignore: -------------------------------------------------------------------------------- 1 | checkpoints/ 2 | *.pth 3 | *.onnx -------------------------------------------------------------------------------- /weights/mb05_120x120.pth: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/weights/mb05_120x120.pth -------------------------------------------------------------------------------- /weights/mb1_120x120.pth: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cleardusk/3DDFA_V2/1b6c67601abffc1e9f248b291708aef0e43b55ae/weights/mb1_120x120.pth -------------------------------------------------------------------------------- /weights/readme.md: -------------------------------------------------------------------------------- 1 | ## Pre-converted onnx model 2 | 3 | | Model | Link | 4 | | :-: | :-: | 5 | | `mb1_120x120.onnx` | [Google Drive](https://drive.google.com/file/d/1YpO1KfXvJHRmCBkErNa62dHm-CUjsoIk/view?usp=sharing) or [Baidu Drive](https://pan.baidu.com/s/1qpQBd5KOS0-5lD6jZKXZ-Q) (Password: cqbx) | 6 | | `mb05_120x120.onnx` | [Google Drive](https://drive.google.com/file/d/1orJFiZPshmp7jmCx_D0tvIEtPYtnFvHS/view?usp=sharing) or [Baidu Drive](https://pan.baidu.com/s/1sRaBOA5wHu6PFS1Qd-TBFA) (Password: 8qst) | 7 | | `resnet22.onnx` | [Google Drive](https://drive.google.com/file/d/1rRyrd7Ar-QYTi1hRHOYHspT8PTyXQ5ds/view?usp=sharing) or [Baidu Drive](https://pan.baidu.com/s/1Nzkw7Ie_5trKvi1JYxymJA) (Password: 1op6) | 8 | | `resnet22.pth` | [Google Drive](https://drive.google.com/file/d/1dh7JZgkj1IaO4ZcSuBOBZl2suT9EPedV/view?usp=sharing) or [Baidu Drive](https://pan.baidu.com/s/1IS7ncVxhw0f955ySg67Y4A) (Password: lv1a) | --------------------------------------------------------------------------------