├── LICENSE ├── README.md ├── assets ├── J_regressor.pkl ├── basic_vtx_clr_symm_map.npy ├── bodyparts.pkl ├── face_regressor.pkl └── smpl_sampling.pkl ├── config.py ├── data ├── dance.png ├── hiphop.png ├── im1010.jpg ├── im1010_bbox.json ├── im1010_openpose.json └── parkour.png ├── demo.py ├── model_arch.py ├── model_composer.py ├── render ├── __init__.py ├── render_layer_ortho.py ├── render_ortho.py └── vertex_normal_expose.py ├── requirements.txt ├── smpl ├── __init__.py ├── batch_lbs.py ├── batch_smpl.py ├── bodyparts.py ├── joints.py ├── lib │ ├── __init__.py │ ├── geometry.py │ └── io.py └── smpl_layer.py ├── teaser_img.png └── utils.py /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | This code base itself is MIT, but for other components please follow the 4 | respective licenses of SMPL, DIRT and datasets (Human3.6M, 3DPW, LSP). 5 | 6 | Copyright (c) 2020 VAL-IISc 7 | 8 | Permission is hereby granted, free of charge, to any person obtaining a copy 9 | of this software and associated documentation files (the "Software"), to deal 10 | in the Software without restriction, including without limitation the rights 11 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | copies of the Software, and to permit persons to whom the Software is 13 | furnished to do so, subject to the following conditions: 14 | 15 | The above copyright notice and this permission notice shall be included in all 16 | copies or substantial portions of the Software. 17 | 18 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 19 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 20 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 21 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 22 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 23 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 24 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Appearance Consensus Driven Self-Supervised Human Mesh Recovery 3 | Code repository for the paper: 4 | **Appearance Consensus Driven Self-Supervised Human Mesh Recovery** 5 | [Jogendra N Kundu](https://sites.google.com/view/jogendra)\*, [Mugalodi Rakesh](https://www.linkedin.com/in/rakesh-mugalodi-179476191/?originalSubdomain=in)\*, [Varun Jampani](https://varunjampani.github.io/), [Rahul M V](https://www.linkedin.com/in/rahul-mysore-venkatesh-64b27380/), [R. Venkatesh Babu](http://cds.iisc.ac.in/faculty/venky/) 6 | 7 | ECCV 2020 8 | [[paper](https://arxiv.org/pdf/2008.01341.pdf)] [[project page](https://sites.google.com/view/ss-human-mesh)] 9 | 10 | ![teaser](teaser_img.png) 11 | 12 | ## Installation 13 | 14 | Clone the repo or download it as a zip from the GitHub GUI. 15 | 16 | ``` 17 | git clone https://github.com/val-iisc/ss_human_mesh.git 18 | ``` 19 | 20 | We have tested the full pipeline on linux with `python2`, hence we suggest you create a `python2` virtual environment and install the relevant pip packages as follows: 21 | 22 | #### Linux Setup: 23 | ``` 24 | mkdir ./ss_h_mesh_venv 25 | python -m virtualenv ./ss_h_mesh_venv 26 | source ./ss_h_mesh_venv/bin/activate 27 | pip install -U pip 28 | pip install -r requirements.txt 29 | ``` 30 | Following external packages are required to realize the full pipeline: 31 | 32 | 1) Install Dirt-renderer from the original repo [here](https://github.com/pmh47/dirt) or from our fork [here](https://github.com/rakeshramesha/dirt_renderer) 33 | Instructions to install Dirt-renderer can be found on the respective repo pages. 34 | 35 | 2) SMPL Model 36 | Download the neutral SMPL model from [here](http://smplify.is.tue.mpg.de/) and place it in the `assets` folder. 37 | ``` 38 | cp /code/models/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl assets/neutral_smpl.pkl 39 | ``` 40 | 41 | Download pre-trained model weights from [here](https://drive.google.com/drive/folders/1Wef_UA1XV5rgSDUIJn7nQT-6-lTvrk7V), extract and place them in the `weights` folder. Check if the weights path matches the path in `config.py`. 42 | 43 | ``` 44 | tar -xvf -C ./weights/ 45 | ``` 46 | 47 | ## Run demo code 48 | Images should be properly cropped, where the person bounding box is image-centered & scaled to get a bbox size of roughly 180px-200px (along the longer bbox dimension). Also single unoccluded person with full body visible (not truncated) yields best overlays and coloured mesh. 49 | 50 | There are 4 ways to run our inference code: 51 | 1. Bounding box as a json file along with image, Bbox would be used internally to obtain a proper crop of the image. 52 | ``` 53 | python demo.py --img_path --bbox 54 | ``` 55 | 2. OpenPose/CenterTrack detection json file along with image, J2D detections would be used to obtain a proper crop of image. 56 | ``` 57 | python demo.py --img_path --j2d_det 58 | ``` 59 | 3. Direct single image inference, Note: Proper crop (as mentioned above) is assumed. 60 | ``` 61 | python demo.py --img_path 62 | ``` 63 | 4. Direct webcam inference, person is assumed to be at the center of the feed. All renderings are performed in real-time, including colored mesh and mesh overlays. Note: Although we provide video inference code, we highly recommend use of a person detector in order to feed proper cropped images to the network. Also note that our model is not trained on video data, hence it might exhibit flicking artifacts. 64 | ``` 65 | python demo.py --webcam 66 | ``` 67 | 68 | ## Citing 69 | If you find our work helpful in your research, please cite the following paper: 70 | 71 | @Inproceedings{kundu_human_mesh, 72 | Title = {Appearance Consensus Driven Self-Supervised Human Mesh Recovery}, 73 | Author = {Kundu, Jogendra Nath and Rakesh, Mugalodi and Jampani, Varun and Venkatesh, Rahul M and Babu, R. Venkatesh}, 74 | Booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)}, 75 | Year = {2020} 76 | } 77 | -------------------------------------------------------------------------------- /assets/J_regressor.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/assets/J_regressor.pkl -------------------------------------------------------------------------------- /assets/basic_vtx_clr_symm_map.npy: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/assets/basic_vtx_clr_symm_map.npy -------------------------------------------------------------------------------- /assets/bodyparts.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/assets/bodyparts.pkl -------------------------------------------------------------------------------- /assets/face_regressor.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/assets/face_regressor.pkl -------------------------------------------------------------------------------- /config.py: -------------------------------------------------------------------------------- 1 | 2 | """ Project Configuration file. """ 3 | 4 | import os, sys, numpy as np, datetime, time 5 | 6 | def add_pypath(path): 7 | """ Insert path to system path. """ 8 | if path not in sys.path: 9 | sys.path.insert(0, path) 10 | 11 | def make_folder(path): 12 | """ Make folder. """ 13 | if not os.path.exists(path): 14 | os.makedirs(path) 15 | 16 | ROOT_DIR = './' 17 | if ROOT_DIR not in sys.path: 18 | sys.path.append(ROOT_DIR) 19 | 20 | os.environ['CUDA_VISIBLE_DEVICES'] = '0' 21 | EXP_NAME = 'release' 22 | 23 | ### INFERENCE 24 | overlay_clr = [0.5, 0.5, 1.0] ##Purple 25 | TF_version = 1.13 26 | restore_str = "./weights/weak_sup_train" 27 | 28 | ### DATALOADER 29 | TRANS_MAX_JITTER = 20 #Pix space 30 | SC_MAX_JITTER = 1.0 31 | SC_MIN_JITTER = 0.75 32 | 33 | ### ARCH 34 | JOINT_RES = 24 35 | IMG_W = 224 36 | IMG_H = 224 37 | IMG_SIZE = 224 38 | emb_size = 32 39 | n_preds = emb_size + 16 ##32+10+3+3 40 | num_stages = 3 41 | PRED_DYN_SCALE_AND_ALIGN = False ## Auto scale and align 42 | GT_DYN_SCALE_AND_ALIGN = True 43 | 44 | ### Organize and Save 45 | LOG_PATH = ROOT_DIR + 'logs/' 46 | 47 | tf_log_path = os.path.join(LOG_PATH, EXP_NAME, 'tf_logs/') 48 | 49 | current_time = datetime.datetime.now().strftime('M:%m_D:%d_T:%H_%M') 50 | 51 | train_log_dir = tf_log_path + current_time + '/train/' 52 | val_log_dir = tf_log_path + current_time + '/val/' 53 | test_log_dir = tf_log_path + current_time + '/test/' 54 | 55 | MODEL_WT_PATH = ROOT_DIR + 'weights/' 56 | model_save_path = os.path.join(MODEL_WT_PATH, EXP_NAME + '/') 57 | 58 | #make_folder(model_save_path) 59 | model_load_path = model_save_path 60 | 61 | SMPL_NEUTRAL = os.path.join(ROOT_DIR, 'assets', 'neutral_smpl.pkl') 62 | SMPL_MALE = os.path.join(ROOT_DIR, 'assets', 'basicmodel_m_lbs_10_207_0_v1.0.0.pkl') 63 | SMPL_FEMALE = os.path.join(ROOT_DIR, 'assets', 'basicModel_f_lbs_10_207_0_v1.0.0.pkl') 64 | 65 | #make_folder(os.path.join(LOG_PATH, EXP_NAME)) 66 | 67 | #os.system(('cp {} {}/').format(os.path.join(ROOT_DIR, '*.py'), os.path.join(LOG_PATH, EXP_NAME))) 68 | -------------------------------------------------------------------------------- /data/dance.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/data/dance.png -------------------------------------------------------------------------------- /data/hiphop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/data/hiphop.png -------------------------------------------------------------------------------- /data/im1010.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/data/im1010.jpg -------------------------------------------------------------------------------- /data/im1010_bbox.json: -------------------------------------------------------------------------------- 1 | { 2 | "bbox": [17, 37, 117, 142] 3 | } 4 | -------------------------------------------------------------------------------- /data/im1010_openpose.json: -------------------------------------------------------------------------------- 1 | {"version":1.2,"people":[{"pose_keypoints_2d":[66.4171,47.0342,0.614349,78.5691,57.8892,0.628322,74.1715,57.8883,0.568593,0,0,0,0,0,0,82.9632,58.1474,0.693141,87.8742,80.3805,0.703207,77.7917,97.9535,0.769904,77.7931,96.9218,0.588614,71.5884,96.9234,0.574803,53.2337,120.705,0.680349,35.916,151.465,0.627472,84.5121,96.9231,0.560366,98.2142,129.75,0.62086,118.893,155.601,0.699111,65.3889,44.6995,0.12635,67.4553,44.1846,0.616014,0,0,0,73.9174,42.636,0.769856,111.14,160.512,0.555395,114.242,160.771,0.531492,123.032,158.188,0.588194,25.3161,149.915,0.444102,26.6083,149.657,0.417128,34.8817,157.153,0.490182],"face_keypoints_2d":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"hand_left_keypoints_2d":[76.6547,100.644,0.0328468,73.4047,101.35,0.0266014,77.5732,96.7581,0.0534945,78.7743,95.7689,0.0531399,71.9917,109.052,0.0295517,71.4265,105.166,0.244298,71.8504,108.274,0.281847,71.9917,109.829,0.269919,72.2036,110.747,0.257855,73.1221,105.166,0.162519,73.1928,108.274,0.163806,73.6167,110.041,0.174101,73.1928,111.03,0.134029,75.3123,105.166,0.0856559,75.2417,107.709,0.11412,74.3939,110.041,0.130178,74.1819,111.242,0.22699,76.3721,105.378,0.0752514,76.2308,107.427,0.0975623,76.4428,109.617,0.0762423,76.0895,109.829,0.0484347],"hand_right_keypoints_2d":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"pose_keypoints_3d":[],"face_keypoints_3d":[],"hand_left_keypoints_3d":[],"hand_right_keypoints_3d":[]}]} -------------------------------------------------------------------------------- /data/parkour.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/data/parkour.png -------------------------------------------------------------------------------- /demo.py: -------------------------------------------------------------------------------- 1 | 2 | """ Demo/Inference file. """ 3 | 4 | import sys 5 | import os 6 | 7 | import numpy as np 8 | import scipy.io as scio 9 | import cv2 10 | import pickle as pkl 11 | import argparse 12 | 13 | import tensorflow as tf 14 | from tensorflow.contrib import slim 15 | 16 | from model_composer import ModelComposer 17 | import utils 18 | import config as cfg 19 | 20 | ### 21 | BATCH_SIZE = 1 22 | IMG_H = cfg.IMG_H 23 | IMG_W = cfg.IMG_W 24 | 25 | out_dir= "./data/" 26 | 27 | ### Argparse 28 | parser = argparse.ArgumentParser(description='Tester') 29 | parser.add_argument('--img_path', required=False, type=str, 30 | help='Path to image, performs a single image inference. Assumes proper cropping is done already.') 31 | parser.add_argument('--bbox', required=False, type=str, 32 | help='Bbox as a json file, assumes the format of [Topleft_x, Topleft_y, bbox_width, bbox_height].') 33 | parser.add_argument('--j2d_det', required=False, type=str, 34 | help='OpenPose/CenterTrack detections as a json file, assumes the format as dumped by OpenPose when run with "--write_json" option.') 35 | parser.add_argument('--webcam', required=False, type=int, 36 | help='Camera ID to read, performs webcam inference.') 37 | 38 | def preprocess_image(in_img, bbox=None, j2d_det=None, cvt_clr=False): 39 | """ Preprocess image. """ 40 | if(type(bbox) != type(None)): 41 | in_img = utils.bbox_crop(in_img, bbox) 42 | elif(type(j2d_det) != type(None)): 43 | in_img = utils.j2d_crop(in_img, j2d_det) 44 | else: 45 | in_img = cv2.resize(in_img, (IMG_W, IMG_H)) 46 | 47 | if(cvt_clr): 48 | in_img = cv2.cvtColor(in_img, cv2.COLOR_BGR2RGB) 49 | 50 | in_proc_img = in_img / 255.0 51 | # Normalize image to [-1, 1] 52 | in_proc_img = 2 * (in_proc_img - 0.5) 53 | 54 | return in_img, in_proc_img 55 | 56 | class Tester(object): 57 | 58 | def __init__(self): 59 | self.model_c = ModelComposer(BATCH_SIZE, is_training=False, is_fine_tune=False) 60 | 61 | def restore_from_path(self, session, load_dir): 62 | """ Restore Model from path. """ 63 | print ("\nRestoring checkpoint %s \n" %(load_dir)) 64 | if(self.model_c.restore_path_weights(session, load_dir)): 65 | sys.exit(0) 66 | 67 | def restore_prev_iter_model(self, session, load_dir): 68 | """ Restore Model from latest checkpoint. """ 69 | print ("\nRestoring latest checkpoint...\n") 70 | self.iteration = utils.get_latest_iter( wt_dir=load_dir, wt_type='iter') 71 | print ('LOADING iter weights from iter-%d' %(self.iteration)) 72 | if(self.model_c.restore_previter_weights(self.iteration, session, load_dir)): 73 | sys.exit(0) 74 | 75 | def restore_prev_best_model(self, session, load_dir): 76 | """ Restore Model from best validation checkpoint. """ 77 | print ("\nRestoring best checkpoint...\n") 78 | self.iteration = utils.get_latest_iter( wt_dir=load_dir, wt_type='best') 79 | print ('LOADING best weights which is from iter-%d' %(self.iteration)) 80 | if(self.model_c.restore_prevbest_weights(self.iteration, session, load_dir)): 81 | sys.exit(0) 82 | 83 | def test(self, session, cmd_args): 84 | """ Restore Model and run inference. """ 85 | self.model_c.compose_model(session) 86 | img_path = args.img_path 87 | webcam_id = args.webcam 88 | restore_str = cfg.restore_str 89 | 90 | init_op = tf.global_variables_initializer() 91 | session.run(init_op) 92 | 93 | if(restore_str == 'iter'): 94 | self.restore_prev_iter_model(session=session,load_dir=cfg.model_load_path) 95 | elif(restore_str == 'best'): 96 | self.restore_prev_best_model(session=session,load_dir=cfg.model_load_path) 97 | elif(type(restore_str) != type(None)): 98 | self.restore_from_path(session=session,load_dir=restore_str) 99 | else: 100 | print ("\nERROR: Unable to restore model based on given restore string...\n") 101 | exit(0) 102 | 103 | if(type(webcam_id) == type(None)): 104 | ## Single image read 105 | if(img_path == None): 106 | img_path = "./data/hiphop.png" 107 | in_img = cv2.imread(img_path) 108 | else: 109 | in_img = cv2.imread(img_path) 110 | 111 | in_img, in_proc_img = preprocess_image(in_img, bbox=args.bbox, j2d_det=args.j2d_det, cvt_clr=True) 112 | test_panel = self.run_test_iter(session, in_img, in_proc_img) 113 | 114 | #### Save panel image 115 | test_out = cv2.cvtColor(test_panel, cv2.COLOR_BGR2RGB).astype(np.uint8) 116 | cv2.imwrite(out_dir + "out_panel.png", test_out) 117 | cv2.imshow("Output Panel", test_out) 118 | cv2.waitKey(0) 119 | 120 | else: 121 | ## Webcam inference 122 | vidcap = cv2.VideoCapture(webcam_id) 123 | print('\nPress "Esc", "q" or "Q" to exit.\n') 124 | 125 | while (1): 126 | success, in_img = vidcap.read() 127 | if not success: 128 | print("Frame not read. ") 129 | break 130 | 131 | ## Center square crop 132 | o_h, o_w = in_img.shape[:2] 133 | c_y, c_x = o_h/2, o_w/2 134 | res = np.min([o_w, o_h]) 135 | 136 | in_img = in_img[c_y-(res/2):c_y+(res/2), c_x-(res/2):c_x+(res/2)] 137 | 138 | in_img, in_proc_img = preprocess_image(in_img, cvt_clr=True) 139 | test_panel = self.run_test_iter(session, in_img, in_proc_img) 140 | 141 | test_out = cv2.resize(cv2.cvtColor(test_panel, cv2.COLOR_BGR2RGB), (IMG_W*3, IMG_H*3)) 142 | 143 | cv2.imshow("Output panel", test_out) 144 | key = cv2.waitKey(1) 145 | if (key & 0xFF == ord('q') or key == 27): 146 | break 147 | 148 | def run_test_iter(self, session, in_image, proc_img): 149 | """ Perform inference and panel creation. """ 150 | model_c = self.model_c 151 | in_model = model_c.in_gt_nodes 152 | 153 | output_feed = [ model_c.ren_img, model_c.ren_side1, model_c.ren_olay, model_c.ren_olside1, model_c.ren_img_raw ] 154 | 155 | proc_img = np.expand_dims(proc_img, axis=0) 156 | 157 | input_feed = { in_model['in_img']: proc_img } 158 | 159 | ## Get predictions 160 | [ ren_fr, ren_s1, ren_olay, ren_ols1, ren_fr_dir ] = session.run(output_feed, input_feed) 161 | 162 | ## Collage rendered images 163 | img_id = -1 164 | ren_fr = (ren_fr[img_id]).astype(np.uint8) 165 | ren_s1 = (ren_s1[img_id]).astype(np.uint8) 166 | 167 | ren_ol = (ren_olay[img_id]).astype(np.uint8) 168 | ren_ols1 = (ren_ols1[img_id]).astype(np.uint8) 169 | 170 | ren_fr_dir = (ren_fr_dir[img_id]).astype(np.uint8) 171 | 172 | row1 = utils.create_collage([in_image, ren_ol, ren_ols1]) 173 | row2 = utils.create_collage([ren_fr_dir, ren_fr, ren_s1]) 174 | test_panel = utils.create_collage([row1, row2],axis=0) 175 | 176 | return test_panel 177 | 178 | if __name__== "__main__": 179 | args = parser.parse_args() 180 | 181 | ### Setup TF session 182 | gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.8) 183 | config = tf.ConfigProto(gpu_options=gpu_options) 184 | config.gpu_options.allow_growth = True 185 | session = tf.InteractiveSession(config=config) 186 | 187 | ### Run inference 188 | tester = Tester() 189 | tester.test(session=session, cmd_args=args) 190 | 191 | print ("\n........................TESTING ENDED.............................\n") 192 | -------------------------------------------------------------------------------- /model_arch.py: -------------------------------------------------------------------------------- 1 | 2 | """ Network Architecture file. """ 3 | 4 | import sys 5 | import os 6 | import numpy as np 7 | import time as time 8 | import pickle as pkl 9 | 10 | import tensorflow as tf 11 | from tensorflow.contrib import slim 12 | from tensorflow.contrib.layers.python.layers.initializers import variance_scaling_initializer 13 | from tensorflow.contrib.slim.python.slim.nets import resnet_v2 14 | 15 | from smpl.smpl_layer import SmplTPoseLayer 16 | from smpl.batch_lbs import batch_rodrigues 17 | 18 | from render.render_layer_ortho import RenderLayer 19 | import render.vertex_normal_expose as dirt_expose 20 | 21 | import utils 22 | import config as cfg 23 | 24 | ### Arch specific flags 25 | JOINT_RES = cfg.JOINT_RES 26 | IMG_W = cfg.IMG_W 27 | IMG_H = cfg.IMG_H 28 | 29 | emb_size = cfg.emb_size 30 | n_preds = cfg.n_preds 31 | 32 | PI = np.pi 33 | 34 | def Res50_backbone_setup(img_in, is_training=True, weight_decay=0.001, reuse=False): 35 | """ Resnet v2-50, CNN backbone to process image. """ 36 | with slim.arg_scope(resnet_v2.resnet_arg_scope(weight_decay=weight_decay)): 37 | net, end_points = resnet_v2.resnet_v2_50(img_in, num_classes=None, is_training=is_training, reuse=reuse, scope='resnet_v2_50') 38 | net = tf.squeeze(net, axis=[1, 2]) 39 | 40 | return net 41 | 42 | def FC_Encoder( cnn_ft, l_neurons=1024, num_preds=n_preds, is_training=True, reuse=False, name="FC_Encoder"): 43 | """ FC layer to process image features. """ 44 | with tf.variable_scope(name, reuse=reuse) as scope: 45 | net = slim.fully_connected(cnn_ft, l_neurons, scope='fc1') 46 | net = slim.dropout(net, 0.5, is_training=is_training, scope='dropout1') 47 | net = slim.fully_connected(net, l_neurons, scope='fc2') 48 | net = slim.dropout(net, 0.5, is_training=is_training, scope='dropout2') 49 | small_xavier = variance_scaling_initializer(factor=.01, mode='FAN_AVG', uniform=True) 50 | net_out = slim.fully_connected( net, num_preds, activation_fn=None, weights_initializer=small_xavier, scope='fc3') 51 | 52 | return net_out 53 | 54 | class NetModel(object): 55 | 56 | def __init__(self, scope_name, batch_size, is_training=True, is_fine_tune=False): 57 | """ Init for Model-arch """ 58 | self.scope = scope_name 59 | self.batch_size = batch_size 60 | self.is_training = is_training 61 | self.is_fine_tune = is_fine_tune 62 | self.num_stages = cfg.num_stages 63 | 64 | with tf.variable_scope(self.scope): 65 | self.add_placeholders() 66 | 67 | self.build_network() 68 | 69 | def add_placeholders(self): 70 | """ Input Placeholders """ 71 | self.in_img = tf.placeholder(tf.float32, shape=[self.batch_size, IMG_H, IMG_W, 3]) 72 | 73 | def setup_cnn_layers(self): 74 | """ CNN + FC regressor, Img -> [Pose_emb, Beta, Cam, Scale, Trans] """ 75 | ### CNN backbone 76 | res_img_ft = Res50_backbone_setup(self.in_img, is_training = self.is_training and not self.is_fine_tune, reuse=False) 77 | 78 | with tf.variable_scope(self.scope) as scope: 79 | FC_layer = FC_Encoder 80 | 81 | ### Base init params 82 | base_pose_emb = np.zeros([1, cfg.emb_size], dtype=np.float32) 83 | base_betas = np.array([[0.20560974, 0.33556297, -0.35068282, 0.35612896, 0.41754073, 0.03088791, 0.30475676, 0.23613405, 0.20912662, 0.31212646]], dtype=np.float32) 84 | base_cam = np.array([[ 1.0, 0, 0]], dtype=np.float32) 85 | base_sc_trans = np.array([[ 0, 0, 0]], dtype=np.float32) 86 | base_params = np.concatenate([base_pose_emb, base_cam, base_betas, base_sc_trans], axis=1) 87 | 88 | base_params = tf.Variable(base_params, name="base_params", dtype=tf.float32) 89 | 90 | prev_pred = tf.tile(base_params, [self.batch_size, 1]) 91 | 92 | ### FC layers 93 | for i in np.arange(self.num_stages): 94 | curr_state = tf.concat([res_img_ft, prev_pred], 1) 95 | 96 | if i == 0: 97 | delta_pred = FC_layer(curr_state, l_neurons=2048, is_training=self.is_training, reuse=False) 98 | else: 99 | delta_pred = FC_layer(curr_state, l_neurons=2048, is_training=self.is_training, reuse=True) 100 | 101 | # Curr pred 102 | curr_pred = prev_pred + delta_pred 103 | prev_pred = curr_pred 104 | 105 | ### Final output 106 | net_pred = curr_pred 107 | 108 | ### Theta/Pose 109 | self.pred_pose_emb = tf.math.tanh( net_pred[:, :emb_size], name='tanh_emb') ### Nxemb_size 110 | 111 | ### Betas/Shape 112 | self.pred_betas = net_pred[:,emb_size+3:emb_size+13] ### Nx10 113 | 114 | ### Global Camera/person orientation 115 | self.pred_cam_axan = tf.math.tanh(net_pred[:, emb_size:emb_size+3], name='tanh_cam') * PI ### Nx3 116 | 117 | ### Scale and Trans 118 | mean_scale = np.full((self.batch_size, 1), 0.8*cfg.IMG_H, dtype=np.float32) 119 | var_scale = mean_scale/3 120 | self.pred_scale = mean_scale + ( tf.math.tanh(net_pred[:, emb_size+13:emb_size+14], name='tanh_scale') * var_scale ) ### Nx1 121 | self.pred_trans = tf.math.tanh(net_pred[:, emb_size+14:], name='tanh_trans') * (cfg.IMG_W/4) ### Nx2 122 | 123 | self.pred_sc_trans = tf.concat([self.pred_scale, self.pred_trans],axis=1) ### Nx3 124 | 125 | def setup_pose_emb_layer(self): 126 | """ Pose-prior module, [Pose_emb,Cam] -> Pose. """ 127 | with tf.variable_scope('AAE_Decoder', reuse=tf.AUTO_REUSE): 128 | 129 | fc_out = tf.layers.dense(self.pred_pose_emb, 512, activation=tf.nn.relu) 130 | fc_out = tf.layers.dense(fc_out, 1024, activation=tf.nn.relu) 131 | fc_out = tf.layers.dense(fc_out, 1024, activation=tf.nn.relu) 132 | 133 | theta_dec = tf.layers.dense(fc_out, 23*3) 134 | 135 | self.repose = theta_dec ### Nx23x3 136 | self.pred_pose = tf.concat( (self.pred_cam_axan, self.repose), axis = 1 ) ### Nx24x3 137 | 138 | def setup_smpl_layers(self): 139 | """ SMPL + Projection module, [Theta, Beta, Cam]-> [Mesh, J3D, J2D]. """ 140 | smpl = SmplTPoseLayer(theta_in_rodrigues=False, theta_is_perfect_rotmtx=True) 141 | 142 | offsets = tf.zeros(None, 1) ### Nx1 143 | zero_trans = tf.zeros_like(self.pred_cam_axan) ### Nx3 144 | 145 | ### Pred SMPL out 146 | pred_pose_reshape = tf.reshape(batch_rodrigues(tf.reshape(self.pred_pose, [-1, 3])), [-1, 24, 3, 3]) ### Nx24x3x3 147 | self.cam_smpl_out = smpl([pred_pose_reshape, self.pred_betas, zero_trans, offsets]) 148 | 149 | ## Raw Vertices [in world space] 150 | self.pred_verts = self.cam_smpl_out[0] # Nx6890x3 151 | ## Raw Joints 3d [in world space] 152 | self.pred_j3d = self.cam_smpl_out[1] # Nx24x3 153 | 154 | ### Scale and Trans 155 | if(cfg.PRED_DYN_SCALE_AND_ALIGN): 156 | ## For known cropping [200/224] 157 | self.scaled_pred_verts, self.scaled_pred_j3d, self.app_scale_pred, self.app_trans_pred = utils.tf_dyn_scale_and_align(vertices=self.pred_verts, joints_3d=self.pred_j3d, scale=200, add_trans=0) 158 | self.app_sc_trans_pred = tf.concat([self.app_scale_pred, self.app_trans_pred], axis=1) 159 | 160 | else: 161 | ## Apply predicted scale and trans 162 | self.scaled_pred_verts, self.scaled_pred_j3d = utils.for_tpix_tf_do_scale_and_align(vertices=self.pred_verts, joints_3d=self.pred_j3d, scale=self.pred_scale, trans=self.pred_trans) 163 | 164 | ### Project onto 2D for Joints2D 165 | self.pred_j2d = utils.tf_orthographic_project(self.scaled_pred_j3d) # Nx24x2 166 | #self.pred_j2d = utils.tf_align_with_image_j2d(self.pred_j2d, self.in_img.shape[1], self.in_img.shape[2]) 167 | 168 | def setup_cam_mesh_relation_module(self): 169 | """ Mesh to Image relation + Reflectional Symmetry module. [Vtx, Img] -> [Vtx_clr, Vtx_clr_symm] """ 170 | ### Unprocess image 171 | self.denorm_image = utils.denormalize_image(self.in_img) 172 | 173 | ### Occlusion-aware weights 174 | pred_camfront_occ_resolved = utils.get_occ_aware_cam_facing_mask(self.scaled_pred_verts, self.batch_size) 175 | 176 | pred_img_clr_picked = utils.colour_pick_img(self.denorm_image, self.scaled_pred_verts, self.batch_size) 177 | pred_img_clr_picked_resolved = tf.multiply(pred_img_clr_picked, pred_camfront_occ_resolved) 178 | 179 | ### Apply Reflectional Symmetry 180 | self.pred_vclr_cm = pred_img_clr_picked 181 | self.pred_vclr_cm_symm = utils.apply_ref_symmetry(pred_img_clr_picked_resolved, pred_camfront_occ_resolved, self.batch_size) 182 | 183 | def setup_renderer_layer(self): 184 | """ Rendering Module, Init for differentiable-renderers. """ 185 | MESH_PROP_FACES_FL = './assets/smpl_sampling.pkl' 186 | 187 | with open(os.path.join(os.path.dirname(__file__), MESH_PROP_FACES_FL), 'rb') as f: 188 | sampling = pkl.load(f) 189 | 190 | M = sampling['meshes'] 191 | 192 | self.faces = M[0]['f'].astype(np.int32) 193 | self.faces = tf.convert_to_tensor(self.faces,dtype=tf.int32) 194 | 195 | bgcolor = tf.zeros(3) ## Black bg 196 | fixed_t = [0.0, 0.0, 0.0] 197 | 198 | ### View 1, front view 199 | fixed_rt = np.array([1.0, 0.0, 0.0]) * PI 200 | self.renderer = RenderLayer(IMG_W, IMG_H, 3, bgcolor=bgcolor, f=self.faces, camera_f=[IMG_W, IMG_H], camera_c=[IMG_W/2.0, IMG_H/2.0], camera_rt=fixed_rt, camera_t=fixed_t) 201 | 202 | ### Overlay Renderer 203 | bg_overlay = self.denorm_image 204 | self.renderer_olay = RenderLayer(IMG_W, IMG_H, 3, bgcolor=bg_overlay, f=self.faces, camera_f=[IMG_W, IMG_H], camera_c=[IMG_W/2.0, IMG_H/2.0], camera_rt=fixed_rt, camera_t=fixed_t) 205 | 206 | ### View 2, -60 deg side view 207 | fixed_rt = np.array([2.72, 0.0, -1.57]) 208 | self.renderer2 = RenderLayer(IMG_W, IMG_H, 3, bgcolor=bgcolor, f=self.faces, camera_f=[IMG_W, IMG_H], camera_c=[IMG_W/2.0, IMG_H/2.0], camera_rt=fixed_rt, camera_t=fixed_t) 209 | 210 | ''' 211 | ### View 3, +60 deg side view 212 | #fixed_rt = np.array([ 2.72, 0.0, 1.57]) 213 | self.renderer3 = RenderLayer(IMG_W, IMG_H, 3, bgcolor=bgcolor, f=self.faces, camera_f=[IMG_W, IMG_H], camera_c=[IMG_W/2.0, IMG_H/2.0], camera_rt=fixed_rt, camera_t=fixed_t) 214 | ''' 215 | ######## 216 | def call_main_render_layer(self, verts, vclr): 217 | """ Render front view, [Vtx, Vtx_clr] -> [Ren_img] """ 218 | return self.renderer.call(v=verts, vc=vclr) 219 | 220 | def call_overlay_render_layer(self, verts): 221 | """ Render Mesh Overlays, [Vtx, Img] -> [Overlay_img] """ 222 | fixed_clr_2 = np.array(cfg.overlay_clr).astype(np.float32) #### clr of overlay mesh 223 | 224 | vert_norms = dirt_expose.get_vertex_normals(verts, self.faces) 225 | 226 | s_norm = tf.reduce_mean(vert_norms, axis=2, keepdims=True) 227 | s_norm = utils.tf_norm(s_norm, axis=1) 228 | overlay_vclr = tf.image.adjust_gamma( tf.tile(s_norm, [1,1,3]) , 0.35) * fixed_clr_2 229 | 230 | return [self.renderer_olay.call(v=verts, vc=overlay_vclr, is_img_bg=True), self.renderer2.call(v=verts, vc=overlay_vclr)] 231 | 232 | def call_vis_render_layer(self, verts, vclr): 233 | """ Render multiple views, [Vtx, Vtx_clr] -> [ren_V1_img, ....] """ 234 | return [self.renderer.call(v=verts, vc=vclr), self.renderer2.call(v=verts, vc=vclr)]#, self.renderer3.call(v=verts, vc=vclr)] 235 | 236 | def build_network(self): 237 | """ Setup Arch and initialize sub modules. """ 238 | self.setup_cnn_layers() 239 | self.setup_pose_emb_layer() 240 | self.setup_smpl_layers() 241 | self.setup_cam_mesh_relation_module() 242 | self.setup_renderer_layer() 243 | 244 | def get_network_nodes(self): 245 | """ Important Nodes to tap into. """ 246 | inputs = { "in_img": self.in_img 247 | } 248 | 249 | cnn_outs = { "pred_pose": self.pred_pose, "pred_betas": self.pred_betas, 250 | "pred_cam_axan": self.pred_cam_axan , "pred_scale": self.pred_scale, 251 | "pred_trans": self.pred_trans, "pred_sc_trans": self.pred_sc_trans, 252 | } 253 | 254 | smpl_outs = { "pred_verts": self.pred_verts, "pred_j3d": self.pred_j3d, "pred_j2d": self.pred_j2d, 255 | "scaled_pred_verts": self.scaled_pred_verts, "scaled_pred_j3d": self.scaled_pred_j3d 256 | } 257 | 258 | render_outs = { "renderer": self.renderer 259 | } 260 | 261 | cam_mesh_outs = { "pred_vclr_cm": self.pred_vclr_cm, "pred_vclr_cm_symm": self.pred_vclr_cm_symm 262 | } 263 | 264 | return { "inputs_and_gt": inputs, "cnn_layer": cnn_outs, "smpl_layer": smpl_outs, "renderer_layer": render_outs, "cam_mesh_module": cam_mesh_outs } 265 | 266 | ''' 267 | def print_DEBUG_STR(self): 268 | print ("\n\n:::::::: NetModel DEBUG_STR ::::::::\n") 269 | ### Can add debug string options here 270 | print ("\n\n") 271 | ''' -------------------------------------------------------------------------------- /model_composer.py: -------------------------------------------------------------------------------- 1 | 2 | """ Pipeline Setup file. """ 3 | 4 | import sys 5 | import os 6 | import numpy as np 7 | import time as time 8 | 9 | import tensorflow as tf 10 | 11 | from model_arch import NetModel 12 | import utils 13 | import config as cfg 14 | 15 | class ModelComposer(object): 16 | 17 | def __init__(self, batch_size, is_training=True, is_fine_tune=False): 18 | """ Init for Pipeline """ 19 | self.batch_size = batch_size 20 | self.part_batch = (self.batch_size / 2) 21 | self.is_training = is_training 22 | 23 | ### Create a new network model 24 | self.net_scope = "netmodel_prime" 25 | self.net_model = NetModel(self.net_scope, batch_size=self.batch_size, is_training=is_training, is_fine_tune=is_fine_tune) 26 | 27 | ### Get suitable params 28 | self.net_params = utils.get_network_params(self.net_scope) 29 | self.resnet_params = utils.get_net_train_params("resnet_v2_50") 30 | 31 | ### Define all trainable params 32 | self.trainable_params = self.net_params + self.resnet_params 33 | self.var_list = tf.global_variables() 34 | 35 | ### Weight save & restore 36 | self.bestwt_saver = tf.train.Saver(self.var_list, max_to_keep=5) 37 | self.iterwt_saver = tf.train.Saver(self.var_list, max_to_keep=5) 38 | 39 | ### Scope/Name 40 | self.scope = "composer" 41 | self.name = "model_composer" 42 | 43 | ### Get critical network nodes 44 | self.nodes_net = self.net_model.get_network_nodes() 45 | 46 | ## Nodes 47 | self.in_gt_nodes = self.nodes_net['inputs_and_gt'] 48 | self.cnn_nodes = self.nodes_net['cnn_layer'] 49 | self.smpl_nodes = self.nodes_net['smpl_layer'] 50 | self.cam_mesh_nodes = self.nodes_net['cam_mesh_module'] 51 | self.ren_nodes = self.nodes_net['renderer_layer'] 52 | 53 | def gen_mask_render(self): 54 | """ Render FG-BG binary mask. """ 55 | vclr_white = tf.ones_like(self.smpl_nodes['scaled_pred_verts']) 56 | self.ren_mask = self.net_model.call_main_render_layer(verts=self.smpl_nodes['scaled_pred_verts'], vclr=vclr_white)*255 57 | 58 | def gen_non_symm_render(self): 59 | """ Render direct picked vtx_clr image. """ 60 | vclr_cp = self.cam_mesh_nodes['pred_vclr_cm'] 61 | self.ren_img_raw = self.net_model.call_main_render_layer(verts=self.smpl_nodes['scaled_pred_verts'], vclr=vclr_cp)*255 62 | 63 | def gen_overlay_render(self): 64 | """ Render mesh overlay image. """ 65 | listy_ol = self.net_model.call_overlay_render_layer(verts=self.smpl_nodes['scaled_pred_verts']) 66 | self.ren_olay = listy_ol[0]*255 67 | self.ren_olside1 = listy_ol[1]*255 68 | 69 | def vis_render_img(self): 70 | """ Render occlusion-aware vtx_clr image. """ 71 | pred_vclrs = self.cam_mesh_nodes['pred_vclr_cm_symm'] 72 | listy = self.net_model.call_vis_render_layer(verts=self.smpl_nodes['scaled_pred_verts'], vclr=pred_vclrs) 73 | 74 | self.ren_img = listy[0]*255 75 | self.ren_side1 = listy[1]*255 76 | #self.ren_side2 = listy[2]*255 77 | 78 | def compose_model(self, session, en_render=True): 79 | """ Setup processing pipeline. """ 80 | if(en_render): 81 | self.vis_render_img() 82 | self.gen_overlay_render() 83 | self.gen_non_symm_render() 84 | 85 | ## Raw outs 86 | self.pred_pose = self.cnn_nodes['pred_pose'] 87 | self.pred_betas = self.cnn_nodes['pred_betas'] 88 | self.pred_proj_cam = self.cnn_nodes['pred_sc_trans'] 89 | 90 | ## SMPL outs 91 | self.pred_verts = self.smpl_nodes['pred_verts'] 92 | self.pred_j3d = self.smpl_nodes['pred_j3d'] 93 | 94 | ## SMPL->cam outs 95 | self.scaled_pred_verts = self.smpl_nodes['scaled_pred_verts'] 96 | self.scaled_pred_j3d = self.smpl_nodes['scaled_pred_j3d'] 97 | self.pred_j2d = self.smpl_nodes['pred_j2d'] 98 | 99 | ## Mesh vtx_clrs 100 | self.pred_vclr_pick = self.cam_mesh_nodes['pred_vclr_cm'] 101 | self.pred_vclr_pick_symm = self.cam_mesh_nodes['pred_vclr_cm_symm'] 102 | 103 | def restore_path_weights(self, session, load_dir): 104 | """ Restore network from a path. """ 105 | try: 106 | print ('Trying to load path weights...') 107 | self.iterwt_saver.restore(session, save_path = '%s' %(load_dir)) 108 | print ("LOADED path weights successfully.") 109 | return False 110 | except Exception as ex: 111 | print('Could not load weights in path: ', load_dir) 112 | return True 113 | 114 | def restore_previter_weights(self, iter_no, session, load_dir): 115 | """ Restore network from recent/latest iteration. """ 116 | try: 117 | print ('Trying to load iter weights...') 118 | self.iterwt_saver.restore(session, save_path = '%siter-%d' % (load_dir, iter_no)) 119 | print ("LOADED iter weights successfully.") 120 | return False 121 | except Exception as ex: 122 | print('Could not load iter weights') 123 | return True 124 | 125 | def restore_prevbest_weights(self, iter_no, session, load_dir): 126 | """ Restore network from best val iteration. """ 127 | try: 128 | print ('Trying to load best weights...') 129 | self.bestwt_saver.restore(session, save_path = '%sbest-%d' % (load_dir, iter_no)) 130 | print ("LOADED best weights successfully.") 131 | return False 132 | except Exception as ex: 133 | print('Could not load best weights') 134 | return True 135 | -------------------------------------------------------------------------------- /render/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/render/__init__.py -------------------------------------------------------------------------------- /render/render_layer_ortho.py: -------------------------------------------------------------------------------- 1 | 2 | """ Renderering layer wrapper file. """ 3 | 4 | import numpy as np 5 | from keras.engine.topology import Layer 6 | from render_ortho import render_colored_batch, render_overlay_colored_batch 7 | 8 | class RenderLayer(Layer): 9 | 10 | def __init__(self, width, height, num_channels, bgcolor, f, camera_f, camera_c, 11 | camera_t=np.zeros(3), camera_rt=np.zeros(3), **kwargs): 12 | """ Init wrapper of renderer """ 13 | self.width = width 14 | self.height = height 15 | self.num_channels = num_channels 16 | 17 | self.f = f 18 | self.bgcolor = bgcolor 19 | 20 | self.camera_f = np.array(camera_f).astype(np.float32) 21 | self.camera_c = np.array(camera_c).astype(np.float32) 22 | self.camera_t = np.array(camera_t).astype(np.float32) 23 | self.camera_rt = np.array(camera_rt).astype(np.float32) 24 | 25 | super(RenderLayer, self).__init__(**kwargs) 26 | 27 | def call(self, v, vc ,cam_pred=None, is_img_bg=False): 28 | """ Render mesh and mesh overlays """ 29 | if(not is_img_bg): 30 | assert(self.num_channels == vc.shape[-1] == self.bgcolor.shape[0]) 31 | 32 | return render_colored_batch(m_v=v, m_f=self.f, m_vc=vc, width=self.width, height=self.height, 33 | camera_f=self.camera_f, camera_c=self.camera_c, num_channels=self.num_channels, 34 | camera_t=self.camera_t, camera_rt=self.camera_rt, bgcolor=self.bgcolor,cam_pred=cam_pred) 35 | 36 | else: 37 | return render_overlay_colored_batch(m_v=v, m_f=self.f, m_vc=vc, width=self.width, height=self.height, 38 | camera_f=self.camera_f, camera_c=self.camera_c, num_channels=self.num_channels, 39 | camera_t=self.camera_t, camera_rt=self.camera_rt, bgcolor=self.bgcolor,cam_pred=cam_pred) 40 | 41 | def compute_output_shape(self, input_shape): 42 | """ Final output shape """ 43 | return input_shape[0], self.height, self.width, self.num_channels 44 | 45 | 46 | -------------------------------------------------------------------------------- /render/render_ortho.py: -------------------------------------------------------------------------------- 1 | 2 | """ Batched Render file. """ 3 | 4 | import dirt 5 | import numpy as np 6 | import tensorflow as tf 7 | 8 | from dirt import matrices 9 | import dirt.lighting as lighting 10 | from tensorflow.python.framework import ops 11 | 12 | def orthgraphic_projection(w, h, near=0.1, far=10., name=None): 13 | """Constructs a orthographic projection matrix. 14 | This function returns a orthographic projection matrix, using the OpenGL convention that the camera 15 | looks along the negative-z axis in view/camera space, and the positive-z axis in clip space. 16 | Multiplying view-space homogeneous coordinates by this matrix maps them into clip space. 17 | """ 18 | with ops.name_scope(name, 'OrthographicProjection', [w, h, near, far]) as scope: 19 | ### symmetric view case 20 | right = w / 2 21 | left = -right 22 | top = h / 2 23 | bottom = -top 24 | 25 | elements = [ 26 | [2. / (right-left), 0., 0, -(right+left)/(right-left) ], 27 | [0., 2. / (top-bottom), 0, -(top+bottom)/(top-bottom) ], 28 | [0., 0., -2. / (far - near), -(far+near)/(far-near) ], 29 | [0.,0.,0.,1.] 30 | ] 31 | 32 | return tf.transpose(tf.convert_to_tensor(elements, dtype=tf.float32)) 33 | 34 | def perspective_projection(f, c, w, h, near=0.1, far=10., name=None): 35 | """Constructs a perspective projection matrix. 36 | This function returns a perspective projection matrix, using the OpenGL convention that the camera 37 | looks along the negative-z axis in view/camera space, and the positive-z axis in clip space. 38 | Multiplying view-space homogeneous coordinates by this matrix maps them into clip space. 39 | """ 40 | with ops.name_scope(name, 'PerspectiveProjection', [f, c, w, h, near, far]) as scope: 41 | f = 0.5 * (f[0] + f[1]) 42 | pixel_center_offset = 0.5 43 | right = (w - (c[0] + pixel_center_offset)) * (near / f) 44 | left = -(c[0] + pixel_center_offset) * (near / f) 45 | top = (c[1] + pixel_center_offset) * (near / f) 46 | bottom = -(h - c[1] + pixel_center_offset) * (near / f) 47 | 48 | elements = [ 49 | [2. * near / (right - left), 0., (right + left) / (right - left), 0.], 50 | [0., 2. * near / (top - bottom), (top + bottom) / (top - bottom), 0.], 51 | [0., 0., -(far + near) / (far - near), -2. * far * near / (far - near)], 52 | [0., 0., -1., 0.] 53 | ] 54 | 55 | return tf.transpose(tf.convert_to_tensor(elements, dtype=tf.float32)) 56 | 57 | def render_colored_batch(m_v, m_f, m_vc, width, height, camera_f, camera_c, bgcolor=np.zeros(3, dtype=np.float32), 58 | num_channels=3, camera_t=np.zeros(3, dtype=np.float32), 59 | camera_rt=np.zeros(3, dtype=np.float32), name=None,batch_size=None,cam_pred=None): 60 | """ Render a batch of meshes with fixed BG. Supported projection types 1) Perspective, 2) Orthographic. """ 61 | 62 | with ops.name_scope(name, "render_batch", [m_v]) as name: 63 | assert (num_channels == m_vc.shape[-1] == bgcolor.shape[0]) 64 | 65 | #projection_matrix = perspective_projection(camera_f, camera_c, width, height, .1, 10) 66 | projection_matrix = orthgraphic_projection(width, height, -(width/2), (width/2)) ### im_w x im_h x im_w cube 67 | 68 | ## Camera Extrinsics, rotate & trans 69 | view_matrix = matrices.compose( matrices.rodrigues(camera_rt.astype(np.float32)), 70 | matrices.translation(camera_t.astype(np.float32)), 71 | ) 72 | ## Fixed clr BG 73 | bg = tf.tile(bgcolor[tf.newaxis,tf.newaxis,tf.newaxis,...],[tf.shape(m_v)[0],width,height,1]) 74 | 75 | m_v = tf.cast(m_v, tf.float32) 76 | m_v = tf.concat([m_v, tf.ones_like(m_v[:, :, -1:])], axis=2) 77 | 78 | ## Extrinsic multiplication 79 | m_v = tf.matmul(m_v, tf.tile(view_matrix[np.newaxis, ...], (tf.shape(m_v)[0], 1, 1))) 80 | 81 | ## Intrinsic Camera projection 82 | m_v = tf.matmul(m_v, tf.tile(projection_matrix[np.newaxis, ...], (tf.shape(m_v)[0], 1, 1))) 83 | 84 | m_f = tf.tile(tf.cast(m_f, tf.int32)[tf.newaxis, ...], (tf.shape(m_v)[0], 1, 1)) 85 | 86 | ## Rasterize 87 | return dirt.rasterise_batch(bg, m_v, m_vc, m_f, name=name) 88 | 89 | def render_overlay_colored_batch(m_v, m_f, m_vc, width, height, camera_f, camera_c, bgcolor=np.zeros(3, dtype=np.float32), 90 | num_channels=3, camera_t=np.zeros(3, dtype=np.float32), 91 | camera_rt=np.zeros(3, dtype=np.float32), name=None,batch_size=None,cam_pred=None): 92 | """ Render a batch of meshes with corresponding BG images. Supported projection types 1) Perspective, 2) Orthographic. """ 93 | 94 | with ops.name_scope(name, "render_batch", [m_v]) as name: 95 | 96 | #projection_matrix = perspective_projection(camera_f, camera_c, width, height, .1, 10) 97 | projection_matrix = orthgraphic_projection(width, height, -(width/2), (width/2)) ### im_w x im_h x im_w cube 98 | 99 | ## Camera Extrinsics, rotate & trans 100 | view_matrix = matrices.compose( matrices.rodrigues(camera_rt.astype(np.float32)), 101 | matrices.translation(camera_t.astype(np.float32)), 102 | ) 103 | ## Image BG 104 | bg = bgcolor 105 | 106 | m_v = tf.cast(m_v, tf.float32) 107 | m_v = tf.concat([m_v, tf.ones_like(m_v[:, :, -1:])], axis=2) 108 | 109 | ## Extrinsic multiplication 110 | m_v = tf.matmul(m_v, tf.tile(view_matrix[np.newaxis, ...], (tf.shape(m_v)[0], 1, 1))) 111 | 112 | ## Intrinsic Camera projection 113 | m_v = tf.matmul(m_v, tf.tile(projection_matrix[np.newaxis, ...], (tf.shape(m_v)[0], 1, 1))) 114 | 115 | m_f = tf.tile(tf.cast(m_f, tf.int32)[tf.newaxis, ...], (tf.shape(m_v)[0], 1, 1)) 116 | 117 | ## Rasterize 118 | return dirt.rasterise_batch(bg, m_v, m_vc, m_f, name=name) 119 | -------------------------------------------------------------------------------- /render/vertex_normal_expose.py: -------------------------------------------------------------------------------- 1 | 2 | """ Expose Normals and other rasterizer properties. """ 3 | 4 | import sys 5 | import numpy as np 6 | import tensorflow as tf 7 | from tensorflow.python.framework import ops 8 | 9 | import dirt 10 | from dirt import matrices 11 | import dirt.lighting as lighting 12 | 13 | def get_vertex_normals(vertices, faces): 14 | """ Get vertex normals. [Vtx, Face_def] -> [Vtx_normals] """ 15 | vtx_nrm = lighting.vertex_normals(vertices=vertices, faces=faces) 16 | 17 | return vtx_nrm 18 | 19 | def get_pre_split_vertex_normals(vertices, faces): 20 | """ Get Pre-split vertex normals, computationlly slightly more efficient. [Vtx, Face_def] -> [Vtx_normals] """ 21 | norms_by_vertex = lighting.vertex_normals_pre_split(vertices=vertices, faces=faces) 22 | 23 | return norms_by_vertex 24 | 25 | def get_face_normals(vertices, faces): 26 | """ Get face normals along with constituent vertex IDs. [Vtx, Face_def] -> [Face_normals, vtx_ids] """ 27 | face_nrm, verts_idx = lighting._get_face_normals(vertices=vertices, faces=faces) 28 | 29 | return [face_nrm, verts_idx] 30 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | chumpy==0.68 2 | h5py==2.9.0 3 | imageio==2.5.0 4 | Keras==2.2.4 5 | matplotlib==2.2.4 6 | numpy==1.16.4 7 | opencv-contrib-python==4.1.0.25 8 | Pillow==6.1.0 9 | scikit-image==0.14.3 10 | scipy==1.2.2 11 | tensorflow-gpu==1.13.1 12 | tensorflow-graphics-gpu==1.0.0 13 | tqdm==4.33.0 14 | -------------------------------------------------------------------------------- /smpl/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/smpl/__init__.py -------------------------------------------------------------------------------- /smpl/batch_lbs.py: -------------------------------------------------------------------------------- 1 | # Code partly taken from https://github.com/akanazawa/hmr. 2 | # The following license applies: 3 | # 4 | # 5 | # MIT License 6 | # 7 | # This code base itself is MIT, but please follow the license for SMPL, MoSh data, 8 | # and the respective dataset. 9 | # 10 | # Copyright (c) 2018 akanazawa 11 | # 12 | # Permission is hereby granted, free of charge, to any person obtaining a copy 13 | # of this software and associated documentation files (the "Software"), to deal 14 | # in the Software without restriction, including without limitation the rights 15 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 16 | # copies of the Software, and to permit persons to whom the Software is 17 | # furnished to do so, subject to the following conditions: 18 | # 19 | # The above copyright notice and this permission notice shall be included in all 20 | # copies or substantial portions of the Software. 21 | # 22 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 23 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 24 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 25 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 26 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 27 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 28 | # SOFTWARE. 29 | 30 | 31 | """ Util functions for SMPL 32 | @@batch_skew 33 | @@batch_rodrigues 34 | @@batch_lrotmin 35 | @@batch_global_rigid_transformation 36 | """ 37 | 38 | from __future__ import absolute_import 39 | from __future__ import division 40 | from __future__ import print_function 41 | 42 | import tensorflow as tf 43 | 44 | 45 | def batch_skew(vec, batch_size=None): 46 | """ 47 | vec is N x 3, batch_size is int 48 | 49 | returns N x 3 x 3. Skew_sym version of each matrix. 50 | """ 51 | with tf.name_scope(name="batch_skew", values=[vec]): 52 | if batch_size is None: 53 | batch_size = vec.shape.as_list()[0] 54 | col_inds = tf.constant([1, 2, 3, 5, 6, 7]) 55 | indices = tf.reshape( 56 | tf.reshape(tf.range(0, batch_size) * 9, [-1, 1]) + col_inds, 57 | [-1, 1]) 58 | updates = tf.reshape( 59 | tf.stack( 60 | [ 61 | -vec[:, 2], vec[:, 1], vec[:, 2], -vec[:, 0], -vec[:, 1], 62 | vec[:, 0] 63 | ], 64 | axis=1), [-1]) 65 | out_shape = [batch_size * 9] 66 | res = tf.scatter_nd(indices, updates, out_shape) 67 | res = tf.reshape(res, [batch_size, 3, 3]) 68 | 69 | return res 70 | 71 | 72 | def batch_rodrigues(theta, name=None): 73 | """ 74 | Theta is N x 3 75 | """ 76 | with tf.name_scope(name, "batch_rodrigues", [theta]): 77 | batch_size = tf.shape(theta)[0] 78 | 79 | # angle = tf.norm(theta, axis=1) 80 | # r = tf.expand_dims(tf.div(theta, tf.expand_dims(angle + 1e-8, -1)), -1) 81 | # angle = tf.expand_dims(tf.norm(theta, axis=1) + 1e-8, -1) 82 | angle = tf.expand_dims(tf.norm(theta + 1e-8, axis=1), -1) 83 | r = tf.expand_dims(tf.div(theta, angle), -1) 84 | 85 | angle = tf.expand_dims(angle, -1) 86 | cos = tf.cos(angle) 87 | sin = tf.sin(angle) 88 | 89 | outer = tf.matmul(r, r, transpose_b=True, name="outer") 90 | 91 | #eyes = tf.tile(tf.expand_dims(tf.cast(tf.eye(3),tf.float64), 0), [batch_size, 1, 1]) 92 | 93 | eyes = tf.tile(tf.expand_dims(tf.cast(tf.eye(3),tf.float32), 0), [batch_size, 1, 1]) 94 | R = cos * eyes + (1 - cos) * outer + sin * batch_skew( 95 | r, batch_size=batch_size) 96 | return R 97 | 98 | 99 | def batch_lrotmin(theta, name=None): 100 | """ NOTE: not used bc I want to reuse R and this is simple. 101 | Output of this is used to compute joint-to-pose blend shape mapping. 102 | Equation 9 in SMPL paper. 103 | 104 | 105 | Args: 106 | pose: `Tensor`, N x 72 vector holding the axis-angle rep of K joints. 107 | This includes the global rotation so K=24 108 | 109 | Returns 110 | diff_vec : `Tensor`: N x 207 rotation matrix of 23=(K-1) joints with identity subtracted., 111 | """ 112 | with tf.name_scope(name, "batch_lrotmin", [theta]): 113 | with tf.name_scope("ignore_global"): 114 | theta = theta[:, 3:] 115 | 116 | # N*23 x 3 x 3 117 | Rs = batch_rodrigues(tf.reshape(theta, [-1, 3])) 118 | lrotmin = tf.reshape(Rs - tf.eye(3), [-1, 207]) 119 | 120 | return lrotmin 121 | 122 | 123 | def batch_global_rigid_transformation(Rs, Js, parent, rotate_base=False): 124 | """ 125 | Computes absolute joint locations given pose. 126 | 127 | rotate_base: if True, rotates the global rotation by 90 deg in x axis. 128 | if False, this is the original SMPL coordinate. 129 | 130 | Args: 131 | Rs: N x 24 x 3 x 3 rotation vector of K joints 132 | Js: N x 24 x 3, joint locations before posing 133 | parent: 24 holding the parent id for each index 134 | 135 | Returns 136 | new_J : `Tensor`: N x 24 x 3 location of absolute joints 137 | A : `Tensor`: N x 24 4 x 4 relative joint transformations for LBS. 138 | """ 139 | with tf.name_scope(name="batch_forward_kinematics", values=[Rs, Js]): 140 | N = tf.shape(Rs)[0] 141 | if rotate_base: 142 | print('Flipping the SMPL coordinate frame!!!!') 143 | rot_x = tf.constant( 144 | [[1, 0, 0], [0, -1, 0], [0, 0, -1]], dtype=Rs.dtype) 145 | rot_x = tf.reshape(tf.tile(rot_x, [N, 1]), [N, 3, 3]) 146 | root_rotation = tf.matmul(Rs[:, 0, :, :], rot_x) 147 | else: 148 | root_rotation = Rs[:, 0, :, :] 149 | 150 | # Now Js is N x 24 x 3 x 1 151 | Js = tf.expand_dims(Js, -1) 152 | 153 | def make_A(R, t, name=None): 154 | # Rs is N x 3 x 3, ts is N x 3 x 1 155 | with tf.name_scope(name, "Make_A", [R, t]): 156 | R_homo = tf.pad(R, [[0, 0], [0, 1], [0, 0]]) 157 | t_homo = tf.concat([t, tf.ones([N, 1, 1])], 1) 158 | return tf.concat([R_homo, t_homo], 2) 159 | 160 | A0 = make_A(root_rotation, Js[:, 0]) 161 | results = [A0] 162 | for i in range(1, parent.shape[0]): 163 | j_here = Js[:, i] - Js[:, parent[i]] 164 | A_here = make_A(Rs[:, i], j_here) 165 | res_here = tf.matmul( 166 | results[parent[i]], A_here, name="propA%d" % i) 167 | results.append(res_here) 168 | 169 | # 10 x 24 x 4 x 4 170 | results = tf.stack(results, axis=1) 171 | 172 | new_J = results[:, :, :3, 3] 173 | 174 | # --- Compute relative A: Skinning is based on 175 | # how much the bone moved (not the final location of the bone) 176 | # but (final_bone - init_bone) 177 | # --- 178 | Js_w0 = tf.concat([Js, tf.zeros([N, 24, 1, 1])], 2) 179 | init_bone = tf.matmul(results, Js_w0) 180 | # Append empty 4 x 3: 181 | init_bone = tf.pad(init_bone, [[0, 0], [0, 0], [0, 0], [3, 0]]) 182 | A = results - init_bone 183 | 184 | return new_J, A 185 | -------------------------------------------------------------------------------- /smpl/batch_smpl.py: -------------------------------------------------------------------------------- 1 | # Code partly taken from https://github.com/akanazawa/hmr. 2 | # The following license applies: 3 | # 4 | # 5 | # MIT License 6 | # 7 | # This code base itself is MIT, but please follow the license for SMPL, MoSh data, 8 | # and the respective dataset. 9 | # 10 | # Copyright (c) 2018 akanazawa 11 | # 12 | # Permission is hereby granted, free of charge, to any person obtaining a copy 13 | # of this software and associated documentation files (the "Software"), to deal 14 | # in the Software without restriction, including without limitation the rights 15 | # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 16 | # copies of the Software, and to permit persons to whom the Software is 17 | # furnished to do so, subject to the following conditions: 18 | # 19 | # The above copyright notice and this permission notice shall be included in all 20 | # copies or substantial portions of the Software. 21 | # 22 | # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 23 | # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 24 | # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 25 | # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 26 | # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 27 | # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 28 | # SOFTWARE. 29 | 30 | 31 | """ 32 | Tensorflow SMPL implementation as batch. 33 | Specify joint types: 34 | 'coco': Returns COCO+ 19 joints 35 | 'lsp': Returns H3.6M-LSP 14 joints 36 | Note: To get original smpl joints, use self.J_transformed 37 | """ 38 | 39 | from __future__ import absolute_import 40 | from __future__ import division 41 | from __future__ import print_function 42 | 43 | import sys 44 | import numpy as np 45 | 46 | import tensorflow as tf 47 | from .batch_lbs import batch_rodrigues, batch_global_rigid_transformation 48 | 49 | if sys.version_info[0] == 3: 50 | import _pickle as pickle 51 | else: 52 | import cPickle as pickle 53 | 54 | 55 | # There are chumpy variables so convert them to numpy. 56 | def undo_chumpy(x): 57 | return x if isinstance(x, np.ndarray) else x.r 58 | 59 | 60 | def sparse_to_tensor(x, dtype=tf.float32): 61 | coo = x.tocoo() 62 | indices = np.mat([coo.row, coo.col]).transpose() 63 | return tf.SparseTensor(indices, tf.convert_to_tensor(coo.data, dtype=dtype), coo.shape) 64 | 65 | 66 | class SMPL(object): 67 | def __init__(self, pkl_path, theta_in_rodrigues=True, theta_is_perfect_rotmtx=True, dtype=tf.float32): 68 | """ 69 | pkl_path is the path to a SMPL model 70 | """ 71 | # -- Load SMPL params -- 72 | with open(pkl_path, 'r') as f: 73 | dd = pickle.load(f) 74 | # Mean template vertices 75 | self.v_template = tf.Variable( 76 | undo_chumpy(dd['v_template']), 77 | name='v_template', 78 | dtype=dtype, 79 | trainable=False) 80 | # Size of mesh [Number of vertices, 3] 81 | self.size = [self.v_template.shape[0].value, 3] 82 | self.num_betas = dd['shapedirs'].shape[-1] 83 | # Shape blend shape basis: 6980 x 3 x 10 84 | # reshaped to 6980*30 x 10, transposed to 10x6980*3 85 | shapedir = np.reshape( 86 | undo_chumpy(dd['shapedirs']), [-1, self.num_betas]).T 87 | self.shapedirs = tf.Variable( 88 | shapedir, name='shapedirs', dtype=dtype, trainable=False) 89 | 90 | # Regressor for joint locations given shape - 6890 x 24 91 | self.J_regressor = sparse_to_tensor(dd['J_regressor'], dtype=dtype) 92 | #print(self.J_regressor.get_shape(),"THIS IS SHAPE OF J_REGRESSOR") 93 | 94 | # Pose blend shape basis: 6890 x 3 x 207, reshaped to 6890*30 x 207 95 | num_pose_basis = dd['posedirs'].shape[-1] 96 | # 207 x 20670 97 | posedirs = np.reshape( 98 | undo_chumpy(dd['posedirs']), [-1, num_pose_basis]).T 99 | self.posedirs = tf.Variable( 100 | posedirs, name='posedirs', dtype=dtype, trainable=False) 101 | 102 | # indices of parents for each joints 103 | self.parents = dd['kintree_table'][0].astype(np.int32) 104 | 105 | # LBS weights 106 | self.weights = tf.Variable( 107 | undo_chumpy(dd['weights']), 108 | name='lbs_weights', 109 | dtype=dtype, 110 | trainable=False) 111 | 112 | # expect theta in rodrigues form 113 | self.theta_in_rodrigues = theta_in_rodrigues 114 | 115 | # if in matrix form, is it already rotmax? 116 | self.theta_is_perfect_rotmtx = theta_is_perfect_rotmtx 117 | 118 | #print("THIS IS INITILIAZED") 119 | 120 | def __call__(self, theta, beta, trans, v_personal, name=None): 121 | 122 | """ 123 | Obtain SMPL with shape (beta) & pose (theta) inputs. 124 | Theta includes the global rotation. 125 | Args: 126 | beta: N x 10 127 | theta: N x 72 (with 3-D axis-angle rep) 128 | 129 | Updates: 130 | self.J_transformed: N x 24 x 3 joint location after shaping 131 | & posing with beta and theta 132 | Returns: 133 | - joints: N x 19 or 14 x 3 joint locations depending on joint_type 134 | If get_skin is True, also returns 135 | - Verts: N x 6980 x 3 136 | """ 137 | #print(theta.shape,"theta") 138 | #print(beta.shape,"beta") 139 | #print(trans.shape,"trans") 140 | 141 | with tf.name_scope(name, "smpl_main", [beta, theta, trans, v_personal]): 142 | num_batch = tf.shape(beta)[0] 143 | 144 | # 1. Add shape blend shapes 145 | # (N x 10) x (10 x 6890*3) = N x 6890 x 3 146 | 147 | #print(beta.shape) 148 | #print(self.shapedirs.shape,"shapedirs") 149 | 150 | self.shapedirs = tf.cast(self.shapedirs,dtype=tf.float32) 151 | self.v_template = tf.cast(self.v_template,dtype=tf.float32) 152 | #beta = tf.cast(beta,dtype=tf.float32) 153 | v_shaped_scaled = tf.reshape( 154 | tf.matmul(beta, self.shapedirs, name='shape_bs'), 155 | [-1, self.size[0], self.size[1]]) + self.v_template 156 | 157 | body_height = (v_shaped_scaled[:, 2802, 1] + v_shaped_scaled[:, 6262, 1]) - (v_shaped_scaled[:, 2237, 1] + v_shaped_scaled[:, 6728, 1]) 158 | scale = tf.reshape(1.66 / body_height, (-1, 1, 1)) 159 | 160 | self.v_shaped = scale * v_shaped_scaled 161 | self.v_shaped_personal = self.v_shaped + v_personal 162 | 163 | # 2. Infer shape-dependent joint locations. 164 | Jx = tf.transpose(tf.sparse_tensor_dense_matmul(self.J_regressor, tf.transpose(v_shaped_scaled[:, :, 0]))) 165 | Jy = tf.transpose(tf.sparse_tensor_dense_matmul(self.J_regressor, tf.transpose(v_shaped_scaled[:, :, 1]))) 166 | Jz = tf.transpose(tf.sparse_tensor_dense_matmul(self.J_regressor, tf.transpose(v_shaped_scaled[:, :, 2]))) 167 | J = scale * tf.stack([Jx, Jy, Jz], axis=2) 168 | 169 | # 3. Add pose blend shapes 170 | # N x 24 x 3 x 3 171 | if self.theta_in_rodrigues: 172 | Rs = tf.reshape( 173 | batch_rodrigues(tf.reshape(theta, [-1, 3])), [-1, 24, 3, 3]) 174 | else: 175 | if self.theta_is_perfect_rotmtx: 176 | Rs = theta 177 | else: 178 | #print("Here") 179 | s, u, v = tf.svd(theta) 180 | Rs = tf.matmul(u, tf.transpose(v, perm=[0, 1, 3, 2])) 181 | 182 | with tf.name_scope("lrotmin"): 183 | # Ignore global rotation. 184 | pose_feature = tf.reshape(Rs[:, 1:, :, :] - tf.eye(3), [-1, 207]) 185 | 186 | # (N x 207) x (207, 20670) -> N x 6890 x 3 187 | self.v_posed = tf.reshape( 188 | tf.matmul(pose_feature, self.posedirs), 189 | [-1, self.size[0], self.size[1]]) + self.v_shaped_personal 190 | 191 | #4. Get the global joint location 192 | self.J_transformed, A = batch_global_rigid_transformation(Rs, J, self.parents,rotate_base=False) 193 | 194 | self.J_transformed += tf.expand_dims(trans, axis=1) 195 | 196 | 197 | # 5. Do skinning: 198 | # W is N x 6890 x 24 199 | W = tf.reshape( 200 | tf.tile(self.weights, [num_batch, 1]), [num_batch, -1, 24]) 201 | # (N x 6890 x 24) x (N x 24 x 16) 202 | T = tf.reshape( 203 | tf.matmul(W, tf.reshape(A, [num_batch, 24, 16])), 204 | [num_batch, -1, 4, 4]) 205 | v_posed_homo = tf.concat( 206 | [self.v_posed, tf.ones([num_batch, self.v_posed.shape[1], 1])], 2) 207 | v_homo = tf.matmul(T, tf.expand_dims(v_posed_homo, -1)) 208 | 209 | verts = v_homo[:, :, :3, 0] 210 | #print(trans.shape,"this is trans shape inside batch smpl") 211 | verts_t = verts + tf.expand_dims(trans, axis=1) 212 | 213 | return verts_t, self.J_transformed 214 | 215 | -------------------------------------------------------------------------------- /smpl/bodyparts.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python2 2 | # -*- coding: utf-8 -*- 3 | 4 | import os 5 | import sys 6 | import numpy as np 7 | 8 | if sys.version_info[0] == 3: 9 | import _pickle as pkl 10 | else: 11 | import cPickle as pkl 12 | 13 | 14 | _cache = None 15 | 16 | 17 | def get_bodypart_vertex_ids(): 18 | global _cache 19 | 20 | if _cache is None: 21 | with open(os.path.join(os.path.dirname(__file__), '../assets/bodyparts.pkl'), 'rb') as fp: 22 | _cache = pkl.load(fp) 23 | 24 | return _cache 25 | 26 | 27 | def regularize_laplace(): 28 | reg = np.ones(6890) 29 | v_ids = get_bodypart_vertex_ids() 30 | 31 | reg[v_ids['face']] = 8. 32 | reg[v_ids['hand_l']] = 5. 33 | reg[v_ids['hand_r']] = 5. 34 | reg[v_ids['fingers_l']] = 8. 35 | reg[v_ids['fingers_r']] = 8. 36 | reg[v_ids['foot_l']] = 5. 37 | reg[v_ids['foot_r']] = 5. 38 | reg[v_ids['toes_l']] = 8. 39 | reg[v_ids['toes_r']] = 8. 40 | reg[v_ids['ear_l']] = 10. 41 | reg[v_ids['ear_r']] = 10. 42 | 43 | return reg 44 | 45 | 46 | def regularize_symmetry(): 47 | reg = np.ones(6890) 48 | v_ids = get_bodypart_vertex_ids() 49 | 50 | reg[v_ids['face']] = 10. 51 | reg[v_ids['hand_l']] = 10. 52 | reg[v_ids['hand_r']] = 10. 53 | reg[v_ids['foot_l']] = 10. 54 | reg[v_ids['foot_r']] = 10. 55 | reg[v_ids['ear_l']] = 5. 56 | reg[v_ids['ear_r']] = 5. 57 | 58 | return reg 59 | -------------------------------------------------------------------------------- /smpl/joints.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | 4 | from lib.geometry import sparse_to_tensor, sparse_dense_matmul_batch_tile 5 | 6 | if sys.version_info[0] == 3: 7 | import _pickle as pkl 8 | else: 9 | import cPickle as pkl 10 | 11 | body_25_reg = None 12 | face_reg = None 13 | 14 | 15 | def joints_body25(v): 16 | global body_25_reg 17 | 18 | if body_25_reg is None: 19 | 20 | body_25_reg = sparse_to_tensor( 21 | pkl.load(open(os.path.join(os.path.dirname(__file__), '../assets/J_regressor.pkl'), 'rb')).T 22 | ) 23 | 24 | return sparse_dense_matmul_batch_tile(body_25_reg, v) 25 | 26 | 27 | def face_landmarks(v): 28 | global face_reg 29 | 30 | if face_reg is None: 31 | face_reg = sparse_to_tensor( 32 | pkl.load(open(os.path.join(os.path.dirname(__file__), '../assets/face_regressor.pkl'), 'rb')).T 33 | ) 34 | 35 | return sparse_dense_matmul_batch_tile(face_reg, v) 36 | -------------------------------------------------------------------------------- /smpl/lib/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/smpl/lib/__init__.py -------------------------------------------------------------------------------- /smpl/lib/geometry.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import tensorflow as tf 3 | 4 | 5 | def sparse_to_tensor(x): 6 | coo = x.tocoo() 7 | indices = np.mat([coo.row, coo.col]).transpose() 8 | return tf.sparse_reorder(tf.SparseTensor(indices, coo.data, coo.shape)) 9 | 10 | 11 | def sparse_dense_matmul_batch(a, b): 12 | num_b = tf.shape(b)[0] 13 | shape = a.dense_shape 14 | 15 | indices = tf.reshape(a.indices, (num_b, -1, 3)) 16 | values = tf.reshape(a.values, (num_b, -1)) 17 | 18 | def matmul((i, bb)): 19 | sp = tf.SparseTensor(indices[i, :, 1:], values[i], shape[1:]) 20 | return i, tf.sparse_tensor_dense_matmul(sp, bb) 21 | 22 | _, p = tf.map_fn(matmul, (tf.range(num_b), b)) 23 | 24 | return p 25 | 26 | 27 | def sparse_dense_matmul_batch_tile(a, b): 28 | return tf.map_fn(lambda x: tf.sparse_tensor_dense_matmul(a, x), b) 29 | 30 | 31 | def batch_laplacian(v, f): 32 | # v: B x N x 3 33 | # f: M x 3 34 | 35 | num_b = tf.shape(v)[0] 36 | num_v = tf.shape(v)[1] 37 | num_f = tf.shape(f)[0] 38 | 39 | v_a = f[:, 0] 40 | v_b = f[:, 1] 41 | v_c = f[:, 2] 42 | 43 | a = tf.gather(v, v_a, axis=1) 44 | b = tf.gather(v, v_b, axis=1) 45 | c = tf.gather(v, v_c, axis=1) 46 | 47 | ab = a - b 48 | bc = b - c 49 | ca = c - a 50 | 51 | cot_a = -1 * tf.reduce_sum(ab * ca, axis=2) / tf.sqrt(tf.reduce_sum(tf.cross(ab, ca) ** 2, axis=-1)) 52 | cot_b = -1 * tf.reduce_sum(bc * ab, axis=2) / tf.sqrt(tf.reduce_sum(tf.cross(bc, ab) ** 2, axis=-1)) 53 | cot_c = -1 * tf.reduce_sum(ca * bc, axis=2) / tf.sqrt(tf.reduce_sum(tf.cross(ca, bc) ** 2, axis=-1)) 54 | 55 | I = tf.tile(tf.expand_dims(tf.concat((v_a, v_c, v_a, v_b, v_b, v_c), axis=0), 0), (num_b, 1)) 56 | J = tf.tile(tf.expand_dims(tf.concat((v_c, v_a, v_b, v_a, v_c, v_b), axis=0), 0), (num_b, 1)) 57 | 58 | W = 0.5 * tf.concat((cot_b, cot_b, cot_c, cot_c, cot_a, cot_a), axis=1) 59 | 60 | batch_dim = tf.tile(tf.expand_dims(tf.range(num_b), 1), (1, num_f * 6)) 61 | 62 | indices = tf.reshape(tf.stack((batch_dim, J, I), axis=2), (num_b, 6, -1, 3)) 63 | W = tf.reshape(W, (num_b, 6, -1)) 64 | 65 | l_indices = [tf.cast(tf.reshape(indices[:, i], (-1, 3)), tf.int64) for i in range(6)] 66 | shape = tf.cast(tf.stack((num_b, num_v, num_v)), tf.int64) 67 | sp_L_raw = [tf.sparse_reorder(tf.SparseTensor(l_indices[i], tf.reshape(W[:, i], (-1,)), shape)) for i in range(6)] 68 | 69 | L = sp_L_raw[0] 70 | for i in range(1, 6): 71 | L = tf.sparse_add(L, sp_L_raw[i]) 72 | 73 | dia_values = tf.sparse_reduce_sum(L, axis=-1) * -1 74 | 75 | I = tf.tile(tf.expand_dims(tf.range(num_v), 0), (num_b, 1)) 76 | batch_dim = tf.tile(tf.expand_dims(tf.range(num_b), 1), (1, num_v)) 77 | indices = tf.reshape(tf.stack((batch_dim, I, I), axis=2), (-1, 3)) 78 | 79 | dia = tf.sparse_reorder(tf.SparseTensor(tf.cast(indices, tf.int64), tf.reshape(dia_values, (-1,)), shape)) 80 | 81 | return tf.sparse_add(L, dia) 82 | 83 | 84 | def compute_laplacian_diff(v0, v1, f): 85 | L0 = batch_laplacian(v0, f) 86 | L1 = batch_laplacian(v1, f) 87 | 88 | return sparse_dense_matmul_batch(L0, v0) - sparse_dense_matmul_batch(L1, v1) 89 | -------------------------------------------------------------------------------- /smpl/lib/io.py: -------------------------------------------------------------------------------- 1 | import cv2 2 | import json 3 | import numpy as np 4 | 5 | 6 | LABELS_FULL = { 7 | 'Sunglasses': [170, 0, 51], 8 | 'LeftArm': [51, 170, 221], 9 | 'RightArm': [0, 255, 255], 10 | 'LeftLeg': [85, 255, 170], 11 | 'RightLeg': [170, 255, 85], 12 | 'LeftShoe': [255, 255, 0], 13 | 'RightShoe': [255, 170, 0], 14 | } 15 | 16 | LABELS_CLOTHING= { 17 | 'Face': [0, 0, 255], 18 | 'Arms': [51, 170, 221], 19 | 'Legs': [85, 255, 170], 20 | 'Shoes': [255, 255, 0] 21 | } 22 | 23 | 24 | def read_segmentation(file): 25 | segm = cv2.imread(file)[:, :, ::-1] 26 | 27 | segm[np.all(segm == LABELS_FULL['Sunglasses'], axis=2)] = LABELS_CLOTHING['Face'] 28 | segm[np.all(segm == LABELS_FULL['LeftArm'], axis=2)] = LABELS_CLOTHING['Arms'] 29 | segm[np.all(segm == LABELS_FULL['RightArm'], axis=2)] = LABELS_CLOTHING['Arms'] 30 | segm[np.all(segm == LABELS_FULL['LeftLeg'], axis=2)] = LABELS_CLOTHING['Legs'] 31 | segm[np.all(segm == LABELS_FULL['RightLeg'], axis=2)] = LABELS_CLOTHING['Legs'] 32 | segm[np.all(segm == LABELS_FULL['LeftShoe'], axis=2)] = LABELS_CLOTHING['Shoes'] 33 | segm[np.all(segm == LABELS_FULL['RightShoe'], axis=2)] = LABELS_CLOTHING['Shoes'] 34 | 35 | return segm[:, :, ::-1] / 255. 36 | 37 | 38 | def openpose_from_file(file, resolution=(1080, 1080), person=0): 39 | with open(file) as f: 40 | data = json.load(f)['people'][person] 41 | 42 | pose = np.array(data['pose_keypoints_2d']).reshape(-1, 3) 43 | pose[:, 2] /= np.expand_dims(np.mean(pose[:, 2][pose[:, 2] > 0.1]), -1) 44 | pose = pose * np.array([2. / resolution[1], -2. / resolution[0], 1.]) + np.array([-1., 1., 0.]) 45 | pose[:, 0] *= 1. * resolution[1] / resolution[0] 46 | 47 | face = np.array(data['face_keypoints_2d']).reshape(-1, 3) 48 | face = face * np.array([2. / resolution[1], -2. / resolution[0], 1.]) + np.array([-1., 1., 0.]) 49 | face[:, 0] *= 1. * resolution[1] / resolution[0] 50 | 51 | return pose, face 52 | 53 | 54 | def write_mesh(filename, v, f): 55 | with open(filename, 'w') as fp: 56 | fp.write(('v {:f} {:f} {:f}\n' * len(v)).format(*v.reshape(-1))) 57 | fp.write(('f {:d} {:d} {:d}\n' * len(f)).format(*(f.reshape(-1) + 1))) 58 | -------------------------------------------------------------------------------- /smpl/smpl_layer.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | from batch_smpl import SMPL 3 | #from joints import joints_body25, face_landmarks 4 | from keras.engine.topology import Layer 5 | 6 | 7 | class SmplTPoseLayer(Layer): 8 | 9 | def __init__(self, model='assets/neutral_smpl.pkl', theta_in_rodrigues=False, theta_is_perfect_rotmtx=True, **kwargs): 10 | self.smpl = SMPL(model, theta_in_rodrigues, theta_is_perfect_rotmtx) 11 | super(SmplTPoseLayer, self).__init__(**kwargs) 12 | 13 | def call(self, (pose, betas, trans, v_personal)): 14 | smpl_out = self.smpl(pose, betas, trans, v_personal) 15 | verts = smpl_out[0] 16 | joints_3d = smpl_out[1] 17 | 18 | return [verts, joints_3d ]#, self.smpl.v_shaped_personal, self.smpl.v_shaped] 19 | 20 | def compute_output_shape(self, input_shape): 21 | shape = input_shape[0][0], 6890, 3 22 | 23 | return [shape, shape, shape] 24 | 25 | ''' 26 | class SmplBody25FaceLayer(Layer): 27 | 28 | def __init__(self, model='assets/neutral_smpl.pkl', theta_in_rodrigues=False, theta_is_perfect_rotmtx=True, **kwargs): 29 | self.smpl = SMPL(model, theta_in_rodrigues, theta_is_perfect_rotmtx) 30 | super(SmplBody25FaceLayer, self).__init__(**kwargs) 31 | 32 | def call(self, (pose, betas, trans)): 33 | v_personal = tf.tile(tf.zeros((1, 6890, 3)), (tf.shape(betas)[0], 1, 1)) 34 | 35 | v = self.smpl(pose, betas, trans, v_personal) 36 | 37 | return tf.concat((joints_body25(v), face_landmarks(v)), axis=1) 38 | 39 | def compute_output_shape(self, input_shape): 40 | return input_shape[0][0], 95, 3 41 | ''' -------------------------------------------------------------------------------- /teaser_img.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/val-iisc/ss_human_mesh/f9c7fcf577c83316eb610753e3f5678b7b5e24c5/teaser_img.png -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | 2 | """ General Utilities file. """ 3 | 4 | import sys 5 | import os 6 | 7 | ############################ NON-TF UTILS ########################## 8 | 9 | from skimage.util import img_as_float 10 | import numpy as np 11 | import cv2 12 | import pickle 13 | from PIL import Image 14 | from io import BytesIO 15 | import math 16 | import tqdm 17 | import scipy 18 | import json 19 | 20 | import matplotlib 21 | gui_env = ['Agg','TKAgg','GTKAgg','Qt4Agg','WXAgg'] 22 | for gui in gui_env: 23 | try: 24 | print ("testing", gui) 25 | matplotlib.use(gui,warn=False, force=True) 26 | from matplotlib import pyplot as plt 27 | break 28 | except: 29 | continue 30 | print ("utils.py Using:",matplotlib.get_backend()) 31 | 32 | from matplotlib.backends.backend_agg import FigureCanvasAgg as Canvas 33 | from mpl_toolkits.mplot3d import Axes3D 34 | 35 | import config as cfg 36 | 37 | ######### Basic Utils ######### 38 | 39 | def adjust_gamma(image, gamma=1.0): 40 | """ Gamma correct images. """ 41 | ## Build a LUT mapping the pixel values [0, 255] to their adjusted gamma values 42 | invGamma = 1.0 / gamma 43 | table = np.array([((i / 255.0) ** invGamma) * 255 44 | for i in np.arange(0, 256)]).astype("uint8") 45 | 46 | ## Apply gamma correction using the LUT 47 | return cv2.LUT(image, table) 48 | 49 | 50 | def scipy_sharpen(img_flt, alpha=30): 51 | """ Sharpen images. """ 52 | from scipy import ndimage 53 | 54 | blurred_f = ndimage.gaussian_filter(img_flt, 3) 55 | filter_blurred_f = ndimage.gaussian_filter(blurred_f, 1) 56 | 57 | img_flt = blurred_f + alpha * (blurred_f - filter_blurred_f) 58 | return img_flt 59 | 60 | def read_pickle(path): 61 | """ Load Pickle file. """ 62 | with open(path, 'rb') as f: 63 | data = pickle.load(f) 64 | 65 | return data 66 | 67 | def save_pickle(data, path): 68 | """ Save Pickle file. """ 69 | with open(path, 'wb') as f: 70 | pickle.dump(data, f, protocol=pickle.HIGHEST_PROTOCOL) 71 | 72 | ######### Pose quality and Metrics ######### 73 | 74 | def compute_similarity_transform(S1, S2): 75 | """ Computes a similarity transform (sR, t) that takes 76 | a set of 3D points S1 (3 x N) closest to a set of 3D points S2, 77 | where R is an 3x3 rotation matrix, t 3x1 translation, s scale. 78 | i.e. solves the orthogonal Procrutes problem. """ 79 | transposed = False 80 | if S1.shape[0] != 3 and S2.shape[0] != 3: 81 | S1 = S1.T 82 | S2 = S2.T 83 | transposed = True 84 | assert(S2.shape[1] == S1.shape[1]) 85 | 86 | ## Mean 87 | mu1 = S1.mean(axis=1, keepdims=True) 88 | mu2 = S2.mean(axis=1, keepdims=True) 89 | X1 = S1 - mu1 90 | X2 = S2 - mu2 91 | 92 | ## Compute variance of X1 used for scale 93 | var1 = np.sum(X1**2) 94 | 95 | ## The outer product of X1 and X2 96 | K = X1.dot(X2.T) 97 | 98 | ## Solution that Maximizes trace(R'K) is R=U*V', where U, V are 99 | ## Singular vectors of K 100 | U, s, Vh = np.linalg.svd(K) 101 | V = Vh.T 102 | ## Construct Z that fixes the orientation of R to get det(R)=1 103 | Z = np.eye(U.shape[0]) 104 | Z[-1, -1] *= np.sign(np.linalg.det(U.dot(V.T))) 105 | ## Construct R 106 | R = V.dot(Z.dot(U.T)) 107 | 108 | ## Recover scale 109 | scale = np.trace(R.dot(K)) / var1 110 | 111 | ## Recover translation 112 | t = mu2 - scale*(R.dot(mu1)) 113 | 114 | ## Error 115 | S1_hat = scale*R.dot(S1) + t 116 | 117 | if transposed: 118 | S1_hat = S1_hat.T 119 | 120 | return S1_hat 121 | 122 | 123 | 124 | def compute_error(pred_3d_all, gt_3d_all, full_out=True): 125 | """ MPJPE and PA_MPJPE metric computation. """ 126 | pred_3d_all_flat = pred_3d_all.copy() 127 | pred_3d_all_flat = pred_3d_all_flat - pred_3d_all_flat[:, 0:1,:] 128 | gt_3d_all_flat = gt_3d_all.copy() 129 | gt_3d_all_flat = gt_3d_all_flat - gt_3d_all_flat[:, 0:1,:] 130 | 131 | joint_wise_error = [] 132 | error = [] 133 | pa_joint_wise_error = [] 134 | pa_error = [] 135 | 136 | for i in range(len(pred_3d_all_flat)): 137 | each_pred_3d = pred_3d_all_flat[i] 138 | each_gt_3d = gt_3d_all_flat[i] 139 | 140 | tmp_err = np.linalg.norm(each_pred_3d-each_gt_3d, axis=1) 141 | joint_wise_error.append(tmp_err) 142 | error.append(np.mean(tmp_err)) 143 | 144 | pred3d_sym = compute_similarity_transform(each_pred_3d.copy(), each_gt_3d.copy()) 145 | tmp_pa_err = np.linalg.norm(pred3d_sym-each_gt_3d, axis=1) 146 | pa_joint_wise_error.append(tmp_pa_err) 147 | pa_error.append(np.mean(tmp_pa_err)) 148 | 149 | joint_wise_error = np.array(joint_wise_error) 150 | 151 | if(full_out): 152 | mpjpe = np.mean(error)*1000 ### Note: unit is mm 153 | pampjpe = np.mean(pa_error)*1000 ### Note: unit is mm 154 | return mpjpe, pampjpe 155 | 156 | else: 157 | return error, pa_error 158 | 159 | ###### Alternative manual regressors ###### 160 | def smplx45_to_17j(pose_smpl): 161 | """ SMPLX 45 joint J3D to 17 joint J3D. """ 162 | ## Remove fingers 163 | pose_smpl = pose_smpl[:-10] 164 | ## Remove extra def feet 165 | pose_smpl = pose_smpl[:-6] 166 | ## Remove face 167 | pose_smpl = pose_smpl[:-5] 168 | ## Remove wrist 169 | pose_smpl = pose_smpl[:-2] 170 | ## Remove extra def spine 171 | pose_smpl = np.delete(pose_smpl, 3, 0) ## 3 172 | pose_smpl = np.delete(pose_smpl, 5, 0) ## 6 173 | pose_smpl = np.delete(pose_smpl, 7, 0) ## 9 174 | ## Remove torso 175 | pose_smpl = np.delete(pose_smpl, 10, 0) ## 10 176 | pose_smpl = np.delete(pose_smpl, 10, 0) ## 11 177 | 178 | ## Hip altitude increase and widen 179 | alt_f = 0.8 180 | wide_f = 8.0 181 | 182 | pelvis = pose_smpl[0].copy() 183 | r_hip = pose_smpl[2].copy() 184 | l_hip = pose_smpl[1].copy() 185 | 186 | ## Alt inc 187 | r_p_dir = pelvis - r_hip 188 | l_p_dir = pelvis - l_hip 189 | 190 | mag_rp = np.linalg.norm(r_p_dir) 191 | r_p_dir /= mag_rp 192 | mag_lp = np.linalg.norm(l_p_dir) 193 | l_p_dir /= mag_lp 194 | 195 | r_hip = r_hip + (r_p_dir*mag_rp*alt_f) 196 | l_hip = l_hip + (l_p_dir*mag_lp*alt_f) 197 | 198 | ## H-Widen 199 | hip_ctr = (r_hip + l_hip) / 2.0 200 | r_dir = r_hip - hip_ctr 201 | l_dir = l_hip - hip_ctr 202 | 203 | ## Unit vec 204 | mag = np.linalg.norm(r_dir) 205 | r_dir /= mag 206 | l_dir /= np.linalg.norm(l_dir) 207 | 208 | r_hip = r_hip + (r_dir*mag*wide_f) 209 | l_hip = l_hip + (l_dir*mag*wide_f) 210 | 211 | ## place back 212 | pose_smpl[2] = r_hip 213 | pose_smpl[1] = l_hip 214 | 215 | return pose_smpl 216 | 217 | def smpl23_to_17j_3d(pose_smpl): 218 | """ Simple SMPL 23 joint J3D to 17 joint J3D. """ 219 | smpl_to_17j = [ [0,1],[8,11], 220 | [12],[17],[19], ### or 15 , 17 221 | [13],[18], [20], ### or 16 , 18 222 | [14],[0],[3], 223 | [9,6],[9],[1], 224 | [4],[10,7],[10] ] 225 | 226 | pose_17j = np.zeros((len(smpl_to_17j),3)) 227 | for idx in range(len(smpl_to_17j)): 228 | sel_idx = smpl_to_17j[idx] 229 | if(len(sel_idx) == 2): 230 | pose_17j[idx] = (pose_smpl[sel_idx[0]] + pose_smpl[sel_idx[1]]) / 2.0 231 | 232 | else: 233 | pose_17j[idx] = pose_smpl[sel_idx[0]] 234 | 235 | return pose_17j 236 | 237 | """ SMPL J17 reordering vec. """ 238 | smpl_reorder_vec = [0, 9, 239 | 12, 14, 16, 240 | 11, 13, 15, 241 | 10, 242 | 2, 4, 6, 8, 243 | 1, 3, 5, 7 ] 244 | 245 | def reorder_smpl17_to_j17(pose_3d): 246 | """ SMPL reorder SMPL J17 to standard J17. """ 247 | pose_3d = pose_3d[smpl_reorder_vec] 248 | return pose_3d 249 | 250 | def smpl24_to_17j_adv(pose_smpl): 251 | """ Improved SMPL 23 joint J3D to 17 joint J3D. """ 252 | ## Hip altitude increase and widen 253 | alt_f = 0.8 254 | wide_f = 8.0 255 | 256 | pelvis = pose_smpl[0].copy() 257 | r_hip = pose_smpl[2].copy() 258 | l_hip = pose_smpl[1].copy() 259 | 260 | ## Alt inc 261 | r_p_dir = pelvis - r_hip 262 | l_p_dir = pelvis - l_hip 263 | 264 | mag_rp = np.linalg.norm(r_p_dir) 265 | r_p_dir /= mag_rp 266 | mag_lp = np.linalg.norm(l_p_dir) 267 | l_p_dir /= mag_lp 268 | 269 | r_hip = r_hip + (r_p_dir*mag_rp*alt_f) 270 | l_hip = l_hip + (l_p_dir*mag_lp*alt_f) 271 | 272 | ## H-Widen 273 | hip_ctr = (r_hip + l_hip) / 2.0 274 | r_dir = r_hip - hip_ctr 275 | l_dir = l_hip - hip_ctr 276 | 277 | ## Unit vec 278 | mag = np.linalg.norm(r_dir) 279 | r_dir /= mag 280 | l_dir /= np.linalg.norm(l_dir) 281 | 282 | r_hip = r_hip + (r_dir*mag*wide_f) 283 | l_hip = l_hip + (l_dir*mag*wide_f) 284 | 285 | ## Place back 286 | pose_smpl[2] = r_hip 287 | pose_smpl[1] = l_hip 288 | 289 | ## Neck to head raise with tilt towards nose 290 | alt_f = 0.7 291 | head = pose_smpl[15].copy() 292 | neck = pose_smpl[12].copy() 293 | 294 | ## Alt inc 295 | n_h_dir = head - neck 296 | mag_nh = np.linalg.norm(n_h_dir) 297 | n_h_dir /= mag_nh 298 | head = head + (n_h_dir*mag_nh*alt_f) 299 | 300 | ## Place back 301 | pose_smpl[15] = head 302 | 303 | ## Remove wrist 304 | pose_smpl = pose_smpl[:-2] 305 | 306 | ## Remove extra def spine 307 | pose_smpl = np.delete(pose_smpl, 3, 0) ## 3 308 | pose_smpl = np.delete(pose_smpl, 5, 0) ## 6 309 | pose_smpl = np.delete(pose_smpl, 7, 0) ## 9 310 | 311 | ## Remove torso 312 | pose_smpl = np.delete(pose_smpl, 10, 0) ## 10 313 | pose_smpl = np.delete(pose_smpl, 10, 0) ## 11 314 | 315 | return pose_smpl 316 | 317 | def hip_straighten(pose_smpl): 318 | """ Straighten Hip in J17. """ 319 | #pelvis = pose_smpl[0].copy() 320 | r_hip = pose_smpl[2].copy() 321 | l_hip = pose_smpl[1].copy() 322 | 323 | pelvis = (r_hip + l_hip) / 2 324 | 325 | pose_smpl[0] = pelvis 326 | 327 | return pose_smpl 328 | 329 | """ Limb parents for SMPL joints. """ 330 | limb_parents = [ 0, 331 | 0, 0, 0, 332 | 1, 2, 3, 4, 333 | 5, 6, 7, 8, 334 | 9, 9, 9, 335 | 12,12,12, 336 | 16,17,18,19,20,21 337 | ] 338 | 339 | """ 3D skeleton plot colours for SMPL joints. """ 340 | colors = np.array([[0,0,255], [0,255,0], [255,0,0], [255,0,255], [0,255,255], [255,255,0], [127,127,0], [0,127,0], [100,0,100], 341 | [255,0,255], [0,255,0], [0,0,255], [255,255,0], [127,127,0], [100,0,100], [175,100,195], 342 | [0,0,255], [0,255,0], [255,0,0], [255,0,255], [0,255,255], [255,255,0], [127,127,0], [0,127,0], [100,0,100], 343 | [255,0,255], [0,255,0], [0,0,255], [255,255,0], [127,127,0], [100,0,100], [175,100,195]]) 344 | 345 | def fig2data(fig): 346 | """ Convert a Matplotlib figure to a 4D numpy array with RGBA channels. """ 347 | ## Draw the renderer 348 | fig.canvas.draw() 349 | 350 | ## Get the RGBA buffer from the figure 351 | w, h = fig.canvas.get_width_height() 352 | buf = np.fromstring(fig.canvas.tostring_argb(), dtype=np.uint8) 353 | buf.shape = (w, h, 4) 354 | 355 | ## Roll the ALPHA channel to have it in RGBA mode 356 | buf = np.roll(buf, 3, axis=2) 357 | return buf 358 | 359 | def draw_limbs_3d_plt(joints_3d, ax, limb_parents=limb_parents): 360 | ## Direct 3d plotting 361 | for i in range(joints_3d.shape[0]): 362 | x_pair = [joints_3d[i, 0], joints_3d[limb_parents[i], 0]] 363 | y_pair = [joints_3d[i, 1], joints_3d[limb_parents[i], 1]] 364 | z_pair = [joints_3d[i, 2], joints_3d[limb_parents[i], 2]] 365 | #ax.text(joints_3d[i, 0], joints_3d[i, 1], joints_3d[i, 2], s=str(i)) 366 | ax.plot(x_pair, y_pair, z_pair, color=colors[i]/255.0, linewidth=3, antialiased=True) 367 | 368 | def plot_skeleton_3d(joints_3d, flag=-1, limb_parents=limb_parents, title=""): 369 | ## 3D Skeleton plotting 370 | fig = plt.figure(frameon=False, figsize=(7, 7)) 371 | ax = fig.add_subplot(1, 1, 1, projection='3d') 372 | ax.clear() 373 | 374 | ## Axis setup 375 | if (flag == 0): 376 | ax.view_init(azim=0, elev=0) 377 | elif (flag == 1): 378 | ax.view_init(azim=90, elev=0) 379 | 380 | ax.set_xlim(-200, 200) 381 | ax.set_ylim(-200, 200) 382 | ax.set_zlim(-200, 200) 383 | 384 | scale = 1 385 | ax.set_xlabel('x') 386 | ax.set_ylabel('y') 387 | ax.set_zlabel('z') 388 | draw_limbs_3d_plt(joints_3d * scale, ax, limb_parents) 389 | 390 | ax.set_title(title) 391 | plt_img = fig2data(fig) 392 | plt.close(fig) 393 | 394 | return plt_img 395 | 396 | def skeleton_image(joints_2d, img): 397 | """ 2D Joint skeleton Overlay. """ 398 | img_copy = img.copy() 399 | 400 | for i in range(joints_2d.shape[0]): 401 | x_pair = [joints_2d[i, 0], joints_2d[limb_parents[i], 0]] 402 | y_pair = [joints_2d[i, 1], joints_2d[limb_parents[i], 1]] 403 | img_copy = cv2.line(img_copy, (int(x_pair[0]),int(y_pair[0])), (int(x_pair[1]),int(y_pair[1])), colors[i],4) 404 | 405 | return img_copy 406 | 407 | def create_collage(img_list, axis=1): 408 | """ Collage a set of images to form a panel. (numpy) """ 409 | np_new_array = np.concatenate([i for i in img_list], axis=axis) 410 | return np_new_array 411 | 412 | def align_by_pelvis(joints): 413 | """ Center by pelvis joint. """ 414 | hip_id = 0 415 | joints -= joints[hip_id, :] 416 | return joints 417 | 418 | def mesh2d_center_by_nose(mesh2d,w=224 ,h=224): 419 | """ Simple mesh centering by nose/pelvis vtx. (numpy) """ 420 | #hip_id = 0 421 | nose_id = 0 422 | ctr = mesh2d[nose_id,:] 423 | mesh_ret = mesh2d - ctr + np.array([ w/2, h/5 ]) 424 | return mesh_ret 425 | 426 | def align_with_image_j2d(points2d, img_width, img_height): 427 | """ Perform center alignment to image coordinate system. (numpy) """ 428 | points2d[:,0] += img_width/2 429 | points2d[:,1] += img_height/2 430 | return points2d 431 | 432 | """ Input preprocess """ 433 | def get_transform(center, scale, res, rot=0): 434 | """ Generate transformation matrix. """ 435 | h = 224 * scale 436 | t = np.zeros((3, 3)) 437 | t[0, 0] = float(res[1]) / h 438 | t[1, 1] = float(res[0]) / h 439 | t[0, 2] = res[1] * (-float(center[0]) / h + .5) 440 | t[1, 2] = res[0] * (-float(center[1]) / h + .5) 441 | t[2, 2] = 1 442 | if not rot == 0: 443 | rot = -rot ## To match direction of rotation from cropping 444 | rot_mat = np.zeros((3,3)) 445 | rot_rad = rot * np.pi / 180 446 | sn,cs = np.sin(rot_rad), np.cos(rot_rad) 447 | rot_mat[0,:2] = [cs, -sn] 448 | rot_mat[1,:2] = [sn, cs] 449 | rot_mat[2,2] = 1 450 | ## Need to rotate around center 451 | t_mat = np.eye(3) 452 | t_mat[0,2] = -res[1]/2 453 | t_mat[1,2] = -res[0]/2 454 | t_inv = t_mat.copy() 455 | t_inv[:2,2] *= -1 456 | t = np.dot(t_inv,np.dot(rot_mat,np.dot(t_mat,t))) 457 | return t 458 | 459 | def transform(pt, center, scale, res, invert=0, rot=0): 460 | """ Transform pixel location to different reference. """ 461 | t = get_transform(center, scale, res, rot=rot) 462 | if invert: 463 | t = np.linalg.inv(t) 464 | new_pt = np.array([pt[0] - 1, pt[1] - 1, 1.]).T 465 | new_pt = np.dot(t, new_pt) 466 | return new_pt[:2].astype(int) + 1 467 | 468 | def crop(img, center, scale, res, rot=0): 469 | """ Crop image according to the supplied bounding box. """ 470 | ## Upper left point 471 | ul = np.array(transform([1, 1], center, scale, res, invert=1)) - 1 472 | ## Bottom right point 473 | br = np.array(transform([res[0]+1, res[1]+1], center, scale, res, invert=1)) - 1 474 | 475 | ## Padding so that when rotated proper amount of context is included 476 | pad = int(np.linalg.norm(br - ul) / 2 - float(br[1] - ul[1]) / 2) 477 | if not rot == 0: 478 | ul -= pad 479 | br += pad 480 | 481 | new_shape = [br[1] - ul[1], br[0] - ul[0]] 482 | if len(img.shape) > 2: 483 | new_shape += [img.shape[2]] 484 | new_img = np.zeros(new_shape) 485 | 486 | ## Range to fill new array 487 | new_x = max(0, -ul[0]), min(br[0], len(img[0])) - ul[0] 488 | new_y = max(0, -ul[1]), min(br[1], len(img)) - ul[1] 489 | ## Range to sample from original image 490 | old_x = max(0, ul[0]), min(len(img[0]), br[0]) 491 | old_y = max(0, ul[1]), min(len(img), br[1]) 492 | new_img[new_y[0]:new_y[1], new_x[0]:new_x[1]] = img[old_y[0]:old_y[1], old_x[0]:old_x[1]] 493 | 494 | if not rot == 0: 495 | ## Remove padding 496 | new_img = scipy.misc.imrotate(new_img, rot) 497 | new_img = new_img[pad:-pad, pad:-pad] 498 | 499 | new_img = scipy.misc.imresize(new_img, res) 500 | return new_img 501 | 502 | def j2d_crop(img, j2d_file, rescale=1.2, detection_thresh=0.2): 503 | """ Get center and scale for Bbox from OpenPose/Centertrack detections.""" 504 | with open(j2d_file, 'r') as f: 505 | keypoints = json.load(f)['people'][0]['pose_keypoints_2d'] 506 | keypoints = np.reshape(np.array(keypoints), (-1,3)) 507 | valid = keypoints[:,-1] > detection_thresh 508 | valid_keypoints = keypoints[valid][:,:-1] 509 | center = valid_keypoints.mean(axis=0) 510 | bbox_size = (valid_keypoints.max(axis=0) - valid_keypoints.min(axis=0)).max() 511 | ## Adjust bounding box tightness 512 | scale = bbox_size / 200.0 513 | scale *= rescale 514 | 515 | img = crop(img, center, scale, (cfg.IMG_W, cfg.IMG_H)) 516 | return img 517 | 518 | def bbox_crop(img, bbox): 519 | """ Crop, center and scale image based on BBox """ 520 | with open(bbox, 'r') as f: 521 | bbox = np.array(json.load(f)['bbox']).astype(np.float32) 522 | 523 | ul_corner = bbox[:2] 524 | center = ul_corner + 0.5 * bbox[2:] 525 | width = max(bbox[2], bbox[3]) 526 | scale = width / 200.0 527 | 528 | img = crop(img, center, scale, (cfg.IMG_W, cfg.IMG_H)) 529 | return img 530 | 531 | ########################### TF UTILS ############################# 532 | 533 | import pickle as pkl 534 | import tensorflow as tf 535 | import tensorflow_graphics as tfg 536 | 537 | from render.render_layer_ortho import RenderLayer 538 | import render.vertex_normal_expose as dirt_expose 539 | 540 | PI = np.pi 541 | 542 | def tfread_image(image,fmt='png', channels=3): 543 | """ Simple read and decode image. """ 544 | if (fmt == 'png'): 545 | return tf.image.decode_png(image, channels=channels) 546 | elif (fmt == 'jpg'): 547 | return tf.image.decode_jpeg(image, channels=channels) 548 | else: 549 | print ("ERROR specified format not found....") 550 | 551 | def tf_norm(tensor, axis=1): 552 | """ Min-Max normalize image. """ 553 | min_val = tf.reduce_min(tensor, axis=axis, keepdims=True) 554 | normalized_tensor = tf.div( tf.subtract(tensor, min_val), tf.subtract(tf.reduce_max(tensor, axis=axis, keepdims=True), min_val)) 555 | return normalized_tensor 556 | 557 | def tfresize_image(image, size=(cfg.IMG_W, cfg.IMG_H)): 558 | """ Resize image. """ 559 | return tf.image.resize(image, size) 560 | 561 | def denormalize_image(image): 562 | """ Undo normalization of image. """ 563 | image = (image / 2) + 0.5 564 | return image 565 | 566 | def unprocess_image(image): 567 | """ Undo preprocess image. """ 568 | # Normalize image to [0, 1] 569 | image = (image / 2) + 0.5 570 | image = image * 255.0 #[0,1] to [0,255] range 571 | 572 | return image 573 | 574 | def preprocess_image(image, do_znorm=True): 575 | """ Preprocess image. """ 576 | image = tf.image.decode_jpeg(image, channels=3) 577 | image = tf.image.resize(image, (cfg.IMG_W, cfg.IMG_H)) 578 | image /= 255.0 # normalize to [0,1] range 579 | 580 | if(do_znorm): 581 | # Normalize image to [-1, 1] 582 | image = 2 * (image - 0.5) 583 | 584 | return image 585 | 586 | def load_and_preprocess_image(path): 587 | """ Simple read and preprocess for just image. """ 588 | image = tf.io.read_file(path) 589 | processed_image = preprocess_image(image) 590 | return processed_image 591 | 592 | def load_and_preprocess_image_and_mask(path, j2d, j3d, beta, mask_path, pose, camera, data_id): 593 | """ Simple read and preprocess for image and mask. """ 594 | image = tf.io.read_file(path) 595 | proc_image = preprocess_image(image) 596 | 597 | ## For Mask 598 | mask = tf.io.read_file(mask_path) 599 | proc_mask = preprocess_image(mask, do_znorm=False) 600 | 601 | return proc_image, j2d, j3d, beta, proc_mask, pose, camera, data_id 602 | 603 | def tf_create_collage(img_list, axis=2): 604 | """ Collage a set of images to form a panel. """ 605 | tf_new_array = tf.concat([i for i in img_list], axis=axis) 606 | return tf_new_array 607 | 608 | def log_images(tag, image, step, writer): 609 | """ Logs a list of images to tensorboard. """ 610 | height, width, channel = image.shape 611 | image = Image.fromarray(image) 612 | output = BytesIO() 613 | image.save(output, format='PNG') 614 | image_string = output.getvalue() 615 | output.close() 616 | 617 | ## Create an Image object 618 | img_sum = tf.Summary.Image(height=height, 619 | width=width, 620 | colorspace=channel, 621 | encoded_image_string=image_string) 622 | ## Create a Summary value 623 | im_summary = tf.Summary.Value(tag='%s' % (tag), image=img_sum) 624 | 625 | ## Create and write Summary 626 | summary = tf.Summary(value=[im_summary]) 627 | writer.add_summary(summary, step) 628 | 629 | def get_network_params(scope): 630 | """ Get all accessable variables. """ 631 | return tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope=scope) 632 | 633 | def get_net_train_params(scope): 634 | """ Get Trainable params. """ 635 | return tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope=scope) 636 | 637 | def copy_weights(iter_no, wt_dir, label='best'): 638 | """ Backup the Weights to pretrained_weights/ given iteration number and label i.e 'iter' or 'best' """ 639 | files = os.listdir(wt_dir+label+"wt_") 640 | match_substr = '%s-%d' % (label, iter_no) 641 | files = [f for f in files if match_substr in f] 642 | for f in files: 643 | cmd = 'cp %s%s pretrained_weights/' % (wt_dir, f) 644 | print (cmd) 645 | os.system(cmd) 646 | 647 | def get_most_recent_iteration(wt_dir, label='iter'): 648 | """ Gets the most recent iteration number from weights/ dir of given label: ('best' or 'iter') """ 649 | files = os.listdir(wt_dir) 650 | files = [f for f in files if label in f] 651 | numbers = {long(f[f.index('-') + 1:f.index('.')]) for f in files} 652 | return max(numbers) 653 | 654 | def copy_latest(wt_dir, wt_type='best'): 655 | """ Backup latest weights. """ 656 | latest_iter = get_most_recent_iteration(label=wt_type, wt_dir=wt_dir) 657 | copy_weights(latest_iter, label=wt_type, wt_dir=wt_dir) 658 | return latest_iter 659 | 660 | def get_latest_iter(wt_dir, wt_type='best'): 661 | """ Get latest weights. """ 662 | latest_iter = get_most_recent_iteration(label=wt_type, wt_dir=wt_dir) 663 | return latest_iter 664 | 665 | def tf_align_by_pelvis(joints): 666 | """ Simple centering by pelvis location. """ 667 | hip_id = 0 668 | pelvis = joints[:, hip_id:hip_id+1, :] 669 | return tf.subtract(joints, pelvis) 670 | 671 | def tf_mesh2d_center_by_nose(mesh2d,w=224 ,h=224): 672 | """ Simple mesh centering by nose/pelvis vtx. """ 673 | #hip_id = 0 674 | nose_id = 0 675 | ctr = mesh2d[nose_id:nose_id+1,:] 676 | mesh_ret = tf.add(tf.subtract(mesh2d, ctr), [[ w/2, h/5 ]]) 677 | return mesh_ret 678 | 679 | def tf_perspective_project(points3d, focal, prin_pt, name="perspective_project"): 680 | """ Simple Perspective Projection. """ 681 | fx = focal[0] 682 | fy = focal[1] 683 | tx = prin_pt[0] 684 | ty = prin_pt[1] 685 | 686 | intrin = tf.convert_to_tensor(np.array([ [fx, 0., tx], 687 | [0., fy, ty], 688 | [0., 0., 1.]])) 689 | intrin = tf.tile(intrin,[points3d.shape[0]]) 690 | p_cam3d = tf.matmul(points3d, intrin, name=name) 691 | points2d = (points3d[:,:,0:2] / points3d[:,:,2]) ### project 692 | return points2d 693 | 694 | def tf_orthographic_project(points3d, name="orthographic_project"): 695 | """ Simple Orthographic Projection. """ 696 | return points3d[:,:,0:2] ## X,Y,Z 697 | 698 | def tf_dyn_scale_and_align(vertices, joints_3d, scale, add_trans): 699 | """ Dynamic scale and trans adjust. """ 700 | xy_max = tf.expand_dims(tf.reduce_max(vertices, axis=1), axis=1) 701 | xy_min = tf.expand_dims(tf.reduce_min(vertices, axis=1), axis=1) 702 | #person_ctr = (xy_max + xy_min)/2.0 703 | person_range = tf.abs(xy_max-xy_min) 704 | person_sc = tf.expand_dims(tf.reduce_max(person_range[:,:,0:2], axis=2), axis=2) 705 | 706 | ### Scale person to detector scale 707 | vertices = tf.div(vertices, person_sc) 708 | vertices = vertices * scale 709 | 710 | joints_3d = tf.div(joints_3d, person_sc) 711 | joints_3d = joints_3d * scale 712 | 713 | ### Bbox center 714 | xy_max = tf.expand_dims(tf.reduce_max(vertices, axis=1), axis=1) 715 | xy_min = tf.expand_dims(tf.reduce_min(vertices, axis=1), axis=1) 716 | person_ctr = (xy_max + xy_min)/2.0 717 | 718 | add_trans = tf.concat([add_trans, tf.zeros_like(add_trans[:,:,0:1])], axis=2) 719 | 720 | vertices = vertices - person_ctr + add_trans 721 | joints_3d = joints_3d - person_ctr + add_trans 722 | 723 | return vertices, joints_3d, scale[:,0], ((add_trans-person_ctr)[:,0,:2]) 724 | 725 | def tf_do_scale_and_align(vertices, joints_3d, scale, trans): 726 | """ Perform Scale and trans. (in world space) """ 727 | scale = tf.reshape(scale, [-1, 1, 1]) 728 | trans = tf.reshape(trans, [-1, 1, 2]) 729 | 730 | z = tf.zeros_like(trans[:,:,0:1]) 731 | shift = tf.concat([trans, z], axis=2) 732 | 733 | ### Trans in world space 734 | vertices = vertices + shift 735 | joints_3d = joints_3d + shift 736 | 737 | ### Scale person 738 | vertices = vertices * scale 739 | joints_3d = joints_3d * scale 740 | 741 | return vertices, joints_3d 742 | 743 | def for_tpix_tf_do_scale_and_align(vertices, joints_3d, scale, trans): 744 | """ Perform Scale and trans. (in Pixel space) """ 745 | xy_max = tf.expand_dims(tf.reduce_max(vertices, axis=1), axis=1) 746 | xy_min = tf.expand_dims(tf.reduce_min(vertices, axis=1), axis=1) 747 | #person_ctr = (xy_max + xy_min)/2.0 748 | person_range = tf.abs(xy_max-xy_min) 749 | person_sc = tf.expand_dims(tf.reduce_max(person_range[:,:,0:2], axis=2), axis=2) ##ignore z 750 | 751 | ### Unit scale 752 | vertices = tf.div(vertices, person_sc) 753 | joints_3d = tf.div(joints_3d, person_sc) 754 | 755 | ### 756 | scale = tf.reshape(scale, [-1, 1, 1]) 757 | trans = tf.reshape(trans, [-1, 1, 2]) 758 | 759 | z = tf.zeros_like(trans[:,:,0:1]) 760 | shift = tf.concat([trans, z], axis=2) 761 | 762 | ### Scale person 763 | vertices = vertices * scale 764 | joints_3d = joints_3d * scale 765 | 766 | ### Trans in cam space 767 | vertices = vertices + shift 768 | joints_3d = joints_3d + shift 769 | 770 | return vertices, joints_3d 771 | 772 | def tf_align_with_image_j2d(points2d, img_width, img_height): 773 | """ Perform center alignment to image coordinate system. (in Pixel space) """ 774 | if(img_width == img_height): 775 | points2d = points2d + (img_width/2) 776 | else: 777 | width_tf = tf.zeros((points2d.shape[0], points2d.shape[1], 1),dtype=tf.int32) + (img_width/2) 778 | height_tf = tf.zeros((points2d.shape[0], points2d.shape[1], 1),dtype=tf.int32) + (img_height/2) 779 | concatd = tf.concat([width_tf, height_tf], axis=2) 780 | points2d = points2d + concatd 781 | 782 | return points2d 783 | 784 | ############ Render pipeline utils ############ 785 | MESH_PROP_FACES_FL = './assets/smpl_sampling.pkl' 786 | 787 | """ Read face definition. Fixed for a SMPL model. """ 788 | with open(os.path.join(os.path.dirname(__file__), MESH_PROP_FACES_FL), 'rb') as f: 789 | sampling = pkl.load(f) 790 | 791 | M = sampling['meshes'] 792 | 793 | faces = M[0]['f'].astype(np.int32) 794 | faces = tf.convert_to_tensor(faces,dtype=tf.int32) 795 | 796 | def_bgcolor = tf.zeros(3) + [0, 0.5, 0] ## Green BG 797 | 798 | def colour_pick_img(img_batch, vertices, batch_size): 799 | """ Pick clr based on mesh registration. [Vtx, Img] -> [Vtx_clr] """ 800 | proj_verts = tf_orthographic_project(vertices) 801 | verts_pix_space = tf_align_with_image_j2d(proj_verts, cfg.IMG_W, cfg.IMG_H) 802 | 803 | #### Pick colours and resolve occlusion softly 804 | verts_pix_space = tf.cast(verts_pix_space, dtype=tf.int32) 805 | verts_pix_space = tf.concat([verts_pix_space[:,:,1:], verts_pix_space[:,:,0:1]], axis=2) 806 | 807 | if(cfg.TF_version >= 1.14): 808 | #### Alternative colour pick for TF 1.14 & above, faster inference. 809 | clr_picked = tf.gather_nd(params=occ_aware_mask, indices=verts_pix_space, batch_dims=1) ### NOTE: only for tf 1.14 and above 810 | 811 | else: 812 | ### For TF 1.13 and older 813 | for b in range(batch_size): 814 | if b == 0: 815 | clr_picked = [tf.gather_nd(params=img_batch[b], indices=verts_pix_space[b])] 816 | else: 817 | curr_clr_pick = [tf.gather_nd(params=img_batch[b], indices=verts_pix_space[b])] 818 | clr_picked = tf.concat([clr_picked, curr_clr_pick], axis=0) 819 | 820 | img_clr_picked = tf.cast(clr_picked, dtype=tf.float32) 821 | 822 | return img_clr_picked 823 | 824 | def get_occ_aware_cam_facing_mask(vertices, batch_size, part_based_occlusion_resolve=False, bgcolor=def_bgcolor): 825 | """ Occlusion-aware vtx weighting, depth based or part-based. [Vtx] -> [Vtx_occ_wtmap] """ 826 | if (part_based_occlusion_resolve): 827 | vertex_colors = np.zeros((batch_size, 6890, 3)) 828 | 829 | ### Part segmentation_generation 830 | vtx_prts = np.load("vtx_clr_smpl_proj_final_part_segmentations.npy") 831 | 832 | ### Vertex parts modify for maximal seperation 833 | vtx_prts = vtx_prts + 1 834 | vtx_prts[vtx_prts == 2] = 5 835 | vtx_prts[vtx_prts == 22] = 7 836 | vtx_prts[vtx_prts == 8] = 22 837 | vtx_prts[vtx_prts == 12] = 2 838 | vtx_prts[vtx_prts == 23] = 13 839 | vtx_prts[vtx_prts == 19] = 4 840 | vtx_prts[vtx_prts == 21] = 18 841 | 842 | #### part labelled 843 | vtx_part_labels = np.zeros(vertices.shape) 844 | vtx_prts = np.expand_dims(vtx_prts, axis=1) 845 | vtx_prts = vtx_prts / 24.0 846 | part_label = np.concatenate([vtx_prts, vtx_prts, vtx_prts], axis=1) 847 | vtx_part_labels[:] = part_label ##broadcast to form batch 848 | 849 | #### Render cam setup 850 | fixed_rt = np.array([1.0, 0.0, 0.0]) ### tilt,pan,roll 851 | angle = np.linalg.norm(fixed_rt) 852 | axis = fixed_rt / angle 853 | ang = np.pi 854 | new_an_ax = axis * (ang) 855 | fixed_rt = new_an_ax 856 | fixed_t = [0., 0., 0.] 857 | ## 858 | 859 | fixed_renderer = RenderLayer(cfg.IMG_W, cfg.IMG_H, 3, bgcolor=bgcolor, f=faces, camera_f=[cfg.IMG_W, cfg.IMG_H], camera_c=[cfg.IMG_W/2.0, cfg.IMG_H/2.0], camera_rt=fixed_rt, camera_t=fixed_t) 860 | 861 | vert_norms = dirt_expose.get_vertex_normals(vertices, faces) 862 | 863 | #### Verts selection based on norm 864 | vert_norms_flat = tf.reshape(vert_norms, [-1, 3]) 865 | fake_angle = tf.ones_like(vert_norms_flat[:,0:1], dtype=tf.float32) ## unit mag 866 | euler_angles = tfg.geometry.transformation.euler.from_axis_angle(axis=vert_norms_flat, angle=fake_angle) 867 | vert_norms_euler = tf.reshape(euler_angles, [-1, 6890, 3]) 868 | 869 | ### Diff. margin formulation 870 | quant_sharpness_factor = 50 871 | verts_ndiff = vert_norms_euler[:,:,2:] * -1 ## invert as cam faces 872 | verts_ndiff = verts_ndiff * quant_sharpness_factor ## centrifugal from 0.0 to get quantization effect 873 | 874 | #verts_ndiff = tf.math.sign(verts_ndiff) 875 | #verts_ndiff = tf.nn.relu(verts_ndiff) 876 | verts_ndiff = tf.nn.sigmoid(verts_ndiff) 877 | 878 | if(part_based_occlusion_resolve): 879 | vtx_part_labels= tf.convert_to_tensor(vtx_part_labels, dtype=tf.float32) 880 | 881 | ## Normal part based resolving occlusion based render 882 | cam_facing_vtx_clrs = tf.multiply(vtx_part_labels, verts_ndiff) 883 | else: 884 | ## Depth based occlusion aware picking to be debugged 885 | depth_vertices = vertices[:,:,2:] 886 | 887 | ## Normalize the depth between 0 and 1 888 | min_val = tf.reduce_min(depth_vertices, axis=1, keepdims=True) 889 | normalized_depth_vertices = tf.div( tf.subtract(depth_vertices, min_val), tf.subtract(tf.reduce_max(depth_vertices, axis=1, keepdims=True), min_val)) 890 | 891 | cam_facing_vtx_clrs = tf.tile(normalized_depth_vertices, [1,1,3]) 892 | cam_facing_vtx_clrs = tf.multiply(cam_facing_vtx_clrs, verts_ndiff) 893 | 894 | ## Mask render for occlusion resolution 895 | occ_aware_mask = fixed_renderer.call(vertices, vc=cam_facing_vtx_clrs) ## occulsion aware z-buffered parts masks 896 | 897 | clr_picked = colour_pick_img(occ_aware_mask, vertices, batch_size) 898 | 899 | ## Occlusion resolution based on z-buffered parts 900 | if(part_based_occlusion_resolve): 901 | occ_sel_diff = (vtx_part_labels[:,:,0:1] - clr_picked[:,:,0:1] ) * 10.0 902 | 903 | else: 904 | ### Depth based colour pick 905 | occ_sel_diff = (normalized_depth_vertices[:,:,0:1] - clr_picked[:,:,0:1] ) * 10.0 906 | 907 | ### Diff. margin soft selection 908 | occ_sel = tf.nn.sigmoid(occ_sel_diff) * tf.nn.sigmoid(-1 * occ_sel_diff) * 4.0 909 | 910 | #### Select front facing 911 | final_front_facing_occ_resolved = tf.multiply(occ_sel, verts_ndiff) 912 | 913 | return final_front_facing_occ_resolved 914 | 915 | def apply_ref_symmetry(vclr_picked_resolved, front_facing_occ_resolved_mask, batch_size): 916 | """ Reflectional symmetry module. [Vtx_clr, Vtx_wtmap] -> [Vtx_clr_symm] """ 917 | symm_arr = np.load("./assets/basic_vtx_clr_symm_map.npy") 918 | symm_arr_transpose = np.transpose(symm_arr) 919 | 920 | sym_map = tf.expand_dims(symm_arr, axis=0) 921 | sym_map = tf.tile(sym_map, [batch_size,1,1]) 922 | 923 | sym_map_transpose = tf.expand_dims(symm_arr_transpose, axis=0) 924 | sym_map_transpose = tf.tile(sym_map_transpose, [batch_size, 1, 1]) 925 | 926 | ## Group clr value calc 927 | num = tf.matmul(sym_map, vclr_picked_resolved) 928 | den = tf.matmul(sym_map, front_facing_occ_resolved_mask) 929 | 930 | den = den + 0.00001 931 | calc_val = tf.truediv(num, den) 932 | ### Value assign using symmtery 933 | vclr_symm = tf.matmul(sym_map_transpose, calc_val) 934 | 935 | return vclr_symm 936 | --------------------------------------------------------------------------------