├── README.md ├── docs ├── rich_in_worldframe.png └── rich_visualization.gif ├── hsc_params.py ├── multicam2world.py ├── requirements.txt ├── resource ├── gender.json ├── imgext.json └── meta │ ├── test.txt │ ├── train.txt │ └── val.txt ├── samples └── tmp.jpg ├── smplx2images.py └── utils.py /README.md: -------------------------------------------------------------------------------- 1 | # RICH: Real scenes, Interaction, Contacts and Humans 2 | 3 | 4 | This is the toolkit for RICH dataset in [Capturing and Inferring Dense Full-BodyHuman-Scene Contact](https://rich.is.tue.mpg.de/index.html). It consists of a few light weight scripts demonstrating how to use the released files, e.g., visualizing body keypoints on images, parsing meta data etc. 5 | 6 | The body-scene contact network (BSTRO) is released in another [repo.](https://github.com/paulchhuang/bstro) 7 | 8 | ## Install Dependencies 9 | ``` 10 | python3 -m venv PATH/2/VENV 11 | source PATH/2/VENV/bin/activate 12 | pip install -r requirements.txt 13 | ``` 14 | 15 | ## Download necessary files 16 | Please download the [RICH dataset](https://rich.is.tue.mpg.de/) and [SMPL-X model](https://smpl-x.is.tue.mpg.de/) from the official websites and organize them following the structure below: 17 | ``` 18 | ${REPO_DIR} 19 | |-- body_models 20 | | |-- smplx 21 | | | |-- SMPLX_FEMALE.pkl 22 | | | |-- SMPLX_FEMALE.npz 23 | | | |-- SMPLX_MALE.pkl 24 | | | |-- SMPLX_MALE.npz 25 | | | |-- SMPLX_NEUTRAL.pkl 26 | | | |-- SMPLX_NEUTRAL.npz 27 | | | |-- ... 28 | |-- data 29 | | |-- bodies 30 | | | |-- train 31 | | | | |--BBQ_001_guitar 32 | | | | |--BBQ_001_juggle 33 | | | | |--... 34 | | | |-- val 35 | | | |-- test 36 | | |-- human_scene_contact 37 | | | |-- train 38 | | | | |--BBQ_001_guitar 39 | | | | |--BBQ_001_juggle 40 | | | | |--... 41 | | | |-- val 42 | | | |-- test 43 | | |-- images 44 | | | |-- train 45 | | | | |--BBQ_001_guitar 46 | | | | |--BBQ_001_juggle 47 | | | | |--... 48 | | | |-- val 49 | | | |-- test 50 | | |-- multicam2world 51 | | | |-- BBQ_multicam2world.json 52 | | | |-- Gym_multicam2world.json 53 | | | |-- ... 54 | | | |-- ... 55 | | |-- scan_calibration 56 | | | |-- BBQ 57 | | | |-- Gym 58 | | | |-- ... 59 | | | |-- ... 60 | ``` 61 | 62 | ## Examples 63 | 1. Get 3D joints from SMPL-X params and project them onto an image: 64 | ``` 65 | python smplx2images.py 66 | ``` 67 | and check the results in `samples` folder. 68 | 69 | 2. Load human-scene contact annotations: 70 | ``` 71 | python hsc_params.py 72 | ``` 73 | and check the variables `hsc_vert_id_smpl` and `hsc_vert_id_smplx`. 74 | 75 | 3. Visualize scans and SMPL-X bodies in world frames: 76 | The released SMPL-X params and the scene scan reside in the calibrated multi-camera coordinate, where the first camera is conventionally chosen as the reference (R=I, t=0) so the ground plane is often not axis-aligned. When an axis-aligned ground plane is required, one can consider transforming bodies and the scene mesh to the world frame defined during the scanning process: 77 | ``` 78 | python multicam2world.py 79 | ``` 80 | Visualizing the generated `body_scene_world.ply` in `samples` folder with meshlab, one shall see: 81 | 82 | 83 | ## Citations 84 | If you find our work useful in your research, please consider citing: 85 | 86 | ```bibtex 87 | @inproceedings{huang2022rich, 88 | title = {Capturing and Inferring Dense Full-Body Human-Scene Contact}, 89 | author = {Huang, Chun-Hao P. and Yi, Hongwei and H{\"o}schle, Markus and Safroshkin, Matvey and Alexiadis, Tsvetelina and Polikovsky, Senya and Scharstein, Daniel and Black, Michael J.}, 90 | booktitle = {IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR) }, 91 | pages = {13274-13285}, 92 | month = jun, 93 | year = {2022}, 94 | month_numeric = {6} 95 | } 96 | ``` 97 | -------------------------------------------------------------------------------- /docs/rich_in_worldframe.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/paulchhuang/rich_toolkit/3b6cf83b0f09e2551f5a20cb180073dea6132cfc/docs/rich_in_worldframe.png -------------------------------------------------------------------------------- /docs/rich_visualization.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/paulchhuang/rich_toolkit/3b6cf83b0f09e2551f5a20cb180073dea6132cfc/docs/rich_visualization.gif -------------------------------------------------------------------------------- /hsc_params.py: -------------------------------------------------------------------------------- 1 | import pickle 2 | import numpy as np 3 | 4 | ## input arguments 5 | SET = 'train' 6 | SEQ_NAME = 'ParkingLot1_005_pushup2' 7 | SCENE_NAME, SUB_ID, _ = SEQ_NAME.split('_') 8 | FRAME_ID = 150 9 | 10 | hsc_params = pickle.load(open(f'data/human_scene_contact/{SET}/{SEQ_NAME}/{FRAME_ID:05d}/{SUB_ID}.pkl', 'rb')) 11 | 12 | contact_labels = hsc_params['contact'] # (6890,): per-vertex 0/1 contact label in smpl format 13 | hsc_vert_id_smpl = np.where(contact_labels > 0.)[0] 14 | distplacement_vec_smplx = hsc_params['s2m_dist_id'] # (10475,3): the vector that points from each vertex in smplx to the closest point on the scene scan 15 | closest_faces_on_scene_smplx = hsc_params['closest_triangles_id'] # (10475,3,3): the triangle which stores the point above 16 | 17 | import trimesh 18 | smplx_mesh = trimesh.load(f'data/human_scene_contact/{SET}/{SEQ_NAME}/{FRAME_ID:05d}/{SUB_ID}.obj', process=False) 19 | hsc_vert_id_smplx = np.where((smplx_mesh.visual.vertex_colors == (0, 255, 0, 255)).all(axis=1))[0] 20 | -------------------------------------------------------------------------------- /multicam2world.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | import torch 4 | import pickle 5 | import trimesh 6 | from smplx import SMPLX 7 | import numpy as np 8 | 9 | ## input arguments 10 | SET = 'train' 11 | SEQ_NAME = 'ParkingLot1_005_pushup2' 12 | SCENE_NAME, SUB_ID, _ = SEQ_NAME.split('_') 13 | FRAME_ID = 150 14 | CAMERA_ID = 0 15 | gender_mapping = json.load(open('resource/gender.json','r')) 16 | GENDER = gender_mapping[f'{int(SUB_ID)}'] 17 | imgext = json.load(open('resource/imgext.json','r')) 18 | EXT = imgext[SCENE_NAME] 19 | 20 | ## SMPLX model 21 | SMPLX_MODEL_DIR = 'body_models/smplx' 22 | body_model = SMPLX( 23 | SMPLX_MODEL_DIR, 24 | gender=GENDER, 25 | num_pca_comps=12, 26 | flat_hand_mean=False, 27 | create_expression=True, 28 | create_jaw_pose=True, 29 | ) 30 | 31 | ## passing the parameters through SMPL-X 32 | smplx_params_fn = os.path.join('data/bodies',SET, SEQ_NAME, f'{FRAME_ID:05d}', f'{SUB_ID}.pkl') 33 | body_params = pickle.load(open(smplx_params_fn,'rb')) 34 | body_params = {k: torch.from_numpy(v) for k, v in body_params.items()} 35 | body_model.reset_params(**body_params) 36 | model_output = body_model(return_verts=True, 37 | body_pose=body_params['body_pose'], 38 | return_full_pose=True) 39 | mesh = trimesh.Trimesh(vertices = model_output.vertices.detach().cpu().squeeze().numpy(), faces = body_model.faces, process=False) 40 | 41 | with open(f'data/multicam2world/{SCENE_NAME}_multicam2world.json', 'r') as f: 42 | cam2scan = json.load(f) 43 | rot_mat = np.array(cam2scan['R']) 44 | translation = cam2scan['t'] 45 | 46 | 47 | ## scan 48 | scene_scan = trimesh.load(f'data/scan_calibration/{SCENE_NAME}/scan_camcoord.ply', process=False) 49 | scan = scene_scan + mesh 50 | print(scan.vertices.shape) 51 | scan.vertices = cam2scan['c'] * scan.vertices @ rot_mat + translation 52 | scan.export(f'samples/body_scene_world.ply') 53 | 54 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | opencv-python 2 | tqdm 3 | numpy 4 | trimesh 5 | pytorch==1.4.0 6 | torchvision==0.5.0 7 | smplx -------------------------------------------------------------------------------- /resource/gender.json: -------------------------------------------------------------------------------- 1 | {"4": "male", "14": "male", "5": "male", "6": "male", "8": "male", "18": "female", "20": "male", "0": "male", "1": "male", "7": "male", "2": "male", "13": "female", "3": "male", "15": "male", "16": "female", "11": "male", "17": "female", "19": "female", "21": "female", "9": "female", "12": "female", "10": "female"} -------------------------------------------------------------------------------- /resource/imgext.json: -------------------------------------------------------------------------------- 1 | {"Gym": "bmp", "BBQ": "bmp", "ParkingLot2": "bmp", "Pavallion": "png", "ParkingLot1": "bmp", "LectureHall": "png"} -------------------------------------------------------------------------------- /resource/meta/test.txt: -------------------------------------------------------------------------------- 1 | sequence_name capture_name scan_name id moving_cam gender scene action/scene-interaction subjects view_id 2 | ParkingLot2_017_burpeejump2 ParkingLot2 scan_camcoord 017 V female V V X 0,2,3 3 | ParkingLot2_017_burpeejump1 ParkingLot2 scan_camcoord 017 V female V V X 0,1,5 4 | ParkingLot2_017_overfence1 ParkingLot2 scan_camcoord 017 V female V V X 0,3,4 5 | ParkingLot2_017_overfence2 ParkingLot2 scan_camcoord 017 V female V V X 0,1,4 6 | ParkingLot2_017_eating1 ParkingLot2 scan_camcoord 017 V female V V X 0,2,4 7 | ParkingLot2_017_pushup1 ParkingLot2 scan_camcoord 017 X female V V X 0,1,4,5 8 | ParkingLot2_017_pushup2 ParkingLot2 scan_camcoord 017 V female V V X 0,4,5 9 | ParkingLot2_009_burpeejump1 ParkingLot2 scan_camcoord 009 X female V V X 0,1,2,3 10 | ParkingLot2_009_burpeejump2 ParkingLot2 scan_camcoord 009 X female V V X 0,2,3,4 11 | ParkingLot2_009_overfence1 ParkingLot2 scan_camcoord 009 X female V V X 0,3,4,5 12 | ParkingLot2_009_overfence2 ParkingLot2 scan_camcoord 009 X female V V X 0,1,4,5 13 | LectureHall_009_sidebalancerun1 LectureHall scan_yoga_scene_camcoord 009 X female V V X 0,1,4,5 14 | LectureHall_010_plankjack1 LectureHall scan_yoga_scene_camcoord 010 X female V V X 0,2,4,6 15 | LectureHall_010_sidebalancerun1 LectureHall scan_yoga_scene_camcoord 010 X female V V X 0,1,2,4 16 | LectureHall_021_plankjack1 LectureHall scan_yoga_scene_camcoord 021 X female V V X 0,3,5,6 17 | LectureHall_021_sidebalancerun1 LectureHall scan_yoga_scene_camcoord 021 X female V V X 0,4,5,6 18 | LectureHall_019_wipingchairs1 LectureHall scan_chair_scene_camcoord 019 X female V V X 0,1,2,3 19 | LectureHall_009_021_reparingprojector1 LectureHall scan_yoga_scene_camcoord 009 X female V X X 0,3,4,5 20 | LectureHall_009_021_reparingprojector1 LectureHall scan_yoga_scene_camcoord 021 X female V X X 0,3,4,5 21 | ParkingLot2_009_spray1 ParkingLot2 scan_camcoord 009 X female V X X 0,1,2,3 22 | ParkingLot2_009_impro1 ParkingLot2 scan_camcoord 009 X female V X X 0,2,3,4 23 | ParkingLot2_009_impro2 ParkingLot2 scan_camcoord 009 X female V X X 0,3,4,5 24 | ParkingLot2_009_impro5 ParkingLot2 scan_camcoord 009 X female V X X 0,2,4,5 25 | Gym_010_pushup1 Gym scan_camcoord 010 X female X V X 3,4,5,6 26 | Gym_010_pushup2 Gym scan_camcoord 010 X female X V X 2,3,4,5 27 | Gym_011_pushup1 Gym scan_camcoord 011 X male X V X 2,3,4,5 28 | Gym_011_pushup2 Gym scan_camcoord 011 X male X V X 2,3,4,5 29 | Gym_011_burpee2 Gym scan_camcoord 011 X male X V X 2,3,4,5 30 | Gym_012_pushup2 Gym scan_camcoord 012 X female X V X 3,4,5,6 31 | Gym_010_mountainclimber1 Gym scan_camcoord 010 X female X V X 3,4,5,6 32 | Gym_010_mountainclimber2 Gym scan_camcoord 010 X female X V X 3,4,5,6 33 | Gym_013_dips1 Gym scan_camcoord 013 X female X X V 0,3,4,5 34 | Gym_013_dips2 Gym scan_camcoord 013 X female X X V 1,2,4,5 35 | Gym_013_dips3 Gym scan_camcoord 013 X female X X V 1,2,4,5 36 | Gym_013_lunge1 Gym scan_camcoord 013 X female X X V 1,4,5,6 37 | Gym_013_lunge2 Gym scan_camcoord 013 X female X X V 0,4,5,6 38 | Gym_013_pushup1 Gym scan_camcoord 013 X female X V V 0,3,4,5 39 | Gym_013_pushup2 Gym scan_camcoord 013 X female X V V 1,2,4,5 40 | Gym_013_burpee4 Gym scan_camcoord 013 X female X V V 0,4,5,6 41 | Gym_010_lunge1 Gym scan_camcoord 010 X female X X X 1,4,5,6 42 | Gym_010_lunge2 Gym scan_camcoord 010 X female X X X 0,2,4,5 43 | Gym_010_dips1 Gym scan_camcoord 010 X female X X X 0,4,5,6 44 | Gym_010_dips2 Gym scan_camcoord 010 X female X X X 1,2,4,5 45 | Gym_010_cooking1 Gym scan_table_camcoord 010 X female X X X 1,3,4,5 46 | Gym_011_cooking1 Gym scan_table_camcoord 011 V male X X X 4,5,6 47 | Gym_011_cooking2 Gym scan_table_camcoord 011 V male X X X 2,4,5 48 | Gym_011_dips1 Gym scan_camcoord 011 X male X X X 1,3,4,5 49 | Gym_011_dips4 Gym scan_camcoord 011 X male X X X 0,2,4,5 50 | Gym_011_dips3 Gym scan_camcoord 011 X male X X X 0,3,4,5 51 | Gym_011_dips2 Gym scan_camcoord 011 X male X X X 1,3,4,5 52 | Gym_012_lunge1 Gym scan_camcoord 012 X female X X X 0,3,4,5 53 | Gym_012_lunge2 Gym scan_camcoord 012 X female X X X 0,4,5,6 54 | Gym_012_cooking2 Gym scan_table_camcoord 012 V female X X X 3,4,5 -------------------------------------------------------------------------------- /resource/meta/train.txt: -------------------------------------------------------------------------------- 1 | sequence_name capture_name scan_name id moving_cam gender view_id 2 | ParkingLot1_002_burpee3 ParkingLot1 scan_camcoord 002 X male 0,1,2,3,4,5,6,7 3 | ParkingLot1_002_overfence1 ParkingLot1 scan_camcoord 002 X male 0,1,2,3,4,5,6,7 4 | ParkingLot1_002_overfence2 ParkingLot1 scan_camcoord 002 X male 0,1,2,3,4,5,6,7 5 | ParkingLot1_002_stretching1 ParkingLot1 scan_camcoord 002 X male 0,1,2,3,4,5,6,7 6 | ParkingLot1_002_pushup1 ParkingLot1 scan_camcoord 002 X male 0,1,2,3,4,5,6,7 7 | ParkingLot1_004_pushup2 ParkingLot1 scan_camcoord 004 X male 0,1,2,3,4,5,6,7 8 | ParkingLot1_004_burpeejump1 ParkingLot1 scan_camcoord 004 X male 0,1,2,3,4,5,6,7 9 | ParkingLot1_004_eating1 ParkingLot1 scan_camcoord 004 X male 0,1,2,3,4,5,6,7 10 | ParkingLot1_004_takingphotos1 ParkingLot1 scan_camcoord 004 X male 0,1,2,3,4,5,6,7 11 | ParkingLot1_004_phonetalk1 ParkingLot1 scan_camcoord 004 X male 0,1,2,3,4,5,6,7 12 | ParkingLot1_005_burpeejump2 ParkingLot1 scan_camcoord 005 X male 0,1,2,3,4,5,6,7 13 | ParkingLot1_005_overfence1 ParkingLot1 scan_camcoord 005 X male 0,1,2,3,4,5,6,7 14 | ParkingLot1_005_pushup2 ParkingLot1 scan_camcoord 005 X male 0,1,2,3,4,5,6,7 15 | ParkingLot1_005_pushup3 ParkingLot1 scan_camcoord 005 X male 0,1,2,3,4,5,6,7 16 | ParkingLot1_004_005_greetingchattingeating1 ParkingLot1 scan_camcoord 004 X male 0,1,2,3,4,5,6,7 17 | ParkingLot1_004_005_greetingchattingeating1 ParkingLot1 scan_camcoord 005 X male 0,1,2,3,4,5,6,7 18 | ParkingLot1_007_overfence2 ParkingLot1 scan_camcoord 007 X male 0,1,2,3,4,5,6,7 19 | ParkingLot1_007_eating1 ParkingLot1 scan_camcoord 007 X male 0,1,2,3,4,5,6,7 20 | ParkingLot1_007_eating2 ParkingLot1 scan_camcoord 007 X male 0,1,2,3,4,5,6,7 21 | ParkingLot2_008_phonetalk1 ParkingLot2 scan_camcoord 008 V male 0,1,2,3,4,5 22 | ParkingLot2_008_burpeejump1 ParkingLot2 scan_camcoord 008 V male 0,1,2,3,4,5 23 | ParkingLot2_008_overfence1 ParkingLot2 scan_camcoord 008 V male 0,1,2,3,4,5 24 | ParkingLot2_008_pushup1 ParkingLot2 scan_camcoord 008 V male 0,1,2,3,4,5 25 | ParkingLot2_008_pushup2 ParkingLot2 scan_camcoord 008 V male 0,1,2,3,4,5 26 | ParkingLot2_008_overfence2 ParkingLot2 scan_camcoord 008 V male 0,1,2,3,4,5 27 | ParkingLot2_008_overfence3 ParkingLot2 scan_camcoord 008 V male 0,1,2,3,4,5 28 | ParkingLot2_008_eating1 ParkingLot2 scan_camcoord 008 V male 0,1,2,3,4,5 29 | ParkingLot2_014_pushup2 ParkingLot2 scan_camcoord 014 X male 0,1,2,3,4,5 30 | ParkingLot2_014_burpeejump1 ParkingLot2 scan_camcoord 014 X male 0,1,2,3,4,5 31 | ParkingLot2_014_burpeejump2 ParkingLot2 scan_camcoord 014 X male 0,1,2,3,4,5 32 | ParkingLot2_014_phonetalk2 ParkingLot2 scan_camcoord 014 X male 0,1,2,3,4,5 33 | ParkingLot2_014_takingphotos2 ParkingLot2 scan_camcoord 014 X male 0,1,2,3,4,5 34 | ParkingLot2_014_overfence3 ParkingLot2 scan_camcoord 014 X male 0,1,2,3,4,5 35 | ParkingLot2_015_overfence1 ParkingLot2 scan_camcoord 015 X male 0,1,2,3,4,5 36 | ParkingLot2_015_burpeejump2 ParkingLot2 scan_camcoord 015 X male 0,1,2,3,4,5 37 | ParkingLot2_015_pushup1 ParkingLot2 scan_camcoord 015 X male 0,1,2,3,4,5 38 | ParkingLot2_015_eating2 ParkingLot2 scan_camcoord 015 X male 0,1,2,3,4,5 39 | ParkingLot2_016_burpeejump2 ParkingLot2 scan_camcoord 016 V female 0,1,2,3,4,5 40 | ParkingLot2_016_overfence2 ParkingLot2 scan_camcoord 016 V female 0,1,2,3,4,5 41 | ParkingLot2_016_pushup1 ParkingLot2 scan_camcoord 016 V female 0,1,2,3,4,5 42 | ParkingLot2_016_pushup2 ParkingLot2 scan_camcoord 016 V female 0,1,2,3,4,5 43 | ParkingLot2_016_stretching1 ParkingLot2 scan_camcoord 016 V female 0,1,2,3,4,5 44 | Pavallion_000_yoga2 Pavallion scan_camcoord 000 X male 0,1,2,3,4,5,6 45 | Pavallion_000_plankjack Pavallion scan_camcoord 000 X male 0,1,2,3,4,5,6 46 | Pavallion_000_phonesiteat Pavallion scan_camcoord 000 X male 0,1,3,4,6 47 | Pavallion_000_sidebalancerun Pavallion scan_camcoord 000 X male 0,1,2,3,4,5,6 48 | Pavallion_002_plankjack Pavallion scan_camcoord 002 V male 0,1,2,3,4,5,6 49 | Pavallion_002_phonesiteat Pavallion scan_camcoord 002 V male 0,1,3,4,6 50 | Pavallion_003_plankjack Pavallion scan_camcoord 003 V male 0,1,2,3,4,5,6 51 | Pavallion_003_phonesiteat Pavallion scan_camcoord 003 V male 0,1,3,4,6 52 | Pavallion_003_sidebalancerun Pavallion scan_camcoord 003 V male 0,1,2,3,4,5,6 53 | Pavallion_006_phonesiteat Pavallion scan_camcoord 006 V male 0,1,3,4,6 54 | Pavallion_006_sidebalancerun Pavallion scan_camcoord 006 V male 0,1,2,3,4,5,6 55 | Pavallion_006_plankjack Pavallion scan_camcoord 006 V male 0,1,2,3,4,5,6 56 | Pavallion_013_phonesiteat Pavallion scan_camcoord 013 X female 0,1,3,4,6 57 | Pavallion_013_plankjack Pavallion scan_camcoord 013 X female 0,1,2,3,4,5,6 58 | Pavallion_013_yoga2 Pavallion scan_camcoord 013 V female 0,1,2,3,4,5,6 59 | Pavallion_003_018_tossball Pavallion scan_camcoord 003 X male 0,1,2,3,4,5,6 60 | Pavallion_003_018_tossball Pavallion scan_camcoord 018 X female 0,1,2,3,4,5,6 61 | LectureHall_018_wipingchairs1 LectureHall scan_chair_scene_camcoord 018 X female 0,1,2,3,4,5,6 62 | LectureHall_018_wipingspray1 LectureHall scan_chair_scene_camcoord 018 X female 2,3,4 63 | LectureHall_020_wipingtable1 LectureHall scan_chair_scene_camcoord 020 X male 0,2,4,5,6 64 | BBQ_001_juggle BBQ scan_camcoord 001 X male 0,1,2,3,4,5,6,7 65 | BBQ_001_guitar BBQ scan_camcoord 001 X male 0,1,2,3,4,5,6,7 -------------------------------------------------------------------------------- /resource/meta/val.txt: -------------------------------------------------------------------------------- 1 | sequence_name capture_name scan_name id moving_cam gender scene action/scene-interaction subjects view_id 2 | ParkingLot1_002_stretching2 ParkingLot1 scan_camcoord 002 X male V V V 0,1,2,3,4,5,6,7 3 | ParkingLot1_002_burpee1 ParkingLot1 scan_camcoord 002 X male V V V 0,1,2,3,4,5,6,7 4 | ParkingLot1_002_burpee2 ParkingLot1 scan_camcoord 002 X male V V V 0,1,2,3,4,5,6,7 5 | ParkingLot1_004_pushup1 ParkingLot1 scan_camcoord 004 X male V V V 0,1,2,3,4,5,6,7 6 | ParkingLot1_004_eating2 ParkingLot1 scan_camcoord 004 X male V V V 0,1,2,3,4,5,6,7 7 | ParkingLot1_004_phonetalk2 ParkingLot1 scan_camcoord 004 X male V V V 0,1,2,3,4,5,6,7 8 | ParkingLot1_004_takingphotos2 ParkingLot1 scan_camcoord 004 X male V V V 0,1,2,3,4,5,6,7 9 | ParkingLot1_004_stretching2 ParkingLot1 scan_camcoord 004 X male V V V 0,1,2,3,4,5,6,7 10 | ParkingLot1_005_overfence2 ParkingLot1 scan_camcoord 005 X male V V V 0,1,2,3,4,5,6,7 11 | ParkingLot1_005_pushup1 ParkingLot1 scan_camcoord 005 X male V V V 0,1,2,3,4,5,6,7 12 | ParkingLot1_005_burpeejump1 ParkingLot1 scan_camcoord 005 X male V V V 0,1,2,3,4,5,6,7 13 | ParkingLot1_007_burpee2 ParkingLot1 scan_camcoord 007 X male V V V 0,1,2,3,4,5,6,7 14 | ParkingLot2_008_eating2 ParkingLot2 scan_camcoord 008 V male V V V 0,1,2,3,4,5 15 | ParkingLot2_008_burpeejump2 ParkingLot2 scan_camcoord 008 V male V V V 0,1,2,3,4,5 16 | ParkingLot2_014_overfence1 ParkingLot2 scan_camcoord 014 X male V V V 0,1,2,3,4,5 17 | ParkingLot2_014_eating2 ParkingLot2 scan_camcoord 014 X male V V V 0,1,2,3,4,5 18 | ParkingLot2_016_phonetalk5 ParkingLot2 scan_camcoord 016 V female V V V 0,1,2,3,4,5 19 | Pavallion_002_sidebalancerun Pavallion scan_camcoord 002 V male V V V 0,1,2,3,4,5,6 20 | Pavallion_013_sidebalancerun Pavallion scan_camcoord 013 X female V V V 0,1,2,3,4,5,6 21 | Pavallion_018_sidebalancerun Pavallion scan_camcoord 018 V female V V V 0,1,2,3,4,5,6 22 | LectureHall_018_wipingtable1 LectureHall scan_chair_scene_camcoord 018 X female V V V 0,2,4,5,6 23 | LectureHall_020_wipingchairs1 LectureHall scan_chair_scene_camcoord 020 X male V V V 0,1,2,3,4,5,6 24 | LectureHall_003_wipingchairs1 LectureHall scan_chair_scene_camcoord 003 X male V V V 0,1,2,3,4,5,6 25 | Pavallion_000_yoga1 Pavallion scan_camcoord 000 X male V X V 0,1,2,3,4,5,6 26 | Pavallion_002_yoga1 Pavallion scan_camcoord 002 V male V X V 0,1,2,3,4,5,6 27 | Pavallion_003_yoga1 Pavallion scan_camcoord 003 V male V X V 0,1,2,3,4,5,6 28 | Pavallion_006_yoga1 Pavallion scan_camcoord 006 V male V X V 0,1,2,3,4,5,6 29 | Pavallion_018_yoga1 Pavallion scan_camcoord 018 V female V X V 0,1,2,3,4,5,6 -------------------------------------------------------------------------------- /samples/tmp.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/paulchhuang/rich_toolkit/3b6cf83b0f09e2551f5a20cb180073dea6132cfc/samples/tmp.jpg -------------------------------------------------------------------------------- /smplx2images.py: -------------------------------------------------------------------------------- 1 | import os 2 | import cv2 3 | import json 4 | import pickle 5 | import torch 6 | import trimesh 7 | from smplx import SMPLX 8 | from utils import CalibratedCamera 9 | 10 | ## input arguments 11 | SET = 'train' 12 | SEQ_NAME = 'ParkingLot1_005_pushup2' 13 | SCENE_NAME, SUB_ID, _ = SEQ_NAME.split('_') 14 | FRAME_ID = 150 15 | CAMERA_ID = 0 16 | gender_mapping = json.load(open('resource/gender.json','r')) 17 | GENDER = gender_mapping[f'{int(SUB_ID)}'] 18 | imgext = json.load(open('resource/imgext.json','r')) 19 | EXT = imgext[SCENE_NAME] 20 | 21 | ## SMPLX model 22 | SMPLX_MODEL_DIR = 'body_models/smplx' 23 | body_model = SMPLX( 24 | SMPLX_MODEL_DIR, 25 | gender=GENDER, 26 | num_pca_comps=12, 27 | flat_hand_mean=False, 28 | create_expression=True, 29 | create_jaw_pose=True, 30 | ) 31 | 32 | ## passing the parameters through SMPL-X 33 | smplx_params_fn = os.path.join('data/bodies',SET, SEQ_NAME, f'{FRAME_ID:05d}', f'{SUB_ID}.pkl') 34 | body_params = pickle.load(open(smplx_params_fn,'rb')) 35 | body_params = {k: torch.from_numpy(v) for k, v in body_params.items()} 36 | body_model.reset_params(**body_params) 37 | model_output = body_model(return_verts=True, 38 | body_pose=body_params['body_pose'], 39 | return_full_pose=True) 40 | mesh = trimesh.Trimesh(vertices = model_output.vertices.detach().cpu().squeeze().numpy(), faces = body_model.faces, process=False) 41 | mesh.export('samples/tmp.obj') 42 | 43 | ## project to image 44 | calib_path = os.path.join('data/scan_calibration', SCENE_NAME, 'calibration', f'{CAMERA_ID:03d}.xml') 45 | cam = CalibratedCamera(calib_path=calib_path) 46 | j_2D = cam(model_output.joints).squeeze().detach().numpy() 47 | img_fn = os.path.join('data/images', SET, SEQ_NAME, f'cam_{CAMERA_ID:02d}', f'{FRAME_ID:05d}_{CAMERA_ID:02d}.{EXT}') 48 | img = cv2.imread(img_fn) 49 | for j in j_2D[:25]: 50 | cv2.circle(img, (int(j[0]), int(j[1])), 6, (255, 0, 255), thickness=-1) 51 | cv2.imwrite('samples/tmp.jpg',img) -------------------------------------------------------------------------------- /utils.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | from smplx.lbs import transform_mat 4 | 5 | def extract_cam_param_xml(xml_path='', dtype=torch.float32): 6 | 7 | import xml.etree.ElementTree as ET 8 | tree = ET.parse(xml_path) 9 | 10 | extrinsics_mat = [float(s) for s in tree.find('./CameraMatrix/data').text.split()] 11 | intrinsics_mat = [float(s) for s in tree.find('./Intrinsics/data').text.split()] 12 | distortion_vec = [float(s) for s in tree.find('./Distortion/data').text.split()] 13 | 14 | focal_length_x = intrinsics_mat[0] 15 | focal_length_y = intrinsics_mat[4] 16 | center = torch.tensor([[intrinsics_mat[2], intrinsics_mat[5]]], dtype=dtype) 17 | 18 | rotation = torch.tensor([[extrinsics_mat[0], extrinsics_mat[1], extrinsics_mat[2]], 19 | [extrinsics_mat[4], extrinsics_mat[5], extrinsics_mat[6]], 20 | [extrinsics_mat[8], extrinsics_mat[9], extrinsics_mat[10]]], dtype=dtype) 21 | 22 | translation = torch.tensor([[extrinsics_mat[3], extrinsics_mat[7], extrinsics_mat[11]]], dtype=dtype) 23 | 24 | # t = -Rc --> c = -R^Tt 25 | cam_center = [ -extrinsics_mat[0]*extrinsics_mat[3] - extrinsics_mat[4]*extrinsics_mat[7] - extrinsics_mat[8]*extrinsics_mat[11], 26 | -extrinsics_mat[1]*extrinsics_mat[3] - extrinsics_mat[5]*extrinsics_mat[7] - extrinsics_mat[9]*extrinsics_mat[11], 27 | -extrinsics_mat[2]*extrinsics_mat[3] - extrinsics_mat[6]*extrinsics_mat[7] - extrinsics_mat[10]*extrinsics_mat[11]] 28 | 29 | cam_center = torch.tensor([cam_center], dtype=dtype) 30 | 31 | k1 = torch.tensor([distortion_vec[0]], dtype=dtype) 32 | k2 = torch.tensor([distortion_vec[1]], dtype=dtype) 33 | 34 | return focal_length_x, focal_length_y, center, rotation, translation, cam_center, k1, k2 35 | 36 | class CalibratedCamera(nn.Module): 37 | 38 | def __init__(self, calib_path='', rotation=None, translation=None, 39 | focal_length_x=None, focal_length_y=None, 40 | batch_size=1, 41 | center=None, dtype=torch.float32, **kwargs): 42 | super(CalibratedCamera, self).__init__() 43 | self.batch_size = batch_size 44 | self.dtype = dtype 45 | self.calib_path = calib_path 46 | # Make a buffer so that PyTorch does not complain when creating 47 | # the camera matrix 48 | self.register_buffer('zero', 49 | torch.zeros([batch_size], dtype=dtype)) 50 | 51 | import os.path as osp 52 | if not osp.exists(calib_path): 53 | raise FileNotFoundError('Could''t find {}.'.format(calib_path)) 54 | else: 55 | focal_length_x, focal_length_y, center, rotation, translation, cam_center, _, _ \ 56 | = extract_cam_param_xml(xml_path=calib_path, dtype=dtype) 57 | 58 | if focal_length_x is None or type(focal_length_x) == float: 59 | focal_length_x = torch.full( 60 | [batch_size], 61 | focal_length_x, 62 | dtype=dtype) 63 | 64 | if focal_length_y is None or type(focal_length_y) == float: 65 | focal_length_y = torch.full( 66 | [batch_size], 67 | focal_length_y, 68 | dtype=dtype) 69 | 70 | self.register_buffer('focal_length_x', focal_length_x) 71 | self.register_buffer('focal_length_y', focal_length_y) 72 | 73 | if center is None: 74 | center = torch.zeros([batch_size, 2], dtype=dtype) 75 | self.register_buffer('center', center) 76 | 77 | rotation = rotation.unsqueeze(dim=0).repeat(batch_size, 1, 1) 78 | rotation = nn.Parameter(rotation, requires_grad=False) 79 | 80 | self.register_parameter('rotation', rotation) 81 | 82 | if translation is None: 83 | translation = torch.zeros([batch_size, 3], dtype=dtype) 84 | 85 | translation = translation.view(3, -1).repeat(batch_size, 1, 1).squeeze(dim=-1) 86 | translation = nn.Parameter(translation, requires_grad=False) 87 | self.register_parameter('translation', translation) 88 | 89 | cam_center = nn.Parameter(cam_center, requires_grad=False) 90 | self.register_parameter('cam_center', cam_center) 91 | 92 | def forward(self, points): 93 | device = points.device 94 | 95 | with torch.no_grad(): 96 | camera_mat = torch.zeros([self.batch_size, 2, 2], 97 | dtype=self.dtype, device=points.device) 98 | camera_mat[:, 0, 0] = self.focal_length_x 99 | camera_mat[:, 1, 1] = self.focal_length_y 100 | 101 | camera_transform = transform_mat(self.rotation, 102 | self.translation.unsqueeze(dim=-1)) 103 | homog_coord = torch.ones(list(points.shape)[:-1] + [1], 104 | dtype=points.dtype, 105 | device=device) 106 | # Convert the points to homogeneous coordinates 107 | points_h = torch.cat([points, homog_coord], dim=-1) 108 | 109 | projected_points = torch.einsum('bki,bji->bjk', 110 | [camera_transform, points_h]) 111 | 112 | img_points = torch.div(projected_points[:, :, :2], 113 | projected_points[:, :, 2].unsqueeze(dim=-1)) 114 | img_points = torch.einsum('bki,bji->bjk', [camera_mat, img_points]) \ 115 | + self.center.unsqueeze(dim=1) 116 | return img_points 117 | --------------------------------------------------------------------------------