├── hoidini ├── closd │ ├── __init__.py │ ├── utils │ │ ├── poselib │ │ │ ├── __init__.py │ │ │ ├── poselib │ │ │ │ ├── core │ │ │ │ │ ├── tests │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ └── test_rotation.py │ │ │ │ │ ├── backend │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ └── logger.py │ │ │ │ │ ├── __init__.py │ │ │ │ │ └── tensor_utils.py │ │ │ │ ├── skeleton │ │ │ │ │ ├── __init__.py │ │ │ │ │ ├── tests │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ └── transfer_npy.py │ │ │ │ │ └── backend │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ └── fbx │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ └── fbx_read_wrapper.py │ │ │ │ ├── visualization │ │ │ │ │ ├── __init__.py │ │ │ │ │ ├── tests │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ └── test_plotter.py │ │ │ │ │ └── core.py │ │ │ │ └── __init__.py │ │ │ ├── setup.py │ │ │ └── README.md │ │ ├── smpllib │ │ │ ├── __init__.py │ │ │ ├── khrylib │ │ │ │ ├── __init__.py │ │ │ │ ├── models │ │ │ │ │ ├── __init__.py │ │ │ │ │ ├── discriminator.py │ │ │ │ │ ├── simple_cnn.py │ │ │ │ │ ├── cmlp.py │ │ │ │ │ ├── resnet.py │ │ │ │ │ ├── erd_net.py │ │ │ │ │ ├── mobile_net.py │ │ │ │ │ ├── mlp.py │ │ │ │ │ ├── rnn.py │ │ │ │ │ ├── video_reg_net.py │ │ │ │ │ └── tcn.py │ │ │ │ ├── rl │ │ │ │ │ ├── __init__.py │ │ │ │ │ ├── agents │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ └── agent_pg.py │ │ │ │ │ ├── core │ │ │ │ │ │ ├── critic.py │ │ │ │ │ │ ├── trajbatch.py │ │ │ │ │ │ ├── __init__.py │ │ │ │ │ │ ├── policy.py │ │ │ │ │ │ ├── common.py │ │ │ │ │ │ ├── policy_disc.py │ │ │ │ │ │ ├── running_norm.py │ │ │ │ │ │ ├── distributions.py │ │ │ │ │ │ ├── policy_gaussian.py │ │ │ │ │ │ └── logger_rl.py │ │ │ │ │ ├── envs │ │ │ │ │ │ └── visual │ │ │ │ │ │ │ └── humanoid_vis.py │ │ │ │ │ └── utils │ │ │ │ │ │ └── visualizer.py │ │ │ │ ├── utils │ │ │ │ │ ├── __init__.py │ │ │ │ │ ├── memory.py │ │ │ │ │ ├── logger.py │ │ │ │ │ ├── zfilter.py │ │ │ │ │ ├── mujoco.py │ │ │ │ │ └── tools.py │ │ │ │ ├── scripts │ │ │ │ │ └── create_vis_model.py │ │ │ │ └── mocap │ │ │ │ │ ├── vis_joint_range.py │ │ │ │ │ └── pose.py │ │ │ └── utils │ │ │ │ ├── flags.py │ │ │ │ ├── geom.py │ │ │ │ ├── vis_model_utils.py │ │ │ │ ├── lightning_utils.py │ │ │ │ └── replay_data.py │ │ ├── flags.py │ │ ├── closd_util.py │ │ ├── hf_handler.py │ │ ├── __init__.py │ │ ├── draw_utils.py │ │ └── parse_task.py │ └── diffusion_planner │ │ ├── data_loaders │ │ ├── humanml │ │ │ ├── data │ │ │ │ └── __init__.py │ │ │ ├── networks │ │ │ │ └── __init__.py │ │ │ ├── motion_loaders │ │ │ │ ├── __init__.py │ │ │ │ └── dataset_motion_loader.py │ │ │ ├── README.md │ │ │ └── utils │ │ │ │ ├── get_opt.py │ │ │ │ └── paramUtil.py │ │ └── humanml_utils.py │ │ ├── assets │ │ ├── example_action_names_humanact12.txt │ │ ├── example_action_names_uestc.txt │ │ └── example_text_prompts.txt │ │ ├── dataset │ │ ├── HumanML3D │ │ ├── README.md │ │ ├── kit_opt.txt │ │ └── humanml_opt.txt │ │ ├── body_models │ │ └── README.md │ │ ├── utils │ │ ├── fixseed.py │ │ ├── config.py │ │ ├── PYTORCH3D_LICENSE │ │ ├── dist_util.py │ │ ├── loss_util.py │ │ ├── misc.py │ │ └── sampler_util.py │ │ ├── LICENSE │ │ ├── model │ │ └── BERT │ │ │ └── BERT_encoder.py │ │ ├── diffusion │ │ └── losses.py │ │ └── train │ │ ├── train_mdm.py │ │ └── train_platforms.py ├── datasets │ ├── __init__.py │ └── resources │ │ ├── gen_test_seq_paths_debug.json │ │ ├── gen_train_seq_paths.json │ │ └── gen_test_seq_paths.json ├── geometry3d │ ├── __init__.py │ └── plot_mesh.py ├── cphoi │ └── cphoi_types.py ├── object_contact_prediction │ └── pointwise_embeddings_extractor.py ├── .DS_Store ├── configs │ ├── 1_ours.yaml │ ├── 1_ours_compare.yaml │ ├── 7_single_phase.yaml │ ├── 6_wo_jitter_loss.yaml │ ├── 2_phase1_inference_phase2_dno.yaml │ ├── 5_nearest_neighbors_instead_of_cps.yaml │ ├── 4b_contact_vs_penetration.yaml │ ├── 2_phase1_inference_phase2_dno_compare.yaml │ ├── 5_nearest_neighbors_instead_of_cps_compare.yaml │ ├── 4_contact_vs_penetration.yaml │ ├── 3_classifier_guidance.yaml │ ├── 1b_inference_compare.yaml │ ├── 3_classifier_guidance_compare.yaml │ ├── 2b_phase1_inference_phase2_dno_no_table_losses.yaml │ ├── 1b_inference_only.yaml │ ├── sampling_cphoi_kitchen.yaml │ ├── 0_base_config_comparison.yaml │ ├── 0_base_config.yaml │ └── experiments │ │ └── resample_result.ipynb ├── __init__.py ├── skeletons │ ├── vertices_segments │ │ ├── smplx_segmentation.png │ │ ├── smpl_segmentation_on_template.png │ │ └── vertex_ids.py │ ├── mano_anchors │ │ ├── mano_anchors_visualization.jpg │ │ └── mano_contact_anchors.py │ ├── smplx_hand_16.py │ └── smpl_24.py ├── model_utils.py ├── smplx │ ├── __init__.py │ ├── vertex_ids.py │ └── vertex_joint_selector.py ├── amasstools │ ├── loop_amass.py │ ├── extract_joints.py │ ├── joints.py │ ├── smpl_mirroring.py │ └── slerp.py ├── blender_utils │ ├── create_table.py │ ├── visualize_mesh.py │ └── visualize_mesh_figure_blender.py ├── resource_paths.py ├── optimize_latent │ └── ol_utils.py ├── normalizer.py ├── objects_fk.py └── object_conditioning │ └── object_encoder_pointwise.py ├── assets └── teaser.png ├── scripts ├── eval.sh ├── train_cphoi.sh └── inference.sh ├── pyrightconfig.json ├── LICENSE ├── cspell.json ├── setup.sh └── INSTALL.md /hoidini/closd/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/datasets/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/geometry3d/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/core/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/skeleton/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/models/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/skeleton/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/visualization/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/cphoi/cphoi_types.py: -------------------------------------------------------------------------------- 1 | from typing import Optional 2 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/skeleton/backend/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/skeleton/backend/fbx/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/visualization/tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/object_contact_prediction/pointwise_embeddings_extractor.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/data_loaders/humanml/data/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/data_loaders/humanml/networks/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/data_loaders/humanml/motion_loaders/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /assets/teaser.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hoidini/HOIDiNi/HEAD/assets/teaser.png -------------------------------------------------------------------------------- /hoidini/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hoidini/HOIDiNi/HEAD/hoidini/.DS_Store -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/__init__.py: -------------------------------------------------------------------------------- 1 | __all__ = ["agents", "core", "envs"] 2 | -------------------------------------------------------------------------------- /hoidini/configs/1_ours.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config.yaml 3 | - _self_ 4 | 5 | 6 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/assets/example_action_names_humanact12.txt: -------------------------------------------------------------------------------- 1 | drink 2 | lift_dumbbell 3 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/dataset/HumanML3D: -------------------------------------------------------------------------------- 1 | /home/dcor/datasets/motion/HumanML3D_from_bermano/ -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/__init__.py: -------------------------------------------------------------------------------- 1 | __version__ = "0.0.1" 2 | 3 | from .core import * 4 | -------------------------------------------------------------------------------- /hoidini/configs/1_ours_compare.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config_comparison.yaml 3 | - _self_ 4 | 5 | 6 | -------------------------------------------------------------------------------- /hoidini/configs/7_single_phase.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config.yaml 3 | - _self_ 4 | 5 | one_phase_experiment: true -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/data_loaders/humanml/README.md: -------------------------------------------------------------------------------- 1 | This code is based on https://github.com/EricGuo5513/text-to-motion.git -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/body_models/README.md: -------------------------------------------------------------------------------- 1 | ## Body models 2 | 3 | Put SMPL models here (full instractions in the main README) -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/core/backend/__init__.py: -------------------------------------------------------------------------------- 1 | from .abstract import Serializable 2 | 3 | from .logger import logger 4 | -------------------------------------------------------------------------------- /scripts/eval.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | conda activate hoidini 3 | export PYTHONPATH="${PYTHONPATH}:$(pwd)" 4 | 5 | python hoidini/eval/eval.py 6 | 7 | -------------------------------------------------------------------------------- /hoidini/configs/6_wo_jitter_loss.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config.yaml 3 | - _self_ 4 | 5 | dno_loss_coefficients_phase2: 6 | jitter_joints: 0.0 -------------------------------------------------------------------------------- /hoidini/__init__.py: -------------------------------------------------------------------------------- 1 | import warnings 2 | import matplotlib 3 | 4 | warnings.filterwarnings("ignore", category=matplotlib.MatplotlibDeprecationWarning) 5 | -------------------------------------------------------------------------------- /hoidini/configs/2_phase1_inference_phase2_dno.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config.yaml 3 | - _self_ 4 | 5 | use_dno_object: false 6 | use_dno_body: true -------------------------------------------------------------------------------- /hoidini/configs/5_nearest_neighbors_instead_of_cps.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config.yaml 3 | - _self_ 4 | 5 | replace_cps_with_nearest_neighbors: true -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/core/__init__.py: -------------------------------------------------------------------------------- 1 | from .tensor_utils import * 2 | from .rotation3d import * 3 | from .backend import Serializable, logger 4 | -------------------------------------------------------------------------------- /hoidini/configs/4b_contact_vs_penetration.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config.yaml 3 | - _self_ 4 | 5 | 6 | dno_loss_coefficients_phase2: 7 | penetration: 0.0 -------------------------------------------------------------------------------- /hoidini/skeletons/vertices_segments/smplx_segmentation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hoidini/HOIDiNi/HEAD/hoidini/skeletons/vertices_segments/smplx_segmentation.png -------------------------------------------------------------------------------- /hoidini/skeletons/mano_anchors/mano_anchors_visualization.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hoidini/HOIDiNi/HEAD/hoidini/skeletons/mano_anchors/mano_anchors_visualization.jpg -------------------------------------------------------------------------------- /hoidini/configs/2_phase1_inference_phase2_dno_compare.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config_comparison.yaml 3 | - _self_ 4 | 5 | use_dno_object: false 6 | use_dno_body: true -------------------------------------------------------------------------------- /hoidini/configs/5_nearest_neighbors_instead_of_cps_compare.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config_comparison.yaml 3 | - _self_ 4 | 5 | replace_cps_with_nearest_neighbors: true -------------------------------------------------------------------------------- /hoidini/configs/4_contact_vs_penetration.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config.yaml 3 | - _self_ 4 | 5 | 6 | dno_loss_coefficients_phase2: 7 | contact: 0.1 8 | penetration: 0.9 -------------------------------------------------------------------------------- /hoidini/skeletons/vertices_segments/smpl_segmentation_on_template.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hoidini/HOIDiNi/HEAD/hoidini/skeletons/vertices_segments/smpl_segmentation_on_template.png -------------------------------------------------------------------------------- /hoidini/datasets/resources/gen_test_seq_paths_debug.json: -------------------------------------------------------------------------------- 1 | [ 2 | "s10/pyramidsmall_inspect_1", 3 | "s10/toothpaste_pass_1", 4 | "s10/spheremedium_inspect_1", 5 | "s10/spheremedium_pass_1" 6 | ] -------------------------------------------------------------------------------- /hoidini/configs/3_classifier_guidance.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config.yaml 3 | - _self_ 4 | 5 | use_classifier_guidance: true 6 | classifier_guidance_lr_factor: null 7 | classifier_guidance_n_steps: 500 -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/assets/example_action_names_uestc.txt: -------------------------------------------------------------------------------- 1 | jumping-jack 2 | left-lunging 3 | left-stretching 4 | raising-hand-and-jumping 5 | rotation-clapping 6 | front-raising 7 | pulling-chest-expanders 8 | -------------------------------------------------------------------------------- /hoidini/configs/1b_inference_compare.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config_comparison.yaml 3 | - _self_ 4 | 5 | 6 | use_dno_object: false 7 | use_dno_body: false 8 | 9 | # Taken from ours phase0 10 | 11 | 12 | -------------------------------------------------------------------------------- /hoidini/configs/3_classifier_guidance_compare.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config_comparison.yaml 3 | - _self_ 4 | 5 | use_classifier_guidance: true 6 | classifier_guidance_lr_factor: null 7 | classifier_guidance_n_steps: 500 -------------------------------------------------------------------------------- /hoidini/configs/2b_phase1_inference_phase2_dno_no_table_losses.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config.yaml 3 | - _self_ 4 | 5 | use_dno_object: false 6 | use_dno_body: true 7 | 8 | 9 | dno_loss_coefficients_phase2: 10 | above_table: 0.0 11 | side_table: 0.0 -------------------------------------------------------------------------------- /hoidini/configs/1b_inference_only.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - 0_base_config.yaml 3 | - _self_ 4 | 5 | 6 | result_phases_to_animate: [phase0] 7 | result_phases_to_save: [phase0] 8 | 9 | use_dno_object: false 10 | use_dno_body: false 11 | 12 | # Taken from ours phase0 13 | 14 | 15 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/agents/__init__.py: -------------------------------------------------------------------------------- 1 | from hoidini.closd.utils.smpllib.khrylib.rl.agents.agent_pg import AgentPG 2 | from hoidini.closd.utils.smpllib.khrylib.rl.agents.agent_ppo import AgentPPO 3 | from hoidini.closd.utils.smpllib.khrylib.rl.agents.agent_trpo import AgentTRPO 4 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/utils/flags.py: -------------------------------------------------------------------------------- 1 | __all__ = ["flags", "summation"] 2 | 3 | 4 | class Flags(object): 5 | def __init__(self, *items): 6 | for key, val in zip(items[:-1], items[1:]): 7 | setattr(self, key, val) 8 | 9 | 10 | flags = Flags("debug", False) 11 | -------------------------------------------------------------------------------- /hoidini/closd/utils/flags.py: -------------------------------------------------------------------------------- 1 | __all__ = ["flags", "summation"] 2 | 3 | 4 | class Flags(object): 5 | def __init__(self, items): 6 | for key, val in items.items(): 7 | setattr(self, key, val) 8 | 9 | 10 | flags = Flags( 11 | { 12 | "test": False, 13 | "debug": False, 14 | "real_traj": False, 15 | "im_eval": False, 16 | } 17 | ) 18 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/dataset/README.md: -------------------------------------------------------------------------------- 1 | ## Data 2 | 3 | * Data dirs should be placed here. 4 | 5 | * The `opt` files are configurations for how to read the data according to [text-to-motion](https://github.com/EricGuo5513/text-to-motion). 6 | * The `*_mean.npy` and `*_std.npy` files, are stats used for evaluation only, according to [text-to-motion](https://github.com/EricGuo5513/text-to-motion). -------------------------------------------------------------------------------- /hoidini/closd/utils/closd_util.py: -------------------------------------------------------------------------------- 1 | import enum 2 | 3 | 4 | class STATES(enum.IntEnum): 5 | """ 6 | For the state machine 7 | """ 8 | 9 | NO_STATE = enum.auto() 10 | REACH = enum.auto() 11 | STRIKE_KICK = enum.auto() 12 | STRIKE_PUNCH = enum.auto() 13 | HALT = enum.auto() 14 | SIT = enum.auto() 15 | GET_UP = enum.auto() 16 | TEXT2MOTION = enum.auto() 17 | -------------------------------------------------------------------------------- /hoidini/model_utils.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch_geometric 3 | 4 | 5 | def model_kwargs_to_device(model_kwargs, device): 6 | model_kwargs["y"] = { 7 | key: ( 8 | val.to(device) 9 | if torch.is_tensor(val) or isinstance(val, torch_geometric.data.Data) 10 | else val 11 | ) 12 | for key, val in model_kwargs["y"].items() 13 | } 14 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/utils/__init__.py: -------------------------------------------------------------------------------- 1 | from hoidini.closd.utils.smpllib.khrylib.utils.memory import * 2 | from hoidini.closd.utils.smpllib.khrylib.utils.zfilter import * 3 | from hoidini.closd.utils.smpllib.khrylib.utils.torch import * 4 | from hoidini.closd.utils.smpllib.khrylib.utils.tools import * 5 | from hoidini.closd.utils.smpllib.khrylib.utils.logger import * 6 | from hoidini.closd.utils.smpllib.khrylib.utils.mujoco import * 7 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/utils/fixseed.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | import random 4 | 5 | 6 | def fixseed(seed): 7 | torch.backends.cudnn.benchmark = False 8 | random.seed(seed) 9 | np.random.seed(seed) 10 | torch.manual_seed(seed) 11 | 12 | 13 | # SEED = 10 14 | # EVALSEED = 0 15 | # # Provoc warning: not fully functionnal yet 16 | # # torch.set_deterministic(True) 17 | # torch.backends.cudnn.benchmark = False 18 | # fixseed(SEED) 19 | -------------------------------------------------------------------------------- /pyrightconfig.json: -------------------------------------------------------------------------------- 1 | { 2 | "include": ["hoidini"], 3 | "exclude": [ 4 | "hoidini_training/**", 5 | "hoidini_results/**", 6 | "hoidini_data/**", 7 | "**/node_modules", 8 | "**/__pycache__", 9 | "**/output", 10 | "wandb/**", 11 | "results/**", 12 | "rendering/**", 13 | "cache/**", 14 | "user_study*/**", 15 | "results/**", 16 | "*.obj" 17 | ], 18 | "verboseOutput": true 19 | } 20 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/assets/example_text_prompts.txt: -------------------------------------------------------------------------------- 1 | person got down and is crawling across the floor. 2 | a person walks forward with wide steps. 3 | a person drops their hands then brings them together in front of their face clasped. 4 | a person lifts their right arm and slaps something, then repeats the motion again. 5 | a person walks forward and stops. 6 | a person marches forward, turns around, and then marches back. 7 | a person is stretching their arms. 8 | person is making attention gesture -------------------------------------------------------------------------------- /scripts/train_cphoi.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | conda activate hoidini 3 | export PYTHONPATH="${PYTHONPATH}:$(pwd)" 4 | SAVE_DIR=hoidini_training/cphoi_v0 5 | 6 | python hoidini/cphoi/cphoi_train.py \ 7 | save_dir=$SAVE_DIR \ 8 | debug_mode=False \ 9 | device=0 \ 10 | batch_size=64 \ 11 | pcd_n_points=512 \ 12 | pcd_augment_rot_z=True \ 13 | pcd_augment_jitter=True \ 14 | pred_len=100 \ 15 | context_len=15 \ 16 | diffusion_steps=8 \ 17 | augment_xy_plane_prob=0.5 \ 18 | mixed_dataset=False \ 19 | overwrite=True 20 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/utils/config.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | SMPL_DATA_PATH = "./body_models/smpl" 4 | 5 | SMPL_KINTREE_PATH = os.path.join(SMPL_DATA_PATH, "kintree_table.pkl") 6 | SMPL_MODEL_PATH = os.path.join(SMPL_DATA_PATH, "SMPL_NEUTRAL.pkl") 7 | JOINT_REGRESSOR_TRAIN_EXTRA = os.path.join(SMPL_DATA_PATH, "J_regressor_extra.npy") 8 | 9 | ROT_CONVENTION_TO_ROT_NUMBER = { 10 | "legacy": 23, 11 | "no_hands": 21, 12 | "full_hands": 51, 13 | "mitten_hands": 33, 14 | } 15 | 16 | GENDERS = ["neutral", "male", "female"] 17 | NUM_BETAS = 10 18 | -------------------------------------------------------------------------------- /hoidini/skeletons/vertices_segments/vertex_ids.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | from typing import Dict, List 4 | 5 | 6 | def get_segments_vertex_ids_dict(model_name) -> Dict[str, List[int]]: 7 | assert model_name in ["smpl", "smplx"] 8 | file_path = os.path.join( 9 | os.path.dirname(__file__), f"{model_name}_vert_segmentation.json" 10 | ) 11 | with open(file_path, "r") as f: 12 | part_segm = json.load(f) 13 | return part_segm 14 | 15 | 16 | if __name__ == "__main__": 17 | d = get_segments_vertex_ids_dict("smpl") 18 | print(d) 19 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/core/critic.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch 3 | 4 | 5 | class Value(nn.Module): 6 | def __init__(self, net, net_out_dim=None): 7 | super().__init__() 8 | self.net = net 9 | if net_out_dim is None: 10 | net_out_dim = net.out_dim 11 | self.value_head = nn.Linear(net_out_dim, 1) 12 | self.value_head.weight.data.mul_(0.1) 13 | self.value_head.bias.data.mul_(0.0) 14 | 15 | def forward(self, x): 16 | x = self.net(x) 17 | value = self.value_head(x) 18 | return value 19 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/models/discriminator.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch 3 | 4 | 5 | class Discriminator(nn.Module): 6 | def __init__(self, net, net_out_dim=None): 7 | super().__init__() 8 | self.net = net 9 | if net_out_dim is None: 10 | net_out_dim = net.out_dim 11 | self.logic = nn.Linear(net_out_dim, 1) 12 | self.logic.weight.data.mul_(0.1) 13 | self.logic.bias.data.mul_(0.0) 14 | 15 | def forward(self, x): 16 | x = self.net(x) 17 | prob = torch.sigmoid(self.logic(x)) 18 | return prob 19 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/core/trajbatch.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | 4 | class TrajBatch: 5 | def __init__(self, memory_list): 6 | memory = memory_list[0] 7 | for x in memory_list[1:]: 8 | memory.append(x) 9 | self.batch = zip(*memory.sample()) 10 | self.states = np.stack(next(self.batch)) 11 | self.actions = np.stack(next(self.batch)) 12 | self.masks = np.stack(next(self.batch)) 13 | self.next_states = np.stack(next(self.batch)) 14 | self.rewards = np.stack(next(self.batch)) 15 | self.exps = np.stack(next(self.batch)) 16 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/utils/memory.py: -------------------------------------------------------------------------------- 1 | import random 2 | 3 | 4 | class Memory(object): 5 | def __init__(self): 6 | self.memory = [] 7 | 8 | def push(self, *args): 9 | """Saves a tuple.""" 10 | self.memory.append([*args]) 11 | 12 | def sample(self, batch_size=None): 13 | if batch_size is None: 14 | return self.memory 15 | else: 16 | random_batch = random.sample(self.memory, batch_size) 17 | return random_batch 18 | 19 | def append(self, new_memory): 20 | self.memory += new_memory.memory 21 | 22 | def __len__(self): 23 | return len(self.memory) 24 | -------------------------------------------------------------------------------- /scripts/inference.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | conda activate hoidini 3 | export PYTHONPATH="${PYTHONPATH}:$(pwd)" 4 | OUT_DIR=hoidini_results 5 | 6 | 7 | python hoidini/cphoi/cphoi_inference.py \ 8 | --config-name="0_base_config.yaml" \ 9 | model_path=hoidini_data/models/cphoi_05011024_c15p100_v0/model000120000.pt \ 10 | out_dir=$OUT_DIR \ 11 | dno_options_phase1.num_opt_steps=200 \ 12 | dno_options_phase2.num_opt_steps=200 \ 13 | sampler_config.n_samples=10 \ 14 | sampler_config.n_frames=115 15 | # n_simplify_hands=700 \ 16 | # n_simplify_object=700 \ 17 | 18 | # Uncomment n_simplify_hands and n_simplify_object to run with low GPU memory (will harm the quality) -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/core/__init__.py: -------------------------------------------------------------------------------- 1 | from hoidini.closd.utils.smpllib.khrylib.rl.core.common import * 2 | from hoidini.closd.utils.smpllib.khrylib.rl.core.critic import Value 3 | from hoidini.closd.utils.smpllib.khrylib.rl.core.distributions import ( 4 | DiagGaussian, 5 | Categorical, 6 | ) 7 | from hoidini.closd.utils.smpllib.khrylib.rl.core.logger_rl import LoggerRL 8 | from hoidini.closd.utils.smpllib.khrylib.rl.core.policy import Policy 9 | from hoidini.closd.utils.smpllib.khrylib.rl.core.policy_disc import PolicyDiscrete 10 | from hoidini.closd.utils.smpllib.khrylib.rl.core.policy_gaussian import PolicyGaussian 11 | from hoidini.closd.utils.smpllib.khrylib.rl.core.trajbatch import TrajBatch 12 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup 2 | 3 | setup( 4 | name="poselib", 5 | packages=["poselib"], 6 | version="0.0.42", 7 | description="Framework Agnostic Tensor Programming", 8 | author="Qiyang Li, Kelly Guo, Brendon Matusch", 9 | classifiers=[ 10 | "Programming Language :: Python", 11 | "Programming Language :: Python :: 3", 12 | "License :: OSI Approved :: GNU General Public License (GPL)", 13 | "Operating System :: OS Independent", 14 | "Development Status :: 1 - Planning", 15 | "Environment :: Console", 16 | "Intended Audience :: Science/Research", 17 | "Topic :: Scientific/Engineering :: GIS", 18 | ], 19 | ) 20 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/core/policy.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | 3 | 4 | class Policy(nn.Module): 5 | def __init__(self): 6 | super().__init__() 7 | 8 | def forward(self, x): 9 | """This function should return a distribution to sample action from""" 10 | raise NotImplementedError 11 | 12 | def select_action(self, x, mean_action=False): 13 | dist = self.forward(x) 14 | action = dist.mean_sample() if mean_action else dist.sample() 15 | return action 16 | 17 | def get_kl(self, x): 18 | dist = self.forward(x) 19 | return dist.kl() 20 | 21 | def get_log_prob(self, x, action): 22 | dist = self.forward(x) 23 | return dist.log_prob(action) 24 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/visualization/tests/test_plotter.py: -------------------------------------------------------------------------------- 1 | from typing import cast 2 | 3 | import matplotlib.pyplot as plt 4 | import numpy as np 5 | 6 | from ..core import BasePlotterTask, BasePlotterTasks 7 | from ..plt_plotter import Matplotlib3DPlotter 8 | from ..simple_plotter_tasks import Draw3DDots, Draw3DLines 9 | 10 | task = Draw3DLines( 11 | task_name="test", 12 | lines=np.array([[[0, 0, 0], [0, 0, 1]], [[0, 1, 1], [0, 1, 0]]]), 13 | color="blue", 14 | ) 15 | task2 = Draw3DDots( 16 | task_name="test2", 17 | dots=np.array([[0, 0, 0], [0, 0, 1], [0, 1, 1], [0, 1, 0]]), 18 | color="red", 19 | ) 20 | task3 = BasePlotterTasks([task, task2]) 21 | plotter = Matplotlib3DPlotter(cast(BasePlotterTask, task3)) 22 | plt.show() 23 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/core/backend/logger.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. 2 | # NVIDIA CORPORATION and its licensors retain all intellectual property 3 | # and proprietary rights in and to this software, related documentation 4 | # and any modifications thereto. Any use, reproduction, disclosure or 5 | # distribution of this software and related documentation without an express 6 | # license agreement from NVIDIA CORPORATION is strictly prohibited. 7 | 8 | import logging 9 | 10 | logger = logging.getLogger("poselib") 11 | logger.setLevel(logging.INFO) 12 | 13 | if not len(logger.handlers): 14 | formatter = logging.Formatter( 15 | fmt="%(asctime)-15s - %(levelname)s - %(module)s - %(message)s" 16 | ) 17 | handler = logging.StreamHandler() 18 | handler.setFormatter(formatter) 19 | logger.addHandler(handler) 20 | logger.info("logger initialized") 21 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/models/simple_cnn.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | 5 | class SimpleCNN(nn.Module): 6 | 7 | def __init__(self, out_dim): 8 | super().__init__() 9 | self.conv1 = nn.Conv2d(3, 32, kernel_size=4, stride=4) 10 | self.conv2 = nn.Conv2d(32, 32, kernel_size=4, stride=4) 11 | self.conv3 = nn.Conv2d(32, 16, kernel_size=4, stride=4) 12 | self.fc = nn.Linear(144, out_dim) 13 | 14 | def forward(self, x): 15 | x = torch.relu(self.conv1(x)) 16 | x = torch.relu(self.conv2(x)) 17 | x = torch.relu(self.conv3(x)) 18 | x = self.fc(x.view(x.size(0), -1)) 19 | return x 20 | 21 | 22 | if __name__ == "__main__": 23 | import time 24 | 25 | torch.set_grad_enabled(False) 26 | net = SimpleCNN(128) 27 | t0 = time.time() 28 | input = torch.zeros(1, 3, 224, 224) 29 | out = net(input) 30 | print(time.time() - t0) 31 | print(out.shape) 32 | -------------------------------------------------------------------------------- /hoidini/skeletons/mano_anchors/mano_contact_anchors.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import json 3 | import os 4 | from functools import lru_cache 5 | 6 | from hoidini.general_utils import SRC_DIR 7 | 8 | 9 | @lru_cache() 10 | def get_contact_anchors_info(): 11 | """ 12 | R2x meaning that the indices belong to x's mesh 13 | """ 14 | with open( 15 | os.path.join(SRC_DIR, "skeletons/mano_anchors/mano_contact_anchors_data.json"), 16 | "r", 17 | ) as f: 18 | anchors_data = json.load(f) 19 | anchor_inds_R2hands = torch.tensor(anchors_data["anchor_indices"]) 20 | closest_anchor_per_vertex_R2anchors = torch.tensor( 21 | anchors_data["closest_anchor_per_vertex"] 22 | ) 23 | 24 | closest_anchor_per_vertex_R2hands = anchor_inds_R2hands[ 25 | closest_anchor_per_vertex_R2anchors 26 | ] 27 | return ( 28 | anchor_inds_R2hands, 29 | closest_anchor_per_vertex_R2anchors, 30 | closest_anchor_per_vertex_R2hands, 31 | ) 32 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/models/cmlp.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch 3 | 4 | 5 | class CMLP(nn.Module): 6 | def __init__(self, input_dim, cond_dim, hidden_dims=(128, 128), activation="tanh"): 7 | super().__init__() 8 | if activation == "tanh": 9 | self.activation = torch.tanh 10 | elif activation == "relu": 11 | self.activation = torch.relu 12 | elif activation == "sigmoid": 13 | self.activation = torch.sigmoid 14 | 15 | self.cond_dim = cond_dim 16 | self.out_dim = hidden_dims[-1] 17 | self.affine_layers = nn.ModuleList() 18 | last_dim = input_dim 19 | for nh in hidden_dims: 20 | self.affine_layers.append(nn.Linear(last_dim + cond_dim, nh)) 21 | last_dim = nh 22 | 23 | def forward(self, c, x): 24 | for affine in self.affine_layers: 25 | x = torch.cat((c, x), dim=1) 26 | x = self.activation(affine(x)) 27 | return x 28 | -------------------------------------------------------------------------------- /hoidini/smplx/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is 4 | # holder of all proprietary rights on this computer program. 5 | # You can only use this computer program if you have closed 6 | # a license agreement with MPG or you get the right to use the computer 7 | # program from someone who is authorized to grant you that right. 8 | # Any use of the computer program without a valid license is prohibited and 9 | # liable to prosecution. 10 | # 11 | # Copyright©2019 Max-Planck-Gesellschaft zur Förderung 12 | # der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute 13 | # for Intelligent Systems. All rights reserved. 14 | # 15 | # Contact: ps-license@tuebingen.mpg.de 16 | 17 | from .body_models import ( 18 | create, 19 | SMPL, 20 | SMPLH, 21 | SMPLX, 22 | MANO, 23 | FLAME, 24 | build_layer, 25 | SMPLLayer, 26 | SMPLHLayer, 27 | SMPLXLayer, 28 | MANOLayer, 29 | FLAMELayer, 30 | ) 31 | -------------------------------------------------------------------------------- /hoidini/skeletons/smplx_hand_16.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from smplx.joint_names import JOINT_NAMES as SMPLX_ALL_JOINS 3 | from skeletons.smplx_52 import BONES_52_NAMES 4 | 5 | 6 | RIGHT_HAND_JOINT_NAMES = [ 7 | "right_wrist", 8 | "right_index1", 9 | "right_index2", 10 | "right_index3", 11 | "right_middle1", 12 | "right_middle2", 13 | "right_middle3", 14 | "right_pinky1", 15 | "right_pinky2", 16 | "right_pinky3", 17 | "right_ring1", 18 | "right_ring2", 19 | "right_ring3", 20 | "right_thumb1", 21 | "right_thumb2", 22 | "right_thumb3", 23 | ] 24 | 25 | len(SMPLX_ALL_JOINS) 26 | 27 | RIGHT_HAND_JOINT_INDS = np.array( 28 | [SMPLX_ALL_JOINS.index(jname) for jname in RIGHT_HAND_JOINT_NAMES] 29 | ) 30 | 31 | BONE_NAMES_RIGHT_HAND = [ 32 | (h, t) for h, t in BONES_52_NAMES if h in RIGHT_HAND_JOINT_NAMES 33 | ] 34 | BONE_LIST_RIGHT_HAND = [ 35 | (RIGHT_HAND_JOINT_NAMES.index(h), RIGHT_HAND_JOINT_NAMES.index(t)) 36 | for h, t in BONE_NAMES_RIGHT_HAND 37 | ] 38 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/core/common.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from hoidini.closd.utils.smpllib.khrylib.utils import batch_to 3 | 4 | 5 | def estimate_advantages(rewards, masks, values, gamma, tau): 6 | device = rewards.device 7 | rewards, masks, values = batch_to(torch.device("cpu"), rewards, masks, values) 8 | tensor_type = type(rewards) 9 | deltas = tensor_type(rewards.size(0), 1) 10 | advantages = tensor_type(rewards.size(0), 1) 11 | 12 | prev_value = 0 13 | prev_advantage = 0 14 | for i in reversed(range(rewards.size(0))): 15 | deltas[i] = rewards[i] + gamma * prev_value * masks[i] - values[i] 16 | advantages[i] = deltas[i] + gamma * tau * prev_advantage * masks[i] 17 | 18 | prev_value = values[i, 0] 19 | prev_advantage = advantages[i, 0] 20 | 21 | returns = values + advantages 22 | advantages = (advantages - advantages.mean()) / advantages.std() 23 | 24 | advantages, returns = batch_to(device, advantages, returns) 25 | return advantages, returns 26 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/core/policy_disc.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | from hoidini.closd.utils.smpllib.utils.math_utils import * 3 | from hoidini.closd.utils.smpllib.khrylib.rl.core.distributions import Categorical 4 | from hoidini.closd.utils.smpllib.khrylib.rl.core.policy import Policy 5 | 6 | 7 | class PolicyDiscrete(Policy): 8 | def __init__(self, net, action_num, net_out_dim=None): 9 | super().__init__() 10 | self.type = "discrete" 11 | if net_out_dim is None: 12 | net_out_dim = net.out_dim 13 | self.net = net 14 | self.action_head = nn.Linear(net_out_dim, action_num) 15 | self.action_head.weight.data.mul_(0.1) 16 | self.action_head.bias.data.mul_(0.0) 17 | 18 | def forward(self, x): 19 | x = self.net(x) 20 | action_prob = torch.softmax(self.action_head(x), dim=1) 21 | return Categorical(probs=action_prob) 22 | 23 | def get_fim(self, x): 24 | action_prob = self.forward(x) 25 | M = action_prob.pow(-1).view(-1).detach() 26 | return M, action_prob, {} 27 | -------------------------------------------------------------------------------- /hoidini/amasstools/loop_amass.py: -------------------------------------------------------------------------------- 1 | import os 2 | from glob import glob 3 | from tqdm import tqdm 4 | 5 | 6 | def loop_amams( 7 | base_folder, 8 | new_base_folder, 9 | ext=".npz", 10 | newext=".npz", 11 | force_redo=False, 12 | only_mirror=False, 13 | exclude=None, 14 | ): 15 | match_str = f"**/*{ext}" 16 | 17 | if only_mirror: 18 | match_str = f"M2/**/*{ext}" 19 | 20 | for motion_file in tqdm(glob(match_str, root_dir=base_folder, recursive=True)): 21 | if exclude and exclude in motion_file: 22 | continue 23 | 24 | motion_path = os.path.join(base_folder, motion_file) 25 | 26 | if motion_path.endswith("shape.npz"): 27 | continue 28 | 29 | new_motion_path = os.path.join( 30 | new_base_folder, motion_file.replace(ext, newext) 31 | ) 32 | if not force_redo and os.path.exists(new_motion_path): 33 | continue 34 | 35 | new_folder = os.path.split(new_motion_path)[0] 36 | os.makedirs(new_folder, exist_ok=True) 37 | 38 | yield motion_path, new_motion_path 39 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2025 hoidini 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/utils/logger.py: -------------------------------------------------------------------------------- 1 | import logging 2 | import os 3 | 4 | 5 | def create_logger(filename, file_handle=True): 6 | # create logger 7 | for handler in logging.root.handlers[:]: 8 | logging.root.removeHandler(handler) 9 | 10 | logger = logging.getLogger(filename) 11 | if logger.hasHandlers(): 12 | logger.handlers.clear() 13 | logger.propagate = False 14 | logger.setLevel(logging.DEBUG) 15 | # create console handler with a higher log level 16 | ch = logging.StreamHandler() 17 | ch.setLevel(logging.INFO) 18 | stream_formatter = logging.Formatter("%(message)s") 19 | ch.setFormatter(stream_formatter) 20 | logger.addHandler(ch) 21 | 22 | if file_handle: 23 | # create file handler which logs even debug messages 24 | os.makedirs(os.path.dirname(filename), exist_ok=True) 25 | fh = logging.FileHandler(filename, mode="a") 26 | fh.setLevel(logging.DEBUG) 27 | file_formatter = logging.Formatter("[%(asctime)s] %(message)s") 28 | fh.setFormatter(file_formatter) 29 | logger.addHandler(fh) 30 | 31 | return logger 32 | -------------------------------------------------------------------------------- /cspell.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": "0.2", 3 | "language": "en", 4 | "words": [ 5 | "amasstools", 6 | "autoregressive", 7 | "cachedir", 8 | "closd", 9 | "COEFFS", 10 | "cpdm", 11 | "Cphoi", 12 | "dataclass", 13 | "datapoints", 14 | "embd", 15 | "embds", 16 | "embedder", 17 | "fullpose", 18 | "gamecontroller", 19 | "grabhoi", 20 | "grav", 21 | "hoidini", 22 | "HUMANML", 23 | "imos", 24 | "Inpaint", 25 | "inpainted", 26 | "Inpainting", 27 | "keypoints", 28 | "mahoi", 29 | "numpify", 30 | "OMOMO", 31 | "pointcloud", 32 | "Pointwise", 33 | "pythonpath", 34 | "retarget", 35 | "retargeted", 36 | "retargeting", 37 | "rodrigues", 38 | "rotmat", 39 | "seqs", 40 | "smpldata", 41 | "smplrifke", 42 | "smplrifkefeats", 43 | "tfms", 44 | "tgts", 45 | "timestep", 46 | "torchify", 47 | "tpose", 48 | "ungroup" 49 | ] 50 | } 51 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 Guy Tevet 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/models/resnet.py: -------------------------------------------------------------------------------- 1 | from torch import nn 2 | from torchvision import models 3 | from hoidini.closd.utils.smpllib.khrylib.utils.torch import * 4 | 5 | 6 | class ResNet(nn.Module): 7 | 8 | def __init__(self, out_dim, fix_params=False, running_stats=False): 9 | super().__init__() 10 | self.out_dim = out_dim 11 | self.resnet = models.resnet18(pretrained=True) 12 | if fix_params: 13 | for param in self.resnet.parameters(): 14 | param.requires_grad = False 15 | self.resnet.fc = nn.Linear(self.resnet.fc.in_features, out_dim) 16 | self.bn_stats(running_stats) 17 | 18 | def forward(self, x): 19 | return self.resnet(x) 20 | 21 | def bn_stats(self, track_running_stats): 22 | for m in self.modules(): 23 | if type(m) == nn.BatchNorm2d: 24 | m.track_running_stats = track_running_stats 25 | 26 | 27 | if __name__ == "__main__": 28 | import time 29 | 30 | net = ResNet(128) 31 | t0 = time.time() 32 | input = ones(1, 3, 224, 224) 33 | out = net(input) 34 | print(time.time() - t0) 35 | print(out.shape) 36 | -------------------------------------------------------------------------------- /setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Source conda and create environment 4 | # source /home/dcor/roeyron/miniconda3/etc/profile.d/conda.sh 5 | conda create -n "hoidini" python=3.11 -y 6 | conda activate hoidini 7 | echo $(which python) 8 | # Update pip to the latest version 9 | pip install --upgrade pip 10 | 11 | # pytorch 12 | conda install pytorch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 pytorch-cuda=12.1 -c pytorch -c nvidia -y 13 | 14 | # pytorch3d 15 | conda install -c fvcore -c iopath -c conda-forge fvcore iopath 16 | conda install -c bottler nvidiacub 17 | 18 | # Select the correct pytorch3d version for the current pytorch version: 19 | # https://github.com/facebookresearch/pytorch3d/discussions/1752 20 | pip install --extra-index-url https://miropsota.github.io/torch_packages_builder pytorch3d==0.7.8+pt2.5.1cu121 21 | 22 | pip install torch-cluster -f https://data.pyg.org/whl/torch-2.5+124.html 23 | 24 | 25 | # pytorch-geometric 26 | conda install pytorch-geometric -c pyg -y 27 | 28 | # pytorch-lightning 29 | pip install git+https://github.com/otaheri/chamfer_distance 30 | pip install git+https://github.com/otaheri/bps_torch 31 | 32 | 33 | pip install -r requirements.txt 34 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/model/BERT/BERT_encoder.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import os 3 | 4 | 5 | def load_bert(model_path): 6 | bert = BERT(model_path) 7 | bert.eval() 8 | bert.text_model.training = False 9 | for p in bert.parameters(): 10 | p.requires_grad = False 11 | return bert 12 | 13 | 14 | class BERT(nn.Module): 15 | def __init__(self, modelpath: str): 16 | super().__init__() 17 | 18 | from transformers import AutoTokenizer, AutoModel 19 | from transformers import logging 20 | 21 | logging.set_verbosity_error() 22 | # Tokenizer 23 | os.environ["TOKENIZERS_PARALLELISM"] = "false" 24 | # Tokenizer 25 | self.tokenizer = AutoTokenizer.from_pretrained(modelpath) 26 | # Text model 27 | self.text_model = AutoModel.from_pretrained(modelpath) 28 | 29 | def forward(self, texts): 30 | encoded_inputs = self.tokenizer(texts, return_tensors="pt", padding=True) 31 | output = self.text_model( 32 | **encoded_inputs.to(self.text_model.device) 33 | ).last_hidden_state 34 | mask = encoded_inputs.attention_mask.to(dtype=bool) 35 | # output = output * mask.unsqueeze(-1) 36 | return output, mask 37 | -------------------------------------------------------------------------------- /hoidini/blender_utils/create_table.py: -------------------------------------------------------------------------------- 1 | import bpy 2 | import torch 3 | import bmesh 4 | 5 | 6 | def add_table_from_4_points(points, name="TablePlane"): 7 | """ 8 | Adds a quad surface mesh to the scene, formed by four given 3D points. 9 | 10 | Args: 11 | points (list of tuple): A list of 4 tuples, each representing (x, y, z) coordinates. 12 | name (str): Name for the created mesh and object. 13 | """ 14 | assert len(points) == 4, "Expected exactly 4 points to form a quad." 15 | if isinstance(points, torch.Tensor): 16 | points = points.cpu().numpy() 17 | # Create mesh and object 18 | mesh = bpy.data.meshes.new(name) 19 | obj = bpy.data.objects.new(name, mesh) 20 | bpy.context.collection.objects.link(obj) 21 | 22 | # Build mesh using bmesh 23 | bm = bmesh.new() 24 | verts = [bm.verts.new(co) for co in points] 25 | bm.faces.new(verts) 26 | bm.to_mesh(mesh) 27 | bm.free() 28 | 29 | 30 | def main(): 31 | table_points = [ 32 | [-0.3222, -0.9099, 0.7420], 33 | [0.1253, -0.9193, 0.7352], 34 | [0.1353, -0.3779, 0.7300], 35 | [-0.3135, -0.3743, 0.7349], 36 | ] 37 | add_table_from_4_points(table_points) 38 | 39 | 40 | if __name__ == "__main__": 41 | main() 42 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/dataset/kit_opt.txt: -------------------------------------------------------------------------------- 1 | ------------ Options ------------- 2 | batch_size: 32 3 | checkpoints_dir: ./checkpoints 4 | dataset_name: kit 5 | decomp_name: Decomp_SP001_SM001_H512 6 | dim_att_vec: 512 7 | dim_dec_hidden: 1024 8 | dim_movement2_dec_hidden: 512 9 | dim_movement_dec_hidden: 512 10 | dim_movement_enc_hidden: 512 11 | dim_movement_latent: 512 12 | dim_msd_hidden: 512 13 | dim_pos_hidden: 1024 14 | dim_pri_hidden: 1024 15 | dim_seq_de_hidden: 512 16 | dim_seq_en_hidden: 512 17 | dim_text_hidden: 512 18 | dim_z: 128 19 | early_stop_count: 3 20 | estimator_mod: bigru 21 | eval_every_e: 5 22 | feat_bias: 5 23 | fixed_steps: 5 24 | gpu_id: 2 25 | input_z: False 26 | is_continue: True 27 | is_train: True 28 | lambda_fake: 10 29 | lambda_gan_l: 0.1 30 | lambda_gan_mt: 0.1 31 | lambda_gan_mv: 0.1 32 | lambda_kld: 0.005 33 | lambda_rec: 1 34 | lambda_rec_init: 1 35 | lambda_rec_mot: 1 36 | lambda_rec_mov: 1 37 | log_every: 50 38 | lr: 0.0002 39 | max_sub_epoch: 50 40 | max_text_len: 20 41 | n_layers_dec: 1 42 | n_layers_msd: 2 43 | n_layers_pos: 1 44 | n_layers_pri: 1 45 | n_layers_seq_de: 2 46 | n_layers_seq_en: 1 47 | name: Comp_v6_KLD005 48 | num_experts: 4 49 | save_every_e: 10 50 | save_latest: 500 51 | text_enc_mod: bigru 52 | tf_ratio: 0.4 53 | unit_length: 4 54 | -------------- End ---------------- 55 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/dataset/humanml_opt.txt: -------------------------------------------------------------------------------- 1 | ------------ Options ------------- 2 | batch_size: 32 3 | checkpoints_dir: ./checkpoints 4 | dataset_name: t2m 5 | decomp_name: Decomp_SP001_SM001_H512 6 | dim_att_vec: 512 7 | dim_dec_hidden: 1024 8 | dim_movement2_dec_hidden: 512 9 | dim_movement_dec_hidden: 512 10 | dim_movement_enc_hidden: 512 11 | dim_movement_latent: 512 12 | dim_msd_hidden: 512 13 | dim_pos_hidden: 1024 14 | dim_pri_hidden: 1024 15 | dim_seq_de_hidden: 512 16 | dim_seq_en_hidden: 512 17 | dim_text_hidden: 512 18 | dim_z: 128 19 | early_stop_count: 3 20 | estimator_mod: bigru 21 | eval_every_e: 5 22 | feat_bias: 5 23 | fixed_steps: 5 24 | gpu_id: 3 25 | input_z: False 26 | is_continue: True 27 | is_train: True 28 | lambda_fake: 10 29 | lambda_gan_l: 0.1 30 | lambda_gan_mt: 0.1 31 | lambda_gan_mv: 0.1 32 | lambda_kld: 0.01 33 | lambda_rec: 1 34 | lambda_rec_init: 1 35 | lambda_rec_mot: 1 36 | lambda_rec_mov: 1 37 | log_every: 50 38 | lr: 0.0002 39 | max_sub_epoch: 50 40 | max_text_len: 20 41 | n_layers_dec: 1 42 | n_layers_msd: 2 43 | n_layers_pos: 1 44 | n_layers_pri: 1 45 | n_layers_seq_de: 2 46 | n_layers_seq_en: 1 47 | name: Comp_v6_KLD01 48 | num_experts: 4 49 | save_every_e: 10 50 | save_latest: 500 51 | text_enc_mod: bigru 52 | tf_ratio: 0.4 53 | unit_length: 4 54 | -------------- End ---------------- 55 | -------------------------------------------------------------------------------- /hoidini/skeletons/smpl_24.py: -------------------------------------------------------------------------------- 1 | JOINTS_24 = [ # SMPL-24 Skeleton 24 joints 2 | "Pelvis", 3 | "LHip", 4 | "RHip", 5 | "Spine1", 6 | "LKnee", 7 | "RKnee", 8 | "Spine2", 9 | "LAnkle", 10 | "RAnkle", 11 | "Spine3", 12 | "LFoot", 13 | "RFoot", 14 | "Neck", 15 | "LCollar", 16 | "RCollar", 17 | "Head", 18 | "LShoulder", 19 | "RShoulder", 20 | "LElbow", 21 | "RElbow", 22 | "LWrist", 23 | "RWrist", 24 | "LHand", 25 | "RHand", 26 | ] 27 | BONES_24_NAMES = [ 28 | ("Pelvis", "LHip"), 29 | ("Pelvis", "RHip"), 30 | ("Pelvis", "Spine1"), 31 | ("LHip", "LKnee"), 32 | ("RHip", "RKnee"), 33 | ("Spine1", "Spine2"), 34 | ("LKnee", "LAnkle"), 35 | ("RKnee", "RAnkle"), 36 | ("Spine2", "Spine3"), 37 | ("LAnkle", "LFoot"), 38 | ("RAnkle", "RFoot"), 39 | ("Spine3", "Neck"), 40 | ("Spine3", "LCollar"), 41 | ("Spine3", "RCollar"), 42 | ("Neck", "Head"), 43 | ("LCollar", "LShoulder"), 44 | ("RCollar", "RShoulder"), 45 | ("LShoulder", "LElbow"), 46 | ("RShoulder", "RElbow"), 47 | ("LElbow", "LWrist"), 48 | ("RElbow", "RWrist"), 49 | ("LWrist", "LHand"), 50 | ("RWrist", "RHand"), 51 | ] 52 | 53 | BONES_24_INDS = [(JOINTS_24.index(e[0]), JOINTS_24.index(e[1])) for e in BONES_24_NAMES] 54 | -------------------------------------------------------------------------------- /hoidini/resource_paths.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | # Get the project root directory (where this file is located relative to hoidini/) 4 | PROJECT_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) 5 | 6 | # Default paths relative to the Hugging Face dataset clone 7 | GRAB_DATA_PATH = os.environ.get( 8 | "GRAB_DATA_PATH", 9 | os.path.join( 10 | PROJECT_ROOT, "hoidini_data", "datasets", "GRAB_RETARGETED_compressed" 11 | ), 12 | ) 13 | MANO_SMPLX_VERTEX_IDS_PATH = os.environ.get( 14 | "MANO_SMPLX_VERTEX_IDS_PATH", 15 | os.path.join(PROJECT_ROOT, "hoidini_data", "datasets", "MANO_SMPLX_vertex_ids.pkl"), 16 | ) 17 | MODEL_PATH = os.environ.get( 18 | "MODEL_PATH", 19 | os.path.join( 20 | PROJECT_ROOT, 21 | "hoidini_data", 22 | "models", 23 | "cphoi_05011024_c15p100_v0", 24 | "model000120000.pt", 25 | ), 26 | ) 27 | SMPL_MODELS_DATA = os.environ.get( 28 | "SMPL_MODELS_DATA", 29 | os.path.join(PROJECT_ROOT, "hoidini_data", "datasets", "smpl_models"), 30 | ) 31 | 32 | # Optional datasets (can be downloaded separately if needed) 33 | GRAB_ORIG_DATA_PATH = os.environ.get("GRAB_ORIG_DATA_PATH", "/path/to/GRAB/original") 34 | HUMANML3D_DATASET_PATH = os.environ.get("HUMANML3D_DATASET_PATH", "/path/to/HumanML3D") 35 | OMOMO_DATA_PATH = os.environ.get("OMOMO_DATA_PATH", "/path/to/OMOMO") 36 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/data_loaders/humanml/motion_loaders/dataset_motion_loader.py: -------------------------------------------------------------------------------- 1 | from t2m.data.dataset import Text2MotionDatasetV2, collate_fn 2 | from t2m.utils.word_vectorizer import WordVectorizer 3 | import numpy as np 4 | from os.path import join as pjoin 5 | from torch.utils.data import DataLoader 6 | from t2m.utils.get_opt import get_opt 7 | 8 | 9 | def get_dataset_motion_loader(opt_path, batch_size, device): 10 | opt = get_opt(opt_path, device) 11 | 12 | # Configurations of T2M dataset and KIT dataset is almost the same 13 | if opt.dataset_name == "t2m" or opt.dataset_name == "kit": 14 | print("Loading dataset %s ..." % opt.dataset_name) 15 | 16 | mean = np.load(pjoin(opt.meta_dir, "mean.npy")) 17 | std = np.load(pjoin(opt.meta_dir, "std.npy")) 18 | 19 | w_vectorizer = WordVectorizer("./glove", "our_vab") 20 | split_file = pjoin(opt.data_root, "test.txt") 21 | dataset = Text2MotionDatasetV2(opt, mean, std, split_file, w_vectorizer) 22 | dataloader = DataLoader( 23 | dataset, 24 | batch_size=batch_size, 25 | num_workers=4, 26 | drop_last=True, 27 | collate_fn=collate_fn, 28 | shuffle=True, 29 | ) 30 | else: 31 | raise KeyError("Dataset not Recognized !!") 32 | 33 | print("Ground Truth Dataset Loading Completed!!!") 34 | return dataloader, dataset 35 | -------------------------------------------------------------------------------- /hoidini/configs/sampling_cphoi_kitchen.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - sampling_cphoi.yaml 3 | - _self_ 4 | 5 | 6 | out_dir: /home/dcor/roeyron/trumans_utils/results/cphoi/kitchen_debug 7 | model_path: /home/dcor/roeyron/trumans_utils/src/Experiments/cphoi_05011024_c15p100_v0/model000120000.pt 8 | use_dno_object: true 9 | use_dno_body: true 10 | use_classifier_guidance: false 11 | 12 | batch_size_gen: 1 13 | 14 | result_phases_to_save: [final] # ,phase1,phase2,final] 15 | result_phases_to_animate: [final] # ,phase1,phase2,final] 16 | n_simplify_hands: 1500 17 | n_simplify_object: 1500 18 | 19 | dno_options_phase1: 20 | num_opt_steps: 120 21 | lr: 0.7 22 | lr_warm_up_steps: 20 23 | perturb_scale: 0.000001 24 | diff_penalty_scale: 0.0000001 25 | 26 | dno_loss_coefficients_phase1: # ignored 27 | above_table: 0.2 28 | specific_6dof: 0.2 29 | 30 | dno_options_phase2: 31 | num_opt_steps: 120 32 | lr: 0.7 33 | lr_warm_up_steps: 20 34 | perturb_scale: 0.00001 35 | diff_penalty_scale: 0.0000001 36 | decorrelate_scale: 10 37 | 38 | 39 | dno_loss_coefficients_phase2: 40 | contact: 0.95 41 | penetration: 0.5 42 | jitter_joints: 0.0001 43 | above_table: 0.2 44 | side_table: 0.2 45 | foot_skate: 0.5 46 | 47 | sampler_config: 48 | _target_: hoidini.cphoi.samplers.samplers_simple.KitchenSamplerSimple 49 | grab_seq_id: s3/lightbulb_screw_1 50 | surface_name: stove 51 | 52 | replace_cps_with_nearest_neighbors: false -------------------------------------------------------------------------------- /hoidini/optimize_latent/ol_utils.py: -------------------------------------------------------------------------------- 1 | from matplotlib import pyplot as plt 2 | from hoidini.optimize_latent.dno import DNO, DNOOptions 3 | from hoidini.optimize_latent.dno_loss_functions import ( 4 | DnoCondHandsIntersection, 5 | DnoCondVertsDist, 6 | ) 7 | import torch 8 | 9 | 10 | def get_static_criterion(smpl_fk, normalizer, n_joints, n_simplify_faces=700): 11 | loss_fn_lst = [ 12 | (0.2, DnoCondVertsDist(smpl_fk, normalizer, n_joints)), 13 | ( 14 | 1, 15 | DnoCondHandsIntersection( 16 | smpl_fk, normalizer, n_joints, n_simplify_faces=700 17 | ), 18 | ), 19 | ] 20 | 21 | def criterion(x, **model_kwargs): 22 | result = 0 23 | for alpha, loss_fn in loss_fn_lst: 24 | result += alpha * loss_fn(x, **model_kwargs) 25 | return result 26 | 27 | return criterion 28 | 29 | 30 | def plot_lr_scheduler(dno_opts: DNOOptions): 31 | dno = DNO(model=None, criterion=None, start_z=torch.ones(1), conf=dno_opts) 32 | lr_scheduler = dno.lr_scheduler 33 | lrs = [] 34 | steps = [] 35 | for step in range(dno.conf.num_opt_steps): 36 | lr_frac = 1 37 | if len(lr_scheduler) > 0: 38 | for scheduler in lr_scheduler: 39 | lr_frac *= scheduler(step) 40 | lr = dno.conf.lr * lr_frac 41 | 42 | lrs.append(lr) 43 | steps.append(step) 44 | plt.figure() 45 | plt.plot(steps, lrs) 46 | plt.show() 47 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/scripts/create_vis_model.py: -------------------------------------------------------------------------------- 1 | from lxml import etree 2 | from lxml.etree import XMLParser, parse, ElementTree, Element, SubElement 3 | from copy import deepcopy 4 | import argparse 5 | 6 | 7 | parser = argparse.ArgumentParser() 8 | parser.add_argument("--in-model", type=str, default="human36m_vis_single_v1") 9 | parser.add_argument("--out-model", type=str, default="human36m_vis_sample_single_v1") 10 | parser.add_argument("--num", type=int, default=10) 11 | parser.add_argument("--trans-range", type=int, default=(-1, -1)) 12 | 13 | args = parser.parse_args() 14 | 15 | in_file = "assets/mujoco_models/%s.xml" % args.in_model 16 | out_file = "assets/mujoco_models/%s.xml" % args.out_model 17 | parser = XMLParser(remove_blank_text=True) 18 | tree = parse(in_file, parser=parser) 19 | root = tree.getroot().find("worldbody") 20 | body = root.find("body") 21 | for i in range(1, args.num): 22 | new_body = deepcopy(body) 23 | if args.trans_range[0] <= i < args.trans_range[1]: 24 | new_body.attrib["childclass"] = "trans" 25 | new_body.attrib["name"] = "%d_%s" % (i, new_body.attrib["name"]) 26 | for node in new_body.findall(".//body"): 27 | node.attrib["name"] = "%d_%s" % (i, node.attrib["name"]) 28 | for node in new_body.findall(".//joint"): 29 | node.attrib["name"] = "%d_%s" % (i, node.attrib["name"]) 30 | root.append(new_body) 31 | if args.trans_range[0] == 0: 32 | body.attrib["childclass"] = "trans" 33 | 34 | tree.write(out_file, pretty_print=True) 35 | -------------------------------------------------------------------------------- /hoidini/closd/utils/hf_handler.py: -------------------------------------------------------------------------------- 1 | from huggingface_hub import hf_hub_download 2 | from huggingface_hub._snapshot_download import snapshot_download 3 | import os 4 | 5 | REPO_NAME = "guytevet/CLoSD" 6 | 7 | 8 | def get_dependencies(): 9 | dependencies_path = snapshot_download(repo_id=REPO_NAME) 10 | print("Data dependencies are cached at [{}]".format(dependencies_path)) 11 | link_all_checkpoints(dependencies_path) 12 | return dependencies_path 13 | 14 | 15 | def link_all_checkpoints(dependencies_path): 16 | link_checkpoints( 17 | os.path.join(dependencies_path, "checkpoints", "dip"), 18 | os.path.join("closd", "diffusion_planner", "save"), 19 | ) # DiP checkpoints 20 | link_checkpoints( 21 | os.path.join(dependencies_path, "evaluation"), 22 | os.path.join("closd", "diffusion_planner", "saved_motions"), 23 | ) # DiP checkpoints 24 | link_checkpoints( 25 | os.path.join(dependencies_path, "checkpoints", "closd"), 26 | os.path.join("output", "CLoSD"), 27 | ) # CLoSD checkpoints 28 | 29 | 30 | def link_checkpoints(src_dir, dst_dir): 31 | assert os.path.isdir(src_dir) 32 | os.makedirs(dst_dir, exist_ok=True) 33 | all_subdirs = [ 34 | subdir 35 | for subdir in os.listdir(src_dir) 36 | if os.path.isdir(os.path.join(src_dir, subdir)) 37 | ] 38 | for subdir in all_subdirs: 39 | if not os.path.exists(os.path.join(dst_dir, subdir)): 40 | os.symlink(os.path.join(src_dir, subdir), os.path.join(dst_dir, subdir)) 41 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/core/running_norm.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | 5 | class RunningNorm(nn.Module): 6 | """ 7 | y = (x-mean)/std 8 | using running estimates of mean,std 9 | """ 10 | 11 | def __init__(self, dim, demean=True, destd=True, clip=5.0): 12 | super().__init__() 13 | self.dim = dim 14 | self.demean = demean 15 | self.destd = destd 16 | self.clip = clip 17 | self.register_buffer("n", torch.tensor(0, dtype=torch.long)) 18 | self.register_buffer("mean", torch.zeros(dim)) 19 | self.register_buffer("var", torch.zeros(dim)) 20 | self.register_buffer("std", torch.zeros(dim)) 21 | 22 | def update(self, x): 23 | var_x, mean_x = torch.var_mean(x, dim=0, unbiased=False) 24 | m = x.shape[0] 25 | w = self.n.to(x.dtype) / (m + self.n).to(x.dtype) 26 | self.var[:] = ( 27 | w * self.var + (1 - w) * var_x + w * (1 - w) * (mean_x - self.mean).pow(2) 28 | ) 29 | self.mean[:] = w * self.mean + (1 - w) * mean_x 30 | self.std[:] = torch.sqrt(self.var) 31 | self.n += m 32 | 33 | def forward(self, x): 34 | if self.training: 35 | with torch.no_grad(): 36 | self.update(x) 37 | if self.n > 0: 38 | if self.demean: 39 | x = x - self.mean 40 | if self.destd: 41 | x = x / (self.std + 1e-8) 42 | if self.clip: 43 | x = torch.clamp(x, -self.clip, self.clip) 44 | return x 45 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/skeleton/tests/transfer_npy.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. 2 | # NVIDIA CORPORATION and its licensors retain all intellectual property 3 | # and proprietary rights in and to this software, related documentation 4 | # and any modifications thereto. Any use, reproduction, disclosure or 5 | # distribution of this software and related documentation without an express 6 | # license agreement from NVIDIA CORPORATION is strictly prohibited. 7 | 8 | import numpy as np 9 | from ...core import Tensor, SO3, Quaternion, Vector3D 10 | from ..skeleton3d import SkeletonTree, SkeletonState, SkeletonMotion 11 | 12 | tpose = np.load( 13 | "/home/serfcx/DL_Animation/rl_mimic/data/skeletons/flex_tpose.npy" 14 | ).item() 15 | 16 | local_rotation = SO3.from_numpy(tpose["local_rotation"], dtype="float32") 17 | root_translation = Vector3D.from_numpy(tpose["root_translation"], dtype="float32") 18 | skeleton_tree = tpose["skeleton_tree"] 19 | parent_indices = Tensor.from_numpy(skeleton_tree["parent_indices"], dtype="int32") 20 | local_translation = Vector3D.from_numpy( 21 | skeleton_tree["local_translation"], dtype="float32" 22 | ) 23 | node_names = skeleton_tree["node_names"] 24 | skeleton_tree = SkeletonTree(node_names, parent_indices, local_translation) 25 | skeleton_state = SkeletonState.from_rotation_and_root_translation( 26 | skeleton_tree=skeleton_tree, r=local_rotation, t=root_translation, is_local=True 27 | ) 28 | 29 | skeleton_state.to_file( 30 | "/home/serfcx/DL_Animation/rl_mimic/data/skeletons/flex_tpose_new.npy" 31 | ) 32 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/core/tensor_utils.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. 4 | # NVIDIA CORPORATION and its licensors retain all intellectual property 5 | # and proprietary rights in and to this software, related documentation 6 | # and any modifications thereto. Any use, reproduction, disclosure or 7 | # distribution of this software and related documentation without an express 8 | # license agreement from NVIDIA CORPORATION is strictly prohibited. 9 | 10 | from collections import OrderedDict 11 | from .backend import Serializable 12 | import torch 13 | 14 | 15 | class TensorUtils(Serializable): 16 | @classmethod 17 | def from_dict(cls, dict_repr, *args, **kwargs): 18 | """Read the object from an ordered dictionary 19 | 20 | :param dict_repr: the ordered dictionary that is used to construct the object 21 | :type dict_repr: OrderedDict 22 | :param kwargs: the arguments that need to be passed into from_dict() 23 | :type kwargs: additional arguments 24 | """ 25 | return torch.from_numpy(dict_repr["arr"].astype(dict_repr["context"]["dtype"])) 26 | 27 | def to_dict(self): 28 | """Construct an ordered dictionary from the object 29 | 30 | :rtype: OrderedDict 31 | """ 32 | return NotImplemented 33 | 34 | 35 | def tensor_to_dict(x): 36 | """Construct an ordered dictionary from the object 37 | 38 | :rtype: OrderedDict 39 | """ 40 | x_np = x.numpy() 41 | return {"arr": x_np, "context": {"dtype": x_np.dtype.name}} 42 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/README.md: -------------------------------------------------------------------------------- 1 | # poselib 2 | 3 | `poselib` is a library for loading, manipulating, and retargeting skeleton poses and motions. It is separated into three modules: `poselib.poselib.core` for basic data loading and tensor operations, `poselib.poselib.skeleton` for higher-level skeleton operations, and `poselib.poselib.visualization` for displaying skeleton poses. 4 | 5 | ## closd.utils.poselib.poselib.core 6 | - `poselib.poselib.core.rotation3d`: A set of Torch JIT functions for dealing with quaternions, transforms, and rotation/transformation matrices. 7 | - `quat_*` manipulate and create quaternions in [x, y, z, w] format (where w is the real component). 8 | - `transform_*` handle 7D transforms in [quat, pos] format. 9 | - `rot_matrix_*` handle 3x3 rotation matrices. 10 | - `euclidean_*` handle 4x4 Euclidean transformation matrices. 11 | - `poselib.poselib.core.tensor_utils`: Provides loading and saving functions for PyTorch tensors. 12 | 13 | ## closd.utils.poselib.poselib.skeleton 14 | - `poselib.poselib.skeleton.skeleton3d`: Utilities for loading and manipulating skeleton poses, and retargeting poses to different skeletons. 15 | - `SkeletonTree` is a class that stores a skeleton as a tree structure. 16 | - `SkeletonState` describes the static state of a skeleton, and provides both global and local joint angles. 17 | - `SkeletonMotion` describes a time-series of skeleton states and provides utilities for computing joint velocities. 18 | 19 | ## closd.utils.poselib.poselib.visualization 20 | - `poselib.poselib.visualization.common`: Functions used for visualizing skeletons interactively in `matplotlib`. 21 | -------------------------------------------------------------------------------- /hoidini/closd/utils/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2018-2023, NVIDIA Corporation 2 | # All rights reserved. 3 | # 4 | # Redistribution and use in source and binary forms, with or without 5 | # modification, are permitted provided that the following conditions are met: 6 | # 7 | # 1. Redistributions of source code must retain the above copyright notice, this 8 | # list of conditions and the following disclaimer. 9 | # 10 | # 2. Redistributions in binary form must reproduce the above copyright notice, 11 | # this list of conditions and the following disclaimer in the documentation 12 | # and/or other materials provided with the distribution. 13 | # 14 | # 3. Neither the name of the copyright holder nor the names of its 15 | # contributors may be used to endorse or promote products derived from 16 | # this software without specific prior written permission. 17 | # 18 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 22 | # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 23 | # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 25 | # CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 26 | # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/utils/PYTORCH3D_LICENSE: -------------------------------------------------------------------------------- 1 | BSD License 2 | 3 | For PyTorch3D software 4 | 5 | Copyright (c) Facebook, Inc. and its affiliates. All rights reserved. 6 | 7 | Redistribution and use in source and binary forms, with or without modification, 8 | are permitted provided that the following conditions are met: 9 | 10 | * Redistributions of source code must retain the above copyright notice, this 11 | list of conditions and the following disclaimer. 12 | 13 | * Redistributions in binary form must reproduce the above copyright notice, 14 | this list of conditions and the following disclaimer in the documentation 15 | and/or other materials provided with the distribution. 16 | 17 | * Neither the name Facebook nor the names of its contributors may be used to 18 | endorse or promote products derived from this software without specific 19 | prior written permission. 20 | 21 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 22 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 23 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 24 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR 25 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 26 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 27 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 28 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 29 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 30 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/models/erd_net.py: -------------------------------------------------------------------------------- 1 | from hoidini.closd.utils.smpllib.khrylib.utils.torch import * 2 | from torch import nn 3 | from hoidini.closd.utils.smpllib.khrylib.models.rnn import RNN 4 | from hoidini.closd.utils.smpllib.khrylib.models.mlp import MLP 5 | 6 | 7 | class ERDNet(nn.Module): 8 | 9 | def __init__(self, state_dim): 10 | super().__init__() 11 | self.state_dim = state_dim 12 | self.encoder_mlp = MLP(state_dim, (500,), "relu") 13 | self.encoder_linear = nn.Linear(500, 500) 14 | self.lstm1 = RNN(500, 1000, "lstm") 15 | self.lstm2 = RNN(1000, 1000, "lstm") 16 | self.decoder_mlp = MLP(1000, (500, 100), "relu") 17 | self.decoder_linear = nn.Linear(100, state_dim) 18 | self.mode = "batch" 19 | 20 | def initialize(self, mode): 21 | self.mode = mode 22 | self.lstm1.set_mode(mode) 23 | self.lstm2.set_mode(mode) 24 | self.lstm1.initialize() 25 | self.lstm2.initialize() 26 | 27 | def forward(self, x): 28 | if self.mode == "batch": 29 | batch_size = x.shape[1] 30 | x = x.view(-1, x.shape[-1]) 31 | x = self.encoder_mlp(x) 32 | x = self.encoder_linear(x) 33 | if self.mode == "batch": 34 | x = x.view(-1, batch_size, x.shape[-1]) 35 | x = self.lstm1(x) 36 | x = self.lstm2(x) 37 | if self.mode == "batch": 38 | x = x.view(-1, x.shape[-1]) 39 | x = self.decoder_mlp(x) 40 | x = self.decoder_linear(x) 41 | return x 42 | 43 | 44 | if __name__ == "__main__": 45 | net = ERDNet(64) 46 | input = ones(32, 3, 64) 47 | out = net(input) 48 | print(out.shape) 49 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/core/distributions.py: -------------------------------------------------------------------------------- 1 | import torch 2 | from torch.distributions import Normal 3 | from torch.distributions import Categorical as TorchCategorical 4 | 5 | 6 | class DiagGaussian(Normal): 7 | 8 | def __init__(self, loc, scale): 9 | super().__init__(loc, scale) 10 | 11 | def kl(self): 12 | loc1 = self.loc 13 | scale1 = self.scale 14 | log_scale1 = self.scale.log() 15 | loc0 = self.loc.detach() 16 | scale0 = self.scale.detach() 17 | log_scale0 = log_scale1.detach() 18 | kl = ( 19 | log_scale1 20 | - log_scale0 21 | + (scale0.pow(2) + (loc0 - loc1).pow(2)) / (2.0 * scale1.pow(2)) 22 | - 0.5 23 | ) 24 | return kl.sum(1, keepdim=True) 25 | 26 | def log_prob(self, value): 27 | return super().log_prob(value).sum(1, keepdim=True) 28 | 29 | def mean_sample(self): 30 | return self.loc 31 | 32 | 33 | class Categorical(TorchCategorical): 34 | 35 | def __init__(self, probs=None, logits=None): 36 | super().__init__(probs, logits) 37 | 38 | def kl(self): 39 | loc1 = self.loc 40 | scale1 = self.scale 41 | log_scale1 = self.scale.log() 42 | loc0 = self.loc.detach() 43 | scale0 = self.scale.detach() 44 | log_scale0 = log_scale1.detach() 45 | kl = ( 46 | log_scale1 47 | - log_scale0 48 | + (scale0.pow(2) + (loc0 - loc1).pow(2)) / (2.0 * scale1.pow(2)) 49 | - 0.5 50 | ) 51 | return kl.sum(1, keepdim=True) 52 | 53 | def log_prob(self, value): 54 | return super().log_prob(value).unsqueeze(1) 55 | 56 | def mean_sample(self): 57 | return self.probs.argmax(dim=1) 58 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/utils/zfilter.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | # from https://github.com/joschu/modular_rl 4 | # http://www.johndcook.com/blog/standard_deviation/ 5 | 6 | 7 | class RunningStat(object): 8 | def __init__(self, shape): 9 | self._n = 0 10 | self._M = np.zeros(shape) 11 | self._S = np.zeros(shape) 12 | 13 | def push(self, x): 14 | x = np.asarray(x) 15 | assert x.shape == self._M.shape 16 | self._n += 1 17 | if self._n == 1: 18 | self._M[...] = x 19 | else: 20 | oldM = self._M.copy() 21 | self._M[...] = oldM + (x - oldM) / self._n 22 | self._S[...] = self._S + (x - oldM) * (x - self._M) 23 | 24 | @property 25 | def n(self): 26 | return self._n 27 | 28 | @property 29 | def mean(self): 30 | return self._M 31 | 32 | @property 33 | def var(self): 34 | return self._S / (self._n - 1) if self._n > 1 else np.square(self._M) 35 | 36 | @property 37 | def std(self): 38 | return np.sqrt(self.var) 39 | 40 | @property 41 | def shape(self): 42 | return self._M.shape 43 | 44 | 45 | class ZFilter: 46 | """ 47 | y = (x-mean)/std 48 | using running estimates of mean,std 49 | """ 50 | 51 | def __init__(self, shape, demean=True, destd=True, clip=10.0): 52 | self.demean = demean 53 | self.destd = destd 54 | self.clip = clip 55 | 56 | self.rs = RunningStat(shape) 57 | 58 | def __call__(self, x, update=True): 59 | if update: 60 | self.rs.push(x) 61 | if self.demean: 62 | x = x - self.rs.mean 63 | if self.destd: 64 | x = x / (self.rs.std + 1e-8) 65 | if self.clip: 66 | x = np.clip(x, -self.clip, self.clip) 67 | return x 68 | 69 | def set_mean_std(self, mean, std, n): 70 | self.rs._n = n 71 | self.rs._M[...] = mean 72 | self.rs._S[...] = std 73 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/core/policy_gaussian.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | from hoidini.closd.utils.smpllib.khrylib.rl.core.distributions import DiagGaussian 3 | from hoidini.closd.utils.smpllib.khrylib.rl.core.policy import Policy 4 | from hoidini.closd.utils.smpllib.utils.math_utils import * 5 | from hoidini.closd.utils.smpllib.khrylib.models.mlp import MLP 6 | 7 | 8 | class PolicyGaussian(Policy): 9 | def __init__(self, cfg, action_dim, state_dim, net_out_dim=None): 10 | super().__init__() 11 | self.type = "gaussian" 12 | policy_hsize = cfg.policy_hsize 13 | policy_htype = cfg.policy_htype 14 | fix_std = cfg.fix_std 15 | log_std = cfg.log_std 16 | self.net = net = MLP(state_dim, policy_hsize, policy_htype) 17 | if net_out_dim is None: 18 | net_out_dim = net.out_dim 19 | self.action_mean = nn.Linear(net_out_dim, action_dim) 20 | self.action_mean.weight.data.mul_(0.1) 21 | self.action_mean.bias.data.mul_(0.0) 22 | self.action_log_std = nn.Parameter( 23 | torch.ones(1, action_dim) * log_std, requires_grad=not fix_std 24 | ) 25 | 26 | def forward(self, x): 27 | x = self.net(x) 28 | action_mean = self.action_mean(x) 29 | action_log_std = self.action_log_std.expand_as(action_mean) 30 | action_std = torch.exp(action_log_std) 31 | return DiagGaussian(action_mean, action_std) 32 | 33 | def get_fim(self, x): 34 | dist = self.forward(x) 35 | cov_inv = self.action_log_std.exp().pow(-2).squeeze(0).repeat(x.size(0)) 36 | param_count = 0 37 | std_index = 0 38 | id = 0 39 | for name, param in self.named_parameters(): 40 | if name == "action_log_std": 41 | std_id = id 42 | std_index = param_count 43 | param_count += param.view(-1).shape[0] 44 | id += 1 45 | return cov_inv.detach(), dist.loc, {"std_id": std_id, "std_index": std_index} 46 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/models/mobile_net.py: -------------------------------------------------------------------------------- 1 | from torch import nn 2 | from hoidini.closd.utils.smpllib.khrylib.utils.torch import * 3 | 4 | 5 | class MobileNet(nn.Module): 6 | def __init__(self, out_dim): 7 | super().__init__() 8 | self.out_dim = out_dim 9 | 10 | def conv_bn(inp, oup, stride): 11 | return nn.Sequential( 12 | nn.Conv2d(inp, oup, 3, stride, 1, bias=False), 13 | nn.BatchNorm2d(oup), 14 | nn.ReLU(inplace=True), 15 | ) 16 | 17 | def conv_dw(inp, oup, stride): 18 | return nn.Sequential( 19 | nn.Conv2d(inp, inp, 3, stride, 1, groups=inp, bias=False), 20 | nn.BatchNorm2d(inp), 21 | nn.ReLU(inplace=True), 22 | nn.Conv2d(inp, oup, 1, 1, 0, bias=False), 23 | nn.BatchNorm2d(oup), 24 | nn.ReLU(inplace=True), 25 | ) 26 | 27 | self.model = nn.Sequential( 28 | conv_bn(3, 32, 2), 29 | conv_dw(32, 64, 1), 30 | conv_dw(64, 128, 2), 31 | conv_dw(128, 128, 1), 32 | conv_dw(128, 256, 2), 33 | conv_dw(256, 256, 1), 34 | conv_dw(256, 512, 2), 35 | conv_dw(512, 512, 1), 36 | conv_dw(512, 512, 1), 37 | conv_dw(512, 512, 1), 38 | conv_dw(512, 512, 1), 39 | conv_dw(512, 512, 1), 40 | conv_dw(512, 1024, 2), 41 | conv_dw(1024, 1024, 1), 42 | nn.AvgPool2d(7), 43 | ) 44 | self.fc = nn.Linear(1024, out_dim) 45 | 46 | def forward(self, x): 47 | x = self.model(x) 48 | x = x.view(-1, 1024) 49 | x = self.fc(x) 50 | return x 51 | 52 | 53 | if __name__ == "__main__": 54 | import time 55 | 56 | torch.set_grad_enabled(False) 57 | net = MobileNet(128) 58 | input = ones(1, 3, 224, 224) 59 | for i in range(10): 60 | t0 = time.time() 61 | out = net(input) 62 | print(time.time() - t0) 63 | print(out.shape) 64 | -------------------------------------------------------------------------------- /hoidini/configs/0_base_config_comparison.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - _self_ 3 | 4 | _target_: hoidini.cphoi.CPHOIInferenceConfig 5 | 6 | model_path: /home/dcor/roeyron/trumans_utils/src/Experiments/cphoi_05011024_c15p100_v0/model000120000.pt 7 | out_dir: null 8 | # grab_dataset_path: 9 | batch_size_gen: 2 10 | device: 0 11 | sampler_config: 12 | _target_: hoidini.cphoi.samplers.samplers.SpecificSampler 13 | grab_seq_ids: null 14 | 15 | 16 | seed: null 17 | # autoregressive_include_prefix: 18 | guidance_param: 3.0 19 | n_simplify_object: 3000 20 | th_contact_force: 0.4 21 | n_simplify_hands: 1100 22 | 23 | dno_options_phase1: 24 | _target_: hoidini.optimize_latent.dno.DNOOptions 25 | num_opt_steps: 350 26 | lr: 0.8 27 | lr_warm_up_steps: 20 28 | perturb_scale: 0.000001 29 | diff_penalty_scale: 1e-06 30 | 31 | 32 | dno_loss_coefficients_phase1: 33 | keep_object_static: 1.2 34 | above_table: 1.2 35 | similar_6dof_1: 0.5 36 | similar_6dof_2: 0.5 37 | 38 | dno_options_phase2: 39 | _target_: hoidini.optimize_latent.dno.DNOOptions 40 | num_opt_steps: 350 41 | lr: 0.8 42 | lr_warm_up_steps: 20 43 | perturb_scale: 0.00001 44 | diff_penalty_scale: 1e-06 45 | decorrelate_scale: 10 46 | 47 | 48 | dno_loss_coefficients_phase2: 49 | contact: 0.95 50 | penetration: 0.15 51 | jitter_joints: 0.00008 52 | above_table: 0.2 53 | side_table: 0.4 54 | foot_skate: 0.7 55 | 56 | use_dno_object: true 57 | use_dno_body: true 58 | anim_setup: MESH_ALL 59 | result_phases_to_animate: [final] 60 | result_phases_to_save: [final] 61 | 62 | anim_save_every_n_gens: null 63 | 64 | 65 | use_classifier_guidance: false 66 | classifier_guidance_n_steps: null 67 | classifier_guidance_lr_factor: null 68 | 69 | # train: 70 | 71 | # Extra keys not in CphoiInferenceConfig, kept at the end in their current order 72 | output_dir: null 73 | n_frames: null 74 | grab_dataset_path: /home/dcor/roeyron/trumans_utils/DATASETS/DATA_GRAB_RETARGETED 75 | autoregressive_include_prefix: true 76 | save_result_pickle: true 77 | 78 | 79 | replace_cps_with_nearest_neighbors: false -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/utils/mujoco.py: -------------------------------------------------------------------------------- 1 | from hoidini.closd.utils.smpllib.utils.math_utils import * 2 | 3 | 4 | def get_body_qveladdr(model): 5 | body_qposaddr = dict() 6 | for i, body_name in enumerate(model.body_names): 7 | start_joint = model.body_jntadr[i] 8 | if start_joint < 0: 9 | continue 10 | end_joint = start_joint + model.body_jntnum[i] 11 | start_qposaddr = model.jnt_dofadr[start_joint] 12 | if end_joint < len(model.jnt_dofadr): 13 | end_qposaddr = model.jnt_dofadr[end_joint] 14 | else: 15 | end_qposaddr = model.nv 16 | body_qposaddr[body_name] = (start_qposaddr, end_qposaddr) 17 | return body_qposaddr 18 | 19 | 20 | def get_body_qposaddr(model): 21 | body_qposaddr = dict() 22 | for i, body_name in enumerate(model.body_names): 23 | start_joint = model.body_jntadr[i] 24 | if start_joint < 0: 25 | continue 26 | end_joint = start_joint + model.body_jntnum[i] 27 | start_qposaddr = model.jnt_qposadr[start_joint] 28 | if end_joint < len(model.jnt_qposadr): 29 | end_qposaddr = model.jnt_qposadr[end_joint] 30 | else: 31 | end_qposaddr = model.nq 32 | body_qposaddr[body_name] = (start_qposaddr, end_qposaddr) 33 | return body_qposaddr 34 | 35 | 36 | def align_human_state(qpos, qvel, ref_qpos): 37 | qpos[:2] = ref_qpos[:2] 38 | hq = get_heading_q(ref_qpos[3:7]) 39 | qpos[3:7] = quaternion_multiply(hq, qpos[3:7]) 40 | qvel[:3] = quat_mul_vec(hq, qvel[:3]) 41 | 42 | 43 | def get_traj_pos(orig_traj): 44 | traj_pos = orig_traj[:, 2:].copy() 45 | for i in range(traj_pos.shape[0]): 46 | traj_pos[i, 1:5] = de_heading(traj_pos[i, 1:5]) 47 | return traj_pos 48 | 49 | 50 | def get_traj_vel(orig_traj, dt): 51 | traj_vel = [] 52 | for i in range(orig_traj.shape[0] - 1): 53 | vel = get_qvel_fd(orig_traj[i, :], orig_traj[i + 1, :], dt, "heading") 54 | traj_vel.append(vel) 55 | traj_vel.append(traj_vel[-1].copy()) 56 | traj_vel = np.vstack(traj_vel) 57 | return traj_vel 58 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/mocap/vis_joint_range.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import os 3 | import sys 4 | 5 | 6 | from hoidini.closd.utils.smpllib.khrylib.utils import * 7 | from mujoco_py import load_model_from_path, MjSim 8 | from hoidini.closd.utils.smpllib.khrylib.rl.envs.common.mjviewer import MjViewer 9 | 10 | 11 | parser = argparse.ArgumentParser() 12 | parser.add_argument("--model", default="assets/mujoco_models/human36m_orig.xml") 13 | args = parser.parse_args() 14 | 15 | model = load_model_from_path(args.model) 16 | sim = MjSim(model) 17 | viewer = MjViewer(sim) 18 | 19 | jind = -1 20 | jang = 30.0 21 | 22 | 23 | def key_callback(key, action, mods): 24 | global jind, jang 25 | 26 | if action != glfw.RELEASE: 27 | return False 28 | elif key == glfw.KEY_LEFT: 29 | jind = max(jind - 1, -1) 30 | print( 31 | "{} {} {}".format( 32 | model.joint_names[jind + 1] if jind >= 0 else "rest", jind, jang 33 | ) 34 | ) 35 | return True 36 | elif key == glfw.KEY_RIGHT: 37 | jind = min(jind + 1, len(model.joint_names) - 2) 38 | print( 39 | "{} {} {}".format( 40 | model.joint_names[jind + 1] if jind >= 0 else "rest", jind, jang 41 | ) 42 | ) 43 | return True 44 | elif key == glfw.KEY_UP: 45 | jang += 5.0 46 | print( 47 | "{} {} {}".format( 48 | model.joint_names[jind + 1] if jind >= 0 else "rest", jind, jang 49 | ) 50 | ) 51 | return True 52 | elif key == glfw.KEY_DOWN: 53 | jang -= 5.0 54 | print( 55 | "{} {} {}".format( 56 | model.joint_names[jind + 1] if jind >= 0 else "rest", jind, jang 57 | ) 58 | ) 59 | return True 60 | return False 61 | 62 | 63 | viewer._hide_overlay = True 64 | viewer.custom_key_callback = key_callback 65 | while True: 66 | sim.data.qpos[:] = 0.0 67 | sim.data.qpos[2] = 1.0 68 | if jind >= 0: 69 | sim.data.qpos[7 + jind] = math.radians(jang) 70 | sim.forward() 71 | viewer.render() 72 | -------------------------------------------------------------------------------- /hoidini/configs/0_base_config.yaml: -------------------------------------------------------------------------------- 1 | defaults: 2 | - _self_ 3 | 4 | _target_: hoidini.cphoi.CPHOIInferenceConfig 5 | 6 | seed: null 7 | model_path: hoidini_data/models/cphoi_05011024_c15p100_v0/model000120000.pt 8 | out_dir: null 9 | # grab_dataset_path: 10 | batch_size_gen: 2 11 | device: 0 12 | sampler_config: 13 | _target_: hoidini.cphoi.samplers.samplers.TestSampler 14 | n_samples: -1 15 | split_name: test 16 | seed: 1 17 | n_frames: null 18 | 19 | # autoregressive_include_prefix: 20 | guidance_param: 3.0 21 | n_simplify_object: 3000 22 | th_contact_force: 0.4 23 | n_simplify_hands: 1100 24 | 25 | one_phase_experiment: false 26 | 27 | dno_options_phase1: 28 | _target_: hoidini.optimize_latent.dno.DNOOptions 29 | num_opt_steps: 350 30 | lr: 0.9 31 | lr_warm_up_steps: 20 32 | perturb_scale: 0.000001 33 | diff_penalty_scale: 1e-06 34 | 35 | 36 | dno_loss_coefficients_phase1: 37 | keep_object_static: 0.9 38 | above_table: 1.2 39 | similar_6dof_1: 0.5 40 | similar_6dof_2: 0.5 41 | 42 | dno_options_phase2: 43 | _target_: hoidini.optimize_latent.dno.DNOOptions 44 | num_opt_steps: 350 45 | lr: 0.9 46 | lr_warm_up_steps: 20 47 | perturb_scale: 0.00001 48 | diff_penalty_scale: 1e-06 49 | decorrelate_scale: 10 50 | 51 | 52 | dno_loss_coefficients_phase2: 53 | contact: 0.95 54 | penetration: 0.05 55 | jitter_joints: 0.00008 56 | above_table: 0.2 57 | side_table: 0.2 58 | foot_skate: 0.5 59 | 60 | use_dno_object: true 61 | use_dno_body: true 62 | anim_setup: NO_MESH 63 | result_phases_to_animate: [final] 64 | result_phases_to_save: [final] 65 | 66 | anim_save_every_n_gens: 7 67 | 68 | 69 | use_classifier_guidance: false 70 | classifier_guidance_n_steps: null 71 | classifier_guidance_lr_factor: null 72 | 73 | # train: 74 | 75 | # Extra keys not in CphoiInferenceConfig, kept at the end in their current order 76 | output_dir: null 77 | n_frames: null 78 | grab_dataset_path: hoidini_data/datasets/GRAB_RETARGETED_compressed 79 | autoregressive_include_prefix: true 80 | save_result_pickle: true 81 | store_loss_data: false 82 | 83 | 84 | replace_cps_with_nearest_neighbors: false -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/models/mlp.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | import torch 3 | 4 | 5 | class MLP(nn.Module): 6 | def __init__(self, input_dim, hidden_dims=(128, 128), activation="tanh"): 7 | super().__init__() 8 | if activation == "tanh": 9 | self.activation = torch.tanh 10 | elif activation == "relu": 11 | self.activation = torch.relu 12 | elif activation == "sigmoid": 13 | self.activation = torch.sigmoid 14 | elif activation == "gelu": 15 | self.activation = torch.nn.GELU() 16 | 17 | self.out_dim = hidden_dims[-1] 18 | self.affine_layers = nn.ModuleList() 19 | last_dim = input_dim 20 | for nh in hidden_dims: 21 | self.affine_layers.append(nn.Linear(last_dim, nh)) 22 | last_dim = nh 23 | 24 | def forward(self, x): 25 | for affine in self.affine_layers: 26 | x = self.activation(affine(x)) 27 | return x 28 | 29 | 30 | class MLPWithInputSkips(torch.nn.Module): 31 | def __init__( 32 | self, 33 | n_layers: int, 34 | input_dim: int, 35 | output_dim: int, 36 | skip_dim: int, 37 | hidden_dim: int, 38 | input_skips, 39 | ): 40 | super().__init__() 41 | 42 | layers = [] 43 | 44 | for layeri in range(n_layers): 45 | if layeri == 0: 46 | dimin = input_dim 47 | dimout = hidden_dim 48 | elif layeri in input_skips: 49 | dimin = hidden_dim + skip_dim 50 | dimout = hidden_dim 51 | else: 52 | dimin = hidden_dim 53 | dimout = hidden_dim 54 | 55 | linear = torch.nn.Linear(dimin, dimout) 56 | layers.append(torch.nn.Sequential(linear, torch.nn.ReLU(True))) 57 | 58 | self.mlp = torch.nn.ModuleList(layers) 59 | self._input_skips = set(input_skips) 60 | 61 | def forward(self, x: torch.Tensor, z: torch.Tensor) -> torch.Tensor: 62 | y = x 63 | 64 | for li, layer in enumerate(self.mlp): 65 | if li in self._input_skips: 66 | y = torch.cat((y, z), dim=-1) 67 | 68 | y = layer(y) 69 | 70 | return y 71 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/utils/geom.py: -------------------------------------------------------------------------------- 1 | from vtk import ( 2 | vtkQuadricDecimation, 3 | vtkPolyData, 4 | vtkSTLReader, 5 | vtkSTLWriter, 6 | vtkTransform, 7 | vtkCenterOfMass, 8 | vtkTransformPolyDataFilter, 9 | ) 10 | 11 | 12 | def quadric_mesh_decimation(fname, reduction_rate, verbose=False): 13 | reader = vtkSTLReader() 14 | reader.SetFileName(fname) 15 | reader.Update() 16 | inputPoly = reader.GetOutput() 17 | 18 | decimate = vtkQuadricDecimation() 19 | decimate.SetInputData(inputPoly) 20 | decimate.SetTargetReduction(reduction_rate) 21 | decimate.Update() 22 | decimatedPoly = vtkPolyData() 23 | decimatedPoly.ShallowCopy(decimate.GetOutput()) 24 | 25 | if verbose: 26 | print( 27 | f"Mesh Decimation: (points, faces) goes from ({inputPoly.GetNumberOfPoints(), inputPoly.GetNumberOfPolys()}) " 28 | f"to ({decimatedPoly.GetNumberOfPoints(), decimatedPoly.GetNumberOfPolys()})" 29 | ) 30 | 31 | stlWriter = vtkSTLWriter() 32 | stlWriter.SetFileName(fname) 33 | stlWriter.SetFileTypeToBinary() 34 | stlWriter.SetInputData(decimatedPoly) 35 | stlWriter.Write() 36 | 37 | 38 | def center_scale_mesh(fname, scale): 39 | reader = vtkSTLReader() 40 | reader.SetFileName(fname) 41 | reader.Update() 42 | inputPoly = reader.GetOutputPort() 43 | 44 | centerOfMassFilter = vtkCenterOfMass() 45 | centerOfMassFilter.SetInputConnection(inputPoly) 46 | centerOfMassFilter.SetUseScalarsAsWeights(False) 47 | centerOfMassFilter.Update() 48 | center = centerOfMassFilter.GetCenter() 49 | 50 | transform = vtkTransform() 51 | transform.PostMultiply() 52 | transform.Translate(-center[0], -center[1], -center[2]) 53 | transform.Scale(scale, scale, scale) 54 | transform.Translate(center[0], center[1], center[2]) 55 | transform.Update() 56 | 57 | transformFilter = vtkTransformPolyDataFilter() 58 | transformFilter.SetInputConnection(inputPoly) 59 | transformFilter.SetTransform(transform) 60 | transformFilter.Update() 61 | 62 | stlWriter = vtkSTLWriter() 63 | stlWriter.SetFileName(fname) 64 | stlWriter.SetFileTypeToBinary() 65 | stlWriter.SetInputConnection(transformFilter.GetOutputPort()) 66 | stlWriter.Write() 67 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/utils/dist_util.py: -------------------------------------------------------------------------------- 1 | """ 2 | Helpers for distributed training. 3 | """ 4 | 5 | import socket 6 | 7 | import torch as th 8 | import torch.distributed as dist 9 | 10 | # Change this to reflect your cluster layout. 11 | # The GPU for a given rank is (rank % GPUS_PER_NODE). 12 | GPUS_PER_NODE = 8 13 | 14 | SETUP_RETRY_COUNT = 3 15 | 16 | used_device = 0 17 | 18 | 19 | def setup_dist(device=0): 20 | """ 21 | Setup a distributed process group. 22 | """ 23 | global used_device 24 | used_device = device 25 | if dist.is_initialized(): 26 | return 27 | # os.environ["CUDA_VISIBLE_DEVICES"] = str(device) # f"{MPI.COMM_WORLD.Get_rank() % GPUS_PER_NODE}" 28 | 29 | # comm = MPI.COMM_WORLD 30 | # backend = "gloo" if not th.cuda.is_available() else "nccl" 31 | 32 | # if backend == "gloo": 33 | # hostname = "localhost" 34 | # else: 35 | # hostname = socket.gethostbyname(socket.getfqdn()) 36 | # os.environ["MASTER_ADDR"] = comm.bcast(hostname, root=0) 37 | # os.environ["RANK"] = str(comm.rank) 38 | # os.environ["WORLD_SIZE"] = str(comm.size) 39 | 40 | # port = comm.bcast(_find_free_port(), root=used_device) 41 | # os.environ["MASTER_PORT"] = str(port) 42 | # dist.init_process_group(backend=backend, init_method="env://") 43 | 44 | 45 | def dev(): 46 | """ 47 | Get the device to use for torch.distributed. 48 | """ 49 | global used_device 50 | if th.cuda.is_available() and used_device >= 0: 51 | return th.device(f"cuda:{used_device}") 52 | return th.device("cpu") 53 | 54 | 55 | def load_state_dict(path, **kwargs): 56 | """ 57 | Load a PyTorch file without redundant fetches across MPI ranks. 58 | """ 59 | return th.load(path, **kwargs) 60 | 61 | 62 | def sync_params(params): 63 | """ 64 | Synchronize a sequence of Tensors across ranks from rank 0. 65 | """ 66 | for p in params: 67 | with th.no_grad(): 68 | dist.broadcast(p, 0) 69 | 70 | 71 | def _find_free_port(): 72 | try: 73 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 74 | s.bind(("", 0)) 75 | s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) 76 | return s.getsockname()[1] 77 | finally: 78 | s.close() 79 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/envs/visual/humanoid_vis.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import mujoco_py 3 | 4 | from hoidini.closd.utils.smpllib.khrylib.rl.envs.common import mujoco_env 5 | 6 | 7 | class HumanoidVisEnv(mujoco_env.MujocoEnv): 8 | def __init__(self, vis_model_file, nframes=6, focus=True): 9 | mujoco_env.MujocoEnv.__init__(self, vis_model_file, nframes) 10 | 11 | self.set_cam_first = set() 12 | self.focus = focus 13 | 14 | def step(self, a): 15 | return np.zeros((10, 1)), 0, False, dict() 16 | 17 | def reset_model(self): 18 | c = 0 19 | self.set_state( 20 | self.np_random.uniform(low=-c, high=c, size=self.model.nq), 21 | self.np_random.uniform(low=-c, high=c, size=self.model.nv), 22 | ) 23 | return None 24 | 25 | def sim_forward(self): 26 | self.sim.forward() 27 | 28 | def set_video_path( 29 | self, image_path="/tmp/image_%07d.png", video_path="/tmp/video_%07d.mp4" 30 | ): 31 | self.viewer._image_path = image_path 32 | self.viewer._video_path = video_path 33 | 34 | def viewer_setup(self, mode): 35 | self.viewer.cam.trackbodyid = 1 36 | if self.focus: 37 | self.viewer.cam.lookat[:2] = self.data.qpos[:2] 38 | self.viewer.cam.lookat[2] = 0.8 39 | if mode not in self.set_cam_first: 40 | self.viewer.video_fps = 30 41 | self.viewer.frame_skip = self.frame_skip 42 | self.viewer.cam.distance = self.model.stat.extent * 1.5 43 | self.viewer.cam.elevation = -10 44 | self.viewer.cam.azimuth = 45 45 | self.set_cam_first.add(mode) 46 | 47 | def reload_sim_model(self, xml_str): 48 | del self.sim 49 | del self.model 50 | del self.data 51 | del self.viewer 52 | del self._viewers 53 | self.model = mujoco_py.load_model_from_xml(xml_str) 54 | self.sim = mujoco_py.MjSim(self.model) 55 | self.data = self.sim.data 56 | self.init_qpos = self.sim.data.qpos.copy() 57 | self.init_qvel = self.sim.data.qvel.copy() 58 | self.viewer = None 59 | self._viewers = {} 60 | self._get_viewer("human")._hide_overlay = True 61 | self.reset() 62 | print("Reloading Vis Sim") 63 | -------------------------------------------------------------------------------- /hoidini/geometry3d/plot_mesh.py: -------------------------------------------------------------------------------- 1 | import open3d as o3d 2 | import numpy as np 3 | import plotly.graph_objects as go 4 | 5 | 6 | def plot_mesh(v=None, f=None, mesh: o3d.geometry.TriangleMesh = None): 7 | # Suppose we already have: 8 | # mesh_in.vertices = o3d.utility.Vector3dVector(verts) 9 | # mesh_in.triangles = o3d.utility.Vector3iVector(faces) 10 | 11 | if mesh is not None: 12 | v = np.asarray(mesh.vertices) 13 | f = np.asarray(mesh.triangles) 14 | 15 | # ---- 16 | # 1) Create the main mesh trace (surface) 17 | # ---- 18 | mesh_trace = go.Mesh3d( 19 | x=v[:, 0], 20 | y=v[:, 1], 21 | z=v[:, 2], 22 | i=f[:, 0], 23 | j=f[:, 1], 24 | k=f[:, 2], 25 | color="lightblue", 26 | opacity=0.8, 27 | ) 28 | 29 | # ---- 30 | # 2) Build edge information for a wireframe overlay 31 | # ---- 32 | edges = set() 33 | for tri in f: 34 | # Each triangle has three edges: (i, j), (j, k), (k, i) 35 | for e in [(tri[0], tri[1]), (tri[1], tri[2]), (tri[2], tri[0])]: 36 | # Sort the edge so (1, 5) and (5, 1) aren't treated as duplicates 37 | edge = tuple(sorted(e)) 38 | edges.add(edge) 39 | 40 | # Now create line coordinates for Plotly's Scatter3d 41 | edge_x = [] 42 | edge_y = [] 43 | edge_z = [] 44 | 45 | for p1, p2 in edges: 46 | edge_x += [v[p1, 0], v[p2, 0], None] 47 | edge_y += [v[p1, 1], v[p2, 1], None] 48 | edge_z += [v[p1, 2], v[p2, 2], None] 49 | 50 | wireframe_trace = go.Scatter3d( 51 | x=edge_x, 52 | y=edge_y, 53 | z=edge_z, 54 | mode="lines", 55 | line=dict(color="black", width=1), 56 | showlegend=False, 57 | ) 58 | 59 | # ---- 60 | # 3) Add points trace with size 0 61 | # ---- 62 | points_trace = go.Scatter3d( 63 | x=v[:, 0], 64 | y=v[:, 1], 65 | z=v[:, 2], 66 | mode="markers", 67 | marker=dict(size=0, color="black"), 68 | showlegend=False, 69 | ) 70 | 71 | # ---- 72 | # 4) Combine all traces in one figure 73 | # ---- 74 | fig = go.Figure(data=[mesh_trace, wireframe_trace, points_trace]) 75 | fig.update_layout(width=500, height=500, scene=dict(aspectmode="data")) 76 | fig.show() 77 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/core/tests/test_rotation.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. 2 | # NVIDIA CORPORATION and its licensors retain all intellectual property 3 | # and proprietary rights in and to this software, related documentation 4 | # and any modifications thereto. Any use, reproduction, disclosure or 5 | # distribution of this software and related documentation without an express 6 | # license agreement from NVIDIA CORPORATION is strictly prohibited. 7 | 8 | from ..rotation3d import * 9 | import numpy as np 10 | import torch 11 | 12 | q = torch.from_numpy(np.array([[0, 1, 2, 3], [-2, 3, -1, 5]], dtype=np.float32)) 13 | print("q", q) 14 | r = quat_normalize(q) 15 | x = torch.from_numpy(np.array([[1, 0, 0], [0, -1, 0]], dtype=np.float32)) 16 | print(r) 17 | print(quat_rotate(r, x)) 18 | 19 | angle = torch.from_numpy(np.array(np.random.rand() * 10.0, dtype=np.float32)) 20 | axis = torch.from_numpy( 21 | np.array([1, np.random.rand() * 10.0, np.random.rand() * 10.0], dtype=np.float32), 22 | ) 23 | 24 | print(repr(angle)) 25 | print(repr(axis)) 26 | 27 | rot = quat_from_angle_axis(angle, axis) 28 | x = torch.from_numpy(np.random.rand(5, 6, 3)) 29 | y = quat_rotate(quat_inverse(rot), quat_rotate(rot, x)) 30 | print(x.numpy()) 31 | print(y.numpy()) 32 | assert np.allclose(x.numpy(), y.numpy()) 33 | 34 | m = torch.from_numpy(np.array([[1, 0, 0], [0, 0, -1], [0, 1, 0]], dtype=np.float32)) 35 | r = quat_from_rotation_matrix(m) 36 | t = torch.from_numpy(np.array([0, 1, 0], dtype=np.float32)) 37 | se3 = transform_from_rotation_translation(r=r, t=t) 38 | print(se3) 39 | print(transform_apply(se3, t)) 40 | 41 | rot = quat_from_angle_axis( 42 | torch.from_numpy(np.array([45, -54], dtype=np.float32)), 43 | torch.from_numpy(np.array([[1, 0, 0], [0, 1, 0]], dtype=np.float32)), 44 | degree=True, 45 | ) 46 | trans = torch.from_numpy(np.array([[1, 1, 0], [1, 1, 0]], dtype=np.float32)) 47 | transform = transform_from_rotation_translation(r=rot, t=trans) 48 | 49 | t = transform_mul(transform, transform_inverse(transform)) 50 | gt = np.zeros((2, 7)) 51 | gt[:, 0] = 1.0 52 | print(t.numpy()) 53 | print(gt) 54 | # assert np.allclose(t.numpy(), gt) 55 | 56 | transform2 = torch.from_numpy( 57 | np.array( 58 | [[1, 0, 0, 1], [0, 0, -1, 0], [0, 1, 0, 0], [0, 0, 0, 1]], dtype=np.float32 59 | ), 60 | ) 61 | transform2 = euclidean_to_transform(transform2) 62 | print(transform2) 63 | -------------------------------------------------------------------------------- /hoidini/smplx/vertex_ids.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is 4 | # holder of all proprietary rights on this computer program. 5 | # You can only use this computer program if you have closed 6 | # a license agreement with MPG or you get the right to use the computer 7 | # program from someone who is authorized to grant you that right. 8 | # Any use of the computer program without a valid license is prohibited and 9 | # liable to prosecution. 10 | # 11 | # Copyright©2019 Max-Planck-Gesellschaft zur Förderung 12 | # der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute 13 | # for Intelligent Systems. All rights reserved. 14 | # 15 | # Contact: ps-license@tuebingen.mpg.de 16 | 17 | from __future__ import print_function 18 | from __future__ import absolute_import 19 | from __future__ import division 20 | 21 | # Joint name to vertex mapping. SMPL/SMPL-H/SMPL-X vertices that correspond to 22 | # MSCOCO and OpenPose joints 23 | vertex_ids = { 24 | "smplh": { 25 | "nose": 332, 26 | "reye": 6260, 27 | "leye": 2800, 28 | "rear": 4071, 29 | "lear": 583, 30 | "rthumb": 6191, 31 | "rindex": 5782, 32 | "rmiddle": 5905, 33 | "rring": 6016, 34 | "rpinky": 6133, 35 | "lthumb": 2746, 36 | "lindex": 2319, 37 | "lmiddle": 2445, 38 | "lring": 2556, 39 | "lpinky": 2673, 40 | "LBigToe": 3216, 41 | "LSmallToe": 3226, 42 | "LHeel": 3387, 43 | "RBigToe": 6617, 44 | "RSmallToe": 6624, 45 | "RHeel": 6787, 46 | }, 47 | "smplx": { 48 | "nose": 9120, 49 | "reye": 9929, 50 | "leye": 9448, 51 | "rear": 616, 52 | "lear": 6, 53 | "rthumb": 8079, 54 | "rindex": 7669, 55 | "rmiddle": 7794, 56 | "rring": 7905, 57 | "rpinky": 8022, 58 | "lthumb": 5361, 59 | "lindex": 4933, 60 | "lmiddle": 5058, 61 | "lring": 5169, 62 | "lpinky": 5286, 63 | "LBigToe": 5770, 64 | "LSmallToe": 5780, 65 | "LHeel": 8846, 66 | "RBigToe": 8463, 67 | "RSmallToe": 8474, 68 | "RHeel": 8635, 69 | }, 70 | "mano": { 71 | "thumb": 744, 72 | "index": 320, 73 | "middle": 443, 74 | "ring": 554, 75 | "pinky": 671, 76 | }, 77 | } 78 | -------------------------------------------------------------------------------- /INSTALL.md: -------------------------------------------------------------------------------- 1 | # Installation Guide 2 | 3 | This guide provides detailed installation instructions for HOIDiNi: Human–Object Interaction through Diffusion Noise Optimization. 4 | 5 | ## Prerequisites 6 | 7 | - **Operating System**: Linux (tested on Ubuntu) 8 | - **Python**: 3.11 (recommended) 9 | - **CUDA**: 12.1 or compatible 10 | - **Conda**: Miniconda or Anaconda installed 11 | 12 | ## 1. Clone the Repository 13 | 14 | First, clone the HOIDiNi repository: 15 | 16 | ```bash 17 | git clone git@github.com:hoidini/HOIDiNi.git 18 | cd HOIDiNi 19 | ``` 20 | 21 | ## 2. Quick Installation (Recommended) 22 | 23 | The easiest way to install HOIDiNi is using the provided setup script: 24 | 25 | ```bash 26 | bash setup.sh 27 | ``` 28 | 29 | This script will automatically handle all dependencies and create the proper environment. 30 | 31 | ## 2. Manual Installation (Alternative) 32 | 33 | If you prefer to install manually or need to customize the installation, follow these steps: 34 | 35 | ### 1. Create Conda Environment 36 | 37 | ```bash 38 | conda create -n hoidini python=3.11 -y 39 | conda activate hoidini 40 | ``` 41 | 42 | ### 2. Update pip 43 | 44 | ```bash 45 | pip install --upgrade pip 46 | ``` 47 | 48 | ### 3. Install PyTorch with CUDA Support 49 | 50 | ```bash 51 | conda install pytorch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 pytorch-cuda=12.1 -c pytorch -c nvidia -y 52 | ``` 53 | 54 | ### 4. Install PyTorch3D Dependencies 55 | 56 | ```bash 57 | conda install -c fvcore -c iopath -c conda-forge fvcore iopath 58 | conda install -c bottler nvidiacub 59 | ``` 60 | 61 | ### 5. Install PyTorch3D 62 | 63 | Install the specific PyTorch3D version compatible with PyTorch 2.5.1: 64 | 65 | ```bash 66 | pip install --extra-index-url https://miropsota.github.io/torch_packages_builder pytorch3d==0.7.8+pt2.5.1cu121 67 | ``` 68 | 69 | ### 6. Install PyTorch Cluster 70 | 71 | ```bash 72 | pip install torch-cluster -f https://data.pyg.org/whl/torch-2.5+124.html 73 | ``` 74 | 75 | ### 7. Install PyTorch Geometric 76 | 77 | ```bash 78 | conda install pytorch-geometric -c pyg -y 79 | ``` 80 | 81 | ### 8. Install Git-based Dependencies 82 | 83 | ```bash 84 | pip install git+https://github.com/otaheri/chamfer_distance 85 | pip install git+https://github.com/otaheri/bps_torch 86 | pip install git+https://github.com/openai/CLIP.git 87 | ``` 88 | 89 | ### 9. Install Remaining Dependencies 90 | 91 | ```bash 92 | pip install -r requirements.txt 93 | ``` 94 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/utils/loss_util.py: -------------------------------------------------------------------------------- 1 | from hoidini.closd.diffusion_planner.diffusion.nn import mean_flat, sum_flat 2 | import torch 3 | import numpy as np 4 | 5 | 6 | def angle_l2(angle1, angle2): 7 | a = angle1 - angle2 8 | a = (a + (torch.pi / 2)) % torch.pi - (torch.pi / 2) 9 | return a**2 10 | 11 | 12 | def diff_l2(a, b): 13 | return (a - b) ** 2 14 | 15 | 16 | def masked_l2(a, b, mask, loss_fn=diff_l2, epsilon=1e-8, entries_norm=True): 17 | # assuming a.shape == b.shape == bs, J, Jdim, seqlen 18 | # assuming mask.shape == bs, 1, 1, seqlen 19 | loss = loss_fn(a, b) 20 | loss = sum_flat( 21 | loss * mask.float() 22 | ) # gives \sigma_euclidean over unmasked elements 23 | n_entries = a.shape[1] 24 | if len(a.shape) > 3: 25 | n_entries *= a.shape[2] 26 | non_zero_elements = sum_flat(mask) 27 | if entries_norm: 28 | # In cases the mask is per frame, and not specifying the number of entries per frame, this normalization is needed, 29 | # Otherwise set it to False 30 | non_zero_elements *= n_entries 31 | # print('mask', mask.shape) 32 | # print('non_zero_elements', non_zero_elements) 33 | # print('loss', loss) 34 | mse_loss_val = loss / ( 35 | non_zero_elements + epsilon 36 | ) # Add epsilon to avoid division by zero 37 | # print('mse_loss_val', mse_loss_val) 38 | return mse_loss_val 39 | 40 | 41 | def masked_goal_l2(pred_goal, ref_goal, cond, all_goal_joint_names): 42 | all_goal_joint_names_w_traj = np.append(all_goal_joint_names, "traj") 43 | target_joint_idx = [ 44 | [np.where(all_goal_joint_names_w_traj == j)[0][0] for j in sample_joints] 45 | for sample_joints in cond["target_joint_names"] 46 | ] 47 | loc_mask = torch.zeros_like(pred_goal[:, :-1], dtype=torch.bool) 48 | for sample_idx in range(loc_mask.shape[0]): 49 | loc_mask[sample_idx, target_joint_idx[sample_idx]] = True 50 | loc_mask[:, -1, 1] = False # vertical joint of 'traj' is always masked out 51 | loc_loss = masked_l2( 52 | pred_goal[:, :-1], ref_goal[:, :-1], loc_mask, entries_norm=False 53 | ) 54 | 55 | heading_loss = masked_l2( 56 | pred_goal[:, -1:, :1], 57 | ref_goal[:, -1:, :1], 58 | cond["is_heading"].unsqueeze(1).unsqueeze(1), 59 | loss_fn=angle_l2, 60 | entries_norm=False, 61 | ) 62 | 63 | loss = loc_loss + heading_loss 64 | return loss 65 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/utils/vis_model_utils.py: -------------------------------------------------------------------------------- 1 | from lxml import etree 2 | from lxml.etree import XMLParser, parse, ElementTree, Element, SubElement 3 | from copy import deepcopy 4 | 5 | 6 | def create_vis_model(in_file, out_file, num=10): 7 | xml_parser = XMLParser(remove_blank_text=True) 8 | tree = parse(in_file, parser=xml_parser) 9 | # remove_elements = ['actuator', 'contact', 'equality', 'sensor'] 10 | remove_elements = ["actuator", "contact", "equality"] 11 | for elem in remove_elements: 12 | node = tree.getroot().find(elem) 13 | if node is None: 14 | print(f"has no elem: {elem}") 15 | else: 16 | node.getparent().remove(node) 17 | 18 | option = tree.getroot().find("option") 19 | flag = SubElement(option, "flag", {"contact": "disable"}) 20 | option.addnext(Element("size", {"njmax": "1000"})) 21 | 22 | worldbody = tree.getroot().find("worldbody") 23 | body = worldbody.find("body") 24 | for i in range(1, num): 25 | new_body = deepcopy(body) 26 | new_body.attrib["name"] = "%d_%s" % (i, new_body.attrib["name"]) 27 | for node in new_body.findall(".//body"): 28 | node.attrib["name"] = "%d_%s" % (i, node.attrib["name"]) 29 | for node in new_body.findall(".//joint"): 30 | node.attrib["name"] = "%d_%s" % (i, node.attrib["name"]) 31 | for node in new_body.findall(".//site"): 32 | node.attrib["name"] = "%d_%s" % (i, node.attrib["name"]) 33 | worldbody.append(new_body) 34 | tree.write(out_file, pretty_print=True) 35 | 36 | 37 | if __name__ == "__main__": 38 | import argparse 39 | 40 | parser = argparse.ArgumentParser() 41 | parser.add_argument("--cfg", type=str, default=None) 42 | parser.add_argument( 43 | "--in_model", 44 | type=str, 45 | default="assets/mujoco_models/models/character1/model.xml", 46 | ) 47 | parser.add_argument( 48 | "--out_model", 49 | type=str, 50 | default="assets/mujoco_models/models/character1/model_vis.xml", 51 | ) 52 | args = parser.parse_args() 53 | 54 | in_model = ( 55 | f"assets/mujoco_models/models/{args.cfg}/model.xml" 56 | if args.cfg is not None 57 | else args.in_model 58 | ) 59 | out_model = ( 60 | f"assets/mujoco_models/models/{args.cfg}/model_vis.xml" 61 | if args.cfg is not None 62 | else args.out_model 63 | ) 64 | 65 | create_vis_model(in_model, out_model) 66 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/data_loaders/humanml_utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | HML_JOINT_NAMES = [ 4 | "pelvis", 5 | "left_hip", 6 | "right_hip", 7 | "spine1", 8 | "left_knee", 9 | "right_knee", 10 | "spine2", 11 | "left_ankle", 12 | "right_ankle", 13 | "spine3", 14 | "left_foot", 15 | "right_foot", 16 | "neck", 17 | "left_collar", 18 | "right_collar", 19 | "head", 20 | "left_shoulder", 21 | "right_shoulder", 22 | "left_elbow", 23 | "right_elbow", 24 | "left_wrist", 25 | "right_wrist", 26 | ] 27 | 28 | NUM_HML_JOINTS = len(HML_JOINT_NAMES) # 22 SMPLH body joints 29 | 30 | HML_EE_JOINT_NAMES = ["left_foot", "right_foot", "left_wrist", "right_wrist", "head"] 31 | HML_LOWER_BODY_JOINTS = [ 32 | HML_JOINT_NAMES.index(name) 33 | for name in [ 34 | "pelvis", 35 | "left_hip", 36 | "right_hip", 37 | "left_knee", 38 | "right_knee", 39 | "left_ankle", 40 | "right_ankle", 41 | "left_foot", 42 | "right_foot", 43 | ] 44 | ] 45 | SMPL_UPPER_BODY_JOINTS = [ 46 | i for i in range(len(HML_JOINT_NAMES)) if i not in HML_LOWER_BODY_JOINTS 47 | ] 48 | 49 | 50 | # Recover global angle and positions for rotation data 51 | # root_rot_velocity (B, seq_len, 1) 52 | # root_linear_velocity (B, seq_len, 2) 53 | # root_y (B, seq_len, 1) 54 | # ric_data (B, seq_len, (joint_num - 1)*3) 55 | # rot_data (B, seq_len, (joint_num - 1)*6) 56 | # local_velocity (B, seq_len, joint_num*3) 57 | # foot contact (B, seq_len, 4) 58 | HML_ROOT_BINARY = np.array([True] + [False] * (NUM_HML_JOINTS - 1)) 59 | HML_ROOT_MASK = np.concatenate( 60 | ( 61 | [True] * (1 + 2 + 1), 62 | HML_ROOT_BINARY[1:].repeat(3), 63 | HML_ROOT_BINARY[1:].repeat(6), 64 | HML_ROOT_BINARY.repeat(3), 65 | [False] * 4, 66 | ) 67 | ) 68 | HML_ROOT_HORIZONTAL_MASK = np.concatenate( 69 | ( 70 | [True] * (1 + 2) + [False], 71 | np.zeros_like(HML_ROOT_BINARY[1:].repeat(3)), 72 | np.zeros_like(HML_ROOT_BINARY[1:].repeat(6)), 73 | np.zeros_like(HML_ROOT_BINARY.repeat(3)), 74 | [False] * 4, 75 | ) 76 | ) 77 | HML_LOWER_BODY_JOINTS_BINARY = np.array( 78 | [i in HML_LOWER_BODY_JOINTS for i in range(NUM_HML_JOINTS)] 79 | ) 80 | HML_LOWER_BODY_MASK = np.concatenate( 81 | ( 82 | [True] * (1 + 2 + 1), 83 | HML_LOWER_BODY_JOINTS_BINARY[1:].repeat(3), 84 | HML_LOWER_BODY_JOINTS_BINARY[1:].repeat(6), 85 | HML_LOWER_BODY_JOINTS_BINARY.repeat(3), 86 | [True] * 4, 87 | ) 88 | ) 89 | HML_UPPER_BODY_MASK = ~HML_LOWER_BODY_MASK 90 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/utils/tools.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import os 3 | import shutil 4 | import datetime 5 | import subprocess 6 | from os import path 7 | from PIL import Image 8 | from hoidini.closd.utils.smpllib.utils.math_utils import * 9 | import cv2 10 | 11 | 12 | def assets_dir(): 13 | return path.abspath(path.join(path.dirname(path.abspath(__file__)), "../assets")) 14 | 15 | 16 | def out_dir(): 17 | return path.abspath(path.join(path.dirname(path.abspath(__file__)), "../out")) 18 | 19 | 20 | def log_dir(): 21 | return path.abspath(path.join(path.dirname(path.abspath(__file__)), "../logs")) 22 | 23 | 24 | def recreate_dirs(*dirs): 25 | for d in dirs: 26 | if os.path.exists(d): 27 | shutil.rmtree(d, ignore_errors=True) 28 | os.makedirs(d) 29 | 30 | 31 | def load_img(path): 32 | # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835) 33 | with open(path, "rb") as f: 34 | I = Image.open(f) 35 | img = I.resize((224, 224), Image.ANTIALIAS).convert("RGB") 36 | return img 37 | 38 | 39 | def save_screen_shots(window, file_name, transparent=False, autogui=False): 40 | import glfw 41 | 42 | xpos, ypos = glfw.get_window_pos(window) 43 | width, height = glfw.get_window_size(window) 44 | if autogui: 45 | import pyautogui 46 | 47 | image = pyautogui.screenshot(region=(xpos * 2, ypos * 2, width * 2, height * 2)) 48 | image = cv2.cvtColor( 49 | np.array(image), cv2.COLOR_RGB2BGRA if transparent else cv2.COLOR_RGB2BGR 50 | ) 51 | if transparent: 52 | image[np.all(image >= [240, 240, 240, 240], axis=2)] = [255, 255, 255, 0] 53 | cv2.imwrite(file_name, image) 54 | else: 55 | print(width * 2, height * 2) 56 | subprocess.call( 57 | [ 58 | "screencapture", 59 | "-x", 60 | "-m", 61 | f"-R {xpos},{ypos},{width},{height}", 62 | file_name, 63 | ] 64 | ) 65 | 66 | 67 | def get_eta_str(cur_iter, total_iter, time_per_iter): 68 | eta = time_per_iter * (total_iter - cur_iter - 1) 69 | return str(datetime.timedelta(seconds=round(eta))) 70 | 71 | 72 | class AverageMeter(object): 73 | """Computes and stores the average and current value""" 74 | 75 | def __init__(self): 76 | self.reset() 77 | 78 | def reset(self): 79 | self.val = 0 80 | self.avg = 0 81 | self.sum = 0 82 | self.count = 0 83 | 84 | def update(self, val, n=1): 85 | self.val = val 86 | self.sum += val * n 87 | self.count += n 88 | self.avg = self.sum / self.count 89 | -------------------------------------------------------------------------------- /hoidini/blender_utils/visualize_mesh.py: -------------------------------------------------------------------------------- 1 | import glob 2 | import numpy as np 3 | import bpy 4 | from pytorch3d.structures import Meshes 5 | import trimesh 6 | import os 7 | 8 | from hoidini.blender_utils.general_blender_utils import blend_scp_and_run 9 | from hoidini.datasets.grab.grab_utils import simplify_trimesh 10 | from hoidini.general_utils import TMP_DIR 11 | from hoidini.resource_paths import GRAB_DATA_PATH 12 | 13 | 14 | def create_mesh(vertices, faces, mesh_name="MyMesh", obj_name="MyMesh"): 15 | mesh = bpy.data.meshes.new(mesh_name) 16 | obj = bpy.data.objects.new(obj_name, mesh) 17 | bpy.context.collection.objects.link(obj) 18 | mesh.from_pydata(vertices, [], faces) 19 | mesh.update() 20 | return obj 21 | 22 | 23 | def visualize_mesh( 24 | verts: np.ndarray = None, 25 | faces: np.ndarray = None, 26 | mesh: Meshes = None, 27 | save_blend_fname=None, 28 | unq_name="", 29 | reset_blender=True, 30 | ): 31 | if reset_blender: 32 | bpy.ops.wm.read_factory_settings(use_empty=True) 33 | # Example usage: 34 | if mesh is not None: 35 | verts = mesh.verts_list()[0].detach().cpu().numpy() 36 | faces = mesh.faces_list()[0].detach().cpu().numpy() 37 | create_mesh(verts, faces, mesh_name=unq_name, obj_name=unq_name) 38 | save_blend_fname = save_blend_fname or "/home/dcor/roeyron/tmp/mymesh.blend" 39 | bpy.ops.wm.save_as_mainfile(filepath=save_blend_fname) 40 | print( 41 | f"scp roeyron@c-005.cs.tau.ac.il:{os.path.abspath(save_blend_fname)} ~/Downloads/{os.path.basename(save_blend_fname)}\nblender ~/Downloads/{os.path.basename(save_blend_fname)}" 42 | ) 43 | 44 | 45 | def show_static_mesh(): 46 | obj_mesh_paths = [ 47 | os.path.join(GRAB_DATA_PATH, p) 48 | for p in sorted(glob.glob("**/*.ply", root_dir=GRAB_DATA_PATH)) 49 | ] 50 | # obj_mesh_path = obj_mesh_paths[0] 51 | n_simplify_object = 1000 52 | 53 | for i, obj_mesh_path in enumerate(obj_mesh_paths): 54 | print(obj_mesh_path) 55 | save_path = os.path.join( 56 | TMP_DIR, f"{os.path.basename(obj_mesh_path).replace('.ply', '.blend')}" 57 | ) 58 | obj_mesh = trimesh.load(obj_mesh_path) 59 | obj_mesh = simplify_trimesh(obj_mesh, tgt_faces=n_simplify_object) 60 | unq_name = obj_mesh_path.split("/")[-1].replace(".ply", "") 61 | visualize_mesh( 62 | np.array(obj_mesh.vertices) + np.array([0.5 * i, 0, 0]), 63 | np.array(obj_mesh.faces), 64 | unq_name=unq_name, 65 | reset_blender=False, 66 | ) 67 | bpy.ops.wm.save_mainfile(filepath=save_path) 68 | blend_scp_and_run(save_path) 69 | 70 | 71 | if __name__ == "__main__": 72 | show_static_mesh() 73 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/skeleton/backend/fbx/fbx_read_wrapper.py: -------------------------------------------------------------------------------- 1 | """ 2 | Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. 3 | 4 | NVIDIA CORPORATION and its licensors retain all intellectual property and proprietary 5 | rights in and to this software, related documentation and any modifications thereto. Any 6 | use, reproduction, disclosure or distribution of this software and related documentation 7 | without an express license agreement from NVIDIA CORPORATION is strictly prohibited. 8 | """ 9 | 10 | """ 11 | Script that reads in fbx files from python 2 12 | 13 | This requires a configs file, which contains the command necessary to switch conda 14 | environments to run the fbx reading script from python 2 15 | """ 16 | 17 | from ....core import logger 18 | 19 | import inspect 20 | import os 21 | 22 | import numpy as np 23 | 24 | # Get the current folder to import the config file 25 | current_folder = os.path.realpath( 26 | os.path.abspath(os.path.split(inspect.getfile(inspect.currentframe()))[0]) 27 | ) 28 | 29 | 30 | def fbx_to_array(fbx_file_path, fbx_configs, root_joint, fps): 31 | """ 32 | Reads an fbx file to an array. 33 | 34 | Currently reading of the frame time is not supported. 120 fps is hard coded TODO 35 | 36 | :param fbx_file_path: str, file path to fbx 37 | :return: tuple with joint_names, parents, transforms, frame time 38 | """ 39 | 40 | # Ensure the file path is valid 41 | fbx_file_path = os.path.abspath(fbx_file_path) 42 | assert os.path.exists(fbx_file_path) 43 | 44 | # Switch directories to the utils folder to ensure the reading works 45 | previous_cwd = os.getcwd() 46 | os.chdir(current_folder) 47 | 48 | # Call the python 2.7 script 49 | temp_file_path = os.path.join(current_folder, fbx_configs["tmp_path"]) 50 | python_path = fbx_configs["fbx_py27_path"] 51 | logger.info("executing python script to read fbx data using Autodesk FBX SDK...") 52 | command = '{} fbx_py27_backend.py "{}" "{}" "{}" "{}"'.format( 53 | python_path, fbx_file_path, temp_file_path, root_joint, fps 54 | ) 55 | logger.debug("executing command: {}".format(command)) 56 | os.system(command) 57 | logger.info( 58 | "executing python script to read fbx data using Autodesk FBX SDK... done" 59 | ) 60 | 61 | with open(temp_file_path, "rb") as f: 62 | data = np.load(f) 63 | output = ( 64 | data["names"], 65 | data["parents"], 66 | data["transforms"], 67 | data["fps"], 68 | ) 69 | 70 | # Remove the temporary file 71 | os.remove(temp_file_path) 72 | 73 | # Return the os to its previous cwd, otherwise reading multiple files might fail 74 | os.chdir(previous_cwd) 75 | return output 76 | -------------------------------------------------------------------------------- /hoidini/amasstools/extract_joints.py: -------------------------------------------------------------------------------- 1 | import os 2 | import argparse 3 | import numpy as np 4 | 5 | import torch 6 | from smplh_layer import load_smplh_gender 7 | from loop_amass import loop_amams 8 | 9 | 10 | def extract_joints( 11 | base_folder, 12 | new_base_folder, 13 | smplh_folder, 14 | jointstype, 15 | batch_size, 16 | gender, 17 | use_betas, 18 | device, 19 | force_redo, 20 | ): 21 | print( 22 | "Extract joint position ({}) from SMPL pose parameter, {} betas and {}".format( 23 | jointstype, 24 | "with" if use_betas else "without", 25 | "with {gender} body shape" 26 | if gender != "gendered" 27 | else "with original gender", 28 | ) 29 | ) 30 | print("The processed motions will be stored in this folder:") 31 | print(new_base_folder) 32 | 33 | smplh = load_smplh_gender(gender, smplh_folder, jointstype, batch_size, device) 34 | 35 | iterator = loop_amams( 36 | base_folder, 37 | new_base_folder, 38 | ext=".npz", 39 | newext=".npy", 40 | force_redo=force_redo, 41 | ) 42 | 43 | for motion_path, new_motion_path in iterator: 44 | data = np.load(motion_path) 45 | 46 | # process sequences 47 | poses = torch.from_numpy(data["poses"]).to(torch.float).to(device) 48 | trans = torch.from_numpy(data["trans"]).to(torch.float).to(device) 49 | 50 | if use_betas and "betas" in data and data["betas"] is not None: 51 | betas = torch.from_numpy(data["betas"]).to(torch.float).to(device) 52 | else: 53 | betas = None 54 | 55 | if gender == "gendered": 56 | gender_motion = str(data["gender"]) 57 | smplh_layer = smplh[gender_motion] 58 | else: 59 | smplh_layer = smplh 60 | 61 | joints = smplh_layer(poses, trans, betas).cpu().numpy() 62 | np.save(new_motion_path, joints) 63 | 64 | 65 | def main(): 66 | base_folder = "datasets/motions/AMASS_20.0_fps_nh" 67 | smplh_folder = "deps/smplh" 68 | jointstype = "smpljoints" 69 | batch_size = 4096 70 | gender = "neutral" 71 | use_betas = False 72 | force_redo = False 73 | 74 | name = os.path.split(base_folder)[1] 75 | new_base_folder = f"datasets/motions/{name}_{jointstype}_{gender}_{'betas' if use_betas else 'nobetas'}" 76 | 77 | device = torch.device("cuda" if torch.cuda.is_available() else "cpu") 78 | 79 | extract_joints( 80 | base_folder, 81 | new_base_folder, 82 | smplh_folder, 83 | jointstype, 84 | batch_size, 85 | gender, 86 | use_betas, 87 | device, 88 | force_redo, 89 | ) 90 | 91 | 92 | if __name__ == "__main__": 93 | main() 94 | -------------------------------------------------------------------------------- /hoidini/amasstools/joints.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | 4 | JOINT_NAMES = { 5 | "smpljoints": [ 6 | "pelvis", 7 | "left_hip", 8 | "right_hip", 9 | "spine1", 10 | "left_knee", 11 | "right_knee", 12 | "spine2", 13 | "left_ankle", 14 | "right_ankle", 15 | "spine3", 16 | "left_foot", 17 | "right_foot", 18 | "neck", 19 | "left_collar", 20 | "right_collar", 21 | "head", 22 | "left_shoulder", 23 | "right_shoulder", 24 | "left_elbow", 25 | "right_elbow", 26 | "left_wrist", 27 | "right_wrist", 28 | "left_hand", 29 | "right_hand", 30 | ], 31 | } 32 | 33 | JOINT_NAMES_IDX = { 34 | jointstype: {x: i for i, x in enumerate(JOINT_NAMES[jointstype])} 35 | for jointstype in JOINT_NAMES 36 | } 37 | 38 | 39 | # EXTRACTOR from SMPLH layer 40 | # replace the "left_hand", "right_hand" by "left_index1", "right_index1" of SMPLH 41 | JOINTS_EXTRACTOR = { 42 | "smpljoints": np.array( 43 | [ 44 | 0, 45 | 1, 46 | 2, 47 | 3, 48 | 4, 49 | 5, 50 | 6, 51 | 7, 52 | 8, 53 | 9, 54 | 10, 55 | 11, 56 | 12, 57 | 13, 58 | 14, 59 | 15, 60 | 16, 61 | 17, 62 | 18, 63 | 19, 64 | 20, 65 | 21, 66 | 22, 67 | 37, 68 | ] 69 | ) 70 | } 71 | 72 | # [1, 4, 7, 10, 13, 16, 18, 20, 22] for smpljoints 73 | LEFT_CHAIN = { 74 | jointstype: np.array( 75 | [i for x, i in JOINT_NAMES_IDX[jointstype].items() if "left" in x] 76 | ) 77 | for jointstype in JOINT_NAMES 78 | } 79 | 80 | # [2, 5, 8, 11, 14, 17, 19, 21, 23] for smpljoints 81 | RIGHT_CHAIN = { 82 | jointstype: np.array( 83 | [i for x, i in JOINT_NAMES_IDX[jointstype].items() if "right" in x] 84 | ) 85 | for jointstype in JOINT_NAMES 86 | } 87 | 88 | 89 | INFOS = { 90 | "smpljoints": { 91 | "LM": JOINT_NAMES["smpljoints"].index("left_ankle"), 92 | "RM": JOINT_NAMES["smpljoints"].index("right_ankle"), 93 | "LF": JOINT_NAMES["smpljoints"].index("left_foot"), 94 | "RF": JOINT_NAMES["smpljoints"].index("right_foot"), 95 | "LS": JOINT_NAMES["smpljoints"].index("left_shoulder"), 96 | "RS": JOINT_NAMES["smpljoints"].index("right_shoulder"), 97 | "LH": JOINT_NAMES["smpljoints"].index("left_hip"), 98 | "RH": JOINT_NAMES["smpljoints"].index("right_hip"), 99 | "njoints": len(JOINT_NAMES["smpljoints"]), 100 | } 101 | } 102 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/models/rnn.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | from hoidini.closd.utils.smpllib.khrylib.utils.torch import * 3 | 4 | 5 | class RNN(nn.Module): 6 | def __init__(self, input_dim, out_dim, cell_type="lstm", bi_dir=False): 7 | super().__init__() 8 | self.input_dim = input_dim 9 | self.out_dim = out_dim 10 | self.cell_type = cell_type 11 | self.bi_dir = bi_dir 12 | self.mode = "batch" 13 | rnn_cls = nn.LSTMCell if cell_type == "lstm" else nn.GRUCell 14 | hidden_dim = out_dim // 2 if bi_dir else out_dim 15 | self.rnn_f = rnn_cls(self.input_dim, hidden_dim) 16 | if bi_dir: 17 | self.rnn_b = rnn_cls(self.input_dim, hidden_dim) 18 | self.hx, self.cx = None, None 19 | 20 | def set_mode(self, mode): 21 | self.mode = mode 22 | 23 | def initialize(self, batch_size=1, hx=None, cx=None): 24 | if self.mode == "step": 25 | self.hx = zeros((batch_size, self.rnn_f.hidden_size)) if hx is None else hx 26 | if self.cell_type == "lstm": 27 | self.cx = ( 28 | zeros((batch_size, self.rnn_f.hidden_size)) if cx is None else cx 29 | ) 30 | 31 | def forward(self, x): 32 | if self.mode == "step": 33 | self.hx, self.cx = batch_to(x.device, self.hx, self.cx) 34 | if self.cell_type == "lstm": 35 | self.hx, self.cx = self.rnn_f(x, (self.hx, self.cx)) 36 | else: 37 | self.hx = self.rnn_f(x, self.hx) 38 | rnn_out = self.hx 39 | else: 40 | rnn_out_f = self.batch_forward(x) 41 | if not self.bi_dir: 42 | return rnn_out_f 43 | rnn_out_b = self.batch_forward(x, reverse=True) 44 | rnn_out = torch.cat((rnn_out_f, rnn_out_b), 2) 45 | return rnn_out 46 | 47 | def batch_forward(self, x, reverse=False): 48 | rnn = self.rnn_b if reverse else self.rnn_f 49 | rnn_out = [] 50 | hx = zeros((x.size(1), rnn.hidden_size), device=x.device) 51 | if self.cell_type == "lstm": 52 | cx = zeros((x.size(1), rnn.hidden_size), device=x.device) 53 | ind = reversed(range(x.size(0))) if reverse else range(x.size(0)) 54 | for t in ind: 55 | if self.cell_type == "lstm": 56 | hx, cx = rnn(x[t, ...], (hx, cx)) 57 | else: 58 | hx = rnn(x[t, ...], hx) 59 | rnn_out.append(hx.unsqueeze(0)) 60 | if reverse: 61 | rnn_out.reverse() 62 | rnn_out = torch.cat(rnn_out, 0) 63 | return rnn_out 64 | 65 | 66 | if __name__ == "__main__": 67 | rnn = RNN(12, 24, "gru", bi_dir=True) 68 | input = zeros(5, 3, 12) 69 | out = rnn(input) 70 | print(out.shape) 71 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/agents/agent_pg.py: -------------------------------------------------------------------------------- 1 | from hoidini.closd.utils.smpllib.khrylib.rl.core import estimate_advantages 2 | from hoidini.closd.utils.smpllib.khrylib.rl.agents.agent import Agent 3 | from hoidini.closd.utils.smpllib.khrylib.utils.torch import * 4 | import time 5 | 6 | 7 | class AgentPG(Agent): 8 | 9 | def __init__( 10 | self, 11 | tau=0.95, 12 | optimizer_policy=None, 13 | optimizer_value=None, 14 | opt_num_epochs=1, 15 | value_opt_niter=1, 16 | **kwargs 17 | ): 18 | super().__init__(**kwargs) 19 | self.tau = tau 20 | self.optimizer_policy = optimizer_policy 21 | self.optimizer_value = optimizer_value 22 | self.opt_num_epochs = opt_num_epochs 23 | self.value_opt_niter = value_opt_niter 24 | 25 | def update_value(self, states, returns): 26 | """update critic""" 27 | for _ in range(self.value_opt_niter): 28 | values_pred = self.value_net(self.trans_value(states)) 29 | value_loss = (values_pred - returns).pow(2).mean() 30 | self.optimizer_value.zero_grad() 31 | value_loss.backward() 32 | self.optimizer_value.step() 33 | 34 | def update_policy(self, states, actions, returns, advantages, exps): 35 | """update policy""" 36 | # use a2c by default 37 | ind = exps.nonzero().squeeze(1) 38 | for _ in range(self.opt_num_epochs): 39 | self.update_value(states, returns) 40 | log_probs = self.policy_net.get_log_prob( 41 | self.trans_policy(states)[ind], actions[ind] 42 | ) 43 | policy_loss = -(log_probs * advantages[ind]).mean() 44 | self.optimizer_policy.zero_grad() 45 | policy_loss.backward() 46 | self.optimizer_policy.step() 47 | 48 | def update_params(self, batch): 49 | t0 = time.time() 50 | to_train(*self.update_modules) 51 | states = torch.from_numpy(batch.states).to(self.dtype).to(self.device) 52 | actions = torch.from_numpy(batch.actions).to(self.dtype).to(self.device) 53 | rewards = torch.from_numpy(batch.rewards).to(self.dtype).to(self.device) 54 | masks = torch.from_numpy(batch.masks).to(self.dtype).to(self.device) 55 | exps = torch.from_numpy(batch.exps).to(self.dtype).to(self.device) 56 | with to_test(*self.update_modules): 57 | with torch.no_grad(): 58 | values = self.value_net(self.trans_value(states)) 59 | 60 | """get advantage estimation from the trajectories""" 61 | advantages, returns = estimate_advantages( 62 | rewards, masks, values, self.gamma, self.tau 63 | ) 64 | 65 | self.update_policy(states, actions, returns, advantages, exps) 66 | 67 | return time.time() - t0 68 | -------------------------------------------------------------------------------- /hoidini/closd/utils/poselib/poselib/visualization/core.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. 2 | # NVIDIA CORPORATION and its licensors retain all intellectual property 3 | # and proprietary rights in and to this software, related documentation 4 | # and any modifications thereto. Any use, reproduction, disclosure or 5 | # distribution of this software and related documentation without an express 6 | # license agreement from NVIDIA CORPORATION is strictly prohibited. 7 | 8 | """ 9 | The base abstract classes for plotter and the plotting tasks. It describes how the plotter 10 | deals with the tasks in the general cases 11 | """ 12 | from typing import List 13 | 14 | 15 | class BasePlotterTask(object): 16 | _task_name: str # unique name of the task 17 | _task_type: str # type of the task is used to identify which callable 18 | 19 | def __init__(self, task_name: str, task_type: str) -> None: 20 | self._task_name = task_name 21 | self._task_type = task_type 22 | 23 | @property 24 | def task_name(self): 25 | return self._task_name 26 | 27 | @property 28 | def task_type(self): 29 | return self._task_type 30 | 31 | def get_scoped_name(self, name): 32 | return self._task_name + "/" + name 33 | 34 | def __iter__(self): 35 | """Should override this function to return a list of task primitives""" 36 | raise NotImplementedError 37 | 38 | 39 | class BasePlotterTasks(object): 40 | def __init__(self, tasks) -> None: 41 | self._tasks = tasks 42 | 43 | def __iter__(self): 44 | for task in self._tasks: 45 | yield from task 46 | 47 | 48 | class BasePlotter(object): 49 | """An abstract plotter which deals with a plotting task. The children class needs to implement 50 | the functions to create/update the objects according to the task given 51 | """ 52 | 53 | _task_primitives: List[BasePlotterTask] 54 | 55 | def __init__(self, task: BasePlotterTask) -> None: 56 | self._task_primitives = [] 57 | self.create(task) 58 | 59 | @property 60 | def task_primitives(self): 61 | return self._task_primitives 62 | 63 | def create(self, task: BasePlotterTask) -> None: 64 | """Create more task primitives from a task for the plotter""" 65 | new_task_primitives = list(task) # get all task primitives 66 | self._task_primitives += new_task_primitives # append them 67 | self._create_impl(new_task_primitives) 68 | 69 | def update(self) -> None: 70 | """Update the plotter for any updates in the task primitives""" 71 | self._update_impl(self._task_primitives) 72 | 73 | def _update_impl(self, task_list: List[BasePlotterTask]) -> None: 74 | raise NotImplementedError 75 | 76 | def _create_impl(self, task_list: List[BasePlotterTask]) -> None: 77 | raise NotImplementedError 78 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/utils/misc.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import torch.nn as nn 3 | 4 | 5 | class WeightedSum(nn.Module): 6 | def __init__(self, num_rows): 7 | super(WeightedSum, self).__init__() 8 | # Initialize learnable weights 9 | self.weights = nn.Parameter(torch.randn(num_rows)) 10 | 11 | def forward(self, x): 12 | # Ensure weights are normalized (optional) 13 | normalized_weights = ( 14 | self.weights / self.weights.sum() 15 | ) # torch.softmax(self.weights, dim=0) 16 | # Compute the weighted sum of the rows 17 | weighted_sum = torch.matmul(normalized_weights, x) 18 | return weighted_sum 19 | 20 | 21 | def wrapped_getattr(self, name, default=None, wrapped_member_name="model"): 22 | """should be called from wrappers of model classes such as ClassifierFreeSampleModel""" 23 | 24 | if isinstance(self, torch.nn.Module): 25 | # for descendants of nn.Module, name may be in self.__dict__[_parameters/_buffers/_modules] 26 | # so we activate nn.Module.__getattr__ first. 27 | # Otherwise, we might encounter an infinite loop 28 | try: 29 | attr = torch.nn.Module.__getattr__(self, name) 30 | except AttributeError: 31 | wrapped_member = torch.nn.Module.__getattr__(self, wrapped_member_name) 32 | attr = getattr(wrapped_member, name, default) 33 | else: 34 | # the easy case, where self is not derived from nn.Module 35 | wrapped_member = getattr(self, wrapped_member_name) 36 | attr = getattr(wrapped_member, name, default) 37 | return attr 38 | 39 | 40 | def to_numpy(tensor): 41 | if torch.is_tensor(tensor): 42 | return tensor.cpu().numpy() 43 | elif type(tensor).__module__ != "numpy": 44 | raise ValueError("Cannot convert {} to numpy array".format(type(tensor))) 45 | return tensor 46 | 47 | 48 | def to_torch(ndarray): 49 | if type(ndarray).__module__ == "numpy": 50 | return torch.from_numpy(ndarray) 51 | elif not torch.is_tensor(ndarray): 52 | raise ValueError("Cannot convert {} to torch tensor".format(type(ndarray))) 53 | return ndarray 54 | 55 | 56 | def cleanexit(): 57 | import sys 58 | import os 59 | 60 | try: 61 | sys.exit(0) 62 | except SystemExit: 63 | os._exit(0) 64 | 65 | 66 | def load_model_wo_clip(model, state_dict): 67 | missing_keys, unexpected_keys = model.load_state_dict(state_dict, strict=False) 68 | assert len(unexpected_keys) == 0 69 | assert all([k.startswith("clip_model.") for k in missing_keys]) 70 | 71 | 72 | def freeze_joints(x, joints_to_freeze): 73 | # Freezes selected joint *rotations* as they appear in the first frame 74 | # x [bs, [root+n_joints], joint_dim(6), seqlen] 75 | frozen = x.detach().clone() 76 | frozen[:, joints_to_freeze, :, :] = frozen[:, joints_to_freeze, :, :1] 77 | return frozen 78 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/diffusion/losses.py: -------------------------------------------------------------------------------- 1 | # This code is based on https://github.com/openai/guided-diffusion 2 | """ 3 | Helpers for various likelihood-based losses. These are ported from the original 4 | Ho et al. diffusion models codebase: 5 | https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/utils.py 6 | """ 7 | 8 | import numpy as np 9 | import torch as th 10 | 11 | 12 | def normal_kl(mean1, logvar1, mean2, logvar2): 13 | """ 14 | Compute the KL divergence between two gaussians. 15 | 16 | Shapes are automatically broadcasted, so batches can be compared to 17 | scalars, among other use cases. 18 | """ 19 | tensor = None 20 | for obj in (mean1, logvar1, mean2, logvar2): 21 | if isinstance(obj, th.Tensor): 22 | tensor = obj 23 | break 24 | assert tensor is not None, "at least one argument must be a Tensor" 25 | 26 | # Force variances to be Tensors. Broadcasting helps convert scalars to 27 | # Tensors, but it does not work for th.exp(). 28 | logvar1, logvar2 = [ 29 | x if isinstance(x, th.Tensor) else th.tensor(x).to(tensor) 30 | for x in (logvar1, logvar2) 31 | ] 32 | 33 | return 0.5 * ( 34 | -1.0 35 | + logvar2 36 | - logvar1 37 | + th.exp(logvar1 - logvar2) 38 | + ((mean1 - mean2) ** 2) * th.exp(-logvar2) 39 | ) 40 | 41 | 42 | def approx_standard_normal_cdf(x): 43 | """ 44 | A fast approximation of the cumulative distribution function of the 45 | standard normal. 46 | """ 47 | return 0.5 * (1.0 + th.tanh(np.sqrt(2.0 / np.pi) * (x + 0.044715 * th.pow(x, 3)))) 48 | 49 | 50 | def discretized_gaussian_log_likelihood(x, *, means, log_scales): 51 | """ 52 | Compute the log-likelihood of a Gaussian distribution discretizing to a 53 | given image. 54 | 55 | :param x: the target images. It is assumed that this was uint8 values, 56 | rescaled to the range [-1, 1]. 57 | :param means: the Gaussian mean Tensor. 58 | :param log_scales: the Gaussian log stddev Tensor. 59 | :return: a tensor like x of log probabilities (in nats). 60 | """ 61 | assert x.shape == means.shape == log_scales.shape 62 | centered_x = x - means 63 | inv_stdv = th.exp(-log_scales) 64 | plus_in = inv_stdv * (centered_x + 1.0 / 255.0) 65 | cdf_plus = approx_standard_normal_cdf(plus_in) 66 | min_in = inv_stdv * (centered_x - 1.0 / 255.0) 67 | cdf_min = approx_standard_normal_cdf(min_in) 68 | log_cdf_plus = th.log(cdf_plus.clamp(min=1e-12)) 69 | log_one_minus_cdf_min = th.log((1.0 - cdf_min).clamp(min=1e-12)) 70 | cdf_delta = cdf_plus - cdf_min 71 | log_probs = th.where( 72 | x < -0.999, 73 | log_cdf_plus, 74 | th.where(x > 0.999, log_one_minus_cdf_min, th.log(cdf_delta.clamp(min=1e-12))), 75 | ) 76 | assert log_probs.shape == x.shape 77 | return log_probs 78 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/utils/sampler_util.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import torch 3 | import torch.nn as nn 4 | from copy import deepcopy 5 | from hoidini.closd.diffusion_planner.utils.misc import wrapped_getattr 6 | import joblib 7 | 8 | 9 | # A wrapper model for Classifier-free guidance **SAMPLING** only 10 | # https://arxiv.org/abs/2207.12598 11 | class ClassifierFreeSampleModel(nn.Module): 12 | 13 | def __init__(self, model, guidance_type="text"): 14 | super().__init__() 15 | self.model = model # model is the actual model to run 16 | 17 | assert ( 18 | model.cond_mask_prob > 0 19 | ), "Cannot run a guided diffusion on a model that has not been trained with no conditions" 20 | 21 | self.guidance_type = guidance_type 22 | 23 | def forward(self, x, timesteps, y=None): 24 | cond_mode = self.model.cond_mode 25 | assert cond_mode in ["text", "action"] 26 | # y_uncond = deepcopy(y) # (Roey) enable gradient propagation by shallow copy instead of deep 27 | y_uncond = {k: v for k, v in y.items()} 28 | if "text" in self.guidance_type: 29 | y_uncond["text_uncond"] = True 30 | if "target" in self.guidance_type: 31 | y_uncond["target_uncond"] = True 32 | out = self.model(x, timesteps, y) 33 | out_uncond = self.model(x, timesteps, y_uncond) 34 | return out_uncond + (y["scale"].view(-1, 1, 1, 1) * (out - out_uncond)) 35 | 36 | def __getattr__(self, name, default=None): 37 | # this method is reached only if name is not in self.__dict__. 38 | return wrapped_getattr(self, name, default=None) 39 | 40 | 41 | class AutoRegressiveSampler: 42 | def __init__(self, args, sample_fn, required_frames=196): 43 | self.sample_fn = sample_fn 44 | self.args = args 45 | self.required_frames = required_frames 46 | 47 | def sample(self, model, shape, **kargs): 48 | n_iterations = (self.required_frames // self.args.pred_len) + 1 49 | samples_buf = [] 50 | cur_prefix = deepcopy(kargs["model_kwargs"]["y"]["prefix"]) # init with data 51 | if self.args.autoregressive_include_prefix: 52 | samples_buf.append(cur_prefix) 53 | autoregressive_shape = list(deepcopy(shape)) 54 | autoregressive_shape[-1] = self.args.pred_len 55 | for _ in range(n_iterations): 56 | cur_kargs = deepcopy(kargs) 57 | cur_kargs["model_kwargs"]["y"]["prefix"] = cur_prefix 58 | sample = self.sample_fn(model, autoregressive_shape, **cur_kargs) 59 | sample = sample.detach() 60 | samples_buf.append(sample.clone()[..., -self.args.pred_len :]) 61 | cur_prefix = sample.clone()[..., -self.args.context_len :] # update 62 | 63 | full_batch = torch.cat(samples_buf, dim=-1)[ 64 | ..., : self.required_frames 65 | ] # 200 -> 196 66 | return full_batch 67 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/train/train_mdm.py: -------------------------------------------------------------------------------- 1 | # This code is based on https://github.com/openai/guided-diffusion 2 | """ 3 | Train a diffusion model on images. 4 | """ 5 | 6 | import os 7 | import json 8 | from hoidini.closd.diffusion_planner.utils.fixseed import fixseed 9 | from hoidini.closd.diffusion_planner.utils.parser_util import train_args 10 | from hoidini.closd.diffusion_planner.utils import dist_util 11 | from hoidini.closd.diffusion_planner.train.training_loop import TrainLoop 12 | from hoidini.closd.diffusion_planner.data_loaders.get_data import get_dataset_loader 13 | from hoidini.closd.diffusion_planner.utils.model_util import create_model_and_diffusion 14 | from hoidini.closd.diffusion_planner.train.train_platforms import ( 15 | ClearmlPlatform, 16 | TensorboardPlatform, 17 | NoPlatform, 18 | WandBPlatform, 19 | ) # required for the eval operation 20 | 21 | PLATFORM_REGISTRY = { 22 | "WandBPlatform": WandBPlatform, 23 | "TensorboardPlatform": TensorboardPlatform, 24 | "ClearmlPlatform": ClearmlPlatform, 25 | "NoPlatform": NoPlatform, 26 | } 27 | 28 | 29 | def main(): 30 | args = train_args() 31 | fixseed(args.seed) 32 | TrainPlatform = PLATFORM_REGISTRY[args.train_platform_type] 33 | train_platform = TrainPlatform(args.save_dir) 34 | train_platform.report_args(args, name="Args") 35 | 36 | if args.save_dir is None: 37 | raise FileNotFoundError("save_dir was not specified.") 38 | elif os.path.exists(args.save_dir) and not args.overwrite: 39 | raise FileExistsError("save_dir [{}] already exists.".format(args.save_dir)) 40 | elif not os.path.exists(args.save_dir): 41 | os.makedirs(args.save_dir) 42 | args_path = os.path.join(args.save_dir, "args.json") 43 | with open(args_path, "w") as fw: 44 | json.dump(vars(args), fw, indent=4, sort_keys=True) 45 | 46 | dist_util.setup_dist(args.device) 47 | 48 | print("creating data loader...") 49 | 50 | data = get_dataset_loader( 51 | name=args.dataset, 52 | batch_size=args.batch_size, 53 | num_frames=args.num_frames, 54 | fixed_len=args.context_len + args.pred_len, 55 | pred_len=args.pred_len, 56 | hml_type=args.hml_type, 57 | device=dist_util.dev(), 58 | experiment_dir=args.save_dir, 59 | is_training=True, 60 | dataset_data=args.dataset_data, 61 | seed=args.seed, 62 | features_string=args.features_string, 63 | data_load_lim=args.data_load_lim, 64 | grab_split="train", 65 | condition_input_feature=args.condition_input_feature, 66 | ) 67 | 68 | print("creating model and diffusion...") 69 | model, diffusion = create_model_and_diffusion(args, data.dataset.n_feats) 70 | model.to(dist_util.dev()) 71 | 72 | print( 73 | "Total params: %.2fM" 74 | % (sum(p.numel() for p in model.parameters_wo_clip()) / 1e6) 75 | ) 76 | print("Training...") 77 | TrainLoop(args, train_platform, model, diffusion, data).run_loop() 78 | train_platform.close() 79 | 80 | 81 | if __name__ == "__main__": 82 | main() 83 | -------------------------------------------------------------------------------- /hoidini/amasstools/smpl_mirroring.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | 4 | import torch 5 | from loop_amass import loop_amams 6 | 7 | # FROM 8 | # https://github.com/mkocabas/PARE/blob/master/pare/core/constants.py 9 | # Permutation of SMPL pose parameters when flipping the shape 10 | # fmt: off 11 | SMPL_JOINTS_FLIP_PERM = [ 12 | 0, 2, 1, 3, 5, 4, 6, 8, 7, 9, 13 | 11, 10, 12, 14, 13, 15, 17, 16, 19, 18, 14 | 21, 20, 15 | ] # Hands are removed: # , 23, 22] 16 | # fmt: on 17 | 18 | SMPL_POSE_FLIP_PERM = [] 19 | for i in SMPL_JOINTS_FLIP_PERM: 20 | SMPL_POSE_FLIP_PERM.append(3 * i) 21 | SMPL_POSE_FLIP_PERM.append(3 * i + 1) 22 | SMPL_POSE_FLIP_PERM.append(3 * i + 2) 23 | 24 | # FROM 25 | # https://github.com/mkocabas/PARE/blob/master/pare/utils/image_utils.py#L220 26 | 27 | 28 | def flip_pose(_pose): 29 | """Flip pose. 30 | The flipping is based on SMPL parameters. 31 | """ 32 | pose = _pose.clone() 33 | flipped_parts = SMPL_POSE_FLIP_PERM 34 | pose = pose[..., flipped_parts] 35 | # we also negate the second and the third dimension of the axis-angle 36 | pose[..., 1::3] = -pose[..., 1::3] 37 | pose[..., 2::3] = -pose[..., 2::3] 38 | return pose 39 | 40 | 41 | def mirror_smpl( 42 | base_folder, 43 | new_base_folder, 44 | force_redo, 45 | ): 46 | print("Mirror SMPL pose parameters") 47 | print("The processed motions will be stored in this folder:") 48 | print(new_base_folder) 49 | 50 | iterator = loop_amams( 51 | base_folder, 52 | new_base_folder, 53 | ext=".npz", 54 | newext=".npz", 55 | force_redo=force_redo, 56 | ) 57 | 58 | for motion_path, new_motion_path in iterator: 59 | if "humanact12" in motion_path: 60 | continue 61 | 62 | if new_base_folder in motion_path: 63 | continue 64 | 65 | # not mirroing again 66 | if motion_path.startswith("M"): 67 | continue 68 | 69 | data = {x: y for x, y in np.load(motion_path).items()} 70 | 71 | # process sequences 72 | poses = torch.from_numpy(data["poses"]) 73 | trans = torch.from_numpy(data["trans"]) 74 | 75 | # flip poses 76 | assert poses.shape[-1] == 66 77 | mirror_poses = flip_pose(poses) 78 | 79 | # flip trans 80 | x, y, z = torch.unbind(trans, dim=-1) 81 | mirrored_trans = torch.stack((-x, y, z), axis=-1) 82 | 83 | data["poses"] = mirror_poses.numpy() 84 | data["trans"] = mirrored_trans.numpy() 85 | 86 | np.savez(new_motion_path, **data) 87 | 88 | return new_base_folder 89 | 90 | 91 | def main(): 92 | base_folder = "datasets/motions/AMASS_20.0_fps_nh" 93 | new_base_folder = os.path.join(base_folder, "M") 94 | # put the mirror motions in the M/ subfolder 95 | 96 | force_redo = False 97 | 98 | mirror_smpl( 99 | base_folder, 100 | new_base_folder, 101 | force_redo, 102 | ) 103 | 104 | 105 | if __name__ == "__main__": 106 | main() 107 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/utils/visualizer.py: -------------------------------------------------------------------------------- 1 | from hoidini.closd.utils.smpllib.khrylib.rl.envs.visual.humanoid_vis import ( 2 | HumanoidVisEnv, 3 | ) 4 | import glfw 5 | import math 6 | 7 | 8 | class Visualizer: 9 | def __init__(self, vis_file): 10 | self.fr = 0 11 | self.num_fr = 0 12 | self.T_arr = [1, 2, 4, 6, 8, 10, 12, 15, 20, 30, 40, 50, 60] 13 | self.T = 12 14 | self.paused = False 15 | self.reverse = False 16 | self.repeat = False 17 | self.vis_file = vis_file 18 | 19 | self.env_vis = HumanoidVisEnv(vis_file, 1, focus=False) 20 | 21 | self.env_vis._get_viewer("human")._hide_overlay = True 22 | self.env_vis.set_custom_key_callback(self.key_callback) 23 | 24 | def data_generator(self): 25 | raise NotImplementedError 26 | 27 | def update_pose(self): 28 | raise NotImplementedError 29 | 30 | def key_callback(self, key, action, mods): 31 | 32 | if action != glfw.RELEASE: 33 | return False 34 | if key == glfw.KEY_D: 35 | self.T = self.T_arr[(self.T_arr.index(self.T) + 1) % len(self.T_arr)] 36 | print(f"T: {self.T}") 37 | elif key == glfw.KEY_F: 38 | self.T = self.T_arr[(self.T_arr.index(self.T) - 1) % len(self.T_arr)] 39 | print(f"T: {self.T}") 40 | elif key == glfw.KEY_Q: 41 | self.data = next(self.data_gen, None) 42 | if self.data is None: 43 | print("end of data!!") 44 | exit() 45 | self.fr = 0 46 | self.update_pose() 47 | elif key == glfw.KEY_W: 48 | self.fr = 0 49 | self.update_pose() 50 | elif key == glfw.KEY_E: 51 | self.fr = self.num_fr - 1 52 | self.update_pose() 53 | elif key == glfw.KEY_G: 54 | self.repeat = not self.repeat 55 | self.update_pose() 56 | 57 | elif key == glfw.KEY_S: 58 | self.reverse = not self.reverse 59 | elif key == glfw.KEY_RIGHT: 60 | if self.fr < self.num_fr - 1: 61 | self.fr += 1 62 | self.update_pose() 63 | elif key == glfw.KEY_LEFT: 64 | if self.fr > 0: 65 | self.fr -= 1 66 | self.update_pose() 67 | elif key == glfw.KEY_SPACE: 68 | self.paused = not self.paused 69 | else: 70 | return False 71 | return True 72 | 73 | def render(self): 74 | self.env_vis.render() 75 | 76 | def show_animation(self): 77 | self.t = 0 78 | while True: 79 | if self.t >= math.floor(self.T): 80 | if not self.reverse: 81 | if self.fr < self.num_fr - 1: 82 | self.fr += 1 83 | elif self.repeat: 84 | self.fr = 0 85 | elif self.reverse and self.fr > 0: 86 | self.fr -= 1 87 | self.update_pose() 88 | self.t = 0 89 | self.render() 90 | if not self.paused: 91 | self.t += 1 92 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/models/video_reg_net.py: -------------------------------------------------------------------------------- 1 | from hoidini.closd.utils.smpllib.khrylib.utils.torch import * 2 | from torch import nn 3 | from hoidini.closd.utils.smpllib.khrylib.models.resnet import ResNet 4 | from hoidini.closd.utils.smpllib.khrylib.models.tcn import TemporalConvNet 5 | from hoidini.closd.utils.smpllib.khrylib.models.rnn import RNN 6 | from hoidini.closd.utils.smpllib.khrylib.models.mlp import MLP 7 | from hoidini.closd.utils.smpllib.khrylib.models.mobile_net import MobileNet 8 | 9 | 10 | class VideoRegNet(nn.Module): 11 | 12 | def __init__( 13 | self, 14 | out_dim, 15 | v_hdim, 16 | cnn_fdim, 17 | no_cnn=False, 18 | frame_shape=(3, 224, 224), 19 | mlp_dim=(300, 200), 20 | cnn_type="resnet", 21 | v_net_type="lstm", 22 | v_net_param=None, 23 | cnn_rs=True, 24 | causal=False, 25 | ): 26 | super().__init__() 27 | self.out_dim = out_dim 28 | self.cnn_fdim = cnn_fdim 29 | self.v_hdim = v_hdim 30 | self.no_cnn = no_cnn 31 | self.frame_shape = frame_shape 32 | if no_cnn: 33 | self.cnn = None 34 | elif cnn_type == "resnet": 35 | self.cnn = ResNet(cnn_fdim, running_stats=cnn_rs) 36 | elif cnn_type == "mobile": 37 | self.cnn = MobileNet(cnn_fdim) 38 | 39 | self.v_net_type = v_net_type 40 | if v_net_type == "lstm": 41 | self.v_net = RNN(cnn_fdim, v_hdim, v_net_type, bi_dir=not causal) 42 | elif v_net_type == "tcn": 43 | if v_net_param is None: 44 | v_net_param = {} 45 | tcn_size = v_net_param.get("size", [64, 128]) 46 | dropout = v_net_param.get("dropout", 0.2) 47 | kernel_size = v_net_param.get("kernel_size", 3) 48 | assert tcn_size[-1] == v_hdim 49 | self.v_net = TemporalConvNet( 50 | cnn_fdim, 51 | tcn_size, 52 | kernel_size=kernel_size, 53 | dropout=dropout, 54 | causal=causal, 55 | ) 56 | self.mlp = MLP(v_hdim, mlp_dim, "relu") 57 | self.linear = nn.Linear(self.mlp.out_dim, out_dim) 58 | 59 | def forward_v_net(self, x): 60 | if self.v_net_type == "tcn": 61 | x = x.permute(1, 2, 0).contiguous() 62 | x = self.v_net(x) 63 | if self.v_net_type == "tcn": 64 | x = x.permute(2, 0, 1).contiguous() 65 | return x 66 | 67 | def forward(self, x): 68 | # CNN 69 | if self.cnn is not None: 70 | x = self.cnn(x.view((-1,) + self.frame_shape)).view( 71 | -1, x.size(1), self.cnn_fdim 72 | ) 73 | x = self.forward_v_net(x).view(-1, self.v_hdim) 74 | x = self.mlp(x) 75 | x = self.linear(x) 76 | return x 77 | 78 | def get_cnn_feature(self, x): 79 | return self.cnn(x.view((-1,) + self.frame_shape)) 80 | 81 | 82 | if __name__ == "__main__": 83 | net = VideoRegNet(64, 128, 128, v_net_type="tcn", cnn_type="mobile") 84 | input = ones(32, 1, 3, 224, 224) 85 | out = net(input) 86 | print(out.shape) 87 | -------------------------------------------------------------------------------- /hoidini/closd/utils/draw_utils.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import skimage 3 | from skimage.draw import polygon 4 | from skimage.draw import bezier_curve 5 | from skimage.draw import circle_perimeter 6 | from skimage.draw import disk 7 | from scipy import ndimage 8 | import matplotlib 9 | import matplotlib.pyplot as plt 10 | import matplotlib.cm 11 | 12 | 13 | def get_color_gradient(percent, color="Blues"): 14 | # return mpl.colormaps[color](percent)[:3] # for newer versions of plt 15 | return matplotlib.cm.get_cmap(color)(percent)[:3] 16 | 17 | 18 | def agt_color(aidx): 19 | return matplotlib.colors.to_rgb( 20 | plt.rcParams["axes.prop_cycle"].by_key()["color"][aidx % 10] 21 | ) 22 | 23 | 24 | def draw_disk(img_size=80, max_r=10, iterations=3): 25 | shape = (img_size, img_size) 26 | img = np.zeros(shape, dtype=np.uint8) 27 | x, y = np.random.uniform(max_r, img_size - max_r, size=(2)) 28 | radius = int(np.random.uniform(max_r)) 29 | rr, cc = disk((x, y), radius, shape=shape) 30 | np.clip(rr, 0, img_size - 1, out=rr) 31 | np.clip(cc, 0, img_size - 1, out=cc) 32 | img[rr, cc] = 1 33 | return img 34 | 35 | 36 | def draw_circle(img_size=80, max_r=10, iterations=3): 37 | img = np.zeros((img_size, img_size), dtype=np.uint8) 38 | r, c = np.random.uniform(max_r, img_size - max_r, size=(2,)).astype(int) 39 | radius = int(np.random.uniform(max_r)) 40 | rr, cc = circle_perimeter(r, c, radius) 41 | np.clip(rr, 0, img_size - 1, out=rr) 42 | np.clip(cc, 0, img_size - 1, out=cc) 43 | img[rr, cc] = 1 44 | img = ndimage.binary_dilation(img, iterations=1).astype(int) 45 | return img 46 | 47 | 48 | def draw_curve(img_size=80, max_sides=10, iterations=3): 49 | img = np.zeros((img_size, img_size), dtype=np.uint8) 50 | r0, c0, r1, c1, r2, c2 = np.random.uniform(0, img_size, size=(6,)).astype(int) 51 | w = np.random.random() 52 | rr, cc = bezier_curve(r0, c0, r1, c1, r2, c2, w) 53 | np.clip(rr, 0, img_size - 1, out=rr) 54 | np.clip(cc, 0, img_size - 1, out=cc) 55 | img[rr, cc] = 1 56 | img = ndimage.binary_dilation(img, iterations=iterations).astype(int) 57 | return img 58 | 59 | 60 | def draw_polygon(img_size=80, max_sides=10): 61 | img = np.zeros((img_size, img_size), dtype=np.uint8) 62 | num_coord = int(np.random.uniform(3, max_sides)) 63 | r = np.random.uniform(0, img_size, size=(num_coord,)).astype(int) 64 | c = np.random.uniform(0, img_size, size=(num_coord,)).astype(int) 65 | rr, cc = polygon(r, c) 66 | np.clip(rr, 0, img_size - 1, out=rr) 67 | np.clip(cc, 0, img_size - 1, out=cc) 68 | img[rr, cc] = 1 69 | return img 70 | 71 | 72 | def draw_ellipse(img_size=80, max_size=10): 73 | img = np.zeros((img_size, img_size), dtype=np.uint8) 74 | r, c, rradius, cradius = ( 75 | np.random.uniform(max_size, img_size - max_size), 76 | np.random.uniform(max_size, img_size - max_size), 77 | np.random.uniform(1, max_size), 78 | np.random.uniform(1, max_size), 79 | ) 80 | rr, cc = skimage.draw.ellipse(r, c, rradius, cradius) 81 | np.clip(rr, 0, img_size - 1, out=rr) 82 | np.clip(cc, 0, img_size - 1, out=cc) 83 | img[rr, cc] = 1 84 | return img 85 | -------------------------------------------------------------------------------- /hoidini/normalizer.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | import os 3 | import torch 4 | import torch.nn as nn 5 | 6 | 7 | class Normalizer(nn.Module): 8 | """ 9 | Standardizes inputs: x ↦ (x - mean) / std 10 | * If std < eps for a channel (i.e. constant feature) we clamp std to1, 11 | so the feature is only centred (not scaled) and we avoid NaNs/Inf. 12 | """ 13 | 14 | def __init__( 15 | self, 16 | mean: torch.Tensor, 17 | std: torch.Tensor, 18 | eps: float = 1e-5, 19 | disable: bool = False, 20 | ): 21 | super().__init__() 22 | 23 | # ─── safe std: clamp tiny values ──────────────────────────────────────── 24 | safe_std = std.clone() 25 | safe_std[safe_std < eps] = 1.0 # “no‑scale” for constant features 26 | 27 | if (safe_std < eps).any(): 28 | print(f"Warning, std smaller than eps: {safe_std < eps}") 29 | 30 | # ─── register buffers ────────────────────────────────────────────────── 31 | self.register_buffer("mean", mean) 32 | self.register_buffer("std", safe_std) 33 | self.eps = eps 34 | self.disable = disable 35 | 36 | @property 37 | def n_feats(self): 38 | return self.mean.shape[0] 39 | 40 | # ------------------------------------------------------------------------- 41 | # I/O helpers 42 | # ------------------------------------------------------------------------- 43 | @staticmethod 44 | def exists(base_dir: str, split: str = "") -> bool: 45 | return all( 46 | os.path.exists(Normalizer._get_path(obj, base_dir, split)) 47 | for obj in ("mean", "std", "eps") 48 | ) 49 | 50 | @classmethod 51 | def from_dir( 52 | cls, base_dir: str, split: str = "", device=None, disable: bool = False 53 | ): 54 | mean = torch.load(cls._get_path("mean", base_dir, split)).to(device) 55 | std = torch.load(cls._get_path("std", base_dir, split)).to(device) 56 | eps = torch.load(cls._get_path("eps", base_dir, split)) 57 | return cls(mean, std, eps, disable) 58 | 59 | @staticmethod 60 | def _get_path(obj_name: str, base_dir: str, split: str = "") -> str: 61 | fname = f"{obj_name}.pt" if split == "" else f"{obj_name}_{split}.pt" 62 | return os.path.join(base_dir, fname) 63 | 64 | def save(self, base_dir: str, split: str = "") -> None: 65 | os.makedirs(base_dir, exist_ok=True) 66 | torch.save(self.mean, self._get_path("mean", base_dir, split)) 67 | torch.save(self.std, self._get_path("std", base_dir, split)) 68 | torch.save(self.eps, self._get_path("eps", base_dir, split)) 69 | 70 | # ------------------------------------------------------------------------- 71 | # API 72 | # ------------------------------------------------------------------------- 73 | def normalize(self, x: torch.Tensor) -> torch.Tensor: 74 | if self.disable: 75 | return x 76 | return (x - self.mean) / self.std # std already “safe” 77 | 78 | def denormalize(self, x: torch.Tensor) -> torch.Tensor: 79 | if self.disable: 80 | return x 81 | return x * self.std + self.mean 82 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/rl/core/logger_rl.py: -------------------------------------------------------------------------------- 1 | import math 2 | 3 | 4 | class LoggerRL: 5 | 6 | def __init__(self): 7 | self.num_steps = 0 8 | self.num_episodes = 0 9 | self.avg_episode_len = 0 10 | self.total_reward = 0 11 | self.min_episode_reward = math.inf 12 | self.max_episode_reward = -math.inf 13 | self.total_c_reward = 0 14 | self.min_c_reward = math.inf 15 | self.max_c_reward = -math.inf 16 | self.episode_reward = 0 17 | self.avg_episode_reward = 0 18 | self.avg_c_reward = 0 19 | self.avg_episode_c_reward = 0 20 | self.total_c_info = 0 21 | self.avg_c_info = 0 22 | self.avg_episode_c_info = 0 23 | self.sample_time = 0 24 | 25 | def start_episode(self, env): 26 | self.episode_reward = 0 27 | 28 | def step(self, env, reward, c_reward, c_info, info): 29 | self.episode_reward += reward 30 | self.total_c_reward += c_reward 31 | self.total_c_info += c_info 32 | self.min_c_reward = min(self.min_c_reward, c_reward) 33 | self.max_c_reward = max(self.max_c_reward, c_reward) 34 | self.num_steps += 1 35 | 36 | def end_episode(self, env): 37 | self.num_episodes += 1 38 | self.total_reward += self.episode_reward 39 | self.min_episode_reward = min(self.min_episode_reward, self.episode_reward) 40 | self.max_episode_reward = max(self.max_episode_reward, self.episode_reward) 41 | 42 | def end_sampling(self): 43 | self.avg_episode_len = self.num_steps / self.num_episodes 44 | self.avg_episode_reward = self.total_reward / self.num_episodes 45 | self.avg_c_reward = self.total_c_reward / self.num_steps 46 | self.avg_c_info = self.total_c_info / self.num_steps 47 | self.avg_episode_c_reward = self.total_c_reward / self.num_episodes 48 | self.avg_episode_c_info = self.total_c_info / self.num_episodes 49 | 50 | @classmethod 51 | def merge(cls, logger_list): 52 | logger = cls() 53 | logger.total_reward = sum([x.total_reward for x in logger_list]) 54 | logger.num_episodes = sum([x.num_episodes for x in logger_list]) 55 | logger.num_steps = sum([x.num_steps for x in logger_list]) 56 | logger.avg_episode_len = logger.num_steps / logger.num_episodes 57 | logger.avg_episode_reward = logger.total_reward / logger.num_episodes 58 | logger.max_episode_reward = max([x.max_episode_reward for x in logger_list]) 59 | logger.min_episode_reward = max([x.min_episode_reward for x in logger_list]) 60 | logger.total_c_reward = sum([x.total_c_reward for x in logger_list]) 61 | logger.avg_c_reward = logger.total_c_reward / logger.num_steps 62 | logger.max_c_reward = max([x.max_c_reward for x in logger_list]) 63 | logger.min_c_reward = min([x.min_c_reward for x in logger_list]) 64 | logger.total_c_info = sum([x.total_c_info for x in logger_list]) 65 | logger.avg_c_info = logger.total_c_info / logger.num_steps 66 | logger.avg_episode_c_reward = logger.total_c_reward / logger.num_episodes 67 | logger.avg_episode_c_info = logger.total_c_info / logger.num_episodes 68 | return logger 69 | -------------------------------------------------------------------------------- /hoidini/smplx/vertex_joint_selector.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is 4 | # holder of all proprietary rights on this computer program. 5 | # You can only use this computer program if you have closed 6 | # a license agreement with MPG or you get the right to use the computer 7 | # program from someone who is authorized to grant you that right. 8 | # Any use of the computer program without a valid license is prohibited and 9 | # liable to prosecution. 10 | # 11 | # Copyright©2019 Max-Planck-Gesellschaft zur Förderung 12 | # der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute 13 | # for Intelligent Systems. All rights reserved. 14 | # 15 | # Contact: ps-license@tuebingen.mpg.de 16 | 17 | from __future__ import absolute_import 18 | from __future__ import print_function 19 | from __future__ import division 20 | 21 | import numpy as np 22 | 23 | import torch 24 | import torch.nn as nn 25 | 26 | from .utils import to_tensor 27 | 28 | 29 | class VertexJointSelector(nn.Module): 30 | 31 | def __init__( 32 | self, vertex_ids=None, use_hands=True, use_feet_keypoints=True, **kwargs 33 | ): 34 | super(VertexJointSelector, self).__init__() 35 | 36 | extra_joints_idxs = [] 37 | 38 | face_keyp_idxs = np.array( 39 | [ 40 | vertex_ids["nose"], 41 | vertex_ids["reye"], 42 | vertex_ids["leye"], 43 | vertex_ids["rear"], 44 | vertex_ids["lear"], 45 | ], 46 | dtype=np.int64, 47 | ) 48 | 49 | extra_joints_idxs = np.concatenate([extra_joints_idxs, face_keyp_idxs]) 50 | 51 | if use_feet_keypoints: 52 | feet_keyp_idxs = np.array( 53 | [ 54 | vertex_ids["LBigToe"], 55 | vertex_ids["LSmallToe"], 56 | vertex_ids["LHeel"], 57 | vertex_ids["RBigToe"], 58 | vertex_ids["RSmallToe"], 59 | vertex_ids["RHeel"], 60 | ], 61 | dtype=np.int32, 62 | ) 63 | 64 | extra_joints_idxs = np.concatenate([extra_joints_idxs, feet_keyp_idxs]) 65 | 66 | if use_hands: 67 | self.tip_names = ["thumb", "index", "middle", "ring", "pinky"] 68 | 69 | tips_idxs = [] 70 | for hand_id in ["l", "r"]: 71 | for tip_name in self.tip_names: 72 | tips_idxs.append(vertex_ids[hand_id + tip_name]) 73 | 74 | extra_joints_idxs = np.concatenate([extra_joints_idxs, tips_idxs]) 75 | 76 | self.register_buffer( 77 | "extra_joints_idxs", to_tensor(extra_joints_idxs, dtype=torch.long) 78 | ) 79 | 80 | def forward(self, vertices, joints): 81 | extra_joints = torch.index_select( 82 | vertices, 1, self.extra_joints_idxs.to(torch.long) 83 | ) # The '.to(torch.long)'. 84 | # added to make the trace work in c++, 85 | # otherwise you get a runtime error in c++: 86 | # 'index_select(): Expected dtype int32 or int64 for index' 87 | joints = torch.cat([joints, extra_joints], dim=1) 88 | 89 | return joints 90 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/data_loaders/humanml/utils/get_opt.py: -------------------------------------------------------------------------------- 1 | import os 2 | from argparse import Namespace 3 | import re 4 | from os.path import join as pjoin 5 | from hoidini.closd.diffusion_planner.data_loaders.humanml.utils.word_vectorizer import ( 6 | POS_enumerator, 7 | ) 8 | 9 | 10 | def is_float(numStr): 11 | flag = False 12 | numStr = str(numStr).strip().lstrip("-").lstrip("+") # 去除正数(+)、负数(-)符号 13 | try: 14 | reg = re.compile(r"^[-+]?[0-9]+\.[0-9]+$") 15 | res = reg.match(str(numStr)) 16 | if res: 17 | flag = True 18 | except Exception as ex: 19 | print("is_float() - error: " + str(ex)) 20 | return flag 21 | 22 | 23 | def is_number(numStr): 24 | flag = False 25 | numStr = str(numStr).strip().lstrip("-").lstrip("+") # 去除正数(+)、负数(-)符号 26 | if str(numStr).isdigit(): 27 | flag = True 28 | return flag 29 | 30 | 31 | def get_opt(opt_path, device, hml_type=None): 32 | opt = Namespace() 33 | opt_dict = vars(opt) 34 | 35 | skip = ( 36 | "-------------- End ----------------", 37 | "------------ Options -------------", 38 | "\n", 39 | ) 40 | print("Reading", opt_path) 41 | with open(opt_path) as f: 42 | for line in f: 43 | if line.strip() not in skip: 44 | # print(line.strip()) 45 | key, value = line.strip().split(": ") 46 | if value in ("True", "False"): 47 | opt_dict[key] = bool(value) 48 | elif is_float(value): 49 | opt_dict[key] = float(value) 50 | elif is_number(value): 51 | opt_dict[key] = int(value) 52 | else: 53 | opt_dict[key] = str(value) 54 | 55 | # print(opt) 56 | opt_dict["which_epoch"] = "latest" 57 | opt.save_root = pjoin(opt.checkpoints_dir, opt.dataset_name, opt.name) 58 | opt.model_dir = pjoin(opt.save_root, "model") 59 | opt.meta_dir = pjoin(opt.save_root, "meta") 60 | opt.hml_type = hml_type 61 | 62 | if opt.dataset_name == "t2m": 63 | opt.data_root = "./dataset/HumanML3D" 64 | data_dir = ( 65 | "new_joint_vecs_global_root" 66 | if hml_type == "global_root" 67 | else "new_joint_vecs" 68 | ) 69 | opt.motion_dir = pjoin(opt.data_root, data_dir) 70 | opt.text_dir = pjoin(opt.data_root, "texts") 71 | opt.joints_num = 22 72 | opt.dim_pose = 263 73 | opt.max_motion_length = 196 74 | elif opt.dataset_name == "kit": 75 | assert ( 76 | opt.hml_type is None 77 | ) # "KIT dataset does not support global representation" 78 | opt.data_root = "./dataset/KIT-ML" 79 | opt.motion_dir = pjoin(opt.data_root, "new_joint_vecs") 80 | opt.text_dir = pjoin(opt.data_root, "texts") 81 | opt.joints_num = 21 82 | opt.dim_pose = 251 83 | opt.max_motion_length = 196 84 | else: 85 | raise KeyError("Dataset not recognized") 86 | 87 | opt.dim_word = 300 88 | opt.num_classes = 200 // opt.unit_length 89 | opt.dim_pos_ohot = len(POS_enumerator) 90 | opt.is_train = False 91 | opt.is_continue = False 92 | opt.device = device 93 | 94 | return opt 95 | -------------------------------------------------------------------------------- /hoidini/objects_fk.py: -------------------------------------------------------------------------------- 1 | from collections import namedtuple 2 | import numpy as np 3 | import torch 4 | import torch.nn as nn 5 | from smplx.lbs import batch_rodrigues 6 | 7 | model_output = namedtuple("output", ["vertices", "global_orient", "transl"]) 8 | 9 | 10 | class ObjectModel(nn.Module): 11 | def __init__(self, v_template, batch_size=1, dtype=torch.float32): 12 | """3D rigid object model 13 | 14 | Parameters 15 | ---------- 16 | v_template: np.array Vx3, dtype = np.float32 17 | The vertices of the object 18 | batch_size: int, N, optional 19 | The batch size used for creating the model variables 20 | 21 | dtype: torch.dtype 22 | The data type for the created variables 23 | """ 24 | 25 | super().__init__() 26 | 27 | self.dtype = dtype 28 | 29 | # Mean template vertices 30 | v_template = np.repeat(v_template[np.newaxis], batch_size, axis=0) 31 | self.register_buffer("v_template", torch.tensor(v_template, dtype=dtype)) 32 | 33 | transl = torch.tensor( 34 | np.zeros((batch_size, 3)), dtype=dtype, requires_grad=True 35 | ) 36 | self.register_parameter("transl", nn.Parameter(transl, requires_grad=True)) 37 | 38 | global_orient = torch.tensor( 39 | np.zeros((batch_size, 3)), dtype=dtype, requires_grad=True 40 | ) 41 | self.register_parameter( 42 | "global_orient", nn.Parameter(global_orient, requires_grad=True) 43 | ) 44 | 45 | self.batch_size = batch_size 46 | 47 | def forward(self, global_orient=None, transl=None, v_template=None, **kwargs): 48 | """Forward pass for the object model 49 | 50 | Parameters 51 | ---------- 52 | global_orient: torch.tensor, optional, shape Bx3 53 | If given, ignore the member variable and use it as the global 54 | rotation of the body. Useful if someone wishes to predicts this 55 | with an external model. (default=None) 56 | 57 | transl: torch.tensor, optional, shape Bx3 58 | If given, ignore the member variable `transl` and use it 59 | instead. For example, it can used if the translation 60 | `transl` is predicted from some external model. 61 | (default=None) 62 | v_template: torch.tensor, optional, shape BxVx3 63 | The new object vertices to overwrite the default vertices 64 | 65 | Returns 66 | ------- 67 | output: ModelOutput 68 | A named tuple of type `ModelOutput` 69 | """ 70 | if global_orient is None: 71 | global_orient = self.global_orient 72 | if transl is None: 73 | transl = self.transl 74 | if v_template is None: 75 | v_template = self.v_template 76 | v_template = torch.tensor(v_template, dtype=self.dtype) 77 | 78 | rot_mats = batch_rodrigues(global_orient.view(-1, 3)).view( 79 | [self.batch_size, 3, 3] 80 | ) 81 | 82 | vertices = torch.matmul(v_template, rot_mats) + transl.unsqueeze(dim=1) 83 | 84 | output = model_output( 85 | vertices=vertices, global_orient=global_orient, transl=transl 86 | ) 87 | 88 | return output 89 | -------------------------------------------------------------------------------- /hoidini/closd/utils/parse_task.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2018-2023, NVIDIA Corporation 2 | # All rights reserved. 3 | # 4 | # Redistribution and use in source and binary forms, with or without 5 | # modification, are permitted provided that the following conditions are met: 6 | # 7 | # 1. Redistributions of source code must retain the above copyright notice, this 8 | # list of conditions and the following disclaimer. 9 | # 10 | # 2. Redistributions in binary form must reproduce the above copyright notice, 11 | # this list of conditions and the following disclaimer in the documentation 12 | # and/or other materials provided with the distribution. 13 | # 14 | # 3. Neither the name of the copyright holder nor the names of its 15 | # contributors may be used to endorse or promote products derived from 16 | # this software without specific prior written permission. 17 | # 18 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 19 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 20 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 22 | # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 23 | # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 25 | # CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 26 | # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | 29 | from hoidini.closd.env.tasks.humanoid import Humanoid 30 | from hoidini.closd.env.tasks.humanoid_amp import HumanoidAMP 31 | from hoidini.closd.env.tasks.humanoid_im import HumanoidIm 32 | from hoidini.closd.env.tasks.vec_task_wrappers import VecTaskPythonWrapper 33 | from hoidini.closd.env.tasks.closd import CLoSD 34 | from hoidini.closd.env.tasks.closd_task import CLoSDTask 35 | from hoidini.closd.env.tasks.closd_t2m import CLoSDT2M 36 | from hoidini.closd.env.tasks.closd_multitask import CLoSDMultiTask 37 | from hoidini.closd.env.tasks.closd_sequence import CLoSDSequence 38 | 39 | from isaacgym import rlgpu 40 | 41 | import json 42 | import numpy as np 43 | 44 | 45 | def warn_task_name(): 46 | raise Exception( 47 | "Unrecognized task!\nTask should be one of: [BallBalance, Cartpole, CartpoleYUp, Ant, Humanoid, Anymal, FrankaCabinet, Quadcopter, ShadowHand, ShadowHandLSTM, ShadowHandFFOpenAI, ShadowHandFFOpenAITest, ShadowHandOpenAI, ShadowHandOpenAITest, Ingenuity]" 48 | ) 49 | 50 | 51 | def parse_task(args, cfg, cfg_train, sim_params): 52 | 53 | # create native task and pass custom config 54 | device_id = args.device_id 55 | rl_device = args.rl_device 56 | 57 | cfg["seed"] = cfg_train.get("seed", -1) 58 | cfg_task = cfg["env"] 59 | cfg_task["seed"] = cfg["seed"] 60 | 61 | task = eval(args.task)( 62 | cfg=cfg, 63 | sim_params=sim_params, 64 | physics_engine=args.physics_engine, 65 | device_type=args.device, 66 | device_id=device_id, 67 | headless=args.headless, 68 | ) 69 | env = VecTaskPythonWrapper( 70 | task, rl_device, cfg_train.get("clip_observations", np.inf) 71 | ) 72 | 73 | return task, env 74 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/utils/lightning_utils.py: -------------------------------------------------------------------------------- 1 | import glob 2 | import os 3 | import sys 4 | import pdb 5 | import os.path as osp 6 | 7 | 8 | import pytorch_lightning as pl 9 | from pytorch_lightning.callbacks import ModelCheckpoint 10 | from pathlib import Path 11 | from pytorch_lightning.utilities import rank_zero_only 12 | from pytorch_lightning.loggers import LightningLoggerBase 13 | from pytorch_lightning.loggers.base import rank_zero_experiment 14 | import logging 15 | 16 | 17 | class PeriodicCheckpoint(ModelCheckpoint): 18 | def __init__(self, every: int, *args, **kwargs): 19 | super().__init__(*args, **kwargs) 20 | self.every = every 21 | 22 | def on_train_batch_end( 23 | self, trainer: pl.Trainer, pl_module: pl.LightningModule, *args, **kwargs 24 | ): 25 | if (pl_module.global_step + 1) % self.every == 0: 26 | assert self.dirpath is not None 27 | current = Path(self.dirpath) / f"model_{pl_module.global_step}.ckpt" 28 | trainer.save_checkpoint(current) 29 | 30 | 31 | class TextLogger(LightningLoggerBase): 32 | def __init__(self, cfg, filename, file_handle=True): 33 | super().__init__() 34 | self.cfg = cfg 35 | self.logger = logger = logging.getLogger(filename) 36 | logger.propagate = False 37 | logger.setLevel(logging.DEBUG) 38 | # create console handler with a higher log level 39 | ch = logging.StreamHandler() 40 | ch.setLevel(logging.INFO) 41 | stream_formatter = logging.Formatter("%(message)s") 42 | ch.setFormatter(stream_formatter) 43 | logger.addHandler(ch) 44 | 45 | if file_handle: 46 | # create file handler which logs even debug messages 47 | os.makedirs(os.path.dirname(filename), exist_ok=True) 48 | fh = logging.FileHandler(filename, mode="a") 49 | fh.setLevel(logging.DEBUG) 50 | file_formatter = logging.Formatter("[%(asctime)s] %(message)s") 51 | fh.setFormatter(file_formatter) 52 | logger.addHandler(fh) 53 | 54 | @property 55 | def name(self): 56 | 57 | return "TextLogger" 58 | 59 | @property 60 | @rank_zero_experiment 61 | def experiment(self): 62 | # Return the experiment object associated with this logger. 63 | pass 64 | 65 | @property 66 | def version(self): 67 | # Return the experiment version, int or str. 68 | return "0.1" 69 | 70 | @rank_zero_only 71 | def log_hyperparams(self, params): 72 | # params is an argparse.Namespace 73 | # your code to record hyperparameters goes here 74 | pass 75 | 76 | @rank_zero_only 77 | def log_metrics(self, metrics, step): 78 | # metrics is a dictionary of metric names and values 79 | # your code to record metrics goes here 80 | log_str = "".join([f"{k} : {v:.3f} \t" for k, v in metrics.items()]) 81 | self.logger.info(log_str) 82 | 83 | @rank_zero_only 84 | def save(self): 85 | # Optional. Any code necessary to save logger data goes here 86 | # If you implement this, remember to call `super().save()` 87 | # at the start of the method (important for aggregation of metrics) 88 | # super().save() 89 | pass 90 | 91 | @rank_zero_only 92 | def finalize(self, status): 93 | # Optional. Any code that needs to be run after training 94 | # finishes goes here 95 | pass 96 | -------------------------------------------------------------------------------- /hoidini/amasstools/slerp.py: -------------------------------------------------------------------------------- 1 | from torch import FloatTensor, LongTensor, Tensor, Size, lerp, zeros_like 2 | from torch.linalg import norm 3 | 4 | # from this gist: https://gist.github.com/Birch-san/230ac46f99ec411ed5907b0a3d728efa 5 | 6 | # adapted to PyTorch from: 7 | # https://gist.github.com/dvschultz/3af50c40df002da3b751efab1daddf2c 8 | # most of the extra complexity is to support: 9 | # - many-dimensional vectors 10 | # - v0 or v1 with last dim all zeroes, or v0 ~colinear with v1 11 | # - falls back to lerp() 12 | # - conditional logic implemented with parallelism rather than Python loops 13 | # - many-dimensional tensor for t 14 | # - you can ask for batches of slerp outputs by making t more-dimensional than the vectors 15 | # - slerp( 16 | # v0: torch.Size([2,3]), 17 | # v1: torch.Size([2,3]), 18 | # t: torch.Size([4,1,1]), 19 | # ) 20 | # - this makes it interface-compatible with lerp() 21 | 22 | 23 | def slerp( 24 | v0: FloatTensor, v1: FloatTensor, t: float | FloatTensor, DOT_THRESHOLD=0.9995 25 | ): 26 | """ 27 | Spherical linear interpolation 28 | Args: 29 | v0: Starting vector 30 | v1: Final vector 31 | t: Float value between 0.0 and 1.0 32 | DOT_THRESHOLD: Threshold for considering the two vectors as 33 | colinear. Not recommended to alter this. 34 | Returns: 35 | Interpolation vector between v0 and v1 36 | """ 37 | assert v0.shape == v1.shape, "shapes of v0 and v1 must match" 38 | 39 | # Normalize the vectors to get the directions and angles 40 | v0_norm: FloatTensor = norm(v0, dim=-1) 41 | v1_norm: FloatTensor = norm(v1, dim=-1) 42 | 43 | v0_normed: FloatTensor = v0 / v0_norm.unsqueeze(-1) 44 | v1_normed: FloatTensor = v1 / v1_norm.unsqueeze(-1) 45 | 46 | # Dot product with the normalized vectors 47 | dot: FloatTensor = (v0_normed * v1_normed).sum(-1) 48 | dot_mag: FloatTensor = dot.abs() 49 | 50 | # if dp is NaN, it's because the v0 or v1 row was filled with 0s 51 | # If absolute value of dot product is almost 1, vectors are ~colinear, so use lerp 52 | gotta_lerp: LongTensor = dot_mag.isnan() | (dot_mag > DOT_THRESHOLD) 53 | can_slerp: LongTensor = ~gotta_lerp 54 | 55 | t_batch_dim_count: int = max(0, t.dim() - v0.dim()) if isinstance(t, Tensor) else 0 56 | t_batch_dims: Size = ( 57 | t.shape[:t_batch_dim_count] if isinstance(t, Tensor) else Size([]) 58 | ) 59 | out: FloatTensor = zeros_like(v0.expand(*t_batch_dims, *[-1] * v0.dim())) 60 | 61 | # if no elements are lerpable, our vectors become 0-dimensional, preventing broadcasting 62 | if gotta_lerp.any(): 63 | lerped: FloatTensor = lerp(v0, v1, t) 64 | out: FloatTensor = lerped.where(gotta_lerp.unsqueeze(-1), out) 65 | 66 | # if no elements are slerpable, our vectors become 0-dimensional, preventing broadcasting 67 | if can_slerp.any(): 68 | # Calculate initial angle between v0 and v1 69 | theta_0: FloatTensor = dot.arccos().unsqueeze(-1) 70 | sin_theta_0: FloatTensor = theta_0.sin() 71 | # Angle at timestep t 72 | theta_t: FloatTensor = theta_0 * t 73 | sin_theta_t: FloatTensor = theta_t.sin() 74 | # Finish the slerp algorithm 75 | s0: FloatTensor = (theta_0 - theta_t).sin() / sin_theta_0 76 | s1: FloatTensor = sin_theta_t / sin_theta_0 77 | slerped: FloatTensor = s0 * v0 + s1 * v1 78 | out: FloatTensor = slerped.where(can_slerp.unsqueeze(-1), out) 79 | 80 | return out 81 | -------------------------------------------------------------------------------- /hoidini/object_conditioning/object_encoder_pointwise.py: -------------------------------------------------------------------------------- 1 | """ 2 | Adapted from https://github.com/pyg-team/pytorch_geometric/blob/master/examples/pointnet2_segmentation.py 3 | """ 4 | 5 | from typing import List 6 | import torch 7 | from torch_geometric.nn import MLP, knn_interpolate 8 | from torch_geometric.typing import WITH_TORCH_CLUSTER 9 | import torch_geometric 10 | 11 | from hoidini.object_conditioning.object_encoder_global import GlobalSAModule, SAModule 12 | from hoidini.object_conditioning.object_pointcloud_dataset import ( 13 | get_grab_point_cloud_dataset, 14 | pyg_collate_wrapper, 15 | ) 16 | from hoidini.resource_paths import GRAB_DATA_PATH 17 | 18 | if not WITH_TORCH_CLUSTER: 19 | quit("This example requires 'torch-cluster'") 20 | 21 | 22 | class FPModule(torch.nn.Module): 23 | def __init__(self, k, nn): 24 | super().__init__() 25 | self.k = k 26 | self.nn = nn 27 | 28 | def forward(self, x, pos, batch, x_skip, pos_skip, batch_skip): 29 | x = knn_interpolate(x, pos, pos_skip, batch, batch_skip, k=self.k) 30 | if x_skip is not None: 31 | x = torch.cat([x, x_skip], dim=1) 32 | x = self.nn(x) 33 | return x, pos_skip, batch_skip 34 | 35 | 36 | class ObjectPointwiseEncoder(torch.nn.Module): 37 | def __init__(self, out_dim: int): 38 | super().__init__() 39 | 40 | self.sa1_module = SAModule(0.2, 0.2, MLP([3 + 3, 64, 64, 128])) 41 | self.sa2_module = SAModule(0.25, 0.4, MLP([128 + 3, 128, 128, 256])) 42 | self.sa3_module = GlobalSAModule(MLP([256 + 3, 256, 512, 1024])) 43 | 44 | self.fp3_module = FPModule(1, MLP([1024 + 256, 256, 256])) 45 | self.fp2_module = FPModule(3, MLP([256 + 128, 256, 128])) 46 | self.fp1_module = FPModule(3, MLP([128 + 3, 128, 128, out_dim])) 47 | 48 | def forward(self, data): 49 | if data.x is not None: 50 | sa0_out = (data.x, data.pos, data.batch) 51 | elif data.normal is not None: 52 | sa0_out = (data.normal, data.pos, data.batch) 53 | else: 54 | raise ValueError("Data object must have either 'x' or 'normal' attribute") 55 | sa1_out = self.sa1_module(*sa0_out) 56 | sa2_out = self.sa2_module(*sa1_out) 57 | sa3_out = self.sa3_module(*sa2_out) 58 | 59 | fp3_out = self.fp3_module(*sa3_out, *sa2_out) 60 | fp2_out = self.fp2_module(*fp3_out, *sa1_out) 61 | x, _, _ = self.fp1_module(*fp2_out, *sa0_out) 62 | 63 | # split to batches 64 | x = torch.stack( 65 | torch.split(x, (data.ptr[1:] - data.ptr[:-1]).tolist(), dim=0), dim=0 66 | ) 67 | return x 68 | 69 | 70 | def pyg_batch_to_torch_batch( 71 | data: torch_geometric.data.Batch, attr: str 72 | ) -> torch.Tensor: 73 | x = torch.split(getattr(data, attr), (data.ptr[1:] - data.ptr[:-1]).tolist(), dim=0) 74 | return torch.stack(x, dim=0) 75 | 76 | 77 | def main(): 78 | dataset = get_grab_point_cloud_dataset(GRAB_DATA_PATH, use_cache=False) 79 | data_list = [dataset[0], dataset[1], dataset[2]] 80 | batch = pyg_collate_wrapper(data_list) 81 | 82 | print("Batch normal shape:", batch.normal.shape) 83 | print("Batch pos shape:", batch.pos.shape) 84 | print("Batch batch shape:", batch.batch.shape) 85 | encoder = ObjectPointwiseEncoder(512) 86 | print("encoder #params (M):", sum(p.numel() for p in encoder.parameters()) / 1e6) 87 | embd = encoder(batch) 88 | print(embd.shape) 89 | 90 | 91 | if __name__ == "__main__": 92 | main() 93 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/data_loaders/humanml/utils/paramUtil.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | # Define a kinematic tree for the skeletal struture 4 | kit_kinematic_chain = [ 5 | [0, 11, 12, 13, 14, 15], 6 | [0, 16, 17, 18, 19, 20], 7 | [0, 1, 2, 3, 4], 8 | [3, 5, 6, 7], 9 | [3, 8, 9, 10], 10 | ] 11 | 12 | kit_raw_offsets = np.array( 13 | [ 14 | [0, 0, 0], 15 | [0, 1, 0], 16 | [0, 1, 0], 17 | [0, 1, 0], 18 | [0, 1, 0], 19 | [1, 0, 0], 20 | [0, -1, 0], 21 | [0, -1, 0], 22 | [-1, 0, 0], 23 | [0, -1, 0], 24 | [0, -1, 0], 25 | [1, 0, 0], 26 | [0, -1, 0], 27 | [0, -1, 0], 28 | [0, 0, 1], 29 | [0, 0, 1], 30 | [-1, 0, 0], 31 | [0, -1, 0], 32 | [0, -1, 0], 33 | [0, 0, 1], 34 | [0, 0, 1], 35 | ] 36 | ) 37 | 38 | t2m_raw_offsets = np.array( 39 | [ 40 | [0, 0, 0], 41 | [1, 0, 0], 42 | [-1, 0, 0], 43 | [0, 1, 0], 44 | [0, -1, 0], 45 | [0, -1, 0], 46 | [0, 1, 0], 47 | [0, -1, 0], 48 | [0, -1, 0], 49 | [0, 1, 0], 50 | [0, 0, 1], 51 | [0, 0, 1], 52 | [0, 1, 0], 53 | [1, 0, 0], 54 | [-1, 0, 0], 55 | [0, 0, 1], 56 | [0, -1, 0], 57 | [0, -1, 0], 58 | [0, -1, 0], 59 | [0, -1, 0], 60 | [0, -1, 0], 61 | [0, -1, 0], 62 | ] 63 | ) 64 | 65 | t2m_kinematic_chain = [ 66 | [0, 2, 5, 8, 11], 67 | [0, 1, 4, 7, 10], 68 | [0, 3, 6, 9, 12, 15], 69 | [9, 14, 17, 19, 21], 70 | [9, 13, 16, 18, 20], 71 | ] 72 | t2m_left_hand_chain = [ 73 | [20, 22, 23, 24], 74 | [20, 34, 35, 36], 75 | [20, 25, 26, 27], 76 | [20, 31, 32, 33], 77 | [20, 28, 29, 30], 78 | ] 79 | t2m_right_hand_chain = [ 80 | [21, 43, 44, 45], 81 | [21, 46, 47, 48], 82 | [21, 40, 41, 42], 83 | [21, 37, 38, 39], 84 | [21, 49, 50, 51], 85 | ] 86 | 87 | 88 | smpl_24_kinematic_chain = [ 89 | [0, 2, 5, 8, 11], 90 | [0, 1, 4, 7, 10], 91 | [0, 3, 6, 9, 12, 15], 92 | [9, 14, 17, 19, 21, 23], 93 | [9, 13, 16, 18, 20, 22], 94 | ] 95 | 96 | 97 | smplx_52_kinematic_chain = [ 98 | [0, 1, 4, 7, 10], 99 | [0, 2, 5, 8, 11], 100 | [0, 3, 6, 9, 12, 15], 101 | [12, 13, 16, 18, 20, 34, 35, 36], 102 | [20, 22, 23, 24], 103 | [20, 25, 26, 27], 104 | [20, 31, 32, 33], 105 | [20, 28, 29, 30], 106 | [12, 14, 17, 19, 21, 49, 50, 51], 107 | [21, 37, 38, 39], 108 | [21, 40, 41, 42], 109 | [21, 46, 47, 48], 110 | [21, 43, 44, 45], 111 | ] 112 | 113 | 114 | kit_tgt_skel_id = "03950" 115 | 116 | t2m_tgt_skel_id = "000021" 117 | 118 | 119 | def get_kinematic_chain_smplx_52(BONE_52_52_INDS): 120 | from collections import defaultdict 121 | 122 | chidren = defaultdict(list) 123 | for f, s in BONE_52_52_INDS: 124 | chidren[f].append(s) 125 | 126 | def scan(cur, cur_chain): 127 | if len(chidren[cur]) > 1: 128 | if cur != 0: 129 | chains.append(cur_chain) 130 | for child in chidren[cur]: 131 | new_chain = [cur, child] 132 | scan(child, new_chain) 133 | elif len(chidren[cur]) == 1: 134 | cur_chain.append(chidren[cur][0]) 135 | scan(chidren[cur][0], cur_chain) 136 | else: 137 | chains.append(cur_chain) 138 | 139 | chains = [] 140 | scan(0, []) 141 | return chains 142 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/mocap/pose.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import math 3 | from bvh import Bvh 4 | from hoidini.closd.utils.smpllib.khrylib.utils.transformation import ( 5 | quaternion_slerp, 6 | quaternion_from_euler, 7 | euler_from_quaternion, 8 | ) 9 | 10 | 11 | def load_amc_file(fname, scale): 12 | 13 | with open(fname) as f: 14 | content = f.readlines() 15 | 16 | bone_addr = dict() 17 | poses = [] 18 | cur_pos = None 19 | fr = 1 20 | for line in content: 21 | line_words = line.split() 22 | cmd = line_words[0] 23 | if cmd == str(fr): 24 | if cur_pos: 25 | poses.append(np.array(cur_pos)) 26 | cur_pos = [] 27 | fr += 1 28 | elif cur_pos is not None: 29 | start_ind = len(cur_pos) 30 | if cmd == "root": 31 | cur_pos += [float(word) * scale for word in line_words[1:4]] 32 | cur_pos += [math.radians(float(word)) for word in line_words[4:]] 33 | elif cmd == "lfoot" or cmd == "rfoot": 34 | cur_pos += reversed( 35 | [math.radians(float(word)) for word in line_words[1:]] 36 | ) 37 | if len(cur_pos) < 3: 38 | cur_pos.insert(-1, 0.0) 39 | else: 40 | cur_pos += reversed( 41 | [math.radians(float(word)) for word in line_words[1:]] 42 | ) 43 | if fr == 2: 44 | end_ind = len(cur_pos) 45 | bone_addr[cmd] = (start_ind, end_ind) 46 | 47 | if cur_pos: 48 | poses.append(np.array(cur_pos)) 49 | poses = np.vstack(poses) 50 | return poses, bone_addr 51 | 52 | 53 | def load_bvh_file(fname, skeleton): 54 | with open(fname) as f: 55 | mocap = Bvh(f.read()) 56 | 57 | # build bone_addr 58 | bone_addr = dict() 59 | start_ind = 0 60 | for bone in skeleton.bones: 61 | end_ind = start_ind + len(bone.channels) 62 | bone_addr[bone.name] = (start_ind, end_ind) 63 | start_ind = end_ind 64 | dof_num = start_ind 65 | 66 | poses = np.zeros((mocap.nframes, dof_num)) 67 | for i in range(mocap.nframes): 68 | for bone in skeleton.bones: 69 | trans = np.array(mocap.frame_joint_channels(i, bone.name, bone.channels)) 70 | if bone == skeleton.root: 71 | trans[:3] *= skeleton.len_scale 72 | trans[3:6] = np.deg2rad(trans[3:6]) 73 | else: 74 | trans = np.deg2rad(trans) 75 | start_ind, end_ind = bone_addr[bone.name] 76 | poses[i, start_ind:end_ind] = trans 77 | 78 | return poses, bone_addr 79 | 80 | 81 | def lin_interp(pose1, pose2, t): 82 | pose_t = (1 - t) * pose1 + t * pose2 83 | if np.any(np.abs(pose2[3:] - pose1[3:]) > np.pi * 0.5): 84 | pose_t[3:] = pose1[3:] if t < 0.5 else pose2[3:] 85 | return pose_t 86 | 87 | 88 | def interpolated_traj(poses, sample_t=0.030, mocap_fr=120, interp_func=lin_interp): 89 | N = poses.shape[0] 90 | T = float(N - 1) / mocap_fr 91 | num = int(math.floor(T / sample_t)) 92 | sampling_times = np.arange(num + 1) * sample_t * mocap_fr 93 | 94 | poses_samp = [] 95 | for t in sampling_times: 96 | start = int(math.floor(t)) 97 | end = min(int(math.ceil(t)), poses.shape[0] - 1) 98 | pose_interp = interp_func(poses[start, :], poses[end, :], t - math.floor(t)) 99 | poses_samp.append(pose_interp) 100 | poses_samp = np.vstack(poses_samp) 101 | 102 | return poses_samp 103 | -------------------------------------------------------------------------------- /hoidini/blender_utils/visualize_mesh_figure_blender.py: -------------------------------------------------------------------------------- 1 | import bpy 2 | import torch 3 | from tqdm import tqdm 4 | import numpy as np 5 | import pickle 6 | from hoidini.amasstools.geometry import axis_angle_to_quaternion 7 | import hoidini.smplx as smplx 8 | from hoidini.datasets.smpldata import get_smpl_model_path 9 | 10 | 11 | def get_smpl_template(model_name, gender="neutral", smpl_models_path=None): 12 | smpl_models_path = get_smpl_model_path(model_name, gender, smpl_models_path) 13 | with open(smpl_models_path, "rb") as f: 14 | d = pickle.load(f, encoding="latin") 15 | sbj_mesh_faces = d["f"] 16 | sbj_mesh_vertices = d["v_template"] 17 | return sbj_mesh_faces, sbj_mesh_vertices 18 | 19 | 20 | def create_new_blender_object_with_mesh(name, initial_verts, faces, color=None): 21 | mesh = bpy.data.meshes.new(name=f"Mesh_{name}") 22 | mesh.from_pydata(initial_verts, [], faces) 23 | mesh.update() 24 | obj = bpy.data.objects.new(name=f"Mesh_{name}", object_data=mesh) 25 | scene = bpy.context.scene 26 | scene.collection.objects.link(obj) 27 | 28 | if color is not None: 29 | material_name = f"Mat_{name}" 30 | material = bpy.data.materials.get(material_name) 31 | if material is None: 32 | material = bpy.data.materials.new(name=material_name) 33 | material.diffuse_color = color if len(color) == 4 else color + (1.0,) 34 | 35 | obj.data.materials.clear() 36 | obj.data.materials.append(material) 37 | return obj 38 | 39 | 40 | def animate_mesh(name, mesh_faces, verts_anim, color=None): 41 | if isinstance(mesh_faces, torch.Tensor): 42 | mesh_faces = mesh_faces.cpu().numpy() 43 | if isinstance(verts_anim, torch.Tensor): 44 | verts_anim = verts_anim.cpu().numpy() 45 | sbj_blend_obj = create_new_blender_object_with_mesh( 46 | name, verts_anim[0], mesh_faces, color=color 47 | ) 48 | sbj_blend_verts = sbj_blend_obj.data.vertices 49 | for frame in tqdm(range(len(verts_anim)), desc=f"animate mesh {name}"): 50 | for v_i, v in enumerate(sbj_blend_verts): 51 | v.co = verts_anim[frame][v_i] 52 | v.keyframe_insert("co", frame=frame) 53 | 54 | 55 | def animate_rigid_mesh(name, mesh_faces, v_template, poses, transl, color=None): 56 | """ 57 | poses is in axis-angle format 58 | """ 59 | if isinstance(poses, np.ndarray): 60 | poses = torch.from_numpy(poses) 61 | else: 62 | poses = poses.cpu() 63 | quats_wxyz = axis_angle_to_quaternion(poses) 64 | quats_wxyz = quats_wxyz.numpy() 65 | quats_xyzw = quats_wxyz[:, [1, 2, 3, 0]] 66 | mesh_faces = ( 67 | mesh_faces.cpu().numpy() if isinstance(mesh_faces, torch.Tensor) else mesh_faces 68 | ) 69 | v_template = ( 70 | v_template.cpu().numpy() if isinstance(v_template, torch.Tensor) else v_template 71 | ) 72 | transl = transl.cpu().numpy() if isinstance(transl, torch.Tensor) else transl 73 | blend_obj = create_new_blender_object_with_mesh( 74 | name, v_template, mesh_faces, color=color 75 | ) 76 | for frame in tqdm(range(len(transl)), desc=f"animate rigid mesh {name}"): 77 | blend_obj.location = transl[frame] 78 | blend_obj.rotation_quaternion = quats_xyzw[frame] 79 | blend_obj.keyframe_insert("location", frame=frame) 80 | blend_obj.keyframe_insert("rotation_quaternion", frame=frame) 81 | 82 | 83 | def main(): 84 | smplx_output = smplx.utils.SMPLOutput() 85 | sbj_mesh_faces, _ = get_smpl_template() 86 | bpy.ops.wm.read_factory_settings(use_empty=True) 87 | sbj_verts_anim = smplx_output.vertices.detach().cpu().numpy() 88 | animate_mesh("SMPLX", sbj_mesh_faces, sbj_verts_anim) 89 | -------------------------------------------------------------------------------- /hoidini/configs/experiments/resample_result.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 6, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "from datasets.smpldata import SmplData\n", 10 | "from inference_hoi_model import HoiResult\n", 11 | "from object_contact_prediction.cpdm_dno_conds import find_contiguous_static_blocks\n", 12 | "import torch\n", 13 | "import torch.nn.functional as F\n", 14 | "from typing import Optional\n", 15 | "\n", 16 | "def resample(smpl_data: SmplData, n_frames: int) -> SmplData:\n", 17 | " def _resample_tensor(x: Optional[torch.Tensor]) -> Optional[torch.Tensor]:\n", 18 | " if x is None:\n", 19 | " return None\n", 20 | " T = x.shape[0]\n", 21 | " if T == n_frames:\n", 22 | " return x\n", 23 | " # flatten non-time dims into \"features\"\n", 24 | " rest = x.shape[1:]\n", 25 | " x_flat = x.reshape(T, -1).transpose(0, 1).unsqueeze(0) # (1, features, T)\n", 26 | " y_flat = F.interpolate(x_flat, size=n_frames, mode='linear', align_corners=True)\n", 27 | " y = y_flat.squeeze(0).transpose(0, 1).reshape(n_frames, *rest)\n", 28 | " return y\n", 29 | "\n", 30 | " data_dict = smpl_data.to_dict()\n", 31 | " for key, val in data_dict.items():\n", 32 | " if isinstance(val, torch.Tensor):\n", 33 | " data_dict[key] = _resample_tensor(val)\n", 34 | " return SmplData(**data_dict)\n", 35 | "\n", 36 | "def remove_short_false_segments(arr: torch.Tensor, min_length: int) -> torch.Tensor:\n", 37 | " arr = arr.bool()\n", 38 | " padded = torch.cat([torch.tensor([False]), arr, torch.tensor([False])])\n", 39 | " diffs = padded[1:] != padded[:-1]\n", 40 | " idxs = torch.nonzero(diffs).flatten()\n", 41 | " starts, ends = idxs[::2], idxs[1::2]\n", 42 | " for s, e in zip(starts, ends):\n", 43 | " if not arr[s] and (e - s) < min_length:\n", 44 | " arr[s:e] = True\n", 45 | " return arr\n", 46 | "\n", 47 | "def get_longest_contact_range(smpldata: SmplData):\n", 48 | " has_contact = (smpldata.contact > 0.5).any(dim=1) # (seq, n_anchors)\n", 49 | " has_contact = remove_short_false_segments(has_contact, min_length=2)\n", 50 | " contact_blocks = find_contiguous_static_blocks(~has_contact)\n", 51 | " start, end = sorted(contact_blocks, key=lambda rng: rng[1] - rng[0])[-1]\n", 52 | " return start, end\n", 53 | "\n", 54 | "result_path = \"/home/dcor/roeyron/trumans_utils/results/Results_May20/1b_inference_only_0/cphoi__cphoi_05011024_c15p100_v0__model000120000__0014__s10_bowl_pass_1__bowl__The_person_is_passing_a_bowl__phase0.pickle\"\n", 55 | "\n", 56 | "# Option 1 - with resampling\n", 57 | "result = HoiResult.load(result_path)\n", 58 | "smpldata = result.smpldata\n", 59 | "start, end = get_longest_contact_range(smpldata)\n", 60 | "smpldata = smpldata.cut(start, end)\n", 61 | "smpldata = resample(smpldata, 30)\n", 62 | "\n", 63 | "# Option 2 - with resampling\n", 64 | "result = HoiResult.load(result_path)\n", 65 | "smpldata = result.smpldata\n", 66 | "start, end = get_longest_contact_range(smpldata)\n", 67 | "mid = (start + end) // 2\n", 68 | "smpldata = smpldata.cut(mid-15, mid+15)" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [] 77 | } 78 | ], 79 | "metadata": { 80 | "language_info": { 81 | "name": "python" 82 | } 83 | }, 84 | "nbformat": 4, 85 | "nbformat_minor": 2 86 | } 87 | -------------------------------------------------------------------------------- /hoidini/closd/diffusion_planner/train/train_platforms.py: -------------------------------------------------------------------------------- 1 | import os 2 | import glob 3 | 4 | 5 | class TrainPlatform: 6 | def __init__(self, save_dir, *args, **kwargs): 7 | self.path, file = os.path.split(save_dir) 8 | self.name = kwargs.get("name", file) 9 | 10 | def report_scalar(self, name, value, iteration, group_name=None): 11 | pass 12 | 13 | def report_media(self, title, series, iteration, local_path): 14 | pass 15 | 16 | def report_args(self, args, name): 17 | pass 18 | 19 | def close(self): 20 | pass 21 | 22 | 23 | class ClearmlPlatform(TrainPlatform): 24 | def __init__(self, save_dir): 25 | from clearml import Task 26 | 27 | path, name = os.path.split(save_dir) 28 | self.task = Task.init(project_name="RL", task_name=name) 29 | self.logger = self.task.get_logger() 30 | 31 | def report_scalar(self, name, value, iteration, group_name): 32 | self.logger.report_scalar( 33 | title=group_name, series=name, iteration=iteration, value=value 34 | ) 35 | 36 | def report_media(self, title, series, iteration, local_path): 37 | self.logger.report_media( 38 | title=title, series=series, iteration=iteration, local_path=local_path 39 | ) 40 | 41 | def report_args(self, args, name): 42 | self.task.connect(args, name=name) 43 | 44 | def close(self): 45 | self.task.close() 46 | 47 | 48 | class TensorboardPlatform(TrainPlatform): 49 | def __init__(self, save_dir): 50 | from torch.utils.tensorboard import SummaryWriter 51 | 52 | self.writer = SummaryWriter(log_dir=save_dir) 53 | 54 | def report_scalar(self, name, value, iteration, group_name=None): 55 | self.writer.add_scalar(f"{group_name}/{name}", value, iteration) 56 | 57 | def close(self): 58 | self.writer.close() 59 | 60 | 61 | class NoPlatform(TrainPlatform): 62 | def __init__(self, save_dir, *args, **kwargs): 63 | pass 64 | 65 | 66 | class WandBPlatform(TrainPlatform): 67 | import wandb 68 | 69 | def __init__(self, save_dir, config=None, *args, **kwargs): 70 | super().__init__(save_dir, *args, **kwargs) 71 | self.wandb.login( 72 | host=os.getenv("WANDB_BASE_URL"), key=os.getenv("WANDB_API_KEY") 73 | ) 74 | self.wandb.init( 75 | project=kwargs.get("project", "amass_smplrifke"), 76 | name=self.name, 77 | id=self.name, # in order to send continued runs to the same record 78 | resume="allow", # in order to send continued runs to the same record 79 | # entity='tau-motion', 80 | save_code=True, 81 | config=config, 82 | ) # config can also be sent via report_args() 83 | 84 | def report_scalar(self, name, value, iteration, group_name=None): 85 | self.wandb.log({name: value}, step=iteration) 86 | 87 | def report_media(self, title, series, iteration, local_path): 88 | files = glob.glob(f"{local_path}/*.mp4") 89 | self.wandb.log( 90 | {series: [self.wandb.Video(file, format="mp4", fps=20) for file in files]}, 91 | step=iteration, 92 | ) 93 | 94 | def report_args(self, args, name, allow_val_change=False): 95 | self.wandb.config.update( 96 | args, allow_val_change=allow_val_change 97 | ) # , allow_val_change=True) # use allow_val_change ONLY if you want to change existing args (e.g., overwrite) 98 | 99 | def watch_model(self, *args, **kwargs): 100 | self.wandb.watch(args, kwargs) 101 | 102 | def close(self): 103 | self.wandb.finish() 104 | -------------------------------------------------------------------------------- /hoidini/datasets/resources/gen_train_seq_paths.json: -------------------------------------------------------------------------------- 1 | [ 2 | "s9/lightbulb_pass_1", 3 | "s9/cubemedium_pass_1", 4 | "s9/torusmedium_pass_1", 5 | "s9/mouse_pass_1", 6 | "s7/mouse_pass_1", 7 | "s9/pyramidmedium_inspect_1", 8 | "s9/elephant_inspect_1", 9 | "s4/duck_pass_1", 10 | "s9/gamecontroller_pass_1", 11 | "s9/cylindermedium_inspect_1", 12 | "s4/waterbottle_shake_1", 13 | "s3/airplane_pass_1", 14 | "s4/cylindersmall_pass_1", 15 | "s7/spherelarge_pass_1", 16 | "s7/elephant_pass_1", 17 | "s9/apple_eat_1", 18 | "s4/mug_drink_2", 19 | "s5/duck_pass_1", 20 | "s9/hand_pass_1", 21 | "s9/flashlight_pass_1", 22 | "s1/flute_pass_1", 23 | "s3/stamp_stamp_1", 24 | "s5/spheresmall_inspect_1", 25 | "s4/gamecontroller_pass_1", 26 | "s6/camera_pass_1", 27 | "s3/waterbottle_pass_1", 28 | "s3/toothbrush_brush_1", 29 | "s5/alarmclock_pass_1", 30 | "s7/hand_pass_1", 31 | "s7/spheresmall_pass_1", 32 | "s2/apple_eat_1", 33 | "s5/cylindersmall_inspect_1", 34 | "s6/stapler_pass_1", 35 | "s4/teapot_pour_2", 36 | "s5/waterbottle_open_1", 37 | "s8/headphones_pass_1", 38 | "s5/cubelarge_pass_1", 39 | "s3/stanfordbunny_pass_1", 40 | "s2/cubemedium_inspect_1", 41 | "s6/bowl_pass_1", 42 | "s9/waterbottle_shake_1", 43 | "s8/airplane_fly_1", 44 | "s2/stapler_pass_1", 45 | "s5/spheremedium_inspect_1", 46 | "s6/phone_pass_1", 47 | "s6/airplane_pass_1", 48 | "s8/torusmedium_pass_1", 49 | "s1/cylindermedium_pass_1", 50 | "s3/gamecontroller_play_1", 51 | "s2/cylinderlarge_pass_1", 52 | "s6/stamp_pass_1", 53 | "s6/airplane_fly_1", 54 | "s8/binoculars_pass_1", 55 | "s1/phone_pass_1", 56 | "s4/elephant_inspect_1", 57 | "s8/watch_pass_1", 58 | "s1/fryingpan_cook_2", 59 | "s3/spheremedium_inspect_1", 60 | "s6/cylinderlarge_pass_1", 61 | "s2/pyramidmedium_inspect_1", 62 | "s3/doorknob_use_1", 63 | "s6/waterbottle_pass_1", 64 | "s6/torusmedium_pass_1", 65 | "s7/piggybank_pass_1", 66 | "s1/pyramidsmall_pass_1", 67 | "s8/wineglass_pass_1", 68 | "s5/pyramidmedium_pass_1", 69 | "s4/bowl_drink_1", 70 | "s3/lightbulb_screw_1", 71 | "s6/headphones_use_1", 72 | "s8/piggybank_use_1", 73 | "s7/headphones_use_1", 74 | "s8/stapler_staple_2", 75 | "s2/scissors_use_1", 76 | "s5/flashlight_on_2", 77 | "s1/eyeglasses_pass_1", 78 | "s7/waterbottle_drink_1", 79 | "s3/stanfordbunny_inspect_1", 80 | "s2/stamp_stamp_1", 81 | "s4/airplane_pass_1", 82 | "s7/cup_drink_1", 83 | "s5/bowl_drink_1", 84 | "s6/waterbottle_shake_1", 85 | "s2/camera_takepicture_2", 86 | "s9/piggybank_use_1", 87 | "s8/mug_offhand_1", 88 | "s9/wineglass_drink_2", 89 | "s3/mug_pass_1", 90 | "s2/mug_drink_2", 91 | "s8/hand_pass_1", 92 | "s6/stanfordbunny_inspect_1", 93 | "s7/doorknob_use_1", 94 | "s8/banana_peel_1", 95 | "s6/knife_peel_1", 96 | "s1/torussmall_inspect_1", 97 | "s3/eyeglasses_clean_1", 98 | "s8/doorknob_use_2", 99 | "s9/banana_eat_1", 100 | "s1/cubemedium_offhand_1", 101 | "s6/binoculars_see_1", 102 | "s8/hammer_use_2", 103 | "s1/torusmedium_inspect_1", 104 | "s7/doorknob_use_2", 105 | "s6/teapot_pour_1", 106 | "s6/toothpaste_squeeze_1_Retake", 107 | "s1/camera_takepicture_1", 108 | "s1/flashlight_on_1", 109 | "s1/cylinderlarge_inspect_1", 110 | "s1/mug_drink_2", 111 | "s6/flashlight_on_2", 112 | "s1/eyeglasses_offhand_1", 113 | "s8/eyeglasses_wear_1", 114 | "s1/hammer_use_3", 115 | "s1/gamecontroller_play_1", 116 | "s1/mouse_use_1" 117 | ] -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/khrylib/models/tcn.py: -------------------------------------------------------------------------------- 1 | import torch.nn as nn 2 | from torch.nn.utils import weight_norm 3 | from hoidini.closd.utils.smpllib.khrylib.utils.torch import * 4 | 5 | 6 | class Chomp1d(nn.Module): 7 | def __init__(self, chomp_size): 8 | super().__init__() 9 | self.chomp_size = chomp_size 10 | 11 | def forward(self, x): 12 | return x[:, :, : -self.chomp_size].contiguous() 13 | 14 | 15 | class TemporalBlock(nn.Module): 16 | def __init__( 17 | self, n_inputs, n_outputs, kernel_size, stride, dilation, dropout, causal 18 | ): 19 | super().__init__() 20 | padding = (kernel_size - 1) * dilation // (1 if causal else 2) 21 | modules = [] 22 | self.conv1 = weight_norm( 23 | nn.Conv1d( 24 | n_inputs, 25 | n_outputs, 26 | kernel_size, 27 | stride=stride, 28 | padding=padding, 29 | dilation=dilation, 30 | ) 31 | ) 32 | modules.append(self.conv1) 33 | if causal: 34 | modules.append(Chomp1d(padding)) 35 | modules.append(nn.ReLU()) 36 | if dropout > 0: 37 | modules.append(nn.Dropout(dropout)) 38 | 39 | self.conv2 = weight_norm( 40 | nn.Conv1d( 41 | n_outputs, 42 | n_outputs, 43 | kernel_size, 44 | stride=stride, 45 | padding=padding, 46 | dilation=dilation, 47 | ) 48 | ) 49 | modules.append(self.conv2) 50 | if causal: 51 | modules.append(Chomp1d(padding)) 52 | modules.append(nn.ReLU()) 53 | if dropout > 0: 54 | modules.append(nn.Dropout(dropout)) 55 | 56 | self.net = nn.Sequential(*modules) 57 | 58 | self.downsample = ( 59 | nn.Conv1d(n_inputs, n_outputs, 1) if n_inputs != n_outputs else None 60 | ) 61 | self.relu = nn.ReLU() 62 | self.init_weights() 63 | 64 | def init_weights(self): 65 | self.conv1.weight.data.normal_(0, 0.01) 66 | self.conv2.weight.data.normal_(0, 0.01) 67 | if self.downsample is not None: 68 | self.downsample.weight.data.normal_(0, 0.01) 69 | 70 | def forward(self, x): 71 | out = self.net(x) 72 | res = x if self.downsample is None else self.downsample(x) 73 | return self.relu(out + res) 74 | 75 | 76 | class TemporalConvNet(nn.Module): 77 | def __init__( 78 | self, num_inputs, num_channels, kernel_size=3, dropout=0.2, causal=False 79 | ): 80 | super().__init__() 81 | assert kernel_size % 2 == 1 82 | layers = [] 83 | num_levels = len(num_channels) 84 | for i in range(num_levels): 85 | dilation_size = 2**i 86 | in_channels = num_inputs if i == 0 else num_channels[i - 1] 87 | out_channels = num_channels[i] 88 | layers += [ 89 | TemporalBlock( 90 | in_channels, 91 | out_channels, 92 | kernel_size, 93 | stride=1, 94 | dilation=dilation_size, 95 | dropout=dropout, 96 | causal=causal, 97 | ) 98 | ] 99 | 100 | self.network = nn.Sequential(*layers) 101 | 102 | def forward(self, x): 103 | return self.network(x) 104 | 105 | 106 | if __name__ == "__main__": 107 | tcn = TemporalConvNet(4, [1, 2, 8], kernel_size=3, causal=False) 108 | input = zeros(3, 4, 80) 109 | out = tcn(input) 110 | print(tcn) 111 | print(out.shape) 112 | -------------------------------------------------------------------------------- /hoidini/closd/utils/smpllib/utils/replay_data.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | 4 | 5 | from mujoco_py import load_model_from_path, MjSim 6 | from hoidini.closd.utils.smpllib.khrylib.rl.envs.common.mjviewer import MjViewer 7 | import pickle 8 | import argparse 9 | import glfw 10 | import math 11 | 12 | parser = argparse.ArgumentParser() 13 | parser.add_argument("--model_id", type=str, default="human36m_v1") 14 | parser.add_argument("--offset_z", type=float, default=0.0) 15 | parser.add_argument("--start_take", type=str, default=None) 16 | parser.add_argument("--dataset", type=str, default="h36m/data_qpos_h36m") 17 | args = parser.parse_args() 18 | 19 | model_file = f"assets/mujoco_models/{args.model_id}.xml" 20 | model = load_model_from_path(model_file) 21 | sim = MjSim(model) 22 | viewer = MjViewer(sim) 23 | 24 | 25 | def key_callback(key, action, mods): 26 | global T, fr, paused, stop, offset_z, take_ind, reverse 27 | 28 | if action != glfw.RELEASE: 29 | return False 30 | elif key == glfw.KEY_D: 31 | T *= 1.5 32 | elif key == glfw.KEY_F: 33 | T = max(1, T / 1.5) 34 | elif key == glfw.KEY_R: 35 | stop = True 36 | elif key == glfw.KEY_W: 37 | fr = 0 38 | update_mocap() 39 | elif key == glfw.KEY_S: 40 | reverse = not reverse 41 | elif key == glfw.KEY_C: 42 | take_ind = (take_ind + 1) % len(takes) 43 | load_take() 44 | update_mocap() 45 | elif key == glfw.KEY_Z: 46 | take_ind = (take_ind - 1) % len(takes) 47 | load_take() 48 | update_mocap() 49 | elif key == glfw.KEY_RIGHT: 50 | if fr < qpos_traj.shape[0] - 1: 51 | fr += 1 52 | update_mocap() 53 | elif key == glfw.KEY_LEFT: 54 | if fr > 0: 55 | fr -= 1 56 | update_mocap() 57 | elif key == glfw.KEY_UP: 58 | offset_z += 0.001 59 | update_mocap() 60 | elif key == glfw.KEY_DOWN: 61 | offset_z -= 0.001 62 | update_mocap() 63 | elif key == glfw.KEY_SPACE: 64 | paused = not paused 65 | else: 66 | return False 67 | return True 68 | 69 | 70 | def update_mocap(): 71 | print(f"{take[0]} {take[1]}: [{fr}, {qpos_traj.shape[0]}] dz: {offset_z:.3f}") 72 | print(qpos_traj.shape) 73 | sim.data.qpos[:] = qpos_traj[fr] 74 | sim.data.qpos[2] += offset_z 75 | sim.forward() 76 | 77 | 78 | def load_take(): 79 | global qpos_traj, fr, take 80 | take = takes[take_ind] 81 | fr = 0 82 | qpos_traj = data[take[0]][take[1]] 83 | 84 | 85 | data = pickle.load(open(os.path.expanduser("data/{}.p").format(args.dataset), "rb")) 86 | takes = [ 87 | (subject, action) for subject, s_data in data.items() for action in s_data.keys() 88 | ] 89 | 90 | qpos_traj = None 91 | take = None 92 | take_ind = ( 93 | 0 if args.start_take is None else takes.index(tuple(args.start_take.split(","))) 94 | ) 95 | fr = 0 96 | offset_z = args.offset_z 97 | # load_take() 98 | 99 | T = 10 100 | paused = False 101 | stop = False 102 | reverse = False 103 | glfw.set_window_size(viewer.window, 1000, 960) 104 | glfw.set_window_pos(viewer.window, 400, 0) 105 | viewer._hide_overlay = True 106 | viewer.cam.distance = 10 107 | viewer.cam.elevation = -20 108 | viewer.cam.azimuth = 90 109 | viewer.custom_key_callback = key_callback 110 | 111 | load_take() 112 | update_mocap() 113 | t = 0 114 | while not stop: 115 | if t >= math.floor(T): 116 | if not reverse and fr < qpos_traj.shape[0] - 1: 117 | fr += 1 118 | update_mocap() 119 | elif reverse and fr > 0: 120 | fr -= 1 121 | update_mocap() 122 | t = 0 123 | 124 | viewer.render() 125 | if not paused: 126 | t += 1 127 | -------------------------------------------------------------------------------- /hoidini/datasets/resources/gen_test_seq_paths.json: -------------------------------------------------------------------------------- 1 | [ 2 | "s10/pyramidsmall_inspect_1", 3 | "s10/toothpaste_pass_1", 4 | "s10/spheremedium_inspect_1", 5 | "s10/spheremedium_pass_1", 6 | "s10/stanfordbunny_pass_1", 7 | "s10/headphones_pass_1", 8 | "s10/torussmall_inspect_1", 9 | "s10/torusmedium_inspect_1", 10 | "s10/waterbottle_pass_1", 11 | "s10/toruslarge_inspect_1", 12 | "s10/spherelarge_pass_1", 13 | "s10/stanfordbunny_inspect_1", 14 | "s10/spheresmall_inspect_1", 15 | "s10/watch_pass_1", 16 | "s10/train_pass_1", 17 | "s10/duck_pass_1", 18 | "s10/torussmall_pass_1", 19 | "s10/phone_pass_1", 20 | "s10/alarmclock_pass_1", 21 | "s10/stamp_stamp_1", 22 | "s10/wineglass_toast_1", 23 | "s10/stapler_pass_1", 24 | "s10/toruslarge_pass_1", 25 | "s10/hammer_pass_1", 26 | "s10/gamecontroller_pass_1", 27 | "s10/spheresmall_pass_1", 28 | "s10/teapot_pass_1", 29 | "s10/cubemedium_pass_1", 30 | "s10/banana_pass_1", 31 | "s10/torusmedium_pass_1", 32 | "s10/mouse_pass_1", 33 | "s10/scissors_pass_1", 34 | "s10/elephant_pass_1", 35 | "s10/cylindersmall_pass_1", 36 | "s10/pyramidmedium_inspect_1", 37 | "s10/cubemedium_inspect_1", 38 | "s10/apple_pass_1", 39 | "s10/lightbulb_pass_1", 40 | "s10/airplane_pass_1", 41 | "s10/train_play_1", 42 | "s10/waterbottle_pour_1", 43 | "s10/stamp_pass_1", 44 | "s10/bowl_pass_1", 45 | "s10/flute_pass_1", 46 | "s10/cubelarge_pass_1", 47 | "s10/alarmclock_see_1", 48 | "s10/pyramidmedium_pass_1", 49 | "s10/camera_takepicture_3", 50 | "s10/mug_toast_1", 51 | "s10/hand_pass_1", 52 | "s10/piggybank_pass_1", 53 | "s10/camera_pass_1", 54 | "s10/cylindersmall_inspect_1", 55 | "s10/cubesmall_inspect_1", 56 | "s10/cup_pour_1", 57 | "s10/wineglass_pass_1", 58 | "s10/stapler_staple_1", 59 | "s10/cubesmall_pass_1", 60 | "s10/cylindermedium_inspect_1", 61 | "s10/mug_pass_1", 62 | "s10/cup_pass_1", 63 | "s10/headphones_use_1", 64 | "s10/mouse_use_1", 65 | "s10/knife_pass_1", 66 | "s10/pyramidlarge_pass_1", 67 | "s10/flashlight_on_1", 68 | "s10/mug_drink_1", 69 | "s10/wineglass_drink_1", 70 | "s10/camera_takepicture_2", 71 | "s10/cylindermedium_pass_1", 72 | "s10/scissors_use_1", 73 | "s10/cylinderlarge_pass_1", 74 | "s10/teapot_pour_2", 75 | "s10/binoculars_pass_1", 76 | "s10/stapler_staple_2", 77 | "s10/airplane_fly_1", 78 | "s10/cup_drink_1", 79 | "s10/spherelarge_inspect_1", 80 | "s10/mug_drink_2", 81 | "s10/apple_eat_1", 82 | "s10/teapot_pour_1", 83 | "s10/camera_takepicture_1", 84 | "s10/bowl_drink_2", 85 | "s10/hammer_use_1", 86 | "s10/waterbottle_drink_1", 87 | "s10/knife_chop_1", 88 | "s10/hammer_use_2", 89 | "s10/wineglass_drink_2", 90 | "s10/doorknob_use_2", 91 | "s10/fryingpan_cook_2", 92 | "s10/fryingpan_cook_1", 93 | "s10/hammer_use_3", 94 | "s10/cubelarge_inspect_1", 95 | "s10/binoculars_see_1", 96 | "s10/lightbulb_screw_1", 97 | "s10/knife_peel_1", 98 | "s10/elephant_inspect_1", 99 | "s10/flashlight_on_2", 100 | "s10/gamecontroller_play_1", 101 | "s10/cup_drink_2", 102 | "s10/banana_peel_1", 103 | "s10/eyeglasses_wear_1", 104 | "s10/hand_inspect_1", 105 | "s10/toothpaste_squeeze_2", 106 | "s10/flute_play_1", 107 | "s10/camera_browse_1", 108 | "s10/doorknob_use_1", 109 | "s10/banana_peel_2", 110 | "s10/cylinderlarge_inspect_1", 111 | "s10/bowl_drink_1_Retake", 112 | "s10/hand_shake_1", 113 | "s10/toothpaste_squeeze_1", 114 | "s10/piggybank_use_1", 115 | "s10/phone_call_1", 116 | "s10/banana_eat_1" 117 | ] --------------------------------------------------------------------------------