16 |
15 |
16 |
30 |
31 | - :trophy: **SOTA simulation performance** Seer achieves state-of-the-art performance on simulation benchmarks CALVIN ABC-D and LIBERO-LONG.
32 | - :muscle: **Impressive Real-World performance** Seer demonstrates strong effectiveness and generalization across diverse real-world downstream tasks.
33 |
34 | ## :door: Getting Started
35 | We provide step-by-step guidance for running Seer in simulations and real-world experiments.
36 | Follow the specific instructions for a seamless setup.
37 |
38 | ### Simulation
39 | #### CALVIN ABC-D
40 | - [Installation](docs/CALVIN_ABC-D_INSTALL.md)
41 | - [Running Code](docs/CALVIN_ABC-D_RUN.md)
42 | #### LIBERO LONG
43 | - [Installation](docs/LIBERO_LONG_INSTALL.md)
44 | - [Running Code](docs/LIBERO_LONG_RUN.md)
45 | ### Real-World
46 | #### Real-World (Quick Training w & w/o pre-training)
47 | For users aiming to train Seer from scratch or fine-tune it, we provide comprehensive instructions for environment setup, downstream task data preparation, training, and deployment.
48 | - [Installation](docs/REAL-WORLD_INSTALL.md)
49 | - [Post-processing](docs/REAL-WORLD_POSTPROCESS.md)
50 | - [Fine-tuning & Scratch](docs/REAL-WORLD_FT_SC.md)
51 | - [Inference](docs/REAL-WORLD_INFERENCE.md)
52 |
53 | #### Real-World (Pre-training)
54 | This section details the pre-training process of Seer in real-world experiments, including environment setup, dataset preparation, and training procedures. Downstream task processing and fine-tuning are covered in [Real-World (Quick Training w & w/o pre-training)](#real-world-qs).
55 | - [Installation](docs/REAL-WORLD_INSTALL.md)
56 | - [Pre-processing](docs/REAL-WORLD_PREPROCESS.md)
57 | - [Pre-training](docs/REAL-WORLD_PRETRAIN.md)
58 |
59 |
60 | ## :pencil2: Checkpoints
61 | Relevant checkpoints are available on the [website](https://drive.google.com/drive/folders/1F3IE95z2THAQ_lt3DKUFdRGc86Thsnc7?usp=sharing).
62 | |Model|Checkpoint|
63 | |:------:|:------:|
64 | |CALVIN ABC-D|[Seer](https://drive.google.com/drive/folders/17Gv9snGCkViuhHmzN3eTWlI0tMfGSGT3?usp=sharing) (Avg.Len. : 3.98) / [Seer Large](https://drive.google.com/drive/folders/1AFabqfDEi69oMo0FTGhEiH2QSRLYBR9r?usp=drive_link) (Avg.Len. : 4.30)|
65 | |Real-World|[Seer (Droid Pre-trained)](https://drive.google.com/drive/folders/1rT8JKLhJGIo97jfYUm2JiFUrogOq-dgJ?usp=drive_link)|
66 |
67 | ## 📆 TODO
68 | - [x] Release real-world expriment code.
69 | - [x] Release CALVIN ABC-D experiment code (Seer).
70 | - [x] Release the evaluation code of Seer-Large on CALVIN ABC-D experiment.
71 | - [x] Release the training code of Seer-Large on CALVIN ABC-D experiment.
72 | - [x] Release LIBERO-LONG experiment code.
73 | - [ ] Release simpleseer, a quick scratch training & deploying code.
74 |
75 | ## License
76 |
77 | All assets and code are under the [Apache 2.0 license](./LICENSE) unless specified otherwise.
78 |
79 | ## Citation
80 | If you find the project helpful for your research, please consider citing our paper:
81 | ```bibtex
82 | @article{tian2024predictive,
83 | title={Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation},
84 | author={Tian, Yang and Yang, Sizhe and Zeng, Jia and Wang, Ping and Lin, Dahua and Dong, Hao and Pang, Jiangmiao},
85 | journal={arXiv preprint arXiv:2412.15109},
86 | year={2024}
87 | }
88 | ```
89 |
90 | ## Acknowledgment
91 | This project builds upon [GR-1](https://github.com/bytedance/GR-1) and [Roboflamingo](https://github.com/RoboFlamingo/RoboFlamingo). We thank these teams for their open-source contributions.
92 |
--------------------------------------------------------------------------------
/real_preprocess/octo_oxe_data_utils/obs_transforms.py:
--------------------------------------------------------------------------------
1 | """
2 | Contains observation-level transforms used in the octo data pipeline. These transforms operate on the
3 | "observation" dictionary, and are applied at a per-frame level.
4 | """
5 | from typing import Mapping, Tuple, Union
6 |
7 | from absl import logging
8 | import dlimp as dl
9 | import tensorflow as tf
10 |
11 |
12 | def augment(
13 | obs: dict, seed: tf.Tensor, augment_kwargs: Union[dict, Mapping[str, dict]]
14 | ) -> dict:
15 | """Augments images, skipping padding images."""
16 | image_names = {key[6:] for key in obs if key.startswith("image_")}
17 |
18 | # "augment_order" is required in augment_kwargs, so if it's there, we can assume that the user has passed
19 | # in a single augmentation dict (otherwise, we assume that the user has passed in a mapping from image
20 | # name to augmentation dict)
21 | if "augment_order" in augment_kwargs:
22 | augment_kwargs = {name: augment_kwargs for name in image_names}
23 |
24 | for i, name in enumerate(image_names):
25 | if name not in augment_kwargs:
26 | continue
27 | kwargs = augment_kwargs[name]
28 | logging.debug(f"Augmenting image_{name} with kwargs {kwargs}")
29 | obs[f"image_{name}"] = tf.cond(
30 | obs["pad_mask_dict"][f"image_{name}"],
31 | lambda: dl.transforms.augment_image(
32 | obs[f"image_{name}"],
33 | **kwargs,
34 | seed=seed + i, # augment each image differently
35 | ),
36 | lambda: obs[f"image_{name}"], # skip padding images
37 | )
38 |
39 | return obs
40 |
41 |
42 | def decode_and_resize(
43 | obs: dict,
44 | resize_size: Union[Tuple[int, int], Mapping[str, Tuple[int, int]]],
45 | depth_resize_size: Union[Tuple[int, int], Mapping[str, Tuple[int, int]]],
46 | ) -> dict:
47 | """Decodes images and depth images, and then optionally resizes them."""
48 | # just gets the part after "image_" or "depth_"
49 | image_names = {key[6:] for key in obs if key.startswith("image_")}
50 | depth_names = {key[6:] for key in obs if key.startswith("depth_")}
51 |
52 | if isinstance(resize_size, tuple):
53 | resize_size = {name: resize_size for name in image_names}
54 | if isinstance(depth_resize_size, tuple):
55 | depth_resize_size = {name: depth_resize_size for name in depth_names}
56 |
57 | for name in image_names:
58 | if name not in resize_size:
59 | logging.warning(
60 | f"No resize_size was provided for image_{name}. This will result in 1x1 "
61 | "padding images, which may cause errors if you mix padding and non-padding images."
62 | )
63 | image = obs[f"image_{name}"]
64 | if image.dtype == tf.string:
65 | if tf.strings.length(image) == 0:
66 | # this is a padding image
67 | image = tf.zeros((*resize_size.get(name, (1, 1)), 3), dtype=tf.uint8)
68 | else:
69 | image = tf.io.decode_image(
70 | image, expand_animations=False, dtype=tf.uint8
71 | )
72 | elif image.dtype != tf.uint8:
73 | raise ValueError(
74 | f"Unsupported image dtype: found image_{name} with dtype {image.dtype}"
75 | )
76 | if name in resize_size:
77 | image = dl.transforms.resize_image(image, size=resize_size[name])
78 | obs[f"image_{name}"] = image
79 |
80 | for name in depth_names:
81 | if name not in depth_resize_size:
82 | logging.warning(
83 | f"No depth_resize_size was provided for depth_{name}. This will result in 1x1 "
84 | "padding depth images, which may cause errors if you mix padding and non-padding images."
85 | )
86 | depth = obs[f"depth_{name}"]
87 | if depth.dtype == tf.string:
88 | if tf.strings.length(depth) == 0:
89 | # this is a padding image
90 | depth = tf.zeros(
91 | (*depth_resize_size.get(name, (1, 1)), 1), dtype=tf.float32
92 | )
93 | else:
94 | depth = tf.io.decode_image(
95 | depth, expand_animations=False, dtype=tf.float32
96 | )[..., 0]
97 | elif depth.dtype != tf.float32:
98 | raise ValueError(
99 | f"Unsupported depth dtype: found depth_{name} with dtype {depth.dtype}"
100 | )
101 | if name in depth_resize_size:
102 | depth = dl.transforms.resize_depth_image(
103 | depth, size=depth_resize_size[name]
104 | )
105 | obs[f"depth_{name}"] = depth
106 |
107 | return obs
108 |
--------------------------------------------------------------------------------
/eval_calvin.py:
--------------------------------------------------------------------------------
1 |
2 | import os
3 | import random
4 | import numpy as np
5 | import torch
6 | import wandb
7 | import clip
8 | from torch.nn.parallel import DistributedDataParallel as DDP
9 | from torch.distributed.elastic.multiprocessing.errors import record
10 | from models.seer_model import SeerAgent
11 | from utils.data_utils import get_calvin_dataset, get_calvin_val_dataset
12 | from utils.distributed_utils import init_distributed_device, world_info_from_env
13 | from utils.eval_utils_calvin import eval_one_epoch_calvin_ddp
14 | from utils.arguments_utils import get_parser
15 |
16 |
17 | def random_seed(seed=42, rank=0):
18 | torch.manual_seed(seed + rank)
19 | np.random.seed(seed + rank)
20 | random.seed(seed + rank)
21 |
22 | @record
23 | def main():
24 | parser = get_parser(is_eval=True)
25 | args = parser.parse_args()
26 | if args.save_checkpoints_to_wandb and args.save_checkpoint and not args.report_to_wandb:
27 | raise ValueError("save_checkpoints_to_wandb requires report_to_wandb")
28 | if args.offline:
29 | os.environ["WANDB_MODE"] = "offline"
30 | os.environ["TRANSFORMERS_OFFLINE"] = "1"
31 | args.local_rank, args.rank, args.world_size = world_info_from_env()
32 | device_id = init_distributed_device(args)
33 | print("device_id: ", device_id)
34 | random_seed(args.seed)
35 | model = SeerAgent(
36 | finetune_type=args.finetune_type,
37 | clip_device=device_id,
38 | vit_checkpoint_path=args.vit_checkpoint_path,
39 | sequence_length=args.sequence_length,
40 | num_resampler_query=args.num_resampler_query,
41 | num_obs_token_per_image=args.num_obs_token_per_image,
42 | calvin_input_image_size=args.calvin_input_image_size,
43 | patch_size=args.patch_size,
44 | action_pred_steps=args.action_pred_steps,
45 | obs_pred=args.obs_pred,
46 | atten_only_obs=args.atten_only_obs,
47 | attn_robot_proprio_state=args.attn_robot_proprio_state,
48 | atten_goal=args.atten_goal,
49 | atten_goal_state=args.atten_goal_state,
50 | mask_l_obs_ratio=args.mask_l_obs_ratio,
51 | transformer_layers=args.transformer_layers,
52 | hidden_dim=args.hidden_dim,
53 | transformer_heads=args.transformer_heads,
54 | phase=args.phase,
55 | gripper_width=args.gripper_width,
56 | )
57 | calvin_dataset = get_calvin_dataset(args, model.image_processor, clip, epoch=0)
58 | random_seed(args.seed, args.rank)
59 | print(f"Start running training on rank {args.rank}.")
60 | if args.rank == 0 and args.report_to_wandb:
61 | wandb.init(
62 | project=args.wandb_project,
63 | entity=args.wandb_entity,
64 | name=args.run_name,
65 | config=vars(args),
66 | )
67 | device_id = args.rank % torch.cuda.device_count()
68 | if args.precision == "bf16" or args.precision == "amp_bfloat16" or args.precision == "amp_bf16":
69 | model = model.bfloat16()
70 | elif args.precision == "fp16":
71 | model = model.half()
72 | elif args.precision == "fp32":
73 | model = model.float()
74 | if 'vision_encoder' in args.bf16_module:
75 | model.vision_encoder.bfloat16()
76 | if "causal_transformer" in args.bf16_module:
77 | model.transformer_backbone.bfloat16()
78 | if "image_decoder" in args.bf16_module:
79 | model.image_decoder.bfloat16()
80 | model.image_decoder_obs_pred_projector.bfloat16()
81 | model.clip_model.requires_grad_(False)
82 | model.vision_encoder.requires_grad_(False)
83 | model = model.to(device_id)
84 | model._init_model_type()
85 | ddp_model = DDP(model, device_ids=[device_id], find_unused_parameters=True)
86 | if args.resume_from_checkpoint is not None:
87 | if args.rank == 0:
88 | print(f"Loading checkpoint from {args.resume_from_checkpoint}")
89 | checkpoint = torch.load(args.resume_from_checkpoint, map_location="cpu")
90 | ddp_model.load_state_dict(checkpoint["model_state_dict"], False)
91 | ddp_model.eval()
92 | eval_log_dir = 'evaluate'
93 | if args.finetune_type == "calvin":
94 | eval_one_epoch_calvin_ddp(
95 | args=args,
96 | model=ddp_model,
97 | image_processor=model.image_processor,
98 | tokenizer=clip,
99 | dataset_path=args.calvin_dataset,
100 | future_act_len=args.future_act_len,
101 | eval_log_dir=eval_log_dir,
102 | debug=args.visualize,
103 | reset=args.reset,
104 | diverse_inst=args.diverse_inst
105 | )
106 | else:
107 | raise NotImplementedError
108 |
109 | if __name__ == "__main__":
110 | os.environ["NCCL_BLOCKING_WAIT"] = "0"
111 | main()
112 |
--------------------------------------------------------------------------------
/real_preprocess/octo_oxe_data_utils/traj_transforms.py:
--------------------------------------------------------------------------------
1 | """
2 | Contains trajectory transforms used in the octo data pipeline. Trajectory transforms operate on a dictionary
3 | that represents a single trajectory, meaning each tensor has the same leading dimension (the trajectory
4 | length).
5 | """
6 | import logging
7 |
8 | import tensorflow as tf
9 |
10 |
11 | def chunk_act_obs(
12 | traj: dict,
13 | window_size: int,
14 | future_action_window_size: int = 0,
15 | ) -> dict:
16 | """Chunks actions and observations into the given window_size.
17 |
18 | "observation" keys are given a new axis (at index 1) of size `window_size` containing `window_size - 1`
19 | observations from the past and the current observation. "action" is given a new axis (at index 1) of size
20 | `window_size + future_action_window_size` containing `window_size - 1` actions from the past, the current
21 | action, and `future_action_window_size` actions from the future. "pad_mask" is added to "observation" and
22 | indicates whether an observation should be considered padding (i.e. if it would have come from a timestep
23 | before the start of the trajectory).
24 | """
25 | traj_len = tf.shape(traj["action"])[0]
26 | action_dim = traj["action"].shape[-1]
27 | chunk_indices = tf.broadcast_to(
28 | tf.range(-window_size + 1, 1), [traj_len, window_size]
29 | ) + tf.broadcast_to(tf.range(traj_len)[:, None], [traj_len, window_size])
30 |
31 | action_chunk_indices = tf.broadcast_to(
32 | tf.range(-window_size + 1, 1 + future_action_window_size),
33 | [traj_len, window_size + future_action_window_size],
34 | ) + tf.broadcast_to(
35 | tf.range(traj_len)[:, None],
36 | [traj_len, window_size + future_action_window_size],
37 | )
38 |
39 | floored_chunk_indices = tf.maximum(chunk_indices, 0)
40 |
41 | if "timestep" in traj["task"]:
42 | goal_timestep = traj["task"]["timestep"]
43 | else:
44 | goal_timestep = tf.fill([traj_len], traj_len - 1)
45 |
46 | floored_action_chunk_indices = tf.minimum(
47 | tf.maximum(action_chunk_indices, 0), goal_timestep[:, None]
48 | )
49 |
50 | traj["observation"] = tf.nest.map_structure(
51 | lambda x: tf.gather(x, floored_chunk_indices), traj["observation"]
52 | )
53 | traj["action"] = tf.gather(traj["action"], floored_action_chunk_indices)
54 |
55 | # indicates whether an entire observation is padding
56 | traj["observation"]["pad_mask"] = chunk_indices >= 0
57 |
58 | # if no absolute_action_mask was provided, assume all actions are relative
59 | if "absolute_action_mask" not in traj and future_action_window_size > 0:
60 | logging.warning(
61 | "future_action_window_size > 0 but no absolute_action_mask was provided. "
62 | "Assuming all actions are relative for the purpose of making neutral actions."
63 | )
64 | absolute_action_mask = traj.get(
65 | "absolute_action_mask", tf.zeros([traj_len, action_dim], dtype=tf.bool)
66 | )
67 | neutral_actions = tf.where(
68 | absolute_action_mask[:, None, :],
69 | traj["action"], # absolute actions are repeated (already done during chunking)
70 | tf.zeros_like(traj["action"]), # relative actions are zeroed
71 | )
72 |
73 | # actions past the goal timestep become neutral
74 | action_past_goal = action_chunk_indices > goal_timestep[:, None]
75 | traj["action"] = tf.where(
76 | action_past_goal[:, :, None], neutral_actions, traj["action"]
77 | )
78 | return traj
79 |
80 |
81 | def subsample(traj: dict, subsample_length: int) -> dict:
82 | """Subsamples trajectories to the given length."""
83 | traj_len = tf.shape(traj["action"])[0]
84 | if traj_len > subsample_length:
85 | indices = tf.random.shuffle(tf.range(traj_len))[:subsample_length]
86 | traj = tf.nest.map_structure(lambda x: tf.gather(x, indices), traj)
87 | return traj
88 |
89 |
90 | def add_pad_mask_dict(traj: dict) -> dict:
91 | """Adds a dictionary indicating which elements of the observation/task should be treated as padding.
92 |
93 | traj["observation"|"task"]["pad_mask_dict"] = {k: traj["observation"|"task"][k] is not padding}
94 | """
95 | traj_len = tf.shape(traj["action"])[0]
96 | for key in ["observation", "task"]:
97 | pad_mask_dict = {}
98 | for subkey in traj[key]:
99 | if traj[key][subkey].dtype == tf.string:
100 | # handles "language_instruction", "image_*", and "depth_*"
101 | pad_mask_dict[subkey] = tf.strings.length(traj[key][subkey]) != 0
102 | else:
103 | # all other keys should not be treated as padding
104 | pad_mask_dict[subkey] = tf.ones([traj_len], dtype=tf.bool)
105 | traj[key]["pad_mask_dict"] = pad_mask_dict
106 | return traj
107 |
--------------------------------------------------------------------------------
/docs/REAL-WORLD_POSTPROCESS.md:
--------------------------------------------------------------------------------
1 | # Post-process
2 | Self-collected data and pre-training datasets often exhibit certain challenges that can negatively affect model performance. Below, we outline these issues and provide standardized solutions for data formatting and post-processing.
3 | * :warning: **Varied Action Labels:**
4 | Different embodiments, and sometimes even identical ones, may use diverse action labels such as:
5 | * **Absolute target joint positions** (qpos)
6 | * **Absolute target joint velocites** (qvel)
7 | * **Absolute end-effector poses** (ee-pose)
8 | * **Delta target end-effector poses** (delta ee-pose)
9 | Furthermore, rotation representations may vary, including quaternions, Euler angles, rotation vectors, and rotation matrices.
10 | * :warning: **Jittering and Long Pauses:** Fresh data collectors often introduce hesitation, leading to long pauses or jittering during data collection. Without proper filtering, such data significantly degrades model performance.
11 | * :warning: **Quick Gripper Open/Close Actions:** A frequency mismatch between camera capture and gripper control often results in abrupt changes in gripper states, especially during grasping or releasing motions.
12 |
13 | To address these issues, we recommend a uniform, clear, and effective format for saving self-collected data and provide tools for post-processing.
14 |
15 | ## :exclamation: Data Format
16 | For each task, we collect 100 demos. The recommended directory structure is:
17 | ```
18 | 0000 (exp_id)
19 | |—— 000000 (episode_id)
20 | |—— steps
21 | |—— 0000 (timestep_id, start)
22 | |—— image_primary.jpg (Eye-on-Base camera rgb image)
23 | |—— image_wrist.jpg (Eye-on-Hand camera rgb image)
24 | └── other.npz (robot state, language, action)
25 | |—— ......
26 | └── xxxx (timestep_id, end)
27 | |—— 000001 (episode_id)
28 | |—— steps
29 | |—— ......
30 | |—— ......
31 | └── 000099 (episode_id)
32 | |—— steps
33 | |—— ......
34 | ```
35 | ### File Details:
36 | * **image_primary.jpg** and **image_wrist.jpg**: Images saved with a resolution of 640 x 480 pixels.
37 | * **other.npz**: Contains key robot metadata. An example of the saved format is:
38 | ```python
39 | # at each timestep i
40 | npz_path = f"other.npz"
41 |
42 | # absolute current gripper pose in robot space, position + euler angles, the unit is m and rad.
43 | gripper_pose = np.array([x, y, z, euler_x, euler_y, euler_z])
44 |
45 | # absolute current gripper open state
46 | gripper_open_state = np.array([1.0]) if gripper is opened else np.array([-1.0])
47 |
48 | # absolute current joints position (qpos)
49 | joints = np.array([q0, q1, q2, q3, q4, q5, q6])
50 |
51 | # language instruction
52 | language_instruction = f"Pick the apple."
53 |
54 | # absolute target pose action label (target_gripper_open_or_close is 1.0 if targetting open, else -1.0)
55 | action_gripper_pose = np.array([target_x, target_y, target_z, target_euler_x, target_euler_y, target_euler_z, target_gripper_open_or_close])
56 |
57 | # delta pose action label
58 | delta_cur_2_last_action = np.array([target_delta_x, target_delta_y, target_delta_z, target_delta_euler_x, target_delta_euler_y, target_delta_euler_z, target_gripper_open_or_close])
59 |
60 | # save npz
61 | np.savez_compressed(
62 | npz_path,
63 | joints=joints,
64 | gripper_pose=gripper_pose,
65 | gripper_open_state=gripper_open_state,
66 | action_gripper_pose=action_gripper_pose,
67 | delta_cur_2_last_action=delta_cur_2_last_action,
68 | language_instruction=language_instruction,
69 | )
70 | ```
71 | For most robotic systems, all metadata except delta_cur_2_last_action can be directly extracted. We provide a helper function to compute the delta pose action label in the [script](../utils/real_ft_data.py):
72 | ```python
73 | compute_delta_action(
74 | data_list,
75 | )
76 | ```
77 |
78 | ## :star: Post-processing Self-collected Data
79 | * **Filtering Jitter and Pauses:** To filter out jittering and long pauses, use the following function in the [script](../utils/real_ft_data.py):
80 | ```python
81 | filter_real_data(
82 | exp_id,
83 | root_path, # path to your raw data
84 | save_data_path, # a desired path to save filterd data
85 | save_gif_path # a desired path to save the filtered gif (only for visualization and debugging)
86 | )
87 | ```
88 | * **Data Augmentation for Gripper Actions:** To augment data by increasing sampling ratios during gripper open/close events, use the following function in the same [script](../utils/real_ft_data.py):
89 | ```python
90 | make_aug_short_real_dataset_info(
91 | root_path, # path to your filterd data
92 | root_info_path, # path to your data info, it should be like xxx/Seer/data_info
93 | dataset_name, # your dataset name, e.g. "ft"
94 | select_ratio=1.0,
95 | sequence_length=7,
96 | action_pred_steps=3,
97 | replicate_steps=10
98 | )
99 | ```
100 |
101 |
--------------------------------------------------------------------------------
/data_info/.hydra/hydra.yaml:
--------------------------------------------------------------------------------
1 | hydra:
2 | run:
3 | dir: /work/dlclarge2/meeso-lfp/calvin_data/task_A_A/
4 | sweep:
5 | dir: multirun/${now:%Y-%m-%d}/${now:%H-%M-%S}
6 | subdir: ${hydra.job.num}
7 | launcher:
8 | _target_: hydra._internal.core_plugins.basic_launcher.BasicLauncher
9 | sweeper:
10 | _target_: hydra._internal.core_plugins.basic_sweeper.BasicSweeper
11 | max_batch_size: null
12 | help:
13 | app_name: ${hydra.job.name}
14 | header: '${hydra.help.app_name} is powered by Hydra.
15 |
16 | '
17 | footer: 'Powered by Hydra (https://hydra.cc)
18 |
19 | Use --hydra-help to view Hydra specific help
20 |
21 | '
22 | template: '${hydra.help.header}
23 |
24 | == Configuration groups ==
25 |
26 | Compose your configuration from those groups (group=option)
27 |
28 |
29 | $APP_CONFIG_GROUPS
30 |
31 |
32 | == Config ==
33 |
34 | Override anything in the config (foo.bar=value)
35 |
36 |
37 | $CONFIG
38 |
39 |
40 | ${hydra.help.footer}
41 |
42 | '
43 | hydra_help:
44 | template: 'Hydra (${hydra.runtime.version})
45 |
46 | See https://hydra.cc for more info.
47 |
48 |
49 | == Flags ==
50 |
51 | $FLAGS_HELP
52 |
53 |
54 | == Configuration groups ==
55 |
56 | Compose your configuration from those groups (For example, append hydra/job_logging=disabled
57 | to command line)
58 |
59 |
60 | $HYDRA_CONFIG_GROUPS
61 |
62 |
63 | Use ''--cfg hydra'' to Show the Hydra config.
64 |
65 | '
66 | hydra_help: ???
67 | hydra_logging:
68 | version: 1
69 | formatters:
70 | colorlog:
71 | (): colorlog.ColoredFormatter
72 | format: '[%(cyan)s%(asctime)s%(reset)s][%(purple)sHYDRA%(reset)s] %(message)s'
73 | handlers:
74 | console:
75 | class: logging.StreamHandler
76 | formatter: colorlog
77 | stream: ext://sys.stdout
78 | root:
79 | level: INFO
80 | handlers:
81 | - console
82 | disable_existing_loggers: false
83 | job_logging:
84 | version: 1
85 | formatters:
86 | simple:
87 | format: '[%(asctime)s][%(name)s][%(levelname)s] - %(message)s'
88 | colorlog:
89 | (): colorlog.ColoredFormatter
90 | format: '[%(cyan)s%(asctime)s%(reset)s][%(blue)s%(name)s%(reset)s][%(log_color)s%(levelname)s%(reset)s]
91 | - %(message)s'
92 | log_colors:
93 | DEBUG: purple
94 | INFO: green
95 | WARNING: yellow
96 | ERROR: red
97 | CRITICAL: red
98 | handlers:
99 | console:
100 | class: logging.StreamHandler
101 | formatter: colorlog
102 | stream: ext://sys.stdout
103 | file:
104 | class: logging.FileHandler
105 | formatter: simple
106 | filename: ${hydra.job.name}.log
107 | root:
108 | level: INFO
109 | handlers:
110 | - console
111 | - file
112 | disable_existing_loggers: false
113 | env: {}
114 | searchpath: []
115 | callbacks: {}
116 | output_subdir: .hydra
117 | overrides:
118 | hydra:
119 | - hydra.run.dir=/work/dlclarge2/meeso-lfp/calvin_data/task_A_A/
120 | task:
121 | - load_dir=/work/dlclarge2/meeso-lfp/calvin_recordings/play_env_A/2021-10-06/16-23-57/
122 | - set_static_cam=false
123 | - processes=16
124 | - +scene=calvin_scene_A
125 | - show_gui=false
126 | job:
127 | name: datarenderer
128 | override_dirname: +scene=calvin_scene_A,load_dir=/work/dlclarge2/meeso-lfp/calvin_recordings/play_env_A/2021-10-06/16-23-57/,processes=16,set_static_cam=false,show_gui=false
129 | id: ???
130 | num: ???
131 | config_name: config_rendering
132 | env_set: {}
133 | env_copy: []
134 | config:
135 | override_dirname:
136 | kv_sep: '='
137 | item_sep: ','
138 | exclude_keys: []
139 | runtime:
140 | version: 1.1.1
141 | cwd: /home/meeso/VRENVCalvinRender/vr_env
142 | config_sources:
143 | - path: hydra.conf
144 | schema: pkg
145 | provider: hydra
146 | - path: /home/meeso/VRENVCalvinRender/conf
147 | schema: file
148 | provider: main
149 | - path: hydra_plugins.hydra_colorlog.conf
150 | schema: pkg
151 | provider: hydra-colorlog
152 | - path: ''
153 | schema: structured
154 | provider: schema
155 | choices:
156 | scene: calvin_scene_A
157 | cameras: static_gripper_tactile
158 | cameras/cameras@cameras.tactile: tactile
159 | cameras/cameras@cameras.gripper: gripper
160 | cameras/cameras@cameras.static: static
161 | hydra/env: default
162 | hydra/callbacks: null
163 | hydra/job_logging: colorlog
164 | hydra/hydra_logging: colorlog
165 | hydra/hydra_help: default
166 | hydra/help: default
167 | hydra/sweeper: basic
168 | hydra/launcher: basic
169 | hydra/output: default
170 | verbose: false
171 |
--------------------------------------------------------------------------------
/real_preprocess/octo_oxe_data_utils/oxe/__init__.py:
--------------------------------------------------------------------------------
1 | import copy
2 | import logging
3 | from typing import Any, Dict, List, Sequence, Tuple, Union
4 |
5 | from octo_oxe_data_utils.oxe.oxe_dataset_configs import ActionEncoding, OXE_DATASET_CONFIGS
6 | from octo_oxe_data_utils.oxe.oxe_dataset_mixes import OXE_NAMED_MIXES
7 | from octo_oxe_data_utils.oxe.oxe_standardization_transforms import OXE_STANDARDIZATION_TRANSFORMS
8 | from octo_oxe_data_utils.utils.data_utils import NormalizationType
9 |
10 |
11 | def make_oxe_dataset_kwargs(
12 | name: str,
13 | data_dir: str,
14 | load_camera_views: Sequence[str] = ("primary",),
15 | load_depth: bool = False,
16 | load_proprio: bool = True,
17 | load_language: bool = True,
18 | action_proprio_normalization_type: NormalizationType = NormalizationType.NORMAL,
19 | ) -> Dict[str, Any]:
20 | """Generates dataset kwargs for a given dataset from Open X-Embodiment. The returned kwargs can be passed
21 | directly into `octo.data.dataset.make_dataset_from_rlds`.
22 |
23 | Args:
24 | name: Name of the dataset to load. See `oxe_dataset_configs.py` for available datasets.
25 | data_dir: Base data directory that contains the dataset.
26 | load_camera_views: Which views to load. See `oxe_dataset_configs.py` for available views.
27 | load_depth: If True, loads corresponding depth channels for each RGB channel.
28 | load_proprio: If True, loads proprioceptive information.
29 | load_language: If True, loads language instructions.
30 | action_proprio_normalization_type: Normalization type to use for proprioceptive actions.
31 | """
32 | dataset_kwargs = copy.deepcopy(OXE_DATASET_CONFIGS[name])
33 | if dataset_kwargs["action_encoding"] is not ActionEncoding.EEF_POS:
34 | raise ValueError(
35 | f"Cannot load {name} since only EEF pose delta action encoding is supported."
36 | )
37 |
38 | # with EEF_POS actions, only the last action dimension (the gripper) is absolute
39 | dataset_kwargs["absolute_action_mask"] = [False] * 6 + [True]
40 |
41 | # we also want to skip normalizing the gripper action
42 | dataset_kwargs["action_normalization_mask"] = [True] * 6 + [False]
43 |
44 | # adjust loaded camera views
45 | if missing_keys := (set(load_camera_views) - set(dataset_kwargs["image_obs_keys"])):
46 | raise ValueError(
47 | f"Cannot load {name} with views {missing_keys} since they are not available."
48 | )
49 | dataset_kwargs["image_obs_keys"] = {
50 | k: v
51 | for k, v in dataset_kwargs["image_obs_keys"].items()
52 | if k in load_camera_views
53 | }
54 | dataset_kwargs["depth_obs_keys"] = {
55 | k: v
56 | for k, v in dataset_kwargs["depth_obs_keys"].items()
57 | if k in load_camera_views
58 | }
59 |
60 | if not load_depth:
61 | dataset_kwargs.pop("depth_obs_keys")
62 | if not load_proprio:
63 | dataset_kwargs.pop("state_obs_keys")
64 |
65 | if load_language:
66 | dataset_kwargs["language_key"] = "language_instruction"
67 |
68 | dataset_kwargs[
69 | "action_proprio_normalization_type"
70 | ] = action_proprio_normalization_type
71 |
72 | del dataset_kwargs["state_encoding"]
73 | del dataset_kwargs["action_encoding"]
74 |
75 | dataset_kwargs["standardize_fn"] = OXE_STANDARDIZATION_TRANSFORMS[name]
76 |
77 | return {"name": name, "data_dir": data_dir, **dataset_kwargs}
78 |
79 |
80 | def make_oxe_dataset_kwargs_and_weights(
81 | data_mix: Union[str, Sequence[Tuple[str, float]]],
82 | data_dir: str,
83 | load_camera_views: Sequence[str] = ("primary",),
84 | load_depth: bool = False,
85 | load_proprio: bool = True,
86 | load_language: bool = True,
87 | action_proprio_normalization_type: NormalizationType = NormalizationType.NORMAL,
88 | ) -> Tuple[Dict[str, Any], List[float]]:
89 | """
90 | Generates dataset kwargs for a given dataset mix from the Open X-Embodiment dataset. The returned kwargs
91 | and weights can be passed directly into `octo.data.dataset.make_interleaved_dataset`.
92 |
93 | Args:
94 | data_mix: List of (dataset name, sampling weight) tuples, or a string specifying a pre-defined mix to
95 | load from `OXE_NAMED_MIXES`.
96 | data_dir: Base data directory that contains the datasets.
97 | load_camera_views: Which views to load. See `oxe_dataset_configs.py` for available views.
98 | load_depth: If True, loads corresponding depth channels for each RGB channel.
99 | load_proprio: If True, loads proprioceptive information.
100 | load_language: If True, loads language instructions.
101 | action_proprio_normalization_type: Normalization type to use for proprioceptive actions.
102 | Returns:
103 | Tuple of (dataset_kwargs_list, sampling weights).
104 | """
105 | if isinstance(data_mix, str):
106 | data_mix = OXE_NAMED_MIXES[data_mix]
107 |
108 | filtered_datasets, included_dataset_names = [], []
109 | for name, weight in data_mix:
110 | if name not in included_dataset_names:
111 | filtered_datasets.append((name, weight))
112 | included_dataset_names.append(name)
113 | else:
114 | logging.warning(f"Skipping duplicate: {(name, weight)}.")
115 | data_mix = filtered_datasets
116 |
117 | data_kwargs_list, weights = [], []
118 | for name, weight in data_mix:
119 | try:
120 | data_kwargs_list.append(
121 | make_oxe_dataset_kwargs(
122 | name,
123 | data_dir,
124 | load_camera_views,
125 | load_depth,
126 | load_proprio,
127 | load_language,
128 | action_proprio_normalization_type,
129 | )
130 | )
131 | weights.append(weight)
132 | except ValueError as e:
133 | logging.warning(f"Skipping {name} due to error: {e}")
134 |
135 | return data_kwargs_list, weights
136 |
--------------------------------------------------------------------------------
/utils/distributed_utils.py:
--------------------------------------------------------------------------------
1 | """
2 | Util functions for setting up distributed training.
3 | Credit: https://github.com/mlfoundations/open_clip/blob/main/src/training/distributed.py
4 | """
5 | import os
6 | import torch
7 | from datetime import timedelta
8 |
9 | def is_global_master(args):
10 | return args.rank == 0
11 |
12 | def is_local_master(args):
13 | return args.local_rank == 0
14 |
15 | def is_master(args, local=False):
16 | return is_local_master(args) if local else is_global_master(args)
17 |
18 | def is_using_distributed():
19 | if "WORLD_SIZE" in os.environ:
20 | return int(os.environ["WORLD_SIZE"]) > 1
21 | if "SLURM_NTASKS" in os.environ:
22 | return int(os.environ["SLURM_NTASKS"]) > 1
23 | return False
24 |
25 | def world_info_from_env():
26 | local_rank = 0
27 | for v in (
28 | "LOCAL_RANK",
29 | "MPI_LOCALRANKID",
30 | "SLURM_LOCALID",
31 | "OMPI_COMM_WORLD_LOCAL_RANK",
32 | ):
33 | if v in os.environ:
34 | local_rank = int(os.environ[v])
35 | break
36 | global_rank = 0
37 | for v in ("RANK", "PMI_RANK", "SLURM_PROCID", "OMPI_COMM_WORLD_RANK"):
38 | if v in os.environ:
39 | global_rank = int(os.environ[v])
40 | break
41 | world_size = 1
42 | for v in ("WORLD_SIZE", "PMI_SIZE", "SLURM_NTASKS", "OMPI_COMM_WORLD_SIZE"):
43 | if v in os.environ:
44 | world_size = int(os.environ[v])
45 | break
46 |
47 | return local_rank, global_rank, world_size
48 |
49 | # def init_distributed_device(args):
50 | # # Distributed training = training on more than one GPU.
51 | # # Works in both single and multi-node scenarios.
52 | # args.distributed = False
53 | # args.world_size = 1
54 | # args.rank = 0 # global rank
55 | # args.local_rank = 0
56 |
57 | # if is_using_distributed():
58 | # if "SLURM_PROCID" in os.environ:
59 | # # DDP via SLURM
60 | # args.local_rank, args.rank, args.world_size = world_info_from_env()
61 | # # SLURM var -> torch.distributed vars in case needed
62 | # os.environ["LOCAL_RANK"] = str(args.local_rank)
63 | # os.environ["RANK"] = str(args.rank)
64 | # os.environ["WORLD_SIZE"] = str(args.world_size)
65 | # torch.distributed.init_process_group(
66 | # backend=args.dist_backend,
67 | # init_method=args.dist_url,
68 | # world_size=args.world_size,
69 | # rank=args.rank,
70 | # timeout=timedelta(seconds=36000000)
71 | # )
72 | # else:
73 | # # DDP via torchrun, torch.distributed.launch
74 | # args.local_rank, _, _ = world_info_from_env()
75 | # torch.distributed.init_process_group(
76 | # backend=args.dist_backend, init_method=args.dist_url, timeout=timedelta(seconds=36000000)
77 | # )
78 | # args.world_size = torch.distributed.get_world_size()
79 | # args.rank = torch.distributed.get_rank()
80 | # args.distributed = True
81 | # else:
82 | # # needed to run on single gpu
83 | # torch.distributed.init_process_group(
84 | # backend=args.dist_backend,
85 | # init_method=args.dist_url,
86 | # world_size=1,
87 | # rank=0,
88 | # timeout=timedelta(seconds=36000000)
89 | # )
90 |
91 | # if torch.cuda.is_available():
92 | # if args.distributed and not args.no_set_device_rank:
93 | # device = "cuda:%d" % args.local_rank
94 | # else:
95 | # device = "cuda:0"
96 | # torch.cuda.set_device(device)
97 | # else:
98 | # device = "cpu"
99 | # args.device = device
100 | # device = torch.device(device)
101 | # return device
102 |
103 | def init_distributed_device(args):
104 | # Distributed training = training on more than one GPU.
105 | # Works in both single and multi-node scenarios.
106 | args.distributed = False
107 | args.world_size = 1
108 | args.rank = 0 # global rank
109 | args.local_rank = 0
110 | from pdb import set_trace
111 | import datetime
112 | if is_using_distributed():
113 | if "SLURM_PROCID" in os.environ:
114 | # DDP via SLURM
115 | args.local_rank, args.rank, args.world_size = world_info_from_env()
116 | # SLURM var -> torch.distributed vars in case needed
117 | os.environ["LOCAL_RANK"] = str(args.local_rank)
118 | os.environ["RANK"] = str(args.rank)
119 | os.environ["WORLD_SIZE"] = str(args.world_size)
120 | os.environ['NCCL_BLOCKING_WAIT'] = '0' # not to enforce timeout
121 | torch.distributed.init_process_group(
122 | # timeout=timedelta(seconds=7200000), # was 1800000
123 | timeout=datetime.timedelta(seconds=7200),
124 | backend=args.dist_backend,
125 | init_method=args.dist_url,
126 | world_size=args.world_size,
127 | rank=args.rank,
128 | )
129 | else:
130 | # DDP via torchrun, torch.distributed.launch
131 | os.environ['NCCL_BLOCKING_WAIT'] = '0' # not to enforce timeout
132 | args.local_rank, _, _ = world_info_from_env()
133 | torch.distributed.init_process_group(
134 | # timeout=timedelta(seconds=7200000), # was 18000
135 | timeout=datetime.timedelta(seconds=17200),
136 | backend=args.dist_backend,
137 | init_method=args.dist_url
138 | )
139 | args.world_size = torch.distributed.get_world_size()
140 | args.rank = torch.distributed.get_rank()
141 | args.distributed = True
142 | else:
143 | # needed to run on single gpu
144 | torch.distributed.init_process_group(
145 | backend=args.dist_backend,
146 | init_method=args.dist_url,
147 | world_size=1,
148 | rank=0,
149 | )
150 |
151 | if torch.cuda.is_available():
152 | if args.distributed and not args.no_set_device_rank:
153 | device = "cuda:%d" % args.local_rank
154 | else:
155 | device = "cuda:0"
156 | torch.cuda.set_device(device)
157 | else:
158 | device = "cpu"
159 | args.device = device
160 | device = torch.device(device)
161 | return device
162 |
--------------------------------------------------------------------------------
/slurm_train_intern.py:
--------------------------------------------------------------------------------
1 | # Copyright (c) Facebook, Inc. and its affiliates.
2 | #
3 | # Licensed under the Apache License, Version 2.0 (the "License");
4 | # you may not use this file except in compliance with the License.
5 | # You may obtain a copy of the License at
6 | #
7 | # http://www.apache.org/licenses/LICENSE-2.0
8 | #
9 | # Unless required by applicable law or agreed to in writing, software
10 | # distributed under the License is distributed on an "AS IS" BASIS,
11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | # See the License for the specific language governing permissions and
13 | # limitations under the License.
14 | """
15 | A script to run multinode training with submitit.
16 | Almost copy-paste from https://github.com/facebookresearch/deit/blob/main/run_with_submitit.py
17 | """
18 | import argparse
19 | import os
20 | import re
21 | import random
22 | import uuid
23 | from pathlib import Path
24 | import train
25 | from utils.arguments_utils import get_parser
26 | import submitit
27 |
28 |
29 | def parse_args():
30 | parser = argparse.ArgumentParser("Submitit for DINO", parents=[get_parser()])
31 | print("!!!")
32 | parser.add_argument("--ngpus", default=8, type=int, help="Number of gpus to request on each node")
33 | parser.add_argument("--nodes", default=4, type=int, help="Number of nodes to request")
34 | parser.add_argument("--timeout", default=72000, type=int, help="Duration of the job")
35 | parser.add_argument("--partition", default="mozi-S1", type=str, help="Partition where to submit")
36 | parser.add_argument("--use_volta32", action='store_true', help="Big models? Use this")
37 | parser.add_argument('--comment', default="", type=str,
38 | help='Comment to pass to scheduler, e.g. priority message')
39 | parser.add_argument("--exclude", default="", type=str, help="Nodes to exclude")
40 | parser.add_argument("--output_dir", default="/mnt/petrelfs/tianyang/Code/ICLR_Manipulation/out", type=str)
41 | return parser.parse_args()
42 |
43 | def get_shared_folder() -> Path:
44 | user = os.getenv("USER")
45 | if Path(f"/ailab/user/{user}/").is_dir():
46 | p = Path(f"/ailab/user/{user}/experiments")
47 | p.mkdir(exist_ok=True)
48 | return p
49 | raise RuntimeError("No shared folder available")
50 |
51 | def get_init_file():
52 | # Init file must not exist, but it's parent dir must exist.
53 | os.makedirs(str(get_shared_folder()), exist_ok=True)
54 | init_file = get_shared_folder() / f"{uuid.uuid4().hex}_init"
55 | if init_file.exists():
56 | os.remove(str(init_file))
57 | return init_file
58 |
59 | def _get_master_port(seed):
60 | MIN_MASTER_PORT, MAX_MASTER_PORT = (20_000, 60_000)
61 | master_port_str = os.environ.get("MASTER_PORT")
62 | if master_port_str is None:
63 | rng = random.Random(seed)
64 | return rng.randint(MIN_MASTER_PORT, MAX_MASTER_PORT)
65 | return int(master_port_str)
66 |
67 | def _parse_slurm_node_list(s):
68 | nodes = []
69 | # Extract "hostname", "hostname[1-2,3,4-5]," substrings
70 | p = re.compile(r"(([^\[]+)(?:\[([^\]]+)\])?),?")
71 | for m in p.finditer(s):
72 | prefix, suffixes = s[m.start(2) : m.end(2)], s[m.start(3) : m.end(3)]
73 | prefix_list = prefix.split(',')
74 | if len(prefix_list) > 1:
75 | nodes += prefix_list[:-1]
76 | prefix = prefix_list[-1]
77 | for suffix in suffixes.split(","):
78 | span = suffix.split("-")
79 | if len(span) == 1:
80 | nodes.append(prefix + suffix)
81 | else:
82 | width = len(span[0])
83 | start, end = int(span[0]), int(span[1]) + 1
84 | for i in range(start, end):
85 | nodes.append(prefix + f"{i:0{width}}")
86 |
87 | return nodes
88 |
89 | class Trainer(object):
90 | def __init__(self, args):
91 | self.args = args
92 | def __call__(self):
93 | # import run_beit_pretraining
94 | self._setup_gpu_args()
95 | train.main(self.args)
96 |
97 | def checkpoint(self):
98 | import os
99 | import submitit
100 | # self.args.dist_url = get_init_file().as_uri()
101 | print("Requeuing ", self.args)
102 | empty_trainer = type(self)(self.args)
103 | return submitit.helpers.DelayedSubmission(empty_trainer)
104 |
105 | def _setup_gpu_args(self):
106 | import submitit
107 | from pathlib import Path
108 | job_id = int(os.environ["SLURM_JOB_ID"])
109 | node_count = int(os.environ["SLURM_JOB_NUM_NODES"])
110 | print("node_list :", os.environ["SLURM_JOB_NODELIST"])
111 | nodes = _parse_slurm_node_list(os.environ["SLURM_JOB_NODELIST"])
112 | print("node_count :", node_count)
113 | print("nodes :", nodes)
114 | assert len(nodes) == node_count
115 | master_addr = nodes[0]
116 | master_port = _get_master_port(seed=job_id)
117 | os.environ['MASTER_ADDR'] = master_addr
118 | os.environ['MASTER_PORT'] = str(master_port)
119 | job_env = submitit.JobEnvironment()
120 | self.args.output_dir = Path(str(self.args.output_dir).replace("%j", str(job_env.job_id)))
121 | self.args.gpu = job_env.local_rank
122 | self.args.rank = job_env.global_rank
123 | self.args.world_size = job_env.num_tasks
124 | print(f"Process group: {job_env.num_tasks} tasks, rank: {job_env.global_rank}")
125 |
126 | def main():
127 | args = parse_args()
128 | if args.output_dir == "":
129 | args.output_dir = get_shared_folder() / "%j"
130 | Path(args.output_dir).mkdir(parents=True, exist_ok=True)
131 | executor = submitit.SlurmExecutor(folder=args.output_dir, max_num_timeout=30)
132 | num_gpus_per_node = args.ngpus
133 | nodes = args.nodes
134 | timeout_min = args.timeout
135 | partition = args.partition
136 | kwargs = {}
137 | if args.use_volta32:
138 | kwargs['slurm_constraint'] = 'volta32gb'
139 | if args.comment:
140 | kwargs['slurm_comment'] = args.comment
141 | if args.exclude:
142 | kwargs["exclude"] = args.exclude
143 | executor.update_parameters(
144 | gres=f"gpu:{num_gpus_per_node}",
145 | ntasks_per_node=num_gpus_per_node, # one task per GPU
146 | cpus_per_task=6,
147 | nodes=nodes,
148 | time=timeout_min,
149 | # Below are cluster dependent parameters
150 | signal_delay_s=120,
151 | partition=partition,
152 | **kwargs
153 | )
154 | executor.update_parameters(job_name="seer")
155 | # args.dist_url = get_init_file().as_uri()
156 | trainer = Trainer(args)
157 | job = executor.submit(trainer)
158 | print(f"Submitted job_id: {job.job_id}")
159 | print(f"Logs and checkpoints will be saved at: {args.output_dir}")
160 |
161 | if __name__ == "__main__":
162 | main()
--------------------------------------------------------------------------------
/real_preprocess/octo_oxe_data_utils/oxe/oxe_dataset_mixes.py:
--------------------------------------------------------------------------------
1 | """Defines dataset mixtures and weights for the Open X-Embodiment Datasets."""
2 |
3 |
4 | BRIDGE_MIX = [
5 | ("bridge_dataset", 1.0),
6 | ]
7 |
8 | RT_X_MIX = [
9 | ("fractal20220817_data", 0.54087122203),
10 | ("kuka", 0.8341046294),
11 | ("bridge_dataset", 1.0),
12 | ("taco_play", 2.0),
13 | ("jaco_play", 2.0),
14 | ("berkeley_cable_routing", 3.0),
15 | ("roboturk", 1.0),
16 | ("nyu_door_opening_surprising_effectiveness", 5.0),
17 | ("viola", 2.0),
18 | ("berkeley_autolab_ur5", 1.0),
19 | ("toto", 1.0),
20 | ]
21 |
22 |
23 | OXE_FRANKA_MIX = [
24 | ("taco_play", 1.0),
25 | ("berkeley_cable_routing", 1.0),
26 | ("viola", 1.0),
27 | ("toto", 1.0),
28 | ("stanford_hydra_dataset_converted_externally_to_rlds", 1.0),
29 | ("austin_buds_dataset_converted_externally_to_rlds", 3.0),
30 | ("nyu_franka_play_dataset_converted_externally_to_rlds", 3.0),
31 | ("maniskill_dataset_converted_externally_to_rlds", 0.1),
32 | ("furniture_bench_dataset_converted_externally_to_rlds", 0.1),
33 | ("cmu_franka_exploration_dataset_converted_externally_to_rlds", 5.0),
34 | ("austin_sailor_dataset_converted_externally_to_rlds", 1.0),
35 | ("austin_sirius_dataset_converted_externally_to_rlds", 1.0),
36 | ("berkeley_rpt_converted_externally_to_rlds", 1.0),
37 | ("kaist_nonprehensile_converted_externally_to_rlds", 3.0),
38 | ("stanford_robocook_converted_externally_to_rlds", 1.0),
39 | ("iamlab_cmu_pickup_insert_converted_externally_to_rlds", 1.0),
40 | ("utaustin_mutex", 1.0),
41 | # ("cmu_playing_with_food", 1.0),
42 | ("cmu_play_fusion", 1.0),
43 | ]
44 |
45 |
46 | OXE_MAGIC_SOUP = [
47 | ("fractal20220817_data", 0.54087122203),
48 | ("kuka", 0.8341046294),
49 | ("bridge_dataset", 1.0),
50 | ("taco_play", 2.0),
51 | ("jaco_play", 1.0),
52 | ("berkeley_cable_routing", 1.0),
53 | ("roboturk", 2.0),
54 | ("nyu_door_opening_surprising_effectiveness", 1.0),
55 | ("viola", 2.0),
56 | ("berkeley_autolab_ur5", 2.0),
57 | ("toto", 1.0),
58 | ("language_table", 0.1),
59 | ("stanford_hydra_dataset_converted_externally_to_rlds", 2.0),
60 | ("austin_buds_dataset_converted_externally_to_rlds", 1.0),
61 | ("nyu_franka_play_dataset_converted_externally_to_rlds", 3.0),
62 | ("furniture_bench_dataset_converted_externally_to_rlds", 0.1),
63 | ("ucsd_kitchen_dataset_converted_externally_to_rlds", 2.0),
64 | ("austin_sailor_dataset_converted_externally_to_rlds", 1.0),
65 | ("austin_sirius_dataset_converted_externally_to_rlds", 1.0),
66 | ("bc_z", 0.2),
67 | ("dlr_edan_shared_control_converted_externally_to_rlds", 1.0),
68 | ("iamlab_cmu_pickup_insert_converted_externally_to_rlds", 1.0),
69 | ("utaustin_mutex", 1.0),
70 | ("berkeley_fanuc_manipulation", 2.0),
71 | ("cmu_stretch", 1.0),
72 | ]
73 |
74 | OXE_MAGIC_SOUP_EEFP_ONLY = [
75 | ("fractal20220817_data", 0.54087122203),
76 | ("kuka", 0.8341046294),
77 | ("bridge_dataset", 1.0),
78 | ("taco_play", 2.0),
79 | ("jaco_play", 1.0),
80 | # ("berkeley_cable_routing", 1.0),
81 | ("roboturk", 2.0),
82 | ("nyu_door_opening_surprising_effectiveness", 1.0),
83 | ("viola", 2.0),
84 | ("berkeley_autolab_ur5", 2.0),
85 | # ("toto", 1.0),
86 | ("language_table", 0.1),
87 | ("stanford_hydra_dataset_converted_externally_to_rlds", 2.0),
88 | ("austin_buds_dataset_converted_externally_to_rlds", 1.0),
89 | # ("nyu_franka_play_dataset_converted_externally_to_rlds", 3.0),
90 | # ("furniture_bench_dataset_converted_externally_to_rlds", 0.1),
91 | ("ucsd_kitchen_dataset_converted_externally_to_rlds", 2.0),
92 | # ("austin_sailor_dataset_converted_externally_to_rlds", 1.0),
93 | # ("austin_sirius_dataset_converted_externally_to_rlds", 1.0),
94 | ("bc_z", 0.2),
95 | ("dlr_edan_shared_control_converted_externally_to_rlds", 1.0),
96 | ("iamlab_cmu_pickup_insert_converted_externally_to_rlds", 1.0),
97 | ("utaustin_mutex", 1.0),
98 | ("berkeley_fanuc_manipulation", 2.0),
99 | ("cmu_stretch", 1.0),
100 | ]
101 |
102 | OXE_FULL_MIX = [
103 | ("fractal20220817_data", 1.0),
104 | ("kuka", 1.0),
105 | ("bridge_dataset", 1),
106 | ("taco_play", 1.0),
107 | ("jaco_play", 1.0),
108 | ("berkeley_cable_routing", 1.0),
109 | ("roboturk", 1.0),
110 | ("nyu_door_opening_surprising_effectiveness", 1.0),
111 | ("viola", 1.0),
112 | ("berkeley_autolab_ur5", 1.0),
113 | ("toto", 1.0),
114 | ("language_table", 1.0),
115 | ("columbia_cairlab_pusht_real", 1.0),
116 | ("stanford_kuka_multimodal_dataset_converted_externally_to_rlds", 1.0),
117 | ("nyu_rot_dataset_converted_externally_to_rlds", 1.0),
118 | ("stanford_hydra_dataset_converted_externally_to_rlds", 1.0),
119 | ("austin_buds_dataset_converted_externally_to_rlds", 1.0),
120 | ("nyu_franka_play_dataset_converted_externally_to_rlds", 1.0),
121 | ("maniskill_dataset_converted_externally_to_rlds", 1.0),
122 | ("furniture_bench_dataset_converted_externally_to_rlds", 1.0),
123 | ("cmu_franka_exploration_dataset_converted_externally_to_rlds", 1.0),
124 | ("ucsd_kitchen_dataset_converted_externally_to_rlds", 1.0),
125 | ("ucsd_pick_and_place_dataset_converted_externally_to_rlds", 1.0),
126 | ("austin_sailor_dataset_converted_externally_to_rlds", 1.0),
127 | ("austin_sirius_dataset_converted_externally_to_rlds", 1.0),
128 | ("bc_z", 1.0),
129 | ("utokyo_pr2_opening_fridge_converted_externally_to_rlds", 1.0),
130 | ("utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds", 1.0),
131 | ("utokyo_xarm_pick_and_place_converted_externally_to_rlds", 1.0),
132 | ("utokyo_xarm_bimanual_converted_externally_to_rlds", 1.0),
133 | ("robo_net", 1.0),
134 | ("berkeley_mvp_converted_externally_to_rlds", 1.0),
135 | ("berkeley_rpt_converted_externally_to_rlds", 1.0),
136 | ("kaist_nonprehensile_converted_externally_to_rlds", 1.0),
137 | ("stanford_mask_vit_converted_externally_to_rlds", 1.0),
138 | ("tokyo_u_lsmo_converted_externally_to_rlds", 1.0),
139 | ("dlr_sara_pour_converted_externally_to_rlds", 1.0),
140 | ("dlr_sara_grid_clamp_converted_externally_to_rlds", 1.0),
141 | ("dlr_edan_shared_control_converted_externally_to_rlds", 1.0),
142 | ("asu_table_top_converted_externally_to_rlds", 1.0),
143 | ("stanford_robocook_converted_externally_to_rlds", 1.0),
144 | ("imperialcollege_sawyer_wrist_cam", 1.0),
145 | ("iamlab_cmu_pickup_insert_converted_externally_to_rlds", 1.0),
146 | ("uiuc_d3field", 1.0),
147 | ("utaustin_mutex", 1.0),
148 | ("berkeley_fanuc_manipulation", 1.0),
149 | ("cmu_playing_with_food", 1.0),
150 | ("cmu_play_fusion", 1.0),
151 | ("cmu_stretch", 1.0),
152 | ("berkeley_gnm_recon", 1.0),
153 | ("berkeley_gnm_cory_hall", 1.0),
154 | ("berkeley_gnm_sac_son", 1.0),
155 | ]
156 |
157 | MYTEST_MIX = [
158 | ("cmu_stretch", 1.0),
159 | ]
160 | MYTEST_MIX1 = [
161 | ("berkeley_autolab_ur5", 1.0),
162 | ]
163 | MYTEST_MIX2 = [
164 | ("viola", 1.0),
165 | ]
166 | MYTEST_MIX3 = [
167 | ("nyu_door_opening_surprising_effectiveness", 1.0),
168 | ]
169 |
170 | OXE_NAMED_MIXES = {
171 | "bridge": BRIDGE_MIX,
172 | "rtx": RT_X_MIX,
173 | "rtx_franka": RT_X_MIX + OXE_FRANKA_MIX,
174 | "oxe_magic_soup": OXE_MAGIC_SOUP,
175 | "oxe_magic_soup_eefp_only": OXE_MAGIC_SOUP_EEFP_ONLY,
176 | "mytest": MYTEST_MIX,
177 | }
178 |
--------------------------------------------------------------------------------
/data_info/.hydra/merged_config.yaml:
--------------------------------------------------------------------------------
1 | cameras:
2 | static:
3 | _target_: calvin_env.camera.static_camera.StaticCamera
4 | name: static
5 | fov: 10
6 | aspect: 1
7 | nearval: 0.01
8 | farval: 10
9 | width: 200
10 | height: 200
11 | look_at:
12 | - -0.026242351159453392
13 | - -0.0302329882979393
14 | - 0.3920000493526459
15 | look_from:
16 | - 2.871459009488717
17 | - -2.166602199425597
18 | - 2.555159848480571
19 | up_vector:
20 | - 0.4041403970338857
21 | - 0.22629790978217404
22 | - 0.8862616969685161
23 | gripper:
24 | _target_: calvin_env.camera.gripper_camera.GripperCamera
25 | name: gripper
26 | fov: 75
27 | aspect: 1
28 | nearval: 0.01
29 | farval: 2
30 | width: 84
31 | height: 84
32 | tactile:
33 | _target_: calvin_env.camera.tactile_sensor.TactileSensor
34 | name: tactile
35 | width: 120
36 | height: 160
37 | digit_link_ids:
38 | - 10
39 | - 12
40 | visualize_gui: false
41 | config_path: conf/digit_sensor/config_digit.yml
42 | vr_input:
43 | vr_controller:
44 | POSITION: 1
45 | ORIENTATION: 2
46 | ANALOG: 3
47 | BUTTONS: 6
48 | BUTTON_A: 2
49 | BUTTON_B: 1
50 | vr_controller_id: 3
51 | gripper_orientation_offset:
52 | - 0
53 | - 3
54 | - 3.14
55 | gripper_position_offset:
56 | - -0.2
57 | - 0.3
58 | - 0
59 | _target_: calvin_env.io_utils.vr_input.VrInput
60 | limit_angle:
61 | - 90
62 | - 0
63 | - 0
64 | - -1
65 | visualize_vr_pos: true
66 | reset_button_queue_len: 60
67 | env:
68 | _target_: calvin_env.envs.play_table_env.PlayTableSimEnv
69 | _recursive_: false
70 | cameras: ${cameras}
71 | seed: 0
72 | bullet_time_step: 240.0
73 | use_vr: false
74 | show_gui: ${show_gui}
75 | robot_cfg: ${robot}
76 | scene_cfg: ${scene}
77 | use_scene_info: false
78 | use_egl: true
79 | control_freq: 30
80 | scene:
81 | _target_: calvin_env.scene.play_table_scene.PlayTableScene
82 | _recursive_: false
83 | data_path: ${data_path}
84 | global_scaling: 0.8
85 | euler_obs: ${robot.euler_obs}
86 | robot_base_position:
87 | - -0.34
88 | - -0.46
89 | - 0.24
90 | robot_base_orientation:
91 | - 0
92 | - 0
93 | - 0
94 | robot_initial_joint_positions:
95 | - -1.21779206
96 | - 1.03987646
97 | - 2.11978261
98 | - -2.34205014
99 | - -0.87015947
100 | - 1.64119353
101 | - 0.55344866
102 | surfaces:
103 | table:
104 | - - 0.0
105 | - -0.15
106 | - 0.46
107 | - - 0.35
108 | - -0.03
109 | - 0.46
110 | slider_left:
111 | - - -0.32
112 | - 0.05
113 | - 0.46
114 | - - -0.16
115 | - 0.12
116 | - 0.46
117 | slider_right:
118 | - - -0.05
119 | - 0.05
120 | - 0.46
121 | - - 0.13
122 | - 0.12
123 | - 0.46
124 | objects:
125 | fixed_objects:
126 | table:
127 | file: calvin_table_D/urdf/calvin_table_D.urdf
128 | initial_pos:
129 | - 0
130 | - 0
131 | - 0
132 | initial_orn:
133 | - 0
134 | - 0
135 | - 0
136 | joints:
137 | base__slide:
138 | initial_state: 0
139 | base__drawer:
140 | initial_state: 0
141 | buttons:
142 | base__button:
143 | initial_state: 0
144 | effect: led
145 | switches:
146 | base__switch:
147 | initial_state: 0
148 | effect: lightbulb
149 | lights:
150 | lightbulb:
151 | link: light_link
152 | color:
153 | - 1
154 | - 1
155 | - 0
156 | - 1
157 | led:
158 | link: led_link
159 | color:
160 | - 0
161 | - 1
162 | - 0
163 | - 1
164 | movable_objects:
165 | block_red:
166 | file: blocks/block_red_middle.urdf
167 | initial_pos: any
168 | initial_orn: any
169 | block_blue:
170 | file: blocks/block_blue_small.urdf
171 | initial_pos: any
172 | initial_orn: any
173 | block_pink:
174 | file: blocks/block_pink_big.urdf
175 | initial_pos: any
176 | initial_orn: any
177 | name: calvin_scene_D
178 | robot:
179 | _target_: calvin_env.robot.robot.Robot
180 | filename: franka_panda/panda_longer_finger.urdf
181 | base_position: ${scene.robot_base_position}
182 | base_orientation: ${scene.robot_base_orientation}
183 | initial_joint_positions: ${scene.robot_initial_joint_positions}
184 | max_joint_force: 200.0
185 | gripper_force: 200
186 | arm_joint_ids:
187 | - 0
188 | - 1
189 | - 2
190 | - 3
191 | - 4
192 | - 5
193 | - 6
194 | gripper_joint_ids:
195 | - 9
196 | - 11
197 | gripper_joint_limits:
198 | - 0
199 | - 0.04
200 | tcp_link_id: 15
201 | end_effector_link_id: 7
202 | gripper_cam_link: 12
203 | use_nullspace: true
204 | max_velocity: 2
205 | use_ik_fast: false
206 | magic_scaling_factor_pos: 1
207 | magic_scaling_factor_orn: 1
208 | use_target_pose: true
209 | euler_obs: true
210 | tasks:
211 | _target_: calvin_env.envs.tasks.Tasks
212 | tasks:
213 | rotate_red_block_right:
214 | - rotate_object
215 | - block_red
216 | - -60
217 | rotate_red_block_left:
218 | - rotate_object
219 | - block_red
220 | - 60
221 | rotate_blue_block_right:
222 | - rotate_object
223 | - block_blue
224 | - -60
225 | rotate_blue_block_left:
226 | - rotate_object
227 | - block_blue
228 | - 60
229 | rotate_pink_block_right:
230 | - rotate_object
231 | - block_pink
232 | - -60
233 | rotate_pink_block_left:
234 | - rotate_object
235 | - block_pink
236 | - 60
237 | push_red_block_right:
238 | - push_object
239 | - block_red
240 | - 0.1
241 | - 0
242 | push_red_block_left:
243 | - push_object
244 | - block_red
245 | - -0.1
246 | - 0
247 | push_blue_block_right:
248 | - push_object
249 | - block_blue
250 | - 0.1
251 | - 0
252 | push_blue_block_left:
253 | - push_object
254 | - block_blue
255 | - -0.1
256 | - 0
257 | push_pink_block_right:
258 | - push_object
259 | - block_pink
260 | - 0.1
261 | - 0
262 | push_pink_block_left:
263 | - push_object
264 | - block_pink
265 | - -0.1
266 | - 0
267 | move_slider_left:
268 | - move_door_rel
269 | - base__slide
270 | - 0.15
271 | move_slider_right:
272 | - move_door_rel
273 | - base__slide
274 | - -0.15
275 | open_drawer:
276 | - move_door_rel
277 | - base__drawer
278 | - 0.12
279 | close_drawer:
280 | - move_door_rel
281 | - base__drawer
282 | - -0.12
283 | lift_red_block_table:
284 | - lift_object
285 | - block_red
286 | - 0.05
287 | - table
288 | - base_link
289 | lift_red_block_slider:
290 | - lift_object
291 | - block_red
292 | - 0.03
293 | - table
294 | - plank_link
295 | lift_red_block_drawer:
296 | - lift_object
297 | - block_red
298 | - 0.05
299 | - table
300 | - drawer_link
301 | lift_blue_block_table:
302 | - lift_object
303 | - block_blue
304 | - 0.05
305 | - table
306 | - base_link
307 | lift_blue_block_slider:
308 | - lift_object
309 | - block_blue
310 | - 0.03
311 | - table
312 | - plank_link
313 | lift_blue_block_drawer:
314 | - lift_object
315 | - block_blue
316 | - 0.05
317 | - table
318 | - drawer_link
319 | lift_pink_block_table:
320 | - lift_object
321 | - block_pink
322 | - 0.05
323 | - table
324 | - base_link
325 | lift_pink_block_slider:
326 | - lift_object
327 | - block_pink
328 | - 0.03
329 | - table
330 | - plank_link
331 | lift_pink_block_drawer:
332 | - lift_object
333 | - block_pink
334 | - 0.05
335 | - table
336 | - drawer_link
337 | place_in_slider:
338 | - place_object
339 | - table
340 | - plank_link
341 | place_in_drawer:
342 | - place_object
343 | - table
344 | - drawer_link
345 | stack_block:
346 | - stack_objects
347 | unstack_block:
348 | - unstack_objects
349 | turn_on_lightbulb:
350 | - toggle_light
351 | - lightbulb
352 | - 0
353 | - 1
354 | turn_off_lightbulb:
355 | - toggle_light
356 | - lightbulb
357 | - 1
358 | - 0
359 | turn_on_led:
360 | - toggle_light
361 | - led
362 | - 0
363 | - 1
364 | turn_off_led:
365 | - toggle_light
366 | - led
367 | - 1
368 | - 0
369 | push_into_drawer:
370 | - push_object_into
371 | - - block_red
372 | - block_blue
373 | - block_pink
374 | - table
375 | - base_link
376 | - table
377 | - drawer_link
378 | recorder:
379 | record: ${record}
380 | record_fps: 30.0
381 | show_fps: false
382 | enable_tts: true
383 | seed: 0
384 | use_vr: true
385 | data_path: data
386 | save_dir: /home/meeso/recordings
387 | record: true
388 | load_dir: /work/dlclarge2/meeso-lfp/calvin_recordings/play_env_A/2021-09-12/11-36-05/
389 | show_gui: false
390 | processes: 16
391 | max_episode_frames: 1
392 | save_body_infos: true
393 | set_static_cam: false
394 |
--------------------------------------------------------------------------------
/real_preprocess/mujoco_menagerie/robotiq_2f85/2f85.xml:
--------------------------------------------------------------------------------
1 |