185 |
186 |
187 | The photo is from [this URL](https://commons.wikimedia.org/wiki/Category:360%C2%B0_panoramas_with_equirectangular_projection#/media/File:Braunschweig_Sankt-%C3%84gidien_Panorama_02.jpg)
188 |
6 | 
7 | 
8 |
9 | 
12 |
13 | MoGe is a powerful model for recovering 3D geometry from monocular open-domain images. The model consists of a ViT encoder and a convolutional decoder. It directly predicts an affine-invariant point map as well as a mask that excludes regions with undefined geometry (e.g., sky), from which the camera shift, camera focal length and depth map can be further derived.
14 |
15 | ***Check our [website](https://wangrc.site/MoGePage) for videos and interactive results!***
16 |
17 | ## Features
18 |
19 | * **Accurate 3D geometry estimation**: Estimate point maps from single images with high precision. Capable of capturing depth variations up to 1000×, ensuring a comprehensive scene representation.
20 | * **Optional ground-truth FOV input**: Enhance model accuracy further by providing the true field of view.
21 | * **Flexible resolution support**: Works seamlessly with various resolutions and aspect ratios, from 2:1 to 1:2.
22 | * **Optimized for speed**: Achieves <0.1s latency per image on an A100 / RTX 3090 GPU with fp16, and 0.2s with fp32.
23 |
24 | ## TODO List
25 |
26 | - [x] Release inference code & ViT-Large model.
27 | - [x] Release evaluation and training code.
28 | - [ ] Release ViT-Base and ViT-Giant models.
29 |
30 | 🌟*Updated on 2025/03/18* [CHANGELOG](CHANGELOG.md)
31 | - **Training and evaluation code released!**
32 | - Installation via pip and CLI usage supported.
33 |
34 | ## Installation
35 |
36 | ### Install via pip
37 |
38 | ```bash
39 | pip install git+https://github.com/microsoft/MoGe.git
40 | ```
41 |
42 | ### Or clone this repository
43 |
44 | ```bash
45 | git clone https://github.com/microsoft/MoGe.git
46 | cd MoGe
47 | ```
48 |
49 | and install the requirements
50 |
51 | ```bash
52 | pip install -r requirements.txt
53 | ```
54 |
55 | MoGe should be compatible with most requirements versions. Please check the `requirements.txt` for more details if you have concerns.
56 |
57 | ## Usage
58 |
59 | ### Pretrained model
60 |
61 | The ViT-Large model has been uploaded to Hugging Face hub at [Ruicheng/moge-vitl](https://huggingface.co/Ruicheng/moge-vitl).
62 | You may load the model via `MoGeModel.from_pretrained("Ruicheng/moge-vitl")` without manually downloading.
63 |
64 | If loading the model from a local file is preferred, you may manually download the model from the huggingface hub and load it via `MoGeModel.from_pretrained("PATH_TO_LOCAL_MODEL.pt")`.
65 |
66 | ### Minimal code example
67 |
68 | Here is a minimal example for loading the model and inferring on a single image.
69 |
70 | ```python
71 | import cv2
72 | import torch
73 | from moge.model.v1 import MoGeModel
74 |
75 | device = torch.device("cuda")
76 |
77 | # Load the model from huggingface hub (or load from local).
78 | model = MoGeModel.from_pretrained("Ruicheng/moge-vitl").to(device)
79 |
80 | # Read the input image and convert to tensor (3, H, W) and normalize to [0, 1]
81 | input_image = cv2.cvtColor(cv2.imread("PATH_TO_IMAGE.jpg"), cv2.COLOR_BGR2RGB)
82 | input_image = torch.tensor(input_image / 255, dtype=torch.float32, device=device).permute(2, 0, 1)
83 |
84 | # Infer
85 | output = model.infer(input_image)
86 | # `output` has keys "points", "depth", "mask" and "intrinsics",
87 | # The maps are in the same size as the input image.
88 | # {
89 | # "points": (H, W, 3), # scale-invariant point map in OpenCV camera coordinate system (x right, y down, z forward)
90 | # "depth": (H, W), # scale-invariant depth map
91 | # "mask": (H, W), # a binary mask for valid pixels.
92 | # "intrinsics": (3, 3), # normalized camera intrinsics
93 | # }
94 | # For more usage details, see the `MoGeModel.infer` docstring.
95 | ```
96 |
97 | ### Gradio demo | `moge app`
98 |
99 | The demo is also available at our [Hugging Face space](https://huggingface.co/spaces/Ruicheng/MoGe).
100 |
101 | ```bash
102 | # Using the command line tool
103 | moge app
104 |
105 | # In this repo
106 | python moge/scripts/app.py # --share for Gradio public sharing
107 | ```
108 |
109 | See also [`moge/scripts/app.py`](moge/scripts/app.py)
110 |
111 |
112 | ### Inference | `moge infer`
113 |
114 | Run the script `moge/scripts/infer.py` via the following command:
115 |
116 | ```bash
117 | # Save the output [maps], [glb] and [ply] files
118 | moge infer -i IMAGES_FOLDER_OR_IMAGE_PATH --o OUTPUT_FOLDER --maps --glb --ply
119 |
120 | # Show the result in a window (requires pyglet < 2.0, e.g. pip install pyglet==1.5.29)
121 | moge infer -i IMAGES_FOLDER_OR_IMAGE_PATH --o OUTPUT_FOLDER --show
122 | ```
123 |
124 | For detailed options, run `moge infer --help`:
125 |
126 | ```
127 | Usage: moge infer [OPTIONS]
128 |
129 | Inference script for the MoGe model.
130 |
131 | Options:
132 | -i, --input PATH Input image or folder path. "jpg" and "png" are
133 | supported.
134 | --fov_x FLOAT If camera parameters are known, set the
135 | horizontal field of view in degrees. Otherwise,
136 | MoGe will estimate it.
137 | -o, --output PATH Output folder path
138 | --pretrained TEXT Pretrained model name or path. Defaults to
139 | "Ruicheng/moge-vitl"
140 | --device TEXT Device name (e.g. "cuda", "cuda:0", "cpu").
141 | Defaults to "cuda"
142 | --fp16 Use fp16 precision for 2x faster inference.
143 | --resize INTEGER Resize the image(s) & output maps to a specific
144 | size. Defaults to None (no resizing).
145 | --resolution_level INTEGER An integer [0-9] for the resolution level for
146 | inference. Higher value means more tokens and
147 | the finer details will be captured, but
148 | inference can be slower. Defaults to 9. Note
149 | that it is irrelevant to the output size, which
150 | is always the same as the input size.
151 | `resolution_level` actually controls
152 | `num_tokens`. See `num_tokens` for more details.
153 | --num_tokens INTEGER number of tokens used for inference. A integer
154 | in the (suggested) range of `[1200, 2500]`.
155 | `resolution_level` will be ignored if
156 | `num_tokens` is provided. Default: None
157 | --threshold FLOAT Threshold for removing edges. Defaults to 0.03.
158 | Smaller value removes more edges. "inf" means no
159 | thresholding.
160 | --maps Whether to save the output maps and fov(image,
161 | depth, mask, points, fov).
162 | --glb Whether to save the output as a.glb file. The
163 | color will be saved as a texture.
164 | --ply Whether to save the output as a.ply file. The
165 | color will be saved as vertex colors.
166 | --show Whether show the output in a window. Note that
167 | this requires pyglet<2 installed as required by
168 | trimesh.
169 | --help Show this message and exit.
170 | ```
171 |
172 | See also [`moge/scripts/infer.py`](moge/scripts/infer.py)
173 |
174 | ### 360° panorama images | `moge infer_panorama`
175 |
176 | > *NOTE: This is an experimental extension of MoGe.*
177 |
178 | The script will split the 360-degree panorama image into multiple perspective views and infer on each view separately.
179 | The output maps will be combined to produce a panorama depth map and point map.
180 |
181 | Note that the panorama image must have spherical parameterization (e.g., environment maps or equirectangular images). Other formats must be converted to spherical format before using this script. Run `moge infer_panorama --help` for detailed options.
182 |
183 |
184 |