└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # laion-3d 2 | Collect large 3d dataset and build models 3 | 4 | 5 | https://github.com/LAION-AI/project-menu/issues/23 6 | 7 | 8 | ## Dataset specs 9 | 10 | Can be: 11 | * 3d alone 12 | * 3d text 13 | * 3d image 14 | 15 | Examples: 16 | * https://github.com/rom1504/minecraft-schematics-dataset 17 | 18 | ## Datasets 19 | 20 | * [30k samples from thingiverse](https://zenodo.org/record/1098527) (3d printing STL model files) 21 | * [Fusion360Gallery](https://github.com/AutodeskAILab/Fusion360GalleryDataset) 22 | * [Amazon Berkeley Objects (ABO)](https://amazon-berkeley-objects.s3.amazonaws.com/index.html#download), a dataset of Amazon products with metadata, catalog images, and 3D models. 23 | * [Large Geometric Models Archive](https://www.cc.gatech.edu/projects/large_models/about.html) 24 | * [FaceScape](https://facescape.nju.edu.cn/Page_Download/), a large-scale detailed 3D face dataset (application required). 25 | * [Redwood 3DScan](https://github.com/isl-org/redwood-3dscan), more than ten thousand 3D scans of real objects. 26 | * [Human3.6M](http://vision.imar.ro/human3.6m/description.php), 3.6 million 3D human poses and corresponding images. 27 | * [Semantic3D](https://www.semantic3d.net/), a large labelled 3D point cloud data set of natural scenes with over 4 billion points in total. 28 | * [SceneNN / ObjectNN](https://github.com/hkust-vgd/scenenn), an RGB-D dataset with more than 100 indoor scenes along with RGB-D objects extracted and split into 20 categories. 29 | * [3D-FRONT](https://tianchi.aliyun.com/specials/promotion/alibaba-3d-scene-dataset), a dataset of 3D furnished rooms with layouts and semantics. 30 | * [3D-FUTURE](https://tianchi.aliyun.com/specials/promotion/alibaba-3d-future), a dataset of 3D furniture shapes with textures. 31 | * [ABC](https://deep-geometry.github.io/abc-dataset/0), a collection of one million Computer-Aided Design (CAD) models 32 | * [Structured3D](https://structured3d-dataset.org/#download), a large-scale photo-realistic dataset containing 3.5K house designs with a variety of ground truth 3D structure annotations. 33 | * [ShapeNet](https://shapenet.org/), a richly-annotated, large-scale dataset of 3D shapes. 34 | * [FixIt!](https://drive.google.com/drive/folders/1h9kMRilQcjbD4Tyt58pmMUEnMIicNATi), a dataset that contains about 5k poorly-designed 3D physical objects paired with choices to fix them. 35 | * [ModelNet](http://modelnet.cs.princeton.edu/#), a comprehensive clean collection of 3D CAD models for objects. 36 | 37 | ## Models 38 | 39 | ### Depth Estimation 40 | 41 | * Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging 42 | * Paper: [https://arxiv.org/abs/2105.14021](https://arxiv.org/abs/2105.14021) 43 | * Code: [https://github.com/compphoto/BoostingMonocularDepth](https://github.com/compphoto/BoostingMonocularDepth) 44 | * Follow up paper: [https://arxiv.org/abs/2012.09365](https://arxiv.org/abs/2012.09365) 45 | * Follow up code: [https://github.com/aim-uofa/AdelaiDepth](https://github.com/aim-uofa/AdelaiDepth) (although firs code includes it) 46 | * Self-supervised Learning of Depth Inference for Multi-view Stereo 47 | * Paper: [https://arxiv.org/abs/2104.02972](https://arxiv.org/abs/2104.02972) 48 | * Code: [https://github.com/JiayuYANG/Self-supervised-CVP-MVSNet](https://github.com/JiayuYANG/Self-supervised-CVP-MVSNet) 49 | 50 | ### Generation 51 | 52 | * Sketch2Model - View-Aware 3D Modeling from Single Free-Hand Sketches 53 | * Paper: [https://arxiv.org/abs/2105.06663](https://arxiv.org/abs/2105.06663) 54 | * Code: [https://github.com/bennyguo/sketch2model](https://github.com/bennyguo/sketch2model) 55 | * SceneFormer - Indoor Scene Generation with Transformers 56 | * Paper: [https://arxiv.org/abs/2012.09793](https://arxiv.org/abs/2012.09793) 57 | * Code: [https://github.com/cy94/sceneformer](https://github.com/cy94/sceneformer) 58 | * Image2Lego - Customized LEGO Set Generation from Images 59 | * Paper: [https://arxiv.org/abs/2108.08477](https://arxiv.org/abs/2108.08477) 60 | * Code: 😥 61 | * Neural RGB-D Surface Reconstruction 62 | * Paper: [https://arxiv.org/abs/2104.04532](https://arxiv.org/abs/2104.04532) 63 | * Code: 😥 64 | * SP-GAN - Sphere-Guided 3D Shape Generation and Manipulation 65 | * Paper: [https://arxiv.org/abs/2108.04476](https://arxiv.org/abs/2108.04476) 66 | * Code: [https://github.com/liruihui/sp-gan](https://github.com/liruihui/sp-gan) 67 | * Style-based Point Generator with Adversarial Rendering for Point Cloud Completion 68 | * Paper: [https://arxiv.org/abs/2103.02535](https://arxiv.org/abs/2103.02535) 69 | * Code: [https://github.com/microsoft/SpareNet](https://github.com/microsoft/SpareNet) 70 | * Learning to Stylize Novel Views 71 | * Paper: [https://arxiv.org/abs/2105.13509](https://arxiv.org/abs/2105.13509) 72 | * Code: [https://github.com/hhsinping/stylescene](https://github.com/hhsinping/stylescene) 73 | * RetrievalFuse - Neural 3D Scene Reconstruction with a Database 74 | * Paper: [https://arxiv.org/abs/2104.00024](https://arxiv.org/abs/2104.00024) 75 | * Code: [https://github.com/nihalsid/retrieval-fuse](https://github.com/nihalsid/retrieval-fuse) 76 | * Geometry-Free View Synthesis - Transformers and no 3D Priors 77 | * Paper: [https://arxiv.org/abs/2104.07652](https://arxiv.org/abs/2104.07652) 78 | * Code: [https://github.com/CompVis/geometry-free-view-synthesis](https://github.com/CompVis/geometry-free-view-synthesis) 79 | * ShapeFormer - Transformer-based Shape Completion via Sparse Representation 80 | * Paper: [https://arxiv.org/abs/2201.10326](https://arxiv.org/abs/2201.10326) 81 | * Code: 😥 82 | 83 | ### Representations (🙄) 84 | 85 | #### SDF-based (mostly) 86 | 87 | * Neural Parts - Learning Expressive 3D Shape Abstractions with Invertible Neural Representations 88 | * Paper: [https://arxiv.org/abs/2103.10429](https://arxiv.org/abs/2103.10429) 89 | * Code: [https://github.com/paschalidoud/neural_parts](https://github.com/paschalidoud/neural_parts) 90 | * DeepSDF - Learning Continuous Signed Distance Functions for Shape Representation 91 | * Paper: [https://arxiv.org/abs/1901.05103](https://arxiv.org/abs/1901.05103) 92 | * Code: [https://github.com/Facebookresearch/deepsdf](https://github.com/Facebookresearch/deepsdf) 93 | * Spline Positional Encoding for Learning 3D Implicit Signed Distance Fields 94 | * Paper: https://arxiv.org/abs/2106.01553 95 | * Code: [https://github.com/microsoft/SplinePosEnc](https://github.com/microsoft/SplinePosEnc) 96 | 97 | #### Hashing 98 | 99 | * Instant Neural Graphics Primitives with a Multiresolution Hash Encoding 100 | * Paper: [https://arxiv.org/abs/2201.05989](https://arxiv.org/abs/2201.05989) 101 | * Code: [https://github.com/nvlabs/instant-ngp](https://github.com/nvlabs/instant-ngp) 102 | 103 | 104 | #### Implicit 105 | 106 | * Differentiable Gradient Sampling for Learning Implicit 3D Scene Reconstructions from a Single Image 107 | * Paper: [https://openreview.net/forum?id=U8pbd00cCWB](https://openreview.net/forum?id=U8pbd00cCWB) 108 | * Code: maskedURL lol 109 | * NeuS - Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction 110 | * Paper: [https://arxiv.org/abs/2106.10689](https://arxiv.org/abs/2106.10689) 111 | * Code: [https://github.com/Totoro97/NeuS](https://github.com/Totoro97/NeuS) 112 | * From data to functa - Your data point is a function 113 | * Paper: [https://arxiv.org/abs/2201.12204](https://arxiv.org/abs/2201.12204) 114 | * Code: 😥 115 | * Previous work: [https://arxiv.org/abs/2102.04776](https://arxiv.org/abs/2102.04776) 116 | * Code: https://github.com/EmilienDupont/neural-function-distributions 117 | * Multiresolution Deep Implicit Functions for 3D Shape Representation 118 | * Paper: [https://arxiv.org/abs/2109.05591](https://arxiv.org/abs/2109.05591) 119 | * Code: 😥 120 | * Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields 121 | * Paper: [https://arxiv.org/abs/2106.05187](https://arxiv.org/abs/2106.05187) 122 | * Code: [https://github.com/yifita/idf](https://github.com/yifita/idf) 123 | * Implicit Neural Representations with Periodic Activation Functions 124 | * Paper: [https://arxiv.org/abs/2006.09661](https://arxiv.org/abs/2006.09661) 125 | * Follow up: [https://arxiv.org/abs/2104.03960](https://arxiv.org/abs/2104.03960) 126 | * Code: [https://github.com/lucidrains/siren-pytorch](https://github.com/lucidrains/siren-pytorch) 127 | * Volume Rendering of Neural Implicit Surfaces 128 | * Paper: [https://arxiv.org/abs/2106.12052](https://arxiv.org/abs/2106.12052) 129 | * Code: [https://github.com/ventusff/neurecon](https://github.com/ventusff/neurecon) 130 | * HyperCube - Implicit Field Representations of Voxelized 3D Models 131 | * Paper: [https://arxiv.org/abs/2110.05770](https://arxiv.org/abs/2110.05770) 132 | * Code: [https://github.com/mproszewska/hypercube](https://github.com/mproszewska/hypercube) 133 | * Convolutional Occupancy Networks 134 | * Paper: [https://arxiv.org/abs/2003.04618](https://arxiv.org/abs/2003.04618) 135 | * Code: https://github.com/autonomousvision/convolutional_occupancy_networks 136 | 137 | ### Neural/Differential Rendering 138 | 139 | * GANcraft - Unsupervised 3D Neural Rendering of Minecraft Worlds 140 | * Paper: [https://arxiv.org/abs/2104.07659](https://arxiv.org/abs/2104.07659) 141 | * Code: [https://github.com/NVlabs/imaginaire](https://github.com/NVlabs/imaginaire) 142 | * ADOP: Approximate Differentiable One-Pixel Point Rendering 143 | * Paper: [https://arxiv.org/abs/2110.06635](https://arxiv.org/abs/2110.06635) 144 | * Code: [https://github.com/darglein/ADOP](https://github.com/darglein/ADOP) 145 | * youtube: 146 | 147 | 148 | ### NeRF 149 | 150 | * Editing Conditional Radiance Fields 151 | * Paper: [http://editnerf.csail.mit.edu/paper.pdf](http://editnerf.csail.mit.edu/paper.pdf) 152 | * Code: [https://github.com/stevliu/editnerf](https://github.com/stevliu/editnerf) 153 | * GIRAFFE - Representing Scenes as Compositional Generative Neural Feature Fields 154 | * Paper: [https://arxiv.org/abs/2011.12100](https://arxiv.org/abs/2011.12100) 155 | * Code: [https://github.com/autonomousvision/giraffe](https://github.com/autonomousvision/giraffe) 156 | * NeX - Real-time View Synthesis with Neural Basis Expansion 157 | * Paper: [https://arxiv.org/abs/2103.05606](https://arxiv.org/abs/2103.05606) 158 | * Code: [https://github.com/nex-mpi/nex-code](https://github.com/nex-mpi/nex-code) 159 | * Putting NeRF on a Diet - Semantically Consistent Few-Shot View Synthesis 160 | * Paper: [https://arxiv.org/abs/2104.00677](https://arxiv.org/abs/2104.00677) 161 | * Code: [https://github.com/ajayjain/DietNeRF](https://github.com/ajayjain/DietNeRF) 162 | * Unconstrained Scene Generation with Locally Conditioned Radiance Fields 163 | * Paper: [https://arxiv.org/abs/2104.00670](https://arxiv.org/abs/2104.00670) 164 | * Code: [https://github.com/apple/ml-gsn](https://github.com/apple/ml-gsn) 165 | * Zero-Shot Text-Guided Object Generation with Dream Fields 166 | * Paper: [https://arxiv.org/abs/2112.01455](https://arxiv.org/abs/2112.01455) 167 | * Code: [https://github.com/google-research/google-research/tree/master/dreamfields](https://github.com/google-research/google-research/tree/master/dreamfields) 168 | 169 | 170 | ### Diffusion 171 | 172 | * Diffusion Probabilistic Models for 3D Point Cloud Generation 173 | * Paper: [https://arxiv.org/abs/2103.01458](https://arxiv.org/abs/2103.01458) 174 | * Code: [https://github.com/luost26/diffusion-point-cloud](https://github.com/luost26/diffusion-point-cloud) 175 | * 3D Shape Generation and Completion through Point-Voxel Diffusion 176 | * Paper: [https://arxiv.org/abs/2104.03670](https://arxiv.org/abs/2104.03670) 177 | * Code: [https://github.com/alexzhou907/PVD](https://github.com/alexzhou907/PVD) 178 | * A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion 179 | * Paper: [https://arxiv.org/abs/2112.03530](https://arxiv.org/abs/2112.03530) 180 | * Code: [https://github.com/zhaoyanglyu/point_diffusion_refinement](https://github.com/zhaoyanglyu/point_diffusion_refinement) (empty) 181 | 182 | 183 | ### GANs 184 | 185 | * MOGAN - Morphologic-structure-aware Generative Learning from a Single Image 186 | * Paper: [https://arxiv.org/abs/2103.02997](https://arxiv.org/abs/2103.02997) 187 | * Code: https://github.com/JinshuChen/MOGAN 188 | 189 | 190 | ### Design / Practice / Creative - Related 191 | 192 | * Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images 193 | * Paper: [https://arxiv.org/abs/2108.09022](https://arxiv.org/abs/2108.09022) 194 | * Code: [https://github.com/mingjiayang/sgsdi](https://github.com/mingjiayang/sgsdi) 195 | * ATISS - Autoregressive Transformers for Indoor Scene Synthesis 196 | * Paper: [https://arxiv.org/abs/2110.03675](https://arxiv.org/abs/2110.03675) 197 | * Code: 😥 198 | * Computer-Aided Design as Language 199 | * Paper: [https://arxiv.org/abs/2105.02769](https://arxiv.org/abs/2105.02769) 200 | * Code: 😥 201 | * Patch2CAD - Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image 202 | * Paper: [https://arxiv.org/abs/2108.09368](https://arxiv.org/abs/2108.09368) 203 | * Code: 😥 204 | * Modeling Artistic Workflows for Image Generation and Editing 205 | * Paper: [https://arxiv.org/abs/2007.07238](https://arxiv.org/abs/2007.07238) 206 | * Code: [https://github.com/hytseng0509/ArtEditing](https://github.com/hytseng0509/ArtEditing) 207 | 208 | 209 | ### Benchmarks 210 | 211 | ### contrastive 212 | 213 | * (3d, text) 214 | * (3d, image) 215 | 216 | ### Synthetic Data Generation 217 | 218 | https://github.com/google-research/kubric from uwu1 219 | 220 | 3d -> text 221 | 222 | text -> 3d 223 | 224 | 3d -> image 225 | 226 | image -> 3d 227 | --------------------------------------------------------------------------------