├── .gitignore ├── INSTALL.md ├── LICENSE.txt ├── README.md ├── cpp_wrappers ├── compile_wrappers.sh ├── cpp_neighbors │ ├── build.bat │ ├── neighbors │ │ ├── neighbors.cpp │ │ └── neighbors.h │ ├── setup.py │ └── wrapper.cpp ├── cpp_subsampling │ ├── build.bat │ ├── grid_subsampling │ │ ├── grid_subsampling.cpp │ │ └── grid_subsampling.h │ ├── setup.py │ └── wrapper.cpp └── cpp_utils │ ├── cloud │ ├── cloud.cpp │ └── cloud.h │ └── nanoflann │ └── nanoflann.hpp ├── datasets ├── ModelNet40.py ├── NPM3D.py ├── S3DIS.py ├── SemanticKitti.py ├── SensatUrban.py ├── Toronto3D.py └── common.py ├── doc ├── Github_intro.png ├── object_classification_guide.md ├── pretrained_models_guide.md ├── scene_segmentation_guide.md ├── slam_segmentation_guide.md └── visualization_guide.md ├── kernels └── kernel_points.py ├── models ├── architectures.py └── blocks.py ├── plot_convergence.py ├── test_models.py ├── train_ModelNet40.py ├── train_NPM3D.py ├── train_S3DIS.py ├── train_SemanticKitti.py ├── train_SensatUrban.py ├── train_Toronto3D.py ├── utils ├── config.py ├── mayavi_visu.py ├── metrics.py ├── ply.py ├── tester.py ├── trainer.py └── visualizer.py └── visualize_deformations.py /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | **/build 3 | **/desktop.ini 4 | /results 5 | /test 6 | /docker_scripts 7 | /kernels/dispositions 8 | core 9 | 10 | # VSCode related 11 | *.code-workspace 12 | 13 | # Byte-compiled / optimized / DLL files 14 | __pycache__/ 15 | *.py[cod] 16 | 17 | # C extensions 18 | *.so 19 | -------------------------------------------------------------------------------- /INSTALL.md: -------------------------------------------------------------------------------- 1 | 2 | # Installation instructions 3 | 4 | ## Ubuntu 18.04 5 | 6 | * Make sure CUDA and cuDNN are installed. One configuration has been tested: 7 | - PyTorch 1.4.0, CUDA 10.1 and cuDNN 7.6 8 | 9 | * Ensure all python packages are installed : 10 | 11 | sudo apt update 12 | sudo apt install python3-dev python3-pip python3-tk 13 | 14 | * Follow PyTorch installation procedure. 15 | 16 | * Install the other dependencies with pip: 17 | - numpy 18 | - scikit-learn 19 | - PyYAML 20 | - matplotlib (for visualization) 21 | - mayavi (for visualization) 22 | - PyQt5 (for visualization) 23 | 24 | * Compile the C++ extension modules for python located in `cpp_wrappers`. Open a terminal in this folder, and run: 25 | 26 | sh compile_wrappers.sh 27 | 28 | You should now be able to train Kernel-Point Convolution models 29 | 30 | ## Windows 10 31 | 32 | * Make sure CUDA and cuDNN are installed. One configuration has been tested: 33 | - PyTorch 1.4.0, CUDA 10.1 and cuDNN 7.5 34 | 35 | * Follow PyTorch installation procedure. 36 | 37 | * We used the PyCharm IDE to pip install all python dependencies (including PyTorch) in a venv: 38 | - torch 39 | - torchvision 40 | - numpy 41 | - scikit-learn 42 | - PyYAML 43 | - matplotlib (for visualization) 44 | - mayavi (for visualization) 45 | - PyQt5 (for visualization) 46 | 47 | * Compile the C++ extension modules for python located in `cpp_wrappers`. You just have to execute two .bat files: 48 | 49 | cpp_wrappers/cpp_neighbors/build.bat 50 | 51 | and 52 | 53 | cpp_wrappers/cpp_subsampling/build.bat 54 | 55 | You should now be able to train Kernel-Point Convolution models 56 | 57 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2019 HuguesTHOMAS 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | ![Intro figure](https://github.com/HuguesTHOMAS/KPConv-PyTorch/blob/master/doc/Github_intro.png) 3 | 4 | Created by Hugues THOMAS 5 | 6 | ## Introduction 7 | 8 | This repository contains the implementation of **Kernel Point Convolution** (KPConv) in [PyTorch](https://pytorch.org/). 9 | 10 | KPConv is also available in [Tensorflow](https://github.com/HuguesTHOMAS/KPConv) (original but older implementation). 11 | 12 | Another implementation of KPConv is available in [PyTorch-Points-3D](https://github.com/nicolas-chaulet/torch-points3d) 13 | 14 | KPConv is a point convolution operator presented in our ICCV2019 paper ([arXiv](https://arxiv.org/abs/1904.08889)). If you find our work useful in your 15 | research, please consider citing: 16 | 17 | ``` 18 | @article{thomas2019KPConv, 19 | Author = {Thomas, Hugues and Qi, Charles R. and Deschaud, Jean-Emmanuel and Marcotegui, Beatriz and Goulette, Fran{\c{c}}ois and Guibas, Leonidas J.}, 20 | Title = {KPConv: Flexible and Deformable Convolution for Point Clouds}, 21 | Journal = {Proceedings of the IEEE International Conference on Computer Vision}, 22 | Year = {2019} 23 | } 24 | ``` 25 | 26 | ## Installation 27 | 28 | This implementation has been tested on Ubuntu 18.04 and Windows 10. Details are provided in [INSTALL.md](./INSTALL.md). 29 | 30 | 31 | ## Experiments 32 | 33 | We provide scripts for three experiments: ModelNet40, S3DIS and SemanticKitti. The instructions to run these 34 | experiments are in the [doc](./doc) folder. 35 | 36 | * [Object Classification](./doc/object_classification_guide.md): Instructions to train KP-CNN on an object classification 37 | task (Modelnet40). 38 | 39 | * [Scene Segmentation](./doc/scene_segmentation_guide.md): Instructions to train KP-FCNN on a scene segmentation 40 | task (S3DIS). 41 | 42 | * [SLAM Segmentation](./doc/slam_segmentation_guide.md): Instructions to train KP-FCNN on a slam segmentation 43 | task (SemanticKitti). 44 | 45 | * [Pretrained models](./doc/pretrained_models_guide.md): We provide pretrained weights and instructions to load them. 46 | 47 | * [Visualization scripts](./doc/visualization_guide.md): For now only one visualization script has been implemented: 48 | the kernel deformations display. 49 | 50 | ## Acknowledgment 51 | 52 | Our code uses the nanoflann library. 53 | 54 | ## License 55 | Our code is released under MIT License (see LICENSE file for details). 56 | 57 | ## Updates 58 | * 27/04/2020: Initial release. 59 | * 27/04/2020: Added NPM3D support thanks to @GeoSur. 60 | -------------------------------------------------------------------------------- /cpp_wrappers/compile_wrappers.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Compile cpp subsampling 4 | cd cpp_subsampling 5 | python3 setup.py build_ext --inplace 6 | cd .. 7 | 8 | # Compile cpp neighbors 9 | cd cpp_neighbors 10 | python3 setup.py build_ext --inplace 11 | cd .. -------------------------------------------------------------------------------- /cpp_wrappers/cpp_neighbors/build.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | py setup.py build_ext --inplace 3 | 4 | 5 | pause -------------------------------------------------------------------------------- /cpp_wrappers/cpp_neighbors/neighbors/neighbors.cpp: -------------------------------------------------------------------------------- 1 | 2 | #include "neighbors.h" 3 | 4 | 5 | void brute_neighbors(vector& queries, vector& supports, vector& neighbors_indices, float radius, int verbose) 6 | { 7 | 8 | // Initialize variables 9 | // ****************** 10 | 11 | // square radius 12 | float r2 = radius * radius; 13 | 14 | // indices 15 | int i0 = 0; 16 | 17 | // Counting vector 18 | int max_count = 0; 19 | vector> tmp(queries.size()); 20 | 21 | // Search neigbors indices 22 | // *********************** 23 | 24 | for (auto& p0 : queries) 25 | { 26 | int i = 0; 27 | for (auto& p : supports) 28 | { 29 | if ((p0 - p).sq_norm() < r2) 30 | { 31 | tmp[i0].push_back(i); 32 | if (tmp[i0].size() > max_count) 33 | max_count = tmp[i0].size(); 34 | } 35 | i++; 36 | } 37 | i0++; 38 | } 39 | 40 | // Reserve the memory 41 | neighbors_indices.resize(queries.size() * max_count); 42 | i0 = 0; 43 | for (auto& inds : tmp) 44 | { 45 | for (int j = 0; j < max_count; j++) 46 | { 47 | if (j < inds.size()) 48 | neighbors_indices[i0 * max_count + j] = inds[j]; 49 | else 50 | neighbors_indices[i0 * max_count + j] = -1; 51 | } 52 | i0++; 53 | } 54 | 55 | return; 56 | } 57 | 58 | void ordered_neighbors(vector& queries, 59 | vector& supports, 60 | vector& neighbors_indices, 61 | float radius) 62 | { 63 | 64 | // Initialize variables 65 | // ****************** 66 | 67 | // square radius 68 | float r2 = radius * radius; 69 | 70 | // indices 71 | int i0 = 0; 72 | 73 | // Counting vector 74 | int max_count = 0; 75 | float d2; 76 | vector> tmp(queries.size()); 77 | vector> dists(queries.size()); 78 | 79 | // Search neigbors indices 80 | // *********************** 81 | 82 | for (auto& p0 : queries) 83 | { 84 | int i = 0; 85 | for (auto& p : supports) 86 | { 87 | d2 = (p0 - p).sq_norm(); 88 | if (d2 < r2) 89 | { 90 | // Find order of the new point 91 | auto it = std::upper_bound(dists[i0].begin(), dists[i0].end(), d2); 92 | int index = std::distance(dists[i0].begin(), it); 93 | 94 | // Insert element 95 | dists[i0].insert(it, d2); 96 | tmp[i0].insert(tmp[i0].begin() + index, i); 97 | 98 | // Update max count 99 | if (tmp[i0].size() > max_count) 100 | max_count = tmp[i0].size(); 101 | } 102 | i++; 103 | } 104 | i0++; 105 | } 106 | 107 | // Reserve the memory 108 | neighbors_indices.resize(queries.size() * max_count); 109 | i0 = 0; 110 | for (auto& inds : tmp) 111 | { 112 | for (int j = 0; j < max_count; j++) 113 | { 114 | if (j < inds.size()) 115 | neighbors_indices[i0 * max_count + j] = inds[j]; 116 | else 117 | neighbors_indices[i0 * max_count + j] = -1; 118 | } 119 | i0++; 120 | } 121 | 122 | return; 123 | } 124 | 125 | void batch_ordered_neighbors(vector& queries, 126 | vector& supports, 127 | vector& q_batches, 128 | vector& s_batches, 129 | vector& neighbors_indices, 130 | float radius) 131 | { 132 | 133 | // Initialize variables 134 | // ****************** 135 | 136 | // square radius 137 | float r2 = radius * radius; 138 | 139 | // indices 140 | int i0 = 0; 141 | 142 | // Counting vector 143 | int max_count = 0; 144 | float d2; 145 | vector> tmp(queries.size()); 146 | vector> dists(queries.size()); 147 | 148 | // batch index 149 | int b = 0; 150 | int sum_qb = 0; 151 | int sum_sb = 0; 152 | 153 | 154 | // Search neigbors indices 155 | // *********************** 156 | 157 | for (auto& p0 : queries) 158 | { 159 | // Check if we changed batch 160 | if (i0 == sum_qb + q_batches[b]) 161 | { 162 | sum_qb += q_batches[b]; 163 | sum_sb += s_batches[b]; 164 | b++; 165 | } 166 | 167 | // Loop only over the supports of current batch 168 | vector::iterator p_it; 169 | int i = 0; 170 | for(p_it = supports.begin() + sum_sb; p_it < supports.begin() + sum_sb + s_batches[b]; p_it++ ) 171 | { 172 | d2 = (p0 - *p_it).sq_norm(); 173 | if (d2 < r2) 174 | { 175 | // Find order of the new point 176 | auto it = std::upper_bound(dists[i0].begin(), dists[i0].end(), d2); 177 | int index = std::distance(dists[i0].begin(), it); 178 | 179 | // Insert element 180 | dists[i0].insert(it, d2); 181 | tmp[i0].insert(tmp[i0].begin() + index, sum_sb + i); 182 | 183 | // Update max count 184 | if (tmp[i0].size() > max_count) 185 | max_count = tmp[i0].size(); 186 | } 187 | i++; 188 | } 189 | i0++; 190 | } 191 | 192 | // Reserve the memory 193 | neighbors_indices.resize(queries.size() * max_count); 194 | i0 = 0; 195 | for (auto& inds : tmp) 196 | { 197 | for (int j = 0; j < max_count; j++) 198 | { 199 | if (j < inds.size()) 200 | neighbors_indices[i0 * max_count + j] = inds[j]; 201 | else 202 | neighbors_indices[i0 * max_count + j] = supports.size(); 203 | } 204 | i0++; 205 | } 206 | 207 | return; 208 | } 209 | 210 | 211 | void batch_nanoflann_neighbors(vector& queries, 212 | vector& supports, 213 | vector& q_batches, 214 | vector& s_batches, 215 | vector& neighbors_indices, 216 | float radius) 217 | { 218 | 219 | // Initialize variables 220 | // ****************** 221 | 222 | // indices 223 | int i0 = 0; 224 | 225 | // Square radius 226 | float r2 = radius * radius; 227 | 228 | // Counting vector 229 | int max_count = 0; 230 | float d2; 231 | vector>> all_inds_dists(queries.size()); 232 | 233 | // batch index 234 | int b = 0; 235 | int sum_qb = 0; 236 | int sum_sb = 0; 237 | 238 | // Nanoflann related variables 239 | // *************************** 240 | 241 | // CLoud variable 242 | PointCloud current_cloud; 243 | 244 | // Tree parameters 245 | nanoflann::KDTreeSingleIndexAdaptorParams tree_params(10 /* max leaf */); 246 | 247 | // KDTree type definition 248 | typedef nanoflann::KDTreeSingleIndexAdaptor< nanoflann::L2_Simple_Adaptor , 249 | PointCloud, 250 | 3 > my_kd_tree_t; 251 | 252 | // Pointer to trees 253 | my_kd_tree_t* index; 254 | 255 | // Build KDTree for the first batch element 256 | current_cloud.pts = vector(supports.begin() + sum_sb, supports.begin() + sum_sb + s_batches[b]); 257 | index = new my_kd_tree_t(3, current_cloud, tree_params); 258 | index->buildIndex(); 259 | 260 | 261 | // Search neigbors indices 262 | // *********************** 263 | 264 | // Search params 265 | nanoflann::SearchParams search_params; 266 | search_params.sorted = true; 267 | 268 | for (auto& p0 : queries) 269 | { 270 | 271 | // Check if we changed batch 272 | if (i0 == sum_qb + q_batches[b]) 273 | { 274 | sum_qb += q_batches[b]; 275 | sum_sb += s_batches[b]; 276 | b++; 277 | 278 | // Change the points 279 | current_cloud.pts.clear(); 280 | current_cloud.pts = vector(supports.begin() + sum_sb, supports.begin() + sum_sb + s_batches[b]); 281 | 282 | // Build KDTree of the current element of the batch 283 | delete index; 284 | index = new my_kd_tree_t(3, current_cloud, tree_params); 285 | index->buildIndex(); 286 | } 287 | 288 | // Initial guess of neighbors size 289 | all_inds_dists[i0].reserve(max_count); 290 | 291 | // Find neighbors 292 | float query_pt[3] = { p0.x, p0.y, p0.z}; 293 | size_t nMatches = index->radiusSearch(query_pt, r2, all_inds_dists[i0], search_params); 294 | 295 | // Update max count 296 | if (nMatches > max_count) 297 | max_count = nMatches; 298 | 299 | // Increment query idx 300 | i0++; 301 | } 302 | 303 | // Reserve the memory 304 | neighbors_indices.resize(queries.size() * max_count); 305 | i0 = 0; 306 | sum_sb = 0; 307 | sum_qb = 0; 308 | b = 0; 309 | for (auto& inds_dists : all_inds_dists) 310 | { 311 | // Check if we changed batch 312 | if (i0 == sum_qb + q_batches[b]) 313 | { 314 | sum_qb += q_batches[b]; 315 | sum_sb += s_batches[b]; 316 | b++; 317 | } 318 | 319 | for (int j = 0; j < max_count; j++) 320 | { 321 | if (j < inds_dists.size()) 322 | neighbors_indices[i0 * max_count + j] = inds_dists[j].first + sum_sb; 323 | else 324 | neighbors_indices[i0 * max_count + j] = supports.size(); 325 | } 326 | i0++; 327 | } 328 | 329 | delete index; 330 | 331 | return; 332 | } 333 | 334 | -------------------------------------------------------------------------------- /cpp_wrappers/cpp_neighbors/neighbors/neighbors.h: -------------------------------------------------------------------------------- 1 | 2 | 3 | #include "../../cpp_utils/cloud/cloud.h" 4 | #include "../../cpp_utils/nanoflann/nanoflann.hpp" 5 | 6 | #include 7 | #include 8 | 9 | using namespace std; 10 | 11 | 12 | void ordered_neighbors(vector& queries, 13 | vector& supports, 14 | vector& neighbors_indices, 15 | float radius); 16 | 17 | void batch_ordered_neighbors(vector& queries, 18 | vector& supports, 19 | vector& q_batches, 20 | vector& s_batches, 21 | vector& neighbors_indices, 22 | float radius); 23 | 24 | void batch_nanoflann_neighbors(vector& queries, 25 | vector& supports, 26 | vector& q_batches, 27 | vector& s_batches, 28 | vector& neighbors_indices, 29 | float radius); 30 | -------------------------------------------------------------------------------- /cpp_wrappers/cpp_neighbors/setup.py: -------------------------------------------------------------------------------- 1 | from distutils.core import setup, Extension 2 | import numpy.distutils.misc_util 3 | 4 | # Adding OpenCV to project 5 | # ************************ 6 | 7 | # Adding sources of the project 8 | # ***************************** 9 | 10 | SOURCES = ["../cpp_utils/cloud/cloud.cpp", 11 | "neighbors/neighbors.cpp", 12 | "wrapper.cpp"] 13 | 14 | module = Extension(name="radius_neighbors", 15 | sources=SOURCES, 16 | extra_compile_args=['-std=c++11', 17 | '-D_GLIBCXX_USE_CXX11_ABI=0']) 18 | 19 | 20 | setup(ext_modules=[module], include_dirs=numpy.distutils.misc_util.get_numpy_include_dirs()) 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | -------------------------------------------------------------------------------- /cpp_wrappers/cpp_neighbors/wrapper.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include "neighbors/neighbors.h" 4 | #include 5 | 6 | 7 | 8 | // docstrings for our module 9 | // ************************* 10 | 11 | static char module_docstring[] = "This module provides two methods to compute radius neighbors from pointclouds or batch of pointclouds"; 12 | 13 | static char batch_query_docstring[] = "Method to get radius neighbors in a batch of stacked pointclouds"; 14 | 15 | 16 | // Declare the functions 17 | // ********************* 18 | 19 | static PyObject *batch_neighbors(PyObject *self, PyObject *args, PyObject *keywds); 20 | 21 | 22 | // Specify the members of the module 23 | // ********************************* 24 | 25 | static PyMethodDef module_methods[] = 26 | { 27 | { "batch_query", (PyCFunction)batch_neighbors, METH_VARARGS | METH_KEYWORDS, batch_query_docstring }, 28 | {NULL, NULL, 0, NULL} 29 | }; 30 | 31 | 32 | // Initialize the module 33 | // ********************* 34 | 35 | static struct PyModuleDef moduledef = 36 | { 37 | PyModuleDef_HEAD_INIT, 38 | "radius_neighbors", // m_name 39 | module_docstring, // m_doc 40 | -1, // m_size 41 | module_methods, // m_methods 42 | NULL, // m_reload 43 | NULL, // m_traverse 44 | NULL, // m_clear 45 | NULL, // m_free 46 | }; 47 | 48 | PyMODINIT_FUNC PyInit_radius_neighbors(void) 49 | { 50 | import_array(); 51 | return PyModule_Create(&moduledef); 52 | } 53 | 54 | 55 | // Definition of the batch_subsample method 56 | // ********************************** 57 | 58 | static PyObject* batch_neighbors(PyObject* self, PyObject* args, PyObject* keywds) 59 | { 60 | 61 | // Manage inputs 62 | // ************* 63 | 64 | // Args containers 65 | PyObject* queries_obj = NULL; 66 | PyObject* supports_obj = NULL; 67 | PyObject* q_batches_obj = NULL; 68 | PyObject* s_batches_obj = NULL; 69 | 70 | // Keywords containers 71 | static char* kwlist[] = { "queries", "supports", "q_batches", "s_batches", "radius", NULL }; 72 | float radius = 0.1; 73 | 74 | // Parse the input 75 | if (!PyArg_ParseTupleAndKeywords(args, keywds, "OOOO|$f", kwlist, &queries_obj, &supports_obj, &q_batches_obj, &s_batches_obj, &radius)) 76 | { 77 | PyErr_SetString(PyExc_RuntimeError, "Error parsing arguments"); 78 | return NULL; 79 | } 80 | 81 | 82 | // Interpret the input objects as numpy arrays. 83 | PyObject* queries_array = PyArray_FROM_OTF(queries_obj, NPY_FLOAT, NPY_IN_ARRAY); 84 | PyObject* supports_array = PyArray_FROM_OTF(supports_obj, NPY_FLOAT, NPY_IN_ARRAY); 85 | PyObject* q_batches_array = PyArray_FROM_OTF(q_batches_obj, NPY_INT, NPY_IN_ARRAY); 86 | PyObject* s_batches_array = PyArray_FROM_OTF(s_batches_obj, NPY_INT, NPY_IN_ARRAY); 87 | 88 | // Verify data was load correctly. 89 | if (queries_array == NULL) 90 | { 91 | Py_XDECREF(queries_array); 92 | Py_XDECREF(supports_array); 93 | Py_XDECREF(q_batches_array); 94 | Py_XDECREF(s_batches_array); 95 | PyErr_SetString(PyExc_RuntimeError, "Error converting query points to numpy arrays of type float32"); 96 | return NULL; 97 | } 98 | if (supports_array == NULL) 99 | { 100 | Py_XDECREF(queries_array); 101 | Py_XDECREF(supports_array); 102 | Py_XDECREF(q_batches_array); 103 | Py_XDECREF(s_batches_array); 104 | PyErr_SetString(PyExc_RuntimeError, "Error converting support points to numpy arrays of type float32"); 105 | return NULL; 106 | } 107 | if (q_batches_array == NULL) 108 | { 109 | Py_XDECREF(queries_array); 110 | Py_XDECREF(supports_array); 111 | Py_XDECREF(q_batches_array); 112 | Py_XDECREF(s_batches_array); 113 | PyErr_SetString(PyExc_RuntimeError, "Error converting query batches to numpy arrays of type int32"); 114 | return NULL; 115 | } 116 | if (s_batches_array == NULL) 117 | { 118 | Py_XDECREF(queries_array); 119 | Py_XDECREF(supports_array); 120 | Py_XDECREF(q_batches_array); 121 | Py_XDECREF(s_batches_array); 122 | PyErr_SetString(PyExc_RuntimeError, "Error converting support batches to numpy arrays of type int32"); 123 | return NULL; 124 | } 125 | 126 | // Check that the input array respect the dims 127 | if ((int)PyArray_NDIM(queries_array) != 2 || (int)PyArray_DIM(queries_array, 1) != 3) 128 | { 129 | Py_XDECREF(queries_array); 130 | Py_XDECREF(supports_array); 131 | Py_XDECREF(q_batches_array); 132 | Py_XDECREF(s_batches_array); 133 | PyErr_SetString(PyExc_RuntimeError, "Wrong dimensions : query.shape is not (N, 3)"); 134 | return NULL; 135 | } 136 | if ((int)PyArray_NDIM(supports_array) != 2 || (int)PyArray_DIM(supports_array, 1) != 3) 137 | { 138 | Py_XDECREF(queries_array); 139 | Py_XDECREF(supports_array); 140 | Py_XDECREF(q_batches_array); 141 | Py_XDECREF(s_batches_array); 142 | PyErr_SetString(PyExc_RuntimeError, "Wrong dimensions : support.shape is not (N, 3)"); 143 | return NULL; 144 | } 145 | if ((int)PyArray_NDIM(q_batches_array) > 1) 146 | { 147 | Py_XDECREF(queries_array); 148 | Py_XDECREF(supports_array); 149 | Py_XDECREF(q_batches_array); 150 | Py_XDECREF(s_batches_array); 151 | PyErr_SetString(PyExc_RuntimeError, "Wrong dimensions : queries_batches.shape is not (B,) "); 152 | return NULL; 153 | } 154 | if ((int)PyArray_NDIM(s_batches_array) > 1) 155 | { 156 | Py_XDECREF(queries_array); 157 | Py_XDECREF(supports_array); 158 | Py_XDECREF(q_batches_array); 159 | Py_XDECREF(s_batches_array); 160 | PyErr_SetString(PyExc_RuntimeError, "Wrong dimensions : supports_batches.shape is not (B,) "); 161 | return NULL; 162 | } 163 | if ((int)PyArray_DIM(q_batches_array, 0) != (int)PyArray_DIM(s_batches_array, 0)) 164 | { 165 | Py_XDECREF(queries_array); 166 | Py_XDECREF(supports_array); 167 | Py_XDECREF(q_batches_array); 168 | Py_XDECREF(s_batches_array); 169 | PyErr_SetString(PyExc_RuntimeError, "Wrong number of batch elements: different for queries and supports "); 170 | return NULL; 171 | } 172 | 173 | // Number of points 174 | int Nq = (int)PyArray_DIM(queries_array, 0); 175 | int Ns= (int)PyArray_DIM(supports_array, 0); 176 | 177 | // Number of batches 178 | int Nb = (int)PyArray_DIM(q_batches_array, 0); 179 | 180 | // Call the C++ function 181 | // ********************* 182 | 183 | // Convert PyArray to Cloud C++ class 184 | vector queries; 185 | vector supports; 186 | vector q_batches; 187 | vector s_batches; 188 | queries = vector((PointXYZ*)PyArray_DATA(queries_array), (PointXYZ*)PyArray_DATA(queries_array) + Nq); 189 | supports = vector((PointXYZ*)PyArray_DATA(supports_array), (PointXYZ*)PyArray_DATA(supports_array) + Ns); 190 | q_batches = vector((int*)PyArray_DATA(q_batches_array), (int*)PyArray_DATA(q_batches_array) + Nb); 191 | s_batches = vector((int*)PyArray_DATA(s_batches_array), (int*)PyArray_DATA(s_batches_array) + Nb); 192 | 193 | // Create result containers 194 | vector neighbors_indices; 195 | 196 | // Compute results 197 | //batch_ordered_neighbors(queries, supports, q_batches, s_batches, neighbors_indices, radius); 198 | batch_nanoflann_neighbors(queries, supports, q_batches, s_batches, neighbors_indices, radius); 199 | 200 | // Check result 201 | if (neighbors_indices.size() < 1) 202 | { 203 | PyErr_SetString(PyExc_RuntimeError, "Error"); 204 | return NULL; 205 | } 206 | 207 | // Manage outputs 208 | // ************** 209 | 210 | // Maximal number of neighbors 211 | int max_neighbors = neighbors_indices.size() / Nq; 212 | 213 | // Dimension of output containers 214 | npy_intp* neighbors_dims = new npy_intp[2]; 215 | neighbors_dims[0] = Nq; 216 | neighbors_dims[1] = max_neighbors; 217 | 218 | // Create output array 219 | PyObject* res_obj = PyArray_SimpleNew(2, neighbors_dims, NPY_INT); 220 | PyObject* ret = NULL; 221 | 222 | // Fill output array with values 223 | size_t size_in_bytes = Nq * max_neighbors * sizeof(int); 224 | memcpy(PyArray_DATA(res_obj), neighbors_indices.data(), size_in_bytes); 225 | 226 | // Merge results 227 | ret = Py_BuildValue("N", res_obj); 228 | 229 | // Clean up 230 | // ******** 231 | 232 | Py_XDECREF(queries_array); 233 | Py_XDECREF(supports_array); 234 | Py_XDECREF(q_batches_array); 235 | Py_XDECREF(s_batches_array); 236 | 237 | return ret; 238 | } 239 | -------------------------------------------------------------------------------- /cpp_wrappers/cpp_subsampling/build.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | py setup.py build_ext --inplace 3 | 4 | 5 | pause -------------------------------------------------------------------------------- /cpp_wrappers/cpp_subsampling/grid_subsampling/grid_subsampling.cpp: -------------------------------------------------------------------------------- 1 | 2 | #include "grid_subsampling.h" 3 | 4 | 5 | void grid_subsampling(vector& original_points, 6 | vector& subsampled_points, 7 | vector& original_features, 8 | vector& subsampled_features, 9 | vector& original_classes, 10 | vector& subsampled_classes, 11 | float sampleDl, 12 | int verbose) { 13 | 14 | // Initialize variables 15 | // ****************** 16 | 17 | // Number of points in the cloud 18 | size_t N = original_points.size(); 19 | 20 | // Dimension of the features 21 | size_t fdim = original_features.size() / N; 22 | size_t ldim = original_classes.size() / N; 23 | 24 | // Limits of the cloud 25 | PointXYZ minCorner = min_point(original_points); 26 | PointXYZ maxCorner = max_point(original_points); 27 | PointXYZ originCorner = floor(minCorner * (1/sampleDl)) * sampleDl; 28 | 29 | // Dimensions of the grid 30 | size_t sampleNX = (size_t)floor((maxCorner.x - originCorner.x) / sampleDl) + 1; 31 | size_t sampleNY = (size_t)floor((maxCorner.y - originCorner.y) / sampleDl) + 1; 32 | //size_t sampleNZ = (size_t)floor((maxCorner.z - originCorner.z) / sampleDl) + 1; 33 | 34 | // Check if features and classes need to be processed 35 | bool use_feature = original_features.size() > 0; 36 | bool use_classes = original_classes.size() > 0; 37 | 38 | 39 | // Create the sampled map 40 | // ********************** 41 | 42 | // Verbose parameters 43 | int i = 0; 44 | int nDisp = N / 100; 45 | 46 | // Initialize variables 47 | size_t iX, iY, iZ, mapIdx; 48 | unordered_map data; 49 | 50 | for (auto& p : original_points) 51 | { 52 | // Position of point in sample map 53 | iX = (size_t)floor((p.x - originCorner.x) / sampleDl); 54 | iY = (size_t)floor((p.y - originCorner.y) / sampleDl); 55 | iZ = (size_t)floor((p.z - originCorner.z) / sampleDl); 56 | mapIdx = iX + sampleNX*iY + sampleNX*sampleNY*iZ; 57 | 58 | // If not already created, create key 59 | if (data.count(mapIdx) < 1) 60 | data.emplace(mapIdx, SampledData(fdim, ldim)); 61 | 62 | // Fill the sample map 63 | if (use_feature && use_classes) 64 | data[mapIdx].update_all(p, original_features.begin() + i * fdim, original_classes.begin() + i * ldim); 65 | else if (use_feature) 66 | data[mapIdx].update_features(p, original_features.begin() + i * fdim); 67 | else if (use_classes) 68 | data[mapIdx].update_classes(p, original_classes.begin() + i * ldim); 69 | else 70 | data[mapIdx].update_points(p); 71 | 72 | // Display 73 | i++; 74 | if (verbose > 1 && i%nDisp == 0) 75 | std::cout << "\rSampled Map : " << std::setw(3) << i / nDisp << "%"; 76 | 77 | } 78 | 79 | // Divide for barycentre and transfer to a vector 80 | subsampled_points.reserve(data.size()); 81 | if (use_feature) 82 | subsampled_features.reserve(data.size() * fdim); 83 | if (use_classes) 84 | subsampled_classes.reserve(data.size() * ldim); 85 | for (auto& v : data) 86 | { 87 | subsampled_points.push_back(v.second.point * (1.0 / v.second.count)); 88 | if (use_feature) 89 | { 90 | float count = (float)v.second.count; 91 | transform(v.second.features.begin(), 92 | v.second.features.end(), 93 | v.second.features.begin(), 94 | [count](float f) { return f / count;}); 95 | subsampled_features.insert(subsampled_features.end(),v.second.features.begin(),v.second.features.end()); 96 | } 97 | if (use_classes) 98 | { 99 | for (int i = 0; i < ldim; i++) 100 | subsampled_classes.push_back(max_element(v.second.labels[i].begin(), v.second.labels[i].end(), 101 | [](const pair&a, const pair&b){return a.second < b.second;})->first); 102 | } 103 | } 104 | 105 | return; 106 | } 107 | 108 | 109 | void batch_grid_subsampling(vector& original_points, 110 | vector& subsampled_points, 111 | vector& original_features, 112 | vector& subsampled_features, 113 | vector& original_classes, 114 | vector& subsampled_classes, 115 | vector& original_batches, 116 | vector& subsampled_batches, 117 | float sampleDl, 118 | int max_p) 119 | { 120 | // Initialize variables 121 | // ****************** 122 | 123 | int b = 0; 124 | int sum_b = 0; 125 | 126 | // Number of points in the cloud 127 | size_t N = original_points.size(); 128 | 129 | // Dimension of the features 130 | size_t fdim = original_features.size() / N; 131 | size_t ldim = original_classes.size() / N; 132 | 133 | // Handle max_p = 0 134 | if (max_p < 1) 135 | max_p = N; 136 | 137 | // Loop over batches 138 | // ***************** 139 | 140 | for (b = 0; b < original_batches.size(); b++) 141 | { 142 | 143 | // Extract batch points features and labels 144 | vector b_o_points = vector(original_points.begin () + sum_b, 145 | original_points.begin () + sum_b + original_batches[b]); 146 | 147 | vector b_o_features; 148 | if (original_features.size() > 0) 149 | { 150 | b_o_features = vector(original_features.begin () + sum_b * fdim, 151 | original_features.begin () + (sum_b + original_batches[b]) * fdim); 152 | } 153 | 154 | vector b_o_classes; 155 | if (original_classes.size() > 0) 156 | { 157 | b_o_classes = vector(original_classes.begin () + sum_b * ldim, 158 | original_classes.begin () + sum_b + original_batches[b] * ldim); 159 | } 160 | 161 | 162 | // Create result containers 163 | vector b_s_points; 164 | vector b_s_features; 165 | vector b_s_classes; 166 | 167 | // Compute subsampling on current batch 168 | grid_subsampling(b_o_points, 169 | b_s_points, 170 | b_o_features, 171 | b_s_features, 172 | b_o_classes, 173 | b_s_classes, 174 | sampleDl, 175 | 0); 176 | 177 | // Stack batches points features and labels 178 | // **************************************** 179 | 180 | // If too many points remove some 181 | if (b_s_points.size() <= max_p) 182 | { 183 | subsampled_points.insert(subsampled_points.end(), b_s_points.begin(), b_s_points.end()); 184 | 185 | if (original_features.size() > 0) 186 | subsampled_features.insert(subsampled_features.end(), b_s_features.begin(), b_s_features.end()); 187 | 188 | if (original_classes.size() > 0) 189 | subsampled_classes.insert(subsampled_classes.end(), b_s_classes.begin(), b_s_classes.end()); 190 | 191 | subsampled_batches.push_back(b_s_points.size()); 192 | } 193 | else 194 | { 195 | subsampled_points.insert(subsampled_points.end(), b_s_points.begin(), b_s_points.begin() + max_p); 196 | 197 | if (original_features.size() > 0) 198 | subsampled_features.insert(subsampled_features.end(), b_s_features.begin(), b_s_features.begin() + max_p * fdim); 199 | 200 | if (original_classes.size() > 0) 201 | subsampled_classes.insert(subsampled_classes.end(), b_s_classes.begin(), b_s_classes.begin() + max_p * ldim); 202 | 203 | subsampled_batches.push_back(max_p); 204 | } 205 | 206 | // Stack new batch lengths 207 | sum_b += original_batches[b]; 208 | } 209 | 210 | return; 211 | } 212 | -------------------------------------------------------------------------------- /cpp_wrappers/cpp_subsampling/grid_subsampling/grid_subsampling.h: -------------------------------------------------------------------------------- 1 | 2 | 3 | #include "../../cpp_utils/cloud/cloud.h" 4 | 5 | #include 6 | #include 7 | 8 | using namespace std; 9 | 10 | class SampledData 11 | { 12 | public: 13 | 14 | // Elements 15 | // ******** 16 | 17 | int count; 18 | PointXYZ point; 19 | vector features; 20 | vector> labels; 21 | 22 | 23 | // Methods 24 | // ******* 25 | 26 | // Constructor 27 | SampledData() 28 | { 29 | count = 0; 30 | point = PointXYZ(); 31 | } 32 | 33 | SampledData(const size_t fdim, const size_t ldim) 34 | { 35 | count = 0; 36 | point = PointXYZ(); 37 | features = vector(fdim); 38 | labels = vector>(ldim); 39 | } 40 | 41 | // Method Update 42 | void update_all(const PointXYZ p, vector::iterator f_begin, vector::iterator l_begin) 43 | { 44 | count += 1; 45 | point += p; 46 | transform (features.begin(), features.end(), f_begin, features.begin(), plus()); 47 | int i = 0; 48 | for(vector::iterator it = l_begin; it != l_begin + labels.size(); ++it) 49 | { 50 | labels[i][*it] += 1; 51 | i++; 52 | } 53 | return; 54 | } 55 | void update_features(const PointXYZ p, vector::iterator f_begin) 56 | { 57 | count += 1; 58 | point += p; 59 | transform (features.begin(), features.end(), f_begin, features.begin(), plus()); 60 | return; 61 | } 62 | void update_classes(const PointXYZ p, vector::iterator l_begin) 63 | { 64 | count += 1; 65 | point += p; 66 | int i = 0; 67 | for(vector::iterator it = l_begin; it != l_begin + labels.size(); ++it) 68 | { 69 | labels[i][*it] += 1; 70 | i++; 71 | } 72 | return; 73 | } 74 | void update_points(const PointXYZ p) 75 | { 76 | count += 1; 77 | point += p; 78 | return; 79 | } 80 | }; 81 | 82 | void grid_subsampling(vector& original_points, 83 | vector& subsampled_points, 84 | vector& original_features, 85 | vector& subsampled_features, 86 | vector& original_classes, 87 | vector& subsampled_classes, 88 | float sampleDl, 89 | int verbose); 90 | 91 | void batch_grid_subsampling(vector& original_points, 92 | vector& subsampled_points, 93 | vector& original_features, 94 | vector& subsampled_features, 95 | vector& original_classes, 96 | vector& subsampled_classes, 97 | vector& original_batches, 98 | vector& subsampled_batches, 99 | float sampleDl, 100 | int max_p); 101 | 102 | -------------------------------------------------------------------------------- /cpp_wrappers/cpp_subsampling/setup.py: -------------------------------------------------------------------------------- 1 | from distutils.core import setup, Extension 2 | import numpy.distutils.misc_util 3 | 4 | # Adding OpenCV to project 5 | # ************************ 6 | 7 | # Adding sources of the project 8 | # ***************************** 9 | 10 | SOURCES = ["../cpp_utils/cloud/cloud.cpp", 11 | "grid_subsampling/grid_subsampling.cpp", 12 | "wrapper.cpp"] 13 | 14 | module = Extension(name="grid_subsampling", 15 | sources=SOURCES, 16 | extra_compile_args=['-std=c++11', 17 | '-D_GLIBCXX_USE_CXX11_ABI=0']) 18 | 19 | 20 | setup(ext_modules=[module], include_dirs=numpy.distutils.misc_util.get_numpy_include_dirs()) 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | -------------------------------------------------------------------------------- /cpp_wrappers/cpp_utils/cloud/cloud.cpp: -------------------------------------------------------------------------------- 1 | // 2 | // 3 | // 0==========================0 4 | // | Local feature test | 5 | // 0==========================0 6 | // 7 | // version 1.0 : 8 | // > 9 | // 10 | //--------------------------------------------------- 11 | // 12 | // Cloud source : 13 | // Define usefull Functions/Methods 14 | // 15 | //---------------------------------------------------- 16 | // 17 | // Hugues THOMAS - 10/02/2017 18 | // 19 | 20 | 21 | #include "cloud.h" 22 | 23 | 24 | // Getters 25 | // ******* 26 | 27 | PointXYZ max_point(std::vector points) 28 | { 29 | // Initialize limits 30 | PointXYZ maxP(points[0]); 31 | 32 | // Loop over all points 33 | for (auto p : points) 34 | { 35 | if (p.x > maxP.x) 36 | maxP.x = p.x; 37 | 38 | if (p.y > maxP.y) 39 | maxP.y = p.y; 40 | 41 | if (p.z > maxP.z) 42 | maxP.z = p.z; 43 | } 44 | 45 | return maxP; 46 | } 47 | 48 | PointXYZ min_point(std::vector points) 49 | { 50 | // Initialize limits 51 | PointXYZ minP(points[0]); 52 | 53 | // Loop over all points 54 | for (auto p : points) 55 | { 56 | if (p.x < minP.x) 57 | minP.x = p.x; 58 | 59 | if (p.y < minP.y) 60 | minP.y = p.y; 61 | 62 | if (p.z < minP.z) 63 | minP.z = p.z; 64 | } 65 | 66 | return minP; 67 | } -------------------------------------------------------------------------------- /cpp_wrappers/cpp_utils/cloud/cloud.h: -------------------------------------------------------------------------------- 1 | // 2 | // 3 | // 0==========================0 4 | // | Local feature test | 5 | // 0==========================0 6 | // 7 | // version 1.0 : 8 | // > 9 | // 10 | //--------------------------------------------------- 11 | // 12 | // Cloud header 13 | // 14 | //---------------------------------------------------- 15 | // 16 | // Hugues THOMAS - 10/02/2017 17 | // 18 | 19 | 20 | # pragma once 21 | 22 | #include 23 | #include 24 | #include 25 | #include 26 | #include 27 | #include 28 | #include 29 | #include 30 | 31 | #include 32 | 33 | 34 | 35 | 36 | // Point class 37 | // *********** 38 | 39 | 40 | class PointXYZ 41 | { 42 | public: 43 | 44 | // Elements 45 | // ******** 46 | 47 | float x, y, z; 48 | 49 | 50 | // Methods 51 | // ******* 52 | 53 | // Constructor 54 | PointXYZ() { x = 0; y = 0; z = 0; } 55 | PointXYZ(float x0, float y0, float z0) { x = x0; y = y0; z = z0; } 56 | 57 | // array type accessor 58 | float operator [] (int i) const 59 | { 60 | if (i == 0) return x; 61 | else if (i == 1) return y; 62 | else return z; 63 | } 64 | 65 | // opperations 66 | float dot(const PointXYZ P) const 67 | { 68 | return x * P.x + y * P.y + z * P.z; 69 | } 70 | 71 | float sq_norm() 72 | { 73 | return x*x + y*y + z*z; 74 | } 75 | 76 | PointXYZ cross(const PointXYZ P) const 77 | { 78 | return PointXYZ(y*P.z - z*P.y, z*P.x - x*P.z, x*P.y - y*P.x); 79 | } 80 | 81 | PointXYZ& operator+=(const PointXYZ& P) 82 | { 83 | x += P.x; 84 | y += P.y; 85 | z += P.z; 86 | return *this; 87 | } 88 | 89 | PointXYZ& operator-=(const PointXYZ& P) 90 | { 91 | x -= P.x; 92 | y -= P.y; 93 | z -= P.z; 94 | return *this; 95 | } 96 | 97 | PointXYZ& operator*=(const float& a) 98 | { 99 | x *= a; 100 | y *= a; 101 | z *= a; 102 | return *this; 103 | } 104 | }; 105 | 106 | 107 | // Point Opperations 108 | // ***************** 109 | 110 | inline PointXYZ operator + (const PointXYZ A, const PointXYZ B) 111 | { 112 | return PointXYZ(A.x + B.x, A.y + B.y, A.z + B.z); 113 | } 114 | 115 | inline PointXYZ operator - (const PointXYZ A, const PointXYZ B) 116 | { 117 | return PointXYZ(A.x - B.x, A.y - B.y, A.z - B.z); 118 | } 119 | 120 | inline PointXYZ operator * (const PointXYZ P, const float a) 121 | { 122 | return PointXYZ(P.x * a, P.y * a, P.z * a); 123 | } 124 | 125 | inline PointXYZ operator * (const float a, const PointXYZ P) 126 | { 127 | return PointXYZ(P.x * a, P.y * a, P.z * a); 128 | } 129 | 130 | inline std::ostream& operator << (std::ostream& os, const PointXYZ P) 131 | { 132 | return os << "[" << P.x << ", " << P.y << ", " << P.z << "]"; 133 | } 134 | 135 | inline bool operator == (const PointXYZ A, const PointXYZ B) 136 | { 137 | return A.x == B.x && A.y == B.y && A.z == B.z; 138 | } 139 | 140 | inline PointXYZ floor(const PointXYZ P) 141 | { 142 | return PointXYZ(std::floor(P.x), std::floor(P.y), std::floor(P.z)); 143 | } 144 | 145 | 146 | PointXYZ max_point(std::vector points); 147 | PointXYZ min_point(std::vector points); 148 | 149 | 150 | struct PointCloud 151 | { 152 | 153 | std::vector pts; 154 | 155 | // Must return the number of data points 156 | inline size_t kdtree_get_point_count() const { return pts.size(); } 157 | 158 | // Returns the dim'th component of the idx'th point in the class: 159 | // Since this is inlined and the "dim" argument is typically an immediate value, the 160 | // "if/else's" are actually solved at compile time. 161 | inline float kdtree_get_pt(const size_t idx, const size_t dim) const 162 | { 163 | if (dim == 0) return pts[idx].x; 164 | else if (dim == 1) return pts[idx].y; 165 | else return pts[idx].z; 166 | } 167 | 168 | // Optional bounding-box computation: return false to default to a standard bbox computation loop. 169 | // Return true if the BBOX was already computed by the class and returned in "bb" so it can be avoided to redo it again. 170 | // Look at bb.size() to find out the expected dimensionality (e.g. 2 or 3 for point clouds) 171 | template 172 | bool kdtree_get_bbox(BBOX& /* bb */) const { return false; } 173 | 174 | }; 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | -------------------------------------------------------------------------------- /doc/Github_intro.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/HuguesTHOMAS/KPConv-PyTorch/bf19211adf52098a90617ef21a677745d9775b10/doc/Github_intro.png -------------------------------------------------------------------------------- /doc/object_classification_guide.md: -------------------------------------------------------------------------------- 1 | 2 | ## Object classification on ModelNet40 3 | 4 | ### Data 5 | 6 | We consider our experiment folder is located at `XXXX/Experiments/KPConv-PyTorch`. And we use a common Data folder 7 | loacated at `XXXX/Data`. Therefore the relative path to the Data folder is `../../Data`. 8 | 9 | Regularly sampled clouds from ModelNet40 dataset can be downloaded 10 | here (1.6 GB). 11 | Uncompress the data and move it inside the folder `../../Data/ModelNet40`. 12 | 13 | N.B. If you want to place your data anywhere else, you just have to change the variable 14 | `self.path` of `ModelNet40Dataset` class ([here](https://github.com/HuguesTHOMAS/KPConv-PyTorch/blob/e9d328135c0a3818ee0cf1bb5bb63434ce15c22e/datasets/ModelNet40.py#L113)). 15 | 16 | 17 | ### Training a model 18 | 19 | Simply run the following script to start the training: 20 | 21 | python3 training_ModelNet40.py 22 | 23 | This file contains a configuration subclass `ModelNet40Config`, inherited from the general configuration class `Config` defined in `utils/config.py`. The value of every parameter can be modified in the subclass. The first run of this script will precompute structures for the dataset which might take some time. 24 | 25 | ### Plot a logged training 26 | 27 | When you start a new training, it is saved in a `results` folder. A dated log folder will be created, containing many information including loss values, validation metrics, model checkpoints, etc. 28 | 29 | In `plot_convergence.py`, you will find detailed comments explaining how to choose which training log you want to plot. Follow them and then run the script : 30 | 31 | python3 plot_convergence.py 32 | 33 | 34 | ### Test the trained model 35 | 36 | The test script is the same for all models (segmentation or classification). In `test_any_model.py`, you will find detailed comments explaining how to choose which logged trained model you want to test. Follow them and then run the script : 37 | 38 | python3 test_any_model.py 39 | -------------------------------------------------------------------------------- /doc/pretrained_models_guide.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## S3DIS Pretrained Models 4 | 5 | ### Models 6 | 7 | We provide pretrained weights for S3DIS dataset. The raw weights come with a parameter file describing the architecture and network hyperparameters. The code can thus load the network automatically. 8 | 9 | 10 | | Name (link) | KPConv Type | Description | Score | 11 | |:-------------|:-------------:|:-----|:-----:| 12 | | [Light_KPFCNN](https://drive.google.com/file/d/14sz0hdObzsf_exxInXdOIbnUTe0foOOz/view?usp=sharing) | rigid | A network with small `in_radius` for light GPU consumption (~8GB) | 65.4% | 13 | | [Heavy_KPFCNN](https://drive.google.com/file/d/1ySQq3SRBgk2Vt5Bvj-0N7jDPi0QTPZiZ/view?usp=sharing) | rigid | A network with better performances but needing bigger GPU (>18GB). | 66.4% | 14 | | [Deform_KPFCNN](https://drive.google.com/file/d/1ObGr2Srfj0f7Bd3bBbuQzxtjf0ULbpSA/view?usp=sharing) | deform | Deformable convolution network needing big GPU (>20GB). | 67.3% | 15 | | [Deform_Light_KPFCNN](https://drive.google.com/file/d/1gZfv6q6lUT9STFh7Fk4qVa5IVTgwmWIr/view?usp=sharing) | deform | Lighter version of the deformable architecture (~8GB). | 66.7% | 16 | 17 | 18 | 19 | ### Instructions 20 | 21 | 1. Unzip and place the folder in your 'results' folder. 22 | 23 | 2. In the test script `test_any_model.py`, set the variable `chosen_log` to the path were you placed the folder. 24 | 25 | 3. Run the test script 26 | 27 | python3 test_any_model.py 28 | 29 | 4. You will see the performance (on the subsampled input clouds) increase as the test goes on. 30 | 31 | Confusion on sub clouds 32 | 65.08 | 92.11 98.40 81.83 0.00 18.71 55.41 68.65 90.93 79.79 74.83 65.31 63.41 56.62 33 | 34 | 35 | 5. After a few minutes, the script will reproject the results form the subsampled input clouds to the real data and get you the real score 36 | 37 | Reproject Vote #9 38 | Done in 2.6 s 39 | 40 | Confusion on full clouds 41 | Done in 2.1 s 42 | 43 | -------------------------------------------------------------------------------------- 44 | 65.38 | 92.62 98.39 81.77 0.00 18.87 57.80 67.93 91.52 80.27 74.24 66.14 64.01 56.42 45 | -------------------------------------------------------------------------------------- 46 | 47 | 6. The test script creates a folder `test/name-of-your-log`, where it saves the predictions, potentials, and probabilities per class. You can load them with CloudCompare for visualization. 48 | 49 | -------------------------------------------------------------------------------- /doc/scene_segmentation_guide.md: -------------------------------------------------------------------------------- 1 | 2 | ## Scene Segmentation on S3DIS 3 | 4 | ### Data 5 | 6 | We consider our experiment folder is located at `XXXX/Experiments/KPConv-PyTorch`. And we use a common Data folder 7 | loacated at `XXXX/Data`. Therefore the relative path to the Data folder is `../../Data`. 8 | 9 | S3DIS dataset can be downloaded here (4.8 GB). 10 | Download the file named `Stanford3dDataset_v1.2.zip`, uncompress the data and move it to `../../Data/S3DIS`. 11 | 12 | N.B. If you want to place your data anywhere else, you just have to change the variable 13 | `self.path` of `S3DISDataset` class ([here](https://github.com/HuguesTHOMAS/KPConv-PyTorch/blob/afa18c92f00c6ed771b61cb08b285d2f93446ea4/datasets/S3DIS.py#L88)). 14 | 15 | ### Training 16 | 17 | Simply run the following script to start the training: 18 | 19 | python3 training_S3DIS.py 20 | 21 | Similarly to ModelNet40 training, the parameters can be modified in a configuration subclass called `S3DISConfig`, and the first run of this script might take some time to precompute dataset structures. 22 | 23 | 24 | ### Plot a logged training 25 | 26 | When you start a new training, it is saved in a `results` folder. A dated log folder will be created, containing many information including loss values, validation metrics, model checkpoints, etc. 27 | 28 | In `plot_convergence.py`, you will find detailed comments explaining how to choose which training log you want to plot. Follow them and then run the script : 29 | 30 | python3 plot_convergence.py 31 | 32 | 33 | ### Test the trained model 34 | 35 | The test script is the same for all models (segmentation or classification). In `test_any_model.py`, you will find detailed comments explaining how to choose which logged trained model you want to test. Follow them and then run the script : 36 | 37 | python3 test_any_model.py 38 | -------------------------------------------------------------------------------- /doc/slam_segmentation_guide.md: -------------------------------------------------------------------------------- 1 | 2 | ## Scene Segmentation on SemanticKitti 3 | 4 | ### Data 5 | 6 | We consider our experiment folder is located at `XXXX/Experiments/KPConv-PyTorch`. And we use a common Data folder 7 | loacated at `XXXX/Data`. Therefore the relative path to the Data folder is `../../Data`. 8 | 9 | SemanticKitti dataset can be downloaded here (80 GB). 10 | Download the three file named: 11 | * [`data_odometry_velodyne.zip` (80 GB)](http://www.cvlibs.net/download.php?file=data_odometry_velodyne.zip) 12 | * [`data_odometry_calib.zip` (1 MB)](http://www.cvlibs.net/download.php?file=data_odometry_calib.zip) 13 | * [`data_odometry_labels.zip` (179 MB)](http://semantic-kitti.org/assets/data_odometry_labels.zip) 14 | 15 | uncompress the data and move it to `../../Data/SemanticKitti`. 16 | 17 | You also need to download the files 18 | [`semantic-kitti-all.yaml`](https://github.com/PRBonn/semantic-kitti-api/blob/master/config/semantic-kitti-all.yaml) 19 | and 20 | [`semantic-kitti.yaml`](https://github.com/PRBonn/semantic-kitti-api/blob/master/config/semantic-kitti.yaml). 21 | Place them in your `../../Data/SemanticKitti` folder. 22 | 23 | N.B. If you want to place your data anywhere else, you just have to change the variable 24 | `self.path` of `SemanticKittiDataset` class ([here](https://github.com/HuguesTHOMAS/KPConv-PyTorch/blob/c32e6ce94ed34a3dd9584f98d8dc0be02535dfb4/datasets/SemanticKitti.py#L65)). 25 | 26 | ### Training 27 | 28 | Simply run the following script to start the training: 29 | 30 | python3 training_SemanticKitti.py 31 | 32 | Similarly to ModelNet40 training, the parameters can be modified in a configuration subclass called `SemanticKittiConfig`, and the first run of this script might take some time to precompute dataset structures. 33 | 34 | 35 | ### Plot a logged training 36 | 37 | When you start a new training, it is saved in a `results` folder. A dated log folder will be created, containing many information including loss values, validation metrics, model checkpoints, etc. 38 | 39 | In `plot_convergence.py`, you will find detailed comments explaining how to choose which training log you want to plot. Follow them and then run the script : 40 | 41 | python3 plot_convergence.py 42 | 43 | 44 | ### Test the trained model 45 | 46 | The test script is the same for all models (segmentation or classification). In `test_any_model.py`, you will find detailed comments explaining how to choose which logged trained model you want to test. Follow them and then run the script : 47 | 48 | python3 test_any_model.py 49 | -------------------------------------------------------------------------------- /doc/visualization_guide.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Visualize kernel deformations 4 | 5 | ### Intructions 6 | 7 | In order to visualize features you need a dataset and a pretrained model that uses deformable KPConv. 8 | 9 | To start this visualization run the script: 10 | 11 | python3 visualize_deformations.py 12 | 13 | ### Details 14 | 15 | The visualization script runs the model on a batch of test examples (forward pass), and then show these 16 | examples in an interactive window. Here is a list of all keyboard shortcuts: 17 | 18 | - 'b' / 'n': smaller or larger point size. 19 | - 'g' / 'h': previous or next example in current batch. 20 | - 'k': switch between the rigid kernel (original kernel points positions) and the deformed kernel (position of the 21 | kernel points after shift are applied) 22 | - 'z': Switch between the points displayed (input points, current layer points or both). 23 | - '0': Saves the example and deformed kernel as ply files. 24 | - mouse left click: select a point and show kernel at its location. 25 | - exit window: compute next batch. 26 | -------------------------------------------------------------------------------- /kernels/kernel_points.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Functions handling the disposition of kernel points. 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 11/06/2018 15 | # 16 | 17 | 18 | # ------------------------------------------------------------------------------------------ 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | 25 | # Import numpy package and name it "np" 26 | import time 27 | import numpy as np 28 | import matplotlib.pyplot as plt 29 | from matplotlib import cm 30 | from os import makedirs 31 | from os.path import join, exists 32 | 33 | from utils.ply import read_ply, write_ply 34 | from utils.config import bcolors 35 | 36 | 37 | # ------------------------------------------------------------------------------------------ 38 | # 39 | # Functions 40 | # \***************/ 41 | # 42 | # 43 | 44 | def create_3D_rotations(axis, angle): 45 | """ 46 | Create rotation matrices from a list of axes and angles. Code from wikipedia on quaternions 47 | :param axis: float32[N, 3] 48 | :param angle: float32[N,] 49 | :return: float32[N, 3, 3] 50 | """ 51 | 52 | t1 = np.cos(angle) 53 | t2 = 1 - t1 54 | t3 = axis[:, 0] * axis[:, 0] 55 | t6 = t2 * axis[:, 0] 56 | t7 = t6 * axis[:, 1] 57 | t8 = np.sin(angle) 58 | t9 = t8 * axis[:, 2] 59 | t11 = t6 * axis[:, 2] 60 | t12 = t8 * axis[:, 1] 61 | t15 = axis[:, 1] * axis[:, 1] 62 | t19 = t2 * axis[:, 1] * axis[:, 2] 63 | t20 = t8 * axis[:, 0] 64 | t24 = axis[:, 2] * axis[:, 2] 65 | R = np.stack([t1 + t2 * t3, 66 | t7 - t9, 67 | t11 + t12, 68 | t7 + t9, 69 | t1 + t2 * t15, 70 | t19 - t20, 71 | t11 - t12, 72 | t19 + t20, 73 | t1 + t2 * t24], axis=1) 74 | 75 | return np.reshape(R, (-1, 3, 3)) 76 | 77 | 78 | def spherical_Lloyd(radius, num_cells, dimension=3, fixed='center', approximation='monte-carlo', 79 | approx_n=5000, max_iter=500, momentum=0.9, verbose=0): 80 | """ 81 | Creation of kernel point via Lloyd algorithm. We use an approximation of the algorithm, and compute the Voronoi 82 | cell centers with discretization of space. The exact formula is not trivial with part of the sphere as sides. 83 | :param radius: Radius of the kernels 84 | :param num_cells: Number of cell (kernel points) in the Voronoi diagram. 85 | :param dimension: dimension of the space 86 | :param fixed: fix position of certain kernel points ('none', 'center' or 'verticals') 87 | :param approximation: Approximation method for Lloyd's algorithm ('discretization', 'monte-carlo') 88 | :param approx_n: Number of point used for approximation. 89 | :param max_iter: Maximum nu;ber of iteration for the algorithm. 90 | :param momentum: Momentum of the low pass filter smoothing kernel point positions 91 | :param verbose: display option 92 | :return: points [num_kernels, num_points, dimension] 93 | """ 94 | 95 | ####################### 96 | # Parameters definition 97 | ####################### 98 | 99 | # Radius used for optimization (points are rescaled afterwards) 100 | radius0 = 1.0 101 | 102 | ####################### 103 | # Kernel initialization 104 | ####################### 105 | 106 | # Random kernel points (Uniform distribution in a sphere) 107 | kernel_points = np.zeros((0, dimension)) 108 | while kernel_points.shape[0] < num_cells: 109 | new_points = np.random.rand(num_cells, dimension) * 2 * radius0 - radius0 110 | kernel_points = np.vstack((kernel_points, new_points)) 111 | d2 = np.sum(np.power(kernel_points, 2), axis=1) 112 | kernel_points = kernel_points[np.logical_and(d2 < radius0 ** 2, (0.9 * radius0) ** 2 < d2), :] 113 | kernel_points = kernel_points[:num_cells, :].reshape((num_cells, -1)) 114 | 115 | # Optional fixing 116 | if fixed == 'center': 117 | kernel_points[0, :] *= 0 118 | if fixed == 'verticals': 119 | kernel_points[:3, :] *= 0 120 | kernel_points[1, -1] += 2 * radius0 / 3 121 | kernel_points[2, -1] -= 2 * radius0 / 3 122 | 123 | ############################## 124 | # Approximation initialization 125 | ############################## 126 | 127 | # Initialize figure 128 | if verbose > 1: 129 | fig = plt.figure() 130 | 131 | # Initialize discretization in this method is chosen 132 | if approximation == 'discretization': 133 | side_n = int(np.floor(approx_n ** (1. / dimension))) 134 | dl = 2 * radius0 / side_n 135 | coords = np.arange(-radius0 + dl/2, radius0, dl) 136 | if dimension == 2: 137 | x, y = np.meshgrid(coords, coords) 138 | X = np.vstack((np.ravel(x), np.ravel(y))).T 139 | elif dimension == 3: 140 | x, y, z = np.meshgrid(coords, coords, coords) 141 | X = np.vstack((np.ravel(x), np.ravel(y), np.ravel(z))).T 142 | elif dimension == 4: 143 | x, y, z, t = np.meshgrid(coords, coords, coords, coords) 144 | X = np.vstack((np.ravel(x), np.ravel(y), np.ravel(z), np.ravel(t))).T 145 | else: 146 | raise ValueError('Unsupported dimension (max is 4)') 147 | elif approximation == 'monte-carlo': 148 | X = np.zeros((0, dimension)) 149 | else: 150 | raise ValueError('Wrong approximation method chosen: "{:s}"'.format(approximation)) 151 | 152 | # Only points inside the sphere are used 153 | d2 = np.sum(np.power(X, 2), axis=1) 154 | X = X[d2 < radius0 * radius0, :] 155 | 156 | ##################### 157 | # Kernel optimization 158 | ##################### 159 | 160 | # Warning if at least one kernel point has no cell 161 | warning = False 162 | 163 | # moving vectors of kernel points saved to detect convergence 164 | max_moves = np.zeros((0,)) 165 | 166 | for iter in range(max_iter): 167 | 168 | # In the case of monte-carlo, renew the sampled points 169 | if approximation == 'monte-carlo': 170 | X = np.random.rand(approx_n, dimension) * 2 * radius0 - radius0 171 | d2 = np.sum(np.power(X, 2), axis=1) 172 | X = X[d2 < radius0 * radius0, :] 173 | 174 | # Get the distances matrix [n_approx, K, dim] 175 | differences = np.expand_dims(X, 1) - kernel_points 176 | sq_distances = np.sum(np.square(differences), axis=2) 177 | 178 | # Compute cell centers 179 | cell_inds = np.argmin(sq_distances, axis=1) 180 | centers = [] 181 | for c in range(num_cells): 182 | bool_c = (cell_inds == c) 183 | num_c = np.sum(bool_c.astype(np.int32)) 184 | if num_c > 0: 185 | centers.append(np.sum(X[bool_c, :], axis=0) / num_c) 186 | else: 187 | warning = True 188 | centers.append(kernel_points[c]) 189 | 190 | # Update kernel points with low pass filter to smooth mote carlo 191 | centers = np.vstack(centers) 192 | moves = (1 - momentum) * (centers - kernel_points) 193 | kernel_points += moves 194 | 195 | # Check moves for convergence 196 | max_moves = np.append(max_moves, np.max(np.linalg.norm(moves, axis=1))) 197 | 198 | # Optional fixing 199 | if fixed == 'center': 200 | kernel_points[0, :] *= 0 201 | if fixed == 'verticals': 202 | kernel_points[0, :] *= 0 203 | kernel_points[:3, :-1] *= 0 204 | 205 | if verbose: 206 | print('iter {:5d} / max move = {:f}'.format(iter, np.max(np.linalg.norm(moves, axis=1)))) 207 | if warning: 208 | print('{:}WARNING: at least one point has no cell{:}'.format(bcolors.WARNING, bcolors.ENDC)) 209 | if verbose > 1: 210 | plt.clf() 211 | plt.scatter(X[:, 0], X[:, 1], c=cell_inds, s=20.0, 212 | marker='.', cmap=plt.get_cmap('tab20')) 213 | #plt.scatter(kernel_points[:, 0], kernel_points[:, 1], c=np.arange(num_cells), s=100.0, 214 | # marker='+', cmap=plt.get_cmap('tab20')) 215 | plt.plot(kernel_points[:, 0], kernel_points[:, 1], 'k+') 216 | circle = plt.Circle((0, 0), radius0, color='r', fill=False) 217 | fig.axes[0].add_artist(circle) 218 | fig.axes[0].set_xlim((-radius0 * 1.1, radius0 * 1.1)) 219 | fig.axes[0].set_ylim((-radius0 * 1.1, radius0 * 1.1)) 220 | fig.axes[0].set_aspect('equal') 221 | plt.draw() 222 | plt.pause(0.001) 223 | plt.show(block=False) 224 | 225 | ################### 226 | # User verification 227 | ################### 228 | 229 | # Show the convergence to ask user if this kernel is correct 230 | if verbose: 231 | if dimension == 2: 232 | fig, (ax1, ax2) = plt.subplots(1, 2, figsize=[10.4, 4.8]) 233 | ax1.plot(max_moves) 234 | ax2.scatter(X[:, 0], X[:, 1], c=cell_inds, s=20.0, 235 | marker='.', cmap=plt.get_cmap('tab20')) 236 | # plt.scatter(kernel_points[:, 0], kernel_points[:, 1], c=np.arange(num_cells), s=100.0, 237 | # marker='+', cmap=plt.get_cmap('tab20')) 238 | ax2.plot(kernel_points[:, 0], kernel_points[:, 1], 'k+') 239 | circle = plt.Circle((0, 0), radius0, color='r', fill=False) 240 | ax2.add_artist(circle) 241 | ax2.set_xlim((-radius0 * 1.1, radius0 * 1.1)) 242 | ax2.set_ylim((-radius0 * 1.1, radius0 * 1.1)) 243 | ax2.set_aspect('equal') 244 | plt.title('Check if kernel is correct.') 245 | plt.draw() 246 | plt.show() 247 | 248 | if dimension > 2: 249 | plt.figure() 250 | plt.plot(max_moves) 251 | plt.title('Check if kernel is correct.') 252 | plt.show() 253 | 254 | # Rescale kernels with real radius 255 | return kernel_points * radius 256 | 257 | 258 | def kernel_point_optimization_debug(radius, num_points, num_kernels=1, dimension=3, 259 | fixed='center', ratio=0.66, verbose=0): 260 | """ 261 | Creation of kernel point via optimization of potentials. 262 | :param radius: Radius of the kernels 263 | :param num_points: points composing kernels 264 | :param num_kernels: number of wanted kernels 265 | :param dimension: dimension of the space 266 | :param fixed: fix position of certain kernel points ('none', 'center' or 'verticals') 267 | :param ratio: ratio of the radius where you want the kernels points to be placed 268 | :param verbose: display option 269 | :return: points [num_kernels, num_points, dimension] 270 | """ 271 | 272 | ####################### 273 | # Parameters definition 274 | ####################### 275 | 276 | # Radius used for optimization (points are rescaled afterwards) 277 | radius0 = 1 278 | diameter0 = 2 279 | 280 | # Factor multiplicating gradients for moving points (~learning rate) 281 | moving_factor = 1e-2 282 | continuous_moving_decay = 0.9995 283 | 284 | # Gradient threshold to stop optimization 285 | thresh = 1e-5 286 | 287 | # Gradient clipping value 288 | clip = 0.05 * radius0 289 | 290 | ####################### 291 | # Kernel initialization 292 | ####################### 293 | 294 | # Random kernel points 295 | kernel_points = np.random.rand(num_kernels * num_points - 1, dimension) * diameter0 - radius0 296 | while (kernel_points.shape[0] < num_kernels * num_points): 297 | new_points = np.random.rand(num_kernels * num_points - 1, dimension) * diameter0 - radius0 298 | kernel_points = np.vstack((kernel_points, new_points)) 299 | d2 = np.sum(np.power(kernel_points, 2), axis=1) 300 | kernel_points = kernel_points[d2 < 0.5 * radius0 * radius0, :] 301 | kernel_points = kernel_points[:num_kernels * num_points, :].reshape((num_kernels, num_points, -1)) 302 | 303 | # Optionnal fixing 304 | if fixed == 'center': 305 | kernel_points[:, 0, :] *= 0 306 | if fixed == 'verticals': 307 | kernel_points[:, :3, :] *= 0 308 | kernel_points[:, 1, -1] += 2 * radius0 / 3 309 | kernel_points[:, 2, -1] -= 2 * radius0 / 3 310 | 311 | ##################### 312 | # Kernel optimization 313 | ##################### 314 | 315 | # Initialize figure 316 | if verbose>1: 317 | fig = plt.figure() 318 | 319 | saved_gradient_norms = np.zeros((10000, num_kernels)) 320 | old_gradient_norms = np.zeros((num_kernels, num_points)) 321 | step = -1 322 | while step < 10000: 323 | 324 | # Increment 325 | step += 1 326 | 327 | # Compute gradients 328 | # ***************** 329 | 330 | # Derivative of the sum of potentials of all points 331 | A = np.expand_dims(kernel_points, axis=2) 332 | B = np.expand_dims(kernel_points, axis=1) 333 | interd2 = np.sum(np.power(A - B, 2), axis=-1) 334 | inter_grads = (A - B) / (np.power(np.expand_dims(interd2, -1), 3/2) + 1e-6) 335 | inter_grads = np.sum(inter_grads, axis=1) 336 | 337 | # Derivative of the radius potential 338 | circle_grads = 10*kernel_points 339 | 340 | # All gradients 341 | gradients = inter_grads + circle_grads 342 | 343 | if fixed == 'verticals': 344 | gradients[:, 1:3, :-1] = 0 345 | 346 | # Stop condition 347 | # ************** 348 | 349 | # Compute norm of gradients 350 | gradients_norms = np.sqrt(np.sum(np.power(gradients, 2), axis=-1)) 351 | saved_gradient_norms[step, :] = np.max(gradients_norms, axis=1) 352 | 353 | # Stop if all moving points are gradients fixed (low gradients diff) 354 | 355 | if fixed == 'center' and np.max(np.abs(old_gradient_norms[:, 1:] - gradients_norms[:, 1:])) < thresh: 356 | break 357 | elif fixed == 'verticals' and np.max(np.abs(old_gradient_norms[:, 3:] - gradients_norms[:, 3:])) < thresh: 358 | break 359 | elif np.max(np.abs(old_gradient_norms - gradients_norms)) < thresh: 360 | break 361 | old_gradient_norms = gradients_norms 362 | 363 | # Move points 364 | # *********** 365 | 366 | # Clip gradient to get moving dists 367 | moving_dists = np.minimum(moving_factor * gradients_norms, clip) 368 | 369 | # Fix central point 370 | if fixed == 'center': 371 | moving_dists[:, 0] = 0 372 | if fixed == 'verticals': 373 | moving_dists[:, 0] = 0 374 | 375 | # Move points 376 | kernel_points -= np.expand_dims(moving_dists, -1) * gradients / np.expand_dims(gradients_norms + 1e-6, -1) 377 | 378 | if verbose: 379 | print('step {:5d} / max grad = {:f}'.format(step, np.max(gradients_norms[:, 3:]))) 380 | if verbose > 1: 381 | plt.clf() 382 | plt.plot(kernel_points[0, :, 0], kernel_points[0, :, 1], '.') 383 | circle = plt.Circle((0, 0), radius, color='r', fill=False) 384 | fig.axes[0].add_artist(circle) 385 | fig.axes[0].set_xlim((-radius*1.1, radius*1.1)) 386 | fig.axes[0].set_ylim((-radius*1.1, radius*1.1)) 387 | fig.axes[0].set_aspect('equal') 388 | plt.draw() 389 | plt.pause(0.001) 390 | plt.show(block=False) 391 | print(moving_factor) 392 | 393 | # moving factor decay 394 | moving_factor *= continuous_moving_decay 395 | 396 | # Remove unused lines in the saved gradients 397 | if step < 10000: 398 | saved_gradient_norms = saved_gradient_norms[:step+1, :] 399 | 400 | # Rescale radius to fit the wanted ratio of radius 401 | r = np.sqrt(np.sum(np.power(kernel_points, 2), axis=-1)) 402 | kernel_points *= ratio / np.mean(r[:, 1:]) 403 | 404 | # Rescale kernels with real radius 405 | return kernel_points * radius, saved_gradient_norms 406 | 407 | 408 | def load_kernels(radius, num_kpoints, dimension, fixed, lloyd=False): 409 | 410 | # Kernel directory 411 | kernel_dir = 'kernels/dispositions' 412 | if not exists(kernel_dir): 413 | makedirs(kernel_dir) 414 | 415 | # To many points switch to Lloyds 416 | if num_kpoints > 30: 417 | lloyd = True 418 | 419 | # Kernel_file 420 | kernel_file = join(kernel_dir, 'k_{:03d}_{:s}_{:d}D.ply'.format(num_kpoints, fixed, dimension)) 421 | 422 | # Check if already done 423 | if not exists(kernel_file): 424 | if lloyd: 425 | # Create kernels 426 | kernel_points = spherical_Lloyd(1.0, 427 | num_kpoints, 428 | dimension=dimension, 429 | fixed=fixed, 430 | verbose=0) 431 | 432 | else: 433 | # Create kernels 434 | kernel_points, grad_norms = kernel_point_optimization_debug(1.0, 435 | num_kpoints, 436 | num_kernels=100, 437 | dimension=dimension, 438 | fixed=fixed, 439 | verbose=0) 440 | 441 | # Find best candidate 442 | best_k = np.argmin(grad_norms[-1, :]) 443 | 444 | # Save points 445 | kernel_points = kernel_points[best_k, :, :] 446 | 447 | write_ply(kernel_file, kernel_points, ['x', 'y', 'z']) 448 | 449 | else: 450 | data = read_ply(kernel_file) 451 | kernel_points = np.vstack((data['x'], data['y'], data['z'])).T 452 | 453 | # Random roations for the kernel 454 | # N.B. 4D random rotations not supported yet 455 | R = np.eye(dimension) 456 | theta = np.random.rand() * 2 * np.pi 457 | if dimension == 2: 458 | if fixed != 'vertical': 459 | c, s = np.cos(theta), np.sin(theta) 460 | R = np.array([[c, -s], [s, c]], dtype=np.float32) 461 | 462 | elif dimension == 3: 463 | if fixed != 'vertical': 464 | c, s = np.cos(theta), np.sin(theta) 465 | R = np.array([[c, -s, 0], [s, c, 0], [0, 0, 1]], dtype=np.float32) 466 | 467 | else: 468 | phi = (np.random.rand() - 0.5) * np.pi 469 | 470 | # Create the first vector in carthesian coordinates 471 | u = np.array([np.cos(theta) * np.cos(phi), np.sin(theta) * np.cos(phi), np.sin(phi)]) 472 | 473 | # Choose a random rotation angle 474 | alpha = np.random.rand() * 2 * np.pi 475 | 476 | # Create the rotation matrix with this vector and angle 477 | R = create_3D_rotations(np.reshape(u, (1, -1)), np.reshape(alpha, (1, -1)))[0] 478 | 479 | R = R.astype(np.float32) 480 | 481 | # Add a small noise 482 | kernel_points = kernel_points + np.random.normal(scale=0.01, size=kernel_points.shape) 483 | 484 | # Scale kernels 485 | kernel_points = radius * kernel_points 486 | 487 | # Rotate kernels 488 | kernel_points = np.matmul(kernel_points, R) 489 | 490 | return kernel_points.astype(np.float32) -------------------------------------------------------------------------------- /models/architectures.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Define network architectures 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 06/03/2020 15 | # 16 | 17 | from models.blocks import * 18 | import numpy as np 19 | 20 | 21 | def p2p_fitting_regularizer(net): 22 | 23 | fitting_loss = 0 24 | repulsive_loss = 0 25 | 26 | for m in net.modules(): 27 | 28 | if isinstance(m, KPConv) and m.deformable: 29 | 30 | ############## 31 | # Fitting loss 32 | ############## 33 | 34 | # Get the distance to closest input point and normalize to be independant from layers 35 | KP_min_d2 = m.min_d2 / (m.KP_extent ** 2) 36 | 37 | # Loss will be the square distance to closest input point. We use L1 because dist is already squared 38 | fitting_loss += net.l1(KP_min_d2, torch.zeros_like(KP_min_d2)) 39 | 40 | ################ 41 | # Repulsive loss 42 | ################ 43 | 44 | # Normalized KP locations 45 | KP_locs = m.deformed_KP / m.KP_extent 46 | 47 | # Point should not be close to each other 48 | for i in range(net.K): 49 | other_KP = torch.cat([KP_locs[:, :i, :], KP_locs[:, i + 1:, :]], dim=1).detach() 50 | distances = torch.sqrt(torch.sum((other_KP - KP_locs[:, i:i + 1, :]) ** 2, dim=2)) 51 | rep_loss = torch.sum(torch.clamp_max(distances - net.repulse_extent, max=0.0) ** 2, dim=1) 52 | repulsive_loss += net.l1(rep_loss, torch.zeros_like(rep_loss)) / net.K 53 | 54 | return net.deform_fitting_power * (2 * fitting_loss + repulsive_loss) 55 | 56 | 57 | class KPCNN(nn.Module): 58 | """ 59 | Class defining KPCNN 60 | """ 61 | 62 | def __init__(self, config): 63 | super(KPCNN, self).__init__() 64 | 65 | ##################### 66 | # Network opperations 67 | ##################### 68 | 69 | # Current radius of convolution and feature dimension 70 | layer = 0 71 | r = config.first_subsampling_dl * config.conv_radius 72 | in_dim = config.in_features_dim 73 | out_dim = config.first_features_dim 74 | self.K = config.num_kernel_points 75 | 76 | # Save all block operations in a list of modules 77 | self.block_ops = nn.ModuleList() 78 | 79 | # Loop over consecutive blocks 80 | block_in_layer = 0 81 | for block_i, block in enumerate(config.architecture): 82 | 83 | # Check equivariance 84 | if ('equivariant' in block) and (not out_dim % 3 == 0): 85 | raise ValueError('Equivariant block but features dimension is not a factor of 3') 86 | 87 | # Detect upsampling block to stop 88 | if 'upsample' in block: 89 | break 90 | 91 | # Apply the good block function defining tf ops 92 | self.block_ops.append(block_decider(block, 93 | r, 94 | in_dim, 95 | out_dim, 96 | layer, 97 | config)) 98 | 99 | 100 | # Index of block in this layer 101 | block_in_layer += 1 102 | 103 | # Update dimension of input from output 104 | if 'simple' in block: 105 | in_dim = out_dim // 2 106 | else: 107 | in_dim = out_dim 108 | 109 | 110 | # Detect change to a subsampled layer 111 | if 'pool' in block or 'strided' in block: 112 | # Update radius and feature dimension for next layer 113 | layer += 1 114 | r *= 2 115 | out_dim *= 2 116 | block_in_layer = 0 117 | 118 | self.head_mlp = UnaryBlock(out_dim, 1024, False, 0) 119 | self.head_softmax = UnaryBlock(1024, config.num_classes, False, 0, no_relu=True) 120 | 121 | ################ 122 | # Network Losses 123 | ################ 124 | 125 | self.criterion = torch.nn.CrossEntropyLoss() 126 | self.deform_fitting_mode = config.deform_fitting_mode 127 | self.deform_fitting_power = config.deform_fitting_power 128 | self.deform_lr_factor = config.deform_lr_factor 129 | self.repulse_extent = config.repulse_extent 130 | self.output_loss = 0 131 | self.reg_loss = 0 132 | self.l1 = nn.L1Loss() 133 | 134 | return 135 | 136 | def forward(self, batch, config): 137 | 138 | # Save all block operations in a list of modules 139 | x = batch.features.clone().detach() 140 | 141 | # Loop over consecutive blocks 142 | for block_op in self.block_ops: 143 | x = block_op(x, batch) 144 | 145 | # Head of network 146 | x = self.head_mlp(x, batch) 147 | x = self.head_softmax(x, batch) 148 | 149 | return x 150 | 151 | def loss(self, outputs, labels): 152 | """ 153 | Runs the loss on outputs of the model 154 | :param outputs: logits 155 | :param labels: labels 156 | :return: loss 157 | """ 158 | 159 | # Cross entropy loss 160 | self.output_loss = self.criterion(outputs, labels) 161 | 162 | # Regularization of deformable offsets 163 | if self.deform_fitting_mode == 'point2point': 164 | self.reg_loss = p2p_fitting_regularizer(self) 165 | elif self.deform_fitting_mode == 'point2plane': 166 | raise ValueError('point2plane fitting mode not implemented yet.') 167 | else: 168 | raise ValueError('Unknown fitting mode: ' + self.deform_fitting_mode) 169 | 170 | # Combined loss 171 | return self.output_loss + self.reg_loss 172 | 173 | @staticmethod 174 | def accuracy(outputs, labels): 175 | """ 176 | Computes accuracy of the current batch 177 | :param outputs: logits predicted by the network 178 | :param labels: labels 179 | :return: accuracy value 180 | """ 181 | 182 | predicted = torch.argmax(outputs.data, dim=1) 183 | total = labels.size(0) 184 | correct = (predicted == labels).sum().item() 185 | 186 | return correct / total 187 | 188 | 189 | class KPFCNN(nn.Module): 190 | """ 191 | Class defining KPFCNN 192 | """ 193 | 194 | def __init__(self, config, lbl_values, ign_lbls): 195 | super(KPFCNN, self).__init__() 196 | 197 | ############ 198 | # Parameters 199 | ############ 200 | 201 | # Current radius of convolution and feature dimension 202 | layer = 0 203 | r = config.first_subsampling_dl * config.conv_radius 204 | in_dim = config.in_features_dim 205 | out_dim = config.first_features_dim 206 | self.K = config.num_kernel_points 207 | self.C = len(lbl_values) - len(ign_lbls) 208 | 209 | ##################### 210 | # List Encoder blocks 211 | ##################### 212 | 213 | # Save all block operations in a list of modules 214 | self.encoder_blocks = nn.ModuleList() 215 | self.encoder_skip_dims = [] 216 | self.encoder_skips = [] 217 | 218 | # Loop over consecutive blocks 219 | for block_i, block in enumerate(config.architecture): 220 | 221 | # Check equivariance 222 | if ('equivariant' in block) and (not out_dim % 3 == 0): 223 | raise ValueError('Equivariant block but features dimension is not a factor of 3') 224 | 225 | # Detect change to next layer for skip connection 226 | if np.any([tmp in block for tmp in ['pool', 'strided', 'upsample', 'global']]): 227 | self.encoder_skips.append(block_i) 228 | self.encoder_skip_dims.append(in_dim) 229 | 230 | # Detect upsampling block to stop 231 | if 'upsample' in block: 232 | break 233 | 234 | # Apply the good block function defining tf ops 235 | self.encoder_blocks.append(block_decider(block, 236 | r, 237 | in_dim, 238 | out_dim, 239 | layer, 240 | config)) 241 | 242 | # Update dimension of input from output 243 | if 'simple' in block: 244 | in_dim = out_dim // 2 245 | else: 246 | in_dim = out_dim 247 | 248 | # Detect change to a subsampled layer 249 | if 'pool' in block or 'strided' in block: 250 | # Update radius and feature dimension for next layer 251 | layer += 1 252 | r *= 2 253 | out_dim *= 2 254 | 255 | ##################### 256 | # List Decoder blocks 257 | ##################### 258 | 259 | # Save all block operations in a list of modules 260 | self.decoder_blocks = nn.ModuleList() 261 | self.decoder_concats = [] 262 | 263 | # Find first upsampling block 264 | start_i = 0 265 | for block_i, block in enumerate(config.architecture): 266 | if 'upsample' in block: 267 | start_i = block_i 268 | break 269 | 270 | # Loop over consecutive blocks 271 | for block_i, block in enumerate(config.architecture[start_i:]): 272 | 273 | # Add dimension of skip connection concat 274 | if block_i > 0 and 'upsample' in config.architecture[start_i + block_i - 1]: 275 | in_dim += self.encoder_skip_dims[layer] 276 | self.decoder_concats.append(block_i) 277 | 278 | # Apply the good block function defining tf ops 279 | self.decoder_blocks.append(block_decider(block, 280 | r, 281 | in_dim, 282 | out_dim, 283 | layer, 284 | config)) 285 | 286 | # Update dimension of input from output 287 | in_dim = out_dim 288 | 289 | # Detect change to a subsampled layer 290 | if 'upsample' in block: 291 | # Update radius and feature dimension for next layer 292 | layer -= 1 293 | r *= 0.5 294 | out_dim = out_dim // 2 295 | 296 | self.head_mlp = UnaryBlock(out_dim, config.first_features_dim, False, 0) 297 | self.head_softmax = UnaryBlock(config.first_features_dim, self.C, False, 0, no_relu=True) 298 | 299 | ################ 300 | # Network Losses 301 | ################ 302 | 303 | # List of valid labels (those not ignored in loss) 304 | self.valid_labels = np.sort([c for c in lbl_values if c not in ign_lbls]) 305 | 306 | # Choose segmentation loss 307 | if len(config.class_w) > 0: 308 | class_w = torch.from_numpy(np.array(config.class_w, dtype=np.float32)) 309 | self.criterion = torch.nn.CrossEntropyLoss(weight=class_w, ignore_index=-1) 310 | else: 311 | self.criterion = torch.nn.CrossEntropyLoss(ignore_index=-1) 312 | self.deform_fitting_mode = config.deform_fitting_mode 313 | self.deform_fitting_power = config.deform_fitting_power 314 | self.deform_lr_factor = config.deform_lr_factor 315 | self.repulse_extent = config.repulse_extent 316 | self.output_loss = 0 317 | self.reg_loss = 0 318 | self.l1 = nn.L1Loss() 319 | 320 | return 321 | 322 | def forward(self, batch, config): 323 | 324 | # Get input features 325 | x = batch.features.clone().detach() 326 | 327 | # Loop over consecutive blocks 328 | skip_x = [] 329 | for block_i, block_op in enumerate(self.encoder_blocks): 330 | if block_i in self.encoder_skips: 331 | skip_x.append(x) 332 | x = block_op(x, batch) 333 | 334 | for block_i, block_op in enumerate(self.decoder_blocks): 335 | if block_i in self.decoder_concats: 336 | x = torch.cat([x, skip_x.pop()], dim=1) 337 | x = block_op(x, batch) 338 | 339 | # Head of network 340 | x = self.head_mlp(x, batch) 341 | x = self.head_softmax(x, batch) 342 | 343 | return x 344 | 345 | def loss(self, outputs, labels): 346 | """ 347 | Runs the loss on outputs of the model 348 | :param outputs: logits 349 | :param labels: labels 350 | :return: loss 351 | """ 352 | 353 | # Set all ignored labels to -1 and correct the other label to be in [0, C-1] range 354 | target = - torch.ones_like(labels) 355 | for i, c in enumerate(self.valid_labels): 356 | target[labels == c] = i 357 | 358 | # Reshape to have a minibatch size of 1 359 | outputs = torch.transpose(outputs, 0, 1) 360 | outputs = outputs.unsqueeze(0) 361 | target = target.unsqueeze(0) 362 | 363 | # Cross entropy loss 364 | self.output_loss = self.criterion(outputs, target) 365 | 366 | # Regularization of deformable offsets 367 | if self.deform_fitting_mode == 'point2point': 368 | self.reg_loss = p2p_fitting_regularizer(self) 369 | elif self.deform_fitting_mode == 'point2plane': 370 | raise ValueError('point2plane fitting mode not implemented yet.') 371 | else: 372 | raise ValueError('Unknown fitting mode: ' + self.deform_fitting_mode) 373 | 374 | # Combined loss 375 | return self.output_loss + self.reg_loss 376 | 377 | def accuracy(self, outputs, labels): 378 | """ 379 | Computes accuracy of the current batch 380 | :param outputs: logits predicted by the network 381 | :param labels: labels 382 | :return: accuracy value 383 | """ 384 | 385 | # Set all ignored labels to -1 and correct the other label to be in [0, C-1] range 386 | target = - torch.ones_like(labels) 387 | for i, c in enumerate(self.valid_labels): 388 | target[labels == c] = i 389 | 390 | predicted = torch.argmax(outputs.data, dim=1) 391 | total = target.size(0) 392 | correct = (predicted == target).sum().item() 393 | 394 | return correct / total 395 | 396 | 397 | 398 | 399 | 400 | 401 | 402 | 403 | 404 | 405 | 406 | 407 | 408 | 409 | 410 | 411 | 412 | 413 | 414 | 415 | 416 | -------------------------------------------------------------------------------- /test_models.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Callable script to start a training on ModelNet40 dataset 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 06/03/2020 15 | # 16 | 17 | 18 | # ---------------------------------------------------------------------------------------------------------------------- 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | # Common libs 25 | import signal 26 | import os 27 | import numpy as np 28 | import sys 29 | import torch 30 | 31 | # Dataset 32 | from datasets.ModelNet40 import * 33 | from datasets.S3DIS import * 34 | from datasets.SensatUrban import * 35 | from datasets.SemanticKitti import * 36 | from datasets.Toronto3D import * 37 | from torch.utils.data import DataLoader 38 | 39 | from utils.config import Config 40 | from utils.tester import ModelTester 41 | from models.architectures import KPCNN, KPFCNN 42 | 43 | 44 | # ---------------------------------------------------------------------------------------------------------------------- 45 | # 46 | # Main Call 47 | # \***************/ 48 | # 49 | 50 | def model_choice(chosen_log): 51 | 52 | ########################### 53 | # Call the test initializer 54 | ########################### 55 | 56 | # Automatically retrieve the last trained model 57 | if chosen_log in ['last_ModelNet40', 'last_ShapeNetPart', 'last_S3DIS', 'last_sensaturban']: 58 | 59 | # Dataset name 60 | test_dataset = '_'.join(chosen_log.split('_')[1:]) 61 | 62 | # List all training logs 63 | logs = np.sort([os.path.join('results', f) for f in os.listdir('results') if f.startswith('Log')]) 64 | 65 | # Find the last log of asked dataset 66 | for log in logs[::-1]: 67 | log_config = Config() 68 | log_config.load(log) 69 | if log_config.dataset.startswith(test_dataset): 70 | chosen_log = log 71 | break 72 | 73 | if chosen_log in ['last_ModelNet40', 'last_ShapeNetPart', 'last_S3DIS', 'last_SensatUrban']: 74 | raise ValueError('No log of the dataset "' + test_dataset + '" found') 75 | 76 | # Check if log exists 77 | if not os.path.exists(chosen_log): 78 | raise ValueError('The given log does not exists: ' + chosen_log) 79 | 80 | return chosen_log 81 | 82 | 83 | # ---------------------------------------------------------------------------------------------------------------------- 84 | # 85 | # Main Call 86 | # \***************/ 87 | # 88 | 89 | if __name__ == '__main__': 90 | 91 | ############################### 92 | # Choose the model to visualize 93 | ############################### 94 | 95 | # Here you can choose which model you want to test with the variable test_model. Here are the possible values : 96 | # 97 | # > 'last_XXX': Automatically retrieve the last trained model on dataset XXX 98 | # > '(old_)results/Log_YYYY-MM-DD_HH-MM-SS': Directly provide the path of a trained model 99 | 100 | chosen_log = 'results/Log_2024-05-14_21-04-36' 101 | 102 | # Choose the index of the checkpoint to load OR None if you want to load the current checkpoint 103 | chkp_idx = -1 104 | 105 | # Choose to test on validation or test split 106 | on_val = True 107 | 108 | # Deal with 'last_XXXXXX' choices 109 | chosen_log = model_choice(chosen_log) 110 | 111 | ############################ 112 | # Initialize the environment 113 | ############################ 114 | 115 | # Set which gpu is going to be used 116 | GPU_ID = '0' 117 | 118 | # Set GPU visible device 119 | os.environ['CUDA_VISIBLE_DEVICES'] = GPU_ID 120 | 121 | ############### 122 | # Previous chkp 123 | ############### 124 | 125 | # Find all checkpoints in the chosen training folder 126 | chkp_path = os.path.join(chosen_log, 'checkpoints') 127 | chkps = [f for f in os.listdir(chkp_path) if f[:4] == 'chkp'] 128 | 129 | # Find which snapshot to restore 130 | if chkp_idx is None: 131 | chosen_chkp = 'current_chkp.tar' 132 | else: 133 | chosen_chkp = np.sort(chkps)[chkp_idx] 134 | chosen_chkp = os.path.join(chosen_log, 'checkpoints', chosen_chkp) 135 | 136 | # Initialize configuration class 137 | config = Config() 138 | config.load(chosen_log) 139 | 140 | ################################## 141 | # Change model parameters for test 142 | ################################## 143 | 144 | # Change parameters for the test here. For example, you can stop augmenting the input data. 145 | 146 | #config.augment_noise = 0.0001 147 | #config.augment_symmetries = False 148 | #config.batch_num = 3 149 | #config.in_radius = 4 150 | config.validation_size = 200 151 | config.input_threads = 10 152 | 153 | ############## 154 | # Prepare Data 155 | ############## 156 | 157 | print() 158 | print('Data Preparation') 159 | print('****************') 160 | 161 | if on_val: 162 | set = 'validation' 163 | else: 164 | set = 'test' 165 | 166 | # Initiate dataset 167 | if config.dataset == 'ModelNet40': 168 | test_dataset = ModelNet40Dataset(config, train=False) 169 | test_sampler = ModelNet40Sampler(test_dataset) 170 | collate_fn = ModelNet40Collate 171 | elif config.dataset == 'S3DIS': 172 | test_dataset = S3DISDataset(config, set='validation', use_potentials=True) 173 | test_sampler = S3DISSampler(test_dataset) 174 | collate_fn = S3DISCollate 175 | elif config.dataset == 'SensatUrban': 176 | test_dataset = SensatUrbanDataset(config, set='validation', use_potentials=True) 177 | test_sampler = SensatUrbanSampler(test_dataset) 178 | collate_fn = SensatUrbanCollate 179 | elif config.dataset == 'Toronto3D': 180 | test_dataset = Toronto3DDataset(config, set='test', use_potentials=True) 181 | test_sampler = Toronto3DSampler(test_dataset) 182 | collate_fn = Toronto3DCollate 183 | elif config.dataset == 'SemanticKitti': 184 | test_dataset = SemanticKittiDataset(config, set=set, balance_classes=False) 185 | test_sampler = SemanticKittiSampler(test_dataset) 186 | collate_fn = SemanticKittiCollate 187 | else: 188 | raise ValueError('Unsupported dataset : ' + config.dataset) 189 | 190 | # Data loader 191 | test_loader = DataLoader(test_dataset, 192 | batch_size=1, 193 | sampler=test_sampler, 194 | collate_fn=collate_fn, 195 | num_workers=config.input_threads, 196 | pin_memory=True) 197 | 198 | # Calibrate samplers 199 | test_sampler.calibration(test_loader, verbose=True) 200 | 201 | print('\nModel Preparation') 202 | print('*****************') 203 | 204 | # Define network model 205 | t1 = time.time() 206 | if config.dataset_task == 'classification': 207 | net = KPCNN(config) 208 | elif config.dataset_task in ['cloud_segmentation', 'slam_segmentation']: 209 | net = KPFCNN(config, test_dataset.label_values, test_dataset.ignored_labels) 210 | else: 211 | raise ValueError('Unsupported dataset_task for testing: ' + config.dataset_task) 212 | 213 | # Define a visualizer class 214 | tester = ModelTester(net, chkp_path=chosen_chkp) 215 | print('Done in {:.1f}s\n'.format(time.time() - t1)) 216 | 217 | print('\nStart test') 218 | print('**********\n') 219 | 220 | # Training 221 | if config.dataset_task == 'classification': 222 | tester.classification_test(net, test_loader, config) 223 | elif config.dataset_task == 'cloud_segmentation': 224 | tester.cloud_segmentation_test(net, test_loader, config) 225 | elif config.dataset_task == 'slam_segmentation': 226 | tester.slam_segmentation_test(net, test_loader, config) 227 | else: 228 | raise ValueError('Unsupported dataset_task for testing: ' + config.dataset_task) 229 | -------------------------------------------------------------------------------- /train_ModelNet40.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Callable script to start a training on ModelNet40 dataset 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 06/03/2020 15 | # 16 | 17 | 18 | # ---------------------------------------------------------------------------------------------------------------------- 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | # Common libs 25 | import signal 26 | import os 27 | import numpy as np 28 | import sys 29 | import torch 30 | 31 | # Dataset 32 | from datasets.ModelNet40 import * 33 | from torch.utils.data import DataLoader 34 | 35 | from utils.config import Config 36 | from utils.trainer import ModelTrainer 37 | from models.architectures import KPCNN 38 | 39 | 40 | # ---------------------------------------------------------------------------------------------------------------------- 41 | # 42 | # Config Class 43 | # \******************/ 44 | # 45 | 46 | class Modelnet40Config(Config): 47 | """ 48 | Override the parameters you want to modify for this dataset 49 | """ 50 | 51 | #################### 52 | # Dataset parameters 53 | #################### 54 | 55 | # Dataset name 56 | dataset = 'ModelNet40' 57 | 58 | # Number of classes in the dataset (This value is overwritten by dataset class when Initializating dataset). 59 | num_classes = None 60 | 61 | # Type of task performed on this dataset (also overwritten) 62 | dataset_task = '' 63 | 64 | # Number of CPU threads for the input pipeline 65 | input_threads = 10 66 | 67 | ######################### 68 | # Architecture definition 69 | ######################### 70 | 71 | # Define layers 72 | architecture = ['simple', 73 | 'resnetb', 74 | 'resnetb_strided', 75 | 'resnetb', 76 | 'resnetb', 77 | 'resnetb_strided', 78 | 'resnetb', 79 | 'resnetb', 80 | 'resnetb_strided', 81 | 'resnetb', 82 | 'resnetb', 83 | 'resnetb_strided', 84 | 'resnetb', 85 | 'resnetb', 86 | 'global_average'] 87 | 88 | ################### 89 | # KPConv parameters 90 | ################### 91 | 92 | # Number of kernel points 93 | num_kernel_points = 15 94 | 95 | # Size of the first subsampling grid in meter 96 | first_subsampling_dl = 0.02 97 | 98 | # Radius of convolution in "number grid cell". (2.5 is the standard value) 99 | conv_radius = 2.5 100 | 101 | # Radius of deformable convolution in "number grid cell". Larger so that deformed kernel can spread out 102 | deform_radius = 6.0 103 | 104 | # Radius of the area of influence of each kernel point in "number grid cell". (1.0 is the standard value) 105 | KP_extent = 1.2 106 | 107 | # Behavior of convolutions in ('constant', 'linear', 'gaussian') 108 | KP_influence = 'linear' 109 | 110 | # Aggregation function of KPConv in ('closest', 'sum') 111 | aggregation_mode = 'sum' 112 | 113 | # Choice of input features 114 | in_features_dim = 1 115 | 116 | # Can the network learn modulations 117 | modulated = True 118 | 119 | # Batch normalization parameters 120 | use_batch_norm = True 121 | batch_norm_momentum = 0.05 122 | 123 | # Deformable offset loss 124 | # 'point2point' fitting geometry by penalizing distance from deform point to input points 125 | # 'point2plane' fitting geometry by penalizing distance from deform point to input point triplet (not implemented) 126 | deform_fitting_mode = 'point2point' 127 | deform_fitting_power = 1.0 # Multiplier for the fitting/repulsive loss 128 | deform_lr_factor = 0.1 # Multiplier for learning rate applied to the deformations 129 | repulse_extent = 1.2 # Distance of repulsion for deformed kernel points 130 | 131 | ##################### 132 | # Training parameters 133 | ##################### 134 | 135 | # Maximal number of epochs 136 | max_epoch = 500 137 | 138 | # Learning rate management 139 | learning_rate = 1e-2 140 | momentum = 0.98 141 | lr_decays = {i: 0.1**(1/100) for i in range(1, max_epoch)} 142 | grad_clip_norm = 100.0 143 | 144 | # Number of batch 145 | batch_num = 10 146 | 147 | # Number of steps per epochs 148 | epoch_steps = 300 149 | 150 | # Number of validation examples per epoch 151 | validation_size = 30 152 | 153 | # Number of epoch between each checkpoint 154 | checkpoint_gap = 50 155 | 156 | # Augmentations 157 | augment_scale_anisotropic = True 158 | augment_symmetries = [True, True, True] 159 | augment_rotation = 'none' 160 | augment_scale_min = 0.8 161 | augment_scale_max = 1.2 162 | augment_noise = 0.001 163 | augment_color = 1.0 164 | 165 | # The way we balance segmentation loss 166 | # > 'none': Each point in the whole batch has the same contribution. 167 | # > 'class': Each class has the same contribution (points are weighted according to class balance) 168 | # > 'batch': Each cloud in the batch has the same contribution (points are weighted according cloud sizes) 169 | segloss_balance = 'none' 170 | 171 | # Do we nee to save convergence 172 | saving = True 173 | saving_path = None 174 | 175 | 176 | # ---------------------------------------------------------------------------------------------------------------------- 177 | # 178 | # Main Call 179 | # \***************/ 180 | # 181 | 182 | if __name__ == '__main__': 183 | 184 | ############################ 185 | # Initialize the environment 186 | ############################ 187 | 188 | # Set which gpu is going to be used 189 | GPU_ID = '0' 190 | 191 | # Set GPU visible device 192 | os.environ['CUDA_VISIBLE_DEVICES'] = GPU_ID 193 | 194 | ############### 195 | # Previous chkp 196 | ############### 197 | 198 | # Choose here if you want to start training from a previous snapshot (None for new training) 199 | #previous_training_path = 'Log_2020-03-19_19-53-27' 200 | previous_training_path = '' 201 | 202 | # Choose index of checkpoint to start from. If None, uses the latest chkp 203 | chkp_idx = None 204 | if previous_training_path: 205 | 206 | # Find all snapshot in the chosen training folder 207 | chkp_path = os.path.join('results', previous_training_path, 'checkpoints') 208 | chkps = [f for f in os.listdir(chkp_path) if f[:4] == 'chkp'] 209 | 210 | # Find which snapshot to restore 211 | if chkp_idx is None: 212 | chosen_chkp = 'current_chkp.tar' 213 | else: 214 | chosen_chkp = np.sort(chkps)[chkp_idx] 215 | chosen_chkp = os.path.join('results', previous_training_path, 'checkpoints', chosen_chkp) 216 | 217 | else: 218 | chosen_chkp = None 219 | 220 | ############## 221 | # Prepare Data 222 | ############## 223 | 224 | print() 225 | print('Data Preparation') 226 | print('****************') 227 | 228 | # Initialize configuration class 229 | config = Modelnet40Config() 230 | if previous_training_path: 231 | config.load(os.path.join('results', previous_training_path)) 232 | config.saving_path = None 233 | 234 | # Get path from argument if given 235 | if len(sys.argv) > 1: 236 | config.saving_path = sys.argv[1] 237 | 238 | # Initialize datasets 239 | training_dataset = ModelNet40Dataset(config, train=True) 240 | test_dataset = ModelNet40Dataset(config, train=False) 241 | 242 | # Initialize samplers 243 | training_sampler = ModelNet40Sampler(training_dataset, balance_labels=True) 244 | test_sampler = ModelNet40Sampler(test_dataset, balance_labels=True) 245 | 246 | # Initialize the dataloader 247 | training_loader = DataLoader(training_dataset, 248 | batch_size=1, 249 | sampler=training_sampler, 250 | collate_fn=ModelNet40Collate, 251 | num_workers=config.input_threads, 252 | pin_memory=True) 253 | test_loader = DataLoader(test_dataset, 254 | batch_size=1, 255 | sampler=test_sampler, 256 | collate_fn=ModelNet40Collate, 257 | num_workers=config.input_threads, 258 | pin_memory=True) 259 | 260 | # Calibrate samplers 261 | training_sampler.calibration(training_loader) 262 | test_sampler.calibration(test_loader) 263 | 264 | #debug_timing(test_dataset, test_sampler, test_loader) 265 | #debug_show_clouds(training_dataset, training_sampler, training_loader) 266 | 267 | print('\nModel Preparation') 268 | print('*****************') 269 | 270 | # Define network model 271 | t1 = time.time() 272 | net = KPCNN(config) 273 | 274 | # Define a trainer class 275 | trainer = ModelTrainer(net, config, chkp_path=chosen_chkp) 276 | print('Done in {:.1f}s\n'.format(time.time() - t1)) 277 | 278 | print('\nStart training') 279 | print('**************') 280 | 281 | # Training 282 | try: 283 | trainer.train(net, training_loader, test_loader, config) 284 | except: 285 | print('Caught an error') 286 | os.kill(os.getpid(), signal.SIGINT) 287 | 288 | print('Forcing exit now') 289 | os.kill(os.getpid(), signal.SIGINT) 290 | 291 | 292 | 293 | -------------------------------------------------------------------------------- /train_NPM3D.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Callable script to start a training on NPM3D dataset 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 06/03/2020 15 | # 16 | 17 | 18 | # ---------------------------------------------------------------------------------------------------------------------- 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | # Common libs 25 | import signal 26 | import os 27 | 28 | # Dataset 29 | from datasets.NPM3D import * 30 | from torch.utils.data import DataLoader 31 | 32 | from utils.config import Config 33 | from utils.trainer import ModelTrainer 34 | from models.architectures import KPFCNN 35 | 36 | 37 | # ---------------------------------------------------------------------------------------------------------------------- 38 | # 39 | # Config Class 40 | # \******************/ 41 | # 42 | 43 | class NPM3DConfig(Config): 44 | """ 45 | Override the parameters you want to modify for this dataset 46 | """ 47 | 48 | #################### 49 | # Dataset parameters 50 | #################### 51 | 52 | # Dataset name 53 | dataset = 'NPM3D' 54 | 55 | # Number of classes in the dataset (This value is overwritten by dataset class when Initializating dataset). 56 | num_classes = None 57 | 58 | # Type of task performed on this dataset (also overwritten) 59 | dataset_task = '' 60 | 61 | # Number of CPU threads for the input pipeline 62 | input_threads = 10 63 | 64 | ######################### 65 | # Architecture definition 66 | ######################### 67 | 68 | # # Define layers 69 | architecture = ['simple', 70 | 'resnetb', 71 | 'resnetb_strided', 72 | 'resnetb', 73 | 'resnetb', 74 | 'resnetb_strided', 75 | 'resnetb', 76 | 'resnetb', 77 | 'resnetb_strided', 78 | 'resnetb', 79 | 'resnetb', 80 | 'resnetb_strided', 81 | 'resnetb', 82 | 'resnetb', 83 | 'nearest_upsample', 84 | 'unary', 85 | 'nearest_upsample', 86 | 'unary', 87 | 'nearest_upsample', 88 | 'unary', 89 | 'nearest_upsample', 90 | 'unary'] 91 | 92 | ################### 93 | # KPConv parameters 94 | ################### 95 | 96 | # Number of kernel points 97 | num_kernel_points = 15 98 | 99 | # Radius of the input sphere (decrease value to reduce memory cost) 100 | in_radius = 3.0 101 | 102 | # Size of the first subsampling grid in meter (increase value to reduce memory cost) 103 | first_subsampling_dl = 0.06 104 | 105 | # Radius of convolution in "number grid cell". (2.5 is the standard value) 106 | conv_radius = 2.5 107 | 108 | # Radius of deformable convolution in "number grid cell". Larger so that deformed kernel can spread out 109 | deform_radius = 5.0 110 | 111 | # Radius of the area of influence of each kernel point in "number grid cell". (1.0 is the standard value) 112 | KP_extent = 1.2 113 | 114 | # Behavior of convolutions in ('constant', 'linear', 'gaussian') 115 | KP_influence = 'linear' 116 | 117 | # Aggregation function of KPConv in ('closest', 'sum') 118 | aggregation_mode = 'sum' 119 | 120 | # Choice of input features 121 | first_features_dim = 128 122 | in_features_dim = 1 123 | 124 | # Can the network learn modulations 125 | modulated = False 126 | 127 | # Batch normalization parameters 128 | use_batch_norm = True 129 | batch_norm_momentum = 0.02 130 | 131 | # Deformable offset loss 132 | # 'point2point' fitting geometry by penalizing distance from deform point to input points 133 | # 'point2plane' fitting geometry by penalizing distance from deform point to input point triplet (not implemented) 134 | deform_fitting_mode = 'point2point' 135 | deform_fitting_power = 1.0 # Multiplier for the fitting/repulsive loss 136 | deform_lr_factor = 0.1 # Multiplier for learning rate applied to the deformations 137 | repulse_extent = 1.2 # Distance of repulsion for deformed kernel points 138 | 139 | ##################### 140 | # Training parameters 141 | ##################### 142 | 143 | # Maximal number of epochs 144 | max_epoch = 500 145 | 146 | # Learning rate management 147 | learning_rate = 1e-2 148 | momentum = 0.98 149 | lr_decays = {i: 0.1 ** (1 / 150) for i in range(1, max_epoch)} 150 | grad_clip_norm = 100.0 151 | 152 | # Number of batch (decrease to reduce memory cost, but it should remain > 3 for stability) 153 | batch_num = 6 154 | 155 | # Number of steps per epochs 156 | epoch_steps = 500 157 | 158 | # Number of validation examples per epoch 159 | validation_size = 50 160 | 161 | # Number of epoch between each checkpoint 162 | checkpoint_gap = 50 163 | 164 | # Augmentations 165 | augment_scale_anisotropic = True 166 | augment_symmetries = [True, False, False] 167 | augment_rotation = 'vertical' 168 | augment_scale_min = 0.9 169 | augment_scale_max = 1.1 170 | augment_noise = 0.001 171 | augment_color = 0.8 172 | 173 | # The way we balance segmentation loss 174 | # > 'none': Each point in the whole batch has the same contribution. 175 | # > 'class': Each class has the same contribution (points are weighted according to class balance) 176 | # > 'batch': Each cloud in the batch has the same contribution (points are weighted according cloud sizes) 177 | segloss_balance = 'none' 178 | 179 | # Do we nee to save convergence 180 | saving = True 181 | saving_path = None 182 | 183 | 184 | # ---------------------------------------------------------------------------------------------------------------------- 185 | # 186 | # Main Call 187 | # \***************/ 188 | # 189 | 190 | if __name__ == '__main__': 191 | 192 | ############################ 193 | # Initialize the environment 194 | ############################ 195 | 196 | # Set which gpu is going to be used 197 | GPU_ID = '0' 198 | 199 | # Set GPU visible device 200 | os.environ['CUDA_VISIBLE_DEVICES'] = GPU_ID 201 | 202 | ############### 203 | # Previous chkp 204 | ############### 205 | 206 | # Choose here if you want to start training from a previous snapshot (None for new training) 207 | # previous_training_path = 'Log_2020-03-19_19-53-27' 208 | previous_training_path = '' 209 | 210 | # Choose index of checkpoint to start from. If None, uses the latest chkp 211 | chkp_idx = None 212 | if previous_training_path: 213 | 214 | # Find all snapshot in the chosen training folder 215 | chkp_path = os.path.join('results', previous_training_path, 'checkpoints') 216 | chkps = [f for f in os.listdir(chkp_path) if f[:4] == 'chkp'] 217 | 218 | # Find which snapshot to restore 219 | if chkp_idx is None: 220 | chosen_chkp = 'current_chkp.tar' 221 | else: 222 | chosen_chkp = np.sort(chkps)[chkp_idx] 223 | chosen_chkp = os.path.join('results', previous_training_path, 'checkpoints', chosen_chkp) 224 | 225 | else: 226 | chosen_chkp = None 227 | 228 | ############## 229 | # Prepare Data 230 | ############## 231 | 232 | print() 233 | print('Data Preparation') 234 | print('****************') 235 | 236 | # Initialize configuration class 237 | config = NPM3DConfig() 238 | if previous_training_path: 239 | config.load(os.path.join('results', previous_training_path)) 240 | config.saving_path = None 241 | 242 | # Get path from argument if given 243 | if len(sys.argv) > 1: 244 | config.saving_path = sys.argv[1] 245 | 246 | # Initialize datasets 247 | training_dataset = NPM3DDataset(config, set='training', use_potentials=True) 248 | test_dataset = NPM3DDataset(config, set='validation', use_potentials=True) 249 | 250 | # Initialize samplers 251 | training_sampler = NPM3DSampler(training_dataset) 252 | test_sampler = NPM3DSampler(test_dataset) 253 | 254 | # Initialize the dataloader 255 | training_loader = DataLoader(training_dataset, 256 | batch_size=1, 257 | sampler=training_sampler, 258 | collate_fn=NPM3DCollate, 259 | num_workers=config.input_threads, 260 | pin_memory=True) 261 | test_loader = DataLoader(test_dataset, 262 | batch_size=1, 263 | sampler=test_sampler, 264 | collate_fn=NPM3DCollate, 265 | num_workers=config.input_threads, 266 | pin_memory=True) 267 | 268 | # Calibrate samplers 269 | training_sampler.calibration(training_loader, verbose=True) 270 | test_sampler.calibration(test_loader, verbose=True) 271 | 272 | # Optional debug functions 273 | # debug_timing(training_dataset, training_loader) 274 | # debug_timing(test_dataset, test_loader) 275 | # debug_upsampling(training_dataset, training_loader) 276 | 277 | print('\nModel Preparation') 278 | print('*****************') 279 | 280 | # Define network model 281 | t1 = time.time() 282 | net = KPFCNN(config, training_dataset.label_values, training_dataset.ignored_labels) 283 | 284 | debug = False 285 | if debug: 286 | print('\n*************************************\n') 287 | print(net) 288 | print('\n*************************************\n') 289 | for param in net.parameters(): 290 | if param.requires_grad: 291 | print(param.shape) 292 | print('\n*************************************\n') 293 | print("Model size %i" % sum(param.numel() for param in net.parameters() if param.requires_grad)) 294 | print('\n*************************************\n') 295 | 296 | # Define a trainer class 297 | trainer = ModelTrainer(net, config, chkp_path=chosen_chkp) 298 | print('Done in {:.1f}s\n'.format(time.time() - t1)) 299 | 300 | print('\nStart training') 301 | print('**************') 302 | 303 | # Training 304 | trainer.train(net, training_loader, test_loader, config) 305 | 306 | print('Forcing exit now') 307 | os.kill(os.getpid(), signal.SIGINT) 308 | -------------------------------------------------------------------------------- /train_S3DIS.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Callable script to start a training on S3DIS dataset 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 06/03/2020 15 | # 16 | 17 | 18 | # ---------------------------------------------------------------------------------------------------------------------- 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | # Common libs 25 | import signal 26 | import os 27 | 28 | # Dataset 29 | from datasets.S3DIS import * 30 | from torch.utils.data import DataLoader 31 | 32 | from utils.config import Config 33 | from utils.trainer import ModelTrainer 34 | from models.architectures import KPFCNN 35 | 36 | 37 | # ---------------------------------------------------------------------------------------------------------------------- 38 | # 39 | # Config Class 40 | # \******************/ 41 | # 42 | 43 | class S3DISConfig(Config): 44 | """ 45 | Override the parameters you want to modify for this dataset 46 | """ 47 | 48 | #################### 49 | # Dataset parameters 50 | #################### 51 | 52 | # Dataset name 53 | dataset = 'S3DIS' 54 | 55 | # Number of classes in the dataset (This value is overwritten by dataset class when Initializating dataset). 56 | num_classes = None 57 | 58 | # Type of task performed on this dataset (also overwritten) 59 | dataset_task = '' 60 | 61 | # Number of CPU threads for the input pipeline 62 | input_threads = 10 63 | 64 | ######################### 65 | # Architecture definition 66 | ######################### 67 | 68 | # # Define layers 69 | architecture = ['simple', 70 | 'resnetb', 71 | 'resnetb_strided', 72 | 'resnetb', 73 | 'resnetb', 74 | 'resnetb_strided', 75 | 'resnetb', 76 | 'resnetb', 77 | 'resnetb_strided', 78 | 'resnetb_deformable', 79 | 'resnetb_deformable', 80 | 'resnetb_deformable_strided', 81 | 'resnetb_deformable', 82 | 'resnetb_deformable', 83 | 'nearest_upsample', 84 | 'unary', 85 | 'nearest_upsample', 86 | 'unary', 87 | 'nearest_upsample', 88 | 'unary', 89 | 'nearest_upsample', 90 | 'unary'] 91 | 92 | # Define layers 93 | # architecture = ['simple', 94 | # 'resnetb', 95 | # 'resnetb_strided', 96 | # 'resnetb', 97 | # 'resnetb', 98 | # 'resnetb_strided', 99 | # 'resnetb', 100 | # 'resnetb', 101 | # 'resnetb_strided', 102 | # 'resnetb', 103 | # 'resnetb', 104 | # 'resnetb_strided', 105 | # 'resnetb', 106 | # 'resnetb', 107 | # 'nearest_upsample', 108 | # 'unary', 109 | # 'nearest_upsample', 110 | # 'unary', 111 | # 'nearest_upsample', 112 | # 'unary', 113 | # 'nearest_upsample', 114 | # 'unary'] 115 | 116 | ################### 117 | # KPConv parameters 118 | ################### 119 | 120 | # Number of kernel points 121 | num_kernel_points = 15 122 | 123 | # Radius of the input sphere (decrease value to reduce memory cost) 124 | in_radius = 1.2 125 | 126 | # Size of the first subsampling grid in meter (increase value to reduce memory cost) 127 | first_subsampling_dl = 0.03 128 | 129 | # Radius of convolution in "number grid cell". (2.5 is the standard value) 130 | conv_radius = 2.5 131 | 132 | # Radius of deformable convolution in "number grid cell". Larger so that deformed kernel can spread out 133 | deform_radius = 5.0 134 | 135 | # Radius of the area of influence of each kernel point in "number grid cell". (1.0 is the standard value) 136 | KP_extent = 1.2 137 | 138 | # Behavior of convolutions in ('constant', 'linear', 'gaussian') 139 | KP_influence = 'linear' 140 | 141 | # Aggregation function of KPConv in ('closest', 'sum') 142 | aggregation_mode = 'sum' 143 | 144 | # Choice of input features 145 | first_features_dim = 128 146 | in_features_dim = 5 147 | 148 | # Can the network learn modulations 149 | modulated = False 150 | 151 | # Batch normalization parameters 152 | use_batch_norm = True 153 | batch_norm_momentum = 0.02 154 | 155 | # Deformable offset loss 156 | # 'point2point' fitting geometry by penalizing distance from deform point to input points 157 | # 'point2plane' fitting geometry by penalizing distance from deform point to input point triplet (not implemented) 158 | deform_fitting_mode = 'point2point' 159 | deform_fitting_power = 1.0 # Multiplier for the fitting/repulsive loss 160 | deform_lr_factor = 0.1 # Multiplier for learning rate applied to the deformations 161 | repulse_extent = 1.2 # Distance of repulsion for deformed kernel points 162 | 163 | ##################### 164 | # Training parameters 165 | ##################### 166 | 167 | # Maximal number of epochs 168 | max_epoch = 500 169 | 170 | # Learning rate management 171 | learning_rate = 1e-2 172 | momentum = 0.98 173 | lr_decays = {i: 0.1 ** (1 / 150) for i in range(1, max_epoch)} 174 | grad_clip_norm = 100.0 175 | 176 | # Number of batch (decrease to reduce memory cost, but it should remain > 3 for stability) 177 | batch_num = 6 178 | 179 | # Number of steps per epochs 180 | epoch_steps = 500 181 | 182 | # Number of validation examples per epoch 183 | validation_size = 50 184 | 185 | # Number of epoch between each checkpoint 186 | checkpoint_gap = 50 187 | 188 | # Augmentations 189 | augment_scale_anisotropic = True 190 | augment_symmetries = [True, False, False] 191 | augment_rotation = 'vertical' 192 | augment_scale_min = 0.9 193 | augment_scale_max = 1.1 194 | augment_noise = 0.001 195 | augment_color = 0.8 196 | 197 | # The way we balance segmentation loss 198 | # > 'none': Each point in the whole batch has the same contribution. 199 | # > 'class': Each class has the same contribution (points are weighted according to class balance) 200 | # > 'batch': Each cloud in the batch has the same contribution (points are weighted according cloud sizes) 201 | segloss_balance = 'none' 202 | 203 | # Do we nee to save convergence 204 | saving = True 205 | saving_path = None 206 | 207 | 208 | # ---------------------------------------------------------------------------------------------------------------------- 209 | # 210 | # Main Call 211 | # \***************/ 212 | # 213 | 214 | if __name__ == '__main__': 215 | 216 | ############################ 217 | # Initialize the environment 218 | ############################ 219 | 220 | # Set which gpu is going to be used 221 | GPU_ID = '0' 222 | 223 | # Set GPU visible device 224 | os.environ['CUDA_VISIBLE_DEVICES'] = GPU_ID 225 | 226 | ############### 227 | # Previous chkp 228 | ############### 229 | 230 | # Choose here if you want to start training from a previous snapshot (None for new training) 231 | # previous_training_path = 'Log_2020-03-19_19-53-27' 232 | previous_training_path = '' 233 | 234 | # Choose index of checkpoint to start from. If None, uses the latest chkp 235 | chkp_idx = None 236 | if previous_training_path: 237 | 238 | # Find all snapshot in the chosen training folder 239 | chkp_path = os.path.join('results', previous_training_path, 'checkpoints') 240 | chkps = [f for f in os.listdir(chkp_path) if f[:4] == 'chkp'] 241 | 242 | # Find which snapshot to restore 243 | if chkp_idx is None: 244 | chosen_chkp = 'current_chkp.tar' 245 | else: 246 | chosen_chkp = np.sort(chkps)[chkp_idx] 247 | chosen_chkp = os.path.join('results', previous_training_path, 'checkpoints', chosen_chkp) 248 | 249 | else: 250 | chosen_chkp = None 251 | 252 | ############## 253 | # Prepare Data 254 | ############## 255 | 256 | print() 257 | print('Data Preparation') 258 | print('****************') 259 | 260 | # Initialize configuration class 261 | config = S3DISConfig() 262 | if previous_training_path: 263 | config.load(os.path.join('results', previous_training_path)) 264 | config.saving_path = None 265 | 266 | # Get path from argument if given 267 | if len(sys.argv) > 1: 268 | config.saving_path = sys.argv[1] 269 | 270 | # Initialize datasets 271 | training_dataset = S3DISDataset(config, set='training', use_potentials=True) 272 | test_dataset = S3DISDataset(config, set='validation', use_potentials=True) 273 | 274 | # Initialize samplers 275 | training_sampler = S3DISSampler(training_dataset) 276 | test_sampler = S3DISSampler(test_dataset) 277 | 278 | # Initialize the dataloader 279 | training_loader = DataLoader(training_dataset, 280 | batch_size=1, 281 | sampler=training_sampler, 282 | collate_fn=S3DISCollate, 283 | num_workers=config.input_threads, 284 | pin_memory=True) 285 | test_loader = DataLoader(test_dataset, 286 | batch_size=1, 287 | sampler=test_sampler, 288 | collate_fn=S3DISCollate, 289 | num_workers=config.input_threads, 290 | pin_memory=True) 291 | 292 | # Calibrate samplers 293 | training_sampler.calibration(training_loader, verbose=True) 294 | test_sampler.calibration(test_loader, verbose=True) 295 | 296 | # Optional debug functions 297 | # debug_timing(training_dataset, training_loader) 298 | # debug_timing(test_dataset, test_loader) 299 | # debug_upsampling(training_dataset, training_loader) 300 | 301 | print('\nModel Preparation') 302 | print('*****************') 303 | 304 | # Define network model 305 | t1 = time.time() 306 | net = KPFCNN(config, training_dataset.label_values, training_dataset.ignored_labels) 307 | 308 | debug = False 309 | if debug: 310 | print('\n*************************************\n') 311 | print(net) 312 | print('\n*************************************\n') 313 | for param in net.parameters(): 314 | if param.requires_grad: 315 | print(param.shape) 316 | print('\n*************************************\n') 317 | print("Model size %i" % sum(param.numel() for param in net.parameters() if param.requires_grad)) 318 | print('\n*************************************\n') 319 | 320 | # Define a trainer class 321 | trainer = ModelTrainer(net, config, chkp_path=chosen_chkp) 322 | print('Done in {:.1f}s\n'.format(time.time() - t1)) 323 | 324 | print('\nStart training') 325 | print('**************') 326 | 327 | # Training 328 | trainer.train(net, training_loader, test_loader, config) 329 | 330 | print('Forcing exit now') 331 | os.kill(os.getpid(), signal.SIGINT) 332 | -------------------------------------------------------------------------------- /train_SemanticKitti.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Callable script to start a training on SemanticKitti dataset 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 06/03/2020 15 | # 16 | 17 | 18 | # ---------------------------------------------------------------------------------------------------------------------- 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | # Common libs 25 | import signal 26 | import os 27 | import numpy as np 28 | import sys 29 | import torch 30 | 31 | # Dataset 32 | from datasets.SemanticKitti import * 33 | from torch.utils.data import DataLoader 34 | 35 | from utils.config import Config 36 | from utils.trainer import ModelTrainer 37 | from models.architectures import KPFCNN 38 | 39 | 40 | # ---------------------------------------------------------------------------------------------------------------------- 41 | # 42 | # Config Class 43 | # \******************/ 44 | # 45 | 46 | class SemanticKittiConfig(Config): 47 | """ 48 | Override the parameters you want to modify for this dataset 49 | """ 50 | 51 | #################### 52 | # Dataset parameters 53 | #################### 54 | 55 | # Dataset name 56 | dataset = 'SemanticKitti' 57 | 58 | # Number of classes in the dataset (This value is overwritten by dataset class when Initializating dataset). 59 | num_classes = None 60 | 61 | # Type of task performed on this dataset (also overwritten) 62 | dataset_task = '' 63 | 64 | # Number of CPU threads for the input pipeline 65 | input_threads = 10 66 | 67 | ######################### 68 | # Architecture definition 69 | ######################### 70 | 71 | # Define layers 72 | architecture = ['simple', 73 | 'resnetb', 74 | 'resnetb_strided', 75 | 'resnetb', 76 | 'resnetb', 77 | 'resnetb_strided', 78 | 'resnetb', 79 | 'resnetb', 80 | 'resnetb_strided', 81 | 'resnetb', 82 | 'resnetb', 83 | 'resnetb_strided', 84 | 'resnetb', 85 | 'nearest_upsample', 86 | 'unary', 87 | 'nearest_upsample', 88 | 'unary', 89 | 'nearest_upsample', 90 | 'unary', 91 | 'nearest_upsample', 92 | 'unary'] 93 | 94 | ################### 95 | # KPConv parameters 96 | ################### 97 | 98 | # Radius of the input sphere 99 | in_radius = 4.0 100 | val_radius = 4.0 101 | n_frames = 1 102 | max_in_points = 100000 103 | max_val_points = 100000 104 | 105 | # Number of batch 106 | batch_num = 8 107 | val_batch_num = 8 108 | 109 | # Number of kernel points 110 | num_kernel_points = 15 111 | 112 | # Size of the first subsampling grid in meter 113 | first_subsampling_dl = 0.06 114 | 115 | # Radius of convolution in "number grid cell". (2.5 is the standard value) 116 | conv_radius = 2.5 117 | 118 | # Radius of deformable convolution in "number grid cell". Larger so that deformed kernel can spread out 119 | deform_radius = 6.0 120 | 121 | # Radius of the area of influence of each kernel point in "number grid cell". (1.0 is the standard value) 122 | KP_extent = 1.2 123 | 124 | # Behavior of convolutions in ('constant', 'linear', 'gaussian') 125 | KP_influence = 'linear' 126 | 127 | # Aggregation function of KPConv in ('closest', 'sum') 128 | aggregation_mode = 'sum' 129 | 130 | # Choice of input features 131 | first_features_dim = 128 132 | in_features_dim = 2 133 | 134 | # Can the network learn modulations 135 | modulated = False 136 | 137 | # Batch normalization parameters 138 | use_batch_norm = True 139 | batch_norm_momentum = 0.02 140 | 141 | # Deformable offset loss 142 | # 'point2point' fitting geometry by penalizing distance from deform point to input points 143 | # 'point2plane' fitting geometry by penalizing distance from deform point to input point triplet (not implemented) 144 | deform_fitting_mode = 'point2point' 145 | deform_fitting_power = 1.0 # Multiplier for the fitting/repulsive loss 146 | deform_lr_factor = 0.1 # Multiplier for learning rate applied to the deformations 147 | repulse_extent = 1.2 # Distance of repulsion for deformed kernel points 148 | 149 | ##################### 150 | # Training parameters 151 | ##################### 152 | 153 | # Maximal number of epochs 154 | max_epoch = 800 155 | 156 | # Learning rate management 157 | learning_rate = 1e-2 158 | momentum = 0.98 159 | lr_decays = {i: 0.1 ** (1 / 150) for i in range(1, max_epoch)} 160 | grad_clip_norm = 100.0 161 | 162 | # Number of steps per epochs 163 | epoch_steps = 500 164 | 165 | # Number of validation examples per epoch 166 | validation_size = 200 167 | 168 | # Number of epoch between each checkpoint 169 | checkpoint_gap = 50 170 | 171 | # Augmentations 172 | augment_scale_anisotropic = True 173 | augment_symmetries = [True, False, False] 174 | augment_rotation = 'vertical' 175 | augment_scale_min = 0.8 176 | augment_scale_max = 1.2 177 | augment_noise = 0.001 178 | augment_color = 0.8 179 | 180 | # Choose weights for class (used in segmentation loss). Empty list for no weights 181 | # class proportion for R=10.0 and dl=0.08 (first is unlabeled) 182 | # 19.1 48.9 0.5 1.1 5.6 3.6 0.7 0.6 0.9 193.2 17.7 127.4 6.7 132.3 68.4 283.8 7.0 78.5 3.3 0.8 183 | # 184 | # 185 | 186 | # sqrt(Inverse of proportion * 100) 187 | # class_w = [1.430, 14.142, 9.535, 4.226, 5.270, 11.952, 12.910, 10.541, 0.719, 188 | # 2.377, 0.886, 3.863, 0.869, 1.209, 0.594, 3.780, 1.129, 5.505, 11.180] 189 | 190 | # sqrt(Inverse of proportion * 100) capped (0.5 < X < 5) 191 | # class_w = [1.430, 5.000, 5.000, 4.226, 5.000, 5.000, 5.000, 5.000, 0.719, 2.377, 192 | # 0.886, 3.863, 0.869, 1.209, 0.594, 3.780, 1.129, 5.000, 5.000] 193 | 194 | # Do we nee to save convergence 195 | saving = True 196 | saving_path = None 197 | 198 | 199 | # ---------------------------------------------------------------------------------------------------------------------- 200 | # 201 | # Main Call 202 | # \***************/ 203 | # 204 | 205 | if __name__ == '__main__': 206 | 207 | ############################ 208 | # Initialize the environment 209 | ############################ 210 | 211 | # Set which gpu is going to be used 212 | GPU_ID = '0' 213 | 214 | # Set GPU visible device 215 | os.environ['CUDA_VISIBLE_DEVICES'] = GPU_ID 216 | 217 | ############### 218 | # Previous chkp 219 | ############### 220 | 221 | # Choose here if you want to start training from a previous snapshot (None for new training) 222 | # previous_training_path = 'Log_2020-03-19_19-53-27' 223 | previous_training_path = '' 224 | 225 | # Choose index of checkpoint to start from. If None, uses the latest chkp 226 | chkp_idx = None 227 | if previous_training_path: 228 | 229 | # Find all snapshot in the chosen training folder 230 | chkp_path = os.path.join('results', previous_training_path, 'checkpoints') 231 | chkps = [f for f in os.listdir(chkp_path) if f[:4] == 'chkp'] 232 | 233 | # Find which snapshot to restore 234 | if chkp_idx is None: 235 | chosen_chkp = 'current_chkp.tar' 236 | else: 237 | chosen_chkp = np.sort(chkps)[chkp_idx] 238 | chosen_chkp = os.path.join('results', previous_training_path, 'checkpoints', chosen_chkp) 239 | 240 | else: 241 | chosen_chkp = None 242 | 243 | ############## 244 | # Prepare Data 245 | ############## 246 | 247 | print() 248 | print('Data Preparation') 249 | print('****************') 250 | 251 | # Initialize configuration class 252 | config = SemanticKittiConfig() 253 | if previous_training_path: 254 | config.load(os.path.join('results', previous_training_path)) 255 | config.saving_path = None 256 | 257 | # Get path from argument if given 258 | if len(sys.argv) > 1: 259 | config.saving_path = sys.argv[1] 260 | 261 | # Initialize datasets 262 | training_dataset = SemanticKittiDataset(config, set='training', 263 | balance_classes=True) 264 | test_dataset = SemanticKittiDataset(config, set='validation', 265 | balance_classes=False) 266 | 267 | # Initialize samplers 268 | training_sampler = SemanticKittiSampler(training_dataset) 269 | test_sampler = SemanticKittiSampler(test_dataset) 270 | 271 | # Initialize the dataloader 272 | training_loader = DataLoader(training_dataset, 273 | batch_size=1, 274 | sampler=training_sampler, 275 | collate_fn=SemanticKittiCollate, 276 | num_workers=config.input_threads, 277 | pin_memory=True) 278 | test_loader = DataLoader(test_dataset, 279 | batch_size=1, 280 | sampler=test_sampler, 281 | collate_fn=SemanticKittiCollate, 282 | num_workers=config.input_threads, 283 | pin_memory=True) 284 | 285 | # Calibrate max_in_point value 286 | training_sampler.calib_max_in(config, training_loader, verbose=True) 287 | test_sampler.calib_max_in(config, test_loader, verbose=True) 288 | 289 | # Calibrate samplers 290 | training_sampler.calibration(training_loader, verbose=True) 291 | test_sampler.calibration(test_loader, verbose=True) 292 | 293 | # debug_timing(training_dataset, training_loader) 294 | # debug_timing(test_dataset, test_loader) 295 | # debug_class_w(training_dataset, training_loader) 296 | 297 | print('\nModel Preparation') 298 | print('*****************') 299 | 300 | # Define network model 301 | t1 = time.time() 302 | net = KPFCNN(config, training_dataset.label_values, training_dataset.ignored_labels) 303 | 304 | debug = False 305 | if debug: 306 | print('\n*************************************\n') 307 | print(net) 308 | print('\n*************************************\n') 309 | for param in net.parameters(): 310 | if param.requires_grad: 311 | print(param.shape) 312 | print('\n*************************************\n') 313 | print("Model size %i" % sum(param.numel() for param in net.parameters() if param.requires_grad)) 314 | print('\n*************************************\n') 315 | 316 | # Define a trainer class 317 | trainer = ModelTrainer(net, config, chkp_path=chosen_chkp) 318 | print('Done in {:.1f}s\n'.format(time.time() - t1)) 319 | 320 | print('\nStart training') 321 | print('**************') 322 | 323 | # Training 324 | trainer.train(net, training_loader, test_loader, config) 325 | 326 | print('Forcing exit now') 327 | os.kill(os.getpid(), signal.SIGINT) 328 | -------------------------------------------------------------------------------- /train_SensatUrban.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Callable script to start a training on SensatUrban dataset 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Bene Köhler - 07/05/2024 15 | # 16 | 17 | 18 | # ---------------------------------------------------------------------------------------------------------------------- 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | # Common libs 25 | import signal 26 | import os 27 | 28 | # Dataset 29 | from datasets.SensatUrban import * 30 | from torch.utils.data import DataLoader 31 | 32 | from utils.config import Config 33 | from utils.trainer import ModelTrainer 34 | from models.architectures import KPFCNN 35 | 36 | 37 | # ---------------------------------------------------------------------------------------------------------------------- 38 | # 39 | # Config Class 40 | # \******************/ 41 | # 42 | 43 | 44 | class SensatUrbanConfig(Config): 45 | """ 46 | Override the parameters you want to modify for this dataset 47 | """ 48 | 49 | #################### 50 | # Dataset parameters 51 | #################### 52 | 53 | # Dataset name 54 | dataset = "SensatUrban" 55 | 56 | # Number of classes in the dataset (This value is overwritten by dataset class when Initializating dataset). 57 | num_classes = None 58 | 59 | # Type of task performed on this dataset (also overwritten) 60 | dataset_task = "segmentation" 61 | 62 | # Number of CPU threads for the input pipeline 63 | input_threads = 10 64 | 65 | ######################### 66 | # Architecture definition 67 | ######################### 68 | 69 | # # Define layers 70 | architecture = [ 71 | "simple", 72 | "resnetb", 73 | "resnetb_strided", 74 | "resnetb", 75 | "resnetb", 76 | "resnetb_strided", 77 | "resnetb", 78 | "resnetb", 79 | "resnetb_strided", 80 | "resnetb_deformable", 81 | "resnetb_deformable", 82 | "resnetb_deformable_strided", 83 | "resnetb_deformable", 84 | "resnetb_deformable", 85 | "nearest_upsample", 86 | "unary", 87 | "nearest_upsample", 88 | "unary", 89 | "nearest_upsample", 90 | "unary", 91 | "nearest_upsample", 92 | "unary", 93 | ] 94 | 95 | # Define layers 96 | # architecture = ['simple', 97 | # 'resnetb', 98 | # 'resnetb_strided', 99 | # 'resnetb', 100 | # 'resnetb', 101 | # 'resnetb_strided', 102 | # 'resnetb', 103 | # 'resnetb', 104 | # 'resnetb_strided', 105 | # 'resnetb', 106 | # 'resnetb', 107 | # 'resnetb_strided', 108 | # 'resnetb', 109 | # 'resnetb', 110 | # 'nearest_upsample', 111 | # 'unary', 112 | # 'nearest_upsample', 113 | # 'unary', 114 | # 'nearest_upsample', 115 | # 'unary', 116 | # 'nearest_upsample', 117 | # 'unary'] 118 | 119 | ################### 120 | # KPConv parameters 121 | ################### 122 | 123 | # Number of kernel points 124 | num_kernel_points = 15 125 | 126 | # Radius of the input sphere (decrease value to reduce memory cost) 127 | in_radius = 4.0 128 | 129 | # Size of the first subsampling grid in meter (increase value to reduce memory cost) 130 | first_subsampling_dl = 0.2 131 | 132 | # Radius of convolution in "number grid cell". (2.5 is the standard value) 133 | conv_radius = 2.5 134 | 135 | # Radius of deformable convolution in "number grid cell". Larger so that deformed kernel can spread out 136 | deform_radius = 6.0 137 | 138 | # Radius of the area of influence of each kernel point in "number grid cell". (1.0 is the standard value) 139 | KP_extent = 1.2 140 | 141 | # Behavior of convolutions in ('constant', 'linear', 'gaussian') 142 | KP_influence = "linear" 143 | 144 | # Aggregation function of KPConv in ('closest', 'sum') 145 | aggregation_mode = "sum" 146 | 147 | # Choice of input features 148 | first_features_dim = 128 149 | in_features_dim = 5 150 | 151 | # Can the network learn modulations 152 | modulated = False 153 | 154 | # Batch normalization parameters 155 | use_batch_norm = True 156 | batch_norm_momentum = 0.02 157 | 158 | # Deformable offset loss 159 | # 'point2point' fitting geometry by penalizing distance from deform point to input points 160 | # 'point2plane' fitting geometry by penalizing distance from deform point to input point triplet (not implemented) 161 | deform_fitting_mode = "point2point" 162 | deform_fitting_power = 1.0 # Multiplier for the fitting/repulsive loss 163 | deform_lr_factor = 0.1 # Multiplier for learning rate applied to the deformations 164 | repulse_extent = 1.2 # Distance of repulsion for deformed kernel points 165 | 166 | ##################### 167 | # Training parameters 168 | ##################### 169 | 170 | # Maximal number of epochs 171 | max_epoch = 500 172 | 173 | # Learning rate management 174 | learning_rate = 1e-2 175 | momentum = 0.98 176 | lr_decays = {i: 0.1 ** (1 / 150) for i in range(1, max_epoch)} 177 | grad_clip_norm = 100.0 178 | 179 | # Number of batch (decrease to reduce memory cost, but it should remain > 3 for stability) 180 | batch_num = 6 181 | 182 | # Number of steps per epochs 183 | epoch_steps = 500 184 | 185 | # Number of validation examples per epoch 186 | validation_size = 50 187 | 188 | # Number of epoch between each checkpoint 189 | checkpoint_gap = 50 190 | 191 | # Augmentations 192 | augment_scale_anisotropic = True 193 | augment_symmetries = [True, False, False] 194 | augment_rotation = "vertical" 195 | augment_scale_min = 0.9 196 | augment_scale_max = 1.1 197 | augment_noise = 0.001 198 | augment_color = 0.8 199 | 200 | # The way we balance segmentation loss 201 | # > 'none': Each point in the whole batch has the same contribution. 202 | # > 'class': Each class has the same contribution (points are weighted according to class balance) 203 | # > 'batch': Each cloud in the batch has the same contribution (points are weighted according cloud sizes) 204 | segloss_balance = "class" 205 | proportions = [0.2028088885, 206 | 0.2513084539, 207 | 0.3979947284, 208 | 0.0095143598, 209 | 0.0014058568, 210 | 0.0228332248, 211 | 0.0002303890, 212 | 0.0607027858, 213 | 0.0123139005, 214 | 0.0173409825, 215 | 0.0201929531, 216 | 0.0000803975, 217 | 0.0032730794] 218 | class_w = np.sqrt([1.0 / p for p in proportions]) 219 | 220 | # Do we nee to save convergence 221 | saving = True 222 | saving_path = None 223 | 224 | 225 | # ---------------------------------------------------------------------------------------------------------------------- 226 | # 227 | # Main Call 228 | # \***************/ 229 | # 230 | 231 | if __name__ == "__main__": 232 | 233 | ############################ 234 | # Initialize the environment 235 | ############################ 236 | 237 | # Set which gpu is going to be used 238 | GPU_ID = "1" 239 | 240 | # Set GPU visible device 241 | os.environ["CUDA_VISIBLE_DEVICES"] = GPU_ID 242 | device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') 243 | print("Device: %s" % device) 244 | 245 | ############### 246 | # Previous chkp 247 | ############### 248 | 249 | # Choose here if you want to start training from a previous snapshot (None for new training) 250 | # previous_training_path = 'Log_2024-06-21_09-09-55' 251 | previous_training_path = None 252 | 253 | # Choose index of checkpoint to start from. If None, uses the latest chkp 254 | chkp_idx = None 255 | if previous_training_path: 256 | 257 | # Find all snapshot in the chosen training folder 258 | chkp_path = os.path.join("results", previous_training_path, "checkpoints") 259 | chkps = [f for f in os.listdir(chkp_path) if f[:4] == "chkp"] 260 | 261 | # Find which snapshot to restore 262 | if chkp_idx is None: 263 | chosen_chkp = "current_chkp.tar" 264 | else: 265 | chosen_chkp = np.sort(chkps)[chkp_idx] 266 | chosen_chkp = os.path.join( 267 | "results", previous_training_path, "checkpoints", chosen_chkp 268 | ) 269 | 270 | else: 271 | chosen_chkp = None 272 | 273 | ############## 274 | # Prepare Data 275 | ############## 276 | 277 | print() 278 | print("Data Preparation") 279 | print("****************") 280 | 281 | # Initialize configuration class 282 | config = SensatUrbanConfig() 283 | if previous_training_path: 284 | config.load(os.path.join("results", previous_training_path)) 285 | config.saving_path = None 286 | 287 | # Get path from argument if given 288 | if len(sys.argv) > 1: 289 | config.saving_path = sys.argv[1] 290 | 291 | # Initialize datasets 292 | training_dataset = SensatUrbanDataset(config, set="training", use_potentials=True) 293 | test_dataset = SensatUrbanDataset(config, set="validation", use_potentials=True) 294 | 295 | # Initialize samplers 296 | training_sampler = SensatUrbanSampler(training_dataset) 297 | test_sampler = SensatUrbanSampler(test_dataset) 298 | 299 | # Initialize the dataloader 300 | training_loader = DataLoader( 301 | training_dataset, 302 | batch_size=1, 303 | sampler=training_sampler, 304 | collate_fn=SensatUrbanCollate, 305 | num_workers=config.input_threads, 306 | pin_memory=True, 307 | ) 308 | test_loader = DataLoader( 309 | test_dataset, 310 | batch_size=1, 311 | sampler=test_sampler, 312 | collate_fn=SensatUrbanCollate, 313 | num_workers=config.input_threads, 314 | pin_memory=True, 315 | ) 316 | 317 | # Calibrate samplers 318 | training_sampler.calibration(training_loader, verbose=True) 319 | test_sampler.calibration(test_loader, verbose=True) 320 | 321 | # Optional debug functions 322 | # debug_timing(training_dataset, training_loader) 323 | # debug_timing(test_dataset, test_loader) 324 | # debug_upsampling(training_dataset, training_loader) 325 | 326 | print("\nModel Preparation") 327 | print("*****************") 328 | 329 | # Define network model 330 | t1 = time.time() 331 | net = KPFCNN(config, training_dataset.label_values, training_dataset.ignored_labels) 332 | 333 | debug = True 334 | if debug: 335 | print("\n*************************************\n") 336 | print(net) 337 | print("\n*************************************\n") 338 | for param in net.parameters(): 339 | if param.requires_grad: 340 | print(param.shape) 341 | print("\n*************************************\n") 342 | print( 343 | "Model size %i" 344 | % sum(param.numel() for param in net.parameters() if param.requires_grad) 345 | ) 346 | print("\n*************************************\n") 347 | 348 | # Define a trainer class 349 | trainer = ModelTrainer(net, config, chkp_path=chosen_chkp) 350 | print("Done in {:.1f}s\n".format(time.time() - t1)) 351 | 352 | print("\nStart training") 353 | print("**************") 354 | 355 | # Training 356 | trainer.train(net, training_loader, test_loader, config) 357 | 358 | print("Forcing exit now") 359 | os.kill(os.getpid(), signal.SIGINT) 360 | -------------------------------------------------------------------------------- /train_Toronto3D.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Callable script to start a training on Toronto3D dataset 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Anass YARROUDH - 20/08/2023 15 | # 16 | 17 | 18 | # ---------------------------------------------------------------------------------------------------------------------- 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | # Common libs 25 | import signal 26 | import os 27 | 28 | # Dataset 29 | from datasets.Toronto3D import * 30 | from torch.utils.data import DataLoader 31 | 32 | from utils.config import Config 33 | from utils.trainer import ModelTrainer 34 | from models.architectures import KPFCNN 35 | 36 | 37 | # ---------------------------------------------------------------------------------------------------------------------- 38 | # 39 | # Config Class 40 | # \******************/ 41 | # 42 | 43 | class Toronto3DConfig(Config): 44 | """ 45 | Override the parameters you want to modify for this dataset 46 | """ 47 | 48 | #################### 49 | # Dataset parameters 50 | #################### 51 | 52 | # Dataset name 53 | dataset = 'Toronto3D' 54 | 55 | # Number of classes in the dataset (This value is overwritten by dataset class when Initializating dataset). 56 | num_classes = None 57 | 58 | # Type of task performed on this dataset (also overwritten) 59 | dataset_task = '' 60 | 61 | # Number of CPU threads for the input pipeline 62 | input_threads = 20 63 | 64 | ######################### 65 | # Architecture definition 66 | ######################### 67 | 68 | # # Define layers 69 | architecture = ['simple', 70 | 'resnetb', 71 | 'resnetb_strided', 72 | 'resnetb', 73 | 'resnetb', 74 | 'resnetb_strided', 75 | 'resnetb', 76 | 'resnetb', 77 | 'resnetb_strided', 78 | 'resnetb', 79 | 'resnetb', 80 | 'resnetb_strided', 81 | 'resnetb', 82 | 'resnetb', 83 | 'nearest_upsample', 84 | 'unary', 85 | 'nearest_upsample', 86 | 'unary', 87 | 'nearest_upsample', 88 | 'unary', 89 | 'nearest_upsample', 90 | 'unary'] 91 | 92 | ################### 93 | # KPConv parameters 94 | ################### 95 | 96 | # Number of kernel points 97 | num_kernel_points = 15 98 | 99 | # Radius of the input sphere (decrease value to reduce memory cost) 100 | in_radius = 3.0 101 | 102 | # Size of the first subsampling grid in meter (increase value to reduce memory cost) 103 | first_subsampling_dl = 0.08 104 | 105 | # Radius of convolution in "number grid cell". (2.5 is the standard value) 106 | conv_radius = 2.5 107 | 108 | # Radius of deformable convolution in "number grid cell". Larger so that deformed kernel can spread out 109 | deform_radius = 5.0 110 | 111 | # Radius of the area of influence of each kernel point in "number grid cell". (1.0 is the standard value) 112 | KP_extent = 1.0 113 | 114 | # Behavior of convolutions in ('constant', 'linear', 'gaussian') 115 | KP_influence = 'linear' 116 | 117 | # Aggregation function of KPConv in ('closest', 'sum') 118 | aggregation_mode = 'closest' 119 | 120 | # Choice of input features 121 | first_features_dim = 128 122 | in_features_dim = 4 123 | 124 | # Can the network learn modulations 125 | modulated = False 126 | 127 | # Batch normalization parameters 128 | use_batch_norm = True 129 | batch_norm_momentum = 0.02 130 | 131 | # Deformable offset loss 132 | # 'point2point' fitting geometry by penalizing distance from deform point to input points 133 | # 'point2plane' fitting geometry by penalizing distance from deform point to input point triplet (not implemented) 134 | deform_fitting_mode = 'point2point' 135 | deform_fitting_power = 1.0 # Multiplier for the fitting/repulsive loss 136 | deform_lr_factor = 0.1 # Multiplier for learning rate applied to the deformations 137 | repulse_extent = 1.2 # Distance of repulsion for deformed kernel points 138 | 139 | ##################### 140 | # Training parameters 141 | ##################### 142 | 143 | # Maximal number of epochs 144 | max_epoch = 400 145 | 146 | # Learning rate management 147 | learning_rate = 1e-2 148 | momentum = 0.98 149 | lr_decays = {i: 0.1 ** (1 / 150) for i in range(1, max_epoch)} 150 | grad_clip_norm = 100.0 151 | 152 | # Number of batch (decrease to reduce memory cost, but it should remain > 3 for stability) 153 | batch_num = 4 154 | 155 | # Number of steps per epochs 156 | epoch_steps = 500 157 | 158 | # Number of validation examples per epoch 159 | validation_size = 50 160 | 161 | # Number of epoch between each checkpoint 162 | checkpoint_gap = 50 163 | 164 | # Augmentations 165 | augment_scale_anisotropic = True 166 | augment_symmetries = [True, False, False] 167 | augment_rotation = 'vertical' 168 | augment_scale_min = 0.9 169 | augment_scale_max = 1.1 170 | augment_noise = 0.001 171 | augment_color = 0.8 172 | 173 | # The way we balance segmentation loss 174 | # > 'none': Each point in the whole batch has the same contribution. 175 | # > 'class': Each class has the same contribution (points are weighted according to class balance) 176 | # > 'batch': Each cloud in the batch has the same contribution (points are weighted according cloud sizes) 177 | segloss_balance = 'none' 178 | 179 | # Do we need to save convergence 180 | saving = True 181 | saving_path = None 182 | 183 | 184 | # ---------------------------------------------------------------------------------------------------------------------- 185 | # 186 | # Main Call 187 | # \***************/ 188 | # 189 | 190 | if __name__ == '__main__': 191 | 192 | ############################ 193 | # Initialize the environment 194 | ############################ 195 | 196 | start = time.time() 197 | 198 | # Set which gpu is going to be used 199 | GPU_ID = '0' 200 | 201 | # Set GPU visible device 202 | os.environ['CUDA_VISIBLE_DEVICES'] = GPU_ID 203 | 204 | ############### 205 | # Previous chkp 206 | ############### 207 | 208 | # Choose here if you want to start training from a previous snapshot (None for new training) 209 | previous_training_path = '' 210 | 211 | # Choose index of checkpoint to start from. If None, uses the latest chkp 212 | chkp_idx = None 213 | if previous_training_path: 214 | 215 | # Find all snapshot in the chosen training folder 216 | chkp_path = os.path.join('results/Toronto3D', previous_training_path, 'checkpoints') 217 | chkps = [f for f in os.listdir(chkp_path) if f[:4] == 'chkp'] 218 | 219 | # Find which snapshot to restore 220 | if chkp_idx is None: 221 | chosen_chkp = 'current_chkp.tar' 222 | else: 223 | chosen_chkp = np.sort(chkps)[chkp_idx] 224 | chosen_chkp = os.path.join('results/Toronto3D', previous_training_path, 'checkpoints', chosen_chkp) 225 | 226 | else: 227 | chosen_chkp = None 228 | 229 | ############## 230 | # Prepare Data 231 | ############## 232 | 233 | print() 234 | print('Data Preparation') 235 | print('****************') 236 | 237 | # Initialize configuration class 238 | config = Toronto3DConfig() 239 | if previous_training_path: 240 | config.load(os.path.join('results/Toronto3D', previous_training_path)) 241 | config.saving_path = None 242 | 243 | # Get path from argument if given 244 | if len(sys.argv) > 1: 245 | config.saving_path = sys.argv[1] 246 | 247 | # Initialize datasets 248 | training_dataset = Toronto3DDataset(config, set='training', use_potentials=True) 249 | test_dataset = Toronto3DDataset(config, set='validation', use_potentials=True) 250 | 251 | # Initialize samplers 252 | training_sampler = Toronto3DSampler(training_dataset) 253 | test_sampler = Toronto3DSampler(test_dataset) 254 | 255 | # Initialize the dataloader 256 | training_loader = DataLoader(training_dataset, 257 | batch_size=1, 258 | sampler=training_sampler, 259 | collate_fn=Toronto3DCollate, 260 | num_workers=config.input_threads, 261 | pin_memory=True) 262 | test_loader = DataLoader(test_dataset, 263 | batch_size=1, 264 | sampler=test_sampler, 265 | collate_fn=Toronto3DCollate, 266 | num_workers=config.input_threads, 267 | pin_memory=True) 268 | 269 | # Calibrate samplers 270 | training_sampler.calibration(training_loader, verbose=True) 271 | test_sampler.calibration(test_loader, verbose=True) 272 | 273 | # Optional debug functions 274 | # debug_timing(training_dataset, training_loader) 275 | # debug_timing(test_dataset, test_loader) 276 | # debug_upsampling(training_dataset, training_loader) 277 | 278 | print('\nModel Preparation') 279 | print('*****************') 280 | 281 | # Define network model 282 | t1 = time.time() 283 | net = KPFCNN(config, training_dataset.label_values, training_dataset.ignored_labels) 284 | 285 | debug = False 286 | if debug: 287 | print('\n*************************************\n') 288 | print(net) 289 | print('\n*************************************\n') 290 | for param in net.parameters(): 291 | if param.requires_grad: 292 | print(param.shape) 293 | print('\n*************************************\n') 294 | print("Model size %i" % sum(param.numel() for param in net.parameters() if param.requires_grad)) 295 | print('\n*************************************\n') 296 | 297 | # Define a trainer class 298 | trainer = ModelTrainer(net, config, chkp_path=chosen_chkp) 299 | print('Done in {:.1f}s\n'.format(time.time() - t1)) 300 | 301 | print('\nStart training') 302 | print('**************') 303 | 304 | # Training 305 | trainer.train(net, training_loader, test_loader, config) 306 | 307 | print('Forcing exit now') 308 | os.kill(os.getpid(), signal.SIGINT) 309 | 310 | end = time.time() 311 | print(time.strftime("%H:%M:%S", time.gmtime(end - start))) 312 | -------------------------------------------------------------------------------- /utils/config.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Configuration class 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 11/06/2018 15 | # 16 | 17 | 18 | from os.path import join 19 | import numpy as np 20 | 21 | 22 | # Colors for printing 23 | class bcolors: 24 | HEADER = '\033[95m' 25 | OKBLUE = '\033[94m' 26 | OKGREEN = '\033[92m' 27 | WARNING = '\033[93m' 28 | FAIL = '\033[91m' 29 | ENDC = '\033[0m' 30 | BOLD = '\033[1m' 31 | UNDERLINE = '\033[4m' 32 | 33 | 34 | class Config: 35 | """ 36 | Class containing the parameters you want to modify for this dataset 37 | """ 38 | 39 | ################## 40 | # Input parameters 41 | ################## 42 | 43 | # Dataset name 44 | dataset = '' 45 | 46 | # Type of network model 47 | dataset_task = '' 48 | 49 | # Number of classes in the dataset 50 | num_classes = 0 51 | 52 | # Dimension of input points 53 | in_points_dim = 3 54 | 55 | # Dimension of input features 56 | in_features_dim = 1 57 | 58 | # Radius of the input sphere (ignored for models, only used for point clouds) 59 | in_radius = 1.0 60 | 61 | # Number of CPU threads for the input pipeline 62 | input_threads = 8 63 | 64 | ################## 65 | # Model parameters 66 | ################## 67 | 68 | # Architecture definition. List of blocks 69 | architecture = [] 70 | 71 | # Decide the mode of equivariance and invariance 72 | equivar_mode = '' 73 | invar_mode = '' 74 | 75 | # Dimension of the first feature maps 76 | first_features_dim = 64 77 | 78 | # Batch normalization parameters 79 | use_batch_norm = True 80 | batch_norm_momentum = 0.99 81 | 82 | # For segmentation models : ratio between the segmented area and the input area 83 | segmentation_ratio = 1.0 84 | 85 | ################### 86 | # KPConv parameters 87 | ################### 88 | 89 | # Number of kernel points 90 | num_kernel_points = 15 91 | 92 | # Size of the first subsampling grid in meter 93 | first_subsampling_dl = 0.02 94 | 95 | # Radius of convolution in "number grid cell". (2.5 is the standard value) 96 | conv_radius = 2.5 97 | 98 | # Radius of deformable convolution in "number grid cell". Larger so that deformed kernel can spread out 99 | deform_radius = 5.0 100 | 101 | # Kernel point influence radius 102 | KP_extent = 1.0 103 | 104 | # Influence function when d < KP_extent. ('constant', 'linear', 'gaussian') When d > KP_extent, always zero 105 | KP_influence = 'linear' 106 | 107 | # Aggregation function of KPConv in ('closest', 'sum') 108 | # Decide if you sum all kernel point influences, or if you only take the influence of the closest KP 109 | aggregation_mode = 'sum' 110 | 111 | # Fixed points in the kernel : 'none', 'center' or 'verticals' 112 | fixed_kernel_points = 'center' 113 | 114 | # Use modulateion in deformable convolutions 115 | modulated = False 116 | 117 | # For SLAM datasets like SemanticKitti number of frames used (minimum one) 118 | n_frames = 1 119 | 120 | # For SLAM datasets like SemanticKitti max number of point in input cloud + validation 121 | max_in_points = 0 122 | val_radius = 51.0 123 | max_val_points = 50000 124 | 125 | ##################### 126 | # Training parameters 127 | ##################### 128 | 129 | # Network optimizer parameters (learning rate and momentum) 130 | learning_rate = 1e-3 131 | momentum = 0.9 132 | 133 | # Learning rate decays. Dictionary of all decay values with their epoch {epoch: decay}. 134 | lr_decays = {200: 0.2, 300: 0.2} 135 | 136 | # Gradient clipping value (negative means no clipping) 137 | grad_clip_norm = 100.0 138 | 139 | # Augmentation parameters 140 | augment_scale_anisotropic = True 141 | augment_scale_min = 0.9 142 | augment_scale_max = 1.1 143 | augment_symmetries = [False, False, False] 144 | augment_rotation = 'vertical' 145 | augment_noise = 0.005 146 | augment_color = 0.7 147 | 148 | # Augment with occlusions (not implemented yet) 149 | augment_occlusion = 'none' 150 | augment_occlusion_ratio = 0.2 151 | augment_occlusion_num = 1 152 | 153 | # Regularization loss importance 154 | weight_decay = 1e-3 155 | 156 | # The way we balance segmentation loss DEPRECATED 157 | segloss_balance = 'none' 158 | 159 | # Choose weights for class (used in segmentation loss). Empty list for no weights 160 | class_w = [] 161 | 162 | # Deformable offset loss 163 | # 'point2point' fitting geometry by penalizing distance from deform point to input points 164 | # 'point2plane' fitting geometry by penalizing distance from deform point to input point triplet (not implemented) 165 | deform_fitting_mode = 'point2point' 166 | deform_fitting_power = 1.0 # Multiplier for the fitting/repulsive loss 167 | deform_lr_factor = 0.1 # Multiplier for learning rate applied to the deformations 168 | repulse_extent = 1.0 # Distance of repulsion for deformed kernel points 169 | 170 | # Number of batch 171 | batch_num = 10 172 | val_batch_num = 10 173 | 174 | # Maximal number of epochs 175 | max_epoch = 1000 176 | 177 | # Number of steps per epochs 178 | epoch_steps = 1000 179 | 180 | # Number of validation examples per epoch 181 | validation_size = 100 182 | 183 | # Number of epoch between each checkpoint 184 | checkpoint_gap = 50 185 | 186 | # Do we nee to save convergence 187 | saving = True 188 | saving_path = None 189 | 190 | def __init__(self): 191 | """ 192 | Class Initialyser 193 | """ 194 | 195 | # Number of layers 196 | self.num_layers = len([block for block in self.architecture if 'pool' in block or 'strided' in block]) + 1 197 | 198 | ################### 199 | # Deform layer list 200 | ################### 201 | # 202 | # List of boolean indicating which layer has a deformable convolution 203 | # 204 | 205 | layer_blocks = [] 206 | self.deform_layers = [] 207 | arch = self.architecture 208 | for block_i, block in enumerate(arch): 209 | 210 | # Get all blocks of the layer 211 | if not ('pool' in block or 'strided' in block or 'global' in block or 'upsample' in block): 212 | layer_blocks += [block] 213 | continue 214 | 215 | # Convolution neighbors indices 216 | # ***************************** 217 | 218 | deform_layer = False 219 | if layer_blocks: 220 | if np.any(['deformable' in blck for blck in layer_blocks]): 221 | deform_layer = True 222 | 223 | if 'pool' in block or 'strided' in block: 224 | if 'deformable' in block: 225 | deform_layer = True 226 | 227 | self.deform_layers += [deform_layer] 228 | layer_blocks = [] 229 | 230 | # Stop when meeting a global pooling or upsampling 231 | if 'global' in block or 'upsample' in block: 232 | break 233 | 234 | def load(self, path): 235 | 236 | filename = join(path, 'parameters.txt') 237 | with open(filename, 'r') as f: 238 | lines = f.readlines() 239 | 240 | # Class variable dictionary 241 | for line in lines: 242 | line_info = line.split() 243 | if len(line_info) > 2 and line_info[0] != '#': 244 | 245 | if line_info[2] == 'None': 246 | setattr(self, line_info[0], None) 247 | 248 | elif line_info[0] == 'lr_decay_epochs': 249 | self.lr_decays = {int(b.split(':')[0]): float(b.split(':')[1]) for b in line_info[2:]} 250 | 251 | elif line_info[0] == 'architecture': 252 | self.architecture = [b for b in line_info[2:]] 253 | 254 | elif line_info[0] == 'augment_symmetries': 255 | self.augment_symmetries = [bool(int(b)) for b in line_info[2:]] 256 | 257 | elif line_info[0] == 'num_classes': 258 | if len(line_info) > 3: 259 | self.num_classes = [int(c) for c in line_info[2:]] 260 | else: 261 | self.num_classes = int(line_info[2]) 262 | 263 | elif line_info[0] == 'class_w': 264 | self.class_w = [float(w) for w in line_info[2:]] 265 | 266 | elif hasattr(self, line_info[0]): 267 | attr_type = type(getattr(self, line_info[0])) 268 | if attr_type == bool: 269 | setattr(self, line_info[0], attr_type(int(line_info[2]))) 270 | else: 271 | setattr(self, line_info[0], attr_type(line_info[2])) 272 | 273 | self.saving = True 274 | self.saving_path = path 275 | self.__init__() 276 | 277 | def save(self): 278 | 279 | with open(join(self.saving_path, 'parameters.txt'), "w") as text_file: 280 | 281 | text_file.write('# -----------------------------------#\n') 282 | text_file.write('# Parameters of the training session #\n') 283 | text_file.write('# -----------------------------------#\n\n') 284 | 285 | # Input parameters 286 | text_file.write('# Input parameters\n') 287 | text_file.write('# ****************\n\n') 288 | text_file.write('dataset = {:s}\n'.format(self.dataset)) 289 | text_file.write('dataset_task = {:s}\n'.format(self.dataset_task)) 290 | if type(self.num_classes) is list: 291 | text_file.write('num_classes =') 292 | for n in self.num_classes: 293 | text_file.write(' {:d}'.format(n)) 294 | text_file.write('\n') 295 | else: 296 | text_file.write('num_classes = {:d}\n'.format(self.num_classes)) 297 | text_file.write('in_points_dim = {:d}\n'.format(self.in_points_dim)) 298 | text_file.write('in_features_dim = {:d}\n'.format(self.in_features_dim)) 299 | text_file.write('in_radius = {:.6f}\n'.format(self.in_radius)) 300 | text_file.write('input_threads = {:d}\n\n'.format(self.input_threads)) 301 | 302 | # Model parameters 303 | text_file.write('# Model parameters\n') 304 | text_file.write('# ****************\n\n') 305 | 306 | text_file.write('architecture =') 307 | for a in self.architecture: 308 | text_file.write(' {:s}'.format(a)) 309 | text_file.write('\n') 310 | text_file.write('equivar_mode = {:s}\n'.format(self.equivar_mode)) 311 | text_file.write('invar_mode = {:s}\n'.format(self.invar_mode)) 312 | text_file.write('num_layers = {:d}\n'.format(self.num_layers)) 313 | text_file.write('first_features_dim = {:d}\n'.format(self.first_features_dim)) 314 | text_file.write('use_batch_norm = {:d}\n'.format(int(self.use_batch_norm))) 315 | text_file.write('batch_norm_momentum = {:.6f}\n\n'.format(self.batch_norm_momentum)) 316 | text_file.write('segmentation_ratio = {:.6f}\n\n'.format(self.segmentation_ratio)) 317 | 318 | # KPConv parameters 319 | text_file.write('# KPConv parameters\n') 320 | text_file.write('# *****************\n\n') 321 | 322 | text_file.write('first_subsampling_dl = {:.6f}\n'.format(self.first_subsampling_dl)) 323 | text_file.write('num_kernel_points = {:d}\n'.format(self.num_kernel_points)) 324 | text_file.write('conv_radius = {:.6f}\n'.format(self.conv_radius)) 325 | text_file.write('deform_radius = {:.6f}\n'.format(self.deform_radius)) 326 | text_file.write('fixed_kernel_points = {:s}\n'.format(self.fixed_kernel_points)) 327 | text_file.write('KP_extent = {:.6f}\n'.format(self.KP_extent)) 328 | text_file.write('KP_influence = {:s}\n'.format(self.KP_influence)) 329 | text_file.write('aggregation_mode = {:s}\n'.format(self.aggregation_mode)) 330 | text_file.write('modulated = {:d}\n'.format(int(self.modulated))) 331 | text_file.write('n_frames = {:d}\n'.format(self.n_frames)) 332 | text_file.write('max_in_points = {:d}\n\n'.format(self.max_in_points)) 333 | text_file.write('max_val_points = {:d}\n\n'.format(self.max_val_points)) 334 | text_file.write('val_radius = {:.6f}\n\n'.format(self.val_radius)) 335 | 336 | # Training parameters 337 | text_file.write('# Training parameters\n') 338 | text_file.write('# *******************\n\n') 339 | 340 | text_file.write('learning_rate = {:f}\n'.format(self.learning_rate)) 341 | text_file.write('momentum = {:f}\n'.format(self.momentum)) 342 | text_file.write('lr_decay_epochs =') 343 | for e, d in self.lr_decays.items(): 344 | text_file.write(' {:d}:{:f}'.format(e, d)) 345 | text_file.write('\n') 346 | text_file.write('grad_clip_norm = {:f}\n\n'.format(self.grad_clip_norm)) 347 | 348 | 349 | text_file.write('augment_symmetries =') 350 | for a in self.augment_symmetries: 351 | text_file.write(' {:d}'.format(int(a))) 352 | text_file.write('\n') 353 | text_file.write('augment_rotation = {:s}\n'.format(self.augment_rotation)) 354 | text_file.write('augment_noise = {:f}\n'.format(self.augment_noise)) 355 | text_file.write('augment_occlusion = {:s}\n'.format(self.augment_occlusion)) 356 | text_file.write('augment_occlusion_ratio = {:.6f}\n'.format(self.augment_occlusion_ratio)) 357 | text_file.write('augment_occlusion_num = {:d}\n'.format(self.augment_occlusion_num)) 358 | text_file.write('augment_scale_anisotropic = {:d}\n'.format(int(self.augment_scale_anisotropic))) 359 | text_file.write('augment_scale_min = {:.6f}\n'.format(self.augment_scale_min)) 360 | text_file.write('augment_scale_max = {:.6f}\n'.format(self.augment_scale_max)) 361 | text_file.write('augment_color = {:.6f}\n\n'.format(self.augment_color)) 362 | 363 | text_file.write('weight_decay = {:f}\n'.format(self.weight_decay)) 364 | text_file.write('segloss_balance = {:s}\n'.format(self.segloss_balance)) 365 | text_file.write('class_w =') 366 | for a in self.class_w: 367 | text_file.write(' {:.6f}'.format(a)) 368 | text_file.write('\n') 369 | text_file.write('deform_fitting_mode = {:s}\n'.format(self.deform_fitting_mode)) 370 | text_file.write('deform_fitting_power = {:.6f}\n'.format(self.deform_fitting_power)) 371 | text_file.write('deform_lr_factor = {:.6f}\n'.format(self.deform_lr_factor)) 372 | text_file.write('repulse_extent = {:.6f}\n'.format(self.repulse_extent)) 373 | text_file.write('batch_num = {:d}\n'.format(self.batch_num)) 374 | text_file.write('val_batch_num = {:d}\n'.format(self.val_batch_num)) 375 | text_file.write('max_epoch = {:d}\n'.format(self.max_epoch)) 376 | if self.epoch_steps is None: 377 | text_file.write('epoch_steps = None\n') 378 | else: 379 | text_file.write('epoch_steps = {:d}\n'.format(self.epoch_steps)) 380 | text_file.write('validation_size = {:d}\n'.format(self.validation_size)) 381 | text_file.write('checkpoint_gap = {:d}\n'.format(self.checkpoint_gap)) 382 | 383 | -------------------------------------------------------------------------------- /utils/mayavi_visu.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Script for various visualization with mayavi 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 11/06/2018 15 | # 16 | 17 | 18 | # ---------------------------------------------------------------------------------------------------------------------- 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | 25 | # Basic libs 26 | import torch 27 | import numpy as np 28 | from sklearn.neighbors import KDTree 29 | from os import makedirs, remove, rename, listdir 30 | from os.path import exists, join 31 | import time 32 | 33 | import sys 34 | 35 | # PLY reader 36 | from utils.ply import write_ply, read_ply 37 | 38 | # Configuration class 39 | from utils.config import Config 40 | 41 | 42 | def show_ModelNet_models(all_points): 43 | from mayavi import mlab 44 | 45 | ########################### 46 | # Interactive visualization 47 | ########################### 48 | 49 | # Create figure for features 50 | fig1 = mlab.figure('Models', bgcolor=(1, 1, 1), size=(1000, 800)) 51 | fig1.scene.parallel_projection = False 52 | 53 | # Indices 54 | global file_i 55 | file_i = 0 56 | 57 | def update_scene(): 58 | 59 | # clear figure 60 | mlab.clf(fig1) 61 | 62 | # Plot new data feature 63 | points = all_points[file_i] 64 | 65 | # Rescale points for visu 66 | points = (points * 1.5 + np.array([1.0, 1.0, 1.0])) * 50.0 67 | 68 | # Show point clouds colorized with activations 69 | activations = mlab.points3d(points[:, 0], 70 | points[:, 1], 71 | points[:, 2], 72 | points[:, 2], 73 | scale_factor=3.0, 74 | scale_mode='none', 75 | figure=fig1) 76 | 77 | # New title 78 | mlab.title(str(file_i), color=(0, 0, 0), size=0.3, height=0.01) 79 | text = '<--- (press g for previous)' + 50 * ' ' + '(press h for next) --->' 80 | mlab.text(0.01, 0.01, text, color=(0, 0, 0), width=0.98) 81 | mlab.orientation_axes() 82 | 83 | return 84 | 85 | def keyboard_callback(vtk_obj, event): 86 | global file_i 87 | 88 | if vtk_obj.GetKeyCode() in ['g', 'G']: 89 | 90 | file_i = (file_i - 1) % len(all_points) 91 | update_scene() 92 | 93 | elif vtk_obj.GetKeyCode() in ['h', 'H']: 94 | 95 | file_i = (file_i + 1) % len(all_points) 96 | update_scene() 97 | 98 | return 99 | 100 | # Draw a first plot 101 | update_scene() 102 | fig1.scene.interactor.add_observer('KeyPressEvent', keyboard_callback) 103 | mlab.show() 104 | 105 | 106 | def show_ModelNet_examples(clouds, cloud_normals=None, cloud_labels=None): 107 | from mayavi import mlab 108 | 109 | ########################### 110 | # Interactive visualization 111 | ########################### 112 | 113 | # Create figure for features 114 | fig1 = mlab.figure('Models', bgcolor=(1, 1, 1), size=(1000, 800)) 115 | fig1.scene.parallel_projection = False 116 | 117 | if cloud_labels is None: 118 | cloud_labels = [points[:, 2] for points in clouds] 119 | 120 | # Indices 121 | global file_i, show_normals 122 | file_i = 0 123 | show_normals = True 124 | 125 | def update_scene(): 126 | 127 | # clear figure 128 | mlab.clf(fig1) 129 | 130 | # Plot new data feature 131 | points = clouds[file_i] 132 | labels = cloud_labels[file_i] 133 | if cloud_normals is not None: 134 | normals = cloud_normals[file_i] 135 | else: 136 | normals = None 137 | 138 | # Rescale points for visu 139 | points = (points * 1.5 + np.array([1.0, 1.0, 1.0])) * 50.0 140 | 141 | # Show point clouds colorized with activations 142 | activations = mlab.points3d(points[:, 0], 143 | points[:, 1], 144 | points[:, 2], 145 | labels, 146 | scale_factor=3.0, 147 | scale_mode='none', 148 | figure=fig1) 149 | if normals is not None and show_normals: 150 | activations = mlab.quiver3d(points[:, 0], 151 | points[:, 1], 152 | points[:, 2], 153 | normals[:, 0], 154 | normals[:, 1], 155 | normals[:, 2], 156 | scale_factor=10.0, 157 | scale_mode='none', 158 | figure=fig1) 159 | 160 | # New title 161 | mlab.title(str(file_i), color=(0, 0, 0), size=0.3, height=0.01) 162 | text = '<--- (press g for previous)' + 50 * ' ' + '(press h for next) --->' 163 | mlab.text(0.01, 0.01, text, color=(0, 0, 0), width=0.98) 164 | mlab.orientation_axes() 165 | 166 | return 167 | 168 | def keyboard_callback(vtk_obj, event): 169 | global file_i, show_normals 170 | 171 | if vtk_obj.GetKeyCode() in ['g', 'G']: 172 | file_i = (file_i - 1) % len(clouds) 173 | update_scene() 174 | 175 | elif vtk_obj.GetKeyCode() in ['h', 'H']: 176 | file_i = (file_i + 1) % len(clouds) 177 | update_scene() 178 | 179 | elif vtk_obj.GetKeyCode() in ['n', 'N']: 180 | show_normals = not show_normals 181 | update_scene() 182 | 183 | return 184 | 185 | # Draw a first plot 186 | update_scene() 187 | fig1.scene.interactor.add_observer('KeyPressEvent', keyboard_callback) 188 | mlab.show() 189 | 190 | 191 | def show_neighbors(query, supports, neighbors): 192 | from mayavi import mlab 193 | 194 | ########################### 195 | # Interactive visualization 196 | ########################### 197 | 198 | # Create figure for features 199 | fig1 = mlab.figure('Models', bgcolor=(1, 1, 1), size=(1000, 800)) 200 | fig1.scene.parallel_projection = False 201 | 202 | # Indices 203 | global file_i 204 | file_i = 0 205 | 206 | def update_scene(): 207 | 208 | # clear figure 209 | mlab.clf(fig1) 210 | 211 | # Rescale points for visu 212 | p1 = (query * 1.5 + np.array([1.0, 1.0, 1.0])) * 50.0 213 | p2 = (supports * 1.5 + np.array([1.0, 1.0, 1.0])) * 50.0 214 | 215 | l1 = p1[:, 2]*0 216 | l1[file_i] = 1 217 | 218 | l2 = p2[:, 2]*0 + 2 219 | l2[neighbors[file_i]] = 3 220 | 221 | # Show point clouds colorized with activations 222 | activations = mlab.points3d(p1[:, 0], 223 | p1[:, 1], 224 | p1[:, 2], 225 | l1, 226 | scale_factor=2.0, 227 | scale_mode='none', 228 | vmin=0.0, 229 | vmax=3.0, 230 | figure=fig1) 231 | 232 | activations = mlab.points3d(p2[:, 0], 233 | p2[:, 1], 234 | p2[:, 2], 235 | l2, 236 | scale_factor=3.0, 237 | scale_mode='none', 238 | vmin=0.0, 239 | vmax=3.0, 240 | figure=fig1) 241 | 242 | # New title 243 | mlab.title(str(file_i), color=(0, 0, 0), size=0.3, height=0.01) 244 | text = '<--- (press g for previous)' + 50 * ' ' + '(press h for next) --->' 245 | mlab.text(0.01, 0.01, text, color=(0, 0, 0), width=0.98) 246 | mlab.orientation_axes() 247 | 248 | return 249 | 250 | def keyboard_callback(vtk_obj, event): 251 | global file_i 252 | 253 | if vtk_obj.GetKeyCode() in ['g', 'G']: 254 | 255 | file_i = (file_i - 1) % len(query) 256 | update_scene() 257 | 258 | elif vtk_obj.GetKeyCode() in ['h', 'H']: 259 | 260 | file_i = (file_i + 1) % len(query) 261 | update_scene() 262 | 263 | return 264 | 265 | # Draw a first plot 266 | update_scene() 267 | fig1.scene.interactor.add_observer('KeyPressEvent', keyboard_callback) 268 | mlab.show() 269 | 270 | 271 | def show_input_batch(batch): 272 | from mayavi import mlab 273 | 274 | ########################### 275 | # Interactive visualization 276 | ########################### 277 | 278 | # Create figure for features 279 | fig1 = mlab.figure('Input', bgcolor=(1, 1, 1), size=(1000, 800)) 280 | fig1.scene.parallel_projection = False 281 | 282 | # Unstack batch 283 | all_points = batch.unstack_points() 284 | all_neighbors = batch.unstack_neighbors() 285 | all_pools = batch.unstack_pools() 286 | 287 | # Indices 288 | global b_i, l_i, neighb_i, show_pools 289 | b_i = 0 290 | l_i = 0 291 | neighb_i = 0 292 | show_pools = False 293 | 294 | def update_scene(): 295 | 296 | # clear figure 297 | mlab.clf(fig1) 298 | 299 | # Rescale points for visu 300 | p = (all_points[l_i][b_i] * 1.5 + np.array([1.0, 1.0, 1.0])) * 50.0 301 | labels = p[:, 2]*0 302 | 303 | if show_pools: 304 | p2 = (all_points[l_i+1][b_i][neighb_i:neighb_i+1] * 1.5 + np.array([1.0, 1.0, 1.0])) * 50.0 305 | p = np.vstack((p, p2)) 306 | labels = np.hstack((labels, np.ones((1,), dtype=np.int32)*3)) 307 | pool_inds = all_pools[l_i][b_i][neighb_i] 308 | pool_inds = pool_inds[pool_inds >= 0] 309 | labels[pool_inds] = 2 310 | else: 311 | neighb_inds = all_neighbors[l_i][b_i][neighb_i] 312 | neighb_inds = neighb_inds[neighb_inds >= 0] 313 | labels[neighb_inds] = 2 314 | labels[neighb_i] = 3 315 | 316 | # Show point clouds colorized with activations 317 | mlab.points3d(p[:, 0], 318 | p[:, 1], 319 | p[:, 2], 320 | labels, 321 | scale_factor=2.0, 322 | scale_mode='none', 323 | vmin=0.0, 324 | vmax=3.0, 325 | figure=fig1) 326 | 327 | 328 | """ 329 | mlab.points3d(p[-2:, 0], 330 | p[-2:, 1], 331 | p[-2:, 2], 332 | labels[-2:]*0 + 3, 333 | scale_factor=0.16 * 1.5 * 50, 334 | scale_mode='none', 335 | mode='cube', 336 | vmin=0.0, 337 | vmax=3.0, 338 | figure=fig1) 339 | mlab.points3d(p[-1:, 0], 340 | p[-1:, 1], 341 | p[-1:, 2], 342 | labels[-1:]*0 + 2, 343 | scale_factor=0.16 * 2 * 2.5 * 1.5 * 50, 344 | scale_mode='none', 345 | mode='sphere', 346 | vmin=0.0, 347 | vmax=3.0, 348 | figure=fig1) 349 | 350 | """ 351 | 352 | # New title 353 | title_str = '<([) b_i={:d} (])> <(,) l_i={:d} (.)> <(N) n_i={:d} (M)>'.format(b_i, l_i, neighb_i) 354 | mlab.title(title_str, color=(0, 0, 0), size=0.3, height=0.90) 355 | if show_pools: 356 | text = 'pools (switch with G)' 357 | else: 358 | text = 'neighbors (switch with G)' 359 | mlab.text(0.01, 0.01, text, color=(0, 0, 0), width=0.3) 360 | mlab.orientation_axes() 361 | 362 | return 363 | 364 | def keyboard_callback(vtk_obj, event): 365 | global b_i, l_i, neighb_i, show_pools 366 | 367 | if vtk_obj.GetKeyCode() in ['[', '{']: 368 | b_i = (b_i - 1) % len(all_points[l_i]) 369 | neighb_i = 0 370 | update_scene() 371 | 372 | elif vtk_obj.GetKeyCode() in [']', '}']: 373 | b_i = (b_i + 1) % len(all_points[l_i]) 374 | neighb_i = 0 375 | update_scene() 376 | 377 | elif vtk_obj.GetKeyCode() in [',', '<']: 378 | if show_pools: 379 | l_i = (l_i - 1) % (len(all_points) - 1) 380 | else: 381 | l_i = (l_i - 1) % len(all_points) 382 | neighb_i = 0 383 | update_scene() 384 | 385 | elif vtk_obj.GetKeyCode() in ['.', '>']: 386 | if show_pools: 387 | l_i = (l_i + 1) % (len(all_points) - 1) 388 | else: 389 | l_i = (l_i + 1) % len(all_points) 390 | neighb_i = 0 391 | update_scene() 392 | 393 | elif vtk_obj.GetKeyCode() in ['n', 'N']: 394 | neighb_i = (neighb_i - 1) % all_points[l_i][b_i].shape[0] 395 | update_scene() 396 | 397 | elif vtk_obj.GetKeyCode() in ['m', 'M']: 398 | neighb_i = (neighb_i + 1) % all_points[l_i][b_i].shape[0] 399 | update_scene() 400 | 401 | elif vtk_obj.GetKeyCode() in ['g', 'G']: 402 | if l_i < len(all_points) - 1: 403 | show_pools = not show_pools 404 | neighb_i = 0 405 | update_scene() 406 | 407 | return 408 | 409 | # Draw a first plot 410 | update_scene() 411 | fig1.scene.interactor.add_observer('KeyPressEvent', keyboard_callback) 412 | mlab.show() 413 | 414 | 415 | 416 | 417 | 418 | 419 | 420 | 421 | 422 | 423 | 424 | 425 | 426 | 427 | 428 | 429 | 430 | 431 | 432 | 433 | 434 | 435 | 436 | 437 | -------------------------------------------------------------------------------- /utils/metrics.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Metric utility functions 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 11/06/2018 15 | # 16 | 17 | 18 | # ---------------------------------------------------------------------------------------------------------------------- 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | 25 | # Basic libs 26 | import numpy as np 27 | 28 | 29 | # ---------------------------------------------------------------------------------------------------------------------- 30 | # 31 | # Utilities 32 | # \***************/ 33 | # 34 | 35 | def fast_confusion(true, pred, label_values=None): 36 | """ 37 | Fast confusion matrix (100x faster than Scikit learn). But only works if labels are la 38 | :param true: 39 | :param false: 40 | :param num_classes: 41 | :return: 42 | """ 43 | 44 | # Ensure data is in the right format 45 | true = np.squeeze(true) 46 | pred = np.squeeze(pred) 47 | if len(true.shape) != 1: 48 | raise ValueError('Truth values are stored in a {:d}D array instead of 1D array'. format(len(true.shape))) 49 | if len(pred.shape) != 1: 50 | raise ValueError('Prediction values are stored in a {:d}D array instead of 1D array'. format(len(pred.shape))) 51 | if true.dtype not in [np.int32, np.int64]: 52 | raise ValueError('Truth values are {:s} instead of int32 or int64'.format(true.dtype)) 53 | if pred.dtype not in [np.int32, np.int64]: 54 | raise ValueError('Prediction values are {:s} instead of int32 or int64'.format(pred.dtype)) 55 | true = true.astype(np.int32) 56 | pred = pred.astype(np.int32) 57 | 58 | # Get the label values 59 | if label_values is None: 60 | # From data if they are not given 61 | label_values = np.unique(np.hstack((true, pred))) 62 | else: 63 | # Ensure they are good if given 64 | if label_values.dtype not in [np.int32, np.int64]: 65 | raise ValueError('label values are {:s} instead of int32 or int64'.format(label_values.dtype)) 66 | if len(np.unique(label_values)) < len(label_values): 67 | raise ValueError('Given labels are not unique') 68 | 69 | # Sort labels 70 | label_values = np.sort(label_values) 71 | 72 | # Get the number of classes 73 | num_classes = len(label_values) 74 | 75 | #print(num_classes) 76 | #print(label_values) 77 | #print(np.max(true)) 78 | #print(np.max(pred)) 79 | #print(np.max(true * num_classes + pred)) 80 | 81 | # Start confusion computations 82 | if label_values[0] == 0 and label_values[-1] == num_classes - 1: 83 | 84 | # Vectorized confusion 85 | vec_conf = np.bincount(true * num_classes + pred) 86 | 87 | # Add possible missing values due to classes not being in pred or true 88 | #print(vec_conf.shape) 89 | if vec_conf.shape[0] < num_classes ** 2: 90 | vec_conf = np.pad(vec_conf, (0, num_classes ** 2 - vec_conf.shape[0]), 'constant') 91 | #print(vec_conf.shape) 92 | 93 | # Reshape confusion in a matrix 94 | return vec_conf.reshape((num_classes, num_classes)) 95 | 96 | 97 | else: 98 | 99 | # Ensure no negative classes 100 | if label_values[0] < 0: 101 | raise ValueError('Unsupported negative classes') 102 | 103 | # Get the data in [0,num_classes[ 104 | label_map = np.zeros((label_values[-1] + 1,), dtype=np.int32) 105 | for k, v in enumerate(label_values): 106 | label_map[v] = k 107 | 108 | pred = label_map[pred] 109 | true = label_map[true] 110 | 111 | # Vectorized confusion 112 | vec_conf = np.bincount(true * num_classes + pred) 113 | 114 | # Add possible missing values due to classes not being in pred or true 115 | if vec_conf.shape[0] < num_classes ** 2: 116 | vec_conf = np.pad(vec_conf, (0, num_classes ** 2 - vec_conf.shape[0]), 'constant') 117 | 118 | # Reshape confusion in a matrix 119 | return vec_conf.reshape((num_classes, num_classes)) 120 | 121 | def metrics(confusions, ignore_unclassified=False): 122 | """ 123 | Computes different metrics from confusion matrices. 124 | :param confusions: ([..., n_c, n_c] np.int32). Can be any dimension, the confusion matrices should be described by 125 | the last axes. n_c = number of classes 126 | :param ignore_unclassified: (bool). True if the the first class should be ignored in the results 127 | :return: ([..., n_c] np.float32) precision, recall, F1 score, IoU score 128 | """ 129 | 130 | # If the first class (often "unclassified") should be ignored, erase it from the confusion. 131 | if (ignore_unclassified): 132 | confusions[..., 0, :] = 0 133 | confusions[..., :, 0] = 0 134 | 135 | # Compute TP, FP, FN. This assume that the second to last axis counts the truths (like the first axis of a 136 | # confusion matrix), and that the last axis counts the predictions (like the second axis of a confusion matrix) 137 | TP = np.diagonal(confusions, axis1=-2, axis2=-1) 138 | TP_plus_FP = np.sum(confusions, axis=-1) 139 | TP_plus_FN = np.sum(confusions, axis=-2) 140 | 141 | # Compute precision and recall. This assume that the second to last axis counts the truths (like the first axis of 142 | # a confusion matrix), and that the last axis counts the predictions (like the second axis of a confusion matrix) 143 | PRE = TP / (TP_plus_FN + 1e-6) 144 | REC = TP / (TP_plus_FP + 1e-6) 145 | 146 | # Compute Accuracy 147 | ACC = np.sum(TP, axis=-1) / (np.sum(confusions, axis=(-2, -1)) + 1e-6) 148 | 149 | # Compute F1 score 150 | F1 = 2 * TP / (TP_plus_FP + TP_plus_FN + 1e-6) 151 | 152 | # Compute IoU 153 | IoU = F1 / (2 - F1) 154 | 155 | return PRE, REC, F1, IoU, ACC 156 | 157 | 158 | def smooth_metrics(confusions, smooth_n=0, ignore_unclassified=False): 159 | """ 160 | Computes different metrics from confusion matrices. Smoothed over a number of epochs. 161 | :param confusions: ([..., n_c, n_c] np.int32). Can be any dimension, the confusion matrices should be described by 162 | the last axes. n_c = number of classes 163 | :param smooth_n: (int). smooth extent 164 | :param ignore_unclassified: (bool). True if the the first class should be ignored in the results 165 | :return: ([..., n_c] np.float32) precision, recall, F1 score, IoU score 166 | """ 167 | 168 | # If the first class (often "unclassified") should be ignored, erase it from the confusion. 169 | if ignore_unclassified: 170 | confusions[..., 0, :] = 0 171 | confusions[..., :, 0] = 0 172 | 173 | # Sum successive confusions for smoothing 174 | smoothed_confusions = confusions.copy() 175 | if confusions.ndim > 2 and smooth_n > 0: 176 | for epoch in range(confusions.shape[-3]): 177 | i0 = max(epoch - smooth_n, 0) 178 | i1 = min(epoch + smooth_n + 1, confusions.shape[-3]) 179 | smoothed_confusions[..., epoch, :, :] = np.sum(confusions[..., i0:i1, :, :], axis=-3) 180 | 181 | # Compute TP, FP, FN. This assume that the second to last axis counts the truths (like the first axis of a 182 | # confusion matrix), and that the last axis counts the predictions (like the second axis of a confusion matrix) 183 | TP = np.diagonal(smoothed_confusions, axis1=-2, axis2=-1) 184 | TP_plus_FP = np.sum(smoothed_confusions, axis=-2) 185 | TP_plus_FN = np.sum(smoothed_confusions, axis=-1) 186 | 187 | # Compute precision and recall. This assume that the second to last axis counts the truths (like the first axis of 188 | # a confusion matrix), and that the last axis counts the predictions (like the second axis of a confusion matrix) 189 | PRE = TP / (TP_plus_FN + 1e-6) 190 | REC = TP / (TP_plus_FP + 1e-6) 191 | 192 | # Compute Accuracy 193 | ACC = np.sum(TP, axis=-1) / (np.sum(smoothed_confusions, axis=(-2, -1)) + 1e-6) 194 | 195 | # Compute F1 score 196 | F1 = 2 * TP / (TP_plus_FP + TP_plus_FN + 1e-6) 197 | 198 | # Compute IoU 199 | IoU = F1 / (2 - F1) 200 | 201 | return PRE, REC, F1, IoU, ACC 202 | 203 | 204 | def IoU_from_confusions(confusions): 205 | """ 206 | Computes IoU from confusion matrices. 207 | :param confusions: ([..., n_c, n_c] np.int32). Can be any dimension, the confusion matrices should be described by 208 | the last axes. n_c = number of classes 209 | :param ignore_unclassified: (bool). True if the the first class should be ignored in the results 210 | :return: ([..., n_c] np.float32) IoU score 211 | """ 212 | 213 | # Compute TP, FP, FN. This assume that the second to last axis counts the truths (like the first axis of a 214 | # confusion matrix), and that the last axis counts the predictions (like the second axis of a confusion matrix) 215 | TP = np.diagonal(confusions, axis1=-2, axis2=-1) 216 | TP_plus_FN = np.sum(confusions, axis=-1) 217 | TP_plus_FP = np.sum(confusions, axis=-2) 218 | 219 | # Compute IoU 220 | IoU = TP / (TP_plus_FP + TP_plus_FN - TP + 1e-6) 221 | 222 | # Compute mIoU with only the actual classes 223 | mask = TP_plus_FN < 1e-3 224 | counts = np.sum(1 - mask, axis=-1, keepdims=True) 225 | mIoU = np.sum(IoU, axis=-1, keepdims=True) / (counts + 1e-6) 226 | 227 | # If class is absent, place mIoU in place of 0 IoU to get the actual mean later 228 | IoU += mask * mIoU 229 | 230 | return IoU 231 | -------------------------------------------------------------------------------- /utils/ply.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0===============================0 4 | # | PLY files reader/writer | 5 | # 0===============================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # function to read/write .ply files 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 10/02/2017 15 | # 16 | 17 | 18 | # ---------------------------------------------------------------------------------------------------------------------- 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | 25 | # Basic libs 26 | import numpy as np 27 | import sys 28 | 29 | 30 | # Define PLY types 31 | ply_dtypes = dict([ 32 | (b'int8', 'i1'), 33 | (b'char', 'i1'), 34 | (b'uint8', 'u1'), 35 | (b'uchar', 'u1'), 36 | (b'int16', 'i2'), 37 | (b'short', 'i2'), 38 | (b'uint16', 'u2'), 39 | (b'ushort', 'u2'), 40 | (b'int32', 'i4'), 41 | (b'int', 'i4'), 42 | (b'uint32', 'u4'), 43 | (b'uint', 'u4'), 44 | (b'float32', 'f4'), 45 | (b'float', 'f4'), 46 | (b'float64', 'f8'), 47 | (b'double', 'f8') 48 | ]) 49 | 50 | # Numpy reader format 51 | valid_formats = {'ascii': '', 'binary_big_endian': '>', 52 | 'binary_little_endian': '<'} 53 | 54 | 55 | # ---------------------------------------------------------------------------------------------------------------------- 56 | # 57 | # Functions 58 | # \***************/ 59 | # 60 | 61 | 62 | def parse_header(plyfile, ext): 63 | # Variables 64 | line = [] 65 | properties = [] 66 | num_points = None 67 | 68 | while b'end_header' not in line and line != b'': 69 | line = plyfile.readline() 70 | 71 | if b'element' in line: 72 | line = line.split() 73 | num_points = int(line[2]) 74 | 75 | elif b'property' in line: 76 | line = line.split() 77 | properties.append((line[2].decode(), ext + ply_dtypes[line[1]])) 78 | 79 | return num_points, properties 80 | 81 | 82 | def parse_mesh_header(plyfile, ext): 83 | # Variables 84 | line = [] 85 | vertex_properties = [] 86 | num_points = None 87 | num_faces = None 88 | current_element = None 89 | 90 | 91 | while b'end_header' not in line and line != b'': 92 | line = plyfile.readline() 93 | 94 | # Find point element 95 | if b'element vertex' in line: 96 | current_element = 'vertex' 97 | line = line.split() 98 | num_points = int(line[2]) 99 | 100 | elif b'element face' in line: 101 | current_element = 'face' 102 | line = line.split() 103 | num_faces = int(line[2]) 104 | 105 | elif b'property' in line: 106 | if current_element == 'vertex': 107 | line = line.split() 108 | vertex_properties.append((line[2].decode(), ext + ply_dtypes[line[1]])) 109 | elif current_element == 'vertex': 110 | if not line.startswith('property list uchar int'): 111 | raise ValueError('Unsupported faces property : ' + line) 112 | 113 | return num_points, num_faces, vertex_properties 114 | 115 | 116 | def read_ply(filename, triangular_mesh=False): 117 | """ 118 | Read ".ply" files 119 | 120 | Parameters 121 | ---------- 122 | filename : string 123 | the name of the file to read. 124 | 125 | Returns 126 | ------- 127 | result : array 128 | data stored in the file 129 | 130 | Examples 131 | -------- 132 | Store data in file 133 | 134 | >>> points = np.random.rand(5, 3) 135 | >>> values = np.random.randint(2, size=10) 136 | >>> write_ply('example.ply', [points, values], ['x', 'y', 'z', 'values']) 137 | 138 | Read the file 139 | 140 | >>> data = read_ply('example.ply') 141 | >>> values = data['values'] 142 | array([0, 0, 1, 1, 0]) 143 | 144 | >>> points = np.vstack((data['x'], data['y'], data['z'])).T 145 | array([[ 0.466 0.595 0.324] 146 | [ 0.538 0.407 0.654] 147 | [ 0.850 0.018 0.988] 148 | [ 0.395 0.394 0.363] 149 | [ 0.873 0.996 0.092]]) 150 | 151 | """ 152 | 153 | with open(filename, 'rb') as plyfile: 154 | 155 | 156 | # Check if the file start with ply 157 | if b'ply' not in plyfile.readline(): 158 | raise ValueError('The file does not start whith the word ply') 159 | 160 | # get binary_little/big or ascii 161 | fmt = plyfile.readline().split()[1].decode() 162 | if fmt == "ascii": 163 | raise ValueError('The file is not binary') 164 | 165 | # get extension for building the numpy dtypes 166 | ext = valid_formats[fmt] 167 | 168 | # PointCloud reader vs mesh reader 169 | if triangular_mesh: 170 | 171 | # Parse header 172 | num_points, num_faces, properties = parse_mesh_header(plyfile, ext) 173 | 174 | # Get point data 175 | vertex_data = np.fromfile(plyfile, dtype=properties, count=num_points) 176 | 177 | # Get face data 178 | face_properties = [('k', ext + 'u1'), 179 | ('v1', ext + 'i4'), 180 | ('v2', ext + 'i4'), 181 | ('v3', ext + 'i4')] 182 | faces_data = np.fromfile(plyfile, dtype=face_properties, count=num_faces) 183 | 184 | # Return vertex data and concatenated faces 185 | faces = np.vstack((faces_data['v1'], faces_data['v2'], faces_data['v3'])).T 186 | data = [vertex_data, faces] 187 | 188 | else: 189 | 190 | # Parse header 191 | num_points, properties = parse_header(plyfile, ext) 192 | 193 | # Get data 194 | data = np.fromfile(plyfile, dtype=properties, count=num_points) 195 | 196 | return data 197 | 198 | 199 | def header_properties(field_list, field_names): 200 | 201 | # List of lines to write 202 | lines = [] 203 | 204 | # First line describing element vertex 205 | lines.append('element vertex %d' % field_list[0].shape[0]) 206 | 207 | # Properties lines 208 | i = 0 209 | for fields in field_list: 210 | for field in fields.T: 211 | lines.append('property %s %s' % (field.dtype.name, field_names[i])) 212 | i += 1 213 | 214 | return lines 215 | 216 | 217 | def write_ply(filename, field_list, field_names, triangular_faces=None): 218 | """ 219 | Write ".ply" files 220 | 221 | Parameters 222 | ---------- 223 | filename : string 224 | the name of the file to which the data is saved. A '.ply' extension will be appended to the 225 | file name if it does no already have one. 226 | 227 | field_list : list, tuple, numpy array 228 | the fields to be saved in the ply file. Either a numpy array, a list of numpy arrays or a 229 | tuple of numpy arrays. Each 1D numpy array and each column of 2D numpy arrays are considered 230 | as one field. 231 | 232 | field_names : list 233 | the name of each fields as a list of strings. Has to be the same length as the number of 234 | fields. 235 | 236 | Examples 237 | -------- 238 | >>> points = np.random.rand(10, 3) 239 | >>> write_ply('example1.ply', points, ['x', 'y', 'z']) 240 | 241 | >>> values = np.random.randint(2, size=10) 242 | >>> write_ply('example2.ply', [points, values], ['x', 'y', 'z', 'values']) 243 | 244 | >>> colors = np.random.randint(255, size=(10,3), dtype=np.uint8) 245 | >>> field_names = ['x', 'y', 'z', 'red', 'green', 'blue', values'] 246 | >>> write_ply('example3.ply', [points, colors, values], field_names) 247 | 248 | """ 249 | 250 | # Format list input to the right form 251 | field_list = list(field_list) if (type(field_list) == list or type(field_list) == tuple) else list((field_list,)) 252 | for i, field in enumerate(field_list): 253 | if field.ndim < 2: 254 | field_list[i] = field.reshape(-1, 1) 255 | if field.ndim > 2: 256 | print('fields have more than 2 dimensions') 257 | return False 258 | 259 | # check all fields have the same number of data 260 | n_points = [field.shape[0] for field in field_list] 261 | if not np.all(np.equal(n_points, n_points[0])): 262 | print('wrong field dimensions') 263 | return False 264 | 265 | # Check if field_names and field_list have same nb of column 266 | n_fields = np.sum([field.shape[1] for field in field_list]) 267 | if (n_fields != len(field_names)): 268 | print('wrong number of field names') 269 | return False 270 | 271 | # Add extension if not there 272 | if not filename.endswith('.ply'): 273 | filename += '.ply' 274 | 275 | # open in text mode to write the header 276 | with open(filename, 'w') as plyfile: 277 | 278 | # First magical word 279 | header = ['ply'] 280 | 281 | # Encoding format 282 | header.append('format binary_' + sys.byteorder + '_endian 1.0') 283 | 284 | # Points properties description 285 | header.extend(header_properties(field_list, field_names)) 286 | 287 | # Add faces if needded 288 | if triangular_faces is not None: 289 | header.append('element face {:d}'.format(triangular_faces.shape[0])) 290 | header.append('property list uchar int vertex_indices') 291 | 292 | # End of header 293 | header.append('end_header') 294 | 295 | # Write all lines 296 | for line in header: 297 | plyfile.write("%s\n" % line) 298 | 299 | # open in binary/append to use tofile 300 | with open(filename, 'ab') as plyfile: 301 | 302 | # Create a structured array 303 | i = 0 304 | type_list = [] 305 | for fields in field_list: 306 | for field in fields.T: 307 | type_list += [(field_names[i], field.dtype.str)] 308 | i += 1 309 | data = np.empty(field_list[0].shape[0], dtype=type_list) 310 | i = 0 311 | for fields in field_list: 312 | for field in fields.T: 313 | data[field_names[i]] = field 314 | i += 1 315 | 316 | data.tofile(plyfile) 317 | 318 | if triangular_faces is not None: 319 | triangular_faces = triangular_faces.astype(np.int32) 320 | type_list = [('k', 'uint8')] + [(str(ind), 'int32') for ind in range(3)] 321 | data = np.empty(triangular_faces.shape[0], dtype=type_list) 322 | data['k'] = np.full((triangular_faces.shape[0],), 3, dtype=np.uint8) 323 | data['0'] = triangular_faces[:, 0] 324 | data['1'] = triangular_faces[:, 1] 325 | data['2'] = triangular_faces[:, 2] 326 | data.tofile(plyfile) 327 | 328 | return True 329 | 330 | 331 | def describe_element(name, df): 332 | """ Takes the columns of the dataframe and builds a ply-like description 333 | 334 | Parameters 335 | ---------- 336 | name: str 337 | df: pandas DataFrame 338 | 339 | Returns 340 | ------- 341 | element: list[str] 342 | """ 343 | property_formats = {'f': 'float', 'u': 'uchar', 'i': 'int'} 344 | element = ['element ' + name + ' ' + str(len(df))] 345 | 346 | if name == 'face': 347 | element.append("property list uchar int points_indices") 348 | 349 | else: 350 | for i in range(len(df.columns)): 351 | # get first letter of dtype to infer format 352 | f = property_formats[str(df.dtypes[i])[0]] 353 | element.append('property ' + f + ' ' + df.columns.values[i]) 354 | 355 | return element -------------------------------------------------------------------------------- /visualize_deformations.py: -------------------------------------------------------------------------------- 1 | # 2 | # 3 | # 0=================================0 4 | # | Kernel Point Convolutions | 5 | # 0=================================0 6 | # 7 | # 8 | # ---------------------------------------------------------------------------------------------------------------------- 9 | # 10 | # Callable script to start a training on ModelNet40 dataset 11 | # 12 | # ---------------------------------------------------------------------------------------------------------------------- 13 | # 14 | # Hugues THOMAS - 06/03/2020 15 | # 16 | 17 | 18 | # ---------------------------------------------------------------------------------------------------------------------- 19 | # 20 | # Imports and global variables 21 | # \**********************************/ 22 | # 23 | 24 | # Common libs 25 | import signal 26 | import os 27 | import numpy as np 28 | import sys 29 | import torch 30 | 31 | # Dataset 32 | from datasets.ModelNet40 import * 33 | from datasets.S3DIS import * 34 | from torch.utils.data import DataLoader 35 | 36 | from utils.config import Config 37 | from utils.visualizer import ModelVisualizer 38 | from models.architectures import KPCNN, KPFCNN 39 | 40 | 41 | # ---------------------------------------------------------------------------------------------------------------------- 42 | # 43 | # Main Call 44 | # \***************/ 45 | # 46 | 47 | def model_choice(chosen_log): 48 | 49 | ########################### 50 | # Call the test initializer 51 | ########################### 52 | 53 | # Automatically retrieve the last trained model 54 | if chosen_log in ['last_ModelNet40', 'last_ShapeNetPart', 'last_S3DIS']: 55 | 56 | # Dataset name 57 | test_dataset = '_'.join(chosen_log.split('_')[1:]) 58 | 59 | # List all training logs 60 | logs = np.sort([os.path.join('results', f) for f in os.listdir('results') if f.startswith('Log')]) 61 | 62 | # Find the last log of asked dataset 63 | for log in logs[::-1]: 64 | log_config = Config() 65 | log_config.load(log) 66 | if log_config.dataset.startswith(test_dataset): 67 | chosen_log = log 68 | break 69 | 70 | if chosen_log in ['last_ModelNet40', 'last_ShapeNetPart', 'last_S3DIS']: 71 | raise ValueError('No log of the dataset "' + test_dataset + '" found') 72 | 73 | # Check if log exists 74 | if not os.path.exists(chosen_log): 75 | raise ValueError('The given log does not exists: ' + chosen_log) 76 | 77 | return chosen_log 78 | 79 | 80 | # ---------------------------------------------------------------------------------------------------------------------- 81 | # 82 | # Main Call 83 | # \***************/ 84 | # 85 | 86 | if __name__ == '__main__': 87 | 88 | ############################### 89 | # Choose the model to visualize 90 | ############################### 91 | 92 | # Here you can choose which model you want to test with the variable test_model. Here are the possible values : 93 | # 94 | # > 'last_XXX': Automatically retrieve the last trained model on dataset XXX 95 | # > 'results/Log_YYYY-MM-DD_HH-MM-SS': Directly provide the path of a trained model 96 | 97 | chosen_log = 'results/Log_2020-04-23_19-42-18' 98 | 99 | # Choose the index of the checkpoint to load OR None if you want to load the current checkpoint 100 | chkp_idx = None 101 | 102 | # Eventually you can choose which feature is visualized (index of the deform convolution in the network) 103 | deform_idx = 0 104 | 105 | # Deal with 'last_XXX' choices 106 | chosen_log = model_choice(chosen_log) 107 | 108 | ############################ 109 | # Initialize the environment 110 | ############################ 111 | 112 | # Set which gpu is going to be used 113 | GPU_ID = '0' 114 | 115 | # Set GPU visible device 116 | os.environ['CUDA_VISIBLE_DEVICES'] = GPU_ID 117 | 118 | ############### 119 | # Previous chkp 120 | ############### 121 | 122 | # Find all checkpoints in the chosen training folder 123 | chkp_path = os.path.join(chosen_log, 'checkpoints') 124 | chkps = [f for f in os.listdir(chkp_path) if f[:4] == 'chkp'] 125 | 126 | # Find which snapshot to restore 127 | if chkp_idx is None: 128 | chosen_chkp = 'current_chkp.tar' 129 | else: 130 | chosen_chkp = np.sort(chkps)[chkp_idx] 131 | chosen_chkp = os.path.join(chosen_log, 'checkpoints', chosen_chkp) 132 | 133 | # Initialize configuration class 134 | config = Config() 135 | config.load(chosen_log) 136 | 137 | ################################## 138 | # Change model parameters for test 139 | ################################## 140 | 141 | # Change parameters for the test here. For example, you can stop augmenting the input data. 142 | 143 | config.augment_noise = 0.0001 144 | config.batch_num = 1 145 | config.in_radius = 2.0 146 | config.input_threads = 0 147 | 148 | ############## 149 | # Prepare Data 150 | ############## 151 | 152 | print() 153 | print('Data Preparation') 154 | print('****************') 155 | 156 | # Initiate dataset 157 | if config.dataset.startswith('ModelNet40'): 158 | test_dataset = ModelNet40Dataset(config, train=False) 159 | test_sampler = ModelNet40Sampler(test_dataset) 160 | collate_fn = ModelNet40Collate 161 | elif config.dataset == 'S3DIS': 162 | test_dataset = S3DISDataset(config, set='validation', use_potentials=True) 163 | test_sampler = S3DISSampler(test_dataset) 164 | collate_fn = S3DISCollate 165 | else: 166 | raise ValueError('Unsupported dataset : ' + config.dataset) 167 | 168 | # Data loader 169 | test_loader = DataLoader(test_dataset, 170 | batch_size=1, 171 | sampler=test_sampler, 172 | collate_fn=collate_fn, 173 | num_workers=config.input_threads, 174 | pin_memory=True) 175 | 176 | # Calibrate samplers 177 | test_sampler.calibration(test_loader, verbose=True) 178 | 179 | print('\nModel Preparation') 180 | print('*****************') 181 | 182 | # Define network model 183 | t1 = time.time() 184 | if config.dataset_task == 'classification': 185 | net = KPCNN(config) 186 | elif config.dataset_task in ['cloud_segmentation', 'slam_segmentation']: 187 | net = KPFCNN(config, test_dataset.label_values, test_dataset.ignored_labels) 188 | else: 189 | raise ValueError('Unsupported dataset_task for deformation visu: ' + config.dataset_task) 190 | 191 | # Define a visualizer class 192 | visualizer = ModelVisualizer(net, config, chkp_path=chosen_chkp, on_gpu=False) 193 | print('Done in {:.1f}s\n'.format(time.time() - t1)) 194 | 195 | print('\nStart visualization') 196 | print('*******************') 197 | 198 | # Training 199 | visualizer.show_deformable_kernels(net, test_loader, config, deform_idx) 200 | 201 | 202 | 203 | --------------------------------------------------------------------------------