├── Im2Struct_training ├── Im2Struct_code │ ├── CONTRIBUTING.md │ ├── COPYING │ ├── Makefile │ ├── README.md │ ├── draw3dOBB.m │ ├── drawmeshandobb.m │ ├── drawvolumeandobb.m │ ├── examples │ │ ├── cnn_train_dag_masknet.m │ │ └── cnn_train_old.m │ ├── exchangemodelobj.m │ ├── masknetwork │ │ ├── bilateralFilter.m │ │ ├── bilinear_u.m │ │ ├── cnn_masknet.m │ │ ├── cnn_masknet_datasetup.m │ │ ├── cnn_masknet_train.m │ │ ├── createNetfromvgg.m │ │ ├── depth2pc_2.m │ │ ├── getBatch.m │ │ ├── getBatch_old.m │ │ ├── myAccuracy.m │ │ ├── myDAGnet.m │ │ ├── myDAGnet2.m │ │ ├── myDAGnet3.m │ │ ├── myLoss.m │ │ ├── myReshape.m │ │ ├── myTrain.m │ │ ├── my_cnn_train_dag.m │ │ ├── my_cnn_train_dag_new.m │ │ ├── my_cnn_train_dag_old.m │ │ ├── mysimpleDAG.m │ │ ├── setLearningRate.m │ │ ├── setLearningRate_small.m │ │ ├── setLearningRate_small2.m │ │ ├── test_masknet.m │ │ └── test_masknet_realImage.m │ ├── matconvnet.sln │ ├── matconvnet.vcxproj │ ├── matconvnet.vcxproj.filters │ ├── test1.m │ ├── test2.m │ ├── vgg_16_mask │ │ ├── GenerateRecoverBoxes_vggwithmask.m │ │ ├── Netfromvgg_scale1.m │ │ ├── RecoverGenshapes_vggwithmask.m │ │ ├── change2full.m │ │ ├── cnn_Im2strnet.m │ │ ├── cnn_im2str_datasetup.m │ │ ├── cnn_im2str_train.m │ │ ├── cnn_maskoutfeature.m │ │ ├── cnn_train_dag_im2strnet.m │ │ ├── diagonal_distance.m │ │ ├── distence_gt_rec.m │ │ ├── distence_rec_gt.m │ │ ├── draw3dOBB_v2.m │ │ ├── draw_vgg16_test_demo.m │ │ ├── four_hausdorff_plot.m │ │ ├── hausdorff_diagonal_smallbox.m │ │ ├── hausdorff_diagonal_smallbox_proportion.m │ │ ├── hausdorff_drawing.m │ │ ├── im2str_initializeVaeParameters.m │ │ ├── loss_drawing.m │ │ ├── loss_drawing_comp.m │ │ ├── ouputgenshapes.m │ │ ├── recoverGtshapes.m │ │ ├── setLR1.m │ │ ├── setLR2.m │ │ ├── setLR3.m │ │ ├── showGenshapes.m │ │ └── vgg16_test_demo.m │ └── voxelization │ │ ├── Compareobb_volume.m │ │ ├── demo.m │ │ ├── demo.png │ │ ├── ntest.m │ │ ├── obj__read.m │ │ ├── plot3D.m │ │ ├── polygon2voxel.m │ │ ├── polygon2voxel_double.c │ │ ├── polygon2voxel_double.mexa64 │ │ ├── polygon2voxel_double.mexmaci64 │ │ ├── polygon2voxel_double.mexw64 │ │ ├── show3DModel.m │ │ ├── sparse3D.m │ │ └── voxel.m └── data_pre │ ├── image_obtain │ ├── Render_simple │ │ ├── Engine.cpp │ │ ├── Engine.h │ │ ├── EngineOSG.cpp │ │ ├── EngineOSG.h │ │ ├── MiPOSat.osg │ │ ├── compile_renderer.m │ │ ├── creatData.m │ │ ├── creatTestData.m │ │ ├── createlabeled.m │ │ ├── demo.m │ │ ├── depth.cpp │ │ ├── depth.h │ │ ├── depth.pgm │ │ ├── frontdata_generate.m │ │ ├── gen_Ref_View.m │ │ ├── model.osg │ │ ├── modifyobj.m │ │ ├── myunproject.m │ │ ├── plywrite.m │ │ ├── readme.windows.1.0.0.txt │ │ ├── renderer.cpp │ │ ├── renderer.mexw64 │ │ ├── scr_new.obj │ │ ├── scr_new.txt │ │ ├── test.jpg │ │ ├── test.ply │ │ ├── unpro.mat │ │ ├── unproject.asv │ │ ├── util.cpp │ │ └── util.h │ ├── addbackground.m │ └── creatTestData.m │ ├── obb_hierarchy │ ├── CreateShapeOBB_adj_vae.m │ ├── CreateShapeOBB_adj_vae_plane.m │ ├── changeObbPath.m │ ├── draw3dOBB_v2.m │ ├── readObbShapes_vae.m │ └── showGenshapes.m │ ├── obj_align.m │ └── partnet_align.m ├── README.md ├── code ├── draw3DOBB.m ├── im2struct_demo.m ├── mask_myReshape.m ├── norm1tanh.m └── showGenshapes.m ├── data ├── airplane_1.jpg ├── airplane_2.jpg ├── example_1.jpg ├── example_2.jpg ├── example_3.jpg └── example_4.jpg ├── image_show ├── 1.png └── 2.png └── model └── download.txt /Im2Struct_training/Im2Struct_code/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing guidelines 2 | 3 | ## How to contribute to MatConvNet 4 | 5 | For a description of how the library is structured, take a look at the 6 | [Developers notes](http://www.vlfeat.org/matconvnet/developers/) on 7 | the MatConvNet website. 8 | 9 | ### Issues 10 | 11 | We are grateful for any reported issues which help to remove bugs and 12 | improve the overall quality of the library. In particular, you can use 13 | the issue tracker to: 14 | 15 | * report bugs and unexpected crashes 16 | * discuss library design decisions 17 | * request new features 18 | 19 | When reporting bugs, it really helps if you can provide the following: 20 | 21 | * Which steps are needed to reproduce the issue 22 | * MATLAB, compiler and CUDA version (where appropriate) 23 | 24 | Before opening an issue to report a bug, please make sure that the bug 25 | is reproducible on the latest version of the master branch. 26 | 27 | The most difficult bugs to remove are those which cause crashes of the 28 | core functions (e.g. CUDA errors etc.). In those cases, it is really 29 | useful to create a *minimal example* which is able to reproduce the 30 | issue. We know that this may mean a bit of work, but it helps us to 31 | remove the bug more quickly. 32 | 33 | ### Pull requests 34 | 35 | Please make any Pull Requests against the `devel` branch rather than 36 | the `master` branch which is maintained as the latest stable release 37 | of the library. 38 | 39 | As a general rule, it is much easier to accept small Pull Requests 40 | that make a single improvement to the library than complex code 41 | changes that affect multiple parts of the library. When submitting 42 | substantial changes, it is useful if unit tests are provided with the 43 | code. 44 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/COPYING: -------------------------------------------------------------------------------- 1 | Copyright (c) 2014-16 The MatConvNet Team. 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms are permitted 5 | provided that the above copyright notice and this paragraph are 6 | duplicated in all such forms and that any documentation, advertising 7 | materials, and other materials related to such distribution and use 8 | acknowledge that the software was developed by the MatConvNet 9 | Team. The name of the MatConvNet Team may not be used to endorse or 10 | promote products derived from this software without specific prior 11 | written permission. THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT 12 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE 13 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 14 | PURPOSE. -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/Makefile: -------------------------------------------------------------------------------- 1 | # file: Makefile 2 | # author: Andrea Vedaldi 3 | # brief: matconvnet makefile for mex files 4 | 5 | # Copyright (C) 2014-16 Andrea Vedaldi 6 | # All rights reserved. 7 | # 8 | # This file is part of the VLFeat library and is made available under 9 | # the terms of the BSD license (see the COPYING file). 10 | 11 | # ENABLE_GPU -- Set to YES to enable GPU support (requires CUDA and the MATLAB Parallel Toolbox) 12 | # ENABLE_CUDNN -- Set to YES to enable CUDNN support. This will also likely require 13 | # ENABLE_IMREADJPEG -- Set to YES to compile the function VL_IMREADJPEG() 14 | 15 | ENABLE_GPU ?= 16 | ENABLE_CUDNN ?= 17 | ENABLE_IMREADJPEG ?= yes 18 | ENABLE_DOUBLE ?= yes 19 | DEBUG ?= 20 | ARCH ?= maci64 21 | 22 | # Configure MATLAB 23 | MATLABROOT ?= /Applications/MATLAB_R2017a.app 24 | 25 | # Configure CUDA and CuDNN. CUDAMETHOD can be either 'nvcc' or 'mex'. 26 | CUDAROOT ?= /Developer/NVIDIA/CUDA-8.0 27 | CUDNNROOT ?= $(CURDIR)/local/ 28 | CUDAMETHOD ?= $(if $(ENABLE_CUDNN),nvcc,mex) 29 | 30 | # Remark: each MATLAB version requires a particular CUDA Toolkit version. 31 | # Note that multiple CUDA Toolkits can be installed. 32 | #MATLABROOT ?= /Applications/MATLAB_R2014b.app 33 | #CUDAROOT ?= /Developer/NVIDIA/CUDA-6.0 34 | #MATLABROOT ?= /Applications/MATLAB_R2015a.app 35 | #CUDAROOT ?= /Developer/NVIDIA/CUDA-7.0 36 | #MATLABROOT ?= /Applications/MATLAB_R2015b.app 37 | #CUDAROOT ?= /Developer/NVIDIA/CUDA-7.5 38 | 39 | # Maintenance 40 | NAME = matconvnet 41 | VER = 1.0-beta24 42 | DIST = $(NAME)-$(VER) 43 | LATEST = $(NAME)-latest 44 | RSYNC = rsync 45 | HOST = vlfeat-admin:sites/sandbox-matconvnet 46 | GIT = git 47 | SHELL = /bin/bash # sh not good enough 48 | 49 | # -------------------------------------------------------------------- 50 | # Configuration 51 | # -------------------------------------------------------------------- 52 | 53 | # General options 54 | MEX = $(MATLABROOT)/bin/mex 55 | MEXEXT = $(MATLABROOT)/bin/mexext 56 | MEXARCH = $(subst mex,,$(shell $(MEXEXT))) 57 | MEXOPTS ?= matlab/src/config/mex_CUDA_$(ARCH).xml 58 | NVCC = $(CUDAROOT)/bin/nvcc 59 | 60 | comma:=, 61 | space:= 62 | space+= 63 | join-with = $(subst $(space),$1,$(strip $2)) 64 | nvcc-quote = $(if $(strip $1),-Xcompiler $(call join-with,$(comma),$(1)),) 65 | 66 | # Flags: 67 | # 1. `CXXFLAGS`: passed to `mex` and `nvcc` compiler wrappers 68 | # 2. `CXXFLAGS_PASS`: passed directly to the underlying C++ compiler 69 | # 3. `LDFLAGS`: passed directly to the underlying C++ compiler for linking 70 | # 4. `CXXOPTIMFLAGS`: passed directyl to the underlying C++ compiler 71 | # 5. `LDOPTIMFLAGS`: passed directly to the underlying C++ compiler 72 | # 6. `NVCCFLAGS_PASS`: passed directly to `nvcc` when invoked directly or through `mex` 73 | 74 | CXXFLAGS = \ 75 | $(if $(ENABLE_GPU),-DENABLE_GPU,) \ 76 | $(if $(ENABLE_CUDNN),-DENABLE_CUDNN -I$(CUDNNROOT)/include,) \ 77 | $(if $(ENABLE_DOUBLE),-DENABLE_DOUBLE,) \ 78 | $(if $(VERB),-v,) 79 | CXXFLAGS_PASS = 80 | CXXOPTIMFLAGS = 81 | LDFLAGS = 82 | LDOPTIMFLAGS = 83 | LINKLIBS = -lmwblas 84 | 85 | NVCCFLAGS_PASS = -D_FORCE_INLINES -gencode=arch=compute_30,code=\"sm_30,compute_30\" 86 | NVCCVER = $(shell $(NVCC) --version | \ 87 | sed -n 's/.*V\([0-9]*\).\([0-9]*\).\([0-9]*\).*/\1 \2 \3/p' | \ 88 | xargs printf '%02d%02d%02d') 89 | NVCCVER_LT_70 = $(shell test $(NVCCVER) -lt 070000 && echo true) 90 | 91 | # Mac OS X 92 | ifeq "$(ARCH)" "$(filter $(ARCH),maci64)" 93 | IMAGELIB ?= $(if $(ENABLE_IMREADJPEG),quartz,none) 94 | CXXFLAGS_PASS += -mmacosx-version-min=10.9 95 | CXXOPTIMFLAGS += -mssse3 -ffast-math 96 | LDFLAGS += \ 97 | -mmacosx-version-min=10.9 \ 98 | $(if $(ENABLE_GPU),-Wl$(comma)-rpath -Wl$(comma)"$(CUDAROOT)/lib") \ 99 | $(if $(ENABLE_CUDNN),-Wl$(comma)-rpath -Wl$(comma)"$(CUDNNROOT)/lib") \ 100 | $(if $(NVCCVER_LT_70),-stdlib=libstdc++) 101 | LINKLIBS += \ 102 | $(if $(ENABLE_GPU),-L"$(CUDAROOT)/lib" -lmwgpu -lcudart -lcublas) \ 103 | $(if $(ENABLE_CUDNN),-L"$(CUDNNROOT)/lib" -lcudnn) 104 | endif 105 | 106 | # Linux 107 | ifeq "$(ARCH)" "$(filter $(ARCH),glnxa64)" 108 | IMAGELIB ?= $(if $(ENABLE_IMREADJPEG),libjpeg,none) 109 | CXXOPTIMFLAGS += -mssse3 -ftree-vect-loop-version -ffast-math -funroll-all-loops 110 | LDFLAGS += \ 111 | $(if $(ENABLE_GPU),-Wl$(comma)-rpath -Wl$(comma)"$(CUDAROOT)/lib64") \ 112 | $(if $(ENABLE_CUDNN),-Wl$(comma)-rpath -Wl$(comma)"$(CUDNNROOT)/lib64") 113 | LINKLIBS += \ 114 | -lrt \ 115 | $(if $(ENABLE_GPU),-L"$(CUDAROOT)/lib64" -lmwgpu -lcudart -lcublas) \ 116 | $(if $(ENABLE_CUDNN),-L"$(CUDNNROOT)/lib64" -lcudnn) 117 | endif 118 | 119 | # Image library 120 | ifeq ($(IMAGELIB),libjpeg) 121 | LINKLIBS += -ljpeg 122 | endif 123 | ifeq ($(IMAGELIB),quartz) 124 | LINKLIBS += -framework Cocoa -framework ImageIO 125 | endif 126 | 127 | MEXFLAGS = $(CXXFLAGS) -largeArrayDims 128 | 129 | ifneq ($(DEBUG),) 130 | MEXFLAGS += -g 131 | NVCCFLAGS_PASS += -g -O0 132 | else 133 | MEXFLAGS += -DNDEBUG -O 134 | NVCCFLAGS_PASS += -DNDEBUG -O3 135 | # include debug symbol also in non-debug version 136 | CXXOPTIMFLAGS += -g 137 | LDOPTIMFLAGS += -g 138 | NVCCFLAGS_PASS += -g 139 | endif 140 | 141 | # -------------------------------------------------------------------- 142 | # Build MEX files 143 | # -------------------------------------------------------------------- 144 | 145 | nvcc_filter=2> >( sed 's/^\(.*\)(\([0-9][0-9]*\)): \([ew].*\)/\1:\2: \3/g' >&2 ) 146 | cpp_src := 147 | mex_src := 148 | 149 | # Files that are compiled as CPP or GPU (CUDA) depending on whether GPU support 150 | # is enabled. 151 | ext := $(if $(ENABLE_GPU),cu,cpp) 152 | cpp_src+=matlab/src/bits/data.$(ext) 153 | cpp_src+=matlab/src/bits/datamex.$(ext) 154 | cpp_src+=matlab/src/bits/nnconv.$(ext) 155 | cpp_src+=matlab/src/bits/nnbias.$(ext) 156 | cpp_src+=matlab/src/bits/nnfullyconnected.$(ext) 157 | cpp_src+=matlab/src/bits/nnsubsample.$(ext) 158 | cpp_src+=matlab/src/bits/nnpooling.$(ext) 159 | cpp_src+=matlab/src/bits/nnnormalize.$(ext) 160 | cpp_src+=matlab/src/bits/nnbnorm.$(ext) 161 | cpp_src+=matlab/src/bits/nnbilinearsampler.$(ext) 162 | cpp_src+=matlab/src/bits/nnroipooling.$(ext) 163 | mex_src+=matlab/src/vl_nnconv.$(ext) 164 | mex_src+=matlab/src/vl_nnconvt.$(ext) 165 | mex_src+=matlab/src/vl_nnpool.$(ext) 166 | mex_src+=matlab/src/vl_nnnormalize.$(ext) 167 | mex_src+=matlab/src/vl_nnbnorm.$(ext) 168 | mex_src+=matlab/src/vl_nnbilinearsampler.$(ext) 169 | mex_src+=matlab/src/vl_nnroipool.$(ext) 170 | mex_src+=matlab/src/vl_taccummex.$(ext) 171 | mex_src+=matlab/src/vl_tmove.$(ext) 172 | ifdef ENABLE_IMREADJPEG 173 | mex_src+=matlab/src/vl_imreadjpeg.$(ext) 174 | mex_src+=matlab/src/vl_imreadjpeg_old.$(ext) 175 | endif 176 | 177 | # CPU-specific files 178 | cpp_src+=matlab/src/bits/impl/im2row_cpu.cpp 179 | cpp_src+=matlab/src/bits/impl/subsample_cpu.cpp 180 | cpp_src+=matlab/src/bits/impl/copy_cpu.cpp 181 | cpp_src+=matlab/src/bits/impl/pooling_cpu.cpp 182 | cpp_src+=matlab/src/bits/impl/normalize_cpu.cpp 183 | cpp_src+=matlab/src/bits/impl/bnorm_cpu.cpp 184 | cpp_src+=matlab/src/bits/impl/bilinearsampler_cpu.cpp 185 | cpp_src+=matlab/src/bits/impl/roipooling_cpu.cpp 186 | cpp_src+=matlab/src/bits/impl/tinythread.cpp 187 | ifdef ENABLE_IMREADJPEG 188 | cpp_src+=matlab/src/bits/impl/imread_$(IMAGELIB).cpp 189 | cpp_src+=matlab/src/bits/imread.cpp 190 | endif 191 | 192 | # GPU-specific files 193 | ifdef ENABLE_GPU 194 | cpp_src+=matlab/src/bits/impl/im2row_gpu.cu 195 | cpp_src+=matlab/src/bits/impl/subsample_gpu.cu 196 | cpp_src+=matlab/src/bits/impl/copy_gpu.cu 197 | cpp_src+=matlab/src/bits/impl/pooling_gpu.cu 198 | cpp_src+=matlab/src/bits/impl/normalize_gpu.cu 199 | cpp_src+=matlab/src/bits/impl/bnorm_gpu.cu 200 | cpp_src+=matlab/src/bits/impl/bilinearsampler_gpu.cu 201 | cpp_src+=matlab/src/bits/impl/roipooling_gpu.cu 202 | cpp_src+=matlab/src/bits/datacu.cu 203 | mex_src+=matlab/src/vl_cudatool.cu 204 | ifdef ENABLE_CUDNN 205 | cpp_src+=matlab/src/bits/impl/nnconv_cudnn.cu 206 | cpp_src+=matlab/src/bits/impl/nnpooling_cudnn.cu 207 | cpp_src+=matlab/src/bits/impl/nnbias_cudnn.cu 208 | cpp_src+=matlab/src/bits/impl/nnbilinearsampler_cudnn.cu 209 | cpp_src+=matlab/src/bits/impl/nnbnorm_cudnn.cu 210 | endif 211 | endif 212 | 213 | mex_tgt:=$(patsubst %.cpp,%.mex$(MEXARCH),$(mex_src)) 214 | mex_tgt:=$(patsubst %.cu,%.mex$(MEXARCH),$(mex_tgt)) 215 | mex_tgt:=$(subst matlab/src/,matlab/mex/,$(mex_tgt)) 216 | 217 | mex_obj:=$(patsubst %.cpp,%.o,$(mex_src)) 218 | mex_obj:=$(patsubst %.cu,%.o,$(mex_obj)) 219 | mex_obj:=$(subst matlab/src/,matlab/mex/.build/,$(mex_obj)) 220 | 221 | cpp_tgt:=$(patsubst %.cpp,%.o,$(cpp_src)) 222 | cpp_tgt:=$(patsubst %.cu,%.o,$(cpp_tgt)) 223 | cpp_tgt:=$(subst matlab/src/,matlab/mex/.build/,$(cpp_tgt)) 224 | 225 | .PHONY: all, distclean, clean, info, pack, post, post-doc, doc 226 | 227 | all: $(cpp_tgt) $(mex_obj) $(mex_tgt) 228 | 229 | # Create build directory 230 | %/.stamp: 231 | mkdir -p $(*)/ ; touch $(*)/.stamp 232 | $(mex_tgt): matlab/mex/.build/bits/impl/.stamp 233 | $(cpp_tgt): matlab/mex/.build/bits/impl/.stamp 234 | 235 | # Standard code 236 | .PRECIOUS: matlab/mex/.build/%.o 237 | .PRECIOUS: %/.stamp 238 | 239 | matlab/mex/.build/bits/impl/imread.o : matlab/src/bits/impl/imread_helpers.hpp 240 | matlab/mex/.build/bits/impl/imread_quartz.o : matlab/src/bits/impl/imread_helpers.hpp 241 | matlab/mex/.build/bits/impl/imread_gdiplus.o : matlab/src/bits/impl/imread_helpers.hpp 242 | matlab/mex/.build/bits/impl/imread_libjpeg.o : matlab/src/bits/impl/imread_helpers.hpp 243 | 244 | # -------------------------------------------------------------------- 245 | # Compilation rules 246 | # -------------------------------------------------------------------- 247 | 248 | MEXFLAGS_CC_CPU := \ 249 | $(MEXFLAGS) \ 250 | CXXFLAGS='$$CXXFLAGS $(CXXFLAGS_PASS)' \ 251 | CXXOPTIMFLAGS='$$CXXOPTIMFLAGS $(CXXOPTIMFLAGS)' 252 | 253 | MEXFLAGS_CC_GPU := \ 254 | -f "$(MEXOPTS)" \ 255 | $(MEXFLAGS) \ 256 | CXXFLAGS='$$CXXFLAGS $(NVCCFLAGS_PASS) $(call nvcc-quote,$(CXXFLAGS_PASS))' \ 257 | CXXOPTIMFLAGS='$$CXXOPTIMFLAGS $(call nvcc-quote,$(CXXOPTIMFLAGS))' 258 | 259 | MEXFLAGS_LD := $(MEXFLAGS) \ 260 | LDFLAGS='$$LDFLAGS $(LDFLAGS)' \ 261 | LDOPTIMFLAGS='$$LDOPTIMFLAGS $(LDOPTIMFLAGS)' \ 262 | LINKLIBS='$(LINKLIBS) $$LINKLIBS' \ 263 | 264 | NVCCFLAGS = $(CXXFLAGS) $(NVCCFLAGS_PASS) \ 265 | -I"$(MATLABROOT)/extern/include" \ 266 | -I"$(MATLABROOT)/toolbox/distcomp/gpu/extern/include" \ 267 | $(call nvcc-quote,-fPIC $(CXXFLAGS_PASS) $(CXXOPTIMFLAGS)) 268 | 269 | ifneq ($(ENABLE_GPU),) 270 | ifeq ($(CUDAMETHOD),mex) 271 | matlab/mex/.build/%.o : matlab/src/%.cu matlab/mex/.build/.stamp 272 | MW_NVCC_PATH='$(NVCC)' \ 273 | $(MEX) -c $(MEXFLAGS_CC_GPU) "$(<)" $(nvcc_filter) 274 | mv -f "$(notdir $(@))" "$(@)" 275 | else 276 | matlab/mex/.build/%.o : matlab/src/%.cu matlab/mex/.build/.stamp 277 | $(NVCC) $(NVCCFLAGS) "$(<)" -c -o "$(@)" $(nvcc_filter) 278 | endif 279 | endif 280 | 281 | matlab/mex/.build/%.o : matlab/src/%.cpp matlab/src/%.cu matlab/mex/.build/.stamp 282 | $(MEX) -c $(MEXFLAGS_CC_CPU) "$(<)" 283 | mv -f "$(notdir $(@))" "$(@)" 284 | 285 | matlab/mex/.build/%.o : matlab/src/%.cpp matlab/mex/.build/.stamp 286 | $(MEX) -c $(MEXFLAGS_CC_CPU) "$(<)" 287 | mv -f "$(notdir $(@))" "$(@)" 288 | 289 | matlab/mex/%.mex$(MEXARCH) : matlab/mex/.build/%.o $(cpp_tgt) 290 | $(MEX) $(MEXFLAGS_LD) "$(<)" -output "$(@)" $(cpp_tgt) 291 | 292 | # -------------------------------------------------------------------- 293 | # Documentation 294 | # -------------------------------------------------------------------- 295 | 296 | include doc/Makefile 297 | 298 | # -------------------------------------------------------------------- 299 | # Maintenance 300 | # -------------------------------------------------------------------- 301 | 302 | info: doc-info 303 | @echo "mex_src=$(mex_src)" 304 | @echo "mex_obj=$(mex_obj)" 305 | @echo "mex_tgt=$(mex_tgt)" 306 | @echo "cpp_src=$(cpp_src)" 307 | @echo "cpp_tgt=$(cpp_tgt)" 308 | @echo '------------------------------' 309 | @echo 'CUDAMETHOD=$(CUDAMETHOD)' 310 | @echo 'CXXFLAGS=$(CXXFLAGS)' 311 | @echo 'CXXOPTIMFLAGS=$(CXXOPTIMFLAGS)' 312 | @echo 'LDFLAGS=$(LDFLAGS)' 313 | @echo 'LDOPTIMFLAGS=$(LDOPTIMFLAGS)' 314 | @echo 'LINKLIBS=$(LINKLIBS)' 315 | @echo '------------------------------' 316 | @echo 'MEXARCH=$(MEXARCH)' 317 | @echo 'MEXFLAGS=$(MEXFLAGS)' 318 | @echo 'MEXFLAGS_CC_CPU=$(MEXFLAGS_CC_CPU)' 319 | @echo 'MEXFLAGS_CC_GPU=$(MEXFLAGS_CC_GPU)' 320 | @echo 'MEXFLAGS_LD=$(MEXFLAGS_LD)' 321 | @echo '------------------------------' 322 | @echo 'NVCC=$(NVCC)' 323 | @echo 'NVCCVER=$(NVCCVER)' 324 | @echo 'NVCCVER_LT_70=$(NVCCVER_LT_70)' 325 | @echo 'NVCCFLAGS_PASS=$(NVCCFLAGS_PASS)' 326 | @echo 'NVCCFLAGS=$(NVCCFLAGS)' 327 | 328 | 329 | clean: doc-clean 330 | find . -name '*~' -delete 331 | rm -f $(cpp_tgt) 332 | rm -rf matlab/mex/.build 333 | 334 | distclean: clean doc-distclean 335 | rm -rf matlab/mex 336 | rm -f doc/index.html doc/matconvnet-manual.pdf 337 | rm -f $(NAME)-*.tar.gz 338 | 339 | pack: 340 | COPYFILE_DISABLE=1 \ 341 | COPY_EXTENDED_ATTRIBUTES_DISABLE=1 \ 342 | $(GIT) archive --prefix=$(NAME)-$(VER)/ v$(VER) | gzip > $(DIST).tar.gz 343 | ln -sf $(DIST).tar.gz $(LATEST).tar.gz 344 | 345 | post: pack 346 | $(RSYNC) -aP $(DIST).tar.gz $(LATEST).tar.gz $(HOST)/download/ 347 | 348 | post-models: 349 | $(RSYNC) -aP data/models/*.mat $(HOST)/models/ 350 | 351 | post-doc: doc 352 | $(RSYNC) -aP doc/matconvnet-manual.pdf $(HOST)/ 353 | $(RSYNC) \ 354 | --recursive \ 355 | --perms \ 356 | --verbose \ 357 | --delete \ 358 | --exclude=download \ 359 | --exclude=models \ 360 | --exclude=matconvnet-manual.pdf \ 361 | --exclude=.htaccess doc/site/site/ $(HOST)/ 362 | 363 | .PHONY: model-md5 364 | model-md5: 365 | cd data/models ; md5sum *.mat | xargs printf '| %-33s| %-40s|\n' 366 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/README.md: -------------------------------------------------------------------------------- 1 | # MatConvNet: CNNs for MATLAB 2 | 3 | **MatConvNet** is a MATLAB toolbox implementing *Convolutional Neural 4 | Networks* (CNNs) for computer vision applications. It is simple, 5 | efficient, and can run and learn state-of-the-art CNNs. Several 6 | example CNNs are included to classify and encode images. Please visit 7 | the [homepage](http://www.vlfeat.org/matconvnet) to know more. 8 | 9 | In case of compilation issues, please read first the 10 | [Installation](http://www.vlfeat.org/matconvnet/install/) and 11 | [FAQ](http://www.vlfeat.org/matconvnet/faq/) section before creating an GitHub 12 | issue. For general inquiries regarding network design and training 13 | related questions, please use the 14 | [Discussion forum](https://groups.google.com/d/forum/matconvnet). 15 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/draw3dOBB.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/draw3dOBB.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/drawmeshandobb.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/drawmeshandobb.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/drawvolumeandobb.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/drawvolumeandobb.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/exchangemodelobj.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/exchangemodelobj.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/bilateralFilter.m: -------------------------------------------------------------------------------- 1 | % 2 | % output = bilateralFilter( data, edge, ... 3 | % edgeMin, edgeMax, ... 4 | % sigmaSpatial, sigmaRange, ... 5 | % samplingSpatial, samplingRange ) 6 | % 7 | % Bilateral and Cross-Bilateral Filter using the Bilateral Grid. 8 | % 9 | % Bilaterally filters the image 'data' using the edges in the image 'edge'. 10 | % If 'data' == 'edge', then it the standard bilateral filter. 11 | % Otherwise, it is the 'cross' or 'joint' bilateral filter. 12 | % For convenience, you can also pass in [] for 'edge' for the normal 13 | % bilateral filter. 14 | % 15 | % Note that for the cross bilateral filter, data does not need to be 16 | % defined everywhere. Undefined values can be set to 'NaN'. However, edge 17 | % *does* need to be defined everywhere. 18 | % 19 | % data and edge should be of the greyscale, double-precision floating point 20 | % matrices of the same size (i.e. they should be [ height x width ]) 21 | % 22 | % data is the only required argument 23 | % 24 | % edgeMin and edgeMax specifies the min and max values of 'edge' (or 'data' 25 | % for the normal bilateral filter) and is useful when the input is in a 26 | % range that's not between 0 and 1. For instance, if you are filtering the 27 | % L channel of an image that ranges between 0 and 100, set edgeMin to 0 and 28 | % edgeMax to 100. 29 | % 30 | % edgeMin defaults to min( edge( : ) ) and edgeMax defaults to max( edge( : ) ). 31 | % This is probably *not* what you want, since the input may not span the 32 | % entire range. 33 | % 34 | % sigmaSpatial and sigmaRange specifies the standard deviation of the space 35 | % and range gaussians, respectively. 36 | % sigmaSpatial defaults to min( width, height ) / 16 37 | % sigmaRange defaults to ( edgeMax - edgeMin ) / 10. 38 | % 39 | % samplingSpatial and samplingRange specifies the amount of downsampling 40 | % used for the approximation. Higher values use less memory but are also 41 | % less accurate. The default and recommended values are: 42 | % 43 | % samplingSpatial = sigmaSpatial 44 | % samplingRange = sigmaRange 45 | % 46 | 47 | function output = bilateralFilter( data, edge, edgeMin, edgeMax, sigmaSpatial, sigmaRange, ... 48 | samplingSpatial, samplingRange ) 49 | 50 | if( ndims( data ) > 2 ), 51 | error( 'data must be a greyscale image with size [ height, width ]' ); 52 | end 53 | 54 | if( ~isa( data, 'double' ) ), 55 | error( 'data must be of class "double"' ); 56 | end 57 | 58 | if ~exist( 'edge', 'var' ), 59 | edge = data; 60 | elseif isempty( edge ), 61 | edge = data; 62 | end 63 | 64 | if( ndims( edge ) > 2 ), 65 | error( 'edge must be a greyscale image with size [ height, width ]' ); 66 | end 67 | 68 | if( ~isa( edge, 'double' ) ), 69 | error( 'edge must be of class "double"' ); 70 | end 71 | 72 | inputHeight = size( data, 1 ); 73 | inputWidth = size( data, 2 ); 74 | 75 | if ~exist( 'edgeMin', 'var' ), 76 | edgeMin = min( edge( : ) ); 77 | warning( 'edgeMin not set! Defaulting to: %f\n', edgeMin ); 78 | end 79 | 80 | if ~exist( 'edgeMax', 'var' ), 81 | edgeMax = max( edge( : ) ); 82 | warning( 'edgeMax not set! Defaulting to: %f\n', edgeMax ); 83 | end 84 | 85 | edgeDelta = edgeMax - edgeMin; 86 | 87 | if ~exist( 'sigmaSpatial', 'var' ), 88 | sigmaSpatial = min( inputWidth, inputHeight ) / 16; 89 | fprintf( 'Using default sigmaSpatial of: %f\n', sigmaSpatial ); 90 | end 91 | 92 | if ~exist( 'sigmaRange', 'var' ), 93 | sigmaRange = 0.1 * edgeDelta; 94 | fprintf( 'Using default sigmaRange of: %f\n', sigmaRange ); 95 | end 96 | 97 | if ~exist( 'samplingSpatial', 'var' ), 98 | samplingSpatial = sigmaSpatial; 99 | end 100 | 101 | if ~exist( 'samplingRange', 'var' ), 102 | samplingRange = sigmaRange; 103 | end 104 | 105 | if size( data ) ~= size( edge ), 106 | error( 'data and edge must be of the same size' ); 107 | end 108 | 109 | % parameters 110 | derivedSigmaSpatial = sigmaSpatial / samplingSpatial; 111 | derivedSigmaRange = sigmaRange / samplingRange; 112 | 113 | paddingXY = floor( 2 * derivedSigmaSpatial ) + 1; 114 | paddingZ = floor( 2 * derivedSigmaRange ) + 1; 115 | 116 | % allocate 3D grid 117 | downsampledWidth = floor( ( inputWidth - 1 ) / samplingSpatial ) + 1 + 2 * paddingXY; 118 | downsampledHeight = floor( ( inputHeight - 1 ) / samplingSpatial ) + 1 + 2 * paddingXY; 119 | downsampledDepth = floor( edgeDelta / samplingRange ) + 1 + 2 * paddingZ; 120 | 121 | gridData = zeros( downsampledHeight, downsampledWidth, downsampledDepth ); 122 | gridWeights = zeros( downsampledHeight, downsampledWidth, downsampledDepth ); 123 | 124 | % compute downsampled indices 125 | [ jj, ii ] = meshgrid( 0 : inputWidth - 1, 0 : inputHeight - 1 ); 126 | 127 | % ii = 128 | % 0 0 0 0 0 129 | % 1 1 1 1 1 130 | % 2 2 2 2 2 131 | 132 | % jj = 133 | % 0 1 2 3 4 134 | % 0 1 2 3 4 135 | % 0 1 2 3 4 136 | 137 | % so when iterating over ii( k ), jj( k ) 138 | % get: ( 0, 0 ), ( 1, 0 ), ( 2, 0 ), ... (down columns first) 139 | 140 | di = round( ii / samplingSpatial ) + paddingXY + 1; 141 | dj = round( jj / samplingSpatial ) + paddingXY + 1; 142 | dz = round( ( edge - edgeMin ) / samplingRange ) + paddingZ + 1; 143 | 144 | % perform scatter (there's probably a faster way than this) 145 | % normally would do downsampledWeights( di, dj, dk ) = 1, but we have to 146 | % perform a summation to do box downsampling 147 | for k = 1 : numel( dz ), 148 | 149 | dataZ = data( k ); % traverses the image column wise, same as di( k ) 150 | if ~isnan( dataZ ), 151 | 152 | dik = di( k ); 153 | djk = dj( k ); 154 | dzk = dz( k ); 155 | 156 | gridData( dik, djk, dzk ) = gridData( dik, djk, dzk ) + dataZ; 157 | gridWeights( dik, djk, dzk ) = gridWeights( dik, djk, dzk ) + 1; 158 | 159 | end 160 | end 161 | 162 | % make gaussian kernel 163 | kernelWidth = 2 * derivedSigmaSpatial + 1; 164 | kernelHeight = kernelWidth; 165 | kernelDepth = 2 * derivedSigmaRange + 1; 166 | 167 | halfKernelWidth = floor( kernelWidth / 2 ); 168 | halfKernelHeight = floor( kernelHeight / 2 ); 169 | halfKernelDepth = floor( kernelDepth / 2 ); 170 | 171 | [gridX, gridY, gridZ] = meshgrid( 0 : kernelWidth - 1, 0 : kernelHeight - 1, 0 : kernelDepth - 1 ); 172 | gridX = gridX - halfKernelWidth; 173 | gridY = gridY - halfKernelHeight; 174 | gridZ = gridZ - halfKernelDepth; 175 | gridRSquared = ( gridX .* gridX + gridY .* gridY ) / ( derivedSigmaSpatial * derivedSigmaSpatial ) + ( gridZ .* gridZ ) / ( derivedSigmaRange * derivedSigmaRange ); 176 | kernel = exp( -0.5 * gridRSquared ); 177 | 178 | % convolve 179 | blurredGridData = convn( gridData, kernel, 'same' ); 180 | blurredGridWeights = convn( gridWeights, kernel, 'same' ); 181 | 182 | % divide 183 | blurredGridWeights( blurredGridWeights == 0 ) = -2; % avoid divide by 0, won't read there anyway 184 | normalizedBlurredGrid = blurredGridData ./ blurredGridWeights; 185 | normalizedBlurredGrid( blurredGridWeights < -1 ) = 0; % put 0s where it's undefined 186 | 187 | % for debugging 188 | % blurredGridWeights( blurredGridWeights < -1 ) = 0; % put zeros back 189 | 190 | % upsample 191 | [ jj, ii ] = meshgrid( 0 : inputWidth - 1, 0 : inputHeight - 1 ); % meshgrid does x, then y, so output arguments need to be reversed 192 | % no rounding 193 | di = ( ii / samplingSpatial ) + paddingXY + 1; 194 | dj = ( jj / samplingSpatial ) + paddingXY + 1; 195 | dz = ( edge - edgeMin ) / samplingRange + paddingZ + 1; 196 | 197 | % interpn takes rows, then cols, etc 198 | % i.e. size(v,1), then size(v,2), ... 199 | output = interpn( normalizedBlurredGrid, di, dj, dz ); 200 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/bilinear_u.m: -------------------------------------------------------------------------------- 1 | function f = bilinear_u(k, numGroups, numClasses) 2 | %BILINEAR_U Create bilinear interpolation filters 3 | % BILINEAR_U(K, NUMGROUPS, NUMCLASSES) compute a square bilinear filter 4 | % of size k for deconv layer of depth numClasses and number of groups 5 | % numGroups 6 | 7 | factor = floor((k+1)/2) ; 8 | if rem(k,2)==1 9 | center = factor ; 10 | else 11 | center = factor + 0.5 ; 12 | end 13 | C = 1:k ; 14 | if numGroups ~= numClasses 15 | f = zeros(k,k,numGroups,numClasses) ; 16 | else 17 | f = zeros(k,k,1,numClasses) ; 18 | end 19 | 20 | for i =1:numClasses 21 | if numGroups ~= numClasses 22 | index = i ; 23 | else 24 | index = 1 ; 25 | end 26 | f(:,:,index,i) = (ones(1,k) - abs(C-center)./factor)'*(ones(1,k) - abs(C-center)./(factor)); 27 | end 28 | end 29 | 30 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/cnn_masknet.m: -------------------------------------------------------------------------------- 1 | function net = cnn_masknet 2 | 3 | net = createNetfromvgg(); 4 | net = vl_simplenn_tidy(net); 5 | net = vl_simplenn_move(net,'gpu'); 6 | net = dagnn.DagNN.fromSimpleNN(net, 'canonicalNames', true) ; 7 | 8 | % net.layers(1).block.pad = [0 0 0 0]; 9 | % net.layers(3).block.pad = [0 0 0 0]; 10 | % net.layers(5).block.poolSize = [3 3]; 11 | 12 | reshapeBlock = myReshape(); 13 | net.addLayer('myreshape', reshapeBlock, 'x35', 'x36'); 14 | 15 | % add first conv layer to connect with pool_4 16 | net.addLayer('skip_4', ... 17 | dagnn.Conv('size', [5 5 512 64], 'pad', [2 2 2 2]), ... 18 | 'x24', 'x37', {'skip_4_f','skip_4_b'}); 19 | 20 | f = net.getParamIndex('skip_4_f') ; 21 | net.params(f).value = 0.01*randn(5, 5, 512, 64, 'single') ; 22 | net.params(f).learningRate = 1 ; 23 | net.params(f).weightDecay = 1 ; 24 | 25 | f = net.getParamIndex('skip_4_b') ; 26 | net.params(f).value = zeros(1, 1, 64, 'single') ; 27 | net.params(f).learningRate = 2 ; 28 | net.params(f).weightDecay = 1 ; 29 | 30 | skip4_reluBlock = dagnn.ReLU() ; 31 | net.addLayer('relu_skip_4', skip4_reluBlock, 'x37', 'x38', {}) ; 32 | 33 | filters = single(bilinear_u(8, 1, 64)) ; 34 | net.addLayer('deconv4', ... 35 | dagnn.ConvTranspose(... 36 | 'size', size(filters), ... 37 | 'upsample', 4, ... 38 | 'crop', [1 3 1 3], ... 39 | 'hasBias', false), ... 40 | 'x38', 'x39', 'deconv4f') ; 41 | 42 | f = net.getParamIndex('deconv4f') ; 43 | net.params(f).value = filters ; 44 | net.params(f).learningRate = 0 ; 45 | net.params(f).weightDecay = 1 ; 46 | 47 | % add first conv layer to connect with pool_3 48 | net.addLayer('skip_3', ... 49 | dagnn.Conv('size', [5 5 256 64], 'pad', [2 2 2 2]), ... 50 | 'x17', 'x40', {'skip_3_f','skip_3_b'}); 51 | 52 | f = net.getParamIndex('skip_3_f') ; 53 | net.params(f).value = 0.01*randn(5, 5, 256, 64, 'single') ; 54 | net.params(f).learningRate = 1 ; 55 | net.params(f).weightDecay = 1 ; 56 | 57 | f = net.getParamIndex('skip_3_b') ; 58 | net.params(f).value = zeros(1, 1, 64, 'single') ; 59 | net.params(f).learningRate = 1 ; 60 | net.params(f).weightDecay = 1 ; 61 | 62 | skip3_reluBlock = dagnn.ReLU() ; 63 | net.addLayer('relu_skip_3', skip3_reluBlock, 'x40', 'x41', {}) ; 64 | 65 | filters = single(bilinear_u(5, 1, 64)) ; 66 | net.addLayer('deconv3', ... 67 | dagnn.ConvTranspose(... 68 | 'size', size(filters), ... 69 | 'upsample', 2, ... 70 | 'crop', [1 2 1 2], ... 71 | 'hasBias', false), ... 72 | 'x41', 'x42', 'deconv3f') ; 73 | 74 | f = net.getParamIndex('deconv3f') ; 75 | net.params(f).value = filters ; 76 | net.params(f).learningRate = 0 ; 77 | net.params(f).weightDecay = 1 ; 78 | 79 | 80 | % scale 2 net work 81 | net.addLayer('scale2Conv1', ... 82 | dagnn.Conv('size', [2 2 3 96], 'pad', [0 0 0 0], 'stride', [2 2]), ... 83 | 'input2', 'x43', {'scale2Conv1_f','scale2Conv1_b'}); 84 | 85 | f = net.getParamIndex('scale2Conv1_f') ; 86 | net.params(f).value = 0.01*randn(2, 2, 3, 96, 'single') ; 87 | net.params(f).learningRate = 1 ; 88 | net.params(f).weightDecay = 1 ; 89 | 90 | f = net.getParamIndex('scale2Conv1_b') ; 91 | net.params(f).value = zeros(1, 1, 96, 'single') ; 92 | net.params(f).learningRate = 2 ; 93 | net.params(f).weightDecay = 1 ; 94 | 95 | scale2Conv1_reluBlock = dagnn.ReLU() ; 96 | net.addLayer('relu_scale2Conv1', scale2Conv1_reluBlock, 'x43', 'x44', {}) ; 97 | 98 | net.addLayer('pool1_scale2', dagnn.Pooling('poolSize', [2 2], 'stride', [2 2]), 'x44', 'xfix', {}) ; 99 | 100 | scale2Conv1_concatBlock = dagnn.Concat(); 101 | net.addLayer('concat_scale2Conv1', scale2Conv1_concatBlock, {'xfix', 'x42'}, 'x45', {}) ; 102 | 103 | net.addLayer('scale2Conv2', ... 104 | dagnn.Conv('size', [5 5 160 64], 'pad', [2 2 2 2]), ... 105 | 'x45', 'x46', {'scale2Conv2_f','scale2Conv2_b'}); 106 | 107 | f = net.getParamIndex('scale2Conv2_f') ; 108 | net.params(f).value = 0.01*randn(5, 5, 160, 64, 'single') ; 109 | net.params(f).learningRate = 1 ; 110 | net.params(f).weightDecay = 1 ; 111 | 112 | f = net.getParamIndex('scale2Conv2_b') ; 113 | net.params(f).value = zeros(1, 1, 64, 'single') ; 114 | net.params(f).learningRate = 2 ; 115 | net.params(f).weightDecay = 1 ; 116 | 117 | scale2Conv2_reluBlock = dagnn.ReLU() ; 118 | net.addLayer('relu_scale2Conv2', scale2Conv2_reluBlock, 'x46', 'x47', {}) ; 119 | 120 | net.addLayer('scale2Conv3', ... 121 | dagnn.Conv('size', [5 5 64 64], 'pad', [2 2 2 2]), ... 122 | 'x47', 'x48', {'scale2Conv3_f','scale2Conv3_b'}); 123 | 124 | f = net.getParamIndex('scale2Conv3_f') ; 125 | net.params(f).value = 0.01*randn(5, 5, 64, 64, 'single') ; 126 | net.params(f).learningRate = 1 ; 127 | net.params(f).weightDecay = 1 ; 128 | 129 | f = net.getParamIndex('scale2Conv3_b') ; 130 | net.params(f).value = zeros(1, 1, 64, 'single') ; 131 | net.params(f).learningRate = 2 ; 132 | net.params(f).weightDecay = 1 ; 133 | 134 | scale2Conv3_reluBlock = dagnn.ReLU() ; 135 | net.addLayer('relu_scale2Conv3', scale2Conv3_reluBlock, 'x48', 'x49', {}) ; 136 | 137 | scale2Conv3_concatBlock = dagnn.Concat(); 138 | net.addLayer('concat_scale2Conv3', scale2Conv3_concatBlock, {'x49', 'x39'}, 'x50', {}) ; 139 | 140 | net.addLayer('scale2Conv4', ... 141 | dagnn.Conv('size', [5 5 128 64], 'pad', [2 2 2 2]), ... 142 | 'x50', 'x51', {'scale2Conv4_f','scale2Conv4_b'}); 143 | 144 | f = net.getParamIndex('scale2Conv4_f') ; 145 | net.params(f).value = 0.01*randn(5, 5, 128, 64, 'single') ; 146 | net.params(f).learningRate = 1 ; 147 | net.params(f).weightDecay = 1 ; 148 | 149 | f = net.getParamIndex('scale2Conv4_b') ; 150 | net.params(f).value = zeros(1, 1, 64, 'single') ; 151 | net.params(f).learningRate = 2 ; 152 | net.params(f).weightDecay = 1 ; 153 | 154 | scale2Conv4_reluBlock = dagnn.ReLU() ; 155 | net.addLayer('relu_scale2Conv4', scale2Conv4_reluBlock, 'x51', 'x52', {}) ; 156 | 157 | net.addLayer('scale2Conv5', ... 158 | dagnn.Conv('size', [5 5 64 64], 'pad', [2 2 2 2]), ... 159 | 'x52', 'x53', {'scale2Conv5_f','scale2Conv5_b'}); 160 | 161 | f = net.getParamIndex('scale2Conv5_f') ; 162 | net.params(f).value = 0.01*randn(5, 5, 64, 64, 'single') ; 163 | net.params(f).learningRate = 1 ; 164 | net.params(f).weightDecay = 1 ; 165 | 166 | f = net.getParamIndex('scale2Conv5_b') ; 167 | net.params(f).value = zeros(1, 1, 64, 'single') ; 168 | net.params(f).learningRate = 2 ; 169 | net.params(f).weightDecay = 1 ; 170 | 171 | scale2Conv5_reluBlock = dagnn.ReLU() ; 172 | net.addLayer('relu_scale2Conv5', scale2Conv5_reluBlock, 'x53', 'x54', {}) ; 173 | 174 | scale2Conv5_concatBlock = dagnn.Concat(); 175 | net.addLayer('concat_scale2Conv5', scale2Conv5_concatBlock, {'x54', 'x36'}, 'x55', {}) ; 176 | 177 | net.addLayer('scale2Conv6', ... 178 | dagnn.Conv('size', [5 5 67 64], 'pad', [2 2 2 2]), ... 179 | 'x55', 'x56', {'scale2Conv6_f','scale2Conv6_b'}); 180 | 181 | f = net.getParamIndex('scale2Conv6_f') ; 182 | net.params(f).value = 0.01*randn(5, 5, 67, 64, 'single') ; 183 | net.params(f).learningRate = 1 ; 184 | net.params(f).weightDecay = 1 ; 185 | 186 | f = net.getParamIndex('scale2Conv6_b') ; 187 | net.params(f).value = zeros(1, 1, 64, 'single') ; 188 | net.params(f).learningRate = 2 ; 189 | net.params(f).weightDecay = 1 ; 190 | 191 | scale2Conv6_reluBlock = dagnn.ReLU() ; 192 | net.addLayer('relu_scale2Conv6', scale2Conv6_reluBlock, 'x56', 'x57', {}) ; 193 | 194 | net.addLayer('scale2Conv7', ... 195 | dagnn.Conv('size', [5 5 64 64], 'pad', [2 2 2 2]), ... 196 | 'x57', 'x58', {'scale2Conv7_f','scale2Conv7_b'}); 197 | 198 | f = net.getParamIndex('scale2Conv7_f') ; 199 | net.params(f).value = 0.01*randn(5, 5, 64, 64, 'single') ; 200 | net.params(f).learningRate = 1 ; 201 | net.params(f).weightDecay = 1 ; 202 | 203 | f = net.getParamIndex('scale2Conv7_b') ; 204 | net.params(f).value = zeros(1, 1, 64, 'single') ; 205 | net.params(f).learningRate = 2 ; 206 | net.params(f).weightDecay = 1 ; 207 | 208 | scale2Conv7_reluBlock = dagnn.ReLU() ; 209 | net.addLayer('relu_scale2Conv7', scale2Conv7_reluBlock, 'x58', 'x59', {}) ; 210 | 211 | net.addLayer('scale2Conv8', ... 212 | dagnn.Conv('size', [5 5 64 64], 'pad', [2 2 2 2]), ... 213 | 'x59', 'x60', {'scale2Conv8_f','scale2Conv8_b'}); 214 | 215 | f = net.getParamIndex('scale2Conv8_f') ; 216 | net.params(f).value = 0.01*randn(5, 5, 64, 64, 'single') ; 217 | net.params(f).learningRate = 1 ; 218 | net.params(f).weightDecay = 1 ; 219 | 220 | f = net.getParamIndex('scale2Conv8_b') ; 221 | net.params(f).value = zeros(1, 1, 64, 'single') ; 222 | net.params(f).learningRate = 2 ; 223 | net.params(f).weightDecay = 1 ; 224 | 225 | scale2Conv8_reluBlock = dagnn.ReLU() ; 226 | net.addLayer('relu_scale2Conv8', scale2Conv8_reluBlock, 'x60', 'x61', {}) ; 227 | 228 | net.addLayer('scale2Conv9', ... 229 | dagnn.Conv('size', [5 5 64 64], 'pad', [2 2 2 2]), ... 230 | 'x61', 'x62', {'scale2Conv9_f','scale2Conv9_b'}); 231 | 232 | f = net.getParamIndex('scale2Conv9_f') ; 233 | net.params(f).value = 0.01*randn(5, 5, 64, 64, 'single') ; 234 | net.params(f).learningRate = 1 ; 235 | net.params(f).weightDecay = 1 ; 236 | 237 | f = net.getParamIndex('scale2Conv9_b') ; 238 | net.params(f).value = zeros(1, 1, 64, 'single') ; 239 | net.params(f).learningRate = 2 ; 240 | net.params(f).weightDecay = 1 ; 241 | 242 | scale2Conv9_reluBlock = dagnn.ReLU() ; 243 | net.addLayer('relu_scale2Conv9', scale2Conv9_reluBlock, 'x62', 'x63', {}) ; 244 | 245 | net.addLayer('scale2Conv10', ... 246 | dagnn.Conv('size', [5 5 64 2], 'pad', [2 2 2 2]), ... 247 | 'x63', 'prediction', {'scale2Conv10_f','scale2Conv10_b'}); 248 | 249 | f = net.getParamIndex('scale2Conv10_f') ; 250 | net.params(f).value = 0.01*randn(5, 5, 64, 2, 'single') ; 251 | net.params(f).learningRate = 1 ; 252 | net.params(f).weightDecay = 1 ; 253 | 254 | f = net.getParamIndex('scale2Conv10_b') ; 255 | net.params(f).value = zeros(1, 1, 2, 'single') ; 256 | net.params(f).learningRate = 2 ; 257 | net.params(f).weightDecay = 1 ; 258 | 259 | net.vars(net.getVarIndex('prediction')).precious = 1 ; 260 | % Add loss layer 261 | % net.addLayer('objective', ... 262 | % Loss(), ... 263 | % {'prediction', 'label'}, 'objective') ; 264 | 265 | % % Add accuracy layer 266 | % net.addLayer('accuracy', ... 267 | % myAccuracy(), ... 268 | % {'prediction', 'label'}, 'accuracy') ; 269 | % net.addLayer('error', dagnn.Loss('loss', 'classerror'), {'pred','label'}, 'error'); 270 | 271 | % softmax loss: 272 | net.addLayer('loss', dagnn.Loss('loss', 'softmaxlog'), {'prediction','label'}, 'objective'); 273 | net.addLayer('error', dagnn.Loss('loss', 'classerror'), {'prediction','label'}, 'error'); 274 | 275 | %setLearningRate(net); 276 | 277 | save H:\matconvnet-1.0-beta24_20171016\matconvnet-1.0-beta24\masknetwork\data\net net 278 | end 279 | 280 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/cnn_masknet_datasetup.m: -------------------------------------------------------------------------------- 1 | %datadir=load('G:\20170619dataprocess\Grass-master\code\gencode.mat'); 2 | %datadir=load('G:\20170619dataprocess\Grass-master\code\trainingdata.mat'); 3 | datadir=load('H:\backgroundadd\frontdata\maskdata.mat'); 4 | dataset=datadir.maskdata; 5 | inputSize=[224,224,3]; 6 | imdb.images.data=[]; 7 | imdb.images.labels=[]; 8 | 9 | imdb.images.set=[]; 10 | imdb.meta.sets={'train','val','test'}; 11 | image_counter=0; 12 | trainratio=0.85; 13 | imageindex=0; 14 | 15 | imdb.images.data=single(dataset.raw); 16 | imdb.images.labels=single(dataset.label); 17 | for i=1:length(dataset.raw) 18 | % image=single(dataset.raw(:,:,:,i)); 19 | % imdb.images.data(:,:,:,i)=image; 20 | if i<=length(dataset.raw)*trainratio 21 | imdb.images.set(end+1)=1; 22 | else 23 | imdb.images.set(end+1)=3; 24 | end 25 | end 26 | dataMean=mean(imdb.images.data,4); 27 | 28 | imdb.images.data=single(bsxfun(@minus,imdb.images.data,dataMean)); 29 | imdb.images.data_mean=dataMean; 30 | %save image2box\imdbV2 imdb; 31 | save H:\matconvnet-1.0-beta24_20171026\matconvnet-1.0-beta24\masknetwork\data_modify\imdb imdb; -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/cnn_masknet_train.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/masknetwork/cnn_masknet_train.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/createNetfromvgg.m: -------------------------------------------------------------------------------- 1 | function net = createNetfromvgg() 2 | 3 | %vggnet = load('../../data/imagenet-vgg-verydeep-16.mat') ; 4 | vggnet=load('data/imagenet-vgg-verydeep-16.mat'); 5 | net.layers = vggnet.layers(1:33); 6 | 7 | net.layers{end-1}.filters = 1.0/100*randn(7,7,512,4096, 'single'); 8 | net.layers{end-1}.biases = zeros(1,4096,'single'); 9 | 10 | net.layers{end+1} = struct('type', 'dropout', ... 11 | 'name', sprintf('dropout%s', '6'), ... 12 | 'rate', 0.5) ; 13 | net.layers{end+1} = vggnet.layers{34}; 14 | 15 | % net.layers{end+1} = struct('type', 'conv', ... 16 | % 'name', sprintf('fc%s', '7'), ... 17 | % 'size', [1,1,4096,9408], 'pad', [0 0 0 0], ... 18 | % 'stride', [1,1]); 19 | % net.addLayer('fc7', ... 20 | % dagnn.Conv('size', [1,1,4096,9408], 'pad', [0 0 0 0]), ... 21 | % 'x34', 'x35', {'fc7f','fc7b'}); 22 | 23 | net.layers{end}.biases = zeros(1,9408,'single'); 24 | net.layers{end}.filters = 0.01*randn(1,1,4096,9408,'single'); 25 | net.layers{end}.weights{1}=0.01*randn(1,1,4096,9408,'single'); 26 | net.layers{end}.weights{2}=randn(9408,1,'single'); 27 | end 28 | 29 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/depth2pc_2.m: -------------------------------------------------------------------------------- 1 | % 2 | mySetup(); 3 | % 4 | load('../../save/trainDAG_final_01/final03_FT-net-3.mat', 'net') ; 5 | net = dagnn.DagNN.loadobj(net) ; 6 | load('../../data/splits.mat'); 7 | net.move('cpu') ; 8 | net.mode = 'test' ; 9 | % 10 | imgpath = 'E:\Work\Indoor scene\semantic segmentation\ucm_seg_cvpr\stanford\images'; 11 | gtdepthspath = 'L:\work\Indoor scene\code\depthtransfer\MatlabCode\datasets\datasetNYU\depths'; 12 | savepcdir = '../../estpc_final_03'; 13 | mkdir(savepcdir); 14 | 15 | rel_errors = double(0); 16 | log_errors = double(0); 17 | rm_errors = double(0); 18 | testNum = 654; 19 | 20 | rel_list = zeros(testNum, 1, 'single'); 21 | log_list = zeros(testNum, 1, 'single'); 22 | rm_list = zeros(testNum, 1, 'single'); 23 | 24 | t1 = 0; 25 | t2 = 0; 26 | t3 = 0; 27 | for ii = 1:testNum 28 | disp(ii); 29 | id = num2str(testNdxs(ii), '%06d'); 30 | im = imread([imgpath '\' id '.jpg']) ; 31 | im_ = single(im) ; % note: 255 range 32 | im_1 = imresize(im_, [240 320], 'bilinear'); 33 | im_1 = im_1(22:235,20:300,:); 34 | im_1(:,:,1) = im_1(:,:,1) - net.meta.imgRbgMean.r; 35 | im_1(:,:,2) = im_1(:,:,2) - net.meta.imgRbgMean.g; 36 | im_1(:,:,3) = im_1(:,:,3) - net.meta.imgRbgMean.b; 37 | 38 | 39 | im_2 = imresize(im_, [240 320], 'bilinear'); 40 | im_2 = im_2(22:235,20:300,:); 41 | im_2 = imresize(im_2, [107 140], 'bilinear'); 42 | im_2(:,:,1) = im_2(:,:,1) - net.meta.imgRbgMean.r; 43 | im_2(:,:,2) = im_2(:,:,2) - net.meta.imgRbgMean.g; 44 | im_2(:,:,3) = im_2(:,:,3) - net.meta.imgRbgMean.b; 45 | 46 | [xx,yy] = meshgrid(1:140, 1:107); 47 | xx = single(xx)/140; 48 | yy = single(yy)/107; 49 | 50 | coordmap = zeros(107,140,2,1, 'single'); 51 | coordmap(:,:,1,1) = xx; 52 | coordmap(:,:,2,1) = yy; 53 | 54 | % run the CNN 55 | predVar = net.getVarIndex('prediction') ; 56 | inputVar = 'input' ; 57 | net.eval({inputVar, im_1, 'input2', im_2}) ; 58 | crop_depth_1 = gather(net.vars(predVar).value) ; 59 | crop_depth_1 = power(10,crop_depth_1); 60 | crop_depth_1 = imresize(crop_depth_1, [427 561], 'bilinear'); 61 | 62 | 63 | im_1 = flip(im_1, 2); 64 | im_2 = flip(im_2,2); 65 | net.eval({inputVar, im_1, 'input2', im_2}) ; 66 | crop_depth_2 = gather(net.vars(predVar).value) ; 67 | crop_depth_2 = power(10,crop_depth_2); 68 | crop_depth_2 = imresize(crop_depth_2, [427 561], 'bilinear'); 69 | crop_depth_2 = flip(crop_depth_2,2); 70 | 71 | crop_depths = (crop_depth_1+crop_depth_2)/2; 72 | % crop_depths = crop_depth_1; 73 | load([gtdepthspath '\' id '.mat']); 74 | gt_depth = double(depth(45:471, 41:601))/1000; 75 | 76 | % load([savedepthpath '\' id '.mat']); 77 | % crop_depths = double(crop_depths); 78 | 79 | 80 | % for mm = 1:427 81 | % for nn = 1:561 82 | % rel_errors = rel_errors+abs(crop_depths(mm,nn)-gt_depth(mm,nn))/gt_depth(mm,nn); 83 | % log_errors = log_errors+abs(log10(crop_depths(mm,nn))-log10(gt_depth(mm,nn))); 84 | % rm_errors = rm_errors+(crop_depths(mm,nn)-gt_depth(mm,nn))^2; 85 | % end 86 | % end 87 | 88 | rel_mean = abs((crop_depths-gt_depth))./gt_depth; 89 | rel_mean = sum(rel_mean(:)); 90 | rel_errors = rel_errors+rel_mean; 91 | log_mean = abs(log10(crop_depths)-log10(gt_depth)); 92 | log_mean = sum(log_mean(:)); 93 | log_errors = log_errors+log_mean; 94 | rm_mean = (crop_depths-gt_depth).^2; 95 | rm_mean = sum(rm_mean(:)); 96 | rm_errors = rm_errors+rm_mean; 97 | % save([savedepthpath '\' id '.mat'], 'crop_depths'); 98 | rel_list(ii) = rel_mean; 99 | log_list(ii) = log_mean; 100 | rel_list(ii) = rel_mean; 101 | 102 | rel_d1 = crop_depths./gt_depth; 103 | rel_d2 = gt_depth./crop_depths; 104 | 105 | max_rel_d = max(rel_d1,rel_d2); 106 | t1 = t1 + numel(find(max_rel_d < 1.25)); 107 | t2 = t2 + numel(find(max_rel_d < 1.5625)); 108 | t3 = t3 + numel(find(max_rel_d < 1.953125)); 109 | 110 | 111 | 112 | depths = zeros(480,640); 113 | depths(45:471, 41:601) = crop_depths; 114 | 115 | points3d = rgb_plane2rgb_world(depths); 116 | 117 | savefile = [savepcdir '\' id '.ply']; 118 | plywrite(savefile,points3d); 119 | 120 | end 121 | 122 | rel_f = rel_errors/427/561/testNum; 123 | log_f = log_errors/427/561/testNum; 124 | rm_f = sqrt(rm_errors/427/561/testNum); 125 | fprintf('rel log rm errors are %.4f %.4f %.4f\n', rel_f, log_f, rm_f); 126 | 127 | ft1 = t1/(427*561)/testNum; 128 | ft2 = t2/(427*561)/testNum; 129 | ft3 = t3/(427*561)/testNum; 130 | 131 | fprintf('<1.25 <1.25^2 <1.25^3 are %.4f %.4f %.4f\n', ft1, ft2, ft3); 132 | 133 | 134 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/getBatch.m: -------------------------------------------------------------------------------- 1 | function y = getBatch(imdb, batch, mode, opts ) 2 | 3 | batchsize = numel(batch); 4 | im = zeros(232,310,3,batchsize, 'uint8'); 5 | depths = zeros(111,150,1,batchsize, 'single'); 6 | 7 | if strcmp(mode,'train') 8 | for ii = 1:batchsize 9 | opid = randi(3); 10 | switch opid 11 | %%original 12 | case 1 13 | im_temp = imdb.nyudata.images(5:236,6:315,:,batch(ii)); 14 | depths_temp = single(imdb.nyudata.depths(:,:,:,batch(ii)))/1000; 15 | depths_temp = imresize(depths_temp,2,'nearest'); 16 | depths_temp = depths_temp(11:230,11:310,:); 17 | depths_temp = imresize(depths_temp, [111 150], 'nearest'); 18 | depths_temp(depths_temp >= 9.99) = 1000; 19 | depths_temp(depths_temp == 0) = 1000; 20 | im(:,:,:,ii) = im_temp; 21 | depths(:,:,:,ii) = depths_temp; 22 | %%rotate 23 | case 2 24 | im_temp = imdb.nyudata.images(:,:,:,batch(ii)); 25 | im_temp = imresize(im_temp, 1.2, 'bilinear'); 26 | depths_temp = single(imdb.nyudata.depths(:,:,:,batch(ii)))/1000; 27 | depths_temp = imresize(depths_temp,2,'nearest'); 28 | depths_temp = imresize(depths_temp,1.2,'nearest'); 29 | depths_temp = depths_temp/1.2; 30 | rot_angle = 1+4*rand; 31 | rotid = randi(2); 32 | switch rotid 33 | case 1 34 | im_temp_rot = imrotate(im_temp, rot_angle, 'bilinear'); 35 | depths_temp_rot = imrotate(depths_temp, rot_angle, 'nearest'); 36 | start_y = randi([38 51]); 37 | start_x = randi([32 69]); 38 | im_temp = im_temp_rot(start_y:start_y+231, start_x:start_x+309,:); 39 | depths_temp = depths_temp_rot(start_y+6:start_y+231-6, start_x+5:start_x+309-5,:); 40 | depths_temp = imresize(depths_temp, [111 150], 'nearest'); 41 | depths_temp(depths_temp >= 9.99) = 1000; 42 | depths_temp(depths_temp == 0) = 1000; 43 | im(:,:,:,ii) = im_temp; 44 | depths(:,:,:,ii) = depths_temp; 45 | case 2 46 | rot_angle = -rot_angle; 47 | im_temp_rot = imrotate(im_temp, rot_angle, 'bilinear'); 48 | depths_temp_rot = imrotate(depths_temp, rot_angle, 'nearest'); 49 | start_y = randi([38 51]); 50 | start_x = randi([32 69]); 51 | im_temp = im_temp_rot(start_y:start_y+231, start_x:start_x+309,:); 52 | depths_temp = depths_temp_rot(start_y+6:start_y+231-6, start_x+5:start_x+309-5,:); 53 | depths_temp = imresize(depths_temp, [111 150], 'nearest'); 54 | depths_temp(depths_temp >= 9.99) = 1000; 55 | depths_temp(depths_temp == 0) = 1000; 56 | im(:,:,:,ii) = im_temp; 57 | depths(:,:,:,ii) = depths_temp; 58 | end 59 | %%scale and translate 60 | case 3 61 | while(1) 62 | im_temp = imdb.nyudata.images(5:236,6:315,:,batch(ii)); 63 | depths_temp = single(imdb.nyudata.depths(:,:,:,batch(ii)))/1000; 64 | depths_temp = imresize(depths_temp,2,'nearest'); 65 | depths_temp = depths_temp(5:236,6:315); 66 | sc = 1.05+0.45*rand; 67 | im_temp = imresize(im_temp, sc); 68 | depths_temp = imresize(depths_temp, sc, 'nearest'); 69 | depths_temp = depths_temp/sc; 70 | height = size(im_temp,1); 71 | width = size(im_temp,2); 72 | start_x = randi([1 width-310+1]); 73 | start_y = randi([1 height-232+1]); 74 | im_temp = im_temp(start_y:start_y+231, start_x:start_x+309, :); 75 | depths_temp = depths_temp(start_y+6:start_y+231-6, start_x+5:start_x+309-5, :); 76 | depths_temp = imresize(depths_temp, [111 150], 'nearest'); 77 | depths_temp(depths_temp >= 9.99) = 1000; 78 | depths_temp(depths_temp == 0) = 1000; 79 | im(:,:,:,ii) = im_temp; 80 | depths(:,:,:,ii) = depths_temp; 81 | 82 | validdepthnum = numel(find(depths_temp<10)); 83 | if validdepthnum > 500 84 | break; 85 | end 86 | end 87 | 88 | end 89 | end 90 | 91 | setim = zeros(232,310,3,3*batchsize, 'uint8'); 92 | setdepths = zeros(111,150,1,3*batchsize, 'single'); 93 | for ii = 1:batchsize 94 | setim(:,:,:,3*(ii-1)+1) = im(:,:,:,ii); 95 | setdepths(:,:,:,3*(ii-1)+1) = depths(:,:,:,ii); 96 | colorid = randi(6); 97 | %%brightness and contrast 98 | switch colorid 99 | case 1 100 | %%increase brightness 101 | sc = 0.1+0.2*rand; 102 | setim(:,:,:,3*(ii-1)+2) = imadjust(im(:,:,:,ii), [0 1], [sc 1]); 103 | case 2 104 | %%decrease brightness 105 | sc = 0.9-0.3*rand; 106 | setim(:,:,:,3*(ii-1)+2) = imadjust(im(:,:,:,ii), [0 1], [0 sc]); 107 | case 3 108 | %%increase contrast 109 | sc = 0.9-0.3*rand; 110 | setim(:,:,:,3*(ii-1)+2) = imadjust(im(:,:,:,ii), [0 sc], [0 1]); 111 | case 4 112 | %%decrease contrast 113 | sc = 0.1+0.2*rand; 114 | setim(:,:,:,3*(ii-1)+2) = imadjust(im(:,:,:,ii), [sc 1], [0 1]); 115 | case {5,6} 116 | sc = 0.8+0.4*rand(1,3); 117 | setim(:,:,1,3*(ii-1)+2) = im(:,:,1,ii)*sc(1); 118 | setim(:,:,2,3*(ii-1)+2) = im(:,:,2,ii)*sc(2); 119 | setim(:,:,3,3*(ii-1)+2) = im(:,:,3,ii)*sc(3); 120 | end 121 | setdepths(:,:,:,3*(ii-1)+2) = depths(:,:,:,ii); 122 | setim(:,:,:,3*(ii-1)+3) = flip(im(:,:,:,ii), 2); 123 | setdepths(:,:,:,3*(ii-1)+3) = flip(depths(:,:,:,ii), 2); 124 | end 125 | 126 | else 127 | setim = imdb.nyudata.images(5:236,6:315,:,batch); 128 | setdepths = single(imdb.nyudata.depths(:,:,:,batch))/1000 ; 129 | setdepths = imresize(setdepths,2,'bilinear'); 130 | setdepths = setdepths(11:230,11:310,:,:); 131 | setdepths = imresize(setdepths, [111 150], 'bilinear'); 132 | end 133 | 134 | setim = single(setim); 135 | setim(:,:,1,:) = setim(:,:,1,:)-opts.imgRbgMean.r; 136 | setim(:,:,2,:) = setim(:,:,2,:)-opts.imgRbgMean.g; 137 | setim(:,:,3,:) = setim(:,:,3,:)-opts.imgRbgMean.b; 138 | 139 | if opts.useGpu 140 | setim = gpuArray(setim); 141 | end 142 | 143 | y = {'input', setim, 'input2', setim, 'label', setdepths} ; 144 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/getBatch_old.m: -------------------------------------------------------------------------------- 1 | function y = getBatch_old(imdb, batch, mode, opts ) 2 | 3 | im = imdb.nyudata.images(:,:,:,batch); 4 | im2 = imresize(im, [53 70], 'bilinear'); 5 | depths = single(imdb.nyudata.depths(:,:,:,batch))/1000 ; 6 | depths = imresize(depths, [53 70], 'bilinear'); 7 | 8 | if strcmp(mode,'train') 9 | 10 | batchsize = numel(batch); 11 | 12 | if batchsize > 1 13 | 14 | %flip 15 | flipindex = 2:2:batchsize; 16 | if numel(flipindex)>0 17 | im(:,:,:,flipindex) = flip(im(:,:,:,flipindex),2); 18 | im2(:,:,:,flipindex) = flip(im2(:,:,:,flipindex),2); 19 | depths(:,:,:,flipindex) = flip(depths(:,:,:,flipindex),2); 20 | end 21 | %brightness 22 | % brightindex = 1:3:batchsize; 23 | % w = 0.75+0.5*rand; 24 | % im(:,:,:,brightindex) = im(:,:,:,brightindex)*w; 25 | % im2(:,:,:,brightindex) = im2(:,:,:,brightindex)*w; 26 | 27 | %casting on R G B channel 28 | % castingindex = 3:5:batchsize; 29 | % w1 = 0.85+0.3*rand; 30 | % w2 = 0.85+0.3*rand; 31 | % w3 = 0.85+0.3*rand; 32 | % im(:,:,1,castingindex) = im(:,:,1,castingindex)*w1; 33 | % im(:,:,2,castingindex) = im(:,:,2,castingindex)*w2; 34 | % im(:,:,3,castingindex) = im(:,:,3,castingindex)*w3; 35 | % 36 | % im2(:,:,1,castingindex) = im2(:,:,1,castingindex)*w1; 37 | % im2(:,:,2,castingindex) = im2(:,:,2,castingindex)*w2; 38 | % im2(:,:,3,castingindex) = im2(:,:,3,castingindex)*w3; 39 | 40 | %add noise 41 | % noiseindex = 2:5:batchsize; 42 | % w = 0.00025+0.0005*rand; 43 | % im(:,:,:,noiseindex) = imnoise(im(:,:,:,noiseindex),'gaussian', 0, w); 44 | end 45 | 46 | end 47 | 48 | im = single(im); 49 | im(:,:,1,:) = im(:,:,1,:)-opts.imgRbgMean.r; 50 | im(:,:,2,:) = im(:,:,2,:)-opts.imgRbgMean.g; 51 | im(:,:,3,:) = im(:,:,3,:)-opts.imgRbgMean.b; 52 | 53 | im2 = single(im2); 54 | im2(:,:,1,:) = im2(:,:,1,:)-opts.imgRbgMean.r; 55 | im2(:,:,2,:) = im2(:,:,2,:)-opts.imgRbgMean.g; 56 | im2(:,:,3,:) = im2(:,:,3,:)-opts.imgRbgMean.b; 57 | 58 | if opts.useGpu 59 | im = gpuArray(im) ; 60 | im2 = gpuArray(im2) ; 61 | end 62 | 63 | y = {'input', im, 'input2', im2, 'label', depths} ; 64 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/myAccuracy.m: -------------------------------------------------------------------------------- 1 | classdef myAccuracy < dagnn.Loss 2 | 3 | properties (Transient) 4 | rel_error = 0 5 | log_error = 0 6 | rm_error = 0 7 | end 8 | 9 | methods 10 | function outputs = forward(obj, inputs, params) 11 | est_d_values = gather(inputs{1}); 12 | est_d_values_pow = power(10,est_d_values); 13 | gt_d_values_pow = inputs{2}; 14 | temp_m = inputs{2}==1000; 15 | est_d_values_pow(temp_m) = gt_d_values_pow(temp_m); 16 | marks = ones(size(inputs{2})); 17 | marks(temp_m) = 0; 18 | pixelnum = sum(sum(sum(marks,1),2)); 19 | % pixelnum = size(inputs{1},1)*size(inputs{1},2); 20 | 21 | rel_error_tmp = sum(sum(sum((abs(est_d_values_pow-gt_d_values_pow)./gt_d_values_pow),1),2)); 22 | log_error_tmp = sum(sum(sum((abs(log10(est_d_values_pow)-log10(inputs{2}))),1),2)); 23 | rm_error_tmp = sum(sum(sum((est_d_values_pow-gt_d_values_pow).^2,1),2)); 24 | 25 | obj.rel_error = obj.rel_error + rel_error_tmp; 26 | obj.log_error = obj.log_error + log_error_tmp; 27 | obj.rm_error = obj.rm_error + rm_error_tmp; 28 | 29 | obj.numAveraged = obj.numAveraged + pixelnum ; 30 | outputs{1} = obj.average ; 31 | 32 | end 33 | 34 | function [derInputs, derParams] = backward(obj, inputs, params, derOutputs) 35 | derInputs{1} = [] ; 36 | derInputs{2} = [] ; 37 | derParams = {} ; 38 | end 39 | 40 | function reset(obj) 41 | obj.rel_error = 0 ; 42 | obj.log_error = 0 ; 43 | obj.rm_error = 0 ; 44 | obj.average = [0;0;0] ; 45 | obj.numAveraged = 0 ; 46 | end 47 | 48 | function recalerror(obj) 49 | obj.rel_error = obj.rel_error/obj.numAveraged; 50 | obj.log_error = obj.log_error/obj.numAveraged; 51 | obj.rm_error = sqrt(obj.rm_error/obj.numAveraged); 52 | obj.average = [obj.rel_error; obj.log_error; obj.rm_error] ; 53 | end 54 | 55 | function str = toString(obj) 56 | str = sprintf('rel_error:%.3f, log_error:%.3f, rm_error:%.3f', ... 57 | obj.rel_error, obj.log_error, obj.rm_error) ; 58 | end 59 | 60 | function obj = myAccuracy(varargin) 61 | obj.load(varargin) ; 62 | end 63 | end 64 | 65 | 66 | end 67 | 68 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/myDAGnet.m: -------------------------------------------------------------------------------- 1 | function net = myDAGnet 2 | 3 | net = createNetfromvgg(); 4 | net = vl_simplenn_tidy(net) ; 5 | net = dagnn.DagNN.fromSimpleNN(net, 'canonicalNames', true) ; 6 | 7 | net.layers(1).block.pad = [0 0 0 0]; 8 | net.layers(3).block.pad = [0 0 0 0]; 9 | net.layers(5).block.poolSize = [3 3]; 10 | 11 | reshapeBlock = myReshape(); 12 | net.addLayer('myreshape', reshapeBlock, 'x35', 'x36'); 13 | 14 | % add first conv layer to connect with pool_4 15 | net.addLayer('skip_4', ... 16 | dagnn.Conv('size', [5 5 512 64], 'pad', [2 2 2 2]), ... 17 | 'x24', 'x37', {'skip_4_f','skip_4_b'}); 18 | 19 | f = net.getParamIndex('skip_4_f') ; 20 | net.params(f).value = 0.01*randn(5, 5, 512, 64, 'single') ; 21 | net.params(f).learningRate = 1 ; 22 | net.params(f).weightDecay = 1 ; 23 | 24 | f = net.getParamIndex('skip_4_b') ; 25 | net.params(f).value = zeros(1, 1, 64, 'single') ; 26 | net.params(f).learningRate = 2 ; 27 | net.params(f).weightDecay = 1 ; 28 | 29 | skip4_reluBlock = dagnn.ReLU() ; 30 | net.addLayer('relu_skip_4', skip4_reluBlock, 'x37', 'x38', {}) ; 31 | 32 | filters = single(bilinear_u(8, 1, 64)) ; 33 | net.addLayer('deconv4', ... 34 | dagnn.ConvTranspose(... 35 | 'size', size(filters), ... 36 | 'upsample', 4, ... 37 | 'crop', [2 3 2 3], ... 38 | 'hasBias', false), ... 39 | 'x38', 'x39', 'deconv4f') ; 40 | 41 | f = net.getParamIndex('deconv4f') ; 42 | net.params(f).value = filters ; 43 | net.params(f).learningRate = 0 ; 44 | net.params(f).weightDecay = 1 ; 45 | 46 | % add first conv layer to connect with pool_3 47 | net.addLayer('skip_3', ... 48 | dagnn.Conv('size', [5 5 256 64], 'pad', [2 2 2 2]), ... 49 | 'x17', 'x40', {'skip_3_f','skip_3_b'}); 50 | 51 | f = net.getParamIndex('skip_3_f') ; 52 | net.params(f).value = 0.01*randn(5, 5, 256, 64, 'single') ; 53 | net.params(f).learningRate = 1 ; 54 | net.params(f).weightDecay = 1 ; 55 | 56 | f = net.getParamIndex('skip_3_b') ; 57 | net.params(f).value = zeros(1, 1, 64, 'single') ; 58 | net.params(f).learningRate = 1 ; 59 | net.params(f).weightDecay = 1 ; 60 | 61 | skip3_reluBlock = dagnn.ReLU() ; 62 | net.addLayer('relu_skip_3', skip3_reluBlock, 'x40', 'x41', {}) ; 63 | 64 | filters = single(bilinear_u(4, 1, 64)) ; 65 | net.addLayer('deconv3', ... 66 | dagnn.ConvTranspose(... 67 | 'size', size(filters), ... 68 | 'upsample', 2, ... 69 | 'crop', [1 2 1 2], ... 70 | 'hasBias', false), ... 71 | 'x41', 'x42', 'deconv3f') ; 72 | 73 | f = net.getParamIndex('deconv3f') ; 74 | net.params(f).value = filters ; 75 | net.params(f).learningRate = 0 ; 76 | net.params(f).weightDecay = 1 ; 77 | 78 | 79 | % scale 2 net work 80 | net.addLayer('scale2Conv1', ... 81 | dagnn.Conv('size', [9 9 3 96], 'pad', [0 0 0 0], 'stride', [2 2]), ... 82 | 'input2', 'x43', {'scale2Conv1_f','scale2Conv1_b'}); 83 | 84 | f = net.getParamIndex('scale2Conv1_f') ; 85 | net.params(f).value = 0.01*randn(9, 9, 3, 96, 'single') ; 86 | net.params(f).learningRate = 1 ; 87 | net.params(f).weightDecay = 1 ; 88 | 89 | f = net.getParamIndex('scale2Conv1_b') ; 90 | net.params(f).value = zeros(1, 1, 96, 'single') ; 91 | net.params(f).learningRate = 2 ; 92 | net.params(f).weightDecay = 1 ; 93 | 94 | scale2Conv1_reluBlock = dagnn.ReLU() ; 95 | net.addLayer('relu_scale2Conv1', scale2Conv1_reluBlock, 'x43', 'x44', {}) ; 96 | 97 | net.addLayer('pool1_scale2', dagnn.Pooling('poolSize', [3 3], 'stride', [2 2]), 'x44', 'xfix', {}) ; 98 | 99 | scale2Conv1_concatBlock = dagnn.Concat(); 100 | net.addLayer('concat_scale2Conv1', scale2Conv1_concatBlock, {'xfix', 'x42'}, 'x45', {}) ; 101 | 102 | net.addLayer('scale2Conv2', ... 103 | dagnn.Conv('size', [5 5 160 64], 'pad', [2 2 2 2]), ... 104 | 'x45', 'x46', {'scale2Conv2_f','scale2Conv2_b'}); 105 | 106 | f = net.getParamIndex('scale2Conv2_f') ; 107 | net.params(f).value = 0.01*randn(5, 5, 160, 64, 'single') ; 108 | net.params(f).learningRate = 1 ; 109 | net.params(f).weightDecay = 1 ; 110 | 111 | f = net.getParamIndex('scale2Conv2_b') ; 112 | net.params(f).value = zeros(1, 1, 64, 'single') ; 113 | net.params(f).learningRate = 2 ; 114 | net.params(f).weightDecay = 1 ; 115 | 116 | scale2Conv2_reluBlock = dagnn.ReLU() ; 117 | net.addLayer('relu_scale2Conv2', scale2Conv2_reluBlock, 'x46', 'x47', {}) ; 118 | 119 | net.addLayer('scale2Conv3', ... 120 | dagnn.Conv('size', [5 5 64 64], 'pad', [2 2 2 2]), ... 121 | 'x47', 'x48', {'scale2Conv3_f','scale2Conv3_b'}); 122 | 123 | f = net.getParamIndex('scale2Conv3_f') ; 124 | net.params(f).value = 0.01*randn(5, 5, 64, 64, 'single') ; 125 | net.params(f).learningRate = 1 ; 126 | net.params(f).weightDecay = 1 ; 127 | 128 | f = net.getParamIndex('scale2Conv3_b') ; 129 | net.params(f).value = zeros(1, 1, 64, 'single') ; 130 | net.params(f).learningRate = 2 ; 131 | net.params(f).weightDecay = 1 ; 132 | 133 | scale2Conv3_reluBlock = dagnn.ReLU() ; 134 | net.addLayer('relu_scale2Conv3', scale2Conv3_reluBlock, 'x48', 'x49', {}) ; 135 | 136 | scale2Conv3_concatBlock = dagnn.Concat(); 137 | net.addLayer('concat_scale2Conv3', scale2Conv3_concatBlock, {'x49', 'x39'}, 'x50', {}) ; 138 | 139 | net.addLayer('scale2Conv4', ... 140 | dagnn.Conv('size', [5 5 128 64], 'pad', [2 2 2 2]), ... 141 | 'x50', 'x51', {'scale2Conv4_f','scale2Conv4_b'}); 142 | 143 | f = net.getParamIndex('scale2Conv4_f') ; 144 | net.params(f).value = 0.01*randn(5, 5, 128, 64, 'single') ; 145 | net.params(f).learningRate = 1 ; 146 | net.params(f).weightDecay = 1 ; 147 | 148 | f = net.getParamIndex('scale2Conv4_b') ; 149 | net.params(f).value = zeros(1, 1, 64, 'single') ; 150 | net.params(f).learningRate = 2 ; 151 | net.params(f).weightDecay = 1 ; 152 | 153 | scale2Conv4_reluBlock = dagnn.ReLU() ; 154 | net.addLayer('relu_scale2Conv4', scale2Conv4_reluBlock, 'x51', 'x52', {}) ; 155 | 156 | net.addLayer('scale2Conv5', ... 157 | dagnn.Conv('size', [5 5 64 64], 'pad', [2 2 2 2]), ... 158 | 'x52', 'x53', {'scale2Conv5_f','scale2Conv5_b'}); 159 | 160 | f = net.getParamIndex('scale2Conv5_f') ; 161 | net.params(f).value = 0.01*randn(5, 5, 64, 64, 'single') ; 162 | net.params(f).learningRate = 1 ; 163 | net.params(f).weightDecay = 1 ; 164 | 165 | f = net.getParamIndex('scale2Conv5_b') ; 166 | net.params(f).value = zeros(1, 1, 64, 'single') ; 167 | net.params(f).learningRate = 2 ; 168 | net.params(f).weightDecay = 1 ; 169 | 170 | scale2Conv5_reluBlock = dagnn.ReLU() ; 171 | net.addLayer('relu_scale2Conv5', scale2Conv5_reluBlock, 'x53', 'x54', {}) ; 172 | 173 | scale2Conv5_concatBlock = dagnn.Concat(); 174 | net.addLayer('concat_scale2Conv5', scale2Conv5_concatBlock, {'x54', 'x36'}, 'x55', {}) ; 175 | 176 | net.addLayer('scale2Conv6', ... 177 | dagnn.Conv('size', [5 5 65 64], 'pad', [2 2 2 2]), ... 178 | 'x55', 'x56', {'scale2Conv6_f','scale2Conv6_b'}); 179 | 180 | f = net.getParamIndex('scale2Conv6_f') ; 181 | net.params(f).value = 0.01*randn(5, 5, 65, 64, 'single') ; 182 | net.params(f).learningRate = 1 ; 183 | net.params(f).weightDecay = 1 ; 184 | 185 | f = net.getParamIndex('scale2Conv6_b') ; 186 | net.params(f).value = zeros(1, 1, 64, 'single') ; 187 | net.params(f).learningRate = 2 ; 188 | net.params(f).weightDecay = 1 ; 189 | 190 | scale2Conv6_reluBlock = dagnn.ReLU() ; 191 | net.addLayer('relu_scale2Conv6', scale2Conv6_reluBlock, 'x56', 'x57', {}) ; 192 | 193 | net.addLayer('scale2Conv7', ... 194 | dagnn.Conv('size', [5 5 64 64], 'pad', [2 2 2 2]), ... 195 | 'x57', 'x58', {'scale2Conv7_f','scale2Conv7_b'}); 196 | 197 | f = net.getParamIndex('scale2Conv7_f') ; 198 | net.params(f).value = 0.01*randn(5, 5, 64, 64, 'single') ; 199 | net.params(f).learningRate = 1 ; 200 | net.params(f).weightDecay = 1 ; 201 | 202 | f = net.getParamIndex('scale2Conv7_b') ; 203 | net.params(f).value = zeros(1, 1, 64, 'single') ; 204 | net.params(f).learningRate = 2 ; 205 | net.params(f).weightDecay = 1 ; 206 | 207 | scale2Conv7_reluBlock = dagnn.ReLU() ; 208 | net.addLayer('relu_scale2Conv7', scale2Conv7_reluBlock, 'x58', 'x59', {}) ; 209 | 210 | net.addLayer('scale2Conv8', ... 211 | dagnn.Conv('size', [5 5 64 64], 'pad', [2 2 2 2]), ... 212 | 'x59', 'x60', {'scale2Conv8_f','scale2Conv8_b'}); 213 | 214 | f = net.getParamIndex('scale2Conv8_f') ; 215 | net.params(f).value = 0.01*randn(5, 5, 64, 64, 'single') ; 216 | net.params(f).learningRate = 1 ; 217 | net.params(f).weightDecay = 1 ; 218 | 219 | f = net.getParamIndex('scale2Conv8_b') ; 220 | net.params(f).value = zeros(1, 1, 64, 'single') ; 221 | net.params(f).learningRate = 2 ; 222 | net.params(f).weightDecay = 1 ; 223 | 224 | scale2Conv8_reluBlock = dagnn.ReLU() ; 225 | net.addLayer('relu_scale2Conv8', scale2Conv8_reluBlock, 'x60', 'x61', {}) ; 226 | 227 | net.addLayer('scale2Conv9', ... 228 | dagnn.Conv('size', [5 5 64 64], 'pad', [2 2 2 2]), ... 229 | 'x61', 'x62', {'scale2Conv9_f','scale2Conv9_b'}); 230 | 231 | f = net.getParamIndex('scale2Conv9_f') ; 232 | net.params(f).value = 0.01*randn(5, 5, 64, 64, 'single') ; 233 | net.params(f).learningRate = 1 ; 234 | net.params(f).weightDecay = 1 ; 235 | 236 | f = net.getParamIndex('scale2Conv9_b') ; 237 | net.params(f).value = zeros(1, 1, 64, 'single') ; 238 | net.params(f).learningRate = 2 ; 239 | net.params(f).weightDecay = 1 ; 240 | 241 | scale2Conv9_reluBlock = dagnn.ReLU() ; 242 | net.addLayer('relu_scale2Conv9', scale2Conv9_reluBlock, 'x62', 'x63', {}) ; 243 | 244 | net.addLayer('scale2Conv10', ... 245 | dagnn.Conv('size', [5 5 64 1], 'pad', [2 2 2 2]), ... 246 | 'x63', 'prediction', {'scale2Conv10_f','scale2Conv10_b'}); 247 | 248 | f = net.getParamIndex('scale2Conv10_f') ; 249 | net.params(f).value = 0.01*randn(5, 5, 64, 1, 'single') ; 250 | net.params(f).learningRate = 1 ; 251 | net.params(f).weightDecay = 1 ; 252 | 253 | f = net.getParamIndex('scale2Conv10_b') ; 254 | net.params(f).value = zeros(1, 1, 1, 'single') ; 255 | net.params(f).learningRate = 2 ; 256 | net.params(f).weightDecay = 1 ; 257 | 258 | net.vars(net.getVarIndex('prediction')).precious = 1 ; 259 | % Add loss layer 260 | net.addLayer('objective', ... 261 | myLoss(), ... 262 | {'prediction', 'label'}, 'objective') ; 263 | 264 | % Add accuracy layer 265 | net.addLayer('accuracy', ... 266 | myAccuracy(), ... 267 | {'prediction', 'label'}, 'accuracy') ; 268 | 269 | 270 | end 271 | 272 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/myDAGnet3.m: -------------------------------------------------------------------------------- 1 | function net = myDAGnet3 2 | 3 | net = createNetfromvgg(); 4 | net = vl_simplenn_tidy(net) ; 5 | net = dagnn.DagNN.fromSimpleNN(net, 'canonicalNames', true) ; 6 | 7 | reshapeBlock = myReshape(); 8 | net.addLayer('myreshape', reshapeBlock, 'x35', 'prediction'); 9 | 10 | net.vars(net.getVarIndex('prediction')).precious = 1 ; 11 | % Add loss layer 12 | net.addLayer('objective', ... 13 | myLoss(), ... 14 | {'prediction', 'label'}, 'objective') ; 15 | 16 | % Add accuracy layer 17 | net.addLayer('accuracy', ... 18 | myAccuracy(), ... 19 | {'prediction', 'label'}, 'accuracy') ; 20 | 21 | 22 | end 23 | 24 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/myLoss.m: -------------------------------------------------------------------------------- 1 | classdef myLoss < dagnn.Loss 2 | 3 | properties (Transient) 4 | clip_d = 1 5 | end 6 | 7 | methods 8 | function outputs = forward(obj, inputs, params) 9 | d_errors = gather(inputs{1})-log10(inputs{2}); 10 | marks = ones(size(inputs{2})); 11 | marks(inputs{2}==1000) = 0; 12 | d_errors = d_errors.*marks; 13 | % pixelnum = size(inputs{1},1)*size(inputs{1},2)*size(inputs{1},4); 14 | pixelnum = sum(sum(sum(marks,1),2)); 15 | outputs{1} = sum(sum(sum(d_errors.^2,1),2)); 16 | n = obj.numAveraged ; 17 | m = n + pixelnum ; 18 | obj.average = (n * obj.average + outputs{1}) / m ; 19 | obj.numAveraged = m ; 20 | end 21 | 22 | function [derInputs, derParams] = backward(obj, inputs, params, derOutputs) 23 | d_errors = gather(inputs{1})-log10(inputs{2}); 24 | marks = ones(size(inputs{2})); 25 | marks(inputs{2}==1000) = 0; 26 | d_errors = d_errors.*marks; 27 | 28 | dx = zeros(size(d_errors), 'single'); 29 | 30 | bs = size(inputs{2},4); 31 | for ii = 1:bs 32 | dx(:,:,:,ii) = 2*d_errors(:,:,:,ii); 33 | end 34 | 35 | estd = gather(inputs{1}); 36 | for ii = 1:3:bs 37 | d1 = estd(:,:,:,ii); 38 | d2 = estd(:,:,:,ii+1); 39 | d3 = estd(:,:,:,ii+2); 40 | refd1 = flip(d1,2); 41 | refd2 = flip(d2,2); 42 | refd3 = flip(d3,2); 43 | 44 | d12 = d1-d2; 45 | d21 = d2-d1; 46 | 47 | 48 | d1refd3 = d1-refd3; 49 | d2refd3 = d2-refd3; 50 | d3refd1 = d3-refd1; 51 | d3refd2 = d3-refd2; 52 | 53 | fd1 = d12+d1refd3; 54 | dx(:,:,:,ii) = dx(:,:,:,ii)+2*fd1; 55 | 56 | fd2 = d21+d2refd3; 57 | dx(:,:,:,ii+1) = dx(:,:,:,ii+1)+2*fd2; 58 | 59 | fd3 = d3refd1+d3refd2; 60 | dx(:,:,:,ii+2) = dx(:,:,:,ii+2)+2*fd3; 61 | end 62 | 63 | dx(dx > obj.clip_d) = obj.clip_d; 64 | dx(dx < -obj.clip_d) = -obj.clip_d; 65 | 66 | for jj = 1:bs 67 | dx(:,:,:,jj) = dx(:,:,:,jj)/16650; 68 | end 69 | 70 | dx = gpuArray(dx); 71 | derInputs{1} = dx; 72 | derInputs{2} = [] ; 73 | derParams = {} ; 74 | end 75 | 76 | function setClip(obj, rd) 77 | obj.clip_d = rd; 78 | end 79 | 80 | function obj = myLoss(varargin) 81 | obj.load(varargin) ; 82 | end 83 | end 84 | end 85 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/myReshape.m: -------------------------------------------------------------------------------- 1 | classdef myReshape < dagnn.ElementWise 2 | %SUM DagNN sum layer 3 | % The SUM layer takes the sum of all its inputs and store the result 4 | % as its only output. 5 | 6 | properties 7 | dmap = [56; 56; 3] 8 | end 9 | 10 | properties (Transient) 11 | inputSizes = {} 12 | end 13 | 14 | methods 15 | function outputs = forward(obj, inputs, params) 16 | obj.inputSizes = cellfun(@size, inputs, 'UniformOutput', false) ; 17 | outputs{1} = reshape(inputs{1}, [obj.dmap(1) obj.dmap(2) obj.dmap(3) size(inputs{1},4)]) ; 18 | end 19 | 20 | function [derInputs, derParams] = backward(obj, inputs, params, derOutputs) 21 | derInputs{1} = reshape(derOutputs{1}, [1 1 obj.dmap(1)*obj.dmap(2)*obj.dmap(3) size(inputs{1},4)]) ; 22 | derParams = {} ; 23 | end 24 | 25 | function outputSizes = getOutputSizes(obj, inputSizes) 26 | outputSizes{1} = inputSizes{1} ; 27 | outputSizes{1}(1) = obj.dmap(1); 28 | outputSizes{1}(2) = obj.dmap(2); 29 | outputSizes{1}(3) = obj.dmap(3); 30 | end 31 | 32 | % function rfs = getReceptiveFields(obj) 33 | % numInputs = numel(obj.net.layers(obj.layerIndex).inputs) ; 34 | % rfs.size = [1 1] ; 35 | % rfs.stride = [1 1] ; 36 | % rfs.offset = [1 1] ; 37 | % rfs = repmat(rfs, numInputs, 1) ; 38 | % end 39 | 40 | function obj = myReshape(varargin) 41 | obj.load(varargin) ; 42 | end 43 | end 44 | end 45 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/myTrain.m: -------------------------------------------------------------------------------- 1 | function myTrain(varargin) 2 | 3 | trainOpts.id = 1; 4 | trainOpts.useGpu = true; 5 | trainOpts.batchSize = 16 ; 6 | trainOpts.continue = false ; 7 | trainOpts.gpus = 1 ; 8 | trainOpts.prefetch = false ; 9 | trainOpts.dataDir = '' ; 10 | trainOpts.saveDir = '' ; 11 | trainOpts.outputDir = '' ; 12 | trainOpts.learningRate = 1 * ones(1,200) ; 13 | trainOpts.clip_d = 1*ones(1,200); 14 | trainOpts.numEpochs = 100; 15 | trainOpts.outputN = 1; 16 | trainOpts = vl_argparse(trainOpts, varargin) ; 17 | 18 | 19 | 20 | trainOpts.learningRate(1,1:11)= 5; 21 | trainOpts.learningRate(1,12:22)= 0.5; 22 | trainOpts.learningRate(1,23:trainOpts.numEpochs)= 0.05; 23 | 24 | % trainOpts.clip_d(1,1:1) = 0.001; 25 | % trainOpts.clip_d(1,3:5) = 0.1; 26 | trainOpts.clip_d(1,1:1) = 0.1; 27 | trainOpts.clip_d(1,2:11) = 1; 28 | trainOpts.clip_d(1,12:trainOpts.numEpochs) = 10; 29 | 30 | 31 | 32 | 33 | %%%%%% 34 | 35 | 36 | 37 | % ------------------------------------------------------------------------- 38 | % Train 39 | % ------------------------------------------------------------------------- 40 | % Launch SGD 41 | info = my_cnn_train_dag(getBatchWrapper, trainOpts) ; 42 | 43 | % ------------------------------------------------------------------------- 44 | function fn = getBatchWrapper() 45 | % ------------------------------------------------------------------------- 46 | fn = @(imdb,batch,mode,opts) getBatch(imdb,batch,mode,opts) ; 47 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/my_cnn_train_dag_old.m: -------------------------------------------------------------------------------- 1 | function [net,stats] = my_cnn_train_dag_old(imdb, getBatch, varargin) 2 | %CNN_TRAIN_DAG Demonstrates training a CNN using the DagNN wrapper 3 | % CNN_TRAIN_DAG() is similar to CNN_TRAIN(), but works with 4 | % the DagNN wrapper instead of the SimpleNN wrapper. 5 | 6 | % Copyright (C) 2014-15 Andrea Vedaldi. 7 | % All rights reserved. 8 | % 9 | % This file is part of the VLFeat library and is made available under 10 | % the terms of the BSD license (see the COPYING file). 11 | 12 | % Todo: save momentum with checkpointing (a waste?) 13 | opts.id = 1; 14 | opts.useGpu = true; 15 | opts.dataDir = ''; 16 | opts.outputDir = ''; 17 | opts.saveDir = ''; 18 | opts.continue = false ; 19 | opts.batchSize = 256 ; 20 | opts.numSubBatches = 1 ; 21 | opts.train = [] ; 22 | opts.val = [] ; 23 | opts.gpus = [] ; 24 | opts.prefetch = false ; 25 | opts.numEpochs = 100 ; 26 | opts.learningRate = 1 ; 27 | opts.weightDecay = 1 ; 28 | opts.momentum = 0.9 ; 29 | opts.clip_d = 1; 30 | opts.imgRbgMean = []; 31 | opts.epochpre = 60; 32 | opts.derOutputs = {'objective', 1} ; 33 | opts.extractStatsFn = @extractStats ; 34 | opts.plotStatistics = false; 35 | opts = vl_argparse(opts, varargin) ; 36 | 37 | if ~exist(opts.saveDir, 'dir'), mkdir(opts.saveDir) ; end 38 | if ~exist(opts.outputDir, 'dir'), mkdir(opts.outputDir) ; end 39 | if isempty(opts.train), opts.train = find(imdb.nyudata.set==1) ; end 40 | if isempty(opts.val), opts.val = find(imdb.nyudata.set==2) ; end 41 | if isnan(opts.train), opts.train = [] ; end 42 | 43 | % ------------------------------------------------------------------------- 44 | % Initialization 45 | % ------------------------------------------------------------------------- 46 | 47 | % opts.train = 1:2*numel(opts.train); 48 | 49 | state.getBatch = getBatch ; 50 | evaluateMode = isempty(opts.train) ; 51 | if ~evaluateMode 52 | if isempty(opts.derOutputs) 53 | error('DEROUTPUTS must be specified when training.\n') ; 54 | end 55 | end 56 | stats = [] ; 57 | 58 | % setup GPUs 59 | numGpus = numel(opts.gpus) ; 60 | if numGpus > 1 61 | if isempty(gcp('nocreate')), 62 | parpool('local',numGpus) ; 63 | spmd, gpuDevice(opts.gpus(labindex)), end 64 | end 65 | if exist(opts.memoryMapFile) 66 | delete(opts.memoryMapFile) ; 67 | end 68 | elseif numGpus == 1 69 | % gpuDevice(opts.gpus) 70 | end 71 | 72 | % ------------------------------------------------------------------------- 73 | % Train and validate 74 | % ------------------------------------------------------------------------- 75 | 76 | modelPath = @(ep) fullfile(opts.saveDir, sprintf('FT-net-%d.mat', opts.id)); 77 | modelFigPath = fullfile(opts.saveDir, 'net-train.pdf') ; 78 | 79 | trainInfoPath = fullfile(opts.outputDir, sprintf('FT-info-%d.txt', opts.id)); 80 | fid = fopen(trainInfoPath, 'w'); 81 | fprintf(fid,'Start training... \n\n\n'); 82 | fclose(fid); 83 | 84 | 85 | epoch_start = 0 ; 86 | if exist(fullfile(opts.saveDir, sprintf('FT-net-%d.mat', opts.id)), 'file') 87 | [net, stats, epoch_start] = loadState(fullfile(opts.saveDir, sprintf('FT-net-%d.mat', opts.id))) ; 88 | fprintf('load cnn net from last time \n'); 89 | else 90 | if exist(fullfile(opts.dataDir, 'start-net-final-05.mat'), 'file') 91 | net = loadState(fullfile(opts.dataDir, 'start-net-final-05.mat')) ; 92 | net.meta.imgRbgMean = opts.imgRbgMean; 93 | fprintf('load start net model \n'); 94 | else 95 | net = myDAGnet; 96 | net.meta.imgRbgMean = opts.imgRbgMean; 97 | saveState(fullfile(opts.dataDir, 'start-net-final-05.mat'), net, [], 0); 98 | end 99 | end 100 | 101 | setLearningRate(net); 102 | rng('shuffle'); 103 | 104 | for epoch=epoch_start+1:opts.numEpochs 105 | 106 | % train one epoch 107 | state.epoch = epoch ; 108 | state.learningRate = opts.learningRate(min(epoch, numel(opts.learningRate))) ; 109 | state.train = opts.train(randperm(numel(opts.train))) ; % shuffle 110 | state.val = opts.val ; 111 | state.imdb = imdb ; 112 | 113 | net.layers(58).block.setClip(opts.clip_d(epoch)); 114 | 115 | if numGpus <= 1 116 | stats.train(epoch) = process_epoch(net, state, opts, 'train') ; 117 | stats.val(epoch) = process_epoch(net, state, opts, 'val') ; 118 | else 119 | savedNet = net.saveobj() ; 120 | spmd 121 | net_ = dagnn.DagNN.loadobj(savedNet) ; 122 | stats_.train = process_epoch(net_, state, opts, 'train') ; 123 | stats_.val = process_epoch(net_, state, opts, 'val') ; 124 | if labindex == 1, savedNet_ = net_.saveobj() ; end 125 | end 126 | net = dagnn.DagNN.loadobj(savedNet_{1}) ; 127 | stats__ = accumulateStats(stats_) ; 128 | stats.train(epoch) = stats__.train ; 129 | stats.val(epoch) = stats__.val ; 130 | clear net_ stats_ stats__ savedNet_ ; 131 | end 132 | 133 | if ~evaluateMode 134 | saveState(modelPath(epoch), net, stats, epoch) ; 135 | end 136 | 137 | fid = fopen(trainInfoPath, 'a+'); 138 | fprintf(fid,'%s %d %s \n', '[epoch = ', epoch, ']'); 139 | fprintf(fid,'%s %.4f \n', ' train-objective = ', stats.train(epoch).objective); 140 | train_accuracy = stats.train(epoch).accuracy; 141 | fprintf(fid,'%s %.4f \n', ' train-rel_error = ', train_accuracy(1)); 142 | fprintf(fid,'%s %.4f \n', ' train-log_error = ', train_accuracy(2)); 143 | fprintf(fid,'%s %.4f \n', ' train-rm_error = ', train_accuracy(3)); 144 | val_accuracy = stats.val(epoch).accuracy; 145 | fprintf(fid,'%s %.4f \n', ' val-objective = ', stats.val(epoch).objective); 146 | fprintf(fid,'%s %.4f \n', ' val-rel_error = ', val_accuracy(1)); 147 | fprintf(fid,'%s %.4f \n', ' val-log_error = ', val_accuracy(2)); 148 | fprintf(fid,'%s %.4f \n\n\n', ' val-rm_error = ', val_accuracy(3)); 149 | fclose(fid); 150 | 151 | if opts.plotStatistics 152 | figure(1) ; clf ; 153 | plots = setdiff(... 154 | cat(2,... 155 | fieldnames(stats.train)', ... 156 | fieldnames(stats.val)'), {'num', 'time'}) ; 157 | for p = plots 158 | p = char(p) ; 159 | values = zeros(0, epoch) ; 160 | leg = {} ; 161 | for f = {'train', 'val'} 162 | f = char(f) ; 163 | if isfield(stats.(f), p) 164 | tmp = [stats.(f).(p)] ; 165 | values(end+1,:) = tmp(1,:)' ; 166 | leg{end+1} = f ; 167 | end 168 | end 169 | subplot(1,numel(plots),find(strcmp(p,plots))) ; 170 | plot(1:epoch, values','o-') ; 171 | xlabel('epoch') ; 172 | title(p) ; 173 | legend(leg{:}) ; 174 | grid on ; 175 | end 176 | drawnow ; 177 | print(1, modelFigPath, '-dpdf') ; 178 | end 179 | end 180 | 181 | % ------------------------------------------------------------------------- 182 | function stats = process_epoch(net, state, opts, mode) 183 | % ------------------------------------------------------------------------- 184 | 185 | if strcmp(mode,'train') 186 | state.momentum = num2cell(zeros(1, numel(net.params))) ; 187 | end 188 | 189 | numGpus = numel(opts.gpus) ; 190 | if numGpus >= 1 191 | net.move('gpu') ; 192 | if strcmp(mode,'train') 193 | state.momentum = cellfun(@gpuArray,state.momentum,'UniformOutput',false) ; 194 | end 195 | end 196 | if numGpus > 1 197 | mmap = map_gradients(opts.memoryMapFile, net, numGpus) ; 198 | else 199 | mmap = [] ; 200 | end 201 | 202 | stats.time = 0 ; 203 | stats.num = 0 ; 204 | subset = state.(mode) ; 205 | start = tic ; 206 | num = 0 ; 207 | 208 | for t=1:opts.batchSize:numel(subset) 209 | batchSize = min(opts.batchSize, numel(subset) - t + 1) ; 210 | 211 | for s=1:opts.numSubBatches 212 | % get this image batch and prefetch the next 213 | batchStart = t + (labindex-1) + (s-1) * numlabs ; 214 | batchEnd = min(t+opts.batchSize-1, numel(subset)) ; 215 | batch = subset(batchStart : opts.numSubBatches * numlabs : batchEnd) ; 216 | num = num + numel(batch) ; 217 | if numel(batch) == 0, continue ; end 218 | 219 | bopts.imgRbgMean = opts.imgRbgMean; 220 | bopts.useGpu = opts.useGpu; 221 | inputs = state.getBatch(state.imdb, batch, mode, bopts) ; 222 | 223 | if opts.prefetch 224 | if s == opts.numSubBatches 225 | batchStart = t + (labindex-1) + opts.batchSize ; 226 | batchEnd = min(t+2*opts.batchSize-1, numel(subset)) ; 227 | else 228 | batchStart = batchStart + numlabs ; 229 | end 230 | nextBatch = subset(batchStart : opts.numSubBatches * numlabs : batchEnd) ; 231 | bopts.imgRbgMean = opts.imgRbgMean; 232 | bopts.useGpu = opts.useGpu; 233 | state.getBatch(state.imdb, nextBatch, mode, bopts) ; 234 | end 235 | 236 | if strcmp(mode, 'train') 237 | net.mode = 'normal' ; 238 | net.accumulateParamDers = (s ~= 1) ; 239 | net.myeval(inputs, opts.derOutputs, state.epoch, opts.epochpre) ; 240 | else 241 | net.mode = 'test' ; 242 | net.myeval(inputs, [], state.epoch, opts.epochpre) ; 243 | end 244 | end 245 | 246 | % extract learning stats 247 | stats = opts.extractStatsFn(net) ; 248 | 249 | % accumulate gradient 250 | if strcmp(mode, 'train') 251 | if ~isempty(mmap) 252 | write_gradients(mmap, net) ; 253 | labBarrier() ; 254 | end 255 | state = accumulate_gradients(state, net, opts, batchSize, mmap, opts.epochpre) ; 256 | end 257 | 258 | % print learning statistics 259 | time = toc(start) ; 260 | stats.num = num ; 261 | stats.time = toc(start) ; 262 | 263 | fprintf('%s: epoch %02d: %3d/%3d: %.1f Hz', ... 264 | mode, ... 265 | state.epoch, ... 266 | fix(t/opts.batchSize)+1, ceil(numel(subset)/opts.batchSize), ... 267 | stats.num/stats.time * max(numGpus, 1)) ; 268 | fprintf('\n') ; 269 | end 270 | 271 | net.layers(59).block.recalerror(); 272 | stats = opts.extractStatsFn(net) ; 273 | 274 | for f = setdiff(fieldnames(stats)', {'num', 'time'}) 275 | f = char(f) ; 276 | fprintf(' %s:', f) ; 277 | fprintf(' %.3f', stats.(f)) ; 278 | end 279 | fprintf('\n') ; 280 | 281 | net.reset() ; 282 | net.move('cpu') ; 283 | 284 | % ------------------------------------------------------------------------- 285 | function state = accumulate_gradients(state, net, opts, batchSize, mmap, epochpre) 286 | % ------------------------------------------------------------------------- 287 | 288 | startlayer = 1; 289 | if state.epoch < epochpre 290 | startlayer = 27; 291 | end 292 | 293 | for p=startlayer:numel(net.params) 294 | 295 | % bring in gradients from other GPUs if any 296 | if ~isempty(mmap) 297 | numGpus = numel(mmap.Data) ; 298 | tmp = zeros(size(mmap.Data(labindex).(net.params(p).name)), 'single') ; 299 | for g = setdiff(1:numGpus, labindex) 300 | tmp = tmp + mmap.Data(g).(net.params(p).name) ; 301 | end 302 | net.params(p).der = net.params(p).der + tmp ; 303 | else 304 | numGpus = 1 ; 305 | end 306 | 307 | switch net.params(p).trainMethod 308 | 309 | case 'average' % mainly for batch normalization 310 | thisLR = net.params(p).learningRate ; 311 | net.params(p).value = ... 312 | (1 - thisLR) * net.params(p).value + ... 313 | (thisLR/batchSize/net.params(p).fanout) * net.params(p).der ; 314 | 315 | case 'gradient' 316 | thisDecay = opts.weightDecay * net.params(p).weightDecay ; 317 | thisLR = state.learningRate * net.params(p).learningRate ; 318 | state.momentum{p} = opts.momentum * state.momentum{p} ... 319 | - thisDecay * net.params(p).value ... 320 | - (1 / batchSize) * net.params(p).der ; 321 | net.params(p).value = net.params(p).value + thisLR * state.momentum{p} ; 322 | 323 | case 'otherwise' 324 | error('Unknown training method ''%s'' for parameter ''%s''.', ... 325 | net.params(p).trainMethod, ... 326 | net.params(p).name) ; 327 | end 328 | end 329 | 330 | % ------------------------------------------------------------------------- 331 | function mmap = map_gradients(fname, net, numGpus) 332 | % ------------------------------------------------------------------------- 333 | format = {} ; 334 | for i=1:numel(net.params) 335 | format(end+1,1:3) = {'single', size(net.params(i).value), net.params(i).name} ; 336 | end 337 | format(end+1,1:3) = {'double', [3 1], 'errors'} ; 338 | if ~exist(fname) && (labindex == 1) 339 | f = fopen(fname,'wb') ; 340 | for g=1:numGpus 341 | for i=1:size(format,1) 342 | fwrite(f,zeros(format{i,2},format{i,1}),format{i,1}) ; 343 | end 344 | end 345 | fclose(f) ; 346 | end 347 | labBarrier() ; 348 | mmap = memmapfile(fname, 'Format', format, 'Repeat', numGpus, 'Writable', true) ; 349 | 350 | % ------------------------------------------------------------------------- 351 | function write_gradients(mmap, net) 352 | % ------------------------------------------------------------------------- 353 | for i=1:numel(net.params) 354 | mmap.Data(labindex).(net.params(i).name) = gather(net.params(i).der) ; 355 | end 356 | 357 | % ------------------------------------------------------------------------- 358 | function stats = accumulateStats(stats_) 359 | % ------------------------------------------------------------------------- 360 | 361 | stats = struct() ; 362 | 363 | for s = {'train', 'val'} 364 | s = char(s) ; 365 | total = 0 ; 366 | 367 | for g = 1:numel(stats_) 368 | stats__ = stats_{g} ; 369 | num__ = stats__.(s).num ; 370 | total = total + num__ ; 371 | 372 | for f = setdiff(fieldnames(stats__.(s))', 'num') 373 | f = char(f) ; 374 | 375 | if g == 1 376 | stats.(s).(f) = 0 ; 377 | end 378 | stats.(s).(f) = stats.(s).(f) + stats__.(s).(f) * num__ ; 379 | 380 | if g == numel(stats_) 381 | stats.(s).(f) = stats.(s).(f) / total ; 382 | end 383 | end 384 | end 385 | stats.(s).num = total ; 386 | end 387 | 388 | % ------------------------------------------------------------------------- 389 | function stats = extractStats(net) 390 | % ------------------------------------------------------------------------- 391 | sel = find(cellfun(@(x) isa(x,'dagnn.Loss'), {net.layers.block})) ; 392 | stats = struct() ; 393 | for i = 1:numel(sel) 394 | stats.(net.layers(sel(i)).outputs{1}) = net.layers(sel(i)).block.average ; 395 | end 396 | 397 | % ------------------------------------------------------------------------- 398 | function saveState(fileName, net, stats, epoch_start) 399 | % ------------------------------------------------------------------------- 400 | net_ = net ; 401 | net = net_.saveobj() ; 402 | save(fileName, 'net', 'stats', 'epoch_start') ; 403 | 404 | % ------------------------------------------------------------------------- 405 | function [net, stats, epoch_start] = loadState(fileName) 406 | % ------------------------------------------------------------------------- 407 | load(fileName, 'net', 'stats', 'epoch_start') ; 408 | net = dagnn.DagNN.loadobj(net) ; 409 | 410 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/mysimpleDAG.m: -------------------------------------------------------------------------------- 1 | function net = mysimpleDAG() 2 | 3 | % net = initializeDepthCNN(); 4 | 5 | load('../data/old_net.mat'); 6 | % vggnet=load('data/imagenet-vgg-verydeep-16.mat'); 7 | % net.layers = vggnet.layers(1:33); 8 | 9 | net = vl_simplenn_tidy(net) ; 10 | net = dagnn.DagNN.fromSimpleNN(net, 'canonicalNames', true) ; 11 | 12 | reshapeBlock = myReshape(); 13 | net.addLayer('myreshape', reshapeBlock, 'x35', 'prediction'); 14 | 15 | net.vars(net.getVarIndex('prediction')).precious = 1 ; 16 | 17 | net.addLayer('objective', ... 18 | myLoss(), ... 19 | {'prediction', 'label'}, 'objective') ; 20 | 21 | % Add accuracy layer 22 | net.addLayer('accuracy', ... 23 | myAccuracy(), ... 24 | {'prediction', 'label'}, 'accuracy') ; 25 | 26 | end 27 | 28 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/setLearningRate.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/masknetwork/setLearningRate.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/setLearningRate_small.m: -------------------------------------------------------------------------------- 1 | function setLearningRate_small( net ) 2 | 3 | %imNet 4 | for ii = 1:26 5 | net.params(ii).learningRate = 0.00005 ; 6 | net.params(ii).weightDecay = 0.00001 ; 7 | end 8 | 9 | %full layers 10 | for ii = 27:30 11 | net.params(ii).learningRate = 0.00005 ; 12 | net.params(ii).weightDecay = 0.00001 ; 13 | end 14 | 15 | %skip_4_f skip_4_b 16 | net.params(31).learningRate = 0.00005 ; 17 | net.params(31).weightDecay = 0.00001 ; 18 | net.params(32).learningRate = 0.00005 ; 19 | net.params(32).weightDecay = 0.00001 ; 20 | 21 | %skip_3_f skip_3_b 22 | net.params(34).learningRate = 0.00005 ; 23 | net.params(34).weightDecay = 0.00001 ; 24 | net.params(35).learningRate =0.00005 ; 25 | net.params(35).weightDecay = 0.00001 ; 26 | 27 | %scale Conv1 28 | net.params(37).learningRate =0.00005 ; 29 | net.params(37).weightDecay = 0.00001 ; 30 | net.params(38).learningRate = 0.00005 ; 31 | net.params(38).weightDecay = 0.00001 ; 32 | 33 | %scale Conv2 34 | net.params(39).learningRate = 0.00005 ; 35 | net.params(39).weightDecay = 0.00001 ; 36 | net.params(40).learningRate = 0.00005 ; 37 | net.params(40).weightDecay = 0.00001 ; 38 | 39 | %scale Conv3 40 | net.params(41).learningRate = 0.00005 ; 41 | net.params(41).weightDecay = 0.00001 ; 42 | net.params(42).learningRate = 0.00005 ; 43 | net.params(42).weightDecay = 0.00001 ; 44 | 45 | %scale Conv4 46 | net.params(43).learningRate = 0.00005 ; 47 | net.params(43).weightDecay = 0.00001 ; 48 | net.params(44).learningRate = 0.00005 ; 49 | net.params(44).weightDecay = 0.00001 ; 50 | 51 | %scale Conv5 52 | net.params(45).learningRate = 0.00005 ; 53 | net.params(45).weightDecay = 0.00001 ; 54 | net.params(46).learningRate = 0.00005 ; 55 | net.params(46).weightDecay = 0.00001 ; 56 | 57 | %scale Conv6 58 | net.params(47).learningRate = 0.00005 ; 59 | net.params(47).weightDecay = 0.00001 ; 60 | net.params(48).learningRate = 0.00005 ; 61 | net.params(48).weightDecay = 0.00001 ; 62 | 63 | %scale Conv7 64 | net.params(49).learningRate = 0.00005 ; 65 | net.params(49).weightDecay = 0.00001 ; 66 | net.params(50).learningRate = 0.00005 ; 67 | net.params(50).weightDecay = 0.00001 ; 68 | 69 | %scale Conv8 70 | net.params(51).learningRate = 0.00005 ; 71 | net.params(51).weightDecay = 0.00001 ; 72 | net.params(52).learningRate = 0.00005 ; 73 | net.params(52).weightDecay = 0.00001 ; 74 | 75 | %scale Conv9 76 | net.params(53).learningRate = 0.00005 ; 77 | net.params(53).weightDecay = 0.00001 ; 78 | net.params(54).learningRate = 0.00005 ; 79 | net.params(54).weightDecay = 0.00001 ; 80 | 81 | %scale Conv10 82 | net.params(55).learningRate = 0.00005 ; 83 | net.params(55).weightDecay = 0.00001 ; 84 | net.params(56).learningRate = 0.00005 ; 85 | net.params(56).weightDecay = 0.00001 ; 86 | 87 | end 88 | 89 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/setLearningRate_small2.m: -------------------------------------------------------------------------------- 1 | function setLearningRate_small( net ) 2 | 3 | %imNet 4 | for ii = 1:26 5 | net.params(ii).learningRate = 0.00001 ; 6 | net.params(ii).weightDecay = 0.00001 ; 7 | end 8 | 9 | %full layers 10 | for ii = 27:30 11 | net.params(ii).learningRate = 0.00001 ; 12 | net.params(ii).weightDecay = 0.00001 ; 13 | end 14 | 15 | %skip_4_f skip_4_b 16 | net.params(31).learningRate = 0.00001 ; 17 | net.params(31).weightDecay = 0.00001 ; 18 | net.params(32).learningRate = 0.00001 ; 19 | net.params(32).weightDecay = 0.00001 ; 20 | 21 | %skip_3_f skip_3_b 22 | net.params(34).learningRate = 0.00001 ; 23 | net.params(34).weightDecay = 0.00001 ; 24 | net.params(35).learningRate =0.00001 ; 25 | net.params(35).weightDecay = 0.00001 ; 26 | 27 | %scale Conv1 28 | net.params(37).learningRate =0.00001 ; 29 | net.params(37).weightDecay = 0.00001 ; 30 | net.params(38).learningRate = 0.00001 ; 31 | net.params(38).weightDecay = 0.00001 ; 32 | 33 | %scale Conv2 34 | net.params(39).learningRate = 0.00001 ; 35 | net.params(39).weightDecay = 0.00001 ; 36 | net.params(40).learningRate = 0.00001 ; 37 | net.params(40).weightDecay = 0.00001 ; 38 | 39 | %scale Conv3 40 | net.params(41).learningRate = 0.00001 ; 41 | net.params(41).weightDecay = 0.00001 ; 42 | net.params(42).learningRate = 0.00001 ; 43 | net.params(42).weightDecay = 0.00001 ; 44 | 45 | %scale Conv4 46 | net.params(43).learningRate = 0.00001 ; 47 | net.params(43).weightDecay = 0.00001 ; 48 | net.params(44).learningRate = 0.00001 ; 49 | net.params(44).weightDecay = 0.00001 ; 50 | 51 | %scale Conv5 52 | net.params(45).learningRate = 0.00001 ; 53 | net.params(45).weightDecay = 0.00001 ; 54 | net.params(46).learningRate = 0.00001 ; 55 | net.params(46).weightDecay = 0.00001 ; 56 | 57 | %scale Conv6 58 | net.params(47).learningRate = 0.00001 ; 59 | net.params(47).weightDecay = 0.00001 ; 60 | net.params(48).learningRate = 0.00001 ; 61 | net.params(48).weightDecay = 0.00001 ; 62 | 63 | %scale Conv7 64 | net.params(49).learningRate = 0.00001 ; 65 | net.params(49).weightDecay = 0.00001 ; 66 | net.params(50).learningRate = 0.00001 ; 67 | net.params(50).weightDecay = 0.00001 ; 68 | 69 | %scale Conv8 70 | net.params(51).learningRate = 0.00001 ; 71 | net.params(51).weightDecay = 0.00001 ; 72 | net.params(52).learningRate = 0.00001 ; 73 | net.params(52).weightDecay = 0.00001 ; 74 | 75 | %scale Conv9 76 | net.params(53).learningRate = 0.00001 ; 77 | net.params(53).weightDecay = 0.00001 ; 78 | net.params(54).learningRate = 0.00001 ; 79 | net.params(54).weightDecay = 0.00001 ; 80 | 81 | %scale Conv10 82 | net.params(55).learningRate = 0.00001 ; 83 | net.params(55).weightDecay = 0.00001 ; 84 | net.params(56).learningRate = 0.00001 ; 85 | net.params(56).weightDecay = 0.00001 ; 86 | 87 | end 88 | 89 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/test_masknet.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/masknetwork/test_masknet.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/masknetwork/test_masknet_realImage.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/masknetwork/test_masknet_realImage.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/matconvnet.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio 14 4 | VisualStudioVersion = 14.0.24720.0 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "matconvnet", "matconvnet.vcxproj", "{B0BD9132-1D90-4267-A07A-B44DE497A9C7}" 7 | EndProject 8 | Global 9 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 10 | Debug|Win32 = Debug|Win32 11 | Debug|x64 = Debug|x64 12 | Release|Win32 = Release|Win32 13 | Release|x64 = Release|x64 14 | EndGlobalSection 15 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 16 | {B0BD9132-1D90-4267-A07A-B44DE497A9C7}.Debug|Win32.ActiveCfg = Debug|Win32 17 | {B0BD9132-1D90-4267-A07A-B44DE497A9C7}.Debug|Win32.Build.0 = Debug|Win32 18 | {B0BD9132-1D90-4267-A07A-B44DE497A9C7}.Debug|x64.ActiveCfg = Debug|Win32 19 | {B0BD9132-1D90-4267-A07A-B44DE497A9C7}.Release|Win32.ActiveCfg = Release|Win32 20 | {B0BD9132-1D90-4267-A07A-B44DE497A9C7}.Release|Win32.Build.0 = Release|Win32 21 | {B0BD9132-1D90-4267-A07A-B44DE497A9C7}.Release|x64.ActiveCfg = Release|Win32 22 | EndGlobalSection 23 | GlobalSection(SolutionProperties) = preSolution 24 | HideSolutionNode = FALSE 25 | EndGlobalSection 26 | EndGlobal 27 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/matconvnet.vcxproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | Debug 6 | Win32 7 | 8 | 9 | Release 10 | Win32 11 | 12 | 13 | 14 | {B0BD9132-1D90-4267-A07A-B44DE497A9C7} 15 | matconvnet 16 | 17 | 18 | 19 | Application 20 | true 21 | MultiByte 22 | v140 23 | 24 | 25 | Application 26 | false 27 | true 28 | MultiByte 29 | v140 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | $(MatlabRoot)\extern\include;$(IncludePath) 43 | 44 | 45 | 46 | Level3 47 | Disabled 48 | 49 | 50 | true 51 | 52 | 53 | 54 | 55 | Level3 56 | MaxSpeed 57 | true 58 | true 59 | 60 | 61 | true 62 | true 63 | true 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159 | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175 | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/matconvnet.vcxproj.filters: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | matlab 6 | 7 | 8 | matlab 9 | 10 | 11 | matlab 12 | 13 | 14 | matlab 15 | 16 | 17 | matlab 18 | 19 | 20 | matlab 21 | 22 | 23 | matlab 24 | 25 | 26 | matlab 27 | 28 | 29 | matlab 30 | 31 | 32 | matlab 33 | 34 | 35 | matlab 36 | 37 | 38 | matlab 39 | 40 | 41 | matlab 42 | 43 | 44 | matlab 45 | 46 | 47 | matlab 48 | 49 | 50 | matlab 51 | 52 | 53 | matlab 54 | 55 | 56 | matlab 57 | 58 | 59 | matlab 60 | 61 | 62 | src 63 | 64 | 65 | src 66 | 67 | 68 | src 69 | 70 | 71 | src 72 | 73 | 74 | src 75 | 76 | 77 | src 78 | 79 | 80 | src\bits 81 | 82 | 83 | src\bits 84 | 85 | 86 | src\bits 87 | 88 | 89 | src\bits 90 | 91 | 92 | src\bits 93 | 94 | 95 | src\bits 96 | 97 | 98 | src\bits 99 | 100 | 101 | src\bits 102 | 103 | 104 | src\bits 105 | 106 | 107 | src\bits 108 | 109 | 110 | src\bits\impl 111 | 112 | 113 | src\bits\impl 114 | 115 | 116 | src\bits\impl 117 | 118 | 119 | src\bits\impl 120 | 121 | 122 | src\bits\impl 123 | 124 | 125 | src\bits\impl 126 | 127 | 128 | src\bits\impl 129 | 130 | 131 | src\bits\impl 132 | 133 | 134 | src\bits\impl 135 | 136 | 137 | src 138 | 139 | 140 | src 141 | 142 | 143 | src 144 | 145 | 146 | src 147 | 148 | 149 | src 150 | 151 | 152 | src 153 | 154 | 155 | 156 | 157 | {b811c080-cb75-4593-9fc5-d07b3566a222} 158 | 159 | 160 | {e325fc55-af0f-49b5-a02d-37eb6b182a74} 161 | 162 | 163 | {7abb3e48-e978-4d99-8f12-6af14582eebf} 164 | 165 | 166 | {ba2f396e-289f-47bc-b12c-2f07fece498c} 167 | 168 | 169 | 170 | 171 | src 172 | 173 | 174 | src 175 | 176 | 177 | src 178 | 179 | 180 | src 181 | 182 | 183 | src 184 | 185 | 186 | src 187 | 188 | 189 | src\bits 190 | 191 | 192 | src\bits 193 | 194 | 195 | src\bits 196 | 197 | 198 | src\bits 199 | 200 | 201 | src\bits 202 | 203 | 204 | src\bits 205 | 206 | 207 | src\bits 208 | 209 | 210 | src\bits 211 | 212 | 213 | src\bits 214 | 215 | 216 | src\bits\impl 217 | 218 | 219 | src\bits\impl 220 | 221 | 222 | src\bits\impl 223 | 224 | 225 | src\bits\impl 226 | 227 | 228 | src\bits\impl 229 | 230 | 231 | src\bits\impl 232 | 233 | 234 | src\bits\impl 235 | 236 | 237 | src\bits\impl 238 | 239 | 240 | src\bits\impl 241 | 242 | 243 | src\bits\impl 244 | 245 | 246 | src\bits 247 | 248 | 249 | src 250 | 251 | 252 | src 253 | 254 | 255 | src 256 | 257 | 258 | src 259 | 260 | 261 | src 262 | 263 | 264 | 265 | 266 | src\bits 267 | 268 | 269 | src\bits 270 | 271 | 272 | src\bits 273 | 274 | 275 | src\bits 276 | 277 | 278 | src\bits 279 | 280 | 281 | src\bits 282 | 283 | 284 | src\bits 285 | 286 | 287 | src\bits 288 | 289 | 290 | src\bits 291 | 292 | 293 | src\bits 294 | 295 | 296 | src\bits 297 | 298 | 299 | src\bits 300 | 301 | 302 | src\bits\impl 303 | 304 | 305 | src\bits\impl 306 | 307 | 308 | src\bits\impl 309 | 310 | 311 | src\bits\impl 312 | 313 | 314 | src\bits\impl 315 | 316 | 317 | src\bits\impl 318 | 319 | 320 | src\bits\impl 321 | 322 | 323 | src\bits\impl 324 | 325 | 326 | src\bits\impl 327 | 328 | 329 | src\bits\impl 330 | 331 | 332 | src\bits\impl 333 | 334 | 335 | src\bits\impl 336 | 337 | 338 | src\bits\impl 339 | 340 | 341 | src\bits\impl 342 | 343 | 344 | src\bits\impl 345 | 346 | 347 | src 348 | 349 | 350 | 351 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/test1.m: -------------------------------------------------------------------------------- 1 | % setup MatConvNet 2 | run matlab/vl_setupnn 3 | 4 | % download a pre-trained CNN from the web (needed once) 5 | % urlwrite(... 6 | % 'http://www.vlfeat.org/matconvnet/models/imagenet-googlenet-dag.mat', ... 7 | % 'imagenet-googlenet-dag.mat') ; 8 | 9 | % load the pre-trained CNN 10 | net = dagnn.DagNN.loadobj(load('imagenet-googlenet-dag.mat')) ; 11 | %net = dagnn.DagNN.loadobj(load('imagenet-vgg-verydeep-16.mat')) ; 12 | net.mode = 'test' ; 13 | 14 | % load and preprocess an image 15 | im = imread('peppers.png') ; 16 | im_ = single(im) ; % note: 0-255 range 17 | im_ = imresize(im_, net.meta.normalization.imageSize(1:2)) ; 18 | im_ = bsxfun(@minus, im_, net.meta.normalization.averageImage) ; 19 | 20 | % run the CNN 21 | net.eval({'data', im_}) ; 22 | 23 | % obtain the CNN otuput 24 | scores = net.vars(net.getVarIndex('prob')).value ; 25 | scores = squeeze(gather(scores)) ; 26 | 27 | % show the classification results 28 | [bestScore, best] = max(scores) ; 29 | figure(1) ; clf ; imagesc(im) ; 30 | title(sprintf('%s (%d), score %.3f',... 31 | net.meta.classes.description{best}, best, bestScore)) ; -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/test2.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/test2.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/GenerateRecoverBoxes_vggwithmask.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/GenerateRecoverBoxes_vggwithmask.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/Netfromvgg_scale1.m: -------------------------------------------------------------------------------- 1 | function net = Netfromvgg_scale1() 2 | 3 | %vggnet = load('../../data/imagenet-vgg-verydeep-16.mat') ; 4 | vggnet=load('data/imagenet-vgg-verydeep-16.mat'); 5 | net.layers = vggnet.layers(1:31); 6 | end 7 | 8 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/RecoverGenshapes_vggwithmask.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/RecoverGenshapes_vggwithmask.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/change2full.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/change2full.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/cnn_Im2strnet.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/cnn_Im2strnet.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/cnn_im2str_datasetup.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/cnn_im2str_datasetup.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/cnn_im2str_train.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/cnn_im2str_train.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/cnn_maskoutfeature.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/cnn_maskoutfeature.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/cnn_train_dag_im2strnet.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/cnn_train_dag_im2strnet.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/diagonal_distance.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/diagonal_distance.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/distence_gt_rec.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/distence_gt_rec.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/distence_rec_gt.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/distence_rec_gt.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/draw3dOBB_v2.m: -------------------------------------------------------------------------------- 1 | function draw3dOBB_v2(p, col) 2 | 3 | center = p(1:3); 4 | lengths = p(4:6); 5 | dir_1 = p(7:9); 6 | dir_2 = p(10:12); 7 | 8 | dir_1 = dir_1/norm(dir_1); 9 | dir_2 = dir_2/norm(dir_2); 10 | dir_3 = cross(dir_1,dir_2); 11 | dir_3 = dir_3/norm(dir_3); 12 | cornerpoints = zeros(8,3); 13 | 14 | d1 = 0.5*lengths(1)*dir_1; 15 | d2 = 0.5*lengths(2)*dir_2; 16 | d3 = 0.5*lengths(3)*dir_3; 17 | cornerpoints(1,:) = center-d1-d2-d3; 18 | cornerpoints(2,:) = center-d1+d2-d3; 19 | cornerpoints(3,:) = center+d1-d2-d3; 20 | cornerpoints(4,:) = center+d1+d2-d3; 21 | cornerpoints(5,:) = center-d1-d2+d3; 22 | cornerpoints(6,:) = center-d1+d2+d3; 23 | cornerpoints(7,:) = center+d1-d2+d3; 24 | cornerpoints(8,:) = center+d1+d2+d3; 25 | 26 | plot3([cornerpoints(1,1),cornerpoints(2,1)],[cornerpoints(1,2),cornerpoints(2,2)],[cornerpoints(1,3),cornerpoints(2,3)],col);hold on; 27 | plot3([cornerpoints(1,1),cornerpoints(3,1)],[cornerpoints(1,2),cornerpoints(3,2)],[cornerpoints(1,3),cornerpoints(3,3)],col);hold on; 28 | plot3([cornerpoints(2,1),cornerpoints(4,1)],[cornerpoints(2,2),cornerpoints(4,2)],[cornerpoints(2,3),cornerpoints(4,3)],col);hold on; 29 | plot3([cornerpoints(3,1),cornerpoints(4,1)],[cornerpoints(3,2),cornerpoints(4,2)],[cornerpoints(3,3),cornerpoints(4,3)],col);hold on; 30 | plot3([cornerpoints(5,1),cornerpoints(6,1)],[cornerpoints(5,2),cornerpoints(6,2)],[cornerpoints(5,3),cornerpoints(6,3)],col);hold on; 31 | plot3([cornerpoints(5,1),cornerpoints(7,1)],[cornerpoints(5,2),cornerpoints(7,2)],[cornerpoints(5,3),cornerpoints(7,3)],col);hold on; 32 | plot3([cornerpoints(6,1),cornerpoints(8,1)],[cornerpoints(6,2),cornerpoints(8,2)],[cornerpoints(6,3),cornerpoints(8,3)],col);hold on; 33 | plot3([cornerpoints(7,1),cornerpoints(8,1)],[cornerpoints(7,2),cornerpoints(8,2)],[cornerpoints(7,3),cornerpoints(8,3)],col);hold on; 34 | plot3([cornerpoints(1,1),cornerpoints(5,1)],[cornerpoints(1,2),cornerpoints(5,2)],[cornerpoints(1,3),cornerpoints(5,3)],col);hold on; 35 | plot3([cornerpoints(2,1),cornerpoints(6,1)],[cornerpoints(2,2),cornerpoints(6,2)],[cornerpoints(2,3),cornerpoints(6,3)],col);hold on; 36 | plot3([cornerpoints(3,1),cornerpoints(7,1)],[cornerpoints(3,2),cornerpoints(7,2)],[cornerpoints(3,3),cornerpoints(7,3)],col);hold on; 37 | plot3([cornerpoints(4,1),cornerpoints(8,1)],[cornerpoints(4,2),cornerpoints(8,2)],[cornerpoints(4,3),cornerpoints(8,3)],col);hold on; 38 | 39 | end 40 | 41 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/draw_vgg16_test_demo.m: -------------------------------------------------------------------------------- 1 | function draw_vgg16_test_demo 2 | %load('D:\exp_vgg16_mask\test_image\candidate\genShapes.mat'); 3 | load('C:\Users\niuchengjie\Desktop\chair\mask\genShapes.mat'); 4 | for ii=1:26 5 | %savefilepath_pre = 'D:\exp_vgg16_mask\test_image\candidate\'; 6 | savefilepath_pre = 'C:\Users\niuchengjie\Desktop\chair\mask\'; 7 | savefilepath=[savefilepath_pre num2str(ii) '.obb']; 8 | 9 | recover_boxes = genShapes{ii}.boxes; 10 | %recover_boxes=feature{1}; 11 | 12 | outputboxes = zeros(15, size(recover_boxes,2)); 13 | for jj = 1:size(recover_boxes,2) 14 | p = recover_boxes(:,jj); 15 | outputboxes(:,jj) = change2full(p); 16 | end 17 | 18 | fid = fopen(savefilepath, 'w'); 19 | fprintf(fid, '# interpolation shape \r\n'); 20 | fprintf(fid, 'N %d\r\n', size(outputboxes,2)); 21 | for jj = 1:size(outputboxes,2) 22 | box = outputboxes(:,jj); 23 | center = box(1:3); 24 | boxlength = box(4:6); 25 | 26 | dir_1 = box(7:9); 27 | dir_2 = box(10:12); 28 | dir_1 = dir_1/norm(dir_1); 29 | dir_2 = dir_2/norm(dir_2); 30 | dir_3 = cross(dir_1,dir_2); 31 | dir_3 = dir_3/norm(dir_3); 32 | 33 | fprintf(fid, '%f %f %f %f %f %f %f %f %f %f %f %f %f %f %f \r\n', center(1),center(2),center(3),dir_1(1),dir_1(2),dir_1(3),dir_2(1),dir_2(2),dir_2(3),... 34 | dir_3(1),dir_3(2),dir_3(3),boxlength(1),boxlength(2),boxlength(3)); 35 | end 36 | fclose(fid); 37 | end 38 | 39 | end -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/four_hausdorff_plot.m: -------------------------------------------------------------------------------- 1 | function four_hausdorff_plot 2 | vgg19mask=load('D:\eval_vgg_data\vgg_19_mask\hausdorff-300.mat'); 3 | vgg16mask=load('D:\eval_vgg_data\vgg_16_mask\hausdorff-200.mat'); 4 | vgg19only=load('D:\eval_vgg_data\vgg_19_only\hausdorff-300.mat'); 5 | vgg16only=load('D:\eval_vgg_data\vgg_16_only\hausdorff-200.mat'); 6 | 7 | huas_vgg19mask=vgg19mask.tmp; 8 | huas_vgg16mask=vgg16mask.tmp; 9 | huas_vgg19only=vgg19only.tmp; 10 | huas_vgg16only=vgg16only.tmp; 11 | for ii=1:length(huas_vgg19mask) 12 | huas_vgg19mask(5,ii)= 0.5*(huas_vgg19mask(3,ii)+ huas_vgg19mask(4,ii)); 13 | huas_vgg19only(5,ii)= 0.5*(huas_vgg19only(3,ii)+ huas_vgg19only(4,ii)); 14 | end 15 | 16 | for ii=1:length(huas_vgg16mask) 17 | huas_vgg16mask(5,ii)= 0.5*(huas_vgg16mask(3,ii)+ huas_vgg16mask(4,ii)); 18 | huas_vgg16only(5,ii)= 0.5*(huas_vgg16only(3,ii)+ huas_vgg16only(4,ii)); 19 | end 20 | 21 | huas_vgg19mask=[ huas_vgg19mask(:,1:40)]; 22 | huas_vgg19only=[ huas_vgg19only(:,1:40) ]; 23 | 24 | step=5; 25 | sumr=200; 26 | start=5; 27 | leg={}; 28 | 29 | hausdorff_plot(3,:)=huas_vgg16only(5,:); 30 | hausdorff_plot(4,:)=huas_vgg16mask(5,:); 31 | hausdorff_plot(1,:)=huas_vgg19only(5,:); 32 | hausdorff_plot(2,:)=huas_vgg19mask(5,:); 33 | for j=1:4 34 | y(j,:)=interp1(5:5:200,sort(hausdorff_plot(j,:),'descend'),5:200,'spline'); 35 | end 36 | 37 | %y(:,1:4)=y(:,5).*(1+rand(4:4)) 38 | % plot(start:5:sumr,hausdorff_plot,'o-',2:5:200,y1,'o-'); 39 | % plot(start:step:sumr,hausdorff_plot,'o-'); 40 | % plot(2:5:200,y1,'o-'); 41 | % y=smoothts(y,'b',2); 42 | plot(5:200,y,'-'); 43 | leg={'vgg16only','vgg16mask','vgg19only','vgg19mask'}; 44 | axis([0 200 0.05 0.25]); 45 | % if strcmp(p,'re2gtdist_') 46 | % title('re2gtdist') ; 47 | % end 48 | % if strcmp(p,'gt2redist_') 49 | % title('gt2redist') ; 50 | % end 51 | 52 | legend(leg{:}) ; 53 | end 54 | -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/hausdorff_diagonal_smallbox.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/Im2Struct_code/vgg_16_mask/hausdorff_diagonal_smallbox.m -------------------------------------------------------------------------------- /Im2Struct_training/Im2Struct_code/vgg_16_mask/hausdorff_diagonal_smallbox_proportion.m: -------------------------------------------------------------------------------- 1 | function hausdorff_diagonal_smallbox_proportion 2 | load('D:\eval_vgg_data\vgg_16_mask\ld_rec_gt_train'); 3 | load('D:\eval_vgg_data\vgg_16_mask\ld_rec_gt_test'); 4 | load('D:\eval_vgg_data\vgg_16_mask\ld_gt_rec_train'); 5 | load('D:\eval_vgg_data\vgg_16_mask\ld_gt_rec_test'); 6 | 7 | prop=0.1; 8 | 9 | switch prop 10 | case 0.2 11 | proportion=0.2; 12 | case 0.15 13 | proportion=0.15; 14 | case 0.1 15 | proportion=0.1; 16 | end 17 | ld_train_tem1=(ld_rec_gt_train (Y) ? (X) : (Y)) 6 | 7 | int mindex3(int x, int y, int z, int sizx, int sizy, int sizz, int wrap) 8 | { 9 | int index; 10 | if(wrap==1) 11 | { 12 | /* Positive modules */ 13 | x=(x % sizx + sizx) % sizx; 14 | y=(y % sizy + sizy) % sizy; 15 | z=(z % sizz + sizz) % sizz; 16 | } 17 | else if(wrap>1) 18 | { 19 | /* Clamp */ 20 | x=max(x,0); 21 | y=max(y,0); 22 | z=max(z,0); 23 | x=min(x,sizx-1); 24 | y=min(y,sizy-1); 25 | z=min(z,sizz-1); 26 | } 27 | index=z*sizx*sizy+y*sizx+x; 28 | return index; 29 | } 30 | 31 | mxLogical *draw_or_split(mxLogical *Volume,double AX,double AY,double AZ,double BX,double BY,double BZ,double CX,double CY,double CZ, int *VolumeSize, int wrap) 32 | { 33 | bool checkA, checkB, checkC; 34 | bool check1, check2, check3, check4, check5, check6; 35 | 36 | double dist1, dist2, dist3, maxdist; 37 | double DX,DY,DZ; 38 | /* Check if vertices outside */ 39 | if(wrap==0) 40 | { 41 | checkA=(AX<0)||(AY<0)||(AZ<0)||(AX>(VolumeSize[0]-1))||(AY>(VolumeSize[1]-1))||(AZ>(VolumeSize[2]-1)); 42 | checkB=(BX<0)||(BY<0)||(BZ<0)||(BX>(VolumeSize[0]-1))||(BY>(VolumeSize[1]-1))||(BZ>(VolumeSize[2]-1)); 43 | checkC=(CX<0)||(CY<0)||(CZ<0)||(CX>(VolumeSize[0]-1))||(CY>(VolumeSize[1]-1))||(CZ>(VolumeSize[2]-1)); 44 | 45 | check1=(AX<0)&&(BX<0)&&(CX<0); 46 | check2=(AY<0)&&(BY<0)&&(CY<0); 47 | check3=(AZ<0)&&(BZ<0)&&(CZ<0); 48 | check4=(AX>(VolumeSize[0]-1))&&(BX>(VolumeSize[0]-1))&&(CX>(VolumeSize[0]-1)); 49 | check5=(AY>(VolumeSize[1]-1))&&(BY>(VolumeSize[1]-1))&&(CY>(VolumeSize[1]-1)); 50 | check6=(AZ>(VolumeSize[2]-1))&&(BZ>(VolumeSize[2]-1))&&(CZ>(VolumeSize[2]-1)); 51 | 52 | /* Return if all vertices outside, on the same side */ 53 | if(check1||check2||check3||check4||check5||check6) 54 | { 55 | return Volume; 56 | } 57 | } 58 | 59 | dist1=(AX-BX)*(AX-BX)+(AY-BY)*(AY-BY)+(AZ-BZ)*(AZ-BZ); 60 | dist2=(CX-BX)*(CX-BX)+(CY-BY)*(CY-BY)+(CZ-BZ)*(CZ-BZ); 61 | dist3=(AX-CX)*(AX-CX)+(AY-CY)*(AY-CY)+(AZ-CZ)*(AZ-CZ); 62 | if(dist1>dist2) 63 | { 64 | if(dist1>dist3) 65 | { 66 | maxdist=dist1; 67 | if(maxdist>0.5) 68 | { 69 | DX=(AX+BX)/2; DY=(AY+BY)/2; DZ=(AZ+BZ)/2; 70 | Volume=draw_or_split(Volume,DX,DY,DZ,BX,BY,BZ,CX,CY,CZ,VolumeSize, wrap); 71 | Volume=draw_or_split(Volume,AX,AY,AZ,DX,DY,DZ,CX,CY,CZ,VolumeSize, wrap); 72 | } 73 | } 74 | else 75 | { 76 | maxdist=dist3; 77 | if(maxdist>0.5) 78 | { 79 | DX=(AX+CX)/2; DY=(AY+CY)/2; DZ=(AZ+CZ)/2; 80 | Volume=draw_or_split(Volume,DX,DY,DZ,BX,BY,BZ,CX,CY,CZ,VolumeSize, wrap); 81 | Volume=draw_or_split(Volume,AX,AY,AZ,BX,BY,BZ,DX,DY,DZ,VolumeSize, wrap); 82 | } 83 | 84 | } 85 | } 86 | else 87 | { 88 | if(dist2>dist3) 89 | { 90 | maxdist=dist2; 91 | DX=(CX+BX)/2; DY=(CY+BY)/2; DZ=(CZ+BZ)/2; 92 | if(maxdist>0.5) 93 | { 94 | Volume=draw_or_split(Volume,AX,AY,AZ,DX,DY,DZ,CX,CY,CZ,VolumeSize, wrap); 95 | Volume=draw_or_split(Volume,AX,AY,AZ,BX,BY,BZ,DX,DY,DZ,VolumeSize, wrap); 96 | } 97 | } 98 | else 99 | { 100 | maxdist=dist3; 101 | if(maxdist>0.5) 102 | { 103 | DX=(AX+CX)/2; DY=(AY+CY)/2; DZ=(AZ+CZ)/2; 104 | Volume=draw_or_split(Volume,DX,DY,DZ,BX,BY,BZ,CX,CY,CZ,VolumeSize, wrap); 105 | Volume=draw_or_split(Volume,AX,AY,AZ,BX,BY,BZ,DX,DY,DZ,VolumeSize, wrap); 106 | } 107 | 108 | } 109 | } 110 | if(wrap==0) 111 | { 112 | if(checkA==false) 113 | { 114 | Volume[mindex3((int)(AX+0.5),(int)(AY+0.5), (int)(AZ+0.5), VolumeSize[0], VolumeSize[1], VolumeSize[2], wrap)]=1; 115 | } 116 | if(checkB==false) 117 | { 118 | Volume[mindex3((int)(BX+0.5),(int)(BY+0.5), (int)(BZ+0.5), VolumeSize[0], VolumeSize[1], VolumeSize[2], wrap)]=1; 119 | } 120 | if(checkC==false) 121 | { 122 | Volume[mindex3((int)(CX+0.5),(int)(CY+0.5), (int)(CZ+0.5), VolumeSize[0], VolumeSize[1], VolumeSize[2], wrap)]=1; 123 | } 124 | } 125 | else 126 | { 127 | Volume[mindex3((int)(AX+0.5),(int)(AY+0.5), (int)(AZ+0.5), VolumeSize[0], VolumeSize[1], VolumeSize[2], wrap)]=1; 128 | Volume[mindex3((int)(BX+0.5),(int)(BY+0.5), (int)(BZ+0.5), VolumeSize[0], VolumeSize[1], VolumeSize[2], wrap)]=1; 129 | Volume[mindex3((int)(CX+0.5),(int)(CY+0.5), (int)(CZ+0.5), VolumeSize[0], VolumeSize[1], VolumeSize[2], wrap)]=1; 130 | } 131 | return Volume; 132 | } 133 | 134 | 135 | 136 | /* The matlab mex function */ 137 | void mexFunction( int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[] ) 138 | { 139 | double *FacesA, *FacesB, *FacesC, *VerticesX, *VerticesY, *VerticesZ, *VolumeSize; 140 | mxLogical *Volume; 141 | double *Wrapd; 142 | int wrap; 143 | double AX,AY,AZ; 144 | double BX,BY,BZ; 145 | double CX,CY,CZ; 146 | int i; 147 | int VolumeDims[3]={0,0,0}; 148 | 149 | const mwSize *FacesDims; 150 | int FacesN=0; 151 | 152 | /* Check for proper number of arguments. */ 153 | if(nrhs!=8) { 154 | mexErrMsgTxt("Eight inputs are required."); 155 | } else if(nlhs!=1) { 156 | mexErrMsgTxt("One output required"); 157 | } 158 | 159 | /* Read all inputs */ 160 | FacesA=mxGetPr(prhs[0]); 161 | FacesB=mxGetPr(prhs[1]); 162 | FacesC=mxGetPr(prhs[2]); 163 | VerticesX=mxGetPr(prhs[3]); 164 | VerticesY=mxGetPr(prhs[4]); 165 | VerticesZ=mxGetPr(prhs[5]); 166 | VolumeSize=mxGetPr(prhs[6]); 167 | Wrapd=mxGetPr(prhs[7]); 168 | wrap=(int)Wrapd[0]; 169 | 170 | FacesDims = mxGetDimensions(prhs[0]); 171 | FacesN=FacesDims[0]*FacesDims[1]; 172 | 173 | /* Create Output array */ 174 | VolumeDims[0]=(int)VolumeSize[0]; 175 | VolumeDims[1]=(int)VolumeSize[1]; 176 | VolumeDims[2]=(int)VolumeSize[2]; 177 | plhs[0] = mxCreateLogicalArray(3, VolumeDims); 178 | Volume = mxGetLogicals(plhs[0]); 179 | 180 | 181 | for (i=0; iwidth = width; 14 | this->height = height; 15 | this->depthOutput = depthOutput; 16 | this->imageOutput = imageOutput; 17 | this->filename = filename; 18 | this->distance = distance; 19 | this->elevation = elevation; 20 | this->azimuth = azimuth; 21 | this->yaw = yaw; 22 | this->sphereOrientation = sphereOrientation; 23 | this->A = A; 24 | this->R = R; 25 | this->T = T; 26 | this->unprojectOutput = unprojectOutput; 27 | this->meshData = meshData; 28 | this->writeFiles = writeFiles; 29 | this->offScreen = offScreen; 30 | this->AOutput = AOutput; 31 | this->ROutput = ROutput; 32 | this->TOutput = TOutput; 33 | this->lighting = lighting; 34 | this->distanceInc = distanceInc; 35 | this->deg_order = deg_order; 36 | this->getUnproject = getUnproject; 37 | } 38 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/Engine.h: -------------------------------------------------------------------------------- 1 | #ifndef __engine__ 2 | #define __engine__ 3 | 4 | #include 5 | #include 6 | #include 7 | using namespace std; 8 | 9 | const bool USE_CACHE = true; 10 | //if enabled, the first specified resolution remains regardless of further resolutions 11 | //#define INIT_ONCE 12 | 13 | 14 | class MeshData { 15 | vector vertices; 16 | vector colors; 17 | vector normals; 18 | int cbind; 19 | int nbind; 20 | 21 | public: 22 | 23 | MeshData() { 24 | } 25 | 26 | MeshData(vector _vertices, vector _colors, int _cbind, vector _normals, 27 | int _nbind) : 28 | vertices(_vertices), colors(_colors), normals(_normals), cbind(_cbind), nbind(_nbind) { 29 | } 30 | 31 | void copy(const MeshData &md) { 32 | vertices = md.vertices; 33 | colors = md.colors; 34 | normals = md.normals; 35 | cbind = md.cbind; 36 | nbind = md.nbind; 37 | } 38 | 39 | void clear() { 40 | vertices.clear(); 41 | normals.clear(); 42 | colors.clear(); 43 | 44 | } 45 | void setCbind(const int& cbind) { 46 | this->cbind = cbind; 47 | } 48 | void setColors(const vector& colors) { 49 | this->colors = colors; 50 | } 51 | void setNbind(const int& nbind) { 52 | this->nbind = nbind; 53 | } 54 | void setNormals(const vector& normals) { 55 | this->normals = normals; 56 | } 57 | void setVertices(const vector& vertices) { 58 | this->vertices = vertices; 59 | } 60 | const int& getCbind() const { 61 | return cbind; 62 | } 63 | const vector& getColors() const { 64 | return colors; 65 | } 66 | const int& getNbind() const { 67 | return nbind; 68 | } 69 | const vector& getNormals() const { 70 | return normals; 71 | } 72 | const vector& getVertices() const { 73 | return vertices; 74 | } 75 | }; 76 | 77 | class Engine { 78 | protected: 79 | int width; 80 | int height; 81 | double *depthOutput; 82 | unsigned char *imageOutput; 83 | string filename; 84 | double distance; 85 | double elevation; 86 | double azimuth; 87 | double yaw; 88 | bool sphereOrientation; 89 | double *A; 90 | double *R; 91 | double *T; 92 | double *AOutput; 93 | double *ROutput; 94 | double *TOutput; 95 | double *unprojectOutput; 96 | MeshData meshData; 97 | bool writeFiles; 98 | bool lighting; 99 | bool offScreen; 100 | bool getUnproject; 101 | double distanceInc; 102 | string deg_order; 103 | public: 104 | Engine(); 105 | virtual ~Engine(); 106 | 107 | void setParams(int width, int height, double *depthOutput, unsigned char *imageOutput, 108 | string filename, double distance, double elevation, double azimuth, double yaw, 109 | bool sphereOrientation, double *A, double *R, double *T, double *unprojectOutput, 110 | MeshData meshData, bool writeFiles, bool offScreen, double *AOutput, double *ROutput, 111 | double *TOutput, bool lighting, 112 | double distanceInc, string deg_order, bool getUnproject); 113 | 114 | const MeshData& getMeshData() const { 115 | return meshData; 116 | } 117 | 118 | virtual void init()=0; 119 | virtual void initDataFromFile()=0; 120 | virtual void initData()=0; 121 | virtual void initCamera()=0; 122 | virtual void initCanvas()=0; 123 | virtual void draw()=0; 124 | virtual void shutdown()=0; 125 | 126 | }; 127 | 128 | #endif // __engine__ 129 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/EngineOSG.h: -------------------------------------------------------------------------------- 1 | #ifndef __engineosg__ 2 | #define __engineosg__ 3 | 4 | #include 5 | #include "Engine.h" 6 | #include 7 | 8 | class EngineOSG : public Engine { 9 | osg::ref_ptr viewer; 10 | 11 | void drawFromFileAndData(); 12 | void drawPoints(vector &vertices); 13 | void setupSphereOrientation(); 14 | void getIntrisnicMatrix(); 15 | public: 16 | EngineOSG(); 17 | virtual ~EngineOSG(); 18 | void init(); 19 | void initDataFromFile(); 20 | void initData(); 21 | void initCamera(); 22 | void initCanvas(); 23 | void draw(); 24 | void shutdown(); 25 | friend class CaptureCB; 26 | friend class TextureCB; 27 | friend class KeyboardEventHandler; 28 | }; 29 | #endif // __engineosg__ 30 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/compile_renderer.m: -------------------------------------------------------------------------------- 1 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2 | % Modified by Chao Yao 3 | % compile the C++ code to .mexw64 file to use in matlab 2013a 64bit 4 | % make sure that your OpensceneGraph is x64 version and can run successfully 5 | % edit the libpath and incpath to your OSG installation directory 6 | %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 7 | 8 | % clc 9 | cmd1='mex -O -largeArrayDims -DA64BITS renderer.cpp depth.cpp Engine.cpp EngineOSG.cpp util.cpp -lOpenGL32 -lglu32 -losg -losgViewer -losgDB -losgGA -losgUtil -lOpenThreads'; 10 | 11 | %libpath=fullfile('E:\Depth_Project\Single_Object\code\OSG\OpenSceneGraph-3.0.1','lib'); 12 | %incpath=fullfile('E:\Depth_Project\Single_Object\code\OSG\OpenSceneGraph-3.0.1','include'); 13 | 14 | cmd=[cmd1 sprintf(' -L%s -I%s',libpath,incpath)]; 15 | 16 | disp('Executing:'); 17 | fprintf('%s\n',cmd); 18 | eval(cmd); 19 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/creatData.m: -------------------------------------------------------------------------------- 1 | %datapath = 'G:\Download\chair_data\training'; 2 | datapath='H:\0retry_cv\chair_300'; 3 | width=1024; 4 | height=width; 5 | folderlist = dir(datapath); 6 | 7 | shapenetdata = struct( 'images', zeros(256,256,3,4500,'uint8'), ... 8 | 'depths', zeros(256,256,1,4500,'uint16'),... 9 | 'set', zeros(1,4500,'uint8')); 10 | 11 | count = 1; 12 | for ii = 3:270 13 | disp(ii); 14 | if strcmp('.',folderlist(ii,1).name) || strcmp('..',folderlist(ii,1).name) 15 | continue; 16 | end 17 | 18 | objectname = fullfile(datapath, folderlist(ii,1).name, 'model.obj'); 19 | for jj = 1:6 20 | [depth, rendered, unproject, A, R, T]=renderer(width,height, objectname,0,0,-0.3,15,-30*(jj-1),1,'zxy'); 21 | rendered = imresize(rendered, [256 256]); 22 | depth = uint16(depth*10000); 23 | depth = imresize(depth, [256 256], 'nearest'); 24 | shapenetdata.images(:,:,:,count) = rendered; 25 | shapenetdata.depths(:,:,:,count) = uint16(depth); 26 | shapenetdata.set(:,count) = 1; 27 | count = count+1; 28 | end 29 | 30 | for jj = 1:6 31 | [depth, rendered, unproject, A, R, T]=renderer(width,height, objectname,0,0,-0.3,30,-30*(jj-1),0,'zxy'); 32 | rendered = imresize(rendered, [256 256]); 33 | depth = uint16(depth*10000); 34 | depth = imresize(depth, [256 256], 'nearest'); 35 | shapenetdata.images(:,:,:,count) = rendered; 36 | shapenetdata.depths(:,:,:,count) = uint16(depth); 37 | shapenetdata.set(:,count) = 1; 38 | count = count+1; 39 | end 40 | 41 | for jj = 1:6 42 | [depth, rendered, unproject, A, R, T]=renderer(width,height, objectname,0,0,-0.3,45,-30*(jj-1),0,'zxy'); 43 | rendered = imresize(rendered, [256 256]); 44 | depth = uint16(depth*10000); 45 | depth = imresize(depth, [256 256], 'nearest'); 46 | shapenetdata.images(:,:,:,count) = rendered; 47 | shapenetdata.depths(:,:,:,count) = uint16(depth); 48 | shapenetdata.set(:,count) = 1; 49 | count = count+1; 50 | end 51 | 52 | for kk = 1:2 53 | for jj = 1:6 54 | w1 = rand*60; 55 | w2 = -rand*90-(kk-1)*90; 56 | [depth, rendered, unproject, A, R, T]=renderer(width,height, objectname,0,0,-0.3,w1,w2,0,'zxy'); 57 | rendered = imresize(rendered, [256 256]); 58 | depth = uint16(depth*10000); 59 | depth = imresize(depth, [256 256], 'nearest'); 60 | shapenetdata.images(:,:,:,count) = rendered; 61 | shapenetdata.depths(:,:,:,count) = uint16(depth); 62 | shapenetdata.set(:,count) = 1; 63 | count = count+1; 64 | end 65 | end 66 | 67 | % 68 | % 69 | % [depth, rendered, unproject, A, R, T]=renderer(width,height, objectname,0,0,-0.3,30,150,0,'zxy'); 70 | % % Displays the results. 71 | % rendered = imresize(rendered, [256 256]); 72 | % depth = imresize(depth, [256 256], 'nearest'); 73 | % %figure, imshow(depth); 74 | % figure, imshow(rendered); 75 | % imwrite(rendered, 'test.jpg'); 76 | % figure, imshow(depth); 77 | 78 | end 79 | 80 | save('shapenetData-18.mat', '-v7.3','shapenetdata'); 81 | 82 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/creatTestData.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/Render_simple/creatTestData.m -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/createlabeled.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/Render_simple/createlabeled.m -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/demo.m: -------------------------------------------------------------------------------- 1 | %% General initialization 2 | width=512; 3 | height=width; 4 | model_name='MiPOSat.osg'; 5 | %% Basic examples 6 | % Performs the rendering. However, since no output parameters were passed nothing is returned. 7 | % renderer(width,height,model_name); 8 | 9 | % Performs the rendering and writes data files (depth.pgm and rendered.png). Note that this is _not_ the preferred way of usage. 10 | % renderer(width,height,model_name,1); 11 | 12 | % Returns the depth matrix and rendered image to MATLAB. No output files are written. This is the preferred usage. 13 | %[depth, rendered]=renderer(width,height, model_name); 14 | 15 | % Displays the results. 16 | % figure, imshow(depth); figure, imshow(rendered); 17 | 18 | % Also returns the camera matrices - A,R and T as well as the unprojection matrix. 19 | % Now 'unproject(125,149,1:3)' returns the world XYZ coordinate of the image point (x=148,y=124) 20 | % [depth, rendered, unproject, A, R, T]=renderer(width,height, model_name); 21 | 22 | % Renders the mesh with a distance of 0.5, an elevation of 10 degrees, azimuth of 20 degrees and yaw of 30 degrees. 23 | [depth, rendered, unproject, A, R, T]=renderer(width,height, model_name,1,0,-0.3,15,-30,0,'zxy'); 24 | % Displays the results. 25 | figure, imshow(depth); figure, imshow(rendered); 26 | 27 | x = unproject(:,:,1); 28 | y = unproject(:,:,2); 29 | z = unproject(:,:,3); 30 | 31 | x =x(1:10:end); 32 | y =y(1:10:end); 33 | z =z(1:10:end); 34 | point3d = [x(:) y(:) z(:)]; 35 | 36 | plywrite('test.ply',point3d); 37 | 38 | unpro = A; 39 | save('unpro.mat', 'unpro'); 40 | 41 | % Providing the camera matrices returned from the previous example will yield the same results. 42 | % [depth, rendered, unproject]=renderer(width,height, model_name,0,0,A,R,T); 43 | % figure, imshow(rendered); 44 | 45 | 46 | % View the 3D points of the model 47 | % figure, plot3(unproject(:,:,1),unproject(:,:,2),unproject(:,:,3),'.'); 48 | %% Pose estimation sanity 49 | % [depth, rendered, unproject, A, R, T]=renderer(width, height, model_name); 50 | % figure, imshow(rendered); 51 | % 52 | % % collect dummy landmarks from the 2D rendered image and the 3D model points 53 | % % If you are fitting a different 2D image (texture), first resize it. e.g. 54 | % % texture=imresize(your_texture,[height width]); 55 | % % Next, annotate m points (landmarks) from 'texture' (2D points) into pts_2D (m by 2 matrix) and 'unproject' (3D points) into pts_3D (m by 3 matrix). 56 | % pts_2D=[183,22; 135,106; 159,83; 132,187; 141,210; 130,229]; 57 | % hold on; plot(pts_2D(:,1),pts_2D(:,2),'+') 58 | % pts_3D=zeros(size(pts_2D,1),3); 59 | % for j=1:size(pts_2D,1) 60 | % pts_3D(j,:)=unproject(pts_2D(j,2),pts_2D(j,1),1:3); 61 | % end; 62 | % 63 | % % calibrate - estimate the new camera matrices 64 | % addpath('../calib/') 65 | % [A,R_new,T_new]=doCalib(width,height,pts_2D,pts_3D,A,[],[]); 66 | % 67 | % % rendered_new should be almost equal to the original rendered 68 | % [~, rendered_new]=renderer(width, height, model_name,0,0,A,R_new,T_new); 69 | % figure, imshow(rendered_new); 70 | % %% Roate the model. First approach 71 | % for yaw=0:10:360 72 | % [~, rendered]=renderer(width,height, model_name,0,0,0.5,10,20,yaw,'zxy'); 73 | % figure(1); imshow(rendered); pause; 74 | % end 75 | % %% Roate the model. Another approach 76 | % [~, rendered, ~, A, R, T]=renderer(width,height, 'example.wrl'); 77 | % for yaw=deg2rad(0:10:360) 78 | % dcm = angle2dcm(-yaw, 0, 0); 79 | % [tmp, rendered]=renderer(width,height, 'example.wrl',0,0,A,dcm*R,T); 80 | % figure(1); imshow(rendered); pause; 81 | % end 82 | % %% Re-render the mesh using a different image size 83 | % [~, rendered, ~, A, R, T]=renderer(width,height, model_name); 84 | % new_width=610; new_height=914; 85 | % A(1:2,3)=[new_width/2;new_height/2]; 86 | % [~, rendered_new]=renderer(new_width, new_height, model_name,0,0,A,R,T); 87 | % imshow(rendered_new) 88 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/depth.cpp: -------------------------------------------------------------------------------- 1 | #include "depth.h" 2 | #include "util.h" 3 | #include 4 | 5 | 6 | 7 | void getRenderedImage(GLubyte *image, int gWidth , int gHeight, GLubyte *imageOutput) 8 | { 9 | if (NULL == imageOutput) { 10 | return; 11 | } 12 | 13 | // OpenGL images are bottom to top. Have to reverse. Plus, Matlab matrices are transposed 14 | for (int i = 0;i < gHeight;i++) { 15 | GLubyte *rowPtr2 = image + (gHeight - 1 - i) * gWidth * 3; 16 | for (int w = 0;w < gWidth;w++) { 17 | // red 18 | imageOutput[w*gHeight+i] = rowPtr2[w*3]; 19 | // green 20 | imageOutput[gHeight*gWidth+w*gHeight+i] = rowPtr2[w*3+1]; 21 | // blue 22 | imageOutput[gHeight*gWidth*2+w*gHeight+i] = rowPtr2[w*3+2]; 23 | } 24 | } 25 | } 26 | 27 | 28 | 29 | void printDebugGetDepthOutput(GLfloat *depth, int gWidth , int gHeight) 30 | { 31 | if (DEBUG) { 32 | GLfloat max = -1; 33 | GLfloat min = 1; 34 | for (int i = 0;i < gHeight*gWidth;i++) { 35 | double d = 1 - depth[i]; 36 | if (d > max && d != 1) { 37 | max = d; 38 | } 39 | if (d < min && d != 0) { 40 | min = d; 41 | } 42 | } 43 | 44 | sout << "max,min: " << max << ", " << min << endl; 45 | } 46 | 47 | } 48 | 49 | void CopyAndModifyDepth(GLfloat *depth, int gWidth , int gHeight, double *imgDepth) 50 | { 51 | printDebugGetDepthOutput(depth, gWidth , gHeight); 52 | 53 | //reverse depth color 54 | for (int i = 0;i < gWidth*gHeight;i++) { 55 | imgDepth[i] = 1 - depth[i]; 56 | } 57 | /*fix: if scaling, scale imgDepth, not depth 58 | if (min > 0) { 59 | float scale = 0.99 / (max - min + 0.01); 60 | for (i = 0;i < gHeight*gWidth;i++) { 61 | if (depth[i] > 0 && depth[i] < 1) { 62 | depth[i] -= (min - 0.01); 63 | depth[i] *= scale; 64 | } 65 | } 66 | }*/ 67 | } 68 | 69 | void getDepthOutput(double *imgDepth, int gWidth , int gHeight, double *depthOutput) 70 | { 71 | if (depthOutput != NULL) { 72 | transposeAndFlipY(imgDepth, gHeight, gWidth, depthOutput); 73 | } 74 | } 75 | 76 | void flipY(double *in, int m, int n, GLubyte *out) 77 | { 78 | for (int i = 0;i < m;i++) { 79 | for (int j = 0;j < n;j++) { 80 | out[i*n+j] = (GLubyte)in[(m-1-i)*n+j]; 81 | } 82 | } 83 | } 84 | 85 | void printDebugWriteDepthFile(int gWidth , int gHeight, GLubyte *newd, int MAX) 86 | { 87 | if (DEBUG) { 88 | int max = -1; 89 | int min = MAX + 1; 90 | for (int i = 0;i < gHeight*gWidth;i++) { 91 | if (newd[i] > max && newd[i] != MAX) { 92 | max = newd[i]; 93 | } 94 | if (newd[i] < min && newd[i] != 0) { 95 | min = newd[i]; 96 | } 97 | } 98 | sout << "new max,min: " << max << ", " << min << endl; 99 | } 100 | } 101 | 102 | 103 | void WriteDepthFile(const char *filename, double *imgDepth, int gWidth , int gHeight) 104 | { 105 | FILE *f; 106 | 107 | f = fopen(filename, "w"); 108 | if (!f) { 109 | printf("Couldn't open image file: %s\n", filename); 110 | return; 111 | } 112 | int MAX = 255; 113 | fprintf(f, "P5\n"); 114 | fprintf(f, "# pgm-file created by %s\n", "renderer"); 115 | fprintf(f, "%i %i\n", gWidth, gHeight); 116 | fprintf(f, "255\n"); 117 | fclose(f); 118 | f = fopen(filename, "ab"); /* now append binary data */ 119 | if (!f) { 120 | printf("Couldn't append to image file: %s\n", filename); 121 | return; 122 | } 123 | 124 | GLubyte *newd = (GLubyte *) malloc(gWidth * gHeight * sizeof(GLubyte)); 125 | for (int i = 0;i < gHeight*gWidth;i++) { 126 | imgDepth[i] *= MAX; 127 | } 128 | flipY(imgDepth, gWidth, gHeight, newd); 129 | 130 | printDebugWriteDepthFile(gWidth, gHeight, newd, MAX); 131 | 132 | fwrite(newd, sizeof(GLubyte), gWidth*gHeight, f); 133 | 134 | fclose(f); 135 | 136 | if (DEBUG) { 137 | sout << "Wrote " << gWidth << " by " << gHeight << " image file: " << filename << endl; 138 | } 139 | free(newd); 140 | 141 | } 142 | 143 | 144 | 145 | Depth::Depth() 146 | { 147 | } 148 | 149 | Depth::~Depth() 150 | { 151 | } 152 | 153 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/depth.h: -------------------------------------------------------------------------------- 1 | #ifndef __depth__ 2 | #define __depth__ 3 | #include 4 | 5 | #pragma comment(lib, "opengl32") 6 | #pragma comment(lib, "glu32") 7 | 8 | #include 9 | #include 10 | void getRenderedImage(GLubyte *image, int gWidth , int gHeight, unsigned char *imageOutput); 11 | void CopyAndModifyDepth(GLfloat *depth, int gWidth , int gHeight, double *imgDepth); 12 | void getDepthOutput(double *imgDepth, int gWidth , int gHeight, double *depthOutput); 13 | void WriteDepthFile(const char *filename, double *imgDepth, int gWidth , int gHeight); 14 | 15 | 16 | class Depth { 17 | 18 | public: 19 | Depth(); 20 | ~Depth(); 21 | 22 | }; 23 | #endif // __depth__ 24 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/depth.pgm: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/Render_simple/depth.pgm -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/frontdata_generate.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/Render_simple/frontdata_generate.m -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/gen_Ref_View.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/Render_simple/gen_Ref_View.m -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/modifyobj.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/Render_simple/modifyobj.m -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/myunproject.m: -------------------------------------------------------------------------------- 1 | 2 | 3 | width = 1024; 4 | height = 1024; 5 | 6 | depth = zeros(size(in), 'single'); 7 | for i = 1:width 8 | for j = 1:height 9 | depth(j,i) = in((height-j)*width+i); 10 | end 11 | end 12 | 13 | depth = 1-depth; 14 | 15 | load('unpro.mat'); 16 | 17 | % f_unpro = zeros(3,3,'single'); 18 | 19 | % for i = 1:3 20 | % for j = 1:3 21 | % f_unpro((j-1)*3+i) = unpro((i-1)*3+j); 22 | % end 23 | % end 24 | 25 | t = unpro(:,1); 26 | 27 | unpro(:,1) = unpro(:,2); 28 | 29 | unpro(:,2) = t; 30 | 31 | p3d = zeros(height,width,3,'single'); 32 | for winy = height:-1:1 33 | for winx = 1:width 34 | winz = depth((winx-1)*height+winy); 35 | if (winz>0 && winz < 1) 36 | p = [ winy+1,winx, winz]; 37 | pos = p*(unpro); 38 | p3d(height+1-winy,winx,1) = pos(1); 39 | p3d(height+1-winy,winx,2) = pos(2); 40 | p3d(height+1-winy,winx,3) = pos(3); 41 | end 42 | 43 | end 44 | end 45 | 46 | x = p3d(:,:,1); 47 | y = p3d(:,:,2); 48 | z = p3d(:,:,3); 49 | 50 | x = x(:); 51 | x = x(1:10:end); 52 | 53 | y = y(:); 54 | y = y(1:10:end); 55 | 56 | z = z(:); 57 | z = z(1:10:end); 58 | 59 | 60 | 61 | point3d = [x(:) y(:) z(:)]; 62 | plywrite('recover.ply',point3d); 63 | 64 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/plywrite.m: -------------------------------------------------------------------------------- 1 | function plywrite(PLYfilename, coordinate, rgb, normals) 2 | % coordinate is 3 * n single matrix for n points 3 | % rgb is 3 * n uint8 matrix for n points range [0, 255] 4 | 5 | if size(coordinate,2)==3 && size(coordinate,1)~=3 6 | coordinate = coordinate'; 7 | end 8 | 9 | isValid = (~isnan(coordinate(1,:))) & (~isnan(coordinate(2,:))) & (~isnan(coordinate(3,:))); 10 | coordinate = coordinate(:,isValid); 11 | 12 | data = reshape(typecast(reshape(single(coordinate),1,[]),'uint8'),3*4,[]); 13 | 14 | if exist('normals','var') && ~isempty(normals) 15 | normals = reshape(typecast(reshape(single(normals),1,[]),'uint8'),3*4,[]); 16 | data = [data; normals]; 17 | end 18 | 19 | if exist('rgb','var') && ~isempty(rgb) 20 | if size(rgb,2)==3 && size(rgb,1)~=3 21 | rgb = rgb'; 22 | end 23 | 24 | if ~isa(rgb,'uint8') 25 | if max(rgb(:))<=1 26 | rgb = rgb * 255; 27 | end 28 | end 29 | 30 | if isa(rgb,'double') 31 | rgb = uint8(rgb); 32 | end 33 | 34 | rgb = rgb(:,isValid); 35 | data = [data; rgb]; 36 | end 37 | 38 | file = fopen(PLYfilename,'w'); 39 | fprintf (file, 'ply\n'); 40 | fprintf (file, 'format binary_little_endian 1.0\n'); 41 | fprintf (file, 'element vertex %d\n', size(data,2)); 42 | fprintf (file, 'property float x\n'); 43 | fprintf (file, 'property float y\n'); 44 | fprintf (file, 'property float z\n'); 45 | if exist('normals','var') && ~isempty(normals) 46 | fprintf (file, 'property float nx\n'); 47 | fprintf (file, 'property float ny\n'); 48 | fprintf (file, 'property float nz\n'); 49 | end 50 | if exist('rgb','var') && ~isempty(rgb) 51 | fprintf (file, 'property uchar red\n'); 52 | fprintf (file, 'property uchar green\n'); 53 | fprintf (file, 'property uchar blue\n'); 54 | end 55 | fprintf (file, 'end_header\n'); 56 | 57 | fwrite(file, data,'uint8'); 58 | fclose(file); 59 | 60 | end -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/renderer.mexw64: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/Render_simple/renderer.mexw64 -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/scr_new.txt: -------------------------------------------------------------------------------- 1 | sdafdasfdafd 2 | adsfdsfsdfdsfds 3 | dfgdgfdrg 4 | mtl sdljfslf 5 | mtllib model.mtl 6 | sdfsfdsagdsgfdg 7 | dgffdgfgfd 8 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/test.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/Render_simple/test.jpg -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/test.ply: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/Render_simple/test.ply -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/unpro.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/Render_simple/unpro.mat -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/unproject.asv: -------------------------------------------------------------------------------- 1 | returnl 2 | width = 1024; 3 | height = 1024; 4 | 5 | depth = zeros(size(in)); 6 | for i = 0:width 7 | for j = 0:height 8 | depth((height-1-j)*width+i) = in(i*height+j); 9 | end 10 | end 11 | 12 | depth = 1-depth; 13 | 14 | load(unpro.mat); 15 | 16 | p3d = zeros(height,width,1,'single'); 17 | for winy = height-1:-1:0 18 | for winx = 0:width-1 19 | winz = depth(winy*width+winx); 20 | if (winz>0 && winz < 1) 21 | p = [winx, winy+1,winz]; 22 | p3d(winx+1,height-winy) = p*unpro; 23 | end 24 | 25 | end 26 | end -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/Render_simple/util.cpp: -------------------------------------------------------------------------------- 1 | #include "util.h" 2 | 3 | ostringstream sout(ostringstream::out); 4 | 5 | 6 | ostream& tab(ostream& output) 7 | { 8 | return output << '\t'; 9 | } 10 | 11 | void printMatrix(double *in, int m, int n) 12 | { 13 | for (int i = 0;i < m;i++) { 14 | for (int j = 0;j < n;j++) { 15 | sout << in[i*n+j] << tab; 16 | } 17 | 18 | sout << ";" << endl; 19 | } 20 | sout < 5 | #include 6 | #include 7 | //#include 8 | 9 | //#define printf mexPrintf 10 | 11 | using namespace std; 12 | 13 | //const bool DEBUG = true; 14 | const bool DEBUG = false; 15 | 16 | const unsigned int TWO = 2; 17 | const unsigned int TRIPLET = 3; 18 | const unsigned int QUADLET = 4; 19 | 20 | //debug output 21 | extern ostringstream sout; 22 | 23 | template < class T > 24 | string toString(const T &arg) 25 | { 26 | ostringstream out; 27 | 28 | out << arg; 29 | 30 | return (out.str()); 31 | } 32 | 33 | ostream& tab(ostream& output); 34 | 35 | void printMatrix(double *in, int m, int n); 36 | void printMatrix(float *in, int m, int n); 37 | 38 | void transposeAndFlipY(double *in, int m, int n, double *out); 39 | 40 | void transpose(double *in, int m, int n, double *out); 41 | 42 | void transpose3dim(unsigned char *image, int gWidth , int gHeight,unsigned char *imageOutput); 43 | void transpose3dimBGR(unsigned char *image, int gWidth , int gHeight,unsigned char *imageOutput); 44 | 45 | void getOpenGLMatrices(double *A, double *R, double *T, int width, int height, double mv[16], 46 | double projectionMatrix[16]); 47 | 48 | void getCameraMatricesFromOpenGL(double *A, double *R, double *T, int width, int height, 49 | double mv[16], double projectionMatrix[16]); 50 | 51 | #endif 52 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/addbackground.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/addbackground.m -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/image_obtain/creatTestData.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/image_obtain/creatTestData.m -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/obb_hierarchy/CreateShapeOBB_adj_vae.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/obb_hierarchy/CreateShapeOBB_adj_vae.m -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/obb_hierarchy/CreateShapeOBB_adj_vae_plane.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/obb_hierarchy/CreateShapeOBB_adj_vae_plane.m -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/obb_hierarchy/changeObbPath.m: -------------------------------------------------------------------------------- 1 | % datapath='H:\0retry_cv\training_data_models' 2 | % savepath='H:\0retry_cv\data\obb'; 3 | % folderlist = dir(datapath); 4 | % delete H:\0retry_cv\data\obb\*.obb; 5 | datapath = 'C:\Users\niuchengjie\Desktop\partNet_data\airplane-labeled\airplane-labeled'; 6 | savepath = 'C:\Users\niuchengjie\Desktop\partNet_data\airplane-labeled\obb_plane' 7 | folderlist = dir(datapath); 8 | for ii = 3:length(folderlist) 9 | disp(ii); 10 | if strcmp('.',folderlist(ii,1).name) || strcmp('..',folderlist(ii,1).name) 11 | continue; 12 | end 13 | objectname = fullfile(datapath,folderlist(ii,1).name,'labeled\model_seg_2.obb'); 14 | index=ii-2; 15 | savename=fullfile(savepath,[folderlist(ii,1).name,'.obb']); 16 | copyfile(objectname,savename); 17 | end -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/obb_hierarchy/draw3dOBB_v2.m: -------------------------------------------------------------------------------- 1 | function draw3dOBB_v2(p, col) 2 | 3 | center = p(1:3); 4 | lengths = p(13:15); 5 | dir_1 = p(4:6); 6 | dir_2 = p(7:9); 7 | 8 | dir_1 = dir_1/norm(dir_1); 9 | dir_2 = dir_2/norm(dir_2); 10 | dir_3 = cross(dir_1,dir_2); 11 | dir_3 = dir_3/norm(dir_3); 12 | cornerpoints = zeros(8,3); 13 | 14 | d1 = 0.5*lengths(1)*dir_1; 15 | d2 = 0.5*lengths(2)*dir_2; 16 | d3 = 0.5*lengths(3)*dir_3; 17 | cornerpoints(1,:) = center-d1-d2-d3; 18 | cornerpoints(2,:) = center-d1+d2-d3; 19 | cornerpoints(3,:) = center+d1-d2-d3; 20 | cornerpoints(4,:) = center+d1+d2-d3; 21 | cornerpoints(5,:) = center-d1-d2+d3; 22 | cornerpoints(6,:) = center-d1+d2+d3; 23 | cornerpoints(7,:) = center+d1-d2+d3; 24 | cornerpoints(8,:) = center+d1+d2+d3; 25 | 26 | plot3([cornerpoints(1,1),cornerpoints(2,1)],[cornerpoints(1,2),cornerpoints(2,2)],[cornerpoints(1,3),cornerpoints(2,3)],col);hold on; 27 | plot3([cornerpoints(1,1),cornerpoints(3,1)],[cornerpoints(1,2),cornerpoints(3,2)],[cornerpoints(1,3),cornerpoints(3,3)],col);hold on; 28 | plot3([cornerpoints(2,1),cornerpoints(4,1)],[cornerpoints(2,2),cornerpoints(4,2)],[cornerpoints(2,3),cornerpoints(4,3)],col);hold on; 29 | plot3([cornerpoints(3,1),cornerpoints(4,1)],[cornerpoints(3,2),cornerpoints(4,2)],[cornerpoints(3,3),cornerpoints(4,3)],col);hold on; 30 | plot3([cornerpoints(5,1),cornerpoints(6,1)],[cornerpoints(5,2),cornerpoints(6,2)],[cornerpoints(5,3),cornerpoints(6,3)],col);hold on; 31 | plot3([cornerpoints(5,1),cornerpoints(7,1)],[cornerpoints(5,2),cornerpoints(7,2)],[cornerpoints(5,3),cornerpoints(7,3)],col);hold on; 32 | plot3([cornerpoints(6,1),cornerpoints(8,1)],[cornerpoints(6,2),cornerpoints(8,2)],[cornerpoints(6,3),cornerpoints(8,3)],col);hold on; 33 | plot3([cornerpoints(7,1),cornerpoints(8,1)],[cornerpoints(7,2),cornerpoints(8,2)],[cornerpoints(7,3),cornerpoints(8,3)],col);hold on; 34 | plot3([cornerpoints(1,1),cornerpoints(5,1)],[cornerpoints(1,2),cornerpoints(5,2)],[cornerpoints(1,3),cornerpoints(5,3)],col);hold on; 35 | plot3([cornerpoints(2,1),cornerpoints(6,1)],[cornerpoints(2,2),cornerpoints(6,2)],[cornerpoints(2,3),cornerpoints(6,3)],col);hold on; 36 | plot3([cornerpoints(3,1),cornerpoints(7,1)],[cornerpoints(3,2),cornerpoints(7,2)],[cornerpoints(3,3),cornerpoints(7,3)],col);hold on; 37 | plot3([cornerpoints(4,1),cornerpoints(8,1)],[cornerpoints(4,2),cornerpoints(8,2)],[cornerpoints(4,3),cornerpoints(8,3)],col);hold on; 38 | 39 | end 40 | 41 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/obb_hierarchy/readObbShapes_vae.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/obb_hierarchy/readObbShapes_vae.m -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/obb_hierarchy/showGenshapes.m: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/Im2Struct_training/data_pre/obb_hierarchy/showGenshapes.m -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/obj_align.m: -------------------------------------------------------------------------------- 1 | % align obj 2 | % datapath = 'H:\0retry_cv\04379243\04379243' 3 | % folderlist = dir(datapath); 4 | % count = 1; 5 | % for ii =1:502 6 | % disp(ii); 7 | % if strcmp('.',folderlist(ii,1).name) || strcmp('..',folderlist(ii,1).name) 8 | % continue; 9 | % end 10 | % objectname = fullfile(datapath,folderlist(ii,1).name,'model.obj'); 11 | % [v,f]=obj__read(objectname); 12 | % vertices=v; 13 | % MULT1 = (max(vertices(1,:)) - min(vertices(1,:))); 14 | % MULT2 = (max(vertices(2,:)) - min(vertices(2,:))); 15 | % MULT3 = (max(vertices(3,:)) - min(vertices(3,:))); 16 | % MULT = max(max(MULT1,MULT2), MULT3); 17 | % disp([max(vertices(1,:)),max(vertices(2,:)),max(vertices(3,:))]); 18 | % disp([min(vertices(1,:)),min(vertices(2,:)),min(vertices(3,:))]); 19 | % % vertices(1,:) = vertices(1,:) / MULT*64 +32; 20 | % % vertices(2,:) = vertices(2,:) / MULT*64 +32; 21 | % % vertices(3,:) = vertices(3,:) / MULT*64 +32; 22 | % % FV.vertices = vertices'; 23 | % end 24 | 25 | datapath = 'H:\0retry_cv\PartNet_data\Table\Table\models'; 26 | savepath = 'H:\0retry_cv\PartNet_data\Table\Table\alignmodels'; 27 | folderlist = dir(datapath); 28 | count = 1; 29 | for ii =1:length(folderlist) 30 | disp(ii); 31 | if strcmp('.',folderlist(ii,1).name) || strcmp('..',folderlist(ii,1).name) 32 | continue; 33 | end 34 | objectname = fullfile(datapath,folderlist(ii,1).name); 35 | [v,f]=obj__read(objectname); 36 | vertices=v; 37 | MULT1 = (max(vertices(1,:)) - min(vertices(1,:))); 38 | MULT2 = (max(vertices(2,:)) - min(vertices(2,:))); 39 | MULT3 = (max(vertices(3,:)) - min(vertices(3,:))); 40 | MULT = max(max(MULT1,MULT2), MULT3); 41 | % disp([max(vertices(1,:)),max(vertices(2,:)),max(vertices(3,:))]); 42 | % disp([min(vertices(1,:)),min(vertices(2,:)),min(vertices(3,:))]); 43 | vertices(1,:) = (vertices(1,:) + (MULT1/2 - max(vertices(1,:)))) / MULT; 44 | vertices(2,:) = (vertices(2,:) + (MULT2/2 - max(vertices(2,:)))) / MULT; 45 | vertices(3,:) = (vertices(3,:) + (MULT3/2 - max(vertices(3,:)))) / MULT; 46 | vertices = flipud(vertices); 47 | FV.vertices = vertices'; 48 | obj_write(fullfile(savepath,folderlist(ii,1).name), FV.vertices, f'); 49 | end 50 | 51 | 52 | -------------------------------------------------------------------------------- /Im2Struct_training/data_pre/partnet_align.m: -------------------------------------------------------------------------------- 1 | %align partnet and shapenet 2 | shapenet = load('C:\Users\niuchengjie\Desktop\partNet_data\tabel_image\shapenet\shapenet_depth.mat'); 3 | partnet = load('C:\Users\niuchengjie\Desktop\partNet_data\tabel_image\partnet\partnet_depth.mat'); 4 | count = 0; 5 | for i = 1:length(shapenet.rendereddata) 6 | %disp(i) 7 | for j =1 : length(partnet.rendereddata) 8 | m = abs(shapenet.rendereddata{i}.images{1} - partnet.rendereddata{j}.images{1}); 9 | if sum(sum(m))< 1.05e+03 10 | count = count+1; 11 | disp('ok') 12 | disp(sum(sum(m))) 13 | spalign{count}.shapename = shapenet.rendereddata{i}.modelname; 14 | spalign{count}.partname = partnet.rendereddata{j}.modelname; 15 | break 16 | end 17 | end 18 | end 19 | % spath = 'C:\Users\niuchengjie\Desktop\partNet_data\tabel_image\shapenet'; 20 | % ppath = 'C:\Users\niuchengjie\Desktop\partNet_data\tabel_image\partnet'; 21 | % imagesavepath = 'C:\Users\niuchengjie\Desktop\partNet_data\tabel_image\compare'; 22 | % for i = 1: length(spalign) 23 | % disp(i) 24 | % sdepth = imread([ fullfile(spath, spalign{i}.shapename) '.jpg']); 25 | % imwrite(sdepth,[imagesavepath '\' num2str(i) '_s.jpg']) 26 | % pdepth = imread( [fullfile(ppath, spalign{i}.partname) '.jpg']); 27 | % imwrite(pdepth,[imagesavepath '\' num2str(i) '_p.jpg']) 28 | % end -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Im2Struct: Recovering 3D Shape Structure from a Single RGB Image 2 | By Chengjie Niu, Jun Li, Kai Xu. 3 | 4 | This repository contains the pre-trained models and testing code for recovering 3D shape structures from single RGB images. 5 | 6 | 7 | ## Citation 8 | If you find our work useful in your research, please consider citing: 9 | 10 | > @inProceedings{niu_cvpr18, 11 | > title={Im2Struct: Recovering 3D Shape Structure from a Single RGB Image}, 12 | > author = {Chengjie Niu 13 | > and Jun Li 14 | > and Kai Xu}, 15 | > booktitle={Computer Vision and Pattern Regognition (CVPR)}, 16 | > year={2018} 17 | > } 18 | 19 | ## Guide: 20 | 21 | Our current release has been tested on Matlab 2015b 22 | 23 | #### 1. Download and compile MatConvNet(http://www.vlfeat.org/matconvnet/) 24 | run matlab/vl_setupnn; 25 | 26 | vl_compilenn('enableGpu', true, ... 27 | 'cudaRoot', '/Developer/NVIDIA/CUDA-X.X', ... 28 | 'cudaMethod', 'nvcc'); 29 | 30 | 31 | #### 2. Download our pre-trained model 32 | 33 | Please download the pre-trained model according to the description in model/download.txt. 34 | 35 | #### 3. Run im2struct_demo.m 36 | Run im2struct_demo.m to recover 3D shape structure from an single RGB image with our pre-trained model. The recovered 3D shape structures can be visulized in Matlab. 37 | 38 | The recovered 3D shape structures for example images(data/example_1/2.jpg) should look as follows: 39 | 40 | 41 | ![Alt text](https://github.com/chengjieniu/Im2Struct/raw/master/data/example_1.jpg) 42 | ![Alt text](https://github.com/chengjieniu/Im2Struct/raw/master/data/example_2.jpg) 43 | ![Alt text](https://github.com/chengjieniu/Im2Struct/raw/master/image_show/1.png) 44 | ![Alt text](https://github.com/chengjieniu/Im2Struct/raw/master/image_show/2.png) 45 | 46 | 47 | For any questions, please contact Chengjie Niu(nchengjie@gmail.com). 48 | 49 | 50 | -------------------------------------------------------------------------------- /code/draw3DOBB.m: -------------------------------------------------------------------------------- 1 | function draw3DOBB(p, col) 2 | %draw obbs 3 | center = p(1:3); 4 | lengths = p(4:6); 5 | dir_1 = p(7:9); 6 | dir_2 = p(10:12); 7 | 8 | dir_1 = dir_1/norm(dir_1); 9 | dir_2 = dir_2/norm(dir_2); 10 | dir_3 = cross(dir_1,dir_2); 11 | dir_3 = dir_3/norm(dir_3); 12 | cornerpoints = zeros(8,3); 13 | 14 | d1 = 0.5*lengths(1)*dir_1; 15 | d2 = 0.5*lengths(2)*dir_2; 16 | d3 = 0.5*lengths(3)*dir_3; 17 | cornerpoints(1,:) = center-d1-d2-d3; 18 | cornerpoints(2,:) = center-d1+d2-d3; 19 | cornerpoints(3,:) = center+d1-d2-d3; 20 | cornerpoints(4,:) = center+d1+d2-d3; 21 | cornerpoints(5,:) = center-d1-d2+d3; 22 | cornerpoints(6,:) = center-d1+d2+d3; 23 | cornerpoints(7,:) = center+d1-d2+d3; 24 | cornerpoints(8,:) = center+d1+d2+d3; 25 | 26 | plot3([cornerpoints(1,1),cornerpoints(2,1)],[cornerpoints(1,2),cornerpoints(2,2)],[cornerpoints(1,3),cornerpoints(2,3)],col);hold on; 27 | plot3([cornerpoints(1,1),cornerpoints(3,1)],[cornerpoints(1,2),cornerpoints(3,2)],[cornerpoints(1,3),cornerpoints(3,3)],col);hold on; 28 | plot3([cornerpoints(2,1),cornerpoints(4,1)],[cornerpoints(2,2),cornerpoints(4,2)],[cornerpoints(2,3),cornerpoints(4,3)],col);hold on; 29 | plot3([cornerpoints(3,1),cornerpoints(4,1)],[cornerpoints(3,2),cornerpoints(4,2)],[cornerpoints(3,3),cornerpoints(4,3)],col);hold on; 30 | plot3([cornerpoints(5,1),cornerpoints(6,1)],[cornerpoints(5,2),cornerpoints(6,2)],[cornerpoints(5,3),cornerpoints(6,3)],col);hold on; 31 | plot3([cornerpoints(5,1),cornerpoints(7,1)],[cornerpoints(5,2),cornerpoints(7,2)],[cornerpoints(5,3),cornerpoints(7,3)],col);hold on; 32 | plot3([cornerpoints(6,1),cornerpoints(8,1)],[cornerpoints(6,2),cornerpoints(8,2)],[cornerpoints(6,3),cornerpoints(8,3)],col);hold on; 33 | plot3([cornerpoints(7,1),cornerpoints(8,1)],[cornerpoints(7,2),cornerpoints(8,2)],[cornerpoints(7,3),cornerpoints(8,3)],col);hold on; 34 | plot3([cornerpoints(1,1),cornerpoints(5,1)],[cornerpoints(1,2),cornerpoints(5,2)],[cornerpoints(1,3),cornerpoints(5,3)],col);hold on; 35 | plot3([cornerpoints(2,1),cornerpoints(6,1)],[cornerpoints(2,2),cornerpoints(6,2)],[cornerpoints(2,3),cornerpoints(6,3)],col);hold on; 36 | plot3([cornerpoints(3,1),cornerpoints(7,1)],[cornerpoints(3,2),cornerpoints(7,2)],[cornerpoints(3,3),cornerpoints(7,3)],col);hold on; 37 | plot3([cornerpoints(4,1),cornerpoints(8,1)],[cornerpoints(4,2),cornerpoints(8,2)],[cornerpoints(4,3),cornerpoints(8,3)],col);hold on; 38 | 39 | end 40 | 41 | -------------------------------------------------------------------------------- /code/im2struct_demo.m: -------------------------------------------------------------------------------- 1 | function im2struct_demo 2 | 3 | %load structure masking network and recursive structure recovery network 4 | load('model\net_m_chair.mat'); 5 | net_m=dagnn.DagNN.loadobj(net_m); 6 | net_m.mode = 'test'; 7 | load('model\net_v_chair.mat'); 8 | net_v=dagnn.DagNN.loadobj(net_v); 9 | 10 | % for ii=1:54 11 | % net_m.params(ii).value=gpuArray( net_m.params(ii).value); 12 | % end 13 | 14 | vae_theta=net_v.meta.vae_theta; 15 | net_v.mode = 'test'; 16 | 17 | % load and preprocess an example image 18 | im=imread('data\example_3.jpg'); 19 | im_ = single(im) ; % note: 0-255 range 20 | im_ = imresize(im_, [224 224]) ; 21 | im_ = bsxfun(@minus, im_, net_m.meta.normalization.averageImage) ; 22 | im_ = gpuArray(im_); 23 | 24 | %get the feature map of the maskingnetwork 25 | inputs = {'input', im_, 'input2', im_} ; 26 | net_m.eval(inputs); 27 | 28 | % if isnan(net_m.getVarIndex('feature')) 29 | % mask_feature = net_m.vars(net_m.getVarIndex('x63')).value; 30 | % else 31 | % mask_feature = net_m.vars(net_m.getVarIndex('feature')).value; 32 | % end 33 | mask_feature = net_m.vars(net_m.getVarIndex('feature')).value; 34 | 35 | %get the 80D fused feature 36 | inputs_v={'input', inputs{2}, 'input2', mask_feature}; 37 | net_v.eval(inputs_v); 38 | res=net_v.vars(net_v.getVarIndex('prediction')).value; 39 | feature=gather(squeeze(res)); 40 | 41 | %decode the fused feature recursively into a box structure 42 | WdecoS1Left = vae_theta.WdecoS1Left; WdecoS1Right = vae_theta.WdecoS1Right; WdecoS2 = vae_theta.WdecoS2; 43 | WsymdecoS1 = vae_theta.WsymdecoS1; WsymdecoS2 = vae_theta.WsymdecoS2; 44 | WdecoBox = vae_theta.WdecoBox; 45 | bdecoS1Left = vae_theta.bdecoS1Left; bdecoS1Right = vae_theta.bdecoS1Right; bdecoS2 = vae_theta.bdecoS2; 46 | bsymdecoS1 = vae_theta.bsymdecoS1; bsymdecoS2 = vae_theta.bsymdecoS2; 47 | bdecoBox = vae_theta.bdecoBox; 48 | Wcat1 = vae_theta.Wcat1; bcat1 = vae_theta.bcat1; 49 | Wcat2 = vae_theta.Wcat2; bcat2 = vae_theta.bcat2; 50 | 51 | f = @norm1tanh; 52 | 53 | num = 1; 54 | genShapes = cell(1,1); 55 | for ii = 1:num 56 | %generate the shapes 57 | count = 1; 58 | features = gather(feature(:,ii)); 59 | 60 | symlist = 10*ones(8,1); 61 | while (size(features,2)) 62 | p = double(features(:,1)); 63 | sfm = f(Wcat1*p+bcat1); 64 | sm = softmax(Wcat2*sfm + bcat2); 65 | [~,l_index] = max(sm); 66 | if(l_index == 1) 67 | re_box = f(WdecoBox*p+bdecoBox); 68 | genShapes{ii}.boxes(:,count) = re_box; 69 | count = count+1; 70 | symfeature = symlist(:,1); 71 | if (abs(symfeature(1)+1) < 0.15 ) 72 | folds = uint8(1/symfeature(8)); 73 | newbox = re_box; 74 | symfeature(2:4) = symfeature(2:4)/norm(symfeature(2:4)); 75 | for kk = 1:folds-1 76 | rotvector = [symfeature(2:4); symfeature(8)*2*pi*single(kk)]; 77 | rotm = vrrotvec2mat(rotvector); 78 | center = re_box(1:3); 79 | dir_1 = re_box(7:9); 80 | dir_2 = re_box(10:12); 81 | newcenter = rotm*(center-symfeature(5:7))+symfeature(5:7); 82 | newbox(1:3) = newcenter; 83 | newbox(7:9) = rotm*dir_1; 84 | newbox(10:12) = rotm*dir_2; 85 | genShapes{ii}.boxes(:,count) = newbox; 86 | count = count+1; 87 | end 88 | end 89 | if (abs(symfeature(1)) < 0.25 ) 90 | trans = symfeature(2:4); 91 | trans_end = symfeature(5:7); 92 | center = re_box(1:3); 93 | [Transmax,TransmaxIndex]=max(abs(trans)); 94 | dimon=zeros(1,3)'; 95 | dimon(TransmaxIndex)=trans(TransmaxIndex); 96 | trans_length=Transmax; 97 | trans_total = sqrt(sum(trans_end-center).^2); 98 | folds = trans_total/trans_length; 99 | for kk = 1:folds 100 | newbox = re_box; 101 | newcenter=center+dimon*single(kk); 102 | newbox(1:3) = newcenter; 103 | genShapes{ii}.boxes(:,count) = newbox; 104 | count = count+1; 105 | end 106 | end 107 | if (abs(symfeature(1)-1) < 0.15 ) 108 | ref_normal = symfeature(2:4); 109 | ref_normal = ref_normal/norm(ref_normal); 110 | ref_point = symfeature(5:7); 111 | newbox = re_box; 112 | center = re_box(1:3); 113 | if (dot(ref_normal, ref_point-center) < 0) 114 | ref_normal = -ref_normal; 115 | end 116 | 117 | newcenter = abs(sum((ref_point-center).*ref_normal))*ref_normal*2+center; 118 | newbox(1:3) = newcenter; 119 | dir_1 = re_box(7:9); 120 | if (dot(ref_normal, dir_1) > 0) 121 | ref_normal = -ref_normal; 122 | end 123 | newbox(7:9) = dir_1 - 2*dot(dir_1, ref_normal)*ref_normal; 124 | dir_2 = re_box(10:12); 125 | if (dot(ref_normal, dir_2) > 0) 126 | ref_normal = -ref_normal; 127 | end 128 | newbox(10:12) = dir_2 - 2*dot(dir_2, ref_normal)*ref_normal; 129 | 130 | genShapes{ii}.boxes(:,count) = newbox; 131 | count = count+1; 132 | 133 | end 134 | symlist(:,1) = []; 135 | features(:,1) = []; 136 | else 137 | if (l_index == 3) 138 | ym = f(WsymdecoS2*p + bsymdecoS2); 139 | yp = f(WsymdecoS1*ym + bsymdecoS1); 140 | y1 = yp(1:end-8); 141 | features(:,1) = []; 142 | features = [features y1]; 143 | symfeature = yp(end-7:end); 144 | symlist(:,1) = []; 145 | symlist = [symlist symfeature]; 146 | else 147 | if(l_index == 2) 148 | ym = f(WdecoS2*p + bdecoS2); 149 | y1 = f(WdecoS1Left*ym + bdecoS1Left); 150 | y2 = f(WdecoS1Right*ym + bdecoS1Right); 151 | features(:,1) = []; 152 | features = [features y1 y2]; 153 | symlist = [symlist symlist(:,1) symlist(:,1)]; 154 | symlist(:,1) = []; 155 | end 156 | end 157 | 158 | end 159 | end 160 | end 161 | %show the result in figure 162 | showGenshapes(genShapes); 163 | end 164 | -------------------------------------------------------------------------------- /code/mask_myReshape.m: -------------------------------------------------------------------------------- 1 | classdef mask_myReshape < dagnn.ElementWise 2 | %one layer in net_mask 3 | 4 | properties 5 | dmap = [56; 56; 3] 6 | end 7 | 8 | properties (Transient) 9 | inputSizes = {} 10 | end 11 | 12 | methods 13 | function outputs = forward(obj, inputs, params) 14 | obj.inputSizes = cellfun(@size, inputs, 'UniformOutput', false) ; 15 | outputs{1} = reshape(inputs{1}, [obj.dmap(1) obj.dmap(2) obj.dmap(3) size(inputs{1},4)]) ; 16 | end 17 | 18 | function [derInputs, derParams] = backward(obj, inputs, params, derOutputs) 19 | derInputs{1} = reshape(derOutputs{1}, [1 1 obj.dmap(1)*obj.dmap(2)*obj.dmap(3) size(inputs{1},4)]) ; 20 | derParams = {} ; 21 | end 22 | 23 | function outputSizes = getOutputSizes(obj, inputSizes) 24 | outputSizes{1} = inputSizes{1} ; 25 | outputSizes{1}(1) = obj.dmap(1); 26 | outputSizes{1}(2) = obj.dmap(2); 27 | outputSizes{1}(3) = obj.dmap(3); 28 | end 29 | 30 | function obj = myReshape(varargin) 31 | obj.load(varargin) ; 32 | end 33 | end 34 | end 35 | -------------------------------------------------------------------------------- /code/norm1tanh.m: -------------------------------------------------------------------------------- 1 | function out = norm1tanh(x) 2 | out = tanh(x); 3 | end -------------------------------------------------------------------------------- /code/showGenshapes.m: -------------------------------------------------------------------------------- 1 | %show the structure hierarchy 2 | function showGenshapes(genshapes) 3 | 4 | for ii =1:length(genshapes) 5 | recover_boxes = genshapes{ii}.boxes; 6 | figure(ii+101); 7 | axis equal; 8 | axis([-0.7 0.7 -0.7 0.7 -0.7 0.7]); 9 | 10 | xlabel ( '--X axis--' ) 11 | ylabel ( '--Y axis--' ) 12 | zlabel ( '--Z axis--' ) 13 | hold on; 14 | 15 | view(145,-30); 16 | for jj = 1:size(recover_boxes,2) 17 | p = recover_boxes(:,jj); 18 | draw3DOBB(p,'r'); 19 | end 20 | 21 | end 22 | end 23 | -------------------------------------------------------------------------------- /data/airplane_1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/data/airplane_1.jpg -------------------------------------------------------------------------------- /data/airplane_2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/data/airplane_2.jpg -------------------------------------------------------------------------------- /data/example_1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/data/example_1.jpg -------------------------------------------------------------------------------- /data/example_2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/data/example_2.jpg -------------------------------------------------------------------------------- /data/example_3.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/data/example_3.jpg -------------------------------------------------------------------------------- /data/example_4.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/data/example_4.jpg -------------------------------------------------------------------------------- /image_show/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/image_show/1.png -------------------------------------------------------------------------------- /image_show/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/chengjieniu/Im2Struct/55c357e8242c5d626a90ecf2a0724bed024cf7ef/image_show/2.png -------------------------------------------------------------------------------- /model/download.txt: -------------------------------------------------------------------------------- 1 | You can download the pre-trained model(for chair) on dropbox: 2 | 3 | https://www.dropbox.com/sh/q007g6wu4jdgakl/AABasIx1C8OucFt2pGCnXqT-a?dl=0 4 | 5 | or this link for chair and airplane on google drive: 6 | https://drive.google.com/drive/folders/1zKhxfyPgOJ4-ocLiVcTlqXwGWYUcsalX?usp=sharing 7 | --------------------------------------------------------------------------------