├── README.md ├── __init__.py ├── bell2014 ├── __pycache__ │ └── density.cpython-35.pyc └── density.py ├── decompose.py ├── decomposition.py ├── density.py ├── docs ├── TheFastAndTheCurious_FinalPresentation1.pdf ├── TheFastAndTheCurious_FinalPresentation1.pptx └── proposal.pdf ├── energy ├── __init__.py ├── bsdfs.csv ├── energy.py ├── prob_abs_r.dat ├── prob_abs_r.py └── prob_abs_s.py ├── image_util.py ├── input.py ├── install_deps.sh ├── judgements.py ├── krahenbuhl2013 ├── .gitignore ├── Makefile ├── __init__.py ├── __init__.pyc ├── __pycache__ │ └── __init__.cpython-35.pyc ├── include │ ├── densecrf.h │ ├── densecrf_wrapper.h │ ├── labelcompatibility.h │ ├── objective.h │ ├── optimization.h │ ├── pairwise.h │ ├── permutohedral.h │ └── unary.h ├── krahenbuhl2013.cpython-35m-x86_64-linux-gnu.so ├── krahenbuhl2013.pyx ├── memory-test │ ├── memory_test.cpp │ ├── premake4 │ └── premake4.lua ├── setup.py ├── setup.py.bak └── src │ ├── densecrf.cpp │ ├── densecrf_wrapper.cpp │ ├── labelcompatibility.cpp │ ├── objective.cpp │ ├── optimization.cpp │ ├── pairwise.cpp │ ├── permutohedral.cpp │ ├── unary.cpp │ ├── util.cpp │ └── util.h ├── lmse.py ├── optimization.py ├── params.py ├── samples ├── 89e1aca9c871a9cd785c918c70fafc6f-r.png ├── 89e1aca9c871a9cd785c918c70fafc6f-s.png ├── 89e1aca9c871a9cd785c918c70fafc6f.jpg ├── baku.jpeg ├── bakul-r.png ├── bakul-s.png ├── gh-r.png ├── gh.jpeg └── ghs.png └── solver.py /README.md: -------------------------------------------------------------------------------- 1 | Intrinsic Image Decomposition 2 | ============================= 3 | 4 | This repository contains the decomposition algorithm presented in the [paper](https://dl.acm.org/citation.cfm?id=2601206): 5 | 6 | Sean Bell, Kavita Bala, Noah Snavely 7 | "Intrinsic Images in the Wild" 8 | ACM Transactions on Graphics (SIGGRAPH 2014) 9 | 10 | @article{bell14intrinsic, 11 | author = "Sean Bell and Kavita Bala and Noah Snavely", 12 | title = "Intrinsic Images in the Wild", 13 | journal = "ACM Trans. on Graphics (SIGGRAPH)", 14 | volume = "33", 15 | number = "4", 16 | year = "2014", 17 | } 18 | as well as a simple Python wrapper to the C++ dense CRF inference code from [Krahenbuhl et al 2013](http://graphics.stanford.edu/projects/drf/): 19 | 20 | Philipp Krähenbühl and Vladlen Koltun 21 | "Parameter Learning and Convergent Inference for Dense Random Fields" 22 | International Conference on Machine Learning (ICML) 2013 23 | 24 | The dataset is hosted at http://intrinsic.cs.cornell.edu/. 25 | - Note : The paper linked here is from ACM Digital Library. Make sure you have a valid subscription. 26 | 27 | Dependencies 28 | ------------ 29 | 30 | The following libraries are needed: 31 | 32 | :heavy_check_mark: Eigen (http://eigen.tuxfamily.org/) 33 | 34 | On Ubuntu 16.04, you can install with: `sudo apt-get install libeigen3-dev` 35 | 36 | :heavy_check_mark: SnakeViz 37 | 38 | On Ubuntu 16.04 you can use PyPI's command: `pip install snakeviz` 39 | 40 | :rotating_light: Numba (Proceed with caution) (https://pypi.org/project/numba/) 41 | 42 | Please check dependencies and proceed. 43 | 44 | :heavy_check_mark: Python 3.5.2 45 | 46 | :heavy_check_mark: Python packages (newer packages will likely work, though these are the exact versions that we used): 47 | 48 | Pillow==6.2.1 49 | Cython==0.29.14 50 | numpy==1.17.3 51 | scipy==1.3.2 52 | scikit-image==0.15.0 53 | scikit-learn==0.21.3 54 | 55 | :heavy_check_mark: cProfile is an optional requirement for advanced users. It is a package included with Python, however before proceeding please make sure cProfile exists. 56 | 57 | Setting up a virtual environment like `virtualenv` will help keep your Python environment safe. We recommend installing all dependencies using this. 58 | 59 | :heavy_check_mark: For people who want a `plug and play` solution to all installation hassles, an installation-cum-running script named install_deps.sh is provided. Run it using the following command: 60 | 61 | bash install_deps.sh decompose.py [name_of_image_file] 62 | 63 | :rotating_light: Note :- This installs `virtualenv` and `virtualenvwrapper` and edits your .bashrc on your machine.You have been warned. 64 | 65 | Compile 66 | ------- 67 | 68 | If on Ubuntu and you have installed Eigen3 to its default directory (/usr/include/eigen3), then you can build the C++ extension with: 69 | 70 | cd krahenbuhl2013/ 71 | make 72 | 73 | :rotating_light: Here you might get a ton of warnings, make sure you have properly specified the python version you are using. 74 | 75 | If you are on another operating system or `eigen3` is in another directory, edit `krahenbuhl2013/setup.py` to change the directory. 76 | 77 | 78 | Running 79 | ------- 80 | 81 | :heavy_check_mark: Basic usage: 82 | 83 | python3 decompose.py [image_filename] 84 | 85 | :rotating_light: Advanced (Checking memory stack): 86 | 87 | python3 -m cProfile -o decompose.prof decompose.py [image_filename] 88 | 89 | :heavy_check_mark: Visualising and seeing important memory heuristics (to target CPU/GPU optimisations): 90 | 91 | snakeviz decompose.prof 92 | 93 | All arguments: 94 | 95 | usage: decompose.py [-h] [-r ] [-s ] [-m ] [-j ] 96 | [-p ] [-l] [-q] [--show-labels] 97 | 98 | 99 | positional arguments: 100 | Input image 101 | 102 | optional arguments: 103 | -h, --help show this help message and exit 104 | -r , --reflectance 105 | Reflectance layer output name (saved as sRGB image) 106 | -s , --shading 107 | Shading layer output name (saved as sRGB image) 108 | -m , --mask 109 | Mask filename 110 | -j , --judgements 111 | Judgements file from the Intrinsic Images in the Wild 112 | dataset 113 | -p , --parameters 114 | Parameters file (JSON format). See params.py 115 | for a list of parameters. 116 | -l, --linear if specified, assume input is linear and generate 117 | linear output, otherwise assume input is sRGB and 118 | generate sRGB output 119 | -q, --quiet if specified, don't print logging info 120 | --show-labels if specified, also output labels 121 | 122 | 123 | Image style management 124 | ----------------------- 125 | 126 | All input images are assumed to be sRGB, and the output reflectance and shading layers are tone-mapped to sRGB. If using linear images (e.g., the MIT Intrinsic Images Dataset, http://www.cs.toronto.edu/~rgrosse/intrinsic/), specify `--linear` and both input/output will remain linear. 127 | 128 | Memory Optimisation 129 | ----------------------- 130 | 131 | - For advanced users who want more performance out of their rigs, we have added a memory profiling for you to target those number-crunching intensive loops and use Just-in-time compiling `jit` to optimise the runtime. A popular library is `numba` which we have attempted to use, but is moderately tough to configure. 132 | 133 | :rotating_light: To use `numba` make sure you have proper CUDA drivers and CUDA toolkit installed. For CPU based optimisations `llvm` is used. 134 | 135 | :heavy_check_mark: Snakeviz will give you memeory profiling based on which you can target the function you want 136 | 137 | :rotating_light: Note - `numba` is still an experimental library, with potential chances to corrupt your system. You have been warned. 138 | 139 | Embedding in other projects 140 | --------------------------- 141 | 142 | Our code was written to be modular and can be embedded in larger projects. The basic code for constructing the input and parameters can be found in `decompose.py` or is listed below: 143 | 144 | ```python 145 | from solver import IntrinsicSolver 146 | from input import IntrinsicInput 147 | from params import IntrinsicParameters 148 | import image_util 149 | 150 | input = IntrinsicInput.from_file( 151 | image_filename, 152 | image_is_srgb=sRGB, 153 | mask_filename=mask_filename, 154 | judgements_filename=judgements_filename, 155 | ) 156 | 157 | params = IntrinsicParameters.from_dict({ 158 | 'param1': ..., 159 | 'param2': ..., 160 | }) 161 | 162 | solver = IntrinsicSolver(input,params) 163 | r, s, decomposition = solver.solve() 164 | 165 | image_util.save(r_filename, r, mask_nz=input.mask_nz, rescale=True) 166 | image_util.save(s_filename, s, mask_nz=input.mask_nz, rescale=True) 167 | ``` 168 | 169 | Some results 170 | --------------------------------- 171 | :heavy_check_mark: Input Image: 172 | ![Paul Walker](https://github.com/DefUs3r/Intrinsic-Image-Decomposition/blob/master/samples/89e1aca9c871a9cd785c918c70fafc6f.jpg) 173 | 174 | :heavy_check_mark: Reflectance Layer: 175 | ![Paul Walker reflectance](https://github.com/DefUs3r/Intrinsic-Image-Decomposition/blob/master/samples/89e1aca9c871a9cd785c918c70fafc6f-r.png) 176 | 177 | :heavy_check_mark: Shading Layer: 178 | ![Paul Walker Shading](https://github.com/DefUs3r/Intrinsic-Image-Decomposition/blob/master/samples/89e1aca9c871a9cd785c918c70fafc6f-s.png) 179 | -------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/__init__.py -------------------------------------------------------------------------------- /bell2014/__pycache__/density.cpython-35.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/bell2014/__pycache__/density.cpython-35.pyc -------------------------------------------------------------------------------- /bell2014/density.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from scipy.ndimage.filters import gaussian_filter 3 | from numba import njit 4 | 5 | class ProbDensityHistogram(object): 6 | 7 | def train(self, training_data, bins=100, bandwidth=3, smoothing=1e-2): 8 | self.ndim = training_data.shape[1] 9 | self.bins = bins 10 | self.hist, self.edges = np.histogramdd( 11 | training_data, bins=bins, normed=True) 12 | self.hist[self.hist < smoothing] = smoothing 13 | if bandwidth: 14 | self.hist = gaussian_filter(self.hist, sigma=bandwidth) 15 | #self.hist /= np.sum(self.hist) 16 | self.hist = np.log(self.hist) 17 | 18 | def logprob(self, samples): 19 | indices = [np.digitize(samples[:, i], self.edges[i]) 20 | for i in range(self.ndim)] 21 | for i in range(self.ndim): 22 | np.clip(indices[i], 0, self.bins - 1, out=indices[i]) 23 | ret = self.hist[indices] 24 | assert ret.shape[0] == samples.shape[0] 25 | return ret 26 | -------------------------------------------------------------------------------- /decompose.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import argparse 4 | 5 | 6 | 7 | if __name__ == '__main__' and __package__ is None: 8 | sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) 9 | 10 | try: 11 | from krahenbuhl2013.krahenbuhl2013 import DenseCRF 12 | except ImportError: 13 | print("") 14 | print("Error: cannot import 'krahenbuhl2013'.") 15 | print("") 16 | print("This is a custom C++ extension and can be compiled with:") 17 | print("") 18 | print((" cd %s" % os.path.join(os.path.dirname(os.path.abspath(__file__)), 'krahenbuhl2013'))) 19 | print(" make") 20 | sys.exit(1) 21 | 22 | from solver import IntrinsicSolver 23 | from input import IntrinsicInput 24 | from params import IntrinsicParameters 25 | import image_util 26 | 27 | 28 | if __name__ == '__main__': 29 | parser = argparse.ArgumentParser( 30 | description=( 31 | 'Decompose an image using the algorithm presented in:\n' 32 | ' Sean Bell, Kavita Bala, Noah Snavely. "Intrinsic Images in the Wild".\n' 33 | ' ACM Transactions on Graphics (SIGGRAPH 2014).\n' 34 | ' http://intrinsic.cs.cornell.edu.\n' 35 | '\n' 36 | 'The output is rescaled for viewing and encoded as sRGB PNG images' 37 | '(unless --linear is specified).' 38 | ) 39 | ) 40 | 41 | parser.add_argument( 42 | 'input', metavar='', help='Input image') 43 | 44 | parser.add_argument( 45 | '-r', '--reflectance', metavar='', 46 | help='Reflectance layer output name (saved as sRGB image)', required=False) 47 | 48 | parser.add_argument( 49 | '-s', '--shading', metavar='', 50 | help='Shading layer output name (saved as sRGB image)', required=False) 51 | 52 | parser.add_argument( 53 | '-m', '--mask', metavar='', type=str, 54 | help='Mask filename', required=False, default=None) 55 | 56 | parser.add_argument( 57 | '-j', '--judgements', metavar='', 58 | help='Judgements file from the Intrinsic Images in the Wild dataset', required=False, default=None) 59 | 60 | parser.add_argument( 61 | '-p', '--parameters', metavar='', 62 | help='Parameters file (JSON format). See params.py for a list of parameters.', required=False, default=None) 63 | 64 | parser.add_argument( 65 | '-l', '--linear', action='store_true', 66 | help='if specified, assume input is linear and generate linear output, otherwise assume input is sRGB and generate sRGB output', required=False) 67 | 68 | parser.add_argument( 69 | '-q', '--quiet', action='store_true', 70 | help="if specified, don't print logging info", required=False) 71 | 72 | parser.add_argument( 73 | '--show-labels', action='store_true', 74 | help="if specified, also output labels", required=False) 75 | 76 | if len(sys.argv) <= 1: 77 | parser.print_help() 78 | sys.exit(1) 79 | 80 | # obtain arguments 81 | args = parser.parse_args() 82 | image_filename = args.input 83 | base, _ = os.path.splitext(image_filename) 84 | r_filename = args.reflectance if args.reflectance else base + '-r.png' 85 | s_filename = args.shading if args.shading else base + '-s.png' 86 | mask_filename = args.mask 87 | judgements_filename = args.judgements 88 | parameters_filename = args.parameters 89 | sRGB = not args.linear 90 | if not r_filename.endswith('.png'): 91 | r_filename += '.png' 92 | if not s_filename.endswith('.png'): 93 | s_filename += '.png' 94 | 95 | print('Input:') 96 | print((' image_filename:', image_filename)) 97 | print((' mask_filename:', mask_filename)) 98 | print((' judgements_filename:', judgements_filename)) 99 | print((' parameters_filename:', parameters_filename)) 100 | print('Output:') 101 | print((' r_filename:', r_filename)) 102 | print((' s_filename:', s_filename)) 103 | 104 | # load input 105 | input = IntrinsicInput.from_file( 106 | image_filename, 107 | image_is_srgb=sRGB, 108 | mask_filename=mask_filename, 109 | judgements_filename=judgements_filename, 110 | ) 111 | 112 | print(('mask_nnz: %s' % input.mask_nnz)) 113 | print(('rows * cols: %s' % (input.rows * input.cols))) 114 | 115 | # load parameters 116 | if parameters_filename: 117 | params = IntrinsicParameters.from_file(parameters_filename) 118 | else: 119 | params = IntrinsicParameters() 120 | 121 | params.logging = not args.quiet 122 | 123 | # solve 124 | solver = IntrinsicSolver(input, params) 125 | r, s, decomposition = solver.solve() 126 | 127 | # save output 128 | image_util.save(r_filename, r, mask_nz=input.mask_nz, rescale=True, srgb=sRGB) 129 | image_util.save(s_filename, s, mask_nz=input.mask_nz, rescale=True, srgb=sRGB) 130 | if args.show_labels: 131 | labels_vis = decomposition.get_labels_visualization() 132 | r_path, r_ext = os.path.splitext(r_filename) 133 | image_util.save('%s_labels%s' % (r_path, r_ext), labels_vis, mask_nz=solver.input.mask_nz, rescale=True) 134 | 135 | # compute error 136 | if judgements_filename: 137 | print(('WHDR: %.1f%%' % (input.compute_whdr(r) * 100.0))) 138 | -------------------------------------------------------------------------------- /decomposition.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | import image_util 4 | # from numba import vectorize,jitclass,jit 5 | 6 | class IntrinsicDecomposition(object): 7 | """ Current state of a reconstruction. All entries (except ``input``) are 8 | mutable. """ 9 | 10 | def __init__(self, params, input): 11 | self._input = input 12 | self.params = params 13 | 14 | # iteration number 15 | self.iter_num = None 16 | 17 | # stage 1 or 2 (each iteration has 2 stages) 18 | self.stage_num = None 19 | 20 | # labels ("x" variable in the paper), where "_nz" indicates that only the 21 | # nonmasked entries are stored. 22 | self.labels_nz = None 23 | 24 | # reflectance intensity (obtained from kmeans) 25 | self.intensities = None 26 | # reflectance chromaticity (obtained from kmeans) 27 | self.chromaticities = None 28 | 29 | # store here for visualization only 30 | self.shading_target = None 31 | 32 | def copy(self): 33 | ret = IntrinsicDecomposition(self.params, self.input) 34 | ret.iter_num = self.iter_num 35 | ret.stage_num = self.stage_num 36 | ret.labels_nz = self.labels_nz.copy() 37 | ret.intensities = self.intensities.copy() 38 | ret.chromaticities = self.chromaticities.copy() 39 | if self.shading_target is not None: 40 | ret.shading_target = self.shading_target.copy() 41 | return ret 42 | 43 | def get_r_s_nz(self): 44 | """ Return (reflectance, shading), with just the nonmasked entries """ 45 | s_nz = self.input.image_gray_nz / self.intensities[self.labels_nz] 46 | r_nz = self.input.image_rgb_nz / np.clip(s_nz, 1e-4, 1e5)[:, np.newaxis] 47 | assert s_nz.ndim == 1 and r_nz.ndim == 2 and r_nz.shape[1] == 3 48 | return r_nz, s_nz 49 | 50 | def get_r_s(self): 51 | """ Return (reflectance, shading), in the full (rows, cols) shape """ 52 | r_nz, s_nz = self.get_r_s_nz() 53 | r = np.zeros((self.input.rows, self.input.cols, 3), dtype=r_nz.dtype) 54 | s = np.zeros((self.input.rows, self.input.cols), dtype=s_nz.dtype) 55 | r[self.input.mask_nz] = r_nz 56 | s[self.input.mask_nz] = s_nz 57 | assert s.ndim == 2 and r.ndim == 3 and r.shape[2] == 3 58 | return r, s 59 | 60 | def get_r_gray(self): 61 | r_nz = self.intensities[self.labels_nz] 62 | r = np.zeros((self.input.rows, self.input.cols), dtype=r_nz.dtype) 63 | r[self.input.mask_nz] = r_nz 64 | return r 65 | 66 | def get_labels_visualization(self): 67 | #colors = image_util.n_distinct_colors(self.nlabels + 1) 68 | colors = self.get_reflectances_rgb() 69 | colors = np.vstack((colors, [0.0, 0.0, 0.0])) 70 | labels = self.get_labels() 71 | labels[labels == -1] = self.nlabels 72 | v = colors[labels, :] 73 | return v 74 | 75 | def get_reflectances_rgb(self): 76 | nlabels = self.intensities.shape[0] 77 | rgb = np.zeros((nlabels, 3)) 78 | s = 3.0 * self.intensities 79 | r = self.chromaticities[:, 0] 80 | g = self.chromaticities[:, 1] 81 | b = 1.0 - r - g 82 | rgb[:, 0] = s * r 83 | rgb[:, 1] = s * g 84 | rgb[:, 2] = s * b 85 | return rgb 86 | 87 | 88 | @property 89 | def nlabels(self): 90 | return self.intensities.shape[0] 91 | 92 | @property 93 | def input(self): 94 | return self._input 95 | 96 | def get_labels(self): 97 | """ Returns labels, expanded to the full image shape, with masked 98 | entries having a label of -1 """ 99 | labels = np.empty((self.input.rows, self.input.cols), dtype=np.int32) 100 | labels.fill(-1) 101 | labels[self.input.mask_nz] = self.labels_nz 102 | return labels 103 | 104 | def save(self, solver, out_dir, save_extra=False, id=None): 105 | """ Save results to a directory """ 106 | if not id: 107 | id = self.input.id 108 | if not id: 109 | raise ValueError("Need an id for saving") 110 | if not os.path.exists(out_dir): 111 | os.makedirs(out_dir) 112 | basename = os.path.join(out_dir, str(self.input.id)) 113 | 114 | r, s = self.get_r_s() 115 | r_filename = '%s-r.png' % basename 116 | image_util.save(r_filename, r, mask_nz=solver.input.mask_nz, 117 | rescale=True) 118 | 119 | s_filename = '%s-s.png' % basename 120 | image_util.save(s_filename, s, mask_nz=solver.input.mask_nz, 121 | rescale=True) 122 | 123 | if save_extra: 124 | r_gray_filename = '%s-r-gray.png' % basename 125 | r_gray = self.get_r_gray() 126 | image_util.save(r_gray_filename, r_gray, 127 | mask_nz=solver.input.mask_nz, rescale=True) 128 | 129 | labels_filename = '%s-labels.png' % basename 130 | labels_image = self.get_labels_visualization() 131 | image_util.save(labels_filename, labels_image, 132 | mask_nz=solver.input.mask_nz, rescale=True) 133 | -------------------------------------------------------------------------------- /density.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from scipy.ndimage.filters import gaussian_filter 3 | 4 | class ProbDensityHistogram(object): 5 | 6 | def train(self, training_data, bins=100, bandwidth=3, smoothing=1e-2): 7 | self.ndim = training_data.shape[1] 8 | self.bins = bins 9 | self.hist, self.edges = np.histogramdd( 10 | training_data, bins=bins, normed=True) 11 | self.hist[self.hist < smoothing] = smoothing 12 | if bandwidth: 13 | self.hist = gaussian_filter(self.hist, sigma=bandwidth) 14 | #self.hist /= np.sum(self.hist) 15 | self.hist = np.log(self.hist) 16 | 17 | def logprob(self, samples): 18 | indices = [np.digitize(samples[:, i], self.edges[i]) 19 | for i in range(self.ndim)] 20 | for i in range(self.ndim): 21 | np.clip(indices[i], 0, self.bins - 1, out=indices[i]) 22 | ret = self.hist[indices] 23 | assert ret.shape[0] == samples.shape[0] 24 | return ret 25 | -------------------------------------------------------------------------------- /docs/TheFastAndTheCurious_FinalPresentation1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/docs/TheFastAndTheCurious_FinalPresentation1.pdf -------------------------------------------------------------------------------- /docs/TheFastAndTheCurious_FinalPresentation1.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/docs/TheFastAndTheCurious_FinalPresentation1.pptx -------------------------------------------------------------------------------- /docs/proposal.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/docs/proposal.pdf -------------------------------------------------------------------------------- /energy/__init__.py: -------------------------------------------------------------------------------- 1 | from .energy import IntrinsicEnergy 2 | from .prob_abs_r import ProbAbsoluteReflectance 3 | from .prob_abs_s import ProbAbsoluteShading 4 | -------------------------------------------------------------------------------- /energy/energy.py: -------------------------------------------------------------------------------- 1 | import timeit 2 | import numpy as np 3 | 4 | from image_util import gaussian_blur_gray_image_nz 5 | from .prob_abs_r import ProbAbsoluteReflectance 6 | from .prob_abs_s import ProbAbsoluteShading 7 | 8 | 9 | class IntrinsicEnergy(object): 10 | 11 | def __init__(self, input, params): 12 | self.input = input 13 | self.params = params 14 | self.prob_abs_r = ProbAbsoluteReflectance(params) 15 | self.prob_abs_s = ProbAbsoluteShading(params) 16 | 17 | def compute_unary_costs(self, decomposition, prev_decomposition): 18 | """ Returns unary costs: nnz x nlabels matrix """ 19 | 20 | if self.params.logging: 21 | t0 = timeit.default_timer() 22 | print("compute_unary_costs...") 23 | 24 | intensities = decomposition.intensities 25 | chromaticities = decomposition.chromaticities 26 | nlabels = intensities.shape[0] 27 | unary_costs = np.zeros( 28 | (self.input.mask_nnz, nlabels), 29 | dtype=np.float32) 30 | 31 | sigma_spatial = ( 32 | self.params.shading_blur_sigma * 33 | self.input.diag / ( 34 | 1.0 + decomposition.iter_num ** 35 | self.params.shading_blur_iteration_pow 36 | ) 37 | ) 38 | if self.params.logging: 39 | print(('blur sigma: %s pixels (image diagonal: %s pixels)' % 40 | (sigma_spatial, self.input.diag))) 41 | 42 | # obtain previous shading layer, or use a method to create a proxy 43 | if prev_decomposition: 44 | prev_r_nz, prev_s_nz = prev_decomposition.get_r_s_nz() 45 | elif self.params.shading_blur_init_method == "constant": 46 | prev_s_nz = 0.5 * np.ones_like(self.input.image_gray_nz) 47 | elif self.params.shading_blur_init_method == "image": 48 | prev_s_nz = self.input.image_gray_nz 49 | elif self.params.shading_blur_init_method == "none": 50 | prev_s_nz = None 51 | else: 52 | raise ValueError("Unknown shading_blur_init_method: %s" % 53 | self.params.shading_blur_init_method) 54 | 55 | if prev_s_nz is not None: 56 | if self.params.shading_blur_log: 57 | # blur in log space 58 | blur_input = np.log(prev_s_nz) 59 | else: 60 | # blur in linear space, then convert to log 61 | blur_input = prev_s_nz 62 | 63 | blur_output = gaussian_blur_gray_image_nz( 64 | image_nz=blur_input, 65 | image_shape=self.input.shape, 66 | mask_nz=self.input.mask_nz, 67 | sigma=sigma_spatial, 68 | ) 69 | 70 | if self.params.shading_blur_log: 71 | log_s_target_nz = blur_output 72 | else: 73 | log_s_target_nz = np.log(blur_output) 74 | else: 75 | log_s_target_nz = None 76 | 77 | # (used below) 78 | if self.params.shading_target_chromaticity: 79 | labels_rgb = np.clip( 80 | decomposition.get_reflectances_rgb(), 1e-5, np.inf) 81 | 82 | # shading and chromaticity terms 83 | for i in range(nlabels): 84 | s_nz = self.input.image_gray_nz / intensities[i] 85 | r_nz = (self.input.image_rgb_nz / 86 | np.clip(s_nz, 1e-4, 1e5)[:, np.newaxis]) 87 | 88 | # absolute reflectance and shading 89 | unary_costs[:, i] += ( 90 | self.prob_abs_s.cost(s_nz) + 91 | self.prob_abs_r.cost(r_nz) 92 | ) 93 | 94 | # chromaticity: encourage reflectance intensities to be assigned to 95 | # pixels that share the same chromaticity as the original kmeans 96 | # cluster from which the reflectance intensity was obtained. 97 | if self.params.chromaticity_weight: 98 | if self.params.chromaticity_norm == "L1": 99 | f = np.abs 100 | elif self.params.chromaticity_norm == "L2": 101 | f = np.square 102 | else: 103 | raise ValueError( 104 | "Invalid value of chromaticity_norm: %s" % 105 | self.params.chromaticity_norm) 106 | 107 | unary_costs[:, i] += self.params.chromaticity_weight * ( 108 | np.sum( 109 | f(self.input.image_irg_nz[:, 1:3] - 110 | chromaticities[i, :]), 111 | axis=1 112 | ) 113 | ) 114 | 115 | # shading smoothness: discourage shading discontinuities 116 | if self.params.shading_target_weight and log_s_target_nz is not None: 117 | if self.params.shading_target_norm == "L2": 118 | f = np.square 119 | elif self.params.shading_target_norm == "L1": 120 | f = np.abs 121 | else: 122 | raise ValueError("Invalid value of shading_target_norm: %s" % 123 | self.params.shading_target_norm) 124 | 125 | if self.params.shading_target_chromaticity: 126 | # interpret labels as RGB (intensity with chromaticity), 127 | # thereby penalizing deviations from grayscale in the 128 | # shading channel (though the final answer is always 129 | # grayscale anyway) 130 | label_rgb = labels_rgb[i, :] 131 | s_rgb_nz = self.input.image_rgb_nz / label_rgb[np.newaxis, :] 132 | log_s_rgb_nz = np.log(np.clip(s_rgb_nz, 1e-5, np.inf)) 133 | unary_costs[:, i] += ( 134 | self.params.shading_target_weight * 135 | np.sum(f(log_s_rgb_nz - log_s_target_nz[:, np.newaxis]), axis=-1) 136 | ) 137 | else: 138 | # interpret labels as intensities 139 | log_s_nz = np.log(s_nz) 140 | unary_costs[:, i] += ( 141 | self.params.shading_target_weight * 142 | f(log_s_nz - log_s_target_nz) 143 | ) 144 | 145 | if self.params.logging: 146 | t1 = timeit.default_timer() 147 | print(("compute_unary_costs: done (%s s)" % (t1 - t0))) 148 | 149 | return unary_costs 150 | 151 | def compute_pairwise_costs(self, decomposition): 152 | """ Returns the pairwise cost matrix: nlabels x nlabels matrix. 153 | Entry ij is ``abs(intensity[i] - intensity[j])`` """ 154 | 155 | if self.params.pairwise_intensity_chromaticity: 156 | # interpret labels as RGB (intensity with chromaticity) 157 | nlabels = decomposition.intensities.shape[0] 158 | R = decomposition.get_reflectances_rgb() 159 | if self.params.pairwise_intensity_log: 160 | R = np.log(np.clip(R, 1e-5, np.inf)) 161 | binary_costs = np.zeros((nlabels, nlabels), dtype=np.float32) 162 | for i in range(nlabels): 163 | for j in range(i): 164 | cost = np.sum(np.abs(R[i, :] - R[j, :])) 165 | binary_costs[i, j] = cost 166 | binary_costs[j, i] = cost 167 | else: 168 | # interpret labels as intensities 169 | R = decomposition.intensities 170 | if self.params.pairwise_intensity_log: 171 | R = np.log(np.clip(R, 1e-5, np.inf)) 172 | binary_costs = np.abs(R[:, np.newaxis] - R[np.newaxis, :]) 173 | 174 | return binary_costs 175 | 176 | def get_features(self): 177 | """ Return an nnz x nfeatures matrix containing the features """ 178 | 179 | if not hasattr(self, '_features'): 180 | mask_nz = self.input.mask_nz 181 | mask_nnz = self.input.mask_nnz 182 | features = np.zeros((mask_nnz, 5), dtype=np.float32) 183 | 184 | # image intensity 185 | features[:, 0] = ( 186 | self.input.image_irg[mask_nz[0], mask_nz[1], 0] / 187 | self.params.theta_l) 188 | 189 | # image chromaticity 190 | features[:, 1] = ( 191 | self.input.image_irg[mask_nz[0], mask_nz[1], 1] / 192 | self.params.theta_c) 193 | features[:, 2] = ( 194 | self.input.image_irg[mask_nz[0], mask_nz[1], 2] / 195 | self.params.theta_c) 196 | 197 | # pixel location 198 | features[:, 3] = ( 199 | mask_nz[0] / (self.params.theta_p * self.input.diag)) 200 | features[:, 4] = ( 201 | mask_nz[1] / (self.params.theta_p * self.input.diag)) 202 | 203 | self._features = features 204 | self._features.setflags(write=False) 205 | 206 | return self._features 207 | -------------------------------------------------------------------------------- /energy/prob_abs_r.dat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/energy/prob_abs_r.dat -------------------------------------------------------------------------------- /energy/prob_abs_r.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import csv 4 | import gzip 5 | import _pickle as cPickle 6 | import numpy as np 7 | from density import ProbDensityHistogram 8 | 9 | 10 | class ProbAbsoluteReflectance(object): 11 | """ Implements the absolute reflectance term p(R_x). """ 12 | 13 | def __init__(self, params): 14 | self.params = params 15 | self._load() 16 | 17 | def cost(self, r_nz): 18 | if self.params.abs_reflectance_weight: 19 | return self.params.abs_reflectance_weight * \ 20 | (-self.density.logprob(np.log(r_nz))) 21 | else: 22 | return 0 23 | 24 | def _load(self): 25 | if self.params.logging: 26 | print("loading reflectances...") 27 | 28 | cur_dir = os.path.dirname(os.path.abspath(__file__)) 29 | print('We are here = ',cur_dir) 30 | data_filename = os.path.join(cur_dir, 'prob_abs_r.dat') 31 | if not os.path.exists(data_filename): 32 | rows = [] 33 | to_id = {} 34 | with open(os.path.join(cur_dir, 'bsdfs.csv'), 'rb') as csvfile: 35 | first = True 36 | for row in csv.reader(csvfile): 37 | if first: 38 | to_id = {name: i for i, name in enumerate(row)} 39 | first = False 40 | else: 41 | if row[to_id['colored_reflection']] == 'False': 42 | r = float(row[to_id['rho_d_r']]) 43 | g = float(row[to_id['rho_d_g']]) 44 | b = float(row[to_id['rho_d_b']]) 45 | if r > 1e-4 and g > 1e-4 and b > 1e-4: 46 | rows.append([r, g, b]) 47 | data_raw = np.array(rows) 48 | 49 | data = np.clip(np.log(data_raw), np.log(1e-4), 0) 50 | self.density = ProbDensityHistogram() 51 | self.density.train(data, bins=100, bandwidth=3) 52 | 53 | cPickle.dump( 54 | obj=self.density, 55 | file=gzip.open(data_filename, "wb"), 56 | protocol=cPickle.HIGHEST_PROTOCOL 57 | ) 58 | else: 59 | # make sure that 'density' is on the path so that it 60 | # unpickles correclty 61 | path = os.path.dirname(os.path.dirname(os.path.dirname(__file__))) 62 | print('data_filename = ',data_filename) 63 | if path not in sys.path: 64 | sys.path.append(path) 65 | 66 | 67 | self.density = cPickle.load(gzip.open(data_filename, "rb"), encoding='latin1') 68 | 69 | if self.params.logging: 70 | print("loaded reflectances") 71 | -------------------------------------------------------------------------------- /energy/prob_abs_s.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | 3 | 4 | class ProbAbsoluteShading(object): 5 | def __init__(self, params): 6 | self.params = params 7 | 8 | def cost(self, s_nz): 9 | if self.params.abs_shading_weight: 10 | if self.params.abs_shading_log: 11 | return self.params.abs_shading_weight * \ 12 | np.abs(np.log(s_nz) - np.log(self.params.abs_shading_gray_point)) 13 | else: 14 | return self.params.abs_shading_weight * \ 15 | np.abs(s_nz - self.params.abs_shading_gray_point) 16 | else: 17 | return 0 18 | -------------------------------------------------------------------------------- /image_util.py: -------------------------------------------------------------------------------- 1 | import os 2 | import numpy as np 3 | from imageio import imread 4 | from imageio import imwrite as imsave 5 | from scipy.ndimage import gaussian_filter 6 | from skimage.restoration import denoise_bilateral 7 | 8 | 9 | 10 | # relative luminance for sRGB: 11 | RGB_TO_Y = np.array([0.2126, 0.7152, 0.0722]) 12 | 13 | def load(filename, is_srgb=True): 14 | if not filename: 15 | raise ValueError("Empty filename") 16 | image = imread(filename).astype(np.float32) / 255.0 17 | if is_srgb: 18 | return srgb_to_rgb(image) 19 | else: 20 | return image 21 | 22 | def load_mask(filename): 23 | if not filename: 24 | raise ValueError("Empty filename") 25 | image = imread(filename) 26 | if image.ndim == 2: 27 | return (image >= 128) 28 | elif image.ndim == 3: 29 | return (np.mean(image, axis=-1) >= 128) 30 | else: 31 | raise ValueError("Unknown mask format") 32 | 33 | def save(filename, image, mask_nz=None, rescale=False, srgb=True): 34 | """ (Optionally) Remap to sRGB and save """ 35 | dirname = os.path.dirname(filename) 36 | if dirname and not os.path.exists(dirname): 37 | os.makedirs(dirname) 38 | if rescale: 39 | image = rescale_for_display(image, mask_nz=mask_nz) 40 | 41 | if mask_nz is not None: 42 | image2 = np.zeros_like(image) 43 | if srgb: 44 | image2[mask_nz] = rgb_to_srgb(image[mask_nz]) * 255.0 45 | else: 46 | image2[mask_nz] = image[mask_nz] * 255.0 47 | else: 48 | if srgb: 49 | image2 = rgb_to_srgb(image) * 255.0 50 | else: 51 | image2 = image * 255.0 52 | 53 | assert not np.isnan(image2).any() 54 | image2[image2 > 255] = 255 55 | image2[image2 < 0] = 0 56 | imsave(filename, image2.astype(np.uint8)) 57 | 58 | def gray_to_rgb(gray): 59 | rgb = np.zeros((gray.shape[0], gray.shape[1], 3), dtype=gray.dtype) 60 | rgb[:, :, :] = gray[:, :, np.newaxis] 61 | return rgb 62 | 63 | def luminance(image): 64 | """ Returns the luminance image """ 65 | if image.ndim == 2: 66 | return np.dot(RGB_TO_Y, image.T).T 67 | else: 68 | rows, cols, _ = image.shape 69 | image_flat = image.reshape(rows * cols, 3) 70 | Y_flat = np.dot(RGB_TO_Y, image_flat.T).T 71 | return Y_flat.reshape(image.shape[0:2]) 72 | 73 | def rescale_for_display(image, mask_nz=None, percentile=99.9): 74 | """ Rescales an image so that a particular perenctile is mapped to pure 75 | white """ 76 | if mask_nz is not None: 77 | return image / np.percentile(image, percentile) 78 | else: 79 | return image / np.percentile(image[mask_nz], percentile) 80 | 81 | def rgb_to_srgb(rgb): 82 | ret = np.zeros_like(rgb) 83 | idx0 = rgb <= 0.0031308 84 | idx1 = rgb > 0.0031308 85 | ret[idx0] = rgb[idx0] * 12.92 86 | ret[idx1] = np.power(1.055 * rgb[idx1], 1.0 / 2.4) - 0.055 87 | return ret 88 | 89 | def srgb_to_rgb(srgb): 90 | ret = np.zeros_like(srgb) 91 | idx0 = srgb <= 0.04045 92 | idx1 = srgb > 0.04045 93 | ret[idx0] = srgb[idx0] / 12.92 94 | ret[idx1] = np.power((srgb[idx1] + 0.055) / 1.055, 2.4) 95 | return ret 96 | 97 | def rgb_to_irg(rgb): 98 | """ converts rgb to (mean of channels, red chromaticity, green chromaticity) """ 99 | irg = np.zeros_like(rgb) 100 | s = np.sum(rgb, axis=-1) 101 | irg[..., 0] = s / 3.0 102 | irg[..., 1] = rgb[..., 0] / s 103 | irg[..., 2] = rgb[..., 1] / s 104 | return irg 105 | 106 | def irg_to_rgb(irg): 107 | """ converts (mean of channels, red chromaticity, green chromaticity) to rgb """ 108 | rgb = np.zeros_like(irg) 109 | s = irg[..., 0] * 3.0 110 | rgb[..., 0] = irg[..., 1] * s 111 | rgb[..., 1] = irg[..., 2] * s 112 | rgb[..., 2] = (1.0 - irg[..., 1] - irg[..., 2]) * s 113 | 114 | #np.testing.assert_array_almost_equal( 115 | #irg, rgb_to_irg(rgb)) 116 | 117 | return rgb 118 | 119 | def gaussian_blur_gray_image_nz(image_nz, image_shape, mask_nz, sigma): 120 | """ Blur a masked grayscale image """ 121 | 122 | # deal with the mask -- set the unmasked entries to the average 123 | orig_mean = np.mean(image_nz) 124 | image = np.empty(image_shape[0:2]) 125 | image.fill(orig_mean) 126 | image[mask_nz] = image_nz 127 | 128 | blurred = gaussian_filter(image, sigma=sigma) 129 | blurred_nz = blurred[mask_nz] 130 | 131 | # adjust to keep the mean the same 132 | new_mean = np.mean(blurred_nz) 133 | blurred_nz *= orig_mean / new_mean 134 | return blurred_nz 135 | 136 | def bilateral_blur_gray_image_nz(image_nz, image_shape, mask_nz, sigma_range, sigma_spatial): 137 | """ Blur a masked grayscale image """ 138 | 139 | # deal with the mask -- set the unmasked entries to the average 140 | orig_mean = np.mean(image_nz) 141 | image = np.empty(image_shape[0:2]) 142 | image.fill(orig_mean) 143 | image[mask_nz] = image_nz 144 | 145 | blurred = denoise_bilateral( 146 | image, 147 | sigma_range=sigma_range, 148 | sigma_spatial=sigma_spatial, 149 | win_size=max(int(sigma_spatial * 2), 3), 150 | mode='reflect', 151 | ) 152 | blurred_nz = blurred[mask_nz] 153 | 154 | # adjust to keep the mean the same 155 | new_mean = np.mean(blurred_nz) 156 | blurred_nz *= orig_mean / new_mean 157 | return blurred_nz 158 | 159 | def n_distinct_colors(n): 160 | # return choices from well-distributed precomputed colors 161 | if n <= DISTINCT_COLORS.shape[0]: 162 | return DISTINCT_COLORS[:n, :] 163 | 164 | # add random colors for remaining 165 | return np.vstack([ 166 | DISTINCT_COLORS, 167 | np.random.rand(n - DISTINCT_COLORS.shape[0], 3) 168 | ]) 169 | 170 | # start off with some distinct colors 171 | DISTINCT_COLORS = np.array( 172 | [(int(s[0:2], 16), int(s[2:4], 16), int(s[4:6], 16)) for s in [ 173 | "ff0000", "ff6c0f", "9d9158", "345f1d", "3b7e64", "aeefff", "57adff", 174 | "00093b", "ba62ff", "520047", "ff84ba", "ff8484", "ff8031", "fff4b5", 175 | "050f00", "00ad6b", "00aad6", "8cc7ff", "00062b", "6400b0", "46003c", 176 | "f7529a", "ff8f8e", "ffb78b", "fbf5cd", "acff81", "002b1a", "008fb4", 177 | "6db6f9", "6276ff", "520090", "3e0536", "f30069", "ee0000", "fa5e00", 178 | "d9d3ac", "9fe47b", "006f46", "0081a0", "0076e6", "001bc4", "3c006a", 179 | "f344d9", "c2427a", "ee0000", "e5a277", "a89000", "7dc25c", "00462c", 180 | "367b8c", "6595c1", "0016a2", "e2b9ff", "932283", "960042", "ed0000", 181 | "c28259", "a59024", "5da13e", "1a6047", "004151", "0056a9", "000e6f", 182 | "8b00e7", "ff1cd9", "813f5c", "d9a3a3", "c34d00", "867200", "1e6200", 183 | "78e5c1", "00222b", "003769", "b4bdff", "9d03ff", "ffa7f1", "5b0027", 184 | "d30000", "9f643c", "807222", "0d2900", "00cf89", "daf7ff", "00284c", 185 | "2f41d2", "61009d", "ff68e5", "e66399", "d10000", "ffc296", "7d723b", 186 | "93ff65", "003523", "00c8ff", "0081ff", "dbdfff", "30004d", "b546a2", 187 | "b2517a", "d20000", "ff7006", "ffe225", "89e861", "00f6a9", "64deff", 188 | "86b5e3", "000d82", "bf54ff", "36002c", "b1004b", "b90000", "8f3c00", 189 | "ffeb6b", "288400", "55c3a1", "79bacd", "005ebd", "5a65df", "f3ddff", 190 | "ffb3f1", "a25d7a", "b78383", "7e4721", "ccb000", "3c811e", "009162", 191 | "589aac", "004588", "000d95", "d78dff", "df00b7", "ff4691", "a00000", 192 | "652a00", "b7b18c", "48f106", "006746", "006681", "002f5d", "2c3276", 193 | "d07aff", "d867c3", "ff3886", "920000", "3b1900", "97926d", "66c640", 194 | "4dffca", "003646", "0077f4", "00085d", "ac26f5", "ba0097", "e59bb9", 195 | "860000", "fd9749", "5e551f", "75ff41", "9affe3", "9adcef", "0064d1", 196 | "2a37e7", "a73ce0", "5b2452", "d57099", "790000", "ffb980", "fff39e", 197 | "6eed42", "2ea281", "00b5eb", "0055af", "9095ff", "8800d2", "fb87e5", 198 | "c37b99", "6e0000", "ffae69", "ffea4d", "42a51d", "008963", "a5b3b8", 199 | "004c9c", "1e2bfc", "51007d", "a30083", "be004c", "620000", "ff8c2e", 200 | "787350", "042700", "000d09", "0089b5", "003976", "00056f", "3a0058", 201 | "960077", "a30043", "5b0000", "d8782a", "605500", "89ff81", "00b080", 202 | "007ca1", "00244c", "7e83ff", "ca5fff", "7a406f", "cc004d", "540000", 203 | "602b00", "fff385", "a2ff9b", "003024", "859397", "00152b", "4c4d95", 204 | "c26bed", "ffeffb", "660027", "4e0000", "381900", "5a5634", "011400", 205 | "00251b", "007294", "000e1c", "a2a2ff", "a04bcb", "ffbdf0", "4e001e", 206 | "440000", "ffa151", "403a02", "7fe97b", "43eac1", "006281", "004089", 207 | "5d5df5", "8411be", "bb7cad", "e70055", "3a0000", "ffe5ce", "fff26a", 208 | "69ff64", "00d69f", "677478", "002958", "0000b6", "7e2aaa", "9a5e8d", 209 | "d62868", "350000", "efcbac", "3d3a1a", "b7ffb4", "00a681", "004c62", 210 | "001b3b", "000082", "c441ff", "800064", "800030", "2f0000", "ea6c00", 211 | "292500", "64ed60", "008263", "4a575b", "5da7ff", "000082", "7700b1", 212 | "720059", "ff98bb", "280000", "ccaa8d", "242000", "07ce00", "00ffca", 213 | "2f3b3f", "0070f4", "00005e", "783896", "ff9ae8", "ff75a5", "240000", 214 | "ab8b6e", "fff21c", "95e493", "00cca0", "002935", "0058c3", "00003c", 215 | "5d0089", "dd9cce", "ff86b0", "1d0000", "ffb672", "090800", "5cc65b", 216 | "6effe2", "162125", "004eaf", "00002c", "591976", "d600a3", "ffa5c5", 217 | "180000", "e9a160", "fef969", "00f800", "00efc0", "00c1ff", "00469c", 218 | "6c6ab6", "b100ff", "c631a3", "f60056", "ffb0ae", "c58242", "fff94a", 219 | "74c274", "004c3e", "e7f6fb", "00387c", "2e2b89", "dd95fa", "281c25", 220 | "a31a4a", "fcc4c3", "8a6c51", "ddd71e", "00b100", "00221b", "00afeb", 221 | "003069", "1e1d35", "ba75d8", "26001d", "8c0030", "ff4038", "6b4f35", 222 | "dad747", "00b100", "008f77", "c5d4d9", "d4e6ff", "c6c3ff", "9956b6", 223 | "ff75dd", "730c2f", "ffe6e5", "ff8202", "b9b600", "00af00", "006c5a", 224 | "00a1d6", "006dff", "5e55ff", "1f002c", "ff12c4", "720028", "ff4e47", 225 | "ffdbb7", "f9fa84", "00ac00", "002b24", "002c3b", "a6cbff", "8c88d7", 226 | "532664", "5f515b", "58001f", "f3605c", "ffd0a1", "b7b622", "008f00", 227 | "00d7b5", "72d9ff", "e7f1ff", "04004c", "350947", "5f0047", "ffb9d0", 228 | "ffa8a4", "ff972c", "969600", "008a00", "00b195", "00bbff", "78b1ff", 229 | "bab5ff", "eab1ff", "51003d", "20000b", "ffd1cf", "ffc48a", "d6d864", 230 | "008800", "00d1b6", "4fcfff", "91c0ff", "847bff", "de83ff", "43353f", 231 | "fb5186", "a9150f", "b45900", "b3b644", "008800", "008a77", "009cd6", 232 | "5191ec", "ada8fa", "d04fff", "ea57c3", "b30039", "68201d", "a26325", 233 | "747700", "006a00", "00675a", "0095c9", "0053c3", "1200d9", "9c00d3", 234 | "decdd9", "990031", "ff968d", "4d341b", "565a00", "006c00", "00ffe1", 235 | "005e81", "003a8f", "383651", "74009d", "bcacb8", "7d0029", "ff8179", 236 | "3a1d00", "3c3e00", "006a00", "00ab96", "004863", "00347c", "02001d", 237 | "714282", "b20083", "ff6090", "ff7164", "341a00", "919622", "006800", 238 | "00463e", "00a7ec", "002e6f", "1100a3", "4e006a", "9c8c98", "ff709b", 239 | "ff5d4f", "190d00", "efff0f", "004e00", "60e6d8", "389bc1", "002558", 240 | "2e219c", "d39de4", "7d6e79", "ff447c", "de2312", "ffc27f", "f3fa9d", 241 | "004700", "8ae1d8", "0082b5", "031e46", "54526e", "b17ec2", "6a1152", 242 | "ff3672", "8a3d36", "ffb868", "cadd00", "003400", "00c9b6", "006a94", 243 | "5fa0ff", "5e4aff", "915fa2", "e600a3", "c1003a", "ffa399", "ffad4f", 244 | "d0d87d", "002c00", "35c4b7", "003246", "0065fb", "918dad", "2e003c", 245 | "8e0065", "a50032", "ac5b52", "ff8c00", "202400", "002300", "69bfb7", 246 | "002635", "005de6", "5146a9", "c000f6", "4b0036", "460216", "9a2f24", 247 | "ffa639", "ecff48", "002000", "002824", "001e2b", "0053d2", "726f8d", 248 | "61007d", "ff89dc", "ea0042", "260400", "f5cb96", "aeb75e", "001c00", 249 | "00ebd7", "00141c", "003d9c", "a99bff", "e577ff", "ff66d0", "ce003b", 250 | "0c0100", "eca148", "8d973f", "001c00", "00a396", "00b4ff", "295495", 251 | "d2cef0", "d63aff", "fb40c4", "c74067", "ff3015", "db7700", "6d7821", 252 | "001700", "479e97", "60bbe3", "b3d0ff", "b1adce", "ba1ee0", "40002d", 253 | "520017", "cf3e2a", "814600", "a4bc00", "001600", "008378", "008ec9", 254 | "276ddf", "8873ff", "9700be", "ff04af", "ff83a4", "ce796f", "371e00", 255 | "4f5a00", "001800", "001e1b", "005275", "004bc4", "7363ca", "6f008a", 256 | "c00084", "f80043", "be4e3e", "311b00", "d8ff06", "001800", "00fced", 257 | "004563", "0045b0", "2b13b0", "3d004d", "8b336f", "2b000b", "ab1400", 258 | "ff9200", "e7ff68", "001000", "227f78", "d6f2ff", "003281", "9481ed", 259 | "de4bff", "780052", "ffb1c4", "ff9483", "ffb657", "c4dd46", "000b00", 260 | "00605a", "b6e8ff", "001d4c", "2300c4", "2f1536", "ac528e", "ffdce5", 261 | "ff4d30", "d2aa76", "f2f5e4", "000200", "003834", "54c9ff", "00173b", 262 | "553ebd", "f1a7ff", "a90070", "eb6285", "ff6d56", "c78128", "d0d3c2", 263 | "3aca3f", "85fffa", "00aeff", "e0ecff", "8b69ff", "b338cb", "6b0048", 264 | "d92654", "e36e5b", "b08a58", "afb2a2", "54a156", "acfffa", "00a3ec", 265 | "0056e6", "eae4ff", "694a6f", "ffbae7", "95304a", "ffbaae", "593200", 266 | "7f9c00", "c9ffcc", "ccfdfb", "0096de", "004ed8", "785be0", "4a0058", 267 | "ff99db", "5d0017", "ff8e79", "8f6c3c", "8f9282", "39f140", "aadbd8", 268 | "007db5", "003ba2", "1f008f", "4b2f51", "dc008f", "ff96af", "ff836d", 269 | "341e00", "717464", "37a53d", "00d8cb", "002f46", "00276a", "b6a1ff", 270 | "d75cee", "ce72ae", "680018", "f55f46", "ffad2f", "535648", "a7dfab", 271 | "8ab9b7", "7dd3ff", "00235d", "5733d2", "c8a5ce", "9c0065", "34000c", 272 | "e02100", "ffc167", "1f2600", "87be8b", "6b9997", "00a9ff", "00030a", 273 | "0e003c", "a785ad", "9a1f70", "b75066", "210500", "ffb84d", "b3e200", 274 | "689d6c", "00928b", "cbedff", "0059fc", "9b79ff", "9104aa", "0f000a", 275 | "b50026", "ff8163", "ffcf89", "323e00", "338139", "00d1cc", "0087c9", 276 | "4d71b5", "af92ff", "85009d", "ff4abb", "9a0021", "ff3000", "eea029", 277 | "383a2d", "497e4f", "00b2ab", "0070a8", "002d81", "cebbff", "87678d", 278 | "eb0090", "8e0020", "ffc5b9", "a46300", "ecfbb5", "2b5f33", "4c7a78", 279 | "006395", "000f2b", "7d51f5", "ee80ff", "cc2e8f", "7f001a", "4a0d00", 280 | "6e4f20", "cad994", "008b1b", "2f5c5b", "004d75", "80abff", "5723e7", 281 | "ea68ff", "bc448e", "740019", "ff5d36", "ffdaa0", "a8b775", "7cff9a", 282 | "002b2a", "004263", "004ee7", "231747", "eac6f0", "b70070", "ff5071", 283 | "ff714d", "d6a95f", "617c00", "59ff80", "00fff9", "00a4ff", "0043c4", 284 | "e2d7ff", "5d006b", "860053", "ff0034", "ffaf98", "533300", "1e2114", 285 | "006e19", "00f0ee", "008fde", "003eb6", "1b005e", "a800be", "570037", 286 | "ff345e", "1b0600", "ffe4b7", "d4ff46", "08631c", "006e6d", "002d46", 287 | "003394", "f4f0ff", "6a0076", "ff86cf", "ff7390", "ff9575", "ffa000", 288 | "e2ff83", "00f83e", "0f403f", "002235", "00226a", "25007c", "ac48b6", 289 | "ffa6db", "ffa9ba", "ffa88c", "ffb535", "889757", "55ed7a", "002525", 290 | "91d7ff", "001f5d", "403063", "8a2696", "ff73c7", "eb002e", "ffb7a3", 291 | "ffedcd", "beff00", "72e993", "62ffff", "007eca", "0052fc", "0d002c", 292 | "1a001d", "ff009b", "db6f84", "9b2e0b", "c98100", "bfde63", "95ffb3", 293 | "bcfeff", "006ba8", "6f90d7", "8045ff", "f189fa", "e065ae", "cf0029", 294 | "421000", "b48a41", "90c100", "00d53d", "00faff", "005f95", "0032a2", 295 | "5501fc", "cd00e1", "770048", "c20028", "ff3e00", "301f00", "9dbc43", 296 | "d9fce3", "00c9cc", "003757", "00194c", "4800d9", "ce69d8", "4b002d", 297 | "a70022", "f75f2e", "ffc14c", "7c9c20", "b7dac2", "00a8ab", "0095f3", 298 | "598bff", "7c68a1", "7f008a", "30001d", "66212e", "e7ccc3", "fec966", 299 | "69783a", "4ec773", "9afbff", "0089de", "91aff9", "5d4b82", "f22cff", 300 | "ff3daf", "3d000c", "e56d45", "926b24", "4b5b1e", "97b9a1", "00d9e1", 301 | "0074bc", "0044e7", "260d58", "a200aa", "ffc7e6", "ff90a4", "e0a28d", 302 | "f1a000", "cfff67", "789882", "00878c", "1378b5", "002882", "b389ff", 303 | "c614cb", "ff64bb", "ff5f7b", "d2795a", "daa946", "9ae800", "004b18", 304 | "004d50", "005388", "aec5ff", "a170ff", "4a004d", "c50071", "46000d", 305 | "d03d0e", "ffac00", "6aa000", "001908", "00292a", "003d63", "0048fc", 306 | "bca6e4", "3a003c", "930053", "ffc6d0", "c5aba2", "ffda88", "3f5f00", 307 | "23cb5b", "00686d", "001a2b", "0031b0", "9b87c2", "2b002c", "ff99d2", 308 | "ff859a", "c04e28", "b68926", "ace344", "00b23a", "38f3ff", "0079ca", 309 | "263763", "be99ff", "fc63ff", "e1007c", "5a000f", "bd836e", "724e00", 310 | "b9ff44", "5a7964", "7df7ff", "0065a8", "011958", "3b00a3", "ea47ee", 311 | "630037", "50000e", "af5a3d", "322200", "a1ff00", "3d5b47", "00b3c0", 312 | "005a95", "00113b", "452876", "631564", "ffb2da", "ff6f85", "a48b83", 313 | "ffc231", "dbff9c", "003512", "003135", "004675", "0039d8", "f2eaff", 314 | "580059", "ff0087", "db2541", "9b6451", "ffc94a", "b9de7c", "00a93c", 315 | "00cfe2", "002d4b", "0035c4", "8135ff", "ff9eff", "ad005d", "ca3f53", 316 | "8c3d22", "ffe39f", "98bd5d", "004619", "209fac", "00233b", "2e4fa9", 317 | "4b00c4", "ffe9ff", "55002d", "863e49", "856d65", "ffb700", "779c3e", 318 | "89e5aa", "00939f", "0097ff", "002ca2", "644495", "ffb6ff", "430326", 319 | "ffbbc4", "7b4736", "dda927", "587d20", "223f2d", "00808c", "85cdff", 320 | "002894", "491d89", "ffcbff", "ff48a6", "ff97a4", "665048", "ffd96e", 321 | "274200", "aaffcb", "00191c", "006fbc", "001e6f", "1f004d", "ff88ff", 322 | "ff87c6", "a75c66", "5b2c1c", "ba8900", "c8ff82", "00f95e", "00e6ff", 323 | "004f88", "5e85ff", "dec7ff", "ffd5ff", "ef007c", "ffa4af", "49342d", 324 | "946b00", "172900", "67c38b", "4ec0cd", "1f5981", "87a4ff", "a665ff", 325 | "ffe1ff", "a00054", "ef6170", "ffdcce", "ffc000", "a6e362", "003013", 326 | "00adc0", "1e3c51", "002182", "a57fd7", "ffdbff", "7d0041", "ec9aa3", 327 | "ffc3ad", "ffecb6", "63a11f", "008b37", "a5f5ff", "008fff", "00134c", 328 | "8461b6", "ffbeff", "6f0e40", "c97a84", "ffa282", "e0a800", "458100", 329 | "00d559", "87f1ff", "50b2ff", "003cfc", "2e0070", "ff70ff", "3a001e", 330 | "a80013", "ff8254", "4c3900", "7bec00", "46a26c", "008ea0", "0076d1", 331 | "002bb6", "c69ffa", "ff93ff", "361325", "ff828f", "ff946c", "2e2200", 332 | "84c142", "072514", "007180", "0061a9", "455381", "4c00b0", "833582", 333 | "ff76ba", "fa0019", "2d1b15", "ffe387", "132400", "001308", "00616e", 334 | "004375", "00165d", "6a3aa9", "6b006b", "bb005e", "c30014", "ff804c", 335 | "ffce2e", "b3ff66", "3fff9a", "004750", "3b586e", "0030d8", "3d008f", 336 | "ffadfe", "9f1c5d", "9b0010", "d13d00", "ffd951", "7dff00", "00b253", 337 | "002f35", "003b69", "002bc4", "d5b3ff", "ff53fc", "6f0037", "8f000e", 338 | "9d2e00", "ffc900", "e3fbcc", "21824f", "00262b", "002b4c", "546bca", 339 | "ad77ed", "a354a2", "ff3a9b", "ff4151", "6d1f00", "282000", "c2d9ab", 340 | "006b32", "00dcff", "bde1ff", "0020a2", "8b58cb", "790077", "d22b7b", 341 | "ff4e5d", "3d1200", "ffec9e", "6dc61a", "69ffb3", "c9f8ff", "007bdf", 342 | "6370a1", "6e2ebe", "ff7bfb", "532d40", "83000c", "ff7033", "ffe26d", 343 | "a1b88b", "3cee92", "00c6e2", "4796d7", "001e95", "4b089d", "e893e4", 344 | "4f0026", "982f37", "ff4c00", "e6d17e", "81986d", "00ff7f", "00b6d5", 345 | "0066bd", "001982", "7500fc", "c573c3", "c9005f", "3a1215", "ff5c10", 346 | "483900", "9bff43", "00a955", "00a5c1", "005ca9", "617eff", "9640f5", 347 | "9b0096", "8a0041", "ff727a", "ffb997", "110e00", "90e843", "002814", 348 | "0088a0", "4476a1", "c4cfff", "914ee0", "8e008a", "71495c", "ff5e66", 349 | "ffeee4", "eed149", "637950", "006333", "004450", "59758d", "838ec2", 350 | "721bd2", "d700cc", "622240", "dd232c", "ffd1b8", "ebd165", "465b34", 351 | "60eaaa", "51e3ff", "004b88", "00146f", "b46dff", "be34b7", "ff97c5", 352 | "ff9599", "fa9861", "e0d295", "49a500", "00904d", "00b1d6", "003157", 353 | "a3aee3", "a958ff", "b200aa", "ffbcda", "bb4f53", "e86d2c", "c7b045", 354 | "2a3f1a", "00d272", "00a0c1", "a7d6ff", "0022d9", "7400e7", "fc24ee", 355 | "ffabd1", "582d2e", "431700", "c3b05e", "d3ffb4", "85ffcb", "006b80", 356 | "0082f4", "3048bd", "3e007d", "e159d8", "f4c5da", "ffb8ba", "ffae81", 357 | "beb176", "1c4400", "9be0c2", "0c5d6e", "b8d5ef", "00105d", "9a2eff", 358 | "ff8bf2", "e40068", "df6e70", "ffc4a2", "675400", "b1df94", "7abea1", 359 | "002c35", "006ed1", "000c4c", "6500c5", "cf07b7", "d2a4b8", "cd3e3f", 360 | "ffa774", "ffe24f", "4eca00", "00894e", "00d1ff", "97b4cd", "00051c", 361 | "30005e", "ab0096", "b08498", "774849", "ff8d4a", "ffeb86", "90bd74", 362 | "004d2b", "8eeaff", "7894ac", "9aa9ff", "cf97ff", "880077", "90325c", 363 | "ff6e6e", "d57844", "f1d022", "719d56", "bcffe3", "c0f4ff", "00529c", 364 | "8c9dff", "20003c", "720064", "906679", "976565", "b15a27", "cab024", 365 | "c0ff9b", "37c78a", "0094b4", "003f76", "788aec", "dbaaff", "650059", 366 | "7b0038", "140700", "a19140", "527d39", "5b9e82", "005162", "001f3b", 367 | "001cb6", "cb86ff", "ff9af2", "44001e", 368 | ]], dtype=np.float32 369 | ) 370 | DISTINCT_COLORS = DISTINCT_COLORS / 255.0 371 | DISTINCT_COLORS.setflags(write=False) 372 | -------------------------------------------------------------------------------- /input.py: -------------------------------------------------------------------------------- 1 | import os 2 | import math 3 | import numpy as np 4 | from skimage.color import rgb2lab 5 | from skimage.transform import resize 6 | import image_util 7 | from judgements import HumanReflectanceJudgements 8 | 9 | class IntrinsicInput(object): 10 | """ Input to a decomposition. All properties are read-only. """ 11 | 12 | def __init__(self, image_rgb, mask=None, r_gt=None, s_gt=None, 13 | judgements=None, dataset=None, id=None): 14 | 15 | self._dataset = dataset 16 | self._id = id 17 | 18 | # convert to color 19 | if image_rgb.ndim == 2: 20 | self._image_rgb = np.zeros((image_rgb.shape[0], image_rgb.shape[1], 3)) 21 | self._image_rgb[:, :, :] = image_rgb[:, :, np.newaxis] 22 | elif image_rgb.ndim == 3: 23 | self._image_rgb = image_rgb.copy() 24 | else: 25 | raise ValueError("Invalid image") 26 | 27 | # drop alpha channel 28 | if self._image_rgb.shape[2] == 4: 29 | self._image_rgb = self._image_rgb[:, :, 0:3] 30 | elif self._image_rgb.shape[2] != 3: 31 | raise ValueError("Invalid image") 32 | 33 | # Clip to 1e-4 since a sRGB value of 1/255 is (1/255)/12.92 ~= 3e-4 34 | # in linear RGB space. This avoids a huge log-space jump between 35 | # intensity 0/255 and intensity 1/255. 36 | self._image_rgb[self._image_rgb < 1e-4] = 1e-4 37 | self._image_rgb.setflags(write=False) 38 | 39 | # load binary image mask 40 | if mask is not None: 41 | if mask.ndim != 2: 42 | raise ValueError("Invalid mask") 43 | self._mask = mask.astype(np.bool, copy=True) 44 | else: 45 | self._mask = np.ones((self.rows, self.cols), dtype=bool) 46 | self._mask.setflags(write=False) 47 | 48 | if self.mask.shape != self.image_rgb.shape[0:2]: 49 | raise ValueError("Shape of Mask and Image is unequal") 50 | 51 | # load ground truth 52 | if r_gt is not None: 53 | self._r_gt = r_gt.copy() 54 | self._r_gt.setflags(write=False) 55 | else: 56 | self._r_gt = None 57 | 58 | if s_gt is not None: 59 | self._s_gt = s_gt.copy() 60 | self._s_gt.setflags(write=False) 61 | else: 62 | self._s_gt = None 63 | 64 | # human judgements 65 | self._judgements = judgements 66 | 67 | @staticmethod 68 | def from_dataset(dataset, in_dir, id): 69 | """ Load input from a dataset """ 70 | 71 | if dataset == "mit": 72 | return IntrinsicInput.from_file( 73 | image_filename=os.path.join(in_dir, id, 'diffuse.png'), 74 | image_is_srgb=False, 75 | mask_filename=os.path.join(in_dir, id, 'mask.png'), 76 | r_gt_filename=os.path.join(in_dir, id, 'reflectance.png'), 77 | s_gt_filename=os.path.join(in_dir, id, 'shading.png'), 78 | gt_is_srgb=False, 79 | dataset=dataset, 80 | id=id, 81 | ) 82 | elif dataset == "iiw": 83 | return IntrinsicInput.from_file( 84 | image_filename=os.path.join(in_dir, '%s.png' % id), 85 | image_is_srgb=True, 86 | judgements_filename=os.path.join(in_dir, '%s.json' % id), 87 | dataset=dataset, 88 | id=id, 89 | ) 90 | else: 91 | raise ValueError("Unknown dataset") 92 | 93 | @staticmethod 94 | def from_file(image_filename, image_is_srgb=True, mask_filename=None, 95 | r_gt_filename=None, s_gt_filename=None, gt_is_srgb=False, 96 | judgements_filename=None, dataset=None, id=None): 97 | """ Load input from files """ 98 | 99 | image_rgb = image_util.load(image_filename, is_srgb=image_is_srgb) 100 | 101 | if mask_filename: 102 | mask = image_util.load_mask(mask_filename) 103 | else: 104 | mask = None 105 | 106 | if r_gt_filename: 107 | r_gt = image_util.load(r_gt_filename, is_srgb=gt_is_srgb) 108 | else: 109 | r_gt = None 110 | 111 | if s_gt_filename: 112 | s_gt = image_util.load(s_gt_filename, is_srgb=gt_is_srgb) 113 | else: 114 | s_gt = None 115 | 116 | if judgements_filename: 117 | judgements = HumanReflectanceJudgements.from_file(judgements_filename) 118 | else: 119 | judgements = None 120 | 121 | return IntrinsicInput( 122 | image_rgb=image_rgb, mask=mask, 123 | r_gt=r_gt, s_gt=s_gt, 124 | judgements=judgements, 125 | dataset=dataset, id=id, 126 | ) 127 | 128 | def downsample(self, factor=2.0): 129 | """ Return a new input at a lower resolution (while keeping ground 130 | truth (gt) resolution the same). """ 131 | 132 | if factor <= 1.0: 133 | return self 134 | 135 | rows = int(self.rows / factor) 136 | cols = int(self.cols / factor) 137 | image_rgb = resize(self.image_rgb, (rows, cols, 3), mode='reflect') 138 | 139 | if self.mask_nnz == self.rows * self.cols: 140 | mask = None 141 | else: 142 | mask = resize(self.mask.astype(float), (rows, cols), mode='reflect') 143 | 144 | return IntrinsicInput(image_rgb, mask, self.r_gt, self.s_gt) 145 | 146 | ## IMAGE TAG ## 147 | 148 | @property 149 | def id(self): 150 | return self._id 151 | 152 | @property 153 | def dataset(self): 154 | return self._dataset 155 | 156 | ## IMAGE MASK ## 157 | 158 | @property 159 | def mask(self): 160 | return self._mask 161 | 162 | @property 163 | def mask_nz(self): 164 | if not hasattr(self, '_mask_nz'): 165 | self._mask_nz = np.nonzero(self.mask) 166 | return self._mask_nz 167 | 168 | @property 169 | def mask_nnz(self): 170 | """ Shorthand for the number of nonzero entries """ 171 | return self.mask_nz[0].size 172 | 173 | ## IMAGE SIZE ## 174 | 175 | @property 176 | def shape(self): 177 | return self._image_rgb.shape 178 | 179 | @property 180 | def rows(self): 181 | return self.shape[0] 182 | 183 | @property 184 | def cols(self): 185 | return self.shape[1] 186 | 187 | @property 188 | def diag(self): 189 | """ diagonal of the inner bounding box of nonzero mask pixels """ 190 | if not hasattr(self, '_diag'): 191 | if self.mask_nz: 192 | self._diag = math.sqrt(np.sum([ 193 | (np.max(nz) - np.min(nz)) ** 2 194 | for nz in self.mask_nz 195 | ])) 196 | else: 197 | self._diag = math.sqrt(self.rows ** 2 + self.cols ** 2) 198 | return self._diag 199 | 200 | ## IMAGE COLORSPACES ## 201 | 202 | @property 203 | def image_rgb(self): 204 | """ Image in linear RGB space """ 205 | return self._image_rgb 206 | 207 | @property 208 | def image_rgb_nz(self): 209 | """ Image linear RGB space with only unmasked (_nz = "nonzero mask") 210 | entries """ 211 | return self._image_rgb[self.mask_nz] 212 | 213 | @property 214 | def image_gray(self): 215 | """ Image in grayscale space with only unmasked (_nz = "nonzero 216 | mask") entries """ 217 | if not hasattr(self, '_image_gray'): 218 | self._image_gray = np.mean(self._image_rgb, axis=2) 219 | self._image_gray.setflags(write=False) 220 | return self._image_gray 221 | 222 | @property 223 | def log_image_gray(self): 224 | """ Image in log-grayscale space """ 225 | if not hasattr(self, '_log_image_gray'): 226 | # no clip necessary since we clip at load time 227 | self._log_image_gray = np.log(self.image_gray) 228 | self._log_image_gray.setflags(write=False) 229 | return self._log_image_gray 230 | 231 | @property 232 | def log_image_rgb(self): 233 | """ Image in log(linear RGB) space """ 234 | if not hasattr(self, '_log_image_rgb'): 235 | # no clip necessary since we clip at load time 236 | self._log_image_rgb = np.log(self.image_rgb) 237 | self._log_image_rgb.setflags(write=False) 238 | return self._log_image_rgb 239 | 240 | @property 241 | def image_gray_nz(self): 242 | """ Image in grayscale space with only unmasked (_nz = "nonzero 243 | mask") entries """ 244 | return self.image_gray[self.mask_nz] 245 | 246 | @property 247 | def image_irg(self): 248 | """ Image in 'irg' space (intensity, red chromaticity, green 249 | chromaticity) """ 250 | 251 | if not hasattr(self, '_image_irg'): 252 | self._image_irg = image_util.rgb_to_irg(self._image_rgb) 253 | self._image_irg.setflags(write=False) 254 | return self._image_irg 255 | 256 | @property 257 | def image_irg_nz(self): 258 | """ Image in 'irg' space with only unmasked (_nz = "nonzero 259 | mask") entries """ 260 | 261 | return self.image_irg[self.mask_nz] 262 | 263 | @property 264 | def image_lab(self): 265 | """ Image in L*a*b* space """ 266 | if not hasattr(self, '_image_lab'): 267 | self._image_lab = rgb2lab(self._image_rgb) 268 | self._image_lab.setflags(write=False) 269 | return self._image_lab 270 | 271 | def image(self, colorspace='rgb'): 272 | return getattr(self, 'image_%s' % colorspace) 273 | 274 | ## GROUND TRUTH ## 275 | 276 | @property 277 | def judgements(self): 278 | return self._judgements 279 | 280 | @property 281 | def r_gt(self): 282 | return self._r_gt 283 | 284 | @property 285 | def s_gt(self): 286 | return self._s_gt 287 | 288 | def compute_lmse(self, r, s, window_size=20): 289 | """ Compute LMSE error, as per [Grosse et al 2009] MIT Intrinsic Images 290 | dataset """ 291 | r = np.mean(r, axis=-1) 292 | r_gt = self.r_gt 293 | if r_gt.ndim == 3: 294 | r_gt = np.mean(r_gt, axis=-1) 295 | from .lmse import score_image 296 | return score_image( 297 | self.s_gt, r_gt, 298 | s, r, self.mask, window_size) 299 | 300 | def compute_whdr(self, r, delta=0.10): 301 | return self.judgements.compute_whdr(r, delta) 302 | 303 | def compute_error(self, r, s): 304 | """ Compute error for the decomposition using the error metric 305 | associated with the dataset """ 306 | 307 | if self.dataset == "mit": 308 | return self.compute_lmse(r, s) 309 | elif self.dataset == "iiw": 310 | return self.compute_whdr(r) 311 | else: 312 | raise ValueError("Unknown dataset: %s" % self.dataset) 313 | -------------------------------------------------------------------------------- /install_deps.sh: -------------------------------------------------------------------------------- 1 | #! /bin/bash 2 | sudo pip install virtualenv virtualenvwrapper 3 | echo -e "\n# virtualenv and virtualenvwrapper" >> ~/.bashrc 4 | echo "export WORKON_HOME=$HOME/.virtualenvs" >> ~/.bashrc 5 | echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc 6 | export WORKON_HOME=$HOME/.virtualenvs 7 | source /usr/local/bin/virtualenvwrapper.sh 8 | source ~/.bashrc 9 | mkvirtualenv intrinsic -p python3 10 | workon intrinsic 11 | sudo apt-get install -y libeigen3-dev 12 | pip install Pillow Cython numpy scipy scikit-image scikit-learn snakeviz numba 13 | cd krahenbuhl2013 14 | make 15 | cd .. 16 | python3 $1 $2 17 | deactivate 18 | 19 | -------------------------------------------------------------------------------- /judgements.py: -------------------------------------------------------------------------------- 1 | import json 2 | import numpy as np 3 | 4 | class HumanReflectanceJudgements(object): 5 | 6 | def __init__(self, judgements): 7 | if not isinstance(judgements, dict): 8 | raise ValueError("Invalid judgements: %s" % judgements) 9 | 10 | self.judgements = judgements 11 | self.id_to_points = {p['id']: p for p in self.points} 12 | 13 | @staticmethod 14 | def from_file(filename): 15 | judgements = json.load(open(filename)) 16 | return HumanReflectanceJudgements(judgements) 17 | 18 | @property 19 | def points(self): 20 | return self.judgements['intrinsic_points'] 21 | 22 | @property 23 | def comparisons(self): 24 | return self.judgements['intrinsic_comparisons'] 25 | 26 | def compute_whdr(self, r, delta=0.10): 27 | """ Compute the Weighted Human Disagreement for a reflectance image 28 | ``r``. """ 29 | 30 | error_sum = 0.0 31 | weight_sum = 0.0 32 | 33 | for c in self.comparisons: 34 | point1 = self.id_to_points[c['point1']] 35 | point2 = self.id_to_points[c['point2']] 36 | darker = c['darker'] 37 | weight = c['darker_score'] 38 | 39 | if not point1['opaque'] or not point2['opaque']: 40 | continue 41 | if weight < 0 or weight is None: 42 | raise ValueError("Invalid darker_score: %s" % weight) 43 | if darker not in ('1', '2', 'E'): 44 | raise ValueError("Invalid darker: %s" % darker) 45 | 46 | l1 = np.mean(r[ 47 | int(point1['y'] * r.shape[0]), 48 | int(point1['x'] * r.shape[1]), 49 | ...]) 50 | l2 = np.mean(r[ 51 | int(point2['y'] * r.shape[0]), 52 | int(point2['x'] * r.shape[1]), 53 | ...]) 54 | 55 | l1 = max(l1, 1e-10) 56 | l2 = max(l2, 1e-10) 57 | 58 | if l2 / l1 > 1.0 + delta: 59 | alg_darker = '1' 60 | elif l1 / l2 > 1.0 + delta: 61 | alg_darker = '2' 62 | else: 63 | alg_darker = 'E' 64 | 65 | if darker != alg_darker: 66 | error_sum += weight 67 | weight_sum += weight 68 | 69 | if weight_sum: 70 | return error_sum / weight_sum 71 | else: 72 | return None 73 | -------------------------------------------------------------------------------- /krahenbuhl2013/.gitignore: -------------------------------------------------------------------------------- 1 | build/ 2 | krahenbuhl2013.so 3 | krahenbuhl2013.cpp 4 | -------------------------------------------------------------------------------- /krahenbuhl2013/Makefile: -------------------------------------------------------------------------------- 1 | all: 2 | python3 setup.py build_ext -i 3 | python3 -c "import krahenbuhl2013" 4 | 5 | clean: 6 | rm -rf build/ krahenbuhl2013.so krahenbuhl2013.cpp 7 | -------------------------------------------------------------------------------- /krahenbuhl2013/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/krahenbuhl2013/__init__.py -------------------------------------------------------------------------------- /krahenbuhl2013/__init__.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/krahenbuhl2013/__init__.pyc -------------------------------------------------------------------------------- /krahenbuhl2013/__pycache__/__init__.cpython-35.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/krahenbuhl2013/__pycache__/__init__.cpython-35.pyc -------------------------------------------------------------------------------- /krahenbuhl2013/include/densecrf.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2013, Philipp Krähenbühl 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | * Neither the name of the Stanford University nor the 13 | names of its contributors may be used to endorse or promote products 14 | derived from this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY Philipp Krähenbühl ''AS IS'' AND ANY 17 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL Philipp Krähenbühl BE LIABLE FOR ANY 20 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 23 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | */ 27 | 28 | #pragma once 29 | #include "unary.h" 30 | #include "labelcompatibility.h" 31 | #include "objective.h" 32 | #include "pairwise.h" 33 | #include 34 | 35 | /**** DenseCRF ****/ 36 | class DenseCRF{ 37 | protected: 38 | // Number of variables and labels 39 | int N_, M_; 40 | 41 | // Store the unary term 42 | UnaryEnergy * unary_; 43 | 44 | // Store all pairwise potentials 45 | std::vector pairwise_; 46 | 47 | // Don't copy this object, bad stuff will happen 48 | DenseCRF( DenseCRF & o ){} 49 | public: 50 | // Create a dense CRF model of size N with M labels 51 | DenseCRF( int N, int M ); 52 | virtual ~DenseCRF(); 53 | 54 | // Add a pairwise potential defined over some feature space 55 | // The potential will have the form: w*exp(-0.5*|f_i - f_j|^2) 56 | // The kernel shape should be captured by transforming the 57 | // features before passing them into this function 58 | // (ownership of LabelCompatibility will be transfered to this class) 59 | void addPairwiseEnergy( const MatrixXf & features, LabelCompatibility * function, KernelType kernel_type=DIAG_KERNEL, NormalizationType normalization_type=NORMALIZE_SYMMETRIC ); 60 | 61 | // Add your own favorite pairwise potential (ownership will be transfered to this class) 62 | void addPairwiseEnergy( PairwisePotential* potential ); 63 | 64 | // Set the unary potential (ownership will be transfered to this class) 65 | void setUnaryEnergy( UnaryEnergy * unary ); 66 | // Add a constant unary term 67 | void setUnaryEnergy( const MatrixXf & unary ); 68 | // Add a logistic unary term 69 | void setUnaryEnergy( const MatrixXf & L, const MatrixXf & f ); 70 | 71 | // Run inference and return the probabilities 72 | MatrixXf inference( int n_iterations ) const; 73 | 74 | // Run MAP inference and return the map for each pixel 75 | VectorXs map( int n_iterations ) const; 76 | 77 | // Step by step inference 78 | MatrixXf startInference() const; 79 | void stepInference( MatrixXf & Q, MatrixXf & tmp1, MatrixXf & tmp2 ) const; 80 | VectorXs currentMap( const MatrixXf & Q ) const; 81 | 82 | // Learning functions 83 | // Compute the gradient of the objective function over mean-field marginals with 84 | // respect to the model parameters 85 | double gradient( int n_iterations, const ObjectiveFunction & objective, VectorXf * unary_grad, VectorXf * lbl_cmp_grad, VectorXf * kernel_grad=NULL ) const; 86 | public: /* Debugging functions */ 87 | // Compute the unary energy of an assignment l 88 | VectorXf unaryEnergy( const VectorXs & l ); 89 | 90 | // Compute the pairwise energy of an assignment l (half of each pairwise potential is added to each of it's endpoints) 91 | VectorXf pairwiseEnergy( const VectorXs & l, int term=-1 ); 92 | 93 | // Compute the KL-divergence of a set of marginals 94 | double klDivergence( const MatrixXf & Q ) const; 95 | 96 | public: /* Parameters */ 97 | VectorXf unaryParameters() const; 98 | void setUnaryParameters( const VectorXf & v ); 99 | VectorXf labelCompatibilityParameters() const; 100 | void setLabelCompatibilityParameters( const VectorXf & v ); 101 | VectorXf kernelParameters() const; 102 | void setKernelParameters( const VectorXf & v ); 103 | }; 104 | 105 | class DenseCRF2D:public DenseCRF{ 106 | protected: 107 | // Width, height of the 2d grid 108 | int W_, H_; 109 | public: 110 | // Create a 2d dense CRF model of size W x H with M labels 111 | DenseCRF2D( int W, int H, int M ); 112 | virtual ~DenseCRF2D(); 113 | // Add a Gaussian pairwise potential with standard deviation sx and sy 114 | void addPairwiseGaussian( float sx, float sy, LabelCompatibility * function=NULL, KernelType kernel_type=DIAG_KERNEL, NormalizationType normalization_type=NORMALIZE_SYMMETRIC ); 115 | 116 | // Add a Bilateral pairwise potential with spacial standard deviations sx, sy and color standard deviations sr,sg,sb 117 | void addPairwiseBilateral( float sx, float sy, float sr, float sg, float sb, const unsigned char * im, LabelCompatibility * function=NULL, KernelType kernel_type=DIAG_KERNEL, NormalizationType normalization_type=NORMALIZE_SYMMETRIC ); 118 | 119 | // Set the unary potential for a specific variable 120 | using DenseCRF::setUnaryEnergy; 121 | }; 122 | -------------------------------------------------------------------------------- /krahenbuhl2013/include/densecrf_wrapper.h: -------------------------------------------------------------------------------- 1 | #include "densecrf.h" 2 | 3 | class DenseCRFWrapper { 4 | public: 5 | DenseCRFWrapper(int npixels, int nlabels); 6 | virtual ~DenseCRFWrapper(); 7 | 8 | void set_unary_energy(float* unary_costs_ptr); 9 | 10 | void add_pairwise_energy(float* pairwise_costs_ptr, 11 | float* features_ptr, int nfeatures); 12 | 13 | void map(int n_iters, int* result); 14 | 15 | int npixels(); 16 | int nlabels(); 17 | 18 | private: 19 | DenseCRF* m_crf; 20 | int m_npixels; 21 | int m_nlabels; 22 | }; 23 | -------------------------------------------------------------------------------- /krahenbuhl2013/include/labelcompatibility.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2013, Philipp Krähenbühl 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | * Neither the name of the Stanford University nor the 13 | names of its contributors may be used to endorse or promote products 14 | derived from this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY Philipp Krähenbühl ''AS IS'' AND ANY 17 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL Philipp Krähenbühl BE LIABLE FOR ANY 20 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 23 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | */ 27 | #pragma once 28 | #include 29 | using namespace Eigen; 30 | 31 | /**** LabelCompatibility models a function \mu(a,b) ****/ 32 | // To create your own label compatibility implement an "apply" function 33 | // than computes out(a) = sum_{x_j} \mu( a, b ) Q(b) (where Q(b) is the mean-field 34 | // marginal of a specific variable) 35 | // See below for examples 36 | class LabelCompatibility { 37 | public: 38 | virtual ~LabelCompatibility(); 39 | virtual void apply( MatrixXf & out, const MatrixXf & Q ) const = 0; 40 | // For non-symmetric pairwise potentials we would need to use the transpose of the pairwise term 41 | // for parameter learning 42 | virtual void applyTranspose( MatrixXf & out, const MatrixXf & Q ) const; 43 | 44 | // Training and parameters 45 | virtual VectorXf parameters() const; 46 | virtual void setParameters( const VectorXf & v ); 47 | virtual VectorXf gradient( const MatrixXf & b, const MatrixXf & Q ) const; 48 | }; 49 | /**** Implements potts \mu(a,b) = -w[a==b] ****/ 50 | class PottsCompatibility: public LabelCompatibility { 51 | protected: 52 | float w_; 53 | public: 54 | PottsCompatibility( float weight=1.0 ); 55 | virtual void apply( MatrixXf & out_values, const MatrixXf & in_values ) const; 56 | 57 | // Training and parameters 58 | virtual VectorXf parameters() const; 59 | virtual void setParameters( const VectorXf & v ); 60 | virtual VectorXf gradient( const MatrixXf & b, const MatrixXf & Q ) const; 61 | }; 62 | /**** Implements diagonal \mu(a,b) = -[a==b]v(a) ****/ 63 | class DiagonalCompatibility: public LabelCompatibility { 64 | protected: 65 | VectorXf w_; 66 | public: 67 | DiagonalCompatibility( const VectorXf & v ); 68 | virtual void apply( MatrixXf & out_values, const MatrixXf & in_values ) const; 69 | 70 | // Training and parameters 71 | virtual VectorXf parameters() const; 72 | virtual void setParameters( const VectorXf & v ); 73 | virtual VectorXf gradient( const MatrixXf & b, const MatrixXf & Q ) const; 74 | }; 75 | /**** Implements matrix \mu(a,b) [enforces symmetry, but not positive definitness] ****/ 76 | class MatrixCompatibility: public LabelCompatibility { 77 | protected: 78 | MatrixXf w_; 79 | public: 80 | MatrixCompatibility( const MatrixXf & m ); 81 | virtual void apply( MatrixXf & out_values, const MatrixXf & in_values ) const; 82 | virtual void applyTranspose( MatrixXf & out_values, const MatrixXf & in_values ) const; 83 | 84 | // Training and parameters 85 | virtual VectorXf parameters() const; 86 | virtual void setParameters( const VectorXf & v ); 87 | virtual VectorXf gradient( const MatrixXf & b, const MatrixXf & Q ) const; 88 | }; 89 | -------------------------------------------------------------------------------- /krahenbuhl2013/include/objective.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2013, Philipp Krähenbühl 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | * Neither the name of the Stanford University nor the 13 | names of its contributors may be used to endorse or promote products 14 | derived from this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY Philipp Krähenbühl ''AS IS'' AND ANY 17 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL Philipp Krähenbühl BE LIABLE FOR ANY 20 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 23 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | */ 27 | #pragma once 28 | #include 29 | using namespace Eigen; 30 | 31 | typedef Matrix VectorXs; 32 | 33 | /**** Learning Objective ****/ 34 | class ObjectiveFunction { 35 | public: 36 | virtual ~ObjectiveFunction(); 37 | // Evaluate an objective function L(Q) and its gradient \nabla L(Q) 38 | // Return the objetive value L(Q) and set gradient[i*M+l] to Q_i(l)*\partial L / \partial Q_i(l) 39 | // We use the scales gradient here for numerical reasons! 40 | virtual double evaluate( MatrixXf & d_mul_Q, const MatrixXf & Q ) const = 0; 41 | }; 42 | // Log likelihood objective 43 | class LogLikelihood: public ObjectiveFunction { 44 | protected: 45 | VectorXs gt_; 46 | float robust_; 47 | public: 48 | // Give a ground_truth labeling of size N, optional use a robustness term robust>0 49 | LogLikelihood( const VectorXs & gt, float robust=0 ); 50 | // The objective value is sum_i log( Q_i( ground_truth_i ) + robust ) 51 | virtual double evaluate( MatrixXf & d_mul_Q, const MatrixXf & Q ) const; 52 | }; 53 | // Log likelihood objective 54 | class Hamming: public ObjectiveFunction { 55 | protected: 56 | VectorXs gt_; 57 | VectorXf class_weight_; 58 | public: 59 | // Give a ground_truth labeling of size N, reweight classes to cope with an invariance 60 | // weight by w_c = pow( #labels_c, -class_weight_pow ) 61 | Hamming( const VectorXs & gt, float class_weight_pow=0 ); 62 | Hamming( const VectorXs & gt, const VectorXf & class_weight ); 63 | // The objective value is sum_i Q_i( ground_truth_i ) 64 | virtual double evaluate( MatrixXf & d_mul_Q, const MatrixXf & Q ) const; 65 | }; 66 | // Intersection over union objective 67 | class IntersectionOverUnion: public ObjectiveFunction { 68 | protected: 69 | VectorXs gt_; 70 | public: 71 | // Give a ground_truth labeling of size N 72 | IntersectionOverUnion( const VectorXs & gt ); 73 | // The objective value is sum_l ( sum_i [ground_truth_i == l] Q_i( l ) ) / ( |ground_truth_i == l| + sum_i [ground_truth_i != l] Q_i( l ) ) 74 | virtual double evaluate( MatrixXf & d_mul_Q, const MatrixXf & Q ) const; 75 | }; 76 | -------------------------------------------------------------------------------- /krahenbuhl2013/include/optimization.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2013, Philipp Krähenbühl 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | * Neither the name of the Stanford University nor the 13 | names of its contributors may be used to endorse or promote products 14 | derived from this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY Philipp Krähenbühl ''AS IS'' AND ANY 17 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL Philipp Krähenbühl BE LIABLE FOR ANY 20 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 23 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | */ 27 | #pragma once 28 | #include 29 | using namespace Eigen; 30 | 31 | class EnergyFunction { 32 | public: 33 | virtual VectorXf initialValue() = 0; 34 | virtual double gradient( const VectorXf & x, VectorXf & dx ) = 0; 35 | }; 36 | VectorXf minimizeLBFGS( EnergyFunction & efun, int restart=0, bool verbose=false ); 37 | VectorXf numericGradient( EnergyFunction & efun, const VectorXf & x, float EPS=1e-3 ); 38 | VectorXf gradient( EnergyFunction & efun, const VectorXf & x ); 39 | double gradCheck( EnergyFunction & efun, const VectorXf & x, float EPS=1e-3 ); 40 | VectorXf computeFunction( EnergyFunction & efun, const VectorXf & x, const VectorXf & dx, int n_samples = 100 ); 41 | -------------------------------------------------------------------------------- /krahenbuhl2013/include/pairwise.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2013, Philipp Krähenbühl 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | * Neither the name of the Stanford University nor the 13 | names of its contributors may be used to endorse or promote products 14 | derived from this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY Philipp Krähenbühl ''AS IS'' AND ANY 17 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL Philipp Krähenbühl BE LIABLE FOR ANY 20 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 23 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | */ 27 | #pragma once 28 | #include "labelcompatibility.h" 29 | #include "permutohedral.h" 30 | 31 | // The filter in the dense CRF can be normalized in a few different ways 32 | enum NormalizationType { 33 | NO_NORMALIZATION, // No normalization whatsoever (will lead to a substantial approximation error) 34 | NORMALIZE_BEFORE, // Normalize before filtering (Not used, just there for completeness) 35 | NORMALIZE_AFTER, // Normalize after filtering (original normalization in NIPS 11 work) 36 | NORMALIZE_SYMMETRIC, // Normalize before and after (ICML 2013, low approximation error and preserves the symmetry of CRF) 37 | }; 38 | enum KernelType { 39 | CONST_KERNEL, // Constant kernel, no parameters 40 | DIAG_KERNEL, // Diagonal kernel (scaling features) 41 | FULL_KERNEL, // Full kernel matrix (arbitrary squared matrix) 42 | }; 43 | 44 | class Kernel { 45 | public: 46 | virtual ~Kernel(); 47 | virtual void apply( MatrixXf & out, const MatrixXf & Q ) const = 0; 48 | virtual void applyTranspose( MatrixXf & out, const MatrixXf & Q ) const = 0; 49 | virtual VectorXf parameters() const = 0; 50 | virtual void setParameters( const VectorXf & p ) = 0; 51 | virtual VectorXf gradient( const MatrixXf & b, const MatrixXf & Q ) const = 0; 52 | }; 53 | 54 | class PairwisePotential{ 55 | protected: 56 | LabelCompatibility * compatibility_; 57 | Kernel * kernel_; 58 | PairwisePotential( const PairwisePotential &o ){} 59 | void filter( MatrixXf & out, const MatrixXf & in, bool transpose=false ) const; 60 | public: 61 | virtual ~PairwisePotential(); 62 | PairwisePotential(const MatrixXf & features, LabelCompatibility * compatibility, KernelType ktype=CONST_KERNEL, NormalizationType ntype=NORMALIZE_SYMMETRIC); 63 | void apply(MatrixXf & out, const MatrixXf & Q) const; 64 | void applyTranspose(MatrixXf & out, const MatrixXf & Q) const; 65 | 66 | // Get the parameters 67 | virtual VectorXf parameters() const; 68 | virtual VectorXf kernelParameters() const; 69 | virtual void setParameters( const VectorXf & v ); 70 | virtual void setKernelParameters( const VectorXf & v ); 71 | virtual VectorXf gradient( const MatrixXf & b, const MatrixXf & Q ) const; 72 | virtual VectorXf kernelGradient( const MatrixXf & b, const MatrixXf & Q ) const; 73 | }; 74 | -------------------------------------------------------------------------------- /krahenbuhl2013/include/permutohedral.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2013, Philipp Krähenbühl 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | * Neither the name of the Stanford University nor the 13 | names of its contributors may be used to endorse or promote products 14 | derived from this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY Philipp Krähenbühl ''AS IS'' AND ANY 17 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL Philipp Krähenbühl BE LIABLE FOR ANY 20 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 23 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | */ 27 | #pragma once 28 | #include 29 | #include 30 | #include 31 | #include 32 | #include 33 | #include 34 | #include 35 | using namespace Eigen; 36 | 37 | /************************************************/ 38 | /*** Permutohedral Lattice ***/ 39 | /************************************************/ 40 | 41 | class Permutohedral 42 | { 43 | protected: 44 | struct Neighbors{ 45 | int n1, n2; 46 | Neighbors( int n1=0, int n2=0 ):n1(n1),n2(n2){ 47 | } 48 | }; 49 | std::vector offset_, rank_; 50 | std::vector barycentric_; 51 | std::vector blur_neighbors_; 52 | // Number of elements, size of sparse discretized space, dimension of features 53 | int N_, M_, d_; 54 | void sseCompute ( float* out, const float* in, int value_size, bool reverse=false ) const; 55 | void seqCompute ( float* out, const float* in, int value_size, bool reverse=false ) const; 56 | public: 57 | Permutohedral(); 58 | void init ( const MatrixXf & features ); 59 | MatrixXf compute ( const MatrixXf & v, bool reverse=false ) const; 60 | void compute ( MatrixXf & out, const MatrixXf & in, bool reverse=false ) const; 61 | // Compute the gradient of a^T K b 62 | void gradient ( float* df, const float * a, const float* b, int value_size ) const; 63 | }; 64 | -------------------------------------------------------------------------------- /krahenbuhl2013/include/unary.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2013, Philipp Krähenbühl 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | * Neither the name of the Stanford University nor the 13 | names of its contributors may be used to endorse or promote products 14 | derived from this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY Philipp Krähenbühl ''AS IS'' AND ANY 17 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL Philipp Krähenbühl BE LIABLE FOR ANY 20 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 23 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | */ 27 | #pragma once 28 | #include 29 | using namespace Eigen; 30 | 31 | class UnaryEnergy { 32 | public: 33 | virtual ~UnaryEnergy(); 34 | // Set the unary 35 | virtual MatrixXf get( ) const = 0; 36 | // Gradient computation 37 | virtual VectorXf parameters() const; 38 | virtual void setParameters( const VectorXf & v ); 39 | virtual VectorXf gradient( const MatrixXf & b ) const; 40 | }; 41 | class ConstUnaryEnergy: public UnaryEnergy { 42 | protected: 43 | MatrixXf unary_; 44 | public: 45 | ConstUnaryEnergy( const MatrixXf & unary ); 46 | virtual MatrixXf get( ) const; 47 | }; 48 | class LogisticUnaryEnergy: public UnaryEnergy { 49 | protected: 50 | MatrixXf L_, f_; 51 | public: 52 | LogisticUnaryEnergy( const MatrixXf & L, const MatrixXf & feature ); 53 | virtual MatrixXf get( ) const; 54 | virtual VectorXf parameters() const; 55 | virtual void setParameters( const VectorXf & v ); 56 | virtual VectorXf gradient( const MatrixXf & b ) const; 57 | }; 58 | -------------------------------------------------------------------------------- /krahenbuhl2013/krahenbuhl2013.cpython-35m-x86_64-linux-gnu.so: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/krahenbuhl2013/krahenbuhl2013.cpython-35m-x86_64-linux-gnu.so -------------------------------------------------------------------------------- /krahenbuhl2013/krahenbuhl2013.pyx: -------------------------------------------------------------------------------- 1 | # distutils: sources = src/densecrf_wrapper.cpp 2 | 3 | cimport numpy as np 4 | 5 | cdef extern from "include/densecrf_wrapper.h": 6 | cdef cppclass DenseCRFWrapper: 7 | DenseCRFWrapper(int, int) except + 8 | void set_unary_energy(float*) 9 | void add_pairwise_energy(float*, float*, int) 10 | void map(int, int*) 11 | int npixels() 12 | int nlabels() 13 | 14 | cdef class DenseCRF: 15 | cdef DenseCRFWrapper *thisptr 16 | 17 | def __cinit__(self, int npixels, int nlabels): 18 | self.thisptr = new DenseCRFWrapper(npixels, nlabels) 19 | 20 | def __dealloc__(self): 21 | del self.thisptr 22 | 23 | def set_unary_energy(self, float[:, ::1] unary_costs): 24 | if (unary_costs.shape[0] != self.thisptr.npixels() or 25 | unary_costs.shape[1] != self.thisptr.nlabels()): 26 | raise ValueError("Invalid unary_costs shape") 27 | 28 | self.thisptr.set_unary_energy(&unary_costs[0, 0]) 29 | 30 | def add_pairwise_energy(self, float[:, ::1] pairwise_costs, 31 | float[:, ::1] features): 32 | if (pairwise_costs.shape[0] != self.thisptr.nlabels() or 33 | pairwise_costs.shape[1] != self.thisptr.nlabels()): 34 | raise ValueError("Invalid pairwise_costs shape") 35 | if (features.shape[0] != self.thisptr.npixels()): 36 | raise ValueError("Invalid features shape") 37 | 38 | self.thisptr.add_pairwise_energy( 39 | &pairwise_costs[0, 0], 40 | &features[0, 0], 41 | features.shape[1] 42 | ) 43 | 44 | def map(self, int n_iters=10): 45 | import numpy as np 46 | labels = np.empty(self.thisptr.npixels(), dtype=np.int32) 47 | cdef int[::1] labels_view = labels 48 | self.thisptr.map(n_iters, &labels_view[0]) 49 | return labels 50 | -------------------------------------------------------------------------------- /krahenbuhl2013/memory-test/memory_test.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | int main(const int argc, const char **argv) { 4 | int npixels = 1000; 5 | int nfeatures = 5; 6 | int nlabels = 20; 7 | 8 | float* unary_costs = new float[npixels * nlabels]; 9 | float* pairwise_costs = new float[nlabels * nlabels]; 10 | float* features = new float[npixels * nfeatures]; 11 | int* result = new int[npixels]; 12 | int n_crf_iters = 10; 13 | 14 | for (int i = 0; i < npixels * nlabels; i++) 15 | unary_costs[i] = i; 16 | for (int i = 0; i < nlabels * nlabels; i++) 17 | pairwise_costs[i] = i; 18 | for (int i = 0; i < nlabels * nfeatures; i++) 19 | features[i] = i; 20 | for (int i = 0; i < npixels; i++) 21 | result[i] = 0; 22 | 23 | densecrf_map_impl( 24 | unary_costs, 25 | pairwise_costs, 26 | features, 27 | nfeatures, 28 | npixels, 29 | nlabels, 30 | n_crf_iters, 31 | result 32 | ); 33 | 34 | delete[] unary_costs; 35 | delete[] pairwise_costs; 36 | delete[] features; 37 | delete[] result; 38 | 39 | return 0; 40 | } 41 | -------------------------------------------------------------------------------- /krahenbuhl2013/memory-test/premake4: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/krahenbuhl2013/memory-test/premake4 -------------------------------------------------------------------------------- /krahenbuhl2013/memory-test/premake4.lua: -------------------------------------------------------------------------------- 1 | -- A solution contains projects, and defines the available configurations 2 | -- 3 | solution "texture-clean" 4 | 5 | -- global config 6 | language "C++" 7 | configurations { "debug" } 8 | location "build" 9 | includedirs { "../include", "/usr/local/include", "/usr/include" } 10 | libdirs { "/usr/lib", "/usr/local/lib" } 11 | flags { "Symbols", "FatalWarnings", "ExtraWarnings" } 12 | 13 | -- Stack traces 14 | linkoptions { "-rdynamic" } 15 | 16 | -- Eigen 17 | includedirs { "/usr/include/eigen3" } 18 | 19 | -- debug: make config=debug 20 | configuration { "debug" } 21 | kind "ConsoleApp" 22 | defines { "DEBUG" } 23 | buildoptions { "-g3", "-O0" } 24 | targetdir "build/debug" 25 | 26 | project "densecrf_memory_test" 27 | files { 28 | "memory_test.cpp", 29 | "../include/*.h", 30 | "../src/densecrf.cpp", 31 | "../src/labelcompatibility.cpp", 32 | "../src/pairwise.cpp", 33 | "../src/permutohedral.cpp", 34 | "../src/unary.cpp", 35 | "../src/util.cpp", 36 | "../src/densecrf_map.cpp" 37 | } 38 | -------------------------------------------------------------------------------- /krahenbuhl2013/setup.py: -------------------------------------------------------------------------------- 1 | from distutils.core import setup 2 | from distutils.extension import Extension 3 | try: 4 | from Cython.Build import cythonize 5 | import numpy 6 | except ImportError: 7 | print("You must have Cython >=0.17 and NumPy to build!") 8 | import sys 9 | sys.exit(1) 10 | 11 | setup( 12 | ext_modules=cythonize(Extension( 13 | 'krahenbuhl2013', 14 | sources=[ 15 | 'krahenbuhl2013.pyx', 16 | "src/densecrf.cpp", 17 | "src/labelcompatibility.cpp", 18 | "src/pairwise.cpp", 19 | "src/permutohedral.cpp", 20 | "src/unary.cpp", 21 | "src/util.cpp", 22 | "src/densecrf_wrapper.cpp", 23 | ], 24 | include_dirs=[ 25 | numpy.get_include(), 26 | "include", 27 | "/usr/include/eigen3", 28 | ], 29 | language="c++", 30 | )), 31 | ) 32 | -------------------------------------------------------------------------------- /krahenbuhl2013/setup.py.bak: -------------------------------------------------------------------------------- 1 | from distutils.core import setup 2 | from distutils.extension import Extension 3 | try: 4 | from Cython.Build import cythonize 5 | import numpy 6 | except ImportError: 7 | print "You must have Cython >=0.17 and NumPy to build!" 8 | import sys 9 | sys.exit(1) 10 | 11 | setup( 12 | ext_modules=cythonize(Extension( 13 | 'krahenbuhl2013', 14 | sources=[ 15 | 'krahenbuhl2013.pyx', 16 | "src/densecrf.cpp", 17 | "src/labelcompatibility.cpp", 18 | "src/pairwise.cpp", 19 | "src/permutohedral.cpp", 20 | "src/unary.cpp", 21 | "src/util.cpp", 22 | "src/densecrf_wrapper.cpp", 23 | ], 24 | include_dirs=[ 25 | numpy.get_include(), 26 | "include", 27 | "/usr/include/eigen3", 28 | ], 29 | language="c++", 30 | )), 31 | ) 32 | -------------------------------------------------------------------------------- /krahenbuhl2013/src/densecrf.cpp: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2013, Philipp Krähenbühl 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | * Neither the name of the Stanford University nor the 13 | names of its contributors may be used to endorse or promote products 14 | derived from this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY Philipp Krähenbühl ''AS IS'' AND ANY 17 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL Philipp Krähenbühl BE LIABLE FOR ANY 20 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 23 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | */ 27 | 28 | #include "densecrf.h" 29 | #include "permutohedral.h" 30 | #include "util.h" 31 | #include "pairwise.h" 32 | #include 33 | #include 34 | #include 35 | 36 | ///////////////////////////// 37 | ///// Alloc / Dealloc ///// 38 | ///////////////////////////// 39 | DenseCRF::DenseCRF(int N, int M) : N_(N), M_(M), unary_(0) { 40 | } 41 | DenseCRF::~DenseCRF() { 42 | if (unary_) 43 | delete unary_; 44 | for( unsigned int i=0; iget(); 120 | expAndNormalize( Q, -unary ); 121 | 122 | for( int it=0; itapply( tmp2, Q ); 126 | tmp1 -= tmp2; 127 | } 128 | expAndNormalize( Q, tmp1 ); 129 | } 130 | return Q; 131 | } 132 | VectorXs DenseCRF::map ( int n_iterations ) const { 133 | // Run inference 134 | MatrixXf Q = inference( n_iterations ); 135 | // Find the map 136 | return currentMap( Q ); 137 | } 138 | /////////////////// 139 | ///// Debug ///// 140 | /////////////////// 141 | VectorXf DenseCRF::unaryEnergy(const VectorXs & l) { 142 | assert( l.cols() == N_ ); 143 | VectorXf r( N_ ); 144 | r.fill(0.f); 145 | if( unary_ ) { 146 | MatrixXf unary = unary_->get(); 147 | 148 | for( int i=0; iapply( Q, Q ); 171 | for( int i=0; iget() ); 185 | return Q; 186 | } 187 | void DenseCRF::stepInference( MatrixXf & Q, MatrixXf & tmp1, MatrixXf & tmp2 ) const{ 188 | tmp1.resize( Q.rows(), Q.cols() ); 189 | tmp1.fill(0); 190 | if( unary_ ) 191 | tmp1 -= unary_->get(); 192 | 193 | // Add up all pairwise potentials 194 | for( unsigned int k=0; kapply( tmp2, Q ); 196 | tmp1 -= tmp2; 197 | } 198 | 199 | // Exponentiate and normalize 200 | expAndNormalize( Q, tmp1 ); 201 | } 202 | VectorXs DenseCRF::currentMap( const MatrixXf & Q ) const{ 203 | VectorXs r(Q.cols()); 204 | // Find the map 205 | for( int i=0; iget(); 223 | for( int i=0; iapply( tmp, Q ); 232 | kl += (Q.array()*tmp.array()).sum(); 233 | } 234 | return kl; 235 | } 236 | 237 | // Gradient computations 238 | double DenseCRF::gradient( int n_iterations, const ObjectiveFunction & objective, VectorXf * unary_grad, VectorXf * lbl_cmp_grad, VectorXf * kernel_grad) const { 239 | // Run inference 240 | std::vector< MatrixXf > Q(n_iterations+1); 241 | MatrixXf tmp1, unary( M_, N_ ), tmp2; 242 | unary.fill(0); 243 | if( unary_ ) 244 | unary = unary_->get(); 245 | expAndNormalize( Q[0], -unary ); 246 | for( int it=0; itapply( tmp2, Q[it] ); 250 | tmp1 -= tmp2; 251 | } 252 | expAndNormalize( Q[it+1], tmp1 ); 253 | } 254 | 255 | // Compute the objective value 256 | MatrixXf b( M_, N_ ); 257 | double r = objective.evaluate( b, Q[n_iterations] ); 258 | sumAndNormalize( b, b, Q[n_iterations] ); 259 | 260 | // Compute the gradient 261 | if(unary_grad && unary_) 262 | *unary_grad = unary_->gradient( b ); 263 | if( lbl_cmp_grad ) 264 | *lbl_cmp_grad = 0*labelCompatibilityParameters(); 265 | if( kernel_grad ) 266 | *kernel_grad = 0*kernelParameters(); 267 | 268 | for( int it=n_iterations-1; it>=0; it-- ) { 269 | // Do the inverse message passing 270 | tmp1.fill(0); 271 | int ip = 0, ik = 0; 272 | // Add up all pairwise potentials 273 | for( unsigned int k=0; kgradient( b, Q[it] ); 277 | lbl_cmp_grad->segment( ip, pg.rows() ) += pg; 278 | ip += pg.rows(); 279 | } 280 | // Compute the kernel gradient expression 281 | if( kernel_grad ) { 282 | VectorXf pg = pairwise_[k]->kernelGradient( b, Q[it] ); 283 | kernel_grad->segment( ik, pg.rows() ) += pg; 284 | ik += pg.rows(); 285 | } 286 | // Compute the new b 287 | pairwise_[k]->applyTranspose( tmp2, b ); 288 | tmp1 += tmp2; 289 | } 290 | sumAndNormalize( b, tmp1.array()*Q[it].array(), Q[it] ); 291 | 292 | // Add the gradient 293 | if(unary_grad && unary_) 294 | *unary_grad += unary_->gradient( b ); 295 | } 296 | return r; 297 | } 298 | VectorXf DenseCRF::unaryParameters() const { 299 | if( unary_ ) 300 | return unary_->parameters(); 301 | return VectorXf(); 302 | } 303 | void DenseCRF::setUnaryParameters( const VectorXf & v ) { 304 | if( unary_ ) 305 | unary_->setParameters( v ); 306 | } 307 | VectorXf DenseCRF::labelCompatibilityParameters() const { 308 | std::vector< VectorXf > terms; 309 | for( unsigned int k=0; kparameters() ); 311 | int np=0; 312 | for( unsigned int k=0; k n; 323 | for( unsigned int k=0; kparameters().rows() ); 325 | int np=0; 326 | for( unsigned int k=0; ksetParameters( v.segment( i, n[k] ) ); 331 | i += n[k]; 332 | } 333 | } 334 | VectorXf DenseCRF::kernelParameters() const { 335 | std::vector< VectorXf > terms; 336 | for( unsigned int k=0; kkernelParameters() ); 338 | int np=0; 339 | for( unsigned int k=0; k n; 350 | for( unsigned int k=0; kkernelParameters().rows() ); 352 | int np=0; 353 | for( unsigned int k=0; ksetKernelParameters( v.segment( i, n[k] ) ); 358 | i += n[k]; 359 | } 360 | } 361 | -------------------------------------------------------------------------------- /krahenbuhl2013/src/densecrf_wrapper.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | #include "densecrf.h" 3 | #include "densecrf_wrapper.h" 4 | 5 | DenseCRFWrapper::DenseCRFWrapper(int npixels, int nlabels) 6 | : m_npixels(npixels), m_nlabels(nlabels) { 7 | m_crf = new DenseCRF(npixels, nlabels); 8 | } 9 | 10 | DenseCRFWrapper::~DenseCRFWrapper() { 11 | delete m_crf; 12 | } 13 | 14 | int DenseCRFWrapper::npixels() { return m_npixels; } 15 | int DenseCRFWrapper::nlabels() { return m_nlabels; } 16 | 17 | void DenseCRFWrapper::add_pairwise_energy(float* pairwise_costs_ptr, float* features_ptr, int nfeatures) { 18 | m_crf->addPairwiseEnergy( 19 | Eigen::Map(features_ptr, nfeatures, m_npixels), 20 | new MatrixCompatibility( 21 | Eigen::Map(pairwise_costs_ptr, m_nlabels, m_nlabels) 22 | ), 23 | DIAG_KERNEL, 24 | NORMALIZE_SYMMETRIC 25 | ); 26 | } 27 | 28 | void DenseCRFWrapper::set_unary_energy(float* unary_costs_ptr) { 29 | m_crf->setUnaryEnergy( 30 | Eigen::Map( 31 | unary_costs_ptr, m_nlabels, m_npixels) 32 | ); 33 | } 34 | 35 | void DenseCRFWrapper::map(int n_iters, int* labels) { 36 | VectorXs labels_vec = m_crf->map(n_iters); 37 | for (int i = 0; i < m_npixels; i ++) 38 | labels[i] = labels_vec(i); 39 | } 40 | -------------------------------------------------------------------------------- /krahenbuhl2013/src/labelcompatibility.cpp: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2013, Philipp Krähenbühl 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | * Neither the name of the Stanford University nor the 13 | names of its contributors may be used to endorse or promote products 14 | derived from this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY Philipp Krähenbühl ''AS IS'' AND ANY 17 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL Philipp Krähenbühl BE LIABLE FOR ANY 20 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 23 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | */ 27 | #include "labelcompatibility.h" 28 | 29 | LabelCompatibility::~LabelCompatibility() { 30 | } 31 | void LabelCompatibility::applyTranspose( MatrixXf & out, const MatrixXf & Q ) const { 32 | apply( out, Q ); 33 | } 34 | VectorXf LabelCompatibility::parameters() const { 35 | return VectorXf(); 36 | } 37 | void LabelCompatibility::setParameters( const VectorXf & v ) { 38 | } 39 | VectorXf LabelCompatibility::gradient( const MatrixXf & b, const MatrixXf & Q ) const { 40 | return VectorXf(); 41 | } 42 | 43 | 44 | PottsCompatibility::PottsCompatibility( float weight ): w_(weight) { 45 | } 46 | void PottsCompatibility::apply( MatrixXf & out, const MatrixXf & Q ) const { 47 | out = -w_*Q; 48 | } 49 | VectorXf PottsCompatibility::parameters() const { 50 | VectorXf r(1); 51 | r[0] = w_; 52 | return r; 53 | } 54 | void PottsCompatibility::setParameters( const VectorXf & v ) { 55 | w_ = v[0]; 56 | } 57 | VectorXf PottsCompatibility::gradient( const MatrixXf & b, const MatrixXf & Q ) const { 58 | VectorXf r(1); 59 | r[0] = -(b.array()*Q.array()).sum(); 60 | return r; 61 | } 62 | 63 | 64 | DiagonalCompatibility::DiagonalCompatibility( const VectorXf & v ): w_(v) { 65 | } 66 | void DiagonalCompatibility::apply( MatrixXf & out, const MatrixXf & Q ) const { 67 | assert( w_.rows() == Q.rows() ); 68 | out = w_.asDiagonal()*Q; 69 | } 70 | VectorXf DiagonalCompatibility::parameters() const { 71 | return w_; 72 | } 73 | void DiagonalCompatibility::setParameters( const VectorXf & v ) { 74 | w_ = v; 75 | } 76 | VectorXf DiagonalCompatibility::gradient( const MatrixXf & b, const MatrixXf & Q ) const { 77 | return (b.array()*Q.array()).rowwise().sum(); 78 | } 79 | MatrixCompatibility::MatrixCompatibility( const MatrixXf & m ): w_(0.5*(m + m.transpose())) { 80 | assert( m.cols() == m.rows() ); 81 | } 82 | void MatrixCompatibility::apply( MatrixXf & out, const MatrixXf & Q ) const { 83 | out = w_*Q; 84 | } 85 | void MatrixCompatibility::applyTranspose( MatrixXf & out, const MatrixXf & Q ) const { 86 | out = w_.transpose()*Q; 87 | } 88 | VectorXf MatrixCompatibility::parameters() const { 89 | VectorXf r( w_.cols()*(w_.rows()+1)/2 ); 90 | for( int i=0,k=0; i 29 | #include 30 | #include 31 | 32 | /**** Learning Objectives ****/ 33 | ObjectiveFunction::~ObjectiveFunction(){ 34 | } 35 | LogLikelihood::LogLikelihood( const VectorXs & gt, float robust ):gt_( gt ),robust_(robust){ 36 | } 37 | double LogLikelihood::evaluate( MatrixXf & d_mul_Q, const MatrixXf & Q ) const { 38 | assert( gt_.rows() == Q.cols() ); 39 | const int N = Q.cols(), M = Q.rows(); 40 | double r = 0; 41 | d_mul_Q = 0*Q; 42 | for( int i=0; i= M ) 55 | M = gt[i]+1; 56 | VectorXf cnt = VectorXf::Zero( M ); 57 | for( int i=0; i= 0 ) 59 | cnt[gt[i]] += 1; 60 | class_weight_ = cnt.array() / cnt.array().sum(); 61 | class_weight_ = class_weight_.array().pow( -class_weight_pow ); 62 | class_weight_ = class_weight_.array() / (cnt.array()*class_weight_.array()).sum(); 63 | } 64 | Hamming::Hamming( const VectorXs & gt, const VectorXf & w ):gt_( gt ),class_weight_(w){ 65 | } 66 | double Hamming::evaluate( MatrixXf & d_mul_Q, const MatrixXf & Q ) const { 67 | assert( gt_.rows() == Q.cols() ); 68 | const int N = Q.cols(), M = Q.rows(); 69 | double r = 0; 70 | d_mul_Q = 0*Q; 71 | for( int i=0; i 30 | #include 31 | 32 | static lbfgsfloatval_t evaluate( 33 | void *instance, 34 | const lbfgsfloatval_t *x, 35 | lbfgsfloatval_t *g, 36 | const int n, 37 | const lbfgsfloatval_t step 38 | ) 39 | { 40 | EnergyFunction * efun = static_cast( instance ); 41 | 42 | VectorXf vx( n ), vg( n ); 43 | std::copy( x, x+n, vx.data() ); 44 | lbfgsfloatval_t r = efun->gradient( vx, vg ); 45 | 46 | std::copy( vg.data(), vg.data()+n, g ); 47 | return r; 48 | } 49 | 50 | static int progress( 51 | void *instance, 52 | const lbfgsfloatval_t *x, 53 | const lbfgsfloatval_t *g, 54 | const lbfgsfloatval_t fx, 55 | const lbfgsfloatval_t xnorm, 56 | const lbfgsfloatval_t gnorm, 57 | const lbfgsfloatval_t step, 58 | int n, 59 | int k, 60 | int ls 61 | ) 62 | { 63 | printf("Iteration %d:\n", k); 64 | printf(" fx = %f, xnorm = %f, gnorm = %f, step = %f\n", fx, xnorm, gnorm, step); 65 | printf("\n"); 66 | return 0; 67 | } 68 | VectorXf minimizeLBFGS( EnergyFunction & efun, int restart, bool verbose ) { 69 | VectorXf x0 = efun.initialValue(); 70 | const int n = x0.rows(); 71 | 72 | lbfgsfloatval_t *x = lbfgs_malloc(n); 73 | if (x == NULL) { 74 | printf("ERROR: Failed to allocate a memory block for variables.\n"); 75 | return x0; 76 | } 77 | std::copy( x0.data(), x0.data()+n, x ); 78 | 79 | lbfgs_parameter_t param; 80 | lbfgs_parameter_init(¶m); 81 | // You might want to adjust the parameters to your problem 82 | param.epsilon = 1e-6; 83 | param.max_iterations = 50; 84 | 85 | double last_f = 1e100; 86 | int ret; 87 | for( int i=0; i<=restart; i++ ) { 88 | lbfgsfloatval_t fx; 89 | ret = lbfgs(n, x, &fx, evaluate, verbose?progress:NULL, &efun, ¶m); 90 | if( last_f > fx ) 91 | last_f = fx; 92 | else 93 | break; 94 | } 95 | 96 | if ( verbose ) { 97 | printf("L-BFGS optimization terminated with status code = %d\n", ret); 98 | } 99 | 100 | std::copy( x, x+n, x0.data() ); 101 | lbfgs_free(x); 102 | return x0; 103 | } 104 | VectorXf numericGradient( EnergyFunction & efun, const VectorXf & x, float EPS ) { 105 | VectorXf g( x.rows() ), tmp; 106 | for( int i=0; i 29 | 30 | Kernel::~Kernel() { 31 | } 32 | class DenseKernel: public Kernel { 33 | protected: 34 | MatrixXf f_; 35 | KernelType ktype_; 36 | NormalizationType ntype_; 37 | Permutohedral lattice_; 38 | VectorXf norm_; 39 | MatrixXf parameters_; 40 | void initLattice( const MatrixXf & f ) { 41 | const int N = f.cols(); 42 | lattice_.init( f ); 43 | 44 | norm_ = lattice_.compute( VectorXf::Ones( N ).transpose() ).transpose(); 45 | 46 | if ( ntype_ == NO_NORMALIZATION ) { 47 | float mean_norm = 0; 48 | for ( int i=0; iapply( out, Q ); 175 | 176 | // Apply the compatibility 177 | compatibility_->apply( out, out ); 178 | } 179 | void PairwisePotential::applyTranspose(MatrixXf & out, const MatrixXf & Q) const { 180 | kernel_->applyTranspose( out, Q ); 181 | // Apply the compatibility 182 | compatibility_->applyTranspose( out, out ); 183 | } 184 | VectorXf PairwisePotential::parameters() const { 185 | return compatibility_->parameters(); 186 | } 187 | void PairwisePotential::setParameters( const VectorXf & v ) { 188 | compatibility_->setParameters( v ); 189 | } 190 | VectorXf PairwisePotential::gradient( const MatrixXf & b, const MatrixXf & Q ) const { 191 | MatrixXf filtered_Q = 0*Q; 192 | // You could reuse the filtered_b from applyTranspose 193 | kernel_->apply( filtered_Q, Q ); 194 | return compatibility_->gradient(b,filtered_Q); 195 | } 196 | VectorXf PairwisePotential::kernelParameters() const { 197 | return kernel_->parameters(); 198 | } 199 | void PairwisePotential::setKernelParameters( const VectorXf & v ) { 200 | kernel_->setParameters( v ); 201 | } 202 | VectorXf PairwisePotential::kernelGradient( const MatrixXf & b, const MatrixXf & Q ) const { 203 | MatrixXf lbl_Q = 0*Q; 204 | // You could reuse the filtered_b from applyTranspose 205 | compatibility_->apply( lbl_Q, Q ); 206 | return kernel_->gradient(b,lbl_Q); 207 | } 208 | -------------------------------------------------------------------------------- /krahenbuhl2013/src/permutohedral.cpp: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2013, Philipp Krähenbühl 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | * Neither the name of the Stanford University nor the 13 | names of its contributors may be used to endorse or promote products 14 | derived from this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY Philipp Krähenbühl ''AS IS'' AND ANY 17 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL Philipp Krähenbühl BE LIABLE FOR ANY 20 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 23 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | */ 27 | 28 | #include "permutohedral.h" 29 | 30 | #ifdef WIN32 31 | inline int round(double X) { 32 | return int(X+.5); 33 | } 34 | #endif 35 | 36 | #ifdef __SSE__ 37 | // SSE Permutoheral lattice 38 | # define SSE_PERMUTOHEDRAL 39 | #endif 40 | 41 | #if defined(SSE_PERMUTOHEDRAL) 42 | # include 43 | # include 44 | # ifdef __SSE4_1__ 45 | # include 46 | # endif 47 | #endif 48 | 49 | 50 | /************************************************/ 51 | /*** Hash Table ***/ 52 | /************************************************/ 53 | 54 | class HashTable{ 55 | protected: 56 | size_t key_size_, filled_, capacity_; 57 | std::vector< short > keys_; 58 | std::vector< int > table_; 59 | void grow(){ 60 | // Create the new memory and copy the values in 61 | int old_capacity = capacity_; 62 | capacity_ *= 2; 63 | std::vector old_keys( (old_capacity+10)*key_size_ ); 64 | std::copy( keys_.begin(), keys_.end(), old_keys.begin() ); 65 | std::vector old_table( capacity_, -1 ); 66 | 67 | // Swap the memory 68 | table_.swap( old_table ); 69 | keys_.swap( old_keys ); 70 | 71 | // Reinsert each element 72 | for( int i=0; i= 0){ 74 | int e = old_table[i]; 75 | size_t h = hash( getKey(e) ) % capacity_; 76 | for(; table_[h] >= 0; h = h= capacity_) grow(); 100 | // Get the hash value 101 | size_t h = hash( k ) % capacity_; 102 | // Find the element with he right key, using linear probing 103 | while(1){ 104 | int e = table_[h]; 105 | if (e==-1){ 106 | if (create){ 107 | // Insert a new key and return the new id 108 | for( size_t i=0; i0; j-- ){ 203 | __m128 cf = f[j-1]*scale_factor[j-1]; 204 | elevated[j] = sm - _mm_set1_ps(j)*cf; 205 | sm += cf; 206 | } 207 | elevated[0] = sm; 208 | 209 | // Find the closest 0-colored simplex through rounding 210 | __m128 sum = Zero; 211 | for( int i=0; i<=d_; i++ ){ 212 | __m128 v = invdplus1 * elevated[i]; 213 | #ifdef __SSE4_1__ 214 | v = _mm_round_ps( v, _MM_FROUND_TO_NEAREST_INT ); 215 | #else 216 | v = _mm_cvtepi32_ps( _mm_cvtps_epi32( v ) ); 217 | #endif 218 | rem0[i] = v*dplus1; 219 | sum += v; 220 | } 221 | 222 | // Find the simplex we are in and store it in rank (where rank describes what position coorinate i has in the sorted order of the features values) 223 | for( int i=0; i<=d_; i++ ) 224 | rank[i] = Zero; 225 | for( int i=0; i0; j-- ){ 366 | float cf = f[j-1]*scale_factor[j-1]; 367 | elevated[j] = sm - j*cf; 368 | sm += cf; 369 | } 370 | elevated[0] = sm; 371 | 372 | // Find the closest 0-colored simplex through rounding 373 | float down_factor = 1.0f / (d_+1); 374 | float up_factor = (d_+1); 375 | int sum = 0; 376 | for( int i=0; i<=d_; i++ ){ 377 | //int rd1 = round( down_factor * elevated[i]); 378 | int rd2; 379 | float v = down_factor * elevated[i]; 380 | float up = ceilf(v)*up_factor; 381 | float down = floorf(v)*up_factor; 382 | if (up - elevated[i] < elevated[i] - down) rd2 = (short)up; 383 | else rd2 = (short)down; 384 | 385 | //if(rd1!=rd2) 386 | // break; 387 | 388 | rem0[i] = rd2; 389 | sum += rd2*down_factor; 390 | } 391 | 392 | // Find the simplex we are in and store it in rank (where rank describes what position coorinate i has in the sorted order of the features values) 393 | for( int i=0; i<=d_; i++ ) 394 | rank[i] = 0; 395 | for( int i=0; i d_ ){ 412 | rank[i] -= d_+1; 413 | rem0[i] -= d_+1; 414 | } 415 | } 416 | 417 | // Compute the barycentric coordinates (p.10 in [Adams etal 2010]) 418 | for( int i=0; i<=d_+1; i++ ) 419 | barycentric[i] = 0; 420 | for( int i=0; i<=d_; i++ ){ 421 | float v = (elevated[i] - rem0[i])*down_factor; 422 | barycentric[d_-rank[i] ] += v; 423 | barycentric[d_-rank[i]+1] -= v; 424 | } 425 | // Wrap around 426 | barycentric[0] += 1.0 + barycentric[d_+1]; 427 | 428 | // Compute all vertices and their offset 429 | for( int remainder=0; remainder<=d_; remainder++ ){ 430 | for( int i=0; i 0 (used for blurring) 479 | float * values = new float[ (M_+2)*value_size ]; 480 | float * new_values = new float[ (M_+2)*value_size ]; 481 | 482 | for( int i=0; i<(M_+2)*value_size; i++ ) 483 | values[i] = new_values[i] = 0; 484 | 485 | // Splatting 486 | for( int i=0; i=0; reverse?j--:j++ ){ 496 | for( int i=0; i 0 (used for blurring) 533 | __m128 * sse_val = (__m128*) _mm_malloc( sse_value_size*sizeof(__m128), 16 ); 534 | __m128 * values = (__m128*) _mm_malloc( (M_+2)*sse_value_size*sizeof(__m128), 16 ); 535 | __m128 * new_values = (__m128*) _mm_malloc( (M_+2)*sse_value_size*sizeof(__m128), 16 ); 536 | 537 | __m128 Zero = _mm_set1_ps( 0 ); 538 | 539 | for( int i=0; i<(M_+2)*sse_value_size; i++ ) 540 | values[i] = new_values[i] = Zero; 541 | for( int i=0; i=0; reverse?j--:j++ ){ 557 | for( int i=0; i 0 (used for blurring) 615 | float * values = new float[ (M_+2)*value_size ]; 616 | float * new_values = new float[ (M_+2)*value_size ]; 617 | 618 | // Set the results to 0 619 | std::fill( df, df+N_*d_, 0.f ); 620 | 621 | // Initialize some constants 622 | std::vector scale_factor( d_ ); 623 | float inv_std_dev = sqrt(2.0 / 3.0)*(d_+1); 624 | for( int i=0; i=0; dir?j--:j++ ){ 646 | for( int i=0; i r_a( (d_+1)*value_size ), sm( value_size ); 662 | 663 | for( int i=0; id_?0:r0+1; 669 | int o0 = offset_[i*(d_+1)+r0]+1; 670 | int o1 = offset_[i*(d_+1)+r1]+1; 671 | for( int k=0; k 30 | 31 | float* allocate(size_t N) { 32 | float * r = NULL; 33 | if (N>0) 34 | #ifdef SSE_DENSE_CRF 35 | r = (float*)_mm_malloc( N*sizeof(float)+16, 16 ); 36 | #else 37 | r = new float[N]; 38 | #endif 39 | memset( r, 0, sizeof(float)*N); 40 | return r; 41 | } 42 | void deallocate(float*& ptr) { 43 | if (ptr) 44 | #ifdef SSE_DENSE_CRF 45 | _mm_free( ptr ); 46 | #else 47 | delete[] ptr; 48 | #endif 49 | ptr = NULL; 50 | } 51 | -------------------------------------------------------------------------------- /krahenbuhl2013/src/util.h: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright (c) 2013, Philipp Krähenbühl 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without 6 | modification, are permitted provided that the following conditions are met: 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright 10 | notice, this list of conditions and the following disclaimer in the 11 | documentation and/or other materials provided with the distribution. 12 | * Neither the name of the Stanford University nor the 13 | names of its contributors may be used to endorse or promote products 14 | derived from this software without specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY Philipp Krähenbühl ''AS IS'' AND ANY 17 | EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL Philipp Krähenbühl BE LIABLE FOR ANY 20 | DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 23 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | */ 27 | 28 | #pragma once 29 | 30 | #include "densecrf.h" 31 | #include "permutohedral.h" 32 | 33 | #ifdef __SSE__ 34 | # define SSE_DENSE_CRF 35 | #endif 36 | 37 | #if defined(SSE_DENSE_CRF) 38 | # include 39 | # include 40 | #endif 41 | 42 | // Memory handling switches between SSE and new 43 | float* allocate ( size_t N ) ; 44 | void deallocate ( float *& ptr ) ; 45 | -------------------------------------------------------------------------------- /lmse.py: -------------------------------------------------------------------------------- 1 | """ 2 | LMSE code from MIT Intrinsic Images dataset: 3 | 4 | Roger Grosse, Micah K. Johnson, Edward H. Adelson, and William T. Freeman, 5 | Ground truth dataset and baseline evaluations for intrinsic image 6 | algorithms, in Proceedings of the International Conference on Computer 7 | Vision (ICCV), 2009. 8 | http://people.csail.mit.edu/rgrosse/intrinsic/ 9 | 10 | NOTE: a small fix was included (see comment in local_error) 11 | 12 | """ 13 | 14 | import numpy as np 15 | 16 | 17 | def ssq_error(correct, estimate, mask): 18 | """Compute the sum-squared-error for an image, where the estimate is 19 | multiplied by a scalar which minimizes the error. Sums over all pixels 20 | where mask is True. If the inputs are color, each color channel can be 21 | rescaled independently.""" 22 | assert correct.ndim == 2 23 | if np.sum(estimate**2 * mask) > 1e-5: 24 | alpha = np.sum(correct * estimate * mask) / np.sum(estimate**2 * mask) 25 | else: 26 | alpha = 0. 27 | return np.sum(mask * (correct - alpha*estimate) ** 2) 28 | 29 | 30 | def local_error(correct, estimate, mask, window_size, window_shift): 31 | """Returns the sum of the local sum-squared-errors, where the estimate may 32 | be rescaled within each local region to minimize the error. The windows are 33 | window_size x window_size, and they are spaced by window_shift.""" 34 | M, N = correct.shape[:2] 35 | ssq = total = 0. 36 | for i in range(0, M - window_size + 1, window_shift): 37 | for j in range(0, N - window_size + 1, window_shift): 38 | correct_curr = correct[i:i+window_size, j:j+window_size] 39 | estimate_curr = estimate[i:i+window_size, j:j+window_size] 40 | mask_curr = mask[i:i+window_size, j:j+window_size] 41 | ssq += ssq_error(correct_curr, estimate_curr, mask_curr) 42 | # FIX: in the original codebase, this was outdented, which allows 43 | # for scores greater than 1 (which should not be possible). On the 44 | # MIT dataset images, this makes a negligible difference, but on 45 | # larger images, this can have a significant effect. 46 | total += np.sum(mask_curr * correct_curr**2) 47 | assert -np.isnan(ssq/total) 48 | 49 | return ssq / total 50 | 51 | 52 | def score_image(true_shading, true_refl, estimate_shading, estimate_refl, mask, window_size=20): 53 | return 0.5 * local_error(true_shading, estimate_shading, mask, window_size, window_size//2) + \ 54 | 0.5 * local_error(true_refl, estimate_refl, mask, window_size, window_size//2) 55 | -------------------------------------------------------------------------------- /optimization.py: -------------------------------------------------------------------------------- 1 | import scipy 2 | import numpy as np 3 | 4 | def minimize_l2(A_data, A_rows, A_cols, A_shape, b, damp=1e-8, logging=False): 5 | A = scipy.sparse.csr_matrix((A_data, (A_rows, A_cols)), shape=A_shape) 6 | return scipy.sparse.linalg.lsmr(A, b, damp=damp)[0] 7 | 8 | 9 | def minimize_l1(A_data, A_rows, A_cols, A_shape, b, x0=None, 10 | tol=1e-6, irls_epsilon=1e-6, damp=1e-8, 11 | max_iters=100, logging=False): 12 | """ 13 | Perform L1 minimization of ``sum(|A.dot(x) - b|)`` via iteratively 14 | reweighted least squares. 15 | """ 16 | 17 | if logging: 18 | print(('solving sparse linear system (%s x %s, %s nnz)...' % ( 19 | A_shape[0], A_shape[1], len(A_data)))) 20 | if A_shape[0] == 0 or A_shape[1] == 0 or b.shape[0] == 0: 21 | print('Warning: empty linear system! returning 0') 22 | return np.zeros(A_shape[1]) 23 | 24 | # construct matrix 25 | A = scipy.sparse.csr_matrix((A_data, (A_rows, A_cols)), shape=A_shape, dtype=np.float32) 26 | 27 | # initial solution 28 | if x0 is not None: 29 | x = x0 30 | else: 31 | x = scipy.sparse.linalg.lsmr(A, b, damp=damp)[0] 32 | 33 | prev_x = x 34 | prev_mean_error = float('inf') 35 | 36 | for i in range(max_iters): 37 | error = np.abs(A.dot(x) - b) 38 | mean_error = np.mean(error) 39 | 40 | if logging and i % 10 == 0: 41 | print(('l1 optimization: (iter %s) mean_error: %s' % (i, mean_error))) 42 | 43 | # exit conditions 44 | delta_error = prev_mean_error - mean_error 45 | if delta_error < 0: 46 | if logging: 47 | print(('l1 optimization: (iter %s) mean_error increased: %s --> %s (exit)' % 48 | (i, prev_mean_error, mean_error))) 49 | return prev_x 50 | elif delta_error < tol: 51 | if logging: 52 | print(('l1 optimization: (iter %s) mean_error: %s, delta_error: %s < %s (exit)' % 53 | (i, mean_error, delta_error, tol))) 54 | return x 55 | 56 | prev_x = x 57 | prev_mean_error = mean_error 58 | 59 | # solve next problem 60 | w = np.sqrt(np.reciprocal(error + irls_epsilon)) 61 | Aw_data = A_data * w[A_rows] 62 | Aw = scipy.sparse.csr_matrix((Aw_data, (A_rows, A_cols)), shape=A_shape) 63 | bw = b * w 64 | x = scipy.sparse.linalg.lsmr(Aw, bw, damp=damp)[0] 65 | 66 | if logging: 67 | print(('l1 optimization: did not converge within %s iterations' % 68 | max_iters)) 69 | return x 70 | -------------------------------------------------------------------------------- /params.py: -------------------------------------------------------------------------------- 1 | import copy 2 | import json 3 | import random 4 | import hashlib 5 | import numpy as np 6 | 7 | 8 | 9 | class IntrinsicParameters(): 10 | """ Global parameter values for the algorithm """ 11 | 12 | def __init__(self): 13 | 14 | #: if True, print progress to the console 15 | self.logging = False 16 | 17 | #: if True, use a fixed seed for k-means clustering 18 | self.fixed_seed = False 19 | 20 | #: number of iterations for the global loop 21 | self.n_iters = 25 22 | 23 | #: number of iterations for the dense CRF 24 | self.n_crf_iters = 10 25 | 26 | #: if ``True``, split clusters at the end 27 | self.split_clusters = True 28 | 29 | #: Pixels k units apart vertically or horizontally are smoothed. 30 | #: The paper only uses k=1. 31 | self.shading_smooth_k = 1 32 | 33 | #: method used to initialize the shading smoothness term: 34 | #: "none": omit this term for the first iteration 35 | #: "image": use the image itself (intensity channel) 36 | #: "constant": constant 0.5 37 | self.shading_blur_init_method = 'none' 38 | 39 | #: standard deviation for blurring the shading channel 40 | self.shading_blur_sigma = 0.1 41 | 42 | #: exponent by which the blur size decreases each iteration 43 | self.shading_blur_iteration_pow = 1 44 | 45 | #: if ``True``, blur in log space. if ``False``, blur in linear 46 | #: space and then convert to log. 47 | self.shading_blur_log = True 48 | 49 | #: kmeans initialization: weight given to the intensity channel 50 | self.kmeans_intensity_scale = 0.5 51 | 52 | #: kmeans initialization: number of clusters (labels) to use 53 | self.kmeans_n_clusters = 20 54 | 55 | #: kmeans initialization: max pixels to consider at once 56 | #: (if the image has more than this, the image is randomly subsampled) 57 | self.kmeans_max_samples = 2000000 58 | 59 | #: weight of the absolute reflectance prior 60 | self.abs_reflectance_weight = 0 61 | 62 | #: weight of the absolute shading prior 63 | self.abs_shading_weight = 500.0 64 | 65 | #: gray-point of absolute shading term 66 | self.abs_shading_gray_point = 0.5 67 | 68 | #: if ``True``, compute shading error in log space 69 | self.abs_shading_log = True 70 | 71 | #: weight of the shading smoothness unary term 72 | self.shading_target_weight = 20000.0 73 | 74 | #: norm used to penalize shading smoothness deviations 75 | self.shading_target_norm = "L2" 76 | 77 | #: interpret labels as RGB (intensity with chromaticity), thereby 78 | #: penalizing deviations from grayscale in the shading channel (though 79 | #: the final answer is always grayscale anyway) 80 | self.shading_target_chromaticity = False 81 | 82 | #: weight of the chromaticity term: each reflectance intensity is 83 | #: assigned a chromaticity (from the kmeans initialization) and is 84 | #: encouraged to be assigned to image pixels that share the same 85 | #: chromaticity. 86 | self.chromaticity_weight = 0 87 | 88 | #: which norm is used for chromaticity 89 | self.chromaticity_norm = "L1" 90 | 91 | #: compute reflectance distance in log space for the pairwise terms 92 | self.pairwise_intensity_log = True 93 | 94 | #: include chromaticity in pairwise term 95 | self.pairwise_intensity_chromaticity = True 96 | 97 | #: weight of the pairwise term 98 | self.pairwise_weight = 10000.0 99 | 100 | #: bilateral standard deviation: pairwise pixel distance 101 | self.theta_p = 0.1 102 | 103 | #: bilateral standard deviation: intensity 104 | self.theta_l = 0.1 105 | 106 | #: bilateral standard deviation: chromaticity 107 | self.theta_c = 0.025 108 | 109 | #: if True, keep the median of all intensities fixed in stage 2. This 110 | #: doesn't really change much, since the solver is damped anyway. 111 | self.stage2_maintain_median_intensity = True 112 | 113 | #: which norm to use when minimizing shading differences in stage 2 114 | self.stage2_norm = "L1" 115 | 116 | #: if True, interpret labels as RGB instead of intensity 117 | self.stage2_chromaticity = False 118 | 119 | #: parameters to be saved/loaded 120 | ALL_PARAMS = [ 121 | 'n_iters', 122 | 'n_crf_iters', 123 | 'split_clusters', 124 | 'kmeans_n_clusters', 125 | 'kmeans_max_samples', 126 | 'shading_blur_init_method', 127 | 'shading_blur_method', 128 | 'shading_blur_log', 129 | 'shading_blur_sigma', 130 | 'shading_blur_bilateral_sigma_range', 131 | 'shading_blur_iteration_pow', 132 | 'shading_smooth_k', 133 | 'kmeans_intensity_scale', 134 | 'abs_reflectance_weight', 135 | 'abs_shading_log', 136 | 'abs_shading_weight', 137 | 'abs_shading_gray_point', 138 | 'shading_target_weight', 139 | 'shading_target_norm', 140 | 'shading_target_chromaticity', 141 | 'chromaticity_weight', 142 | 'chromaticity_norm', 143 | 'pairwise_intensity_log', 144 | 'pairwise_intensity_chromaticity', 145 | 'pairwise_weight', 146 | 'theta_p', 147 | 'theta_l', 148 | 'theta_c', 149 | 'stage2_norm', 150 | 'stage2_chromaticity', 151 | 'stage2_maintain_median_intensity', 152 | ] 153 | 154 | #: parameters to be adjusted during training 155 | TRAIN_PARAMS = [ 156 | 'n_iters', 157 | #'n_crf_iters', 158 | 159 | 'split_clusters', 160 | 161 | 'kmeans_intensity_scale', 162 | 'kmeans_n_clusters', 163 | 164 | 'shading_blur_init_method', 165 | #'shading_blur_log', 166 | #'pairwise_intensity_log', 167 | 168 | 'shading_blur_sigma', 169 | 'shading_smooth_k', 170 | 171 | 'abs_reflectance_weight', 172 | #'abs_shading_log', 173 | 'abs_shading_weight', 174 | 'abs_shading_gray_point', 175 | 'shading_target_weight', 176 | 'chromaticity_weight', 177 | 'pairwise_weight', 178 | 179 | 'theta_p', 180 | 'theta_l', 181 | 'theta_c', 182 | ] 183 | 184 | #: these parameters are discrete 1-of-N choices 185 | PARAM_CHOICES = { 186 | 'shading_blur_init_method': ( 187 | "none", 188 | "image", 189 | "constant", 190 | ), 191 | } 192 | 193 | #: bounds on paramters 194 | PARAM_BOUNDS = { 195 | 'n_iters': (1, 30), 196 | 'n_crf_iters': (1, 10), 197 | 'shading_blur_sigma': (1e-8, 1.0), 198 | 'shading_smooth_k': (1, 4), 199 | 'kmeans_intensity_scale': (1e-8, 1e10), 200 | 'kmeans_n_clusters': (2, 50), 201 | 'abs_reflectance_weight': (0, 1e10), 202 | 'abs_shading_weight': (0, 1e10), 203 | 'abs_shading_gray_point': (0, 1e10), 204 | 'shading_target_weight': (0, 1e10), 205 | 'chromaticity_weight': (0, 1e10), 206 | 'pairwise_weight': (0, 1e16), 207 | 'theta_p': (1e-8, 1e10), 208 | 'theta_l': (1e-8, 1e10), 209 | 'theta_c': (1e-8, 1e10), 210 | } 211 | 212 | WEIGHT_PARAMS = [ 213 | 'abs_reflectance_weight', 214 | 'abs_shading_weight', 215 | 'shading_target_weight', 216 | 'chromaticity_weight', 217 | 'pairwise_weight', 218 | ] 219 | 220 | THETA_PARAMS = [ 221 | 'theta_p', 222 | 'theta_l', 223 | 'theta_c', 224 | ] 225 | 226 | def to_json(self, indent=4, **extra_kwargs): 227 | """ Convert paramters to a JSON-encoded string """ 228 | obj = {k: getattr(self, k) 229 | for k in IntrinsicParameters.ALL_PARAMS} 230 | if extra_kwargs: 231 | obj.update(extra_kwargs) 232 | return json.dumps(obj, sort_keys=True, indent=indent) 233 | 234 | def __str__(self): 235 | return self.to_json() 236 | 237 | def __unicode__(self): 238 | return self.to_json() 239 | 240 | @staticmethod 241 | def from_file(filename): 242 | """ Load paramers from ``filename`` (in JSON format) """ 243 | return IntrinsicParameters.from_dict(json.load(open(filename))) 244 | 245 | @staticmethod 246 | def from_dict(d): 247 | """ Load paramers from a dictionary """ 248 | ret = IntrinsicParameters() 249 | for k, v in d.items(): 250 | if not k.startswith('_') and k not in IntrinsicParameters.ALL_PARAMS: 251 | raise ValueError("Invalid parameter: %s" % k) 252 | setattr(ret, k, d[k]) 253 | return ret 254 | 255 | def md5(self): 256 | dump = self.to_json() 257 | m = hashlib.md5() 258 | m.update(dump) 259 | return m.hexdigest() 260 | 261 | def save(self, filename, **extra_kwargs): 262 | """ Save paramers to ``filename`` (in JSON format) """ 263 | with open(filename, 'w') as f: 264 | f.write(self.to_json(**extra_kwargs)) 265 | 266 | def clip(self): 267 | """ Clip parameters to be within bounds """ 268 | for k, bounds in IntrinsicParameters.PARAM_BOUNDS.items(): 269 | v = getattr(self, k) 270 | t = type(v) 271 | setattr(self, k, t(np.clip(v, bounds[0], bounds[1]))) 272 | 273 | def random_perterbation( 274 | self, mean_num_params=8, std_delta=0.5, seed=None): 275 | """ Return a new set of parameters with a random perterbation. The 276 | number of variables modified is Poisson-distributed with mean 277 | ``mean_num_params`` , and each changed variable is multiplied by exp(x) 278 | where x is normally distributed with mean 0 and standard deviation 279 | ``std_delta`` """ 280 | 281 | if seed is not None: 282 | random.seed(seed) 283 | np.random.seed(seed) 284 | 285 | # choose a random subset to modify 286 | num_params = len(IntrinsicParameters.TRAIN_PARAMS) 287 | n = np.clip(np.random.poisson(mean_num_params), 1, num_params) 288 | keys = random.sample(IntrinsicParameters.TRAIN_PARAMS, n) 289 | 290 | # modify the subset 291 | ret = copy.deepcopy(self) 292 | for k in keys: 293 | v = getattr(ret, k) 294 | t = type(v) 295 | 296 | if k in IntrinsicParameters.PARAM_CHOICES: 297 | v = random.choice(IntrinsicParameters.PARAM_CHOICES[k]) 298 | elif t == bool: 299 | v = random.choice((False, True)) 300 | else: 301 | v *= np.exp(random.normalvariate(0, std_delta)) 302 | 303 | if t in (int, int): 304 | v = round(v) 305 | setattr(ret, k, t(v)) 306 | 307 | ret.clip() 308 | return ret 309 | -------------------------------------------------------------------------------- /samples/89e1aca9c871a9cd785c918c70fafc6f-r.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/samples/89e1aca9c871a9cd785c918c70fafc6f-r.png -------------------------------------------------------------------------------- /samples/89e1aca9c871a9cd785c918c70fafc6f-s.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/samples/89e1aca9c871a9cd785c918c70fafc6f-s.png -------------------------------------------------------------------------------- /samples/89e1aca9c871a9cd785c918c70fafc6f.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/samples/89e1aca9c871a9cd785c918c70fafc6f.jpg -------------------------------------------------------------------------------- /samples/baku.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/samples/baku.jpeg -------------------------------------------------------------------------------- /samples/bakul-r.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/samples/bakul-r.png -------------------------------------------------------------------------------- /samples/bakul-s.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/samples/bakul-s.png -------------------------------------------------------------------------------- /samples/gh-r.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/samples/gh-r.png -------------------------------------------------------------------------------- /samples/gh.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/samples/gh.jpeg -------------------------------------------------------------------------------- /samples/ghs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DefUs3r/Intrinsic-Image-Decomposition/8d2f27451b3faefebb8388feca83041d7dd8093d/samples/ghs.png -------------------------------------------------------------------------------- /solver.py: -------------------------------------------------------------------------------- 1 | import timeit 2 | import numpy as np 3 | import sklearn 4 | from skimage import morphology 5 | from sklearn.cluster import MiniBatchKMeans 6 | 7 | from params import IntrinsicParameters 8 | from decomposition import IntrinsicDecomposition 9 | from energy import IntrinsicEnergy 10 | from optimization import minimize_l1, minimize_l2 11 | 12 | from krahenbuhl2013.krahenbuhl2013 import DenseCRF 13 | # from pydensecrf.densecrf import DenseCRF2D as DenseCRF 14 | 15 | # from numba import vectorize 16 | 17 | 18 | class IntrinsicSolver(object): 19 | 20 | def __init__(self, input, params): 21 | """ Create a new solver with given input and parameters. Nothing 22 | happens until you call ``solve``. """ 23 | 24 | if isinstance(params, dict): 25 | params = IntrinsicParameters.from_dict(params) 26 | 27 | self.params = params 28 | self.input = input 29 | self.energy = IntrinsicEnergy(self.input, params) 30 | 31 | def solve(self): 32 | """ Perform all steps. """ 33 | 34 | if self.params.logging: 35 | t0 = timeit.default_timer() 36 | print("solve...") 37 | 38 | # Initialize 39 | self.decomposition = IntrinsicDecomposition(self.params, self.input) 40 | self.decomposition_history = [] 41 | self.initialize_intensities() 42 | 43 | for i in range(self.params.n_iters): 44 | if self.params.logging: 45 | print(("\nrun: starting iteration %s/%s" % (i+1, self.params.n_iters))) 46 | self.decomposition.iter_num = i 47 | 48 | # STAGE 1 49 | self.decomposition.stage_num = 1 50 | self.stage1_optimize_r() 51 | self.remove_unused_intensities() 52 | self.decomposition_history.append(self.decomposition.copy()) 53 | 54 | if self.decomposition.intensities.shape[0] <= 1: 55 | if self.params.logging: 56 | print("Warning: only 1 reflectance -- exit early") 57 | break 58 | 59 | # STAGE 2 60 | self.decomposition.stage_num = 2 61 | if self.params.split_clusters and i == self.params.n_iters - 1: 62 | self.split_label_clusters() 63 | self.stage2_smooth_s() 64 | self.decomposition_history.append(self.decomposition.copy()) 65 | 66 | # prepare final solution 67 | r, s = self.decomposition.get_r_s() 68 | 69 | if self.params.logging: 70 | t1 = timeit.default_timer() 71 | print(("solve (%s s)" % (t1 - t0))) 72 | 73 | return r, s, self.decomposition 74 | 75 | 76 | def prev_decomposition(self): 77 | """ Return the previous decomposition (used to compute the blurred 78 | shading target). """ 79 | 80 | if self.decomposition_history: 81 | return self.decomposition_history[-1] 82 | else: 83 | return None 84 | 85 | def initialize_intensities(self): 86 | """ Initialization: k-means of the input image """ 87 | 88 | if self.params.logging: 89 | t0 = timeit.default_timer() 90 | print(("initialization: k-means clustering with %s centers..." % 91 | self.params.kmeans_n_clusters)) 92 | 93 | image_irg = self.input.image_irg 94 | mask_nz = self.input.mask_nz 95 | 96 | if self.params.fixed_seed: 97 | # fix the seed when computing things like gradients across 98 | # hyperparameters 99 | random_state = np.random.RandomState(seed=59173) 100 | else: 101 | random_state = None 102 | 103 | samples = image_irg[mask_nz[0], mask_nz[1], :] 104 | if samples.shape[0] > self.params.kmeans_max_samples: 105 | print(("image is large: subsampling %s/%s random pixels" % 106 | (self.params.kmeans_max_samples, samples.shape[0]))) 107 | samples = sklearn.utils \ 108 | .shuffle(samples)[:self.params.kmeans_max_samples, :] 109 | samples[:, 0] *= self.params.kmeans_intensity_scale 110 | 111 | kmeans = MiniBatchKMeans( 112 | n_clusters=self.params.kmeans_n_clusters, 113 | compute_labels=True, random_state=random_state) 114 | kmeans.fit(samples) 115 | assert self.params.kmeans_intensity_scale > 0 116 | self.decomposition.intensities = ( 117 | kmeans.cluster_centers_[:, 0] / 118 | self.params.kmeans_intensity_scale 119 | ) 120 | self.decomposition.chromaticities = ( 121 | kmeans.cluster_centers_[:, 1:3] 122 | ) 123 | if self.params.logging: 124 | t1 = timeit.default_timer() 125 | print(("clustering done (%s s). intensities:\n%s" % 126 | (t1 - t0, self.decomposition.intensities))) 127 | 128 | 129 | def stage1_optimize_r(self): 130 | """ Stage 1: dense CRF optimization """ 131 | 132 | if self.params.logging: 133 | t0 = timeit.default_timer() 134 | print("stage1_optimize_r: compute costs...") 135 | 136 | nlabels = self.decomposition.intensities.shape[0] 137 | npixels = self.input.mask_nnz 138 | 139 | # use a Python wrapper around the code from [Krahenbuhl et al 2013] 140 | densecrf = DenseCRF(npixels, nlabels) 141 | 142 | # unary costs 143 | unary_costs = self.energy.compute_unary_costs( 144 | decomposition=self.decomposition, 145 | prev_decomposition=self.prev_decomposition(), 146 | ) 147 | densecrf.set_unary_energy(unary_costs) 148 | 149 | # pairwise costs 150 | if self.params.pairwise_weight: 151 | pairwise_costs = self.energy.compute_pairwise_costs( 152 | decomposition=self.decomposition, 153 | ) 154 | densecrf.add_pairwise_energy( 155 | pairwise_costs=(self.params.pairwise_weight * pairwise_costs).astype(np.float32), 156 | features=self.energy.get_features().copy(), 157 | ) 158 | 159 | if self.params.logging: 160 | print(("stage1_optimize_r: optimizing dense crf (%s iters)..." % 161 | self.params.n_crf_iters)) 162 | t0crf = timeit.default_timer() 163 | 164 | # maximum aposteriori labeling ("x" variable in the paper) 165 | self.decomposition.labels_nz = densecrf.map(self.params.n_crf_iters) 166 | 167 | if self.params.logging: 168 | t1crf = timeit.default_timer() 169 | print(("stage1_optimize_r: dense crf done (%s s)" % (t1crf - t0crf))) 170 | 171 | if self.params.logging: 172 | t1 = timeit.default_timer() 173 | print(("stage1_optimize_r: done (%s s)" % (t1 - t0))) 174 | 175 | def stage2_smooth_s(self): 176 | """ Stage 2: L1 shading smoothness """ 177 | 178 | if self.params.logging: 179 | t0 = timeit.default_timer() 180 | print('stage2_smooth_s: constructing linear system...') 181 | 182 | if self.params.stage2_maintain_median_intensity: 183 | median_intensity = np.median(self.decomposition.intensities) 184 | 185 | log_intensities = np.log(self.decomposition.intensities) 186 | 187 | # the 'A' matrix (in Ax = b) is in CSR sparse format 188 | A_data, A_rows, A_cols, A_shape, b = \ 189 | self.construct_shading_smoothness_system(log_intensities) 190 | 191 | if len(b) < 1: 192 | if self.params.logging: 193 | print(('Warning: empty linear system (%s, nlabels=%s)' % ( 194 | self.basename, self.cur_intensities.shape[0]))) 195 | return 196 | 197 | if self.params.logging: 198 | print('solving linear system...') 199 | 200 | # solve for the change to the variables, so that we can slightly 201 | # regularize the variables to be near zero (i.e. near the previous 202 | # value). 203 | if self.params.stage2_norm == "L1": 204 | minimize = minimize_l1 205 | elif self.params.stage2_norm == "L2": 206 | minimize = minimize_l2 207 | else: 208 | raise ValueError("Invalid stage2_norm: %s" % self.params.stage2_norm) 209 | delta_intensities = minimize( 210 | A_data, A_rows, A_cols, A_shape, b, 211 | damp=1e-8, logging=self.params.logging, 212 | ) 213 | intensities = np.exp(log_intensities + delta_intensities) 214 | 215 | if self.params.stage2_maintain_median_intensity: 216 | # Since there's a scale ambiguity and stage1 includes a term that 217 | # depends on absolute shading, keep the median intensity constant. 218 | # This is a pretty small adjustment. 219 | intensities *= median_intensity / np.median(intensities) 220 | 221 | self.decomposition.intensities = intensities 222 | 223 | if self.params.logging: 224 | t1 = timeit.default_timer() 225 | print(('stage2_smooth_s: done (%s s)' % (t1 - t0))) 226 | 227 | def construct_shading_smoothness_system(self, log_intensities): 228 | """ Create a sparse matrix (CSR format) to minimize discontinuities in 229 | the shading channel (by adjusting ``decomposition.intensities``). 230 | 231 | :return: A_data, A_rows, A_cols, A_shape, b 232 | """ 233 | 234 | rows, cols = self.input.shape[0:2] 235 | mask = self.input.mask 236 | labels = self.decomposition.get_labels() 237 | 238 | if self.params.stage2_chromaticity: 239 | # labels represent RGB colors (but we are still only adjusting 240 | # intensity) 241 | log_image_rgb = self.input.log_image_rgb 242 | log_reflectances_rgb = np.log(np.clip(self.decomposition.get_reflectances_rgb(), 1e-5, np.inf)) 243 | else: 244 | # labels represent intensities 245 | log_image_gray = self.input.log_image_gray 246 | 247 | A_rows = [] 248 | A_cols = [] 249 | A_data = [] 250 | b = [] 251 | 252 | # Note that in the paper, params.shading_smooth_k = 1, i.e. only the 253 | # immediate pixel neighbors are smoothed. This code is slightly more 254 | # general in that it allows to smooth pixels k units away if you set 255 | # shading_smooth_k > 1, weighted by 1/(k*k). 256 | for k in range(1, self.params.shading_smooth_k + 1): 257 | weight = 1.0 / (k * k) 258 | for i in range(rows - k): 259 | for j in range(cols - k): 260 | if not mask[i, j]: 261 | continue 262 | if mask[i + k, j]: 263 | l0 = labels[i, j] 264 | l1 = labels[i + k, j] 265 | if l0 != l1: 266 | if self.params.stage2_chromaticity: 267 | # RGB interpretation 268 | for c in range(3): 269 | A_rows.append(len(b)) 270 | A_cols.append(l0) 271 | A_data.append(weight) 272 | A_rows.append(len(b)) 273 | A_cols.append(l1) 274 | A_data.append(-weight) 275 | bval = (log_image_rgb[i, j, c] - 276 | log_image_rgb[i + k, j, c] + 277 | log_reflectances_rgb[l1, c] - 278 | log_reflectances_rgb[l0, c]) 279 | b.append(weight * bval) 280 | else: 281 | # intensity interpretation 282 | A_rows.append(len(b)) 283 | A_cols.append(l0) 284 | A_data.append(weight) 285 | A_rows.append(len(b)) 286 | A_cols.append(l1) 287 | A_data.append(-weight) 288 | bval = (log_image_gray[i, j] - 289 | log_image_gray[i + k, j] + 290 | log_intensities[l1] - 291 | log_intensities[l0]) 292 | b.append(weight * bval) 293 | if mask[i, j + k]: 294 | l0 = labels[i, j] 295 | l1 = labels[i, j + k] 296 | if l0 != l1: 297 | if self.params.stage2_chromaticity: 298 | # RGB interpretation 299 | for c in range(3): 300 | A_rows.append(len(b)) 301 | A_cols.append(l0) 302 | A_data.append(weight) 303 | A_rows.append(len(b)) 304 | A_cols.append(l1) 305 | A_data.append(-weight) 306 | bval = (log_image_rgb[i, j, c] - 307 | log_image_rgb[i, j + k, c] + 308 | log_reflectances_rgb[l1, c] - 309 | log_reflectances_rgb[l0, c]) 310 | b.append(weight * bval) 311 | else: 312 | # intensity interpretation 313 | A_rows.append(len(b)) 314 | A_cols.append(l0) 315 | A_data.append(weight) 316 | A_rows.append(len(b)) 317 | A_cols.append(l1) 318 | A_data.append(-weight) 319 | bval = (log_image_gray[i, j] - 320 | log_image_gray[i, j + k] + 321 | log_intensities[l1] - 322 | log_intensities[l0]) 323 | b.append(weight * bval) 324 | 325 | A_shape = (len(b), log_intensities.shape[0]) 326 | return ( 327 | np.array(A_data), 328 | np.array(A_rows), 329 | np.array(A_cols), 330 | A_shape, 331 | np.array(b, dtype=np.float32) 332 | ) 333 | 334 | def remove_unused_intensities(self): 335 | """ Remove any intensities that are not currently assigned to a pixel, 336 | and then re-number all labels so they are contiguous again. """ 337 | 338 | if self.params.logging: 339 | prev_r_s = self.decomposition.get_r_s() 340 | 341 | labels_nz = self.decomposition.labels_nz 342 | intensities = self.decomposition.intensities 343 | chromaticities = self.decomposition.chromaticities 344 | nlabels = intensities.shape[0] 345 | 346 | new_to_old = np.nonzero(np.bincount( 347 | labels_nz, minlength=nlabels))[0] 348 | old_to_new = np.empty(nlabels, dtype=np.int32) 349 | old_to_new.fill(-1) 350 | for new, old in enumerate(new_to_old): 351 | old_to_new[old] = new 352 | 353 | self.decomposition.labels_nz = old_to_new[labels_nz] 354 | self.decomposition.intensities = intensities[new_to_old] 355 | self.decomposition.chromaticities = chromaticities[new_to_old] 356 | 357 | if self.params.logging: 358 | print(('remove_unused_intensities: %s/%s labels kept' % ( 359 | len(self.decomposition.intensities), len(intensities)))) 360 | 361 | if self.params.logging: 362 | np.testing.assert_equal(self.decomposition.get_r_s(), prev_r_s) 363 | assert (self.decomposition.chromaticities.shape[0] == 364 | self.decomposition.intensities.shape[0]) 365 | 366 | def split_label_clusters(self, neighbors=4): 367 | """ Expand the set of labels by looking at each connected component in 368 | the labels. Assign each component a new label number, and copy its old 369 | intensity value to its new label. This typically expands the number of 370 | labels from ~30 to ~3000, so you should only really do it on the last 371 | iteration. """ 372 | 373 | if self.params.logging: 374 | prev_r_s = self.decomposition.get_r_s() 375 | 376 | rows, cols = self.input.shape[0:2] 377 | labels = self.decomposition.get_labels() 378 | intensities = self.decomposition.intensities 379 | chromaticities = self.decomposition.chromaticities 380 | 381 | # split labels 382 | new_labels = morphology.label(labels, neighbors=neighbors) 383 | 384 | # map labels 385 | self.decomposition.labels_nz = new_labels[self.input.mask_nz] 386 | 387 | # map intensities 388 | _, indices = np.unique(new_labels.ravel(), return_index=True) 389 | new_to_old = labels.ravel()[indices] 390 | new_to_old = new_to_old[new_to_old != -1] 391 | self.decomposition.intensities = intensities[new_to_old] 392 | self.decomposition.chromaticities = chromaticities[new_to_old] 393 | 394 | if self.params.logging: 395 | print(('split_label_clusters: %s --> %s' % ( 396 | intensities.shape[0], self.decomposition.intensities.shape[0]))) 397 | 398 | self.remove_unused_intensities() 399 | 400 | if self.params.logging: 401 | np.testing.assert_equal(self.decomposition.get_r_s(), prev_r_s) 402 | assert (self.decomposition.chromaticities.shape[0] == 403 | self.decomposition.intensities.shape[0]) 404 | --------------------------------------------------------------------------------