├── LICENSE.md ├── README.md ├── _config.yml ├── images ├── imageGT.png ├── imageSR.png ├── inputImage_interpolated.png ├── peppers256.png └── peppers256.tif └── src ├── SuperResolution.cpp └── SuperResolution.h /LICENSE.md: -------------------------------------------------------------------------------- 1 | Copyright (c) 2011-2017 GitHub Inc. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 4 | 5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 6 | 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Single Image Example-based Super-Resolution 2 | 3 | ## Description 4 | 5 | Performs example-based Super-resolution on an image using direct mapping between high and low resolution patches. 6 | 7 | Learning is done utilizing a self-example high-resolution, low-resolution coupled dictionary. Self-example means the dictionary is created from sampling patches from the input low resolution image. This approach is based on the observation that similar patches of a natural image occur within the same scale and between different scales. 8 | 9 | An affine mapping approach is learned to transform a low resolution patch to its high resolution counterpart. This problem is ill-posed and hence we add a regularization term based on Tikhonov regularization. 10 | 11 | We take patches of the image in the wavelet domain to perform training and prediction. We expand the discrete wavelet basis by using a discrete rational wavelet transform. Results suggest that the increased sparsity and the larger basis help to improve the ill-posed nature of the problem. 12 | 13 | Super-resolution is performed through gradual upscaling. This method better retains low resolution patch structures and outperforms single step upscaling at higher upscaling rates. 14 | 15 | 16 | ## API 17 | 18 | ### gs::superResolution 19 | 20 | cv::Mat* **gs::superResolution**(cv::Mat inputImage, float upscale, int iterations, double lambda, unsigned int patchSize 21 | unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight); 22 | 23 | > gs::superResolution uses example patches from the input low resolution image to perform super-resolution. 24 | 25 | #### Paramaters 26 | **cv::Mat inputImage**: *Input low resolution image.* 27 | 28 | **float upscale**: *Upscaling factor. Must be greater than 1.* 29 | 30 | **int iterations**: *Number of upscaling iterations performed. The final product is upscaled by a factor of 'upscale'.* 31 | 32 | **double lambda**: *Regularization weight.* 33 | 34 | **unsigned int patchSize**: *Patch size.* 35 | 36 | **unsigned int patchOverlap**: *Patch overlap.* 37 | 38 | **unsigned int neighborhoodSize**: *K-nearest neighbors used.* 39 | 40 | **double neighborhoodWeight**: *K-nearest neighbor scaling factor. Dictionary atoms are scaled based on the atom's distance from a low resolution patch.* 41 | 42 | 43 | ### gs::superResolutionWavelet 44 | 45 | cv::Mat* **gs::superResolutionWavelet**(cv::Mat inputImage, float upscale, int iterations, double lambda, unsigned int patchSize 46 | unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight, int waveletP, int waveletQ); 47 | 48 | > gs::superResolutionWavelet uses example wavelet patches from the input low resolution image to perform super-resolution. 49 | 50 | #### Paramaters 51 | **cv::Mat inputImage**: *Input low resolution image.* 52 | 53 | **float upscale**: *Upscaling factor. Must be greater than 1.* 54 | 55 | **int iterations**: *Number of upscaling iterations performed. The final product is upscaled by a factor of 'upscale'.* 56 | 57 | **double lambda**: *Regularization weight.* 58 | 59 | **unsigned int patchSize**: *Patch size.* 60 | 61 | **unsigned int patchOverlap**: *Patch overlap.* 62 | 63 | **unsigned int neighborhoodSize**: *K-nearest neighbors used.* 64 | 65 | **double neighborhoodWeight**: *K-nearest neighbor scaling factor. Dictionary atoms are scaled based on the atom's distance from a low resolution patch.* 66 | 67 | **int waveletP**: *upsampling wavelet factor.* 68 | 69 | **int waveletQ**: *downsampling wavelet factor. WaveletQ must be larger than waveletP.* 70 | 71 | ## Example 72 | 73 | ``` 74 | #include "SuperResolution.h" 75 | #define LR_IMAGE_PATH "../images/peppers256.tif" 76 | #define GT_IMAGE_PATH "../images/peppers.png" 77 | 78 | int main() 79 | { 80 | 81 | cv::Mat imageLR; 82 | cv::Mat imageGT; 83 | 84 | /*load the test images*/ 85 | cv::Mat imageLR = imread(LR_IMAGE_PATH, CV_LOAD_IMAGE_GRAYSCALE); 86 | imageLR = gs::to32F(imageLR); 87 | imageGT = imread(GT_IMAGE_PATH, CV_LOAD_IMAGE_GRAYSCALE); 88 | imageGT = gs::to32F(imageGT); 89 | 90 | /*SR variables*/ 91 | float upscale = 2.0; 92 | int iterations = 1; 93 | unsigned int patchSize = 4; 94 | unsigned int patchOverlap = 3; 95 | double lambda = 1e-12; 96 | unsigned int neighborhoodSize = 200; 97 | double neighborhoodWeight = 1.0; 98 | 99 | /*Super-resolve the image*/ 100 | Mat* imageSR = gs::superResolve(imageLR, upscale, iterations, lambda, patchSize, patchOverlap, neighborhoodSize, neighborhoodWeight); 101 | /*export the report and images*/ 102 | gs::exportReport(imageLR, *imageSR, imageGT, upscale, iterations, lambda, patchSize, patchOverlap, neighborhoodSize, neighborhoodWeight); 103 | 104 | int waveletP = 7; 105 | int waveletQ = 8; 106 | neighborhoodWeight = 4.0; 107 | /*Super-resolve the image*/ 108 | Mat* imageSRWavelet = gs::superResolveWavelet(imageLR, upscale, iterations, lambda, patchSize, patchOverlap, neighborhoodSize, neighborhoodWeight, waveletP, waveletQ); 109 | /*export the report and images*/ 110 | gs::exportReportWavelet(imageLR, *s, imageGT, 2, 1, lambda, patchSize, patchOverlap, neighborhoodSize, neighborhoodWeight,waveletP,waveletQ); 111 | } 112 | ``` 113 | 114 | ## Results 115 | 116 | ###Input Image: 117 | 118 | ![input image](https://raw.githubusercontent.com/Gregjksmith/Single-Image-Example-Based-Super-Resolution/master/images/peppers256.png?raw=true) 119 | 120 | ###Super-resolved: 121 | 122 | ![super-resolved](https://github.com/Gregjksmith/Single-Image-Example-Based-Super-Resolution/blob/master/images/imageSR.png?raw=true) 123 | 124 | ###Bicubic Interpolation: 125 | 126 | ![bicubic interpolation](https://raw.githubusercontent.com/Gregjksmith/Single-Image-Example-Based-Super-Resolution/master/images/inputImage_interpolated.png?raw=true) 127 | 128 | ###Ground Truth: 129 | 130 | ![ground truth](https://raw.githubusercontent.com/Gregjksmith/Single-Image-Example-Based-Super-Resolution/master/images/imageGT.png?raw=true) 131 | 132 | ## License 133 | 134 | This project is licensed under the terms of the MIT license. 135 | 136 | [1] Bevilacqua, Marco, et al. "Single-image super-resolution via linear mapping of interpolated self-examples." IEEE Transactions on Image Processing 23.12 (2014): 5334-5347. -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-minimal -------------------------------------------------------------------------------- /images/imageGT.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gregjksmith/Single-Image-Example-Based-Super-Resolution/41d695bf46ad0f4842446d438649f34a9bea69fe/images/imageGT.png -------------------------------------------------------------------------------- /images/imageSR.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gregjksmith/Single-Image-Example-Based-Super-Resolution/41d695bf46ad0f4842446d438649f34a9bea69fe/images/imageSR.png -------------------------------------------------------------------------------- /images/inputImage_interpolated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gregjksmith/Single-Image-Example-Based-Super-Resolution/41d695bf46ad0f4842446d438649f34a9bea69fe/images/inputImage_interpolated.png -------------------------------------------------------------------------------- /images/peppers256.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gregjksmith/Single-Image-Example-Based-Super-Resolution/41d695bf46ad0f4842446d438649f34a9bea69fe/images/peppers256.png -------------------------------------------------------------------------------- /images/peppers256.tif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Gregjksmith/Single-Image-Example-Based-Super-Resolution/41d695bf46ad0f4842446d438649f34a9bea69fe/images/peppers256.tif -------------------------------------------------------------------------------- /src/SuperResolution.cpp: -------------------------------------------------------------------------------- 1 | #include "SuperResolution.h" 2 | 3 | #define FILTER_SIZE 151 4 | using namespace cv; 5 | 6 | gs::RationalWavelet::RationalWavelet(cv::Mat image, int J, int p, int q, int s) 7 | { 8 | /*create the filter banks*/ 9 | cv::Mat H, G, h, g; 10 | makeFreqResp(FILTER_SIZE, 1, (float)q / (float)p, s, H, G, h, g); 11 | this->J = J; 12 | this->p = p; 13 | this->q = q; 14 | this->s = s; 15 | 16 | /*init the wavelet array*/ 17 | wavelet = new Wavelet*[J]; 18 | cv::Mat im1 = image; 19 | 20 | for (int j = 0; j < J; j++) 21 | { 22 | wavelet[j] = new Wavelet(); 23 | 24 | int N = image.rows; 25 | int M = image.cols; 26 | Size imSize = cv::Size(N, M); 27 | int c = lcm(q, s); 28 | Size l; 29 | l.width = c*ceil(imSize.width / c); 30 | l.height = c*ceil(imSize.height / c); 31 | cv::Mat xp; 32 | /*resize the image such that the resulting image can be evenly downsampled*/ 33 | resize(image, xp, l, 0.0, 0.0, CV_INTER_CUBIC); 34 | 35 | cv::Mat lo; 36 | /*downsample the image im1, and compute the wavelets*/ 37 | afb(&im1, &lo, wavelet[j], &h, &g, p, q, s); 38 | 39 | im1 = lo; 40 | } 41 | 42 | this->lowband = im1; 43 | } 44 | 45 | gs::RationalWavelet::~RationalWavelet() 46 | { 47 | for (int i = 0; i < J; i++) 48 | { 49 | delete wavelet[i]; 50 | } 51 | 52 | } 53 | 54 | Mat gs::RationalWavelet::inverseRationalWavelet() 55 | { 56 | /*create the filter banks*/ 57 | cv::Mat H, G, h, g; 58 | makeFreqResp(FILTER_SIZE, 1, (float)q / (float)p, s, H, G, h, g); 59 | 60 | /* compute the conjugate filters*/ 61 | cv::Mat hconj(1, FILTER_SIZE, CV_32F); 62 | cv::Mat gconj(1, FILTER_SIZE, CV_32F); 63 | for (int i = 0; i < FILTER_SIZE; i++) 64 | { 65 | hconj.at(0, i) = h.at(0, FILTER_SIZE - 1 - i); 66 | gconj.at(0, i) = g.at(0, FILTER_SIZE - 1 - i); 67 | } 68 | 69 | cv::Mat lo = this->lowband; 70 | for (int i = J - 1; i >= 0; i--) 71 | { 72 | Wavelet* hi = this->wavelet[i]; 73 | /*iterative reconstruct the image*/ 74 | sfb(&lo, hi, &hconj, &gconj, p, q, s); 75 | } 76 | return lo; 77 | } 78 | 79 | void gs::RationalWavelet::afb(cv::Mat* x, cv::Mat* lo, Wavelet* w, cv::Mat* h, cv::Mat* g, int p, int q, int s) 80 | { 81 | Size N(ceil(p*x->rows / q), ceil(p*x->cols / q)); 82 | cv::Mat ht; 83 | transpose(*h, ht); 84 | cv::Mat gt; 85 | transpose(*g, gt); 86 | 87 | /*anti alias the input image*/ 88 | filter2D(*x, *lo, -1, *h, Point(-1, -1), 0.0, BORDER_REPLICATE); 89 | filter2D(*lo, *lo, -1, ht, Point(-1, -1), 0.0, BORDER_REPLICATE); 90 | /*down sample the image*/ 91 | resize(*lo, *lo, N, 0.0, 0.0, CV_INTER_CUBIC); 92 | 93 | /*filter the wavelet LH*/ 94 | filter2D(*x, w->highbandLH, -1, ht, Point(-1, -1), 0.0, BORDER_REPLICATE); 95 | filter2D(w->highbandLH, w->highbandLH, -1, *g, Point(-1, -1), 0.0, BORDER_REPLICATE); 96 | 97 | /*filter the wavelet HL*/ 98 | filter2D(*x, w->highbandHL, -1, gt, Point(-1, -1), 0.0, BORDER_REPLICATE); 99 | filter2D(w->highbandHL, w->highbandHL, -1, *h, Point(-1, -1), 0.0, BORDER_REPLICATE); 100 | 101 | /*filter the wavelet HH*/ 102 | filter2D(*x, w->highbandHH, -1, *g, Point(-1, -1), 0.0, BORDER_REPLICATE); 103 | filter2D(w->highbandHH, w->highbandHH, -1, gt, Point(-1, -1), 0.0, BORDER_REPLICATE); 104 | } 105 | 106 | void gs::RationalWavelet::sfb(cv::Mat* lo, Wavelet* hi, cv::Mat* h, cv::Mat* g, int p, int q, int s) 107 | { 108 | cv::Mat ht; 109 | transpose(*h, ht); 110 | cv::Mat gt; 111 | transpose(*g, gt); 112 | Size N(hi->highbandHH.rows, hi->highbandHH.cols); 113 | /*upsacle the image*/ 114 | resize(*lo, *lo, N, 0.0, 0.0, CV_INTER_CUBIC); 115 | /*anti alias*/ 116 | filter2D(*lo, *lo, -1, *h, Point(-1, -1), 0.0, BORDER_REPLICATE); 117 | filter2D(*lo, *lo, -1, ht, Point(-1, -1), 0.0, BORDER_REPLICATE); 118 | 119 | cv::Mat hiHL; 120 | cv::Mat hiLH; 121 | cv::Mat hiHH; 122 | 123 | /*filter the wavelets*/ 124 | filter2D(hi->highbandHL, hiHL, -1, *g, Point(-1, -1), 0.0, BORDER_REPLICATE); 125 | filter2D(hiHL, hiHL, -1, ht, Point(-1, -1), 0.0, BORDER_REPLICATE); 126 | 127 | filter2D(hi->highbandLH, hiLH, -1, *h, Point(-1, -1), 0.0, BORDER_REPLICATE); 128 | filter2D(hiLH, hiLH, -1, gt, Point(-1, -1), 0.0, BORDER_REPLICATE); 129 | 130 | filter2D(hi->highbandHH, hiHH, -1, *g, Point(-1, -1), 0.0, BORDER_REPLICATE); 131 | filter2D(hiHH, hiHH, -1, gt, Point(-1, -1), 0.0, BORDER_REPLICATE); 132 | 133 | //imwrite("../images/testConjFilter0.png", to8U(hiLH)); 134 | //imwrite("../images/testConjFilter1.png", to8U(hiHL)); 135 | //imwrite("../images/testConjFilter2.png", to8U(hiHH)); 136 | 137 | /*reconstruct*/ 138 | *lo = *lo + hiHL + hiLH + hiHH; 139 | //*lo = *lo + hi->highbandHL + hi->highbandLH + hi->highbandHH; 140 | } 141 | 142 | int gs::RationalWavelet::gcd(int a, int b) 143 | { 144 | int c; 145 | while (a != 0) 146 | { 147 | c = a; 148 | a = b%a; 149 | b = c; 150 | } 151 | return b; 152 | } 153 | int gs::RationalWavelet::lcm(int a, int b) 154 | { 155 | return a*(b / gcd(a, b)); 156 | } 157 | 158 | void gs::makeFreqResp(int filterSize, float p, float q, float s, cv::Mat& H, cv::Mat& G, cv::Mat& h, cv::Mat& g) 159 | { 160 | /*init filters*/ 161 | H = cv::Mat::zeros(cv::Size(1, filterSize), CV_32F); 162 | G = cv::Mat::zeros(cv::Size(1, filterSize), CV_32F); 163 | h = cv::Mat::zeros(cv::Size(1, filterSize), CV_32F); 164 | g = cv::Mat::zeros(cv::Size(1, filterSize), CV_32F); 165 | 166 | float pi = 3.14159265359; 167 | float wp = ((float)s - 1.0)*pi / s; 168 | float ws = pi / q; 169 | 170 | float* w; 171 | int wSize; 172 | wSize = filterSize; 173 | 174 | /*create sample array, w*/ 175 | w = new float[wSize]; 176 | for (int i = 0; i < wSize; i++) 177 | { 178 | w[i] = 2.0*pi*(float)i / float(filterSize); 179 | } 180 | 181 | /* 182 | create kPass, pass band magnitude for the low pass filter. 183 | create kStop, stop band magnitude for the low pass filter. 184 | create kTrans, stop band magnitude for the low pass filter. 185 | */ 186 | float* kPass = new float[wSize]; 187 | float* kStop = new float[wSize]; 188 | float* kTrans = new float[wSize]; 189 | for (int i = 0; i < wSize; i++) 190 | { 191 | if (abs(w[i]) <= wp) 192 | kPass[i] = 1.0; 193 | else 194 | kPass[i] = 0.0; 195 | 196 | if (abs(w[i]) >= ws) 197 | kStop[i] = 1.0; 198 | else 199 | kStop[i] = 0.0; 200 | 201 | if (abs(w[i]) > wp && abs(w[i]) < ws) 202 | kTrans[i] = 1.0; 203 | else 204 | kTrans[i] = 0.0; 205 | } 206 | 207 | float a = (1 - 1.0 / s)*pi; 208 | float b = (float)p / (float)q - (1.0 - 1.0 / s); 209 | float* wScaled = new float[wSize]; 210 | for (int i = 0; i < wSize; i++) 211 | { 212 | wScaled[i] = (abs(w[i]) - a) / b; 213 | } 214 | 215 | 216 | /*create the low pass magnitude response*/ 217 | for (int i = 0; i < wSize; i++) 218 | { 219 | if (kPass[i] == 1.0) 220 | { 221 | H.at(0, i) = 1.0; 222 | } 223 | else if (kTrans[i] == 1.0) 224 | { 225 | float d; 226 | d = (1.0 + cos(wScaled[i]))*sqrt(2.0 - cos(wScaled[i])) / 2; 227 | H.at(0, i) = d; 228 | } 229 | else 230 | { 231 | H.at(0, i) = 0.0; 232 | } 233 | } 234 | 235 | 236 | if ((filterSize % 2) == 0) 237 | { 238 | H.at(0, wSize) = 0.0; 239 | } 240 | 241 | /*copy and padd the dft with zeros for imag components, set up for the idft*/ 242 | cv::Mat Hpadd = cv::Mat::zeros(cv::Size(1, filterSize), CV_32F); 243 | int filterMid = floor(filterSize / 2); 244 | Hpadd.at(0, 0) = H.at(0, 0); 245 | for (int i = 0; i < filterMid; i++) 246 | { 247 | Hpadd.at(0, i * 2 + 1) = H.at(0, i + 1); 248 | } 249 | 250 | /* 251 | create kPass, pass band magnitude for the high pass filter. 252 | create kStop, stop band magnitude for the high pass filter. 253 | create kTrans, stop band magnitude for the high pass filter. 254 | */ 255 | ws = ((float)s - 1.0)*pi / (float)s; 256 | wp = (float)p*pi / q; 257 | for (int i = 0; i < wSize; i++) 258 | { 259 | if (abs(w[i]) >= wp) 260 | kPass[i] = 1.0; 261 | else 262 | kPass[i] = 0.0; 263 | 264 | if (abs(w[i]) <= ws) 265 | kStop[i] = 1.0; 266 | else 267 | kStop[i] = 0.0; 268 | 269 | if (abs(w[i]) < wp && abs(w[i]) > ws) 270 | kTrans[i] = 1.0; 271 | else 272 | kTrans[i] = 0.0; 273 | } 274 | a = (1.0 - 1.0 / s)*pi; 275 | b = (float)p / q - (1.0 - 1.0 / s); 276 | for (int i = 0; i < wSize; i++) 277 | { 278 | wScaled[i] = (abs(w[i]) - a) / b; 279 | } 280 | 281 | /*create the high pass magnitude response*/ 282 | for (int i = 0; i < wSize; i++) 283 | { 284 | if (kPass[i] == 1.0) 285 | { 286 | G.at(0, i) = 1.0; 287 | } 288 | else if (kTrans[i] == 1.0) 289 | { 290 | float d; 291 | d = (1.0 - cos(wScaled[i]))*sqrt(2.0 + cos(wScaled[i])) / 2; 292 | G.at(0, i) = d; 293 | } 294 | else 295 | { 296 | G.at(0, i) = 0.0; 297 | } 298 | } 299 | 300 | if ((filterSize % 2) == 0) 301 | { 302 | G.at(0, wSize) = 1.0; 303 | } 304 | 305 | /*copy and padd the dft with zeros for imag components, set up for the idft*/ 306 | cv::Mat Gpadd = cv::Mat::zeros(cv::Size(1, filterSize), CV_32F); 307 | Gpadd.at(0, 0) = G.at(0, 0); 308 | for (int i = 0; i < filterMid; i++) 309 | { 310 | Gpadd.at(0, i * 2 + 1) = G.at(0, i + 1); 311 | } 312 | Gpadd.at(0, filterSize) = G.at(0, filterMid); 313 | cv::Mat Hifft; 314 | cv::Mat Gifft; 315 | 316 | /*take the idft*/ 317 | cv::idft(Hpadd, Hifft, cv::DFT_SCALE); 318 | cv::idft(Gpadd, Gifft, cv::DFT_SCALE); 319 | 320 | /*shift and center the filters*/ 321 | for (int i = 0; i < filterMid + 1; i++) 322 | { 323 | h.at(0, filterMid + i) = Hifft.at(0, i); 324 | g.at(0, filterMid + i) = Gifft.at(0, i); 325 | } 326 | for (int i = 0; i < filterMid; i++) 327 | { 328 | h.at(0, i) = Hifft.at(0, i + filterMid + 1); 329 | g.at(0, i) = Gifft.at(0, i + filterMid + 1); 330 | } 331 | 332 | /*clean up*/ 333 | delete kPass; 334 | delete kStop; 335 | delete kTrans; 336 | delete w; 337 | delete wScaled; 338 | 339 | return; 340 | } 341 | 342 | cv::Mat* gs::RationalWavelet::waveletBand(int scale, int band) 343 | { 344 | switch (band) 345 | { 346 | case 0: 347 | return &this->wavelet[scale]->highbandLH; 348 | break; 349 | case 1: 350 | return &this->wavelet[scale]->highbandHL; 351 | break; 352 | case 2: 353 | return &this->wavelet[scale]->highbandHH; 354 | break; 355 | } 356 | return nullptr; 357 | 358 | } 359 | 360 | cv::Mat gs::RationalWavelet::to32F(cv::Mat image) 361 | { 362 | Mat copy(image.rows, image.cols, CV_32F); 363 | for (int i = 0; i < image.rows; i++) 364 | { 365 | for (int j = 0; j < image.cols; j++) 366 | { 367 | copy.at(i, j) = ((float)image.at(i, j)) / 255.0; 368 | } 369 | } 370 | return copy; 371 | } 372 | cv::Mat gs::RationalWavelet::to8U(cv::Mat image) 373 | { 374 | Mat copy(image.rows, image.cols, CV_8U); 375 | for (int i = 0; i < image.rows; i++) 376 | { 377 | for (int j = 0; j < image.cols; j++) 378 | { 379 | int sample = (int)round(abs(image.at(i, j)) * 255.0); 380 | if (sample > 255) 381 | sample = 255; 382 | copy.at(i, j) = (unsigned char)sample; 383 | } 384 | } 385 | return copy; 386 | } 387 | 388 | cv::Mat* gs::superResolveWavelet(cv::Mat inputImage, float upscale, int iterations, double lambda, int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight, int waveletP, int waveletQ) 389 | { 390 | time_t t = time(0); 391 | int currentSRIter = 0; 392 | 393 | float gradualUpscale = pow((float)upscale, 1.0 / (float)iterations); 394 | int waveletJ = ceil(log(1 / gradualUpscale) / log((float)waveletP / (float)waveletQ)); 395 | Mat** imageSR = new Mat*[iterations + 1]; 396 | 397 | int numPatches = ceil((inputImage.size().width - patchSize)*(inputImage.size().height - patchSize)); 398 | 399 | cv::Mat** dLow; 400 | cv::Mat** dHigh; 401 | cv::flann::Index** tree; 402 | 403 | int waveletScale; 404 | int waveletBand; 405 | int numDictionaries = 3 * waveletJ; 406 | dLow = new Mat*[numDictionaries]; 407 | dHigh = new Mat*[numDictionaries]; 408 | tree = new cv::flann::Index*[numDictionaries]; 409 | 410 | for (int i = 0; i < waveletJ; i++) 411 | { 412 | for (int j = 0; j < 3; j++) 413 | { 414 | int dictionaryIndex = i * 3 + j; 415 | dLow[dictionaryIndex] = new Mat(3 * numPatches, patchSize*patchSize, CV_32F); 416 | dHigh[dictionaryIndex] = new Mat(3 * numPatches, patchSize*patchSize, CV_32F); 417 | createDictionaryWavelet(inputImage, gradualUpscale, patchSize, waveletJ, waveletP, waveletQ, dLow[dictionaryIndex], dHigh[dictionaryIndex], i, j); 418 | tree[dictionaryIndex] = new cv::flann::Index(*(dLow[dictionaryIndex]), flann::KDTreeIndexParams(), cvflann::FLANN_DIST_EUCLIDEAN); 419 | } 420 | } 421 | 422 | 423 | imageSR[0] = new Mat(inputImage.rows,inputImage.cols, CV_32F); 424 | inputImage.copyTo(*(imageSR[0])); 425 | 426 | int iter; 427 | for (iter = 0; iter < iterations; iter++) 428 | { 429 | currentSRIter = iter; 430 | 431 | /*compute the SR image size*/ 432 | unsigned int NupX = ceil(imageSR[iter]->size().width * gradualUpscale); 433 | unsigned int NupY = ceil(imageSR[iter]->size().height * gradualUpscale); 434 | cv::Size Nup(NupX, NupY); 435 | 436 | Mat imageInterp(NupX, NupY, CV_32F); 437 | /*resize the input image using bicubic interpolation*/ 438 | resize(*(imageSR[iter]), imageInterp, Nup, 0.0, 0.0, INTER_CUBIC); 439 | 440 | Mat imageFiltered; 441 | imageFiltered = imageInterp; 442 | 443 | Mat aLowTrans; 444 | Mat invTemp; 445 | Mat inv; 446 | Mat M; 447 | Mat reconPatch; 448 | 449 | int patchIndex = 0; 450 | int totalPatches = (floor((float)NupX / (patchSize - patchOverlap)) - 1)*(floor((float)NupY / (patchSize - patchOverlap)) - 1); 451 | 452 | /*init the HR/LR atoms*/ 453 | Mat atomsHigh((unsigned int)(patchSize*patchSize), neighborhoodSize, CV_64F); 454 | Mat atomsLow((unsigned int)(patchSize*patchSize), neighborhoodSize, CV_64F); 455 | 456 | /*init the patches. We utilize double precision for DM calculations but the final product is single precision*/ 457 | Mat patch(1, patchSize*patchSize, CV_32F); 458 | Mat patchDouble(patchSize*patchSize, 1, CV_64F); 459 | 460 | /*init the final SR image*/ 461 | Mat* y = new Mat(NupX, NupY, CV_32F); 462 | 463 | 464 | /*take the rational wavelet of the filtered image*/ 465 | RationalWavelet* rwUp = new RationalWavelet(imageInterp, waveletJ, waveletP, waveletQ, 1); 466 | 467 | Mat imageFilteredResize; 468 | 469 | for (int scale = 0; scale < waveletJ; scale++) 470 | { 471 | cv::Size sizeLow(rwUp->waveletBand(scale, 0)->rows, rwUp->waveletBand(scale, 0)->cols); 472 | 473 | /* calculate the variance of the dictionary*/ 474 | for (int band = 0; band < 3; band++) 475 | { 476 | Mat wLow = *(rwUp->waveletBand(0, band)); 477 | resize(imageInterp, imageFilteredResize, sizeLow, 0.0, 0.0, CV_INTER_CUBIC); 478 | 479 | int dictIndex = scale * 3 + band; 480 | patchIndex = 0; 481 | 482 | Mat wImageUp; 483 | 484 | Mat* wSR = rwUp->waveletBand(scale, band); 485 | Size wSRsize(wSR->rows, wSR->cols); 486 | 487 | resize(imageFilteredResize, wImageUp, wSRsize, 0.0, 0.0, CV_INTER_CUBIC); 488 | 489 | #if DEBUG_TEST 490 | imwrite("../images/waveletInterp.png", to8U(wImageUp)); 491 | #endif 492 | 493 | int wX = wImageUp.size().width; 494 | int wY = wImageUp.size().height; 495 | 496 | Mat recon = Mat::zeros(wX, wY, CV_32F); 497 | Mat reconGain = Mat::zeros(wX, wY, CV_64F); 498 | totalPatches = (floor((float)wX / (patchSize - patchOverlap)) - 1)*(floor((float)wY / (patchSize - patchOverlap)) - 1); 499 | 500 | for (int x = 0; x < wX - patchSize + 1; x = x + (patchSize - patchOverlap)) 501 | { 502 | for (int y = 0; y < wY - patchSize + 1; y = y + (patchSize - patchOverlap)) 503 | { 504 | double sampleSum = 0.0; 505 | int pixelIndex = 0; 506 | for (int i = 0; i < patchSize; i++) 507 | { 508 | for (int j = 0; j < patchSize; j++) 509 | { 510 | int sampleIndexX = (x + i); 511 | int sampleIndexY = (y + j); 512 | float interpSample = (float)wImageUp.at(sampleIndexX, sampleIndexY); 513 | patch.at(0, pixelIndex) = interpSample; 514 | patchDouble.at(pixelIndex, 0) = (double)interpSample; 515 | sampleSum = sampleSum + (double)abs(interpSample); 516 | pixelIndex++; 517 | } 518 | } 519 | 520 | pixelIndex = 0; 521 | sampleSum = sampleSum / (float)(patchSize*patchSize); 522 | if (sampleSum != 0.0) 523 | { 524 | int pixelIndex = 0; 525 | for (int i = 0; i < patchSize; i++) 526 | { 527 | for (int j = 0; j < patchSize; j++) 528 | { 529 | patch.at(0, pixelIndex) = patch.at(0, pixelIndex) - sampleSum; 530 | patchDouble.at(pixelIndex, 0) = patchDouble.at(pixelIndex, 0) - sampleSum; 531 | pixelIndex++; 532 | } 533 | } 534 | } 535 | 536 | /* get the k nearest neighbors*/ 537 | nearestNeighborAtoms(&patch, patchSize, tree[dictIndex], dLow[dictIndex], dHigh[dictIndex], neighborhoodSize, neighborhoodWeight, &atomsLow, &atomsHigh); 538 | 539 | /*compute the direct mapping matrix*/ 540 | transpose(atomsLow, aLowTrans); 541 | invTemp = atomsLow*aLowTrans + lambda*Mat::eye(patchSize*patchSize, patchSize*patchSize, CV_64F); 542 | inv = invTemp.inv(DECOMP_SVD); 543 | M = atomsHigh*aLowTrans*inv; 544 | reconPatch = M*patchDouble; 545 | 546 | pixelIndex = 0; 547 | for (int i = 0; i < patchSize; i++) 548 | { 549 | for (int j = 0; j < patchSize; j++) 550 | { 551 | double f = reconPatch.at(pixelIndex, 0); 552 | recon.at(x + i, y + j) = recon.at(x + i, y + j) + (float)reconPatch.at(pixelIndex, 0); 553 | reconGain.at(x + i, y + j) = reconGain.at(x + i, y + j) + 1.0; 554 | pixelIndex++; 555 | } 556 | } 557 | patchIndex++; 558 | 559 | /*print the progress*/ 560 | if (patchIndex % 1000 == 0) 561 | { 562 | //system("cls"); 563 | float progress = 100.0*(float)patchIndex / (float)totalPatches; 564 | printf("Current Iteration: "); 565 | printf(std::to_string(currentSRIter + 1).c_str()); 566 | printf("/"); 567 | printf(std::to_string(iterations).c_str()); 568 | printf(" scale: "); 569 | printf(std::to_string(scale + 1).c_str()); 570 | printf("/"); 571 | printf(std::to_string(waveletJ).c_str()); 572 | printf(" band: "); 573 | printf(std::to_string(band + 1).c_str()); 574 | printf("/3"); 575 | printf(", progress: %0.1f percent \n", progress); 576 | } 577 | } 578 | } 579 | 580 | /*correct the overlapping gain at each pixel*/ 581 | for (int i = 0; i < wX; i++) 582 | { 583 | for (int j = 0; j < wY; j++) 584 | { 585 | if (reconGain.at(i, j) != 0.0) 586 | { 587 | wSR->at(i, j) = (float)(recon.at(i, j) / reconGain.at(i, j)); 588 | } 589 | } 590 | } 591 | 592 | #if DEBUG_TEST 593 | imwrite("../images/waveletRecon.png", to8U(*wSR)); 594 | #endif 595 | 596 | } 597 | } 598 | *y = rwUp->inverseRationalWavelet(); 599 | 600 | imageInterp.release(); 601 | delete rwUp; 602 | 603 | imageSR[iter + 1] = y; 604 | 605 | Mat imageSR_ibp; 606 | /*back project the image for refinement*/ 607 | iterativeBackProjection(*(imageSR[iter + 1]), inputImage, imageSR_ibp, upscale); 608 | *(imageSR[iter + 1]) = imageSR_ibp; 609 | 610 | 611 | std::string iterfp("../images/SR_iteration_"); 612 | iterfp.append(std::to_string(iter)); 613 | iterfp.append(".png"); 614 | imwrite(iterfp.c_str(), to8U(*imageSR[iter + 1])); 615 | } 616 | 617 | 618 | for (int i = 0; i < waveletJ; i++) 619 | { 620 | dLow[i]->release(); 621 | dHigh[i]->release(); 622 | delete tree[i]; 623 | } 624 | delete dLow; 625 | delete dHigh; 626 | delete tree; 627 | 628 | return imageSR[iter]; 629 | } 630 | 631 | cv::Mat* gs::superResolve(cv::Mat inputImage, float upscale, int iterations, double lambda, unsigned int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight) 632 | { 633 | time_t t = time(0); 634 | float gradualUpscale = pow((float)upscale, 1.0 / (float)iterations); 635 | cv::Mat** imageSR = new cv::Mat*[iterations + 1]; 636 | cv::Mat trainingImage; 637 | trainingImage = inputImage; 638 | unsigned int numPatches = ceil((trainingImage.size().width - patchSize)*(trainingImage.size().height - patchSize)); 639 | int currentSRIter = 0; 640 | cv::Mat* dLow; 641 | cv::Mat* dHigh; 642 | cv::flann::Index* tree; 643 | 644 | dLow = new cv::Mat(numPatches, patchSize*patchSize, CV_32F); 645 | dHigh = new cv::Mat(numPatches, patchSize*patchSize, CV_32F); 646 | createDictionary(trainingImage, gradualUpscale, patchSize, dLow, dHigh); 647 | tree = new cv::flann::Index(*dLow, flann::KDTreeIndexParams(1), cvflann::FLANN_DIST_EUCLIDEAN); 648 | 649 | 650 | imageSR[0] = new Mat(inputImage.rows, inputImage.cols, CV_32F); 651 | inputImage.copyTo(*(imageSR[0])); 652 | int iter; 653 | 654 | for (iter = 0; iter < iterations; iter++) 655 | { 656 | currentSRIter = iter; 657 | /*Super resolve the image*/ 658 | 659 | /*compute the SR image size*/ 660 | unsigned int NupX = ceil(imageSR[iter]->size().width * gradualUpscale); 661 | unsigned int NupY = ceil(imageSR[iter]->size().height * gradualUpscale); 662 | cv::Size Nup(NupX, NupY); 663 | 664 | cv::Mat imageInterp(NupX, NupY, CV_32F); 665 | /*resize the input image using bicubic interpolation*/ 666 | resize(*(imageSR[iter]), imageInterp, Nup, 0.0, 0.0, INTER_CUBIC); 667 | 668 | cv::Mat imageFiltered = imageInterp; 669 | 670 | cv::Mat aLowTrans; 671 | cv::Mat invTemp; 672 | cv::Mat inv; 673 | cv::Mat M; 674 | cv::Mat reconPatch; 675 | 676 | int patchIndex = 0; 677 | int totalPatches = (floor((float)NupX / (patchSize - patchOverlap)) - 1)*(floor((float)NupY / (patchSize - patchOverlap)) - 1); 678 | 679 | /*init the HR/LR atoms*/ 680 | cv::Mat atomsHigh((unsigned int)(patchSize*patchSize), neighborhoodSize, CV_64F); 681 | cv::Mat atomsLow((unsigned int)(patchSize*patchSize), neighborhoodSize, CV_64F); 682 | 683 | /*init the patches. We utilize double precision for DM calculations but the final product is single precision*/ 684 | cv::Mat patch(1, patchSize*patchSize, CV_32F); 685 | cv::Mat patchDouble(patchSize*patchSize, 1, CV_64F); 686 | 687 | /*init the final SR image*/ 688 | cv::Mat* y = new cv::Mat(NupX, NupY, CV_32F); 689 | 690 | #if DEBUG_TEST 691 | imwrite("../images/imageInterpolate.png", to8U(imageFiltered)); 692 | #endif 693 | 694 | cv::Mat recon = cv::Mat::zeros(cv::Size(NupX, NupY), CV_64F); 695 | cv::Mat reconGain = cv::Mat::zeros(cv::Size(NupX, NupY), CV_64F); 696 | for (int x = 0; x < NupX - patchSize + 1; x = x + (patchSize - patchOverlap)) 697 | { 698 | for (int y = 0; y < NupY - patchSize + 1; y = y + (patchSize - patchOverlap)) 699 | { 700 | /* sample patch at (x,y) with size (patchSize,patchSize) */ 701 | 702 | int pixelIndex = 0; 703 | for (int i = 0; i < patchSize; i++) 704 | { 705 | for (int j = 0; j < patchSize; j++) 706 | { 707 | int sampleIndexX = (x + i); 708 | int sampleIndexY = (y + j); 709 | float interpSample = (float)imageFiltered.at(sampleIndexX, sampleIndexY); 710 | patch.at(0, pixelIndex) = interpSample; 711 | patchDouble.at(pixelIndex, 0) = (double)interpSample; 712 | pixelIndex++; 713 | } 714 | } 715 | 716 | /*find the k nearest neighbors*/ 717 | nearestNeighborAtoms(&patch, patchSize, tree, dLow, dHigh, neighborhoodSize, neighborhoodWeight, &atomsLow, &atomsHigh); 718 | 719 | /*compute the mapping 'M' between the K-nearest low res atoms and their corresponding K high res atoms*/ 720 | transpose(atomsLow, aLowTrans); 721 | invTemp = atomsLow*aLowTrans + lambda*cv::Mat::eye(patchSize*patchSize, patchSize*patchSize, CV_64F); 722 | inv = invTemp.inv(DECOMP_SVD); 723 | 724 | M = atomsHigh*aLowTrans*inv; 725 | /*estimate the high resolution patch of the input low resolution patch*/ 726 | reconPatch = M*patchDouble; 727 | 728 | pixelIndex = 0; 729 | for (int i = 0; i < patchSize; i++) 730 | { 731 | for (int j = 0; j < patchSize; j++) 732 | { 733 | double f = reconPatch.at(pixelIndex, 0); 734 | /*add the HR estimate patch to the final result*/ 735 | recon.at(x + i, y + j) = recon.at(x + i, y + j) + reconPatch.at(pixelIndex, 0); 736 | /* compute the scaling factor at each pixel to be divided by at the end*/ 737 | reconGain.at(x + i, y + j) = reconGain.at(x + i, y + j) + 1.0; 738 | pixelIndex++; 739 | } 740 | } 741 | patchIndex++; 742 | 743 | /*print the progress*/ 744 | if (patchIndex % 500 == 0) 745 | { 746 | //system("cls"); 747 | float progress = 100.0*(float)patchIndex / (float)totalPatches; 748 | printf("Current Iteration: "); 749 | printf(std::to_string(currentSRIter + 1).c_str()); 750 | printf("/"); 751 | printf(std::to_string(iterations).c_str()); 752 | printf(", progress: %0.1f percent \n", progress); 753 | } 754 | 755 | } 756 | } 757 | 758 | for (int i = 0; i < NupX; i++) 759 | { 760 | for (int j = 0; j < NupY; j++) 761 | { 762 | /*correct the overlapping gain at each pixel*/ 763 | if (reconGain.at(i, j) == 0.0) 764 | { 765 | y->at(i, j) = imageInterp.at(i, j); 766 | } 767 | else 768 | { 769 | y->at(i, j) = (float)(recon.at(i, j) / reconGain.at(i, j)); 770 | } 771 | } 772 | } 773 | 774 | imageSR[iter + 1] = y; 775 | 776 | cv::Mat imageSR_ibp; 777 | /*back project the image for refinement*/ 778 | iterativeBackProjection(*(imageSR[iter + 1]), inputImage, imageSR_ibp, upscale); 779 | *(imageSR[iter + 1]) = imageSR_ibp; 780 | 781 | 782 | std::string iterfp("../images/SR_iteration_"); 783 | iterfp.append(std::to_string(iter)); 784 | iterfp.append(".png"); 785 | #if DEBUG_TEST 786 | imwrite(iterfp.c_str(), to8U(*imageSR[iter + 1])); 787 | #endif 788 | 789 | } 790 | 791 | dLow->release(); 792 | dHigh->release(); 793 | delete tree; 794 | 795 | for (int i = 0; i < iter - 1; i++) 796 | { 797 | imageSR[i]->release(); 798 | } 799 | 800 | return imageSR[iter]; 801 | } 802 | 803 | void gs::createDictionary(cv::Mat inputImage, float gradualUpscale, int patchSize, cv::Mat* dLow, cv::Mat* dHigh) 804 | { 805 | int nx = (int)ceil(inputImage.size().width); 806 | int ny = (int)ceil(inputImage.size().height); 807 | int nx_low = (int)ceil(inputImage.size().width / gradualUpscale); 808 | int ny_low = (int)ceil(inputImage.size().height / gradualUpscale); 809 | 810 | cv::Mat H, G, h, g; 811 | /*create the LPF*/ 812 | makeFreqResp(151, 1, (float)gradualUpscale / 1.75, 1, H, G, h, g); 813 | cv::Mat ht; 814 | transpose(h, ht); 815 | cv::Mat imageFiltered; 816 | 817 | /*filter the image, obatin a low res estimate*/ 818 | filter2D(inputImage, imageFiltered, -1, h, Point(-1, -1), 0.0, BORDER_REPLICATE); 819 | filter2D(imageFiltered, imageFiltered, -1, ht, Point(-1, -1), 0.0, BORDER_REPLICATE); 820 | 821 | cv::Mat tt; 822 | resize(inputImage, tt, Size(nx * 2, ny * 2), 0.0, 0.0, CV_INTER_CUBIC); 823 | 824 | resize(imageFiltered, imageFiltered, Size(nx_low, ny_low), 0.0, 0.0); 825 | resize(imageFiltered, imageFiltered, Size(nx, ny), 0.0, 0.0, CV_INTER_CUBIC); 826 | 827 | #if DEBUG_TEST 828 | imwrite("../images/trainingImageLR.png", to8U(imageFiltered)); 829 | imwrite("../images/trainingImageHR.png", to8U(inputImage)); 830 | imwrite("../images/trainingImageUpscale.png", to8U(tt)); 831 | #endif 832 | 833 | /*iterate through each overlapping patch*/ 834 | int patchIndex = 0; 835 | for (int i = 0; i < ny - patchSize; i++) 836 | { 837 | for (int j = 0; j < nx - patchSize; j++) 838 | { 839 | int pixelIndex = 0; 840 | for (int x = 0; x < patchSize; x++) 841 | { 842 | for (int y = 0; y < patchSize; y++) 843 | { 844 | int sampleX = (x + i); 845 | int sampleY = (y + j); 846 | float sampleLow = (float)imageFiltered.at(sampleX, sampleY); 847 | float sampleHigh = (float)inputImage.at(sampleX, sampleY); 848 | 849 | /*append to dictionaries*/ 850 | dLow->at(patchIndex, pixelIndex) = sampleLow; 851 | dHigh->at(patchIndex, pixelIndex) = sampleHigh; 852 | pixelIndex++; 853 | } 854 | } 855 | 856 | patchIndex++; 857 | 858 | } 859 | } 860 | 861 | imageFiltered.release(); 862 | 863 | 864 | } 865 | 866 | void gs::createDictionaryWavelet(cv::Mat inputImage, float gradualUpscale, int patchSize, int waveletJ, int waveletP, int waveletQ, cv::Mat* dLow, cv::Mat* dHigh, int waveletScale, int waveletBand) 867 | { 868 | int nx = (int)ceil(inputImage.size().width); 869 | int ny = (int)ceil(inputImage.size().height); 870 | int nx_low = (int)ceil(inputImage.size().width / gradualUpscale); 871 | int ny_low = (int)ceil(inputImage.size().height / gradualUpscale); 872 | 873 | cv::Mat H, G, h, g; 874 | /*create the LPF*/ 875 | gs::makeFreqResp(151, 1, (float)gradualUpscale / 1.75, 1, H, G, h, g); 876 | cv::Mat ht; 877 | transpose(h, ht); 878 | cv::Mat imageFiltered; 879 | 880 | /*filter the image, obatin a low res estimate*/ 881 | filter2D(inputImage, imageFiltered, -1, h, Point(-1, -1), 0.0, BORDER_REPLICATE); 882 | filter2D(imageFiltered, imageFiltered, -1, ht, Point(-1, -1), 0.0, BORDER_REPLICATE); 883 | 884 | cv::Mat tt; 885 | resize(inputImage, tt, Size(nx * 2, ny * 2), 0.0, 0.0, CV_INTER_CUBIC); 886 | 887 | resize(imageFiltered, imageFiltered, Size(nx_low, ny_low), 0.0, 0.0); 888 | resize(imageFiltered, imageFiltered, Size(nx, ny), 0.0, 0.0, CV_INTER_CUBIC); 889 | 890 | /*take the rational wavelet transform of the input image*/ 891 | gs::RationalWavelet* rwHR = new gs::RationalWavelet(inputImage, waveletJ, waveletP, waveletQ, 1); 892 | 893 | cv::Mat imageFilteredResize; 894 | 895 | cv::Mat* wImageHR; 896 | cv::Mat wImageLR; 897 | 898 | int patchIndex = 0; 899 | //int waveletScale, int waveletBand 900 | 901 | cv::Size sizeLow(rwHR->waveletBand(waveletScale, 0)->rows, rwHR->waveletBand(waveletScale, 0)->cols); 902 | resize(imageFiltered, imageFilteredResize, sizeLow, 0.0, 0.0, CV_INTER_CUBIC); 903 | 904 | int dictIndex = waveletScale * 3 + waveletBand; 905 | patchIndex = 0; 906 | wImageHR = rwHR->waveletBand(waveletScale, waveletBand); 907 | Size hrSize(wImageHR->rows, wImageHR->cols); 908 | 909 | resize(imageFilteredResize, imageFilteredResize, hrSize, 0.0, 0.0, CV_INTER_CUBIC); 910 | 911 | #if DEBUG_TEST 912 | std::string fpLow("../images/trainingImage_waveletLR_"); 913 | std::string fpHigh("../images/trainingImage_waveletHR_"); 914 | 915 | fpLow.append(std::to_string(waveletScale)); 916 | fpLow.append("_"); 917 | fpLow.append(std::to_string(waveletBand)); 918 | fpLow.append(".png"); 919 | 920 | fpHigh.append(std::to_string(waveletScale)); 921 | fpHigh.append("_"); 922 | fpHigh.append(std::to_string(waveletBand)); 923 | fpHigh.append(".png"); 924 | 925 | //imwrite(fpLow.c_str(), to8U(wImageLR)); 926 | imwrite(fpLow.c_str(), to8U(imageFilteredResize)); 927 | imwrite(fpHigh.c_str(), to8U(*wImageHR)); 928 | #endif 929 | 930 | nx = wImageHR->rows; 931 | ny = wImageHR->cols; 932 | 933 | for (int i = 0; i < nx - patchSize; i++) 934 | { 935 | for (int j = 0; j < ny - patchSize; j++) 936 | { 937 | int pixelIndex = 0; 938 | float sumLow = 0.0; 939 | float sumHigh = 0.0; 940 | for (int x = 0; x < patchSize; x++) 941 | { 942 | for (int y = 0; y < patchSize; y++) 943 | { 944 | int sampleX = (x + i); 945 | int sampleY = (y + j); 946 | float sampleLow = (float)imageFilteredResize.at(sampleX, sampleY); 947 | float sampleHigh = (float)wImageHR->at(sampleX, sampleY); 948 | sumLow = sumLow + (sampleLow); 949 | sumHigh = sumHigh + abs(sampleHigh); 950 | 951 | /*append the dictionary for the current scale*/ 952 | dLow->at(patchIndex, pixelIndex) = sampleLow; 953 | dHigh->at(patchIndex, pixelIndex) = sampleHigh; 954 | pixelIndex++; 955 | } 956 | } 957 | 958 | sumLow = sumLow / (float)(patchSize*patchSize); 959 | 960 | pixelIndex = 0; 961 | if (sumLow != 0.0) 962 | { 963 | for (int x = 0; x < patchSize; x++) 964 | { 965 | for (int y = 0; y < patchSize; y++) 966 | { 967 | //enforce unity dc gain 968 | dLow->at(patchIndex, pixelIndex) = dLow->at(patchIndex, pixelIndex) - sumLow; 969 | pixelIndex++; 970 | } 971 | } 972 | } 973 | 974 | /*threshold the patches to avoid too many low enery patches*/ 975 | if (sumHigh > WAVELET_THRESHOLD / ((float)(patchSize*patchSize))) 976 | { 977 | patchIndex++; 978 | } 979 | 980 | } 981 | } 982 | 983 | cv::Mat dLowTrim(patchIndex, patchSize*patchSize, CV_32F); 984 | cv::Mat dHighTrim(patchIndex, patchSize*patchSize, CV_32F); 985 | int pixelIndex = 0; 986 | /*trim the dictionary to the appropriate size*/ 987 | for (int i = 0; i < patchIndex; i++) 988 | { 989 | for (int j = 0; j < patchSize*patchSize; j++) 990 | { 991 | dLowTrim.at(i, j) = dLow->at(i, j); 992 | dHighTrim.at(i, j) = dHigh->at(i, j); 993 | } 994 | } 995 | 996 | dLow->release(); 997 | dHigh->release(); 998 | *(dLow) = dLowTrim; 999 | *(dHigh) = dHighTrim; 1000 | 1001 | 1002 | 1003 | delete rwHR; 1004 | } 1005 | 1006 | void gs::nearestNeighborAtoms(cv::Mat* patch, int patchSize, cv::flann::Index* tree, cv::Mat* dLow, cv::Mat* dHigh, int K, double weight, cv::Mat* atomsLow, cv::Mat* atomsHigh) 1007 | { 1008 | /*index of the nearest patches patches, sorted by distance*/ 1009 | cv::Mat ind; 1010 | /*Eculidean distances of nearest patches*/ 1011 | /* NOTE!: DISTANCE IS EUCLIDEAN DISTANCE SQUARED! */ 1012 | cv::Mat dist; 1013 | 1014 | long temp = getTickCount(); 1015 | /*search the tree for the K nearest patches of 'patch'*/ 1016 | tree->knnSearch(*patch, ind, dist, K); 1017 | long t = getTickCount() - temp; 1018 | 1019 | double* scaleVec = new double[K]; 1020 | for (int i = 0; i < K; i++) 1021 | { 1022 | /* for each K-nearest patches*/ 1023 | double d = (double)dist.at(0, i); 1024 | 1025 | /*compute the atom weighting*/ 1026 | double e = -(d) / weight; 1027 | double scale = exp(e); 1028 | scaleVec[i] = scale; 1029 | } 1030 | 1031 | /*compute the sum of the atoms weights for normalization*/ 1032 | double scaleSum = 0.0; 1033 | for (int i = 0; i < K; i++) 1034 | { 1035 | scaleSum = scaleSum + scaleVec[i]; 1036 | } 1037 | double tempSum = 0.0; 1038 | for (int i = 0; i < K; i++) 1039 | { 1040 | scaleVec[i] = scaleVec[i] / scaleSum; 1041 | tempSum += scaleVec[i]; 1042 | } 1043 | 1044 | for (int k = 0; k < K; k++) 1045 | { 1046 | int index = ind.at(0, k); 1047 | /*if an error exists, append a zero atom*/ 1048 | if (index < dLow->rows && index >= 0 && scaleSum != 0.0) 1049 | { 1050 | /*append the LR atoms to "atomsLow', and HR atoms to 'atomsHigh'. Weight and normalize*/ 1051 | for (int j = 0; j < patchSize*patchSize; j++) 1052 | { 1053 | atomsLow->at(j, k) = (double)(dLow->at(index, j)*(scaleVec[k])); 1054 | atomsHigh->at(j, k) = (double)(dHigh->at(index, j)*(scaleVec[k])); 1055 | } 1056 | } 1057 | else 1058 | { 1059 | for (int j = 0; j < patchSize*patchSize; j++) 1060 | { 1061 | atomsLow->at(j, k) = 0.0; 1062 | atomsHigh->at(j, k) = 0.0; 1063 | } 1064 | } 1065 | } 1066 | 1067 | /*clean up*/ 1068 | delete scaleVec; 1069 | ind.release(); 1070 | dist.release(); 1071 | } 1072 | 1073 | #define IBP_ITERS 8 1074 | void gs::iterativeBackProjection(cv::Mat imageSR, cv::Mat &imageLR, cv::Mat &result, float upscale) 1075 | { 1076 | result = cv::Mat(imageSR.rows, imageSR.cols, CV_32F); 1077 | imageSR.copyTo(result); 1078 | int filterSize = 251; 1079 | 1080 | /*create the lpf*/ 1081 | cv::Mat H, G, h, g; 1082 | makeFreqResp(filterSize, 1.0, upscale , 1.0, H, G, h, g); 1083 | cv::Mat ht; 1084 | transpose(h, ht); 1085 | cv::Mat srFiltered; 1086 | cv::Mat imageErr; 1087 | 1088 | cv::Mat lr; 1089 | resize(imageLR, lr, Size(imageSR.rows, imageSR.cols), 0.0, 0.0, CV_INTER_CUBIC); 1090 | filter2D(lr, lr, -1, h, Point(-1, -1), 0.0, BORDER_REPLICATE); 1091 | filter2D(lr, lr, -1, ht, Point(-1, -1), 0.0, BORDER_REPLICATE); 1092 | 1093 | for (int i = 0; i < IBP_ITERS; i++) 1094 | { 1095 | /* filter and downsample the sr image */ 1096 | filter2D(result, srFiltered, -1, h, Point(-1, -1), 0.0, BORDER_REPLICATE); 1097 | filter2D(srFiltered, srFiltered, -1, ht, Point(-1, -1), 0.0, BORDER_REPLICATE); 1098 | 1099 | /*compute the error*/ 1100 | imageErr = lr - srFiltered; 1101 | 1102 | /*add the error back*/ 1103 | result = result + imageErr; 1104 | } 1105 | 1106 | /*clean up*/ 1107 | H.release(); 1108 | G.release(); 1109 | h.release(); 1110 | g.release(); 1111 | ht.release(); 1112 | imageErr.release(); 1113 | srFiltered.release(); 1114 | } 1115 | 1116 | cv::Mat gs::to32F(cv::Mat image) 1117 | { 1118 | cv::Mat copy(image.rows, image.cols, CV_32F); 1119 | for (int i = 0; i < image.rows; i++) 1120 | { 1121 | for (int j = 0; j < image.cols; j++) 1122 | { 1123 | copy.at(i, j) = ((float)image.at(i, j)) / 255.0; 1124 | } 1125 | } 1126 | return copy; 1127 | } 1128 | 1129 | cv::Mat gs::to8U(cv::Mat image) 1130 | { 1131 | cv::Mat copy(image.rows, image.cols, CV_8U); 1132 | for (int i = 0; i < image.rows; i++) 1133 | { 1134 | for (int j = 0; j < image.cols; j++) 1135 | { 1136 | int sample = (int)round(abs(image.at(i, j)) * 255.0); 1137 | if (sample > 255) 1138 | sample = 255; 1139 | copy.at(i, j) = (unsigned char)sample; 1140 | } 1141 | } 1142 | return copy; 1143 | } 1144 | 1145 | float gs::psnr32F(cv::Mat imageGT, cv::Mat imageRec) 1146 | { 1147 | cv::Mat gtResize; 1148 | resize(imageGT, gtResize, imageRec.size(), 0.0, 0.0, CV_INTER_CUBIC); 1149 | 1150 | 1151 | double err = 0.0; 1152 | for (int i = 0; i < imageRec.rows; i++) 1153 | { 1154 | for (int j = 0; j < imageRec.cols; j++) 1155 | { 1156 | float recSample = imageRec.at(i, j); 1157 | float gtSample = gtResize.at(i, j); 1158 | float errSample = recSample - gtSample; 1159 | err = err + (double)pow(abs(errSample), 2.0); 1160 | } 1161 | } 1162 | err = err / ((double)imageRec.rows*(double)imageRec.cols); 1163 | double result = 20.0*log10(1.0 / sqrt(err)); 1164 | return (float)result; 1165 | } 1166 | 1167 | float gs::psnr8U(cv::Mat imageGT, cv::Mat imageRec) 1168 | { 1169 | cv::Mat gtResize; 1170 | resize(imageGT, gtResize, imageRec.size(), 0.0, 0.0, CV_INTER_CUBIC); 1171 | 1172 | unsigned char err = 0; 1173 | for (int i = 0; i < imageRec.rows; i++) 1174 | { 1175 | for (int j = 0; j < imageRec.cols; j++) 1176 | { 1177 | unsigned char recSample = imageRec.at(i, j); 1178 | unsigned char gtSample = imageGT.at(i, j); 1179 | unsigned char errSample = recSample - gtSample; 1180 | err = err + (double)pow(abs((double)errSample), 2.0); 1181 | } 1182 | } 1183 | double derr = (double)err / ((double)imageRec.rows*(double)imageRec.cols); 1184 | double result = 20.0*log10(255.0 / sqrt(err)); 1185 | return (float)result; 1186 | } 1187 | 1188 | void gs::exportReport(Mat inputImage, Mat srImage, double upscale, int iterations, double lambda, unsigned int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight) 1189 | { 1190 | time_t t = time(0); 1191 | std::string basePath("../exports/"); 1192 | basePath.append(std::to_string((long)t)); 1193 | int err = _mkdir(basePath.c_str()); 1194 | 1195 | std::string textPath = basePath; 1196 | textPath.append("/report.txt"); 1197 | 1198 | std::ofstream myfile; 1199 | myfile.open(textPath, std::ofstream::app); 1200 | 1201 | struct tm * now = localtime(&t); 1202 | myfile << "Date: "; 1203 | myfile << (now->tm_year + 1900); 1204 | myfile << '-'; 1205 | myfile << (now->tm_mon + 1); 1206 | myfile << '-'; 1207 | myfile << now->tm_mday; 1208 | myfile << "\n"; 1209 | 1210 | myfile << "SR algorithm: Direct Mapping of Self Examples\n"; 1211 | 1212 | myfile << "patch size: "; 1213 | myfile << std::to_string(patchSize); 1214 | myfile << "\n"; 1215 | 1216 | myfile << "patch overlap: "; 1217 | myfile << std::to_string(patchOverlap); 1218 | myfile << "\n"; 1219 | 1220 | myfile << "neighborhood size: "; 1221 | myfile << std::to_string(neighborhoodSize); 1222 | myfile << "\n"; 1223 | 1224 | myfile << "neighborhood weight: "; 1225 | myfile << std::to_string(neighborhoodWeight); 1226 | myfile << "\n"; 1227 | 1228 | myfile << "upscale factor: "; 1229 | myfile << std::to_string(upscale); 1230 | myfile << "\n"; 1231 | 1232 | myfile << "lambda: "; 1233 | myfile << (double)lambda; 1234 | myfile << "\n"; 1235 | 1236 | myfile << "iterations: "; 1237 | myfile << std::to_string(iterations); 1238 | myfile << "\n"; 1239 | 1240 | myfile.close(); 1241 | 1242 | Mat interpImage; 1243 | resize(inputImage, interpImage, srImage.size(), 0.0, 0.0, CV_INTER_CUBIC); 1244 | cv::imwrite(basePath + std::string("/inputImage.png"), gs::to8U(inputImage)); 1245 | imwrite(basePath + std::string("/inputImage_interpolated.png"), gs::to8U(interpImage)); 1246 | imwrite(basePath + std::string("/imageSR.png"), gs::to8U(srImage)); 1247 | 1248 | basePath.clear(); 1249 | textPath.clear(); 1250 | interpImage.release(); 1251 | } 1252 | 1253 | void gs::exportReport(Mat inputImage, Mat srImage, Mat imageGT, double upscale, int iterations, double lambda, unsigned int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight) 1254 | { 1255 | time_t t = time(0); 1256 | std::string basePath("../exports/"); 1257 | basePath.append(std::to_string((long)t)); 1258 | int err = _mkdir(basePath.c_str()); 1259 | 1260 | std::string textPath = basePath; 1261 | textPath.append("/report.txt"); 1262 | 1263 | std::ofstream myfile; 1264 | myfile.open(textPath, std::ofstream::app); 1265 | 1266 | struct tm * now = localtime(&t); 1267 | myfile << "Date: "; 1268 | myfile << (now->tm_year + 1900); 1269 | myfile << '-'; 1270 | myfile << (now->tm_mon + 1); 1271 | myfile << '-'; 1272 | myfile << now->tm_mday; 1273 | myfile << "\n"; 1274 | 1275 | 1276 | myfile << "SR algorithm: Direct Mapping of Self Examples\n"; 1277 | 1278 | myfile << "patch size: "; 1279 | myfile << std::to_string(patchSize); 1280 | myfile << "\n"; 1281 | 1282 | myfile << "patch overlap: "; 1283 | myfile << std::to_string(patchOverlap); 1284 | myfile << "\n"; 1285 | 1286 | myfile << "neighborhood size: "; 1287 | myfile << std::to_string(neighborhoodSize); 1288 | myfile << "\n"; 1289 | 1290 | myfile << "neighborhood weight: "; 1291 | myfile << std::to_string(neighborhoodWeight); 1292 | myfile << "\n"; 1293 | 1294 | myfile << "upscale factor: "; 1295 | myfile << std::to_string(upscale); 1296 | myfile << "\n"; 1297 | 1298 | myfile << "lambda: "; 1299 | myfile << (double)lambda; 1300 | myfile << "\n"; 1301 | 1302 | myfile << "iterations: "; 1303 | myfile << std::to_string(iterations); 1304 | myfile << "\n"; 1305 | 1306 | 1307 | Mat interpImage; 1308 | resize(inputImage, interpImage, srImage.size(), 0.0, 0.0, CV_INTER_CUBIC); 1309 | cv::imwrite(basePath + std::string("/inputImage.png"), gs::to8U(inputImage)); 1310 | imwrite(basePath + std::string("/inputImage_interpolated.png"), gs::to8U(interpImage)); 1311 | imwrite(basePath + std::string("/imageSR.png"), gs::to8U(srImage)); 1312 | imwrite(basePath + "/imageGT.png", gs::to8U(imageGT)); 1313 | 1314 | Mat imageInter; 1315 | resize(inputImage, imageInter, srImage.size(), 0.0, 0.0, CV_INTER_CUBIC); 1316 | 1317 | float psnrBC = gs::psnr32F(imageGT, imageInter); 1318 | float psnrSR = gs::psnr32F(imageGT, srImage); 1319 | 1320 | myfile << "PSNR Super Resoltion: "; 1321 | myfile << std::to_string(psnrSR); 1322 | myfile << " PSNR Interpolation: "; 1323 | myfile << std::to_string(psnrBC); 1324 | myfile << " difference: "; 1325 | myfile << std::to_string(psnrSR - psnrBC); 1326 | myfile << "\n"; 1327 | 1328 | myfile.close(); 1329 | 1330 | imageInter.release(); 1331 | basePath.clear(); 1332 | textPath.clear(); 1333 | } 1334 | 1335 | void gs::exportReportWavelet(Mat inputImage, Mat srImage, double upscale, int iterations, double lambda, unsigned int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight, int waveletP, int waveletQ) 1336 | { 1337 | time_t t = time(0); 1338 | 1339 | std::string basePath("../exports/"); 1340 | basePath.append(std::to_string((long)t)); 1341 | int err = _mkdir(basePath.c_str()); 1342 | 1343 | std::string textPath = basePath; 1344 | textPath.append("/report.txt"); 1345 | 1346 | std::ofstream myfile; 1347 | myfile.open(textPath, std::ofstream::app); 1348 | 1349 | struct tm * now = localtime(&t); 1350 | myfile << "Date: "; 1351 | myfile << (now->tm_year + 1900); 1352 | myfile << '-'; 1353 | myfile << (now->tm_mon + 1); 1354 | myfile << '-'; 1355 | myfile << now->tm_mday; 1356 | myfile << "\n"; 1357 | 1358 | myfile << "SR algorithm: Direct Mapping of Self Wavelet Examples\n"; 1359 | 1360 | myfile << "patch size: "; 1361 | myfile << std::to_string(patchSize); 1362 | myfile << "\n"; 1363 | 1364 | myfile << "patch overlap: "; 1365 | myfile << std::to_string(patchOverlap); 1366 | myfile << "\n"; 1367 | 1368 | myfile << "neighborhood size: "; 1369 | myfile << std::to_string(neighborhoodSize); 1370 | myfile << "\n"; 1371 | 1372 | myfile << "neighborhood weight: "; 1373 | myfile << std::to_string(neighborhoodWeight); 1374 | myfile << "\n"; 1375 | 1376 | myfile << "upscale factor: "; 1377 | myfile << std::to_string(upscale); 1378 | myfile << "\n"; 1379 | 1380 | myfile << "lambda: "; 1381 | myfile << (double)lambda; 1382 | myfile << "\n"; 1383 | 1384 | 1385 | myfile << "iterations: "; 1386 | myfile << std::to_string(iterations); 1387 | myfile << "\n"; 1388 | 1389 | myfile << "wavelet dilation factor: "; 1390 | myfile << std::to_string((float)waveletP / (float)waveletQ); 1391 | myfile << "\n"; 1392 | 1393 | myfile.close(); 1394 | 1395 | Mat interpImage; 1396 | resize(inputImage, interpImage, srImage.size(), 0.0, 0.0, CV_INTER_CUBIC); 1397 | cv::imwrite(basePath + std::string("/inputImage.png"), gs::to8U(inputImage)); 1398 | imwrite(basePath + std::string("/inputImage_interpolated.png"), gs::to8U(interpImage)); 1399 | imwrite(basePath + std::string("/imageSR.png"), gs::to8U(srImage)); 1400 | 1401 | 1402 | int J = ceil(log(1 / (float)upscale) / log((float)waveletP / (float)waveletQ)); 1403 | gs::RationalWavelet* rwSR = new gs::RationalWavelet(srImage, J, waveletP, waveletQ, 1); 1404 | gs::RationalWavelet* rwBC = new gs::RationalWavelet(interpImage, J, waveletP, waveletQ, 1); 1405 | 1406 | for (int scale = 0; scale < J; scale++) 1407 | { 1408 | for (int band = 0; band < 3; band++) 1409 | { 1410 | Mat* wSR = rwSR->waveletBand(scale, band); 1411 | Mat* wBC = rwBC->waveletBand(scale, band); 1412 | std::string waveletSRpath("/waveletSR_scale"); 1413 | waveletSRpath.append(std::to_string(scale)); 1414 | waveletSRpath.append("_band"); 1415 | waveletSRpath.append(std::to_string(band)); 1416 | waveletSRpath.append(".png"); 1417 | imwrite(basePath + waveletSRpath, gs::to8U(*wSR)); 1418 | 1419 | std::string waveletBCpath("/waveletInterpolated_scale"); 1420 | waveletBCpath.append(std::to_string(scale)); 1421 | waveletBCpath.append("_band"); 1422 | waveletBCpath.append(std::to_string(band)); 1423 | waveletBCpath.append(".png"); 1424 | imwrite(basePath + waveletBCpath, gs::to8U(*wBC)); 1425 | } 1426 | } 1427 | 1428 | interpImage.release(); 1429 | textPath.clear(); 1430 | basePath.clear(); 1431 | delete rwSR; 1432 | delete rwBC; 1433 | } 1434 | 1435 | void gs::exportReportWavelet(Mat inputImage, Mat srImage, Mat imageGT, double upscale, int iterations, double lambda, unsigned int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight, int waveletP, int waveletQ) 1436 | { 1437 | time_t t = time(0); 1438 | 1439 | std::string basePath("../exports/"); 1440 | basePath.append(std::to_string((long)t)); 1441 | int err = _mkdir(basePath.c_str()); 1442 | 1443 | std::string textPath = basePath; 1444 | textPath.append("/report.txt"); 1445 | 1446 | int J = ceil(log(1 / (float)upscale) / log((float)waveletP / (float)waveletQ)); 1447 | gs::RationalWavelet* rwGT = new gs::RationalWavelet(imageGT, J, waveletP, waveletQ, 1); 1448 | 1449 | for (int scale = 0; scale < J; scale++) 1450 | { 1451 | for (int band = 0; band < 3; band++) 1452 | { 1453 | Mat* wGT = rwGT->waveletBand(scale, band); 1454 | 1455 | std::string waveletSRpath("/waveletGT_scale"); 1456 | waveletSRpath.append(std::to_string(scale)); 1457 | waveletSRpath.append("_band"); 1458 | waveletSRpath.append(std::to_string(band)); 1459 | waveletSRpath.append(".png"); 1460 | imwrite(basePath + waveletSRpath, gs::to8U(*wGT)); 1461 | } 1462 | } 1463 | 1464 | imwrite(basePath + "/imageGT.png", gs::to8U(imageGT)); 1465 | imwrite(basePath + "/imageSR.png", gs::to8U(srImage)); 1466 | 1467 | 1468 | std::ofstream myfile; 1469 | myfile.open(textPath, std::ofstream::app); 1470 | 1471 | 1472 | struct tm * now = localtime(&t); 1473 | myfile << "Date: "; 1474 | myfile << (now->tm_year + 1900); 1475 | myfile << '-'; 1476 | myfile << (now->tm_mon + 1); 1477 | myfile << '-'; 1478 | myfile << now->tm_mday; 1479 | myfile << "\n"; 1480 | 1481 | myfile << "SR algorithm: Direct Mapping of Self Wavelet Examples\n"; 1482 | 1483 | myfile << "patch size: "; 1484 | myfile << std::to_string(patchSize); 1485 | myfile << "\n"; 1486 | 1487 | myfile << "patch overlap: "; 1488 | myfile << std::to_string(patchOverlap); 1489 | myfile << "\n"; 1490 | 1491 | myfile << "neighborhood size: "; 1492 | myfile << std::to_string(neighborhoodSize); 1493 | myfile << "\n"; 1494 | 1495 | myfile << "neighborhood weight: "; 1496 | myfile << std::to_string(neighborhoodWeight); 1497 | myfile << "\n"; 1498 | 1499 | myfile << "upscale factor: "; 1500 | myfile << std::to_string(upscale); 1501 | myfile << "\n"; 1502 | 1503 | myfile << "lambda: "; 1504 | myfile << (double)lambda; 1505 | myfile << "\n"; 1506 | 1507 | 1508 | myfile << "iterations: "; 1509 | myfile << std::to_string(iterations); 1510 | myfile << "\n"; 1511 | 1512 | myfile << "wavelet dilation factor: "; 1513 | myfile << std::to_string((float)waveletP / (float)waveletQ); 1514 | myfile << "\n"; 1515 | 1516 | Mat imageInter; 1517 | resize(inputImage, imageInter, srImage.size(), 0.0, 0.0, CV_INTER_CUBIC); 1518 | 1519 | float psnrBC = gs::psnr32F(imageGT, imageInter); 1520 | float psnrSR = gs::psnr32F(imageGT, srImage); 1521 | 1522 | myfile << "PSNR Super Resoltion: "; 1523 | myfile << std::to_string(psnrSR); 1524 | myfile << " PSNR Interpolation: "; 1525 | myfile << std::to_string(psnrBC); 1526 | myfile << " difference: "; 1527 | myfile << std::to_string(psnrSR - psnrBC); 1528 | myfile << "\n"; 1529 | 1530 | imwrite(basePath + "/inputImage_interpolated.png", gs::to8U(imageInter)); 1531 | 1532 | gs::RationalWavelet* rwBC = new gs::RationalWavelet(imageInter, J, waveletP, waveletQ, 1); 1533 | gs::RationalWavelet* rwSR = new gs::RationalWavelet(srImage, J, waveletP, waveletQ, 1); 1534 | 1535 | for (int scale = 0; scale < J; scale++) 1536 | { 1537 | for (int band = 0; band < 3; band++) 1538 | { 1539 | Mat* wBC = rwBC->waveletBand(scale, band); 1540 | Mat* wSR = rwSR->waveletBand(scale, band); 1541 | Mat* wGT = rwGT->waveletBand(scale, band); 1542 | 1543 | psnrBC = gs::psnr32F(*wGT, *wBC); 1544 | psnrSR = gs::psnr32F(*wGT, *wSR); 1545 | 1546 | myfile << "PSNR scale "; 1547 | myfile << std::to_string(scale); 1548 | myfile << ", band "; 1549 | myfile << std::to_string(band); 1550 | myfile << ", Super Resolution: "; 1551 | myfile << std::to_string(psnrSR); 1552 | myfile << " Interpolation: "; 1553 | myfile << std::to_string(psnrBC); 1554 | myfile << " difference: "; 1555 | myfile << std::to_string(psnrSR - psnrBC); 1556 | myfile << "\n"; 1557 | 1558 | 1559 | std::string waveletSRpath("/waveletSR_scale"); 1560 | waveletSRpath.append(std::to_string(scale)); 1561 | waveletSRpath.append("_band"); 1562 | waveletSRpath.append(std::to_string(band)); 1563 | waveletSRpath.append(".png"); 1564 | imwrite(basePath + waveletSRpath, to8U(*wSR)); 1565 | 1566 | std::string waveletBCpath("/waveletInterpolated_scale"); 1567 | waveletBCpath.append(std::to_string(scale)); 1568 | waveletBCpath.append("_band"); 1569 | waveletBCpath.append(std::to_string(band)); 1570 | waveletBCpath.append(".png"); 1571 | imwrite(basePath + waveletBCpath, to8U(*wBC)); 1572 | } 1573 | } 1574 | 1575 | textPath.clear(); 1576 | basePath.clear(); 1577 | imageInter.release(); 1578 | 1579 | myfile.close(); 1580 | 1581 | delete rwBC; 1582 | delete rwSR; 1583 | delete rwGT; 1584 | 1585 | } 1586 | -------------------------------------------------------------------------------- /src/SuperResolution.h: -------------------------------------------------------------------------------- 1 | /* 2 | Greg Smith 2016. 3 | 4 | Single image example-based Super-resolution using direct mapping of self-examples. based on : 5 | Bevilacqua, Marco, et al. "Single-image super-resolution via linear mapping of interpolated self-examples." IEEE Transactions on Image Processing 23.12 (2014): 5334-5347. 6 | 7 | We create a two dictionaries of self example wavelet patches: a high resolution dictionary and a low resolution dictionary. 8 | The high resolution patches are sampled directly from the input image and the low resolution patches are sample from the input 9 | image passed through a low pass filter. We first upscale the image using bicubic interpolation. For each wavelet patch, he K-nearest 10 | neighbor atoms are searched from the low resolution dictionary. The corresponding high resolution dictionary atoms are 11 | combined to estimate a high resolution patch. The patches for each wavelet band and scale are combined to form the super-resolved 12 | image. 13 | 14 | Gradual upscaling is performed to improve the SR results. 15 | 16 | Hyperparameters: 17 | 18 | float. Upscale. Upscaling factor. 19 | int. Iterations. Number of iterations performed to achieve the desired upscaling. 20 | int. Patch Size. Dimension in pixels of the patch size. 21 | int. Patch Overlap. Dimension in pixels of the patch overlap. 22 | float. Lambda. Regularization weight. 23 | int. Neighborhood Size. Number of nearest neighbors used in K-nearest neighbor search. 24 | int. Neighborhood Weight. Atom exponential weighting variance. 25 | 26 | Wavelet only hyperparameters: 27 | int. P. wavelet dilation factor p. 28 | int. Q. wavelet dilation factor q. 29 | 30 | */ 31 | 32 | #pragma once 33 | 34 | #define DEBUG_TEST 1 35 | #define PI (double)3.14159265359 36 | #define WAVELET_THRESHOLD 0.001 37 | 38 | #include 39 | #include 40 | #include 41 | #include 42 | #include 43 | #include 44 | #include 45 | 46 | namespace gs 47 | { 48 | /*Rational Wavelet 49 | Implements a wavelet transform and inverse transform with a rational dilation factor. 50 | author: gsmith 51 | */ 52 | 53 | /* 54 | Wavelet struct 55 | @members, variables: 56 | Mat highbandLH, wavelet band, horizontal low frequency, verical high frequency. 57 | Mat highbandHL, wavelet band, horizontal high frequency, verical low frequency. 58 | Mat highbandHH, wavelet band, horizontal high frequency, verical high frequency. 59 | */ 60 | struct Wavelet 61 | { 62 | cv::Mat highbandLH; 63 | cv::Mat highbandHL; 64 | cv::Mat highbandHH; 65 | }; 66 | 67 | /* 68 | MakeFreqResp 69 | creates two wavelet filters h and g. h is a lowpass filter and g is a high pass filter. 70 | 71 | @param int filtersize, N 72 | @param float p, Wavelet upscale 73 | @param float q, Wavelet downscale 74 | @param float s, Wavelet band downsample fator, normally set to 1. 75 | @param Mat& H, Magnitude of the dft of filter h. 76 | @param Mat& G, Magnitude of the dft of filter g. 77 | @param Mat& h, low pass filter 78 | @param Mat& g, high pass filter. 79 | */ 80 | void makeFreqResp(int filterSize, float p, float q, float s, cv::Mat& H, cv::Mat& G, cv::Mat& h, cv::Mat& g); 81 | 82 | /* 83 | class RationalWavelet 84 | performes the rational wavelet wavelet of an image using the number of scales J, dilation factors p,q and s. 85 | 86 | @members, 87 | functions: 88 | RationalWavelet, constructor, overload 0. 89 | ~RationalWavelet(), destructor. 90 | Mat inverseRationalWavlet(), performs the inverse rational wavelet using Wavelet* wavelet and lowband. 91 | Mat* waveletBand(int scale, int band), returns a pointer to the wavelet band at a specified scale and band 92 | 93 | variables: 94 | Mat lowband, baseband, lowresolution image. 95 | */ 96 | class RationalWavelet 97 | { 98 | public: 99 | 100 | /*constructor 101 | @param Mat image, input image to be decomposed 102 | @param int J, number of scales 103 | @param int p, upscale factor 104 | @param int q, downscale factor 105 | @param int s, wavelet downsampling factor 106 | */ 107 | RationalWavelet(cv::Mat image, int J, int p, int q, int s); 108 | 109 | /*destructor*/ 110 | ~RationalWavelet(); 111 | 112 | /*inverseRationalWavelet 113 | @return Mat, returns the inverse rational wavelet of Wavelet** wavelet 114 | */ 115 | cv::Mat inverseRationalWavelet(); 116 | 117 | /* 118 | waveletBand 119 | @param int scale, 120 | @oaram int band 121 | @return Mat*, returns a pointer to the wavelet band at the specificed scale and band. 122 | */ 123 | cv::Mat* waveletBand(int scale, int band); 124 | 125 | /*baseband low resolution image*/ 126 | cv::Mat lowband; 127 | private: 128 | /*scale and dilation factors*/ 129 | int J, p, q, s; 130 | 131 | /* 132 | analysis filter bank 133 | performs the radwt. Filters and downsamples image x. 134 | 135 | @param Mat* x, input image. 136 | @param Mat* lo, resulting baseband low resolution image. 137 | @param Wavelet* w, resulting wavelet bands. 138 | @param Mat* h, low pass filter bank. 139 | @param Mat* g, high pass filter bank. 140 | @param int p, upscale factor. 141 | @param int q, downscale factor. 142 | @param int s, wavelet downsampling factor. 143 | */ 144 | void afb(cv::Mat* x, cv::Mat* lo, Wavelet* w, cv::Mat* h, cv::Mat* g, int p, int q, int s); 145 | 146 | /* 147 | synthesis filter bank 148 | Reconstructs an image using wavelet bands an a downsampled image. 149 | 150 | @param Mat* lo, input low resolution base band image. Reconstructed image is returned in lo. 151 | @param Wavelet* hi, input wavelet bands. 152 | @param Mat* h, low pass filter bank. 153 | @param Mat* g, high pass filter bank. 154 | @param int p, upscale factor. 155 | @param int q, downscale factor. 156 | @param int s, wavelet downsampling factor. 157 | */ 158 | void sfb(cv::Mat* lo, Wavelet* hi, cv::Mat* h, cv::Mat* g, int p, int q, int s); 159 | 160 | /*array of wavelet bands*/ 161 | Wavelet** wavelet; 162 | 163 | /*converts an unsigned char image into a floating point image.*/ 164 | cv::Mat to32F(cv::Mat im); 165 | 166 | /*converts a floating point image into an unsigned char image*/ 167 | cv::Mat to8U(cv::Mat im); 168 | 169 | /*computes the greatest common denominator between int a and int b*/ 170 | int gcd(int a, int b); 171 | 172 | /*computes the lowest common denominator between int a and int b*/ 173 | int lcm(int a, int b); 174 | }; 175 | 176 | 177 | /* 178 | superResolve 179 | 180 | Takes the input image and applied the super resolution algorithm. 181 | 182 | @param Mat. inputImage. 183 | @param float. upscale. 184 | @param double. lambda. 185 | @param unsigned int. patchSize. 186 | @param unsigned int. patchOverlap. 187 | @param unsigned int. neighborhoodSize. 188 | @param double. neighborhoodWeight. 189 | 190 | @return Mat* 191 | */ 192 | cv::Mat* superResolve(cv::Mat inputImage, float upscale, int iterations, double lambda, unsigned int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight); 193 | 194 | 195 | 196 | /* 197 | superResolve 198 | 199 | Takes the input image and applied the super resolution algorithm. uses a wavelet reconstruction approach. 200 | 201 | @param Mat. inputImage. 202 | @param float. upscale. 203 | @param double. lambda. 204 | @param unsigned int. patchSize. 205 | @param unsigned int. patchOverlap. 206 | @param unsigned int. neighborhoodSize. 207 | @param double. neighborhoodWeight. 208 | 209 | @return Mat* 210 | */ 211 | cv::Mat* superResolveWavelet(cv::Mat inputImage, float upscale, int iterations, double lambda, int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight, int waveletP, int waveletQ); 212 | 213 | 214 | 215 | /* 216 | nearestNeighborAtoms 217 | 218 | nearestNeighborAtoms takes an input patch and searches the low resolution dictionary for the K nearest neighbors. The found low resolution and corresponding high resolution 219 | atoms are weighted exponentially with a variance of 'weight'. 220 | 221 | @param Mat*. patch. The input path to be searched. 222 | @param cv::flann::Index*. tree. Binary tree built from the low resolution dictionary used for efficient searching of the dictionary 223 | @param Mat*. dLow. Low resolution dictionary. 224 | @param Mat*. dHigh. high resolution dictionary. 225 | @param int. K. Number of nearest neighbors to be searched. 226 | @param double. weight. Exponential weighting variance. 227 | @param Mat*. atomsLow. Resulting searched and weighted low resolution atoms. 228 | @param Mat*. atomsHigh. Resulting searched and weighted high resolution atoms. 229 | 230 | @return void 231 | */ 232 | void nearestNeighborAtoms(cv::Mat* patch, int patchSize, cv::flann::Index* tree, cv::Mat* dLow, cv::Mat* dHigh, int K, double weight, cv::Mat* atomsLow, cv::Mat* atomsHigh); 233 | 234 | 235 | /* 236 | createDictionary 237 | creates a high/low resolution coupled dictionary from an image (inputImage). With the SR Wavelet build configs, a wavelet patch dictionary is created. 238 | With wavelets an array of dictionaries is created, one for each scale of the wavelet. 239 | 240 | @param cv::Mat. inputImage 241 | @param float. upscale 242 | 243 | @param Mat*. dLow. Low resolution dictionary (SR full band) 244 | @param Mat*. dHigh. High resolution dictionary (SR full band) 245 | 246 | @param Mat**. dLow. Low resolution dictionary (SR wavelet) 247 | @param Mat**. dHigh. High resolution dictionary (SR wavelet) 248 | 249 | @return void 250 | */ 251 | void createDictionary(cv::Mat inputImage, float gradualUpscale, int patchSize, cv::Mat* dLow, cv::Mat* dHigh); 252 | 253 | void createDictionaryWavelet(cv::Mat inputImage, float gradualUpscale, int patchSize, int waveletJ, int waveletP, int waveletQ, cv::Mat* dLow, cv::Mat* dHigh, int waveletScale, int waveletBand); 254 | 255 | 256 | /* 257 | iterativeBackProjection 258 | 259 | Iterative back projection computes the error between the input image and the low resolution super resolution estimate. We back project the 260 | error back onto the estime for a better estimate. We add an early stopping criteria to avoid gibbs phenomena. 261 | 262 | @param Mat. imageSR. Super resolution estimate. 263 | @param &Mat. imageLR. Input low resolution image. 264 | @param &Mat. result. Backprojected super resolution image. 265 | @param float. upscale. 266 | 267 | @return void 268 | */ 269 | void iterativeBackProjection(cv::Mat imageSR, cv::Mat &imageLR, cv::Mat &result, float upscale); 270 | 271 | 272 | 273 | /* to32F 274 | converts an unsigned char image to a floating point image. 275 | 276 | @param Mat image. an unsigned char image. 277 | 278 | @return Mat, a floating point image. 279 | */ 280 | cv::Mat to32F(cv::Mat image); 281 | 282 | 283 | /* to8U 284 | converts an floating point image to an unsigned char image. 285 | 286 | @param Mat image. a floating point image. 287 | 288 | @return Mat, an unsigned char image. 289 | */ 290 | cv::Mat to8U(cv::Mat image); 291 | 292 | 293 | /* psnr32F 294 | computes the peak signal to noise ration between two images: imageGT, imageRec. 295 | both images are asumed to be floating point images. 296 | 297 | @param Mat imageGT. 298 | @param Mat imageRec. 299 | 300 | @return flaot, psnr 301 | */ 302 | float psnr32F(cv::Mat imageGT, cv::Mat imageRec); 303 | 304 | 305 | /* psnr8U 306 | computes the peak signal to noise ration between two images: imageGT, imageRec. 307 | both images are asumed to unsigned char images. 308 | 309 | @param Mat imageGT. 310 | @param Mat imageRec. 311 | 312 | @return flaot, psnr 313 | */ 314 | float psnr8U(cv::Mat imageGT, cv::Mat imageRec); 315 | 316 | 317 | void exportReport(cv::Mat inputImage, cv::Mat srImage, double upscale, int iterations, double lambda, unsigned int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight); 318 | void exportReport(cv::Mat inputImage, cv::Mat srImage, cv::Mat imageGT, double upscale, int iterations, double lambda, unsigned int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight); 319 | 320 | void exportReportWavelet(cv::Mat inputImage, cv::Mat srImage, double upscale, int iterations, double lambda, unsigned int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight, int waveletP, int waveletQ); 321 | void exportReportWavelet(cv::Mat inputImage, cv::Mat srImage, cv::Mat imageGT, double upscale, int iterations, double lambda, unsigned int patchSize, unsigned int patchOverlap, unsigned int neighborhoodSize, double neighborhoodWeight, int waveletP, int waveletQ); 322 | } --------------------------------------------------------------------------------