8 |
9 | @article{chang2017projector,
10 | title={One Network to Solve Them All ---
11 | Solving Linear Inverse Problems using Deep Projection Models},
12 | author={J. H. Rick Chang and
13 | Chun-Liang Li and
14 | Barnab{\'a}s P{\'o}czos and
15 | B. V. K. Vijaya Kumar and
16 | Aswin C. Sankaranarayanan},
17 | journal={arXiv preprint arXiv:1703.09912},
18 | year={2017}
19 | }
20 |
21 | |
22 |  |
23 |
29 |
where y is the linear measurements, e.g., a low-resolution image, A is the linear operator that generates y from an image x. In image super-resolution, A can indicate direct downsampling or box averaging. The optimization problem is solved with a constraint that the solution x lies in the natural image set X. 30 | 31 | To solve the optimization problem, we found that in proximal algorithms like alternating direction method of multipliers (ADMM), the constraint usually appears in solving a subproblem --- projecting current estimate of x to the set X. Thereby, we propose to learn the projection operator with a deep neural net. Since the projection opereator is independent to individual linear operator A, once the network is trained, it can be used in any linear inverse problem. Since we do not have the exaxt definition of the natural image set, we use the decision boundary of a classifier to approximate the set. 32 | 33 | There are multiple methods to achieve this approximation. For example, given a large image dataset, we can create nonimage signals by perturbing the images in the dataset and then train a classifier to differentiate the two classes. While this method is simple, the decision boundary will be loose. To get a tighter approximation, we found that during the training process, the projected images of the projection network become closer and closer to the natural image set. Thus, if we use these projected images as negative instances, we will learn a tighter decision boundary. This framework is motivated by adversarial generative net. 34 | 35 | Once the projection network is trained, we can solve any linear inverse problem with ADMM. An illustration of the testing process is shown below. 36 |
37 | 
38 |
39 |
40 |
41 |
42 | ## Prerequest
43 | The code is tested under Ubuntu 16.04 with CUDA 8.0 on Titan X Pascal and GTX 1080. We use Python 2.7 and Tensorflow 0.12.1.
44 |
45 | We train the model on two datasets, MS-CELEB-1M dataset and ImageNet dataset. The dataset should be plased under ~/dataset. For example, we put MS-CELEB-1M dataset at ~/dataset/celeb-1m. We load the datasets via load_celeb.py and load_imagenet.py, both can be easily adapt to other datasets. In this tutorial, we take MS-CELEB-1M as an example to illustrate how to use our code. For ImageNet, the usage is alomst the same by replacing ``import load_celeb`` with ``import load_imagenet``. Please see the comments in these files for more information.
46 |
47 | ## Train a Projector
48 | ```bash
49 | cd projector
50 | source run_celeb.sh
51 | ```
52 | The above steps train a projection network on MS-CELEB-1M dataset. Please refer to our paper for the details of parameters. The model files are saved in ```model``` directory.
53 |
54 | ## Run ADMM to solve the Linear Inverse Problems
55 | We have to preprocess the reference batch used in virtual batch normalization for testing. Note that you need to modifiy the filepath of your trained model!
56 | ```bash
57 | cd admm
58 | python update_popmean.py
59 | ```
60 | We then run demo script for differet linear inverse problems. Likewise, you need to modifiy the filepath of your updated model!
61 | ```bash
62 | python paper_demo.py
63 | ```
64 | In our experience, models trained for 50,000 iterations should give you the result similar to we reported in the paper. Note that you may need to adjust the value of alpha (the penalty parameter) for each task and different hyper-parameters used to train the model.
65 |
66 | Here are some sampled result reported in the papers.
67 |
68 |
38 | 
69 |
70 |
71 | ## Trained models
72 | The trained model used in the paper can be found here. A newer version of the model that uses imresize by nearest neighbor algorithm to replace upsampling/downsampling by stride is here. We found that using imresize to perform upsampling and downsampling provides more stable projectors. We have not fully explored this method, so the resulted images may look blurrier than the original model used in the paper.
73 |
74 | To use these models, you need to modify the filepath in admm/update_popmean.py and admm/paper_demo.py. Check the comments in these files. Also note that the alpha in admm/paper_demo.py depends on the dataset, the model, and the problem itself. So like solving the traditional LASSO problems, you need to tune alpha to get nice results. In the file admm/paper_demo.py we provide the values used in the paper to solve the ms-celeb-1m dataset. They provide a good starting point for other datasets and models.
75 |
76 | ## Acknowledgement
77 | Part of our code is based on https://github.com/jazzsaxmafia/Inpainting
78 |
79 |
--------------------------------------------------------------------------------
/admm/add_noise.py:
--------------------------------------------------------------------------------
1 |
2 | import numpy as np
3 |
4 | def exe(x, noise_mean = 0.0, noise_std = 0.1):
5 | noise = np.random.randn(*x.shape) * noise_std + noise_mean;
6 | y = x + noise
7 | return y, noise
8 |
--------------------------------------------------------------------------------
/admm/cs.py:
--------------------------------------------------------------------------------
1 |
2 | import numpy as np
3 | import scipy as sp
4 | from vec import vec
5 | import matplotlib.pyplot as plt
6 |
7 |
8 |
9 | def setup(x_shape, compress_ratio):
10 |
11 | d = np.prod(x_shape).astype(int)
12 | m = np.round(compress_ratio * d).astype(int)
13 |
14 | A = np.random.randn(m,d) / np.sqrt(m)
15 |
16 |
17 | def A_fun(x):
18 | y = np.dot(A, x.ravel(order='F'))
19 | y = np.reshape(y, [1, m], order='F')
20 | return y
21 |
22 | def AT_fun(y):
23 | y = np.reshape(y, [m, 1], order='F')
24 | x = np.dot(A.T, y)
25 | x = np.reshape(x, x_shape, order='F')
26 | return x
27 |
28 | return (A_fun, AT_fun, A)
29 |
30 |
--------------------------------------------------------------------------------
/admm/inpaint.py:
--------------------------------------------------------------------------------
1 |
2 | import numpy as np
3 | from vec import vec
4 |
5 |
6 | def setup(x_shape, drop_prob = 0.5):
7 |
8 | mask = np.random.rand(*x_shape) > drop_prob;
9 | mask = mask.astype('double')
10 |
11 | def A_fun(x):
12 | y = np.multiply(x, mask);
13 | return y
14 |
15 | def AT_fun(y):
16 | x = np.multiply(y, mask);
17 | return x
18 |
19 | return (A_fun, AT_fun, mask)
--------------------------------------------------------------------------------
/admm/inpaint_block.py:
--------------------------------------------------------------------------------
1 |
2 | import numpy as np
3 | import scipy as sp
4 | from vec import vec
5 | import matplotlib.pyplot as plt
6 |
7 |
8 | """ currently only support width (and height) * resize_ratio is an interger! """
9 | def setup(x_shape, box_size, total_box = 1):
10 |
11 | spare = 0.25 * box_size
12 |
13 | mask = np.ones(x_shape)
14 |
15 | for i in range(total_box):
16 |
17 | start_row = spare
18 | end_row = x_shape[1] - spare - box_size - 1
19 | start_col = spare
20 | end_col = x_shape[2] - spare - box_size - 1
21 |
22 | idx_row = int(np.random.rand(1) * (end_row - start_row) + start_row)
23 | idx_col = int(np.random.rand(1) * (end_col - start_col) + start_col)
24 |
25 | mask[0,idx_row:idx_row+box_size,idx_col:idx_col+box_size,:] = 0.
26 |
27 |
28 | def A_fun(x):
29 | y = np.multiply(x, mask);
30 | return y
31 |
32 | def AT_fun(y):
33 | x = np.multiply(y, mask);
34 | return x
35 |
36 | return (A_fun, AT_fun, mask)
37 |
38 |
39 |
--------------------------------------------------------------------------------
/admm/inpaint_center.py:
--------------------------------------------------------------------------------
1 |
2 | import numpy as np
3 | import scipy as sp
4 | from vec import vec
5 | import matplotlib.pyplot as plt
6 |
7 |
8 | def setup(x_shape, box_size):
9 |
10 |
11 | mask = np.ones(x_shape)
12 |
13 |
14 | idx_row = np.round(float(x_shape[1]) / 2.0 - float(box_size) / 2.0).astype(int)
15 | idx_col = np.round(float(x_shape[2]) / 2.0 - float(box_size) / 2.0).astype(int)
16 |
17 | mask[0,idx_row:idx_row+box_size,idx_col:idx_col+box_size,:] = 0.
18 |
19 |
20 | def A_fun(x):
21 | y = np.multiply(x, mask);
22 | return y
23 |
24 | def AT_fun(y):
25 | x = np.multiply(y, mask);
26 | return x
27 |
28 | return (A_fun, AT_fun, mask)
29 |
30 |
31 |
--------------------------------------------------------------------------------
/admm/paper_demo.py:
--------------------------------------------------------------------------------
1 | import sys
2 | sys.path.append("../projector")
3 | import main as model
4 | import tensorflow as tf
5 | import numpy as np
6 | import scipy as sp
7 | import scipy.io
8 | import math
9 | import load_celeb as load_data
10 | import os
11 | import timeit
12 | import matplotlib.pyplot as plt
13 | import add_noise
14 | import solver_paper as solver
15 | import solver_l1 as solver_l1
16 | from PIL import Image
17 | from PIL import ImageFont
18 | from PIL import ImageDraw
19 |
20 |
21 | def save_results(folder, infos, x, z, u):
22 | filename = '%s/infos.mat' % folder
23 | sp.io.savemat(filename, infos)
24 | filename = '%s/x.jpg' % folder
25 | sp.misc.imsave(filename, sp.misc.imresize((reshape_img(np.clip(x, 0.0, 1.0))*255).astype(np.uint8), 4.0, interp='nearest'))
26 | filename = '%s/z.jpg' % folder
27 | sp.misc.imsave(filename, sp.misc.imresize((reshape_img(np.clip(z, 0.0, 1.0))*255).astype(np.uint8), 4.0, interp='nearest'))
28 | filename = '%s/u.jpg' % folder
29 | sp.misc.imsave(filename, sp.misc.imresize((reshape_img(np.clip(u, 0.0, 1.0))*255).astype(np.uint8), 4.0, interp='nearest'))
30 |
31 |
32 | # index of test images
33 | idxs = np.random.randint(391854, size=1)
34 |
35 | # result folder
36 | clean_paper_results = 'clean_paper_results'
37 |
38 | # filename of the trained model. If using virtual batch normalization,
39 | # the popmean and popvariance need to be updated first via update_popmean.py!
40 | iter = 49999
41 | pretrained_folder = os.path.expanduser("../projector/model/imsize64_ratio0.010000_dis0.005000_latent0.000100_img0.001000_de1.000000_derate1.000000_dp1_gd1_softpos0.850000_wdcy_0.000000_seed0")
42 | pretrained_model_file = '%s/update/model_iter-%d' % (pretrained_folder, iter)
43 |
44 |
45 | for idx in idxs :
46 | print 'idx = %d --------' % idx
47 |
48 | np.random.seed(idx)
49 |
50 | img_size = (64,64,3)
51 |
52 | show_img_progress = False # whether the plot intermediate results (may slow down the process)
53 | run_ours = True # whether the run the proposed method
54 | run_l1 = False # whether the run the traditional wavelet sparsity method
55 |
56 | def load_image(filepath):
57 | img = sp.misc.imread(filepath)
58 | img = sp.misc.imresize(img, [64,64]).astype(float) / 255.0
59 | if len(img.shape) < 3:
60 | img = np.tile(img, [1,1,3])
61 | return img
62 |
63 |
64 | #def load_image(filepath):
65 | #img = sp.misc.imread(filepath)
66 | ##