├── .gitignore ├── LICENSE.md ├── README.md ├── examples ├── README.md ├── gauss │ ├── clean.m │ ├── instance.m │ └── model.m ├── nmr1d │ ├── adjoint.m │ ├── clean.m │ ├── fid.dat.gz │ ├── forward.m │ ├── instance.m │ └── model.m ├── nmr2d │ ├── adjoint.m │ ├── fid.dat.gz │ ├── forward.m │ ├── instance.m │ ├── model.m │ ├── mxconj.m │ ├── mxconj2.m │ ├── mxdot.m │ ├── mxdot2.m │ ├── mxfft2.m │ ├── mxifft2.m │ ├── nesta_mx2.m │ └── vbcs_mx2.m ├── spikes-lg │ ├── clean.m │ ├── instance.m │ └── model.m └── spikes-sm │ ├── clean.m │ ├── instance.m │ └── model.m ├── figures ├── README.md ├── gauss │ ├── prepare.m │ └── render.gp ├── nmr1d │ ├── prepare.m │ └── render.gp ├── nmr2d │ ├── cwrite.m │ ├── prepare.m │ └── render.gp └── spikes │ ├── prepare.m │ └── render.gp └── sources ├── README.md ├── build.m ├── gssbl.m ├── ifsbl.m ├── nesta.m ├── util_krylov.m ├── util_lanczos.m ├── vbcs.m ├── vrvm.m ├── vrvm_const.m ├── vrvm_krylov.m └── vrvm_lowrank.m /.gitignore: -------------------------------------------------------------------------------- 1 | # .gitignore 2 | .DS_Store 3 | .*.swp 4 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | 2 | ## MIT License 3 | 4 | Copyright (c) 2017 Bradley Worley 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. 23 | 24 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Variational Bayesian Compressed Sensing 3 | 4 | An extension of the variational relevance vector machine (VRVM) for 5 | highly scalable sparse Bayesian learning in compressed sensing 6 | recovery problems. 7 | 8 | **Note:** This repository is woefully out of date. For peer-reviewed 9 | code, please see [sbl-sandbox](https://github.com/geekysuavo/sbl-sandbox). 10 | 11 | ## Introduction 12 | 13 | Compressed sensing deals with the recovery of signals from incomplete 14 | measurements, using knowledge of the signal structure in some other 15 | domain (e.g. frequency, wavelet, _etc._). While recovery problems 16 | in compressed sensing have been analyzed from the perspective of 17 | Bayesian inference, most of the resulting algorithms do not yield 18 | fully Bayesian estimates, and fully Bayesian approaches scale 19 | poorly. 20 | 21 | The variational Bayesian compressed sensing (VBCS) algorithm extends 22 | the VRVM to bring the computational requirements of fully Bayesian 23 | estimation (or rather its variational approximation) down to levels 24 | that enable modeling of large problem instances. 25 | 26 | ### Source code 27 | 28 | The main source code of **vbcs** is stored in [sources](sources), where 29 | implementations of both VBCS and the VRVM are provided. 30 | 31 | ### Example problems 32 | 33 | Numerical problems demonstrating the features of **vbcs** are stored in 34 | subdirectories of [examples](examples). 35 | 36 | ### Figures 37 | 38 | Scripts for generating the figures in the **vbcs** publication are stored 39 | in subdirectories of [figures](figures). 40 | 41 | ## Licensing 42 | 43 | The **vbcs** sources and example files released under the 44 | [MIT license](https://opensource.org/licenses/MIT). See the 45 | [LICENSE.md](LICENSE.md) file for the complete license terms. 46 | -------------------------------------------------------------------------------- /examples/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## variational-bayes-cs/examples 3 | 4 | This directory contains a small selection of numerical example problems 5 | that VBCS (and potentially the VRVM) may be used to solve. The list of 6 | problems is: 7 | 8 | * [gauss](gauss): Unit impulse positive spikes, Gauss matrix measurements. 9 | * [nmr1d](nmr1d): 1D NMR trace, incomplete Fourier operator measurements. 10 | * [nmr2d](nmr2d): 2D NMR plane, incomplete Fourier operator measurements. 11 | * [spikes-sm](spikes-sm): Small spikes instance, normal matrix measurements. 12 | * [spikes-lg](spikes-lg): Large spikes instance, normal matrix measurements. 13 | 14 | ### Running the scripts 15 | 16 | Within each directory, there are two scripts: **instance.m** and **model.m**. 17 | The instance script is used to generate an instance of the given problem. 18 | Using octave, this is called as: 19 | 20 | ```bash 21 | octave instance.m 22 | ``` 23 | 24 | The model script loads the current problem instance and either builds a 25 | VBCS model or a VRVM model. Using octave, the models are built by calling: 26 | 27 | ```bash 28 | octave model.m vbcs 29 | octave model.m vrvm 30 | ``` 31 | 32 | The VRVM model script can take a very long time to complete, and typically 33 | requires an order of magnitude more memory to work. Results can be cleaned 34 | up by calling: 35 | 36 | ```bash 37 | octave clean.m 38 | ``` 39 | 40 | -------------------------------------------------------------------------------- /examples/gauss/clean.m: -------------------------------------------------------------------------------- 1 | 2 | % remove all output files. 3 | unlink('instance.dat.gz'); 4 | unlink('model-vbcs.dat.gz'); 5 | unlink('model-vrvm.dat.gz'); 6 | 7 | -------------------------------------------------------------------------------- /examples/gauss/instance.m: -------------------------------------------------------------------------------- 1 | 2 | % set the problem sizes. 3 | m = 200; 4 | n = 1000; 5 | k = 5; 6 | 7 | % set the prior parameters. 8 | nu0 = 200; 9 | lambda0 = 0.02; 10 | alpha0 = 1e-9; 11 | beta0 = 1e-9; 12 | 13 | % set the iteration count. 14 | iters = 10000; 15 | 16 | % set the smoothing kernel width. 17 | delta = 50; 18 | 19 | % build a measurement matrix. 20 | A = exp(-([1 : n/m : n]' - [1 : n]).^2 ./ delta^2); 21 | A = diag(1 ./ sqrt(sumsq(A, 2))) * A; 22 | At = []; 23 | 24 | % build the ground truth vector. 25 | id0 = [100 : n - 100]'; 26 | idx = id0(randperm(length(id0))(1 : k)); 27 | x0 = zeros(n, 1); 28 | x0(idx) = 1; 29 | 30 | % sample a noise vector. 31 | z = normrnd(0, 0.01, m, 1); 32 | 33 | % construct the measurement. 34 | y = A * x0 + z; 35 | 36 | % save the current instance. 37 | save('-binary', '-z', 'instance.dat.gz'); 38 | 39 | -------------------------------------------------------------------------------- /examples/gauss/model.m: -------------------------------------------------------------------------------- 1 | 2 | % use the default build mechanism. 3 | addpath('../../sources'); 4 | build 5 | 6 | -------------------------------------------------------------------------------- /examples/nmr1d/adjoint.m: -------------------------------------------------------------------------------- 1 | 2 | % define the adjoint operator. 3 | function x = adjoint (y, schd, n) 4 | % create an infilled vector of y-values. 5 | f = zeros(n, columns(y)); 6 | f(schd,:) = y; 7 | 8 | % compute the fourier transform. 9 | x = fft(f) ./ sqrt(n); 10 | end 11 | 12 | -------------------------------------------------------------------------------- /examples/nmr1d/clean.m: -------------------------------------------------------------------------------- 1 | 2 | % remove all output files. 3 | unlink('instance.dat.gz'); 4 | unlink('model-vbcs.dat.gz'); 5 | unlink('model-vrvm.dat.gz'); 6 | 7 | -------------------------------------------------------------------------------- /examples/nmr1d/fid.dat.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekysuavo/variational-bayes-cs/3f92f2a2bfc29e098aa188b45c0a4e02a820d1d0/examples/nmr1d/fid.dat.gz -------------------------------------------------------------------------------- /examples/nmr1d/forward.m: -------------------------------------------------------------------------------- 1 | 2 | % define the forward operator. 3 | function y = forward (x, schd, n) 4 | % inverse fourier transform and subsample. 5 | f = ifft(x) .* sqrt(n); 6 | y = f(schd,:); 7 | end 8 | 9 | -------------------------------------------------------------------------------- /examples/nmr1d/instance.m: -------------------------------------------------------------------------------- 1 | 2 | % load the input data. 3 | load('fid.dat.gz'); 4 | x0 = x0 ./ max(real(x0)); 5 | 6 | % set the problem sizes. 7 | m = length(sched); 8 | n = 2 * length(x0); 9 | 10 | % set the prior parameters. 11 | nu0 = 50; 12 | lambda0 = 0.005; 13 | alpha0 = 1e-3; 14 | beta0 = 1e-3; 15 | 16 | % set the iteration count. 17 | iters = 500; 18 | 19 | % define the final operators. 20 | A = @(x) forward(x, sched, n); 21 | At = @(y) adjoint(y, sched, n); 22 | 23 | % construct the subsampled measurement. 24 | y = x0(sched); 25 | 26 | % save the current instance. 27 | save('-binary', '-z', 'instance.dat.gz'); 28 | 29 | -------------------------------------------------------------------------------- /examples/nmr1d/model.m: -------------------------------------------------------------------------------- 1 | 2 | % use the default build mechanism. 3 | addpath('../../sources'); 4 | build 5 | 6 | -------------------------------------------------------------------------------- /examples/nmr2d/adjoint.m: -------------------------------------------------------------------------------- 1 | 2 | % define the adjoint operator. 3 | function x = adjoint (y, schd, n) 4 | % create an infilled vector of y-values. 5 | f = zeros(n(1), n(2), 4); 6 | f(schd) = y; 7 | 8 | % compute the fourier transform. 9 | x = mxfft2(f, n); 10 | end 11 | 12 | -------------------------------------------------------------------------------- /examples/nmr2d/fid.dat.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/geekysuavo/variational-bayes-cs/3f92f2a2bfc29e098aa188b45c0a4e02a820d1d0/examples/nmr2d/fid.dat.gz -------------------------------------------------------------------------------- /examples/nmr2d/forward.m: -------------------------------------------------------------------------------- 1 | 2 | % define the forward operator. 3 | function y = forward (x, schd, n) 4 | % inverse fourier transform and subsample. 5 | f = mxifft2(x, n); 6 | y = f(schd); 7 | end 8 | 9 | -------------------------------------------------------------------------------- /examples/nmr2d/instance.m: -------------------------------------------------------------------------------- 1 | 2 | % load the input data. 3 | load('fid.dat.gz'); 4 | 5 | % set the problem sizes. 6 | m = length(sched); 7 | n = size(y0)(1:2); 8 | 9 | % set the linearized sizes. 10 | N = prod(n); 11 | 12 | % build an index set for compacting to m-vectors. 13 | S0 = sched(:,1) + (sched(:,2) - 1) * n(1); 14 | S = bsxfun(@plus, S0, N * [0:3]); 15 | 16 | % set the prior parameters. 17 | nu0 = m; 18 | sigma0 = 0.001; 19 | lambda0 = m * sigma0^2; 20 | alpha0 = 1e-3; 21 | beta0 = 1e-3; 22 | 23 | % set the iteration count. 24 | iters = 500; 25 | 26 | % define the final operators. 27 | A = @(x) forward(x, S, n); 28 | At = @(y) adjoint(y, S, n); 29 | 30 | % construct the subsampled measurement. 31 | y = y0(S); 32 | 33 | % save the current instance. 34 | save('-binary', '-z', 'instance.dat.gz'); 35 | 36 | -------------------------------------------------------------------------------- /examples/nmr2d/model.m: -------------------------------------------------------------------------------- 1 | 2 | % get the command line arguments. 3 | args = argv(); 4 | 5 | % set the model type to its default value. 6 | mtype = 'vbcs'; 7 | 8 | % check if arguments were specified. 9 | if (length(args) && ... 10 | (strcmp(args{end}, 'vbcs') || ... 11 | strcmp(args{end}, 'nesta'))) 12 | % set the new model type. 13 | mtype = args{end}; 14 | end 15 | 16 | % get the model function handle. 17 | mfunc = str2func([mtype, '_mx2']); 18 | 19 | % load the instance data. 20 | load('instance.dat.gz'); 21 | 22 | % execute a timed reconstruction. 23 | tic; 24 | [x, elbo, eta] = mfunc(y, A, At, nu0, lambda0, alpha0, beta0, iters); 25 | runtime = toc; 26 | 27 | % write the results. 28 | save('-binary', '-z', ['model-', mtype, '.dat.gz']); 29 | 30 | -------------------------------------------------------------------------------- /examples/nmr2d/mxconj.m: -------------------------------------------------------------------------------- 1 | 2 | function y = mxconj (x) 3 | y = x; 4 | y(:,1) *= -1; 5 | y(:,2) *= -1; 6 | end 7 | 8 | -------------------------------------------------------------------------------- /examples/nmr2d/mxconj2.m: -------------------------------------------------------------------------------- 1 | 2 | function y = mxconj2 (x) 3 | y = x; 4 | y(:,:,1) *= -1; 5 | y(:,:,2) *= -1; 6 | end 7 | 8 | -------------------------------------------------------------------------------- /examples/nmr2d/mxdot.m: -------------------------------------------------------------------------------- 1 | 2 | % multicomplex one-dimensional inner product. 3 | function d = mxdot (x, y) 4 | z = zeros(size(x)); 5 | 6 | x = mxconj(x); 7 | 8 | z(:,1) = x(:,1) .* y(:,1) ... 9 | - x(:,2) .* y(:,2) ... 10 | - x(:,3) .* y(:,3) ... 11 | - x(:,4) .* y(:,4); 12 | 13 | z(:,2) = x(:,1) .* y(:,2) ... 14 | + x(:,2) .* y(:,1) ... 15 | - x(:,3) .* y(:,4) ... 16 | - x(:,4) .* y(:,3); 17 | 18 | z(:,3) = x(:,1) .* y(:,3) ... 19 | + x(:,3) .* y(:,1) ... 20 | - x(:,2) .* y(:,4) ... 21 | - x(:,4) .* y(:,2); 22 | 23 | z(:,4) = x(:,1) .* y(:,4) ... 24 | + x(:,4) .* y(:,1) ... 25 | + x(:,2) .* y(:,3) ... 26 | + x(:,3) .* y(:,2); 27 | 28 | d = reshape(sum(z, 1), 4, 1); 29 | end 30 | 31 | -------------------------------------------------------------------------------- /examples/nmr2d/mxdot2.m: -------------------------------------------------------------------------------- 1 | 2 | % multicomplex two-dimensional inner product. 3 | function d = mxdot2 (x, y) 4 | z = zeros(size(x)); 5 | 6 | x = mxconj2(x); 7 | 8 | z(:,:,1) = x(:,:,1) .* y(:,:,1) ... 9 | - x(:,:,2) .* y(:,:,2) ... 10 | - x(:,:,3) .* y(:,:,3) ... 11 | - x(:,:,4) .* y(:,:,4); 12 | 13 | z(:,:,2) = x(:,:,1) .* y(:,:,2) ... 14 | + x(:,:,2) .* y(:,:,1) ... 15 | - x(:,:,3) .* y(:,:,4) ... 16 | - x(:,:,4) .* y(:,:,3); 17 | 18 | z(:,:,3) = x(:,:,1) .* y(:,:,3) ... 19 | + x(:,:,3) .* y(:,:,1) ... 20 | - x(:,:,2) .* y(:,:,4) ... 21 | - x(:,:,4) .* y(:,:,2); 22 | 23 | z(:,:,4) = x(:,:,1) .* y(:,:,4) ... 24 | + x(:,:,4) .* y(:,:,1) ... 25 | + x(:,:,2) .* y(:,:,3) ... 26 | + x(:,:,3) .* y(:,:,2); 27 | 28 | d = reshape(sum(sum(z, 1), 2), 4, 1); 29 | end 30 | 31 | -------------------------------------------------------------------------------- /examples/nmr2d/mxfft2.m: -------------------------------------------------------------------------------- 1 | 2 | % two-dimensional multicomplex fft. 3 | function y = mxfft2 (x, n) 4 | % compute the first-dimension fourier transform. 5 | x12 = fft(complex(x(:,:,1), x(:,:,2))) ./ sqrt(n(1)); 6 | x34 = fft(complex(x(:,:,3), x(:,:,4))) ./ sqrt(n(1)); 7 | 8 | % repack. 9 | x(:,:,1) = real(x12); 10 | x(:,:,2) = imag(x12); 11 | x(:,:,3) = real(x34); 12 | x(:,:,4) = imag(x34); 13 | 14 | % compute the second dimension fourier transform. 15 | x13 = fft(complex(x(:,:,1), x(:,:,3)), [], 2) ./ sqrt(n(2)); 16 | x24 = fft(complex(x(:,:,2), x(:,:,4)), [], 2) ./ sqrt(n(2)); 17 | 18 | % repack. 19 | x(:,:,1) = real(x13); 20 | x(:,:,3) = imag(x13); 21 | x(:,:,2) = real(x24); 22 | x(:,:,4) = imag(x24); 23 | 24 | % return. 25 | y = x; 26 | end 27 | 28 | -------------------------------------------------------------------------------- /examples/nmr2d/mxifft2.m: -------------------------------------------------------------------------------- 1 | 2 | % two-dimensional multicomplex inverse fft. 3 | function y = mxifft2 (x, n) 4 | % compute the first-dimension inverse fourier transform. 5 | x12 = ifft(complex(x(:,:,1), x(:,:,2))) .* sqrt(n(1)); 6 | x34 = ifft(complex(x(:,:,3), x(:,:,4))) .* sqrt(n(1)); 7 | 8 | % repack. 9 | x(:,:,1) = real(x12); 10 | x(:,:,2) = imag(x12); 11 | x(:,:,3) = real(x34); 12 | x(:,:,4) = imag(x34); 13 | 14 | % compute the second dimension inverse fourier transform. 15 | x13 = ifft(complex(x(:,:,1), x(:,:,3)), [], 2) .* sqrt(n(2)); 16 | x24 = ifft(complex(x(:,:,2), x(:,:,4)), [], 2) .* sqrt(n(2)); 17 | 18 | % repack. 19 | x(:,:,1) = real(x13); 20 | x(:,:,3) = imag(x13); 21 | x(:,:,2) = real(x24); 22 | x(:,:,4) = imag(x24); 23 | 24 | % return. 25 | y = x; 26 | end 27 | 28 | -------------------------------------------------------------------------------- /examples/nmr2d/nesta_mx2.m: -------------------------------------------------------------------------------- 1 | 2 | % nesterov's algorithm for quadratically constrained l1-minimization, 3 | % for two-dimensional multicomplex data. 4 | % 5 | function [x, elbo, eta] = ... 6 | nesta_mx2 (b, A, At, mu0, lambda0, alpha0, beta0, iters) 7 | % check for the minimum number of arguments. 8 | if (nargin < 3) 9 | error('at least two arguments required'); 10 | end 11 | 12 | % check the required measurement vector. 13 | if (isempty(b) || !ismatrix(b) || columns(b) != 4) 14 | error('measurement must be a 4-column matrix'); 15 | end 16 | 17 | % check the required function handles. 18 | if (isempty(A) || !is_function_handle(A) || ... 19 | isempty(At) || !is_function_handle(At)) 20 | error('invalid measurement: expected function handles'); 21 | end 22 | 23 | % create a function handle for projection. 24 | AtA = @(x) At(A(x)); 25 | 26 | % check for a prior mu parameter. 27 | if (nargin < 4 || isempty(mu0)) 28 | % none specified. use a default value. 29 | mu0 = 1e-3; 30 | end 31 | 32 | % check for a prior lambda parameter. 33 | if (nargin < 5 || isempty(lambda0)) 34 | % none specified. use a default value. 35 | lambda0 = 1e-3; 36 | end 37 | 38 | % check for an iteration count argument. 39 | if (nargin < 8 || isempty(iters)) 40 | % none specified. use a default value. 41 | iters = [10, 50]; 42 | 43 | elseif (isscalar(iters)) 44 | % adjust the iteration scheme to accomodate nesta. 45 | iters = [round(iters / 50), 50]; 46 | end 47 | 48 | % define the gradient of the l1 term. 49 | df = @(x, mu) ... 50 | bsxfun(@times, x ./ mu, sqrt(sumsq(x, 3)) < mu) ... 51 | + bsxfun(@times, sign(x), sqrt(sumsq(x, 3)) >= mu); 52 | 53 | % initialize the transformed data vector. 54 | h = At(b); 55 | 56 | % get the problem sizes. 57 | m = rows(b); 58 | n = prod(size(h)(1:2)); 59 | 60 | % initialize the iterates. 61 | x = zeros(size(h)); 62 | y = zeros(size(h)); 63 | z = zeros(size(h)); 64 | 65 | % set up the vector of thresholds, and compute the max threshold. 66 | muv = linspace(0.9, 0.01, iters(1)); 67 | mu0 = max(vec(sqrt(sumsq(h, 3)))); 68 | 69 | % compute the constant for the quadratic constraint. 70 | vareps = 0.01 * sqrt(m / (mu0 / lambda0)); 71 | 72 | % iterate, outer loop. 73 | for it = 1 : iters(1) 74 | % set the current threshold value and lipschitz constant. 75 | mu = mu0 * muv(it); 76 | L = 1 / mu; 77 | 78 | % initialize the gradient accumulator. 79 | xc = x; 80 | adf = zeros(size(x)); 81 | 82 | % iterate, inner loop. 83 | for jt = 1 : iters(2) 84 | % compute the gradient. 85 | dfdx = df(x, mu); 86 | 87 | % compute the scale factors for the accumulator and iterates. 88 | alpha = (jt + 1) / 2; 89 | tau = 2 / (jt + 3); 90 | 91 | % update the accumulator. 92 | adf += alpha .* dfdx; 93 | 94 | % update the y-iterate. 95 | ly = max(0, (L/vareps) * norm(b - A(x - dfdx ./ L)) - L); 96 | y = x + (ly / L) .* h - dfdx ./ L; 97 | y -= (ly / (ly + L)) .* AtA(y); 98 | 99 | % update the z-iterate. 100 | lz = max(0, (L/vareps) * norm(b - A(xc - adf ./ L)) - L); 101 | z = xc + (lz / L) .* h - adf ./ L; 102 | z -= (lz / (lz + L)) .* AtA(z); 103 | 104 | % update the x-iterate. 105 | x = tau .* z + (1 - tau) .* y; 106 | end 107 | end 108 | 109 | % return empty values for the other outputs. 110 | elbo = []; 111 | eta = []; 112 | end 113 | 114 | -------------------------------------------------------------------------------- /examples/nmr2d/vbcs_mx2.m: -------------------------------------------------------------------------------- 1 | 2 | % modification of vbcs() for two-dimensional multicomplex data, 3 | % under the assumption of a partial fourier measurement. 4 | % 5 | function [x, elbo, eta] = ... 6 | vbcs_mx2 (y, A, At, mu0, lambda0, alpha0, beta0, iters) 7 | % check for the minimum number of arguments. 8 | if (nargin < 3) 9 | error('at least three arguments required'); 10 | end 11 | 12 | % check the required measurement vector. 13 | if (isempty(y) || !ismatrix(y) || columns(y) != 4) 14 | error('measurement must be a 4-column matrix'); 15 | end 16 | 17 | % check the required function handles. 18 | if (isempty(A) || !is_function_handle(A) || ... 19 | isempty(At) || !is_function_handle(At)) 20 | error('invalid measurement: expected function handles'); 21 | end 22 | 23 | % create a function handle for projection. 24 | AtA = @(x) At(A(x)); 25 | 26 | % check for a prior mu parameter. 27 | if (nargin < 4 || isempty(mu0)) 28 | % none specified. use a default value. 29 | mu0 = 1e-3; 30 | end 31 | 32 | % check for a prior lambda parameter. 33 | if (nargin < 5 || isempty(lambda0)) 34 | % none specified. use a default value. 35 | lambda0 = 1e-3; 36 | end 37 | 38 | % check for a prior alpha parameter. 39 | if (nargin < 6 || isempty(alpha0)) 40 | % none specified. use a default value. 41 | alpha0 = 1e-3; 42 | end 43 | 44 | % check for a prior beta parameter. 45 | if (nargin < 7 || isempty(beta0)) 46 | % none specified. use a default value. 47 | beta0 = 1e-3; 48 | end 49 | 50 | % check for an iteration count argument. 51 | if (nargin < 8 || isempty(iters)) 52 | % none specified. use a default value. 53 | iters = 100; 54 | end 55 | 56 | % initialize the transformed data vector. 57 | h = At(y); 58 | 59 | % get the problem sizes. 60 | m = rows(y); 61 | n = prod(size(h)(1:2)); 62 | 63 | % initialize the weight means. 64 | x = zeros(size(h)); 65 | 66 | % initialize the precisions. 67 | xi = repmat(alpha0 ./ beta0, size(h,1), size(h,2)); 68 | 69 | % initialize the noise. 70 | tau = mu0 / lambda0; 71 | 72 | % initialize the projected weight means. 73 | u = AtA(x); 74 | 75 | % compute the projector diagonal, valid only for partial fourier. 76 | g = m / n; 77 | 78 | % compute the constant updated parameters. 79 | alpha = alpha0 + (1/2); 80 | mu = mu0 + m/2; 81 | 82 | % compute the constant portion of the lower bound. 83 | L0 = n * alpha0 * log(beta0) + mu0 * log(lambda0) ... 84 | + n * (lgamma(alpha) - lgamma(alpha0)) ... 85 | + (lgamma(mu) - lgamma(mu0)) ... 86 | + n * alpha + mu - (m/2) * log(2*pi) + n/2; 87 | 88 | % initialize the lower bound vector. 89 | elbo = repmat(L0, iters, 1); 90 | 91 | % compute the inner product of the measurement. 92 | yy = mxdot(y, y)(1); 93 | 94 | % iterate. 95 | for it = 1 : iters 96 | % update the variances. 97 | v = 1 ./ (xi + tau .* g); 98 | 99 | % compute the parallel means and their projection. 100 | xp = tau .* bsxfun(@times, v, h - u + g .* x); 101 | up = AtA(xp); 102 | 103 | % compute the terms of the step size. 104 | T1 = tau .* mxdot2(h, x - xp); 105 | T2 = -mxdot2(x, tau .* u + bsxfun(@times, xi, x)); 106 | T3 = -mxdot2(xp, tau .* up + bsxfun(@times, xi, xp)); 107 | T4 = mxdot2(x, tau .* up + bsxfun(@times, xi, xp)); 108 | 109 | % compute the bounded step size for the mean update. 110 | gamma = (T1 + T2 + T4)(1) / (T2 + T3 + 2*T4)(1); 111 | gamma = min(max(gamma, 1e-3), 1); 112 | 113 | % update the means and their projection. 114 | x = (1 - gamma) .* x + gamma .* xp; 115 | u = (1 - gamma) .* u + gamma .* up; 116 | 117 | % compute some intermediate values. 118 | trGxx = mxdot2(x, u)(1) + sum(vec(g .* v)); 119 | hx = mxdot2(h, x)(1); 120 | x2 = sumsq(x, 3) + v; 121 | 122 | % update the precisions. 123 | beta = beta0 + 0.5 .* x2; 124 | xi = alpha ./ beta; 125 | 126 | % update the noise. 127 | lambda = lambda0 + 0.5 .* yy - hx + 0.5 .* trGxx; 128 | lambda = real(lambda); 129 | tau = mu / lambda; 130 | 131 | % compute the new value of the lower bound. 132 | L = -(tau/2) * (yy - 2 * hx + trGxx) ... 133 | - sum(vec(((alpha/2) * x2 + alpha * beta0) ./ beta)) ... 134 | - mu * log(lambda) - mu * lambda0 / lambda ... 135 | - alpha * sum(vec(log(beta))) ... 136 | + 0.5 * sum(vec(log(v))); 137 | 138 | % store the bound. 139 | elbo(it) += L; 140 | end 141 | 142 | % check if the complete set of variational parameters was requested. 143 | if (nargout >= 3) 144 | % store the parameters over x. 145 | eta.xhat = x; 146 | eta.v = v; 147 | 148 | % store the parameters over xi. 149 | eta.alpha = alpha; 150 | eta.beta = beta; 151 | 152 | % store the parameters over tau. 153 | eta.mu = mu; 154 | eta.lambda = lambda; 155 | end 156 | end 157 | 158 | -------------------------------------------------------------------------------- /examples/spikes-lg/clean.m: -------------------------------------------------------------------------------- 1 | 2 | % remove all output files. 3 | unlink('instance.dat.gz'); 4 | unlink('model-vbcs.dat.gz'); 5 | unlink('model-vrvm.dat.gz'); 6 | 7 | -------------------------------------------------------------------------------- /examples/spikes-lg/instance.m: -------------------------------------------------------------------------------- 1 | 2 | % set the problem sizes. 3 | m = 2000; 4 | n = 10000; 5 | k = 200; 6 | 7 | % set the prior parameters. 8 | nu0 = 200; 9 | lambda0 = 0.005; 10 | alpha0 = 1e-3; 11 | beta0 = 1e-3; 12 | 13 | % set the iteration count. 14 | iters = 500; 15 | 16 | % build a measurement matrix. 17 | A = normrnd(0, 1, m, n); 18 | A = diag(1 ./ sqrt(sumsq(A, 2))) * A; 19 | At = []; 20 | 21 | % sample properties for a ground truth vector. 22 | sgn = 2 * (randi(2, k, 1) - 1.5); 23 | idx = randperm(n)(1 : k)'; 24 | 25 | % build the ground truth vector. 26 | x0 = zeros(n, 1); 27 | x0(idx) = sgn; 28 | 29 | % sample a noise vector. 30 | z = normrnd(0, 0.005, m, 1); 31 | 32 | % construct the measurement. 33 | y = A * x0 + z; 34 | 35 | % save the current instance. 36 | save('-binary', '-z', 'instance.dat.gz'); 37 | 38 | -------------------------------------------------------------------------------- /examples/spikes-lg/model.m: -------------------------------------------------------------------------------- 1 | 2 | % use the default build mechanism. 3 | addpath('../../sources'); 4 | build 5 | 6 | -------------------------------------------------------------------------------- /examples/spikes-sm/clean.m: -------------------------------------------------------------------------------- 1 | 2 | % remove all output files. 3 | unlink('instance.dat.gz'); 4 | unlink('model-vbcs.dat.gz'); 5 | unlink('model-vrvm.dat.gz'); 6 | 7 | -------------------------------------------------------------------------------- /examples/spikes-sm/instance.m: -------------------------------------------------------------------------------- 1 | 2 | % set the problem sizes. 3 | m = 200; 4 | n = 1000; 5 | k = 20; 6 | 7 | % set the prior parameters. 8 | nu0 = 200; 9 | lambda0 = 0.005; 10 | alpha0 = 1e-3; 11 | beta0 = 1e-3; 12 | 13 | % set the iteration count. 14 | iters = 500; 15 | 16 | % build a measurement matrix. 17 | A = normrnd(0, 1, m, n); 18 | A = diag(1 ./ sqrt(sumsq(A, 2))) * A; 19 | At = []; 20 | 21 | % sample properties for a ground truth vector. 22 | sgn = 2 * (randi(2, k, 1) - 1.5); 23 | amp = unifrnd(0.1, 1.1, k, 1); 24 | idx = randperm(n)(1 : k)'; 25 | 26 | % build the ground truth vector. 27 | x0 = zeros(n, 1); 28 | x0(idx) = sgn .* amp; 29 | 30 | % sample a noise vector. 31 | z = normrnd(0, 0.005, m, 1); 32 | 33 | % construct the measurement. 34 | y = A * x0 + z; 35 | 36 | % save the current instance. 37 | save('-binary', '-z', 'instance.dat.gz'); 38 | 39 | -------------------------------------------------------------------------------- /examples/spikes-sm/model.m: -------------------------------------------------------------------------------- 1 | 2 | % use the default build mechanism. 3 | addpath('../../sources'); 4 | build 5 | 6 | -------------------------------------------------------------------------------- /figures/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## variational-bayes-cs/figures 3 | 4 | This directory contains a set of scripts for summarizing some of the 5 | numerical example problems in the [examples](../examples) directory. 6 | The correspondence is: 7 | 8 | * [gauss](gauss) => [examples/gauss](../examples/gauss) 9 | * [nmr1d](nmr1d) => [examples/nmr1d](../examples/nmr1d) 10 | * [nmr2d](nmr2d) => [examples/nmr2d](../examples/nmr2d) 11 | * [spikes](spikes) => [examples/spikes-lg](../examples/spikes-lg) 12 | 13 | ### Generating the figures 14 | 15 | Within each directory, there are two scripts: **prepare.m** and **render.gp**. 16 | The prepare script is used to collect all required data from the corresponding 17 | example directory into a gnuplot-readable text file. Using octave, this is 18 | called as: 19 | 20 | ```bash 21 | octave prepare.m 22 | ``` 23 | 24 | The render script reads the prepared text file and constructs two files: 25 | an EPS file and a TEX file. The files are built by calling: 26 | 27 | ```bash 28 | gnuplot render.gp 29 | ``` 30 | 31 | -------------------------------------------------------------------------------- /figures/gauss/prepare.m: -------------------------------------------------------------------------------- 1 | 2 | % set the input directory. 3 | D = '../../examples/gauss/'; 4 | 5 | % load the instance data. 6 | load([D, 'instance.dat.gz']); 7 | 8 | % load the models. 9 | vbcs = load([D, 'model-vbcs.dat.gz']); 10 | vrvm = load([D, 'model-vrvm.dat.gz']); 11 | 12 | % collect the necessary data. 13 | datx = [[1 : n]', x0, vrvm.x, vbcs.x]; 14 | daty = [[1 : m]', y, A * vrvm.x, A * vbcs.x]; 15 | 16 | % write the data to a text file. 17 | save('-ascii', 'gauss-x.dat', 'datx'); 18 | save('-ascii', 'gauss-y.dat', 'daty'); 19 | 20 | -------------------------------------------------------------------------------- /figures/gauss/render.gp: -------------------------------------------------------------------------------- 1 | 2 | set terminal epslatex color size 2.5in, 2in header \ 3 | '\newcommand{\ft}[0]{\footnotesize}' 4 | 5 | set output 'gauss.tex' 6 | 7 | set lmargin 0.2 8 | set rmargin 0.2 9 | 10 | unset key 11 | set xtics format '' 12 | set xrange [1 : 200] 13 | set x2range [1 : 1000] 14 | set xlabel '$i,j$' off 0, 0.5 15 | set ylabel '$\m{y}^0$, $\m{y}_{\textsc{vbcs}}$' off 3 16 | 17 | p 'gauss-y.dat' u 1:2 w p pt 6 ps 0.5 lc rgb 'black', \ 18 | 'gauss-y.dat' u 1:4 w l lc rgb 'black', \ 19 | "0.1' gauss-x.dat" u 1:(0) \ 20 | w p ax x2y1 pt 1 lw 2.5 lc rgb 'black', \ 21 | "0.1' gauss-x.dat" u 1:(-0.007) \ 22 | w p ax x2y1 pt 9 ps 0.8 lw 2 lc rgb 'black' 23 | 24 | -------------------------------------------------------------------------------- /figures/nmr1d/prepare.m: -------------------------------------------------------------------------------- 1 | 2 | % set the input directory. 3 | D = '../../examples/nmr1d/'; 4 | 5 | % load the instance data. 6 | load([D, 'instance.dat.gz']); 7 | 8 | % load the models. 9 | vbcs = load([D, 'model-vbcs.dat.gz']); 10 | vrvm = load([D, 'model-vrvm.dat.gz']); 11 | nesta = load([D, 'model-nesta.dat.gz']); 12 | 13 | % collect the necessary data. 14 | ax = linspace(-pi, pi, n)'; 15 | dat = [fft(x0, n) ./ sqrt(n), nesta.x, vrvm.x, vbcs.x]; 16 | dat = [ax, real(fftshift(dat, 1))]; 17 | 18 | % write the data to a text file. 19 | save('-ascii', 'nmr1d.dat', 'dat'); 20 | 21 | -------------------------------------------------------------------------------- /figures/nmr1d/render.gp: -------------------------------------------------------------------------------- 1 | 2 | set terminal epslatex color size 2.5in, 5in header \ 3 | '\newcommand{\ft}[0]{\footnotesize}' 4 | 5 | set output 'nmr1d.tex' 6 | 7 | set lmargin 0.2 8 | set rmargin 0.2 9 | 10 | set multiplot layout 3, 1 11 | 12 | unset key 13 | set xlabel '$\omega$' off 0, 0.5 14 | set xtics ('{\ft $-\pi$}' -pi, '{\ft %g}' 0, '{\ft $\pi$}' pi) 15 | set ytics ('{\ft %g}' 0) 16 | unset ytics 17 | set xrange [-pi : pi] 18 | 19 | set yrange [-0.02 : 0.7] 20 | set ylabel '$\m{\Phi}^* \m{y}^0$' 21 | p 'nmr1d.dat' u 1:2 w l lc rgb 'black' 22 | 23 | set yrange [-0.05 : 1.4] 24 | set ylabel '$\xhat_{\textsc{nesta}}$' 25 | p 'nmr1d.dat' u 1:3 w l lc rgb 'black' 26 | 27 | set yrange [-0.1 : 4.4] 28 | set ylabel '$\xhat_{\textsc{vbcs}}$' 29 | p 'nmr1d.dat' u 1:5 w l lc rgb 'black' 30 | 31 | unset multiplot 32 | 33 | -------------------------------------------------------------------------------- /figures/nmr2d/cwrite.m: -------------------------------------------------------------------------------- 1 | 2 | % cwrite: write the contours of a matrix to a text file. 3 | % 4 | function cwrite (x, lev, fname) 5 | % compute the contours. 6 | C = contourc(x, lev); 7 | N = columns(C); 8 | ii = 1; 9 | 10 | % open the output file. 11 | fh = fopen(fname, 'w'); 12 | 13 | % loop over each curve. 14 | while (ii <= N) 15 | % break each curve with a blank line. 16 | if (ii > 1) 17 | fprintf(fh, '\n'); 18 | end 19 | 20 | % get the current curve size and height. 21 | z = C(1, ii); 22 | n = C(2, ii); 23 | 24 | % get the current curve points. 25 | x = C(1, ii+1 : ii+n); 26 | y = C(2, ii+1 : ii+n); 27 | 28 | % write the current curve. 29 | for jj = 1 : n 30 | fprintf(fh, '%e %e %e\n', x(jj), y(jj), z); 31 | end 32 | 33 | % increment the counter. 34 | ii += n + 1; 35 | end 36 | 37 | % close the output file. 38 | fclose(fh); 39 | end 40 | 41 | -------------------------------------------------------------------------------- /figures/nmr2d/prepare.m: -------------------------------------------------------------------------------- 1 | 2 | % set the input directory. 3 | D = '../../examples/nmr2d/'; 4 | addpath(D); 5 | 6 | % load the instance data. 7 | load([D, 'instance.dat.gz']); 8 | 9 | % load the models. 10 | vbcs = load([D, 'model-vbcs.dat.gz']); 11 | nesta = load([D, 'model-nesta.dat.gz']); 12 | 13 | % set the slice indices. 14 | hset = [1453, 1584, 1657, 1736]; 15 | nset = [866, 867, 868]; 16 | 17 | % set the contours. 18 | clev0 = logspace(log10(0.01), log10(0.5), 20); 19 | vbcs.clev = logspace(log10(0.02), log10(4), 20); 20 | nesta.clev = logspace(log10(0.01), log10(2), 20); 21 | 22 | % set the vbcs variance contours. 23 | vbcs.vlev = linspace(0.16, 0.26, 10); 24 | 25 | % compute the ground truth spectrum. 26 | x0 = fftshift(mxifft2(y0, n)(:,:,1)); 27 | 28 | % compute the estimated spectra. 29 | vbcs.xr = fftshift(vbcs.x(:,:,1)); 30 | nesta.xr = fftshift(nesta.x(:,:,1)); 31 | 32 | % compute the standard deviation. 33 | vbcs.vr = sqrt(fftshift(vbcs.eta.v)); 34 | 35 | % compute the slices. 36 | vbcs.sh = [[1 : n(2)]', vbcs.xr(hset,:)', vbcs.vr(hset,:)']; 37 | vbcs.sn = [[1 : n(1)]', vbcs.xr(:,nset), vbcs.vr(:,nset) ]; 38 | nesta.sh = [[1 : n(2)]', nesta.xr(hset,:)']; 39 | nesta.sn = [[1 : n(1)]', nesta.xr(:,nset) ]; 40 | 41 | % write contours from each spectrum. 42 | cwrite(x0, clev0, 'nmr2d-orig.dat'); 43 | cwrite(vbcs.xr, vbcs.clev, 'nmr2d-vbcs.dat'); 44 | cwrite(nesta.xr, nesta.clev, 'nmr2d-nesta.dat'); 45 | 46 | % write the variance contours. 47 | cwrite(vbcs.vr, vbcs.vlev, 'nmr2d-var.dat'); 48 | 49 | % write the slices. 50 | dat = vbcs.sh; save('-ascii', 'nmr2d-vbcs-h1.dat', 'dat'); 51 | dat = vbcs.sn; save('-ascii', 'nmr2d-vbcs-n15.dat', 'dat'); 52 | dat = nesta.sh; save('-ascii', 'nmr2d-nesta-h1.dat', 'dat'); 53 | dat = nesta.sn; save('-ascii', 'nmr2d-nesta-n15.dat', 'dat'); 54 | 55 | -------------------------------------------------------------------------------- /figures/nmr2d/render.gp: -------------------------------------------------------------------------------- 1 | 2 | set terminal epslatex color size 6in, 2in header \ 3 | '\newcommand{\ft}[0]{\footnotesize}' 4 | 5 | lx = 0.05 6 | ly = 0.9 7 | 8 | set output 'nmr2d.tex' 9 | 10 | #set lmargin 0.2 11 | set rmargin 0.2 12 | 13 | set multiplot layout 1, 3 14 | 15 | unset key 16 | set xlabel '$\omega_1$' off 0, 0.5 17 | set ylabel '$\omega_2$' 18 | unset xtics 19 | unset ytics 20 | unset colorbox 21 | set palette gray 22 | 23 | set xrange [-1475 : -415] 24 | set yrange [200 : 710] 25 | 26 | set label 1 '$\m{\Phi}^* \m{y}^0$' at graph lx, ly 27 | p 'nmr2d-orig.dat' u (-$1):2:3 w l lc pal 28 | 29 | set xrange [595 : 1630] 30 | set yrange [-1855 : -1344] 31 | 32 | set label 1 '$\xhat_{\textsc{nesta}}$' at graph lx, ly 33 | p 'nmr2d-nesta.dat' u 1:(-$2):3 w l lc pal 34 | 35 | set label 1 '$\xhat_{\textsc{vbcs}}$' at graph lx, ly 36 | p 'nmr2d-vbcs.dat' u 1:(-$2):3 w l lc pal 37 | 38 | unset multiplot 39 | 40 | -------------------------------------------------------------------------------- /figures/spikes/prepare.m: -------------------------------------------------------------------------------- 1 | 2 | % set the input directory. 3 | D = '../../examples/spikes-lg/'; 4 | 5 | % load the instance data. 6 | load([D, 'instance.dat.gz']); 7 | 8 | % load the models. 9 | vbcs = load([D, 'model-vbcs.dat.gz']); 10 | vrvm = load([D, 'model-vrvm.dat.gz']); 11 | 12 | % collect the necessary data. 13 | dat = [[1 : n]', x0, vrvm.x, vbcs.x]; 14 | 15 | % write the data to a text file. 16 | save('-ascii', 'spikes.dat', 'dat'); 17 | 18 | -------------------------------------------------------------------------------- /figures/spikes/render.gp: -------------------------------------------------------------------------------- 1 | 2 | set terminal epslatex color size 2.5in, 5in header \ 3 | '\newcommand{\ft}[0]{\footnotesize}' 4 | 5 | set output 'spikes.tex' 6 | 7 | set lmargin 0.2 8 | set rmargin 0.2 9 | 10 | set multiplot layout 3, 1 11 | 12 | unset key 13 | set xlabel '$i$' off 0, 0.5 14 | set xtics 5000 format '{\ft %g}' 15 | set ytics 1 format '{\ft %g}' 16 | set xrange [1 : 10000] 17 | set yrange [-1.2 : 1.2] 18 | 19 | set ylabel '$\m{x}^0$' off 3 20 | p 'spikes.dat' u 1:2 w l lc rgb 'black' 21 | 22 | set ylabel '$\xhat_{\textsc{vrvm}}$' 23 | p 'spikes.dat' u 1:3 w l lc rgb 'black' 24 | 25 | set ylabel '$\xhat_{\textsc{vbcs}}$' 26 | p 'spikes.dat' u 1:4 w l lc rgb 'black' 27 | 28 | unset multiplot 29 | 30 | -------------------------------------------------------------------------------- /sources/README.md: -------------------------------------------------------------------------------- 1 | 2 | ## variational-bayes-cs/sources 3 | 4 | This directory contains the main source code of the VBCS and VRVM 5 | implementations used in the associated publication on VBCS. Each 6 | of the two implementations may be used as a standalone Octave or 7 | MATLAB function file. 8 | 9 | -------------------------------------------------------------------------------- /sources/build.m: -------------------------------------------------------------------------------- 1 | 2 | % get the command line arguments. 3 | args = argv(); 4 | 5 | % set the model type to its default value. 6 | mtype = 'vbcs'; 7 | 8 | % check if arguments were specified. 9 | if (length(args) && ... 10 | (strcmp(args{end}, 'vbcs') || ... 11 | strcmp(args{end}, 'vrvm') || ... 12 | strcmp(args{end}, 'vrvm_lowrank') || ... 13 | strcmp(args{end}, 'vrvm_krylov') || ... 14 | strcmp(args{end}, 'gssbl') || ... 15 | strcmp(args{end}, 'ifsbl') || ... 16 | strcmp(args{end}, 'nesta'))) 17 | % set the new model type. 18 | mtype = args{end}; 19 | end 20 | 21 | % get the model function handle. 22 | mfunc = str2func(mtype); 23 | 24 | % read in the source files. 25 | addpath('../../sources'); 26 | 27 | % load the instance data. 28 | load('instance.dat.gz'); 29 | 30 | % execute a timed reconstruction. 31 | tic; 32 | [mu, obj, parms] = mfunc(y, A, At, nu0, lambda0, alpha0, beta0, iters); 33 | runtime = toc; 34 | 35 | % write the results. 36 | save('-binary', '-z', ['model-', mtype, '.dat.gz']); 37 | 38 | -------------------------------------------------------------------------------- /sources/gssbl.m: -------------------------------------------------------------------------------- 1 | 2 | % -*- texinfo -*- 3 | % @deftypefn {} {[@var{mu}, @dots{}] =} gssbl (@var{y}, @var{A}, @dots{}) 4 | % Recover a sparse signal @var{x} from incomplete data @var{y} and a 5 | % measurement matrix (@var{A} or @var{A},@var{At}) using a Gibbs 6 | % sampler for the sparse Bayesian learning model. 7 | % 8 | % See @ref{vrvm} for detailed usage information. 9 | % @end deftypefn 10 | % 11 | function [mu, obj, parms] = ... 12 | gssbl (y, A, At, nu0, lambda0, alpha0, beta0, iters, thin) 13 | % check for the minimum number of arguments. 14 | if (nargin < 2) 15 | error('at least two arguments required'); 16 | end 17 | 18 | % check the required measurement vector. 19 | if (isempty(y) || !isvector(y) || !iscolumn(y)) 20 | error('measurement must be a vector'); 21 | end 22 | 23 | % check the required measurement operator. 24 | if (!isempty(A) && (nargin < 3 || isempty(At))) 25 | % matrix specification. check the data type. 26 | if (!ismatrix(A)) 27 | error('invalid measurement: expected matrix'); 28 | end 29 | 30 | % store copies of the measurement matrix and its gramian. 31 | B = A; 32 | G = A' * A; 33 | 34 | % create function handles for forward, inverse, and projection. 35 | A = @(x) B * x; 36 | At = @(y) B' * y; 37 | AtA = @(x) G * x; 38 | 39 | elseif (!isempty(A) && nargin >= 3 && !isempty(At)) 40 | % function handle specification. check the data types. 41 | if (!is_function_handle(A) || !is_function_handle(At)) 42 | error('invalid measurement: expected function handles'); 43 | end 44 | 45 | % create a function handle for projection. 46 | AtA = @(x) At(A(x)); 47 | end 48 | 49 | % check for a prior nu parameter. 50 | if (nargin < 4 || isempty(nu0)) 51 | % none specified. use a default value. 52 | nu0 = 1e-3; 53 | end 54 | 55 | % check for a prior lambda parameter. 56 | if (nargin < 5 || isempty(lambda0)) 57 | % none specified. use a default value. 58 | lambda0 = 1e-3; 59 | end 60 | 61 | % check for a prior alpha parameter. 62 | if (nargin < 6 || isempty(alpha0)) 63 | % none specified. use a default value. 64 | alpha0 = 1e-3; 65 | end 66 | 67 | % check for a prior beta parameter. 68 | if (nargin < 7 || isempty(beta0)) 69 | % none specified. use a default value. 70 | beta0 = 1e-3; 71 | end 72 | 73 | % check for an iteration count argument. 74 | if (nargin < 8 || isempty(iters)) 75 | % none specified. use a default value. 76 | iters = 100; 77 | end 78 | 79 | % check for a thinning rate argument. 80 | if (nargin < 9 || isempty(thin)) 81 | % none specified. use a default value. 82 | thin = 1; 83 | end 84 | 85 | % initialize the current sample. 86 | x = At(y); 87 | 88 | % get the problem sizes. 89 | m = length(y); 90 | n = length(x); 91 | 92 | % initialize the sample store. 93 | X = zeros(n, floor(iters / thin)); 94 | 95 | % compute the constant updated parameters. 96 | alpha = alpha0 + (1/2); 97 | nu = nu0 + m/2; 98 | 99 | % initialize the objective vector. 100 | stdnrm = @(N) normrnd(0, 1, [N, 1]); 101 | obj = []; 102 | 103 | figure(1); 104 | drawnow(); 105 | 106 | % iterate. 107 | for it = 1 : iters 108 | % update the tau rate parameter. 109 | lambda = lambda0 + 0.5 * norm(y - A(x))^2; 110 | 111 | % sample tau. 112 | tau = gamrnd(nu, 1 / lambda); 113 | 114 | % update the xi rate parameter. 115 | beta = beta0 + 0.5 .* abs(x).^2; 116 | 117 | % sample xi, with a lower bound. 118 | xi = bsxfun(@gamrnd, repmat(alpha, n, 1), 1 ./ beta); 119 | 120 | % sample independent normal vectors for sampling x. 121 | z1 = stdnrm(m); 122 | z2 = stdnrm(n); 123 | 124 | % build a sample from the distribution N(0, Sigma^-1). 125 | z3 = sqrt(tau) .* At(z1) + sqrt(xi) .* z2; 126 | 127 | % compute the right-hand vector. 128 | u = tau .* At(y) + z3; 129 | v = A(u ./ xi); 130 | 131 | % define the kernel matrix operator, and solve K*d = v. 132 | K = @(z) z ./ tau + A(At(z) ./ xi); 133 | d = K(eye(m)) \ v; 134 | 135 | % sample x. 136 | x = (u - At(d)) ./ xi; 137 | 138 | % store the sample. 139 | if (mod(it, thin) == 0) 140 | X(:,it/thin) = x; 141 | end 142 | end 143 | 144 | % compute the mean result. 145 | mu = mean(X, 2); 146 | 147 | % check if the complete set of variational parameters was requested. 148 | if (nargout >= 3) 149 | % store the parameters over x. 150 | parms.mu = mu; 151 | parms.sigma = var(X, [], 2); 152 | 153 | % store the samples. 154 | parms.X = X; 155 | end 156 | end 157 | 158 | -------------------------------------------------------------------------------- /sources/ifsbl.m: -------------------------------------------------------------------------------- 1 | 2 | % -*- texinfo -*- 3 | % @deftypefn {} {[@var{mu}, @dots{}] =} ifsbl (@var{y}, @var{A}, @dots{}) 4 | % Recover a sparse signal @var{x} from incomplete data @var{y} and a 5 | % measurement matrix (@var{A} or @var{A},@var{At}) using 6 | % Inverse-Free Sparse Bayesian Learning. 7 | % 8 | % See @ref{vrvm} for detailed usage information. 9 | % @end deftypefn 10 | % 11 | function [mu, obj, parms] = ... 12 | ifsbl (y, A, At, nu0, lambda0, alpha0, beta0, iters) 13 | % check for the minimum number of arguments. 14 | if (nargin < 2) 15 | error('at least two arguments required'); 16 | end 17 | 18 | % check the required measurement vector. 19 | if (isempty(y) || !isvector(y) || !iscolumn(y)) 20 | error('measurement must be a vector'); 21 | end 22 | 23 | % check the required measurement operator. 24 | if (!isempty(A) && (nargin < 3 || isempty(At))) 25 | % matrix specification. check the data type. 26 | if (!ismatrix(A)) 27 | error('invalid measurement: expected matrix'); 28 | end 29 | 30 | % store copies of the measurement matrix and its gramian. 31 | B = A; 32 | G = A' * A; 33 | 34 | % create function handles for forward, inverse, and projection. 35 | A = @(x) B * x; 36 | At = @(y) B' * y; 37 | AtA = @(x) G * x; 38 | 39 | elseif (!isempty(A) && nargin >= 3 && !isempty(At)) 40 | % function handle specification. check the data types. 41 | if (!is_function_handle(A) || !is_function_handle(At)) 42 | error('invalid measurement: expected function handles'); 43 | end 44 | 45 | % create a function handle for projection. 46 | AtA = @(x) At(A(x)); 47 | end 48 | 49 | % check for a prior nu parameter. 50 | if (nargin < 4 || isempty(nu0)) 51 | % none specified. use a default value. 52 | nu0 = 1e-3; 53 | end 54 | 55 | % check for a prior lambda parameter. 56 | if (nargin < 5 || isempty(lambda0)) 57 | % none specified. use a default value. 58 | lambda0 = 1e-3; 59 | end 60 | 61 | % check for a prior alpha parameter. 62 | if (nargin < 6 || isempty(alpha0)) 63 | % none specified. use a default value. 64 | alpha0 = 1e-3; 65 | end 66 | 67 | % check for a prior beta parameter. 68 | if (nargin < 7 || isempty(beta0)) 69 | % none specified. use a default value. 70 | beta0 = 1e-3; 71 | end 72 | 73 | % check for an iteration count argument. 74 | if (nargin < 8 || isempty(iters)) 75 | % none specified. use a default value. 76 | iters = 100; 77 | end 78 | 79 | % initialize the transformed data vector. 80 | h = At(y); 81 | 82 | % get the problem sizes. 83 | m = length(y); 84 | n = length(h); 85 | 86 | % initialize the weight means. 87 | mu = zeros(n, 1); 88 | z = mu; 89 | 90 | % initialize the precisions. 91 | xi = repmat(alpha0 / beta0, n, 1); 92 | 93 | % initialize the noise. 94 | tau = nu0 / lambda0; 95 | 96 | % compute the bounding lipschitz constant and projector diagonal, 97 | % in the most general way possible. 98 | if (isreal(A(eye(n)))) 99 | eigopt = 'lm'; 100 | else 101 | eigopt = 'lr'; 102 | end 103 | T = 2 * real(eigs(AtA(eye(n)), 1, eigopt)); 104 | 105 | % define the expectation of the bound on the norm, E[g(x,z)] 106 | g = @(z, mu, sigma) ... 107 | norm(y - A(z))^2 + 2 * (mu - z)' * At(A(z) - y) + ... 108 | (T/2) * (norm(mu - z)^2 + sum(sigma)); 109 | 110 | % compute the constant updated parameters. 111 | alpha = alpha0 + (1/2); 112 | nu = nu0 + m/2; 113 | 114 | % compute the constant portion of the objective. 115 | phi0 = vrvm_const(m, n, nu0, lambda0, alpha0, beta0); 116 | 117 | % initialize the objective vector. 118 | obj = repmat(phi0, iters, 1); 119 | 120 | % iterate. 121 | for it = 1 : iters 122 | % update the variances. 123 | sigma = 1 ./ (tau * (T/2) + xi); 124 | 125 | % update the means. 126 | mu = tau .* sigma .* (h + (T/2) .* z - AtA(z)); 127 | 128 | % compute the log-determinant. 129 | lndetS = sum(log(sigma)); 130 | 131 | % update the precisions. 132 | mu2 = conj(mu) .* mu + sigma; 133 | beta = beta0 + 0.5 .* mu2; 134 | xi = alpha ./ beta; 135 | 136 | % update the noise. 137 | lambda = lambda0 + 0.5 * real(g(z, mu, sigma)); 138 | tau = nu / lambda; 139 | 140 | % update the bounding parameter. 141 | z = mu; 142 | 143 | % compute the objective function value. 144 | phi = nu * log(lambda) + alpha * sum(log(beta)) - 0.5 * lndetS; 145 | 146 | % store the objective. 147 | obj(it) += phi; 148 | end 149 | 150 | % check if the complete set of variational parameters was requested. 151 | if (nargout >= 3) 152 | % store the parameters over x. 153 | parms.mu = mu; 154 | parms.sigma = sigma; 155 | 156 | % store the parameters over tau. 157 | parms.nu = nu; 158 | parms.lambda = lambda; 159 | 160 | % store the parameters over xi. 161 | parms.alpha = alpha; 162 | parms.beta = beta; 163 | end 164 | end 165 | 166 | -------------------------------------------------------------------------------- /sources/nesta.m: -------------------------------------------------------------------------------- 1 | 2 | % -*- texinfo -*- 3 | % @deftypefn {} {[@var{mu}, @dots{}] =} nesta (@var{y}, @var{A}, @dots{}) 4 | % Recover a sparse signal @var{x} from incomplete data @var{y} and a 5 | % measurement matrix (@var{A} or @var{A},@var{At}) using Nesterov's 6 | % algorithm for quadratically constrained l1-norm minimization. 7 | % 8 | % See @ref{vrvm} for detailed usage information. 9 | % @end deftypefn 10 | % 11 | function [x, obj, parms] = ... 12 | nesta (b, A, At, nu0, lambda0, alpha0, beta0, iters) 13 | % check for the minimum number of arguments. 14 | if (nargin < 2) 15 | error('at least two arguments required'); 16 | end 17 | 18 | % check the required measurement vector. 19 | if (isempty(b) || !isvector(b) || !iscolumn(b)) 20 | error('measurement must be a vector'); 21 | end 22 | 23 | % check the required measurement operator. 24 | if (!isempty(A) && (nargin < 3 || isempty(At))) 25 | % matrix specification. check the data type. 26 | if (!ismatrix(A)) 27 | error('invalid measurement: expected matrix'); 28 | end 29 | 30 | % store copies of the measurement matrix and its gramian. 31 | B = A; 32 | G = A' * A; 33 | 34 | % create function handles for forward, inverse, and projection. 35 | A = @(x) B * x; 36 | At = @(y) B' * y; 37 | AtA = @(x) G * x; 38 | 39 | elseif (!isempty(A) && nargin >= 3 && !isempty(At)) 40 | % function handle specification. check the data types. 41 | if (!is_function_handle(A) || !is_function_handle(At)) 42 | error('invalid measurement: expected function handles'); 43 | end 44 | 45 | % create a function handle for projection. 46 | AtA = @(x) At(A(x)); 47 | end 48 | 49 | % check for a prior nu parameter. 50 | if (nargin < 4 || isempty(nu0)) 51 | % none specified. use a default value. 52 | nu0 = 1e-3; 53 | end 54 | 55 | % check for a prior lambda parameter. 56 | if (nargin < 5 || isempty(lambda0)) 57 | % none specified. use a default value. 58 | lambda0 = 1e-3; 59 | end 60 | 61 | % check for an iteration count argument. 62 | if (nargin < 8 || isempty(iters)) 63 | % none specified. use a default value. 64 | iters = [10, 50]; 65 | 66 | elseif (isscalar(iters)) 67 | % adjust the iteration scheme to accomodate nesta. 68 | iters = [round(iters / 50), 50]; 69 | end 70 | 71 | % define the gradient of the l1 term. 72 | df = @(x, nu) (x ./ nu) .* (abs(x) < nu) + sign(x) .* (abs(x) >= nu); 73 | 74 | % initialize the transformed data vector. 75 | h = At(b); 76 | 77 | % get the problem sizes. 78 | m = length(b); 79 | n = length(h); 80 | 81 | % initialize the iterates. 82 | x = zeros(n, 1); 83 | y = zeros(n, 1); 84 | z = zeros(n, 1); 85 | 86 | % set up the vector of thresholds, and compute the max threshold. 87 | nuv = linspace(0.9, 0.01, iters(1)); 88 | nu0 = max(abs(h)); 89 | 90 | % compute the constant for the quadratic constraint. 91 | vareps = 0.1 * sqrt(m / (nu0 / lambda0)); 92 | 93 | % initialize the objective vector. 94 | obj = repmat(0, prod(iters), 1); 95 | 96 | % iterate, outer loop. 97 | for it = 1 : iters(1) 98 | % set the current threshold value and lipschitz constant. 99 | nu = nu0 * nuv(it); 100 | L = 1 / nu; 101 | 102 | % initialize the gradient accumulator. 103 | xc = x; 104 | adf = zeros(size(x)); 105 | 106 | % iterate, inner loop. 107 | for jt = 1 : iters(2) 108 | % compute the gradient. 109 | dfdx = df(x, nu); 110 | 111 | % compute the scale factors for the accumulator and iterates. 112 | alpha = (jt + 1) / 2; 113 | tau = 2 / (jt + 3); 114 | 115 | % update the accumulator. 116 | adf += alpha .* dfdx; 117 | 118 | % update the y-iterate. 119 | ly = max(0, (L/vareps) * norm(b - A(x - dfdx ./ L)) - L); 120 | y = x + (ly / L) .* h - dfdx ./ L; 121 | y -= (ly / (ly + L)) .* AtA(y); 122 | 123 | % update the z-iterate. 124 | lz = max(0, (L/vareps) * norm(b - A(xc - adf ./ L)) - L); 125 | z = xc + (lz / L) .* h - adf ./ L; 126 | z -= (lz / (lz + L)) .* AtA(z); 127 | 128 | % update the x-iterate. 129 | x = tau .* z + (1 - tau) .* y; 130 | 131 | % store the objective. 132 | obj((it-1) * iters(2) + jt) = sum(abs(x)); 133 | end 134 | end 135 | 136 | % return an empty set of variational parameters. 137 | parms = []; 138 | end 139 | 140 | -------------------------------------------------------------------------------- /sources/util_krylov.m: -------------------------------------------------------------------------------- 1 | 2 | % -*- texinfo -*- 3 | % @deftypefn {} {[@var{mu}, @var{sigma}, @var{f}] =} util_krylov (@ 4 | % @var{A}, @var{At}, @var{tau}, @var{xi}, @var{y}, @var{y0}) 5 | % Compute the solution to a linear system and the diagonal of the 6 | % inverse coefficient matrix using Lanczos iteration. 7 | % 8 | % The outputs @var{mu} and @var{sigma} are the solution and 9 | % diagonal, and @var{f} holds an estimate of the determinant 10 | % of the kernel matrix. 11 | % @end deftypefn 12 | % 13 | function [mu, sigma, f] = util_krylov (A, At, tau, xi, y, y0) 14 | % get the problem sizes. 15 | n = length(xi); 16 | m = length(y); 17 | 18 | % define the matrix operators. 19 | J = @(x) tau .* At(A(x)) + xi .* x; 20 | K = @(y) y ./ tau + A(At(y) ./ xi); 21 | 22 | % initialize the tridiagonal matrix elements. 23 | omega = []; 24 | gamma = []; 25 | 26 | % initialize the lanczos vector. 27 | q = y ./ norm(y); 28 | Q = q; 29 | 30 | % initialize the transformed lanczos vector. 31 | v = At(q); 32 | V = v; 33 | 34 | % compute the first conjugate vector. 35 | p = K(q); 36 | omega = real(q' * p); 37 | p = p - omega .* q; 38 | 39 | % initialize the recurrence variables. 40 | f2 = 0; 41 | f1 = 1; 42 | f = omega; 43 | 44 | % run lanczos iterations. 45 | for j = 2 : m 46 | % fully reorthogonalize. 47 | h = Q(:, 1 : j-1)' * p; 48 | p = p - Q(:, 1 : j-1) * h; 49 | 50 | % compute the next lanczos vector. 51 | gamma = [gamma; norm(p)]; 52 | q = p ./ gamma(end); 53 | Q = [Q, q]; 54 | 55 | % compute the next transformed lanczos vector. 56 | v = At(q); 57 | V = [V, v]; 58 | 59 | % compute the next conjugate vector. 60 | p = K(q); 61 | omega = [omega; real(q' * p)]; 62 | p = p - omega(end) .* q - gamma(end) .* Q(:, j-1); 63 | 64 | % update the recurrence variables. 65 | f2 = f1; 66 | f1 = f; 67 | f = omega(end) * f1 - gamma(end)^2 * f2; 68 | end 69 | 70 | % construct the tridiagonal matrix. 71 | T = sparse(diag(omega) + diag(gamma, 1) + diag(gamma, -1)); 72 | 73 | % compute the marginal means. 74 | Qty = sparse(norm(y) .* eye(m, 1)); 75 | t = Q * (T \ Qty); 76 | mu = At(t) ./ xi; 77 | 78 | % compute the marginal variances. 79 | S = sum(real(V .* (T \ V').'), 2); 80 | sigma = 1./xi - S ./ xi.^2; 81 | end 82 | 83 | -------------------------------------------------------------------------------- /sources/util_lanczos.m: -------------------------------------------------------------------------------- 1 | 2 | % -*- texinfo -*- 3 | % @deftypefn {} {[@var{mu}, @var{sigma}, @var{f}] =} util_lanczos (@var{A}, @ 4 | % @var{At}, @var{tau}, @var{xi}, @var{y}, @var{y0}) 5 | % Compute the solution to a linear system and the diagonal of the 6 | % inverse coefficient matrix using Lanczos iteration. 7 | % 8 | % The outputs @var{mu} and @var{sigma} are the solution and 9 | % diagonal, and @var{f} holds an estimate of the determinant 10 | % of the kernel matrix. 11 | % @end deftypefn 12 | % 13 | function [mu, sigma, f] = util_lanczos (A, At, tau, xi, y, y0) 14 | % get the problem sizes. 15 | n = length(xi); 16 | m = length(y); 17 | 18 | % define the kernel matrix operator. 19 | K = @(z) z ./ tau + A(At(z) ./ xi); 20 | 21 | % initialize the tridiagonal matrix elements. 22 | omega = []; 23 | gamma = []; 24 | 25 | % initialize the lanczos vector. 26 | q = y ./ norm(y); 27 | Q = q; 28 | 29 | % initialize the transformed lanczos vector. 30 | v = At(q); 31 | V = v; 32 | 33 | % compute the first conjugate vector. 34 | p = K(q); 35 | omega = real(q' * p); 36 | p = p - omega .* q; 37 | 38 | % initialize the recurrence variables. 39 | f2 = 0; 40 | f1 = 1; 41 | f = omega; 42 | 43 | % run lanczos iterations. 44 | for j = 2 : m 45 | % fully reorthogonalize. 46 | h = Q(:, 1 : j-1)' * p; 47 | p = p - Q(:, 1 : j-1) * h; 48 | 49 | % compute the next lanczos vector. 50 | gamma = [gamma; norm(p)]; 51 | q = p ./ gamma(end); 52 | Q = [Q, q]; 53 | 54 | % compute the next transformed lanczos vector. 55 | v = At(q); 56 | V = [V, v]; 57 | 58 | % compute the next conjugate vector. 59 | p = K(q); 60 | omega = [omega; real(q' * p)]; 61 | p = p - omega(end) .* q - gamma(end) .* Q(:, j-1); 62 | 63 | % update the recurrence variables. 64 | f2 = f1; 65 | f1 = f; 66 | f = omega(end) * f1 - gamma(end)^2 * f2; 67 | end 68 | 69 | % construct the tridiagonal matrix. 70 | T = sparse(diag(omega) + diag(gamma, 1) + diag(gamma, -1)); 71 | 72 | % compute the marginal means. 73 | Qty = sparse(norm(y) .* eye(m, 1)); 74 | t = Q * (T \ Qty); 75 | mu = At(t) ./ xi; 76 | 77 | % compute the marginal variances. 78 | S = sum(real(V .* (T \ V').'), 2); 79 | sigma = 1./xi - S ./ xi.^2; 80 | end 81 | 82 | -------------------------------------------------------------------------------- /sources/vbcs.m: -------------------------------------------------------------------------------- 1 | 2 | % -*- texinfo -*- 3 | % @deftypefn {} {[@var{mu}, @dots{}] =} vbcs (@var{y}, @var{A}, @dots{}) 4 | % Recover a sparse signal @var{x} from incomplete data @var{y} and a 5 | % measurement matrix (@var{A} or @var{A},@var{At}) using Variational 6 | % Bayesian Compressed Sensing. 7 | % 8 | % See @ref{vrvm} for detailed usage information. 9 | % @end deftypefn 10 | % 11 | function [x, obj, parms] = ... 12 | vbcs (y, A, At, nu0, lambda0, alpha0, beta0, iters) 13 | % check for the minimum number of arguments. 14 | if (nargin < 2) 15 | error('at least two arguments required'); 16 | end 17 | 18 | % check the required measurement vector. 19 | if (isempty(y) || !isvector(y) || !iscolumn(y)) 20 | error('measurement must be a vector'); 21 | end 22 | 23 | % check the required measurement operator. 24 | if (!isempty(A) && (nargin < 3 || isempty(At))) 25 | % matrix specification. check the data type. 26 | if (!ismatrix(A)) 27 | error('invalid measurement: expected matrix'); 28 | end 29 | 30 | % store copies of the measurement matrix and its gramian. 31 | B = A; 32 | G = A' * A; 33 | 34 | % create function handles for forward, inverse, and projection. 35 | A = @(x) B * x; 36 | At = @(y) B' * y; 37 | AtA = @(x) G * x; 38 | 39 | elseif (!isempty(A) && nargin >= 3 && !isempty(At)) 40 | % function handle specification. check the data types. 41 | if (!is_function_handle(A) || !is_function_handle(At)) 42 | error('invalid measurement: expected function handles'); 43 | end 44 | 45 | % create a function handle for projection. 46 | AtA = @(x) At(A(x)); 47 | end 48 | 49 | % check for a prior nu parameter. 50 | if (nargin < 4 || isempty(nu0)) 51 | % none specified. use a default value. 52 | nu0 = 1e-3; 53 | end 54 | 55 | % check for a prior lambda parameter. 56 | if (nargin < 5 || isempty(lambda0)) 57 | % none specified. use a default value. 58 | lambda0 = 1e-3; 59 | end 60 | 61 | % check for a prior alpha parameter. 62 | if (nargin < 6 || isempty(alpha0)) 63 | % none specified. use a default value. 64 | alpha0 = 1e-3; 65 | end 66 | 67 | % check for a prior beta parameter. 68 | if (nargin < 7 || isempty(beta0)) 69 | % none specified. use a default value. 70 | beta0 = 1e-3; 71 | end 72 | 73 | % check for an iteration count argument. 74 | if (nargin < 8 || isempty(iters)) 75 | % none specified. use a default value. 76 | iters = 100; 77 | end 78 | 79 | % initialize the transformed data vector. 80 | h = At(y); 81 | 82 | % get the problem sizes. 83 | m = length(y); 84 | n = length(h); 85 | 86 | % initialize the weight means. 87 | x = zeros(n, 1); 88 | 89 | % initialize the precisions. 90 | xi = repmat(alpha0 / beta0, n, 1); 91 | 92 | % initialize the noise. 93 | tau = nu0 / lambda0; 94 | 95 | % initialize the projected weight means. 96 | u = AtA(x); 97 | 98 | % compute the projector diagonal, in the most general way possible. 99 | g = real(diag(AtA(eye(n)))); 100 | 101 | % compute the constant updated parameters. 102 | alpha = alpha0 + (1/2); 103 | nu = nu0 + m/2; 104 | 105 | % compute the constant portion of the objective. 106 | phi0 = vrvm_const(m, n, nu0, lambda0, alpha0, beta0); 107 | 108 | % initialize the objective vector. 109 | obj = repmat(phi0, iters, 1); 110 | 111 | % iterate. 112 | for it = 1 : iters 113 | % update the variances. 114 | v = 1 ./ (tau .* g + xi); 115 | 116 | % compute the parallel means and their projection. 117 | xp = tau .* v .* (h - u + g .* x); 118 | up = AtA(xp); 119 | 120 | % compute the terms of the step size. 121 | T1 = tau .* h' * (x - xp); 122 | T2 = -x' * (tau .* u + xi .* x); 123 | T3 = -xp' * (tau .* up + xi .* xp); 124 | T4 = x' * (tau .* up + xi .* xp); 125 | 126 | % compute the bounded step size for the mean update. 127 | gamma = real(T1 + T2 + T4) / real(T2 + T3 + 2*T4); 128 | gamma = min(max(gamma, 1e-3), 1); 129 | 130 | % update the means and their projection. 131 | x = (1 - gamma) .* x + gamma .* xp; 132 | u = (1 - gamma) .* u + gamma .* up; 133 | 134 | % compute the log-determinant. 135 | lndetS = sum(log(v)); 136 | 137 | % compute the trace term. 138 | trGS = g' * v; 139 | 140 | % update the precisions. 141 | mu2 = conj(x) .* x + v; 142 | beta = beta0 + 0.5 .* mu2; 143 | xi = alpha ./ beta; 144 | 145 | % update the noise. 146 | lambda = lambda0 + 0.5 * norm(y - A(x))^2 + 0.5 * trGS; 147 | tau = nu / lambda; 148 | 149 | % compute the objective function value. 150 | phi = nu * log(lambda) + alpha * sum(log(beta)) - 0.5 * lndetS; 151 | 152 | % store the objective. 153 | obj(it) += phi; 154 | end 155 | 156 | % check if the complete set of variational parameters was requested. 157 | if (nargout >= 3) 158 | % store the parameters over x. 159 | parms.mu = x; 160 | parms.sigma = v; 161 | 162 | % store the parameters over tau. 163 | parms.nu = nu; 164 | parms.lambda = lambda; 165 | 166 | % store the parameters over xi. 167 | parms.alpha = alpha; 168 | parms.beta = beta; 169 | end 170 | end 171 | 172 | -------------------------------------------------------------------------------- /sources/vrvm.m: -------------------------------------------------------------------------------- 1 | 2 | % -*- texinfo -*- 3 | % @deftypefn {} {@var{mu} =} vrvm (@var{y}, @var{A}) 4 | % @deftypefnx {} {@var{mu} =} vrvm (@var{y}, @var{A}, @var{At}) 5 | % @deftypefnx {} {@var{mu} =} vrvm (@var{y}, @var{A}, @var{At}, @var{nu0}) 6 | % @deftypefnx {} {@var{mu} =} vrvm (@var{y}, @var{A}, @var{At}, @var{nu0}, @ 7 | % @var{lambda0}) 8 | % @deftypefnx {} {@var{mu} =} vrvm (@var{y}, @var{A}, @var{At}, @var{nu0}, @ 9 | % @var{lambda0}, @var{alpha0}) 10 | % @deftypefnx {} {@var{mu} =} vrvm (@var{y}, @var{A}, @var{At}, @var{nu0}, @ 11 | % @var{lambda0}, @var{alpha0}, @var{beta0}) 12 | % @deftypefnx {} {@var{mu} =} vrvm (@var{y}, @var{A}, @var{At}, @var{nu0}, @ 13 | % @var{lambda0}, @var{alpha0}, @var{beta0}, @var{iters}) 14 | % @deftypefnx {} {[@var{mu}, @var{obj}] =} vrvm (@dots{}) 15 | % @deftypefnx {} {[@var{mu}, @var{obj}, @var{parms}] =} vrvm (@dots{}) 16 | % Recover a sparse signal @var{x} from incomplete data @var{y} and a 17 | % measurement matrix (@var{A} or @var{A},@var{At}) using a direct 18 | % implementation of the Variational Relevance Vector Machine. 19 | % 20 | % For matrix-type measurements, only @var{A} is required. For custom 21 | % measurements, the function handles @var{A} and @var{At} are both 22 | % required. 23 | % 24 | % When function handles are supplied, they must be capable of handling 25 | % both column vectors and matrices, where the operations are performed 26 | % independently to each column of the matrix. 27 | % 28 | % The optional hyperparameters @var{nu0} and @var{lambda0}, which both 29 | % default to 1e-3, control the prior distribution over the noise 30 | % precision. 31 | % 32 | % The optional hyperparameters @var{alpha0} and @var{beta0}, which also 33 | % both default to 1e-3, control the prior distribution over the signal 34 | % coefficient precisions. 35 | % 36 | % The optional argument @var{iters} controls the iteration count, and 37 | % defaults to 100 iterations. 38 | % 39 | % The function outputs @var{mu}, the mean estimate of the regression weights, 40 | % and (optionally) the objective function values @var{obj} and the set of 41 | % inferred parameters @var{parms}. 42 | % @end deftypefn 43 | % 44 | function [mu, obj, parms] = ... 45 | vrvm (y, A, At, nu0, lambda0, alpha0, beta0, iters) 46 | % check for the minimum number of arguments. 47 | if (nargin < 2) 48 | error('at least two arguments required'); 49 | end 50 | 51 | % check the required measurement vector. 52 | if (isempty(y) || !isvector(y) || !iscolumn(y)) 53 | error('measurement must be a vector'); 54 | end 55 | 56 | % check the required measurement operator. 57 | if (!isempty(A) && (nargin < 3 || isempty(At))) 58 | % matrix specification. check the data type. 59 | if (!ismatrix(A)) 60 | error('invalid measurement: expected matrix'); 61 | end 62 | 63 | % store copies of the measurement matrix and its gramian. 64 | B = A; 65 | G = A' * A; 66 | 67 | % create function handles for forward, inverse, and projection. 68 | A = @(x) B * x; 69 | At = @(y) B' * y; 70 | AtA = @(x) G * x; 71 | 72 | elseif (!isempty(A) && nargin >= 3 && !isempty(At)) 73 | % function handle specification. check the data types. 74 | if (!is_function_handle(A) || !is_function_handle(At)) 75 | error('invalid measurement: expected function handles'); 76 | end 77 | 78 | % create a function handle for projection. 79 | AtA = @(x) At(A(x)); 80 | end 81 | 82 | % check for a prior nu parameter. 83 | if (nargin < 4 || isempty(nu0)) 84 | % none specified. use a default value. 85 | nu0 = 1e-3; 86 | end 87 | 88 | % check for a prior lambda parameter. 89 | if (nargin < 5 || isempty(lambda0)) 90 | % none specified. use a default value. 91 | lambda0 = 1e-3; 92 | end 93 | 94 | % check for a prior alpha parameter. 95 | if (nargin < 6 || isempty(alpha0)) 96 | % none specified. use a default value. 97 | alpha0 = 1e-3; 98 | end 99 | 100 | % check for a prior beta parameter. 101 | if (nargin < 7 || isempty(beta0)) 102 | % none specified. use a default value. 103 | beta0 = 1e-3; 104 | end 105 | 106 | % check for an iteration count argument. 107 | if (nargin < 8 || isempty(iters)) 108 | % none specified. use a default value. 109 | iters = 100; 110 | end 111 | 112 | % initialize the transformed data vector. 113 | h = At(y); 114 | 115 | % get the problem sizes. 116 | m = length(y); 117 | n = length(h); 118 | 119 | % if the projector was not defined, define it. 120 | if (!exist('G', 'var')) 121 | G = AtA(eye(n)); 122 | end 123 | 124 | % initialize the weight means. 125 | mu = zeros(n, 1); 126 | 127 | % initialize the precisions. 128 | xi = repmat(alpha0 / beta0, n, 1); 129 | 130 | % initialize the noise. 131 | tau = nu0 / lambda0; 132 | 133 | % compute the constant updated parameters. 134 | alpha = alpha0 + (1/2); 135 | nu = nu0 + m/2; 136 | 137 | % compute the constant portion of the objective. 138 | phi0 = vrvm_const(m, n, nu0, lambda0, alpha0, beta0); 139 | 140 | % initialize the objective vector. 141 | obj = repmat(phi0, iters, 1); 142 | 143 | % iterate. 144 | for it = 1 : iters 145 | % update the variances. 146 | U = chol(tau .* G + diag(xi)); 147 | Sigma = chol2inv(U); 148 | 149 | % update the means. 150 | mu = tau .* Sigma * h; 151 | 152 | % compute the log-determinant. 153 | lndetS = -2 * sum(log(real(diag(U)))); 154 | 155 | % compute the trace term. 156 | trGS = real(sum(vec(G .* Sigma))); 157 | 158 | % update the precisions. 159 | mu2 = conj(mu) .* mu + real(diag(Sigma)); 160 | beta = beta0 + 0.5 .* mu2; 161 | xi = alpha ./ beta; 162 | 163 | % update the noise. 164 | lambda = lambda0 + 0.5 * norm(y - A(mu))^2 + 0.5 * trGS; 165 | tau = nu / lambda; 166 | 167 | % compute the objective function value. 168 | phi = nu * log(lambda) + alpha * sum(log(beta)) - 0.5 * lndetS; 169 | 170 | % store the objective. 171 | obj(it) += phi; 172 | end 173 | 174 | % check if the complete set of variational parameters was requested. 175 | if (nargout >= 3) 176 | % store the parameters over x. 177 | parms.mu = mu; 178 | parms.sigma = real(diag(Sigma)); 179 | 180 | % store the parameters over tau. 181 | parms.nu = nu; 182 | parms.lambda = lambda; 183 | 184 | % store the parameters over xi. 185 | parms.alpha = alpha; 186 | parms.beta = beta; 187 | end 188 | end 189 | 190 | -------------------------------------------------------------------------------- /sources/vrvm_const.m: -------------------------------------------------------------------------------- 1 | 2 | % -*- texinfo -*- 3 | % @deftypefn {} {@var{phi0} =} vrvm_const (@var{m}, @var{n}, @ 4 | % @var{nu0}, @var{lambda0}, @var{alpha0}, @var{beta0}) 5 | % Compute the constant offset term of the Variational Relevance 6 | % Vector Machine objective function, based only on prior values 7 | % and the problem size. 8 | % @end deftypefn 9 | % 10 | function phi0 = vrvm_const (m, n, nu0, lambda0, alpha0, beta0) 11 | % compute Cp, the constant offset of the Kullback-Liebler corrected 12 | % variational lower bound. 13 | Cp = nu0 * log(lambda0) + n * alpha0 * log(beta0) ... 14 | + (lgamma(nu0 + m/2) - lgamma(nu0)) ... 15 | + n * (lgamma(alpha0 + 1/2) - lgamma(alpha0)) ... 16 | - ((m + n) / 2) * log(2 * pi); 17 | 18 | % compute phi0, the constant offset of the objective function, 19 | % which includes the constant term of the normal entropy. 20 | phi0 = -(n/2) * log(2 * pi * e) - Cp; 21 | end 22 | 23 | -------------------------------------------------------------------------------- /sources/vrvm_krylov.m: -------------------------------------------------------------------------------- 1 | 2 | % -*- texinfo -*- 3 | % @deftypefn {} {@var{mu} =} vrvm_krylov (@var{y}, @var{A}) 4 | % @deftypefnx {} {@var{mu} =} vrvm_krylov (@var{y}, @var{A}, @var{At}) 5 | % @deftypefnx {} {@var{mu} =} vrvm_krylov (@var{y}, @var{A}, @var{At}, @ 6 | % @var{nu0}) 7 | % @deftypefnx {} {@var{mu} =} vrvm_krylov (@var{y}, @var{A}, @var{At}, @ 8 | % @var{nu0}, @var{lambda0}) 9 | % @deftypefnx {} {@var{mu} =} vrvm_krylov (@var{y}, @var{A}, @var{At}, @ 10 | % @var{nu0}, @var{lambda0}, @var{alpha0}) 11 | % @deftypefnx {} {@var{mu} =} vrvm_krylov (@var{y}, @var{A}, @var{At}, @ 12 | % @var{nu0}, @var{lambda0}, @var{alpha0}, @var{beta0}) 13 | % @deftypefnx {} {@var{mu} =} vrvm_krylov (@var{y}, @var{A}, @var{At}, @ 14 | % @var{nu0}, @var{lambda0}, @var{alpha0}, @var{beta0}, @var{iters}) 15 | % @deftypefnx {} {[@var{mu}, @var{obj}] =} vrvm_krylov (@dots{}) 16 | % @deftypefnx {} {[@var{mu}, @var{obj}, @var{parms}] =} vrvm_krylov (@dots{}) 17 | % Recover a sparse signal @var{x} from incomplete data @var{y} and a 18 | % measurement matrix (@var{A} or @var{A},@var{At}) using an 19 | % implementation of the VRVM that solves the marginalization 20 | % subproblem of each iteration using lanczos iteration. 21 | % 22 | % See @ref{vrvm} for detailed usage information. 23 | % @end deftypefn 24 | % 25 | function [mu, obj, parms] = ... 26 | vrvm_krylov (y, A, At, nu0, lambda0, alpha0, beta0, iters) 27 | % check for the minimum number of arguments. 28 | if (nargin < 2) 29 | error('at least two arguments required'); 30 | end 31 | 32 | % check the required measurement vector. 33 | if (isempty(y) || !isvector(y) || !iscolumn(y)) 34 | error('measurement must be a vector'); 35 | end 36 | 37 | % check the required measurement operator. 38 | if (!isempty(A) && (nargin < 3 || isempty(At))) 39 | % matrix specification. check the data type. 40 | if (!ismatrix(A)) 41 | error('invalid measurement: expected matrix'); 42 | end 43 | 44 | % store copies of the measurement matrix and its gramian. 45 | B = A; 46 | G = A' * A; 47 | 48 | % create function handles for forward, inverse, and projection. 49 | A = @(x) B * x; 50 | At = @(y) B' * y; 51 | AtA = @(x) G * x; 52 | 53 | elseif (!isempty(A) && nargin >= 3 && !isempty(At)) 54 | % function handle specification. check the data types. 55 | if (!is_function_handle(A) || !is_function_handle(At)) 56 | error('invalid measurement: expected function handles'); 57 | end 58 | 59 | % create a function handle for projection. 60 | AtA = @(x) At(A(x)); 61 | end 62 | 63 | % check for a prior nu parameter. 64 | if (nargin < 4 || isempty(nu0)) 65 | % none specified. use a default value. 66 | nu0 = 1e-3; 67 | end 68 | 69 | % check for a prior lambda parameter. 70 | if (nargin < 5 || isempty(lambda0)) 71 | % none specified. use a default value. 72 | lambda0 = 1e-3; 73 | end 74 | 75 | % check for a prior alpha parameter. 76 | if (nargin < 6 || isempty(alpha0)) 77 | % none specified. use a default value. 78 | alpha0 = 1e-3; 79 | end 80 | 81 | % check for a prior beta parameter. 82 | if (nargin < 7 || isempty(beta0)) 83 | % none specified. use a default value. 84 | beta0 = 1e-3; 85 | end 86 | 87 | % check for an iteration count argument. 88 | if (nargin < 8 || isempty(iters)) 89 | % none specified. use a default value. 90 | iters = 100; 91 | end 92 | 93 | % initialize the transformed data vector. 94 | h = At(y); 95 | 96 | % get the problem sizes. 97 | m = length(y); 98 | n = length(h); 99 | 100 | % initialize the precisions. 101 | xi = repmat(alpha0 / beta0, n, 1); 102 | 103 | % initialize the noise. 104 | tau = nu0 / lambda0; 105 | 106 | % compute the constant updated parameters. 107 | alpha = alpha0 + (1/2); 108 | nu = nu0 + m/2; 109 | 110 | % compute the constant portion of the objective. 111 | phi0 = vrvm_const(m, n, nu0, lambda0, alpha0, beta0); 112 | 113 | % initialize the objective vector. 114 | obj = repmat(phi0, iters, 1); 115 | 116 | % initialize the marginals. 117 | mu = zeros(n, 1); 118 | sigma = zeros(n, 1); 119 | yhat = zeros(m, 1); 120 | 121 | % FIXME 122 | Tau = zeros(iters, 1); 123 | Xi = zeros(n, iters); 124 | Mu = zeros(n, iters); 125 | Sigma = zeros(n, iters); 126 | 127 | % iterate. 128 | for it = 1 : iters 129 | % solve the marginalization subproblem. 130 | [mu, sigma, f] = util_krylov(A, At, tau, xi, y, yhat); 131 | 132 | % FIXME 133 | Tau(it) = tau; 134 | Xi(:,it) = xi; 135 | Mu(:,it) = mu; 136 | Sigma(:,it) = sigma; 137 | 138 | % compute the log-determinant. 139 | lndetS = log(f) + m * log(tau) + sum(log(xi)); 140 | 141 | % compute the trace term. 142 | trGS = (n - xi' * sigma) ./ tau; 143 | 144 | % update the precisions. 145 | beta = beta0 + 0.5 .* (abs(mu).^2 + sigma); 146 | xi = alpha ./ beta; 147 | 148 | % update the noise. 149 | yhat = A(mu); 150 | lambda = lambda0 + 0.5 * norm(y - yhat)^2 + 0.5 * trGS; 151 | tau = nu / lambda; 152 | 153 | % compute the objective function value. 154 | phi = nu * log(lambda) + alpha * sum(log(beta)) - 0.5 * lndetS; 155 | 156 | % store the objective. 157 | obj(it) += phi; 158 | end 159 | 160 | % check if the complete set of variational parameters was requested. 161 | if (nargout >= 3) 162 | % store the parameters over x. 163 | parms.mu = mu; 164 | parms.sigma = sigma; 165 | 166 | % store the parameters over tau. 167 | parms.nu = nu; 168 | parms.lambda = lambda; 169 | 170 | % store the parameters over xi. 171 | parms.alpha = alpha; 172 | parms.beta = beta; 173 | 174 | % FIXME 175 | parms.Tau = Tau; 176 | parms.Xi = Xi; 177 | parms.Mu = Mu; 178 | parms.Sigma = Sigma; 179 | end 180 | end 181 | 182 | -------------------------------------------------------------------------------- /sources/vrvm_lowrank.m: -------------------------------------------------------------------------------- 1 | 2 | % -*- texinfo -*- 3 | % @deftypefn {} {[@var{mu}, @dots{}] =} vrvm_lowrank (@var{y}, @var{A}, @ 4 | % @dots{}) 5 | % Recover a sparse signal @var{x} from incomplete data @var{y} and a 6 | % measurement matrix (@var{A} or @var{A},@var{At}) using an implementation 7 | % of the Variational Relevance Vector Machine that leverages the low-rank 8 | % properties of the measurement operator to reduce computational demands. 9 | % 10 | % See @ref{vrvm} for detailed usage information. 11 | % @end deftypefn 12 | % 13 | function [mu, obj, parms] = ... 14 | vrvm_lowrank (y, A, At, nu0, lambda0, alpha0, beta0, iters) 15 | % check for the minimum number of arguments. 16 | if (nargin < 2) 17 | error('at least two arguments required'); 18 | end 19 | 20 | % check the required measurement vector. 21 | if (isempty(y) || !isvector(y) || !iscolumn(y)) 22 | error('measurement must be a vector'); 23 | end 24 | 25 | % check the required measurement operator. 26 | if (!isempty(A) && (nargin < 3 || isempty(At))) 27 | % matrix specification. check the data type. 28 | if (!ismatrix(A)) 29 | error('invalid measurement: expected matrix'); 30 | end 31 | 32 | % store copies of the measurement matrix and its gramian. 33 | B = A; 34 | G = A' * A; 35 | 36 | % create function handles for forward, inverse, and projection. 37 | A = @(x) B * x; 38 | At = @(y) B' * y; 39 | AtA = @(x) G * x; 40 | 41 | elseif (!isempty(A) && nargin >= 3 && !isempty(At)) 42 | % function handle specification. check the data types. 43 | if (!is_function_handle(A) || !is_function_handle(At)) 44 | error('invalid measurement: expected function handles'); 45 | end 46 | 47 | % create a function handle for projection. 48 | AtA = @(x) At(A(x)); 49 | end 50 | 51 | % check for a prior nu parameter. 52 | if (nargin < 4 || isempty(nu0)) 53 | % none specified. use a default value. 54 | nu0 = 1e-3; 55 | end 56 | 57 | % check for a prior lambda parameter. 58 | if (nargin < 5 || isempty(lambda0)) 59 | % none specified. use a default value. 60 | lambda0 = 1e-3; 61 | end 62 | 63 | % check for a prior alpha parameter. 64 | if (nargin < 6 || isempty(alpha0)) 65 | % none specified. use a default value. 66 | alpha0 = 1e-3; 67 | end 68 | 69 | % check for a prior beta parameter. 70 | if (nargin < 7 || isempty(beta0)) 71 | % none specified. use a default value. 72 | beta0 = 1e-3; 73 | end 74 | 75 | % check for an iteration count argument. 76 | if (nargin < 8 || isempty(iters)) 77 | % none specified. use a default value. 78 | iters = 100; 79 | end 80 | 81 | % initialize the transformed data vector. 82 | h = At(y); 83 | 84 | % get the problem sizes. 85 | m = length(y); 86 | n = length(h); 87 | 88 | % initialize the weight means. 89 | mu = zeros(n, 1); 90 | 91 | % initialize the precisions. 92 | xi = repmat(alpha0 / beta0, n, 1); 93 | 94 | % initialize the noise. 95 | tau = nu0 / lambda0; 96 | 97 | % compute the constant updated parameters. 98 | alpha = alpha0 + (1/2); 99 | nu = nu0 + m/2; 100 | 101 | % compute the constant portion of the objective. 102 | phi0 = vrvm_const(m, n, nu0, lambda0, alpha0, beta0); 103 | 104 | % initialize the lower bound vector. 105 | obj = repmat(phi0, iters, 1); 106 | 107 | % iterate. 108 | for it = 1 : iters 109 | % build the kernel matrix, K = A E[Xi]^-1 A* 110 | K = A(A(diag(1 ./ xi))'); 111 | 112 | % factorize the shifted kernel matrix. 113 | U = chol(eye(m) ./ tau + K); 114 | Q = chol2inv(U); 115 | 116 | % update the means. 117 | t = Q * K * y; 118 | mu = tau .* At(y - t) ./ xi; 119 | 120 | % compute the log-determinant. 121 | lndetS = m * log(tau) + sum(log(xi)) + 2 * sum(log(real(diag(U)))); 122 | 123 | % compute the trace term. 124 | trKQK = real(trace(K * Q * K)); 125 | trK = real(trace(K)); 126 | trGS = trK - trKQK; 127 | 128 | % compute the marginal variances. 129 | S = sum(real(At(eye(m)) .* conj(At(Q))), 2); 130 | sigma = 1./xi - S ./ xi.^2; 131 | 132 | % update the precisions. 133 | mu2 = conj(mu) .* mu + sigma; 134 | beta = beta0 + 0.5 .* mu2; 135 | xi = alpha ./ beta; 136 | 137 | % update the noise. 138 | lambda = lambda0 + 0.5 * norm(y - A(mu))^2 + 0.5 * trGS; 139 | tau = nu / lambda; 140 | 141 | % compute the objective function value. 142 | phi = nu * log(lambda) + alpha * sum(log(beta)) - 0.5 * lndetS; 143 | 144 | % store the objective. 145 | obj(it) += phi; 146 | end 147 | 148 | % check if the complete set of variational parameters was requested. 149 | if (nargout >= 3) 150 | % store the parameters over x. 151 | parms.mu = mu; 152 | parms.sigma = sigma; 153 | 154 | % store the parameters over tau. 155 | parms.nu = nu; 156 | parms.lambda = lambda; 157 | 158 | % store the parameters over xi. 159 | parms.alpha = alpha; 160 | parms.beta = beta; 161 | end 162 | end 163 | 164 | --------------------------------------------------------------------------------