├── LICENSE ├── README.md ├── VGC-BivariateLogNormal ├── BPcdf_basis.m ├── BPpdf_basis.m ├── BPprime_basis.m ├── Cal_mcELBO.m ├── Grad_MuC.m ├── Grad_W.m ├── MvLogNRand.m ├── Phi_f.m ├── SimplxProj.m ├── demo_BivariateLN.m ├── derivatives.m ├── dphi_f.m ├── dpsi_f.m ├── hist1D2D.m ├── iPsi_f.m ├── ini_WinSet.m ├── logdet.m ├── logmodel.m ├── logqpost.m ├── logqpost_derivatives.m ├── modelLN.m ├── mzscore_test.m ├── post2vec.m ├── randBPw.m ├── randchol.m ├── sampleEVG.m ├── sampleGC.m ├── sphi_f.m ├── spsi_f.m ├── svgln.m ├── vec2post.m └── vgcbp_MuCW.m ├── VGC-FlexibleMargins ├── BPcdf_basis.m ├── BPpdf_basis.m ├── BPprime_basis.m ├── Cal_mcELBO.m ├── Grad_MuC.m ├── Grad_W.m ├── Phi_f.m ├── SimplxProj.m ├── demo_Beta.m ├── demo_Gamma.m ├── demo_SkewNormal.m ├── demo_StudentT.m ├── derivatives.m ├── dphi_f.m ├── dpsi_f.m ├── iPsi_f.m ├── ini_WinSet.m ├── logdet.m ├── logmodel.m ├── logqpost.m ├── logqpost_derivatives.m ├── mzscore_test.m ├── randBPw.m ├── sampleGC.m ├── sphi_f.m ├── spsi_f.m ├── vgcbp_MuCW.m └── vgcbp_w.m ├── VGC-HorseshoeShrinkage ├── BPcdf_basis.m ├── BPpdf_basis.m ├── BPprime_basis.m ├── Cal_mcELBO.m ├── Grad_MuCW.m ├── Horseshoe_Gibbs.m ├── Horseshoe_Mfvb.m ├── Horseshoe_VCD_LN.m ├── Phi_f.m ├── SimplxProj.m ├── demo_Horseshoe.m ├── derivatives.m ├── dphi_f.m ├── dpsi_f.m ├── hist1D2D.m ├── iPsi_f.m ├── ini_WinSet.m ├── logdet.m ├── logmodel.m ├── logqpost.m ├── logqpost_derivatives.m ├── mzscore_test.m ├── post2vec.m ├── randBPw.m ├── randchol.m ├── sampleGC.m ├── sphi_f.m ├── spsi_f.m ├── vcval_Horseshoe.m ├── vcval_Horseshoe_diag.m ├── vec2post.m └── vgcbp_MuCW.m ├── VGC-PoissonLogLinear ├── BPcdf_basis.m ├── BPpdf_basis.m ├── BPprime_basis.m ├── Cal_mcELBO.m ├── Grad_MuCW.m ├── Phi_f.m ├── PoissonLogLinearModel.txt ├── PoissonLogMCMC.mat ├── PoissonLogVGC.mat ├── SimplxProj.m ├── TreeData50.mat ├── demo_JAGS_PoissonLogLinear.R ├── demo_VGC_PoissonLogLinear.m ├── derivatives.m ├── dphi_f.m ├── dpsi_f.m ├── dpsi_f4.m ├── hist1D2D.m ├── iPsi_f.m ├── iPsi_f4.m ├── ini_WinSet.m ├── logdet.m ├── logmodel.m ├── logqpost.m ├── logqpost_derivatives.m ├── mzscore_test.m ├── post2vec.m ├── randBPw.m ├── randchol.m ├── sampleGC.m ├── sphi_f.m ├── spsi_f.m ├── spsi_f4.m ├── vec2post.m └── vgcbp_MuCW.m └── figure ├── VGC-JAGS.png ├── horseshoe.png ├── lognormal.png └── margins.png /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 shaobo Han 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Variational Gaussian Copula Inference 2 | 3 | We use Gaussian copulas (combined with fixed/free-form margins) as **automated inference engines** for variational approximation in generic hierarchical Bayesian models (the only two model-specific terms are the log likelihood & prior term and its derivatives). We evaluate the **peculiarities** reproduced in the univariate margins and the **posterior dependence** captured broadly across latent variables. 4 | 5 | ## Matlab code for the paper 6 | 7 | Shaobo Han, Xuejun Liao, David B. Dunson, and Lawrence Carin, "Variational Gaussian Copula Inference", *The 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016)*, Cadiz, Spain, May, 2016 8 | 9 | ## Examples 10 | 11 | #### Demo 1: Marginal Adaptation (Skew normal, Student's t, Beta, Gamma) 12 | 13 | ```Matlab 14 | >> demo_SkewNormal 15 | >> demo_StudentT 16 | >> demo_Gamma 17 | >> demo_Beta 18 | ``` 19 | The accuracy of marginal approximation for real, positive real, and truncated [0,1] variables is shown as follows, 20 | 21 | 22 | 23 | 24 | --- 25 | #### Demo 2: Bivariate Log-Normal 26 | 27 | ```Matlab 28 | >> demo_BivariateLN 29 | ``` 30 | We approximate bivariate log-normal distributions using a bivariate Gaussian copula with (1) fixed-form log-normal distributed margins (2) free-form Bernstein polynomial based margins, 31 | 32 | 33 | 34 | --- 35 | #### Demo 3: Horseshoe Shrinkage 36 | 37 | Baseline comparisons include: 38 | * Gibbs sampler 39 | * Mean-field VB 40 | * VGC-LN-full: Gaussian copula with log-normal margins 41 | * VGC-LN-diag: Independence copula with Log-normal margins 42 | * VGC-BP-full: Gaussian copula with Bernstein polynomial margins 43 | 44 | ```Matlab 45 | >> demo_Horseshoe 46 | ``` 47 | 48 | 49 | 50 | --- 51 | #### Demo 4: Poisson Log-Linear Regression 52 | 53 | MCMC sampler is implemented in JAGS: 54 | 55 | ```r 56 | >> demo_JAGS_PoissonLogLinear 57 | ``` 58 | Variational Gaussian copula (VGC) inference: 59 | 60 | ```Matlab 61 | >> demo_VGC_PoissonLogLinear 62 | ``` 63 | The univaraite margins and pairwise posteriors (JAGS v.s. VGC-BP) are shown below: 64 | 65 | 66 | 67 | 68 | --- 69 | 70 | ## Citations 71 | 72 | If you find this code helpful, please cite the work using the following information: 73 | 74 | @inproceedings{VGC_2016, 75 | title={Variational Gaussian Copula Inference}, 76 | author={Shaobo Han and Xuejun Liao and David B. Dunson and Lawrence Carin}, 77 | booktitle={The 19th International Conference on Artificial Intelligence and Statistics (AISTATS)}, 78 | year={2016}, 79 | } 80 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/BPcdf_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial CDF basis function 2 | % Regularized incomplete beta function 3 | 4 | % u: evaluating point, could be a p by 1 vector 5 | % k: degree of BPs 6 | % opt: BP/exBP, standard or extended BP 7 | 8 | % V: a p by d matrix storing basis funcitons 9 | 10 | % Shaobo Han 11 | % 08/07/2015 12 | 13 | function V = BPcdf_basis(u, k, opt) 14 | if nargin<3, 15 | opt = 'BP'; % Standard BP by default 16 | end 17 | p = length(u); % # of variables 18 | switch opt 19 | case 'BP' % Standard BP 20 | d = k; % # of basis functions 21 | a_seq = 1:k; b_seq = k-a_seq+1; 22 | case 'exBP' % extended BP 23 | d = k*(k+1)/2; % # of basis functions 24 | a_seq = zeros(1, d); b_seq = zeros(1, d); 25 | tmp_idx = cumsum(1:k); 26 | for j = 1:k 27 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 28 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 29 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 30 | end 31 | end 32 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 33 | U = repmat(u, 1, d); % p by d matrix 34 | V = betacdf(U , A_seq, B_seq); % p by d matrix 35 | end 36 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/BPpdf_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial PDF basis function 2 | % Beta densities 3 | 4 | % u: evaluating point, could be a p by 1 vector 5 | % k: degree of BPs 6 | % opt: BP/exBP, standard or extended BP 7 | 8 | % V: a p by d matrix storing basis funcitons 9 | 10 | % Shaobo Han 11 | % 08/08/2015 12 | 13 | function V = BPpdf_basis(u, k, opt) 14 | if nargin<3, 15 | opt = 'BP'; % Standard BP by default 16 | end 17 | p = length(u); % # of variables 18 | switch opt 19 | case 'BP' % Standard BP 20 | d = k; % # of basis functions 21 | a_seq = 1:k; b_seq = k-a_seq+1; 22 | case 'exBP' % extended BP 23 | d = k*(k+1)/2; % # of basis functions 24 | a_seq = zeros(1, d); b_seq = zeros(1, d); 25 | tmp_idx = cumsum(1:k); 26 | for j = 1:k 27 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 28 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 29 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 30 | end 31 | end 32 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 33 | U = repmat(u, 1, d); % p by d matrix 34 | V = betapdf(U , A_seq, B_seq); % p by d matrix 35 | end 36 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/BPprime_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial prime basis function 2 | % First-order of Derivative Beta densities 3 | 4 | % u: evaluating point, could be a p by 1 vector 5 | % k: degree of BPs 6 | % opt: BP/exBP, standard or extended BP 7 | 8 | % V: a P by D matrix storing basis funcitons 9 | 10 | % Shaobo Han 11 | % 08/10/2015 12 | 13 | function V = BPprime_basis(u, k, opt) 14 | if nargin<3, 15 | opt = 'BP'; % Standard BP by default 16 | end 17 | p = length(u); % # of variables 18 | switch opt 19 | case 'BP' % Standard BP 20 | d = k; % # of basis functions 21 | a_seq = 1:k; b_seq = k-a_seq+1; 22 | case 'exBP' % extended BP 23 | d = k*(k+1)/2; % # of basis functions 24 | a_seq = zeros(1, d); b_seq = zeros(1, d); 25 | tmp_idx = cumsum(1:k); 26 | for j = 1:k 27 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 28 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 29 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 30 | end 31 | end 32 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 33 | U = repmat(u, 1, d); % p by d matrix 34 | V = (A_seq+B_seq-1).*(betapdf(U , A_seq-1, B_seq)-betapdf(U , A_seq, B_seq-1)); % p by d matrix 35 | % Remove NaN's 36 | % Defining betapdf(u, a, 0) = betapdf(u,0,b) = 0; 37 | Idx = (A_seq == 1)|(B_seq == 1); 38 | V(Idx) = 0; 39 | end 40 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/Cal_mcELBO.m: -------------------------------------------------------------------------------- 1 | function sELBO = Cal_mcELBO(trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par) 2 | tmpMu=par.Mu; tmpC = par.C; tmpW=par.W; 3 | N_mc = opt.N_mc; PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; k = opt.k; 4 | P = length(tmpMu); sELBOM = zeros(1, N_mc); 5 | v_t = sqrt(diag(tmpC*tmpC')); 6 | 7 | for i = 1: N_mc 8 | % Generate a P by 1 multivariate Normal vector 9 | w_t = randn(P,1); Z_t = tmpMu + tmpC*w_t; 10 | if opt.adaptivePhi == 1 11 | Z = diag(1./v_t)*(Z_t-tmpMu); 12 | u = Phi_f(Z, PhiPar, PhiType); 13 | sphi = diag(1./v_t)*sphi_f(Z, PhiPar, PhiType); 14 | else 15 | u = Phi_f(Z_t, PhiPar, PhiType); 16 | sphi = sphi_f(Z_t, PhiPar, PhiType); 17 | end 18 | BPcdf_basiseval = BPcdf_basis(u, k, BPtype); 19 | tmpB = sum(BPcdf_basiseval.*tmpW, 2); 20 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 21 | logp = logmodel(tmpx, trueModel, fix); 22 | BPpdf_basiseval = BPpdf_basis(u, k, BPtype); 23 | tmpb = sum(BPpdf_basiseval.*tmpW, 2); 24 | spsi = spsi_f(tmpx, PsiPar, PsiType); 25 | hdz1 = diag(sphi./spsi)*tmpb; 26 | sELBOM(i) = logp + sum(log(hdz1))+ P/2*log(2*pi)+P/2 + sum(log(diag(tmpC))); 27 | end 28 | sELBO = median(sELBOM); 29 | end 30 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/Grad_MuC.m: -------------------------------------------------------------------------------- 1 | function [Delta_Mu, Delta_C, WinSet] = Grad_MuC(VGmethod, trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par, WinSet) 2 | NumberZ = opt.NumberZ; P = fix.P; 3 | PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; k = opt.k; 4 | Mu_old = par.Mu; C_old = par.C; W_old = par.W; 5 | MuMC = zeros(P, NumberZ); 6 | CMC = zeros(P,P, NumberZ); 7 | v_t = sqrt(diag(C_old*C_old')); 8 | for jj = 1:NumberZ 9 | %% Draw Sample w_t and Z_t 10 | Flag_outlier = 1; 11 | while Flag_outlier == 1 12 | % Generate a P by 1 multivariate Normal vector 13 | w_t = randn(P,1); Z_t = Mu_old + C_old*w_t; 14 | if opt.adaptivePhi == 1 15 | Z = diag(1./v_t)*(Z_t-Mu_old); 16 | u = Phi_f(Z, PhiPar, PhiType); 17 | else 18 | u = Phi_f(Z_t, PhiPar, PhiType); 19 | end 20 | 21 | BPcdf_basiseval = BPcdf_basis(u, k, BPtype); 22 | tmpB = sum(BPcdf_basiseval.*W_old, 2); 23 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 24 | logp = logmodel(tmpx, trueModel, fix); 25 | % Online outlier detection 26 | score = mzscore_test(logp, WinSet); 27 | if abs(score) < opt.OutlierTol 28 | Flag_outlier = 0; WinSet_old = WinSet; 29 | WinSet = [WinSet_old, logp]; 30 | WinSet(1) = []; 31 | end 32 | end 33 | if opt.adaptivePhi == 1 34 | sphi = diag(1./v_t)*sphi_f(Z, PhiPar, PhiType); 35 | dphi = -diag(1./v_t)*(Z.*sphi); 36 | else 37 | sphi = sphi_f(Z_t, PhiPar, PhiType); 38 | dphi = dphi_f(Z_t, PhiPar, PhiType); 39 | end 40 | BPpdf_basiseval = BPpdf_basis(u, k, BPtype); 41 | tmpb = sum(BPpdf_basiseval.*W_old, 2); 42 | BPprime_basiseval = BPprime_basis(u, k, BPtype); 43 | tmpbprime = sum(BPprime_basiseval.*W_old, 2); 44 | rho1prime = sphi.*tmpbprime; 45 | 46 | spsi = spsi_f(tmpx, PsiPar, PsiType); 47 | dpsi = dpsi_f(tmpx,PsiPar, PsiType); 48 | g = derivatives(tmpx, trueModel, fix); 49 | 50 | hdz1 = diag(sphi./spsi)*tmpb; 51 | rho3prime = hdz1.*dpsi; 52 | hdz2 = (rho1prime.*sphi.*spsi... 53 | +tmpb.*dphi.*spsi... 54 | -tmpb.*sphi.*rho3prime)./(spsi.^2); 55 | lnh1dz = hdz2./hdz1; 56 | ls_z = g.*hdz1+lnh1dz; 57 | switch VGmethod 58 | case 'Analytic' 59 | dmu = ls_z; 60 | dC = tril(ls_z*w_t')+ diag(1./diag(C_old)); 61 | case 'Numeric' 62 | tmppar.Mu = Mu_old; tmppar.Sigma = C_old*C_old'; 63 | g2 = logqpost_derivatives(Z_t, inferModel, tmppar); 64 | dmu = ls_z-g2; 65 | dC = tril(dmu*w_t'); 66 | end 67 | MuMC(:, jj) = dmu; 68 | CMC(:, :, jj) = dC; 69 | end 70 | Delta_Mu = mean(MuMC, 2); 71 | Delta_C= mean(CMC, 3); 72 | end 73 | 74 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/Grad_W.m: -------------------------------------------------------------------------------- 1 | function [Delta_w, WinSet] = Grad_W(trueModel, PhiType, PsiType, BPtype, fix, opt, par, WinSet) 2 | NumberZ = opt.NumberZ; P = fix.P; D = opt.D; 3 | PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; k = opt.k; 4 | Mu_old = par.Mu; C_old = par.C; W_old = par.W; 5 | WMC = zeros(P, D, NumberZ); 6 | vt = sqrt(diag(C_old*C_old')); 7 | for jj = 1:NumberZ 8 | %% Draw Sample w_t and Z_t 9 | Flag_outlier = 1; 10 | while Flag_outlier == 1 11 | % Generate a P by 1 multivariate Normal vector 12 | w_t = randn(P,1); Z_t = Mu_old + C_old*w_t; 13 | if opt.adaptivePhi == 1 14 | Z = diag(1./vt)*(Z_t-Mu_old); 15 | u = Phi_f(Z, PhiPar, PhiType); 16 | else 17 | u = Phi_f(Z_t, PhiPar, PhiType); 18 | end 19 | BPcdf_basiseval = BPcdf_basis(u, k, BPtype); 20 | tmpB = sum(BPcdf_basiseval.*W_old, 2); 21 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 22 | logp = logmodel(tmpx, trueModel, fix); 23 | % Online outlier detection 24 | score = mzscore_test(logp, WinSet); 25 | if abs(score) < opt.OutlierTol 26 | Flag_outlier = 0; WinSet_old = WinSet; 27 | WinSet = [WinSet_old, logp]; 28 | WinSet(1) = []; 29 | end 30 | end 31 | BPpdf_basiseval = BPpdf_basis(u, k, BPtype); 32 | tmpb = sum(BPpdf_basiseval.*W_old, 2); 33 | spsi = spsi_f(tmpx, PsiPar, PsiType); 34 | dpsi = dpsi_f(tmpx,PsiPar, PsiType); 35 | g = derivatives(tmpx, trueModel, fix); 36 | hdw = diag(1./spsi)*BPcdf_basiseval; 37 | lnh1dw = diag(1./tmpb)*BPpdf_basiseval - diag(dpsi./spsi.^2)*BPcdf_basiseval; 38 | WMC(:,:,jj) = diag(g)*hdw+lnh1dw; 39 | end 40 | Delta_w = median(WMC, 3); 41 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/MvLogNRand.m: -------------------------------------------------------------------------------- 1 | function y = MvLogNRand( Mu , Sigma , Simulations , CorrMat ) 2 | %MVLOGNRAND MultiVariant Lognormal random numbers with correlation 3 | % 4 | % Mu: The Lognormal parameter Mu (can be column or row vector) 5 | % 6 | % Sigma: The Lognormal parameter Sigma (can be column or row vector) 7 | % 8 | % Simulations: The Number of simulations to run (scalar) 9 | % 10 | % CorrMat: OPTIONAL A square matrix with the number of rows and columns 11 | % equal to the number of elements in Mu/Sigma. Each element on the 12 | % diagonal is equal to one, with the off diagonal cells equal to the 13 | % correlation of the marginal Lognormal distributions. If not specified, 14 | % then assume zero correlation. 15 | % 16 | % To check the simulation run corrcoef(Y) and that should be the same as 17 | % your CorrMat. 18 | % 19 | % REQUIRES THE STATISTICS TOOLBOX 20 | % 21 | % Example: 22 | % Mu = [ 11 12 13 ]; 23 | % Sigma = [ .1 .3 .5 ]; 24 | % Simulations = 1e6; 25 | % CorrMat = [1 .2 .4 ; .2 1 .5 ; .4 .5 1]; 26 | % y = MvLogNRand( Mu , Sigma , Simulations , CorrMat ); 27 | % 28 | % corrcoef(y) 29 | % ans = 30 | % 1 0.19927 0.40156 31 | % 0.19927 1 0.50008 32 | % 0.40156 0.50008 1 33 | % 34 | % CorrMat = 35 | % 1 0.2 0.4 36 | % 0.2 1 0.5 37 | % 0.4 0.5 1 38 | % 39 | % For more information see: Aggregration of Correlated Risk Portfolios: 40 | % Models and Algorithms; Shaun S. Wang, Phd. Casualty Actuarial Society 41 | % Proceedings Volume LXXXV www.casact.org 42 | % 43 | % Author: Stephen Lienhard 44 | 45 | % Error checking 46 | if nargin < 3 47 | error('Must have at least 3 input arguements') 48 | end 49 | 50 | if numel(Simulations) ~= 1 || Simulations < 0 51 | error('The number of simulations must be greater then zero and a scalar') 52 | end 53 | 54 | if nargin == 3 55 | CorrMat = eye(numel(Mu)); 56 | elseif size(CorrMat,1) ~= size(CorrMat,2) 57 | error('The correlation matrix must have the same number of rows as columns') 58 | end 59 | 60 | if numel(Mu) ~= numel(Sigma) 61 | error('Mu and Sigma must have the same number of elements') 62 | end 63 | 64 | % Force column vectors 65 | Mu = Mu(:); 66 | Sigma = Sigma(:); 67 | 68 | % Calculate the covariance structure 69 | sigma_down = repmat( Sigma' , numel(Sigma), 1 ); 70 | sigma_acrs = repmat( Sigma , 1 , numel(Sigma) ); 71 | covv = log( CorrMat .* sqrt(exp(sigma_down.^2)-1) .* ... 72 | sqrt(exp(sigma_acrs.^2)-1) + 1 ); 73 | 74 | % The Simulation 75 | y = exp( mvnrnd( Mu , covv , Simulations )); -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/Phi_f.m: -------------------------------------------------------------------------------- 1 | % Phi function: mapping from (-infty, infty) to unit closed interval [0,1] 2 | % Shaobo Han 3 | % 09/17/2015 4 | function u = Phi_f(z, par, opt) 5 | switch opt 6 | case 'Normal' 7 | m0 = par(1); v0 = par(2); 8 | u = normcdf(z, m0, sqrt(v0)); 9 | end 10 | end 11 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/SimplxProj.m: -------------------------------------------------------------------------------- 1 | function X = SimplxProj(Y) 2 | % Projects each row vector in the N by D marix Y to the probability 3 | % simplex in D-1 dimensions 4 | [N, D] = size(Y); 5 | X = sort(Y, 2, 'descend'); 6 | Xtmp = (cumsum(X,2)-1)*diag(sparse(1./(1:D))); 7 | X = max(bsxfun(@minus, Y, Xtmp(sub2ind([N,D], (1:N)', sum(X>Xtmp,2)))),0); 8 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/demo_BivariateLN.m: -------------------------------------------------------------------------------- 1 | % Bivariate Log-Normal example for the paper 2 | % "Variational Gaussian Copula Inference", 3 | % Shaobo Han, Xuejun Liao, David. B. Dunson, and Lawrence Carin, 4 | % The 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016) 5 | % ----------------------------- 6 | % Code written by 7 | % Shaobo Han, Duke University 8 | % shaobohan@gmail.com 9 | 10 | % 09/23/2015 11 | 12 | % Method 13 | % 1 Ground Truth 14 | % 2 VGC-LN with Analytic VG updates 15 | % 3 VGC-LN with Numeric VG updates 16 | % 4 VGC-BP with Numeric VG updates 17 | % 5 VGC-BP with Analytic VG updates 18 | 19 | % We use MVLOGNRAND written by Stephen Lienhard 20 | % as Random Samples Generator for Bivaraite Lognormal Distributions 21 | % http://www.mathworks.com/matlabcentral/fileexchange/6426-multivariate-lognormal-simulation-with-correlation/content/MvLogNRand.m 22 | 23 | clear all; close all; clc; 24 | % addpath(genpath(pwd)) % Add all sub-directories to the Matlab path 25 | fix. P = 2; % Number of latent variables 26 | 27 | % True SN Posterior 28 | trueModel = 'LogNormal2'; 29 | fix.LNsigmaY1 = 0.5; fix.LNsigmaY2 = 0.5; 30 | fix.LNmuY1 = 0.1; fix.LNmuY2 = 0.1; 31 | fix.LNrho = -0.4; % fix.LNrho = 0.4; 32 | 33 | fix.trueMu = [fix.LNmuY1; fix.LNmuY2]; 34 | fix.trueSigma=[fix.LNsigmaY1^2,fix.LNrho*fix.LNsigmaY1*fix.LNsigmaY2;... 35 | fix.LNrho*fix.LNsigmaY1*fix.LNsigmaY2,fix.LNsigmaY2^2]; 36 | fix.trueUpsilon = corrcov( fix.trueSigma); 37 | opt.adaptivePhi = 0; opt.normalize = 0; 38 | fix.c = 2; % unnormalizing constant 39 | %% 40 | opt.k = 5; % Degree/Maximum Degree of Bernstein Polynomials 41 | opt.MaxIter = 100; % Number of SGD stages 42 | opt.NumberZ = 1; % Average Gradients 43 | opt.InnerIter = 250; % Number of iteration 44 | 45 | stageMax = 100; iterMax = 25; 46 | fix.rho = 0.005; fix.dec = 0.95; 47 | 48 | opt.N_mc = 1; % Number of median average in sELBO stepsize search 49 | opt.nsample = 5e5; 50 | figplot.nbins = 50; 51 | BPtype = 'BP'; % Bernstein Polynomials 52 | % BPtype = 'exBP'; % Extended Bernstein Polynomials 53 | 54 | % PsiType = 'Normal'; opt.PsiPar(1) = 0; opt.PsiPar(2) = 1; % variance 55 | PsiType = 'Exp'; opt.PsiPar = 1; 56 | PhiType = 'Normal'; opt.PhiPar(1) = 0; opt.PhiPar(2) =1; % variance 57 | VGmethod = 'Numeric'; 58 | opt.Wthreshold = 1e12; 59 | 60 | % Learning rate 61 | opt.LearnRate.Mu = 0.005; opt.LearnRate.C = 0.005; 62 | opt.LearnRate.W = 0.5*1e-3; opt.LearnRate.dec = 0.95; % decreasing base learning rate 63 | 64 | switch BPtype 65 | case 'BP' 66 | opt.D = opt.k; 67 | case 'exBP' 68 | opt.D = opt.k*(opt.k+1)/2; % # of basis functions 69 | end 70 | 71 | % Diagonal constraint on Upsilon 72 | opt.diagUpsilon = 0; 73 | 74 | %% Initialization 75 | ini.Mu = opt.PhiPar(1).*ones(fix.P,1); 76 | ini.C = sqrt(opt.PhiPar(2)).*eye(fix.P); 77 | % ini.w = randBPw(fix.P, opt.D, 1, 1); 78 | ini.w = 1./opt.D.*ones(fix.P, opt.D); 79 | % ini.w = randBPw(fix.P, opt.D, 1, 1); 80 | 81 | % Median Outlier Removal 82 | opt.OutlierTol = 10; % Threshold for online outlier detection 83 | opt.WinSize = 20; % Size of the window 84 | ini.WinSet = ini_WinSet(trueModel, PhiType, PsiType, BPtype, fix, opt, ini); 85 | 86 | %% 1 VGC LN Analytic 87 | fix.scale = 1; % The scale of covariance 88 | inferModel2 = 'MVN'; % Multivariate Normal 89 | VG = svgln(trueModel, inferModel2, stageMax, iterMax, 'Analytic', fix); 90 | figplot.Y_vg = sampleEVG(opt, VG); 91 | figplot.Y_vg(sum(~isfinite(figplot.Y_vg),2)~=0,:) = []; 92 | VGsamples = hist1D2D(figplot.Y_vg, figplot.nbins); 93 | 94 | %% VG Numeric 95 | VG2 = svgln(trueModel, inferModel2, stageMax, iterMax, 'Numeric', fix); 96 | figplot.Y_vg2 = sampleEVG(opt, VG2); 97 | figplot.Y_vg2(sum(~isfinite(figplot.Y_vg2),2)~=0,:) = []; 98 | VGsamples2 = hist1D2D(figplot.Y_vg2, figplot.nbins); 99 | 100 | %% VGC MuCW-Numeric 101 | opt.updateW = 1; opt.updateMuC = 1; 102 | [ELBO3, par3] = vgcbp_MuCW(trueModel, inferModel2, PhiType, PsiType, BPtype, 'Numeric', fix, ini, opt); 103 | % ini.Mu = par.Mu; ini.C = par.C; ini.WinSet = par.WinSet; 104 | 105 | figplot.Y_svc3 = sampleGC(PhiType, PsiType, BPtype, opt, par3); 106 | figplot.Y_svc3(sum(~isfinite(figplot.Y_svc3),2)~=0,:) = []; 107 | VGCBPall3 = hist1D2D(figplot.Y_svc3, figplot.nbins); 108 | 109 | %% VGC MuCW-Analytic 110 | opt.updateW = 1; opt.updateMuC = 1; 111 | [ELBO5, par5] = vgcbp_MuCW(trueModel, inferModel2, PhiType, PsiType, BPtype, 'Analytic', fix, ini, opt); 112 | figplot.Y_svc5 = sampleGC(PhiType, PsiType, BPtype, opt, par5); 113 | figplot.Y_svc5(sum(~isfinite(figplot.Y_svc5),2)~=0,:) = []; 114 | VGCBPall5 = hist1D2D(figplot.Y_svc5, figplot.nbins); 115 | 116 | %% Plot 117 | figplot.nb = 1000; 118 | contourvector = [0.05, 0.1, 0.25,0.5, 0.75, 0.9]; 119 | figplot.Seqx = linspace(0.001, 5, figplot.nb)'; 120 | figplot.Seqy = linspace(0.001, 5, figplot.nb+1)'; 121 | [figplot.X1,figplot.X2] = meshgrid(figplot.Seqx,figplot.Seqy); 122 | figplot.log_pM = modelLN(figplot.X1, figplot.X2, fix); 123 | plevs1=contourvector.*max(exp(figplot.log_pM (:))); 124 | 125 | figure (1) 126 | figplot.ls = 2; lfs =26; 127 | set(gca, 'fontsize', 16) 128 | contour(figplot.Seqx,figplot.Seqy, exp(figplot.log_pM), plevs1, 'linewidth', 2*figplot.ls); 129 | xlabel('x_1'); ylabel('x_2'); 130 | axis([0,3.5,0,3.5]) 131 | AX=legend('Ground Truth'); 132 | LEG = findobj(AX,'type','text'); 133 | set(LEG,'fontsize',lfs) 134 | 135 | figure (2) 136 | F25 =VGsamples.count./sum(sum(VGsamples.count)); 137 | plevs3=contourvector.*max(F25(:)); 138 | set(gca, 'fontsize', 16) 139 | contour(VGsamples.Seqx,VGsamples.Seqy, F25, plevs3, 'linewidth', 2*figplot.ls); 140 | xlabel('x_1'); ylabel('x_2'); 141 | axis([0,3.5,0,3.5]) 142 | AX= legend('VGC-LN1'); 143 | LEG = findobj(AX,'type','text'); 144 | set(LEG,'fontsize',lfs) 145 | 146 | figure (3) 147 | F26 =VGsamples2.count./sum(sum(VGsamples2.count)); 148 | plevs4=contourvector.*max(F26(:)); 149 | set(gca, 'fontsize', 16) 150 | contour(VGsamples2.Seqx,VGsamples2.Seqy, F26, plevs4, 'linewidth', 2*figplot.ls); 151 | xlabel('x_1'); ylabel('x_2'); 152 | axis([0,3.5,0,3.5]) 153 | AX= legend('VGC-LN2'); 154 | LEG = findobj(AX,'type','text'); 155 | set(LEG,'fontsize',lfs) 156 | 157 | figure (4) 158 | set(gca, 'fontsize', 16) 159 | F38 =VGCBPall5.count./sum(sum(VGCBPall5.count)); 160 | plevs36=contourvector.*max(F38(:)); 161 | contour(VGCBPall5.Seqx,VGCBPall5.Seqy, F38, plevs36, 'linewidth', 2*figplot.ls); 162 | xlabel('x_1'); ylabel('x_2'); 163 | axis([0,3.5,0,3.5]) 164 | AX=legend('VGC-BP1'); 165 | LEG = findobj(AX,'type','text'); 166 | set(LEG,'fontsize',lfs) 167 | 168 | figure (5) 169 | set(gca, 'fontsize', 16) 170 | F28 =VGCBPall3.count./sum(sum(VGCBPall3.count)); 171 | plevs6=contourvector.*max(F28(:)); 172 | contour(VGCBPall3.Seqx,VGCBPall3.Seqy, F28, plevs6, 'linewidth', 2*figplot.ls); 173 | xlabel('x_1'); ylabel('x_2'); 174 | axis([0,3.5,0,3.5]) 175 | AX=legend('VGC-BP2'); 176 | LEG = findobj(AX,'type','text'); 177 | set(LEG,'fontsize',lfs) 178 | 179 | figure (6) 180 | figplot.fs = 16; figplot.ms =3; 181 | set(gca, 'fontsize', figplot.fs) 182 | plot(median(VG.RMSECTC,2), '--r*', 'linewidth', figplot.ls, 'Markersize', 3+figplot.ms); 183 | hold on 184 | plot(median(VG2.RMSECTC,2), '-bo', 'linewidth', figplot.ls, 'Markersize', 3+figplot.ms); 185 | hold off 186 | grid on 187 | axis([0,100, 0,1]) 188 | AX=legend('VGC-LN1', 'VGC-LN2'); 189 | LEG = findobj(AX,'type','text'); 190 | set(LEG,'fontsize',lfs) 191 | xlabel('Iterations') 192 | ylabel('\rho') 193 | 194 | figure (7) 195 | figplot.fs = 16; figplot.ms = 3; 196 | set(gca, 'fontsize', figplot.fs) 197 | plot([par5.RMSE(1,1), median(par5.RMSE,1)], '--r*', 'linewidth', figplot.ls, 'Markersize', 3+figplot.ms); 198 | hold on 199 | plot([par3.RMSE(1,1),median(par3.RMSE,1)], '-bo', 'linewidth', figplot.ls, 'Markersize', 3+figplot.ms); 200 | hold off 201 | grid on 202 | axis([0,100, 0,1]) 203 | AX=legend('VGC-BP1', 'VGC-BP2'); 204 | LEG = findobj(AX,'type','text'); 205 | set(LEG,'fontsize',lfs) 206 | xlabel('Iterations') 207 | ylabel('\rho') -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/derivatives.m: -------------------------------------------------------------------------------- 1 | % Derivatives of Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function g = derivatives(x, model, fix) 7 | switch model 8 | case 'Horseshoe' 9 | y = fix.y; 10 | g = zeros(2,1); 11 | g(1) = -2/x(1)+y^2/2/(x(1)^2)+x(2)/(x(1)^2); 12 | g(2) = -1/x(1)-1; 13 | case 'SkewNormal' 14 | m = fix.snMu; 15 | Lambda = fix.snSigma; 16 | alpha = fix.snAlpha; 17 | tmp = alpha'*(x-m); 18 | g = -(Lambda\(x-m))+normpdf(tmp).*alpha./normcdf(tmp); 19 | case 'LogNormal2' 20 | sigmaY1 = fix.LNsigmaY1; sigmaY2 = fix.LNsigmaY2; 21 | muY1 = fix.LNmuY1; muY2 = fix.LNmuY2; LNrho = fix.LNrho; 22 | alpha1 = (log(x(1))-muY1)./sigmaY1; 23 | alpha2= (log(x(2))-muY2)./sigmaY2; 24 | g = zeros(2,1); 25 | g(1)=-1./x(1)-(alpha1-LNrho*alpha2)/(1-LNrho.^2)./x(1)./sigmaY1; 26 | g(2)=-1./x(2)-(alpha2-LNrho*alpha1)/(1-LNrho.^2)./x(2)./sigmaY2; 27 | case 'Gamma' 28 | alpha = fix.alpha; 29 | beta = fix.beta; 30 | g = (alpha-1)./x-beta; 31 | case 'Student' 32 | nu = fix.nu; 33 | g = -(nu+1)*x/(nu+x^2); 34 | case 'Beta'; 35 | alpha = fix.alpha; beta = fix.beta; 36 | g = (alpha-1)/x-(beta-1)/(1-x); 37 | end 38 | end 39 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/dphi_f.m: -------------------------------------------------------------------------------- 1 | % dphi function: derivative of sphi 2 | % Shaobo Han 3 | % 09/24/2015 4 | function output = dphi_f(z, par, opt) 5 | switch opt 6 | case 'Normal' 7 | m0 = par(1); v0 = par(2); 8 | output = -(z./v0).*normpdf(z, m0, sqrt(v0)); 9 | end 10 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/dpsi_f.m: -------------------------------------------------------------------------------- 1 | % dpsi function: derivative of small psi 2 | % Shaobo Han 3 | % 08/10/2015 4 | 5 | function output = dpsi_f(x, par, opt) 6 | if nargin<3, 7 | opt = 'Exp'; % exponential distribution by default 8 | end 9 | if nargin<2, 10 | par = 0.1; % lambda = 0.1 by default 11 | end 12 | switch opt 13 | case 'Exp' % exp(lambda) 14 | output = -par.*exppdf(x,1/par); 15 | case 'Normal' % 16 | m0 = par(1); % mean 17 | v0 = par(2); % variance 18 | output = (-x./v0).*normpdf(x,m0,sqrt(v0)); 19 | case 'Beta' 20 | a0 = par(1); b0 = par(2); 21 | output =(a0+b0-1)*(betapdf(x, a0-1, b0)- betapdf(x, a0, b0-1)); 22 | end 23 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/hist1D2D.m: -------------------------------------------------------------------------------- 1 | function Output=hist1D2D(Matrix, nbins) 2 | % 1D and 2D histogram 3 | % Matrix N by 2 matrix 4 | [f_x1,x_x1] = hist(Matrix(:,1), nbins); 5 | [f_x2,x_x2] = hist(Matrix(:,2), nbins); 6 | [count,Dy] = hist3(Matrix,[nbins+2,nbins]); 7 | Seqx= Dy{1,1}; Seqy = Dy{1,2}; 8 | Output.f_x1 = f_x1; Output.x_x1 = x_x1; 9 | Output.f_x2 = f_x2; Output.x_x2 = x_x2; 10 | Output.count =count'; Output.Seqx =Seqx; Output.Seqy =Seqy; 11 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/iPsi_f.m: -------------------------------------------------------------------------------- 1 | % Invserse Psi function: 2 | % Shaobo Han 3 | % 08/10/2015 4 | function output = iPsi_f(p, par, opt) 5 | if nargin<3, 6 | opt = 'Exp'; % exponential distribution by default 7 | end 8 | if nargin<2, 9 | par = 0.1; % lambda = 0.1 by default 10 | end 11 | p(p>1-1e-16) = 1-1e-16; % numerical stability 12 | switch opt 13 | case 'Exp' % exp(lambda) 14 | % output = -log(1-p)./par; 15 | output = expinv(p,1/par); 16 | case 'Normal' % 17 | m0 = par(1); v0 = par(2); 18 | output = norminv(p,m0,sqrt(v0)); 19 | case 'Beta' 20 | a0 = par(1); b0 = par(2); 21 | output = betainv(p, a0, b0); 22 | end 23 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/ini_WinSet.m: -------------------------------------------------------------------------------- 1 | function WinSet = ini_WinSet(trueModel, PhiType, PsiType, BPtype, fix, opt, ini) 2 | k = opt.k; PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; 3 | P = fix.P; Mu_old = ini.Mu; C_old = ini.C; W_old = ini.w; 4 | WinSet = zeros(1, opt.WinSize); 5 | vt = sqrt(diag(C_old*C_old')); 6 | for i = 1:opt.WinSize 7 | w_t0 = randn(P,1); Z_t0 = Mu_old + C_old*w_t0; 8 | if opt.adaptivePhi == 1 9 | Z0 = diag(1./vt)*(Z_t0-Mu_old); 10 | u0 = Phi_f(Z0, PhiPar, PhiType); 11 | else 12 | u0 = Phi_f(Z_t0, PhiPar, PhiType); 13 | end 14 | BPcdf_basiseval0 = BPcdf_basis(u0, k, BPtype); 15 | tmpB0 = sum(BPcdf_basiseval0.*W_old, 2); 16 | tmpx0 = iPsi_f(tmpB0, PsiPar, PsiType); 17 | WinSet(i) = logmodel(tmpx0, trueModel, fix); 18 | end 19 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/logdet.m: -------------------------------------------------------------------------------- 1 | function y = logdet(A) 2 | %LOGDET Logarithm of determinant for positive-definite matrix 3 | % logdet(A) returns log(det(A)) where A is positive-definite. 4 | % This is faster and more stable than using log(det(A)). 5 | % Note that logdet does not check if A is positive-definite. 6 | % If A is not positive-definite, the result will not be the same as log(det(A)). 7 | % Written by Tom Minka 8 | % (c) Microsoft Corporation. All rights reserved. 9 | U = chol(A); 10 | y = 2*sum(log(diag(U))); -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/logmodel.m: -------------------------------------------------------------------------------- 1 | % Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function p = logmodel(x, model, fix) 7 | switch model 8 | case 'Horseshoe' 9 | y = fix.y; 10 | c0 = -0.5*log(2*pi)-2*log(gamma(0.5)); 11 | p = c0-2*log(x(1))-y.^2./2./x(1)-x(2)./x(1)-x(2); 12 | case 'LogNormal2' 13 | sigmaY1 = fix.LNsigmaY1; sigmaY2 = fix.LNsigmaY2; 14 | muY1 = fix.LNmuY1; muY2 = fix.LNmuY2; LNrho = fix.LNrho; 15 | c0 = -log(2*pi)-log(sigmaY1*sigmaY2*sqrt(1-LNrho^2)); 16 | alpha1 = (log(x(1))-muY1)/sigmaY1; 17 | alpha2= (log(x(2))-muY2)/sigmaY2; 18 | q = 1/(1-LNrho^2)*(alpha1^2-2*LNrho*alpha1*alpha2+alpha2^2); 19 | p = c0-log(x(1))-log(x(2))-q/2; 20 | case 'LogNormal' 21 | sigmaY = fix.LNsigmaY; 22 | muY = fix.LNmuY; 23 | c0 = -0.5*log(2*pi)-log(sigmaY); 24 | alpha = (log(x)-muY)./sigmaY; 25 | p = c0-log(x)- alpha.^2/2; 26 | case 'SkewNormal' 27 | m = fix.snMu; 28 | Lambda = fix.snSigma; 29 | alpha = fix.snAlpha; 30 | c = fix.c; 31 | p = log(2)+log(mvnpdf(x,m,Lambda))+log(normcdf(alpha'*(x-m)))+log(c); 32 | case 'Gamma' 33 | alpha = fix.alpha; 34 | beta = fix.beta; 35 | c = fix.c; 36 | p = log(gampdf(x, alpha, 1/beta))+log(c); 37 | case 'Student' 38 | nu = fix.nu; c = fix.c; 39 | p = log(tpdf(x,nu))+log(c); 40 | case 'Exp' 41 | lambda = fix.lambda; c = fix.c; 42 | p = log(exppdf(x,1/lambda))+log(c); 43 | case 'Beta'; 44 | alpha = fix.alpha; beta = fix.beta; c = fix.c; 45 | p = log(betapdf(x, alpha, beta))+log(c); 46 | end 47 | end 48 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/logqpost.m: -------------------------------------------------------------------------------- 1 | % Log q posteiror 2 | % Model dependent term 3 | % Shaobo Han 4 | % 09/11/2015 5 | 6 | function p = logqpost(x, model, par) 7 | switch model 8 | case 'Horseshoe' 9 | y = par.y; 10 | c0 = -0.5*log(2*pi)-2*log(gamma(0.5)); 11 | p = c0-2*log(x(1))-y.^2./2./x(1)-x(2)./x(1)-x(2); 12 | case 'SkewNormal' 13 | m = par.snMu; 14 | Lambda = par.snSigma; 15 | alpha = par.snAlpha; 16 | p = log(2)+log(mvnpdf(x,m,Lambda))+log(normcdf(alpha'*(x-m))) ; 17 | case 'MVN' 18 | mu = par.Mu; C= par.C; Sigma=C*C'; 19 | d = length(mu); 20 | p = -d*log(2*pi)/2-0.5*logdet(Sigma)-0.5*(x-mu)'*(Sigma\(x-mu)); 21 | case 'MVNdiag' 22 | mu = par.Mu; C= par.C; Sigma=C*C'; 23 | p = sum(log(normpdf(x, mu, sqrt(diag(Sigma))))); 24 | end 25 | end 26 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/logqpost_derivatives.m: -------------------------------------------------------------------------------- 1 | % Derivatives of Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function g = logqpost_derivatives(x, model, par) 7 | switch model 8 | case 'Horseshoe' 9 | y = par.y; 10 | g = zeros(2,1); 11 | g(1) = -2/x(1)+y^2/2/(x(1)^2)+x(2)/(x(1)^2); 12 | g(2) = -1/x(1)-1; 13 | case 'SkewNormal' 14 | m = par.snMu; 15 | Lambda = par.snSigma; 16 | alpha = par.snAlpha; 17 | g = -inv(Lambda)*(x-m)+mvnpdf(alpha'*(x-m)).*alpha./mvncdf(alpha'*(x-m)); 18 | case 'MVN' 19 | mu = par.Mu; Sigma= par.Sigma; 20 | g = Sigma\(mu-x); 21 | case 'MVNdiag' 22 | mu = par.Mu; Sigma= par.Sigma; 23 | g = diag(1./diag(Sigma))*(mu-x); 24 | end 25 | end 26 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/modelLN.m: -------------------------------------------------------------------------------- 1 | function p = modelLN(pi_1V, pi_2V, fix) 2 | [m,n] = size(pi_1V); 3 | sigmaY1 = fix.LNsigmaY1; sigmaY2 = fix.LNsigmaY2; 4 | muY1 = fix.LNmuY1; muY2 = fix.LNmuY2; LNrho = fix.LNrho; 5 | x1 = reshape(pi_1V, m*n,1); 6 | x2 = reshape(pi_2V, m*n,1); 7 | c0 = -log(2*pi)-log(sigmaY1*sigmaY2*sqrt(1-LNrho^2)); 8 | alpha1 = (log(x1)-muY1)./sigmaY1; 9 | alpha2 = (log(x2)-muY2)./sigmaY2; 10 | q = 1/(1-LNrho^2).*(alpha1.^2-2.*LNrho.*alpha1.*alpha2+alpha2.^2); 11 | logpV = c0-log(x1)-log(x2)-q/2; 12 | p = reshape(logpV, m,n); 13 | end 14 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/mzscore_test.m: -------------------------------------------------------------------------------- 1 | % Calculate the modified Z-score of a query point from a window set 2 | % Shaobo Han 3 | % 08/13/2015 4 | function score = mzscore_test(x, windowset) 5 | tmpX = [x, windowset]; 6 | tmpm = median(tmpX); 7 | mad = median(abs(tmpX-tmpm)); 8 | score = 0.6745*(x-tmpm)/mad; 9 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/post2vec.m: -------------------------------------------------------------------------------- 1 | function vec=post2vec(post) 2 | % convert posterior stucture to a 1=-d col. vector 3 | vec=[post.m;post.c(:)]; 4 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/randBPw.m: -------------------------------------------------------------------------------- 1 | % Generate a P*D BP weight matrix 2 | % (every column is a probability vector which sums up to one) 3 | % P: number of variables 4 | % D: number of BP basis functions 5 | % a: shape parameter; b: scale parameter 6 | 7 | % Shaobo Han 8 | % 08/08/2015 9 | function w = randBPw(P, D, a, b) 10 | if nargin<4 11 | a = 1; b = 1; % by default 12 | end 13 | if D==1 14 | w = ones(P,1); 15 | else 16 | w0 = gamrnd(a,b,P,D); 17 | w = diag(1./sum(w0,2))*w0; 18 | end 19 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/randchol.m: -------------------------------------------------------------------------------- 1 | % Generate a P*P lower triangular Cholesky factor matrix 2 | % Shaobo Han 3 | % 08/07/2015 4 | function C = randchol(P) 5 | a = rand(P, P); 6 | aTa = a'*a; 7 | [V, D] = eig(aTa); 8 | aTa_PD = V*(D+1e-5)*V'; 9 | C=chol(aTa_PD, 'lower'); 10 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/sampleEVG.m: -------------------------------------------------------------------------------- 1 | function Y_svc = sampleEVG(opt, VG) 2 | nsample = opt.nsample; 3 | Mu = VG.mu; C = VG.C; 4 | % wtz2 = mvnrnd(Mu', C*C', nsample); 5 | % Y_svc = exp(wtz2); 6 | Sigma = sqrt(diag(C*C')); Simulations = nsample; 7 | CorrMat = corrcov(C*C'); 8 | Y_svc = MvLogNRand( Mu , Sigma , Simulations , CorrMat ); 9 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/sampleGC.m: -------------------------------------------------------------------------------- 1 | function Y_svc = sampleGC(PhiType, PsiType, BPtype, opt, par) 2 | PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; 3 | nsample = opt.nsample; k = opt.k; 4 | Mu = par.Mu; C = par.C; W = par.W; 5 | P = length(Mu); 6 | % Sample from Gaussian Copula 7 | optmu_s2 = Mu; optSig_s2 = C*C'; 8 | wtz2 = mvnrnd(optmu_s2, optSig_s2, nsample); 9 | Y_svc = zeros(nsample,P); 10 | if opt.adaptivePhi == 0 11 | for ns = 1:nsample 12 | tmpz = wtz2(ns,:)'; 13 | tmpu = Phi_f(tmpz, PhiPar, PhiType); 14 | BPcdf_basiseval = BPcdf_basis(tmpu, k, BPtype); 15 | tmpB = sum(BPcdf_basiseval.*W, 2); 16 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 17 | Y_svc(ns,:) = tmpx; 18 | end 19 | else 20 | v_t = sqrt(diag(C*C')); 21 | for ns = 1:nsample 22 | tmpz0 = wtz2(ns,:)'; 23 | tmpz = diag(1./v_t)*(tmpz0-Mu); 24 | tmpu = Phi_f(tmpz, PhiPar, PhiType); 25 | BPcdf_basiseval = BPcdf_basis(tmpu, k, BPtype); 26 | tmpB = sum(BPcdf_basiseval.*W, 2); 27 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 28 | Y_svc(ns,:) = tmpx; 29 | end 30 | end 31 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/sphi_f.m: -------------------------------------------------------------------------------- 1 | % phi function: small phi, derivative of Phi 2 | % Shaobo Han 3 | % 08/07/2015 4 | function output = sphi_f(z, par, opt) 5 | switch opt 6 | case 'Normal' 7 | m0 = par(1); v0 = par(2); 8 | output = normpdf(z, m0, sqrt(v0)); 9 | end 10 | end 11 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/spsi_f.m: -------------------------------------------------------------------------------- 1 | % psi function: small psi, derivative of Psi 2 | % Shaobo Han 3 | % 08/10/2015 4 | 5 | function output = spsi_f(x, par, opt) 6 | if nargin<3, 7 | opt = 'Exp'; % exponential distribution by default 8 | end 9 | if nargin<2, 10 | par = 0.1; % lambda = 0.1 by default 11 | end 12 | switch opt 13 | case 'Exp' % exp(lambda) 14 | output = exppdf(x,1/par); 15 | case 'Normal' % 16 | m0 = par(1); v0 = par(2); 17 | output = normpdf(x, m0, sqrt(v0)); 18 | case 'Beta' 19 | a0 = par(1); b0 = par(2); 20 | output = betapdf(x, a0, b0); 21 | end 22 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/svgln.m: -------------------------------------------------------------------------------- 1 | function output = svgln(truemodel, infermodel, stageMax, iterMax, method, fix) 2 | % method = 'Analytic' | 'Numeric'; 3 | 4 | P = fix.P; rho = fix.rho; 5 | dec = fix.dec; scale = fix.scale; 6 | % Initialize 7 | tmpMu = zeros(P, 1); 8 | tmpC = sqrt(scale).*0.1.*eye(P); 9 | 10 | ELBO = zeros(stageMax, iterMax); 11 | RMSE1 = zeros(stageMax, iterMax); 12 | RMSE2 = zeros(stageMax, iterMax); 13 | 14 | for stage = 1: stageMax 15 | for iter = 1:iterMax 16 | if (iter ==1) && (stage==1) 17 | Mu = tmpMu; C = tmpC; 18 | end 19 | tmpw = randn(P,1); 20 | tmpz = Mu + C*tmpw; 21 | switch method 22 | case 'Analytic' 23 | tmpx = exp(tmpz); 24 | g = derivatives(tmpx, truemodel, fix).*exp(tmpz)+ones(2,1); 25 | dmu = g; 26 | dC = tril(g*tmpw') + diag(1./diag(C)); 27 | case 'Numeric' 28 | tmpx = exp(tmpz); 29 | g1 = derivatives(tmpx, truemodel, fix).*exp(tmpz)+ones(2,1); 30 | par.Mu = Mu; par.Sigma = C*C'; 31 | g2 = logqpost_derivatives(tmpz, infermodel, par); 32 | dmu = g1-g2; 33 | dC = tril(dmu*tmpw'); 34 | end 35 | Mu = Mu + rho.*dmu; 36 | C = C + rho.*dC; 37 | par.Mu = Mu; par.Sigma = C*C'; par.C = C; 38 | tmpw = randn(P,1); 39 | tmpz = Mu + C*tmpw; 40 | tmpx = exp(tmpz); 41 | sELBO = logmodel(tmpx, truemodel, fix)+sum(tmpz); 42 | logq = logqpost(tmpz, infermodel, par); 43 | ELBO(stage, iter) = sELBO-logq; 44 | RMSE1(stage, iter) = norm(Mu- fix.trueMu, 'fro')./norm(fix.trueMu, 'fro'); 45 | RMSE2(stage, iter) = norm(diag(corrcov(C*C'),-1)- diag(fix.trueUpsilon,-1), 'fro')./norm(diag(fix.trueUpsilon,-1), 'fro'); 46 | end 47 | rho = rho*dec; 48 | end 49 | fELBO = median(ELBO(end, :)); 50 | output.RMSEmu = RMSE1; 51 | output.RMSECTC = RMSE2; 52 | output.fsELBO = fELBO; 53 | output.sELBO = ELBO; 54 | output.mu = Mu; 55 | output.C = C; 56 | end -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/vec2post.m: -------------------------------------------------------------------------------- 1 | function post=vec2post(vec,postDim) 2 | % convert vector of posterior params to posterior structure 3 | post.m=vec(1:postDim); 4 | vec=vec(postDim+1:end); 5 | post.c=reshape(vec(:),postDim,postDim); 6 | end 7 | -------------------------------------------------------------------------------- /VGC-BivariateLogNormal/vgcbp_MuCW.m: -------------------------------------------------------------------------------- 1 | function [ELBO, par] = vgcbp_MuCW(trueModel, inferModel, PhiType, PsiType, BPtype, VGmethod, fix, ini, opt) 2 | % VGmethod = 'Analytic' | 'Numeric'; 3 | 4 | % Parameters Unpack 5 | InnerIter = opt.InnerIter; MaxIter = opt. MaxIter; 6 | r1 = opt.LearnRate.Mu; r2 = opt.LearnRate.C; 7 | r3 = opt.LearnRate.W; dec = opt.LearnRate.dec; 8 | 9 | if opt.diagUpsilon ==1 10 | display(['[VGC(S): diag], BPtype = ', BPtype,' PsiType = ', PsiType,' PhiType = ', PhiType]) 11 | else 12 | display(['[VGC(S): full], BPtype = ', BPtype,' PsiType = ', PsiType,' PhiType = ', PhiType]) 13 | end 14 | ELBO = zeros(InnerIter, MaxIter); 15 | RMSE = zeros(InnerIter, MaxIter); 16 | for iter = 1: MaxIter 17 | tic; 18 | for inner = 1:InnerIter 19 | % First Iteration 20 | if (iter == 1)&&(inner==1) 21 | par.Mu_old = ini.Mu; par.C_old = ini.C; par.W_old = ini.w; 22 | WinSet = ini.WinSet; 23 | par.Mu = par.Mu_old; par.C = par.C_old; par.W = par.W_old; 24 | else 25 | par.Mu_old = par.Mu; par.C_old = par.C; par.W_old = par.W; 26 | end 27 | 28 | if opt.updateMuC==1 29 | %% Draw Samples and Calculate (Mu, C) 30 | [Delta_Mu, Delta_C, WinSet] = Grad_MuC(VGmethod, trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par,WinSet); 31 | if opt.normalize ==1 32 | if norm(Delta_Mu,'fro')<=1e-5 33 | Delta_Mu = zeros(P,1); 34 | else 35 | Delta_Mu = Delta_Mu./norm(Delta_Mu,'fro'); 36 | end 37 | if norm(Delta_C,'fro')<=1e-5 38 | Delta_C = zeros(P,P); 39 | else 40 | Delta_C = Delta_C./norm(Delta_C,'fro'); 41 | end 42 | end 43 | %% Update Mu 44 | par.Mu = par.Mu_old + r1.*Delta_Mu; 45 | if opt.adaptivePhi == 1 46 | par.Mu = zeros(size( par.Mu)); 47 | end 48 | %% Update Mu 49 | par.C = par.C_old + r2.*Delta_C; 50 | else 51 | par.Mu = par.Mu_old; par.C = par.C_old; 52 | end 53 | if opt.updateW==1 54 | %% Draw Samples and Calculate Delta_W 55 | [Delta_w, WinSet] = Grad_W(trueModel, PhiType, PsiType, BPtype, fix, opt, par,WinSet); 56 | if opt.normalize ==1 57 | % Normalize 58 | if norm(Delta_w,'fro')<=1e-5 59 | Delta_w = zeros(size(Delta_w)); 60 | else 61 | Delta_w = diag(1./sqrt(sum(abs(Delta_w).^2,2)))*Delta_w; 62 | end 63 | end 64 | %% Update W 65 | tmpNorm = sqrt(sum(abs(Delta_w).^2,2)); 66 | UnstableIndex = (tmpNorm>opt.Wthreshold); 67 | tmpsum = sum(UnstableIndex); 68 | if tmpsum ~=0 69 | Delta_w(UnstableIndex,:) = zeros(tmpsum, opt.D); 70 | display('Warning: Unstable Delta W Omitted!') 71 | tmpNorm(UnstableIndex) 72 | end 73 | W0 = par.W_old + r3.*Delta_w; 74 | % Project Gradient Descent 75 | par.W = SimplxProj(W0); 76 | if sum(abs(sum(par.W,2)-1)>1e-5*ones(size(sum(par.W,2))))>0 77 | display('Error: Weight not Sum up to 1!') 78 | sum(par.W,2) 79 | par.W = par.W_old; 80 | end 81 | else 82 | par.W = par.W_old; 83 | end 84 | ELBO(inner, iter) = Cal_mcELBO(trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par); 85 | RMSE(inner, iter) = norm(diag(corrcov(par.C *par.C'),-1)- diag(fix.trueUpsilon,-1), 'fro')./norm(diag(fix.trueUpsilon,-1), 'fro'); 86 | end 87 | r1 = r1*dec; r2 = r2*dec; r3 = r3*dec; 88 | t1= toc; 89 | display(['ELBO = ', num2str(median(ELBO(:,iter))), ', Iter: ' num2str(iter), '/' num2str(MaxIter),' Elapsed: ', num2str(t1) ' sec']); 90 | end 91 | par.WinSet = WinSet; 92 | par.RMSE = RMSE; 93 | end 94 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/BPcdf_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial CDF basis function 2 | % Regularized incomplete beta function 3 | % u: evaluating point, could be a p by 1 vector 4 | % k: degree of BPs 5 | % opt: BP/exBP, standard or extended BP 6 | 7 | % V: a p by d matrix storing basis funcitons 8 | 9 | % Shaobo Han 10 | % 08/07/2015 11 | 12 | function V = BPcdf_basis(u, k, opt) 13 | if nargin<3, 14 | opt = 'BP'; % Standard BP by default 15 | end 16 | p = length(u); % # of variables 17 | switch opt 18 | case 'BP' % Standard BP 19 | d = k; % # of basis functions 20 | a_seq = 1:k; b_seq = k-a_seq+1; 21 | case 'exBP' % extended BP 22 | d = k*(k+1)/2; % # of basis functions 23 | a_seq = zeros(1, d); b_seq = zeros(1, d); 24 | tmp_idx = cumsum(1:k); 25 | for j = 1:k 26 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 27 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 28 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 29 | end 30 | end 31 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 32 | U = repmat(u, 1, d); % p by d matrix 33 | V = betacdf(U , A_seq, B_seq); % p by d matrix 34 | end 35 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/BPpdf_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial PDF basis function 2 | % Beta densities 3 | % u: evaluating point, could be a p by 1 vector 4 | % k: degree of BPs 5 | % opt: BP/exBP, standard or extended BP 6 | % V: a p by d matrix storing basis funcitons 7 | 8 | % Shaobo Han 9 | % 08/08/2015 10 | 11 | function V = BPpdf_basis(u, k, opt) 12 | if nargin<3, 13 | opt = 'BP'; % Standard BP by default 14 | end 15 | p = length(u); % # of variables 16 | switch opt 17 | case 'BP' % Standard BP 18 | d = k; % # of basis functions 19 | a_seq = 1:k; b_seq = k-a_seq+1; 20 | case 'exBP' % extended BP 21 | d = k*(k+1)/2; % # of basis functions 22 | a_seq = zeros(1, d); b_seq = zeros(1, d); 23 | tmp_idx = cumsum(1:k); 24 | for j = 1:k 25 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 26 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 27 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 28 | end 29 | end 30 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 31 | U = repmat(u, 1, d); % p by d matrix 32 | V = betapdf(U , A_seq, B_seq); % p by d matrix 33 | end 34 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/BPprime_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial prime basis function 2 | % First-order of Derivative Beta densities 3 | % u: evaluating point, could be a p by 1 vector 4 | % k: degree of BPs 5 | % opt: BP/exBP, standard or extended BP 6 | 7 | % V: a P by D matrix storing basis funcitons 8 | 9 | % Shaobo Han 10 | % 08/10/2015 11 | 12 | function V = BPprime_basis(u, k, opt) 13 | if nargin<3, 14 | opt = 'BP'; % Standard BP by default 15 | end 16 | p = length(u); % # of variables 17 | switch opt 18 | case 'BP' % Standard BP 19 | d = k; % # of basis functions 20 | a_seq = 1:k; b_seq = k-a_seq+1; 21 | case 'exBP' % extended BP 22 | d = k*(k+1)/2; % # of basis functions 23 | a_seq = zeros(1, d); b_seq = zeros(1, d); 24 | tmp_idx = cumsum(1:k); 25 | for j = 1:k 26 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 27 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 28 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 29 | end 30 | end 31 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 32 | U = repmat(u, 1, d); % p by d matrix 33 | V = (A_seq+B_seq-1).*(betapdf(U , A_seq-1, B_seq)-betapdf(U , A_seq, B_seq-1)); % p by d matrix 34 | % Remove NaN's 35 | % Defining betapdf(u, a, 0) = betapdf(u,0,b) = 0; 36 | Idx = (A_seq == 1)|(B_seq == 1); 37 | V(Idx) = 0; 38 | end 39 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/Cal_mcELBO.m: -------------------------------------------------------------------------------- 1 | function val = Cal_mcELBO(trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par) 2 | tmpMu=par.Mu; tmpC = par.C; tmpW=par.W; 3 | N_mc = opt.N_mc; PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; k = opt.k; 4 | P = length(tmpMu); sELBOM = zeros(1, N_mc); 5 | for i = 1: N_mc 6 | % Generate a P by 1 multivariate Normal vector 7 | w_t = randn(P,1); Z_t = tmpMu + tmpC*w_t; 8 | u = Phi_f(Z_t, PhiPar, PhiType); 9 | BPcdf_basiseval = BPcdf_basis(u, k, BPtype); 10 | tmpB = sum(BPcdf_basiseval.*tmpW, 2); 11 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 12 | logp = logmodel(tmpx, trueModel, fix); 13 | BPpdf_basiseval = BPpdf_basis(u, k, BPtype); 14 | tmpb = sum(BPpdf_basiseval.*tmpW, 2); 15 | sphi = sphi_f(Z_t, PhiPar, PhiType); 16 | spsi = spsi_f(tmpx, PsiPar, PsiType); 17 | hdz1 = diag(sphi./spsi)*tmpb; 18 | logq = logqpost(Z_t, inferModel, par); 19 | sELBOM(i) = logp + sum(log(hdz1))-logq; 20 | end 21 | sELBO = median(sELBOM); 22 | val = sELBO; 23 | end -------------------------------------------------------------------------------- /VGC-FlexibleMargins/Grad_MuC.m: -------------------------------------------------------------------------------- 1 | function [Delta_Mu, Delta_C, WinSet] = Grad_MuC(VGmethod, trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par, WinSet) 2 | NumberZ = opt.NumberZ; P = fix.P; 3 | PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; k = opt.k; 4 | Mu_old = par.Mu; C_old = par.C; W_old = par.W; 5 | MuMC = zeros(P, NumberZ); 6 | CMC = zeros(P,P, NumberZ); 7 | v_t = sqrt(diag(C_old*C_old')); 8 | 9 | for jj = 1:NumberZ 10 | %% Draw Sample w_t and Z_t 11 | Flag_outlier = 1; 12 | while Flag_outlier == 1 13 | % Generate a P by 1 multivariate Normal vector 14 | w_t = randn(P,1); Z_t = Mu_old + C_old*w_t; 15 | if opt.adaptivePhi == 1 16 | Z = diag(1./v_t)*(Z_t-Mu_old); 17 | u = Phi_f(Z, PhiPar, PhiType); 18 | else 19 | u = Phi_f(Z_t, PhiPar, PhiType); 20 | end 21 | 22 | BPcdf_basiseval = BPcdf_basis(u, k, BPtype); 23 | tmpB = sum(BPcdf_basiseval.*W_old, 2); 24 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 25 | logp = logmodel(tmpx, trueModel, fix); 26 | % Online outlier detection 27 | score = mzscore_test(logp, WinSet); 28 | if abs(score) < opt.OutlierTol 29 | Flag_outlier = 0; WinSet_old = WinSet; 30 | WinSet = [WinSet_old, logp]; 31 | WinSet(1) = []; 32 | end 33 | end 34 | if opt.adaptivePhi == 1 35 | sphi = diag(1./v_t)*sphi_f(Z, PhiPar, PhiType); 36 | dphi = -diag(1./v_t)*(Z.*sphi); 37 | else 38 | sphi = sphi_f(Z_t, PhiPar, PhiType); 39 | dphi = dphi_f(Z_t, PhiPar, PhiType); 40 | end 41 | 42 | BPpdf_basiseval = BPpdf_basis(u, k, BPtype); 43 | tmpb = sum(BPpdf_basiseval.*W_old, 2); 44 | BPprime_basiseval = BPprime_basis(u, k, BPtype); 45 | tmpbprime = sum(BPprime_basiseval.*W_old, 2); 46 | rho1prime = sphi.*tmpbprime; 47 | 48 | spsi = spsi_f(tmpx, PsiPar, PsiType); 49 | dpsi = dpsi_f(tmpx,PsiPar, PsiType); 50 | g = derivatives(tmpx, trueModel, fix); 51 | 52 | hdz1 = diag(sphi./spsi)*tmpb; 53 | rho3prime = hdz1.*dpsi; 54 | hdz2 = (rho1prime.*sphi.*spsi... 55 | +tmpb.*dphi.*spsi... 56 | -tmpb.*sphi.*rho3prime)./(spsi.^2); 57 | lnh1dz = hdz2./hdz1; 58 | ls_z = g.*hdz1+lnh1dz; 59 | switch VGmethod 60 | case 'Analytic' 61 | dmu = ls_z; 62 | dC = tril(ls_z*w_t')+ diag(1./diag(C_old)); 63 | case 'Numeric' 64 | tmppar.Mu = Mu_old; tmppar.Sigma = C_old*C_old'; 65 | g2 = logqpost_derivatives(Z_t, inferModel, tmppar); 66 | dmu = ls_z-g2; 67 | dC = tril(dmu*w_t'); 68 | end 69 | MuMC(:, jj) = dmu; 70 | CMC(:, :, jj) = dC; 71 | end 72 | Delta_Mu = mean(MuMC, 2); 73 | Delta_C= mean(CMC, 3); 74 | end 75 | 76 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/Grad_W.m: -------------------------------------------------------------------------------- 1 | function [Delta_w, WinSet] = Grad_W(trueModel, PhiType, PsiType, BPtype, fix, opt, par, WinSet) 2 | NumberZ = opt.NumberZ; P = fix.P; D = opt.D; 3 | PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; k = opt.k; 4 | Mu_old = par.Mu; C_old = par.C; W_old = par.W_old; 5 | WMC = zeros(P, D, NumberZ); 6 | for jj = 1:NumberZ 7 | %% Draw Sample w_t and Z_t 8 | Flag_outlier = 1; 9 | while Flag_outlier == 1 10 | % Generate a P by 1 multivariate Normal vector 11 | w_t = randn(P,1); Z_t = Mu_old + C_old*w_t; 12 | u = Phi_f(Z_t, PhiPar, PhiType); 13 | BPcdf_basiseval = BPcdf_basis(u, k, BPtype); 14 | tmpB = sum(BPcdf_basiseval.*W_old, 2); 15 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 16 | logp = logmodel(tmpx, trueModel, fix); 17 | % Online outlier detection 18 | score = mzscore_test(logp, WinSet); 19 | if abs(score) < opt.OutlierTol 20 | Flag_outlier = 0; WinSet_old = WinSet; 21 | WinSet = [WinSet_old, logp]; 22 | WinSet(1) = []; 23 | end 24 | end 25 | BPpdf_basiseval = BPpdf_basis(u, k, BPtype); 26 | tmpb = sum(BPpdf_basiseval.*W_old, 2); 27 | spsi = spsi_f(tmpx, PsiPar, PsiType); 28 | dpsi = dpsi_f(tmpx,PsiPar, PsiType); 29 | g = derivatives(tmpx, trueModel, fix); 30 | hdw = diag(1./spsi)*BPcdf_basiseval; 31 | lnh1dw = diag(1./tmpb)*BPpdf_basiseval - diag(dpsi./spsi.^2)*BPcdf_basiseval; 32 | ls_omega = diag(g)*hdw+lnh1dw; 33 | WMC(:,:,jj) = ls_omega; 34 | end 35 | Delta_w = mean(WMC, 3); 36 | end -------------------------------------------------------------------------------- /VGC-FlexibleMargins/Phi_f.m: -------------------------------------------------------------------------------- 1 | % Phi function: mapping from (-infty, infty) to unit closed interval [0,1] 2 | % Shaobo Han 3 | % 09/17/2015 4 | function u = Phi_f(z, par, opt) 5 | switch opt 6 | case 'Normal' 7 | m0 = par(1); v0 = par(2); 8 | u = normcdf(z, m0, sqrt(v0)); 9 | end 10 | end 11 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/SimplxProj.m: -------------------------------------------------------------------------------- 1 | function X = SimplxProj(Y) 2 | % Projects each row vector in the N by D marix Y to the probability 3 | % simplex in D-1 dimensions 4 | [N, D] = size(Y); 5 | X = sort(Y, 2, 'descend'); 6 | Xtmp = (cumsum(X,2)-1)*diag(sparse(1./(1:D))); 7 | X = max(bsxfun(@minus, Y, Xtmp(sub2ind([N,D], (1:N)', sum(X>Xtmp,2)))),0); 8 | end -------------------------------------------------------------------------------- /VGC-FlexibleMargins/demo_Beta.m: -------------------------------------------------------------------------------- 1 | % Flexible Margins example for the paper 2 | % "Variational Gaussian Copula Inference", 3 | % Shaobo Han, Xuejun Liao, David. B. Dunson, and Lawrence Carin, 4 | % The 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016) 5 | % ----------------------------- 6 | % Code written by 7 | % Shaobo Han, Duke University 8 | % shaobohan@gmail.com 9 | % 09/22/2015 10 | 11 | % Examples: 12 | % Beta Distribution 13 | 14 | clear all; close all; clc; 15 | % addpath(genpath(pwd)) % Add all sub-directories to the Matlab path 16 | fix. P = 1; % Number of latent variables 17 | trueModel = 'Beta'; 18 | fix.alpha = 0.5; fix.beta =0.5; 19 | inferModel ='MVNdiag'; 20 | 21 | fix.c = 1; % unnormalizing constant 22 | %% 23 | opt.k = 5; % Degree/Maximum Degree of Bernstein Polynomials 24 | opt.MaxIter = 10; % Number of SGD stages 25 | opt.NumberZ = 1; % Average Gradients 26 | opt.InnerIter = 500; % Number of iteration 27 | opt.N_mc = 1; % Number of median average in sELBO stepsize search 28 | 29 | BPtype = 'BP'; % Bernstein Polynomials 30 | % BPtype = 'exBP'; % Extended Bernstein Polynomials 31 | 32 | PsiType = 'Beta'; opt.PsiPar(1) = 2; opt.PsiPar(2) = 2; 33 | 34 | % PsiType = 'Normal'; opt.PsiPar(1) = 0; opt.PsiPar(2) = 1; % variance 35 | % PsiType = 'Exp'; opt.PsiPar = 0.5; 36 | PhiType = 'Normal'; opt.PhiPar(1) = 0; opt.PhiPar(2) =1; % variance 37 | 38 | % Learning rate 39 | opt.LearnRate.Mu = 0.001; opt.LearnRate.C =1e-3; 40 | opt.LearnRate.W = 1e-3; opt.LearnRate.dec = 0.95; % decreasing base learning rate 41 | 42 | switch BPtype 43 | case 'BP' 44 | opt.D = opt.k; 45 | case 'exBP' 46 | opt.D = opt.k*(opt.k+1)/2; % # of basis functions 47 | end 48 | 49 | % Diagonal constraint on Upsilon 50 | opt.diagUpsilon = 0; 51 | 52 | %% Initialization 53 | ini.Mu = opt.PhiPar(1).*ones(fix.P,1); 54 | ini.C = opt.PhiPar(2).*eye(fix.P); 55 | % ini.w = randBPw(fix.P, opt.D, 1, 1); 56 | ini.w = 1./opt.D.*ones(fix.P, opt.D); 57 | 58 | % Median Outlier Removal 59 | opt.OutlierTol = 10; % Threshold for online outlier detection 60 | opt.WinSize = 20; % Size of the window 61 | opt.Wthreshold = 1e8; opt.normalize = 0; 62 | ini.WinSet = ini_WinSet(trueModel, PhiType, PsiType, BPtype, fix, opt, ini); 63 | %% VGC-(Uniform)Adaptive Algorithm 64 | [ELBO, par] = vgcbp_w(trueModel, inferModel, PhiType, PsiType, BPtype,fix, ini, opt); 65 | opt.nsample = 2e4; 66 | Y_svc = sampleGC(PhiType, PsiType, BPtype, opt, par); 67 | Y_svc(sum(~isfinite(Y_svc),2)~=0,:) = []; 68 | figplot.nbins = 50; 69 | [figplot.f_x1,figplot.x_x1] = hist(Y_svc, figplot.nbins); 70 | 71 | %% VGC-(Non-Uniform) Algorithm 72 | opt.adaptivePhi = 0; VGmethod = 'Numeric'; 73 | [ELBO2, par2] = vgcbp_MuCW(trueModel, inferModel, PhiType, PsiType, BPtype, VGmethod, fix, ini, opt); 74 | Y_svc2 = sampleGC(PhiType, PsiType, BPtype, opt, par2); 75 | Y_svc2(sum(~isfinite(Y_svc2),2)~=0,:) = []; 76 | [figplot.f_x2,figplot.x_x2] = hist(Y_svc2, figplot.nbins); 77 | 78 | %% Plot 79 | figplot.nb = 100; 80 | figplot.Seqx = linspace(0, 1, figplot.nb)'; fix2.c = 1; 81 | figplot.log_pM = logmodel(figplot.Seqx, trueModel, fix); 82 | trueModel2 = PsiType; 83 | fix2.alpha = opt.PsiPar(1) ; fix2.beta = opt.PsiPar(2) ; fix2.c = 1; 84 | figplot.log_pM2 = logmodel(figplot.Seqx, trueModel2, fix2); 85 | 86 | figure 87 | figplot.ls =1; figplot.ms =1; 88 | set(gca,'fontsize', 14) 89 | plot(figplot.Seqx, exp(figplot.log_pM), '-', 'linewidth', 3.5*figplot.ls, 'Markersize', 4*figplot.ms, 'Color', 'k'); 90 | hold on 91 | plot(figplot.x_x1,figplot.f_x1/trapz(figplot.x_x1,figplot.f_x1), '-o', 'linewidth', 2*figplot.ls, 'Markersize', 5*figplot.ms, 'Color', 'g'); 92 | hold on 93 | plot(figplot.x_x2,figplot.f_x2/trapz(figplot.x_x2,figplot.f_x2), '--', 'linewidth', 3*figplot.ls, 'Markersize', 4*figplot.ms, 'Color', 'b'); 94 | hold on 95 | plot(figplot.Seqx, exp(figplot.log_pM2)./trapz(figplot.Seqx, exp(figplot.log_pM2)), '-*', 'linewidth', 2*figplot.ls, 'Markersize', 4.5*figplot.ms, 'Color', 'r'); 96 | hold off 97 | legend('Ground Truth', 'VIT-BP(k=5)', 'VGC-BP(k=5)' , 'Predefined \Psi') 98 | 99 | 100 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/demo_Gamma.m: -------------------------------------------------------------------------------- 1 | % Flexible Margins example for the paper 2 | % "Variational Gaussian Copula Inference", 3 | % Shaobo Han, Xuejun Liao, David. B. Dunson, and Lawrence Carin, 4 | % The 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016) 5 | % ----------------------------- 6 | % Code written by 7 | % Shaobo Han, Duke University 8 | % shaobohan@gmail.com 9 | % 09/22/2015 10 | 11 | % Examples: 12 | % Gamma Distribution 13 | 14 | clear all; close all; clc; 15 | 16 | % addpath(genpath(pwd)) % Add all sub-directories to the Matlab path 17 | 18 | fix. P = 1; % Number of latent variables 19 | trueModel = 'Gamma'; 20 | fix.alpha = 5; fix.beta = 2; 21 | inferModel ='MVNdiag'; 22 | fix.c = 1; % unnormalizing constant 23 | %% 24 | opt.k = 15; % Degree/Maximum Degree of Bernstein Polynomials 25 | opt.MaxIter = 50; % Number of SGD stages 26 | opt.NumberZ = 1; % Average Gradients 27 | opt.InnerIter = 500; % Number of iteration 28 | opt.N_mc = 1; % Number of median average in sELBO stepsize search 29 | 30 | BPtype = 'BP'; % Bernstein Polynomials 31 | % BPtype = 'exBP'; % Extended Bernstein Polynomials 32 | 33 | % PsiType = 'Normal'; opt.PsiPar(1) = 0; opt.PsiPar(2) = 1; % variance 34 | PsiType = 'Exp'; opt.PsiPar = 1; 35 | PhiType = 'Normal'; opt.PhiPar(1) = 0; opt.PhiPar(2) =1; % variance 36 | 37 | % Learning rate 38 | opt.LearnRate.Mu = 0.001; opt.LearnRate.C =1e-3; 39 | opt.LearnRate.W = 1e-3; opt.LearnRate.dec = 0.95; % decreasing base learning rate 40 | 41 | switch BPtype 42 | case 'BP' 43 | opt.D = opt.k; 44 | case 'exBP' 45 | opt.D = opt.k*(opt.k+1)/2; % # of basis functions 46 | end 47 | 48 | % Diagonal constraint on Upsilon 49 | opt.diagUpsilon = 0; 50 | 51 | %% Initialization 52 | ini.Mu = opt.PhiPar(1).*ones(fix.P,1); 53 | ini.C = opt.PhiPar(2).*eye(fix.P); 54 | ini.w = randBPw(fix.P, opt.D, 1, 1); 55 | % ini.w = 1./opt.D.*ones(fix.P, opt.D); 56 | 57 | % Median Outlier Removal 58 | opt.OutlierTol = 10; % Threshold for online outlier detection 59 | opt.WinSize = 20; % Size of the window 60 | 61 | opt.Wthreshold = 1e8; opt.normalize = 0; 62 | ini.WinSet = ini_WinSet(trueModel, PhiType, PsiType, BPtype, fix, opt, ini); 63 | %% VGC-(Uniform)Adaptive Algorithm 64 | [ELBO, par] = vgcbp_w(trueModel, inferModel, PhiType, PsiType, BPtype,fix, ini, opt); 65 | opt.nsample = 2e4; 66 | Y_svc = sampleGC(PhiType, PsiType, BPtype, opt, par); 67 | Y_svc(sum(~isfinite(Y_svc),2)~=0,:) = []; 68 | figplot.nbins = 50; 69 | [figplot.f_x1,figplot.x_x1] = hist(Y_svc, figplot.nbins); 70 | % VCSBPC = hist1D2D(Y_svc, nbins); 71 | 72 | %% VGC-(Non-Uniform) Algorithm 73 | opt.adaptivePhi = 0; VGmethod = 'Numeric'; 74 | [ELBO2, par2] = vgcbp_MuCW(trueModel, inferModel, PhiType, PsiType, BPtype, VGmethod, fix, ini, opt); 75 | Y_svc2 = sampleGC(PhiType, PsiType, BPtype, opt, par2); 76 | Y_svc2(sum(~isfinite(Y_svc2),2)~=0,:) = []; 77 | [figplot.f_x2,figplot.x_x2] = hist(Y_svc2, figplot.nbins); 78 | 79 | %% Plot 80 | figplot.nb = 100; figplot.Seqx = linspace(0, 20, figplot.nb)'; 81 | figplot.log_pM = logmodel(figplot.Seqx, trueModel, fix); 82 | trueModel2 = PsiType; fix2.lambda = opt.PsiPar; fix2.c = 1; 83 | figplot.log_pM2 = logmodel(figplot.Seqx, trueModel2, fix2); 84 | 85 | figure 86 | figplot.ls =1; figplot.ms =1; 87 | set(gca,'fontsize', 14) 88 | plot(figplot.Seqx, exp(figplot.log_pM), '-', 'linewidth', 3.5*figplot.ls, 'Markersize', 4*figplot.ms, 'Color', 'k'); 89 | hold on 90 | plot(figplot.x_x1,figplot.f_x1/trapz(figplot.x_x1,figplot.f_x1), '-o', 'linewidth', 2*figplot.ls, 'Markersize', 5*figplot.ms, 'Color', 'g'); 91 | hold on 92 | plot(figplot.x_x2,figplot.f_x2/trapz(figplot.x_x2,figplot.f_x2), '--', 'linewidth', 3*figplot.ls, 'Markersize', 4*figplot.ms, 'Color', 'b'); 93 | hold on 94 | plot(figplot.Seqx, exp(figplot.log_pM2)./trapz(figplot.Seqx, exp(figplot.log_pM2)), '-*', 'linewidth', 2*figplot.ls, 'Markersize', 4.5*figplot.ms, 'Color', 'r'); 95 | hold off 96 | legend('Ground Truth', 'VIT-BP(k=15)', 'VGC-BP(k=15)' , 'Predefined \Psi') 97 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/demo_SkewNormal.m: -------------------------------------------------------------------------------- 1 | % Flexible Margins example for the paper 2 | % "Variational Gaussian Copula Inference", 3 | % Shaobo Han, Xuejun Liao, David. B. Dunson, and Lawrence Carin, 4 | % The 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016) 5 | % ----------------------------- 6 | % Code written by 7 | % Shaobo Han, Duke University 8 | % shaobohan@gmail.com 9 | % 09/22/2015 10 | 11 | % Examples: 12 | % Skew Normal Distribution 13 | 14 | clear all; close all; clc; 15 | % addpath(genpath(pwd)) % Add all sub-directories to the Matlab path 16 | 17 | fix. P = 1; % Number of latent variables 18 | 19 | % True SN Posterior 20 | trueModel = 'SkewNormal'; 21 | fix.snMu = 0; fix.snSigma = 1; fix.snAlpha = 5; 22 | inferModel ='MVNdiag'; 23 | fix.c = 2; % unnormalizing constant 24 | %% 25 | opt.k = 15; % Degree/Maximum Degree of Bernstein Polynomials 26 | opt.MaxIter = 50; % Number of SGD stages 27 | opt.NumberZ = 1; % Average Gradients 28 | opt.InnerIter = 500; % Number of iteration 29 | opt.N_mc = 1; % Number of median average in sELBO stepsize search 30 | BPtype = 'BP'; % Bernstein Polynomials 31 | % BPtype = 'exBP'; % Extended Bernstein Polynomials 32 | 33 | PsiType = 'Normal'; opt.PsiPar(1) = 0; opt.PsiPar(2) = 1; % variance 34 | % PsiType = 'Exp'; opt.PsiPar = 1; 35 | PhiType = 'Normal'; opt.PhiPar(1) = 0; opt.PhiPar(2) =1; % variance 36 | 37 | % Learning rate 38 | opt.LearnRate.Mu = 0.001; opt.LearnRate.C =1e-3; 39 | opt.LearnRate.W = 0.5*1e-3; opt.LearnRate.dec = 0.95; % decreasing base learning rate 40 | 41 | switch BPtype 42 | case 'BP' 43 | opt.D = opt.k; 44 | case 'exBP' 45 | opt.D = opt.k*(opt.k+1)/2; % # of basis functions 46 | end 47 | 48 | % Diagonal constraint on Upsilon 49 | opt.diagUpsilon = 0; 50 | %% Initialization 51 | ini.Mu = opt.PhiPar(1).*ones(fix.P,1); 52 | ini.C = opt.PhiPar(2).*eye(fix.P); 53 | % ini.w = randBPw(fix.P, opt.D, 1, 1); 54 | ini.w = 1./opt.D.*ones(fix.P, opt.D); 55 | 56 | % Median Outlier Removal 57 | opt.OutlierTol = 10; % Threshold for online outlier detection 58 | opt.WinSize = 20; % Size of the window 59 | opt.Wthreshold = 1e12; opt.normalize = 0; 60 | ini.WinSet = ini_WinSet(trueModel, PhiType, PsiType, BPtype, fix, opt, ini); 61 | %% VGC-(Uniform)Adaptive Algorithm 62 | [ELBO, par] = vgcbp_w(trueModel, inferModel, PhiType, PsiType, BPtype,fix, ini, opt); 63 | opt.nsample = 5e5; 64 | Y_svc = sampleGC(PhiType, PsiType, BPtype, opt, par); 65 | Y_svc(sum(~isfinite(Y_svc),2)~=0,:) = []; 66 | figplot.nbins = 50; 67 | [figplot.f_x1,figplot.x_x1] = hist(Y_svc, figplot.nbins); 68 | 69 | %% VGC-(Non-Uniform) Algorithm 70 | opt.adaptivePhi = 0; VGmethod = 'Numeric'; 71 | [ELBO2, par2] = vgcbp_MuCW(trueModel, inferModel, PhiType, PsiType, BPtype, VGmethod, fix, ini, opt); 72 | Y_svc2 = sampleGC(PhiType, PsiType, BPtype, opt, par2); 73 | Y_svc2(sum(~isfinite(Y_svc2),2)~=0,:) = []; 74 | [figplot.f_x2,figplot.x_x2] = hist(Y_svc2, figplot.nbins); 75 | 76 | %% Plot 77 | figplot.nb = 100; figplot.Seqx = linspace(-5, 5, figplot.nb)'; 78 | figplot.log_pM = logmodel(figplot.Seqx, trueModel, fix); 79 | trueModel2 = 'SkewNormal'; 80 | fix2.snMu = opt.PsiPar(1) ; fix2.snSigma = opt.PsiPar(2) ; fix2.snAlpha =0; fix2.c = 1; 81 | figplot.log_pM2 = logmodel(figplot.Seqx, trueModel2, fix2); 82 | 83 | figure 84 | figplot.ls =1; figplot.ms =1; 85 | set(gca,'fontsize', 14) 86 | plot(figplot.Seqx, exp(figplot.log_pM)./trapz(figplot.Seqx, exp(figplot.log_pM)), '-', 'linewidth', 3.5*figplot.ls, 'Markersize', 4*figplot.ms, 'Color', 'k'); 87 | hold on 88 | plot(figplot.x_x1,figplot.f_x1/trapz(figplot.x_x1,figplot.f_x1), '-o', 'linewidth', 2*figplot.ls, 'Markersize', 5*figplot.ms, 'Color', 'g'); 89 | hold on 90 | plot(figplot.x_x2,figplot.f_x2/trapz(figplot.x_x2,figplot.f_x2), '--', 'linewidth', 3*figplot.ls, 'Markersize', 4*figplot.ms, 'Color', 'b'); 91 | hold on 92 | plot(figplot.Seqx, exp(figplot.log_pM2)./trapz(figplot.Seqx, exp(figplot.log_pM2)), '-*', 'linewidth', 2*figplot.ls, 'Markersize', 4.5*figplot.ms, 'Color', 'r'); 93 | hold off 94 | legend('Ground Truth', 'VIT-BP(k=15)', 'VGC-BP(k=15)' , 'Predefined \Psi') 95 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/demo_StudentT.m: -------------------------------------------------------------------------------- 1 | % Flexible Margins example for the paper 2 | % "Variational Gaussian Copula Inference", 3 | % Shaobo Han, Xuejun Liao, David. B. Dunson, and Lawrence Carin, 4 | % The 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016) 5 | % ----------------------------- 6 | % Code written by 7 | % Shaobo Han, Duke University 8 | % shaobohan@gmail.com 9 | % 09/22/2015 10 | 11 | % Examples: 12 | % Student's t Distribution 13 | 14 | clear all; close all; clc; 15 | % addpath(genpath(pwd)) % Add all sub-directories to the Matlab path 16 | 17 | fix. P = 1; % Number of latent variables 18 | trueModel = 'Student'; 19 | fix.nu = 1; 20 | inferModel ='MVNdiag'; 21 | fix.c = 1; % unnormalizing constant 22 | %% 23 | opt.k =15; % Degree/Maximum Degree of Bernstein Polynomials 24 | opt.MaxIter = 20; % Number of SGD stages 25 | opt.NumberZ = 1; % Average Gradients 26 | opt.InnerIter = 500; % Number of iteration 27 | opt.N_mc = 1; % Number of median average in sELBO stepsize search 28 | 29 | BPtype = 'BP'; % Bernstein Polynomials 30 | % BPtype = 'exBP'; % Extended Bernstein Polynomials 31 | 32 | PsiType = 'Normal'; opt.PsiPar(1) = 0; opt.PsiPar(2) = 1; % variance 33 | % PsiType = 'Exp'; opt.PsiPar = 0.5; 34 | PhiType = 'Normal'; opt.PhiPar(1) = 0; opt.PhiPar(2) =1; % variance 35 | 36 | % Learning rate 37 | opt.LearnRate.Mu = 0.001; opt.LearnRate.C =1e-3; 38 | opt.LearnRate.W = 0.5*1e-3; opt.LearnRate.dec = 0.95; % decreasing base learning rate 39 | 40 | switch BPtype 41 | case 'BP' 42 | opt.D = opt.k; 43 | case 'exBP' 44 | opt.D = opt.k*(opt.k+1)/2; % # of basis functions 45 | end 46 | 47 | % Diagonal constraint on Upsilon 48 | opt.diagUpsilon = 0; 49 | 50 | %% Initialization 51 | ini.Mu = opt.PhiPar(1).*ones(fix.P,1); 52 | ini.C = opt.PhiPar(2).*eye(fix.P); 53 | % ini.w = randBPw(fix.P, opt.D, 1, 1); 54 | ini.w = 1./opt.D.*ones(fix.P, opt.D); 55 | 56 | % Median Outlier Removal 57 | opt.OutlierTol = 10; % Threshold for online outlier detection 58 | opt.WinSize = 20; % Size of the window 59 | 60 | opt.Wthreshold = 1e12; opt.normalize = 0; 61 | ini.WinSet = ini_WinSet(trueModel, PhiType, PsiType, BPtype, fix, opt, ini); 62 | %% VGC-(Uniform)Adaptive Algorithm 63 | [ELBO, par] = vgcbp_w(trueModel, inferModel, PhiType, PsiType, BPtype,fix, ini, opt); 64 | opt.nsample = 5e5; 65 | Y_svc = sampleGC(PhiType, PsiType, BPtype, opt, par); 66 | Y_svc(sum(~isfinite(Y_svc),2)~=0,:) = []; 67 | figplot.nbins = 50; 68 | [figplot.f_x1,figplot.x_x1] = hist(Y_svc, figplot.nbins); 69 | 70 | %% VGC-(Non-Uniform) Algorithm 71 | opt.adaptivePhi = 0; VGmethod = 'Numeric'; 72 | [ELBO2, par2] = vgcbp_MuCW(trueModel, inferModel, PhiType, PsiType, BPtype, VGmethod, fix, ini, opt); 73 | Y_svc2 = sampleGC(PhiType, PsiType, BPtype, opt, par2); 74 | Y_svc2(sum(~isfinite(Y_svc2),2)~=0,:) = []; 75 | [figplot.f_x2,figplot.x_x2] = hist(Y_svc2, figplot.nbins); 76 | 77 | %% Plot 78 | figplot.nb = 100; figplot.Seqx = linspace(-8, 8, figplot.nb)'; 79 | figplot.log_pM = logmodel(figplot.Seqx, trueModel, fix); 80 | trueModel2 = 'SkewNormal'; 81 | fix2.snMu = 0; fix2.snSigma = 1; fix2.snAlpha =0; fix2.c = 1; 82 | figplot.log_pM2 = logmodel(figplot.Seqx, trueModel2, fix2); 83 | 84 | figure 85 | figplot.ls =1; figplot.ms =1; 86 | set(gca,'fontsize', 16) 87 | plot(figplot.Seqx, exp(figplot.log_pM)./trapz(figplot.Seqx, exp(figplot.log_pM)), '-', 'linewidth', 3.5*figplot.ls, 'Markersize', 4*figplot.ms, 'Color', 'k'); 88 | hold on 89 | plot(figplot.x_x1,figplot.f_x1/trapz(figplot.x_x1,figplot.f_x1), '--', 'linewidth', 2.5*figplot.ls, 'Markersize', 4*figplot.ms, 'Color', 'g'); 90 | hold on 91 | plot(figplot.x_x2,figplot.f_x2/trapz(figplot.x_x2,figplot.f_x2), '-o', 'linewidth', 2*figplot.ls, 'Markersize', 4*figplot.ms, 'Color', 'b'); 92 | hold on 93 | plot(figplot.Seqx, exp(figplot.log_pM2)./trapz(figplot.Seqx, exp(figplot.log_pM2)), '-.', 'linewidth', 4*figplot.ls, 'Markersize', 4*figplot.ms, 'Color', 'r'); 94 | hold off 95 | legend('Ground Truth', 'VIT-BP(k=15)', 'VGC-BP(k=15)' , 'Predefined \Psi') 96 | 97 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/derivatives.m: -------------------------------------------------------------------------------- 1 | % Derivatives of Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function g = derivatives(x, model, fix) 7 | switch model 8 | case 'SkewNormal' 9 | m = fix.snMu; 10 | Lambda = fix.snSigma; 11 | alpha = fix.snAlpha; 12 | g = -(Lambda\(x-m))+mvnpdf(alpha'*(x-m)).*alpha./mvncdf(alpha'*(x-m)); 13 | case 'Gamma' 14 | alpha = fix.alpha; 15 | beta = fix.beta; 16 | g = (alpha-1)./x-beta; 17 | case 'Student' 18 | nu = fix.nu; 19 | g = -(nu+1)*x/(nu+x^2); 20 | case 'Beta'; 21 | alpha = fix.alpha; beta = fix.beta; 22 | g = (alpha-1)/x-(beta-1)/(1-x); 23 | end 24 | end 25 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/dphi_f.m: -------------------------------------------------------------------------------- 1 | % dphi function: derivative of sphi 2 | % Shaobo Han 3 | % 09/24/2015 4 | 5 | function output = dphi_f(z, par, opt) 6 | switch opt 7 | case 'Normal' 8 | m0 = par(1); v0 = par(2); 9 | output = -(z./v0).*normpdf(z, m0, sqrt(v0)); 10 | end 11 | end -------------------------------------------------------------------------------- /VGC-FlexibleMargins/dpsi_f.m: -------------------------------------------------------------------------------- 1 | % dpsi function: derivative of small psi 2 | % Shaobo Han 3 | % 08/10/2015 4 | 5 | function output = dpsi_f(x, par, opt) 6 | if nargin<3, 7 | opt = 'Exp'; % exponential distribution by default 8 | end 9 | if nargin<2, 10 | par = 0.1; % lambda = 0.1 by default 11 | end 12 | switch opt 13 | case 'Exp' % exp(lambda) 14 | output = -par.*exppdf(x,1/par); 15 | case 'Normal' % 16 | m0 = par(1); % mean 17 | v0 = par(2); % variance 18 | output = (-x./v0).*normpdf(x,m0,sqrt(v0)); 19 | case 'Beta' 20 | a0 = par(1); b0 = par(2); 21 | output =(a0+b0-1)*(betapdf(x, a0-1, b0)- betapdf(x, a0, b0-1)); 22 | end 23 | end -------------------------------------------------------------------------------- /VGC-FlexibleMargins/iPsi_f.m: -------------------------------------------------------------------------------- 1 | % Invserse Psi function: 2 | % Shaobo Han 3 | % 08/10/2015 4 | 5 | function output = iPsi_f(p, par, opt) 6 | if nargin<3, 7 | opt = 'Exp'; % exponential distribution by default 8 | end 9 | if nargin<2, 10 | par = 0.1; % lambda = 0.1 by default 11 | end 12 | 13 | p(p>1-1e-16) = 1-1e-16; % numerical stability 14 | 15 | switch opt 16 | case 'Exp' % exp(lambda) 17 | % output = -log(1-p)./par; 18 | output = expinv(p,1/par); 19 | case 'Normal' % 20 | m0 = par(1); v0 = par(2); 21 | output = norminv(p,m0,sqrt(v0)); 22 | case 'Beta' 23 | a0 = par(1); b0 = par(2); 24 | output = betainv(p, a0, b0); 25 | end 26 | end -------------------------------------------------------------------------------- /VGC-FlexibleMargins/ini_WinSet.m: -------------------------------------------------------------------------------- 1 | function WinSet = ini_WinSet(trueModel, PhiType, PsiType, BPtype, fix, opt, ini) 2 | k = opt.k; PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; 3 | P = fix.P; Mu_old = ini.Mu; C_old = ini.C; W_old = ini.w; 4 | WinSet = zeros(1, opt.WinSize); 5 | for i = 1:opt.WinSize 6 | w_t0 = randn(P,1); Z_t0 = Mu_old + C_old*w_t0; 7 | u0 = Phi_f(Z_t0, PhiPar, PhiType); 8 | BPcdf_basiseval0 = BPcdf_basis(u0, k, BPtype); 9 | tmpB0 = sum(BPcdf_basiseval0.*W_old, 2); 10 | tmpx0 = iPsi_f(tmpB0, PsiPar, PsiType); 11 | WinSet(i) = logmodel(tmpx0, trueModel, fix); 12 | end 13 | end -------------------------------------------------------------------------------- /VGC-FlexibleMargins/logdet.m: -------------------------------------------------------------------------------- 1 | function y = logdet(A) 2 | %LOGDET Logarithm of determinant for positive-definite matrix 3 | % logdet(A) returns log(det(A)) where A is positive-definite. 4 | % This is faster and more stable than using log(det(A)). 5 | % Note that logdet does not check if A is positive-definite. 6 | % If A is not positive-definite, the result will not be the same as log(det(A)). 7 | 8 | % Written by Tom Minka 9 | % (c) Microsoft Corporation. All rights reserved. 10 | 11 | U = chol(A); 12 | y = 2*sum(log(diag(U))); -------------------------------------------------------------------------------- /VGC-FlexibleMargins/logmodel.m: -------------------------------------------------------------------------------- 1 | % Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function p = logmodel(x, model, fix) 7 | switch model 8 | case 'SkewNormal' 9 | m = fix.snMu; 10 | Lambda = fix.snSigma; 11 | alpha = fix.snAlpha; 12 | c = fix.c; 13 | p = log(2)+log(mvnpdf(x,m,Lambda))+log(normcdf(alpha'*(x-m)))+log(c) ; 14 | case 'Gamma' 15 | alpha = fix.alpha; 16 | beta = fix.beta; 17 | c = fix.c; 18 | % p = alpha*log(beta)-log(gamma(alpha))+(alpha-1)*log(x)-beta*x+log(c) ; 19 | p = log(gampdf(x, alpha, 1/beta))+log(c); 20 | case 'Student' 21 | nu = fix.nu; c = fix.c; 22 | p = log(tpdf(x,nu))+log(c); 23 | case 'Exp' 24 | lambda = fix.lambda; c = fix.c; 25 | p = log(exppdf(x,1/lambda))+log(c); 26 | case 'Beta'; 27 | alpha = fix.alpha; beta = fix.beta; c = fix.c; 28 | p = log(betapdf(x, alpha, beta))+log(c); 29 | end 30 | end 31 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/logqpost.m: -------------------------------------------------------------------------------- 1 | % Log q posteiror 2 | % Model dependent term 3 | % Shaobo Han 4 | % 09/11/2015 5 | 6 | function p = logqpost(x, model, par) 7 | switch model 8 | case 'SkewNormal' 9 | m = par.snMu; 10 | Lambda = par.snSigma; 11 | alpha = par.snAlpha; 12 | p = log(2)+log(mvnpdf(x,m,Lambda))+log(normcdf(alpha'*(x-m))) ; 13 | case 'MVN' 14 | mu = par.Mu; C= par.C; Sigma=C*C'; 15 | d = length(mu); 16 | p = -d*log(2*pi)/2-0.5*logdet(Sigma)-0.5*(x-mu)'*(Sigma\(x-mu)); 17 | case 'MVNdiag' 18 | mu = par.Mu; C= par.C; Sigma=C*C'; 19 | p = sum(log(normpdf(x, mu, sqrt(diag(Sigma))))); 20 | end 21 | end 22 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/logqpost_derivatives.m: -------------------------------------------------------------------------------- 1 | % Derivatives of Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function g = logqpost_derivatives(x, model, par) 7 | switch model 8 | case 'SkewNormal' 9 | m = par.snMu; 10 | Lambda = par.snSigma; 11 | alpha = par.snAlpha; 12 | g = -inv(Lambda)*(x-m)+mvnpdf(alpha'*(x-m)).*alpha./mvncdf(alpha'*(x-m)); 13 | case 'MVN' 14 | mu = par.Mu; Sigma= par.Sigma; 15 | g = Sigma\(mu-x); 16 | case 'MVNdiag' 17 | mu = par.Mu; Sigma= par.Sigma; 18 | g = diag(1./diag(Sigma))*(mu-x); 19 | end 20 | end 21 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/mzscore_test.m: -------------------------------------------------------------------------------- 1 | % Calculate the modified Z-score of a query point from a window set 2 | % Shaobo Han 3 | % 08/13/2015 4 | function score = mzscore_test(x, windowset) 5 | tmpX = [x, windowset]; 6 | tmpm = median(tmpX); 7 | mad = median(abs(tmpX-tmpm)); 8 | score = 0.6745*(x-tmpm)/mad; 9 | end -------------------------------------------------------------------------------- /VGC-FlexibleMargins/randBPw.m: -------------------------------------------------------------------------------- 1 | % Generate a P*D BP weight matrix 2 | % (every column is a probability vector which sums up to one) 3 | % P: number of variables 4 | % D: number of BP basis functions 5 | % a: shape parameter; b: scale parameter 6 | % Shaobo Han 7 | % 08/08/2015 8 | 9 | function w = randBPw(P, D, a, b) 10 | if nargin<4 11 | a = 1; b = 1; % by default 12 | end 13 | if D==1 14 | w = ones(P,1); 15 | else 16 | w0 = gamrnd(a,b,P,D); 17 | w = diag(1./sum(w0,2))*w0; 18 | end 19 | end -------------------------------------------------------------------------------- /VGC-FlexibleMargins/sampleGC.m: -------------------------------------------------------------------------------- 1 | function Y_svc = sampleGC(PhiType, PsiType, BPtype, opt, par) 2 | PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; 3 | nsample = opt.nsample; k = opt.k; 4 | Mu = par.Mu; C = par.C; W = par.W; 5 | % Sample from Gaussian Copula 6 | optmu_s2 = Mu; optSig_s2 = C*C'; 7 | wtz2 = mvnrnd(optmu_s2, optSig_s2, nsample); 8 | tau_svc2 = zeros(nsample,1); 9 | for ns = 1:nsample 10 | tmpz = wtz2(ns,:)'; 11 | tmpu = Phi_f(tmpz, PhiPar, PhiType); 12 | BPcdf_basiseval = BPcdf_basis(tmpu, k, BPtype); 13 | tmpB = sum(BPcdf_basiseval.*W, 2); 14 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 15 | tau_svc2(ns,1) = tmpx(1); 16 | end 17 | Y_svc = tau_svc2; 18 | end -------------------------------------------------------------------------------- /VGC-FlexibleMargins/sphi_f.m: -------------------------------------------------------------------------------- 1 | % phi function: small phi, derivative of Phi 2 | % Shaobo Han 3 | % 08/07/2015 4 | function output = sphi_f(z, par, opt) 5 | switch opt 6 | case 'Normal' 7 | m0 = par(1); v0 = par(2); 8 | output = normpdf(z, m0, sqrt(v0)); 9 | end 10 | end 11 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/spsi_f.m: -------------------------------------------------------------------------------- 1 | % psi function: small psi, derivative of Psi 2 | % Shaobo Han 3 | % 08/10/2015 4 | 5 | function output = spsi_f(x, par, opt) 6 | if nargin<3, 7 | opt = 'Exp'; % exponential distribution by default 8 | end 9 | if nargin<2, 10 | par = 0.1; % lambda = 0.1 by default 11 | end 12 | switch opt 13 | case 'Exp' % exp(lambda) 14 | output = exppdf(x,1/par); 15 | case 'Normal' % 16 | m0 = par(1); v0 = par(2); 17 | output = normpdf(x,m0,sqrt(v0)); 18 | case 'Beta' 19 | a0 = par(1); b0 = par(2); 20 | output = betapdf(x, a0, b0); 21 | end 22 | end -------------------------------------------------------------------------------- /VGC-FlexibleMargins/vgcbp_MuCW.m: -------------------------------------------------------------------------------- 1 | function [ELBO, par] = vgcbp_MuCW(trueModel, inferModel, PhiType, PsiType, BPtype, VGmethod, fix, ini, opt) 2 | % Parameters Unpack 3 | InnerIter = opt.InnerIter; MaxIter = opt. MaxIter; 4 | r1 = opt.LearnRate.Mu; r2 = opt.LearnRate.C; 5 | r3 = opt.LearnRate.W; dec = opt.LearnRate.dec; 6 | 7 | if opt.diagUpsilon ==1 8 | display(['[VGC(S): diag], BPtype = ', BPtype,' PsiType = ', PsiType,' PhiType = ', PhiType]) 9 | else 10 | display(['[VGC(S): full], BPtype = ', BPtype,' PsiType = ', PsiType,' PhiType = ', PhiType]) 11 | end 12 | ELBO = zeros(InnerIter, MaxIter); 13 | for iter = 1: MaxIter 14 | tic; 15 | for inner = 1:InnerIter 16 | % First Iteration 17 | if (iter == 1)&&(inner==1) 18 | par.Mu_old = ini.Mu; par.C_old = ini.C; par.W_old = ini.w; 19 | WinSet = ini.WinSet; 20 | par.Mu = par.Mu_old; par.C = par.C_old; par.W = par.W_old; 21 | else 22 | par.Mu_old = par.Mu; par.C_old = par.C; par.W_old = par.W; 23 | end 24 | %% Draw Samples and Calculate (Mu, C) 25 | [Delta_Mu, Delta_C, WinSet] = Grad_MuC(VGmethod, trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par,WinSet); 26 | if opt.normalize ==1 27 | P= fix.P; 28 | if norm(Delta_Mu,'fro')<=1e-5 29 | Delta_Mu = zeros(P,1); 30 | else 31 | Delta_Mu = Delta_Mu./norm(Delta_Mu,'fro'); 32 | end 33 | if norm(Delta_C,'fro')<=1e-5 34 | Delta_C = zeros(P,P); 35 | else 36 | Delta_C = Delta_C./norm(Delta_C,'fro'); 37 | end 38 | end 39 | %% Update Mu 40 | par.Mu = par.Mu_old + r1.*Delta_Mu; 41 | if opt.adaptivePhi == 1 42 | par.Mu = zeros(size( par.Mu)); 43 | end 44 | %% Update Mu 45 | par.C = par.C_old + r2.*Delta_C; 46 | %% Draw Samples and Calculate Delta_W 47 | [Delta_w, WinSet] = Grad_W(trueModel, PhiType, PsiType, BPtype, fix, opt, par,WinSet); 48 | if opt.normalize ==1 49 | % Normalize 50 | if norm(Delta_w,'fro')<=1e-5 51 | Delta_w = zeros(size(Delta_w)); 52 | else 53 | Delta_w = diag(1./sqrt(sum(abs(Delta_w).^2,2)))*Delta_w; 54 | end 55 | end 56 | %% Update W 57 | tmpNorm = sqrt(sum(abs(Delta_w).^2,2)); 58 | UnstableIndex = (tmpNorm>opt.Wthreshold); 59 | tmpsum = sum(UnstableIndex); 60 | if tmpsum ~=0 61 | Delta_w(UnstableIndex,:) = zeros(tmpsum, opt.D); 62 | display('Warning: Unstable Delta W Omitted!') 63 | tmpNorm(UnstableIndex) 64 | end 65 | W0 = par.W_old + r3.*Delta_w; 66 | % Project Gradient Descent 67 | par.W = SimplxProj(W0); 68 | if sum(abs(sum(par.W,2)-1)>1e-5*ones(size(sum(par.W,2))))>0 69 | display('Error: Weight not Sum up to 1!') 70 | sum(par.W,2) 71 | par.W = par.W_old; 72 | end 73 | ELBO(inner, iter) = Cal_mcELBO(trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par); 74 | end 75 | r1 = r1*dec; r2 = r2*dec; r3 = r3*dec; 76 | t1= toc; 77 | display(['ELBO = ', num2str(median(ELBO(:,iter))), ', Iter: ' num2str(iter), '/' num2str(MaxIter),' Elapsed: ', num2str(t1) ' sec']); 78 | end 79 | par.WinSet = WinSet; 80 | end 81 | -------------------------------------------------------------------------------- /VGC-FlexibleMargins/vgcbp_w.m: -------------------------------------------------------------------------------- 1 | function [ELBO, par] = vgcbp_w(trueModel, inferModel, PhiType, PsiType, BPtype, fix, ini, opt) 2 | % Parameters Unpack 3 | InnerIter = opt.InnerIter; MaxIter = opt. MaxIter; 4 | r3 = opt.LearnRate.W; dec = opt.LearnRate.dec; 5 | if opt.diagUpsilon ==1 6 | display(['[VGC(S): diag], BPtype = ', BPtype,' PsiType = ', PsiType,' PhiType = ', PhiType]) 7 | else 8 | display(['[VGC(S): full], BPtype = ', BPtype,' PsiType = ', PsiType,' PhiType = ', PhiType]) 9 | end 10 | ELBO = zeros(InnerIter, MaxIter); 11 | for iter = 1: MaxIter 12 | tic; 13 | for inner = 1:InnerIter 14 | % First Iteration 15 | if (iter == 1)&&(inner==1) 16 | par.Mu_old = ini.Mu; par.C_old = ini.C; par.W_old = ini.w; 17 | WinSet = ini.WinSet; 18 | else 19 | par.Mu_old = par.Mu; par.C_old = par.C; par.W_old = par.W; 20 | end 21 | % Do Not Update (Mu, C) Here 22 | par.Mu = par.Mu_old; par.C = par.C_old; 23 | %% Draw Samples and Calculates 24 | [Delta_w, WinSet] = Grad_W(trueModel, PhiType, PsiType, BPtype, fix, opt, par,WinSet); 25 | if opt.normalize ==1 26 | if norm(Delta_w,'fro')<=1e-5 27 | Delta_w = zeros(size(Delta_w)); 28 | else 29 | Delta_w = diag(1./sqrt(sum(abs(Delta_w).^2,2)))*Delta_w; 30 | end 31 | end 32 | 33 | %% Update W 34 | tmpNorm = sqrt(sum(abs(Delta_w).^2,2)); 35 | UnstableIndex = (tmpNorm>opt.Wthreshold); 36 | tmpsum = sum(UnstableIndex); 37 | if tmpsum ~=0 38 | Delta_w(UnstableIndex,:) = zeros(tmpsum, opt.D); 39 | display('Warning: Unstable Delta W Omitted!') 40 | tmpNorm(UnstableIndex) 41 | end 42 | W0 = par.W_old + r3.*Delta_w; 43 | % Project Gradient Descent 44 | par.W = SimplxProj(W0); 45 | if sum(abs(sum(par.W,2)-1)>1e-5*ones(size(sum(par.W,2))))>0 46 | display('Error: weight not sum up to 1!') 47 | sum(par.W,2) 48 | par.W = par.W_old; 49 | end 50 | ELBO(inner, iter) = Cal_mcELBO(trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par); 51 | end 52 | r3 = r3*dec; 53 | t1= toc; 54 | display(['ELBO = ', num2str(median(ELBO(:,iter))), ', Iter: ' num2str(iter), '/' num2str(MaxIter),' Elapsed: ', num2str(t1) ' sec']); 55 | end 56 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/BPcdf_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial CDF basis function 2 | % Regularized incomplete beta function 3 | 4 | % u: evaluating point, could be a p by 1 vector 5 | % k: degree of BPs 6 | % opt: BP/exBP, standard or extended BP 7 | 8 | % V: a p by d matrix storing basis funcitons 9 | 10 | % Shaobo Han 11 | % 08/07/2015 12 | 13 | function V = BPcdf_basis(u, k, opt) 14 | if nargin<3, 15 | opt = 'BP'; % Standard BP by default 16 | end 17 | p = length(u); % # of variables 18 | switch opt 19 | case 'BP' % Standard BP 20 | d = k; % # of basis functions 21 | a_seq = 1:k; b_seq = k-a_seq+1; 22 | case 'exBP' % extended BP 23 | d = k*(k+1)/2; % # of basis functions 24 | a_seq = zeros(1, d); b_seq = zeros(1, d); 25 | tmp_idx = cumsum(1:k); 26 | for j = 1:k 27 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 28 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 29 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 30 | end 31 | end 32 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 33 | U = repmat(u, 1, d); % p by d matrix 34 | V = betacdf(U , A_seq, B_seq); % p by d matrix 35 | end 36 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/BPpdf_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial PDF basis function 2 | % Beta densities 3 | 4 | % u: evaluating point, could be a p by 1 vector 5 | % k: degree of BPs 6 | % opt: BP/exBP, standard or extended BP 7 | 8 | % V: a p by d matrix storing basis funcitons 9 | 10 | % Shaobo Han 11 | % 08/08/2015 12 | 13 | function V = BPpdf_basis(u, k, opt) 14 | if nargin<3, 15 | opt = 'BP'; % Standard BP by default 16 | end 17 | p = length(u); % # of variables 18 | switch opt 19 | case 'BP' % Standard BP 20 | d = k; % # of basis functions 21 | a_seq = 1:k; b_seq = k-a_seq+1; 22 | case 'exBP' % extended BP 23 | d = k*(k+1)/2; % # of basis functions 24 | a_seq = zeros(1, d); b_seq = zeros(1, d); 25 | tmp_idx = cumsum(1:k); 26 | for j = 1:k 27 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 28 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 29 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 30 | end 31 | end 32 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 33 | U = repmat(u, 1, d); % p by d matrix 34 | V = betapdf(U , A_seq, B_seq); % p by d matrix 35 | end 36 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/BPprime_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial prime basis function 2 | % First-order of Derivative Beta densities 3 | 4 | % u: evaluating point, could be a p by 1 vector 5 | % k: degree of BPs 6 | % opt: BP/exBP, standard or extended BP 7 | 8 | % V: a P by D matrix storing basis funcitons 9 | 10 | % Shaobo Han 11 | % 08/10/2015 12 | 13 | function V = BPprime_basis(u, k, opt) 14 | if nargin<3, 15 | opt = 'BP'; % Standard BP by default 16 | end 17 | p = length(u); % # of variables 18 | switch opt 19 | case 'BP' % Standard BP 20 | d = k; % # of basis functions 21 | a_seq = 1:k; b_seq = k-a_seq+1; 22 | case 'exBP' % extended BP 23 | d = k*(k+1)/2; % # of basis functions 24 | a_seq = zeros(1, d); b_seq = zeros(1, d); 25 | tmp_idx = cumsum(1:k); 26 | for j = 1:k 27 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 28 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 29 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 30 | end 31 | end 32 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 33 | U = repmat(u, 1, d); % p by d matrix 34 | V = (A_seq+B_seq-1).*(betapdf(U , A_seq-1, B_seq)-betapdf(U , A_seq, B_seq-1)); % p by d matrix 35 | % Remove NaN's 36 | % Defining betapdf(u, a, 0) = betapdf(u,0,b) = 0; 37 | Idx = (A_seq == 1)|(B_seq == 1); 38 | V(Idx) = 0; 39 | end 40 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/Cal_mcELBO.m: -------------------------------------------------------------------------------- 1 | function sELBO = Cal_mcELBO(trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par) 2 | tmpMu=par.Mu; tmpC = par.C; tmpW=par.W; 3 | N_mc = opt.N_mc; PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; k = opt.k; 4 | P = length(tmpMu); sELBOM = zeros(1, N_mc); 5 | v_t = sqrt(diag(tmpC*tmpC')); 6 | 7 | for i = 1: N_mc 8 | % Generate a P by 1 multivariate Normal vector 9 | w_t = randn(P,1); Z_t = tmpMu + tmpC*w_t; 10 | if opt.adaptivePhi == 1 11 | Z = diag(1./v_t)*(Z_t-tmpMu); 12 | u = Phi_f(Z, PhiPar, PhiType); 13 | sphi = diag(1./v_t)*sphi_f(Z, PhiPar, PhiType); 14 | else 15 | u = Phi_f(Z_t, PhiPar, PhiType); 16 | sphi = sphi_f(Z_t, PhiPar, PhiType); 17 | end 18 | BPcdf_basiseval = BPcdf_basis(u, k, BPtype); 19 | tmpB = sum(BPcdf_basiseval.*tmpW, 2); 20 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 21 | logp = logmodel(tmpx, trueModel, fix); 22 | BPpdf_basiseval = BPpdf_basis(u, k, BPtype); 23 | tmpb = sum(BPpdf_basiseval.*tmpW, 2); 24 | spsi = spsi_f(tmpx, PsiPar, PsiType); 25 | hdz1 = diag(sphi./spsi)*tmpb; 26 | sELBOM(i) = logp + sum(log(hdz1))+ P/2*log(2*pi)+P/2 + sum(log(diag(tmpC))); 27 | end 28 | 29 | % if opt.sELBO ==1 30 | sELBO = median(sELBOM); 31 | % % val = (P/2)*log(2*pi)+logdet(tmpC)+sELBO; % negative ELBO 32 | % else 33 | % sELBO+ P/2*log(2*pi)+P/2 + sum(log(diag(C))); 34 | % end 35 | 36 | end 37 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/Grad_MuCW.m: -------------------------------------------------------------------------------- 1 | function [Delta_Mu, Delta_C, Delta_w, WinSet] = Grad_MuCW(VGmethod, trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par, WinSet) 2 | NumberZ = opt.NumberZ; P = fix.P; D = opt.D; 3 | PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; k = opt.k; 4 | Mu_old = par.Mu; C_old = par.C; W_old = par.W; 5 | MuMC = zeros(P, NumberZ); 6 | CMC = zeros(P,P, NumberZ); 7 | v_t = sqrt(diag(C_old*C_old')); 8 | WMC = zeros(P, D, NumberZ); 9 | for jj = 1:NumberZ 10 | %% Draw Sample w_t and Z_t 11 | Flag_outlier = 1; 12 | while Flag_outlier == 1 13 | % Generate a P by 1 multivariate Normal vector 14 | w_t = randn(P,1); Z_t = Mu_old + C_old*w_t; 15 | if opt.adaptivePhi == 1 16 | Z = diag(1./v_t)*(Z_t-Mu_old); 17 | u = Phi_f(Z, PhiPar, PhiType); 18 | else 19 | u = Phi_f(Z_t, PhiPar, PhiType); 20 | end 21 | BPcdf_basiseval = BPcdf_basis(u, k, BPtype); % Step 3 22 | tmpB = sum(BPcdf_basiseval.*W_old, 2); % Step 5 23 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); % Step 8 24 | logp = logmodel(tmpx, trueModel, fix); 25 | % Online outlier detection 26 | score = mzscore_test(logp, WinSet); 27 | if abs(score) < opt.OutlierTol 28 | Flag_outlier = 0; WinSet_old = WinSet; 29 | WinSet = [WinSet_old, logp]; 30 | WinSet(1) = []; 31 | end 32 | end 33 | if opt.adaptivePhi == 1 34 | sphi = diag(1./v_t)*sphi_f(Z, PhiPar, PhiType); 35 | dphi = -diag(1./v_t)*(Z.*sphi); 36 | else 37 | sphi = sphi_f(Z_t, PhiPar, PhiType); 38 | dphi = dphi_f(Z_t, PhiPar, PhiType); 39 | end 40 | BPpdf_basiseval = BPpdf_basis(u, k, BPtype); 41 | tmpb = sum(BPpdf_basiseval.*W_old, 2); 42 | BPprime_basiseval = BPprime_basis(u, k, BPtype); 43 | tmpbprime = sum(BPprime_basiseval.*W_old, 2); 44 | rho1prime = sphi.*tmpbprime; 45 | spsi = spsi_f(tmpx, PsiPar, PsiType); 46 | dpsi = dpsi_f(tmpx,PsiPar, PsiType); 47 | g = derivatives(tmpx, trueModel, fix); 48 | 49 | hdz1 = diag(sphi./spsi)*tmpb; 50 | rho3prime = hdz1.*dpsi; 51 | hdz2 = (rho1prime.*sphi.*spsi... 52 | +tmpb.*dphi.*spsi... 53 | -tmpb.*sphi.*rho3prime)./(spsi.^2); 54 | lnh1dz = hdz2./hdz1; 55 | ls_z = g.*hdz1+lnh1dz; 56 | switch VGmethod 57 | case 'Analytic' 58 | dmu = ls_z; 59 | dC = tril(ls_z*w_t')+ diag(1./diag(C_old)); 60 | case 'Numeric' 61 | tmppar.Mu = Mu_old; tmppar.Sigma = C_old*C_old'; 62 | g2 = logqpost_derivatives(Z_t, inferModel, tmppar); 63 | dmu = ls_z-g2; 64 | dC = tril(dmu*w_t'); 65 | end 66 | hdw = diag(1./spsi)*BPcdf_basiseval; 67 | lnh1dw = diag(1./tmpb)*BPpdf_basiseval - diag(dpsi./spsi.^2)*BPcdf_basiseval; 68 | MuMC(:, jj) = dmu; 69 | CMC(:, :, jj) = dC; 70 | WMC(:,:,jj) = diag(g)*hdw+lnh1dw; 71 | end 72 | Delta_Mu = mean(MuMC, 2); 73 | Delta_C= mean(CMC, 3); 74 | Delta_w = median(WMC, 3); 75 | end 76 | 77 | 78 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/Horseshoe_Gibbs.m: -------------------------------------------------------------------------------- 1 | function Y_mc=Horseshoe_Gibbs(x, nsample) 2 | tic; 3 | Y_mc = zeros(nsample,2); 4 | for iter = 1:nsample 5 | if iter==1; ga_mc = 1; end 6 | tau_mc = 1/gamrnd(1,1/(x^2/2+ga_mc)); 7 | ga_mc = gamrnd(1,1/(1/tau_mc+1)); 8 | Y_mc(iter,:) = [tau_mc; ga_mc]; 9 | end 10 | t2=toc; 11 | display(['[Gibbs Sampler] Num of Iters: ' num2str(nsample),' Elapsed Time: ', num2str(t2) ' sec']); 12 | tempC=corrcoef(log(Y_mc)); 13 | display(['[Gibbs Sampler] Correlation Coefs: C11 = ', num2str(tempC(1,1)), ... 14 | ', C21 = ', num2str(tempC(2,1)), ', C22 = ', num2str(tempC(2,2))]) 15 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/Horseshoe_Mfvb.m: -------------------------------------------------------------------------------- 1 | function [Y_vb,ELBO]=Horseshoe_Mfvb(x,nsample, iterMax,tol) 2 | tic; 3 | c0 = -0.5*log(2*pi)-2*log(gamma(0.5)); 4 | iter = 1; ga_exp = 1; 5 | tau_a = 1; ga_a = 1; Flag=1; 6 | ELBO = zeros(1, iterMax); 7 | while (iter<=iterMax) && (Flag==1) 8 | tau_b = x^2/2+ga_exp; 9 | invtau_exp = tau_a/tau_b; 10 | ga_b = invtau_exp+1; 11 | ga_exp = ga_a/ga_b; 12 | H1 = tau_a+log(tau_b)+log(gamma(tau_a))-(1+tau_a)*psi(tau_a); 13 | H2 = ga_a-log(ga_b)+log(gamma(ga_a))+(1-ga_a)*psi(ga_a); 14 | ELBO(iter) = c0-2*(log(tau_b)-psi(tau_a))-x^2/2*tau_a/tau_b-1+H1+H2; 15 | if iter>1 16 | if norm(ELBO(iter)-ELBO(iter-1),'fro')/norm(ELBO(iter),'fro')Xtmp,2)))),0); 8 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/demo_Horseshoe.m: -------------------------------------------------------------------------------- 1 | % Horseshoe shrinakge example for the paper 2 | % "Variational Gaussian Copula Inference", 3 | % Shaobo Han, Xuejun Liao, David. B. Dunson, and Lawrence Carin, 4 | % The 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016) 5 | % ----------------------------- 6 | % Code written by 7 | % Shaobo Han, Duke University 8 | % shaobohan@gmail.com 9 | % 09/29/2015 10 | 11 | % Methods 12 | % 1 Ground Truth (Gibbs Sampler) 13 | % 2 Mean-field VB 14 | % 3 Deterministic VGC with Log-normal Margins 15 | % 4 Deterministic VGC with Log-normal Margins and Diagonal Covariance 16 | % 5 Stochastic VGC with Bernstein Polynomials 17 | 18 | % We used minFunc package for in deterministic implementation of VGC-LN, 19 | % M. Schmidt. minFunc: unconstrained differentiable multivariate 20 | % optimization in Matlab. 21 | % http://www.cs.ubc.ca/~schmidtm/Software/minFunc.html, 2005. 22 | % To run this code, make sure minFunc is installed 23 | 24 | clear all; close all; clc; 25 | 26 | addpath(genpath(pwd)) % Add all sub-directories to the Matlab path 27 | 28 | fix. P = 2; % Number of latent variables 29 | 30 | trueModel = 'Horseshoe'; fix.y = 0.01; 31 | inferModel = 'MVN'; % Multivariate Gaussian 32 | 33 | fix.c = 1; % unnormalizing constant 34 | %% 35 | opt.k = 10; % Degree/Maximum Degree of Bernstein Polynomials 36 | opt.MaxIter = 50; % Number of SGD stages 37 | opt.NumberZ = 1; % Average Gradients 38 | opt.InnerIter = 500; % Number of iteration 39 | opt.N_mc = 1; % Number of median average in sELBO stepsize search 40 | opt.nsample = 1e6; 41 | figplot.nbins = 60; 42 | BPtype = 'BP'; % Bernstein Polynomials 43 | % BPtype = 'exBP'; % Extended Bernstein Polynomials 44 | % PsiType = 'Normal'; opt.PsiPar(1) = 0; opt.PsiPar(2) = 1; % variance 45 | PsiType = 'Exp'; opt.PsiPar = 0.1; 46 | PhiType = 'Normal'; opt.PhiPar(1) = 0; opt.PhiPar(2) =1; % variance 47 | 48 | VGmethod = 'Numeric'; 49 | 50 | % Learning rate 51 | opt.LearnRate.Mu = 1e-3; opt.LearnRate.C =1e-3; 52 | opt.LearnRate.W = 0.5*1e-3; opt.LearnRate.dec = 0.95; % decreasing base learning rate 53 | 54 | switch BPtype 55 | case 'BP' 56 | opt.D = opt.k; 57 | case 'exBP' 58 | opt.D = opt.k*(opt.k+1)/2; % # of basis functions 59 | end 60 | 61 | % Diagonal constraint on Upsilon 62 | opt.diagUpsilon = 0; 63 | opt.adaptivePhi = 0; 64 | opt.normalize = 1; 65 | 66 | %% Initialization 67 | ini.Mu = opt.PhiPar(1).*ones(fix.P,1); 68 | % ini.C = sqrt(opt.PhiPar(2)).*eye(fix.P); 69 | ini.C = randchol(fix.P); 70 | % ini.w = randBPw(fix.P, opt.D, 1, 1); 71 | ini.w = 1./opt.D.*ones(fix.P, opt.D); 72 | % Median Outlier Removal 73 | % Use modified Z score to determine and remove outliers 74 | opt.OutlierTol = 100; % Threshold for online outlier detection 75 | opt.WinSize = 20; % Size of the window 76 | ini.WinSet = ini_WinSet(trueModel, PhiType, PsiType, BPtype, fix, opt, ini); 77 | opt.Wthreshold = 1e12; 78 | 79 | %% 1. Gibbs Sampler 80 | Y_mc = Horseshoe_Gibbs(fix.y, opt.nsample); 81 | MCMC = hist1D2D(log(Y_mc), figplot.nbins); 82 | %% 2. MFVB 83 | iterMax = 500; tol = 1e-6; 84 | [Y_vb,ELBO_vb]=Horseshoe_Mfvb(fix.y, opt.nsample, iterMax, tol); 85 | MFVB = hist1D2D(log(Y_vb), figplot.nbins); 86 | 87 | %% 3. VC(D): Log-normal 88 | % -------------------full covariance (+C)------------------- 89 | nsample = 30000; nbins =50; 90 | post.m = ini.Mu; 91 | post.c = ini.C; 92 | c1 = 1+0.5*log(2*pi)-2*log(gamma(0.5)); 93 | c0 = -0.5*log(2*pi)-2*log(gamma(0.5)); 94 | 95 | fix.c1 = c1; fix.x = fix.y; fix.c0 = c0; 96 | fix.post.dim = fix.P; 97 | 98 | options = []; 99 | % options.display = 'full'; 100 | options.display = 'none'; 101 | options.maxFunEvals = 2000; 102 | options.Method = 'qnewton'; 103 | fix.Flagdiag = 0; % full covariance 104 | [Y_vcdlnc,ELBO_vcdlnc,postoptimized] = Horseshoe_VCD_LN(post, options,nsample, fix); 105 | VCDLNC = hist1D2D(log(Y_vcdlnc), nbins); 106 | %% 4. VC(D): Log-normal + diagonal 107 | % -------------------diag covariance (+I)------------------- 108 | fix.Flagdiag = 1; % diagonal covariance 109 | [Y_vcdlni,ELBO_vcdlni] = Horseshoe_VCD_LN(post, options,nsample, fix); 110 | VCDLNI = hist1D2D(log(Y_vcdlni), nbins); 111 | 112 | %% 5.(S)VGC-BP 113 | fix.scale = 1; % The scale of covariance 114 | inferModel2 = 'MVN'; % Multivariate Normal 115 | % ini.Mu = VG.mu; 116 | % ini.C = VG.C; 117 | opt.updateMuC=1; opt.updateW=1; 118 | [ELBO1, par] = vgcbp_MuCW(trueModel, inferModel2, PhiType, PsiType, BPtype,VGmethod, fix, ini, opt); 119 | figplot.Y_svc2 = sampleGC(PhiType, PsiType, BPtype, opt, par); 120 | figplot.Y_svc2(sum(~isfinite(log(figplot.Y_svc2)),2)~=0,:) = []; 121 | VGCBPall = hist1D2D(log(figplot.Y_svc2), figplot.nbins); 122 | 123 | %% Plot 124 | contourvector = [0.1, 0.25,0.5, 0.75, 0.9]; 125 | T1 =MCMC.count'; F1 =T1./sum(sum(T1)); plevs1=contourvector.*max(F1(:)); 126 | T2 =MFVB.count'; F2 =T2./sum(sum(T2)); plevs2=contourvector.*max(F2(:)); 127 | T3 =VCDLNC.count'; F3 =T3./sum(sum(T3)); plevs3=contourvector.*max(F3(:)); 128 | T4 =VCDLNI.count'; F4 =T4./sum(sum(T4)); plevs4=contourvector.*max(F4(:)); 129 | T5 =VGCBPall.count'; F5 =T5./sum(sum(T5)); plevs5=contourvector.*max(F5(:)); 130 | 131 | figure(1) 132 | fs = 16; ncontour = 4; ls = 1.5; ms = 2.5; 133 | main=subplot(4,4,[5,6,7,9,10,11,13,14,15]); 134 | set(gca, 'fontsize',fs) 135 | contour(MCMC.Seqy,MCMC.Seqx,MCMC.count'./sum(sum(MCMC.count)),ncontour, 'Color', 'k', 'LineWidth', 1.2*ls); 136 | hold on 137 | contour(MFVB.Seqy,MFVB.Seqx,MFVB.count'./sum(sum(MFVB.count)),ncontour, '--', 'Color', 'r', 'LineWidth', 2*ls); 138 | hold on 139 | contour(VCDLNC.Seqy,VCDLNC.Seqx,VCDLNC.count'./sum(sum(VCDLNC.count)), ncontour, 'Color', 'm', 'LineWidth', 2*ls); 140 | hold on 141 | contour(VCDLNI.Seqy,VCDLNI.Seqx,VCDLNI.count'./sum(sum(VCDLNI.count)),ncontour, 'Color', 'c', 'LineWidth', 2*ls); 142 | hold on 143 | contour(VGCBPall.Seqy,VGCBPall.Seqx,VGCBPall.count'./sum(sum(VGCBPall.count)),ncontour, 'Color', 'b', 'LineWidth', 1.5*ls); 144 | hold off 145 | grid on 146 | grid minor 147 | axis([-15,5,-15,5]) 148 | legend('Gibbs Sampler', 'MFVB', 'VGCLN-full', 'VGCLN-diag', 'VGC-BP-full') 149 | 150 | ylabel('log(\gamma)', 'fontsize',fs) 151 | xlabel('log(\tau)', 'fontsize',fs) 152 | pion=subplot(4,4,[8,12,16]); %right plot (rotated) 153 | set(gca, 'fontsize',fs) 154 | plot(MCMC.x_x1,MCMC.f_x1/trapz(MCMC.x_x1,MCMC.f_x1), 'Color', 'k','linewidth', 1.2*ls, 'Markersize', 4*ms); 155 | hold on 156 | plot(MFVB.x_x1,MFVB.f_x1/trapz(MFVB.x_x1,MFVB.f_x1),'--', 'Color', 'r','linewidth', 2.5*ls, 'Markersize', 4*ms); 157 | hold on 158 | plot(VCDLNC.x_x1,VCDLNC.f_x1/trapz(VCDLNC.x_x1,VCDLNC.f_x1), '-o', 'Color', 'm','linewidth', 1.2*ls, 'Markersize', 2*ms); 159 | hold on 160 | plot(VCDLNI.x_x1,VCDLNI.f_x1/trapz(VCDLNI.x_x1,VCDLNI.f_x1), '-s', 'Color', 'c','linewidth', 1*ls, 'Markersize', 1.2*ms); 161 | hold on 162 | plot(VGCBPall.x_x1,VGCBPall.f_x1/trapz(VGCBPall.x_x1,VGCBPall.f_x1), '-x', 'Color', 'b','linewidth', 1*ls, 'Markersize', 4*ms); 163 | hold off 164 | grid on 165 | grid minor 166 | axis([-15,5,... 167 | 0, 1.05.*max([MCMC.f_x1/trapz(MCMC.x_x1,MCMC.f_x1),... 168 | MFVB.f_x1/trapz(MFVB.x_x1,MFVB.f_x1),... 169 | VCDLNC.f_x1/trapz(VCDLNC.x_x1,VCDLNC.f_x1),... 170 | VCDLNI.f_x1/trapz(VCDLNI.x_x1,VCDLNI.f_x1),])]) 171 | legend('Gibbs Sampler', 'MFVB', 'VGC-LN-full', 'VGC-LN-diag', 'VGC-BP-full') 172 | view(90, 270) 173 | 174 | poz=subplot(4,4,1:3); %upper plot 175 | set(gca, 'fontsize',fs) 176 | plot(MCMC.x_x2,MCMC.f_x2/trapz(MCMC.x_x2,MCMC.f_x2), 'Color', 'k', 'linewidth', 1.2*ls, 'Markersize', 4*ms); 177 | hold on 178 | plot(MFVB.x_x2,MFVB.f_x2/trapz(MFVB.x_x2,MFVB.f_x2), '--','Color', 'r', 'linewidth', 2.5*ls, 'Markersize', 4*ms); 179 | hold on 180 | plot(VCDLNC.x_x2,VCDLNC.f_x2/trapz(VCDLNC.x_x2,VCDLNC.f_x2),'-o', 'Color', 'm', 'linewidth', 1.2*ls, 'Markersize', 2*ms); 181 | hold on 182 | plot(VCDLNI.x_x2,VCDLNI.f_x2/trapz(VCDLNI.x_x2,VCDLNI.f_x2), '-s', 'Color', 'c','linewidth', 1*ls, 'Markersize', 1.2*ms); 183 | hold on 184 | plot(VGCBPall.x_x2,VGCBPall.f_x2/trapz(VGCBPall.x_x2,VGCBPall.f_x2), '-x', 'Color', 'b', 'linewidth', 1*ls, 'Markersize', 4*ms); 185 | hold off 186 | grid on 187 | grid minor 188 | axis([-15,5,... 189 | 0, 1.05.*max([MCMC.f_x2/trapz(MCMC.x_x2,MCMC.f_x2),... 190 | MFVB.f_x2/trapz(MFVB.x_x2,MFVB.f_x2),... 191 | VCDLNC.f_x2/trapz(VCDLNC.x_x2,VCDLNC.f_x2),... 192 | VCDLNI.f_x2/trapz(VCDLNI.x_x2,VCDLNI.f_x2) ])]) 193 | pos1=get(poz,'Position'); pos2=get(main,'Position'); pos3=get(pion,'Position'); 194 | pos1(3) = pos2(3); %width for the upper plot 195 | set(poz,'Position',pos1) 196 | pos3(4) = pos2(4); %height for the right plot 197 | set(pion,'Position',pos3) 198 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/derivatives.m: -------------------------------------------------------------------------------- 1 | % Derivatives of Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function g = derivatives(x, model, fix) 7 | switch model 8 | case 'Horseshoe' 9 | y = fix.y; 10 | g = zeros(2,1); 11 | g(1) = -2/x(1)+y^2/2/(x(1)^2)+x(2)/(x(1)^2); 12 | g(2) = -1/x(1)-1; 13 | case 'SkewNormal' 14 | m = fix.snMu; 15 | Lambda = fix.snSigma; 16 | alpha = fix.snAlpha; 17 | tmp = alpha'*(x-m); 18 | g = -(Lambda\(x-m))+normpdf(tmp).*alpha./normcdf(tmp); 19 | case 'LogNormal2' 20 | sigmaY1 = fix.LNsigmaY1; sigmaY2 = fix.LNsigmaY2; 21 | muY1 = fix.LNmuY1; muY2 = fix.LNmuY2; LNrho = fix.LNrho; 22 | alpha1 = (log(x(1))-muY1)./sigmaY1; 23 | alpha2= (log(x(2))-muY2)./sigmaY2; 24 | g = zeros(2,1); 25 | g(1)=-1./x(1)-(alpha1-LNrho*alpha2)/(1-LNrho.^2)./x(1)./sigmaY1; 26 | g(2)=-1./x(2)-(alpha2-LNrho*alpha1)/(1-LNrho.^2)./x(2)./sigmaY2; 27 | case 'Gamma' 28 | alpha = fix.alpha; 29 | beta = fix.beta; 30 | g = (alpha-1)./x-beta; 31 | case 'Student' 32 | nu = fix.nu; 33 | g = -(nu+1)*x/(nu+x^2); 34 | case 'Beta'; 35 | alpha = fix.alpha; beta = fix.beta; 36 | g = (alpha-1)/x-(beta-1)/(1-x); 37 | end 38 | end 39 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/dphi_f.m: -------------------------------------------------------------------------------- 1 | % dphi function: derivative of sphi 2 | % Shaobo Han 3 | % 09/24/2015 4 | function output = dphi_f(z, par, opt) 5 | switch opt 6 | case 'Normal' 7 | m0 = par(1); v0 = par(2); 8 | output = -(z./v0).*normpdf(z, m0, sqrt(v0)); 9 | end 10 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/dpsi_f.m: -------------------------------------------------------------------------------- 1 | % dpsi function: derivative of small psi 2 | % Shaobo Han 3 | % 08/10/2015 4 | 5 | function output = dpsi_f(x, par, opt) 6 | if nargin<3, 7 | opt = 'Exp'; % exponential distribution by default 8 | end 9 | if nargin<2, 10 | par = 0.1; % lambda = 0.1 by default 11 | end 12 | switch opt 13 | case 'Exp' % exp(lambda) 14 | output = -par.*exppdf(x,1/par); 15 | case 'Normal' % 16 | m0 = par(1); % mean 17 | v0 = par(2); % variance 18 | output = (-x./v0).*normpdf(x,m0,sqrt(v0)); 19 | case 'Beta' 20 | a0 = par(1); b0 = par(2); 21 | output =(a0+b0-1)*(betapdf(x, a0-1, b0)- betapdf(x, a0, b0-1)); 22 | end 23 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/hist1D2D.m: -------------------------------------------------------------------------------- 1 | function Output=hist1D2D(Matrix, nbins) 2 | % 1D and 2D histogram 3 | % Matrix N by 2 matrix 4 | [f_x1,x_x1] = hist(Matrix(:,1), nbins); 5 | [f_x2,x_x2] = hist(Matrix(:,2), nbins); 6 | [count,Dy] = hist3(Matrix,[nbins+2,nbins]); 7 | Seqx= Dy{1,1}; Seqy = Dy{1,2}; 8 | Output.f_x1 = f_x1; Output.x_x1 = x_x1; 9 | Output.f_x2 = f_x2; Output.x_x2 = x_x2; 10 | Output.count =count'; Output.Seqx =Seqx; Output.Seqy =Seqy; 11 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/iPsi_f.m: -------------------------------------------------------------------------------- 1 | % Invserse Psi function: 2 | % Shaobo Han 3 | % 08/10/2015 4 | 5 | function output = iPsi_f(p, par, opt) 6 | if nargin<3, 7 | opt = 'Exp'; % exponential distribution by default 8 | end 9 | if nargin<2, 10 | par = 0.1; % lambda = 0.1 by default 11 | end 12 | p(p>1-1e-16) = 1-1e-16; % numerical stability 13 | switch opt 14 | case 'Exp' % exp(lambda) 15 | output = expinv(p,1/par); 16 | case 'Normal' % 17 | m0 = par(1); v0 = par(2); 18 | output = norminv(p,m0,sqrt(v0)); 19 | case 'Beta' 20 | a0 = par(1); b0 = par(2); 21 | output = betainv(p, a0, b0); 22 | end 23 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/ini_WinSet.m: -------------------------------------------------------------------------------- 1 | function WinSet = ini_WinSet(trueModel, PhiType, PsiType, BPtype, fix, opt, ini) 2 | k = opt.k; PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; 3 | P = fix.P; Mu_old = ini.Mu; C_old = ini.C; W_old = ini.w; 4 | WinSet = zeros(1, opt.WinSize); 5 | vt = sqrt(diag(C_old*C_old')); 6 | for i = 1:opt.WinSize 7 | w_t0 = randn(P,1); Z_t0 = Mu_old + C_old*w_t0; 8 | if opt.adaptivePhi == 1 9 | Z0 = diag(1./vt)*(Z_t0-Mu_old); 10 | u0 = Phi_f(Z0, PhiPar, PhiType); 11 | else 12 | u0 = Phi_f(Z_t0, PhiPar, PhiType); 13 | end 14 | BPcdf_basiseval0 = BPcdf_basis(u0, k, BPtype); 15 | tmpB0 = sum(BPcdf_basiseval0.*W_old, 2); 16 | tmpx0 = iPsi_f(tmpB0, PsiPar, PsiType); 17 | WinSet(i) = logmodel(tmpx0, trueModel, fix); 18 | end 19 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/logdet.m: -------------------------------------------------------------------------------- 1 | function y = logdet(A) 2 | %LOGDET Logarithm of determinant for positive-definite matrix 3 | % logdet(A) returns log(det(A)) where A is positive-definite. 4 | % This is faster and more stable than using log(det(A)). 5 | % Note that logdet does not check if A is positive-definite. 6 | % If A is not positive-definite, the result will not be the same as log(det(A)). 7 | 8 | % Written by Tom Minka 9 | % (c) Microsoft Corporation. All rights reserved. 10 | 11 | U = chol(A); 12 | y = 2*sum(log(diag(U))); -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/logmodel.m: -------------------------------------------------------------------------------- 1 | % Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function p = logmodel(x, model, fix) 7 | switch model 8 | case 'Horseshoe' 9 | y = fix.y; 10 | c0 = -0.5*log(2*pi)-2*log(gamma(0.5)); 11 | p = c0-2*log(x(1))-y.^2./2./x(1)-x(2)./x(1)-x(2); 12 | case 'LogNormal2' 13 | sigmaY1 = fix.LNsigmaY1; sigmaY2 = fix.LNsigmaY2; 14 | muY1 = fix.LNmuY1; muY2 = fix.LNmuY2; LNrho = fix.LNrho; 15 | c0 = -log(2*pi)-log(sigmaY1*sigmaY2*sqrt(1-LNrho^2)); 16 | alpha1 = (log(x(1))-muY1)/sigmaY1; 17 | alpha2= (log(x(2))-muY2)/sigmaY2; 18 | q = 1/(1-LNrho^2)*(alpha1^2-2*LNrho*alpha1*alpha2+alpha2^2); 19 | p = c0-log(x(1))-log(x(2))-q/2; 20 | case 'LogNormal' 21 | sigmaY = fix.LNsigmaY; 22 | muY = fix.LNmuY; 23 | c0 = -0.5*log(2*pi)-log(sigmaY); 24 | alpha = (log(x)-muY)./sigmaY; 25 | p = c0-log(x)- alpha.^2/2; 26 | case 'SkewNormal' 27 | m = fix.snMu; 28 | Lambda = fix.snSigma; 29 | alpha = fix.snAlpha; 30 | c = fix.c; 31 | p = log(2)+log(mvnpdf(x,m,Lambda))+log(normcdf(alpha'*(x-m)))+log(c); 32 | case 'Gamma' 33 | alpha = fix.alpha; 34 | beta = fix.beta; 35 | c = fix.c; 36 | p = log(gampdf(x, alpha, 1/beta))+log(c); 37 | case 'Student' 38 | nu = fix.nu; c = fix.c; 39 | p = log(tpdf(x,nu))+log(c); 40 | case 'Exp' 41 | lambda = fix.lambda; c = fix.c; 42 | p = log(exppdf(x,1/lambda))+log(c); 43 | case 'Beta'; 44 | alpha = fix.alpha; beta = fix.beta; c = fix.c; 45 | p = log(betapdf(x, alpha, beta))+log(c); 46 | end 47 | end 48 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/logqpost.m: -------------------------------------------------------------------------------- 1 | % Log q posteiror 2 | % Model dependent term 3 | % Shaobo Han 4 | % 09/11/2015 5 | 6 | function p = logqpost(x, model, par) 7 | switch model 8 | case 'Horseshoe' 9 | y = par.y; 10 | c0 = -0.5*log(2*pi)-2*log(gamma(0.5)); 11 | p = c0-2*log(x(1))-y.^2./2./x(1)-x(2)./x(1)-x(2); 12 | case 'SkewNormal' 13 | m = par.snMu; 14 | Lambda = par.snSigma; 15 | alpha = par.snAlpha; 16 | p = log(2)+log(mvnpdf(x,m,Lambda))+log(normcdf(alpha'*(x-m))) ; 17 | case 'MVN' 18 | mu = par.Mu; C= par.C; Sigma=C*C'; 19 | d = length(mu); 20 | p = -d*log(2*pi)/2-0.5*logdet(Sigma)-0.5*(x-mu)'*(Sigma\(x-mu)); 21 | case 'MVNdiag' 22 | mu = par.Mu; C= par.C; Sigma=C*C'; 23 | p = sum(log(normpdf(x, mu, sqrt(diag(Sigma))))); 24 | end 25 | end 26 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/logqpost_derivatives.m: -------------------------------------------------------------------------------- 1 | % Derivatives of Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function g = logqpost_derivatives(x, model, par) 7 | switch model 8 | case 'Horseshoe' 9 | y = par.y; 10 | g = zeros(2,1); 11 | g(1) = -2/x(1)+y^2/2/(x(1)^2)+x(2)/(x(1)^2); 12 | g(2) = -1/x(1)-1; 13 | case 'SkewNormal' 14 | m = par.snMu; 15 | Lambda = par.snSigma; 16 | alpha = par.snAlpha; 17 | g = -inv(Lambda)*(x-m)+mvnpdf(alpha'*(x-m)).*alpha./mvncdf(alpha'*(x-m)); 18 | case 'MVN' 19 | mu = par.Mu; Sigma= par.Sigma; 20 | g = Sigma\(mu-x); 21 | case 'MVNdiag' 22 | mu = par.Mu; Sigma= par.Sigma; 23 | g = diag(1./diag(Sigma))*(mu-x); 24 | end 25 | end 26 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/mzscore_test.m: -------------------------------------------------------------------------------- 1 | % Calculate the modified Z-score of a query point from a window set 2 | % Shaobo Han 3 | % 08/13/2015 4 | function score = mzscore_test(x, windowset) 5 | tmpX = [x, windowset]; 6 | tmpm = median(tmpX); 7 | mad = median(abs(tmpX-tmpm)); 8 | score = 0.6745*(x-tmpm)/mad; 9 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/post2vec.m: -------------------------------------------------------------------------------- 1 | function vec=post2vec(post) 2 | % convert posterior stucture to a 1=-d col. vector 3 | vec=[post.m;post.c(:)]; 4 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/randBPw.m: -------------------------------------------------------------------------------- 1 | % Generate a P*D BP weight matrix 2 | % (every column is a probability vector which sums up to one) 3 | % P: number of variables 4 | % D: number of BP basis functions 5 | % a: shape parameter; b: scale parameter 6 | 7 | % Shaobo Han 8 | % 08/08/2015 9 | 10 | function w = randBPw(P, D, a, b) 11 | if nargin<4 12 | a = 1; b = 1; % by default 13 | end 14 | if D==1 15 | w = ones(P,1); 16 | else 17 | w0 = gamrnd(a,b,P,D); 18 | w = diag(1./sum(w0,2))*w0; 19 | end 20 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/randchol.m: -------------------------------------------------------------------------------- 1 | % Generate a P*P lower triangular Cholesky factor matrix 2 | % Shaobo Han 3 | % 08/07/2015 4 | function C = randchol(P) 5 | a = rand(P, P); 6 | aTa = a'*a; 7 | [V, D] = eig(aTa); 8 | aTa_PD = V*(D+1e-5)*V'; 9 | C=chol(aTa_PD, 'lower'); 10 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/sampleGC.m: -------------------------------------------------------------------------------- 1 | function Y_svc = sampleGC(PhiType, PsiType, BPtype, opt, par) 2 | PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; 3 | nsample = opt.nsample; k = opt.k; 4 | Mu = par.Mu; C = par.C; W = par.W; 5 | P = length(Mu); 6 | % Sample from Gaussian Copula 7 | optmu_s2 = Mu; optSig_s2 = C*C'; 8 | wtz2 = mvnrnd(optmu_s2, optSig_s2, nsample); 9 | Y_svc = zeros(nsample,P); 10 | if opt.adaptivePhi == 0 11 | for ns = 1:nsample 12 | tmpz = wtz2(ns,:)'; 13 | tmpu = Phi_f(tmpz, PhiPar, PhiType); 14 | BPcdf_basiseval = BPcdf_basis(tmpu, k, BPtype); 15 | tmpB = sum(BPcdf_basiseval.*W, 2); 16 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 17 | Y_svc(ns,:) = tmpx; 18 | end 19 | else 20 | v_t = sqrt(diag(C*C')); 21 | for ns = 1:nsample 22 | tmpz0 = wtz2(ns,:)'; 23 | tmpz = diag(1./v_t)*(tmpz0-Mu); 24 | tmpu = Phi_f(tmpz, PhiPar, PhiType); 25 | BPcdf_basiseval = BPcdf_basis(tmpu, k, BPtype); 26 | tmpB = sum(BPcdf_basiseval.*W, 2); 27 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 28 | Y_svc(ns,:) = tmpx; 29 | end 30 | end 31 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/sphi_f.m: -------------------------------------------------------------------------------- 1 | % phi function: small phi, derivative of Phi 2 | % Shaobo Han 3 | % 08/07/2015 4 | function output = sphi_f(z, par, opt) 5 | switch opt 6 | case 'Normal' 7 | m0 = par(1); v0 = par(2); 8 | output = normpdf(z, m0, sqrt(v0)); 9 | end 10 | end 11 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/spsi_f.m: -------------------------------------------------------------------------------- 1 | % psi function: small psi, derivative of Psi 2 | % Shaobo Han 3 | % 08/10/2015 4 | 5 | function output = spsi_f(x, par, opt) 6 | if nargin<3, 7 | opt = 'Exp'; % exponential distribution by default 8 | end 9 | if nargin<2, 10 | par = 0.1; % lambda = 0.1 by default 11 | end 12 | switch opt 13 | case 'Exp' % exp(lambda) 14 | output = exppdf(x,1/par); 15 | case 'Normal' % 16 | m0 = par(1); v0 = par(2); 17 | output = normpdf(x, m0, sqrt(v0)); 18 | case 'Beta' 19 | a0 = par(1); b0 = par(2); 20 | output = betapdf(x, a0, b0); 21 | end 22 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/vcval_Horseshoe.m: -------------------------------------------------------------------------------- 1 | function [val, grad] = vcval_Horseshoe(opt,fix) 2 | % Evaluate ELBO and Gradient 3 | optl = length(opt); % get size of opt params 4 | % break if opt vals not real 5 | if ~isreal(opt), val = Inf; grad = Inf(optl,1); return; end 6 | post = vec2post(opt,fix.post.dim); % unpack opt. params. 7 | % unpack fix. parameters 8 | c1 = fix.c1; 9 | x = fix.x; 10 | %% ELBO 11 | % ENTROPY TERM 12 | Eh=log(det(post.c)); 13 | m1 = post.m(1); m2 = post.m(2); 14 | C11 = post.c(1,1); C21 = post.c(2,1); C22 = post.c(2,2); 15 | % break if diag(C) negative 16 | if ~isreal(Eh), val = Inf; grad = Inf(optl,1); return; end 17 | ell1 = exp(-m1+(C11^2)/2); 18 | tempC = C11^2-2*C11*C21+C21^2+C22^2; 19 | ell2 = exp(m2-m1+0.5.*tempC); 20 | ell3 = exp(m2+0.5*(C21^2+C22^2)); 21 | val1 = c1-m1+m2-x^2/2*ell1-ell2-ell3; 22 | val2 = Eh; 23 | val = -(val1+val2); 24 | if ~isreal(val), val = Inf; grad = Inf(optl,1); return; end 25 | %% Gradients 26 | grad_m1 = -1+x^2/2*ell1+ell2; 27 | grad_m2 = 1-ell2-ell3; 28 | tempgrad_c11 = -C11*(x^2)/2*ell1-(C11-C21)*ell2+1/C11; 29 | tempgrad_c21 = (C11-C21)*ell2-C21*ell3; 30 | tempgrad_c22 = -C22*ell2-C22*ell3+1/C22; 31 | gradc = [tempgrad_c11, 0; tempgrad_c21, tempgrad_c22]; 32 | gradm = [grad_m1; grad_m2]; 33 | grad = [-gradm; -gradc(:)]; 34 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/vcval_Horseshoe_diag.m: -------------------------------------------------------------------------------- 1 | function [val, grad] = vcval_NIGG_diag(opt,fix) 2 | % Evaluate ELBO and Gradient 3 | optl = length(opt); % get size of opt params 4 | % break if opt vals not real 5 | if ~isreal(opt), val = Inf; grad = Inf(optl,1); return; end 6 | post = vec2post(opt,fix.post.dim); % unpack opt. params. 7 | % unpack fix. parameters 8 | c1 = fix.c1; 9 | x = fix.x; 10 | 11 | %% ELBO 12 | % ENTROPY TERM 13 | Eh=log(det(post.c)); 14 | m1 = post.m(1); m2 = post.m(2); 15 | C11 = post.c(1,1); C21 = 0; C22 = post.c(2,2); 16 | % break if diag(C) negative 17 | if ~isreal(Eh), val = Inf; grad = Inf(optl,1); return; end 18 | ell1 = exp(-m1+(C11^2)/2); 19 | tempC = C11^2-2*C11*C21+C21^2+C22^2; 20 | ell2 = exp(m2-m1+0.5.*tempC); 21 | ell3 = exp(m2+0.5*(C21^2+C22^2)); 22 | val1 = c1-m1+m2-x^2/2*ell1-ell2-ell3; 23 | val2 = Eh; 24 | val = -(val1+val2); 25 | if ~isreal(val), val = Inf; grad = Inf(optl,1); return; end 26 | %% Gradients 27 | grad_m1 = -1+x^2/2*ell1+ell2; 28 | grad_m2 = 1-ell2-ell3; 29 | tempgrad_c11 = -C11*(x^2)/2*ell1-(C11-C21)*ell2+1/C11; 30 | tempgrad_c21 = 0; 31 | tempgrad_c22 = -C22*ell2-C22*ell3+1/C22; 32 | gradc = [tempgrad_c11, 0; tempgrad_c21, tempgrad_c22]; 33 | gradm = [grad_m1; grad_m2]; 34 | grad = [-gradm; -gradc(:)]; 35 | end -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/vec2post.m: -------------------------------------------------------------------------------- 1 | function post=vec2post(vec,postDim) 2 | % convert vector of posterior params to posterior structure 3 | post.m=vec(1:postDim); 4 | vec=vec(postDim+1:end); 5 | post.c=reshape(vec(:),postDim,postDim); 6 | end 7 | -------------------------------------------------------------------------------- /VGC-HorseshoeShrinkage/vgcbp_MuCW.m: -------------------------------------------------------------------------------- 1 | function [ELBO, par] = vgcbp_MuCW(trueModel, inferModel, PhiType, PsiType, BPtype, VGmethod, fix, ini, opt) 2 | % Parameters Unpack 3 | InnerIter = opt.InnerIter; MaxIter = opt. MaxIter; 4 | r1 = opt.LearnRate.Mu; r2 = opt.LearnRate.C; 5 | r3 = opt.LearnRate.W; dec = opt.LearnRate.dec; 6 | if opt.diagUpsilon ==1 7 | display(['[VGC(S): diag], BPtype = ', BPtype,' PsiType = ', PsiType,' PhiType = ', PhiType]) 8 | else 9 | display(['[VGC(S): full], BPtype = ', BPtype,' PsiType = ', PsiType,' PhiType = ', PhiType]) 10 | end 11 | ELBO = zeros(InnerIter, MaxIter); 12 | RMSE = zeros(InnerIter, MaxIter); 13 | for iter = 1: MaxIter 14 | tic; 15 | for inner = 1:InnerIter 16 | % First Iteration 17 | if (iter == 1)&&(inner==1) 18 | par.Mu_old = ini.Mu; par.C_old = ini.C; par.W_old = ini.w; 19 | WinSet = ini.WinSet; 20 | par.Mu = par.Mu_old; par.C = par.C_old; par.W = par.W_old; 21 | else 22 | par.Mu_old = par.Mu; par.C_old = par.C; par.W_old = par.W; 23 | end 24 | %% Draw Samples and Calculate (Mu, C) 25 | [Delta_Mu, Delta_C, Delta_w, WinSet] = Grad_MuCW(VGmethod, trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par,WinSet); 26 | if opt.normalize ==1 27 | if norm(Delta_Mu,'fro')<=1e-5 28 | Delta_Mu = zeros(P,1); 29 | else 30 | Delta_Mu = Delta_Mu./norm(Delta_Mu,'fro'); 31 | end 32 | if norm(Delta_C,'fro')<=1e-5 33 | Delta_C = zeros(P,P); 34 | else 35 | Delta_C = Delta_C./norm(Delta_C,'fro'); 36 | end 37 | end 38 | if opt.updateMuC~=1 39 | Delta_C = zeros(size(Delta_C)); 40 | Delta_Mu = zeros(size(Delta_Mu)); 41 | end 42 | %% Update Mu 43 | par.Mu = par.Mu_old + r1.*Delta_Mu; 44 | if opt.adaptivePhi == 1 45 | par.Mu = zeros(size( par.Mu)); 46 | end 47 | %% Update Mu 48 | par.C = par.C_old + r2.*Delta_C; 49 | %% Draw Samples and Calculate Delta_W 50 | if opt.normalize ==1 51 | % Normalize 52 | if norm(Delta_w,'fro')<=1e-5 53 | Delta_w = zeros(size(Delta_w)); 54 | else 55 | Delta_w = diag(1./sqrt(sum(abs(Delta_w).^2,2)))*Delta_w; 56 | end 57 | end 58 | %% Update W 59 | tmpNorm = sqrt(sum(abs(Delta_w).^2,2)); 60 | UnstableIndex = (tmpNorm>opt.Wthreshold); 61 | tmpsum = sum(UnstableIndex); 62 | if tmpsum ~=0 63 | Delta_w(UnstableIndex,:) = zeros(tmpsum, opt.D); 64 | display('Warning: Unstable Delta W Omitted!') 65 | tmpNorm(UnstableIndex) 66 | end 67 | W0 = par.W_old + r3.*Delta_w; 68 | % Project Gradient Descent 69 | par.W = SimplxProj(W0); 70 | if sum(abs(sum(par.W,2)-1)>1e-5*ones(size(sum(par.W,2))))>0 71 | display('Error: Weight not Sum up to 1!') 72 | sum(par.W,2) 73 | par.W = par.W_old; 74 | end 75 | ELBO(inner, iter) = Cal_mcELBO(trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par); 76 | if strcmp(trueModel, 'LogNormal2')==1 77 | RMSE(inner, iter) = norm(corrcov(par.C *par.C')- fix.trueUpsilon, 'fro')./norm(fix.trueUpsilon, 'fro'); 78 | end 79 | end 80 | r1 = r1*dec; r2 = r2*dec; r3 = r3*dec; 81 | t1= toc; 82 | display(['ELBO = ', num2str(median(ELBO(:,iter))), ', Iter: ' num2str(iter), '/' num2str(MaxIter),' Elapsed: ', num2str(t1) ' sec']); 83 | end 84 | par.WinSet = WinSet; 85 | par.RMSE = RMSE; 86 | end 87 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/BPcdf_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial CDF basis function 2 | % Regularized incomplete beta function 3 | % u: evaluating point, could be a p by 1 vector 4 | % k: degree of BPs 5 | % opt: BP/exBP, standard or extended BP 6 | 7 | % V: a p by d matrix storing basis funcitons 8 | 9 | % Shaobo Han 10 | % 08/07/2015 11 | 12 | function V = BPcdf_basis(u, k, opt) 13 | if nargin<3, 14 | opt = 'BP'; % Standard BP by default 15 | end 16 | p = length(u); % # of variables 17 | switch opt 18 | case 'BP' % Standard BP 19 | d = k; % # of basis functions 20 | a_seq = 1:k; b_seq = k-a_seq+1; 21 | case 'exBP' % extended BP 22 | d = k*(k+1)/2; % # of basis functions 23 | a_seq = zeros(1, d); b_seq = zeros(1, d); 24 | tmp_idx = cumsum(1:k); 25 | for j = 1:k 26 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 27 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 28 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 29 | end 30 | end 31 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 32 | U = repmat(u, 1, d); % p by d matrix 33 | V = betacdf(U , A_seq, B_seq); % p by d matrix 34 | end 35 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/BPpdf_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial PDF basis function 2 | % Beta densities 3 | % u: evaluating point, could be a p by 1 vector 4 | % k: degree of BPs 5 | % opt: BP/exBP, standard or extended BP 6 | 7 | % V: a p by d matrix storing basis funcitons 8 | 9 | % Shaobo Han 10 | % 08/08/2015 11 | 12 | function V = BPpdf_basis(u, k, opt) 13 | if nargin<3, 14 | opt = 'BP'; % Standard BP by default 15 | end 16 | p = length(u); % # of variables 17 | switch opt 18 | case 'BP' % Standard BP 19 | d = k; % # of basis functions 20 | a_seq = 1:k; b_seq = k-a_seq+1; 21 | case 'exBP' % extended BP 22 | d = k*(k+1)/2; % # of basis functions 23 | a_seq = zeros(1, d); b_seq = zeros(1, d); 24 | tmp_idx = cumsum(1:k); 25 | for j = 1:k 26 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 27 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 28 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 29 | end 30 | end 31 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 32 | U = repmat(u, 1, d); % p by d matrix 33 | V = betapdf(U , A_seq, B_seq); % p by d matrix 34 | end 35 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/BPprime_basis.m: -------------------------------------------------------------------------------- 1 | % Bernstein polynomial prime basis function 2 | % First-order of Derivative Beta densities 3 | % u: evaluating point, could be a p by 1 vector 4 | % k: degree of BPs 5 | % opt: BP/exBP, standard or extended BP 6 | 7 | % V: a P by D matrix storing basis funcitons 8 | 9 | % Shaobo Han 10 | % 08/10/2015 11 | 12 | function V = BPprime_basis(u, k, opt) 13 | if nargin<3, 14 | opt = 'BP'; % Standard BP by default 15 | end 16 | p = length(u); % # of variables 17 | switch opt 18 | case 'BP' % Standard BP 19 | d = k; % # of basis functions 20 | a_seq = 1:k; b_seq = k-a_seq+1; 21 | case 'exBP' % extended BP 22 | d = k*(k+1)/2; % # of basis functions 23 | a_seq = zeros(1, d); b_seq = zeros(1, d); 24 | tmp_idx = cumsum(1:k); 25 | for j = 1:k 26 | tmp2=tmp_idx(j); tmp1=tmp_idx(j)-j+1; 27 | tmp_a = 1:j; tmp_b = j-tmp_a+1; 28 | a_seq(1, tmp1:tmp2) = tmp_a; b_seq(1, tmp1:tmp2) = tmp_b; 29 | end 30 | end 31 | A_seq = repmat(a_seq, p,1); B_seq = repmat(b_seq, p,1); 32 | U = repmat(u, 1, d); % p by d matrix 33 | V = (A_seq+B_seq-1).*(betapdf(U , A_seq-1, B_seq)-betapdf(U , A_seq, B_seq-1)); % p by d matrix 34 | % Remove NaN's 35 | % Defining betapdf(u, a, 0) = betapdf(u,0,b) = 0; 36 | Idx = (A_seq == 1)|(B_seq == 1); 37 | V(Idx) = 0; 38 | end 39 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/Cal_mcELBO.m: -------------------------------------------------------------------------------- 1 | function sELBO = Cal_mcELBO(trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par) 2 | tmpMu=par.Mu; tmpC = par.C; tmpW=par.W; 3 | N_mc = opt.N_mc; PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; k = opt.k; 4 | P = length(tmpMu); sELBOM = zeros(1, N_mc); 5 | v_t = sqrt(diag(tmpC*tmpC')); 6 | for i = 1: N_mc 7 | % Generate a P by 1 multivariate Normal vector 8 | w_t = randn(P,1); Z_t = tmpMu + tmpC*w_t; 9 | if opt.adaptivePhi == 1 10 | Z = diag(1./v_t)*(Z_t-tmpMu); 11 | u = Phi_f(Z, PhiPar, PhiType); 12 | sphi = diag(1./v_t)*sphi_f(Z, PhiPar, PhiType); 13 | else 14 | u = Phi_f(Z_t, PhiPar, PhiType); 15 | sphi = sphi_f(Z_t, PhiPar, PhiType); 16 | end 17 | 18 | BPcdf_basiseval = BPcdf_basis(u, k, BPtype); 19 | tmpB = sum(BPcdf_basiseval.*tmpW, 2); 20 | if strcmp(trueModel, 'PoLog4')==1 21 | tmpx = iPsi_f4(tmpB, opt); 22 | else 23 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 24 | end 25 | logp = logmodel(tmpx, trueModel, fix); 26 | BPpdf_basiseval = BPpdf_basis(u, k, BPtype); 27 | tmpb = sum(BPpdf_basiseval.*tmpW, 2); 28 | if strcmp(trueModel, 'PoLog4')==1 29 | spsi = spsi_f4(tmpx, opt); 30 | else 31 | spsi = spsi_f(tmpx, PsiPar, PsiType); 32 | end 33 | hdz1 = diag(sphi./spsi)*tmpb; 34 | sELBOM(i) = logp + sum(log(hdz1))+ P/2*log(2*pi)+P/2 + sum(log(diag(tmpC))); 35 | end 36 | sELBO = median(sELBOM); 37 | end 38 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/Grad_MuCW.m: -------------------------------------------------------------------------------- 1 | function [Delta_Mu, Delta_C, Delta_w, WinSet] = Grad_MuCW(VGmethod, trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par, WinSet) 2 | NumberZ = opt.NumberZ; P = fix.P; D = opt.D; 3 | PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; k = opt.k; 4 | Mu_old = par.Mu; C_old = par.C; W_old = par.W; 5 | MuMC = zeros(P, NumberZ); 6 | CMC = zeros(P,P, NumberZ); 7 | v_t = sqrt(diag(C_old*C_old')); 8 | WMC = zeros(P, D, NumberZ); 9 | for jj = 1:NumberZ 10 | 11 | %% Draw Sample w_t and Z_t 12 | Flag_outlier = 1; 13 | while Flag_outlier == 1 14 | % Generate a P by 1 multivariate Normal vector 15 | w_t = randn(P,1); Z_t = Mu_old + C_old*w_t; 16 | if opt.adaptivePhi == 1 17 | Z = diag(1./v_t)*(Z_t-Mu_old); 18 | u = Phi_f(Z, PhiPar, PhiType); 19 | else 20 | u = Phi_f(Z_t, PhiPar, PhiType); 21 | end 22 | BPcdf_basiseval = BPcdf_basis(u, k, BPtype); 23 | tmpB = sum(BPcdf_basiseval.*W_old, 2); 24 | 25 | if strcmp(trueModel, 'PoLog4')==1 26 | tmpx = iPsi_f4(tmpB, opt); 27 | else 28 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 29 | end 30 | 31 | logp = logmodel(tmpx, trueModel, fix); 32 | % Online outlier detection 33 | score = mzscore_test(logp, WinSet); 34 | if abs(score) < opt.OutlierTol 35 | Flag_outlier = 0; WinSet_old = WinSet; 36 | WinSet = [WinSet_old, logp]; 37 | WinSet(1) = []; 38 | end 39 | end 40 | if opt.adaptivePhi == 1 41 | sphi = diag(1./v_t)*sphi_f(Z, PhiPar, PhiType); 42 | dphi = -diag(1./v_t)*(Z.*sphi); 43 | else 44 | sphi = sphi_f(Z_t, PhiPar, PhiType); 45 | dphi = dphi_f(Z_t, PhiPar, PhiType); 46 | end 47 | BPpdf_basiseval = BPpdf_basis(u, k, BPtype); 48 | tmpb = sum(BPpdf_basiseval.*W_old, 2); 49 | BPprime_basiseval = BPprime_basis(u, k, BPtype); 50 | tmpbprime = sum(BPprime_basiseval.*W_old, 2); 51 | rho1prime = sphi.*tmpbprime; 52 | if strcmp(trueModel, 'PoLog4')==1 53 | spsi = spsi_f4(tmpx, opt); 54 | dpsi = dpsi_f4(tmpx,opt); 55 | else 56 | spsi = spsi_f(tmpx, PsiPar, PsiType); 57 | dpsi = dpsi_f(tmpx,PsiPar, PsiType); 58 | end 59 | 60 | g = derivatives(tmpx, trueModel, fix); 61 | 62 | hdz1 = diag(sphi./spsi)*tmpb; 63 | rho3prime = hdz1.*dpsi; 64 | hdz2 = (rho1prime.*sphi.*spsi... 65 | +tmpb.*dphi.*spsi... 66 | -tmpb.*sphi.*rho3prime)./(spsi.^2); 67 | lnh1dz = hdz2./hdz1; 68 | ls_z = g.*hdz1+lnh1dz; 69 | switch VGmethod 70 | case 'Analytic' 71 | dmu = ls_z; 72 | dC = tril(ls_z*w_t')+ diag(1./diag(C_old)); 73 | case 'Numeric' 74 | tmppar.Mu = Mu_old; tmppar.Sigma = C_old*C_old'; 75 | g2 = logqpost_derivatives(Z_t, inferModel, tmppar); 76 | dmu = ls_z-g2; 77 | dC = tril(dmu*w_t'); 78 | hdw = diag(1./spsi)*BPcdf_basiseval; 79 | lnh1dw = diag(1./tmpb)*BPpdf_basiseval - diag(dpsi./spsi.^2)*BPcdf_basiseval; 80 | end 81 | MuMC(:, jj) = dmu; 82 | CMC(:, :, jj) = dC; 83 | WMC(:,:,jj) = diag(g)*hdw+lnh1dw; 84 | end 85 | Delta_Mu = mean(MuMC, 2); 86 | Delta_C= mean(CMC, 3); 87 | Delta_w = median(WMC, 3); 88 | end 89 | 90 | 91 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/Phi_f.m: -------------------------------------------------------------------------------- 1 | % Phi function: mapping from (-infty, infty) to unit closed interval [0,1] 2 | % Shaobo Han 3 | % 09/17/2015 4 | function u = Phi_f(z, par, opt) 5 | switch opt 6 | case 'Normal' 7 | m0 = par(1); v0 = par(2); 8 | u = normcdf(z, m0, sqrt(v0)); 9 | end 10 | end 11 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/PoissonLogLinearModel.txt: -------------------------------------------------------------------------------- 1 | 2 | model 3 | { 4 | # priors 5 | 6 | beta0 ~ dnorm(0,sigma) 7 | 8 | beta1 ~ dnorm(0,sigma) 9 | 10 | beta2 ~ dnorm(0,sigma) 11 | 12 | sigma <- pow(tau, -1) 13 | 14 | tau ~ dgamma(1, 1) 15 | 16 | 17 | # likelihood 18 | 19 | for(i in 1:N.cells) 20 | 21 | { 22 | n50[i] ~ dpois(lambda[i]) 23 | 24 | log(lambda[i]) <- beta0 + beta1*elev50[i] + beta2*pow(elev50[i],2) 25 | 26 | # this part is here in order to make nice prediction curves: 27 | 28 | prediction[i] ~ dpois(lambda[i]) 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/PoissonLogMCMC.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shaobohan/VariationalGaussianCopula/a31aa2cc265b317979dfa4659dc5f75909d073f7/VGC-PoissonLogLinear/PoissonLogMCMC.mat -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/PoissonLogVGC.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shaobohan/VariationalGaussianCopula/a31aa2cc265b317979dfa4659dc5f75909d073f7/VGC-PoissonLogLinear/PoissonLogVGC.mat -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/SimplxProj.m: -------------------------------------------------------------------------------- 1 | function X = SimplxProj(Y) 2 | % Projects each row vector in the N by D marix Y to the probability 3 | % simplex in D-1 dimensions 4 | [N, D] = size(Y); 5 | X = sort(Y, 2, 'descend'); 6 | Xtmp = (cumsum(X,2)-1)*diag(sparse(1./(1:D))); 7 | X = max(bsxfun(@minus, Y, Xtmp(sub2ind([N,D], (1:N)', sum(X>Xtmp,2)))),0); 8 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/TreeData50.mat: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shaobohan/VariationalGaussianCopula/a31aa2cc265b317979dfa4659dc5f75909d073f7/VGC-PoissonLogLinear/TreeData50.mat -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/demo_JAGS_PoissonLogLinear.R: -------------------------------------------------------------------------------- 1 | # Poisson Log Linear Regression example for the paper 2 | # "Variational Gaussian Copula Inference", 3 | # Shaobo Han, Xuejun Liao, David. B. Dunson, and Lawrence Carin, 4 | # The 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016) 5 | # ----------------------------- 6 | # Download and install JAGS as per operating system requriements. 7 | # http://mcmc-jags.sourceforge.net/ 8 | # ----------------------------- 9 | # JAGS (Implemented via RJAGS) 10 | # For more information about this dataset, see 11 | # https://github.com/petrkeil/Statistics/blob/master/Lecture%203%20-%20poisson_regression/poisson_regression.Rmd 12 | # ----------------------------- 13 | # Shaobo Han, Duke University 14 | # shaobohan@gmail.com 15 | # 10/01/2015 16 | 17 | library(R.matlab) 18 | library(rjags) 19 | 20 | path <- getwd() 21 | pathname <- file.path(path, "TreeData50.mat") 22 | data <- readMat(pathname) 23 | attach(data) 24 | 25 | ## JAGS 26 | jags.data <- list(N.cells = length(y), n50 = y, elev50 = x) 27 | cat("\n model\n {\n # priors\n 28 | beta0 ~ dnorm(0,sigma)\n 29 | beta1 ~ dnorm(0,sigma)\n 30 | beta2 ~ dnorm(0,sigma)\n 31 | sigma <- pow(tau, -1)\n 32 | tau ~ dgamma(1, 1)\n 33 | \n # likelihood\n 34 | for(i in 1:N.cells)\n 35 | {\n n50[i] ~ dpois(lambda[i])\n 36 | log(lambda[i]) <- beta0 + beta1*elev50[i] + beta2*pow(elev50[i],2)\n 37 | # this part is here in order to make nice prediction curves:\n 38 | prediction[i] ~ dpois(lambda[i])\n } \n }\n ", 39 | file = "PoissonLogLinearModel.txt") 40 | params <- c("beta0", "beta1", "beta2", "tau", "prediction") 41 | jm <- jags.model("PoissonLogLinearModel.txt", data = jags.data, n.chains = 10, n.adapt = 1000) 42 | update(jm, n.iter = 10000) 43 | jm.sample <- jags.samples(jm, variable.names = params, n.iter = 10000, thin = 1) 44 | 45 | summary(as.mcmc.list(jm.sample$beta0)) 46 | summary(as.mcmc.list(jm.sample$beta1)) 47 | summary(as.mcmc.list(jm.sample$beta2)) 48 | summary(as.mcmc.list(jm.sample$tau)) 49 | 50 | # Save MCMC samples to mat file 51 | filename <- paste("PoissonLogMCMC", ".mat", sep="") 52 | writeMat(filename, beta0=jm.sample$beta0, beta1=jm.sample$beta1, beta2=jm.sample$beta2, 53 | tau=jm.sample$tau) -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/demo_VGC_PoissonLogLinear.m: -------------------------------------------------------------------------------- 1 | % Poisson Log Linear Regression example for the paper 2 | % "Variational Gaussian Copula Inference", 3 | % Shaobo Han, Xuejun Liao, David. B. Dunson, and Lawrence Carin, 4 | % The 19th International Conference on Artificial Intelligence and Statistics (AISTATS 2016) 5 | % ----------------------------- 6 | % Code written by 7 | % Shaobo Han, Duke University 8 | % shaobohan@gmail.com 9 | % 10/01/2015 10 | 11 | % Methods 12 | % 1 JAGS (Implemented via RJAGS) 13 | % 2 Stochastic VGC with Bernstein Polynomials 14 | 15 | % clear all; close all; clc; 16 | load Treedata50.mat 17 | % For more information about this dataset, see 18 | % https://github.com/petrkeil/Statistics/blob/master/Lecture%203%20-%20poisson_regression/poisson_regression.Rmd 19 | 20 | xm = x'; 21 | fix.n = length(y); 22 | fix.Xmatrix = [ones(1, fix.n); xm; xm.^2]; % X is a 3 by N matrix 23 | fix.y = y; fix.logfac_y =log(factorial(y)); 24 | fix.a0 = 1; fix.b0 = 1; 25 | fix. P = 4; % Number of latent variables 26 | figplot.nbins = 50; 27 | 28 | trueModel = 'PoLog4'; opt.trueModel =trueModel; 29 | VGmethod = 'Numeric'; 30 | inferModel2 = 'MVN'; % Multivariate Normal 31 | 32 | %% VGC Numeric 33 | opt.adaptivePhi = 0; opt.normalize = 1; 34 | fix.c = 1; % unnormalizing constant 35 | opt.k = 10; % Degree/Maximum Degree of Bernstein Polynomials 36 | opt.MaxIter = 50; % Number of SGD stages 37 | opt.NumberZ = 1; % Average Gradients 38 | opt.InnerIter = 500; % Number of iteration 39 | opt.N_mc = 1; % Number of median average in sELBO stepsize search 40 | opt.nsample = 1e5; 41 | 42 | BPtype = 'BP'; % Bernstein Polynomials 43 | % BPtype = 'exBP'; % Extended Bernstein Polynomials 44 | PsiType = 'Normal'; opt.PsiPar(1) = 0; opt.PsiPar(2) = 1; % variance 45 | 46 | opt.PoM.PsiType1 = 'Normal'; 47 | opt.PoM.PsiPar1(1) = 0; 48 | opt.PoM.PsiPar1(2) = 1; % variance 49 | opt.PoM.PsiType2 = 'Exp'; 50 | opt.PoM.PsiPar2 = 1; 51 | 52 | % PsiType = 'Exp'; opt.PsiPar = 1; 53 | PhiType = 'Normal'; opt.PhiPar(1) = 0; opt.PhiPar(2) =1; % variance 54 | opt.Wthreshold = 1e4; 55 | 56 | % Learning rate 57 | opt.LearnRate.Mu = 1e-3; opt.LearnRate.C = 1e-4; 58 | opt.LearnRate.W = 0.25.*1e-3; opt.LearnRate.dec = 0.95; % decreasing base learning rate 59 | 60 | switch BPtype 61 | case 'BP' 62 | opt.D = opt.k; 63 | case 'exBP' 64 | opt.D = opt.k*(opt.k+1)/2; % # of basis functions 65 | end 66 | 67 | % Diagonal constraint on Upsilon 68 | opt.diagUpsilon = 0; 69 | 70 | %% Initialization 71 | ini.Mu = opt.PhiPar(1).*ones(fix.P,1); 72 | ini.C = sqrt(opt.PhiPar(2)).*0.1.*eye(fix.P); 73 | % ini.w = randBPw(fix.P, opt.D, 1, 1); 74 | ini.w = 1./opt.D.*ones(fix.P, opt.D); 75 | % ini.w = randBPw(fix.P, opt.D, 1, 1); 76 | 77 | % Median Outlier Removal 78 | opt.OutlierTol = 100; % Threshold for online outlier detection 79 | opt.WinSize = 20; % Size of the window 80 | ini.WinSet = ini_WinSet(trueModel, PhiType, PsiType, BPtype, fix, opt, ini); 81 | 82 | %% VGC MuCW-Numeric 83 | opt.updateW = 1; opt.updateMuC = 1; 84 | [ELBO3, par3] = vgcbp_MuCW(trueModel, inferModel2, PhiType, PsiType, BPtype, VGmethod, fix, ini, opt); 85 | 86 | tic; 87 | figplot.Y_svc3 = sampleGC(PhiType, PsiType, BPtype, opt, par3); 88 | figplot.Y_svc3(sum(~isfinite(figplot.Y_svc3),2)~=0,:) = []; 89 | toc; 90 | 91 | PoissonLogVGC = figplot.Y_svc3'; 92 | save PoissonLogVGC.mat PoissonLogVGC 93 | %% ------------------------------ 94 | load PoissonLogVGC.mat 95 | load PoissonLogMCMC.mat 96 | 97 | beta0 = beta0(:)'; beta1 = beta1(:)'; 98 | beta2 = beta2(:)'; tau = tau(:)'; 99 | 100 | X_mcmc = [beta0; beta1; beta2; tau]; 101 | X_vgc =PoissonLogVGC; 102 | ncoutour = 6; figplot.ls = 1.5; figplot.nbins = 50; 103 | 104 | figure 105 | for i = 1:4 106 | for j = 1:4 107 | k = (i-1)*4+j; 108 | subplot(4,4,k) 109 | if i==j 110 | [f_x1,x_x1] = hist(X_mcmc(i,:), figplot.nbins); 111 | [f_x2,x_x2] = hist(X_vgc(i,:), figplot.nbins); 112 | plot(x_x1, f_x1./trapz(x_x1,f_x1), 'r--', 'linewidth', 2) 113 | hold on 114 | plot(x_x2, f_x2./trapz(x_x2,f_x2), 'b-', 'linewidth', 2) 115 | hold off 116 | legend('JAGS','VGC-BP') 117 | else 118 | tmp = hist1D2D([X_mcmc(i,:); X_mcmc(j,:)]', figplot.nbins); 119 | tmp2 = hist1D2D([X_vgc(i,:); X_vgc(j,:)]', figplot.nbins); 120 | contour(tmp.Seqx,tmp.Seqy, tmp.count./sum(sum(tmp.count)), ncoutour, 'r--' , 'linewidth', 2*figplot.ls); 121 | hold on 122 | contour(tmp2.Seqx,tmp2.Seqy, tmp2.count./sum(sum(tmp2.count)), ncoutour, 'b' , 'linewidth', 2*figplot.ls); 123 | hold off 124 | legend('JAGS', 'VGC-BP') 125 | end 126 | end 127 | end 128 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/derivatives.m: -------------------------------------------------------------------------------- 1 | % Derivatives of Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function g = derivatives(x, model, fix) 7 | switch model 8 | case 'PoLog4' 9 | y = fix.y; Xmatrix = fix.Xmatrix; 10 | mu = exp(Xmatrix'*x(1:3)); 11 | tmp_tau = x(4); g = zeros(4,1); 12 | tmp_ymu = y-mu; 13 | g(1) = sum(tmp_ymu)-1./tmp_tau.*x(1); 14 | g(2) = sum(Xmatrix(2,:)'.*tmp_ymu)-1./tmp_tau.*x(2); 15 | g(3) = sum(Xmatrix(3,:)'.*tmp_ymu)-1./tmp_tau.*x(3); 16 | g(4) = -3./(2.*tmp_tau)+(x(1)^2+x(2)^2+x(3)^2)./2./(tmp_tau.^2)... 17 | +(fix.a0-1)./tmp_tau-fix.b0; 18 | case 'Horseshoe' 19 | y = fix.y; 20 | g = zeros(2,1); 21 | g(1) = -2/x(1)+y^2/2/(x(1)^2)+x(2)/(x(1)^2); 22 | g(2) = -1/x(1)-1; 23 | case 'SkewNormal' 24 | m = fix.snMu; 25 | Lambda = fix.snSigma; 26 | alpha = fix.snAlpha; 27 | tmp = alpha'*(x-m); 28 | g = -(Lambda\(x-m))+normpdf(tmp).*alpha./normcdf(tmp); 29 | case 'LogNormal2' 30 | sigmaY1 = fix.LNsigmaY1; sigmaY2 = fix.LNsigmaY2; 31 | muY1 = fix.LNmuY1; muY2 = fix.LNmuY2; LNrho = fix.LNrho; 32 | alpha1 = (log(x(1))-muY1)./sigmaY1; 33 | alpha2= (log(x(2))-muY2)./sigmaY2; 34 | g = zeros(2,1); 35 | g(1)=-1./x(1)-(alpha1-LNrho*alpha2)/(1-LNrho.^2)./x(1)./sigmaY1; 36 | g(2)=-1./x(2)-(alpha2-LNrho*alpha1)/(1-LNrho.^2)./x(2)./sigmaY2; 37 | case 'Gamma' 38 | alpha = fix.alpha; 39 | beta = fix.beta; 40 | g = (alpha-1)./x-beta; 41 | case 'Student' 42 | nu = fix.nu; 43 | g = -(nu+1)*x/(nu+x^2); 44 | case 'Beta'; 45 | alpha = fix.alpha; beta = fix.beta; 46 | g = (alpha-1)/x-(beta-1)/(1-x); 47 | end 48 | end 49 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/dphi_f.m: -------------------------------------------------------------------------------- 1 | % dphi function: derivative of sphi 2 | % Shaobo Han 3 | % 09/24/2015 4 | function output = dphi_f(z, par, opt) 5 | switch opt 6 | case 'Normal' 7 | m0 = par(1); v0 = par(2); 8 | output = -(z./v0).*normpdf(z, m0, sqrt(v0)); 9 | end 10 | end 11 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/dpsi_f.m: -------------------------------------------------------------------------------- 1 | % dpsi function: derivative of small psi 2 | % Shaobo Han 3 | % 08/10/2015 4 | function output = dpsi_f(x, par, opt) 5 | if nargin<3, 6 | opt = 'Exp'; % exponential distribution by default 7 | end 8 | if nargin<2, 9 | par = 0.1; % lambda = 0.1 by default 10 | end 11 | switch opt 12 | case 'Exp' % exp(lambda) 13 | output = -par.*exppdf(x,1/par); 14 | case 'Normal' % 15 | m0 = par(1); % mean 16 | v0 = par(2); % variance 17 | output = (-x./v0).*normpdf(x,m0,sqrt(v0)); 18 | case 'Beta' 19 | a0 = par(1); b0 = par(2); 20 | output =(a0+b0-1)*(betapdf(x, a0-1, b0)- betapdf(x, a0, b0-1)); 21 | end 22 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/dpsi_f4.m: -------------------------------------------------------------------------------- 1 | % dpsi function: derivative of small psi 2 | % Shaobo Han 3 | % 08/10/2015 4 | function output = dpsi_f4(x, opt) 5 | output = zeros(4,1); 6 | output(1)= dpsi_f(x(1), opt.PoM.PsiPar1, opt.PoM.PsiType1); 7 | output(2)= dpsi_f(x(2), opt.PoM.PsiPar1, opt.PoM.PsiType1); 8 | output(3)= dpsi_f(x(3), opt.PoM.PsiPar1, opt.PoM.PsiType1); 9 | output(4)= dpsi_f(x(4), opt.PoM.PsiPar2, opt.PoM.PsiType2); 10 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/hist1D2D.m: -------------------------------------------------------------------------------- 1 | function Output=hist1D2D(Matrix, nbins) 2 | % 1D and 2D histogram 3 | % Matrix N by 2 matrix 4 | [f_x1,x_x1] = hist(Matrix(:,1), nbins); 5 | [f_x2,x_x2] = hist(Matrix(:,2), nbins); 6 | [count,Dy] = hist3(Matrix,[nbins+2,nbins]); 7 | Seqx= Dy{1,1}; Seqy = Dy{1,2}; 8 | Output.f_x1 = f_x1; Output.x_x1 = x_x1; 9 | Output.f_x2 = f_x2; Output.x_x2 = x_x2; 10 | Output.count =count'; Output.Seqx =Seqx; Output.Seqy =Seqy; 11 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/iPsi_f.m: -------------------------------------------------------------------------------- 1 | % Invserse Psi function: 2 | % Shaobo Han 3 | % 08/10/2015 4 | function output = iPsi_f(p, par, opt) 5 | if nargin<3, 6 | opt = 'Exp'; % exponential distribution by default 7 | end 8 | if nargin<2, 9 | par = 0.1; % lambda = 0.1 by default 10 | end 11 | 12 | p(p>1-1e-16) = 1-1e-16; % numerical stability 13 | 14 | switch opt 15 | case 'Exp' % exp(lambda) 16 | % output = -log(1-p)./par; 17 | output = expinv(p,1/par); 18 | case 'Normal' % 19 | m0 = par(1); v0 = par(2); 20 | output = norminv(p,m0,sqrt(v0)); 21 | case 'Beta' 22 | a0 = par(1); b0 = par(2); 23 | output = betainv(p, a0, b0); 24 | end 25 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/iPsi_f4.m: -------------------------------------------------------------------------------- 1 | % Invserse Psi function: 2 | % Shaobo Han 3 | % 08/10/2015 4 | function output = iPsi_f4(p, opt) 5 | output = zeros(4,1); 6 | output(1)= iPsi_f(p(1), opt.PoM.PsiPar1, opt.PoM.PsiType1); 7 | output(2)= iPsi_f(p(2), opt.PoM.PsiPar1, opt.PoM.PsiType1); 8 | output(3)= iPsi_f(p(3), opt.PoM.PsiPar1, opt.PoM.PsiType1); 9 | output(4)= iPsi_f(p(4), opt.PoM.PsiPar2, opt.PoM.PsiType2); 10 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/ini_WinSet.m: -------------------------------------------------------------------------------- 1 | function WinSet = ini_WinSet(trueModel, PhiType, PsiType, BPtype, fix, opt, ini) 2 | k = opt.k; PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; 3 | P = fix.P; Mu_old = ini.Mu; C_old = ini.C; W_old = ini.w; 4 | WinSet = zeros(1, opt.WinSize); 5 | vt = sqrt(diag(C_old*C_old')); 6 | for i = 1:opt.WinSize 7 | w_t0 = randn(P,1); Z_t0 = Mu_old + C_old*w_t0; 8 | if opt.adaptivePhi == 1 9 | Z0 = diag(1./vt)*(Z_t0-Mu_old); 10 | u0 = Phi_f(Z0, PhiPar, PhiType); 11 | else 12 | u0 = Phi_f(Z_t0, PhiPar, PhiType); 13 | end 14 | BPcdf_basiseval0 = BPcdf_basis(u0, k, BPtype); 15 | tmpB0 = sum(BPcdf_basiseval0.*W_old, 2); 16 | if strcmp(trueModel, 'PoLog4')==1 17 | tmpx0 = iPsi_f4(tmpB0, opt); 18 | else 19 | tmpx0 = iPsi_f(tmpB0, PsiPar, PsiType); 20 | end 21 | WinSet(i) = logmodel(tmpx0, trueModel, fix); 22 | end 23 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/logdet.m: -------------------------------------------------------------------------------- 1 | function y = logdet(A) 2 | %LOGDET Logarithm of determinant for positive-definite matrix 3 | % logdet(A) returns log(det(A)) where A is positive-definite. 4 | % This is faster and more stable than using log(det(A)). 5 | % Note that logdet does not check if A is positive-definite. 6 | % If A is not positive-definite, the result will not be the same as log(det(A)). 7 | 8 | % Written by Tom Minka 9 | % (c) Microsoft Corporation. All rights reserved. 10 | 11 | U = chol(A); 12 | y = 2*sum(log(diag(U))); -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/logmodel.m: -------------------------------------------------------------------------------- 1 | % Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function p = logmodel(x, model, fix) 7 | switch model 8 | case 'PoLog4' 9 | y = fix.y; Xmatrix = fix.Xmatrix; logfac_y=fix.logfac_y; 10 | mu = exp(Xmatrix'*x(1:3)); 11 | tmp0 = sum(y.*log(mu)-mu-logfac_y); 12 | tmp_tau = x(4); tmp_sqrttau = sqrt(tmp_tau); 13 | tmp1 = log(normpdf(x(1), 0, tmp_sqrttau)); 14 | tmp2 = log(normpdf(x(2), 0, tmp_sqrttau)); 15 | tmp3 = log(normpdf(x(3), 0, tmp_sqrttau)); 16 | tmp4 = log(gampdf(x(4), fix.a0, 1./fix.b0)); 17 | p = tmp0+tmp1+tmp2+tmp3+tmp4; 18 | case 'Horseshoe' 19 | y = fix.y; 20 | c0 = -0.5*log(2*pi)-2*log(gamma(0.5)); 21 | p = c0-2*log(x(1))-y.^2./2./x(1)-x(2)./x(1)-x(2); 22 | case 'LogNormal2' 23 | sigmaY1 = fix.LNsigmaY1; sigmaY2 = fix.LNsigmaY2; 24 | muY1 = fix.LNmuY1; muY2 = fix.LNmuY2; LNrho = fix.LNrho; 25 | c0 = -log(2*pi)-log(sigmaY1*sigmaY2*sqrt(1-LNrho^2)); 26 | alpha1 = (log(x(1))-muY1)/sigmaY1; 27 | alpha2= (log(x(2))-muY2)/sigmaY2; 28 | q = 1/(1-LNrho^2)*(alpha1^2-2*LNrho*alpha1*alpha2+alpha2^2); 29 | p = c0-log(x(1))-log(x(2))-q/2; 30 | case 'LogNormal' 31 | sigmaY = fix.LNsigmaY; 32 | muY = fix.LNmuY; 33 | c0 = -0.5*log(2*pi)-log(sigmaY); 34 | alpha = (log(x)-muY)./sigmaY; 35 | p = c0-log(x)- alpha.^2/2; 36 | case 'SkewNormal' 37 | m = fix.snMu; 38 | Lambda = fix.snSigma; 39 | alpha = fix.snAlpha; 40 | c = fix.c; 41 | p = log(2)+log(mvnpdf(x,m,Lambda))+log(normcdf(alpha'*(x-m)))+log(c); 42 | case 'Gamma' 43 | alpha = fix.alpha; 44 | beta = fix.beta; 45 | c = fix.c; 46 | p = log(gampdf(x, alpha, 1/beta))+log(c); 47 | case 'Student' 48 | nu = fix.nu; c = fix.c; 49 | p = log(tpdf(x,nu))+log(c); 50 | case 'Exp' 51 | lambda = fix.lambda; c = fix.c; 52 | p = log(exppdf(x,1/lambda))+log(c); 53 | case 'Beta'; 54 | alpha = fix.alpha; beta = fix.beta; c = fix.c; 55 | p = log(betapdf(x, alpha, beta))+log(c); 56 | end 57 | end 58 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/logqpost.m: -------------------------------------------------------------------------------- 1 | % Log q posteiror 2 | % Model dependent term 3 | % Shaobo Han 4 | % 09/11/2015 5 | 6 | function p = logqpost(x, model, par) 7 | switch model 8 | case 'Horseshoe' 9 | y = par.y; 10 | c0 = -0.5*log(2*pi)-2*log(gamma(0.5)); 11 | p = c0-2*log(x(1))-y.^2./2./x(1)-x(2)./x(1)-x(2); 12 | case 'SkewNormal' 13 | m = par.snMu; 14 | Lambda = par.snSigma; 15 | alpha = par.snAlpha; 16 | p = log(2)+log(mvnpdf(x,m,Lambda))+log(normcdf(alpha'*(x-m))) ; 17 | case 'MVN' 18 | mu = par.Mu; C= par.C; Sigma=C*C'; 19 | d = length(mu); 20 | p = -d*log(2*pi)/2-0.5*logdet(Sigma)-0.5*(x-mu)'*(Sigma\(x-mu)); 21 | case 'MVNdiag' 22 | mu = par.Mu; C= par.C; Sigma=C*C'; 23 | p = sum(log(normpdf(x, mu, sqrt(diag(Sigma))))); 24 | end 25 | end 26 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/logqpost_derivatives.m: -------------------------------------------------------------------------------- 1 | % Derivatives of Log likelihood and Log prior 2 | % Model dependent term 3 | % Shaobo Han 4 | % 08/10/2015 5 | 6 | function g = logqpost_derivatives(x, model, par) 7 | switch model 8 | case 'Horseshoe' 9 | y = par.y; 10 | g = zeros(2,1); 11 | g(1) = -2/x(1)+y^2/2/(x(1)^2)+x(2)/(x(1)^2); 12 | g(2) = -1/x(1)-1; 13 | case 'SkewNormal' 14 | m = par.snMu; 15 | Lambda = par.snSigma; 16 | alpha = par.snAlpha; 17 | g = -inv(Lambda)*(x-m)+mvnpdf(alpha'*(x-m)).*alpha./mvncdf(alpha'*(x-m)); 18 | case 'MVN' 19 | mu = par.Mu; Sigma= par.Sigma; 20 | g = Sigma\(mu-x); 21 | case 'MVNdiag' 22 | mu = par.Mu; Sigma= par.Sigma; 23 | g = diag(1./diag(Sigma))*(mu-x); 24 | end 25 | end 26 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/mzscore_test.m: -------------------------------------------------------------------------------- 1 | % Calculate the modified Z-score of a query point from a window set 2 | % Shaobo Han 3 | % 08/13/2015 4 | function score = mzscore_test(x, windowset) 5 | tmpX = [x, windowset]; 6 | tmpm = median(tmpX); 7 | mad = median(abs(tmpX-tmpm)); 8 | score = 0.6745*(x-tmpm)/mad; 9 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/post2vec.m: -------------------------------------------------------------------------------- 1 | function vec=post2vec(post) 2 | % convert posterior stucture to a 1=-d col. vector 3 | vec=[post.m;post.c(:)]; 4 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/randBPw.m: -------------------------------------------------------------------------------- 1 | % Generate a P*D BP weight matrix 2 | % (every column is a probability vector which sums up to one) 3 | % P: number of variables 4 | % D: number of BP basis functions 5 | % a: shape parameter; b: scale parameter 6 | 7 | % Shaobo Han 8 | % 08/08/2015 9 | 10 | function w = randBPw(P, D, a, b) 11 | if nargin<4 12 | a = 1; b = 1; % by default 13 | end 14 | if D==1 15 | w = ones(P,1); 16 | else 17 | w0 = gamrnd(a,b,P,D); 18 | w = diag(1./sum(w0,2))*w0; 19 | end 20 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/randchol.m: -------------------------------------------------------------------------------- 1 | % Generate a P*P lower triangular Cholesky factor matrix 2 | % Shaobo Han 3 | % 08/07/2015 4 | function C = randchol(P) 5 | a = rand(P, P); 6 | aTa = a'*a; 7 | [V, D] = eig(aTa); 8 | aTa_PD = V*(D+1e-5)*V'; 9 | C=chol(aTa_PD, 'lower'); 10 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/sampleGC.m: -------------------------------------------------------------------------------- 1 | function Y_svc = sampleGC(PhiType, PsiType, BPtype, opt, par) 2 | PhiPar = opt.PhiPar; PsiPar = opt.PsiPar; 3 | nsample = opt.nsample; k = opt.k; 4 | Mu = par.Mu; C = par.C; W = par.W; 5 | P = length(Mu); 6 | % Sample from Gaussian Copula 7 | optmu_s2 = Mu; optSig_s2 = C*C'; 8 | wtz2 = mvnrnd(optmu_s2, optSig_s2, nsample); 9 | Y_svc = zeros(nsample,P); 10 | if opt.adaptivePhi == 0 11 | for ns = 1:nsample 12 | tmpz = wtz2(ns,:)'; 13 | tmpu = Phi_f(tmpz, PhiPar, PhiType); 14 | BPcdf_basiseval = BPcdf_basis(tmpu, k, BPtype); 15 | tmpB = sum(BPcdf_basiseval.*W, 2); 16 | if strcmp(opt.trueModel, 'PoLog4')==1 17 | tmpx = iPsi_f4(tmpB, opt); 18 | else 19 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 20 | end 21 | Y_svc(ns,:) = tmpx; 22 | end 23 | else 24 | v_t = sqrt(diag(C*C')); 25 | for ns = 1:nsample 26 | tmpz0 = wtz2(ns,:)'; 27 | tmpz = diag(1./v_t)*(tmpz0-Mu); 28 | tmpu = Phi_f(tmpz, PhiPar, PhiType); 29 | BPcdf_basiseval = BPcdf_basis(tmpu, k, BPtype); 30 | tmpB = sum(BPcdf_basiseval.*W, 2); 31 | tmpx = iPsi_f(tmpB, PsiPar, PsiType); 32 | Y_svc(ns,:) = tmpx; 33 | end 34 | end 35 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/sphi_f.m: -------------------------------------------------------------------------------- 1 | % phi function: small phi, derivative of Phi 2 | % Shaobo Han 3 | % 08/07/2015 4 | function output = sphi_f(z, par, opt) 5 | switch opt 6 | case 'Normal' 7 | m0 = par(1); v0 = par(2); 8 | output = normpdf(z, m0, sqrt(v0)); 9 | end 10 | end 11 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/spsi_f.m: -------------------------------------------------------------------------------- 1 | % psi function: small psi, derivative of Psi 2 | % Shaobo Han 3 | % 08/10/2015 4 | 5 | function output = spsi_f(x, par, opt) 6 | if nargin<3, 7 | opt = 'Exp'; % exponential distribution by default 8 | end 9 | if nargin<2, 10 | par = 0.1; % lambda = 0.1 by default 11 | end 12 | switch opt 13 | case 'Exp' % exp(lambda) 14 | output = exppdf(x,1/par); 15 | case 'Normal' % 16 | m0 = par(1); v0 = par(2); 17 | output = normpdf(x, m0, sqrt(v0)); 18 | case 'Beta' 19 | a0 = par(1); b0 = par(2); 20 | output = betapdf(x, a0, b0); 21 | end 22 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/spsi_f4.m: -------------------------------------------------------------------------------- 1 | % psi function: small psi, derivative of Psi 2 | % Shaobo Han 3 | % 08/10/2015 4 | function output = spsi_f4(x, opt) 5 | output = zeros(4,1); 6 | output(1)= spsi_f(x(1), opt.PoM.PsiPar1, opt.PoM.PsiType1); 7 | output(2)= spsi_f(x(2), opt.PoM.PsiPar1, opt.PoM.PsiType1); 8 | output(3)= spsi_f(x(3), opt.PoM.PsiPar1, opt.PoM.PsiType1); 9 | output(4)= spsi_f(x(4), opt.PoM.PsiPar2, opt.PoM.PsiType2); 10 | end -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/vec2post.m: -------------------------------------------------------------------------------- 1 | function post=vec2post(vec,postDim) 2 | % convert vector of posterior params to posterior structure 3 | post.m=vec(1:postDim); 4 | vec=vec(postDim+1:end); 5 | post.c=reshape(vec(:),postDim,postDim); 6 | end 7 | -------------------------------------------------------------------------------- /VGC-PoissonLogLinear/vgcbp_MuCW.m: -------------------------------------------------------------------------------- 1 | function [ELBO, par] = vgcbp_MuCW(trueModel, inferModel, PhiType, PsiType, BPtype, VGmethod, fix, ini, opt) 2 | % Parameters Unpack 3 | InnerIter = opt.InnerIter; MaxIter = opt. MaxIter; 4 | r1 = opt.LearnRate.Mu; r2 = opt.LearnRate.C; 5 | r3 = opt.LearnRate.W; dec = opt.LearnRate.dec; 6 | 7 | if opt.diagUpsilon ==1 8 | display(['[VGC(S): diag], BPtype = ', BPtype,' PsiType = ', PsiType,' PhiType = ', PhiType]) 9 | else 10 | display(['[VGC(S): full], BPtype = ', BPtype,' PsiType = ', PsiType,' PhiType = ', PhiType]) 11 | end 12 | ELBO = zeros(InnerIter, MaxIter); 13 | RMSE = zeros(InnerIter, MaxIter); 14 | for iter = 1: MaxIter 15 | tic; 16 | for inner = 1:InnerIter 17 | % First Iteration 18 | if (iter == 1)&&(inner==1) 19 | par.Mu_old = ini.Mu; par.C_old = ini.C; par.W_old = ini.w; 20 | WinSet = ini.WinSet; 21 | par.Mu = par.Mu_old; par.C = par.C_old; par.W = par.W_old; 22 | else 23 | par.Mu_old = par.Mu; par.C_old = par.C; par.W_old = par.W; 24 | end 25 | %% Draw Samples and Calculate (Mu, C) 26 | [Delta_Mu, Delta_C, Delta_w, WinSet] = Grad_MuCW(VGmethod, trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par,WinSet); 27 | if opt.normalize ==1 28 | if norm(Delta_Mu,'fro')<=1e-5 29 | Delta_Mu = zeros(P,1); 30 | else 31 | Delta_Mu = Delta_Mu./norm(Delta_Mu,'fro'); 32 | end 33 | if norm(Delta_C,'fro')<=1e-5 34 | Delta_C = zeros(P,P); 35 | else 36 | Delta_C = Delta_C./norm(Delta_C,'fro'); 37 | end 38 | end 39 | if opt.updateMuC~=1 40 | Delta_C = zeros(size(Delta_C)); 41 | Delta_Mu = zeros(size(Delta_Mu)); 42 | end 43 | %% Update Mu 44 | par.Mu = par.Mu_old + r1.*Delta_Mu; 45 | if opt.adaptivePhi == 1 46 | par.Mu = zeros(size( par.Mu)); 47 | end 48 | %% Update Mu 49 | par.C = par.C_old + r2.*Delta_C; 50 | %% Draw Samples and Calculate Delta_W 51 | if opt.normalize ==1 52 | % Normalize 53 | if norm(Delta_w,'fro')<=1e-5 54 | Delta_w = zeros(size(Delta_w)); 55 | else 56 | Delta_w = diag(1./sqrt(sum(abs(Delta_w).^2,2)))*Delta_w; 57 | end 58 | end 59 | %% Update W 60 | tmpNorm = sqrt(sum(abs(Delta_w).^2,2)); 61 | UnstableIndex = (tmpNorm>opt.Wthreshold); 62 | tmpsum = sum(UnstableIndex); 63 | if tmpsum ~=0 64 | Delta_w(UnstableIndex,:) = zeros(tmpsum, opt.D); 65 | display('Warning: Unstable Delta W Omitted!') 66 | tmpNorm(UnstableIndex) 67 | end 68 | W0 = par.W_old + r3.*Delta_w; 69 | % Project Gradient Descent 70 | par.W = SimplxProj(W0); 71 | if sum(abs(sum(par.W,2)-1)>1e-5*ones(size(sum(par.W,2))))>0 72 | display('Error: Weight not Sum up to 1!') 73 | sum(par.W,2) 74 | par.W = par.W_old; 75 | end 76 | ELBO(inner, iter) = Cal_mcELBO(trueModel, inferModel, PhiType, PsiType, BPtype, fix, opt, par); 77 | if strcmp(trueModel, 'LogNormal2')==1 78 | RMSE(inner, iter) = norm(corrcov(par.C *par.C')- fix.trueUpsilon, 'fro')./norm(fix.trueUpsilon, 'fro'); 79 | end 80 | end 81 | r1 = r1*dec; r2 = r2*dec; r3 = r3*dec; 82 | t1= toc; 83 | display(['ELBO = ', num2str(median(ELBO(:,iter))), ', Iter: ' num2str(iter), '/' num2str(MaxIter),' Elapsed: ', num2str(t1) ' sec']); 84 | end 85 | par.WinSet = WinSet; 86 | par.RMSE = RMSE; 87 | end 88 | -------------------------------------------------------------------------------- /figure/VGC-JAGS.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shaobohan/VariationalGaussianCopula/a31aa2cc265b317979dfa4659dc5f75909d073f7/figure/VGC-JAGS.png -------------------------------------------------------------------------------- /figure/horseshoe.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shaobohan/VariationalGaussianCopula/a31aa2cc265b317979dfa4659dc5f75909d073f7/figure/horseshoe.png -------------------------------------------------------------------------------- /figure/lognormal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shaobohan/VariationalGaussianCopula/a31aa2cc265b317979dfa4659dc5f75909d073f7/figure/lognormal.png -------------------------------------------------------------------------------- /figure/margins.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shaobohan/VariationalGaussianCopula/a31aa2cc265b317979dfa4659dc5f75909d073f7/figure/margins.png --------------------------------------------------------------------------------