├── bayesvarsel_notes.pdf ├── .gitignore ├── sim.py ├── README.md ├── mcmc.py ├── linear_regression.py ├── linear_averaging.py ├── core.py └── LICENSE /bayesvarsel_notes.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/martinaragoneses/bma/HEAD/bayesvarsel_notes.pdf -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | 5 | # C extensions 6 | *.so 7 | 8 | # Distribution / packaging 9 | .Python 10 | env/ 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | *.egg-info/ 23 | .installed.cfg 24 | *.egg 25 | 26 | # PyInstaller 27 | # Usually these files are written by a python script from a template 28 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 29 | *.manifest 30 | *.spec 31 | 32 | # Installer logs 33 | pip-log.txt 34 | pip-delete-this-directory.txt 35 | 36 | # Unit test / coverage reports 37 | htmlcov/ 38 | .tox/ 39 | .coverage 40 | .coverage.* 41 | .cache 42 | nosetests.xml 43 | coverage.xml 44 | *,cover 45 | 46 | # Translations 47 | *.mo 48 | *.pot 49 | 50 | # Django stuff: 51 | *.log 52 | 53 | # Sphinx documentation 54 | docs/_build/ 55 | 56 | # PyBuilder 57 | target/ 58 | -------------------------------------------------------------------------------- /sim.py: -------------------------------------------------------------------------------- 1 | """ 2 | Provides testing utilities. 3 | """ 4 | 5 | import numpy as np 6 | import matplotlib.pyplot as plt 7 | import linear_averaging 8 | 9 | plt.style.use("ggplot") 10 | plt.rcParams.update({'font.size': 16}) 11 | 12 | 13 | 14 | def simulate_data(nobs, weights, cor=0.5): 15 | """Simulate a simple dataset where 10 predictors are irrelevant and 5 increasingly relevant. 16 | 17 | Predictors are jointly gaussian with 0 mean, given correlation and variance 1. 18 | 19 | Residuals are gaussian with 0 mean and variance 1. 20 | 21 | Parameters 22 | ---------- 23 | nobs : int {1, .., inf} 24 | number of samples to be drawn 25 | weights : np.ndarray 26 | model weights 27 | cor : float (-1, 1) 28 | correlation of the predictors 29 | 30 | Returns 31 | ------- 32 | tup 33 | with [0] feature matrix and [1] response 34 | """ 35 | 36 | ndim = len(weights) 37 | cov = (1 - cor) * np.identity(ndim) + cor * np.ones((ndim, ndim)) 38 | mean = np.zeros(ndim) 39 | 40 | X = np.random.multivariate_normal(mean, cov, nobs) 41 | e = np.random.normal(0, 1, nobs) 42 | y = np.dot(X, weights) + e 43 | 44 | return (X, y) 45 | 46 | 47 | def replicate_trial(trial, n): 48 | """Repeat a random trial n times. 49 | 50 | Parameters 51 | ---------- 52 | trial : func 53 | call to the generating function 54 | n : int {1, .., inf} 55 | number of trials 56 | 57 | Returns 58 | ------- 59 | np.ndarray 60 | where rows are trials and columns are variables 61 | """ 62 | 63 | return np.array([trial() for i in range(n)]) 64 | 65 | 66 | 67 | # set coefficients 68 | weights = np.hstack((np.zeros(10), np.arange(0.2, 1.2, 0.2))) 69 | 70 | # simulate data 71 | np.random.seed(2015) 72 | X, y = simulate_data(100, weights, 0.5) 73 | 74 | # full enumeration 75 | enumerator = linear_averaging.LinearEnumerator(X, y, 15**2, 1/3) 76 | enumerator.select() 77 | enumerator.estimate() 78 | 79 | # mcmc approximation 80 | mc3 = linear_averaging.LinearMC3(X, y, 15**2, 1/3) 81 | mc3.select(niter=10000, method="random") 82 | mc3.estimate() 83 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Bayesian Model Averaging 2 | ======================== 3 | 4 | Provides routines for Bayesian Model Averaging (BMA). BMA searches a model space (e.g. linear regression models) for promising models and computes the posterior probability distribution over that space. Coefficients are then estimated from a weighted average over the model space. 5 | 6 | Running BMA is as simple as fitting a regression model. Estimates will be close to the ones you would obtain from fitting the "true" nested model, and no knowledge of that model is required. 7 | 8 | 9 | TOC 10 | --- 11 | 12 | The following scripts are relevant for end users: 13 | - `linear_regression.py` contains routines for Bayesian linear regression. 14 | - `linear_averaging.py` contains routines for linear BMA. 15 | - `sim.py` demonstrates basic usage of linear BMA. 16 | 17 | The following scripts are useful if you wish to adapt BMA to other model spaces: 18 | - `core.py` contains routines for generic BMA. 19 | - `mcmc.py` contains generic MCMC routines. 20 | 21 | 22 | Usage 23 | ----- 24 | 25 | The specific Bayesian regression model I use expects 2 hyperparameters: 26 | - *g* is a parameter that penalizes model size. I recommend setting it to max(n_obs, n_dim^2). 27 | - *p* is your prior expectation of how many relevant variables your dataset contains. If you expected 10% of the variables in *X* to be relevant, you would set it to 1/10. 28 | 29 | Basic usage is demonstrated in `sim.py`. Given regressors *X* and response *y* You can fit the model by executing 30 | 31 | ```python 32 | mod = linear_averaging.LinearMC3(X, y, g, p) 33 | 34 | mod.select() 35 | 36 | mod.estimate() 37 | ``` 38 | 39 | The first step computes the posterior model distribution, the second computes the posterior distributions over the model parameters. 40 | 41 | Please consult the docstrings for further documentation. 42 | 43 | 44 | Dependencies 45 | ------------- 46 | 47 | All scripts were written with Python 3 in mind and require the usual set of scientific Python libraries. They can be converted to Python 2.7 with minimal changes. It is crucial to enable *true division* by adding the following line to all scripts: 48 | 49 | ```python 50 | from __future__ import division 51 | ``` 52 | 53 | 54 | References 55 | ---------- 56 | 57 | - Kass and Wassermann (1995) and Kass and Raftery (1995) for Bayesian model averaging and MC3. 58 | - Liang et al. (2008) for the Bayesian Linear Regression model. 59 | - Hastings (1970) for details on the Metropolis-Hastings algorithm. 60 | - Sokal (1997) for MCMC diagnostics. -------------------------------------------------------------------------------- /mcmc.py: -------------------------------------------------------------------------------- 1 | """ 2 | Provides routines for Markov Chain Monte Carlo (MCMC) sampling. 3 | 4 | References 5 | ---------- 6 | See Hastings (1970) for details on the Metropolis-Hastings algorithm. 7 | See Sokal (1997) for MCMC diagnostics. 8 | """ 9 | 10 | import numpy as np 11 | from collections import Counter 12 | 13 | 14 | 15 | def get_kendalltau_bin(sample1, sample2): 16 | """Calculate Kendall's tau-b in O(n) time. 17 | 18 | Parameters 19 | ---------- 20 | sample1/2 : array_like in {0, 1}^ndim 21 | binary sequences 22 | 23 | Returns 24 | ------- 25 | float 26 | Kendall's tau-b or 0 if undefined 27 | """ 28 | 29 | c = Counter() 30 | for i in range(len(sample1)): 31 | c[(sample1[i], sample2[i])] += 1 32 | 33 | conc = c[(1, 1)] * c[(0, 0)] 34 | nconc = c[(1, 0)] * c[(0, 1)] 35 | ties1 = c[(0, 0)] * c[(0, 1)] + c[(1, 1)] * c[(1, 0)] 36 | ties2 = c[(0, 0)] * c[(1, 0)] + c[(1, 1)] * c[(0, 1)] 37 | 38 | try: 39 | return (conc - nconc) / ((conc + nconc + ties1) * (conc + nconc + ties2)) ** 0.5 40 | except ZeroDivisionError: 41 | return 0 42 | 43 | 44 | def est_bin_acf(series): 45 | """Estimate the binary autocorrelation function. 46 | 47 | Parameters 48 | ---------- 49 | series : array_like in {0, 1}^d 50 | binary time series 51 | 52 | Returns 53 | ------- 54 | np.ndarray 55 | acf up to lag n // 100, starting at lag 1 56 | """ 57 | 58 | return [ 59 | get_kendalltau_bin(series[lag:], series[:-lag]) 60 | for lag in range(1, 100 + 1) 61 | ] 62 | 63 | 64 | def est_acf(series): 65 | """Estimate the autocorrelation function. 66 | 67 | Parameters 68 | ---------- 69 | series : array_like in R^d 70 | binary time series 71 | 72 | Returns 73 | ------- 74 | np.ndarray 75 | acf up to lag n // 100, starting at lag 1 76 | """ 77 | 78 | mean = np.mean(series) 79 | var = np.mean(series ** 2) - mean ** 2 80 | 81 | acf = np.array([ 82 | (np.mean(series[lag:] * series[:-lag]) - mean ** 2) / var 83 | for lag in range(1, 100 + 1) 84 | ]) 85 | 86 | acf[np.isnan(acf)] = 0 87 | return acf 88 | 89 | 90 | def est_int_autocor(acf, tradeoff_par=6): 91 | """Estimate the integrated autocorrelation time. 92 | Since there is a bias-variance tradeoff involved in the estimation, the acf is integrated up to lag l such that l is the smallest int for which 93 | l >= tradeoff_par * (0.5 + sum[t = 1][l](acf(t)) 94 | 95 | Parameters 96 | ---------- 97 | acf : array_like in (0, 1)^ndim 98 | autocorrelation function starting at lag 1 99 | tradeoff_par : int {1, .., inf}, default 6 100 | governs the bias-variance tradeoff in the estimation. A higher parameter lowers the bias and increases the variance 101 | 102 | Returns 103 | ------- 104 | float 105 | estimate of the acf's integrated autocorrelation time 106 | """ 107 | 108 | int_autocor = 0.5 109 | for i in range(len(acf)): 110 | int_autocor += acf[i] 111 | if i + 1 >= tradeoff_par * int_autocor: 112 | return int_autocor 113 | return int_autocor 114 | 115 | 116 | def est_exp_autocor(acf): 117 | """Estimate the exponential autocorrelation time. 118 | This method is very sensitive to post-decay noise in the acf. 119 | 120 | Parameters 121 | ---------- 122 | acf : array_like in (0, 1)^ndim 123 | autocorrelation function starting at lag 1 124 | 125 | Returns 126 | ------- 127 | float 128 | estimate of the acf's exponential autocorrelation time 129 | """ 130 | 131 | return np.nanmax([ 132 | (lag + 1) / -np.log(np.abs(acf[lag])) 133 | for lag in range(len(acf)) 134 | ]) 135 | 136 | 137 | 138 | class MetropolisSampler(object): 139 | """Generic implementation of the Metropolis-Hastings algorithm. 140 | 141 | Draws dependent samples from a probability distribution. 142 | 143 | Parameters 144 | ---------- 145 | rv_prob_func : func 146 | log probability measure on the random variable 147 | proposal_func : func 148 | draw from the proposal distribution given the current state 149 | proposal_prob_func : func 150 | log probability measure on the proposal distribution 151 | 152 | Attributes 153 | ---------- 154 | draws : np.ndarray 155 | array of draws from the random variable 156 | diagnostics : dict 157 | summary statistics for all dimensions including 158 | "ess" (effective sample size) 159 | "ndiscard" (estimated number of pre-equilibrium samples) 160 | "stderr" (standard error of the mean estimator) 161 | """ 162 | 163 | def __init__(self, rv_prob_func, proposal_func, proposal_prob_func): 164 | 165 | self._get_rv_prob = rv_prob_func 166 | self._propose = proposal_func 167 | self._get_proposal_prob = proposal_prob_func 168 | 169 | 170 | def _run(self, init, niter=100000): 171 | """Run the sampler. 172 | 173 | Parameters 174 | ---------- 175 | init : float or array_like in R^ndim 176 | initial state of the Markov chain 177 | niter : int {1, .., inf} 178 | number of iterations 179 | """ 180 | 181 | self.draws = np.empty((niter, len(init)), dtype=np.array(init).dtype) 182 | state = {"draw": init, "prob": self._get_rv_prob(init)} 183 | 184 | for i in range(niter): 185 | candidate = {"draw": self._propose(state["draw"])} 186 | candidate["prob"] = self._get_rv_prob(candidate["draw"]) 187 | if self._decide(state, candidate): 188 | state = candidate 189 | self.draws[i, :] = state["draw"] 190 | 191 | # discard burn-in 192 | self._diagnose() 193 | self.draws = self.draws[np.max(self.diagnostics["ndiscard"]):,:] 194 | 195 | 196 | def _decide(self, state, proposal): 197 | """Apply the Metropolis-Hastings decision rule to a candidate state. 198 | 199 | Parameters 200 | ---------- 201 | proposal : np.ndarray in R^ndim 202 | candidate MC state 203 | state : np.ndarray in R^ndim 204 | current MC state 205 | 206 | Returns 207 | ------- 208 | bool 209 | decision 210 | """ 211 | 212 | odds_ratio = proposal["prob"] - state["prob"] 213 | proposal_ratio = self._get_proposal_ratio(state, proposal) 214 | 215 | if odds_ratio + proposal_ratio > np.log(np.random.uniform()): 216 | return True 217 | else: 218 | return False 219 | 220 | 221 | def _get_proposal_ratio(self, state, candidate): 222 | """Compute the ratio of proposal probabilities. 223 | 224 | Parameters 225 | ---------- 226 | proposal : np.ndarray in R^ndim 227 | candidate MC state 228 | state : np.ndarray in R^ndim 229 | current MC state 230 | 231 | Returns 232 | ------- 233 | float 234 | log ratio 235 | """ 236 | 237 | forward_prob = self._get_proposal_prob( 238 | candidate["draw"], 239 | state["draw"] 240 | ) 241 | backward_prob = self._get_proposal_prob( 242 | state["draw"], 243 | candidate["draw"] 244 | ) 245 | 246 | return backward_prob - forward_prob 247 | 248 | 249 | def _diagnose(self): 250 | """Compute performance statistics. 251 | """ 252 | 253 | acfs = [est_acf(series) for series in self.draws.T] 254 | int_autocor = np.array([est_int_autocor(acf) for acf in acfs]) 255 | ndiscard = 20 * int_autocor 256 | ess = (self.draws.shape[0] - ndiscard) / 2 / int_autocor 257 | means = np.array([np.mean(series) for series in self.draws.T]) 258 | stderr = (np.array([np.var(series) for series in self.draws.T]) / ess) ** 0.5 259 | 260 | self.diagnostics = { 261 | "ess": ess, 262 | "ndiscard": ndiscard, 263 | "stderr": stderr 264 | } 265 | -------------------------------------------------------------------------------- /linear_regression.py: -------------------------------------------------------------------------------- 1 | """ 2 | Provides routines for Bayesian linear regression, such that 3 | E[y|x_1, ..., x_n] = a + Xb 4 | where y is the response, b are coefficients and X are predictors. 5 | 6 | The probability model is given by 7 | p[y, a, b, q] = p[y | a, b, q] * p[b | q] * p[a, q] 8 | where q is the residual precision. 9 | """ 10 | 11 | import numpy as np 12 | from scipy.optimize import brentq 13 | 14 | 15 | 16 | def draw_student(loc, scale, df, ndraws): 17 | """Draw from the multivariate student-t distribution. 18 | 19 | Parameters 20 | ---------- 21 | loc : np.ndarray in R^ndim 22 | vector of means 23 | scale : np.ndarray in R^(ndim x ndim) 24 | positive definite scale matrix 25 | df : int {1, .., inf} 26 | degrees of freedom 27 | ndraws : int {1, .., inf} 28 | number of draws 29 | 30 | Returns 31 | ------- 32 | np.ndarray 33 | matrix of draws 34 | """ 35 | 36 | scaling = np.random.gamma(df / 2, 2 / df, ndraws) ** 0.5 37 | 38 | if type(loc) != np.ndarray: 39 | return loc + np.random.normal(0, scale, ndraws) / scaling 40 | 41 | normal_draws = np.random.multivariate_normal( 42 | np.zeros(loc.shape[0]), 43 | scale, 44 | ndraws 45 | ) 46 | 47 | return loc + normal_draws / scaling[:, np.newaxis] 48 | 49 | 50 | def draw_normal_gamma(params, ndraws): 51 | """Draw from the multivariate student-t distribution. 52 | 53 | Parameters 54 | ---------- 55 | params : dict 56 | parameters including "shape", "rate", "precision", "location" 57 | ndraws : int {1, .., inf} 58 | number of draws 59 | 60 | Returns 61 | ------- 62 | dict 63 | including "gamma" and "normal" draws 64 | """ 65 | 66 | gamma_draws = np.random.gamma( 67 | params["shape"], 68 | 1 / params["rate"], 69 | ndraws 70 | ) 71 | 72 | if type(params["precision"]) != np.ndarray: 73 | normal_draws = np.array([ 74 | np.random.normal( 75 | params["location"], 76 | 1 / params["precision"] / gamma_draws[i] 77 | ) 78 | for i in range(ndraws) 79 | ]) 80 | 81 | else: 82 | normal_draws = np.array([ 83 | np.random.multivariate_normal( 84 | params["location"], 85 | np.linalg.inv(params["precision"]) / gamma_draws[i] 86 | ) 87 | for i in range(ndraws) 88 | ]) 89 | 90 | return {"gamma": gamma_draws, "normal": normal_draws} 91 | 92 | 93 | def integrate(draws, lo, hi): 94 | """Approximate the probability integral from a series of draws. 95 | 96 | Parameters 97 | ---------- 98 | draws : np.ndarray in R^nobs or R^(ndim x nobs) 99 | vector or matrix of draws from a random variable or vector 100 | lo : float or np.ndarray in R^ndim 101 | lower bound of the integral 102 | hi : float or np.ndarray in R^ndim 103 | upper bound of the integral 104 | 105 | Returns 106 | ------- 107 | float 108 | probability integral 109 | """ 110 | 111 | return np.mean(np.logical_and(lo <= draws, draws <= hi)) 112 | 113 | 114 | def get_prediction_interval(draws, alpha=0.05): 115 | """Approximate the (1 - alpha)-probability interval from a series of draws. 116 | 117 | If you pass a series of multivariate draws, the dimensions will be scaled according to their standard deviation. 118 | 119 | Parameters 120 | ---------- 121 | draws : np.ndarray in R^nobs or R^(ndim x nobs) 122 | vector or matrix of draws from a random variable or vector 123 | alpha : float (0, 1) 124 | error probability 125 | 126 | Returns 127 | ------- 128 | tup 129 | bounds of the prediction interval 130 | """ 131 | 132 | integral = lambda x, mean, std, alpha: alpha - 1 + integrate( 133 | draws, 134 | mean - x * std, 135 | mean + x * std 136 | ) 137 | mean = np.mean(draws, axis=0) 138 | std = np.var(draws, axis=0) ** 0.5 139 | offset = brentq(integral, 1 - alpha, 1 / alpha, args=(mean, std, alpha)) 140 | 141 | return (mean - offset * std, mean + offset * std) 142 | 143 | 144 | 145 | class LinearModel(object): 146 | """Computes the posterior distribution of the models' coefficients. 147 | 148 | Provides routines to draw from relevant posterior distributions. 149 | 150 | Parameters 151 | ---------- 152 | X : np.ndarray in R^(nobs x ndim) 153 | predictor matrix 154 | y : np.ndarray in R^nobs 155 | response vector 156 | penalty_par : float (0, inf) 157 | dimensionality penalty ("g") 158 | 159 | Attributes 160 | ---------- 161 | nobs : int 162 | number of observations 163 | ndim : int 164 | number of predictors 165 | X : np.ndarray 166 | prediction matrix 167 | y : np.ndarray 168 | response vector 169 | posterior: dict 170 | posterior distribution over the model parameters including 171 | "shape", "rate", "location", "precision" 172 | estimates: dict 173 | point estimate of the model parameters including 174 | "coefficients" and "res_precision" 175 | """ 176 | 177 | 178 | def __init__(self, X, y, penalty_par): 179 | 180 | self.nobs, self.ndim = X.shape 181 | self.X = X 182 | self.y = y 183 | self.penalty_par = penalty_par 184 | 185 | 186 | def estimate(self): 187 | """Compute the parameters of the posterior distribution of the model coefficients. 188 | """ 189 | 190 | design = np.hstack((np.ones((self.nobs, 1)), self.X)) 191 | sub_quadrant = (1 + 1 / self.nobs / self.penalty_par) * np.dot(self.X.T, self.X) 192 | 193 | precision = np.vstack(( 194 | np.hstack((self.nobs, np.sum(self.X, 0).T)), 195 | np.hstack((np.sum(self.X, 0, keepdims=True).T, sub_quadrant)) 196 | )) 197 | location = np.linalg.solve(precision, np.dot(design.T, self.y)) 198 | shape = (self.nobs - 1) / 2 199 | rate = 0.5 * np.dot(self.y - np.dot(design, location), self.y) 200 | 201 | self.posterior = { 202 | "shape": shape, 203 | "rate": rate, 204 | "location": location, 205 | "precision": precision 206 | } 207 | self.estimates = { 208 | "coefficients": location, 209 | "res_precision": shape / rate 210 | } 211 | 212 | 213 | def predict(self, X_new): 214 | """Give point predictions for a new set of observations. 215 | 216 | Parameters 217 | ---------- 218 | X_new : np.ndarray in R^(nobs x ndim) 219 | prediction matrix 220 | 221 | Returns 222 | ------- 223 | np.ndarray 224 | vector of point estimates 225 | """ 226 | 227 | design = np.hstack((np.ones((X_new.shape[0], 1)), X_new)) 228 | 229 | return np.dot(design, self.estimates["coefficients"]) 230 | 231 | 232 | def residual_dist(self, ndraws=1000): 233 | """Draw from the posterior distribution of the variance of residuals. 234 | 235 | Parameters 236 | ---------- 237 | ndraws : int {1, .., inf}, default 1000 238 | number of draws 239 | 240 | Retrurns 241 | -------- 242 | np.ndarray 243 | vector of draws 244 | """ 245 | 246 | return np.random.gamma( 247 | self.posterior["shape"], 248 | 1 / self.posterior["rate"], 249 | ndraws 250 | ) 251 | 252 | 253 | def coef_dist(self, ndraws=1000): 254 | """Draw from the posterior distribution of the regression coefficients. 255 | 256 | Parameters 257 | ---------- 258 | ndraws : int {1, .., inf}, default 1000 259 | number of draws 260 | 261 | Retrurns 262 | -------- 263 | np.ndarray 264 | matrix of draws 265 | """ 266 | 267 | return draw_normal_gamma(self.posterior, ndraws)["normal"] 268 | 269 | 270 | def predictive_dist(self, x_new, ndraws=1000): 271 | """Draw from the predictive distribution of a new observation. 272 | 273 | Parameters 274 | ---------- 275 | x_new : np.ndarray in R^ndim 276 | prediction vector 277 | ndraws : int {1, .., inf}, default 1000 278 | number of draws 279 | 280 | Retrurns 281 | -------- 282 | np.ndarray 283 | vector of draws 284 | """ 285 | 286 | ng_draws = draw_normal_gamma(self.posterior, ndraws) 287 | res_draws = np.random.randn(ndraws) / ng_draws["gamma"] ** 0.5 288 | 289 | return np.dot(ng_draws["normal"], np.hstack((1, x_new))) + res_draws 290 | -------------------------------------------------------------------------------- /linear_averaging.py: -------------------------------------------------------------------------------- 1 | """ 2 | Provides routines for Bayesian model choice in a linear regression context, such that 3 | E[y|x_1, ..., x_n] = a + Xb 4 | where y is the response, b are coefficients and X are predictors. 5 | 6 | The full probability model is given by 7 | p[y, a, b, q, M] = p[y | a, b, q, M] * p[b | q, M] * p[a, q] * p[M] 8 | where M is the model/hypothesis and q is the residual precision 9 | 10 | You may compute Pr[M|y] directly by finding the posterior probability of all possible models. You may also approximate it through MCMC simulation for better scaling. 11 | 12 | References 13 | ---------- 14 | See Liang et al. (2008) for linear model averaging. 15 | """ 16 | 17 | import numpy as np 18 | import core 19 | import linear_regression 20 | 21 | 22 | 23 | def log_gamma(x): 24 | """Compute the logarithm of the gamma function defined as log Gamma(x) = sum[from i = 1 to x - 1] log i 25 | 26 | Parameters 27 | ---------- 28 | x : int {1, ..., inf} 29 | parameter of the gamma function 30 | 31 | Returns 32 | ------- 33 | int 34 | value of the gamma function 35 | """ 36 | 37 | return np.sum(np.log(np.arange(1, x))) 38 | 39 | 40 | 41 | class LinearEnumerator(core.Enumerator): 42 | """Computes the posterior probability distribution over the space of linear regression models. 43 | 44 | This method computes 2^d probabilities, where d is the number of predictors. Use MC3 for larger d. 45 | 46 | Parameters 47 | ---------- 48 | X : np.ndarray in R^(nobs x ndim) 49 | predictor matrix 50 | y : np.ndarray in R^nobs 51 | response vector 52 | penalty_par : float (0, inf) 53 | dimensionality penalty ("g") 54 | incl_par : float (0, 1) 55 | prior inclusion probability ("p") 56 | the default implies a uniform prior over the model space 57 | 58 | Attributes 59 | ---------- 60 | nobs : int 61 | number of observations 62 | ndim : int 63 | number of predictors 64 | X : np.ndarray 65 | prediction matrix 66 | y : np.ndarray 67 | response vector 68 | posterior: Counter 69 | posterior distribution over the model space where 70 | str(model) is the key and the posterior probability is the value 71 | estimates: dict 72 | point estimates of the model parameters including 73 | "coefficients" and "res_precision" 74 | """ 75 | 76 | def __init__(self, X, y, penalty_par, incl_par): 77 | 78 | self.nobs, self.ndim = X.shape 79 | self.X = X 80 | self.y = y 81 | self.par = {"penalty": penalty_par, "incl": incl_par} 82 | 83 | 84 | def estimate(self): 85 | """Compute point estimates of the model parameters. 86 | """ 87 | 88 | self.estimates = { 89 | "coefficients": np.zeros(self.X.shape[1] + 1), 90 | "res_precision": 0 91 | } 92 | 93 | for model, weight in self.posterior.items(): 94 | mask = np.array(model) == 1 95 | model = linear_regression.LinearModel( 96 | self.X[:, mask], 97 | self.y, 98 | self.par["penalty"] 99 | ) 100 | model.estimate() 101 | 102 | self.estimates["coefficients"][np.hstack((True, mask))] += weight * model.estimates["coefficients"] 103 | self.estimates["res_precision"] += weight * model.estimates["res_precision"] 104 | 105 | 106 | def predict(self, X_new): 107 | """Give point predictions for a new set of observations. 108 | 109 | Parameters 110 | ---------- 111 | X_new : np.ndarray in R^(nobs x ndim) 112 | prediction matrix 113 | 114 | Returns 115 | ------- 116 | np.ndarray 117 | vector of point estimates 118 | """ 119 | 120 | design = np.hstack((np.ones((X_new.shape[0], 1)), X_new)) 121 | return np.dot(design, self.estimates["coefficients"]) 122 | 123 | 124 | def residual_dist(self, ndraws=1000): 125 | """Draw from the posterior distribution of the variance of residuals. 126 | 127 | Parameters 128 | ---------- 129 | ndraws : int {1, .., inf}, default 1000 130 | number of draws 131 | 132 | Retrurns 133 | -------- 134 | np.ndarray 135 | vector of draws 136 | """ 137 | 138 | model_draws = np.random.multinomial( 139 | ndraws, 140 | list(self.posterior.values()) 141 | ) 142 | 143 | draws = np.empty(0) 144 | for i, model in enumerate(self.posterior.keys()): 145 | 146 | if model_draws[i] == 0: 147 | continue 148 | 149 | subm = linear_regression.LinearModel( 150 | self.X[:, np.array(model) == 1], 151 | self.y, 152 | self.par["penalty"] 153 | ) 154 | subm.estimate() 155 | 156 | draws = np.append( 157 | draws, 158 | subm.residual_dist(model_draws[i]) 159 | ) 160 | 161 | return draws 162 | 163 | 164 | def predictive_dist(self, x_new, ndraws=1000): 165 | """Draw from the predictive distribution of a new observation. 166 | 167 | Parameters 168 | ---------- 169 | x_new : np.ndarray in R^ndim 170 | prediction vector 171 | ndraws : int {1, .., inf}, default 1000 172 | number of draws 173 | 174 | Retrurns 175 | -------- 176 | np.ndarray 177 | vector of draws 178 | """ 179 | 180 | model_draws = np.random.multinomial( 181 | ndraws, 182 | list(self.posterior.values()) 183 | ) 184 | 185 | draws = np.empty(0) 186 | for i, model in enumerate(self.posterior.keys()): 187 | 188 | if model_draws[i] == 0: 189 | continue 190 | 191 | subm = linear_regression.LinearModel( 192 | self.X[:, np.array(model) == 1], 193 | self.y, 194 | self.par["penalty"] 195 | ) 196 | subm.estimate() 197 | 198 | draws = np.append(draws, subm.predictive_dist( 199 | x_new[np.array(model) == 1], 200 | model_draws[i] 201 | )) 202 | 203 | return draws 204 | 205 | 206 | def _get_prior_prob(self, k, n): 207 | """Compute the prior probability of a model with k variables. 208 | 209 | Parameters 210 | ---------- 211 | k : int {1, ..., n} 212 | number of variables in the model 213 | n : int {1, ..., inf} 214 | total number of predictors 215 | 216 | Returns 217 | ------- 218 | float 219 | prior probability 220 | """ 221 | 222 | if k < 0 or n < k: 223 | return 0 224 | return self.par["incl"] ** k * (1 - self.par["incl"]) ** (n - k) 225 | 226 | 227 | def _get_likelihood(self, model): 228 | """Compute the marginal likelihood of the linear model with a g-prior on betas. 229 | 230 | Parameters 231 | ---------- 232 | model : np.ndarray in R^ndim 233 | vector of variable inclusion indicators 234 | 235 | Returns 236 | ------- 237 | float 238 | log marginal likelihood 239 | """ 240 | 241 | X = self.X[:, model == 1] 242 | y = self.y 243 | nobs, ndim = X.shape 244 | design = np.hstack((np.ones((nobs, 1)), X)) 245 | 246 | mle = np.linalg.solve(np.dot(design.T, design), np.dot(design.T, y)) 247 | residuals = y - np.dot(design, mle) 248 | rsquared = 1 - np.var(residuals) / np.var(y) 249 | 250 | return (log_gamma((nobs - 1) / 2) 251 | - (nobs - 1) / 2 * np.log(np.pi) 252 | - 0.5 * np.log(nobs) 253 | - (nobs - 1) / 2 * np.log(np.dot(residuals, residuals)) 254 | + (nobs - ndim - 1) / 2 * np.log(1 + self.par["penalty"]) 255 | - (nobs - 1) / 2 * np.log(1 + self.par["penalty"] * (1 - rsquared))) 256 | 257 | 258 | 259 | class LinearMC3(core.MC3, LinearEnumerator): 260 | """Computes the posterior probability distribution over the space of linear regression models. 261 | 262 | Suitable for high-dimensional models. 263 | 264 | Parameters 265 | ---------- 266 | X : np.ndarray in R^(nobs x ndim) 267 | predictor matrix 268 | y : np.ndarray in R^nobs 269 | response vector 270 | penalty_par : float (0, inf) 271 | dimensionality penalty ("g") 272 | incl_par : float (0, 1) 273 | prior inclusion probability ("p") 274 | the default implies a uniform prior over the model space 275 | 276 | Attributes 277 | ---------- 278 | nobs : int 279 | number of observations 280 | ndim : int 281 | number of predictors 282 | X : np.ndarray 283 | prediction matrix 284 | y : np.ndarray 285 | response vector 286 | posterior: Counter 287 | posterior distribution over the model space where 288 | str(model) is the key and the posterior probability is the value 289 | estimates: dict 290 | point estimates of the model parameters including 291 | "coefficients" and "res_precision" 292 | """ 293 | 294 | def __init__(self, X, y, penalty_par, incl_par): 295 | 296 | # wrap parent constructor 297 | LinearEnumerator.__init__(self, X, y, penalty_par, incl_par) 298 | -------------------------------------------------------------------------------- /core.py: -------------------------------------------------------------------------------- 1 | """ 2 | Provides routines for Bayesian model averaging. They compute the posterior distribution 3 | Pr[M|D] = p[D|M] * p[M] / p[D] 4 | where M is a member of some model space (e.g. linear regression models) and D is the observed data. 5 | 6 | You may compute Pr[M|D] directly by finding the posterior probability of all possible models. You may also approximate it through MCMC simulation for better scaling. 7 | 8 | You can adapt the routines to any model space by providing its marginal likelihood p[D|M] and its prior probability measure p[M]. 9 | 10 | References 11 | ---------- 12 | See Kass and Wassermann (1995) and Kass and Raftery (1995) for Bayesian model averaging and MC3. 13 | """ 14 | 15 | import numpy as np 16 | import mcmc 17 | from collections import Counter 18 | from itertools import product 19 | from scipy.special import binom 20 | 21 | 22 | 23 | def log_sum_exp(sequence): 24 | """Compute the logarithm of a sum of exponentials in a stable way. 25 | If the difference between the smallest and the largest exponential is larger than the float-64 range, the routine will drop low-order terms. 26 | 27 | Parameters 28 | ---------- 29 | sequence : array_like 30 | sequence of numbers to be exponentiated and added 31 | 32 | Returns 33 | ------- 34 | float 35 | logarithm of the sum of exponentials of the sequence's elements 36 | """ 37 | 38 | float_range = (-745, 705) 39 | lower = np.min(sequence) 40 | upper = np.max(sequence) 41 | 42 | if upper - lower < float_range[1] - float_range[0]: 43 | offset = (lower + upper) / 2 44 | else: 45 | print("Warning: Some very small terms have been dropped from a sum of exponentials to prevent overflow.") 46 | offset = upper - float_range[1] 47 | sequence = sequence[sequence - offset > float_range[0]] 48 | 49 | return offset + np.log(np.sum(np.e ** (sequence - offset))) 50 | 51 | 52 | 53 | class InputError(Exception): 54 | 55 | def __init__(self, expr, msg): 56 | self.expr = expr 57 | self.msg = msg 58 | 59 | def __str__(self): 60 | return str(self.expr) + ": " + str(self.msg) 61 | 62 | 63 | 64 | class Enumerator(object): 65 | """Generic model averaging routine. 66 | 67 | Computes the posterior distribution over the model space. Full enumeration requires the computation of 2^ndim probabilities. Thus, the method does not scale well beyond 15 dimensions. 68 | 69 | Parameters 70 | ---------- 71 | X : np.ndarray in R^(nobs x ndim) 72 | predictor matrix 73 | y : np.ndarray in R^nobs 74 | response vector 75 | likelihood_func : func 76 | function that returns the log marginal likelihood of a given model 77 | prior_func : func 78 | function that returns the prior probability of a given model 79 | 80 | Attributes 81 | ---------- 82 | nobs : int 83 | number of observations 84 | ndim : int 85 | number of predictors 86 | X : np.ndarray 87 | prediction matrix 88 | y : np.ndarray 89 | response vector 90 | posterior: Counter 91 | posterior distribution over the model space where 92 | tuple(model) is the key and the posterior probability is the value 93 | """ 94 | 95 | def __init__(self, X, y, likelihood_func, prior_func): 96 | 97 | self.X = X 98 | self.y = y 99 | self.nobs, self.ndim = X.shape 100 | 101 | self._get_likelihood = likelihood_func 102 | self._get_prior_prob = prior_func 103 | 104 | 105 | def select(self): 106 | """Compute the posterior probability distribution by enumerating all 2^ndim models. 107 | """ 108 | 109 | models = np.array(list(product((0, 1), repeat = self.ndim))) 110 | 111 | # compute model probabilities 112 | priors = np.array([ 113 | np.log(self._get_prior_prob(np.sum(model), self.ndim)) 114 | for model in models 115 | ]) 116 | likelihoods = np.array([ 117 | self._get_likelihood(model) 118 | for model in models 119 | ]) 120 | posteriors = np.e ** (priors + likelihoods - log_sum_exp(priors + likelihoods)) 121 | 122 | # summarize 123 | self.posterior = Counter({ 124 | tuple(models[i]):posteriors[i] 125 | for i in range(len(models)) 126 | }) 127 | 128 | 129 | def test_single_coefficients(self): 130 | """Evaluate the inclusion probability of single coefficients. 131 | 132 | Returns 133 | ------- 134 | np.ndarray 135 | vector of individual inclusion probabilities 136 | """ 137 | 138 | weighted_models = np.array([ 139 | weight * np.array(model) 140 | for model, weight in self.posterior.items() 141 | ]) 142 | 143 | return np.sum(weighted_models, 0) 144 | 145 | 146 | def test_joint_coefficients(self, indices): 147 | """Evaluate the joint inclusion probability of multiple coefficients. 148 | 149 | Parameters 150 | ---------- 151 | indices : array_like in {0, .., ndim - 1} 152 | indices of variables in X to be included in the test 153 | 154 | Returns 155 | ------- 156 | float 157 | joint inclusion probability 158 | """ 159 | 160 | return sum( 161 | weight 162 | for model, weight in self.posterior.items() 163 | if np.all(np.array(model)[indices]) 164 | ) 165 | 166 | 167 | def get_model_size_dist(self): 168 | """Evaluate the posterior model size distribution. 169 | 170 | Returns 171 | ------- 172 | np.ndarray 173 | (ndim + 1 x 1) posterior model size probabilities 174 | """ 175 | 176 | dist = np.zeros(self.ndim + 1) 177 | 178 | for model, weight in self.posterior.items(): 179 | dist[sum(model)] += weight 180 | 181 | return dist 182 | 183 | 184 | 185 | class MC3(Enumerator, mcmc.MetropolisSampler): 186 | """Generic model averaging routine based on the Metropolis-Hastings algorithm. 187 | 188 | Approximates the posterior distribution over the model space. Scales well to high dimensions. 189 | 190 | Parameters 191 | ---------- 192 | X : np.ndarray in R^(nobs x ndim) 193 | predictor matrix 194 | y : np.ndarray in R^nobs 195 | response vector 196 | likelihood_func : func 197 | function that returns the log marginal likelihood of a given model 198 | prior_func : func 199 | function that returns the prior probability of a given model 200 | 201 | Attributes 202 | ---------- 203 | nobs : int 204 | number of observations 205 | ndim : int 206 | number of predictors 207 | X : np.ndarray 208 | prediction matrix 209 | y : np.ndarray 210 | response vector 211 | posterior: Counter 212 | posterior distribution over the model space where 213 | tuple(model) is the key and the posterior probability is the value 214 | """ 215 | 216 | def select(self, niter=10000, method="random"): 217 | """Estimate the posterior probability distribution through MCMC simulation. 218 | 219 | Parameters 220 | ---------- 221 | niter : int {1, .., inf} 222 | number of draws from the distribution 223 | proposal : str {"random", "prior"} 224 | strategy that determines MCMC proposal probabilities 225 | """ 226 | 227 | # execute mcmc search 228 | self.method = method 229 | self._run(np.zeros(self.ndim), niter) 230 | 231 | # summarize 232 | counts = Counter() 233 | for i in range(self.draws.shape[0]): 234 | counts[tuple(self.draws[i,:])] += 1 235 | 236 | self.posterior = Counter({ 237 | key:(counts[key] / sum(counts.values())) 238 | for key in counts 239 | }) 240 | 241 | 242 | def _get_rv_prob(self, model): 243 | """Compute the posterior probability (up to the normalizing constant) of a given model. 244 | 245 | Parameters 246 | ---------- 247 | model : np.ndarray in {0, 1}^ndim 248 | vector of variable inclusion indicators 249 | 250 | Returns 251 | ------- 252 | float 253 | log posterior probability 254 | """ 255 | 256 | prior = np.log(self._get_prior_prob(np.sum(model), self.ndim)) 257 | likelihood = self._get_likelihood(model) 258 | 259 | return likelihood + prior 260 | 261 | 262 | def _propose(self, state): 263 | """Draw a candidate from the proposal distrubtion. 264 | 265 | Parameters 266 | ---------- 267 | state : np.ndarray in {0, 1}^ndim 268 | current MC state 269 | 270 | Returns 271 | ------- 272 | np.ndarray 273 | candidate vector of variable inclusion indicators 274 | """ 275 | 276 | if self.method == "prior": 277 | prob_dplus = binom( 278 | len(state), 279 | np.sum(state) + 1 280 | ) * self._get_prior_prob( 281 | np.sum(state) + 1, 282 | len(state) 283 | ) 284 | prob_dminus = binom( 285 | len(state), 286 | np.sum(state) - 1 287 | ) * self._get_prior_prob( 288 | np.sum(state) - 1, 289 | len(state) 290 | ) 291 | growth_prob = prob_dplus / (prob_dplus + prob_dminus) 292 | else: 293 | growth_prob = 1 - np.sum(state) / len(state) 294 | 295 | # decide on an action 296 | add = np.random.binomial(1, growth_prob) 297 | 298 | # pick entering/leaving variable 299 | pick = np.random.choice(np.arange(len(state))[state != add]) 300 | candidate = np.copy(state) 301 | candidate[pick] = not candidate[pick] 302 | 303 | return candidate 304 | 305 | 306 | def _get_proposal_prob(self, proposal, state): 307 | """Compute the probability of proposing "proposal" given "state". 308 | 309 | Parameters 310 | ---------- 311 | proposal : np.ndarray in {0, 1}^ndim 312 | candidate MC state 313 | state : np.ndarray in {0, 1}^ndim 314 | current MC state 315 | 316 | Returns 317 | ------- 318 | float 319 | probability of proposal 320 | """ 321 | 322 | if self.method == "prior": 323 | prob_dplus = binom( 324 | len(state), 325 | np.sum(state) + 1 326 | ) * self._get_prior_prob( 327 | np.sum(state) + 1, 328 | len(state) 329 | ) 330 | prob_dminus = binom( 331 | len(state), 332 | np.sum(state) - 1 333 | ) * self._get_prior_prob( 334 | np.sum(state) - 1, 335 | len(state) 336 | ) 337 | growth_prob = prob_dplus / (prob_dplus + prob_dminus) 338 | else: 339 | growth_prob = 1 - np.sum(state) / len(state) 340 | 341 | if np.sum(state) < np.sum(proposal): 342 | forward_prob = growth_prob / (len(state) - np.sum(state)) 343 | else: 344 | forward_prob = (1 - growth_prob) / np.sum(state) 345 | 346 | return np.log(forward_prob) 347 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | {description} 294 | Copyright (C) {year} {fullname} 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | {signature of Ty Coon}, 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | 341 | --------------------------------------------------------------------------------