├── .editorconfig ├── .gitignore ├── .travis.yml ├── AUTHORS.rst ├── CONTRIBUTING.rst ├── HISTORY.rst ├── LICENSE ├── MANIFEST.in ├── Makefile ├── README.rst ├── bayes_logistic ├── __init__.py └── bayes_logistic.py ├── docs ├── Makefile ├── authors.rst ├── conf.py ├── contributing.rst ├── history.rst ├── index.rst ├── installation.rst ├── make.bat ├── readme.rst └── usage.rst ├── notebooks ├── bayeslogistic_demo.ipynb └── parkinsons_sample.ipynb ├── requirements.txt ├── setup.cfg ├── setup.py ├── tests ├── __init__.py └── test_bayes_logistic.py └── tox.ini /.editorconfig: -------------------------------------------------------------------------------- 1 | # http://editorconfig.org 2 | 3 | root = true 4 | 5 | [*] 6 | indent_style = space 7 | indent_size = 4 8 | trim_trailing_whitespace = true 9 | insert_final_newline = true 10 | charset = utf-8 11 | end_of_line = lf 12 | 13 | [*.bat] 14 | indent_style = tab 15 | end_of_line = crlf 16 | 17 | [LICENSE] 18 | insert_final_newline = false 19 | 20 | [Makefile] 21 | indent_style = tab 22 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.py[cod] 2 | 3 | # C extensions 4 | *.so 5 | 6 | # Packages 7 | *.egg 8 | *.egg-info 9 | dist 10 | build 11 | eggs 12 | parts 13 | bin 14 | var 15 | sdist 16 | develop-eggs 17 | .installed.cfg 18 | lib 19 | lib64 20 | 21 | # Installer logs 22 | pip-log.txt 23 | 24 | # Unit test / coverage reports 25 | .coverage 26 | .tox 27 | nosetests.xml 28 | htmlcov 29 | 30 | # Translations 31 | *.mo 32 | 33 | # Mr Developer 34 | .mr.developer.cfg 35 | .project 36 | .pydevproject 37 | 38 | # Complexity 39 | output/*.html 40 | output/*/index.html 41 | 42 | # Sphinx 43 | docs/_build 44 | 45 | # Idea 46 | .idea 47 | 48 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | # Config file for automatic testing at travis-ci.org 2 | 3 | language: python 4 | 5 | python: 6 | - "3.5" 7 | - "3.4" 8 | - "2.7" 9 | 10 | # command to install dependencies, e.g. pip install -r requirements.txt --use-mirrors 11 | install: 12 | - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then 13 | wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh; 14 | else 15 | wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh; 16 | fi 17 | - bash miniconda.sh -b -p $HOME/miniconda 18 | - export PATH="$HOME/miniconda/bin:$PATH" 19 | - hash -r 20 | - conda config --set always_yes yes --set changeps1 no 21 | - conda update -q conda 22 | # Useful for debugging any issues with conda 23 | - conda info -a 24 | 25 | # Replace dep1 dep2 ... with your dependencies 26 | - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION scipy numpy 27 | - source activate test-environment 28 | - python setup.py install 29 | 30 | 31 | # command to run tests, e.g. python setup.py test 32 | script: python setup.py test 33 | -------------------------------------------------------------------------------- /AUTHORS.rst: -------------------------------------------------------------------------------- 1 | ======= 2 | Credits 3 | ======= 4 | 5 | Development Lead 6 | ---------------- 7 | 8 | * Rob Haslinger 9 | 10 | Contributors 11 | ------------ 12 | 13 | * Marius van Niekerk 14 | -------------------------------------------------------------------------------- /CONTRIBUTING.rst: -------------------------------------------------------------------------------- 1 | ============ 2 | Contributing 3 | ============ 4 | 5 | Contributions are welcome, and they are greatly appreciated! Every 6 | little bit helps, and credit will always be given. 7 | 8 | You can contribute in many ways: 9 | 10 | Types of Contributions 11 | ---------------------- 12 | 13 | Report Bugs 14 | ~~~~~~~~~~~ 15 | 16 | Report bugs at https://github.com/maxpoint/bayes_logistic/issues. 17 | 18 | If you are reporting a bug, please include: 19 | 20 | * Your operating system name and version. 21 | * Any details about your local setup that might be helpful in troubleshooting. 22 | * Detailed steps to reproduce the bug. 23 | 24 | Fix Bugs 25 | ~~~~~~~~ 26 | 27 | Look through the GitHub issues for bugs. Anything tagged with "bug" 28 | is open to whoever wants to implement it. 29 | 30 | Implement Features 31 | ~~~~~~~~~~~~~~~~~~ 32 | 33 | Look through the GitHub issues for features. Anything tagged with "feature" 34 | is open to whoever wants to implement it. 35 | 36 | Write Documentation 37 | ~~~~~~~~~~~~~~~~~~~ 38 | 39 | Bayes Logistic Regression could always use more documentation, whether as part of the 40 | official Bayes Logistic Regression docs, in docstrings, or even on the web in blog posts, 41 | articles, and such. 42 | 43 | Submit Feedback 44 | ~~~~~~~~~~~~~~~ 45 | 46 | The best way to send feedback is to file an issue at https://github.com/maxpoint/bayes_logistic/issues. 47 | 48 | If you are proposing a feature: 49 | 50 | * Explain in detail how it would work. 51 | * Keep the scope as narrow as possible, to make it easier to implement. 52 | * Remember that this is a volunteer-driven project, and that contributions 53 | are welcome :) 54 | 55 | Get Started! 56 | ------------ 57 | 58 | Ready to contribute? Here's how to set up `bayes_logistic` for local development. 59 | 60 | 1. Fork the `bayes_logistic` repo on GitHub. 61 | 2. Clone your fork locally:: 62 | 63 | $ git clone git@github.com:your_name_here/bayes_logistic.git 64 | 65 | 3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development:: 66 | 67 | $ mkvirtualenv bayes_logistic 68 | $ cd bayes_logistic/ 69 | $ python setup.py develop 70 | 71 | 4. Create a branch for local development:: 72 | 73 | $ git checkout -b name-of-your-bugfix-or-feature 74 | 75 | Now you can make your changes locally. 76 | 77 | 5. When you're done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox:: 78 | 79 | $ flake8 bayes_logistic tests 80 | $ python setup.py test 81 | $ tox 82 | 83 | To get flake8 and tox, just pip install them into your virtualenv. 84 | 85 | 6. Commit your changes and push your branch to GitHub:: 86 | 87 | $ git add . 88 | $ git commit -m "Your detailed description of your changes." 89 | $ git push origin name-of-your-bugfix-or-feature 90 | 91 | 7. Submit a pull request through the GitHub website. 92 | 93 | Pull Request Guidelines 94 | ----------------------- 95 | 96 | Before you submit a pull request, check that it meets these guidelines: 97 | 98 | 1. The pull request should include tests. 99 | 2. If the pull request adds functionality, the docs should be updated. Put 100 | your new functionality into a function with a docstring, and add the 101 | feature to the list in README.rst. 102 | 3. The pull request should work for Python 2.6, 2.7, 3.3, and 3.4, and for PyPy. Check 103 | https://travis-ci.org/MaxPoint/bayes_logistic/pull_requests 104 | and make sure that the tests pass for all supported Python versions. 105 | 106 | Tips 107 | ---- 108 | 109 | To run a subset of tests:: 110 | 111 | $ python -m unittest tests.test_bayes_logistic 112 | -------------------------------------------------------------------------------- /HISTORY.rst: -------------------------------------------------------------------------------- 1 | .. :changelog: 2 | 3 | History 4 | ------- 5 | 6 | 0.2.0 (2015-09-02) 7 | --------------------- 8 | 9 | * First release on PyPI. 10 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015 MaxPoint Interactive, Inc. 2 | 3 | All rights reserved. 4 | 5 | Redistribution and use in source and binary forms, with or without modification, are permitted provided that the 6 | following conditions are met: 7 | 8 | 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following 9 | disclaimer. 10 | 11 | 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following 12 | disclaimer in the documentation and/or other materials provided with the distribution. 13 | 14 | 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products 15 | derived from this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, 18 | INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 20 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 21 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, 22 | WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 23 | THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 24 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include AUTHORS.rst 2 | include CONTRIBUTING.rst 3 | include HISTORY.rst 4 | include LICENSE 5 | include README.rst 6 | 7 | recursive-include tests * 8 | recursive-exclude * __pycache__ 9 | recursive-exclude * *.py[co] 10 | 11 | recursive-include docs *.rst conf.py Makefile make.bat 12 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .PHONY: clean-pyc clean-build docs clean 2 | 3 | help: 4 | @echo "clean - remove all build, test, coverage and Python artifacts" 5 | @echo "clean-build - remove build artifacts" 6 | @echo "clean-pyc - remove Python file artifacts" 7 | @echo "clean-test - remove test and coverage artifacts" 8 | @echo "lint - check style with flake8" 9 | @echo "test - run tests quickly with the default Python" 10 | @echo "test-all - run tests on every Python version with tox" 11 | @echo "coverage - check code coverage quickly with the default Python" 12 | @echo "docs - generate Sphinx HTML documentation, including API docs" 13 | @echo "release - package and upload a release" 14 | @echo "dist - package" 15 | @echo "install - install the package to the active Python's site-packages" 16 | 17 | clean: clean-build clean-pyc clean-test 18 | 19 | clean-build: 20 | rm -fr build/ 21 | rm -fr dist/ 22 | rm -fr .eggs/ 23 | find . -name '*.egg-info' -exec rm -fr {} + 24 | find . -name '*.egg' -exec rm -f {} + 25 | 26 | clean-pyc: 27 | find . -name '*.pyc' -exec rm -f {} + 28 | find . -name '*.pyo' -exec rm -f {} + 29 | find . -name '*~' -exec rm -f {} + 30 | find . -name '__pycache__' -exec rm -fr {} + 31 | 32 | clean-test: 33 | rm -fr .tox/ 34 | rm -f .coverage 35 | rm -fr htmlcov/ 36 | 37 | lint: 38 | flake8 bayes_logistic tests 39 | 40 | test: 41 | python setup.py test 42 | 43 | test-all: 44 | tox 45 | 46 | coverage: 47 | coverage run --source bayes_logistic setup.py test 48 | coverage report -m 49 | coverage html 50 | open htmlcov/index.html 51 | 52 | docs: 53 | rm -f docs/bayes_logistic.rst 54 | rm -f docs/modules.rst 55 | sphinx-apidoc -o docs/ bayes_logistic 56 | $(MAKE) -C docs clean 57 | $(MAKE) -C docs html 58 | open docs/_build/html/index.html 59 | 60 | release: clean 61 | python setup.py sdist upload 62 | python setup.py bdist_wheel upload 63 | 64 | dist: clean 65 | python setup.py sdist 66 | python setup.py bdist_wheel 67 | ls -l dist 68 | 69 | install: clean 70 | python setup.py install 71 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ========================= 2 | Bayes Logistic Regression 3 | ========================= 4 | 5 | +------+-----------+ 6 | |asdasd|asdasdasd | 7 | +======+===========+ 8 | 9 | 10 | .. image:: https://img.shields.io/travis/MaxPoint/bayes_logistic.svg 11 | :target: https://travis-ci.org/MaxPoint/bayes_logistic 12 | 13 | .. image:: https://img.shields.io/pypi/v/bayes_logistic.svg 14 | :target: https://pypi.python.org/pypi/bayes_logistic 15 | 16 | .. image:: https://img.shields.io/pypi/pyversions/bayes_logistic.svg 17 | :target: https://pypi.python.org/pypi/bayes_logistic 18 | 19 | 20 | This package will fit Bayesian logistic regression models with arbitrary 21 | prior means and covariance matrices, although we work with the inverse covariance matrix which is the log-likelihood 22 | Hessian. 23 | 24 | Either the full Hessian or a diagonal approximation may be used. 25 | 26 | Individual data points may be weighted in an arbitrary manner. 27 | 28 | Finally, p-values on each fitted parameter may be calculated and this can be used 29 | for variable selection of sparse models. 30 | 31 | * Free software (BSD): |lic| 32 | * Documentation: https://bayes_logistic.readthedocs.org. 33 | * See related presentation video `here`_. 34 | 35 | .. |lic| image:: https://img.shields.io/github/license/MaxPoint/bayes_logistic.svg 36 | .. _here: http://www.opendatascience.com/conferences/rob-haslinger-at-bdf-2015-bayes_logistic-a-python-package-for-bayesian-logistic-regression/ 37 | 38 | Demo 39 | ---- 40 | 41 | `Example Notebook`_ 42 | 43 | .. _Example Notebook: http://nbviewer.ipython.org/github/MaxPoint/bayes_logistic/blob/master/notebooks/bayeslogistic_demo.ipynb 44 | -------------------------------------------------------------------------------- /bayes_logistic/__init__.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2015 MaxPoint Interactive, Inc. 2 | # 3 | # All rights reserved. 4 | # 5 | # Redistribution and use in source and binary forms, with or without modification, are permitted provided that the 6 | # following conditions are met: 7 | # 8 | # 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following 9 | # disclaimer. 10 | # 11 | # 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following 12 | # disclaimer in the documentation and/or other materials provided with the distribution. 13 | # 14 | # 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products 15 | # derived from this software without specific prior written permission. 16 | # 17 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, 18 | # INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 20 | # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 21 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, 22 | # WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 23 | # THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 24 | 25 | 26 | from __future__ import absolute_import 27 | 28 | __author__ = 'Rob Haslinger' 29 | __email__ = 'rob.haslinger@maxpoint.com' 30 | __version__ = '0.1.0' 31 | 32 | from .bayes_logistic import * -------------------------------------------------------------------------------- /bayes_logistic/bayes_logistic.py: -------------------------------------------------------------------------------- 1 | # Copyright (c) 2015 MaxPoint Interactive, Inc. 2 | # 3 | # All rights reserved. 4 | # 5 | # Redistribution and use in source and binary forms, with or without modification, are permitted provided that the 6 | # following conditions are met: 7 | # 8 | # 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following 9 | # disclaimer. 10 | # 11 | # 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following 12 | # disclaimer in the documentation and/or other materials provided with the distribution. 13 | # 14 | # 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products 15 | # derived from this software without specific prior written permission. 16 | # 17 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, 18 | # INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 20 | # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 21 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, 22 | # WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 23 | # THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 24 | 25 | 26 | from __future__ import absolute_import, print_function 27 | import numpy as np 28 | from scipy.optimize import minimize 29 | from scipy.stats import norm 30 | 31 | def logistic_prob(X, w): 32 | """ MAP (Bayes point) logistic regression probability with overflow prevention via exponent truncation 33 | 34 | Parameters 35 | ---------- 36 | X : array-like, shape (N, p) 37 | Feature matrix 38 | w : array-like, shape (p, ) 39 | Parameter vector 40 | 41 | Returns 42 | ------- 43 | 44 | pr : array-like, shape (N, ) 45 | vector of logistic regression probabilities 46 | 47 | References 48 | ---------- 49 | Chapter 8 of Murphy, K. 'Machine Learning a Probabilistic Perspective', MIT Press (2012) 50 | Chapter 4 of Bishop, C. 'Pattern Recognition and Machine Learning', Springer (2006) 51 | 52 | """ 53 | 54 | # set a truncation exponent. 55 | trunc = 8. # exp(8)/(1+exp(8)) = 0.9997 which is close enough to 1 as to not matter in most cases. 56 | 57 | # calculate argument of logit 58 | z = np.dot(X, w) 59 | 60 | # truncate to avoid numerical over/underflow 61 | z = np.clip(z, -trunc, trunc) 62 | 63 | # calculate logitstic probability 64 | pr = np.exp(z) 65 | pr = pr / (1. + pr) 66 | 67 | return pr 68 | 69 | 70 | def f_log_posterior(w, wprior, H, y, X, weights=None): 71 | """Returns negative log posterior probability. 72 | 73 | Parameters 74 | ---------- 75 | w : array-like, shape (p, ) 76 | vector of parameters at which the negative log posterior is to be evaluated 77 | wprior : array-like, shape (p, ) 78 | vector of prior means on the parameters to be fit 79 | H : array-like, shape (p, p) or (p, ) 80 | Array of prior Hessian (inverse covariance of prior distribution of parameters) 81 | y : array-like, shape (N, ) 82 | vector of binary ({0,1} responses) 83 | X : array-like, shape (N, p) 84 | array of features 85 | weights : array-like, shape (N, ) 86 | vector of data point weights. Should be within [0,1] 87 | 88 | Returns 89 | ------- 90 | neg_log_post : float 91 | negative log posterior probability 92 | 93 | References 94 | ---------- 95 | Chapter 8 of Murphy, K. 'Machine Learning a Probabilistic Perspective', MIT Press (2012) 96 | Chapter 4 of Bishop, C. 'Pattern Recognition and Machine Learning', Springer (2006) 97 | 98 | """ 99 | 100 | # fill in weights if need be 101 | if weights is None: 102 | weights = np.ones(len(np.atleast_1d(y)), ) 103 | if len(np.atleast_1d(weights)) != len(np.atleast_1d(y)): 104 | raise ValueError(' weight vector must be same length as response vector') 105 | 106 | # calculate negative log posterior 107 | 108 | eps = 1e-6 # this defined to ensure that we never take a log of zero 109 | 110 | mu = logistic_prob(X, w) 111 | 112 | if len(H.shape) == 2: 113 | neg_log_post = (- (np.dot(y.T, weights * np.log(mu + eps)) 114 | + np.dot((1. - y).T, weights * np.log(1. - mu + eps))) 115 | + 0.5 * np.dot((w - wprior).T, np.dot(H, (w - wprior)))) 116 | elif len(H.shape) == 1: 117 | neg_log_post = (- (np.dot(y.T, weights * np.log(mu + eps)) 118 | + np.dot((1. - y).T, weights * np.log(1. - mu + eps))) 119 | + 0.5 * np.dot((w - wprior).T, H * (w - wprior))) 120 | else: 121 | raise ValueError('Incompatible Hessian') 122 | 123 | return float(neg_log_post) 124 | 125 | 126 | def g_log_posterior(w, wprior, H, y, X, weights=None): 127 | """Returns gradient of the negative log posterior probability. 128 | 129 | Parameters 130 | ---------- 131 | w : array-like, shape (p, ) 132 | parameter vector at which the gradient is to be evaluated 133 | wprior : array-like, shape (p, ) 134 | array of prior means on the parameters to be fit 135 | H : array-like, shape (p, p) or (p, ) 136 | array of prior Hessian (inverse covariance of prior distribution of parameters) 137 | y : array-like, shape (N, ) 138 | array of binary ({0,1} responses) 139 | X : array-like, shape (N, p) 140 | array of features 141 | weights : array-like, shape (N, ) 142 | array of data point weights. Should be within [0,1] 143 | 144 | Returns 145 | ------- 146 | grad_log_post : array-like, shape (p, ) 147 | gradient of negative log posterior 148 | 149 | References 150 | ---------- 151 | Chapter 8 of Murphy, K. 'Machine Learning a Probabilistic Perspective', MIT Press (2012) 152 | Chapter 4 of Bishop, C. 'Pattern Recognition and Machine Learning', Springer (2006) 153 | 154 | """ 155 | 156 | # fill in weights if need be 157 | if weights is None: 158 | weights = np.ones(len(np.atleast_1d(y)), ) 159 | if len(np.atleast_1d(weights)) != len(np.atleast_1d(y)): 160 | raise ValueError(' weight vector must be same length as response vector') 161 | 162 | # calculate gradient 163 | 164 | mu_ = logistic_prob(X, w) 165 | 166 | if len(H.shape) == 2: 167 | grad_log_post = np.dot(X.T, weights * (mu_ - y)) + np.dot(H, (w - wprior)) 168 | elif len(H.shape) == 1: 169 | grad_log_post = np.dot(X.T, weights * (mu_ - y)) + H * (w - wprior) 170 | else: 171 | raise ValueError('Incompatible Hessian') 172 | 173 | return grad_log_post 174 | 175 | 176 | def g_log_posterior_small(w, wprior, H, y, X, weights=None): 177 | """Returns normalized (to 1) gradient of the negative log posterior probability. 178 | This is used for BFGS and L-BFGS-B solvers which tend to not converge unless 179 | the gradient is normalized. 180 | 181 | Parameters 182 | ---------- 183 | w : array-like, shape (p, ) 184 | parameter vector at which the gradient is to be evaluated 185 | wprior : array-like, shape (p, ) 186 | array of prior means on the parameters to be fit 187 | H : array-like, shape (p, p) or (p, ) 188 | array of prior Hessian (inverse covariance of prior distribution of parameters) 189 | y : array-like, shape (N, ) 190 | array of binary ({0,1} responses) 191 | X : array-like, shape (N, p) 192 | array of features 193 | weights : array-like, shape (N, ) 194 | array of data point weights. Should be within [0,1] 195 | 196 | Returns 197 | ------- 198 | grad_log_post : array-like, shape (p, ) 199 | normalized (to 1) gradient of negative log posterior 200 | 201 | References 202 | ---------- 203 | Chapter 8 of Murphy, K. 'Machine Learning a Probabilistic Perspective', MIT Press (2012) 204 | Chapter 4 of Bishop, C. 'Pattern Recognition and Machine Learning', Springer (2006) 205 | 206 | """ 207 | 208 | # fill in weights if need be 209 | if weights is None: 210 | weights = np.ones(len(np.atleast_1d(y)), ) 211 | if len(np.atleast_1d(weights)) != len(np.atleast_1d(y)): 212 | raise ValueError(' weight vector must be same length as response vector') 213 | 214 | # calculate gradient 215 | 216 | mu = logistic_prob(X, w) 217 | 218 | if len(H.shape) == 2: 219 | grad_log_post = np.dot(X.T, weights * (mu - y)) + np.dot(H, (w - wprior)) 220 | elif len(H.shape) == 1: 221 | grad_log_post = np.dot(X.T, weights * (mu - y)) + H * (w - wprior) 222 | else: 223 | raise ValueError('Incompatible Hessian') 224 | 225 | # normalize gradient to length 1 226 | grad_log_post = grad_log_post / np.sqrt(np.sum(grad_log_post * grad_log_post)) 227 | 228 | return grad_log_post 229 | 230 | 231 | def H_log_posterior(w, wprior, H, y, X, weights=None): 232 | """Returns Hessian (either full or diagonal) of the negative log posterior probability. 233 | 234 | Parameters 235 | ---------- 236 | w : array-like, shape (p, ) 237 | parameter vector at which the Hessian is to be evaluated 238 | wprior : array-like, shape (p, ) 239 | array of prior means on the parameters to be fit 240 | H : array-like, shape (p, p) or (p, ) 241 | array of log prior Hessian (inverse covariance of prior distribution of parameters) 242 | y : array-like, shape (N, ) 243 | array of binary ({0,1} responses) 244 | X : array-like, shape (N, p) 245 | array of features 246 | weights : array-like, shape (N, ) 247 | array of data point weights. Should be within [0,1] 248 | 249 | Returns 250 | ------- 251 | H_log_post : array-like, shape like `H` 252 | Hessian of negative log posterior 253 | 254 | References 255 | ---------- 256 | Chapter 8 of Murphy, K. 'Machine Learning a Probabilistic Perspective', MIT Press (2012) 257 | Chapter 4 of Bishop, C. 'Pattern Recognition and Machine Learning', Springer (2006) 258 | 259 | """ 260 | 261 | # fill in weights if need be 262 | if weights is None: 263 | weights = np.ones(len(np.atleast_1d(y)), ) 264 | if len(np.atleast_1d(weights)) != len(np.atleast_1d(y)): 265 | raise ValueError(' weight vector must be same length as response vector') 266 | 267 | # calculate log posterior Hessian 268 | 269 | mu = logistic_prob(X, w) 270 | 271 | S = mu * (1. - mu) * weights 272 | 273 | if len(H.shape) == 2: 274 | H_log_post = np.dot(X.T, X * S[:, np.newaxis]) + H 275 | elif len(H.shape) == 1: 276 | H_log_post = np.diag(np.dot(X.T, X * S[:, np.newaxis])) + H 277 | else: 278 | raise ValueError('Incompatible Hessian') 279 | 280 | return H_log_post 281 | 282 | 283 | def HP_log_posterior(w, q, wprior, H, y, X, weights=None): 284 | """Returns diagonal Hessian of the negative log posterior probability multiplied by an arbitrary vector. 285 | This is useful for the Newton-CG solver, particularly when we only want to store a diagonal Hessian. 286 | 287 | Parameters 288 | ---------- 289 | w : array-like, shape (p, ) 290 | parameter vector at which the Hessian is to be evaluated 291 | q : array-like, shape (p, ) 292 | arbitrary vector to multiply Hessian by 293 | wprior : array-like, shape (p, ) 294 | array of prior means on the parameters to be fit 295 | H : array-like, shape (p, ) 296 | array of diagonal log prior Hessian (inverse covariance of prior distribution of parameters) 297 | y : array-like, shape (N, ) 298 | array of binary ({0,1} responses) 299 | X : array-like, shape (N, p) 300 | array of features 301 | weights : array-like, shape (N, ) 302 | array of data point weights. Should be within [0,1] 303 | 304 | Returns 305 | ------- 306 | HP : array-like, shape (p, ) 307 | Hessian of log posterior (diagonal approx) multiplied by arbitrary vector 308 | 309 | References 310 | ---------- 311 | Chapter 8 of Murphy, K. 'Machine Learning a Probabilistic Perspective', MIT Press (2012) 312 | Chapter 4 of Bishop, C. 'Pattern Recognition and Machine Learning', Springer (2006) 313 | 314 | """ 315 | 316 | # fill in weights if need be 317 | if weights is None: 318 | weights = np.ones(len(np.atleast_1d(y)), ) 319 | if len(np.atleast_1d(weights)) != len(np.atleast_1d(y)): 320 | raise ValueError(' weight vector must be same length as response vector') 321 | 322 | HP = H_log_posterior(w, wprior, H, y, X, weights) 323 | HP = HP * q 324 | 325 | return HP 326 | 327 | 328 | def fit_bayes_logistic(y, X, wprior, H, weights=None, solver='Newton-CG', bounds=None, maxiter=100): 329 | """ Bayesian Logistic Regression Solver. Assumes Laplace (Gaussian) Approximation 330 | to the posterior of the fitted parameter vector. Uses scipy.optimize.minimize 331 | 332 | Parameters 333 | ---------- 334 | y : array-like, shape (N, ) 335 | array of binary {0,1} responses 336 | X : array-like, shape (N, p) 337 | array of features 338 | wprior : array-like, shape (p, ) 339 | array of prior means on the parameters to be fit 340 | H : array-like, shape (p, p) or (p, ) 341 | array of prior Hessian (inverse covariance of prior distribution of parameters) 342 | weights : array-like, shape (N, ) 343 | array of data point weights. Should be within [0,1] 344 | solver : string 345 | scipy optimize solver used. this should be either 'Newton-CG', 'BFGS' or 'L-BFGS-B'. 346 | The default is Newton-CG. 347 | bounds : iterable of length p 348 | a length p list (or tuple) of tuples each of length 2. 349 | This is only used if the solver is set to 'L-BFGS-B'. In that case, a tuple 350 | (lower_bound, upper_bound), both floats, is defined for each parameter. See the 351 | scipy.optimize.minimize docs for further information. 352 | maxiter : int 353 | maximum number of iterations for scipy.optimize.minimize solver. 354 | 355 | Returns 356 | ------- 357 | w_fit : array-like, shape (p, ) 358 | posterior parameters (MAP estimate) 359 | H_fit : array-like, shape like `H` 360 | posterior Hessian (Hessian of negative log posterior evaluated at MAP parameters) 361 | 362 | References 363 | ---------- 364 | Chapter 8 of Murphy, K. 'Machine Learning a Probabilistic Perspective', MIT Press (2012) 365 | Chapter 4 of Bishop, C. 'Pattern Recognition and Machine Learning', Springer (2006) 366 | 367 | """ 368 | 369 | # Check that dimensionality of inputs agrees 370 | 371 | # check X 372 | if len(X.shape) != 2: 373 | raise ValueError('X must be a N*p matrix') 374 | (nX, pX) = X.shape 375 | 376 | # check y 377 | if len(y.shape) > 1: 378 | raise ValueError('y must be a vector of shape (p, )') 379 | if len(np.atleast_1d(y)) != nX: 380 | raise ValueError('y and X do not have the same number of rows') 381 | 382 | # check wprior 383 | if len(wprior.shape) > 1: 384 | raise ValueError('prior should be a vector of shape (p, )') 385 | if len(np.atleast_1d(wprior)) != pX: 386 | raise ValueError('prior mean has incompatible length') 387 | 388 | # check H 389 | if len(H.shape) == 1: 390 | if np.atleast_1d(H).shape[0] != pX: 391 | raise ValueError('prior Hessian is diagonal but has incompatible length') 392 | elif len(H.shape) == 2: 393 | (h1,h2) = np.atleast_2d(H).shape 394 | if h1 != h2: 395 | raise ValueError('prior Hessian must either be a p*p square matrix or a vector or shape (p, ) ') 396 | if h1 != pX: 397 | raise ValueError('prior Hessian is square but has incompatible size') 398 | 399 | # fill in weights if need be 400 | if weights is None: 401 | weights = np.ones(len(np.atleast_1d(y)), ) 402 | if len(np.atleast_1d(weights)) != len(np.atleast_1d(y)): 403 | raise ValueError(' weight vector must be same length as response vector') 404 | 405 | # Do the regression 406 | 407 | if solver == 'Newton-CG': 408 | 409 | if len(H.shape) == 2: 410 | 411 | ww = minimize(f_log_posterior, wprior, args=(wprior, H, y, X, weights), jac=g_log_posterior, 412 | hess=H_log_posterior, method='Newton-CG', options={'maxiter': maxiter}) 413 | w_fit = ww.x 414 | H_fit = H_log_posterior(w_fit, wprior, H, y, X, weights) 415 | 416 | elif len(H.shape) == 1: 417 | 418 | ww = minimize(f_log_posterior, wprior, args=(wprior, H, y, X, weights), jac=g_log_posterior, 419 | hessp=HP_log_posterior, method='Newton-CG', options={'maxiter': maxiter}) 420 | w_fit = ww.x 421 | H_fit = H_log_posterior(w_fit, wprior, H, y, X, weights) 422 | 423 | else: 424 | raise ValueError(' You must either use the full Hessian or its diagonal as a vector') 425 | 426 | elif solver == 'BFGS': 427 | ww = minimize(f_log_posterior, wprior, args=(wprior, H, y, X, weights), jac=g_log_posterior_small, 428 | method='BFGS', options={'maxiter': maxiter}) 429 | w_fit = ww.x 430 | H_fit = H_log_posterior(w_fit, wprior, H, y, X, weights) 431 | 432 | elif solver == 'L-BFGS-B': 433 | ww = minimize(f_log_posterior, wprior, args=(wprior, H, y, X, weights), jac=g_log_posterior_small, 434 | method='L-BFGS-B', bounds=bounds, options={'maxiter': maxiter}) 435 | w_fit = ww.x 436 | H_fit = H_log_posterior(w_fit, wprior, H, y, X, weights) 437 | 438 | else: 439 | raise ValueError('Unknown solver specified: "{0}"'.format(solver)) 440 | 441 | return w_fit, H_fit 442 | 443 | 444 | def get_pvalues(w, H): 445 | """ Calculates p-values on fitted parameters. This can be used for variable selection by, 446 | for example, discarding every parameter with a p-value less than 0.05 (or some other cutoff) 447 | 448 | Parameters 449 | ---------- 450 | w : array-like, shape (p, ) 451 | array of posterior means on the fitted parameters 452 | H : array-like, shape (p, p) or (p, ) 453 | array of log posterior Hessian 454 | 455 | Returns 456 | ------- 457 | pvals : array-like, shape (p, ) 458 | array of p-values for each of the fitted parameters 459 | 460 | References 461 | ---------- 462 | Chapter 2 of Pawitan, Y. 'In All Likelihood', Oxford University Press (2013) 463 | Also see: Gerhard, F. 'Extraction of network topology from multi-electrode recordings: is there 464 | a small world effect', Frontiers in Computational Neuroscience (2011) for a use case of 465 | p-value based variable selection. 466 | 467 | """ 468 | 469 | # get inverse standard error of each parameter from the square root of the Hessian, 470 | # which is equal to the Fisher information 471 | if len(H.shape) == 2: 472 | inv_std_err = np.sqrt(np.diag(H)) 473 | elif len(H.shape) == 1: 474 | inv_std_err = np.sqrt(H) 475 | else: 476 | raise ValueError("Incompatible Hessian provided") 477 | 478 | # calculate Wald statistic 479 | z_ = w * inv_std_err 480 | 481 | # get p-value by comparing Wald statistic to cdf of Normal distribution 482 | pvals = 2. * (1. - norm.cdf(np.abs(z_))) 483 | 484 | return pvals 485 | 486 | 487 | def bayes_logistic_prob(X, w, H): 488 | """ Posterior predictive logistic regression probability. Uses probit approximation 489 | to the logistic regression sigmoid. Also has overflow prevention via exponent truncation. 490 | 491 | Parameters 492 | ---------- 493 | X : array-like, shape (N, p) 494 | array of covariates 495 | w : array-like, shape (p, ) 496 | array of fitted MAP parameters 497 | H : array-like, shape (p, p) or (p, ) 498 | array of log posterior Hessian (covariance matrix of fitted MAP parameters) 499 | 500 | Returns 501 | ------- 502 | pr : array-like, shape (N, ) 503 | moderated (by full distribution) logistic probability 504 | 505 | References 506 | ---------- 507 | Chapter 8 of Murphy, K. 'Machine Learning a Probabilistic Perspective', MIT Press (2012) 508 | Chapter 4 of Bishop, C. 'Pattern Recognition and Machine Learning', Springer (2006) 509 | 510 | """ 511 | 512 | # set a truncation exponent 513 | trunc = 8. # exp(8)/(1+exp(8)) = 0.9997 which is close enough to 1 as to not matter in most cases. 514 | 515 | # unmoderated argument of exponent 516 | z_a = np.dot(X, w) 517 | 518 | # find the moderation 519 | if len(H.shape) == 2: 520 | H_inv_ = np.linalg.inv(H) 521 | sig2_a = np.sum(X * np.dot(H_inv_, X.T).T, axis=1) 522 | elif len(H.shape) == 1: 523 | H_inv_ = 1. / H 524 | sig2_a = np.sum(X * (H_inv_ * X), axis=1) 525 | else: 526 | raise ValueError(' You must either use the full Hessian or its diagonal as a vector') 527 | 528 | # get the moderation factor. Implicit in here is approximating the logistic sigmoid with 529 | # a probit by setting the probit and sigmoid slopes to be equal at the origin. This is where 530 | # the factor of pi/8 comes from. 531 | kappa_sig2_a = 1. / np.sqrt(1. + 0.125 * np.pi * sig2_a) 532 | 533 | # calculate the moderated argument of the logit 534 | z = z_a * kappa_sig2_a 535 | 536 | # do a truncation to prevent exp overflow 537 | z = np.clip(z, -trunc, trunc) 538 | 539 | # get the moderated logistic probability 540 | pr = np.exp(z) 541 | pr = pr / (1. + pr) 542 | 543 | return pr 544 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = _build 9 | 10 | # User-friendly check for sphinx-build 11 | ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) 12 | $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) 13 | endif 14 | 15 | # Internal variables. 16 | PAPEROPT_a4 = -D latex_paper_size=a4 17 | PAPEROPT_letter = -D latex_paper_size=letter 18 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 19 | # the i18n builder cannot share the environment and doctrees with the others 20 | I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 21 | 22 | .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext 23 | 24 | help: 25 | @echo "Please use \`make ' where is one of" 26 | @echo " html to make standalone HTML files" 27 | @echo " dirhtml to make HTML files named index.html in directories" 28 | @echo " singlehtml to make a single large HTML file" 29 | @echo " pickle to make pickle files" 30 | @echo " json to make JSON files" 31 | @echo " htmlhelp to make HTML files and a HTML help project" 32 | @echo " qthelp to make HTML files and a qthelp project" 33 | @echo " devhelp to make HTML files and a Devhelp project" 34 | @echo " epub to make an epub" 35 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 36 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 37 | @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" 38 | @echo " text to make text files" 39 | @echo " man to make manual pages" 40 | @echo " texinfo to make Texinfo files" 41 | @echo " info to make Texinfo files and run them through makeinfo" 42 | @echo " gettext to make PO message catalogs" 43 | @echo " changes to make an overview of all changed/added/deprecated items" 44 | @echo " xml to make Docutils-native XML files" 45 | @echo " pseudoxml to make pseudoxml-XML files for display purposes" 46 | @echo " linkcheck to check all external links for integrity" 47 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 48 | 49 | clean: 50 | rm -rf $(BUILDDIR)/* 51 | 52 | html: 53 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 54 | @echo 55 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 56 | 57 | dirhtml: 58 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 59 | @echo 60 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 61 | 62 | singlehtml: 63 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 64 | @echo 65 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 66 | 67 | pickle: 68 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 69 | @echo 70 | @echo "Build finished; now you can process the pickle files." 71 | 72 | json: 73 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 74 | @echo 75 | @echo "Build finished; now you can process the JSON files." 76 | 77 | htmlhelp: 78 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 79 | @echo 80 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 81 | ".hhp project file in $(BUILDDIR)/htmlhelp." 82 | 83 | qthelp: 84 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 85 | @echo 86 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 87 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 88 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/bayes_logistic.qhcp" 89 | @echo "To view the help file:" 90 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/bayes_logistic.qhc" 91 | 92 | devhelp: 93 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 94 | @echo 95 | @echo "Build finished." 96 | @echo "To view the help file:" 97 | @echo "# mkdir -p $$HOME/.local/share/devhelp/bayes_logistic" 98 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/bayes_logistic" 99 | @echo "# devhelp" 100 | 101 | epub: 102 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 103 | @echo 104 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 105 | 106 | latex: 107 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 108 | @echo 109 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 110 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 111 | "(use \`make latexpdf' here to do that automatically)." 112 | 113 | latexpdf: 114 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 115 | @echo "Running LaTeX files through pdflatex..." 116 | $(MAKE) -C $(BUILDDIR)/latex all-pdf 117 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 118 | 119 | latexpdfja: 120 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 121 | @echo "Running LaTeX files through platex and dvipdfmx..." 122 | $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja 123 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 124 | 125 | text: 126 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 127 | @echo 128 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 129 | 130 | man: 131 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 132 | @echo 133 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 134 | 135 | texinfo: 136 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 137 | @echo 138 | @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." 139 | @echo "Run \`make' in that directory to run these through makeinfo" \ 140 | "(use \`make info' here to do that automatically)." 141 | 142 | info: 143 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 144 | @echo "Running Texinfo files through makeinfo..." 145 | make -C $(BUILDDIR)/texinfo info 146 | @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." 147 | 148 | gettext: 149 | $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale 150 | @echo 151 | @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." 152 | 153 | changes: 154 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 155 | @echo 156 | @echo "The overview file is in $(BUILDDIR)/changes." 157 | 158 | linkcheck: 159 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 160 | @echo 161 | @echo "Link check complete; look for any errors in the above output " \ 162 | "or in $(BUILDDIR)/linkcheck/output.txt." 163 | 164 | doctest: 165 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 166 | @echo "Testing of doctests in the sources finished, look at the " \ 167 | "results in $(BUILDDIR)/doctest/output.txt." 168 | 169 | xml: 170 | $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml 171 | @echo 172 | @echo "Build finished. The XML files are in $(BUILDDIR)/xml." 173 | 174 | pseudoxml: 175 | $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml 176 | @echo 177 | @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." 178 | -------------------------------------------------------------------------------- /docs/authors.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../AUTHORS.rst 2 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # bayes_logistic documentation build configuration file, created by 5 | # sphinx-quickstart on Tue Jul 9 22:26:36 2013. 6 | # 7 | # This file is execfile()d with the current directory set to its 8 | # containing dir. 9 | # 10 | # Note that not all possible configuration values are present in this 11 | # autogenerated file. 12 | # 13 | # All configuration values have a default; values that are commented out 14 | # serve to show the default. 15 | 16 | import sys 17 | import os 18 | 19 | # If extensions (or modules to document with autodoc) are in another 20 | # directory, add these directories to sys.path here. If the directory is 21 | # relative to the documentation root, use os.path.abspath to make it 22 | # absolute, like shown here. 23 | #sys.path.insert(0, os.path.abspath('.')) 24 | 25 | # Get the project root dir, which is the parent dir of this 26 | cwd = os.getcwd() 27 | project_root = os.path.dirname(cwd) 28 | 29 | # Insert the project root dir as the first element in the PYTHONPATH. 30 | # This lets us ensure that the source package is imported, and that its 31 | # version is used. 32 | sys.path.insert(0, project_root) 33 | 34 | import bayes_logistic 35 | 36 | # -- General configuration --------------------------------------------- 37 | 38 | # If your documentation needs a minimal Sphinx version, state it here. 39 | #needs_sphinx = '1.0' 40 | 41 | # Add any Sphinx extension module names here, as strings. They can be 42 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. 43 | extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode'] 44 | 45 | # Add any paths that contain templates here, relative to this directory. 46 | templates_path = ['_templates'] 47 | 48 | # The suffix of source filenames. 49 | source_suffix = '.rst' 50 | 51 | # The encoding of source files. 52 | #source_encoding = 'utf-8-sig' 53 | 54 | # The master toctree document. 55 | master_doc = 'index' 56 | 57 | # General information about the project. 58 | project = u'Bayes Logistic Regression' 59 | copyright = u'2015, MaxPoint Interactive, Inc.' 60 | 61 | # The version info for the project you're documenting, acts as replacement 62 | # for |version| and |release|, also used in various other places throughout 63 | # the built documents. 64 | # 65 | # The short X.Y version. 66 | version = bayes_logistic.__version__ 67 | # The full version, including alpha/beta/rc tags. 68 | release = bayes_logistic.__version__ 69 | 70 | # The language for content autogenerated by Sphinx. Refer to documentation 71 | # for a list of supported languages. 72 | #language = None 73 | 74 | # There are two options for replacing |today|: either, you set today to 75 | # some non-false value, then it is used: 76 | #today = '' 77 | # Else, today_fmt is used as the format for a strftime call. 78 | #today_fmt = '%B %d, %Y' 79 | 80 | # List of patterns, relative to source directory, that match files and 81 | # directories to ignore when looking for source files. 82 | exclude_patterns = ['_build'] 83 | 84 | # The reST default role (used for this markup: `text`) to use for all 85 | # documents. 86 | #default_role = None 87 | 88 | # If true, '()' will be appended to :func: etc. cross-reference text. 89 | #add_function_parentheses = True 90 | 91 | # If true, the current module name will be prepended to all description 92 | # unit titles (such as .. function::). 93 | #add_module_names = True 94 | 95 | # If true, sectionauthor and moduleauthor directives will be shown in the 96 | # output. They are ignored by default. 97 | #show_authors = False 98 | 99 | # The name of the Pygments (syntax highlighting) style to use. 100 | pygments_style = 'sphinx' 101 | 102 | # A list of ignored prefixes for module index sorting. 103 | #modindex_common_prefix = [] 104 | 105 | # If true, keep warnings as "system message" paragraphs in the built 106 | # documents. 107 | #keep_warnings = False 108 | 109 | 110 | # -- Options for HTML output ------------------------------------------- 111 | 112 | # The theme to use for HTML and HTML Help pages. See the documentation for 113 | # a list of builtin themes. 114 | html_theme = 'default' 115 | 116 | # Theme options are theme-specific and customize the look and feel of a 117 | # theme further. For a list of options available for each theme, see the 118 | # documentation. 119 | #html_theme_options = {} 120 | 121 | # Add any paths that contain custom themes here, relative to this directory. 122 | #html_theme_path = [] 123 | 124 | # The name for this set of Sphinx documents. If None, it defaults to 125 | # " v documentation". 126 | #html_title = None 127 | 128 | # A shorter title for the navigation bar. Default is the same as 129 | # html_title. 130 | #html_short_title = None 131 | 132 | # The name of an image file (relative to this directory) to place at the 133 | # top of the sidebar. 134 | #html_logo = None 135 | 136 | # The name of an image file (within the static path) to use as favicon 137 | # of the docs. This file should be a Windows icon file (.ico) being 138 | # 16x16 or 32x32 pixels large. 139 | #html_favicon = None 140 | 141 | # Add any paths that contain custom static files (such as style sheets) 142 | # here, relative to this directory. They are copied after the builtin 143 | # static files, so a file named "default.css" will overwrite the builtin 144 | # "default.css". 145 | html_static_path = ['_static'] 146 | 147 | # If not '', a 'Last updated on:' timestamp is inserted at every page 148 | # bottom, using the given strftime format. 149 | #html_last_updated_fmt = '%b %d, %Y' 150 | 151 | # If true, SmartyPants will be used to convert quotes and dashes to 152 | # typographically correct entities. 153 | #html_use_smartypants = True 154 | 155 | # Custom sidebar templates, maps document names to template names. 156 | #html_sidebars = {} 157 | 158 | # Additional templates that should be rendered to pages, maps page names 159 | # to template names. 160 | #html_additional_pages = {} 161 | 162 | # If false, no module index is generated. 163 | #html_domain_indices = True 164 | 165 | # If false, no index is generated. 166 | #html_use_index = True 167 | 168 | # If true, the index is split into individual pages for each letter. 169 | #html_split_index = False 170 | 171 | # If true, links to the reST sources are added to the pages. 172 | #html_show_sourcelink = True 173 | 174 | # If true, "Created using Sphinx" is shown in the HTML footer. 175 | # Default is True. 176 | #html_show_sphinx = True 177 | 178 | # If true, "(C) Copyright ..." is shown in the HTML footer. 179 | # Default is True. 180 | #html_show_copyright = True 181 | 182 | # If true, an OpenSearch description file will be output, and all pages 183 | # will contain a tag referring to it. The value of this option 184 | # must be the base URL from which the finished HTML is served. 185 | #html_use_opensearch = '' 186 | 187 | # This is the file name suffix for HTML files (e.g. ".xhtml"). 188 | #html_file_suffix = None 189 | 190 | # Output file base name for HTML help builder. 191 | htmlhelp_basename = 'bayes_logisticdoc' 192 | 193 | 194 | # -- Options for LaTeX output ------------------------------------------ 195 | 196 | latex_elements = { 197 | # The paper size ('letterpaper' or 'a4paper'). 198 | #'papersize': 'letterpaper', 199 | 200 | # The font size ('10pt', '11pt' or '12pt'). 201 | #'pointsize': '10pt', 202 | 203 | # Additional stuff for the LaTeX preamble. 204 | #'preamble': '', 205 | } 206 | 207 | # Grouping the document tree into LaTeX files. List of tuples 208 | # (source start file, target name, title, author, documentclass 209 | # [howto/manual]). 210 | latex_documents = [ 211 | ('index', 'bayes_logistic.tex', 212 | u'Bayes Logistic Regression Documentation', 213 | u'Rob Haslinger', 'manual'), 214 | ] 215 | 216 | # The name of an image file (relative to this directory) to place at 217 | # the top of the title page. 218 | #latex_logo = None 219 | 220 | # For "manual" documents, if this is true, then toplevel headings 221 | # are parts, not chapters. 222 | #latex_use_parts = False 223 | 224 | # If true, show page references after internal links. 225 | #latex_show_pagerefs = False 226 | 227 | # If true, show URL addresses after external links. 228 | #latex_show_urls = False 229 | 230 | # Documents to append as an appendix to all manuals. 231 | #latex_appendices = [] 232 | 233 | # If false, no module index is generated. 234 | #latex_domain_indices = True 235 | 236 | 237 | # -- Options for manual page output ------------------------------------ 238 | 239 | # One entry per manual page. List of tuples 240 | # (source start file, name, description, authors, manual section). 241 | man_pages = [ 242 | ('index', 'bayes_logistic', 243 | u'Bayes Logistic Regression Documentation', 244 | [u'Rob Haslinger'], 1) 245 | ] 246 | 247 | # If true, show URL addresses after external links. 248 | #man_show_urls = False 249 | 250 | 251 | # -- Options for Texinfo output ---------------------------------------- 252 | 253 | # Grouping the document tree into Texinfo files. List of tuples 254 | # (source start file, target name, title, author, 255 | # dir menu entry, description, category) 256 | texinfo_documents = [ 257 | ('index', 'bayes_logistic', 258 | u'Bayes Logistic Regression Documentation', 259 | u'Rob Haslinger', 260 | 'bayes_logistic', 261 | 'One line description of project.', 262 | 'Miscellaneous'), 263 | ] 264 | 265 | # Documents to append as an appendix to all manuals. 266 | #texinfo_appendices = [] 267 | 268 | # If false, no module index is generated. 269 | #texinfo_domain_indices = True 270 | 271 | # How to display URL addresses: 'footnote', 'no', or 'inline'. 272 | #texinfo_show_urls = 'footnote' 273 | 274 | # If true, do not generate a @detailmenu in the "Top" node's menu. 275 | #texinfo_no_detailmenu = False 276 | -------------------------------------------------------------------------------- /docs/contributing.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../CONTRIBUTING.rst 2 | -------------------------------------------------------------------------------- /docs/history.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../HISTORY.rst 2 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. bayes_logistic documentation master file, created by 2 | sphinx-quickstart on Tue Jul 9 22:26:36 2013. 3 | You can adapt this file completely to your liking, but it should at least 4 | contain the root `toctree` directive. 5 | 6 | Welcome to Bayes Logistic Regression's documentation! 7 | ====================================== 8 | 9 | Contents: 10 | 11 | .. toctree:: 12 | :maxdepth: 2 13 | 14 | readme 15 | installation 16 | usage 17 | contributing 18 | authors 19 | history 20 | 21 | Indices and tables 22 | ================== 23 | 24 | * :ref:`genindex` 25 | * :ref:`modindex` 26 | * :ref:`search` 27 | 28 | -------------------------------------------------------------------------------- /docs/installation.rst: -------------------------------------------------------------------------------- 1 | ============ 2 | Installation 3 | ============ 4 | 5 | At the command line:: 6 | 7 | $ easy_install bayes_logistic 8 | 9 | Or, if you have virtualenvwrapper installed:: 10 | 11 | $ mkvirtualenv bayes_logistic 12 | $ pip install bayes_logistic 13 | -------------------------------------------------------------------------------- /docs/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | REM Command file for Sphinx documentation 4 | 5 | if "%SPHINXBUILD%" == "" ( 6 | set SPHINXBUILD=sphinx-build 7 | ) 8 | set BUILDDIR=_build 9 | set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% . 10 | set I18NSPHINXOPTS=%SPHINXOPTS% . 11 | if NOT "%PAPER%" == "" ( 12 | set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% 13 | set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% 14 | ) 15 | 16 | if "%1" == "" goto help 17 | 18 | if "%1" == "help" ( 19 | :help 20 | echo.Please use `make ^` where ^ is one of 21 | echo. html to make standalone HTML files 22 | echo. dirhtml to make HTML files named index.html in directories 23 | echo. singlehtml to make a single large HTML file 24 | echo. pickle to make pickle files 25 | echo. json to make JSON files 26 | echo. htmlhelp to make HTML files and a HTML help project 27 | echo. qthelp to make HTML files and a qthelp project 28 | echo. devhelp to make HTML files and a Devhelp project 29 | echo. epub to make an epub 30 | echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter 31 | echo. text to make text files 32 | echo. man to make manual pages 33 | echo. texinfo to make Texinfo files 34 | echo. gettext to make PO message catalogs 35 | echo. changes to make an overview over all changed/added/deprecated items 36 | echo. xml to make Docutils-native XML files 37 | echo. pseudoxml to make pseudoxml-XML files for display purposes 38 | echo. linkcheck to check all external links for integrity 39 | echo. doctest to run all doctests embedded in the documentation if enabled 40 | goto end 41 | ) 42 | 43 | if "%1" == "clean" ( 44 | for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i 45 | del /q /s %BUILDDIR%\* 46 | goto end 47 | ) 48 | 49 | 50 | %SPHINXBUILD% 2> nul 51 | if errorlevel 9009 ( 52 | echo. 53 | echo.The 'sphinx-build' command was not found. Make sure you have Sphinx 54 | echo.installed, then set the SPHINXBUILD environment variable to point 55 | echo.to the full path of the 'sphinx-build' executable. Alternatively you 56 | echo.may add the Sphinx directory to PATH. 57 | echo. 58 | echo.If you don't have Sphinx installed, grab it from 59 | echo.http://sphinx-doc.org/ 60 | exit /b 1 61 | ) 62 | 63 | if "%1" == "html" ( 64 | %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html 65 | if errorlevel 1 exit /b 1 66 | echo. 67 | echo.Build finished. The HTML pages are in %BUILDDIR%/html. 68 | goto end 69 | ) 70 | 71 | if "%1" == "dirhtml" ( 72 | %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml 73 | if errorlevel 1 exit /b 1 74 | echo. 75 | echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. 76 | goto end 77 | ) 78 | 79 | if "%1" == "singlehtml" ( 80 | %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml 81 | if errorlevel 1 exit /b 1 82 | echo. 83 | echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. 84 | goto end 85 | ) 86 | 87 | if "%1" == "pickle" ( 88 | %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle 89 | if errorlevel 1 exit /b 1 90 | echo. 91 | echo.Build finished; now you can process the pickle files. 92 | goto end 93 | ) 94 | 95 | if "%1" == "json" ( 96 | %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json 97 | if errorlevel 1 exit /b 1 98 | echo. 99 | echo.Build finished; now you can process the JSON files. 100 | goto end 101 | ) 102 | 103 | if "%1" == "htmlhelp" ( 104 | %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp 105 | if errorlevel 1 exit /b 1 106 | echo. 107 | echo.Build finished; now you can run HTML Help Workshop with the ^ 108 | .hhp project file in %BUILDDIR%/htmlhelp. 109 | goto end 110 | ) 111 | 112 | if "%1" == "qthelp" ( 113 | %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp 114 | if errorlevel 1 exit /b 1 115 | echo. 116 | echo.Build finished; now you can run "qcollectiongenerator" with the ^ 117 | .qhcp project file in %BUILDDIR%/qthelp, like this: 118 | echo.^> qcollectiongenerator %BUILDDIR%\qthelp\bayes_logistic.qhcp 119 | echo.To view the help file: 120 | echo.^> assistant -collectionFile %BUILDDIR%\qthelp\bayes_logistic.ghc 121 | goto end 122 | ) 123 | 124 | if "%1" == "devhelp" ( 125 | %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp 126 | if errorlevel 1 exit /b 1 127 | echo. 128 | echo.Build finished. 129 | goto end 130 | ) 131 | 132 | if "%1" == "epub" ( 133 | %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub 134 | if errorlevel 1 exit /b 1 135 | echo. 136 | echo.Build finished. The epub file is in %BUILDDIR%/epub. 137 | goto end 138 | ) 139 | 140 | if "%1" == "latex" ( 141 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 142 | if errorlevel 1 exit /b 1 143 | echo. 144 | echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. 145 | goto end 146 | ) 147 | 148 | if "%1" == "latexpdf" ( 149 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 150 | cd %BUILDDIR%/latex 151 | make all-pdf 152 | cd %BUILDDIR%/.. 153 | echo. 154 | echo.Build finished; the PDF files are in %BUILDDIR%/latex. 155 | goto end 156 | ) 157 | 158 | if "%1" == "latexpdfja" ( 159 | %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex 160 | cd %BUILDDIR%/latex 161 | make all-pdf-ja 162 | cd %BUILDDIR%/.. 163 | echo. 164 | echo.Build finished; the PDF files are in %BUILDDIR%/latex. 165 | goto end 166 | ) 167 | 168 | if "%1" == "text" ( 169 | %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text 170 | if errorlevel 1 exit /b 1 171 | echo. 172 | echo.Build finished. The text files are in %BUILDDIR%/text. 173 | goto end 174 | ) 175 | 176 | if "%1" == "man" ( 177 | %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man 178 | if errorlevel 1 exit /b 1 179 | echo. 180 | echo.Build finished. The manual pages are in %BUILDDIR%/man. 181 | goto end 182 | ) 183 | 184 | if "%1" == "texinfo" ( 185 | %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo 186 | if errorlevel 1 exit /b 1 187 | echo. 188 | echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. 189 | goto end 190 | ) 191 | 192 | if "%1" == "gettext" ( 193 | %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale 194 | if errorlevel 1 exit /b 1 195 | echo. 196 | echo.Build finished. The message catalogs are in %BUILDDIR%/locale. 197 | goto end 198 | ) 199 | 200 | if "%1" == "changes" ( 201 | %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes 202 | if errorlevel 1 exit /b 1 203 | echo. 204 | echo.The overview file is in %BUILDDIR%/changes. 205 | goto end 206 | ) 207 | 208 | if "%1" == "linkcheck" ( 209 | %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck 210 | if errorlevel 1 exit /b 1 211 | echo. 212 | echo.Link check complete; look for any errors in the above output ^ 213 | or in %BUILDDIR%/linkcheck/output.txt. 214 | goto end 215 | ) 216 | 217 | if "%1" == "doctest" ( 218 | %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest 219 | if errorlevel 1 exit /b 1 220 | echo. 221 | echo.Testing of doctests in the sources finished, look at the ^ 222 | results in %BUILDDIR%/doctest/output.txt. 223 | goto end 224 | ) 225 | 226 | if "%1" == "xml" ( 227 | %SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml 228 | if errorlevel 1 exit /b 1 229 | echo. 230 | echo.Build finished. The XML files are in %BUILDDIR%/xml. 231 | goto end 232 | ) 233 | 234 | if "%1" == "pseudoxml" ( 235 | %SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml 236 | if errorlevel 1 exit /b 1 237 | echo. 238 | echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml. 239 | goto end 240 | ) 241 | 242 | :end 243 | -------------------------------------------------------------------------------- /docs/readme.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../README.rst 2 | -------------------------------------------------------------------------------- /docs/usage.rst: -------------------------------------------------------------------------------- 1 | ======== 2 | Usage 3 | ======== 4 | 5 | To use Bayes Logistic Regression in a project:: 6 | 7 | import bayes_logistic 8 | 9 | 10 | 11 | Methods 12 | ------- 13 | 14 | .. currentmodule:: bayes_logistic 15 | 16 | .. autofunction:: bayes_logistic_prob 17 | 18 | .. autofunction:: get_pvalues 19 | 20 | .. autofunction:: fit_bayes_logistic 21 | -------------------------------------------------------------------------------- /notebooks/parkinsons_sample.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 37, 6 | "metadata": { 7 | "collapsed": true, 8 | "deletable": true, 9 | "editable": true 10 | }, 11 | "outputs": [], 12 | "source": [ 13 | "import numpy as np\n", 14 | "import copy\n", 15 | "import bayes_logistic as bl\n", 16 | "import pandas as pd" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 38, 22 | "metadata": { 23 | "collapsed": true, 24 | "deletable": true, 25 | "editable": true 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "# Download and process data\n", 30 | "\n", 31 | "url=\"http://archive.ics.uci.edu/ml/machine-learning-databases/parkinsons/parkinsons.data\"\n", 32 | "df = pd.read_csv(url)\n", 33 | "y = df.values[:, 17] \n", 34 | "y = y.astype(int)\n", 35 | "X = np.delete(df.values, 17, 1)\n", 36 | "X = np.delete(X, 0, 1)\n", 37 | "\n", 38 | "n_samples, n_features = X.shape\n", 39 | "\n", 40 | "# Add bias column to the feature matrix\n", 41 | "B = np.ones((n_samples, n_features + 1)) \n", 42 | "B[:, 1:] = X \n", 43 | "X = B \n" 44 | ] 45 | }, 46 | { 47 | "cell_type": "code", 48 | "execution_count": 39, 49 | "metadata": { 50 | "collapsed": true, 51 | "deletable": true, 52 | "editable": true 53 | }, 54 | "outputs": [], 55 | "source": [ 56 | "# Perform feature scaling using mean normalization\n", 57 | "for col in range(1, n_features):\n", 58 | " v = X[:, col]\n", 59 | " mean = v.mean()\n", 60 | " std = v.std()\n", 61 | " X[:, col] = (X[:, col] - mean)/std" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": 40, 67 | "metadata": { 68 | "collapsed": true, 69 | "deletable": true, 70 | "editable": true 71 | }, 72 | "outputs": [], 73 | "source": [ 74 | "# The data is divided into training and test sets\n", 75 | "TRAINING_PERCENTAGE = 0.7 \n", 76 | "TEST_PERCENTAGE = 0.3 \n", 77 | "\n", 78 | "training_cnt = int(n_samples*TRAINING_PERCENTAGE)\n", 79 | "training_X = X[:training_cnt,:]\n", 80 | "training_y = y[:training_cnt]\n", 81 | "\n", 82 | "test_cnt = int(n_samples*TEST_PERCENTAGE)\n", 83 | "test_X = X[training_cnt:,:]\n", 84 | "test_y = y[training_cnt:]\n" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": 41, 90 | "metadata": { 91 | "collapsed": true, 92 | "deletable": true, 93 | "editable": true 94 | }, 95 | "outputs": [], 96 | "source": [ 97 | "# Train the model \n", 98 | "w_prior = np.zeros(n_features + 1)\n", 99 | "H_prior = np.diag(np.ones(n_features + 1))*0.001\n", 100 | "\n", 101 | "GD_BATCH_SIZE = training_cnt\n", 102 | "ITERATION_CNT = 5\n", 103 | "w = training_X.shape[1]\n", 104 | "w_prior = np.zeros(w)\n", 105 | "H_prior = np.diag(np.ones(w))*0.001\n", 106 | "\n", 107 | "for i in range(0, ITERATION_CNT):\n", 108 | " for idx in range(0, training_cnt, GD_BATCH_SIZE):\n", 109 | " batch_size = GD_BATCH_SIZE if (idx + GD_BATCH_SIZE) < training_cnt else training_cnt - idx\n", 110 | " w_posterior, H_posterior = bl.fit_bayes_logistic(training_y[idx:batch_size],\n", 111 | " training_X[idx:batch_size,:],\n", 112 | " w_prior, H_prior, solver = 'BFGS')\n", 113 | " w_prior = copy.copy(w_posterior)\n", 114 | " H_prior = copy.copy(H_posterior)\n", 115 | "\n", 116 | "w_fit = w_prior\n", 117 | "H_fit = H_prior" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": 42, 123 | "metadata": { 124 | "collapsed": false, 125 | "deletable": true, 126 | "editable": true 127 | }, 128 | "outputs": [ 129 | { 130 | "name": "stdout", 131 | "output_type": "stream", 132 | "text": [ 133 | "Prediction Accuracy for test set 60.34\n" 134 | ] 135 | } 136 | ], 137 | "source": [ 138 | "# Perform Test\n", 139 | "y_cnt = 0\n", 140 | "test_probs = bl.bayes_logistic_prob(test_X, w_fit, H_fit)\n", 141 | "prediction_cnt = 0\n", 142 | "for idx in range(0, test_cnt):\n", 143 | " if test_probs[idx] > 0.5 and test_y[idx] == 1:\n", 144 | " prediction_cnt += 1\n", 145 | " y_cnt += 1\n", 146 | "\n", 147 | "prediction_accuracy = (100.0*prediction_cnt)/y_cnt\n", 148 | "print \"Prediction Accuracy for test set %.02f\" % prediction_accuracy" 149 | ] 150 | } 151 | ], 152 | "metadata": { 153 | "kernelspec": { 154 | "display_name": "Python 2", 155 | "language": "python", 156 | "name": "python2" 157 | }, 158 | "language_info": { 159 | "codemirror_mode": { 160 | "name": "ipython", 161 | "version": 2 162 | }, 163 | "file_extension": ".py", 164 | "mimetype": "text/x-python", 165 | "name": "python", 166 | "nbconvert_exporter": "python", 167 | "pygments_lexer": "ipython2", 168 | "version": "2.7.12" 169 | } 170 | }, 171 | "nbformat": 4, 172 | "nbformat_minor": 2 173 | } 174 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | scipy 2 | numpy 3 | wheel==0.23.0 4 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [wheel] 2 | universal = 1 3 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | # Copyright (c) 2015 MaxPoint Interactive, Inc. 5 | # 6 | # All rights reserved. 7 | # 8 | # Redistribution and use in source and binary forms, with or without modification, are permitted provided that the 9 | # following conditions are met: 10 | # 11 | # 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following 12 | # disclaimer. 13 | # 14 | # 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following 15 | # disclaimer in the documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products 18 | # derived from this software without specific prior written permission. 19 | # 20 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, 21 | # INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 22 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 23 | # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, 25 | # WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 26 | # THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 27 | 28 | try: 29 | from setuptools import setup 30 | except ImportError: 31 | from distutils.core import setup 32 | 33 | 34 | with open('README.rst') as readme_file: 35 | readme = readme_file.read() 36 | 37 | with open('HISTORY.rst') as history_file: 38 | history = history_file.read().replace('.. :changelog:', '') 39 | 40 | requirements = [ 41 | "numpy", "scipy" 42 | ] 43 | 44 | test_requirements = [ 45 | "numpy", "scipy" 46 | ] 47 | 48 | setup( 49 | name='bayes_logistic', 50 | version='0.2.0', 51 | description="Bayesian Logistic Regression using Laplace approximations to the posterior.", 52 | long_description=readme + '\n\n' + history, 53 | author="MaxPoint Interactive, Inc.", 54 | author_email='rob.haslinger@maxpoint.com', 55 | url='https://github.com/maxpoint/bayes_logistic', 56 | packages=[ 57 | 'bayes_logistic', 58 | ], 59 | package_dir={'bayes_logistic': 60 | 'bayes_logistic'}, 61 | include_package_data=True, 62 | install_requires=requirements, 63 | license="BSD", 64 | zip_safe=False, 65 | keywords='bayes_logistic', 66 | classifiers=[ 67 | 'Development Status :: 4 - Beta', 68 | 'Intended Audience :: Developers', 69 | 'License :: OSI Approved :: BSD License', 70 | 'Natural Language :: English', 71 | "Programming Language :: Python :: 2", 72 | 'Programming Language :: Python :: 2.6', 73 | 'Programming Language :: Python :: 2.7', 74 | 'Programming Language :: Python :: 3', 75 | 'Programming Language :: Python :: 3.3', 76 | 'Programming Language :: Python :: 3.4', 77 | ], 78 | test_suite='tests', 79 | tests_require=test_requirements 80 | ) 81 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Copyright (c) 2015 MaxPoint Interactive, Inc. 4 | # 5 | # All rights reserved. 6 | # 7 | # Redistribution and use in source and binary forms, with or without modification, are permitted provided that the 8 | # following conditions are met: 9 | # 10 | # 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following 11 | # disclaimer. 12 | # 13 | # 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following 14 | # disclaimer in the documentation and/or other materials provided with the distribution. 15 | # 16 | # 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products 17 | # derived from this software without specific prior written permission. 18 | # 19 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, 20 | # INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 21 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 22 | # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 23 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, 24 | # WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 25 | # THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /tests/test_bayes_logistic.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | # Copyright (c) 2015 MaxPoint Interactive, Inc. 5 | # 6 | # All rights reserved. 7 | # 8 | # Redistribution and use in source and binary forms, with or without modification, are permitted provided that the 9 | # following conditions are met: 10 | # 11 | # 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following 12 | # disclaimer. 13 | # 14 | # 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following 15 | # disclaimer in the documentation and/or other materials provided with the distribution. 16 | # 17 | # 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products 18 | # derived from this software without specific prior written permission. 19 | # 20 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, 21 | # INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 22 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 23 | # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 24 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, 25 | # WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 26 | # THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 27 | 28 | 29 | """ 30 | test_bayes_logistic 31 | ---------------------------------- 32 | 33 | Tests for `bayes_logistic` module. 34 | """ 35 | 36 | import unittest 37 | 38 | from bayes_logistic import bayes_logistic 39 | 40 | class TestBayes_logistic(unittest.TestCase): 41 | 42 | def setUp(self): 43 | pass 44 | 45 | def test_something(self): 46 | pass 47 | 48 | def tearDown(self): 49 | pass 50 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py26, py27, py33, py34 3 | 4 | [testenv] 5 | setenv = 6 | PYTHONPATH = {toxinidir}:{toxinidir}/bayes_logistic 7 | commands = python setup.py test 8 | deps = 9 | -r{toxinidir}/requirements.txt 10 | --------------------------------------------------------------------------------