└── README.md /README.md: -------------------------------------------------------------------------------- 1 | ## Implementations of Uncertainty Quantification Methods for Neural Networks 2 | This repository collects available implementations for uncertainty quantification and calibration methods in neural networks. The list mainly links to resources which are provided by the authors of the corresponding approaches or to other collections of implementations. 3 | 4 | | Method Type |             Approch            | Paper(s) | Authors official Repository | Framework | Link to Repository 5 | | --- | --- | --- | --- | --- | --- | 6 | Calibration | Temperature Scaling | C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, "On calibration of modern neural networks", International Conference on Machine Learning (2017). [[paper](https://dl.acm.org/doi/pdf/10.5555/3305381.3305518)] | yes | PyTorch | https://github.com/gpleiss/temperature_scaling 7 | Single Deterministic Neural Network | Misclassification and Out-of-Distribution detection based on the softmax output. | D. Hendrycks and K. Gimpel. "A baseline for detecting misclassified and out-of-distribution examples in neural networks." arXiv preprint arXiv:1610.02136 (2016). [[paper](https://arxiv.org/abs/1610.02136)] | yes | TensorFlow | https://github.com/hendrycks/error-detection 8 | Single Deterministic Neural Network & Calibration| Implementation of Odin and Temperature Scalling for out-of-distribution detection. |S. Liang, Y. Li and R. Srikant, "Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks", International Conference on Learning Representations (2018). [[paper](https://openreview.net/pdf?id=H1VGkIxRZ)] | yes | PyTorch | https://github.com/facebookresearch/odin 9 | | Single Deterministic Neural Network | Outlier exposure - Learn heuristic for out-of-distribution detection from out-of-distribution examples. | Hendrycks, Dan, Mantas Mazeika, and Thomas Dietterich. "Deep anomaly detection with outlier exposure." arXiv preprint arXiv:1812.04606 (2018). [[paper](https://arxiv.org/abs/1812.04606)]| yes | PyTorch | https://github.com/hendrycks/outlier-exposure 10 | | Single Deterministic Neural Network | Deep Mahalanobis Detector for Out-of-Dsitribution detection| K. Lee, K. Lee, H. Lee, J. Shin, “A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks”, NeurIPS 2018. [[paper](https://papers.nips.cc/paper/2018/file/abdeb6f575ac5c6676b747bca8d09cc2-Paper.pdf)] | yes | PyTorch | https://github.com/pokaxpoka/deep_Mahalanobis_detector 11 | | Single Deterministic Neural Network | Evidential Neural Networks for Classification tasks | M. Sensoy, L. Kaplan, and M. Kandemir, “Evidential deep learning to quantify classification uncertainty” , NeurIPS 2018. [[paper](https://papers.nips.cc/paper/2018/file/a981f2b708044d6fb4a71a1463242520-Paper.pdf)] | partially | Tensorflow & PyTorch | https://github.com/dougbrion/pytorch-classification-uncertainty

https://muratsensoy.github.io/uncertainty.html 12 | | Single Deterministic Neural Network | Dirichlet Prior Networks | A. Malinin and M. Gales, “Predictive uncertainty estimation via prior networks” NeurIPS 2018. [[paper](https://papers.nips.cc/paper/2018/file/3ea2db50e62ceefceaf70a9d9a56a6f4-Paper.pdf)]

A. Malinin and M. Gales, “Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness” , NeurIPS 2019. [[paper](https://papers.nips.cc/paper/2019/file/7dd2ae7db7d18ee7c9425e38df1af5e2-Paper.pdf)] | yes | PyTorch | https://github.com/KaosEngineer/PriorNetworks 13 | | Bayesian Neural Network | Bayes By Backprop (BBB) | C. Blundell, J. Cornebise, K. Kavukcuoglu & D. Wierstra "Weight Uncertainty in Neural Network", ICML 2015. [[paper](http://proceedings.mlr.press/v37/blundell15.pdf)] | no | PyTorch | https://github.com/nitarshan/bayes-by-backprop

https://github.com/ThirstyScholar/bayes-by-backprop 14 | | Bayesian Neural Network | Example implementation of Multiplicative Normalizing Flows | C. Louizos and M. Welling, "Multiplicative Normalizing Flows for Variational Bayesian Neural Networks", ICML 2017. [[paper](http://proceedings.mlr.press/v70/louizos17a/louizos17a.pdf)] | no | TensorFlow & PyTorch | https://github.com/AMLab-Amsterdam/MNF_VBNN

https://github.com/janosh/torch-mnf 15 | | Bayesian Neural Network | Monte Carlo Dropout | Gal, Yarin, and Zoubin Ghahramani. "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." ICML 2016. [[paper](http://www.cs.ox.ac.uk/people/yarin.gal/website/publications.html#Gal2015Dropout)]| yes | TensorFlow | https://github.com/yaringal/DropoutUncertaintyExps 16 | | Bayesian Neural Network | VOGN Optimizer and presentation of Bayesian principles in neural networks. | M. Khan, D. Nielsen, V. Tangkaratt, W. Lin, Y. Gal and A. Y. Srivastava, "Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam", ICML 2018. [[paper](http://proceedings.mlr.press/v80/khan18a.html)] | yes | PyTorch | https://github.com/team-approx-bayes/dl-with-bayes 17 | | Bayesian Neural Network | Provides facilities to easily train your PyTorch neural network models using variational inference. | - | no | PyTorch | https://github.com/ctallec/pyvarinf 18 | Bayesian Neural Network | Stochastic Weight Averaging-Gaussian (SWAG) | W. J. Maddox, et al. "A simple baseline for bayesian uncertainty in deep learning." NeurIPS 2019: 13153-13164. [[paper](https://proceedings.neurips.cc/paper/2019/file/118921efba23fc329e6560b27861f0c2-Paper.pdf)]| yes | PyTorch | https://github.com/wjmaddox/swa_gaussian | 19 | | Bayesian Neural Network | Deterministic Variational Inference | A. Wu, S. Nowozin, E. Meeds, R. E. Turner, J. M. Hernandez-Lobato, A. L. Gaunt, Deterministic Variational Inference for Robust Bayesian Neural Networks, ICLR 2019. [[paper](https://openreview.net/pdf?id=B1l08oAct7)] | partially | TensorFlow & PyTorch | https://github.com/Microsoft/deterministic-variational-inference

https://github.com/markovalexander/DVI 20 | | Bayesian Neural Network | Laplace Approximation with four different approximations of the curvature | Estimating Model Uncertainty of Neural Networks in Sparse Information Form J. Lee, M. Humt, J. Feng, R. Triebel, ICLR 2020. [[paper](https://proceedings.icml.cc/static/paper_files/icml/2020/2525-Paper.pdf)]

Bayesian Optimization Meets Laplace Approximation for Robotic Introspection M. Humt, J. Lee, R. Triebel, IROS 2020 Workshop. [[paper](https://elib.dlr.de/137021/1/ICRA2020LTAWS_paper_2.pdf)]

"Learning Multiplicative Interactions with Bayesian Neural Networks for Visual-Inertial Odometry", K. Shinde, J. Lee, M. Humt, A. Sezgin, R. Triebel, ICLR 2020 Workshop. [[paper](https://elib.dlr.de/135547/1/Learning%20Multiplicative%20Interactions%20with%20Bayesian%20Neural%20Networks%20for%20Visual-Inertial%20Odometry.pdf)] | yes | PyTorch | https://github.com/DLR-RM/curvature 21 | | Ensembles | Deep Ensembles | B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles” NeurIPS 2017. [[paper](https://papers.nips.cc/paper/2017/file/9ef2ed4b7fd2c810847ffa5fa85bce38-Paper.pdf)]| no | TensorFlow | https://github.com/vvanirudh/deep-ensembles-uncertainty

https://github.com/axelbrando/Mixture-Density-Networks-for-distribution-and-uncertainty-estimation

https://github.com/Kyushik/Predictive-Uncertainty-Estimation-using-Deep-Ensemble 22 | | Collection | Collection containing
  • Deep Ensembles
  • Stochastic Weight-Averaging Gaussian
  • Kronecker-Factored Laplace Approximation
| A. Ashukha, A. Lyzhov, D. Molchanov and D. Vetrov, "Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning", ICLR 2020 | yes | PyTorch | https://github.com/bayesgroup/pytorch-ensembles 23 | | Collection | Collection containing
  • Deep Ensembles
  • Bayes by Backprop
  • Monte Carlo Dropout
  • Temperature scaling
| - | no | PyTorch | https://github.com/cpark321/uncertainty-deep-learning 24 | | Collection | Collection containing
  • Bayes by Backprop
  • Monte Carlo Dropout
  • Stochastic gradient Langevin dynamics
  • Kronecker-Factored Laplace Approximation
  • Stochastic Gradient Hamiltonian Monte Carlo
| - | no | PyTorch | https://github.com/JavierAntoran/Bayesian-Neural-Networks 25 | --------------------------------------------------------------------------------