├── .gitignore ├── assets ├── oodtype.png ├── benchmark.jpg ├── taxonomy.jpg └── timeline.jpg ├── Recruit.md ├── 2_Taxonomy.md ├── visualization.ipynb ├── 6_Outlier.md ├── README.md ├── 7_Outlook.md ├── 4_OSR.md ├── 3_AD.md └── 5_OOD.md /.gitignore: -------------------------------------------------------------------------------- 1 | *.txt 2 | .DS_Store 3 | */.DS_Store 4 | -------------------------------------------------------------------------------- /assets/oodtype.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jingkang50/OODSurvey/HEAD/assets/oodtype.png -------------------------------------------------------------------------------- /assets/benchmark.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jingkang50/OODSurvey/HEAD/assets/benchmark.jpg -------------------------------------------------------------------------------- /assets/taxonomy.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jingkang50/OODSurvey/HEAD/assets/taxonomy.jpg -------------------------------------------------------------------------------- /assets/timeline.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Jingkang50/OODSurvey/HEAD/assets/timeline.jpg -------------------------------------------------------------------------------- /Recruit.md: -------------------------------------------------------------------------------- 1 | # OOD-Related Researcher Openings at SenseTime EIG Research 2 | [![sensetime](https://img.shields.io/badge/Company-SenseTime-DD3E34)](https://arxiv.org) 3 |   4 | [![hongkong](https://img.shields.io/badge/Location-Hong%20Kong-EE1C25)](https://arxiv.org) 5 | 6 | > MMLab@NTU has a very close collaboration with SenseTime Research. 7 | > 8 | > Especially, SenseTime EIG Research pays particular attention to model deployment in real open world scenarios. 9 | > Therefore, they have recently paid great attention to research related to generalized OOD detection 10 | > and plan to recruit a group of full-time researchers to explore this direction with us. 11 | > Below we post their recruitment notice in English and Chinese. 12 | > 13 | > Please send questions or expressions of interest to wayne.zhang (at) sensetime.com 14 | > 15 | 16 | The Emerging Innovation Business Group (EIG), as the new engine of SenseTime's future business growth, focuses on transforming breakthroughs in artificial intelligence (AI) technology innovations into innovative business incubation. EIG has developed innovative AI technologies in multi-modal video understanding, visual inspection for smart manufacturing, remote sensing, and built business partnersihp with leading corporations in media, energy and remote sensing industries, etc. 17 | 18 | ### YOUR CHALLENGES? 19 | 20 | - Formulate research problems from products and deliver research results that will be transferred into products 21 | - Push state-of-the-art in deep learning for multi-modal video understanding, visual inspection for smart manufacturing, remote sensing 22 | - Publish key research findings in top-tier conferences and journals 23 | - Be a contributor of proprietary R&D toolboxes and open source projects 24 | 25 | ### LET'S TALK ABOUT YOU... 26 | 27 | - PhD in STEM (AI related field is defintely a plus), with strong publication record 28 | - Passionate about AI and self-driven to make an impact 29 | - Capable to start a new research topic independently 30 | - Have strong coding skills in Python (knowledge of ML/DL frameworks is a plus) 31 | 32 | ### WE OFFER: 33 | 34 | - Work with a team of highly experienced AI researchers and engineers 35 | - Outstanding R&D environment including cutting-edge technologies and proprietary R&D toolboxes 36 | - Personal development through research seminars, technical trainings and mentorship 37 | - Huge computational resources including 1000+ pieces of GPUs 38 | - Competitive package 39 | - Possibility to cooperate with our joint university laboratories worldwide 40 | 41 | ### Join us! 42 | 43 | --- 44 | 45 | 笔者是在商汤新兴创新事业群（EIG）研究中心实习时开始接触“开放世界识别”领域。我们为了更好地解决具体业务问题，团队会将业务问题抽象成学术问题进行深挖和充分的研究。探索得到的思考，洞察，和新方法不仅形成了顶会论文，也直接在业务上进行落地。实习结束后，笔者也被EIG研究中心直推至MMLab进行深造，目前仍然和EIG研究中心保持紧密合作，共同在开放世界识别的领域探索新颖，有效，可落地的扎实工作。 46 | 47 | EIG研究中心也荟聚了众多顶尖的人工智能技术人才，有若干名校博士，顶会论文作者，openmmlab开源项目主要贡献者。大家秉承着敢为人先的理念，致力于人工智能赋能百业，创新氛围浓厚。 48 | 49 | 目前团队正招募全职研究员（工作地点：香港），希望入职后能够： 50 | - 从产品中抽象研究问题，并将研究成果转化到产品 51 | - 推动前沿的深度学习的进展，课题包括：开放世界识别、视频OCR、文档关键信息提取、半监督检测、分割等，应用方向包括：多模态视频理解、智能制造视觉检测、遥感 52 | - 在顶级会议和期刊上发表关键研究成果 53 | - 成为内部研发工具箱和开源项目的贡献者 54 | 55 | 通常我们期望候选人是： 56 | - STEM 博士，在所在的科研领域（不限于AI相关领域）有突出成果，能够独立进行新的研究课题，或者，本科以上学历，在AI领域发表过至少一篇一作论文或在知名的AI竞赛中获得过前五名的成绩。 57 | - 对人工智能充满热情并有很强的自我驱动力 58 | - 具有较强的 Python 编码能力（了解 ML/DL 框架者优先） 59 | 60 | 我们能够提供： 61 | - 与研究和产业界丰富经验的研究员和工程师团队合作的宽松环境 62 | - 卓越的研发环境，包括多年积累的领先技术和内部研发工具箱 63 | - 论文研讨会、技术培训和个性化指导 64 | - 海量的计算资源，有1000多个GPU的集群可供使用 65 | - 有竞争力的package 66 | - 可以与我们在全球的联合实验室合作，表现优异的同学有推荐读博的机会 67 | 68 | 请感兴趣的同学通过wayne.zhang (at) sensetime.com联系我们！期待与你的合作！ 69 | -------------------------------------------------------------------------------- /2_Taxonomy.md: -------------------------------------------------------------------------------- 1 | 2 | # 2. Taxonomy 3 | - [2.1 Anomaly Detection](#2.1) 4 | - [2.1.1 Sensory Anomaly Detection](#2.1.1) 5 | - [2.1.2 Semantic Anomaly Detection](#2.1.2) 6 | - [2.2 Novelty Detection](#2.2) 7 | - [2.2.1 One-Class Novelty Detection](#2.2.1) 8 | - [2.2.2 Multi-Class Novelty Detection](#2.2.2) 9 | - [2.3 Open Set Recognition](#2.3) 10 | - [2.4 Out-of-Distribution Detection](#2.4) 11 | - [2.5 Outlier Detection](#2.5) 12 | 13 | 14 | 15 | ## 2.1 Anomaly Detection 16 | > *"All normals are alike; each anomaly is abnormal in its own way." - Adapted from "Anna Karenina", by Leo Tolstoy* 17 | 18 | Anomaly detection (AD) aims to detect any anomalous samples that are deviated from the predefined normality during testing. The deviation can happen due to either covariate shift or semantic shift, while assuming the other distribution shift do not exist. This leads to two sub-tasks: sensory AD and semantic AD, respectively. 19 | 20 | 21 | ### 2.1.1 Sensory AD 22 | Sensory AD detects test samples with covariate shift, under the assumption that normalities come from the same covariate distribution. No semantic shift takes place in sensory AD settings. 23 | 24 | 25 | ### 2.1.2 Semantic AD 26 | Semantic AD detects test samples with label shift, assuming that normalities come from the same semantic distribution (category), i.e., normalities should belong to only one class. No covariate shift happens in semantic AD settings. 27 | 28 | Two broad categories of anomaly detection techniques exist. In the standard unsupervised AD setting, all given training samples are normal samples. The (semi-)supervised AD setting requires a dataset that has been labeled as `normal` and `abnormal`, and involves training a model explicitly. 29 | 30 | 31 | 32 | ## 2.2 Novelty Detection 33 | > *"Admitting one’s ignorance is the first step in acquiring knowledge.” - Socrates* 34 | 35 | Novelty detection aims to detect any test samples that do not fall into any training category. 36 | The detected novel samples are usually prepared for future constructive procedures, such as later specialized analysis, or incremental learning of the model itself. 37 | Based on the number of training classes, ND contains two different settings: 38 | 39 | ### 2.2.1 One-class novelty detection 40 | One-class novelty detection (`one-class ND`): only one class exists in the training set; 41 | 42 | ### 2.2.2 Multi-class novelty detection 43 | Multi-class novelty detection (`multi-class ND`): multiple classes exist in the training set. It is worth noting that despite having many ID classes, the goal of multi-class ND is only to distinguish novel samples from ID. 44 | 45 | Both one-class and multi-class ND are formulated as binary classification problems. 46 | 47 | 48 | ## 2.3 Open Set Recognition 49 | > *”To know what you know and what you do not know, that is true knowledge.” - Confucius* 50 | 51 | OSR requires the multi-class classifier to simultaneously: 52 | 1) accurately classify test samples that from `known known classes`; 53 | 2) detect test samples from `unknown unknown classes`. 54 | 55 | 56 | 57 | ## 2.4 Out-of-Distribution Detection 58 | Out-of-distribution detection aims to detect test samples with non-overlapping labels w.r.t training data. 59 | Formally, test samples in the OOD detection setting come from the distribution with semantic shift from ID. 60 | The ID can contain a single class or multiple classes. 61 | When multiple classes exist in training, OOD detection should NOT harm the ID classification capability. 62 | 63 | 64 | 65 | ## 2.5 Outlier Detection 66 | > *”Outliers: Escape from Ordinary.”* 67 | 68 | Different from all previous sub-tasks, whose in-distribution is defined during training, the `in-distribution` for outlier detection refers to the majority of the observations. Outliers may exist due to semantic shift or covariate shift. 69 | -------------------------------------------------------------------------------- /visualization.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "import re\n", 10 | "import numpy as np\n", 11 | "import pandas as pd\n", 12 | "import matplotlib.pyplot as plt\n", 13 | "from matplotlib.pyplot import MultipleLocator\n", 14 | "import matplotlib as mpl\n", 15 | "from collections import Counter\n", 16 | "from operator import itemgetter\n", 17 | "from scipy.interpolate import make_interp_spline" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": null, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "class DataVisualization:\n", 27 | " def __init__(self):\n", 28 | " pass\n", 29 | " \n", 30 | " def extract_data(self, name):\n", 31 | " self.name = name\n", 32 | " journal_list = []\n", 33 | " year_list = []\n", 34 | " with open(name, 'r') as f:\n", 35 | " for line in f.readlines():\n", 36 | " if '**[' in line:\n", 37 | " content = re.search('\\[(.*)\\]', line).group(1)\n", 38 | " year = re.search('\\d+', content).group()\n", 39 | " journal = re.search('\\D+', content).group()[:-1]\n", 40 | " journal_list.append(journal)\n", 41 | " year_list.append(year) \n", 42 | " return journal_list, year_list\n", 43 | " \n", 44 | " def get_dummies(self, journal_list, year_list):\n", 45 | " self.journal = Counter(journal_list).items()\n", 46 | " self.journal = sorted(self.journal, key=itemgetter(1), reverse=True)\n", 47 | " self.year = Counter(year_list).items()\n", 48 | " self.year = sorted(self.year, key=itemgetter(0), reverse=False)\n", 49 | " self.count_journal = len(self.journal)\n", 50 | " self.count_year = len(self.year)\n", 51 | " \n", 52 | " \n", 53 | " def journal_bar_plot(self, color):\n", 54 | " cmap = mpl.cm.get_cmap(color, self.count_journal)\n", 55 | " colors = cmap(np.linspace(0, 1, self.count_journal))\n", 56 | " fig = plt.figure(figsize=(20,10))\n", 57 | " labels, values = zip(*self.journal)\n", 58 | " plt.bar(labels, values, color = colors) \n", 59 | " plt.xticks(rotation=90)\n", 60 | " plt.title(\"Ammounts of journals and booktitles used for \"+ self.name[:-3], fontsize=20)\n", 61 | " plt.tick_params(labelsize=15)\n", 62 | " plt.show()\n", 63 | " \n", 64 | " def year_line_chart(self, color):\n", 65 | " plt.figure(figsize = (20,10))\n", 66 | " labels, values = zip(*self.year)\n", 67 | " labels = list(int(year) for year in labels)\n", 68 | " x_smooth = np.linspace(min(labels), max(labels), 500) \n", 69 | " y_smooth = make_interp_spline(labels, values)(x_smooth)\n", 70 | " plt.plot(x_smooth, y_smooth, color=color, linewidth=1)\n", 71 | " plt.title(\"Published year of paper used for \" + self.name[:-3], fontsize = 18)\n", 72 | " plt.tick_params(labelsize = 15)\n", 73 | " plt.xticks(rotation = 60)\n", 74 | " x_major_locator = MultipleLocator(2)\n", 75 | " ax = plt.gca()\n", 76 | " ax.xaxis.set_major_locator(x_major_locator)\n", 77 | " plt.show()" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": null, 83 | "metadata": {}, 84 | "outputs": [], 85 | "source": [ 86 | "file = ['AD.md', 'OOD.md', 'OSR.md']\n", 87 | "colors = [\"plasma\", \"viridis\", \"RdBu\"]\n", 88 | "for id, name in enumerate(file):\n", 89 | " d = DataVisualization()\n", 90 | " journal, year = d.extract_data(name)\n", 91 | " d.get_dummies(journal, year)\n", 92 | " d.journal_bar_plot(colors[id])\n", 93 | " d.year_line_chart(\"darkblue\")" 94 | ] 95 | } 96 | ], 97 | "metadata": { 98 | "kernelspec": { 99 | "display_name": "Python 3", 100 | "language": "python", 101 | "name": "python3" 102 | }, 103 | "language_info": { 104 | "codemirror_mode": { 105 | "name": "ipython", 106 | "version": 3 107 | }, 108 | "file_extension": ".py", 109 | "mimetype": "text/x-python", 110 | "name": "python", 111 | "nbconvert_exporter": "python", 112 | "pygments_lexer": "ipython3", 113 | "version": "3.8.5" 114 | } 115 | }, 116 | "nbformat": 4, 117 | "nbformat_minor": 4 118 | } 119 | -------------------------------------------------------------------------------- /6_Outlier.md: -------------------------------------------------------------------------------- 1 | 2 | # 6. Outlier Detection 3 | - [6.1 Density-based Method](#6.1) 4 | - [6.2 Distance](#6.2) 5 | - [6.2.1 Cluster-based Method](#6.2.1) 6 | - [6.2.2 Graph-based Method](#6.2.2) 7 | - [6.3 Classification-based Method](#6.3) 8 | 9 | 10 | Outlier detection (OD) requires the observation of all samples and aims to detect those that deviate significantly from the majority distribution. 11 | Therefore, their approaches are usually transductive, rather than inductive. 12 | 13 | 14 | 15 | 16 | ## 6.1 Density-based Method 17 | 18 | **[BMC-2014]** 19 | [Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range](https://link.springer.com/article/10.1186/1471-2288-14-135). 20 |
21 | **Authors:** Xiang Wan, Wenqian Wang, Jiming Liu, Tiejun Tong 22 |
23 | **Institution:** Hong Kong Baptist University; Northwestern University 24 | 25 | 26 | **[SIGMOD-2000]** 27 | [Lof: identifying density-based local outliers](https://dl.acm.org/doi/abs/10.1145/342009.335388). 28 |
29 | **Authors:** Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, Jorg Sander 30 |
31 | **Institution:** University of Munich; University of British Columbia 32 | 33 | 34 | **[PAKDD-2002]** 35 | [Enhancing effectiveness of outlier detections for low density patterns](https://link.springer.com/chapter/10.1007/3-540-47887-6_53). 36 |
37 | **Authors:** Jian Tang, Zhixiang Chen, Ada Wai-chee Fu, David W. Cheung 38 |
39 | **Institution:** Chinese University of Hong Kong; University of Texas; University of Hong Kong 40 | 41 | 42 | **[ACM-2009]** 43 | [Loop: local outlier probabilities](https://dl.acm.org/doi/abs/10.1145/1645953.1646195). 44 |
45 | **Authors:** Hans-Peter Kriegel, Peer Kroger, Erich Schubert, Arthur Zimek 46 |
47 | **Institution:** Ludwig-Maximilians University 48 | 49 | 50 | **[DMKD-2012]** 51 | [Local outlier detection reconsidered: a generalized view on locality with applications to spatial, video, and network outlier detection](https://link.springer.com/article/10.1007/s10618-012-0300-z). 52 |
53 | **Authors:** Erich Schubert, Arthur Zimek, Hans-Peter Kriegel 54 |
55 | **Institution:** Ludwig-Maximilians-University; University of Alberta 56 | 57 | 58 | **[ACM-1981]** 59 | [Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography](https://dl.acm.org/doi/abs/10.1145/358669.358692). 60 |
61 | **Authors:** Martin A. Fischler, Robert C. Bolles 62 |
63 | **Institution:** SRI International 64 | 65 | 66 | **[WIREs-2011]** 67 | [Robust statistics for outlier de- tection](https://wires.onlinelibrary.wiley.com/doi/abs/10.1002/widm.2). 68 |
69 | **Authors:** Peter J. Rousseeuw, Mia Hubert 70 |
71 | **Institution:** Katholieke University 72 | 73 | 74 | 75 | **[NeurIPS-2018]** 76 | [Efficient anomaly detection via matrix sketching](https://arxiv.org/abs/1804.03065). 77 |
78 | **Authors:** Vatsal Sharan, Parikshit Gopalan, Udi Wieder 79 |
80 | **Institution:** Stanford University; VMware Research 81 | 82 | 83 | ## 6.2 Distance 84 | 85 | 86 | 87 | ### 6.2.1 Cluster-based Method 88 | The most basic OD method model the entire dataset with the Gaussian distribution, and flag the samples over three standard deviations from the mean. 89 | 90 | 91 | **[KDD-1996]** 92 | [A density-based algorithm for discovering clusters in large spatial databases with noise](https://www.aaai.org/Papers/KDD/1996/KDD96-037.pdf?source=post_page). 93 |
94 | **Authors:** Martin Ester, Hans-Peter Kriegel, Jiirg Sander, Xiaowei Xu 95 |
96 | **Institution:** University of Munich 97 | 98 | 99 | 100 | **[ECML-2007]** 101 | [Class noise mitigation through instance weighting](https://link.springer.com/chapter/10.1007/978-3-540-74958-5_71). 102 |
103 | **Authors:** Umaa Rebbapragada, Carla E. Brodley 104 |
105 | **Institution:** Tufts University 106 | 107 | 108 | 109 | 110 | 111 | ### 6.2.2 Graph-based Method 112 | Similar to "three standard deviations" rules under the assumption that the data follows normal distribution, interquartile range can also be used to identify outliers. 113 | 114 | **[DMKD-2014]** 115 | [Graph based anomaly detection and description: a survey](https://link.springer.com/article/10.1007/s10618-014-0365-y). 116 |
117 | **Authors:** Leman Akoglu; Hanghang Tong; Danai Koutra 118 |
119 | **Institution:** Stony Brook University, City University of New York, Carnegie Mellon University 120 | 121 | 122 | 123 | **[SIGKDD-2003]** 124 | [Graph-based anomaly detection](https://dl.acm.org/doi/abs/10.1145/956750.956831). 125 |
126 | **Authors:** Caleb C. Noble, Diane J. Cook 127 |
128 | **Institution:** University of Texas 129 | 130 | 131 | 132 | **[ICTAI-2007]** 133 | [Spatial outlier detection: a graph-based approach](https://ieeexplore.ieee.org/abstract/document/4410296). 134 |
135 | **Authors:** Yufeng Kou, Chang-Tien Lu, Raimundo F. Dos Santos 136 |
137 | **Institution:** Virginia Polytechnic Institute and State University 138 | 139 | 140 | 141 | **[ICCSE-2012]** 142 | [A graph-based clustering algorithm for anomaly intrusion detection](https://ieeexplore.ieee.org/abstract/document/6295306). 143 |
144 | **Authors:** Zhou Mingqiang, Huang Hui, Wang Qian 145 |
146 | **Institution:** Chongqing University 147 | 148 | 149 | 150 | **[ACM-2020]** 151 | [Webly supervised image classification with metadata: Automatic noisy label correction via visual-semantic graph](https://dl.acm.org/doi/abs/10.1145/3394171.3413952). 152 |
153 | **Authors:** Jingkang Yang, Weirong Chen, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang 154 |
155 | **Institution:** Sensetime Research; Rice University; The Chinese University of Hong Kong; Shanghai Jiao Tong University 156 | 157 | 158 | 159 | 160 | 161 | ## 6.3 Classification-based Method 162 | 163 | **[-2002]** 164 | [One-class classification: Concept learning in the absence of counter-examples](https://www.elibrary.ru/item.asp?id=5230402). 165 |
166 | **Authors:** Tax D.M.J 167 |
168 | **Institution:** Technische Universiteit Delft 169 | 170 | 171 | **[ICMI-2018]** 172 | [Deep one-class classification](http://proceedings.mlr.press/v80/ruff18a). 173 |
174 | **Authors:** Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Muller, Marius Kloft 175 |
176 | **Institution:** Humboldt University; Hasso Plattner Institute; TU Kaiserslautern; TU Berlin; University of Edinburgh; DFKI GmbH; Singapore University of Technology and Design 177 | 178 | 179 | **[ICDM-2008]** 180 | [Isolation forest](https://ieeexplore.ieee.org/abstract/document/4781136/). 181 |
182 | **Authors:** Fei Tony Liu, Kai Ming Ting, Zhi-Hua Zhou 183 |
184 | **Institution:** Monash University; Nanjing University 185 | 186 | 187 | **[CVPR-2017]** 188 | [Learning from noisy labels with distillation](https://openaccess.thecvf.com/content_iccv_2017/html/Li_Learning_From_Noisy_ICCV_2017_paper.html). 189 |
190 | **Authors:** Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, Li-Jia Li 191 |
192 | **Institution:** Snap Inc.; Yahoo Research 193 | 194 | 195 | **[ICLR-2020]** 196 | [Self: Learning to filter noisy labels with self-ensembling](https://arxiv.org/abs/1910.01842). 197 |
198 | **Authors:** Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, Thomas Brox 199 |
200 | **Institution:** University of Freiburg; Bosch Research; Bosch Center for AI; Karlsruhe Institute of Technology 201 | 202 | 203 | **[NIPS-2018]** 204 | [Co-teaching: Robust training of deep neural networks with extremely noisy labels](https://arxiv.org/abs/1804.06872). 205 |
206 | **Authors:** Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, Masashi Sugiyama 207 |
208 | **Institution:** University of Technology Sydney; RIKEN; 4Paradigm Inc.; Stanford University; University of Tokyo 209 | 210 | 211 | **[ECCV-2020]** 212 | [Webly supervised image classification with self- contained confidence](https://link.springer.com/chapter/10.1007%2F978-3-030-58598-3_46). 213 |
214 | **Authors:** Jingkang Yang, Litong Feng, Weirong Chen, Xiaopeng Yan, Huabin Zheng, Ping Luo, Wayne Zhang 215 |
216 | **Institution:** SenseTime Research; Rice University; The Chinese University of Hong Kong; The University of Hong Kong 217 | 218 | 219 | 220 | 221 | 222 | 223 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Generalized Out-of-Distribution Detection: A Survey 2 | 3 | [![paper](https://img.shields.io/badge/Paper-arxiv-b31b1b)](https://arxiv.org/abs/2110.11334) 4 |   5 | [![recruit](https://img.shields.io/badge/OOD–Related%20Researchers-Recruiting!-1463BD)](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/Recruit.md) 6 |   7 | [![welcome](https://img.shields.io/badge/Issues,%20Comments,%20and%20Questions-are%20all%20welcomed!-f39f37)](https://github.com/Jingkang50/OODSurvey/issues) 8 | 9 | ![benchmark](assets/benchmark.jpg) 10 | 11 | 12 | ## 1. Overview 13 | This repository is with our survey paper: 14 | 15 | 16 | > **Title:** [Generalized Out-of-Distribution Detection: A Survey](https://arxiv.org/abs/2110.11334)
17 | > **Authors:** [Jingkang Yang1](https://jingkang50.github.io/), [Kaiyang Zhou1](https://kaiyangzhou.github.io/), [Yixuan Li2](http://pages.cs.wisc.edu/~sharonli/), [Ziwei Liu1](https://github.com/liuziwei7)
18 | > **Institutions:** [1MMLab@NTU](https://www.mmlab-ntu.com/), [2University of Wisconsin-Madison](https://www.cs.wisc.edu/). 19 | 20 | 21 | This survey comprehensively reviews the similar topics of **outlier detection (OD)**, **anomaly detection (AD)**, **novelty detection (ND)**, **open set recognition (OSR)**, and **out-of-distribution (OOD) detection**, extensively compares their commomality and differences, and eventually unifies them under a big umbrella of "generalized OOD detection" framework. 22 | 23 | We hope that this survey can help readers and participants better understand the open-world field centered on OOD detection. At the same time, it urges future work to learn, compare, and develop ideas and methods from the broader scope of generalized OOD detection, with clear problem definition and proper benchmarking. 24 | 25 | We prepare this repository for the following two reasons: 26 | 1. We consider it an awesome list to easily access the references mentioned in the paper Table 1. We also believe this list will continue to include more promising works as new works appear. Please feel free to nominate good related works with [Pull Requests](https://github.com/Jingkang50/OOD_Detection_Survey/pulls). 27 | 2. We hope this repository becomes a discussion panel for readers to ask questions, raise concerns, and make constructive comments for the broad generalized OOD detection field. Please feel free to post your ideas in the [Issues](https://github.com/Jingkang50/OOD_Detection_Survey/issues). 28 | 29 | We are also planning to build an evaluation benchmark to compare representative generalized OOD detection methods from *every* sub-task to further unify the field. The work will be collaborated with SenseTime EIG Research, which recently have many full-time researcher openings for this benchmarking project and other OOD-related research. Check their [Recruitment Info](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/Recruit.md) for more information. 30 | 31 | 32 | ![benchmark](assets/oodtype.png) | ![benchmark](assets/taxonomy.jpg) 33 | :-----------------------------:|:-------------------------: 34 | **Fig.1.1**: Two kinds of distribution shift to assist better understanding of our framework. | **Fig.1.2**: Taxonomy diagram of generalized OOD detection framework. 35 | 36 | ## 2. Taxonomy 37 | - [2.1 Anomaly Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/2_Taxonomy.md#2.1) 38 | - [2.1.1 Sensory Anomaly Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/2_Taxonomy.md#2.1.1) 39 | - [2.1.2 Semantic Anomaly Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/2_Taxonomy.md#2.1.2) 40 | - [2.2 Novelty Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/2_Taxonomy.md#2.2) 41 | - [2.2.1 One-Class Novelty Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/2_Taxonomy.md#2.2.1) 42 | - [2.2.2 Multi-Class Novelty Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/2_Taxonomy.md#2.2.2) 43 | - [2.3 Open Set Recognition](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/2_Taxonomy.md#2.3) 44 | - [2.4 Out-of-Distribution Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/2_Taxonomy.md#2.4) 45 | - [2.5 Outlier Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/2_Taxonomy.md#2.5) 46 | - [2.6 Discussion](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/2_Taxonomy.md#2.5) 47 | 48 | 49 | ## 3. Anomaly Detection & One-Class Novelty Detection 50 | - [3.1 Density-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.1) 51 | - [3.1.1 Classic Density Estimation](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.1.1) 52 | - [3.1.2 NN-based Density Estimation](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.1.2) 53 | - [3.1.3 Energy-based Model](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.1.3) 54 | - [3.1.4 Frequency-based Model](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.1.4) 55 | - [3.2 Reconstruction-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.2) 56 | - [3.2.1 Sparse Representation Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.2.1) 57 | - [3.2.2 Reconstruction-Error Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.2.2) 58 | - [3.3 Classification-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.3) 59 | - [3.3.1 One-Class Classification](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.3.1) 60 | - [3.3.2 Positive-Unlabeled Learning](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.3.2) 61 | - [3.3.3 Self-Supervised Learning](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.3.3) 62 | - [3.4 Distance-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.4) 63 | - [3.5 Gradient-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.5) 64 | - [3.6 Discussion and Theoretical Analysis](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/3_AD.md#3.6) 65 | 66 | 67 | ## 4. Multi-Class Novelty Detection & Open Set Recognition 68 | - [4.1 Classfication-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/4_OSR.md#4.1) 69 | - [4.1.1 EVT-based Calibration](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/4_OSR.md#4.1.1) 70 | - [4.1.2 EVT-free Calibration](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/4_OSR.md#4.1.2) 71 | - [4.1.3 Unknown Class Generation](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/4_OSR.md#4.1.3) 72 | - [4.1.4 Label Space Redesign](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/4_OSR.md#4.1.4) 73 | - [4.2 Distance-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/4_OSR.md#4.2) 74 | - [4.3 Reconstruction-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/4_OSR.md#4.3) 75 | - [4.3.1 Sparse Representation Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/4_OSR.md#4.3.1) 76 | - [4.3.2 Reconstruction-Error Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/4_OSR.md#4.3.2) 77 | 78 | 79 | 80 | ## 5. Out-of-Distribution Detection 81 | - [5.1 Classfication-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.1) 82 | - [5.1.0 Baseline](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.1.0) 83 | - [5.1.1 Output-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.1.1) 84 | - [5.1.1.1 Post-hoc Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.1.1.1) 85 | - [5.1.1.2 Confidence Enhancement](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.1.1.2) 86 | - [5.1.1.3 Outlier Exposure](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.1.1.3) 87 | - [5.1.2 OOD Data Generation](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.1.3) 88 | - [5.1.3 Gradient-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.1.4) 89 | - [5.1.4 Bayesian Models](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.1.4) 90 | - [5.1.5 Large-scale OOD Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.1.5) 91 | - [5.2 Density-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.2) 92 | - [5.3 Distance-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/5_OOD.md#5.3) 93 | 94 | 95 | ## 6. Outlier Detection 96 | - [6.1 Density-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/6_Outlier.md#6.1) 97 | - [6.2 Distance-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/6_Outlier.md#6.2) 98 | - [6.2.1 Cluster-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/6_Outlier.md#6.2.1) 99 | - [6.2.2 Graph-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/6_Outlier.md#6.2.2) 100 | - [6.3 Classification-based Methods](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/6_Outlier.md#6.3) 101 | 102 | ## 7. Challenges and Future Direction 103 | - [7.1 Challenges](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/7_Outlook.md#7.1) 104 | - [7.1.1 Proper Evaluation and Benchmarking](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/7_Outlook.md#7.1.1) 105 | - [7.1.2 Outlier-free OOD Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/7_Outlook.md#7.1.2) 106 | - [7.1.3 Tradeoff Between Classification and OOD Detection](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/7_Outlook.md#7.1.3) 107 | - [7.1.4 Real-world Benchmarks and Evaluations](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/7_Outlook.md#7.1.4) 108 | - [7.2 Future Directions](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/7_Outlook.md#7.2) 109 | - [7.2.1 Methodologies across Sub-tasks](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/7_Outlook.md#7.2.1) 110 | - [7.2.2 OOD Detection & OOD Generalization](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/7_Outlook.md#7.2.2) 111 | - [7.2.3 OOD Detection & Open-Set Noisy Labels](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/7_Outlook.md#7.2.3) 112 | - [7.2.4 Theoretical Analysis](https://github.com/Jingkang50/OOD_Detection_Survey/blob/main/7_Outlook.md#7.2.4) 113 | 114 | 115 | ## 8. Conclusion 116 | In this survey, we comprehensively review five topics: AD, ND, OSR, OOD detection, and OD, and unify them as a framework of *generalized OOD detection*. By articulating the motivations and definitions of each sub-task, we encourage follow-up works to accurately locate their target problems and find the most suitable benchmarks. 117 | By sorting out the methodologies for each sub-task, we hope that readers can easily grasp the mainstream methods, identify suitable baselines, and contribute future solutions in light of existing ones. 118 | By providing insights, challenges, and future directions, we hope that future works will pay more attention to the existing problems and explore more interactions across other tasks within or even outside the scope of generalized OOD detection. 119 | 120 | 121 | ## 122 | 123 | ## Citation 124 | If you find our survey and repository useful for your research, please consider citing our paper: 125 | ```bibtex 126 | @article{yang2021oodsurvey, 127 | title={Generalized Out-of-Distribution Detection: A Survey}, 128 | author={Yang, Jingkang and Zhou, Kaiyang and Li, Yixuan and Liu, Ziwei}, 129 | journal={arXiv preprint arXiv:2110.11334}, 130 | year={2021} 131 | } 132 | ``` 133 | 134 | 135 | ## Acknowledgements 136 | This repository is created and maintained by Jingkang Yang and Peng Wenxuan from NTU; Kunyuan Ding, Zixu Song, Pengyun Wang, Zitang Zhou, and Dejian Zou from BUPT. 137 | -------------------------------------------------------------------------------- /7_Outlook.md: -------------------------------------------------------------------------------- 1 | # 7. Challenges and Future Direction 2 | - [7.1 Challenges](#7.1) 3 | - [7.1.1 Proper Evaluation and Benchmarking](#7.1.1) 4 | - [7.1.2 Outlier-free OOD Detection](#7.1.2) 5 | - [7.1.3 Tradeoff Between Classification and OOD Detection](#7.1.3) 6 | - [7.1.4 Real-world Benchmarks and Evaluations](#7.1.4) 7 | - [7.2 Future Directions](#7.2) 8 | - [7.2.1 Methodologies across Sub-tasks](#7.2.1) 9 | - [7.2.2 OOD Detection & OOD Generalization](#7.2.2) 10 | - [7.2.3 OOD Detection & Open-Set Noisy Labels](#7.2.3) 11 | - [7.2.4 Theoretical Analysis](#7.2.4) 12 | 13 | 14 | ## 7.1 Challenges 15 | 16 | ### 7.1.1 Proper Evaluation and Benchmarking 17 | We hope this survey can clarify the distinctions and connections of various sub-tasks, and help future works properly identify the target problem and benchmarks within the framework. The mainstream OOD detection works primarily focus on detecting semantic shifts. 18 | Admittedly, the field of OOD detection can be very broad due to the diverse nature of distribution shifts. 19 | Such a broad OOD definition also leads to some challenges and concerns [[1, 2]](#7.1.1.ref), which advocate a clear specification of OOD type in consideration (e.g. semantic OOD, adversarial OOD, etc.) so that proposed solutions can be more specialized. 20 | Besides, the motivation of detecting a certain distribution shift also requires clarification. While rejecting classifying samples with semantic shift is apparent, detecting sensory OOD should be specified to some meaningful scenarios to contextualize the necessity and practical relevance of the task. 21 | 22 | We also urge the community to carefully construct the benchmarks and evaluations. It is noticed that early work [[3]](#7.1.1.ref) ignored the fact that some OOD datasets may contain images with ID categories, causing inaccurate performance evaluation. 23 | Fortunately, recent OOD detection works [[4, 5]](#7.1.1.ref) have realized this flaw and pay special attention to removing ID classes from OOD samples to ensure proper evaluation. 24 | 25 | --- 26 | 27 | [1] W. Gan, [“Language guided out-of-distribution detection,”](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-139.pdf) 2021. 28 | 29 | [2] F. Ahmed and A. Courville, [“Detecting semantic anomalies,”](https://ojs.aaai.org/index.php/AAAI/article/download/5712/5568) in AAAI, 2020 30 | 31 | [3] D. Hendrycks and K. Gimpel, [“A baseline for detecting misclassified and out-of-distribution examples in neural networks,”](https://arxiv.org/pdf/1610.02136) in ICLR, 2017 32 | 33 | [4] R. Huang and Y. Li, [“Mos: Towards scaling out-of-distribution detection for large semantic space,”](https://openaccess.thecvf.com/content/CVPR2021/papers/Huang_MOS_Towards_Scaling_Out-of-Distribution_Detection_for_Large_Semantic_Space_CVPR_2021_paper.pdf) in CVPR, 2021 34 | 35 | [5] J. Yang, H. Wang, L. Feng, X. Yan, H. Zheng, W. Zhang, and Z. Liu, [“Semantically coherent out-of-distribution detection,”](http://openaccess.thecvf.com/content/ICCV2021/papers/Yang_Semantically_Coherent_Out-of-Distribution_Detection_ICCV_2021_paper.pdf) in ICCV, 2021 36 | 37 | 38 | ### 7.1.2 Outlier-free OOD Detection 39 | The outlier exposure approach [[6]](#7.1.2.ref) imposes a strong assumption of the availability of OOD training data, which can be difficult to obtain in practice. Moreover, one needs to perform careful de-duplication to ensure that the outlier training data does not contain ID data. These restrictions may lead to inflexible solutions and prevent the adoption of methods in the real world. As with the recent taken-down of TinyImages dataset [[7]](#7.1.2.ref), it poses a reproducibility crisis for OE-based methods. Going forward, a major challenge for the field is to devise outlier-free learning objectives that are less dependent on auxiliary outlier dataset. 40 | 41 | --- 42 | 43 | [6] D. Hendrycks, M. Mazeika, and T. Dietterich, [“Deep anomaly detection with outlier exposure,”](https://arxiv.org/pdf/1812.04606) in ICLR, 2019 44 | 45 | [7] A. Torralba, R. Fergus, and W. T. Freeman, [“80 million tiny images: A large data set for nonparametric object and scene recognition,”](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.73.4858&rep=rep1&type=pdf) TPAMI, 2008. 46 | 47 | 48 | ### 7.1.3 Tradeoff Between Classification and OOD Detection 49 | In OSR and OOD detection, it is important to achieve the dual objectives simultaneously: one for the ID task (e.g. image classification), another for the OOD detection task. For a shared network, an inherent trade-off may exist between the two tasks. Promising solutions should strive for both. These two tasks may or may not contradict each other, depending on the methodologies. For example, [[8]](#7.1.3.ref) advocated the integration of image classification and open-set recognition so that the model will possess the capability of discriminative recognition on known classes and sensitivity to novel classes at the same time. 50 | [[9]](#7.1.3.ref) also showed that the ability of detecting novel classes can be highly correlated with its accuracy on the closed-set classes. 51 | [[10]](#7.1.3.ref) demonstrated that optimizing for the cluster compactness of ID classes may facilitate both improved classification and distance-based OOD detection performance. Such solutions may be more desirable than ND, which develops a binary OOD detector separately from the classification model, and requires deploying two models. 52 | 53 | --- 54 | 55 | [8] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu, [“Largescale long-tailed recognition in an open world,”](http://openaccess.thecvf.com/content_CVPR_2019/papers/Liu_Large-Scale_Long-Tailed_Recognition_in_an_Open_World_CVPR_2019_paper.pdf) 56 | in CVPR, 2019. 57 | [9] S. Vaze, K. Han, A. Vedaldi, and A. Zisserman, [“Open-set recognition: A good closed-set classifier is all you need,”](https://arxiv.org/pdf/2110.06207) arXiv preprint arXiv:2110.06207, 2021. 58 | 59 | [10] J. Yang, H. Wang, L. Feng, X. Yan, H. Zheng, W. Zhang, and Z. Liu, [“Semantically coherent out-of-distribution detection,”](http://openaccess.thecvf.com/content/ICCV2021/papers/Yang_Semantically_Coherent_Out-of-Distribution_Detection_ICCV_2021_paper.pdf) in ICCV, 2021. 60 | 61 | 62 | ### 7.1.4 Real-world Benchmarks and Evaluations 63 | Current methods have been primarily evaluated on small data sets such as CIFAR. 64 | It's been shown that approaches developed on the CIFAR benchmark might not translate effectively into ImageNet benchmark with a large semantic space, highlighting the need to evaluate OOD detection in a large-scale real-world setting. 65 | Therefore, we encourage future research to evaluate on ImageNet-based OOD detection benchmark [[11]](#7.1.4.ref), as well as large-scale OSR benchmark [[12]](#7.1.4.ref), and test the limits of the method developed. Moreover, real-world benchmarks that go beyond image classification can be valuable for the research community. In particular, for safety-critical settings such as autonomous driving and medical imaging diagnosis, more specialized benchmarks are needed and should be carefully constructed. 66 | 67 | --- 68 | 69 | [11] R. Huang and Y. Li, [“Mos: Towards scaling out-of-distribution detection for large semantic space,”](https://openaccess.thecvf.com/content/CVPR2021/papers/Huang_MOS_Towards_Scaling_Out-of-Distribution_Detection_for_Large_Semantic_Space_CVPR_2021_paper.pdf) in CVPR, 2021. 70 | 71 | [12] S. Vaze, K. Han, A. Vedaldi, and A. Zisserman, [“Open-set recognition: A good closed-set classifier is all you need,”](https://arxiv.org/pdf/2110.06207) arXiv preprint arXiv:2110.06207, 2021. 72 | 73 | 74 | 75 | ## 7.2 Future Directions 76 | 77 | ### 7.2.1 Methodologies across Sub-tasks 78 | Due to the inherent connections among different sub-tasks, their solution space can be shared and inspired from each other. For example, the recent emerging density-based OOD detection research can draw insights from the density-based AD methods that have been around for a long time. 79 | 80 | 81 | ### 7.2.2 OOD Detection & OOD Generalization 82 | An open-world classifier should consider two tasks, i.e., being robust to covariate shift while being aware of the semantic shift. Existing works pursue these two goals independently. Recent work proposes a semantically coherent OOD detection framework [[1]](#7.2.2.ref) that encourages detecting semantic OOD samples while being robust to negligible covariate shift. Given the vague definition of OOD, [[2]](#7.2.2.ref) proposed a new formalization of OOD detection by explicitly taking into account the separation 83 | between invariant features (semantic related) and environmental features (non-semantic). The work highlighted that spurious environmental features in the training set can significantly impact 84 | OOD detection, especially when the label-shifted OOD data contains the spurious feature. Recent works on open long-tailed recognition [[3]](#7.2.2.ref), open compound domain adaptation [[4]](#7.2.2.ref), open-set domain adaptation [[5]](#7.2.2.ref) and open-set domain generalization [[6]](#7.2.2.ref) consider the potential existence of open-class samples. 85 | Looking ahead, we envision great research opportunities on how OOD detection and OOD generalization can better enable each other [[7]](#7.2.2.ref), in terms of both algorithmic design and comprehensive performance evaluation. 86 | 87 | --- 88 | 89 | [1] J. Yang, H. Wang, L. Feng, X. Yan, H. Zheng, W. Zhang, and Z. Liu, [“Semantically coherent out-of-distribution detection,”](http://openaccess.thecvf.com/content/ICCV2021/papers/Yang_Semantically_Coherent_Out-of-Distribution_Detection_ICCV_2021_paper.pdf) in ICCV, 2021. 90 | 91 | [2] Y. Ming, H. Yin, and Y. Li, [“On the impact of spurious correlation for out-of-distribution detection,”](https://arxiv.org/pdf/2109.05642) in AAAI, 2022 92 | 93 | [3] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu, [“Large scale long-tailed recognition in an open world,”](http://openaccess.thecvf.com/content_CVPR_2019/papers/Liu_Large-Scale_Long-Tailed_Recognition_in_an_Open_World_CVPR_2019_paper.pdf) in CVPR, 2019. 94 | 95 | [4] Z. Liu, Z. Miao, X. Pan, X. Zhan, D. Lin, S. X. Yu, and B. Gong, [“Open compound domain adaptation,”](http://openaccess.thecvf.com/content_CVPR_2020/papers/Liu_Open_Compound_Domain_Adaptation_CVPR_2020_paper.pdf) in CVPR, 2020. 96 | 97 | [5] P. Panareda Busto and J. Gall, [“Open set domain adaptation,”](http://openaccess.thecvf.com/content_ICCV_2017/papers/Busto_Open_Set_Domain_ICCV_2017_paper.pdf) in ICCV, 2017. 98 | 99 | [6] Y. Shu, Z. Cao, C. Wang, J. Wang, and M. Long, [“Open domain generalization with domain-augmented meta-learning,”](https://openaccess.thecvf.com/content/CVPR2021/papers/Shu_Open_Domain_Generalization_with_Domain-Augmented_Meta-Learning_CVPR_2021_paper.pdf) in CVPR, 2021 100 | 101 | [7] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu, [“Largescale long-tailed recognition in an open world,”](http://openaccess.thecvf.com/content_CVPR_2019/papers/Liu_Large-Scale_Long-Tailed_Recognition_in_an_Open_World_CVPR_2019_paper.pdf) 102 | in CVPR, 2019. 103 | 104 | ### 7.2.3 OOD Detection & Open-Set Noisy Labels 105 | Existing methods of learning from open-set noisy labels focus on suppressing the negative effects of noise [[1, 2]](#7.2.3.ref). However, 106 | the open-set noisy samples can be useful for outlier exposure [[3]](#7.2.3.ref) and potentially benefit OOD detection. 107 | With a similar idea, the setting of open-set semi-supervised learning can be promising for OOD detection. 108 | We believe the combination between OOD detection and the previous two fields can provide more insights and possibilities. 109 | 110 | --- 111 | 112 | [1] Y. Wang, W. Liu, X. Ma, J. Bailey, H. Zha, L. Song, and S.-T. Xia, [“Iterative learning with open-set noisy labels,”](http://openaccess.thecvf.com/content_cvpr_2018/papers/Wang_Iterative_Learning_With_CVPR_2018_paper.pdf) 113 | in CVPR, 2018. 114 | 115 | [2] J. Li, C. Xiong, and S. C. Hoi, [“Mopro: Webly supervised learning with momentum prototypes,”](https://arxiv.org/pdf/2009.07995) 116 | ICLR, 2021. 117 | 118 | [3] Z.-F. Wu, T. Wei, J. Jiang, C. Mao, M. Tang, and Y.-F. Li, [“Ngc: A unified framework for learning with open-world noisy data,”](https://openaccess.thecvf.com/content/ICCV2021/papers/Wu_NGC_A_Unified_Framework_for_Learning_With_Open-World_Noisy_Data_ICCV_2021_paper.pdf) in ICCV, 2021. 119 | 120 | 121 | ### 7.2.4 Theoretical Analysis 122 | While most of the existing OOD detection works focus on developing effective approaches to obtain better empirical performance, the theoretical analysis remains largely underexplored. We hope future research can also contribute theoretical analyses and provide in-depth insights that help guide algorithmic development with rigorous guarantees. 123 | 124 | [1] Z. Fang, J. Lu, A. Liu, F. Liu, G. Zhang, [“Learning Bounds for Open-Set Learning”](https://arxiv.org/abs/2106.15792) in ICML, 2021. 125 | 126 | [2] Peyman Morteza and Yixuan Li, [“Provable Guarantees for Understanding Out-of-distribution Detection,”](https://arxiv.org/abs/2112.00787) in AAAI, 2022 127 | 128 | -------------------------------------------------------------------------------- /4_OSR.md: -------------------------------------------------------------------------------- 1 | 2 | # 4. Multi-Class Novelty Detection & Open Set Recognition 3 | - [4.1 Classfication](#4.1) 4 | - [4.1.1 EVT-based Calibration](#4.1.1) 5 | - [4.1.2 EVT-free Calibration](#4.1.2) 6 | - [4.1.3 Unknown Generation](#4.1.3) 7 | - [4.1.4 Label Space Redesign](#4.1.4) 8 | - [4.2 Distance-based Method](#4.2) 9 | - [4.3 Reconstruction](#4.3) 10 | - [4.3.1 Sparse Representation](#4.3.1) 11 | - [4.3.2 Reconstruction-Error](#4.3.2) 12 | 13 | 14 | 15 | 16 | ## 4.1 Classfication 17 | 18 | **[TPAMI-2013]** 19 | [Toward Open Set Recognition](https://ieeexplore.ieee.org/abstract/document/6365193) 20 |
21 | **Authors:** Walter J. Scheirer, Anderson de Rezende Rocha, Archana Sapkota, Terrance E. Boult 22 |
23 | **Institution:** Harvard University; University of Campinas; University of Colorado, Colorado Springs 24 | >
25 | > Kicking-off paper using 1-vs-set machine for OSR. 26 | >

27 | > This paper highlights the practicality of OSR by showing the difference between classification and recognition: classification only has a given set of classes between which we must discriminate; Recognition has some classes we can recognize in a much larger space of things we do not recognize. The paper shows the validity of 1-class SVM and binary SVM for OSR, and proposes 1-vs-Set SVM to manage the open-set risk by solving a two-plane optimization problem instead of the classic half-space of a binary linear classifier. 28 | >

29 | >
30 | 31 | 32 | ### 4.1.1 EVT-based Uncertainty Calibration 33 | 34 | **[TPAMI-2014]** 35 | [Probability models for open set recognition](https://ieeexplore.ieee.org/abstract/document/6809169) 36 |
37 | **Authors:** Walter J. Scheirer, Lalit P. Jain, Terrance E. Boult 38 |
39 | **Institution:** Harvard University; University of Colorado, Colorado Springs 40 | >
41 | > W-SVM using CAP and EVT for score calibration on one-class and binary SVM. 42 | >

43 | > CAP explicitly models the probability of class membership abating from ID points to OOD points, as classic probabilistic model lacks the consideration of open space, and EVT exactly focuses on modeling the tailed distribution with extreme high/low values. The novel Weibull-calibrated SVM (W-SVM) algorithm is introduced, combining the useful properties of CAP and EVT. 44 | >

45 | >
46 | 47 | **[ECCV-2014]** 48 | [Multi-class open set recognition using probability of inclusion](https://link.springer.com/content/pdf/10.1007/978-3-319-10578-9_26.pdf) 49 |
50 | **Authors:** Lalit P. Jain, Walter J. Scheirer, Terrance E. Boult 51 |
52 | **Institution:** University of Colorado, Colorado Springs; Harvard University; Securics 53 | >
54 | > PI-SVM estimating the unnormalized posterior probability of class inclusion. 55 | >

56 | > Modeling positive training data at the decision boundary, where we can invoke the statistical EVT. A new algorithm called the PI-SVM is introduced for estimating the unnormalized posterior probability of multiple class inclusion. 57 | >

58 | >
59 | 60 | 61 | **[CVPR-2016]** 62 | [Towards open set deep networks](https://www.cv-foundation.org/openaccess/content_cvpr_2016/html/Bendale_Towards_Open_Set_CVPR_2016_paper.html) 63 |
64 | **Authors:** Abhijit Bendale, Terrance E. Boult 65 |
66 | **Institution:** University of Colorado, Colorado Springs 67 | >
68 | > OpenMax: Replacing softmax layer with OpenMax and calibrating the confidence to predict novel class. 69 | >

70 | > This method uses the scores from the penultimate layer to estimate if the input is “far” from known training data. 71 | >

72 | >
73 | 74 | 75 | **[BMVC-2017]** 76 | [Adversarial robustness: Softmax versus openmax](https://arxiv.org/abs/1708.01697) 77 |
78 | **Authors:** Andras Rozsa, Manuel Gunther, Terrance E. Boult 79 |
80 | **Institution:** University of Colorado, Colorado Springs 81 | 82 | 83 | ### 4.1.2 EVT-free Calibration 84 | 85 | **[ICCV-2021]** 86 | [Evidential Deep Learning for Open Set Action Recognition](https://arxiv.org/abs/2107.10161). 87 |
88 | **Authors:** Wentao Bao, Qi Yu and Yu Kong 89 |
90 | **Institution:** Rochester Institute of Technology, Rochester, NY 14623, USA 91 | 92 | **[CVPR-2019]** 93 | [Deep transfer learning for multiple class novelty detection](https://openaccess.thecvf.com/content_CVPR_2019/html/Perera_Deep_Transfer_Learning_for_Multiple_Class_Novelty_Detection_CVPR_2019_paper.html) 94 |
95 | **Authors:** Pramuditha Perera, Vishal M. Patel 96 |
97 | **Institution:** Johns Hopkins University 98 | 99 | **[CVPR-2020]** 100 | [Generative-discriminative feature representations for open-set recognition](https://openaccess.thecvf.com/content_CVPR_2020/html/Perera_Generative-Discriminative_Feature_Representations_for_Open-Set_Recognition_CVPR_2020_paper.html) 101 |
102 | **Authors:** Pramuditha Perera, Vlad I. Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, Vishal M. Patel 103 |
104 | **Institution:** Johns Hopkins University; Adobe Research; University of Virginia 105 | 106 | 107 | 108 | **[arXiv-2021]** 109 | [M2iosr: Maximal mutual information open set recognition](https://arxiv.org/abs/2108.02373) 110 |
111 | **Authors:** Xin Sun, Henghui Ding, Chi Zhang, Guosheng Lin, Keck-Voon Ling 112 |
113 | **Institution:** Nanyang Technological University 114 | 115 | 116 | 117 | ### 4.1.3 Unknown Generation 118 | 119 | **[BMVC-2017]** 120 | [Generative openmax for multi-class open set classification](https://arxiv.org/abs/1707.07418) 121 |
122 | **Authors:** ZongYuan Ge, Sergey Demyanov, Zetao Chen, Rahil Garnavi 123 |
124 | **Institution:** IBM Research; Vision for Robotics Lab 125 | 126 | 127 | **[ECCV-2018]** 128 | [Open set learning with counterfactual images](https://openaccess.thecvf.com/content_ECCV_2018/html/Lawrence_Neal_Open_Set_Learning_ECCV_2018_paper.html) 129 |
130 | **Authors:** Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng-Keen Wong, Fuxin Li; 131 |
132 | **Institution:** Oregon State University 133 | 134 | 135 | **[CVPR-2021]** 136 | [Learning Placeholders for Open-Set Recognition](https://openaccess.thecvf.com/content/CVPR2021/html/Zhou_Learning_Placeholders_for_Open-Set_Recognition_CVPR_2021_paper.html) 137 |
138 | **Authors:** Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan 139 |
140 | **Institution:** Nanjing University 141 | 142 | 143 | **[TKDE-2020]** 144 | [Collective decision for open set recognition](https://ieeexplore.ieee.org/abstract/document/9023939/) 145 |
146 | **Authors:** Chuanxing Geng, Songcan Chen 147 |
148 | **Institution:** Nanjing University 149 | 150 | 151 | **[arXiv-2020]** 152 | [One-vs-rest network-based deep probabil- ity model for open set recognition](https://arxiv.org/abs/2004.08067) 153 |
154 | **Authors:** Jaeyeon Jang, Chang Ouk Kim 155 |
156 | **Institution:** Yonsei University 157 | 158 | 159 | **[EUSIPCO-2019]** 160 | [Open-set recognition using intra-class splitting](https://ieeexplore.ieee.org/abstract/document/8902738) 161 |
162 | **Authors:** Patrick Schlachter, Yiwen Liao, Bin Yang 163 |
164 | **Institution:** University of Stuttgart 165 | 166 | 167 | **[ICCV-2021]** 168 | [OpenGAN: Open-Set Recognition via Open Data Generation](https://openaccess.thecvf.com/content/ICCV2021/papers/Kong_OpenGAN_Open-Set_Recognition_via_Open_Data_Generation_ICCV_2021_paper.pdf) 169 |
170 | **Authors:** Shu Kong, Deva Ramanan 171 |
172 | **Institution:** Carnegie Mellon University; Argo AI 173 | 174 | 175 | 176 | ### 4.1.4 Label Space Redesign 177 | 178 | **[Report-2021]** 179 | [Language guided out-of-distribution detection](https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-139.pdf) 180 |
181 | **Authors:** William Gan 182 |
183 | **Institution:** UC, Berkeley 184 | 185 | 186 | **[CVPR-2018]** 187 | [Hierarchical novelty detection for visual object recognition](https://openaccess.thecvf.com/content_cvpr_2018/html/Lee_Hierarchical_Novelty_Detection_CVPR_2018_paper.html) 188 |
189 | **Authors:** Kibok Lee, Kimin Lee† Kyle Min, Yuting Zhang, Jinwoo Shin† Honglak Lee 190 |
191 | **Institution:** University of Michigan; Korea Advanced Institute of Science and Technology; Google Brain 192 | 193 | 194 | **[CVPR-2021]** 195 | [Mos: Towards scaling out-of-distribution detection for large semantic space](https://openaccess.thecvf.com/content/CVPR2021/html/Huang_MOS_Towards_Scaling_Out-of-Distribution_Detection_for_Large_Semantic_Space_CVPR_2021_paper.html) 196 |
197 | **Authors:** Rui Huang, Yixuan Li 198 |
199 | **Institution:** University of Wisconsin-Madison 200 | 201 | 202 | 203 | **[NeurIPS-2018]** 204 | [Out-of-distribution detection using multiple semantic label representations](https://arxiv.org/abs/1808.06664) 205 |
206 | **Authors:** Gabi Shalev, Yossi Adi, Joseph Keshet 207 |
208 | **Institution:** Bar-Ilan University 209 | 210 | 211 | **[arXiv-2021]** 212 | [Learning transferable visual models from natural language supervision](https://arxiv.org/abs/2103.00020) 213 |
214 | **Authors:** Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever 215 |
216 | **Institution:** OpenAI 217 | 218 | 219 | 220 | **[arXiv-2021]** 221 | [Exploring the limits of out-of-distribution detection](https://arxiv.org/abs/2106.03004) 222 |
223 | **Authors:** Stanislav Fort, Jie Ren, Balaji Lakshminarayanan 224 |
225 | **Institution:** Stanford University; Google Research 226 | 227 | 228 | 229 | 230 | 231 | ## 4.2 Distance-based Method 232 | 233 | **[TPAMI-2021]** 234 | [Adversarial Reciprocal Points Learning for Open Set Recognition](https://arxiv.org/abs/2103.00953). 235 |
236 | **Authors:** Guangyao Chen, Peixi Peng, Xiangqian Wang and Yonghong Tian 237 |
238 | **Institution:** Peking University, Peng Cheng Laboratory 239 | 240 | **[TPAMI-2020]** 241 | [Convolutional Prototype Network for Open Set Recognition](https://ieeexplore.ieee.org/document/9296325). 242 |
243 | **Authors:** Hong-Ming Yang, Xu-Yao Zhang, Fei Yin, Qing Yang and Cheng-Lin Liu 244 |
245 | **Institution:** Institute of Automation Chinese Academy of Sciences 246 | 247 | 248 | **[BMVC-2018]** 249 | [Metric learning for novelty and anomaly detection](https://arxiv.org/abs/1808.05492) 250 |
251 | **Authors:** Masana, Marc and Ruiz, Idoia and Serrat, Joan and van de Weijer, Joost and Lopez, Antonio M 252 |
253 | **Institution:** Universitat Autonoma de Barcelona, Bellaterra, Spain 254 | 255 | 256 | **[Report]** 257 | [p-odn: prototype- based open deep network for open set recognition](https://www.nature.com/articles/s41598-020-63649-6) 258 |
259 | **Authors:** Yu Shu, Yemin Shi, Yaowei Wang, Tiejun Huang, Yonghong Tian 260 |
261 | **Institution:** Peking University; Peng Cheng Laboratory 262 | 263 | 264 | **[CVPR-2020]** 265 | [Few-shot open-set recognition using meta-learning](https://openaccess.thecvf.com/content_CVPR_2020/html/Liu_Few-Shot_Open-Set_Recognition_Using_Meta-Learning_CVPR_2020_paper.html) 266 |
267 | **Authors:** Bo Liu, Hao Kang, Haoxiang Li, Gang Hua, Nuno Vasconcelos 268 |
269 | **Institution:** Wormpex AI Research; UC, San Diego 270 | 271 | 272 | **[ECCV-2020]** 273 | [Learning open set network with discriminative reciprocal points](https://link.springer.com/chapter/10.1007%2F978-3-030-58580-8_30) 274 |
275 | **Authors:** Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, Yonghong Tian 276 |
277 | **Institution:** Peking University; Beihang University; Peng Cheng Laboratory; Hikvision Research Institute 278 | 279 | 280 | 281 | 282 | **[CVPR-2019]** 283 | [Classification-reconstruction learning for open-set recognition](https://openaccess.thecvf.com/content_CVPR_2019/html/Yoshihashi_Classification-Reconstruction_Learning_for_Open-Set_Recognition_CVPR_2019_paper.html) 284 |
285 | **Authors:** Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You2, Makoto Iida, Takeshi Naemura1 286 |
287 | **Institution:** The University of Tokyo; Data61-CSIRO 288 | 289 | 290 | 291 | **[AAAI-2020]** 292 | [Open-set recognition with gaussian mixture variational autoencoders](https://www.aaai.org/AAAI21Papers/AAAI-3823.CaoA.pdf) 293 |
294 | **Authors:** Alexander Cao, Yuan Luo, Diego Klabjan 295 |
296 | **Institution:** Northwestern University 297 | 298 | 299 | **[Machine Learning-2017]** 300 | [Nearest neighbors distance ratio open-set classifier](https://link.springer.com/article/10.1007/s10994-016-5610-8) 301 |
302 | **Authors:** Pedro R. Mendes Junior, Rafael de O. Werneck, Bernardo V. Stein, Daniel V. Pazinato, Waldir R. de Almeida, Otavio A. B. Penatti, Ricardo da S. Torres, Anderson Rocha, Roberto M. de Souza, Otavio A. B. Penatti 303 |
304 | **Institution:** University of Campinas; SAMSUNG Research Institute 305 | 306 | 307 | 308 | 309 | 310 | ## 4.3 Reconstruction 311 | 312 | 313 | ### 4.3.1 Sparse Representation 314 | 315 | **[TPAMI-2016]** 316 | [Sparse representation-based open set recognition](https://ieeexplore.ieee.org/abstract/document/7577876) 317 |
318 | **Authors:** He Zhang, Vishal M. Patel 319 |
320 | **Institution:** Rutgers University 321 | >
322 | > SROSR models the tails of the matched and sum of non-matched reconstruction error distributions. 323 | >

324 | > This method model the tail of the above two error distributions using the statistical EVT, and the confidence scores corresponding to the tail distributions of a novel test sample are then fused to determine its identity. Notice that the hidden embedding during reconstruction is regularized by sparsity. 325 | >

326 | >
327 | 328 | **[CVPR-2013]** 329 | [Kernel null space methods for novelty detection](https://openaccess.thecvf.com/content_cvpr_2013/html/Bodesheim_Kernel_Null_Space_2013_CVPR_paper.html) 330 |
331 | **Authors:** Paul Bodesheim, Alexander Freytag, Erik Rodner, Michael Kemmler, Joachim Denzler 332 |
333 | **Institution:** University Jena; UC Berkeley 334 | 335 | 336 | **[CVPR-2017]** 337 | [Incremental kernel null space discriminant analysis for novelty detection](https://openaccess.thecvf.com/content_cvpr_2017/html/Liu_Incremental_Kernel_Null_CVPR_2017_paper.html) 338 |
339 | **Authors:** Juncheng Liu, Zhouhui Lian, Yi Wang, Jianguo Xiao 340 |
341 | **Institution:** Peking University; Dalian University 342 | 343 | 344 | 345 | 346 | 347 | ### 4.3.2 Reconstruction-Error 348 | 349 | 350 | **[CVPR-2019]** 351 | [C2AE: Class Conditioned Auto-Encoder for Open-Set Recognition](https://openaccess.thecvf.com/content_CVPR_2019/html/Oza_C2AE_Class_Conditioned_Auto-Encoder_for_Open-Set_Recognition_CVPR_2019_paper.html) 352 |
353 | **Authors:** Poojan Oza, Vishal M. Patel 354 |
355 | **Institution:** Johns Hopkins University 356 | 357 | 358 | 359 | **[CVPR-2020]** 360 | [Conditional gaussian distribution learning for open set recognition](https://openaccess.thecvf.com/content_CVPR_2020/html/Sun_Conditional_Gaussian_Distribution_Learning_for_Open_Set_Recognition_CVPR_2020_paper.html) 361 |
362 | **Authors:** Xin Sun, Zhenning Yang, Chi Zhang, Keck-Voon Ling, Guohao Peng 363 |
364 | **Institution:** Nanyang Technological University 365 | 366 | 367 | **[CVPR-2021]** 368 | [Counterfactual zero-shot and open-set visual recognition](https://openaccess.thecvf.com/content/CVPR2021/html/Yue_Counterfactual_Zero-Shot_and_Open-Set_Visual_Recognition_CVPR_2021_paper.html) 369 |
370 | **Authors:** Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, Hanwang Zhang 371 |
372 | **Institution:** Nanyang Technological University; Singapore Management University; Alibaba Damo Academy 373 | 374 | 375 | 376 | **[ECCV-2020]** 377 | [Open-set adversarial defense](https://link.springer.com/chapter/10.1007%2F978-3-030-58520-4_40) 378 |
379 | **Authors:** Rui Shao, Pramuditha Perera, Pong C. Yuen, Vishal M. Patel 380 |
381 | **Institution:** Hong Kong Baptist University; AWS AI Labs; Johns Hopkins University 382 | -------------------------------------------------------------------------------- /3_AD.md: -------------------------------------------------------------------------------- 1 | 2 | # 3. Anomaly Detection & One-Class Novelty Detection 3 | - [3.1 Density-based Methods](#3.1) 4 | - [3.1.1 Classic Density Estimation](#3.1.1) 5 | - [3.1.2 NN-based Density Estimation](#3.1.2) 6 | - [3.1.3 Energy-based Models](#3.1.3) 7 | - [3.1.4 Frequency-based Methods](#3.1.4) 8 | - [3.2 Reconstruction-based Methods](#3.2) 9 | - [3.2.1 Sparse Representation Methods](#3.2.1) 10 | - [3.2.2 Reconstruction-Error Methods](#3.2.2) 11 | - [3.3 Classification-based Methods](#3.3) 12 | - [3.3.1 One-Class Classification](#3.3.1) 13 | - [3.3.2 Positive-Unlabeled Learning](#3.3.2) 14 | - [3.3.3 Self-Supervised Learning](#3.3.3) 15 | - [3.4 Distance-based Methods](#3.4) 16 | - [3.5 Gradient-based Methods](#3.5) 17 | - [3.6 Discussion and Theoretical Analysis](#3.6) 18 | 19 | 20 | ## 3.1 Density-based Methods 21 | 22 | ### 3.1.1 Classic Density Estimation ### 23 | **[TPAMI-1998]** 24 | [Parametric model fitting: from inlier characterization to outlier detection](https://ieeexplore.ieee.org/abstract/document/667884) 25 |
26 | **Authors:** Gaudenz Danuser, M. Stricker 27 |
28 | **Institution:** Marine Biological Laboratory; Analytical, and Mathematical Services 29 | 30 | 31 | **[JESP-2018]** 32 | [Detecting multivariate outliers: Use a robust variant of the mahalanobis distance](https://www.sciencedirect.com/science/article/abs/pii/S0022103117302123#!) 33 |
34 | **Authors:** Christophe Leys, Olivier Klein, Yves Dominicy 35 |
36 | **Institution:** University libre de Bruxelles; Ghent University 37 | 38 | 39 | **[ICML-2000]** 40 | [Anomaly detection over noisy data using learned probability distributions](https://academiccommons.columbia.edu/doi/10.7916/D8C53SKF) 41 |
42 | **Authors:** Eskin Eleazar 43 |
44 | **Institution:** Columbia University 45 | 46 | 47 | **[ISI-2016]** 48 | [Poisson factorization for peer-based anomaly detection](https://ieeexplore.ieee.org/abstract/document/7745472) 49 |
50 | **Authors:** Melissa Turcotte, Juston Moore, Nick Heard, Aaron McPhall 51 |
52 | **Institution:** Los Alamos National Laboratory; University of Bristol 53 | 54 | 55 | **[JASA-1991]** 56 | [Review papers: Recent developments in non-parametric density estimation](https://www.tandfonline.com/doi/abs/10.1080/01621459.1991.10475021) 57 |
58 | **Authors:** Alan Julian Izenman 59 |
60 | **Institution:** Temple University 61 | 62 | 63 | **[TKDE-2018]** 64 | [Anomaly detection using local kernel density estimation and context-based regression](https://ieeexplore.ieee.org/abstract/document/8540843) 65 |
66 | **Authors:** Weiming Hu, Jun Gao, Bing Li, Ou Wu, Junping Du, Stephen Maybank 67 |
68 | **Institution:** Chinese Academy of Sciences; University of Chinese Academy of Sciences; Tianjin University; Birkbeck College 69 | 70 | 71 |
72 | 73 | [Back to Top](#top) 74 | 75 |
76 | 77 | 78 | ### 3.1.2 NN-based Density Est. ### 79 | **[ICLR-2018]** 80 | [Deep autoencoding gaussian mixture model for Deep autoencoding gaussian mixture model for unsupervised anomaly detection](https://openreview.net/forum?id=BJJLHbb0-) 81 |
82 | **Authors:** Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, Haifeng Chen 83 |
84 | **Institution:** Washington State University; NEC Laboratories America 85 | 86 | 87 | 88 | **[CVPR-2019]** 89 | [Latent Space Autoregression for Novelty Detection](https://openaccess.thecvf.com/content_CVPR_2019/html/Abati_Latent_Space_Autoregression_for_Novelty_Detection_CVPR_2019_paper.html) 90 |
91 | **Authors:** Davide Abati, Angelo Porrello, Simone Calderara, Rita Cucchiara 92 |
93 | **Institution:** University of Modena and Reggio Emilia 94 | 95 | **[NeurIPS-2018]** 96 | [Generative probabilistic novelty detection with adversarial autoencoders](https://arxiv.org/abs/1807.02588) 97 |
98 | **Authors:** Stanislav Pidhorskyi, Ranya Almohsen, Donald A Adjeroh, Gianfranco Doretto 99 |
100 | **Institution:** West Virginia University 101 | 102 | 103 | **[ECMLPKDD-2018]** 104 | [Image anomaly detection with generative adversarial networks](https://link.springer.com/chapter/10.1007/978-3-030-10925-7_1) 105 |
106 | **Authors:** Lucas Deecke, Robert VandermeulenLukas, RuffStephan Mandt, Marius Kloft 107 |
108 | **Institution:** University of EdinburghEdinburghScotland; TU Kaiserslautern; Hasso Plattner Institute; University of California 109 | 110 | 111 | **[ICML-2015]** 112 | [Variational inference with normalizing flows](http://proceedings.mlr.press/v37/rezende15.html) 113 |
114 | **Authors:** Danilo Rezende, Shakir Mohamed 115 |
116 | **Institution:** Google DeepMind 117 | > 118 | 119 | **[TPAMI-2020]** 120 | [Normalizing flows: An introduction and review of current methods](https://ieeexplore.ieee.org/abstract/document/9089305) 121 |
122 | **Authors:** Ivan Kobyzev, Simon J.D. Prince, Marcus A. Brubaker 123 |
124 | **Institution:** Borealis AI 125 | 126 | 127 | **[CVPR-2021]** 128 | [Cutpaste: Self-supervised learning for anomaly detection and localization](https://openaccess.thecvf.com/content/CVPR2021/html/Li_CutPaste_SelfSupervised_Learning_for_Anomaly_Detection_and_Localization_CVPR_2021_paper.html) 129 |
130 | **Authors:** Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, Tomas Pfister 131 |
132 | **Institution:** Google Cloud AI Research 133 | 134 | 135 | **[CVPR-2021]** 136 | [Multiresolution knowledge distillation for anomaly detection](https://openaccess.thecvf.com/content/CVPR2021/html/Salehi_Multiresolution_Knowledge_Distillation_for_Anomaly_Detection_CVPR_2021_paper.html) 137 |
138 | **Authors:** Mohammadreza Salehi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad H. Rohban, Hamid R. Rabiee 139 |
140 | **Institution:** Sharif University of Technology 141 | 142 | 143 | **[NeurIPS-2018]** 144 | [A loss framework for calibrated anomaly detection](https://proceedings.neurips.cc/paper/2018/file/959a557f5f6beb411fd954f3f34b21c3-Paper.pdf) 145 |
146 | **Authors:** Aditya Krishna Menon, Robert C. Williamson 147 |
148 | **Institution:** Australian National University 149 | 150 | 151 | **[CVPR-2021]** 152 | [Multiattentional deepfake detection](https://openaccess.thecvf.com/content/CVPR2021/html/Zhao_Multi-Attentional_Deepfake_Detection_CVPR_2021_paper.html) 153 |
154 | **Authors:** Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, Nenghai Yu 155 |
156 | **Institution:** University of Science and Technology of China; Microsoft Cloud AI 157 | 158 | **[AAAI-2020]** 159 | [Ml-loo:Detecting adversarial examples with feature attribution](https://ojs.aaai.org/index.php/AAAI/article/view/6140) 160 |
161 | **Authors:** Puyudi Yang, Jianbo Chen, Cho-Jui Hsieh, Jane-Ling Wang, Michael Jordan 162 |
163 | **Institution:** University of California 164 | 165 | 166 | **[CIKM-2020]** 167 | [Towards generalizable deepfake detection with locality-aware autoencoder](https://dl.acm.org/doi/abs/10.1145/3340531.3411892) 168 |
169 | **Authors:** Mengnan Du, Shiva Pentyala, Yuening Li, Xia Hu 170 |
171 | **Institution:** Texas A&M University 172 | 173 | 174 |
175 | 176 | [Back to Top](#top) 177 | 178 |
179 | 180 | 181 | 182 | ### 3.1.3 Energy-based Models ### 183 | **[ICML-2016]** 184 | [Deep structured energy based models for anomaly detection](http://proceedings.mlr.press/v48/zhai16.html) 185 |
186 | **Authors:** Shuangfei Zhai, Yu Cheng, Weining Lu, Zhongfei Zhang 187 |
188 | **Institution:** Binghamton Univeristy; IBM T. J. Watson Research Center; Tsinghua University 189 | 190 | 191 | **[2005]** 192 | [Estimation of non-normalized statistical models by score matching](https://www.jmlr.org/papers/volume6/hyvarinen05a/hyvarinen05a.pdf) 193 |
194 | **Authors:** Aapo Hyv¡§arinen 195 |
196 | **Institution:** BHelsinki Institute for Information Technology 197 | 198 | **[ICML-2011]** 199 | [Bayesian learning via stochastic gradient langevin dynamics](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441.3813&rep=rep1&type=pdf) 200 |
201 | **Authors:** Max Welling, Yee Whye Teh 202 |
203 | **Institution:** University of California; UCL 204 | 205 |
206 | 207 | [Back to Top](#top) 208 | 209 |
210 | 211 | 212 | ### 3.1.4 Frequency-based Models ### 213 | 214 | **[CVPR-2020]** 215 | [High-frequency component helps explain the generalization of convolutional neural networks](https://openaccess.thecvf.com/content_CVPR_2020/html/Wang_High-Frequency_Component_Helps_Explain_the_Generalization_of_Convolutional_Neural_Networks_CVPR_2020_paper.html) 216 |
217 | **Authors:** Haohan Wang, Xindi Wu, Zeyi Huang, Eric P. Xing 218 |
219 | **Institution:** UCarnegie Mellon University 220 | 221 | **[CNeurIPS-2019]** 222 | [Adversarial examples are not bugs, they are features](https://arxiv.org/abs/1905.02175) 223 |
224 | **Authors:** Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, Aleksander Madry 225 |
226 | **Institution:** MIT 227 | 228 | 229 | **[ICCV-2021]** 230 | [Amplitudephase recombination: Rethinking robustness of convolutional neural networks in frequency domain](https://openaccess.thecvf.com/content/ICCV2021/html/Chen_AmplitudePhase_Recombination_Rethinking_Robustness_of_Convolutional_Neural_Networks_in_Frequency_ICCV_2021_paper.html) 231 |
232 | **Authors:** Guangyao Chen, Peixi Peng, Li Ma, Jia Li, Lin Du, Yonghong Tian 233 |
234 | **Institution:** Peking University; Beihang University; AI Application Research Center Huawei 235 | 236 | **[CVPR-2021]** 237 | [Spatial-phase shallow learning: rethinking face forgery detection in frequency domain](https://openaccess.thecvf.com/content/CVPR2021/html/Liu_Spatial-Phase_Shallow_Learning_Rethinking_Face_Forgery_Detection_in_Frequency_Domain_CVPR_2021_paper.html) 238 |
239 | **Authors:** Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, Nenghai Yu 240 |
241 | **Institution:** University of Science and Technology of China; Alibaba Group 242 | 243 | 244 |
245 | 246 | [Back to Top](#top) 247 | 248 |
249 | 250 | 251 | ## 3.2 Reconstruction-based Methods 252 | 253 | 254 | ### 3.2.1 Sparse Representation 255 | 256 | [//]: 106 257 | **[J. Signal Process. Syst.-2015]** 258 | [Sparse coding with anomaly detection](https://link.springer.com/article/10.1007/s11265-014-0913-0). 259 |
260 | **Authors:** Amir Adler, Michael Elad, Yacov Hel-Or, Ehud Rivlin 261 |
262 | **Institution:** Technion 263 | 264 | 265 | [//]: 107 266 | **[Multimedia Tools and Applications-2017]** 267 | [Anomaly detection using sparse reconstruction in crowded scenes](https://link.springer.com/article/10.1007/s11042-016-4115-6). 268 |
269 | **Authors:** Ang Li, Zhenjiang Miao, Yigang Cen, Yi Cen 270 |
271 | **Institution:** Beijing Jiaotong University, Beijing Key Laboratory, Minzu University of China 272 | 273 | 274 | [//]: 108 275 | **[IEEE-2014]** 276 | [Adaptive Sparse Representations for Video Anomaly Detection](https://ieeexplore.ieee.org/abstract/document/6587741). 277 |
278 | **Authors:** Xuan Mo, Vishal Monga, Raja Bala, Zhigang Fan 279 |
280 | **Institution:** Pennsylvania State University 281 | 282 | 283 | [//]: 109 284 | **[Pattern Recognition-2013]** 285 | [AticleL1 norm based kpca for novelty detection](https://www.sciencedirect.com/science/article/pii/S0031320312002877). 286 |
287 | **Authors:** Yingchao Xiao, Huangang Wanga, Wenli Xu, Junwu Zhou 288 |
289 | **Institution:** Tsinghua University, Beijing General Research Institute of Mining & Metallurgy 290 | 291 | 292 | [//]: 110 293 | **[AAAI-2021]** 294 | [Lren: Low-rank embedded network for sample-free hyperspectral anomaly detection](https://www.aaai.org/AAAI21Papers/AAAI-766.JiangK.pdf). 295 |
296 | **Authors:** Kai Jiang, Weiying Xie, Jie Lei, Tao Jiang, Yunsong Li 297 |
298 | **Institution:** Xidian University 299 | 300 | 301 |
302 | 303 | [Back to Top](#top) 304 | 305 |
306 | 307 | 308 | 309 | ### 3.2.2 Reconstruction-Error Methods 310 | 311 | [//]:89 312 | **[NeurIPS-2018]** 313 | [Generative probabilistic novelty detection with adversarial autoencoders](https://arxiv.org/abs/1807.02588). 314 |
315 | **Authors:** Stanislav Pidhorskyi, Ranya Almohsen, Donald A Adjeroh, Gianfranco Doretto 316 |
317 | **Institution:** West Virginia University 318 | 319 | 320 | [//]: 111 321 | **[Wireless Telecommunications Symposium-2018]** 322 | [Autoencoderbased network anomaly detection](https://ieeexplore.ieee.org/abstract/document/8363930). 323 |
324 | **Authors:** Zhaomin Chen, Chai Kiat Yeo, Bu Sung Lee, Chiew Tong Lau 325 |
326 | **Institution:** Nanyang Technological University 327 | 328 | 329 | [//]: 112 330 | **[Special Lecture on IE-2015]** 331 | [Variational autoencoder based anomaly detection using reconstruction probability](http://dm.snu.ac.kr/static/docs/TR/SNUDM-TR-2015-03.pdf). 332 |
333 | **Authors:** J. An and S. Cho 334 |
335 | **Institution:** cannot open 336 | 337 | 338 | [//]: 113 339 | **[ICLR-W-2018]** 340 | [Efficient GAN-Based Anomaly Detection](https://arxiv.org/abs/1802.06222). 341 |
342 | **Authors:** Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, Vijay Ramaseshan Chandrasekhar 343 |
344 | **Institution:** CentraleSup¡äelec, Nanyang Technological University, Carnegie Mellon University, Institute for Infocomm Research 345 | 346 | 347 | [//]: 114 348 | **[CVPR-2018]** 349 | [Future frame prediction for anomaly detection¨Ca new baseline](http://openaccess.thecvf.com/content_cvpr_2018/html/Liu_Future_Frame_Prediction_CVPR_2018_paper.html). 350 |
351 | **Authors:** Wen Liu, Weixin Luo, Dongze Lian, Shenghua Gao 352 |
353 | **Institution:** ShanghaiTech University 354 | 355 | 356 | [//]: 115 357 | **[CVPR-2019]** 358 | [Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection](https://openaccess.thecvf.com/content_ICCV_2019/html/Gong_Memorizing_Normality_to_Detect_Anomaly_Memory-Augmented_Deep_Autoencoder_for_Unsupervised_ICCV_2019_paper.html). 359 |
360 | **Authors:** Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, Anton van den Hengel 361 |
362 | **Institution:** University of Adelaide, Deakin University, University of Western Australia 363 | 364 | 365 | [//]: 116 366 | **[CVPR-2020]** 367 | [Learning Memory Guided Normality for Anomaly Detection](https://arxiv.org/abs/2101.12382). 368 |
369 | **Authors:** Kevin Stephen, Varun Menon 370 |
371 | **Institution:** Department of Information Technology, Pune Institute of Computer Technology, New York University 372 | 373 | 374 | 375 | [//]: 117 376 | **[ICLR-2020]** 377 | [Robust subspace recovery layer for unsupervised anomaly detection](https://arxiv.org/abs/1904.00152). 378 |
379 | **Authors:** Chieh-Hsin Lai, Dongmian Zou, Gilad Lerman 380 |
381 | **Institution:** School of Mathematics University of Minnesota 382 | 383 | 384 | [//]: 118 385 | **[AAAI-2021]** 386 | [Learning semantic context from normal samples for unsupervised anomaly detection](https://www.aaai.org/AAAI21Papers/AAAI-4221.YanX.pdf). 387 |
388 | **Authors:** Xudong Yan, Huaidong Zhang, Xuemiao Xu1, Xiaowei Hu, Pheng-Ann Heng 389 |
390 | **Institution:** South China University of Technology, Ministry of Education Key Laboratory of Big Data and Intelligent Robot, Guangdong Provincial Key Lab of Computational Intelligence and Cyberspace Information 391 | 392 | 393 | [//]: 119 394 | **[ICML-2019]** 395 | [Anomaly detection with multiple-hypotheses predictions](http://proceedings.mlr.press/v97/nguyen19b.html). 396 |
397 | **Authors:** Duc Tam Nguyen, Zhongyu Lou, Michael Klar, Thomas Brox 398 |
399 | **Institution:** University of Freiburg, Germany Corporate Research 400 | 401 | 402 | [//]: 120 403 | **[AAAI-2019]** 404 | [Learning competitive and discriminative reconstructions for anomaly detection](https://ojs.aaai.org/index.php/AAAI/article/view/4451). 405 |
406 | **Authors:** Kai Tian, Shuigeng Zhou, Jianping Fan, Jihong Guan 407 |
408 | **Institution:** Fudan University, University of North Carolina at Charlotte, Tongji University 409 | 410 | 411 | [//]: 121 412 | **[CVPR-2018]** 413 | [Adversarially learned one-class classifier for novelty detection](https://openaccess.thecvf.com/content_cvpr_2018/html/Sabokrou_Adversarially_Learned_One-Class_CVPR_2018_paper.html). 414 |
415 | **Authors:** Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, Ehsan Adeli 416 |
417 | **Institution:** Institute for Research in Fundamental Sciences, Amirkabir University of Technology, Stanford University 418 | 419 | 420 | 421 | [//]: 122 422 | **[IEEE/CVF-2019]** 423 | [Ocgan: One-class novelty detection using gans with constrained latent representations](http://openaccess.thecvf.com/content_CVPR_2019/html/Perera_OCGAN_One-Class_Novelty_Detection_Using_GANs_With_Constrained_Latent_Representations_CVPR_2019_paper.html). 424 |
425 | **Authors:** Pramuditha Perera, Ramesh Nallapati, Bing Xiang 426 |
427 | **Institution:** Johns Hopkins University, AWS AI 428 | 429 | 430 | 431 | [//]: 123 432 | **[ECCV-2020]** 433 | [Encoding structure-texture relation with p-net for anomaly detection in retinal images](https://arxiv.org/pdf/2008.03632). 434 |
435 | **Authors:** Kang Zhou, Yuting Xiao, Jianlong Yang, Jun Cheng, Wen Liu, Weixin Luo, Zaiwang Gu, Jiang Liu, Shenghua Gao 436 |
437 | **Institution:** ShanghaiTech University, Chinese Academy of Sciences, Southern University, Shanghai Engineering Research Center of Intelligent Vision and Imaging 438 | 439 | 440 | 441 | [//]: 124 442 | **[arXiv preprint arXiv-2020]** 443 | [Gan ensemble for anomaly detection](https://www.aaai.org/AAAI21Papers/AAAI-1883.HanX.pdf). 444 |
445 | **Authors:** Xu Han, Xiaohui Chen, Li-Ping Liu 446 |
447 | **Institution:** Tufts University 448 | 449 | 450 |
451 | 452 | [Back to Top](#top) 453 | 454 |
455 | 456 | 457 | 458 | ## 3.1 Classification-based Methods 459 | 460 | ### 3.3.1 One-Class Classification 461 | **[Journal of Artificial Intelligence Research-2002]** 462 | [One-class classification: Concept learning in the absence of counter-examples.](https://elibrary.ru/item.asp?id=5230402) 463 |
464 | **Authors:** Tax, David Martinus Johannes 465 |
466 | **Institution:** TU Delft 467 | 468 | 469 | 470 | **[ICML-2018]** 471 | [Deep one-class classification](http://proceedings.mlr.press/v80/ruff18a) 472 |
473 | **Authors:** Ruff, Lukas and Vandermeulen, Robert and Goernitz, Nico and Deecke, Lucas and Siddiqui, Shoaib Ahmed and Binder, Alexander and Muller, Emmanuel and Kloft, Marius 474 |
475 | **Institution:** Humboldt University of Berlin; Hasso Plattner Institute; TU Kaiserslautern; TU Berlin; University of Edinburgh; Singapore University of Technology and Design 476 | 477 | 478 | **[CVPR-2021]** 479 | [PANDA: Adapting Pretrained Features for Anomaly Detection and Segmentation](https://openaccess.thecvf.com/content/CVPR2021/html/Reiss_PANDA_Adapting_Pretrained_Features_for_Anomaly_Detection_and_Segmentation_CVPR_2021_paper.html) 480 |
481 | **Authors:** Reiss, Tal and Cohen, Niv and Bergman, Liron and Hoshen, Yedid 482 |
483 | **Institution:** The Hebrew University of Jerusalem 484 | 485 | 486 | **[CVPR-2019]** 487 | [Gods: Generalized one-class discriminative subspaces for anomaly detection](https://openaccess.thecvf.com/content_ICCV_2019/html/Wang_GODS_Generalized_One-Class_Discriminative_Subspaces_for_Anomaly_Detection_ICCV_2019_paper.html) 488 |
489 | **Authors:** Wang, Jue and Cherian, Anoop 490 |
491 | **Institution:** Australian National University; Mitsubishi Electric Research Labs 492 | 493 | 494 |
495 | 496 | [Back to Top](#top) 497 | 498 |
499 | 500 | 501 | 502 | ### 3.3.2 Positive-Unlabeled Learning 503 | **[Machine Learning-2020]** 504 | [Learning from positive and unlabeled data: A survey](https://link.springer.com/article/10.1007/s10994-020-05877-5) 505 |
506 | **Authors:** Bekker, Jessa and Davis, Jesse 507 |
508 | **Institution:** KU Leuven 509 | 510 | 511 | **[International Symposiums on Information Processing-2008]** 512 | [Learning from positive and unlabeled examples: A survey](https://ieeexplore.ieee.org/abstract/document/4554167) 513 |
514 | **Authors:** Zhang, Bangzuo and Zuo, Wanli 515 |
516 | **Institution:** Jilin University; Northeast Normal University 517 | 518 | 519 | **[International Conference on Information, Intelligence, Systems and Applications-2019]** 520 | [Positive and unlabeled learning algorithms and applications: A survey](https://ieeexplore.ieee.org/abstract/document/8900698) 521 |
522 | **Authors:** Jaskie, Kristen and Spanias, Andreas 523 |
524 | **Institution:** Arizona State University 525 | 526 | 527 | 528 | **[IJCAI-2003]** 529 | [Learning to classify texts using positive and unlabeled data](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.9914&rep=rep1&type=pdf) 530 |
531 | **Authors:** Li, Xiaoli and Liu, Bing 532 |
533 | **Institution:** National University of Singapore; University of Illinois at Chicago 534 | 535 | 536 | **[Bioinformatics-2006]** 537 | [PSoL: a positive sample only learning algorithm for finding non-coding RNA genes](https://academic.oup.com/bioinformatics/article/22/21/2590/250725?login=true) 538 |
539 | **Authors:** Wang, Chunlin and Ding, Chris and Meraz, Richard F and Holbrook, Stephen R 540 |
541 | **Institution:** Lawrence Berkeley National Laboratory 542 | 543 | 544 | **[ICONIP-2012]** 545 | [Learning from positive and unlabelled examples using maximum margin clustering](https://link.springer.com/chapter/10.1007/978-3-642-34487-9_56) 546 |
547 | **Authors:** Chaudhari, Sneha and Shevade, Shirish 548 |
549 | **Institution:** IBM Research; Indian Institute of Science 550 | 551 | 552 | **[Journal of Computers-2009]** 553 | [Reliable Negative Extracting Based on kNN for Learning from Positive and Unlabeled Examples.](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.415.7161&rep=rep1&type=pdf) 554 |
555 | **Authors:** Zhang, Bangzuo and Zuo, Wanli 556 |
557 | **Institution:** Jilin University; Northeast Normal University 558 | 559 | 560 | **[Journal of Information Science and Engineering-2014]** 561 | [Clustering-based Method for Positive and Unlabeled Text Categorization Enhanced by Improved TFIDF.](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.684.162&rep=rep1&type=pdf) 562 |
563 | **Authors:** Liu, Lu and Peng, Tao 564 |
565 | **Institution:** University of Illinois at Urbana-Champaign Urbana; Jilin University 566 | 567 | 568 | **[arXiv-2018]** 569 | [Instance-dependent pu learning by bayesian optimal relabeling](https://arxiv.org/abs/1808.02180) 570 |
571 | **Authors:** He, Fengxiang and Liu, Tongliang and Webb, Geoffrey I and Tao, Dacheng 572 |
573 | **Institution:** University of Sydney 574 | 575 | 576 | **[AAAI-2019]** 577 | [Learning competitive and discriminative reconstructions for anomaly detection](https://ojs.aaai.org/index.php/AAAI/article/view/4451) 578 |
579 | **Authors:** Tian, Kai and Zhou, Shuigeng and Fan, Jianping and Guan, Jihong 580 |
581 | **Institution:** Fudan University; University of North Carolina; Tongji University 582 | 583 | 584 | **[CVPR-2019]** 585 | [Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection](https://openaccess.thecvf.com/content_CVPR_2019/html/Zhong_Graph_Convolutional_Label_Noise_Cleaner_Train_a_Plug-And-Play_Action_Classifier_CVPR_2019_paper.html) 586 |
587 | **Authors:** Zhong, Jia-Xing and Li, Nannan and Kong, Weijie and Liu, Shan and Li, Thomas H and Li, Ge 588 |
589 | **Institution:** Peking University 590 | 591 | 592 | **[ICML-2015]** 593 | [Learning from corrupted binary labels via class-probability estimation](http://proceedings.mlr.press/v37/menon15.html) 594 |
595 | **Authors:** Menon, Aditya and Van Rooyen, Brendan and Ong, Cheng Soon and Williamson, Bob 596 |
597 | **Institution:** National ICT Australia; The Australian National University 598 | 599 | 600 | **[Artificial Intelligence and Statistics-2015]** 601 | [A rate of convergence for mixture proportion estimation, with application to learning from noisy labels](http://proceedings.mlr.press/v38/scott15.html) 602 |
603 | **Authors:** Scott, Clayton 604 |
605 | **Institution:** University of Michigan 606 | 607 |
608 | 609 | [Back to Top](#top) 610 | 611 |
612 | 613 | 614 | ### 3.3.3 Self-Supervised Learning 615 | **[ICDM-2008]** 616 | [Isolation forest](https://ieeexplore.ieee.org/abstract/document/4781136) 617 |
618 | **Authors:** Liu, Fei Tony and Ting, Kai Ming and Zhou, Zhi-Hua 619 |
620 | **Institution:** Monash University; Nanjing University 621 | 622 | 623 | **[NeurIPS-2018]** 624 | [Deep anomaly detection using geometric transformations](https://arxiv.org/abs/1805.10917) 625 |
626 | **Authors:** Golan, Izhak and El-Yaniv, Ran 627 |
628 | **Institution:** Israel Institute of Technology 629 | 630 | 631 | **[ICLR-2020]** 632 | [Classification-based anomaly detection for general data](https://arxiv.org/abs/2005.02359) 633 |
634 | **Authors:** Bergman, Liron and Hoshen, Yedid 635 |
636 | **Institution:** The Hebrew University of Jerusalem 637 | 638 | 639 | **[NeurIPS-2020]** 640 | [Csi: Novelty detection via contrastive learning on distributionally shifted instances](https://arxiv.org/abs/2007.08176) 641 |
642 | **Authors:** Tack, Jihoon and Mo, Sangwoo and Jeong, Jongheon and Shin, Jinwoo 643 |
644 | **Institution:** KAIST 645 | 646 | 647 | 648 | **[CVPR-2021]** 649 | [Anomaly detection in video via self-supervised and multi-task learning](https://openaccess.thecvf.com/content/CVPR2021/html/Georgescu_Anomaly_Detection_in_Video_via_Self-Supervised_and_Multi-Task_Learning_CVPR_2021_paper.html) 650 |
651 | **Authors:** Georgescu, Mariana-Iuliana and Barbalau, Antonio and Ionescu, Radu Tudor and Khan, Fahad Shahbaz and Popescu, Marius and Shah, Mubarak 652 |
653 | **Institution:** University of Bucharest; Abu Dhabi; SecurifAI; University of Central Florida 654 | 655 | 656 | **[CVPR-2019]** 657 | [Object-centric auto-encoders and dummy anomalies for abnormal event detection in video](https://openaccess.thecvf.com/content_CVPR_2019/html/Ionescu_Object-Centric_Auto-Encoders_and_Dummy_Anomalies_for_Abnormal_Event_Detection_in_CVPR_2019_paper.html) 658 |
659 | **Authors:** Ionescu, Radu Tudor and Khan, Fahad Shahbaz and Georgescu, Mariana-Iuliana and Shao, Ling 660 |
661 | **Institution:** IIAI; University of Bucharest; SecurifAI 662 | 663 |
664 | 665 | [Back to Top](#top) 666 | 667 |
668 | 669 | 670 | ## 3.4 Distance-based Methods 671 | 672 | **[PHM Society European Conference, 2014]** 673 | [Anomaly detection using self-organizing maps-based k-nearest neighbor algorithm](https://papers.phmsociety.org/index.php/phme/article/download/1554/522) 674 |
675 | **Authors:** J. Tian, M. H. Azarian, and M. Pecht 676 |
677 | **Institution:** Center for Advanced Life Cycle Engineering (CALCE), University of Maryland, College Park, MD, 20742, U.S.A. 678 | 679 | 680 | **[GI/ITG Workshop MMBnet, pp. 13¨C14, 2007]** 681 | [Traffic anomaly detection using k-means clustering](https://www.net.in.tum.de/projects/dfg-lupus/files/muenz07k-means.pdf) 682 |
683 | **Authors:** G. Munz, S. Li, and G. Carle 684 |
685 | **Institution:** Wilhelm Schickard Institute for Computer Science; University of Tuebingen, Germany 686 | 687 | 688 | **[International conference on networked digital technologies, pp. 135¨C145, Springer,2012]** 689 | [Unsupervised clustering approach for network anomaly detection](https://eprints.soton.ac.uk/338221/1/Unsupervised_Clustering_and_Outlier_Detection_approach_for_network_anomaly_detection_-_camera_ready_new.pdf) 690 |
691 | **Authors:** I. Syarif, A. Prugel-Bennett, and G. Wills 692 |
693 | **Institution:** School of Electronics and Computer Science, University of Southampton, UK; Eletronics Engineering Polytechnics Institute of Surabaya, Indonesia 694 | 695 | 696 | 697 | ## 3.5 Gradient-based Methods 698 | 699 | **[ECCV-2020]** 700 | [Back-propagated gradient representations for anomaly detection](https://arxiv.org/pdf/2007.09507) 701 |
702 | **Authors:** G. Kwon, M. Prabhushankar, D. Temel, and G. AlRegib 703 |
704 | **Institution:** Georgia Institute of Technology, Atlanta, GA 30332, USA 705 | 706 | 707 | 708 | 709 | ## 3.6 Discussion and Theoretical Analysis 710 | 711 | **[ICML-2018]** 712 | [Open category detection with pac guarantees](http://proceedings.mlr.press/v80/liu18e/liu18e.pdf) 713 |
714 | **Authors:** S. Liu, R. Garrepalli, T. Dietterich, A. Fern, and D. Hendrycks 715 |
716 | **Institution:** Department of Statistics, Oregon State University, Oregn, USA School of EECS, Oregon State University, Oregon, USA University of California, Berkeley, California USA 717 | 718 | **[ICML-2021]** 719 | [Learning bounds for open-set learning](http://proceedings.mlr.press/v139/fang21c/fang21c.pdf) 720 |
721 | **Authors:** Z. Fang, J. Lu, A. Liu, F. Liu, and G. Zhang 722 |
723 | **Institution:** AAII, University of Technology Sydney. 724 | -------------------------------------------------------------------------------- /5_OOD.md: -------------------------------------------------------------------------------- 1 | 2 | # 5. OOD Detection 3 | - [5.1 Classfication-based Method](#5.1) 4 | - [5.1.1 Output-based Methods](#5.1.1) 5 | - [5.1.1.1 Post-hoc Methods](#5.1.1.1) 6 | - [5.1.1.2 Confidence Enhancement](#5.1.1.2) 7 | - [5.1.1.3 Outlier Explosure](#5.1.1.3) 8 | - [5.1.2 OOD Data Generation](#5.1.2) 9 | - [5.1.3 Gradient-based Methods](#5.1.3) 10 | - [5.1.4 Bayesian Models](#5.1.4) 11 | - [5.1.5 Big Pretrained Model](#5.1.5) 12 | - [5.2 Density-based Method](#5.2) 13 | - [5.3 Distance-based Method](#5.3) 14 | 15 | 16 | 17 | ## 5.1 Classfication-based Method 18 | 19 | 20 | ### 5.1.1 Output-based Methods 21 | 22 | #### 5.1.1.1 Post-hoc Methods 23 | 24 | **[ICLR-2017]** 25 | [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://arxiv.org/abs/1610.02136). 26 |
27 | **Authors:** Dan Hendrycks, Kevin Gimpel 28 |
29 | **Institution:** University of California, Berkeley; Toyota Technological Institute at Chicago 30 | >
31 | > The starting point of OOD detection, proposing a baseline simply uses softmax probabilities to detect OOD. 32 | >

33 | > Correctly classified examples tend to have greater maximum softmax probabilities than erroneously classified and out-of-distribution examples, allowing for their detection. However, due to the overconfidence characteristics of deep models, the baseline cannot be well performed. The overconfidence property comes from softmax always modeling sharp distribution for predictions. 34 | >

35 | >
36 | 37 | 38 | **[ICLR-2018]** 39 | [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://github.com/facebookresearch/odin). 40 |
41 | **Authors:** Shiyu Liang, Yixuan Li, R.rikant 42 |
43 | **Institution:** University of Illinois at Urbana-Champaign; University of Wisconsin-Madison 44 | >
45 | > Using temperature scaling on softmax probabilities with small perturbations for reliability. 46 | >

47 | > Temperature scaling has a strong smoothing effect that transforms the softmax score back to the logit space, which effectively distinguishes ID vs.OOD. A perturbation on each sample at test time can further increase the separability between ID and OOD data. 48 | >

49 | >
50 | 51 | **[NeurIPS-2020]** 52 | [Energy-based Out-of-distribution Detection](https://arxiv.org/abs/2010.03759). 53 |
54 | **Authors:** Weitang Liu, Xiaoyun Wang, John D. Owens, Yixuan Li 55 |
56 | **Institution:** University of California, San Diego; University of California, Davis; University of Wisconsin-Madison 57 | >
58 | > Using energy scores instead of softmax scores to conveniently achieve good results. 59 | >

60 | > Unlike softmax confidence scores, energy scores are theoretically aligned with the probability density of the inputs and are less susceptible to the overconfidence issue. The paper shows that energy can conveniently replace softmax confidence for any pre-trained neural network, and proposes an energy-bounded learning objective to fine-tune the network. 61 | >

62 | >
63 | 64 | 65 | **[ICML-2020]** 66 | [Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices](https://arxiv.org/abs/1912.12510). 67 |
68 | **Authors:** Chandramouli Shama Sastry, Sageev Oore 69 |
70 | **Institution:** Dalhousie University, Halifax 71 | 72 | 73 | **[CVPR-2021]** 74 | [MOOD: Multi-level Out-of-distribution Detection](https://arxiv.org/abs/2104.14726). 75 |
76 | **Authors:** Ziqian Lin, Sreya Dutta Roy, Yixuan Li 77 |
78 | **Institution:** University of Wisconsin-Madison 79 | >
80 | > Accelerate training by finding optimal exit level via data complexity. 81 | >

82 | >We explore and establish a direct relationship between the OOD data complexity and optimal exit level, and show that easy OOD examples can be effectively detected early without propagating to deeper layers. 83 | >

84 | >
85 | 86 | **[NeurIPS-2021]** 87 | [ReAct: Out-of-distribution detection with rectified activations](http://pages.cs.wisc.edu/~sharonli/). 88 |
89 | **Authors:** Yiyou Sun, Chuan Guo and Yixuan Li 90 |
91 | **Institution:** University of Wisconsin-Madison, Facebook AI Research 92 | 93 | 94 | **[NeurIPS-2021]** 95 | [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://arxiv.org/abs/2110.00218). 96 |
97 | **Authors:** Rui Huang, Andrew Geng, Yixuan Li 98 |
99 | **Institution:** University of Wisconsin-Madison 100 | 101 | 102 | 103 | **[NeurIPS-2021]** 104 | [Can multi-label classification networks know what they don’t know?](https://arxiv.org/abs/2109.14162). 105 |
106 | **Authors:** Haoran Wang, Weitang Liu, Alex Bocchieri, Yixuan Li 107 |
108 | **Institution:** Carnegie Mellon University; University of California, San Diego; Department of Computer Sciences 109 | University of Wisconsin-Madison 110 | 111 | **[arXiv-2021]** 112 | [On the Effectiveness of Sparsification for Detecting the Deep Unknowns](https://arxiv.org/abs/2111.09805). 113 |
114 | **Authors:** Yiyou Sun, Yixuan Li 115 |
116 | **Institution:** University of Wisconsin-Madison 117 | 118 | **[NeurIPS-2022]** 119 | [RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection](https://arxiv.org/abs/2209.08590). 120 |
121 | **Authors:** Yue Song, Nicu Sebe, Wei Wang 122 |
123 | **Institution:** University of Trento 124 | 125 |
126 | 127 | 128 | #### 5.1.1.2 Confidence Enhancement 129 | 130 | **[arXiv-2017]** 131 | [Improved Regularization of Convolutional Neural Networks with Cutout](https://arxiv.org/abs/1708.04552). 132 |
133 | **Authors:** Terrance DeVries, Graham W. Taylor 134 |
135 | **Institution:** University of Guelph; Canadian Institute for Advanced Research and Vector Institute 136 | 137 | 138 | **[arXiv-2018]** 139 | [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://arxiv.org/abs/1802.04865). 140 |
141 | **Authors:** Terrance DeVries, Graham W. Taylor 142 |
143 | **Institution:** University of Guelph; Vector Institute 144 | >
145 | > Neural network augmented with a confidence estimation branch. 146 | >

147 | > During training, the predictions are modified according to the confidence of the network such that they are closer to the target probability distribution y. The gradual training procedure helps a better estimation of confidence. 148 | >

149 | >
150 | 151 | 152 | **[ECCV-2018]** 153 | [Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers](https://arxiv.org/abs/1809.03576). 154 |
155 | **Authors:** Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, Theodore L. Willke 156 |
157 | **Institution:** Intel labs, Bangalore, India; Intel labs, Hillsboro; Idiap Research Institute, Switzerland 158 | >
159 | > Training multiple classifers by leaving out a random subset of training data as OOD data and the rest as in-distribution for ensembling. 160 | >

161 | >They add a novel margin-based loss term to maintain a margin between the average entropy of OOD and ID samples respectively. An ensemble of K leave-out classifiers is used for OOD detection. The weakness is that the large computational cost and extra OOD dataset for hyper-parameter search. 162 | >

163 | >
164 | 165 | 166 | **[CVPR-2019]** 167 | [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://arxiv.org/abs/1812.05720). 168 |
169 | **Authors:** Matthias Hein, Maksym Andriushchenko, Julian Bitterwolf 170 |
171 | **Institution:** University of T¨ubingen; Saarland University 172 | >
173 | > ReLU-networks lead to over-confident predictions 174 | >

175 | > ReLU-networks lead to over-confident predictions even for samples that are far away from the in-domain distributions and propose methods to mitigate this problem 176 | >

177 | >
178 | 179 | 180 | **[NeurIPS-2019]** 181 | [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://arxiv.org/abs/1905.11001). 182 |
183 | **Authors:** Sunil Thulasidasan, Gopinath Chennupati, Jeff Bilmes, Tanmoy Bhattacharya, Sarah Michalak 184 |
185 | **Institution:** Los Alamos National Laboratory; University of Washington 186 | >
187 | > Mixup-training helps. 188 | >

189 | > We also observe that mixup-trained DNNs are less prone to over-confident predictions on out-of-distribution and random-noise data. 190 | >

191 | >
192 | 193 | 194 | 195 | **[CVPR-2019]** 196 | [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://arxiv.org/abs/1905.04899). 197 |
198 | **Authors:** Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, Youngjoon Yoo 199 |
200 | **Institution:** 1Clova AI Research, NAVER Corp; Clova AI Research, LINE Plus Corp; Yonsei University 201 | 202 | 203 | **[arXiv-2019]** 204 | [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://arxiv.org/abs/1912.02781). 205 |
206 | **Authors:** Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, Balaji Lakshminarayanan 207 |
208 | **Institution:** DeepMind; Google 209 | 210 | 211 | **[arXiv-2019]** 212 | [Towards neural networks that provably know when they don't know](https://arxiv.org/abs/1909.12180). 213 |
214 | **Authors:** Alexander Meinke, Matthias Hein 215 |
216 | **Institution:** University of Tübingen 217 | >
218 | > Certified Certain Uncertainty 219 | >

220 | >We propose a Certified Certain Uncertainty (CCU) model with which one can train deep neural networks that provably make low-confidence predictions far away from the training data. 221 | >

222 | >
223 | 224 | 225 | 226 | **[CVPR-W-2020]** 227 | [On Out-of-Distribution Detection Algorithms With Deep Neural Skin Cancer Classifiers](https://openaccess.thecvf.com/content_CVPRW_2020/html/w42/Pacheco_On_Out-of-Distribution_Detection_Algorithms_With_Deep_Neural_Skin_Cancer_Classifiers_CVPRW_2020_paper.html). 228 |
229 | **Authors:** Andre G. C. Pacheco, Chandramouli S. Sastry, Thomas Trappenberg, Sageev Oore, Renato A. Krohling 230 |
231 | **Institution:** Federal University of Espirito Santo; Dalhousie University; Vector Institute 232 | 233 | 234 | **[CVPR-2020]** 235 | [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://arxiv.org/abs/2002.11297). 236 |
237 | **Authors:** Yen-Chang Hsu, Yilin Shen, Hongxia Jin, Zsolt Kira 238 |
239 | **Institution:** Georgia Institute of Technology; Samsung Research 240 | >
241 | > Improving ODIN by decomposed confidence scoring and a modified input pre-processing method. 242 | >

243 | > The method find that previous work relies on the class posterior probability p(y|x), which does not consider the domain variable at all. Therefore, they use the explicit variable in the classifier, rewriting it as the quotient of the joint class-domain probability and the domain probability using the rule of conditional probability, and take the decomposed confidence scores for OOD detection. The decomposed confidence in the end is the probability of an input being in-distribution, computed by the cosine similarity between sample features and class features. The method also modifies the input preprocessing by only optimizing in-distribution data, therefore extra OOD validation samples are not required. 244 | >

245 | >
246 | 247 | **[NeurIPS-2020]** 248 | [Certifiably Adversarially Robust Detection of Out-of-Distribution Data](https://arxiv.org/abs/2007.08473). 249 |
250 | **Authors:** Julian Bitterwolf, Alexander Meinke, Matthias Hein 251 |
252 | **Institution:** University of Tübingen 253 | 254 | 255 | **[arXiv-2020]** 256 | [Robust Out-of-distribution Detection for Neural Networks](https://arxiv.org/abs/2003.09711). 257 |
258 | **Authors:** Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, Somesh Jha 259 |
260 | **Institution:** University of Wisconsin-Madison; Stanford University; Google 261 | >
262 | > Add a optimized adversarial perturbations on in-distribution and OOD samples for robust model. 263 | >

264 | >This paper shows that existing detection mechanisms can be extremely brittle when evaluating on inputs with minimal adversarial perturbations which don’t change their semantics. To address the problem, the method performs robust training by exposing the model to perturbed adversarial in-distribution and outlier examples. 265 | >

266 | >
267 | 268 | **[ICLR-2020]** 269 | [Novelty Detection Via Blurring](https://arxiv.org/abs/1911.11943). 270 |
271 | **Authors:** Sungik Choi, Sae-Young Chung 272 |
273 | **Institution:** Korea Advanced Institute of Science and Technology 274 | 275 | 276 | 277 |
278 | 279 | [Back to Top](#top) 280 | 281 |
282 | 283 | 284 | #### 5.1.1.3 Outlier Exposure (OE) 285 | 286 | **[NeurIPS-2018]** 287 | [Reducing network agnostophobia](https://arxiv.org/abs/1811.04110). 288 |
289 | **Authors:** Akshay Raj Dhamija, Manuel Günther, Terrance E. Boult 290 |
291 | **Institution:** Vision and Security Technology Lab; University of Colorado Colorado Springs 292 | >
293 | > An entropic open-set loss and an OOD-feature-magnitudes-suppression loss on the additional background class. 294 | >

295 | > The paper designs novel losses to maximize entropy for unknown inputs while increasing separation in deep feature space by modifying magnitudes of known and unknown samples. In sum, logits entropy and feature magnitudes are used for OOD detection. 296 | >

297 | >
298 | 299 | 300 | 301 | **[ICLR-2019]** 302 | [Deep anomaly detection with outlier exposure](https://arxiv.org/abs/1812.04606) 303 |
304 | **Authors:** Dan Hendrycks, Mantas Mazeika, Thomas Dietterich 305 |
306 | **Institution:** University of California, Berkeley; University of Chicago; Oregon State University 307 | >
308 | > A baseline model to produce a uniform posterior distribution on auxiliary dataset of outliers. 309 | >

310 | > It can learn effective heuristics for detecting out-of-distribution inputs by exposing the model to OOD examples, thus learning a more conservative concept of the inliers and enabling the detection of novel forms of anomalies. The result is shown effective on both CV and NLP tasks. 311 | >

312 | >
313 | 314 | 315 | 316 | **[ICCV-2019]** 317 | [Unsupervised out-of-distribution detection by maximum classifier discrepancy](https://arxiv.org/abs/1908.04951) 318 |
319 | **Authors:** Qing Yu, Kiyoharu Aizawa 320 |
321 | **Institution:** The University of Tokyo 322 | >
323 | > A network with two branches, between which entropy discrepancy is enlarged for OOD training data. 324 | >

325 | > It trains a two-head CNN consisting of one common feature extractor and two classifiers which have different decision boundaries but can classify in-distribution samples correctly. Then it uses the unlabeled data to maximize the discrepancy between the decision boundaries of two classifiers to push OOD samples outside the manifold of the in-distribution samples, which enables to detect OOD samples that are far from the support of the ID samples. 326 | >

327 | >
328 | 329 | **[BMVC-2019]** 330 | [A Less Biased Evaluation of Out-of-distribution Sample Detectors](https://arxiv.org/abs/1809.04729). 331 |
332 | **Authors:** Alireza Shafaei, Mark Schmidt, James J. Little 333 |
334 | **Institution:** University of British Columbia 335 | 336 | 337 | **[AAAI-2020]** 338 | [Self-Supervised Learning for Generalizable Out-of-Distribution Detection](https://ojs.aaai.org/index.php/AAAI/article/view/5966) 339 |
340 | **Authors:** Sina Mohseni, Mandar Pitale, JBS Yadawa, Zhangyang Wang 341 |
342 | **Institution:** Texas A&M University; NVIDIA 343 | >
344 | > Pseudo-labeling external unlabeled set for later OOD training. 345 | >

346 | > It simultaneously trains in-distribution classifiers and out-of-distribution detectors in one network. By pseudo-labeling, the unlabeled data can be assigned with label index or reject label for later training. 347 | >

348 | >
349 | 350 | 351 | 352 | **[CVPR-2020]** 353 | [Background data resampling for outlier-aware classification](https://openaccess.thecvf.com/content_CVPR_2020/html/Li_Background_Data_Resampling_for_Outlier-Aware_Classification_CVPR_2020_paper.html) 354 |
355 | **Authors:** Yi Li, Nuno Vasconcelos 356 |
357 | **Institution:** University of California, San Diego 358 | >
359 | > Using adversarial resampling approach to obtain a compact yet representative set of background data points. 360 | >

361 | > This work focuses on training with background and claims that using all background data leads to inefficient or even impractical solution due to imbalance and computational complexity. The resampling algorithm takes inspiration from prior work on hard negative mining, performing an iterative adversarial weighting on the background examples and using the learned weights to obtain the subset of desired size. 362 | >

363 | >
364 | 365 | 366 | **[AAAI-2020]** 367 | [Self-Supervised Learning for Generalizable Out-of-Distribution Detection](https://ojs.aaai.org/index.php/AAAI/article/view/5966) 368 |
369 | **Authors:** Sina Mohseni，Mandar Pitale, JBS Yadawa，Zhangyang Wang 370 |
371 | **Institution:** Texas A&M University; NVIDIA 372 | >
373 | > Pseudo-labeling external unlabeled set for later OOD training. 374 | >

375 | > It simultaneously trains in-distribution classifiers and out-of-distribution detectors in one network. By pseudo-labeling, the unlabeled data can be assigned with label index or reject label for later training. 376 | >

377 | >
378 | 379 | **[Neurocomputing-2021]** 380 | [Outlier exposure with confidence control for out-of-distribution detection](https://arxiv.org/abs/1906.03509) 381 |
382 | **Authors:** Aristotelis-Angelos Papadopoulos, Mohammad Reza Rajati, Nazim Shaikh, Jiamian Wang 383 |
384 | **Institution:** University of Southern California 385 | >
386 | > Performing prediction confidence calibration on the top of OE. 387 | >

388 | > Based on the loss function of OE, this work add the second regularization term to minimize the Euclidean distance between the training accuracy of a DNN and the average confidence in its predictions on the training set. 389 | >

390 | >
391 | 392 | 393 | **[ICCV-2021]** 394 | [Semantically Coherent Out-of-Distribution Detection](https://arxiv.org/abs/2108.11941). 395 |
396 | **Authors:** Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, Ziwei Liu 397 |
398 | **Institution:** Nanyang Technological University; SenseTime Research, Shanghai Jiaotong Univerisity; Shanghai AI Lab. 399 | 400 | 401 | **[ECML PKDD-2021]** 402 | [ATOM: Robustifying Out-of-distribution Detection Using Outlier Mining](https://arxiv.org/abs/2006.15207) 403 |
404 | **Authors:** Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, Somesh Jha 405 |
406 | **Institution:** University of Wisconsin-Madison; Google 407 | >
408 | > Using informative auxiliary outlier data to learn a tight decision boundary between ID and OOD data. 409 | >

410 | >By mining informative auxiliary OOD data, one can significantly improve OOD detection performance, and somewhat surprisingly, generalize to unseen adversarial attack. The key idea is to selectively utilize auxiliary outlier data for estimating a tight decision boundary between ID and OOD data, which leads to robust OOD detection performance. 411 | >

412 | >
413 | > 414 | **[arXiv-2021]** 415 | [An Effective Baseline for Robustness to Distributional Shift](https://arxiv.org/abs/2105.07107) 416 |
417 | **Authors:** Sunil Thulasidasan, Sushil Thapa, Sayera Dhaubhadel, Gopinath Chennupati, Tanmoy Bhattacharya, Jeff Bilmes 418 |
419 | **Institution:** Los Alamos; New Mexico Tech; University of Washington 420 | >
421 | > An extra abstention (or rejection class) in combination with outlier training data for effective OoD detection. 422 | >

423 | > This work demonstrates the efficacy of using an extra abstention (or rejection class) in combination with outlier training data for effective OoD detection. 424 | >

425 | >
426 | 427 | 428 | 429 | 430 |
431 | 432 | [Back to Top](#top) 433 | 434 |
435 | 436 | 437 | ### 5.1.2 OOD Data Generation 438 | 439 | **[ICLR-2018]** 440 | [Training confidence-calibrated classifiers for detecting out-of-distribution samples](https://arxiv.org/abs/1711.09325) 441 |
442 | **Authors:** Kimin Lee, Honglak Lee, Kibok Lee, Jinwoo Shin 443 |
444 | **Institution:** KAIST; University of Michigan, Ann Arbor; Google Brain 445 | >
446 | > An confidence loss to encourage a uniform prediction of GAN-generated ‘boundary’ OOD samples. 447 | >

448 | > It proposes an confidence loss to minimize the KL divergence from the predictive distribution on the GAN-generated OOD samples to the uniform one in order to give less confident predictions on them. The proposed GAN generates ‘boundary’ samples in the in-distribution low-density area. 449 | >

450 | >
451 | 452 | 453 | **[NeurIPSW-2018]** 454 | [Building robust classifiers through generation of confident out of distribution examples](https://arxiv.org/abs/1812.00239) 455 |
456 | **Authors:** Kumar Sricharan, Ashok Srivastava 457 |
458 | **Institution:** Central Data Science Organization, Intuit Inc 459 | 460 | 461 | **[NeurIPSW-2019]** 462 | [Out-of-distribution detection in classifiers via generation](https://arxiv.org/abs/1910.04241) 463 |
464 | **Authors:** Sachin Vernekar, Ashish Gaurav, Vahdat Abdelzad, Taylor Denouden, Rick Salay, Krzysztof Czarnecki 465 |
466 | **Institution:** University of Waterloo 467 | 468 | 469 | **[CVPR-2019]** 470 | [Out-of-distribution detection for generalized zero-shot action recognition](https://arxiv.org/abs/1904.08703) 471 |
472 | **Authors:** Devraj Mandal, Sanath Narayan, Saikumar Dwivedi, Vikram Gupta, Shuaib Ahmed, Fahad Shahbaz Khan, Ling Shao 473 |
474 | **Institution:** Indian Institute of Science, Bangalore; Inception Institute of Artificial Intelligence, UAE; Mercedes-Benz R&D India, Bangalore 475 | 476 | 477 | **[ICLR-2022]** 478 | [VOS: Learning What You Don't Know By Virtual Outlier Synthesis](https://arxiv.org/abs/2202.01197) 479 |
480 | **Authors:** Xuefeng Du, Zhaoning Wang, Mu Cai, Yixuan Li 481 |
482 | **Institution:** University of Wisconsin - Madison 483 | >
484 | > A novel framework for OOD detection by adaptively synthesizing virtual outliers that can meaningfully regularize the model's decision boundary during training. 485 | >

486 | > VOS samples virtual outliers from the low-likelihood region of the class-conditional distribution estimated in the feature space. VOS achieves strong performance on both object detection and image classification models. 487 | >

488 | >
489 | 490 | 491 |
492 | 493 | [Back to Top](#top) 494 | 495 |
496 | 497 | 498 | ### 5.1.3 Gradient-based Methods 499 | 500 | **[ICLR-2018]** 501 | [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://github.com/facebookresearch/odin). 502 |
503 | **Authors:** Shiyu Liang, Yixuan Li, R. Srikant 504 |
505 | **Institution:** University of Illinois at Urbana-Champaign; University of Wisconsin-Madison 506 | >
507 | > Using temperature scaling on softmax probabilities with small perturbations for robustness. 508 | >

509 | > Temperature scaling can calibrate the softmax probabilities so the model takes the calibrated maximum softmax probabilities as the indicator for OOD detection. A perturbation on each sample at test time can further exploit the model robustness in detecting ID samples. However, it requires an OOD validation set for hyperparameter tuning. 510 | >

511 | >
512 | 513 | **[NeurIPS-2021]** 514 | [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://arxiv.org/abs/2110.00218). 515 |
516 | **Authors:** Rui Huang, Andrew Geng, Yixuan Li 517 |
518 | **Institution:** University of Wisconsin-Madison 519 | 520 | 521 | 522 |
523 | 524 | [Back to Top](#top) 525 | 526 |
527 | 528 | 529 | 530 | ### 5.1.4 Bayesian Models 531 | 532 | **[ICML-2016]** 533 | [Dropout as a bayesian approximation: Representing model uncertainty in deep learning](http://proceedings.mlr.press/v48/gal16.pdf). 534 |
535 | **Authors:** Yarin Gal , Zoubin Ghahramani 536 |
537 | **Institution:** University of Cambridge 538 | 539 | 540 | 541 | **[NeurIPS-2017]** 542 | [Simple and scalable predictive uncertainty estimation using deep ensembles](https://arxiv.org/pdf/1612.01474.pdf). 543 |
544 | **Authors:** Balaji Lakshminarayanan , Alexander Pritzel , Charles Blundell 545 |
546 | **Institution:** DeepMind 547 | 548 | 549 | 550 | **[NeurIPS-2018]** 551 | [Predictive Uncertainty Estimation via Prior Networks](https://arxiv.org/pdf/1802.10501.pdf) 552 |
553 | **Authors:** Andrey Malinin, Mark Gales 554 |
555 | **Institution:** University of Cambridge 556 | 557 | 558 | 559 | **[NeurIPS-2019]** 560 | [Practical deep learning with bayesian principles](https://arxiv.org/pdf/1906.02506.pdf) 561 |
562 | **Authors:** Kazuki Osawa, Siddharth Swaroop, Anirudh Jain, Runa Eschenhagen, Richard E. Turner, Rio Yokota, Mohammad Emtiyaz Khan 563 |
564 | **Institution:** Tokyo Institute of Technology; University of Cambridge; Indian Institute of Technology (ISM); University of Osnabrück; RIKEN Center for AI Project 565 | 566 | 567 | **[NeurIPS-2019]** 568 | [Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness](https://arxiv.org/pdf/1905.13472.pdf) 569 |
570 | **Authors:** Andrey Malinin, Mark Gales 571 |
572 | **Institution:** Yandex; University of Cambridge 573 | 574 | 575 | 576 | **[NeurIPS-2020]** 577 | [Towards maximizing the representation gap between in-domain & out-of-distribution examples](https://arxiv.org/pdf/2010.10474.pdf) 578 |
579 | **Authors:** Jay Nandy, Wynne Hsu, Mong Li Lee 580 |
581 | **Institution:** National University of Singapore 582 | 583 | 584 | 585 |
586 | 587 | [Back to Top](#top) 588 | 589 |
590 | 591 | 592 | 593 | ### 5.1.5 Large-scale OOD Detection 594 | 595 | **[CVPR-2021]** 596 | [Mos: Towards scaling out-of-distribution detection for large semantic space](https://openaccess.thecvf.com/content/CVPR2021/papers/Huang_MOS_Towards_Scaling_Out-of-Distribution_Detection_for_Large_Semantic_Space_CVPR_2021_paper.pdf) 597 |
598 | **Authors:** Rui Huang, Yixuan Li 599 |
600 | **Institution:** University of Wisconsin-Madison 601 | 602 | 603 | 604 | **[arXiv-2021]** 605 | [Exploring the limits of out-of-distribution detection](https://arxiv.org/pdf/2106.03004.pdf) 606 |
607 | **Authors:** Stanislav Fort, Jie Ren, Balaji Lakshminarayanan 608 |
609 | **Institution:** Stanford University; Google Research 610 | >
611 | > Large-scale pre-trained transformers significantly improve near-OOD tasks 612 | >

613 | > This work explores the effectiveness of large-scale pre-trained transformers, especially when few-shot outlier exposure is available. It also shows that the pre-trained multi-modal image-text transformers CLIP is also effective on OOD detection if using the names of outlier classes as candidate text labels. 614 | >

615 | >
616 | 617 | 618 | 619 | **[arXiv-2020]** 620 | [Pretrained transformers improve out-of-distribution robustness](https://arxiv.org/pdf/2004.06100.pdf) 621 |
622 | **Authors:** Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, Dawn Song 623 |
624 | **Institution:** UC Berkeley; Shanghai Jiao Tong University; University of Chicago 625 | 626 | 627 | 628 | **[arXiv-2021]** 629 | [Oodformer: Out-of-distribution detection transformer](https://arxiv.org/pdf/2107.08976.pdf) 630 |
631 | **Authors**: Rajat Koner, Poulami Sinhamahapatra, Karsten Roscher, Stephan Günnemann, Volker Tresp 632 |
633 | **Institution:** Ludwig Maximilian University; Technical University; Fraunhofer, IKS; Siemens AG 634 | 635 | 636 | 637 |
638 | 639 | [Back to Top](#top) 640 | 641 |
642 | 643 | 644 | ## 5.2 Density-based Methods 645 | 646 | 647 | 648 | **[ICML-2016]** 649 | [Pixel recurrent neural networks](http://proceedings.mlr.press/v48/oord16.pdf) 650 |
651 | **Authors:** Aaron van den Oord, Nal Kalchbrenner, Koray Kavukcuoglu 652 |
653 | **Institution:** Google DeepMind 654 | 655 | 656 | 657 | **[NeurIPS-2018]** 658 | [Generative probabilistic novelty detection with adversarial autoencoders](https://arxiv.org/pdf/1807.02588.pdf) 659 |
660 | **Authors:** Stanislav Pidhorskyi, Ranya Almohsen, Donald A. Adjeroh, Gianfranco Doretto 661 |
662 | **Institution:** West Virginia University 663 | 664 | 665 | 666 | **[NeurIPS-2018]** 667 | [Glow: Generative flow with invertible 1x1 convolutions](https://arxiv.org/pdf/1807.03039.pdf) 668 |
669 | **Authors:** Diederik P. Kingma, Prafulla Dhariwal 670 |
671 | **Institution:** OpenAI 672 | 673 | 674 | 675 | 676 | **[ECML/KDD-2018]** 677 | [Image anomaly detection with generative adversarial networks](https://openreview.net/pdf?id=S1EfylZ0Z) 678 |
679 | **Authors:** Lucas Deecke, authorRobert Vandermeulen, Lukas RuffStephan Mandt, Marius Kloft 680 |
681 | **Institution:** University of Edinburgh; TU Kaiserslautern; Hasso Plattner Institute; University of California 682 | 683 | 684 | 685 | 686 | **[ICLR-2018]** 687 | [Deep autoencoding gaussian mixture model for unsupervised anomaly detection](https://openreview.net/pdf?id=BJJLHbb0-) 688 |
689 | **Authors:** Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, Haifeng Chen 690 |
691 | **Institution:** NEC Laboratories America; Washington State University 692 | 693 | **[NeurIPS-2018]** 694 | [Do deep generative models know what they don’t know?](https://arxiv.org/pdf/1810.09136.pdf) 695 |
696 | **Authors:** Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, Balaji Lakshminarayanan 697 |
698 | **Institution:** DeepMind 699 | 700 | 701 | **[arXiv-2018]** 702 | [Waic, but why? generative ensembles for robust anomaly detection](https://arxiv.org/pdf/1810.01392.pdf) 703 |
704 | **Authors:** Hyunsun Choi, Eric Jang, Alexander A. Alemi 705 |
706 | **Institution:** Google Inc. 707 | 708 | 709 | **[CVPR-2018]** 710 | [Adversarially learned one-class classifier for novelty detection](https://openaccess.thecvf.com/content_cvpr_2018/papers/Sabokrou_Adversarially_Learned_One-Class_CVPR_2018_paper.pdf) 711 |
712 | **Authors:** Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, Ehsan Adeli 713 |
714 | **Institution:** Institute for Research in Fundamental Sciences; Amirkabir University of Technology; Stanford University 715 | 716 | 717 | 718 | **[NeurIPS-2019]** 719 | [Likelihood ratios for out-of-distribution detection](https://arxiv.org/pdf/1906.02845.pdf) 720 |
721 | **Authors:** Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, Balaji Lakshminarayanan 722 |
723 | **Institution:** Google Research; DeepMind 724 | >
725 | > Using likelihood ratios to cancel out background influence. 726 | >

727 | > This work finds the likelihood score is heavily affected by background, so likelihood ratios are used to cancel out background influence. The Likelihood Ratio (LR) is the likelihood that a given test result would be expected in a patient with the target disorder compared to the likelihood that that same result would be expected in a patient without the target disorder. 728 | >

729 | >
730 | 731 | 732 | **[CVPR-2019]** 733 | [Latent space autoregression for novelty detection](https://openaccess.thecvf.com/content_CVPR_2019/papers/Abati_Latent_Space_Autoregression_for_Novelty_Detection_CVPR_2019_paper.pdf) 734 |
735 | **Authors:** Davide Abati, Angelo Porrello, Simone Calderara, Rita Cucchiara 736 |
737 | **Institution:** University of Modena and Reggio Emilia 738 | 739 | 740 | 741 | **[CVPR-2020]** 742 | [Deep residual flow for out of distribution detection](?) 743 |
744 | **Authors:** Ev Zisselman, Aviv Tamar 745 |
746 | **Institution:** Technion 747 | 748 | 749 | 750 | 751 | **[NeurIPS-2020]** 752 | [Why normalizing flows fail to detect out-of-distribution data](https://arxiv.org/pdf/2006.08545.pdf) 753 |
754 | **Authors:** Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson 755 |
756 | **Institution:** New York University 757 | 758 | 759 | 760 | 761 | **[ICLR-2020]** 762 | [Input complexity and out-of-distribution detection with likelihood-based generative models](https://arxiv.org/pdf/1909.11480.pdf) 763 |
764 | **Authors:** Joan Serra, David Alvarez, Vicenc¸ Gomez, Olga Slizovskaia, Jose F. Nunez, Jordi Luque 765 |
766 | **Institution:** Dolby Laboratories; Telefonica Research; Universitat Politecnica de Catalunya; Universitat Pompeu Fabra 767 | 768 | 769 | **[NeurIPS-2020]** 770 | [Likelihood regret: An out-ofdistribution detection score for variational auto-encoder](https://arxiv.org/pdf/2003.02977.pdf) 771 |
772 | **Authors:** Zhisheng Xiao, Qing Yan, Yali Amit 773 |
774 | **Institution:** University of Chicago 775 | 776 | 777 | **[TPAMI-2020]** 778 | [Normalizing flows: An introduction and review of current methods](https://arxiv.org/pdf/1908.09257.pdf) 779 |
780 | **Authors:** Ivan Kobyzev, Simon J.D. Prince, Marcus A. Brubaker 781 |
782 | **Institution:** Borealis AI 783 | 784 | 785 | 786 |
787 | 788 | [Back to Top](#top) 789 | 790 |
791 | 792 | 793 | ## 5.3 Distance-based Methods 794 | **[NeurIPS-2018]** 795 | [A simple unified framework for detecting out-of-distribution samples and adversarial attacks](https://proceedings.neurips.cc/paper/2018/file/abdeb6f575ac5c6676b747bca8d09cc2-Paper.pdf) 796 |
797 | **Authors:** Kimin Lee, Kibok Lee, Honglak Lee, Jinwoo Shin 798 |
799 | **Institution:** Korea Advanced Institute of Science and Technology (KAIST); University of Michigan; Google Brain; AItrics 800 | 801 | 802 | 803 | **[ACCV-2020]** 804 | [Hyperparameter-free out-of-distribution detection using cosine similarity](https://openaccess.thecvf.com/content/ACCV2020/papers/Techapanurak_Hyperparameter-Free_Out-of-Distribution_Detection_Using_Cosine_Similarity_ACCV_2020_paper.pdf) 805 |
806 | **Authors:** Engkarat Techapanurak, Masanori Suganuma, Takayuki Okatani 807 |
808 | **Institution:** Tohoku University; RIKEN Center for AIP 809 | >
810 | > Using scaled cosine similarity between test sample features and class features to determine OOD samples. 811 | >

812 | > The first work employs softmax of scaled cosine similarity instead of ordinary softmax of logits. Taking the metric learning idea into OOD detection. It is also the concurrent work of Generalized ODIN. 813 | >

814 | >
815 | 816 | **[ECCV-2020]** 817 | [A boundary based out-of-distribution classifier for generalized zero-shot learning](https://arxiv.org/abs/2008.04872) 818 |
819 | **Authors:** Xingyu Chen, Xuguang Lan, Fuchun Sun, Nanning Zheng 820 |
821 | **Institution:** Xian Jiaotong University; Tsinghua University 822 | 823 | **[ICML-2020]** 824 | [Uncertainty estimation using a single deep deterministic neural network](https://arxiv.org/abs/2008.04872) 825 |
826 | **Authors:** Joost van Amersfoort, Lewis Smith, Yee Whye Teh, Yarin Gal 827 |
828 | **Institution:** University of Oxford 829 | 830 | 831 | **[arXiv-2020]** 832 | [Feature Space Singularity for Out-of-Distribution Detection](https://arxiv.org/abs/2011.14654) 833 |
834 | **Authors:** Haiwen Huang, Zhihan Li, Lulu Wang, Sishuo Chen, Bin Dong, Xinyu Zhou 835 |
836 | **Institution:** University of Oxford; Peking University; MEGVII Technology; etc. 837 | >
838 | > Distance to Feature Space Singularity can measure OOD. 839 | >

840 | > It is observed that in feature spaces, OOD samples concentrate near a Feature Space Singularity (FSS) point, and the distance from a sample to FSS measures the degree of OOD. It can be exlained that moving speeds of features of other data depend on their similarity to the training data. During training, they use generated uniform noise or validation data as OOD. 841 | >

842 | >
843 | 844 | 845 | 846 | **[CVPR-2021]** 847 | [Out-of-Distribution Detection Using Union of 1-Dimensional Subspaces](https://openaccess.thecvf.com/content/CVPR2021/papers/Zaeemzadeh_Out-of-Distribution_Detection_Using_Union_of_1-Dimensional_Subspaces_CVPR_2021_paper.pdf) 848 |
849 | **Authors:** Alireza Zaeemzadeh, Niccolò Bisagno, Zeno Sambugaro, Nicola Conci, Nazanin Rahnavard, Mubarak Shah 850 |
851 | **Institution:** University of Central Florida; University of Trento 852 | >
853 | > Calculating class membership probabilities in a union of 1-dimensional subspaces. 854 | >

855 | > The cosine similarities between the extracted feature and the class vectors are used to compute the class membership probabilities, using a Union of 1-dimensional subspaces. The 1-dimensional subspaces is spanned by the first singular vector of the feature vectors extracted from the training set. Feature vectors lie on a union of 1-dimensional subspaces helps OOD samples to be robustly detected. 856 | >

857 | >
858 | 859 | 860 | 861 | **[arXiv-2021]** 862 | [A simple fix to mahalanobis distance for improving near-ood detection](https://arxiv.org/pdf/2106.09022.pdf) 863 |
864 | **Authors:** Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, Balaji Lakshminarayanan 865 |
866 | **Institution:** Google Research; Stanford University; Harvard University; Google Health 867 | 868 | 869 | 870 |
871 | 872 | [Back to Top](#top) 873 | 874 |
875 | --------------------------------------------------------------------------------