├── .gitignore ├── README.md ├── calc_NUDFT.ipynb ├── cfg ├── cfg_mnist.yaml └── cfg_run.yaml ├── data ├── pbmc_x.zip └── pbmc_y.zip ├── dataset.py ├── idc_evaluate.ipynb ├── idc_example.ipynb ├── img ├── freq_bias.png ├── img.png ├── nudft_ALLAML.png ├── pbmc.gif ├── supervised_train_plots.png └── supervised_train_plots2.png ├── interpretability_metrics.py ├── lspin_pbmc.py ├── model.py ├── requirements.txt ├── run.py └── train_evaluate.py /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | # Interpretable Deep Clustering 7 | 8 |

9 | 10 |

11 | 12 | ## An official implementation of the ICML 2024 accepted paper: [Interpretable Deep Clustering for Tabular Data](https://openreview.net/pdf?id=QPy7zLfvof) 13 |

14 | 15 |

16 | 17 | ## UPDATES: 18 | 19 | - 2024-10-08: 20 | * we add an example notebook that produces a NUDFT plot and compares gated vs non-gated supervised models 21 | * we present below a fixed figure 4 from the paper: the frequency values are normalized for each model and gated features are used for IDC model rather than raw features. 22 | 23 | 24 | ## How to run on your data: 25 | 26 | 1. Do you have a dataset without labels? Use our colab example notebook:

27 | 2. If you have a labeled dataset, please follow the colab with evaluation example:

28 | 29 | 30 | ## Fixed Figure 4. Spectral properties of the learned predictive function using ALLAML dataset. 31 | 32 | The model trained with the gating network (IDC) has higher Fourier amplitudes at all frequency levels than 33 | without gates (IDCw/o_gates) the baseline (TELL). This suggests that IDC can better handle the inductive bias of tabular data. 34 |

35 | 36 |

37 | ### Citation: 38 | Please cite our paper if you use this code: 39 | 40 | 41 | ``` 42 | 43 | @InProceedings{pmlr-v235-svirsky24a, 44 | title = {Interpretable Deep Clustering for Tabular Data}, 45 | author = {Svirsky, Jonathan and Lindenbaum, Ofir}, 46 | booktitle = {Proceedings of the 41st International Conference on Machine Learning}, 47 | pages = {47314--47330}, 48 | year = {2024}, 49 | editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, 50 | volume = {235}, 51 | series = {Proceedings of Machine Learning Research}, 52 | month = {21--27 Jul}, 53 | publisher = {PMLR}, 54 | pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/svirsky24a/svirsky24a.pdf}, 55 | url = {https://proceedings.mlr.press/v235/svirsky24a.html}, 56 | abstract = {Clustering is a fundamental learning task widely used as a first step in data analysis. For example, biologists use cluster assignments to analyze genome sequences, medical records, or images. Since downstream analysis is typically performed at the cluster level, practitioners seek reliable and interpretable clustering models. We propose a new deep-learning framework for general domain tabular data that predicts interpretable cluster assignments at the instance and cluster levels. First, we present a self-supervised procedure to identify the subset of the most informative features from each data point. Then, we design a model that predicts cluster assignments and a gate matrix that provides cluster-level feature selection. Overall, our model provides cluster assignments with an indication of the driving feature for each sample and each cluster. We show that the proposed method can reliably predict cluster assignments in biological, text, image, and physics tabular datasets. Furthermore, using previously proposed metrics, we verify that our model leads to interpretable results at a sample and cluster level. Our code is available on https://github.com/jsvir/idc.} 57 | } 58 | ``` 59 | 60 | 61 | 62 | ### TODO list: 63 | - [x] Add interpretability evaluation scripts 64 | - [ ] Add experiments configs 65 | - [ ] Add features index outputs 66 | - [x] Add synthetic dataset deneration code -------------------------------------------------------------------------------- /calc_NUDFT.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "source": [ 6 | "# Calculation of NUDFT \n", 7 | "\n", 8 | "We proved an example for pre-trained supervised model [LSPIN](https://proceedings.mlr.press/v162/yang22i/yang22i.pdf) trained on PBMC dataset compared against deep classifier.\n", 9 | "\n", 10 | "We provide the checkpoints in ckpts directory.\n" 11 | ], 12 | "metadata": { 13 | "collapsed": false 14 | }, 15 | "id": "9e52f756a2211364" 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "source": [], 20 | "metadata": { 21 | "collapsed": false 22 | }, 23 | "id": "e40cd92806e61978" 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "source": [ 28 | "### Imports and util functions\n" 29 | ], 30 | "metadata": { 31 | "collapsed": false 32 | }, 33 | "id": "9dbae214166550cd" 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 1, 38 | "outputs": [], 39 | "source": [ 40 | "import platform\n", 41 | "from nfft import nfft_adjoint\n", 42 | "import torch\n", 43 | "import numpy as np\n", 44 | "from matplotlib import pyplot as plt\n", 45 | "from omegaconf import OmegaConf\n", 46 | "import seaborn as sns\n", 47 | "from lspin_pbmc import PBMC, Classifier, GatingNet\n", 48 | "\n", 49 | "# which dimension use in the y_hat outputs (normalized with softmax)\n", 50 | "TARGET_IDX = 1\n", 51 | "COLUMNS_SUBSET=100000\n", 52 | "\n", 53 | "\n", 54 | "def spectrum_NUDFT(x, y, kmax=50, nk=1000):\n", 55 | " kvals = np.linspace(0.1, kmax, nk+1)\n", 56 | " nufft = (1 / len(x)) * nfft_adjoint(-(x * kmax / nk), y, 2 * (nk + 1))[nk + 1:]\n", 57 | " return [kvals, np.array(nufft, dtype=\"complex_\")]\n", 58 | "\n", 59 | "\n", 60 | "def select_k_columns_with_max_variance(matrix):\n", 61 | " variance_per_column = torch.var(matrix, dim=0)\n", 62 | " _, sorted_indices = torch.sort(variance_per_column, descending=True)\n", 63 | " top_k_indices = sorted_indices[:COLUMNS_SUBSET]\n", 64 | " selected_columns = matrix[:, top_k_indices]\n", 65 | " return selected_columns" 66 | ], 67 | "metadata": { 68 | "collapsed": false, 69 | "ExecuteTime": { 70 | "end_time": "2024-10-08T12:57:48.715663Z", 71 | "start_time": "2024-10-08T12:57:44.858113500Z" 72 | } 73 | }, 74 | "id": "e1c3233b55927f30" 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "source": [ 79 | "### Load pre-trained checkpoints" 80 | ], 81 | "metadata": { 82 | "collapsed": false 83 | }, 84 | "id": "697ba1c508082d" 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 2, 89 | "outputs": [], 90 | "source": [ 91 | "gated_cfg = OmegaConf.create(dict(\n", 92 | " gated=True,\n", 93 | " input_dim=17126,\n", 94 | " n_clusters=2,\n", 95 | " dataset=\"PBMC\",\n", 96 | " data_dir=\"C:/data/fs/pbmc\" if platform.system() == \"Windows\" else \".\" ,\n", 97 | " batch_size=256,\n", 98 | " repitions=5,\n", 99 | " sigma=0.5,\n", 100 | " reg_beta=100,\n", 101 | " devices=1,\n", 102 | " accelerator=\"gpu\",\n", 103 | " max_epochs=100,\n", 104 | " deterministic=True,\n", 105 | " logger=True,\n", 106 | " log_every_n_steps=10,\n", 107 | " check_val_every_n_epoch=1,\n", 108 | " enable_checkpointing=False,\n", 109 | "))\n", 110 | "\n", 111 | "nongated_cfg = OmegaConf.create(dict(\n", 112 | " gated=False,\n", 113 | " input_dim=17126,\n", 114 | " n_clusters=2,\n", 115 | " dataset=\"PBMC\",\n", 116 | " data_dir=\"C:/data/fs/pbmc\" if platform.system() == \"Windows\" else \".\" ,\n", 117 | " batch_size=256,\n", 118 | " repitions=5,\n", 119 | " devices=1,\n", 120 | " accelerator=\"gpu\",\n", 121 | " max_epochs=100,\n", 122 | " deterministic=True,\n", 123 | " logger=True,\n", 124 | " log_every_n_steps=10,\n", 125 | " check_val_every_n_epoch=1,\n", 126 | " enable_checkpointing=False,\n", 127 | "))\n", 128 | "\n", 129 | "\n" 130 | ], 131 | "metadata": { 132 | "collapsed": false, 133 | "ExecuteTime": { 134 | "end_time": "2024-10-08T12:57:48.730048700Z", 135 | "start_time": "2024-10-08T12:57:48.711662700Z" 136 | } 137 | }, 138 | "id": "545756af156b20df" 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "source": [ 143 | "The models were trained for ~400 epochs and we present the training/validation plots:\n", 144 | "\n", 145 | " $\"Alt$ \n", 146 | " $\"Alt$ \n" 147 | ], 148 | "metadata": { 149 | "collapsed": false 150 | }, 151 | "id": "101afcef26dfe116" 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": 8, 156 | "outputs": [ 157 | { 158 | "name": "stdout", 159 | "output_type": "stream", 160 | "text": [ 161 | "Dataset PBMC stats:\n", 162 | "X.shape: (20742, 17126)\n", 163 | "Y.shape: (20742,)\n", 164 | "X.min=-2.734075760955218, X.max=144.01736006468113\n", 165 | "Y.min=0, Y.max=1\n", 166 | "Label 0 has 10479 samples\n", 167 | "Label 1 has 10263 samples\n", 168 | "Split to train/test: train 16594 test 4148\n" 169 | ] 170 | } 171 | ], 172 | "source": [ 173 | "gating_net = GatingNet(gated_cfg)\n", 174 | "classifier_gated = Classifier(gated_cfg)\n", 175 | "gating_net.load_state_dict(torch.load(\"ckpts/supervised/sparse_model_last_pbmc_beta_10_seed_0.pth\")[\"gating\"])\n", 176 | "classifier_gated.load_state_dict(torch.load(\"ckpts/supervised/sparse_model_last_pbmc_beta_10_seed_0.pth\")[\"clustering\"])\n", 177 | "\n", 178 | "# # load clustering model without gates:\n", 179 | "classifier = Classifier(nongated_cfg)\n", 180 | "classifier.load_state_dict(torch.load(\"ckpts/supervised/sparse_model_nogates_best_pbmc_seed_0.pth\")[\"clustering\"])\n", 181 | "\n", 182 | "classifier = classifier.to('cpu')\n", 183 | "classifier_gated = classifier_gated.to('cpu')\n", 184 | "\n", 185 | "classifier.eval()\n", 186 | "classifier_gated.eval()\n", 187 | "gating_net.eval()\n", 188 | "\n", 189 | "_, test_dataset = PBMC.setup(gated_cfg.data_dir)\n", 190 | "\n" 191 | ], 192 | "metadata": { 193 | "collapsed": false, 194 | "ExecuteTime": { 195 | "end_time": "2024-10-08T13:17:14.874885100Z", 196 | "start_time": "2024-10-08T13:16:56.649883800Z" 197 | } 198 | }, 199 | "id": "a055dcc6d21c43e6" 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": 10, 204 | "outputs": [ 205 | { 206 | "data": { 207 | "text/plain": "