From github, navigate to the folder you want reval library in, open terminal and run:
git clone https://github.com/IIT-LAND/reval_clustering
164 | pip install -r requirements.txt
165 | PyPI alternative:
168 |pip install reval
169 | | 167 | r | ||
| 171 | |
172 | reval | 173 | |
| 176 | |
177 | reval.best_nclust_cv | 178 | |
| 181 | |
182 | reval.internal_baselines | 183 | |
| 186 | |
187 | reval.utils | 188 | |
| 191 | |
192 | reval.visualization | 193 | |
reval","Examples of how things can go wrong","Performance on benchmark datasets","Stability-based relative clustering validation to determine the best number of cluster","Installing"],titleterms:{"class":0,"function":0,base:4,benchmark:3,best:4,blob:3,can:2,cite:4,classifi:1,cluster:[1,4],code:0,dataset:3,descript:0,determin:4,enough:2,exampl:[2,3],featur:2,few:2,gaussian:3,good:2,how:[1,2],indic:4,instal:5,mnist:3,more:3,nois:3,number:[2,4],paramet:1,perform:3,rel:4,reval:1,sampl:2,select:1,stabil:4,tabl:4,thing:2,too:2,use:1,valid:4,visual:0,when:2,wrong:2}})
--------------------------------------------------------------------------------
/docs/make.bat:
--------------------------------------------------------------------------------
1 | @ECHO OFF
2 |
3 | pushd %~dp0
4 |
5 | REM Command file for Sphinx documentation
6 |
7 | if "%SPHINXBUILD%" == "" (
8 | set SPHINXBUILD=sphinx-build
9 | )
10 | set SOURCEDIR=source
11 | set BUILDDIR=build
12 |
13 | if "%1" == "" goto help
14 |
15 | %SPHINXBUILD% >NUL 2>NUL
16 | if errorlevel 9009 (
17 | echo.
18 | echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
19 | echo.installed, then set the SPHINXBUILD environment variable to point
20 | echo.to the full path of the 'sphinx-build' executable. Alternatively you
21 | echo.may add the Sphinx directory to PATH.
22 | echo.
23 | echo.If you don't have Sphinx installed, grab it from
24 | echo.http://sphinx-doc.org/
25 | exit /b 1
26 | )
27 |
28 | %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
29 | goto end
30 |
31 | :help
32 | %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
33 |
34 | :end
35 | popd
36 |
--------------------------------------------------------------------------------
/docs/source/code_description.rst:
--------------------------------------------------------------------------------
1 | Code description
2 | ================
3 |
4 | ``reval`` module has one superclass :class:`FindBestClustCV` and a subclass :class:`RelativeValidation`.
5 | :class:`SCParamSelection` and :class:`ParamSelection` have been added to later releases to perform
6 | hyperparameter selections.
7 |
8 | Classes
9 | -------
10 |
11 | .. autoclass:: reval.relative_validation.RelativeValidation
12 | :members:
13 | :private-members:
14 |
15 | .. autoclass:: reval.best_nclust_cv.FindBestClustCV
16 | :members:
17 | :private-members:
18 |
19 | .. autoclass:: reval.param_selection.SCParamSelection
20 | :members:
21 | :private-members:
22 |
23 | .. autoclass:: reval.param_selection.ParamSelection
24 | :members:
25 | :private-members:
26 |
27 | Functions
28 | ---------
29 |
30 | Useful functions that can be used on their own are also available. In particular,
31 | ``reval.utils.kuhn_munkres_algorithm`` is an implementation of the Kuhn-Munkres algorithm
32 | (Kuhn, 1955; Munkres, 1957), that performs consistent permutation of predicted labels
33 | in order to minimize the misclassification error with respect to true labels. ``reval.utils.compute_metrics``
34 | takes as input clustering and classification labels and returns classification metrics, such as F1 score,
35 | accuracy and Matthews correlation coefficient for generalization.
36 |
37 | Kuhn, H. W. (1955). The Hungarian method for the assignment problem. *Naval research logistics quarterly*,
38 | 2(1‐2), 83-97.
39 |
40 | Munkres, J. (1957). Algorithms for the assignment and transportation problems.
41 | *Journal of the society for industrial and applied mathematics*, 5(1), 32-38.
42 |
43 | .. automodule:: reval.utils
44 | :members: kuhn_munkres_algorithm, compute_metrics
45 | :private-members: _build_weight_mat
46 |
47 | The ``reval.best_nclust_cv._confint`` computes 95% confidence interval using ``scipy.stats.t.ppf()`` function.
48 |
49 | .. automodule:: reval.best_nclust_cv
50 | :members: _confint
51 | :private-members:
52 |
53 |
54 | The module ``reval.internal_baselines`` includes functions ``select_best`` and ``evaluate_best`` that allow comparisons
55 | between ``reval`` method and internal validation measures.
56 |
57 | .. automodule:: reval.internal_baselines
58 | :members: select_best, evaluate_best
59 |
60 | Visualization
61 | -------------
62 |
63 | ``reval.visualization`` enables plotting the cross-validation performance.
64 |
65 | .. automodule:: reval.visualization
66 | :members:
67 | :private-members:
68 |
--------------------------------------------------------------------------------
/docs/source/code_usage.rst:
--------------------------------------------------------------------------------
1 | How to use ``reval``
2 | ====================
3 |
4 | In the following, we are going to simulate N = 1,000 sample dataset with two groups and two features
5 | (for visualization purposes), then we will show how to apply the ``reval`` package and investigate
6 | the result types. We will use hierarchical clustering and KNN classification algorithms.
7 |
8 | First (after starting ``ipython`` or a jupyter notebook),
9 | let us import a bunch of useful libraries and our class ``reval.best_nclust_cv.FindBestClustCV``:
10 |
11 | .. code:: python3
12 |
13 | from reval.best_nclust_cv import FindBestClustCV
14 | from sklearn.datasets import make_blobs
15 | from sklearn.neighbors import KNeighborsClassifier
16 | from sklearn.cluster import AgglomerativeClustering
17 | from sklearn.model_selection import train_test_split
18 | import matplotlib.pyplot as plt
19 |
20 | Then we simulate the toy dataset and visualize it:
21 |
22 | .. code:: python3
23 |
24 | data = make_blobs(1000, 2, centers=2, random_state=42)
25 | plt.scatter(data[0][:, 0], data[0][:, 1],
26 | c=data[1], cmap='rainbow_r')
27 | plt.show()
28 |
29 | .. image:: images/useblobs.png
30 | :align: center
31 |
32 | Then, we split the dataset into training and test sets:
33 |
34 | .. code:: python3
35 |
36 | X_tr, X_ts, y_tr, y_ts = train_test_split(data[0], data[1],
37 | test_size=0.30,
38 | random_state=42,
39 | stratify=data[1])
40 |
41 | We apply the stability-based relative clustering validation approach with 10x2 repeated cross-validation,
42 | 10 iterations of random labeling, and number of clusters ranging from 2 to 10.
43 |
44 | .. code:: python3
45 |
46 | classifier = KNeighborsClassifier()
47 | clustering = AgglomerativeClustering()
48 | findbestclust = FindBestClustCV(nfold=2,
49 | nclust_range=list(range(2, 11)),
50 | s=classifier,
51 | c=clustering,
52 | nrand=100)
53 | metrics, nbest = findbestclust.best_nclust(X_tr, iter_cv=10, strat_vect=y_tr)
54 | out = findbestclust.evaluate(X_tr, X_ts, nbest)
55 |
56 | To obtain the training stability and the normalized validation stability for the
57 | selected number of clusters we need to call:
58 |
59 | .. code:: python3
60 |
61 | nbest
62 | # 2
63 | metrics['train'][nbest]
64 | # (0.0, (0.0, 0.0)) (stab, (stab, error))
65 | metrics['val'][nbest]
66 | # (0.0, (0.0, 0.0)) (stab, (stab, error))
67 |
68 | ``out`` returns train/test accuracies and test set clustering labels.
69 |
70 | .. code:: python3
71 |
72 | out.train_cllab
73 | # array([0, 1, 0, 1, 0, 0, 1...
74 | out.test_cllab
75 | # array([0, 0, 0, 0, 1...
76 | out.train_acc
77 | # 1.0
78 | out.test_acc
79 | # 1.0
80 |
81 | Attribute ``cv_results_`` of :class:`FindBestClustCV` returns a dataframe with training and validation
82 | misclassification errors.
83 |
84 | .. code:: python3
85 |
86 | findbestclust.cv_results_
87 |
88 | To visualize performance metrics during cross-validation, i.e., training stability and validation normalized stability
89 | with confidence intervals:
90 |
91 | .. code:: python3
92 |
93 | from reval.visualization import plot_metrics
94 | plot_metrics(metrics, title="Reval metrics")
95 |
96 | .. image:: images/performanceexample.png
97 | :align: center
98 |
99 | The plot can be customized and also show the normalized stability of a random classifier for each number of clusters
100 | to evaluate the model performance.
101 |
102 | .. image:: images/performanceexample2.png
103 | :align: center
104 |
105 | Classifier/clustering selection
106 | -------------------------------
107 |
108 | Let us now suppose that we are not sure which combination of clustering and classifier to use
109 | for the blobs dataset. We might want to try both hierarchical clustering and k-means and KNN and
110 | logistic regression. We import the libraries we have not imported before including the
111 | :class:`SCParamSelection` from the ``param_selection.py`` module.
112 |
113 | .. code:: python3
114 |
115 | from sklearn.linear_model import LogisticRegression
116 | from sklearn.cluster import KMeans
117 | from reval.param_selection import SCParamSelection
118 |
119 | We initialize the parameter selection class with a dictionary that includes the classification and
120 | clustering algorithms we want to run and we initialize a 10x2 repeated cross validation with 10 runs of random
121 | labeling. We set the number of parallel processes to 7 to speed up computations.
122 |
123 | .. code:: python3
124 |
125 | sc_params = {'s': [LogisticRegression(max_iter=1000), KNeighborsClassifier()],
126 | 'c': [AgglomerativeClustering(), KMeans()]}
127 | scparsel = SCParamSelection(sc_params, cv=2, nrand=10, n_jobs=7,
128 | iter_cv=10, clust_range=list(range(2, 11)),
129 | strat=y_tr)
130 | scparsel.fit(X_tr, nclass=2)
131 |
132 | In this case we knew the true number of clusters a priori, so we passed it to the ``fit()`` method in
133 | order to prioritize the parameter combinations that select the true number of clusters, along with the
134 | combinations with global minimum stability. As a result, four different combinations are run and all of
135 | them selected two as the best number of clusters with minimum stability.
136 |
137 | Parameter selection
138 | -------------------
139 |
140 | Let us now settle with hierarchical clustering and KNN and suppose we want to try different number of
141 | neighbors for KNN, i.e., 5 and 15, and different methods for hierarchical clustering,
142 | i.e., Ward and single-linkage. We can then use the :class:`ParamSelection` as follows:
143 |
144 | .. code:: python3
145 |
146 | from reval.param_selection import ParamSelection
147 | params = {'s': {'n_neighbors': [5, 15]},
148 | 'c': {'linkage': ['ward', 'single']}}
149 | parsel = ParamSelection(params, cv=2, s=KNeighborsClassifier(), c=AgglomerativeClustering(),
150 | nrand=10,
151 | n_jobs=7,
152 | iter_cv=10,
153 | strat=y_tr, clust_range=list(range(2, 11)))
154 | parsel.fit(X_tr, nclass=2)
155 |
156 | Also in this case we run four different hyperparameter combinations which all report 2 as the best number
157 | of clusters with minimum stability.
158 |
159 |
160 |
161 |
162 |
163 |
164 |
165 |
166 |
167 |
168 |
169 |
170 |
--------------------------------------------------------------------------------
/docs/source/conf.py:
--------------------------------------------------------------------------------
1 | # Configuration file for the Sphinx documentation builder.
2 | #
3 | # This file only contains a selection of the most common options. For a full
4 | # list see the documentation:
5 | # https://www.sphinx-doc.org/en/master/usage/configuration.html
6 |
7 | # -- Path setup --------------------------------------------------------------
8 |
9 | # If extensions (or modules to document with autodoc) are in another directory,
10 | # add these directories to sys.path here. If the directory is relative to the
11 | # documentation root, use os.path.abspath to make it absolute, like shown here.
12 | #
13 | import os
14 | import sys
15 |
16 | sys.path.insert(0, os.path.abspath('../../'))
17 | sys.setrecursionlimit(1500)
18 |
19 | # -- Project information -----------------------------------------------------
20 |
21 | project = 'Relative clustering validation'
22 | copyright = '2020, Isotta Landi'
23 | authors = 'Isotta Landi'
24 |
25 | # The full version, including alpha/beta/rc tags
26 | release = '0.0.1'
27 |
28 | # Add any Sphinx extension module names here, as strings. They can be
29 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
30 | # ones.
31 | extensions = ['sphinx.ext.autodoc',
32 | 'sphinx_rtd_theme',
33 | "sphinx.ext.intersphinx",
34 | "sphinx.ext.mathjax",
35 | "sphinx.ext.viewcode",
36 | "sphinx.ext.doctest",
37 | ]
38 |
39 | # Add any paths that contain templates here, relative to this directory.
40 | templates_path = ['_templates']
41 |
42 | master_doc = "index"
43 |
44 | # List of patterns, relative to source directory, that match files and
45 | # directories to ignore when looking for source files.
46 | # This pattern also affects html_static_path and html_extra_path.
47 | exclude_patterns = []
48 |
49 | pygments_style = "sphinx"
50 |
51 | # -- Options for HTML output -------------------------------------------------
52 |
53 | # The theme to use for HTML and HTML Help pages. See the documentation for
54 | # a list of builtin themes.
55 | #
56 |
57 | html_theme = 'sphinx_rtd_theme'
58 |
59 | # Add any paths that contain custom static files (such as style sheets) here,
60 | # relative to this directory. They are copied after the builtin static files,
61 | # so a file named "default.css" will overwrite the builtin "default.css".
62 | html_static_path = ['_static']
63 |
--------------------------------------------------------------------------------
/docs/source/images/blobs1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/blobs1.png
--------------------------------------------------------------------------------
/docs/source/images/blobsnoisy.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/blobsnoisy.png
--------------------------------------------------------------------------------
/docs/source/images/classes10.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/classes10.png
--------------------------------------------------------------------------------
/docs/source/images/classes100500500.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/classes100500500.png
--------------------------------------------------------------------------------
/docs/source/images/classes1005050.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/classes1005050.png
--------------------------------------------------------------------------------
/docs/source/images/classes20.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/classes20.png
--------------------------------------------------------------------------------
/docs/source/images/performance100500500.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/performance100500500.png
--------------------------------------------------------------------------------
/docs/source/images/performance1005050.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/performance1005050.png
--------------------------------------------------------------------------------
/docs/source/images/performance10features.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/performance10features.png
--------------------------------------------------------------------------------
/docs/source/images/performance20features.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/performance20features.png
--------------------------------------------------------------------------------
/docs/source/images/performanceblobs1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/performanceblobs1.png
--------------------------------------------------------------------------------
/docs/source/images/performanceexample.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/performanceexample.png
--------------------------------------------------------------------------------
/docs/source/images/performanceexample2.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/performanceexample2.png
--------------------------------------------------------------------------------
/docs/source/images/performancemnist.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/performancemnist.png
--------------------------------------------------------------------------------
/docs/source/images/performancenoisy.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/performancenoisy.png
--------------------------------------------------------------------------------
/docs/source/images/performanceumap.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/performanceumap.png
--------------------------------------------------------------------------------
/docs/source/images/predlab10.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/predlab10.png
--------------------------------------------------------------------------------
/docs/source/images/predlab100500500.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/predlab100500500.png
--------------------------------------------------------------------------------
/docs/source/images/predlab1005050.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/predlab1005050.png
--------------------------------------------------------------------------------
/docs/source/images/predlab20.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/predlab20.png
--------------------------------------------------------------------------------
/docs/source/images/predlabblobs1.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/predlabblobs1.png
--------------------------------------------------------------------------------
/docs/source/images/predlabmnist.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/predlabmnist.png
--------------------------------------------------------------------------------
/docs/source/images/predlabnoisy.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/predlabnoisy.png
--------------------------------------------------------------------------------
/docs/source/images/predlabumap.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/predlabumap.png
--------------------------------------------------------------------------------
/docs/source/images/revalv0.0.2pipeline.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/revalv0.0.2pipeline.png
--------------------------------------------------------------------------------
/docs/source/images/test100500500.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/test100500500.png
--------------------------------------------------------------------------------
/docs/source/images/testmnist.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/testmnist.png
--------------------------------------------------------------------------------
/docs/source/images/testumap.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/testumap.png
--------------------------------------------------------------------------------
/docs/source/images/trainmnist.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/trainmnist.png
--------------------------------------------------------------------------------
/docs/source/images/trainumap.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/trainumap.png
--------------------------------------------------------------------------------
/docs/source/images/useblobs.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/IIT-LAND/reval_clustering/e69015b50bf4cdd44c8c89dce37597b639c5bb8c/docs/source/images/useblobs.png
--------------------------------------------------------------------------------
/docs/source/index.rst:
--------------------------------------------------------------------------------
1 | .. Relative clustering validation documentation master file, created by
2 | sphinx-quickstart on Mon May 11 18:17:16 2020.
3 |
4 | Stability-based relative clustering validation to determine the best number of cluster
5 | ======================================================================================
6 |
7 | ``reval`` allows to determine the best clustering solution without a priori knowledge.
8 | It leverages a stability-based relative clustering validation method (Lange et al., 2004) that transforms
9 | a clustering algorithm into a supervised classification problem and selects the number of clusters
10 | that leads to the minimum expected misclassification error, i.e., stability.
11 |
12 | This library allows to:
13 |
14 | 1. Select any classification algorithm from ``sklearn`` library;
15 | 2. Select a clustering algorithm with ``n_clusters`` parameter or HDBSCAN density-based algorithm,
16 | i.e., choose among ``sklearn.cluster.KMeans``,
17 | ``sklearn.cluster.AgglomerativeClustering``, ``sklearn.cluster.SpectralClustering``, ``hdbscan.HDBSCAN``;
18 | 3. Perform (repeated) *k*-fold cross-validation to determine the best number of clusters;
19 | 4. Test the final model on an held-out dataset.
20 |
21 | Theoretical background can be found in (Lange et al., 2004), whereas code can be found on `github
22 |