Two previously published datasets are used as examples of the standardised PLS workflow and the proposed equivalent ANN workflow. The first, by Chan et al. (2016) is a urine NMR dataset comprised of 149 named metabolites, publicly available on Metabolomics Workbench (Study ID: ST0001047). Two classes were used: gastric cancer (n=43) vs. healthy controls (n=40). The second, by Ganna et al. (2014) and Ganna et al. (2015) is a plasma LC-MS with 189 named metabolites, publicly available on Metabolights (Study ID: MTBLS90). Samples were split into two classes by sex: males (n=485) and females (n=483).
10 |
11 | Due to the structural equivalence with PLS, a shallow (2-layer) ANN is used in this study. Provided the success of this approach towards visualisation and interrogation in shallow ANNs, it may then be possible to adapt this further to deeper ANN architectures. This shallow (2-layer) ANN architecture has a hidden layer consisting of multiple neurons (n = 2 to 6) with a sigmoidal activation, and an output layer consisting of a single neuron with a sigmoidal activation function.
12 |
13 | The standardised PLS workflow and the proposed equivalent ANN workflow include the following steps: hyperparameter optimisation, building and training the model, bootstrap resampling of the model, model evaluation, and model visualisation. All steps and accompanying visualisation methods are described in detail above each corresponding code cell within the workflows. These workflows were implemented using the Python programming language, and are presented as Jupyter Notebooks. There are three ways to that these can be accessed: as a static HTML file, in the cloud (using Binder), or downloaded and run on a local machine.
14 |
15 | This requires Python 3.x and Jupyter to be installed on your local machine. We recommend using the Anaconda Distribution, which can be download from the Anaconda Webpage (https://www.anaconda.com/distribution/). For information on installing Python and using Jupyter Notebooks, refer to the tutorial, "Toward collaborative open data science in metabolomics using Jupyter Notebooks and cloud computing" by Mendez et al. (2019).
35 |
36 | Note: If you are using Windows, you need to install git using the following: [Git for Windows](https://gitforwindows.org/)
37 |
38 | 1. Open Terminal on Linux/MacOS or Command Prompt on Windows
39 | 2. Enter the following into the console (one line at a time)
40 |
41 | ```console
42 | git clone https://github.com/cimcb/MetabProjectionViz
43 | cd MetabProjectionViz
44 | conda env create -f environment.yml
45 | conda activate MetabProjectionViz
46 | jupyter notebook
47 | ```
48 |
49 |
50 |
51 |
--------------------------------------------------------------------------------
/notebooks/PLSDA_MTBLS90.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {
6 | "toc-hr-collapsed": false
7 | },
8 | "source": [
9 | "\n",
10 | " To begin: Click anywhere in this cell and press Run on the menu bar. This executes the current cell and then highlights the next cell. There are two types of cells. A text cell and a code cell. When you Run a text cell (we are in a text cell now), you advance to the next cell without executing any code. When you Run a code cell (identified by In[ ]: to the left of the cell) you advance to the next cell after executing all the Python code within that cell. Any visual results produced by the code (text/figures) are reported directly below that cell. Press Run again. Repeat this process until the end of the notebook. NOTE: All the cells in this notebook can be automatically executed sequentially by clicking Kernel→Restart and Run All. Should anything crash then restart the Jupyter Kernal by clicking Kernel→Restart, and start again from the top.\n",
11 | " \n",
12 | "
"
13 | ]
14 | },
15 | {
16 | "cell_type": "markdown",
17 | "metadata": {},
18 | "source": [
19 | "\n",
20 | "
![](\"https://github.com/CIMCB/MetabComparisonBinaryML/blob/master/cimcb_logo.png?raw=true\")
\n",
21 | "\n",
22 | "
\n",
23 | "\n",
24 | "
Metabolomics Data Visualisation Workflow for PLS-DA
\n",
25 | "\n",
26 | "
\n",
27 | "
\n",
28 | "
\n",
29 | "
This Jupyter Notebook described a metabolomics data analysis and visualisation workflow for partial least squares regression (a.k.a. projection to latent structure) with a binary classification outcome.
\n",
30 | "\n",
31 | "
This computational workflow is described using a previously published NMR dataset by Ganna et al. (2014) and Ganna et al. (2015).The study compared the plasma metabolomic profile comparison across a large prospective epidemiological study of men (n=485) and women (n=483) at age 70 living in Uppsala, Sweden. For the purpose of this computational workflow, we compare only the males (Class=1) and females (Class=0) in a binary discriminant analysis. The deconvolved and annotated data from this study is deposited on Metabolights, and can be accessed directly via its Study ID: MTBLS90. The Excel file used in this workflow can be accessed via the following link: MTBLS90.xlsx.
\n",
32 | "\n",
33 | "
This computational workflow requires a dataset to be in, or converted to, a previously described standardised Excel file format (Mendez et al. 2019). This format uses the Tidy Data Framework (Wickham, 2014), where each row represents an observation (e.g. sample) and each column represents a variable (e.g. age or metabolite). Each excel file (per study) contains two sheets; a data sheet and a peak sheet. The data sheet contains the metabolite concentration together with the metadata associated for each observation (requiring the inclusion of the columns: Idx, SampleID, and Class). The peak sheet contains the additional metadata that pertains to the metabolites in the data sheet (requiring the inclusion of the columns: Idx, Name, and Label). The standardisation of this format allows for the efficient re-use of this computational workflow.
\n",
34 | "\n",
35 | "
\n",
36 | "The steps included in this data analysis and visualisation workflow are: \n",
37 | "
\n",
38 | "\n",
39 | "1.
Import Packages\n",
40 | "2.
Load Data & Peak Sheet\n",
41 | "3.
Extract X & Y\n",
42 | "4.
Split Data into Train & Test Set\n",
43 | "5.
Extract, Transform, & Scale X Data with Missing Values Imputed\n",
44 | "6.
Hyperparameter Optimisation\n",
45 | " 6.1.
Plot R² & Q²\n",
46 | " 6.2.
Plot Latent Projections: Full & CV\n",
47 | "7.
Build Model & Evaluate\n",
48 | "8.
Permutation Test\n",
49 | "9.
Bootstrap Resampling of the Model \n",
50 | "10.
Model Evaluation using Bootstrap Resampling \n",
51 | "11.
Model Visualisation \n",
52 | " 11.1.
Plot Latent Projections: in-bag & out-of-bag\n",
53 | " 11.2.
Plot Weight Vectors\n",
54 | "12.
Variable Contribution Plots \n",
55 | "13.
Export Results\n",
56 | "\n",
57 | "
\n",
67 | " \n",
68 | "
\n",
69 | "
1. Import Packages
\n",
70 | "\n",
71 | "
Packages provide additional tools that extend beyond the basic functionality of the Python programming. Prior to usage, packages need to be imported into the Jupyter environment. The following packages need to be imported for this computational workflow:
\n",
72 | "\n",
73 | "
\n",
74 | "numpy: A standard package primarily used for the manipulation of arrayspandas: A standard package primarily used for the manipulation of data tablescimcb: A library of helpful functions and tools provided by the authorssklearn: A standard package with tools for machine learning\n",
81 | "\n",
82 | "\n",
83 | "train_test_split: A method to split arrays into training and test subsets
\n",
84 | "\n",
85 | "
\n",
88 | "\n",
89 | "
\n",
90 | "\n",
91 | "
\n",
113 | " \n",
114 | "
\n",
115 | "
Optional: Set Random Seed for Splitting Data into Training & Test sets
\n",
116 | "\n",
117 | "
To reproduce the figures in the research article, set the random seed to 8. This seed is used in the train_test_split method to reproducibly split the source data into a training and test set.
\n",
118 | "\n",
119 | "
\n",
122 | "
\n",
123 | "\n",
124 | "
\n",
142 | "\n",
143 | "
\n",
144 | "
2. Load Data & Peak Sheet
\n",
145 | "\n",
146 | "
This CIMCB helper function load_dataXL() loads the Data and Peak sheet from an Excel file. In addition, this helper function checks that the data is in the standardised Excel file format described above. After the initial checks, load_dataXL() outputs two individual Pandas DataFrames (i.e. tables) called DataTable and PeakTable from the Excel file MTBLS90.xlsx. This helper function requires values for the following parameters:
\n",
147 | "
\n",
148 | " filename: The name of the excel file (.xlsx file)DataSheet: The name of the data sheet in the filePeakSheet: The name of the peak sheet in the file
\n",
152 | "
\n",
153 | "\n",
154 | "
\n",
174 | "\n",
175 | "
\n",
176 | "
3. Extract X & Y
\n",
177 | "\n",
178 | "
Prior to performing any statistical or machine learning modelling, it is best practice to assess the quality of the data and remove metabolites that lack reproducible measurements (Broadhurst et al. 2018). \n",
179 | "\n",
180 | "
\n",
181 | "
The following steps are needed to extract the X matrix of metabolite concentrations and associated Y vector of classification labels (“M”=1 and “F”=0):\n",
182 | " \n",
183 | "
\n",
184 | " \n",
185 | "- Create a subset of
DataTable called DataTable2, with samples only in the Class “M” or “F” \n",
186 | " \n",
187 | "\n",
188 | "- Create the variable
PeakList to hold the names (M1...Mn) of the metabolites to be used \n",
189 | "\n",
190 | "- Using this
PeakList, extract all corresponding columns (i.e. metabolite data) from DataTable2, and place it in matrix X \n",
191 | "\n",
192 | "- Set
Y to a list (or 1D array) of binary outcomes based on the Class column from DataTable2 (“M”=1 and “F”=0) \n",
193 | "\n",
194 | "
\n",
195 | "\n",
196 | "
\n",
197 | "
\n",
228 | "\n",
229 | "
\n",
230 | "
4. Split Data into Train & Test Set
\n",
231 | "\n",
232 | "\n",
233 | "
The train_test_split method is used to split the X and Y data into training (2/3rd) and test (1/3rd) sets using stratified random selection. Additionally, the Class data is split for use in figure legends. The seed is selected in the optional section above. For further information on this method, refer to the scikit learn documentation.\n",
234 | "\n",
235 | "
\n",
236 | "
\n",
259 | " \n",
260 | "
\n",
261 | "
5. Extract, Transform, & Scale X Data with Missing Values Imputed
\n",
262 | "\n",
263 | "
The X Data (XTrain and XTest) is log transformed, mean centred, and scaled to unit variance (with missing values imputed using K-Nearest Neighbour) prior to modelling following standard protocols for metabolomics (Broadhurst and Kell, 2006).
\n",
264 | "
\n",
265 | " \n",
266 | "- Log-transform the values in
XTrain \n",
267 | "\n",
268 | "- Using the helper function
cb.utils.scale(), scale the log-transformed data (XTrainLog) to the unit variance (a.k.a. auto scaling), while also returning mu & sigma. \n",
269 | "\n",
270 | "- Impute the missing values by using a k-nearest neighbour approach (with three neighbours) using the helper function
cb.utils.knnimpute() to give the final matrix, XTrainKnn \n",
271 | "\n",
272 | "- Log-transform the values in
XTest \n",
273 | "\n",
274 | "- Using the helper function
cb.utils.scale(), scale the log-transformed data (XTestLog) to the unit variance (a.k.a. auto scaling) using the mu & sigma from above.\n",
275 | " \n",
276 | " - Impute the missing values by using a k-nearest neighbour approach (with three neighbours) using the helper function
cb.utils.knnimpute() to give the final matrix, XTestKnn \n",
277 | " \n",
278 | "
\n",
279 | "\n",
280 | "
\n",
305 | " \n",
306 | "
\n",
307 | "
6. Hyperparameter Optimisation
\n",
308 | "\n",
309 | "
The CIMCB helper function cb.cross_val.KFold() is used to carry out k-fold cross-validation (k=5) on a set of PLS-DA models with varying number of latent variables (1 to 6) to determine the optimal number. In k-fold cross-validation, the original dataset is randomly split into k sized folds and subsequently trained for k iterations, where the model is trained on 1 – k folds and tested on the k fold (Kohavi 1995). This helper function requires values for the following parameters:
\n",
310 | " \n",
311 | "
\n",
312 | " model: The class of model used by the function, cb.model.PLS_SIMPLSX: The metabolite data matrix, XTrainKnnY: The binary outcome vector, YTrainparam_dict: a dictionary, param_dict, that describes all key:value pairs to search, with the key name corresponding to the hyperparameter in the model class and the value as the list of possible valuesfolds: The number of folds in the k-fold cross validationn_mc: The number of Monte Carlo repetitions of the k-fold CV
\n",
319 | "
\n",
320 | "
\n",
349 | " \n",
350 | "
\n",
351 | "
6.1. Plot R² & Q²
\n",
352 | "\n",
353 | "
When cv.plot(metric='r2q2', method='absolute') is run, 2 plots of $R^2$ and $Q^2$ statistics are displayed: (a) the absolute difference of ($R^2 - Q^2$) vs. $Q^2$, and (b) $R^2$ and $Q^2$ against the number of latent variables. Alternatively, if method='ratio', plot (a) is the absolute difference of ($R^2 - Q^2$) / $R^2$ vs. $Q^2$. The optimal number of hyperparameters is selected based on the point of inflection in figure b, or if a clear inflection point is not present, where | ($R^2 - Q^2$) | = 0.2. Note, the $R^2$ is the mean coefficient of determination for the full dataset, and the $Q^2$ is the mean coefficient of determination for cross-validated prediction dataset over the 10 Monte Carlo repetitions. The following parameters of cv.plot() can be altered:
\n",
354 | "
\n",
355 | " metric: The metric used for the plots (default = 'r2q2'). Alternative metrics include 'auc', 'acc', 'f1score', 'prec', 'sens', and 'spec'\n",
356 | " method: The types of plots displayed (default = 'absolute'). Alternative value is 'ratio'\n",
357 | " ci: The confidence interval in figure b (default = 95)\n",
358 | " legend: to show legend (default = True). Alternative value is False\n",
359 | "
\n",
360 | "\n",
361 | "
\n",
362 | "
\n",
382 | "\n",
383 | "
\n",
384 | "
6.2. Plot Latent Projections: Full & CV
\n",
385 | " \n",
386 | "
When cv.plot_projections() is run, an n x n grid of plots are displayed, where n is the number of latent variables (LV) to interrogate. These plots include score plots, distribution plots, and receiver operating characteristic (ROC) curves.
\n",
387 | "\n",
388 | "
There are C(n,2) score plots (i.e. a score plot for every combination of 2 LVs e.g. LV1 scores vs. LV2 scores). Each score plot includes the full scores (as circles) and CV scores (as crosses) coloured by group, as well as the 95% confidence interval ellipses for the full scores (as solid lines) and CV scores (as dashed lines). Additionally, the optimal line of separation (dashed grey line) and orthogonal line (solid grey line) are shown.
\n",
389 | "\n",
390 | "
There are n distribution plots (a distribution plot for each LV scores). The distribution of the full and CV scores for each corresponding group (i.e. 4 discrete distributions overlayed for 2 groups). Each distribution is calculated using kernel density estimation, a standard non-parametric method used for estimation of a probability density function based on a set of data points (Silverman 1986).
\n",
391 | "\n",
392 | "
There are C(n,2) ROC curves (a ROC curve for every combination of 2 LVs e.g. LV1 scores vs. LV2 scores). As the ROC curves are for every combination of 2 LVs, the discrimination is calculated based on optimal separation (i.e. the grey line from the corresponding score plot). For each ROC curve plot there is a ROC curve for the full model (green), and ROC curve for the cv model with 95% confidence intervals (yellow). Additionally, the equal distribution line (dashed black line) is shown.
\n",
393 | "\n",
394 | "
\n",
395 | " components: LVs to plot (default = \"all\" ; plot all components). Alternatively, list the components to plot e.g. [1,3,4]plot: Data to show (default = 'ci' ; plot only 95% confidence interval ellipses). Alternative values include 'meanci', 'full', 'cv', and 'all'label: Add labels to groups (default = None ; refers to groups as 0/1)\n",
398 | " legend: Show legends for plots (default = 'all'). Alternative values are 'scatter', 'dist', 'roc', and 'none'.
\n",
400 | "\n",
401 | "
\n",
402 | "
\n",
434 | "\n",
435 | "
\n",
436 | "
7. Build Model & Evaluate
\n",
437 | "\n",
438 | "
A PLS-DA model using cb.model.PLS_SIMPLS is created and initialised using the optimal hyperparameter values determined in step 4 (i.e. the number of latent variables). The implementation of PLS in the cb.model.PLS_SIMPLS class uses the SIMPLS algorithm (De Jong, 1993).
\n",
439 | " \n",
440 | "
Following this initialisation, the PLS-DA model is trained using the .train(X, Y) method where the X matrix is XTrainKnn and the Y vector is YTrain, returning the Y predicted value YPredTrain. This model is then tested using the .test(X, Y) method where the X matrix is XTestKnn and the Y vector is YTest, returning the Y predicted value YPredTest.
\n",
441 | "\n",
442 | "
The .evaluate() method can be used to evaluate the predictability of the model using the train and test set. There are three plots produced when this method is run including a violin plot, probability density function, and a ROC curve. The violin plots show the predicted score for the train and test (by group). The distribution plot shows the probability density function of the predicted scores for the train and test (by group). The ROC curve shows the ROC curve for the train (green) and test (yellow). The following parameter values in .evaluate() can be altered:\n",
443 | " \n",
444 | "
\n",
445 | " testset: Plot test dataset (default = None). Alternative, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
446 | " label: Add labels to groups (default = None ; refer to groups as 0/1)\n",
447 | " legend: Show legends for plots (default = 'all'). Alternative values are 'roc', 'dist', 'violin', and 'none'
\n",
450 | "
\n",
485 | "\n",
486 | "
\n",
487 | "
8. Permutation Test
\n",
488 | "\n",
489 | "
After a model has been trained, the .permutation_test() method can be used to assess the reliability of the trained model (after selecting the number of latent variables). For the permutation test, the metabolite data matrix is randomised (permuted or 'shuffled'), while the Y (i.e. outcome) is fixed, and subsequently trained and tested on this randomised data (Szymańska et al. 2012). This process is repeated (in this case, n=100) to construct a distribution to fairly access the model. For a dataset with features that have with no meaningful contribution, we would expect a similar $R^2$ and $Q^2$ to a randomised dataset, while for a dataset with features with meaningful contribution, we would expect a $R^2$ and $Q^2$ significantly higher than that of the randomised dataset. When .permutation_test() is run, 2 plots are displayed: (a) $R^2$ and $Q^2$ against \"correlation of permuted data against original data\", and (b) probability density functions for $R^2$ and $Q^2$, with the $R^2$ and $Q^2$ values found for the model trained on original data presented as ball-and-stick. The following parameter value of .permutation_test() can be altered: \n",
490 | "\n",
491 | "
\n",
492 | " metric: The metric used for the plots (default = 'r2q2'). Alternative metrics include 'auc', 'acc', 'f1score', 'prec', 'sens', and 'spec'. Multiple metrics can be plotted using a list e.g. ['r2q2', 'auc]\n",
493 | " nperm: The number of permutations. (default = 100)\n",
494 | " legend: To show legend (default = True). Alternative value is False\n",
495 | "
\n",
496 | "\n",
497 | "
\n",
498 | "
\n",
517 | "\n",
518 | "
\n",
519 | "
9. Bootstrap Resampling of the Model
\n",
520 | "\n",
521 | "
Bootstrap resampling is a resampling method based on random resampling with replacement, commonly used to provide an estimate of sampling distribution of a test statistic (Efron, 1982). In the context of this workflow, the PLS model from step 5 with its fixed hyperparameter values (i.e. number of LVs = 2) is retrained on the resampled with replacement data (in-bag) and evaluated on the unused data (out-of-bag) for 100 resamples. After the model is evaluated for each bootstrap, metrics including the predicted values (ypred), LV scores, LV loadings, and feature importance (VIP and coefficients) are stored and used to calculate 95% confidence intervals. To calculate the 95% confidence intervals, various methods can be used including the basic percentile method, corrected percentile method (a.k.a. bias-corrected method), and the commonly used bias-corrected and accelerated (BCA) method. In this example, the BCA method is used with the class cb.boostrap.BCA. Alternatively, use cb.boostrap.Per to use the percentile method, or cb.bootstrap.CPer for the corrected percentile method. To create and run the bootmodel for any method, the following parameter values need to be set:\n",
522 | " \n",
523 | "
\n",
524 | " model: A model with fixed hyperparameter values for boostrap resamplingbootnum: The number of bootstrap resamples (default = 100)
\n",
528 | "
\n",
554 | "\n",
555 | "
\n",
556 | "
10. Model Evaluation using Bootstrap Resampling
\n",
557 | "\n",
558 | "
After the bootmodel has been run, the .evaluate() method can be used to provide an estimate of the robustness and a measure of the generalised predictability of the model. There are three plots produced when this method is run including a violin plot, probability density function, and a ROC curve. The violin plots show the distribution of the median predicted score for the in-bag and out-of-bag (i.e. train and test) by group. The distribution plot shows the probability density function of the median predicted score for the in-bag and out-of-bag (i.e. train and test) by group. The ROC curve shows the ROC curve with the median (green) and 95% CI for the in-bag (light green band) and the median (yellow) and 95% CI for the out-of-bag (light yellow band). The method used to calculate the 95% CI for the in-bag (green) is the class selected in the previous cell. In this example, the bias-corrected and accelerated method is used as cb.bootstrap.BCA was used in the previous cell to create bootmodel. \n",
559 | " \n",
560 | "
\n",
561 | " label: Add labels to groups (default = None ; refer to groups as 0/1)\n",
562 | " legend: Show legends for plots (default = 'all'). Alternative values are 'roc', 'dist', 'violin', and 'none'trainset: Plot train dataset instead of median in-bag (default = None). Alternatively, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
564 | " testset: Plot test dataset instead of median out-of-bag (default = None). Alternatively, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
565 | "
\n",
566 | "
\n",
586 | "\n",
587 | "
\n",
588 | "
11. Model Visualisation
\n",
589 | "\n",
590 | "
\n",
591 | "
\n",
599 | " \n",
600 | "
\n",
601 | "
11.1 Plot Latent Projections: in-bag & out-of-bag
\n",
602 | " \n",
603 | "
After the bootmodel has been run, the .plot_projections() method can be used to visualise the latent variable (LV) scores. When this method is run, a n x n grid of plots are displayed, where n is the number of LVs. These plots include score plots, distribution plots, and receiver operating characteristic (ROC) curves.
\n",
604 | "\n",
605 | "
There are C(n,2) score plots (i.e. a score plot for every combination of 2 LVs e.g. LV1 scores vs. LV2 scores). Each score plot includes the in-bag scores (as circles) and out-of-bag scores (as crosses) coloured by group, as well as the 95% confidence interval ellipses for the in-bag scores (as solid lines) and out-of-bag scores (as dashed lines). Additionally, the optimal line of separation (dashed grey line) and orthogonal line (solid grey line) are shown.
\n",
606 | "\n",
607 | "
There are n distribution plots (a distribution plot for each LV scores). The distribution of the in-bag and out-of-bag scores for each corresponding group (i.e. 4 discrete distributions overlayed for 2 groups). Each distribution is calculated using kernel density estimation, a standard non-parametric method used for estimation of a probability density function based on a set of data points (Silverman 1986).
\n",
608 | "\n",
609 | "
There are C(n,2) ROC curves (a ROC curve for every combination of 2 LVs e.g. LV1 scores vs. LV2 scores). As the ROC curves are for every combination of 2 LVs, the discrimination is calculated based on optimal separation (i.e. the grey line from the corresponding score plot). For each ROC curve plot there is a ROC curve with the LV score for the initial model with the 95% confidence intervals using the in-bag LV scores (green), and a ROC curve for the out-of-bag LV scores with 95% confidence intervals. The method used to calculate the 95% CI for the in-bag (green) is the class used to create the bootmodel. In this example, the bias-corrected and accelerated method is used (cb.bootstrap.BCA). Additionally, the equal distribution line (dashed black line) is shown. \n",
610 | "\n",
611 | "
\n",
612 | "\n",
613 | "
\n",
614 | " plot: Data to show in plot (default = \"ci\" ; plot only 95% confidence interval ellipses). Alternative values include 'meanci', 'ib', 'oob', and 'all'label: Add labels to groups in scores plot (default = None ; refer to groups as 0/1).\n",
616 | " legend: Show legends for plots (default = 'all'). Alternative values are 'scatter', 'dist', 'roc', and 'none'
\n",
619 | "
\n",
660 | " \n",
661 | "
\n",
662 | "
11.2 Plot Weight Vectors
\n",
663 | "\n",
664 | "
After the bootmodel has been run, the .plot_weights() method can be used to visualise the latent variable (LV) weight vectors. When this method is run, n plots are displayed, where n is the number of LVs. The circles in each plot represent the LV weight vectors for the initial model. The 95% confidence intervals are calculated using bias-correct (BC) bootstrap method in step 6. Any metabolite weights with a confidence interval crossing the zero line are considered non-significant to the latent variable. This method requires values for the following parameters:
\n",
665 | " \n",
666 | "
\n",
667 | " PeakTable: Cleaned PeakTable from step 3PeakList: Peaks to include in plot (default = None; include all samples).\n",
669 | " plot: To plot the data or median as circles (default 'data'). Alternative values include 'median', and a list structured as [LV1, LV2, etc.]\n",
670 | " ylabel: Name of column in PeakTable to use as the ylabel (default = 'Label')\n",
671 | " sort: Whether to sort plots in absolute descending order (default = True)
\n",
673 | " \n",
674 | "
\n",
675 | "
\n",
698 | "\n",
699 | "
\n",
700 | "
12. Variable Contribution Plots
\n",
701 | "\n",
702 | "
After the bootmodel has been run, the .plot_featureimportance() method can be used to visualise the feature importance metrics. When this method is run, 2 plots are displayed; the Coefficient Plot and Variable Importance in Projection (VIP) plot. The circles represent the values in the initial model. The 95% confidence intervals are calculated using bias-correct (BC) bootstrap method in step 6.
\n",
703 | " \n",
704 | "
The coefficients (in the Coefficient plot) contain information about the overall contribution of each metabolite. The coefficient values can either a positive or negative number, and therefore, negatively or positively contribute to the model. Any metabolite coefficient value with a confidence interval crossing the zero line is considered non-significant to the model.
\n",
705 | " \n",
706 | "
The values in the VIP plot contain information about the overall contribution of each metabolite. Unlike the coefficient values, the VIP is absolute, with the higher values representing a higher significance to the model. Typically, metabolites with a VIP greater than 1 are considered \"important\" in the model.
\n",
707 | " \n",
708 | "
This method, bootmodel exports the feature importance metrics as a pandas DataFrame (table). This method also requires values for the following parameters:
\n",
709 | " \n",
710 | "
\n",
711 | " PeakTable: Cleaned PeakTable from step 3PeakList: Peaks to include in plot (default = None; include all samples).\n",
713 | " plot: To plot the data or median as circles (default 'data'). Alternative values include 'median', and a list structured as [Coef, VIP].\n",
714 | " ylabel: Name of column in PeakTable to use as the ylabel (default = 'Label')\n",
715 | " sort: Whether to sort plots in absolute descending order (default = True)
\n",
717 | " \n",
718 | "
\n",
719 | "
\n",
740 | "\n",
741 | "
\n",
742 | "
13. Export Results
\n",
743 | "\n",
744 | "
The feature importance table created in step 8.3 can be exported using the inbuilt .to_excel() function within a pandas DataFrame. This function requires an input with the name of the file to create, and it can include directories by using the ‘ / ’ symbol. In the cell below, the table feature_importance is exported as an excel file called 'PLSDA_ST001047.xlsx' in the 'results' folder.
\n",
745 | "\n",
746 | "
\n",
10 | " To begin: Click anywhere in this cell and press Run on the menu bar. This executes the current cell and then highlights the next cell. There are two types of cells. A text cell and a code cell. When you Run a text cell (we are in a text cell now), you advance to the next cell without executing any code. When you Run a code cell (identified by In[ ]: to the left of the cell) you advance to the next cell after executing all the Python code within that cell. Any visual results produced by the code (text/figures) are reported directly below that cell. Press Run again. Repeat this process until the end of the notebook. NOTE: All the cells in this notebook can be automatically executed sequentially by clicking Kernel→Restart and Run All. Should anything crash then restart the Jupyter Kernal by clicking Kernel→Restart, and start again from the top.\n",
11 | " \n",
12 | "
"
13 | ]
14 | },
15 | {
16 | "cell_type": "markdown",
17 | "metadata": {},
18 | "source": [
19 | "\n",
20 | "
\n",
21 | "\n",
22 | "
\n",
23 | "\n",
24 | "
Metabolomics Data Visualisation Workflow for PLS-DA
\n",
25 | "\n",
26 | "
\n",
27 | "
\n",
28 | "
\n",
29 | "
This Jupyter Notebook described a metabolomics data analysis and visualisation workflow for partial least squares regression (a.k.a. projection to latent structure) with a binary classification outcome.
\n",
30 | "\n",
31 | "
This computational workflow is described using a previously published NMR dataset by Chan et al. (2016). The study compared the urine metabolomic profile comparison across patients characterised as Gastric Cancer (GC; n=43), Benign Gastric Disease (BN; n=40), and Healthy Control (HE; n=40) using 149 named metabolites. For the purpose of this computational workflow, we compare only the GC vs HE samples in a binary discriminant analysis. The deconvolved and annotated data from this study are deposited on Metabolomics Workbench (Study ID: ST001047), and can be accessed directly via its Project DOI: 10.21228/M8B10B. The Excel file used in this workflow can be accessed via the following link: ST001047.xlsx.
\n",
32 | "\n",
33 | "
This computational workflow requires a dataset to be in, or converted to, a previously described standardised Excel file format (Mendez et al. 2019). This format uses the Tidy Data Framework (Wickham, 2014), where each row represents an observation (e.g. sample) and each column represents a variable (e.g. age or metabolite). Each excel file (per study) contains two sheets; a data sheet and a peak sheet. The data sheet contains the metabolite concentration together with the metadata associated for each observation (requiring the inclusion of the columns: Idx, SampleID, and Class). The peak sheet contains the additional metadata that pertains to the metabolites in the data sheet (requiring the inclusion of the columns: Idx, Name, and Label). The standardisation of this format allows for the efficient re-use of this computational workflow.
\n",
34 | "\n",
35 | "
\n",
36 | "The steps included in this data analysis and visualisation workflow are: \n",
37 | "
\n",
38 | "\n",
39 | "1.
Import Packages\n",
40 | "2.
Load Data & Peak Sheet\n",
41 | "3.
Extract X & Y\n",
42 | "4.
Split Data into Train & Test Set\n",
43 | "5.
Extract, Transform, & Scale X Data with Missing Values Imputed\n",
44 | "6.
Hyperparameter Optimisation\n",
45 | " 6.1.
Plot R² & Q²\n",
46 | " 6.2.
Plot Latent Projections: Full & CV\n",
47 | "7.
Build Model & Evaluate\n",
48 | "8.
Permutation Test\n",
49 | "9.
Bootstrap Resampling of the Model \n",
50 | "10.
Model Evaluation using Bootstrap Resampling \n",
51 | "11.
Model Visualisation \n",
52 | " 11.1.
Plot Latent Projections: in-bag & out-of-bag\n",
53 | " 11.2.
Plot Weight Vectors\n",
54 | "12.
Variable Contribution Plots \n",
55 | "13.
Export Results\n",
56 | "\n",
57 | "
\n",
67 | " \n",
68 | "
\n",
69 | "
1. Import Packages
\n",
70 | "\n",
71 | "
Packages provide additional tools that extend beyond the basic functionality of the Python programming. Prior to usage, packages need to be imported into the Jupyter environment. The following packages need to be imported for this computational workflow:
\n",
72 | "\n",
73 | "
\n",
74 | "numpy: A standard package primarily used for the manipulation of arrayspandas: A standard package primarily used for the manipulation of data tablescimcb: A library of helpful functions and tools provided by the authorssklearn: A standard package with tools for machine learning\n",
81 | "\n",
82 | "\n",
83 | "train_test_split: A method to split arrays into training and test subsets
\n",
84 | "\n",
85 | "
\n",
88 | "\n",
89 | "
\n",
90 | "\n",
91 | "
\n",
113 | " \n",
114 | "
\n",
115 | "
Optional: Set Random Seed for Splitting Data into Training & Test sets
\n",
116 | "\n",
117 | "
To reproduce the figures in the research article, set the random seed to 40. This seed is used in the train_test_split method to reproducibly split the source data into a training and test set.
\n",
118 | "\n",
119 | "
\n",
122 | "
\n",
123 | "\n",
124 | "
\n",
142 | "\n",
143 | "
\n",
144 | "
2. Load Data & Peak Sheet
\n",
145 | "\n",
146 | "
This CIMCB helper function load_dataXL() loads the Data and Peak sheet from an Excel file. In addition, this helper function checks that the data is in the standardised Excel file format described above. After the initial checks, load_dataXL() outputs two individual Pandas DataFrames (i.e. tables) called DataTable and PeakTable from the Excel file ST001047.xlsx. This helper function requires values for the following parameters:
\n",
147 | "
\n",
148 | " filename: The name of the excel file (.xlsx file)DataSheet: The name of the data sheet in the filePeakSheet: The name of the peak sheet in the file
\n",
152 | "
\n",
153 | "\n",
154 | "
\n",
174 | "\n",
175 | "
\n",
176 | "
3. Extract X & Y
\n",
177 | "\n",
178 | "
Prior to performing any statistical or machine learning modelling, it is best practice to assess the quality of the data and remove metabolites that lack reproducible measurements (Broadhurst et al. 2018). In this dataset ST001047.xlsx, we can find that the QC-RSD and percentage of missing value has been previously calculated (refer to the peak sheet). In this Jupyter Notebook, we remove all metabolites that do not meet the following criteria:
\n",
179 | "\n",
180 | "
\n",
181 | "- QC-RSD less than 20%
\n",
182 | "\n",
183 | "- Fewer than 10% of values are missing
\n",
184 | "
\n",
185 | "\n",
186 | "
\n",
187 | "
The following steps are needed to extract the X matrix of metabolite concentrations and associated Y vector of classification labels (“GC”=1 and “HE”=0):\n",
188 | " \n",
189 | "
\n",
190 | " \n",
191 | "- Create a subset of
DataTable called DataTable2, with samples only in the Class “GC” or “HE” \n",
192 | " \n",
193 | "\n",
194 | "- Create the variable
PeakList to hold the names (M1...Mn) of the metabolites to be used \n",
195 | "\n",
196 | "- Using this
PeakList, extract all corresponding columns (i.e. metabolite data) from DataTable2, and place it in matrix X \n",
197 | "\n",
198 | "- Set
Y to a list (or 1D array) of binary outcomes based on the Class column from DataTable2 (“GC”=1 and “HE”=0) \n",
199 | "\n",
200 | "
\n",
201 | "\n",
202 | "
\n",
203 | "
\n",
238 | "\n",
239 | "
\n",
240 | "
4. Split Data into Train & Test Set
\n",
241 | "\n",
242 | "\n",
243 | "
The train_test_split method is used to split the X and Y data into training (2/3rd) and test (1/3rd) sets using stratified random selection. Additionally, the Class data is split for use in figure legends. The seed is selected in the optional section above. For further information on this method, refer to the scikit learn documentation.\n",
244 | "\n",
245 | "
\n",
246 | "
\n",
269 | " \n",
270 | "
\n",
271 | "
5. Extract, Transform, & Scale X Data with Missing Values Imputed
\n",
272 | "\n",
273 | "
The X Data (XTrain and XTest) is log10 transformed, mean centred, and scaled to unit variance (with missing values imputed using K-Nearest Neighbour) prior to modelling following standard protocols for metabolomics (Broadhurst and Kell, 2006).
\n",
274 | "
\n",
275 | " \n",
276 | "- Log-transform the values in
XTrain \n",
277 | "\n",
278 | "- Using the helper function
cb.utils.scale(), scale the log-transformed data (XTrainLog) to the unit variance (a.k.a. auto scaling), while also returning mu & sigma. \n",
279 | "\n",
280 | "- Impute the missing values by using a k-nearest neighbour approach (with three neighbours) using the helper function
cb.utils.knnimpute() to give the final matrix, XTrainKnn \n",
281 | "\n",
282 | "- Log-transform the values in
XTest \n",
283 | "\n",
284 | "- Using the helper function
cb.utils.scale(), scale the log-transformed data (XTestLog) to the unit variance (a.k.a. auto scaling) using the mu & sigma from above.\n",
285 | " \n",
286 | " - Impute the missing values by using a k-nearest neighbour approach (with three neighbours) using the helper function
cb.utils.knnimpute() to give the final matrix, XTestKnn \n",
287 | " \n",
288 | "
\n",
289 | "\n",
290 | "
\n",
315 | " \n",
316 | "
\n",
317 | "
6. Hyperparameter Optimisation
\n",
318 | "\n",
319 | "
The CIMCB helper function cb.cross_val.KFold() is used to carry out k-fold cross-validation (k=5) on a set of PLS-DA models with varying number of latent variables (1 to 6) to determine the optimal number. In k-fold cross-validation, the original dataset is randomly split into k sized folds and subsequently trained for k iterations, where the model is trained on 1 – k folds and tested on the k fold (Kohavi 1995). This helper function requires values for the following parameters:
\n",
320 | " \n",
321 | "
\n",
322 | " model: The class of model used by the function, cb.model.PLS_SIMPLSX: The metabolite data matrix, XTrainKnnY: The binary outcome vector, YTrainparam_dict: a dictionary, param_dict, that describes all key:value pairs to search, with the key name corresponding to the hyperparameter in the model class and the value as the list of possible valuesfolds: The number of folds in the k-fold cross validationn_mc: The number of Monte Carlo repetitions of the k-fold CV
\n",
329 | "
\n",
330 | "
\n",
359 | " \n",
360 | "
\n",
361 | "
6.1. Plot R² & Q²
\n",
362 | "\n",
363 | "
When cv.plot(metric='r2q2', method='absolute') is run, 2 plots of $R^2$ and $Q^2$ statistics are displayed: (a) the absolute difference of ($R^2 - Q^2$) vs. $Q^2$, and (b) $R^2$ and $Q^2$ against the number of latent variables. Alternatively, if method='ratio', plot (a) is the absolute difference of ($R^2 - Q^2$) / $R^2$ vs. $Q^2$. The optimal number of hyperparameters is selected based on the point of inflection in figure b, or if a clear inflection point is not present, where | ($R^2 - Q^2$) | = 0.2. Note, the $R^2$ is the mean coefficient of determination for the full dataset, and the $Q^2$ is the mean coefficient of determination for cross-validated prediction dataset over the 10 Monte Carlo repetitions. The following parameters of cv.plot() can be altered:
\n",
364 | "
\n",
365 | " metric: The metric used for the plots (default = 'r2q2'). Alternative metrics include 'auc', 'acc', 'f1score', 'prec', 'sens', and 'spec'\n",
366 | " method: The types of plots displayed (default = 'absolute'). Alternative value is 'ratio'\n",
367 | " ci: The confidence interval in figure b (default = 95)\n",
368 | " legend: to show legend (default = True). Alternative value is False\n",
369 | "
\n",
370 | "\n",
371 | "
\n",
372 | "
\n",
392 | "\n",
393 | "
\n",
394 | "
6.2. Plot Latent Projections: Full & CV
\n",
395 | " \n",
396 | "
When cv.plot_projections() is run, an n x n grid of plots are displayed, where n is the number of latent variables (LV) to interrogate. These plots include score plots, distribution plots, and receiver operating characteristic (ROC) curves.
\n",
397 | "\n",
398 | "
There are C(n,2) score plots (i.e. a score plot for every combination of 2 LVs e.g. LV1 scores vs. LV2 scores). Each score plot includes the full scores (as circles) and CV scores (as crosses) coloured by group, as well as the 95% confidence interval ellipses for the full scores (as solid lines) and CV scores (as dashed lines). Additionally, the optimal line of separation (dashed grey line) and orthogonal line (solid grey line) are shown.
\n",
399 | "\n",
400 | "
There are n distribution plots (a distribution plot for each LV scores). The distribution of the full and CV scores for each corresponding group (i.e. 4 discrete distributions overlayed for 2 groups). Each distribution is calculated using kernel density estimation, a standard non-parametric method used for estimation of a probability density function based on a set of data points (Silverman 1986).
\n",
401 | "\n",
402 | "
There are C(n,2) ROC curves (a ROC curve for every combination of 2 LVs e.g. LV1 scores vs. LV2 scores). As the ROC curves are for every combination of 2 LVs, the discrimination is calculated based on optimal separation (i.e. the grey line from the corresponding score plot). For each ROC curve plot there is a ROC curve for the full model (green), and ROC curve for the cv model with 95% confidence intervals (yellow). Additionally, the equal distribution line (dashed black line) is shown.
\n",
403 | "\n",
404 | "
\n",
405 | " components: LVs to plot (default = \"all\" ; plot all components). Alternatively, list the components to plot e.g. [1,3,4]plot: Data to show (default = 'ci' ; plot only 95% confidence interval ellipses). Alternative values include 'meanci', 'full', 'cv', and 'all'label: Add labels to groups (default = None ; refers to groups as 0/1)\n",
408 | " legend: Show legends for plots (default = 'all'). Alternative values are 'scatter', 'dist', 'roc', and 'none'.
\n",
410 | "\n",
411 | "
\n",
412 | "
\n",
432 | "\n",
433 | "
\n",
434 | "
7. Build Model & Evaluate
\n",
435 | "\n",
436 | "
A PLS-DA model using cb.model.PLS_SIMPLS is created and initialised using the optimal hyperparameter values determined in step 4 (i.e. the number of latent variables). The implementation of PLS in the cb.model.PLS_SIMPLS class uses the SIMPLS algorithm (De Jong, 1993).
\n",
437 | " \n",
438 | "
Following this initialisation, the PLS-DA model is trained using the .train(X, Y) method where the X matrix is XTrainKnn and the Y vector is YTrain, returning the Y predicted value YPredTrain. This model is then tested using the .test(X, Y) method where the X matrix is XTestKnn and the Y vector is YTest, returning the Y predicted value YPredTest.
\n",
439 | "\n",
440 | "
The .evaluate() method can be used to evaluate the predictability of the model using the train and test set. There are three plots produced when this method is run including a violin plot, probability density function, and a ROC curve. The violin plots show the predicted score for the train and test (by group). The distribution plot shows the probability density function of the predicted scores for the train and test (by group). The ROC curve shows the ROC curve for the train (green) and test (yellow). The following parameter values in .evaluate() can be altered:\n",
441 | " \n",
442 | "
\n",
443 | " testset: Plot test dataset (default = None). Alternative, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
444 | " label: Add labels to groups (default = None ; refer to groups as 0/1)\n",
445 | " legend: Show legends for plots (default = 'all'). Alternative values are 'roc', 'dist', 'violin', and 'none'
\n",
448 | "
\n",
483 | "\n",
484 | "
\n",
485 | "
8. Permutation Test
\n",
486 | "\n",
487 | "
After a model has been trained, the .permutation_test() method can be used to assess the reliability of the trained model (after selecting the number of latent variables). For the permutation test, the metabolite data matrix is randomised (permuted or 'shuffled'), while the Y (i.e. outcome) is fixed, and subsequently trained and tested on this randomised data (Szymańska et al. 2012). This process is repeated (in this case, n=100) to construct a distribution to fairly access the model. For a dataset with features that have with no meaningful contribution, we would expect a similar $R^2$ and $Q^2$ to a randomised dataset, while for a dataset with features with meaningful contribution, we would expect a $R^2$ and $Q^2$ significantly higher than that of the randomised dataset. When .permutation_test() is run, 2 plots are displayed: (a) $R^2$ and $Q^2$ against \"correlation of permuted data against original data\", and (b) probability density functions for $R^2$ and $Q^2$, with the $R^2$ and $Q^2$ values found for the model trained on original data presented as ball-and-stick. The following parameter value of .permutation_test() can be altered: \n",
488 | "\n",
489 | "
\n",
490 | " metric: The metric used for the plots (default = 'r2q2'). Alternative metrics include 'auc', 'acc', 'f1score', 'prec', 'sens', and 'spec'. Multiple metrics can be plotted using a list e.g. ['r2q2', 'auc]\n",
491 | " nperm: The number of permutations. (default = 100)\n",
492 | " legend: To show legend (default = True). Alternative value is False\n",
493 | "
\n",
494 | "\n",
495 | "
\n",
496 | "
\n",
515 | "\n",
516 | "
\n",
517 | "
9. Bootstrap Resampling of the Model
\n",
518 | "\n",
519 | "
Bootstrap resampling is a resampling method based on random resampling with replacement, commonly used to provide an estimate of sampling distribution of a test statistic (Efron, 1982). In the context of this workflow, the PLS model from step 5 with its fixed hyperparameter values (i.e. number of LVs = 2) is retrained on the resampled with replacement data (in-bag) and evaluated on the unused data (out-of-bag) for 100 resamples. After the model is evaluated for each bootstrap, metrics including the predicted values (ypred), LV scores, LV loadings, and feature importance (VIP and coefficients) are stored and used to calculate 95% confidence intervals. To calculate the 95% confidence intervals, various methods can be used including the basic percentile method, corrected percentile method (a.k.a. bias-corrected method), and the commonly used bias-corrected and accelerated (BCA) method. In this example, the BCA method is used with the class cb.boostrap.BCA. Alternatively, use cb.boostrap.Per to use the percentile method, or cb.bootstrap.CPer for the corrected percentile method. To create and run the bootmodel for any method, the following parameter values need to be set:\n",
520 | " \n",
521 | "
\n",
522 | " model: A model with fixed hyperparameter values for boostrap resamplingbootnum: The number of bootstrap resamples (default = 100)
\n",
526 | "
\n",
552 | "\n",
553 | "
\n",
554 | "
10. Model Evaluation using Bootstrap Resampling
\n",
555 | "\n",
556 | "
After the bootmodel has been run, the .evaluate() method can be used to provide an estimate of the robustness and a measure of the generalised predictability of the model. There are three plots produced when this method is run including a violin plot, probability density function, and a ROC curve. The violin plots show the distribution of the median predicted score for the in-bag and out-of-bag (i.e. train and test) by group. The distribution plot shows the probability density function of the median predicted score for the in-bag and out-of-bag (i.e. train and test) by group. The ROC curve shows the ROC curve with the median (green) and 95% CI for the in-bag (light green band) and the median (yellow) and 95% CI for the out-of-bag (light yellow band). The method used to calculate the 95% CI for the in-bag (green) is the class selected in the previous cell. In this example, the bias-corrected and accelerated method is used as cb.bootstrap.BCA was used in the previous cell to create bootmodel. \n",
557 | " \n",
558 | "
\n",
559 | " label: Add labels to groups (default = None ; refer to groups as 0/1)\n",
560 | " legend: Show legends for plots (default = 'all'). Alternative values are 'roc', 'dist', 'violin', and 'none'trainset: Plot train dataset instead of median in-bag (default = None). Alternatively, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
562 | " testset: Plot test dataset instead of median out-of-bag (default = None). Alternatively, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
563 | "
\n",
564 | "
\n",
584 | "\n",
585 | "
\n",
586 | "
11. Model Visualisation
\n",
587 | "\n",
588 | "
\n",
589 | "
\n",
597 | " \n",
598 | "
\n",
599 | "
11.1 Plot Latent Projections: in-bag & out-of-bag
\n",
600 | " \n",
601 | "
After the bootmodel has been run, the .plot_projections() method can be used to visualise the latent variable (LV) scores. When this method is run, a n x n grid of plots are displayed, where n is the number of LVs. These plots include score plots, distribution plots, and receiver operating characteristic (ROC) curves.
\n",
602 | "\n",
603 | "
There are C(n,2) score plots (i.e. a score plot for every combination of 2 LVs e.g. LV1 scores vs. LV2 scores). Each score plot includes the in-bag scores (as circles) and out-of-bag scores (as crosses) coloured by group, as well as the 95% confidence interval ellipses for the in-bag scores (as solid lines) and out-of-bag scores (as dashed lines). Additionally, the optimal line of separation (dashed grey line) and orthogonal line (solid grey line) are shown.
\n",
604 | "\n",
605 | "
There are n distribution plots (a distribution plot for each LV scores). The distribution of the in-bag and out-of-bag scores for each corresponding group (i.e. 4 discrete distributions overlayed for 2 groups). Each distribution is calculated using kernel density estimation, a standard non-parametric method used for estimation of a probability density function based on a set of data points (Silverman 1986).
\n",
606 | "\n",
607 | "
There are C(n,2) ROC curves (a ROC curve for every combination of 2 LVs e.g. LV1 scores vs. LV2 scores). As the ROC curves are for every combination of 2 LVs, the discrimination is calculated based on optimal separation (i.e. the grey line from the corresponding score plot). For each ROC curve plot there is a ROC curve with the LV score for the initial model with the 95% confidence intervals using the in-bag LV scores (green), and a ROC curve for the out-of-bag LV scores with 95% confidence intervals. The method used to calculate the 95% CI for the in-bag (green) is the class used to create the bootmodel. In this example, the bias-corrected and accelerated method is used (cb.bootstrap.BCA). Additionally, the equal distribution line (dashed black line) is shown. \n",
608 | "\n",
609 | "
\n",
610 | "\n",
611 | "
\n",
612 | " plot: Data to show in plot (default = \"ci\" ; plot only 95% confidence interval ellipses). Alternative values include 'meanci', 'ib', 'oob', and 'all'label: Add labels to groups in scores plot (default = None ; refer to groups as 0/1).\n",
614 | " legend: Show legends for plots (default = 'all'). Alternative values are 'scatter', 'dist', 'roc', and 'none'
\n",
617 | "
\n",
658 | " \n",
659 | "
\n",
660 | "
11.2 Plot Weight Vectors
\n",
661 | "\n",
662 | "
After the bootmodel has been run, the .plot_weights() method can be used to visualise the latent variable (LV) weight vectors. When this method is run, n plots are displayed, where n is the number of LVs. The circles in each plot represent the LV weight vectors for the initial model. The 95% confidence intervals are calculated using bias-correct (BC) bootstrap method in step 6. Any metabolite weights with a confidence interval crossing the zero line are considered non-significant to the latent variable. This method requires values for the following parameters:
\n",
663 | " \n",
664 | "
\n",
665 | " PeakTable: Cleaned PeakTable from step 3PeakList: Peaks to include in plot (default = None; include all samples).\n",
667 | " plot: To plot the data or median as circles (default 'data'). Alternative values include 'median', and a list structured as [LV1, LV2, etc.]\n",
668 | " ylabel: Name of column in PeakTable to use as the ylabel (default = 'Label')\n",
669 | " sort: Whether to sort plots in absolute descending order (default = True)
\n",
671 | " \n",
672 | "
\n",
673 | "
\n",
694 | "\n",
695 | "
\n",
696 | "
12. Variable Contribution Plots
\n",
697 | "\n",
698 | "
After the bootmodel has been run, the .plot_featureimportance() method can be used to visualise the feature importance metrics. When this method is run, 2 plots are displayed; the Coefficient Plot and Variable Importance in Projection (VIP) plot. The circles represent the values in the initial model. The 95% confidence intervals are calculated using bias-correct (BC) bootstrap method in step 6.
\n",
699 | " \n",
700 | "
The coefficients (in the Coefficient plot) contain information about the overall contribution of each metabolite. The coefficient values can either a positive or negative number, and therefore, negatively or positively contribute to the model. Any metabolite coefficient value with a confidence interval crossing the zero line is considered non-significant to the model.
\n",
701 | " \n",
702 | "
The values in the VIP plot contain information about the overall contribution of each metabolite. Unlike the coefficient values, the VIP is absolute, with the higher values representing a higher significance to the model. Typically, metabolites with a VIP greater than 1 are considered \"important\" in the model.
\n",
703 | " \n",
704 | "
This method, bootmodel exports the feature importance metrics as a pandas DataFrame (table). This method also requires values for the following parameters:
\n",
705 | " \n",
706 | "
\n",
707 | " PeakTable: Cleaned PeakTable from step 3PeakList: Peaks to include in plot (default = None; include all samples).\n",
709 | " plot: To plot the data or median as circles (default 'data'). Alternative values include 'median', and a list structured as [Coef, VIP].\n",
710 | " ylabel: Name of column in PeakTable to use as the ylabel (default = 'Label')\n",
711 | " sort: Whether to sort plots in absolute descending order (default = True)
\n",
713 | " \n",
714 | "
\n",
715 | "
\n",
736 | "\n",
737 | "
\n",
738 | "
13. Export Results
\n",
739 | "\n",
740 | "
The feature importance table created in step 8.3 can be exported using the inbuilt .to_excel() function within a pandas DataFrame. This function requires an input with the name of the file to create, and it can include directories by using the ‘ / ’ symbol. In the cell below, the table feature_importance is exported as an excel file called 'PLSDA_ST001047.xlsx' in the 'results' folder.
\n",
741 | "\n",
742 | "
\n",
10 | " To begin: Click anywhere in this cell and press Run on the menu bar. This executes the current cell and then highlights the next cell. There are two types of cells. A text cell and a code cell. When you Run a text cell (we are in a text cell now), you advance to the next cell without executing any code. When you Run a code cell (identified by In[ ]: to the left of the cell) you advance to the next cell after executing all the Python code within that cell. Any visual results produced by the code (text/figures) are reported directly below that cell. Press Run again. Repeat this process until the end of the notebook. NOTE: All the cells in this notebook can be automatically executed sequentially by clicking Kernel→Restart and Run All. Should anything crash then restart the Jupyter Kernal by clicking Kernel→Restart, and start again from the top.\n",
11 | " \n",
12 | "
"
13 | ]
14 | },
15 | {
16 | "cell_type": "markdown",
17 | "metadata": {},
18 | "source": [
19 | "\n",
20 | "
\n",
21 | "\n",
22 | "
\n",
23 | "\n",
24 | "
Metabolomics Data Visualisation Workflow for ANN-SS
\n",
25 | "\n",
26 | "
\n",
27 | "
\n",
28 | "
\n",
29 | "
This Jupyter Notebook described a metabolomics data analysis and visualisation workflow for a 2 layer artificial neural network with layer 1 consisting of multiple neurons (n = 2 to 6) with a sigmoidal activation, and layer 2 (output layer) consisting of a single neuron with a sigmoidal activation function (ANN-SS) for a binary classification outcome.
\n",
30 | "\n",
31 | "
This computational workflow is described using a previously published NMR dataset by Ganna et al. (2014) and Ganna et al. (2015).The study compared the plasma metabolomic profile comparison across a large prospective epidemiological study of men (n=485) and women (n=483) at age 70 living in Uppsala, Sweden. For the purpose of this computational workflow, we compare only the males (Class=1) and females (Class=0) in a binary discriminant analysis. The deconvolved and annotated data from this study is deposited on Metabolights, and can be accessed directly via its Study ID: MTBLS90. The Excel file used in this workflow can be accessed via the following link: MTBLS90.xlsx.
\n",
32 | "\n",
33 | "
This computational workflow requires a dataset to be in, or converted to, a previously described standardised Excel file format (Mendez et al. 2019). This format uses the Tidy Data Framework (Wickham, 2014), where each row represents an observation (e.g. sample) and each column represents a variable (e.g. age or metabolite). Each excel file (per study) contains two sheets; a data sheet and a peak sheet. The data sheet contains the metabolite concentration together with the metadata associated for each observation (requiring the inclusion of the columns: Idx, SampleID, and Class). The peak sheet contains the additional metadata that pertains to the metabolites in the data sheet (requiring the inclusion of the columns: Idx, Name, and Label). The standardisation of this format allows for the efficient re-use of this computational workflow.
\n",
34 | "\n",
35 | "
\n",
36 | "The steps included in this data analysis and visualisation workflow are: \n",
37 | "
\n",
38 | "\n",
39 | "1.
Import Packages\n",
40 | "2.
Load Data & Peak Sheet\n",
41 | "3.
Extract X & Y\n",
42 | "4.
Split Data into Train & Test Set\n",
43 | "5.
Extract, Transform, & Scale X Data with Missing Values Imputed\n",
44 | "6.
Hyperparameter Optimisation\n",
45 | " 6.1.
Plot R² & Q²\n",
46 | " 6.2.
Plot Latent Projections: Full & CV\n",
47 | "7.
Build Model & Evaluate\n",
48 | "8.
Permutation Test\n",
49 | "9.
Bootstrap Resampling of the Model \n",
50 | "10.
Model Evaluation using Bootstrap Resampling \n",
51 | "11.
Model Visualisation \n",
52 | " 11.1.
Plot Latent Projections: in-bag & out-of-bag\n",
53 | " 11.2.
Plot Weight Vectors\n",
54 | "12.
Variable Contribution Plots \n",
55 | "13.
Export Results\n",
56 | "\n",
57 | "
\n",
67 | " \n",
68 | "
\n",
69 | "
1. Import Packages
\n",
70 | "\n",
71 | "
Packages provide additional tools that extend beyond the basic functionality of the Python programming. Prior to usage, packages need to be imported into the Jupyter environment. The following packages need to be imported for this computational workflow:
\n",
72 | "\n",
73 | "
\n",
74 | "numpy: A standard package primarily used for the manipulation of arrayspandas: A standard package primarily used for the manipulation of data tablescimcb: A library of helpful functions and tools provided by the authorssklearn: A standard package with tools for machine learning\n",
81 | "\n",
82 | "\n",
83 | "train_test_split: A method to split arrays into training and test subsets
\n",
84 | "\n",
85 | "
\n",
88 | "\n",
89 | "
\n",
90 | "\n",
91 | "
\n",
113 | " \n",
114 | "
\n",
115 | "
Optional: Set Random Seed for Splitting Data into Training & Test sets
\n",
116 | "\n",
117 | "
To reproduce the figures in the research article, set the random seed to 8. This seed is used in the train_test_split method to reproducibly split the source data into a training and test set.
\n",
118 | "\n",
119 | "
\n",
122 | "
\n",
123 | "\n",
124 | " \n",
125 | "
\n",
126 | "
Optional: Set Random Seed for Weight Initialisation
\n",
127 | "\n",
128 | "
To reproduce the figures in the research article, set the random seed to 8. When a neural network is first compilied, the weights are initialised. By default in Keras, the glorot normal initializer (a.k.a. Xavier normal initializer) is used where the weights are randomly drawn from a truncated normal distribution. The seed is used to set reproducible initial weights from this distribution.
\n",
129 | "\n",
130 | "
\n",
133 | "
\n",
134 | "\n",
135 | "
\n",
155 | "\n",
156 | "
\n",
157 | "
2. Load Data & Peak Sheet
\n",
158 | "\n",
159 | "
This CIMCB helper function load_dataXL() loads the Data and Peak sheet from an Excel file. In addition, this helper function checks that the data is in the standardised Excel file format described above. After the initial checks, load_dataXL() outputs two individual Pandas DataFrames (i.e. tables) called DataTable and PeakTable from the Excel file MTBLS90.xlsx. This helper function requires values for the following parameters:
\n",
160 | "
\n",
161 | " filename: The name of the excel file (.xlsx file)DataSheet: The name of the data sheet in the filePeakSheet: The name of the peak sheet in the file
\n",
165 | "
\n",
166 | "\n",
167 | "
\n",
187 | "\n",
188 | "
\n",
189 | "
3. Extract X & Y
\n",
190 | "\n",
191 | "
Prior to performing any statistical or machine learning modelling, it is best practice to assess the quality of the data and remove metabolites that lack reproducible measurements (Broadhurst et al. 2018). \n",
192 | "\n",
193 | "
\n",
194 | "
The following steps are needed to extract the X matrix of metabolite concentrations and associated Y vector of classification labels (“M”=1 and “F”=0):\n",
195 | " \n",
196 | "
\n",
197 | " \n",
198 | "- Create a subset of
DataTable called DataTable2, with samples only in the Class “M” or “F” \n",
199 | " \n",
200 | "\n",
201 | "- Create the variable
PeakList to hold the names (M1...Mn) of the metabolites to be used \n",
202 | "\n",
203 | "- Using this
PeakList, extract all corresponding columns (i.e. metabolite data) from DataTable2, and place it in matrix X \n",
204 | "\n",
205 | "- Set
Y to a list (or 1D array) of binary outcomes based on the Class column from DataTable2 (“M”=1 and “F”=0) \n",
206 | "\n",
207 | "
\n",
208 | "\n",
209 | "
\n",
210 | "
\n",
241 | "\n",
242 | "
\n",
243 | "
4. Split Data into Train & Test Set
\n",
244 | "\n",
245 | "\n",
246 | "
The train_test_split method is used to split the X and Y data into training (2/3rd) and test (1/3rd) sets using stratified random selection. Additionally, the Class data is split for use in figure legends. The seed is selected in the optional section above. For further information on this method, refer to the scikit learn documentation.\n",
247 | "\n",
248 | "
\n",
249 | "
\n",
272 | " \n",
273 | "
\n",
274 | "
5. Extract, Transform, & Scale X Data with Missing Values Imputed
\n",
275 | "\n",
276 | "
The X Data (XTrain and XTest) is log transformed, mean centred, and scaled to unit variance (with missing values imputed using K-Nearest Neighbour) prior to modelling following standard protocols for metabolomics (Broadhurst and Kell, 2006).
\n",
277 | "
\n",
278 | " \n",
279 | "- Log-transform the values in
XTrain \n",
280 | "\n",
281 | "- Using the helper function
cb.utils.scale(), scale the log-transformed data (XTrainLog) to the unit variance (a.k.a. auto scaling), while also returning mu & sigma. \n",
282 | "\n",
283 | "- Impute the missing values by using a k-nearest neighbour approach (with three neighbours) using the helper function
cb.utils.knnimpute() to give the final matrix, XTrainKnn \n",
284 | "\n",
285 | "- Log-transform the values in
XTest \n",
286 | "\n",
287 | "- Using the helper function
cb.utils.scale(), scale the log-transformed data (XTestLog) to the unit variance (a.k.a. auto scaling) using the mu & sigma from above.\n",
288 | " \n",
289 | " - Impute the missing values by using a k-nearest neighbour approach (with three neighbours) using the helper function
cb.utils.knnimpute() to give the final matrix, XTestKnn \n",
290 | " \n",
291 | "
\n",
292 | "\n",
293 | "
\n",
318 | " \n",
319 | "
\n",
320 | "
6. Hyperparameter Optimisation
\n",
321 | "\n",
322 | "
The CIMCB helper function cb.cross_val.kfold() is used to carry out k-fold cross-validation (k=5) on a set of ANN-SS models with varying number of neurons (1 to 6) and learning rate (0.01 to 0.05) to determine the optimal hyperparamater values. In k-fold cross-validation, the original dataset is randomly split into k sized folds and subsequently trained for k iterations, where the model is trained on 1 – k folds and tested on the k fold (Kohavi 1995). This helper function requires values for the following parameters:
\n",
323 | " \n",
324 | "
\n",
325 | " model: the class of model used by the function, cb.model.NN_SigmoidSigmoidX: The metabolite data matrix, XTrainKnnY: The binary outcome vector, YTrainparam_dict: a dictionary, param_dict, that describes all key:value pairs to search, with the key name corresponding to the hyperparameter in the model class and the value as the list of possible valuesfolds: the number of folds in the k-fold cross validationn_mc: the number of Monte Carlo repetitions of the k-fold CV
\n",
332 | "
\n",
333 | "
\n",
374 | " \n",
375 | "
\n",
376 | "
6.1. Plot R² & Q²
\n",
377 | "\n",
378 | "
When cv.plot(metric='r2q2', method='absolute') is run, 6 plots of $R^2$ and $Q^2$ statistics are displayed: (a) heatmap of $R^2$, (b) heatmap of $Q^2$, (c) heatmap of 1 - | ($R^2 - Q^2$) |, (d) | ($R^2 - Q^2$) | vs. $Q^2$, (e) $R^2$ and $Q^2$ against the learning rate, and (f) $R^2$ and $Q^2$ against the number of neurons. Alternatively, if method='ratio', | ($R^2 - Q^2$) / $R^2$ | is used instead of | ($R^2 - Q^2$) | . The optimal number of hyperparameters is selected based on the point of inflection in figure b, or if a clear inflection point is not present, where | ($R^2 - Q^2$) | = 0.2. Note, the $R^2$ is the mean coefficient of determination for the full dataset, and the $Q^2$ is the mean coefficient of determination for cross-validated prediction dataset over the 100 Monte Carlo repetitions. When cv.plot(metric='auc') is run, the predictability of the model is presented as area under the ROC curve (AUC), $AUC(full)$ & $AUC(cv)$, a non-parametric alternative to $R^2$ & $Q^2$. The following parameters of cv.plot() can be altered:
\n",
379 | "\n",
380 | "
\n",
381 | " metric: The metric used for the plots (default = 'r2q2'). Alternative metrics include 'auc', 'acc', 'f1score', 'prec', 'sens', and 'spec'\n",
382 | " method: The types of plots displayed (default = 'absolute'). Alternative value is 'ratio'\n",
383 | " ci: The confidence interval in figure e & f (default = 95)\n",
384 | "
\n",
385 | "\n",
386 | "
\n",
387 | "
\n",
405 | "\n",
406 | "
\n",
407 | "
6.2. Plot Latent Projections: Full & CV
\n",
408 | " \n",
409 | "
When cv.plot_projections() is run, an n x n grid of plots are displayed, where n is the number of neurons in the hidden layer to interrogate. These plots include score plots, distribution plots, and receiver operating characteristic (ROC) curves.
\n",
410 | "\n",
411 | "
There are C(n,2) score plots (i.e. a score plot for every combination of 2 neurons e.g. neuron 1 scores vs. neuron 2 scores). Each score plot includes the full scores (as circles) and CV scores (as crosses) coloured by group, as well as the 95% confidence interval ellipses for the full scores (as solid lines) and CV scores (as dashed lines). Additionally, the optimal line of separation (dashed grey line) and orthogonal line (solid grey line) are shown.
\n",
412 | "\n",
413 | "
There are n distribution plots (a distribution plot for each neuron scores). The distribution of the full and CV scores for each corresponding group (i.e. 4 discrete distributions overlayed for 2 groups). Each distribution is calculated using kernel density estimation, a standard non-parametric method used for estimation of a probability density function based on a set of data points (Silverman 1986).
\n",
414 | "\n",
415 | "
There are C(n,2) ROC curves (a ROC curve for every combination of 2 neurons e.g. neuron 1 scores vs. neuron 2 scores). As the ROC curves are for every combination of 2 neurons, the discrimination is calculated based on optimal separation (i.e. the grey line from the corresponding score plot). For each ROC curve plot there is a ROC curve for the full model (green), and ROC curve for the cv model with 95% confidence intervals (yellow). Additionally, the equal distribution line (dashed black line) is shown.
\n",
416 | "\n",
417 | "
\n",
418 | " **optional_arguments: optional arguments to specify model hyperparameters if they are changed in this search e.g. learning_rate=0.02 (except number of components). By default, the max value of each hyperparameter is used (unless specificied).components: Neurons to plot (default = \"all\" ; plot all components). Alternatively, list the components to plot e.g. [1,3,4]plot: Data to show (default = 'ci' ; plot only 95% confidence interval ellipses). Alternative values include 'meanci', 'full', 'cv', and 'all'label: Add labels to groups in scores plot (default = None ; refers to groups as 0/1)\n",
422 | " legend: Show legends for plots (default = 'all'). Alternative values are 'scatter', 'dist', 'roc', and 'none'
\n",
424 | "\n",
425 | "
\n",
426 | "
\n",
460 | "\n",
461 | "
\n",
462 | "
7. Build Model & Evaluate
\n",
463 | "\n",
464 | "
A ANN-SS model using cb.model.NN_SigmoidSigmoid is created and initialised using the optimal hyperparameter values determined in step 4. The implementation of ANN-SS in the cb.model.NN_SigmoidSigmoid class uses using Keras with a Theano backend.
\n",
465 | "\n",
466 | "
Following this initialisation, the ANN-SS model is trained using the .train(X, Y) method where the X matrix is XTrainKnn and the Y vector is YTrain, returning the Y predicted value YPredTrain. This model is then tested using the .test(X, Y) method where the X matrix is XTestKnn and the Y vector is YTest, returning the Y predicted value YPredTest.
\n",
467 | "\n",
468 | "
The .evaluate() method can be used to evaluate the predictability of the model using the train and test set. There are three plots produced when this method is run including a violin plot, probability density function, and a ROC curve. The violin plots show the predicted score for the train and test (by group). The distribution plot shows the probability density function of the predicted scores for the train and test (by group). The ROC curve shows the ROC curve for the train (green) and test (yellow). The following parameter values in .evaluate() can be altered:\n",
469 | " \n",
470 | "
\n",
471 | " testset: Plot test dataset (default = None). Alternative, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
472 | " label: Add labels to groups (default = None ; refer to groups as 0/1)\n",
473 | " legend: Show legends for plots (default = 'all'). Alternative values are 'roc', 'dist', 'violin', and 'none'
\n",
476 | "
\n",
519 | "\n",
520 | "
\n",
521 | "
8. Permutation Test
\n",
522 | "\n",
523 | "
After a model has been trained, the .permutation_test() method can be used to assess the reliability of the trained model (after selecting the number of latent variables). For the permutation test, the metabolite data matrix is randomised (permuted or 'shuffled'), while the Y (i.e. outcome) is fixed, and subsequently trained and tested on this randomised data (Szymańska et al. 2012). This process is repeated (in this case, n=100) to construct a distribution to fairly access the model. For a dataset with features that have with no meaningful contribution, we would expect a similar $R^2$ and $Q^2$ to a randomised dataset, while for a dataset with features with meaningful contribution, we would expect a $R^2$ and $Q^2$ significantly higher than that of the randomised dataset. When .permutation_test() is run, 2 plots are displayed: (a) $R^2$ and $Q^2$ against \"correlation of permuted data against original data\", and (b) probability density functions for $R^2$ and $Q^2$, with the $R^2$ and $Q^2$ values found for the model trained on original data presented as ball-and-stick. The following parameter value of .permutation_test() can be altered: \n",
524 | "\n",
525 | "
\n",
526 | " metric: The metric used for the plots (default = 'r2q2'). Alternative metrics include 'auc', 'acc', 'f1score', 'prec', 'sens', and 'spec'. Multiple metrics can be plotted using a list e.g. ['r2q2', 'auc]\n",
527 | " nperm: The number of permutations. (default = 100)\n",
528 | " legend: To show legend (default = True). Alternative value is False\n",
529 | "
\n",
530 | "\n",
531 | "
\n",
532 | "
\n",
553 | "\n",
554 | "
\n",
555 | "
9. Bootstrap Resampling of the Model
\n",
556 | "\n",
557 | "
Bootstrap resampling is a resampling method based on random resampling with replacement, commonly used to provide an estimate of sampling distribution of a test statistic (Efron, 1982). In the context of this workflow, the PLS model from step 5 with its fixed hyperparameter values (i.e. number of LVs = 2) is retrained on the resampled with replacement data (in-bag) and evaluated on the unused data (out-of-bag) for 100 resamples. After the model is evaluated for each bootstrap, metrics including the predicted values (ypred), LV scores, LV loadings, and feature importance (VIP and coefficients) are stored and used to calculate 95% confidence intervals. To calculate the 95% confidence intervals, various methods can be used including the basic percentile method, corrected percentile method (a.k.a. bias-corrected method), and the commonly used bias-corrected and accelerated (BCA) method. In this example, the BCA method is used with the class cb.boostrap.BCA. Alternatively, use cb.boostrap.Per to use the percentile method, or cb.bootstrap.CPer for the corrected percentile method. To create and run the bootmodel for any method, the following parameter values need to be set:\n",
558 | " \n",
559 | "
\n",
560 | " model: A model with fixed hyperparameter values for boostrap resamplingbootnum: The number of bootstrap resamples (default = 100)
\n",
564 | "
\n",
592 | "\n",
593 | "
\n",
594 | "
10. Model Evaluation using Bootstrap Resampling
\n",
595 | "\n",
596 | "
After the bootmodel has been run, the .evaluate() method can be used to provide an estimate of the robustness and a measure of the generalised predictability of the model. There are three plots produced when this method is run including a violin plot, probability density function, and a ROC curve. The violin plots show the distribution of the median predicted score for the in-bag and out-of-bag (i.e. train and test) by group. The distribution plot shows the probability density function of the median predicted score for the in-bag and out-of-bag (i.e. train and test) by group. The ROC curve shows the ROC curve with the median (green) and 95% CI for the in-bag (light green band) and the median (yellow) and 95% CI for the out-of-bag (light yellow band). The method used to calculate the 95% CI for the in-bag (green) is the class selected in the previous cell. In this example, the bias-corrected and accelerated method is used as cb.bootstrap.BCA was used in the previous cell to create bootmodel. \n",
597 | " \n",
598 | "
\n",
599 | " label: Add labels to groups (default = None ; refer to groups as 0/1)\n",
600 | " legend: Show legends for plots (default = 'all'). Alternative values are 'roc', 'dist', 'violin', and 'none'trainset: Plot train dataset instead of median in-bag (default = None). Alternatively, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
602 | " testset: Plot test dataset instead of median out-of-bag (default = None). Alternatively, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
603 | " \n",
604 | "
\n",
605 | "
\n",
627 | "\n",
628 | "
\n",
629 | "
11. Model Visualisation
\n",
630 | "\n",
631 | "
\n",
632 | "
\n",
640 | " \n",
641 | "
\n",
642 | "
11.1 Plot Latent Projections: in-bag & out-of-bag
\n",
643 | " \n",
644 | "
After the bootmodel has been run, the .plot_projections() method can be used to visualise the latent variable (LV) scores. When this method is run, an n x n grid of plots are displayed, where n is the number of neurons in the hidden layer. These plots include score plots, distribution plots, and receiver operating characteristic (ROC) curves.
\n",
645 | "\n",
646 | "
There are C(n,2) score plots (i.e. a score plot for every combination of 2 neurons e.g. neuron 1 scores vs. neuron 2 scores). Each score plot includes the in-bag scores (as circles) and out-of-bag scores (as crosses) coloured by group, as well as the 95% confidence interval ellipses for the in-bag scores (as solid lines) and out-of-bag scores (as dashed lines). Additionally, the optimal line of separation (dashed grey line) and orthogonal line (solid grey line) are shown.
\n",
647 | "\n",
648 | "
There are n distribution plots (a distribution plot for each neuron scores). The distribution of the in-bag and out-of-bag scores for each corresponding group (i.e. 4 discrete distributions overlayed for 2 groups). Each distribution is calculated using kernel density estimation, a standard non-parametric method used for estimation of a probability density function based on a set of data points (Silverman 1986).
\n",
649 | "\n",
650 | "
There are C(n,2) ROC curves (a ROC curve for every combination of 2 neurons e.g. neuron 1 scores vs. neuron 2 scores). As the ROC curves are for every combination of 2 neurons, the discrimination is calculated based on optimal separation (i.e. the grey line from the corresponding score plot). For each ROC curve plot there is a ROC curve with the LV score for the initial model with the 95% confidence intervals using the in-bag LV scores (green), and a ROC curve for the out-of-bag LV scores with 95% confidence intervals. The method used to calculate the 95% CI for the in-bag (green) is the class used to create the bootmodel. In this example, the bias corrected and accelerated method is used (cb.bootstrap.BCA). Additionally, the equal distribution line (dashed black line) is shown. \n",
651 | "\n",
652 | "
\n",
653 | " plot: Data to show in plot (default = \"ci\" ; plot only 95% confidence interval ellipses). Alternative values include 'meanci', 'ib', 'oob', and 'all'label: Add labels to groups in scores plot (default = None ; refer to groups as 0/1).\n",
655 | " legend: Show legends for plots (default = 'all'). Alternative values are 'scatter', 'dist', 'roc', and 'none'
\n",
658 | "
\n",
699 | " \n",
700 | "
\n",
701 | "
11.2 Plot Weight Vectors
\n",
702 | "\n",
703 | "
After the bootmodel has been run, the .plot_weights() method can be used to visualise the neuron weight vectors. When this method is run, n plots are displayed, where n is the number of neurons. The circles in each plot represent the neuron weight vectors for the model. The 95% confidence intervals are calculated using bias-correct (BC) bootstrap method in step 6. Any metabolite weights with a confidence interval crossing the zero line are considered non-significant to the neuron. This method requires values for the following parameters:
\n",
704 | " \n",
705 | "
\n",
706 | " PeakTable: Cleaned PeakTable from step 3PeakList: Peaks to include in plot (default = None; include all samples).\n",
708 | " plot: To plot the data or median as circles (default 'data'). Alternative values include 'median', and a list structured as [N1, N2, etc.]\n",
709 | " ylabel: Name of column in PeakTable to use as the ylabel (default = 'Label')\n",
710 | " sort: Whether to sort plots in absolute descending order (default = True)
\n",
712 | " \n",
713 | "
\n",
714 | "
\n",
737 | "\n",
738 | "
\n",
739 | "
12. Variable Contribution Plots
\n",
740 | "\n",
741 | "
After the bootmodel has been run, the .plot_featureimportance() method can be used to visualise the feature importance metrics. When this method is run, 2 plots are displayed; Connection Weight plot and Garson's Algorithm plot. These feature importance metrics are alternatives to the coefficient and variable importance in projection (VIP) in PLS.\n",
742 | " \n",
743 | "
The values in the Connection Weight plot contain information about the overall contribution of each metabolite (Olden et al. 2004). The values can either a positive or negative number, and therefore, negatively or positively contribute to the model. Any metabolite coefficient value with a confidence interval crossing the zero line is considered non-significant to the model.
\n",
744 | " \n",
745 | "
The values in the Garson's Algorithm plot contain information about the overall contribution of each metabolite (Garson 1991). These values are absolute, with the higher values representing a higher significance to the model. Unlike in a VIP plot, there is no standard cut-off used to determine whether metabolites are considered \"important\" in the model. One method (used below) is to use the average value across the metabolites as the cut-off.
\n",
746 | " \n",
747 | "
This method, bootmodel exports the feature importance metrics as a pandas DataFrame (table). This method also requires values for the following parameters:
\n",
748 | " \n",
749 | "
\n",
750 | " PeakTable: Cleaned PeakTable from step 3PeakList: Peaks to include in plot (default = None; include all samples).\n",
752 | " plot: To plot the data or median as circles (default 'data'). Alternative values include 'median', and a list structured as [CW, GA].\n",
753 | " ylabel: Name of column in PeakTable to use as the ylabel (default = 'Label')\n",
754 | " sort: Whether to sort plots in absolute descending order (default = True)
\n",
756 | " \n",
757 | "
\n",
758 | "
\n",
781 | "\n",
782 | "
\n",
783 | "
13. Export Results
\n",
784 | "\n",
785 | "
The feature importance table created in step 8.3 can be exported using the inbuilt .to_excel() function within a pandas DataFrame. This function requires an input with the name of the file to create, and it can include directories by using the ‘ / ’ symbol. In the cell below, the table feature_importance is exported as an excel file called 'ANNSigSig_ST001047.xlsx' in the 'results' folder.
\n",
786 | "\n",
787 | "
\n",
10 | " To begin: Click anywhere in this cell and press Run on the menu bar. This executes the current cell and then highlights the next cell. There are two types of cells. A text cell and a code cell. When you Run a text cell (we are in a text cell now), you advance to the next cell without executing any code. When you Run a code cell (identified by In[ ]: to the left of the cell) you advance to the next cell after executing all the Python code within that cell. Any visual results produced by the code (text/figures) are reported directly below that cell. Press Run again. Repeat this process until the end of the notebook. NOTE: All the cells in this notebook can be automatically executed sequentially by clicking Kernel→Restart and Run All. Should anything crash then restart the Jupyter Kernal by clicking Kernel→Restart, and start again from the top.\n",
11 | " \n",
12 | "
"
13 | ]
14 | },
15 | {
16 | "cell_type": "markdown",
17 | "metadata": {},
18 | "source": [
19 | "\n",
20 | "
\n",
21 | "\n",
22 | "
\n",
23 | "\n",
24 | "
Metabolomics Data Visualisation Workflow for ANN-SS
\n",
25 | "\n",
26 | "
\n",
27 | "
\n",
28 | "
\n",
29 | "
This Jupyter Notebook described a metabolomics data analysis and visualisation workflow for a 2 layer artificial neural network with layer 1 consisting of multiple neurons (n = 2 to 6) with a sigmoidal activation, and layer 2 (output layer) consisting of a single neuron with a sigmoidal activation function (ANN-SS) for a binary classification outcome.
\n",
30 | "\n",
31 | "
This computational workflow is described using a previously published NMR dataset by Chan et al. (2016). The study compared the urine metabolomic profile comparison across patients characterised as Gastric Cancer (GC; n=43), Benign Gastric Disease (BN; n=40), and Healthy Control (HE; n=40) using 149 named metabolites. For the purpose of this computational workflow, we compare only the GC vs HE samples in a binary discriminant analysis. The deconvolved and annotated data from this study are deposited on Metabolomics Workbench (Study ID: ST001047), and can be accessed directly via its Project DOI: 10.21228/M8B10B. The Excel file used in this workflow can be accessed via the following link: ST001047.xlsx.
\n",
32 | "\n",
33 | "
This computational workflow requires a dataset to be in, or converted to, a previously described standardised Excel file format (Mendez et al. 2019). This format uses the Tidy Data Framework (Wickham, 2014), where each row represents an observation (e.g. sample) and each column represents a variable (e.g. age or metabolite). Each excel file (per study) contains two sheets; a data sheet and a peak sheet. The data sheet contains the metabolite concentration together with the metadata associated for each observation (requiring the inclusion of the columns: Idx, SampleID, and Class). The peak sheet contains the additional metadata that pertains to the metabolites in the data sheet (requiring the inclusion of the columns: Idx, Name, and Label). The standardisation of this format allows for the efficient re-use of this computational workflow.
\n",
34 | "\n",
35 | "
\n",
36 | "The steps included in this data analysis and visualisation workflow are: \n",
37 | "
\n",
38 | "\n",
39 | "1.
Import Packages\n",
40 | "2.
Load Data & Peak Sheet\n",
41 | "3.
Extract X & Y\n",
42 | "4.
Split Data into Train & Test Set\n",
43 | "5.
Extract, Transform, & Scale X Data with Missing Values Imputed\n",
44 | "6.
Hyperparameter Optimisation\n",
45 | " 6.1.
Plot R² & Q²\n",
46 | " 6.2.
Plot Latent Projections: Full & CV\n",
47 | "7.
Build Model & Evaluate\n",
48 | "8.
Permutation Test\n",
49 | "9.
Bootstrap Resampling of the Model \n",
50 | "10.
Model Evaluation using Bootstrap Resampling \n",
51 | "11.
Model Visualisation \n",
52 | " 11.1.
Plot Latent Projections: in-bag & out-of-bag\n",
53 | " 11.2.
Plot Weight Vectors\n",
54 | "12.
Variable Contribution Plots \n",
55 | "13.
Export Results\n",
56 | "\n",
57 | "
\n",
67 | " \n",
68 | "
\n",
69 | "
1. Import Packages
\n",
70 | "\n",
71 | "
Packages provide additional tools that extend beyond the basic functionality of the Python programming. Prior to usage, packages need to be imported into the Jupyter environment. The following packages need to be imported for this computational workflow:
\n",
72 | "\n",
73 | "
\n",
74 | "numpy: A standard package primarily used for the manipulation of arrayspandas: A standard package primarily used for the manipulation of data tablescimcb: A library of helpful functions and tools provided by the authorssklearn: A standard package with tools for machine learning\n",
81 | "\n",
82 | "\n",
83 | "train_test_split: A method to split arrays into training and test subsets
\n",
84 | "\n",
85 | "
\n",
88 | "\n",
89 | "
\n",
90 | "\n",
91 | "
\n",
113 | " \n",
114 | "
\n",
115 | "
Optional: Set Random Seed for Splitting Data into Training & Test sets
\n",
116 | "\n",
117 | "
To reproduce the figures in the research article, set the random seed to 100. This seed is used in the train_test_split method to reproducibly split the source data into a training and test set.
\n",
118 | "\n",
119 | "
\n",
122 | "
\n",
123 | "\n",
124 | " \n",
125 | "
\n",
126 | "
Optional: Set Random Seed for Weight Initialisation
\n",
127 | "\n",
128 | "
To reproduce the figures in the research article, set the random seed to 4. When a neural network is first compilied, the weights are initialised. By default in Keras, the glorot normal initializer (a.k.a. Xavier normal initializer) is used where the weights are randomly drawn from a truncated normal distribution. The seed is used to set reproducible initial weights from this distribution.
\n",
129 | "\n",
130 | "
\n",
133 | "
\n",
134 | "\n",
135 | "
\n",
155 | "\n",
156 | "
\n",
157 | "
2. Load Data & Peak Sheet
\n",
158 | "\n",
159 | "
This CIMCB helper function load_dataXL() loads the Data and Peak sheet from an Excel file. In addition, this helper function checks that the data is in the standardised Excel file format described above. After the initial checks, load_dataXL() outputs two individual Pandas DataFrames (i.e. tables) called DataTable and PeakTable from the Excel file ST001047.xlsx. This helper function requires values for the following parameters:
\n",
160 | "
\n",
161 | " filename: The name of the excel file (.xlsx file)DataSheet: The name of the data sheet in the filePeakSheet: The name of the peak sheet in the file
\n",
165 | "
\n",
166 | "\n",
167 | "
\n",
187 | "\n",
188 | "
\n",
189 | "
3. Extract X & Y
\n",
190 | "\n",
191 | "
Prior to performing any statistical or machine learning modelling, it is best practice to assess the quality of the data and remove metabolites that lack reproducible measurements (Broadhurst et al. 2018). In this dataset ST001047.xlsx, we can find that the QC-RSD and percentage of missing value has been previously calculated (refer to the peak sheet). In this Jupyter Notebook, we remove all metabolites that do not meet the following criteria:
\n",
192 | "\n",
193 | "
\n",
194 | "- QC-RSD less than 20%
\n",
195 | "\n",
196 | "- Fewer than 10% of values are missing
\n",
197 | "
\n",
198 | "\n",
199 | "
\n",
200 | "
The following steps are needed to extract the X matrix of metabolite concentrations and associated Y vector of classification labels (“GC”=1 and “HE”=0):\n",
201 | " \n",
202 | "
\n",
203 | " \n",
204 | "- Create a subset of
DataTable called DataTable2, with samples only in the Class “GC” or “HE” \n",
205 | " \n",
206 | "\n",
207 | "- Create the variable
PeakList to hold the names (M1...Mn) of the metabolites to be used \n",
208 | "\n",
209 | "- Using this
PeakList, extract all corresponding columns (i.e. metabolite data) from DataTable2, and place it in matrix X \n",
210 | "\n",
211 | "- Set
Y to a list (or 1D array) of binary outcomes based on the Class column from DataTable2 (“GC”=1 and “HE”=0) \n",
212 | "\n",
213 | "
\n",
214 | "\n",
215 | "
\n",
216 | "
\n",
251 | "\n",
252 | "
\n",
253 | "
4. Split Data into Train & Test Set
\n",
254 | "\n",
255 | "\n",
256 | "
The train_test_split method is used to split the X and Y data into training (2/3rd) and test (1/3rd) sets using stratified random selection. Additionally, the Class data is split for use in figure legends. The seed is selected in the optional section above. For further information on this method, refer to the scikit learn documentation.\n",
257 | "\n",
258 | "
\n",
259 | "
\n",
282 | " \n",
283 | "
\n",
284 | "
5. Extract, Transform, & Scale X Data with Missing Values Imputed
\n",
285 | "\n",
286 | "
The X Data (XTrain and XTest) is log10 transformed, mean centred, and scaled to unit variance (with missing values imputed using K-Nearest Neighbour) prior to modelling following standard protocols for metabolomics (Broadhurst and Kell, 2006).
\n",
287 | "
\n",
288 | " \n",
289 | "- Log-transform the values in
XTrain \n",
290 | "\n",
291 | "- Using the helper function
cb.utils.scale(), scale the log-transformed data (XTrainLog) to the unit variance (a.k.a. auto scaling), while also returning mu & sigma. \n",
292 | "\n",
293 | "- Impute the missing values by using a k-nearest neighbour approach (with three neighbours) using the helper function
cb.utils.knnimpute() to give the final matrix, XTrainKnn \n",
294 | "\n",
295 | "- Log-transform the values in
XTest \n",
296 | "\n",
297 | "- Using the helper function
cb.utils.scale(), scale the log-transformed data (XTestLog) to the unit variance (a.k.a. auto scaling) using the mu & sigma from above.\n",
298 | " \n",
299 | " - Impute the missing values by using a k-nearest neighbour approach (with three neighbours) using the helper function
cb.utils.knnimpute() to give the final matrix, XTestKnn \n",
300 | " \n",
301 | "
\n",
302 | "\n",
303 | "
\n",
328 | " \n",
329 | "
\n",
330 | "
6. Hyperparameter Optimisation
\n",
331 | "\n",
332 | "
The CIMCB helper function cb.cross_val.kfold() is used to carry out k-fold cross-validation (k=5) on a set of ANN-SS models with varying number of neurons (1 to 6) and learning rate (0.01 to 0.05) to determine the optimal hyperparamater values. In k-fold cross-validation, the original dataset is randomly split into k sized folds and subsequently trained for k iterations, where the model is trained on 1 – k folds and tested on the k fold (Kohavi 1995). This helper function requires values for the following parameters:
\n",
333 | " \n",
334 | "
\n",
335 | " model: the class of model used by the function, cb.model.NN_SigmoidSigmoidX: The metabolite data matrix, XTrainKnnY: The binary outcome vector, YTrainparam_dict: a dictionary, param_dict, that describes all key:value pairs to search, with the key name corresponding to the hyperparameter in the model class and the value as the list of possible valuesfolds: the number of folds in the k-fold cross validationn_mc: the number of Monte Carlo repetitions of the k-fold CV
\n",
342 | "
\n",
343 | "
\n",
383 | " \n",
384 | "
\n",
385 | "
6.1. Plot R² & Q²
\n",
386 | "\n",
387 | "
When cv.plot(metric='r2q2', method='absolute') is run, 6 plots of $R^2$ and $Q^2$ statistics are displayed: (a) heatmap of $R^2$, (b) heatmap of $Q^2$, (c) heatmap of 1 - | ($R^2 - Q^2$) |, (d) | ($R^2 - Q^2$) | vs. $Q^2$, (e) $R^2$ and $Q^2$ against the learning rate, and (f) $R^2$ and $Q^2$ against the number of neurons. Alternatively, if method='ratio', | ($R^2 - Q^2$) / $R^2$ | is used instead of | ($R^2 - Q^2$) | . The optimal number of hyperparameters is selected based on the point of inflection in figure b, or if a clear inflection point is not present, where | ($R^2 - Q^2$) | = 0.2. Note, the $R^2$ is the mean coefficient of determination for the full dataset, and the $Q^2$ is the mean coefficient of determination for cross-validated prediction dataset over the 100 Monte Carlo repetitions. When cv.plot(metric='auc') is run, the predictability of the model is presented as area under the ROC curve (AUC), $AUC(full)$ & $AUC(cv)$, a non-parametric alternative to $R^2$ & $Q^2$. The following parameters of cv.plot() can be altered:
\n",
388 | "\n",
389 | "
\n",
390 | " metric: The metric used for the plots (default = 'r2q2'). Alternative metrics include 'auc', 'acc', 'f1score', 'prec', 'sens', and 'spec'\n",
391 | " method: The types of plots displayed (default = 'absolute'). Alternative value is 'ratio'\n",
392 | " ci: The confidence interval in figure e & f (default = 95)\n",
393 | "
\n",
394 | "\n",
395 | "
\n",
396 | "
\n",
414 | "\n",
415 | "
\n",
416 | "
6.2. Plot Latent Projections: Full & CV
\n",
417 | " \n",
418 | "
When cv.plot_projections() is run, an n x n grid of plots are displayed, where n is the number of neurons in the hidden layer to interrogate. These plots include score plots, distribution plots, and receiver operating characteristic (ROC) curves.
\n",
419 | "\n",
420 | "
There are C(n,2) score plots (i.e. a score plot for every combination of 2 neurons e.g. neuron 1 scores vs. neuron 2 scores). Each score plot includes the full scores (as circles) and CV scores (as crosses) coloured by group, as well as the 95% confidence interval ellipses for the full scores (as solid lines) and CV scores (as dashed lines). Additionally, the optimal line of separation (dashed grey line) and orthogonal line (solid grey line) are shown.
\n",
421 | "\n",
422 | "
There are n distribution plots (a distribution plot for each neuron scores). The distribution of the full and CV scores for each corresponding group (i.e. 4 discrete distributions overlayed for 2 groups). Each distribution is calculated using kernel density estimation, a standard non-parametric method used for estimation of a probability density function based on a set of data points (Silverman 1986).
\n",
423 | "\n",
424 | "
There are C(n,2) ROC curves (a ROC curve for every combination of 2 neurons e.g. neuron 1 scores vs. neuron 2 scores). As the ROC curves are for every combination of 2 neurons, the discrimination is calculated based on optimal separation (i.e. the grey line from the corresponding score plot). For each ROC curve plot there is a ROC curve for the full model (green), and ROC curve for the cv model with 95% confidence intervals (yellow). Additionally, the equal distribution line (dashed black line) is shown.
\n",
425 | "\n",
426 | "
\n",
427 | " **optional_arguments: optional arguments to specify model hyperparameters if they are changed in this search e.g. learning_rate=0.02 (except number of components). By default, the max value of each hyperparameter is used (unless specificied).components: Neurons to plot (default = \"all\" ; plot all components). Alternatively, list the components to plot e.g. [1,3,4]plot: Data to show (default = 'ci' ; plot only 95% confidence interval ellipses). Alternative values include 'meanci', 'full', 'cv', and 'all'label: Add labels to groups in scores plot (default = None ; refers to groups as 0/1)\n",
431 | " legend: Show legends for plots (default = 'all'). Alternative values are 'scatter', 'dist', 'roc', and 'none'
\n",
433 | "\n",
434 | "
\n",
435 | "
\n",
456 | "\n",
457 | "
\n",
458 | "
7. Build Model & Evaluate
\n",
459 | "\n",
460 | "
A ANN-SS model using cb.model.NN_SigmoidSigmoid is created and initialised using the optimal hyperparameter values determined in step 4. The implementation of ANN-SS in the cb.model.NN_SigmoidSigmoid class uses using Keras with a Theano backend.
\n",
461 | "\n",
462 | "
Following this initialisation, the ANN-SS model is trained using the .train(X, Y) method where the X matrix is XTrainKnn and the Y vector is YTrain, returning the Y predicted value YPredTrain. This model is then tested using the .test(X, Y) method where the X matrix is XTestKnn and the Y vector is YTest, returning the Y predicted value YPredTest.
\n",
463 | "\n",
464 | "
The .evaluate() method can be used to evaluate the predictability of the model using the train and test set. There are three plots produced when this method is run including a violin plot, probability density function, and a ROC curve. The violin plots show the predicted score for the train and test (by group). The distribution plot shows the probability density function of the predicted scores for the train and test (by group). The ROC curve shows the ROC curve for the train (green) and test (yellow). The following parameter values in .evaluate() can be altered:\n",
465 | " \n",
466 | "
\n",
467 | " testset: Plot test dataset (default = None). Alternative, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
468 | " label: Add labels to groups (default = None ; refer to groups as 0/1)\n",
469 | " legend: Show legends for plots (default = 'all'). Alternative values are 'roc', 'dist', 'violin', and 'none'
\n",
472 | "
\n",
515 | "\n",
516 | "
\n",
517 | "
8. Permutation Test
\n",
518 | "\n",
519 | "
After a model has been trained, the .permutation_test() method can be used to assess the reliability of the trained model (after selecting the number of latent variables). For the permutation test, the metabolite data matrix is randomised (permuted or 'shuffled'), while the Y (i.e. outcome) is fixed, and subsequently trained and tested on this randomised data (Szymańska et al. 2012). This process is repeated (in this case, n=100) to construct a distribution to fairly access the model. For a dataset with features that have with no meaningful contribution, we would expect a similar $R^2$ and $Q^2$ to a randomised dataset, while for a dataset with features with meaningful contribution, we would expect a $R^2$ and $Q^2$ significantly higher than that of the randomised dataset. When .permutation_test() is run, 2 plots are displayed: (a) $R^2$ and $Q^2$ against \"correlation of permuted data against original data\", and (b) probability density functions for $R^2$ and $Q^2$, with the $R^2$ and $Q^2$ values found for the model trained on original data presented as ball-and-stick. The following parameter value of .permutation_test() can be altered: \n",
520 | "\n",
521 | "
\n",
522 | " metric: The metric used for the plots (default = 'r2q2'). Alternative metrics include 'auc', 'acc', 'f1score', 'prec', 'sens', and 'spec'. Multiple metrics can be plotted using a list e.g. ['r2q2', 'auc]\n",
523 | " nperm: The number of permutations. (default = 100)\n",
524 | " legend: To show legend (default = True). Alternative value is False\n",
525 | "
\n",
526 | "\n",
527 | "
\n",
528 | "
\n",
549 | "\n",
550 | "
\n",
551 | "
9. Bootstrap Resampling of the Model
\n",
552 | "\n",
553 | "
Bootstrap resampling is a resampling method based on random resampling with replacement, commonly used to provide an estimate of sampling distribution of a test statistic (Efron, 1982). In the context of this workflow, the PLS model from step 5 with its fixed hyperparameter values (i.e. number of LVs = 2) is retrained on the resampled with replacement data (in-bag) and evaluated on the unused data (out-of-bag) for 100 resamples. After the model is evaluated for each bootstrap, metrics including the predicted values (ypred), LV scores, LV loadings, and feature importance (VIP and coefficients) are stored and used to calculate 95% confidence intervals. To calculate the 95% confidence intervals, various methods can be used including the basic percentile method, corrected percentile method (a.k.a. bias-corrected method), and the commonly used bias-corrected and accelerated (BCA) method. In this example, the BCA method is used with the class cb.boostrap.BCA. Alternatively, use cb.boostrap.Per to use the percentile method, or cb.bootstrap.CPer for the corrected percentile method. To create and run the bootmodel for any method, the following parameter values need to be set:\n",
554 | " \n",
555 | "
\n",
556 | " model: A model with fixed hyperparameter values for boostrap resamplingbootnum: The number of bootstrap resamples (default = 100)
\n",
560 | "
\n",
588 | "\n",
589 | "
\n",
590 | "
10. Model Evaluation using Bootstrap Resampling
\n",
591 | "\n",
592 | "
After the bootmodel has been run, the .evaluate() method can be used to provide an estimate of the robustness and a measure of the generalised predictability of the model. There are three plots produced when this method is run including a violin plot, probability density function, and a ROC curve. The violin plots show the distribution of the median predicted score for the in-bag and out-of-bag (i.e. train and test) by group. The distribution plot shows the probability density function of the median predicted score for the in-bag and out-of-bag (i.e. train and test) by group. The ROC curve shows the ROC curve with the median (green) and 95% CI for the in-bag (light green band) and the median (yellow) and 95% CI for the out-of-bag (light yellow band). The method used to calculate the 95% CI for the in-bag (green) is the class selected in the previous cell. In this example, the bias-corrected and accelerated method is used as cb.bootstrap.BCA was used in the previous cell to create bootmodel. \n",
593 | " \n",
594 | "
\n",
595 | " label: Add labels to groups (default = None ; refer to groups as 0/1)\n",
596 | " legend: Show legends for plots (default = 'all'). Alternative values are 'roc', 'dist', 'violin', and 'none'trainset: Plot train dataset instead of median in-bag (default = None). Alternatively, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
598 | " testset: Plot test dataset instead of median out-of-bag (default = None). Alternatively, add YTrue and YPredicted as a list e.g. [YTrue, YPredicted].\n",
599 | "
\n",
600 | "
\n",
622 | "\n",
623 | "
\n",
624 | "
11. Model Visualisation
\n",
625 | "\n",
626 | "
\n",
627 | "
\n",
635 | " \n",
636 | "
\n",
637 | "
11.1 Plot Latent Projections: in-bag & out-of-bag
\n",
638 | " \n",
639 | "
After the bootmodel has been run, the .plot_projections() method can be used to visualise the latent variable (LV) scores. When this method is run, an n x n grid of plots are displayed, where n is the number of neurons in the hidden layer. These plots include score plots, distribution plots, and receiver operating characteristic (ROC) curves.
\n",
640 | "\n",
641 | "
There are C(n,2) score plots (i.e. a score plot for every combination of 2 neurons e.g. neuron 1 scores vs. neuron 2 scores). Each score plot includes the in-bag scores (as circles) and out-of-bag scores (as crosses) coloured by group, as well as the 95% confidence interval ellipses for the in-bag scores (as solid lines) and out-of-bag scores (as dashed lines). Additionally, the optimal line of separation (dashed grey line) and orthogonal line (solid grey line) are shown.
\n",
642 | "\n",
643 | "
There are n distribution plots (a distribution plot for each neuron scores). The distribution of the in-bag and out-of-bag scores for each corresponding group (i.e. 4 discrete distributions overlayed for 2 groups). Each distribution is calculated using kernel density estimation, a standard non-parametric method used for estimation of a probability density function based on a set of data points (Silverman 1986).
\n",
644 | "\n",
645 | "
There are C(n,2) ROC curves (a ROC curve for every combination of 2 neurons e.g. neuron 1 scores vs. neuron 2 scores). As the ROC curves are for every combination of 2 neurons, the discrimination is calculated based on optimal separation (i.e. the grey line from the corresponding score plot). For each ROC curve plot there is a ROC curve with the LV score for the initial model with the 95% confidence intervals using the in-bag LV scores (green), and a ROC curve for the out-of-bag LV scores with 95% confidence intervals. The method used to calculate the 95% CI for the in-bag (green) is the class used to create the bootmodel. In this example, the bias corrected and accelerated method is used (cb.bootstrap.BCA). Additionally, the equal distribution line (dashed black line) is shown. \n",
646 | "\n",
647 | "
\n",
648 | " plot: Data to show in plot (default = \"ci\" ; plot only 95% confidence interval ellipses). Alternative values include 'meanci', 'ib', 'oob', and 'all'label: Add labels to groups in scores plot (default = None ; refer to groups as 0/1).\n",
650 | " legend: Show legends for plots (default = 'all'). Alternative values are 'scatter', 'dist', 'roc', and 'none'
\n",
653 | "
\n",
694 | " \n",
695 | "
\n",
696 | "
11.2 Plot Weight Vectors
\n",
697 | "\n",
698 | "
After the bootmodel has been run, the .plot_weights() method can be used to visualise the neuron weight vectors. When this method is run, n plots are displayed, where n is the number of neurons. The circles in each plot represent the neuron weight vectors for the model. The 95% confidence intervals are calculated using bias-correct (BC) bootstrap method in step 6. Any metabolite weights with a confidence interval crossing the zero line are considered non-significant to the neuron. This method requires values for the following parameters:
\n",
699 | " \n",
700 | "
\n",
701 | " PeakTable: Cleaned PeakTable from step 3PeakList: Peaks to include in plot (default = None; include all samples).\n",
703 | " plot: To plot the data or median as circles (default 'data'). Alternative values include 'median', and a list structured as [N1, N2, etc.]\n",
704 | " ylabel: Name of column in PeakTable to use as the ylabel (default = 'Label')\n",
705 | " sort: Whether to sort plots in absolute descending order (default = True)
\n",
707 | " \n",
708 | "
\n",
709 | "
\n",
732 | "\n",
733 | "
\n",
734 | "
12. Variable Contribution Plots
\n",
735 | "\n",
736 | "
After the bootmodel has been run, the .plot_featureimportance() method can be used to visualise the feature importance metrics. When this method is run, 2 plots are displayed; Connection Weight plot and Garson's Algorithm plot. These feature importance metrics are alternatives to the coefficient and variable importance in projection (VIP) in PLS.\n",
737 | " \n",
738 | "
The values in the Connection Weight plot contain information about the overall contribution of each metabolite (Olden et al. 2004). The values can either a positive or negative number, and therefore, negatively or positively contribute to the model. Any metabolite coefficient value with a confidence interval crossing the zero line is considered non-significant to the model.
\n",
739 | " \n",
740 | "
The values in the Garson's Algorithm plot contain information about the overall contribution of each metabolite (Garson 1991). These values are absolute, with the higher values representing a higher significance to the model. Unlike in a VIP plot, there is no standard cut-off used to determine whether metabolites are considered \"important\" in the model. One method (used below) is to use the average value across the metabolites as the cut-off.
\n",
741 | " \n",
742 | "
This method, bootmodel exports the feature importance metrics as a pandas DataFrame (table). This method also requires values for the following parameters:
\n",
743 | " \n",
744 | "
\n",
745 | " PeakTable: Cleaned PeakTable from step 3PeakList: Peaks to include in plot (default = None; include all samples).\n",
747 | " plot: To plot the data or median as circles (default 'data'). Alternative values include 'median', and a list structured as [CW, GA].\n",
748 | " ylabel: Name of column in PeakTable to use as the ylabel (default = 'Label')\n",
749 | " sort: Whether to sort plots in absolute descending order (default = True)
\n",
751 | " \n",
752 | "
\n",
753 | "
\n",
776 | "\n",
777 | "
\n",
778 | "
13. Export Results
\n",
779 | "\n",
780 | "
The feature importance table created in step 8.3 can be exported using the inbuilt .to_excel() function within a pandas DataFrame. This function requires an input with the name of the file to create, and it can include directories by using the ‘ / ’ symbol. In the cell below, the table feature_importance is exported as an excel file called 'ANNSigSig_ST001047.xlsx' in the 'results' folder.
\n",
781 | "\n",
782 | "
2 |
3 | # README.md - `SI_Mendez_etal_2020`
4 |
5 |