├── README.md
└── editorial-data
├── editorial-stats.png
├── plot-publication-stats.ipynb
└── publication-stats.xlsx
/README.md:
--------------------------------------------------------------------------------
1 | # ML Papers Checklist
2 |
3 | At npj Computational Materials we are striving to set the highest standards for all research published in the journal. To help with this goal we have a checklist of factors that can help you to review the paper. We understand that there are certain cases where some checklist items may be impossible or inappropriate, but where reviewers request a checklist item and authors do not comply or provide what we consider a reasonable excuse, this may be taken into account in the final editorial decision.
4 |
5 | We believe that the points regarding dataset and model descriptions should be fulfilled by all papers using ML. The points relating to reproducibility are strong suggestions meaning that we believe that they should be followed, unless there are valid reasons for not doing so (e.g. sensitive data). Much of this checklist has been adapted from
6 |
7 | * [Best practices in machine learning for chemistry](https://static-content.springer.com/esm/art%3A10.1038%2Fs41557-021-00716-z/MediaObjects/41557_2021_716_MOESM1_ESM.pdf)
8 | * [The JARVIS-Leaderboard reproducibility guide](https://pages.nist.gov/jarvis_leaderboard/guide/guide_short/)
9 | * [The machine learning reproducibility checklist](https://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf)
10 | * [The REFORMS checklist](https://reforms.cs.princeton.edu/appendices.pdf)
11 |
12 | ## Dataset descriptions
13 | ### Required:
14 | * Source(s) of data, separately for the training and evaluation datasets (if applicable), along with the time when the dataset(s) are collected
15 | * Number of samples in the dataset
16 | * The details of train / validation / test splits
17 | * An explanation of any data that were excluded, and all pre-processing steps
18 | * An evaluation of the amount of removed source data is reported
19 | * Instances of combining data from multiple sources clearly identified, and potential issues mitigated
20 | * Methods for representing data as features or descriptors clearly articulated, ideally with software implementations
21 | ## Model descriptions
22 | ### Required:
23 | * Detailed descriptions of all models trained, including:
24 | * All features used in the model (including any feature selection)
25 | * Types of models implemented (e.g., Random Forests, Neural Networks)
26 | * Loss function used
27 | * Justification for the choice of model types implemented
28 | * For the model(s) reported in the paper, specify details about the hyperparameter tuning:
29 | * Method to select the best hyper-parameter configuration
30 | * Specification of all hyper-parameters used to generate results reported in the paper
31 | * Justification that any model comparisons are against appropriate baselines.
32 | * Discussion of any possible paths for data leakage and possible impact on assessment (e.g., hyperparameter optimization using whole database before train/test split).
33 | ## Reproducibility and Usability
34 | ### Strongly suggest:
35 | * Open dataset used for training and evaluating the model along with link or DOI to uniquely identify the dataset (such as FigShare, Zenodo etc.)
36 | * Open code used to train and evaluate the model and produce the results reported in the paper along with link or DOI/hash-id to uniquely identify the version of the code used
37 | * Where relevant benchmarks and benchmarking frameworks are available, a report of the performance of the model on the benchmark data (such as JARVIS-Leaderboard, MatBench, OpenCatalyst).
38 | * Description of the computing infrastructure used
39 | * Hardware infrastructure: CPU, GPU, RAM, disk space etc.
40 | * Operating system
41 | * Software environment: Programming language and version, documentation of all packages used along with versions and dependencies (e.g., through a requirements.txt/environment.yaml/nix/docker file)
42 | * An estimate of the time taken to generate the results
43 | * Saved experimental data traces for automated experiments, e.g. sequence of data files and model states with the absolute or relative timestamps
44 | * Artifacts (e.g. predicted properties, generated structures etc) produced by the model and used as results in the manuscript made available
45 | * Any provided data should be in a recognised and usable format
46 | * Discussion of model domain of applicability, if practical
47 |
48 | ### Nice to have:
49 | * A cloud-based implementation of the code (e.g. JARVIS-Leaderboard, Colab, Huggingface, Garden, etc)
50 | * As much uncertainty quantification of the model as practical
51 |
52 |
--------------------------------------------------------------------------------
/editorial-data/editorial-stats.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ML-Materials-Standards/ml-materials-checklist/b0d52be53e9f9319478262952768444d28b1bb07/editorial-data/editorial-stats.png
--------------------------------------------------------------------------------
/editorial-data/plot-publication-stats.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 7,
6 | "id": "49df915a-f7f1-45be-912b-9780d84fee49",
7 | "metadata": {
8 | "tags": []
9 | },
10 | "outputs": [],
11 | "source": [
12 | "import matplotlib.pyplot as plt\n",
13 | "import pandas as pd\n",
14 | "import numpy as np\n",
15 | "\n",
16 | "#plt.style.use('mdi-style')\n",
17 | "cols = plt.rcParams['axes.prop_cycle'].by_key()['color']"
18 | ]
19 | },
20 | {
21 | "cell_type": "code",
22 | "execution_count": 2,
23 | "id": "648676e1-a4b6-4933-b0ff-4dcea8fc0ad5",
24 | "metadata": {
25 | "tags": []
26 | },
27 | "outputs": [],
28 | "source": [
29 | "data = pd.read_excel('publication-stats.xlsx')"
30 | ]
31 | },
32 | {
33 | "cell_type": "markdown",
34 | "id": "14af74b0-3f42-450c-b788-df1d58020763",
35 | "metadata": {},
36 | "source": [
37 | "# Data generation\n",
38 | "\n",
39 | "These data were generated from Web of Science with the following search terms:\n",
40 | "\n",
41 | "- ML + Mat Sci: `Machine Learning Materials Science`\n",
42 | "- Mat Sci: `Materials Science`\n",
43 | "- ML: `Machine Learning`"
44 | ]
45 | },
46 | {
47 | "cell_type": "code",
48 | "execution_count": 3,
49 | "id": "dcc7a9dd-d8ef-48f2-bd09-e2a476ee4fcd",
50 | "metadata": {
51 | "tags": []
52 | },
53 | "outputs": [
54 | {
55 | "data": {
56 | "text/html": [
57 | "\n",
58 | "\n",
71 | "
\n",
72 | " \n",
73 | " \n",
74 | " Year \n",
76 | " ML + Mat Sci \n",
77 | " Mat Sci \n",
78 | " ML \n",
79 | " \n",
80 | " \n",
81 | " \n",
82 | " \n",
83 | " 0 \n",
84 | " 2023 \n",
85 | " 4827 \n",
86 | " 262655 \n",
87 | " 83264 \n",
88 | " \n",
89 | " \n",
90 | " 1 \n",
91 | " 2022 \n",
92 | " 4268 \n",
93 | " 268723 \n",
94 | " 83295 \n",
95 | " \n",
96 | " \n",
97 | " 2 \n",
98 | " 2021 \n",
99 | " 3038 \n",
100 | " 251571 \n",
101 | " 70014 \n",
102 | " \n",
103 | " \n",
104 | " 3 \n",
105 | " 2020 \n",
106 | " 1992 \n",
107 | " 235922 \n",
108 | " 53087 \n",
109 | " \n",
110 | " \n",
111 | " 4 \n",
112 | " 2019 \n",
113 | " 1252 \n",
114 | " 222686 \n",
115 | " 44830 \n",
116 | " \n",
117 | " \n",
118 | "
\n",
119 | "
"
181 | ]
182 | },
183 | "metadata": {},
184 | "output_type": "display_data"
185 | }
186 | ],
187 | "source": [
188 | "offset = 0.20\n",
189 | "width = 0.4 \n",
190 | "fig, ax = plt.subplots(figsize=(8, 6))\n",
191 | "secy = ax.twinx()\n",
192 | "\n",
193 | "bar_a = ax.bar(x = years - offset, height = norm_by_mat_sci, width=width, edgecolor='black', label = '% mat sci papers')\n",
194 | "bar_b = ax.bar(x = years + offset, height = norm_by_ml, width=width, edgecolor='black', label = '% ml papers')\n",
195 | "secy.plot(years, mat_sci, marker='o', markeredgecolor='black', c=cols[2], label = 'Total mat sci papers')\n",
196 | "secy.plot(years, ml, marker='s', markeredgecolor='black', c=cols[3], label = 'Total ML papers')\n",
197 | "ax.vlines(2011, 0, 4.2, color='black', ls='--', lw=1)\n",
198 | "ax.text(2011.1, 4.1, 'Materials Genome \\n Initiative')\n",
199 | "\n",
200 | "ax.vlines(2016, 0, 2.8, color='black', ls='--', lw=1)\n",
201 | "ax.text(2016.1, 2.7, 'Alpha Go')\n",
202 | "ax.set_ylabel('% papers')\n",
203 | "secy.set_ylabel('Total # papers')\n",
204 | "\n",
205 | "ax.set_xticks(range(2006, 2024))\n",
206 | "ax.set_xticklabels(range(2006, 2024), rotation = 75)\n",
207 | "ax.legend(loc=(0.01, 0.9))\n",
208 | "secy.legend(loc=(0.4, 0.9))\n",
209 | "\n",
210 | "plt.savefig('editorial-stats.png')"
211 | ]
212 | },
213 | {
214 | "cell_type": "code",
215 | "execution_count": null,
216 | "id": "41f0bed9-5dc8-4f1e-b1a3-c324c38fd293",
217 | "metadata": {},
218 | "outputs": [],
219 | "source": []
220 | }
221 | ],
222 | "metadata": {
223 | "kernelspec": {
224 | "display_name": "Python 3 (ipykernel)",
225 | "language": "python",
226 | "name": "python3"
227 | },
228 | "language_info": {
229 | "codemirror_mode": {
230 | "name": "ipython",
231 | "version": 3
232 | },
233 | "file_extension": ".py",
234 | "mimetype": "text/x-python",
235 | "name": "python",
236 | "nbconvert_exporter": "python",
237 | "pygments_lexer": "ipython3",
238 | "version": "3.9.16"
239 | }
240 | },
241 | "nbformat": 4,
242 | "nbformat_minor": 5
243 | }
244 |
--------------------------------------------------------------------------------
/editorial-data/publication-stats.xlsx:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ML-Materials-Standards/ml-materials-checklist/b0d52be53e9f9319478262952768444d28b1bb07/editorial-data/publication-stats.xlsx
--------------------------------------------------------------------------------