![\"Parity](\"assets/result.gif\")
\n",
498 | "\n",
499 | "This exercise requires a basic knowledge of Python, e.g. that you can write list comprehensions, and are able to read documentation of functions provided by Python packages.\n",
500 | "You will be asked to provide some function arguments (indicated by `#fillme` comments).\n",
501 | "\n",
502 | "You can execute all the following code cells by pressing SHIFT and ENTER and get informations about the functions by pressing TAB when you are between the parentheses (see the notes for more tips). \n",
503 | "\n",
504 | "Also the [sklearn documentation](https://scikit-learn.org/stable/user_guide.html) is a great source of reference with many explanations and examples.\n",
505 | "\n",
506 | "In pandas dataframe (df) you can select columns using their name by running `df[columnname]`. If at any point you think that the dataset is too large for your computer, you can select a subset using `df.sample()` or by making the test set larger in the train/test split (section 2). "
507 | ]
508 | },
509 | {
510 | "cell_type": "markdown",
511 | "metadata": {},
512 | "source": [
513 | "## 1. Import the data"
514 | ]
515 | },
516 | {
517 | "cell_type": "code",
518 | "execution_count": 3,
519 | "metadata": {},
520 | "outputs": [],
521 | "source": [
522 | "df = pd.read_csv(DATA_FILE)"
523 | ]
524 | },
525 | {
526 | "cell_type": "markdown",
527 | "metadata": {},
528 | "source": [
529 | "Let's take a look at the first few rows to see if everythings seems reasonable ..."
530 | ]
531 | },
532 | {
533 | "cell_type": "code",
534 | "execution_count": 4,
535 | "metadata": {},
536 | "outputs": [
537 | {
538 | "data": {
539 | "text/html": [
540 | "![\"RACs](\"assets/racs.png\")
"
859 | ]
860 | },
861 | {
862 | "cell_type": "markdown",
863 | "metadata": {},
864 | "source": [
865 | "The number in the descriptornames shows the coordination shell that was considered in the calculation of the RACs."
866 | ]
867 | },
868 | {
869 | "cell_type": "markdown",
870 | "metadata": {},
871 | "source": [
872 | "The target we use for this application is the high-pressure CO$_2$ uptake. This is the amount of CO$_2$ (mmol) the MOF can load per gram."
873 | ]
874 | },
875 | {
876 | "cell_type": "markdown",
877 | "metadata": {},
878 | "source": [
879 | "## 2. Split the data"
880 | ]
881 | },
882 | {
883 | "cell_type": "markdown",
884 | "metadata": {},
885 | "source": [
886 | "Next, we split our data into a training set and a test set.\n",
887 | "\n",
888 | "In order to prevent *any* information of the test set from leaking into our model, we split *before* starting to analyze or transform our data. For more details on why this matters, see [chapter 7.10.2 of Elements of Statistical Learning](https://web.stanford.edu/~hastie/ElemStatLearn//printings/ESLII_print10.pdf)."
889 | ]
890 | },
891 | {
892 | "cell_type": "markdown",
893 | "metadata": {},
894 | "source": [
895 | "### 2.1. Split with stratification"
896 | ]
897 | },
898 | {
899 | "cell_type": "markdown",
900 | "metadata": {},
901 | "source": [
902 | "[Stratification](https://en.wikipedia.org/wiki/Stratified_sampling) ensures that the class distributions (ratio of \"good\" to \"bad\" materials) are the same in the training and test set.\n",
903 | "\n",
904 | " $\\color{DarkBlue}{\\textsf{Short question}}$\n",
905 | "\n",
906 | "- Why is this important? What could happen if we would not do this? \n",
907 | " \n",
908 | "\n",
909 | "For stratification to work, we to define what makes a \"good\" or a \"bad\" material. We will use 15 mmol CO$_2$ / g as the threshold for the uptake, thus binarizing our continuous target variable. (You can choose it based on the histogram of the variables)."
910 | ]
911 | },
912 | {
913 | "cell_type": "markdown",
914 | "metadata": {},
915 | "source": [
916 | " $\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
917 | " - add a column 'target_binary' that encodes whether a material is low performing (`0`) or high perfoming (`1`) by comparing the uptake with the `THRESHOLD`"
918 | ]
919 | },
920 | {
921 | "cell_type": "markdown",
922 | "metadata": {},
923 | "source": [
924 | "\n",
925 | "
"
932 | ]
933 | },
934 | {
935 | "cell_type": "code",
936 | "execution_count": null,
937 | "metadata": {},
938 | "outputs": [],
939 | "source": [
940 | "THRESHOLD = 15 # in units of mmol CO2/g\n",
941 | "df[TARGET_BINARY] = # add your code"
942 | ]
943 | },
944 | {
945 | "cell_type": "markdown",
946 | "metadata": {},
947 | "source": [
948 | "Now, we can perform the actual split into training and test set."
949 | ]
950 | },
951 | {
952 | "cell_type": "markdown",
953 | "metadata": {},
954 | "source": [
955 | " $\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
956 | "- select reasonable values for `XX` and `XY` and then perform the test/train splits. What do you consider when making this decision (think about what you would do with really small and really big datasets, what happens if you have only one test point, what happens to the model performance if you have more test points than training points)? \n",
957 | "- why do we need to perform the split into a training and test set? \n",
958 | "- would we use the test set to tune the hyperparameters of our model?\n",
959 | "\n",
960 | "Click here for a hint
\n", 926 | "- \n",
927 | "
- you can use pd.cut, \n", 928 | " list comprehension, the binarizer in sklearn ...) \n", 929 | "
- a list comprehension example:
[1 if value > THRESHOLD else 0 for value in df[TARGET]]\n",
930 | "
\n",
961 | "
"
968 | ]
969 | },
970 | {
971 | "cell_type": "code",
972 | "execution_count": null,
973 | "metadata": {},
974 | "outputs": [],
975 | "source": [
976 | "df_train_stratified, df_test_stratified = train_test_split(\n",
977 | " df,\n",
978 | " train_size=XX,\n",
979 | " test_size=XY,\n",
980 | " random_state=RANDOM_SEED,\n",
981 | " stratify=df[TARGET_BINARY],\n",
982 | ")\n"
983 | ]
984 | },
985 | {
986 | "cell_type": "markdown",
987 | "metadata": {},
988 | "source": [
989 | "## 3. Exploratory data analysis (EDA) "
990 | ]
991 | },
992 | {
993 | "cell_type": "markdown",
994 | "metadata": {},
995 | "source": [
996 | "After we have put the test set aside, we can give the training set a closer look."
997 | ]
998 | },
999 | {
1000 | "cell_type": "markdown",
1001 | "metadata": {},
1002 | "source": [
1003 | "### 3.1. Correlations"
1004 | ]
1005 | },
1006 | {
1007 | "cell_type": "markdown",
1008 | "metadata": {},
1009 | "source": [
1010 | " $\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
1011 | "- Plot some features against the target property and calculate the Pearson and Spearman correlation coefficient (what is the different between those correlation coefficients?) \n",
1012 | "- What are the strongest correlations? Did you expect them? \n",
1013 | "- What can be a problem when features are correlated?\n",
1014 | "- *Optional:* Do they change if you switch from CO$_2$ to CH$_4$ uptake as the target instead? Explain your observation."
1015 | ]
1016 | },
1017 | {
1018 | "cell_type": "markdown",
1019 | "metadata": {},
1020 | "source": [
1021 | "To get the correlation matrices, you can use the `df.corr(method=)`method on your dataframe (`df`). You might want to calculate not the full correlation matrix but just the correlation of the features with the targets"
1022 | ]
1023 | },
1024 | {
1025 | "cell_type": "markdown",
1026 | "metadata": {},
1027 | "source": [
1028 | "Click here for a hint
\n", 962 | "- \n",
963 | "
- The `size` arguments can either be integers or, often more convenient, decimals like 0.1 \n", 964 | "
- When you perform the split into training and test set you need to trade-off bias (pessimistic bias due to little training data) and variance (due to little test data) \n", 965 | "
- A typical split cloud be 70/30, but for huge dataset the test set might be too big and for small datasets the training set might be too small in this way \n", 966 | "
\n",
1029 | "
"
1036 | ]
1037 | },
1038 | {
1039 | "cell_type": "code",
1040 | "execution_count": null,
1041 | "metadata": {},
1042 | "outputs": [],
1043 | "source": [
1044 | "# add code here"
1045 | ]
1046 | },
1047 | {
1048 | "cell_type": "markdown",
1049 | "metadata": {},
1050 | "source": [
1051 | "## 4. Baselines"
1052 | ]
1053 | },
1054 | {
1055 | "cell_type": "markdown",
1056 | "metadata": {},
1057 | "source": [
1058 | "For machine learning, it is important to have some *baselines* to which one then compares the results of a model. Think of a classification model for some rare disease where we only have 1% postives. A classification model that only predictes the negatives *all the time* will still have a amazingly high accuracy. To be able to understand if our model is really better than such a simple prediction we need to make the simple prediction first. This is what we call a baseline.\n",
1059 | "\n",
1060 | "A baseline could be a really simple model, a basic heuristic or the current state of the art.\n",
1061 | "this. We will use a heuristic.\n",
1062 | "\n",
1063 | "For this we use sklearn `Dummy` objects that simply calculate the mean, the median or the most frequent case of the training set, when you run the `fit()` method on them (which takes the features matrix $\\mathbf{X}$ and the labels $\\mathbf{y}$ as arguments.\n",
1064 | "This is, the prediction of a `DummyRegressor` with `mean` strategy will always be the mean, independent of the input (it will not look at the feature matrix!). \n",
1065 | "\n",
1066 | "Instead of using those `sklearn` objects you could also just manually compute the the mean or median of the dataset. But we will use those objects as we can learn in this way how to use estimators in `sklearn` and it is also allows you to test your full pipeline with different (baseline) models. \n",
1067 | "What does this mean? In practice this means that you can use all the regression and classification models shown in the figure below in the same way, they will all have a `fit()` method that accepts `X` and `y` and a predict method that accepts `X` and returns the predictions. \n",
1068 | "\n",
1069 | "\n",
1070 | "Click here for a hint
\n", 1030 | "- \n",
1031 | "
- To get the correlation with a target, you can use indexing. E.g.
df.corr(method='spearman')[TARGET]\n",
1032 | " - use
.sort_values()method on the output of `df.corr()` to sort by the value of the correlation coefficient \n",
1033 | " - You can use something like
scatter = hv.Scatter(df, 'Di', [TARGET, 'density [g/cm^3]']).opts(color='density [g/cm^3]', cmap='rainbow')for plotting. Also consider theholoviewsdocumentation. In caseholoviewsis too new for you, you can of course just usematplotliband something likeplt.scatter(x,y)\n",
1034 | "
![\"ML](\"https://scikit-learn.org/stable/assets/ml_map.png\")
\n",
1071 | "\n",
1072 | "The estimator objects can be always used in the same way \n",
1073 | "\n",
1074 | "![\"ML](\"https://static.packt-cdn.com/products/9781789800265/graphics/d49a2e95-8f22-42ed-89f1-474b3d028787.png\")
\n",
1075 | "\n",
1076 | "Using these objects, instead of the mean directly, allows you to easily swap them with other models in pipelines, where one chains many data transformation steps (see section 6)."
1077 | ]
1078 | },
1079 | {
1080 | "cell_type": "markdown",
1081 | "metadata": {},
1082 | "source": [
1083 | "### 4.1. Build dummy models"
1084 | ]
1085 | },
1086 | {
1087 | "cell_type": "markdown",
1088 | "metadata": {},
1089 | "source": [
1090 | "$\\color{DarkBlue}{\\textsf{Short Question}}$\n",
1091 | "- If you call `.fit(X, y)` on a `DummyRegressor` does it actually use the `X`? If not, why is there still the place for the `X` in the function? If yes, how does it use it?\n",
1092 | "\n",
1093 | "$\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
1094 | "- Create [`DummyRegressor`](https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyRegressor.html) instances for `mean`, `median`. (e.g. `dummyinstance = DummyRegressor(strategy='mean')`)\n",
1095 | "- Train them on the training data (`dummyinstance.fit(df_train[FEATURES], df_train[TARGET])`)"
1096 | ]
1097 | },
1098 | {
1099 | "cell_type": "markdown",
1100 | "metadata": {},
1101 | "source": [
1102 | "\n",
1103 | "
"
1109 | ]
1110 | },
1111 | {
1112 | "cell_type": "code",
1113 | "execution_count": null,
1114 | "metadata": {},
1115 | "outputs": [],
1116 | "source": [
1117 | "# Build DummyRegressors\n",
1118 | "dummyregressor_mean = DummyRegressor(strategy='mean')\n",
1119 | "dummyregressor_median = DummyRegressor(#fillme)"
1120 | ]
1121 | },
1122 | {
1123 | "cell_type": "code",
1124 | "execution_count": null,
1125 | "metadata": {},
1126 | "outputs": [],
1127 | "source": [
1128 | "# Fit Dummy Regressors\n",
1129 | "dummyregressor_mean.fit(df_train_stratified[FEATURES], df_train_stratified[TARGET])\n",
1130 | "dummyregressor_median.fit(#fillme)"
1131 | ]
1132 | },
1133 | {
1134 | "cell_type": "markdown",
1135 | "metadata": {},
1136 | "source": [
1137 | "#### Evaluate the performance of the dummy models"
1138 | ]
1139 | },
1140 | {
1141 | "cell_type": "markdown",
1142 | "metadata": {},
1143 | "source": [
1144 | " $\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
1145 | "- Calculate maximum error, mean absolute error and mean square error for the dummy regressors on training and test set. What would you expect those numbers to be?\n",
1146 | "- Do the actual values surprise you? \n",
1147 | "- What does this mean in practice for reporting of metrics/the reasoning behind using baseline models\n",
1148 | "\n",
1149 | "It can be handy to store our metrics of choice in a nested dictionary ([Python dictionaries are key-value pairs](https://www.tutorialspoint.com/python/python_dictionary.htm)): \n",
1150 | "\n",
1151 | "```python\n",
1152 | "{\n",
1153 | " 'dummyestimator1': {\n",
1154 | " 'metric_a_key': metric_a_value, \n",
1155 | " 'metric_b_key': metric_b_value\n",
1156 | " },\n",
1157 | " 'dummyestimator2': {\n",
1158 | " 'metric_a_key': metric_a_value, \n",
1159 | " 'metric_b_key': metric_b_value\n",
1160 | " },\n",
1161 | " }\n",
1162 | "``` \n",
1163 | "\n",
1164 | "You will now write functions `get_regression_metrics(model, X, y_true)` that compute the metrics and return this dictionary for a given model. The `predict` method takes the feature matrix $\\mathbf{X}$ as input.\n",
1165 | "\n",
1166 | "In them, we calculate \n",
1167 | "\n",
1168 | "$\\mathrm {MAE} =\\frac{\\sum _{i=1}^{n}\\left|Y_{i}-\\hat{y}_{i}\\right|}{n}.$\n",
1169 | "\n",
1170 | "and \n",
1171 | "\n",
1172 | "$\\mathrm {MSE} = {\\frac {1}{n}}\\sum _{i=1}^{n}(Y_{i}-{\\hat {Y_{i}}})^{2}.$ \n",
1173 | "\n",
1174 | "where $\\hat{y}$ are the predictions and, $Y_{i}$ the true values.\n",
1175 | "\n",
1176 | "as well as the maximum error."
1177 | ]
1178 | },
1179 | {
1180 | "cell_type": "markdown",
1181 | "metadata": {},
1182 | "source": [
1183 | "Click here for hints
\n", 1104 | "- \n",
1105 | "
- to create
DummyRegressoryou can for example usedummyregressor_mean = DummyRegressor(strategy='mean')\n",
1106 | " - to see the implementation of the
DummyRegressoryou can check out the source code on GitHub \n",
1107 | "
\n",
1184 | "
"
1190 | ]
1191 | },
1192 | {
1193 | "cell_type": "code",
1194 | "execution_count": null,
1195 | "metadata": {},
1196 | "outputs": [],
1197 | "source": [
1198 | "def get_regression_metrics(model, X, y_true):\n",
1199 | " \"\"\"\n",
1200 | " Get a dicionary with regression metrics:\n",
1201 | "\n",
1202 | " model: sklearn model with predict method\n",
1203 | " X: feature matrix\n",
1204 | " y_true: ground truth labels\n",
1205 | " \"\"\"\n",
1206 | " y_predicted = model.predict(#fillme)\n",
1207 | "\n",
1208 | " mae = mean_absolute_error(#fillme)\n",
1209 | " mse = mean_squared_error(#fillme)\n",
1210 | " maximum_error = max_error(#fillme)\n",
1211 | "\n",
1212 | " metrics_dict = {\n",
1213 | " 'mae': mae,\n",
1214 | " 'mse': mse,\n",
1215 | " 'max_error': maximum_error\n",
1216 | " }\n",
1217 | "\n",
1218 | " return metrics_dict"
1219 | ]
1220 | },
1221 | {
1222 | "cell_type": "code",
1223 | "execution_count": null,
1224 | "metadata": {},
1225 | "outputs": [],
1226 | "source": [
1227 | "dummy_regressors = [\n",
1228 | " ('mean', dummyregressor_mean),\n",
1229 | " ('median', dummyregressor_median)\n",
1230 | "]"
1231 | ]
1232 | },
1233 | {
1234 | "cell_type": "code",
1235 | "execution_count": null,
1236 | "metadata": {},
1237 | "outputs": [],
1238 | "source": [
1239 | "dummy_regressor_results_test = {} # initialize empty dictionary\n",
1240 | "dummy_regressor_results_train = {}\n",
1241 | "\n",
1242 | "# loop over the dummy_regressor list\n",
1243 | "# if you have a tuple regressorname, regressor = (a, b) that is automatically expanded into the variables\n",
1244 | "# a = regressorname, b = regressor\n",
1245 | "for regressorname, regressor in dummy_regressors:\n",
1246 | " print(f\"Calculating metrics for {regressorname}\")\n",
1247 | " dummy_regressor_results_test[regressorname] = get_regression_metrics(#fillme)\n",
1248 | " dummy_regressor_results_train[regressorname] = get_regression_metrics(#fillme)"
1249 | ]
1250 | },
1251 | {
1252 | "cell_type": "markdown",
1253 | "metadata": {},
1254 | "source": [
1255 | "## 5. Build actual regression models"
1256 | ]
1257 | },
1258 | {
1259 | "cell_type": "markdown",
1260 | "metadata": {},
1261 | "source": [
1262 | "Let's build a simple [kernel ridge regression (KRR)](https://emtiyaz.github.io/pcml15/kernel-ridge-regression.pdf) machine learning model and train it with our raw data.\n",
1263 | "You can try different kernels, but we recommend to start with the Gaussian radial basis function ('rbf') kernel.\n",
1264 | " \n",
1265 | " $\\color{DarkBlue}{\\textsf{Short Question}}$\n",
1266 | "- Do you expect this model to perform better than the dummy models?\n",
1267 | "- Train it and then calculate the performance metrics on the training and test set. How do they compare to the performance of the dummy models?\n",
1268 | "- What is the shape of the Kernel and of the weights? (you can check your answer by looking at the `dual_coef_` attribute of the KRR instance. You can get shapes of objects using the `shape` atrribute "
1269 | ]
1270 | },
1271 | {
1272 | "cell_type": "code",
1273 | "execution_count": null,
1274 | "metadata": {},
1275 | "outputs": [],
1276 | "source": [
1277 | "# Train the model with a Gaussian kernel\n",
1278 | "krr = KernelRidge(kernel='rbf')\n",
1279 | "krr.fit(#fillme)"
1280 | ]
1281 | },
1282 | {
1283 | "cell_type": "code",
1284 | "execution_count": null,
1285 | "metadata": {},
1286 | "outputs": [],
1287 | "source": [
1288 | "# get the metrics on the train and the test set using the get_regression_metrics functions (as above)"
1289 | ]
1290 | },
1291 | {
1292 | "cell_type": "markdown",
1293 | "metadata": {},
1294 | "source": [
1295 | "## 6. Evaluate the model performance in detail"
1296 | ]
1297 | },
1298 | {
1299 | "cell_type": "markdown",
1300 | "metadata": {},
1301 | "source": [
1302 | "We have trained our first machine learning model!\n",
1303 | "We'll first have a closer look at its performance, before learning how to improve it."
1304 | ]
1305 | },
1306 | {
1307 | "cell_type": "markdown",
1308 | "metadata": {},
1309 | "source": [
1310 | " $\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
1311 | "- Create a parity plot (true values against predictions)for the training and test data\n",
1312 | "- Plot a histogram of the distribution of the training and test errors on the training and test set. Plot the errors also as a function of the true value\n",
1313 | "- Let's assume we would like to use our model for pre-screening a library of millions of porous materials to zoom-in on those with the most promising gas uptake. Could you tolerate the errors of your model?\n",
1314 | "- Compare the parity plots for this model with the ones for the dummy models. \n",
1315 | "Use the plotting functions below the evaluate all the following models you train.\n",
1316 | "\n",
1317 | "For this exercise, it can be handy to save the results in a dictionary, e.g. \n",
1318 | "```(python)\n",
1319 | "res_train = {\n",
1320 | " 'y true': [],\n",
1321 | " 'y pred': []\n",
1322 | "}\n",
1323 | "```"
1324 | ]
1325 | },
1326 | {
1327 | "cell_type": "markdown",
1328 | "metadata": {},
1329 | "source": [
1330 | "Click here for hints
\n", 1185 | "- \n",
1186 | "
- to perform a prediction using a estimator object, you can call
classifier.predict(X)\n",
1187 | " - to calculate metrics, you can for example call
accuracy_score(true_values, predicted_values)\n",
1188 | "
\n",
1331 | "
"
1337 | ]
1338 | },
1339 | {
1340 | "cell_type": "code",
1341 | "execution_count": null,
1342 | "metadata": {},
1343 | "outputs": [],
1344 | "source": [
1345 | "# Create dictionaries with training and test results to create parity plots\n",
1346 | "res_train = {\n",
1347 | " 'y true': # fillme using the dataframe,\n",
1348 | " 'y pred': # fillme using the model prediction\n",
1349 | "}\n",
1350 | "\n",
1351 | "res_test = {\n",
1352 | " 'y true': # fillme using the dataframe\n",
1353 | " 'y pred': # fillme using the model prediction\n",
1354 | "}"
1355 | ]
1356 | },
1357 | {
1358 | "cell_type": "markdown",
1359 | "metadata": {},
1360 | "source": [
1361 | "Now, lets calculate the errors"
1362 | ]
1363 | },
1364 | {
1365 | "cell_type": "code",
1366 | "execution_count": null,
1367 | "metadata": {},
1368 | "outputs": [],
1369 | "source": [
1370 | "res_train[\"error\"] = res_train[\"y true\"] - res_train[\"y pred\"]\n",
1371 | "res_test[\"error\"] = res_test[\"y true\"] - res_test[\"y pred\"]\n"
1372 | ]
1373 | },
1374 | {
1375 | "cell_type": "markdown",
1376 | "metadata": {},
1377 | "source": [
1378 | "Now, plot the parity plots and error distributions"
1379 | ]
1380 | },
1381 | {
1382 | "cell_type": "markdown",
1383 | "metadata": {},
1384 | "source": [
1385 | "Click here for hints for plotting
\n", 1332 | "- \n",
1333 | "
- If you want to use matplotlib to make the parity plots, you can use the hist2d function \n", 1334 | "
- To create the frequencies and the edges of a histogram, one can use
np.histogram\n",
1335 | "
\n",
1386 | "
"
1399 | ]
1400 | },
1401 | {
1402 | "cell_type": "code",
1403 | "execution_count": null,
1404 | "metadata": {},
1405 | "outputs": [],
1406 | "source": [
1407 | "# plot it\n",
1408 | "hist_density(res_train['y true'], res_train['y pred'], xlabel='y true', ylabel='y pred', title='Train')"
1409 | ]
1410 | },
1411 | {
1412 | "cell_type": "code",
1413 | "execution_count": null,
1414 | "metadata": {},
1415 | "outputs": [],
1416 | "source": [
1417 | "hist_density(res_test['y true'], res_test['y pred'], xlabel='y true', ylabel='y pred', title='Test')"
1418 | ]
1419 | },
1420 | {
1421 | "cell_type": "markdown",
1422 | "metadata": {},
1423 | "source": [
1424 | "## 7. Improve the model "
1425 | ]
1426 | },
1427 | {
1428 | "cell_type": "markdown",
1429 | "metadata": {},
1430 | "source": [
1431 | "Our training set still has a couple of issues you might have noticed:\n",
1432 | "- The feature values are not scaled (different features are measured in different units ...)\n",
1433 | "- Some features are basically constant, i.e. do not contain relevant information and just increase the dimensionality of the problem \n",
1434 | "- Some feature distributions are skewed (which is more relevant for some models than for others ...)"
1435 | ]
1436 | },
1437 | {
1438 | "cell_type": "markdown",
1439 | "metadata": {},
1440 | "source": [
1441 | "$\\color{DarkBlue}{\\textsf{Short Question}}$\n",
1442 | "- Why might the scaling of the features be relevant for a machine learning model? "
1443 | ]
1444 | },
1445 | {
1446 | "cell_type": "markdown",
1447 | "metadata": {},
1448 | "source": [
1449 | "### 7.1. Standard scaling and building a first pipeline "
1450 | ]
1451 | },
1452 | {
1453 | "cell_type": "markdown",
1454 | "metadata": {},
1455 | "source": [
1456 | "Given that we will now go beyond training a single model, we will build [Pipelines](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html), which are objects that can collect a selection of transformations and estimators. This makes it quite easy to apply the same set of operations to different datasets. A simple pipeline might be built as follows \n",
1457 | "\n",
1458 | "Click here for hints for plotting
\n", 1387 | "If you want interactive plots, you can use the following code:\n", 1388 | "\n",
1389 | "hv.extension(\"bokeh\")\n",
1390 | "hex_train = hv.HexTiles(res_train, [\"y true\", \"y pred\"]).hist(\n",
1391 | " dimension=[\"y true\", \"y pred\"]\n",
1392 | ")\n",
1393 | "hex_test = hv.HexTiles(res_test, [\"y true\", \"y pred\"]).hist(\n",
1394 | " dimension=[\"y true\", \"y pred\"]\n",
1395 | ")\n",
1396 | "hex_train + hex_test\n",
1397 | "\n",
1398 | "![\"Pipeline\"](\"https://vitalflux.com/wp-content/uploads/2020/08/ML-Pipeline-Page-2-1024x307.png\")
\n",
1459 | "\n"
1460 | ]
1461 | },
1462 | {
1463 | "cell_type": "markdown",
1464 | "metadata": {},
1465 | "source": [
1466 | " $\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
1467 | "- Build a pipline that first performs standard scaling and then fits a KRR. Call it `pipe_w_scaling`. \n",
1468 | "- Fit it on the training set \n",
1469 | "- Make predictions, calculate the errors and make the parity plots"
1470 | ]
1471 | },
1472 | {
1473 | "cell_type": "markdown",
1474 | "metadata": {},
1475 | "source": [
1476 | "\n",
1477 | "
"
1482 | ]
1483 | },
1484 | {
1485 | "cell_type": "code",
1486 | "execution_count": null,
1487 | "metadata": {},
1488 | "outputs": [],
1489 | "source": [
1490 | "pipe_w_scaling = Pipeline(\n",
1491 | " [\n",
1492 | " ('scaling', StandardScaler()),\n",
1493 | " ('krr', #fillme)\n",
1494 | " ]\n",
1495 | ")"
1496 | ]
1497 | },
1498 | {
1499 | "cell_type": "markdown",
1500 | "metadata": {},
1501 | "source": [
1502 | "### 7.2. Hyperparameter optimization"
1503 | ]
1504 | },
1505 | {
1506 | "cell_type": "markdown",
1507 | "metadata": {},
1508 | "source": [
1509 | "A key component we did not optimize so far are hyperparameters. Those are parameters of the model that we usually cannot learn from the data but have to fix before we train the model. \n",
1510 | "Since we cannot learn those parameters it is not trivial to select them. Hence, what we typically do in practice is to create another set, a \"validation set\", and use it to test models trained with different hyperparameters.\n",
1511 | "\n",
1512 | "The most common approach to hyperparameter optimization is to define a grid of all relevant parameters and to search over the grid for the best model performance."
1513 | ]
1514 | },
1515 | {
1516 | "cell_type": "markdown",
1517 | "metadata": {},
1518 | "source": [
1519 | "$\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
1520 | "- Think about which parameters you could optimize in the pipeline. Note that your KRR model has two parameters you can optimize. You can also switch off some steps by setting them to `None'.\n",
1521 | "- For each parameter you need to define a resonable grid to search over.\n",
1522 | "- Recall, what k-fold cross-validation does. Run the hyperparameter optimization using 5-fold cross-validation (you can adjust the number of folds according to your computational resources/impatience. It turns out at k=10 is the [best tradeoff between variance and bias](https://arxiv.org/abs/1811.12808)). \n",
1523 | "Tune the hyperparameters until you are statisfied (e.g., until you cannot improve the cross validated error any more)\n",
1524 | "- Why don't we use the test set for hyperparameter tuning but instead test on the validation set? \n",
1525 | "- Evaluate the model performance by calculating the performance metrics (MAE, MSE, max error) on the training and the test set.\n",
1526 | "- *Optional:* Instead of grid search, try to use random search on the same grid (`RandomizedSearchCV`) and fix the number of evaluations (`n_iter`) to a fraction of the number of evaluations of grid search. What do you observe and conclude?"
1527 | ]
1528 | },
1529 | {
1530 | "cell_type": "markdown",
1531 | "metadata": {},
1532 | "source": [
1533 | " $\\color{DarkRed}{\\textsf{Tips}}$\n",
1534 | "- If you want to see what is happening, set the `verbosity` argument of the `GridSearchCV` object to a higher number.\n",
1535 | " \n",
1536 | "- If you want to speed up the optimization, you can run it in parallel by setting the `n_jobs` argument to the number of workers. If you set it to -1 it will use all available cores. *Using all cores might freeze your computer if you do not have enough memory*\n",
1537 | " \n",
1538 | "- If the optimization is too slow, reduce the number of data points in your set, the number of folds or the grid size. Note that it can also be a feasible strategy to first use a coarser grid and the a finer grid for fine-tuning.\n",
1539 | "\n",
1540 | "- For grid search, you need to define a parameter grid, which is a dictionary of the following form: \n",
1541 | "```(python)\n",
1542 | "param_grid = {\n",
1543 | " 'pipelinestage__parameter': np.logspace(-4,1,10),\n",
1544 | " 'pipelinestage': [None, TransformerA(), TransformerB()]\n",
1545 | " }\n",
1546 | "```\n",
1547 | "\n",
1548 | "- After the search, you can access the best model with `.best_estimator_` and the best parameters with `.best_params_` on the GridSearchCV instance. For example `grid_krr.best_estimator_`\n",
1549 | "\n",
1550 | "- If you initialize the GridSearchCV instance with `refit=True` it will automatically train the model with all training data (and not only the training folds from cross-validations)\n",
1551 | "\n",
1552 | "The double underscore (dunder) notation works recursively and specifies the parameters for any pipeline stage. \n",
1553 | "For example, `ovasvm__estimator__cls__C` would specifiy the `C` parameter of the estimator in the one-versus-rest classifier `ovasvm`. \n",
1554 | "\n",
1555 | "You can print all parameters of the pipeline using `print(sorted(pipeline.get_params().keys()))`"
1556 | ]
1557 | },
1558 | {
1559 | "cell_type": "markdown",
1560 | "metadata": {},
1561 | "source": [
1562 | "Click here for hints
\n", 1478 | "- \n",
1479 | "
- the
fit,predictmethods also work for pipelines \n",
1480 | "
\n",
1563 | "Be aware that tight grids will drastically increase the number of experiments you will run! In some cases, it can be useful to perform the optimization in steps, i.e., first use a coarse grid and then refine in interesting regions. \n",
1564 | "Alternatively, there are approached like hyperband that dynamically adjust the number of data points.\n",
1565 | "
"
1566 | ]
1567 | },
1568 | {
1569 | "cell_type": "markdown",
1570 | "metadata": {},
1571 | "source": [
1572 | "\n",
1573 | "
"
1580 | ]
1581 | },
1582 | {
1583 | "cell_type": "code",
1584 | "execution_count": null,
1585 | "metadata": {},
1586 | "outputs": [],
1587 | "source": [
1588 | "# Define the parameter grid and the grid search object\n",
1589 | "param_grid = {\n",
1590 | " 'scaling': [MinMaxScaler(), StandardScaler()], # test different scaling methods\n",
1591 | " 'krr__alpha': #fillme,\n",
1592 | " 'krr__#fillme': #fillme\n",
1593 | " }\n",
1594 | "\n",
1595 | "grid_krr = GridSearchCV(#your pipeline, param_grid=param_grid,\n",
1596 | " cv=#number of folds, verbose=2, n_jobs=2)\n",
1597 | "\n",
1598 | "# optional random search\n",
1599 | "#random_krr = RandomizedSearchCV(#your pipeline, param_distributions=param_grid, n_iter=#number of evaluations,\n",
1600 | "# cv=#number of folds, verbose=2, n_jobs=2\n",
1601 | ")"
1602 | ]
1603 | },
1604 | {
1605 | "cell_type": "code",
1606 | "execution_count": null,
1607 | "metadata": {},
1608 | "outputs": [],
1609 | "source": [
1610 | "# run the grid search by calling the fit method\n",
1611 | "grid_krr.fit(#fillme)\n",
1612 | "# optional random search\n",
1613 | "# random_krr.fit(#fillme)"
1614 | ]
1615 | },
1616 | {
1617 | "cell_type": "code",
1618 | "execution_count": null,
1619 | "metadata": {},
1620 | "outputs": [],
1621 | "source": [
1622 | "# get the performance metrics\n",
1623 | "get_regression_metrics(#fillme)"
1624 | ]
1625 | },
1626 | {
1627 | "cell_type": "markdown",
1628 | "metadata": {},
1629 | "source": [
1630 | "Click here for hints about pipelines and grid search
\n", 1574 | "- \n",
1575 | "
- You can use the
np.logspacefunction to generate a grid for values that you want to vary on a logarithmic scale \n",
1576 | " - There are two hyperparameters for KRR: the regularization strength
alphaand the Gaussian widthgamma\n",
1577 | " - For the regularization strength, values between 1 and 1e-3 can be reasonable. For gamma you can use the median heuristic, gamma = 1 / median, or values between 1e-3 and 1e3 \n", 1578 | "
\n",
1631 | "
"
1634 | ]
1635 | },
1636 | {
1637 | "cell_type": "markdown",
1638 | "metadata": {},
1639 | "source": [
1640 | "Click here for some more information about hyperparameter optimization
\n", 1632 | "Grid search is not the most efficient way to perform hyperparamter optimization. Even random search was shown to be more efficient. Really efficient though are Bayesian optimization approaches like TPE. This is implemented in the hyperopt library, which is also installed in your conda environment.\n", 1633 | "\n",
1641 | "
"
1694 | ]
1695 | },
1696 | {
1697 | "cell_type": "markdown",
1698 | "metadata": {},
1699 | "source": [
1700 | "## 8. Feature Engineering "
1701 | ]
1702 | },
1703 | {
1704 | "cell_type": "markdown",
1705 | "metadata": {},
1706 | "source": [
1707 | "Finally, we would like to remove features with low variance. This can be done by setting a variance threshold."
1708 | ]
1709 | },
1710 | {
1711 | "cell_type": "markdown",
1712 | "metadata": {},
1713 | "source": [
1714 | "$\\color{DarkBlue}{\\textsf{Short Question}}$\n",
1715 | " \n",
1716 | "- What is the reasoning behind doing this? \n",
1717 | "- When might it go wrong and why?"
1718 | ]
1719 | },
1720 | {
1721 | "cell_type": "markdown",
1722 | "metadata": {},
1723 | "source": [
1724 | " $\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
1725 | "- Add a variance threshold to the pipeline (select the correct function argument)\n",
1726 | "- Use random search for hyperparameter optimization, retrain the pipeline, and calculate the performance metrics (max error, MAE, MSE) on the training and test set\n",
1727 | "- If you could improve the predictive performance, do not forget to also run the model for the Kaggle competition!"
1728 | ]
1729 | },
1730 | {
1731 | "cell_type": "code",
1732 | "execution_count": null,
1733 | "metadata": {},
1734 | "outputs": [],
1735 | "source": [
1736 | "# Define the pipeline\n",
1737 | "pipe_variance_threshold = Pipeline(\n",
1738 | " # fillme with the pipeline steps\n",
1739 | " [\n",
1740 | " ('variance_treshold', VarianceThreshold(#fillme with threshold)),\n",
1741 | " #fillme with remaining pipeline steps\n",
1742 | " ]\n",
1743 | ")"
1744 | ]
1745 | },
1746 | {
1747 | "cell_type": "code",
1748 | "execution_count": null,
1749 | "metadata": {},
1750 | "outputs": [],
1751 | "source": [
1752 | "param_grid_variance_threshold = {\n",
1753 | " 'scaling': [None, StandardScaler()],\n",
1754 | " 'krr__alpha': #fillme,\n",
1755 | " 'krr__#fillme': #fillme,\n",
1756 | " 'variance_treshold__threshold': #fillme\n",
1757 | " }\n",
1758 | "\n",
1759 | "random_variance_treshold = RandomizedSearchCV(#your pipeline, param_distributions=param_grid, n_iter=#number of evaluations,\n",
1760 | " cv=#number of folds, verbose=2, n_jobs=2)"
1761 | ]
1762 | },
1763 | {
1764 | "cell_type": "code",
1765 | "execution_count": null,
1766 | "metadata": {},
1767 | "outputs": [],
1768 | "source": [
1769 | "# Fit the pipeline and run the evaluation\n",
1770 | "random_variance_treshold.fit(#fillme)"
1771 | ]
1772 | },
1773 | {
1774 | "cell_type": "markdown",
1775 | "metadata": {},
1776 | "source": [
1777 | "$\\color{DarkBlue}{\\textsf{Short Exercise (optional)}}$\n",
1778 | "- replace the variance threshold with a model-based feature selection \n",
1779 | "`('feature_selection', SelectFromModel(LinearSVC(penalty=\"l1\")))` or [any feature selection method that you would like to try](https://scikit-learn.org/stable/modules/feature_selection.html)"
1780 | ]
1781 | },
1782 | {
1783 | "cell_type": "markdown",
1784 | "metadata": {},
1785 | "source": [
1786 | "## 9. Saving the model"
1787 | ]
1788 | },
1789 | {
1790 | "cell_type": "markdown",
1791 | "metadata": {},
1792 | "source": [
1793 | "Now, that we spent so much time in optimizing our model, we do not want to loose it. "
1794 | ]
1795 | },
1796 | {
1797 | "cell_type": "markdown",
1798 | "metadata": {},
1799 | "source": [
1800 | " $\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
1801 | "- use the [joblib library](https://scikit-learn.org/stable/modules/model_persistence.html) to save your model\n",
1802 | "- make sure you can load it again\n"
1803 | ]
1804 | },
1805 | {
1806 | "cell_type": "code",
1807 | "execution_count": null,
1808 | "metadata": {},
1809 | "outputs": [],
1810 | "source": [
1811 | "# Dump your model\n",
1812 | "joblib.dump(model, filename)"
1813 | ]
1814 | },
1815 | {
1816 | "cell_type": "code",
1817 | "execution_count": null,
1818 | "metadata": {},
1819 | "outputs": [],
1820 | "source": [
1821 | "# Try to load it again\n",
1822 | "model_loaded = joblib.load(filename)"
1823 | ]
1824 | },
1825 | {
1826 | "cell_type": "markdown",
1827 | "metadata": {},
1828 | "source": [
1829 | "## 10. Influence of Regularization"
1830 | ]
1831 | },
1832 | {
1833 | "cell_type": "markdown",
1834 | "metadata": {},
1835 | "source": [
1836 | " $\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
1837 | "- what happens if you set $\\alpha$ to a really small or to large value? Why is this the case explain what the parameter means using the equation derived in the lectures?\n",
1838 | "\n",
1839 | " To test this, fix this value in one of your pipelines, retrain the models (re-optimizing the other hyperparameters) and rerun the performance evaluation."
1840 | ]
1841 | },
1842 | {
1843 | "cell_type": "markdown",
1844 | "metadata": {},
1845 | "source": [
1846 | "Click here for hyperparameter optimization with hyperopt (advanded and optional outlook)
\n", 1642 | " \n", 1643 | "Import the tools we need\n", 1644 | "\n",
1645 | "from hyperopt import fmin, tpe, hp, STATUS_OK, Trials, mix, rand, anneal, space_eval\n",
1646 | "from functools import partial\n",
1647 | " \n",
1648 | "\n",
1649 | "Define the grid\n",
1650 | "\n",
1651 | "param_hyperopt = {\n",
1652 | " \"krr__alpha\": hp.loguniform(\"krr__alpha\", np.log(0.001), np.log(10)),\n",
1653 | " \"krr__gamma\": hp.loguniform(\"krr__gamma\", np.log(0.001), np.log(10)),\n",
1654 | "}\n",
1655 | " \n",
1656 | "\n",
1657 | "Define the objective function\n",
1658 | "\n",
1659 | "def objective_function(params):\n",
1660 | " pipe.set_params(\n",
1661 | " **{\n",
1662 | " \"krr__alpha\": params[\"krr__alpha\"],\n",
1663 | " \"krr__gamma\": params[\"krr__gamma\"],\n",
1664 | " }\n",
1665 | " )\n",
1666 | " score = cross_val_score(\n",
1667 | " pipe, X_train, y_train, cv=10, scoring=\"neg_mean_absolute_error\"\n",
1668 | " ).mean()\n",
1669 | " return {\"loss\": -score, \"status\": STATUS_OK} \n",
1670 | "\n",
1671 | "\n",
1672 | "We will use a search in which we mix random search, annealing and tpe\n",
1673 | "\n",
1674 | "trials = Trials()\n",
1675 | "mix_search = partial(\n",
1676 | " mix.suggest,\n",
1677 | " p_suggest=[(0.15, rand.suggest), (0.15, anneal.suggest), (0.7, tpe.suggest)],\n",
1678 | ")\n",
1679 | "\n",
1680 | "\n",
1681 | "Now, we can minimize the objective function.\n",
1682 | "\n",
1683 | "best_param = fmin(\n",
1684 | " objective_function,\n",
1685 | " param_hyperopt,\n",
1686 | " algo=mix_search,\n",
1687 | " max_evals=MAX_EVALES,\n",
1688 | " trials=trials,\n",
1689 | " rstate=np.random.RandomState(RANDOM_SEED),\n",
1690 | " )\n",
1691 | "\n",
1692 | "\n",
1693 | "\n",
1847 | "
"
1853 | ]
1854 | },
1855 | {
1856 | "cell_type": "markdown",
1857 | "metadata": {},
1858 | "source": [
1859 | "## 11. Interpreting the model "
1860 | ]
1861 | },
1862 | {
1863 | "cell_type": "markdown",
1864 | "metadata": {},
1865 | "source": [
1866 | "Now, that our model performs decently, we would like to know which features are mainly responsible for this, i.e. how the model performs its reasoning. \n",
1867 | "\n",
1868 | "One method to do so is the [permutation feature importance technique](https://christophm.github.io/interpretable-ml-book/feature-importance.html)."
1869 | ]
1870 | },
1871 | {
1872 | "cell_type": "markdown",
1873 | "metadata": {},
1874 | "source": [
1875 | "$\\color{DarkBlue}{\\textsf{Short question}}$\n",
1876 | "\n",
1877 | "We use both descriptors that encode the pore geometry (density, pore diameters, surface areas) as well as some that describe the chemistry of the MOF (the RACs). \n",
1878 | "- Would you expect the relative importance of these features to be different for prediction of gas adsorption at high vs low gas pressure?"
1879 | ]
1880 | },
1881 | {
1882 | "cell_type": "markdown",
1883 | "metadata": {},
1884 | "source": [
1885 | "Click here for hints
\n", 1848 | "- \n",
1849 | "
- Check the derivation for ridge regression and KRR in the notes. \n", 1850 | "
- Also remember the loss landscapes we discussed in the lectures about LASSO. \n", 1851 | "
\n",
1886 | "
"
1891 | ]
1892 | },
1893 | {
1894 | "cell_type": "markdown",
1895 | "metadata": {},
1896 | "source": [
1897 | " $\\color{DarkBlue}{\\textsf{Short Exercise}}$\n",
1898 | "- Complete the function `_calculate_permutation_scores` (which we took from the `sklearn` package) and which is needed to calculate the permutation feature importance using the `permutation_importance` function. "
1899 | ]
1900 | },
1901 | {
1902 | "cell_type": "code",
1903 | "execution_count": null,
1904 | "metadata": {},
1905 | "outputs": [],
1906 | "source": [
1907 | "def _calculate_permutation_scores(estimator, X, y, col_idx, random_state,\n",
1908 | " n_repeats, scorer):\n",
1909 | " \"\"\"Calculate score when `col_idx` is permuted. Based on the sklearn implementation\n",
1910 | "\n",
1911 | " estimator: sklearn estimator object\n",
1912 | " X: pd.Dataframe or np.array\n",
1913 | " y: pd.Dataframe or np.array\n",
1914 | " col_idx: int\n",
1915 | " random_state: int\n",
1916 | " n_repeats: int\n",
1917 | " scorer: function that takes model, X and y_true as arguments\n",
1918 | " \"\"\"\n",
1919 | " random_state = check_random_state(random_state)\n",
1920 | "\n",
1921 | " X_permuted = X.copy()\n",
1922 | " scores = np.zeros(n_repeats)\n",
1923 | " # get the indices\n",
1924 | " shuffling_idx = np.arange(X.shape[0])\n",
1925 | " for n_round in range(n_repeats):\n",
1926 | " # FILL BELOW HERE\n",
1927 | " # shuffle them (fill in what you want to shuffle)\n",
1928 | " random_state.shuffle(#fillme)\n",
1929 | "\n",
1930 | " # Deal with dataframes\n",
1931 | " if hasattr(X_permuted, \"iloc\"):\n",
1932 | " # .iloc selects the indices from a dataframe and you give it [row, column]\n",
1933 | " col = X_permuted.iloc[shuffling_idx, col_idx]\n",
1934 | " col.index = X_permuted.index\n",
1935 | " X_permuted.iloc[:, col_idx] = col\n",
1936 | "\n",
1937 | " # Deal with numpy arrays\n",
1938 | " else:\n",
1939 | " # FILL BELOW HERE\n",
1940 | " # array indexing is [row, column]\n",
1941 | " X_permuted[:, col_idx] = X_permuted[#fillme]\n",
1942 | "\n",
1943 | " # Get the scores\n",
1944 | " feature_score = scorer(estimator, X_permuted, y)\n",
1945 | "\n",
1946 | " # record the scores in array\n",
1947 | " scores[n_round] = feature_score\n",
1948 | "\n",
1949 | " return scores"
1950 | ]
1951 | },
1952 | {
1953 | "cell_type": "markdown",
1954 | "metadata": {},
1955 | "source": [
1956 | "Nothing to change in the function below, it just call the `_calculate_permutation_scores` function you just completed. "
1957 | ]
1958 | },
1959 | {
1960 | "cell_type": "code",
1961 | "execution_count": null,
1962 | "metadata": {},
1963 | "outputs": [],
1964 | "source": [
1965 | "def permutation_importance(\n",
1966 | " estimator,\n",
1967 | " X,\n",
1968 | " y,\n",
1969 | " scoring=\"neg_mean_absolute_error\",\n",
1970 | " n_repeats=5,\n",
1971 | " n_jobs=2,\n",
1972 | " random_state=None,\n",
1973 | "):\n",
1974 | " \"\"\"Permutation importance for feature evaluation\n",
1975 | " estimator : object\n",
1976 | " An estimator that has already been :term:`fitted` and is compatible\n",
1977 | " with :term:`scorer`.\n",
1978 | " X : ndarray or DataFrame, shape (n_samples, n_features)\n",
1979 | " Data on which permutation importance will be computed.\n",
1980 | " y : array-like or None, shape (n_samples, ) or (n_samples, n_classes)\n",
1981 | " Targets for supervised or `None` for unsupervised.\n",
1982 | " scoring : string, callable or None, default=None\n",
1983 | " Scorer to use. It can be a single\n",
1984 | " string (see :ref:`scoring_parameter`) or a callable (see\n",
1985 | " :ref:`scoring`). If None, the estimator's default scorer is used.\n",
1986 | " n_repeats : int, default=5\n",
1987 | " Number of times to permute a feature.\n",
1988 | " n_jobs : int or None, default=2\n",
1989 | " The number of jobs to use for the computation.\n",
1990 | " `None` means 1 unless in a :obj:`joblib.parallel_backend` context.\n",
1991 | " `-1` means using all processors. See :term:`Glossary Click here for a hint
\n", 1887 | "- \n",
1888 | "
- An article from Diego et al. (10.1021/acs.chemmater.8b02257) gives some hints. \n", 1889 | "
\n",
2073 | "
"
2080 | ]
2081 | },
2082 | {
2083 | "cell_type": "markdown",
2084 | "metadata": {},
2085 | "source": [
2086 | "Click here for hints
\n", 2074 | "- \n",
2075 | "
- To get the top
n indices of an arraya, you can usenp.argsort(a)[-n:]\n",
2076 | " - Get the feature names from the
FEATURESlist \n",
2077 | " - combined this might look like
np.array(FEATURES)[np.argsort(a)[-n:]]\n",
2078 | "
\n",
2087 | "
"
2091 | ]
2092 | },
2093 | {
2094 | "cell_type": "markdown",
2095 | "metadata": {},
2096 | "source": [
2097 | "## 12. Submit your best model to Kaggle "
2098 | ]
2099 | },
2100 | {
2101 | "attachments": {},
2102 | "cell_type": "markdown",
2103 | "metadata": {},
2104 | "source": [
2105 | "Join the [Kaggle competition](https://www.kaggle.com/competitions/molsim-2024-ml-challenge/host/launch-checklist) for this course!\n",
2106 | "For this you can: \n",
2107 | "- try to continue optimizing your KRR model \n",
2108 | "- try to use a new model ([browse the sklearn documentation](https://scikit-learn.org/) for ideas or check out [xgboost](https://xgboost.readthedocs.io/en/stable/)\n",
2109 | "\n",
2110 | "The important parts for us here are: \n",
2111 | "- that you make an attempt to improve your model, discuss this attempt, and use proper models to measure potential improvement \n",
2112 | "- we will not grade you based on how \"fancy\" or model is or how well it performs but rather on whether you do something reasonable that is well motivated in your discussion\n",
2113 | "- you do not need to try both a model and continue optimizing your model. Doing one of them is, in principle, \"enough\"\n",
2114 | "\n",
2115 | "Use then your best model to create a `submission.csv` with your predictions to join the competition and upload it to the competition site.\n"
2116 | ]
2117 | },
2118 | {
2119 | "cell_type": "code",
2120 | "execution_count": null,
2121 | "metadata": {},
2122 | "outputs": [],
2123 | "source": [
2124 | "kaggle_data = pd.read_csv('data/features.csv')\n",
2125 | "kaggle_predictions = #fillme.predict(kaggle_data[FEATURES])"
2126 | ]
2127 | },
2128 | {
2129 | "cell_type": "code",
2130 | "execution_count": null,
2131 | "metadata": {},
2132 | "outputs": [],
2133 | "source": [
2134 | "submission = pd.DataFrame({\"id\": kaggle_data[\"id\"], \"prediction\": kaggle_predictions})\n",
2135 | "\n",
2136 | "submission.to_csv(\"submission.csv\", index=False)\n"
2137 | ]
2138 | }
2139 | ],
2140 | "metadata": {
2141 | "kernelspec": {
2142 | "display_name": "Python 3",
2143 | "language": "python",
2144 | "name": "python3"
2145 | },
2146 | "language_info": {
2147 | "codemirror_mode": {
2148 | "name": "ipython",
2149 | "version": 3
2150 | },
2151 | "file_extension": ".py",
2152 | "mimetype": "text/x-python",
2153 | "name": "python",
2154 | "nbconvert_exporter": "python",
2155 | "pygments_lexer": "ipython3",
2156 | "version": "3.10.13"
2157 | },
2158 | "vscode": {
2159 | "interpreter": {
2160 | "hash": "5c7b89af1651d0b8571dde13640ecdccf7d5a6204171d6ab33e7c296e100e08a"
2161 | }
2162 | }
2163 | },
2164 | "nbformat": 4,
2165 | "nbformat_minor": 4
2166 | }
2167 |
--------------------------------------------------------------------------------
/test/test.py:
--------------------------------------------------------------------------------
1 | # basics
2 | import os
3 | import numpy as np
4 |
5 | # data
6 | import pandas as pd
7 |
8 | # machine learning
9 | # scaling of data
10 | from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
11 | # train/test split
12 | from sklearn.model_selection import train_test_split
13 | # model selection
14 | from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
15 | # model
16 | from sklearn.kernel_ridge import KernelRidge
17 | # pipeline
18 | from sklearn.pipeline import Pipeline
19 | # PCA
20 | from sklearn.decomposition import PCA
21 | # Dummy model
22 | from sklearn.dummy import DummyClassifier, DummyRegressor
23 | # Variance Threshold
24 | from sklearn.feature_selection import VarianceThreshold, SelectFromModel
25 | # metrics
26 | from sklearn.metrics import (accuracy_score, precision_score, recall_score, f1_score,
27 | mean_absolute_error, mean_squared_error, max_error)
28 | # feature names
29 | #from descriptornames import *
30 |
31 | # save/load models
32 | import joblib
33 |
34 | # For the permutation importance
35 | from joblib import Parallel
36 | from joblib import delayed
37 | from sklearn.metrics import check_scoring
38 | from sklearn.utils import Bunch
39 | from sklearn.utils import check_random_state
40 | from sklearn.utils import check_array
41 |
42 | # plotting
43 | import matplotlib.pyplot as plt
44 | import seaborn as sns
45 |
46 | # for interactive plots, you can try to use holoviewes
47 | import holoviews as hv
48 | from holoviews import dim, opts
49 | hv.extension('plotly', 'bokeh', 'matplotlib')
50 |
51 | RANDOM_SEED = 4242424242
52 | DATA_DIR = 'data'
53 | DATA_FILE = os.path.join(DATA_DIR, 'data.csv')
54 |
55 | np.random.seed(RANDOM_SEED)
56 |
57 | df = pd.read_csv('data/data.csv')
58 |
59 | print(df.head())
60 |
61 |
--------------------------------------------------------------------------------