├── floyd_requirements.txt ├── floyd.yml ├── README.md ├── bayesian_optimization_example.ipynb ├── random_search_example.ipynb └── grid_search_example.ipynb /floyd_requirements.txt: -------------------------------------------------------------------------------- 1 | wrangle==0.3.1 2 | hyperas -------------------------------------------------------------------------------- /floyd.yml: -------------------------------------------------------------------------------- 1 | env: tensorflow-1.9 2 | machine: cpu 3 | input: 4 | - source: redeipirati/datasets/breast-cancer/ 5 | destination: bcds -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Hyperparameters search for deep learning models 2 | 3 | [![Run on FH](https://static.floydhub.com/button/button-small.svg)](https://floydhub.com/run) 4 | 5 | [![Run on FloydHub](https://static.floydhub.com/button/button.svg)](https://floydhub.com/run) 6 | 7 | Click this button to open a [Workspace](https://blog.floydhub.com/workspaces/) on [FloydHub](https://www.floydhub.com/?utm_medium=readme&utm_source=hyperparameters_search_examples&utm_campaign=sept_2018). You can use the workspace to test the companion examples explained in the [blog post](https://blog.floydhub.com/guide-to-hyperparameters-search-for-deep-learning-models/). -------------------------------------------------------------------------------- /bayesian_optimization_example.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Bayesian Optimization Example\n", 8 | "\n", 9 | "In this example we will perform a Bayesian Optimization using [Hyperas](https://github.com/maxpumperla/hyperas) on the [breast cancer](https://github.com/autonomio/datasets/blob/master/autonomio-datasets/breast_cancer.csv) classification task. You can run this example on CPU. It will take more or less 3 minutes.\n", 10 | "\n", 11 | "We will continue to use the same example of the [Grid Search](./grid_search_example.ipynb) and [Random Search](./random_search_example.ipynb) notebooks." 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "## Initial Setup\n", 19 | "\n", 20 | "Import the packages we need for the computation." 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 1, 26 | "metadata": {}, 27 | "outputs": [ 28 | { 29 | "name": "stderr", 30 | "output_type": "stream", 31 | "text": [ 32 | "Using TensorFlow backend.\n" 33 | ] 34 | } 35 | ], 36 | "source": [ 37 | "import os\n", 38 | "import pandas as pd\n", 39 | "import wrangle as wr\n", 40 | "\n", 41 | "from numpy import nan\n", 42 | "\n", 43 | "from sklearn.model_selection import train_test_split\n", 44 | "from keras.utils import to_categorical" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "### Load the dataset\n", 52 | "\n", 53 | "Load, clean and preprocess the dataset" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": 2, 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "def data():\n", 63 | " \"\"\"\n", 64 | " Data providing function:\n", 65 | " This function is separated from model() so that hyperopt\n", 66 | " won't reload data for each evaluation run.\n", 67 | " \"\"\"\n", 68 | " # Mounting point\n", 69 | " MP = '/floyd/input/bcds'\n", 70 | " \n", 71 | " def breast_cancer():\n", 72 | " \"\"\"Load and preprocess(cleaning) the dataset\"\"\"\n", 73 | " df = pd.read_csv(os.path.join(MP, 'breast_cancer.csv'))\n", 74 | "\n", 75 | " # then some minimal data cleanup\n", 76 | " df.drop(\"Unnamed: 32\", axis=1, inplace=True)\n", 77 | " df.drop(\"id\", axis=1, inplace=True)\n", 78 | "\n", 79 | " # separate to x and y\n", 80 | " y = df.diagnosis.values\n", 81 | " x = df.drop('diagnosis', axis=1).values\n", 82 | "\n", 83 | " # convert the string labels to binary\n", 84 | " y = (y == 'M').astype(int)\n", 85 | "\n", 86 | " return x, y\n", 87 | " \n", 88 | " # Load the dataset\n", 89 | " x, y = breast_cancer()\n", 90 | "\n", 91 | " # Normalize every feature to mean 0, std 1\n", 92 | " x = wr.mean_zero(pd.DataFrame(x)).values\n", 93 | "\n", 94 | " input_dim = x.shape[1] # number of columns\n", 95 | "\n", 96 | " # Train - Test split: 66 - 33\n", 97 | " x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33, random_state=7)\n", 98 | " return x_train, y_train, x_test, y_test" 99 | ] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "metadata": {}, 104 | "source": [ 105 | "### Model definition\n", 106 | "\n", 107 | "Define the model and the variables to search." 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": 3, 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "from keras.models import Sequential\n", 117 | "from keras.layers import Dense, Dropout, Flatten\n", 118 | "\n", 119 | "from hyperas.distributions import choice, uniform\n", 120 | "from hyperopt import Trials, STATUS_OK, tpe\n", 121 | "\n", 122 | "def model(x_train, y_train, x_test, y_test):\n", 123 | " \"\"\"\n", 124 | " Model providing function:\n", 125 | " Create Keras model with double curly brackets dropped-in as needed.\n", 126 | " Return value has to be a valid python dictionary with two customary keys:\n", 127 | " - loss: Specify a numeric evaluation metric to be minimized\n", 128 | " - status: Just use STATUS_OK and see hyperopt documentation if not feasible\n", 129 | " The last one is optional, though recommended, namely:\n", 130 | " - model: specify the model just created so that we can later use it again.\n", 131 | " \"\"\"\n", 132 | " model = Sequential()\n", 133 | " \n", 134 | " # L1\n", 135 | " model.add(Dense({{choice([8,9,10])}}, \n", 136 | " input_dim=input_dim, \n", 137 | " kernel_initializer={{choice(['uniform', 'normal'])}}, \n", 138 | " activation={{choice(['relu', 'elu'])}}))\n", 139 | " # Dropout\n", 140 | " model.add(Dropout({{uniform(0, 1)}}))\n", 141 | " # L2\n", 142 | " model.add(Dense(1, \n", 143 | " kernel_initializer={{choice(['uniform', 'normal'])}}, \n", 144 | " activation='sigmoid'))\n", 145 | " # Compile model\n", 146 | " model.compile(loss='binary_crossentropy', \n", 147 | " optimizer={{choice(['nadam', 'adam', 'sgd'])}}, \n", 148 | " metrics=['accuracy'])\n", 149 | " \n", 150 | " model.fit(x_train, y_train,\n", 151 | " batch_size=1024,\n", 152 | " epochs=10,\n", 153 | " verbose=2,\n", 154 | " validation_data=(x_test, y_test))\n", 155 | " \n", 156 | " score, acc = model.evaluate(x_test, y_test, verbose=0)\n", 157 | " print('Test accuracy:', acc)\n", 158 | " return {'loss': -acc, 'status': STATUS_OK, 'model': model}" 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": {}, 164 | "source": [ 165 | "## SMBO in action\n", 166 | "\n", 167 | "Run 5 iterations using using the [Tree Parzen Estimator or TPE algorithm](https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf) provided with [hyperopt](https://github.com/hyperopt/hyperopt)." 168 | ] 169 | }, 170 | { 171 | "cell_type": "code", 172 | "execution_count": 4, 173 | "metadata": {}, 174 | "outputs": [ 175 | { 176 | "name": "stdout", 177 | "output_type": "stream", 178 | "text": [ 179 | ">>> Imports:\n", 180 | "#coding=utf-8\n", 181 | "\n", 182 | "try:\n", 183 | " import os\n", 184 | "except:\n", 185 | " pass\n", 186 | "\n", 187 | "try:\n", 188 | " import pandas as pd\n", 189 | "except:\n", 190 | " pass\n", 191 | "\n", 192 | "try:\n", 193 | " import wrangle as wr\n", 194 | "except:\n", 195 | " pass\n", 196 | "\n", 197 | "try:\n", 198 | " from numpy import nan\n", 199 | "except:\n", 200 | " pass\n", 201 | "\n", 202 | "try:\n", 203 | " from sklearn.model_selection import train_test_split\n", 204 | "except:\n", 205 | " pass\n", 206 | "\n", 207 | "try:\n", 208 | " from keras.utils import to_categorical\n", 209 | "except:\n", 210 | " pass\n", 211 | "\n", 212 | "try:\n", 213 | " from keras.models import Sequential\n", 214 | "except:\n", 215 | " pass\n", 216 | "\n", 217 | "try:\n", 218 | " from keras.layers import Dense, Dropout, Flatten\n", 219 | "except:\n", 220 | " pass\n", 221 | "\n", 222 | "try:\n", 223 | " from hyperas.distributions import choice, uniform\n", 224 | "except:\n", 225 | " pass\n", 226 | "\n", 227 | "try:\n", 228 | " from hyperopt import Trials, STATUS_OK, tpe\n", 229 | "except:\n", 230 | " pass\n", 231 | "\n", 232 | "try:\n", 233 | " from hyperas import optim\n", 234 | "except:\n", 235 | " pass\n", 236 | "\n", 237 | ">>> Hyperas search space:\n", 238 | "\n", 239 | "def get_space():\n", 240 | " return {\n", 241 | " 'Dense': hp.choice('Dense', [8,9,10]),\n", 242 | " 'kernel_initializer': hp.choice('kernel_initializer', ['uniform', 'normal']),\n", 243 | " 'activation': hp.choice('activation', ['relu', 'elu']),\n", 244 | " 'Dropout': hp.uniform('Dropout', 0, 1),\n", 245 | " 'kernel_initializer_1': hp.choice('kernel_initializer_1', ['uniform', 'normal']),\n", 246 | " 'optimizer': hp.choice('optimizer', ['nadam', 'adam', 'sgd']),\n", 247 | " }\n", 248 | "\n", 249 | ">>> Data\n", 250 | " 1: \n", 251 | " 2: \"\"\"\n", 252 | " 3: Data providing function:\n", 253 | " 4: This function is separated from model() so that hyperopt\n", 254 | " 5: won't reload data for each evaluation run.\n", 255 | " 6: \"\"\"\n", 256 | " 7: # Mounting point\n", 257 | " 8: MP = '/floyd/input/bcds'\n", 258 | " 9: \n", 259 | " 10: def breast_cancer():\n", 260 | " 11: \"\"\"Download and preprocess(cleaning) the dataset\"\"\"\n", 261 | " 12: df = pd.read_csv(os.path.join(MP, 'breast_cancer.csv'))\n", 262 | " 13: \n", 263 | " 14: # then some minimal data cleanup\n", 264 | " 15: df.drop(\"Unnamed: 32\", axis=1, inplace=True)\n", 265 | " 16: df.drop(\"id\", axis=1, inplace=True)\n", 266 | " 17: \n", 267 | " 18: # separate to x and y\n", 268 | " 19: y = df.diagnosis.values\n", 269 | " 20: x = df.drop('diagnosis', axis=1).values\n", 270 | " 21: \n", 271 | " 22: # convert the string labels to binary\n", 272 | " 23: y = (y == 'M').astype(int)\n", 273 | " 24: \n", 274 | " 25: return x, y\n", 275 | " 26: \n", 276 | " 27: # Load the dataset\n", 277 | " 28: x, y = breast_cancer()\n", 278 | " 29: \n", 279 | " 30: # Normalize every feature to mean 0, std 1\n", 280 | " 31: x = wr.mean_zero(pd.DataFrame(x)).values\n", 281 | " 32: \n", 282 | " 33: input_dim = x.shape[1] # number of columns\n", 283 | " 34: \n", 284 | " 35: # Train - Test split: 66 - 33\n", 285 | " 36: x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33, random_state=7)\n", 286 | " 37: \n", 287 | " 38: \n", 288 | " 39: \n", 289 | ">>> Resulting replaced keras model:\n", 290 | "\n", 291 | " 1: def keras_fmin_fnct(space):\n", 292 | " 2: \n", 293 | " 3: \"\"\"\n", 294 | " 4: Model providing function:\n", 295 | " 5: Create Keras model with double curly brackets dropped-in as needed.\n", 296 | " 6: Return value has to be a valid python dictionary with two customary keys:\n", 297 | " 7: - loss: Specify a numeric evaluation metric to be minimized\n", 298 | " 8: - status: Just use STATUS_OK and see hyperopt documentation if not feasible\n", 299 | " 9: The last one is optional, though recommended, namely:\n", 300 | " 10: - model: specify the model just created so that we can later use it again.\n", 301 | " 11: \"\"\"\n", 302 | " 12: model = Sequential()\n", 303 | " 13: \n", 304 | " 14: # L1\n", 305 | " 15: model.add(Dense(space['Dense'], \n", 306 | " 16: input_dim=input_dim, \n", 307 | " 17: kernel_initializer=space['kernel_initializer'], \n", 308 | " 18: activation=space['activation']))\n", 309 | " 19: # Dropout\n", 310 | " 20: model.add(Dropout(space['Dropout']))\n", 311 | " 21: # L2\n", 312 | " 22: model.add(Dense(1, \n", 313 | " 23: kernel_initializer=space['kernel_initializer_1'], \n", 314 | " 24: activation='sigmoid'))\n", 315 | " 25: # Compile model\n", 316 | " 26: model.compile(loss='binary_crossentropy', \n", 317 | " 27: optimizer=space['optimizer'], \n", 318 | " 28: metrics=['accuracy'])\n", 319 | " 29: \n", 320 | " 30: model.fit(x_train, y_train,\n", 321 | " 31: batch_size=1024,\n", 322 | " 32: epochs=10,\n", 323 | " 33: verbose=2,\n", 324 | " 34: validation_data=(x_test, y_test))\n", 325 | " 35: \n", 326 | " 36: score, acc = model.evaluate(x_test, y_test, verbose=0)\n", 327 | " 37: print('Test accuracy:', acc)\n", 328 | " 38: return {'loss': -acc, 'status': STATUS_OK, 'model': model}\n", 329 | " 39: \n", 330 | "Train on 381 samples, validate on 188 samples\n", 331 | "Epoch 1/10\n", 332 | " - 0s - loss: 0.6865 - acc: 0.5906 - val_loss: 0.6899 - val_acc: 0.6383\n", 333 | "Epoch 2/10\n", 334 | " - 0s - loss: 0.6883 - acc: 0.5722 - val_loss: 0.6895 - val_acc: 0.6755\n", 335 | "Epoch 3/10\n", 336 | " - 0s - loss: 0.6871 - acc: 0.6352 - val_loss: 0.6890 - val_acc: 0.6968\n", 337 | "Epoch 4/10\n", 338 | " - 0s - loss: 0.6884 - acc: 0.6142 - val_loss: 0.6885 - val_acc: 0.7021\n", 339 | "Epoch 5/10\n", 340 | " - 0s - loss: 0.6875 - acc: 0.6115 - val_loss: 0.6880 - val_acc: 0.7553\n", 341 | "Epoch 6/10\n", 342 | " - 0s - loss: 0.6873 - acc: 0.6378 - val_loss: 0.6876 - val_acc: 0.8085\n", 343 | "Epoch 7/10\n", 344 | " - 0s - loss: 0.6845 - acc: 0.6667 - val_loss: 0.6871 - val_acc: 0.8191\n", 345 | "Epoch 8/10\n", 346 | " - 0s - loss: 0.6847 - acc: 0.6352 - val_loss: 0.6866 - val_acc: 0.8511\n", 347 | "Epoch 9/10\n", 348 | " - 0s - loss: 0.6835 - acc: 0.7060 - val_loss: 0.6862 - val_acc: 0.8723\n", 349 | "Epoch 10/10\n", 350 | " - 0s - loss: 0.6845 - acc: 0.6772 - val_loss: 0.6857 - val_acc: 0.8723\n", 351 | "Test accuracy: 0.8723404217273631\n", 352 | "Train on 381 samples, validate on 188 samples\n", 353 | "Epoch 1/10\n", 354 | " - 0s - loss: 0.6989 - acc: 0.3858 - val_loss: 0.6985 - val_acc: 0.2766\n", 355 | "Epoch 2/10\n", 356 | " - 0s - loss: 0.6980 - acc: 0.4541 - val_loss: 0.6964 - val_acc: 0.3138\n", 357 | "Epoch 3/10\n", 358 | " - 0s - loss: 0.6948 - acc: 0.4777 - val_loss: 0.6944 - val_acc: 0.3883\n", 359 | "Epoch 4/10\n", 360 | " - 0s - loss: 0.6913 - acc: 0.5354 - val_loss: 0.6924 - val_acc: 0.5000\n", 361 | "Epoch 5/10\n", 362 | " - 0s - loss: 0.6898 - acc: 0.5302 - val_loss: 0.6904 - val_acc: 0.6383\n", 363 | "Epoch 6/10\n", 364 | " - 0s - loss: 0.6903 - acc: 0.5092 - val_loss: 0.6883 - val_acc: 0.7340\n", 365 | "Epoch 7/10\n", 366 | " - 0s - loss: 0.6871 - acc: 0.5748 - val_loss: 0.6861 - val_acc: 0.8723\n", 367 | "Epoch 8/10\n", 368 | " - 0s - loss: 0.6827 - acc: 0.6115 - val_loss: 0.6836 - val_acc: 0.9096\n", 369 | "Epoch 9/10\n", 370 | " - 0s - loss: 0.6841 - acc: 0.6142 - val_loss: 0.6813 - val_acc: 0.9202\n", 371 | "Epoch 10/10\n", 372 | " - 0s - loss: 0.6783 - acc: 0.6693 - val_loss: 0.6786 - val_acc: 0.9309\n", 373 | "Test accuracy: 0.9308510663661551\n", 374 | "Train on 381 samples, validate on 188 samples\n", 375 | "Epoch 1/10\n", 376 | " - 0s - loss: 0.6939 - acc: 0.5591 - val_loss: 0.6927 - val_acc: 0.6223\n", 377 | "Epoch 2/10\n", 378 | " - 0s - loss: 0.6911 - acc: 0.5722 - val_loss: 0.6919 - val_acc: 0.7021\n", 379 | "Epoch 3/10\n", 380 | " - 0s - loss: 0.6900 - acc: 0.5984 - val_loss: 0.6911 - val_acc: 0.7713\n", 381 | "Epoch 4/10\n", 382 | " - 0s - loss: 0.6915 - acc: 0.5958 - val_loss: 0.6904 - val_acc: 0.7872\n", 383 | "Epoch 5/10\n", 384 | " - 0s - loss: 0.6902 - acc: 0.5906 - val_loss: 0.6896 - val_acc: 0.8085\n", 385 | "Epoch 6/10\n", 386 | " - 0s - loss: 0.6899 - acc: 0.6168 - val_loss: 0.6888 - val_acc: 0.8351\n", 387 | "Epoch 7/10\n", 388 | " - 0s - loss: 0.6883 - acc: 0.6115 - val_loss: 0.6880 - val_acc: 0.8404\n", 389 | "Epoch 8/10\n", 390 | " - 0s - loss: 0.6895 - acc: 0.6089 - val_loss: 0.6872 - val_acc: 0.8564\n", 391 | "Epoch 9/10\n", 392 | " - 0s - loss: 0.6862 - acc: 0.6194 - val_loss: 0.6864 - val_acc: 0.8564\n", 393 | "Epoch 10/10\n", 394 | " - 0s - loss: 0.6856 - acc: 0.6247 - val_loss: 0.6856 - val_acc: 0.8617\n", 395 | "Test accuracy: 0.8617021301959423\n", 396 | "Train on 381 samples, validate on 188 samples\n", 397 | "Epoch 1/10\n", 398 | " - 0s - loss: 0.6838 - acc: 0.6430 - val_loss: 0.6787 - val_acc: 0.7819\n", 399 | "Epoch 2/10\n", 400 | " - 0s - loss: 0.6852 - acc: 0.6352 - val_loss: 0.6759 - val_acc: 0.8245\n", 401 | "Epoch 3/10\n", 402 | " - 0s - loss: 0.6782 - acc: 0.6850 - val_loss: 0.6729 - val_acc: 0.8404\n", 403 | "Epoch 4/10\n", 404 | " - 0s - loss: 0.6749 - acc: 0.6877 - val_loss: 0.6700 - val_acc: 0.8617\n", 405 | "Epoch 5/10\n", 406 | " - 0s - loss: 0.6668 - acc: 0.7323 - val_loss: 0.6670 - val_acc: 0.8670\n", 407 | "Epoch 6/10\n", 408 | " - 0s - loss: 0.6691 - acc: 0.7244 - val_loss: 0.6639 - val_acc: 0.8883\n", 409 | "Epoch 7/10\n", 410 | " - 0s - loss: 0.6681 - acc: 0.7297 - val_loss: 0.6608 - val_acc: 0.8936\n", 411 | "Epoch 8/10\n", 412 | " - 0s - loss: 0.6640 - acc: 0.7822 - val_loss: 0.6577 - val_acc: 0.8989\n", 413 | "Epoch 9/10\n", 414 | " - 0s - loss: 0.6616 - acc: 0.7349 - val_loss: 0.6545 - val_acc: 0.9043\n", 415 | "Epoch 10/10\n", 416 | " - 0s - loss: 0.6633 - acc: 0.7533 - val_loss: 0.6513 - val_acc: 0.9043\n", 417 | "Test accuracy: 0.904255321685304\n", 418 | "Train on 381 samples, validate on 188 samples\n", 419 | "Epoch 1/10\n", 420 | " - 0s - loss: 0.6913 - acc: 0.6404 - val_loss: 0.6856 - val_acc: 0.7766\n", 421 | "Epoch 2/10\n", 422 | " - 0s - loss: 0.6858 - acc: 0.7769 - val_loss: 0.6813 - val_acc: 0.8404\n", 423 | "Epoch 3/10\n", 424 | " - 0s - loss: 0.6817 - acc: 0.8268 - val_loss: 0.6771 - val_acc: 0.8723\n", 425 | "Epoch 4/10\n", 426 | " - 0s - loss: 0.6774 - acc: 0.8793 - val_loss: 0.6728 - val_acc: 0.8936\n", 427 | "Epoch 5/10\n", 428 | " - 0s - loss: 0.6731 - acc: 0.8819 - val_loss: 0.6682 - val_acc: 0.8936\n", 429 | "Epoch 6/10\n", 430 | " - 0s - loss: 0.6683 - acc: 0.8898 - val_loss: 0.6632 - val_acc: 0.8989\n", 431 | "Epoch 7/10\n", 432 | " - 0s - loss: 0.6635 - acc: 0.8950 - val_loss: 0.6579 - val_acc: 0.9043\n", 433 | "Epoch 8/10\n", 434 | " - 0s - loss: 0.6583 - acc: 0.8976 - val_loss: 0.6522 - val_acc: 0.9043\n", 435 | "Epoch 9/10\n", 436 | " - 0s - loss: 0.6524 - acc: 0.9108 - val_loss: 0.6462 - val_acc: 0.9096\n", 437 | "Epoch 10/10\n", 438 | " - 0s - loss: 0.6463 - acc: 0.9055 - val_loss: 0.6399 - val_acc: 0.9096\n", 439 | "Test accuracy: 0.9095744706214742\n" 440 | ] 441 | } 442 | ], 443 | "source": [ 444 | "from hyperas import optim\n", 445 | "\n", 446 | "best_run, best_model = optim.minimize(model=model,\n", 447 | " data=data,\n", 448 | " algo=tpe.suggest,\n", 449 | " max_evals=5,\n", 450 | " trials=Trials(),\n", 451 | " notebook_name='bayesian_optimization_example')" 452 | ] 453 | }, 454 | { 455 | "cell_type": "markdown", 456 | "metadata": {}, 457 | "source": [ 458 | "### Results\n", 459 | "\n", 460 | "Let's see which configuration give us the best performance." 461 | ] 462 | }, 463 | { 464 | "cell_type": "code", 465 | "execution_count": 5, 466 | "metadata": {}, 467 | "outputs": [ 468 | { 469 | "name": "stdout", 470 | "output_type": "stream", 471 | "text": [ 472 | "Evalutation of best performing model:\n", 473 | "188/188 [==============================] - 0s 32us/step\n", 474 | "[0.6786171705164807, 0.9308510663661551]\n", 475 | "Best performing model chosen hyper-parameters:\n", 476 | "{'Dense': 2, 'Dropout': 0.8366666847115819, 'activation': 0, 'kernel_initializer': 1, 'kernel_initializer_1': 0, 'optimizer': 0}\n" 477 | ] 478 | } 479 | ], 480 | "source": [ 481 | "x_train, y_train, x_test, y_test = data()\n", 482 | "print(\"Evalutation of best performing model:\")\n", 483 | "print(best_model.evaluate(x_test, y_test))\n", 484 | "print(\"Best performing model chosen hyper-parameters:\")\n", 485 | "print(best_run)" 486 | ] 487 | }, 488 | { 489 | "cell_type": "markdown", 490 | "metadata": {}, 491 | "source": [ 492 | "**That's all folks - don't forget to shutdown your workspace once you're done 🙂**" 493 | ] 494 | } 495 | ], 496 | "metadata": { 497 | "kernelspec": { 498 | "display_name": "Python 2", 499 | "language": "python", 500 | "name": "python2" 501 | }, 502 | "language_info": { 503 | "codemirror_mode": { 504 | "name": "ipython", 505 | "version": 2 506 | }, 507 | "file_extension": ".py", 508 | "mimetype": "text/x-python", 509 | "name": "python", 510 | "nbconvert_exporter": "python", 511 | "pygments_lexer": "ipython2", 512 | "version": "2.7.10" 513 | } 514 | }, 515 | "nbformat": 4, 516 | "nbformat_minor": 2 517 | } 518 | -------------------------------------------------------------------------------- /random_search_example.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Random Search Example\n", 8 | "\n", 9 | "In this example we will perform a RandomSearch using Scikit-learn and Keras on the [breast cancer](https://github.com/autonomio/datasets/blob/master/autonomio-datasets/breast_cancer.csv) classification task. You can run this example on CPU. It will take more or less 5 minutes.\n", 10 | "\n", 11 | "We will continue to use the same example of the [Grid Search](./grid_search_example.ipynb) notebook, but we will search for a different subset of hyperparameters." 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "## Initial Setup\n", 19 | "\n", 20 | "Import the packages we need for the computation." 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 1, 26 | "metadata": {}, 27 | "outputs": [ 28 | { 29 | "name": "stderr", 30 | "output_type": "stream", 31 | "text": [ 32 | "Using TensorFlow backend.\n" 33 | ] 34 | } 35 | ], 36 | "source": [ 37 | "import os\n", 38 | "import pandas as pd\n", 39 | "import wrangle as wr\n", 40 | "\n", 41 | "from numpy import nan\n", 42 | "\n", 43 | "from scipy.stats import randint as sp_randint\n", 44 | "from scipy.stats import uniform\n", 45 | "\n", 46 | "from keras.utils import to_categorical\n", 47 | "from keras.wrappers.scikit_learn import KerasClassifier\n", 48 | "\n", 49 | "# Mounting point\n", 50 | "MP = '/floyd/input/bcds'" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "### Load the dataset\n", 58 | "\n", 59 | "Load, clean and preprocess the dataset" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 2, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "def breast_cancer():\n", 69 | " '''Load and preprocess(cleaning) the dataset'''\n", 70 | " df = pd.read_csv(os.path.join(MP, 'breast_cancer.csv'))\n", 71 | " \n", 72 | " # then some minimal data cleanup\n", 73 | " df.drop(\"Unnamed: 32\", axis=1, inplace=True)\n", 74 | " df.drop(\"id\", axis=1, inplace=True)\n", 75 | "\n", 76 | " # separate to x and y\n", 77 | " y = df.diagnosis.values\n", 78 | " x = df.drop('diagnosis', axis=1).values\n", 79 | "\n", 80 | " # convert the string labels to binary\n", 81 | " y = (y == 'M').astype(int)\n", 82 | "\n", 83 | " return x, y" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 3, 89 | "metadata": {}, 90 | "outputs": [], 91 | "source": [ 92 | "# Load the dataset\n", 93 | "x, y = breast_cancer()\n", 94 | "\n", 95 | "# Normalize every feature to mean 0, std 1\n", 96 | "x = wr.mean_zero(pd.DataFrame(x)).values\n", 97 | "\n", 98 | "input_dim = x.shape[1] # number of columns" 99 | ] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "metadata": {}, 104 | "source": [ 105 | "### Model definition\n", 106 | "\n", 107 | "Define the model and the variables to search." 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": 4, 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "from keras.models import Sequential\n", 117 | "from keras.layers import Dense, Dropout, Flatten\n", 118 | "from keras import optimizers\n", 119 | "\n", 120 | "# Function to create model, required for KerasClassifier\n", 121 | "def create_model(first_neuron=9,\n", 122 | " activation='relu',\n", 123 | " kernel_initializer='uniform',\n", 124 | " dropout_rate=0,\n", 125 | " optimizer='adam'):\n", 126 | " \n", 127 | " # Create model\n", 128 | " model = Sequential()\n", 129 | " # L1\n", 130 | " model.add(Dense(first_neuron, \n", 131 | " input_dim=input_dim, \n", 132 | " kernel_initializer=kernel_initializer, \n", 133 | " activation=activation))\n", 134 | " # Dropout\n", 135 | " model.add(Dropout(dropout_rate))\n", 136 | " # L2\n", 137 | " model.add(Dense(1, kernel_initializer=kernel_initializer, activation='sigmoid'))\n", 138 | " \n", 139 | " # Compile model\n", 140 | " model.compile(loss='binary_crossentropy', \n", 141 | " optimizer=optimizer, \n", 142 | " metrics=['accuracy'])\n", 143 | " return model" 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": 5, 149 | "metadata": {}, 150 | "outputs": [], 151 | "source": [ 152 | "# Create the model\n", 153 | "model = KerasClassifier(build_fn=create_model) " 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "### Range of Values - The Grid\n", 161 | "\n", 162 | "Defining the parameter space boundaries." 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": 6, 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "# Specify parameters and distributions to sample from\n", 172 | "\n", 173 | "# Model Design Components\n", 174 | "first_neurons = sp_randint(8, 15)\n", 175 | "activation = ['relu', 'elu'] # You can also try 'tanh', 'sigmoid', 'hard_sigmoid', 'linear'\n", 176 | "kernel_initializer = ['uniform', 'normal'] # You can also try lecun_uniform', 'zero', 'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform'\n", 177 | "optimizer= ['adam', 'nadam', 'sgd']\n", 178 | "\n", 179 | "# Hyperparameters\n", 180 | "epochs = [10] # You can also try 20, 30, 40, etc...\n", 181 | "batch_size = [1024] # You can also try 2, 4, 8, 16, 32, 64, 128 etc...\n", 182 | "dropout_rate = uniform(0.0, 1) # Sample a value of dropout in the range [0, 0.5]" 183 | ] 184 | }, 185 | { 186 | "cell_type": "code", 187 | "execution_count": 7, 188 | "metadata": {}, 189 | "outputs": [], 190 | "source": [ 191 | "# Prepare for the Search\n", 192 | "param_dist = dict(epochs=epochs, \n", 193 | " batch_size=batch_size, \n", 194 | " optimizer=optimizer,\n", 195 | " dropout_rate=dropout_rate,\n", 196 | " activation=activation,\n", 197 | " kernel_initializer=kernel_initializer,\n", 198 | " first_neuron=first_neurons)" 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": {}, 204 | "source": [ 205 | "### Grid Search\n", 206 | "\n", 207 | "Run the search: 3 folds cross-validation (`cv=3`) on a single process (`n_jobs=1`)." 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": 8, 213 | "metadata": {}, 214 | "outputs": [ 215 | { 216 | "name": "stdout", 217 | "output_type": "stream", 218 | "text": [ 219 | "Fitting 3 folds for each of 8 candidates, totalling 24 fits\n", 220 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.6010381533043048, epochs=10, first_neuron=10, kernel_initializer=normal, optimizer=adam \n", 221 | "Epoch 1/10\n", 222 | "379/379 [==============================] - 0s 877us/step - loss: 0.6857 - acc: 0.6385\n", 223 | "Epoch 2/10\n", 224 | "379/379 [==============================] - 0s 11us/step - loss: 0.6849 - acc: 0.6016\n", 225 | "Epoch 3/10\n", 226 | "379/379 [==============================] - 0s 6us/step - loss: 0.6776 - acc: 0.7098\n", 227 | "Epoch 4/10\n", 228 | "379/379 [==============================] - 0s 7us/step - loss: 0.6761 - acc: 0.7230\n", 229 | "Epoch 5/10\n", 230 | "379/379 [==============================] - 0s 8us/step - loss: 0.6727 - acc: 0.7388\n", 231 | "Epoch 6/10\n", 232 | "379/379 [==============================] - 0s 7us/step - loss: 0.6697 - acc: 0.8047\n", 233 | "Epoch 7/10\n", 234 | "379/379 [==============================] - 0s 7us/step - loss: 0.6667 - acc: 0.8047\n", 235 | "Epoch 8/10\n", 236 | "379/379 [==============================] - 0s 8us/step - loss: 0.6629 - acc: 0.8338\n", 237 | "Epoch 9/10\n", 238 | "379/379 [==============================] - 0s 7us/step - loss: 0.6543 - acc: 0.8734\n", 239 | "Epoch 10/10\n", 240 | "379/379 [==============================] - 0s 7us/step - loss: 0.6553 - acc: 0.8628\n", 241 | "190/190 [==============================] - 0s 213us/step\n", 242 | "379/379 [==============================] - 0s 3us/step\n", 243 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.6010381533043048, epochs=10, first_neuron=10, kernel_initializer=normal, optimizer=adam, total= 1.0s\n", 244 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.6010381533043048, epochs=10, first_neuron=10, kernel_initializer=normal, optimizer=adam \n" 245 | ] 246 | }, 247 | { 248 | "name": "stderr", 249 | "output_type": "stream", 250 | "text": [ 251 | "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 1.0s remaining: 0.0s\n" 252 | ] 253 | }, 254 | { 255 | "name": "stdout", 256 | "output_type": "stream", 257 | "text": [ 258 | "Epoch 1/10\n", 259 | "379/379 [==============================] - 0s 650us/step - loss: 0.7021 - acc: 0.3773\n", 260 | "Epoch 2/10\n", 261 | "379/379 [==============================] - 0s 12us/step - loss: 0.6987 - acc: 0.4327\n", 262 | "Epoch 3/10\n", 263 | "379/379 [==============================] - 0s 11us/step - loss: 0.6939 - acc: 0.4538\n", 264 | "Epoch 4/10\n", 265 | "379/379 [==============================] - 0s 13us/step - loss: 0.6944 - acc: 0.5145\n", 266 | "Epoch 5/10\n", 267 | "379/379 [==============================] - 0s 5us/step - loss: 0.6879 - acc: 0.6095\n", 268 | "Epoch 6/10\n", 269 | "379/379 [==============================] - 0s 11us/step - loss: 0.6848 - acc: 0.6253\n", 270 | "Epoch 7/10\n", 271 | "379/379 [==============================] - 0s 5us/step - loss: 0.6844 - acc: 0.6412\n", 272 | "Epoch 8/10\n", 273 | "379/379 [==============================] - 0s 4us/step - loss: 0.6787 - acc: 0.6992\n", 274 | "Epoch 9/10\n", 275 | "379/379 [==============================] - 0s 6us/step - loss: 0.6726 - acc: 0.7573\n", 276 | "Epoch 10/10\n", 277 | "379/379 [==============================] - 0s 11us/step - loss: 0.6733 - acc: 0.7599\n", 278 | "190/190 [==============================] - 0s 218us/step\n", 279 | "379/379 [==============================] - 0s 10us/step\n", 280 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.6010381533043048, epochs=10, first_neuron=10, kernel_initializer=normal, optimizer=adam, total= 0.8s\n", 281 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.6010381533043048, epochs=10, first_neuron=10, kernel_initializer=normal, optimizer=adam \n", 282 | "Epoch 1/10\n", 283 | "380/380 [==============================] - 0s 654us/step - loss: 0.6965 - acc: 0.4316\n", 284 | "Epoch 2/10\n", 285 | "380/380 [==============================] - 0s 12us/step - loss: 0.6945 - acc: 0.4789\n", 286 | "Epoch 3/10\n", 287 | "380/380 [==============================] - 0s 6us/step - loss: 0.6919 - acc: 0.5342\n", 288 | "Epoch 4/10\n", 289 | "380/380 [==============================] - 0s 6us/step - loss: 0.6909 - acc: 0.5500\n", 290 | "Epoch 5/10\n", 291 | "380/380 [==============================] - 0s 6us/step - loss: 0.6876 - acc: 0.6316\n", 292 | "Epoch 6/10\n", 293 | "380/380 [==============================] - 0s 5us/step - loss: 0.6856 - acc: 0.6895\n", 294 | "Epoch 7/10\n", 295 | "380/380 [==============================] - 0s 6us/step - loss: 0.6830 - acc: 0.7289\n", 296 | "Epoch 8/10\n", 297 | "380/380 [==============================] - 0s 8us/step - loss: 0.6820 - acc: 0.7395\n", 298 | "Epoch 9/10\n", 299 | "380/380 [==============================] - 0s 7us/step - loss: 0.6778 - acc: 0.8132\n", 300 | "Epoch 10/10\n", 301 | "380/380 [==============================] - 0s 5us/step - loss: 0.6755 - acc: 0.7921\n", 302 | "189/189 [==============================] - 0s 303us/step\n", 303 | "380/380 [==============================] - 0s 3us/step\n", 304 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.6010381533043048, epochs=10, first_neuron=10, kernel_initializer=normal, optimizer=adam, total= 0.9s\n", 305 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.39983996327340543, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=adam \n", 306 | "Epoch 1/10\n", 307 | "379/379 [==============================] - 0s 718us/step - loss: 0.6912 - acc: 0.7309\n", 308 | "Epoch 2/10\n", 309 | "379/379 [==============================] - 0s 11us/step - loss: 0.6900 - acc: 0.7678\n", 310 | "Epoch 3/10\n", 311 | "379/379 [==============================] - 0s 6us/step - loss: 0.6891 - acc: 0.7678\n", 312 | "Epoch 4/10\n", 313 | "379/379 [==============================] - 0s 7us/step - loss: 0.6883 - acc: 0.7652\n", 314 | "Epoch 5/10\n", 315 | "379/379 [==============================] - 0s 7us/step - loss: 0.6869 - acc: 0.7573\n", 316 | "Epoch 6/10\n", 317 | "379/379 [==============================] - 0s 7us/step - loss: 0.6853 - acc: 0.7836\n", 318 | "Epoch 7/10\n", 319 | "379/379 [==============================] - 0s 6us/step - loss: 0.6848 - acc: 0.7784\n", 320 | "Epoch 8/10\n", 321 | "379/379 [==============================] - 0s 8us/step - loss: 0.6833 - acc: 0.7731\n", 322 | "Epoch 9/10\n", 323 | "379/379 [==============================] - 0s 7us/step - loss: 0.6820 - acc: 0.7599\n", 324 | "Epoch 10/10\n", 325 | "379/379 [==============================] - 0s 7us/step - loss: 0.6809 - acc: 0.7757\n", 326 | "190/190 [==============================] - 0s 310us/step\n", 327 | "379/379 [==============================] - 0s 10us/step\n", 328 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.39983996327340543, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=adam, total= 0.8s\n", 329 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.39983996327340543, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=adam \n", 330 | "Epoch 1/10\n", 331 | "379/379 [==============================] - 0s 770us/step - loss: 0.6929 - acc: 0.5409\n", 332 | "Epoch 2/10\n", 333 | "379/379 [==============================] - 0s 4us/step - loss: 0.6917 - acc: 0.6412\n", 334 | "Epoch 3/10\n", 335 | "379/379 [==============================] - 0s 5us/step - loss: 0.6902 - acc: 0.7256\n", 336 | "Epoch 4/10\n", 337 | "379/379 [==============================] - 0s 5us/step - loss: 0.6890 - acc: 0.7520\n", 338 | "Epoch 5/10\n", 339 | "379/379 [==============================] - 0s 5us/step - loss: 0.6882 - acc: 0.7889\n", 340 | "Epoch 6/10\n", 341 | "379/379 [==============================] - 0s 5us/step - loss: 0.6868 - acc: 0.8021\n", 342 | "Epoch 7/10\n", 343 | "379/379 [==============================] - 0s 6us/step - loss: 0.6865 - acc: 0.8285\n", 344 | "Epoch 8/10\n", 345 | "379/379 [==============================] - 0s 6us/step - loss: 0.6853 - acc: 0.8575\n", 346 | "Epoch 9/10\n", 347 | "379/379 [==============================] - 0s 6us/step - loss: 0.6849 - acc: 0.8443\n", 348 | "Epoch 10/10\n", 349 | "379/379 [==============================] - 0s 6us/step - loss: 0.6825 - acc: 0.8522\n", 350 | "190/190 [==============================] - 0s 374us/step\n", 351 | "379/379 [==============================] - 0s 3us/step\n", 352 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.39983996327340543, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=adam, total= 1.0s\n", 353 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.39983996327340543, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=adam \n", 354 | "Epoch 1/10\n", 355 | "380/380 [==============================] - 0s 830us/step - loss: 0.6936 - acc: 0.4579\n", 356 | "Epoch 2/10\n", 357 | "380/380 [==============================] - 0s 5us/step - loss: 0.6918 - acc: 0.5421\n", 358 | "Epoch 3/10\n", 359 | "380/380 [==============================] - 0s 5us/step - loss: 0.6908 - acc: 0.6474\n", 360 | "Epoch 4/10\n", 361 | "380/380 [==============================] - 0s 5us/step - loss: 0.6897 - acc: 0.7105\n", 362 | "Epoch 5/10\n", 363 | "380/380 [==============================] - 0s 5us/step - loss: 0.6888 - acc: 0.7395\n", 364 | "Epoch 6/10\n", 365 | "380/380 [==============================] - 0s 5us/step - loss: 0.6880 - acc: 0.7789\n", 366 | "Epoch 7/10\n", 367 | "380/380 [==============================] - 0s 5us/step - loss: 0.6866 - acc: 0.8053\n", 368 | "Epoch 8/10\n", 369 | "380/380 [==============================] - 0s 5us/step - loss: 0.6858 - acc: 0.8184\n", 370 | "Epoch 9/10\n", 371 | "380/380 [==============================] - 0s 5us/step - loss: 0.6851 - acc: 0.8184\n", 372 | "Epoch 10/10\n", 373 | "380/380 [==============================] - 0s 5us/step - loss: 0.6840 - acc: 0.8184\n", 374 | "189/189 [==============================] - 0s 413us/step\n", 375 | "380/380 [==============================] - 0s 11us/step\n", 376 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.39983996327340543, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=adam, total= 0.9s\n", 377 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.7129907024046704, epochs=10, first_neuron=13, kernel_initializer=normal, optimizer=adam \n", 378 | "Epoch 1/10\n", 379 | "379/379 [==============================] - 0s 895us/step - loss: 0.6901 - acc: 0.5805\n", 380 | "Epoch 2/10\n", 381 | "379/379 [==============================] - 0s 10us/step - loss: 0.6877 - acc: 0.6464\n", 382 | "Epoch 3/10\n", 383 | "379/379 [==============================] - 0s 6us/step - loss: 0.6857 - acc: 0.7018\n", 384 | "Epoch 4/10\n", 385 | "379/379 [==============================] - 0s 5us/step - loss: 0.6847 - acc: 0.7098\n", 386 | "Epoch 5/10\n", 387 | "379/379 [==============================] - 0s 5us/step - loss: 0.6811 - acc: 0.7361\n", 388 | "Epoch 6/10\n", 389 | "379/379 [==============================] - 0s 7us/step - loss: 0.6776 - acc: 0.7731\n", 390 | "Epoch 7/10\n", 391 | "379/379 [==============================] - 0s 5us/step - loss: 0.6732 - acc: 0.7784\n", 392 | "Epoch 8/10\n", 393 | "379/379 [==============================] - 0s 11us/step - loss: 0.6732 - acc: 0.7836\n", 394 | "Epoch 9/10\n", 395 | "379/379 [==============================] - 0s 5us/step - loss: 0.6723 - acc: 0.7916\n", 396 | "Epoch 10/10\n", 397 | "379/379 [==============================] - 0s 7us/step - loss: 0.6710 - acc: 0.7757\n", 398 | "190/190 [==============================] - 0s 503us/step\n", 399 | "379/379 [==============================] - 0s 3us/step\n", 400 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.7129907024046704, epochs=10, first_neuron=13, kernel_initializer=normal, optimizer=adam, total= 0.9s\n", 401 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.7129907024046704, epochs=10, first_neuron=13, kernel_initializer=normal, optimizer=adam \n", 402 | "Epoch 1/10\n", 403 | "379/379 [==============================] - 0s 951us/step - loss: 0.6967 - acc: 0.4723\n", 404 | "Epoch 2/10\n", 405 | "379/379 [==============================] - 0s 12us/step - loss: 0.6985 - acc: 0.4433\n", 406 | "Epoch 3/10\n", 407 | "379/379 [==============================] - 0s 7us/step - loss: 0.6947 - acc: 0.5356\n", 408 | "Epoch 4/10\n", 409 | "379/379 [==============================] - 0s 6us/step - loss: 0.6929 - acc: 0.5594\n", 410 | "Epoch 5/10\n", 411 | "379/379 [==============================] - 0s 6us/step - loss: 0.6908 - acc: 0.5963\n", 412 | "Epoch 6/10\n", 413 | "379/379 [==============================] - 0s 6us/step - loss: 0.6884 - acc: 0.6385\n", 414 | "Epoch 7/10\n", 415 | "379/379 [==============================] - 0s 7us/step - loss: 0.6864 - acc: 0.6570\n", 416 | "Epoch 8/10\n", 417 | "379/379 [==============================] - 0s 7us/step - loss: 0.6849 - acc: 0.6807\n", 418 | "Epoch 9/10\n", 419 | "379/379 [==============================] - 0s 5us/step - loss: 0.6837 - acc: 0.7124\n", 420 | "Epoch 10/10\n", 421 | "379/379 [==============================] - 0s 6us/step - loss: 0.6826 - acc: 0.6966\n", 422 | "190/190 [==============================] - 0s 528us/step\n", 423 | "379/379 [==============================] - 0s 3us/step\n", 424 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.7129907024046704, epochs=10, first_neuron=13, kernel_initializer=normal, optimizer=adam, total= 1.1s\n", 425 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.7129907024046704, epochs=10, first_neuron=13, kernel_initializer=normal, optimizer=adam \n", 426 | "Epoch 1/10\n", 427 | "380/380 [==============================] - 0s 998us/step - loss: 0.6964 - acc: 0.5105\n", 428 | "Epoch 2/10\n", 429 | "380/380 [==============================] - 0s 12us/step - loss: 0.6951 - acc: 0.5474\n", 430 | "Epoch 3/10\n", 431 | "380/380 [==============================] - 0s 5us/step - loss: 0.6932 - acc: 0.5500\n", 432 | "Epoch 4/10\n", 433 | "380/380 [==============================] - 0s 7us/step - loss: 0.6906 - acc: 0.5711\n", 434 | "Epoch 5/10\n", 435 | "380/380 [==============================] - 0s 5us/step - loss: 0.6899 - acc: 0.5816\n", 436 | "Epoch 6/10\n", 437 | "380/380 [==============================] - 0s 7us/step - loss: 0.6880 - acc: 0.6079\n", 438 | "Epoch 7/10\n", 439 | "380/380 [==============================] - 0s 6us/step - loss: 0.6858 - acc: 0.6237\n", 440 | "Epoch 8/10\n", 441 | "380/380 [==============================] - 0s 8us/step - loss: 0.6844 - acc: 0.6737\n", 442 | "Epoch 9/10\n", 443 | "380/380 [==============================] - 0s 6us/step - loss: 0.6809 - acc: 0.6868\n", 444 | "Epoch 10/10\n", 445 | "380/380 [==============================] - 0s 7us/step - loss: 0.6801 - acc: 0.7000\n", 446 | "189/189 [==============================] - 0s 623us/step\n", 447 | "380/380 [==============================] - 0s 3us/step\n", 448 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.7129907024046704, epochs=10, first_neuron=13, kernel_initializer=normal, optimizer=adam, total= 1.0s\n", 449 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.3072453873479509, epochs=10, first_neuron=10, kernel_initializer=uniform, optimizer=sgd \n", 450 | "Epoch 1/10\n", 451 | "379/379 [==============================] - 0s 987us/step - loss: 0.6946 - acc: 0.3931\n", 452 | "Epoch 2/10\n", 453 | "379/379 [==============================] - 0s 12us/step - loss: 0.6941 - acc: 0.5066\n", 454 | "Epoch 3/10\n", 455 | "379/379 [==============================] - 0s 5us/step - loss: 0.6937 - acc: 0.5884\n", 456 | "Epoch 4/10\n", 457 | "379/379 [==============================] - 0s 13us/step - loss: 0.6933 - acc: 0.6253\n", 458 | "Epoch 5/10\n", 459 | "379/379 [==============================] - 0s 7us/step - loss: 0.6929 - acc: 0.6702\n", 460 | "Epoch 6/10\n", 461 | "379/379 [==============================] - 0s 7us/step - loss: 0.6920 - acc: 0.6807\n", 462 | "Epoch 7/10\n", 463 | "379/379 [==============================] - 0s 6us/step - loss: 0.6917 - acc: 0.6887\n", 464 | "Epoch 8/10\n", 465 | "379/379 [==============================] - 0s 7us/step - loss: 0.6912 - acc: 0.7045\n", 466 | "Epoch 9/10\n", 467 | "379/379 [==============================] - 0s 7us/step - loss: 0.6906 - acc: 0.6966\n", 468 | "Epoch 10/10\n", 469 | "379/379 [==============================] - 0s 8us/step - loss: 0.6903 - acc: 0.7018\n", 470 | "190/190 [==============================] - 0s 604us/step\n", 471 | "379/379 [==============================] - 0s 3us/step\n", 472 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.3072453873479509, epochs=10, first_neuron=10, kernel_initializer=uniform, optimizer=sgd, total= 0.9s\n", 473 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.3072453873479509, epochs=10, first_neuron=10, kernel_initializer=uniform, optimizer=sgd \n", 474 | "Epoch 1/10\n", 475 | "379/379 [==============================] - 0s 1ms/step - loss: 0.6912 - acc: 0.7203\n", 476 | "Epoch 2/10\n", 477 | "379/379 [==============================] - 0s 11us/step - loss: 0.6909 - acc: 0.7177\n", 478 | "Epoch 3/10\n", 479 | "379/379 [==============================] - 0s 5us/step - loss: 0.6913 - acc: 0.6755\n", 480 | "Epoch 4/10\n", 481 | "379/379 [==============================] - 0s 7us/step - loss: 0.6906 - acc: 0.7018\n", 482 | "Epoch 5/10\n", 483 | "379/379 [==============================] - 0s 7us/step - loss: 0.6907 - acc: 0.6913\n", 484 | "Epoch 6/10\n", 485 | "379/379 [==============================] - 0s 5us/step - loss: 0.6902 - acc: 0.6992\n", 486 | "Epoch 7/10\n", 487 | "379/379 [==============================] - 0s 6us/step - loss: 0.6899 - acc: 0.6913\n", 488 | "Epoch 8/10\n", 489 | "379/379 [==============================] - 0s 7us/step - loss: 0.6897 - acc: 0.6807\n", 490 | "Epoch 9/10\n", 491 | "379/379 [==============================] - 0s 7us/step - loss: 0.6896 - acc: 0.6807\n", 492 | "Epoch 10/10\n", 493 | "379/379 [==============================] - 0s 6us/step - loss: 0.6894 - acc: 0.6649\n", 494 | "190/190 [==============================] - 0s 712us/step\n", 495 | "379/379 [==============================] - 0s 3us/step\n", 496 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.3072453873479509, epochs=10, first_neuron=10, kernel_initializer=uniform, optimizer=sgd, total= 1.0s\n", 497 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.3072453873479509, epochs=10, first_neuron=10, kernel_initializer=uniform, optimizer=sgd \n", 498 | "Epoch 1/10\n", 499 | "380/380 [==============================] - 0s 1ms/step - loss: 0.6938 - acc: 0.4368\n", 500 | "Epoch 2/10\n", 501 | "380/380 [==============================] - 0s 11us/step - loss: 0.6936 - acc: 0.4658\n", 502 | "Epoch 3/10\n", 503 | "380/380 [==============================] - 0s 6us/step - loss: 0.6935 - acc: 0.4474\n", 504 | "Epoch 4/10\n", 505 | "380/380 [==============================] - 0s 8us/step - loss: 0.6935 - acc: 0.4447\n", 506 | "Epoch 5/10\n", 507 | "380/380 [==============================] - 0s 6us/step - loss: 0.6934 - acc: 0.4474\n", 508 | "Epoch 6/10\n", 509 | "380/380 [==============================] - 0s 7us/step - loss: 0.6935 - acc: 0.4763\n", 510 | "Epoch 7/10\n", 511 | "380/380 [==============================] - 0s 7us/step - loss: 0.6930 - acc: 0.4816\n", 512 | "Epoch 8/10\n", 513 | "380/380 [==============================] - 0s 7us/step - loss: 0.6932 - acc: 0.4605\n", 514 | "Epoch 9/10\n", 515 | "380/380 [==============================] - 0s 7us/step - loss: 0.6929 - acc: 0.4711\n", 516 | "Epoch 10/10\n", 517 | "380/380 [==============================] - 0s 5us/step - loss: 0.6927 - acc: 0.5053\n", 518 | "189/189 [==============================] - 0s 812us/step\n", 519 | "380/380 [==============================] - 0s 10us/step\n", 520 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.3072453873479509, epochs=10, first_neuron=10, kernel_initializer=uniform, optimizer=sgd, total= 1.0s\n", 521 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.9201662691113225, epochs=10, first_neuron=12, kernel_initializer=uniform, optimizer=sgd \n", 522 | "Epoch 1/10\n", 523 | "379/379 [==============================] - 0s 1ms/step - loss: 0.6902 - acc: 0.6412\n", 524 | "Epoch 2/10\n", 525 | "379/379 [==============================] - 0s 12us/step - loss: 0.6905 - acc: 0.6332\n", 526 | "Epoch 3/10\n", 527 | "379/379 [==============================] - 0s 5us/step - loss: 0.6902 - acc: 0.6438\n", 528 | "Epoch 4/10\n", 529 | "379/379 [==============================] - 0s 9us/step - loss: 0.6894 - acc: 0.6675\n", 530 | "Epoch 5/10\n", 531 | "379/379 [==============================] - 0s 7us/step - loss: 0.6905 - acc: 0.6438\n", 532 | "Epoch 6/10\n", 533 | "379/379 [==============================] - 0s 6us/step - loss: 0.6884 - acc: 0.6623\n", 534 | "Epoch 7/10\n", 535 | "379/379 [==============================] - 0s 6us/step - loss: 0.6886 - acc: 0.6887\n", 536 | "Epoch 8/10\n", 537 | "379/379 [==============================] - 0s 6us/step - loss: 0.6853 - acc: 0.6939\n", 538 | "Epoch 9/10\n", 539 | "379/379 [==============================] - 0s 11us/step - loss: 0.6857 - acc: 0.6992\n", 540 | "Epoch 10/10\n", 541 | "379/379 [==============================] - 0s 7us/step - loss: 0.6851 - acc: 0.7071\n", 542 | "190/190 [==============================] - 0s 797us/step\n", 543 | "379/379 [==============================] - 0s 4us/step\n", 544 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.9201662691113225, epochs=10, first_neuron=12, kernel_initializer=uniform, optimizer=sgd, total= 1.0s\n", 545 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.9201662691113225, epochs=10, first_neuron=12, kernel_initializer=uniform, optimizer=sgd \n", 546 | "Epoch 1/10\n", 547 | "379/379 [==============================] - 0s 1ms/step - loss: 0.6921 - acc: 0.6016\n", 548 | "Epoch 2/10\n", 549 | "379/379 [==============================] - 0s 10us/step - loss: 0.6903 - acc: 0.6148\n", 550 | "Epoch 3/10\n", 551 | "379/379 [==============================] - 0s 6us/step - loss: 0.6900 - acc: 0.6385\n", 552 | "Epoch 4/10\n", 553 | "379/379 [==============================] - 0s 6us/step - loss: 0.6898 - acc: 0.6069\n", 554 | "Epoch 5/10\n", 555 | "379/379 [==============================] - 0s 6us/step - loss: 0.6888 - acc: 0.6491\n", 556 | "Epoch 6/10\n", 557 | "379/379 [==============================] - 0s 6us/step - loss: 0.6888 - acc: 0.6332\n", 558 | "Epoch 7/10\n", 559 | "379/379 [==============================] - 0s 6us/step - loss: 0.6872 - acc: 0.6332\n", 560 | "Epoch 8/10\n", 561 | "379/379 [==============================] - 0s 6us/step - loss: 0.6893 - acc: 0.6069\n", 562 | "Epoch 9/10\n", 563 | "379/379 [==============================] - 0s 8us/step - loss: 0.6860 - acc: 0.6174\n", 564 | "Epoch 10/10\n", 565 | "379/379 [==============================] - 0s 7us/step - loss: 0.6901 - acc: 0.6253\n", 566 | "190/190 [==============================] - 0s 807us/step\n", 567 | "379/379 [==============================] - 0s 5us/step\n", 568 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.9201662691113225, epochs=10, first_neuron=12, kernel_initializer=uniform, optimizer=sgd, total= 1.0s\n", 569 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.9201662691113225, epochs=10, first_neuron=12, kernel_initializer=uniform, optimizer=sgd \n", 570 | "Epoch 1/10\n", 571 | "380/380 [==============================] - 0s 1ms/step - loss: 0.6954 - acc: 0.4868\n", 572 | "Epoch 2/10\n", 573 | "380/380 [==============================] - 0s 12us/step - loss: 0.6966 - acc: 0.4711\n", 574 | "Epoch 3/10\n", 575 | "380/380 [==============================] - 0s 5us/step - loss: 0.6931 - acc: 0.5342\n", 576 | "Epoch 4/10\n", 577 | "380/380 [==============================] - 0s 9us/step - loss: 0.6896 - acc: 0.5763\n", 578 | "Epoch 5/10\n", 579 | "380/380 [==============================] - 0s 6us/step - loss: 0.6927 - acc: 0.5263\n", 580 | "Epoch 6/10\n", 581 | "380/380 [==============================] - 0s 7us/step - loss: 0.6902 - acc: 0.5447\n", 582 | "Epoch 7/10\n", 583 | "380/380 [==============================] - 0s 6us/step - loss: 0.6933 - acc: 0.5237\n", 584 | "Epoch 8/10\n", 585 | "380/380 [==============================] - 0s 7us/step - loss: 0.6879 - acc: 0.5947\n", 586 | "Epoch 9/10\n", 587 | "380/380 [==============================] - 0s 5us/step - loss: 0.6907 - acc: 0.5447\n", 588 | "Epoch 10/10\n", 589 | "380/380 [==============================] - 0s 10us/step - loss: 0.6886 - acc: 0.5632\n", 590 | "189/189 [==============================] - 0s 901us/step\n", 591 | "380/380 [==============================] - 0s 4us/step\n", 592 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.9201662691113225, epochs=10, first_neuron=12, kernel_initializer=uniform, optimizer=sgd, total= 1.2s\n", 593 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.1246912886611955, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=adam \n", 594 | "Epoch 1/10\n", 595 | "379/379 [==============================] - 1s 1ms/step - loss: 0.6927 - acc: 0.5515\n", 596 | "Epoch 2/10\n", 597 | "379/379 [==============================] - 0s 12us/step - loss: 0.6918 - acc: 0.7203\n", 598 | "Epoch 3/10\n", 599 | "379/379 [==============================] - 0s 5us/step - loss: 0.6908 - acc: 0.7784\n", 600 | "Epoch 4/10\n", 601 | "379/379 [==============================] - 0s 10us/step - loss: 0.6898 - acc: 0.8100\n", 602 | "Epoch 5/10\n", 603 | "379/379 [==============================] - 0s 4us/step - loss: 0.6887 - acc: 0.8338\n", 604 | "Epoch 6/10\n", 605 | "379/379 [==============================] - 0s 8us/step - loss: 0.6875 - acc: 0.8417\n", 606 | "Epoch 7/10\n", 607 | "379/379 [==============================] - 0s 12us/step - loss: 0.6865 - acc: 0.8522\n", 608 | "Epoch 8/10\n", 609 | "379/379 [==============================] - 0s 6us/step - loss: 0.6852 - acc: 0.8707\n", 610 | "Epoch 9/10\n", 611 | "379/379 [==============================] - 0s 5us/step - loss: 0.6838 - acc: 0.8628\n", 612 | "Epoch 10/10\n", 613 | "379/379 [==============================] - 0s 8us/step - loss: 0.6827 - acc: 0.8813\n", 614 | "190/190 [==============================] - 0s 1ms/step\n", 615 | "379/379 [==============================] - 0s 4us/step\n", 616 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.1246912886611955, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=adam, total= 1.2s\n", 617 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.1246912886611955, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=adam \n", 618 | "Epoch 1/10\n", 619 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6957 - acc: 0.2559\n", 620 | "Epoch 2/10\n", 621 | "379/379 [==============================] - 0s 13us/step - loss: 0.6945 - acc: 0.3245\n", 622 | "Epoch 3/10\n", 623 | "379/379 [==============================] - 0s 5us/step - loss: 0.6933 - acc: 0.4908\n", 624 | "Epoch 4/10\n", 625 | "379/379 [==============================] - 0s 6us/step - loss: 0.6922 - acc: 0.6412\n", 626 | "Epoch 5/10\n", 627 | "379/379 [==============================] - 0s 6us/step - loss: 0.6912 - acc: 0.7652\n", 628 | "Epoch 6/10\n", 629 | "379/379 [==============================] - 0s 6us/step - loss: 0.6903 - acc: 0.8021\n", 630 | "Epoch 7/10\n", 631 | "379/379 [==============================] - 0s 5us/step - loss: 0.6893 - acc: 0.8179\n", 632 | "Epoch 8/10\n", 633 | "379/379 [==============================] - 0s 6us/step - loss: 0.6881 - acc: 0.8522\n", 634 | "Epoch 9/10\n", 635 | "379/379 [==============================] - 0s 6us/step - loss: 0.6870 - acc: 0.8602\n", 636 | "Epoch 10/10\n", 637 | "379/379 [==============================] - 0s 5us/step - loss: 0.6857 - acc: 0.8628\n", 638 | "190/190 [==============================] - 0s 1ms/step\n", 639 | "379/379 [==============================] - 0s 4us/step\n", 640 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.1246912886611955, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=adam, total= 1.3s\n", 641 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.1246912886611955, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=adam \n", 642 | "Epoch 1/10\n", 643 | "380/380 [==============================] - 1s 2ms/step - loss: 0.6910 - acc: 0.7500\n", 644 | "Epoch 2/10\n", 645 | "380/380 [==============================] - 0s 6us/step - loss: 0.6901 - acc: 0.7921\n", 646 | "Epoch 3/10\n", 647 | "380/380 [==============================] - 0s 11us/step - loss: 0.6892 - acc: 0.8184\n", 648 | "Epoch 4/10\n", 649 | "380/380 [==============================] - 0s 5us/step - loss: 0.6883 - acc: 0.8263\n", 650 | "Epoch 5/10\n", 651 | "380/380 [==============================] - 0s 6us/step - loss: 0.6872 - acc: 0.8474\n", 652 | "Epoch 6/10\n", 653 | "380/380 [==============================] - 0s 10us/step - loss: 0.6864 - acc: 0.8447\n", 654 | "Epoch 7/10\n", 655 | "380/380 [==============================] - 0s 5us/step - loss: 0.6853 - acc: 0.8684\n", 656 | "Epoch 8/10\n", 657 | "380/380 [==============================] - 0s 6us/step - loss: 0.6839 - acc: 0.8658\n", 658 | "Epoch 9/10\n", 659 | "380/380 [==============================] - 0s 10us/step - loss: 0.6832 - acc: 0.8789\n", 660 | "Epoch 10/10\n", 661 | "380/380 [==============================] - 0s 8us/step - loss: 0.6813 - acc: 0.8789\n", 662 | "189/189 [==============================] - 0s 1ms/step\n", 663 | "380/380 [==============================] - 0s 11us/step\n", 664 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.1246912886611955, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=adam, total= 1.3s\n", 665 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.06949616977881301, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=sgd \n", 666 | "Epoch 1/10\n", 667 | "379/379 [==============================] - 1s 1ms/step - loss: 0.6836 - acc: 0.7361\n", 668 | "Epoch 2/10\n", 669 | "379/379 [==============================] - 0s 11us/step - loss: 0.6827 - acc: 0.7546\n", 670 | "Epoch 3/10\n", 671 | "379/379 [==============================] - 0s 5us/step - loss: 0.6816 - acc: 0.7889\n", 672 | "Epoch 4/10\n", 673 | "379/379 [==============================] - 0s 9us/step - loss: 0.6803 - acc: 0.8047\n", 674 | "Epoch 5/10\n", 675 | "379/379 [==============================] - 0s 6us/step - loss: 0.6794 - acc: 0.8179\n", 676 | "Epoch 6/10\n", 677 | "379/379 [==============================] - 0s 7us/step - loss: 0.6784 - acc: 0.8100\n", 678 | "Epoch 7/10\n", 679 | "379/379 [==============================] - 0s 9us/step - loss: 0.6775 - acc: 0.8364\n", 680 | "Epoch 8/10\n", 681 | "379/379 [==============================] - 0s 7us/step - loss: 0.6763 - acc: 0.8285\n", 682 | "Epoch 9/10\n", 683 | "379/379 [==============================] - 0s 7us/step - loss: 0.6753 - acc: 0.8338\n", 684 | "Epoch 10/10\n", 685 | "379/379 [==============================] - 0s 7us/step - loss: 0.6745 - acc: 0.8311\n", 686 | "190/190 [==============================] - 0s 1ms/step\n", 687 | "379/379 [==============================] - 0s 4us/step\n", 688 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.06949616977881301, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=sgd, total= 1.2s\n", 689 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.06949616977881301, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=sgd \n", 690 | "Epoch 1/10\n", 691 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6864 - acc: 0.6939\n", 692 | "Epoch 2/10\n", 693 | "379/379 [==============================] - 0s 11us/step - loss: 0.6861 - acc: 0.7150\n", 694 | "Epoch 3/10\n", 695 | "379/379 [==============================] - 0s 5us/step - loss: 0.6852 - acc: 0.7177\n", 696 | "Epoch 4/10\n", 697 | "379/379 [==============================] - 0s 6us/step - loss: 0.6846 - acc: 0.7256\n", 698 | "Epoch 5/10\n", 699 | "379/379 [==============================] - 0s 6us/step - loss: 0.6840 - acc: 0.7441\n", 700 | "Epoch 6/10\n", 701 | "379/379 [==============================] - 0s 6us/step - loss: 0.6836 - acc: 0.7573\n", 702 | "Epoch 7/10\n", 703 | "379/379 [==============================] - 0s 6us/step - loss: 0.6830 - acc: 0.7731\n", 704 | "Epoch 8/10\n", 705 | "379/379 [==============================] - 0s 6us/step - loss: 0.6816 - acc: 0.7757\n", 706 | "Epoch 9/10\n", 707 | "379/379 [==============================] - 0s 5us/step - loss: 0.6817 - acc: 0.7625\n", 708 | "Epoch 10/10\n", 709 | "379/379 [==============================] - 0s 6us/step - loss: 0.6813 - acc: 0.7836\n", 710 | "190/190 [==============================] - 0s 1ms/step\n", 711 | "379/379 [==============================] - 0s 4us/step\n", 712 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.06949616977881301, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=sgd, total= 1.4s\n", 713 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.06949616977881301, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=sgd \n", 714 | "Epoch 1/10\n", 715 | "380/380 [==============================] - 1s 2ms/step - loss: 0.6990 - acc: 0.3553\n", 716 | "Epoch 2/10\n", 717 | "380/380 [==============================] - 0s 12us/step - loss: 0.6984 - acc: 0.3500\n", 718 | "Epoch 3/10\n", 719 | "380/380 [==============================] - 0s 7us/step - loss: 0.6971 - acc: 0.3974\n", 720 | "Epoch 4/10\n", 721 | "380/380 [==============================] - 0s 7us/step - loss: 0.6966 - acc: 0.4237\n", 722 | "Epoch 5/10\n", 723 | "380/380 [==============================] - 0s 7us/step - loss: 0.6953 - acc: 0.4500\n", 724 | "Epoch 6/10\n", 725 | "380/380 [==============================] - 0s 6us/step - loss: 0.6947 - acc: 0.4868\n", 726 | "Epoch 7/10\n", 727 | "380/380 [==============================] - 0s 8us/step - loss: 0.6943 - acc: 0.5079\n", 728 | "Epoch 8/10\n", 729 | "380/380 [==============================] - 0s 8us/step - loss: 0.6934 - acc: 0.5342\n", 730 | "Epoch 9/10\n", 731 | "380/380 [==============================] - 0s 7us/step - loss: 0.6926 - acc: 0.5474\n", 732 | "Epoch 10/10\n", 733 | "380/380 [==============================] - 0s 8us/step - loss: 0.6917 - acc: 0.5711\n", 734 | "189/189 [==============================] - 0s 1ms/step\n", 735 | "380/380 [==============================] - 0s 4us/step\n", 736 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.06949616977881301, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=sgd, total= 1.3s\n", 737 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.4902729543087778, epochs=10, first_neuron=13, kernel_initializer=uniform, optimizer=sgd \n", 738 | "Epoch 1/10\n", 739 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6903 - acc: 0.6649\n", 740 | "Epoch 2/10\n", 741 | "379/379 [==============================] - 0s 11us/step - loss: 0.6890 - acc: 0.6385\n", 742 | "Epoch 3/10\n", 743 | "379/379 [==============================] - 0s 6us/step - loss: 0.6879 - acc: 0.7441\n", 744 | "Epoch 4/10\n", 745 | "379/379 [==============================] - 0s 7us/step - loss: 0.6870 - acc: 0.7520\n", 746 | "Epoch 5/10\n", 747 | "379/379 [==============================] - 0s 9us/step - loss: 0.6859 - acc: 0.7704\n", 748 | "Epoch 6/10\n", 749 | "379/379 [==============================] - 0s 5us/step - loss: 0.6858 - acc: 0.7731\n", 750 | "Epoch 7/10\n", 751 | "379/379 [==============================] - 0s 11us/step - loss: 0.6843 - acc: 0.7836\n", 752 | "Epoch 8/10\n", 753 | "379/379 [==============================] - 0s 8us/step - loss: 0.6836 - acc: 0.7889\n", 754 | "Epoch 9/10\n", 755 | "379/379 [==============================] - 0s 7us/step - loss: 0.6828 - acc: 0.8153\n", 756 | "Epoch 10/10\n", 757 | "379/379 [==============================] - 0s 8us/step - loss: 0.6815 - acc: 0.7889\n", 758 | "190/190 [==============================] - 0s 1ms/step\n", 759 | "379/379 [==============================] - 0s 11us/step\n", 760 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.4902729543087778, epochs=10, first_neuron=13, kernel_initializer=uniform, optimizer=sgd, total= 1.3s\n", 761 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.4902729543087778, epochs=10, first_neuron=13, kernel_initializer=uniform, optimizer=sgd \n", 762 | "Epoch 1/10\n", 763 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6932 - acc: 0.4802\n", 764 | "Epoch 2/10\n", 765 | "379/379 [==============================] - 0s 7us/step - loss: 0.6930 - acc: 0.5277\n", 766 | "Epoch 3/10\n", 767 | "379/379 [==============================] - 0s 6us/step - loss: 0.6922 - acc: 0.5805\n", 768 | "Epoch 4/10\n", 769 | "379/379 [==============================] - 0s 9us/step - loss: 0.6922 - acc: 0.5910\n", 770 | "Epoch 5/10\n", 771 | "379/379 [==============================] - 0s 8us/step - loss: 0.6919 - acc: 0.6174\n", 772 | "Epoch 6/10\n", 773 | "379/379 [==============================] - 0s 7us/step - loss: 0.6912 - acc: 0.6227\n", 774 | "Epoch 7/10\n", 775 | "379/379 [==============================] - 0s 6us/step - loss: 0.6906 - acc: 0.6517\n", 776 | "Epoch 8/10\n", 777 | "379/379 [==============================] - 0s 8us/step - loss: 0.6904 - acc: 0.6623\n", 778 | "Epoch 9/10\n", 779 | "379/379 [==============================] - 0s 6us/step - loss: 0.6896 - acc: 0.6781\n", 780 | "Epoch 10/10\n", 781 | "379/379 [==============================] - 0s 8us/step - loss: 0.6890 - acc: 0.6755\n", 782 | "190/190 [==============================] - 0s 2ms/step\n", 783 | "379/379 [==============================] - 0s 4us/step\n", 784 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.4902729543087778, epochs=10, first_neuron=13, kernel_initializer=uniform, optimizer=sgd, total= 1.4s\n", 785 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.4902729543087778, epochs=10, first_neuron=13, kernel_initializer=uniform, optimizer=sgd \n", 786 | "Epoch 1/10\n", 787 | "380/380 [==============================] - 1s 2ms/step - loss: 0.6913 - acc: 0.6132\n", 788 | "Epoch 2/10\n", 789 | "380/380 [==============================] - 0s 11us/step - loss: 0.6906 - acc: 0.6711\n", 790 | "Epoch 3/10\n", 791 | "380/380 [==============================] - 0s 11us/step - loss: 0.6900 - acc: 0.7158\n", 792 | "Epoch 4/10\n", 793 | "380/380 [==============================] - 0s 7us/step - loss: 0.6904 - acc: 0.7053\n", 794 | "Epoch 5/10\n", 795 | "380/380 [==============================] - 0s 7us/step - loss: 0.6894 - acc: 0.7316\n", 796 | "Epoch 6/10\n", 797 | "380/380 [==============================] - 0s 7us/step - loss: 0.6897 - acc: 0.7553\n", 798 | "Epoch 7/10\n", 799 | "380/380 [==============================] - 0s 7us/step - loss: 0.6894 - acc: 0.7447\n", 800 | "Epoch 8/10\n", 801 | "380/380 [==============================] - 0s 7us/step - loss: 0.6888 - acc: 0.7868\n", 802 | "Epoch 9/10\n", 803 | "380/380 [==============================] - 0s 7us/step - loss: 0.6884 - acc: 0.7737\n", 804 | "Epoch 10/10\n", 805 | "380/380 [==============================] - 0s 7us/step - loss: 0.6883 - acc: 0.7947\n", 806 | "189/189 [==============================] - 0s 1ms/step\n", 807 | "380/380 [==============================] - 0s 11us/step\n", 808 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.4902729543087778, epochs=10, first_neuron=13, kernel_initializer=uniform, optimizer=sgd, total= 1.4s\n" 809 | ] 810 | }, 811 | { 812 | "name": "stderr", 813 | "output_type": "stream", 814 | "text": [ 815 | "[Parallel(n_jobs=1)]: Done 24 out of 24 | elapsed: 26.3s finished\n" 816 | ] 817 | }, 818 | { 819 | "name": "stdout", 820 | "output_type": "stream", 821 | "text": [ 822 | "Epoch 1/10\n", 823 | "569/569 [==============================] - 1s 1ms/step - loss: 0.6953 - acc: 0.4728\n", 824 | "Epoch 2/10\n", 825 | "569/569 [==============================] - 0s 4us/step - loss: 0.6928 - acc: 0.5009\n", 826 | "Epoch 3/10\n", 827 | "569/569 [==============================] - 0s 4us/step - loss: 0.6909 - acc: 0.5870\n", 828 | "Epoch 4/10\n", 829 | "569/569 [==============================] - 0s 4us/step - loss: 0.6880 - acc: 0.6028\n", 830 | "Epoch 5/10\n", 831 | "569/569 [==============================] - 0s 4us/step - loss: 0.6845 - acc: 0.6538\n", 832 | "Epoch 6/10\n", 833 | "569/569 [==============================] - 0s 4us/step - loss: 0.6805 - acc: 0.7030\n", 834 | "Epoch 7/10\n", 835 | "569/569 [==============================] - 0s 4us/step - loss: 0.6787 - acc: 0.7557\n", 836 | "Epoch 8/10\n", 837 | "569/569 [==============================] - 0s 4us/step - loss: 0.6763 - acc: 0.7399\n", 838 | "Epoch 9/10\n", 839 | "569/569 [==============================] - 0s 4us/step - loss: 0.6734 - acc: 0.7610\n", 840 | "Epoch 10/10\n", 841 | "569/569 [==============================] - 0s 4us/step - loss: 0.6702 - acc: 0.7979\n" 842 | ] 843 | }, 844 | { 845 | "data": { 846 | "text/plain": [ 847 | "RandomizedSearchCV(cv=3, error_score='raise',\n", 848 | " estimator=,\n", 849 | " fit_params=None, iid=True, n_iter=8, n_jobs=1,\n", 850 | " param_distributions={'epochs': [10], 'batch_size': [1024], 'optimizer': ['adam', 'nadam', 'sgd'], 'dropout_rate': , 'activation': ['relu', 'elu'], 'kernel_initializer': ['uniform', 'normal'], 'first_neuron': },\n", 851 | " pre_dispatch='2*n_jobs', random_state=None, refit=True,\n", 852 | " return_train_score='warn', scoring=None, verbose=2)" 853 | ] 854 | }, 855 | "execution_count": 8, 856 | "metadata": {}, 857 | "output_type": "execute_result" 858 | } 859 | ], 860 | "source": [ 861 | "# Perform the Search!\n", 862 | "from sklearn.model_selection import RandomizedSearchCV\n", 863 | "\n", 864 | "# run randomized search\n", 865 | "n_iter_search = 8 # Use half number of iterations performed by Grid Search\n", 866 | "random_search = RandomizedSearchCV(estimator=model, \n", 867 | " param_distributions=param_dist,\n", 868 | " n_iter=n_iter_search,\n", 869 | " n_jobs=1, cv=3, verbose=2)\n", 870 | "random_search.fit(x, y)" 871 | ] 872 | }, 873 | { 874 | "cell_type": "markdown", 875 | "metadata": {}, 876 | "source": [ 877 | "### Results\n", 878 | "\n", 879 | "Let's see which configuration give us the best performance.\n", 880 | "\n", 881 | "*Note*: it's likely that the Random Search we will perform slightly worse than Random Search in this toy example." 882 | ] 883 | }, 884 | { 885 | "cell_type": "code", 886 | "execution_count": 9, 887 | "metadata": {}, 888 | "outputs": [ 889 | { 890 | "name": "stdout", 891 | "output_type": "stream", 892 | "text": [ 893 | "Best: 0.894552 using {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.6010381533043048, 'epochs': 10, 'first_neuron': 10, 'kernel_initializer': 'normal', 'optimizer': 'adam'}\n", 894 | "0.894552 (0.034227) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.6010381533043048, 'epochs': 10, 'first_neuron': 10, 'kernel_initializer': 'normal', 'optimizer': 'adam'}\n", 895 | "0.806678 (0.184844) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.39983996327340543, 'epochs': 10, 'first_neuron': 8, 'kernel_initializer': 'uniform', 'optimizer': 'adam'}\n", 896 | "0.843585 (0.077833) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.7129907024046704, 'epochs': 10, 'first_neuron': 13, 'kernel_initializer': 'normal', 'optimizer': 'adam'}\n", 897 | "0.581722 (0.065330) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.3072453873479509, 'epochs': 10, 'first_neuron': 10, 'kernel_initializer': 'uniform', 'optimizer': 'sgd'}\n", 898 | "0.773286 (0.112224) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.9201662691113225, 'epochs': 10, 'first_neuron': 12, 'kernel_initializer': 'uniform', 'optimizer': 'sgd'}\n", 899 | "0.873462 (0.066920) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.1246912886611955, 'epochs': 10, 'first_neuron': 9, 'kernel_initializer': 'uniform', 'optimizer': 'adam'}\n", 900 | "0.745167 (0.054693) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.06949616977881301, 'epochs': 10, 'first_neuron': 9, 'kernel_initializer': 'normal', 'optimizer': 'sgd'}\n", 901 | "0.801406 (0.106561) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.4902729543087778, 'epochs': 10, 'first_neuron': 13, 'kernel_initializer': 'uniform', 'optimizer': 'sgd'}\n" 902 | ] 903 | } 904 | ], 905 | "source": [ 906 | "# Show results\n", 907 | "print(\"Best: %f using %s\" % (random_search.best_score_, random_search.best_params_))\n", 908 | "means = random_search.cv_results_['mean_test_score']\n", 909 | "stds = random_search.cv_results_['std_test_score']\n", 910 | "params = random_search.cv_results_['params']\n", 911 | "for mean, stdev, param in zip(means, stds, params):\n", 912 | " print(\"%f (%f) with: %r\" % (mean, stdev, param))" 913 | ] 914 | }, 915 | { 916 | "cell_type": "markdown", 917 | "metadata": {}, 918 | "source": [ 919 | "**That's all folks - don't forget to shutdown your workspace once you're done 🙂**" 920 | ] 921 | } 922 | ], 923 | "metadata": { 924 | "kernelspec": { 925 | "display_name": "Python 2", 926 | "language": "python", 927 | "name": "python2" 928 | }, 929 | "language_info": { 930 | "codemirror_mode": { 931 | "name": "ipython", 932 | "version": 2 933 | }, 934 | "file_extension": ".py", 935 | "mimetype": "text/x-python", 936 | "name": "python", 937 | "nbconvert_exporter": "python", 938 | "pygments_lexer": "ipython2", 939 | "version": "2.7.10" 940 | } 941 | }, 942 | "nbformat": 4, 943 | "nbformat_minor": 2 944 | } 945 | -------------------------------------------------------------------------------- /grid_search_example.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Grid Search Example\n", 8 | "\n", 9 | "In this example we will perform a GridSearch using Scikit-learn and Keras on the [breast cancer](https://github.com/autonomio/datasets/blob/master/autonomio-datasets/breast_cancer.csv) classification task. You can run this example on CPU. It will take more or less 5 minutes.\n", 10 | "\n", 11 | "This example was taken from [Talos](https://github.com/autonomio/talos/blob/master/examples/Hyperparameter%20Optimization%20on%20Keras%20with%20Breast%20Cancer%20Data.ipynb) which is another super interesting library to perfrom Grid and Random Search with Keras." 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "## Initial Setup\n", 19 | "\n", 20 | "Import the packages we need for the computation." 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 1, 26 | "metadata": {}, 27 | "outputs": [ 28 | { 29 | "name": "stderr", 30 | "output_type": "stream", 31 | "text": [ 32 | "Using TensorFlow backend.\n" 33 | ] 34 | } 35 | ], 36 | "source": [ 37 | "import os\n", 38 | "import pandas as pd\n", 39 | "import wrangle as wr\n", 40 | "\n", 41 | "from numpy import nan\n", 42 | "\n", 43 | "from keras.utils import to_categorical\n", 44 | "from keras.wrappers.scikit_learn import KerasClassifier\n", 45 | "\n", 46 | "# Mounting point\n", 47 | "MP = '/floyd/input/bcds'" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "### Load the dataset\n", 55 | "\n", 56 | "Load, clean and preprocess the dataset" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": 2, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "def breast_cancer():\n", 66 | " '''Load and preprocess(cleaning) the dataset'''\n", 67 | " df = pd.read_csv(os.path.join(MP, 'breast_cancer.csv'))\n", 68 | " \n", 69 | " # then some minimal data cleanup\n", 70 | " df.drop(\"Unnamed: 32\", axis=1, inplace=True)\n", 71 | " df.drop(\"id\", axis=1, inplace=True)\n", 72 | "\n", 73 | " # separate to x and y\n", 74 | " y = df.diagnosis.values\n", 75 | " x = df.drop('diagnosis', axis=1).values\n", 76 | "\n", 77 | " # convert the string labels to binary\n", 78 | " y = (y == 'M').astype(int)\n", 79 | "\n", 80 | " return x, y" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 3, 86 | "metadata": {}, 87 | "outputs": [], 88 | "source": [ 89 | "# Load the dataset\n", 90 | "x, y = breast_cancer()\n", 91 | "\n", 92 | "# Normalize every feature to mean 0, std 1\n", 93 | "x = wr.mean_zero(pd.DataFrame(x)).values\n", 94 | "\n", 95 | "input_dim = x.shape[1] # number of columns" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "### Model definition\n", 103 | "\n", 104 | "Define the model and the variables to search." 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 4, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "from keras.models import Sequential\n", 114 | "from keras.layers import Dense, Dropout, Flatten\n", 115 | "\n", 116 | "# Function to create model, required for KerasClassifier\n", 117 | "def create_model(first_neuron=9,\n", 118 | " activation='relu',\n", 119 | " kernel_initializer='uniform',\n", 120 | " dropout_rate=0,\n", 121 | " optimizer='Adam'):\n", 122 | " \n", 123 | " # Create model\n", 124 | " model = Sequential()\n", 125 | " # L1\n", 126 | " model.add(Dense(first_neuron, \n", 127 | " input_dim=input_dim, \n", 128 | " kernel_initializer=kernel_initializer, \n", 129 | " activation=activation))\n", 130 | " # Dropout\n", 131 | " model.add(Dropout(dropout_rate))\n", 132 | " # L2\n", 133 | " model.add(Dense(1, kernel_initializer=kernel_initializer, activation='sigmoid'))\n", 134 | " # Compile model\n", 135 | " model.compile(loss='binary_crossentropy', \n", 136 | " optimizer=optimizer, \n", 137 | " metrics=['accuracy'])\n", 138 | " return model" 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": 5, 144 | "metadata": {}, 145 | "outputs": [], 146 | "source": [ 147 | "# Create the model\n", 148 | "model = KerasClassifier(build_fn=create_model) " 149 | ] 150 | }, 151 | { 152 | "cell_type": "markdown", 153 | "metadata": {}, 154 | "source": [ 155 | "### Range of Values - The Grid\n", 156 | "\n", 157 | "Defining the parameter space boundaries." 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": 6, 163 | "metadata": {}, 164 | "outputs": [], 165 | "source": [ 166 | "# Define the range of values\n", 167 | "\n", 168 | "# Model Design Components\n", 169 | "first_neurons = [8, 9] \n", 170 | "activation = ['relu', 'elu'] # You can also try 'tanh', 'sigmoid', 'hard_sigmoid', 'linear'\n", 171 | "kernel_initializer = ['uniform', 'normal'] # You can also try lecun_uniform', 'zero', 'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform'\n", 172 | "optimizer = ['Adam', 'Nadam'] # You can also try 'SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adamax'\n", 173 | "\n", 174 | "# Hyperparameters\n", 175 | "epochs = [10] # You can also try 20, 30, 40, etc...\n", 176 | "batch_size = [1024] # You can also try 2, 4, 8, 16, 32, 64, 128 etc...\n", 177 | "dropout_rate = [0.0] # No dropout, but you can also try 0.1, 0.2 etc..." 178 | ] 179 | }, 180 | { 181 | "cell_type": "code", 182 | "execution_count": 7, 183 | "metadata": {}, 184 | "outputs": [], 185 | "source": [ 186 | "# Prepare the Grid\n", 187 | "param_grid = dict(epochs=epochs, \n", 188 | " batch_size=batch_size, \n", 189 | " optimizer=optimizer,\n", 190 | " dropout_rate=dropout_rate,\n", 191 | " activation=activation,\n", 192 | " kernel_initializer=kernel_initializer,\n", 193 | " first_neuron=first_neurons)" 194 | ] 195 | }, 196 | { 197 | "cell_type": "markdown", 198 | "metadata": {}, 199 | "source": [ 200 | "### Grid Search\n", 201 | "\n", 202 | "Run the search: 3 folds for cross-validation (`cv=3`) on a single process (`n_jobs=1`)." 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": 8, 208 | "metadata": {}, 209 | "outputs": [ 210 | { 211 | "name": "stdout", 212 | "output_type": "stream", 213 | "text": [ 214 | "Fitting 3 folds for each of 16 candidates, totalling 48 fits\n", 215 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam \n", 216 | "Epoch 1/10\n", 217 | "379/379 [==============================] - 0s 486us/step - loss: 0.6947 - acc: 0.3193\n", 218 | "Epoch 2/10\n", 219 | "379/379 [==============================] - 0s 4us/step - loss: 0.6936 - acc: 0.4485\n", 220 | "Epoch 3/10\n", 221 | "379/379 [==============================] - 0s 9us/step - loss: 0.6925 - acc: 0.5594\n", 222 | "Epoch 4/10\n", 223 | "379/379 [==============================] - 0s 12us/step - loss: 0.6914 - acc: 0.7177\n", 224 | "Epoch 5/10\n", 225 | "379/379 [==============================] - 0s 5us/step - loss: 0.6904 - acc: 0.8074\n", 226 | "Epoch 6/10\n", 227 | "379/379 [==============================] - 0s 5us/step - loss: 0.6893 - acc: 0.8364\n", 228 | "Epoch 7/10\n", 229 | "379/379 [==============================] - 0s 4us/step - loss: 0.6882 - acc: 0.8575\n", 230 | "Epoch 8/10\n", 231 | "379/379 [==============================] - 0s 5us/step - loss: 0.6872 - acc: 0.8628\n", 232 | "Epoch 9/10\n", 233 | "379/379 [==============================] - 0s 4us/step - loss: 0.6861 - acc: 0.8734\n", 234 | "Epoch 10/10\n", 235 | "379/379 [==============================] - 0s 6us/step - loss: 0.6850 - acc: 0.8786\n", 236 | "190/190 [==============================] - 0s 146us/step\n", 237 | "379/379 [==============================] - 0s 5us/step\n", 238 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam, total= 0.7s\n", 239 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam \n" 240 | ] 241 | }, 242 | { 243 | "name": "stderr", 244 | "output_type": "stream", 245 | "text": [ 246 | "[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.7s remaining: 0.0s\n" 247 | ] 248 | }, 249 | { 250 | "name": "stdout", 251 | "output_type": "stream", 252 | "text": [ 253 | "Epoch 1/10\n", 254 | "379/379 [==============================] - 0s 574us/step - loss: 0.6893 - acc: 0.7071\n", 255 | "Epoch 2/10\n", 256 | "379/379 [==============================] - 0s 6us/step - loss: 0.6882 - acc: 0.7836\n", 257 | "Epoch 3/10\n", 258 | "379/379 [==============================] - 0s 4us/step - loss: 0.6871 - acc: 0.8364\n", 259 | "Epoch 4/10\n", 260 | "379/379 [==============================] - 0s 9us/step - loss: 0.6860 - acc: 0.8602\n", 261 | "Epoch 5/10\n", 262 | "379/379 [==============================] - 0s 4us/step - loss: 0.6848 - acc: 0.8681\n", 263 | "Epoch 6/10\n", 264 | "379/379 [==============================] - 0s 5us/step - loss: 0.6835 - acc: 0.8813\n", 265 | "Epoch 7/10\n", 266 | "379/379 [==============================] - 0s 6us/step - loss: 0.6823 - acc: 0.8997\n", 267 | "Epoch 8/10\n", 268 | "379/379 [==============================] - 0s 5us/step - loss: 0.6810 - acc: 0.9024\n", 269 | "Epoch 9/10\n", 270 | "379/379 [==============================] - 0s 7us/step - loss: 0.6796 - acc: 0.9103\n", 271 | "Epoch 10/10\n", 272 | "379/379 [==============================] - 0s 5us/step - loss: 0.6782 - acc: 0.9077\n", 273 | "190/190 [==============================] - 0s 224us/step\n", 274 | "379/379 [==============================] - 0s 3us/step\n", 275 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam, total= 0.7s\n", 276 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam \n", 277 | "Epoch 1/10\n", 278 | "380/380 [==============================] - 0s 618us/step - loss: 0.6910 - acc: 0.7658\n", 279 | "Epoch 2/10\n", 280 | "380/380 [==============================] - 0s 12us/step - loss: 0.6901 - acc: 0.8079\n", 281 | "Epoch 3/10\n", 282 | "380/380 [==============================] - 0s 11us/step - loss: 0.6891 - acc: 0.8447\n", 283 | "Epoch 4/10\n", 284 | "380/380 [==============================] - 0s 12us/step - loss: 0.6880 - acc: 0.8605\n", 285 | "Epoch 5/10\n", 286 | "380/380 [==============================] - 0s 10us/step - loss: 0.6870 - acc: 0.8684\n", 287 | "Epoch 6/10\n", 288 | "380/380 [==============================] - 0s 4us/step - loss: 0.6859 - acc: 0.8711\n", 289 | "Epoch 7/10\n", 290 | "380/380 [==============================] - 0s 4us/step - loss: 0.6847 - acc: 0.8684\n", 291 | "Epoch 8/10\n", 292 | "380/380 [==============================] - 0s 4us/step - loss: 0.6835 - acc: 0.8684\n", 293 | "Epoch 9/10\n", 294 | "380/380 [==============================] - 0s 10us/step - loss: 0.6822 - acc: 0.8711\n", 295 | "Epoch 10/10\n", 296 | "380/380 [==============================] - 0s 6us/step - loss: 0.6808 - acc: 0.8737\n", 297 | "189/189 [==============================] - 0s 233us/step\n", 298 | "380/380 [==============================] - 0s 3us/step\n", 299 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam, total= 0.8s\n", 300 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam \n", 301 | "Epoch 1/10\n", 302 | "379/379 [==============================] - 0s 662us/step - loss: 0.6907 - acc: 0.8997\n", 303 | "Epoch 2/10\n", 304 | "379/379 [==============================] - 0s 11us/step - loss: 0.6886 - acc: 0.9499\n", 305 | "Epoch 3/10\n", 306 | "379/379 [==============================] - 0s 6us/step - loss: 0.6869 - acc: 0.9420\n", 307 | "Epoch 4/10\n", 308 | "379/379 [==============================] - 0s 4us/step - loss: 0.6852 - acc: 0.9340\n", 309 | "Epoch 5/10\n", 310 | "379/379 [==============================] - 0s 4us/step - loss: 0.6833 - acc: 0.9314\n", 311 | "Epoch 6/10\n", 312 | "379/379 [==============================] - 0s 5us/step - loss: 0.6811 - acc: 0.9367\n", 313 | "Epoch 7/10\n", 314 | "379/379 [==============================] - 0s 5us/step - loss: 0.6786 - acc: 0.9314\n", 315 | "Epoch 8/10\n", 316 | "379/379 [==============================] - 0s 6us/step - loss: 0.6758 - acc: 0.9340\n", 317 | "Epoch 9/10\n", 318 | "379/379 [==============================] - 0s 6us/step - loss: 0.6726 - acc: 0.9314\n", 319 | "Epoch 10/10\n", 320 | "379/379 [==============================] - 0s 7us/step - loss: 0.6691 - acc: 0.9340\n", 321 | "190/190 [==============================] - 0s 314us/step\n", 322 | "379/379 [==============================] - 0s 3us/step\n", 323 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam, total= 0.8s\n", 324 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam \n", 325 | "Epoch 1/10\n", 326 | "379/379 [==============================] - 0s 721us/step - loss: 0.6926 - acc: 0.5303\n", 327 | "Epoch 2/10\n", 328 | "379/379 [==============================] - 0s 11us/step - loss: 0.6906 - acc: 0.6570\n", 329 | "Epoch 3/10\n", 330 | "379/379 [==============================] - 0s 5us/step - loss: 0.6889 - acc: 0.6755\n", 331 | "Epoch 4/10\n", 332 | "379/379 [==============================] - 0s 6us/step - loss: 0.6872 - acc: 0.6834\n", 333 | "Epoch 5/10\n", 334 | "379/379 [==============================] - 0s 7us/step - loss: 0.6854 - acc: 0.6939\n", 335 | "Epoch 6/10\n", 336 | "379/379 [==============================] - 0s 7us/step - loss: 0.6833 - acc: 0.7361\n", 337 | "Epoch 7/10\n", 338 | "379/379 [==============================] - 0s 5us/step - loss: 0.6810 - acc: 0.7652\n", 339 | "Epoch 8/10\n", 340 | "379/379 [==============================] - 0s 8us/step - loss: 0.6784 - acc: 0.7863\n", 341 | "Epoch 9/10\n", 342 | "379/379 [==============================] - 0s 6us/step - loss: 0.6754 - acc: 0.8100\n", 343 | "Epoch 10/10\n", 344 | "379/379 [==============================] - 0s 6us/step - loss: 0.6721 - acc: 0.8417\n", 345 | "190/190 [==============================] - 0s 365us/step\n", 346 | "379/379 [==============================] - 0s 3us/step\n", 347 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam, total= 0.8s\n", 348 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam \n", 349 | "Epoch 1/10\n", 350 | "380/380 [==============================] - 0s 750us/step - loss: 0.6952 - acc: 0.2421\n", 351 | "Epoch 2/10\n", 352 | "380/380 [==============================] - 0s 6us/step - loss: 0.6930 - acc: 0.5447\n", 353 | "Epoch 3/10\n", 354 | "380/380 [==============================] - 0s 5us/step - loss: 0.6915 - acc: 0.7316\n", 355 | "Epoch 4/10\n", 356 | "380/380 [==============================] - 0s 7us/step - loss: 0.6899 - acc: 0.8105\n", 357 | "Epoch 5/10\n", 358 | "380/380 [==============================] - 0s 7us/step - loss: 0.6883 - acc: 0.8237\n", 359 | "Epoch 6/10\n", 360 | "380/380 [==============================] - 0s 5us/step - loss: 0.6865 - acc: 0.8395\n", 361 | "Epoch 7/10\n", 362 | "380/380 [==============================] - 0s 6us/step - loss: 0.6845 - acc: 0.8474\n", 363 | "Epoch 8/10\n", 364 | "380/380 [==============================] - 0s 5us/step - loss: 0.6823 - acc: 0.8632\n", 365 | "Epoch 9/10\n", 366 | "380/380 [==============================] - 0s 7us/step - loss: 0.6797 - acc: 0.8789\n", 367 | "Epoch 10/10\n", 368 | "380/380 [==============================] - 0s 5us/step - loss: 0.6769 - acc: 0.8842\n", 369 | "189/189 [==============================] - 0s 399us/step\n", 370 | "380/380 [==============================] - 0s 10us/step\n", 371 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam, total= 0.9s\n", 372 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam \n", 373 | "Epoch 1/10\n", 374 | "379/379 [==============================] - 0s 819us/step - loss: 0.6948 - acc: 0.3087\n", 375 | "Epoch 2/10\n", 376 | "379/379 [==============================] - 0s 12us/step - loss: 0.6934 - acc: 0.4142\n", 377 | "Epoch 3/10\n", 378 | "379/379 [==============================] - 0s 9us/step - loss: 0.6921 - acc: 0.5752\n", 379 | "Epoch 4/10\n", 380 | "379/379 [==============================] - 0s 4us/step - loss: 0.6908 - acc: 0.7150\n", 381 | "Epoch 5/10\n", 382 | "379/379 [==============================] - 0s 6us/step - loss: 0.6896 - acc: 0.7704\n", 383 | "Epoch 6/10\n", 384 | "379/379 [==============================] - 0s 6us/step - loss: 0.6884 - acc: 0.8179\n", 385 | "Epoch 7/10\n", 386 | "379/379 [==============================] - 0s 5us/step - loss: 0.6872 - acc: 0.8522\n", 387 | "Epoch 8/10\n", 388 | "379/379 [==============================] - 0s 6us/step - loss: 0.6861 - acc: 0.8628\n", 389 | "Epoch 9/10\n", 390 | "379/379 [==============================] - 0s 4us/step - loss: 0.6849 - acc: 0.8654\n", 391 | "Epoch 10/10\n", 392 | "379/379 [==============================] - 0s 5us/step - loss: 0.6837 - acc: 0.8707\n", 393 | "190/190 [==============================] - 0s 444us/step\n", 394 | "379/379 [==============================] - 0s 8us/step\n", 395 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam, total= 0.8s\n", 396 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam \n", 397 | "Epoch 1/10\n", 398 | "379/379 [==============================] - 0s 1ms/step - loss: 0.6915 - acc: 0.5330\n", 399 | "Epoch 2/10\n", 400 | "379/379 [==============================] - 0s 6us/step - loss: 0.6900 - acc: 0.6412\n", 401 | "Epoch 3/10\n", 402 | "379/379 [==============================] - 0s 5us/step - loss: 0.6885 - acc: 0.7414\n", 403 | "Epoch 4/10\n", 404 | "379/379 [==============================] - 0s 6us/step - loss: 0.6870 - acc: 0.8127\n", 405 | "Epoch 5/10\n", 406 | "379/379 [==============================] - 0s 5us/step - loss: 0.6855 - acc: 0.8549\n", 407 | "Epoch 6/10\n", 408 | "379/379 [==============================] - 0s 6us/step - loss: 0.6840 - acc: 0.8575\n", 409 | "Epoch 7/10\n", 410 | "379/379 [==============================] - 0s 5us/step - loss: 0.6824 - acc: 0.8786\n", 411 | "Epoch 8/10\n", 412 | "379/379 [==============================] - 0s 6us/step - loss: 0.6808 - acc: 0.8839\n", 413 | "Epoch 9/10\n", 414 | "379/379 [==============================] - 0s 13us/step - loss: 0.6792 - acc: 0.8945\n", 415 | "Epoch 10/10\n", 416 | "379/379 [==============================] - 0s 4us/step - loss: 0.6776 - acc: 0.8971\n", 417 | "190/190 [==============================] - 0s 465us/step\n", 418 | "379/379 [==============================] - 0s 3us/step\n", 419 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam, total= 1.0s\n", 420 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam \n", 421 | "Epoch 1/10\n", 422 | "380/380 [==============================] - 0s 938us/step - loss: 0.6873 - acc: 0.6711\n", 423 | "Epoch 2/10\n", 424 | "380/380 [==============================] - 0s 10us/step - loss: 0.6852 - acc: 0.7763\n", 425 | "Epoch 3/10\n", 426 | "380/380 [==============================] - 0s 4us/step - loss: 0.6831 - acc: 0.8211\n", 427 | "Epoch 4/10\n", 428 | "380/380 [==============================] - 0s 9us/step - loss: 0.6810 - acc: 0.8605\n", 429 | "Epoch 5/10\n", 430 | "380/380 [==============================] - 0s 7us/step - loss: 0.6788 - acc: 0.8868\n", 431 | "Epoch 6/10\n", 432 | "380/380 [==============================] - 0s 5us/step - loss: 0.6766 - acc: 0.8974\n", 433 | "Epoch 7/10\n", 434 | "380/380 [==============================] - 0s 6us/step - loss: 0.6745 - acc: 0.8974\n", 435 | "Epoch 8/10\n", 436 | "380/380 [==============================] - 0s 6us/step - loss: 0.6723 - acc: 0.9026\n", 437 | "Epoch 9/10\n", 438 | "380/380 [==============================] - 0s 7us/step - loss: 0.6700 - acc: 0.9000\n", 439 | "Epoch 10/10\n", 440 | "380/380 [==============================] - 0s 6us/step - loss: 0.6678 - acc: 0.9000\n", 441 | "189/189 [==============================] - 0s 540us/step\n", 442 | "380/380 [==============================] - 0s 4us/step\n", 443 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam, total= 0.9s\n", 444 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam \n", 445 | "Epoch 1/10\n", 446 | "379/379 [==============================] - 0s 1ms/step - loss: 0.6932 - acc: 0.4459\n", 447 | "Epoch 2/10\n", 448 | "379/379 [==============================] - 0s 7us/step - loss: 0.6902 - acc: 0.7441\n", 449 | "Epoch 3/10\n", 450 | "379/379 [==============================] - 0s 4us/step - loss: 0.6878 - acc: 0.8285\n", 451 | "Epoch 4/10\n", 452 | "379/379 [==============================] - 0s 5us/step - loss: 0.6854 - acc: 0.8760\n", 453 | "Epoch 5/10\n", 454 | "379/379 [==============================] - 0s 5us/step - loss: 0.6829 - acc: 0.8971\n", 455 | "Epoch 6/10\n", 456 | "379/379 [==============================] - 0s 7us/step - loss: 0.6802 - acc: 0.9077\n", 457 | "Epoch 7/10\n", 458 | "379/379 [==============================] - 0s 6us/step - loss: 0.6773 - acc: 0.9129\n", 459 | "Epoch 8/10\n", 460 | "379/379 [==============================] - 0s 7us/step - loss: 0.6740 - acc: 0.9182\n", 461 | "Epoch 9/10\n", 462 | "379/379 [==============================] - 0s 6us/step - loss: 0.6705 - acc: 0.9340\n", 463 | "Epoch 10/10\n", 464 | "379/379 [==============================] - 0s 6us/step - loss: 0.6667 - acc: 0.9393\n", 465 | "190/190 [==============================] - 0s 589us/step\n", 466 | "379/379 [==============================] - 0s 3us/step\n", 467 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam, total= 0.9s\n", 468 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam \n", 469 | "Epoch 1/10\n", 470 | "379/379 [==============================] - 1s 1ms/step - loss: 0.6826 - acc: 0.8021\n", 471 | "Epoch 2/10\n", 472 | "379/379 [==============================] - 0s 13us/step - loss: 0.6792 - acc: 0.8311\n", 473 | "Epoch 3/10\n", 474 | "379/379 [==============================] - 0s 4us/step - loss: 0.6765 - acc: 0.8443\n", 475 | "Epoch 4/10\n", 476 | "379/379 [==============================] - 0s 7us/step - loss: 0.6739 - acc: 0.8470\n", 477 | "Epoch 5/10\n", 478 | "379/379 [==============================] - 0s 5us/step - loss: 0.6711 - acc: 0.8602\n", 479 | "Epoch 6/10\n", 480 | "379/379 [==============================] - 0s 6us/step - loss: 0.6681 - acc: 0.8602\n", 481 | "Epoch 7/10\n", 482 | "379/379 [==============================] - 0s 6us/step - loss: 0.6649 - acc: 0.8760\n", 483 | "Epoch 8/10\n", 484 | "379/379 [==============================] - 0s 5us/step - loss: 0.6614 - acc: 0.8839\n", 485 | "Epoch 9/10\n", 486 | "379/379 [==============================] - 0s 8us/step - loss: 0.6577 - acc: 0.8865\n", 487 | "Epoch 10/10\n", 488 | "379/379 [==============================] - 0s 6us/step - loss: 0.6537 - acc: 0.8918\n", 489 | "190/190 [==============================] - 0s 650us/step\n", 490 | "379/379 [==============================] - 0s 11us/step\n", 491 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam, total= 1.1s\n", 492 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam \n", 493 | "Epoch 1/10\n", 494 | "380/380 [==============================] - 0s 1ms/step - loss: 0.6917 - acc: 0.5026\n", 495 | "Epoch 2/10\n", 496 | "380/380 [==============================] - 0s 6us/step - loss: 0.6889 - acc: 0.6842\n", 497 | "Epoch 3/10\n", 498 | "380/380 [==============================] - 0s 6us/step - loss: 0.6867 - acc: 0.7895\n", 499 | "Epoch 4/10\n", 500 | "380/380 [==============================] - 0s 8us/step - loss: 0.6846 - acc: 0.8421\n", 501 | "Epoch 5/10\n", 502 | "380/380 [==============================] - 0s 5us/step - loss: 0.6824 - acc: 0.8605\n", 503 | "Epoch 6/10\n", 504 | "380/380 [==============================] - 0s 7us/step - loss: 0.6800 - acc: 0.8605\n", 505 | "Epoch 7/10\n", 506 | "380/380 [==============================] - 0s 6us/step - loss: 0.6774 - acc: 0.8658\n", 507 | "Epoch 8/10\n", 508 | "380/380 [==============================] - 0s 6us/step - loss: 0.6745 - acc: 0.8684\n", 509 | "Epoch 9/10\n", 510 | "380/380 [==============================] - 0s 5us/step - loss: 0.6714 - acc: 0.8868\n", 511 | "Epoch 10/10\n", 512 | "380/380 [==============================] - 0s 6us/step - loss: 0.6679 - acc: 0.8895\n", 513 | "189/189 [==============================] - 0s 738us/step\n", 514 | "380/380 [==============================] - 0s 10us/step\n", 515 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam, total= 1.0s\n", 516 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam \n", 517 | "Epoch 1/10\n", 518 | "379/379 [==============================] - 0s 1ms/step - loss: 0.6942 - acc: 0.4222\n", 519 | "Epoch 2/10\n", 520 | "379/379 [==============================] - 0s 11us/step - loss: 0.6930 - acc: 0.5858\n", 521 | "Epoch 3/10\n", 522 | "379/379 [==============================] - 0s 5us/step - loss: 0.6918 - acc: 0.6702\n", 523 | "Epoch 4/10\n", 524 | "379/379 [==============================] - 0s 6us/step - loss: 0.6905 - acc: 0.7388\n", 525 | "Epoch 5/10\n", 526 | "379/379 [==============================] - 0s 8us/step - loss: 0.6893 - acc: 0.7863\n", 527 | "Epoch 6/10\n", 528 | "379/379 [==============================] - 0s 6us/step - loss: 0.6880 - acc: 0.8206\n", 529 | "Epoch 7/10\n", 530 | "379/379 [==============================] - 0s 7us/step - loss: 0.6867 - acc: 0.8391\n", 531 | "Epoch 8/10\n", 532 | "379/379 [==============================] - 0s 7us/step - loss: 0.6853 - acc: 0.8575\n", 533 | "Epoch 9/10\n", 534 | "379/379 [==============================] - 0s 5us/step - loss: 0.6839 - acc: 0.8654\n", 535 | "Epoch 10/10\n", 536 | "379/379 [==============================] - 0s 6us/step - loss: 0.6824 - acc: 0.8760\n", 537 | "190/190 [==============================] - 0s 735us/step\n", 538 | "379/379 [==============================] - 0s 10us/step\n", 539 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam, total= 1.0s\n", 540 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam \n", 541 | "Epoch 1/10\n", 542 | "379/379 [==============================] - 0s 1ms/step - loss: 0.6929 - acc: 0.4934\n", 543 | "Epoch 2/10\n", 544 | "379/379 [==============================] - 0s 5us/step - loss: 0.6917 - acc: 0.6042\n", 545 | "Epoch 3/10\n", 546 | "379/379 [==============================] - 0s 6us/step - loss: 0.6906 - acc: 0.7098\n", 547 | "Epoch 4/10\n", 548 | "379/379 [==============================] - 0s 6us/step - loss: 0.6894 - acc: 0.7757\n", 549 | "Epoch 5/10\n", 550 | "379/379 [==============================] - 0s 5us/step - loss: 0.6882 - acc: 0.8047\n", 551 | "Epoch 6/10\n", 552 | "379/379 [==============================] - 0s 5us/step - loss: 0.6869 - acc: 0.8232\n", 553 | "Epoch 7/10\n", 554 | "379/379 [==============================] - 0s 5us/step - loss: 0.6856 - acc: 0.8285\n", 555 | "Epoch 8/10\n", 556 | "379/379 [==============================] - 0s 5us/step - loss: 0.6843 - acc: 0.8443\n", 557 | "Epoch 9/10\n", 558 | "379/379 [==============================] - 0s 5us/step - loss: 0.6829 - acc: 0.8522\n", 559 | "Epoch 10/10\n", 560 | "379/379 [==============================] - 0s 5us/step - loss: 0.6814 - acc: 0.8549\n", 561 | "190/190 [==============================] - 0s 751us/step\n", 562 | "379/379 [==============================] - 0s 10us/step\n", 563 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam, total= 1.0s\n", 564 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam \n", 565 | "Epoch 1/10\n", 566 | "380/380 [==============================] - 0s 1ms/step - loss: 0.6933 - acc: 0.5737\n", 567 | "Epoch 2/10\n", 568 | "380/380 [==============================] - 0s 12us/step - loss: 0.6921 - acc: 0.7211\n", 569 | "Epoch 3/10\n", 570 | "380/380 [==============================] - 0s 5us/step - loss: 0.6909 - acc: 0.7974\n", 571 | "Epoch 4/10\n", 572 | "380/380 [==============================] - 0s 8us/step - loss: 0.6897 - acc: 0.8447\n", 573 | "Epoch 5/10\n", 574 | "380/380 [==============================] - 0s 6us/step - loss: 0.6886 - acc: 0.8579\n", 575 | "Epoch 6/10\n", 576 | "380/380 [==============================] - 0s 9us/step - loss: 0.6874 - acc: 0.8763\n", 577 | "Epoch 7/10\n", 578 | "380/380 [==============================] - 0s 5us/step - loss: 0.6862 - acc: 0.8789\n", 579 | "Epoch 8/10\n", 580 | "380/380 [==============================] - 0s 7us/step - loss: 0.6849 - acc: 0.8763\n", 581 | "Epoch 9/10\n", 582 | "380/380 [==============================] - 0s 7us/step - loss: 0.6836 - acc: 0.8868\n", 583 | "Epoch 10/10\n", 584 | "380/380 [==============================] - 0s 7us/step - loss: 0.6823 - acc: 0.8868\n", 585 | "189/189 [==============================] - 0s 832us/step\n", 586 | "380/380 [==============================] - 0s 3us/step\n", 587 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam, total= 1.2s\n", 588 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam \n", 589 | "Epoch 1/10\n", 590 | "379/379 [==============================] - 1s 1ms/step - loss: 0.6946 - acc: 0.6121\n", 591 | "Epoch 2/10\n", 592 | "379/379 [==============================] - 0s 14us/step - loss: 0.6918 - acc: 0.6755\n", 593 | "Epoch 3/10\n", 594 | "379/379 [==============================] - 0s 5us/step - loss: 0.6897 - acc: 0.6887\n", 595 | "Epoch 4/10\n", 596 | "379/379 [==============================] - 0s 8us/step - loss: 0.6877 - acc: 0.7124\n", 597 | "Epoch 5/10\n", 598 | "379/379 [==============================] - 0s 6us/step - loss: 0.6854 - acc: 0.7493\n", 599 | "Epoch 6/10\n", 600 | "379/379 [==============================] - 0s 6us/step - loss: 0.6830 - acc: 0.7968\n", 601 | "Epoch 7/10\n", 602 | "379/379 [==============================] - 0s 7us/step - loss: 0.6803 - acc: 0.8338\n", 603 | "Epoch 8/10\n", 604 | "379/379 [==============================] - 0s 5us/step - loss: 0.6772 - acc: 0.8496\n", 605 | "Epoch 9/10\n", 606 | "379/379 [==============================] - 0s 10us/step - loss: 0.6737 - acc: 0.8628\n", 607 | "Epoch 10/10\n", 608 | "379/379 [==============================] - 0s 7us/step - loss: 0.6699 - acc: 0.8918\n", 609 | "190/190 [==============================] - 0s 904us/step\n", 610 | "379/379 [==============================] - 0s 4us/step\n", 611 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam, total= 1.1s\n", 612 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam \n", 613 | "Epoch 1/10\n", 614 | "379/379 [==============================] - 1s 1ms/step - loss: 0.6953 - acc: 0.3325\n", 615 | "Epoch 2/10\n", 616 | "379/379 [==============================] - 0s 6us/step - loss: 0.6929 - acc: 0.5567\n", 617 | "Epoch 3/10\n", 618 | "379/379 [==============================] - 0s 6us/step - loss: 0.6912 - acc: 0.6675\n", 619 | "Epoch 4/10\n", 620 | "379/379 [==============================] - 0s 5us/step - loss: 0.6895 - acc: 0.7441\n", 621 | "Epoch 5/10\n", 622 | "379/379 [==============================] - 0s 8us/step - loss: 0.6879 - acc: 0.7863\n", 623 | "Epoch 6/10\n", 624 | "379/379 [==============================] - 0s 5us/step - loss: 0.6860 - acc: 0.8100\n", 625 | "Epoch 7/10\n", 626 | "379/379 [==============================] - 0s 6us/step - loss: 0.6840 - acc: 0.8338\n", 627 | "Epoch 8/10\n", 628 | "379/379 [==============================] - 0s 6us/step - loss: 0.6816 - acc: 0.8654\n", 629 | "Epoch 9/10\n", 630 | "379/379 [==============================] - 0s 12us/step - loss: 0.6789 - acc: 0.8786\n", 631 | "Epoch 10/10\n", 632 | "379/379 [==============================] - 0s 6us/step - loss: 0.6758 - acc: 0.8945\n", 633 | "190/190 [==============================] - 0s 1ms/step\n", 634 | "379/379 [==============================] - 0s 7us/step\n", 635 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam, total= 1.2s\n", 636 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam \n", 637 | "Epoch 1/10\n", 638 | "380/380 [==============================] - 1s 2ms/step - loss: 0.6941 - acc: 0.4868\n", 639 | "Epoch 2/10\n", 640 | "380/380 [==============================] - 0s 5us/step - loss: 0.6914 - acc: 0.6237\n", 641 | "Epoch 3/10\n", 642 | "380/380 [==============================] - 0s 5us/step - loss: 0.6894 - acc: 0.6868\n", 643 | "Epoch 4/10\n", 644 | "380/380 [==============================] - 0s 5us/step - loss: 0.6874 - acc: 0.7184\n", 645 | "Epoch 5/10\n", 646 | "380/380 [==============================] - 0s 5us/step - loss: 0.6854 - acc: 0.7684\n", 647 | "Epoch 6/10\n", 648 | "380/380 [==============================] - 0s 5us/step - loss: 0.6832 - acc: 0.7947\n", 649 | "Epoch 7/10\n", 650 | "380/380 [==============================] - 0s 5us/step - loss: 0.6808 - acc: 0.8053\n", 651 | "Epoch 8/10\n", 652 | "380/380 [==============================] - 0s 5us/step - loss: 0.6782 - acc: 0.8184\n", 653 | "Epoch 9/10\n", 654 | "380/380 [==============================] - 0s 5us/step - loss: 0.6753 - acc: 0.8316\n", 655 | "Epoch 10/10\n", 656 | "380/380 [==============================] - 0s 5us/step - loss: 0.6720 - acc: 0.8474\n", 657 | "189/189 [==============================] - 0s 1ms/step\n", 658 | "380/380 [==============================] - 0s 5us/step\n", 659 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam, total= 1.2s\n", 660 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam \n", 661 | "Epoch 1/10\n", 662 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6894 - acc: 0.6359\n", 663 | "Epoch 2/10\n", 664 | "379/379 [==============================] - 0s 6us/step - loss: 0.6878 - acc: 0.7520\n", 665 | "Epoch 3/10\n", 666 | "379/379 [==============================] - 0s 5us/step - loss: 0.6861 - acc: 0.8259\n", 667 | "Epoch 4/10\n", 668 | "379/379 [==============================] - 0s 4us/step - loss: 0.6845 - acc: 0.8681\n", 669 | "Epoch 5/10\n", 670 | "379/379 [==============================] - 0s 5us/step - loss: 0.6828 - acc: 0.8997\n", 671 | "Epoch 6/10\n", 672 | "379/379 [==============================] - 0s 4us/step - loss: 0.6812 - acc: 0.9235\n", 673 | "Epoch 7/10\n", 674 | "379/379 [==============================] - 0s 4us/step - loss: 0.6795 - acc: 0.9261\n", 675 | "Epoch 8/10\n", 676 | "379/379 [==============================] - 0s 5us/step - loss: 0.6778 - acc: 0.9314\n", 677 | "Epoch 9/10\n", 678 | "379/379 [==============================] - 0s 5us/step - loss: 0.6760 - acc: 0.9367\n", 679 | "Epoch 10/10\n", 680 | "379/379 [==============================] - 0s 9us/step - loss: 0.6742 - acc: 0.9420\n", 681 | "190/190 [==============================] - 0s 1ms/step\n", 682 | "379/379 [==============================] - 0s 3us/step\n", 683 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam, total= 1.2s\n", 684 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam \n", 685 | "Epoch 1/10\n", 686 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6911 - acc: 0.6201\n", 687 | "Epoch 2/10\n", 688 | "379/379 [==============================] - 0s 12us/step - loss: 0.6899 - acc: 0.7124\n", 689 | "Epoch 3/10\n", 690 | "379/379 [==============================] - 0s 6us/step - loss: 0.6887 - acc: 0.8074\n", 691 | "Epoch 4/10\n", 692 | "379/379 [==============================] - 0s 7us/step - loss: 0.6875 - acc: 0.8945\n", 693 | "Epoch 5/10\n", 694 | "379/379 [==============================] - 0s 5us/step - loss: 0.6863 - acc: 0.9050\n", 695 | "Epoch 6/10\n", 696 | "379/379 [==============================] - 0s 7us/step - loss: 0.6851 - acc: 0.9077\n", 697 | "Epoch 7/10\n", 698 | "379/379 [==============================] - 0s 9us/step - loss: 0.6838 - acc: 0.9156\n", 699 | "Epoch 8/10\n", 700 | "379/379 [==============================] - 0s 7us/step - loss: 0.6825 - acc: 0.9129\n", 701 | "Epoch 9/10\n", 702 | "379/379 [==============================] - 0s 7us/step - loss: 0.6811 - acc: 0.9182\n", 703 | "Epoch 10/10\n", 704 | "379/379 [==============================] - 0s 7us/step - loss: 0.6797 - acc: 0.9235\n", 705 | "190/190 [==============================] - 0s 1ms/step\n", 706 | "379/379 [==============================] - 0s 4us/step\n", 707 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam, total= 1.5s\n", 708 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam \n", 709 | "Epoch 1/10\n", 710 | "380/380 [==============================] - 1s 2ms/step - loss: 0.6955 - acc: 0.4605\n", 711 | "Epoch 2/10\n", 712 | "380/380 [==============================] - 0s 7us/step - loss: 0.6933 - acc: 0.5421\n", 713 | "Epoch 3/10\n", 714 | "380/380 [==============================] - 0s 7us/step - loss: 0.6911 - acc: 0.5816\n", 715 | "Epoch 4/10\n", 716 | "380/380 [==============================] - 0s 7us/step - loss: 0.6889 - acc: 0.6395\n", 717 | "Epoch 5/10\n", 718 | "380/380 [==============================] - 0s 6us/step - loss: 0.6868 - acc: 0.6974\n", 719 | "Epoch 6/10\n", 720 | "380/380 [==============================] - 0s 10us/step - loss: 0.6846 - acc: 0.7368\n", 721 | "Epoch 7/10\n", 722 | "380/380 [==============================] - 0s 7us/step - loss: 0.6824 - acc: 0.7789\n", 723 | "Epoch 8/10\n", 724 | "380/380 [==============================] - 0s 7us/step - loss: 0.6801 - acc: 0.8132\n", 725 | "Epoch 9/10\n", 726 | "380/380 [==============================] - 0s 7us/step - loss: 0.6779 - acc: 0.8395\n", 727 | "Epoch 10/10\n", 728 | "380/380 [==============================] - 0s 9us/step - loss: 0.6756 - acc: 0.8553\n", 729 | "189/189 [==============================] - 0s 1ms/step\n", 730 | "380/380 [==============================] - 0s 10us/step\n", 731 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam, total= 1.3s\n", 732 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam \n", 733 | "Epoch 1/10\n", 734 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6937 - acc: 0.4644\n", 735 | "Epoch 2/10\n", 736 | "379/379 [==============================] - 0s 6us/step - loss: 0.6908 - acc: 0.6913\n", 737 | "Epoch 3/10\n", 738 | "379/379 [==============================] - 0s 5us/step - loss: 0.6886 - acc: 0.7678\n", 739 | "Epoch 4/10\n", 740 | "379/379 [==============================] - 0s 5us/step - loss: 0.6864 - acc: 0.8100\n", 741 | "Epoch 5/10\n", 742 | "379/379 [==============================] - 0s 5us/step - loss: 0.6841 - acc: 0.8522\n", 743 | "Epoch 6/10\n", 744 | "379/379 [==============================] - 0s 6us/step - loss: 0.6816 - acc: 0.8760\n", 745 | "Epoch 7/10\n", 746 | "379/379 [==============================] - 0s 5us/step - loss: 0.6788 - acc: 0.8945\n", 747 | "Epoch 8/10\n", 748 | "379/379 [==============================] - 0s 6us/step - loss: 0.6757 - acc: 0.9050\n", 749 | "Epoch 9/10\n", 750 | "379/379 [==============================] - 0s 5us/step - loss: 0.6722 - acc: 0.9103\n", 751 | "Epoch 10/10\n", 752 | "379/379 [==============================] - 0s 6us/step - loss: 0.6685 - acc: 0.9208\n", 753 | "190/190 [==============================] - 0s 1ms/step\n", 754 | "379/379 [==============================] - 0s 10us/step\n", 755 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam, total= 1.4s\n", 756 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam \n", 757 | "Epoch 1/10\n", 758 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6981 - acc: 0.3509\n", 759 | "Epoch 2/10\n", 760 | "379/379 [==============================] - 0s 6us/step - loss: 0.6943 - acc: 0.4565\n", 761 | "Epoch 3/10\n", 762 | "379/379 [==============================] - 0s 6us/step - loss: 0.6913 - acc: 0.5567\n", 763 | "Epoch 4/10\n", 764 | "379/379 [==============================] - 0s 4us/step - loss: 0.6885 - acc: 0.6359\n", 765 | "Epoch 5/10\n", 766 | "379/379 [==============================] - 0s 5us/step - loss: 0.6856 - acc: 0.7414\n", 767 | "Epoch 6/10\n", 768 | "379/379 [==============================] - 0s 5us/step - loss: 0.6825 - acc: 0.7810\n", 769 | "Epoch 7/10\n", 770 | "379/379 [==============================] - 0s 6us/step - loss: 0.6793 - acc: 0.8206\n", 771 | "Epoch 8/10\n", 772 | "379/379 [==============================] - 0s 6us/step - loss: 0.6757 - acc: 0.8443\n", 773 | "Epoch 9/10\n", 774 | "379/379 [==============================] - 0s 8us/step - loss: 0.6719 - acc: 0.8575\n", 775 | "Epoch 10/10\n", 776 | "379/379 [==============================] - 0s 8us/step - loss: 0.6679 - acc: 0.8839\n", 777 | "190/190 [==============================] - 0s 2ms/step\n", 778 | "379/379 [==============================] - 0s 11us/step\n", 779 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam, total= 1.5s\n", 780 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam \n", 781 | "Epoch 1/10\n", 782 | "380/380 [==============================] - 1s 2ms/step - loss: 0.7001 - acc: 0.2395\n", 783 | "Epoch 2/10\n", 784 | "380/380 [==============================] - 0s 8us/step - loss: 0.6953 - acc: 0.3658\n", 785 | "Epoch 3/10\n", 786 | "380/380 [==============================] - 0s 5us/step - loss: 0.6919 - acc: 0.5158\n", 787 | "Epoch 4/10\n", 788 | "380/380 [==============================] - 0s 5us/step - loss: 0.6888 - acc: 0.6368\n", 789 | "Epoch 5/10\n", 790 | "380/380 [==============================] - 0s 6us/step - loss: 0.6856 - acc: 0.7316\n", 791 | "Epoch 6/10\n", 792 | "380/380 [==============================] - 0s 6us/step - loss: 0.6824 - acc: 0.7658\n", 793 | "Epoch 7/10\n", 794 | "380/380 [==============================] - 0s 7us/step - loss: 0.6790 - acc: 0.7763\n", 795 | "Epoch 8/10\n", 796 | "380/380 [==============================] - 0s 13us/step - loss: 0.6755 - acc: 0.8132\n", 797 | "Epoch 9/10\n", 798 | "380/380 [==============================] - 0s 5us/step - loss: 0.6717 - acc: 0.8132\n", 799 | "Epoch 10/10\n", 800 | "380/380 [==============================] - 0s 6us/step - loss: 0.6678 - acc: 0.8316\n", 801 | "189/189 [==============================] - 0s 2ms/step\n", 802 | "380/380 [==============================] - 0s 4us/step\n", 803 | "[CV] activation=relu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam, total= 1.5s\n", 804 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam \n", 805 | "Epoch 1/10\n", 806 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6915 - acc: 0.7520\n", 807 | "Epoch 2/10\n", 808 | "379/379 [==============================] - 0s 12us/step - loss: 0.6901 - acc: 0.8311\n", 809 | "Epoch 3/10\n", 810 | "379/379 [==============================] - 0s 7us/step - loss: 0.6887 - acc: 0.8813\n", 811 | "Epoch 4/10\n", 812 | "379/379 [==============================] - 0s 7us/step - loss: 0.6872 - acc: 0.8918\n", 813 | "Epoch 5/10\n", 814 | "379/379 [==============================] - 0s 7us/step - loss: 0.6857 - acc: 0.9050\n", 815 | "Epoch 6/10\n", 816 | "379/379 [==============================] - 0s 8us/step - loss: 0.6841 - acc: 0.9156\n", 817 | "Epoch 7/10\n", 818 | "379/379 [==============================] - 0s 7us/step - loss: 0.6825 - acc: 0.9156\n", 819 | "Epoch 8/10\n", 820 | "379/379 [==============================] - 0s 6us/step - loss: 0.6807 - acc: 0.9182\n", 821 | "Epoch 9/10\n", 822 | "379/379 [==============================] - 0s 6us/step - loss: 0.6790 - acc: 0.9208\n", 823 | "Epoch 10/10\n", 824 | "379/379 [==============================] - 0s 8us/step - loss: 0.6771 - acc: 0.9261\n", 825 | "190/190 [==============================] - 0s 1ms/step\n", 826 | "379/379 [==============================] - 0s 4us/step\n", 827 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam, total= 1.6s\n", 828 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam \n", 829 | "Epoch 1/10\n", 830 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6990 - acc: 0.0739\n", 831 | "Epoch 2/10\n", 832 | "379/379 [==============================] - 0s 12us/step - loss: 0.6974 - acc: 0.1108\n", 833 | "Epoch 3/10\n", 834 | "379/379 [==============================] - 0s 6us/step - loss: 0.6959 - acc: 0.2375\n", 835 | "Epoch 4/10\n", 836 | "379/379 [==============================] - 0s 5us/step - loss: 0.6945 - acc: 0.4222\n", 837 | "Epoch 5/10\n", 838 | "379/379 [==============================] - 0s 7us/step - loss: 0.6930 - acc: 0.5673\n", 839 | "Epoch 6/10\n", 840 | "379/379 [==============================] - 0s 7us/step - loss: 0.6915 - acc: 0.6570\n", 841 | "Epoch 7/10\n", 842 | "379/379 [==============================] - 0s 7us/step - loss: 0.6901 - acc: 0.7124\n", 843 | "Epoch 8/10\n", 844 | "379/379 [==============================] - 0s 7us/step - loss: 0.6886 - acc: 0.7652\n", 845 | "Epoch 9/10\n", 846 | "379/379 [==============================] - 0s 7us/step - loss: 0.6871 - acc: 0.7942\n", 847 | "Epoch 10/10\n", 848 | "379/379 [==============================] - 0s 7us/step - loss: 0.6855 - acc: 0.8285\n", 849 | "190/190 [==============================] - 0s 2ms/step\n", 850 | "379/379 [==============================] - 0s 4us/step\n", 851 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam, total= 1.5s\n", 852 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam \n", 853 | "Epoch 1/10\n", 854 | "380/380 [==============================] - 1s 2ms/step - loss: 0.6920 - acc: 0.6000\n", 855 | "Epoch 2/10\n", 856 | "380/380 [==============================] - 0s 6us/step - loss: 0.6904 - acc: 0.6974\n", 857 | "Epoch 3/10\n", 858 | "380/380 [==============================] - 0s 6us/step - loss: 0.6888 - acc: 0.7816\n", 859 | "Epoch 4/10\n", 860 | "380/380 [==============================] - 0s 5us/step - loss: 0.6872 - acc: 0.8158\n", 861 | "Epoch 5/10\n", 862 | "380/380 [==============================] - 0s 5us/step - loss: 0.6855 - acc: 0.8447\n", 863 | "Epoch 6/10\n", 864 | "380/380 [==============================] - 0s 6us/step - loss: 0.6837 - acc: 0.8579\n", 865 | "Epoch 7/10\n", 866 | "380/380 [==============================] - 0s 8us/step - loss: 0.6819 - acc: 0.8763\n", 867 | "Epoch 8/10\n", 868 | "380/380 [==============================] - 0s 8us/step - loss: 0.6801 - acc: 0.8816\n", 869 | "Epoch 9/10\n", 870 | "380/380 [==============================] - 0s 11us/step - loss: 0.6782 - acc: 0.8816\n", 871 | "Epoch 10/10\n", 872 | "380/380 [==============================] - 0s 7us/step - loss: 0.6763 - acc: 0.8816\n", 873 | "189/189 [==============================] - 0s 2ms/step\n", 874 | "380/380 [==============================] - 0s 4us/step\n", 875 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Adam, total= 1.6s\n", 876 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam \n", 877 | "Epoch 1/10\n", 878 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6928 - acc: 0.5778\n", 879 | "Epoch 2/10\n", 880 | "379/379 [==============================] - 0s 6us/step - loss: 0.6892 - acc: 0.8127\n", 881 | "Epoch 3/10\n", 882 | "379/379 [==============================] - 0s 6us/step - loss: 0.6863 - acc: 0.8734\n", 883 | "Epoch 4/10\n", 884 | "379/379 [==============================] - 0s 6us/step - loss: 0.6836 - acc: 0.8971\n", 885 | "Epoch 5/10\n", 886 | "379/379 [==============================] - 0s 6us/step - loss: 0.6806 - acc: 0.9103\n", 887 | "Epoch 6/10\n", 888 | "379/379 [==============================] - 0s 5us/step - loss: 0.6773 - acc: 0.9129\n", 889 | "Epoch 7/10\n", 890 | "379/379 [==============================] - 0s 6us/step - loss: 0.6736 - acc: 0.9103\n", 891 | "Epoch 8/10\n", 892 | "379/379 [==============================] - 0s 5us/step - loss: 0.6695 - acc: 0.9182\n", 893 | "Epoch 9/10\n", 894 | "379/379 [==============================] - 0s 6us/step - loss: 0.6650 - acc: 0.9235\n", 895 | "Epoch 10/10\n", 896 | "379/379 [==============================] - 0s 6us/step - loss: 0.6600 - acc: 0.9261\n", 897 | "190/190 [==============================] - 0s 2ms/step\n", 898 | "379/379 [==============================] - 0s 12us/step\n", 899 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam, total= 1.6s\n", 900 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam \n", 901 | "Epoch 1/10\n", 902 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6955 - acc: 0.2401\n", 903 | "Epoch 2/10\n", 904 | "379/379 [==============================] - 0s 13us/step - loss: 0.6922 - acc: 0.6544\n", 905 | "Epoch 3/10\n", 906 | "379/379 [==============================] - 0s 5us/step - loss: 0.6897 - acc: 0.8865\n", 907 | "Epoch 4/10\n", 908 | "379/379 [==============================] - 0s 9us/step - loss: 0.6872 - acc: 0.9077\n", 909 | "Epoch 5/10\n", 910 | "379/379 [==============================] - 0s 9us/step - loss: 0.6845 - acc: 0.9156\n", 911 | "Epoch 6/10\n", 912 | "379/379 [==============================] - 0s 7us/step - loss: 0.6815 - acc: 0.9208\n", 913 | "Epoch 7/10\n", 914 | "379/379 [==============================] - 0s 8us/step - loss: 0.6780 - acc: 0.9182\n", 915 | "Epoch 8/10\n", 916 | "379/379 [==============================] - 0s 9us/step - loss: 0.6740 - acc: 0.9235\n", 917 | "Epoch 9/10\n", 918 | "379/379 [==============================] - 0s 6us/step - loss: 0.6695 - acc: 0.9235\n", 919 | "Epoch 10/10\n", 920 | "379/379 [==============================] - 0s 6us/step - loss: 0.6646 - acc: 0.9208\n", 921 | "190/190 [==============================] - 0s 2ms/step\n", 922 | "379/379 [==============================] - 0s 11us/step\n", 923 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam, total= 1.6s\n", 924 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam \n", 925 | "Epoch 1/10\n", 926 | "380/380 [==============================] - 1s 2ms/step - loss: 0.6875 - acc: 0.7947\n", 927 | "Epoch 2/10\n", 928 | "380/380 [==============================] - 0s 12us/step - loss: 0.6831 - acc: 0.8553\n", 929 | "Epoch 3/10\n", 930 | "380/380 [==============================] - 0s 7us/step - loss: 0.6798 - acc: 0.8658\n", 931 | "Epoch 4/10\n", 932 | "380/380 [==============================] - 0s 8us/step - loss: 0.6765 - acc: 0.8737\n", 933 | "Epoch 5/10\n", 934 | "380/380 [==============================] - 0s 9us/step - loss: 0.6730 - acc: 0.8789\n", 935 | "Epoch 6/10\n", 936 | "380/380 [==============================] - 0s 7us/step - loss: 0.6692 - acc: 0.8868\n", 937 | "Epoch 7/10\n", 938 | "380/380 [==============================] - 0s 10us/step - loss: 0.6651 - acc: 0.8895\n", 939 | "Epoch 8/10\n", 940 | "380/380 [==============================] - 0s 8us/step - loss: 0.6605 - acc: 0.8921\n", 941 | "Epoch 9/10\n", 942 | "380/380 [==============================] - 0s 8us/step - loss: 0.6556 - acc: 0.8921\n", 943 | "Epoch 10/10\n", 944 | "380/380 [==============================] - 0s 7us/step - loss: 0.6502 - acc: 0.8921\n", 945 | "189/189 [==============================] - 0s 2ms/step\n", 946 | "380/380 [==============================] - 0s 5us/step\n", 947 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=uniform, optimizer=Nadam, total= 1.7s\n", 948 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam \n", 949 | "Epoch 1/10\n", 950 | "379/379 [==============================] - 1s 2ms/step - loss: 0.6873 - acc: 0.7441\n", 951 | "Epoch 2/10\n", 952 | "379/379 [==============================] - 0s 10us/step - loss: 0.6845 - acc: 0.8179\n", 953 | "Epoch 3/10\n", 954 | "379/379 [==============================] - 0s 8us/step - loss: 0.6817 - acc: 0.8602\n", 955 | "Epoch 4/10\n", 956 | "379/379 [==============================] - 0s 5us/step - loss: 0.6788 - acc: 0.8734\n", 957 | "Epoch 5/10\n", 958 | "379/379 [==============================] - 0s 8us/step - loss: 0.6760 - acc: 0.8918\n", 959 | "Epoch 6/10\n", 960 | "379/379 [==============================] - 0s 7us/step - loss: 0.6732 - acc: 0.9129\n", 961 | "Epoch 7/10\n", 962 | "379/379 [==============================] - 0s 8us/step - loss: 0.6703 - acc: 0.9156\n", 963 | "Epoch 8/10\n", 964 | "379/379 [==============================] - 0s 7us/step - loss: 0.6674 - acc: 0.9288\n", 965 | "Epoch 9/10\n", 966 | "379/379 [==============================] - 0s 8us/step - loss: 0.6645 - acc: 0.9314\n", 967 | "Epoch 10/10\n", 968 | "379/379 [==============================] - 0s 6us/step - loss: 0.6615 - acc: 0.9261\n", 969 | "190/190 [==============================] - 0s 2ms/step\n", 970 | "379/379 [==============================] - 0s 4us/step\n", 971 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam, total= 1.7s\n", 972 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam \n", 973 | "Epoch 1/10\n", 974 | "379/379 [==============================] - 1s 3ms/step - loss: 0.6905 - acc: 0.6174\n", 975 | "Epoch 2/10\n", 976 | "379/379 [==============================] - 0s 7us/step - loss: 0.6885 - acc: 0.7071\n", 977 | "Epoch 3/10\n", 978 | "379/379 [==============================] - 0s 8us/step - loss: 0.6866 - acc: 0.7520\n", 979 | "Epoch 4/10\n", 980 | "379/379 [==============================] - 0s 5us/step - loss: 0.6846 - acc: 0.7968\n", 981 | "Epoch 5/10\n", 982 | "379/379 [==============================] - 0s 7us/step - loss: 0.6826 - acc: 0.8311\n", 983 | "Epoch 6/10\n", 984 | "379/379 [==============================] - 0s 7us/step - loss: 0.6805 - acc: 0.8575\n", 985 | "Epoch 7/10\n", 986 | "379/379 [==============================] - 0s 7us/step - loss: 0.6784 - acc: 0.8707\n", 987 | "Epoch 8/10\n", 988 | "379/379 [==============================] - 0s 8us/step - loss: 0.6763 - acc: 0.8813\n", 989 | "Epoch 9/10\n", 990 | "379/379 [==============================] - 0s 7us/step - loss: 0.6741 - acc: 0.8918\n", 991 | "Epoch 10/10\n", 992 | "379/379 [==============================] - 0s 7us/step - loss: 0.6719 - acc: 0.8971\n", 993 | "190/190 [==============================] - 0s 2ms/step\n", 994 | "379/379 [==============================] - 0s 5us/step\n", 995 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam, total= 2.0s\n", 996 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam \n", 997 | "Epoch 1/10\n", 998 | "380/380 [==============================] - 1s 3ms/step - loss: 0.6870 - acc: 0.7921\n", 999 | "Epoch 2/10\n", 1000 | "380/380 [==============================] - 0s 6us/step - loss: 0.6850 - acc: 0.8184\n", 1001 | "Epoch 3/10\n", 1002 | "380/380 [==============================] - 0s 6us/step - loss: 0.6829 - acc: 0.8421\n", 1003 | "Epoch 4/10\n", 1004 | "380/380 [==============================] - 0s 5us/step - loss: 0.6809 - acc: 0.8605\n", 1005 | "Epoch 5/10\n", 1006 | "380/380 [==============================] - 0s 6us/step - loss: 0.6788 - acc: 0.8737\n", 1007 | "Epoch 6/10\n", 1008 | "380/380 [==============================] - 0s 6us/step - loss: 0.6766 - acc: 0.8816\n", 1009 | "Epoch 7/10\n", 1010 | "380/380 [==============================] - 0s 8us/step - loss: 0.6744 - acc: 0.8789\n", 1011 | "Epoch 8/10\n", 1012 | "380/380 [==============================] - 0s 7us/step - loss: 0.6722 - acc: 0.8842\n", 1013 | "Epoch 9/10\n", 1014 | "380/380 [==============================] - 0s 8us/step - loss: 0.6699 - acc: 0.8895\n", 1015 | "Epoch 10/10\n", 1016 | "380/380 [==============================] - 0s 8us/step - loss: 0.6675 - acc: 0.8895\n", 1017 | "189/189 [==============================] - 0s 2ms/step\n", 1018 | "380/380 [==============================] - 0s 4us/step\n", 1019 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Adam, total= 1.8s\n", 1020 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam \n", 1021 | "Epoch 1/10\n", 1022 | "379/379 [==============================] - 1s 3ms/step - loss: 0.7012 - acc: 0.1953\n", 1023 | "Epoch 2/10\n", 1024 | "379/379 [==============================] - 0s 13us/step - loss: 0.6969 - acc: 0.3245\n", 1025 | "Epoch 3/10\n", 1026 | "379/379 [==============================] - 0s 7us/step - loss: 0.6938 - acc: 0.5013\n", 1027 | "Epoch 4/10\n", 1028 | "379/379 [==============================] - 0s 9us/step - loss: 0.6908 - acc: 0.6860\n", 1029 | "Epoch 5/10\n", 1030 | "379/379 [==============================] - 0s 7us/step - loss: 0.6878 - acc: 0.7968\n", 1031 | "Epoch 6/10\n", 1032 | "379/379 [==============================] - 0s 10us/step - loss: 0.6846 - acc: 0.8813\n", 1033 | "Epoch 7/10\n", 1034 | "379/379 [==============================] - 0s 8us/step - loss: 0.6812 - acc: 0.9472\n", 1035 | "Epoch 8/10\n", 1036 | "379/379 [==============================] - 0s 6us/step - loss: 0.6777 - acc: 0.9604\n", 1037 | "Epoch 9/10\n", 1038 | "379/379 [==============================] - 0s 11us/step - loss: 0.6738 - acc: 0.9631\n", 1039 | "Epoch 10/10\n", 1040 | "379/379 [==============================] - 0s 8us/step - loss: 0.6696 - acc: 0.9631\n", 1041 | "190/190 [==============================] - 0s 2ms/step\n", 1042 | "379/379 [==============================] - 0s 6us/step\n", 1043 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam, total= 1.9s\n", 1044 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam \n", 1045 | "Epoch 1/10\n", 1046 | "379/379 [==============================] - 1s 3ms/step - loss: 0.6933 - acc: 0.4987\n", 1047 | "Epoch 2/10\n", 1048 | "379/379 [==============================] - 0s 6us/step - loss: 0.6891 - acc: 0.7546\n", 1049 | "Epoch 3/10\n", 1050 | "379/379 [==============================] - 0s 6us/step - loss: 0.6859 - acc: 0.8470\n", 1051 | "Epoch 4/10\n", 1052 | "379/379 [==============================] - 0s 5us/step - loss: 0.6830 - acc: 0.8865\n", 1053 | "Epoch 5/10\n", 1054 | "379/379 [==============================] - 0s 6us/step - loss: 0.6799 - acc: 0.8971\n", 1055 | "Epoch 6/10\n", 1056 | "379/379 [==============================] - 0s 6us/step - loss: 0.6768 - acc: 0.9129\n", 1057 | "Epoch 7/10\n", 1058 | "379/379 [==============================] - 0s 6us/step - loss: 0.6733 - acc: 0.9129\n", 1059 | "Epoch 8/10\n", 1060 | "379/379 [==============================] - 0s 8us/step - loss: 0.6696 - acc: 0.9208\n", 1061 | "Epoch 9/10\n", 1062 | "379/379 [==============================] - 0s 6us/step - loss: 0.6656 - acc: 0.9235\n", 1063 | "Epoch 10/10\n", 1064 | "379/379 [==============================] - 0s 6us/step - loss: 0.6612 - acc: 0.9182\n", 1065 | "190/190 [==============================] - 0s 2ms/step\n", 1066 | "379/379 [==============================] - 0s 12us/step\n", 1067 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam, total= 1.9s\n", 1068 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam \n", 1069 | "Epoch 1/10\n", 1070 | "380/380 [==============================] - 1s 3ms/step - loss: 0.6912 - acc: 0.6184\n", 1071 | "Epoch 2/10\n", 1072 | "380/380 [==============================] - 0s 12us/step - loss: 0.6872 - acc: 0.8184\n", 1073 | "Epoch 3/10\n", 1074 | "380/380 [==============================] - 0s 6us/step - loss: 0.6842 - acc: 0.8658\n", 1075 | "Epoch 4/10\n", 1076 | "380/380 [==============================] - 0s 7us/step - loss: 0.6813 - acc: 0.9079\n", 1077 | "Epoch 5/10\n", 1078 | "380/380 [==============================] - 0s 6us/step - loss: 0.6781 - acc: 0.9237\n", 1079 | "Epoch 6/10\n", 1080 | "380/380 [==============================] - 0s 8us/step - loss: 0.6747 - acc: 0.9368\n", 1081 | "Epoch 7/10\n", 1082 | "380/380 [==============================] - 0s 11us/step - loss: 0.6708 - acc: 0.9368\n", 1083 | "Epoch 8/10\n", 1084 | "380/380 [==============================] - 0s 6us/step - loss: 0.6666 - acc: 0.9342\n", 1085 | "Epoch 9/10\n", 1086 | "380/380 [==============================] - 0s 8us/step - loss: 0.6620 - acc: 0.9316\n", 1087 | "Epoch 10/10\n", 1088 | "380/380 [==============================] - 0s 7us/step - loss: 0.6569 - acc: 0.9263\n", 1089 | "189/189 [==============================] - 0s 2ms/step\n", 1090 | "380/380 [==============================] - 0s 5us/step\n", 1091 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=8, kernel_initializer=normal, optimizer=Nadam, total= 2.0s\n", 1092 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam \n", 1093 | "Epoch 1/10\n", 1094 | "379/379 [==============================] - 1s 3ms/step - loss: 0.6899 - acc: 0.6755\n", 1095 | "Epoch 2/10\n", 1096 | "379/379 [==============================] - 0s 12us/step - loss: 0.6877 - acc: 0.8259\n", 1097 | "Epoch 3/10\n", 1098 | "379/379 [==============================] - 0s 6us/step - loss: 0.6855 - acc: 0.9077\n", 1099 | "Epoch 4/10\n", 1100 | "379/379 [==============================] - 0s 10us/step - loss: 0.6833 - acc: 0.9420\n", 1101 | "Epoch 5/10\n", 1102 | "379/379 [==============================] - 0s 6us/step - loss: 0.6811 - acc: 0.9604\n", 1103 | "Epoch 6/10\n", 1104 | "379/379 [==============================] - 0s 6us/step - loss: 0.6789 - acc: 0.9631\n", 1105 | "Epoch 7/10\n", 1106 | "379/379 [==============================] - 0s 10us/step - loss: 0.6766 - acc: 0.9604\n", 1107 | "Epoch 8/10\n", 1108 | "379/379 [==============================] - 0s 8us/step - loss: 0.6743 - acc: 0.9551\n", 1109 | "Epoch 9/10\n", 1110 | "379/379 [==============================] - 0s 8us/step - loss: 0.6719 - acc: 0.9551\n", 1111 | "Epoch 10/10\n", 1112 | "379/379 [==============================] - 0s 8us/step - loss: 0.6695 - acc: 0.9551\n", 1113 | "190/190 [==============================] - 0s 2ms/step\n", 1114 | "379/379 [==============================] - 0s 12us/step\n", 1115 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam, total= 2.0s\n", 1116 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam \n", 1117 | "Epoch 1/10\n", 1118 | "379/379 [==============================] - 1s 3ms/step - loss: 0.6934 - acc: 0.4037\n", 1119 | "Epoch 2/10\n", 1120 | "379/379 [==============================] - 0s 13us/step - loss: 0.6917 - acc: 0.6570\n", 1121 | "Epoch 3/10\n", 1122 | "379/379 [==============================] - 0s 6us/step - loss: 0.6901 - acc: 0.8496\n", 1123 | "Epoch 4/10\n", 1124 | "379/379 [==============================] - 0s 10us/step - loss: 0.6884 - acc: 0.9235\n", 1125 | "Epoch 5/10\n", 1126 | "379/379 [==============================] - 0s 6us/step - loss: 0.6867 - acc: 0.9367\n", 1127 | "Epoch 6/10\n", 1128 | "379/379 [==============================] - 0s 9us/step - loss: 0.6849 - acc: 0.9578\n", 1129 | "Epoch 7/10\n", 1130 | "379/379 [==============================] - 0s 6us/step - loss: 0.6831 - acc: 0.9604\n", 1131 | "Epoch 8/10\n", 1132 | "379/379 [==============================] - 0s 8us/step - loss: 0.6812 - acc: 0.9578\n", 1133 | "Epoch 9/10\n", 1134 | "379/379 [==============================] - 0s 8us/step - loss: 0.6793 - acc: 0.9499\n", 1135 | "Epoch 10/10\n", 1136 | "379/379 [==============================] - 0s 10us/step - loss: 0.6773 - acc: 0.9420\n", 1137 | "190/190 [==============================] - 0s 2ms/step\n", 1138 | "379/379 [==============================] - 0s 12us/step\n", 1139 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam, total= 2.0s\n", 1140 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam \n", 1141 | "Epoch 1/10\n", 1142 | "380/380 [==============================] - 1s 3ms/step - loss: 0.6986 - acc: 0.1474\n", 1143 | "Epoch 2/10\n", 1144 | "380/380 [==============================] - 0s 7us/step - loss: 0.6967 - acc: 0.2474\n", 1145 | "Epoch 3/10\n", 1146 | "380/380 [==============================] - 0s 8us/step - loss: 0.6948 - acc: 0.4000\n", 1147 | "Epoch 4/10\n", 1148 | "380/380 [==============================] - 0s 6us/step - loss: 0.6929 - acc: 0.5395\n", 1149 | "Epoch 5/10\n", 1150 | "380/380 [==============================] - 0s 7us/step - loss: 0.6910 - acc: 0.6289\n", 1151 | "Epoch 6/10\n", 1152 | "380/380 [==============================] - 0s 5us/step - loss: 0.6891 - acc: 0.7211\n", 1153 | "Epoch 7/10\n", 1154 | "380/380 [==============================] - 0s 6us/step - loss: 0.6872 - acc: 0.7947\n", 1155 | "Epoch 8/10\n", 1156 | "380/380 [==============================] - 0s 7us/step - loss: 0.6852 - acc: 0.8342\n", 1157 | "Epoch 9/10\n", 1158 | "380/380 [==============================] - 0s 6us/step - loss: 0.6832 - acc: 0.8526\n", 1159 | "Epoch 10/10\n", 1160 | "380/380 [==============================] - 0s 8us/step - loss: 0.6811 - acc: 0.8711\n", 1161 | "189/189 [==============================] - 0s 2ms/step\n", 1162 | "380/380 [==============================] - 0s 4us/step\n", 1163 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Adam, total= 2.1s\n", 1164 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam \n", 1165 | "Epoch 1/10\n", 1166 | "379/379 [==============================] - 1s 3ms/step - loss: 0.6948 - acc: 0.3298\n", 1167 | "Epoch 2/10\n", 1168 | "379/379 [==============================] - 0s 13us/step - loss: 0.6908 - acc: 0.7784\n", 1169 | "Epoch 3/10\n", 1170 | "379/379 [==============================] - 0s 6us/step - loss: 0.6879 - acc: 0.9288\n", 1171 | "Epoch 4/10\n", 1172 | "379/379 [==============================] - 0s 10us/step - loss: 0.6851 - acc: 0.9578\n", 1173 | "Epoch 5/10\n", 1174 | "379/379 [==============================] - 0s 8us/step - loss: 0.6821 - acc: 0.9736\n", 1175 | "Epoch 6/10\n", 1176 | "379/379 [==============================] - 0s 7us/step - loss: 0.6789 - acc: 0.9657\n", 1177 | "Epoch 7/10\n", 1178 | "379/379 [==============================] - 0s 12us/step - loss: 0.6753 - acc: 0.9657\n", 1179 | "Epoch 8/10\n", 1180 | "379/379 [==============================] - 0s 10us/step - loss: 0.6712 - acc: 0.9657\n", 1181 | "Epoch 9/10\n", 1182 | "379/379 [==============================] - 0s 11us/step - loss: 0.6667 - acc: 0.9631\n", 1183 | "Epoch 10/10\n", 1184 | "379/379 [==============================] - 0s 8us/step - loss: 0.6616 - acc: 0.9604\n", 1185 | "190/190 [==============================] - 0s 3ms/step\n", 1186 | "379/379 [==============================] - 0s 4us/step\n", 1187 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam, total= 2.2s\n", 1188 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam \n", 1189 | "Epoch 1/10\n", 1190 | "379/379 [==============================] - 1s 3ms/step - loss: 0.6978 - acc: 0.1372\n", 1191 | "Epoch 2/10\n", 1192 | "379/379 [==============================] - 0s 11us/step - loss: 0.6944 - acc: 0.3219\n", 1193 | "Epoch 3/10\n", 1194 | "379/379 [==============================] - 0s 7us/step - loss: 0.6919 - acc: 0.6491\n", 1195 | "Epoch 4/10\n", 1196 | "379/379 [==============================] - 0s 8us/step - loss: 0.6894 - acc: 0.8259\n", 1197 | "Epoch 5/10\n", 1198 | "379/379 [==============================] - 0s 9us/step - loss: 0.6868 - acc: 0.8786\n", 1199 | "Epoch 6/10\n", 1200 | "379/379 [==============================] - 0s 8us/step - loss: 0.6838 - acc: 0.9024\n", 1201 | "Epoch 7/10\n", 1202 | "379/379 [==============================] - 0s 9us/step - loss: 0.6805 - acc: 0.9129\n", 1203 | "Epoch 8/10\n", 1204 | "379/379 [==============================] - 0s 10us/step - loss: 0.6769 - acc: 0.9182\n", 1205 | "Epoch 9/10\n", 1206 | "379/379 [==============================] - 0s 10us/step - loss: 0.6729 - acc: 0.9182\n", 1207 | "Epoch 10/10\n", 1208 | "379/379 [==============================] - 0s 11us/step - loss: 0.6683 - acc: 0.9182\n", 1209 | "190/190 [==============================] - 0s 3ms/step\n", 1210 | "379/379 [==============================] - 0s 5us/step\n", 1211 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam, total= 2.5s\n", 1212 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam \n", 1213 | "Epoch 1/10\n", 1214 | "380/380 [==============================] - 1s 3ms/step - loss: 0.6935 - acc: 0.4579\n", 1215 | "Epoch 2/10\n", 1216 | "380/380 [==============================] - 0s 11us/step - loss: 0.6908 - acc: 0.7763\n", 1217 | "Epoch 3/10\n", 1218 | "380/380 [==============================] - 0s 6us/step - loss: 0.6887 - acc: 0.8474\n", 1219 | "Epoch 4/10\n", 1220 | "380/380 [==============================] - 0s 7us/step - loss: 0.6865 - acc: 0.8816\n", 1221 | "Epoch 5/10\n", 1222 | "380/380 [==============================] - 0s 7us/step - loss: 0.6841 - acc: 0.9026\n", 1223 | "Epoch 6/10\n", 1224 | "380/380 [==============================] - 0s 12us/step - loss: 0.6813 - acc: 0.9132\n", 1225 | "Epoch 7/10\n", 1226 | "380/380 [==============================] - 0s 8us/step - loss: 0.6781 - acc: 0.9184\n", 1227 | "Epoch 8/10\n", 1228 | "380/380 [==============================] - 0s 8us/step - loss: 0.6744 - acc: 0.9184\n", 1229 | "Epoch 9/10\n", 1230 | "380/380 [==============================] - 0s 9us/step - loss: 0.6701 - acc: 0.9211\n", 1231 | "Epoch 10/10\n", 1232 | "380/380 [==============================] - 0s 10us/step - loss: 0.6653 - acc: 0.9211\n", 1233 | "189/189 [==============================] - 1s 3ms/step\n", 1234 | "380/380 [==============================] - 0s 4us/step\n", 1235 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=uniform, optimizer=Nadam, total= 2.3s\n", 1236 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam \n", 1237 | "Epoch 1/10\n", 1238 | "379/379 [==============================] - 1s 3ms/step - loss: 0.6778 - acc: 0.9103\n", 1239 | "Epoch 2/10\n", 1240 | "379/379 [==============================] - 0s 11us/step - loss: 0.6752 - acc: 0.9182\n", 1241 | "Epoch 3/10\n", 1242 | "379/379 [==============================] - 0s 7us/step - loss: 0.6726 - acc: 0.9288\n", 1243 | "Epoch 4/10\n", 1244 | "379/379 [==============================] - 0s 6us/step - loss: 0.6698 - acc: 0.9314\n", 1245 | "Epoch 5/10\n", 1246 | "379/379 [==============================] - 0s 7us/step - loss: 0.6671 - acc: 0.9288\n", 1247 | "Epoch 6/10\n", 1248 | "379/379 [==============================] - 0s 8us/step - loss: 0.6643 - acc: 0.9288\n", 1249 | "Epoch 7/10\n", 1250 | "379/379 [==============================] - 0s 13us/step - loss: 0.6615 - acc: 0.9288\n", 1251 | "Epoch 8/10\n", 1252 | "379/379 [==============================] - 0s 6us/step - loss: 0.6585 - acc: 0.9288\n", 1253 | "Epoch 9/10\n", 1254 | "379/379 [==============================] - 0s 8us/step - loss: 0.6556 - acc: 0.9314\n", 1255 | "Epoch 10/10\n", 1256 | "379/379 [==============================] - 0s 9us/step - loss: 0.6525 - acc: 0.9288\n", 1257 | "190/190 [==============================] - 1s 3ms/step\n", 1258 | "379/379 [==============================] - 0s 12us/step\n", 1259 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam, total= 2.3s\n", 1260 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam \n", 1261 | "Epoch 1/10\n", 1262 | "379/379 [==============================] - 1s 4ms/step - loss: 0.6796 - acc: 0.7546\n", 1263 | "Epoch 2/10\n", 1264 | "379/379 [==============================] - 0s 7us/step - loss: 0.6768 - acc: 0.7968\n", 1265 | "Epoch 3/10\n", 1266 | "379/379 [==============================] - 0s 7us/step - loss: 0.6740 - acc: 0.8259\n", 1267 | "Epoch 4/10\n", 1268 | "379/379 [==============================] - 0s 5us/step - loss: 0.6712 - acc: 0.8470\n", 1269 | "Epoch 5/10\n", 1270 | "379/379 [==============================] - 0s 8us/step - loss: 0.6684 - acc: 0.8628\n", 1271 | "Epoch 6/10\n", 1272 | "379/379 [==============================] - 0s 7us/step - loss: 0.6656 - acc: 0.8813\n", 1273 | "Epoch 7/10\n", 1274 | "379/379 [==============================] - 0s 9us/step - loss: 0.6628 - acc: 0.8839\n", 1275 | "Epoch 8/10\n", 1276 | "379/379 [==============================] - 0s 8us/step - loss: 0.6600 - acc: 0.8865\n", 1277 | "Epoch 9/10\n", 1278 | "379/379 [==============================] - 0s 7us/step - loss: 0.6572 - acc: 0.8918\n", 1279 | "Epoch 10/10\n", 1280 | "379/379 [==============================] - 0s 8us/step - loss: 0.6544 - acc: 0.8945\n", 1281 | "190/190 [==============================] - 1s 3ms/step\n", 1282 | "379/379 [==============================] - 0s 13us/step\n", 1283 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam, total= 2.3s\n", 1284 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam \n", 1285 | "Epoch 1/10\n", 1286 | "380/380 [==============================] - 1s 4ms/step - loss: 0.6943 - acc: 0.4132\n", 1287 | "Epoch 2/10\n", 1288 | "380/380 [==============================] - 0s 11us/step - loss: 0.6912 - acc: 0.5132\n", 1289 | "Epoch 3/10\n", 1290 | "380/380 [==============================] - 0s 9us/step - loss: 0.6881 - acc: 0.6316\n", 1291 | "Epoch 4/10\n", 1292 | "380/380 [==============================] - 0s 9us/step - loss: 0.6851 - acc: 0.7263\n", 1293 | "Epoch 5/10\n", 1294 | "380/380 [==============================] - 0s 7us/step - loss: 0.6820 - acc: 0.7868\n", 1295 | "Epoch 6/10\n", 1296 | "380/380 [==============================] - 0s 8us/step - loss: 0.6789 - acc: 0.8289\n", 1297 | "Epoch 7/10\n", 1298 | "380/380 [==============================] - 0s 9us/step - loss: 0.6757 - acc: 0.8579\n", 1299 | "Epoch 8/10\n", 1300 | "380/380 [==============================] - 0s 10us/step - loss: 0.6726 - acc: 0.8789\n", 1301 | "Epoch 9/10\n", 1302 | "380/380 [==============================] - 0s 10us/step - loss: 0.6694 - acc: 0.8895\n", 1303 | "Epoch 10/10\n", 1304 | "380/380 [==============================] - 0s 7us/step - loss: 0.6661 - acc: 0.9000\n", 1305 | "189/189 [==============================] - 1s 3ms/step\n", 1306 | "380/380 [==============================] - 0s 5us/step\n", 1307 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Adam, total= 2.4s\n", 1308 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam \n", 1309 | "Epoch 1/10\n", 1310 | "379/379 [==============================] - 1s 4ms/step - loss: 0.6934 - acc: 0.5330\n", 1311 | "Epoch 2/10\n", 1312 | "379/379 [==============================] - 0s 12us/step - loss: 0.6872 - acc: 0.6807\n", 1313 | "Epoch 3/10\n", 1314 | "379/379 [==============================] - 0s 8us/step - loss: 0.6825 - acc: 0.7203\n", 1315 | "Epoch 4/10\n", 1316 | "379/379 [==============================] - 0s 8us/step - loss: 0.6780 - acc: 0.7652\n", 1317 | "Epoch 5/10\n", 1318 | "379/379 [==============================] - 0s 9us/step - loss: 0.6735 - acc: 0.7889\n", 1319 | "Epoch 6/10\n", 1320 | "379/379 [==============================] - 0s 8us/step - loss: 0.6686 - acc: 0.8259\n", 1321 | "Epoch 7/10\n", 1322 | "379/379 [==============================] - 0s 6us/step - loss: 0.6635 - acc: 0.8602\n", 1323 | "Epoch 8/10\n", 1324 | "379/379 [==============================] - 0s 7us/step - loss: 0.6581 - acc: 0.8786\n", 1325 | "Epoch 9/10\n", 1326 | "379/379 [==============================] - 0s 9us/step - loss: 0.6523 - acc: 0.8865\n", 1327 | "Epoch 10/10\n", 1328 | "379/379 [==============================] - 0s 9us/step - loss: 0.6461 - acc: 0.8945\n", 1329 | "190/190 [==============================] - 1s 3ms/step\n", 1330 | "379/379 [==============================] - 0s 8us/step\n", 1331 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam, total= 2.5s\n", 1332 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam \n", 1333 | "Epoch 1/10\n", 1334 | "379/379 [==============================] - 2s 4ms/step - loss: 0.6937 - acc: 0.5092\n", 1335 | "Epoch 2/10\n", 1336 | "379/379 [==============================] - 0s 11us/step - loss: 0.6872 - acc: 0.6834\n", 1337 | "Epoch 3/10\n", 1338 | "379/379 [==============================] - 0s 8us/step - loss: 0.6823 - acc: 0.7704\n", 1339 | "Epoch 4/10\n", 1340 | "379/379 [==============================] - 0s 7us/step - loss: 0.6776 - acc: 0.8127\n", 1341 | "Epoch 5/10\n", 1342 | "379/379 [==============================] - 0s 9us/step - loss: 0.6727 - acc: 0.8522\n", 1343 | "Epoch 6/10\n", 1344 | "379/379 [==============================] - 0s 8us/step - loss: 0.6676 - acc: 0.8813\n", 1345 | "Epoch 7/10\n", 1346 | "379/379 [==============================] - 0s 9us/step - loss: 0.6622 - acc: 0.8865\n", 1347 | "Epoch 8/10\n", 1348 | "379/379 [==============================] - 0s 9us/step - loss: 0.6565 - acc: 0.9050\n", 1349 | "Epoch 9/10\n", 1350 | "379/379 [==============================] - 0s 9us/step - loss: 0.6505 - acc: 0.9129\n", 1351 | "Epoch 10/10\n", 1352 | "379/379 [==============================] - 0s 9us/step - loss: 0.6442 - acc: 0.9182\n", 1353 | "190/190 [==============================] - 1s 3ms/step\n", 1354 | "379/379 [==============================] - 0s 5us/step\n", 1355 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam, total= 2.5s\n", 1356 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam \n", 1357 | "Epoch 1/10\n", 1358 | "380/380 [==============================] - 1s 4ms/step - loss: 0.6903 - acc: 0.5868\n", 1359 | "Epoch 2/10\n", 1360 | "380/380 [==============================] - 0s 12us/step - loss: 0.6828 - acc: 0.7158\n", 1361 | "Epoch 3/10\n", 1362 | "380/380 [==============================] - 0s 9us/step - loss: 0.6772 - acc: 0.7842\n", 1363 | "Epoch 4/10\n", 1364 | "380/380 [==============================] - 0s 8us/step - loss: 0.6718 - acc: 0.8263\n", 1365 | "Epoch 5/10\n", 1366 | "380/380 [==============================] - 0s 8us/step - loss: 0.6663 - acc: 0.8658\n", 1367 | "Epoch 6/10\n", 1368 | "380/380 [==============================] - 0s 9us/step - loss: 0.6606 - acc: 0.8763\n", 1369 | "Epoch 7/10\n", 1370 | "380/380 [==============================] - 0s 10us/step - loss: 0.6547 - acc: 0.8842\n", 1371 | "Epoch 8/10\n", 1372 | "380/380 [==============================] - 0s 8us/step - loss: 0.6483 - acc: 0.8921\n", 1373 | "Epoch 9/10\n", 1374 | "380/380 [==============================] - 0s 9us/step - loss: 0.6417 - acc: 0.8921\n", 1375 | "Epoch 10/10\n", 1376 | "380/380 [==============================] - 0s 8us/step - loss: 0.6346 - acc: 0.9000\n", 1377 | "189/189 [==============================] - 1s 3ms/step\n", 1378 | "380/380 [==============================] - 0s 13us/step\n", 1379 | "[CV] activation=elu, batch_size=1024, dropout_rate=0.0, epochs=10, first_neuron=9, kernel_initializer=normal, optimizer=Nadam, total= 2.6s\n" 1380 | ] 1381 | }, 1382 | { 1383 | "name": "stderr", 1384 | "output_type": "stream", 1385 | "text": [ 1386 | "[Parallel(n_jobs=1)]: Done 48 out of 48 | elapsed: 1.2min finished\n" 1387 | ] 1388 | }, 1389 | { 1390 | "name": "stdout", 1391 | "output_type": "stream", 1392 | "text": [ 1393 | "Epoch 1/10\n", 1394 | "569/569 [==============================] - 2s 3ms/step - loss: 0.6932 - acc: 0.4780\n", 1395 | "Epoch 2/10\n", 1396 | "569/569 [==============================] - 0s 10us/step - loss: 0.6910 - acc: 0.6116\n", 1397 | "Epoch 3/10\n", 1398 | "569/569 [==============================] - 0s 7us/step - loss: 0.6888 - acc: 0.7241\n", 1399 | "Epoch 4/10\n", 1400 | "569/569 [==============================] - 0s 5us/step - loss: 0.6866 - acc: 0.7979\n", 1401 | "Epoch 5/10\n", 1402 | "569/569 [==============================] - 0s 5us/step - loss: 0.6843 - acc: 0.8348\n", 1403 | "Epoch 6/10\n", 1404 | "569/569 [==============================] - 0s 6us/step - loss: 0.6820 - acc: 0.8594\n", 1405 | "Epoch 7/10\n", 1406 | "569/569 [==============================] - 0s 6us/step - loss: 0.6797 - acc: 0.8805\n", 1407 | "Epoch 8/10\n", 1408 | "569/569 [==============================] - 0s 6us/step - loss: 0.6773 - acc: 0.8893\n", 1409 | "Epoch 9/10\n", 1410 | "569/569 [==============================] - 0s 9us/step - loss: 0.6749 - acc: 0.8998\n", 1411 | "Epoch 10/10\n", 1412 | "569/569 [==============================] - 0s 5us/step - loss: 0.6724 - acc: 0.9086\n" 1413 | ] 1414 | } 1415 | ], 1416 | "source": [ 1417 | "# Perform the Search!\n", 1418 | "from sklearn.model_selection import GridSearchCV\n", 1419 | "\n", 1420 | "grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1, cv=3, verbose=2)\n", 1421 | "grid_result = grid.fit(x, y) " 1422 | ] 1423 | }, 1424 | { 1425 | "cell_type": "markdown", 1426 | "metadata": {}, 1427 | "source": [ 1428 | "### Results\n", 1429 | "\n", 1430 | "Let's see which configuration give us the best performance." 1431 | ] 1432 | }, 1433 | { 1434 | "cell_type": "code", 1435 | "execution_count": 9, 1436 | "metadata": {}, 1437 | "outputs": [ 1438 | { 1439 | "name": "stdout", 1440 | "output_type": "stream", 1441 | "text": [ 1442 | "Best: 0.938489 using {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 9, 'kernel_initializer': 'uniform', 'optimizer': 'Adam'}\n", 1443 | "0.859402 (0.094322) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 8, 'kernel_initializer': 'uniform', 'optimizer': 'Adam'}\n", 1444 | "0.896309 (0.043754) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 8, 'kernel_initializer': 'uniform', 'optimizer': 'Nadam'}\n", 1445 | "0.866432 (0.058552) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 8, 'kernel_initializer': 'normal', 'optimizer': 'Adam'}\n", 1446 | "0.926186 (0.034017) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 8, 'kernel_initializer': 'normal', 'optimizer': 'Nadam'}\n", 1447 | "0.887522 (0.055293) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 9, 'kernel_initializer': 'uniform', 'optimizer': 'Adam'}\n", 1448 | "0.885765 (0.065705) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 9, 'kernel_initializer': 'uniform', 'optimizer': 'Nadam'}\n", 1449 | "0.901582 (0.019412) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 9, 'kernel_initializer': 'normal', 'optimizer': 'Adam'}\n", 1450 | "0.885765 (0.030188) with: {'activation': 'relu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 9, 'kernel_initializer': 'normal', 'optimizer': 'Nadam'}\n", 1451 | "0.875220 (0.039725) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 8, 'kernel_initializer': 'uniform', 'optimizer': 'Adam'}\n", 1452 | "0.915641 (0.037395) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 8, 'kernel_initializer': 'uniform', 'optimizer': 'Nadam'}\n", 1453 | "0.917399 (0.021626) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 8, 'kernel_initializer': 'normal', 'optimizer': 'Adam'}\n", 1454 | "0.917399 (0.020268) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 8, 'kernel_initializer': 'normal', 'optimizer': 'Nadam'}\n", 1455 | "0.938489 (0.013885) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 9, 'kernel_initializer': 'uniform', 'optimizer': 'Adam'}\n", 1456 | "0.929701 (0.025109) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 9, 'kernel_initializer': 'uniform', 'optimizer': 'Nadam'}\n", 1457 | "0.913884 (0.027976) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 9, 'kernel_initializer': 'normal', 'optimizer': 'Adam'}\n", 1458 | "0.906854 (0.049824) with: {'activation': 'elu', 'batch_size': 1024, 'dropout_rate': 0.0, 'epochs': 10, 'first_neuron': 9, 'kernel_initializer': 'normal', 'optimizer': 'Nadam'}\n" 1459 | ] 1460 | } 1461 | ], 1462 | "source": [ 1463 | "# Show results\n", 1464 | "print(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\n", 1465 | "means = grid_result.cv_results_['mean_test_score']\n", 1466 | "stds = grid_result.cv_results_['std_test_score']\n", 1467 | "params = grid_result.cv_results_['params']\n", 1468 | "for mean, stdev, param in zip(means, stds, params):\n", 1469 | " print(\"%f (%f) with: %r\" % (mean, stdev, param))" 1470 | ] 1471 | }, 1472 | { 1473 | "cell_type": "markdown", 1474 | "metadata": {}, 1475 | "source": [ 1476 | "**That's all folks - don't forget to shutdown your workspace once you're done 🙂**" 1477 | ] 1478 | } 1479 | ], 1480 | "metadata": { 1481 | "kernelspec": { 1482 | "display_name": "Python 2", 1483 | "language": "python", 1484 | "name": "python2" 1485 | }, 1486 | "language_info": { 1487 | "codemirror_mode": { 1488 | "name": "ipython", 1489 | "version": 2 1490 | }, 1491 | "file_extension": ".py", 1492 | "mimetype": "text/x-python", 1493 | "name": "python", 1494 | "nbconvert_exporter": "python", 1495 | "pygments_lexer": "ipython2", 1496 | "version": "2.7.10" 1497 | } 1498 | }, 1499 | "nbformat": 4, 1500 | "nbformat_minor": 2 1501 | } 1502 | --------------------------------------------------------------------------------