├── 1_rsna_2nd_place_data_preparation.ipynb ├── 2_rsna_2nd_place_train_img_classifier.ipynb ├── 3_rsna_2nd_place_train_lstm.ipynb └── README.md /3_rsna_2nd_place_train_lstm.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "This notebook breaks down the code to train the second part of the model, the LSTM, based the code from the [2nd place repo here](https://github.com/darraghdog/rsna/blob/master/scripts/trainlstm.py)\n", 8 | "\n", 9 | "### Main Observations\n", 10 | "I really enjoyed taking a look under the hood of this LSTM training script. The main thing that stood out for me was the need **to be very careful when handling the data with the custom Dataset and Collate Function**. Making sure to keep the correct sequence of images when processing the data and feeding it to the model was the goal here.\n", 11 | "\n", 12 | "Also dealing with batches containing odd sequence lengths was an issue that was nicely dealt with. The LSTM model is surprisingly easy to understand, yet clearly very effective.\n", 13 | "\n", 14 | "### Loading already trained embeddings\n", 15 | "\n", 16 | "The winners also enabled us to download the 2nd Place Stage 1 train and validation embeddings and the Stage 2 test embeddings from their trained Resnext101 model. \n", 17 | "\n", 18 | "As described in their repo, the following code will download the Stage 1 train, validation and test (stage 1) embeddings. The `gdown` package is used as the file size on the google drive is quite large (16gb):\n", 19 | "\n", 20 | "`pip install gdown\n", 21 | "gdown https://drive.google.com/uc?id=13hqPFdCjoMxtAwF863J3Dk33TcBN_wie -O resnext101v12fold1.tar.gz\n", 22 | "gunzip resnext101v12fold1.tar.gz\n", 23 | "tar -xvf resnext101v12fold1.tar`\n", 24 | "\n", 25 | "The Stage 2 test embeddings can then be downloaded: \n", 26 | "\n", 27 | "`wget gdown https://drive.google.com/uc?id=1YxCJ0mWIYXfYLN15DPpQ6OLSt4Y54Hp0`\n", 28 | "\n", 29 | "### Core Modelling Parameters\n", 30 | "- Model : Custom LSTM, from stage 1 Kaggle Toxic comp\n", 31 | "- Epochs : 10\n", 32 | "- Folds : 2, (this notebook doesn't implement folds, the 2nd place solution only got the chance to train for 2 folds but wanted to train for more)\n", 33 | "- Optimizer : Adam\n", 34 | "- Batch Size : 4\n", 35 | "\n", 36 | "### This Notebook\n", 37 | "The initial cells are a little messy are they are a copy and paste from the python script and I don't have a huge amount of time to reformat them to be more suitable for Jupyter.\n", 38 | "\n", 39 | "Also, this notebook is a modification from the original training script in the following ways:\n", 40 | "- The Stage-1 train and validation datasets were used as they were available to download from the 2nd place solution repo\n", 41 | "- The Stage-2 test dataset was used, in order to submit a prediction to Kaggle\n", 42 | "- Folds were not used as I just wanted to get a demo working, and not spend gpu time replicating the initial score\n", 43 | "- You might have to tweak the directory settings to match your own folder names/structure\n", 44 | "- The section around processing and loading the saved embeddings looks different to the original script, but the outcome is the same" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "### Imports..." 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": 1, 57 | "metadata": {}, 58 | "outputs": [], 59 | "source": [ 60 | "import numpy as np\n", 61 | "import csv, gzip, os, sys, gc\n", 62 | "import math\n", 63 | "import torch\n", 64 | "from torch import nn\n", 65 | "import torch.optim as optim\n", 66 | "from torch.nn import functional as F\n", 67 | "\n", 68 | "import logging\n", 69 | "import datetime\n", 70 | "import optparse\n", 71 | "import pandas as pd\n", 72 | "import os\n", 73 | "from sklearn.metrics import log_loss\n", 74 | "import ast\n", 75 | "from torch.utils.data import Dataset\n", 76 | "from sklearn.metrics import log_loss\n", 77 | "from torch.utils.data import DataLoader\n", 78 | "from scipy.ndimage import uniform_filter\n", 79 | "from torch.optim.lr_scheduler import StepLR\n", 80 | "\n", 81 | "from apex.parallel import DistributedDataParallel as DDP\n", 82 | "from apex.fp16_utils import *\n", 83 | "from apex import amp, optimizers\n", 84 | "from apex.multi_tensor_apply import multi_tensor_applier" 85 | ] 86 | }, 87 | { 88 | "cell_type": "markdown", 89 | "metadata": {}, 90 | "source": [ 91 | "### Keep scrolling..." 92 | ] 93 | }, 94 | { 95 | "cell_type": "code", 96 | "execution_count": 2, 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "# Print info about environments\n", 101 | "parser = optparse.OptionParser()\n", 102 | "parser.add_option('-s', '--seed', action=\"store\", dest=\"seed\", help=\"model seed\", default=\"1234\")\n", 103 | "parser.add_option('-o', '--fold', action=\"store\", dest=\"fold\", help=\"Fold for split\", default=\"0\")\n", 104 | "parser.add_option('-p', '--nbags', action=\"store\", dest=\"nbags\", help=\"Number of bags for averaging\", default=\"4\")\n", 105 | "parser.add_option('-e', '--epochs', action=\"store\", dest=\"epochs\", help=\"epochs\", default=\"10\")\n", 106 | "parser.add_option('-b', '--batchsize', action=\"store\", dest=\"batchsize\", help=\"batch size\", default=\"4\")\n", 107 | "parser.add_option('-r', '--rootpath', action=\"store\", dest=\"rootpath\", help=\"root directory\", default=\"\")\n", 108 | "parser.add_option('-i', '--imgpath', action=\"store\", dest=\"imgpath\", help=\"root directory\", default=\"data/mount/512X512X6/\")\n", 109 | "parser.add_option('-w', '--workpath', action=\"store\", dest=\"workpath\", help=\"Working path\", default=\"weights/\")\n", 110 | "parser.add_option('-f', '--weightsname', action=\"store\", dest=\"weightsname\", help=\"Weights file name\", default=\"pytorch_model.bin\")\n", 111 | "parser.add_option('-l', '--lr', action=\"store\", dest=\"lr\", help=\"learning rate\", default=\"0.00005\")\n", 112 | "parser.add_option('-g', '--logmsg', action=\"store\", dest=\"logmsg\", help=\"root directory\", default=\"Recursion-pytorch\")\n", 113 | "parser.add_option('-c', '--size', action=\"store\", dest=\"size\", help=\"model size\", default=\"512\")\n", 114 | "parser.add_option('-a', '--globalepoch', action=\"store\", dest=\"globalepoch\", help=\"root directory\", default=\"3\")\n", 115 | "parser.add_option('-n', '--loadcsv', action=\"store\", dest=\"loadcsv\", help=\"Convert csv embeddings to numpy\", default=\"F\")\n", 116 | "parser.add_option('-j', '--lstm_units', action=\"store\", dest=\"lstm_units\", help=\"Lstm units\", default=\"128\")\n", 117 | "parser.add_option('-d', '--dropout', action=\"store\", dest=\"dropout\", help=\"LSTM input spatial dropout\", default=\"0.3\")\n", 118 | "parser.add_option('-z', '--decay', action=\"store\", dest=\"decay\", help=\"Weight Decay\", default=\"0.0\")\n", 119 | "parser.add_option('-m', '--lrgamma', action=\"store\", dest=\"lrgamma\", help=\"Scheduler Learning Rate Gamma\", default=\"1.0\")\n", 120 | "parser.add_option('-k', '--ttahflip', action=\"store\", dest=\"ttahflip\", help=\"Bag with horizontal flip on and off\", default=\"F\")\n", 121 | "parser.add_option('-q', '--ttatranspose', action=\"store\", dest=\"ttatranspose\", help=\"Bag with horizontal flip on and off\", default=\"F\")\n", 122 | "parser.add_option('-x', '--datapath', action=\"store\", dest=\"datapath\", help=\"Data path\", default=\"data\")\n", 123 | "\n", 124 | "options, args = parser.parse_args()\n", 125 | "package_dir = options.rootpath\n", 126 | "sys.path.append(package_dir)\n", 127 | "sys.path.insert(0, 'scripts')\n", 128 | "from logs import get_logger\n", 129 | "from utils import dumpobj, loadobj, GradualWarmupScheduler" 130 | ] 131 | }, 132 | { 133 | "cell_type": "markdown", 134 | "metadata": {}, 135 | "source": [ 136 | "### Keep scrolling..." 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": 3, 142 | "metadata": { 143 | "scrolled": true 144 | }, 145 | "outputs": [ 146 | { 147 | "name": "stderr", 148 | "output_type": "stream", 149 | "text": [ 150 | "2020-01-29 12:46:44,515 - Recursion-pytorch - INFO - Cuda set up : time 12:46:44.515726\n", 151 | "2020-01-29 12:46:44,524 - Recursion-pytorch - INFO - Device : GeForce RTX 2080 Ti\n", 152 | "2020-01-29 12:46:44,525 - Recursion-pytorch - INFO - Cuda available : True\n", 153 | "2020-01-29 12:46:44,526 - Recursion-pytorch - INFO - Cuda n_gpus : 1\n", 154 | "2020-01-29 12:46:44,526 - Recursion-pytorch - INFO - Load params : time 12:46:44.526832\n", 155 | "2020-01-29 12:46:44,527 - Recursion-pytorch - INFO - seed 1234\n", 156 | "2020-01-29 12:46:44,527 - Recursion-pytorch - INFO - fold 0\n", 157 | "2020-01-29 12:46:44,528 - Recursion-pytorch - INFO - nbags 4\n", 158 | "2020-01-29 12:46:44,528 - Recursion-pytorch - INFO - epochs 10\n", 159 | "2020-01-29 12:46:44,529 - Recursion-pytorch - INFO - batchsize 4\n", 160 | "2020-01-29 12:46:44,529 - Recursion-pytorch - INFO - rootpath \n", 161 | "2020-01-29 12:46:44,530 - Recursion-pytorch - INFO - imgpath data/mount/512X512X6/\n", 162 | "2020-01-29 12:46:44,530 - Recursion-pytorch - INFO - workpath weights/\n", 163 | "2020-01-29 12:46:44,531 - Recursion-pytorch - INFO - weightsname /home/morgan/.local/share/jupyter/runtime/kernel-413dd5aa-64c8-4380-9765-d9843833952f.json\n", 164 | "2020-01-29 12:46:44,531 - Recursion-pytorch - INFO - lr 0.00005\n", 165 | "2020-01-29 12:46:44,532 - Recursion-pytorch - INFO - logmsg Recursion-pytorch\n", 166 | "2020-01-29 12:46:44,532 - Recursion-pytorch - INFO - size 512\n", 167 | "2020-01-29 12:46:44,533 - Recursion-pytorch - INFO - globalepoch 3\n", 168 | "2020-01-29 12:46:44,533 - Recursion-pytorch - INFO - loadcsv F\n", 169 | "2020-01-29 12:46:44,534 - Recursion-pytorch - INFO - lstm_units 128\n", 170 | "2020-01-29 12:46:44,534 - Recursion-pytorch - INFO - dropout 0.3\n", 171 | "2020-01-29 12:46:44,535 - Recursion-pytorch - INFO - decay 0.0\n", 172 | "2020-01-29 12:46:44,535 - Recursion-pytorch - INFO - lrgamma 1.0\n", 173 | "2020-01-29 12:46:44,536 - Recursion-pytorch - INFO - ttahflip F\n", 174 | "2020-01-29 12:46:44,536 - Recursion-pytorch - INFO - ttatranspose F\n", 175 | "2020-01-29 12:46:44,536 - Recursion-pytorch - INFO - datapath data\n" 176 | ] 177 | } 178 | ], 179 | "source": [ 180 | "# Print info about environments\n", 181 | "logger = get_logger(options.logmsg, 'INFO') # noqa\n", 182 | "logger.info('Cuda set up : time {}'.format(datetime.datetime.now().time()))\n", 183 | "\n", 184 | "device=torch.device('cuda')\n", 185 | "logger.info('Device : {}'.format(torch.cuda.get_device_name(0)))\n", 186 | "logger.info('Cuda available : {}'.format(torch.cuda.is_available()))\n", 187 | "n_gpu = torch.cuda.device_count()\n", 188 | "logger.info('Cuda n_gpus : {}'.format(n_gpu ))\n", 189 | "\n", 190 | "\n", 191 | "logger.info('Load params : time {}'.format(datetime.datetime.now().time()))\n", 192 | "for (k,v) in options.__dict__.items():\n", 193 | " logger.info('{}{}'.format(k.ljust(20), v))\n", 194 | "\n", 195 | "WDIR = 'resnext101v01'\n", 196 | "GEPOCH=0\n", 197 | "epochs = 12\n", 198 | "fold = 0\n", 199 | "lr = 0.00001\n", 200 | "batchsize = 4\n", 201 | "workpath = f'scripts/{WDIR}'\n", 202 | "ttahflip = 'T'\n", 203 | "ttatranspose = 'T'\n", 204 | "lrgamma = 0.95\n", 205 | "nbags = 12\n", 206 | "globalepoch = f'{GEPOCH}'\n", 207 | "loadcsv = 'F'\n", 208 | "lstm_units = 2048\n", 209 | " \n", 210 | "SEED = int(options.seed)\n", 211 | "SIZE = 408 # int(options.size)\n", 212 | "EPOCHS = int(options.epochs)\n", 213 | "GLOBALEPOCH= globalepoch #int(options.globalepoch)\n", 214 | "n_epochs = EPOCHS \n", 215 | "lr=float(options.lr)\n", 216 | "#lrgamma=float(options.lrgamma)\n", 217 | "DECAY=float(options.decay)\n", 218 | "batch_size = batchsize #int(options.batchsize)\n", 219 | "ROOT = options.rootpath\n", 220 | "path_data = os.path.join(ROOT, options.datapath)\n", 221 | "path_img = os.path.join(ROOT, options.imgpath)\n", 222 | "WORK_DIR = os.path.join(ROOT, options.workpath)\n", 223 | "path_emb = os.path.join(ROOT, options.workpath)\n", 224 | "WEIGHTS_NAME = options.weightsname\n", 225 | "FOLD = 0 #int(options.fold)\n", 226 | "LOADCSV= options.loadcsv=='T'\n", 227 | "LSTM_UNITS=2048 #int(options.lstm_units)\n", 228 | "#nbags=int(options.nbags)\n", 229 | "DROPOUT=float(options.dropout)\n", 230 | "TTAHFLIP= 'T' #'T' if options.ttahflip=='T' else ''\n", 231 | "TTATRANSPOSE='P' if options.ttatranspose=='T' else ''\n", 232 | "\n", 233 | "n_classes = 6\n", 234 | "label_cols = ['epidural', 'intraparenchymal', 'intraventricular', 'subarachnoid', 'subdural', 'any']\n", 235 | "logmsg = f'Rsna-lstm-{GEPOCH}-{FOLD}-fp16'" 236 | ] 237 | }, 238 | { 239 | "cell_type": "markdown", 240 | "metadata": {}, 241 | "source": [ 242 | "# PyTorch Dataset" 243 | ] 244 | }, 245 | { 246 | "cell_type": "code", 247 | "execution_count": 10, 248 | "metadata": {}, 249 | "outputs": [], 250 | "source": [ 251 | "class IntracranialDataset(Dataset):\n", 252 | " def __init__(self, df, mat, labels=label_cols):\n", 253 | " self.data = df\n", 254 | " self.mat = mat\n", 255 | " #print(self.mat.shape)\n", 256 | " self.labels = labels\n", 257 | " self.patients = df.SliceID.unique()\n", 258 | " self.data = self.data.set_index('SliceID')\n", 259 | "\n", 260 | " def __len__(self):\n", 261 | " return len(self.patients)\n", 262 | "\n", 263 | " def __getitem__(self, idx):\n", 264 | " \n", 265 | " # Get the PatientID from the given index\n", 266 | " patidx = self.patients[idx]\n", 267 | " \n", 268 | " # For a particular PatientID, sort the values according to the seq key\n", 269 | " # Wrap the argument to .loc in a list to ensure a dataframe is returned every time\n", 270 | " patdf = self.data.loc[[patidx]].sort_values('seq') \n", 271 | " \n", 272 | " # Select the embedding index values from the particular Patient Dataframe\n", 273 | " # and index into the dataset .mat with those indices\n", 274 | " patemb = self.mat[patdf['embidx'].values]\n", 275 | "\n", 276 | " # Feed in the embeddings in sequence on key - Patient, Study and Series - \n", 277 | " # also concat on the deltas between current and previous/next embeddings ( and ) \n", 278 | " # to give the model knowledge of changes around the image.\n", 279 | " \n", 280 | " # This will mean that every item from the Dataset will return 3 embeddings:\n", 281 | " # - The patient's embedding for that image (\"patemb\")\n", 282 | " # - The difference between an embedding and its previous embedding(\"patdeltalag\")\n", 283 | " # - The difference between an embedding and its previous embedding(\"patdeltalead\")\n", 284 | " \n", 285 | " patdeltalag = np.zeros(patemb.shape)\n", 286 | " patdeltalead = np.zeros(patemb.shape)\n", 287 | " patdeltalag [1:] = patemb[1:]-patemb[:-1] # e.g. patemb.shape = (36, 2048) , patemb[1:].shape = (35, 2048)\n", 288 | " patdeltalead[:-1] = patemb[:-1]-patemb[1:]\n", 289 | "\n", 290 | " # The 3 embeddings are concatted together going from 3 x (36, 2048) to (36, 6144)\n", 291 | " patemb = np.concatenate((patemb, patdeltalag, patdeltalead), -1)\n", 292 | " \n", 293 | " ids = torch.tensor(patdf['embidx'].values)\n", 294 | "\n", 295 | " if self.labels:\n", 296 | " labels = torch.tensor(patdf[label_cols].values)\n", 297 | " return {'emb': patemb, 'embidx' : ids, 'labels': labels} \n", 298 | " else: \n", 299 | " return {'emb': patemb, 'embidx' : ids}" 300 | ] 301 | }, 302 | { 303 | "cell_type": "markdown", 304 | "metadata": {}, 305 | "source": [ 306 | "# Prep Metadata DataFrames" 307 | ] 308 | }, 309 | { 310 | "cell_type": "markdown", 311 | "metadata": {}, 312 | "source": [ 313 | "### Generate SliceID \n", 314 | "based on ['PatientID', 'SeriesInstanceUID', 'StudyInstanceUID']" 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": 4, 320 | "metadata": { 321 | "scrolled": false 322 | }, 323 | "outputs": [ 324 | { 325 | "name": "stderr", 326 | "output_type": "stream", 327 | "text": [ 328 | "2020-01-29 12:46:44,549 - Recursion-pytorch - INFO - Cuda set up : time 12:46:44.549043\n" 329 | ] 330 | }, 331 | { 332 | "name": "stdout", 333 | "output_type": "stream", 334 | "text": [ 335 | "752803 121232\n" 336 | ] 337 | } 338 | ], 339 | "source": [ 340 | "from random import sample \n", 341 | "\n", 342 | "# Print info about environments\n", 343 | "logger.info('Cuda set up : time {}'.format(datetime.datetime.now().time()))\n", 344 | "\n", 345 | "# Get image sequences\n", 346 | "trnmdf = pd.read_csv(os.path.join(path_data, 'rsna_darraghdog/darraghdog_train_metadata.csv'))\n", 347 | "#trnmdf.Image = 'train_' + trnmdf.Image\n", 348 | "tstmdf = pd.read_csv(os.path.join(path_data, 'rsna_darraghdog/darraghdog_test_metadata.csv'))\n", 349 | "#tstmdf.Image = 'test_' + tstmdf.Image\n", 350 | "\n", 351 | "trnmdf['SliceID'] = trnmdf[['PatientID', 'SeriesInstanceUID', 'StudyInstanceUID']].apply(lambda x: '{}__{}__{}'.format(*x.tolist()), 1)\n", 352 | "tstmdf['SliceID'] = tstmdf[['PatientID', 'SeriesInstanceUID', 'StudyInstanceUID']].apply(lambda x: '{}__{}__{}'.format(*x.tolist()), 1)\n", 353 | "\n", 354 | "print(len(trnmdf), len(tstmdf))" 355 | ] 356 | }, 357 | { 358 | "cell_type": "markdown", 359 | "metadata": {}, 360 | "source": [ 361 | "### Generate Sequence Count\n", 362 | "- Generate Sequence numbers for each series of images based on ['SliceID', 'ImagePos1', 'ImagePos2', 'ImagePos3']\n", 363 | "- Reduce num of colums" 364 | ] 365 | }, 366 | { 367 | "cell_type": "code", 368 | "execution_count": 5, 369 | "metadata": {}, 370 | "outputs": [ 371 | { 372 | "name": "stdout", 373 | "output_type": "stream", 374 | "text": [ 375 | "752803 121232\n" 376 | ] 377 | }, 378 | { 379 | "data": { 380 | "text/html": [ 381 | "
\n", 382 | "\n", 395 | "\n", 396 | " \n", 397 | " \n", 398 | " \n", 399 | " \n", 400 | " \n", 401 | " \n", 402 | " \n", 403 | " \n", 404 | " \n", 405 | " \n", 406 | " \n", 407 | " \n", 408 | " \n", 409 | " \n", 410 | " \n", 411 | " \n", 412 | " \n", 413 | " \n", 414 | " \n", 415 | " \n", 416 | " \n", 417 | " \n", 418 | " \n", 419 | " \n", 420 | " \n", 421 | " \n", 422 | " \n", 423 | " \n", 424 | " \n", 425 | " \n", 426 | " \n", 427 | " \n", 428 | " \n", 429 | " \n", 430 | " \n", 431 | " \n", 432 | " \n", 433 | " \n", 434 | " \n", 435 | " \n", 436 | " \n", 437 | " \n", 438 | " \n", 439 | " \n", 440 | " \n", 441 | " \n", 442 | "
PatientIDSliceIDImageseq
0ID_0002cd41ID_0002cd41__ID_e22a5534e6__ID_66929e09d4train_ID_45785016b1
1ID_0002cd41ID_0002cd41__ID_e22a5534e6__ID_66929e09d4train_ID_37f32aed22
2ID_0002cd41ID_0002cd41__ID_e22a5534e6__ID_66929e09d4train_ID_1b9de29223
3ID_0002cd41ID_0002cd41__ID_e22a5534e6__ID_66929e09d4train_ID_d61a6a7b94
4ID_0002cd41ID_0002cd41__ID_e22a5534e6__ID_66929e09d4train_ID_406c821125
\n", 443 | "
" 444 | ], 445 | "text/plain": [ 446 | " PatientID SliceID Image \\\n", 447 | "0 ID_0002cd41 ID_0002cd41__ID_e22a5534e6__ID_66929e09d4 train_ID_45785016b \n", 448 | "1 ID_0002cd41 ID_0002cd41__ID_e22a5534e6__ID_66929e09d4 train_ID_37f32aed2 \n", 449 | "2 ID_0002cd41 ID_0002cd41__ID_e22a5534e6__ID_66929e09d4 train_ID_1b9de2922 \n", 450 | "3 ID_0002cd41 ID_0002cd41__ID_e22a5534e6__ID_66929e09d4 train_ID_d61a6a7b9 \n", 451 | "4 ID_0002cd41 ID_0002cd41__ID_e22a5534e6__ID_66929e09d4 train_ID_406c82112 \n", 452 | "\n", 453 | " seq \n", 454 | "0 1 \n", 455 | "1 2 \n", 456 | "2 3 \n", 457 | "3 4 \n", 458 | "4 5 " 459 | ] 460 | }, 461 | "execution_count": 5, 462 | "metadata": {}, 463 | "output_type": "execute_result" 464 | } 465 | ], 466 | "source": [ 467 | "# Generate poscols like this: ['ImagePos1', 'ImagePos2', 'ImagePos3']\n", 468 | "poscols = ['ImagePos{}'.format(i) for i in range(1, 4)]\n", 469 | "\n", 470 | "# Parse the ImagePositionPatient string (e.g. \"['-125.000', '-144.700', '109.750']\" ) and add each value\n", 471 | "# to one of 3 \"ImagePos\" columns\n", 472 | "# ast.literal_eval()\n", 473 | "# Safely evaluate an expression node or a string containing a Python literal \n", 474 | "# or container display. The string or node provided may only consist of the \n", 475 | "# following Python literal structures: strings, bytes, numbers, tuples, lists, \n", 476 | "# dicts, sets, booleans, and None.\n", 477 | "# This can be used for safely evaluating strings containing Python values from untrusted sources \n", 478 | "# without the need to parse the values oneself. It is not capable of evaluating arbitrarily \n", 479 | "# complex expressions, for example involving operators or indexing.\n", 480 | "\n", 481 | "trnmdf[poscols] = pd.DataFrame(trnmdf['ImagePositionPatient']\\\n", 482 | " .apply(lambda x: list(map(float, ast.literal_eval(x)))).tolist())\n", 483 | "tstmdf[poscols] = pd.DataFrame(tstmdf['ImagePositionPatient']\\\n", 484 | " .apply(lambda x: list(map(float, ast.literal_eval(x)))).tolist())\n", 485 | "\n", 486 | "# 1. Sort values by ['SliceID', 'ImagePos1', 'ImagePos2', 'ImagePos3']\n", 487 | "# 2. Only select the following columns: [PatientID, SliceID,SOPInstanceUID,ImagePos1,ImagePos2,ImagePos3]\n", 488 | "# 3. Reset index\n", 489 | "trnmdf = trnmdf.sort_values(['SliceID']+poscols)\\\n", 490 | " [['PatientID', 'SliceID', 'SOPInstanceUID']+poscols].reset_index(drop=True)\n", 491 | "tstmdf = tstmdf.sort_values(['SliceID']+poscols)\\\n", 492 | " [['PatientID', 'SliceID', 'SOPInstanceUID']+poscols].reset_index(drop=True)\n", 493 | "\n", 494 | "# Group by the SliceID col and then do a cumulative count for each item that it is grouped by\n", 495 | "# Beacuse these samples have already been sorted by ['SliceID', 'ImagePos1', 'ImagePos2', 'ImagePos3']\n", 496 | "# this value can be inferred to the sequence order of this sequence of images\n", 497 | "trnmdf['seq'] = (trnmdf.groupby(['SliceID']).cumcount() + 1)\n", 498 | "tstmdf['seq'] = (tstmdf.groupby(['SliceID']).cumcount() + 1)\n", 499 | "\n", 500 | "# Further reduce the columns kept\n", 501 | "keepcols = ['PatientID', 'SliceID', 'SOPInstanceUID', 'seq']\n", 502 | "trnmdf = trnmdf[keepcols]\n", 503 | "tstmdf = tstmdf[keepcols]\n", 504 | "\n", 505 | "# rename SOPInstanceUID to Image to prepare to join to the dataframe with labels\n", 506 | "trnmdf.columns = tstmdf.columns = ['PatientID', 'SliceID', 'Image', 'seq']\n", 507 | "\n", 508 | "trnmdf.Image = 'train_' + trnmdf.Image\n", 509 | "tstmdf.Image = 'test_' + tstmdf.Image\n", 510 | "\n", 511 | "print(len(trnmdf), len(tstmdf))\n", 512 | "\n", 513 | "trnmdf.head()" 514 | ] 515 | }, 516 | { 517 | "cell_type": "markdown", 518 | "metadata": {}, 519 | "source": [ 520 | "### Load the datasets" 521 | ] 522 | }, 523 | { 524 | "cell_type": "code", 525 | "execution_count": 6, 526 | "metadata": {}, 527 | "outputs": [], 528 | "source": [ 529 | "def return_stg1_2_embs_data():\n", 530 | " imgpath = 'data/rsna_darraghdog/darraghdog_proc'\n", 531 | "\n", 532 | " stg_1_trn_loader = loadobj('weights/stg1_downloaded_embeddings_resnext101v12fold1/loader_trn_size480_fold1_ep4')\n", 533 | " stg_1_val_loader = loadobj('weights/stg1_downloaded_embeddings_resnext101v12fold1/loader_val_size480_fold1_ep4')\n", 534 | " \n", 535 | " stg_1_trn_loader.dataset.path = imgpath\n", 536 | " stg_1_val_loader.dataset.path = imgpath\n", 537 | "\n", 538 | " stg_1_trn_df = stg_1_trn_loader.dataset.data\n", 539 | " stg_1_val_df = stg_1_val_loader.dataset.data\n", 540 | "\n", 541 | " stg_1_trn_df.Image = ['train_ID_' + x.split('_', 1)[1] for x in stg_1_trn_df.Image]\n", 542 | " stg_1_val_df.Image = ['train_ID_' + x.split('_', 1)[1] for x in stg_1_val_df.Image]\n", 543 | "\n", 544 | " # LOAD Stg2Test\n", 545 | " stg_2_tst_loader = loadobj('weights/stg2tst/loader_tst2_size480_fold1_ep5')\n", 546 | " stg_2_tst_loader.dataset.path = imgpath\n", 547 | " stg_2_tst_df = stg_2_tst_loader.dataset.data\n", 548 | " \n", 549 | " return stg_1_trn_df, stg_1_val_df, stg_2_tst_df" 550 | ] 551 | }, 552 | { 553 | "cell_type": "markdown", 554 | "metadata": {}, 555 | "source": [ 556 | "### More Data Processing\n", 557 | "- Load Stage 1 + 2 Datasets\n", 558 | "- Set the embedding index" 559 | ] 560 | }, 561 | { 562 | "cell_type": "code", 563 | "execution_count": 11, 564 | "metadata": { 565 | "scrolled": true 566 | }, 567 | "outputs": [ 568 | { 569 | "data": { 570 | "text/html": [ 571 | "
\n", 572 | "\n", 585 | "\n", 586 | " \n", 587 | " \n", 588 | " \n", 589 | " \n", 590 | " \n", 591 | " \n", 592 | " \n", 593 | " \n", 594 | " \n", 595 | " \n", 596 | " \n", 597 | " \n", 598 | " \n", 599 | " \n", 600 | " \n", 601 | " \n", 602 | " \n", 603 | " \n", 604 | " \n", 605 | " \n", 606 | " \n", 607 | " \n", 608 | " \n", 609 | " \n", 610 | " \n", 611 | " \n", 612 | " \n", 613 | " \n", 614 | " \n", 615 | " \n", 616 | " \n", 617 | " \n", 618 | " \n", 619 | " \n", 620 | " \n", 621 | " \n", 622 | " \n", 623 | " \n", 624 | " \n", 625 | " \n", 626 | " \n", 627 | " \n", 628 | " \n", 629 | " \n", 630 | " \n", 631 | " \n", 632 | " \n", 633 | " \n", 634 | " \n", 635 | " \n", 636 | " \n", 637 | " \n", 638 | " \n", 639 | " \n", 640 | " \n", 641 | " \n", 642 | " \n", 643 | " \n", 644 | " \n", 645 | " \n", 646 | " \n", 647 | " \n", 648 | " \n", 649 | " \n", 650 | " \n", 651 | " \n", 652 | " \n", 653 | " \n", 654 | " \n", 655 | " \n", 656 | " \n", 657 | " \n", 658 | " \n", 659 | " \n", 660 | " \n", 661 | " \n", 662 | " \n", 663 | " \n", 664 | " \n", 665 | " \n", 666 | " \n", 667 | " \n", 668 | " \n", 669 | " \n", 670 | " \n", 671 | " \n", 672 | " \n", 673 | " \n", 674 | " \n", 675 | " \n", 676 | " \n", 677 | " \n", 678 | " \n", 679 | " \n", 680 | "
ImageembidxanyepiduralintraparenchymalintraventricularsubarachnoidsubduralPatientIDfoldSliceIDseq
0train_ID_45785016b146382000000ID_0002cd414ID_0002cd41__ID_e22a5534e6__ID_66929e09d41
1train_ID_37f32aed2117917000000ID_0002cd414ID_0002cd41__ID_e22a5534e6__ID_66929e09d42
2train_ID_1b9de292258394000000ID_0002cd414ID_0002cd41__ID_e22a5534e6__ID_66929e09d43
3train_ID_d61a6a7b9451680000000ID_0002cd414ID_0002cd41__ID_e22a5534e6__ID_66929e09d44
4train_ID_406c82112135730000000ID_0002cd414ID_0002cd41__ID_e22a5534e6__ID_66929e09d45
\n", 681 | "
" 682 | ], 683 | "text/plain": [ 684 | " Image embidx any epidural intraparenchymal \\\n", 685 | "0 train_ID_45785016b 146382 0 0 0 \n", 686 | "1 train_ID_37f32aed2 117917 0 0 0 \n", 687 | "2 train_ID_1b9de2922 58394 0 0 0 \n", 688 | "3 train_ID_d61a6a7b9 451680 0 0 0 \n", 689 | "4 train_ID_406c82112 135730 0 0 0 \n", 690 | "\n", 691 | " intraventricular subarachnoid subdural PatientID fold \\\n", 692 | "0 0 0 0 ID_0002cd41 4 \n", 693 | "1 0 0 0 ID_0002cd41 4 \n", 694 | "2 0 0 0 ID_0002cd41 4 \n", 695 | "3 0 0 0 ID_0002cd41 4 \n", 696 | "4 0 0 0 ID_0002cd41 4 \n", 697 | "\n", 698 | " SliceID seq \n", 699 | "0 ID_0002cd41__ID_e22a5534e6__ID_66929e09d4 1 \n", 700 | "1 ID_0002cd41__ID_e22a5534e6__ID_66929e09d4 2 \n", 701 | "2 ID_0002cd41__ID_e22a5534e6__ID_66929e09d4 3 \n", 702 | "3 ID_0002cd41__ID_e22a5534e6__ID_66929e09d4 4 \n", 703 | "4 ID_0002cd41__ID_e22a5534e6__ID_66929e09d4 5 " 704 | ] 705 | }, 706 | "execution_count": 11, 707 | "metadata": {}, 708 | "output_type": "execute_result" 709 | } 710 | ], 711 | "source": [ 712 | "use_stg1_embs = True\n", 713 | "\n", 714 | "if use_stg1_embs:\n", 715 | " \n", 716 | " # Retrieve the Stage 1 data and stage 2 test data from the saved dataloaders\n", 717 | " stg_1_trn_df, stg_1_val_df, stg_2_tst_df = return_stg1_2_embs_data()\n", 718 | " \n", 719 | " # THE ORDER OF THE ORIGINAL TRAIN DATASET SHOULD BE PRESERVED\n", 720 | " # Save the current order of the embeddings (as the loaded ones are in this order)\n", 721 | " stg_1_trn_df['embidx'] = range(stg_1_trn_df.shape[0])\n", 722 | " stg_1_val_df['embidx'] = range(stg_1_val_df.shape[0])\n", 723 | " \n", 724 | " # MERGE stg1 train and val sets\n", 725 | " train = pd.concat([stg_1_trn_df, stg_1_val_df], axis=0, sort=False)\n", 726 | " \n", 727 | " # IMPORT STAGE 2 TRAIN (which has all labels)\n", 728 | " stg2_train = pd.read_csv(os.path.join(path_data, 'rsna_darraghdog/darraghdog_train.csv.gz'))\n", 729 | " stg2_test = pd.read_csv(os.path.join(path_data, 'rsna_darraghdog/darraghdog_test.csv.gz'))\n", 730 | " \n", 731 | " stg2_train.Image = 'train_' + stg2_train.Image\n", 732 | " stg2_test.Image = 'test_' + stg2_test.Image\n", 733 | " \n", 734 | " # DROP the fold column as we want to use the \n", 735 | " stg2_train.drop(['fold'], axis=1)\n", 736 | " \n", 737 | " # DROP LABEL COLS (as some are missing due to stg_1_tst_df merge) COLUMNS FROM train\n", 738 | " drop_cols = label_cols.copy()\n", 739 | " drop_cols.append('PatientID')\n", 740 | " drop_cols.append('fold')\n", 741 | " train.drop(drop_cols, axis=1, inplace=True)\n", 742 | " \n", 743 | " # MERGE train with stg2_train\n", 744 | " train = train.merge(stg2_train, on = 'Image', sort=False)\n", 745 | " \n", 746 | " # RENAME STAGE 2 TEST\n", 747 | " test = stg2_test.copy()\n", 748 | " \n", 749 | " # MERGE with METADATA DF to get full picture\n", 750 | " trndf = train.merge(trnmdf.drop('PatientID', 1), on = 'Image')\n", 751 | " tstdf = test.merge(tstmdf, on = 'Image')\n", 752 | " \n", 753 | " # THE ORDER OF THE ORIGINAL TRAIN DATASET SHOULD BE PRESERVED\n", 754 | " # Save the current order of the embeddings (as the loaded ones are in this order)\n", 755 | " tstdf['embidx'] = range(tstdf.shape[0])\n", 756 | "\n", 757 | " # THE ORDER OF THE ORIGINAL TRAIN DATASET SHOULD BE PRESERVED\n", 758 | " \n", 759 | " trndf = trndf.sort_values(['PatientID','SliceID','seq']).reset_index(drop=True)\n", 760 | " tstdf = tstdf.sort_values(['PatientID','SliceID','seq']).reset_index(drop=True)\n", 761 | "\n", 762 | "trndf.head()" 763 | ] 764 | }, 765 | { 766 | "cell_type": "markdown", 767 | "metadata": {}, 768 | "source": [ 769 | "### Train/Val Split\n", 770 | "Split into Train and Val sets again, as per Stg1 split" 771 | ] 772 | }, 773 | { 774 | "cell_type": "code", 775 | "execution_count": 12, 776 | "metadata": {}, 777 | "outputs": [ 778 | { 779 | "data": { 780 | "text/plain": [ 781 | "(539827, 134430)" 782 | ] 783 | }, 784 | "execution_count": 12, 785 | "metadata": {}, 786 | "output_type": "execute_result" 787 | } 788 | ], 789 | "source": [ 790 | "valdf = trndf.loc[trndf.Image.isin(stg_1_val_df.Image.values)].copy()\n", 791 | "trndf = trndf.loc[trndf.Image.isin(stg_1_trn_df.Image.values)].copy()\n", 792 | "len(trndf), len(valdf)" 793 | ] 794 | }, 795 | { 796 | "cell_type": "markdown", 797 | "metadata": {}, 798 | "source": [ 799 | "### Load the Embeddings\n", 800 | "- Load the Stage 1 Train + Val Embeddings and the Stage 2 Test Embeddings\n", 801 | "- Assuming Stage2 embeddings came from the same model as the Stage1 embeddings" 802 | ] 803 | }, 804 | { 805 | "cell_type": "code", 806 | "execution_count": 14, 807 | "metadata": { 808 | "scrolled": true 809 | }, 810 | "outputs": [ 811 | { 812 | "name": "stderr", 813 | "output_type": "stream", 814 | "text": [ 815 | "2020-01-29 13:04:31,858 - Recursion-pytorch - INFO - Load embeddings...\n", 816 | "2020-01-29 13:05:12,360 - Recursion-pytorch - INFO - Trn shape 539827 2048\n", 817 | "2020-01-29 13:05:12,368 - Recursion-pytorch - INFO - Val shape 134430 2048\n", 818 | "2020-01-29 13:05:12,369 - Recursion-pytorch - INFO - Tst shape 121232 2048\n" 819 | ] 820 | } 821 | ], 822 | "source": [ 823 | "def load_saved_emb(e_pth, emb_path):\n", 824 | " return np.load(os.path.join(e_pth, emb_path))['arr_0']\n", 825 | "\n", 826 | "stg_1_emb_path = 'weights/stg1_downloaded_embeddings_resnext101v12fold1'\n", 827 | "stg_2_path_emb = 'weights/stg2tst'\n", 828 | "\n", 829 | "# Paths for the Stage 1 embeddings\n", 830 | "trn_emb_path = 'emb_trn_size480_fold1_ep4.npz'\n", 831 | "val_emb_path = 'emb_val_size480_fold1_ep4.npz'\n", 832 | "\n", 833 | "# Paths for the Stage 2 embeddings\n", 834 | "tst_emb_path = 'emb_tst2_size480_fold1_ep5.npz'\n", 835 | "\n", 836 | "logger.info('Load embeddings...')\n", 837 | "trnembls = [load_saved_emb(stg_1_emb_path, trn_emb_path)]\n", 838 | "valembls = [load_saved_emb(stg_1_emb_path, val_emb_path)]\n", 839 | "tstembls = [load_saved_emb(stg_2_path_emb, tst_emb_path)]\n", 840 | "\n", 841 | "trnemb = sum(trnembls)/len(trnembls)\n", 842 | "valemb = sum(valembls)/len(valembls)\n", 843 | "tstemb = sum(tstembls)/len(tstembls)\n", 844 | "\n", 845 | "logger.info('Trn shape {} {}'.format(*trnemb.shape))\n", 846 | "logger.info('Val shape {} {}'.format(*valemb.shape))\n", 847 | "logger.info('Tst shape {} {}'.format(*tstemb.shape))" 848 | ] 849 | }, 850 | { 851 | "cell_type": "markdown", 852 | "metadata": {}, 853 | "source": [ 854 | "### Collate Function\n", 855 | "- Define a collate function to pass to collate_fn in the DataLoader in order to operate on the Dataset as needed\n", 856 | "- https://www.kaggle.com/bminixhofer/speed-up-your-rnn-with-sequence-bucketing" 857 | ] 858 | }, 859 | { 860 | "cell_type": "code", 861 | "execution_count": null, 862 | "metadata": {}, 863 | "outputs": [], 864 | "source": [ 865 | "def collatefn(batch):\n", 866 | " maxlen = max([l['emb'].shape[0] for l in batch])\n", 867 | " \n", 868 | " embdim = batch[0]['emb'].shape[1]\n", 869 | " withlabel = 'labels' in batch[0]\n", 870 | " if withlabel:\n", 871 | " labdim= batch[0]['labels'].shape[1]\n", 872 | " \n", 873 | " for b in batch:\n", 874 | " # batch size could be (3, 40, 6144)\n", 875 | " # For sequences of different length:\n", 876 | " # - padded them to same length\n", 877 | " # - made a dummy embedding of zeros\n", 878 | " # - then threw the results of this away before calculating loss and saving the predictions.\n", 879 | " #\n", 880 | " # \"masklen\" : The number of dummy image embeddings to add to make sure each dequence in the batch \n", 881 | " # has the same dequence length. Calculated as the difference between:\n", 882 | " # - the number of the LONGEST sequence of embeddings\n", 883 | " # in the batch (longest seq of images from the same patient)\n", 884 | " # MINUS\n", 885 | " # - The number of image embeddings of this particular sequence\n", 886 | " #\n", 887 | " # A batch contains sequences of images from multiple patients. Some patients\n", 888 | " # might have 28 images in their sequence, while some others might have 40. \n", 889 | " # If there a patients with different numbers of images (different lengths of sequences)\n", 890 | " # in a batch then these sequences won't be able to be stacked as items in a batch must \n", 891 | " # the same dimensions. \n", 892 | " # This scenario is addressed by creating dummy images full of zeros and adding them to \n", 893 | " # a patient's sequence of images. The number of dummy images needed is dictated by the \n", 894 | " # the \"masklen\"\n", 895 | " masklen = maxlen-len(b['emb'])\n", 896 | " \n", 897 | " # Stack a number (\"masklen\") of dummy embeddings onto the current sequence of embeddings\n", 898 | " # to make sure all sequences in the batch are the same length\n", 899 | " b['emb'] = np.vstack((np.zeros((masklen, embdim)), b['emb']))\n", 900 | " \n", 901 | " # Adjust the embedding index \"embidx\" by adding a number (\"masklen\") of -1's to it\n", 902 | " # e.g. tensor([-1, -1, -1, -1])\n", 903 | " b['embidx'] = torch.cat((torch.ones((masklen),dtype=torch.long)*-1, b['embidx']))\n", 904 | " \n", 905 | " # \"mask\" is a flag to indicate whether the embedding is a dummy or not \"array([1., 1., 1., 1.])\"\n", 906 | " # Create it to be the length of the longest sequence and fill it with 1's\n", 907 | " b['mask'] = np.ones((maxlen))\n", 908 | " \n", 909 | " # Change the first numbner (\"masklen\") of flags to have a 0 flag, meaning it is a dummy embedding\n", 910 | " # This works because the dummy embeddings were inserted ahead of the real embeddings in \"b['emb']\"\n", 911 | " b['mask'][:masklen] = 0.\n", 912 | " if withlabel:\n", 913 | " # Add dummy labels for the dummy embeddings\n", 914 | " b['labels'] = np.vstack((np.zeros((maxlen-len(b['labels']), labdim)), b['labels']))\n", 915 | " \n", 916 | " # Expand the array dimensions to be the correct dims for the LSTM model.\n", 917 | " # nn.LSTM takes inputs of shape: (batch, seq_len, input_size) when batch_first=True in the lstm definition\n", 918 | " # numpy.expand_dims(a, axis)[source]\n", 919 | " # Expand the shape of an array. Insert a new axis that will appear at the axis position\n", 920 | " # in the expanded array shape.\n", 921 | " # Returns: View of a with the number of dimensions increased by one.\n", 922 | " # e.g. Expands b['emb'] from (36, 6144) to (1, 36, 6144)\n", 923 | " outbatch = {'emb' : torch.tensor(np.vstack([np.expand_dims(b['emb'], 0) \\\n", 924 | " for b in batch])).float()} \n", 925 | " \n", 926 | " outbatch['mask'] = torch.tensor(np.vstack([np.expand_dims(b['mask'], 0) \\\n", 927 | " for b in batch])).float()\n", 928 | " outbatch['embidx'] = torch.tensor(np.vstack([np.expand_dims(b['embidx'], 0) \\\n", 929 | " for b in batch])).float()\n", 930 | " if withlabel:\n", 931 | " outbatch['labels'] = torch.tensor(np.vstack([np.expand_dims(b['labels'], 0) for b in batch])).float()\n", 932 | " return outbatch" 933 | ] 934 | }, 935 | { 936 | "cell_type": "markdown", 937 | "metadata": {}, 938 | "source": [ 939 | "### Create the PyTorch Datasets and Dataloaders based on loaded info" 940 | ] 941 | }, 942 | { 943 | "cell_type": "code", 944 | "execution_count": null, 945 | "metadata": {}, 946 | "outputs": [], 947 | "source": [ 948 | "logger.info('Create loaders...')\n", 949 | "trndataset = IntracranialDataset(trndf, trnemb, labels=True)\n", 950 | "valdataset = IntracranialDataset(valdf, valemb, labels=True)\n", 951 | "tstdataset = IntracranialDataset(tstdf, tstemb, labels=False)\n", 952 | "\n", 953 | "batch_size=4\n", 954 | "trnloader = DataLoader(trndataset, batch_size=batch_size, shuffle=True, num_workers=8, collate_fn=collatefn)\n", 955 | "valloader = DataLoader(valdataset, batch_size=batch_size*4, shuffle=False, num_workers=8, collate_fn=collatefn)\n", 956 | "tstloader = DataLoader(tstdataset, batch_size=batch_size*4, shuffle=False, num_workers=8, collate_fn=collatefn)" 957 | ] 958 | }, 959 | { 960 | "cell_type": "markdown", 961 | "metadata": {}, 962 | "source": [ 963 | "SpatialDropout not used in the final solution, leaving it in here just for reference" 964 | ] 965 | }, 966 | { 967 | "cell_type": "code", 968 | "execution_count": null, 969 | "metadata": {}, 970 | "outputs": [], 971 | "source": [ 972 | "# class SpatialDropout(nn.Dropout2d):\n", 973 | "# def forward(self, x):\n", 974 | "# x = x.unsqueeze(2) # (N, T, 1, K)\n", 975 | "# x = x.permute(0, 3, 2, 1) # (N, K, 1, T)\n", 976 | "# x = super(SpatialDropout, self).forward(x) # (N, K, 1, T), some features are masked\n", 977 | "# x = x.permute(0, 3, 2, 1) # (N, T, 1, K)\n", 978 | "# x = x.squeeze(2) # (N, T, K)\n", 979 | "# return x" 980 | ] 981 | }, 982 | { 983 | "cell_type": "markdown", 984 | "metadata": {}, 985 | "source": [ 986 | "# LSTM Model\n", 987 | "\n", 988 | "Define the LSTM model" 989 | ] 990 | }, 991 | { 992 | "cell_type": "code", 993 | "execution_count": 22, 994 | "metadata": {}, 995 | "outputs": [], 996 | "source": [ 997 | "class NeuralNet(nn.Module):\n", 998 | " def __init__(self, embed_size=trnemb.shape[-1]*3, LSTM_UNITS=64, DO = 0.3): # embed_size=trnemb.shape[-1]*3\n", 999 | " super(NeuralNet, self).__init__()\n", 1000 | " \n", 1001 | " # embed_size=trnemb.shape[-1]*3 because of the earlier creatiion of lagging and leading embeddings\n", 1002 | " # trnemb.shape) (26805, 2048)\n", 1003 | " # trnemb.shape[-1]) is 2048\n", 1004 | " #trnemb.shape[-1]*3) is 6144\n", 1005 | " \n", 1006 | " # This SpatialDropout doesn't seem to be used, commenting out\n", 1007 | " #self.embedding_dropout = SpatialDropout(0.0) #DO)\n", 1008 | " \n", 1009 | " # LSTM\n", 1010 | " # input of shape (batch, seq_len, input_size): tensor containing the \n", 1011 | " # features of the input sequence.\n", 1012 | " # h_0 of shape (batch, num_layers * num_directions, hidden_size): tensor \n", 1013 | " # containing the initial hidden state for each element in the batch. If \n", 1014 | " # the LSTM is bidirectional, num_directions should be 2, else it should be 1.\n", 1015 | " # c_0 of shape (batch, num_layers * num_directions, hidden_size): tensor\n", 1016 | " \n", 1017 | " # Note, bidirectional=True, meaning the lstm runs through the sequence \n", 1018 | " # forwards and backwards, returning the SUM of each output\n", 1019 | " self.lstm1 = nn.LSTM(input_size=embed_size, # 6144\n", 1020 | " hidden_size=LSTM_UNITS, # 64\n", 1021 | " bidirectional=True,\n", 1022 | " batch_first=True)\n", 1023 | " # lstm1 output size will be: torch.Size([3, 36, 4096]), when LSTM_UNITS=2048\n", 1024 | " # output of shape `(batch, seq_len, num_directions * hidden_size), (3, 36, 2 * 64) when LSTM_UNITS=64\n", 1025 | " \n", 1026 | " self.lstm2 = nn.LSTM(LSTM_UNITS * 2, # 128\n", 1027 | " LSTM_UNITS, # 64\n", 1028 | " bidirectional=True, \n", 1029 | " batch_first=True)\n", 1030 | "\n", 1031 | " self.linear1 = nn.Linear(in_features=LSTM_UNITS*2, # 128\n", 1032 | " out_features=LSTM_UNITS*2 # 128\n", 1033 | " )\n", 1034 | " self.linear2 = nn.Linear(LSTM_UNITS*2, # 128\n", 1035 | " LSTM_UNITS*2 # 128\n", 1036 | " )\n", 1037 | "\n", 1038 | " self.linear = nn.Linear(LSTM_UNITS*2, # 128\n", 1039 | " n_classes # 6\n", 1040 | " )\n", 1041 | "\n", 1042 | " def forward(self, x, lengths=None):\n", 1043 | " # x.size() is torch.Size([2, 36, 6144])\n", 1044 | " h_embedding = x\n", 1045 | " \n", 1046 | " n = 2048\n", 1047 | " # while h_embedding[:,:,:n].size() is torch.Size([2, 36, 2048])\n", 1048 | " # Selecting and duplicating the first (and origional) of the 3 embeddings (original, lag, lead) \n", 1049 | " # that were earlier concatted \n", 1050 | " \n", 1051 | " # h_embadd.size is torch.Size([2, 36, 4096]), not needed now though\n", 1052 | " h_embadd = torch.cat((h_embedding[:,:,:n], h_embedding[:,:,:n]), -1)\n", 1053 | " \n", 1054 | " h_lstm1, _ = self.lstm1(h_embedding)\n", 1055 | " h_lstm2, _ = self.lstm2(h_lstm1)\n", 1056 | " \n", 1057 | " h_conc_linear1 = F.relu(self.linear1(h_lstm1))\n", 1058 | " h_conc_linear2 = F.relu(self.linear2(h_lstm2))\n", 1059 | " \n", 1060 | " # SUM the oringal embedding (\"h_embadd\") back to the outputs from the other layers\n", 1061 | " # \"h_embadd\" had to be used due to doubling of lstm_unit in earlier dimensions (where lstm_unit = 2048)\n", 1062 | " # However this sum to calculate \"hidden\" then kills my 13GB memory:\n", 1063 | " hidden = h_lstm1 + h_lstm2 + h_conc_linear1 + h_conc_linear2 + h_embadd\n", 1064 | "\n", 1065 | " output = self.linear(hidden)\n", 1066 | " \n", 1067 | " return output" 1068 | ] 1069 | }, 1070 | { 1071 | "cell_type": "code", 1072 | "execution_count": null, 1073 | "metadata": {}, 1074 | "outputs": [], 1075 | "source": [ 1076 | "# Try free up some memory\n", 1077 | "del(trnemb, valemb, tstemb)\n", 1078 | "gc.collect()" 1079 | ] 1080 | }, 1081 | { 1082 | "cell_type": "markdown", 1083 | "metadata": {}, 1084 | "source": [ 1085 | "### Los function\n", 1086 | "A customer BCEWithLogitsLoss loss function, same as the image classifier loss function " 1087 | ] 1088 | }, 1089 | { 1090 | "cell_type": "code", 1091 | "execution_count": 25, 1092 | "metadata": {}, 1093 | "outputs": [], 1094 | "source": [ 1095 | "def criterion(data, targets, criterion = torch.nn.BCEWithLogitsLoss()):\n", 1096 | " ''' Define custom loss function for weighted BCE on 'target' column '''\n", 1097 | " loss_all = criterion(data, targets)\n", 1098 | " loss_any = criterion(data[:,-1:], targets[:,-1:])\n", 1099 | " return (loss_all*6 + loss_any*1)/7" 1100 | ] 1101 | }, 1102 | { 1103 | "cell_type": "markdown", 1104 | "metadata": {}, 1105 | "source": [ 1106 | "### Reclaim GPU RAM\n", 1107 | "- context manager to reclaim GPU RAM if CUDA out of memory happened, or execution was interrupted\n", 1108 | "- taken from fastai: https://docs.fast.ai/troubleshoot.html" 1109 | ] 1110 | }, 1111 | { 1112 | "cell_type": "code", 1113 | "execution_count": 27, 1114 | "metadata": {}, 1115 | "outputs": [], 1116 | "source": [ 1117 | "class gpu_mem_restore_ctx():\n", 1118 | " \" context manager to reclaim GPU RAM if CUDA out of memory happened, or execution was interrupted\"\n", 1119 | " def __enter__(self): return self\n", 1120 | " def __exit__(self, exc_type, exc_val, exc_tb):\n", 1121 | " if not exc_val: return True\n", 1122 | " traceback.clear_frames(exc_tb)\n", 1123 | " raise exc_type(exc_val).with_traceback(exc_tb) from None" 1124 | ] 1125 | }, 1126 | { 1127 | "cell_type": "code", 1128 | "execution_count": 8, 1129 | "metadata": {}, 1130 | "outputs": [], 1131 | "source": [ 1132 | "# ls = []\n", 1133 | "# for i in range(10):\n", 1134 | "# print(1e-4*((0.95*1.0)**(i)))\n", 1135 | "# ls.append(1e-4*((0.95*1.0)**(i)))\n", 1136 | " \n", 1137 | "# from matplotlib import pyplot as plt\n", 1138 | "# plt.plot(ls)\n", 1139 | "# plt.xlabel('Epoch number')\n", 1140 | "# plt.ylabel('Learning rate')" 1141 | ] 1142 | }, 1143 | { 1144 | "cell_type": "markdown", 1145 | "metadata": {}, 1146 | "source": [ 1147 | "How many epochs are we doing?" 1148 | ] 1149 | }, 1150 | { 1151 | "cell_type": "code", 1152 | "execution_count": 9, 1153 | "metadata": {}, 1154 | "outputs": [ 1155 | { 1156 | "data": { 1157 | "text/plain": [ 1158 | "10" 1159 | ] 1160 | }, 1161 | "execution_count": 9, 1162 | "metadata": {}, 1163 | "output_type": "execute_result" 1164 | } 1165 | ], 1166 | "source": [ 1167 | "EPOCHS" 1168 | ] 1169 | }, 1170 | { 1171 | "cell_type": "markdown", 1172 | "metadata": {}, 1173 | "source": [ 1174 | "### Prediction\n", 1175 | "- Used for retrieving predictions to calculate the validation loss and get the test set predictions\n", 1176 | "- Ignores the mask embeddings" 1177 | ] 1178 | }, 1179 | { 1180 | "cell_type": "code", 1181 | "execution_count": 24, 1182 | "metadata": {}, 1183 | "outputs": [], 1184 | "source": [ 1185 | "def predict(loader):\n", 1186 | " valls = []\n", 1187 | " imgls = []\n", 1188 | " imgdf = loader.dataset.data.reset_index().set_index('embidx')[['Image']].copy()\n", 1189 | " for step, batch in enumerate(loader):\n", 1190 | " inputs = batch[\"emb\"]\n", 1191 | " mask = batch['mask'].to(device, dtype=torch.int)\n", 1192 | " inputs = inputs.to(device, dtype=torch.float)\n", 1193 | " logits = model(inputs)\n", 1194 | " # get the mask for masked labels\n", 1195 | " maskidx = mask.view(-1)==1\n", 1196 | " # reshape for\n", 1197 | " logits = logits.view(-1, n_classes)[maskidx]\n", 1198 | " valls.append(torch.sigmoid(logits).detach().cpu().numpy())\n", 1199 | " # Get the list of images\n", 1200 | " embidx = batch[\"embidx\"].detach().cpu().numpy().astype(np.int32)\n", 1201 | " embidx = embidx.flatten()[embidx.flatten()>-1]\n", 1202 | " images = imgdf.loc[embidx].Image.tolist() \n", 1203 | " imgls += images\n", 1204 | " return np.concatenate(valls, 0), imgls" 1205 | ] 1206 | }, 1207 | { 1208 | "cell_type": "markdown", 1209 | "metadata": {}, 1210 | "source": [ 1211 | "### Make Submission to Kaggle\n", 1212 | "Gets the output predictions formatted correctly for submission" 1213 | ] 1214 | }, 1215 | { 1216 | "cell_type": "code", 1217 | "execution_count": 26, 1218 | "metadata": {}, 1219 | "outputs": [], 1220 | "source": [ 1221 | "def makeSub(ypred, imgs):\n", 1222 | " imgls = np.array(imgs).repeat(len(label_cols)) \n", 1223 | " icdls = pd.Series(label_cols*ypred.shape[0]) \n", 1224 | " yidx = ['{}_{}'.format(i,j) for i,j in zip(imgls, icdls)]\n", 1225 | " subdf = pd.DataFrame({'ID' : yidx, 'Label': ypred.flatten()})\n", 1226 | " return subdf" 1227 | ] 1228 | }, 1229 | { 1230 | "cell_type": "markdown", 1231 | "metadata": {}, 1232 | "source": [ 1233 | "### Initialise Model" 1234 | ] 1235 | }, 1236 | { 1237 | "cell_type": "code", 1238 | "execution_count": 31, 1239 | "metadata": {}, 1240 | "outputs": [ 1241 | { 1242 | "name": "stderr", 1243 | "output_type": "stream", 1244 | "text": [ 1245 | "2020-01-27 18:10:39,282 - Recursion-pytorch - INFO - Create model\n" 1246 | ] 1247 | }, 1248 | { 1249 | "name": "stdout", 1250 | "output_type": "stream", 1251 | "text": [ 1252 | "Selected optimization level O1: Insert automatic casts around Pytorch functions and Tensor methods.\n", 1253 | "\n", 1254 | "Defaults for this optimization level are:\n", 1255 | "enabled : True\n", 1256 | "opt_level : O1\n", 1257 | "cast_model_type : None\n", 1258 | "patch_torch_functions : True\n", 1259 | "keep_batchnorm_fp32 : None\n", 1260 | "master_weights : None\n", 1261 | "loss_scale : dynamic\n", 1262 | "Processing user overrides (additional kwargs that are not None)...\n", 1263 | "After processing overrides, optimization options are:\n", 1264 | "enabled : True\n", 1265 | "opt_level : O1\n", 1266 | "cast_model_type : None\n", 1267 | "patch_torch_functions : True\n", 1268 | "keep_batchnorm_fp32 : None\n", 1269 | "master_weights : None\n", 1270 | "loss_scale : dynamic\n" 1271 | ] 1272 | } 1273 | ], 1274 | "source": [ 1275 | "logger.info('Create model')\n", 1276 | "lrgamma = 0.95\n", 1277 | "\n", 1278 | "#LSTM_UNITS = 1024\n", 1279 | "LSTM_UNITS = 2048\n", 1280 | "model = NeuralNet(LSTM_UNITS=LSTM_UNITS, DO = DROPOUT)\n", 1281 | "model = model.to(device)\n", 1282 | "\n", 1283 | "param_optimizer = list(model.named_parameters())\n", 1284 | "no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']\n", 1285 | "plist = [\n", 1286 | " {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': DECAY},\n", 1287 | " {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}\n", 1288 | " ]\n", 1289 | "\n", 1290 | "optimizer = optim.Adam(plist, lr=lr)\n", 1291 | "scheduler = StepLR(optimizer, step_size=1, gamma=lrgamma, last_epoch=-1)\n", 1292 | "model, optimizer = amp.initialize(model, optimizer, opt_level=\"O1\")\n", 1293 | "\n", 1294 | "ypredls = []" 1295 | ] 1296 | }, 1297 | { 1298 | "cell_type": "markdown", 1299 | "metadata": {}, 1300 | "source": [ 1301 | "### Training Loop" 1302 | ] 1303 | }, 1304 | { 1305 | "cell_type": "code", 1306 | "execution_count": 32, 1307 | "metadata": { 1308 | "scrolled": true 1309 | }, 1310 | "outputs": [ 1311 | { 1312 | "name": "stderr", 1313 | "output_type": "stream", 1314 | "text": [ 1315 | "2020-01-27 18:10:42,198 - Recursion-pytorch - INFO - EPOCH 0\n", 1316 | "2020-01-27 18:10:43,044 - Recursion-pytorch - INFO - Trn step 0 of 1955 trn lossavg 0.67631\n", 1317 | "2020-01-27 18:10:49,770 - Recursion-pytorch - INFO - Trn step 50 of 1955 trn lossavg 0.10079\n", 1318 | "2020-01-27 18:10:56,523 - Recursion-pytorch - INFO - Trn step 100 of 1955 trn lossavg 0.08413\n", 1319 | "2020-01-27 18:11:03,324 - Recursion-pytorch - INFO - Trn step 150 of 1955 trn lossavg 0.07459\n", 1320 | "2020-01-27 18:11:10,198 - Recursion-pytorch - INFO - Trn step 200 of 1955 trn lossavg 0.07127\n", 1321 | "2020-01-27 18:11:17,022 - Recursion-pytorch - INFO - Trn step 250 of 1955 trn lossavg 0.06836\n", 1322 | "2020-01-27 18:11:23,844 - Recursion-pytorch - INFO - Trn step 300 of 1955 trn lossavg 0.06579\n", 1323 | "2020-01-27 18:11:30,691 - Recursion-pytorch - INFO - Trn step 350 of 1955 trn lossavg 0.06505\n", 1324 | "2020-01-27 18:11:37,489 - Recursion-pytorch - INFO - Trn step 400 of 1955 trn lossavg 0.06321\n", 1325 | "2020-01-27 18:11:44,411 - Recursion-pytorch - INFO - Trn step 450 of 1955 trn lossavg 0.06199\n", 1326 | "2020-01-27 18:11:51,282 - Recursion-pytorch - INFO - Trn step 500 of 1955 trn lossavg 0.06179\n", 1327 | "2020-01-27 18:11:58,017 - Recursion-pytorch - INFO - Trn step 550 of 1955 trn lossavg 0.06096\n", 1328 | "2020-01-27 18:12:04,714 - Recursion-pytorch - INFO - Trn step 600 of 1955 trn lossavg 0.06059\n", 1329 | "2020-01-27 18:12:11,541 - Recursion-pytorch - INFO - Trn step 650 of 1955 trn lossavg 0.06041\n", 1330 | "2020-01-27 18:12:18,253 - Recursion-pytorch - INFO - Trn step 700 of 1955 trn lossavg 0.06000\n", 1331 | "2020-01-27 18:12:25,013 - Recursion-pytorch - INFO - Trn step 750 of 1955 trn lossavg 0.05957\n", 1332 | "2020-01-27 18:12:31,913 - Recursion-pytorch - INFO - Trn step 800 of 1955 trn lossavg 0.05910\n", 1333 | "2020-01-27 18:12:38,616 - Recursion-pytorch - INFO - Trn step 850 of 1955 trn lossavg 0.05873\n", 1334 | "2020-01-27 18:12:45,307 - Recursion-pytorch - INFO - Trn step 900 of 1955 trn lossavg 0.05829\n", 1335 | "2020-01-27 18:12:52,163 - Recursion-pytorch - INFO - Trn step 950 of 1955 trn lossavg 0.05772\n", 1336 | "2020-01-27 18:12:58,963 - Recursion-pytorch - INFO - Trn step 1000 of 1955 trn lossavg 0.05717\n", 1337 | "2020-01-27 18:13:05,733 - Recursion-pytorch - INFO - Trn step 1050 of 1955 trn lossavg 0.05665\n", 1338 | "2020-01-27 18:14:28,375 - Recursion-pytorch - INFO - Trn step 1650 of 1955 trn lossavg 0.05500\n", 1339 | "2020-01-27 18:14:35,263 - Recursion-pytorch - INFO - Trn step 1700 of 1955 trn lossavg 0.05498\n", 1340 | "2020-01-27 18:14:42,088 - Recursion-pytorch - INFO - Trn step 1750 of 1955 trn lossavg 0.05480\n", 1341 | "2020-01-27 18:14:48,921 - Recursion-pytorch - INFO - Trn step 1800 of 1955 trn lossavg 0.05485\n", 1342 | "2020-01-27 18:14:55,705 - Recursion-pytorch - INFO - Trn step 1850 of 1955 trn lossavg 0.05489\n", 1343 | "2020-01-27 18:15:02,522 - Recursion-pytorch - INFO - Trn step 1900 of 1955 trn lossavg 0.05455\n", 1344 | "2020-01-27 18:15:09,327 - Recursion-pytorch - INFO - Trn step 1950 of 1955 trn lossavg 0.05448\n", 1345 | "/home/morgan/anaconda3/envs/fastai2_me/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:91: UserWarning: Seems like `optimizer.step()` has been overridden after learning rate scheduler initialization. Please, make sure to call `optimizer.step()` before `lr_scheduler.step()`. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\n", 1346 | " \"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\", UserWarning)\n", 1347 | "2020-01-27 18:15:11,394 - Recursion-pytorch - INFO - Prep val score...\n", 1348 | "2020-01-27 18:15:20,980 - Recursion-pytorch - INFO - Epoch 0 val logloss 0.06363 bagged 0.06363\n", 1349 | "2020-01-27 18:15:21,071 - Recursion-pytorch - INFO - Prep test sub...\n", 1350 | "2020-01-27 18:15:27,252 - Recursion-pytorch - INFO - EPOCH 1\n", 1351 | "2020-01-27 18:15:27,924 - Recursion-pytorch - INFO - Trn step 0 of 1955 trn lossavg 0.05417\n", 1352 | "2020-01-27 18:15:34,871 - Recursion-pytorch - INFO - Trn step 50 of 1955 trn lossavg 0.04680\n", 1353 | "2020-01-27 18:15:41,812 - Recursion-pytorch - INFO - Trn step 100 of 1955 trn lossavg 0.04783\n", 1354 | "2020-01-27 18:15:48,642 - Recursion-pytorch - INFO - Trn step 150 of 1955 trn lossavg 0.04817\n", 1355 | "2020-01-27 18:15:55,580 - Recursion-pytorch - INFO - Trn step 200 of 1955 trn lossavg 0.04970\n", 1356 | "2020-01-27 18:16:02,484 - Recursion-pytorch - INFO - Trn step 250 of 1955 trn lossavg 0.04968\n", 1357 | "2020-01-27 18:16:09,414 - Recursion-pytorch - INFO - Trn step 300 of 1955 trn lossavg 0.04990\n", 1358 | "2020-01-27 18:16:16,348 - Recursion-pytorch - INFO - Trn step 350 of 1955 trn lossavg 0.05013\n", 1359 | "2020-01-27 18:16:23,252 - Recursion-pytorch - INFO - Trn step 400 of 1955 trn lossavg 0.04927\n", 1360 | "2020-01-27 18:16:30,191 - Recursion-pytorch - INFO - Trn step 450 of 1955 trn lossavg 0.04846\n", 1361 | "2020-01-27 18:16:37,108 - Recursion-pytorch - INFO - Trn step 500 of 1955 trn lossavg 0.04813\n", 1362 | "2020-01-27 18:16:44,043 - Recursion-pytorch - INFO - Trn step 550 of 1955 trn lossavg 0.04789\n", 1363 | "2020-01-27 18:16:50,988 - Recursion-pytorch - INFO - Trn step 600 of 1955 trn lossavg 0.04826\n", 1364 | "2020-01-27 18:16:57,991 - Recursion-pytorch - INFO - Trn step 650 of 1955 trn lossavg 0.04837\n", 1365 | "2020-01-27 18:17:04,910 - Recursion-pytorch - INFO - Trn step 700 of 1955 trn lossavg 0.04831\n", 1366 | "2020-01-27 18:17:11,824 - Recursion-pytorch - INFO - Trn step 750 of 1955 trn lossavg 0.04822\n", 1367 | "2020-01-27 18:17:18,746 - Recursion-pytorch - INFO - Trn step 800 of 1955 trn lossavg 0.04865\n", 1368 | "2020-01-27 18:17:25,727 - Recursion-pytorch - INFO - Trn step 850 of 1955 trn lossavg 0.04906\n", 1369 | "2020-01-27 18:17:32,581 - Recursion-pytorch - INFO - Trn step 900 of 1955 trn lossavg 0.04874\n", 1370 | "2020-01-27 18:17:39,450 - Recursion-pytorch - INFO - Trn step 950 of 1955 trn lossavg 0.04973\n", 1371 | "2020-01-27 18:17:46,420 - Recursion-pytorch - INFO - Trn step 1000 of 1955 trn lossavg 0.04992\n", 1372 | "2020-01-27 18:17:53,397 - Recursion-pytorch - INFO - Trn step 1050 of 1955 trn lossavg 0.04970\n", 1373 | "2020-01-27 18:18:00,317 - Recursion-pytorch - INFO - Trn step 1100 of 1955 trn lossavg 0.04951\n", 1374 | "2020-01-27 18:18:07,243 - Recursion-pytorch - INFO - Trn step 1150 of 1955 trn lossavg 0.04947\n", 1375 | "2020-01-27 18:18:14,177 - Recursion-pytorch - INFO - Trn step 1200 of 1955 trn lossavg 0.04966\n", 1376 | "2020-01-27 18:18:21,163 - Recursion-pytorch - INFO - Trn step 1250 of 1955 trn lossavg 0.04960\n", 1377 | "2020-01-27 18:18:28,119 - Recursion-pytorch - INFO - Trn step 1300 of 1955 trn lossavg 0.04925\n", 1378 | "2020-01-27 18:18:35,013 - Recursion-pytorch - INFO - Trn step 1350 of 1955 trn lossavg 0.04932\n", 1379 | "2020-01-27 18:18:42,013 - Recursion-pytorch - INFO - Trn step 1400 of 1955 trn lossavg 0.04927\n", 1380 | "2020-01-27 18:18:48,977 - Recursion-pytorch - INFO - Trn step 1450 of 1955 trn lossavg 0.04942\n", 1381 | "2020-01-27 18:18:55,862 - Recursion-pytorch - INFO - Trn step 1500 of 1955 trn lossavg 0.04968\n", 1382 | "2020-01-27 18:19:02,769 - Recursion-pytorch - INFO - Trn step 1550 of 1955 trn lossavg 0.04956\n", 1383 | "2020-01-27 18:19:09,723 - Recursion-pytorch - INFO - Trn step 1600 of 1955 trn lossavg 0.04962\n", 1384 | "2020-01-27 18:19:16,655 - Recursion-pytorch - INFO - Trn step 1650 of 1955 trn lossavg 0.04979\n", 1385 | "2020-01-27 18:19:23,665 - Recursion-pytorch - INFO - Trn step 1700 of 1955 trn lossavg 0.04976\n", 1386 | "2020-01-27 18:19:30,537 - Recursion-pytorch - INFO - Trn step 1750 of 1955 trn lossavg 0.04977\n", 1387 | "2020-01-27 18:19:37,426 - Recursion-pytorch - INFO - Trn step 1800 of 1955 trn lossavg 0.04991\n", 1388 | "2020-01-27 18:19:44,326 - Recursion-pytorch - INFO - Trn step 1850 of 1955 trn lossavg 0.04997\n", 1389 | "2020-01-27 18:19:51,286 - Recursion-pytorch - INFO - Trn step 1900 of 1955 trn lossavg 0.04991\n", 1390 | "2020-01-27 18:19:58,102 - Recursion-pytorch - INFO - Trn step 1950 of 1955 trn lossavg 0.04969\n", 1391 | "2020-01-27 18:20:00,187 - Recursion-pytorch - INFO - Prep val score...\n", 1392 | "2020-01-27 18:20:09,800 - Recursion-pytorch - INFO - Epoch 1 val logloss 0.06619 bagged 0.06332\n", 1393 | "2020-01-27 18:20:09,893 - Recursion-pytorch - INFO - Prep test sub...\n", 1394 | "2020-01-27 18:20:15,999 - Recursion-pytorch - INFO - EPOCH 2\n", 1395 | "2020-01-27 18:20:16,661 - Recursion-pytorch - INFO - Trn step 0 of 1955 trn lossavg 0.05593\n", 1396 | "2020-01-27 18:20:23,663 - Recursion-pytorch - INFO - Trn step 50 of 1955 trn lossavg 0.05982\n", 1397 | "2020-01-27 18:20:30,546 - Recursion-pytorch - INFO - Trn step 100 of 1955 trn lossavg 0.05657\n", 1398 | "2020-01-27 18:20:37,417 - Recursion-pytorch - INFO - Trn step 150 of 1955 trn lossavg 0.05311\n", 1399 | "2020-01-27 18:20:44,339 - Recursion-pytorch - INFO - Trn step 200 of 1955 trn lossavg 0.05205\n", 1400 | "2020-01-27 18:20:51,263 - Recursion-pytorch - INFO - Trn step 250 of 1955 trn lossavg 0.05066\n", 1401 | "2020-01-27 18:20:58,208 - Recursion-pytorch - INFO - Trn step 300 of 1955 trn lossavg 0.05025\n" 1402 | ] 1403 | }, 1404 | { 1405 | "name": "stderr", 1406 | "output_type": "stream", 1407 | "text": [ 1408 | "2020-01-27 18:21:05,169 - Recursion-pytorch - INFO - Trn step 350 of 1955 trn lossavg 0.04909\n", 1409 | "2020-01-27 18:21:12,139 - Recursion-pytorch - INFO - Trn step 400 of 1955 trn lossavg 0.04983\n", 1410 | "2020-01-27 18:21:19,113 - Recursion-pytorch - INFO - Trn step 450 of 1955 trn lossavg 0.04931\n", 1411 | "2020-01-27 18:21:26,055 - Recursion-pytorch - INFO - Trn step 500 of 1955 trn lossavg 0.04907\n", 1412 | "2020-01-27 18:21:32,960 - Recursion-pytorch - INFO - Trn step 550 of 1955 trn lossavg 0.04937\n", 1413 | "2020-01-27 18:21:39,916 - Recursion-pytorch - INFO - Trn step 600 of 1955 trn lossavg 0.04928\n", 1414 | "2020-01-27 18:21:46,859 - Recursion-pytorch - INFO - Trn step 650 of 1955 trn lossavg 0.04890\n", 1415 | "2020-01-27 18:21:53,774 - Recursion-pytorch - INFO - Trn step 700 of 1955 trn lossavg 0.04905\n", 1416 | "2020-01-27 18:22:00,583 - Recursion-pytorch - INFO - Trn step 750 of 1955 trn lossavg 0.04910\n", 1417 | "2020-01-27 18:22:07,547 - Recursion-pytorch - INFO - Trn step 800 of 1955 trn lossavg 0.04877\n", 1418 | "2020-01-27 18:22:14,489 - Recursion-pytorch - INFO - Trn step 850 of 1955 trn lossavg 0.04867\n", 1419 | "2020-01-27 18:22:21,444 - Recursion-pytorch - INFO - Trn step 900 of 1955 trn lossavg 0.04862\n", 1420 | "2020-01-27 18:22:28,444 - Recursion-pytorch - INFO - Trn step 950 of 1955 trn lossavg 0.04868\n", 1421 | "2020-01-27 18:22:35,280 - Recursion-pytorch - INFO - Trn step 1000 of 1955 trn lossavg 0.04856\n", 1422 | "2020-01-27 18:22:42,119 - Recursion-pytorch - INFO - Trn step 1050 of 1955 trn lossavg 0.04812\n", 1423 | "2020-01-27 18:22:48,947 - Recursion-pytorch - INFO - Trn step 1100 of 1955 trn lossavg 0.04836\n", 1424 | "2020-01-27 18:22:55,822 - Recursion-pytorch - INFO - Trn step 1150 of 1955 trn lossavg 0.04857\n", 1425 | "2020-01-27 18:23:02,704 - Recursion-pytorch - INFO - Trn step 1200 of 1955 trn lossavg 0.04864\n", 1426 | "2020-01-27 18:23:09,551 - Recursion-pytorch - INFO - Trn step 1250 of 1955 trn lossavg 0.04853\n", 1427 | "2020-01-27 18:23:16,415 - Recursion-pytorch - INFO - Trn step 1300 of 1955 trn lossavg 0.04870\n", 1428 | "2020-01-27 18:23:23,398 - Recursion-pytorch - INFO - Trn step 1350 of 1955 trn lossavg 0.04863\n", 1429 | "2020-01-27 18:23:30,349 - Recursion-pytorch - INFO - Trn step 1400 of 1955 trn lossavg 0.04855\n", 1430 | "2020-01-27 18:23:37,237 - Recursion-pytorch - INFO - Trn step 1450 of 1955 trn lossavg 0.04838\n", 1431 | "2020-01-27 18:23:44,123 - Recursion-pytorch - INFO - Trn step 1500 of 1955 trn lossavg 0.04820\n", 1432 | "2020-01-27 18:23:50,921 - Recursion-pytorch - INFO - Trn step 1550 of 1955 trn lossavg 0.04813\n", 1433 | "2020-01-27 18:23:57,799 - Recursion-pytorch - INFO - Trn step 1600 of 1955 trn lossavg 0.04830\n", 1434 | "2020-01-27 18:24:04,633 - Recursion-pytorch - INFO - Trn step 1650 of 1955 trn lossavg 0.04805\n", 1435 | "2020-01-27 18:24:11,489 - Recursion-pytorch - INFO - Trn step 1700 of 1955 trn lossavg 0.04813\n", 1436 | "2020-01-27 18:24:18,285 - Recursion-pytorch - INFO - Trn step 1750 of 1955 trn lossavg 0.04836\n", 1437 | "2020-01-27 18:24:25,122 - Recursion-pytorch - INFO - Trn step 1800 of 1955 trn lossavg 0.04832\n", 1438 | "2020-01-27 18:24:31,966 - Recursion-pytorch - INFO - Trn step 1850 of 1955 trn lossavg 0.04819\n", 1439 | "2020-01-27 18:24:38,895 - Recursion-pytorch - INFO - Trn step 1900 of 1955 trn lossavg 0.04819\n", 1440 | "2020-01-27 18:24:45,650 - Recursion-pytorch - INFO - Trn step 1950 of 1955 trn lossavg 0.04811\n", 1441 | "2020-01-27 18:24:47,668 - Recursion-pytorch - INFO - Prep val score...\n", 1442 | "2020-01-27 18:24:57,244 - Recursion-pytorch - INFO - Epoch 2 val logloss 0.06626 bagged 0.06327\n", 1443 | "2020-01-27 18:24:57,335 - Recursion-pytorch - INFO - Prep test sub...\n", 1444 | "2020-01-27 18:25:03,410 - Recursion-pytorch - INFO - EPOCH 3\n", 1445 | "2020-01-27 18:25:04,063 - Recursion-pytorch - INFO - Trn step 0 of 1955 trn lossavg 0.07544\n", 1446 | "2020-01-27 18:25:11,081 - Recursion-pytorch - INFO - Trn step 50 of 1955 trn lossavg 0.05136\n", 1447 | "2020-01-27 18:25:18,003 - Recursion-pytorch - INFO - Trn step 100 of 1955 trn lossavg 0.04740\n", 1448 | "2020-01-27 18:25:24,971 - Recursion-pytorch - INFO - Trn step 150 of 1955 trn lossavg 0.04695\n", 1449 | "2020-01-27 18:25:31,882 - Recursion-pytorch - INFO - Trn step 200 of 1955 trn lossavg 0.04803\n", 1450 | "2020-01-27 18:25:38,846 - Recursion-pytorch - INFO - Trn step 250 of 1955 trn lossavg 0.04682\n", 1451 | "2020-01-27 18:25:45,773 - Recursion-pytorch - INFO - Trn step 300 of 1955 trn lossavg 0.04690\n", 1452 | "2020-01-27 18:25:52,637 - Recursion-pytorch - INFO - Trn step 350 of 1955 trn lossavg 0.04674\n", 1453 | "2020-01-27 18:25:59,606 - Recursion-pytorch - INFO - Trn step 400 of 1955 trn lossavg 0.04676\n", 1454 | "2020-01-27 18:26:06,569 - Recursion-pytorch - INFO - Trn step 450 of 1955 trn lossavg 0.04669\n", 1455 | "2020-01-27 18:26:13,442 - Recursion-pytorch - INFO - Trn step 500 of 1955 trn lossavg 0.04638\n", 1456 | "2020-01-27 18:26:20,411 - Recursion-pytorch - INFO - Trn step 550 of 1955 trn lossavg 0.04599\n", 1457 | "2020-01-27 18:26:27,278 - Recursion-pytorch - INFO - Trn step 600 of 1955 trn lossavg 0.04588\n", 1458 | "2020-01-27 18:26:34,244 - Recursion-pytorch - INFO - Trn step 650 of 1955 trn lossavg 0.04622\n", 1459 | "2020-01-27 18:26:41,117 - Recursion-pytorch - INFO - Trn step 700 of 1955 trn lossavg 0.04627\n", 1460 | "2020-01-27 18:26:47,935 - Recursion-pytorch - INFO - Trn step 750 of 1955 trn lossavg 0.04623\n", 1461 | "2020-01-27 18:26:54,742 - Recursion-pytorch - INFO - Trn step 800 of 1955 trn lossavg 0.04611\n", 1462 | "2020-01-27 18:27:01,653 - Recursion-pytorch - INFO - Trn step 850 of 1955 trn lossavg 0.04616\n", 1463 | "2020-01-27 18:27:08,506 - Recursion-pytorch - INFO - Trn step 900 of 1955 trn lossavg 0.04650\n", 1464 | "2020-01-27 18:27:15,412 - Recursion-pytorch - INFO - Trn step 950 of 1955 trn lossavg 0.04644\n", 1465 | "2020-01-27 18:27:22,341 - Recursion-pytorch - INFO - Trn step 1000 of 1955 trn lossavg 0.04690\n", 1466 | "2020-01-27 18:27:29,298 - Recursion-pytorch - INFO - Trn step 1050 of 1955 trn lossavg 0.04736\n", 1467 | "2020-01-27 18:27:36,170 - Recursion-pytorch - INFO - Trn step 1100 of 1955 trn lossavg 0.04736\n", 1468 | "2020-01-27 18:27:43,055 - Recursion-pytorch - INFO - Trn step 1150 of 1955 trn lossavg 0.04709\n", 1469 | "2020-01-27 18:27:49,939 - Recursion-pytorch - INFO - Trn step 1200 of 1955 trn lossavg 0.04691\n", 1470 | "2020-01-27 18:27:56,960 - Recursion-pytorch - INFO - Trn step 1250 of 1955 trn lossavg 0.04678\n", 1471 | "2020-01-27 18:28:03,876 - Recursion-pytorch - INFO - Trn step 1300 of 1955 trn lossavg 0.04671\n", 1472 | "2020-01-27 18:28:10,806 - Recursion-pytorch - INFO - Trn step 1350 of 1955 trn lossavg 0.04676\n", 1473 | "2020-01-27 18:28:17,731 - Recursion-pytorch - INFO - Trn step 1400 of 1955 trn lossavg 0.04647\n", 1474 | "2020-01-27 18:28:24,595 - Recursion-pytorch - INFO - Trn step 1450 of 1955 trn lossavg 0.04646\n", 1475 | "2020-01-27 18:28:31,543 - Recursion-pytorch - INFO - Trn step 1500 of 1955 trn lossavg 0.04630\n", 1476 | "2020-01-27 18:28:38,453 - Recursion-pytorch - INFO - Trn step 1550 of 1955 trn lossavg 0.04637\n" 1477 | ] 1478 | }, 1479 | { 1480 | "name": "stdout", 1481 | "output_type": "stream", 1482 | "text": [ 1483 | "Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 262144.0\n" 1484 | ] 1485 | }, 1486 | { 1487 | "name": "stderr", 1488 | "output_type": "stream", 1489 | "text": [ 1490 | "2020-01-27 18:28:45,401 - Recursion-pytorch - INFO - Trn step 1600 of 1955 trn lossavg 0.04624\n", 1491 | "2020-01-27 18:28:52,320 - Recursion-pytorch - INFO - Trn step 1650 of 1955 trn lossavg 0.04625\n", 1492 | "2020-01-27 18:28:59,174 - Recursion-pytorch - INFO - Trn step 1700 of 1955 trn lossavg 0.04636\n", 1493 | "2020-01-27 18:29:06,092 - Recursion-pytorch - INFO - Trn step 1750 of 1955 trn lossavg 0.04657\n", 1494 | "2020-01-27 18:29:12,915 - Recursion-pytorch - INFO - Trn step 1800 of 1955 trn lossavg 0.04662\n", 1495 | "2020-01-27 18:29:19,891 - Recursion-pytorch - INFO - Trn step 1850 of 1955 trn lossavg 0.04659\n", 1496 | "2020-01-27 18:29:26,913 - Recursion-pytorch - INFO - Trn step 1900 of 1955 trn lossavg 0.04650\n", 1497 | "2020-01-27 18:29:33,645 - Recursion-pytorch - INFO - Trn step 1950 of 1955 trn lossavg 0.04653\n", 1498 | "2020-01-27 18:29:35,681 - Recursion-pytorch - INFO - Prep val score...\n", 1499 | "2020-01-27 18:29:45,253 - Recursion-pytorch - INFO - Epoch 3 val logloss 0.06401 bagged 0.06296\n", 1500 | "2020-01-27 18:29:45,346 - Recursion-pytorch - INFO - Prep test sub...\n", 1501 | "2020-01-27 18:29:51,520 - Recursion-pytorch - INFO - EPOCH 4\n", 1502 | "2020-01-27 18:29:52,194 - Recursion-pytorch - INFO - Trn step 0 of 1955 trn lossavg 0.10785\n", 1503 | "2020-01-27 18:29:59,239 - Recursion-pytorch - INFO - Trn step 50 of 1955 trn lossavg 0.04355\n", 1504 | "2020-01-27 18:30:06,260 - Recursion-pytorch - INFO - Trn step 100 of 1955 trn lossavg 0.04139\n", 1505 | "2020-01-27 18:30:13,346 - Recursion-pytorch - INFO - Trn step 150 of 1955 trn lossavg 0.04365\n", 1506 | "2020-01-27 18:30:20,312 - Recursion-pytorch - INFO - Trn step 200 of 1955 trn lossavg 0.04306\n", 1507 | "2020-01-27 18:30:27,236 - Recursion-pytorch - INFO - Trn step 250 of 1955 trn lossavg 0.04212\n", 1508 | "2020-01-27 18:30:34,205 - Recursion-pytorch - INFO - Trn step 300 of 1955 trn lossavg 0.04171\n", 1509 | "2020-01-27 18:30:41,213 - Recursion-pytorch - INFO - Trn step 350 of 1955 trn lossavg 0.04175\n", 1510 | "2020-01-27 18:30:48,174 - Recursion-pytorch - INFO - Trn step 400 of 1955 trn lossavg 0.04185\n", 1511 | "2020-01-27 18:30:55,152 - Recursion-pytorch - INFO - Trn step 450 of 1955 trn lossavg 0.04163\n", 1512 | "2020-01-27 18:31:02,056 - Recursion-pytorch - INFO - Trn step 500 of 1955 trn lossavg 0.04172\n", 1513 | "2020-01-27 18:31:09,000 - Recursion-pytorch - INFO - Trn step 550 of 1955 trn lossavg 0.04251\n", 1514 | "2020-01-27 18:31:15,875 - Recursion-pytorch - INFO - Trn step 600 of 1955 trn lossavg 0.04297\n", 1515 | "2020-01-27 18:31:22,843 - Recursion-pytorch - INFO - Trn step 650 of 1955 trn lossavg 0.04306\n", 1516 | "2020-01-27 18:31:29,749 - Recursion-pytorch - INFO - Trn step 700 of 1955 trn lossavg 0.04374\n", 1517 | "2020-01-27 18:31:36,653 - Recursion-pytorch - INFO - Trn step 750 of 1955 trn lossavg 0.04392\n", 1518 | "2020-01-27 18:31:43,530 - Recursion-pytorch - INFO - Trn step 800 of 1955 trn lossavg 0.04387\n", 1519 | "2020-01-27 18:31:50,432 - Recursion-pytorch - INFO - Trn step 850 of 1955 trn lossavg 0.04386\n", 1520 | "2020-01-27 18:31:57,309 - Recursion-pytorch - INFO - Trn step 900 of 1955 trn lossavg 0.04375\n", 1521 | "2020-01-27 18:32:04,246 - Recursion-pytorch - INFO - Trn step 950 of 1955 trn lossavg 0.04363\n", 1522 | "2020-01-27 18:32:11,196 - Recursion-pytorch - INFO - Trn step 1000 of 1955 trn lossavg 0.04387\n", 1523 | "2020-01-27 18:32:18,193 - Recursion-pytorch - INFO - Trn step 1050 of 1955 trn lossavg 0.04405\n", 1524 | "2020-01-27 18:32:25,192 - Recursion-pytorch - INFO - Trn step 1100 of 1955 trn lossavg 0.04419\n", 1525 | "2020-01-27 18:32:32,232 - Recursion-pytorch - INFO - Trn step 1150 of 1955 trn lossavg 0.04420\n", 1526 | "2020-01-27 18:32:39,413 - Recursion-pytorch - INFO - Trn step 1200 of 1955 trn lossavg 0.04445\n", 1527 | "2020-01-27 18:32:46,475 - Recursion-pytorch - INFO - Trn step 1250 of 1955 trn lossavg 0.04460\n", 1528 | "2020-01-27 18:32:53,516 - Recursion-pytorch - INFO - Trn step 1300 of 1955 trn lossavg 0.04465\n", 1529 | "2020-01-27 18:33:00,514 - Recursion-pytorch - INFO - Trn step 1350 of 1955 trn lossavg 0.04473\n", 1530 | "2020-01-27 18:33:07,523 - Recursion-pytorch - INFO - Trn step 1400 of 1955 trn lossavg 0.04469\n", 1531 | "2020-01-27 18:33:14,567 - Recursion-pytorch - INFO - Trn step 1450 of 1955 trn lossavg 0.04480\n", 1532 | "2020-01-27 18:33:21,562 - Recursion-pytorch - INFO - Trn step 1500 of 1955 trn lossavg 0.04471\n", 1533 | "2020-01-27 18:33:28,650 - Recursion-pytorch - INFO - Trn step 1550 of 1955 trn lossavg 0.04460\n", 1534 | "2020-01-27 18:33:35,768 - Recursion-pytorch - INFO - Trn step 1600 of 1955 trn lossavg 0.04451\n", 1535 | "2020-01-27 18:33:42,736 - Recursion-pytorch - INFO - Trn step 1650 of 1955 trn lossavg 0.04448\n", 1536 | "2020-01-27 18:33:49,691 - Recursion-pytorch - INFO - Trn step 1700 of 1955 trn lossavg 0.04458\n", 1537 | "2020-01-27 18:33:56,663 - Recursion-pytorch - INFO - Trn step 1750 of 1955 trn lossavg 0.04448\n", 1538 | "2020-01-27 18:34:03,688 - Recursion-pytorch - INFO - Trn step 1800 of 1955 trn lossavg 0.04471\n", 1539 | "2020-01-27 18:34:10,692 - Recursion-pytorch - INFO - Trn step 1850 of 1955 trn lossavg 0.04466\n", 1540 | "2020-01-27 18:34:17,697 - Recursion-pytorch - INFO - Trn step 1900 of 1955 trn lossavg 0.04458\n", 1541 | "2020-01-27 18:34:24,604 - Recursion-pytorch - INFO - Trn step 1950 of 1955 trn lossavg 0.04454\n", 1542 | "2020-01-27 18:34:26,626 - Recursion-pytorch - INFO - Prep val score...\n", 1543 | "2020-01-27 18:34:36,249 - Recursion-pytorch - INFO - Epoch 4 val logloss 0.06482 bagged 0.06288\n", 1544 | "2020-01-27 18:34:36,341 - Recursion-pytorch - INFO - Prep test sub...\n", 1545 | "2020-01-27 18:34:42,488 - Recursion-pytorch - INFO - EPOCH 5\n", 1546 | "2020-01-27 18:34:43,135 - Recursion-pytorch - INFO - Trn step 0 of 1955 trn lossavg 0.00137\n", 1547 | "2020-01-27 18:34:49,961 - Recursion-pytorch - INFO - Trn step 50 of 1955 trn lossavg 0.04327\n", 1548 | "2020-01-27 18:34:57,107 - Recursion-pytorch - INFO - Trn step 100 of 1955 trn lossavg 0.04441\n", 1549 | "2020-01-27 18:35:04,159 - Recursion-pytorch - INFO - Trn step 150 of 1955 trn lossavg 0.04374\n", 1550 | "2020-01-27 18:35:11,375 - Recursion-pytorch - INFO - Trn step 200 of 1955 trn lossavg 0.04175\n", 1551 | "2020-01-27 18:35:18,537 - Recursion-pytorch - INFO - Trn step 250 of 1955 trn lossavg 0.04089\n", 1552 | "2020-01-27 18:35:25,650 - Recursion-pytorch - INFO - Trn step 300 of 1955 trn lossavg 0.04083\n", 1553 | "2020-01-27 18:35:32,828 - Recursion-pytorch - INFO - Trn step 350 of 1955 trn lossavg 0.04156\n", 1554 | "2020-01-27 18:35:40,023 - Recursion-pytorch - INFO - Trn step 400 of 1955 trn lossavg 0.04146\n", 1555 | "2020-01-27 18:35:47,088 - Recursion-pytorch - INFO - Trn step 450 of 1955 trn lossavg 0.04125\n", 1556 | "2020-01-27 18:35:54,276 - Recursion-pytorch - INFO - Trn step 500 of 1955 trn lossavg 0.04128\n", 1557 | "2020-01-27 18:36:01,435 - Recursion-pytorch - INFO - Trn step 550 of 1955 trn lossavg 0.04096\n", 1558 | "2020-01-27 18:36:08,481 - Recursion-pytorch - INFO - Trn step 600 of 1955 trn lossavg 0.04098\n", 1559 | "2020-01-27 18:36:15,457 - Recursion-pytorch - INFO - Trn step 650 of 1955 trn lossavg 0.04122\n", 1560 | "2020-01-27 18:36:22,579 - Recursion-pytorch - INFO - Trn step 700 of 1955 trn lossavg 0.04205\n", 1561 | "2020-01-27 18:36:29,697 - Recursion-pytorch - INFO - Trn step 750 of 1955 trn lossavg 0.04183\n", 1562 | "2020-01-27 18:36:36,791 - Recursion-pytorch - INFO - Trn step 800 of 1955 trn lossavg 0.04159\n", 1563 | "2020-01-27 18:36:43,949 - Recursion-pytorch - INFO - Trn step 850 of 1955 trn lossavg 0.04175\n", 1564 | "2020-01-27 18:36:51,054 - Recursion-pytorch - INFO - Trn step 900 of 1955 trn lossavg 0.04156\n", 1565 | "2020-01-27 18:36:58,080 - Recursion-pytorch - INFO - Trn step 950 of 1955 trn lossavg 0.04182\n", 1566 | "2020-01-27 18:37:05,137 - Recursion-pytorch - INFO - Trn step 1000 of 1955 trn lossavg 0.04168\n", 1567 | "2020-01-27 18:37:12,184 - Recursion-pytorch - INFO - Trn step 1050 of 1955 trn lossavg 0.04165\n", 1568 | "2020-01-27 18:37:19,112 - Recursion-pytorch - INFO - Trn step 1100 of 1955 trn lossavg 0.04162\n", 1569 | "2020-01-27 18:37:26,051 - Recursion-pytorch - INFO - Trn step 1150 of 1955 trn lossavg 0.04160\n", 1570 | "2020-01-27 18:37:33,114 - Recursion-pytorch - INFO - Trn step 1200 of 1955 trn lossavg 0.04177\n", 1571 | "2020-01-27 18:37:40,234 - Recursion-pytorch - INFO - Trn step 1250 of 1955 trn lossavg 0.04167\n", 1572 | "2020-01-27 18:37:47,331 - Recursion-pytorch - INFO - Trn step 1300 of 1955 trn lossavg 0.04161\n", 1573 | "2020-01-27 18:37:54,396 - Recursion-pytorch - INFO - Trn step 1350 of 1955 trn lossavg 0.04173\n", 1574 | "2020-01-27 18:38:01,442 - Recursion-pytorch - INFO - Trn step 1400 of 1955 trn lossavg 0.04184\n", 1575 | "2020-01-27 18:38:08,533 - Recursion-pytorch - INFO - Trn step 1450 of 1955 trn lossavg 0.04210\n" 1576 | ] 1577 | }, 1578 | { 1579 | "name": "stdout", 1580 | "output_type": "stream", 1581 | "text": [ 1582 | "Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 262144.0\n" 1583 | ] 1584 | }, 1585 | { 1586 | "name": "stderr", 1587 | "output_type": "stream", 1588 | "text": [ 1589 | "2020-01-27 18:38:15,637 - Recursion-pytorch - INFO - Trn step 1500 of 1955 trn lossavg 0.04211\n", 1590 | "2020-01-27 18:38:22,712 - Recursion-pytorch - INFO - Trn step 1550 of 1955 trn lossavg 0.04203\n", 1591 | "2020-01-27 18:38:29,823 - Recursion-pytorch - INFO - Trn step 1600 of 1955 trn lossavg 0.04200\n", 1592 | "2020-01-27 18:38:36,894 - Recursion-pytorch - INFO - Trn step 1650 of 1955 trn lossavg 0.04218\n", 1593 | "2020-01-27 18:38:43,918 - Recursion-pytorch - INFO - Trn step 1700 of 1955 trn lossavg 0.04212\n", 1594 | "2020-01-27 18:38:51,049 - Recursion-pytorch - INFO - Trn step 1750 of 1955 trn lossavg 0.04220\n", 1595 | "2020-01-27 18:38:58,149 - Recursion-pytorch - INFO - Trn step 1800 of 1955 trn lossavg 0.04223\n", 1596 | "2020-01-27 18:39:05,195 - Recursion-pytorch - INFO - Trn step 1850 of 1955 trn lossavg 0.04214\n", 1597 | "2020-01-27 18:39:12,579 - Recursion-pytorch - INFO - Trn step 1900 of 1955 trn lossavg 0.04216\n", 1598 | "2020-01-27 18:39:19,629 - Recursion-pytorch - INFO - Trn step 1950 of 1955 trn lossavg 0.04207\n", 1599 | "2020-01-27 18:39:21,651 - Recursion-pytorch - INFO - Prep val score...\n", 1600 | "2020-01-27 18:39:31,290 - Recursion-pytorch - INFO - Epoch 5 val logloss 0.06898 bagged 0.06289\n", 1601 | "2020-01-27 18:39:31,383 - Recursion-pytorch - INFO - Prep test sub...\n", 1602 | "2020-01-27 18:39:37,499 - Recursion-pytorch - INFO - EPOCH 6\n", 1603 | "2020-01-27 18:39:38,147 - Recursion-pytorch - INFO - Trn step 0 of 1955 trn lossavg 0.06551\n", 1604 | "2020-01-27 18:39:45,201 - Recursion-pytorch - INFO - Trn step 50 of 1955 trn lossavg 0.04076\n", 1605 | "2020-01-27 18:39:52,259 - Recursion-pytorch - INFO - Trn step 100 of 1955 trn lossavg 0.03768\n", 1606 | "2020-01-27 18:39:59,230 - Recursion-pytorch - INFO - Trn step 150 of 1955 trn lossavg 0.03941\n", 1607 | "2020-01-27 18:40:06,268 - Recursion-pytorch - INFO - Trn step 200 of 1955 trn lossavg 0.03928\n", 1608 | "2020-01-27 18:40:13,280 - Recursion-pytorch - INFO - Trn step 250 of 1955 trn lossavg 0.03909\n", 1609 | "2020-01-27 18:40:20,264 - Recursion-pytorch - INFO - Trn step 300 of 1955 trn lossavg 0.03898\n", 1610 | "2020-01-27 18:40:27,221 - Recursion-pytorch - INFO - Trn step 350 of 1955 trn lossavg 0.03857\n", 1611 | "2020-01-27 18:40:34,044 - Recursion-pytorch - INFO - Trn step 400 of 1955 trn lossavg 0.03782\n", 1612 | "2020-01-27 18:40:40,994 - Recursion-pytorch - INFO - Trn step 450 of 1955 trn lossavg 0.03858\n", 1613 | "2020-01-27 18:40:47,933 - Recursion-pytorch - INFO - Trn step 500 of 1955 trn lossavg 0.03849\n", 1614 | "2020-01-27 18:40:54,766 - Recursion-pytorch - INFO - Trn step 550 of 1955 trn lossavg 0.03827\n", 1615 | "2020-01-27 18:41:01,747 - Recursion-pytorch - INFO - Trn step 600 of 1955 trn lossavg 0.03798\n", 1616 | "2020-01-27 18:41:08,611 - Recursion-pytorch - INFO - Trn step 650 of 1955 trn lossavg 0.03836\n", 1617 | "2020-01-27 18:41:15,566 - Recursion-pytorch - INFO - Trn step 700 of 1955 trn lossavg 0.03844\n", 1618 | "2020-01-27 18:41:22,450 - Recursion-pytorch - INFO - Trn step 750 of 1955 trn lossavg 0.03855\n", 1619 | "2020-01-27 18:41:29,489 - Recursion-pytorch - INFO - Trn step 800 of 1955 trn lossavg 0.03845\n", 1620 | "2020-01-27 18:41:36,460 - Recursion-pytorch - INFO - Trn step 850 of 1955 trn lossavg 0.03831\n", 1621 | "2020-01-27 18:41:43,511 - Recursion-pytorch - INFO - Trn step 900 of 1955 trn lossavg 0.03852\n", 1622 | "2020-01-27 18:41:50,403 - Recursion-pytorch - INFO - Trn step 950 of 1955 trn lossavg 0.03824\n", 1623 | "2020-01-27 18:41:57,341 - Recursion-pytorch - INFO - Trn step 1000 of 1955 trn lossavg 0.03829\n", 1624 | "2020-01-27 18:42:04,310 - Recursion-pytorch - INFO - Trn step 1050 of 1955 trn lossavg 0.03833\n", 1625 | "2020-01-27 18:42:11,160 - Recursion-pytorch - INFO - Trn step 1100 of 1955 trn lossavg 0.03823\n", 1626 | "2020-01-27 18:42:18,049 - Recursion-pytorch - INFO - Trn step 1150 of 1955 trn lossavg 0.03841\n", 1627 | "2020-01-27 18:42:25,074 - Recursion-pytorch - INFO - Trn step 1200 of 1955 trn lossavg 0.03837\n", 1628 | "2020-01-27 18:42:31,988 - Recursion-pytorch - INFO - Trn step 1250 of 1955 trn lossavg 0.03830\n", 1629 | "2020-01-27 18:42:38,987 - Recursion-pytorch - INFO - Trn step 1300 of 1955 trn lossavg 0.03841\n", 1630 | "2020-01-27 18:42:45,896 - Recursion-pytorch - INFO - Trn step 1350 of 1955 trn lossavg 0.03849\n", 1631 | "2020-01-27 18:42:52,823 - Recursion-pytorch - INFO - Trn step 1400 of 1955 trn lossavg 0.03838\n", 1632 | "2020-01-27 18:42:59,688 - Recursion-pytorch - INFO - Trn step 1450 of 1955 trn lossavg 0.03849\n", 1633 | "2020-01-27 18:43:06,691 - Recursion-pytorch - INFO - Trn step 1500 of 1955 trn lossavg 0.03837\n", 1634 | "2020-01-27 18:43:13,603 - Recursion-pytorch - INFO - Trn step 1550 of 1955 trn lossavg 0.03839\n", 1635 | "2020-01-27 18:43:20,533 - Recursion-pytorch - INFO - Trn step 1600 of 1955 trn lossavg 0.03845\n", 1636 | "2020-01-27 18:43:27,531 - Recursion-pytorch - INFO - Trn step 1650 of 1955 trn lossavg 0.03856\n", 1637 | "2020-01-27 18:43:34,445 - Recursion-pytorch - INFO - Trn step 1700 of 1955 trn lossavg 0.03861\n", 1638 | "2020-01-27 18:43:41,379 - Recursion-pytorch - INFO - Trn step 1750 of 1955 trn lossavg 0.03859\n", 1639 | "2020-01-27 18:43:48,372 - Recursion-pytorch - INFO - Trn step 1800 of 1955 trn lossavg 0.03857\n", 1640 | "2020-01-27 18:43:55,216 - Recursion-pytorch - INFO - Trn step 1850 of 1955 trn lossavg 0.03854\n", 1641 | "2020-01-27 18:44:02,051 - Recursion-pytorch - INFO - Trn step 1900 of 1955 trn lossavg 0.03860\n", 1642 | "2020-01-27 18:44:08,907 - Recursion-pytorch - INFO - Trn step 1950 of 1955 trn lossavg 0.03860\n", 1643 | "2020-01-27 18:44:10,955 - Recursion-pytorch - INFO - Prep val score...\n", 1644 | "2020-01-27 18:44:20,551 - Recursion-pytorch - INFO - Epoch 6 val logloss 0.07149 bagged 0.06307\n", 1645 | "2020-01-27 18:44:20,643 - Recursion-pytorch - INFO - Prep test sub...\n", 1646 | "2020-01-27 18:44:26,751 - Recursion-pytorch - INFO - EPOCH 7\n", 1647 | "2020-01-27 18:44:27,436 - Recursion-pytorch - INFO - Trn step 0 of 1955 trn lossavg 0.05759\n", 1648 | "2020-01-27 18:44:34,370 - Recursion-pytorch - INFO - Trn step 50 of 1955 trn lossavg 0.03158\n", 1649 | "2020-01-27 18:44:41,340 - Recursion-pytorch - INFO - Trn step 100 of 1955 trn lossavg 0.03153\n", 1650 | "2020-01-27 18:44:48,304 - Recursion-pytorch - INFO - Trn step 150 of 1955 trn lossavg 0.03251\n", 1651 | "2020-01-27 18:44:55,307 - Recursion-pytorch - INFO - Trn step 200 of 1955 trn lossavg 0.03197\n", 1652 | "2020-01-27 18:45:02,296 - Recursion-pytorch - INFO - Trn step 250 of 1955 trn lossavg 0.03234\n", 1653 | "2020-01-27 18:45:09,330 - Recursion-pytorch - INFO - Trn step 300 of 1955 trn lossavg 0.03242\n", 1654 | "2020-01-27 18:45:16,401 - Recursion-pytorch - INFO - Trn step 350 of 1955 trn lossavg 0.03276\n", 1655 | "2020-01-27 18:45:23,366 - Recursion-pytorch - INFO - Trn step 400 of 1955 trn lossavg 0.03334\n", 1656 | "2020-01-27 18:45:30,339 - Recursion-pytorch - INFO - Trn step 450 of 1955 trn lossavg 0.03338\n", 1657 | "2020-01-27 18:45:37,359 - Recursion-pytorch - INFO - Trn step 500 of 1955 trn lossavg 0.03379\n", 1658 | "2020-01-27 18:45:44,329 - Recursion-pytorch - INFO - Trn step 550 of 1955 trn lossavg 0.03367\n", 1659 | "2020-01-27 18:45:51,325 - Recursion-pytorch - INFO - Trn step 600 of 1955 trn lossavg 0.03418\n", 1660 | "2020-01-27 18:45:58,240 - Recursion-pytorch - INFO - Trn step 650 of 1955 trn lossavg 0.03462\n", 1661 | "2020-01-27 18:46:05,158 - Recursion-pytorch - INFO - Trn step 700 of 1955 trn lossavg 0.03504\n", 1662 | "2020-01-27 18:46:12,059 - Recursion-pytorch - INFO - Trn step 750 of 1955 trn lossavg 0.03512\n", 1663 | "2020-01-27 18:46:19,021 - Recursion-pytorch - INFO - Trn step 800 of 1955 trn lossavg 0.03491\n", 1664 | "2020-01-27 18:46:25,957 - Recursion-pytorch - INFO - Trn step 850 of 1955 trn lossavg 0.03477\n", 1665 | "2020-01-27 18:46:32,950 - Recursion-pytorch - INFO - Trn step 900 of 1955 trn lossavg 0.03494\n", 1666 | "2020-01-27 18:46:39,923 - Recursion-pytorch - INFO - Trn step 950 of 1955 trn lossavg 0.03484\n", 1667 | "2020-01-27 18:46:46,812 - Recursion-pytorch - INFO - Trn step 1000 of 1955 trn lossavg 0.03472\n", 1668 | "2020-01-27 18:46:53,737 - Recursion-pytorch - INFO - Trn step 1050 of 1955 trn lossavg 0.03484\n", 1669 | "2020-01-27 18:47:00,683 - Recursion-pytorch - INFO - Trn step 1100 of 1955 trn lossavg 0.03484\n", 1670 | "2020-01-27 18:47:07,618 - Recursion-pytorch - INFO - Trn step 1150 of 1955 trn lossavg 0.03483\n", 1671 | "2020-01-27 18:47:14,552 - Recursion-pytorch - INFO - Trn step 1200 of 1955 trn lossavg 0.03485\n", 1672 | "2020-01-27 18:47:21,422 - Recursion-pytorch - INFO - Trn step 1250 of 1955 trn lossavg 0.03501\n", 1673 | "2020-01-27 18:47:28,370 - Recursion-pytorch - INFO - Trn step 1300 of 1955 trn lossavg 0.03514\n", 1674 | "2020-01-27 18:47:35,412 - Recursion-pytorch - INFO - Trn step 1350 of 1955 trn lossavg 0.03510\n", 1675 | "2020-01-27 18:47:42,379 - Recursion-pytorch - INFO - Trn step 1400 of 1955 trn lossavg 0.03511\n", 1676 | "2020-01-27 18:47:49,282 - Recursion-pytorch - INFO - Trn step 1450 of 1955 trn lossavg 0.03492\n", 1677 | "2020-01-27 18:47:56,149 - Recursion-pytorch - INFO - Trn step 1500 of 1955 trn lossavg 0.03491\n" 1678 | ] 1679 | }, 1680 | { 1681 | "name": "stderr", 1682 | "output_type": "stream", 1683 | "text": [ 1684 | "2020-01-27 18:48:03,168 - Recursion-pytorch - INFO - Trn step 1550 of 1955 trn lossavg 0.03478\n", 1685 | "2020-01-27 18:48:10,104 - Recursion-pytorch - INFO - Trn step 1600 of 1955 trn lossavg 0.03464\n" 1686 | ] 1687 | }, 1688 | { 1689 | "name": "stdout", 1690 | "output_type": "stream", 1691 | "text": [ 1692 | "Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 524288.0\n" 1693 | ] 1694 | }, 1695 | { 1696 | "name": "stderr", 1697 | "output_type": "stream", 1698 | "text": [ 1699 | "2020-01-27 18:48:16,905 - Recursion-pytorch - INFO - Trn step 1650 of 1955 trn lossavg 0.03473\n", 1700 | "2020-01-27 18:48:23,795 - Recursion-pytorch - INFO - Trn step 1700 of 1955 trn lossavg 0.03467\n", 1701 | "2020-01-27 18:48:30,705 - Recursion-pytorch - INFO - Trn step 1750 of 1955 trn lossavg 0.03452\n", 1702 | "2020-01-27 18:48:37,907 - Recursion-pytorch - INFO - Trn step 1800 of 1955 trn lossavg 0.03443\n", 1703 | "2020-01-27 18:48:44,992 - Recursion-pytorch - INFO - Trn step 1850 of 1955 trn lossavg 0.03427\n", 1704 | "2020-01-27 18:48:52,092 - Recursion-pytorch - INFO - Trn step 1900 of 1955 trn lossavg 0.03432\n", 1705 | "2020-01-27 18:48:59,115 - Recursion-pytorch - INFO - Trn step 1950 of 1955 trn lossavg 0.03432\n", 1706 | "2020-01-27 18:49:01,153 - Recursion-pytorch - INFO - Prep val score...\n", 1707 | "2020-01-27 18:49:10,744 - Recursion-pytorch - INFO - Epoch 7 val logloss 0.07367 bagged 0.06319\n", 1708 | "2020-01-27 18:49:10,841 - Recursion-pytorch - INFO - Prep test sub...\n", 1709 | "2020-01-27 18:49:17,041 - Recursion-pytorch - INFO - EPOCH 8\n", 1710 | "2020-01-27 18:49:17,715 - Recursion-pytorch - INFO - Trn step 0 of 1955 trn lossavg 0.03458\n", 1711 | "2020-01-27 18:49:24,789 - Recursion-pytorch - INFO - Trn step 50 of 1955 trn lossavg 0.03188\n", 1712 | "2020-01-27 18:49:31,945 - Recursion-pytorch - INFO - Trn step 100 of 1955 trn lossavg 0.03036\n", 1713 | "2020-01-27 18:49:38,982 - Recursion-pytorch - INFO - Trn step 150 of 1955 trn lossavg 0.02974\n", 1714 | "2020-01-27 18:49:45,994 - Recursion-pytorch - INFO - Trn step 200 of 1955 trn lossavg 0.02808\n", 1715 | "2020-01-27 18:49:53,048 - Recursion-pytorch - INFO - Trn step 250 of 1955 trn lossavg 0.02722\n", 1716 | "2020-01-27 18:50:00,072 - Recursion-pytorch - INFO - Trn step 300 of 1955 trn lossavg 0.02732\n", 1717 | "2020-01-27 18:50:07,071 - Recursion-pytorch - INFO - Trn step 350 of 1955 trn lossavg 0.02780\n", 1718 | "2020-01-27 18:50:14,130 - Recursion-pytorch - INFO - Trn step 400 of 1955 trn lossavg 0.02821\n", 1719 | "2020-01-27 18:50:21,181 - Recursion-pytorch - INFO - Trn step 450 of 1955 trn lossavg 0.02850\n", 1720 | "2020-01-27 18:50:28,286 - Recursion-pytorch - INFO - Trn step 500 of 1955 trn lossavg 0.02879\n", 1721 | "2020-01-27 18:50:35,405 - Recursion-pytorch - INFO - Trn step 550 of 1955 trn lossavg 0.02855\n", 1722 | "2020-01-27 18:50:42,505 - Recursion-pytorch - INFO - Trn step 600 of 1955 trn lossavg 0.02879\n", 1723 | "2020-01-27 18:50:49,552 - Recursion-pytorch - INFO - Trn step 650 of 1955 trn lossavg 0.02872\n", 1724 | "2020-01-27 18:50:56,554 - Recursion-pytorch - INFO - Trn step 700 of 1955 trn lossavg 0.02841\n", 1725 | "2020-01-27 18:51:03,597 - Recursion-pytorch - INFO - Trn step 750 of 1955 trn lossavg 0.02871\n", 1726 | "2020-01-27 18:51:10,663 - Recursion-pytorch - INFO - Trn step 800 of 1955 trn lossavg 0.02863\n", 1727 | "2020-01-27 18:51:17,791 - Recursion-pytorch - INFO - Trn step 850 of 1955 trn lossavg 0.02869\n", 1728 | "2020-01-27 18:51:24,872 - Recursion-pytorch - INFO - Trn step 900 of 1955 trn lossavg 0.02858\n", 1729 | "2020-01-27 18:51:31,945 - Recursion-pytorch - INFO - Trn step 950 of 1955 trn lossavg 0.02863\n", 1730 | "2020-01-27 18:51:39,014 - Recursion-pytorch - INFO - Trn step 1000 of 1955 trn lossavg 0.02882\n", 1731 | "2020-01-27 18:51:46,003 - Recursion-pytorch - INFO - Trn step 1050 of 1955 trn lossavg 0.02877\n", 1732 | "2020-01-27 18:51:53,136 - Recursion-pytorch - INFO - Trn step 1100 of 1955 trn lossavg 0.02865\n", 1733 | "2020-01-27 18:52:00,288 - Recursion-pytorch - INFO - Trn step 1150 of 1955 trn lossavg 0.02868\n", 1734 | "2020-01-27 18:52:07,334 - Recursion-pytorch - INFO - Trn step 1200 of 1955 trn lossavg 0.02879\n", 1735 | "2020-01-27 18:52:14,410 - Recursion-pytorch - INFO - Trn step 1250 of 1955 trn lossavg 0.02881\n", 1736 | "2020-01-27 18:52:21,498 - Recursion-pytorch - INFO - Trn step 1300 of 1955 trn lossavg 0.02873\n", 1737 | "2020-01-27 18:52:28,655 - Recursion-pytorch - INFO - Trn step 1350 of 1955 trn lossavg 0.02882\n", 1738 | "2020-01-27 18:52:35,680 - Recursion-pytorch - INFO - Trn step 1400 of 1955 trn lossavg 0.02896\n", 1739 | "2020-01-27 18:52:42,851 - Recursion-pytorch - INFO - Trn step 1450 of 1955 trn lossavg 0.02886\n", 1740 | "2020-01-27 18:52:49,867 - Recursion-pytorch - INFO - Trn step 1500 of 1955 trn lossavg 0.02861\n", 1741 | "2020-01-27 18:52:56,877 - Recursion-pytorch - INFO - Trn step 1550 of 1955 trn lossavg 0.02862\n", 1742 | "2020-01-27 18:53:03,957 - Recursion-pytorch - INFO - Trn step 1600 of 1955 trn lossavg 0.02857\n", 1743 | "2020-01-27 18:53:10,994 - Recursion-pytorch - INFO - Trn step 1650 of 1955 trn lossavg 0.02856\n", 1744 | "2020-01-27 18:53:18,011 - Recursion-pytorch - INFO - Trn step 1700 of 1955 trn lossavg 0.02871\n" 1745 | ] 1746 | }, 1747 | { 1748 | "name": "stdout", 1749 | "output_type": "stream", 1750 | "text": [ 1751 | "Gradient overflow. Skipping step, loss scaler 0 reducing loss scale to 524288.0\n" 1752 | ] 1753 | }, 1754 | { 1755 | "name": "stderr", 1756 | "output_type": "stream", 1757 | "text": [ 1758 | "2020-01-27 18:53:25,159 - Recursion-pytorch - INFO - Trn step 1750 of 1955 trn lossavg 0.02874\n", 1759 | "2020-01-27 18:53:32,248 - Recursion-pytorch - INFO - Trn step 1800 of 1955 trn lossavg 0.02877\n", 1760 | "2020-01-27 18:53:39,437 - Recursion-pytorch - INFO - Trn step 1850 of 1955 trn lossavg 0.02865\n", 1761 | "2020-01-27 18:53:46,578 - Recursion-pytorch - INFO - Trn step 1900 of 1955 trn lossavg 0.02867\n", 1762 | "2020-01-27 18:53:53,613 - Recursion-pytorch - INFO - Trn step 1950 of 1955 trn lossavg 0.02862\n", 1763 | "2020-01-27 18:53:55,626 - Recursion-pytorch - INFO - Prep val score...\n", 1764 | "2020-01-27 18:54:05,258 - Recursion-pytorch - INFO - Epoch 8 val logloss 0.08200 bagged 0.06319\n", 1765 | "2020-01-27 18:54:05,352 - Recursion-pytorch - INFO - Prep test sub...\n", 1766 | "2020-01-27 18:54:11,508 - Recursion-pytorch - INFO - EPOCH 9\n", 1767 | "2020-01-27 18:54:12,171 - Recursion-pytorch - INFO - Trn step 0 of 1955 trn lossavg 0.02686\n", 1768 | "2020-01-27 18:54:19,194 - Recursion-pytorch - INFO - Trn step 50 of 1955 trn lossavg 0.01849\n", 1769 | "2020-01-27 18:54:26,191 - Recursion-pytorch - INFO - Trn step 100 of 1955 trn lossavg 0.01930\n", 1770 | "2020-01-27 18:54:33,345 - Recursion-pytorch - INFO - Trn step 150 of 1955 trn lossavg 0.01956\n", 1771 | "2020-01-27 18:54:40,421 - Recursion-pytorch - INFO - Trn step 200 of 1955 trn lossavg 0.02027\n", 1772 | "2020-01-27 18:54:47,433 - Recursion-pytorch - INFO - Trn step 250 of 1955 trn lossavg 0.01990\n", 1773 | "2020-01-27 18:54:54,514 - Recursion-pytorch - INFO - Trn step 300 of 1955 trn lossavg 0.02014\n", 1774 | "2020-01-27 18:55:01,530 - Recursion-pytorch - INFO - Trn step 350 of 1955 trn lossavg 0.02088\n", 1775 | "2020-01-27 18:55:08,510 - Recursion-pytorch - INFO - Trn step 400 of 1955 trn lossavg 0.02095\n", 1776 | "2020-01-27 18:55:15,564 - Recursion-pytorch - INFO - Trn step 450 of 1955 trn lossavg 0.02082\n", 1777 | "2020-01-27 18:55:22,616 - Recursion-pytorch - INFO - Trn step 500 of 1955 trn lossavg 0.02072\n", 1778 | "2020-01-27 18:55:29,668 - Recursion-pytorch - INFO - Trn step 550 of 1955 trn lossavg 0.02090\n", 1779 | "2020-01-27 18:55:36,842 - Recursion-pytorch - INFO - Trn step 600 of 1955 trn lossavg 0.02123\n", 1780 | "2020-01-27 18:55:43,917 - Recursion-pytorch - INFO - Trn step 650 of 1955 trn lossavg 0.02165\n", 1781 | "2020-01-27 18:55:50,946 - Recursion-pytorch - INFO - Trn step 700 of 1955 trn lossavg 0.02174\n", 1782 | "2020-01-27 18:55:58,039 - Recursion-pytorch - INFO - Trn step 750 of 1955 trn lossavg 0.02192\n", 1783 | "2020-01-27 18:56:05,191 - Recursion-pytorch - INFO - Trn step 800 of 1955 trn lossavg 0.02211\n", 1784 | "2020-01-27 18:56:12,272 - Recursion-pytorch - INFO - Trn step 850 of 1955 trn lossavg 0.02198\n", 1785 | "2020-01-27 18:56:19,334 - Recursion-pytorch - INFO - Trn step 900 of 1955 trn lossavg 0.02214\n", 1786 | "2020-01-27 18:56:26,401 - Recursion-pytorch - INFO - Trn step 950 of 1955 trn lossavg 0.02194\n", 1787 | "2020-01-27 18:56:33,479 - Recursion-pytorch - INFO - Trn step 1000 of 1955 trn lossavg 0.02199\n", 1788 | "2020-01-27 18:56:40,511 - Recursion-pytorch - INFO - Trn step 1050 of 1955 trn lossavg 0.02201\n", 1789 | "2020-01-27 18:56:47,556 - Recursion-pytorch - INFO - Trn step 1100 of 1955 trn lossavg 0.02207\n", 1790 | "2020-01-27 18:56:54,576 - Recursion-pytorch - INFO - Trn step 1150 of 1955 trn lossavg 0.02197\n", 1791 | "2020-01-27 18:57:01,655 - Recursion-pytorch - INFO - Trn step 1200 of 1955 trn lossavg 0.02202\n", 1792 | "2020-01-27 18:57:08,746 - Recursion-pytorch - INFO - Trn step 1250 of 1955 trn lossavg 0.02202\n", 1793 | "2020-01-27 18:57:15,823 - Recursion-pytorch - INFO - Trn step 1300 of 1955 trn lossavg 0.02210\n", 1794 | "2020-01-27 18:57:22,919 - Recursion-pytorch - INFO - Trn step 1350 of 1955 trn lossavg 0.02203\n", 1795 | "2020-01-27 18:57:29,953 - Recursion-pytorch - INFO - Trn step 1400 of 1955 trn lossavg 0.02199\n", 1796 | "2020-01-27 18:57:36,936 - Recursion-pytorch - INFO - Trn step 1450 of 1955 trn lossavg 0.02204\n", 1797 | "2020-01-27 18:57:44,011 - Recursion-pytorch - INFO - Trn step 1500 of 1955 trn lossavg 0.02205\n", 1798 | "2020-01-27 18:57:50,963 - Recursion-pytorch - INFO - Trn step 1550 of 1955 trn lossavg 0.02208\n", 1799 | "2020-01-27 18:57:57,910 - Recursion-pytorch - INFO - Trn step 1600 of 1955 trn lossavg 0.02205\n", 1800 | "2020-01-27 18:58:05,019 - Recursion-pytorch - INFO - Trn step 1650 of 1955 trn lossavg 0.02212\n", 1801 | "2020-01-27 18:58:11,924 - Recursion-pytorch - INFO - Trn step 1700 of 1955 trn lossavg 0.02214\n", 1802 | "2020-01-27 18:58:18,964 - Recursion-pytorch - INFO - Trn step 1750 of 1955 trn lossavg 0.02220\n", 1803 | "2020-01-27 18:58:26,071 - Recursion-pytorch - INFO - Trn step 1800 of 1955 trn lossavg 0.02217\n", 1804 | "2020-01-27 18:58:33,096 - Recursion-pytorch - INFO - Trn step 1850 of 1955 trn lossavg 0.02231\n", 1805 | "2020-01-27 18:58:40,110 - Recursion-pytorch - INFO - Trn step 1900 of 1955 trn lossavg 0.02237\n", 1806 | "2020-01-27 18:58:47,015 - Recursion-pytorch - INFO - Trn step 1950 of 1955 trn lossavg 0.02249\n", 1807 | "2020-01-27 18:58:49,052 - Recursion-pytorch - INFO - Prep val score...\n", 1808 | "2020-01-27 18:58:58,833 - Recursion-pytorch - INFO - Epoch 9 val logloss 0.09069 bagged 0.06358\n", 1809 | "2020-01-27 18:58:58,927 - Recursion-pytorch - INFO - Prep test sub...\n" 1810 | ] 1811 | }, 1812 | { 1813 | "name": "stdout", 1814 | "output_type": "stream", 1815 | "text": [ 1816 | "DONE!\n" 1817 | ] 1818 | } 1819 | ], 1820 | "source": [ 1821 | "# Run Training\n", 1822 | "for epoch in range(EPOCHS):\n", 1823 | " logger.info(f'EPOCH {epoch}')\n", 1824 | " tr_loss = 0.\n", 1825 | " for param in model.parameters():\n", 1826 | " param.requires_grad = True\n", 1827 | " model.train() \n", 1828 | " for step, batch in enumerate(trnloader):\n", 1829 | " y = batch['labels'].to(device, dtype=torch.float)\n", 1830 | " mask = batch['mask'].to(device, dtype=torch.int)\n", 1831 | " \n", 1832 | " x = batch['emb'].to(device, dtype=torch.float)\n", 1833 | " x = torch.autograd.Variable(x, requires_grad=True)\n", 1834 | " \n", 1835 | " y = torch.autograd.Variable(y)\n", 1836 | " logits = model(x).to(device, dtype=torch.float)\n", 1837 | " \n", 1838 | " # get the mask for masked labels\n", 1839 | " maskidx = mask.view(-1)==1\n", 1840 | " y = y.view(-1, n_classes)[maskidx]\n", 1841 | " logits = logits.view(-1, n_classes)[maskidx]\n", 1842 | " # Get loss\n", 1843 | " loss = criterion(logits, y)\n", 1844 | " \n", 1845 | " tr_loss += loss.item()\n", 1846 | " optimizer.zero_grad()\n", 1847 | " \n", 1848 | " with amp.scale_loss(loss, optimizer) as scaled_loss:\n", 1849 | " with gpu_mem_restore_ctx():\n", 1850 | " scaled_loss.backward()\n", 1851 | " optimizer.step()\n", 1852 | " if step%50==0:\n", 1853 | " logger.info('Trn step {} of {} trn lossavg {:.5f}'. \\\n", 1854 | " format(step, len(trnloader), (tr_loss/(1+step))))\n", 1855 | " \n", 1856 | " del(batch)\n", 1857 | " gc.collect()\n", 1858 | " # Hitting memory errors, possibly with amp\n", 1859 | " torch.cuda.empty_cache()\n", 1860 | " \n", 1861 | " output_model_file = os.path.join(WORK_DIR, 'lstm_gepoch{}_lstmepoch{}_fold{}.bin'.format(GLOBALEPOCH, epoch, fold))\n", 1862 | " torch.save(model.state_dict(), output_model_file)\n", 1863 | "\n", 1864 | " scheduler.step()\n", 1865 | " \n", 1866 | " # Get Validation Loss\n", 1867 | " model.eval()\n", 1868 | " logger.info('Prep val score...')\n", 1869 | " ypred, imgval = predict(valloader)\n", 1870 | " ypredls.append(ypred)\n", 1871 | " \n", 1872 | " yvalpred = sum(ypredls[-nbags:])/len(ypredls[-nbags:])\n", 1873 | " yvalout = makeSub(yvalpred, imgval)\n", 1874 | " yvalp = makeSub(ypred, imgval)\n", 1875 | " \n", 1876 | " # get Val score\n", 1877 | " weights = ([1, 1, 1, 1, 1, 2] * ypred.shape[0])\n", 1878 | " yact = valloader.dataset.data[label_cols].values#.flatten()\n", 1879 | " yact = makeSub(yact, valloader.dataset.data['Image'].tolist())\n", 1880 | " yact = yact.set_index('ID').loc[yvalout.ID].reset_index()\n", 1881 | " valloss = log_loss(yact['Label'].values, yvalp['Label'].values.clip(.00001,.99999) , sample_weight = weights)\n", 1882 | " vallossavg = log_loss(yact['Label'].values, yvalout['Label'].values.clip(.00001,.99999) , sample_weight = weights)\n", 1883 | " logger.info('Epoch {} val logloss {:.5f} bagged {:.5f}'.format(epoch, valloss, vallossavg))\n", 1884 | "\n", 1885 | " del(ypred, yvalout, yvalp, imgval, yact)\n", 1886 | " gc.collect()\n", 1887 | " \n", 1888 | "print('DONE!')" 1889 | ] 1890 | }, 1891 | { 1892 | "cell_type": "markdown", 1893 | "metadata": {}, 1894 | "source": [ 1895 | "### Make Submission" 1896 | ] 1897 | }, 1898 | { 1899 | "cell_type": "code", 1900 | "execution_count": 63, 1901 | "metadata": {}, 1902 | "outputs": [ 1903 | { 1904 | "name": "stderr", 1905 | "output_type": "stream", 1906 | "text": [ 1907 | "2020-01-27 21:18:58,318 - Recursion-pytorch - INFO - Prep test sub...\n", 1908 | "2020-01-27 21:19:04,697 - Recursion-pytorch - INFO - Write out bagged prediction to preds folder\n" 1909 | ] 1910 | }, 1911 | { 1912 | "name": "stdout", 1913 | "output_type": "stream", 1914 | "text": [ 1915 | "DONE!\n" 1916 | ] 1917 | } 1918 | ], 1919 | "source": [ 1920 | "ypredtstls = []\n", 1921 | "model.eval()\n", 1922 | "logger.info('Prep test submission...')\n", 1923 | "ypred, imgtst = predict(tstloader)\n", 1924 | "ypredtstls.append(ypred)\n", 1925 | "\n", 1926 | "logger.info('Write out bagged prediction to preds folder')\n", 1927 | "ytstpred = sum(ypredtstls[-nbags:])/len(ypredtstls[-nbags:])\n", 1928 | "ytstout = makeSub(ytstpred, imgtst)\n", 1929 | "ytstout.ID = [x.split('_', 1)[1] for x in ytstout.ID.values]\n", 1930 | "\n", 1931 | "ytstout.to_csv('preds/mg_lstm_20200127.csv.gz', index = False, compression = 'gzip')\n", 1932 | "print('DONE!')" 1933 | ] 1934 | }, 1935 | { 1936 | "cell_type": "code", 1937 | "execution_count": 67, 1938 | "metadata": {}, 1939 | "outputs": [ 1940 | { 1941 | "data": { 1942 | "text/html": [ 1943 | "
\n", 1944 | "\n", 1957 | "\n", 1958 | " \n", 1959 | " \n", 1960 | " \n", 1961 | " \n", 1962 | " \n", 1963 | " \n", 1964 | " \n", 1965 | " \n", 1966 | " \n", 1967 | " \n", 1968 | " \n", 1969 | " \n", 1970 | " \n", 1971 | " \n", 1972 | " \n", 1973 | " \n", 1974 | " \n", 1975 | " \n", 1976 | " \n", 1977 | " \n", 1978 | " \n", 1979 | " \n", 1980 | " \n", 1981 | " \n", 1982 | " \n", 1983 | " \n", 1984 | " \n", 1985 | " \n", 1986 | " \n", 1987 | " \n", 1988 | " \n", 1989 | " \n", 1990 | " \n", 1991 | " \n", 1992 | " \n", 1993 | " \n", 1994 | " \n", 1995 | " \n", 1996 | " \n", 1997 | "
IDLabel
0ID_9b2d4a7b3_epidural0.0
1ID_9b2d4a7b3_intraparenchymal0.0
2ID_9b2d4a7b3_intraventricular0.0
3ID_9b2d4a7b3_subarachnoid0.0
4ID_9b2d4a7b3_subdural0.0
5ID_9b2d4a7b3_any0.0
\n", 1998 | "
" 1999 | ], 2000 | "text/plain": [ 2001 | " ID Label\n", 2002 | "0 ID_9b2d4a7b3_epidural 0.0\n", 2003 | "1 ID_9b2d4a7b3_intraparenchymal 0.0\n", 2004 | "2 ID_9b2d4a7b3_intraventricular 0.0\n", 2005 | "3 ID_9b2d4a7b3_subarachnoid 0.0\n", 2006 | "4 ID_9b2d4a7b3_subdural 0.0\n", 2007 | "5 ID_9b2d4a7b3_any 0.0" 2008 | ] 2009 | }, 2010 | "execution_count": 67, 2011 | "metadata": {}, 2012 | "output_type": "execute_result" 2013 | } 2014 | ], 2015 | "source": [ 2016 | "ytstout.head(6)" 2017 | ] 2018 | }, 2019 | { 2020 | "cell_type": "markdown", 2021 | "metadata": {}, 2022 | "source": [ 2023 | "### Create csv download link" 2024 | ] 2025 | }, 2026 | { 2027 | "cell_type": "code", 2028 | "execution_count": 66, 2029 | "metadata": {}, 2030 | "outputs": [ 2031 | { 2032 | "data": { 2033 | "text/html": [ 2034 | "preds/mg_lstm_20200127.csv.gz
" 2035 | ], 2036 | "text/plain": [ 2037 | "/home/morgan/ml/projects/rsna/rsna_2nd_place_solution/preds/mg_lstm_20200127.csv.gz" 2038 | ] 2039 | }, 2040 | "execution_count": 66, 2041 | "metadata": {}, 2042 | "output_type": "execute_result" 2043 | } 2044 | ], 2045 | "source": [ 2046 | "from IPython.display import FileLink, FileLinks\n", 2047 | "\n", 2048 | "FileLink('preds/mg_lstm_20200127.csv.gz')" 2049 | ] 2050 | }, 2051 | { 2052 | "cell_type": "markdown", 2053 | "metadata": {}, 2054 | "source": [ 2055 | "### Result = 1.29793" 2056 | ] 2057 | } 2058 | ], 2059 | "metadata": { 2060 | "kernelspec": { 2061 | "display_name": "Python 3", 2062 | "language": "python", 2063 | "name": "python3" 2064 | }, 2065 | "language_info": { 2066 | "codemirror_mode": { 2067 | "name": "ipython", 2068 | "version": 3 2069 | }, 2070 | "file_extension": ".py", 2071 | "mimetype": "text/x-python", 2072 | "name": "python", 2073 | "nbconvert_exporter": "python", 2074 | "pygments_lexer": "ipython3", 2075 | "version": "3.7.6" 2076 | } 2077 | }, 2078 | "nbformat": 4, 2079 | "nbformat_minor": 2 2080 | } 2081 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Blog Series - Kaggle RNSA Intracranial Haemorrage 2nd Place Solution Breakdown 2 | 3 | Notebooks to accompany my blog series about the 2nd place Kaggle RSNA winners to help me understand theire solution. First check out the 2nd place kaggle solution winners (Darragh and Dmitry) explanations and code here for yourself: 4 | 5 | - [kaggle discussion of their solution](https://www.kaggle.com/c/rsna-intracranial-hemorrhage-detection/discussion/117228) 6 | - [accompanying youtube explainer](https://www.youtube.com/watch?t=&v=U7WjTtGSOKA) 7 | - [code on github](https://github.com/darraghdog/rsna) 8 | 9 | 10 | After performing dismally in the Kaggle RSNA Intracranial Haemorrhage Competition thanks to a pig-headed strategy 11 | relying on brute force (via an expensive cloud instance) and too little thinking I resolved to see what the 12 | winners had done right. This series posts will cover what I learned looking at the code shared by the 2nd 13 | placed team, who’s solution I found both approachable and innovative, hope you enjoy. 14 | 15 | Post 1/4: [Intro](https://www.ntentional.com/technical/2020/1/18/kaggle) 16 | 17 | Post 2/4: [Data Preparation](https://www.ntentional.com/technical/2020/1/20/kaggle-rsna-2nd-place-data-prep) 18 | 19 | Post 3/4: [Image Classifier Training](https://ntentional.com/technical/2020/1/24/kaggle-rsna-2nd-place-train-img-classifier) 20 | 21 | Post 4/4: [LSTM Training](https://ntentional.com/technical/2020/1/26/rsna-2nd-place-lstm) 22 | 23 | --------------------------------------------------------------------------------