├── .gitignore ├── results_attn └── attn_summary.txt ├── results_qreg └── result_summary.txt ├── results_slopes └── result_summary.txt ├── data_download ├── kaggle.json └── dataset_download.py ├── perf_results ├── efnb0_fd_32_af_32_nal_3.txt ├── efnb1_fd_32_af_32_nal_3.txt ├── efnb2_fd_32_af_32_nal_3.txt ├── efnb3_fd_32_af_32_nal_3.txt ├── efnb4_fd_32_af_32_nal_3.txt ├── resnet101_fd_32_af_32_nal_3.txt ├── resnet152_fd_32_af_32_nal_3.txt ├── resnet18_fd_32_af_32_nal_3.txt ├── resnet34_fd_32_af_32_nal_3.txt ├── resnet50_fd_32_af_32_nal_3.txt ├── resnext50_fd_32_af_32_nal_3.txt └── resnext101_fd_32_af_32_nal_3.txt ├── __pycache__ ├── config.cpython-37.pyc └── train_attn_best_config.cpython-37.pyc ├── results_sm └── mlp_fd_32_af_32_nal_3.tar ├── data_train.command ├── LICENSE ├── README.md ├── requirements.txt ├── config.py ├── modal_clinical.py ├── performance.py ├── train_slopes.py ├── train_attn_b2.py ├── train_attn_best_config.py ├── modal_ct.py ├── train_qreg.py ├── train_data_ct_tab.csv └── results_attn_bc └── resnet18_fd_32_af_64_nal_1.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.tar 2 | -------------------------------------------------------------------------------- /results_attn/attn_summary.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /results_qreg/result_summary.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /results_slopes/result_summary.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /data_download/kaggle.json: -------------------------------------------------------------------------------- 1 | {"username":"your-user-name","key":"your-kaggle-key"} 2 | -------------------------------------------------------------------------------- /perf_results/efnb0_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 0.07 GMac params : 4.05 M 2 | infer: 0.6712081800000007 3 | -------------------------------------------------------------------------------- /perf_results/efnb1_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 0.1 GMac params : 6.56 M 2 | infer: 0.7008031043000003 3 | -------------------------------------------------------------------------------- /perf_results/efnb2_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 0.1 GMac params : 7.75 M 2 | infer: 0.7272119500000003 3 | -------------------------------------------------------------------------------- /perf_results/efnb3_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 0.14 GMac params : 10.75 M 2 | infer: 0.7022428334999986 3 | -------------------------------------------------------------------------------- /perf_results/efnb4_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 0.18 GMac params : 17.61 M 2 | infer: 0.7055882494999992 3 | -------------------------------------------------------------------------------- /perf_results/resnet101_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 40.61 GMac params : 42.56 M 2 | infer: 0.6923097215000006 3 | -------------------------------------------------------------------------------- /perf_results/resnet152_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 60.1 GMac params : 58.21 M 2 | infer: 0.7032769890999994 3 | -------------------------------------------------------------------------------- /perf_results/resnet18_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 9.11 GMac params : 11.19 M 2 | infer: 0.6700435730000001 3 | -------------------------------------------------------------------------------- /perf_results/resnet34_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 18.79 GMac params : 21.3 M 2 | infer: 0.6259505765999996 3 | -------------------------------------------------------------------------------- /perf_results/resnet50_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 21.13 GMac params : 23.57 M 2 | infer: 0.6927700813999997 3 | -------------------------------------------------------------------------------- /perf_results/resnext50_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 21.92 GMac params : 23.04 M 2 | infer: 0.6820671836000003 3 | -------------------------------------------------------------------------------- /perf_results/resnext101_fd_32_af_32_nal_3.txt: -------------------------------------------------------------------------------- 1 | macs : 85.84 GMac params : 86.81 M 2 | infer: 0.6952471045999999 3 | -------------------------------------------------------------------------------- /__pycache__/config.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zabir-nabil/Fibro-CoSANet/HEAD/__pycache__/config.cpython-37.pyc -------------------------------------------------------------------------------- /results_sm/mlp_fd_32_af_32_nal_3.tar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zabir-nabil/Fibro-CoSANet/HEAD/results_sm/mlp_fd_32_af_32_nal_3.tar -------------------------------------------------------------------------------- /__pycache__/train_attn_best_config.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zabir-nabil/Fibro-CoSANet/HEAD/__pycache__/train_attn_best_config.cpython-37.pyc -------------------------------------------------------------------------------- /data_train.command: -------------------------------------------------------------------------------- 1 | cd data_download; python dataset_download.py; mv osic-pulmonary-fibrosis-progression.zip ../../; unzip ../../osic-pulmonary-fibrosis-progression.zip; cd ../; python train_slopes.py -------------------------------------------------------------------------------- /data_download/dataset_download.py: -------------------------------------------------------------------------------- 1 | from kaggle.api.kaggle_api_extended import KaggleApi 2 | from shutil import copyfile 3 | 4 | # move kaggle.json file 5 | # copyfile("kaggle.json", "/root/.kaggle") 6 | 7 | api = KaggleApi() 8 | api.authenticate() 9 | 10 | print('authentication done') 11 | 12 | # api.competition_download_files('osic-pulmonary-fibrosis-progression') 13 | 14 | import os 15 | os.system("kaggle competitions download -c osic-pulmonary-fibrosis-progression") 16 | 17 | print('done') 18 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Zabir Al Nazi Nabil 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Fibro-CoSANet: Pulmonary Fibrosis Prognosis Prediction using a Convolutional Self Attention Network 2 | 3 | ### Installation 4 | 5 | 1. `git clone https://github.com/zabir-nabil/osic-pulmonary-fibrosis-progression.git` 6 | 2. `cd osic-pulmonary-fibrosis-progression` 7 | 3. Install Anaconda [Anaconda](https://www.anaconda.com/products/individual) 8 | 4. `conda create -n pulmo python==3.7.5` 9 | 5. `conda activate pulmo` 10 | 6. `conda install -c intel mkl_fft` (opt.) 11 | 7. `conda install -c intel mkl_random` (opt.) 12 | 8. `conda install -c anaconda mkl-service` (opt.) 13 | 9. `pip install -r requirements.txt` 14 | 15 | ### Download Dataset 16 | 17 | 1. Download the kaggle.json from Kaggle account. [Kaggle authentication](https://www.kaggle.com/docs/api) 18 | 2. Keep the kaggle.json file inside data_download folder. 19 | 3. `sudo mkdir /root/.kaggle` 20 | 4. `sudo cp kaggle.json /root/.kaggle/` 21 | 5. `sudo apt install unzip` if not installed already 22 | 23 | * `cd data_download; python dataset_download.py; mv osic-pulmonary-fibrosis-progression.zip ../../; unzip ../../osic-pulmonary-fibrosis-progression.zip -d ../../; cd ../; python train_slopes.py` 24 | 25 | ### Training 26 | 27 | 1. Set the training hyperparameters in `config.py` 28 | 2. Slope Prediction 29 | * To train **slopes model** run `python train_slopes.py` 30 | * trained model weights and results will be saved inside `hyp.results_dir` 31 | 3. Quantile Regression 32 | * To train **qreg model** run `python train_qreg.py` 33 | * trained model weights and results will be saved inside `hyp.results_dir` 34 | 35 | * Volume calculation: https://www.kaggle.com/furcifer/q-regression-with-ct-tabular-features-pytorch 36 | 37 | 38 | ### Arxiv pre-print 39 | 40 | https://arxiv.org/abs/2104.05889 41 | 42 | 43 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | absl-py==0.9.0 2 | argon2-cffi==20.1.0 3 | astunparse==1.6.3 4 | attrs==20.1.0 5 | backcall==0.2.0 6 | beautifulsoup4==4.6.3 7 | bleach==3.1.5 8 | blinker==1.4 9 | brotlipy==0.7.0 10 | cachetools # @ file:///tmp/build/80754af9/cachetools_1596822027882/work 11 | certifi==2020.6.20 12 | cffi==1.14.2 13 | chardet==3.0.4 14 | click==7.1.2 15 | cliff==2.8.3 16 | cmd2==1.3.8 17 | colorama==0.4.3 18 | cryptography # @ file:///tmp/build/80754af9/cryptography_1598892037289/work 19 | cycler==0.10.0 20 | decorator==4.4.2 21 | defusedxml==0.6.0 22 | efficientnet # @ file:///home/ml2/nabil/kag_pul/efficientnet-1.1.0 23 | efficientnet-pytorch==0.7.0 24 | entrypoints==0.3 25 | future==0.18.2 26 | gast==0.3.3 27 | google-auth # @ file:///tmp/build/80754af9/google-auth_1598987460909/work 28 | google-auth-oauthlib==0.4.1 29 | google-pasta==0.2.0 30 | grpcio # @ file:///tmp/build/80754af9/grpcio_1597424467487/work 31 | h5py # @ file:///tmp/build/80754af9/h5py_1593454119955/work 32 | idna # @ file:///tmp/build/80754af9/idna_1593446292537/work 33 | imageio==2.9.0 34 | importlib-metadata # @ file:///tmp/build/80754af9/importlib-metadata_1593446408836/work 35 | ipykernel==5.3.4 36 | ipython==7.17.0 37 | ipython-genutils==0.2.0 38 | ipywidgets==7.5.1 39 | jedi==0.17.2 40 | Jinja2==2.11.2 41 | joblib==0.16.0 42 | jsonschema==3.2.0 43 | jupyter==1.0.0 44 | jupyter-client==6.1.7 45 | jupyter-console==6.1.0 46 | jupyter-core==4.6.3 47 | kaggle==1.5.6 48 | Keras-Applications # @ file:///home/ml2/nabil/kag_pul/keras-team-keras-applications-3b180cb 49 | Keras-Preprocessing==1.1.0 50 | kiwisolver==1.2.0 51 | Markdown # @ file:///tmp/build/80754af9/markdown_1597433218422/work 52 | MarkupSafe==1.1.1 53 | matplotlib==3.3.1 54 | MechanicalSoup==0.8.0 55 | mistune==0.8.4 56 | # mkl-fft # conda install -c intel mkl_fft 57 | # mkl-random 58 | # mkl-service 59 | nbconvert==5.6.1 60 | nbformat==5.0.7 61 | networkx==2.5 62 | notebook==6.1.3 63 | numpy # @ file:///tmp/build/80754af9/numpy_and_numpy_base_1596233707986/work 64 | oauthlib==3.1.0 65 | opencv-python==4.4.0.42 66 | opt-einsum==3.1.0 67 | packaging==20.4 68 | pandas==1.1.1 69 | pandocfilters==1.4.2 70 | parso==0.7.1 71 | pbr==5.4.5 72 | pexpect==4.8.0 73 | pickleshare==0.7.5 74 | Pillow==7.2.0 75 | plotly==4.9.0 76 | prettytable==0.7.2 77 | prometheus-client==0.8.0 78 | prompt-toolkit==3.0.6 79 | protobuf==3.13.0 80 | ptyprocess==0.6.0 81 | pyasn1==0.4.8 82 | pyasn1-modules==0.2.7 83 | pycparser # @ file:///tmp/build/80754af9/pycparser_1594388511720/work 84 | pydicom==2.0.0 85 | Pygments==2.6.1 86 | PyJWT==1.7.1 87 | pyOpenSSL # @ file:///tmp/build/80754af9/pyopenssl_1594392929924/work 88 | pyparsing==2.4.7 89 | pyperclip==1.8.0 90 | pyrsistent==0.16.0 91 | PySocks # @ file:///tmp/build/80754af9/pysocks_1594394576006/work 92 | python-dateutil==2.8.1 93 | python-slugify==4.0.1 94 | pytorch-tabnet==1.2.0 95 | pytz==2020.1 96 | PyWavelets==1.1.1 97 | PyYAML==5.3.1 98 | pyzmq==19.0.2 99 | qtconsole==4.7.6 100 | QtPy==1.9.0 101 | requests # @ file:///tmp/build/80754af9/requests_1592841827918/work 102 | requests-oauthlib==1.3.0 103 | retrying==1.3.3 104 | rsa # @ file:///tmp/build/80754af9/rsa_1596998415516/work 105 | scikit-image==0.17.2 106 | scikit-learn==0.23.2 107 | scipy # @ file:///tmp/build/80754af9/scipy_1597686620742/work 108 | seaborn==0.10.1 109 | Send2Trash==1.5.0 110 | six==1.15.0 111 | slugify==0.0.1 112 | stevedore==3.2.0 113 | termcolor==1.1.0 114 | terminado==0.8.3 115 | testpath==0.4.4 116 | text-unidecode==1.3 117 | threadpoolctl==2.1.0 118 | tifffile==2020.8.25 119 | torch==1.6.0 120 | torchvision==0.7.0 121 | tornado==6.0.4 122 | tqdm==4.48.2 123 | traitlets==4.3.3 124 | typeguard==2.9.1 125 | urllib3 # @ file:///tmp/build/80754af9/urllib3_1597086586889/work 126 | wcwidth==0.2.5 127 | webencodings==0.5.1 128 | Werkzeug==1.0.1 129 | widgetsnbextension==3.5.1 130 | wrapt==1.12.1 131 | zipp==3.1.0 132 | # kaggle 133 | ptflops 134 | thop 135 | -------------------------------------------------------------------------------- /config.py: -------------------------------------------------------------------------------- 1 | # author: github/zabir-nabil 2 | 3 | class HyperP: 4 | def __init__(self, model_type): 5 | # hyperparameters 6 | if model_type == "slope_train": 7 | self.seed = 1997 8 | self.data_folder = '..' # one level up 9 | self.ct_tab_feature_csv = 'train_data_ct_tab.csv' # some extra features 10 | self.strip_ct = .15 # strip this amount of ct slices before randomly choosing 11 | self.n_tab = 5 # number of tabular features used 12 | 13 | self.cnn_dim = 32 # compressed cnn feature dim 14 | 15 | self.fc_dim = 16 16 | 17 | # select which models to train 18 | self.train_models = ['resnet18', 'resnet34', 'resnet50', 'resnet101', 'resnet152', 'resnext50', 'resnext101', 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4', 'efnb5', 'efnb6', 'efnb7'] 19 | 20 | self.gpu_index = 0 21 | self.num_workers = 0 # 0 for bug fix/docker 22 | self.results_dir = "results_slopes" 23 | self.nfold = 5 24 | self.n_epochs = 40 25 | self.batch_size = 16 26 | self.final_lr = 0.0002 27 | elif model_type == "slope_test": 28 | pass 29 | elif model_type == "qreg_train": 30 | self.seed = 1997 31 | self.data_folder = '..' 32 | self.ct_tab_feature_csv = 'train_data_ct_tab.csv' # some extra features 33 | self.strip_ct = .15 # strip this amount of ct slices before randomly choosing 34 | self.n_tab = 7 # number of tabular features used 35 | 36 | # select which models to train 37 | self.train_models = ['resnet18' , 'resnet34', 'resnet50', 'resnet101', 'resnet152', 'resnext50', 'resnext101', 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4', 'efnb5', 'efnb6', 'efnb7'] 38 | 39 | self.gpu_index = 0 40 | self.results_dir = "results_qreg" 41 | self.nfold = 5 42 | self.n_epochs = 40 43 | self.batch_size = 8 44 | self.final_lr = 0.0002 45 | self.loss_weight = 0.7 46 | self.dummy_training = False 47 | self.dummy_train_rows = 400 48 | elif model_type == "attn_train": 49 | # ablation study 50 | self.seed = 1997 51 | self.data_folder = '..' # .. one level up 52 | self.ct_tab_feature_csv = 'train_data_ct_tab.csv' # some extra features 53 | self.strip_ct = .15 # strip this amount of ct slices before randomly choosing 54 | self.n_tab = 5 # number of tabular features used 55 | 56 | # self.cnn_dim = 32 # compressed cnn feature dim 57 | 58 | self.fc_dim = [16, 32] 59 | 60 | # select which models to train 61 | self.train_models = ['efnb2_attn'] 62 | 63 | self.gpu_index = 0 64 | self.num_workers = 0 # 0 for bug fix/docker 65 | self.results_dir = "results_attn" 66 | self.nfold = 5 67 | self.n_epochs = 40 68 | self.batch_size = 10 69 | self.final_lr = 0.0002 70 | 71 | self.attn_filters = [32, 64, 128] # attn_filters and cnn_dim should be same 72 | 73 | self.n_attn_layers = [1, 2, 3] 74 | 75 | elif model_type == "attn_train_best_config": 76 | # ablation study 77 | self.seed = 1997 78 | self.data_folder = '..' # .. one level up 79 | self.ct_tab_feature_csv = 'train_data_ct_tab.csv' # some extra features 80 | self.strip_ct = .15 # strip this amount of ct slices before randomly choosing 81 | self.n_tab = 5 # number of tabular features used 82 | 83 | # self.cnn_dim = 32 # compressed cnn feature dim 84 | 85 | self.fc_dim = [32] 86 | 87 | # select which models to train 88 | self.train_models = [ 'resnet18' , 'resnet34', 'resnet50', 'resnet101', 'resnet152', 'resnext50', 'resnext101', 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4' ] 89 | # train 1 : 'resnet18' , 'resnet34', 'resnet50', 'resnet101', 'resnet152', 'resnext50' 90 | # train 2 : 'resnext101', 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4' 91 | 92 | self.gpu_index = 0 93 | self.num_workers = 0 # 0 for bug fix/docker 94 | self.results_dir = "results_attn_bc" 95 | self.nfold = 5 96 | self.n_epochs = 40 97 | self.batch_size = 10 98 | self.final_lr = 0.0002 99 | 100 | self.attn_filters = [32] # [32, 128] # attn_filters and cnn_dim should be same 101 | 102 | self.n_attn_layers = [3] 103 | elif model_type == "singlemodal_ct": 104 | # ablation study 105 | self.seed = 1997 106 | self.data_folder = '..' # .. one level up 107 | self.ct_tab_feature_csv = 'train_data_ct_tab.csv' # some extra features 108 | self.strip_ct = .15 # strip this amount of ct slices before randomly choosing 109 | self.n_tab = 5 # number of tabular features used 110 | 111 | # self.cnn_dim = 32 # compressed cnn feature dim 112 | 113 | self.fc_dim = [32] 114 | 115 | # select which models to train 116 | self.train_models = [ 'efnb0', 'efnb1', 'efnb2' ] 117 | # train 1 : 'resnet18' , 'resnet34', 'resnet50', 'resnet101', 'resnet152', 'resnext50' 118 | # train 2 : 'resnext101', 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4' 119 | 120 | self.gpu_index = 0 121 | self.num_workers = 0 # 0 for bug fix/docker 122 | self.results_dir = "results_sm" 123 | self.nfold = 5 124 | self.n_epochs = 40 125 | self.batch_size = 10 126 | self.final_lr = 0.0002 127 | 128 | self.attn_filters = [32] # [32, 128] # attn_filters and cnn_dim should be same 129 | 130 | self.n_attn_layers = [3] 131 | elif model_type == "singlemodal_clinical": 132 | # ablation study 133 | self.seed = 1997 134 | self.data_folder = '..' # .. one level up 135 | self.ct_tab_feature_csv = 'train_data_ct_tab.csv' # some extra features 136 | self.strip_ct = .15 # strip this amount of ct slices before randomly choosing 137 | self.n_tab = 5 # number of tabular features used 138 | 139 | # self.cnn_dim = 32 # compressed cnn feature dim 140 | 141 | self.fc_dim = [32] 142 | 143 | # select which models to train 144 | self.train_models = [ 'mlp' ] 145 | # train 1 : 'resnet18' , 'resnet34', 'resnet50', 'resnet101', 'resnet152', 'resnext50' 146 | # train 2 : 'resnext101', 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4' 147 | 148 | self.gpu_index = 0 149 | self.num_workers = 0 # 0 for bug fix/docker 150 | self.results_dir = "results_sm" 151 | self.nfold = 5 152 | self.n_epochs = 40 153 | self.batch_size = 10 154 | self.final_lr = 0.0002 155 | 156 | self.attn_filters = [32] # [32, 128] # attn_filters and cnn_dim should be same 157 | 158 | self.n_attn_layers = [3] -------------------------------------------------------------------------------- /modal_clinical.py: -------------------------------------------------------------------------------- 1 | 2 | # author: github/zabir-nabil 3 | 4 | # relevant imports 5 | 6 | import os 7 | import cv2 8 | 9 | import pydicom 10 | import pandas as pd 11 | import numpy as np 12 | # import tensorflow as tf 13 | # import matplotlib.pyplot as plt 14 | 15 | # torch dataset 16 | import torch 17 | from torch.utils.data import Dataset, DataLoader 18 | from torchvision import transforms, utils 19 | 20 | import random 21 | from tqdm import tqdm 22 | 23 | # k-fold 24 | from sklearn.model_selection import KFold 25 | 26 | # hyperparam object 27 | 28 | from config import HyperP 29 | 30 | hyp = HyperP(model_type = "singlemodal_clinical") # slope prediction 31 | 32 | # seed 33 | seed = hyp.seed 34 | random.seed(seed) 35 | os.environ['PYTHONHASHSEED'] = str(seed) 36 | np.random.seed(seed) 37 | # tf.random.set_seed(seed) 38 | torch.manual_seed(seed) 39 | torch.backends.cudnn.deterministic = True 40 | torch.backends.cudnn.benchmark = False 41 | 42 | 43 | # path 44 | 45 | root_path = hyp.data_folder # ../input/osic-pulmonary-fibrosis-progression 46 | 47 | train = pd.read_csv(f'{root_path}/train.csv') 48 | train_vol = pd.read_csv(f'{hyp.ct_tab_feature_csv}') 49 | 50 | 51 | train['Volume'] = 2000. 52 | 53 | for i in range(len(train)): 54 | pid = train.iloc[i]['Patient'] 55 | try: 56 | train.at[i, 'Volume'] = train_vol[train_vol['Patient']==pid].iloc[0]['Volume'] 57 | except: 58 | print('bug at volume') 59 | 60 | 61 | # tabular feature generation 62 | 63 | def get_tab(df): 64 | vector = [(df.Age.values[0] - train.Age.values.mean()) / train.Age.values.std()] # df.Age.values[0].mean(), df.Age.values[0].std() 65 | 66 | if df.Sex.values[0] == 'Male': 67 | vector.append(0) 68 | else: 69 | vector.append(1) 70 | 71 | if df.SmokingStatus.values[0] == 'Never smoked': 72 | vector.extend([0,0]) 73 | elif df.SmokingStatus.values[0] == 'Ex-smoker': 74 | vector.extend([1,1]) 75 | elif df.SmokingStatus.values[0] == 'Currently smokes': 76 | vector.extend([0,1]) 77 | else: 78 | vector.extend([1,0]) # this is useless 79 | 80 | vector.append((df.Volume.values[0] - train.Volume.values.mean()) / train.Volume.values.std()) 81 | return np.array(vector) 82 | 83 | 84 | A = {} # the slopes 85 | TAB = {} # tabular features 86 | P = [] # patient IDs 87 | 88 | for i, p in tqdm(enumerate(train.Patient.unique())): 89 | sub = train.loc[train.Patient == p, :] 90 | fvc = sub.FVC.values 91 | weeks = sub.Weeks.values 92 | c = np.vstack([weeks, np.ones(len(weeks))]).T 93 | 94 | a, _ = np.linalg.lstsq(c, fvc)[0] # we calculate the best slope with least square 95 | 96 | # ref: https://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html 97 | 98 | A[p] = a 99 | TAB[p] = get_tab(sub) 100 | P.append(p) 101 | 102 | 103 | 104 | 105 | 106 | class OSICData(Dataset): 107 | BAD_ID = ['ID00011637202177653955184', 'ID00052637202186188008618'] 108 | def __init__(self, keys, a, tab): 109 | self.keys = [k for k in keys if k not in self.BAD_ID] 110 | self.a = a 111 | self.tab = tab 112 | 113 | self.train_data = {} 114 | for p in train.Patient.values: 115 | p_n = len(os.listdir(f'{root_path}/train/{p}/')) 116 | self.train_data[p] = os.listdir(f'{root_path}/train/{p}/')[int( hyp.strip_ct * p_n):-int( hyp.strip_ct * p_n)] # removing first and last 15% slices 117 | 118 | 119 | def __len__(self): 120 | return len(self.keys) 121 | 122 | def get_img(self, path): 123 | d = pydicom.dcmread(path) 124 | return cv2.resize(d.pixel_array / 2**11, (512, 512)) # maybe bug in resize 125 | 126 | def __getitem__(self, idx): 127 | x = [] 128 | a, tab = [], [] 129 | k = self.keys[idx] # instead of random id send a specific id 130 | # np.random.choice(self.keys, 1)[0] 131 | 132 | try: 133 | i = np.random.choice(self.train_data[k], size=1)[0] 134 | # print(i) 135 | cp = f'{root_path}/train/{k}/{i}' 136 | #print(cp) 137 | img = self.get_img(cp) 138 | 139 | x.append(img) 140 | a.append(self.a[k]) 141 | tab.append(self.tab[k]) 142 | except: 143 | print('failed') 144 | print(k, i) 145 | 146 | x, a, tab = torch.tensor(x, dtype=torch.float32), torch.tensor(a, dtype=torch.float32), torch.tensor(tab, dtype=torch.float32) 147 | tab = torch.squeeze(tab, axis=0) 148 | return [x, tab] , a, k # k for patient id 149 | 150 | 151 | from torch import nn 152 | 153 | # only clinical 154 | class TabCT(nn.Module): 155 | def __init__(self, cnn, attn_filters, fc_dim, n_attn_layers): 156 | super(TabCT, self).__init__() 157 | 158 | self.n_tab = hyp.n_tab # n tabular features 159 | self.attn_filters = attn_filters 160 | self.fc_dim = fc_dim 161 | self.n_attn_layers = n_attn_layers 162 | 163 | 164 | self.fc_inter = nn.Linear(self.n_tab, self.fc_dim) 165 | 166 | self.fc = nn.Linear(self.fc_dim, 1) 167 | 168 | def forward(self, x_ct, x_tab): 169 | 170 | x = self.fc_inter(x_tab) 171 | 172 | x = self.fc(x) 173 | 174 | return x 175 | 176 | from sklearn.model_selection import train_test_split 177 | from sklearn.metrics import mean_squared_error 178 | 179 | 180 | # score calculation 181 | 182 | def score(fvc_true, fvc_pred, sigma): 183 | sigma_clip = np.maximum(sigma, 70) 184 | delta = np.abs(fvc_true - fvc_pred) 185 | delta = np.minimum(delta, 1000) 186 | sq2 = np.sqrt(2) 187 | metric = (delta / sigma_clip)*sq2 + np.log(sigma_clip* sq2) 188 | return np.mean(metric) 189 | 190 | 191 | def score_avg(p, a): # patient id, predicted a 192 | percent_true = train.Percent.values[train.Patient == p] 193 | fvc_true = train.FVC.values[train.Patient == p] 194 | weeks_true = train.Weeks.values[train.Patient == p] 195 | 196 | fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 197 | percent = percent_true[0] - a * abs(weeks_true - weeks_true[0]) 198 | return score(fvc_true, fvc, percent) 199 | 200 | def rmse_avg(p, a): # patient id, predicted a 201 | percent_true = train.Percent.values[train.Patient == p] 202 | fvc_true = train.FVC.values[train.Patient == p] 203 | weeks_true = train.Weeks.values[train.Patient == p] 204 | 205 | fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 206 | return mean_squared_error(fvc_true, fvc, squared = False) 207 | 208 | 209 | # hyperparams 210 | 211 | result_dir = hyp.results_dir 212 | 213 | # training only resnet models on gpu 0 214 | train_models = hyp.train_models 215 | 216 | # 'resnext101' -> seems too heavy for 1080 217 | # 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4', 'efnb5', 'efnb6', 'efnb7' 218 | 219 | # device 220 | gpu = torch.device(f"cuda:{hyp.gpu_index}" if torch.cuda.is_available() else "cpu") 221 | 222 | 223 | nfold = hyp.nfold # hyper 224 | 225 | # removing noisy data 226 | P = [p for p in P if p not in ['ID00011637202177653955184', 'ID00052637202186188008618']] 227 | 228 | for model in train_models: 229 | for fd in hyp.fc_dim: 230 | for af in hyp.attn_filters: 231 | for nal in hyp.n_attn_layers: 232 | log = open(f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.txt", "a+") 233 | kfold =KFold(n_splits=nfold) 234 | 235 | ifold = 0 236 | for train_index, test_index in kfold.split(P): 237 | # print(train_index, test_index) 238 | 239 | p_train = np.array(P)[train_index] 240 | p_test = np.array(P)[test_index] 241 | 242 | osic_train = OSICData(p_train, A, TAB) 243 | train_loader = DataLoader(osic_train, batch_size=hyp.batch_size, shuffle=True, num_workers=hyp.num_workers) 244 | 245 | osic_val = OSICData(p_test, A, TAB) 246 | val_loader = DataLoader(osic_val, batch_size=hyp.batch_size, shuffle=True, num_workers=hyp.num_workers) 247 | 248 | 249 | tabct = TabCT(cnn = model, fc_dim = fd, attn_filters = af, n_attn_layers = nal).to(gpu) 250 | print(f"creating {model} with {fd} feature_dim, {af} attn_filters, and {nal} n_attn_layers") 251 | print(f"fold: {ifold}") 252 | log.write(f"fold: {ifold}\n") 253 | 254 | ifold += 1 255 | 256 | 257 | n_epochs = hyp.n_epochs # max 30 epochs, patience 5, find the suitable epoch number for later final training 258 | 259 | best_epoch = n_epochs # 30 260 | 261 | 262 | optimizer = torch.optim.AdamW(tabct.parameters()) 263 | criterion = torch.nn.L1Loss() 264 | 265 | max_score = 99999999.0000 # here, max score ]= minimum score 266 | 267 | for epoch in range(n_epochs): # loop over the dataset multiple times 268 | 269 | running_loss = 0.0 270 | tabct.train() 271 | for i, data in tqdm(enumerate(train_loader, 0)): 272 | 273 | [x, t], a, _ = data 274 | print(x.shape) 275 | print(t.shape) 276 | 277 | x = x.to(gpu) 278 | t = t.to(gpu) 279 | a = a.to(gpu) 280 | 281 | # zero the parameter gradients 282 | optimizer.zero_grad() 283 | 284 | # forward + backward + optimize 285 | outputs = tabct(x, t) 286 | loss = criterion(outputs, a) 287 | loss.backward() 288 | optimizer.step() 289 | 290 | # print statistics 291 | running_loss += loss.item() 292 | print(f"epoch {epoch+1} train: {running_loss}") 293 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 294 | 295 | running_loss = 0.0 296 | pred_a = {} 297 | tabct.eval() 298 | for i, data in tqdm(enumerate(val_loader, 0)): 299 | 300 | [x, t], a, pid = data 301 | 302 | x = x.to(gpu) 303 | t = t.to(gpu) 304 | a = a.to(gpu) 305 | 306 | # forward 307 | outputs = tabct(x, t) 308 | loss = criterion(outputs, a) 309 | 310 | pids = pid 311 | preds_a = outputs.detach().cpu().numpy().flatten() 312 | 313 | for j, p_d in enumerate(pids): 314 | pred_a[p_d] = preds_a[j] 315 | 316 | 317 | 318 | 319 | # print statistics 320 | running_loss += loss.item() 321 | print(f"epoch {epoch+1} val: {running_loss}") 322 | log.write(f"epoch {epoch+1} val: {running_loss}\n") 323 | # score calculation 324 | print(pred_a) 325 | print(len(pred_a)) 326 | print(p_test) 327 | print(len(p_test)) 328 | score_v = 0. 329 | rmse = 0. 330 | for p in p_test: 331 | score_v += (score_avg(p, pred_a[p]))/len(p_test) 332 | rmse += (rmse_avg(p, pred_a[p]))/len(p_test) 333 | 334 | print(f"val score: {score_v}") 335 | log.write(f"val score: {score_v}\n") 336 | log.write(f"val rmse: {rmse}\n") 337 | 338 | if score_v <= max_score: 339 | torch.save({ 340 | 'epoch': epoch, 341 | 'model_state_dict': tabct.state_dict(), 342 | 'optimizer_state_dict': optimizer.state_dict(), 343 | 'score': score_v 344 | }, f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.tar") 345 | max_score = score_v 346 | best_epoch = epoch + 1 347 | # destroy model 348 | del tabct 349 | torch.cuda.empty_cache() 350 | 351 | # final training with optimized setting 352 | 353 | osic_all = OSICData(P, A, TAB) 354 | all_loader = DataLoader(osic_all, batch_size=8, shuffle=True, num_workers=hyp.num_workers) 355 | 356 | # load the best model 357 | tabct = TabCT(cnn = model, fc_dim = fd, attn_filters = af, n_attn_layers = nal).to(gpu) 358 | tabct.load_state_dict(torch.load(f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.tar")["model_state_dict"]) 359 | 360 | optimizer = torch.optim.AdamW(tabct.parameters(), lr = hyp.final_lr) # very small learning rate 361 | criterion = torch.nn.L1Loss() 362 | 363 | print(f"Final training") 364 | log.write(f"Final training\n") 365 | for epoch in range(best_epoch + 2): # loop over the dataset multiple times 366 | 367 | running_loss = 0.0 368 | tabct.train() 369 | for i, data in tqdm(enumerate(all_loader, 0)): 370 | 371 | [x, t], a, _ = data 372 | 373 | x = x.to(gpu) 374 | t = t.to(gpu) 375 | a = a.to(gpu) 376 | 377 | # zero the parameter gradients 378 | optimizer.zero_grad() 379 | 380 | # forward + backward + optimize 381 | outputs = tabct(x, t) 382 | loss = criterion(outputs, a) 383 | loss.backward() 384 | optimizer.step() 385 | 386 | # print statistics 387 | running_loss += loss.item() 388 | print(f"epoch {epoch+1} train: {running_loss}") 389 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 390 | 391 | torch.save({ 392 | 'epoch': best_epoch, 393 | 'model_state_dict': tabct.state_dict(), 394 | 'optimizer_state_dict': optimizer.state_dict() 395 | }, f"{result_dir}/{model}.tar") 396 | 397 | print('Finished Training') 398 | # destroy model 399 | del tabct 400 | torch.cuda.empty_cache() 401 | 402 | 403 | 404 | 405 | # ref: https://www.kaggle.com/miklgr500/linear-decay-based-on-resnet-cnn 406 | # https://pytorch.org/docs/stable/index.html -------------------------------------------------------------------------------- /performance.py: -------------------------------------------------------------------------------- 1 | # author: github/zabir-nabil 2 | 3 | # relevant imports 4 | 5 | import os 6 | import cv2 7 | 8 | import pydicom 9 | import pandas as pd 10 | import numpy as np 11 | # import tensorflow as tf 12 | # import matplotlib.pyplot as plt 13 | 14 | # torch dataset 15 | import torch 16 | from torch.utils.data import Dataset, DataLoader 17 | from torchvision import transforms, utils 18 | 19 | import random 20 | from tqdm import tqdm 21 | 22 | # k-fold 23 | from sklearn.model_selection import KFold 24 | 25 | # hyperparam object 26 | 27 | from config import HyperP 28 | from time import process_time 29 | from ptflops import get_model_complexity_info 30 | 31 | hyp = HyperP(model_type = "attn_train_best_config") # slope prediction 32 | 33 | # seed 34 | seed = hyp.seed 35 | random.seed(seed) 36 | os.environ['PYTHONHASHSEED'] = str(seed) 37 | np.random.seed(seed) 38 | # tf.random.set_seed(seed) 39 | torch.manual_seed(seed) 40 | torch.backends.cudnn.deterministic = True 41 | torch.backends.cudnn.benchmark = False 42 | 43 | 44 | # path 45 | 46 | root_path = hyp.data_folder # ../input/osic-pulmonary-fibrosis-progression 47 | 48 | train = pd.read_csv(f'{root_path}/train.csv') 49 | train_vol = pd.read_csv(f'{hyp.ct_tab_feature_csv}') 50 | 51 | 52 | train['Volume'] = 2000. 53 | 54 | for i in range(len(train)): 55 | pid = train.iloc[i]['Patient'] 56 | try: 57 | train.at[i, 'Volume'] = train_vol[train_vol['Patient']==pid].iloc[0]['Volume'] 58 | except: 59 | print('bug at volume') 60 | 61 | 62 | # tabular feature generation 63 | 64 | def get_tab(df): 65 | vector = [(df.Age.values[0] - train.Age.values.mean()) / train.Age.values.std()] # df.Age.values[0].mean(), df.Age.values[0].std() 66 | 67 | if df.Sex.values[0] == 'Male': 68 | vector.append(0) 69 | else: 70 | vector.append(1) 71 | 72 | if df.SmokingStatus.values[0] == 'Never smoked': 73 | vector.extend([0,0]) 74 | elif df.SmokingStatus.values[0] == 'Ex-smoker': 75 | vector.extend([1,1]) 76 | elif df.SmokingStatus.values[0] == 'Currently smokes': 77 | vector.extend([0,1]) 78 | else: 79 | vector.extend([1,0]) # this is useless 80 | 81 | vector.append((df.Volume.values[0] - train.Volume.values.mean()) / train.Volume.values.std()) 82 | return np.array(vector) 83 | 84 | 85 | A = {} # the slopes 86 | TAB = {} # tabular features 87 | P = [] # patient IDs 88 | 89 | for i, p in tqdm(enumerate(train.Patient.unique())): 90 | sub = train.loc[train.Patient == p, :] 91 | fvc = sub.FVC.values 92 | weeks = sub.Weeks.values 93 | c = np.vstack([weeks, np.ones(len(weeks))]).T 94 | 95 | a, _ = np.linalg.lstsq(c, fvc)[0] # we calculate the best slope with least square 96 | 97 | # ref: https://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html 98 | 99 | A[p] = a 100 | TAB[p] = get_tab(sub) 101 | P.append(p) 102 | 103 | 104 | 105 | 106 | 107 | class OSICData(Dataset): 108 | BAD_ID = ['ID00011637202177653955184', 'ID00052637202186188008618'] 109 | def __init__(self, keys, a, tab): 110 | self.keys = [k for k in keys if k not in self.BAD_ID] 111 | self.a = a 112 | self.tab = tab 113 | 114 | self.train_data = {} 115 | for p in train.Patient.values: 116 | p_n = len(os.listdir(f'{root_path}/train/{p}/')) 117 | self.train_data[p] = os.listdir(f'{root_path}/train/{p}/')[int( hyp.strip_ct * p_n):-int( hyp.strip_ct * p_n)] # removing first and last 15% slices 118 | 119 | 120 | def __len__(self): 121 | return len(self.keys) 122 | 123 | def get_img(self, path): 124 | d = pydicom.dcmread(path) 125 | return cv2.resize(d.pixel_array / 2**11, (512, 512)) # maybe bug in resize 126 | 127 | def __getitem__(self, idx): 128 | x = [] 129 | a, tab = [], [] 130 | k = self.keys[idx] # instead of random id send a specific id 131 | # np.random.choice(self.keys, 1)[0] 132 | 133 | try: 134 | i = np.random.choice(self.train_data[k], size=1)[0] 135 | # print(i) 136 | cp = f'{root_path}/train/{k}/{i}' 137 | #print(cp) 138 | img = self.get_img(cp) 139 | 140 | x.append(img) 141 | a.append(self.a[k]) 142 | tab.append(self.tab[k]) 143 | except: 144 | print('failed') 145 | print(k, i) 146 | 147 | x, a, tab = torch.tensor(x, dtype=torch.float32), torch.tensor(a, dtype=torch.float32), torch.tensor(tab, dtype=torch.float32) 148 | tab = torch.squeeze(tab, axis=0) 149 | return [x, tab] , a, k # k for patient id 150 | 151 | 152 | from torchvision import models 153 | from torch import nn 154 | from efficientnet_pytorch import EfficientNet 155 | from efficientnet_pytorch.utils import Conv2dStaticSamePadding 156 | 157 | class Identity(nn.Module): 158 | # credit: ptrblck 159 | def __init__(self): 160 | super(Identity, self).__init__() 161 | 162 | def forward(self, x): 163 | return x 164 | 165 | 166 | class Self_Attn(nn.Module): 167 | """ Self attention Layer""" 168 | def __init__(self,in_dim): 169 | super(Self_Attn,self).__init__() 170 | self.chanel_in = in_dim 171 | self.query_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1) 172 | self.key_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1) 173 | self.value_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim , kernel_size= 1) 174 | self.gamma = nn.Parameter(torch.rand(1)) # random initialization 175 | 176 | self.softmax = nn.Softmax(dim=-1) # 177 | def forward(self,x): 178 | """ 179 | inputs : 180 | x : input feature maps( B X C X W X H) 181 | returns : 182 | out : self attention value + input feature 183 | attention: B X N X N (N is Width*Height) 184 | """ 185 | m_batchsize,C,width ,height = x.size() 186 | proj_query = self.query_conv(x).view(m_batchsize,-1,width*height).permute(0,2,1) # B X CX(N) 187 | proj_key = self.key_conv(x).view(m_batchsize,-1,width*height) # B X C x (*W*H) 188 | energy = torch.bmm(proj_query,proj_key) # transpose check 189 | attention = self.softmax(energy) # BX (N) X (N) 190 | proj_value = self.value_conv(x).view(m_batchsize,-1,width*height) # B X C X N 191 | 192 | out = torch.bmm(proj_value,attention.permute(0,2,1) ) 193 | out = out.view(m_batchsize,C,width,height) 194 | 195 | out = self.gamma*out + x 196 | return out # , attention 197 | 198 | 199 | # only based on best config of b2 200 | class TabCT(nn.Module): 201 | def __init__(self, cnn, attn_filters, fc_dim, n_attn_layers): 202 | super(TabCT, self).__init__() 203 | 204 | # CT features 205 | cnn_dict = {'resnet18': models.resnet18, 'resnet34': models.resnet34, 'resnet50': models.resnet50, 206 | 'resnet101': models.resnet101, 'resnet152': models.resnet152, 'resnext50': models.resnext50_32x4d, 207 | 'resnext101': models.resnext101_32x8d} 208 | 209 | # feature dim 210 | self.out_dict = {'resnet18': 512, 'resnet34': 512, 'resnet50': 2048, 'resnet101': 2048, 'resnet152': 2048, 211 | 'resnext50': 2048, 'resnext101': 2048, "efnb0": 1280, "efnb1": 1280, "efnb2": 1408, 212 | "efnb3": 1536, "efnb4": 1792, "efnb5": 2048, "efnb6": 2304, "efnb7": 2560} 213 | 214 | self.n_tab = hyp.n_tab # n tabular features 215 | self.attn_filters = attn_filters 216 | self.fc_dim = fc_dim 217 | self.n_attn_layers = n_attn_layers 218 | 219 | # efficient net b2 base 220 | if cnn in cnn_dict.keys(): 221 | self.ct_cnn = cnn_dict[cnn](pretrained = True) 222 | 223 | # make single channel 224 | self.ct_cnn.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) 225 | 226 | self.ct_cnn.avgpool = nn.Conv2d(self.out_dict[cnn], self.attn_filters, kernel_size=(1, 1), bias=False) 227 | self.ct_cnn.fc = nn.Identity() 228 | 229 | # 1 self attn layer [stacked] 230 | self.attn = nn.ModuleList() 231 | 232 | for _ in range(self.n_attn_layers): 233 | self.attn.append(Self_Attn(self.attn_filters)) 234 | elif 'efn' in cnn: 235 | if 'b0' in cnn: 236 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b0') 237 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 238 | bias = False, image_size = 512) 239 | elif 'b1' in cnn: 240 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b1') 241 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 242 | bias = False, image_size = 512) 243 | elif 'b2' in cnn: 244 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b2') 245 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 246 | bias = False, image_size = 512) 247 | elif 'b3' in cnn: 248 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b3') 249 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 40, kernel_size = (3,3), stride = (2,2), 250 | bias = False, image_size = 512) 251 | elif 'b4' in cnn: 252 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b4') 253 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 48, kernel_size = (3,3), stride = (2,2), 254 | bias = False, image_size = 512) 255 | # replace avg_pool layer 256 | # 1408 is the number of filters in last conv 257 | self.ct_cnn._avg_pooling = Conv2dStaticSamePadding(self.out_dict[ cnn.split('_')[0] ], self.attn_filters, kernel_size = (1,1), stride = (1,1), 258 | bias = False, image_size = 512) 259 | self.ct_cnn._dropout = nn.Identity() 260 | self.ct_cnn._fc = nn.Identity() 261 | self.ct_cnn._swish = nn.Identity() 262 | 263 | # 1 self attn layer [stacked] 264 | self.attn = nn.ModuleList() 265 | 266 | for _ in range(self.n_attn_layers): 267 | self.attn.append(Self_Attn(self.attn_filters)) 268 | 269 | 270 | else: 271 | raise ValueError("cnn not recognized") 272 | 273 | self.avgpool = nn.AdaptiveAvgPool2d((1,1)) 274 | 275 | self.dropout = nn.Dropout(p=0.25) 276 | 277 | self.fc_inter = nn.Linear(self.attn_filters + self.n_tab, self.fc_dim) 278 | 279 | self.fc = nn.Linear(self.fc_dim, 1) 280 | 281 | def forward(self, x_ct, x_tab): 282 | print(x_ct.shape) 283 | ct_f = self.ct_cnn(x_ct).view(-1, self.attn_filters, 16, 16) # ct features 284 | #print(ct_f.shape) 285 | 286 | for ii in range(len(self.attn)): 287 | ct_f = self.attn[ii](ct_f) 288 | #print(ct_f.shape) 289 | ct_f = self.avgpool(ct_f).view(-1, self.attn_filters) 290 | #print(ct_f.shape) 291 | # print(x_tab.shape) 292 | 293 | # concatenate 294 | x = torch.cat((ct_f, x_tab), -1) # concat on last axis 295 | 296 | # dropout 297 | if self.training: 298 | x = self.dropout(x) 299 | 300 | x = self.fc_inter(x) 301 | 302 | x = self.fc(x) 303 | 304 | return x 305 | 306 | from sklearn.model_selection import train_test_split 307 | from sklearn.metrics import mean_squared_error 308 | 309 | 310 | 311 | 312 | 313 | # hyperparams 314 | 315 | result_dir = "perf_results" 316 | 317 | # training only resnet models on gpu 0 318 | train_models = hyp.train_models 319 | 320 | # 'resnext101' -> seems too heavy for 1080 321 | # 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4', 'efnb5', 'efnb6', 'efnb7' 322 | 323 | # device 324 | gpu = torch.device(f"cuda:{hyp.gpu_index}" if torch.cuda.is_available() else "cpu") 325 | 326 | 327 | 328 | # removing noisy data 329 | P = [p for p in P if p not in ['ID00011637202177653955184', 'ID00052637202186188008618']] 330 | 331 | 332 | # mac and param calc 333 | def prepare_input(resolution): 334 | x1 = torch.FloatTensor(1, 1, 512, 512).to(torch.device("cuda")) 335 | x2 = torch.FloatTensor(1, 5).to(torch.device("cuda")) 336 | return {"x_ct" : x1, "x_tab" : x2} 337 | 338 | for model in train_models: 339 | for fd in hyp.fc_dim: 340 | for af in hyp.attn_filters: 341 | for nal in hyp.n_attn_layers: 342 | log = open(f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.txt", "a+") 343 | kfold =KFold(n_splits=2) 344 | 345 | ifold = 0 346 | for train_index, test_index in kfold.split(P): 347 | # print(train_index, test_index) 348 | 349 | p_train = np.array(P)[train_index] 350 | p_test = np.array(P)[test_index] 351 | 352 | osic_train = OSICData(p_train, A, TAB) 353 | train_loader = DataLoader(osic_train, batch_size=1, shuffle=True, num_workers=hyp.num_workers) 354 | 355 | osic_val = OSICData(p_test, A, TAB) 356 | val_loader = DataLoader(osic_val, batch_size=1, shuffle=True, num_workers=hyp.num_workers) 357 | 358 | 359 | tabct = TabCT(cnn = model, fc_dim = fd, attn_filters = af, n_attn_layers = nal).to(gpu) # net 360 | 361 | macs, params = get_model_complexity_info(tabct, input_res = (1, 512, 512), input_constructor=prepare_input, as_strings=True, print_per_layer_stat=True, verbose=True) 362 | 363 | print(f"creating {model} with {fd} feature_dim, {af} attn_filters, and {nal} n_attn_layers") 364 | 365 | print(f"macs : {macs} params : {params}") 366 | log.write(f"macs : {macs} params : {params}\n") 367 | 368 | g_count = 0 369 | t1 = process_time() 370 | for epoch in range(5): # loop over the dataset multiple times 371 | 372 | tabct.eval() 373 | for i, data in tqdm(enumerate(val_loader, 0)): 374 | 375 | [x, t], a, pid = data 376 | 377 | x = x.to(gpu) 378 | t = t.to(gpu) 379 | a = a.to(gpu) 380 | 381 | # forward 382 | outputs = tabct(x, t) 383 | 384 | g_count += 1 385 | 386 | if g_count >= 10: 387 | break 388 | 389 | if g_count >= 10: 390 | break 391 | t2 = process_time() 392 | print(f"infer: {(t2-t1)/g_count}") 393 | log.write(f"infer: {(t2-t1)/g_count}\n") 394 | # destroy model 395 | del tabct 396 | torch.cuda.empty_cache() 397 | break 398 | 399 | 400 | 401 | 402 | # ref: https://www.kaggle.com/miklgr500/linear-decay-based-on-resnet-cnn 403 | # https://pytorch.org/docs/stable/index.html -------------------------------------------------------------------------------- /train_slopes.py: -------------------------------------------------------------------------------- 1 | # author: github/zabir-nabil 2 | 3 | # relevant imports 4 | 5 | import os 6 | import cv2 7 | 8 | import pydicom 9 | import pandas as pd 10 | import numpy as np 11 | # import tensorflow as tf 12 | # import matplotlib.pyplot as plt 13 | 14 | # torch dataset 15 | import torch 16 | from torch.utils.data import Dataset, DataLoader 17 | from torchvision import transforms, utils 18 | 19 | import random 20 | from tqdm import tqdm 21 | 22 | # k-fold 23 | from sklearn.model_selection import KFold 24 | 25 | # hyperparam object 26 | 27 | from config import HyperP 28 | 29 | hyp = HyperP(model_type = "slope_train") # slope prediction 30 | 31 | # seed 32 | seed = hyp.seed 33 | random.seed(seed) 34 | os.environ['PYTHONHASHSEED'] = str(seed) 35 | np.random.seed(seed) 36 | # tf.random.set_seed(seed) 37 | torch.manual_seed(seed) 38 | torch.backends.cudnn.deterministic = True 39 | torch.backends.cudnn.benchmark = False 40 | 41 | 42 | # path 43 | 44 | root_path = hyp.data_folder # ../input/osic-pulmonary-fibrosis-progression 45 | 46 | train = pd.read_csv(f'{root_path}/train.csv') 47 | train_vol = pd.read_csv(f'{hyp.ct_tab_feature_csv}') 48 | 49 | 50 | train['Volume'] = 2000. 51 | 52 | for i in range(len(train)): 53 | pid = train.iloc[i]['Patient'] 54 | try: 55 | train.at[i, 'Volume'] = train_vol[train_vol['Patient']==pid].iloc[0]['Volume'] 56 | except: 57 | print('bug at volume') 58 | 59 | 60 | # tabular feature generation 61 | 62 | def get_tab(df): 63 | vector = [(df.Age.values[0] - train.Age.values.mean()) / train.Age.values.std()] # df.Age.values[0].mean(), df.Age.values[0].std() 64 | 65 | if df.Sex.values[0] == 'Male': 66 | vector.append(0) 67 | else: 68 | vector.append(1) 69 | 70 | if df.SmokingStatus.values[0] == 'Never smoked': 71 | vector.extend([0,0]) 72 | elif df.SmokingStatus.values[0] == 'Ex-smoker': 73 | vector.extend([1,1]) 74 | elif df.SmokingStatus.values[0] == 'Currently smokes': 75 | vector.extend([0,1]) 76 | else: 77 | vector.extend([1,0]) # this is useless 78 | 79 | vector.append((df.Volume.values[0] - train.Volume.values.mean()) / train.Volume.values.std()) 80 | return np.array(vector) 81 | 82 | 83 | A = {} # the slopes 84 | TAB = {} # tabular features 85 | P = [] # patient IDs 86 | 87 | for i, p in tqdm(enumerate(train.Patient.unique())): 88 | sub = train.loc[train.Patient == p, :] 89 | fvc = sub.FVC.values 90 | weeks = sub.Weeks.values 91 | c = np.vstack([weeks, np.ones(len(weeks))]).T 92 | 93 | a, _ = np.linalg.lstsq(c, fvc)[0] # we calculate the best slope with least square 94 | 95 | # ref: https://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html 96 | 97 | A[p] = a 98 | TAB[p] = get_tab(sub) 99 | P.append(p) 100 | 101 | 102 | def get_img(path): 103 | d = pydicom.dcmread(path) 104 | return cv2.resize(d.pixel_array / 2**11, (512, 512)) 105 | 106 | 107 | class OSICData(Dataset): 108 | BAD_ID = ['ID00011637202177653955184', 'ID00052637202186188008618'] 109 | def __init__(self, keys, a, tab): 110 | self.keys = [k for k in keys if k not in self.BAD_ID] 111 | self.a = a 112 | self.tab = tab 113 | 114 | self.train_data = {} 115 | for p in train.Patient.values: 116 | p_n = len(os.listdir(f'{root_path}/train/{p}/')) 117 | self.train_data[p] = os.listdir(f'{root_path}/train/{p}/')[int( hyp.strip_ct * p_n):-int( hyp.strip_ct * p_n)] # removing first and last 15% slices 118 | 119 | 120 | def __len__(self): 121 | return len(self.keys) 122 | 123 | def __getitem__(self, idx): 124 | x = [] 125 | a, tab = [], [] 126 | k = self.keys[idx] # instead of random id send a specific id 127 | # np.random.choice(self.keys, 1)[0] 128 | 129 | try: 130 | i = np.random.choice(self.train_data[k], size=1)[0] 131 | img = get_img(f'{root_path}/train/{k}/{i}') 132 | x.append(img) 133 | a.append(self.a[k]) 134 | tab.append(self.tab[k]) 135 | except: 136 | print(k, i) 137 | 138 | x, a, tab = torch.tensor(x, dtype=torch.float32), torch.tensor(a, dtype=torch.float32), torch.tensor(tab, dtype=torch.float32) 139 | tab = torch.squeeze(tab, axis=0) 140 | return [x, tab] , a, k # k for patient id 141 | 142 | 143 | from torchvision import models 144 | from torch import nn 145 | from efficientnet_pytorch import EfficientNet 146 | from efficientnet_pytorch.utils import Conv2dStaticSamePadding 147 | 148 | class Identity(nn.Module): 149 | # credit: ptrblck 150 | def __init__(self): 151 | super(Identity, self).__init__() 152 | 153 | def forward(self, x): 154 | return x 155 | 156 | class TabCT(nn.Module): 157 | def __init__(self, cnn): 158 | super(TabCT, self).__init__() 159 | 160 | # CT features 161 | cnn_dict = {'resnet18': models.resnet18, 'resnet34': models.resnet34, 'resnet50': models.resnet50, 162 | 'resnet101': models.resnet101, 'resnet152': models.resnet152, 'resnext50': models.resnext50_32x4d, 163 | 'resnext101': models.resnext101_32x8d} 164 | 165 | # feature dim 166 | self.out_dict = {'resnet18': 512, 'resnet34': 512, 'resnet50': 2048, 'resnet101': 2048, 'resnet152': 2048, 167 | 'resnext50': 2048, 'resnext101': 2048, "efnb0": 1280, "efnb1": 1280, "efnb2": 1408, 168 | "efnb3": 1536, "efnb4": 1792, "efnb5": 2048, "efnb6": 2304, "efnb7": 2560} 169 | 170 | self.n_tab = hyp.n_tab # n tabular features 171 | 172 | # efficient net b0 to b7 173 | 174 | if cnn in cnn_dict.keys(): # resnet or resnext 175 | self.ct_cnn = cnn_dict[cnn](pretrained = True) 176 | 177 | # make single channel 178 | self.ct_cnn.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) 179 | 180 | # remove the fc layer/ add a simple linear layer 181 | self.ct_cnn.fc = nn.Linear(self.out_dict[cnn], hyp.cnn_dim) # mapped to 64 dimensions, Identity() 182 | 183 | elif cnn == "efnb0": 184 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b0') 185 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 186 | bias = False, image_size = 512) 187 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], hyp.cnn_dim) 188 | self.ct_cnn._swish = nn.Identity() 189 | 190 | elif cnn == "efnb1": 191 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b1') 192 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 193 | bias = False, image_size = 512) 194 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], hyp.cnn_dim) 195 | self.ct_cnn._swish = nn.Identity() 196 | 197 | elif cnn == "efnb2": 198 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b2') 199 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 200 | bias = False, image_size = 512) 201 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], hyp.cnn_dim) 202 | self.ct_cnn._swish = nn.Identity() 203 | 204 | elif cnn == "efnb3": 205 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b3') 206 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 40, kernel_size = (3,3), stride = (2,2), 207 | bias = False, image_size = 512) 208 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], hyp.cnn_dim) 209 | self.ct_cnn._swish = nn.Identity() 210 | 211 | elif cnn == "efnb4": 212 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b4') 213 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 48, kernel_size = (3,3), stride = (2,2), 214 | bias = False, image_size = 512) 215 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], hyp.cnn_dim) 216 | self.ct_cnn._swish = nn.Identity() 217 | 218 | elif cnn == "efnb5": 219 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b5') 220 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 48, kernel_size = (3,3), stride = (2,2), 221 | bias = False, image_size = 512) 222 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], hyp.cnn_dim) 223 | self.ct_cnn._swish = nn.Identity() 224 | 225 | elif cnn == "efnb6": 226 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b6') 227 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 56, kernel_size = (3,3), stride = (2,2), 228 | bias = False, image_size = 512) 229 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], hyp.cnn_dim) 230 | self.ct_cnn._swish = nn.Identity() 231 | 232 | elif cnn == "efnb7": 233 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b7') 234 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 64, kernel_size = (3,3), stride = (2,2), 235 | bias = False, image_size = 512) 236 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], hyp.cnn_dim) 237 | self.ct_cnn._swish = nn.Identity() 238 | 239 | else: 240 | raise ValueError("cnn not recognized") 241 | 242 | self.dropout = nn.Dropout(p=0.25) 243 | 244 | self.fc_inter = nn.Linear(hyp.cnn_dim + self.n_tab, hyp.fc_dim) 245 | 246 | self.fc = nn.Linear(hyp.fc_dim, 1) 247 | 248 | def forward(self, x_ct, x_tab): 249 | ct_f = self.ct_cnn(x_ct) # ct features 250 | 251 | # print(ct_f.shape) 252 | # print(x_tab.shape) 253 | 254 | # concatenate 255 | x = torch.cat((ct_f, x_tab), -1) # concat on last axis 256 | 257 | # dropout 258 | if self.training: 259 | x = self.dropout(x) 260 | 261 | x = self.fc_inter(x) 262 | 263 | x = self.fc(x) 264 | 265 | return x 266 | 267 | 268 | from sklearn.model_selection import train_test_split 269 | from sklearn.metrics import mean_squared_error 270 | 271 | # tr_p, val_p = train_test_split(P, 272 | # shuffle=True, 273 | # train_size= 0.75) # 75% training 274 | 275 | 276 | 277 | 278 | # dummy = False 279 | 280 | # if dummy: 281 | # tr_p = tr_p[:4] 282 | # val_p = val_p[:4] 283 | 284 | # osic_train = OSICData(tr_p, A, TAB) 285 | # train_loader = DataLoader(osic_train, batch_size=4, shuffle=True, num_workers=4) 286 | 287 | # osic_val = OSICData(val_p, A, TAB) 288 | # val_loader = DataLoader(osic_val, batch_size=4, shuffle=True, num_workers=4) 289 | 290 | # all data 291 | # osic_all = OSICData(tr_p + val_p, A, TAB) 292 | # all_loader = DataLoader(osic_all, batch_size=4, shuffle=True, num_workers=4) 293 | 294 | # score calculation 295 | 296 | def score(fvc_true, fvc_pred, sigma): 297 | sigma_clip = np.maximum(sigma, 70) 298 | delta = np.abs(fvc_true - fvc_pred) 299 | delta = np.minimum(delta, 1000) 300 | sq2 = np.sqrt(2) 301 | metric = (delta / sigma_clip)*sq2 + np.log(sigma_clip* sq2) 302 | return np.mean(metric) 303 | 304 | 305 | def score_avg(p, a): # patient id, predicted a 306 | percent_true = train.Percent.values[train.Patient == p] 307 | fvc_true = train.FVC.values[train.Patient == p] 308 | weeks_true = train.Weeks.values[train.Patient == p] 309 | 310 | fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 311 | percent = percent_true[0] - a * abs(weeks_true - weeks_true[0]) 312 | return score(fvc_true, fvc, percent) 313 | 314 | def rmse_avg(p, a): # patient id, predicted a 315 | percent_true = train.Percent.values[train.Patient == p] 316 | fvc_true = train.FVC.values[train.Patient == p] 317 | weeks_true = train.Weeks.values[train.Patient == p] 318 | 319 | fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 320 | return mean_squared_error(fvc_true, fvc, squared = False) 321 | 322 | 323 | # hyperparams 324 | 325 | result_dir = hyp.results_dir 326 | 327 | # training only resnet models on gpu 0 328 | train_models = hyp.train_models 329 | 330 | # 'resnext101' -> seems too heavy for 1080 331 | # 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4', 'efnb5', 'efnb6', 'efnb7' 332 | 333 | # device 334 | gpu = torch.device(f"cuda:{hyp.gpu_index}" if torch.cuda.is_available() else "cpu") 335 | 336 | 337 | nfold = hyp.nfold # hyper 338 | 339 | # removing noisy data 340 | P = [p for p in P if p not in ['ID00011637202177653955184', 'ID00052637202186188008618']] 341 | 342 | for model in train_models: 343 | log = open(f"{result_dir}/{model}.txt", "a+") 344 | kfold =KFold(n_splits=nfold) 345 | 346 | ifold = 0 347 | for train_index, test_index in kfold.split(P): 348 | # print(train_index, test_index) 349 | 350 | p_train = np.array(P)[train_index] 351 | p_test = np.array(P)[test_index] 352 | 353 | osic_train = OSICData(p_train, A, TAB) 354 | train_loader = DataLoader(osic_train, batch_size=hyp.batch_size, shuffle=True, num_workers=hyp.num_workers) 355 | 356 | osic_val = OSICData(p_test, A, TAB) 357 | val_loader = DataLoader(osic_val, batch_size=hyp.batch_size, shuffle=True, num_workers=hyp.num_workers) 358 | 359 | 360 | tabct = TabCT(cnn = model).to(gpu) 361 | print(f"creating {model}") 362 | print(f"fold: {ifold}") 363 | log.write(f"fold: {ifold}\n") 364 | 365 | ifold += 1 366 | 367 | 368 | n_epochs = hyp.n_epochs # max 30 epochs, patience 5, find the suitable epoch number for later final training 369 | 370 | best_epoch = n_epochs # 30 371 | 372 | 373 | optimizer = torch.optim.AdamW(tabct.parameters()) 374 | criterion = torch.nn.L1Loss() 375 | 376 | max_score = 99999999.0000 # here, max score ]= minimum score 377 | 378 | for epoch in range(n_epochs): # loop over the dataset multiple times 379 | 380 | running_loss = 0.0 381 | tabct.train() 382 | for i, data in tqdm(enumerate(train_loader, 0)): 383 | 384 | [x, t], a, _ = data 385 | 386 | x = x.to(gpu) 387 | t = t.to(gpu) 388 | a = a.to(gpu) 389 | 390 | # zero the parameter gradients 391 | optimizer.zero_grad() 392 | 393 | # forward + backward + optimize 394 | outputs = tabct(x, t) 395 | loss = criterion(outputs, a) 396 | loss.backward() 397 | optimizer.step() 398 | 399 | # print statistics 400 | running_loss += loss.item() 401 | print(f"epoch {epoch+1} train: {running_loss}") 402 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 403 | 404 | running_loss = 0.0 405 | pred_a = {} 406 | tabct.eval() 407 | for i, data in tqdm(enumerate(val_loader, 0)): 408 | 409 | [x, t], a, pid = data 410 | 411 | x = x.to(gpu) 412 | t = t.to(gpu) 413 | a = a.to(gpu) 414 | 415 | # forward 416 | outputs = tabct(x, t) 417 | loss = criterion(outputs, a) 418 | 419 | pids = pid 420 | preds_a = outputs.detach().cpu().numpy().flatten() 421 | 422 | for j, p_d in enumerate(pids): 423 | pred_a[p_d] = preds_a[j] 424 | 425 | 426 | 427 | 428 | # print statistics 429 | running_loss += loss.item() 430 | print(f"epoch {epoch+1} val: {running_loss}") 431 | log.write(f"epoch {epoch+1} val: {running_loss}\n") 432 | # score calculation 433 | print(pred_a) 434 | print(len(pred_a)) 435 | print(p_test) 436 | print(len(p_test)) 437 | score_v = 0. 438 | rmse = 0. 439 | for p in p_test: 440 | score_v += (score_avg(p, pred_a[p]))/len(p_test) 441 | rmse += (rmse_avg(p, pred_a[p]))/len(p_test) 442 | 443 | print(f"val score: {score_v}") 444 | log.write(f"val score: {score_v}\n") 445 | log.write(f"val rmse: {rmse}\n") 446 | 447 | if score_v <= max_score: 448 | torch.save({ 449 | 'epoch': epoch, 450 | 'model_state_dict': tabct.state_dict(), 451 | 'optimizer_state_dict': optimizer.state_dict(), 452 | 'score': score_v 453 | }, f"{result_dir}/{model}.tar") 454 | max_score = score_v 455 | best_epoch = epoch + 1 456 | # destroy model 457 | del tabct 458 | torch.cuda.empty_cache() 459 | 460 | # final training with optimized setting 461 | 462 | osic_all = OSICData(P, A, TAB) 463 | all_loader = DataLoader(osic_all, batch_size=2, shuffle=True, num_workers=hyp.num_workers) 464 | 465 | # load the best model 466 | tabct = TabCT(cnn = model).to(gpu) 467 | tabct.load_state_dict(torch.load(f"{result_dir}/{model}.tar")["model_state_dict"]) 468 | 469 | optimizer = torch.optim.AdamW(tabct.parameters(), lr = hyp.final_lr) # very small learning rate 470 | criterion = torch.nn.L1Loss() 471 | 472 | print(f"Final training") 473 | log.write(f"Final training\n") 474 | for epoch in range(best_epoch + 2): # loop over the dataset multiple times 475 | 476 | running_loss = 0.0 477 | tabct.train() 478 | for i, data in tqdm(enumerate(all_loader, 0)): 479 | 480 | [x, t], a, _ = data 481 | 482 | x = x.to(gpu) 483 | t = t.to(gpu) 484 | a = a.to(gpu) 485 | 486 | # zero the parameter gradients 487 | optimizer.zero_grad() 488 | 489 | # forward + backward + optimize 490 | outputs = tabct(x, t) 491 | loss = criterion(outputs, a) 492 | loss.backward() 493 | optimizer.step() 494 | 495 | # print statistics 496 | running_loss += loss.item() 497 | print(f"epoch {epoch+1} train: {running_loss}") 498 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 499 | 500 | torch.save({ 501 | 'epoch': best_epoch, 502 | 'model_state_dict': tabct.state_dict(), 503 | 'optimizer_state_dict': optimizer.state_dict() 504 | }, f"{result_dir}/{model}.tar") 505 | 506 | print('Finished Training') 507 | # destroy model 508 | del tabct 509 | torch.cuda.empty_cache() 510 | 511 | 512 | 513 | 514 | # ref: https://www.kaggle.com/miklgr500/linear-decay-based-on-resnet-cnn 515 | # https://pytorch.org/docs/stable/index.html -------------------------------------------------------------------------------- /train_attn_b2.py: -------------------------------------------------------------------------------- 1 | # author: github/zabir-nabil 2 | 3 | # relevant imports 4 | 5 | import os 6 | import cv2 7 | 8 | import pydicom 9 | import pandas as pd 10 | import numpy as np 11 | # import tensorflow as tf 12 | # import matplotlib.pyplot as plt 13 | 14 | # torch dataset 15 | import torch 16 | from torch.utils.data import Dataset, DataLoader 17 | from torchvision import transforms, utils 18 | 19 | import random 20 | from tqdm import tqdm 21 | 22 | # k-fold 23 | from sklearn.model_selection import KFold 24 | 25 | # hyperparam object 26 | 27 | from config import HyperP 28 | 29 | hyp = HyperP(model_type = "attn_train") # slope prediction 30 | 31 | # seed 32 | seed = hyp.seed 33 | random.seed(seed) 34 | os.environ['PYTHONHASHSEED'] = str(seed) 35 | np.random.seed(seed) 36 | # tf.random.set_seed(seed) 37 | torch.manual_seed(seed) 38 | torch.backends.cudnn.deterministic = True 39 | torch.backends.cudnn.benchmark = False 40 | 41 | 42 | # path 43 | 44 | root_path = hyp.data_folder # ../input/osic-pulmonary-fibrosis-progression 45 | 46 | train = pd.read_csv(f'{root_path}/train.csv') 47 | train_vol = pd.read_csv(f'{hyp.ct_tab_feature_csv}') 48 | 49 | 50 | train['Volume'] = 2000. 51 | 52 | for i in range(len(train)): 53 | pid = train.iloc[i]['Patient'] 54 | try: 55 | train.at[i, 'Volume'] = train_vol[train_vol['Patient']==pid].iloc[0]['Volume'] 56 | except: 57 | print('bug at volume') 58 | 59 | 60 | # tabular feature generation 61 | 62 | def get_tab(df): 63 | vector = [(df.Age.values[0] - train.Age.values.mean()) / train.Age.values.std()] # df.Age.values[0].mean(), df.Age.values[0].std() 64 | 65 | if df.Sex.values[0] == 'Male': 66 | vector.append(0) 67 | else: 68 | vector.append(1) 69 | 70 | if df.SmokingStatus.values[0] == 'Never smoked': 71 | vector.extend([0,0]) 72 | elif df.SmokingStatus.values[0] == 'Ex-smoker': 73 | vector.extend([1,1]) 74 | elif df.SmokingStatus.values[0] == 'Currently smokes': 75 | vector.extend([0,1]) 76 | else: 77 | vector.extend([1,0]) # this is useless 78 | 79 | vector.append((df.Volume.values[0] - train.Volume.values.mean()) / train.Volume.values.std()) 80 | return np.array(vector) 81 | 82 | 83 | A = {} # the slopes 84 | TAB = {} # tabular features 85 | P = [] # patient IDs 86 | 87 | for i, p in tqdm(enumerate(train.Patient.unique())): 88 | sub = train.loc[train.Patient == p, :] 89 | fvc = sub.FVC.values 90 | weeks = sub.Weeks.values 91 | c = np.vstack([weeks, np.ones(len(weeks))]).T 92 | 93 | a, _ = np.linalg.lstsq(c, fvc)[0] # we calculate the best slope with least square 94 | 95 | # ref: https://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html 96 | 97 | A[p] = a 98 | TAB[p] = get_tab(sub) 99 | P.append(p) 100 | 101 | 102 | def get_img(path): 103 | d = pydicom.dcmread(path) 104 | return cv2.resize(d.pixel_array / 2**11, (512, 512)) 105 | 106 | 107 | class OSICData(Dataset): 108 | BAD_ID = ['ID00011637202177653955184', 'ID00052637202186188008618'] 109 | def __init__(self, keys, a, tab): 110 | self.keys = [k for k in keys if k not in self.BAD_ID] 111 | self.a = a 112 | self.tab = tab 113 | 114 | self.train_data = {} 115 | for p in train.Patient.values: 116 | p_n = len(os.listdir(f'{root_path}/train/{p}/')) 117 | self.train_data[p] = os.listdir(f'{root_path}/train/{p}/')[int( hyp.strip_ct * p_n):-int( hyp.strip_ct * p_n)] # removing first and last 15% slices 118 | 119 | 120 | def __len__(self): 121 | return len(self.keys) 122 | 123 | def __getitem__(self, idx): 124 | x = [] 125 | a, tab = [], [] 126 | k = self.keys[idx] # instead of random id send a specific id 127 | # np.random.choice(self.keys, 1)[0] 128 | 129 | try: 130 | i = np.random.choice(self.train_data[k], size=1)[0] 131 | img = get_img(f'{root_path}/train/{k}/{i}') 132 | x.append(img) 133 | a.append(self.a[k]) 134 | tab.append(self.tab[k]) 135 | except: 136 | print(k, i) 137 | 138 | x, a, tab = torch.tensor(x, dtype=torch.float32), torch.tensor(a, dtype=torch.float32), torch.tensor(tab, dtype=torch.float32) 139 | tab = torch.squeeze(tab, axis=0) 140 | return [x, tab] , a, k # k for patient id 141 | 142 | 143 | from torchvision import models 144 | from torch import nn 145 | from efficientnet_pytorch import EfficientNet 146 | from efficientnet_pytorch.utils import Conv2dStaticSamePadding 147 | 148 | class Identity(nn.Module): 149 | # credit: ptrblck 150 | def __init__(self): 151 | super(Identity, self).__init__() 152 | 153 | def forward(self, x): 154 | return x 155 | 156 | 157 | class Self_Attn(nn.Module): 158 | """ Self attention Layer""" 159 | def __init__(self,in_dim): 160 | super(Self_Attn,self).__init__() 161 | self.chanel_in = in_dim 162 | self.query_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1) 163 | self.key_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1) 164 | self.value_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim , kernel_size= 1) 165 | self.gamma = nn.Parameter(torch.rand(1)) # random initialization 166 | 167 | self.softmax = nn.Softmax(dim=-1) # 168 | def forward(self,x): 169 | """ 170 | inputs : 171 | x : input feature maps( B X C X W X H) 172 | returns : 173 | out : self attention value + input feature 174 | attention: B X N X N (N is Width*Height) 175 | """ 176 | m_batchsize,C,width ,height = x.size() 177 | proj_query = self.query_conv(x).view(m_batchsize,-1,width*height).permute(0,2,1) # B X CX(N) 178 | proj_key = self.key_conv(x).view(m_batchsize,-1,width*height) # B X C x (*W*H) 179 | energy = torch.bmm(proj_query,proj_key) # transpose check 180 | attention = self.softmax(energy) # BX (N) X (N) 181 | proj_value = self.value_conv(x).view(m_batchsize,-1,width*height) # B X C X N 182 | 183 | out = torch.bmm(proj_value,attention.permute(0,2,1) ) 184 | out = out.view(m_batchsize,C,width,height) 185 | 186 | out = self.gamma*out + x 187 | return out # , attention 188 | 189 | 190 | # only based on efficientnet-b2 191 | class TabCT(nn.Module): 192 | def __init__(self, cnn, attn_filters, fc_dim, n_attn_layers): 193 | super(TabCT, self).__init__() 194 | 195 | # CT features 196 | cnn_dict = {'resnet18': models.resnet18, 'resnet34': models.resnet34, 'resnet50': models.resnet50, 197 | 'resnet101': models.resnet101, 'resnet152': models.resnet152, 'resnext50': models.resnext50_32x4d, 198 | 'resnext101': models.resnext101_32x8d} 199 | 200 | # feature dim 201 | self.out_dict = {'resnet18': 512, 'resnet34': 512, 'resnet50': 2048, 'resnet101': 2048, 'resnet152': 2048, 202 | 'resnext50': 2048, 'resnext101': 2048, "efnb0": 1280, "efnb1": 1280, "efnb2": 1408, 203 | "efnb3": 1536, "efnb4": 1792, "efnb5": 2048, "efnb6": 2304, "efnb7": 2560} 204 | 205 | self.n_tab = hyp.n_tab # n tabular features 206 | self.attn_filters = attn_filters 207 | self.fc_dim = fc_dim 208 | self.n_attn_layers = n_attn_layers 209 | 210 | # efficient net b2 base 211 | 212 | if cnn == "efnb2_attn": 213 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b2') 214 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 215 | bias = False, image_size = 512) 216 | 217 | 218 | # replace avg_pool layer 219 | # 1408 is the number of filters in last conv 220 | self.ct_cnn._avg_pooling = Conv2dStaticSamePadding(1408, self.attn_filters, kernel_size = (1,1), stride = (1,1), 221 | bias = False, image_size = 512) 222 | self.ct_cnn._dropout = nn.Identity() 223 | self.ct_cnn._fc = nn.Identity() 224 | self.ct_cnn._swish = nn.Identity() 225 | 226 | # 1 self attn layer [stacked] 227 | self.attn = nn.ModuleList() 228 | 229 | for _ in range(self.n_attn_layers): 230 | self.attn.append(Self_Attn(self.attn_filters)) 231 | 232 | 233 | else: 234 | raise ValueError("cnn not recognized") 235 | 236 | self.avgpool = nn.AdaptiveAvgPool2d((1,1)) 237 | 238 | self.dropout = nn.Dropout(p=0.25) 239 | 240 | self.fc_inter = nn.Linear(self.attn_filters + self.n_tab, self.fc_dim) 241 | 242 | self.fc = nn.Linear(self.fc_dim, 1) 243 | 244 | def forward(self, x_ct, x_tab): 245 | ct_f = self.ct_cnn(x_ct).view(-1, self.attn_filters, 16, 16) # ct features 246 | #print(ct_f.shape) 247 | 248 | for ii in range(len(self.attn)): 249 | ct_f = self.attn[ii](ct_f) 250 | #print(ct_f.shape) 251 | ct_f = self.avgpool(ct_f).view(-1, self.attn_filters) 252 | #print(ct_f.shape) 253 | # print(x_tab.shape) 254 | 255 | # concatenate 256 | x = torch.cat((ct_f, x_tab), -1) # concat on last axis 257 | 258 | # dropout 259 | if self.training: 260 | x = self.dropout(x) 261 | 262 | x = self.fc_inter(x) 263 | 264 | x = self.fc(x) 265 | 266 | return x 267 | 268 | from sklearn.model_selection import train_test_split 269 | from sklearn.metrics import mean_squared_error 270 | 271 | 272 | # score calculation 273 | 274 | def score(fvc_true, fvc_pred, sigma): 275 | sigma_clip = np.maximum(sigma, 70) 276 | delta = np.abs(fvc_true - fvc_pred) 277 | delta = np.minimum(delta, 1000) 278 | sq2 = np.sqrt(2) 279 | metric = (delta / sigma_clip)*sq2 + np.log(sigma_clip* sq2) 280 | return np.mean(metric) 281 | 282 | 283 | def score_avg(p, a): # patient id, predicted a 284 | percent_true = train.Percent.values[train.Patient == p] 285 | fvc_true = train.FVC.values[train.Patient == p] 286 | weeks_true = train.Weeks.values[train.Patient == p] 287 | 288 | fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 289 | percent = percent_true[0] - a * abs(weeks_true - weeks_true[0]) 290 | return score(fvc_true, fvc, percent) 291 | 292 | def rmse_avg(p, a): # patient id, predicted a 293 | percent_true = train.Percent.values[train.Patient == p] 294 | fvc_true = train.FVC.values[train.Patient == p] 295 | weeks_true = train.Weeks.values[train.Patient == p] 296 | 297 | fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 298 | return mean_squared_error(fvc_true, fvc, squared = False) 299 | 300 | 301 | # hyperparams 302 | 303 | result_dir = hyp.results_dir 304 | 305 | # training only resnet models on gpu 0 306 | train_models = hyp.train_models 307 | 308 | # 'resnext101' -> seems too heavy for 1080 309 | # 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4', 'efnb5', 'efnb6', 'efnb7' 310 | 311 | # device 312 | gpu = torch.device(f"cuda:{hyp.gpu_index}" if torch.cuda.is_available() else "cpu") 313 | 314 | 315 | nfold = hyp.nfold # hyper 316 | 317 | # removing noisy data 318 | P = [p for p in P if p not in ['ID00011637202177653955184', 'ID00052637202186188008618']] 319 | 320 | for model in train_models: 321 | for fd in hyp.fc_dim: 322 | for af in hyp.attn_filters: 323 | for nal in hyp.n_attn_layers: 324 | log = open(f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.txt", "a+") 325 | kfold =KFold(n_splits=nfold) 326 | 327 | ifold = 0 328 | for train_index, test_index in kfold.split(P): 329 | # print(train_index, test_index) 330 | 331 | p_train = np.array(P)[train_index] 332 | p_test = np.array(P)[test_index] 333 | 334 | osic_train = OSICData(p_train, A, TAB) 335 | train_loader = DataLoader(osic_train, batch_size=hyp.batch_size, shuffle=True, num_workers=hyp.num_workers) 336 | 337 | osic_val = OSICData(p_test, A, TAB) 338 | val_loader = DataLoader(osic_val, batch_size=hyp.batch_size, shuffle=True, num_workers=hyp.num_workers) 339 | 340 | 341 | tabct = TabCT(cnn = model, fc_dim = fd, attn_filters = af, n_attn_layers = nal).to(gpu) 342 | print(f"creating {model} with {fd} feature_dim, {af} attn_filters, and {nal} n_attn_layers") 343 | print(f"fold: {ifold}") 344 | log.write(f"fold: {ifold}\n") 345 | 346 | ifold += 1 347 | 348 | 349 | n_epochs = hyp.n_epochs # max 30 epochs, patience 5, find the suitable epoch number for later final training 350 | 351 | best_epoch = n_epochs # 30 352 | 353 | 354 | optimizer = torch.optim.AdamW(tabct.parameters()) 355 | criterion = torch.nn.L1Loss() 356 | 357 | max_score = 99999999.0000 # here, max score ]= minimum score 358 | 359 | for epoch in range(n_epochs): # loop over the dataset multiple times 360 | 361 | running_loss = 0.0 362 | tabct.train() 363 | for i, data in tqdm(enumerate(train_loader, 0)): 364 | 365 | [x, t], a, _ = data 366 | 367 | x = x.to(gpu) 368 | t = t.to(gpu) 369 | a = a.to(gpu) 370 | 371 | # zero the parameter gradients 372 | optimizer.zero_grad() 373 | 374 | # forward + backward + optimize 375 | outputs = tabct(x, t) 376 | loss = criterion(outputs, a) 377 | loss.backward() 378 | optimizer.step() 379 | 380 | # print statistics 381 | running_loss += loss.item() 382 | print(f"epoch {epoch+1} train: {running_loss}") 383 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 384 | 385 | running_loss = 0.0 386 | pred_a = {} 387 | tabct.eval() 388 | for i, data in tqdm(enumerate(val_loader, 0)): 389 | 390 | [x, t], a, pid = data 391 | 392 | x = x.to(gpu) 393 | t = t.to(gpu) 394 | a = a.to(gpu) 395 | 396 | # forward 397 | outputs = tabct(x, t) 398 | loss = criterion(outputs, a) 399 | 400 | pids = pid 401 | preds_a = outputs.detach().cpu().numpy().flatten() 402 | 403 | for j, p_d in enumerate(pids): 404 | pred_a[p_d] = preds_a[j] 405 | 406 | 407 | 408 | 409 | # print statistics 410 | running_loss += loss.item() 411 | print(f"epoch {epoch+1} val: {running_loss}") 412 | log.write(f"epoch {epoch+1} val: {running_loss}\n") 413 | # score calculation 414 | print(pred_a) 415 | print(len(pred_a)) 416 | print(p_test) 417 | print(len(p_test)) 418 | score_v = 0. 419 | rmse = 0. 420 | for p in p_test: 421 | score_v += (score_avg(p, pred_a[p]))/len(p_test) 422 | rmse += (rmse_avg(p, pred_a[p]))/len(p_test) 423 | 424 | print(f"val score: {score_v}") 425 | log.write(f"val score: {score_v}\n") 426 | log.write(f"val rmse: {rmse}\n") 427 | 428 | if score_v <= max_score: 429 | torch.save({ 430 | 'epoch': epoch, 431 | 'model_state_dict': tabct.state_dict(), 432 | 'optimizer_state_dict': optimizer.state_dict(), 433 | 'score': score_v 434 | }, f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.tar") 435 | max_score = score_v 436 | best_epoch = epoch + 1 437 | # destroy model 438 | del tabct 439 | torch.cuda.empty_cache() 440 | 441 | # final training with optimized setting 442 | 443 | osic_all = OSICData(P, A, TAB) 444 | all_loader = DataLoader(osic_all, batch_size=8, shuffle=True, num_workers=hyp.num_workers) 445 | 446 | # load the best model 447 | tabct = TabCT(cnn = model, fc_dim = fd, attn_filters = af, n_attn_layers = nal).to(gpu) 448 | tabct.load_state_dict(torch.load(f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.tar")["model_state_dict"]) 449 | 450 | optimizer = torch.optim.AdamW(tabct.parameters(), lr = hyp.final_lr) # very small learning rate 451 | criterion = torch.nn.L1Loss() 452 | 453 | print(f"Final training") 454 | log.write(f"Final training\n") 455 | for epoch in range(best_epoch + 2): # loop over the dataset multiple times 456 | 457 | running_loss = 0.0 458 | tabct.train() 459 | for i, data in tqdm(enumerate(all_loader, 0)): 460 | 461 | [x, t], a, _ = data 462 | 463 | x = x.to(gpu) 464 | t = t.to(gpu) 465 | a = a.to(gpu) 466 | 467 | # zero the parameter gradients 468 | optimizer.zero_grad() 469 | 470 | # forward + backward + optimize 471 | outputs = tabct(x, t) 472 | loss = criterion(outputs, a) 473 | loss.backward() 474 | optimizer.step() 475 | 476 | # print statistics 477 | running_loss += loss.item() 478 | print(f"epoch {epoch+1} train: {running_loss}") 479 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 480 | 481 | torch.save({ 482 | 'epoch': best_epoch, 483 | 'model_state_dict': tabct.state_dict(), 484 | 'optimizer_state_dict': optimizer.state_dict() 485 | }, f"{result_dir}/{model}.tar") 486 | 487 | print('Finished Training') 488 | # destroy model 489 | del tabct 490 | torch.cuda.empty_cache() 491 | 492 | 493 | 494 | 495 | # ref: https://www.kaggle.com/miklgr500/linear-decay-based-on-resnet-cnn 496 | # https://pytorch.org/docs/stable/index.html -------------------------------------------------------------------------------- /train_attn_best_config.py: -------------------------------------------------------------------------------- 1 | # author: github/zabir-nabil 2 | 3 | # relevant imports 4 | 5 | import os 6 | import cv2 7 | 8 | import pydicom 9 | import pandas as pd 10 | import numpy as np 11 | # import tensorflow as tf 12 | # import matplotlib.pyplot as plt 13 | 14 | # torch dataset 15 | import torch 16 | from torch.utils.data import Dataset, DataLoader 17 | from torchvision import transforms, utils 18 | 19 | import random 20 | from tqdm import tqdm 21 | 22 | # k-fold 23 | from sklearn.model_selection import KFold 24 | 25 | # hyperparam object 26 | 27 | from config import HyperP 28 | 29 | hyp = HyperP(model_type = "attn_train_best_config") # slope prediction 30 | 31 | # seed 32 | seed = hyp.seed 33 | random.seed(seed) 34 | os.environ['PYTHONHASHSEED'] = str(seed) 35 | np.random.seed(seed) 36 | # tf.random.set_seed(seed) 37 | torch.manual_seed(seed) 38 | torch.backends.cudnn.deterministic = True 39 | torch.backends.cudnn.benchmark = False 40 | 41 | 42 | # path 43 | 44 | root_path = hyp.data_folder # ../input/osic-pulmonary-fibrosis-progression 45 | 46 | train = pd.read_csv(f'{root_path}/train.csv') 47 | train_vol = pd.read_csv(f'{hyp.ct_tab_feature_csv}') 48 | 49 | 50 | train['Volume'] = 2000. 51 | 52 | for i in range(len(train)): 53 | pid = train.iloc[i]['Patient'] 54 | try: 55 | train.at[i, 'Volume'] = train_vol[train_vol['Patient']==pid].iloc[0]['Volume'] 56 | except: 57 | print('bug at volume') 58 | 59 | 60 | # tabular feature generation 61 | 62 | def get_tab(df): 63 | vector = [(df.Age.values[0] - train.Age.values.mean()) / train.Age.values.std()] # df.Age.values[0].mean(), df.Age.values[0].std() 64 | 65 | if df.Sex.values[0] == 'Male': 66 | vector.append(0) 67 | else: 68 | vector.append(1) 69 | 70 | if df.SmokingStatus.values[0] == 'Never smoked': 71 | vector.extend([0,0]) 72 | elif df.SmokingStatus.values[0] == 'Ex-smoker': 73 | vector.extend([1,1]) 74 | elif df.SmokingStatus.values[0] == 'Currently smokes': 75 | vector.extend([0,1]) 76 | else: 77 | vector.extend([1,0]) # this is useless 78 | 79 | vector.append((df.Volume.values[0] - train.Volume.values.mean()) / train.Volume.values.std()) 80 | return np.array(vector) 81 | 82 | 83 | A = {} # the slopes 84 | TAB = {} # tabular features 85 | P = [] # patient IDs 86 | 87 | for i, p in tqdm(enumerate(train.Patient.unique())): 88 | sub = train.loc[train.Patient == p, :] 89 | fvc = sub.FVC.values 90 | weeks = sub.Weeks.values 91 | c = np.vstack([weeks, np.ones(len(weeks))]).T 92 | 93 | a, _ = np.linalg.lstsq(c, fvc)[0] # we calculate the best slope with least square 94 | 95 | # ref: https://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html 96 | 97 | A[p] = a 98 | TAB[p] = get_tab(sub) 99 | P.append(p) 100 | 101 | 102 | 103 | 104 | 105 | class OSICData(Dataset): 106 | BAD_ID = ['ID00011637202177653955184', 'ID00052637202186188008618'] 107 | def __init__(self, keys, a, tab): 108 | self.keys = [k for k in keys if k not in self.BAD_ID] 109 | self.a = a 110 | self.tab = tab 111 | 112 | self.train_data = {} 113 | for p in train.Patient.values: 114 | p_n = len(os.listdir(f'{root_path}/train/{p}/')) 115 | self.train_data[p] = os.listdir(f'{root_path}/train/{p}/')[int( hyp.strip_ct * p_n):-int( hyp.strip_ct * p_n)] # removing first and last 15% slices 116 | 117 | 118 | def __len__(self): 119 | return len(self.keys) 120 | 121 | def get_img(self, path): 122 | d = pydicom.dcmread(path) 123 | return cv2.resize(d.pixel_array / 2**11, (512, 512)) # maybe bug in resize 124 | 125 | def __getitem__(self, idx): 126 | x = [] 127 | a, tab = [], [] 128 | k = self.keys[idx] # instead of random id send a specific id 129 | # np.random.choice(self.keys, 1)[0] 130 | 131 | try: 132 | i = np.random.choice(self.train_data[k], size=1)[0] 133 | # print(i) 134 | cp = f'{root_path}/train/{k}/{i}' 135 | #print(cp) 136 | img = self.get_img(cp) 137 | 138 | x.append(img) 139 | a.append(self.a[k]) 140 | tab.append(self.tab[k]) 141 | except: 142 | print('failed') 143 | print(k, i) 144 | 145 | x, a, tab = torch.tensor(x, dtype=torch.float32), torch.tensor(a, dtype=torch.float32), torch.tensor(tab, dtype=torch.float32) 146 | tab = torch.squeeze(tab, axis=0) 147 | return [x, tab] , a, k # k for patient id 148 | 149 | 150 | from torchvision import models 151 | from torch import nn 152 | from efficientnet_pytorch import EfficientNet 153 | from efficientnet_pytorch.utils import Conv2dStaticSamePadding 154 | 155 | class Identity(nn.Module): 156 | # credit: ptrblck 157 | def __init__(self): 158 | super(Identity, self).__init__() 159 | 160 | def forward(self, x): 161 | return x 162 | 163 | 164 | class Self_Attn(nn.Module): 165 | """ Self attention Layer""" 166 | def __init__(self,in_dim): 167 | super(Self_Attn,self).__init__() 168 | self.chanel_in = in_dim 169 | self.query_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1) 170 | self.key_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1) 171 | self.value_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim , kernel_size= 1) 172 | self.gamma = nn.Parameter(torch.rand(1)) # random initialization 173 | 174 | self.softmax = nn.Softmax(dim=-1) # 175 | def forward(self,x): 176 | """ 177 | inputs : 178 | x : input feature maps( B X C X W X H) 179 | returns : 180 | out : self attention value + input feature 181 | attention: B X N X N (N is Width*Height) 182 | """ 183 | m_batchsize,C,width ,height = x.size() 184 | proj_query = self.query_conv(x).view(m_batchsize,-1,width*height).permute(0,2,1) # B X CX(N) 185 | proj_key = self.key_conv(x).view(m_batchsize,-1,width*height) # B X C x (*W*H) 186 | energy = torch.bmm(proj_query,proj_key) # transpose check 187 | attention = self.softmax(energy) # BX (N) X (N) 188 | proj_value = self.value_conv(x).view(m_batchsize,-1,width*height) # B X C X N 189 | 190 | out = torch.bmm(proj_value,attention.permute(0,2,1) ) 191 | out = out.view(m_batchsize,C,width,height) 192 | 193 | out = self.gamma*out + x 194 | return out # , attention 195 | 196 | 197 | # only based on best config of b2 198 | class TabCT(nn.Module): 199 | def __init__(self, cnn, attn_filters, fc_dim, n_attn_layers): 200 | super(TabCT, self).__init__() 201 | 202 | # CT features 203 | cnn_dict = {'resnet18': models.resnet18, 'resnet34': models.resnet34, 'resnet50': models.resnet50, 204 | 'resnet101': models.resnet101, 'resnet152': models.resnet152, 'resnext50': models.resnext50_32x4d, 205 | 'resnext101': models.resnext101_32x8d} 206 | 207 | # feature dim 208 | self.out_dict = {'resnet18': 512, 'resnet34': 512, 'resnet50': 2048, 'resnet101': 2048, 'resnet152': 2048, 209 | 'resnext50': 2048, 'resnext101': 2048, "efnb0": 1280, "efnb1": 1280, "efnb2": 1408, 210 | "efnb3": 1536, "efnb4": 1792, "efnb5": 2048, "efnb6": 2304, "efnb7": 2560} 211 | 212 | self.n_tab = hyp.n_tab # n tabular features 213 | self.attn_filters = attn_filters 214 | self.fc_dim = fc_dim 215 | self.n_attn_layers = n_attn_layers 216 | 217 | # efficient net b2 base 218 | if cnn in cnn_dict.keys(): 219 | self.ct_cnn = cnn_dict[cnn](pretrained = True) 220 | 221 | # make single channel 222 | self.ct_cnn.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) 223 | 224 | self.ct_cnn.avgpool = nn.Conv2d(self.out_dict[cnn], self.attn_filters, kernel_size=(1, 1), bias=False) 225 | self.ct_cnn.fc = nn.Identity() 226 | 227 | # 1 self attn layer [stacked] 228 | self.attn = nn.ModuleList() 229 | 230 | for _ in range(self.n_attn_layers): 231 | self.attn.append(Self_Attn(self.attn_filters)) 232 | elif 'efn' in cnn: 233 | if 'b0' in cnn: 234 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b0') 235 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 236 | bias = False, image_size = 512) 237 | elif 'b1' in cnn: 238 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b1') 239 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 240 | bias = False, image_size = 512) 241 | elif 'b2' in cnn: 242 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b2') 243 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 244 | bias = False, image_size = 512) 245 | elif 'b3' in cnn: 246 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b3') 247 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 40, kernel_size = (3,3), stride = (2,2), 248 | bias = False, image_size = 512) 249 | elif 'b4' in cnn: 250 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b4') 251 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 48, kernel_size = (3,3), stride = (2,2), 252 | bias = False, image_size = 512) 253 | # replace avg_pool layer 254 | # 1408 is the number of filters in last conv 255 | self.ct_cnn._avg_pooling = Conv2dStaticSamePadding(self.out_dict[ cnn.split('_')[0] ], self.attn_filters, kernel_size = (1,1), stride = (1,1), 256 | bias = False, image_size = 512) 257 | self.ct_cnn._dropout = nn.Identity() 258 | self.ct_cnn._fc = nn.Identity() 259 | self.ct_cnn._swish = nn.Identity() 260 | 261 | # 1 self attn layer [stacked] 262 | self.attn = nn.ModuleList() 263 | 264 | for _ in range(self.n_attn_layers): 265 | self.attn.append(Self_Attn(self.attn_filters)) 266 | 267 | 268 | else: 269 | raise ValueError("cnn not recognized") 270 | 271 | self.avgpool = nn.AdaptiveAvgPool2d((1,1)) 272 | 273 | self.dropout = nn.Dropout(p=0.25) 274 | 275 | self.fc_inter = nn.Linear(self.attn_filters + self.n_tab, self.fc_dim) 276 | 277 | self.fc = nn.Linear(self.fc_dim, 1) 278 | 279 | def forward(self, x_ct, x_tab): 280 | print(x_ct.shape) 281 | ct_f = self.ct_cnn(x_ct).view(-1, self.attn_filters, 16, 16) # ct features 282 | #print(ct_f.shape) 283 | 284 | for ii in range(len(self.attn)): 285 | ct_f = self.attn[ii](ct_f) 286 | #print(ct_f.shape) 287 | ct_f = self.avgpool(ct_f).view(-1, self.attn_filters) 288 | #print(ct_f.shape) 289 | # print(x_tab.shape) 290 | 291 | # concatenate 292 | x = torch.cat((ct_f, x_tab), -1) # concat on last axis 293 | 294 | # dropout 295 | if self.training: 296 | x = self.dropout(x) 297 | 298 | x = self.fc_inter(x) 299 | 300 | x = self.fc(x) 301 | 302 | return x 303 | 304 | from sklearn.model_selection import train_test_split 305 | from sklearn.metrics import mean_squared_error 306 | 307 | 308 | # score calculation 309 | 310 | def score(fvc_true, fvc_pred, sigma): 311 | sigma_clip = np.maximum(sigma, 70) 312 | delta = np.abs(fvc_true - fvc_pred) 313 | delta = np.minimum(delta, 1000) 314 | sq2 = np.sqrt(2) 315 | metric = (delta / sigma_clip)*sq2 + np.log(sigma_clip* sq2) 316 | return np.mean(metric) 317 | 318 | 319 | def score_avg(p, a): # patient id, predicted a 320 | percent_true = train.Percent.values[train.Patient == p] 321 | fvc_true = train.FVC.values[train.Patient == p] 322 | weeks_true = train.Weeks.values[train.Patient == p] 323 | 324 | fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 325 | percent = percent_true[0] - a * abs(weeks_true - weeks_true[0]) 326 | return score(fvc_true, fvc, percent) 327 | 328 | def rmse_avg(p, a): # patient id, predicted a 329 | percent_true = train.Percent.values[train.Patient == p] 330 | fvc_true = train.FVC.values[train.Patient == p] 331 | weeks_true = train.Weeks.values[train.Patient == p] 332 | 333 | fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 334 | return mean_squared_error(fvc_true, fvc, squared = False) 335 | 336 | 337 | # hyperparams 338 | 339 | result_dir = hyp.results_dir 340 | 341 | # training only resnet models on gpu 0 342 | train_models = hyp.train_models 343 | 344 | # 'resnext101' -> seems too heavy for 1080 345 | # 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4', 'efnb5', 'efnb6', 'efnb7' 346 | 347 | # device 348 | gpu = torch.device(f"cuda:{hyp.gpu_index}" if torch.cuda.is_available() else "cpu") 349 | 350 | 351 | nfold = hyp.nfold # hyper 352 | 353 | # removing noisy data 354 | P = [p for p in P if p not in ['ID00011637202177653955184', 'ID00052637202186188008618']] 355 | 356 | for model in train_models: 357 | for fd in hyp.fc_dim: 358 | for af in hyp.attn_filters: 359 | for nal in hyp.n_attn_layers: 360 | log = open(f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.txt", "a+") 361 | kfold =KFold(n_splits=nfold) 362 | 363 | ifold = 0 364 | for train_index, test_index in kfold.split(P): 365 | # print(train_index, test_index) 366 | 367 | p_train = np.array(P)[train_index] 368 | p_test = np.array(P)[test_index] 369 | 370 | osic_train = OSICData(p_train, A, TAB) 371 | train_loader = DataLoader(osic_train, batch_size=hyp.batch_size, shuffle=True, num_workers=hyp.num_workers) 372 | 373 | osic_val = OSICData(p_test, A, TAB) 374 | val_loader = DataLoader(osic_val, batch_size=hyp.batch_size, shuffle=True, num_workers=hyp.num_workers) 375 | 376 | 377 | tabct = TabCT(cnn = model, fc_dim = fd, attn_filters = af, n_attn_layers = nal).to(gpu) 378 | print(f"creating {model} with {fd} feature_dim, {af} attn_filters, and {nal} n_attn_layers") 379 | print(f"fold: {ifold}") 380 | log.write(f"fold: {ifold}\n") 381 | 382 | ifold += 1 383 | 384 | 385 | n_epochs = hyp.n_epochs # max 30 epochs, patience 5, find the suitable epoch number for later final training 386 | 387 | best_epoch = n_epochs # 30 388 | 389 | 390 | optimizer = torch.optim.AdamW(tabct.parameters()) 391 | criterion = torch.nn.L1Loss() 392 | 393 | max_score = 99999999.0000 # here, max score ]= minimum score 394 | 395 | for epoch in range(n_epochs): # loop over the dataset multiple times 396 | 397 | running_loss = 0.0 398 | tabct.train() 399 | for i, data in tqdm(enumerate(train_loader, 0)): 400 | 401 | [x, t], a, _ = data 402 | print(x.shape) 403 | print(t.shape) 404 | 405 | x = x.to(gpu) 406 | t = t.to(gpu) 407 | a = a.to(gpu) 408 | 409 | # zero the parameter gradients 410 | optimizer.zero_grad() 411 | 412 | # forward + backward + optimize 413 | outputs = tabct(x, t) 414 | loss = criterion(outputs, a) 415 | loss.backward() 416 | optimizer.step() 417 | 418 | # print statistics 419 | running_loss += loss.item() 420 | print(f"epoch {epoch+1} train: {running_loss}") 421 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 422 | 423 | running_loss = 0.0 424 | pred_a = {} 425 | tabct.eval() 426 | for i, data in tqdm(enumerate(val_loader, 0)): 427 | 428 | [x, t], a, pid = data 429 | 430 | x = x.to(gpu) 431 | t = t.to(gpu) 432 | a = a.to(gpu) 433 | 434 | # forward 435 | outputs = tabct(x, t) 436 | loss = criterion(outputs, a) 437 | 438 | pids = pid 439 | preds_a = outputs.detach().cpu().numpy().flatten() 440 | 441 | for j, p_d in enumerate(pids): 442 | pred_a[p_d] = preds_a[j] 443 | 444 | 445 | 446 | 447 | # print statistics 448 | running_loss += loss.item() 449 | print(f"epoch {epoch+1} val: {running_loss}") 450 | log.write(f"epoch {epoch+1} val: {running_loss}\n") 451 | # score calculation 452 | print(pred_a) 453 | print(len(pred_a)) 454 | print(p_test) 455 | print(len(p_test)) 456 | score_v = 0. 457 | rmse = 0. 458 | for p in p_test: 459 | score_v += (score_avg(p, pred_a[p]))/len(p_test) 460 | rmse += (rmse_avg(p, pred_a[p]))/len(p_test) 461 | 462 | print(f"val score: {score_v}") 463 | log.write(f"val score: {score_v}\n") 464 | log.write(f"val rmse: {rmse}\n") 465 | 466 | if score_v <= max_score: 467 | torch.save({ 468 | 'epoch': epoch, 469 | 'model_state_dict': tabct.state_dict(), 470 | 'optimizer_state_dict': optimizer.state_dict(), 471 | 'score': score_v 472 | }, f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.tar") 473 | max_score = score_v 474 | best_epoch = epoch + 1 475 | # destroy model 476 | del tabct 477 | torch.cuda.empty_cache() 478 | 479 | # final training with optimized setting 480 | 481 | osic_all = OSICData(P, A, TAB) 482 | all_loader = DataLoader(osic_all, batch_size=8, shuffle=True, num_workers=hyp.num_workers) 483 | 484 | # load the best model 485 | tabct = TabCT(cnn = model, fc_dim = fd, attn_filters = af, n_attn_layers = nal).to(gpu) 486 | tabct.load_state_dict(torch.load(f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.tar")["model_state_dict"]) 487 | 488 | optimizer = torch.optim.AdamW(tabct.parameters(), lr = hyp.final_lr) # very small learning rate 489 | criterion = torch.nn.L1Loss() 490 | 491 | print(f"Final training") 492 | log.write(f"Final training\n") 493 | for epoch in range(best_epoch + 2): # loop over the dataset multiple times 494 | 495 | running_loss = 0.0 496 | tabct.train() 497 | for i, data in tqdm(enumerate(all_loader, 0)): 498 | 499 | [x, t], a, _ = data 500 | 501 | x = x.to(gpu) 502 | t = t.to(gpu) 503 | a = a.to(gpu) 504 | 505 | # zero the parameter gradients 506 | optimizer.zero_grad() 507 | 508 | # forward + backward + optimize 509 | outputs = tabct(x, t) 510 | loss = criterion(outputs, a) 511 | loss.backward() 512 | optimizer.step() 513 | 514 | # print statistics 515 | running_loss += loss.item() 516 | print(f"epoch {epoch+1} train: {running_loss}") 517 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 518 | 519 | torch.save({ 520 | 'epoch': best_epoch, 521 | 'model_state_dict': tabct.state_dict(), 522 | 'optimizer_state_dict': optimizer.state_dict() 523 | }, f"{result_dir}/{model}.tar") 524 | 525 | print('Finished Training') 526 | # destroy model 527 | del tabct 528 | torch.cuda.empty_cache() 529 | 530 | 531 | 532 | 533 | # ref: https://www.kaggle.com/miklgr500/linear-decay-based-on-resnet-cnn 534 | # https://pytorch.org/docs/stable/index.html -------------------------------------------------------------------------------- /modal_ct.py: -------------------------------------------------------------------------------- 1 | # author: github/zabir-nabil 2 | 3 | # relevant imports 4 | 5 | import os 6 | import cv2 7 | 8 | import pydicom 9 | import pandas as pd 10 | import numpy as np 11 | # import tensorflow as tf 12 | # import matplotlib.pyplot as plt 13 | 14 | # torch dataset 15 | import torch 16 | from torch.utils.data import Dataset, DataLoader 17 | from torchvision import transforms, utils 18 | 19 | import random 20 | from tqdm import tqdm 21 | 22 | # k-fold 23 | from sklearn.model_selection import KFold 24 | 25 | # hyperparam object 26 | 27 | from config import HyperP 28 | 29 | hyp = HyperP(model_type = "singlemodal_ct") # slope prediction 30 | 31 | # seed 32 | seed = hyp.seed 33 | random.seed(seed) 34 | os.environ['PYTHONHASHSEED'] = str(seed) 35 | np.random.seed(seed) 36 | # tf.random.set_seed(seed) 37 | torch.manual_seed(seed) 38 | torch.backends.cudnn.deterministic = True 39 | torch.backends.cudnn.benchmark = False 40 | 41 | 42 | # path 43 | 44 | root_path = hyp.data_folder # ../input/osic-pulmonary-fibrosis-progression 45 | 46 | train = pd.read_csv(f'{root_path}/train.csv') 47 | train_vol = pd.read_csv(f'{hyp.ct_tab_feature_csv}') 48 | 49 | 50 | train['Volume'] = 2000. 51 | 52 | for i in range(len(train)): 53 | pid = train.iloc[i]['Patient'] 54 | try: 55 | train.at[i, 'Volume'] = train_vol[train_vol['Patient']==pid].iloc[0]['Volume'] 56 | except: 57 | print('bug at volume') 58 | 59 | 60 | # tabular feature generation 61 | 62 | def get_tab(df): 63 | 64 | # passing a zero vector, no features encoded 65 | return np.zeros(5, dtype=np.float) 66 | 67 | vector = [(df.Age.values[0] - train.Age.values.mean()) / train.Age.values.std()] # df.Age.values[0].mean(), df.Age.values[0].std() 68 | 69 | if df.Sex.values[0] == 'Male': 70 | vector.append(0) 71 | else: 72 | vector.append(1) 73 | 74 | if df.SmokingStatus.values[0] == 'Never smoked': 75 | vector.extend([0,0]) 76 | elif df.SmokingStatus.values[0] == 'Ex-smoker': 77 | vector.extend([1,1]) 78 | elif df.SmokingStatus.values[0] == 'Currently smokes': 79 | vector.extend([0,1]) 80 | else: 81 | vector.extend([1,0]) # this is useless 82 | 83 | vector.append((df.Volume.values[0] - train.Volume.values.mean()) / train.Volume.values.std()) 84 | return np.array(vector) 85 | 86 | 87 | A = {} # the slopes 88 | TAB = {} # tabular features 89 | P = [] # patient IDs 90 | 91 | for i, p in tqdm(enumerate(train.Patient.unique())): 92 | sub = train.loc[train.Patient == p, :] 93 | fvc = sub.FVC.values 94 | weeks = sub.Weeks.values 95 | c = np.vstack([weeks, np.ones(len(weeks))]).T 96 | 97 | a, _ = np.linalg.lstsq(c, fvc)[0] # we calculate the best slope with least square 98 | 99 | # ref: https://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html 100 | 101 | A[p] = a 102 | TAB[p] = get_tab(sub) 103 | P.append(p) 104 | 105 | 106 | 107 | 108 | 109 | class OSICData(Dataset): 110 | BAD_ID = ['ID00011637202177653955184', 'ID00052637202186188008618'] 111 | def __init__(self, keys, a, tab): 112 | self.keys = [k for k in keys if k not in self.BAD_ID] 113 | self.a = a 114 | self.tab = tab 115 | 116 | self.train_data = {} 117 | for p in train.Patient.values: 118 | p_n = len(os.listdir(f'{root_path}/train/{p}/')) 119 | self.train_data[p] = os.listdir(f'{root_path}/train/{p}/')[int( hyp.strip_ct * p_n):-int( hyp.strip_ct * p_n)] # removing first and last 15% slices 120 | 121 | 122 | def __len__(self): 123 | return len(self.keys) 124 | 125 | def get_img(self, path): 126 | d = pydicom.dcmread(path) 127 | return cv2.resize(d.pixel_array / 2**11, (512, 512)) # maybe bug in resize 128 | 129 | def __getitem__(self, idx): 130 | x = [] 131 | a, tab = [], [] 132 | k = self.keys[idx] # instead of random id send a specific id 133 | # np.random.choice(self.keys, 1)[0] 134 | 135 | try: 136 | i = np.random.choice(self.train_data[k], size=1)[0] 137 | # print(i) 138 | cp = f'{root_path}/train/{k}/{i}' 139 | #print(cp) 140 | img = self.get_img(cp) 141 | 142 | x.append(img) 143 | a.append(self.a[k]) 144 | tab.append(self.tab[k]) 145 | except: 146 | print('failed') 147 | print(k, i) 148 | 149 | x, a, tab = torch.tensor(x, dtype=torch.float32), torch.tensor(a, dtype=torch.float32), torch.tensor(tab, dtype=torch.float32) 150 | tab = torch.squeeze(tab, axis=0) 151 | return [x, tab] , a, k # k for patient id 152 | 153 | 154 | from torchvision import models 155 | from torch import nn 156 | from efficientnet_pytorch import EfficientNet 157 | from efficientnet_pytorch.utils import Conv2dStaticSamePadding 158 | 159 | class Identity(nn.Module): 160 | # credit: ptrblck 161 | def __init__(self): 162 | super(Identity, self).__init__() 163 | 164 | def forward(self, x): 165 | return x 166 | 167 | 168 | class Self_Attn(nn.Module): 169 | """ Self attention Layer""" 170 | def __init__(self,in_dim): 171 | super(Self_Attn,self).__init__() 172 | self.chanel_in = in_dim 173 | self.query_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1) 174 | self.key_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim//8 , kernel_size= 1) 175 | self.value_conv = nn.Conv2d(in_channels = in_dim , out_channels = in_dim , kernel_size= 1) 176 | self.gamma = nn.Parameter(torch.rand(1)) # random initialization 177 | 178 | self.softmax = nn.Softmax(dim=-1) # 179 | def forward(self,x): 180 | """ 181 | inputs : 182 | x : input feature maps( B X C X W X H) 183 | returns : 184 | out : self attention value + input feature 185 | attention: B X N X N (N is Width*Height) 186 | """ 187 | m_batchsize,C,width ,height = x.size() 188 | proj_query = self.query_conv(x).view(m_batchsize,-1,width*height).permute(0,2,1) # B X CX(N) 189 | proj_key = self.key_conv(x).view(m_batchsize,-1,width*height) # B X C x (*W*H) 190 | energy = torch.bmm(proj_query,proj_key) # transpose check 191 | attention = self.softmax(energy) # BX (N) X (N) 192 | proj_value = self.value_conv(x).view(m_batchsize,-1,width*height) # B X C X N 193 | 194 | out = torch.bmm(proj_value,attention.permute(0,2,1) ) 195 | out = out.view(m_batchsize,C,width,height) 196 | 197 | out = self.gamma*out + x 198 | return out # , attention 199 | 200 | 201 | # only based on best config of b2 202 | class TabCT(nn.Module): 203 | def __init__(self, cnn, attn_filters, fc_dim, n_attn_layers): 204 | super(TabCT, self).__init__() 205 | 206 | # CT features 207 | cnn_dict = {'resnet18': models.resnet18, 'resnet34': models.resnet34, 'resnet50': models.resnet50, 208 | 'resnet101': models.resnet101, 'resnet152': models.resnet152, 'resnext50': models.resnext50_32x4d, 209 | 'resnext101': models.resnext101_32x8d} 210 | 211 | # feature dim 212 | self.out_dict = {'resnet18': 512, 'resnet34': 512, 'resnet50': 2048, 'resnet101': 2048, 'resnet152': 2048, 213 | 'resnext50': 2048, 'resnext101': 2048, "efnb0": 1280, "efnb1": 1280, "efnb2": 1408, 214 | "efnb3": 1536, "efnb4": 1792, "efnb5": 2048, "efnb6": 2304, "efnb7": 2560} 215 | 216 | self.n_tab = hyp.n_tab # n tabular features 217 | self.attn_filters = attn_filters 218 | self.fc_dim = fc_dim 219 | self.n_attn_layers = n_attn_layers 220 | 221 | # efficient net b2 base 222 | if cnn in cnn_dict.keys(): 223 | self.ct_cnn = cnn_dict[cnn](pretrained = True) 224 | 225 | # make single channel 226 | self.ct_cnn.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) 227 | 228 | self.ct_cnn.avgpool = nn.Conv2d(self.out_dict[cnn], self.attn_filters, kernel_size=(1, 1), bias=False) 229 | self.ct_cnn.fc = nn.Identity() 230 | 231 | # 1 self attn layer [stacked] 232 | self.attn = nn.ModuleList() 233 | 234 | for _ in range(self.n_attn_layers): 235 | self.attn.append(Self_Attn(self.attn_filters)) 236 | elif 'efn' in cnn: 237 | if 'b0' in cnn: 238 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b0') 239 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 240 | bias = False, image_size = 512) 241 | elif 'b1' in cnn: 242 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b1') 243 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 244 | bias = False, image_size = 512) 245 | elif 'b2' in cnn: 246 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b2') 247 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 248 | bias = False, image_size = 512) 249 | elif 'b3' in cnn: 250 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b3') 251 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 40, kernel_size = (3,3), stride = (2,2), 252 | bias = False, image_size = 512) 253 | elif 'b4' in cnn: 254 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b4') 255 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 48, kernel_size = (3,3), stride = (2,2), 256 | bias = False, image_size = 512) 257 | # replace avg_pool layer 258 | # 1408 is the number of filters in last conv 259 | self.ct_cnn._avg_pooling = Conv2dStaticSamePadding(self.out_dict[ cnn.split('_')[0] ], self.attn_filters, kernel_size = (1,1), stride = (1,1), 260 | bias = False, image_size = 512) 261 | self.ct_cnn._dropout = nn.Identity() 262 | self.ct_cnn._fc = nn.Identity() 263 | self.ct_cnn._swish = nn.Identity() 264 | 265 | # 1 self attn layer [stacked] 266 | self.attn = nn.ModuleList() 267 | 268 | for _ in range(self.n_attn_layers): 269 | self.attn.append(Self_Attn(self.attn_filters)) 270 | 271 | 272 | else: 273 | raise ValueError("cnn not recognized") 274 | 275 | self.avgpool = nn.AdaptiveAvgPool2d((1,1)) 276 | 277 | self.dropout = nn.Dropout(p=0.25) 278 | 279 | self.fc_inter = nn.Linear(self.attn_filters + self.n_tab, self.fc_dim) 280 | 281 | self.fc = nn.Linear(self.fc_dim, 1) 282 | 283 | def forward(self, x_ct, x_tab): 284 | print(x_ct.shape) 285 | ct_f = self.ct_cnn(x_ct).view(-1, self.attn_filters, 16, 16) # ct features 286 | #print(ct_f.shape) 287 | 288 | for ii in range(len(self.attn)): 289 | ct_f = self.attn[ii](ct_f) 290 | #print(ct_f.shape) 291 | ct_f = self.avgpool(ct_f).view(-1, self.attn_filters) 292 | #print(ct_f.shape) 293 | # print(x_tab.shape) 294 | 295 | # concatenate 296 | x = torch.cat((ct_f, x_tab), -1) # concat on last axis 297 | 298 | # dropout 299 | if self.training: 300 | x = self.dropout(x) 301 | 302 | x = self.fc_inter(x) 303 | 304 | x = self.fc(x) 305 | 306 | return x 307 | 308 | from sklearn.model_selection import train_test_split 309 | from sklearn.metrics import mean_squared_error 310 | 311 | 312 | # score calculation 313 | 314 | def score(fvc_true, fvc_pred, sigma): 315 | sigma_clip = np.maximum(sigma, 70) 316 | delta = np.abs(fvc_true - fvc_pred) 317 | delta = np.minimum(delta, 1000) 318 | sq2 = np.sqrt(2) 319 | metric = (delta / sigma_clip)*sq2 + np.log(sigma_clip* sq2) 320 | return np.mean(metric) 321 | 322 | 323 | def score_avg(p, a): # patient id, predicted a 324 | percent_true = train.Percent.values[train.Patient == p] 325 | fvc_true = train.FVC.values[train.Patient == p] 326 | weeks_true = train.Weeks.values[train.Patient == p] 327 | 328 | fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 329 | percent = percent_true[0] - a * abs(weeks_true - weeks_true[0]) 330 | return score(fvc_true, fvc, percent) 331 | 332 | def rmse_avg(p, a): # patient id, predicted a 333 | percent_true = train.Percent.values[train.Patient == p] 334 | fvc_true = train.FVC.values[train.Patient == p] 335 | weeks_true = train.Weeks.values[train.Patient == p] 336 | 337 | fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 338 | return mean_squared_error(fvc_true, fvc, squared = False) 339 | 340 | 341 | # hyperparams 342 | 343 | result_dir = hyp.results_dir 344 | 345 | # training only resnet models on gpu 0 346 | train_models = hyp.train_models 347 | 348 | # 'resnext101' -> seems too heavy for 1080 349 | # 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4', 'efnb5', 'efnb6', 'efnb7' 350 | 351 | # device 352 | gpu = torch.device(f"cuda:{hyp.gpu_index}" if torch.cuda.is_available() else "cpu") 353 | 354 | 355 | nfold = hyp.nfold # hyper 356 | 357 | # removing noisy data 358 | P = [p for p in P if p not in ['ID00011637202177653955184', 'ID00052637202186188008618']] 359 | 360 | for model in train_models: 361 | for fd in hyp.fc_dim: 362 | for af in hyp.attn_filters: 363 | for nal in hyp.n_attn_layers: 364 | log = open(f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.txt", "a+") 365 | kfold =KFold(n_splits=nfold) 366 | 367 | ifold = 0 368 | for train_index, test_index in kfold.split(P): 369 | # print(train_index, test_index) 370 | 371 | p_train = np.array(P)[train_index] 372 | p_test = np.array(P)[test_index] 373 | 374 | osic_train = OSICData(p_train, A, TAB) 375 | train_loader = DataLoader(osic_train, batch_size=hyp.batch_size, shuffle=True, num_workers=hyp.num_workers) 376 | 377 | osic_val = OSICData(p_test, A, TAB) 378 | val_loader = DataLoader(osic_val, batch_size=hyp.batch_size, shuffle=True, num_workers=hyp.num_workers) 379 | 380 | 381 | tabct = TabCT(cnn = model, fc_dim = fd, attn_filters = af, n_attn_layers = nal).to(gpu) 382 | print(f"creating {model} with {fd} feature_dim, {af} attn_filters, and {nal} n_attn_layers") 383 | print(f"fold: {ifold}") 384 | log.write(f"fold: {ifold}\n") 385 | 386 | ifold += 1 387 | 388 | 389 | n_epochs = hyp.n_epochs # max 30 epochs, patience 5, find the suitable epoch number for later final training 390 | 391 | best_epoch = n_epochs # 30 392 | 393 | 394 | optimizer = torch.optim.AdamW(tabct.parameters()) 395 | criterion = torch.nn.L1Loss() 396 | 397 | max_score = 99999999.0000 # here, max score ]= minimum score 398 | 399 | for epoch in range(n_epochs): # loop over the dataset multiple times 400 | 401 | running_loss = 0.0 402 | tabct.train() 403 | for i, data in tqdm(enumerate(train_loader, 0)): 404 | 405 | [x, t], a, _ = data 406 | print(x.shape) 407 | print(t.shape) 408 | 409 | x = x.to(gpu) 410 | t = t.to(gpu) 411 | a = a.to(gpu) 412 | 413 | # zero the parameter gradients 414 | optimizer.zero_grad() 415 | 416 | # forward + backward + optimize 417 | outputs = tabct(x, t) 418 | loss = criterion(outputs, a) 419 | loss.backward() 420 | optimizer.step() 421 | 422 | # print statistics 423 | running_loss += loss.item() 424 | print(f"epoch {epoch+1} train: {running_loss}") 425 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 426 | 427 | running_loss = 0.0 428 | pred_a = {} 429 | tabct.eval() 430 | for i, data in tqdm(enumerate(val_loader, 0)): 431 | 432 | [x, t], a, pid = data 433 | 434 | x = x.to(gpu) 435 | t = t.to(gpu) 436 | a = a.to(gpu) 437 | 438 | # forward 439 | outputs = tabct(x, t) 440 | loss = criterion(outputs, a) 441 | 442 | pids = pid 443 | preds_a = outputs.detach().cpu().numpy().flatten() 444 | 445 | for j, p_d in enumerate(pids): 446 | pred_a[p_d] = preds_a[j] 447 | 448 | 449 | 450 | 451 | # print statistics 452 | running_loss += loss.item() 453 | print(f"epoch {epoch+1} val: {running_loss}") 454 | log.write(f"epoch {epoch+1} val: {running_loss}\n") 455 | # score calculation 456 | print(pred_a) 457 | print(len(pred_a)) 458 | print(p_test) 459 | print(len(p_test)) 460 | score_v = 0. 461 | rmse = 0. 462 | for p in p_test: 463 | score_v += (score_avg(p, pred_a[p]))/len(p_test) 464 | rmse += (rmse_avg(p, pred_a[p]))/len(p_test) 465 | 466 | print(f"val score: {score_v}") 467 | log.write(f"val score: {score_v}\n") 468 | log.write(f"val rmse: {rmse}\n") 469 | 470 | if score_v <= max_score: 471 | torch.save({ 472 | 'epoch': epoch, 473 | 'model_state_dict': tabct.state_dict(), 474 | 'optimizer_state_dict': optimizer.state_dict(), 475 | 'score': score_v 476 | }, f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.tar") 477 | max_score = score_v 478 | best_epoch = epoch + 1 479 | # destroy model 480 | del tabct 481 | torch.cuda.empty_cache() 482 | 483 | # final training with optimized setting 484 | 485 | osic_all = OSICData(P, A, TAB) 486 | all_loader = DataLoader(osic_all, batch_size=8, shuffle=True, num_workers=hyp.num_workers) 487 | 488 | # load the best model 489 | tabct = TabCT(cnn = model, fc_dim = fd, attn_filters = af, n_attn_layers = nal).to(gpu) 490 | tabct.load_state_dict(torch.load(f"{result_dir}/{model}_fd_{fd}_af_{af}_nal_{nal}.tar")["model_state_dict"]) 491 | 492 | optimizer = torch.optim.AdamW(tabct.parameters(), lr = hyp.final_lr) # very small learning rate 493 | criterion = torch.nn.L1Loss() 494 | 495 | print(f"Final training") 496 | log.write(f"Final training\n") 497 | for epoch in range(best_epoch + 2): # loop over the dataset multiple times 498 | 499 | running_loss = 0.0 500 | tabct.train() 501 | for i, data in tqdm(enumerate(all_loader, 0)): 502 | 503 | [x, t], a, _ = data 504 | 505 | x = x.to(gpu) 506 | t = t.to(gpu) 507 | a = a.to(gpu) 508 | 509 | # zero the parameter gradients 510 | optimizer.zero_grad() 511 | 512 | # forward + backward + optimize 513 | outputs = tabct(x, t) 514 | loss = criterion(outputs, a) 515 | loss.backward() 516 | optimizer.step() 517 | 518 | # print statistics 519 | running_loss += loss.item() 520 | print(f"epoch {epoch+1} train: {running_loss}") 521 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 522 | 523 | torch.save({ 524 | 'epoch': best_epoch, 525 | 'model_state_dict': tabct.state_dict(), 526 | 'optimizer_state_dict': optimizer.state_dict() 527 | }, f"{result_dir}/{model}.tar") 528 | 529 | print('Finished Training') 530 | # destroy model 531 | del tabct 532 | torch.cuda.empty_cache() 533 | 534 | 535 | 536 | 537 | # ref: https://www.kaggle.com/miklgr500/linear-decay-based-on-resnet-cnn 538 | # https://pytorch.org/docs/stable/index.html -------------------------------------------------------------------------------- /train_qreg.py: -------------------------------------------------------------------------------- 1 | # author: github/zabir-nabil 2 | 3 | # relevant imports 4 | 5 | import os 6 | import cv2 7 | 8 | import pydicom 9 | import pandas as pd 10 | import numpy as np 11 | import tensorflow as tf 12 | import matplotlib.pyplot as plt 13 | 14 | # torch dataset 15 | import torch 16 | from torch.utils.data import Dataset, DataLoader 17 | from torchvision import transforms, utils 18 | 19 | import random 20 | from tqdm import tqdm 21 | 22 | # k-fold 23 | from sklearn.model_selection import KFold 24 | 25 | # hyperparam object 26 | 27 | from config import HyperP 28 | 29 | hyp = HyperP(model_type = "qreg_train") # slope prediction 30 | 31 | # seed 32 | seed = hyp.seed 33 | random.seed(seed) 34 | os.environ['PYTHONHASHSEED'] = str(seed) 35 | np.random.seed(seed) 36 | tf.random.set_seed(seed) 37 | torch.manual_seed(seed) 38 | torch.backends.cudnn.deterministic = True 39 | torch.backends.cudnn.benchmark = False 40 | 41 | 42 | # path 43 | 44 | root_path = hyp.data_folder # ../input/osic-pulmonary-fibrosis-progression 45 | 46 | train = pd.read_csv(f'{root_path}/train.csv') 47 | train_vol = pd.read_csv(f'{hyp.ct_tab_feature_csv}') 48 | 49 | 50 | # getting base week for patient 51 | def get_baseline_week(data): 52 | df = data.copy() 53 | df['Weeks'] = df['Weeks'].astype(int) 54 | df['min_week'] = df.groupby('Patient')['Weeks'].transform('min') 55 | df['baseline_week'] = df['Weeks'] - df['min_week'] 56 | return df 57 | 58 | # getting FVC for base week and setting it as base_FVC of patient 59 | def get_base_FVC(data): 60 | df = data.copy() 61 | base = df.loc[df.Weeks == df.min_week][['Patient','FVC']].copy() 62 | base.columns = ['Patient','base_FVC'] 63 | 64 | base['nb']=1 65 | base['nb'] = base.groupby('Patient')['nb'].transform('cumsum') 66 | 67 | base = base[base.nb==1] 68 | base.drop('nb',axis =1,inplace=True) 69 | df = df.merge(base,on="Patient",how='left') 70 | df.drop(['min_week'], axis = 1) 71 | return df 72 | 73 | # getting Percent for base week and setting it as base_Percent of patient 74 | def get_base_Percent(data): 75 | df = data.copy() 76 | 77 | return df 78 | 79 | 80 | train['Volume'] = 2000. 81 | 82 | for i in range(len(train)): 83 | pid = train.iloc[i]['Patient'] 84 | try: 85 | train.at[i, 'Volume'] = train_vol[train_vol['Patient']==pid].iloc[0]['Volume'] 86 | except: 87 | print('bug at volume') 88 | 89 | 90 | # encoding temporal info 91 | train_data = train 92 | train_data.drop_duplicates(keep=False,inplace=True,subset=['Patient','Weeks']) 93 | train_data = get_baseline_week(train_data) 94 | train_data = get_base_FVC(train_data) 95 | 96 | 97 | 98 | 99 | # tabular feature generation 100 | 101 | def get_tab(df): 102 | # print(df) 103 | vector = [(df.Age.values[0] - train_data.Age.values.mean()) / train_data.Age.values.std()] # df.Age.values[0].mean(), df.Age.values[0].std() 104 | 105 | if df.Sex.values[0] == 'Male': 106 | vector.append(0) 107 | else: 108 | vector.append(1) 109 | 110 | if df.SmokingStatus.values[0] == 'Never smoked': 111 | vector.extend([0,0]) 112 | elif df.SmokingStatus.values[0] == 'Ex-smoker': 113 | vector.extend([1,1]) 114 | elif df.SmokingStatus.values[0] == 'Currently smokes': 115 | vector.extend([0,1]) 116 | else: 117 | vector.extend([1,0]) # this is useless 118 | 119 | vector.append((df.Volume.values[0] - train_data.Volume.values.mean()) / train_data.Volume.values.std()) 120 | 121 | vector.append((df.baseline_week.values[0] - train_data.baseline_week.values.mean()) / train_data.baseline_week.values.std()) 122 | 123 | vector.append((df.base_FVC.values[0] - train_data.base_FVC.values.mean()) / train_data.base_FVC.values.std()) 124 | 125 | return np.array(vector) 126 | 127 | 128 | 129 | def get_img(path): 130 | d = pydicom.dcmread(path) 131 | return cv2.resize(d.pixel_array / 2**11, (512, 512)) 132 | 133 | # torch dataset 134 | import torch 135 | from torch.utils.data import Dataset, DataLoader 136 | from torchvision import transforms, utils 137 | 138 | class OSICData(Dataset): 139 | BAD_ID = ['ID00011637202177653955184', 'ID00052637202186188008618'] 140 | def __init__(self, keys, train_df): 141 | self.keys = [k for k in keys if k not in self.BAD_ID] 142 | self.train_df = train_df 143 | 144 | self.train_data = {} 145 | for p in train.Patient.values: 146 | p_n = len(os.listdir(f'{root_path}/train/{p}/')) 147 | self.train_data[p] = os.listdir(f'{root_path}/train/{p}/')[int( hyp.strip_ct * p_n):-int( hyp.strip_ct * p_n)] # removing first and last 15% slices 148 | 149 | def __len__(self): 150 | return len(self.train_df) 151 | 152 | def __getitem__(self, idx): 153 | x = [] 154 | tab = [] 155 | all_features = list(self.train_df.iloc[idx]) 156 | pid = all_features[0] 157 | fvc = [] 158 | fvc.append(all_features[2]) 159 | # print(self.train_df.iloc[[idx]]) 160 | feature_set = get_tab(self.train_df.iloc[[idx]]) 161 | try: 162 | i = np.random.choice(self.train_data[pid], size=1)[0] 163 | img = get_img(f'{root_path}/train/{pid}/{i}') 164 | x.append(img) 165 | tab.append(feature_set) 166 | except Exception as e: 167 | print(e) 168 | print('error') 169 | x, tab, fvc = torch.tensor(x, dtype=torch.float32), torch.tensor(tab, dtype=torch.float32), torch.tensor(fvc, dtype=torch.float32) 170 | tab = torch.squeeze(tab, axis=0) 171 | return [x, tab], fvc, pid 172 | 173 | 174 | from torchvision import models 175 | from torch import nn 176 | from efficientnet_pytorch import EfficientNet 177 | from efficientnet_pytorch.utils import Conv2dStaticSamePadding 178 | 179 | class Identity(nn.Module): 180 | # credit: ptrblck 181 | def __init__(self): 182 | super(Identity, self).__init__() 183 | 184 | def forward(self, x): 185 | return x 186 | 187 | class TabCT(nn.Module): 188 | def __init__(self, cnn): 189 | super(TabCT, self).__init__() 190 | 191 | # CT features 192 | cnn_dict = {'resnet18': models.resnet18, 'resnet34': models.resnet34, 'resnet50': models.resnet50, 193 | 'resnet101': models.resnet101, 'resnet152': models.resnet152, 'resnext50': models.resnext50_32x4d, 194 | 'resnext101': models.resnext101_32x8d} 195 | 196 | # feature dim 197 | self.out_dict = {'resnet18': 512, 'resnet34': 512, 'resnet50': 2048, 'resnet101': 2048, 'resnet152': 2048, 198 | 'resnext50': 2048, 'resnext101': 2048, "efnb0": 1280, "efnb1": 1280, "efnb2": 1408, 199 | "efnb3": 1536, "efnb4": 1792, "efnb5": 2048, "efnb6": 2304, "efnb7": 2560} 200 | 201 | self.n_tab = hyp.n_tab # n tabular features 202 | 203 | # hashtable features 204 | self.cnn_features = {} # a map pid to feature 205 | 206 | # efficient net b0 to b7 207 | 208 | if cnn in cnn_dict.keys(): # resnet or resnext 209 | self.ct_cnn = cnn_dict[cnn](pretrained = True) 210 | 211 | # make single channel 212 | self.ct_cnn.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) 213 | 214 | # remove the fc layer/ add a simple linear layer 215 | self.ct_cnn.fc = nn.Linear(self.out_dict[cnn], 64) # mapped to 64 dimensions, Identity() 216 | 217 | elif cnn == "efnb0": 218 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b0') 219 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 220 | bias = False, image_size = 512) 221 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], 64) 222 | self.ct_cnn._swish = nn.Identity() 223 | 224 | elif cnn == "efnb1": 225 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b1') 226 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 227 | bias = False, image_size = 512) 228 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], 64) 229 | self.ct_cnn._swish = nn.Identity() 230 | 231 | elif cnn == "efnb2": 232 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b2') 233 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 32, kernel_size = (3,3), stride = (2,2), 234 | bias = False, image_size = 512) 235 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], 64) 236 | self.ct_cnn._swish = nn.Identity() 237 | 238 | elif cnn == "efnb3": 239 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b3') 240 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 40, kernel_size = (3,3), stride = (2,2), 241 | bias = False, image_size = 512) 242 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], 64) 243 | self.ct_cnn._swish = nn.Identity() 244 | 245 | elif cnn == "efnb4": 246 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b4') 247 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 48, kernel_size = (3,3), stride = (2,2), 248 | bias = False, image_size = 512) 249 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], 64) 250 | self.ct_cnn._swish = nn.Identity() 251 | 252 | elif cnn == "efnb5": 253 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b5') 254 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 48, kernel_size = (3,3), stride = (2,2), 255 | bias = False, image_size = 512) 256 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], 64) 257 | self.ct_cnn._swish = nn.Identity() 258 | 259 | elif cnn == "efnb6": 260 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b6') 261 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 56, kernel_size = (3,3), stride = (2,2), 262 | bias = False, image_size = 512) 263 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], 64) 264 | self.ct_cnn._swish = nn.Identity() 265 | 266 | elif cnn == "efnb7": 267 | self.ct_cnn = EfficientNet.from_pretrained('efficientnet-b7') 268 | self.ct_cnn._conv_stem = Conv2dStaticSamePadding(1, 64, kernel_size = (3,3), stride = (2,2), 269 | bias = False, image_size = 512) 270 | self.ct_cnn._fc = nn.Linear(self.out_dict[cnn], 64) 271 | self.ct_cnn._swish = nn.Identity() 272 | 273 | else: 274 | raise ValueError("cnn not recognized") 275 | 276 | self.dropout = nn.Dropout(p=0.2) 277 | 278 | self.fc1 = nn.Linear(64 + self.n_tab, 100) 279 | self.relu1 = nn.ReLU() 280 | 281 | self.fc2 = nn.Linear(100, 3) 282 | self.relu2 = nn.ReLU() # quantile adjustment 283 | 284 | 285 | def forward(self, x_ct, x_tab, pid = None): 286 | 287 | if self.training: 288 | ct_f = self.ct_cnn(x_ct) # ct features 289 | else: 290 | # hashtable trick for inference 291 | # print('inference mode') 292 | if pid in self.cnn_features: 293 | ct_f = self.cnn_features[pid] 294 | else: 295 | ct_f = self.ct_cnn(x_ct) 296 | self.cnn_features[pid] = ct_f 297 | 298 | 299 | # print(ct_f.shape) 300 | # print(x_tab.shape) 301 | 302 | # concatenate 303 | x = torch.cat((ct_f, x_tab), -1) # concat on last axis 304 | 305 | # dropout 306 | if self.training: 307 | x = self.dropout(x) 308 | 309 | x = self.fc1(x) 310 | x = self.relu1(x) 311 | 312 | x1 = self.fc2(x) 313 | x2 = self.relu2(x1) 314 | 315 | x_f = x1 + torch.cumsum(x2, dim=-1) 316 | 317 | return x_f 318 | 319 | from sklearn.model_selection import train_test_split 320 | from sklearn.metrics import mean_squared_error 321 | 322 | # tr_p, val_p = train_test_split(P, 323 | # shuffle=True, 324 | # train_size= 0.75) # 75% training 325 | 326 | 327 | 328 | 329 | # dummy = False 330 | 331 | # if dummy: 332 | # tr_p = tr_p[:4] 333 | # val_p = val_p[:4] 334 | 335 | # osic_train = OSICData(tr_p, A, TAB) 336 | # train_loader = DataLoader(osic_train, batch_size=4, shuffle=True, num_workers=4) 337 | 338 | # osic_val = OSICData(val_p, A, TAB) 339 | # val_loader = DataLoader(osic_val, batch_size=4, shuffle=True, num_workers=4) 340 | 341 | # all data 342 | # osic_all = OSICData(tr_p + val_p, A, TAB) 343 | # all_loader = DataLoader(osic_all, batch_size=4, shuffle=True, num_workers=4) 344 | 345 | # score calculation 346 | 347 | def score_np(fvc_true, fvc_pred, sigma): 348 | sigma_clip = np.maximum(sigma, 70) 349 | delta = np.abs(fvc_true - fvc_pred) 350 | delta = np.minimum(delta, 1000) 351 | sq2 = np.sqrt(2) 352 | metric = (delta / sigma_clip)*sq2 + np.log(sigma_clip* sq2) 353 | return np.mean(metric) 354 | 355 | 356 | #def score_avg(p, a): # patient id, predicted a 357 | # percent_true = train.Percent.values[train.Patient == p] 358 | # fvc_true = train.FVC.values[train.Patient == p] 359 | # weeks_true = train.Weeks.values[train.Patient == p] 360 | 361 | # fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 362 | # percent = percent_true[0] - a * abs(weeks_true - weeks_true[0]) 363 | # return score_np(fvc_true, fvc, percent) 364 | 365 | #def rmse_avg(p, a): # patient id, predicted a 366 | # percent_true = train.Percent.values[train.Patient == p] 367 | # fvc_true = train.FVC.values[train.Patient == p] 368 | # weeks_true = train.Weeks.values[train.Patient == p] 369 | 370 | # fvc = a * (weeks_true - weeks_true[0]) + fvc_true[0] 371 | # return mean_squared_error(fvc_true, fvc, squared = False) 372 | 373 | 374 | # hyperparams 375 | 376 | result_dir = hyp.results_dir 377 | 378 | # training only resnet models on gpu 0 379 | train_models = hyp.train_models 380 | 381 | # 'resnext101' -> seems too heavy for 1080 382 | # 'efnb0', 'efnb1', 'efnb2', 'efnb3', 'efnb4', 'efnb5', 'efnb6', 'efnb7' 383 | 384 | # device 385 | gpu = torch.device(f"cuda:{hyp.gpu_index}" if torch.cuda.is_available() else "cpu") 386 | 387 | 388 | nfold = hyp.nfold # hyper 389 | 390 | 391 | # loss function 392 | 393 | def score(outputs,target): 394 | confidence = outputs[:,2] - outputs[:,0] # output -> 0-> 20% percentile, 50% percentile, 80% percentile 395 | clip = torch.clamp(confidence,min=70) 396 | target=torch.reshape(target,outputs[:,1].shape) 397 | delta = torch.abs(outputs[:,1] - target) 398 | delta = torch.clamp(delta,max=1000) 399 | sqrt_2 = torch.sqrt(torch.tensor([2.])).to(gpu) 400 | metrics = (delta*sqrt_2/clip) + torch.log(clip*sqrt_2) 401 | return torch.mean(metrics) 402 | 403 | def qloss(outputs,target): 404 | qs = [0.2,0.5,0.8] 405 | qs = torch.tensor(qs,dtype=torch.float).to(gpu) 406 | e = outputs - target 407 | e = e.to(gpu) 408 | v = torch.max(qs*e,(qs-1)*e) 409 | return torch.mean(v) 410 | 411 | 412 | def hyb_loss(outputs,target,l): 413 | return l * qloss(outputs,target) + (1- l) * score(outputs,target) 414 | 415 | # need to edit from here 416 | 417 | # cut data 418 | if hyp.dummy_training: 419 | train_data = train_data.iloc[range(hyp.dummy_train_rows)] 420 | 421 | for model in train_models: 422 | log = open(f"{result_dir}/{model}.txt", "a+") 423 | kfold = KFold(n_splits=nfold) 424 | 425 | ifold = 0 426 | for train_index, test_index in kfold.split(train_data): 427 | # print(train_index, test_index) 428 | 429 | df_train = train_data.iloc[train_index] 430 | df_test = train_data.iloc[test_index] 431 | 432 | P = list(set(train_data['Patient'])) 433 | 434 | osic_train = OSICData(P, df_train) 435 | train_loader = DataLoader(osic_train, batch_size=hyp.batch_size, shuffle=True, num_workers=4) 436 | 437 | osic_val = OSICData(P, df_test) 438 | val_loader = DataLoader(osic_val, batch_size=hyp.batch_size, shuffle=True, num_workers=4) 439 | 440 | 441 | tabct = TabCT(cnn = model).to(gpu) 442 | print(f"creating {model}") 443 | print(f"fold: {ifold}") 444 | log.write(f"fold: {ifold}\n") 445 | 446 | ifold += 1 447 | 448 | 449 | n_epochs = hyp.n_epochs # max 30 epochs, patience 5, find the suitable epoch number for later final training 450 | 451 | best_epoch = n_epochs # 30 452 | 453 | 454 | optimizer = torch.optim.AdamW(tabct.parameters()) 455 | criterion = hyb_loss # torch.nn.L1Loss() 456 | 457 | max_score = 99999999.0000 # here, max score ]= minimum score 458 | 459 | for epoch in range(n_epochs): # loop over the dataset multiple times 460 | tabct.train() 461 | running_loss = 0.0 462 | for i, data in tqdm(enumerate(train_loader, 0)): 463 | 464 | [x, t], a, _ = data 465 | 466 | x = x.to(gpu) 467 | t = t.to(gpu) 468 | a = a.to(gpu) 469 | 470 | # zero the parameter gradients 471 | optimizer.zero_grad() 472 | 473 | # forward + backward + optimize 474 | outputs = tabct(x, t) 475 | loss = criterion(outputs, a, hyp.loss_weight) 476 | loss.backward() 477 | optimizer.step() 478 | 479 | # print statistics 480 | running_loss += loss.item() 481 | print(f"epoch {epoch+1} train: {running_loss}") 482 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 483 | 484 | running_loss = 0.0 485 | score_avg = 0.0 486 | rmse_avg = 0.0 487 | n_batch = 0 488 | tabct.eval() 489 | for i, data in tqdm(enumerate(val_loader, 0)): 490 | 491 | [x, t], a, pid = data 492 | 493 | x = x.to(gpu) 494 | t = t.to(gpu) 495 | a = a.to(gpu) 496 | 497 | # forward 498 | outputs = tabct(x, t, pid) 499 | loss = criterion(outputs, a, hyp.loss_weight) 500 | 501 | preds_fvc = outputs # .detach().cpu().numpy() # .flatten() 502 | true_fvc = a # .detach().cpu().numpy() # .flatten() 503 | 504 | score_avg += score(preds_fvc, true_fvc) 505 | 506 | 507 | preds_fvc = outputs[:,1].detach().cpu().numpy().flatten() 508 | true_fvc = a.detach().cpu().numpy().flatten() 509 | 510 | rmse_avg += mean_squared_error(true_fvc, preds_fvc, squared = False) 511 | n_batch += 1 512 | 513 | 514 | # print statistics 515 | running_loss += loss.item() 516 | print(f"epoch {epoch+1} val: {running_loss}") 517 | log.write(f"epoch {epoch+1} val: {running_loss}\n") 518 | # score calculation 519 | 520 | score_v = score_avg / n_batch 521 | rmse = rmse_avg / n_batch 522 | 523 | print(f"val score: {score_v}") 524 | log.write(f"val score: {score_v}\n") 525 | log.write(f"val rmse: {rmse}\n") 526 | 527 | if score_v <= max_score: 528 | torch.save({ 529 | 'epoch': epoch, 530 | 'model_state_dict': tabct.state_dict(), 531 | 'optimizer_state_dict': optimizer.state_dict(), 532 | 'score': score_v 533 | }, f"{result_dir}/{model}.tar") 534 | max_score = score_v 535 | best_epoch = epoch + 1 536 | # destroy model 537 | del tabct 538 | torch.cuda.empty_cache() 539 | 540 | # final training with optimized setting 541 | P = list(set(train_data['Patient'])) 542 | osic_all = OSICData(P, train_data) 543 | all_loader = DataLoader(osic_all, batch_size=hyp.batch_size, shuffle=True, num_workers=4) 544 | 545 | # load the best model 546 | tabct = TabCT(cnn = model).to(gpu) 547 | tabct.load_state_dict(torch.load(f"{result_dir}/{model}.tar")["model_state_dict"]) 548 | 549 | optimizer = torch.optim.AdamW(tabct.parameters(), lr = hyp.final_lr) # very small learning rate 550 | criterion = hyb_loss 551 | 552 | print(f"Final training") 553 | log.write(f"Final training\n") 554 | 555 | tabct.train() 556 | for epoch in range(best_epoch + 2): # loop over the dataset multiple times 557 | 558 | running_loss = 0.0 559 | for i, data in tqdm(enumerate(all_loader, 0)): 560 | 561 | [x, t], a, _ = data 562 | 563 | x = x.to(gpu) 564 | t = t.to(gpu) 565 | a = a.to(gpu) 566 | 567 | # zero the parameter gradients 568 | optimizer.zero_grad() 569 | 570 | # forward + backward + optimize 571 | outputs = tabct(x, t) 572 | loss = criterion(outputs, a, hyp.loss_weight) 573 | loss.backward() 574 | optimizer.step() 575 | 576 | # print statistics 577 | running_loss += loss.item() 578 | print(f"epoch {epoch+1} train: {running_loss}") 579 | log.write(f"epoch {epoch+1} train: {running_loss}\n") 580 | 581 | torch.save({ 582 | 'epoch': best_epoch, 583 | 'model_state_dict': tabct.state_dict(), 584 | 'optimizer_state_dict': optimizer.state_dict() 585 | }, f"{result_dir}/{model}.tar") 586 | 587 | print('Finished Training') 588 | # destroy model 589 | del tabct 590 | torch.cuda.empty_cache() 591 | 592 | 593 | 594 | 595 | # ref: https://www.kaggle.com/miklgr500/linear-decay-based-on-resnet-cnn 596 | # https://www.kaggle.com/furcifer/q-regression-with-ct-tabular-features-pytorch 597 | # https://pytorch.org/docs/stable/index.html -------------------------------------------------------------------------------- /train_data_ct_tab.csv: -------------------------------------------------------------------------------- 1 | Patient,Weeks,FVC,Percent,Age,Sex,SmokingStatus,Volume,Mean,Skew,Kurthosis 2 | ID00129637202219868188000,0,2253,59.622102254684,71,Male,Never smoked,1897.6417038549116,89.99672951559745,1.7913186188673669,4.466360008117333 3 | ID00210637202257228694086,0,2846,80.54109123839709,76,Male,Currently smokes,3116.3087242749993,67.60080365445162,1.7169436855454827,3.8960369584839185 4 | ID00276637202271694539978,6,3107,87.14798608773701,77,Male,Ex-smoker,3257.095616455078,129.74950536757711,3.05397697896124,10.481746462632685 5 | ID00099637202206203080121,44,2934,82.2125084061869,68,Male,Ex-smoker,2785.3919953918457,52.23310935722411,1.8467758160928458,4.2629158674422705 6 | ID00342637202287526592911,23,2390,67.8514649102885,72,Male,Ex-smoker,2631.11572265625,105.40277451477128,2.978316781555375,12.661993183051475 7 | ID00119637202215426335765,2,2917,66.7017287112412,57,Male,Ex-smoker,5873.928417587281,71.38552426823675,2.001692730815426,3.8236063551672146 8 | ID00339637202287377736231,0,3574,81.897341888176,64,Male,Ex-smoker,830.6608337402344,96.94107906391588,3.653741612533852,16.3261423710895 9 | ID00124637202217596410344,47,3559,88.8151327610301,60,Male,Ex-smoker,4936.39841944305,68.06775509740186,1.8905147745159732,4.355099294108881 10 | ID00073637202198167792918,3,2547,70.74213976224871,69,Male,Ex-smoker,3298.8904115638734,81.17865189439384,2.3035967035067206,7.021256671552095 11 | ID00228637202259965313869,14,1885,85.57679211876341,58,Female,Ex-smoker,2750.319318237344,61.70825495576166,1.5134760454379628,2.1772126487360586 12 | ID00290637202279304677843,4,2283,67.4087634345104,75,Male,Never smoked,3158.5138549804688,56.18490833386316,1.6884215839570658,2.803759931823544 13 | ID00340637202287399835821,-1,2345,59.0382678751259,68,Male,Ex-smoker,3095.9053417968753,83.55259653079855,2.0459692267557767,5.6271799564336895 14 | ID00125637202218590429387,8,2349,53.757781032588795,65,Male,Never smoked,3254.8555523902937,84.24511714233431,2.235898887536806,6.542812332230616 15 | ID00291637202279398396106,2,4051,100.25242526232401,72,Male,Ex-smoker,5139.5655867953765,75.13334364306786,1.7777398648296254,3.627897807032652 16 | ID00130637202220059448013,11,1690,69.852029428784,65,Female,Never smoked,2204.98363571167,73.70054309629528,2.5970142715362785,7.360682371618742 17 | ID00405637202308359492977,0,1560,61.8213521439328,61,Female,Never smoked,1311.2929757812526,111.30920698924731,1.9229221803054357,3.5800084421177374 18 | ID00341637202287410878488,28,2849,143.381982888777,68,Female,Currently smokes,584.7870719146729,72.9314935341938,2.8154199259654935,8.944286069294858 19 | ID00232637202260377586117,9,1909,115.431128310558,74,Female,Never smoked,4123.46887193,70.13847141411442,3.0872864889740823,11.825533739478146 20 | ID00242637202264759739921,4,1853,54.7447411959348,64,Male,Ex-smoker,4496.3470458984375,91.82267103905525,1.422024413349134,1.6218139629585977 21 | ID00323637202285211956970,26,1412,59.148793565683604,77,Male,Ex-smoker,3390.122545513153,78.10229987557285,2.087973828387001,5.382693958569201 22 | ID00213637202257692916109,6,3035,83.5627753303965,70,Male,Currently smokes,1892.2825856208801,66.50580446548523,1.5840151906634738,2.048565574318303 23 | ID00393637202302431697467,0,3562,89.09454727363679,67,Male,Never smoked,4748.885408779224,91.02838793671052,1.2434530684368201,1.0529107272657372 24 | ID00199637202248141386743,5,2672,64.7286821705426,62,Male,Ex-smoker,6447.825244000001,91.74504561335405,1.737393118136277,3.721981642185807 25 | ID00042637202184406822975,1,3791,103.69256017505501,58,Male,Ex-smoker,3632.36853515625,69.65846417016398,2.232864346622866,7.857945070922996 26 | ID00140637202231728595149,16,4124,90.09087731562391,58,Male,Currently smokes,5522.728065593719,106.78264876525189,2.7364696637095123,8.131731836648475 27 | ID00368637202296470751086,7,3128,83.1472620946305,65,Male,Never smoked,3562.3889099121093,72.90640259305988,1.4178220312327836,1.796451893475302 28 | ID00222637202259066229764,-1,2644,62.8326996197719,70,Male,Ex-smoker,695.997303073883,84.54301306027347,1.5642817913117293,3.095172345397212 29 | ID00126637202218610655908,18,2375,59.375,78,Male,Ex-smoker,3575.8330993652344,100.39502952822413,1.6134366746348334,3.2595583154374976 30 | ID00423637202312137826377,17,3294,79.2589027911453,72,Male,Ex-smoker,3290.1859188079834,82.74627422615927,3.6922604557577037,17.11259665484609 31 | ID00279637202272164826258,8,4141,90.9390372452565,70,Male,Ex-smoker,5454.085580951454,91.9765506703361,1.8496077957677235,3.260612059529425 32 | ID00329637202285906759848,39,2805,66.2494095418044,69,Male,Ex-smoker,2917.6396985054016,65.29219427948964,1.697889913240049,3.271206736415298 33 | ID00020637202178344345685,18,2297,117.77071369975401,66,Female,Never smoked,2889.532421875,69.74758486648243,2.0425870809669635,4.770676274154575 34 | ID00248637202266698862378,14,1272,79.55469385202329,71,Female,Never smoked,3486.262939453125,77.4969435072489,2.2927720866246437,6.145135018342245 35 | ID00196637202246668775836,9,3849,89.9215026633025,65,Male,Ex-smoker,4623.371329307556,75.75918726741095,1.7482808292601522,3.4407998223193363 36 | ID00264637202270643353440,13,1938,114.647420728822,76,Female,Never smoked,1897.0123291015625,65.7171896277803,2.4232892154580687,7.305288376141528 37 | ID00168637202237852027833,7,3186,79.3959330143541,62,Male,Ex-smoker,3718.1548476219177,94.45839925505327,2.9357273974293174,9.983544835839027 38 | ID00014637202177757139317,0,3807,90.076660988075,56,Male,Ex-smoker,4444.618992891654,83.04074032525631,1.9985078640480438,4.801028497100305 39 | ID00355637202295106567614,24,4418,141.22235008311,65,Male,Currently smokes,4371.233284260022,110.48004429241766,1.4309298871728569,1.738197820093209 40 | ID00225637202259339837603,13,1583,80.1721954925298,77,Female,Never smoked,1738.6570672874452,65.94689906203612,2.5177828946468654,7.177022354951173 41 | ID00364637202296074419422,37,3191,91.17142857142859,64,Male,Ex-smoker,3544.1252994537354,85.61267898767899,1.2336933987462568,1.0991272432125436 42 | ID00038637202182690843176,6,3946,97.0296055867021,71,Male,Ex-smoker,5609.444112019087,88.41849309876665,1.1748790248264,1.5426638347185984 43 | ID00026637202179561894768,41,2884,79.57179119302509,57,Male,Ex-smoker,2564.2632806633424,2.8229166666666665,0.0,-3.0 44 | ID00207637202252526380974,33,2379,65.6022501654534,67,Male,Ex-smoker,3826.9404421600007,106.29320804303285,1.6220428839894971,3.076131807471495 45 | ID00202637202249376026949,20,4311,107.281505076647,64,Male,Never smoked,4507.449292266608,93.46286571576366,1.3355596103132272,1.267744448389089 46 | ID00089637202204675567570,7,2478,57.967624216337605,63,Male,Never smoked,3111.671459668961,96.91982893549871,3.0315112090105742,10.777572224871053 47 | ID00052637202186188008618,48,2489,58.969863532979495,54,Male,Ex-smoker,3209.770190429688,102.59473302971003,3.3026461913567844,12.26170436516124 48 | ID00307637202282126172865,3,3170,81.76003301351491,65,Male,Never smoked,3895.9475579833984,104.40099632110993,1.312627270507037,1.5808582384999754 49 | ID00383637202300493233675,48,3237,114.864625102019,64,Female,Never smoked,4149.926279296875,68.42106666666666,1.3590278024748441,1.3556151518858668 50 | ID00365637202296085035729,35,2421,71.72058300746541,71,Male,Ex-smoker,3677.0787767601014,70.17839468217792,2.0196355966126616,5.3168821777913085 51 | ID00376637202297677828573,39,4284,119.664804469274,72,Male,Never smoked,5224.624884878448,106.48145031581358,1.5563359261585707,2.398970383567307 52 | ID00093637202205278167493,-1,3695,84.95815322358129,69,Male,Ex-smoker,7852.265251274109,77.38117391375393,1.7225298367536785,2.896377903557876 53 | ID00285637202278913507108,4,1411,66.4093754412388,61,Female,Never smoked,2007.7178214245632,75.30090265310194,0.9557216190908462,0.11973695282081342 54 | ID00030637202181211009029,19,2506,59.18752952290979,69,Male,Ex-smoker,2486.774876109254,78.5127182383078,1.1593225443765407,1.4482255076743336 55 | ID00128637202219474716089,34,2220,96.9263010827803,87,Female,Never smoked,361.33026562773136,16.53846153846154,0.0,-3.0 56 | ID00288637202279148973731,21,2276,78.7979504223792,63,Female,Ex-smoker,3743.9105080010604,76.95529109750092,1.8225746179188822,3.624089718770949 57 | ID00180637202240177410333,9,3009,85.69719753930279,68,Male,Ex-smoker,4436.536263354,92.45364922781026,1.7952225469712826,3.2341619984886947 58 | ID00061637202188184085559,23,3969,108.501913613997,68,Male,Ex-smoker,5381.3843307675,83.2220475341175,1.3707431125812355,1.6239188269274685 59 | ID00117637202212360228007,4,2933,79.614549402823,68,Male,Currently smokes,4076.6717614746094,126.69636222571903,2.3158720909323502,5.102567954034237 60 | ID00401637202305320178010,42,2184,80.38277511961721,74,Female,Ex-smoker,2829.5667114257812,93.74791948380398,1.8150334272057993,3.523753578865435 61 | ID00183637202241995351650,39,2541,70.46589018302832,71,Male,Ex-smoker,523.7139591283757,102.76262742796229,1.673051884148811,3.6528351176387464 62 | ID00173637202238329754031,11,2648,77.003605909038,73,Male,Ex-smoker,0.0,65.26718746889549,1.8040517804354668,3.0999690578716788 63 | ID00010637202177584971671,0,3523,94.72467197246719,60,Male,Ex-smoker,2160.438276100159,70.77731913867176,1.4210291438813305,2.1939164891376013 64 | ID00169637202238024117706,12,2520,106.445890005914,66,Female,Never smoked,4616.7419787597655,92.49148210891319,1.0796487152458207,0.537049306102785 65 | ID00077637202199102000916,5,3170,82.07332228666121,70,Male,Ex-smoker,3788.112391662598,96.68786641268143,1.472619781433901,2.4372730434136463 66 | ID00229637202260254240583,23,3518,78.7024608501119,71,Male,Ex-smoker,4482.14069704,78.07481154313389,2.211580285095738,6.50837112221004 67 | ID00023637202179104603099,-3,1536,65.3061224489796,71,Female,Ex-smoker,2306.712242126465,87.2276521475186,1.7129994386025567,3.7914287155324455 68 | ID00378637202298597306391,25,3338,77.9178338001867,56,Male,Never smoked,4455.743578552247,83.93197601210304,0.9865595934255527,0.47139421441220497 69 | ID00019637202178323708467,13,2100,92.8587220871103,83,Female,Ex-smoker,3636.4721003699997,79.7061623359108,1.7106575596573406,3.1900375890920865 70 | ID00343637202287577133798,23,1637,71.2637673588438,68,Female,Never smoked,2765.6472656250003,76.1524541193557,2.7710341694919784,10.451302536016119 71 | ID00319637202283897208687,12,4143,121.595444940127,72,Male,Ex-smoker,5193.176833190918,91.21769024206215,1.8478155275416928,4.010365168695162 72 | ID00090637202204766623410,17,3562,95.0576430401367,69,Male,Ex-smoker,4412.298583984375,65.97312605934883,2.762086304280557,14.303059199477193 73 | ID00411637202309374271828,7,1556,43.352279059400395,65,Male,Ex-smoker,3032.8912353515625,96.16088117297141,2.2014343321239713,4.798730090834905 74 | ID00192637202245493238298,4,1728,56.279312141740505,56,Female,Never smoked,2063.244905090332,69.6099873323355,4.8673836730409015,50.360141434549504 75 | ID00167637202237397919352,7,4562,92.6633084172896,58,Male,Ex-smoker,5175.395233154297,84.68883117085072,2.3799706656588966,7.546753193567104 76 | ID00062637202188654068490,44,2301,76.4553429027113,74,Male,Never smoked,270.8906913877043,81.17601885945811,1.4860464675907459,2.3909393773932885 77 | ID00108637202209619669361,11,2914,82.5963718820862,73,Male,Ex-smoker,5409.9305419921875,81.23375877898634,1.6226834401211203,2.2321432198096396 78 | ID00047637202184938901501,2,3313,89.92942453854499,68,Male,Ex-smoker,4187.6943514823915,67.77401144805235,1.3632468599994512,1.8578628108034172 79 | ID00105637202208831864134,44,2479,66.4539995710916,64,Male,Ex-smoker,4028.0815758519107,50.151602466889095,2.262790243621663,5.680761505772301 80 | ID00072637202198161894406,15,2359,72.3530855109803,71,Male,Ex-smoker,3286.3926541299998,82.62137080020263,1.639637127886578,2.9273206224376507 81 | ID00255637202267923028520,1,1563,64.88708070408511,62,Female,Never smoked,1671.7091566467286,67.5151886455302,1.5855415089622484,2.450957749033509 82 | ID00337637202286839091062,29,3959,96.1202291929688,69,Male,Ex-smoker,5363.162303466797,74.22175640667322,1.6920230424865694,2.9137821881665067 83 | ID00371637202296828615743,6,2470,57.0596932175199,71,Male,Ex-smoker,1902.583560937527,59.378424124136885,2.2465791085923406,7.17635164315595 84 | ID00109637202210454292264,53,2073,53.956272774596606,73,Male,Ex-smoker,3065.625596375569,58.31346970209249,2.3413875543880747,6.4256230619274355 85 | ID00134637202223873059688,2,2629,67.7088698877099,67,Male,Ex-smoker,3608.895182725851,83.72685680658127,1.5802907532010282,2.225543105742343 86 | ID00392637202302319160044,5,2231,55.4423459244533,66,Male,Ex-smoker,2672.8625774383545,119.64357562908069,1.1586500434842342,0.8387931142864056 87 | ID00296637202279895784347,5,2223,66.0035629453682,58,Male,Ex-smoker,2207.7992305620187,80.9221613208824,2.417711628102664,7.267141915062828 88 | ID00221637202258717315571,11,1401,75.016063396873,76,Female,Ex-smoker,2565.001706542969,52.31313225377518,1.7356520852058264,3.053260106190578 89 | ID00283637202278714365037,0,1965,54.52275249722529,60,Male,Never smoked,5255.6525768280035,65.02429130400064,1.549469180479602,2.46145962341285 90 | ID00027637202179689871102,12,2472,64.3414888079125,73,Male,Ex-smoker,8883.359026263774,100.47885365656914,1.9143016833627768,3.130974763952616 91 | ID00241637202264294508775,2,1399,78.8969095420708,66,Female,Never smoked,2454.457979176,77.39070247568914,2.597770088172517,13.145405950987517 92 | ID00094637202205333947361,0,4916,117.29337659858801,64,Male,Ex-smoker,4621.049011230469,117.01213901167283,2.7290599751175453,10.096336792699034 93 | ID00032637202181710233084,30,5045,119.629137816561,63,Male,Ex-smoker,3582.057734645251,77.71931269159003,1.76759218383827,2.6082307391064488 94 | ID00127637202219096738943,44,1677,65.8550952287454,55,Female,Ex-smoker,508.48632354736327,127.93337461726357,1.4538765054844616,1.3830746695123688 95 | ID00358637202295388077032,4,2030,48.0723690442361,65,Male,Ex-smoker,2786.618885635084,47.275861978867624,1.6459927916102768,3.571290924795444 96 | ID00132637202222178761324,6,2582,60.1640413831671,69,Male,Ex-smoker,39.51620224681032,36.300292853936426,0.18466934555927983,-1.486132218800967 97 | ID00331637202286306023714,28,3135,94.64436662238859,69,Male,Currently smokes,3516.2934646606445,57.338714970138476,2.3076096839077134,6.987938802199684 98 | ID00426637202313170790466,0,2925,71.824968077792,73,Male,Never smoked,3684.04007625316,81.2158087356019,1.4304856174402765,1.530935844909525 99 | ID00007637202177411956430,-4,2315,58.2536487166583,79,Male,Ex-smoker,2847.190090143611,124.02855267476208,1.4820187951948294,2.5908891932142026 100 | ID00419637202311204720264,6,3020,70.18685507111651,73,Male,Ex-smoker,5553.8773543087045,93.16975869805353,2.8101299700419884,15.489320345553526 101 | ID00025637202179541264076,0,2903,73.7775744637593,65,Male,Ex-smoker,3690.6593101212547,95.35355674964951,2.9492184323167465,11.077300621724575 102 | ID00414637202310318891556,7,2619,69.61722488038279,65,Male,Never smoked,3254.4914219591915,95.61204871966653,2.2573877875516284,5.414957492652739 103 | ID00298637202280361773446,4,1880,107.600732600733,67,Female,Never smoked,2978.7109375,71.5751565855605,1.3035754355415863,2.2101039480433773 104 | ID00240637202264138860065,5,2991,78.4216046145779,63,Male,Ex-smoker,4331.791008021301,79.16625905440516,1.7949288485945227,3.352861830506197 105 | ID00015637202177877247924,12,3042,78.64529472595659,71,Male,Ex-smoker,3590.4281828073263,69.58501407112831,1.4132340343294132,1.4642784203058445 106 | ID00224637202259281193413,0,2992,64.0082149580695,72,Male,Never smoked,772.2570574188233,84.9758363014177,1.933288288312092,4.796052325311556 107 | ID00110637202210673668310,15,1995,62.6885369532429,74,Male,Ex-smoker,3088.556531046842,76.27907649493143,1.5637565115450593,2.7064758965447933 108 | ID00273637202271319294586,4,3020,79.9618724846431,60,Male,Ex-smoker,4030.3313443756106,93.2171878412991,1.7767723707928527,3.1214458725756105 109 | ID00138637202231603868088,14,4510,108.95825280247401,66,Male,Ex-smoker,5683.506870162964,68.23876737523022,1.68978964259488,3.172429667829495 110 | ID00322637202284842245491,3,2460,60.134936931651495,59,Male,Ex-smoker,3451.2750725588267,83.63277594898462,1.6790708408384531,2.813100886730382 111 | ID00381637202299644114027,5,2415,56.914592760180994,62,Male,Ex-smoker,2631.6614268634694,61.46699178158017,4.701388592672017,38.76590901550264 112 | ID00367637202296290303449,35,1366,55.750550975430606,57,Female,Never smoked,1984.174260370826,104.91295149772981,1.4535269029820026,1.5103533493091437 113 | ID00104637202208063407045,20,2377,68.4935454126326,74,Male,Never smoked,3787.4923961875006,81.82356326985892,1.3813499762805503,1.736031395992617 114 | ID00218637202258156844710,20,2439,78.94225789746241,81,Male,Ex-smoker,4485.392472381592,73.61737399185967,1.6670228897581894,2.2693519805879783 115 | ID00133637202223847701934,-2,3195,92.85631248546851,83,Male,Never smoked,644.3491825522319,105.72308589840343,2.273751615493266,5.2486507399147655 116 | ID00299637202280383305867,22,2109,77.1735948477752,78,Male,Ex-smoker,2115.0208560180663,84.47228760890472,1.914478218788479,4.256006947822273 117 | ID00139637202231703564336,16,3793,100.77045696068001,76,Male,Ex-smoker,5752.862152099609,82.08978301333536,1.3786025842537457,1.7530288999804862 118 | ID00267637202270790561585,3,1015,50.158134018580796,70,Female,Never smoked,1965.29296875,57.060732504440494,2.4257294670196745,9.836384345857683 119 | ID00214637202257820847190,3,2869,83.70288248337029,69,Male,Ex-smoker,4686.2039794921875,94.69753593631464,1.5333382394826416,2.5653186202978624 120 | ID00233637202260580149633,-3,3829,100.78437565803301,68,Male,Ex-smoker,2764.928918546763,74.37693694545622,1.615113556684936,2.3017355559271113 121 | ID00068637202190879923934,11,2794,82.63338459718442,73,Male,Ex-smoker,3464.3975830078125,69.93077719044983,1.9834144316030753,5.884071230887365 122 | ID00076637202199015035026,-4,2298,52.749977045266704,51,Male,Never smoked,4470.129686550001,74.7234345633297,1.3221640350204555,1.924848653117345 123 | ID00102637202206574119190,3,2553,68.6437943643794,60,Male,Ex-smoker,3242.1882705688477,83.79891380365417,1.3575928056554132,1.5593296728290653 124 | ID00172637202238316925179,35,2603,70.9418946909408,73,Male,Ex-smoker,3648.45071758,68.44398490983596,2.339352865487938,7.458543330530773 125 | ID00312637202282607344793,0,1879,97.4939033881596,72,Female,Ex-smoker,3504.421517753318,80.37415545145579,1.3573499583495576,1.3309679452303902 126 | ID00216637202257988213445,10,2658,70.65390749601279,65,Male,Never smoked,3293.997802734375,71.34569259271512,2.101754029771628,5.267072041281104 127 | ID00249637202266730854017,34,3255,92.0739986422268,56,Male,Ex-smoker,1841.3802971424261,84.38888939710691,1.2870481866096461,1.9882247010452208 128 | ID00400637202305055099402,29,3962,88.38226108682069,55,Male,Ex-smoker,5289.624597926327,77.61955261603038,1.8553596048060415,3.7675695958504534 129 | ID00115637202211874187958,15,2548,77.74929818137441,77,Male,Ex-smoker,3300.8058909669667,102.42677458976911,1.3737268536458962,1.4827770063724515 130 | ID00012637202177665765362,33,3418,93.7260063617418,65,Male,Never smoked,3604.3054687500003,92.0575910034602,2.2070810277450637,7.372660000171223 131 | ID00335637202286784464927,3,3043,82.22546476437529,74,Male,Ex-smoker,4198.113499319999,93.58719659345272,2.3844282434167714,7.9907635884376145 132 | ID00275637202271440119890,48,2080,82.6807647970744,62,Female,Never smoked,2730.759392517,71.73202001787344,1.19448198025975,1.7879227086128342 133 | ID00060637202187965290703,16,3255,85.34347142108021,63,Male,Ex-smoker,3172.303983581543,66.00001182494678,1.7265976587289347,2.653264557487428 134 | ID00417637202310901214011,8,3357,82.2471579772638,66,Male,Never smoked,4976.455259857178,58.03486931130262,1.8496004006173121,4.784587275020932 135 | ID00123637202217151272140,48,1674,57.8978314253104,69,Female,Never smoked,1711.7969997711182,72.89656061526338,1.4882521970465135,2.088921284831332 136 | ID00234637202261078001846,3,2080,57.445868316394204,66,Male,Ex-smoker,2575.748357000001,108.16843577614958,1.7837969041389794,4.287259152662612 137 | ID00388637202301028491611,0,3157,79.74638779428109,53,Male,Ex-smoker,3714.9165796018997,65.55413003347759,1.3516978287436356,1.4467637684226178 138 | ID00336637202286801879145,6,2569,65.2295348364818,56,Male,Ex-smoker,3207.4705855178836,98.97204169154767,1.5005051862948369,2.4608621868178284 139 | ID00009637202177434476278,8,3660,85.28287818063191,69,Male,Ex-smoker,4323.567561097534,148.58305839266825,2.760336393392726,6.879779679292341 140 | ID00186637202242472088675,8,3096,77.3304026376262,69,Male,Ex-smoker,3494.152456884611,93.01091790509973,1.3971899923559792,1.6546293131431407 141 | ID00135637202224630271439,31,3808,106.09606597570499,65,Male,Never smoked,4933.291601721319,74.23174888958836,2.1333893901999708,6.04798423751061 142 | ID00082637202201836229724,19,2918,99.79480164158691,49,Female,Currently smokes,4989.797314453125,92.68459360540123,1.3526265532105557,1.5234094084864465 143 | ID00184637202242062969203,29,3130,67.3118279569892,52,Male,Ex-smoker,4628.704601287842,58.984420067549365,1.5442775075882171,2.0937340948804346 144 | ID00422637202311677017371,6,1930,76.6724932464643,73,Male,Ex-smoker,2409.8008287838493,85.60367094151863,1.2764191314464526,0.9996845936367373 145 | ID00251637202267455595113,21,2075,62.18532725965,88,Male,Ex-smoker,2774.439239501953,104.90197121187964,1.8829752404060307,4.94791612872204 146 | ID00048637202185016727717,9,1375,60.0594042107102,70,Female,Never smoked,1008.613125,83.57866457337704,2.5574595470513795,8.829435880255348 147 | ID00086637202203494931510,-5,3367,117.62856344326401,65,Female,Never smoked,5952.1142305501635,108.7002564682809,1.5706219482152515,2.7051552932912513 148 | ID00078637202199415319443,0,1860,68.2894591915409,55,Female,Ex-smoker,9722.524652380094,88.11801173318851,7.73022552468764,171.77363885477965 149 | ID00344637202287684217717,0,2066,53.95946510656079,58,Male,Never smoked,2586.683006213858,81.50389156421987,1.2974139515961116,1.310149553026557 150 | ID00317637202283194142136,26,2375,57.45597058254311,64,Male,Ex-smoker,2914.6192498876444,79.46278869169261,1.7591840761074542,3.054749074543409 151 | ID00161637202235731948764,8,2841,68.29983652274261,63,Male,Ex-smoker,3789.6585546875,104.06512678679368,3.5871461069248247,14.91453170546679 152 | ID00131637202220424084844,63,3327,69.49202105439049,61,Male,Never smoked,4171.691418089999,80.90078730312031,1.769407447154241,3.4432698036470857 153 | ID00011637202177653955184,6,3326,85.9875904860393,72,Male,Ex-smoker,4221.476861572266,92.5084137327917,1.5906081605916707,2.975898853695166 154 | ID00408637202308839708961,13,2651,97.40593768371549,74,Male,Ex-smoker,4126.141604454041,65.31147028202292,1.867556571604169,4.279537358122362 155 | ID00351637202289476567312,14,2808,82.0093457943925,78,Male,Ex-smoker,3750.7226504576815,89.91989083672333,1.7369748169221835,3.3413963902017256 156 | ID00165637202237320314458,79,2102,53.80362444967749,54,Male,Ex-smoker,2636.8116796739837,70.02646841509903,2.044187264033753,5.077123118567631 157 | ID00035637202182204917484,11,2728,74.5762711864407,69,Male,Ex-smoker,3213.5128038135,76.12692389275914,1.583248539591013,1.9906169825129787 158 | ID00149637202232704462834,44,3247,130.82191780821898,66,Female,Ex-smoker,3992.3642107056653,101.95095523353416,1.5647728386258548,2.2999908856430924 159 | ID00190637202244450116191,4,4291,112.01315652083099,69,Male,Ex-smoker,5459.043011444886,85.64231956208326,1.9418851100246735,4.093276022720313 160 | ID00235637202261451839085,34,2061,53.8795357105511,67,Male,Ex-smoker,3301.0721295776366,70.75901680302682,1.8482963509619983,3.6231564740752997 161 | ID00407637202308788732304,21,3179,93.7647475224163,66,Male,Ex-smoker,2855.511474609375,90.13191204404139,1.6339334396388272,2.7240452040317127 162 | ID00309637202282195513787,3,3240,83.0854446609909,73,Male,Never smoked,4716.128291913776,66.72488658990956,1.4566573173962174,1.4488785579103194 163 | ID00136637202224951350618,3,3239,93.2354634427173,64,Male,Ex-smoker,3886.404230804696,94.5088152692202,2.021519104915034,6.070456675833322 164 | ID00305637202281772703145,8,2984,78.8917089678511,62,Male,Ex-smoker,4347.301503124238,81.46445499239461,1.6235961079522447,2.711279001865127 165 | ID00398637202303897337979,11,3046,89.5460959548448,70,Male,Ex-smoker,2600.717481613159,76.73091058114325,1.880960404468414,4.164281552520616 166 | ID00360637202295712204040,3,3107,91.4361389052384,79,Male,Never smoked,3420.760540527344,78.26283846403969,1.1720091114407796,0.6580854286861326 167 | ID00421637202311550012437,15,2739,82.0452911574407,68,Male,Ex-smoker,7933.976501473119,78.68866698190833,1.8615366022823099,4.041201517806079 168 | ID00067637202189903532242,4,3456,91.2354804646251,64,Male,Ex-smoker,5922.776166617229,98.01918776368646,1.3176088348693649,1.1433650887594622 169 | ID00370637202296737666151,12,2308,63.8062589848502,75,Male,Ex-smoker,3069.7598643310257,77.6626062625631,2.3613928862749844,6.417095993585166 170 | ID00197637202246865691526,0,2776,73.79053694843171,65,Male,Currently smokes,3204.9341923713687,61.01614924981423,2.4029579137880557,8.196368738426077 171 | ID00075637202198610425520,25,1641,67.0534875168553,52,Female,Never smoked,1975.0075868240096,90.8564483907112,2.198819105312008,5.161619733992421 172 | ID00111637202210956877205,9,2389,68.9505887785731,72,Male,Ex-smoker,3397.314184112549,95.51489514643343,0.9840285884071898,0.5919724198700975 173 | ID00122637202216437668965,-4,2581,69.50129254631621,58,Male,Ex-smoker,5611.258033919006,63.27003340685697,1.3964431194258637,1.6991937920790043 174 | ID00170637202238079193844,38,1970,91.884328358209,79,Female,Ex-smoker,2658.590987991333,57.35635419432889,2.428293566427738,7.752927451324265 175 | ID00051637202185848464638,-1,1697,81.6454173682945,73,Female,Ex-smoker,2555.2351523742677,69.71475161116238,1.5591096048153457,2.5011012056634296 176 | ID00294637202279614924243,5,2327,78.8279132791328,74,Male,Ex-smoker,3316.718524726762,83.65854242872174,1.5170500759395427,2.7771889214799383 177 | ID00219637202258203123958,0,6399,153.012912482066,71,Male,Ex-smoker,7961.942309326172,72.915660668893,1.43954866217281,1.932292689188949 178 | -------------------------------------------------------------------------------- /results_attn_bc/resnet18_fd_32_af_64_nal_1.txt: -------------------------------------------------------------------------------- 1 | fold: 0 2 | epoch 1 train: 64.96971821784973 3 | epoch 1 val: 22.60785722732544 4 | val score: 6.883895315696509 5 | val rmse: 204.41987938112 6 | epoch 2 train: 58.714619159698486 7 | epoch 2 val: 22.989309310913086 8 | val score: 6.74167478456399 9 | val rmse: 198.83041161764962 10 | epoch 3 train: 58.0261812210083 11 | epoch 3 val: 68.01687145233154 12 | val score: 6.926952443279564 13 | val rmse: 476.0772508845134 14 | epoch 4 train: 58.75056457519531 15 | epoch 4 val: 21.90040373802185 16 | val score: 6.827512793947173 17 | val rmse: 201.24801965806344 18 | epoch 5 train: 57.9221887588501 19 | epoch 5 val: 20.469754934310913 20 | val score: 6.767011369404967 21 | val rmse: 197.54937269889135 22 | epoch 6 train: 57.48012590408325 23 | epoch 6 val: 20.48733425140381 24 | val score: 6.721745731668951 25 | val rmse: 195.3726373367204 26 | epoch 7 train: 57.64536094665527 27 | epoch 7 val: 20.20257830619812 28 | val score: 6.768781705689127 29 | val rmse: 198.78217474448934 30 | epoch 8 train: 58.29794239997864 31 | epoch 8 val: 21.213780879974365 32 | val score: 6.74871679298061 33 | val rmse: 196.91210931208957 34 | epoch 9 train: 57.1262092590332 35 | epoch 9 val: 20.136257886886597 36 | val score: 6.773859224617981 37 | val rmse: 197.71189172509966 38 | epoch 10 train: 59.32291924953461 39 | epoch 10 val: 21.753007173538208 40 | val score: 6.72779092781564 41 | val rmse: 194.5186514328947 42 | epoch 11 train: 61.24781274795532 43 | epoch 11 val: 20.75409173965454 44 | val score: 6.82302203868625 45 | val rmse: 202.29277642786428 46 | epoch 12 train: 57.28135573863983 47 | epoch 12 val: 19.689078330993652 48 | val score: 6.715912168723626 49 | val rmse: 194.96821321234722 50 | epoch 13 train: 57.69435358047485 51 | epoch 13 val: 19.71856451034546 52 | val score: 6.7644667543355945 53 | val rmse: 196.26110508358445 54 | epoch 14 train: 58.3093227148056 55 | epoch 14 val: 20.42232322692871 56 | val score: 6.734788974066989 57 | val rmse: 197.28648814924378 58 | epoch 15 train: 56.258116722106934 59 | epoch 15 val: 18.96862006187439 60 | val score: 6.7168673617553845 61 | val rmse: 195.58620207958182 62 | epoch 16 train: 58.267945289611816 63 | epoch 16 val: 20.13893699645996 64 | val score: 6.840996342547207 65 | val rmse: 199.88530177051982 66 | epoch 17 train: 56.636881828308105 67 | epoch 17 val: 20.536177158355713 68 | val score: 6.682182988569278 69 | val rmse: 192.20301815814366 70 | epoch 18 train: 56.25700533390045 71 | epoch 18 val: 23.267988204956055 72 | val score: 6.8563203640841035 73 | val rmse: 203.99328038006283 74 | epoch 19 train: 61.55150830745697 75 | epoch 19 val: 22.976202249526978 76 | val score: 6.901134483925695 77 | val rmse: 205.94635108861954 78 | epoch 20 train: 56.739397048950195 79 | epoch 20 val: 18.58591055870056 80 | val score: 6.659168184128277 81 | val rmse: 192.43016597260515 82 | epoch 21 train: 58.377131938934326 83 | epoch 21 val: 20.386918783187866 84 | val score: 6.739765279342913 85 | val rmse: 197.09695147439766 86 | epoch 22 train: 60.26612639427185 87 | epoch 22 val: 20.612136125564575 88 | val score: 6.712560228136465 89 | val rmse: 195.57536995306782 90 | epoch 23 train: 59.023926734924316 91 | epoch 23 val: 21.39928650856018 92 | val score: 6.7621182960443695 93 | val rmse: 198.38434836341216 94 | epoch 24 train: 59.21179485321045 95 | epoch 24 val: 19.24097216129303 96 | val score: 6.773145740059698 97 | val rmse: 198.9817807284807 98 | epoch 25 train: 57.750040769577026 99 | epoch 25 val: 19.59726357460022 100 | val score: 6.701387584274934 101 | val rmse: 195.1876103113945 102 | epoch 26 train: 59.177435994148254 103 | epoch 26 val: 21.951263427734375 104 | val score: 6.742302232286058 105 | val rmse: 197.3978784952532 106 | epoch 27 train: 58.710195541381836 107 | epoch 27 val: 19.88407611846924 108 | val score: 6.680670016455766 109 | val rmse: 198.21409358138632 110 | epoch 28 train: 57.52652931213379 111 | epoch 28 val: 20.904608964920044 112 | val score: 6.758351737909089 113 | val rmse: 197.68929417928774 114 | epoch 29 train: 59.46340990066528 115 | epoch 29 val: 21.38616442680359 116 | val score: 6.747595181771235 117 | val rmse: 198.16928228955638 118 | epoch 30 train: 56.79719090461731 119 | epoch 30 val: 20.54972815513611 120 | val score: 6.756958112613133 121 | val rmse: 197.89141979723615 122 | epoch 31 train: 58.46683645248413 123 | epoch 31 val: 20.790443420410156 124 | val score: 6.715051689333229 125 | val rmse: 195.60113662735895 126 | epoch 32 train: 58.56350636482239 127 | epoch 32 val: 21.28389596939087 128 | val score: 6.746241730969327 129 | val rmse: 196.91341092919802 130 | epoch 33 train: 58.20165514945984 131 | epoch 33 val: 21.506865978240967 132 | val score: 6.831172286958296 133 | val rmse: 203.29927701947918 134 | epoch 34 train: 58.66703200340271 135 | epoch 34 val: 21.411378622055054 136 | val score: 6.738272860496448 137 | val rmse: 198.18476899935487 138 | epoch 35 train: 57.46269345283508 139 | epoch 35 val: 19.714397430419922 140 | val score: 6.784112776005541 141 | val rmse: 198.61721596109908 142 | epoch 36 train: 58.064886808395386 143 | epoch 36 val: 21.341623544692993 144 | val score: 6.7411692419549265 145 | val rmse: 195.8531473037298 146 | epoch 37 train: 60.096336126327515 147 | epoch 37 val: 20.07378888130188 148 | val score: 6.75046324518301 149 | val rmse: 197.338857797543 150 | epoch 38 train: 58.72186207771301 151 | epoch 38 val: 21.268160820007324 152 | val score: 6.789329746329116 153 | val rmse: 199.81368785854215 154 | epoch 39 train: 57.9124653339386 155 | epoch 39 val: 28.296252250671387 156 | val score: 6.727331263175473 157 | val rmse: 233.00995195103428 158 | epoch 40 train: 57.10844802856445 159 | epoch 40 val: 19.008377075195312 160 | val score: 6.678468369908912 161 | val rmse: 193.70234750068047 162 | fold: 1 163 | epoch 1 train: 62.44024872779846 164 | epoch 1 val: 29.65473985671997 165 | val score: 6.856359424692613 166 | val rmse: 206.30350414123825 167 | epoch 2 train: 67.38722634315491 168 | epoch 2 val: 24.617186307907104 169 | val score: 6.7487404279245276 170 | val rmse: 193.77896197343762 171 | epoch 3 train: 60.35738277435303 172 | epoch 3 val: 17.241989135742188 173 | val score: 6.425448746608218 174 | val rmse: 160.39582500395363 175 | epoch 4 train: 61.113476037979126 176 | epoch 4 val: 18.783568620681763 177 | val score: 6.640013977349293 178 | val rmse: 169.59961147109505 179 | epoch 5 train: 59.462254762649536 180 | epoch 5 val: 16.683509588241577 181 | val score: 6.517627670806958 182 | val rmse: 166.45509609121154 183 | epoch 6 train: 60.99172353744507 184 | epoch 6 val: 18.98461127281189 185 | val score: 6.522628827746667 186 | val rmse: 165.8386422757896 187 | epoch 7 train: 61.9699285030365 188 | epoch 7 val: 19.119922161102295 189 | val score: 6.506105987251446 190 | val rmse: 163.78897815194622 191 | epoch 8 train: 66.09351873397827 192 | epoch 8 val: 17.354548931121826 193 | val score: 6.527891632721243 194 | val rmse: 163.44147729828342 195 | epoch 9 train: 61.20128130912781 196 | epoch 9 val: 18.47905659675598 197 | val score: 6.477815276770164 198 | val rmse: 169.08900963005354 199 | epoch 10 train: 59.98923420906067 200 | epoch 10 val: 22.823837518692017 201 | val score: 6.655261385846787 202 | val rmse: 188.6871854965756 203 | epoch 11 train: 61.8863000869751 204 | epoch 11 val: 19.22650933265686 205 | val score: 6.573166144918978 206 | val rmse: 187.47783618418993 207 | epoch 12 train: 60.88166332244873 208 | epoch 12 val: 17.72073531150818 209 | val score: 6.542945159272933 210 | val rmse: 165.42752988754322 211 | epoch 13 train: 60.46764373779297 212 | epoch 13 val: 18.252128839492798 213 | val score: 6.5599683003573475 214 | val rmse: 167.46856437504303 215 | epoch 14 train: 61.2820405960083 216 | epoch 14 val: 16.8647620677948 217 | val score: 6.452074251338734 218 | val rmse: 161.7928228357389 219 | epoch 15 train: 60.194764852523804 220 | epoch 15 val: 18.14552617073059 221 | val score: 6.439771377439734 222 | val rmse: 167.16279443697198 223 | epoch 16 train: 62.659385442733765 224 | epoch 16 val: 16.979930877685547 225 | val score: 6.524179205542128 226 | val rmse: 165.48752365183276 227 | epoch 17 train: 59.339953899383545 228 | epoch 17 val: 17.06336283683777 229 | val score: 6.462294671640389 230 | val rmse: 166.86679393220004 231 | epoch 18 train: 61.026084661483765 232 | epoch 18 val: 18.131183624267578 233 | val score: 6.510317387182465 234 | val rmse: 172.95115468733783 235 | epoch 19 train: 60.13036596775055 236 | epoch 19 val: 17.55238103866577 237 | val score: 6.474124313074519 238 | val rmse: 165.55883911164685 239 | epoch 20 train: 58.8433678150177 240 | epoch 20 val: 19.425392389297485 241 | val score: 6.528681700019513 242 | val rmse: 167.72914567703725 243 | epoch 21 train: 60.31694793701172 244 | epoch 21 val: 17.742435216903687 245 | val score: 6.530732049753975 246 | val rmse: 170.34766760271503 247 | epoch 22 train: 60.11043655872345 248 | epoch 22 val: 18.88919425010681 249 | val score: 6.538138308801744 250 | val rmse: 172.94209131855843 251 | epoch 23 train: 60.62796866893768 252 | epoch 23 val: 18.080440759658813 253 | val score: 6.537722896233847 254 | val rmse: 170.51162061203522 255 | epoch 24 train: 60.874069690704346 256 | epoch 24 val: 55.04205799102783 257 | val score: 6.869090579748873 258 | val rmse: 368.3690104269625 259 | epoch 25 train: 58.083086252212524 260 | epoch 25 val: 23.9586124420166 261 | val score: 6.599844742856152 262 | val rmse: 211.0242627644548 263 | epoch 26 train: 60.319310426712036 264 | epoch 26 val: 21.906644582748413 265 | val score: 6.594885989824989 266 | val rmse: 193.09614384764924 267 | epoch 27 train: 60.68947076797485 268 | epoch 27 val: 18.52328372001648 269 | val score: 6.498507250226581 270 | val rmse: 166.10660943967684 271 | epoch 28 train: 60.61902916431427 272 | epoch 28 val: 17.382543563842773 273 | val score: 6.478327526672415 274 | val rmse: 165.5203270166263 275 | epoch 29 train: 58.29853892326355 276 | epoch 29 val: 16.901286125183105 277 | val score: 6.47714909877495 278 | val rmse: 162.2553130338177 279 | epoch 30 train: 58.78415620326996 280 | epoch 30 val: 16.86282968521118 281 | val score: 6.527417007925515 282 | val rmse: 164.29694235489066 283 | epoch 31 train: 59.019872188568115 284 | epoch 31 val: 16.941307544708252 285 | val score: 6.51762249521174 286 | val rmse: 164.24485824443835 287 | epoch 32 train: 58.24340581893921 288 | epoch 32 val: 16.6660578250885 289 | val score: 6.483269674966457 290 | val rmse: 164.89073698223618 291 | epoch 33 train: 57.12609100341797 292 | epoch 33 val: 19.843063831329346 293 | val score: 6.558502272839134 294 | val rmse: 168.28111944167975 295 | epoch 34 train: 58.708112478256226 296 | epoch 34 val: 19.911396026611328 297 | val score: 6.556810932116592 298 | val rmse: 170.1611641318168 299 | epoch 35 train: 60.44526278972626 300 | epoch 35 val: 19.37260103225708 301 | val score: 6.564664567562074 302 | val rmse: 170.09510710366644 303 | epoch 36 train: 59.00279998779297 304 | epoch 36 val: 20.37187647819519 305 | val score: 6.531875382360081 306 | val rmse: 172.86139820433024 307 | epoch 37 train: 59.881338119506836 308 | epoch 37 val: 19.733505487442017 309 | val score: 6.498633246837682 310 | val rmse: 176.0167532260386 311 | epoch 38 train: 58.65711796283722 312 | epoch 38 val: 20.3035306930542 313 | val score: 6.479703974784562 314 | val rmse: 182.70913930424405 315 | epoch 39 train: 60.52656149864197 316 | epoch 39 val: 24.744938373565674 317 | val score: 6.450538101330263 318 | val rmse: 204.91160766230539 319 | epoch 40 train: 61.28426456451416 320 | epoch 40 val: 21.646448850631714 321 | val score: 6.421679230413398 322 | val rmse: 191.18173239067212 323 | fold: 2 324 | epoch 1 train: 71.71574974060059 325 | epoch 1 val: 14.17856216430664 326 | val score: 7.192725282191628 327 | val rmse: 215.75978099667873 328 | epoch 2 train: 63.79126262664795 329 | epoch 2 val: 13.242332696914673 330 | val score: 7.244561855646499 331 | val rmse: 219.02765596487563 332 | epoch 3 train: 64.19961404800415 333 | epoch 3 val: 23.056568145751953 334 | val score: 7.083234605471872 335 | val rmse: 250.50243485294425 336 | epoch 4 train: 63.75034701824188 337 | epoch 4 val: 13.01891028881073 338 | val score: 7.11872523256988 339 | val rmse: 216.89852394020835 340 | epoch 5 train: 64.72794723510742 341 | epoch 5 val: 12.713470458984375 342 | val score: 7.101647176792761 343 | val rmse: 217.57780743456507 344 | epoch 6 train: 63.77369272708893 345 | epoch 6 val: 21.656996846199036 346 | val score: 7.2116372082010685 347 | val rmse: 266.137063034339 348 | epoch 7 train: 64.48489785194397 349 | epoch 7 val: 16.16008758544922 350 | val score: 7.182478261564255 351 | val rmse: 234.2132473801052 352 | epoch 8 train: 63.141905069351196 353 | epoch 8 val: 16.795446157455444 354 | val score: 7.12257847575781 355 | val rmse: 234.83888937169763 356 | epoch 9 train: 64.60765063762665 357 | epoch 9 val: 13.880623579025269 358 | val score: 7.134113201430326 359 | val rmse: 219.84013651079633 360 | epoch 10 train: 63.85371279716492 361 | epoch 10 val: 14.808722257614136 362 | val score: 7.160292908181091 363 | val rmse: 223.01558269222116 364 | epoch 11 train: 64.1598596572876 365 | epoch 11 val: 17.050592184066772 366 | val score: 7.233950269684907 367 | val rmse: 225.08255777168995 368 | epoch 12 train: 62.32453155517578 369 | epoch 12 val: 13.083370804786682 370 | val score: 7.143916827282758 371 | val rmse: 220.9539092157171 372 | epoch 13 train: 63.61132073402405 373 | epoch 13 val: 14.025254964828491 374 | val score: 7.194533458719552 375 | val rmse: 217.83836986079447 376 | epoch 14 train: 65.12792110443115 377 | epoch 14 val: 38.32434320449829 378 | val score: 7.185178914502789 379 | val rmse: 327.4383806555876 380 | epoch 15 train: 63.11473846435547 381 | epoch 15 val: 14.217060804367065 382 | val score: 7.160864489969075 383 | val rmse: 224.76175768471882 384 | epoch 16 train: 65.40810680389404 385 | epoch 16 val: 16.075042009353638 386 | val score: 7.115112726964226 387 | val rmse: 225.72219138844903 388 | epoch 17 train: 62.941240072250366 389 | epoch 17 val: 15.158620357513428 390 | val score: 7.143503457045821 391 | val rmse: 223.92425024642094 392 | epoch 18 train: 62.56142067909241 393 | epoch 18 val: 15.97208833694458 394 | val score: 7.133864853138459 395 | val rmse: 224.36461891257443 396 | epoch 19 train: 62.40687131881714 397 | epoch 19 val: 12.129128694534302 398 | val score: 7.150614914414641 399 | val rmse: 216.34028956683312 400 | epoch 20 train: 63.514891147613525 401 | epoch 20 val: 13.702512741088867 402 | val score: 7.142593199037591 403 | val rmse: 221.22387503092 404 | epoch 21 train: 64.20075011253357 405 | epoch 21 val: 13.506792485713959 406 | val score: 7.110019413061831 407 | val rmse: 222.84213104619897 408 | epoch 22 train: 63.199618101119995 409 | epoch 22 val: 12.718915820121765 410 | val score: 7.1861931344233305 411 | val rmse: 219.450695736856 412 | epoch 23 train: 64.27488040924072 413 | epoch 23 val: 13.124852776527405 414 | val score: 7.154358851746924 415 | val rmse: 218.97975761879962 416 | epoch 24 train: 62.31318736076355 417 | epoch 24 val: 22.07652711868286 418 | val score: 7.18075273389596 419 | val rmse: 253.13887696670756 420 | epoch 25 train: 62.157729625701904 421 | epoch 25 val: 14.48632526397705 422 | val score: 7.116811976451142 423 | val rmse: 223.4643407926591 424 | epoch 26 train: 64.27236866950989 425 | epoch 26 val: 15.957724809646606 426 | val score: 7.122438258805875 427 | val rmse: 223.48987322683595 428 | epoch 27 train: 62.24888491630554 429 | epoch 27 val: 14.351091623306274 430 | val score: 7.147614407832893 431 | val rmse: 223.18967516308535 432 | epoch 28 train: 64.39752054214478 433 | epoch 28 val: 15.476150035858154 434 | val score: 7.098657690078083 435 | val rmse: 224.15370293761703 436 | epoch 29 train: 62.434977293014526 437 | epoch 29 val: 15.477422714233398 438 | val score: 7.110637935604798 439 | val rmse: 221.23788131328405 440 | epoch 30 train: 65.00940155982971 441 | epoch 30 val: 14.709250211715698 442 | val score: 7.144325515436995 443 | val rmse: 224.48023504927397 444 | epoch 31 train: 63.930299282073975 445 | epoch 31 val: 13.106693029403687 446 | val score: 7.156302871350145 447 | val rmse: 218.78348017094285 448 | epoch 32 train: 63.271746039390564 449 | epoch 32 val: 20.05545687675476 450 | val score: 7.133254105282583 451 | val rmse: 242.30586466355095 452 | epoch 33 train: 63.95471262931824 453 | epoch 33 val: 12.99117398262024 454 | val score: 7.139646128695488 455 | val rmse: 216.85406516471198 456 | epoch 34 train: 62.87055051326752 457 | epoch 34 val: 15.28401494026184 458 | val score: 7.236108094308856 459 | val rmse: 222.67003372367006 460 | epoch 35 train: 63.37023591995239 461 | epoch 35 val: 14.946204662322998 462 | val score: 7.133743026230233 463 | val rmse: 227.10826096720862 464 | epoch 36 train: 63.221723556518555 465 | epoch 36 val: 16.032171964645386 466 | val score: 7.138005098086352 467 | val rmse: 229.16869524992427 468 | epoch 37 train: 61.835203409194946 469 | epoch 37 val: 15.049345254898071 470 | val score: 7.127663090364313 471 | val rmse: 227.62121176226043 472 | epoch 38 train: 60.92604970932007 473 | epoch 38 val: 17.795825481414795 474 | val score: 7.944993850849465 475 | val rmse: 228.96332033974974 476 | epoch 39 train: 63.2498836517334 477 | epoch 39 val: 16.2826144695282 478 | val score: 7.321407734492529 479 | val rmse: 226.87977948888803 480 | epoch 40 train: 63.23513889312744 481 | epoch 40 val: 16.770180225372314 482 | val score: 7.148532058522681 483 | val rmse: 234.68992662524363 484 | fold: 3 485 | epoch 1 train: 65.26435124874115 486 | epoch 1 val: 23.508511066436768 487 | val score: 6.5079558461851015 488 | val rmse: 181.56576652891255 489 | epoch 2 train: 62.019973278045654 490 | epoch 2 val: 29.254785537719727 491 | val score: 6.625203770099658 492 | val rmse: 216.48060848962686 493 | epoch 3 train: 60.69349694252014 494 | epoch 3 val: 26.704277992248535 495 | val score: 6.565287932530117 496 | val rmse: 205.45392413893734 497 | epoch 4 train: 60.40701174736023 498 | epoch 4 val: 48.26881217956543 499 | val score: 6.850096913207576 500 | val rmse: 322.8347416433317 501 | epoch 5 train: 60.71586513519287 502 | epoch 5 val: 21.058653116226196 503 | val score: 6.523491749094691 504 | val rmse: 185.8680845876077 505 | epoch 6 train: 60.60818636417389 506 | epoch 6 val: 19.196923971176147 507 | val score: 6.522124473835497 508 | val rmse: 175.41776518599352 509 | epoch 7 train: 61.39654541015625 510 | epoch 7 val: 24.02221965789795 511 | val score: 6.530223861599167 512 | val rmse: 187.74353191685387 513 | epoch 8 train: 61.311980962753296 514 | epoch 8 val: 20.674588203430176 515 | val score: 6.4958437074754 516 | val rmse: 174.9837816846217 517 | epoch 9 train: 59.4351270198822 518 | epoch 9 val: 19.486040830612183 519 | val score: 6.4697178617401265 520 | val rmse: 165.82833144670278 521 | epoch 10 train: 60.43686628341675 522 | epoch 10 val: 18.588313817977905 523 | val score: 6.481736297526672 524 | val rmse: 165.5049001383606 525 | epoch 11 train: 60.23481273651123 526 | epoch 11 val: 29.827263355255127 527 | val score: 6.6008645560388866 528 | val rmse: 220.62250448456007 529 | epoch 12 train: 60.37552332878113 530 | epoch 12 val: 18.402823209762573 531 | val score: 6.485168418919716 532 | val rmse: 167.32849682811346 533 | epoch 13 train: 60.07301712036133 534 | epoch 13 val: 19.73308300971985 535 | val score: 6.469199154269604 536 | val rmse: 166.8486662605632 537 | epoch 14 train: 59.283940076828 538 | epoch 14 val: 17.480336666107178 539 | val score: 6.476262178229407 540 | val rmse: 165.29446262965425 541 | epoch 15 train: 58.960456013679504 542 | epoch 15 val: 17.99527907371521 543 | val score: 6.483868877883521 544 | val rmse: 166.37219201707157 545 | epoch 16 train: 59.37215435504913 546 | epoch 16 val: 18.444024801254272 547 | val score: 6.569794109832644 548 | val rmse: 172.0550414005399 549 | epoch 17 train: 60.132853507995605 550 | epoch 17 val: 18.139357805252075 551 | val score: 6.508193432722025 552 | val rmse: 168.62716652069847 553 | epoch 18 train: 60.95835995674133 554 | epoch 18 val: 18.674551010131836 555 | val score: 6.4710689623846624 556 | val rmse: 167.26028299906034 557 | epoch 19 train: 58.45285105705261 558 | epoch 19 val: 20.160449981689453 559 | val score: 6.496627336842919 560 | val rmse: 169.52551063452458 561 | epoch 20 train: 61.78423476219177 562 | epoch 20 val: 16.7478506565094 563 | val score: 6.436797027915626 564 | val rmse: 160.44672079740826 565 | epoch 21 train: 58.593048095703125 566 | epoch 21 val: 16.426332592964172 567 | val score: 6.4492327034363175 568 | val rmse: 163.1657780416038 569 | epoch 22 train: 59.91761255264282 570 | epoch 22 val: 20.272841215133667 571 | val score: 6.69827749024132 572 | val rmse: 177.77532455148753 573 | epoch 23 train: 61.480879068374634 574 | epoch 23 val: 20.20084571838379 575 | val score: 6.5031362706580005 576 | val rmse: 170.3514078769875 577 | epoch 24 train: 59.91995668411255 578 | epoch 24 val: 18.55363154411316 579 | val score: 6.470849137872126 580 | val rmse: 165.27739795869283 581 | epoch 25 train: 58.95390701293945 582 | epoch 25 val: 17.95784854888916 583 | val score: 6.478853951079196 584 | val rmse: 165.42944395258002 585 | epoch 26 train: 60.105199575424194 586 | epoch 26 val: 18.855769395828247 587 | val score: 6.457365994504421 588 | val rmse: 164.9555908097431 589 | epoch 27 train: 60.052210092544556 590 | epoch 27 val: 19.190634727478027 591 | val score: 6.50106280422537 592 | val rmse: 167.202968689525 593 | epoch 28 train: 59.33212113380432 594 | epoch 28 val: 19.75223469734192 595 | val score: 6.499897902864803 596 | val rmse: 168.14311254919963 597 | epoch 29 train: 59.69977355003357 598 | epoch 29 val: 17.552094221115112 599 | val score: 6.462103015630458 600 | val rmse: 164.1606805688412 601 | epoch 30 train: 60.239665508270264 602 | epoch 30 val: 18.229687929153442 603 | val score: 6.452919143735836 604 | val rmse: 163.27529145332707 605 | epoch 31 train: 58.68057584762573 606 | epoch 31 val: 16.739206314086914 607 | val score: 6.4366723443425915 608 | val rmse: 157.4982540590669 609 | epoch 32 train: 59.88064956665039 610 | epoch 32 val: 18.79264211654663 611 | val score: 6.518343394404429 612 | val rmse: 169.39550656968416 613 | epoch 33 train: 58.10928225517273 614 | epoch 33 val: 18.114619970321655 615 | val score: 6.464022024542986 616 | val rmse: 165.90027555983076 617 | epoch 34 train: 60.11639225482941 618 | epoch 34 val: 19.536619424819946 619 | val score: 6.459037549667577 620 | val rmse: 163.9096041142924 621 | epoch 35 train: 59.94880020618439 622 | epoch 35 val: 17.5801260471344 623 | val score: 6.449825421609028 624 | val rmse: 163.13377064592356 625 | epoch 36 train: 59.042219400405884 626 | epoch 36 val: 18.38330316543579 627 | val score: 6.481887976231905 628 | val rmse: 166.12948953941353 629 | epoch 37 train: 58.94150269031525 630 | epoch 37 val: 18.87453842163086 631 | val score: 6.465535554259673 632 | val rmse: 164.9215452068353 633 | epoch 38 train: 60.2912118434906 634 | epoch 38 val: 18.79565668106079 635 | val score: 6.4544523335483595 636 | val rmse: 163.71123378011927 637 | epoch 39 train: 59.20958065986633 638 | epoch 39 val: 18.614283084869385 639 | val score: 6.480616950192098 640 | val rmse: 166.108866547185 641 | epoch 40 train: 59.24623990058899 642 | epoch 40 val: 18.973641395568848 643 | val score: 6.470858690366123 644 | val rmse: 165.1440655704035 645 | fold: 4 646 | epoch 1 train: 70.17238521575928 647 | epoch 1 val: 22.34827470779419 648 | val score: 6.667239898492185 649 | val rmse: 193.89860030099683 650 | epoch 2 train: 63.15628242492676 651 | epoch 2 val: 18.86600947380066 652 | val score: 6.682029903381849 653 | val rmse: 185.94696514595594 654 | epoch 3 train: 62.37942981719971 655 | epoch 3 val: 17.75534749031067 656 | val score: 6.859860997547514 657 | val rmse: 188.34408554371035 658 | epoch 4 train: 60.65038537979126 659 | epoch 4 val: 23.26894235610962 660 | val score: 6.767487827368018 661 | val rmse: 187.8022332084075 662 | epoch 5 train: 59.99577021598816 663 | epoch 5 val: 17.07836627960205 664 | val score: 6.737144269457005 665 | val rmse: 188.56763972937594 666 | epoch 6 train: 60.4095801115036 667 | epoch 6 val: 17.24711275100708 668 | val score: 6.82633576505439 669 | val rmse: 192.46296354272414 670 | epoch 7 train: 60.87045228481293 671 | epoch 7 val: 17.0134379863739 672 | val score: 6.770492769214463 673 | val rmse: 185.0443770730405 674 | epoch 8 train: 61.01167273521423 675 | epoch 8 val: 20.122021913528442 676 | val score: 6.775117498450442 677 | val rmse: 189.07894126176288 678 | epoch 9 train: 61.75462341308594 679 | epoch 9 val: 59.24621391296387 680 | val score: 6.876119324372362 681 | val rmse: 397.0887940499722 682 | epoch 10 train: 61.466769218444824 683 | epoch 10 val: 18.96992039680481 684 | val score: 6.7220742870478665 685 | val rmse: 183.32900629673154 686 | epoch 11 train: 61.979114294052124 687 | epoch 11 val: 15.823312997817993 688 | val score: 6.746095754548985 689 | val rmse: 183.74154430104284 690 | epoch 12 train: 60.65316295623779 691 | epoch 12 val: 16.552685737609863 692 | val score: 6.752775926790695 693 | val rmse: 183.27022119215619 694 | epoch 13 train: 62.69566535949707 695 | epoch 13 val: 18.708300352096558 696 | val score: 6.7075068936811535 697 | val rmse: 184.1109819649906 698 | epoch 14 train: 61.83624768257141 699 | epoch 14 val: 27.752505779266357 700 | val score: 6.752533951200956 701 | val rmse: 222.55493657706668 702 | epoch 15 train: 60.881738901138306 703 | epoch 15 val: 25.763005256652832 704 | val score: 6.74187091934614 705 | val rmse: 208.83286435716445 706 | epoch 16 train: 60.86843824386597 707 | epoch 16 val: 17.666922092437744 708 | val score: 6.686787260563145 709 | val rmse: 182.19117842582065 710 | epoch 17 train: 59.67353594303131 711 | epoch 17 val: 16.297288179397583 712 | val score: 6.705745291615024 713 | val rmse: 184.0956828593705 714 | epoch 18 train: 62.00575840473175 715 | epoch 18 val: 18.372760772705078 716 | val score: 6.748476385991245 717 | val rmse: 182.92189309303077 718 | epoch 19 train: 60.92128622531891 719 | epoch 19 val: 15.495804071426392 720 | val score: 6.72468694111483 721 | val rmse: 182.42329177205198 722 | epoch 20 train: 61.333000898361206 723 | epoch 20 val: 16.01842761039734 724 | val score: 6.722183768577181 725 | val rmse: 182.58029649699048 726 | epoch 21 train: 61.24144959449768 727 | epoch 21 val: 18.718430995941162 728 | val score: 6.765948157495835 729 | val rmse: 191.01955896072715 730 | epoch 22 train: 61.09436893463135 731 | epoch 22 val: 22.270862817764282 732 | val score: 6.76912555478758 733 | val rmse: 213.64817422366332 734 | epoch 23 train: 60.27746891975403 735 | epoch 23 val: 29.399681568145752 736 | val score: 6.694414609407009 737 | val rmse: 242.61383594818435 738 | epoch 24 train: 59.85258722305298 739 | epoch 24 val: 51.229580879211426 740 | val score: 6.769763165907258 741 | val rmse: 334.1467453777578 742 | epoch 25 train: 60.503751158714294 743 | epoch 25 val: 24.01532793045044 744 | val score: 6.697931598523037 745 | val rmse: 205.32176175977162 746 | epoch 26 train: 60.66587686538696 747 | epoch 26 val: 19.033122539520264 748 | val score: 6.751303058767696 749 | val rmse: 189.17054086673753 750 | epoch 27 train: 60.188878536224365 751 | epoch 27 val: 15.371063947677612 752 | val score: 6.72000702938068 753 | val rmse: 182.36746724981353 754 | epoch 28 train: 61.16017127037048 755 | epoch 28 val: 18.756279706954956 756 | val score: 6.736999949353348 757 | val rmse: 182.62676963311134 758 | epoch 29 train: 62.29088020324707 759 | epoch 29 val: 20.152073860168457 760 | val score: 6.7652538843646015 761 | val rmse: 183.19627163644432 762 | epoch 30 train: 62.86806297302246 763 | epoch 30 val: 15.422858953475952 764 | val score: 6.722136977222664 765 | val rmse: 181.8734021775325 766 | epoch 31 train: 60.43858242034912 767 | epoch 31 val: 17.808740377426147 768 | val score: 6.83491700707524 769 | val rmse: 185.71438736748505 770 | epoch 32 train: 61.82444715499878 771 | epoch 32 val: 18.24975824356079 772 | val score: 6.739697790387389 773 | val rmse: 183.46972287559842 774 | epoch 33 train: 61.92837643623352 775 | epoch 33 val: 16.0122013092041 776 | val score: 6.806178631900573 777 | val rmse: 184.83245107478825 778 | epoch 34 train: 61.01805353164673 779 | epoch 34 val: 16.119980335235596 780 | val score: 6.744265842985423 781 | val rmse: 183.18806936017813 782 | epoch 35 train: 61.3659029006958 783 | epoch 35 val: 16.197680950164795 784 | val score: 6.740768972911454 785 | val rmse: 182.87657911018684 786 | epoch 36 train: 60.04664373397827 787 | epoch 36 val: 16.94153618812561 788 | val score: 6.72305442795946 789 | val rmse: 184.17024250017025 790 | epoch 37 train: 59.7602014541626 791 | epoch 37 val: 16.29321026802063 792 | val score: 6.707786351120749 793 | val rmse: 181.15433381409105 794 | epoch 38 train: 60.62006592750549 795 | epoch 38 val: 17.862085103988647 796 | val score: 6.725073963191223 797 | val rmse: 182.15990141045884 798 | epoch 39 train: 60.01054322719574 799 | epoch 39 val: 19.05290699005127 800 | val score: 6.781517074777077 801 | val rmse: 183.65738291576997 802 | epoch 40 train: 61.52290940284729 803 | epoch 40 val: 15.805213212966919 804 | val score: 6.797546411163592 805 | val rmse: 186.6794139291405 806 | Final training 807 | epoch 1 train: 98.18125748634338 808 | epoch 2 train: 97.02088141441345 809 | epoch 3 train: 94.76944661140442 810 | --------------------------------------------------------------------------------