├── Workflow.png
├── cg_generation.png
├── LICENSE
├── README.md
├── Binary_data_preprocessing.ipynb
├── Ternary_data_preprocessing.ipynb
└── scdiag_gintopk_roc_ternary.ipynb


/Workflow.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Docurdt/Cell-Graph_Signature/HEAD/Workflow.png


--------------------------------------------------------------------------------
/cg_generation.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/Docurdt/Cell-Graph_Signature/HEAD/cg_generation.png


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2021 Yanan Wang & Yuguang Wang
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # Cell-Graph Signature (CG<sub>Signature</sub>)
 2 | #### GNN-based cancer prognosis prediction using image-derived Cell-Graphs
 3 | > Git repo for the manuscript of "Cell graph neural networks enable digital staging of tumour microenvironment and precisely predict patient survival in gastric cancer".
 4 | 
 5 | ### Abstract
 6 | Gastric cancer is one of the deadliest cancers worldwide. Accurate prognosis is essential for effective clinical assessment and treatment. Spatial patterns in the tumor microenvironment (TME) are conceptually indicative of the staging and progression of gastric cancer patients. Using spatial patterns of the TME by integrating and transforming the multiplexed immunohistochemistry (mIHC) images as Cell-Graphs, we propose a novel graph neural network-based approach, termed **Cell-Graph Signature** or **CG<sub>Signature</sub>**, powered by artificial intelligence, for digital staging of TME and precise prediction of patient survival in gastric cancer. In this study, patient survival prediction is formulated as either a binary (**short-term** and **long-term**) or ternary (**short-term**, **medium-term**, and **long-term**) classification task. Extensive benchmarking experiments demonstrate that the CG<sub>Signature</sub> achieves outstanding model performance, with Area Under the Receiver-Operating Characteristic curve (AUROC) of 0.960 &pm; 0.01, and 0.771 &pm; 0.024 to 0.904 &pm; 0.012 for the binary- and ternary-classification, respectively. Moreover, Kaplan-Meier survival analysis indicates that the 'digital-grade' cancer staging produced by CG<sub>Signature</sub> provides a remarkable capability in discriminating both binary and ternary classes with statistical significance (<em>P-value</em> < 0.0001), significantly outperforming the AJCC 8th edition Tumor-Node-Metastasis staging system. Using Cell-Graphs extracted from mIHC images, CG<sub>Signature</sub> improves the assessment of the link between the TME spatial patterns and patient prognosis. Our study suggests the feasibility and benefits of such artificial intelligence-powered digital staging system in diagnostic pathology and precision oncology.
 7 | 
 8 | ### Workflow
 9 | ![An overall workflow of graph neural network-based prognosis prediction using Cell-Graphs.](Workflow.png "An overall workflow of graph neural network-based prognosis prediction using Cell-Graphs.")
10 | **Figure 1. An overall workflow of graph neural network-based prognosis prediction using Cell-Graphs.**
11 | **(a)** Specimen processing: The tumor tissues were extracted from gastric cancer, and stained with seven different biomarkers including DAPI, Pan-CK, CD8, CD68, CD163/CD45, Foxp3, and PD-L1. **(b)** Image pre-processing: sub-sampling and cell-graph construction were conducted for image pre-processing. **(c)** An illustration for the cohort, 172 gastric cancer patients were collected. **(d)** Data split. The training, validation and testing datasets were split with the percentages of 64%, 16%, and 20%, respectively. **(f)** Data binning: overall survival time ranged from 0 to 88 months, and two data binning strategies were applied to generate  binary- and ternary-class datasets. **(e)** Model construction: four different GNN model architectures, including GCNSag, GCNTopK, GINSag, and GINTopK, were constructed and compared. Multi-run model training, five-fold cross-validation, and independent test were conducted to evaluate the performance of the constructed GNN models. **(g)** Model architecture: The four models shared the same architecture but employed different types of convolutional unit and pooling layer, which consists of four consecutive convolutional layer and pooling layer blocks, followed by a summary layer and three fully-connected layers, prior to the generation of the final classification outcome. Architecture of the best-performing GINTopK model is illustrated herein, which outperformed the other three model architectures and also achieved the best performance on the test dataset. The corresponding number of hidden layers or feature dimensions are indicated at the bottom of each box. Here, FC stands for "fully connected layer".
12 | 
13 | ![An overview of procedures from multiplexed staining to Cell-Graph generation.](cg_generation.png "An overall workflow of graph neural network-based prognosis prediction using Cell-Graphs.")
14 | **Figure 2. An overview of procedures from multiplexed staining to Cell-Graph generation.** Detailed information can be seen in the descriptions of the figure.
15 | 


--------------------------------------------------------------------------------
/Binary_data_preprocessing.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "code",
  5 |    "execution_count": 2,
  6 |    "metadata": {},
  7 |    "outputs": [],
  8 |    "source": [
  9 |     "#!/usr/bin/env python3\n",
 10 |     "# -*- coding: utf-8 -*-\n",
 11 |     "\"\"\"\n",
 12 |     "Created on Thu Aug  8 01:49:00 2019\n",
 13 |     "\n",
 14 |     "@author: Yuguang Wang & Yanan Wang\n",
 15 |     "\"\"\"\n",
 16 |     "\n",
 17 |     "import pandas as pd\n",
 18 |     "import numpy as np\n",
 19 |     "import scipy.io as sio\n",
 20 |     "import os\n",
 21 |     "from scipy.sparse import csr_matrix"
 22 |    ]
 23 |   },
 24 |   {
 25 |    "cell_type": "code",
 26 |    "execution_count": null,
 27 |    "metadata": {},
 28 |    "outputs": [
 29 |     {
 30 |      "name": "stdout",
 31 |      "output_type": "stream",
 32 |      "text": [
 33 |       "****** Processing data for data_file/test_data_surv.xlsx ******\n"
 34 |      ]
 35 |     },
 36 |     {
 37 |      "name": "stderr",
 38 |      "output_type": "stream",
 39 |      "text": [
 40 |       "/home/song-lab/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:123: RuntimeWarning: divide by zero encountered in double_scalars\n",
 41 |       "/home/song-lab/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:124: RuntimeWarning: divide by zero encountered in double_scalars\n"
 42 |      ]
 43 |     }
 44 |    ],
 45 |    "source": [
 46 |     "def create_samples(file_name):\n",
 47 |     "    subdir = '_{}/'.format(file_name.split(\".\")[0])\n",
 48 |     "    # critical distance 20 \\mu m, pix width 0.5 \\mu m\n",
 49 |     "    critical = 20/0.5\n",
 50 |     "    num_node = 100\n",
 51 |     "    adj = list()\n",
 52 |     "    feature = list()\n",
 53 |     "    label = list()\n",
 54 |     "    pid = list()\n",
 55 |     "    pid_name = list()\n",
 56 |     "    coor = list()\n",
 57 |     "    edge_index = list()\n",
 58 |     "    edge_coor = list()\n",
 59 |     "    edge_attr = list()\n",
 60 |     "    factor_flag = 'selected_new'\n",
 61 |     "    #factor_flag = 'selected'\n",
 62 |     "    #factor_flag = 'full'\n",
 63 |     "    sv_dir = \"data/\" + factor_flag + subdir\n",
 64 |     "    if not os.path.exists(sv_dir):\n",
 65 |     "        os.makedirs(sv_dir)\n",
 66 |     "    \n",
 67 |     "    print('****** Processing data for %s ******' % file_name)\n",
 68 |     "    df_diagnosis = pd.read_excel(file_name)\n",
 69 |     "    id_participant = df_diagnosis['Id']\n",
 70 |     "    num_id_1 = len(id_participant)\n",
 71 |     "    survival_time_all = df_diagnosis['survival_time'].array\n",
 72 |     "    \n",
 73 |     "    for kid, row in df_diagnosis.iterrows():\n",
 74 |     "        ld_csv = row['file_path']\n",
 75 |     "        old_new_flag = ld_csv.split(\"/\")[2]\n",
 76 |     "        #print(ld_csv)\n",
 77 |     "        #print(old_new_flag)\n",
 78 |     "        \n",
 79 |     "        label1 = row['prognosis_label']\n",
 80 |     "        df_csv = pd.read_csv(ld_csv)\n",
 81 |     "        # extract the features\n",
 82 |     "        if factor_flag == 'full': # from \"DAPI (DAPI) Nucleus Intensity\" to end\n",
 83 |     "            lv = np.linspace(22,59,38,dtype=int)\n",
 84 |     "        if factor_flag == 'selected_old':\n",
 85 |     "            lv = np.array([11,12,18,35,36,37,38,39],dtype=np.int)\n",
 86 |     "        if(old_new_flag == 'stomach_csv_1' and factor_flag == 'selected_new'):\n",
 87 |     "            lv = np.concatenate((np.linspace(20,44,25,dtype=int),np.linspace(50,59,10,dtype=int)))\n",
 88 |     "        elif(old_new_flag == 'stomach_csv_2' and factor_flag == 'selected_new'):\n",
 89 |     "            lv = np.concatenate((np.linspace(32,56,25,dtype=int),np.linspace(62,71,10,dtype=int)))\n",
 90 |     "        feature_all = df_csv.take(lv,axis=1).values\n",
 91 |     "        #print(feature_all)\n",
 92 |     "        feature_all = feature_all/(feature_all.max(axis=0)+0.00000000000001) # Add normalization\n",
 93 |     "        #print(feature_all)\n",
 94 |     "        \n",
 95 |     "        \n",
 96 |     "        # compute adjacency matrix\n",
 97 |     "        xmin = np.array(df_csv['XMin'])\n",
 98 |     "        xmax = np.array(df_csv['XMax'])\n",
 99 |     "        ymin = np.array(df_csv['YMin'])\n",
100 |     "        ymax = np.array(df_csv['YMax'])\n",
101 |     "        num_cell = len(xmin)\n",
102 |     "        if np.mod(num_cell,num_node)==0:\n",
103 |     "            num_graph = int(num_cell/num_node)\n",
104 |     "        else:\n",
105 |     "            num_graph = int(num_cell/num_node)+1\n",
106 |     "        # compute the centre coordinates of cells\n",
107 |     "        xc = (xmin+xmax)/2\n",
108 |     "        yc = (ymin+ymax)/2\n",
109 |     "        #% compute the adjacency matrix for each graph\n",
110 |     "        # deal with the graphs except the last\n",
111 |     "        for i in range(num_graph-1):\n",
112 |     "            A = np.zeros([num_node,num_node])\n",
113 |     "            coor1 = list()\n",
114 |     "            coor1.append(xc[i*num_node:(i+1)*num_node])\n",
115 |     "            coor1.append(yc[i*num_node:(i+1)*num_node])\n",
116 |     "            coor1 = np.reshape(np.array(coor1),[num_node,2])\n",
117 |     "            edge_coor_1 = list()\n",
118 |     "            edge_index_1 =list()\n",
119 |     "            edge_attr_1 = list()\n",
120 |     "            for k in range(num_node):\n",
121 |     "                for j in range(k+1,num_node):\n",
122 |     "                    # turn to global coordinates\n",
123 |     "                    k1 = i*num_node + k\n",
124 |     "                    j1 = i*num_node + j\n",
125 |     "                    dist = np.sqrt((xc[k1]-xc[j1])**2+(yc[k1]-yc[j1])**2)\n",
126 |     "                    if dist<critical:\n",
127 |     "                        A[k,j] = critical/dist\n",
128 |     "                        A[j,k] = critical/dist\n",
129 |     "                        edge_coor_temp = np.array([xc[k1],yc[k1],xc[j1],yc[j1]],dtype=np.float64)\n",
130 |     "                        edge_coor_1.append(edge_coor_temp)\n",
131 |     "                        edge_coor_temp = np.array([xc[j1],yc[j1],xc[k1],yc[k1]],dtype=np.float64)\n",
132 |     "                        edge_coor_1.append(edge_coor_temp)\n",
133 |     "                        edge_index_temp = np.array([k,j],dtype=np.int)\n",
134 |     "                        edge_index_1.append(edge_index_temp)\n",
135 |     "                        edge_index_temp = np.array([j,k],dtype=np.int)\n",
136 |     "                        edge_index_1.append(edge_index_temp)\n",
137 |     "                        edge_attr_temp = np.array([A[k,j],A[j,k]],dtype=np.float)\n",
138 |     "                        edge_attr_1.append(edge_attr_temp)\n",
139 |     "            A = csr_matrix(A)\n",
140 |     "            adj.append(A)\n",
141 |     "            #print('kid, id_participant[kid]:',kid, id_participant[kid])\n",
142 |     "            tissue_id = row['组织编码']\n",
143 |     "            #print(tissue_id)\n",
144 |     "            pid_name.append(tissue_id)\n",
145 |     "            pid.append(kid)\n",
146 |     "            label.append(label1)\n",
147 |     "            coor.append(coor1)\n",
148 |     "            edge_coor.append(np.reshape(np.array(edge_coor_1),[len(edge_coor_1),4]))\n",
149 |     "            edge_index.append(np.reshape(np.array(edge_index_1),[len(edge_index_1),2]))\n",
150 |     "            edge_attr.append(np.reshape(np.array(edge_attr_1),[len(edge_index_1),1]))\n",
151 |     "            feature.append(feature_all[i*num_node:(i+1)*num_node,])\n",
152 |     "        # deal with the last graph\n",
153 |     "        if np.mod(num_cell,num_node)>0:\n",
154 |     "            num_node_last = int(np.mod(num_cell,num_node))\n",
155 |     "        else:\n",
156 |     "            num_node_last = num_node\n",
157 |     "        coor1 = list()\n",
158 |     "        coor1.append(xc[(i+1)*num_node:])\n",
159 |     "        coor1.append(yc[(i+1)*num_node:])\n",
160 |     "        coor1 = np.reshape(np.array(coor1),[num_node_last,2])\n",
161 |     "        A = np.zeros([num_node_last,num_node_last])\n",
162 |     "        for k in range(num_node_last):\n",
163 |     "            for j in range(k,num_node_last):\n",
164 |     "                dist = np.sqrt((xc[k1]-xc[j1])**2+(yc[k1]-yc[j1])**2)\n",
165 |     "                k1 = i*num_node + k\n",
166 |     "                j1 = i*num_node + j\n",
167 |     "                if dist<critical:\n",
168 |     "                    A[k,j] = critical/dist\n",
169 |     "                    A[j,k] = critical/dist\n",
170 |     "                    edge_coor_temp = np.array([xc[k1],yc[k1],xc[j1],yc[j1]],dtype=np.float64)\n",
171 |     "                    edge_coor_1.append(edge_coor_temp)\n",
172 |     "                    edge_coor_temp = np.array([xc[j1],yc[j1],xc[k1],yc[k1]],dtype=np.float64)\n",
173 |     "                    edge_coor_1.append(edge_coor_temp)\n",
174 |     "                    edge_index_temp = np.array([k,j],dtype=np.int)\n",
175 |     "                    edge_index_1.append(edge_index_temp)\n",
176 |     "                    edge_index_temp = np.array([j,k],dtype=np.int)\n",
177 |     "                    edge_index_1.append(edge_index_temp)\n",
178 |     "                    edge_attr_temp = np.array([A[k,j],A[j,k]],dtype=np.float)\n",
179 |     "                    edge_attr_1.append(edge_attr_temp)\n",
180 |     "        A = csr_matrix(A)\n",
181 |     "    #    print('kid, id_participant[kid]:',kid, id_participant[kid])\n",
182 |     "        adj.append(A)\n",
183 |     "        \n",
184 |     "        tissue_id = row['组织编码']\n",
185 |     "        #print(tissue_id)\n",
186 |     "        pid.append(kid)\n",
187 |     "        pid_name.append(tissue_id)\n",
188 |     "        \n",
189 |     "\n",
190 |     "        label.append(label1)\n",
191 |     "        coor.append(coor1)\n",
192 |     "        edge_coor.append(np.reshape(np.array(edge_coor_1),[len(edge_coor_1),4]))\n",
193 |     "        edge_index.append(np.reshape(np.array(edge_index_1),[len(edge_index_1),2]))\n",
194 |     "        edge_attr.append(np.reshape(np.array(edge_attr_1),[len(edge_index_1),1]))\n",
195 |     "        feature.append(feature_all[i*num_node:(i+1)*num_node,])\n",
196 |     "        \n",
197 |     "    #%% save data\n",
198 |     "    # adj\n",
199 |     "    sv_adj = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_adj_csr' + '.mat'\n",
200 |     "    sio.savemat(sv_adj,mdict={'adj':adj})\n",
201 |     "    # feature\n",
202 |     "    sv_feature = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_feature' + '.mat'\n",
203 |     "    sio.savemat(sv_feature,mdict={'feature':feature})\n",
204 |     "    # label in survival time classes\n",
205 |     "    sv_label = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_label' + '.mat'\n",
206 |     "    sio.savemat(sv_label,mdict={'label':label})\n",
207 |     "    # label in participant id\n",
208 |     "    sv_pid = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_pid' + '.mat'\n",
209 |     "    sio.savemat(sv_pid,mdict={'pid':pid})\n",
210 |     "    # label in participant id name\n",
211 |     "    sv_pid_name = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_pid_name' + '.mat'\n",
212 |     "    sio.savemat(sv_pid_name,mdict={'pid_name':pid_name})\n",
213 |     "    # coordinates of nodes\n",
214 |     "    sv_node_coor = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_nodecoor' + '.mat'\n",
215 |     "    sio.savemat(sv_node_coor,mdict={'coor':coor})\n",
216 |     "    # index of the two ends of edges\n",
217 |     "    sv_edge = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_edge_index' + '.mat'\n",
218 |     "    sio.savemat(sv_edge,mdict={'edge_index':edge_index})\n",
219 |     "    # weight on each edge\n",
220 |     "    sv_edge_attr = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_edge_attr' + '.mat'\n",
221 |     "    sio.savemat(sv_edge_attr,mdict={'edge_attr':edge_attr})\n",
222 |     "    # coordinates of the two ends of edges\n",
223 |     "    sv_edge_coor = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_edge_coor' + '.mat'\n",
224 |     "    sio.savemat(sv_edge_coor,mdict={'edge_coor':edge_coor})\n",
225 |     "        \n",
226 |     "        \n",
227 |     "        \n",
228 |     "        \n",
229 |     "        \n",
230 |     "#create_samples(\"data_file/test_data.xlsx\")\n",
231 |     "create_samples(\"data_file/test_data_surv.xlsx\")\n",
232 |     "\n",
233 |     "# for i in [0,1,2,3,4]:\n",
234 |     "#     create_samples(\"data_file/val_data_fold_{}.xlsx\".format(i))\n",
235 |     "#     create_samples(\"data_file/train_data_fold_{}.xlsx\".format(i))\n",
236 |     "\n",
237 |     "    \n",
238 |     "    \n",
239 |     "    \n",
240 |     "    "
241 |    ]
242 |   },
243 |   {
244 |    "cell_type": "code",
245 |    "execution_count": null,
246 |    "metadata": {},
247 |    "outputs": [],
248 |    "source": []
249 |   }
250 |  ],
251 |  "metadata": {
252 |   "kernelspec": {
253 |    "display_name": "CellStar",
254 |    "language": "python",
255 |    "name": "cellstar"
256 |   },
257 |   "language_info": {
258 |    "codemirror_mode": {
259 |     "name": "ipython",
260 |     "version": 3
261 |    },
262 |    "file_extension": ".py",
263 |    "mimetype": "text/x-python",
264 |    "name": "python",
265 |    "nbconvert_exporter": "python",
266 |    "pygments_lexer": "ipython3",
267 |    "version": "3.7.6"
268 |   }
269 |  },
270 |  "nbformat": 4,
271 |  "nbformat_minor": 4
272 | }
273 | 


--------------------------------------------------------------------------------
/Ternary_data_preprocessing.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "code",
  5 |    "execution_count": 1,
  6 |    "metadata": {},
  7 |    "outputs": [],
  8 |    "source": [
  9 |     "#!/usr/bin/env python3\n",
 10 |     "# -*- coding: utf-8 -*-\n",
 11 |     "\"\"\"\n",
 12 |     "Created on Thu Aug  8 01:49:00 2019\n",
 13 |     "\n",
 14 |     "@author: Yuguang Wang & Yanan Wang\n",
 15 |     "\"\"\"\n",
 16 |     "\n",
 17 |     "import pandas as pd\n",
 18 |     "import numpy as np\n",
 19 |     "import scipy.io as sio\n",
 20 |     "import os\n",
 21 |     "from scipy.sparse import csr_matrix"
 22 |    ]
 23 |   },
 24 |   {
 25 |    "cell_type": "code",
 26 |    "execution_count": 3,
 27 |    "metadata": {},
 28 |    "outputs": [
 29 |     {
 30 |      "name": "stdout",
 31 |      "output_type": "stream",
 32 |      "text": [
 33 |       "****** Processing data for data_file/test_data.xlsx ******\n"
 34 |      ]
 35 |     },
 36 |     {
 37 |      "name": "stderr",
 38 |      "output_type": "stream",
 39 |      "text": [
 40 |       "/home/song-lab/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:123: RuntimeWarning: divide by zero encountered in double_scalars\n",
 41 |       "/home/song-lab/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:124: RuntimeWarning: divide by zero encountered in double_scalars\n"
 42 |      ]
 43 |     },
 44 |     {
 45 |      "name": "stdout",
 46 |      "output_type": "stream",
 47 |      "text": [
 48 |       "****** Processing data for data_file/val_data_fold_0.xlsx ******\n",
 49 |       "****** Processing data for data_file/train_data_fold_0.xlsx ******\n",
 50 |       "****** Processing data for data_file/val_data_fold_1.xlsx ******\n",
 51 |       "****** Processing data for data_file/train_data_fold_1.xlsx ******\n",
 52 |       "****** Processing data for data_file/val_data_fold_2.xlsx ******\n",
 53 |       "****** Processing data for data_file/train_data_fold_2.xlsx ******\n",
 54 |       "****** Processing data for data_file/val_data_fold_3.xlsx ******\n",
 55 |       "****** Processing data for data_file/train_data_fold_3.xlsx ******\n",
 56 |       "****** Processing data for data_file/val_data_fold_4.xlsx ******\n",
 57 |       "****** Processing data for data_file/train_data_fold_4.xlsx ******\n"
 58 |      ]
 59 |     }
 60 |    ],
 61 |    "source": [
 62 |     "def create_samples(file_name):\n",
 63 |     "    subdir = '_{}/'.format(file_name.split(\".\")[0])\n",
 64 |     "    # critical distance 20 \\mu m, pix width 0.5 \\mu m\n",
 65 |     "    critical = 20/0.5\n",
 66 |     "    num_node = 100\n",
 67 |     "    adj = list()\n",
 68 |     "    feature = list()\n",
 69 |     "    label = list()\n",
 70 |     "    pid = list()\n",
 71 |     "    pid_name = list()\n",
 72 |     "    coor = list()\n",
 73 |     "    edge_index = list()\n",
 74 |     "    edge_coor = list()\n",
 75 |     "    edge_attr = list()\n",
 76 |     "    factor_flag = 'selected_new'\n",
 77 |     "    #factor_flag = 'selected'\n",
 78 |     "    #factor_flag = 'full'\n",
 79 |     "    sv_dir = \"data/\" + factor_flag + subdir\n",
 80 |     "    if not os.path.exists(sv_dir):\n",
 81 |     "        os.makedirs(sv_dir)\n",
 82 |     "    \n",
 83 |     "    print('****** Processing data for %s ******' % file_name)\n",
 84 |     "    df_diagnosis = pd.read_excel(file_name)\n",
 85 |     "    id_participant = df_diagnosis['Id']\n",
 86 |     "    num_id_1 = len(id_participant)\n",
 87 |     "    survival_time_all = df_diagnosis['survival_time'].array\n",
 88 |     "    \n",
 89 |     "    for kid, row in df_diagnosis.iterrows():\n",
 90 |     "        ld_csv = row['file_path']\n",
 91 |     "        old_new_flag = ld_csv.split(\"/\")[2]\n",
 92 |     "        #print(ld_csv)\n",
 93 |     "        #print(old_new_flag)\n",
 94 |     "        \n",
 95 |     "        label1 = row['prognosis_label']\n",
 96 |     "        df_csv = pd.read_csv(ld_csv)\n",
 97 |     "        # extract the features\n",
 98 |     "        if factor_flag == 'full': # from \"DAPI (DAPI) Nucleus Intensity\" to end\n",
 99 |     "            lv = np.linspace(22,59,38,dtype=int)\n",
100 |     "        if factor_flag == 'selected_old':\n",
101 |     "            lv = np.array([11,12,18,35,36,37,38,39],dtype=np.int)\n",
102 |     "        if(old_new_flag == 'stomach_csv_1' and factor_flag == 'selected_new'):\n",
103 |     "            lv = np.concatenate((np.linspace(20,44,25,dtype=int),np.linspace(50,59,10,dtype=int)))\n",
104 |     "        elif(old_new_flag == 'stomach_csv_2' and factor_flag == 'selected_new'):\n",
105 |     "            lv = np.concatenate((np.linspace(32,56,25,dtype=int),np.linspace(62,71,10,dtype=int)))\n",
106 |     "        feature_all = df_csv.take(lv,axis=1).values\n",
107 |     "        #print(feature_all)\n",
108 |     "        feature_all = feature_all/(feature_all.max(axis=0)+0.00000000000001) # Add normalization\n",
109 |     "        #print(feature_all)\n",
110 |     "        \n",
111 |     "        \n",
112 |     "        # compute adjacency matrix\n",
113 |     "        xmin = np.array(df_csv['XMin'])\n",
114 |     "        xmax = np.array(df_csv['XMax'])\n",
115 |     "        ymin = np.array(df_csv['YMin'])\n",
116 |     "        ymax = np.array(df_csv['YMax'])\n",
117 |     "        num_cell = len(xmin)\n",
118 |     "        if np.mod(num_cell,num_node)==0:\n",
119 |     "            num_graph = int(num_cell/num_node)\n",
120 |     "        else:\n",
121 |     "            num_graph = int(num_cell/num_node)+1\n",
122 |     "        # compute the centre coordinates of cells\n",
123 |     "        xc = (xmin+xmax)/2\n",
124 |     "        yc = (ymin+ymax)/2\n",
125 |     "        #% compute the adjacency matrix for each graph\n",
126 |     "        # deal with the graphs except the last\n",
127 |     "        for i in range(num_graph-1):\n",
128 |     "            A = np.zeros([num_node,num_node])\n",
129 |     "            coor1 = list()\n",
130 |     "            coor1.append(xc[i*num_node:(i+1)*num_node])\n",
131 |     "            coor1.append(yc[i*num_node:(i+1)*num_node])\n",
132 |     "            coor1 = np.reshape(np.array(coor1),[num_node,2])\n",
133 |     "            edge_coor_1 = list()\n",
134 |     "            edge_index_1 =list()\n",
135 |     "            edge_attr_1 = list()\n",
136 |     "            for k in range(num_node):\n",
137 |     "                for j in range(k+1,num_node):\n",
138 |     "                    # turn to global coordinates\n",
139 |     "                    k1 = i*num_node + k\n",
140 |     "                    j1 = i*num_node + j\n",
141 |     "                    dist = np.sqrt((xc[k1]-xc[j1])**2+(yc[k1]-yc[j1])**2)\n",
142 |     "                    if dist<critical:\n",
143 |     "                        A[k,j] = critical/dist\n",
144 |     "                        A[j,k] = critical/dist\n",
145 |     "                        edge_coor_temp = np.array([xc[k1],yc[k1],xc[j1],yc[j1]],dtype=np.float64)\n",
146 |     "                        edge_coor_1.append(edge_coor_temp)\n",
147 |     "                        edge_coor_temp = np.array([xc[j1],yc[j1],xc[k1],yc[k1]],dtype=np.float64)\n",
148 |     "                        edge_coor_1.append(edge_coor_temp)\n",
149 |     "                        edge_index_temp = np.array([k,j],dtype=np.int)\n",
150 |     "                        edge_index_1.append(edge_index_temp)\n",
151 |     "                        edge_index_temp = np.array([j,k],dtype=np.int)\n",
152 |     "                        edge_index_1.append(edge_index_temp)\n",
153 |     "                        edge_attr_temp = np.array([A[k,j],A[j,k]],dtype=np.float)\n",
154 |     "                        edge_attr_1.append(edge_attr_temp)\n",
155 |     "            A = csr_matrix(A)\n",
156 |     "            adj.append(A)\n",
157 |     "            #print('kid, id_participant[kid]:',kid, id_participant[kid])\n",
158 |     "            tissue_id = row['组织编码']\n",
159 |     "            #print(tissue_id)\n",
160 |     "            pid_name.append(tissue_id)\n",
161 |     "            pid.append(kid)\n",
162 |     "            label.append(label1)\n",
163 |     "            coor.append(coor1)\n",
164 |     "            edge_coor.append(np.reshape(np.array(edge_coor_1),[len(edge_coor_1),4]))\n",
165 |     "            edge_index.append(np.reshape(np.array(edge_index_1),[len(edge_index_1),2]))\n",
166 |     "            edge_attr.append(np.reshape(np.array(edge_attr_1),[len(edge_index_1),1]))\n",
167 |     "            feature.append(feature_all[i*num_node:(i+1)*num_node,])\n",
168 |     "        # deal with the last graph\n",
169 |     "        if np.mod(num_cell,num_node)>0:\n",
170 |     "            num_node_last = int(np.mod(num_cell,num_node))\n",
171 |     "        else:\n",
172 |     "            num_node_last = num_node\n",
173 |     "        coor1 = list()\n",
174 |     "        coor1.append(xc[(i+1)*num_node:])\n",
175 |     "        coor1.append(yc[(i+1)*num_node:])\n",
176 |     "        coor1 = np.reshape(np.array(coor1),[num_node_last,2])\n",
177 |     "        A = np.zeros([num_node_last,num_node_last])\n",
178 |     "        for k in range(num_node_last):\n",
179 |     "            for j in range(k,num_node_last):\n",
180 |     "                dist = np.sqrt((xc[k1]-xc[j1])**2+(yc[k1]-yc[j1])**2)\n",
181 |     "                k1 = i*num_node + k\n",
182 |     "                j1 = i*num_node + j\n",
183 |     "                if dist<critical:\n",
184 |     "                    A[k,j] = critical/dist\n",
185 |     "                    A[j,k] = critical/dist\n",
186 |     "                    edge_coor_temp = np.array([xc[k1],yc[k1],xc[j1],yc[j1]],dtype=np.float64)\n",
187 |     "                    edge_coor_1.append(edge_coor_temp)\n",
188 |     "                    edge_coor_temp = np.array([xc[j1],yc[j1],xc[k1],yc[k1]],dtype=np.float64)\n",
189 |     "                    edge_coor_1.append(edge_coor_temp)\n",
190 |     "                    edge_index_temp = np.array([k,j],dtype=np.int)\n",
191 |     "                    edge_index_1.append(edge_index_temp)\n",
192 |     "                    edge_index_temp = np.array([j,k],dtype=np.int)\n",
193 |     "                    edge_index_1.append(edge_index_temp)\n",
194 |     "                    edge_attr_temp = np.array([A[k,j],A[j,k]],dtype=np.float)\n",
195 |     "                    edge_attr_1.append(edge_attr_temp)\n",
196 |     "        A = csr_matrix(A)\n",
197 |     "    #    print('kid, id_participant[kid]:',kid, id_participant[kid])\n",
198 |     "        adj.append(A)\n",
199 |     "        \n",
200 |     "        tissue_id = row['组织编码']\n",
201 |     "        #print(tissue_id)\n",
202 |     "        pid.append(kid)\n",
203 |     "        pid_name.append(tissue_id)\n",
204 |     "        \n",
205 |     "\n",
206 |     "        label.append(label1)\n",
207 |     "        coor.append(coor1)\n",
208 |     "        edge_coor.append(np.reshape(np.array(edge_coor_1),[len(edge_coor_1),4]))\n",
209 |     "        edge_index.append(np.reshape(np.array(edge_index_1),[len(edge_index_1),2]))\n",
210 |     "        edge_attr.append(np.reshape(np.array(edge_attr_1),[len(edge_index_1),1]))\n",
211 |     "        feature.append(feature_all[i*num_node:(i+1)*num_node,])\n",
212 |     "        \n",
213 |     "    #%% save data\n",
214 |     "    # adj\n",
215 |     "    sv_adj = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_adj_csr' + '.mat'\n",
216 |     "    sio.savemat(sv_adj,mdict={'adj':adj})\n",
217 |     "    # feature\n",
218 |     "    sv_feature = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_feature' + '.mat'\n",
219 |     "    sio.savemat(sv_feature,mdict={'feature':feature})\n",
220 |     "    # label in survival time classes\n",
221 |     "    sv_label = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_label' + '.mat'\n",
222 |     "    sio.savemat(sv_label,mdict={'label':label})\n",
223 |     "    # label in participant id\n",
224 |     "    sv_pid = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_pid' + '.mat'\n",
225 |     "    sio.savemat(sv_pid,mdict={'pid':pid})\n",
226 |     "    # label in participant id name\n",
227 |     "    sv_pid_name = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_pid_name' + '.mat'\n",
228 |     "    sio.savemat(sv_pid_name,mdict={'pid_name':pid_name})\n",
229 |     "    # coordinates of nodes\n",
230 |     "    sv_node_coor = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_nodecoor' + '.mat'\n",
231 |     "    sio.savemat(sv_node_coor,mdict={'coor':coor})\n",
232 |     "    # index of the two ends of edges\n",
233 |     "    sv_edge = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_edge_index' + '.mat'\n",
234 |     "    sio.savemat(sv_edge,mdict={'edge_index':edge_index})\n",
235 |     "    # weight on each edge\n",
236 |     "    sv_edge_attr = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_edge_attr' + '.mat'\n",
237 |     "    sio.savemat(sv_edge_attr,mdict={'edge_attr':edge_attr})\n",
238 |     "    # coordinates of the two ends of edges\n",
239 |     "    sv_edge_coor = sv_dir + 'graph' + str(len(label)) + '_node' + str(num_node) + '_weighted' + '_edge_coor' + '.mat'\n",
240 |     "    sio.savemat(sv_edge_coor,mdict={'edge_coor':edge_coor})\n",
241 |     "        \n",
242 |     "        \n",
243 |     "        \n",
244 |     "        \n",
245 |     "        \n",
246 |     "create_samples(\"data_file/test_data.xlsx\")\n",
247 |     "\n",
248 |     "for i in [0,1,2,3,4]:\n",
249 |     "    create_samples(\"data_file/val_data_fold_{}.xlsx\".format(i))\n",
250 |     "    create_samples(\"data_file/train_data_fold_{}.xlsx\".format(i))\n",
251 |     "\n",
252 |     "    \n",
253 |     "    \n",
254 |     "    \n",
255 |     "    "
256 |    ]
257 |   },
258 |   {
259 |    "cell_type": "code",
260 |    "execution_count": null,
261 |    "metadata": {},
262 |    "outputs": [],
263 |    "source": []
264 |   }
265 |  ],
266 |  "metadata": {
267 |   "kernelspec": {
268 |    "display_name": "CellStar",
269 |    "language": "python",
270 |    "name": "cellstar"
271 |   },
272 |   "language_info": {
273 |    "codemirror_mode": {
274 |     "name": "ipython",
275 |     "version": 3
276 |    },
277 |    "file_extension": ".py",
278 |    "mimetype": "text/x-python",
279 |    "name": "python",
280 |    "nbconvert_exporter": "python",
281 |    "pygments_lexer": "ipython3",
282 |    "version": "3.7.6"
283 |   }
284 |  },
285 |  "nbformat": 4,
286 |  "nbformat_minor": 4
287 | }
288 | 


--------------------------------------------------------------------------------
/scdiag_gintopk_roc_ternary.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "code",
  5 |    "execution_count": 1,
  6 |    "metadata": {},
  7 |    "outputs": [],
  8 |    "source": [
  9 |     "#!/usr/bin/env python\n",
 10 |     "# coding: utf-8\n",
 11 |     "\n",
 12 |     "# # Stomach Cancer Diagnosis by Graph Neural Networks\n",
 13 |     "# ##  PyTorch Implementation for GCN + TopKPooling with LeNet5 Architecture \n",
 14 |     "# ### Wang, Y. G. @UNSW, Dec 2019\n",
 15 |     "# ### Yanan Wang, @Monash\n",
 16 |     "\n",
 17 |     "# ## Main Program\n",
 18 |     "\n",
 19 |     "# In[ ]:\n",
 20 |     "\n",
 21 |     "# import os\n",
 22 |     "import time\n",
 23 |     "from matplotlib import pyplot as plt\n",
 24 |     "import numpy as np\n",
 25 |     "import scipy.io as sio\n",
 26 |     "from sklearn.metrics import confusion_matrix\n",
 27 |     "import matplotlib\n",
 28 |     "import argparse\n",
 29 |     "from sklearn.metrics import f1_score, accuracy_score, auc, roc_curve\n",
 30 |     "import torch\n",
 31 |     "import torch.nn.functional as F\n",
 32 |     "from torch.nn import Sequential as Seq, Linear as Lin, ReLU\n",
 33 |     "from torch_geometric.data import Data, DataLoader\n",
 34 |     "from torch_geometric.nn import GraphConv, GINConv, TopKPooling\n",
 35 |     "from torch_geometric.nn import global_mean_pool as gap, global_max_pool as gmp\n",
 36 |     "# from torch.utils.data import random_split\n",
 37 |     "# from torch_geometric.data import InMemoryDataset\n",
 38 |     "# import matplotlib.pyplot as plt\n",
 39 |     "# import pandas as pd\n",
 40 |     "\n",
 41 |     "def trim_axs(axs, N):\n",
 42 |     "    \"\"\"little helper to massage the axs list to have correct length...\"\"\"\n",
 43 |     "    axs = axs.flat\n",
 44 |     "    for ax in axs[N:]:\n",
 45 |     "        ax.remove()\n",
 46 |     "    return axs[:N]\n",
 47 |     "\n",
 48 |     "def plot_multi_label_confusion_matrix(_save_path, y_true, y_pred, labels, normalize=False, title=None, cmap=plt.cm.Blues):\n",
 49 |     "    plt.close('all')\n",
 50 |     "    plt.style.use(\"ggplot\")\n",
 51 |     "    plt.rcParams.update({'font.size': 8})\n",
 52 |     "    plt.rcParams.update({'font.family':'Arial'})\n",
 53 |     "    conf_mat_dict={}\n",
 54 |     "\n",
 55 |     "    class_num = len(labels)\n",
 56 |     "\n",
 57 |     "    plot_rows = int(class_num/4)+1\n",
 58 |     "    plot_cols = 4 if class_num>=4 else class_num\n",
 59 |     "\n",
 60 |     "    for label_col in range(class_num):\n",
 61 |     "        y_true_label = y_true[:, label_col]\n",
 62 |     "        y_pred_label = y_pred[:, label_col]\n",
 63 |     "\n",
 64 |     "        print(y_true_label)\n",
 65 |     "        print(y_pred_label)\n",
 66 |     "        conf_mat_dict[labels[label_col]] = confusion_matrix(y_pred=y_pred_label, y_true=y_true_label)\n",
 67 |     "\n",
 68 |     "\n",
 69 |     "    fig, axes = plt.subplots(nrows=plot_rows, ncols=plot_cols, sharex=False, sharey=False,gridspec_kw = {'wspace':0.5, 'hspace':0.05},figsize=(10,10))\n",
 70 |     "    axes = trim_axs(axes, class_num)\n",
 71 |     "    for ii in range(len(labels)):\n",
 72 |     "        _label = labels[ii]\n",
 73 |     "        _matrix = conf_mat_dict[_label]\n",
 74 |     "        axes[ii].imshow(_matrix,interpolation='nearest', cmap=plt.cm.Blues)\n",
 75 |     "        axes[ii].set(xticks=np.arange(_matrix.shape[1]),\n",
 76 |     "               yticks=np.arange(_matrix.shape[0]),\n",
 77 |     "               # ... and label them with the respective list entries\n",
 78 |     "               xticklabels=[\"Neg\",\"Pos\"], yticklabels=[\"Neg\",\"Pos\"],\n",
 79 |     "               title=_label,\n",
 80 |     "               ylabel='True label',\n",
 81 |     "               xlabel='Predicted label')\n",
 82 |     "        fmt = 'd'\n",
 83 |     "        thresh = _matrix.max() / 2.\n",
 84 |     "        for i in range(_matrix.shape[0]):\n",
 85 |     "            for j in range(_matrix.shape[1]):\n",
 86 |     "                axes[ii].text(j, i, format(_matrix[i, j], fmt),\n",
 87 |     "                    ha=\"center\", va=\"center\", fontsize=8,\n",
 88 |     "                    color=\"white\" if _matrix[i, j] > thresh else \"black\")\n",
 89 |     "\n",
 90 |     "    plt.savefig(_save_path, dpi=100,pad_inches = 0.1,bbox_inches = 'tight')\n",
 91 |     "\n",
 92 |     "\n",
 93 |     "# In[ ]:\n",
 94 |     "\n",
 95 |     "def calculate_metrics(gts, ops, preds, class_num, labels, outputs, mode):\n",
 96 |     "    if mode:\n",
 97 |     "        gts = np.vstack([gts, labels.cpu()]) if gts.size else labels.cpu()\n",
 98 |     "        y_pred = outputs.unsqueeze(1)\n",
 99 |     "        y_pred = torch.cat([1.0 - y_pred, y_pred], dim=1)\n",
100 |     "        y_pred = torch.max(y_pred, dim=1)[1]\n",
101 |     "        # print(\"Predict is %s\"%y_pred)\n",
102 |     "        preds = np.vstack([preds, y_pred.cpu()]) if preds.size else y_pred.cpu()\n",
103 |     "    else:\n",
104 |     "        _labels = labels.cpu()\n",
105 |     "        tmp = torch.zeros(len(_labels), class_num)\n",
106 |     "        for idx, ele in enumerate(_labels):\n",
107 |     "            tmp[idx][ele] = 1\n",
108 |     "        gts = np.vstack([gts, tmp]) if gts.size else tmp\n",
109 |     "        view = outputs.view(-1, class_num)\n",
110 |     "        y_pred = (view == view.max(dim=1, keepdim=True)[0]).view_as(outputs).type(torch.ByteTensor)\n",
111 |     "            # y_pred = torch.max(outputs, 1)[1].view(labels.size())\n",
112 |     "            # y_pred = np.argmax(y_pred.cpu())\n",
113 |     "            # print(y_pred)\n",
114 |     "        preds = np.vstack([preds, y_pred.cpu()]) if preds.size else y_pred.cpu()\n",
115 |     "\n",
116 |     "    acc_list = []\n",
117 |     "    auc_list = []\n",
118 |     "    f1 = f1_score(gts, preds, average=\"micro\")\n",
119 |     "    for j in range(0, class_num):\n",
120 |     "        gts_i = gts[:,j]\n",
121 |     "        preds_i = preds[:,j]\n",
122 |     "        ops_i = ops[:,j]\n",
123 |     "        fpr, tpr, thresholds = roc_curve(gts_i, ops_i)\n",
124 |     "        acc_score = accuracy_score(gts_i, preds_i)\n",
125 |     "        auc_score = auc(fpr, tpr)\n",
126 |     "        acc_list.append(acc_score)\n",
127 |     "        auc_list.append(auc_score)\n",
128 |     "        print(\"class_num: %d, acc_score: %f, auc_score: %f\"%(j, acc_score, auc_score))\n",
129 |     "    return acc_list, auc_list, f1, gts, ops, preds\n",
130 |     "\n",
131 |     "\n",
132 |     "def plot_confusion_matrix(_model, y_true, y_pred, classes, normalize=False, title=None, cmap=plt.cm.Blues):\n",
133 |     "\n",
134 |     "    plot_multi_label_confusion_matrix('/home/yuguang/cellstar/figures/%s_Confusion_matrix.png' % _model, y_true, y_pred, classes)\n",
135 |     "\n",
136 |     "    \n",
137 |     "def plot_roc_curve(pred_y, test_y, class_label, n_classes, fig_name=\"roc_auc.png\"):\n",
138 |     "    #pred_y = pred_y/pred_y.max(axis=0)\n",
139 |     "    colors = [\"#E69F00\", \"#56B4E9\", \"#009E73\", \"#F0E442\", \"#0072B2\", \"#D55E00\", \"#CC79A7\", \"#000000\", \"#66CC99\", \"#999999\"]\n",
140 |     "    plt.close('all')\n",
141 |     "    plt.style.use(\"ggplot\")\n",
142 |     "    matplotlib.rcParams['font.family'] = \"Arial\"\n",
143 |     "    plt.figure(figsize=(8, 8), dpi=400)\n",
144 |     "    for i in range(n_classes):\n",
145 |     "        _tmp_pred = pred_y\n",
146 |     "        _tmp_label = test_y\n",
147 |     "        #print(_tmp_label[:, 0], _tmp_pred[:, 0])\n",
148 |     "        _fpr, _tpr, _ = roc_curve(_tmp_label[:, i], _tmp_pred[:, i])\n",
149 |     "        _auc = auc(_fpr, _tpr)\n",
150 |     "        plt.plot(_fpr, _tpr, color=colors[i],\n",
151 |     "                 label=r'%s ROC (AUC = %0.3f)' % (class_label[i], _auc), lw=2, alpha=.9)\n",
152 |     "    plt.plot([0, 1], [0, 1], 'k--', lw=2)\n",
153 |     "    plt.xlim([0.0, 1.01])\n",
154 |     "    plt.ylim([0.0, 1.01])\n",
155 |     "    plt.xlabel('False Positive Rate')\n",
156 |     "    plt.ylabel('True Positive Rate')\n",
157 |     "    #plt.title('ROC curve of')\n",
158 |     "    plt.legend(loc=\"lower right\")\n",
159 |     "    plt.savefig(fig_name, dpi=400)\n",
160 |     "    plt.close('all')\n",
161 |     "\n",
162 |     "##Define Model Class\n",
163 |     "class GCNTopK(torch.nn.Module):\n",
164 |     "    def __init__(self, num_feature, num_class, nhid=256, pooling_ratio=0.75):\n",
165 |     "        super(GCNTopK, self).__init__()\n",
166 |     "        self.nhid = nhid\n",
167 |     "        self.pooling_ratio = pooling_ratio\n",
168 |     "        self.conv1 = GraphConv(int(num_feature), self.nhid)\n",
169 |     "        self.pool1 = TopKPooling(self.nhid, ratio = self.pooling_ratio) # edited by Ming with concern for further extension\n",
170 |     "        self.conv2 = GraphConv(self.nhid, self.nhid)\n",
171 |     "        self.pool2 = TopKPooling(self.nhid, ratio = self.pooling_ratio)\n",
172 |     "        self.conv3 = GraphConv(self.nhid, self.nhid)\n",
173 |     "        self.pool3 = TopKPooling(self.nhid, ratio = self.pooling_ratio)\n",
174 |     "        #add one more conv-pooling block, i.e., conv4 and pool4\n",
175 |     "        self.conv4 = GraphConv(self.nhid, self.nhid)\n",
176 |     "        self.pool4 = TopKPooling(self.nhid, ratio = self.pooling_ratio)\n",
177 |     "\n",
178 |     "        self.lin1 = torch.nn.Linear(self.nhid*2, self.nhid)   # edited by Ming with concern for further extension\n",
179 |     "        self.lin2 = torch.nn.Linear(self.nhid, self.nhid//2)\n",
180 |     "        self.lin3 = torch.nn.Linear(self.nhid//2, num_class)  # edited by Ming with concern for further extension\n",
181 |     "\n",
182 |     "    def forward(self, data):\n",
183 |     "        x, edge_index, edge_attr, batch = data.x, data.edge_index, data.edge_attr, data.batch\n",
184 |     "\n",
185 |     "        x = F.relu(self.conv1(x, edge_index))\n",
186 |     "        x, edge_index, edge_attr, batch, _, _ = self.pool1(x, edge_index, edge_attr, batch)\n",
187 |     "        x1 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
188 |     "\n",
189 |     "        x = F.relu(self.conv2(x, edge_index))\n",
190 |     "        x, edge_index, edge_attr, batch, _, _ = self.pool2(x, edge_index, edge_attr, batch)\n",
191 |     "        x2 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
192 |     "\n",
193 |     "        x = F.relu(self.conv3(x, edge_index))\n",
194 |     "        x, edge_index, edge_attr, batch, _, _ = self.pool3(x, edge_index, edge_attr, batch)\n",
195 |     "        x3 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
196 |     "        #add one more conv-pooling block, corresponding to conv4 and pool4\n",
197 |     "        x = F.relu(self.conv4(x, edge_index))\n",
198 |     "        x, edge_index, edge_attr, batch, _, _ = self.pool4(x, edge_index, edge_attr, batch)\n",
199 |     "        x4 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
200 |     "\n",
201 |     "        x = x1 + x2 + x3 + x4\n",
202 |     "#         x = x1 + x2 + x3\n",
203 |     "\n",
204 |     "        x = F.relu(self.lin1(x))\n",
205 |     "        x = F.dropout(x, p=0.5, training=self.training)\n",
206 |     "        x = F.relu(self.lin2(x))\n",
207 |     "#         print('shape of x before log_softmax: ',x.shape)\n",
208 |     "        y1 = F.log_softmax(self.lin3(x), dim=-1)\n",
209 |     "#         print('shape of x after log_softmax: ',x.shape)\n",
210 |     "        y2 = torch.sigmoid(self.lin3(x))\n",
211 |     "\n",
212 |     "        return y1, y2\n",
213 |     "    \n",
214 |     "##GCNSag\n",
215 |     "class GCNSag(torch.nn.Module):\n",
216 |     "    def __init__(self, num_feature, num_class, nhid=256, pooling_ratio=0.75):\n",
217 |     "        super(GCNSag, self).__init__()\n",
218 |     "        self.nhid = nhid\n",
219 |     "        self.pooling_ratio = pooling_ratio\n",
220 |     "        self.conv1 = GCNConv(int(num_feature), self.nhid)\n",
221 |     "        self.pool1 = SAGPooling(self.nhid, min_score=0.001, GNN=GCNConv) # edited by Ming with concern for further extension\n",
222 |     "        self.conv2 = GCNConv(self.nhid, self.nhid)\n",
223 |     "        self.pool2 = SAGPooling(self.nhid, min_score=0.001, GNN=GCNConv)\n",
224 |     "        self.conv3 = GCNConv(self.nhid, self.nhid)\n",
225 |     "        self.pool3 = SAGPooling(self.nhid, min_score=0.001, GNN=GCNConv)\n",
226 |     "        #add one more conv-pooling block, i.e., conv4 and pool4\n",
227 |     "        self.conv4 = GCNConv(self.nhid, self.nhid)\n",
228 |     "        self.pool4 = SAGPooling(self.nhid, min_score=0.001, GNN=GCNConv)\n",
229 |     "\n",
230 |     "        self.lin1 = torch.nn.Linear(self.nhid*2, self.nhid)   # edited by Ming with concern for further extension\n",
231 |     "        self.lin2 = torch.nn.Linear(self.nhid, self.nhid//2)\n",
232 |     "        self.lin3 = torch.nn.Linear(self.nhid//2, num_class)  # edited by Ming with concern for further extension\n",
233 |     "\n",
234 |     "    def forward(self, data):\n",
235 |     "        x, edge_index, edge_attr, batch = data.x, data.edge_index, data.edge_attr, data.batch\n",
236 |     "\n",
237 |     "        x = F.relu(self.conv1(x, edge_index))\n",
238 |     "        x, edge_index, edge_attr, batch, _, _ = self.pool1(x, edge_index, edge_attr, batch)\n",
239 |     "        x1 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
240 |     "\n",
241 |     "        x = F.relu(self.conv2(x, edge_index))\n",
242 |     "        x, edge_index, edge_attr, batch, _, _ = self.pool2(x, edge_index, edge_attr, batch)\n",
243 |     "        x2 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
244 |     "\n",
245 |     "        x = F.relu(self.conv3(x, edge_index))\n",
246 |     "        x, edge_index, edge_attr, batch, _, _ = self.pool3(x, edge_index, edge_attr, batch)\n",
247 |     "        x3 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
248 |     "        #add one more conv-pooling block, corresponding to conv4 and pool4\n",
249 |     "        x = F.relu(self.conv4(x, edge_index))\n",
250 |     "        x, edge_index, edge_attr, batch, _, _ = self.pool4(x, edge_index, edge_attr, batch)\n",
251 |     "        x4 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
252 |     "\n",
253 |     "        x = x1 + x2 + x3 + x4\n",
254 |     "#         x = x1 + x2 + x3\n",
255 |     "\n",
256 |     "        x = F.relu(self.lin1(x))\n",
257 |     "        x = F.dropout(x, p=0.5, training=self.training)\n",
258 |     "        x = F.relu(self.lin2(x))\n",
259 |     "#         print('shape of x before log_softmax: ',x.shape)\n",
260 |     "        y1 = F.log_softmax(self.lin3(x), dim=-1)\n",
261 |     "#         print('shape of x after log_softmax: ',x.shape)\n",
262 |     "        y2 = torch.sigmoid(self.lin3(x))\n",
263 |     "\n",
264 |     "        return y1, y2    \n",
265 |     "\n",
266 |     "    \n",
267 |     "##GINTopK\n",
268 |     "class GINTopK(torch.nn.Module):\n",
269 |     "    def __init__(self, num_feature, num_class, nhid):\n",
270 |     "        super(GINTopK, self).__init__()\n",
271 |     "        self.conv1 = GINConv(Seq(Lin(num_feature, nhid), ReLU(), Lin(nhid, nhid)))\n",
272 |     "        self.pool1 = TopKPooling(nhid, ratio=0.8)\n",
273 |     "        self.conv2 = GINConv(Seq(Lin(nhid, nhid), ReLU(), Lin(nhid, nhid)))\n",
274 |     "        self.pool2 = TopKPooling(nhid, ratio=0.8)\n",
275 |     "        self.conv3 = GINConv(Seq(Lin(nhid, nhid), ReLU(), Lin(nhid, nhid)))\n",
276 |     "        self.pool3 = TopKPooling(nhid, ratio=0.8)\n",
277 |     "        self.conv4 = GINConv(Seq(Lin(nhid, nhid), ReLU(), Lin(nhid, nhid)))\n",
278 |     "        self.pool4 = TopKPooling(nhid, ratio=0.8)\n",
279 |     "\n",
280 |     "        self.lin1 = torch.nn.Linear(2*nhid, nhid)\n",
281 |     "        self.lin2 = torch.nn.Linear(nhid, nhid//2)\n",
282 |     "        self.lin3 = torch.nn.Linear(nhid//2, num_class)\n",
283 |     "\n",
284 |     "    def forward(self, data):\n",
285 |     "        x, edge_index, batch = data.x, data.edge_index, data.batch\n",
286 |     "\n",
287 |     "        x = F.relu(self.conv1(x, edge_index))\n",
288 |     "        x, edge_index, _, batch, _, _ = self.pool1(x, edge_index, None, batch)\n",
289 |     "        x1 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
290 |     "\n",
291 |     "        x = F.relu(self.conv2(x, edge_index))\n",
292 |     "        x, edge_index, _, batch, _, _ = self.pool2(x, edge_index, None, batch)\n",
293 |     "        x2 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
294 |     "\n",
295 |     "        x = F.relu(self.conv3(x, edge_index))\n",
296 |     "        x, edge_index, _, batch, _, _ = self.pool3(x, edge_index, None, batch)\n",
297 |     "        x3 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
298 |     "        \n",
299 |     "        x = F.relu(self.conv4(x, edge_index))\n",
300 |     "        x, edge_index, _, batch, _, _ = self.pool4(x, edge_index, None, batch)\n",
301 |     "        x4 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
302 |     "\n",
303 |     "        x = x1 + x2 + x3 + x4\n",
304 |     "\n",
305 |     "        x = F.relu(self.lin1(x))\n",
306 |     "        x = F.dropout(x, p=0.5, training=self.training)\n",
307 |     "        x = F.relu(self.lin2(x))\n",
308 |     "        y1 = F.log_softmax(self.lin3(x), dim=-1)\n",
309 |     "        y2 = torch.sigmoid(self.lin3(x))\n",
310 |     "\n",
311 |     "        return y1, y2\n",
312 |     "    \n",
313 |     "\n",
314 |     "##GINSAG\n",
315 |     "class GINSAG(torch.nn.Module):\n",
316 |     "    def __init__(self, num_feature, num_class, nhid):\n",
317 |     "        super(Net, self).__init__()\n",
318 |     "        self.conv1 = GINConv(Seq(Lin(num_feature, nhid), ReLU(), Lin(nhid, nhid)))\n",
319 |     "        self.pool1 = SAGPooling(nhid, min_score=0.001, GNN=GCNConv)\n",
320 |     "        self.conv2 = GINConv(Seq(Lin(nhid, nhid), ReLU(), Lin(nhid, nhid)))\n",
321 |     "        self.pool2 = SAGPooling(nhid, min_score=0.001, GNN=GCNConv)\n",
322 |     "        self.conv3 = GINConv(Seq(Lin(nhid, nhid), ReLU(), Lin(nhid, nhid)))\n",
323 |     "        self.pool3 = SAGPooling(nhid, min_score=0.001, GNN=GCNConv)\n",
324 |     "        self.conv4 = GINConv(Seq(Lin(nhid, nhid), ReLU(), Lin(nhid, nhid)))\n",
325 |     "        self.pool4 = SAGPooling(nhid, min_score=0.001, GNN=GCNConv)\n",
326 |     "\n",
327 |     "        self.lin1 = torch.nn.Linear(2*nhid, nhid)\n",
328 |     "        self.lin2 = torch.nn.Linear(nhid, nhid//2)\n",
329 |     "        self.lin3 = torch.nn.Linear(nhid//2, num_class)\n",
330 |     "\n",
331 |     "    def forward(self, data):\n",
332 |     "        x, edge_index, batch = data.x, data.edge_index, data.batch\n",
333 |     "\n",
334 |     "        x = F.relu(self.conv1(x, edge_index))\n",
335 |     "        x, edge_index, _, batch, _, _ = self.pool1(x, edge_index, None, batch)\n",
336 |     "        x1 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
337 |     "\n",
338 |     "        x = F.relu(self.conv2(x, edge_index))\n",
339 |     "        x, edge_index, _, batch, _, _ = self.pool2(x, edge_index, None, batch)\n",
340 |     "        x2 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
341 |     "\n",
342 |     "        x = F.relu(self.conv3(x, edge_index))\n",
343 |     "        x, edge_index, _, batch, _, _ = self.pool3(x, edge_index, None, batch)\n",
344 |     "        x3 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
345 |     "        \n",
346 |     "        x = F.relu(self.conv4(x, edge_index))\n",
347 |     "        x, edge_index, _, batch, _, _ = self.pool4(x, edge_index, None, batch)\n",
348 |     "        x4 = torch.cat([gmp(x, batch), gap(x, batch)], dim=1)\n",
349 |     "\n",
350 |     "        x = x1 + x2 + x3 + x4\n",
351 |     "\n",
352 |     "        x = F.relu(self.lin1(x))\n",
353 |     "        x = F.dropout(x, p=0.5, training=self.training)\n",
354 |     "        x = F.relu(self.lin2(x))\n",
355 |     "        y1 = F.log_softmax(self.lin3(x), dim=-1)\n",
356 |     "        y2 = torch.sigmoid(self.lin3(x))\n",
357 |     "\n",
358 |     "        return y1, y2    \n",
359 |     "    \n",
360 |     "\n",
361 |     "def train(model,train_loader,device):\n",
362 |     "    model.train()\n",
363 |     "\n",
364 |     "    loss_all = 0\n",
365 |     "    for data in train_loader:\n",
366 |     "        data = data.to(device)\n",
367 |     "        optimizer.zero_grad()\n",
368 |     "        output, _ = model(data)\n",
369 |     "        loss = F.nll_loss(output, data.y)\n",
370 |     "        loss.backward()\n",
371 |     "        loss_all += data.num_graphs * loss.item()\n",
372 |     "        optimizer.step()\n",
373 |     "    return loss_all / len(train_loader.dataset)\n",
374 |     "      \n",
375 |     "def test(model,loader):\n",
376 |     "    model.eval()\n",
377 |     "    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
378 |     "    correct = 0.\n",
379 |     "    loss = 0.   # edited by Ming with concern for further extension\n",
380 |     "    pred_1 = list()\n",
381 |     "    out_1 = np.array([])\n",
382 |     "    gt_l = np.array([])\n",
383 |     "    pred_bi = np.array([])\n",
384 |     "    label = np.array([])\n",
385 |     "    for data in loader:\n",
386 |     "        data = data.to(device)\n",
387 |     "        out, out2 = model(data)\n",
388 |     "#         print('out, out2 in test: ',out,out2)\n",
389 |     "        pred = out.max(dim=1)[1]\n",
390 |     "        correct += pred.eq(data.y).sum().item()\n",
391 |     "        loss += F.nll_loss(out, data.y,reduction='sum').item()\n",
392 |     "        \n",
393 |     "        pred_1.append(pred.cpu().detach().numpy())\n",
394 |     "        out_1 = np.vstack([out_1, out2.cpu().detach().numpy()]) if out_1.size else out2.cpu().detach().numpy()\n",
395 |     "        _tmp_label = data.y.cpu().detach().numpy()\n",
396 |     "        for _label in _tmp_label:\n",
397 |     "            if(_label == 0):\n",
398 |     "                _label_3d = np.array([1, 0, 0])\n",
399 |     "            elif(_label == 1):\n",
400 |     "                _label_3d = np.array([0, 1, 0])\n",
401 |     "            elif(_label == 2):\n",
402 |     "                _label_3d = np.array([0, 0, 1])\n",
403 |     "            gt_l = np.vstack([gt_l, _label_3d]) if gt_l.size else _label_3d\n",
404 |     "        for _pred in pred:\n",
405 |     "            if(_pred == 0):\n",
406 |     "                _pred_bi = np.array([1, 0, 0])\n",
407 |     "            if(_pred == 1):\n",
408 |     "                _pred_bi = np.array([0, 1, 0])\n",
409 |     "            if(_pred == 2):\n",
410 |     "                _pred_bi = np.array([0, 0, 1])\n",
411 |     "            pred_bi = np.vstack([pred_bi,_pred_bi]) if pred_bi.size else _pred_bi\n",
412 |     "        label = np.hstack([label,_tmp_label]) if label.size else _tmp_label\n",
413 |     "    # pred_1 = np.array(pred_1).reshape(pred_1)\n",
414 |     "    return correct *1.0 / len(loader.dataset), loss / len(loader.dataset), pred_1, out_1, gt_l, label, pred_bi"
415 |    ]
416 |   },
417 |   {
418 |    "cell_type": "code",
419 |    "execution_count": 6,
420 |    "metadata": {},
421 |    "outputs": [
422 |     {
423 |      "name": "stdout",
424 |      "output_type": "stream",
425 |      "text": [
426 |       "Device: cuda:0\n"
427 |      ]
428 |     }
429 |    ],
430 |    "source": [
431 |     "# import argparse\n",
432 |     "#def hyperopt_train(batch_size=256, learning_rate=0.01, weight_decay=0.0005, nhid=256, pooling_ratio=0.75, epochs=200, runs=1):\n",
433 |     "         ## Parameter Setting\n",
434 |     "     #added by ming for future pooling extensions\n",
435 |     "    \n",
436 |     "# parser = argparse.ArgumentParser()\n",
437 |     "# parser.add_argument('--batch_size', type=int, default=256,\n",
438 |     "#                  help='batch size')\n",
439 |     "# parser.add_argument('--learning_rate', type=float, default=5e-4,\n",
440 |     "#                  help='learning rate')\n",
441 |     "# parser.add_argument('--weight_decay', type=float, default=1e-4,\n",
442 |     "#                  help='weight decay')\n",
443 |     "# parser.add_argument('--nhid', type=int, default=512,\n",
444 |     "#                  help='hidden size')\n",
445 |     "# parser.add_argument('--pooling_ratio', type=float, default=0.5,\n",
446 |     "#                  help='pooling ratio')\n",
447 |     "# parser.add_argument('--epochs', type=int, default=200,\n",
448 |     "#                  help='maximum number of epochs')\n",
449 |     "# # parser.add_argument('--early_stopping', type=int, default=100,\n",
450 |     "# #                  help='patience for earlystopping')\n",
451 |     "# parser.add_argument('--num_layers', type=int, default=4,\n",
452 |     "#                  help='number of layers')\n",
453 |     "# parser.add_argument('--runs', type=int, default=1,\n",
454 |     "#                  help='number of runs')\n",
455 |     "# args = parser.parse_args()\n",
456 |     "\n",
457 |     "# batch_size = args.batch_size\n",
458 |     "# learning_rate = args.learning_rate\n",
459 |     "# weight_decay = args.weight_decay\n",
460 |     "# nhid = args.nhid\n",
461 |     "# pooling_ratio = args.pooling_ratio\n",
462 |     "# epochs = args.epochs\n",
463 |     "# # early_stopping = args.early_stopping\n",
464 |     "# num_layers = args.num_layers\n",
465 |     "# runs = args.runs\n",
466 |     "\n",
467 |     "batch_size = 256\n",
468 |     "learning_rate = 5e-4\n",
469 |     "weight_decay = 1e-4\n",
470 |     "nhid = 512\n",
471 |     "pooling_ratio = 0.5\n",
472 |     "epochs = 200\n",
473 |     "# early_stopping = args.early_stopping\n",
474 |     "num_layers = 4\n",
475 |     "\n",
476 |     "model_name = \"gintopk\"\n",
477 |     "runs = 1\n",
478 |     "fold = 4\n",
479 |     "\n",
480 |     "# early_stopping = epochs\n",
481 |     "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
482 |     "print('Device: {}'.format(device))"
483 |    ]
484 |   },
485 |   {
486 |    "cell_type": "code",
487 |    "execution_count": 7,
488 |    "metadata": {},
489 |    "outputs": [],
490 |    "source": [
491 |     "import os\n",
492 |     "\n",
493 |     "def load_dataset(dataset_path):\n",
494 |     "    ## load and preprocess data for stomach cancer\n",
495 |     "    ld_edge_index = \"\"\n",
496 |     "    ld_edge_attr = \"\"\n",
497 |     "    ld_feature = \"\"\n",
498 |     "    ld_label = \"\"\n",
499 |     "    ld_pid = \"\"\n",
500 |     "    for _root, _dirs, _files in os.walk(dataset_path):\n",
501 |     "        for _file in _files:\n",
502 |     "            #print(_file)\n",
503 |     "            if(\"weighted_edge_index\" in _file):\n",
504 |     "                ld_edge_index = os.path.join(_root, _file)\n",
505 |     "            elif(\"weighted_edge_attr\" in _file):\n",
506 |     "                ld_edge_attr = os.path.join(_root, _file)\n",
507 |     "            elif(\"weighted_feature\" in _file):\n",
508 |     "                ld_feature = os.path.join(_root, _file)\n",
509 |     "            elif(\"weighted_label\" in _file):\n",
510 |     "                ld_label = os.path.join(_root, _file)\n",
511 |     "            elif(\"weighted_pid.mat\" in _file):\n",
512 |     "                ld_pid = os.path.join(_root, _file)\n",
513 |     "#     print(ld_edge_index)\n",
514 |     "#     print(ld_edge_attr)\n",
515 |     "#     print(ld_feature)\n",
516 |     "#     print(ld_label)\n",
517 |     "#     print(ld_pid)\n",
518 |     "\n",
519 |     "    edge_index = sio.loadmat(ld_edge_index)\n",
520 |     "    edge_index = edge_index['edge_index'][0]\n",
521 |     "    # load edge_attr\n",
522 |     "    edge_attr = sio.loadmat(ld_edge_attr)\n",
523 |     "    edge_attr = edge_attr['edge_attr'][0]\n",
524 |     "    # load feature\n",
525 |     "    feature = sio.loadmat(ld_feature)\n",
526 |     "    feature = feature['feature']\n",
527 |     "    #print(feature)\n",
528 |     "    # load label\n",
529 |     "    label = sio.loadmat(ld_label)\n",
530 |     "    label = label['label'][0]\n",
531 |     "    # load label_pid\n",
532 |     "    pid = sio.loadmat(ld_pid)\n",
533 |     "    pid = pid['pid'][0]\n",
534 |     "    \n",
535 |     "    stomach = list()\n",
536 |     "    num_edge = 0\n",
537 |     "    #num_feature = 0\n",
538 |     "    num_node = 0\n",
539 |     "    num_class = 3\n",
540 |     "    num_graph = edge_index.shape[0]\n",
541 |     "\n",
542 |     "    for i in range(num_graph):\n",
543 |     "        # extract edge index, turn to tensor\n",
544 |     "        edge_index_1 = np.array(edge_index[i][:,0:2],dtype=np.int)\n",
545 |     "        edge_index_1 = torch.tensor(edge_index_1, dtype=torch.long).to(device)\n",
546 |     "        # number of edges\n",
547 |     "        num_edge = num_edge + edge_index_1.shape[0]\n",
548 |     "        # extract edge_attr, turn to tensor\n",
549 |     "        edge_attr_1 = np.array(edge_attr[i][:,0:1],dtype=np.int)\n",
550 |     "        edge_attr_1 = torch.tensor(edge_attr_1, dtype=torch.float).to(device)\n",
551 |     "        # extract feature, turn to tensor\n",
552 |     "        \n",
553 |     "        feature_1 = torch.tensor(feature[i], dtype=torch.float).to(device)\n",
554 |     "        #print(feature_1.shape)\n",
555 |     "        # number of nodes\n",
556 |     "        num_node = num_node + feature_1.shape[0]\n",
557 |     "        # number of features\n",
558 |     "        if i==0:\n",
559 |     "            num_feature = feature_1.shape[1]\n",
560 |     "        # extract label, turn to tensor\n",
561 |     "        label_1 = torch.tensor([label[i]],dtype=torch.long).to(device)\n",
562 |     "        # extract patient id, turn to tensor\n",
563 |     "        \n",
564 |     "        pid_1 = torch.tensor([pid[i]],dtype=torch.long).to(device)\n",
565 |     "        # put edge, feature, label together to form graph information in \"Data\" format\n",
566 |     "        data_1 = Data(x=feature_1, edge_index=edge_index_1.t().contiguous(), edge_attr=edge_attr_1, y=label_1, pid=pid_1)\n",
567 |     "        stomach.append(data_1)\n",
568 |     "    return(stomach, num_feature, num_edge, num_node)\n",
569 |     "        \n",
570 |     "train_data_list, num_feature, num_edge, num_node = load_dataset(\"data/selected_new_data_file/train_data_fold_{}/\".format(fold))\n",
571 |     "val_data_list, _, _, _ = load_dataset(\"data/selected_new_data_file/val_data_fold_{}/\".format(fold))\n",
572 |     "test_data_list, _, _, _ = load_dataset(\"data/selected_new_data_file/test_data/\")\n",
573 |     "\n",
574 |     "train_val_list = train_data_list + val_data_list\n",
575 |     "# generate training, validation and test data sets\n",
576 |     "nv = np.random.permutation(len(train_val_list))\n",
577 |     "stomach_1 = train_val_list\n",
578 |     "stomach = list()\n",
579 |     "for i in nv:\n",
580 |     "    stomach.append(stomach_1[nv[i]])\n",
581 |     "num_train_val = len(stomach)\n",
582 |     "num_train = int(num_train_val * 0.8)\n",
583 |     "#num_val = num_train_val - num_train\n",
584 |     "\n",
585 |     "train_loader = DataLoader(stomach[0:num_train], batch_size=batch_size, shuffle = True)\n",
586 |     "val_loader = DataLoader(stomach[num_train:-1], batch_size=batch_size, shuffle = True)\n",
587 |     "test_loader = DataLoader(test_data_list, batch_size=1, shuffle = False)"
588 |    ]
589 |   },
590 |   {
591 |    "cell_type": "code",
592 |    "execution_count": 8,
593 |    "metadata": {
594 |     "scrolled": false
595 |    },
596 |    "outputs": [
597 |     {
598 |      "name": "stdout",
599 |      "output_type": "stream",
600 |      "text": [
601 |       "**Data Set\n",
602 |       "Ave.#Edge: 5553410.0, Ave.#Feature: 35.0, Ave.#Node: 1053500.0, #Classes: 3\n",
603 |       "Train-val-test ratio: 7:1:2, Shuffle: True\n",
604 |       "- number of training data: 42\n",
605 |       "- number of validation data: 11\n",
606 |       "- number of test data: 3660\n",
607 |       "**Network Parameter Setting\n",
608 |       "- batch size:  256\n",
609 |       "- learning rate:  0.0005\n",
610 |       "- weight decay:  0.0001\n",
611 |       "- hidden size:  512\n",
612 |       "- pooling_ratio:  0.5\n",
613 |       "- maximum number of epochs:  200\n",
614 |       "- graph convolution:  GCNConv\n",
615 |       "- number of graph convoluational layers: 1x4\n",
616 |       "- graph pooling:  TopKPooling\n",
617 |       "- number of pooling layers:  4\n",
618 |       "- number of fully connected layers:  4\n",
619 |       "Run: 001, Epoch: 001, Val loss: 1.05696, Val acc: 0.42664\n",
620 |       "Validation loss decreased (inf --> 1.056962).  Saving model ...\n",
621 |       "Run: 001, Epoch: 002, Val loss: 1.01926, Val acc: 0.45636\n",
622 |       "Validation loss decreased (1.056962 --> 1.019258).  Saving model ...\n",
623 |       "Run: 001, Epoch: 003, Val loss: 0.96705, Val acc: 0.50489\n",
624 |       "Validation loss decreased (1.019258 --> 0.967054).  Saving model ...\n",
625 |       "Run: 001, Epoch: 004, Val loss: 0.97122, Val acc: 0.50263\n",
626 |       "EarlyStopping counter: 1 out of 20\n",
627 |       "Run: 001, Epoch: 005, Val loss: 0.91673, Val acc: 0.54891\n",
628 |       "Validation loss decreased (0.967054 --> 0.916735).  Saving model ...\n",
629 |       "Run: 001, Epoch: 006, Val loss: 0.85152, Val acc: 0.59707\n",
630 |       "Validation loss decreased (0.916735 --> 0.851520).  Saving model ...\n",
631 |       "Run: 001, Epoch: 007, Val loss: 0.84072, Val acc: 0.61362\n",
632 |       "Validation loss decreased (0.851520 --> 0.840718).  Saving model ...\n",
633 |       "Run: 001, Epoch: 008, Val loss: 0.78786, Val acc: 0.65124\n",
634 |       "Validation loss decreased (0.840718 --> 0.787856).  Saving model ...\n",
635 |       "Run: 001, Epoch: 009, Val loss: 0.75665, Val acc: 0.66290\n",
636 |       "Validation loss decreased (0.787856 --> 0.756647).  Saving model ...\n",
637 |       "Run: 001, Epoch: 010, Val loss: 0.68001, Val acc: 0.70542\n",
638 |       "Validation loss decreased (0.756647 --> 0.680006).  Saving model ...\n",
639 |       "Run: 001, Epoch: 011, Val loss: 0.69914, Val acc: 0.68811\n",
640 |       "EarlyStopping counter: 1 out of 20\n",
641 |       "Run: 001, Epoch: 012, Val loss: 0.62928, Val acc: 0.72649\n",
642 |       "Validation loss decreased (0.680006 --> 0.629284).  Saving model ...\n",
643 |       "Run: 001, Epoch: 013, Val loss: 0.62442, Val acc: 0.73890\n",
644 |       "Validation loss decreased (0.629284 --> 0.624422).  Saving model ...\n",
645 |       "Run: 001, Epoch: 014, Val loss: 0.57847, Val acc: 0.75508\n",
646 |       "Validation loss decreased (0.624422 --> 0.578474).  Saving model ...\n",
647 |       "Run: 001, Epoch: 015, Val loss: 0.55423, Val acc: 0.77163\n",
648 |       "Validation loss decreased (0.578474 --> 0.554229).  Saving model ...\n",
649 |       "Run: 001, Epoch: 016, Val loss: 0.56123, Val acc: 0.77351\n",
650 |       "EarlyStopping counter: 1 out of 20\n",
651 |       "Run: 001, Epoch: 017, Val loss: 0.54422, Val acc: 0.76185\n",
652 |       "Validation loss decreased (0.554229 --> 0.544223).  Saving model ...\n",
653 |       "Run: 001, Epoch: 018, Val loss: 0.48902, Val acc: 0.80023\n",
654 |       "Validation loss decreased (0.544223 --> 0.489018).  Saving model ...\n",
655 |       "Run: 001, Epoch: 019, Val loss: 0.51223, Val acc: 0.79834\n",
656 |       "EarlyStopping counter: 1 out of 20\n",
657 |       "Run: 001, Epoch: 020, Val loss: 0.48600, Val acc: 0.80813\n",
658 |       "Validation loss decreased (0.489018 --> 0.486000).  Saving model ...\n",
659 |       "Run: 001, Epoch: 021, Val loss: 0.45923, Val acc: 0.81264\n",
660 |       "Validation loss decreased (0.486000 --> 0.459229).  Saving model ...\n",
661 |       "Run: 001, Epoch: 022, Val loss: 0.44748, Val acc: 0.82054\n",
662 |       "Validation loss decreased (0.459229 --> 0.447480).  Saving model ...\n",
663 |       "Run: 001, Epoch: 023, Val loss: 0.46807, Val acc: 0.81716\n",
664 |       "EarlyStopping counter: 1 out of 20\n",
665 |       "Run: 001, Epoch: 024, Val loss: 0.41482, Val acc: 0.84274\n",
666 |       "Validation loss decreased (0.447480 --> 0.414825).  Saving model ...\n",
667 |       "Run: 001, Epoch: 025, Val loss: 0.43815, Val acc: 0.83747\n",
668 |       "EarlyStopping counter: 1 out of 20\n",
669 |       "Run: 001, Epoch: 026, Val loss: 0.42474, Val acc: 0.84048\n",
670 |       "EarlyStopping counter: 2 out of 20\n",
671 |       "Run: 001, Epoch: 027, Val loss: 0.42213, Val acc: 0.84349\n",
672 |       "EarlyStopping counter: 3 out of 20\n",
673 |       "Run: 001, Epoch: 028, Val loss: 0.38146, Val acc: 0.86494\n",
674 |       "Validation loss decreased (0.414825 --> 0.381457).  Saving model ...\n",
675 |       "Run: 001, Epoch: 029, Val loss: 0.43542, Val acc: 0.85403\n",
676 |       "EarlyStopping counter: 1 out of 20\n",
677 |       "Run: 001, Epoch: 030, Val loss: 0.38875, Val acc: 0.86305\n",
678 |       "EarlyStopping counter: 2 out of 20\n",
679 |       "Run: 001, Epoch: 031, Val loss: 0.44091, Val acc: 0.84462\n",
680 |       "EarlyStopping counter: 3 out of 20\n",
681 |       "Run: 001, Epoch: 032, Val loss: 0.40274, Val acc: 0.87020\n",
682 |       "EarlyStopping counter: 4 out of 20\n",
683 |       "Run: 001, Epoch: 033, Val loss: 0.44874, Val acc: 0.86117\n",
684 |       "EarlyStopping counter: 5 out of 20\n",
685 |       "Run: 001, Epoch: 034, Val loss: 0.39264, Val acc: 0.86268\n",
686 |       "EarlyStopping counter: 6 out of 20\n",
687 |       "Run: 001, Epoch: 035, Val loss: 0.41636, Val acc: 0.86719\n",
688 |       "EarlyStopping counter: 7 out of 20\n",
689 |       "Run: 001, Epoch: 036, Val loss: 0.39624, Val acc: 0.87359\n",
690 |       "EarlyStopping counter: 8 out of 20\n",
691 |       "Run: 001, Epoch: 037, Val loss: 0.46559, Val acc: 0.86230\n",
692 |       "EarlyStopping counter: 9 out of 20\n",
693 |       "Run: 001, Epoch: 038, Val loss: 0.42229, Val acc: 0.87547\n",
694 |       "EarlyStopping counter: 10 out of 20\n",
695 |       "Run: 001, Epoch: 039, Val loss: 0.45045, Val acc: 0.86531\n",
696 |       "EarlyStopping counter: 11 out of 20\n",
697 |       "Run: 001, Epoch: 040, Val loss: 0.38113, Val acc: 0.88751\n",
698 |       "Validation loss decreased (0.381457 --> 0.381129).  Saving model ...\n",
699 |       "Run: 001, Epoch: 041, Val loss: 0.43642, Val acc: 0.87434\n",
700 |       "EarlyStopping counter: 1 out of 20\n",
701 |       "Run: 001, Epoch: 042, Val loss: 0.40980, Val acc: 0.88149\n",
702 |       "EarlyStopping counter: 2 out of 20\n",
703 |       "Run: 001, Epoch: 043, Val loss: 0.38917, Val acc: 0.88864\n",
704 |       "EarlyStopping counter: 3 out of 20\n",
705 |       "Run: 001, Epoch: 044, Val loss: 0.45189, Val acc: 0.87698\n",
706 |       "EarlyStopping counter: 4 out of 20\n",
707 |       "Run: 001, Epoch: 045, Val loss: 0.46373, Val acc: 0.87660\n",
708 |       "EarlyStopping counter: 5 out of 20\n",
709 |       "Run: 001, Epoch: 046, Val loss: 0.54833, Val acc: 0.86569\n",
710 |       "EarlyStopping counter: 6 out of 20\n",
711 |       "Run: 001, Epoch: 047, Val loss: 0.42891, Val acc: 0.87472\n",
712 |       "EarlyStopping counter: 7 out of 20\n",
713 |       "Run: 001, Epoch: 048, Val loss: 0.42434, Val acc: 0.89391\n",
714 |       "EarlyStopping counter: 8 out of 20\n",
715 |       "Run: 001, Epoch: 049, Val loss: 0.41616, Val acc: 0.89052\n",
716 |       "EarlyStopping counter: 9 out of 20\n",
717 |       "Run: 001, Epoch: 050, Val loss: 0.40190, Val acc: 0.89541\n",
718 |       "EarlyStopping counter: 10 out of 20\n",
719 |       "Run: 001, Epoch: 051, Val loss: 0.44651, Val acc: 0.89278\n",
720 |       "EarlyStopping counter: 11 out of 20\n",
721 |       "Run: 001, Epoch: 052, Val loss: 0.46134, Val acc: 0.89278\n",
722 |       "EarlyStopping counter: 12 out of 20\n",
723 |       "Run: 001, Epoch: 053, Val loss: 0.47696, Val acc: 0.87848\n",
724 |       "EarlyStopping counter: 13 out of 20\n",
725 |       "Run: 001, Epoch: 054, Val loss: 0.55726, Val acc: 0.88375\n",
726 |       "EarlyStopping counter: 14 out of 20\n",
727 |       "Run: 001, Epoch: 055, Val loss: 0.51403, Val acc: 0.86305\n",
728 |       "EarlyStopping counter: 15 out of 20\n",
729 |       "Run: 001, Epoch: 056, Val loss: 0.47854, Val acc: 0.89353\n",
730 |       "EarlyStopping counter: 16 out of 20\n",
731 |       "Run: 001, Epoch: 057, Val loss: 0.46898, Val acc: 0.88939\n",
732 |       "EarlyStopping counter: 17 out of 20\n",
733 |       "Run: 001, Epoch: 058, Val loss: 0.47342, Val acc: 0.89014\n",
734 |       "EarlyStopping counter: 18 out of 20\n",
735 |       "Run: 001, Epoch: 059, Val loss: 0.42647, Val acc: 0.89165\n",
736 |       "EarlyStopping counter: 19 out of 20\n",
737 |       "Run: 001, Epoch: 060, Val loss: 0.51179, Val acc: 0.87923\n",
738 |       "EarlyStopping counter: 20 out of 20\n",
739 |       "Early stopping\n",
740 |       "** Run: 001, test loss: 1.96189, test acc: 0.74235\n",
741 |       "Test accuarcy at patient level: 74.29\n",
742 |       "** Model 1600507584.0485966, mean test acc (cell): 0.74235\n"
743 |      ]
744 |     }
745 |    ],
746 |    "source": [
747 |     "# import EarlyStopping\n",
748 |     "from pytorchtools import EarlyStopping\n",
749 |     "\n",
750 |     "\n",
751 |     "sv_dat = '{}/test_data.pt'.format(model_name)\n",
752 |     "torch.save(test_data_list, sv_dat)\n",
753 |     "num_class = 3\n",
754 |     "\n",
755 |     "print('**Data Set')\n",
756 |     "#print('Data name: {}, Data type: {},  #Graph: {}'.format('Stomach',data_type,num_graph))\n",
757 |     "print('Ave.#Edge: {:.1f}, Ave.#Feature: {:.1f}, Ave.#Node: {:.1f}, #Classes: {:d}'.format(num_edge,num_feature,num_node,num_class))\n",
758 |     "print('Train-val-test ratio: 7:1:2, Shuffle: True')\n",
759 |     "print('- number of training data:',len(train_loader))\n",
760 |     "print('- number of validation data:',len(val_loader))\n",
761 |     "print('- number of test data:',len(test_loader))\n",
762 |     "\n",
763 |     "print('**Network Parameter Setting')\n",
764 |     "print('- batch size: ',batch_size)\n",
765 |     "print('- learning rate: ',learning_rate)\n",
766 |     "print('- weight decay: ',weight_decay)\n",
767 |     "print('- hidden size: ',nhid)\n",
768 |     "print('- pooling_ratio: ',pooling_ratio)\n",
769 |     "print('- maximum number of epochs: ',epochs)\n",
770 |     "# print('- patience for earlystopping: ',early_stopping)\n",
771 |     "print('- graph convolution: ','GCNConv')\n",
772 |     "print('- number of graph convoluational layers: {}x{}'.format(1,num_layers))\n",
773 |     "print('- graph pooling: ','TopKPooling')\n",
774 |     "print('- number of pooling layers: ',num_layers)\n",
775 |     "print('- number of fully connected layers: ',num_layers)\n",
776 |     "    \n",
777 |     "###############################################################\n",
778 |     "\n",
779 |     "train_loss = np.zeros((runs,epochs),dtype=np.float)\n",
780 |     "val_acc = np.zeros((runs,epochs))\n",
781 |     "val_loss = np.zeros((runs,epochs))\n",
782 |     "test_acc_c = np.zeros(runs)\n",
783 |     "test_loss_c = np.zeros(runs)\n",
784 |     "test_pred_c = np.zeros(runs)\n",
785 |     "test_out_c = np.zeros((runs,num_class)) \n",
786 |     "groud_truth_c = np.zeros((runs,num_class))\n",
787 |     "test_acc_p = np.zeros(runs)\n",
788 |     "min_loss = 1e10*np.ones(runs)\n",
789 |     "# num_test_p = num_test\n",
790 |     "# pid_test_p  = np.zeros((runs,num_test_p))\n",
791 |     "for run in range(runs):\n",
792 |     "#         print('\\n*** Training ***')\n",
793 |     "#         print('** Run {} of total {} runs ...'.format(run+1,runs))\n",
794 |     "#     device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
795 |     "    model = GINTopK(num_feature=num_feature, num_class=num_class, nhid=nhid).to(device)\n",
796 |     "    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay = weight_decay)\n",
797 |     "    \n",
798 |     "    ## Training\n",
799 |     "\n",
800 |     "    # initialize the early_stopping object\n",
801 |     "    patience = 20\n",
802 |     "    early_stopping = EarlyStopping(patience=patience, verbose=True, path=\"{}/model_{}_fold{}_run{}.pth\".format(model_name, model_name, fold, run))\n",
803 |     "#        val_acc_c = np.zeros((runs,epochs))\n",
804 |     "#        val_loss_c = np.zeros((runs,epochs))\n",
805 |     "#        test_acc_c = np.zeros(runs)\n",
806 |     "#        test_acc_p = np.zeros(runs)\n",
807 |     "    for epoch in range(epochs):\n",
808 |     "        model.train()\n",
809 |     "        loss_all = 0\n",
810 |     "        for i, data in enumerate(train_loader):\n",
811 |     "            data = data.to(device)\n",
812 |     "#             print('data in train: ',data)\n",
813 |     "            out, out2 = model(data)\n",
814 |     "            loss = F.nll_loss(out, data.y)\n",
815 |     "            #print(out, data.y)\n",
816 |     "            #writer.add_scalar('train/loss', loss, len(train_loader)*epoch+i)\n",
817 |     "            #print(\"Training loss: {:.5f}\".format(loss.item()))\n",
818 |     "            loss.backward()\n",
819 |     "            loss_all += data.num_graphs * loss.item()\n",
820 |     "            optimizer.step()\n",
821 |     "            optimizer.zero_grad()\n",
822 |     "        loss = loss_all / len(train_loader.dataset)   \n",
823 |     "        train_loss[run,epoch] = loss\n",
824 |     "        val_acc[run,epoch], val_loss[run, epoch], _, _, _, _, _ = test(model, val_loader)\n",
825 |     "        print(\"Run: {:03d}, Epoch: {:03d}, Val loss: {:.5f}, Val acc: {:.5f}\".format(run+1,epoch+1,val_loss[run,epoch],val_acc[run,epoch]))\n",
826 |     "        \n",
827 |     "        # early_stopping needs the validation loss to check if it has decresed, \n",
828 |     "        # and if it has, it will make a checkpoint of the current model\n",
829 |     "        early_stopping(val_loss[run, epoch], model)\n",
830 |     "        \n",
831 |     "        if early_stopping.early_stop:\n",
832 |     "            print(\"Early stopping\")\n",
833 |     "            break\n",
834 |     "        \n",
835 |     "#         if val_loss[run,epoch] < min_loss[run]:\n",
836 |     "#             torch.save(model.state_dict(), 'model_gintopk.pth')   # save the model and reuse later in test\n",
837 |     "#             #print(\"Model saved at epoch: {:03d}\".format(epoch))\n",
838 |     "#             min_loss[run] = val_loss[run,epoch]\n",
839 |     "    # model = GCNTopK(num_feature=num_feature, num_class=num_class, nhid=nhid, pooling_ratio=pooling_ratio).to(device)\n",
840 |     "    model = GINTopK(num_feature=num_feature, num_class=num_class, nhid=nhid).to(device)\n",
841 |     "    model.load_state_dict(torch.load(\"{}/model_{}_fold{}_run{}.pth\".format(model_name, model_name, fold, run)))\n",
842 |     "    test_acc_c[run], test_loss_c[run], test_pred_c, test_out_c, ground_truth_c, test_label_c, test_pred_bi_c = test(model,test_loader)\n",
843 |     "    print(\"** Run: {:03d}, test loss: {:.5f}, test acc: {:.5f}\".format(run+1,test_loss_c[run],test_acc_c[run]))\n",
844 |     "    pid_list = list()\n",
845 |     "    test_data = list([None] * len(test_loader))\n",
846 |     "    for i, data in enumerate(test_loader):\n",
847 |     "        pid_temp = data.pid.cpu().numpy()\n",
848 |     "        gt = data.y.cpu().numpy()\n",
849 |     "        test_data[i] = [pid_temp,gt,test_pred_c[i]]\n",
850 |     "        if not pid_temp in pid_list:\n",
851 |     "            pid_list.append(pid_temp)\n",
852 |     "    num_test_p = len(pid_list)\n",
853 |     "    test_pred_1 = np.zeros([num_class,num_test_p],dtype=np.int)\n",
854 |     "    pred_p = np.zeros(num_test_p,dtype=np.int)\n",
855 |     "    test_label_p = np.zeros(num_test_p,dtype=np.int)\n",
856 |     "    pid_test = np.array(pid_list)\n",
857 |     "    for j in range(num_test_p):\n",
858 |     "        pid_1 = pid_list[j]\n",
859 |     "        k = 0\n",
860 |     "        for i, data in enumerate(test_loader):\n",
861 |     "            if data.pid.cpu().numpy()==pid_1:\n",
862 |     "                if k==0:\n",
863 |     "                    test_label_p[j] = data.y.cpu().numpy()\n",
864 |     "                    k = 1\n",
865 |     "                test_pred_i = int(test_pred_c[i])\n",
866 |     "                test_pred_1[test_pred_i,j] = test_pred_1[test_pred_i,j] + 1\n",
867 |     "        pred_p[j] = np.argmax(test_pred_1[:,j])\n",
868 |     "  #   print('j: {}, pred_p[j]: {}, test_pred_p[j]: {}'.format(j,pred_p[j],test_label_p[j]))\n",
869 |     "    test_acc_p[run] = (pred_p==test_label_p).sum()*1.0/num_test_p\n",
870 |     "    print(\"Test accuarcy at patient level: {:.2f}\".format(test_acc_p[run]*100))\n",
871 |     "    ## save data\n",
872 |     "    t1 = time.time()\n",
873 |     "    print(\"** Model {}, mean test acc (cell): {:.5f}\".format(t1,np.mean(test_acc_c)))\n",
874 |     "    sv = model_name + '/scdiag_' + model_name + '_fold' + str(fold) + '_runs' + str(runs) + '_run' + str(run) + '_epochs' + str(epochs) + '.mat'\n",
875 |     "    sio.savemat(sv,mdict={'val_loss':val_loss,'val_acc':val_acc,'test_loss_c':test_loss_c,'test_acc_c':test_acc_c,'train_loss':train_loss,'test_pred_c':test_pred_c,'test_out_c':test_out_c,'ground_truth_c':ground_truth_c,'test_label_c':test_label_c,'test_pred_bi_c':test_pred_bi_c,'test_acc_p':test_acc_p,'test_pred_p':pred_p,'pid_test':pid_test,'test_data':test_data})"
876 |    ]
877 |   },
878 |   {
879 |    "cell_type": "code",
880 |    "execution_count": 9,
881 |    "metadata": {},
882 |    "outputs": [
883 |     {
884 |      "data": {
885 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAI4CAYAAAB3OR9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd3gVZeL28e+kkYQUICG0UBMIISGEUCTUUAWUoqKAgmIBFVkr7qLra3fX3WUVe/upKEVAEESqK70JUkLvIXSpBggkkDLvHwcigcBJO5lzkvtzXVwhM3Pm3MlQboZnnscwTRMREREREbFxszqAiIiIiIgzUUEWEREREbmKCrKIiIiIyFVUkEVERERErqKCLCIiIiJyFRVkEREREZGrOKwgG4bxlWEYxw3D2HKD/YZhGO8bhrHHMIxNhmHEOSqLiIiIiEh+OfIO8lig+0329wDqX/4xDPjEgVlERERERPLFYQXZNM2lwOmbHNIH+Na0+RWoYBhGNUflERERERHJDw8L37sGcPCqzw9d3nb02gMNwxiG7S4z3t7ezWrVquWwUIaZjd/5ZDI8/Ej3Drluv2nCodRs3A2o7qch3PmRnZ2Nm5u+V65C18u16Hq5Ft+DBzFNkzQH/j0mxUu/x1xPQa7Zrl27TpqmWfna7VYWZCOPbXmue22a5ufA5wARERHmzp07HZkLZv4FNk+F59aAd+B1u2dvOsoTE9fzUt9oBreq7dgspcDixYtJSEiwOobkk66Xa9H1cjEJCaSkpFAhMdHqJJJP+j3megpyzQzD2J/Xdiv/SXQIqHnV56HAEYuy5NZsCGRcgE1T8tzds3FVWocFMXr+Tk6fv1Sy2URERETEoawsyDOB+y/PZtEKOGOa5nXDKyxRPQ6qxsC6sbYxFdcwDIPXekdx/mIm/5m/o+TziYiIaxozhj0jRlidQkTscOQ0b98Bq4AIwzAOGYbxsGEYjxmG8djlQ+YAScAe4AtguKOyFJhh2O4iH9sCh9fneUj9Kv482KYOk347yMaDKSWbT0REXFNsLKnh4VanEBE7HDYG2TTNgXb2m8ATjnr/Imt8N/z8/2DdVxDaLM9Dnuxcnx8Tj/Dyj1uYPrwNbm55DasWERG57JdfqLhxI5TSMa0ZGRkcOnSI9PR0q6MUm8DAQLZv3251DCmAvK6Zt7c3oaGheHp65uscVj6k59y8A6DxXbaH9W79R54P6/l7e/Jiz0ienpzIlLUHGdBSTyWLiMhNvPkmtVNS4LnnrE7iEIcOHcLf3586depgGKXjptG5c+fw9/e3OoYUwLXXzDRNTp06xaFDh6hbt26+zqF5S27mysN6m7+/4SF9YqvTsk4l/jVvBykX9MCeiIiUXenp6QQFBZWaciylg2EYBAUFFeh/NlSQb+bKw3prx+b5sB5cfmCvTxRn0jL478+7SjafiIiIk1E5FmdU0F+XKsg3k/Ow3uYbPqwHEFktgPvj6zBh9X62HD5TcvlEREREpNipINvT+G7wLG97WO8mnunagIq+XrwycyvZ2XnfbRYRERHHOXXqFLGxscTGxlK1alVq1KiR8/mlS/kbBvnggw9ib0Gyjz76iAkTJhRHZNq2bUuiFo5xOnpIz558PKwHEOjjyd96NOSvUzcxfcNh7moWWsJBRUTE6X32GTtXr+YWq3OUUkFBQTll89VXX8XPz4+RI0fmOsY0TUzTvOFSxF9//bXd93niCeedhEuKh+4g50ezB+0+rAfQLy6UprUq8M+5OzibnlFC4URExGVERJBWSzMelbQ9e/YQHR3NY489RlxcHEePHmXYsGE0b96cqKgoXn/99Zxjr9zRzczMpEKFCrzyyis0adKE+Ph4jh8/DsBLL73EmDFjco4fNWoULVu2JCIigpUrVwJw/vx57rrrLpo0acLAgQNp3rx5vu8Up6Wl8cADD9C4cWPi4uJYunQpAJs3b6ZFixbExsYSExNDUlIS586do0ePHjRp0oTo6GimTp1anN+6Mkt3kPOjetM/H9Zr/rBtbHIe3NwMXu8dTe+PljPmf7t5uVejks0pIiLO7aefCNq8udTOg3y1137ayrYjZ4v1nI2qB/BKr6hCvXbbtm18/fXXfPrppwC8/fbbVKpUiczMTDp27Ei/fv1o1Cj339tnzpyhTZs2vPPOOzz77LN89dVXjBo16rpzm6bJmjVrmDlzJq+//jrz5s3jgw8+oGrVqkybNo2NGzcSFxeX76zvv/8+Xl5ebN68ma1bt9KzZ092797Nxx9/zMiRI+nfvz8XL17ENE1+/PFH6tSpw9y5c3MyS9HpDnJ+5PNhPYDGoYHc36wyW36dz+GVkyA7u2QyioiI8/vvf6k5ZYrVKcqksLAwWrRokfP5d999R1xcHHFxcWzfvp1t27Zd9xofHx+6desGQLNmzUhOTs7z3Hfeeed1xyxfvpwBAwYA0KRJE6Ki8l/sly9fzuDBgwGIioqievXq7Nmzh9atW/Pmm2/y73//m4MHD+Lt7U1MTAzz5s1j1KhRrFixgsDAvIeCSsHoDnJ+5ays93XulfUunYffN8ORRDiaCEcSefXkTgzPbPgZzIrlMSJ7WZdbRETEAoW90+so5cuXz/n57t27ee+991izZg0VKlRg0KBBec6R6+XllfNzd3d3MjMz8zx3uXLlrjvGvMH0sPlxo9cOHjyY+Ph4Zs+eTdeuXfnmm29o3749a9euZc6cOTz//PPcfvvtvPjii4V+b7FRQc6vqx/WC2kERzfaCvHJXWBevkvsVwWqxWI06s2SczVouO5ljs/6mLCwHvh66VstIiLiDM6ePYu/vz8BAQEcPXqU+fPn071792J9j7Zt2zJlyhTatWvH5s2b87xDfSPt27dnwoQJtG/fnu3bt3P06FHCw8NJSkoiPDycp556it27d7Np0ybCwsIIDg5m8ODB+Pj4MGnSpGL9OsoqtbaCaP4QrP8W5r8AflWheiw06mv7WC0WAqrlHNreNNlwcgMx+7/h/o9m868HulKzkq+F4UVERAQgLi6ORo0aER0dTb169WjTpk2xv8df/vIX7r//fmJiYoiLiyM6OvqGwx9uvfVWPD09AWjXrh1fffUVjz76KI0bN8bT05Nvv/0WLy8vJk6cyHfffYenpyfVq1fnzTffZOXKlYwaNQo3Nze8vLxyxlhL0RhF+S8AK0RERJj25id0qGNbwTcI/KvaP/bkHviwGe9yH9+49eWje+NoEx7s+IxOZvHixSSUgQdSSgtdL9ei6+ViEhJISUmhQimd93b79u1ERkZaHaNYnTt3Dn9//wK/LjMzk8zMTLy9vdm9ezfdunVj9+7deHjo3qSj3eia5fXr0zCMdaZpNr/2WF2lgqpSgDFVweFQK54RZ39lbvY9DP5yNS/2jOThtnW1FKeISFk0bhzbV60i3uoc4nCpqal07tyZzMxMTNPks88+Uzl2IbpSjtZ0MJ4/DmfGYE+eXVmVN2dvZ8vhM7x9Vwzenu5WpxMRkZJUsyYX9+61OoWUgAoVKrBu3TqrY0ghaZo3R2vUB7z88N0ykU8GxTGyWwN+3HiEfp+u5HBKmtXpRESkJE2eTOWFC61OISJ2qCA7Wjk/iLoDtkzHuHSeEZ3q8+UDzdl/8gK9PljOqr2nrE4oIiIl5ZNPqDFzptUpRMQOFeSS0HQwZJyHbTMA6NSwCjNGtKGiryeDvlzN2BX7ijRfooiIiIgUHxXkklCzJQTVhw3jczaFVfZjxhNt6BgRwqs/beOFHzarJIuIiIg4ARXkkmAY0HQQHFhlm/rtMn9vTz4f3IwH29Rh0m8H2XnsnIUhRUREXFtCQgLz58/PtW3MmDEMHz78pq/z8/MD4MiRI/Tr1++G5167du1NzzNmzBguXLiQ83nPnj1JSUnJT/SbevXVVxk9enSRzyP5p4JcUpoMAMMdEsfn2uzmZvB4hzAAFmw/bkUyERGRUmHgwIHXrSQ3adIkBg4cmK/XV69enalTpxb6/a8tyHPmzKFChQqFPp9YRwW5pPhXhfrdIPE7yMq9lntIgDcxoYEs2H7MonAiIlIipk5l62uvWZ2i1OrXrx+zZs3i4sWLACQnJ3PkyBHatm2bMy9xXFwcjRs35scff7zu9cnJyURHRwOQlpbGgAEDiI+Pp3///qSl/Tnz1OOPP07z5s2JiorilVdeAeD999/nyJEjdOzYkY4dOwJQp04dTp48CcA777xDdHQ00dHRjBkzJuf9IiMjGTp0KFFRUXTr1i3X+9iT1znPnz/PbbfdRpMmTYiOjmby5MkAjBo1ikaNGhETE8PIkSML9H0tizQPcklqOgh2zYU9v0BE7jXfOzeswpgFuziZepFgv3IWBRQREYcKDibjBssNlzpzR8Hvm4v3nFUbQ4+3b7g7KCiIli1bMm/ePPr06cOkSZPo378/hmHg7e3N9OnTCQgI4OTJk7Rq1YrevXvfcOGuTz75BF9fX1atWsW+ffuIi4vL2ffWW29RqVIlsrKy6Ny5M5s2beLJJ5/knXfeYdGiRQQH5141d926dXz99desXr0a0zS55ZZb6NChAxUrVmT37t189913fPHFF9xzzz1MmzaNQYMG2f1W3OicSUlJVK9endmzZwNw5swZTp8+zfTp09mxYweGYRTLsI/STneQS1KDW6F8Zdgw7rpdnSNDME1YvPOEBcFERKREjB1L1XnzrE5Rql09zOLq4RWmafLiiy8SExNDly5dOHz4MMeO3fh/bpcuXZpTVGNiYoiJicnZN2XKFOLi4mjatClbt25l27ZtN820fPly7rjjDsqXL4+fnx933nkny5YtA6Bu3brExsYC0KxZM5KTk/P1dd7onI0bN+aXX37hb3/7G8uWLSMwMJCAgAC8vb155JFH+OGHH/D19c3Xe5RluoNcktw9IaY/rP4UUk+AX+WcXVHVA6ga4M2C7cfo1yzUwpAiIuIwY8dSNSUF3r7xXdBS4yZ3eh2pb9++PPvss6xfv560tLScO78TJkzgxIkTrFu3Dk9PT+rUqUN6evpNz5XX3eV9+/YxevRofvvtNypWrMiQIUPsnudms1SVK/fn/xq7u7vne4jFjc7ZoEED1q1bx5w5c3jhhRfo1q0bL7/8MmvWrGHBggVMmjSJDz/8kIVasOamdAe5pDUdBNmZsGlyrs2GYdApMoSlu05wMTPLonAiIiKuzc/Pj4SEBB566KFcD+edOXOGkJAQPD09WbRoEfv377/pedq3b8+ECRMA2LJlC5s2bQLg7NmzlC9fnsDAQI4dO8bcuXNzXuPv78+5c9fPSNW+fXtmzJjBhQsXOH/+PNOnT6ddu3ZF+jpvdM4jR47g6+vLoEGDGDlyJOvXryc1NZUzZ87Qs2dPxowZQ2JiYpHeuyzQHeSSFhIJNZrbhlnEP2GbAu6yzg1DmLj6AGv2naZd/co3OYmIiIjcyMCBA7nzzjtzzWhx33330atXL5o3b05sbCwNGza86Tkef/xxHnzwQeLj44mLi6Nly5YANGnShKZNmxIVFUW9evVo06ZNzmuGDRtGjx49qFatGosWLcrZHhcXx5AhQ3LO8cgjj9C0adN8D6cAePPNN3MexAM4dOhQnuecP38+zz//PG5ubnh6evLJJ59w7tw5+vTpQ3p6OqZp8u677+b7fcsqw9UWp4iIiDB37txpdYyiWTcWfnoKHlkIoc1yNqdnZBH7+s8MaFGLV3tHWZevmC1evJiEhASrY0g+6Xq5Fl0vF5OQQEpKChVK6R287du3ExkZaXWMYnXu3Dn8/f2tjiEFcKNrltevT8Mw1pmm2fzaYzXEwgpRd4KHz3UP63l7utM2PJgFO45pVT0RERERi6ggW8E7AKL6wpZpcOlCrl2dGlbh4Ok0dh9PtSiciIg4zJw5bCoLD+iJuDgVZKs0HQQXz8L2mbk2d2oYAsAvWjRERKT08fUl29vb6hQiYocKslVqt4GKdWFD7qWnqwZ6E10jgIVadlpEpPT5+GOqz5hhdQoRsUMF2SqGAU3vg+RlcDop167ODauw/sAfnD5/yaJwIiLiEFOmELJ4sdUpRMQOFWQrNbkXMCBxYq7NXSKrkG3Coh26iywiIiJS0lSQrRRYA8I72wpy9p+Lg0RVDyDEvxwLVZBFREQKxN3dndjY2JwfbxfwochXX32V0aNH5/v4X3/9lVtuuYXY2FgiIyN59dVXAdsUjCtXrizQe+dX69ati+1ca9asoX379kRERNCwYUMeeeQRLly4UODvw40U13lmzpxp91omJyczceLEmx6TX1ooxGpNB8H3QyBpEYR3AcDNzaBzZAg/bTzKpcxsvDz07xgREZH88PHxKfRKcZmZmQV+zQMPPMCUKVNo0qQJWVlZXFmrYfHixfj5+RVrmb2iuIr3sWPHuPvuu5k0aRLx8fGYpsm0adPyXA3Qar1796Z37943PeZKQe7Vq1eR30/Ny2oRPcGn4nUP63VqWIXUi5n8lnzaomAiIiKlx+uvv06LFi2Ijo5m2LBhOesNJCQk8OKLL9KhQwfee++9nOP37t1LXFxczue7d++mWbNm1533+PHjVKtWDbDdvW7UqBHJycl8+umnvPvuu8TGxrJs2TL2799P586diYmJoXPnzhw4cACAIUOG8Nhjj9GuXTsaNGjArFmzABg7dix9+vShe/fuRERE8Nprr+W8p5+fH/DnQkH9+vWjYcOG3HfffTlf15w5c2jYsCFt27blySef5Pbbb78u+0cffcQDDzxAfHw8AIZh0K9fP6pUqQLAtm3bSEhIoF69erz//vs5rxs/fjwtW7YkNjaWRx99lKws2/+Cz5s3j7i4OJo0aULnzp2ve78vvviCHj16kJaWRkJCAk8//TStW7cmOjqaNWvWAHD69Gn69u1LTEwMrVq1ylnie+zYsYwYMSLne/bkk0/SunVr6tWrx9SpUwEYNWoUy5Yto02bNkVeLVAF2Woe5SCmP+yYDWkpOZvbhgdTzsNN072JiJQmixeTeNVywaVeQsL1Pz7+2LbvwoW8948da9t/8uT1+/IhLS0t1xCLyZMnAzBixAh+++03tmzZQlpaWk4RBUhJSWHJkiU899xzOdvCwsIIDAzMKWhff/01Q4YMue79nnnmGSIiIrjjjjv47LPPSE9Pp06dOjz22GM888wzJCYm0q5dO0aMGMH999/Ppk2buO+++3jyySdzzpGcnMySJUuYPXs2jz32GOnp6YBt+MOECRNITEzk+++/Z+3atde9/4YNGxgzZgzbtm0jKSmJFStWkJ6ezqOPPsrcuXNZvnw5J06cyPN7tWXLljxL/xU7duxg/vz5rFmzhtdee42MjAy2b9/O5MmTWbFiBYmJibi7uzNhwgROnDjB0KFDmTZtGhs3buT777/Pda4PP/yQn376iRkzZuDj4wPA+fPnWblyJR9//DEPPfQQAK+88gpNmzZl06ZN/OMf/+D+++/PM9vRo0dZvnw5s2bNYtSoUQC8/fbbtGvXjhUrVvDMM8/c8OvKDxVkZxDdD7Iuwa75OZt8vNxpEx7Mgu3HtaqeiIhIPl0ZYnHlR//+/QFYtGgRt9xyC40bN2bhwoVs3bo15zVXjrnWI488wvjx48nKymLy5Mnce++91x3z8ssvs3btWrp168bEiRPp3r17nudatWpVzusHDx7M8uXLc/bdc889uLm5Ub9+ferVq8eOHTsA6Nq1K0FBQfj4+HDnnXfmes0VLVu2JDQ0FDc3N2JjY0lOTmbHjh3Uq1ePunXrAjBw4MD8fOuuc9ttt1GuXDmCg4MJCQnh2LFjLFiwgHXr1tGiRQtiY2NZsGABSUlJ/Prrr7Rv3z7nPStVqpRznnHjxjF37lymTZtGuXLlcrZfydW+fXvOnj1LSkoKy5cvZ/DgwQB06tSJU6dOcebMmeuy9e3bFzc3Nxo1asSxY8V/M1FjkJ1BjWbgX822aEiTP3+TdmoYwsIdx9l7IpXwEK0DLyLi8kaPpubevfm+G+rybjalna/vzfcHB998fwGkp6czfPhw1q5dS82aNXn11Vdz7tIClC9fPs/X3XXXXbzyyivMmjWLZs2aERQUlOdxYWFhPP744wwdOpTKlStz6tQpu5kMw8jz51d/fqPtV7u6cLq7u5OZmZnvG2tRUVGsW7eOPn365Ln/Rud+4IEH+Oc//5nr2JkzZ+aZDyA6OprExEQOHTqUU6Dz+noMw8gzu72v2xE3EnUH2Rm4uUHD22HPArh0Pmdz58grq+ppNgsRkVJh1iyCVq2yOkWZc6UMBwcHk5qamjNm1R5vb286d+7M448/zoMPPpjnMbNnz84paLt378bd3Z0KFSrg7++f62G31q1bM2nSJAAmTJhA27Ztc/Z9//33ZGdns3fvXpKSkoiIiADgf//7H6dPnyYtLY0ZM2bQpk2bfOVu2LAhSUlJJCcnA+QMM7nWiBEj+Oabb1i9enXOtvHjx/P777/f8NydO3dm6tSpHD9u6yanT59m//79xMfHs2TJEvbt25ez/YqmTZvy2Wef0bt3b44cOZKz/Uqu5cuXExgYSGBgIO3bt2fChAmAbYx1cHAwAQEB+fq6r/2eF4UKsrOI7AWZabaSfFm1QB8aVQtggcYhi4iI5Mu1Y5BHjRpFhQoVGDp0KI0bN6Zv3760aNEi3+e75557MAyDbt265bl/3LhxREREEBsby+DBg5kwYQLu7u706tWL6dOn5zyk9/777/P1118TExPDuHHjcj0QGBERQYcOHejRoweffvop3peXI2/bti2DBw8mNjaWu+66i+bNm+crs4+PDx9//DHdu3enbdu2VKlShcDAwOuOq1KlCpMmTWLkyJFEREQQGRnJsmXLblpIGzVqxJtvvkm3bt2IiYmha9euHD16lMqVK/P5559z55130qRJk+uGrbRt25bRo0dz2223cfLkSQAqVqxI69ateeyxx/jyyy8B27Rwa9euJSYmhlGjRvHNN9/k62sGiImJwcPDg9atWxf5IT3D1ca3RkREmFemUClVsjJhdDiEd4W7vsjZ/M7PO/lw0R7WvdSViuW9LAxYeFeeshXXoOvlWnS9XExCAikpKVQo5DRkzm779u1ERkZaHaNYvfXWW6Snp/PGG2845PxDhgzh9ttvp1+/frm2jx07lrVr1/Lhhx8W6rypqan4+flhmiZPPPEE9evXL/KDa8UpISGB0aNH57v0F8S5c+fw979+aGpevz4Nw1hnmuZ1IXQH2Vm4e0DEbbYH9TL/XGK68+VV9Rbv0jALERGRknTHHXfw3Xff8dRTT1kdpcC++OILYmNjiYqK4syZMzz66KNWR3IpekjPmUT2gsTxkLw0Z9GQxjUCqexfjgXbj3NH01CLA4qISJH4+JCVlmZ1Csmn6dOn3/BuZHEZe2Vau2sMGTIkz2nl8uuZZ55xqjvG11pcTA9gOoruIDuTegng5Qfbf8rZ5OZm0CkihCW7TpCRlW1ZNBERKQZz57L5X/+yOoVDudrQTSkbCvrrUgXZmXh6Q/1utkVDsrNyNneKDOFceia/7dOqeiIi4ry8vb05deqUSrI4FdM0OXXqVM7Dj/mhIRbOJrIXbP0BDq6G2rb129vVD8bLw40FO47TOjzY4oAiIlJob7xB7X37Su08yKGhoRw6dOiGK7e5ovT09AIVK7FeXtfM29ub0ND8D1VVQXY29buCeznbMIvLBdnXy4PWYUEs2H6Ml26LvOFE3CIi4uQWLKBiSorVKRzG09Mz10IQpcHixYtp2rSp1TGkAIrjmmmIhbMp5w9hnWwF+ar/ourcMITkUxfYe+L8TV4sIiIiIkWlguyMInvBmYNw9M95MjtFVgFg4Q4tGiIiIiLiSCrIziiiBxjuuWazqFHBh4ZV/bXstIiIiIiDqSA7I99KUKdNroIM0CWyCuv2/0HKhUs3eKGIiDi1oCAybrKMr4g4BxVkZxXZG07ughN/LqvdOTKErGyTJbtKz9PBIiJlyrRpbH39datTiIgdKsjOquFtto/bZ+ZsahJagWA/Lw2zEBEREXEgFWRnFVAdQltcv6pewxAW7TjO+YuZFoYTEZFCeeEF6n7xhdUpRMQOFWRnFtkLjm6EP/bnbOrfohapFzOZkXjYwmAiIlIoq1YRuHWr1SlExA4VZGfW8Hbbxx2zcjbF1apAo2oBjFu1X0t5ioiIiDiACrIzCwqDKtG5hlkYhsHg+Nrs+P0c6/b/YWE4ERERkdJJBdnZRfaCA79C6p8P5vWJrY5/OQ/G/br/Ji8UERERkcJQQXZ2kb0AE3bMztnk6+XBXc1CmbP5KCdTL1qXTURECiY0lIuVK1udQkTsUEF2diGNoFK96xYNGdSqNhlZJpN/O2hRMBERKbDx49n+979bnUJE7FBBdnaGYbuLvG8JpKXkbA4P8aNNeBATft1PVrYe1hMREREpLirIriCyN2Rnwq75uTYPblWbI2fSWbhDC4eIiLiEp58m/MMPrU4hInaoILuC6nHgXz3XqnoAXSKrUCWgnB7WExFxFYmJ+O3ZY3UKEbFDBdkVuLlB5O2wZwFcOp+z2cPdjXtb1mbprhMknzx/kxOIiIiISH6pILuKyF6QmWYryVcZ2LImHm4GE1brLrKIiIhIcVBBdhW1WoNPpetmswgJ8ObW6KpMWXuI9Iwsi8KJiIiIlB4qyK7C3QMa9oRd8yDzUq5dg1vV5kxaBj9tPGJROBERyZcGDbgQGmp1ChGxQwXZlUT2hotnYd/SXJtvqVuJ+iF+elhPRMTZff45u0aOtDqFiNihguxK6nYAL3/Y/mOuzYZhMDi+NpsOnWHjwZQbvFhERERE8kMF2ZV4etuGWWwYDzOGwx/JObvuaFoDXy933UUWEXFmw4bRYPRoq1OIiB0qyK6mx7+g1XDYPBU+aAaznoGzR/D39uSOpjX4aeMR/jh/yf55RESk5O3ahe+hQ1anEBE7VJBdjU9FuPUteCoR4h6A9ePgvViY9yJDYstzMTObqev0h6+IiIhIYakgu6qA6nD7O/CXtdD4blj9CfUntuWdoJnM+HUr2dmm1QlFREREXJIKsqurWAf6fgRPrIGI7tx5fhLfnR/G/hmvwcVzVqcTERERcTkqyKVFcH3o9xUZQ5ex3i2KupvehfeawNbpVvR/bx0AACAASURBVCcTEZErYmNJDQ+3OoWI2KGCXMp41ohhbauP6HvpdS75VoPZI69bWERERCwyZgx7RoywOoWI2KGCXAoNvKUWm8xwfgx+BC6chJ2zrY4kIiIi4jJUkEuhGhV86BJZhX/vqoYZEArrvrE6koiIAAwaRORbb1mdQkTsUEEupQbH1+bEhSx2VO8LSYtyLSoiIiIWOXSIcidOWJ1CROxQQS6l2oQFUze4PP890RIMN1j/rdWRRERERFyCCnIp5eZm8GCbOvxy2IOU0ATYMAGyMq2OJSIiIuL0VJBLsX7NQgn08WRiRkdI/R12z7c6koiIiIjTU0EuxXy9PBjUqhbv7K9DZvmqelhPRMRq8fGciYqyOoWI2KGCXMo9EF8HNzdPVvjdCnv+B2cOWx1JRKTs+uc/2Td0qNUpRMQOFeRSLiTAmz6x1XnjSHMws2HDeKsjiYiIiDg1FeQyYGj7euzJCOJAhVtgwzjIzrI6kohI2XTXXUS9/LLVKUTEDhXkMqBBFX86NKjMR2fbwpmDsHeR1ZFERMqmU6fwPHvW6hQiYocKchkxtF09frjQhHSvSrDua6vjiIiIiDgtFeQyok14EOHVKjGTDpi75sG5Y1ZHEhEREXFKKshlhGEYDG1Xl0/OtcXIzoTECVZHEhEREXFKKshlyO0x1bngX4dtXjG2paezs62OJCJStnTuzB9xcVanEBE7VJDLEC8PNx5sU5fPUtvCH/sgeZnVkUREypb/9//Yf//9VqcQETtUkMuYgS1rsdQjnvNu/rBeK+uJiIiIXMuhBdkwjO6GYew0DGOPYRij8thfyzCMRYZhbDAMY5NhGD0dmUcg0MeTO1qE831GG8ztP8H5U1ZHEhEpO3r0oPHf/mZ1ChGxw2EF2TAMd+AjoAfQCBhoGEajaw57CZhimmZTYADwsaPyyJ8ebFOHSVkdMbIuwaZJVscRESk70tJwv3jR6hQiYocj7yC3BPaYpplkmuYlYBLQ55pjTCDg8s8DgSMOzCOX1azkS1h0SzZSn+y1X4NpWh1JRERExGl4OPDcNYCDV31+CLjlmmNeBX42DOMvQHmgS14nMgxjGDAMoHLlyixevLi4s5Y5zcpnMT6jI/859TkbfvyEMxWuvblffFJTU3XNXIiul2vR9XItsSkpZGVl6Zq5EP0ecz3Fcc0cWZCNPLZde6tyIDDWNM3/GoYRD4wzDCPaNM1c84+Zpvk58DlARESEmZCQ4Ii8ZUoCMPiwG6nHx9MkeyNuCcMd9l6LFy9G18x16Hq5Fl0vF1OhAikpKbpmLkS/x1xPcVwzRw6xOATUvOrzUK4fQvEwMAXANM1VgDcQ7MBMcpXBHSKZkRmPuXUGpKVYHUdEpPS7/XZOxcdbnUJE7HBkQf4NqG8YRl3DMLywPYQ385pjDgCdAQzDiMRWkE84MJNcpUtkFZb634Z7VjrmpilWxxERKf1GjuRg//5WpxAROxxWkE3TzARGAPOB7dhmq9hqGMbrhmH0vnzYc8BQwzA2At8BQ0xTT4yVFDc3g/YdurA5uw4Xfv1KD+uJiIiI4NgxyJimOQeYc822l6/6+TagjSMzyM3dFRfKO/O78vc/voAj66FGM6sjiYiUXgkJxKakQGKi1UlE5Ca0kl4Z5+PlTmDLezlvliP1l3/rLrKIiIiUeSrIwoB2UXyUfRd+++bBlmlWxxERERGxlAqyEOxXDp8OT7EhO5xLPz0L545ZHUlERETEMirIAsBjHRvwWaXnMC9d4NKPT2mohYiIiJRZKsgCgKe7G88M6MW7WffgtWcubJ5qdSQRkdLnnns4rkUnRJyeCrLkiKjqT0DHp1iXXZ9LPz0H5363OpKISOkyfDhH+va1OoWI2KGCLLkM61CfL4OeJzvjAhdnaKiFiEixunABt/R0q1OIiB0qyJKLh7sbzw7syZis/pTbOw9z02SrI4mIlB49exIzapTVKUTEDhVkuU54iD9BXZ5ibXYDMmY9r6EWIiIiUqaoIEueHmpXn7GVnyc7I42L0/+ioRYiIiJSZqggS57c3QyeHdCTd7MHUC7pZ8yN31kdSURERKREqCDLDdWr7EfVrk/zW3YDMmb/Dc4etTqSiIiIiMOpIMtNPdAmjHEhfyPrUjrp00doqIWISFEMGcLv3btbnUJE7FBBlptyczN4bmAPxpgD8N73C2biBKsjiYi4LhVkEZeggix21Q4qT2j3Z1id3ZCM2aPg7BGrI4mIuKaTJ/E8c8bqFCJihwqy5Mt9reoyqdpfycq4SNq0JzTUQkSkMPr1I+qVV6xOISJ2qCBLvri5GTzbvwfvmvfis38h2Vt/tDqSiIiIiEOoIEu+1azkS52ez3DCDODgyilWxxERERFxCBVkKZCBt9Rmm3cz/I8sI/1ShtVxRERERIqdCrIUiGEYVG/Wk0qcZd4vv1gdR0RERKTYqSBLgdWP7w3A/t9mkXox0+I0IiIu5PHHOdy7t9UpRMQOFWQpOP+qpFWMoFnmBr5evs/qNCIirqN/f0506mR1ChGxQwVZCsWnYVducd/Ft0u3k3LhktVxRERcw8GDlDt+3OoUImKHCrIUTlhHPMmgUcYWPl2SZHUaERHXMHgwkf/4h9UpRMQOFWQpnFqtwb0cQ6ruY+zKfRw/m251IhEREZFioYIshePlC7Va0cbYRGaWyQcL91idSERERKRYqCBL4YV1wuvUDh6J9eG7NQc4cOqC1YlEREREikwFWQovzPYk9mOhB3B3MxizYJfFgURERESKTgVZCq9KNJSvTIWjy3mgdR2mbzjMrmPnrE4lIuK8nnuOg/fcY3UKEbFDBVkKz80N6iVA0iIea1+X8l4evPOz7iKLiNxQr16cat3a6hQiYocKshRNWCc4f4JK53YxtF095m39nY0HU6xOJSLinHbuxOfAAatTiIgdKshSNPU62j4mLeLhdnWpVN6L0T/vtDaTiIizevRRIt55x+oUImKHCrIUTUA1CGkEexfiV86D4QlhLNt9klV7T1mdTERERKRQVJCl6Op1hP2r4NIFBrWqTdUAb/4zfwemaVqdTERERKTAVJCl6MI6QdZFOLASb093nuxcn/UHUli447jVyUREREQKTAVZiq52a3D3gr2LALi7eSi1g3z5z/ydZGfrLrKIiIi4FhVkKbrLy05fKcie7m4827UBO34/x6zNRy0OJyLiRF56if2DB1udQkTsUEGW4hHWCY5vhXO/A9ArpjoNq/rzzs87ydRdZBERmy5d+KNZM6tTiIgdKshSPC4vO33lLrKbm8Fz3SJIPnWBn/dnWBhMRMSJJCbit2eP1SlExA4VZCkeVRqDbzAkLcrZ1CUyhG6NqvDD7gz2HNcS1CIiPP004R9+aHUKEbFDBVmKx5Vlp/cuguxsAAzD4K07GuPtDs99v4nMrGxLI4qIiIjkhwqyFJ+wTnD+uG0s8mWV/csxuFE5Nh5M4fNlSRaGExEREckfFWQpPmGXl53euzDX5pZV3enZuCpj/rebnb9rqIWIiIg4NxVkKT4B1aFyZM6DelcYhsEbfaLx9/Zg5PcbydBQCxEREXFiKshSvMI6wv6VkJGWa3OQXzne6BvN5sNn+GzJXovCiYhY7B//IOmRR6xOISJ2qCBL8bqy7PT+ldft6tm4GrfHVOO9BbvZfvTs9a/N1p1lESnlWrfmbHS01SlExA4VZCleV5adTlqU5+7X+0QT6OOZe6hFdjbM/zuMDs9ZaEREpFRauZKALVusTiEidqggS/HyKp9r2elrVSrvxZt9G7P1yFk+XrQXsjJgxmOw6kO4cAp2zi3hwCIiJejFF6n3f/9ndQoRsUMFWYpfvY5wbMsN7wZ3j65Kn9jqfLFwC+fG3gObJkPHl6BCbdg1v4TDioiIiOSmgizF78qy00mLb3jIa12qM6HcP/E9uJjMnu9Ch+ehQXfba655wE9ERESkJKkgS/GrGgO+QTccZsHZI1SY0pfGRhJPXHqS91Pa2LY3uBUy02Df0pLLKiIiInINFWQpfm5utmEWexeCaebed3IPfHkrpBzAbdA0fGPv4KPFe9l86AzUaQtefhqHLCIiIpZSQRbHCOtoW3b62J/LTnNkA3x1K2RcgCGzoF4HXrk9imA/L0Z+v5GLeNhet2v+9cVaRKQ0GDOGPSNGWJ1CROxQQRbHqHfNstNJS2Ds7eDpCw/Nh+pNAQj09eTtO2PYeewc7y/YDQ16wLkj8Psmi4KLiDhQbCyp4eFWpxARO1SQxTECa0DlhpC0iOATK2FCP6hQCx7+GYJz/+XQsWEIdzcL5ZPFezkY3BYwNJuFiJROv/xCxXXrrE4hInaoIIvjhHWCfUuJ2vof2x3jB+dAQLU8Dx15awRuhsG3m85DaHONQxaR0unNN6k9bpzVKUTEDhVkcZzwLpCdyelKcTB4BvhUvOGhVQK8uTWqKlPWHiIjrBscWQ/njpVgWBEREREbFWRxnLBO8PAvbIl+Abx87R4+OL42Z9IyWJgdZ9uwW8MsREREpOSpIIvjGAbUbIHp5pGvw2+pW4kGVfz4YKsXZkCoxiGLiIiIJVSQxWkYhsHg+DpsOXKOk9Uvz6OckW51LBERESljVJDFqdzRtAZ+5TyYfj7aNl9y8jKrI4mIFJ/PPmPns89anUJE7FBBFqfiV86DO+Nq8H5SNUwPX9g1z+pIIiLFJyKCtFq1rE4hInaoIIvTGdyqNqlZHiQHtoSd87SqnoiUHj/9RNDKlVanEBE7VJDF6dSv4k98vSAmnYmCs4dyL1ctIuLK/vtfak6ZYnUKEbFDBVmc0uD42vyQ2sj2yS4tGiIiIiIlRwVZnFLXRlVwC6jKXq8ITfcmIiIiJUoFWZySp7sb97aszY/nG2MeWgupJ6yOJCIiImWECrI4rYEta7KYOAxM2P2z1XFERESkjFBBFqcVEuBNzUatOEYlsnbMsTqOiEjRjRvH9hdftDqFiNihgixO7f74OvyS2ZTsPQsh86LVcUREiqZmTS6GhFidQkTsUEEWp9aybiV2BrTGM+sCZvJyq+OIiBTN5MlUXrjQ6hQiYocKsjg1wzCIbNOLNNOLE+tnWh1HRKRoPvmEGjP1Z5mIs1NBFqfXu3kYq4nGfbdW1RMRERHHU0EWp1e+nAdna3UhKON3/ti/yeo4IiIiUsqpIItLaNzxbgB2LNESrSIiIuJYKsjiEurWa8A+z3B8k38hK1vDLERERMRxVJDFZWSFdyM6eyfLNu6wOoqISOFMncrW116zOoWI2KGCLC6jTut+uBsmO5b9YHUUEZHCCQ4mIzDQ6hQiYocKsrgMjxpNOe8ZROiJJSSdSLU6johIwY0dS9V586xOISJ2qCCL63Bzw61hdzq4bWLiyr1WpxERKTgVZBGXoIIsLsUn6jb8jTSS1v3MnuO6iywiIiLFTwVZXEu9BEz3cnR238Aj3/xGyoVLVicSERGRUkYFWVyLV3mMuu25x2slA85+zedffkpG6mmrU4mIiEgp4mF1AJECS3gBzzkjGXZpNm6nfoTR/w9CGkGtVlCzle1jhVpgGFYnFRERERekgiyuJ7QZDFuE26ULTPjhB45uXszArMPU2DwV1n5lO8a/OtS6BWrFQ2QvCKhubWYREYA5c9i0dCntrc4hIjelgiyuy8uXAffcx7D0CNrvOsG3Q5rRJuA4HPj1zx9bp8Oqj+Av68Fdv9xFxGK+vmR7e1udQkTs0BhkcWnubgZjBsQSVrk8w7/byD6PetByKPT7Ep7dCnePhZT9sG2G1VFFRODjj6k+Q38eiTg7FWRxef7ennz5QAvcDHj4m984k5bx587IPhBUH1a+D6ZpXUgREYApUwhZvNjqFCJihwqylAo1K/ny6aBmHDx9gRET15OZlW3b4eYGrUfA0Y2QvMzakCIiIuISVJCl1LilXhBv9o1m2e6TvDl7+587YgZA+cqw8gPrwomIiIjLUEGWUqV/i1o83LYuY1cmM3H1AdtGT29o+Sjs/hmOb7/5CURERKTMU0GWUueFHg3p0KAyL/+4hVV7T9k2tngYPHxg5YfWhhMRERGnp4IspY6Huxsf3NuUOsHleXzCOvafOg++laDpINg0Gc79bnVEESmrFi8mccwYq1OIiB0qyFIqBXh78uUDzQEYPmE92dkmxA8HMwtWf2ZxOhEREXFmKshSatUOKs9rvaPYeuQsMxIPQ6V6tlX11n4JF89ZHU9EyqLRo6k5ebLVKUTEDocWZMMwuhuGsdMwjD2GYYy6wTH3GIaxzTCMrYZhTHRkHil7esVUp3GNQP778y7SM7Kg9ZOQfgY2jLc6moiURbNmEbRqldUpRMQOhxVkwzDcgY+AHkAjYKBhGI2uOaY+8ALQxjTNKOBpR+WRssnNzeCFHg05nJLGt6uSIbQ51GoNqz6GrEyr44mIiIgTcuQd5JbAHtM0k0zTvARMAvpcc8xQ4CPTNP8AME3zuAPzSBnVOjyYhIjKfLhwDykXLkHrv8CZA1p+WkRERPLk4cBz1wAOXvX5IeCWa45pAGAYxgrAHXjVNM15157IMIxhwDCAypUrs1jLdLqU1NRUy69Z5+BsluzMZNS3ixgQ4U1Ln+pkzf8H604GgWFYms3ZOMP1kvzT9XItsSkpZGVl6Zq5EP0ecz3Fcc0cWZDzah1mHu9fH0gAQoFlhmFEm6aZkutFpvk58DlARESEmZCQUOxhxXEWL16MM1yzjekbmZl4hL/f0wpf/7/CrKdJqOMBddtZHc2pOMv1kvzR9XIx1apx6vRpXTMXot9jrqc4rpkjh1gcAmpe9XkocCSPY340TTPDNM19wE5shVmk2D3btQGGAe/8bxc0GXh5+en3rY4lImXJ3Lls/te/rE4hInY4siD/BtQ3DKOuYRhewABg5jXHzAA6AhiGEYxtyEWSAzNJGVa9gg8Pta3L9A2H2XL8IrQcpuWnRURE5DoOK8imaWYCI4D5wHZgimmaWw3DeN0wjN6XD5sPnDIMYxuwCHjeNM1Tjsok8liHMCr4evKveTug+eXlp1dp+WkRKSFvvEHtb7+1OoWI2OHQeZBN05xjmmYD0zTDTNN86/K2l03TnHn556Zpms+aptnINM3GpmlOcmQekUAfT/7SqT7Ldp9k6eHsy8tPT9Hy0yJSMhYsoOL69VanEBE7tJKelDmDWtWiZiUf/jl3B9m3DIesDC0/LSIiIjlUkKXMKefhzshuEWw/epYZB7yuWn461epoIiIi4gRUkKVMunoJ6ou3jLi8/PQ4q2OJiIiIE1BBljLJzc3ghZ62Jai/ORAMteK1/LSIOF5QEBkBAVanEBE7VJClzGod9ucS1KnNHtfy0yLieNOmsfX1161OISJ2qCBLmTaqR0POXczkvYNhULkhzHsB/thvdSwRERGxkAqylGkNqwbQLy6Ub1Yd5Oitn0HmRZh4D6Sl2H+xiEhBvfACdb/4wuoUImKHCrKUec92sy1B/e91QP9xcGoPfP+Abfo3EZHitGoVgVu3Wp1CROxQQZYyr1rgVUtQl4uFXu9D0mKY9QyYptXxREREpISpIIsAjyeEUdHXk+ET1rOzWm9o/7xt2rfl71odTUREREqYCrIIEODtyZdDWpCWkcWdH69gfsjD0PhuWPAabJlmdTwREREpQSrIIpfF1arITyPaEh7ix6Pj1/Oh/zOYNeNh+uNwYLXV8USkNAgN5WLlylanEBE7VJBFrlI10JvJj8ZzZ9MajF6YzPMefyU7IBQmDYTTSVbHExFXN3482//+d6tTiIgdKsgi1/D2dOe/9zThpdsi+WFHGo9k/pWs7GyYcDdcOF2wk53eB5umFPx1IiIiYhkVZJE8GIbBI+3qMfbBlqw9V5FHLj5D9h8HYPIg21zJN/NHMiwfA591gPdj4Yeh8EUnOL69RLKLiBN7+mnCP/zQ6hQiYocKsshNtG9QmR9HtOWgfyzPXhoG+1dgzhxx/fRvKQdgxfvweUd4rwn88goYbtD1DRg4CTIuwP91gR2zrflCRMQ5JCbit2eP1SmktEpaAhdTrU5RKnhYHUDE2dUNLs/04a15ZrIvo3cdY+SmKWQG1sGj2f2wbQZsnQ6H19kOrt4Uur4OjfpAxTp/nqRaE5h0H0y6Fzq+BO1HgmFY8vWIiEgp9Md++LY3NH8IbtcUpUWlgiySD/7ennw+uDnv/OzP98uPcfeyf8Oyf9t2VmsCXV6FRn2hUt28TxBQHR6cAz89BYvehGOboe8n4FW+pL4EEREpzfYtsX3cMB7a/xUCqlmbx8WpIIvkk5ubwcjuDZld9QO+/uEFKobUoO+9w6FSvfydwNMH7vgMqkTbhmCc2gsDJkLF2o4NLiIipV/SEvAOtA2xWPkBdP+H1YlcmsYgixTQbbF1ONXmVZ4+1JGt6UEFe7FhQJsn4d7vIeUgfNERkpc7JqiIOJ8GDbgQGmp1CiltTBP2LYX63WyLXK39Cs6ftDqVS1NBFimEoe3rEejjyX9/3lW4E9TvAkMXgm8QfNsHfvu/4g0oIs7p88/ZNXKk1SmstfJD2DzV6hSly4kdcP441O0A7Z6FzHT49WOrU7k0FWSRQgj08eSxDmEs3HGctcmFnOM4OBwe+QXCOsPs5+CnpyHzUvEGFRFxJlkZsOgfsPpTq5OULvuW2j7WbQ+VI2wPiq/+HNL+sDaXC1NBFimkB1rXprJ/Of49fyfmtdO+5Zd3IAz8Dto+C+u+tt1Nzkgr3qAi4jyGDaPB6NFWp7DO0Y2Qcd42L3x2ttVpSo+kJbaZk64809LuObh0DtZ8YWksV6aCLFJIvl4e/KVTOGv2nWbp7iKM9XJzhy6vQJ+P4cBK2DKt+EKKiHPZtQvfQ4esTmGdK89cXEqFMweszVJaZGXavq912/+5rVoMNOhuG2aheZELRQVZpAgGtKhFaEUf/jN/R+HvIl8Rey9UCoPEicUTTkTE2SQvB3cv28+1umjx+H0jXDxjG398tXYjbUMs1n5lTS4Xp4IsUgReHm483aUBWw6fZd6W34t2MsOwleT9K+B0UvEEFBFxFlmZcOBX25zxAMe2WpuntEi6PP/x1XeQAWq2sJXmlR9o6F4hqCCLFNEdTWsQHuLH6J93kpVdxLvITQbalqjWXWQRKW1+32QbF9vgVqhQC45vszpR6bBvKYQ0Ar+Q6/e1f942u8WG8SWfy8WpIIsUkbubwXNdG7D3xHmmbzhctJMF1oB6HSHxO8jOKp6AIuI8YmNJDQ+3OoU19q+wfazT1lbojqkgF1nmRdtd+WuHV1xRpy3UbAXLx2iWpAJSQRYpBt2jq9K4RiDv/m8XFzOLWGyb3gdnD/25bKiIlB5jxrBnxAirU1gjeYXtOQv/qraCfGq3SltRHfoNMtOuH15xhWFA+5G2v1M2TS7ZbC5OBVmkGBiGwchbIzicksbk3w4W7WQRt9mmf9swoXjCiYhYLTsL9q+EOm1sn1eJguxMW0mWwktaYhuWd+X7mpfwLlCtCSx/xzYOXPJFBVmkmLSvH0zLupV4f8EeLlwqwh9Cnt62pUJ3zIK0lOILKCLWGzSIyLfesjpFyTu2xTbTQp12ts9DGl3ermEWRbJvKVRvarupciOGYRuLfDoJtk4vuWwuTgVZpJgYhsFfb43gZOpFxq5MLtrJYu+1LRW69YdiySYiTuLQIcqdOGF1ipKXfHn8ce3LdzqDwsHNA45rJotCu5gKh9feePzx1SJug8qRsOy/WqAln1SQRYpR8zqV6NQwhE8X7+VMWkbhT1Q9zvaHmYZZiEhpsH+FbaW3wBq2zz28ILiB7iAXxf6VtmEqNxp/fDU3N9vqeie2w87Zjs9WCqggixSz57o14Gx6Jl8sLcJcxoZhe1jv8Fo4sbP4womIlLTsbFtBrt029/aQRlospCj2LbEtulKrVf6Oj7oDKtWDpf+Boi5sVQaoIIsUs6jqgdweU42vVuzjxLmLhT9RTH8w3DV/pYi4tuPbbCu6XfsgWZVGtuWm089ak8vV7VsCNW8BT5/8He/uAW2fgaMbYc8Cx2YrBVSQRRzg2a4NuJiZzceL9xT+JH4htgn1N03Wk8cipUV8PGeioqxOUbL2XzP++IorD+rpLnLBXTgNv2/O3/jjq8UMgIBQWPpv3UW2QwVZxAHqVfajX1woE349wOGUIizxGXsvpB6DvfrXvkip8M9/sm/oUKtTlKzk5RBYCyrWzr09pyDrQb0C27fU9rFeAQuyhxe0fRoOrrZdF7khFWQRB3myS30A3vtlV+FPUv9W8A3SMAsRcU2mmXv+46tVqAVe/rqDXBj7loKXn22Kt4JqOgj8qsCy0cWfqxRRQRZxkBoVfLivVS2mrjvEnuOphTuJh5dtLPLOuXD+VPEGFJGSd9ddRL38stUpSs6JnXDh5PXDK8D2MHJIpGayKIx9S2zfU3fPgr/W0wfiR0DSYti/qtijlRYqyCIO9ETHcHy9PHh7bhHukMTeB9kZsPn7gr82OxtmDIcvOsHFc4XPICLF49QpPM+WoYfSkpfZPt5opbeQSNsQC42Hzb8zh+HUnvxN73YjLR4G/+ow72+2VQ5LC9OEL7tR7cjPRT5VvgqyYRhPGYYRYNh8aRjGesMwuhX53UVKuWC/cgzvGMYv24+zYs/Jwp2karRtmdDEAg6zME2Y/wIkToDD62D6Y5ogXkRK1v4VtiJWsW7e+6tE2Wa4OPd7yeZyZYUdf3w1r/LQ7Q3bjBYbxhVPLmdwOsk2vpqi/4Mrv3eQHzJN8yzQDagMPAi8XeR3FykDHmpTl9CKPrwxaxtZ2YX8TRs7yPbE8tFN+X/NijGw+lNoNRy6vWVbunr5O4V7fxGRgjJN2wp6ddrahlPkJedBPQ2zyLd9S2zPpoQUcTaU6LugVmtY8LrtHymlweUHD1MqFH2mmPwW5Cu/snsCX5umufGqbSJyE96e7ozq0ZAdv59jytqDhTtJ4362CeETJ+bv+MSJ8Murtj8Au70F5k7pHwAAIABJREFU8U9A47th4Zuw+3+FyyAiUhCn9sD54zceXgEqyAVlmrY7yHXa2VbHKwrDgJ7/n737Dq+qyvo4/j3phQQIIQFC6CQQem8CEVBBKaIIKs02FnTsbWYso/Pa29gVGcWCIhbsgDX0XqWGltB7SQKEkOS8f2yiCCk3yb05N8nv8zw8Ibecs9LX3WfttZ41yfFvT7knPqelzoXQKE4Ex5T6UK5+dpdalvUjJkGeYVlWGKBrtSIuuqRVbTrWr84LP24gPbMEI6hDIiB+APw+BbKzCn/sxp/g69tMf8xL3zS/RC0LBr0C0S3hi+vNZSgRKXt9+3K4fXunoygbeW3Ezp6gd6bQGqajgjbquebgZkjbWbr64zPVagUdr4PFE2BvBWi3lzoP6ncv+IpFMbiaIF8PPAh0sm37OOCPKbMQERdYlsXDAxM4kJHFG0mbS3aQtqPg+EFInl7wY3YshSljTF3fiI/AL/DP+wJC4MqPwPKBySPhZAk7a4hIyT38MKljxjgdRdlImWOS3xqNC39cVIJ6Ibtq60zztlGi+455/r8gKBx+uL98b5Y8nApHt5uSHjdwNUHuBmywbfuIZVmjgIeAo26JQKSSaBNbjaHtYvjfnK1sP3S8+Ado3Aeq1DKb7vJzYBN8fAWE1oSRn5tfeGer3gCGvQv718M3t5XvX4Yi4r1s21zurt+j6NW86BamHVxF6qbgKVtnQngMRDRy3zFDIqDPw5A6B9ZMdd9xy1pBExtLyNUE+U3guGVZbYD7gVTgA7dEIFKJ3N8/Hh8Lnp6+vvhP9vWDNleaEor0vX+9L30vfDTU/H/0VAiLLvg4jftA30fML8J5rxQ/DhEpuQEDaPXAA05H4XmHtkD6btdW86ISIDsTDm31fFzlWW4ubJ1tyufcUELwFx2uMeUWPz4EWcfce+yykjoXgqtDzWZuOZyrCXK2bds2MAR42bbtl4Ewt0QgUonUrhrMTb0a8/2q3SxJOVT8A7QdCXYOrPr0z9sy02DS5WaQyMjPir6cCdDjTkgYYjbybf61+HGISMmcOIHvyZNOR+F5eat5LiXIzc1blVkUbu9qOHGodO3dCuLjCwOeM/XNc15y//HLQsrpKxal3bx4mqtHSbcs6x/AaOB7y7J8MXXIIlJMN/VuRHR4II9/t5bc4rZ9qxkHdTuZMgvbhuyT8OkoM6p1+AcQ08G141gWDHkDIuPh8+vgcEqxPw4RkQKlzDXlXpFxRT+2ZjPA0ka9ouT1P3bXBr2z1e8GrYbD3FfK32p+2i44vNVs0HMTVxPkEcBJTD/kPUAM8JzbohCpREIC/Lj/omas2nGUr1bsLP4B2o40NcQ7lsBXt5iatCGvQ9N+xTtOYBW4chLYuSbJzipBXbSIyNn+qD92sZtAQIipqVWrt8JtnQk1mkJ4Hc+d44LHwMcPZvzLc+fwhNR55q2b6o/BxQT5dFI8CahqWdZAINO2bdUgi5TQ0HYxtK5blWenb+B4VnbxntzyMvALhslXw+ov4ILHTW1ySdRoDJdNgD2r4ds7tGlPRErvyOluAoW1dztbdIIS5MLknDJJoKdWj/OE14He98GG72HTz549lzulzIHAcFNH7SaujpoeDiwCrgCGAwstyxrmtihEKhkfH4uHLklgT1om42cVsydxUFVoPsg04O86DrrfXrpg4i40bX5+nwIL3izdsUSkcAMHcrBbN6ej8KyUvPrjYqzmRSWYjX2nTngmpvJu5zLIyvBM/fHZuo6DiMYw7cGi++57i9S5UK+rqaV2E1dLLP6F6YE81rbtMUBn4GG3RSFSCXVuGMHFrWrx9swt7DmaWbwnX/AYDPyvmZLnjt3MPe+BZgPhx4eodrgY46xFpHjuvZftI0Y4HYVnpc6F4Aio2dz150QlmHKv/SXo8FMZbJ0JWGaCnqf5BUL/p+HgRlj4lufPV1oZ++FAslvLK8D1BNnHtu19Z7x/sBjPFZECPNi/OTm5Ns/OKOYfhfA60PFat+3WxcfHTN2r0ZiEtc/D0RLURouIgLncXb978X4/Rbcwb/et80xM5d3WWaZ8ICSibM4XdyHE9YeZz0D6nrI5Z0kVp2NKMbj63TvdsqwZlmVdY1nWNcD3wA9ujUSkEqpXI4Rrz2vAl8t2smrHEWeDCQqHER/hk3vSTOPLrgStqETKWmIibe+80+koPOfIdlODXNxkJaIR+AZWjHHH7pZ1HLYv9Hz98dkuehJyskw7UG+WOhf8Q6F2G7ce1tVNevcB44HWQBtgvG3blaDTuYjn3XZ+E2qEBvCf79ZiO71JrmY8G+Jvh51LYMY/nY1FRMqfkk4z8/GFmvHaqJef7QtNotoosWzPW6MxdLsNVn4C2xaW7bmLI3UexHYGX/d2H3b5+odt21/Ytn23bdt32bZdjmcRiniXsCB/7rkwnsUph5m22vlLWfujephfiosnwMrJTocjIuVJyhyzkTivZKI4oluoF3J+ts40rdfqObC5s+c9EFYHfrj33Amu3uD4IXPVwc31x1BEgmxZVrplWWn5/Eu3LCvN7dGIVFIjOsXSrFYYT01bR+apHKfDgX6PmRZN395pWsCJiLgiNW+aWQm6CUQlQMYek/SIkXUckmdATEfTu76sBVaBAc/Ant/hvy3hm7/D/uSyj6Mg2+YDdvE6prio0ATZtu0w27bD8/kXZtt2uNujEamkfH0sHhmUwI7DJ3jgi1XOl1r4+sEV70FwNTNE5ITD9dEi4v3SdptWbSVdzYtOMG9VZmHsWw/v9DEbFzte61wcCYPh70uh3ShYNQVe7wSfXAXbFjgXU57UeaZ23dUpssWgThQiXqJ740juvTCer1fs4s2Zm50OB6pEwRXvm4b/U2+G3FynIxIp/4YPZ19iotNReEZqCfofnynqdFmGyixgxcfwzvlwbD+M/rLkw6DcpUZjGPgS3Lkaej9gkuN3L4IJF8C6byHXxSufR3fC2m/gp0dg4kCY/o/SxZUyB+p2Mq3p3MzP7UcUkRIbl9iY9XvSeW7GBuKiwuiXEO1sQPW6mJ3M0+6HOS9Ar/ucjUekvBs3jl1JScQ5HYcnpMw+Pc2sdcmeH1YLgqrBvkrcySLrGHx/L6z82PQ8vnyC+bx4iyo14fx/Qo87TBI//zVzlTGiMXS/DdpcBf7B5rEn02HXctixBHYuNf/Sd5v7fPyhSrT5nulwLdQswU9E5lHYs8pjf5eUIIt4EcuyePby1qQcOMYdk5cz9dYexEWHORtU5xthx2L49Qmo0x6a9PXs+XJz3dffWcTbHD+OT2YxBwOVFymlnGZmWZV7o96+dTBlrBl60ftB6H2/WyfDuVVAKHT+m0lu130D816B7+6C35403Tb2rD499OV0uWBEY5Pw1+1oyiFqtTIJ9IsJsOANGPTf4sewfZEZLuOBDXqgEgsRrxMc4Mv4MR0ICfTjhveXcPiYw6M+LQsGvQxRzeGLG+DINs+da9kH8GwDM1ZVpCK6+GJaP/ig01G4X/peM3mttMlKVIJJFJ3eh1GWbBuWfwTjz4cTh2HMV3D+P7w3OT6Trx+0vAz+9huM/Q7qtIOts6FaLCT+A0Z9AfdvhduXweXvQJebTJLsFwihkaZ0ZOUncOxg8c+dMsesRNft5P6PCyXIIl6pdtVg3h7dgT1pmYybtIxTOQ7X/waEwvAPITfbDBE55YEVsCXvmh3SmUdh0TvuP77IruXw+fWe+f6t7P6oPy7lKOToBMhKN3sfKoOTGWaPx9e3QmwnuHlO2fc7dgfLgoY9YeRncO8G8zbxAWjSr/Dpf13HQXam+f1fXKlzIaY9BISUPO5CKEEW8VLt61XnqaGtmL/lIP/5zgsuOUY2gaFvmSRjupvnBC16x1yea3oRtLka1kxV5wxxv0UTYPXnsPxDpyOpWDL2w7L3IaBK6aeZRZ3uZFEZyiz2rjEb8VZ9alZbR38FYQ7vOylrUc2gyQWwaHzxprdmHTN/i+p391hoSpBFvNjlHepyY69GfDA/lUkLU50OB5pdAufdBUsnwjI3JRkL3zZN6OMvhhEfmrq27BMmkRFxl9wcSJ5u/j/3Zcg55Ww8FUH2SZjzX3jl9GX1XveZS+6lEdXcvK3IG/Vs2/wOfaePuWI29htIfLB8lFR4Qrdb4dg+WP2F68/Zvshc0axfzJHmxaAEWcTLPdC/GYnxNXn06zUs2FKCOi13O/8haNgbvr/HbMopjfmvmw4ZzQaalnJ+gaaGLbqVqUcWcZedS+H4AWg9wly+X/Wp0xGVX7YNa76C1zrBz4+aVbxxC+C8O0t/7KCqUDXW1CFXRMcPma4P395hNjTePAca9nI6Kmc1SjQt/ua/7nrteepcsHxMpyUPUYIs4uV8fSxeuaod9WuEcMtHS9l+6LjDAfnBsHfNpcCJF8OXN8LRHcU/ztxXYMY/oflguGIi+AWY2y0LOoyF3Sth1wq3hi6V2IZpYPmaqWC1WsPsF13v3epO11zDnv79y/687rJrObx3MXw21uxNGD0VRk4pWZuugkQlVMwSi82/wZvdzWS8C/8PRk01/eYrO8uCbuNg72ozVtsVqfNMOU+g57o8KUEWKQfCg/yZMLYTObk2f/tgCRkns50NKDQSbp4L591tVpJe7QC//p/ZcOKK2S/CTw9Di8tMsu3r/9f7Ww0DvyCtIov7JE83K53B1aHXvXBos6l1L2vlNUFO2w1Tb4HxiaYN2cCX4KbZ0LiP+88V1dyco6KUwWSfhBn/gg8vNX2i//YLdP+72lmeqdUVEBplVpGLcirT9Fb2UHu3PPrqiJQTDSNDeX1kezbuy+DuT1eQm+twG6SgcOj3KPx9iSmRmPUcvNreJLWFrczNeg5+ecz8QrzsnXOTYzBJTMKl8PtnZjOGSGkcTjXji+NOJ6bNBkFkPMx+oewnRB44gP/Ro2V7ztLIOg5Jz5if7dWfmwERty+DjteVvt64INEtIPcUHNjomeOXpX3r4J2+ZqBGpxvgxqTSb2SsiPwCzf6TjT/C/g2FP3bnEsg5CQ08V38MSpBFypWeTWvy0CXN+XHtXl76OdnpcIxq9WDY/+D6n6FafdOq7e1esCXp3McmPW1WmltfCUPfLvwPbPsxcDIN1n7tsdClksjbnBc/wLz18YGed5ukOXla2cYybBgtHn20bM9ZUjuXwmsdIelJ067r1kVwweOmTtiT8jpZ7CvHZRa2bbrzjE800+Ou+hQuecFjLckqhI7XmSuHC94o/HGp8wDL1HB7kBJkkXLmmu4NGNExlld/3cTcTQecDudPsZ3g+h9h2Hsmsf1gCHx8pVkFsm0ziS/pKWg7Ei59o+gd2/W7Q40mKrOQ0tswDWo0hRqN/7yt5TDzgm7W86UbSmHbZsPV9H+UPk5vM+t5Ux5wzQ+mw0xEw7I5b2Qc+PiV3wQ5Yx98PNx052nQE8bNh/hyWFZT1v4YHDK58MEhKXMguqW50uhBSpBFyhnLsnhsSAsa1Qzlvs9WkpbpRXV6lmWmKt26GPo9Zn6RvdEV3h8Es56FdqNh8GuutTOyLLOKvG1+0ZfcxDkfDTMJorfKTDPfh2cnKL5+pmXhrmWw+deSH3/J/0zLriXvmnKEiiIzDTb9DK2HQwPP1nqewy/AvDgujxv1kmfAG91gy0wY8JwZmKGNeK4ranBIdpZp8ebB/sd5lCCLlENB/r68OLwte9IyefxbL/wj4h9kWj7dvhzajzUteTpcC4NeKd7GlDZXmZUkrSJ7p20LYdNPsOJjMyLXG23+1dSzxg049762V0NYHVOLXBK7V8H0f0L1BuaPuqs78MuDDdMgJ8vsBXBCVEL564W88G2zchxWC26aCV1uNC/0xXU14wsfHLJ7hemTXwYv2pQgi5RTbWOrMS6xCZ8v3cFPa/c6HU7+qtSEgS/CA6kw6L/F37VdJcoMEFn5SfGmLEnZmPcK+AWbRKo4Tf7LUvJ0CKoGsfn0S/ULhB63mxdwqfOKd9yT6fD5tWaM7rXTICDMJJUVxdqvzIuHup2cOX90AhzZZj7P5cHJDFNC1rA33PDLnwNPpPjyBof8ns+wqJQ55q2HO1iAEmSRcu32vk1JqB3OP75cxcEML04gg8JL/tz2Y+H4Qdjwg/vikdI7sAnWfw/dbzOrfSs+cTqic+XmmF3xTS8seENo+7EQEmnqbV1l22ZQzqEtcPkECK8DTfqYy+tFdcW45RZ2Dh7s+rmckJkGm36BhCHOtSKLamHe7lvvzPmLa9kH5ipKn4fMFTQpuUaJBQ8OSZ1rOtCERno8DCXIIuVYgJ8PL45ow9ETp3joq9XYpdls5K0an28ma6nMonRWfgpv9XTfityC18E3ADrfaEoVdi6B/V7SWSXPjsXmxVVhG6QCQsyK1eZfYOcy14674mMzia/3g3+2moq/GDL2mEvAhRkxgv19PNA72J2Sp5s2Wi0cKq+A8jVyOjvLtHGr3wNiOzsdTflnWeZnct+av3ZDysk2ZV1lVBOvBFmknGtWK5y7Lohj2uo9fLNyl9PhuJ+PL7QbZaZQHU4t2TH2rjEbqSrK4IHism1TZ7tnFcx7rfTHO3bAJIltrjRlMK2Gmyl1Kz8u/bHdacM0U8PepF/hj+t0g2ld5kot8r71f3Yn6HXvn7c3vdCMvs1rKVeQ7dsJ3Lev6PM4aU1eeYWDyV61+uAfWj426q3+HNJ2Qg83jNoWo9WwcweH7FkFWellUl4BSpBFKoSbejWmfb1qPPzVavYczXQ6HPdrO9K8Xf5R8Z97YBNMHGg6LbxzvhmV6+1OnXDv8bYtgAMbzB+cea9C+p7SHW/xBLMprdtt5v2waGjS16xSOzG+uSB50/OK6tsbFA5dbob13xWekGUdN3XH/iGmtOLMbiwhEabOuag65NGjaf7kk0XHbtvma1XWJQZ53SucLK8Ac+6o5t7f6i03F+b815QENL3A6WgqjrzBIZt++rOLUepc81YJsoi4ytfH4oXhbcnKyeWBL1ZVvFKLarFmFXD5R+Yym6vS98JHQ80lu0tegIz98E4f+PEh723JtW8dPF3PvRu+lr5nRtyOnmounSc9XfJjnTphdpjHDYCacX/e3uYqSN/lPZ0cDm2F/evz716Rny43mxXLwlaRpz9oErbLxptOBWeL629WuY7uLFnMZ0qdZ75PJw0rvCesu3lDeUWe6ATz+fbm32fJ08yLz/PuUscKdzt7cEjqPIhoBOG1y+T0SpBFKoiGkaH8Y0BzZibv55NF250Ox/3ajzEJ2OZfXHt8Ztrp5OIAXP2ZuYx+60LTi3neq/Bmt/yn/TltyXumK8TsF91zvOOHzCXz1sOhVkvoeL2p5y5pb+kVH5u63u5//+vt8ReblVpv2az3x/Q8Fwc0hERAp+thzZdwcPO59//+OSx73yRCTfrmf4y8SX3umM63YpJZqc7YB19cV3Yr895QXpEnKsF8r2V4aUmKbcOcl8w00RZDnY6m4jlzcEjGPpMgl0H/4zxKkEUqkNFd69OjSQ3+7/u1bDvopSukJRXXH0JrurZZLzsLpow2tcfDP4C6HcztwdVg8Csw9jtTL/rBEPjqVu/p4Xsq02z+CqoKOxaZhviltXKyWRHscI15v/f9JvH6+bHiHys3x9QE1ml/7h8q/yBoeTms+9a8OHHahmlmt3tEI9ef0+028PE3Sc+ZDm42JTqxXeD8fxX8/Mg4qN4QNhRRh1yUkxkmUW15OVzyvHkh9+v/le6YrvijvGKws+UVef4YOe2lG/VS55mNoN1vL7hLipRO3uCQ7++BzCNQ/7wyO7UX/ASIiLv4+Fg8O6wNvpbFvZ+tJDfX9UuTtm2zdlcahzOLaFPlFL8A0y1hw7TCa2hzc+HrW01SMfjV/OsCG/aEW+aZ1cCVn8BrnU1C4vSl3PXfmT8CQ94wSfL8Um6os22zOTGmI9RqZW4LjTRDXDZ8X/zevxumwaHNpndwfpeT21xtmviv/ap0cZdW5lFTrxjvYnlFnrBo6DDWfE8cOX0VJvukqTv28YPL/we+/gU/37LMObfOgqxjJY9/7ddw6pjZnNp+jGlFN+dFWPddyY/piuQZp8srvGQ1NPp0q7d5r5orEwc2Of8zeqY5L5kWge1GOR1JxVUz3myAXfeNeb8MpzoqQRapYGKqBfPIoAQWpRzi3blbi3x86sFjvPLLRvq+OJOLX5nNf5ed9N4a5nZjwM4xl/kL8vOj8PsU6PMwtBtZ8OP8g6Hfv+HG30xN22djYfJISHOwE8jyD6FqPVOu0PE6sxp7OKXkx9s239RHdrz2r7d3HQdhteHHh4uXcMx71XQXaDYo//vrdoQaTZ0vs9j0C+RmFz9BBrMaCGYICsBPj8DulXDpm6YWvihx/U2Sufm3/O+/5x62Dx9e+DFWTIKIxn8ON7n4ObNqP/VmOLDRtY+jJNZ6UXkFnL7EfjXsWAJf3QyvdYBnG5rx5knPmK9z5lFnYtvzu9lA1vUW87tEPKfrOPO2aqwpZykjSpBFKqBhHerSr3k0z87YwMa95/a9PZhxkvfnpTD0jbn0fi6JF39KpmaVQC5rF0NqWi7LtnlJycHZIpuYS2zLPsh/IMOCN01i0+kG6HmPa8es3QZu+BUu+I8ZS/x6F/jpUVj2IWyZaTZ7lUV7uMOpZtW73UhzebvzTaZ12oK3Sn7MpRPN5ryzVwQDQkypwM4lZrXSFdsXwfYFpj9pQZeTLQvaXgXb5pkhGk5Jng4hNUo2Ba5arNlwuPR9WPw/WPiW+QPd7GLXnl+/OwRWLbgOedAgDnYvpI7y0Baz+t326j9X6f0CTamQXwB8OsqUYLhbZhps/Ml7yivyDH3TTOIct8BcEWo+yLRUS3oKProMnq5vfma/vtV8v2fsL5u45vwXAqqYunXxrEaJUK+b6axShlQ0I1IBWZbFk5e15KKXZnHPZyv54pbunMrJ5cc1e/lqxU5mbzxATq5Ns1phPDigGYPb1KFOtWCOncxm2u87eX9eKh3qRzj9YeSv/RiYeiOkzoGGvf68ffUXMP0f5g/ogGeLt6Pc18+UDTQfaGrd5r0C9hkJuOVjVtaq1Tvj3+nVjHrdTAJTWismAdafLe3Ca5sa1OUfQuKDpn66OPI257UfAwGh597f9mpTT/zLY2bF2i+g8OPNe8WMbG5byKo8QOsr4Zf/mNrn8/9ZvJjdISfblArED/hrG7biOO8u8/X4/m6o0w76FaNe29cfmvaD5B/Ni7izk80NGwjetq3g56/4BLBMkn6marEw7F34cCh8cxsMe8+9XRPyyisSvKB7xdnyWr5FNTffz2AS+p1LzeryjsWw/gfT5eb7e8z3c4ex0Oj8kn8PFObQVrOZs9utEFzd/ceXv7IsM869jLuEKEEWqaCiwoJ4Ymgrxk1axrC35pO8J50Tp3KIqRbMjb0acWnbGOJrhf3lOaGBfpwX48e01bvZl96cqDAvHJmaMBim3WdWkfMS5K2zzOXnel3hsndK/kcxotHpVminzCrVkW3n/kudZ0o48hLoZgPhykml+5hyc2D5JDM18MzL+N1uhVWTTfeEHncU75h/bM4bm//9Pr5wwePw8RVm5a3LjQUf6+BmU//a824IrFL4eavGQKPepo6394Nlvxq5faGp445zsXtFfmo0Non++u9NUlrUi4ezxQ0wL9h2LTNlJ2e66SbijxyBMWPOfV5urvm8NT7ffB7P1igR+j4CP//b1JV3v614cRVm7Vem7CavrMPbBYWbz1Pj8837tm1awq342Pxb9425JN9ulPlXta77zj3/NVOT3vVW9x1TCudACz0lyCIV2MWtanNFh7r8uHYvl7aL4dK2dejUIAIfn4J/2fSt589PqdlMXrSd2/s2LcNoXeQfDK1HmEvgAw6ZRHbySFOzedUn7qkH9PWH6g3Mv/zknDK1yovfMXW52xdDbAku5+fZ8huk7YAL//PX22u3Ni8CFr5tLvMXtkHsTLZteh+fuTkvP00vMBPhZj5t2ikFhef/uAVvmHN3LiSJPlPbkfDl30ypRYOy23UOmNIGH39oXMpxzoNfhYueMO3fiqtpP1Mes2HauQlyYVJmwdHtpja+ID3uNCunPz1iyoMa9ix+fGc7mW7KKzpe613lFcVhWWZT30VPmBcRG34wvyOSnjJ9v5v0M6vP8QNc/znKT8Y+s1Ld5soy68crziinPwki4qpnh7Vm5aMX8tRlrejSqEahyTFArVAfejaN5OOF28jO8dKOFu3HmNXR2S+YDTsBVWDU52V3udPXH6rXNyukIZHw2xOlO96yDyE4Appdcu593f5uXgSsKUZniG3z4UDyuZvzzmZZZhX5+EGY+3L+jzl20Kxutx6R/3CM/DQbCAFhhW+mLEr6nvzrzIuyYbpJygtK9l3l61ey5BjM92G9bkWPnT7b8kmmfrnZwIIfY1mmy0lEI9Ndwx1DSby5vKIk/AJN3f2Yr+COldDrPtPyccpoeLG52Zx6YFPJjr3wLdPZpHsxr+hIuaMEWaSCs0pwaWpMtwbsScvkp7V7PRCRG9RqZXb1z3/NtBUb9YV7L6G6KrCKaZm25TdImVuyYxw7aC7ltx6Rfy1zk36mv+78V13vOLHkvfw35+Unpj20HGbqkfPr4LHkf+Zz3K0Yl/MDQqDFELMBsCTtzlZ/CS80MwMyirNB8uBmOLixZN0r3C2+P+xdbcpyXJF51JQFtLrc9JQuTFA4jPjITIP8bKxJ2EpjzdTyVV5RHNUbQJ9/wV2rzcCg2C7me/21jmbPQnHGumemwaIJpswrsonHQhbvoARZRM7Rp1kUMdWCeX9+itOhFKzbrWa17cpPzEhap3S8HqrUMqvIJWmPt+pTyD0F7Ufnf7+Pj/lYd6803Q2KcvyQSUxbj8h/c15++j5s2uedvRJ+6oQp72h6EUQ1c+1YedqOhKwM06quOFLmwtSbTC32mqkwpRgJYN547tLUH7tL3ohrV4eGrJlqBiK0dbGnblQ8pxnCAAAgAElEQVQzuPR1s0FtRik2Q+aVVzT3su4V7ubjC3EXmv0Cd68z3ScWvAFv9zIlK65Y+h6cPGrKXKTCq8A/DSJSUr4+FqO61mfBlkMk59Mmziu0Ggb3bynTxvH5CggxLeVS5xZ/dLVtmy4VMR3+HIqQn9YjTCnHPBcGh6z85K+T81xRvYGpL17xsbkU/cexJsPxA+eOlXZFvW7muMUps9i/ASZfZXot3zgTLn7eDDT55CqzWlqU5Olm+lr1+sWP190im0CNJue2e3voIVJH5/NiaPkkqNnMrOi7qsVQ87VZPKHkvaf/GA5SQcorXBEWDZe8YDbkZh2DCRfAr0+YCZwFyT4J89+Ahr2L9zWScksJsojka0SnWAL8fPhgforToRTMW8a7dhgL4XXNOODirCLvXGp23rcrYPU4j3+w6e2cPK3w2sm8yXl1O0Gtlq7HASbJDwwzHRLA1P/Ofw1qty3ZRjvrdKuyrbP+nEpXmPQ9pp7cN9DUk4dEQOe/weDXTH/qj4cX3v/3xGHTYcQbVo/zxPWHlDlmlTZPv34c7tDhr4/bn2xGi5/Z+9hVff9tNlp+dydsW1j8GNdMNVdAYrsW/7nlXeM+ZqJm6+Ew61mY0Bf2rs3/sSsnQ8Ye0wJQKgUlyCKSr4jQAAa1rsPUZTtJzyyDQRnlmV8g9L7PDN7Y+KPrz1v2AfiHmH7HRel0g0keF7xe8GNS55nNecVZPc4TEmGS5I0/mgEpydPh4CazQlnSFkttrgRs06quMCfTTQJ8/CCMnPLX7iHtR5vWfanzzGCIgianbfrFlIl4Q/1xnvgBkJNlEvw8K1ZQZdNZL3JWfmy6XrS+svjn8PUzPZHD65jPz7YFrj/3ZDps+tkMYKjI5RWFCa4GQ9+CEZNMDf743mYISG7On4+xc8wm1tptTas9qRQq6U+EiLhiTLf6HMvK4ctlbtgpX9G1HWkSO1dXkbOOmc1oCZe61nGhSk1oM8JcSj92MP/HLJ1o6rJbXFacyP/U+SbTO/anh81gkKr1StfZoHoDM/lwxScFf05yTsFn18Ce1XDFRDOY42ytr4Ar3oOdy+D9wabO+mwbppkylJgO597nlNiuZrjKmXXId95Jk9fOKJXJzTGrk00vMJf+S6JKTbjme9Nl5MPLzIsJVyTPMHXPlam8oiDNB5ppfU0vNOPq37vYbPoEau5fAIc2m9VjB/rxijM8miBbltXfsqwNlmVtsizrwUIeN8yyLNuyrGI0jBQRT2sTW402sdX4YH4Kdkk2oFUmvv6m7dueVa5tTFvzFWSlF7w5Lz9dbzUdJZa8e+59f2zOG27qokvCPwj6PGQ2BG6bD93Glb6Mpe1VJrnYvujc+2wbvrvLrGIOfMlsoipIwhDTuWHfWpg48K8jhXNOwaafIO4iz0xOKylfP5NwbZzx1xXJM23+FdJ3m/KK0givY5Lk8DqmVCVlTtHPqczlFfmpUtN8jw0dD/vWwVvnweIJ1Nv2hemz3nyQ0xFKGfJYgmxZli/wOjAASACusizrnK3mlmWFAbcDJSieEhFPG9O1Ppv3H2Pe5gJWLeVPrYdDjabw25MFJ0R5ln9oNnHV6+b68aOaQZMLYNH4czs75G3OK6r3cVFaDTdt9IKqmglkpZUwxJSRrMxns96s58znodd9BU/8O1N8f7j6Uzi0BSZeDGm7ze3bFpjSC2+qP84T39+UjuxYkv/9KyaZHthxbigNCatlkuSqdWHSFab+uyAnM06XV1Tw7hXFZVnmSs24+WYy5/f3EJax2Yyi96YXX+Jxnvyp6Axssm17i23bWcBkYEg+j/sP8CyQ6cFYRKSELmldm4jQAN6fl+J0KN7PxxfO/wfsX2dW5wpyYKNZoW03qviXbLvdCsf2we+f/XmbbZvex3U7Fd4NwxU+PnDVp3DtNLNpr7QCw0wLsdVT/9pzdvkk01auzVVw/r9cP17jPqbvddoueG+A6TOcPB18A0o/Pc8TmvQzY4nP7mYBZtV//ffmhVVxx1kXJCwarvnOdAKZNLzgzirJ00+XV7jQK7syqhoDo76ES15kb1RP830qlYont4DHAGduXd4B/KULuWVZ7YBY27a/syzr3oIOZFnWjcCNADVr1iQpKcn90YrHZGRk6GtWjuT39eoWZfPD2r18Me1XagRrtalQdnU6htbH54eHWby/OnY+q06NNr9PLD7MP9aArOL+bNjQMbQB/PwsS47EkHHsGMu/foN2BzeyPv7v7HHnz9o69xyrmpVA25OTWTP1efZH9aT6oeW0+v0/HKneht+rXo49c2axjxne4hFar3qM7Df7ADbHw1uwan4Bq7QOaxOeQMCyz1nsl0jbI0fIyckhKSmJOju/Jy4niyXZ8WS4+Xekf9MHabPyEYI/uoLVLf/J4Yi/1na3WD2B8IDqzN+SCVvde+6KpTEZ9W6mypz5TgcixeCOvMOTCXJ+yyJ/FDFaluUDvARcU9SBbNseD4wHiI+PtxMTE90ToZSJpKQk9DUrP/L7ejVpc5xpz/7GFp86XJ5YzIERlVGtJ+HTkfSuvgfajfzrfTmnYPHfIL4/3S8q4epd9Qfgq1tIjM0laUcV2h1cDoFVaXb5P2lW0vpjT8rtBVvH0yJrBcQPgfdegKjmRFz7Hb1LPBI6ETp1we/DoXDiEEF9HySxc6Ibg3ajwKtgxj9IbF0f3niDZcuWmZ+x8f+G6FZ0HHSdZ87boyd8MIQ2a56CKz+Gpv3M7SczYM5yaD+GxPO9cNXdy+hvWPnjjq+ZJ5eCdgCxZ7xfFzhzjmkY0BJIsiwrBegKfKONeiLep271EPo2j2byou2czC6itlag2SWmJdTMZ84dlbzxR1MiUVTv48K0HGY2V81/Hb9TaaXfnOdpPj6m5dvmX2HSMNO1Y+RnrnXvKEydtqbmtt1o11rlOSX+dG108nTo3p20li1Nv91dy899AeVOoTVg7DdQM94MYEme8Wcc2Zml61AiUsF5MkFeDDS1LKuhZVkBwJXAN3l32rZ91LbtSNu2G9i23QBYAAy2bds7r5GJVHJjutXn4LEsfvh9t9OheD/LMnW1R1Jh+Ud/vW/Zh1Al2nQ3KCm/ADNEY/MvNN78vum1W9rNeZ7W5iqwc+FUJoz83HRbcIfoBBjymunj7K0iGkFkvGlFN28e4atXm815Pv5mU6QnhUTAmK/NhMHJI00Ma78y34P11L1CpCAeS5Bt284GbgNmAOuAKbZtr7Es63HLsgZ76rwi4hk9GkfSKDKU9+elOh1K+dD0Aqjb2XRqOHV6D3L6HrOC3Oaq0rdP63gd+IdQe8/P7tmc52mRTWDQyzBmqklqK5v4/mYc+YP302jCO7DqU9OWLrSG588dEgFjvjLdST4dbVaSE4aoK4NIITy628a27R9s246zbbuxbdtPnL7tEdu2v8nnsYlaPRbxXj4+FqO71WfF9iOs2nHE6XC8n2VBn39B2k5Y9r65bcXHZipXacor8oREmOEkAB28fPU4T4drvGuQR1mKvxhys+HEEfxOpcOx/e5po+eq4OoweirUbmOuOKh7hUihtB1dRFx2eYe6hAT48sF8rSK7pGFvaNATZj1vJuct/wjqdTerqe7Q6z5S6w3z7vpbMep2gpAacOIQAVlHIDTKtIArS8HVzEry6K+gfveyPbdIOaMEWURcFh7kz9B2MXy7cheHj2U5HY73y6tFPrYPvrzRTJQrzuS8ooRFs7XRaDMBT7ybj6+pOz9+CP9T6WZTpa9/2ccRGAaNzy/784qUM0qQRaRYxnRrwMnsXKYs2V70gwXqd4PGfWH9dxAQZmo/pXKK6w+52VjYZVteISLFpgRZRIolvlYYnRtG8OGCVHJy7aKfIH9Oimt1OQSEOhuLOKdxHxhQhYzL4iCqudPRiEghlCCLSLGN7daAHYdP8OOaPU6HUj7U7WDG1vZ91OlIxElB4XDz22xIvMPpSESkCEqQRaTYLmwRTYMaIfz9k+W8+FMyWdm5Tofk/Zr09e5evVI29tfAb2OG01GISBGUIItIsfn7+jB1XA8GtanDK79sZPBrc/h9x1GnwxLxfv/3f9T/8EOnoxCRIihBFpESqR4awEsj2vK/sR05fDyLS9+Yy7PT15N5SqOoRUSkfFOCLCKl0rd5ND/e1ZvL2sXwRtJmBr46h2XbDjsdloiISIkpQRaRUqsa7M9zV7Rh4rWdOH4ym2FvzuOJ79dqNVlERMolJcgi4jaJ8VHMuKsXV3auxzuztzLg5dks2nrI6bBERESKRQmyiLhVWJA/Tw5txaQbunAqJ5cR4+fz72/WaDVZBODtt9lw991ORyEiRVCCLCIe0aNJJDPu7MXYbg2YOC+FF37c4HRIIs6Lj+dEvXpORyEiRVCCLCIeExrox78Ht2BYh7q8Py+VHYePOx2SiLO+/ZYa8+Y5HYWIFEEJsoh43N0XxGFZ8OKPyU6HIuKsF14gdsoUp6MQkSIoQRYRj6tTLZhrejRg6oqdrN2V5nQ4IiIihVKCLCJlYlxiE6oG+/P09PVOhyIiIlIoJcgiUiaqBvtz2/lNmJW8nzkbDzgdjoiISIGUIItImRndrT4x1YJ5ato6cnNtp8MRERHJlxJkESkzgX6+3HtRHGt2pfHtql1OhyNS9j78kHX//KfTUYhIEZQgi0iZGtImhoTa4Tw3YwMnszU8RCqZ2FhORkU5HYWIFEEJsoiUKR8fiwcHNGPH4RN8tGCb0+GIlK1PP6Xmr786HYWIFEEJsoiUuV5xNTmvSSSv/bqRoydOOR2OSNl5801ivvnG6ShEpAhKkEXEEQ8OaMbh46d4a+Zmp0MRERH5CyXIIuKIljFVubRtHd6ds5XdR084HY6IiMgflCCLiGPuuTAe24aXftIIahER8R5KkEXEMbERIYzuVp/Pl+4geW+60+GIiIgASpBFxGG3nd+E0EA/npmmEdRSCXz+OWsee8zpKESkCEqQRcRR1UMDuCWxMb+s38eCLQedDkfEsyIjOVW1qtNRiEgRlCCLiOOu69GQWuFBPDVtPbatEdRSgU2cSK3p052OQkSKoARZRBwX5O/L3RfGsXL7EX74fY/T4Yh4jhJkkXJBCbKIeIXL29clPjqM52as1whqERFxlBJkEfEKvj4WD17cjJSDxxk1YSH70jOdDklERCopJcgi4jXOj4/i5Svb8vvOowx6dQ7Lth12OiQREamElCCLiFcZ0jaGL2/pQYCfDyPens/HC7c5HZKIiFQySpBFxOsk1Ann29vOo2ujGvxz6u88+MUq1SVLxfDDD6x6+mmnoxCRIihBFhGvVC0kgInXdmZcYmMmL97OiLcXsPvoCafDEimdkBByg4KcjkJEiqAEWUS8lq+Pxf39m/HWqPZs3JvOoFfnsFDDRKQ8e+MN6nz1ldNRiEgRlCCLiNfr37I2X93ag7Agf0ZOWMh7c7dqoIiUT1OmEJWU5HQUIlIEJcgiUi40jQ7j69t6kBhfk8e+Xcs9U1aSeUp1ySIi4n5KkEWk3AgP8mf86I7c1S+OL5fvZNhb8zhyPMvpsEREpIJRgiwi5YqPj8Ud/ZoyYUxHkvdkcOMHS7WSLCIibqUEWUTKpX4J0Tw/vA2LUg5xz2cryc1VTbKIiLiHn9MBiIiU1OA2ddh95ARPTVtPnapB/OuSBKdDEilcUhIrkpJIdDoOESmUEmQRKddu7NWIXUdO8M7srdSpFsy1PRo6HZKIiJRzSpBFpFyzLItHBrVg99FMHv9uLbWrBtG/ZW2nwxLJ3/PPE7t5MyQmOh2JiBRCNcgiUu75+li8fGU72sZW447JK1iaesjpkETy99131Jg/3+koRKQISpBFpEIIDvBlwpiO1K4axA3vL2HL/gynQxIRkXJKCbKIVBg1qgQy8drOWJbFNe8t5kDGSadDEhGRckgJsohUKA0iQ/nf2I7sS8/k+omLOZ6V7XRIIiJSzihBFpEKp1296rx6VXt+33mUv3+8nOycXKdDEjGCg8kJDHQ6ChEpghJkEamQLkiI5rHBLfhl/T4e/WYNtq1BIuIFpk3j92eecToKESmC2ryJSIU1ulsDdh7J5K2Zm6kW4s8tiU2oEqhfeyIiUjj9pRCRCu3+i+LZm5bJ679t5n9ztnJhQi2GtovhvKaR+PvqIpqUsf/8h/pbt6oPsoiXU4IsIhWaj4/Fi8PbMLJLPaYu38l3q3bzzcpd1AgNYFCbOgxtF0PrulWxLMvpUKUy+OUXqh854nQUIlIEJcgiUuFZlkXHBhF0bBDBo4NakLRhH1+t2MnHi7YxcV4KjSJDubRdDEPbxRAbEeJ0uCIi4jAlyCJSqQT4+XBhi1pc2KIWR0+cYvrq3UxdvpMXf0rmxZ+S6Vi/Ojf1bswFCdFOhyoiIg5RAZ6IVFpVg/0Z0akek2/sxtwH+3B//3gOHcvipg+X8PPavU6HJyIiDlGCLCICxFQLZlxiE767/TxaxVTltk+WsWzbYafDkoqmRg1OhYc7HYWIFEEJsojIGUIC/Hj3mk7UCg/i+omL2bw/w+mQpCL54gvWPP6401GISBGUIIuInKVGlUDev64zvj4WY99dxL60TKdDEhGRMqQEWUQkH/VrhPLuNZ04dCyLa95bTHrmKadDkorgH/+g4TvvOB2FiBRBCbKISAFa163Gm6M6kLw3nZs/WkpWdq7TIUl5N38+VdescToKESmCEmQRkUL0jqvJM5e3Zu6mg9z3+Upyc22nQxIREQ9TH2QRkSJc3qEue9MzeXb6BqLDg/jnxc2dDklERDxICbKIiAtu6d2YvUczGT9rC9HhQVx/XkOnQxIREQ9Rgiwi4gLLsnhkUAv2Z5zkP9+tJSoskEFt6jgdlpQ3dety0t/f6ShEpAhKkEVEXOTrY/Hi8LYcSF/EPVNWUqNKAN0bRzodlpQnH33EuqQkNMhcxLtpk56ISDEE+fvyzpiONIgM4aYPlrJud5rTIYmIiJspQRYRKaaqIf5MvLYzoYF+/O2DJRw9rh7J4qI776TJa685HYWIFEEJsohICdSpFsybo9qzNy2Tu6esUPs3cc2KFVTZtMnpKESkCEqQRURKqF296jx0SQK/rN/HW7M2Ox2OiIi4iRJkEZFSGNOtPoPa1OH5GRuYt/mA0+GIiIgbKEEWESkFy7J4+rJWNIwM5fZPlrM3LdPpkEREpJSUIIuIlFJooB9vjerA8awc/v7xck7l5DodkniruDiO163rdBQiUgQlyCIibtA0OoynLmvFopRDPD9jg9PhiLcaP57ke+91OgoRKYISZBERNxnSNobRXevz9qwtzFizx+lwRESkhJQgi4i40UMDm9OmblXunbKSlAPHnA5HvM2NNxL3/PNORyEiRVCCLCLiRoF+vrw+sj2+vha3TFpG5qkcp0MSb5KcTMiOHU5HISJFUIIsIuJmdauH8NKItqzbncYjX692OhwRESkmJcgiIh5wfnwUf+/ThClLdjBl8XanwxERkWJQgiwi4iF39ovjvCaRPPz1atbsOup0OCIi4iIlyCIiHuLrY/HylW2pHhLAuEnLOHrilNMhidPatiWjSROnoxCRIihBFhHxoBpVAnl9ZDt2Hj7B8Lfms2V/htMhiZP++1823Xab01GISBGUIIuIeFiH+hG8d20n9qVnMuS1ueqRLCLi5ZQgi4iUgZ5Na/Ld7T1pWDOUmz5cytPT1pOtkdSVz6hRNH/iCaejEJEiKEEWESkjMdWC+ezmblzdpR5vzdzMmHcXcSDjpNNhSVnasYPA/fudjkJEiqAEWUSkDAX6+fLk0FY8N6w1S1MPM/CVOSzbdtjpsERE5AxKkEVEHHBFx1i+uKU7/n4WI96ez4fzU7Bt2+mwREQEJcgiIo5pGVOV727rebpX8hrumbKSE1kaTS0i4jQlyCIiDqoa4s//xnbirn5xTF2xk6FvzCXlwDGnwxJP6daNoy1aOB2FiBTBz+kAREQqOx8fizv6NaVNbFXu/HQFg16dw4BWtejSsAZdG9cgplqw0yGKuzz1FFuTkqjvdBwiUiglyCIiXiIxPopvbzuPp6atY8aavUxZsgOA2Ihgkyw3qkGXhhHERoQ4HKmISMWmBFlExIvERoTwxsgO5ObarN+TzsKtB1mw5SC/rNvL50tNwhxTLZgujSLo2qgGvifUS7lcufxyWuzfD7NmOR2JiBRCCbKIiBfy8bFIqBNOQp1wru3RkNxcm+R96SzccogFWw6StGE/Xy7bia8FUU3207NpTadDFlccPIh/WprTUYhIETy6Sc+yrP6WZW2wLGuTZVkP5nP/3ZZlrbUsa5VlWb9YlqWyLBGRfPj4WDSrFc7Y7g14c1QHlj7Ujx/v6kWtUIs7J69gX1qm0yGKiFQYHkuQLcvyBV4HBgAJwFWWZSWc9bDlQEfbtlsDnwPPeioeEZGKxLIs4qLDGNc2iONZOdw+eTk5ueqjLCLiDp5cQe4MbLJte4tt21nAZGDImQ+wbfs327aPn353AVDXg/GIiFQ4MVV8+M+lLVmw5RAv/7LR6XBERCoET9YgxwDbz3h/B9ClkMdfD0zL7w7Lsm4EbgSoWbMmSUlJbgpRykJGRoa+ZuWIvl7lS0ZGBpFs4rwYP179ZSNBadtJqOHrdFhSgPqNGpGVlcUK/YyVG/qdWP6442vmyQTZyue2fK//WZY1CugI9M7vftu2xwPjAeLj4+3ExEQ3hShlISkpCX3Nyg99vcqXvK9X5+7ZDHltLu+uO8UPd3QlKizI6dAkP4mJ+hkrZ/T1Kn/c8TXzZInFDiD2jPfrArvOfpBlWf2AfwGDbds+6cF4REQqrJAAP14f2Z6Mk6e4c/IK1SOLiJSCJxPkxUBTy7IaWpYVAFwJfHPmAyzLage8jUmO93kwFhGRCi8uOozHh7Rk3uaDvPqr99Ujpx48RuapHKfDcNaAAbR64AGnoxCRIngsQbZtOxu4DZgBrAOm2La9xrKsxy3LGnz6Yc8BVYDPLMtaYVnWNwUcTkREXHBFh7pc1j6Gl3/ZyLxNB5wO5w+/rt9L3xdm8q+pq50OxVknTuB7UhdLRbydR/sg27b9g23bcbZtN7Zt+4nTtz1i2/Y3p//fz7btaNu2257+N7jwI4qISGEsy+I/Q1rSKDKUOz5dwf5055OxhVsOcstHy7As+HrFTvYcVc9mEfFuHk2QRUSk7IUGmnrktBOnuOtTZ+uRV+88yg3vL6Fu9WA+u7k7ubbN+/NTHItHRMQVSpBFRCqgZrXCeXxIC+ZsOsAbv21yJIbN+zMY++4iwoP9+eiGLrSNrcZFLWoxaUEqx05mOxKTiIgrlCCLiFRQwzvGcmnbOrz0czILthws03PvPHKC0RMWYlnw0Q1dqF01GIAbejYkLTObz5fuKNN4vMbAgRzs1s3pKESkCEqQRUQqKMuyeGJoKxpEhnL7J8s5kFE29cgHMk4yesJC0k9m8/51nWkYGfrHfR3qR9CuXjXenbu1craiu/deto8Y4XQUIlIEJcgiIhVYaKAfr1/dnqMnTnHD+0v4cH4KS1MPkeGhEoe0zFOMfXcRu46e4L1rOtGiTtVzHvO3no1IPXicn9bu9UgMIiKl5clJeiIi4gWa1w7nmctb8+9v1/Dw12v+uL1BjRAS6oTTvFa4eVs7nNpVg7Cs/AahFu1EVg43TFxC8t503hnTkY4NIvJ93IUJ0dStHsyE2Vvo37JWic5VbiUm0vbIEVixwulIRKQQSpBFRCqBS9vFMKRtHXYfzWTtrjTW7U5j7e401u5K44ff9/zxuGoh/iTUDqd13Wr0bBpJh/rVCfL3LfL4p3JyGTdpKYtTD/HqVe1IjI8q8LF+vj5c16Mhj3+3luXbDtOuXnW3fIwiIu6iBFlEpJKwLIs61YKpUy2YfgnRf9yecTKbDXtMsrx2dzprdx3lf3O28NbMzQT5+9ClYQ16No2kV1xNmkZVOWeFOSfX5p4pK/ltw36euqwVA1vXKTKW4Z1ieennZCbM2crrVytBFhHvogRZRKSSqxLoR4f6EXSo/2dJxLGT2SzcepBZyQeYvXE///f9Ovh+HdHhgfRsWpOeTSM5r0kkEaEBPPrNar5ZuYsHBzTjqs71XD7n1Z3r8c7sLWw/dJzYiBBPfXgiIsWmBFlERM4RGuhHn2bR9GlmVpp3HjnBnI37mbXxAD+v2/tHm7YGNUJIOXicm3s35ubejYt1jrHdG/C/OVuZOC+FhwcmuP1jEBEpKSXIIiJSpJhqwYzoVI8RneqRk2uzeudRZm/cz5xNB7iwRS0e6B9f7GPWqRbMJa1r8+ni7dzRrynhQf4eiNzLDB/OvuRkqjkdh4gUSgmyiIgUi6+PRZvYarSJrcZtfZqW6lg3nNeIr1fs4tNF2/lbr0ZuitCLjRvHrqQk4pyOQ0QKpT7IIiLimFZ1q9KlYQTvzd1Kdk6u0+F43vHj+GRmOh2FiBRBCbKIiDjqbz0bsetoJj+s3lP0g8u7iy+m9YMPOh2FiBRBCbKIiDiqT7MoGkWGMmH2Fmy7Eo6fFhGvowRZREQc5eNjcd15DVm14yiLUw47HY6IiBJkERFx3uXt61I9xJ93Zm9xOhQRESXIIiLivOAAX0Z1rc/P6/ay9cAxp8MRkUpOCbKIiHiF0d3q4+/jw7tztjodiudccw17+vd3OgoRKYISZBER8QpRYUEMaVuHz5Zu58jxLKfD8QwlyCLlghJkERHxGtf3bEjmqVwmLdzmdCieceAA/kePOh2FiBRBCbKIiHiNZrXC6dk0konzUjiZneN0OO43bBgtHn3U6ShEpAhKkEVExKvc0LMR+9NPMnnRdqdDEZFKSgmyiIh4lV5NI+nZNJJnpq9n+6HjTocjIpWQEmQREfEqlmXx9OWt8bEs7vt8Jbm5mq4nImVLCbKIiHidmGrBPHRJcxZsOcRHC1OdDkdEKhklyCIi4pVGdIqlV1xNnvphPakHK8jwkFtuYefgwU5HISJFUIIsIlPx+xEAAB0PSURBVCJeybIsnrm8FX4+Fvd9vqpilFqMGMH+Pn2cjkJEiqAEWUREvFbtqsE8PCiBRVsP8f78FKfDKb3t2wnct8/pKESkCEqQRUTEq13RoS7nx9fkmenrSTlQzkstRo+m+ZNPOh2FiBRBCbKIiHg1y7J46rLW+Pv6qKuFiJQJJcgiIuL1alUN4t+DWrA45TDvzUtxOhwRqeCUIIuISLlwWfsY+jWP4tnp69myP8PpcESkAlOCLCIi5YJlWTw5tBVB/r7c9/kqclRqISIeogRZRETKjajwIB4b3IKlqYd5d85Wl55j2zardhzh39+sYfT/FvL1ip1k5+R6ONIC3HMP24cPd+bcIuIyP6cDEBERKY4hbevw/e+7ee7HDZzfLIomUVXyfdzOIyf4avlOvly2g837jxHg60PNsEDumLyC53/cwI29GnNFh7oE+fuWXfCDBnEwLKzsziciJaIEWUREyhXLsnhiaEsufGkW9362ki9u6Y6vjwVAWuYppv2+my+X7WTh1kMAdGpQnevPa8QlrWoTFuTHT+v28mbSZh7+ajUv/5zMtT0aMqprfaoG+3s++A0bCN62zfPnEZFSUYIsIiLlTlRYEI8PacntnyznrZmbaV47jC+X7eSntXs5mZ1Lw8hQ7r4gjqHtYoiNCPnLcy9qUYsLE6JZuPUQbyZt5rkZG3gzaTNXd6nH9ec1JDo8qMjzZ2XnknLwGBv2pLNxbzpYFs1qhdGsVhj1a4T+kbCf46abiD9yBMaMccenQUQ8RAmyiIiUS4Na1+aHVbt5bsYGAKqH+DOiUyxD28XQNrYallVAkopZhe7aqAZdG9Vgza6jvD1zCxNmb2Hi3BQuax/Djb0a0ahmFbJzckk9dJzkPekk780geV86yXvS2XrgGNmnNwnm5cJ5ewaD/H2Ijw6jWa1w4muF0ax2GM1rhVM9NMCjnw8RcR8lyCIiUi5ZlsWTl7WibvVgujSqQe+4mgT4FX/veYs6VXnlqnbce2E842dvZsqSHXy6ZDuNIkPZfugEWWds6KsXEUJcdBX6JUQTF12FuOgwGtc0NdAb92awbk8aG/aks35PGj+t28unS7b/8dzo8EAm7kknyLIJyc4tUawiUjaUIIuISLkVERrAQwMT3HKsejVC+L9LW3FH3zgmztvK+t3p9G0eTVx0GHHRVWgSVYWQgIL/bLaqW5VWdav+8b5t2+zPOMn63els2JPOuj1pZGXncDQrl9EvJHF7n6Zc1j4GP18lyiLeRgmyiIjIGWqGBXLfRc1KfRzLsogKCyIqLIhecTUBsN+oxs79h4kIDeD+L1bxRtIm7ujXlMFtYgquWxaRMqeXrSIiImXEeughDl43hq9v7cH40R0I8vflrk9XctF/Z/Hdql3kaviJiFfQCrKIiEhZ6dePw35+WJbFhS1q0a95NNNW7+Gln5O57ePlNKu1ibsuiOPChOhCNxmKiGdpBVlERKSsrFhBlU2b/njXx8fikta1mXFnL/47oi2Zp3K46cOlDH5tLr+t34dta0VZxAlKkEVERMrKnXfS5LXXzrnZ18fi0nYx/Hx3b54d1prDx7O4duJiLn9zHvM2HXAgUJHKTQmyiIiIl/Dz9WF4x1h+vSeRJ4a2ZPfRTK6esJArx89nScohp8MTqTSUIIuIiHiZAD8fRnapz2/3JvLooAQ27TvGsLfmM/bdRazacaRExzyVk8uirYf4YukOcrQZUKRQ2qQnIiLipYL8fbm2R0NGdIrlg/mpvDVzM4Nfm8uFCdHcdUEczWuHF/hc27bZcuAYczYeYPbG/czffJBjWTkAbNqfwQP9S9/KzpNs2+atmVvo2TSSljFVi36CiBspQRYREfFyIQF+3Ny7MSO71OO9uSm8M2sLP66dzcDWtbmzXxxNosw0v8PHspi7+QCzkw8wZ9MBdh45AZgJgJe2i6Fn00h+W7+fN5M206xWGEPaxjj5YRVqxpq9PDN9PV8sq8L0O3pqoIqUKSXIIiIiZeXJJ9mybBntS/j0sCB/bu/blLHdGvDO7C28O3crP/y+mwsSotl9NJPfdx7FtiEsyI8ejSO5JbExPZtGUr9G6B/H6NMsmi0HMnjgi1U0rlnFK1dns7JzeXraOsKC/Ni0L4Mvlu1gRKd6TocllYgSZBERkbLSvTtpWVmlPkzVEH/uvSiea3s04K2Zm/l86Q6aRFXhzr5x9IyLpHVM1QJXXAP8fHhzVAcGvzqHv32whG9uO4+aYYGljsmdJi1MJeXgcd69piMv/397dx5eVXXvf/z9PUkIhkACJAwhCXPCICREBBlFJpE6ixPVS1HkVrGtSKst7a+9UnuvtD8U56Gg4IQgXkRBUUGQSZkJs4Q5YQZJkDnDun/koGEIBEjOQD6v58mTs/dZZ+3veb5Pdr5nnbX3mrGB577K4JbUOlQMC/F3aFJO6PsKERERX5k/nyqrVpVad9Ujw/nzL5qx7K89+fDX7fld98akJVY973SEmMhw3viP1hw4coKH313CibyCUovpUuUczeX5GRl0aFSd65Jr8GSvZHYdPMbY+Vv8HZqUIyqQRUREfGXoUBqMGuXvKAC4sk4U/+qTwuKtB/jbJ6sCZlGSl2duIOdoLkN7N8XMaN8whmuTYnll1kZyjub6OzwpJ1Qgi4iIlFM3pcTxSJeGjFuYybvfbfV3OGT+cIQx87ZwR1o8zeN+nhv9RK9kco7m8to3G/0YnZQnKpBFRETKsSE9k+napAZPfbqGbzfu92ssw6etw+OB3/dMPmV/87gobkmN4615m9l98JifopPyRAWyiIhIORbiMUbek0rd6hEMen8pmT8c8UscS7cdYMqKnQzs1IBaURXPeH5Ij2TyCxwjp2f4ITopb1Qgi4iIlHNVKoYxqt/V5OUX8NDbizl8PM+nx3fO8d9T1xITGc7AaxuetU1i9Qj6tklkwuJMNu495NP4pPxRgSwiIuIrI0ey4dFH/R3FWdWPqcSLfdNYv/tHfv9huk8v2pu2aheLtx7g8R5JRIYXfwfaR7s2JjzUw4gvv/dZbFI+qUAWERHxldRUDjVq5O8oinVtUix/uqEpn6/axYtfb/DJMU/kFfDMtHUk1Yzkrtbx52wbWzmcAZ0a8NnKXaRnZvskPn86dDyPR95bwptzN/s7lHJHBbKIiIivTJ9O1SVL/B3FOQ3oVJ/bWtXh2a/WM23VzjI/3rvfbWXr/iP8qXfTEi0n/VCn+lSvVIFnPl8XMLemKwtHTuTxwFuL+GzlLoZNWcOkZVn+Dqlc0Up6IiIivvL009TNzoYhQ/wdSbHMjP+5vQWb9h7i1+8upUblcFrGR9EyPpoW8VGkxEdTrVKFUjlWzpFcXvg6g46NYuiSFFui11SuGMajXRvx1KdrmJ2xj2tL+Lpgciw3nwFjF7N46w+MuDOFD5dk8sTEFdSOuoJrGlQvlWPkFzhCPFYqfV2OVCCLiIjIKSqGhTD2gTZ8vGw7K7JySM/KZsa6PZwcsK0TfQUpCYVFc8s6UVwZH0WVimEXfJyXZmacsihISfVtm8jouZsZ/vk6OjWKwXMZFXrHcvMZ+M4Svt20nxF3pnB7Wjzdm9bk9lfn8Z/vLOGjh9vTqEbkJR1j3MJt/H3KGob2bsp919QtpcgvLyqQRURE5AzRERX4VYf6P23/eCyXVdsPsnJ7NulZOazMyuGzlbt+er5lfBT92tXjxpTahIeGnLf/bfuPMHb+VvqkxdMsrsoFxRYeGsKQnkkMHp/Opyt2cEtqnQt6faA6kVfAoPeWMnv9Xv55R0tuTyuckx0VEcaY/m247ZV59B+zkEmPdCAmMvyC+3fO8fyMDEZOz6BqRBh/+XgVBc7xH+3qlfI7CX4qkEVEROS8KlcMo13D6rRr+PNX/AcOn2DF9hxWZGYzOX0HQz5M55lp67j/mrr0bZt4ziJu+BfrCPEYQ05bFKSkbkmpw+vfbGLEl+u54craVAgN7suqcvML+M24pcxYt4enb72Su65OOOX5hGoRjOp3Nfe88S0Dxi7mg4HXUDHs/B9ETsrLL+D/TV7NuIXb6HNVPMNuac7vPljOXyevpqDAnfJhSHSRnoiIiFykqpUqcG1SLL/p1pivBnfm7Qfa0DyuCs9+tZ72z3zNExPTWbvz4BmvW7L1AFNX7OShzmdfFKQkPB7jyV5N2PbDET5YtO1S34pf5eUX8Nj45Xyxejd/u6lZsdMeUhOiGXl3K9Kzshk8fjkFBSW7SPFYbj4Pv7eUcQu3Mei6hvyrT0siKoTyct80rm9ek//6dA2jdaeMU6hAFhER8ZXXX+f7xx/3dxRlwszonBTLmP5tmP74tdzVOp5P0ndww/Nz6Pvv75ixdjcFBQ7nHP+YuoaYyHD+s3ODSzpml+RY2tSvxgszMny+uElpyS9w/GHiCqau2MnQ3k3of56R3F5X1uLPvQtvxffMtHXn7T/7yAl+OWoB09fu5qmbm/OH65v8NN+7QqiHl/qmccOVtfj7lDWMmrOpVN7T5UAFsoiIiK8kJ3M0MdHfUZS5RjUiefrWFnz3p2482asJm/cd5sGxi+k6YhZDJ61i6bZshvRMotI5FgUpCTPjjzc0Yd+hE0E5AlpQ4PjjRyuYtGw7v++ZxMDOZ19F8HQPdqxPv3Z1eWP2Jt75bmux7bZnH6XPa9+yMiuHl/um0a99vTPahIV4eOHeVvyiRW2enrqW17/ZeLFv57KiAllERMRXPv2U6vPn+zsKn4mOqMDDXRoy+4nrePHeVlStVIFxC7eRXLMyd7VOOH8HJZCWWJXrm9fkjdmb2JVzrFT69AXnHH+ZvIoPl2Tx226NebRr4xK/1sz4603N6dakBn+bvIqZ6/ac0eb7XT9yxyvz2Z1zjLEPtKF3i9rF9hcW4uH5e1K5sWVt/ufzdbw6S0WyCmQRERFfGTGChAkT/B2Fz4WFeLgpJY5Jj3Rgym86MvaBNqV6D94/XN+E3PwCrh85m/cXbCvx3Nzz2fvjcdbszye/lPo7yTnHU5+u4f0F2/j1tQ0Z3L3kxfFJIR7jhXtb0SyuCoPeX8qq7Tk/Pbdg0376vDYfh2PCr9udcmFlcUJDPIy8O5WbU+IYPm0dL8/0zUqKgUoFsoiIiPjMlXWiLvrCvOI0qhHJ1N92pEmtygydtJI7XpvP6h05539hMbKPnGD4tHV0/udM/rnoGD2e+4ZP0neUSuG9Zd9hHp+Qzpj5W3iwY32e7JV8QfeALqpSeChv9rua6CvCeHDsInZkH+XzlTu5/82F1KgczkcPt6dp7ZLfQi80xMOzd6Vwa2oc//rie16YkXFRcV0OdJs3ERERCXqNalTmg4HXMGnZdv4xdS03vTiXX7Wvz+M9k4gs4VznH4/lMnruZkbP2cyhE3ncnBJHbP4+5uzx8Ntxy3jp6wwGd0/i+ua1LmhxEuccS7Ye4N9zNvHlmt2EeoxHujTkD9dffHF8Uo0qFXmrfxv6vDqfPq/OZ+fBY7RKiGZ0v6upehErHoaGeBhxVyoej/HsV+spcI7HuiddUozBSAWyiIiIXBbMjNvT4unWpCbDv1jHW/M3M3XlDv56Y3N6t6hVbDF65EQeY+dv5fXZG8k+kkuv5rUY3COJ5FqVmTVrFkPv7cTUlTsZOX09D7+3lGa1qzC4RxLdm9Y4Z4GbX+D4YvUu3pi9ieWZ2URdEcYjXRrSr109alQpvVH05FqVefW+q+g/ZiHdmtTgxXvTuKJCye+RfLoQj/GvPil4zBg5PYO8fMdj3RsTGlJ+Jh6oQBYREZHLSlREGP99WwvuvCqeP09axaD3l9I5KZZhNzenXkyln9ody81n3MJtvDxzI/sOHee65Fge75FMi/ioU/rzeIybUuLo3aI2n6Rv5/npGTz09mJaxkcxuEcSXZJiTymUDx/P48PFmYyet5nMH46SWC2Cp25uzp2t44moUDalV8fGMSwY2p3oK8JKZentEI/xzzta4jF4aeYGJi3bTv8O9bj76gQqX8Sy4r6SX+A4kX/pU2FUIIuIiPjKO++w9ttvaefvOMqJVolV+eTRDrzz3VZGfLmeniNnM6hLIwZ0qs/k5Tt48esMduYco12D6rx+fxpX1a12zv5CPMZtreK5qWUc/7tsOy/MyKD/W4tIS4zm8R7JJNWMZMz8Lby3YBs5R3NJS4zmz72b0qNZrVK9KLE41S5iSsW5eDzG8Dta0qNZLf49ZxNPT13L89MzuLdtIr9qX4+46CtK9XgXa9v+I8zdsI+5G/Yyf+N+rk8wel5inyqQRUREfCUhgeMbdQstXwoN8dC/Q316t6jNsClreG76el6amUFuviMtMZoRd6bQvlHMBfd5V+sEbk2tw8QlWbz0dQb3jV6Ax8ABvZrXYkCnBlxVt2rZvCkfMjN6NKtJj2Y1Sc/M5t9zNjF67mbenLuZX7SszYCODc4YcS9rBw6fYP7G/czdsJe5G/aR+cNRAGpHVaR705rU8+y75GOoQBYREfGV8eOJXb0aunTxdyTlTs0qFXm5bxp3t97Lx8u3c1PLOLokx17SRXIVQj30bZvIHVfVYcKiTHbkHOOeqxOoW73S+V8chFISonmpbxpZB44wZt4WPliUyeTlO2hbvxoPdWpA1yY1SmV6x+mO5eazeMsB5mzYy7wN+1i94yDOQeXwUK5pWJ0BHRvQsXEMDWIqYWbMmjXrko+pAllERMRXXn2VOtnZMGyYvyMptzonxdI5KbZU+wwPDeH+dvVKtc9AFl81gr/c2Izfdm/M+IWZvDVvMwPeXkyD2Er8sm1d2tSrRpPalQm7yIv6Cgoca3cdZE7GPuZm7GPhlh84kVdAWIjRKrEqg7sn0bFxDC3rRJXZhYMqkEVERETkglWpGMZDnRvwqw71+GzlTkbN2czfp6wBCkfXr4yrQkpCNKnen8RqEcWO2O/MOfpTQTxvwz72Hz4BQFLNSO5rW5dOjWNoU7/aJS9PXlIqkEVERETkooWFeLgltQ43p8SRdeAo6VnZpGdmszwzm3ELt/HWvC0AVI0IIyUhmpT4aFIToykocIVF8YZ9bNhzCICYyHA6NY6hY+NYOjaKKfVFZUpKBbKIiIiIXDIzI6FaBAnVIrixZRwAefkFrN99iOWZPxfN36zPwHnvxBYe6qFN/Wrc3TqBjo1jaFKr8iUvnlIaVCCLiIiISJkIDfHQLK4KzeKq0LdtIgCHjuexMisHhyMtsSoVwy5+UZOyogJZRETEVyZOZPW8eXTwdxwifhQZHkq7htX9HcY5lZ81A0VERPwtJobcKN/eM1ZELpwKZBEREV8ZM4Za06b5OwoROQ8VyCIiIr6iAlkkKKhAFhEREREpQgWyiIiIiEgRKpBFRERERIpQgSwiIiIiUoTugywiIuIrn33Gitmz6ezvOETknDSCLCIi4isRERRUrOjvKETkPFQgi4iI+MorrxD38cf+jkJEzkNTLERERHxlwgRqZGf7OwoROQ+NIIuIiIiIFFGmBbKZ9TKz781sg5n98SzPh5vZeO/zC8ysXlnGIyIiIiJyPmVWIJtZCPAycAPQDLjXzJqd1uxB4IBzrhHwHDC8rOIRERERESmJshxBbgNscM5tcs6dAD4AbjmtzS3AWO/jiUA3M7MyjElERERE5JzK8iK9OkBmke0soG1xbZxzeWaWA1QH9hVtZGYDgYHezeNmtqpMIpayEsNpOZWApnwFF+Ur+MRgppwFD/2NBZ8LyVnds+0sywL5bCPB7iLa4Jx7A3gDwMwWO+daX3p44ivKWXBRvoKL8hV8lLPgonwFn9LIWVlOscgCEopsxwM7imtjZqFAFPBDGcYkIiIiInJOZVkgLwIam1l9M6sA3AN8clqbT4B+3sd9gK+dc2eMIIuIiIiI+EqZTbHwzil+FPgCCAHedM6tNrNhwGLn3CfAaOAdM9tA4cjxPSXo+o2yilnKjHIWXJSv4KJ8BR/lLLgoX8HnknNmGrAVEREREfmZVtITERERESlCBbKIiIiISBFBVSCfb+lq8S8ze9PM9hS9T7WZVTOzr8wsw/u7qj9jlJ+ZWYKZzTSztWa22sx+592vnAUoM6toZgvNLN2bs6e8++ub2QJvzsZ7L4yWAGFmIWa2zMymeLeVrwBmZlvMbKWZLTezxd59Oi8GKDOLNrOJZrbO+/+sXWnkK2gK5BIuXS3+NQboddq+PwIznHONgRnebQkMecAQ51xT4BpgkPdvSjkLXMeBrs65FCAV6GVm1wDDgee8OTsAPOjHGOVMvwPWFtlWvgLfdc651CL30tV5MXA9D0xzzjUBUij8W7vkfAVNgUzJlq4WP3LOzebM+1gXXU58LHCrT4OSYjnndjrnlnof/0jhSaUOylnAcoUOeTfDvD8O6ApM9O5XzgKImcUDvwBGebcN5SsY6bwYgMysCtCZwrui4Zw74ZzLphTyFUwF8tmWrq7jp1ik5Go653ZCYUEG1PBzPHIWZlYPaAUsQDkLaN6v65cDe4CvgI1AtnMuz9tE58bAMhJ4AijwbldH+Qp0DvjSzJaY2UDvPp0XA1MDYC/wlnca0ygzq0Qp5CuYCuQSLUstIhfGzCKBj4DHnHMH/R2PnJtzLt85l0rh6qRtgKZna+bbqORszOxGYI9zbknR3WdpqnwFlg7OuTQKp3QOMrPO/g5IihUKpAGvOudaAYcppekvwVQgl2Tpagk8u82sNoD39x4/xyNFmFkYhcXxe865//XuVs6CgPdrxFkUzh+PNrOTCz/p3Bg4OgA3m9kWCqcFdqVwRFn5CmDOuR3e33uASRR+ENV5MTBlAVnOuQXe7YkUFsyXnK9gKpBLsnS1BJ6iy4n3Ayb7MRYpwjsXcjSw1jn3bJGnlLMAZWaxZhbtfXwF0J3CueMzgT7eZspZgHDO/ck5F++cq0fh/6yvnXO/RPkKWGZWycwqn3wM9ARWofNiQHLO7QIyzSzZu6sbsIZSyFdQraRnZr0p/PR9cunqf/g5JCnCzMYBXYAYYDfwN+BjYAKQCGwD7nTOnX4hn/iBmXUE5gAr+Xl+5FAK5yErZwHIzFpSeMFJCIUDHBOcc8PMrAGFI5TVgGXAfc654/6LVE5nZl2A3zvnblS+Apc3N5O8m6HA+865f5hZdXReDEhmlkrhRbAVgE1Af7znRy4hX0FVIIuIiIiIlLVgmmIhIiIiIlLmVCCLiIiIiBShAllEREREpAgVyCIiIiIiRahAFhEREREpQgWyiEg5YmZdzGyKv+MQEQlkKpBFRERERIpQgSwiEoDM7D4zW2hmy83sdTMLMbNDZjbCzJaa2Qwzi/W2TTWz78xshZlNMrOq3v2NzGy6maV7X9PQ232kmU00s3Vm9p53VUXM7BkzW+Pt5//76a2LiPidCmQRkQBjZk2Bu4EOzrlUIB/4JVAJWOqcSwO+oXC1SoC3gSedcy0pXBnx5P73gJedcylAe2Cnd38r4DGgGdAA6GBm1YDbgObefp4u23cpIhK4VCCLiASebsBVwCIzW+7dbkDhkuDjvW3eBTqaWRQQ7Zz7xrt/LNDZzCoDdZxzkwCcc8ecc0e8bRY657KccwXAcqAecBA4Bowys9uBk21FRModFcgiIoHHgLHOuVTvT7Jz7r/O0s6dp4/iHC/yOB8Idc7lAW2Aj4BbgWkXGLOIyGVDBbKISOCZAfQxsxoAZlbNzOpSeM7u423TF5jrnMsBDphZJ+/++4FvnHMHgSwzu9XbR7iZRRR3QDOLBKKcc59ROP0itSzemIhIMAj1dwAiInIq59waM/sL8KWZeYBcYBBwGGhuZkuAHArnKQP0A17zFsCbgP7e/fcDr5vZMG8fd57jsJWByWZWkcLR58Gl/LZERIKGOXeub+hERCRQmNkh51ykv+MQEbncaYqFiIiIiEgRGkEWERERESlCI8giIiIiIkWoQBYRERERKUIFsoiIiIhIESqQRURERESKUIEsIiIiIlLE/wEOThoy37oGQwAAAABJRU5ErkJggg==\n",
886 |       "text/plain": [
887 |        "<Figure size 720x576 with 1 Axes>"
888 |       ]
889 |      },
890 |      "metadata": {
891 |       "needs_background": "light"
892 |      },
893 |      "output_type": "display_data"
894 |     }
895 |    ],
896 |    "source": [
897 |     "for run in range(runs):\n",
898 |     "    # visualize the loss as the network trained\n",
899 |     "    fig = plt.figure(figsize=(10,8))\n",
900 |     "    t_loss = train_loss[run][np.where(train_loss[run] > 0)]\n",
901 |     "    v_loss = val_loss[run][np.where(val_loss[run] > 0)]\n",
902 |     "    \n",
903 |     "    plt.plot(range(1,len(t_loss)+1),t_loss, label='Training Loss')\n",
904 |     "    plt.plot(range(1,len(v_loss)+1),v_loss,label='Validation Loss')\n",
905 |     "\n",
906 |     "    # find position of lowest validation loss\n",
907 |     "    #print(np.where(v_loss == np.min(v_loss))[0][0])\n",
908 |     "    minposs = np.where(v_loss == np.min(v_loss))[0][0] + 1\n",
909 |     "    plt.axvline(minposs, linestyle='--', color='r',label='Early Stopping Checkpoint')\n",
910 |     "\n",
911 |     "    plt.xlabel('epochs')\n",
912 |     "    plt.ylabel('loss')\n",
913 |     "    plt.ylim(0, 1) # consistent scale\n",
914 |     "    plt.xlim(0, len(v_loss)+1) # consistent scale\n",
915 |     "    plt.grid(True)\n",
916 |     "    plt.legend()\n",
917 |     "    plt.tight_layout()\n",
918 |     "    plt.show()\n",
919 |     "    fig.savefig('{}/loss_plot_fold{}_run{}.png'.format(model_name, fold, run), bbox_inches='tight',dpi=400)"
920 |    ]
921 |   },
922 |   {
923 |    "cell_type": "code",
924 |    "execution_count": null,
925 |    "metadata": {},
926 |    "outputs": [],
927 |    "source": []
928 |   },
929 |   {
930 |    "cell_type": "code",
931 |    "execution_count": null,
932 |    "metadata": {},
933 |    "outputs": [],
934 |    "source": []
935 |   },
936 |   {
937 |    "cell_type": "code",
938 |    "execution_count": null,
939 |    "metadata": {},
940 |    "outputs": [],
941 |    "source": []
942 |   },
943 |   {
944 |    "cell_type": "code",
945 |    "execution_count": null,
946 |    "metadata": {},
947 |    "outputs": [],
948 |    "source": []
949 |   },
950 |   {
951 |    "cell_type": "code",
952 |    "execution_count": null,
953 |    "metadata": {},
954 |    "outputs": [],
955 |    "source": []
956 |   },
957 |   {
958 |    "cell_type": "code",
959 |    "execution_count": null,
960 |    "metadata": {},
961 |    "outputs": [],
962 |    "source": []
963 |   },
964 |   {
965 |    "cell_type": "code",
966 |    "execution_count": null,
967 |    "metadata": {},
968 |    "outputs": [],
969 |    "source": []
970 |   }
971 |  ],
972 |  "metadata": {
973 |   "kernelspec": {
974 |    "display_name": "CellStar",
975 |    "language": "python",
976 |    "name": "cellstar"
977 |   },
978 |   "language_info": {
979 |    "codemirror_mode": {
980 |     "name": "ipython",
981 |     "version": 3
982 |    },
983 |    "file_extension": ".py",
984 |    "mimetype": "text/x-python",
985 |    "name": "python",
986 |    "nbconvert_exporter": "python",
987 |    "pygments_lexer": "ipython3",
988 |    "version": "3.7.6"
989 |   }
990 |  },
991 |  "nbformat": 4,
992 |  "nbformat_minor": 4
993 | }
994 | 


--------------------------------------------------------------------------------