├── .gitignore ├── CONTRIBUTING.md ├── LICENSE.md ├── README.md ├── book.jpg └── code ├── Chapter-10 ├── README.md └── Testing_and_Remediating_Bias_constrained.ipynb ├── Chapter-11 ├── Backdoor_testing.ipynb ├── Data_Poisoning.ipynb ├── README.md ├── Red_Teaming_an_XGBoost_model.ipynb └── Training_an_Overfit_and_a_Constrained_XGBoost_model.ipynb ├── Chapter-6 ├── Constrained_XGB_and_Post_Hoc_Explanations.ipynb ├── GLM,GAM_and_EBM_code_example.ipynb └── README.md ├── Chapter-7 & 9 ├── 1.Data Preparation.ipynb ├── 2.Transfer learning-Stage_1.ipynb ├── 3.Transfer learning-Stage_2.ipynb ├── 4.Post-Hoc Explanations.ipynb ├── 5.Adding Noise to images .ipynb ├── 6.Label_Randomization.ipynb ├── Adversarial Example Attacks.ipynb ├── README.md └── Retraining on Gaussian Noise.ipynb ├── Chapter-8 ├── README.md ├── Residual_Analysis_for_XGBoost.ipynb ├── Selecting a Better XGBoost Model.ipynb ├── Sensitivity_Analysis_for_XGBoost_Adversarial_Example_Search.ipynb └── Stress_testing_XGBoost.ipynb └── Data.zip /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | *.ipynb 3 | *.jupyterlab-workspace 4 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Desktop (please complete the following information):** 27 | - OS: [e.g. iOS] 28 | - Browser [e.g. chrome, safari] 29 | - Version [e.g. 22] 30 | 31 | **Additional context** 32 | Add any other context about the problem here. 33 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 Machine Learning for High-Risk Applications Book 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Machine-Learning-for-High-Risk-Applications-Book 2 | 3 | This is a companion repository for the book [Machine Learning for High-Risk Applications](https://learning.oreilly.com/library/view/machine-learning-for/9781098102425/) 4 | 5 | 6 | 7 | ![Book Cover](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/book.jpg) 8 | 9 | Buy on Amazon | 10 | Read on O'Reilly 11 | 12 | 13 | The past decade has witnessed the broad adoption of artificial intelligence and machine learning (AI/ML) technologies. However, a lack of oversight in their widespread implementation has resulted in some incidents and harmful outcomes that could have been avoided with proper risk management. Before we can realize AI/ML's true benefit, practitioners must understand how to mitigate its risks. 14 | 15 | This book describes approaches to responsible AI—a holistic framework for improving AI/ML technology, business processes, and cultural competencies that builds on best practices in risk management, cybersecurity, data privacy, and applied social science. Authors Patrick Hall, James Curtis, and Parul Pandey created this guide for data scientists who want to improve real-world AI/ML system outcomes for organizations, consumers, and the public. 16 | 17 | Learn technical approaches for responsible AI across explainability, model validation and debugging, bias management, data privacy, and ML security 18 | Learn how to create a successful and impactful AI risk management practice 19 | Get a basic guide to existing standards, laws, and assessments for adopting AI technologies, including the new NIST AI Risk Management Framework 20 | Engage with interactive resources on GitHub and Colab 21 | 22 | ## Code 23 | 24 | The code for this book can be found in the following sections: 25 | 26 | 27 | | Chapter | Code Notebooks | 28 | | ------- | -------------- | 29 | | 6 | [Explainable Boosting Machines and Explaining XGBoost](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/tree/main/code/Chapter-6) | 30 | | 7 | [Explaining a PyTorch Image Classifier](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/tree/main/code/Chapter-7%20%26%209) | 31 | | 8 | [Debugging XGBoost](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/tree/main/code/Chapter-8) | 32 | | 9 | [Debugging a PyTorch Image Classifier](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/tree/main/code/Chapter-7%20%26%209) | 33 | | 10 | [Testing and Remediating Bias with XGBoost](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/tree/main/code/Chapter-10) | 34 | | 11 | [Red-teaming XGBoost](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/tree/main/code/Chapter-11) | 35 | 36 | 37 | 38 | 39 | 40 | 41 | -------------------------------------------------------------------------------- /book.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/0fbf704b8b2d84579ce9473cdecabf46747a7ac2/book.jpg -------------------------------------------------------------------------------- /code/Chapter-10/README.md: -------------------------------------------------------------------------------- 1 | # Code for Chapter 10 - Testing and Remediating Bias with XGBoost 2 | 3 | This chapter focuses on technical implementations of bias testing and remediation approaches. We’ll start off by training XGBoost on a variant of the credit card data. We’ll then test for bias by checking for differences in performance and outcomes across demographic groups. We’ll additionally try and identify any bias concerns at the individual observation level. Once we confirm the existence of measurable levels of bias in our model predictions, we’ll start trying to fix, or remediate, that bias. We employ pre-, in- and post-processing remediation methods that attempt to fix the training data, model, and outcomes, respectively. We’ll finish off the chapter by conducting bias-aware model selection that leaves us with a model that is both performant and minimally biased. 4 | 5 | # Code 6 | 7 | 1. Testing and Remediating Bias in an XGBoost Credit Model [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-10/Testing_and_Remediating_Bias_constrained.ipynb) [![Open In GitHub](https://img.shields.io/badge/Github-code-green)](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-10/Testing_and_Remediating_Bias_constrained.ipynb) 8 | -------------------------------------------------------------------------------- /code/Chapter-11/README.md: -------------------------------------------------------------------------------- 1 | # Code for Chapter 11 - Red-teaming XGBoost 2 | 3 | In this chapter, we’ll show you how to hack your own models so that you can add red-teaming into to your model debugging repertoire. The main idea of the chapter is when you know what 4 | hackers will try to do to your model, then you can try it out first and devise effective defenses. We’ll start out with a concept refresher that reintroduces common ML 5 | attacks and countermeasures, then we’ll dive into examples of attacking an XGBoost classifier trained on structured data. We’ll then introduce two XGBoost models, one trained with the 6 | standard black-box approach, and one trained with constraints and a high degree of L2 regularization. We’ll use these two models to explain the attacks and to test 7 | whether transparency and L2 regularization are adequate countermeasures. After that, we’ll jump into attacks that are likely to be performed by external adversaries 8 | against a black-box ML API: model extraction and adversarial example attacks. From there, we’ll try out insider attacks that involve making deliberate changes to an ML 9 | modeling pipeline: data poisoning and model backdoors. 10 | 11 | # Code 12 | 13 | 1. Training an Overfit and a Constrained XGBoost model [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-11/Training_an_Overfit_and_a_Constrained_XGBoost_model.ipynb) [![Open In GitHub](https://img.shields.io/badge/Github-code-green)](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-11/Training_an_Overfit_and_a_Constrained_XGBoost_model.ipynb) 14 | 2. Red-Teaming an XGBoost Model [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-11/Data_Poisoning.ipynb) [![Open In GitHub](https://img.shields.io/badge/Github-code-green)](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-11/Red_Teaming_an_XGBoost_model.ipynb) 15 | 3. Data Poisoning [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1tMxexHbgNoUaeTS179bXUA7BbvkbcSOe?usp=sharing) [![Open In GitHub](https://img.shields.io/badge/Github-code-green)](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-11/Data_Poisoning.ipynb) 16 | 4. Backdoor Testing [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-11/Backdoor_testing.ipynb) [![Open In GitHub](https://img.shields.io/badge/Github-code-green)](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-11/Backdoor_testing.ipynb) 17 | -------------------------------------------------------------------------------- /code/Chapter-6/README.md: -------------------------------------------------------------------------------- 1 | 2 | # Code for Chapter 6 - Explainable Boosting Machines and Explaining XGBoost 3 | 4 | This chapter explores interpretable models and post-hoc explanation with examples relating to consumer finance. It also applies the approaches discussed in Chapter 2 using explainable boosting machines (EBMs), monotonically constrained XGBoost models, and post-hoc explanation techniques. 5 | 6 | # Code 7 | 1. GLM, GAM, and EBM code example Open In Colab [![Open In GitHub](https://img.shields.io/badge/Github-code-green)](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-6/Constrained_XGB_and_Post_Hoc_Explanations.ipynb) 8 | 2. Constrained_XGB and PostHoc Explanations Open In Colab [![Open In GitHub](https://img.shields.io/badge/Github-code-green)](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-6/Constrained_XGB_and_Post_Hoc_Explanations.ipynb) 9 | 10 | -------------------------------------------------------------------------------- /code/Chapter-7 & 9/1.Data Preparation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "cd5db84d", 6 | "metadata": { 7 | "id": "cd5db84d" 8 | }, 9 | "source": [ 10 | "# Chapter 7: Explaining a PyTorch Image Classifier\n", 11 | "\n", 12 | "## **Data Preparation**" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "id": "91f6f7c2", 18 | "metadata": { 19 | "id": "91f6f7c2" 20 | }, 21 | "source": [ 22 | "## 1. Setting the environment\n", 23 | "\n", 24 | "If you are using Colab, it comes preinstalled with PyTorch and other commonly used libraries for machine and Deep learning. However if you are executing this notebook in your local system, you will need to install them manually via the following commands:\n", 25 | "\n" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 6, 31 | "id": "2467c0e9", 32 | "metadata": { 33 | "collapsed": true, 34 | "jupyter": { 35 | "outputs_hidden": true 36 | }, 37 | "tags": [] 38 | }, 39 | "outputs": [ 40 | { 41 | "name": "stdout", 42 | "output_type": "stream", 43 | "text": [ 44 | "\n", 45 | "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.2.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m22.3.1\u001b[0m\n", 46 | "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython3.10 -m pip install --upgrade pip\u001b[0m\n" 47 | ] 48 | } 49 | ], 50 | "source": [ 51 | "!pip3 install torch torchvision numpy pandas matplotlib seaborn scikit-learn --quiet" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": 7, 57 | "id": "7b52ade8", 58 | "metadata": { 59 | "id": "7b52ade8" 60 | }, 61 | "outputs": [], 62 | "source": [ 63 | "# Importing the necessary libraries \n", 64 | "import os\n", 65 | "import numpy as np\n", 66 | "import pandas as pd\n", 67 | "import seaborn as sns\n", 68 | "import matplotlib.pyplot as plt\n", 69 | "\n", 70 | "from sklearn.model_selection import train_test_split\n", 71 | "\n", 72 | "import torch\n", 73 | "import torchvision\n", 74 | "import torch.nn as nn\n", 75 | "from torchvision import datasets, transforms\n", 76 | "from torchvision.datasets import ImageFolder\n", 77 | "from torch.utils.data import Dataset, DataLoader, ConcatDataset\n", 78 | "\n", 79 | "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "id": "874f772c", 85 | "metadata": { 86 | "id": "874f772c" 87 | }, 88 | "source": [ 89 | "## 2. Loading the dataset\n", 90 | "\n", 91 | "We'll create an image classifier to diagnose pneumonia in Chest X-Ray images The dataset we’ll use for training has been taken from [Kaggle](https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia), and it consists of 5,863 X-Ray images of patients, which have been split into two distinct categories — one containing pneumonia and the other being normal. In this ection, we'll have a look at the dataset and see if there are any issues with it.\n", 92 | "\n", 93 | "> Download the [zipped dataset from Kaggle](https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia) and save it as `chest_xray_original.zip` on your local systems" 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": 14, 99 | "id": "18adeea3", 100 | "metadata": { 101 | "id": "18adeea3" 102 | }, 103 | "outputs": [], 104 | "source": [ 105 | "import zipfile\n", 106 | "with zipfile.ZipFile(\"chest_xray_original.zip\",\"r\") as zip_ref:\n", 107 | " zip_ref.extractall(\"chest_xray_original\")" 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": 21, 113 | "id": "1e1c3ad1", 114 | "metadata": { 115 | "id": "1e1c3ad1" 116 | }, 117 | "outputs": [], 118 | "source": [ 119 | "\n", 120 | "# Assigning PATH to original dataset\n", 121 | "PATH_original = 'chest_xray_original/chest_xray_original'\n" 122 | ] 123 | }, 124 | { 125 | "cell_type": "code", 126 | "execution_count": 22, 127 | "id": "8df3f9bf", 128 | "metadata": { 129 | "id": "8df3f9bf" 130 | }, 131 | "outputs": [], 132 | "source": [ 133 | "def distribution(data_set):\n", 134 | " #To calculate distribution of the datasets\n", 135 | " \n", 136 | " normal_path = os.path.join(PATH_original+f\"/{data_set}/NORMAL\")\n", 137 | " pneumonia_path = os.path.join(PATH_original+f\"/{data_set}/PNEUMONIA\")\n", 138 | " \n", 139 | " normal = len([filename for filename in os.listdir(normal_path)])\n", 140 | " pneumonia = len([filename for filename in os.listdir(pneumonia_path)])\n", 141 | "\n", 142 | " distribution = dict(zip(['Normal','Pneumonia'],[normal,pneumonia]))\n", 143 | " sns.barplot(x=list(distribution.keys()), y=list(distribution.values())).set_title(f\"{data_set} Data Imbalance\")\n", 144 | " return distribution" 145 | ] 146 | }, 147 | { 148 | "cell_type": "code", 149 | "execution_count": 23, 150 | "id": "dd70b780", 151 | "metadata": { 152 | "id": "dd70b780", 153 | "outputId": "e28e2fa7-367c-4885-a2ac-118d2c2be228" 154 | }, 155 | "outputs": [ 156 | { 157 | "data": { 158 | "text/plain": [ 159 | "{'Normal': 1342, 'Pneumonia': 3876}" 160 | ] 161 | }, 162 | "execution_count": 23, 163 | "metadata": {}, 164 | "output_type": "execute_result" 165 | }, 166 | { 167 | "data": { 168 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAGzCAYAAAAxPS2EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA43klEQVR4nO3de1xVdb7/8TegbC66UbxwSUDLUlHU0hndlaZJbg2dLtbk6IimZTropOblMMfQcCZMzUtqWqdJnI6OaRdPQWp4n5TUmEiysjQKSzdYBtsrKKzfH/1Y4040UQiWvZ6Px3o82Gt91nd9vhthv117rY2XYRiGAAAALMS7phsAAACoLAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMYGHNmzfXsGHDarqNa86wYcNUr169Kh2T7xVQtQgwQDXauXOnpk+frsLCwppu5bJMnz5dXl5e5hIQEKDIyEj1799fy5YtU3Fx8RWP/c4772j69OlV1+z/Vx1hA0DtR4ABqtHOnTv11FNPVVuA2b9/v/7nf/6nysddsmSJXnnlFS1cuFCPPPKIjh07puHDh+u3v/2tDh06dEVjvvPOO3rqqaequFMAv1Z1aroBAD8qKytTSUmJ/Pz8Lnsfm81WLb088MADaty4sfk4KSlJK1asUHx8vB588EG9//771XJcALhcnIEBqsn06dM1adIkSVKLFi3Mt2W++uorSZKXl5fGjBmjFStWqG3btrLZbFq/fr0kac6cObr11lvVqFEj+fv7q1OnTnrttdcuOMZPr6tITU2Vl5eXduzYoQkTJqhJkyYKDAzUfffdp6NHj17VfAYPHqxHHnlEu3btUkZGhrn+X//6lx588EFFRkbKZrMpIiJC48eP1+nTp82aYcOGafHixea8y5dylzvfy9W8eXP169dPW7duVefOneXv76+YmBht3bpVkvTGG28oJiZGfn5+6tSpkz788MMKx/nyyy/ldDoVGBio8PBwJScnyzAMj5or7f3YsWOaOHGiYmJiVK9ePdntdvXt21cfffSRR93WrVvl5eWl1atX629/+5uaNWsmPz8/9erVSwcOHLhg3F27dunuu+9Ww4YNFRgYqPbt22vBggUeNZ999pkeeOABBQcHy8/PT507d9Zbb731sz0DtQlnYIBqcv/99+vzzz/XP//5T82bN888o9GkSROzZvPmzVq9erXGjBmjxo0bq3nz5pKkBQsW6He/+50GDx6skpISrVq1Sg8++KDS0tIUFxf3s8ceO3asGjZsqGnTpumrr77S/PnzNWbMGL366qtXNachQ4boxRdf1Lvvvqu77rpLkrRmzRqdOnVKo0ePVqNGjbR7924tXLhQ33zzjdasWSNJeuyxx3T48GFlZGTolVdeuWDcq51vRQ4cOKBBgwbpscce0x//+EfNmTNH/fv319KlS/WXv/xFf/rTnyRJKSkp+v3vf6/9+/fL2/s//6crLS1Vnz591LVrV82aNUvr16/XtGnTdO7cOSUnJ191719++aXWrl2rBx98UC1atFB+fr5eeOEF3XHHHfrkk08UHh7uUT9z5kx5e3tr4sSJKioq0qxZszR48GDt2rXLrMnIyFC/fv0UFhamxx9/XKGhofr000+Vlpamxx9/XJK0b98+3Xbbbbruuuv0X//1XwoMDNTq1at177336vXXX9d99913Rc838IszAFSb2bNnG5KM3NzcC7ZJMry9vY19+/ZdsO3UqVMej0tKSox27doZd955p8f6qKgoY+jQoebjZcuWGZKM2NhYo6yszFw/fvx4w8fHxygsLLxkv9OmTTMkGUePHq1w+w8//GBIMu67776L9moYhpGSkmJ4eXkZX3/9tbkuISHBuNivnMudb0WGDh1qBAYGeqyLiooyJBk7d+40123YsMGQZPj7+3v09cILLxiSjC1btniMKckYO3asua6srMyIi4szfH19PZ6fK/1enTlzxigtLfWoyc3NNWw2m5GcnGyu27JliyHJaNOmjVFcXGyuX7BggSHJyMnJMQzDMM6dO2e0aNHCiIqKMn744QePcc//t9CrVy8jJibGOHPmjMf2W2+91bjxxhsNwCp4CwmoQXfccYeio6MvWO/v729+/cMPP6ioqEjdunXTv//978sad+TIkR5v0XTr1k2lpaX6+uuvr6rf8rt9jh8/XmGvJ0+e1Hfffadbb71VhmFc9K2Zn7ra+VYkOjpaDofDfNylSxdJ0p133qnIyMgL1n/55ZcXjDFmzBjz6/K3/EpKSrRx48ar7t1ms5lnfEpLS/X999+rXr16atWqVYX7Pvzww/L19TUfd+vWzaPvDz/8ULm5uRo3bpwaNGjgsW/5v4Vjx45p8+bN+v3vf6/jx4/ru+++03fffafvv/9eTqdTX3zxhb799ttL9g3UFryFBNSgFi1aVLg+LS1Nf/3rX5Wdne1x6/L5oeRSzn+BlqSGDRtK+vEF9mqcOHFCklS/fn1zXV5enpKSkvTWW29dMH5RUdFljXu1863IT5+DoKAgSVJERESF63/au7e3t66//nqPdTfddJMkmdcxXU3vZWVlWrBggZ5//nnl5uaqtLTU3NaoUaOfnc9Pv6cHDx6UJLVr1+6ixzxw4IAMw9CTTz6pJ598ssKagoICXXfddZfsHagNCDBADTr/f+/l/vWvf+l3v/udunfvrueff15hYWGqW7euli1bppUrV17WuD4+PhWuN35yAWplffzxx5Kkli1bSvrxzMFdd92lY8eOacqUKWrdurUCAwP17bffatiwYSorK/vZMativhW52HNQlc/N1fT+9NNP68knn9Tw4cM1Y8YMBQcHy9vbW+PGjavweauKvsvHnThxopxOZ4U15d9boLYjwADV6ErOILz++uvy8/PThg0bPG6TXrZsWVW2dkXKL8Atf/HLycnR559/ruXLlys+Pt6sO/8upXIXey5q63zLysr05ZdfmmddJOnzzz+XJPNi66vp/bXXXlPPnj3197//3WN9YWGhxy3sl+uGG26Q9GPIjI2NrbCm/IxS3bp1L1oDWAXXwADVKDAwUJIq9UF2Pj4+8vLy8nhL4auvvtLatWuruLvKWblypV566SU5HA716tVL0n/OCpx/FsAwjAtu25Uu/lzU1vlK0qJFi8yvDcPQokWLVLduXY/5X2nvPj4+F5w9WbNmzRVfg3LLLbeoRYsWmj9//gXPcflxmjZtqh49euiFF17QkSNHLhjjam+1B35JnIEBqlGnTp0kSf/93/+tgQMHqm7duurfv7/5Yl6RuLg4zZ07V3369NGgQYNUUFCgxYsXq2XLltq7d+8v0vdrr72mevXqqaSkRN9++602bNigHTt2qEOHDuat0ZLUunVr3XDDDZo4caK+/fZb2e12vf766xVea1P+XPz5z3+W0+mUj4+PBg4cWCvmWxE/Pz+tX79eQ4cOVZcuXbRu3Tqlp6frL3/5i3kr/NX03q9fPyUnJ+vhhx/WrbfeqpycHK1YseKC624ul7e3t5YsWaL+/furY8eOevjhhxUWFqbPPvtM+/bt04YNGyRJixcv1u23366YmBg9+uijuv7665Wfn6/MzEx98803F3wODVBr1dTtT8CvxYwZM4zrrrvO8Pb29rilWpKRkJBQ4T5///vfjRtvvNGw2WxG69atjWXLlpm3OJ/vYrdR79mzx6Ou/Fbc828Vrkj5McoXPz8/o1mzZka/fv2Ml19+2ePW23KffPKJERsba9SrV89o3Lix8eijjxofffSRIclYtmyZWXfu3Dlj7NixRpMmTQwvLy+PuVzufCtysduo4+LiLqit6DnPzc01JBmzZ8++YMyDBw8avXv3NgICAoyQkBBj2rRpF9z6fKXfqzNnzhhPPPGEERYWZvj7+xu33XabkZmZadxxxx3GHXfcYdaVf+/WrFlTYd/nP8eGYRjvvfeecddddxn169c3AgMDjfbt2xsLFy70qDl48KARHx9vhIaGGnXr1jWuu+46o1+/fsZrr712wXMG1FZehnGVV/UBAAD8wrgGBgAAWA4BBgAAWA4BBgAAWA4BBgAAWA4BBgAAWA4BBgAAWM41+0F2ZWVlOnz4sOrXr39VfxAOAAD8cgzD0PHjxxUeHm7+xfaKXLMB5vDhwxf81VkAAGANhw4dUrNmzS66/ZoNMPXr15f04xNgt9truBsAAHA53G63IiIizNfxi7qaj/FNSUkxJBmPP/64ue706dPGn/70JyM4ONgIDAw07r//fsPlcnns9/XXXxt333234e/vbzRp0sSYOHGicfbsWY+aLVu2GDfffLPh6+tr3HDDDRd8XPbPKSoqMiQZRUVFVzo9AADwC7vc1+8rvoh3z549euGFF9S+fXuP9ePHj9fbb7+tNWvWaNu2bTp8+LDuv/9+c3tpaani4uJUUlKinTt3avny5UpNTVVSUpJZk5ubq7i4OPXs2VPZ2dkaN26cHnnkEfOPkQEAgF+5K0lHx48fN2688UYjIyPDuOOOO8wzMIWFhUbdunU9/ujYp59+akgyMjMzDcMwjHfeecfw9vb2OCuzZMkSw263G8XFxYZhGMbkyZONtm3behzzoYceMpxO52X3yBkYAACsp1rPwCQkJCguLk6xsbEe67OysnT27FmP9a1bt1ZkZKQyMzMlSZmZmYqJiVFISIhZ43Q65Xa7tW/fPrPmp2M7nU5zjIoUFxfL7XZ7LAAA4NpU6Yt4V61apX//+9/as2fPBdtcLpd8fX3VoEEDj/UhISFyuVxmzfnhpXx7+bZL1bjdbp0+fVr+/v4XHDslJUVPPfVUZacDAAAsqFJnYA4dOqTHH39cK1askJ+fX3X1dEUSExNVVFRkLocOHarplgAAQDWpVIDJyspSQUGBbrnlFtWpU0d16tTRtm3b9Nxzz6lOnToKCQlRSUmJCgsLPfbLz89XaGioJCk0NFT5+fkXbC/fdqkau91e4dkXSbLZbLLb7R4LAAC4NlUqwPTq1Us5OTnKzs42l86dO2vw4MHm13Xr1tWmTZvMffbv36+8vDw5HA5JksPhUE5OjgoKCsyajIwM2e12RUdHmzXnj1FeUz4GAAD4davUNTD169dXu3btPNYFBgaqUaNG5voRI0ZowoQJCg4Olt1u19ixY+VwONS1a1dJUu/evRUdHa0hQ4Zo1qxZcrlcmjp1qhISEmSz2SRJo0aN0qJFizR58mQNHz5cmzdv1urVq5Wenl4VcwYAABZX5Z/EO2/ePHl7e2vAgAEqLi6W0+nU888/b2738fFRWlqaRo8eLYfDocDAQA0dOlTJyclmTYsWLZSenq7x48drwYIFatasmV566SU5nc6qbhcAAFiQl2EYRk03UR3cbreCgoJUVFTE9TAAAFjE5b5+X/En8QIAANQUAgwAALAcAgwAALAcAgwAALCcKr8LCQCuFXnJMTXdAlDrRCbl1HQLkjgDAwAALIgAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALIcAAwAALKdSAWbJkiVq37697Ha77Ha7HA6H1q1bZ27v0aOHvLy8PJZRo0Z5jJGXl6e4uDgFBASoadOmmjRpks6dO+dRs3XrVt1yyy2y2Wxq2bKlUlNTr3yGAADgmlOnMsXNmjXTzJkzdeONN8owDC1fvlz33HOPPvzwQ7Vt21aS9Oijjyo5OdncJyAgwPy6tLRUcXFxCg0N1c6dO3XkyBHFx8erbt26evrppyVJubm5iouL06hRo7RixQpt2rRJjzzyiMLCwuR0OqtizgAAwOK8DMMwrmaA4OBgzZ49WyNGjFCPHj3UsWNHzZ8/v8LadevWqV+/fjp8+LBCQkIkSUuXLtWUKVN09OhR+fr6asqUKUpPT9fHH39s7jdw4EAVFhZq/fr1F+2juLhYxcXF5mO3262IiAgVFRXJbrdfzRQB/ErlJcfUdAtArROZlFOt47vdbgUFBf3s6/cVXwNTWlqqVatW6eTJk3I4HOb6FStWqHHjxmrXrp0SExN16tQpc1tmZqZiYmLM8CJJTqdTbrdb+/btM2tiY2M9juV0OpWZmXnJflJSUhQUFGQuERERVzo1AABQy1XqLSRJysnJkcPh0JkzZ1SvXj29+eabio6OliQNGjRIUVFRCg8P1969ezVlyhTt379fb7zxhiTJ5XJ5hBdJ5mOXy3XJGrfbrdOnT8vf37/CvhITEzVhwgTzcfkZGAAAcO2pdIBp1aqVsrOzVVRUpNdee01Dhw7Vtm3bFB0drZEjR5p1MTExCgsLU69evXTw4EHdcMMNVdr4T9lsNtlstmo9BgAAqB0q/RaSr6+vWrZsqU6dOiklJUUdOnTQggULKqzt0qWLJOnAgQOSpNDQUOXn53vUlD8ODQ29ZI3dbr/o2RcAAPDrctWfA1NWVuZx8ez5srOzJUlhYWGSJIfDoZycHBUUFJg1GRkZstvt5ttQDodDmzZt8hgnIyPD4zobAADw61apt5ASExPVt29fRUZG6vjx41q5cqW2bt2qDRs26ODBg1q5cqXuvvtuNWrUSHv37tX48ePVvXt3tW/fXpLUu3dvRUdHa8iQIZo1a5ZcLpemTp2qhIQE8+2fUaNGadGiRZo8ebKGDx+uzZs3a/Xq1UpPT6/62QMAAEuqVIApKChQfHy8jhw5oqCgILVv314bNmzQXXfdpUOHDmnjxo2aP3++Tp48qYiICA0YMEBTp0419/fx8VFaWppGjx4th8OhwMBADR061ONzY1q0aKH09HSNHz9eCxYsULNmzfTSSy/xGTAAAMB01Z8DU1td7n3kAHAxfA4McCHLfw4MAABATSHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAyyHAAAAAy6lUgFmyZInat28vu90uu90uh8OhdevWmdvPnDmjhIQENWrUSPXq1dOAAQOUn5/vMUZeXp7i4uIUEBCgpk2batKkSTp37pxHzdatW3XLLbfIZrOpZcuWSk1NvfIZAgCAa06lAkyzZs00c+ZMZWVl6YMPPtCdd96pe+65R/v27ZMkjR8/Xm+//bbWrFmjbdu26fDhw7r//vvN/UtLSxUXF6eSkhLt3LlTy5cvV2pqqpKSksya3NxcxcXFqWfPnsrOzta4ceP0yCOPaMOGDVU0ZQAAYHVehmEYVzNAcHCwZs+erQceeEBNmjTRypUr9cADD0iSPvvsM7Vp00aZmZnq2rWr1q1bp379+unw4cMKCQmRJC1dulRTpkzR0aNH5evrqylTpig9PV0ff/yxeYyBAweqsLBQ69evv+y+3G63goKCVFRUJLvdfjVTBPArlZccU9MtALVOZFJOtY5/ua/fV3wNTGlpqVatWqWTJ0/K4XAoKytLZ8+eVWxsrFnTunVrRUZGKjMzU5KUmZmpmJgYM7xIktPplNvtNs/iZGZmeoxRXlM+xsUUFxfL7XZ7LAAA4NpU6QCTk5OjevXqyWazadSoUXrzzTcVHR0tl8slX19fNWjQwKM+JCRELpdLkuRyuTzCS/n28m2XqnG73Tp9+vRF+0pJSVFQUJC5REREVHZqAADAIiodYFq1aqXs7Gzt2rVLo0eP1tChQ/XJJ59UR2+VkpiYqKKiInM5dOhQTbcEAACqSZ3K7uDr66uWLVtKkjp16qQ9e/ZowYIFeuihh1RSUqLCwkKPszD5+fkKDQ2VJIWGhmr37t0e45XfpXR+zU/vXMrPz5fdbpe/v/9F+7LZbLLZbJWdDgAAsKCr/hyYsrIyFRcXq1OnTqpbt642bdpkbtu/f7/y8vLkcDgkSQ6HQzk5OSooKDBrMjIyZLfbFR0dbdacP0Z5TfkYAAAAlToDk5iYqL59+yoyMlLHjx/XypUrtXXrVm3YsEFBQUEaMWKEJkyYoODgYNntdo0dO1YOh0Ndu3aVJPXu3VvR0dEaMmSIZs2aJZfLpalTpyohIcE8ezJq1CgtWrRIkydP1vDhw7V582atXr1a6enpVT97AABgSZUKMAUFBYqPj9eRI0cUFBSk9u3ba8OGDbrrrrskSfPmzZO3t7cGDBig4uJiOZ1OPf/88+b+Pj4+SktL0+jRo+VwOBQYGKihQ4cqOTnZrGnRooXS09M1fvx4LViwQM2aNdNLL70kp9NZRVMGAABWd9WfA1Nb8TkwAK4WnwMDXMjynwMDAABQUwgwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcggwAADAcioVYFJSUvSb3/xG9evXV9OmTXXvvfdq//79HjU9evSQl5eXxzJq1CiPmry8PMXFxSkgIEBNmzbVpEmTdO7cOY+arVu36pZbbpHNZlPLli2Vmpp6ZTMEAADXnEoFmG3btikhIUHvv/++MjIydPbsWfXu3VsnT570qHv00Ud15MgRc5k1a5a5rbS0VHFxcSopKdHOnTu1fPlypaamKikpyazJzc1VXFycevbsqezsbI0bN06PPPKINmzYcJXTBQAA14I6lSlev369x+PU1FQ1bdpUWVlZ6t69u7k+ICBAoaGhFY7x7rvv6pNPPtHGjRsVEhKijh07asaMGZoyZYqmT58uX19fLV26VC1atNCzzz4rSWrTpo3ee+89zZs3T06ns7JzBAAA15irugamqKhIkhQcHOyxfsWKFWrcuLHatWunxMREnTp1ytyWmZmpmJgYhYSEmOucTqfcbrf27dtn1sTGxnqM6XQ6lZmZedFeiouL5Xa7PRYAAHBtqtQZmPOVlZVp3Lhxuu2229SuXTtz/aBBgxQVFaXw8HDt3btXU6ZM0f79+/XGG29Iklwul0d4kWQ+drlcl6xxu906ffq0/P39L+gnJSVFTz311JVOBwAAWMgVB5iEhAR9/PHHeu+99zzWjxw50vw6JiZGYWFh6tWrlw4ePKgbbrjhyjv9GYmJiZowYYL52O12KyIiotqOBwAAas4VvYU0ZswYpaWlacuWLWrWrNkla7t06SJJOnDggCQpNDRU+fn5HjXlj8uvm7lYjd1ur/DsiyTZbDbZ7XaPBQAAXJsqFWAMw9CYMWP05ptvavPmzWrRosXP7pOdnS1JCgsLkyQ5HA7l5OSooKDArMnIyJDdbld0dLRZs2nTJo9xMjIy5HA4KtMuAAC4RlUqwCQkJOh///d/tXLlStWvX18ul0sul0unT5+WJB08eFAzZsxQVlaWvvrqK7311luKj49X9+7d1b59e0lS7969FR0drSFDhuijjz7Shg0bNHXqVCUkJMhms0mSRo0apS+//FKTJ0/WZ599pueff16rV6/W+PHjq3j6AADAiioVYJYsWaKioiL16NFDYWFh5vLqq69Kknx9fbVx40b17t1brVu31hNPPKEBAwbo7bffNsfw8fFRWlqafHx85HA49Mc//lHx8fFKTk42a1q0aKH09HRlZGSoQ4cOevbZZ/XSSy9xCzUAAJAkeRmGYdR0E9XB7XYrKChIRUVFXA8D4IrkJcfUdAtArROZlFOt41/u6zd/CwkAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFhOpQJMSkqKfvOb36h+/fpq2rSp7r33Xu3fv9+j5syZM0pISFCjRo1Ur149DRgwQPn5+R41eXl5iouLU0BAgJo2bapJkybp3LlzHjVbt27VLbfcIpvNppYtWyo1NfXKZggAAK45lQow27ZtU0JCgt5//31lZGTo7Nmz6t27t06ePGnWjB8/Xm+//bbWrFmjbdu26fDhw7r//vvN7aWlpYqLi1NJSYl27typ5cuXKzU1VUlJSWZNbm6u4uLi1LNnT2VnZ2vcuHF65JFHtGHDhiqYMgAAsDovwzCMK9356NGjatq0qbZt26bu3burqKhITZo00cqVK/XAAw9Ikj777DO1adNGmZmZ6tq1q9atW6d+/frp8OHDCgkJkSQtXbpUU6ZM0dGjR+Xr66spU6YoPT1dH3/8sXmsgQMHqrCwUOvXr7+s3txut4KCglRUVCS73X6lUwTwK5aXHFPTLQC1TmRSTrWOf7mv31d1DUxRUZEkKTg4WJKUlZWls2fPKjY21qxp3bq1IiMjlZmZKUnKzMxUTEyMGV4kyel0yu12a9++fWbN+WOU15SPUZHi4mK53W6PBQAAXJuuOMCUlZVp3Lhxuu2229SuXTtJksvlkq+vrxo0aOBRGxISIpfLZdacH17Kt5dvu1SN2+3W6dOnK+wnJSVFQUFB5hIREXGlUwMAALXcFQeYhIQEffzxx1q1alVV9nPFEhMTVVRUZC6HDh2q6ZYAAEA1qXMlO40ZM0ZpaWnavn27mjVrZq4PDQ1VSUmJCgsLPc7C5OfnKzQ01KzZvXu3x3jldymdX/PTO5fy8/Nlt9vl7+9fYU82m002m+1KpgMAACymUmdgDMPQmDFj9Oabb2rz5s1q0aKFx/ZOnTqpbt262rRpk7lu//79ysvLk8PhkCQ5HA7l5OSooKDArMnIyJDdbld0dLRZc/4Y5TXlYwAAgF+3Sp2BSUhI0MqVK/V///d/ql+/vnnNSlBQkPz9/RUUFKQRI0ZowoQJCg4Olt1u19ixY+VwONS1a1dJUu/evRUdHa0hQ4Zo1qxZcrlcmjp1qhISEswzKKNGjdKiRYs0efJkDR8+XJs3b9bq1auVnp5exdMHAABWVKkzMEuWLFFRUZF69OihsLAwc3n11VfNmnnz5qlfv34aMGCAunfvrtDQUL3xxhvmdh8fH6WlpcnHx0cOh0N//OMfFR8fr+TkZLOmRYsWSk9PV0ZGhjp06KBnn31WL730kpxOZxVMGQAAWN1VfQ5MbcbnwAC4WnwODHCha+JzYAAAAGoCAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFgOAQYAAFhOpQPM9u3b1b9/f4WHh8vLy0tr16712D5s2DB5eXl5LH369PGoOXbsmAYPHiy73a4GDRpoxIgROnHihEfN3r171a1bN/n5+SkiIkKzZs2q/OwAAMA1qdIB5uTJk+rQoYMWL1580Zo+ffroyJEj5vLPf/7TY/vgwYO1b98+ZWRkKC0tTdu3b9fIkSPN7W63W71791ZUVJSysrI0e/ZsTZ8+XS+++GJl2wUAANegOpXdoW/fvurbt+8la2w2m0JDQyvc9umnn2r9+vXas2ePOnfuLElauHCh7r77bs2ZM0fh4eFasWKFSkpK9PLLL8vX11dt27ZVdna25s6d6xF0AADAr1O1XAOzdetWNW3aVK1atdLo0aP1/fffm9syMzPVoEEDM7xIUmxsrLy9vbVr1y6zpnv37vL19TVrnE6n9u/frx9++KHCYxYXF8vtdnssAADg2lTlAaZPnz76xz/+oU2bNumZZ57Rtm3b1LdvX5WWlkqSXC6XmjZt6rFPnTp1FBwcLJfLZdaEhIR41JQ/Lq/5qZSUFAUFBZlLREREVU8NAADUEpV+C+nnDBw40Pw6JiZG7du31w033KCtW7eqV69eVX04U2JioiZMmGA+drvdhBgAAK5R1X4b9fXXX6/GjRvrwIEDkqTQ0FAVFBR41Jw7d07Hjh0zr5sJDQ1Vfn6+R03544tdW2Oz2WS32z0WAABwbar2APPNN9/o+++/V1hYmCTJ4XCosLBQWVlZZs3mzZtVVlamLl26mDXbt2/X2bNnzZqMjAy1atVKDRs2rO6WAQBALVfpAHPixAllZ2crOztbkpSbm6vs7Gzl5eXpxIkTmjRpkt5//3199dVX2rRpk+655x61bNlSTqdTktSmTRv16dNHjz76qHbv3q0dO3ZozJgxGjhwoMLDwyVJgwYNkq+vr0aMGKF9+/bp1Vdf1YIFCzzeIgIAAL9elQ4wH3zwgW6++WbdfPPNkqQJEybo5ptvVlJSknx8fLR371797ne/00033aQRI0aoU6dO+te//iWbzWaOsWLFCrVu3Vq9evXS3Xffrdtvv93jM16CgoL07rvvKjc3V506ddITTzyhpKQkbqEGAACSJC/DMIyabqI6uN1uBQUFqaioiOthAFyRvOSYmm4BqHUik3KqdfzLff3mbyEBAADLIcAAAADLIcAAAADLIcAAAADLIcAAAADLIcAAAADLIcAAAADLqfI/5vhr02nSP2q6BaDWyZodX9MtALjGcQYGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYDgEGAABYTqUDzPbt29W/f3+Fh4fLy8tLa9eu9dhuGIaSkpIUFhYmf39/xcbG6osvvvCoOXbsmAYPHiy73a4GDRpoxIgROnHihEfN3r171a1bN/n5+SkiIkKzZs2q/OwAAMA1qdIB5uTJk+rQoYMWL15c4fZZs2bpueee09KlS7Vr1y4FBgbK6XTqzJkzZs3gwYO1b98+ZWRkKC0tTdu3b9fIkSPN7W63W71791ZUVJSysrI0e/ZsTZ8+XS+++OIVTBEAAFxr6lR2h759+6pv374VbjMMQ/Pnz9fUqVN1zz33SJL+8Y9/KCQkRGvXrtXAgQP16aefav369dqzZ486d+4sSVq4cKHuvvtuzZkzR+Hh4VqxYoVKSkr08ssvy9fXV23btlV2drbmzp3rEXTOV1xcrOLiYvOx2+2u7NQAAIBFVOk1MLm5uXK5XIqNjTXXBQUFqUuXLsrMzJQkZWZmqkGDBmZ4kaTY2Fh5e3tr165dZk337t3l6+tr1jidTu3fv18//PBDhcdOSUlRUFCQuURERFTl1AAAQC1SpQHG5XJJkkJCQjzWh4SEmNtcLpeaNm3qsb1OnToKDg72qKlojPOP8VOJiYkqKioyl0OHDl39hAAAQK1U6beQaiubzSabzVbTbQAAgF9AlZ6BCQ0NlSTl5+d7rM/Pzze3hYaGqqCgwGP7uXPndOzYMY+aisY4/xgAAODXq0oDTIsWLRQaGqpNmzaZ69xut3bt2iWHwyFJcjgcKiwsVFZWllmzefNmlZWVqUuXLmbN9u3bdfbsWbMmIyNDrVq1UsOGDauyZQAAYEGVDjAnTpxQdna2srOzJf144W52drby8vLk5eWlcePG6a9//aveeust5eTkKD4+XuHh4br33nslSW3atFGfPn306KOPavfu3dqxY4fGjBmjgQMHKjw8XJI0aNAg+fr6asSIEdq3b59effVVLViwQBMmTKiyiQMAAOuq9DUwH3zwgXr27Gk+Lg8VQ4cOVWpqqiZPnqyTJ09q5MiRKiws1O23367169fLz8/P3GfFihUaM2aMevXqJW9vbw0YMEDPPfecuT0oKEjvvvuuEhIS1KlTJzVu3FhJSUkXvYUaAAD8ungZhmHUdBPVwe12KygoSEVFRbLb7dV2nE6T/lFtYwNWlTU7vqZbqBJ5yTE13QJQ60Qm5VTr+Jf7+s3fQgIAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZDgAEAAJZT5QFm+vTp8vLy8lhat25tbj9z5owSEhLUqFEj1atXTwMGDFB+fr7HGHl5eYqLi1NAQICaNm2qSZMm6dy5c1XdKgAAsKg61TFo27ZttXHjxv8cpM5/DjN+/Hilp6drzZo1CgoK0pgxY3T//fdrx44dkqTS0lLFxcUpNDRUO3fu1JEjRxQfH6+6devq6aefro52AQCAxVRLgKlTp45CQ0MvWF9UVKS///3vWrlype68805J0rJly9SmTRu9//776tq1q95991198skn2rhxo0JCQtSxY0fNmDFDU6ZM0fTp0+Xr61sdLQMAAAuplmtgvvjiC4WHh+v666/X4MGDlZeXJ0nKysrS2bNnFRsba9a2bt1akZGRyszMlCRlZmYqJiZGISEhZo3T6ZTb7da+ffsueszi4mK53W6PBQAAXJuqPMB06dJFqampWr9+vZYsWaLc3Fx169ZNx48fl8vlkq+vrxo0aOCxT0hIiFwulyTJ5XJ5hJfy7eXbLiYlJUVBQUHmEhERUbUTAwAAtUaVv4XUt29f8+v27durS5cuioqK0urVq+Xv71/VhzMlJiZqwoQJ5mO3202IAQDgGlXtt1E3aNBAN910kw4cOKDQ0FCVlJSosLDQoyY/P9+8ZiY0NPSCu5LKH1d0XU05m80mu93usQAAgGtTtQeYEydO6ODBgwoLC1OnTp1Ut25dbdq0ydy+f/9+5eXlyeFwSJIcDodycnJUUFBg1mRkZMhutys6Orq62wUAABZQ5W8hTZw4Uf3791dUVJQOHz6sadOmycfHR3/4wx8UFBSkESNGaMKECQoODpbdbtfYsWPlcDjUtWtXSVLv3r0VHR2tIUOGaNasWXK5XJo6daoSEhJks9mqul0AAGBBVR5gvvnmG/3hD3/Q999/ryZNmuj222/X+++/ryZNmkiS5s2bJ29vbw0YMEDFxcVyOp16/vnnzf19fHyUlpam0aNHy+FwKDAwUEOHDlVycnJVtwoAACyqygPMqlWrLrndz89Pixcv1uLFiy9aExUVpXfeeaeqWwMAANcI/hYSAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwHAIMAACwnFodYBYvXqzmzZvLz89PXbp00e7du2u6JQAAUAvU2gDz6quvasKECZo2bZr+/e9/q0OHDnI6nSooKKjp1gAAQA2rtQFm7ty5evTRR/Xwww8rOjpaS5cuVUBAgF5++eWabg0AANSwOjXdQEVKSkqUlZWlxMREc523t7diY2OVmZlZ4T7FxcUqLi42HxcVFUmS3G53tfZaWny6WscHrKi6f+5+KcfPlNZ0C0CtU90/3+XjG4ZxybpaGWC+++47lZaWKiQkxGN9SEiIPvvsswr3SUlJ0VNPPXXB+oiIiGrpEcDFBS0cVdMtAKguKUG/yGGOHz+uoKCLH6tWBpgrkZiYqAkTJpiPy8rKdOzYMTVq1EheXl412Bl+CW63WxERETp06JDsdntNtwOgCvHz/etiGIaOHz+u8PDwS9bVygDTuHFj+fj4KD8/32N9fn6+QkNDK9zHZrPJZrN5rGvQoEF1tYhaym638wsOuEbx8/3rcakzL+Vq5UW8vr6+6tSpkzZt2mSuKysr06ZNm+RwOGqwMwAAUBvUyjMwkjRhwgQNHTpUnTt31m9/+1vNnz9fJ0+e1MMPP1zTrQEAgBpWawPMQw89pKNHjyopKUkul0sdO3bU+vXrL7iwF5B+fAtx2rRpF7yNCMD6+PlGRbyMn7tPCQAAoJapldfAAAAAXAoBBgAAWA4BBgAAWA4BBgAAWA4BBriErVu3ysvLS4WFhTXdCoBfSPPmzTV//vyabgM/gwCDX8ywYcPk5eWlmTNneqxfu3Ytf+4BqEXKf1a9vLzk6+urli1bKjk5WefOnavp1n4Re/bs0ciRI2u6DfwMAgx+UX5+fnrmmWf0ww8/VNmYJSUlVTYWgB/16dNHR44c0RdffKEnnnhC06dP1+zZs2u6rV9EkyZNFBAQUNNt4GcQYPCLio2NVWhoqFJSUi5a8/rrr6tt27ay2Wxq3ry5nn32WY/tzZs314wZMxQfHy+73a6RI0cqNTVVDRo0UFpamlq1aqWAgAA98MADOnXqlJYvX67mzZurYcOG+vOf/6zS0lJzrFdeeUWdO3dW/fr1FRoaqkGDBqmgoKDa5g9Yhc1mU2hoqKKiojR69GjFxsbqrbfe0rBhw3Tvvfdqzpw5CgsLU6NGjZSQkKCzZ8+a+xYXF2vixIm67rrrFBgYqC5dumjr1q3m9unTp6tjx44ex5s/f76aN29uPi4/ztNPP62QkBA1aNDAPAs0adIkBQcHq1mzZlq2bJnHODk5Obrzzjvl7++vRo0aaeTIkTpx4sQF416q/5++hTR37lzFxMQoMDBQERER+tOf/uQxJmoGAQa/KB8fHz399NNauHChvvnmmwu2Z2Vl6fe//70GDhyonJwcTZ8+XU8++aRSU1M96ubMmaMOHTroww8/1JNPPilJOnXqlJ577jmtWrVK69ev19atW3XffffpnXfe0TvvvKNXXnlFL7zwgl577TVznLNnz2rGjBn66KOPtHbtWn311VcaNmxYdT4FgCX5+/ubZzu3bNmigwcPasuWLVq+fLlSU1M9fkbHjBmjzMxMrVq1Snv37tWDDz6oPn366IsvvqjUMTdv3qzDhw9r+/btmjt3rqZNm6Z+/fqpYcOG2rVrl0aNGqXHHnvM/F1y8uRJOZ1ONWzYUHv27NGaNWu0ceNGjRkzxmPcn+v/p7y9vfXcc89p3759Wr58uTZv3qzJkydXai6oBgbwCxk6dKhxzz33GIZhGF27djWGDx9uGIZhvPnmm0b5P8VBgwYZd911l8d+kyZNMqKjo83HUVFRxr333utRs2zZMkOSceDAAXPdY489ZgQEBBjHjx831zmdTuOxxx67aI979uwxJJn7bNmyxZBk/PDDD5WfMGBR5/+slpWVGRkZGYbNZjMmTpxoDB061IiKijLOnTtn1j/44IPGQw89ZBiGYXz99deGj4+P8e2333qM2atXLyMxMdEwDMOYNm2a0aFDB4/t8+bNM6Kiojx6iIqKMkpLS811rVq1Mrp162Y+PnfunBEYGGj885//NAzDMF588UWjYcOGxokTJ8ya9PR0w9vb23C5XB7jXqx/w/jxd8y8efMu+vysWbPGaNSo0UW345fBGRjUiGeeeUbLly/Xp59+6rH+008/1W233eax7rbbbtMXX3zh8dZP586dLxgzICBAN9xwg/k4JCREzZs3V7169TzWnf8WUVZWlvr376/IyEjVr19fd9xxhyQpLy/v6iYIWFxaWprq1asnPz8/9e3bVw899JCmT58uSWrbtq18fHzM2rCwMPPnKicnR6WlpbrppptUr149c9m2bZsOHjxYqR7atm0rb+//vEyFhIQoJibGfOzj46NGjRqZx/7000/VoUMHBQYGmjW33XabysrKtH//fo9xL9Z/RTZu3KhevXrpuuuuU/369TVkyBB9//33OnXqVKXmg6pVa/+YI65t3bt3l9PpVGJi4hW9ZXP+L6hydevW9Xjs5eVV4bqysjJJ/znd7HQ6tWLFCjVp0kR5eXlyOp1cGIxfvZ49e2rJkiXy9fVVeHi46tT5z8vFpX6uTpw4IR8fH2VlZXmEBEnmfya8vb1l/OTP8J1/DcqljnOpY1+uyozx1VdfqV+/fho9erT+9re/KTg4WO+9955GjBihkpISLvatQQQY1JiZM2eqY8eOatWqlbmuTZs22rFjh0fdjh07dNNNN13wy/BqffbZZ/r+++81c+ZMRURESJI++OCDKj0GYFWBgYFq2bJlpfe7+eabVVpaqoKCAnXr1q3CmiZNmsjlcskwDPMjFLKzs6+mXUk//v5ITU3VyZMnzf/k7NixQ97e3h6/ZyojKytLZWVlevbZZ82zQatXr77qXnH1eAsJNSYmJkaDBw/Wc889Z6574okntGnTJs2YMUOff/65li9frkWLFmnixIlVfvzIyEj5+vpq4cKF+vLLL/XWW29pxowZVX4c4Nfkpptu0uDBgxUfH6833nhDubm52r17t1JSUpSeni5J6tGjh44ePapZs2bp4MGDWrx4sdatW3fVxx48eLD8/Pw0dOhQffzxx9qyZYvGjh2rIUOGKCQk5IrGbNmypc6ePWv+nnjllVe0dOnSq+4VV48AgxqVnJzscer2lltu0erVq7Vq1Sq1a9dOSUlJSk5OrpY7g5o0aaLU1FStWbNG0dHRmjlzpubMmVPlxwF+bZYtW6b4+Hg98cQTatWqle69917t2bNHkZGRkn48U/L8889r8eLF6tChg3bv3l0l/0kJCAjQhg0bdOzYMf3mN7/RAw88oF69emnRokVXPGaHDh00d+5cPfPMM2rXrp1WrFhxyY+BwC/Hy/jpG5EAAAC1HGdgAACA5RBgAACA5RBgAACA5RBgAACA5RBgAACA5RBgAACA5RBgAACA5RBgAACA5RBgAACA5RBgAACA5RBgAACA5fw/CPP4p4708yUAAAAASUVORK5CYII=", 169 | "text/plain": [ 170 | "
" 171 | ] 172 | }, 173 | "metadata": {}, 174 | "output_type": "display_data" 175 | } 176 | ], 177 | "source": [ 178 | "distribution('train')" 179 | ] 180 | }, 181 | { 182 | "cell_type": "code", 183 | "execution_count": 24, 184 | "id": "7bacfeab", 185 | "metadata": { 186 | "id": "7bacfeab", 187 | "outputId": "ca740fa9-8120-4eaa-e9b1-4f59a2a6fce2" 188 | }, 189 | "outputs": [ 190 | { 191 | "data": { 192 | "text/plain": [ 193 | "{'Normal': 234, 'Pneumonia': 390}" 194 | ] 195 | }, 196 | "execution_count": 24, 197 | "metadata": {}, 198 | "output_type": "execute_result" 199 | }, 200 | { 201 | "data": { 202 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGzCAYAAAAFROyYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA0VklEQVR4nO3de1RVZf7H8Q8XOQp4QFA4moCmeUHRChs9WVZKopnlhJWTo1iOlIPNL0lzMT/vVqiVWo1p02rEHM3JSlOnNO9OSWWURWqmZmEp4CU4XkYQ2L8/Wu5fJ7QCIR7s/Vprr8V+nmc/+7vRAx/25Rwfy7IsAQAAGMS3tgsAAAD4MQIKAAAwDgEFAAAYh4ACAACMQ0ABAADGIaAAAADjEFAAAIBxCCgAAMA4BBQAAGAcAgqAS97mzZvl4+OjV199tdrmnDx5snx8fKptPgDeCChANdm2bZsmT56swsLCGt3P448/rhUrVvyisV999ZV8fHzspV69emrcuLGuvfZa/fWvf1Vubm6V6zh06JAmT56sHTt2VHmO86mJMAGg7iGgANVk27ZtmjJlilEB5Zw//OEPWrRokV588UVNmDBBl19+uebMmaP27dtr6dKlVarj0KFDmjJlSrUHFACQJP/aLgBAzbv66qv1xz/+0avt66+/Vu/evZWcnKz27durc+fOtVQdAFTEGRSgGkyePFljx46VJLVs2dK+pPLVV1/ZY/75z38qPj5eDRo0UFhYmAYNGqSDBw96zbN3714lJSXJ5XKpfv36at68uQYNGqSioiJJko+Pj06dOqWFCxfa+xg2bFiVao6JiVFmZqZKSko0c+ZMu/348eMaM2aM4uLiFBwcLKfTqb59++qTTz6xx2zevFnXXHONJOnee++1a8nMzJQk/ec//9Gdd96p6OhoORwORUVFafTo0frvf/9bpVrP3e/xxRdf6I9//KNCQkLUpEkTTZgwQZZl6eDBg7r99tvldDrlcrn01FNPnXeesrIy/fWvf5XL5VJQUJBuu+22Cv8GF1P7ggUL1LNnT0VERMjhcCg2Nlbz5s2rMK5Fixa69dZb9c477+h3v/ud6tevr8svv1wvvfRShbGFhYUaPXq0WrRoIYfDoebNm2vo0KE6evSoPaa4uFiTJk1S69at7ZofeeQRFRcX/2zNgKk4gwJUgzvuuENffPGFXn75Zc2ePVuNGzeWJDVp0kSS9Nhjj2nChAm666679Kc//UlHjhzRs88+qx49eujjjz9WaGioSkpKlJiYqOLiYj344INyuVz69ttvtXr1ahUWFiokJESLFi3Sn/70J/3ud79TSkqKJKlVq1ZVrtvtdqtVq1Zat26d3fbll19qxYoVuvPOO9WyZUvl5+fr+eef1w033KBdu3apWbNmat++vaZOnaqJEycqJSVF119/vSTp2muvlSQtW7ZMp0+f1siRIxUeHq4PPvhAzz77rL755hstW7asyvXefffdat++vaZPn65///vfevTRRxUWFqbnn39ePXv21IwZM7R48WKNGTNG11xzjXr06OG1/WOPPSYfHx+NGzdOBQUFmjNnjhISErRjxw41aNDgomufN2+eOnTooNtuu03+/v5atWqV/vznP6u8vFypqaleY/ft26eBAwdq+PDhSk5O1j/+8Q8NGzZM8fHx6tChgyTp5MmTuv7667V7927dd999uvrqq3X06FGtXLlS33zzjRo3bqzy8nLddttteuedd5SSkqL27dsrJydHs2fP1hdffFHpy4GAMSwA1eKJJ56wJFkHDhzwav/qq68sPz8/67HHHvNqz8nJsfz9/e32jz/+2JJkLVu27Cf3ExQUZCUnJ/+img4cOGBJsp544okLjrn99tstSVZRUZFlWZZ15swZq6ysrMI8DofDmjp1qt22fft2S5K1YMGCCnOePn26QltGRobl4+Njff311z9Z86ZNmyp8HyZNmmRJslJSUuy20tJSq3nz5paPj481ffp0u/27776zGjRo4PU9OjfnZZddZnk8Hrv9lVdesSRZTz/9dKVrP1fTzx13YmKidfnll3u1xcTEWJKsrVu32m0FBQWWw+GwHn74Ybtt4sSJliTr9ddfrzBveXm5ZVmWtWjRIsvX19f6z3/+49U/f/58S5L17rvvVtgWqAu4xAPUsNdff13l5eW66667dPToUXtxuVy64oortGnTJklSSEiIJGnt2rU6ffr0r1ZfcHCwJOnEiROSJIfDIV/f7380lJWV6dixYwoODlbbtm310Ucf/aI5z52NkKRTp07p6NGjuvbaa2VZlj7++OMq1/qnP/3J/trPz09dunSRZVkaPny43R4aGqq2bdvqyy+/rLD90KFD1bBhQ3t94MCBatq0qd58881qqf2H2xYVFeno0aO64YYb9OWXX9qX6c6JjY21zzxJ359t+3Hdr732mjp37qzf//73FfZ17hHnZcuWqX379mrXrp3X/6+ePXtKkv3/C6hruMQD1LC9e/fKsixdccUV5+2vV6+epO/vXUlLS9OsWbO0ePFiXX/99brtttvsey5qysmTJyXJ/sVdXl6up59+Ws8995wOHDigsrIye2x4ePgvmjM3N1cTJ07UypUr9d1333n1/fgXdWVER0d7rYeEhKh+/fr2JbUfth87dqzC9j/+N/Dx8VHr1q297hW6mNrfffddTZo0SVlZWRVCZlFRkde/44+PRZIaNWrktc/9+/crKSnpJ/e5d+9e7d69276c+GMFBQU/uT1gKgIKUMPKy8vl4+Ojt956S35+fhX6z53BkKSnnnpKw4YN0xtvvKG3335bf/nLX5SRkaH33ntPzZs3r5H6PvvsM0VERMjpdEr6/jHmCRMm6L777tO0adMUFhYmX19fPfTQQyovL//Z+crKynTzzTfr+PHjGjdunNq1a6egoCB9++23GjZs2C+a40LO9/07X5skWZZV6fkvpvb9+/erV69eateunWbNmqWoqCgFBATozTff1OzZsytsW111l5eXKy4uTrNmzTpvf1RUVKXmA0xBQAGqyYXeVbRVq1ayLEstW7ZUmzZtfnaeuLg4xcXFafz48dq2bZu6d++u+fPn69FHH/3J/VRFVlaW9u/f7/UI8quvvqqbbrpJL774otfYwsJCrzMVF6ojJydHX3zxhRYuXKihQ4fa7T+8Ebe27N2712vdsizt27dPnTp1knRxta9atUrFxcVauXKl19mRi7nE0qpVK3322Wc/O+aTTz5Rr169eGdbXFK4BwWoJkFBQZJU4Y3a7rjjDvn5+WnKlCkV/jq2LMu+FOHxeFRaWurVHxcXJ19fX6/HRYOCgqrlzeC+/vprDRs2TAEBAfYj0tL3f9n/uM5ly5bp22+/9Wq70PGeOzPwwzksy9LTTz990TVfrJdeesm+10b6PowdPnxYffv2lXRxtZ9v26KiIi1YsKDK9SYlJemTTz7R8uXLK/Sd289dd92lb7/9Vi+88EKFMf/973916tSpKu8fqE2cQQGqSXx8vCTpf//3fzVo0CDVq1dP/fv3V6tWrfToo48qPT1dX331lQYMGKCGDRvqwIEDWr58uVJSUjRmzBht3LhRo0aN0p133qk2bdqotLRUixYtkp+fn9d9CPHx8Vq/fr1mzZqlZs2aqWXLluratetP1vbRRx/pn//8p8rLy1VYWKjt27frtddek4+PjxYtWmSfQZCkW2+9VVOnTtW9996ra6+9Vjk5OVq8eLEuv/xyrzlbtWql0NBQzZ8/Xw0bNlRQUJC6du2qdu3aqVWrVhozZoy+/fZbOZ1OvfbaaxXu56gNYWFhuu6663TvvfcqPz9fc+bMUevWrTVixAhJuqjae/furYCAAPXv31/333+/Tp48qRdeeEERERE6fPhwleodO3asXn31Vd1555267777FB8fr+PHj2vlypWaP3++OnfurCFDhuiVV17RAw88oE2bNql79+4qKyvT559/rldeeUVr165Vly5dqrR/oFb96s8NAZewadOmWZdddpnl6+tb4ZHj1157zbruuuusoKAgKygoyGrXrp2Vmppq7dmzx7Isy/ryyy+t++67z2rVqpVVv359KywszLrpppus9evXe+3j888/t3r06GE1aNDAkvSTjxyfe8z43OLv72+FhYVZXbt2tdLT08/7yO+ZM2eshx9+2GratKnVoEEDq3v37lZWVpZ1ww03WDfccIPX2DfeeMOKjY21/P39vR453rVrl5WQkGAFBwdbjRs3tkaMGGF98sknF3ws+Yd+6jHjI0eOeI1NTk62goKCKsxxww03WB06dKgw58svv2ylp6dbERERVoMGDax+/fpV+B780trP95jxypUrrU6dOln169e3WrRoYc2YMcP6xz/+UeH/QkxMjNWvX7/z1v3j7/GxY8esUaNGWZdddpkVEBBgNW/e3EpOTraOHj1qjykpKbFmzJhhdejQwXI4HFajRo2s+Ph4a8qUKfbj40Bd42NZVbiTDAAAoAZxDwoAADAOAQUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHHq5Bu1lZeX69ChQ2rYsCFv7QwAQB1hWZZOnDihZs2a2Z+afiF1MqAcOnSID8ACAKCOOnjw4M9+AGqdDCjnPhb+4MGD9iewAgAAs3k8HkVFRdm/x39KnQwo5y7rOJ1OAgoAAHXML7k9g5tkAQCAcS4qoEyfPl0+Pj566KGH7LYzZ84oNTVV4eHhCg4OVlJSkvLz8722y83NVb9+/RQYGKiIiAiNHTu2wsfMAwCA364qB5Tt27fr+eef9/qYdkkaPXq0Vq1apWXLlmnLli06dOiQ7rjjDru/rKxM/fr1U0lJibZt26aFCxcqMzNTEydOrPpRAACAS0qVAsrJkyc1ePBgvfDCC2rUqJHdXlRUpBdffFGzZs1Sz549FR8frwULFmjbtm167733JElvv/22du3apX/+85+68sor1bdvX02bNk1z585VSUlJ9RwVAACo06oUUFJTU9WvXz8lJCR4tWdnZ+vs2bNe7e3atVN0dLSysrIkSVlZWYqLi1NkZKQ9JjExUR6PRzt37jzv/oqLi+XxeLwWAABw6ar0UzxLly7VRx99pO3bt1foy8vLU0BAgEJDQ73aIyMjlZeXZ4/5YTg513+u73wyMjI0ZcqUypYKAADqqEqdQTl48KD+53/+R4sXL1b9+vVrqqYK0tPTVVRUZC8HDx781fYNAAB+fZUKKNnZ2SooKNDVV18tf39/+fv7a8uWLXrmmWfk7++vyMhIlZSUqLCw0Gu7/Px8uVwuSZLL5arwVM+59XNjfszhcNjvecJ7nwAAcOmrVEDp1auXcnJytGPHDnvp0qWLBg8ebH9dr149bdiwwd5mz549ys3NldvtliS53W7l5OSooKDAHrNu3To5nU7FxsZW02EBAIC6rFL3oDRs2FAdO3b0agsKClJ4eLjdPnz4cKWlpSksLExOp1MPPvig3G63unXrJknq3bu3YmNjNWTIEM2cOVN5eXkaP368UlNT5XA4qumwAABAXVbtb3U/e/Zs+fr6KikpScXFxUpMTNRzzz1n9/v5+Wn16tUaOXKk3G63goKClJycrKlTp1Z3KQAAoI7ysSzLqu0iKsvj8SgkJERFRUXcjwIAQB1Rmd/ffBYPAAAwDgEFAAAYp9rvQQGAuiB3alxtlwAYJ3piTm2XYOMMCgAAMA4BBQAAGIeAAgAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOAQUAABgHAIKAAAwDgEFAAAYh4ACAACMQ0ABAADGIaAAAADjEFAAAIBxCCgAAMA4BBQAAGAcAgoAADAOAQUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGIeAAgAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOAQUAABgnEoFlHnz5qlTp05yOp1yOp1yu91666237P4bb7xRPj4+XssDDzzgNUdubq769eunwMBARUREaOzYsSotLa2eowEAAJcE/8oMbt68uaZPn64rrrhClmVp4cKFuv322/Xxxx+rQ4cOkqQRI0Zo6tSp9jaBgYH212VlZerXr59cLpe2bdumw4cPa+jQoapXr54ef/zxajokAABQ11UqoPTv399r/bHHHtO8efP03nvv2QElMDBQLpfrvNu//fbb2rVrl9avX6/IyEhdeeWVmjZtmsaNG6fJkycrICDgvNsVFxeruLjYXvd4PJUpGwAA1DFVvgelrKxMS5cu1alTp+R2u+32xYsXq3HjxurYsaPS09N1+vRpuy8rK0txcXGKjIy02xITE+XxeLRz584L7isjI0MhISH2EhUVVdWyAQBAHVCpMyiSlJOTI7fbrTNnzig4OFjLly9XbGysJOmee+5RTEyMmjVrpk8//VTjxo3Tnj179Prrr0uS8vLyvMKJJHs9Ly/vgvtMT09XWlqave7xeAgpAABcwiodUNq2basdO3aoqKhIr776qpKTk7VlyxbFxsYqJSXFHhcXF6emTZuqV69e2r9/v1q1alXlIh0OhxwOR5W3BwAAdUulL/EEBASodevWio+PV0ZGhjp37qynn376vGO7du0qSdq3b58kyeVyKT8/32vMufUL3bcCAAB+ey76fVDKy8u9bmD9oR07dkiSmjZtKklyu93KyclRQUGBPWbdunVyOp32ZSIAAIBKXeJJT09X3759FR0drRMnTmjJkiXavHmz1q5dq/3792vJkiW65ZZbFB4erk8//VSjR49Wjx491KlTJ0lS7969FRsbqyFDhmjmzJnKy8vT+PHjlZqayiUcAABgq1RAKSgo0NChQ3X48GGFhISoU6dOWrt2rW6++WYdPHhQ69ev15w5c3Tq1ClFRUUpKSlJ48ePt7f38/PT6tWrNXLkSLndbgUFBSk5OdnrfVMAAAB8LMuyaruIyvJ4PAoJCVFRUZGcTmdtlwOgDsqdGlfbJQDGiZ6YU6PzV+b3N5/FAwAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOAQUAABgHAIKAAAwDgEFAAAYh4ACAACMQ0ABAADGIaAAAADjEFAAAIBxCCgAAMA4BBQAAGAcAgoAADAOAQUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGIeAAgAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOAQUAABgHAIKAAAwDgEFAAAYh4ACAACMQ0ABAADGqVRAmTdvnjp16iSn0ymn0ym326233nrL7j9z5oxSU1MVHh6u4OBgJSUlKT8/32uO3Nxc9evXT4GBgYqIiNDYsWNVWlpaPUcDAAAuCZUKKM2bN9f06dOVnZ2tDz/8UD179tTtt9+unTt3SpJGjx6tVatWadmyZdqyZYsOHTqkO+64w96+rKxM/fr1U0lJibZt26aFCxcqMzNTEydOrN6jAgAAdZqPZVnWxUwQFhamJ554QgMHDlSTJk20ZMkSDRw4UJL0+eefq3379srKylK3bt301ltv6dZbb9WhQ4cUGRkpSZo/f77GjRunI0eOKCAg4Bft0+PxKCQkREVFRXI6nRdTPoDfqNypcbVdAmCc6Ik5NTp/ZX5/V/kelLKyMi1dulSnTp2S2+1Wdna2zp49q4SEBHtMu3btFB0draysLElSVlaW4uLi7HAiSYmJifJ4PPZZmPMpLi6Wx+PxWgAAwKWr0gElJydHwcHBcjgceuCBB7R8+XLFxsYqLy9PAQEBCg0N9RofGRmpvLw8SVJeXp5XODnXf67vQjIyMhQSEmIvUVFRlS0bAADUIZUOKG3bttWOHTv0/vvva+TIkUpOTtauXbtqojZbenq6ioqK7OXgwYM1uj8AAFC7/Cu7QUBAgFq3bi1Jio+P1/bt2/X000/r7rvvVklJiQoLC73OouTn58vlckmSXC6XPvjgA6/5zj3lc27M+TgcDjkcjsqWCgAA6qiLfh+U8vJyFRcXKz4+XvXq1dOGDRvsvj179ig3N1dut1uS5Ha7lZOTo4KCAnvMunXr5HQ6FRsbe7GlAACAS0SlzqCkp6erb9++io6O1okTJ7RkyRJt3rxZa9euVUhIiIYPH660tDSFhYXJ6XTqwQcflNvtVrdu3SRJvXv3VmxsrIYMGaKZM2cqLy9P48ePV2pqKmdIAACArVIBpaCgQEOHDtXhw4cVEhKiTp06ae3atbr55pslSbNnz5avr6+SkpJUXFysxMREPffcc/b2fn5+Wr16tUaOHCm3262goCAlJydr6tSp1XtUAACgTrvo90GpDbwPCoCLxfugABVdEu+DAgAAUFMIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGIeAAgAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOAQUAABgHAIKAAAwDgEFAAAYh4ACAACMQ0ABAADGIaAAAADjEFAAAIBxCCgAAMA4BBQAAGAcAgoAADAOAQUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGIeAAgAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOJUKKBkZGbrmmmvUsGFDRUREaMCAAdqzZ4/XmBtvvFE+Pj5eywMPPOA1Jjc3V/369VNgYKAiIiI0duxYlZaWXvzRAACAS4J/ZQZv2bJFqampuuaaa1RaWqq//vWv6t27t3bt2qWgoCB73IgRIzR16lR7PTAw0P66rKxM/fr1k8vl0rZt23T48GENHTpU9erV0+OPP14NhwQAAOq6SgWUNWvWeK1nZmYqIiJC2dnZ6tGjh90eGBgol8t13jnefvtt7dq1S+vXr1dkZKSuvPJKTZs2TePGjdPkyZMVEBBQhcMAAACXkou6B6WoqEiSFBYW5tW+ePFiNW7cWB07dlR6erpOnz5t92VlZSkuLk6RkZF2W2Jiojwej3bu3Hne/RQXF8vj8XgtAADg0lWpMyg/VF5eroceekjdu3dXx44d7fZ77rlHMTExatasmT799FONGzdOe/bs0euvvy5JysvL8wonkuz1vLy88+4rIyNDU6ZMqWqpAACgjqlyQElNTdVnn32md955x6s9JSXF/jouLk5NmzZVr169tH//frVq1apK+0pPT1daWpq97vF4FBUVVbXCAQCA8ap0iWfUqFFavXq1Nm3apObNm//k2K5du0qS9u3bJ0lyuVzKz8/3GnNu/UL3rTgcDjmdTq8FAABcuioVUCzL0qhRo7R8+XJt3LhRLVu2/NltduzYIUlq2rSpJMntdisnJ0cFBQX2mHXr1snpdCo2NrYy5QAAgEtUpS7xpKamasmSJXrjjTfUsGFD+56RkJAQNWjQQPv379eSJUt0yy23KDw8XJ9++qlGjx6tHj16qFOnTpKk3r17KzY2VkOGDNHMmTOVl5en8ePHKzU1VQ6Ho/qPEAAA1DmVOoMyb948FRUV6cYbb1TTpk3t5V//+pckKSAgQOvXr1fv3r3Vrl07Pfzww0pKStKqVavsOfz8/LR69Wr5+fnJ7Xbrj3/8o4YOHer1vikAAOC3rVJnUCzL+sn+qKgobdmy5WfniYmJ0ZtvvlmZXQMAgN8QPosHAAAYh4ACAACMQ0ABAADGIaAAAADjEFAAAIBxCCgAAMA4BBQAAGCcKn9Y4G9B/NiXarsEwDjZTwyt7RIA/AZwBgUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGIeAAgAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOAQUAABgHAIKAAAwDgEFAAAYh4ACAACMQ0ABAADGIaAAAADjEFAAAIBxCCgAAMA4BBQAAGAcAgoAADAOAQUAABiHgAIAAIxTqYCSkZGha665Rg0bNlRERIQGDBigPXv2eI05c+aMUlNTFR4eruDgYCUlJSk/P99rTG5urvr166fAwEBFRERo7NixKi0tvfijAQAAl4RKBZQtW7YoNTVV7733ntatW6ezZ8+qd+/eOnXqlD1m9OjRWrVqlZYtW6YtW7bo0KFDuuOOO+z+srIy9evXTyUlJdq2bZsWLlyozMxMTZw4sfqOCgAA1Gk+lmVZVd34yJEjioiI0JYtW9SjRw8VFRWpSZMmWrJkiQYOHChJ+vzzz9W+fXtlZWWpW7dueuutt3Trrbfq0KFDioyMlCTNnz9f48aN05EjRxQQEPCz+/V4PAoJCVFRUZGcTmdVy/9Z8WNfqrG5gboq+4mhtV1CtcidGlfbJQDGiZ6YU6PzV+b390Xdg1JUVCRJCgsLkyRlZ2fr7NmzSkhIsMe0a9dO0dHRysrKkiRlZWUpLi7ODieSlJiYKI/Ho507d553P8XFxfJ4PF4LAAC4dFU5oJSXl+uhhx5S9+7d1bFjR0lSXl6eAgICFBoa6jU2MjJSeXl59pgfhpNz/ef6zicjI0MhISH2EhUVVdWyAQBAHVDlgJKamqrPPvtMS5curc56zis9PV1FRUX2cvDgwRrfJwAAqD3+Vdlo1KhRWr16tbZu3armzZvb7S6XSyUlJSosLPQ6i5Kfny+Xy2WP+eCDD7zmO/eUz7kxP+ZwOORwOKpSKgAAqIMqdQbFsiyNGjVKy5cv18aNG9WyZUuv/vj4eNWrV08bNmyw2/bs2aPc3Fy53W5JktvtVk5OjgoKCuwx69atk9PpVGxs7MUcCwAAuERU6gxKamqqlixZojfeeEMNGza07xkJCQlRgwYNFBISouHDhystLU1hYWFyOp168MEH5Xa71a1bN0lS7969FRsbqyFDhmjmzJnKy8vT+PHjlZqaylkSAAAgqZIBZd68eZKkG2+80at9wYIFGjZsmCRp9uzZ8vX1VVJSkoqLi5WYmKjnnnvOHuvn56fVq1dr5MiRcrvdCgoKUnJysqZOnXpxRwIAAC4ZlQoov+QtU+rXr6+5c+dq7ty5FxwTExOjN998szK7BgAAvyF8Fg8AADAOAQUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGIeAAgAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOAQUAABgHAIKAAAwDgEFAAAYh4ACAACMQ0ABAADGIaAAAADjEFAAAIBxCCgAAMA4BBQAAGAcAgoAADAOAQUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGKfSAWXr1q3q37+/mjVrJh8fH61YscKrf9iwYfLx8fFa+vTp4zXm+PHjGjx4sJxOp0JDQzV8+HCdPHnyog4EAABcOiodUE6dOqXOnTtr7ty5FxzTp08fHT582F5efvllr/7Bgwdr586dWrdunVavXq2tW7cqJSWl8tUDAIBLkn9lN+jbt6/69u37k2McDodcLtd5+3bv3q01a9Zo+/bt6tKliyTp2Wef1S233KInn3xSzZo1q2xJAADgElMj96Bs3rxZERERatu2rUaOHKljx47ZfVlZWQoNDbXDiSQlJCTI19dX77///nnnKy4ulsfj8VoAAMClq9oDSp8+ffTSSy9pw4YNmjFjhrZs2aK+ffuqrKxMkpSXl6eIiAivbfz9/RUWFqa8vLzzzpmRkaGQkBB7iYqKqu6yAQCAQSp9iefnDBo0yP46Li5OnTp1UqtWrbR582b16tWrSnOmp6crLS3NXvd4PIQUAAAuYTX+mPHll1+uxo0ba9++fZIkl8ulgoICrzGlpaU6fvz4Be9bcTgccjqdXgsAALh01XhA+eabb3Ts2DE1bdpUkuR2u1VYWKjs7Gx7zMaNG1VeXq6uXbvWdDkAAKAOqPQlnpMnT9pnQyTpwIED2rFjh8LCwhQWFqYpU6YoKSlJLpdL+/fv1yOPPKLWrVsrMTFRktS+fXv16dNHI0aM0Pz583X27FmNGjVKgwYN4gkeAAAgqQpnUD788ENdddVVuuqqqyRJaWlpuuqqqzRx4kT5+fnp008/1W233aY2bdpo+PDhio+P13/+8x85HA57jsWLF6tdu3bq1auXbrnlFl133XX6+9//Xn1HBQAA6rRKn0G58cYbZVnWBfvXrl37s3OEhYVpyZIlld01AAD4jeCzeAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGIeAAgAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOAQUAABgHAIKAAAwDgEFAAAYh4ACAACMQ0ABAADGIaAAAADjEFAAAIBxCCgAAMA4BBQAAGAcAgoAADAOAQUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGIeAAgAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOJUOKFu3blX//v3VrFkz+fj4aMWKFV79lmVp4sSJatq0qRo0aKCEhATt3bvXa8zx48c1ePBgOZ1OhYaGavjw4Tp58uRFHQgAALh0VDqgnDp1Sp07d9bcuXPP2z9z5kw988wzmj9/vt5//30FBQUpMTFRZ86csccMHjxYO3fu1Lp167R69Wpt3bpVKSkpVT8KAABwSfGv7AZ9+/ZV3759z9tnWZbmzJmj8ePH6/bbb5ckvfTSS4qMjNSKFSs0aNAg7d69W2vWrNH27dvVpUsXSdKzzz6rW265RU8++aSaNWt2EYcDAAAuBdV6D8qBAweUl5enhIQEuy0kJERdu3ZVVlaWJCkrK0uhoaF2OJGkhIQE+fr66v333z/vvMXFxfJ4PF4LAAC4dFVrQMnLy5MkRUZGerVHRkbafXl5eYqIiPDq9/f3V1hYmD3mxzIyMhQSEmIvUVFR1Vk2AAAwTJ14iic9PV1FRUX2cvDgwdouCQAA1KBqDSgul0uSlJ+f79Wen59v97lcLhUUFHj1l5aW6vjx4/aYH3M4HHI6nV4LAAC4dFVrQGnZsqVcLpc2bNhgt3k8Hr3//vtyu92SJLfbrcLCQmVnZ9tjNm7cqPLycnXt2rU6ywEAAHVUpZ/iOXnypPbt22evHzhwQDt27FBYWJiio6P10EMP6dFHH9UVV1yhli1basKECWrWrJkGDBggSWrfvr369OmjESNGaP78+Tp79qxGjRqlQYMG8QQPAACQVIWA8uGHH+qmm26y19PS0iRJycnJyszM1COPPKJTp04pJSVFhYWFuu6667RmzRrVr1/f3mbx4sUaNWqUevXqJV9fXyUlJemZZ56phsMBAACXAh/LsqzaLqKyPB6PQkJCVFRUVKP3o8SPfanG5gbqquwnhtZ2CdUid2pcbZcAGCd6Yk6Nzl+Z39914ikeAADw20JAAQAAxiGgAAAA4xBQAACAcQgoAADAOAQUAABgHAIKAAAwDgEFAAAYh4ACAACMQ0ABAADGIaAAAADjEFAAAIBxCCgAAMA4BBQAAGAcAgoAADAOAQUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGIeAAgAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOAQUAABgHAIKAAAwDgEFAAAYh4ACAACMQ0ABAADGqfaAMnnyZPn4+Hgt7dq1s/vPnDmj1NRUhYeHKzg4WElJScrPz6/uMgAAQB1WI2dQOnTooMOHD9vLO++8Y/eNHj1aq1at0rJly7RlyxYdOnRId9xxR02UAQAA6ij/GpnU318ul6tCe1FRkV588UUtWbJEPXv2lCQtWLBA7du313vvvadu3brVRDkAAKCOqZEzKHv37lWzZs10+eWXa/DgwcrNzZUkZWdn6+zZs0pISLDHtmvXTtHR0crKyrrgfMXFxfJ4PF4LAAC4dFV7QOnatasyMzO1Zs0azZs3TwcOHND111+vEydOKC8vTwEBAQoNDfXaJjIyUnl5eRecMyMjQyEhIfYSFRVV3WUDAACDVPslnr59+9pfd+rUSV27dlVMTIxeeeUVNWjQoEpzpqenKy0tzV73eDyEFAAALmE1/phxaGio2rRpo3379snlcqmkpESFhYVeY/Lz8897z8o5DodDTqfTawEAAJeuGg8oJ0+e1P79+9W0aVPFx8erXr162rBhg92/Z88e5ebmyu1213QpAACgjqj2SzxjxoxR//79FRMTo0OHDmnSpEny8/PTH/7wB4WEhGj48OFKS0tTWFiYnE6nHnzwQbndbp7gAQAAtmoPKN98843+8Ic/6NixY2rSpImuu+46vffee2rSpIkkafbs2fL19VVSUpKKi4uVmJio5557rrrLAAAAdVi1B5SlS5f+ZH/9+vU1d+5czZ07t7p3DQAALhF8Fg8AADAOAQUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGIeAAgAAjENAAQAAxiGgAAAA4xBQAACAcQgoAADAOAQUAABgHAIKAAAwDgEFAAAYh4ACAACMQ0ABAADGIaAAAADjEFAAAIBxCCgAAMA4BBQAAGAcAgoAADAOAQUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGKdWA8rcuXPVokUL1a9fX127dtUHH3xQm+UAAABD1FpA+de//qW0tDRNmjRJH330kTp37qzExEQVFBTUVkkAAMAQtRZQZs2apREjRujee+9VbGys5s+fr8DAQP3jH/+orZIAAIAh/GtjpyUlJcrOzlZ6errd5uvrq4SEBGVlZVUYX1xcrOLiYnu9qKhIkuTxeGq0zrLi/9bo/EBdVNOvu1/LiTNltV0CYJyafn2fm9+yrJ8dWysB5ejRoyorK1NkZKRXe2RkpD7//PMK4zMyMjRlypQK7VFRUTVWI4DzC3n2gdouAUBNyQj5VXZz4sQJhYT89L5qJaBUVnp6utLS0uz18vJyHT9+XOHh4fLx8anFyvBr8Hg8ioqK0sGDB+V0Omu7HADViNf3b4tlWTpx4oSaNWv2s2NrJaA0btxYfn5+ys/P92rPz8+Xy+WqMN7hcMjhcHi1hYaG1mSJMJDT6eQHGHCJ4vX92/FzZ07OqZWbZAMCAhQfH68NGzbYbeXl5dqwYYPcbndtlAQAAAxSa5d40tLSlJycrC5duuh3v/ud5syZo1OnTunee++trZIAAIAhai2g3H333Tpy5IgmTpyovLw8XXnllVqzZk2FG2cBh8OhSZMmVbjMB6Du4/WNC/GxfsmzPgAAAL8iPosHAAAYh4ACAACMQ0ABAADGIaAAAADjEFDwm7V582b5+PiosLCwtksB8Ctp0aKF5syZU9tl4BcgoKBaDBs2TD4+Ppo+fbpX+4oVK/g4AsAg516rPj4+CggIUOvWrTV16lSVlpbWdmm/iu3btyslJaW2y8AvQEBBtalfv75mzJih7777rtrmLCkpqba5AHyvT58+Onz4sPbu3auHH35YkydP1hNPPFHbZf0qmjRposDAwNouA78AAQXVJiEhQS6XSxkZGRcc89prr6lDhw5yOBxq0aKFnnrqKa/+Fi1aaNq0aRo6dKicTqdSUlKUmZmp0NBQrV69Wm3btlVgYKAGDhyo06dPa+HChWrRooUaNWqkv/zlLyorK7PnWrRokbp06aKGDRvK5XLpnnvuUUFBQY0dP1BXOBwOuVwuxcTEaOTIkUpISNDKlSs1bNgwDRgwQE8++aSaNm2q8PBwpaam6uzZs/a2xcXFGjNmjC677DIFBQWpa9eu2rx5s90/efJkXXnllV77mzNnjlq0aGGvn9vP448/rsjISIWGhtpnccaOHauwsDA1b95cCxYs8JonJydHPXv2VIMGDRQeHq6UlBSdPHmywrw/Vf+PL/HMmjVLcXFxCgoKUlRUlP785z97zYnaQ0BBtfHz89Pjjz+uZ599Vt98802F/uzsbN11110aNGiQcnJyNHnyZE2YMEGZmZle45588kl17txZH3/8sSZMmCBJOn36tJ555hktXbpUa9as0ebNm/X73/9eb775pt58800tWrRIzz//vF599VV7nrNnz2ratGn65JNPtGLFCn311VcaNmxYTX4LgDqpQYMG9tnKTZs2af/+/dq0aZMWLlyozMxMr9foqFGjlJWVpaVLl+rTTz/VnXfeqT59+mjv3r2V2ufGjRt16NAhbd26VbNmzdKkSZN06623qlGjRnr//ff1wAMP6P7777d/lpw6dUqJiYlq1KiRtm/frmXLlmn9+vUaNWqU17w/V/+P+fr66plnntHOnTu1cOFCbdy4UY888kiljgU1xAKqQXJysnX77bdblmVZ3bp1s+677z7Lsixr+fLl1rn/Zvfcc4918803e203duxYKzY21l6PiYmxBgwY4DVmwYIFliRr3759dtv9999vBQYGWidOnLDbEhMTrfvvv/+CNW7fvt2SZG+zadMmS5L13XffVf6AgTrqh6/V8vJya926dZbD4bDGjBljJScnWzExMVZpaak9/s4777Tuvvtuy7Is6+uvv7b8/Pysb7/91mvOXr16Wenp6ZZlWdakSZOszp07e/XPnj3biomJ8aohJibGKisrs9vatm1rXX/99fZ6aWmpFRQUZL388suWZVnW3//+d6tRo0bWyZMn7TH//ve/LV9fXysvL89r3gvVb1nf/4yZPXv2Bb8/y5Yts8LDwy/Yj18PZ1BQ7WbMmKGFCxdq9+7dXu27d+9W9+7dvdq6d++uvXv3el2a6dKlS4U5AwMD1apVK3s9MjJSLVq0UHBwsFfbDy/hZGdnq3///oqOjlbDhg11ww03SJJyc3Mv7gCBOm716tUKDg5W/fr11bdvX919992aPHmyJKlDhw7y8/OzxzZt2tR+XeXk5KisrExt2rRRcHCwvWzZskX79++vVA0dOnSQr+///wqKjIxUXFycve7n56fw8HB737t371bnzp0VFBRkj+nevbvKy8u1Z88er3kvVP/5rF+/Xr169dJll12mhg0basiQITp27JhOnz5dqeNB9au1DwvEpatHjx5KTExUenp6lS6p/PAH0Dn16tXzWvfx8TlvW3l5uaT/Px2cmJioxYsXq0mTJsrNzVViYiI33uI376abbtK8efMUEBCgZs2ayd///38V/NTr6uTJk/Lz81N2drZXCJBk/7Hg6+sr60cf8fbDe0B+aj8/te9fqjJzfPXVV7r11ls1cuRIPfbYYwoLC9M777yj4cOHq6SkhJtpaxkBBTVi+vTpuvLKK9W2bVu7rX379nr33Xe9xr377rtq06ZNhR92F+vzzz/XsWPHNH36dEVFRUmSPvzww2rdB1BXBQUFqXXr1pXe7qqrrlJZWZkKCgp0/fXXn3dMkyZNlJeXJ8uy7LcY2LFjx8WUK+n7nx+ZmZk6deqU/UfMu+++K19fX6+fM5WRnZ2t8vJyPfXUU/bZnFdeeeWia0X14BIPakRcXJwGDx6sZ555xm57+OGHtWHDBk2bNk1ffPGFFi5cqL/97W8aM2ZMte8/OjpaAQEBevbZZ/Xll19q5cqVmjZtWrXvB/gtadOmjQYPHqyhQ4fq9ddf14EDB/TBBx8oIyND//73vyVJN954o44cOaKZM2dq//79mjt3rt56662L3vfgwYNVv359JScn67PPPtOmTZv04IMPasiQIYqMjKzSnK1bt9bZs2ftnxOLFi3S/PnzL7pWVA8CCmrM1KlTvU6tXn311XrllVe0dOlSdezYURMnTtTUqVNr5MmaJk2aKDMzU8uWLVNsbKymT5+uJ598str3A/zWLFiwQEOHDtXDDz+stm3basCAAdq+fbuio6MlfX+m47nnntPcuXPVuXNnffDBB9XyR0hgYKDWrl2r48eP65prrtHAgQPVq1cv/e1vf6vynJ07d9asWbM0Y8YMdezYUYsXL/7Jt0nAr8vH+vHFQgAAgFrGGRQAAGAcAgoAADAOAQUAABiHgAIAAIxDQAEAAMYhoAAAAOMQUAAAgHEIKAAAwDgEFAAAYBwCCgAAMA4BBQAAGOf/ADPDd/mfjWbSAAAAAElFTkSuQmCC", 203 | "text/plain": [ 204 | "
" 205 | ] 206 | }, 207 | "metadata": {}, 208 | "output_type": "display_data" 209 | } 210 | ], 211 | "source": [ 212 | "distribution('test')" 213 | ] 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": 25, 218 | "id": "8b61bbae", 219 | "metadata": { 220 | "id": "8b61bbae", 221 | "outputId": "334cb43e-959e-4d72-babf-ce1724a86683" 222 | }, 223 | "outputs": [ 224 | { 225 | "data": { 226 | "text/plain": [ 227 | "{'Normal': 9, 'Pneumonia': 9}" 228 | ] 229 | }, 230 | "execution_count": 25, 231 | "metadata": {}, 232 | "output_type": "execute_result" 233 | }, 234 | { 235 | "data": { 236 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAGzCAYAAABzfl4TAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAil0lEQVR4nO3df3zP9f7/8ft7i/d+D8tvs4kQlp+lOEhoypx0Ln4dysYJR34kP07tXC7Mj8MoRPMjdWr2cfFbcVwQSZYSkaRV5HcWMT+W+XEa9n5+/+i79/G2kfc8h+l2vVzel4v36/368dhq223v1+u9t8MYYwQAAGCBz+0eAAAA3D0ICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsIC6CQzZkzRw6HQ4cOHbrdoxQJcXFxCgoKsrrPyMhIxcXFWd0ngPwRFsAdYtSoUXI4HO5bQECAKleurPbt2ys5OVnZ2dkF3vfq1as1atQoe8P+f4URAQCKNsICuMPMmjVLc+fOVVJSkp5//nmdPn1avXr10sMPP6z09PQC7XP16tUaPXq05UkBIK97bvcAADx17NhR9957r/v+yJEjNW/ePPXo0UOdOnXSli1bbuN0AHB9PGMBXGHp0qVyOBz65JNP8jw2e/ZsORwOffvtt5Kkb775RnFxcbrvvvvk5+encuXKqVevXjp16pT1ubp3767nn39eX3zxhdatW+de/umnn6pTp06qXLmynE6nwsPD9dJLL+m///2ve524uDjNmDFDkjxOteSaNGmSmjRporCwMPn7+6thw4ZaunRpgWeNjIxUTEyMUlNT1ahRI/n7+ysqKkqpqamSpPfff19RUVHy8/NTw4YNtWPHjnz3c+DAAUVHRyswMFAVKlTQmDFjdPWbMRd09tOnT2vYsGGKiopSUFCQQkJC9OSTT2rnzp0e66WmpsrhcGjx4sUaN26cKlWqJD8/P7Vq1Ur79u3Ls98vvvhCTz31lEqWLKnAwEA9+OCDmjZtmsc6u3fvVseOHVWqVCn5+fmpUaNGWrFixe/ODBQVhAVwhXbt2ikoKEiLFy/O89iiRYtUu3Zt1alTR5K0bt06HThwQD179lRSUpK6du2qhQsX6qmnnsrzA9CG5557TpL04YcfupctWbJEFy5cUL9+/ZSUlKTo6GglJSWpR48e7nX69u2rNm3aSJLmzp3rvuWaNm2a6tevrzFjxmj8+PG655571KlTJ61atarAs+7bt0/dunVT+/btlZiYqMzMTLVv317z5s3TSy+9pGeffVajR4/W/v371blzZ7lcLo/tc3Jy1LZtW5UtW1avvvqqGjZsqISEBCUkJHisV9DZDxw4oOXLlysmJkZTpkzR8OHDlZaWphYtWujo0aN51p8wYYKWLVumYcOGKT4+Xlu2bFH37t091lm3bp2aN2+u77//Xi+++KImT56sli1bauXKle51vvvuOz3yyCPatWuXXnnlFU2ePFmBgYHq0KGDli1b5u2nGbgzGQAe/vrXv5oyZcqYy5cvu5f9/PPPxsfHx4wZM8a97MKFC3m2XbBggZFkNm7c6F6WnJxsJJmDBw9e97gJCQlGkjlx4kS+j2dmZhpJ5plnnrnuDImJicbhcJgff/zRvax///7mWl/uV+/j4sWLpk6dOubxxx+/7rzGGBMbG2sCAwM9lkVERBhJ5vPPP3cvW7t2rZFk/P39PeaaPXu2kWQ2bNjgsU9JZuDAge5lLpfLtGvXzhQvXtzj83Ojs0dERJjY2Fj3/V9//dXk5OR4rHPw4EHjdDo9/htv2LDBSDIPPPCAyc7Odi+fNm2akWTS0tKMMcZcvnzZVKlSxURERJjMzEyP/bpcLve/W7VqZaKiosyvv/7q8XiTJk3M/fffb4C7Ac9YAFfp0qWLMjIy3E/dS7+dInG5XOrSpYt7mb+/v/vfv/76q06ePKlHHnlEkvTVV19Znyv31Rdnz57Nd4bz58/r5MmTatKkiYwx1zzFcLUr95GZmakzZ86oWbNmN/Ux1KpVS48++qj7fuPGjSVJjz/+uCpXrpxn+YEDB/LsY8CAAe5/OxwODRgwQBcvXtRHH31007M7nU75+Pz27S8nJ0enTp1SUFCQatSoke+2PXv2VPHixd33mzVr5jH3jh07dPDgQQ0ePFglSpTw2Db3tNPp06f18ccfq3Pnzjp79qxOnjypkydP6tSpU4qOjtbevXt15MiR684NFAVcvAlcpW3btgoNDdWiRYvUqlUrSb+dBqlXr56qV6/uXu/06dMaPXq0Fi5cqIyMDI99nDlzxvpc586dkyQFBwe7lx0+fFgjR47UihUrlJmZWaAZVq5cqX/961/6+uuvPV7SeuV1GN66Mh4kKTQ0VJIUHh6e7/KrZ/fx8dF9993nsSz3c3/l3wMp6Owul0vTpk3TzJkzdfDgQeXk5LgfCwsL+92Pp2TJkh5z79+/X5Lcp8nys2/fPhljNGLECI0YMSLfdTIyMlSxYsXrzg7c6QgL4CpOp9N9znvmzJk6fvy4Nm3apPHjx3us17lzZ33++ecaPny46tWrp6CgILlcLrVt2zbPNQM25F40Wq1aNUm//abdpk0bnT59Wi+//LJq1qypwMBAHTlyRHFxcTc0w6effqo///nPat68uWbOnKny5curWLFiSk5O1vz58ws8q6+vr1fLTQGuSbmZ2cePH68RI0aoV69eGjt2rEqVKiUfHx8NHjw438+bjblz9zts2DBFR0fnu07uf1ugKCMsgHx06dJFKSkpWr9+vXbt2iVjjMdpkMzMTK1fv16jR4/WyJEj3cv37t1baDPlXnCZ+0MpLS1Ne/bsUUpKisfFmle+aiTXtX6Df++99+Tn56e1a9fK6XS6lycnJ9sc3Wsul0sHDhzweIZoz549kn571Yl0c7MvXbpULVu21DvvvOOx/JdffvF4qe+Nqlq1qqTf4q9169b5rpP7DEyxYsWuuQ5wN+AaCyAfrVu3VqlSpbRo0SItWrRIDz/8sKpUqeJ+PPc32Kt/Y506dWqhzDN//nz9+9//1qOPPuo+PZPfDMaYPC9vlKTAwEBJv/3gvJKvr68cDofHqYBDhw5p+fLllj8C702fPt39b2OMpk+frmLFinl8/AWd3dfXN89/uyVLlhT4GocGDRqoSpUqmjp1ap7Pce5xypQpo8cee0yzZ8/Wzz//nGcfJ06cKNCxgTsNz1gA+ShWrJj+8pe/aOHChTp//rwmTZrk8XhISIiaN2+uV199VZcuXVLFihX14Ycf6uDBgzd97KVLlyooKEgXL17UkSNHtHbtWm3atEl169bVkiVL3OvVrFlTVatW1bBhw3TkyBGFhITovffey3O9giQ1bNhQkjRo0CBFR0fL19dXXbt2Vbt27TRlyhS1bdtW3bp1U0ZGhmbMmKFq1arpm2++uemPpaD8/Py0Zs0axcbGqnHjxvrggw+0atUq/fOf/1Tp0qUl6aZmj4mJ0ZgxY9SzZ081adJEaWlpmjdvXp7rOm6Uj4+PZs2apfbt26tevXrq2bOnypcvr927d+u7777T2rVrJUkzZszQn/70J0VFRal379667777dPz4cW3evFk//fRTnr+jARRJt+nVKMAdb926dUaScTgcJj09Pc/jP/30k3nmmWdMiRIlTGhoqOnUqZM5evSokWQSEhLc63n7ctPcm5+fn6lUqZKJiYkx7777rsdLFHN9//33pnXr1iYoKMjce++9pnfv3mbnzp1GkklOTnavd/nyZTNw4EBTunRp43A4PF56+s4775j777/fOJ1OU7NmTZOcnOye5fdc6+Wm7dq1y7OuJNO/f3+PZQcPHjSSzGuvvZZnn/v37zdPPPGECQgIMGXLljUJCQl5XiJ6o7Pn93LToUOHmvLlyxt/f3/TtGlTs3nzZtOiRQvTokUL93q5LzddsmRJvnNf+Tk2xpjPPvvMtGnTxgQHB5vAwEDz4IMPmqSkJI919u/fb3r06GHKlStnihUrZipWrGhiYmLM0qVL83zOgKLIYUwh/CUfAADwh8Q1FgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACsISwAAIA1t/wPZLlcLh09elTBwcE39SZHAADg1jHG6OzZs6pQoYL73YHzc8vD4ujRo3ne4RAAABQN6enpqlSp0jUfv+VhkfuWz+np6QoJCbnVhwcAAAWQlZWl8PBw98/xa7nlYZF7+iMkJISwAACgiPm9yxi4eBMAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACw5pa/bfqt0HD4/93uEYA70vbXetzuEW7a4TFRt3sE4I5UeWTa7R5BEs9YAAAAiwgLAABgDWEBAACsISwAAIA1hAUAALCGsAAAANYQFgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACsISwAAIA1hAUAALCGsAAAANYQFgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACsISwAAIA1hAUAALCGsAAAANYQFgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACsISwAAIA1hAUAALCGsAAAANYQFgAAwBrCAgAAWENYAAAAawgLAABgDWEBAACs8SoscnJyNGLECFWpUkX+/v6qWrWqxo4dK2NMYc0HAACKkHu8WXnixImaNWuWUlJSVLt2bX355Zfq2bOnQkNDNWjQoMKaEQAAFBFehcXnn3+up59+Wu3atZMkRUZGasGCBdq6des1t8nOzlZ2drb7flZWVgFHBQAAdzqvToU0adJE69ev1549eyRJO3fu1GeffaYnn3zymtskJiYqNDTUfQsPD7+5iQEAwB3Lq2csXnnlFWVlZalmzZry9fVVTk6Oxo0bp+7du19zm/j4eA0ZMsR9Pysri7gAAOAu5VVYLF68WPPmzdP8+fNVu3Ztff311xo8eLAqVKig2NjYfLdxOp1yOp1WhgUAAHc2r8Ji+PDheuWVV9S1a1dJUlRUlH788UclJiZeMywAAMAfh1fXWFy4cEE+Pp6b+Pr6yuVyWR0KAAAUTV49Y9G+fXuNGzdOlStXVu3atbVjxw5NmTJFvXr1Kqz5AABAEeJVWCQlJWnEiBF64YUXlJGRoQoVKqhv374aOXJkYc0HAACKEK/CIjg4WFOnTtXUqVMLaRwAAFCU8V4hAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrvA6LI0eO6Nlnn1VYWJj8/f0VFRWlL7/8sjBmAwAARcw93qycmZmppk2bqmXLlvrggw9UunRp7d27VyVLliys+QAAQBHiVVhMnDhR4eHhSk5Odi+rUqWK9aEAAEDR5NWpkBUrVqhRo0bq1KmTypQpo/r16+vtt9++7jbZ2dnKysryuAEAgLuTV2Fx4MABzZo1S/fff7/Wrl2rfv36adCgQUpJSbnmNomJiQoNDXXfwsPDb3poAABwZ/IqLFwulxo0aKDx48erfv366tOnj3r37q0333zzmtvEx8frzJkz7lt6evpNDw0AAO5MXoVF+fLlVatWLY9lDzzwgA4fPnzNbZxOp0JCQjxuAADg7uRVWDRt2lQ//PCDx7I9e/YoIiLC6lAAAKBo8iosXnrpJW3ZskXjx4/Xvn37NH/+fL311lvq379/Yc0HAACKEK/C4qGHHtKyZcu0YMEC1alTR2PHjtXUqVPVvXv3wpoPAAAUIV79HQtJiomJUUxMTGHMAgAAijjeKwQAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYM1NhcWECRPkcDg0ePBgS+MAAICirMBhsW3bNs2ePVsPPvigzXkAAEARVqCwOHfunLp37663335bJUuWtD0TAAAoogoUFv3791e7du3UunXr3103OztbWVlZHjcAAHB3usfbDRYuXKivvvpK27Ztu6H1ExMTNXr0aK8HAwAARY9Xz1ikp6frxRdf1Lx58+Tn53dD28THx+vMmTPuW3p6eoEGBQAAdz6vnrHYvn27MjIy1KBBA/eynJwcbdy4UdOnT1d2drZ8fX09tnE6nXI6nXamBQAAdzSvwqJVq1ZKS0vzWNazZ0/VrFlTL7/8cp6oAAAAfyxehUVwcLDq1KnjsSwwMFBhYWF5lgMAgD8e/vImAACwxutXhVwtNTXVwhgAAOBuwDMWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrvAqLxMREPfTQQwoODlaZMmXUoUMH/fDDD4U1GwAAKGK8CotPPvlE/fv315YtW7Ru3TpdunRJTzzxhM6fP19Y8wEAgCLkHm9WXrNmjcf9OXPmqEyZMtq+fbuaN29udTAAAFD0eBUWVztz5owkqVSpUtdcJzs7W9nZ2e77WVlZN3NIAABwByvwxZsul0uDBw9W06ZNVadOnWuul5iYqNDQUPctPDy8oIcEAAB3uAKHRf/+/fXtt99q4cKF110vPj5eZ86ccd/S09MLekgAAHCHK9CpkAEDBmjlypXauHGjKlWqdN11nU6nnE5ngYYDAABFi1dhYYzRwIEDtWzZMqWmpqpKlSqFNRcAACiCvAqL/v37a/78+frPf/6j4OBgHTt2TJIUGhoqf3//QhkQAAAUHV5dYzFr1iydOXNGjz32mMqXL+++LVq0qLDmAwAARYjXp0IAAACuhfcKAQAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYQ1gAAABrCAsAAGANYQEAAKwhLAAAgDWEBQAAsIawAAAA1hAWAADAGsICAABYU6CwmDFjhiIjI+Xn56fGjRtr69attucCAABFkNdhsWjRIg0ZMkQJCQn66quvVLduXUVHRysjI6Mw5gMAAEWI12ExZcoU9e7dWz179lStWrX05ptvKiAgQO+++25hzAcAAIqQe7xZ+eLFi9q+fbvi4+Pdy3x8fNS6dWtt3rw5322ys7OVnZ3tvn/mzBlJUlZWVkHmvSE52f8ttH0DRVlhft3dKmd/zbndIwB3pML++s7dvzHmuut5FRYnT55UTk6OypYt67G8bNmy2r17d77bJCYmavTo0XmWh4eHe3NoABaEJv39do8AoLAkht6Sw5w9e1ahodc+lldhURDx8fEaMmSI+77L5dLp06cVFhYmh8NR2IfHbZaVlaXw8HClp6crJCTkdo8DwCK+vv9YjDE6e/asKlSocN31vAqLe++9V76+vjp+/LjH8uPHj6tcuXL5buN0OuV0Oj2WlShRwpvD4i4QEhLCNx7gLsXX9x/H9Z6pyOXVxZvFixdXw4YNtX79evcyl8ul9evX69FHH/V+QgAAcFfx+lTIkCFDFBsbq0aNGunhhx/W1KlTdf78efXs2bMw5gMAAEWI12HRpUsXnThxQiNHjtSxY8dUr149rVmzJs8FnYD026mwhISEPKfDABR9fH0jPw7ze68bAQAAuEG8VwgAALCGsAAAANYQFgAAwBrCAgAAWENYoEhKTU2Vw+HQL7/8crtHAXALREZGaurUqbd7DNwAwgKKi4uTw+HQhAkTPJYvX76cP7sO3CFyv04dDoeKFy+uatWqacyYMbp8+fLtHu2W2LZtm/r06XO7x8ANICwgSfLz89PEiROVmZlpbZ8XL160ti8AUtu2bfXzzz9r7969Gjp0qEaNGqXXXnvtdo91S5QuXVoBAQG3ewzcAMICkqTWrVurXLlySkxMvOY67733nmrXri2n06nIyEhNnjzZ4/HIyEiNHTtWPXr0UEhIiPr06aM5c+aoRIkSWrlypWrUqKGAgAB17NhRFy5cUEpKiiIjI1WyZEkNGjRIOTn/ezvsuXPnqlGjRgoODla5cuXUrVs3ZWRkFNrHDxQFTqdT5cqVU0REhPr166fWrVtrxYoViouLU4cOHTRp0iSVL19eYWFh6t+/vy5duuTeNjs7W8OGDVPFihUVGBioxo0bKzU11f34qFGjVK9ePY/jTZ06VZGRke77uccZP368ypYtqxIlSrifNRk+fLhKlSqlSpUqKTk52WM/aWlpevzxx+Xv76+wsDD16dNH586dy7Pf681/9amQKVOmKCoqSoGBgQoPD9cLL7zgsU/cPoQFJEm+vr4aP368kpKS9NNPP+V5fPv27ercubO6du2qtLQ0jRo1SiNGjNCcOXM81ps0aZLq1q2rHTt2aMSIEZKkCxcu6I033tDChQu1Zs0apaam6plnntHq1au1evVqzZ07V7Nnz9bSpUvd+7l06ZLGjh2rnTt3avny5Tp06JDi4uIK81MAFDn+/v7uZwY3bNig/fv3a8OGDUpJSdGcOXM8vj4HDBigzZs3a+HChfrmm2/UqVMntW3bVnv37vXqmB9//LGOHj2qjRs3asqUKUpISFBMTIxKliypL774Qn//+9/Vt29f9/eR8+fPKzo6WiVLltS2bdu0ZMkSffTRRxowYIDHfn9v/qv5+PjojTfe0HfffaeUlBR9/PHH+sc//uHVx4JCYvCHFxsba55++mljjDGPPPKI6dWrlzHGmGXLlpnc/0W6detm2rRp47Hd8OHDTa1atdz3IyIiTIcOHTzWSU5ONpLMvn373Mv69u1rAgICzNmzZ93LoqOjTd++fa8547Zt24wk9zYbNmwwkkxmZqb3HzBQBF35depyucy6deuM0+k0w4YNM7GxsSYiIsJcvnzZvX6nTp1Mly5djDHG/Pjjj8bX19ccOXLEY5+tWrUy8fHxxhhjEhISTN26dT0ef/31101ERITHDBERESYnJ8e9rEaNGqZZs2bu+5cvXzaBgYFmwYIFxhhj3nrrLVOyZElz7tw59zqrVq0yPj4+5tixYx77vdb8xvz2/eX111+/5udnyZIlJiws7JqP49bhGQt4mDhxolJSUrRr1y6P5bt27VLTpk09ljVt2lR79+71OIXRqFGjPPsMCAhQ1apV3ffLli2ryMhIBQUFeSy78lTH9u3b1b59e1WuXFnBwcFq0aKFJOnw4cM39wECRdjKlSsVFBQkPz8/Pfnkk+rSpYtGjRolSapdu7Z8fX3d65YvX979NZWWlqacnBxVr15dQUFB7tsnn3yi/fv3ezVD7dq15ePzvx8dZcuWVVRUlPu+r6+vwsLC3MfetWuX6tatq8DAQPc6TZs2lcvl0g8//OCx32vNn5+PPvpIrVq1UsWKFRUcHKznnntOp06d0oULF7z6eGCf129Chrtb8+bNFR0drfj4+AKderjym0euYsWKedx3OBz5LnO5XJL+99RpdHS05s2bp9KlS+vw4cOKjo7mglD8obVs2VKzZs1S8eLFVaFCBd1zz/++hV/va+rcuXPy9fXV9u3bPX54S3IHvo+Pj8xVbx115TUO1zvO9Y59o7zZx6FDhxQTE6N+/fpp3LhxKlWqlD777DP97W9/08WLF7nI8zYjLJDHhAkTVK9ePdWoUcO97IEHHtCmTZs81tu0aZOqV6+e5xvVzdq9e7dOnTqlCRMmKDw8XJL05ZdfWj0GUBQFBgaqWrVqXm9Xv3595eTkKCMjQ82aNct3ndKlS+vYsWMyxrhfZv7111/fzLiSfvveMWfOHJ0/f979i8emTZvk4+Pj8T3GG9u3b5fL5dLkyZPdz54sXrz4pmeFHZwKQR5RUVHq3r273njjDfeyoUOHav369Ro7dqz27NmjlJQUTZ8+XcOGDbN+/MqVK6t48eJKSkrSgQMHtGLFCo0dO9b6cYA/iurVq6t79+7q0aOH3n//fR08eFBbt25VYmKiVq1aJUl67LHHdOLECb366qvav3+/ZsyYoQ8++OCmj929e3f5+fkpNjZW3377rTZs2KCBAwfqueeeU9myZQu0z2rVqunSpUvu7xFz587Vm2++edOzwg7CAvkaM2aMx9OQDRo00OLFi7Vw4ULVqVNHI0eO1JgxYwrllRqlS5fWnDlztGTJEtWqVUsTJkzQpEmTrB8H+CNJTk5Wjx49NHToUNWoUUMdOnTQtm3bVLlyZUm/PbMwc+ZMzZgxQ3Xr1tXWrVut/OIQEBCgtWvX6vTp03rooYfUsWNHtWrVStOnTy/wPuvWraspU6Zo4sSJqlOnjubNm3fdl8rj1nKYq0+qAQAAFBDPWAAAAGsICwAAYA1hAQAArCEsAACANYQFAACwhrAAAADWEBYAAMAawgIAAFhDWAAAAGsICwAAYA1hAQAArPl/nKQvNkgeYe4AAAAASUVORK5CYII=", 237 | "text/plain": [ 238 | "
" 239 | ] 240 | }, 241 | "metadata": {}, 242 | "output_type": "display_data" 243 | } 244 | ], 245 | "source": [ 246 | "distribution('val')" 247 | ] 248 | }, 249 | { 250 | "cell_type": "markdown", 251 | "id": "41bfef58", 252 | "metadata": { 253 | "id": "41bfef58" 254 | }, 255 | "source": [ 256 | "Like many datasets in medical applications, this data has a class imbalance problem. Another cause of the concern is the presence of very few images validation set. The validation dataset consists of only 9 images for Pneumonia class and another 9 for the Normal class. This is not a sufficient number to adequately validate the model. We have to address these and other issues before proceeding with training our model.\n", 257 | "\n", 258 | "> We will manually transfer 461 Normal and 498 Pneumonia unique images from training folder to the validation folder. \n", 259 | " \n", 260 | "*Download and place the preprocessed dataset in a folder named `chest_xray_preprocessed`. The preprocessed dataset can be accessed from [here](https://drive.google.com/drive/folders/1-hvJdh-tONZHYFVdA3ttpB_yuRusZK5C?usp=sharing).*\n", 261 | " " 262 | ] 263 | }, 264 | { 265 | "cell_type": "markdown", 266 | "id": "fd6f56a5", 267 | "metadata": { 268 | "id": "fd6f56a5" 269 | }, 270 | "source": [ 271 | "## 3. Oversampling the data\n", 272 | "\n", 273 | "We'll now augment the remaining training set of the preprocessed dataset by oversampling the minority class inorder to balance the classes" 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "execution_count": 32, 279 | "id": "ef39f361", 280 | "metadata": { 281 | "id": "ef39f361" 282 | }, 283 | "outputs": [], 284 | "source": [ 285 | "TRAIN_DIR = 'chest_xray_preprocessed/train'\n", 286 | "\n", 287 | "IMAGE_SIZE = 224 # Image size of resize when applying transforms.\n", 288 | "BATCH_SIZE = 32\n", 289 | "NUM_WORKERS = 4 # Number of parallel processes for data preparation.\n", 290 | "def get_augmented_data():\n", 291 | " sample1 = ImageFolder(TRAIN_DIR, \n", 292 | " transform = transforms.Compose([transforms.Resize((224,224)),\n", 293 | " transforms.RandomRotation(10),\n", 294 | " transforms.RandomGrayscale(),\n", 295 | " transforms.RandomAffine(translate=(0.05,0.05), degrees=0),\n", 296 | " transforms.ToTensor(),\n", 297 | " transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n", 298 | " ]))\n", 299 | "\n", 300 | " sample2 = ImageFolder(TRAIN_DIR, \n", 301 | " transform=transforms.Compose([transforms.Resize((224,224)),\n", 302 | " transforms.RandomGrayscale(),\n", 303 | " transforms.RandomAffine(translate=(0.1,0.05), degrees=10),\n", 304 | " \n", 305 | " transforms.ToTensor(),\n", 306 | " transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n", 307 | " ]))\n", 308 | "\n", 309 | " sample3 = ImageFolder(TRAIN_DIR, \n", 310 | " transforms.Compose([transforms.Resize((224,224)),\n", 311 | " transforms.RandomRotation(15),\n", 312 | " transforms.RandomGrayscale(p=1),\n", 313 | " transforms.RandomAffine(translate=(0.08,0.1), degrees=15),\n", 314 | " transforms.ToTensor(),\n", 315 | " transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n", 316 | " ]))\n", 317 | " sample4 = ImageFolder(TRAIN_DIR, \n", 318 | " transforms.Compose([transforms.Resize((224,224)),\n", 319 | " transforms.RandomRotation(15),\n", 320 | " transforms.RandomGrayscale(p=1),\n", 321 | " transforms.RandomAffine(translate=(0.09,0.2), degrees=17),\n", 322 | " transforms.ToTensor(),\n", 323 | " transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n", 324 | " ]))\n", 325 | "\n", 326 | "\n", 327 | " normal_1, _ = train_test_split(sample2, test_size= 3377/(880+3377), shuffle=False)\n", 328 | " normal_2, _ = train_test_split(sample3, test_size= 3377/(880+3377), shuffle=False)\n", 329 | " normal_3, _ = train_test_split(sample4, test_size= 3000/(880+3377), shuffle=False)\n", 330 | "\n", 331 | " train_dataset = ConcatDataset([sample1, normal_1, normal_2, normal_3])\n", 332 | " return train_dataset" 333 | ] 334 | }, 335 | { 336 | "cell_type": "code", 337 | "execution_count": 34, 338 | "id": "93df4eda", 339 | "metadata": { 340 | "id": "93df4eda" 341 | }, 342 | "outputs": [], 343 | "source": [ 344 | "train_dataset = get_augmented_data()" 345 | ] 346 | }, 347 | { 348 | "cell_type": "code", 349 | "execution_count": 35, 350 | "id": "647e1a6c", 351 | "metadata": { 352 | "id": "647e1a6c" 353 | }, 354 | "outputs": [], 355 | "source": [ 356 | "# Saving the augmented training set images\n", 357 | "torch.save(train_dataset,'train_dataset.pt')" 358 | ] 359 | }, 360 | { 361 | "cell_type": "markdown", 362 | "id": "e89d79ce", 363 | "metadata": { 364 | "id": "e89d79ce" 365 | }, 366 | "source": [ 367 | "Sanity Check to see if everything works as proposed" 368 | ] 369 | }, 370 | { 371 | "cell_type": "code", 372 | "execution_count": 36, 373 | "id": "c1701d59", 374 | "metadata": { 375 | "id": "c1701d59", 376 | "outputId": "8c150459-a10c-4d76-fe8e-56cdc2f07c4c" 377 | }, 378 | "outputs": [ 379 | { 380 | "name": "stdout", 381 | "output_type": "stream", 382 | "text": [ 383 | "Normal : 3516 and Pneumonia : 3758\n" 384 | ] 385 | } 386 | ], 387 | "source": [ 388 | "pneumonia = 0\n", 389 | "normal = 0\n", 390 | "for i in range(len(train_dataset)):\n", 391 | " if train_dataset[i][1]==1:\n", 392 | " pneumonia += 1\n", 393 | " else:\n", 394 | " normal += 1\n", 395 | " \n", 396 | "print(f'Normal : {normal} and Pneumonia : {pneumonia}')" 397 | ] 398 | }, 399 | { 400 | "attachments": {}, 401 | "cell_type": "markdown", 402 | "id": "c4a07f26", 403 | "metadata": { 404 | "id": "c4a07f26" 405 | }, 406 | "source": [ 407 | "Another preprocessing technique that we can use for the dataset is image cropping. We\n", 408 | "crop some of the images in the training set so as to highlight only the lung region.\n", 409 | "Cropping also helped to eliminate any annotation markers or any other markings on\n", 410 | "the Chest X-Ray images and only focussing on the region of interest in the images.\n", 411 | "We saved these images as a separate dataset to be used to fine tune the network during\n", 412 | "the second stage of transfer learning. \n", 413 | "\n", 414 | "The **Cropped Dataset** can be accessed from [here](https://drive.google.com/drive/folders/15C9RiGnjYpISCPBkoLrHhEzMHtRh7E7J?usp=sharing)" 415 | ] 416 | }, 417 | { 418 | "cell_type": "markdown", 419 | "id": "df38d4fd", 420 | "metadata": { 421 | "id": "df38d4fd" 422 | }, 423 | "source": [ 424 | "## 4. Loading the Cropped Dataset" 425 | ] 426 | }, 427 | { 428 | "cell_type": "code", 429 | "execution_count": 37, 430 | "id": "ad7cc68e", 431 | "metadata": { 432 | "id": "ad7cc68e" 433 | }, 434 | "outputs": [], 435 | "source": [ 436 | "cropped_ds = ImageFolder('Cropped', \n", 437 | " transform = transforms.Compose([transforms.Resize((224,224)),\n", 438 | " transforms.RandomRotation(10),\n", 439 | " transforms.RandomGrayscale(),\n", 440 | " transforms.RandomAffine(translate=(0.05,0.05), degrees=0),\n", 441 | " transforms.ToTensor(),\n", 442 | " transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n", 443 | " ]))\n" 444 | ] 445 | }, 446 | { 447 | "cell_type": "code", 448 | "execution_count": 38, 449 | "id": "bd7f5d42", 450 | "metadata": { 451 | "id": "bd7f5d42", 452 | "outputId": "44f42be6-941e-41e2-f689-353a4759c024" 453 | }, 454 | "outputs": [ 455 | { 456 | "data": { 457 | "text/plain": [ 458 | "Dataset ImageFolder\n", 459 | " Number of datapoints: 422\n", 460 | " Root location: Cropped\n", 461 | " StandardTransform\n", 462 | "Transform: Compose(\n", 463 | " Resize(size=(224, 224), interpolation=bilinear, max_size=None, antialias=None)\n", 464 | " RandomRotation(degrees=[-10.0, 10.0], interpolation=nearest, expand=False, fill=0)\n", 465 | " RandomGrayscale(p=0.1)\n", 466 | " RandomAffine(degrees=[0.0, 0.0], translate=(0.05, 0.05))\n", 467 | " ToTensor()\n", 468 | " Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n", 469 | " )" 470 | ] 471 | }, 472 | "execution_count": 38, 473 | "metadata": {}, 474 | "output_type": "execute_result" 475 | } 476 | ], 477 | "source": [ 478 | "cropped_ds" 479 | ] 480 | }, 481 | { 482 | "cell_type": "code", 483 | "execution_count": null, 484 | "id": "f9e8cbe5-d957-4bb8-9f36-a0d7b3b5cd11", 485 | "metadata": {}, 486 | "outputs": [], 487 | "source": [] 488 | } 489 | ], 490 | "metadata": { 491 | "colab": { 492 | "include_colab_link": true, 493 | "provenance": [] 494 | }, 495 | "kernelspec": { 496 | "display_name": "Python 3 (ipykernel)", 497 | "language": "python", 498 | "name": "python3" 499 | }, 500 | "language_info": { 501 | "codemirror_mode": { 502 | "name": "ipython", 503 | "version": 3 504 | }, 505 | "file_extension": ".py", 506 | "mimetype": "text/x-python", 507 | "name": "python", 508 | "nbconvert_exporter": "python", 509 | "pygments_lexer": "ipython3", 510 | "version": "3.10.8" 511 | } 512 | }, 513 | "nbformat": 4, 514 | "nbformat_minor": 5 515 | } 516 | -------------------------------------------------------------------------------- /code/Chapter-7 & 9/3.Transfer learning-Stage_2.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "id": "8b0c8670", 7 | "metadata": {}, 8 | "source": [ 9 | "# Chapter 7: Explaining a PyTorch Image Classifier\n", 10 | "\n", 11 | "## **Transfer Learning Stage 2**" 12 | ] 13 | }, 14 | { 15 | "attachments": {}, 16 | "cell_type": "markdown", 17 | "id": "7a4690cd", 18 | "metadata": {}, 19 | "source": [ 20 | "## Setting the environment\n", 21 | "\n", 22 | "If you are using Colab, it comes preinstalled with PyTorch and other commonly used libraries for machine and Deep learning. If you are executing this notebook in your local system, you will need to install them manually via the following commands:" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "id": "46d89428", 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "#!pip3 install torch torchvision numpy pandas matplotlib seaborn" 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": 1, 38 | "id": "3cbe0711", 39 | "metadata": {}, 40 | "outputs": [], 41 | "source": [ 42 | "import numpy as np \n", 43 | "import pandas as pd \n", 44 | "import matplotlib.pyplot as plt\n", 45 | "from matplotlib.image import imread\n", 46 | "import seaborn as sns\n", 47 | "import random\n", 48 | "\n", 49 | "\n", 50 | "import copy\n", 51 | "import os\n", 52 | "from sklearn.model_selection import train_test_split,StratifiedShuffleSplit\n", 53 | "from sklearn.metrics import confusion_matrix,ConfusionMatrixDisplay, classification_report\n", 54 | "from skimage.util import random_noise\n", 55 | "import time\n", 56 | "\n", 57 | "\n", 58 | "import torch\n", 59 | "import torch.nn as nn\n", 60 | "import torch.nn.functional as F\n", 61 | "import torchvision\n", 62 | "import torchvision.models as models\n", 63 | "from torchvision.datasets import ImageFolder\n", 64 | "from torchvision.utils import make_grid, save_image\n", 65 | "import torchvision.transforms as transforms\n", 66 | "from torch.utils.data import Dataset, DataLoader, ConcatDataset\n", 67 | "from PIL import Image\n", 68 | "\n", 69 | "from mlxtend.plotting import plot_confusion_matrix\n", 70 | "from sklearn.metrics import confusion_matrix\n", 71 | "\n", 72 | "# Seed\n", 73 | "seed = 123\n", 74 | "torch.manual_seed(seed)\n", 75 | "torch.cuda.manual_seed(seed)\n", 76 | "torch.cuda.manual_seed_all(seed)\n", 77 | "np.random.seed(seed)\n", 78 | "random.seed(seed)\n", 79 | "torch.backends.cudnn.benchmark = False\n", 80 | "torch.backends.cudnn.deterministic = True\n", 81 | "\n", 82 | "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n", 83 | "random_seed = 12345" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 2, 89 | "id": "a73f9c43", 90 | "metadata": {}, 91 | "outputs": [ 92 | { 93 | "data": { 94 | "text/plain": [ 95 | "" 96 | ] 97 | }, 98 | "execution_count": 2, 99 | "metadata": {}, 100 | "output_type": "execute_result" 101 | } 102 | ], 103 | "source": [ 104 | "def seed_worker(worker_id):\n", 105 | " worker_seed = torch.initial_seed() % 2**32\n", 106 | " numpy.random.seed(worker_seed)\n", 107 | " random.seed(worker_seed)\n", 108 | "\n", 109 | "g = torch.Generator()\n", 110 | "g.manual_seed(0)" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "id": "87d9fb2c", 116 | "metadata": {}, 117 | "source": [ 118 | "## Loading the Cropped Dataset\n", 119 | "Refer to the `Data Preparation` notebook" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": 3, 125 | "id": "241236e4", 126 | "metadata": {}, 127 | "outputs": [ 128 | { 129 | "data": { 130 | "text/plain": [ 131 | "(422, 977)" 132 | ] 133 | }, 134 | "execution_count": 3, 135 | "metadata": {}, 136 | "output_type": "execute_result" 137 | } 138 | ], 139 | "source": [ 140 | "cropped_train_ds = ImageFolder('chest_xray_pre-processed/chest_xray/Cropped', \n", 141 | " transform = transforms.Compose([transforms.Resize((224,224)),\n", 142 | " transforms.RandomRotation(10),\n", 143 | " transforms.RandomGrayscale(),\n", 144 | " transforms.RandomAffine(translate=(0.05,0.05), degrees=0),\n", 145 | " transforms.ToTensor(),\n", 146 | " transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n", 147 | " ]))\n", 148 | "val_ds = ImageFolder('chest_xray_pre-processed/chest_xray/val', \n", 149 | " transform = transforms.Compose([transforms.Resize((224,224)),\n", 150 | " transforms.ToTensor(),\n", 151 | " transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n", 152 | " ]))\n", 153 | "\n", 154 | "len(cropped_train_ds), len(val_ds)\n" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": 4, 160 | "id": "0db8d417", 161 | "metadata": {}, 162 | "outputs": [], 163 | "source": [ 164 | "batch_size=32\n", 165 | "\n", 166 | "cropped_train_dl = DataLoader(cropped_train_ds, batch_size, shuffle=True, worker_init_fn=seed_worker)\n", 167 | "val_dl = DataLoader(val_ds, batch_size, worker_init_fn=seed_worker)\n", 168 | "loaders = {'train':cropped_train_dl, 'val':val_dl}\n", 169 | "dataset_sizes = {'train':len(cropped_train_ds), 'val':len(val_ds)}" 170 | ] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "execution_count": 38, 175 | "id": "6b499208", 176 | "metadata": {}, 177 | "outputs": [], 178 | "source": [ 179 | "model = torch.load('Finetuning_Stage1.pt')" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 39, 185 | "id": "b0736ee8", 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "for param in model.parameters():\n", 190 | " param.requires_grad = False" 191 | ] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": 40, 196 | "id": "4c535cea", 197 | "metadata": {}, 198 | "outputs": [], 199 | "source": [ 200 | "criterion = nn.CrossEntropyLoss()\n", 201 | "optimizer = torch.optim.Adam(model.parameters(), lr = 0.001)\n", 202 | "scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 4, gamma=0.1)" 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": 41, 208 | "id": "cd9985fd", 209 | "metadata": {}, 210 | "outputs": [], 211 | "source": [ 212 | "losses = {'train':[], 'val':[]}\n", 213 | "accuracies = {'train':[], 'val':[]}\n", 214 | "\n", 215 | "\n", 216 | "def train(model, criterion, optimizer, scheduler, epochs):\n", 217 | " since = time.time()\n", 218 | " best_model = copy.deepcopy(model.state_dict())\n", 219 | " best_acc = 0.0\n", 220 | " for epoch in range(epochs):\n", 221 | " for phase in ['train', 'val']:\n", 222 | " if phase == 'train':\n", 223 | " model.train()\n", 224 | " else:\n", 225 | " model.eval()\n", 226 | " \n", 227 | " running_loss = 0.0\n", 228 | " running_corrects = 0.0\n", 229 | "\n", 230 | " for inputs, labels in loaders[phase]:\n", 231 | " inputs, labels = inputs.to(device), labels.to(device)\n", 232 | "\n", 233 | " optimizer.zero_grad()\n", 234 | "\n", 235 | " with torch.set_grad_enabled(phase=='train'):\n", 236 | " outp = model(inputs)\n", 237 | " _, pred = torch.max(outp, 1)\n", 238 | " loss = criterion(outp, labels)\n", 239 | " loss.requires_grad = True\n", 240 | " \n", 241 | " if phase == 'train':\n", 242 | " loss.backward()\n", 243 | " optimizer.step()\n", 244 | " \n", 245 | " running_loss += loss.item()*inputs.size(0)\n", 246 | " running_corrects += torch.sum(pred == labels.data)\n", 247 | "\n", 248 | "\n", 249 | " epoch_loss = running_loss / dataset_sizes[phase]\n", 250 | " epoch_acc = running_corrects.double()/dataset_sizes[phase]\n", 251 | " losses[phase].append(epoch_loss)\n", 252 | " accuracies[phase].append(epoch_acc)\n", 253 | " if phase == 'train':\n", 254 | " print('Epoch: {}/{}'.format(epoch+1, epochs))\n", 255 | " print('{} - loss:{}, accuracy{}'.format(phase, epoch_loss, epoch_acc))\n", 256 | " \n", 257 | " \n", 258 | " if phase == 'val' and epoch_acc > best_acc:\n", 259 | " best_acc = epoch_acc\n", 260 | " best_model = copy.deepcopy(model.state_dict())\n", 261 | " scheduler.step() \n", 262 | " print('Best accuracy {}'.format(best_acc))\n", 263 | "\n", 264 | " model.load_state_dict(best_model)\n", 265 | " return model " 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": 42, 271 | "id": "a8cf8954", 272 | "metadata": {}, 273 | "outputs": [ 274 | { 275 | "name": "stdout", 276 | "output_type": "stream", 277 | "text": [ 278 | "Epoch: 1/17\n", 279 | "train - loss:0.16213886227604374, accuracy0.9360189573459716\n", 280 | "val - loss:0.05905295006321144, accuracy0.9815762538382805\n", 281 | "Epoch: 2/17\n", 282 | "train - loss:0.2181292337970146, accuracy0.9218009478672986\n", 283 | "val - loss:0.09303262661174307, accuracy0.9733879222108496\n", 284 | "Epoch: 3/17\n", 285 | "train - loss:0.17090152465336694, accuracy0.9312796208530807\n", 286 | "val - loss:0.09203716716300457, accuracy0.9723643807574207\n", 287 | "Epoch: 4/17\n", 288 | "train - loss:0.17591952452197743, accuracy0.9312796208530807\n", 289 | "val - loss:0.08194925019857345, accuracy0.9744114636642784\n", 290 | "Epoch: 5/17\n", 291 | "train - loss:0.1736818109572781, accuracy0.9241706161137442\n", 292 | "val - loss:0.07081017136284617, accuracy0.9785056294779939\n", 293 | "Epoch: 6/17\n", 294 | "train - loss:0.1652766755689377, accuracy0.9265402843601896\n", 295 | "val - loss:0.06797165959056645, accuracy0.9795291709314228\n", 296 | "Epoch: 7/17\n", 297 | "train - loss:0.17079675276498907, accuracy0.9454976303317536\n", 298 | "val - loss:0.08522375099469713, accuracy0.9744114636642784\n", 299 | "Epoch: 8/17\n", 300 | "train - loss:0.16720860655159112, accuracy0.9454976303317536\n", 301 | "val - loss:0.08462242321445095, accuracy0.9744114636642784\n", 302 | "Epoch: 9/17\n", 303 | "train - loss:0.1643253081076518, accuracy0.9360189573459716\n", 304 | "val - loss:0.0878552215935862, accuracy0.9733879222108496\n", 305 | "Epoch: 10/17\n", 306 | "train - loss:0.18233245928988073, accuracy0.9265402843601896\n", 307 | "val - loss:0.08086182687440344, accuracy0.9744114636642784\n", 308 | "Epoch: 11/17\n", 309 | "train - loss:0.18567422757993376, accuracy0.9241706161137442\n", 310 | "val - loss:0.06240757423052341, accuracy0.9815762538382805\n", 311 | "Epoch: 12/17\n", 312 | "train - loss:0.17861574831732077, accuracy0.9336492890995262\n", 313 | "val - loss:0.06742130681486361, accuracy0.9805527123848516\n", 314 | "Epoch: 13/17\n", 315 | "train - loss:0.1907900203354833, accuracy0.9336492890995262\n", 316 | "val - loss:0.0762084919125589, accuracy0.9744114636642784\n", 317 | "Epoch: 14/17\n", 318 | "train - loss:0.162779003488765, accuracy0.9407582938388627\n", 319 | "val - loss:0.08570337082003851, accuracy0.9733879222108496\n", 320 | "Epoch: 15/17\n", 321 | "train - loss:0.18446631447999115, accuracy0.9336492890995262\n", 322 | "val - loss:0.08623684697826443, accuracy0.9744114636642784\n", 323 | "Epoch: 16/17\n", 324 | "train - loss:0.1861739684496587, accuracy0.9383886255924171\n", 325 | "val - loss:0.07517468557685293, accuracy0.9754350051177073\n", 326 | "Epoch: 17/17\n", 327 | "train - loss:0.1731398391638887, accuracy0.9431279620853081\n", 328 | "val - loss:0.0682922940559919, accuracy0.9805527123848516\n", 329 | "Best accuracy 0.9815762538382805\n" 330 | ] 331 | } 332 | ], 333 | "source": [ 334 | "model.to(device)\n", 335 | "epochs = 17\n", 336 | "model = train(model, criterion, optimizer, scheduler, epochs)" 337 | ] 338 | }, 339 | { 340 | "cell_type": "code", 341 | "execution_count": 46, 342 | "id": "3f022b90", 343 | "metadata": {}, 344 | "outputs": [], 345 | "source": [ 346 | "torch.save(model, 'Finetuning_Stage2.pt')" 347 | ] 348 | }, 349 | { 350 | "cell_type": "markdown", 351 | "id": "7b98bfee", 352 | "metadata": {}, 353 | "source": [ 354 | "## Model Evaluation on Test Set" 355 | ] 356 | }, 357 | { 358 | "cell_type": "code", 359 | "execution_count": 47, 360 | "id": "e654c750", 361 | "metadata": {}, 362 | "outputs": [], 363 | "source": [ 364 | "testset = ImageFolder('chest_xray_pre-processed/chest_xray/test', \n", 365 | " transform=torchvision.transforms.Compose([torchvision.transforms.Resize((224,224)), \n", 366 | " torchvision.transforms.ToTensor(),\n", 367 | " torchvision.transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]), \n", 368 | " \n", 369 | " ]))" 370 | ] 371 | }, 372 | { 373 | "cell_type": "code", 374 | "execution_count": 48, 375 | "id": "c64639c6", 376 | "metadata": {}, 377 | "outputs": [], 378 | "source": [ 379 | "test_dl = DataLoader(testset, batch_size=256)\n", 380 | "model.to(device);\n" 381 | ] 382 | }, 383 | { 384 | "cell_type": "code", 385 | "execution_count": 49, 386 | "id": "beaf99af", 387 | "metadata": {}, 388 | "outputs": [], 389 | "source": [ 390 | "def accuracy(outputs, labels):\n", 391 | " _, preds = torch.max(outputs, dim=1) \n", 392 | " return torch.tensor(torch.sum(preds == labels).item() / len(preds)), preds\n", 393 | "\n", 394 | "def validation(batch):\n", 395 | " images,labels = batch\n", 396 | " images,labels = images.to(device),labels.to(device)\n", 397 | " output = model(images) \n", 398 | " loss = F.cross_entropy(output, labels) \n", 399 | " acc,predictions = accuracy(output, labels) \n", 400 | " \n", 401 | " return {'valid_loss': loss.detach(), 'valid_accuracy':acc.detach(), 'predictions':predictions.detach(), 'labels':labels.detach()}\n", 402 | " \n", 403 | "@torch.no_grad()\n", 404 | "def test_predict(model, test_dataloader):\n", 405 | " model.eval()\n", 406 | " \n", 407 | " outputs = [validation(batch) for batch in test_dataloader] \n", 408 | " batch_losses = [x['valid_loss'] for x in outputs]\n", 409 | " epoch_loss = torch.stack(batch_losses).mean() \n", 410 | " batch_accs = [x['valid_accuracy'] for x in outputs]\n", 411 | " epoch_acc = torch.stack(batch_accs).mean() \n", 412 | " \n", 413 | " batch_preds = [pred for x in outputs for pred in x['predictions'].tolist()] \n", 414 | " batch_labels = [label for x in outputs for label in x['labels'].tolist()] \n", 415 | " \n", 416 | " print('test_loss: {:.4f}, test_acc: {:.4f}'\n", 417 | " .format(epoch_loss.item(), epoch_acc.item()))\n", 418 | " \n", 419 | " return batch_preds, batch_labels " 420 | ] 421 | }, 422 | { 423 | "cell_type": "code", 424 | "execution_count": 50, 425 | "id": "0e4e2e1a", 426 | "metadata": {}, 427 | "outputs": [ 428 | { 429 | "name": "stdout", 430 | "output_type": "stream", 431 | "text": [ 432 | "test_loss: 0.2626, test_acc: 0.9334\n" 433 | ] 434 | } 435 | ], 436 | "source": [ 437 | "preds,labels = test_predict(model, test_dl)" 438 | ] 439 | }, 440 | { 441 | "cell_type": "code", 442 | "execution_count": 51, 443 | "id": "4901a11c", 444 | "metadata": {}, 445 | "outputs": [ 446 | { 447 | "data": { 448 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAFzCAYAAACXXrbmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAZSklEQVR4nO3df7TVdZ3v8ecLORdpIBCVlis0nOq6FERQxNREITOtWyvJZXmbHHK06d6VNXWtWbfphzk3bzk5/bBf4+2XzjJXOmFLs2UQ/kJTlAQEFMcyLX9UmoioiPz43D/29+CW4HiAc/jA2c/HWi729/vdP97HteF59nfv8zkppSBJUg2Dag8gSepcRkiSVI0RkiRVY4QkSdUYIUlSNUZIklTN4NoDDDTDhg0ro0aNqj2GxN577117BAmAhx56iCeeeCKbO2aE+tioUaP4xCc+UXsMiTPPPLP2CBIARx999BaPeTpOklSNEZIkVWOEJEnVGCFJUjVGSJJUjRGSJFVjhCRJ1RghSVI1RkiSVI0RkiRVY4QkSdUYIUlSNUZIklSNEZIkVWOEJEnVGCFJUjVGSJJUjRGSJFVjhCRJ1RghSVI1RkiSVI0RkiRVY4QkSdUYIUlSNUZIklSNEZIkVWOEJEnVGCFJUjVGSJJUjRGSJFVjhCRJ1RghSVI1RkiSVI0RkiRVY4QkSdUYIUlSNUZIklSNEZIkVWOEJEnVGCFJUjVGSJJUjRGSJFVjhCRJ1RghSVI1RkiSVI0RkiRVY4QkSdUYIUlSNUZIklSNEZIkVWOEJEnVGCFJUjVGSJJUjRGSJFVjhCRJ1RghSVI1RkiSVI0RkiRVY4QkSdUYIUlSNUZIklSNEZIkVWOEJEnVGCFJUjVGSJJUjRGSJFVjhCRJ1RghSVI1RkiSVI0RkiRVY4QkSdUYIUlSNUZIklSNEZIkVWOEJEnVDK49gDrP9OnTGTt2LKtXr+byyy8HYM8992TatGl0dXXx9NNPM3v2bNauXcugQYOYNm0ao0ePppTCvHnzeOSRRyp/BRqInn/+eY4//nheeOEF1q1bx8knn8ynP/1pzjrrLObNm8eIESMAuPjiiznkkEMqTztw7DIRSlKAfy2l/K9m+xxgWCnl3B04w43AOaWUBTvqMQei5cuXs2TJEo4//viN+6ZPn86tt97Ko48+yoEHHsihhx7K/PnzGTduHACXX345Q4cO5e1vfztXXHFFrdE1gA0ZMoTrrruOYcOGsXbtWqZPn84JJ5wAwPnnn8+MGTMqTzgw7Uqn49YAM5LstS03TrLLBHege/TRR3n++edfsm/kyJE8+uijAPz+97/nta99LQB77LEHDz/8MACrV69mzZo1jB49escOrI6QhGHDhgGwdu1a1q1bR5LKUw18u1KE1gEXAx/d9ECSsUmuT3J3krlJ9mv2/yDJt5PMBy5otr+V5PYkDyQ5Lsn3ktyb5Adt9/etJAuSLEvyuR31BXayJ598kv333x+A173udRv/Mfjzn//M/vvvTxKGDx/O6NGjGT58eM1RNYCtX7+eI444gv3224/p06czZcoUAM4991wOP/xwPv7xj7NmzZrKUw4su1KEAL4BvDfJiE32XwRcUkqZAFwGfK3t2BjgqFLKx5rtPYAjacXsauDLwDjg4CQTm+v8UyllMjABODbJhJ6GSvKBJloLnnnmmW3/6jrY3LlzOfjggzn11FPp6upiw4YNANxzzz0888wznHrqqRxzzDE89thjG49JfW233XZj/vz5/PrXv2bBggUsW7aM8847j8WLF3PLLbewYsUKLrzwwtpjDii7VIRKKU8DlwIf3uTQkcAPm8v/Dryx7diVpZT1bdvXlFIKsAT4YyllSSllA7AMGNtc59QkdwELaQXqoJeZ6+JSyuRSyuTu7+C1dZ566imuvvpqrrjiCu6//35WrlwJQCmFW265hR/96Ef87Gc/Y8iQITz11FN1h9WAN3LkSI499lhmz57NPvvsQxKGDBnC6aefzoIFviXcl3apCDW+Avwd8Fe9vP6zm2x3v5be0Ha5e3twkv2Bc4A3Na+srgV23+Zp1StDhw7deHny5MksXboUgMGDBzN4cOvtvH333ZcNGzawYsWKKjNqYHv88cc3foOzevVq5s6dywEHHMBjjz0GtL4huvrqqznooB6/J9VW2uXerC+lPJnkCloh+l6z+5fAe2i9CnovMG87HuKVtMK1MsmrgJOAG7fj/rSJE044gVe/+tXsvvvuzJw5k/nz59PV1cWECa2znr/5zW+49957gVac3vGOd1BK4dlnn+UXv/hFzdE1gP3hD3/grLPOYv369WzYsIF3vetdvPWtb+XEE0/kiSeeoJTChAkTuOiii2qPOqDschFqXAh8qG37bOD7ST4OPA68f1vvuJSyOMlCYDnwe+DW7RlUf2n27Nmb3X/33Xf/xb5Vq1Zx2WWX9fdIEgcffDC33377X+y/7rrrKkzTOXaZCJVShrVd/iPwirbth4Dpm7nNzC1tl1IeBMZv4dhLbte2/7itHlyStEW74ntCkqQBwghJkqoxQpKkaoyQJKkaIyRJqsYISZKqMUKSpGqMkCSpGiMkSarGCEmSqjFCkqRqjJAkqRojJEmqxghJkqoxQpKkaoyQJKkaIyRJqsYISZKqMUKSpGqMkCSpGiMkSarGCEmSqjFCkqRqjJAkqRojJEmqxghJkqoxQpKkaoyQJKkaIyRJqsYISZKqMUKSpGqMkCSpGiMkSarGCEmSqjFCkqRqjJAkqRojJEmqxghJkqoxQpKkaoyQJKkaIyRJqsYISZKqMUKSpGqMkCSpGiMkSarGCEmSqjFCkqRqjJAkqRojJEmqxghJkqoxQpKkaoyQJKmawVs6kOTQnm5YSrmr78eRJHWSLUYIuLCHYwWY3sezSJI6zBYjVEqZtiMHkSR1npd9TyjJK5J8KsnFzfbrk/y3/h9NkjTQ9eaDCd8HXgCOarYfAf5Pv00kSeoYvYnQa0spFwBrAUopzwHp16kkSR2hNxF6IclQWh9GIMlrgTX9OpUkqSP09Om4bp8FrgP2TXIZcDQwsz+HkiR1hpeNUCllTpK7gDfQOg33kVLKE/0+mSRpwOvNKyGAY4E30jol1wVc1W8TSZI6Rm8+ov1N4IPAEmAp8PdJvtHfg0mSBr7evBKaDhxYSun+YMIlwLJ+nUqS1BF68+m4XwP7tW3v2+yTJGm79LSA6TW03gMaDtyb5I5m+wjgjh0zniRpIOvpdNyXdtgUkqSO1NMCpjftyEEkSZ2nN5+Oe0OSO5M8k+SFJOuTPL0jhpMkDWy9+WDC14HTgPuBocCZgB/RliRtt179eu9Syq+B3Uop60sp3wdO7N+xJEmdoDc/J/Rckv8CLEpyAfAYvYyXJEk96U1M3tdc70PAs7R+TmhGfw4lSeoMvVnA9KHm4vPA5wCS/Ah4dz/OJUnqANt6Wu3IPp1CktSR0iwJt3U3Sn5XStnv5a/ZeSZPnlwWLFhQewyJxF+ArJ1HKWWzT8ielu05dEuHaP06B0mStktP7wld2MOx5X09iCSp8/S0bM+0HTmIJKnz+PM+kqRqjJAkqRojJEmqpjeraCfJ3yT5TLO9X5Ip/T+aJGmg680roW/S+uHU05rtVbiKtiSpD/RmAdMjSimHJlkIUEpZ0SxoKknSdunNK6G1SXYDCkCSvYEN/TqVJKkj9CZCXwOuAkYn+TxwC3B+v04lSeoIvVlF+7IkvwLeRGvJnneWUu7t98kkSQPey0YoyX7Ac8A17ftKKb/rz8EkSQNfbz6YcC2t94MC7A7sD9wHjOvHuSRJHaA3p+MObt9uVtf+n/02kSSpY2z1igmllLuAI/phFklSh+nNe0Ifa9scBBwKPNpvE0mSOkZv3hMa3nZ5Ha33iH7cP+NIkjpJjxFqfkh1eCnlnB00jySpg2zxPaEkg0sp64Gjd+A8kqQO0tMroTtovf+zKMnVwJXAs90HSymz+nk2SdIA15v3hHYH/gxM58WfFyqAEZIkbZeeIjS6+WTcUl6MT7fSr1NJkjpCTxHaDRjGS+PTzQhJkrZbTxF6rJRy3g6bRJLUcXpaMWFzr4AkSeozPUXoTTtsCklSR9pihEopT+7IQSRJnWerFzCVJKmvGCFJUjVGSJJUjRGSJFVjhCRJ1RghSVI1RkiSVI0RkiRVY4QkSdUYIUlSNUZIklSNEZIkVWOEJEnVGCFJUjVGSJJUjRGSJFVjhCRJ1RghSVI1RkiSVI0RkiRVY4QkSdUYIUlSNUZIklSNEZIkVWOEJEnVGCFJUjVGSJJUjRGSJFVjhCRJ1RghSVI1RkiSVI0RkiRVY4QkSdUYIUlSNUZIklSNEZIkVWOEJEnVGCFJUjVGSJJUjRGSJFVjhCRJ1RghSVI1RkiSVI0RkiRVY4QkSdUYIUlSNUZIklSNEZIkVWOEJEnVGCFVdcYZZzB69GjGjx+/cd+VV17JuHHjGDRoEAsWLKg4nQa6IUOGMH/+fBYtWsTSpUs599xzAbj55ptZuHAhCxcu5JFHHuGqq64CYOTIkcyaNYvFixczf/58xo0bV3H6gaHfIpRkfZJFSZYmuTLJK/rrsfpSkslJvlZ7jk4xc+ZMrrvuupfsGz9+PLNmzWLq1KmVplKnWLNmDdOnT2fixIlMnDiRE088kSOOOIKpU6cyadIkJk2axG233casWbMA+OQnP8miRYs45JBDOP300/nqV79a+SvY9fXnK6HVpZSJpZTxwAvAB/vxsfpMKWVBKeXDtefoFFOnTmXUqFEv2XfggQdywAEHVJpInebZZ58FoKuri66uLkopG48NHz6c6dOn85Of/ASAgw46iOuvvx6A++67j7FjxzJ69OgdPvNAsqNOx80DXpfkuCQ3JvmPJMuTXJYkAEkOS3JTkl8l+XmSfZr9NyaZ3FzeK8mDzeWZSX6SZE6SB5N8KMnHkixMcnuSUc31Jjbbdye5Kskebff7xSR3JPnPJMc0+49L8tPm8pQktzX3+csk/ssoDTCDBg1i4cKF/OlPf2LOnDnccccdG4+9853vZO7cuaxatQqAxYsXM2PGDAAOP/xwXvOa1zBmzJgqcw8U/R6hJIOBk4Alza5JwD8ABwF/DRydpAu4CDillHIY8D3g8724+/HADODw5vrPlVImAbcBpzfXuRT4x1LKhGaGz7bdfnApZUozT/v+bsuBY5r7/Axw/ha+xg8kWZBkweOPP96LsSXtLDZs2MCkSZMYM2YMU6ZMecn7PKeddhqXX375xu0vfOELjBw5koULF3L22WezcOFC1q9fX2PsAWNwP9730CSLmsvzgO8CRwF3lFIeBmiOjwWeohWUOc0Lo92Ax3rxGDeUUlYBq5KsBK5p9i8BJiQZAYwspdzU7L8EuLLt9rOaP3/VzLGpEcAlSV4PFKBrc0OUUi4GLgaYPHly2dx1JO3cVq5cyQ033MCJJ57IsmXL2HPPPZkyZQonn3zyxuusWrWKM844Y+P2b3/7Wx544IEa4w4YO+I9oYmllLNLKS80+9e0XWc9rRAGWNZ2/YNLKSc011nXNufumzxG+31taNveQO8C23397jk29c+0QjceePtmHl/SLmyvvfZixIgRAOy+++68+c1vZvny5QCccsop/PSnP2XNmhf/mRkxYgRdXa3vRc8880xuvvnmjafqtG12lo9o3wfsneRIgCRdSbpfEz8IHNZcPmVr7rSUshJY0f1+D/A+4KYebrKpEcAjzeWZW/PY6p3TTjuNI488kvvuu48xY8bw3e9+l6uuuooxY8Zw22238ba3vY23vOUttcfUALXPPvtwww03sHjxYu68807mzJnDtddeC8B73vOel5yKg9aHZpYuXcry5cs56aST+MhHPlJj7AGlP0/H9Vop5YUkpwBfa06hDQa+AiwDvgRckeQDwLXbcPd/C3y7+Yj4A8D7t+K2F9A6HfepbXxsvYxN/5J3az8FIvWXJUuWcOihh2722LRp0/5i3+233+4nN/tY2j+OqO03efLk4g9YamfQvL8q7RRKKZt9Qu4sp+MkSR3ICEmSqjFCkqRqjJAkqRojJEmqxghJkqoxQpKkaoyQJKkaIyRJqsYISZKqMUKSpGqMkCSpGiMkSarGCEmSqjFCkqRqjJAkqRojJEmqxghJkqoxQpKkaoyQJKkaIyRJqsYISZKqMUKSpGqMkCSpGiMkSarGCEmSqjFCkqRqjJAkqRojJEmqxghJkqoxQpKkaoyQJKkaIyRJqsYISZKqMUKSpGqMkCSpGiMkSarGCEmSqjFCkqRqjJAkqRojJEmqxghJkqoxQpKkaoyQJKkaIyRJqsYISZKqMUKSpGqMkCSpGiMkSarGCEmSqjFCkqRqjJAkqRojJEmqxghJkqoxQpKkaoyQJKkaIyRJqsYISZKqMUKSpGqMkCSpGiMkSarGCEmSqjFCkqRqjJAkqRojJEmqxghJkqoxQpKkaoyQJKkaIyRJqsYISZKqMUKSpGqMkCSpGiMkSarGCEmSqjFCkqRqjJAkqZqUUmrPMKAkeRx4qPYcu7i9gCdqDyE1fD5uv9eUUvbe3AEjpJ1OkgWllMm155DA52N/83ScJKkaIyRJqsYIaWd0ce0BpDY+H/uR7wlJkqrxlZAkqRojpD6VpCS5sG37nCTn7uAZbkzip5kGsCTrkyxKsjTJlUleUXum3kgyOcnXas+xMzFC6mtrgBlJ9tqWGycZ3MfzaGBaXUqZWEoZD7wAfLD2QL1RSllQSvlw7Tl2JkZIfW0drTdyP7rpgSRjk1yf5O4kc5Ps1+z/QZJvJ5kPXNBsfyvJ7UkeSHJcku8luTfJD9ru71tJFiRZluRzO+oL1E5nHvC65nlyY5L/SLI8yWVJApDksCQ3JflVkp8n2afZv/FVc5K9kjzYXJ6Z5CdJ5iR5MMmHknwsycLmeTmqud7EZvvuJFcl2aPtfr+Y5I4k/5nkmGb/cUl+2lyekuS25j5/meSAHf0/bmdghNQfvgG8N8mITfZfBFxSSpkAXAa0n5YYAxxVSvlYs70HcCStmF0NfBkYBxycZGJznX9qfohwAnBskgn98cVo59W8cj4JWNLsmgT8A3AQ8NfA0Um6aD33TimlHAZ8D/h8L+5+PDADOLy5/nOllEnAbcDpzXUuBf6xeU4vAT7bdvvBpZQpzTzt+7stB45p7vMzwPm9mGnA8dSH+lwp5ekklwIfBla3HTqS1l9qgH8HLmg7dmUpZX3b9jWllJJkCfDHUsoSgCTLgLHAIuDUJB+g9Tzeh9Y/PHf3/VekndDQJIuay/OA7wJHAXeUUh4GaI6PBZ6iFZQ5zQuj3YDHevEYN5RSVgGrkqwErmn2LwEmNN9kjSyl3NTsvwS4su32s5o/f9XMsakRwCVJXg8UoKsXMw04Rkj95SvAXcD3e3n9ZzfZXtP8uaHtcvf24CT7A+cAh5dSVjSn6Xbf5mm1q1ldSpnYvqMJTPtzZT2tf+MCLCulHLmZ+1nHi2eENn3+bPq8a39O9ubfzu7rd8+xqX+mFbqTk4wFbuzFfQ44no5TvyilPAlcAfxd2+5fAu9pLr+X1new2+qVtMK1MsmraJ2SkTbnPmDvJEcCJOlKMq459iBwWHP5lK2501LKSmBF9/s9wPuAm3q4yaZGAI80l2duzWMPJEZI/elCWisQdzsbeH+Su2n9hf3Itt5xKWUxsJDWefUfArdux5wawEopL9AKzBeTLKZ1Kveo5vCXgP+RZCEvfa721t8C/9I8pycC523FbS8A/m/z2B17VsoVEyRJ1fhKSJJUjRGSJFVjhCRJ1RghSVI1RkiSVI0RkvpAX67q3Kydd0pz+TtJDurhusclOWpLx3u43YObW2R2S/u3cB8zk3y9Lx5XncsISX2jx1Wdt3V18FLKmaWUe3q4ynG8+DMv0i7HCEl9r31V53lJrgbuSbJbkn9Jcmez6vLfA6Tl60nuS/ILYHT3HW2yyvOJSe5KsjitVcjH0ordR5tXYcck2TvJj5vHuDPJ0c1t90wyO60Vx79DaymbXnmZ1Z73bWa8P8ln227zN80K0ouS/FuS3bb9f6cGso79KV2pP7St6nxds+tQYHwp5bfNYqsrSymHJxkC3JpkNq2Vnw+gtQDrq4B7aK303H6/ewP/D5ja3NeoUsqTSb4NPFNK+VJzvR8CXy6l3JLWr8r4OXAgrVWcbymlnJfkbbx0OaWX073a87okx9Na7fldzbEptBYHfQ64M8m1tJZTejdwdCllbZJv0lqm6dKteEx1CCMk9Y2eVnX+bbP/BFqrL3evUTYCeD0wFbi8WUX80STXb+b+3wDc3H1fzdp8m3M8cFCzmCfAK5MMax5jRnPba5Os2IqvrafVnueUUv4MkGQW8EZai4IeRitKAEOBP23F46mDGCGpb2xpVef21cEDnF1K+fkm13trH84xCHhDKeX5zcyyrXpa7XnTdb8Kra/zklLK/96eB1Vn8D0hacf5Oa3FMrsAkvzXJH8F3Ay8u3nPaB9g2mZuezswtfkVFqT5zZ7AKmB42/Vm01ooluZ6E5uLNwP/vdl3Eq1fGthbPa32/OYko5IMBd5JayHZucApSUZ3z5rkNVvxeOogRkjacb5D6/2eu5IsBf6N1tmIq4D7m2OX0vrNnS9RSnkc+AAwq1kJ+kfNoWuAk7s/mEDrFwlObj74cA8vfkrvc7QitozWabnf9TDn3Ukebv77V3pe7fkO4Me0fpngj0spC5pP830KmN2sLj2H1i8dlP6Cq2hLkqrxlZAkqRojJEmqxghJkqoxQpKkaoyQJKkaIyRJqsYISZKqMUKSpGr+P1ix73y8aePIAAAAAElFTkSuQmCC", 449 | "text/plain": [ 450 | "
" 451 | ] 452 | }, 453 | "metadata": { 454 | "needs_background": "light" 455 | }, 456 | "output_type": "display_data" 457 | } 458 | ], 459 | "source": [ 460 | "cm = confusion_matrix(labels, preds)\n", 461 | "plot_confusion_matrix(cm,figsize=(8,6),cmap=plt.cm.Greys)\n", 462 | "plt.xticks(range(2), ['Normal', 'Pneumonia'])\n", 463 | "plt.yticks(range(2), ['Normal', 'Pneumonia'])\n", 464 | "plt.xlabel('Predicted Label',fontsize=10)\n", 465 | "plt.ylabel('True Label',fontsize=10)\n", 466 | "plt.show()" 467 | ] 468 | }, 469 | { 470 | "cell_type": "code", 471 | "execution_count": 52, 472 | "id": "9c3153ca", 473 | "metadata": {}, 474 | "outputs": [ 475 | { 476 | "name": "stdout", 477 | "output_type": "stream", 478 | "text": [ 479 | " precision recall f1-score support\n", 480 | "\n", 481 | " 0 0.95 0.85 0.90 234\n", 482 | " 1 0.92 0.97 0.94 390\n", 483 | "\n", 484 | " accuracy 0.93 624\n", 485 | " macro avg 0.93 0.91 0.92 624\n", 486 | "weighted avg 0.93 0.93 0.93 624\n", 487 | "\n" 488 | ] 489 | } 490 | ], 491 | "source": [ 492 | "\n", 493 | "print(classification_report(labels, preds))" 494 | ] 495 | }, 496 | { 497 | "attachments": {}, 498 | "cell_type": "markdown", 499 | "id": "4c8132f8", 500 | "metadata": {}, 501 | "source": [ 502 | "The saved model can be accessed from [here](https://drive.google.com/file/d/17pKbfQwLugq60_RGMHUqbg3DI-EC7M24/view?usp=share_link)" 503 | ] 504 | }, 505 | { 506 | "cell_type": "markdown", 507 | "id": "66cfe614", 508 | "metadata": {}, 509 | "source": [] 510 | } 511 | ], 512 | "metadata": { 513 | "kernelspec": { 514 | "display_name": "Python 3 (ipykernel)", 515 | "language": "python", 516 | "name": "python3" 517 | }, 518 | "language_info": { 519 | "codemirror_mode": { 520 | "name": "ipython", 521 | "version": 3 522 | }, 523 | "file_extension": ".py", 524 | "mimetype": "text/x-python", 525 | "name": "python", 526 | "nbconvert_exporter": "python", 527 | "pygments_lexer": "ipython3", 528 | "version": "3.8.10" 529 | } 530 | }, 531 | "nbformat": 4, 532 | "nbformat_minor": 5 533 | } 534 | -------------------------------------------------------------------------------- /code/Chapter-7 & 9/Adversarial Example Attacks.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "attachments": {}, 5 | "cell_type": "markdown", 6 | "id": "df42271f", 7 | "metadata": {}, 8 | "source": [ 9 | "# Chapter 9: Debugging a PyTorch Image Classifier\n", 10 | "\n", 11 | "## **Adversarial Example Attacks.**\n", 12 | "\n", 13 | "The code below performs the FGSM attack on a PyTorch model. This attack involves generating adversarial examples by perturbing input data with small, carefully crafted changes that cause the model to misclassify the input. The code loops over all examples in the test set, calculates the loss, and then uses the FGSM attack to generate adversarial examples. Finally, it returns the accuracy and the adversarial examples." 14 | ] 15 | }, 16 | { 17 | "cell_type": "code", 18 | "execution_count": 1, 19 | "id": "5ee6465c-fd8f-4df2-bb4b-ce8d5f9c671d", 20 | "metadata": {}, 21 | "outputs": [], 22 | "source": [ 23 | "import matplotlib.pyplot as plt\n", 24 | "import numpy as np\n", 25 | "import scipy\n", 26 | "import torch\n", 27 | "import torch.nn as nn\n", 28 | "import torch.nn.functional as F\n", 29 | "import torch.optim as optim\n", 30 | "from torch.utils.data import ConcatDataset, DataLoader, Dataset\n", 31 | "from torchvision import datasets, transforms\n", 32 | "from torchvision.datasets import ImageFolder\n", 33 | "\n", 34 | "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n", 35 | "random_seed = 12345\n", 36 | "\n", 37 | "PATH = \"chest_xray_pre-processed\"" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": 2, 43 | "id": "ef53d588-eba4-4b47-b4f7-3c535bbc416f", 44 | "metadata": {}, 45 | "outputs": [ 46 | { 47 | "data": { 48 | "text/plain": [ 49 | "" 50 | ] 51 | }, 52 | "execution_count": 2, 53 | "metadata": {}, 54 | "output_type": "execute_result" 55 | } 56 | ], 57 | "source": [ 58 | "def seed_worker(worker_id):\n", 59 | " worker_seed = torch.initial_seed() % 2**32\n", 60 | " numpy.random.seed(worker_seed)\n", 61 | " random.seed(worker_seed)\n", 62 | "\n", 63 | "\n", 64 | "g = torch.Generator()\n", 65 | "g.manual_seed(0)" 66 | ] 67 | }, 68 | { 69 | "cell_type": "code", 70 | "execution_count": 3, 71 | "id": "3bf1ef57-f2d0-42c7-b8b1-30ce775b5df6", 72 | "metadata": {}, 73 | "outputs": [], 74 | "source": [ 75 | "import matplotlib.pyplot as plt\n", 76 | "from matplotlib.colors import LinearSegmentedColormap\n", 77 | "\n", 78 | "classes = dict({0: \"NORMAL\", 1: \"PNEUMONIA\"})" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": 4, 84 | "id": "3feab12c-d83f-4258-86d6-f8ff9688f30d", 85 | "metadata": {}, 86 | "outputs": [], 87 | "source": [ 88 | "MEAN = torch.tensor([0.485, 0.456, 0.406])\n", 89 | "STD = torch.tensor([0.229, 0.224, 0.225])\n", 90 | "\n", 91 | "\n", 92 | "from matplotlib.colors import LinearSegmentedColormap\n", 93 | "\n", 94 | "\n", 95 | "def imshow(img, transpose=True):\n", 96 | " plt.figure(figsize=(11, 6))\n", 97 | " x = img.cpu() * STD[:, None, None] + MEAN[:, None, None]\n", 98 | " # x = img.cpu()\n", 99 | " # x= x / 2 + 0.5 # unnormalize\n", 100 | " npimg = x.cpu().detach().numpy()\n", 101 | " plt.imshow(npimg.transpose(1, 2, 0))\n", 102 | " plt.axis(\"off\")\n", 103 | " plt.show()" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": 11, 109 | "id": "74473fc5-ba3e-4c49-88b3-965a469855f2", 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "epsilons = [0, .02, 0.05, .1, .15, .2,]\n", 114 | "pretrained_model = \"model/Finetuning_Stage2.pt\"\n", 115 | "model = torch.load(pretrained_model, map_location=device)\n", 116 | "model.eval();" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": 12, 122 | "id": "3852fc26-80ea-4dcb-93ca-d33615f04828", 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "testset = ImageFolder(\n", 127 | " PATH + \"/test\",\n", 128 | " transform=transforms.Compose(\n", 129 | " [\n", 130 | " transforms.Resize((224, 224)),\n", 131 | " transforms.ToTensor(),\n", 132 | " transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n", 133 | " ]\n", 134 | " ),\n", 135 | ")\n", 136 | "test_dl = DataLoader(testset, batch_size=1)" 137 | ] 138 | }, 139 | { 140 | "cell_type": "code", 141 | "execution_count": 13, 142 | "id": "c1c01940-dce2-4464-be07-d5f8efa84447", 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "# This function performs an FGSM attack on an input image using the given data gradient\n", 147 | "# Code adapted from https://pytorch.org/tutorials/beginner/fgsm_tutorial.html\n", 148 | "\n", 149 | "def fgsm_attack(image, epsilon, data_grad):\n", 150 | " # Compute the sign of the data gradient\n", 151 | " sign_data_grad = torch.sign(data_grad)\n", 152 | " # Create a perturbed image by adding epsilon times the sign of the gradient to the original image\n", 153 | " perturbed_image = image + epsilon * sign_data_grad\n", 154 | " # Clip the pixel values to be in the [0,1] range\n", 155 | " perturbed_image = torch.clamp(perturbed_image, 0, 1)\n", 156 | " # Return the perturbed image\n", 157 | " return perturbed_image" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": null, 163 | "id": "c16bfd1e", 164 | "metadata": {}, 165 | "outputs": [], 166 | "source": [ 167 | "# This function tests the robustness of a deep learning model against FGSM attacks with varying levels of epsilon\n", 168 | "# It returns the accuracy of the model and saves adversarial examples for visualization\n", 169 | "# Code adapted from https://pytorch.org/tutorials/beginner/fgsm_tutorial.html\n", 170 | "\n", 171 | "def test(model, device, test_loader, epsilon):\n", 172 | "\n", 173 | " # Initialize counters and storage\n", 174 | " num_correct = 0\n", 175 | " adversarial_examples = []\n", 176 | "\n", 177 | " # Iterate over the test set\n", 178 | " for images, labels in test_loader:\n", 179 | "\n", 180 | " # Send data and labels to the device\n", 181 | " images, labels = images.to(device), labels.to(device)\n", 182 | "\n", 183 | " # Set requires_grad attribute of tensor\n", 184 | " images.requires_grad = True\n", 185 | "\n", 186 | " # Make predictions\n", 187 | " output = model(images)\n", 188 | " initial_predictions = output.max(1, keepdim=True)[1]\n", 189 | "\n", 190 | " # Check if initial prediction is correct\n", 191 | " if initial_predictions.item() != labels.item():\n", 192 | " continue\n", 193 | "\n", 194 | " # Calculate loss and gradients\n", 195 | " loss = F.nll_loss(output, labels)\n", 196 | " model.zero_grad()\n", 197 | " loss.backward()\n", 198 | " datagrad = images.grad.data\n", 199 | "\n", 200 | " # Generate adversarial examples using FGSM attack\n", 201 | " perturbed_images = fgsm_attack(images, epsilon, datagrad)\n", 202 | "\n", 203 | " # Make predictions on the adversarial examples\n", 204 | " output = model(perturbed_images)\n", 205 | " final_predictions = output.max(1, keepdim=True)[1]\n", 206 | "\n", 207 | " # Check for success\n", 208 | " if final_predictions.item() == labels.item():\n", 209 | " num_correct += 1\n", 210 | " if (epsilon == 0) and (len(adversarial_examples) < 5):\n", 211 | " adversarial_example = perturbed_images.squeeze().detach().cpu().numpy()\n", 212 | " adversarial_examples.append((initial_predictions.item(), final_predictions.item(), adversarial_example))\n", 213 | " else:\n", 214 | " if len(adversarial_examples) < 5:\n", 215 | " adversarial_example = perturbed_images.squeeze().detach().cpu().numpy()\n", 216 | " adversarial_examples.append((initial_predictions.item(), final_predictions.item(), adversarial_example))\n", 217 | "\n", 218 | " # Calculate and print accuracy\n", 219 | " accuracy = num_correct / float(len(test_loader))\n", 220 | " print(\"Epsilon: {}\\tTest Accuracy = {} / {} = {}\".format(epsilon, num_correct, len(test_loader), accuracy))\n", 221 | "\n", 222 | " # Return accuracy and adversarial examples\n", 223 | " return accuracy, adversarial_examples" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": 15, 229 | "id": "b7939be3-6f58-4435-9d50-024e80fef5d8", 230 | "metadata": {}, 231 | "outputs": [ 232 | { 233 | "name": "stderr", 234 | "output_type": "stream", 235 | "text": [ 236 | "/Users/parul/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at ../c10/core/TensorImpl.h:1156.)\n", 237 | " return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)\n" 238 | ] 239 | }, 240 | { 241 | "name": "stdout", 242 | "output_type": "stream", 243 | "text": [ 244 | "Epsilon: 0\tTest Accuracy = 519 / 624 = 0.8317307692307693\n", 245 | "Epsilon: 0.02\tTest Accuracy = 285 / 624 = 0.4567307692307692\n", 246 | "Epsilon: 0.05\tTest Accuracy = 160 / 624 = 0.2564102564102564\n", 247 | "Epsilon: 0.1\tTest Accuracy = 133 / 624 = 0.21314102564102563\n", 248 | "Epsilon: 0.15\tTest Accuracy = 155 / 624 = 0.2483974358974359\n", 249 | "Epsilon: 0.2\tTest Accuracy = 177 / 624 = 0.28365384615384615\n" 250 | ] 251 | } 252 | ], 253 | "source": [ 254 | "accuracies = []\n", 255 | "examples = []\n", 256 | "\n", 257 | "# Run test for each epsilon\n", 258 | "for eps in epsilons:\n", 259 | " acc, ex = test(model, device, test_dl, eps)\n", 260 | " accuracies.append(acc)\n", 261 | " examples.append(ex)" 262 | ] 263 | }, 264 | { 265 | "cell_type": "code", 266 | "execution_count": 19, 267 | "id": "4cc9536c-2593-49aa-ade2-634f90eea06e", 268 | "metadata": {}, 269 | "outputs": [ 270 | { 271 | "data": { 272 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAAFNCAYAAABmLCa9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAqM0lEQVR4nO3de3xddZ3v/9c7SW9Jr7mU3tsEyqXciubQKip6nJGW0RZFFAYVHJXRGXRER8WjD8Zh8Jw5OjMqyvwUPV5RLjLqo45lUBCUUSgNWsC2FNIUeqHQ9EbvlzSf3x9rpd0NafdumtWdnf1+Ph77kb0ue+3PSuDd7/qu9V1LEYGZmR1ZRbELMDPr7xyUZmZ5OCjNzPJwUJqZ5eGgNDPLw0FpZpaHg9KsCCRdKemXOdMh6ZRi1mRH5qAsY5IelLRF0pBi19KfSXpW0m5JO3JeXzuebUbEDyPiTX1Vo2XLQVmmJE0DXgsEMO8Ef3fVify+PvKWiBie87q22AXZieOgLF/vAR4BvgtclbtA0mRJP5HULmlTbutJ0gckLZe0XdIySa9I5x926Cjpu5JuSt+/XtJaSZ+S9ALwHUljJP1n+h1b0veTcj5fK+k7kp5Pl/8snf8nSW/JWW+QpI2Szuu+g2mdb86Zrkq/7xWShkq6Ld2/rZIWSzrpWH+Jkq6W9DtJX5P0kqSnJL2x2/K29Pe1StKVOfP/+wjbHCXp+2mtz0n6rKSK3M9J+pf097JK0txjrduOjYOyfL0H+GH6uqgrJCRVAv8JPAdMAyYCd6TLLgM+l352JElLdFOB3zcOqAWmAteQ/Lf3nXR6CrAbyD2c/QFQDZwJjAW+lM7/PvCunPUuBtZHxB97+M7bgStypi8CNkbEH0j+cRgFTAbqgA+mNfTGLGAlUA/8A/CTNOhrgJuBuRExAng1sKSA7X01ra0JuJDk9/3ebt+3Iv2+LwD/T5J6WbsVIiL8KrMX8BpgP1CfTj8FXJe+fxXQDlT18Ll7gb87wjYDOCVn+rvATen71wP7gKFHqWkmsCV9Px7oBMb0sN4EYDswMp2+G/jkEbZ5SrpudTr9Q+CG9P1fAb8Hzing9/UssAPYmvP6QLrsauB5QDnrPwq8G6hJ170UGNZtm1cD/9399wdUpr+rGTnL/hp4MOdzrTnLqtPPjiv2f1cD+eUWZXm6CvhlRGxMp3/EocPvycBzEdHRw+cmk7SceqM9IvZ0TUiqlvSN9NByG/BbYHTaop0MbI6ILd03EhHPA78DLpU0GphLEoAvExGtwHLgLZKqSVrAP0oX/4Ak+O9ID++/IGnQUeq/JCJG57y+mbNsXaSplXoOmBARO4F3krRW10v6haTTj/IdkLQSB6XbyN3exJzpF3L2cVf6dnie7dpxcFCWGUnDgHcAF0p6Ie0zvA44V9K5wBpgyhFOuKwBTj7CpneRtG66jOu2vPttqj4OnAbMioiRwOu6Sky/pzYNwp58j+Tw+zLg4YhYd4T14NDh93xgWRqeRMT+iPjHiJhBckj8ZpJD3N6Y2O3QdwpJK5OIuDci/pyklfwU8M0ePp9rI0lrf2q37R1tHy1jDsrycwlwAJhBcrg7EzgDeIgkKB4F1gP/LKkmPelxQfrZbwF/L+mVSpwiqet/6CXAX0qqlDSHpG/taEaQ9AlulVRL0rcHQESsB+4B/j096TNI0utyPvsz4BXA35H0WR7NHcCbgA9xqDWJpDdIOjttwW4jCafOPNs6krHAR9I6LyP5fS6UdJKk+Wlf5V6Sw/ejfkdEHADuAj4vaUT6+/0YcFsva7M+4KAsP1cB34mI1RHxQteL5ETKlSQtureQ9JetBtaSHD4SET8GPk8SONtJAqs23e7fpZ/bmm7nZ3nq+DIwjKQF9QjwX92Wv5skvJ4CNgAf7VoQEbuB/wAagZ8c7UvS0H2YpNV4Z86icST9m9tIDs9/Q3I4fiQ/1+HXUf40Z9kiYHq6L58H3h4Rm0j+//oYSetyM8k/Hh86Wr2pDwM7gTbgv0l+398u4HOWER3etWJWGiTdAJwaEe/Ku3K2dVwNvD8iXlPMOixbpXjhr5W59FD9fSStTrPMZXboLenbkjZI+tMRlkvSzZJaJT2h9MJls6OR9AGSkz33RMRvi12PlYfMDr3TzvcdwPcj4qwell9M0hdzMckFtF+JiFmZFGNmdhwya1Gm/9pvPsoq80lCNCLiEZJr6MZnVY+ZWW8V86z3RJJDqC5rOfyiWjOzfqEkTuZIuoZkfDA1NTWvPP30fIMbzMyOzWOPPbYxIhp6WlbMoFxHMlStyySOMPogIm4FbgVobm6OlpaW7Kszs7Ii6bkjLSvmofcC4D3p2e/ZwEvpxcFmZv1KZi1KSbeT3DWmXtJakiFqgwAi4uvAQpIz3q0k44Tf2/OWzMyKK7OgjIgr8iwP4G+z+n4zs77isd5mZnk4KM3M8nBQmpnl4aA0M8vDQWlmloeD0swsDwelmVkeDkozszwclGZmeTgozczycFCameXhoDQzy8NBaWaWh4PSzCwPB6WZWR4OSjOzPByUZmZ5OCjNzPLINCglzZG0QlKrpOt7WD5V0v2SnpD0oKRJWdZjZtYbmQWlpErgFmAuMAO4QtKMbqv9C/D9iDgHuBH4P1nVY2bWW1m2KM8HWiOiLSL2AXcA87utMwP4dfr+gR6Wm5kVXZZBORFYkzO9Np2X63Hgben7twIjJNVlWJOZ2TEr9smcvwculPRH4EJgHXCg+0qSrpHUIqmlvb39RNdoZmUuy6BcB0zOmZ6UzjsoIp6PiLdFxHnAZ9J5W7tvKCJujYjmiGhuaGjIsGQzs5fLMigXA9MlNUoaDFwOLMhdQVK9pK4aPg18O8N6zMx6JbOgjIgO4FrgXmA5cFdELJV0o6R56WqvB1ZIeho4Cfh8VvWYmfWWIqLYNRyT5ubmaGlpKXYZZjbASHosIpp7WlbskzlmZv2eg9LMLA8HpZlZHg5KM7M8HJRmZnk4KM3M8nBQmpnl4aA0M8vDQWlmloeD0swsDwelmVkeAz4oN2zbwzu+8TAbtu8pdilmVqIGfFB++b6nWfzsZm6+75lil2JmJaqq2AVk5bTP3sPejs6D07ctWs1ti1YzpKqCFTfNLWJlZlZqBmyL8qFPvoF5MyegdHrooArmz5zAQ596Q1HrMrPSM2CDcuzIoYwYUkXX3Tb3dnQyYkgVY0cMLWpdZlZ6BmxQAmzcsZfXTq8H4KIZJ9G+Y2+RKzKzUjRg+ygBvvHuZjbu2EvzTfdx7uQxfOj1Jxe7JDMrQQO6RQlQP3wI08cO55G2TcUuxcxKVKZBKWmOpBWSWiVd38PyKZIekPRHSU9IujiLOmY31dHy7GY6DnTmX9nMrJvMglJSJXALMBeYAVwhaUa31T5L8nTG80geZ/vvWdQyq6mWnfsO8Kfnt2WxeTMb4LJsUZ4PtEZEW0TsA+4A5ndbJ4CR6ftRwPNZFDKrsQ7Ah99m1itZBuVEYE3O9Np0Xq7PAe+StBZYCHy4pw1JukZSi6SW9vb2Yy6kYcQQTnE/pZn1UrFP5lwBfDciJgEXAz+Q9LKaIuLWiGiOiOaGhoZefdGsxloWr3I/pZkduyyDch0wOWd6Ujov1/uAuwAi4mFgKFCfRTGzm+rYue8AS91PaWbHKMugXAxMl9QoaTDJyZoF3dZZDbwRQNIZJEF57MfWBZjVVAu4n9LMjl1mQRkRHcC1wL3AcpKz20sl3ShpXrrax4EPSHocuB24OiKi5y0en7EjhtLUUOOgNLNjlunInIhYSHKSJnfeDTnvlwEXZFlDrtlNdfx8yfN0HOikqrLY3bNmVirKKi1mN9WxfW8Hy9a7n9LMCldeQdnofkozO3ZlFZRjRw6lqb6GRW2bi12KmZWQsgpKgFlNdTy6ajMHOjM5Z2RmA1DZBeXsptqkn9LXU5pZgcowKD3u28yOTdkF5Ukjh9JYX8OiVQ5KMytM2QUlJIffi9xPaWYFKtOgrGP7ng6W+3pKMytAWQal709pZseiLINy3KihTKur5hFfT2lmBSjLoITk8PvRVZvcT2lmeZVtUM5qqmWb+ynNrADlG5RpP+WiVT78NrOjK9ugnDB6GFPrqn1Cx8zyKtugBJjdmIz77nQ/pZkdRVkH5aymWl7avZ+nXthe7FLMrB8r86D09ZRmll+mQSlpjqQVklolXd/D8i9JWpK+npa0Nct6ups4ehhTat1PaWZHl9kzcyRVArcAfw6sBRZLWpA+JweAiLguZ/0PA+dlVc+RzGqs5VfLX6SzM6io0In+ejMrAVm2KM8HWiOiLSL2AXcA84+y/hUkT2I8oWY31bF1135WvOh+SjPrWZZBORFYkzO9Np33MpKmAo3ArzOsp0d+3reZ5dNfTuZcDtwdEQd6WijpGkktklra29v79Isnjalmcu0wB6WZHVGWQbkOmJwzPSmd15PLOcphd0TcGhHNEdHc0NDQhyUmZvl6SjM7iiyDcjEwXVKjpMEkYbig+0qSTgfGAA9nWMtRzW6qY8uu/Ty9wf2UZvZymQVlRHQA1wL3AsuBuyJiqaQbJc3LWfVy4I6IKFpzblbX875X+vDbzF4us8uDACJiIbCw27wbuk1/LssaCjG5tpqJo4exaNVmrr6gsdjlmFk/019O5hTd7KY6Frmf0sx64KBMzW6qZfPOfTyzYUexSzGzfsZBmfLzvs3sSByUqUP9lA5KMzucgzLHrKZaHmnbTBFPwJtZP+SgzDG7qc79lGb2Mg7KHLO7nqPjfkozy+GgzDG5dhgTRg31877N7DAOyhySmN1UxyNtm9xPaWYHOSi7mdVUy6ad+2h1P6WZpRyU3Ry8ntLP+zazlIOymym11YwfNdQXnpvZQQ7Kbrr6KRe5n9LMUg7KHsxqrGXjjn2sbN9Z7FLMrB9wUPbA477NLJeDsgdT66oZN9L9lGaWcFD2QBKzmmpZtMrjvs3MQXlEs5vqaN++l7aN7qc0K3eZBqWkOZJWSGqVdP0R1nmHpGWSlkr6UZb1HAv3U5pZl8yCUlIlcAswF5gBXCFpRrd1pgOfBi6IiDOBj2ZVz7GaVlfNSSOHeNy3mWXaojwfaI2ItojYB9wBzO+2zgeAWyJiC0BEbMiwnmMiiVmNvp7SzLINyonAmpzptem8XKcCp0r6naRHJM3JsJ5jNrupjg3b97LK/ZRmZa3YJ3OqgOnA64ErgG9KGt19JUnXSGqR1NLe3n7CipvdlD7v24ffZmUty6BcB0zOmZ6Uzsu1FlgQEfsjYhXwNElwHiYibo2I5ohobmhoyKzg7hrra2gYMcTP0TErc1kG5WJguqRGSYOBy4EF3db5GUlrEkn1JIfibRnWdEx8f0ozgwyDMiI6gGuBe4HlwF0RsVTSjZLmpavdC2yStAx4APhERPSr5tvsplpe3LaXZzftKnYpZlYkVVluPCIWAgu7zbsh530AH0tf/VLu9ZSN9TVFrsbMiqHYJ3P6vab6GuqHD/EDx8zKWN6glPQWSWUbqEk/pZ/3bVbOCgnAdwLPSPqCpNOzLqg/mt1Uxwvb9vCc+ynNylLeoIyIdwHnASuB70p6OL2ucUTm1fUTXddT+jIhs/JU0CF1RGwD7iYZhjgeeCvwB0kfzrC2fuPkhuHUDx/sC8/NylQhfZTzJP0UeBAYBJwfEXOBc4GPZ1te/5Dcn9LXU5qVq0JalJcCX4qIsyPii103roiIXcD7Mq2uH5ndVMf6l/awerP7Kc3KTSFB+Tng0a4JScMkTQOIiPuzKav/md2Y9lP68Nus7BQSlD8GOnOmD6TzysopY4dTVzPYN/I1K0OFBGVVej9JANL3g7MrqX/yuG+z8lVIULbnjM1G0nxgY3Yl9V+zmmp5/qU9rN2yu9ilmNkJVMhY7w8CP5T0NUAkN+N9T6ZV9VNd474fbtvE5NrqIldjZidKIRecr4yI2STPvTkjIl4dEa3Zl9b/TB87nFr3U5qVnYLuHiTpL4AzgaGSAIiIGzOsq19KnqNT6zPfZmWmkAvOv04y3vvDJIfelwFTM66r35rdVMe6rbtZ4+spzcpGISdzXh0R7wG2RMQ/Aq8iuRN5WfLzvs3KTyFBuSf9uUvSBGA/yXjvsnSon9KH32blopA+yp+nT0b8IvAHIIBvZllUf1ZRIc6fVus7CZmVkaO2KNMb9t4fEVsj4j9I+iZPz32cQ57Pz5G0QlKrpOt7WH61pHZJS9LX+3u1FyfY7KZa1m5xP6VZuThqUEZEJ3BLzvTeiHipkA1Lqkw/O5fk0qIrJM3oYdU7I2Jm+vpW4aUXz+yTk37KRat8+G1WDgrpo7xf0qXqui6ocOcDrRHRlg57vAOYf8wV9kOnjh3B6OpBfo6OWZkoJCj/muQmGHslbZO0XdK2Aj43kWQUT5e16bzuLpX0hKS7JU0uYLtFV1GRXE/5iPspzcpCISNzRkRERUQMjoiR6fTIPvr+nwPTIuIc4FfA93paKX30RIuklvb29j766uMzu6mONZt3s3aL+ynNBrq8Z70lva6n+RHx2zwfXQfkthAnpfNyt5HbJPsW8IUjfNetwK0Azc3N/eLWPV3XUy5q28ykV3rct9lAVsjlQZ/IeT+UpO/xMeB/5vncYmC6pEaSgLwc+MvcFSSNj4j16eQ8YHkhRfcHp52U9lOu2sSlr5xU7HLMLEN5gzIi3pI7nfYjfrmAz3VIuha4F6gEvh0RSyXdCLRExALgI+kt3DqAzcDVx7wHRdJ1PaUvPDcb+Aq6KUY3a4EzClkxIhYCC7vNuyHn/aeBT/eihn5hdlMdv1z2Is9v3c2E0cOKXY6ZZaSQPsqvkozGgeTkz0ySETplb1bO877fep4Pv80GqkJalC057zuA2yPidxnVU1LOGDeSUcMG8cjKzQ5KswGskKC8G9gTEQcgGXEjqTp9XG1Zq6gQ5/t6SrMBr6CROUBuB9ww4L5syik9s5vqeG7TLta/5OfomA1UhQTl0IjY0TWRvveFg6lZ6fO+3/WtRWzYvifP2mZWigoJyp2SXtE1IemVgJtPqTPGj2RQpVjZvpOb73um2OWYWQYK6aP8KPBjSc+TPApiHMmjIcreaZ+9h70dnQenb1u0mtsWrWZIVQUrbppbxMrMrC8VMtZ7MXA68CGSR9eeERGPZV1YKXjok29g3swJDK5Kfo2VgvkzJ/DQp95Q5MrMrC8V8nCxvwVqIuJPEfEnYLikv8m+tP5v7MihjBhSxf4DnVRWiAMBW3buY+yIocUuzcz6UCF9lB+IiK1dExGxBfhAZhWVmI079nLlrKn89G9eTW3NYH6/chMvbvNJHbOBpJCgrMy9aW965/LB2ZVUWr7x7mZuuuQszpk0mrv++lVUVYpP3P0EEf3iJkdm1gcKCcr/Au6U9EZJbwRuB+7JtqzSdMrY4fyvi8/gt0+384NHnit2OWbWRwoJyk8BvyY5kfNB4EkOvwDdcrx79lQuPLWBz/9iOa0bduT/gJn1e4Wc9e4EFgHPktyL8n9SQveNPNEk8cW3n0P14Equu3MJ+3IuHzKz0nTEoJR0qqR/kPQU8FVgNUBEvCEivnaiCixFY0cO5f+87WyeXPcSN9/vi9DNSt3RWpRPkbQe3xwRr4mIrwIHTkxZpW/OWeO59BWT+PcHW3nsOd/c16yUHS0o3wasBx6Q9M30RM6xPrK2rH1u3gwmjB7GdXc+zo69HcUux8x66YhBGRE/i4jLSUblPEAylHGspP9P0ptOUH0lbcTQQfzbO2ayZssubvz50mKXY2a9VMjJnJ0R8aP02TmTgD+SnAnPS9IcSSsktUq6/ijrXSopJDUXXHmJOL+xlg9eeDJ3tazl3qUvFLscM+uFQi4POigitkTErRHxxnzrphem3wLMBWYAV0ia0cN6I4C/IzmzPiBd92encuaEkXz6J0/6VmxmJeiYgvIYnQ+0RkRbROwD7gDm97DePwH/FxiwCTK4qoIvv3MmO/d28CmP2jErOVkG5URgTc702nTeQel9LidHxC8yrKNfmH7SCK6fezoPrGjnh4tWF7scMzsGWQblUUmqAP4N+HgB614jqUVSS3t7e/bFZeSqV03jtdPr+fwvltPW7lE7ZqUiy6BcB0zOmZ6UzusyAjgLeFDSs8BsYEFPJ3TSftHmiGhuaGjIsORsVVSIL779XAZXVXDdnUvYf8CjdsxKQZZBuRiYLqlR0mDgcmBB18KIeCki6iNiWkRMAx4B5kVES8+bGxjGjRrK5996Fo+vfYmv/rq12OWYWQEyC8qI6ACuBe4lGRt+V0QslXSjpHlZfW8pePM5E3jreRO55YFW/rB6S7HLMbM8VGpnYJubm6OlpfQbndv27Gfulx9iUKX4xUdeS82QQh5fZGZZkfRYRPR4LXfRTuaUu5FDB/Gv7ziX5zbv4qZf+GZMZv2Zg7KIZjfVcc1rm7j90dXct+zFYpdjZkfgoCyyj73pVM4YP5Lrf/IEG3fsLXY5ZtYDB2WRDamq5MvvnMm2PR1c/x8etWPWHzko+4HTxo3gkxedxn3LN3DH4jX5P2BmJ5SDsp/4qwsaefXJdfzTfy7j2Y07i12OmeVwUPYTFRXiXy47l6oK8dE7l9DhUTtm/YaDsh+ZMHoY/3TJWSxZs5V/f3Blscsxs5SDsp+ZP3Mi886dwFfuf4bH12wtdjlmhoOyX/qn+WcxdsQQrrtzCbv2+Vk7ZsXmoOyHRlUP4l8vO5e2jTv53ws9ases2ByU/dSrT6nn/a9p5LZHVvPAUxuKXY5ZWXNQ9mN/f9FpnD5uBJ+4+wk2edSOWdE4KPuxoYMq+dI7Z7Jt934+/ZMnPWrHrEgclP3cGeNH8vE3ncovl73Ij1vWFrscs7LkoCwB739tE7Maa/nHny9l9aZdxS7HrOw4KEtAZYX413ecS4XEx+5awoFOH4KbnUgOyhIxaUw1N15yJi3PbeHrv/GoHbMTKdOglDRH0gpJrZKu72H5ByU9KWmJpP+WNCPLekrdJTMn8hfnjOdLv3qaJ9e+VOxyzMpGZkEpqRK4BZgLzACu6CEIfxQRZ0fETOALJM/5tiOQxOcvOYv64UP46J1/ZPe+A8UuyawsZNmiPB9ojYi2iNgH3AHMz10hIrblTNYA7nzLY3T1YP7lsnNZ2b6Tf77Ho3bMToQsg3IikHsX2rXpvMNI+ltJK0lalB/JsJ4B4zXT63nvBdP43sPP8Zun24tdjtmAV/STORFxS0ScDHwK+GxP60i6RlKLpJb2dgcDwKfmnM70scP5xI8fZ8vOfcUux2xAyzIo1wGTc6YnpfOO5A7gkp4WRMStEdEcEc0NDQ19V2EJ6xq1s2XXPv7XTz1qxyxLWQblYmC6pEZJg4HLgQW5K0ianjP5F8AzGdYz4Jw1cRTX/fmp3POnF/jJH472b5BZ+diwbQ/v+MbDbNi+p8+2mVlQRkQHcC1wL7AcuCsilkq6UdK8dLVrJS2VtAT4GHBVVvUMVH/9upM5f1ot/7BgKWs2e9SO2c33P8PiZzfzlfv6rt2lUjtka25ujpaWlmKX0a+s2byLuV95iBnjR3L7NbOprFCxSzLL3J79B1i9eRdt7TtY2b6Tf/3lCnoatDakqoIVN83Nuz1Jj0VEc0/Lqo67Wiu6ybXVfG7emfz9jx/n1t+28aHXn1zsksz6RESw/qU9rNq482Agrtq4k7aNO1i7ZTe57by6msEAbN29nwOdwdCqCi46axyf+YszjrsOB+UAcekrJnL/8hf5t1+t4LXT6zlr4qhil2RWsO179tPWFYLtO1i5cSer0und+w8NrKgeXEljfQ0zJ4/hbedNoqmhhqb64TQ21DB8SBWf+emT/OjR1QypqmDvgU5GDKli7Iihx12fg3KAkMT/fuvZPPbcFq67cwk///BrGDqosthlmR20/0AnazbvSsMwaRUmP3fSvv3QjakrlNzboKmhhtlNdTQ21HByfQ1NDcM5aeQQpCN3LW3csZcrZ03lL8+fwo8eXU17H53QcR/lAPObp9u56tuP8t4LpvEPbzmz2OVYmYkINu7YR1v7jvQQOWkhtm3cyepNu+jI6UQcUz2IpobhNKUh2Fhfw8kNNUypq2ZI1Yn/R959lGXkwlMbeM+rpvKd3z3LK6aM4QePPMfX/vK8Pjn8MOuye98BVm08dKicG4jb9xx6cujgqgqm1VVz6tgRzDlzHI1pKDbV1zAm7VMsBQ7KAejTc8/gd60b+eTdT7Cn4wA33/cMN7317GKXZSWmszNYt3U3bRt3supgGCbhuG7r7sPWHT9qKE0NNVwycyJNDTVp63A4E0YPGxBXYfjQewA67bP3sLej82XzB1dV8HQBl0lY6duwbQ/X3v7Hgo4mXtq1n5Vpf+Gqrn7D9p08u2nnYf8dDR9SlZ48OXSo3BWK1YNLv83lQ+8y89An38BNC5dzz5Pr2X/g0D+E+zo6ef0XH2DGhJGcMW5k8nP8SMaPGnrUDnIrPV0XXXcdTezr6GT15p2HLq9p33Gwdbgp514BlRViam01jfU1vO7U+sMCsWH40U+kDGQOygFo7MihjBhSRUdnMKSqgn0dnVx4agOvmDqG5eu3sfT5bSx88oWD64+uHnRYcM4YP5JTxg5ncFXR75lixyAiOO2z/8W+A4dagbctWs1ti1a/bN364UNoqq/hz2eclLYKh9PUUMOU2moGVfrv3p2DcoDq6TKJj7zx0ND6HXs7eGr9Npav38ay9dtYtn47P1z0HHv2J/+TDaoUp4wdwRnjRzAjDc8zxo8sqQ74gWrn3g5WbdzJyq4zy+mlNqvadx4WkgACxo8eypwzx3H2pFE01ictxFHDBhWn+BLlPko76EBnsGrjTpZ1Bejzyc8NOde4jR819GBodrVAp9ZWUzEAOuz7kwOdwdotu2hrf3kgvrjt0N9Dgomjhx08edLUUMODKzbwwFPtDK6qYN+BTq48f4pP5hXAfZRWkMoKccrY4Zwydjjzzp1wcP7GHXtZflh4bufBp9sPPg2yenAlp48bcdih+2njRgyIDv6sbd6572B/YfdrDnNbh6OGDaKpoYYLTqlPArG+hsaGGqbV1bxsYMHvWjdy5ey+v+i6nLlFab2yZ/8BWjfsYNnzXYfuSZB2XUMnQWN9zcHg7GqF5htZMRDt2X+A5zbtyrneMD1U3riTrbv2H1xvUKWYWldzMARPTvsNG+trqK0ZXHa/txPNLUrrc0MHVXLWxFGHjSmPCNZu2X2w33P5+m08sXYrv3hi/cF1amsGH+z37Dp8P7lheMmfQOjsDF7YtudlQ/Pa2newbuvhN284aeQQmuqHc/HZ42lKD5kb62uYNGYYVSX+exio3KK0zG3bs5+n1m8/dOj+wjaeemE7+9Jr9AZXVjD9pOEHW59dP0dV978TDl03b+g6ebIybSE+2+3mDTWDK2nsumFDennNyQ3DmVaf3LzB+p+jtSgdlFYUHQc6D544WpZz4mjjjkPX9E0cPSwNzUP9n5PHZH/iqOvmDW05t/RamV6EvXHH4TdvmFxbnRwqp4fJXYE4dkT5dTGUOgellYwN2/ewfP32g8G5bP022tp3HLwh6/AhVT2eOOrpTklHG52Se/OGto2HX4S9evPhN2+orRmcjkY5FIgnN9QwubY4N2+wbDgoraTt2X+Ap188PDyXr9/Ojr3JiaMKQVND7qF7EqQ33/cMP3x0NXPPGs/FZ49jVbczy91v3tBYV3OwVXiwhVhfw+hqXztaDhyUNuB0diYnjnLPuC97ftvLbtbQ3YRRQ5O716Rnk7vuZDNQbt5gvVe0s96S5gBfASqBb0XEP3db/jHg/UAH0A78VUQ8l2VNNjBUVIgpddVMqatmzlnjDs5/add+ft+2kVseaGX5+u0c6AwGVYoLTqnnxnlnMaWuuohVW6nK7FoESZXALcBcYAZwhaQZ3Vb7I9AcEecAdwNfyKoeKw+jqgcx96zxnDtpNJ2RjHXv6AwmjR7mkLRey/KirfOB1ohoi4h9wB3A/NwVIuKBiOh6xuojwKQM67Ey0jXW/ad/cwFXzppKe87ZarNjleWh90RgTc70WmDWUdZ/H3BPTwskXQNcAzBlypS+qs8GsG+8+1BX002XnFXESmwg6BfDACS9C2gGvtjT8oi4NSKaI6K5oaHhxBZnZmUvyxblOmByzvSkdN5hJP0Z8Bngwojw8ZGZ9TtZtigXA9MlNUoaDFwOLMhdQdJ5wDeAeRGxIcNazMx6LbOgjIgO4FrgXmA5cFdELJV0o6R56WpfBIYDP5a0RNKCI2zOzKxoMr2OMiIWAgu7zbsh5/2fZfn9ZmZ9oV+czDEz688clGZmeTgozczycFCameXhoDQzy8NBaWaWh4PSzCwPB6WZWR4OSjOzPByUZmZ5OCjNzPJwUJqZ5eGgNDPLw0FpZpaHg9LMLA8HpZlZHg5KM7M8HJRmZnlkGpSS5khaIalV0vU9LH+dpD9I6pD09ixrMTPrrcyCUlIlcAswF5gBXCFpRrfVVgNXAz/Kqg4zs+OV5cPFzgdaI6INQNIdwHxgWdcKEfFsuqwzwzrMzI5LlofeE4E1OdNr03lmZiWlJE7mSLpGUouklvb29mKXY2ZlJsugXAdMzpmelM47ZhFxa0Q0R0RzQ0NDnxRnZlaoLINyMTBdUqOkwcDlwIIMv8/MLBOZBWVEdADXAvcCy4G7ImKppBslzQOQ9D8krQUuA74haWlW9ZiZ9VaWZ72JiIXAwm7zbsh5v5jkkNzMrN8qiZM5ZmbF5KA0M8vDQWlmloeD0swsDwelmVkeDkozszwclGZmeTgozczycFCameXhoDQzy8NBaWaWh4PSzCwPB6WZWR4OSjOzPByUZmZ5OCjNzPJwUJqZ5eGgNDPLI9OglDRH0gpJrZKu72H5EEl3pssXSZqWZT1mZr2RWVBKqgRuAeYCM4ArJM3ottr7gC0RcQrwJeD/ZlWPmVlvZdmiPB9ojYi2iNgH3AHM77bOfOB76fu7gTdKUoY1mZkdsyyDciKwJmd6bTqvx3XSx9u+BNRlWJOZ2THL9HG1fUXSNcA16eQOSSuOcRP1wMa+rapklPO+Q3nvfznve29MPdKCLINyHTA5Z3pSOq+nddZKqgJGAZu6bygibgVu7W0hkloiorm3ny9l5bzvUN77X8773teyPPReDEyX1ChpMHA5sKDbOguAq9L3bwd+HRGRYU1mZscssxZlRHRIuha4F6gEvh0RSyXdCLRExALg/wE/kNQKbCYJUzOzfiXTPsqIWAgs7Dbvhpz3e4DLsqwh1evD9gGgnPcdynv/y3nf+5R8pGtmdnQewmhmlkdJB+XxDJGU9Ol0/gpJF53QwvtIb/df0jRJuyUtSV9fP+HFH6cC9v11kv4gqUPS27stu0rSM+nrqu6f7e+Oc98P5Pzdu59ctSOJiJJ8kZwgWgk0AYOBx4EZ3db5G+Dr6fvLgTvT9zPS9YcAjel2Kou9Tydw/6cBfyr2PmS879OAc4DvA2/PmV8LtKU/x6TvxxR7n07EvqfLdhR7H0rxVcotyuMZIjkfuCMi9kbEKqA13V4pKechonn3PSKejYgngM5un70I+FVEbI6ILcCvgDknoug+cjz7br1UykF5PEMkC/lsf3e8Q0QbJf1R0m8kvTbrYvvY8fz9Sv1vf7z1D5XUIukRSZf0aWUDWEkMYbQ+tx6YEhGbJL0S+JmkMyNiW7ELs8xNjYh1kpqAX0t6MiJWFruo/q6UW5THMkSSbkMkC/lsf9fr/U+7HDYBRMRjJH1ep2Zecd85nr9fqf/tj6v+iFiX/mwDHgTO68viBqpSDsrjGSK5ALg8PSvcCEwHHj1BdfeVXu+/pIb0fqGkLYvpJCc1SkUh+34k9wJvkjRG0hjgTem8UtHrfU/3eUj6vh64AFiWWaUDSbHPJh3PC7gYeJqkRfSZdN6NwLz0/VDgxyQnax4FmnI++5n0cyuAucXelxO5/8ClwFJgCfAH4C3F3pcM9v1/kPTf7SQ5ilia89m/Sn8nrcB7i70vJ2rfgVcDT5KcKX8SeF+x96VUXh6ZY2aWRykfepuZnRAOSjOzPByUZmZ5OCjNzPJwUJqZ5eGgtH6r251ulvR0p5wCttEs6eb0/dWSvtb3ldpA5yGM1p/tjoiZx7OBiGgBWvqmHCtXblFayZH0rKQvSHpS0qOSTknnXybpT5Iel/TbdN7rJf1nD9uYJunXkp6QdL+kKen870q6WdLvJbV1v5+jlScHpfVnw7oder8zZ9lLEXE28DXgy+m8G4CLIuJcYF6ebX8V+F5EnAP8ELg5Z9l44DXAm4F/7oP9sBLnQ2/rz4526H17zs8vpe9/B3xX0l3AT/Js+1XA29L3PwC+kLPsZxHRCSyTdNIxV20DjluUVqqi+/uI+CDwWZK76zwmqa6nDxZgb877gXCjYztODkorVe/M+fkwgKSTI2JRJI9Ebufw25F193sOPUf+SuChrAq10udDb+vPhklakjP9XxHRdYnQGElPkLT+rkjnfVHSdJJW4P0kd8m58Ajb/jDwHUmfIAnV9/Z18TZw+O5BVnIkPQs0R8TGYtdi5cGH3mZmebhFaWaWh1uUZmZ5OCjNzPJwUJqZ5eGgNDPLw0FpZpaHg9LMLI//H63GPW7Zqw+UAAAAAElFTkSuQmCC", 273 | "text/plain": [ 274 | "
" 275 | ] 276 | }, 277 | "metadata": { 278 | "needs_background": "light" 279 | }, 280 | "output_type": "display_data" 281 | } 282 | ], 283 | "source": [ 284 | "plt.figure(figsize=(5, 5))\n", 285 | "plt.plot(epsilons, accuracies, \"*-\")\n", 286 | "plt.yticks(np.arange(0, 1.1, step=0.1))\n", 287 | "plt.xticks(np.arange(0, 0.20, step=0.05))\n", 288 | "plt.title(\"Accuracy vs Epsilon\")\n", 289 | "plt.xlabel(\"Epsilon\")\n", 290 | "plt.ylabel(\"Accuracy\")\n", 291 | "plt.savefig('ff.svg')\n", 292 | "plt.show()" 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": null, 298 | "id": "f6f98377-1ab0-484c-8b97-12ee8cc63931", 299 | "metadata": {}, 300 | "outputs": [], 301 | "source": [] 302 | } 303 | ], 304 | "metadata": { 305 | "kernelspec": { 306 | "display_name": "Python 3", 307 | "language": "python", 308 | "name": "python3" 309 | }, 310 | "language_info": { 311 | "codemirror_mode": { 312 | "name": "ipython", 313 | "version": 3 314 | }, 315 | "file_extension": ".py", 316 | "mimetype": "text/x-python", 317 | "name": "python", 318 | "nbconvert_exporter": "python", 319 | "pygments_lexer": "ipython3", 320 | "version": "3.8.11" 321 | } 322 | }, 323 | "nbformat": 4, 324 | "nbformat_minor": 5 325 | } 326 | -------------------------------------------------------------------------------- /code/Chapter-7 & 9/README.md: -------------------------------------------------------------------------------- 1 | # Code for Chapter 7 - Explaining a PyTorch Image Classifier 2 | 3 | 4 | | Section | Notebook | 5 | | :-----------------------------------------------: | ---------------------- | 6 | | Data Preparation | [1.Data Preparation.ipynb](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-7%20%26%209/1.Data%20Preparation.ipynb "1.Data Preparation.ipynb") | 7 | | Model Training | [2.Transfer learning-Stage_1.ipynb](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-7%20%26%209/2.Transfer%20learning-Stage_1.ipynb "2.Transfer learning-Stage_1.ipynb") | 8 | | | [3.Transfer learning-Stage_2.ipynb](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-7/3.Transfer%20learning-Stage_2.ipynb "3.Transfer learning-Stage_2.ipynb") | | 9 | | Generating Post-Hoc Explanations Using Captum | [4.Post-Hoc Explanations.ipynb](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-7/4.Post-Hoc%20Explanations.ipynb "4.Post-Hoc Explanations.ipynb") | | 10 | | Assessing the Robustness of Post-Hoc Explanations | [5.Adding Noise to images](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-7/5.Adding%20Noise%20to%20images%20.ipynb) | | 11 | | | [6.Label Randomization](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-7/6.Label_Randomization.ipynb) | 12 | --- 13 | 14 | # Code for Chapter 9 - Debugging a PyTorch Image Classifier 15 | 16 | This chapter focuses on model debugging techniques for Deep Learning(DL) models using our example pneumonia classifier trained in Chapter 7. We’ll start by discussing data quality and leakage issues in DL systems and why it is important to address them in the very beginning of a project. We’ll then explore some software testing methods and why software quality assurance (QA) is an essential component of debugging DL pipelines. We’ll also perform DL sensitivity analysis approaches, including testing the model on different distributions of pneumonia images and applying adversarial attacks. We’ll close the chapter by addressing our own data quality and leakage issues, discussing interesting new debugging tools for DL, and addressing the results of our own adversarial testing. 17 | 18 | 19 | * [Adversarial Example Attacks](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-7%20%26%209/Adversarial%20Example%20Attacks.ipynb) 20 | * [Noise Injection](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-7%20%26%209/Retraining%20on%20Gaussian%20Noise.ipynb) 21 | 22 | -------------------------------------------------------------------------------- /code/Chapter-8/README.md: -------------------------------------------------------------------------------- 1 | 2 | # Code for Chapter 8 - Debugging XGBoost 3 | This chapter will introduce several methods, that go beyond traditional model assessment, to push models to their limits and find hidden problems and failure modes. The chapter starts with a concept refresher and then focuses on model debugging exercises that better simulate real-world stresses with sensitivity analysis and tests that uncover model errors with residual analysis. The overarching goal of model debugging is to increase trust in model performance with human users, but in the process, we’ll also gain an increased level of transparency into models. 4 | # Code 5 | 1. Selecting a better XGBoost Model [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-8/Selecting%20a%20Better%20XGBoost%20Model.ipynb) [![Open In GitHub](https://img.shields.io/badge/Github-code-green)](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-8/Selecting%20a%20Better%20XGBoost%20Model.ipynb) 6 | 2. Sensitivity Analysis for XGBoost - Stress Testing XGBoost [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-8/Stress_testing_XGBoost.ipynb) [![Open In GitHub](https://img.shields.io/badge/Github-code-green)](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-8/Stress_testing_XGBoost.ipynb) 7 | 3. Sensitivity Analysis for XGBoost - Adversarial Example Search [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-8/Sensitivity_Analysis_for_XGBoost_Adversarial_Example_Search.ipynb) [![Open In GitHub](https://img.shields.io/badge/Github-code-green)](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-8/Sensitivity_Analysis_for_XGBoost_Adversarial_Example_Search.ipynb) 8 | 4. Residual Analysis for XGBoost [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://githubtocolab.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-8/Residual_Analysis_for_XGBoost.ipynb) [![Open In GitHub](https://img.shields.io/badge/Github-code-green)](https://github.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/blob/main/code/Chapter-8/Residual_Analysis_for_XGBoost.ipynb) 9 | 10 | 11 | 12 | -------------------------------------------------------------------------------- /code/Data.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ml-for-high-risk-apps-book/Machine-Learning-for-High-Risk-Applications-Book/0fbf704b8b2d84579ce9473cdecabf46747a7ac2/code/Data.zip --------------------------------------------------------------------------------