├── README.md
├── feature-encoding.ipynb
├── feature-scaling.ipynb
└── missing-values-imputation.ipynb
/README.md:
--------------------------------------------------------------------------------
1 | # Scikit-learn Tutorial
2 |
3 | ## Introduction
4 | [Scikit-learn](https://scikit-learn.org/stable/) is a free software machine learning library for the Python programming language. It features various classification,
5 | regression and clustering algorithms including support vector machines, random forests, gradient boosting and k-means and is
6 | designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy.
7 |
8 | ## Repository description
9 | This repository contains 3 separate notebooks, each covering different aspects of data preprocessing for machine learning
10 | using scikit-learn, namely:
11 | - Feature encoding
12 | - Feature scaling
13 | - Missing values imputation
14 |
15 | ## Medium (Towards Data Science) articles
16 | - [Guide to Encoding Categorical Features using Scikit-Learn for Machine Learning](https://towardsdatascience.com/guide-to-encoding-categorical-features-using-scikit-learn-for-machine-learning-5048997a5c79)
17 | - [What is Feature Scaling & Why is it Important in Machine Learning?](https://towardsdatascience.com/what-is-feature-scaling-why-is-it-important-in-machine-learning-2854ae877048)
18 | - [Stop Wasting Useful Information When Imputing Missing Values](https://towardsdatascience.com/stop-wasting-useful-information-when-imputing-missing-values-d6ef91ef4c21)
19 |
--------------------------------------------------------------------------------
/feature-scaling.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "markdown",
5 | "metadata": {},
6 | "source": [
7 | "# 0. Introduction"
8 | ]
9 | },
10 | {
11 | "cell_type": "markdown",
12 | "metadata": {},
13 | "source": [
14 | "Feature scaling is the process of normalising the range of features in a dataset. \n",
15 | "\n",
16 | "Real-world datasets often contain features that are varying in degrees of magnitude, range and units. Therefore, in order for machine learning models to interpret these features on the same scale, we have to perform feature scaling.\n",
17 | "\n",
18 | "In science, we all know the importance of comparing apples to apples and yet many people, especially beginners, have a tendency to overlook feature scaling as part of the preprocessing steps for machine learning. This has proven to cause models to make inaccurate predictions. \n",
19 | "\n",
20 | "In this tutorial, we will discuss why feature scaling is important, the difference between normalisation and standardisation as well as how feature scaling affects model accuracy. More specifically, we will explore the applications of 3 different types of scalers in the Scikit-learn library: \n",
21 | "\n",
22 | "1. [MixMaxScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html)\n",
23 | "2. [StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)\n",
24 | "3. [RobustScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html)"
25 | ]
26 | },
27 | {
28 | "cell_type": "markdown",
29 | "metadata": {},
30 | "source": [
31 | "# 1. Import libraries"
32 | ]
33 | },
34 | {
35 | "cell_type": "code",
36 | "execution_count": 1,
37 | "metadata": {},
38 | "outputs": [],
39 | "source": [
40 | "# Data wrangling\n",
41 | "import pandas as pd\n",
42 | "import numpy as np\n",
43 | "\n",
44 | "# Data visualisation\n",
45 | "import seaborn as sns\n",
46 | "import matplotlib.pyplot as plt\n",
47 | "\n",
48 | "# Machine learning\n",
49 | "from sklearn.model_selection import train_test_split\n",
50 | "from sklearn.preprocessing import MinMaxScaler, StandardScaler, RobustScaler\n",
51 | "from sklearn.neighbors import KNeighborsRegressor\n",
52 | "from sklearn.svm import SVR\n",
53 | "from sklearn.tree import DecisionTreeRegressor\n",
54 | "from sklearn.pipeline import make_pipeline\n",
55 | "from sklearn.metrics import mean_squared_error"
56 | ]
57 | },
58 | {
59 | "cell_type": "markdown",
60 | "metadata": {},
61 | "source": [
62 | "# 2. Import dataset\n",
63 | "\n",
64 | "For the purpose of this tutorial, we will use one of the toy datasets in the Scikit-learn library, the [Boston house prices dataset](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html).\n",
65 | "\n",
66 | "You can find the description of the features [here](https://scikit-learn.org/stable/datasets/toy_dataset.html#boston-dataset)."
67 | ]
68 | },
69 | {
70 | "cell_type": "code",
71 | "execution_count": 2,
72 | "metadata": {},
73 | "outputs": [
74 | {
75 | "data": {
76 | "text/html": [
77 | "
"
479 | ],
480 | "text/plain": [
481 | " count mean std min 25% 50% \\\n",
482 | "CRIM 506.0 3.613524 8.601545 0.00632 0.082045 0.25651 \n",
483 | "ZN 506.0 11.363636 23.322453 0.00000 0.000000 0.00000 \n",
484 | "INDUS 506.0 11.136779 6.860353 0.46000 5.190000 9.69000 \n",
485 | "CHAS 506.0 0.069170 0.253994 0.00000 0.000000 0.00000 \n",
486 | "NOX 506.0 0.554695 0.115878 0.38500 0.449000 0.53800 \n",
487 | "RM 506.0 6.284634 0.702617 3.56100 5.885500 6.20850 \n",
488 | "AGE 506.0 68.574901 28.148861 2.90000 45.025000 77.50000 \n",
489 | "DIS 506.0 3.795043 2.105710 1.12960 2.100175 3.20745 \n",
490 | "RAD 506.0 9.549407 8.707259 1.00000 4.000000 5.00000 \n",
491 | "TAX 506.0 408.237154 168.537116 187.00000 279.000000 330.00000 \n",
492 | "PTRATIO 506.0 18.455534 2.164946 12.60000 17.400000 19.05000 \n",
493 | "B 506.0 356.674032 91.294864 0.32000 375.377500 391.44000 \n",
494 | "LSTAT 506.0 12.653063 7.141062 1.73000 6.950000 11.36000 \n",
495 | "target 506.0 22.532806 9.197104 5.00000 17.025000 21.20000 \n",
496 | "\n",
497 | " 75% max \n",
498 | "CRIM 3.677083 88.9762 \n",
499 | "ZN 12.500000 100.0000 \n",
500 | "INDUS 18.100000 27.7400 \n",
501 | "CHAS 0.000000 1.0000 \n",
502 | "NOX 0.624000 0.8710 \n",
503 | "RM 6.623500 8.7800 \n",
504 | "AGE 94.075000 100.0000 \n",
505 | "DIS 5.188425 12.1265 \n",
506 | "RAD 24.000000 24.0000 \n",
507 | "TAX 666.000000 711.0000 \n",
508 | "PTRATIO 20.200000 22.0000 \n",
509 | "B 396.225000 396.9000 \n",
510 | "LSTAT 16.955000 37.9700 \n",
511 | "target 25.000000 50.0000 "
512 | ]
513 | },
514 | "execution_count": 4,
515 | "metadata": {},
516 | "output_type": "execute_result"
517 | }
518 | ],
519 | "source": [
520 | "# Summary statistics\n",
521 | "\n",
522 | "data.describe().transpose()"
523 | ]
524 | },
525 | {
526 | "cell_type": "markdown",
527 | "metadata": {},
528 | "source": [
529 | "We can clearly observe that our features span across different range of values. This is largely attributed to the different units in which these features were measured and recorded.\n",
530 | "\n",
531 | "This is where feature scaling can help us solve this issue."
532 | ]
533 | },
534 | {
535 | "cell_type": "markdown",
536 | "metadata": {},
537 | "source": [
538 | "# 4. Understand the effects of different scalers\n",
539 | "\n",
540 | "In this section, we will learn the distinction between normalisation and standardisation. Subsequently, we will look at the effects of 3 different feature scaling techniques in Scikit-learn. "
541 | ]
542 | },
543 | {
544 | "cell_type": "markdown",
545 | "metadata": {},
546 | "source": [
547 | "# 4.1 Theory\n",
548 | "\n",
549 | "Before we examine the effects of feature scaling, let us first go over some theories behind normalisation and standardisation."
550 | ]
551 | },
552 | {
553 | "cell_type": "markdown",
554 | "metadata": {},
555 | "source": [
556 | "## 4.2.1 Normalisation\n",
557 | "\n",
558 | "Normalisation, also known as min-max scaling, is a scaling technique whereby the values in a column are shifted so that they are bounded between a fixed range of 0 and 1.\n",
559 | "\n",
560 | "X_new = (X - X_min) / (X_max - X_min)\n",
561 | "\n",
562 | "[MinMaxScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html) is the Scikit-learn function for normalisation. "
563 | ]
564 | },
565 | {
566 | "cell_type": "markdown",
567 | "metadata": {},
568 | "source": [
569 | "## 4.2.2 Standardisation\n",
570 | "\n",
571 | "On the other hand, standardisation or Z-score normalisation is another scaling technique whereby the values in a column are rescaled so that they demonstrate the properties of a standard Gaussian distribution, that is mean = 0 and variance = 1. \n",
572 | "\n",
573 | "X_new = (X - mean) / std\n",
574 | "\n",
575 | "[StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) is the Scikit-learn function for standardisation.\n",
576 | "\n",
577 | "Unlike StandardScaler, [RobustScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html) scales features using statistics that are robust to outliers. More specifically, this scaler removes the median and scales the data according to the quantile range or by default, the interquartile range, thus making it less susceptible to outliers. "
578 | ]
579 | },
580 | {
581 | "cell_type": "markdown",
582 | "metadata": {},
583 | "source": [
584 | "## 4.2.3 Normalisation vs standardisation \n",
585 | "\n",
586 | "The choice between normalisation or standardisation comes down to the application.\n",
587 | "\n",
588 | "Standardisation is generally preferred over normalisation in most machine learning context as it is especially important in order to compare the similarities between features based on certain distance measures. This is most prominent in Principal Component Analysis (PCA) where we are interested in the components that maximise the variance.\n",
589 | "\n",
590 | "Normalisation, on the other hand, also offers many practical applications particularly in computer vision and image processing where pixel intensities have to be normalised to fit within a the RGB colour range between 0 and 255. Furthermore, neural network algorithms typically require data to be normalised to a 0-1 scale before model training. \n",
591 | "\n",
592 | "At the end of the day, there is no definitive answer as to whether you should normalise or standardise your data. One can always apply both techniques and compare the model performance for the best results. "
593 | ]
594 | },
595 | {
596 | "cell_type": "markdown",
597 | "metadata": {},
598 | "source": [
599 | "# 4.2 Application \n",
600 | "\n",
601 | "Now that we have a theoretical understanding of feature scaling, let's see how they work in practice. "
602 | ]
603 | },
604 | {
605 | "cell_type": "code",
606 | "execution_count": 5,
607 | "metadata": {},
608 | "outputs": [
609 | {
610 | "name": "stdout",
611 | "output_type": "stream",
612 | "text": [
613 | "X shape: (506, 13)\n",
614 | "Y shape: (506,)\n"
615 | ]
616 | }
617 | ],
618 | "source": [
619 | "# Get predictor and target variables\n",
620 | "X = data.drop('target', axis = 1)\n",
621 | "Y = data['target']\n",
622 | "\n",
623 | "# X, Y shape\n",
624 | "print(\"X shape: \", X.shape)\n",
625 | "print(\"Y shape: \", Y.shape)"
626 | ]
627 | },
628 | {
629 | "cell_type": "code",
630 | "execution_count": 6,
631 | "metadata": {},
632 | "outputs": [],
633 | "source": [
634 | "# Instantiate MinMaxScaler, StandardScaler and RobustScaler\n",
635 | "\n",
636 | "norm = MinMaxScaler()\n",
637 | "standard = StandardScaler()\n",
638 | "robust = RobustScaler()"
639 | ]
640 | },
641 | {
642 | "cell_type": "code",
643 | "execution_count": 7,
644 | "metadata": {},
645 | "outputs": [],
646 | "source": [
647 | "# MinMaxScaler\n",
648 | "normalised_features = norm.fit_transform(X)\n",
649 | "normalised_df = pd.DataFrame(normalised_features, index = X.index, columns = X.columns)\n",
650 | "\n",
651 | "# StandardScaler\n",
652 | "standardised_features = standard.fit_transform(X)\n",
653 | "standardised_df = pd.DataFrame(standardised_features, index = X.index, columns = X.columns)\n",
654 | "\n",
655 | "# RobustScaler\n",
656 | "robust_features = robust.fit_transform(X)\n",
657 | "robust_df = pd.DataFrame(robust_features, index = X.index, columns = X.columns)"
658 | ]
659 | },
660 | {
661 | "cell_type": "markdown",
662 | "metadata": {},
663 | "source": [
664 | "To demonstrate the effects of different scalers, I have chosen to examine the following features in our dataset before and after implementing feature scaling: \n",
665 | "\n",
666 | "- ZN\n",
667 | "- AGE\n",
668 | "- TAX\n",
669 | "- B "
670 | ]
671 | },
672 | {
673 | "cell_type": "code",
674 | "execution_count": 8,
675 | "metadata": {
676 | "scrolled": false
677 | },
678 | "outputs": [
679 | {
680 | "data": {
681 | "text/plain": [
682 | "Text(0, 0.5, '')"
683 | ]
684 | },
685 | "execution_count": 8,
686 | "metadata": {},
687 | "output_type": "execute_result"
688 | },
689 | {
690 | "data": {
691 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsUAAAIYCAYAAACFa5HpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdfZxcZXnH/893kxACASLZgMCCsSRW0WorMaCWGMwDhAro7ycKtbhYlNQK1B9YH4JgAkitLT6EWkoQ6gJWiiglaCLZYALaijERiAJKFoiw8pQNCRASyG72+v1xzoaZZR9md2fnnNn5vl+vvHbuc+4555oj3nvtNfe5jyICMzMzM7NaVpd1AGZmZmZmWXNSbGZmZmY1z0mxmZmZmdU8J8VmZmZmVvOcFJuZmZlZzXNSbGZmZmY1z0mxVTVJCyR9u9x9SzhWSJpSjmOZmeWNpP+QdGHWcQwnSTMltWYdh+WHk2LLFUlnSPqNpO2SnpJ0paQJvfWPiMsi4uOlHHsgfc3MRipJGyXtlFTfbfu96R/8kyPi7yLikhKP9530fSd12/6NdPsZZYj5zZJWSNoiaaukdZJOGOpxzQo5KbbckHQ+8M/APwL7AUcDrwOaJe3RQ//RlY3QzGzEeBQ4rash6c+AcUM43kNAY8HxRgOnAA8P4ZiFbgOagQOBA4BzgefLdOwB8++fkclJseWCpH2BRcA5EfGTiGiPiI3Ah0gS47+RtFDSzZJukPQ8cEa67YaC43xU0h8kbZZ0YVoRmZ3u291X0uS0gtEo6TFJbZIuKDjOdEm/SCsST0r6t54SczOzKnU98NGCdiNwXVcjrf5emr6eKalV0vmSnknHxI91O95twLslvSZtHw+sB54qOObhkn6ajs9tkr7b9U1guu9ZSW9P2wenfWamFe3XA1dHxM703/9GxM8Ljn1yWul+XtLDko5Pt39M0oOSXpD0iKT5vV2Q9Jw/kLRJ0qOSzi3Y96rfP6VeaKseTootL94F7An8sHBjRGwDlgNz0k0nAzcDE4DvFvaVdATw78BHgINIqs2H9HPevwT+FJgFXCTpTen2XcD/B9QD70z3//0gPpeZWR7dDewr6U2SRgEfBm7oo/9reWVMPRP4VkECDPASsBQ4NW1/lIIkOyXgn4CDgTcBhwILASLiYeBzwHcl7QX8J/CdiFgNbAZagBskvV/SgUUHlaan5/pHkt8NM4CN6e5ngPcB+wIfA77elXh3O0YdSWJ/X/oZZwGflnRcQbdef//YyOCk2PKiHmiLiI4e9j2Z7gf4RUT8T0R0RsSObv0+CNwWET+PiJ3ARUD0c95FEbEjIu4jGQzfBhAR6yLi7ojoSCvWVwHvGdxHMzPLpa5q8Rzgd8Af++jbDlycfou3DNhGUlAodB3wUUn7kYyX/1O4MyJaIqI5Il6OiE3A1ygYVyPiamAD8EuSwsYF6fYAjiVJdC8HnpR0l6Sp6VvPBK5Nj90ZEX+MiN+l7/1xRDwciTuBFcAxPXy+dwCTIuLitBL9CHA1ryT50PfvHxsBPCfG8qINqJc0uofE+KB0P8DjfRzj4ML9EbFd0uZ+zvtUwevtwHgASW8gGbCnAXuR/H9lXX8fwsysilwP3EUyNaF7Vbe7zd3G5t3jZZeI+LmkScAXgR9FxA5Ju/dLOgBYTJKU7kNSmNvS7TxXk1Scz4qIlwuO3QqcnR7nUGBJGvM7SSrOy3oKWtI84EvAG9Lz7QX8poeurwMOlrS1YNso4GcF7b5+/9gI4Eqx5cUvgJeB/6dwo6S9gXnAHemmviq/TwINBe8dB0wcZDxXklROpkbEvsACkq/+zMxGhIj4A8kNdyfQberaENwAnE/PSfY/kYzhb03H1b+hYFyVNB74BnANsFDS/r3E/TjwLeAt6abHgcO795M0FvgB8K/AgRExgSR57mksfxx4NCImFPzbJyIKV7jo75tHq3JOii0XIuI5khvtrpB0vKQxkiYD3wdaSSoa/bkZOFHSu9Kb4hYx+ER2H5I7m7dJeiPwyUEex8wsz84E3hsRL5bpeItJpmPc1cO+fUimXWyVdAjJHOBC3wTWpUtn/hj4DwBJr5G0SNIUSXXpjXd/SzIvGpIk+mOSZqX7D0nH7T2AscAmoCOtGs/tJe41wPOSPidpnKRRkt4i6R2DvA5WhZwUW25ExFdJKrL/SpKQ/pLkr/dZhV+j9fH++4FzgBtJqsYvkNxk0e97e/AZ4K/TY1wN/PcgjmFmlmvpfNu1ZTzesxFxRzoPuLtFwNuB50iS3t3VaUknk6xY8XfppvOAt0v6CLATmAysJPnd8FuScf2M9JxrSG+iS499J/C6iHiBZOm2m0imafw1ydSMnuLeBZwI/DlJ9bwN+DbJzYVWI9Tzf7dm1S/9Km4ryRSIR7OOx8zMzPLLlWIbUSSdKGmvdC7yv5LcULEx26jMzMws75wU20hzMvBE+m8qcGovX+OZmZmZ7ebpE2ZmZmZW81wpNjMzM7Oa56TYzMzMzGpeLp5oV19fH5MnT846DDOzAVu3bl1bREzKOo5K8phtZtWst3E7F0nx5MmTWbu2bMskmplVjKQ/ZB1DpXnMNrNq1tu47ekTZmZmZlbznBSbmZmZWc1zUmxmZmZmNc9JsZmZmZnVPCfFZmYjmKRrJT0j6be97JekxZJaJK2X9PZKx1gObW1tnHPOOWzevDnrUMysSjkpNjMb2b4DHN/H/nkkj0SfCpwFXFmBmMquqamJ9evX09TUlHUoZlalnBSbmY1gEXEX8GwfXU4GrovE3cAESQdVJrryaGtrY/ny5UQEy5cvd7XYzAal33WKJf0p8N8Fm/4EuAi4Lt0+GdgIfCgitkgS8E3gBGA7cEZE/Lq8YZtlb/HixbS0tAzqva2trQA0NDQM+vxTpkzh3HPPHfT7zVKHAI8XtFvTbU9mE87ANTU1EREAdHZ20tTUxHnnnZdxVMPPY9DADOV6wdCvWRbXa6j/jezYsaPMEZVu3LhxFb/W/SbFEfF74M8BJI0C/gjcAnweuCMiviLp82n7cxR/FXcUyVdxRw04MrMRLMuBxqwb9bAtXtVJOotkegWHHXbYcMc0IM3NzbS3twPQ3t7OihUraiIpHgqPQQNXjdespaWF3917L68dxHtfBjrKHdBAzv/ii2xtaxvw+54awjkH+kS7WcDDEfEHSScDM9PtTcBqkqR491dxwN2SJkg6KCKqpupgVoqh/MXf9d7FixeXKxyzwWoFDi1oNwBPdO8UEUuAJQDTpk17VdKcpTlz5rBs2TLa29sZM2YMc+fOzTqkivAYNDBDrdJW4zVrbW199V+4JZpY1kgqJ3ilqj9QA51TfCrwvfT1gV2JbvrzgHR7b1/FmZlZ/iwFPpquQnE08Fy1FTEaGxtJZu5BXV0djY2NGUdkZtWo5EqxpD2Ak4Av9Ne1h21V9VWcmdlIIel7JN/q1UtqBb4EjAGIiP8AlpHcA9JCch/Ix7KJdPDq6+uZN28eS5cuZd68eUycWK01LrPyamhoYGtbG2f2mJqNTNcQTBjkXOSBTJ+YB/w6Ip5O2093TYtI71R+Jt1e9V/FmZmNFBFxWj/7A/hUhcIZNo2NjWzcuNFVYjMbtIFMnziNV6ZOQPKVW9fo0wjcWrC9qr+KMzOz6lJfX88VV1zhKrGZDVpJSbGkvYA5wA8LNn8FmCNpQ7rvK+n2ZcAjJF/FXQ38fdmiNTMz68HKlSuZMWMGq1atyjoUM6tSJU2fiIjtdLsRMSI2k6xG0b3viPgqzszMqsdll10GwCWXXMKxxx6bcTRmVo38RDszM6tqK1eupKMjWVG1o6PD1WIzGxQnxWZmVtW6qsRdLrnkkowiMbNqNtCHd5iZmeVKV5W4t7ZZLXuKZJmyStuc/qz0ra9PARMG+V4nxVbThvJc+KHYsGEDMPQnLA3WYJ8Lb5ZHo0ePLkqER4+unl9tHoNsOE2ZMiWzc29K/xubMHVqRc87gcF/7uoZOcyGQUtLC/fcf8/g/6wcrM7kxz1/vKfCJwa2Vv6UZsNpwYIFXHzxxbvbF154YYbRDExLSwsP/fbXHDZ+V0XPu0d7MnvypY2/quh5AR7bNqri56xVWf7hUY2PxXZSbDYBOmd2Zh1FxdSt9q0ENrLMnj2byy67jI6ODkaPHl11q08cNn4XX5y2LeswKubSteOzDsGsR/7taGZmVW/BggVAdVWJzSxfXCk2M7OqN3v2bGbPnp11GGZWxZwUm5lZ1bvlllv4+te/zmc+8xlOOumkrMOxYZLVjYmQ7c2JvjGxMpwUm5lZ1fvGN74BwOWXX+6keATL7OZoyO4Gad8cXTFOis3MrKrdcsstRCTrsEYES5cudWI8kvnmaBsmvtJmZlbVuqrEXS6//PKMIjGzauak2MzMqlpXlbi3tplZKZwUm5lZVZPUZ9vMrBROis3MrKp9+tOfLmqff/75GUViZtXMSbGZmVW1D3zgA7urw5J8k51ZDmzfvp3169dntoTeYDgpNjOzqtdVLXaV2CwfNm7cSGdnJ1/84hezDqVkXpLNzMyq3gc+8AE+8IEPZB2G2YgxlAelbN++nZ07dwLwxBNP8PGPf5y99tprQMfI4oElrhSbmVnVe+ihh5g3b15VfVVrNlJt3Lixz3ZeuVJsZmZV79JLL+XFF1/k4osv5rrrrss6HLOqN5Qq7YwZM4raO3fuZPHixUMNadiVVCmWNEHSzZJ+J+lBSe+UtL+kZkkb0p+vSftK0mJJLZLWS3r78H4EMzOrZQ899NDuStTGjRtdLTazQSl1+sQ3gZ9ExBuBtwEPAp8H7oiIqcAdaRtgHjA1/XcWcGVZIzYzMytw6aWXFrUvvvjijCIxs2rWb1IsaV9gBnANQETsjIitwMlAU9qtCXh/+vpk4LpI3A1MkHRQ2SM3MzOjeucvmlm+lFIp/hNgE/Cfku6R9G1JewMHRsSTAOnPA9L+hwCPF7y/Nd1WRNJZktZKWrtp06YhfQgzM6tdkydP7rNtZpVVX1/fZzuvSkmKRwNvB66MiL8AXuSVqRI96en5mq96EH1ELImIaRExbdKkSSUFa2Zm1l33dVAvuuiijCIxM4C2trY+23lVSlLcCrRGxC/T9s0kSfLTXdMi0p/PFPQ/tOD9DcAT5QnXzMzMzKz8+k2KI+Ip4HFJf5pumgU8ACwFGtNtjcCt6eulwEfTVSiOBp7rmmZhZmZWbr7RzszKodR1is8BvitpD+AR4GMkCfVNks4EHgNOSfsuA04AWoDtaV8zM7Nh4RvtzPLloIMO4sknnyxqV4OSkuKIuBeY1sOuWT30DeBTQ4zLzMysJJMnTy5KhH2jnVm2zj//fD7zmc/sbn/2s5/NMJrS+THPZmYjmKTjJf0+faDSq26SlnSYpFXp6kLrJZ2QRZxD4RvtzPKlubm5qH377bdnFMnAOCk2MxuhJI0CvkXyUKUjgNMkHdGt2xeBm9LVhU4F/r2yUQ7dV77ylaL2V7/61YwiMTOAFStWFLWdFJuZWdamAy0R8UhE7ARuJHnAUqEA9k1f70cVrhbU/bHOv/vd7zKKxMwAkpm0vbfzykmxmdnIVcrDlBYCfyOpleRG6XN6OpAfuGRmI52TYjOzkauUhymdBnwnIhpIVg66XtKrfjf4gUtmNtI5KTYzG7lKeZjSmcBNABHxC2BPoDqeyZqaMmVKUfuNb3xjRpGYGYCkPtt55aTYzGzk+hUwVdLr03XmTyV5wFKhx0iX15T0JpKkuKrmR1x77bVF7SVLlmQUiZkBzJ07t6h93HHHZRTJwDgpNjMboSKiAzgbuB14kGSVifslXSzppLTb+cAnJN0HfA84I6rlrpgCXdViV4nNsnfKKacUtT/0oQ9lFMnAlPpEOzMzq0IRsYzkBrrCbRcVvH4AeHel4yq37tViM8vODTfcUNS+/vrrWbRoUUbRlM5JsZmZVb01a9bw2c9+lssvv5wjjzwy63BK1trayosvjOLSteOzDqVi/vDCKPZubc06DBtGq1evLmqvWrWqKpJiT58wM7Oqt3DhQjo7O7nwwguzDsXMqpQrxWZmVtXWrFnDtm3bANi2bRvr1q2rmmpxQ0MDL3U8yRenbcs6lIq5dO149mxoyDoMG0aSih7Y4dUnzMzMKmDhwoVFbVeLzbJ19NFHF7Xf9a53ZRTJwDgpNjOzqtZVJe6tbWaVte+++xa199lnn4wiGRgnxWZmVtXGjx/fZ9vMKuvOO+8sane/8S6vnBSbmVlV6z594pJLLskmEDMDYPTo0X2288pJsZmZVbXp06fvrg6PHz++am6yMxupqnVKk5NiMzOregsXLqSurs5VYrMc2Hvvvfts51V11LPNzMz6MH369KqZt2g20u3YsaPPdl6VVCmWtFHSbyTdK2ltum1/Sc2SNqQ/X5Nul6TFklokrZf09uH8AGZmZmvWrGHmzJmsW7cu61DMal5dXV2f7bwaSJTHRsSfR8S0tP154I6ImArckbYB5gFT039nAVeWK1gzM7Oe+Il2Zvkxa9asovbs2bMzimRghpK6nww0pa+bgPcXbL8uEncDEyQdNITzmJmZ9aqnJ9qZWXbmz5+/uzpcV1fH/PnzM46oNKUmxQGskLRO0lnptgMj4kmA9OcB6fZDgMcL3tuabjMzMys7P9HOLF/q6+uZMWMGAO95z3uYOHFixhGVptQb7d4dEU9IOgBolvS7Pvr29IDreFWnJLk+C+Cwww4rMQwzM7Ni1br8k9lINnbs2KKf1aCkSnFEPJH+fAa4BZgOPN01LSL9+UzavRU4tODtDcATPRxzSURMi4hpkyZNGvwnMDOzmuYn2pnlS1tbG6tWrQJg1apVbN68OeOIStNvUixpb0n7dL0G5gK/BZYCjWm3RuDW9PVS4KPpKhRHA891TbMwMzMrNz/RzixfmpqaiEgmCXR2dtLU1NTPO/KhlErxgcDPJd0HrAF+HBE/Ab4CzJG0AZiTtgGWAY8ALcDVwN+XPWozM7OUn2hnli/Nzc20t7cD0N7ezooVKzKOqDT9JsUR8UhEvC399+aI+HK6fXNEzIqIqenPZ9PtERGfiojDI+LPImLtcH8IMzOrbX6inVl+zJkzh1GjRgEwatQo5s6dm3FEpamO1ZTNzMz60PVEO1eJzbLX2NhIZ2cnkEyfaGxs7Ocd+eCk2MzMzMzK5tlnn909pzgi2LJlS8YRlcZJsZmZmZmVzaWXXlrUvvjiizOKZGCcFJuZmZlZ2WzcuLHPdl45KTYzMzOzspk8eXKf7bxyUmxmZmZmZXP22WcXtf/hH/4ho0gGxkmxmZmZmZXNz372s6L2nXfemVEkA+Ok2MzMzMzKprm5uag9Yh7eYWZmZmZWqmOOOaaoPWPGjIwiGRgnxWZmZmZW85wUm5mZmVnZdJ9TfNddd2UUycA4KTYzMzOzspkzZ05Re+7cuRlFMjBOis3MRjBJx0v6vaQWSZ/vpc+HJD0g6X5J/1XpGMth5cqVzJgxg1WrVmUdilnNO/HEE4vaJ510UkaRDIyTYjOzEUrSKOBbwDzgCOA0SUd06zMV+ALw7oh4M/DpigdaBpdddhkAl1xyScaRmNn3v//9ovZNN92UUSQD46TYzGzkmg60RMQjEbETuBE4uVufTwDfiogtABHxTIVjHLKVK1fS0dEBQEdHh6vFZhm74447itorV67MKJKBcVJsZjZyHQI8XtBuTbcVegPwBkn/K+luScdXLLoy6aoSd3G12CxbEdFnO6+cFJuZjVzqYVv3306jganATOA04NuSJrzqQNJZktZKWrtp06ayBzoUXVXi3tpmVllep9jMzPKmFTi0oN0APNFDn1sjoj0iHgV+T5IkF4mIJRExLSKmTZo0adgCHozRo0f32Tazyho7dmyf7bxyUmxmNnL9Cpgq6fWS9gBOBZZ26/M/wLEAkupJplM8UtEoh2jBggVF7QsvvDCjSMwMvE6xmZnlTER0AGcDtwMPAjdFxP2SLpbUtUbS7cBmSQ8Aq4B/jIjN2UQ8OLNnz95dHR49ejTHHntsxhGZ1bYRP31C0ihJ90j6Udp+vaRfStog6b/TKgSSxqbtlnT/5OEJ3czM+hMRyyLiDRFxeER8Od12UUQsTV9HRJwXEUdExJ9FxI3ZRjw4XdViV4nNbLAGUin+B5JKQ5d/Br4eEVOBLcCZ6fYzgS0RMQX4etrPzMxs2MyePZu77rrLVWKzHBjR0yckNQB/BXw7bQt4L3Bz2qUJeH/6+uS0Tbp/VtrfzMxsWKxZs4aZM2eybt26rEMxq3lz5swpmtJULY95LvUW3W8AnwX2SdsTga3pfDUoXvty97qYEdEh6bm0f1vhASWdBZwFcNhhhw02fjOroMWLF9PS0jKo97a2trJjx44yR1S6cePG0dDQMKj3TpkyhXPPPbfMEVk5LVy4kM7OTi688EKWLVuWdThmNa2xsZHly5cDMGrUKBobGzOOqDT9JsWS3gc8ExHrJM3s2txD1yhh3ysbIpYASwCmTZtWHas6m9W4lpYWfnfvvbx2EO99Gchy9diXX3yRrW1t/Xfs5qlhiMXKa82aNWzbtg2Abdu2sW7dOo488siMozKrXfX19cybN4+lS5cyb948Jk6cmHVIJSmlUvxu4CRJJwB7AvuSVI4nSBqdVosL177sWhezVdJoYD/g2bJHbmYV19ra+uq/cEtUHUPiqwXJ57b8WrhwYVHb1WKz7J144omsXLmSk046qf/OOdHvnOKI+EJENETEZJI1Ln8aER8hWbrng2m3RuDW9PXStE26/6dRLc/3MzOzqtNVJe6tbWaVd9ttt7F9+3aWLu2+NHp+DeWxP58DbpR0KXAPcE26/RrgekktJBXiU4cWopnlRUNDA1vb2jizx1lSI9M1BBMGORfZKmP8+PFFifD48eMzjMbM2traWL58ORHB8uXLaWxsrIopFAN6eEdErI6I96WvH4mI6RExJSJOiYiX0+0vpe0p6f6qejKSmZlVl+7TJy655JJsAjEzAJqamuiaJNDZ2UlTU1M/78gHPyDezAbkKZLqaSV1PV4tizrDU8CEDM5rZlatmpubaW9vB6C9vZ0VK1Zw3nnnZRxV/5wUm1nJpkyZksl5N23YAMCEqVMrfu4JZPe5rTS+0c4sX+bMmcOyZctob29nzJgxI26dYjOzzNbq7Trv4sWLMzm/5ZtvtDPLl8J1iuvq6qpmneIBzSk2MzPLm+431vlGO7Nsda1TLKmq1il2UmxmZlXNN9qZ5U9jYyNvfetbq6ZKDJ4+YWZmVW769Om7l2UbP368n2ZnlgP19fVcccUVWYcxIK4Um1nubd++nfXr19PS0pJ1KJZTCxcupK6uzlViMxs0J8VmlnuPPvoonZ2dfOELX8g6FMup6dOns3r1aleJzWzQnBSbWa499NBDu9e7fPrpp10tNjOzYeE5xWZWEYsXLx5UQnv//fcXtefPn8+b3/zmAR9nypQpmS0pZ2Zm+edKsZnlWleVuLe2mZlZObhSbGYVMdgq7YwZM161zQ/xMDOzcnOl2MzMzMxqnivFVtNaW1vhOahbXUN/H26F1mjNOgqzsrr++uu5+uqr+eQnP8lpp52WdThmNa+trY1FixaxcOFCP9HOzMysUq6++moArrzyyowjMTOApqYm1q9fT1NTU9ahlMyVYqtpDQ0NbNImOmd2Zh1KxdStrqPhkIaswyjZmDFjim6uGzNmTIbRWB5df/31Re3vfe97rhabZaitrY3ly5cTESxfvpzGxsaqqBa7UmxmuebVJ6w/XVXiLq4Wm2WrqamJzs6k2LRr166qqRY7KTYzMzOzsmlubqajowOAjo4OVqxYkXFEpXFSbGZmZmZlc8wxxxS1e1paM4/6nVMsaU/gLmBs2v/miPiSpNcDNwL7A78GTo+InZLGAtcBRwKbgQ9HxMZhit/MRrh99tmHF154YXd73333zTAay6NPfOITRVMoPvnJT2YYzcA9tm0Ul64dX9FzPr09qYkduFfl76d4bNso3lDxs5r1r5Qb7V4G3hsR2ySNAX4uaTlwHvD1iLhR0n8AZwJXpj+3RMQUSacC/wx8eJjiN7MRbuzYsUVJ8R577JFhNJZHp59+elFSXE032U2ZMiWT8+7csAGAPSdPrfi530B2n9sq46677ipq33nnnSxYsCCjaErXb1IcEQFsS5tj0n8BvBf463R7E7CQJCk+OX0NcDPwb5KUHsfMbEDa2tr6bJvBK9XiaqsSD/ZJj+U6r58OacPhwAMPZOPGjUXtalDSkmySRgHrgCnAt4CHga0R0ZF2aQUOSV8fAjwOEBEdkp4DJgL+TWZmZsPi9NNP5/TTT886DDMDnn766T7beVXSjXYRsSsi/hxoAKYDb+qpW/pTfezbTdJZktZKWrtp06ZS4zUzM3uVq666ihkzZnDNNddkHYpZzZs7dy5Skg5K4rjjjss4otIMaPWJiNgKrAaOBiZI6qo0NwBPpK9bgUMB0v37Ac/2cKwlETEtIqZNmjRpcNGb2Yg3bty4PtvWN0nHS/q9pBZJn++j3wclhaRplYyvXL773e8CVM16qGYjWWNj4+4HLY0ZM4bGxsaMIypNv0mxpEmSJqSvxwGzgQeBVcAH026NwK3p66Vpm3T/Tz2f2MwGa8eOHX22rXfp1LdvAfOAI4DTJB3RQ799gHOBX1Y2wvK46qqritquFptlq76+nnnz5iGJE044oSqeZgelVYoPAlZJWg/8CmiOiB8BnwPOk9RCMme4axS6BpiYbj8P6LUyYWZmw2o60BIRj0TETpJlNE/uod8lwFeBlyoZXLl0VYm7uFpslr3Gxkbe+ta3Vk2VGEpbfWI98Bc9bH+EZMDtvv0l4JSyRGdmNa+urm7340K72lay3Tc+p1qBowo7SPoL4NCI+JGkz/R2IElnAWcBHHbYYcMQqpmNJPX19VxxxRVZhzEgJa0+YTaibYW61RVOtLoWOazsev2JrbyyVkwVmDFjBqtXr97dfs973pNdMNWnzxufJdUBXwfO6O9AEbEEWAIwbdo0T4kzsxHHSbHVtKwWkN+QLpw/9ZDKL5zPIdW1cP7YsWP7bFufdt/4nCq8KRpgH+AtwOr0TvHXAkslnRQRaysW5RB95CMfKZpCUU1f14fIVmoAACAASURBVJpZfjgptprmhfPz72c/+1lR+6677qqKJyPlxK+AqZJeD/wROJVXHrpERDwH1He1Ja0GPlNNCTHA/Pnzi5LiM888M8NozKxaeXKemeXa9OnFty4cddRRvfS07tIHLJ0N3E6yatBNEXG/pIslnZRtdOX1kY98BHCV2MwGz5ViM8u1lpaWonbX1BMrTUQsA5Z123ZRL31nViKm4TB//nzmz5+fdRhmVsVcKTazXGttbe2zbQawZs0aZs6cybp167IOxcyqlJNiM8u18ePH99k2A1i4cCGdnZ1ceOGFWYdiZlXKSbGZ5Vp7e3ufbbM1a9awbVuyzuG2bdtcLTazQXFSbGa5dtBBB/XZNlu4cGFR29ViMxsMJ8VmlmtPPfVUn22zripxb20zq7y2tjbOOeccNm/enHUoJXNSbGa5NmnSpD7bZmaWP01NTaxfv56mpqasQymZk2Izy7Unnniiz7aZmeVLW1sby5YtIyJYtmxZ1VSLnRSbmZmZWdk0NTXR0dEBJDdHV0u12EmxmeXaAQcc0GfbzMzyZcWKFUQEABHB7bffnnFEpXFSbGa59swzz/TZNhs9enSfbTOrrAMPPLDPdl45KTazXJPUZ9tswYIFRW0vyWaWraeffrrPdl45KTazXJs1a1ZRe/bs2RlFYnk1e/bs3dXh0aNHc+yxx2YckVltmzt37u4ChiSOO+64jCMqjZNiM8u1+fPn99k2g1eqxa4Sm2WvsbGRMWPGADBmzBgaGxszjqg0nnhlZrlXV1dHZ2cndXX+O34kW7x4MS0tLYN6b2trK/X19dxyyy3ccsstgzrGlClTOPfccwf1XjN7RX19PfPmzWPp0qWccMIJTJw4MeuQSuLfMGaWa01NTbuT4bq6uqpZ2scqa8eOHezYsSPrMMws1djYyFvf+taqqRJDCZViSYcC1wGvBTqBJRHxTUn7A/8NTAY2Ah+KiC1KJpF8EzgB2A6cERG/Hp7wzWyka25u3r3eZUdHBytWrOC8887LOCobDkOp0na9d/HixeUKx8xqTCmV4g7g/Ih4E3A08ClJRwCfB+6IiKnAHWkbYB4wNf13FnBl2aM2s5oxZ86corlpc+fOzTgiMzPrz4h8zHNEPNlV6Y2IF4AHgUOAk4GuT9oEvD99fTJwXSTuBiZIOqjskZtZTWhsbNx9F3NdXV1VfRVnZlaL2traWL58ORHB8uXLR+ZjniVNBv4C+CVwYEQ8CUniDHQ9ZuoQ4PGCt7Wm27of6yxJayWt3bRp08AjN7Oa0HXDhiTmzZtXNTdsmJnVqqampt1PtOvs7KyaanHJq09IGg/8APh0RDzfxwL6Pe2IV22IWAIsAZg2bdqr9puZdWlsbGTjxo2uEpvVuNbWVngO6lbX0DoBW6E1WrOOYkCam5tpb28HoL29vWruBSnpvypJY0gS4u9GxA/TzU93TYtIf3Y9e7UVOLTg7Q3AE+UJ18zMzMzybM6cOYwaNQqAUaNGVc29IKWsPiHgGuDBiPhawa6lQCPwlfTnrQXbz5Z0I3AU8FzXNAuzkWQoa6pu2LABGNrd9rW0pmrhDRvVUG0ws+HR0NDAJm2ic2Zn1qFUTN3qOhoOacg6jAFpbGzk1luTtLCzs7NqvuUrpVL8buB04L2S7k3/nUCSDM+RtAGYk7YBlgGPAC3A1cDflz9ss+o2btw4xo0bl3UYVaHwho1ly5ZVzQ0bZma16tlnn939OiLYsmVLhtGUrt9KcUT8nJ7nCQPM6qF/AJ8aYlxmuVcrVdqsNTU1Fc1Nc7XYzCzfFi1aVNT+0pe+xHe/+92MoildDc1UN7NqtGLFit13MUcEt99+e8YRmZlZXx5//PE+23nlpNjMcq2+vr7PtpmZWTk4KTazXPvjH//YZ9vMzPKl+7K9fSzjmytOis0s1+rq6vpsm5lZvnRfgu24447LKJKB8W8XM8u1WbOK7+edPXt2RpGYmVkpjjrqqKL2O9/5zowiGRgnxWaWa/Pnz99dHa6rq2P+/PkZR2RmZn352te+VtT+l3/5l4wiGRgnxWaWa/X19cyZMwdIvpKbOHFixhGZmVlftm3b1mc7r5wUm1nuzZ8/n7e97W2uEg+CpOMl/V5Si6TP97D/PEkPSFov6Q5Jr8siTjMbOUaPHt1nO6+cFJtZ7tXX13PFFVe4SjxAkkYB3wLmAUcAp0k6olu3e4BpEfFW4Gbgq5WN0sxGmmq9Qbo6ojSzmtbW1sY555zjRzwP3HSgJSIeiYidwI3AyYUdImJVRGxPm3cDDRWO0cxGmIMPPrjPdl45KTaz3GtqamL9+vU0NTVlHUq1OQQofJRUa7qtN2cCy3vaIeksSWslrd20aVMZQzSzkebpp5/us51XTorNLNfa2tpYtmwZEcGyZctcLR6YnlbMjx47Sn8DTAN6vE08IpZExLSImDZp0qQyhmhmI83cuXN3P7BDktcpNjMrh6amJjo6OgBob293tXhgWoFDC9oNwBPdO0maDVwAnBQRL1coNjMboRobG/ts55WTYjPLtRUrVhCRFDcjgttvvz3jiKrKr4Cpkl4vaQ/gVGBpYQdJfwFcRZIQP5NBjGY2AhWO29WiOtbIMLOadeCBB7Jx48aitpUmIjoknQ3cDowCro2I+yVdDKyNiKUk0yXGA99Pv+58LCJOyixos/5shbrVGdT0upbaHV/h826l7zsBcuiqq656VXvBggUZRVM6J8VmlmvVesNGXkTEMmBZt20XFbz2c7OtakyZMiWzc2/YsAGAqYdMreyJD8n2cw/GypUri9rNzc1Ois3Mhmru3Lnceuutu9vVcsOGmZXfueeem/m5Fy9enFkM1aLrJrve2nnlOcVmlmuNjY27F36vq6urmhs2zMxq1axZs4ras2dXxxdSTop74YcFmJmZmQ3cKaecUtT+0Ic+lFEkA9NvUizpWknPSPptwbb9JTVL2pD+fE26XZIWS2qRtF7S24cz+OHkhwWY5UNTUxOdnZ0AdHZ2+v+TZmY5d9tttxWtU7x06dJ+3pEPpVSKvwMc323b54E7ImIqcEfaBpgHTE3/nQVcWZ4wK6utrY3bbruNiOC2225ztdgsQ92XYPvJT36SUSRmZlaK5ubmoiXZVqxYkXFEpen3RruIuEvS5G6bTwZmpq+bgNXA59Lt10VyJe6WNEHSQRHxZLkCroTCytSuXbtoamrivPPOyziq4bd48WJaWloG9d7W1lYAGhoaBvX+KVOmZHoDheVX13zi3tpmZpYvc+bM4cc//jEdHR2MHj2auXPnZh1SSQb72+XArkQ3/XlAuv0Q4PGCfq30srqepLMkrZW0dtOmTYMMY3gsX768qL1s2bJeelqXHTt2sGPHjqzDsBFo+/btfbbNzCxfGhsbi6a9VcsN0uVekq2nNTd6fJRJRCwBlgBMmzYtV4872blzZ5/tPBtKtTdLLS0tQ6oUu9JsZmaWH7X0RLunu6ZFSDoI6Ho0aCtwaEG/BuCJoQRoA9PS0sJDv/01h43fVdHz7tGefOnw0sZfVfS8AI9tG1Xxc1rl7L///jz77LO72xMnTswwGjMz689VV11VlBSP9CfaLQUaga+kP28t2H62pBuBo4Dnqm0+8Uhw2PhdfHHatv47jhCXrq30MzetkrpPl3jxxRczisRKkdW3VV1PG8vqGyN/W2X2ijvuuKOovXLlypGRFEv6HslNdfWSWoEvkSTDN0k6E3gM6FqQbhlwAtACbAc+Ngwxm1kNeemll/psW760tLRwz28eoHOv/St6Xu1MqlLrHn6qoucFqNv+bP+dzGpI9ykT1TKFopTVJ07rZdes7hvSVSc+NdSgzMysenXutT8vHfG+rMOomD0f+FHWIZjlyjHHHMPq1at3t2fMmJFdMANQ7hvtLGOtra28+MKomppS8IcXRrF3uiScjTyjRo1i165dRW0zM8uvsWPH9tnOKy/4aWa5Nn369KL2UUcdlVEkZmZWip/97GdF7bvuuiujSAbGleIRpqGhgZc6nqy5G+32HORDQyz/Wrt9C/D444/30tPMzPJgzpw5LF26lIhA0oh/eIeZWUV0T4KdFJuZ5duJJ55YtCTbSSedlHFEpXFS3IMxY8b02Tazyjn00EP7bJuZWb7cdtttSMnz3CSxdOnSjCMqjZPiHhTe1NNT28wq5/DDDy9qT5kyJaNIzMysFM3NzUWV4hUrVmQcUWmcFPeg63ndvbXNrHJ++ctfFrXvvvvujCIxM7NSHHPMMUXtalmSzUmxmeVafX19n20zM8uX559/vs92XjkpNrNce+KJJ/psm5lZvvziF78oav/f//1fRpEMjJdkG4Ee21b5h3c8vT35++rAvSo/1eSxbaN4Q8XPamZmZiOJk+IRJqubkHZu2ADAnpOnVvzcb8A3X41kBx10UNFaxQcddFCG0ZiZ2UjlpHiEOffcczM97+LFizM5v41cbW1tfbbNzCxfxo0bx44dO4ra1cBzis0s11772tf22TYzs3wpTIh7aufViK4UL168mJaWlrIca6AV2ClTpmRWtTUbSXyjnZmZVYIrxWaWa36YjplZdTn44IP7bOfViK4UD7ZS+6UvfYlVq1btbs+ZM4cLL7ywXGGZ2QA4KTYzqy5btmzps51XrhT3YNGiRUVtJ8RmZmZmpTnqqKOK2kcffXRGkQyMk+Je7LvvvkBSJTYzMzOz0vz+97/vs51XuZ8+Uc6b5QZi165d7L333mzatCmTG+Z8o56ZmZlVoyeffLKoXS03SA9LUizpeOCbwCjg2xHxlcEeq6WlhXt+8wCde+1ftvhKoZ0BwLqHn6roeQHqtj9b8XOa5dUee+zBzp07i9pWuv7GY0ljgeuAI4HNwIcjYmOl4zQzy1rZk2JJo4BvAXOAVuBXkpZGxAODPWbnXvvz0hHvK1eIubfnAz/KOgSz3ChMiHtqW+9KHI/PBLZExBRJpwL/DHy48tGamWVrOCrF04GWiHgEQNKNwMnAoJNiq4yhTFXZkD7mebBTPjxdxGxYlDIenwwsTF/fDPybJEVEVDJQM7OsDUdSfAjweEG7FTiql779am1tpe6Fzey1tmngb+7cBVmN6xLUjRrce3d10NraUd54hlm1PMLRrMaUMh7v7hMRHZKeAyYCg3qedtWO2TD4cTujMTvLQgZUXzFjqPcoufgz8g1HUqwetr1qlJN0FnAWwGGHHdbrwSZMmDDoxwO+/PLLdHZ2Duq9Q1VXV8fYsYOd+7gHEyZMKGs8pfD/Wc1GnFLGY4/ZqcGP29mM2UPhQsbA+ZqNfMORFLcChxa0G4BX3XYYEUuAJQDTpk3rtTRw7bXXljs+M6siRx55JOvWrdvdfsc73pFhNFWnlPG4q0+rpNHAfsCr7vb1mJ0/LmQMjK9X5cycOZPVq1fvbh977LHZBTMAw7FO8a+AqZJeL2kP4FRg6TCcx8xqwAUXXFDUXrBgQUaRVKVSxuOlQGP6+oPATz2f2MyGovsfINXyB0nZk+KI6ADOBm4HHgRuioj7y30eM6sN9fX1HHnkkUBSJZ44cWLGEVWP3sZjSRdLOintdg0wUVILcB7w+WyiNbORor6+npkzZwJJlbhaxu1hWac4IpYBy4bj2GZWey644AIWLVrkKvEg9DQeR8RFBa9fAk6pdFxmNrKde+65bNmypWqqxFAFT7QzM6uvr+eKK67IOgwzMytRNY7bwzGn2MzMzMysqjgpNjMzM7Oa56TYzMzMzGqek2IzMzMzq3nKw3KUkjYBf8g6jh7UM8hHndYoX6+B8fUamLxer9dFxKSsg6gkj9kjhq/XwPmaDUxer1eP43YukuK8krQ2IqZlHUe18PUaGF+vgfH1sv74v5GB8fUaOF+zgam26+XpE2ZmZmZW85wUm5mZmVnNc1LctyVZB1BlfL0GxtdrYHy9rD/+b2RgfL0GztdsYKrqenlOsZmZmZnVPFeKzczMzKzmOSk2MzMzs5pX80mxpA9Iurfbv05Jn5QUks4p6Ptvks7IMNxMpdcqJL2xYNtUST+S9LCkdZJWSZqR7jtD0qZu1/aI7D5BZUiaWPB5n5L0x4L2Hr1cx2mSfitpj7R9uKRHJO2b3SfJB0m70mt3n6RfS3pX1jFZtjxul8Zjduk8bpdPNY/ZNZ8UR8QtEfHnXf+Afwd+BtwOPAP8Q9d/8MZpwM+BUwEk7Qn8GFgSEYdHxJHAOcCfFLznvwuvb0Q8UPGoKywiNhf89/QfwNcLPv9Oul3H9D1rgbuAz6SbvgVcEBHPVzj8PNqRXru3AV8A/inrgCxbHrdL5jG7RB63y6pqx+yaT4oLSXoDcBFwOtAJbALuABqzjCsPJI0H3g2cySuDwkeAX0TE0q5+EfHbiPhO5SOsDr1cxy4LgI9L+iwwJiK+V+n4qsC+wJasg7D88LjdM4/Z5eNxe0iqaswenXUAeSFpDPBfwGci4jFJk9NdXwGWS7o2q9hy4v3ATyLiIUnPSno78Gbg1/2878OS/rKg/c6I2DFsUebfq65jRPwaICK2SvpnkqpXTXxlWaJxku4F9gQOAt6bcTyWEx63++Qxu3w8bg9M1Y7ZrhS/4hLg/oi4sXBjRDwKrAH+OpOo8uM0oOva3Ji2i0i6JZ1f9cOCzd2/iqv1wbW/6zgPeBoProW6vop7I3A8cJ0kZR2U5YLH7d55zC4fj9sDU7VjtivFgKSZwP8LvL2XLpcBN5PMHao5kiaS/KX3FkkBjAICWATM6OoXER+QNA3410wCzbnerqOkz0ZESHofsB9wHHCLpNsjYnuGIedORPxCUj0wiWTuqNUoj9u985hdPh63h6baxuyarxRLeg3wn8BHI+KFnvpExO+AB4D3VTK2HPkgcF1EvC4iJkfEocCjwEPAuyWdVNB3r0wirA69Xce/lDQOuBz4VET8BrgVuCDDWHMpvfN7FLA561gsOx63++Uxu3w8bg9BtY3ZrhTD3wEHAFd2q+53nyz/ZeCeSgWVM6eRzNEr9AOSrybfB3xN0jdIvj56Abi0oF/3+Wl/HxH/N5zB5lhf13Ee8D8Fd3ovBO6V9J2I2FC5EHOpa34agIDGiNiVZUCWOY/bffOYXT4etweuasdsP+bZzMzMzGpezU+fMDMzMzNzUmxmZmZmNc9JsZmZmZnVPCfFZmZmZlbznBSbmZmZWc1zUmxmZmZmNc9JsZmZmZnVPCfFZmZmZlbznBSbmZmZWc1zUmxmZmZmNc9JsZmZmZnVPCfFZmZmZlbznBRb1ZA0U1JrGY83WVJIGl2uY/ZwjrLGbGaWNUlnSPp51nEMlqSFkm7IOg7LHyfFNmCS/lLS/0l6TtKzkv5X0jtGwED5ZkkrJG2RtFXSOkknZB2XmdlwkLRR0g5J2yQ9Jek7ksZnGM9qSR/vtu1kSfdKel5Sm6Q7JE3OJkIb6ZwU24BI2hf4EXAFsD9wCLAIeDnLuPpTYjX4NqAZOBA4ADgXeH444+rLcFawzcxSJ0bEeODPgb8AvpBxPLtJmgJcB5wP7Ae8Hvh3oDOjeCTJedMI5v9xbaDeABAR34uIXRGxIyJWAO3AfwDvTKsOWwEk/ZWke9K/8h+XtLDrQAXTFxolPZZWAS4o2D8urVxskfQA8I7CQCR9XtLDkl6Q9ICkDxTsOyOtYH9d0rPAQkmjJP1rep5HgL8q6F9PMuBeHRE703//GxE/L+hTWLF4WNLx6faPSXowjeMRSfN7u3iSDpb0A0mbJD0q6dyCfQsl3SzpBknPA2cM6H8ZM7NBioingNtJkmMk7SfpunSs+oOkL3ZLCCXpivQbw99JmlWwY6Ok2QXt3dMVJO2ZjnGb02/kfiXpQElfBo4B/i39HfJvaSyPRsQdkXghIn4QEY+lxxolaUHB74F1kg5N930z/Z3zfLr9mN4+u6Sj028/t0q6T9LMgn2rJX1Z0v8C24E/Geq1tvxyUmwD9RCwS1KTpHmSXgMQEQ8Cfwf8IiLGR8SEtP+LwEeBCSRJ6Cclvb/bMf8S+FNgFnCRpDel278EHJ7+Ow5o7Pa+h0kG0f1IqtU3SDqoYP9RwCMkVd8vA58A3kdSDZkGfLCg72agJT3G+yUdWHgiSdNJKhb/mH6WGcDGdPcz6XH3BT4GfF3S27tfuPQXym3AfSQV9lnApyUdV9DtZODm9Bzf7X4MM7PhIKkBmEcyDkLybeB+JEnge0jG8Y8VvKVrfK0nGat/KGn/Ek7VmB73UGAiye+NHRFxAfAz4Oz0d8jZwK+BN6bFjWP16qkd5wGnASeQjL9/S5K4AvyKJKneH/gv4PuS9uzhcx8C/Bi4NO37GeAHkiYVdDsdOAvYB/hDCZ/RqpSTYhuQiHieJIkN4Gpgk6Sl3ZPIgv6rI+I3EdEZEeuB75EMsIUWpRXn+0gSxrel2z8EfDkino2Ix4HF3Y79/Yh4Ij32fwMbgOkFXZ6IiCsioiMidqTH+0ZEPB4RzwL/VHCsAI4lSXQvB56UdJekqWmXM4FrI6I5Pd8fI+J36Xt/HBEPp5WMO4EVJMl6d+8AJkXExWkl+pH0Gp5a0OcXEfE/6Tl29HRNzczK6H8kvQA8TvIH/pckjQI+DHwhrc5uJBkXTy943zMk42l7Ov7+noJv3/rQTpIMT0m/bVyX/l55lXSMnElSRLgJaFPxvOePA1+MiN+n4+99EbE5fe8NEbE5Hf8vB8aSFF+6+xtgWUQsS8fdZmAtSaLd5TsRcX96rPYSPqNVKSfFNmAR8WBEnBERDcBbgIOBb/TUV9JRklalX8E9R1IVqO/W7amC19uBrgHvYJKBukvRX+iSPppOZ9iqZLrGW7odu/C9/R4vIloj4uyIOBx4HUmV+7p096EklemePuM8SXcruelwK8lg2v0zkh7z4K54074LSOYw9xazmdlwen9E7EOSfL6RZOyqB/ageIz8A0ly2uWPaTGhcP/BJZzvepJpGjdKekLSVyWN6a1zRNwdER+KiEkkxYYZQNc0u77G5fPTaW3PpWPtfvQ+Lp/SbVz+S6DwW0ePyzXCSbENSVot/Q5JQho9dPkvYClwaETsRzLvWCUe/kmSQa/LYV0vJL2OpMp6NjAxna7x227H7h5Pr8frLq1Mf4vkc0EyKB7evZ+kscAPgH8FDkzjWEbPn/FxkvlxEwr+7RMRhRWJnq6hmdmwSr/l+g7JWNZGUtF9XUGXw4A/FrQPkaRu+59IX78I7FWw77UF52mPiEURcQTwLpKpZx/t2t1PjL8Cfkj/4/IxwOdIvh18TTouP0fv4/L13cblvSPiK4Wn7isuGzmcFNuASHpj+hd4Q9o+lGRO193A00CDpD0K3rIP8GxEvJTOy/3rAZzuJuALkl6Tnu+cgn17kwxUm9I4PsYrA2VfxztXUkM6F/rzBZ/rNZIWSZoiqU7JjXd/m34ugGuAj0male4/RNIbSaopY9M4OiTNA+b2cv41wPOSPqfkJsJRkt4i6R299Dczq6RvAHOAPyMZL78saZ+0CHEeULi27wEk4+kYSacAbyIpCADcC5ya7iu6fyOdG/xn6RSN50mS713p7qcpuJFNyfKfn5B0QNp+I3ASr4zL3wYukTRVibdKmkjye6eDZFweLekikjnHPbkBOFHScemYvKeS9eUbBnjtbARwUmwD9QLJDRa/lPQiyeD0W5Ilc34K3A88Jakt7f/3wMXpnLWLSAbaUi0i+UruUZJ5utd37YiIB0jmuP2CZCD9M+B/+zne1SRf291HcgPHDwv27QQmAytJBurfkiwzd0Z6vjWkN9GRVBzuBF4XES+QLN12E7CFJOlf2tPJI2IXcCLpHdUk1Zhvk3ytZ2aWqYjYRDJl7EKSIsSLJDfT/ZzkW79rC7r/EphKMo59Gfhg13ze9P2Hk4yJi9L3dnktyc3EzwMPkoylXcn2N4EPKllxaDGwlSQJ/o2kbcBPgFuAr6b9v0Yy9q5Ij3cNMI5knF9OcmP4H4CX6GUKRPqt4MkkU9k2pf3+EedHNUnFU4LMzMzMzGqP/xIyMzMzs5rnpNjMzMzMap6TYjMzMzOreU6KzczMzKzmjc46AID6+vqYPHly1mGYmQ3YunXr2tIHC9QMj9lmVs16G7dzkRRPnjyZtWvXZh2GmdmASfpD/71GFo/ZZlbNehu3h236RLoI9j2SfjRc5zAzMzMzK4fhnFP8DyQLc5uZmZmZ5dqwJMXp4xH/iuRpXWZmZmZmuTZcleJvAJ8FOnvrIOksSWslrd20adMwhWFmZmZm1r+yJ8WS3gc8ExHr+uoXEUsiYlpETJs0qaZu3DYzMzOznBmOSvG7gZMkbQRuBN4r6YZhOM+wamtr45xzzmHz5s1Zh2JmZv3wmG1mQ1X2pDgivhARDRExGTgV+GlE/E25zzPcmpqaWL9+PU1NTVmHYmZm/fCYbWZD5Sfa9aCtrY3ly5cTESxfvtyVBzOzHPOYbWblMKxJcUSsjoj3Dec5hkNTUxMRAUBnZ6crD2ZmOdbU1ER7ezsAO3fu9JhtZoPiSnEPmpubdw+w7e3trFixIuOIzMysN83NzbsLGRHhMdvMBsVJcQ/mzJmDJAAkMXfu3IwjMjOz3rzpTW8qah9xxBEZRWJm1cxJcQ8aGxuLqg6NjY0ZR2RmZr257777itr33ntvRpGYWTVzUtyD7gPq+vXrM4rEzMz609HR0WfbzKwUTop7cNlllxW1L7nkkowiMTMzM7NKcFLcA1cdzMzMzGqLk+IejB49us+2mZmZmY0sTop7sGDBgqL2hRdemFEkZmZmZlYJTop7MHv27KL2sccem1EkZmbWn3333beovd9++2UUiZlVMyfFPbjllluK2kuXLs0oEjMz68/zz///7d1/lFx1mefxz9OdXw0t25t0D0I6mTAmjAM5BE3TCEoA8wOyhyPrzOLCKLZH1jSuiC6Crom4CSC66zhqwMGEhbVVlPWMwwKakHT2EIGzQOgIxBAkaTAmRSB0x0ES8oP+8ewf3R26mupKVfWt+t5bwri/aQAAIABJREFU9X6dUyf9VNW99+lL+Oapb31/vJEW//nPfw6UCYAkoyjO4Hvf+15a/J3vfCdQJgAAACgFiuIMhjbuGC0GAABAeaEozmBoi+fRYgAAAJQXiuIMvvjFL6bFX/rSlwJlAgAAgFKgKM7g9NNPT4tPO+20QJkAAACgFCiKM7jlllvS4ptuuilQJgAAACgFiuIMdu7cmTUGAMTHpEmTssYAkIuiFMVmNsnMNpnZs2b2nJmtKMZ1iuWkk05Ki08++eRAmQAAjoXJ0QCiMK5I5z0i6cPufsDMxkt6zMzWuvsTRbpepGhQASA5Dh06lDUGgFwUpSj2gYV9DwyG4wcfiVnsd8+ePVljACg3ZrZT0n5JfZJ63b0pbEa5q62t1YEDB9JiAMhX0cYUm1m1mT0j6TVJ7e7+ZLGuFbVx48ZljQGgTF3o7mcmqSCWpOXLl6fFN998c5hEACRa0Ypid+9z9zMlNUpqNrPZw183syVm1mFmHV1dXcVKoyC9vb1ZYwBAfDQ3N6fFc+fODZQJgCQr+uoT7v66pI2SLh7x/Gp3b3L3poaGhmKnAQDIziWtN7PNZrZk5Itx7sjYsGFDWvzwww8HygRAkhVr9YkGM6sb/LlG0gJJvy/GtQAAkfigu79f0mJJnzOzecNfjHNHxq233poWM3wCQCGK1VN8kqSHzWyLpKc0MKb4V0W6VuRmzpyZFr/3ve8NlAkAlIa77xn88zVJ90lqzn5EfDDkDUAUilIUu/sWd3+fu5/h7rPdPVFbwt19991p8erVqwNlAgDFZ2bHm9m7hn6WtEjS1rBZ5a66ujprDAC5YEe7DDZt2pQWb968OVAmAFASJ2pgPflnJW2S9Gt3fyhwTjmrqqrKGgNALlhrLIORy/vceOONWrNmTZhkAKDI3P0lSXNC51Gonp6erDEA5IKP0xkMXwQ+UwwAAIDyQlGcwcjdkNgdCQDia9KkSVljAMgFRXEG7I4EAMlxyy23pMXf/OY3A2UCIMkoigEAifbCCy+kxdu3bw+UCYAkoyjOINNEOwBAPN15551p8R133BEoEwBJRlGcARPtAAAAKgtFcQZMtAMAAKgsFMUZMNEOAACgslAUZ9Dc3Hy0d7i2tlZz584NnBEAAACKiaJ4FMuXL1dVVRW9xAAAABWAbZ5H0dzcrI0bN4ZOAwAAACVATzEAAAAqHkUxAAAAKh7DJ0axYcMG3XTTTVqxYoUuvPDC0OkAQNlbuXKlOjs7IznXtddem/cxM2fOLOg4AOWBnuJR3HrrrZJYjg0A4m769OlZYwDIReQ9xWY2TdKPJb1bUr+k1e7+/aivU0wbNmxQb2+vJKm3t1cPP/wwvcUAUGRj6aWdN2/e0Z9/+tOfRpEOgApTjJ7iXklfcve/kfQBSZ8zs9OKcJ2iGeolHkJvMQDE21Dv8IoVKwJnAiCpIu8pdvdXJL0y+PN+M3te0lRJ26K+VrEM9RKPFgMA4mXy5MmaPHky3+oBKFhRxxSb2QxJ75P0ZDGvE7Vx48ZljQEAAFBeilYUm1mtpF9K+qK7v5Hh9SVm1mFmHV1dXcVKoyBLly5Ni2+88cZAmQAAAKAUilIUm9l4DRTE97j7v2R6j7uvdvcmd29qaGgoRhoFO+GEE7LGAAAAKC+RF8VmZpLukvS8u/9j1OcvheXLl6fF9BQDAACUt2L0FH9Q0pWSPmxmzww+/l0RrlM0Bw4cyBoDAACgvBRj9YnHJFnU5y2lmpoaHTp0KC0GRhrL7lupVEqS1NjYWPD12X0LcRTlrnT52LFjh6SxrXU8Fvz/CCQfyypk4O5ZY2Cshn/oAspJZ2ennv7dNvUfN7mk17W3BtrpzS++WtLrSlLVwT+V/JqVaqydESHb3pqamoI7QvjQVRoUxRkcPnw4awxIY+uRGjp25cqVUaUDxEb/cZN1+LRLQqdRMpO2/Sp0ChWjs7NTTz/3tFRXwMFvamB7sUAO9B9Q18sFrLb1evS5IDOK4gxqa2vTxhHX1tYGzAYAABxVJ/Vf0B86i5Kp2ljULSUwDEVxBsuXL9f1119/NGabZwBA3DCvAYgWRXEGzc3NR3uLa2trNXfu3NApAQDKUKgxskPHjWWMbSqVKjh3CmrEEUXxKM477zytXbtWF154YehUAKDozOxiSd+XVC3pf7r7twKnVBE6Ozu1fetvNb22L+9j6yVpfGHX3dsz8JX8ieOPFHYCSep9XYd3vpL3YbsOVBd+TeSFiYn5oSgexdq1ayVJDz74oG644YbA2aBYWD4KkMysWtIPJC2UlJL0lJk94O7bwmZWGabX9ulrTZWzHv4tHczTKZXOzk79/pln9O4Cjj2ioPMSdeTNN/V6d3fex41l/RmK4gx+8pOfpMU///nPdcUVVwTKBsU0ppnMYzE4R+Tpl58u8YXFTGZk0iyp091fkiQzu1fSpZLyLopTqZSq9u/TcR1t+WfR3yeFXALTTKoqoBezr1epVGHlQyqV0pv7qyuqUPzj/modPzimOV+pVEr6c4VNPntdSnnh96vQ/6OmFHhcaK63x8zni6I4gzvvvDMtvuOOOyiKyxkzmYGpknYPi1OSzh7+BjNbImmJJE2fPn3UE9XV1RX8leuRI0fU3x/u/8WqqipNnDihgCMnqK6u1J+sK1ivwny4HxrhUurRHyG7aysMRTEAINMupGkdTO6+WtJqSWpqahq18+nuu++ONrMy19jYqMO9r1Tc8IlJBY4VveCCC4IMeZPeHvY2a9askl975syZBR3X2Nio17u7dVWyNxrOy11y1RX494uiGACQkjRtWNwoaU+gXIBRjWUuRKg5JEOYyxF/fI+awWc+85m0+LOf/WygTACgJJ6SNMvMTjGzCZIul/RA4JyAWKmpqVFNTU3oNFBE9BRncOWVV6aNK2Y8MYBy5u69ZnaNpHUaGDF5t7s/FzitirHrQOkn2u09OLgk23GlH8O960C1Ti35VcOt9oPkoCjOYNWqVWnxXXfdpauuuipQNgBQfO6+RtKa0HlUmkLHio7VW4PjYyfNKP342FMV7veuRK9qYJxtqe0b/LPUq1i8qsIXlKIozuCee+5Ji9va2iiKAQCRC9V7OXTdlStXBrk+SiPkh4+uwQ9edSWemFinwn9vimIAAIAyFHLISBI/eDHRDgAAABWPnuIMPv7xj6cNoWhpaQmYTemMdY90SSXfpxwAACAKRSmKzexuSZdIes3dZxfjGsXU2tqaVhQznvjYCt3BCgBQmLF0ZAxtRDGWzgg6M1BuitVT/CNJt0v6cZHOX1SbNm1Kizdv3qy5c+cGyqZ0xtK4JXHsEABUKtbbBd6pKEWxuz9iZjOKce5SuPHGG9PipUuXat26dYGyAQDgneilBaIVbKKdmS0xsw4z6+jq6gqVRkYjhwIwNAAAACB3Bw8e1JYtW4JurZ2vYBPt3H21pNWS1NTUVPpVpctUqL3doxifNhaMbQMAID527typ/v5+fe1rX9O9994bOp2csPpEmens7NT2rb/V9Nq+kl53Qs/Alw6Hdz5V0utKA1uGAgCAeNi+fbveeustSdKePXvU2dmZiF0MKYozqK6uVl9fX1qcJNNr+/S1pgOh0yiZWzpqQ6cAAEBZGcs3z9u2bUuLr776ap122ml5nSPEN8DFWpLt55IukFRvZilJ/83d7yrGtbIp9D/o1KlTtWvXrrQ43/8wfJ0PAAAq0VAv8WhxXBVr9YkrinHeUpk8eXJaUTx58uSA2QAAAJTWWDr25s2b947nkrBka1kPnxjLf9BPfOIT2rVrl1asWKELL7wwwqwAAAAQN2VdFI/F5MmTNXnyZApiAACAChBsnWIAAAAgLiiKAQAAUPEoigEAABCZmpqarHFcURQDAAAgMjNmzEiLTznllDCJ5ImiGAAAAJF5/vnn0+KRm3nEFUUxAAAAKh5LsgHI2Vi2/UylUjp06FDEGeWupqZGjY2NBR3LDpUAUP4oigHkbOPGjdrX3a0JBRzbK6k/6oTycPjNN7W/uzvv497SQEFPUQwA5Y2iGEBeJkg6KXQSJfRK6AQAIGHOOeccPf7440fjc889N2A2uaMoLjOpVEpv7q/WLR21oVMpmT/ur9bxqVToNCpCY2OjXu/u1lWy0KmUzF1y1RU47AIAKtFbb72VNY4rJtoBAAAgMps3b06LOzo6AmWSH3qKy0xjY6MO976irzUdCJ1KydzSUatJ9OQBAIAxoCgGkJdXNTCkoJT2Df45paRXHfCqpLoA1wWApBo3bpx6e3vT4iRIRpYAYmHmzJlBrtu1Y4ckqW7WrJJfu07hfu9SMLPlkj4jqWvwqaXuviZcRgCSzsyyxnFFUQwgZ6GWJRu67sqVK4NcvwJ8193/IXQSAMrDhAkT1NPTkxYnQVEm2pnZxWb2gpl1mtl/LcY1AAAAED9vvvlm1jiuIu8pNrNqST+QtFBSStJTZvaAuxe08fVYdtAaix2DX9eG6hljBy0AJXSNmX1SUoekL7n7v4ZOCABKrRjDJ5oldbr7S5JkZvdKulRSQUVxZ2ennv7dNvUfNznCFI/N3hqYSLT5xVdLel1Jqjr4p5JfE0D5MrMNkt6d4aVlku6QdLMkH/zzO5I+neEcSyQtkaTp06cXLVcAyVdVVaX+/v60OAmKURRPlbR7WJySdPbIN+XTwPYfN1mHT7skwhTjbdK2X4VOAYiVgwcPqrOzU52dnWU96a1Y3H1BLu8zszslZWyA3H21pNWS1NTUVNrlRwAkysKFC7Vu3bqj8aJFiwJmk7tilO6Zphi+owF199Xu3uTuTQ0NDUVIA0C5eOmll9Tf36+vfOUroVMpO2Y2fNfuj0raGioXAOWhtbU1axxXxSiKU5KmDYsbJe0pwnUAVIDt27cfXe+yq6sryByDMvc/zOx3ZrZF0oWS/kvohAAk39CQiaQMnZCKM3ziKUmzzOwUSS9LulzS3xfhOgASpNBJs1u3pndcLlmyRLNnz877PExezczdrwydA4Dy0tbWdnRccVVVldra2nTdddeFTuuYIi/f3b1X0jWS1kl6XtIv3P25qK8DoDIM3xUpUwwAiJf29vajbXVvb6/Wr18fOKPcFGXzjsHdkNgRCcBRhfbSzps37x3PsYkHAMTXwoULdf/99x+NK3miHQAAACrUeeedlxaff/75gTLJD0UxAAAAInP77benxd///vcDZZIfimIAAABEZufOnVnjuKIoBhBr48ePzxoDAOKltrY2axxXFMUAYq2npydrDACIl6SuGlSU1SeApEilUtKfpaqNFfT58HUp5anQWQAAytT555+fts3zBRdcEC6ZPMS+KE6lUqrav0/HdbTlf3B/n+Tv2GG6NMykqurCju3rVSqVjE9VAAAA5SD2RXFdXZ0OHTpU0LFHjhxRf39/xBnlpqqqShMnTijw6Amqq6uLNB9k1tjYqC7rUv8FYf6ehFC1sUqNUxtDp5Gzd73rXdq/f//R+IQTTgiYDQDgWB599NG0+JFHHtHSpUsDZZO72BfFd999d+gUAAQ0ceLEtKJ4woRCP2wCAEqhublZGzduPBqfffbZ4ZLJQwUNpASQRN3d3VljAEC8vPjii2lxZ2dnoEzyQ1EMAACAyOzevTtrHFcUxQAAAIjMtGnTssZxRVEMINZqamqyxgCAeGlsTJ/MnZSiOPYT7YCiez3AOsUHBv8MscnP65KmBrhugUauPlPoajQAgNLYtGlTWvzkk08GyiQ/FMWoaDNnzgxy3R07dkiSZk2dVfqLTw33ewMAyp+ZZY3jiqK4DO06UK1bOkrbBbn34EBP64nHlX69310HqnVqgcdee+21keaS73VXrlwZ5PpJUlVVlbbeeFUVo74AIM7mz5+ftqPdggULAmaTO4riMhOqB/CtwZ7PSTNK3/N5quj5LGcLFy5Ma1wXLVoUMBsAwLG0trZq/fr1cneZmVpbW0OnlJNIi2Izu0zSckl/I6nZ3TuiPD+OjZ5PlJvLLrssrSj+2Mc+FjAbAMCx1NfXq7GxUbt371ZjY6OmTJkSOqWcRP095FZJfyvpkYjPC6BCPfjgg2nxAw88ECgTAEAuuru7tXfvXknS3r17tW/fvsAZ5SbSotjdn3f3F6I8J4DKNryXWJIeeuihQJkAAHLR1tYmd5ckubva2toCZ5SbYDNWzGyJmXWYWUdXV1eoNADE3Lhx47LGAIB4aW9vV09PjySpp6dH69evD5xRbvIuis1sg5ltzfC4NJ/zuPtqd29y96aGhoZ80wBQIQ4cOJA1BgDEy8KFCzV+/HhJ0vjx4xMzQTrvLhd3T8a6GgDKQm1tbVohXFsbYscTAECuWlpatHbtWkkDy2i2tLQEzig3LPgJINaGvoIbLQYAxEt9fb0WL14sM9PixYsrc/UJM/uomaUknSPp12a27ljHAEA2J510UtYYABA/LS0tOuOMMxLTSyxFvE6xu98n6b4ozwmgsg0t6zNaDACIn/r6et12222h08gLwycAxNrZZ5+dFn/gAx8IlAkAoJxRFAOItRdffDEt7uzsDJRJcpnZZWb2nJn1m1nTiNe+amadZvaCmV0UKkcACI0FP4ECrVy5suACbceOHZLGti33zJkzg23rXUq7d+/OGiMnQ7uNrhr+pJmdJulySadLOlnSBjM71d37Sp8igHLS3d2tFStWaPny5ZU50Q5AbmpqalRTUxM6jUQ4/vjjs8Y4tiy7jV4q6V53P+Luf5DUKam5tNkBKEdtbW3asmVLYnazk+gpBgpWCb20cXDo0KGsMcZkqqQnhsWpwecAoGDd3d1au3at3F1r1qxRS0tLInqL6SkGEGtVVVVZYwwocLdRy/Ccj3L+JWbWYWYdXV1d0SQNoCy1tbWlbfOclN5i/nUBEGvz589PixcsYFPNTNx9gbvPzvC4P8thKUnThsWNkvaMcv7V7t7k7k0NDQ1Rpg6gzKxfv17uA5+v3V3r1iVj2wqKYgCx1tramjXGmDwg6XIzm2hmp0iaJWlT4JwAJFx9fX3WOK4oigHE3tCQCYZOFGa03Ubd/TlJv5C0TdJDkj7HyhMAxurll1/OGscV/8IAiLW2tra0ojgpY9PixN3vc/dGd5/o7ie6+0XDXvuGu7/H3f/a3deGzBNAeUjqXJBkZAmgYrW3t6u3t1eS1Nvbq/Xr1wfOCACQTVLnglAUA4i1hQsXavz48ZKk8ePHa9GiRYEzAgBk09ramvYNX1LmglAUA4i1lpYWmQ2sHFZVVaWWlpbAGQEAsqmvr9fChQslSYsWLUrEGsUSRTGAmKuvr9fixYtlZlq8eHFiGlcAqGStra2aM2dOYnqJJXa0A5AALS0t2rlzJ73EAJAQ9fX1uu2220KnkRd6igEAAFDxKIoBxF5bW5u2bNnCcmwAgKKJtCg2s2+b2e/NbIuZ3WdmdVGeH0Dl6e7u1tq1a+XuWrNmjfbt2xc6JQBAGYq6p7hd0mx3P0PSdklfjfj8ACpMW1ubenp6JEk9PT30FgMAiiLSotjd17t772D4hKTGKM8PoPKsX79e7i5JcnetW7cucEYAgGPp7u7W5z//+UR9u1fMMcWflsSWoQDGpL6+PmsMAIifJM4FybsoNrMNZrY1w+PSYe9ZJqlX0j1ZzrPEzDrMrKOrq6uw7AGUvZdffjlrDACIl+7ubq1ZsyZxc0HyLordfYG7z87wuF+SzKxF0iWSPu5D33lmPs9qd29y96aGhobCfwMAZW1oq9DRYgBAvLS1tam3d2A0bZLmgkS9+sTFkr4i6SPufjDKcwOoTPPnz0+LFyxYECgTAEAukjoXJOoul9slvUtSu5k9Y2Y/jPj8ACpMa2vr0d7hqqqqRG0ZCgCV6MQTT8wax1Wk2zy7+8wozwcA9fX1WrhwodatW6dFixZpypQpoVMCAGSxd+/erHFcMTgPQOy1trZqzpw59BIDQAIsWrRIZiZJMjNddNFFgTPKDUUxgNirr6/XbbfdRi8xACRAS0uLxo0bGIwwbtw4tbS0BM4oN5EOn0CyrVy5Up2dnQUdu2PHDknStddeW9DxM2fOLPhYAAAQH/X19Zo6dap27typqVOnJqZDg55iRKKmpkY1NTWh00CZSuLOSABQqbq7u7Vnzx5J0p49exLTdtNTjKPoqUVcrVq1Ss8++6xWrVqlpUuXhk4HAJBFW1ub+vv7JUn9/f1qa2vTddddFzirY6OnGECsdXd3q729XdLA2pdJ6XEAgErV3t5+dPOO3t5erV+/PnBGuaEoBhBrq1atSutxWLVqVeCMAADZnHfeeWnxvHnzAmWSH4piALG2YcOGtHio1xgAEE9vvPFG1jiuKIoBxNrQWpejxQCAeHniiSfS4scffzxQJvmhKAYQa/Pnz0+LFyxYECgTAEAu3D1rHFcUxQBibeQuduxqBwAoBopiALFXVVWV9icAIL7OOOOMtPjMM88MlEl++BcGQKy1tbUdHUdsZmprawucEQAgm5G7427fvj1QJvmhKAYQa+3t7err65Mk9fX1JWa9yzgxs8vM7Dkz6zezpmHPzzCzQ2b2zODjhyHzBFAeDh48mDWOK4piALHW3NycFp999tmBMkm0rZL+VtIjGV570d3PHHxcXeK8AJShpK4axDbPAGJt27ZtWWMcm7s/LyXnHyYAycbqEwBQBK+99lpavHfv3kCZlK1TzOxpM/uNmZ032pvMbImZdZhZR1dXVynzA5AwM2bMyBrHFUUxAJQBM9tgZlszPC7Nctgrkqa7+/skXSfpZ2Z2QqY3uvtqd29y96aGhoZi/AoAysQ111yTFn/hC18IlEl+Ih8+YWY3S7pUUr+k1yR9yt33RH0dAMDb3D3vXU3c/YikI4M/bzazFyWdKqkj4vQAVJD29va0eN26dZo7d26gbHJXjJ7ib7v7Ge5+pqRfSfp6Ea4BABgjM2sws+rBn/9K0ixJL4XNCkDSbdiwIS0eWSTHVeRFsbu/MSw8XlIyRlcDiKWkzmKOEzP7qJmlJJ0j6ddmtm7wpXmStpjZs5L+WdLV7v6nUHkCKA9JbbeLsvqEmX1D0icl/VnShaO8Z4mkJZI0ffr0YqQBoAwkdRZznLj7fZLuy/D8LyX9svQZAShnH/rQh7Rx48aj8XnnjTqHN1YK6ik+1oQOd1/m7tMk3SPpmkznYNIGgFxMmzYtawwAiJeJEydmjeOqoKLY3Re4++wMj/tHvPVnkv5u7GkCqFSNjY1pMUUxAMTbo48+mhY/8kimfYPiJ/IxxWY2a1j4EUm/j/oaACrHpk2b0uInn3wyUCYAgFyMHC4xb968QJnkpxhjir9lZn+tgSXZ/iiJbUMBAAAqxJEjR7LGcRV5UezuDJcAEJmTTz5Zu3fvTosBAPH12GOPpcUjh1PEFTvaAYi17u7urDEAIF6SumoQRTGAWBs5Fu38888PlAkAIBcnnXRS1jiuKIoBxFpSx6YBQKXat29f1jiuKIoBxFpSl/YBgEqV1G/4KIoBxFpfX1/WGACAKFAUA4i16urqrDEAIF5GfqP3m9/8JlAm+aEoBhBrSV0EHgAqVV1dXdY4riiKAcTaxIkTs8YAgHh55ZVXssZxRVEMINY2btyYFj/88MNhEgEAlDWKYgCx1tPTkzUGACAKFMUAYq2/vz9rDABAFCiKAQAAUPEoigEAAFDxKIoBAAAQmUmTJqXFNTU1gTLJD0UxgFhjSTYASJbDhw+nxYcOHQqUSX4oigHE2pEjR7LGAABEgaIYAAAAFa9oRbGZXW9mbmb1xboGAAAAEIWiFMVmNk3SQkm7inF+AAAAIErF6in+rqQvS/IinR9AhZg7d25afNZZZwXKBACQi3POOSctPvfccwNlkp/Ii2Iz+4ikl9392WO8b4mZdZhZR1dXV9RpACgTy5YtS4uXLl0aKBMAQC5uuOGGrHFcFVQUm9kGM9ua4XGppGWSvn6sc7j7andvcvemhoaGQtIAUAHq6+uP9hafddZZmjJlSuCMAADZ1NfXH+0tPvfccxPTbhdUFLv7AnefPfIh6SVJp0h61sx2SmqU9Fsze3d0KQOoNMuWLdOcOXPoJQaAhLjhhhs0Z86cxPQSS9K4KE/m7r+T9BdD8WBh3OTu3VFeB0Blqa+v12233RY6DQBAjpLYbrNOMQCUOTP7tpn93sy2mNl9ZlY37LWvmlmnmb1gZheFzBMAQipqUezuM+glBoDg2iXNdvczJG2X9FVJMrPTJF0u6XRJF0v6JzOrDpYlAARETzEAlDl3X+/uvYPhExqY7yFJl0q6192PuPsfJHVKag6RIwCERlEMAJXl05LWDv48VdLuYa+lBp97B5bRBFDuIp1oV6jNmzd3m9kfQ+eRQb0khn/kjvuVH+5XfuJ6v/4ydALSwFKZkjKt9LPM3e8ffM8ySb2S7hk6LMP7M2665O6rJa0ePE8XbXZZ4H7lj3uWn7jer4ztdiyKYneP5ULFZtbh7k2h80gK7ld+uF/54X5l5+4Lsr1uZi2SLpE0392HCt+UpGnD3tYoaU8O16LNLgPcr/xxz/KTtPvF8AkAKHNmdrGkr0j6iLsfHPbSA5IuN7OJZnaKpFmSNoXIEQBCi0VPMQCgqG6XNFFSu5lJ0hPufrW7P2dmv5C0TQPDKj7n7n0B8wSAYCiKs1sdOoGE4X7lh/uVH+5Xgdx9ZpbXviHpGyVMp5j4O5If7lf+uGf5SdT9sreHlgEAAACViTHFAAAAqHgVXxSb2UfN7JkRj34z+6yZuZl9fth7bzezTwVMN6jBe+Vm9t5hz80ys1+Z2YtmttnMHjazeYOvfWpw6abh9/a0cL9BaZjZlGG/76tm9vKweMIo97HJzLaa2YTB+D1m9pKZnRDuN4kHM+sbvHfPmtlvzezc0DkhLNrt3NBm5452OzpJbrMrvih29/vc/cyhh6R/kvSopHWSXpP0haG/8NAVkh7TwLawMrNJkn4tabW7v8fd50r6vKS/GnbM/x5+f919W8mzLjF33zfs79OiW3JMAAAC7UlEQVQPJX132O//lkbcx8FjOiQ9Iun6wad+oIH1Zd8ocfpxdGjw3s3RwPbE3wydEMKi3c4ZbXaOaLcjldg2u+KL4uHM7FRJX5d0paR+SV2S/q+klpB5xYGZ1Ur6oKSr9Haj8HFJj7v7A0Pvc/et7v6j0meYDKPcxyFLJf0nM/uypPHu/vNS55cAJ0j619BJID5otzOjzY4O7faYJKrNZvWJQWY2XtLPJF3v7rvMbMbgS9+StNbM7g6VW0z8e0kPuft2M/uTmb1f0umSfnuM4/6jmX1oWHyOux8qWpbx94776O6/lSR3f93M/rsGer0q4ivLHNWY2TOSJkk6SdKHA+eDmKDdzoo2Ozq02/lJbJtNT/Hbbpb0nLvfO/xJd/+DBhaz//sgWcXHFZKG7s29g3EaM7tvcHzVvwx7euRXcZXeuB7rPi6WtFc0rsMNfRX3XkkXS/qxmWXanhiVh3Z7dLTZ0aHdzk9i22x6iiWZ2QWS/k7S+0d5y62S/lkDY4cqjplN0cAnvdlm5pKqJbmkFZLmDb3P3T9qZk2S/iFIojE32n00sy+7u5vZJZL+jaSLJN1nZutG7D5W8dz9cTOrl9SggbGjqFC026OjzY4O7fbYJK3NrvieYjP7t5L+l6RPuvv+TO9x999rYMenS0qZW4z8B0k/dve/dPcZ7j5N0h8kbZf0QTP7yLD3Hhckw2QY7T5+yMxqJH1HAzuK/U7S/ZKWBcw1lgZnfldL2hc6F4RDu31MtNnRod0eg6S12fQUS1dL+gtJd4zo3R85WP4bkp4uVVIxc4UGxugN90sNfDV5iaR/NLPvaeDro/2Sbhn2vpHj0/6zu/+/YiYbY9nu42JJ/2fYTO/lkp4xsx+5+47SpRhLQ+PTJMkktbAVccWj3c6ONjs6tNv5S2ybzY52AAAAqHgVP3wCAAAAoCgGAABAxaMoBgAAQMWjKAYAAEDFoygGAABAxaMoBgAAQMWjKAYAAEDFoygGAABAxfv/TsXuaMbCZxAAAAAASUVORK5CYII=\n",
692 | "text/plain": [
693 | "
"
694 | ]
695 | },
696 | "metadata": {
697 | "needs_background": "light"
698 | },
699 | "output_type": "display_data"
700 | }
701 | ],
702 | "source": [
703 | "# Create subplots\n",
704 | "fig, ax = plt.subplots(2, 2, figsize = (12, 9))\n",
705 | "\n",
706 | "# Original\n",
707 | "sns.boxplot(x = 'variable', y = 'value', data = pd.melt(data[['ZN', 'AGE', 'TAX', 'B']]), ax = ax[0, 0])\n",
708 | "ax[0, 0].set_title('Original')\n",
709 | "ax[0, 0].set_xlabel('')\n",
710 | "ax[0, 0].set_ylabel('')\n",
711 | "\n",
712 | "# MinMaxScaler\n",
713 | "sns.boxplot(x = 'variable', y = 'value', data = pd.melt(normalised_df[['ZN', 'AGE', 'TAX', 'B']]), ax = ax[0, 1])\n",
714 | "ax[0, 1].set_title('MinMaxScaler')\n",
715 | "ax[0, 1].set_xlabel('')\n",
716 | "ax[0, 1].set_ylabel('')\n",
717 | "\n",
718 | "# StandardScaler\n",
719 | "sns.boxplot(x = 'variable', y = 'value', data = pd.melt(standardised_df[['ZN', 'AGE', 'TAX', 'B']]), ax = ax[1, 0])\n",
720 | "ax[1, 0].set_title('StandardScaler')\n",
721 | "ax[1, 0].set_xlabel('')\n",
722 | "ax[1, 0].set_ylabel('')\n",
723 | "\n",
724 | "# RobustScaler\n",
725 | "sns.boxplot(x = 'variable', y = 'value', data = pd.melt(robust_df[['ZN', 'AGE', 'TAX', 'B']]), ax = ax[1, 1])\n",
726 | "ax[1, 1].set_title('RobustScaler')\n",
727 | "ax[1, 1].set_xlabel('')\n",
728 | "ax[1, 1].set_ylabel('')"
729 | ]
730 | },
731 | {
732 | "cell_type": "markdown",
733 | "metadata": {},
734 | "source": [
735 | "As we can see, our original features have wildly different ranges.\n",
736 | "\n",
737 | "MinMaxScaler has rescaled our features so that their values are bounded between 0 and 1.\n",
738 | "\n",
739 | "StandardScaler and RobustScaler, on the other hand, have rescaled our features so that they are distributed around the mean of 0. "
740 | ]
741 | },
742 | {
743 | "cell_type": "markdown",
744 | "metadata": {},
745 | "source": [
746 | "# 5. Compare model accuracy"
747 | ]
748 | },
749 | {
750 | "cell_type": "markdown",
751 | "metadata": {},
752 | "source": [
753 | "I mentioned in the introduction of this tutorial that unscaled data can adversely impact a model's ability to make accurate predictions but so far, we have not discussed exactly how and why they do. In fact, feature scaling does not always improve a model's performance. Some models do not require feature scaling. \n",
754 | "\n",
755 | "In this section, we will explore the following classes of machine learning algorithms and discuss whether or not feature scaling impact their performance:\n",
756 | "\n",
757 | "1. Gradient descent based algorithms\n",
758 | "2. Distance-based algorithms\n",
759 | "3. Tree-based algorithms "
760 | ]
761 | },
762 | {
763 | "cell_type": "markdown",
764 | "metadata": {},
765 | "source": [
766 | "# 5.1 Theory\n",
767 | "\n",
768 | "Let's first go over some concepts behind those algorithms and think about how and why feature scaling might be important to each of them."
769 | ]
770 | },
771 | {
772 | "cell_type": "markdown",
773 | "metadata": {},
774 | "source": [
775 | "## 5.1.1 Gradient descent based algorithms\n",
776 | "\n",
777 | "Gradient desent is an iterative optimisation algorithm that takes us to the minimum of a function. Machine learning algorithms like linear regression and logistic regression rely on gradient descent to minimise their loss functions or in other words, to reduce the error between the predicted values and the actual values. \n",
778 | "\n",
779 | "Having features with varying range of values will cause different step sizes for each feature. Therefore, to ensure that gradient descent converges more smoothly and quickly, we need to scale our features so that they have a similar scale."
780 | ]
781 | },
782 | {
783 | "cell_type": "markdown",
784 | "metadata": {},
785 | "source": [
786 | "## 5.1.2 Distance-based algorithms\n",
787 | "\n",
788 | "The underlying algorithms to distance-based models make them the most susceptible to unscaled data. \n",
789 | "\n",
790 | "Algorithms like k-nearest neighbours, support vector machines and k-means clustering use the distance between data points to determine their similarity. Hence, features with a greater magnitude will be given a higher weightage by the model. This is not an ideal scenario as we do not want our algorithm to be heavily biased towards a single feature.\n",
791 | "\n",
792 | "Evidently, it is important that we implement feature scaling to our data before fitting them to distance-based algorithms to ensure that all features contribute equally to the result. "
793 | ]
794 | },
795 | {
796 | "cell_type": "markdown",
797 | "metadata": {},
798 | "source": [
799 | "## 5.1.3 Tree-based algorithms \n",
800 | "\n",
801 | "Each node in a classification and regression trees (CART) model, otherwise known as decision trees represents a single feature in a dataset. The tree splits each node in such a way that it increases the homogeneity of that node. This split is not affected by the other features in the dataset. \n",
802 | "\n",
803 | "For that reason, we can conclude that decision trees are invariant to the scale of the features and therefore do not require feature scaling. This includes other ensemble models that are also tree-based such as random forest and gradient boosting. "
804 | ]
805 | },
806 | {
807 | "cell_type": "markdown",
808 | "metadata": {},
809 | "source": [
810 | "# 5.2 Proof of concept\n",
811 | "\n",
812 | "Now that we understand the types of models that are sensitive and insensitive to feature scaling, let us now convince ourselves with a concrete example using the Boston house prices dataset. \n",
813 | "\n",
814 | "Here, I have chosen 2 distance-based algorithms (KNN and SVR) as well as 1 tree-based algorithm (decision trees regressor) to predict the house prices.\n",
815 | "\n",
816 | "We should expect to see an improved model performance with feature scaling under KNN and SVR and a constant model performance under decision trees with and without feature scaling.\n",
817 | "\n",
818 | "Feel free to experiment with other types of models like linear regression, random forest and gradient boosting!"
819 | ]
820 | },
821 | {
822 | "cell_type": "code",
823 | "execution_count": 9,
824 | "metadata": {},
825 | "outputs": [],
826 | "source": [
827 | "# Instantiate models \n",
828 | "knn = KNeighborsRegressor()\n",
829 | "svr = SVR()\n",
830 | "tree = DecisionTreeRegressor(max_depth = 10, random_state = 42)\n",
831 | "\n",
832 | "# Create a list which contains different scalers \n",
833 | "scalers = [norm, standard, robust]"
834 | ]
835 | },
836 | {
837 | "cell_type": "code",
838 | "execution_count": 10,
839 | "metadata": {},
840 | "outputs": [
841 | {
842 | "name": "stdout",
843 | "output_type": "stream",
844 | "text": [
845 | "X_train shape: (354, 13)\n",
846 | "Y_train shape: (354,)\n",
847 | "X_test shape: (152, 13)\n",
848 | "Y_test shape: (152,)\n"
849 | ]
850 | }
851 | ],
852 | "source": [
853 | "# Train test split\n",
854 | "\n",
855 | "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.3, random_state = 42)\n",
856 | "\n",
857 | "print(\"X_train shape: \", X_train.shape)\n",
858 | "print(\"Y_train shape: \", Y_train.shape)\n",
859 | "print(\"X_test shape: \", X_test.shape)\n",
860 | "print(\"Y_test shape: \", Y_test.shape)"
861 | ]
862 | },
863 | {
864 | "cell_type": "markdown",
865 | "metadata": {},
866 | "source": [
867 | "Before we get started, I think it is important to highlight the good practice of first fitting the scalers to the training set and then use that to transform the data in the test set. This is to prevent any data leakage and misleading accuracy scores.\n",
868 | "\n",
869 | "Here, I will construct a pipeline which contains a scaler and a model to fit and transform the features and subsequently make predictions using each model. The accuracy of these predictions are then evaluated using root mean squared error. The smaller the error, the better the model performance. "
870 | ]
871 | },
872 | {
873 | "cell_type": "markdown",
874 | "metadata": {},
875 | "source": [
876 | "## 5.2.1 KNN"
877 | ]
878 | },
879 | {
880 | "cell_type": "code",
881 | "execution_count": 11,
882 | "metadata": {},
883 | "outputs": [
884 | {
885 | "data": {
886 | "text/html": [
887 | "
"
692 | ],
693 | "text/plain": [
694 | " SibSp Fare Age\n",
695 | "0 1 7.2500 22.0\n",
696 | "1 1 71.2833 38.0\n",
697 | "2 0 7.9250 26.0\n",
698 | "3 1 53.1000 35.0\n",
699 | "4 0 8.0500 35.0\n",
700 | "5 0 8.4583 NaN"
701 | ]
702 | },
703 | "execution_count": 11,
704 | "metadata": {},
705 | "output_type": "execute_result"
706 | }
707 | ],
708 | "source": [
709 | "# Create sample dataframe\n",
710 | "\n",
711 | "df = pd.DataFrame({'SibSp': [1, 1, 0, 1, 0, 0], \n",
712 | " 'Fare': [7.25, 71.2833, 7.925, 53.1, 8.05, 8.4583], \n",
713 | " 'Age': [22, 38, 26, 35, 35, np.nan]})\n",
714 | "df"
715 | ]
716 | },
717 | {
718 | "cell_type": "markdown",
719 | "metadata": {},
720 | "source": [
721 | "# 4.1 Simple imputer\n",
722 | "\n",
723 | "Simple imputer follows a univariate approach to imputing missing values i.e. it only takes a single feature into consideration. Some of the most common uses of simple imputer are:\n",
724 | "\n",
725 | "- Mean\n",
726 | "- Median\n",
727 | "- Most frequent (mode)"
728 | ]
729 | },
730 | {
731 | "cell_type": "code",
732 | "execution_count": 12,
733 | "metadata": {},
734 | "outputs": [
735 | {
736 | "name": "stdout",
737 | "output_type": "stream",
738 | "text": [
739 | "Average age: 31.2\n"
740 | ]
741 | },
742 | {
743 | "data": {
744 | "text/plain": [
745 | "array([[ 1. , 7.25 , 22. ],\n",
746 | " [ 1. , 71.2833, 38. ],\n",
747 | " [ 0. , 7.925 , 26. ],\n",
748 | " [ 1. , 53.1 , 35. ],\n",
749 | " [ 0. , 8.05 , 35. ],\n",
750 | " [ 0. , 8.4583, 31.2 ]])"
751 | ]
752 | },
753 | "execution_count": 12,
754 | "metadata": {},
755 | "output_type": "execute_result"
756 | }
757 | ],
758 | "source": [
759 | "print(\"Average age: \", df['Age'].mean())\n",
760 | "simple_imp = SimpleImputer(missing_values = np.nan, strategy = 'mean')\n",
761 | "simple_imp.fit_transform(df)"
762 | ]
763 | },
764 | {
765 | "cell_type": "markdown",
766 | "metadata": {},
767 | "source": [
768 | "As we can see, simple imputer has filled the missing value in the Age column with the average age which is 31.2."
769 | ]
770 | },
771 | {
772 | "cell_type": "markdown",
773 | "metadata": {},
774 | "source": [
775 | "# 4.2 Iterative imputer\n",
776 | "\n",
777 | "Iterative imputer is an example of a multivariate approach to imputation. It models the missing values in a column by using information from the other columns in a dataset. More specifically, it treats the column with missing values as a target variable while the remaining columns are used are predictor variables to predict the target variable. \n",
778 | "\n",
779 | "In our sample data frame, the Age column has one missing value on row 6 and is therefore assigned as the target variable in this scenario. This leaves the SibSp and Fare columns as our predictor variables. \n",
780 | "\n",
781 | "Iterative imputer will use the first 5 rows of the data frame to train a predictive model. Once the model is ready, it will then values in the SibSp and Fare columns of row 6 as inputs and predict the Age value for that row."
782 | ]
783 | },
784 | {
785 | "cell_type": "code",
786 | "execution_count": 13,
787 | "metadata": {
788 | "scrolled": true
789 | },
790 | "outputs": [
791 | {
792 | "data": {
793 | "text/plain": [
794 | "array([[ 1. , 7.25 , 22. ],\n",
795 | " [ 1. , 71.2833 , 38. ],\n",
796 | " [ 0. , 7.925 , 26. ],\n",
797 | " [ 1. , 53.1 , 35. ],\n",
798 | " [ 0. , 8.05 , 35. ],\n",
799 | " [ 0. , 8.4583 , 28.50639495]])"
800 | ]
801 | },
802 | "execution_count": 13,
803 | "metadata": {},
804 | "output_type": "execute_result"
805 | }
806 | ],
807 | "source": [
808 | "iterative_imp = IterativeImputer()\n",
809 | "iterative_imp.fit_transform(df)"
810 | ]
811 | },
812 | {
813 | "cell_type": "markdown",
814 | "metadata": {},
815 | "source": [
816 | "As we can see, the value predicted under iterative imputer is different to that under simple imputer.\n",
817 | "\n",
818 | "This is a more accurate approach to predict the missing Age value as it takes other features in our dataframe into account. "
819 | ]
820 | },
821 | {
822 | "cell_type": "markdown",
823 | "metadata": {},
824 | "source": [
825 | "# 4.3 KNN imputer\n",
826 | "\n",
827 | "Last but not least, we have KNN Imputer which is another multivariate imputation technique. KNN Imputer scans our dataframe for k nearest observations to the row with missing value. It will then proceed to fill the missing value with the average of those nearest observations. \n",
828 | "\n",
829 | "Here, I have set k to equal to 2 or in other words, I want KNN imputer to look for 2 observations that are nearest to row 6 and fill the missing age with the average age of those 2 rows."
830 | ]
831 | },
832 | {
833 | "cell_type": "code",
834 | "execution_count": 14,
835 | "metadata": {},
836 | "outputs": [
837 | {
838 | "data": {
839 | "text/plain": [
840 | "array([[ 1. , 7.25 , 22. ],\n",
841 | " [ 1. , 71.2833, 38. ],\n",
842 | " [ 0. , 7.925 , 26. ],\n",
843 | " [ 1. , 53.1 , 35. ],\n",
844 | " [ 0. , 8.05 , 35. ],\n",
845 | " [ 0. , 8.4583, 30.5 ]])"
846 | ]
847 | },
848 | "execution_count": 14,
849 | "metadata": {},
850 | "output_type": "execute_result"
851 | }
852 | ],
853 | "source": [
854 | "knn_imp = KNNImputer(n_neighbors = 2)\n",
855 | "knn_imp.fit_transform(df)"
856 | ]
857 | },
858 | {
859 | "cell_type": "markdown",
860 | "metadata": {},
861 | "source": [
862 | "As a result, KNN imputer has taken row 3 and row 5 as the nearest observations for row 6.\n",
863 | "\n",
864 | "Therefore, the average age between row 3 and row 5 is (26 + 35) / 2 = 30.5. "
865 | ]
866 | },
867 | {
868 | "cell_type": "markdown",
869 | "metadata": {},
870 | "source": [
871 | "# 5. Model accuracy under simple imputer and iterative imputer\n",
872 | "\n",
873 | "Now that we have a better understanding of how the different imputers work, we can move on to apply these techniques to our Titanic dataset and compare the model accuracy under each approach.\n",
874 | "\n",
875 | "We should expect to see our model perform better under multivariate imputation than univariate imputation as multivariate imputation provides a more accurate prediction of the missing values and thus allowing our model to make better predictions. \n",
876 | "\n",
877 | "In this section, we will build a column transformer which consists of a OneHotEncoder for encoding the Sex and Embarked columns as well as an imputer to impute the missing values in the Age column.\n",
878 | "\n",
879 | "Following that, we will chain the column transformer with a random forest classifier to predict the surival of the passengers on the Titanic. Finally, we will perform 10-fold cross-validation to compare the prediction results under univariate imputation versus under multivariate imputation. "
880 | ]
881 | },
882 | {
883 | "cell_type": "code",
884 | "execution_count": 15,
885 | "metadata": {},
886 | "outputs": [
887 | {
888 | "data": {
889 | "text/html": [
890 | "