├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── data
    └── text.csv
├── model
    └── params.yaml
├── notebook
    └── test_churn_prediction.ipynb
├── requirements.txt
└── scripts
    ├── create_dataset.py
    ├── interpret.py
    ├── preprocess.py
    └── train.py


/CODE_OF_CONDUCT.md:
--------------------------------------------------------------------------------
1 | ## Code of Conduct
2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
4 | opensource-codeofconduct@amazon.com with any additional questions or comments.
5 | 


--------------------------------------------------------------------------------
/CONTRIBUTING.md:
--------------------------------------------------------------------------------
 1 | # Contributing Guidelines
 2 | 
 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional
 4 | documentation, we greatly value feedback and contributions from our community.
 5 | 
 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
 7 | information to effectively respond to your bug report or contribution.
 8 | 
 9 | 
10 | ## Reporting Bugs/Feature Requests
11 | 
12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features.
13 | 
14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already
15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful:
16 | 
17 | * A reproducible test case or series of steps
18 | * The version of our code being used
19 | * Any modifications you've made relevant to the bug
20 | * Anything unusual about your environment or deployment
21 | 
22 | 
23 | ## Contributing via Pull Requests
24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
25 | 
26 | 1. You are working against the latest source on the *main* branch.
27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already.
28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted.
29 | 
30 | To send us a pull request, please:
31 | 
32 | 1. Fork the repository.
33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
34 | 3. Ensure local tests pass.
35 | 4. Commit to your fork using clear commit messages.
36 | 5. Send us a pull request, answering any default questions in the pull request interface.
37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
38 | 
39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/).
41 | 
42 | 
43 | ## Finding contributions to work on
44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.
45 | 
46 | 
47 | ## Code of Conduct
48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
50 | opensource-codeofconduct@amazon.com with any additional questions or comments.
51 | 
52 | 
53 | ## Security issue notifications
54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
55 | 
56 | 
57 | ## Licensing
58 | 
59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution.
60 | 


--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
 2 | 
 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of
 4 | this software and associated documentation files (the "Software"), to deal in
 5 | the Software without restriction, including without limitation the rights to
 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
 7 | the Software, and to permit persons to whom the Software is furnished to do so.
 8 | 
 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
15 | 
16 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | ## Churn Prediction with Text and Interpretability
 2 | 
 3 | Customer churn, the loss of current customers, is a problem faced by a wide range of companies. When trying to retain customers, it is in a company’s best interest to focus their efforts on customers who are more likely to leave, but companies need a way to detect customers who are likely to leave before they have decided to leave. Users prone to churn often leave clues to their disposition in user behavior and customer support chat logs which can be detected and understood using Natural Language Processing (NLP) tools.
 4 | 
 5 | Here, we demonstrate how to build a churn prediction model that leverages both text and structured data (numerical and categorical) which we call a bi-modal model architecture. We use Amazon SageMaker to prepare, build, and train the model. Detecting customers who are likely to churn is only part of the battle, finding the root cause is an essential part of actually solving the issue. Since we are not only interested in the likelihood of a customer churning but also in the driving factors, we complement the prediction model with an analysis into feature importance for both text and non-text inputs.
 6 | 
 7 | The categorical and numerical data is from Kaggle: Customer Churn Prediction 2020 and was combined with a synthetic text dataset we created using GPT-2.
 8 | 
 9 | ## Blog Post
10 | 
11 | [Medium / Towards Data Science blog post](https://towardsdatascience.com/customer-churn-prediction-with-text-and-interpretability-bd3d57af34b1)
12 | 
13 | ## Installation
14 | 
15 | ```
16 | git clone https://github.com/aws-samples/churn-prediction-with-text-and-interpretability.git
17 | conda create -n py39 python=3.9
18 | conda activate py39
19 | cd churn-prediction-with-text-and-interpretability
20 | pip install -r requirements.txt
21 | ```
22 | 
23 | ## Download categorical/numerical data and combine with synthetic text data
24 | 
25 | 1. Download categorical/numerical data - [Customer Churn Prediction 2020](https://www.kaggle.com/c/customer-churn-prediction-2020/data) 
26 |     May require Kaggle account.
27 |     Download train.csv and store in data folder.
28 | 
29 | 2. Run script to combine categorical data with synthetic text data (../scripts)
30 |     ```
31 |     python create_dataset.py
32 |     ```
33 | 
34 | ## Run in Notebook
35 | 
36 | An example notebook to run the entire pipeline and print/visualize the results in included in ../notebook.
37 | 
38 | ## Run in Terminal
39 | 
40 | The python scripts to prepare the data, train and evaluate the model, as well as interpret the model, are stored in ../scripts.
41 | The parameters used for training and interpreting the model are stored in ../model/params.yaml.
42 | 
43 | 
44 | 1. Prepare the data:
45 |     ```
46 |     python preprocess.py
47 |     ```
48 | 2. Train and evaluate the model:
49 |     ```
50 |     python train.py
51 |     ```
52 | 3. Interpret the trained model (text):
53 |     ```
54 |     python interpret.py --churn 1 --speaker Customer
55 |     ```
56 | 
57 | ## Credits
58 | 
59 | * Packages:
60 |     * [Spacy](https://spacy.io/usage/linguistic-features/) 
61 |     * [PyTorch](https://pytorch.org/)
62 |     * [XGBoost](https://xgboost.readthedocs.io/en/latest/)
63 |     * [Hugging Face Sentence Transformers](https://huggingface.co/sentence-transformers)
64 | 
65 | * Datasets:
66 |     * [Customer Churn Prediction 2020](https://www.kaggle.com/c/customer-churn-prediction-2020/data) (with synthetic text dataset)
67 |     
68 | * Models:
69 |     * GPT2, Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever
70 |     * BERT, Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova
71 |     * Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, Reimers, Nils and Gurevych, Iryna
72 | 
73 | ## Security
74 | 
75 | See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.
76 | 
77 | ## License
78 | 
79 | This library is licensed under the MIT-0 License. See the LICENSE file.
80 | 
81 | 


--------------------------------------------------------------------------------
/model/params.yaml:
--------------------------------------------------------------------------------
 1 | data_dir: ../data
 2 | model_dir: ../model
 3 | batch_size: 10
 4 | batch_size_test: 1000
 5 | epochs: 10
 6 | pos_weight: 10
 7 | lr: 0.001
 8 | momentum: 0.9
 9 | topn_relevant_keywords: 1000
10 | frac_relevant_keywords: 0.25
11 | w_marg_contr: 0.3
12 | w_count: 0.2
13 | w_sim : 0.5


--------------------------------------------------------------------------------
/notebook/test_churn_prediction.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "id": "df276921",
  6 |    "metadata": {},
  7 |    "source": [
  8 |     "# Churn Prediction with Text and Interpretability"
  9 |    ]
 10 |   },
 11 |   {
 12 |    "cell_type": "markdown",
 13 |    "id": "1c1f7791",
 14 |    "metadata": {},
 15 |    "source": [
 16 |     "This notebook runs the entire churn prediction pipeline from data preparation to model evaluation and interpretation.\n",
 17 |     "\n",
 18 |     "Alternatively, everything can be run from the terminal as well (see README.md).\n",
 19 |     "\n",
 20 |     "Prerequisite: Dataset has been created (see README.md)."
 21 |    ]
 22 |   },
 23 |   {
 24 |    "cell_type": "markdown",
 25 |    "id": "032bb6ce",
 26 |    "metadata": {},
 27 |    "source": [
 28 |     "### Setup"
 29 |    ]
 30 |   },
 31 |   {
 32 |    "cell_type": "code",
 33 |    "execution_count": 1,
 34 |    "id": "a32ade35",
 35 |    "metadata": {},
 36 |    "outputs": [],
 37 |    "source": [
 38 |     "import os\n",
 39 |     "import pandas as pd\n",
 40 |     "from matplotlib import pyplot as plt\n",
 41 |     "\n",
 42 |     "os.chdir(\"../scripts\")\n",
 43 |     "\n",
 44 |     "import preprocess\n",
 45 |     "import train\n",
 46 |     "import interpret"
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "code",
 51 |    "execution_count": 3,
 52 |    "id": "9d500e76",
 53 |    "metadata": {},
 54 |    "outputs": [],
 55 |    "source": [
 56 |     "%load_ext autoreload\n",
 57 |     "%autoreload 2"
 58 |    ]
 59 |   },
 60 |   {
 61 |    "cell_type": "markdown",
 62 |    "id": "4b8b1336",
 63 |    "metadata": {},
 64 |    "source": [
 65 |     "### Load and Prepare the Data"
 66 |    ]
 67 |   },
 68 |   {
 69 |    "cell_type": "code",
 70 |    "execution_count": 4,
 71 |    "id": "94c30e10",
 72 |    "metadata": {},
 73 |    "outputs": [
 74 |     {
 75 |      "data": {
 76 |       "text/html": [
 77 |        "<div>\n",
 78 |        "<style scoped>\n",
 79 |        "    .dataframe tbody tr th:only-of-type {\n",
 80 |        "        vertical-align: middle;\n",
 81 |        "    }\n",
 82 |        "\n",
 83 |        "    .dataframe tbody tr th {\n",
 84 |        "        vertical-align: top;\n",
 85 |        "    }\n",
 86 |        "\n",
 87 |        "    .dataframe thead th {\n",
 88 |        "        text-align: right;\n",
 89 |        "    }\n",
 90 |        "</style>\n",
 91 |        "<table border=\"1\" class=\"dataframe\">\n",
 92 |        "  <thead>\n",
 93 |        "    <tr style=\"text-align: right;\">\n",
 94 |        "      <th></th>\n",
 95 |        "      <th>churn</th>\n",
 96 |        "      <th>chat_log</th>\n",
 97 |        "      <th>state</th>\n",
 98 |        "      <th>account_length</th>\n",
 99 |        "      <th>area_code</th>\n",
100 |        "      <th>international_plan</th>\n",
101 |        "      <th>voice_mail_plan</th>\n",
102 |        "      <th>number_vmail_messages</th>\n",
103 |        "      <th>total_day_minutes</th>\n",
104 |        "      <th>total_day_calls</th>\n",
105 |        "      <th>...</th>\n",
106 |        "      <th>total_eve_minutes</th>\n",
107 |        "      <th>total_eve_calls</th>\n",
108 |        "      <th>total_eve_charge</th>\n",
109 |        "      <th>total_night_minutes</th>\n",
110 |        "      <th>total_night_calls</th>\n",
111 |        "      <th>total_night_charge</th>\n",
112 |        "      <th>total_intl_minutes</th>\n",
113 |        "      <th>total_intl_calls</th>\n",
114 |        "      <th>total_intl_charge</th>\n",
115 |        "      <th>number_customer_service_calls</th>\n",
116 |        "    </tr>\n",
117 |        "  </thead>\n",
118 |        "  <tbody>\n",
119 |        "    <tr>\n",
120 |        "      <th>0</th>\n",
121 |        "      <td>no</td>\n",
122 |        "      <td>Customer: Well, the only thing that I'm consid...</td>\n",
123 |        "      <td>CT</td>\n",
124 |        "      <td>134</td>\n",
125 |        "      <td>area_code_408</td>\n",
126 |        "      <td>no</td>\n",
127 |        "      <td>no</td>\n",
128 |        "      <td>0</td>\n",
129 |        "      <td>177.2</td>\n",
130 |        "      <td>91</td>\n",
131 |        "      <td>...</td>\n",
132 |        "      <td>228.7</td>\n",
133 |        "      <td>105</td>\n",
134 |        "      <td>19.44</td>\n",
135 |        "      <td>194.3</td>\n",
136 |        "      <td>113</td>\n",
137 |        "      <td>8.74</td>\n",
138 |        "      <td>8.9</td>\n",
139 |        "      <td>3</td>\n",
140 |        "      <td>2.40</td>\n",
141 |        "      <td>2</td>\n",
142 |        "    </tr>\n",
143 |        "    <tr>\n",
144 |        "      <th>1</th>\n",
145 |        "      <td>yes</td>\n",
146 |        "      <td>Customer: Well, I just want to be able to canc...</td>\n",
147 |        "      <td>WV</td>\n",
148 |        "      <td>78</td>\n",
149 |        "      <td>area_code_408</td>\n",
150 |        "      <td>no</td>\n",
151 |        "      <td>no</td>\n",
152 |        "      <td>0</td>\n",
153 |        "      <td>226.3</td>\n",
154 |        "      <td>88</td>\n",
155 |        "      <td>...</td>\n",
156 |        "      <td>306.2</td>\n",
157 |        "      <td>81</td>\n",
158 |        "      <td>26.03</td>\n",
159 |        "      <td>200.9</td>\n",
160 |        "      <td>120</td>\n",
161 |        "      <td>9.04</td>\n",
162 |        "      <td>7.8</td>\n",
163 |        "      <td>11</td>\n",
164 |        "      <td>2.11</td>\n",
165 |        "      <td>1</td>\n",
166 |        "    </tr>\n",
167 |        "    <tr>\n",
168 |        "      <th>2</th>\n",
169 |        "      <td>no</td>\n",
170 |        "      <td>Customer: I would like data.\\nTelCom Agent: Ok...</td>\n",
171 |        "      <td>IN</td>\n",
172 |        "      <td>88</td>\n",
173 |        "      <td>area_code_415</td>\n",
174 |        "      <td>no</td>\n",
175 |        "      <td>no</td>\n",
176 |        "      <td>0</td>\n",
177 |        "      <td>183.5</td>\n",
178 |        "      <td>93</td>\n",
179 |        "      <td>...</td>\n",
180 |        "      <td>170.5</td>\n",
181 |        "      <td>80</td>\n",
182 |        "      <td>14.49</td>\n",
183 |        "      <td>193.8</td>\n",
184 |        "      <td>88</td>\n",
185 |        "      <td>8.72</td>\n",
186 |        "      <td>8.3</td>\n",
187 |        "      <td>5</td>\n",
188 |        "      <td>2.24</td>\n",
189 |        "      <td>3</td>\n",
190 |        "    </tr>\n",
191 |        "  </tbody>\n",
192 |        "</table>\n",
193 |        "<p>3 rows × 21 columns</p>\n",
194 |        "</div>"
195 |       ],
196 |       "text/plain": [
197 |        "  churn                                           chat_log state  \\\n",
198 |        "0    no  Customer: Well, the only thing that I'm consid...    CT   \n",
199 |        "1   yes  Customer: Well, I just want to be able to canc...    WV   \n",
200 |        "2    no  Customer: I would like data.\\nTelCom Agent: Ok...    IN   \n",
201 |        "\n",
202 |        "   account_length      area_code international_plan voice_mail_plan  \\\n",
203 |        "0             134  area_code_408                 no              no   \n",
204 |        "1              78  area_code_408                 no              no   \n",
205 |        "2              88  area_code_415                 no              no   \n",
206 |        "\n",
207 |        "   number_vmail_messages  total_day_minutes  total_day_calls  ...  \\\n",
208 |        "0                      0              177.2               91  ...   \n",
209 |        "1                      0              226.3               88  ...   \n",
210 |        "2                      0              183.5               93  ...   \n",
211 |        "\n",
212 |        "   total_eve_minutes  total_eve_calls  total_eve_charge  total_night_minutes  \\\n",
213 |        "0              228.7              105             19.44                194.3   \n",
214 |        "1              306.2               81             26.03                200.9   \n",
215 |        "2              170.5               80             14.49                193.8   \n",
216 |        "\n",
217 |        "   total_night_calls  total_night_charge  total_intl_minutes  \\\n",
218 |        "0                113                8.74                 8.9   \n",
219 |        "1                120                9.04                 7.8   \n",
220 |        "2                 88                8.72                 8.3   \n",
221 |        "\n",
222 |        "   total_intl_calls  total_intl_charge  number_customer_service_calls  \n",
223 |        "0                 3               2.40                              2  \n",
224 |        "1                11               2.11                              1  \n",
225 |        "2                 5               2.24                              3  \n",
226 |        "\n",
227 |        "[3 rows x 21 columns]"
228 |       ]
229 |      },
230 |      "execution_count": 4,
231 |      "metadata": {},
232 |      "output_type": "execute_result"
233 |     }
234 |    ],
235 |    "source": [
236 |     "df = pd.read_csv('../data/churn_dataset.csv')\n",
237 |     "df.head(3)"
238 |    ]
239 |   },
240 |   {
241 |    "cell_type": "code",
242 |    "execution_count": null,
243 |    "id": "ede78f6b",
244 |    "metadata": {},
245 |    "outputs": [],
246 |    "source": [
247 |     "X_train, X_test, y_train, y_test = preprocess.prep_data(df, use_existing=False, test_size=0.33)"
248 |    ]
249 |   },
250 |   {
251 |    "cell_type": "code",
252 |    "execution_count": 27,
253 |    "id": "5d0f6d4f",
254 |    "metadata": {},
255 |    "outputs": [
256 |     {
257 |      "data": {
258 |       "text/plain": [
259 |        "((2233, 841), (1100, 841), (2233, 1), (1100, 1))"
260 |       ]
261 |      },
262 |      "execution_count": 27,
263 |      "metadata": {},
264 |      "output_type": "execute_result"
265 |     }
266 |    ],
267 |    "source": [
268 |     "X_train.shape, X_test.shape, y_train.shape, y_test.shape"
269 |    ]
270 |   },
271 |   {
272 |    "cell_type": "markdown",
273 |    "id": "c477f751",
274 |    "metadata": {},
275 |    "source": [
276 |     "### Train and Evaluate the Model"
277 |    ]
278 |   },
279 |   {
280 |    "cell_type": "code",
281 |    "execution_count": 28,
282 |    "id": "0d68bf05",
283 |    "metadata": {},
284 |    "outputs": [
285 |     {
286 |      "name": "stdout",
287 |      "output_type": "stream",
288 |      "text": [
289 |       "/home/ec2-user/SageMaker/churn_test/scripts\n"
290 |      ]
291 |     }
292 |    ],
293 |    "source": [
294 |     "!pwd"
295 |    ]
296 |   },
297 |   {
298 |    "cell_type": "code",
299 |    "execution_count": 29,
300 |    "id": "eb2e2878",
301 |    "metadata": {
302 |     "collapsed": true,
303 |     "jupyter": {
304 |      "outputs_hidden": true
305 |     }
306 |    },
307 |    "outputs": [
308 |     {
309 |      "name": "stdout",
310 |      "output_type": "stream",
311 |      "text": [
312 |       "starting epoch:  1\n",
313 |       "Train Epoch: 1, train-auc-score: 0.9561\n",
314 |       "test_auc_score: 0.9422\n",
315 |       "starting epoch:  2\n",
316 |       "Train Epoch: 2, train-auc-score: 0.9571\n",
317 |       "test_auc_score: 0.9453\n",
318 |       "starting epoch:  3\n",
319 |       "Train Epoch: 3, train-auc-score: 0.9602\n",
320 |       "test_auc_score: 0.9480\n",
321 |       "starting epoch:  4\n",
322 |       "Train Epoch: 4, train-auc-score: 0.9594\n",
323 |       "test_auc_score: 0.9467\n",
324 |       "starting epoch:  5\n",
325 |       "Train Epoch: 5, train-auc-score: 0.9628\n",
326 |       "test_auc_score: 0.9529\n",
327 |       "starting epoch:  6\n",
328 |       "Train Epoch: 6, train-auc-score: 0.9711\n",
329 |       "test_auc_score: 0.9555\n",
330 |       "starting epoch:  7\n",
331 |       "Train Epoch: 7, train-auc-score: 0.9756\n",
332 |       "test_auc_score: 0.9586\n",
333 |       "starting epoch:  8\n",
334 |       "Train Epoch: 8, train-auc-score: 0.9804\n",
335 |       "test_auc_score: 0.9598\n",
336 |       "starting epoch:  9\n",
337 |       "Train Epoch: 9, train-auc-score: 0.9810\n",
338 |       "test_auc_score: 0.9618\n",
339 |       "starting epoch:  10\n",
340 |       "Train Epoch: 10, train-auc-score: 0.9644\n",
341 |       "test_auc_score: 0.9552\n",
342 |       "saving scores\n",
343 |       "saving model\n"
344 |      ]
345 |     }
346 |    ],
347 |    "source": [
348 |     "# train the model\n",
349 |     "train.train(\n",
350 |     "    X=X_train,\n",
351 |     "    y=y_train,\n",
352 |     "    X_test=X_test,\n",
353 |     "    y_test=y_test\n",
354 |     ")"
355 |    ]
356 |   },
357 |   {
358 |    "cell_type": "code",
359 |    "execution_count": 30,
360 |    "id": "294bd939",
361 |    "metadata": {},
362 |    "outputs": [
363 |     {
364 |      "data": {
365 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABBqElEQVR4nO3deXxU5fX48c/JThJCIAlhCZCwyo6sKoggFcSVpVVwRxF3qba2aq1+tVZtq/7UiiBaVBTBFaVKhWpFQJAthB0lYU3YEraQQPbz++MOYZJMIEAmk+W8X695MXPvc+89mZfeM8997n2OqCrGGGNMaX6+DsAYY0z1ZAnCGGOMR5YgjDHGeGQJwhhjjEeWIIwxxnhkCcIYY4xHliCMMcZ4ZAnC1EkiskBEDolIsIfl40stGyQiqW6fRUQeFJH1IpItIqki8omIdC3nWJ1FZL7reIdFZJWIXOGdv8yYymMJwtQ5IhIPXAwocM1Z7OJVYCLwINAIaA98AVxZTvt/A/8FYoHGru0yz+K45RKRgMrcnzFgCcLUTbcAPwHvAreeyYYi0g64Dxirqv9T1VxVPaaqM1T1BQ/to4EE4C1VzXO9flTVxW5trhWRJBHJFJEUEbnctbyZiMwRkYMikiwid7pt838i8qmIfCAimcBtItJARP4lIntEJE1EnhUR/7P4fowBLEGYuukWYIbrNUxEYs9g2yFAqqour2D7A0Ay8IGIjCh9LBHpC0wHHgEigYHAdtfqmUAq0Az4NfCciAxx2/xa4FPXdjOA94ACoC1wPjAUKHG5zJgzYQnC1CkiMgBoBXysqquAFOCGM9hFFLCnoo3VmexsMM5J/yVgj4gsdPVEAO4Apqnqf1W1SFXTVHWziLQABgB/VNUcVU0C3gZudtv9UlX9QlWLgAhgOPBbVc1W1f3A/wPGnMHfZkwJliBMXXMrMF9VM1yfP6TkZaYCILDUNoFAvuv9AaDpmRxQVVNV9X5VbYOTnLJxeg0ALXCSVGnNgIOqetRt2Q6gudvnXW7vW7ni3OMaCD8MvIkz5mHMWbGBLVNniEg94DrAX0T2uhYHA5Ei0l1V1wA7gfhSmybgnJwBvgMmiUhvVV15pjGo6i4RmYRz+Qick3wbD013A41EpL5bkmgJpLnvzu39LiAXiFbVgjONyxhPrAdh6pIRQCHQCejhenUEFuGMSwB8BIwTkb6u21nbAw8BswBUdQvwBjDTdftrkIiEiMgYEXm09AFFpKGIPC0ibUXEzzVofTvOIDnAv1zHG+Ja31xEzlPVXcAS4HnX/rvhXI6a4ekPU9U9wHzgJRGJcO2rjYhccm5fmanLLEGYuuRW4B1V3amqe0+8gNeBG0UkQFXnAY8C7wBHgLk4g79T3fbzoGubScBhnEtEI3FuZy0tD6dH8i3Ora3rcX7p3wbgGuwehzNecAT4AedyEcBY17a7gdnAU6r631P8fbcAQcBG4BDOAPYZXQ4zxp1YwSBjjDGeWA/CGGOMR5YgjDHGeGQJwhhjjEeWIIwxxnhUq56DiI6O1vj4eF+HYYwxNcaqVasyVDXG07palSDi4+NZufKMn10yxpg6S0R2lLfOLjEZY4zxyBKEMcYYjyxBGGOM8ahWjUF4kp+fT2pqKjk5Ob4Opc4JCQkhLi6OwMDSk6MaY2qCWp8gUlNTqV+/PvHx8YiIr8OpM1SVAwcOkJqaSkJCgq/DMcachVp/iSknJ4eoqChLDlVMRIiKirKemzE1mFcThIhcLiI/u+rpljcV8mwRWSsiy0Wki9u6h0Rkg4isF5GZIhJyDnGc7abmHNj3bkzN5rVLTK5i6ZOAy3Dq6q4QkTmqutGt2eNAkqqOFJHzXO2HiEhznCmVO6nqcRH5GKd04rveitcYY04lt6CQ9KO57MvMZX9mDnszcxDguj4tCA2qnVfrvflX9QWSVXUrgIjMwimy7p4gOgHPA7jq8Ma7FXUPAOqJSD4QijMnvjHGVKrCIuVAlnPi35uZw77MHPZn5rAvM5d9R3PYeySH/UdzOZid53H7txdv49kRXRjUofZVd/VmgmhOyZq5qUC/Um3WAKOAxSLSF6dQSpyqrhKRF3HKPx7HqSE839NBRGQCMAGgZcuWlfsXVILDhw/z4Ycfcu+9957RdldccQUffvghkZGR3gnMmFpOVTl8LN/tpJ/Lvswc10k/l/1HneXpR3MpKlUWx08gOjyY2IgQ4hrWo1erhsRGhBAbEUzjiBBi6zvvk/dn8fjsddz2zgqu7t6MJ6/qREz9YN/8wV7gzQTh6QJ06epELwCvikgSsA5YDRSISEOc3kYCTsWuT0TkJlX9oMwOVafiqvbVu3fvalf96PDhw7zxxhtlEkRhYSH+/v7lbjd37lxvh1Yhp4vTGF/Iyi1wftm7Tvj7Tpz8M0++35+ZS15hUZltG4UF0bi+c/I/r0l9YiNCaBwRQhNXAoiNCCEqLIgA/9MP0UaFBzN34sVMXpDCG9+n8MPP+3n8io5c17sFfn41fwzOmwkiFWjh9jmOUpeJVDUTp9wi4oxobnO9hgHbVDXdte5z4CKgTII4E0//ewMbd2eeyy7K6NQsgqeu7lzu+kcffZSUlBR69OhBYGAg4eHhNG3alKSkJDZu3MiIESPYtWsXOTk5TJw4kQkTJgAn55XKyspi+PDhDBgwgCVLltC8eXO+/PJL6tWr5/F4r732GlOmTCEgIIBOnToxa9YssrKyeOCBB1i5ciUiwlNPPcXo0aOZOXMmzz33HKrKlVdeyd/+9jcAwsPDefjhh5k3bx4vvfQS27dv57XXXiMvL49+/frxxhtvAHDHHXcU7/P222/noYceqtTv1tRtRUXKnswcUvZnkZLueu3PJjk9i/SjuWXa1w8OoLHrBN83vpHzS9/1OTYimMb1Q2gcEUxwQOX+4AkO8Oe3v2rPVd2a8fjsdTz6+To+X53GcyO70rZxeKUeq6p5M0GsANqJSAKQhjPIfIN7AxGJBI6pah4wHlioqpkishO4QERCcS4xDQFq5Cx8L7zwAuvXrycpKYkFCxZw5ZVXsn79+uJnA6ZNm0ajRo04fvw4ffr0YfTo0URFRZXYx5YtW5g5cyZvvfUW1113HZ999hk33XRTucfbtm0bwcHBHD58GIC//OUvNGjQgHXr1gFw6NAhdu/ezR//+EdWrVpFw4YNGTp0KF988QUjRowgOzubLl268Mwzz7Bp0yb+9re/8eOPPxIYGMi9997LjBkz6Ny5M2lpaaxfvx6g+FjGnKmc/EK2H8gmZX/2yUTgSgbH8wuL20WEBNC2cTiDO8SQEB1Os8gQ18k/hMb1gwkL9u1AcdvG4cy68wI+WbWL5+Zu5opXF3HPoDbcO7hNpSelquK1b1RVC0TkfmAe4A9MU9UNInK3a/0UoCMwXUQKcQav73CtWyYinwKJQAHOpaepHg5zRk71S7+q9O3bt8SDY6+99hqzZ88GYNeuXWzZsqVMgkhISKBHjx4A9OrVi+3bt5e7/27dunHjjTcyYsQIRowYAcC3337LrFmzits0bNiQhQsXMmjQIGJinFl+b7zxRhYuXMiIESPw9/dn9OjRAHz33XesWrWKPn36AHD8+HEaN27M1VdfzdatW3nggQe48sorGTp06Dl9L6b2O5id5zrxn0gCTkLYdfBY8RiACDSPrEebmHD69YuiTUw4bWLCaNM4nKiwoGp/67Sfn3B9n5Zcel4sf/lqI69+t4Wv1u7muZFd6dc66vQ7qGa8mnJVdS4wt9SyKW7vlwLtytn2KeApb8bnC2FhYcXvFyxYwLfffsvSpUsJDQ1l0KBBHh8sCw4+Oejl7+/P8ePHy93/119/zcKFC5kzZw5/+ctf2LBhA6pa5n8s1fKHa0JCQorHHVSVW2+9leeff75MuzVr1jBv3jwmTZrExx9/zLRp08r/w02dUFikpB46VtwDONkjyC5xF1BwgB+tY8Lp2rwBI3o0p23jcNrEhJMQHUa9oJr5a9tdTP1gXht7PqN6NueJL9Zz/dSfGNOnBY8N70iD0Joz9UztvHm3Gqlfvz5Hjx71uO7IkSM0bNiQ0NBQNm/ezE8//XROxyoqKmLXrl0MHjyYAQMG8OGHH5KVlcXQoUN5/fXXeeWVVwDnElO/fv2YOHEiGRkZNGzYkJkzZ/LAAw+U2eeQIUO49tpreeihh2jcuDEHDx7k6NGjhIWFERQUxOjRo2nTpg233XbbOcVuapZjeQVsdfUAnB6B835rRjZ5BScHhqPDg2gdE86wzk1oExNWnAiaR9arFYO4pzOoQ2PmPzSQV7/dwtuLt/Htpn08eXVnru7WtNr3hsAShNdFRUXRv39/unTpQr169YiNjS1ed/nllzNlyhS6detGhw4duOCCC87pWIWFhdx0000cOXIEVeWhhx4iMjKSJ554gvvuu48uXbrg7+/PU089xahRo3j++ecZPHgwqsoVV1zBtddeW2afnTp14tlnn2Xo0KEUFRURGBjIpEmTqFevHuPGjaOoyDkZeOphmNqjqEj54Zd0ZizbyaY9maQdPtmL9RNoFRVGm5gwLmkf41wWauxcGooMDfJh1NVDaFAAj13RkWt6NOOxz9fx4MzVfLYqlWdHdKFFo1Bfh3dKcqpLDTVN7969tXRFuU2bNtGxY0cfRWTs+6/ZsnML+DwxlXd+3M7WjGxiI4K5qE20My4QE07bxuG0jAqtsYOwVa2wSJm+dDsvzvuZQlUevqw9t/dPqNAttd4iIqtUtbenddaDMMaUkXb4ONOXbGfm8p1k5hTQPa4Br47pwRVdmxLow5NZTefvJ4zrn8Cwzk148sv1PDd3M1+s3s3zo7rSvUWkr8MrwxJEDXXffffx448/llg2ceJExo0b56OITE2nqiTuPMS0xdv5ZsNeAC7v0oTb+yfQs2VkjbhmXlM0i6zHW7f05pv1e3lqzgZGvvEjt14Uz++GdiDcx7fruqs+kZgzMmnSJF+HYGqJ/MIi5q7bw7TF21iTeoSIkADGX5zALRfG0zzS8wOZ5tyJCMO7NqV/u2j+/s1m3l2ynXnr9/LMtV34VafY0++gCliCMKaOOpSdx4fLdzJ96Xb2ZebSOjqMv1zbmdG94mrt7KQVVpALxw6Ueh10/s3OOLksNxP8Akq+/APBLxD8/N3eB4D/iTaBJd5H+AfybJQ/d/fP59/r9/PDjDnsa96Iq3u0ICIstNR+S7937SsgBBpX/lhfHf+vwJi6Z8u+o0z7cTuzV6eSk1/Exe2ieWFUNy5pH1M7bz0tKoTjhzyc8A9Atodlxw5Cnudb0wEIiYTQqJOvokIoKnBe+cdPvi/Md73Pd9oU5nt4X1C82zjgHoBAYD/gcXrScoTFwCPJZ/PtnJIlCGPqgKIiZeGWdP61eBuLtmQQHODHqJ7Nue2iBDo0qe/r8M5MYT4cSfVwss8o+Uv/xOv4IcrOE+oSGOY60Tdy/o1uV/JziVc01Gvo/GKvLKpuCSbflTgK2ZlxhBf/s4E1OzPo3iyMh4e0Jj4yyNU23y35uF7inbvILEF42dlO9w3wyiuvMGHCBEJDq/e90qb6OpZXwOeJabzz4zZS0rNpXD+YR4Z1YGzfljQKq8bPKBTkweEdcHBrydeBFDi8E7Sw7DZ+gSdP5mFR0KSLh5N8I+dEf+J9oI/HWESchOMfAJwsmtkyPIZX727D54lpPPv1Rv4zI4O7Brbh/kvbEhJYdbcU23MQXrZ9+3auuuqq4kntzsSJGV2jo6O9EFlZ3pja29fff121+/Bxpi/dwczlOzlyPJ9ucQ24vX8CV3RtSlBANblNNT+nZBI4kHLy/ZFdoG5TdQfVh6jW0Kg1NGoDDeMhPLbkr/3g+s4Jt5Y5mJ3Hs19v5PPENOKjQnluZFcualt55wR7DsKH3Kf7vuyyy2jcuDEff/wxubm5jBw5kqeffprs7Gyuu+46UlNTKSws5M9//jP79u1j9+7dDB48mOjoaL7//vsy+y4sLPQ45XZycjJ333036enp+Pv788knn9C6dWv+8Ic/8J///AcR4YknnuD6669nwYIFPP3008VTkK9bt45HH32UBQsWkJuby3333cddd93Fnj17uP7668nMzKSgoIDJkydz8cUX++AbNafi3Ka6jf+s34uqFt+m2qtVQ9/cppp/HA5tL3nyP7gVDm5zkoD7pZ+QBs7JP64PdLveSQZRbZx/Q6Nq5cm/IhqFBfHydT0YdX4cf/piHTe8vYzRPeP405Udvd4LrFsJ4j+Pwt51lbvPJl1h+Avlrnaf7nv+/Pl8+umnLF++HFXlmmuuYeHChaSnp9OsWTO+/vprwJmjqUGDBrz88st8//335fYgkpKSPE65feONN/Loo48ycuRIcnJyKCoq4vPPPycpKYk1a9aQkZFBnz59GDhwIADLly8vnoJ86tSpNGjQgBUrVpCbm0v//v0ZOnQon3/+OcOGDeNPf/oThYWFHDt2rBK/RHMu8guL+Gb9Xv61eBtJuw5TPySAOwYkcMuFrYhrWAWXJ/OynRN+8ck/5eTnzLSSbes1ck74LS+AqBtdPQLXK7SR92OtwQa0i2bebwfyz/9t4c0ftvL9z/t54sqOjDy/udeSf91KED42f/585s+fz/nnnw9AVlYWW7Zs4eKLL+b3v/89f/zjH7nqqqsq/Mu8devWZabcPnr0KGlpaYwcORJwZmYFWLx4MWPHjsXf35/Y2FguueQSVqxYQURERIkpyOfPn8/atWv59NNPASdZbdmyhT59+nD77beTn5/PiBEjiqcfN75z+FgeM5fvYvrS7ew5kkNCdBjPXNuZ0T3jKrc2QmEBZO2FzN3O4PChbSd7AQdSnHXuwmKcE37CwJIJoFGCM8hrzlpIoD+PDDuPq7s78zo9/PEaPk9M468ju9AqKuz0OzhDdStBnOKXflVQVR577DHuuuuuMutWrVrF3Llzeeyxxxg6dChPPvnkaffXsGHDMlNun5ix1dOxy+M+Bbmq8s9//pNhw4aVabdw4UK+/vprbr75Zh555BFuueWW08ZoKl/y/ize+XEbnyU6t6n2bxvFX0d2YVD7xmd+m2pBHhzd45z8M9Nc/5547/qcta/keAA41/8btYa2v3JO/O5JIKRB5f2xxqPzmkTw2d0XMWPZDv7+zc+MmPQjPz56aaU/v1K3EoQPuE/3PWzYMP785z9z4403Eh4eTlpaGoGBgRQUFNCoUSNuuukmwsPDeffdd0tsW94lpoyMjDJTbkdERBAXF1dcHS43N5fCwkIGDhzIm2++ya233srBgwdZuHAh//jHP9i8eXOJfQ4bNozJkydz6aWXEhgYyC+//ELz5s3JyMigefPm3HnnnWRnZ5OYmGgJogqpKgu3ZDBt8TZ++CWdoAA/RvZozrgB8ZzXJMLzRgW5bif8Uif94pP/fsrcAhoUDhHNIaIZtOkIDVzvTyyLbAXBNbuUZm3g5yfcfGE8Qzs3YW3qEa883GgJwsvcp/sePnw4N9xwAxdeeCHg1H7+4IMPSE5O5pFHHsHPz4/AwEAmT54MwIQJExg+fDhNmzb1OEidlpbmccrt999/n7vuuosnn3ySwMBAPvnkE0aOHMnSpUvp3r07IsLf//53mjRpUiZBjB8/nu3bt9OzZ09UlZiYGL744gsWLFjAP/7xj+K62tOnT/fm12Zc8guLmJO0mzcXpvDLvixi6gfzu8vac0PPGKKKDkDmGliT5uHX/27ITi+7w+AGzkm+QXNn/OzESb84ATSHkHISjqmWYiNCuKxTyOkbngW7zdV4lX3/Z+d4XiEfrdjJW4u2sf/wUR6P/JbLG+wglgP4Ze6G4wfLbhQSCQ3iSp3w3U78EU2dW0GNceOz21xF5HLgVZya1G+r6gul1jcEpgFtgBzgdlVdLyIdgI/cmrYGnlTVV7wZrzG+duRYPtOXbuedJds5mJ3HNXHZPFvvVSIOrYeIzhDZAlr0dTvpNzt58g+q/EFKU7d5LUGIiD8wCbgMSAVWiMgcVd3o1uxxIElVR4rIea72Q1T1Z6CH237SgNneirUm6NevH7m5uSWWvf/++3Tt2tVHEZnKtC8zh7cXbeXDZTvJzivk0g4x/Ln5KhJWPAMBwXDd+9DpGl+HaeoYb/Yg+gLJqroVQERmAdcC7gmiE/A8gKpuFpF4EYlV1X1ubYYAKaq6w4uxVnvLli3zdQjGC7ZlZPPmDyl8nphGoSpXd2vKPf0a0WH5E7BkjnOr6Mg3nZ6CMVXMmwmiObDL7XMq0K9UmzXAKGCxiPQFWuFMauieIMYAM88lEFW1Yic+UJvGtyrb+rQjTF6Qwtz1ewj09+P6Pi248+LWtMxcCZ/f4AwwX/YMXPgA+FWTqTFMnePNBOHpjFz6jPEC8KqIJAHrgNVA8fy3IhIEXAM8Vu5BRCYAEwBatmxZZn1ISAgHDhwgKirKkkQVUlUOHDhQ/KCecb6TpVsPMHlBCou2ZFA/OIB7LmnDuP4JxNQT+P6v8OOrzvQSY/8Lzc73dcimjvNmgkgFWrh9jgN2uzdQ1UxgHIA4Z+9trtcJw4HEUpecSlDVqcBUcO5iKr0+Li6O1NRU0tM93PJnvCokJIS4uDhfh+FzRUXKfzft440FKazZdZjo8GD+ePl53HhBSyJCAiEjGT68A/YkQc9b4fLnbcDZVAveTBArgHYikoAzyDwGuMG9gYhEAsdUNQ8YDyx0JY0TxnKOl5cCAwOLp5EwNUhBHhQcr9FP5eYVFPFlUhpTfkghJT2blo1C+evILozuGedM2awKq96Dbx51BqKv/wA6Xu3rsI0p5rUEoaoFInI/MA/nNtdpqrpBRO52rZ8CdASmi0ghzuD1HSe2F5FQnDugys5LYWqf7AOwa5nrtRx2JzqFUHrcAAMedqZwqCGO5RUwa/ku3l60ld1HcujYNILXxp7PFV2aEODvGk84dhD+/SBs+rcNRJtqq9Y/KGeqoaIiOLAFdv7kJINdP8EBV7lEv0Bo2p0j0T3Ze/AwbdO+wE8Lodv1yMDfO9fnq6nDx/J4b8kO3l2yjUPH8ukb34h7BrdhUPuYkuNfW3+A2Xc7A9FD/mwD0canrB6E8a28Y5C26mTvYNcyyDnsrKvXCFr0g/Nvcv5tdj6z1x/gT7PXcyyvkMZcwl0BX3Fj0icErplFYsQQfm5/F9HxXWjbOJxWUWEE+vv25LrnyHH+tWgbHy7fybG8Qn7VsTF3X9KG3vGlpq8uyIPvn4UfX3MNRH8LzXr4JGZjKsIShKl8R9LcksFPTg2OE8XZozs4D3y16Oe8otoWF4I5llfAU19s4JNVqfSNb8RfR3bh0LF8kvcPZFLaDjpue4/BmXPoteJbvlp2AfcUjGSbtKBVVCjtGtenbePw4lfrmDCvTF7mLiU9izd/SGH26jSKFK7p3oy7LmntefK8jC3w2XhnILrXbTDsORuINtWeXWIy56awAPatP5kMdi13VQoDAupBXG9naogW/ZxKYeUUhfll31Hum5FIcnoW9w9uy8Qh7U5er3eXnUHe4tfwX/k2/vnZbG50KR/VG8MPmbHsOHCMwqKT/z03j6xHu9hw2saEl0gekaHnVoVrbephJi9I4ZsNewlye4ahRSMPxXlUIfE9+OYxZyD6mn/aQLSpVk51ickShDkzxw9D6kpXMlgGqasgP9tZV78ZtOx3snfQpCv4B55yd6rKJ6tSefLL9YQHB/LK9T0Y0K4C9XaPHYSf3oBlb0JuJpx3Ffn9f8+2oLYk788qfm3Zn8XW9CxyC07WM4gODzqZMGLCaevqfcRGBJf7rIyqsiTFeYZhcXIG9UMCuPXCeG7rH090eHD5Mc55ADZ/BQmXwMgpNhBtqh1LEObsqDqVw9x7B/s3AQriD026QIsLTvYQIlucdpfusnMLeOKL9cxencZFbaJ4ZUwPGtc/wwfrjh+Cn6bAT5Mh9wi0vxwG/gHiehU3KSxS0g4dJzn9qJM09mWRnO4kkKM5xc9lUj84gDZuPY0TPY/NezOZvCCFNalHiKkfzPgBCdzQryX1Q06R/EoMRD8JF95vA9GmWrIEYSquMB9WvA3bFzs9hBM1BYIbnEwELfpC817nVDRm4+5M7v8wke0Hsvntr9pz3+C2+J9pNTR3OUdg2VRY+rozAN72V3DJH51Yy6GqpB/NLe5pFPc80rNIP1pyYsRWUaHcNbANo3o2d55hKE9BHvzvL7Dkn874yui3bSDaVGuWIEzFLZ0E8x53ykee6B20vMAZXK6EX8CqyofLd/L0vzcSWS+QV8ecz4VtoiohcJecTCfBLfmnUzOh9SAnUbS66Ix2c+RYvquXcZQG9YK4rFPs6RNYxhb47A7YswZ6jYNhf7WBaFPtWYIwFaMKb1zgFJUZ/22l7/5oTj6Pfr6Or9fuYWD7GF6+rnv51+/PVW4WrJwGS15zekHxF8Mlf3D+rew5ucoMRL8OHa+q3GMY4yX2HISpmNQVkL7ZudOmkq1PO8J9HyaSeug4f7i8A3cPbIPfuVxSOp3gcOj/IPQZD6vehR9fgfeuhpYXOYmi9aDKSRRlBqLfdIr3GFMLWIIwJyW+B4Fh0Hlkpe1SVXlvyXaem7uZqPAgZk24gD6lHyDzpqBQuPBe6D0OEt+Hxf8P3h8BcX2dS09th5x9oti6wDUQnQGX/cUGok2tYwnCOHKPwvrZ0GVUpdUtPnIsnz98toZ5G/Yx5LzGvPib7jQMO7dnEM5aYD3oNwF63QqrP3ASxYzR0KynkyjaD6t4oig9ED12lg1Em1rJEoRxrP/ceZ6h562VsrvVOw/xwMzV7D2SwxNXduSOAQnVox5HQDD0uQPOvxnWzIRFL8LM66FJNydRdLji1L2A9F/g8/FuA9HPOb0UY2ohSxDGkTgdYs5znnw+B6rKvxZv44X/bCY2IoRP7r6Q81s2rKQgK1FAkNOb6HEDrP0IFr4IH90IsV1g4CPQ8ZqSiULVGcv45jGnN3L9DBuINrWeJQgD+zZC2krn1/A5/Mo/lJ3H7z9Zw3eb9zO0Uyz/+HV3GoSe+klqn/MPdCYK7DYG1n8KC/8Bn9wKMR1h4O+d8ZicIycHolsPghFTbCDa1AmWIAysft+ZZrvbmLPexcrtB3lw5moysvL4v6s7cetF8dXjklJF+QdA9zHQ9TewYTb88HfnmYYf/uaMz2RnwNBn4YL7bCDa1BmWIOq6glznWvx5V0LYmT+wVlSkTFmYwkvzf6F5ZD0+u+ciusbV3Cpw+PlD119D51Gw6UtY9JJzSemGj6Bpd19HZ0yVsgRR123+2pnPqOctZ7zpgaxcHv54DT/8ks6VXZvy/OiuTo3l2sDPz7m8VIm3/BpT01iCqOsSp0ODFtB68Blt9tPWA0yctZpDx/J5dkQXbuzXsmZdUjLGnJYliLrs0A7Y+j0MeqzC19ULi5RJ3yfzyre/EB8Vxju39aVTMw8FcowxNZ5XR9tE5HIR+VlEkkXkUQ/rG4rIbBFZKyLLRaSL27pIEflURDaLyCYRudCbsdZJSTMAgR43Vqj5/qM53DJtGS//9xeu6d6MOQ8MsORgTC3mtR6EiPgDk4DLgFRghYjMUdWNbs0eB5JUdaSInOdqP8S17lXgG1X9tYgEAfY0UmUqKnSeKG5zaYXqOCzeksFvP0oiKzefv43uynW9W9glJWNqOW/2IPoCyaq6VVXzgFnAtaXadAK+A1DVzUC8iMSKSAQwEPiXa12eqh72Yqx1T8r/IDPttIPTBYVFvDT/Z26etozI0EC+vG8A1/ex8QZj6gJvjkE0B3a5fU4F+pVqswYYBSwWkb5AKyAOKATSgXdEpDuwCpioqtlejLduSZwOoVHO1BLl2HskhwdnrWb5toP8ulccz1zbmdAgG7Yypq7wZg/C00/M0sUnXgAaikgS8ACwGijASVw9gcmqej6QDZQZwwAQkQkislJEVqanp1dW7LVbVjr8PBe6j3WmnPBgwc/7ueK1RaxLPcJLv+nOi7/pbsnBmDrGm//HpwLuF7fjgN3uDVQ1ExgHIM41i22uVyiQqqrLXE0/pZwEoapTgangFAyqxPhrrzUzoajAmbCulMIi5R/zfmbKDymc16Q+r9/Qk7aNz760qDGm5vJmD2IF0E5EElyDzGOAOe4NXHcqnfgJOx5YqKqZqroX2CUiHVzrhgDug9vmbKk6U2u06AeNzyuz+oOfdjDlhxTG9m3BF/f1t+RgTB3mtR6EqhaIyP3APMAfmKaqG0Tkbtf6KUBHYLqIFOIkgDvcdvEAMMOVQLbi6mmYc7RrGWT84pTFLCW/sIipC7fSu1VDnhvZ1QaijanjvHpRWVXnAnNLLZvi9n4p0K6cbZOAc5t72pSVOB2Cwj1OIfFl0m7SDh/nLyM6W3Iwxnj3QTlTzeRkOjOVdhnt1Gx2U1SkTF6QzHlN6jO4Q2MfBWiMqU4sQdQl6z+D/GMen32Yv3EvKenZ3Du4rfUejDGAJYi6JXE6NO4EzXuVWKyqvLEghVZRoVzRpYmPgjPGVDeWIOqKvethd6LTeyjVQ1icnMHa1CPcfUkbAvztPwljjMPOBnXF6vfBPwi6XV9m1RvfpxAbEcyons19EJgxprqyBFEX5OfAmllw3lUQ2qjEqsSdh1i69QDjB7QmOMDfRwEaY6ojSxB1weavIOewx8HpN75PoUG9QG7o17Lq4zLGVGuWIOqCxOkQ2RISLimx+Oe9R/l20z5uuyiesGCbZ8kYU5IliNru4DbY9oMz71KpqnGTFyQTGuTPbRfF+yY2Y0y1ZgmitkuaAeIHPW4osXjXwWP8e+0ebujbkoZhnmd0NcbUbZYgarOiQlg9A9oMgQZxJVa9uTAFP4HxF7f2UXDGmOrOEkRtlvwdHN1dZnB6/9EcPl6Zyq97xdGkQYiPgjPGVHeWIGqzxPcgLAbaX15i8b8Wb6OgsIi7BrbxUWDGmJqgQglCRAaIyInCPjEikuDdsMw5y9oPv3wD3ceUqBp35Fg+HyzdwRVdmxIfHebDAI0x1d1pE4SIPAX8EXjMtSgQ+MCbQZlKUFw1ruTlpelLt5OdV8i9g9r6KDBjTE1RkR7ESOAanLrQqOpuoL43gzLnSNV59qHlhRDTvnjxsbwC3lmyncEdYujULMKHARpjaoKKJIg8VVVAAUTErktUdzuXwoHkMjWnZy3fxcHsPO4bbL0HY8zpVSRBfCwibwKRInIn8C3wlnfDMuck8X0Iqg+dRxQvyiso4q1FW+kb34je8Y3K39YYY1xOOb+COJVjPgLOAzKBDsCTqvrfKojNnI2cI07VuO7XQ9DJzt4XSWnsOZLDc6O6+jA4Y0xNcsoEoaoqIl+oai/gjJOCiFwOvAr4A2+r6gul1jcEpgFtgBzgdlVd71q3HTgKFAIFqmr1qSti3adQcLzEsw+FRcqUBSl0ahrBoPYxPgzOGFOTVOQS008i0udMdywi/sAkYDjQCRgrIp1KNXscSFLVbsAtOMnE3WBV7WHJ4Qysfh9iu0CznsWL5m3Yy9aMbO4d3MbKiRpjKqwiCWIwTpJIEZG1IrJORNZWYLu+QLKqblXVPGAWcG2pNp2A7wBUdTMQLyKxZxC/cbdnLexe7QxOuxKBqjLp+2QSosMY3qWpjwM0xtQkFZnjefhZ7rs5sMvtcyrQr1SbNcAoYLGI9AVaAXHAPpy7puaLiAJvqurUs4yj7lj9PvgHQ7frihct3JLBht2Z/G10V/z9rPdgjKm40yYIVd0hIt2Bi12LFqnqmgrs29PZSEt9fgF4VUSSgHXAaqDAta6/qu4WkcbAf0Vks6ouLHMQkQnABICWLetw0Zv847D2I+h4dYmqcW98n0yTiBBGnh93io2NMaasijxJPRGYATR2vT4QkQcqsO9UoIXb5zhgt3sDVc1U1XGq2gNnDCIG2OZat9v1735gNs4lqzJUdaqq9lbV3jExdXgAdtNXzh1MPU8++7Bqx0GWbTvInQNbExRg024ZY85MRc4adwD9VPVJVX0SuAC4swLbrQDaiUiCiAQBY4A57g1EJNK1DmA8sFBVM0UkTETqu9qEAUOB9RX7k+qoxPcgshXEDyxe9Mb3KTQMDWRs3xan2NAYYzyryBiE4NxqekIhni8flaCqBSJyPzAP5zbXaaq6QUTudq2fAnQEpotIIbARJxkBxAKzXXfcBAAfquo3FfuT6qCDW2H7Irj0ieKqcZv2ZPLd5v089Kv2hAZZOVFjzJmryJnjHWCZiMx2fR4B/KsiO1fVucDcUsumuL1fCrTzsN1WoHtFjmGA1R+4qsbdWLxo8oIUwoL8ufWiVj4MzBhTk1VkkPplEVkADMDpOYxT1dXeDsxUUGGBUzWu7WUQ0QyAHQey+WrtbsZf3JrIUCsnaow5O6dNECJyAbBBVRNdn+uLSD9VXeb16MzpJX8LWXuh50vFi6b8sJUAPz/GD7CyHcaYs1eRQerJQJbb52zXMlMdJE6HsMbQfhgA+zJz+GxVKr/uHUfjCCsnaow5exVJEOKa7hsAVS2iYmMXxtuO7nWqxvUYC/6BgKucaFERdw1s7ePgjDE1XUUSxFYReVBEAl2vicBWbwdmKmDNTNDC4qpxh4/l8cFPO7i6ezNaRVnZDmPMualIgrgbuAhIc7364Xpy2fhQcdW4iyDaKQD03pIdHMsr5J5BbXwcnDGmNqjIXUz7cR5yM9XJjh+d5x8G/gGA7NwC3lmyjV91bMx5TaycqDHm3JXbgxCRO0Wkneu9iMg0ETnimtG1Z3nbmSqS+D4ER0AnZ4Lcmct3cvhYPvcMsnKixpjKcapLTBOB7a73Y3EeXGsNPEzZug2mKh0/DBu/gK6/hqBQcgsKeXvRNvolNKJXq4a+js4YU0ucKkEUqGq+6/1VwHRVPaCq3wI2AupL6z6BgpziqnGzE9PYm5nDfYOt92CMqTynShBFItJUREKAIcC3buvqeTcsc0qr34cmXaFpDwqLlDcXbqVL8wgubhft68iMMbXIqRLEk8BKnMtMc1R1A4CIXILd5uo7u5Ngzxrn1lYR/rN+D9sysrl3UFsrJ2qMqVTl3sWkql+JSCugvqoeclu1Erje65EZz4qrxv3GVU40hdYxYQzr3MTXkRljaplTPgehqgWlkgOqmq2qWeVtY7wo/zis/cS5c6leQxb8ks6mPZncfUkbKydqjKl0VmasJtk4B3JPVo174/tkmjUIYUSP5j4OzBhTG1mCqEkSp0PDBGg1gBXbD7Ji+yErJ2qM8ZpTPSg3TER+7WH5jSJymXfDMmUcSIEdi53eg58fb3yfTKOwIMb0aenryIwxtdSpfno+DfzgYfl3wDPeCceUa/X7TtW47jewYfcRvv85ndv7x1MvyN/XkRljaqlTJYhQVU0vvVBV92IPylWtwgJI+hDaDYOIpkxekEJ4cAA3Xxjv68iMMbXYqRJEiIiUuQ1WRAKp4INyInK5iPwsIski8qiH9Q1FZLZrfqflItKl1Hp/EVktIl9V5Hi11pb5kLUPet7Ctoxs5q7bw00XtKJBvUBfR2aMqcVOlSA+B94SkeLeguv9FNe6UxIRf2ASMBzoBIwVkU6lmj0OJKlqN+AWys7xNBHYdLpj1XqJ0yE8FtoN5c0fUgjw9+P2AfG+jsoYU8udKkE8AewDdojIKhFJxHmqOt217nT6AsmqulVV84BZwLWl2nTCGdNAVTcD8SISCyAiccCVwNsV/3Nqocw9sGUe9LiBvVkFfJaYynW942hc38qJGmO861RPUhcAj4rI08CJWeCSVfV4BffdHNjl9jkVp9iQuzXAKGCxiPQFWgFxOInpFeAPQP0KHq92WvMhaBGcfzNvLdpKkcJdA60gkDHG+8pNECIyqtQiBSJFJElVj1Zg354e7dVSn18AXhWRJGAdsBooEJGrgP2qukpEBp3yICITcFW4a9mylt3yWVTk1H1oNYBDIS2Yufx/XNO9GS0ahfo6MmNMHXCqinJXe1jWCOgmIneo6v9Os+9UoIXb5zhgt3sDVc0ExoFTlAjY5nqNAa4RkSuAECBCRD5Q1ZtKH0RVpwJTAXr37l06AdVsOxbDoW0w6DHeXbLdyokaY6rUqS4xjfO03DWB38eUvVxU2gqgnYgk4NSyHgPcUGpfkcAx1xjFeGChK2k85nrh6kH83lNyqPUS34fgBmS1uYJ3v1jKZZ1iaR9bt6+4GWOqzmlrUpemqjtct7qerl2BiNwPzAP8gWmqukFE7natnwJ0BKaLSCGwEbjjTOOptY4fgo1fQs+bmZmYzpHj+dxrvQdjTBU64wQhIh2A3Iq0VdW5wNxSy6a4vV8KtDvNPhYAC840zhpv7SdQmEte95t4672tXNQmivNbWjlRY0zVOdUg9b8pO6jcCGgK3OzNoOo8VefZhybd+DQtiv1Hd/PydT18HZUxpo45VQ/ixVKfFTgAbHGNGRhv2ZME+9ZROPwfvLkwhW5xDejfNsrXURlj6phTDVJ7mqgPEekvIjeo6n3eC6uOS5wOASF8Ixez48BWptzUy8qJGmOqXIXGIESkB84dSNfh3IZ62qk2zFnKOwbrPkU7XcM/l6TTtnE4QzvF+joqY0wddKoxiPY4t6aOxbm09BEgqjq4imKrmzZ+CbmZrI6+hs3Lj/LSb7rjZ+VEjTE+cKoexGZgEXC1qiYDiMhDVRJVXZY4HW3Umr+ub0TzyFyu6dHM1xEZY+qoU03WNxrYC3wvIm+JyBA8T59hKktGMuxcwq5Wo1m18zATBrYm0N/KiRpjfKPcs4+qzlbV64HzcJ5DeAiIFZHJIjK0iuKrW1ZPB/HnH/t7ER0exPV9Wpx+G2OM8ZLT/jxV1WxVnaGqV+HMp5QElCn+Y85RYT4kzSSz5RD+nVLEuP4JhARaOVFjjO+c0fULVT2oqm+q6qXeCqjOWvcpZO/ng/zB1A8O4OYLW/k6ImNMHWcXuKuDoiJY/DK5UR35x7aW3HxhKyJCrJyoMca3LEFUB5v/DRm/8Gm96wjy9+f2AQm+jsgYYyxB+JwqLHqJvIh4ntnWnjF9WhAdHuzrqIwxxhKEzyV/B3vW8HLOlTQIDeG+S9uefhtjjKkCZzzdt6lcuuhFDvpH8372hXxwVy8a1w/xdUjGGANYD8K3dixBdi7lnzlX8NTIHlbvwRhTrVgPwof2fvVXAjSCwD63cV1veyjOGFO9WA/CR35ZvYgm6Yv5NmI0f7j6fF+HY4wxZViC8IH9R3NInfMsRwll2LgnbL4lY0y15NUzk4hcLiI/i0iyiJSZnkNEGorIbBFZKyLLRaSLa3mI6/MaEdkgIk97M86qlFdQxF/f/YJBRcvIOf8OGjaK9nVIxhjjkdcShIj4A5OA4UAnYKyIdCrV7HEgSVW7AbcAr7qW5wKXqmp3oAdwuYhc4K1Yq9JTczYwcP8HaEAwMb/6ra/DMcaYcnmzB9EXSFbVra4a1rOAa0u16QR8B6Cqm4F4EYlVR5arTaDrpV6MtUp88NMOFq1YxUj/Jfj3Hgdh1nswxlRf3kwQzYFdbp9TXcvcrQFGAYhIX6AVzoyxiIi/iCQB+4H/quoyL8bqdcu3HeT/5mzg6ahvET8/uOgBX4dkjDGn5M0E4am4UOlewAtAQ1cieABYDRQAqGqhqvbASRh9T4xPlDmIyAQRWSkiK9PT0ysr9kq1+/Bx7p2xih6ROVx6fD7SYyw0KJ0rjTGmevFmgkgF3G/ujwN2uzdQ1UxVHedKBLcAMcC2Um0O4xQsutzTQVR1qqr2VtXeMTExlRZ8ZcnJL2TC+yvJyS/irfbLkKJ86P9bX4dljDGn5c0EsQJoJyIJIhIEjAHmuDcQkUjXOoDxwEJVzRSRGBGJdLWpB/wKp0Z2jaKqPPrZWjbszuT1EfE03PgBdB4FUW18HZoxxpyW156kVtUCEbkfmAf4A9NUdYOI3O1aPwXoCEwXkUJgI3CHa/OmwHuuO6H8gI9V9Stvxeotby/axhdJu/ndZe0ZdOQzyMuCix/2dVjGGFMhXp1qQ1XnAnNLLZvi9n4p0M7DdmuBGv148Q+/pPP8fzYxvEsT7u8fC69Mhg5XQGxnX4dmjDEVYo/wesH2jGwe+DCR9rH1efE33ZFV70LOYbj4d74OzRhjKswSRCXLyi3gzukr8fMT3rqlN2F+BbD0dUi4BOJ6+zo8Y4ypMEsQlaioSHn4oyS2ZmQz6YaetGgUCkkfQNY+6z0YY2ocSxCV6LX/bWH+xn08fkVH+reNhsJ8+PFViOsDCQN9HZ4xxpwRSxCVZN6Gvbzy7RZG94zj9v7xzsJ1n8LhnU7vQTw9N2iMMdWXJYhK8Mu+ozz8URLd4xrw15FdEBEoKoLFL0NsF2jv8Rk/Y4yp1ixBnKMjx/KZMH0l9YICmHJzL0IC/Z0Vm7+CjF9gwEPWezDG1EiWIM5BYZFy/8xE0g4f582be9K0QT1nhSosehEatYbOI30bpDHGnCVLEOfg799sZtGWDJ65tgu9WjU6uSLlO9izxuk9+Pn7LkBjjDkHliDO0pdJaby5cCs3XdCSsX1blly58CWIaA7dxvgmOGOMqQSWIM7C+rQj/OHTtfSNb8STV5WaOmPHEti5BC56EAKCPO/AGGNqAEsQZygjK5cJ01cSFRbEGzf1JCig1Fe46CUIjYaet/gmQGOMqSSWIM5AfmER985I5EB2Hm/e3Jvo8OCSDXYnQfK3cOG9EBTqkxiNMaayeHU219rmmX9vZPm2g7w6pgdd4xqUbbD4ZQhuAH3GV31wxhhTyawHUUGzlu/k/Z92MGFga67t4aFcaPrPsHEO9L0TQjwkD2OMqWEsQVTAqh0H+fOX67m4XTR/vPw8z40WvwIBIXDBPVUamzHGeIsliNPYeySHuz9IpFlkPV4f2xN/Pw9PRR/aAWs/gl63QVh0lcdojDHeYGMQp5CTX8hd76/kWG4BM8b3o0FooOeGS14D8YOLHqjaAI0xxossQZRDVfnT7PWsST3Cmzf3on1sfc8Nj+6FxPehx1ho4GFswhhjaiivXmISkctF5GcRSRaRRz2sbygis0VkrYgsF5EuruUtROR7EdkkIhtEZKI34/TknR+381liKhOHtGNY5yblN1w6CYryof9vqyw2Y4ypCl5LECLiD0wChgOdgLEi0qlUs8eBJFXtBtwCvOpaXgD8TlU7AhcA93nY1mt+TM7gr3M3MbRTLBOHtCu/4bGDsHIadB4FUW2qKjxjjKkS3uxB9AWSVXWrquYBs4BrS7XpBHwHoKqbgXgRiVXVPaqa6Fp+FNgEVMn1m10Hj3Hfh4m0jg7j5et74OdpUPqE5VMhLwsufrgqQjPGmCrlzQTRHNjl9jmVsif5NcAoABHpC7QC4twbiEg8cD6wzFuBnnAsr4A7p6+kqEh565behAefYogm9yj8NBk6XAGxnctvZ4wxNZQ3E4Snn95a6vMLQEMRSQIeAFbjXF5ydiASDnwG/FZVMz0eRGSCiKwUkZXp6elnHayq8sgna/ll31H+eUNP4qPDTr3Byncg57BTTtQYY2ohb97FlAq0cPscB+x2b+A66Y8DEBEBtrleiEggTnKYoaqfl3cQVZ0KTAXo3bt36QRUYW8sSOHrdXt4bPh5XNI+5tSN83Ng6euQcAnE9T7bQxpjTLXmzR7ECqCdiCSISBAwBpjj3kBEIl3rAMYDC1U105Us/gVsUtWXvRgjAP/bvI8X5//MNd2bMWFg69NvkDQDsvZZ78EYU6t5rQehqgUicj8wD/AHpqnqBhG527V+CtARmC4ihcBG4A7X5v2Bm4F1rstPAI+r6tzKjvPwsTwmzkqiU9MI/ja6G3K6+tGF+fDjKxDXBxIGVnY4xhhTbXj1QTnXCX1uqWVT3N4vBcrcR6qqi/E8hlHpIkOD+PvobnRrEUm9oAqUB13/GRzeCcP/DqdLJsYYU4PZk9TA8K5NK9awqAgWvQyxXaD95d4NyhhjfMwm6zsTm7+CjJ9hwEPWezDG1HqWICpKFRa9CI1aQ+eRvo7GGGO8zhJERaV8B3vWOL0HvwqMVRhjTA1nCaKiFr0MEc2h2xhfR2KMMVXCEkRF7FgKO36Eix6EgKDTtzfGmFrAEkRFLHoJQqOh5y2+jsQYY6qMJYjT2Z0Eyf+FC++FoFBfR2OMMVXGEsTpLH4ZghtAn/G+jsQYY6qUJYhTSf8ZNs6BvndCSANfR2OMMVXKEsSpLH4FAkLggnt8HYkxxlQ5SxDlObQD1n4EvW6DsGhfR2OMMVXOEkR5lrwG4gcXPeDrSIwxxicsQXhydB8kvg89xkKDKimFbYwx1Y4lCE+Wvg5F+dD/t76OxBhjfMYSRGnHDsLKadB5FES18XU0xhjjM5YgSls+FfKy4OKHfR2JMcb4lCUId7lH4afJ0OEKiO3s62iMMcanLEG4W/Uu5ByGi3/n60iMMcbnvJogRORyEflZRJJF5FEP6xuKyGwRWSsiy0Wki9u6aSKyX0TWezPGYvk5sOSfkHAJxPWukkMaY0x15rUEISL+wCRgONAJGCsinUo1exxIUtVuwC3Aq27r3gWqrvBz0gzI2me9B2OMcfFmD6IvkKyqW1U1D5gFXFuqTSfgOwBV3QzEi0is6/NC4KAX4zupMB9+fAXi+kDCwCo5pDHGVHfeTBDNgV1un1Ndy9ytAUYBiEhfoBUQdyYHEZEJIrJSRFamp6efXaTrP4PDO53eg8jZ7cMYY2oZbyYIT2daLfX5BaChiCQBDwCrgYIzOYiqTlXV3qraOyYm5syjLCpyyonGdoH2VXdFyxhjqrsAL+47FWjh9jkO2O3eQFUzgXEAIiLANter6uRnQ4u+0HaI9R6MMcaNNxPECqCdiCQAacAY4Ab3BiISCRxzjVGMBxa6kkbVCa4P175epYc0xpiawGuXmFS1ALgfmAdsAj5W1Q0icreI3O1q1hHYICKbce52mnhiexGZCSwFOohIqojc4a1YjTHGlCWqpYcFaq7evXvrypUrfR2GMcbUGCKySlU9PvxlT1IbY4zxyBKEMcYYjyxBGGOM8cgShDHGGI8sQRhjjPHIEoQxxhiPatVtriKSDuw4y82jgYxKDKcms++iJPs+SrLv46Ta8F20UlWP8xTVqgRxLkRkZXn3Atc19l2UZN9HSfZ9nFTbvwu7xGSMMcYjSxDGGGM8sgRx0lRfB1CN2HdRkn0fJdn3cVKt/i5sDMIYY4xH1oMwxhjjkSUIY4wxHtX5BCEil4vIzyKSLCKP+joeXxKRFiLyvYhsEpENIjLx9FvVbiLiLyKrReQrX8fiayISKSKfishm138jF/o6Jl8SkYdc/5+sF5GZIhLi65gqW51OECLiD0zCKVbUCRgrIp18G5VPFQC/U9WOwAXAfXX8+wCniNUmXwdRTbwKfKOq5wHdqcPfi4g0Bx4EeqtqF8Afp2pmrVKnEwTQF0hW1a2usqezgGt9HJPPqOoeVU10vT+KcwJo7tuofEdE4oArgbd9HYuviUgEMBD4F4Cq5qnqYZ8G5XsBQD0RCQBCgd0+jqfS1fUE0RzY5fY5lTp8QnQnIvHA+cAyH4fiS68AfwCKfBxHddAaSAfecV1ye1tEwnwdlK+oahrwIrAT2AMcUdX5vo2q8tX1BCEeltX5+35FJBz4DPitqmb6Oh5fEJGrgP2qusrXsVQTAUBPYLKqng9kA3V2zE5EGuJcbUgAmgFhInKTb6OqfHU9QaQCLdw+x1ELu4lnQkQCcZLDDFX93Nfx+FB/4BoR2Y5z6fFSEfnAtyH5VCqQqqonepSf4iSMuupXwDZVTVfVfOBz4CIfx1Tp6nqCWAG0E5EEEQnCGWSa4+OYfEZEBOca8yZVfdnX8fiSqj6mqnGqGo/z38X/VLXW/UKsKFXdC+wSkQ6uRUOAjT4Mydd2AheISKjr/5sh1MJB+wBfB+BLqlogIvcD83DuQpimqht8HJYv9QduBtaJSJJr2eOqOtd3IZlq5AFghuvH1FZgnI/j8RlVXSYinwKJOHf/raYWTrthU20YY4zxqK5fYjLGGFMOSxDGGGM8sgRhjDHGI0sQxhhjPLIEYYwxxiNLEMachogUikiS26vSniAWkXgRWV9Z+zOmMtXp5yCMqaDjqtrD10EYU9WsB2HMWRKR7SLyNxFZ7nq1dS1vJSLficha178tXctjRWS2iKxxvU5MzeAvIm+5agvMF5F6rvYPishG135m+ejPNHWYJQhjTq9eqUtM17uty1TVvsDrOLO/4no/XVW7ATOA11zLXwN+UNXuOPMYnXhqvx0wSVU7A4eB0a7ljwLnu/Zzt3f+NGPKZ09SG3MaIpKlquEelm8HLlXVra5JDveqapSIZABNVTXftXyPqkaLSDoQp6q5bvuIB/6rqu1cn/8IBKrqsyLyDZAFfAF8oapZXv5TjSnBehDGnBst5315bTzJdXtfyMmxwStxKh72Ala5CtMYU2UsQRhzbq53+3ep6/0STpafvBFY7Hr/HXAPFNe6jihvpyLiB7RQ1e9xihZFAmV6McZ4k/0iMeb06rnNbgtOXeYTt7oGi8gynB9bY13LHgSmicgjOFXYTsx6OhGYKiJ34PQU7sGpRuaJP/CBiDTAKWz1/6zEp6lqNgZhzFlyjUH0VtUMX8dijDfYJSZjjDEeWQ/CGGOMR9aDMMYY45ElCGOMMR5ZgjDGGOORJQhjjDEeWYIwxhjj0f8HFwEh3lwoi0IAAAAASUVORK5CYII=\n",
366 |       "text/plain": [
367 |        "<Figure size 432x288 with 1 Axes>"
368 |       ]
369 |      },
370 |      "metadata": {
371 |       "needs_background": "light"
372 |      },
373 |      "output_type": "display_data"
374 |     }
375 |    ],
376 |    "source": [
377 |     "# plot training stats\n",
378 |     "train.plot_train_stats()"
379 |    ]
380 |   },
381 |   {
382 |    "cell_type": "code",
383 |    "execution_count": 31,
384 |    "id": "75e22bb5",
385 |    "metadata": {},
386 |    "outputs": [
387 |     {
388 |      "data": {
389 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAFNCAYAAAANRGjoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABAxElEQVR4nO3dd3wU1frH8c+TTkJCCqEmVOldQxEFARGpIl4LimIDxF7u9er1d1X02q7dawfEioCKIipSRKqgEgTpJfTQEyAJ6eX8/pglhpDAEnYz2d3n/Xrlld2Zye53Qphn55yZc8QYg1JKKd/lZ3cApZRS9tJCoJRSPk4LgVJK+TgtBEop5eO0ECillI/TQqCUUj5OC4HyGCIyQkTmOrHdeyLyeGVkqgwislNE+joejxORz+zOpLyLFgLlEo6DVbaIHBeRgyLyoYhUd+V7GGMmG2P6ObHdWGPMf1z53ieIiBGRTMd+7hWRV0XE3x3vVREiEiEir4vIbkfGJMfzmnZnU1WXFgLlSkOMMdWB84HOwL9LbyAiAZWeyvU6OPbzEuA64Dab8wAgIkHAfKAN0B+IALoDqUCXCryeN/xbKSdoIVAuZ4zZC/wItIXiT9F3i8hWYKtj2WARWS0ix0RkmYi0P/HzIhIvIl+LyGERSRWRtxzLbxGRpY7HIiKvicghEUkTkTUicuL9PhKRZ0q83mjHJ+MjIjJTROqVWGdEZKyIbBWRoyLytoiIk/uZBPwCdCzxehXZr6Yi8rNjWYqITBaRyLP8tQOMBBoAw4wxG4wxRcaYQ8aY/xhjZpXY3/NKZCr+XYlILxFJFpFHROQA8KGIbBSRwSW2D3BkPN/xvJtjP4+JyJ8i0qsCuZXNtBAolxOReGAgsKrE4iuBrkBrx0FkEnAHEAO8D8wUkWBHM8v3wC6gEVAfmFrG2/QDegLNgUisT+apZWTpAzwPXAvUdbxu6dcbjHUG08Gx3eVO7mdLoAeQ5Hhe0f0SR8Z6QCsgHhjnTIZS+gKzjTHHK/CzJ9QBooGGwBhgCnB9ifWXAynGmD9EpD7wA/CM42f+AUwXkdhzeH9lAy0EypVmiMgxYCmwCHiuxLrnjTFHjDHZwGjgfWPMb8aYQmPMx0Au0A2rCaMe8LAxJtMYk2OMWVrGe+UD4UBLQIwxG40x+8vYbgQwyRjzhzEmF/gXcKGINCqxzQvGmGPGmN3AAkp8wi/HHyKSCWwEFgLvOJZXaL+MMUnGmHnGmFxjzGHgVaxmp7MVA5T1OzgbRcCTjizZwOfAFSIS6lh/g2MZwI3ALGPMLMfZxzwgEetDgPIgWgiUK11pjIk0xjQ0xtzlOJCcsKfE44bA3x3NCcccxSMe60AZD+wyxhSc7o2MMT8DbwFvAwdFZLyIRJSxaT2sT+Enfu441plD/RLbHCjxOAuoDiAi6x0drsdFpEeJbc53bHMd1llO2Lnsl4jUEpGpjs7ndOAzoCKdu6lYZz3n4rAxJufEE0fz10ZgiKMYXMFfhaAhcE2p/b3YBRlUJdNCoCpLyWFu9wDPOorGia9QY8wUx7oGznRUGmP+Z4y5AKtztDnwcBmb7cM6YAEgImFYn5z3OvH6bYwx1R1fS0qtM8aYL4DlwBPnuF/PY/1+2htjIrA+aTvVT1HKT8Dljn0sTxYQWuJ5nVLryxqO+ETz0FBgg6M4gLVPn5ba3zBjzAsVyK5spIVA2WECMFZEujo6fcNEZJCIhAO/YzVvvOBYHiIiF5V+ARHp7Pj5QCATyAEKy3ivz4FbRaSjiARjNVf9ZozZ6aJ9eQEYIyJ1zmG/woHjwDFHu3tZBc0Zn2IdnKeLSEsR8RORGBF5TERONNesBm4QEX8R6Y9zTVBTsfpk7uSvswGwzlyGiMjljtcLcXQ4x1Uwv7KJFgJV6YwxiVjt6W8BR7E6W29xrCsEhgDnAbuBZKwmmNIisA68R7GaflKBl8t4r/nA48B0rANxU2C4C/dlLVZ/yMPnsF9PYTU3pWF1vn5dwSy5WB3Gm4B5QDpWAaoJ/ObY7H5HjmNY/ScznHjd/VhnPt2BaSWW78E6S3gMOIxVhB5GjyseR3RiGqWU8m1auZVSysdpIVBKKR+nhUAppXycFgKllPJxWgiUUsrHedzogjVr1jSNGjWyO4ZSSnmUlStXphhjyhwHyuMKQaNGjUhMTLQ7hlJKeRQR2VXeOm0aUkopH6eFQCmlfJwWAqWU8nFaCJRSysdpIVBKKR+nhUAppXycFgKllPJxbisEIjJJRA6JyLpy1ouI/E9EkkRkjWPib6WUUpXMnWcEHwH9T7N+ANDM8TUGeNeNWZRSSpXDbXcWG2MWi0ij02wyFPjEWDPj/CoikSJS1zEbksvtTs3i1827KfILwvh53A3VVYq/CP3a1CYyNMjuKEopF7DziFgfa2q7E5Idy04pBCIyBuusgQYNGlTozdYmHyPqxzsJJp978u8lneoVeh1lOZrVkjsuaWp3DKWUC9hZCKSMZWXOm2mMGQ+MB0hISKjQ3JqXtq5N7uUjiPj5UZZf8CsZvZ+vyMv4vPzCInq8uID8wiK7oyilXMTOQpAMxJd4Hgfsc9ebhQT6E3LxKGjUibDY5oQFh0BBHgRo88bZ0AKglPex8/LRmcBIx9VD3YA0d/UPnCTuAggOh7ws+KAvLHoRivTgppTyXW47IxCRKUAvoKaIJANPAoEAxpj3gFnAQCAJyAJudVeWcgJCbEtY8CwcWAtXvgvB2m9QEfmFRaRn5wMQUz3Y5jRKqbMl1kU7niMhIcG4bD4CY2D52zDvcYhtBdd/DlGNXPPaXiq/sIhm//cjUaGBBAX4kZ5dQHZ+YfH6Bf/oReOaYTYmVEqVRURWGmMSylrn29dRikD3e6B2a/jyVvj2Hrjle7tTVWkBfsKtFzXiQFoOESGBRFQLICIkkAPpOUz+bTdHMvOqTCEwxiBS1jUJSqmSfLsQnNC0D4z+GcTRZVKYD34BVqFQJxERnhzS5pTli7YcZvJvu13+fvmFRaRl5xd/pZf6bj0uOHmbHOt7Zm4Bjw1sxageTVyeSylvooXghBjHNfHGwPRRVn/BoFchQNu8XSmvoIiU47kcycwjNTOPI5m5pB53PD7+17IT6zNyCk77eiGBftSoFkiNaoFEhARSt0YILeuEE1EtkGkr9rA9JbOS9kwpz6WFoDRjoGZzWPwiHN4M130G4XXsTuUxthzMIOV4LofScziYnsuhDOv7wfQcDmVYB/iyBPgJ0WFBxFQPJiYsiLioUKLDgogOCyIy1HGgdxzsiw/81QIIDvAvN8v3a8q/CK2wyHAsK4+jWfkcy8rjSGYex7LyOZqVR50aIQztWP+cfxdKeQotBKX5+UGf/4M6beGbO2F8L7husnXZqSpXoL/VjPavr9cWL/P3E2KrB1M7Ipj46FASGkVROzyEmuHBRIcFUbN6ENFh1uOIkAC3tOcn7jzC37/4k6NZedZXpnXwT8/Jp7zrJAL8RAuB8ilaCMrTeihEN4Wp18O0G+H+1dpMdBqdG0Xz1g2dqBboT+2IEGpFBBMTFoy/n339LM1qVWf1nmMczykgKiyIqFDrTCMqNJCo0CDru2N5VGgQUWGBfLJ8FxOWbLcts1J20EJwOnXawphFcHSHVQSMgaJC8NdfW2mB/n4Mbl/P7hgn+Xx017M+ywgJ8MMYeHXeFlKO55J6PJfuTWtyc/dG7gmpVBWgR7QzCY22vgAWvww7l8A1H/21TFVZFWlqio0IAeDNn7cSHRpEVl4hu1KztBAor6aF4GxE1IPdy2FCbxg+xbr/QHmVm7o1ZEj7uoSHBOLvJ9zxaSK7UrPsjqWUW+lUlWej0wi4ZRbk58DEvrDxO7sTKTeIDA2ytW9DqcqmheBsxXeGMQuhVkvrbuRje874I0opVZVp01BFRNS1zgx2L4dIx0jahQXaiezFsvIK2Hcsh/1p2ew7ls3B9FwGta9L01gdqFB5Pj1yVVRgCDTtbT3e+D0seA6GfwbROpyBt9l0IIPWT8w5ZXladj6PD9Z+IuX5tBC4QnB1SN8L43tbVxSdKBDK493UrRGx4cHUrVGN+pHVqBdZjXqRIQx4fQlFxlBYZLQ/QXk83x6G2pWObIcpN0DKZuj3LHS7Uwet82Ltxs0hN7+IgqIiXruu41ndiZyRk0/y0WySj2bTJDZMm5dUpdBhqCtDdBMYNQ++GQtz/gW1WumZgRcbe0lTth/OZPofyew5cvLlpZm5BY4DfVbx9z1Hskk+Zj0/lpVfvG3nRlF8ObZ7ZcdX6iRaCFwpOByu/RS2zoEmvaxlRYXgV/7AaMoz3d37PPILi5j+RzI/bzrExv0Z7HEc+EsPrBcc4EdcVDXio0PpGB9JXFQo8VGhTFiynYycAjYdSCf5yF+F40B6Drdd3JjzG0TZtHfK12ghcDU/P2gxwHp8cD1MuwmGvW9ddqq8ir8IseHBrElO40hmHvHRobSpV4P46GrERYVaB/+oUGpWDyrzLudvViXz08ZD9H99SfGykEA/cvKLqB9VTQuBqjTaR+BOhzbClOGQvg8GvwadbrQ7kXKxvIIi/AQC/M/+lpw1ycdYti2VuKi/CkdMWBAtH59Ns9rVaVYrnLt7N+W8WuFuSK58jfYR2KVWKxi9AL66Fb69Gw6shX7PgH+g3cmUiwQFVPyezPZxkbSPizxleVxUNTYfyGDd3nRa140oLgT5hUXsO5bN7iNZ7ErNwk+EG7o2qPD7K3WCFgJ3C42GEdNh3hPw69sQ2RAuvMvuVKoKm/vgJWTmFdB+3Fxm/rmPhVsOsftIFvuO5VBYdPIZfN/WtagVHmJTUuUttBBUBv8A6P8cNO5pzY8M2omsyuXvJ1QL9Ccuqhr7jmUT4B/K+Q2iuLJjKPHRoTSMDmXFziO8PHcLz8/aRL3IEB6+vKXdsZUH00JQmVr0t75npsLHQ6DXI9YEOEqVEujvx5J/9i53KO2s/EJE4Ls/91FQZHigb3MCK9BPoRTooHP2KCqAoFD4YiT8/CwUFdmdSFVBp5tPoXeLWmz6T38e6NusEhMpb6WFwA7hteGWH6DjjbD4RZg2AnLS7U6lPExwgDYtKtfQQmCXgGAY+hYMeBG2zIE5j9mdSHmwvq8u4tYPf7c7hvJQ2kdgJxHoegfUbgOxjs6+oiLrpjSlnHBBw2g6xEeSkpHLip1H+WP3UWKrBxMfHWp3NOVB9IhTFTS6GMJqQmE+fHolLHsTPOxGP2WPC5vG8O3dF9G/bR2O5xZw1TvLGPvZyuL1nnbDqLKHnhFUJYX5EFID5v7buvlsyBsQWM3uVMoD3HxhI+pHVuO7NfvYdug4909dRdKh4+xIyeTWixrp5aXqtPSMoCoJCoVrP4He/4Y10+DDAZC21+5UygM0iAnltosb06VRNOk5BSTuPEpM9WD8/YRthzLtjqeqOD0jqGpE4JKHoXZr+HoMfHkz3D5P5zZQTnl0QEsevKw5IYHWFUWXv7b4pPXGmNNelqp8kxaCqqrlIBg137rnQEQ7kZVTRKS4CJyw6UA693z+B1sPHmfXkUyeG9aOq86Psymhqoq0EFRltUq06/74MCDQ/3kdtE45rVZEMEu2plBoDM1rhbP5YAY7U7SpSJ1MC4EnMMbqNF72JhzeZM2LHFbT7lTKA0y6pTP5hUWEBln/1Rv/64fidcdzCziYnkPjmDD8dN5ln6aFwBOIWMNX124HM++F8b3h+s+hTju7k6kqLtDf75QxiKb/sZevV+0l+Wg2AJ/e3oUezWLtiKeqCG109iQdroPbfrT6DT4dBnlZZ/4ZpUro3DCa6sEBdGoQxcgLGwLw1HcbGPjGEg6m59icTtlFzwg8Tf0LYMxCOLTButwUrKYjvRJEOeGLsRcWP07LzmfxlsPkFhSSdOg4r/+0hdoRIdzbpxn+2lTkU7QQeKLw2tYXQOIka6yiq8ZbN6Mp5aQa1QJZ+HBv1u1NY/CbS5ny+x4AmsRWJ07nTPYp2jTk6YyBpJ9gYl9ISbI7jfJAbepFMOeBnjw3zOpzum/KKq5+dxkZOfk2J1OVRc8IPF3n26Fmc+vGswl94OpJ0Kyv3amUBxERWtQJp35UNXLyC1m3L42v/9hLfqGOU+Qr3HpGICL9RWSziCSJyKNlrK8hIt+JyJ8isl5EbnVnHq/VuAeMXgCRDeDzayF1m92JlAeqHhzAbRc3pkNcJAAfLN3OPZ//QZ9XFnLHp4n2hlNu5bZCICL+wNvAAKA1cL2ItC612d3ABmNMB6AX8IqIBLkrk1eLagi3z7HOCGKaWst05ElVAdWDrYaCtxdsY/WeY2TmFrBw82FGTvqdN+dvtTmdcgd3Ng11AZKMMdsBRGQqMBTYUGIbA4SLNfhJdeAIUODGTN4tKAzaXGk93vkL/PSkdfNZDR1OQDnvio71aFEnnAYxoUSEBPL+om28MncLv+9IZf3eNHYfyaJn81iGdKhnd1TlIu5sGqoP7CnxPNmxrKS3gFbAPmAtcL8xRifwdYX8LDi0Ccb3gl3L7U6jPEigvx9t69cgIsQayuSOS5qy6T/9GdapPqmZeXy5MplX523hme838PuOIzanVa7gzkJQ1oXIpdsqLgdWA/WAjsBbIhJxyguJjBGRRBFJPHz4sKtzeqdml8Ho+RAcAR8PsS4zVaqC/PyEp4e2JfHffendIpYdKZlMXLqDyb/tKnP71OO5LN2awvuLtvHcrI3kFhRWcmJ1NtzZNJQMxJd4Hof1yb+kW4EXjDWNUpKI7ABaAidNvmqMGQ+MB0hISNCGb2fFtoDRP8P02+H7ByG8HrTob3cq5aEC/f2oWT2Y16/rREpmLrd9tILCIsPOlEw27E9nw7704u8HSt2lPKBtHTrpfQlVljsLwQqgmYg0BvYCw4EbSm2zG7gUWCIitYEWwHY3ZvI91SLhhi9gzRfQrJ+1TO9EVuegRmggNUID8RPh+zX7+X7NfgD8/YTzYqvTvWkMretF0LpuBIeP53L/1NWnNAWoqsVthcAYUyAi9wBzAH9gkjFmvYiMdax/D/gP8JGIrMVqSnrEGJPirkw+y88fOl5vPT62G6bdCEP+B/U62hpLebbRPZqw6UA6bepF0LpuDZrVrn7KXAgLNx+yKZ06G269ocwYMwuYVWrZeyUe7wP6uTODKiUnHTJTYVJ/GPoWtLva7kTKQ93QtYHdEZSL6BATvqZOW2vQunqdrL6Dn8ZBkXbkKeXLtBD4ouqxMPJbSLgNlr4Gv7xhdyKllI10rCFfFRAEg1+DBt2hxQBrmXYiK+WT9IzA17W/BoKrQ14mfDTIGtJaKTc4lpXHkq2HWbxF7wWqavSMQFlyMyDvOHx+HVz6OFz8kJ4dKJcZ/XEiqZl5APgJPHRZc2pWD2Z4F+1wrgq0EChLeB24dbY1J/L8p+HAOuuqoqAwu5MpD9a8djgd4yOpH1mNtvVrcCAtm4+X7+LluVsI8BMWbD5Ei9rhPNSvhd1RfZoYDxuhMiEhwSQm6pC4bmOM1Xn80zhof60185lSLpKTX8j6feksS0rhlXlbipfHhAXRp2UtXrqmg43pvJuIrDTGJJS1Ts8I1MlE4OIHoHYbqNnMWqadyMpFQgL9uaBhFBc0jGJ0zybM/HMfH/6yk8MZOWw5mGF3PJ+lncWqbM0ug6hGVhGYfjusmKjzGyiXCgn059qEeH68vwdt6+t823bSMwJ1egU5kHscfvg77F8DA1+2Lj1VysU2H8yg1eOzuTYhjsEd6rEmOY3NB9K5sVtD2jtmTVPuoX0E6syKCmHBs7DkFYjvBtd9CtVr2Z1KeZH3F21j5p/7WL8vvcz10WFBvHxNe/q0rF3JybyH9hGoc+PnD5c+YfUbzLgbPvsb3LFY+w2Uy9xxSVPuuKQpP204yNq9abSPq0G7+jX494x17EvLZt3edLYdyqRPS7uTeictBMp5bf8GMc2sm89EtBNZuVzf1rXp2/qvT/3jRyZwPLeAtk/qjY7upIVAnZ267f96vOi/1k1ofZ+yzhqUUh5JrxpSFWMMZKXCsjdh8jWQfdTuREqpCtJCoCpGBAa+BEPegB2LYUIfOLTJ7lRKqQrQQqDOzQW3wC3fW5eYftjfmvhGKeVRtI9AnbsG3azJbpJXQEiE3WmUUmdJzwiUa9SoD22utB5vmAlf3GydJSilqjwtBMr1MvbDxpkw6XI4utPuNEqpM9BCoFyv6x0w4ktI2wPje1udyUqdowPpOczfeJDcAp1j29W0ECj3OK8vjF4AYbHwyZVwaKPdiZSHOnHL4gdLd3D7x4nM33jI1jzeSDuLlfvENIVRP8G66VCrld1plIcKCw7g2WFtScnI47WftpCTr2cErqZnBMq9QiIg4Vbr8YF18MlQyDhgbyblcUZ0bciVnerZHcNraSFQlSctGfb8DuN7QfJKu9MopRy0EKjK06I/3D4X/APhwwGweordiZQH2p+Wwy9JKRQVedYQ+lWZ9hGoylWnHYxeCF/eDDPGQlAYtL7C7lTKA4ij2/ilOZsBCA3yJzosiL/3a06gvx+D22vTUUVpIVCVLywGbvoGVnwALQbYnUZ5iPjoavyjX3My8wp5d+E2svIKycrL5sFpfwLQtXEMseHBNqf0TFoIlD38A6HbWOtxZgp8dRsM+K9eXaTKJSLc06cZAP+8vAW5BUXMWLWXjfvT+Xj5Lgq1qajCtI9A2S99HxzeBBP7wsbv7U6jPICIEBLoz/AuDWhZV8e3OldaCJT96ra3Bq2r2RymjYCF/4WiIrtTKQ/z0pzNjJu5ntV7jrHpgI6Ceza0aUhVDRH14NYf4fsHYOFz1nwHl/zT7lTKA9SoFgjA9D+SAfho2U5qRwTz22N97YzlUbQQqKojMASufBfiu0IrvZJIOad/mzosergXhUWGb1btZeWuo6zbm2Z3LI+iTUOqahGx7kQOi4HCfJg6ArYvtDuVqsL8/ISGMWE0ia3O3/u1oHntcLsjeRwtBKrqykqF1G3w6VXw67vWPMlKKZfTQqCqrvA6MGqeda/B7Efh27shP8fuVEp5HS0EqmoLDodrP4VLHoXVk2HGnXYnUsrraGexqvr8/KD3v6B2G4huYnca5SGO5xYQ4Gfdb6BOT88IlOdofQXUaWs9/vFRWDXZ3jyqykrPKaD9uDnc9MFvdkfxCHpGoDxPfg4c2gC/vQsH1kK/Z8Bf/5SVpVeLWHalZrIjJZPU43l2x/EIbj0jEJH+IrJZRJJE5NFytuklIqtFZL2ILHJnHuUlAkPgxq+h211WMfjsKsg6YncqVUX0alGLD2/tQru4SLujeAy3FQIR8QfeBgYArYHrRaR1qW0igXeAK4wxbYBr3JVHeRn/AOj/PAx9B3Yvh4+HQJFOYahURbjzfLoLkGSM2Q4gIlOBocCGEtvcAHxtjNkNYIzRWanV2ek0AmJbWCOY+mmnoDrZsex8Hp+xji6NoxnSQecrKI87m4bqA3tKPE92LCupORAlIgtFZKWIjHRjHuWt4hKs2c8AVkyEBc/poHWK6sEBHMnM49Nfd/HJ8p12x6nS3FkIpIxlpW8NDQAuAAYBlwOPi0jzU15IZIyIJIpI4uHDh12fVHmPA2th0X9h2o2Qm2F3GmWjfw1syewHetC1cbTdUao8dxaCZCC+xPM4YF8Z28w2xmQaY1KAxUCH0i9kjBlvjEkwxiTExsa6LbDyAoNfh/7/hS2zYeJl1hAVyidFhATSsk4E/n5lfSZVJbmzEKwAmolIYxEJAoYDM0tt8y3QQ0QCRCQU6ApsdGMm5e1ErJnPbvoajh+AiZfqFUVKnYHbOouNMQUicg8wB/AHJhlj1ovIWMf694wxG0VkNrAGKAImGmPWuSuT8iFNesHoBbBjEYRq04BSpyPGw0Z0TEhIMImJiXbHUJ5mxxJYMxUGvmLdh6B8xg0TfiX5aDaXta5N/chqhAX7071pTeKjQ+2OVqlEZKUxJqGsdU6dEYjIRcA4oKHjZwQwxhgd+EV5hgNrYNVncHADDJ9szYimfEJokD+7j2TxwdIdxctu6d6IcVe0sTFV1eJs09AHwIPASkDv2lGe58K7IbIhfHMHjO8F130G8V3sTqUqwXPD2nFPnxyy8wrZn5bNuJnryS/Uy4tLcrazOM0Y86Mx5pAxJvXEl1uTKeVqrQbDqJ8gMBQ+GgT7/7Q7kaoEtSJC6BgfyYVNY7jq/DiCAnSszdKcPSNYICIvAV8DuScWGmP+cEsqpdylVisY/TOs/AjqtLc7jVJVgrOFoKvje8mOBgP0cW0cpSpBaDT0eMh6fHQnzH3cuv8gLMbOVKoSJR06znOzNtK8djh5BUUMbFeHyNAgu2PZxqlCYIzp7e4gStni4AbYMgcm9ILhn0OddnYnUm4WHODPbzuO8NuOv+4vMRhGdG1oYyp7OXX5qIjUAJ4EejoWLQKeNsakuTFbmfTyUeVye1fC1BGQkwZXvgNthtmdSLnRpgPpZOYWcig9h/ScfB6ZvpYezWpSKzyER/q3oFaEd15efLrLR53tNZkEZADXOr7SgQ9dE08pm9W/AMYshNpt4ctbYN3XdidSbtSyTgQXNIxiQLu6DGpfj0B/YcnWFKb/kcyKnUftjmcLZ/sImhpj/lbi+VMistoNeZSyR3gduOV7WPo6NL/c7jSqklQPDuCnhy5hf1oOw8f/ancc2zh7RpAtIhefeOK4wSzbPZGUsklAMPR6BILCrJFLvxipg9b5gIYxYUSH+W5HMThfCO4E3haRnSKyC3gLGOu+WErZ7MgOa1iKCb0h6Se70yjlVk4VAmPMamNMB6A90M4Y08kYo3fjKO9Vt73Vb1AjHiZfA7/8DzxsXC6lnHXaPgIRudEY85mIPFRqOQDGmFfdmE0pe0U1hNvnwow7Yd7jkJ8FvR61O5VSLnemzuIwx/dwdwdRqkoKCoNrPoZlb0KbK+1Oo5RbnLYQGGPed3x/qnLiKFUFicBF91mPi4rg+weg4w3QoJutsZRyFaf6CETkRRGJEJFAEZkvIikicqO7wylV5WSlwM4l8NFga7wipbyAs1cN9TPGpAODseYZbg487LZUSlVV1WtZg9Y17gnf3Q8//B0K8+1OpdQ5cbYQBDq+DwSmGGN0Eljlu6pFwYgvoft9sGIiTB9ldyKlzomzdxZ/JyKbsG4iu0tEYoEc98VSqorz84d+/7EGqasRZ3capc6Js6OPPioi/wXSjTGFIpIJDHVvNKU8QPtr/3q8+CWIbgJt/1b+9kpVQadtGhKRPo7vVwG9gaGOx/2B7u6Pp5SHKMyHpPnw1W3w01NQpDO6eqL/zt7Ehc/PJ/lolt1RKtWZzgguAX4GhpSxzmDNWKaU8g+EkTPhx4dh6atwcB38bSKE1LA7mXJCrfBg6kdWA2B/Wg7JR7OJiwq1OVXlOdN9BE86vt9aOXGU8mABQTDkDWsKzB//CZMGwB2Lwd/Zrjhll8jQIH55tA/LtqVww4Tf7I5T6Zy9j+A5EYks8TxKRJ5xWyqlPFnn262zg+73ahHwcBk5+RxI8/7rYpz9Kx1gjHnsxBNjzFERGQj82z2xlPJwjS4CLrIer58BR7bDxQ9adymrKu/9RdsYN3M9mw5kANC9aQyx4cG8MbyTzcncw9n7CPxFJPjEExGpBgSfZnul1AlJ82D+U1ZHcp5vdUJ6mhPzEqzYeZTY8GBa1Y1wPD/Cd3/uszOaWzl7RvAZMF9EPsTqJL4N+NhtqZTyJle8BdFNYf7TkJoEwz+HyHi7U6kytKwTwYr/60t0WBD+ftbZW25BIW/9nMTbC5JsTuc+zs5H8CLwDNAKaAP8x7FMKXUmItDjIbjhCzi6E8b3goyDdqdS5YgNDy4uAgDBAf42pqkcZ9OTtREoMMb8JCKhIhJujMlwVzClvE7zftY4RRtnQnhtu9Oos1RkrPsM6kVW46ZuDe2O41LOXjU0GvgKeN+xqD4ww02ZlPJeNZtBj79bj/evgVn/hII8ezOpMwoLtj4zv7twG/+bv9XmNK7nbGfx3ViXQKQDGGO2ArXcFUopn7BjEfz+PnxyBRw/ZHcadRq3dG/EzHsu4qpO9e2O4hbOFoJcY0zxxxYRCcDqNFZKVVT3e+FvH8C+1TC+t/VdVUkhgf60j4skONDqL8jIySe3wHuGEXG2ECwSkceAaiJyGfAl8J37YinlI9pdDbfPsTqUJ10OySvtTqTO4HBGLu3GzaXduLmMX7yNFTs9f1R+ZwvBI8BhYC1wBzALvZlMKdeo2wFGL4Auo6Fue7vTqNPo26oWfVtZreJ5BUU8N2uTV/QZiDGnb+ERET9gjTGmbeVEOr2EhASTmJhodwyl3Of4YZg/Dvo9Y02Co6qcvcey2XIgg5fnbqZaoD+PDmhJk9jqxTekVUUistIYk1DWujOeERhjioA/RaSBy5MppU6VvAL+nAYTLoXDm+1Oo8pQP7IavVvWIjTIn8RdR7n6veW8OHuT3bEqzNmmobrAesfE9TNPfLkzmFI+q+VAuPk7yE23isHmH+1OpMpxV6/zuO/SZtSsHkRWnud2Hjt7Q9lTbk2hlDpZwwthzEKYegNMud6a26Dd1XanUqX0blmL3i1refw4RKctBCISAowFzsPqKP7AGFNQGcGU8nk14uDW2bDgWWjax+40youdqWnoYyABqwgMAF5xeyKl1F+CQuHyZyE02roD+bsH4Oguu1OpMhxIy+GT5TvZc8QaYdYYw9HMPM50QU5VcKamodbGmHYAIvIB8Lv7IymlypSyBdZ9bY1VdO0n0OhiuxMph0B/4fedR/h95xFgPb1axPLr9lRy8osAayrMz0Z1pXntcHuDluNMZwT5Jx5UpElIRPqLyGYRSRKRR0+zXWcRKRQRbQRVqjx12lqD1oXGwCdD4fcJ4AGfNn3BK9d0ZMLIBMKCrDuPD6TlcHmbOgA0iQ3jUEYuL8/ZzCNfrSG7CnYqn/Y+AhEpBDJPPAWqAVmOx8YYE3Gan/UHtgCXAcnACuB6Y8yGMrabB+QAk4wxX50usN5HoHxeTjp8PRq2zIZLHoHej535Z1SlyMjJx0+keJA6gN2pWfR8acFJ29WtEcLcB3sSHhJYadlOdx/BmSavP5eBuLsAScaY7Y4QU4GhwIZS290LTAc6n8N7KeU7QiJg+BRY9F9oPdTuNKqEsg7sDWJC+eXRPmTmFnDv56vIKyxiR0omRzPzK7UQnI6z9xFURH1gT4nnyY5lxUSkPjAMeM+NOZTyPn5+0PtfULu11Tw0/2nYq+MUVVX1I6vRvHY4cx7syT29z7M7zincWQjKmqW7dDvU68AjxpjTNpqJyBgRSRSRxMOHD7sqn1LeIesIrPkSJg2w7khWHuH6Cb/S99VFFBQW2R3FrYUgGSg5MWscUPquiwRgqojsBK4G3hGRK0u/kDFmvDEmwRiTEBsb66a4SnmosBgYswDiu8A3Y2Duv6Go6nVIKkuLOuG0qB1OoL+QdOg4uQXeXQhWAM1EpLGIBAHDgZOGpTDGNDbGNDLGNMKaAe0uY8wMN2ZSyjuF1YSbvoEuY2DZm/DVrXYnUuVoW78Gcx7syQ1dreHbrn1/OaM+XkFeQZFtZwdnM2fxWTHGFIjIPcAcwB/riqD1IjLWsV77BZRyJf9AGPgS1GlnXWKqqrRmtcKpFR7MwfRc1u9Lp82Ts2kfF8n0O7tXepYzDkNd1ejlo0qdhcQPoXotaDnI7iSqHMuSUpi4dAc7UzJJOZ7LdZ3j6dQgioHt6rr0fc5pGGqllIcqKoTVn1sD1y16EYrsb4tWp+p+Xk0m3dKZns1jSc8pYMKSHby/aFulZtBCoJS38vOHm2dC++usgeu+vBlyj9udSpXjn/1bMO/BnvRoVrPS31sLgVLeLLAaDHsf+j0Lm76HSf2hINfuVKoMoUEBNKsdjr9fWVfeu5fbOouVUlWECHS/B2q1gsObICDY7kSqitFCoJSvOO9S6wtg+yKrKHQZYxUKVaXkFxrWJqexYX8amw8cZ1D7OlzQMNpt76eFQClftOYLWP0Z7F8Dg1/Vs4QqRIAN+9MZ8tbS4mUZOflaCJRSLnbFm1CjvjVwXcpmuO4zCK9jdyoFjO7ZhLb1a9CqbgSt60Zw/YRf3f6eWgiU8kV+ftbw1bXbwDd3wvheMGq+VRyUrbo3rUn3pn9dOVQZDXdaCJTyZa2HQsx51v0GEfXsTqNsopePKuXrarex5kUWgSM7rCGtC896QkLlwbQQKKX+sul7WPIKTP6bNby18glaCJRSf+l+Lwx9G3Ytgwm94WDpCQWVN9JCoJQ6Wacb4ZZZkJ8DE/vCnhV2J1JupoVAKXWq+M4wZiG0vcrqQ1BeTQuBUqpsEXVh6FsQFAq5GTD7Meu78jpaCJRSZ7bzF/jtPZh4GRzZbnca5WJaCJRSZ9aiP9w4HTL2w/jesG2B3YmUC2khUEo5p2lvGLMAwuvCZ1fB2q/sTqRcRAuBUsp50U1g1DzoOALiu9qdRrmIFgKl1NkJDrc6kSPjrekvfxoH6fvsTqXOgRYCpVTFpWyG3ydY/QZ6v4FbFRUZcvIL3fLaWgiUUhVXqxXcPg8CQ+CjgbDqM7sTeaXZ6w7Q5sk5vLvQPZPaayFQSp2b2q1h9AJo2B2+vRsWvmB3Iq8yqH1dOjaI5PouDejS2D2T0+gw1EqpcxcaDSOmw09PQpNedqfxKv83qLXb30PPCJRSruEfYA1n3aCb9Xz5O3Bgnb2ZlFO0ECilXC8nDZa9CR9cBhu+tTuNOgMtBEop1wupYd18VrsNfDESfn7WutRUVUlaCJRS7hFeB275wRrWevGL8NWtYIzdqVQZtLNYKeU+AcFwxVtQpwMEVrOmw1RVjhYCpZR7iUDXMX893/AtBFWH8y61L5M6iTYNKaUqT1GR1Yk8+WpY9pY2FVURWgiUUpXHzw9umgEtB8Pc/4Nv7oD8bLtT+TwtBEqpyhVcHa79BHr/G9ZMgw8HQF6W3al8mvYRKKUqnwhc8rB1eWnyCms6TGUbLQRKKfu0HGh9AexbBQfWwvkj7c3kg7RpSClVNfw+EWbeCz/8Awrz7U7jU/SMQClVNQx5A0KjrKuKDm+Caz6CsJp2p/IJekaglKoa/AOg3zMwbDzs+d2a7ObYbrtT+QQ9I1BKVS0droOazeD38RBez+40PkHPCJRSVU/982HYe9ZZQsZBWPKqDlrnRm4tBCLSX0Q2i0iSiDxaxvoRIrLG8bVMRDq4M49SygOt/QLmPwVTr7eGt1Yu57ZCICL+wNvAAKA1cL2IlJ5qZwdwiTGmPfAfYLy78iilPNSF98DAlyHpJ5jYF1KS7E7kddx5RtAFSDLGbDfG5AFTgaElNzDGLDPGHHU8/RWIc2MepZQnEoEuo62hKbJSYUIfqzNZuYw7C0F9YE+J58mOZeW5HfjRjXmUUp6scQ8YvcD6HnOe3Wm8ijsLQVkDj5c51KCI9MYqBI+Us36MiCSKSOLhw4ddGFEp5VGiGsLwyRAaDQW5sOhFHbTOBdxZCJKB+BLP44B9pTcSkfbARGCoMSa1rBcyxow3xiQYYxJiY2PdElYp5WG2L4IFz8Gky+HYnjNvr8rlzkKwAmgmIo1FJAgYDswsuYGINAC+Bm4yxmxxYxallLdp3g+unwqp22FCb9i1zO5EHstthcAYUwDcA8wBNgJfGGPWi8hYERnr2OwJIAZ4R0RWi0iiu/IopbxQi/4w+mcIqQEfD4G1X9mdyCOJ8bAZghISEkxiotYLpVQJ2cfg+wfgkkehVku701RJIrLSGJNQ1jq9s1gp5fmqRVqD1NVqaU1/uewtOK4XljhLC4FSyrsc2Q4/PwPje8G+1Xan8QhaCJRS3iWmKdw223o8qb/2GzhBC4FSyvvU6whjFlrfp98OC/9rc6CqTQuBUso7VY+FkTMh4Xao297uNFWazkeglPJeAUEw+NW/nv85FeqdD7HN7ctUBekZgVLKN+Rlwk/jYOKlsHm23WmqFC0ESinfEBQGt8+D6MYwZTgsecW61FRpIVBK+ZDIeLh1NrT9G8x/GqaP0mKA9hEopXxNUCj8bSLUaQcYa74DH6eFQCnle0Tg4gf+er59Efj5Q6OLbYtkJ20aUkr5NmNgwbPwyVBYMdEnm4q0ECilfJsIjPgSml4KP/zdGryuIM/uVJVKC4FSSoXUgOunwMUPwcqPrCGtczPsTlVptI9AKaXA6iPo+yTUaQvbfoag6nYnqjR6RqCUUiW1/RsMfdtqMkrd5hOD1mkhUEqp8vzyhjVo3dx/Q1Gh3WncRpuGlFKqPINeAf8gWPYmHNwAV38A1aLsTuVyekaglFLl8Q+EQS/DkDdgx2KY0AeO7rI7lctpIVBKqTO54Ba45XuIaQbVa9mdxuW0ECillDMadIMRX0BgNchJh98neM3NZ1oIlFLqbK2eDLP+AV/eYg1v7eG0s1gppc5W17FQmA8/PQmpSTB8MkQ1sjtVhekZgVJKnS0RuOg+a2iKtD0wvjfs+d3uVBWmhUAppSrqvL4wegHUbgMR9e1OU2FaCJRS6lzENLWuKKpR37rp7Lf3oSDX7lRnRQuBUkq5yo7F8OM/4aPBkHHA7jRO00KglFKu0rQ3XPMxHFxn9RvsXWl3IqdoIVBKKVdqcyXcPhf8A2DSAFj3td2JzkgLgVJKuVqddjB6ITTuATXi7E5zRloIlFLKHcJi4MbpEN/Fer5qMmQdsTdTObQQKKWUux3bbU2BOaE3HNpod5pTaCFQSil3i2wAt/wA+dkwsS9s/N7uRCfRQqCUUpUhvguMWQg1m8O0EbD4ZbsTFfOKsYby8/NJTk4mJyfH7ihKnSQkJIS4uDgCAwPtjqKqgoh6cOuP8N39EF7X7jTFvKIQJCcnEx4eTqNGjRARu+MoBYAxhtTUVJKTk2ncuLHdcVRVERgCw96zxisC2DwbYltAtH1/I17RNJSTk0NMTIwWAVWliAgxMTF6pqpOdeJYlZ8D3z9odSJvX2hbHK8oBIAWAVUl6d+lOq3AELj1B6heBz69Cn5915bJbrymENhNRLjpppuKnxcUFBAbG8vgwYPP6nUaNWpESkrKOW9TnilTpvDss8+etGz16tXMmjWrQq8HMGPGDDZs2OD09vfffz/169enqKioeNm4ceN4+eWTO89K7ueBAwcYPnw4TZs2pXXr1gwcOJAtW7ZUODNAbm4u1113Heeddx5du3Zl586dZW43bdo02rdvT5s2bfjnP/950rovvviC1q1b06ZNG2644YZzyqN8VHQTGDUPWgyA2Y/Ct3dDif8blUELgYuEhYWxbt06srOzAZg3bx7161e9YWlnz55N//79T1pWmYWgqKiIb775hvj4eBYvXuzUzxhjGDZsGL169WLbtm1s2LCB5557joMHD1Y4M8AHH3xAVFQUSUlJPPjggzzyyCOnbJOamsrDDz/M/PnzWb9+PQcPHmT+/PkAbN26leeff55ffvmF9evX8/rrr59THuXDgsPh2k/hkkehWhT4Ve6hWQuBCw0YMIAffvgBsD55X3/99cXrjhw5wpVXXkn79u3p1q0ba9asAawDTb9+/ejUqRN33HEHpsRp4WeffUaXLl3o2LEjd9xxB4WFheW+9xdffMFDDz0EwBtvvEGTJk0A2LZtGxdffDFgHVBXr17N+eefX/xzeXl5PPHEE0ybNo2OHTsybdo0MjMzue222+jcuTOdOnXi22+/BeC+++7j6aefBmDOnDn07NmTZcuWMXPmTB5++GE6duzItm3bTvs7WrBgAW3btuXOO+9kypQpTv1eFyxYQGBgIGPHji1e1rFjR3r06OHUz5fn22+/5eabbwbg6quvZv78+Sf9/gG2b99O8+bNiY2NBaBv375Mnz4dgAkTJnD33XcTFRUFQK1a3jepuapEfn7Q+1/Q7xnr+b5VsGdFpby1W68aEpH+wBuAPzDRGPNCqfXiWD8QyAJuMcb8cS7v+dR369mwL/1cXuIUretF8OSQNmfcbvjw4Tz99NMMHjyYNWvWcNttt7FkyRIAnnzySTp16sSMGTP4+eefGTlyJKtXr+app57i4osv5oknnuCHH35g/PjxAGzcuJFp06bxyy+/EBgYyF133cXkyZMZOXJkme/ds2dPXnrpJQCWLFlCTEwMe/fuZenSpcUHzFWrVtGhQ4eT2q2DgoJ4+umnSUxM5K233gLgscceo0+fPkyaNIljx47RpUsX+vbtywsvvEDnzp3p0aMH9913H7NmzaJp06ZcccUVDB48mKuvvhqA9957D+CkA/cJJwrk0KFDeeyxx8jPzz/jpZXr1q3jggsuOOPvH6BHjx5kZGScsvzll1+mb9++Jy3bu3cv8fHxAAQEBFCjRg1SU1OpWbNm8TbnnXcemzZtYufOncTFxTFjxgzy8vIAipumLrroIgoLCxk3btwpZ1tKnbUT/z/n/B8kr4DBr0OnEW59S7cVAhHxB94GLgOSgRUiMtMYU7INYQDQzPHVFXjX8d0jtW/fnp07dzJlyhQGDhx40rqlS5cWf5Ls06cPqamppKWlsXjxYr7+2hqdcNCgQcWfLufPn8/KlSvp3LkzANnZ2af9xFmnTh2OHz9ORkYGe/bs4YYbbmDx4sUsWbKEq666CrCahQYMGHDG/Zg7dy4zZ84sbrPPyclh9+7dtGrVigkTJtCzZ09ee+01mjZtWubPl1UAwDr7mDVrFq+99hrh4eF07dqVuXPnMmjQoHI7Vc+2s/VE4XVG6U//Zb1fVFQU7777Ltdddx1+fn50796d7du3A1Y/0NatW1m4cCHJycn06NGDdevWERkZeVaZlSrTdZ/BlzfDt3fBgbXWmYK/ew7Z7jwj6AIkGWO2A4jIVGAoULIQDAU+Mdb/yF9FJFJE6hpj9lf0TZ355O5OV1xxBf/4xz9YuHAhqampxctPd9Ap62BnjOHmm2/m+eefd/q9L7zwQj788ENatGhBjx49mDRpEsuXL+eVV14BrAP8iWJ0OsYYpk+fTosWLU5Zt3btWmJiYti3b5/TuU6YPXs2aWlptGvXDoCsrCxCQ0MZNGgQMTEx7N9/8j97RkYGkZGRtGnThq+++sqp9zibM4K4uDj27NlDXFwcBQUFpKWlER0dfcrPDhkyhCFDhgAwfvx4/P39i3++W7duBAYG0rhxY1q0aMHWrVuLi7dS5yQ0Gm78BuY9Dr++A4c2WMUhJMLlb+XOPoL6wJ4Sz5Mdy852G0RkjIgkikji4cOHXR7UlW677TaeeOKJ4oPdCT179mTy5MkALFy4kJo1axIREXHS8h9//JGjR48CcOmll/LVV19x6NAhwOpj2LVr12nfu2fPnrz88sv07NmTTp06sWDBAoKDg6lRowZpaWkUFBQQExNzys+Fh4efdPC8/PLLefPNN4uL16pVqwDYtWsXr7zyCqtWreLHH3/kt99+K/PnyzNlyhQmTpzIzp072blzJzt27GDu3LlkZWXRs2dPZs6cWfw6X3/9NR06dMDf358+ffqQm5vLhAkTil9rxYoVLFq06JT3WLJkCatXrz7lq3QRAKtof/zxxwB89dVX9OnTp8yifOLf4OjRo7zzzjuMGjUKgCuvvJIFCxYAkJKSwpYtW4r7ZpRyCf8A6P88DH0HgsIgMNQ972OMccsXcA1Wv8CJ5zcBb5ba5gfg4hLP5wMXnO51L7jgAlPahg0bTllW2cLCwk5ZtmDBAjNo0CBjjDGpqanmiiuuMO3atTNdu3Y1f/75pzHGmJSUFHPZZZeZTp06mQceeMA0aNDAHD582BhjzNSpU02HDh1Mu3btzPnnn2+WL19ujDGmYcOGxduUlJSUZACzefNmY4wxl112mbn33nuNMcZ8+eWX5sknnywze2pqqklISDAdOnQwU6dONVlZWWbMmDGmbdu2pk2bNmbQoEGmqKjIXHrppebbb781xhiTmJho2rZta7Kzs83SpUtNq1atTMeOHU1SUpJ59913zbvvvnvSe2RmZpqoqCiTlpZ20vJhw4aZqVOnGmOMee+990z79u1Nhw4dzGWXXWa2bdtWvN3evXvNNddcY5o0aWJat25tBg4caLZs2XKaf5Ezy87ONldffbVp2rSp6dy580nv16FDh+LHw4cPN61atTKtWrUyU6ZMKV5eVFRkHnzwQdOqVSvTtm3bk9aVVBX+PpUXKCo6px8HEk05x1Uxbrp5QUQuBMYZYy53PP+Xo/A8X2Kb94GFxpgpjuebgV7mNE1DCQkJJjEx8aRlGzdupFWrVq7fCS8yatQoRo0aRbdu3eyO4nP071NVBSKy0hiTUNY6d/YRrACaiUhjYC8wHCh9x81M4B5H/0FXIO10RUBV3MSJE+2OoJSqotxWCIwxBSJyDzAH6/LRScaY9SIy1rH+PWAW1qWjSViXj97qrjxKKaXK5tb7CIwxs7AO9iWXvVfisQHudmcGpZRSp+c1dxa7q69DqXOhf5fKE3hFIQgJCSE1NVX/06kqxTjmIwgJCbE7ilKn5RUT08TFxZGcnExVv8dA+Z4TM5QpVZV5RSE4cWenUkqps+cVTUNKKaUqTguBUkr5OC0ESinl49w2xIS7iMhh4PSjr5WvJlCxOR49l+6zb9B99g3nss8NjTGxZa3wuEJwLkQksbyxNryV7rNv0H32De7aZ20aUkopH6eFQCmlfJyvFYLxdgewge6zb9B99g1u2Wef6iNQSil1Kl87I1BKKVWKVxYCEekvIptFJElEHi1jvYjI/xzr14jI+XbkdCUn9nmEY1/XiMgyEelgR05XOtM+l9ius4gUisjVlZnPHZzZZxHpJSKrRWS9iJw6sbOHceJvu4aIfCcifzr22aPnNRGRSSJySETWlbPe9cev8uaw9NQvrElwtgFNgCDgT6B1qW0GAj8CAnQDfrM7dyXsc3cgyvF4gC/sc4ntfsaaF+Nqu3NXwr9zJLABaOB4Xsvu3JWwz48B/3U8jgWOAEF2Zz+Hfe4JnA+sK2e9y49f3nhG0AVIMsZsN8bkAVOBoaW2GQp8Yiy/ApEiUreyg7rQGffZGLPMGHPU8fRXwNOHxHTm3xngXmA6cKgyw7mJM/t8A/C1MWY3gDHG0/fbmX02QLiICFAdqxAUVG5M1zHGLMbah/K4/PjljYWgPrCnxPNkx7Kz3caTnO3+3I71icKTnXGfRaQ+MAx4D+/gzL9zcyBKRBaKyEoRGVlp6dzDmX1+C2gF7APWAvcbY4oqJ54tXH788ophqEuRMpaVvjTKmW08idP7IyK9sQrBxW5N5H7O7PPrwCPGmELrw6LHc2afA4ALgEuBasByEfnVGLPF3eHcxJl9vhxYDfQBmgLzRGSJMSbdzdns4vLjlzcWgmQgvsTzOKxPCme7jSdxan9EpD0wERhgjEmtpGzu4sw+JwBTHUWgJjBQRAqMMTMqJaHrOfu3nWKMyQQyRWQx0AHw1ELgzD7fCrxgrAb0JBHZAbQEfq+ciJXO5ccvb2waWgE0E5HGIhIEDAdmltpmJjDS0fveDUgzxuyv7KAudMZ9FpEGwNfATR786bCkM+6zMaaxMaaRMaYR8BVwlwcXAXDub/tboIeIBIhIKNAV2FjJOV3JmX3ejXUGhIjUBloA2ys1ZeVy+fHL684IjDEFInIPMAfrioNJxpj1IjLWsf49rCtIBgJJQBbWJwqP5eQ+PwHEAO84PiEXGA8esMvJffYqzuyzMWajiMwG1gBFwERjTJmXIXoCJ/+d/wN8JCJrsZpNHjHGeOyopCIyBegF1BSRZOBJIBDcd/zSO4uVUsrHeWPTkFJKqbOghUAppXycFgKllPJxWgiUUsrHaSFQSikfp4VAqTI4RitdLSLrHCNbRrr49XeKSE3H4+OufG2lzpYWAqXKlm2M6WiMaYs1ANjddgdSyl20ECh1ZstxDOolIk1FZLZjQLclItLSsby2iHzjGBP/TxHp7lg+w7HtehEZY+M+KFUur7uzWClXEhF/rOELPnAsGg+MNcZsFZGuwDtYg539D1hkjBnm+Jnqju1vM8YcEZFqwAoRme4F4zwpL6OFQKmyVROR1UAjYCXWiJbVsSb4+bLEaKbBju99gJEAxphCIM2x/D4RGeZ4HA80A7QQqCpFC4FSZcs2xnQUkRrA91h9BB8Bx4wxHZ15ARHpBfQFLjTGZInIQiDEHWGVOhfaR6DUaRhj0oD7gH8A2cAOEbkGiueOPTH383zgTsdyfxGJAGoARx1FoCXWtIJKVTlaCJQ6A2PMKqy5cocDI4DbReRPYD1/TZt4P9DbMQLmSqANMBsIEJE1WCNk/lrZ2ZVyho4+qpRSPk7PCJRSysdpIVBKKR+nhUAppXycFgKllPJxWgiUUsrHaSFQSikfp4VAKaV8nBYCpZTycf8PnJmxOFUMJRAAAAAASUVORK5CYII=\n",
390 |       "text/plain": [
391 |        "<Figure size 432x360 with 1 Axes>"
392 |       ]
393 |      },
394 |      "metadata": {
395 |       "needs_background": "light"
396 |      },
397 |      "output_type": "display_data"
398 |     }
399 |    ],
400 |    "source": [
401 |     "# plot pr curve\n",
402 |     "train.plot_pr_curve(X_test, y_test)"
403 |    ]
404 |   },
405 |   {
406 |    "cell_type": "markdown",
407 |    "id": "fa34b9d3",
408 |    "metadata": {},
409 |    "source": [
410 |     "### Interpret the Model"
411 |    ]
412 |   },
413 |   {
414 |    "cell_type": "markdown",
415 |    "id": "2ca793cf",
416 |    "metadata": {},
417 |    "source": [
418 |     "#### Categorical and Numerical Features"
419 |    ]
420 |   },
421 |   {
422 |    "cell_type": "code",
423 |    "execution_count": 32,
424 |    "id": "5ad792da",
425 |    "metadata": {},
426 |    "outputs": [
427 |     {
428 |      "name": "stderr",
429 |      "output_type": "stream",
430 |      "text": [
431 |       "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/xgboost/sklearn.py:1146: UserWarning: The use of label encoder in XGBClassifier is deprecated and will be removed in a future release. To remove this warning, do the following: 1) Pass option use_label_encoder=False when constructing XGBClassifier object; and 2) Encode your labels (y) as integers starting with 0, i.e. 0, 1, 2, ..., [num_class - 1].\n",
432 |       "  warnings.warn(label_encoder_deprecation_msg, UserWarning)\n",
433 |       "/home/ec2-user/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sklearn/utils/validation.py:63: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
434 |       "  return f(*args, **kwargs)\n"
435 |      ]
436 |     },
437 |     {
438 |      "name": "stdout",
439 |      "output_type": "stream",
440 |      "text": [
441 |       "[16:15:16] WARNING: ../src/learner.cc:1095: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n"
442 |      ]
443 |     },
444 |     {
445 |      "data": {
446 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgMAAAEICAYAAADC9PcJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAstklEQVR4nO3deZgcVb3/8fcnwxJIQgISvCGgAxhlC4QwgMhyg6wCKjyg8cqS4MJFEBV/CFFBIOIVhCtc9IoGBGTf4QIREwRCkDUTSDKAgkCi7Mg2BAKRJN/fH3VGOk33TC8z0zOpz+t5+qG66tQ531M9pL51TnWXIgIzMzPLrwGNDsDMzMway8mAmZlZzjkZMDMzyzknA2ZmZjnnZMDMzCznnAyYmZnlnJMBM6uZpJD0sUbHYWb1cTJgllOSBktaIOnLBeuGSPq7pAMbGVs5kk6WdGkXZRZIekfSWwWvdetsd4Gk3eqpo8r2uuxnb5E0UdKfGh2H9SwnA2Y5FRFvAYcD/yNpeFr9M6A1Iq5tXGTd4rMRMbjg9Xwjg5G0UiPbr1V/jduq52TALMciYjowFThH0jjgi8BRHdslfUjSzZLelDRL0qklrhL3lvS0pFcknSFpQNp3gKQTJP1N0suSLpY0tKDuz0l6VNIbkmZI2qRg2/GSnpO0UNLjknaVtBfwA2B8utqfW01fJQ2V9FtJL6S6T5XUlLZtJOkOSa+mflwmaVjadgnwEeDm1O5xksZJerao/n+NHqQr+2slXSrpTWBiZ+1XEHtIOlLSX9Mx+XGK+b702VwtaZVUdpykZyX9IPVlgaSDio7DxZL+kT6bEwo+s4mS7pF0lqTXgKuAXwPbp76/kcrtI+nh1PYzkk4uqL85xTshjTK9IumHBdubUmxPpb7MlrR+2raxpNskvZY+9y9W8xlbHSLCL7/8yvELWBN4AXgFOKxo25XptTqwKfAM8KeC7QHcCaxFdsJ8Avha2vYV4ElgQ2AwcD1wSdr2ceBtYHdgZeC4VHYV4BOpnXVT2WZgo7R8MnBpF/1ZAOxWYv2NwG+AQcA6wIPAf6ZtH0uxrAoMB2YCZ5erExgHPFuu3RTne8B+ZBddq3XWfolYl+tnOs43AWsAmwGLgdvTsR0KPAZMKIhtCfDz1J9/T8f6E2n7xcD/AUPSsX0C+GraNjHtezSwUop7YuFnXtDG6NS3LYCXgP0KPq8Azkv7b5ni3SRt/x7Qlj5npe0fSsflGeCw1PZYsr/JzRr9/0geXg0PwC+//Gr8C/gjsAgYWrCuKZ3QPlGw7lQ+mAzsVfD+SOD2tHw7cGTBtk+k+lYCTgSuLtg2AHgunWQ+BrwM7AasXBTncifJMn1ZALwFvJFeNwIfTiek1QrK/QdwZ5k69gMeLqqz2mRgZsG2attfrp/pOO9Q8H42cHzB+/8mJS+8nwwMKth+dTrmTSmOTQu2/ScwIy1PBP5eFMtEipKBEvGeDZyVlptTvOsVbH8Q+FJafhz4fIk6xgN3F637DXBSo///yMPL80FmOSfpYLJ/wP8InA4ckTYNJztxP1NQ/Bk+qHDd34COm/XWTe8Lt61EdmJcbltELJP0DDAyImZI+g7ZCXEzSdOA70Z18/77RcQfC/q4LdkIxAuSOlYP6Ihd0jrAOcBOZFfMA4DXq2ivlMLj8tHO2q/QSwXL75R4/28F71+PiLcL3nd8LmuTjb4Ufy4jy8RdkqTtgNOAzVN9qwLXFBV7sWB5EdnoEMD6wFMlqv0osF3HVESyEnBJV/FY/XzPgFmOpZPgWcDXya4Qvyhp57T5H2RXmOsV7LJ+iWoK130E6DhpP0/2D3zhtiVkJ7Hltik7Q65PNjpARFweETumMkGWpJCWa/EM2RXx2hExLL3WiIjN0vafprq3iIg1gIPJhrA7FLf7NtnUSUf8TWTJU6HCfbpqv7utKWlQwfuOz+UVstGZ4s/luTJxl3oPcDnZtMX6ETGU7L4ClShXyjPARmXW31VwfIZFdvPnNyqs1+rgZMAs334J3BgRd0bEC2Rz9+dJWjUilpLN858saXVJGwOHlqjje5LWTDeBfZvspjOAK4BjJG0gaTDwX8BVEbGEbNh6n3Rj4MrA/yM7Wd4r6ROSPi1pVeBdsqvepanOl4DmjhveKpX6Nh34b0lrKLu5cSNJ/56KDCFNLUgaSTavXeglsvn5Dk8AA9ONdCsDJ5BdHdfafk84RdIqknYC9gWuSZ/p1cBPlH2N9KPAd4HOvsb4ErBexw2KyRDgtYh4N426fLn0riWdD/xY0ihltpD0IeAW4OOSDpG0cnpto4IbS63nOBkwyylJ+wE7UnDii4jzgWeBH6VV3yS7Qe1FsuHaK8hO2oX+j2wOew7ZNxN+m9ZfkPaZCcwnO7Efndp5nOzq+xdkV6ufJfs64D/JTqqnpfUvkt1s94NUZ8dQ9KuSHqqyy4eSDWk/RjYFcC0wIm07heyGtfbUh+uL9v0pcIKybz4cGxHtZPdHnE92Vf022XGrtf3u9mJq43ngMuCIiPhL2nY0WbxPA38iu8q/oJO67gAeBV6U9EpadyQwWdJCsr+Vq6uI7eep/HTgTbK/l9UiYiGwB/ClFPeLZCNCZZMs6z6KqHXUzczyRtLpwL9FxIRGx2KlKfuK6KURsV4XRc3+xSMDZlZW+t73Fmk4d1vgq8ANjY7LzLqXv01gZp0ZQjY1sC7Z1/3+m2xawMxWIJ4mMDMzyzlPE5iZmeWcpwms31l77bWjubm50WGYmfUrs2fPfiUiin8PA3AyYP1Qc3Mzra2tjQ7DzKxfkfS3cts8TWBmZpZzTgbMzMxyzsmAmZlZzjkZMDMzyzknA2ZmZjnnZMDMzCznnAyYmZnlnJMBMzOznPOPDlm/0/ZcO82TpjY6DDOzXrXgtH16rG6PDJiZmeWckwEzM7OcczJgZmaWc04GzMzMcs7JgJmZWc7lOhmQNENSS6PjqISkIyQdmpYvknRgo2MyM7MVg79aWCNJK0XEkt5qLyJ+3VttmZlZvvSLkQFJzZL+LOk8SY9Kmi5ptcIre0lrS1qQlidKulHSzZLmS/qmpO9KeljS/ZLWKqj+YEn3SnpE0rZp/0GSLpA0K+3z+YJ6r5F0MzC9TKxXSdq74P1Fkg6oNCZJX0/tzpV0naTV0/qTJR1b4fFaIOm/JN0nqVXSWEnTJD0l6YiCct9Lbc2TdEpB36em9h+RND6tP03SY6nsmWndZyU9kPrwR0kfTuuHS7pN0kOSfiPpb5LWTtsOlvSgpDlpW1N6XZTaa5N0TIk+HZ760rp0UXslh8HMzCrUL5KBZBTwvxGxGfAGcEAX5TcHvgxsC/wEWBQRWwH3AYcWlBsUEZ8CjgQuSOt+CNwREdsAuwBnSBqUtm0PTIiIT5dp90qg4wS6CrAr8PsqYro+IraJiC2BPwNf7aKf5TwTEdsDdwMXAQcCnwQmp9j2IDum2wJjgK0l7QzsBTwfEVtGxObAH1Kisj+wWURsAZya2vgT8MnUhyuB49L6k8iO31jgBuAjqc1N0rHZISLGAEuBg1L7IyNi84gYDVxY3JmImBIRLRHR0rT60BoPiZmZldKfpgnmR8SctDwbaO6i/J0RsRBYKKkduDmtbwO2KCh3BUBEzJS0hqRhwB7A5wquxAeSTmjAbRHxWift3gqcI2lVshPrzIh4R1KlMW0u6VRgGDAYmNZFP8u5qaDuwQXtvlvQxz2Ah1O5wWTJwd3AmZJOB26JiLslrQS8C5wvaSpwS9pnPeAqSSOAVYD5af2OZMkDEfEHSa+n9bsCWwOz0vFYDXg5HYcNJf0CmEqZURczM+sZ/SkZWFywvJTsRLKE90c3BnZSflnB+2Us3+8o2i8AAQdExOOFGyRtB7zdWZAR8a6kGcCeZFfBV1QZ00XAfhExV9JEYFxn7XWisO7idlci6+NPI+I3xTtK2hrYG/ippOkRMTlNoewKfAn4JvBp4BfAzyPiJknjgJM7qigTk4DfRcT3S7S5JdkxOwr4IvCVintqZmZ16U/TBKUsILvShGwYvBYdQ/o7Au0R0U52NX600uWrpK2qrPNK4DBgJ6q/sh8CvCBpZbIh9J4yDfiKpMEAkkZKWkfSumTTF5cCZwJjU5mhEfF74Dtkw/oAQ4Hn0vKEgrr/RHZC75iOWDOtvx04UNI6adtakj6a7icYEBHXAScCY3uiw2ZmVlp/Ghko5UzgakmHAHfUWMfrku4F1uD9q9EfA2cD81JCsADYt4o6pwMXAzdFxD+rjOdE4AHgb2RD/EOq3L8iETE9zeHfl3Ket4CDgY+R3SOxDHgP+EaK4f8kDSS7uu+4we9k4BpJzwH3Axuk9acAV6SbD+8CXgAWRsQrkk4ApksakOo/CngHuDCtA/jAyIGZmfUcRRSPkpvVJ90vsTQilkjaHjg33TDYLVYdMSpGTDi7u6ozM+sX6n1qoaTZEVHyt3X6+8iA9U0fIRuxGQD8E/h6g+MxM7NOOBmokaTRwCVFqxdHxHa9GMMNvD803+H4iKj1GwjdIiL+ClR7n0XFRo8cSmsPPtfbzCxvnAzUKCLaeP9GukbFsH8j2zczsxVDf/82gZmZmdXJyYCZmVnOeZrA+p2259ppnjS10WGYmfWoer89UA2PDJiZmeWckwEzM7OcczJgZmaWc04GzMzMcs7JgJmZWc41LBmQNENSyd9IbiRJwyQd2eg4epKk30sa1qC2//W5S1qQnlhoZmYN1C9HBiT15FcihwG9mgz0RH8kNZXbFhF7R8Qb3d2mmZn1T10mA5KaJf1Z0nmSHpU0XdJqRVd4a0takJYnSrpR0s2S5kv6pqTvSnpY0v2S1iqo/mBJ90p6RNK2af9Bki6QNCvt8/mCeq+RdDPZI4LLxXucpDZJcyWdltaVi3UzSQ9KmiNpnqRRwGnARmndGcqckWJsS4/lRdI4SXdJulrSE5JOk3RQqq9N0kap3HBJ16X+zJK0Q1p/sqQpkjoed1yqL6XiQ9LBBet/03Hil/SWpMmSHgB+IOnqgrrGpWO33BW5pENT3XMlXdJZzGViHCzpwtTneZIOSOvPldSa/mZOKbd/wWc+NcXwSMcxLipzeKqvdemi9s6qMzOzKlV6RToK+I+I+Ho6wRzQRfnNyR5UMxB4kuzhOVtJOgs4FDg7lRsUEZ+StDNwQdrvh8AdEfEVZUPZD0r6Yyq/PbBFRLxWqlFJnwH2A7aLiEVFiUcpRwD/ExGXSVoFaAImAZt3PHI3ndzGAFsCawOzJM1M+28JbAK8BjwNnB8R20r6NnA08B3gf4CzIuJPkj4CTEv7AGwN7BgR71Qan6RNgPHADhHxnqRfAQeRJRSDgEci4kfKRhueljQoIt5O+1xVdLw2IzveO0TEKwXHq7OYi50ItEfE6FTnmmn9DyPitZSo3C5pi4iYV6aOvYDnI2KfVMfQ4gIRMQWYAtkjjMvUY2ZmNag0GZgfEXPS8myguYvyd0bEQmChpHbg5rS+DdiioNwVABExU9Ia6eS/B/A5ScemMgPJHokLcFu5RCDZDbgwIhalejsrC3Af8ENJ6wHXR8RfJRWX2RG4IiKWAi9JugvYBngTmBURLwBIeor3RyzagF0KYtq0oN41JA1Jyzd1kgiUi29XsiRiVqpzNeDlVH4pcF3q+xJJfwA+K+laYB/guKL6Pw1cGxGvpH06jlfJmNNnWmw34EsdbyLi9bT4RUmHk/2NjQA2BcolA23AmZJOB26JiLs7OSZmZtbNKk0GFhcsLyU7AS3h/WmGgZ2UX1bwfllRm8VXeAEIOCAiHi/cIGk74O0u4lSJOikXa0RcnobU9wGmSfoa2RV+cZ3lVNLPAcD2xSf9dKLttD9l4hPwu4j4fold3k1JS4ergKPIRi5mlTiZlzteJWMu4wN1SNoAOBbYJiJel3QRH/wb+ZeIeELS1sDewE8lTY+IyRW0bWZm3aCeGwgXkF2hAhxYYx0d8+87kg01t5MNSR+tdLaUtFUV9U0HviJp9bRvx7B3yVglbQg8HRHnADeRjVosBDqu3AFmAuMlNUkaDuwMPFhlTN8saHNMpTuWie924EBJ66Qya0n6aJkqZgBjga9TNEWQ3E52Bf+hjrpqiLm47JrAGmSJTrukDwOf6aKf6wKLIuJS4MwUs5mZ9ZJ6koEzgW9IupdsLr0Wr6f9fw18Na37MbAyME/SI+l9RSLiD2QnzVZJc8iuTjuLdTzwSCq7MXBxRLwK3JNuZDsDuIFseHsucAdwXES8WEUfvwW0pJvrHiO7D6BSpeJ7DDgBmC5pHnAb2TD8B6RRglvITsa3lNj+KPAT4C5Jc4Gf1xDzqcCa6XjNBXaJiLnAw8CjZPeC3NNFP0eT3Rsyh+wehlO7KG9mZt1IEb4Xy/qXVUeMihETzm50GGZmPaq7n1ooaXZElPx9n375OwNmZmbWfXryx3t6jKTRwCVFqxdHxHaNiKdekvYETi9aPT8i9m9EPKVIOgz4dtHqeyLiqEbEY2Zm3cfTBNbvtLS0RGtra6PDMDPrVzxNYGZmZmU5GTAzM8s5JwNmZmY51y9vILR8a3uuneZJUxsdhpmtoLr7K339gUcGzMzMcs7JgJmZWc45GTAzM8s5JwNmZmY552TAaiJpjKT7JD2aHmg0vtExmZlZbfxtAqvVIuDQiPhregTxbEnTIuKNBsdlZmZV8siAdUnSNunqf6CkQZIeBVaJiL8CRMTzwMvA8E7qWCDpFEkPSWqTtHFav5akG1P990vaolc6ZWZm/+JkwLoUEbOAm4BTgZ8Bl0bEIx3bJW0LrAI81UVVr0TEWOBc4Ni07hTg4YjYAvgBcHGpHSUdLqlVUuvSRe119cfMzJbnZMAqNRnYHWghSwgAkDSC7AmSh0XEsi7quD79dzbQnJZ3TPsTEXcAH5I0tHjHiJgSES0R0dK0+gc2m5lZHZwMWKXWAgYDQ4CBAJLWAKYCJ0TE/RXUsTj9dynv36+iEuX8KE0zs17kZMAqNQU4EbgMOF3SKsANwMURcU0d9c4EDgKQNI5sKuHN+kI1M7Nq+NsE1iVJhwJLIuJySU3AvcCXgJ3JhvUnpqITI2JOldWfDFwoaR7ZNxQmdEvQZmZWMScD1qWIuJh0Y19ELAW2S5tK3uxXpo7mguVWYFxafg34fDeFamZmNfA0gZmZWc55ZMC6laQbgA2KVh8fEdMaEY+ZmXXNyYB1q4jYv6fbGD1yKK05fN64mVlP8TSBmZlZzjkZMDMzyzknA2ZmZjnnewas32l7rp3mSVMbHYaZ1WiB7/npczwyYGZmlnNOBszMzHLOyYCZmVnOORkwMzPLOScDZmZmOedkoAqShkk6sosyzZK+XEFdzZIeqaLtGZJaKi3fHSS1SDqnjv0nSlq3O2MyM7Pu52SgOsOATpMBoBnoMhnoDyKiNSK+VUcVEwEnA2ZmfZyTgeqcBmwkaY6kM9LrEUltksYXlNkplTkmjQDcLemh9PpUJQ1JWk3SlZLmSboKWK1g27mSWiU9KumUtG7X9JCgjjK7S7q+k/rfknS6pNmS/ihp2zT68LSkz6Uy4yTdkpZPlnRBQZlvpfXLjXBIOjaVPRBoAS5Lx2I1SVtLuiu1OU3SiLTPtyQ9lvp6ZZl4D099bl26qL2SQ2hmZhVyMlCdScBTETEGuB8YA2wJ7AackU5uk4C7I2JMRJwFvAzsHhFjgfFApcPu3wAWRcQWwE+ArQu2/TAiWoAtgH+XtAVwB7CJpOGpzGHAhZ3UPwiYERFbAwuBU4Hdgf2ByWX22RjYE9gWOEnSyuUqj4hrgVbgoHS8lgC/AA5MbV6Q+gXZMdsq9fWIMvVNiYiWiGhpWn1oJ90yM7Nq+RcIa7cjcEVELAVeknQXsA3wZlG5lYFfShoDLAU+XmH9O5MSh4iYJ2lewbYvSjqc7PMbAWyaylwCHCzpQmB74NBO6v8n8Ie03AYsjoj3JLWRTXWUMjUiFgOLJb0MfLjCvgB8AtgcuE0SQBPwQto2j2wE4UbgxirqNDOzbuBkoHaqsNwxwEtkIwgDgHeraCM+0Ki0AXAssE1EvC7pImBg2nwhcHNq45qIWNJJ3e9FREf9y4DFABGxTFK5v4vFBctLyf5+lrD8CNNAShPwaERsX2LbPmTJz+eAEyVt1kXsZmbWjTxNUJ2FwJC0PBMYL6kpDc3vDDxYVAZgKPBCRCwDDiG7Iq7ETOAgAEmbk00JAKwBvA20S/ow8JmOHSLieeB54ATgomo7V6OXgHUkfUjSqsC+BdsKj8XjwHBJ2wNIWlnSZpIGAOtHxJ3AcWQ3aQ7updjNzAyPDFQlIl6VdE+6Ye5WsuHtuWRX8MdFxIuSXgWWSJpLdkL+FXCdpC8Ad5KdyCtxLnBhmh6YQ5ZoEBFzJT0MPAo8DdxTtN9lwPCIeKz2nlYuTS1MBh4A5gN/Kdh8EfBrSe+QTVscCJwjaSjZ397ZwBPApWmdgLMi4o3eiN3MzDJ6f6TYVgSSfgk8HBG/bXQsPWXVEaNixISzGx2GmdXITy1sDEmz083nH+CRgRWIpNlkIw//r9GxmJlZ/+FkoMEk7QmcXrR6fkTsX21d6St7xfU/AKxatPqQiGirtv6+YvTIobT6ysLMrNs4GWiwiJgGTOvB+rfrqbrNzGzF4G8TmJmZ5ZyTATMzs5zzNIH1O23PtdM8aWqjwzCzIv6WQP/lkQEzM7OcczJgZmaWc04GzMzMcs7JgJmZWc45GTAzM8s5JwMNJGmYpCO7KNMs6csV1NWcHqBUbnuLpHOqiaerOqsl6SJJB6blGZJK/ka2mZn1LicDjTUM6DQZAJqBLpOBrkREa0R8qxviMTOzFYyTgcY6DdhI0hxJZ6TXI5LaJI0vKLNTKnNMulq/W9JD6fWpShqSNE7SLWn5ZEkXpKvzpyV1JAnLxVNBnU2SzkzxzpN0dFr/I0mzUl+mSFIXdVxU0O9jypQ7XFKrpNali9or6bKZmVXIPzrUWJOAzSNijKQDgCOALYG1gVmSZqYyx0bEvgCSVgd2j4h3JY0CrgBqGW7fGNgFGAI8LuncwnhSW81d1HE4sAGwVUQskbRWWv/LiJic6rgE2Be4uUwdY4CREbF5Kj+sVKGImAJMgewRxl13z8zMKuWRgb5jR+CKiFgaES8BdwHblCi3MnCepDbgGmDTGtubGhGLI+IV4GXgwzXUsRvw64hYAhARr6X1u0h6IMX4aWCzTup4GthQ0i8k7QW8WUMcZmZWBycDfUfZofQixwAvkY0gtACr1Nje4oLlpdQ2SiRguat0SQOBXwEHRsRo4DxgYLkKIuJ1sr7MAI4Czq8hDjMzq4OTgcZaSDZMDzATGJ/m0IcDOwMPFpUBGAq8EBHLgEOAph6KpxLTgSMkrQSQpgk6TvyvSBoMHNhZBZLWBgZExHXAicDYqqM2M7O6+J6BBoqIVyXdk76+dyswD5hLdrV9XES8KOlVYImkucBFZFfd10n6AnAn8HYPxvO/XexyPvBxYJ6k94DzIuKXks4D2oAFwKwu6hgJXCipIzH9fs0dMDOzmijC92JZ/7LqiFExYsLZjQ7DzIr4qYV9m6TZEVHyhnNPE5iZmeWcpwlWMJL2BE4vWj0/IvbvS3XWY/TIobT6CsTMrNs4GVjBRMQ0YFpfr9PMzPoOTxOYmZnlnJMBMzOznPM0gfU7bc+10zxpaqPDsBWM74S3PPPIgJmZWc45GTAzM8s5JwNmZmY552TAzMws55wMmJmZ5ZyTgSpIGibpyC7KNEv6cgV1NacHAvVZko6QdGgd+/+gO+MxM7Oe4WSgOsOATpMBoBnoMhnoDyLi1xFxcR1VOBkwM+sHnAxU5zRgI0lzJJ2RXo9IapM0vqDMTqnMMWkE4G5JD6XXpyppSFJTqn+WpHmS/jOtv0rS3gXlLpJ0QLnyZeoeJ+kuSVdLekLSaZIOkvRg6stGqdzJko5NyzMknZ7KPCFpp7R+oqRfFtR9S6r/NGC1dBwuS9sOTvvPkfSbFHNT6kPHcTymTMyHS2qV1Lp0UXslh9DMzCrkZKA6k4CnImIMcD8wBtgS2A04Q9KIVObuiBgTEWcBLwO7R8RYYDxwToVtfRVoj4htgG2Ar0vaALgy1YOkVYBdgd93Ur6cLYFvA6OBQ4CPR8S2wPnA0WX2WSmV+Q5wUmfBR8Qk4J10HA6StEmKe4d0/JYCB5Edw5ERsXlEjAYuLFPflIhoiYiWptWHdta0mZlVyb9AWLsdgSsiYinwkqS7yE7CbxaVWxn4paQxZCfAj1dY/x7AFpIOTO+HAqOAW4FzJK0K7AXMjIh3JJUrP79M/bMi4gUASU8B09P6NmCXMvtcn/47m2w6pBq7AlsDsyQBrEaWKN0MbCjpF8DUgjjMzKyXOBmonSosdwzwEtmV+ADg3SrqPzo9MXD5DdIMYE+yK+0ruipfxuKC5WUF75dR/u+io8zSgjJLWH6EaWCZfQX8LiK+/4EN0pZk/TkK+CLwla6CNzOz7uNpguosBIak5ZnA+DTnPRzYGXiwqAxkV+gvRMQysuH4pgrbmgZ8Q9LKAJI+LmlQ2nYlcBiwE+8/Wriz8j1pATBG0gBJ6wPbFmx7ryMe4HbgQEnrpPjWkvRRSWsDAyLiOuBEYGwvxGxmZgU8MlCFiHhV0j3pK4G3AvOAuUAAx0XEi5JeBZZImgtcBPwKuE7SF4A7gbcrbO58sqH4h5SNq/8D2C9tmw5cDNwUEf+soHxPuodsKqINeAR4qGDbFGCepIfSfQMnANMlDQDeIxsJeAe4MK0D+MDIgZmZ9SxFRKNjMKvKqiNGxYgJZzc6DFvB+KmFtqKTNDsiWkpt8zSBmZlZznmaoMEk7QmcXrR6fkTs3w11jwYuKVq9OCK2q7fuRho9ciitvoozM+s2TgYaLN39X+k3AKqtu43se/xmZmZleZrAzMws55wMmJmZ5ZynCazfaXuuneZJUxsdhq1A/E0CyzuPDJiZmeWckwEzM7OcczJgZmaWc04GzMzMcs7JgJmZWc45GaiBpGGSjuyiTLOkL1dQV3N68FG57S2Szqkmnq7qrJSkyZJ2q3HfLo+RmZn1DU4GajMM6OpE1wx0mQx0JSJaI+Jb3RBPLW3/KCL+WOPuw+iBmMzMrPs5GajNacBGkuZIOiO9HpHUJml8QZmdUplj0tX63ZIeSq9PVdKQpHGSbknLJ0u6QNIMSU9L6kgSloungjonSrpR0s2S5kv6pqTvSnpY0v2S1krlLpJ0YFpeIOmUFHubpI0LYjq2oO5HJDWXiknS9yTNkjRP0ilp3SBJUyXNTfuOpwRJh0tqldS6dFF7JYfOzMwq5B8dqs0kYPOIGCPpAOAIYEtgbWCWpJmpzLERsS+ApNWB3SPiXUmjgCuAko+S7MLGwC7AEOBxSecWxpPaaq6gns2BrYCBwJPA8RGxlaSzgEOBs0vs80pEjE3D/8cCX+uk/uKY9gBGAdsCAm6StDMwHHg+IvZJ5YaWqiwipgBTIHuEcQX9MzOzCnlkoH47AldExNKIeAm4C9imRLmVgfMktQHXAJvW2N7UiFgcEa8ALwMfrrGeOyNiYUT8A2gHbk7r28imOEq5Pv13didlytkjvR4GHiJLakal9naTdLqknSLCl/1mZr3MIwP1U4XljgFeIhtBGAC8W2N7iwuWl1L7Z1hYz7KC98s6qbOjTGG7S1g+qRxYZl8BP42I33xgg7Q1sDfwU0nTI2Jy1+GbmVl38chAbRaSDdMDzATGS2qSNBzYGXiwqAzAUOCFiFgGHAI09VA8vW0BMBZA0lhggzIxTQO+ImlwKjtS0jqS1gUWRcSlwJkddZmZWe/xyEANIuJVSfekr+/dCswD5gIBHBcRL0p6FVgiaS5wEfAr4DpJXwDuBN7uwXj+t7vqrsB1wKGS5gCzgCdKxRQR35O0CXCfJIC3gIOBjwFnSFoGvAd8oxdjNzMzQBG+F8v6l1VHjIoRE85udBi2AvFTCy0PJM2OiJI3rnuawMzMLOc8TdBHSNoTOL1o9fyI2L8v1dkXjB45lFZfyZmZdRsnA31EREwju8muT9dpZmYrHk8TmJmZ5ZyTATMzs5zzNIH1O23PtdM8aWqjw7B+yN8aMCvNIwNmZmY552TAzMws55wMmJmZ5ZyTATMzs5xzMmBmZpZzTgasbpImSPprek3oouwMSS1peYGktvR6TNKpklbtnajNzKyDkwGri6S1gJOA7YBtgZMkrVlFFbtExOi074bAlO6P0szMOuNkwComaRtJ8yQNlDRI0qPAUcBtEfFaRLwO3AbsVW3dEfEWcASwX0owzMysl/hHh6xiETFL0k3AqcBqwKXAe8AzBcWeBUbWWP+bkuYDo4AHCrdJOhw4HKBpjeG1VG9mZmV4ZMCqNRnYHWgBfgaoRJmoo/5S9RERUyKiJSJamlYfWkf1ZmZWzMmAVWstYDAwBBhINhKwfsH29YDna6lY0hCgGXiivhDNzKwaTgasWlOAE4HLgNPJHpG8h6Q1042De1DDY5MlDQZ+BdyY7j0wM7Ne4nsGrGKSDgWWRMTlkpqAe4ExwI+BWanY5Ih4rYpq75QkssT0hlSXmZn1IicDVrGIuBi4OC0vJfs6YYcLKqxjXMFyczeGZ2ZmNfI0gZmZWc55ZMB6hKQbgA2KVh8fEVXfT2BmZj3LyYD1iIjYv6fqHj1yKK2n7dNT1ZuZ5Y6nCczMzHLOyYCZmVnOORkwMzPLOd8zYP1O23PtNE+a2ugwrA9b4HtKzKrikQEzM7OcczJgZmaWc04GzMzMcs7JgJmZWc45GbCaSZog6a/pNaGTcjdImiPpSUntaXmOpB0kzZa0c0HZ6ZK+0Ds9MDMz8LcJrEaS1gJOAlqAAGZLuqnU44c7fo1Q0jjg2IjYt6CeI4HzJY0FDsyKxzU93wMzM+vgkQHrkqRtJM2TNFDSIEmPAkcBt0XEaykBuA3Yq9q6I+IBskchnwz8V6rXzMx6kUcGrEsRMUvSTcCpwGrApcB7wDMFxZ4FRtbYxPdTXWdHxJOlCkg6HDgcoGmN4TU2Y2ZmpXhkwCo1GdidbFrgZ4BKlIka694ZaAc2L1cgIqZEREtEtDStPrTGZszMrBQnA1aptYDBwBBgINlIwPoF29cDnq+2UkmDyJKLTwPDJe1df6hmZlYNJwNWqSnAicBlwOnANGAPSWtKWhPYI62r1o+AqyPiL8CRwFmSBnZTzGZmVgHfM2BdknQosCQiLpfURHbD3xjgx8CsVGxyRLxWZb2bAvsDWwJExBxJ04DjgVO6KXwzM+uCImqd5jVrjFVHjIoRE85udBjWh/lBRWYfJGl2RLSU2uZpAjMzs5zzNIF1K0k3ABsUrT4+Imq5n8DMzHqBkwHrVh2/NtiTRo8cSquHgc3Muo2nCczMzHLOyYCZmVnOORkwMzPLOd8zYP1O23PtNE+a2ugwrAx/rc+s//HIgJmZWc45GTAzM8s5JwNmZmY552TAzMws55wMmJmZ5ZyTAauJpAmS/ppeEzopd4OkOZKelNSeludI+pSkGZJaC8q2SJrRKx0wM7N/8VcLrWqS1gJOAlqAAGZLuikiXi8u2/HzxJLGAcdGxL4F9QCsI+kzEXFrL4RuZmYleGTAOiVpG0nzJA2UNEjSo8BRwG0R8VpKAG4D9qqxiTOAEyqI43BJrZJaly5qr7EpMzMrxcmAdSoiZgE3AacCPwMuBd4Bniko9iwwssYm7gMWS9qlizimRERLRLQ0rT60xqbMzKwUJwNWicnA7mTTAj8DVKJM1FH/qVQwOmBmZj3DyYBVYi1gMDAEGEg2ErB+wfb1gOdrrTwi7kj1frKOGM3MrEZOBqwSU4ATgcuA04FpwB6S1pS0JrBHWlePnwDH1VmHmZnVwN8msE5JOhRYEhGXS2oC7gXGAD8GZqVikyPitXraiYjfS/pHXcGamVlNnAxYpyLiYuDitLwU2K5g8wVV1DMDmFG0blzR+61rDNPMzOrgaQIzM7Oc88iAdRtJNwAbFK0+PiLqvZ9gOaNHDqX1tH26s0ozs1xzMmDdpuPXBs3MrH/xNIGZmVnOORkwMzPLOScDZmZmOedkwMzMLOecDJiZmeWckwEzM7OcczJgZmaWc04GzMzMck4R9TyG3qz3SVoIPN7oOHrJ2sArjQ6il+Slr3npJ7ivfc1HI2J4qQ3+BULrjx6PiJZGB9EbJLW6ryuWvPQT3Nf+xNMEZmZmOedkwMzMLOecDFh/NKXRAfQi93XFk5d+gvvab/gGQjMzs5zzyICZmVnOORkwMzPLOScD1qdI2kvS45KelDSpxHZJOidtnydpbKX79jW19lXS+pLulPRnSY9K+nbvR1+5ej7TtL1J0sOSbum9qGtT59/vMEnXSvpL+my3793oq1NnX49Jf7uPSLpC0sDejb5yFfRzY0n3SVos6dhq9u1TIsIvv/rEC2gCngI2BFYB5gKbFpXZG7gVEPBJ4IFK9+1Lrzr7OgIYm5aHAE/01b7W08+C7d8FLgduaXR/erKvwO+Ar6XlVYBhje5TT/QVGAnMB1ZL768GJja6T3X0cx1gG+AnwLHV7NuXXh4ZsL5kW+DJiHg6Iv4JXAl8vqjM54GLI3M/MEzSiAr37Utq7mtEvBARDwFExELgz2T/wPZF9XymSFoP2Ac4vzeDrlHNfZW0BrAz8FuAiPhnRLzRi7FXq67PlewH71aTtBKwOvB8bwVepS77GREvR8Qs4L1q9+1LnAxYXzISeKbg/bN88CRXrkwl+/Yl9fT1XyQ1A1sBD3R/iN2i3n6eDRwHLOuh+LpTPX3dEPgHcGGaEjlf0qCeDLZONfc1Ip4DzgT+DrwAtEfE9B6MtR71/LvSr/5NcjJgfYlKrCv+7mu5MpXs25fU09dsozQYuA74TkS82Y2xdaea+ylpX+DliJjd/WH1iHo+05WAscC5EbEV8DbQl+eY6/lc1yS7Qt4AWBcYJOngbo6vu9Tz70q/+jfJyYD1Jc8C6xe8X48PDh+WK1PJvn1JPX1F0spkicBlEXF9D8ZZr3r6uQPwOUkLyIZYPy3p0p4LtW71/v0+GxEdIzzXkiUHfVU9fd0NmB8R/4iI94DrgU/1YKz1qOfflX71b5KTAetLZgGjJG0gaRXgS8BNRWVuAg5Ndyp/kmyI8YUK9+1Lau6rJJHNLf85In7eu2FXreZ+RsT3I2K9iGhO+90REX31ChLq6+uLwDOSPpHK7Qo81muRV6+e/1f/DnxS0urpb3lXsvte+qJ6/l3pV/8m+amF1mdExBJJ3wSmkd2Je0FEPCrpiLT918Dvye5SfhJYBBzW2b4N6EZF6ukr2RXzIUCbpDlp3Q8i4ve92IWK1NnPfqUb+no0cFk6cTxNHz4Odf6/+oCka4GHgCXAw/TRn/KtpJ+S/g1oBdYAlkn6Dtm3Bt7sT/8m+eeIzczMcs7TBGZmZjnnZMDMzCznnAyYmZnlnJMBMzOznHMyYGZmlnNOBszMzHLOyYCZmVnO/X88qiWCoamOfAAAAABJRU5ErkJggg==\n",
447 |       "text/plain": [
448 |        "<Figure size 432x288 with 1 Axes>"
449 |       ]
450 |      },
451 |      "metadata": {
452 |       "needs_background": "light"
453 |      },
454 |      "output_type": "display_data"
455 |     }
456 |    ],
457 |    "source": [
458 |     "preds_xgb = interpret.train_xgb()"
459 |    ]
460 |   },
461 |   {
462 |    "cell_type": "markdown",
463 |    "id": "9e80253e",
464 |    "metadata": {},
465 |    "source": [
466 |     "#### Textual Features (focus on customer chats that result in churn)"
467 |    ]
468 |   },
469 |   {
470 |    "cell_type": "code",
471 |    "execution_count": 9,
472 |    "id": "7f18dcf5",
473 |    "metadata": {},
474 |    "outputs": [],
475 |    "source": [
476 |     "chats, df_sub = interpret.get_chats(df=df, churn=1, speaker='Customer')"
477 |    ]
478 |   },
479 |   {
480 |    "cell_type": "code",
481 |    "execution_count": 10,
482 |    "id": "0b65267d",
483 |    "metadata": {},
484 |    "outputs": [
485 |     {
486 |      "data": {
487 |       "text/plain": [
488 |        "[\"Well, I just want to be able to cancel the contract because I don't think that I want to stay. My local provider has been terrible and I really would like to switch. Sure, I can.\",\n",
489 |        " \"Well, it's the old TelCom billing system for the last 5 years. I don't trust anymore and I think you should change to the newer billing system. I would like to give you a call back number. Okay, I can see why you need the new billing system, but I don't know if I can do that. I would like to know your cancellation policy.\",\n",
490 |        " \"Well, I've been getting phone calls from a very good friend who's a TelCom agent and I have told him the same thing and the problem has not been resolved. He has offered me a $20/mo deal but that's not good enough for me because I'm getting $20 out of his pocket. Sure. $60 to cancel for nine months with a $20/mo bonus.\"]"
491 |       ]
492 |      },
493 |      "execution_count": 10,
494 |      "metadata": {},
495 |      "output_type": "execute_result"
496 |     }
497 |    ],
498 |    "source": [
499 |     "chats[:3]"
500 |    ]
501 |   },
502 |   {
503 |    "cell_type": "markdown",
504 |    "id": "8c73e7dd",
505 |    "metadata": {},
506 |    "source": [
507 |     "##### Candidate keywords (POS tagging, lower casing, lemmatization)"
508 |    ]
509 |   },
510 |   {
511 |    "cell_type": "code",
512 |    "execution_count": 11,
513 |    "id": "3db8f636",
514 |    "metadata": {},
515 |    "outputs": [
516 |     {
517 |      "name": "stdout",
518 |      "output_type": "stream",
519 |      "text": [
520 |       "['want', 'able', 'cancel', 'contract', 'think', 'want', 'stay', 'local', 'provider', 'terrible']\n"
521 |      ]
522 |     }
523 |    ],
524 |    "source": [
525 |     "# find candidate keywords\n",
526 |     "keywords, tokens = interpret.get_keywords(chats)\n",
527 |     "print(keywords[:10])\n",
528 |     "\n",
529 |     "# map keywords to original tokens\n",
530 |     "keywords_dict = interpret.map_to_orig_tok(keywords, tokens)"
531 |    ]
532 |   },
533 |   {
534 |    "cell_type": "markdown",
535 |    "id": "172eed77",
536 |    "metadata": {},
537 |    "source": [
538 |     "##### Relevant keywords (semantic similarity)"
539 |    ]
540 |   },
541 |   {
542 |    "cell_type": "code",
543 |    "execution_count": 12,
544 |    "id": "07be6aa0",
545 |    "metadata": {},
546 |    "outputs": [],
547 |    "source": [
548 |     "relevant_keywords, simMat = interpret.get_relevant_keywords(\n",
549 |     "    text = chats, \n",
550 |     "    keywords_dict = keywords_dict\n",
551 |     ")"
552 |    ]
553 |   },
554 |   {
555 |    "cell_type": "code",
556 |    "execution_count": 13,
557 |    "id": "267a00b0",
558 |    "metadata": {},
559 |    "outputs": [
560 |     {
561 |      "name": "stdout",
562 |      "output_type": "stream",
563 |      "text": [
564 |       "['voicemail', 'bored', 'spam', 'backlog', 'sick', 'frustrated', 'incompetence', 'overcharge', 'angry', 'disappointed', 'scam', 'lag', 'termination', 'resign', 'nightmare', 'incompetent', 'frustration', 'lagging', 'yesterday', 'friday', 'monday', 'discontinue', 'cheat', 'dead', 'annoyed']\n"
565 |      ]
566 |     }
567 |    ],
568 |    "source": [
569 |     "print(relevant_keywords[:25])"
570 |    ]
571 |   },
572 |   {
573 |    "cell_type": "markdown",
574 |    "id": "3ba0da26",
575 |    "metadata": {},
576 |    "source": [
577 |     "##### Impactful kewords (marginal contribution)"
578 |    ]
579 |   },
580 |   {
581 |    "cell_type": "code",
582 |    "execution_count": 14,
583 |    "id": "8c877937",
584 |    "metadata": {},
585 |    "outputs": [
586 |     {
587 |      "name": "stderr",
588 |      "output_type": "stream",
589 |      "text": [
590 |       "100%|██████████| 250/250 [11:25<00:00,  2.74s/it]\n"
591 |      ]
592 |     }
593 |    ],
594 |    "source": [
595 |     "# get marginal contribution to prediction for each keyword\n",
596 |     "marg_contr_df = interpret.perform_ablation(\n",
597 |     "    df = df_sub,\n",
598 |     "    keywords = relevant_keywords,\n",
599 |     "    keywords_dict = keywords_dict\n",
600 |     ")"
601 |    ]
602 |   },
603 |   {
604 |    "cell_type": "markdown",
605 |    "id": "d43f8aa6",
606 |    "metadata": {},
607 |    "source": [
608 |     "##### Create joint metric (semantic similarity + marginal contribution + count)"
609 |    ]
610 |   },
611 |   {
612 |    "cell_type": "code",
613 |    "execution_count": null,
614 |    "id": "c02d300a",
615 |    "metadata": {},
616 |    "outputs": [],
617 |    "source": [
618 |     "# load from local disc if available\n",
619 |     "#results_df = pd.read_csv('model/ablation_results.csv')\n",
620 |     "#results_df = results_df.rename(columns={'Unnamed: 0' : 'keyword'})"
621 |    ]
622 |   },
623 |   {
624 |    "cell_type": "code",
625 |    "execution_count": 15,
626 |    "id": "2e112e00",
627 |    "metadata": {},
628 |    "outputs": [],
629 |    "source": [
630 |     "results_df = interpret.get_important_keywords(\n",
631 |     "    simMat_df=simMat,\n",
632 |     "    marg_contr_df=marg_contr_df\n",
633 |     ")"
634 |    ]
635 |   },
636 |   {
637 |    "cell_type": "code",
638 |    "execution_count": 16,
639 |    "id": "efd90445",
640 |    "metadata": {},
641 |    "outputs": [
642 |     {
643 |      "data": {
644 |       "text/html": [
645 |        "<div>\n",
646 |        "<style scoped>\n",
647 |        "    .dataframe tbody tr th:only-of-type {\n",
648 |        "        vertical-align: middle;\n",
649 |        "    }\n",
650 |        "\n",
651 |        "    .dataframe tbody tr th {\n",
652 |        "        vertical-align: top;\n",
653 |        "    }\n",
654 |        "\n",
655 |        "    .dataframe thead th {\n",
656 |        "        text-align: right;\n",
657 |        "    }\n",
658 |        "</style>\n",
659 |        "<table border=\"1\" class=\"dataframe\">\n",
660 |        "  <thead>\n",
661 |        "    <tr style=\"text-align: right;\">\n",
662 |        "      <th></th>\n",
663 |        "      <th>keyword</th>\n",
664 |        "      <th>sim</th>\n",
665 |        "      <th>chg</th>\n",
666 |        "      <th>count</th>\n",
667 |        "      <th>joint</th>\n",
668 |        "    </tr>\n",
669 |        "  </thead>\n",
670 |        "  <tbody>\n",
671 |        "    <tr>\n",
672 |        "      <th>0</th>\n",
673 |        "      <td>voicemail</td>\n",
674 |        "      <td>90.076553</td>\n",
675 |        "      <td>0.006695</td>\n",
676 |        "      <td>5</td>\n",
677 |        "      <td>0.628774</td>\n",
678 |        "    </tr>\n",
679 |        "    <tr>\n",
680 |        "      <th>1</th>\n",
681 |        "      <td>cancel</td>\n",
682 |        "      <td>61.187359</td>\n",
683 |        "      <td>-0.081321</td>\n",
684 |        "      <td>168</td>\n",
685 |        "      <td>0.545633</td>\n",
686 |        "    </tr>\n",
687 |        "    <tr>\n",
688 |        "      <th>2</th>\n",
689 |        "      <td>sick</td>\n",
690 |        "      <td>74.789459</td>\n",
691 |        "      <td>-0.127242</td>\n",
692 |        "      <td>1</td>\n",
693 |        "      <td>0.538919</td>\n",
694 |        "    </tr>\n",
695 |        "    <tr>\n",
696 |        "      <th>3</th>\n",
697 |        "      <td>turnover</td>\n",
698 |        "      <td>60.896118</td>\n",
699 |        "      <td>-0.286630</td>\n",
700 |        "      <td>1</td>\n",
701 |        "      <td>0.533321</td>\n",
702 |        "    </tr>\n",
703 |        "    <tr>\n",
704 |        "      <th>4</th>\n",
705 |        "      <td>disappointed</td>\n",
706 |        "      <td>70.248131</td>\n",
707 |        "      <td>-0.091740</td>\n",
708 |        "      <td>5</td>\n",
709 |        "      <td>0.522520</td>\n",
710 |        "    </tr>\n",
711 |        "    <tr>\n",
712 |        "      <th>5</th>\n",
713 |        "      <td>spam</td>\n",
714 |        "      <td>77.940460</td>\n",
715 |        "      <td>-0.000022</td>\n",
716 |        "      <td>3</td>\n",
717 |        "      <td>0.506429</td>\n",
718 |        "    </tr>\n",
719 |        "    <tr>\n",
720 |        "      <th>6</th>\n",
721 |        "      <td>bored</td>\n",
722 |        "      <td>78.131271</td>\n",
723 |        "      <td>-0.038932</td>\n",
724 |        "      <td>1</td>\n",
725 |        "      <td>0.502213</td>\n",
726 |        "    </tr>\n",
727 |        "    <tr>\n",
728 |        "      <th>7</th>\n",
729 |        "      <td>unhappy</td>\n",
730 |        "      <td>65.601990</td>\n",
731 |        "      <td>-0.024782</td>\n",
732 |        "      <td>37</td>\n",
733 |        "      <td>0.493910</td>\n",
734 |        "    </tr>\n",
735 |        "    <tr>\n",
736 |        "      <th>8</th>\n",
737 |        "      <td>frustrated</td>\n",
738 |        "      <td>73.496033</td>\n",
739 |        "      <td>-0.006023</td>\n",
740 |        "      <td>5</td>\n",
741 |        "      <td>0.486930</td>\n",
742 |        "    </tr>\n",
743 |        "    <tr>\n",
744 |        "      <th>9</th>\n",
745 |        "      <td>mistake</td>\n",
746 |        "      <td>66.247879</td>\n",
747 |        "      <td>-0.093309</td>\n",
748 |        "      <td>3</td>\n",
749 |        "      <td>0.470609</td>\n",
750 |        "    </tr>\n",
751 |        "    <tr>\n",
752 |        "      <th>10</th>\n",
753 |        "      <td>late</td>\n",
754 |        "      <td>65.971649</td>\n",
755 |        "      <td>-0.003873</td>\n",
756 |        "      <td>18</td>\n",
757 |        "      <td>0.458023</td>\n",
758 |        "    </tr>\n",
759 |        "    <tr>\n",
760 |        "      <th>11</th>\n",
761 |        "      <td>error</td>\n",
762 |        "      <td>63.123695</td>\n",
763 |        "      <td>-0.065609</td>\n",
764 |        "      <td>7</td>\n",
765 |        "      <td>0.448412</td>\n",
766 |        "    </tr>\n",
767 |        "    <tr>\n",
768 |        "      <th>12</th>\n",
769 |        "      <td>faulty</td>\n",
770 |        "      <td>55.486092</td>\n",
771 |        "      <td>-0.239817</td>\n",
772 |        "      <td>1</td>\n",
773 |        "      <td>0.448232</td>\n",
774 |        "    </tr>\n",
775 |        "    <tr>\n",
776 |        "      <th>13</th>\n",
777 |        "      <td>angry</td>\n",
778 |        "      <td>70.865860</td>\n",
779 |        "      <td>-0.002967</td>\n",
780 |        "      <td>3</td>\n",
781 |        "      <td>0.444018</td>\n",
782 |        "    </tr>\n",
783 |        "    <tr>\n",
784 |        "      <th>14</th>\n",
785 |        "      <td>backlog</td>\n",
786 |        "      <td>74.808548</td>\n",
787 |        "      <td>0.000020</td>\n",
788 |        "      <td>1</td>\n",
789 |        "      <td>0.442185</td>\n",
790 |        "    </tr>\n",
791 |        "    <tr>\n",
792 |        "      <th>15</th>\n",
793 |        "      <td>customer</td>\n",
794 |        "      <td>52.072552</td>\n",
795 |        "      <td>-0.006985</td>\n",
796 |        "      <td>480</td>\n",
797 |        "      <td>0.439737</td>\n",
798 |        "    </tr>\n",
799 |        "    <tr>\n",
800 |        "      <th>16</th>\n",
801 |        "      <td>lag</td>\n",
802 |        "      <td>69.912178</td>\n",
803 |        "      <td>-0.004650</td>\n",
804 |        "      <td>3</td>\n",
805 |        "      <td>0.436584</td>\n",
806 |        "    </tr>\n",
807 |        "    <tr>\n",
808 |        "      <th>17</th>\n",
809 |        "      <td>overpay</td>\n",
810 |        "      <td>65.573105</td>\n",
811 |        "      <td>-0.093846</td>\n",
812 |        "      <td>1</td>\n",
813 |        "      <td>0.429261</td>\n",
814 |        "    </tr>\n",
815 |        "    <tr>\n",
816 |        "      <th>18</th>\n",
817 |        "      <td>disconnect</td>\n",
818 |        "      <td>64.002014</td>\n",
819 |        "      <td>-0.010485</td>\n",
820 |        "      <td>11</td>\n",
821 |        "      <td>0.429104</td>\n",
822 |        "    </tr>\n",
823 |        "    <tr>\n",
824 |        "      <th>19</th>\n",
825 |        "      <td>incompetence</td>\n",
826 |        "      <td>73.107452</td>\n",
827 |        "      <td>-0.000754</td>\n",
828 |        "      <td>1</td>\n",
829 |        "      <td>0.427229</td>\n",
830 |        "    </tr>\n",
831 |        "  </tbody>\n",
832 |        "</table>\n",
833 |        "</div>"
834 |       ],
835 |       "text/plain": [
836 |        "         keyword        sim       chg  count     joint\n",
837 |        "0      voicemail  90.076553  0.006695      5  0.628774\n",
838 |        "1         cancel  61.187359 -0.081321    168  0.545633\n",
839 |        "2           sick  74.789459 -0.127242      1  0.538919\n",
840 |        "3       turnover  60.896118 -0.286630      1  0.533321\n",
841 |        "4   disappointed  70.248131 -0.091740      5  0.522520\n",
842 |        "5           spam  77.940460 -0.000022      3  0.506429\n",
843 |        "6          bored  78.131271 -0.038932      1  0.502213\n",
844 |        "7        unhappy  65.601990 -0.024782     37  0.493910\n",
845 |        "8     frustrated  73.496033 -0.006023      5  0.486930\n",
846 |        "9        mistake  66.247879 -0.093309      3  0.470609\n",
847 |        "10          late  65.971649 -0.003873     18  0.458023\n",
848 |        "11         error  63.123695 -0.065609      7  0.448412\n",
849 |        "12        faulty  55.486092 -0.239817      1  0.448232\n",
850 |        "13         angry  70.865860 -0.002967      3  0.444018\n",
851 |        "14       backlog  74.808548  0.000020      1  0.442185\n",
852 |        "15      customer  52.072552 -0.006985    480  0.439737\n",
853 |        "16           lag  69.912178 -0.004650      3  0.436584\n",
854 |        "17       overpay  65.573105 -0.093846      1  0.429261\n",
855 |        "18    disconnect  64.002014 -0.010485     11  0.429104\n",
856 |        "19  incompetence  73.107452 -0.000754      1  0.427229"
857 |       ]
858 |      },
859 |      "execution_count": 16,
860 |      "metadata": {},
861 |      "output_type": "execute_result"
862 |     }
863 |    ],
864 |    "source": [
865 |     "results_df.head(20)"
866 |    ]
867 |   },
868 |   {
869 |    "cell_type": "markdown",
870 |    "id": "7036dc83",
871 |    "metadata": {},
872 |    "source": [
873 |     "##### Context of keywords"
874 |    ]
875 |   },
876 |   {
877 |    "cell_type": "code",
878 |    "execution_count": 17,
879 |    "id": "5c0f0d2b",
880 |    "metadata": {},
881 |    "outputs": [],
882 |    "source": [
883 |     "keyword_of_interest = 'spam'"
884 |    ]
885 |   },
886 |   {
887 |    "cell_type": "code",
888 |    "execution_count": 18,
889 |    "id": "041befbf",
890 |    "metadata": {},
891 |    "outputs": [
892 |     {
893 |      "name": "stdout",
894 |      "output_type": "stream",
895 |      "text": [
896 |       "Basically, I'm getting a lot of spam calls every day from a guy named Michael who's calling from a really weird number.\n",
897 |       "TelCom started to flood me with emails and phone calls, spamming me with thousands of phony invoices.\n",
898 |       "I just got some spam messages last night, and today it's been getting a lot of texts that I \"don't have my SIM card\" and \"I need my SIM card.\n"
899 |      ]
900 |     }
901 |    ],
902 |    "source": [
903 |     "interpret.obtain_context(\n",
904 |     "    chats_list = chats,\n",
905 |     "    keyword = keyword_of_interest\n",
906 |     ")"
907 |    ]
908 |   },
909 |   {
910 |    "cell_type": "code",
911 |    "execution_count": null,
912 |    "id": "a294f494",
913 |    "metadata": {},
914 |    "outputs": [],
915 |    "source": []
916 |   }
917 |  ],
918 |  "metadata": {
919 |   "kernelspec": {
920 |    "display_name": "conda_pytorch_p36",
921 |    "language": "python",
922 |    "name": "conda_pytorch_p36"
923 |   },
924 |   "language_info": {
925 |    "codemirror_mode": {
926 |     "name": "ipython",
927 |     "version": 3
928 |    },
929 |    "file_extension": ".py",
930 |    "mimetype": "text/x-python",
931 |    "name": "python",
932 |    "nbconvert_exporter": "python",
933 |    "pygments_lexer": "ipython3",
934 |    "version": "3.6.13"
935 |   }
936 |  },
937 |  "nbformat": 4,
938 |  "nbformat_minor": 5
939 | }
940 | 


--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | pandas
2 | matplotlib
3 | sentence_transformers==2.0.0
4 | xgboost==1.4.2
5 | spacy>=3.0.0,<4.0.0
6 | https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.0.0/en_core_web_sm-3.0.0.tar.gz#egg=en_core_web_sm


--------------------------------------------------------------------------------
/scripts/create_dataset.py:
--------------------------------------------------------------------------------
 1 | """ Module for creating the dataset by combining categorical/numerical and text csv. files.
 2 | 
 3 | First, download categorical/numerical data into data folder as described in README.md, then:
 4 | 
 5 | Run in CLI example:
 6 |     'python create_dataset.py'
 7 | 
 8 | """
 9 | 
10 | import yaml
11 | import pandas as pd
12 | from pathlib import Path
13 | 
14 | with open("../model/params.yaml", "r") as params_file:
15 |     params = yaml.safe_load(params_file)
16 | 
17 | 
18 | def create_joint_dataset(
19 |     df_categorical, 
20 |     df_text
21 | ):
22 |     df_text_no = df_text[df_text.churn == 'no'].reset_index(drop=True)
23 |     df_text_yes = df_text[df_text.churn == 'yes'].reset_index(drop=True)
24 | 
25 |     df_cat_no = df_categorical[df_categorical.churn == 'no'].reset_index(drop=True)[:len(df_text_no)]
26 |     df_cat_yes = df_categorical[df_categorical.churn == 'yes'].reset_index(drop=True)[:len(df_text_yes)]
27 | 
28 |     df_no = pd.concat([df_text_no, df_cat_no.iloc[:,:-1]], axis=1)
29 |     df_yes = pd.concat([df_text_yes, df_cat_yes.iloc[:,:-1]], axis=1)
30 | 
31 |     df = pd.concat([df_no, df_yes], axis=0)
32 | 
33 |     # shuffle data
34 |     df = df.sample(frac=1).reset_index(drop=True)
35 | 
36 |     return df
37 | 
38 | 
39 | if __name__ == "__main__":
40 |     data_dir = params['data_dir']
41 |     
42 |     # load data
43 |     df_categorical = pd.read_csv(Path(data_dir, "train.csv"))
44 |     df_text = pd.read_csv(Path(data_dir, "text.csv"))
45 |     
46 |     # create joint dataset
47 |     df = create_joint_dataset(df_categorical, df_text)
48 |     
49 |     # save data
50 |     df.to_csv(Path(data_dir, "churn_dataset.csv"), index=False)
51 | 


--------------------------------------------------------------------------------
/scripts/interpret.py:
--------------------------------------------------------------------------------
  1 | """ Module for interpreting the trained churn prediction model with text. 
  2 | 
  3 | The parameters for model interpretation are stored in 'params.yaml'.
  4 | 
  5 | Run in CLI example:
  6 |     'python interpret.py --churn 1 --speaker Customer'
  7 | 
  8 | """
  9 | 
 10 | 
 11 | import json
 12 | import yaml
 13 | import spacy
 14 | import torch
 15 | import argparse
 16 | import numpy as np
 17 | import pandas as pd
 18 | from tqdm import tqdm
 19 | from pathlib import Path
 20 | from sklearn import preprocessing
 21 | from xgboost import XGBClassifier
 22 | from collections import defaultdict
 23 | from matplotlib import pyplot as plt
 24 | import preprocess
 25 | from preprocess import BertEncoder
 26 | import train
 27 | 
 28 | nlp = spacy.load('en_core_web_sm')
 29 | 
 30 | with open("../model/params.yaml", "r") as params_file:
 31 |     params = yaml.safe_load(params_file)
 32 | 
 33 | model_dir = params['model_dir']
 34 | 
 35 | 
 36 | def get_chats(
 37 |     df,
 38 |     churn=None,
 39 |     speaker=None
 40 | ):
 41 |     """
 42 |     Args:
 43 |         df: dataframe with all data
 44 |         churn (int): 1 for churn, 0 for no churn, None (default) for all chats
 45 |         customer (str): 'Customer' for only customer chats,
 46 |                         'Agent' for only agent chats, 
 47 |                          None (default) for all chats
 48 |     
 49 |     Returns:
 50 |         list of chats (strings)
 51 |     """
 52 |     # convert labels to binary numeric
 53 |     df = preprocess.convert_label(df)
 54 |     # keep only churn/no churn chat logs
 55 |     if churn is not None:
 56 |         df = df[df.churn == churn]
 57 |     # drop short chat logs
 58 |     df = df[df.chat_log.apply(lambda x: len(str(x))>=5)]
 59 |     
 60 |     # select chats
 61 |     chat_logs = list(df['chat_log'])
 62 |     chat_logs = [i if isinstance(i, str) else "nan" for i in chat_logs]
 63 |     
 64 |     if speaker is not None:
 65 |         chats = []
 66 |         for chat in chat_logs:
 67 |             sents = chat.split('\n')
 68 |             cchat = []
 69 |             for sent in sents:
 70 |                 if str(sent).split(':')[0] == speaker:
 71 |                     cchat.append(sent[10:])
 72 |             chats.append(' '.join(cchat))
 73 |     else:
 74 |         chats = chat_logs
 75 | 
 76 |     return chats, df
 77 | 
 78 | 
 79 | def get_keywords(texts):
 80 |     """Returns candidate keywords based on POS as well as original form of keyword.
 81 |     """
 82 |     candidate_pos = ['ADJ', 'VERB', 'NOUN', 'PROPN']
 83 |     keywords = []
 84 |     tokens = [] # for referencing keywords in original text later on
 85 |     for text in texts:
 86 |         text_keywords = []
 87 |         text_tokens = []
 88 |         doc = nlp(text)
 89 |         for token in doc:
 90 |             if token.pos_ in candidate_pos and token.is_stop is False:
 91 |                 text_tokens.append(str(token))
 92 |                 text_keywords.append(token.lemma_.lower())
 93 | 
 94 |         keywords.extend(text_keywords)
 95 |         tokens.extend(text_tokens)
 96 |         
 97 |     return keywords, tokens
 98 | 
 99 | 
100 | def map_to_orig_tok(keywords, tokens):
101 |     """Create dictionary mapping keywords to original tokens.
102 |     """
103 |     keywords_dict = defaultdict(list)
104 |     for kw, t in zip(keywords, tokens):
105 |         keywords_dict[kw].append(t)
106 |     for kw, l in keywords_dict.items():
107 |         keywords_dict[kw] = list(set(l))
108 | 
109 |     keywords_dict = dict(keywords_dict)
110 |     
111 |     return keywords_dict
112 | 
113 | 
114 | def get_relevant_keywords(
115 |     text,
116 |     keywords_dict
117 | ):
118 |     """Returns relevant keywords based on semantic similarity to class embedding.
119 |     """
120 |     # obtain class embedding
121 |     textual_transformer = BertEncoder()
122 |     textual_features = textual_transformer.transform(text)
123 |     class_embedding = np.mean(np.array(textual_features), axis=0)
124 |     
125 |     # obtain relevant keywords
126 |     unique_keywords = list(keywords_dict.keys())
127 |     topn_relevant_keywords = params['topn_relevant_keywords']
128 |     relevant_keywords, simMat = relevant_keywords_helper(unique_keywords,
129 |                                                          class_embedding,
130 |                                                          topn_relevant_keywords)
131 |     return relevant_keywords, simMat
132 | 
133 | 
134 | def relevant_keywords_helper(
135 |     keywords,
136 |     class_embedding,
137 |     topn_relevant_keywords
138 | ):
139 |     """Helper function for obtaining embedding similarity.
140 |     """
141 |     textual_transformer = BertEncoder()
142 |     keyword_embeddings = textual_transformer.transform(keywords)
143 | 
144 |     simMatrix = np.dot(keyword_embeddings, class_embedding.T)
145 |     d = {"keyword" : keywords, "sim" : list(simMatrix)}
146 |     df_simMatrix = pd.DataFrame(d).sort_values(by="sim", ascending=False).reset_index(drop=True)
147 |     relevant_keywords = list(df_simMatrix['keyword'])[:topn_relevant_keywords]
148 | 
149 |     return relevant_keywords, df_simMatrix
150 | 
151 | 
152 | def prep_ablation(df):
153 |     """Prepare data for making predictions."""
154 |     
155 |     # convert df to list of dicts
156 |     data = df.to_json(orient="records")
157 |     data = json.loads(data)
158 |     
159 |     # load model assets from training job
160 |     model_assets = train.get_train_assets()
161 |     #print('extracting features')
162 |     numerical_features, categorical_features, textual_features = preprocess.extract_features(
163 |         data,
164 |         model_assets['numerical_feature_names'],
165 |         model_assets['categorical_feature_names'],
166 |         model_assets['textual_feature_names']
167 |     )
168 | 
169 |     # extract labels
170 |     _, _, _, label_name = preprocess.get_feature_names(df)
171 |     labels = preprocess.extract_labels(
172 |         data,
173 |         label_name
174 |     )
175 | 
176 |     # preprocess the data
177 |     #print('transforming numerical_features')
178 |     numerical_features = model_assets['numerical_transformer'].transform(numerical_features)
179 |     #print('transforming categorical_features')
180 |     categorical_features = model_assets['categorical_transformer'].transform(categorical_features)
181 |     #print('transforming textual_features')
182 |     textual_features = model_assets['textual_transformer'].transform(textual_features)
183 | 
184 |     #print('concatenating features')
185 |     categorical_features = categorical_features.toarray()
186 |     textual_features = np.array(textual_features)
187 |     textual_features = textual_features.reshape(textual_features.shape[0], -1)
188 |     features = np.concatenate([
189 |         numerical_features,
190 |         categorical_features,
191 |         textual_features
192 |     ], axis=1)
193 | 
194 |     return features, labels
195 | 
196 | 
197 | def perform_ablation(
198 |     df,
199 |     keywords,
200 |     keywords_dict
201 | ):
202 |     """Predict w/ and w/o keywords ablated.
203 |     """
204 |     # select subset of relevant keywords
205 |     frac_relevant_keywords = params['frac_relevant_keywords']
206 |     topn = int(params['topn_relevant_keywords'] * frac_relevant_keywords)
207 |     keywords_select = keywords[:topn]
208 |     keywords_select_dict = {}
209 |     for kw in keywords_select:
210 |         keywords_select_dict[kw] = keywords_dict[kw]
211 | 
212 |     # loop through keywords and perform ablation analysis
213 |     results_dict = {}
214 |     for keyword, keywords_list in tqdm(keywords_select_dict.items()):
215 |         # get portion of df where keywords occur
216 |         df_incl = pd.DataFrame()
217 |         df_excl = pd.DataFrame()
218 |         for i, row in df.iterrows():
219 |             for kw in keywords_list:
220 |                 if kw in row['chat_log']:
221 |                     # df including keyword in chat
222 |                     df_incl = df_incl.append(row)
223 |                     # df excluding keyword in chat
224 |                     chat_wkw = row['chat_log'].replace(kw, ' ')
225 |                     row['chat_log'] = chat_wkw
226 |                     df_excl = df_excl.append(row)
227 | 
228 |         # prep data incl/excl keyword in text (loads trained preprocessors)
229 |         features_incl, labels_incl = prep_ablation(
230 |             df_incl
231 |         )
232 | 
233 |         features_excl, labels_excl = prep_ablation(
234 |             df_excl
235 |         )
236 | 
237 |         # predict using previously trained model
238 |         pred_incl = train.predict(features_incl, labels_incl)
239 |         pred_excl = train.predict(features_excl, labels_excl)
240 | 
241 |         # save results
242 |         results_dict[keyword] = {'incl' : np.average(pred_incl),
243 |                                  'excl' : np.average(pred_excl),
244 |                                  'chg' : np.average(pred_excl) - np.average(pred_incl),
245 |                                  'count' : len(pred_incl)}
246 | 
247 |     # store results
248 |     results_df = pd.DataFrame.from_dict(results_dict, orient='index')
249 |     results_df = results_df.sort_values(by=["chg", "count"], ascending=(True, False))
250 |     results_df.to_csv(Path(model_dir, "ablation_results.csv"))
251 |     
252 |     return results_df
253 | 
254 | 
255 | def get_important_keywords(
256 |     simMat_df,
257 |     marg_contr_df
258 | ):
259 |     # merge dataframes
260 |     marg_contr_df.insert(0, 'keyword', marg_contr_df.index)
261 |     marg_contr_df = marg_contr_df.drop(['incl', 'excl'], axis=1)
262 |     results_df = marg_contr_df.merge(simMat_df, how='left', on='keyword')
263 | 
264 |     # rescale metrics
265 |     temp_df = results_df[['chg','count','sim']].copy()
266 |     temp_df['chg'] = temp_df['chg'] * (-1)
267 |     temp_df['count'] = np.log(temp_df['count']) # taking log transformation on keyword counts due to extreme outliers
268 |     min_max_scaler = preprocessing.MinMaxScaler(feature_range=(0, 1))
269 |     temp_df = min_max_scaler.fit_transform(temp_df)
270 |     temp_df = pd.DataFrame(temp_df, columns=['chg','count','sim'])
271 |     
272 |     # calculate weighted average
273 |     w_marg_contr = params['w_marg_contr']
274 |     w_count = params['w_count']
275 |     w_sim = params['w_sim']
276 |     temp_df['joint'] = temp_df['chg'] * w_marg_contr + temp_df['count'] * w_count + temp_df['sim'] * w_sim
277 |     
278 |     results_df['joint'] = temp_df['joint']
279 |     results_df = results_df.sort_values(by="joint", ascending=False).reset_index(drop=True)
280 |     results_df = results_df[['keyword','sim','chg','count','joint']]
281 |     
282 |     # store results
283 |     results_df.to_csv(Path(model_dir, "important_keywords.csv"))
284 |     
285 |     return results_df
286 | 
287 | 
288 | def obtain_context(
289 |     chats_list,
290 |     keyword,
291 |     limit=3
292 | ):
293 |     """Prints out limited number of chats where keyword occurs.
294 |     """
295 |     nlp = spacy.load('en_core_web_sm')
296 |     counter = 0
297 |     for chat in chats_list:
298 |         doc = nlp(chat)
299 |         for sent in doc.sents:
300 |             if keyword in sent.text and counter < limit:
301 |                 print(sent.text)
302 |                 print('\n')
303 |                 counter+=1
304 |         if counter >= limit:
305 |             break
306 | 
307 |     return None
308 | 
309 | 
310 | def train_xgb():
311 |     """Train XGBoost model to explain categorical and numerical feature importance.
312 |     """
313 |     # load one-hot feature names
314 |     filepath = Path(model_dir, "one_hot_feature_names.json")
315 |     numerical_feature_names, categorical_feature_names, _ = preprocess.load_feature_names(filepath)
316 |     one_hot_feature_names = numerical_feature_names + categorical_feature_names
317 |     
318 |     # load train data and exclude text embeddings
319 |     train = pd.read_csv(Path(model_dir, "train.csv"))
320 |     train_notext = train.iloc[:, :len(one_hot_feature_names)]
321 |     labels = pd.read_csv(Path(model_dir, "labels.csv"))
322 |     test = pd.read_csv(Path(model_dir, "test.csv"))
323 |     test_notext = test.iloc[:, :len(one_hot_feature_names)]
324 |     labels_test = pd.read_csv(Path(model_dir, "labels_test.csv"))
325 | 
326 |     # train XGBoost model
327 |     xgb = XGBClassifier()
328 |     xgb.fit(train_notext, labels)
329 |     
330 |     y_pred = xgb.predict_proba(test_notext)
331 |     y_pred = [p[1] for p in y_pred]
332 | 
333 |     # plot important features
334 |     topn = 10
335 |     sorted_idx = xgb.feature_importances_.argsort()
336 |     plt.barh(np.array(one_hot_feature_names)[sorted_idx[-topn:]],
337 |              xgb.feature_importances_[sorted_idx[-topn:]])
338 |     plt.title("Xgboost Feature Importance")
339 |     plt.show()
340 | 
341 |     return y_pred
342 | 
343 | 
344 | if __name__ == "__main__":
345 |     parser = argparse.ArgumentParser()
346 |     parser.add_argument("--churn", type=int, default=1)
347 |     parser.add_argument("--speaker", type=str, default='Customer')
348 |     args = parser.parse_args()
349 | 
350 |     data_dir = params['data_dir'] 
351 |     df = pd.read_csv(Path(data_dir, "churn_dataset.csv"))
352 | 
353 |     churn = args.churn
354 |     speaker = args.speaker
355 |     chats, df_sub = get_chats(df, churn, speaker)
356 |     keywords, tokens = get_keywords(chats)
357 |     keywords_dict = map_to_orig_tok(keywords, tokens)
358 | 
359 |     relevant_keywords, simMat_df = get_relevant_keywords(chats, keywords_dict)
360 |     marg_contr_df = perform_ablation(df_sub, relevant_keywords, keywords_dict)
361 | 
362 |     get_important_keywords(simMat_df, marg_contr_df)
363 | 


--------------------------------------------------------------------------------
/scripts/preprocess.py:
--------------------------------------------------------------------------------
  1 | """ Module for preparing the data for the churn prediction model with text. 
  2 | 
  3 | Run in CLI example:
  4 |     'python preprocess.py --test-size 0.33'
  5 | 
  6 | """
  7 | 
  8 | 
  9 | import os
 10 | import sys
 11 | import json
 12 | import yaml
 13 | import joblib
 14 | import logging
 15 | import argparse
 16 | import numpy as np
 17 | import pandas as pd
 18 | from pathlib import Path
 19 | from matplotlib import pyplot as plt
 20 | from sklearn.model_selection import train_test_split
 21 | from sklearn.impute import SimpleImputer
 22 | from sklearn.preprocessing import OneHotEncoder
 23 | from sklearn.base import BaseEstimator, TransformerMixin
 24 | from sentence_transformers import SentenceTransformer
 25 | 
 26 | 
 27 | with open("../model/params.yaml", "r") as params_file:
 28 |     params = yaml.safe_load(params_file)
 29 | 
 30 | model_dir = params['model_dir']
 31 | 
 32 | 
 33 | class BertEncoder(BaseEstimator, TransformerMixin):
 34 |     def __init__(self, model_name='bert-base-nli-mean-tokens'):
 35 |         self.model = SentenceTransformer(model_name)
 36 |         self.model.parallel_tokenization = False
 37 | 
 38 |     def fit(self, X, y=None):
 39 |         return self
 40 | 
 41 |     def transform(self, X):
 42 |         output = []
 43 |         for sample in X:
 44 |             encodings = self.model.encode(sample)
 45 |             output.append(encodings)
 46 |         return output
 47 | 
 48 | 
 49 | def extract_labels(
 50 |     data,
 51 |     label_name
 52 | ):
 53 |     labels = []
 54 |     for sample in data:
 55 |         value = sample[label_name]
 56 |         labels.append(value)
 57 |     labels = np.array(labels).astype('int')
 58 | 
 59 |     return labels.reshape(labels.shape[0],-1)
 60 | 
 61 | 
 62 | def convert_label(
 63 |     df
 64 | ):
 65 |     df.churn = df.churn.replace("no", 0)
 66 |     df.churn = df.churn.replace("yes", 1)
 67 |     return df
 68 | 
 69 | def extract_numerical_features(
 70 |     sample,
 71 |     numerical_feature_names
 72 | ):
 73 |     output = []
 74 |     for feature_name in numerical_feature_names:
 75 |         if feature_name in sample.keys():
 76 |             value = sample[feature_name]
 77 |             if value is None:
 78 |                 value = np.nan
 79 |         else:
 80 |             value = np.nan
 81 |         output.append(value)
 82 |     return output
 83 | 
 84 | 
 85 | def extract_categorical_features(
 86 |     sample,
 87 |     categorical_feature_names
 88 | ):
 89 |     output = []
 90 |     for feature_name in categorical_feature_names:
 91 |         if feature_name in sample.keys():
 92 |             value = sample[feature_name]
 93 |             if value is None:
 94 |                 value = ""
 95 |         else:
 96 |             value = ""
 97 |         output.append(value)
 98 |     return output
 99 | 
100 | 
101 | def extract_textual_features(
102 |     sample,
103 |     textual_feature_names
104 | ):
105 |     output = []
106 |     for feature_name in textual_feature_names:
107 |         if feature_name in sample.keys():
108 |             value = sample[feature_name]
109 |             if value is None:
110 |                 value = ""
111 |         else:
112 |             value = ""
113 |         output.append(value)
114 |     return output
115 | 
116 | 
117 | def split_data(
118 |     df,
119 |     label_name,
120 |     test_size
121 | ):
122 |     """Splits data and creates json format.
123 |     """
124 |     X = df.drop(columns=[label_name], axis=1)
125 |     y = df[label_name]
126 |     X_train, X_test, y_train, y_test = train_test_split(
127 |         X, y, test_size=test_size, random_state=123, stratify=y)
128 |     
129 |     train = pd.DataFrame(X_train, columns = X.columns)
130 |     train[label_name] = y_train
131 |     test = pd.DataFrame(X_test, columns = X.columns)
132 |     test[label_name] = y_test
133 |     
134 |     # create list of dicts
135 |     train = train.to_json(orient="records")
136 |     train = json.loads(train)
137 |     test = test.to_json(orient="records")
138 |     test = json.loads(test)
139 |     
140 |     return train, test
141 | 
142 | 
143 | def extract_features(
144 |     data,
145 |     numerical_feature_names,
146 |     categorical_feature_names,
147 |     textual_feature_names
148 | ):
149 |     """extract features by given feature names.
150 |     """
151 |     numerical_features = []
152 |     categorical_features = []
153 |     textual_features = []
154 |     for sample in data:
155 |         num_feat = extract_numerical_features(sample, numerical_feature_names)
156 |         numerical_features.append(num_feat)
157 |         cat_feat = extract_categorical_features(sample, categorical_feature_names)
158 |         categorical_features.append(cat_feat)
159 |         text_feat = extract_textual_features(sample, textual_feature_names)
160 |         textual_features.append(text_feat)
161 |         
162 |     textual_features = [i if isinstance(i[0], str) else ["nan"] for i in textual_features]
163 |     textual_features = [i[0] for i in textual_features]
164 | 
165 |     return numerical_features, categorical_features, textual_features
166 | 
167 | 
168 | def save_feature_names(
169 |     numerical_feature_names,
170 |     categorical_feature_names,
171 |     textual_feature_names,
172 |     filepath
173 | ):
174 |     feature_names = {
175 |         'numerical': numerical_feature_names,
176 |         'categorical': categorical_feature_names,
177 |         'textual': textual_feature_names
178 |     }
179 |     with open(filepath, 'w') as f:
180 |         json.dump(feature_names, f)
181 | 
182 | 
183 | def load_feature_names(filepath):
184 |     with open(filepath, 'r') as f:
185 |         feature_names = json.load(f)
186 |     numerical_feature_names = feature_names['numerical']
187 |     categorical_feature_names = feature_names['categorical']
188 |     textual_feature_names = feature_names['textual']
189 |     return numerical_feature_names, categorical_feature_names, textual_feature_names
190 | 
191 | 
192 | def get_feature_names(
193 |     df
194 | ):
195 |     num_columns = df.select_dtypes(include=np.number).columns.tolist()
196 |     numerical_feature_names = [i for i in num_columns if i not in ['churn']]
197 | 
198 |     cat_columns = df.select_dtypes(include='object').columns.tolist()
199 |     categorical_feature_names = [i for i in cat_columns if i not in ['chat_log']]
200 | 
201 |     textual_feature_names = ['chat_log']
202 |     label_name = 'churn'
203 | 
204 |     return numerical_feature_names, categorical_feature_names, textual_feature_names, label_name
205 |     
206 | 
207 | def prep_data(
208 |     df, use_existing=False, test_size=0.33
209 | ):  
210 |     """
211 |     Args:
212 |         df: Pandas dataframe with raw data
213 |         use_existing: Set to True if you want to use locally stored, 
214 |         already prepared train/test data. Set to False if you want 
215 |         to rerun the data preparation pipeline.
216 |     Returns:
217 |         Train and test data as well as train labels and test labels.
218 |     """
219 |     # if prepared data exists, don't prepare again if use_existing set to True
220 |     train_file = Path(model_dir, 'train.csv')
221 |     labels_file = Path(model_dir, 'labels.csv')
222 |     test_file = Path(model_dir, 'test.csv')
223 |     labels_test_file = Path(model_dir, 'labels_test.csv')
224 |     feature_names_file = Path(model_dir, "feature_names.json")
225 |     oh_feature_names_file = Path(model_dir, "one_hot_feature_names.json")
226 |     all_file_paths = [train_file, labels_file, test_file, labels_test_file,
227 |                      feature_names_file, oh_feature_names_file]
228 |     
229 |     if use_existing == True and all(file.exists() for file in all_file_paths):
230 |         features = np.array(pd.read_csv('../model/train.csv'))
231 |         labels = np.array(pd.read_csv('../model/labels.csv'))
232 |         features_test = np.array(pd.read_csv('../model/test.csv'))
233 |         labels_test = np.array(pd.read_csv('../model/labels_test.csv'))
234 |         print("Using already prepared data.")
235 | 
236 |     else:
237 |         print("Running data preparation pipeline...")
238 |         # convert label to binary numeric
239 |         df = convert_label(df)
240 | 
241 |         # extract feature names
242 |         numerical_feature_names, categorical_feature_names, textual_feature_names, label_name = get_feature_names(
243 |             df
244 |         )
245 |         # train/test split and convert to json format (list of dicts)
246 |         train, test = split_data(
247 |             df,
248 |             label_name,
249 |             test_size
250 |         )
251 |         # extract features & label
252 |         print('extracting features')
253 |         numerical_features, categorical_features, textual_features = extract_features(
254 |             train,
255 |             numerical_feature_names,
256 |             categorical_feature_names,
257 |             textual_feature_names
258 |         )
259 |         labels = extract_labels(
260 |             train,
261 |             label_name
262 |         )
263 |         # extract features & label (for test data)
264 |         numerical_features_test, categorical_features_test, textual_features_test = extract_features(
265 |             test,
266 |             numerical_feature_names,
267 |             categorical_feature_names,
268 |             textual_feature_names
269 |         )
270 |         labels_test = extract_labels(
271 |             test,
272 |             label_name
273 |         )
274 |         # define preprocessors
275 |         print('defining preprocessors')
276 |         numerical_transformer = SimpleImputer(missing_values=np.nan, strategy='mean', add_indicator=True)
277 |         categorical_transformer = OneHotEncoder(handle_unknown="ignore")
278 |         textual_transformer = BertEncoder()
279 | 
280 |         # fit preprocessors
281 |         print('fitting numerical_transformer')
282 |         numerical_transformer.fit(numerical_features)
283 |         print('saving numerical_transformer')
284 |         joblib.dump(numerical_transformer, Path(model_dir, "numerical_transformer.joblib"))
285 |         print('fitting categorical_transformer')
286 |         categorical_transformer.fit(categorical_features)
287 |         print('saving categorical_transformer')
288 |         joblib.dump(categorical_transformer, Path(model_dir, "categorical_transformer.joblib"))
289 | 
290 |         # transform features
291 |         print('transforming numerical_features')
292 |         numerical_features = numerical_transformer.transform(numerical_features)
293 |         print('transforming categorical_features')
294 |         categorical_features = categorical_transformer.transform(categorical_features)
295 |         print('transforming textual_features')
296 |         textual_features = textual_transformer.transform(textual_features)
297 | 
298 |         # transform features (for test data)
299 |         print('transforming numerical_features_test')
300 |         numerical_features_test = numerical_transformer.transform(numerical_features_test)
301 |         print('transforming categorical_features_test')
302 |         categorical_features_test = categorical_transformer.transform(categorical_features_test)
303 |         print('transforming textual_features_test')
304 |         textual_features_test = textual_transformer.transform(textual_features_test)
305 | 
306 |         # concat features
307 |         print('concatenating features')
308 |         categorical_features = categorical_features.toarray()
309 |         textual_features = np.array(textual_features)
310 |         textual_features = textual_features.reshape(textual_features.shape[0], -1)
311 |         features = np.concatenate([
312 |             numerical_features,
313 |             categorical_features,
314 |             textual_features
315 |         ], axis=1)
316 | 
317 |         # concat features (test data)
318 |         print('concatenating features of test data')
319 |         categorical_features_test = categorical_features_test.toarray()
320 |         textual_features_test = np.array(textual_features_test)
321 |         textual_features_test = textual_features_test.reshape(textual_features_test.shape[0], -1)
322 |         features_test = np.concatenate([
323 |             numerical_features_test,
324 |             categorical_features_test,
325 |             textual_features_test
326 |         ], axis=1)
327 | 
328 |         # save to disk
329 |         pd.DataFrame(features).to_csv(Path(model_dir, "train.csv"), index=False)
330 |         pd.DataFrame(labels).to_csv(Path(model_dir, "labels.csv"), index=False)
331 |         pd.DataFrame(features_test).to_csv(Path(model_dir, "test.csv"), index=False)
332 |         pd.DataFrame(labels_test).to_csv(Path(model_dir, "labels_test.csv"), index=False)
333 | 
334 |         save_feature_names(
335 |             numerical_feature_names,
336 |             categorical_feature_names,
337 |             textual_feature_names,
338 |             Path(model_dir, "feature_names.json")
339 |         )
340 |         # one-hot encoded feature names (for feat_imp)
341 |         save_feature_names(
342 |             numerical_feature_names,
343 |             categorical_transformer.get_feature_names().tolist(),
344 |             textual_feature_names,
345 |             Path(model_dir, "one_hot_feature_names.json")
346 |         )
347 | 
348 |     return features, features_test, labels, labels_test
349 | 
350 | 
351 | if __name__ == "__main__":
352 |     parser = argparse.ArgumentParser()
353 |     parser.add_argument("--use-existing", action="store_true")
354 |     parser.add_argument("--test-size", type=float, default=0.33)
355 |     args = parser.parse_args()
356 |     
357 |     data_dir = params['data_dir']
358 |     df = pd.read_csv(Path(data_dir, "churn_dataset.csv"))
359 |     
360 |     if args.use_existing:
361 |         use_existing = True
362 |     else:
363 |         use_existing = False
364 |     test_size = args.test_size
365 |     
366 |     prep_data(df, use_existing, test_size)
367 | 


--------------------------------------------------------------------------------
/scripts/train.py:
--------------------------------------------------------------------------------
  1 | """ Module for training the churn prediction model with text. 
  2 | 
  3 | Training parameters are stored in 'params.yaml'.
  4 | 
  5 | Run in CLI example:
  6 |     'python train.py'
  7 | 
  8 | """
  9 | 
 10 | 
 11 | import os
 12 | import sys
 13 | import json
 14 | import yaml
 15 | import joblib
 16 | import logging
 17 | import argparse
 18 | import numpy as np
 19 | import pandas as pd
 20 | from pathlib import Path
 21 | from matplotlib import pyplot as plt
 22 | from sklearn.model_selection import train_test_split
 23 | from sklearn.impute import SimpleImputer
 24 | from sklearn.preprocessing import OneHotEncoder
 25 | from sklearn.metrics import roc_auc_score, roc_curve, auc
 26 | from sklearn.metrics import precision_recall_curve
 27 | import torch
 28 | import torch.nn as nn
 29 | import torch.nn.functional as F
 30 | import torch.optim as optim
 31 | from torch.utils.data import DataLoader, TensorDataset
 32 | from torch import Tensor
 33 | import preprocess
 34 | from preprocess import BertEncoder
 35 | 
 36 | 
 37 | logger = logging.getLogger(__name__)
 38 | logger.setLevel(logging.DEBUG)
 39 | logger.addHandler(logging.StreamHandler(sys.stdout))
 40 | 
 41 | 
 42 | with open("../model/params.yaml", "r") as params_file:
 43 |     params = yaml.safe_load(params_file)
 44 | 
 45 | model_dir = params['model_dir']
 46 | 
 47 |     
 48 | class Net(nn.Module):
 49 |     def __init__(self, x1_size, x2_size):
 50 |         super(Net, self).__init__()
 51 |         self.batch_norm = nn.BatchNorm1d(x1_size)
 52 |         self.fc1 = nn.Linear(x1_size, 10)
 53 |         self.fc2 = nn.Linear(10 + x2_size, 10)
 54 |         self.fc3 = nn.Linear(10, 1)
 55 | 
 56 |     def forward(self, x1, x2):
 57 |         x1 = self.batch_norm(x1)
 58 |         x1 = F.relu(self.fc1(x1))
 59 |         x1 = F.dropout(x1, p=0.2, training=self.training)
 60 |         x12 = torch.cat((x1.view(x1.size(0), -1),
 61 |                         x2.view(x2.size(0), -1)), dim=1)
 62 |         x12 = F.dropout(x12, p=0.1, training=self.training)
 63 |         x12 = self.fc2(x12)
 64 |         out = self.fc3(x12)
 65 |         return out
 66 | 
 67 | 
 68 | def train(
 69 |     X,
 70 |     y,
 71 |     X_test,
 72 |     y_test
 73 | ):
 74 |     # get parameters
 75 |     batch_size = params['batch_size']
 76 |     batch_size_test = params['batch_size_test']
 77 |     epochs = params['epochs']
 78 |     pos_weight = params['pos_weight']
 79 |     lr = params['lr']
 80 |     momentum = params['momentum']
 81 |     
 82 |     # prepare training job
 83 |     X = np.array(X)
 84 |     y = np.array(y)
 85 |     X_test = np.array(X_test)
 86 |     y_test = np.array(y_test)
 87 |     training_data = TensorDataset( Tensor(X), Tensor(y) )
 88 |     train_loader = DataLoader(training_data, batch_size=batch_size,
 89 |                                               shuffle=True,
 90 |                                               num_workers=4)
 91 |     test_data = TensorDataset( Tensor(X_test), Tensor(y_test) )
 92 |     test_loader = DataLoader(test_data, batch_size=batch_size_test,
 93 |                                               shuffle=True,
 94 |                                               num_workers=4)
 95 |     
 96 |     # get size of num/cat & text data
 97 |     numerical_feature_names, categorical_feature_names, _ = preprocess.load_feature_names(Path(model_dir, "one_hot_feature_names.json"))
 98 |     number_cat_num_features = len(numerical_feature_names) + len(categorical_feature_names)
 99 |     x1_size = X[:, :number_cat_num_features].shape[1]
100 |     x2_size = X[:, number_cat_num_features:].shape[1]
101 | 
102 |     model = Net(x1_size, x2_size)
103 |     criterion = nn.BCEWithLogitsLoss(pos_weight=torch.as_tensor(pos_weight, dtype=torch.float))
104 |     #criterion = nn.BCELoss(pos_weight=torch.as_tensor(pos_weight, dtype=torch.float))
105 |     optimizer = optim.SGD(model.parameters(), lr=lr, momentum=momentum)
106 |     
107 |     # train NN model
108 |     scores_df = pd.DataFrame()
109 |     train_scores = []
110 |     test_scores = []
111 |     for epoch in range(1, epochs + 1):  # loop over the dataset multiple times
112 |         model.train()
113 |         print("starting epoch: ", epoch)
114 |         
115 |         for batch_idx, (data, target) in enumerate(train_loader, 1):
116 |             # zero the parameter gradients
117 |             optimizer.zero_grad()
118 | 
119 |             # split data inputs into cat/num features and text features
120 |             x1 = data[:, :number_cat_num_features]
121 |             x2 = data[:, number_cat_num_features:]
122 |             
123 |             # forward + backward + optimize
124 |             output = model(x1, x2)
125 |             loss = criterion(output, target) #target.type_as(output)) #labels_batch.long())
126 |             loss.backward()
127 |             optimizer.step()
128 | 
129 |         # training performance after each epoch
130 |         with torch.no_grad():
131 |             output = model(Tensor(X[:, :number_cat_num_features]), Tensor(X[:, number_cat_num_features:])).reshape(-1)
132 |             preds = torch.sigmoid(output)
133 |             train_score = roc_auc_score(Tensor(y), preds)
134 |             train_scores.append(train_score)
135 |             logger.info('Train Epoch: {}, train-auc-score: {:.4f}'.format(epoch, train_score))
136 | 
137 |         # test performance after each epoch
138 |         test_score = test(model, test_loader)
139 |         test_scores.append(test_score)
140 |     
141 |     # save scores
142 |     print('saving scores')
143 |     scores_df['train_scores'] = train_scores
144 |     scores_df['test_scores'] = test_scores
145 |     scores_df.to_csv(Path(model_dir, 'training_scores.csv'), index=False)
146 |     
147 |     # save model
148 |     print('saving model')
149 |     torch.save(model.state_dict(), Path(model_dir, 'model.pth'))
150 | 
151 |     return None
152 | 
153 | 
154 | def test(
155 |     model,
156 |     test_loader
157 | ):
158 |     model.eval()
159 |     correct = 0
160 |     preds_all = []
161 |     targets_all = []
162 |     with torch.no_grad():
163 |         for data, targets in test_loader:
164 |             
165 |             # split data inputs
166 |             numerical_feature_names, categorical_feature_names, _ = preprocess.load_feature_names(Path(model_dir, "one_hot_feature_names.json"))
167 |             number_cat_num_features = len(numerical_feature_names) + len(categorical_feature_names)
168 |             x1 = data[:, :number_cat_num_features]
169 |             x2 = data[:, number_cat_num_features:]
170 |             
171 |             output = model(x1, x2).reshape(-1)
172 |             preds = torch.sigmoid(output)
173 |             preds_all.extend(preds)
174 |             targets_all.extend(targets)
175 |         test_score = roc_auc_score(Tensor(targets_all), Tensor(preds_all))
176 | 
177 |     logger.info('test_auc_score: {:.4f}'.format(test_score))
178 | 
179 |     return test_score
180 | 
181 | 
182 | def get_train_assets(
183 | ):
184 |     #print('loading feature_names')
185 |     numerical_feature_names, categorical_feature_names, textual_feature_names = preprocess.load_feature_names(Path(model_dir, "feature_names.json"))
186 |     #print('loading numerical_transformer')
187 |     numerical_transformer = joblib.load(Path(model_dir, "numerical_transformer.joblib"))
188 |     #print('loading categorical_transformer')
189 |     categorical_transformer = joblib.load(Path(model_dir, "categorical_transformer.joblib"))
190 |     #print('loading textual_transformer')
191 |     textual_transformer = BertEncoder()
192 | 
193 |     model_assets = {
194 |         'numerical_feature_names': numerical_feature_names,
195 |         'numerical_transformer': numerical_transformer,
196 |         'categorical_feature_names': categorical_feature_names,
197 |         'categorical_transformer': categorical_transformer,
198 |         'textual_feature_names': textual_feature_names,
199 |         'textual_transformer': textual_transformer
200 |     }
201 |     return model_assets
202 | 
203 | 
204 | def predict(
205 |     features,
206 |     labels
207 | ):
208 |     # get size of num/cat & text data to specify neural net
209 |     numerical_feature_names, categorical_feature_names, _ = preprocess.load_feature_names(Path(model_dir, "one_hot_feature_names.json"))
210 |     number_cat_num_features = len(numerical_feature_names) + len(categorical_feature_names)
211 |     x1_size = features[:, :number_cat_num_features].shape[1]
212 |     x2_size = features[:, number_cat_num_features:].shape[1]
213 |     
214 |     model = Net(x1_size, x2_size)
215 |     with open(os.path.join(model_dir, 'model.pth'), 'rb') as f:
216 |         model.load_state_dict(torch.load(f))
217 | 
218 |     model.eval()
219 |     with torch.no_grad():
220 |         output = model(Tensor(features[:, :number_cat_num_features]), Tensor(features[:, number_cat_num_features:])).reshape(-1)
221 |         preds = torch.sigmoid(output)
222 | 
223 |     return preds
224 | 
225 | 
226 | def plot_train_stats(
227 | ):
228 |     scores_df = pd.read_csv(Path(model_dir, 'scores.csv'))
229 |     scores_df.plot(xlabel='Epochs', ylabel='AUC Score', title='AUC Score')
230 |     return None
231 | 
232 | 
233 | def plot_pr_curve(
234 |     features,
235 |     labels
236 | ):
237 |     preds = predict(features, labels)
238 |     precisions, recalls, thresholds = precision_recall_curve(labels, preds)
239 |     roc_auc = roc_auc_score(labels, preds)
240 | 
241 |     fig, ax = plt.subplots(figsize=(6,5))
242 |     ax.plot(recalls, precisions, label='Model w/ text: AUC = %0.2f' % roc_auc)
243 |     ax.plot([1, 0], [0, 1],'--')
244 |     ax.set_xlabel('Recall')
245 |     ax.set_ylabel('Precision')
246 |     ax.legend(loc='lower left')
247 |     plt.title('Precision-Recall Curve')
248 |     plt.show()
249 |     return None
250 | 
251 | 
252 | def plot_roc_curve(
253 |     features,
254 |     labels
255 | ):
256 |     preds = predict(features, labels)
257 |     fpr, tpr, threshold = roc_curve(labels, preds)
258 |     roc_auc = auc(fpr, tpr)
259 |     
260 |     plt.title('ROC Curve')
261 |     plt.plot(fpr, tpr, 'b', label='AUC = %0.2f' % roc_auc)
262 |     plt.legend(loc = 'lower right')
263 |     plt.plot([0, 1], [0, 1],'r--')
264 |     plt.xlim([0, 1])
265 |     plt.ylim([0, 1])
266 |     plt.ylabel('True Positive Rate')
267 |     plt.xlabel('False Positive Rate')
268 |     plt.show()
269 |     return None
270 | 
271 | 
272 | if __name__ == "__main__":
273 | 
274 |     model_dir = params['model_dir']
275 | 
276 |     X_train = pd.read_csv(Path(model_dir, "train.csv"))
277 |     y_train = pd.read_csv(Path(model_dir, "labels.csv"))
278 |     X_test = pd.read_csv(Path(model_dir, "test.csv"))
279 |     y_test = pd.read_csv(Path(model_dir, "labels_test.csv"))
280 | 
281 |     train(X=X_train, y=y_train, X_test=X_test, y_test=y_test)
282 | 


--------------------------------------------------------------------------------