├── 1. ML workflow.ipynb
├── 2. Linear Regression.ipynb
├── 3. Regression accuracy metrics.ipynb
├── 4. Logistic Regression.ipynb
├── 5. Classification: Accuracy Metrics.ipynb
├── 6. Decision Tree.ipynb
├── 7. Cross-validation and Grid Search.ipynb
├── 8. K-means Clustering.ipynb
├── Churn.csv
├── README.md
├── bmw.csv
└── processed.cleveland.data


/2. Linear Regression.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# Expedition to Data Science and Machine Learning\n",
  8 |     "## Module 4: Machine Learning with Python\n",
  9 |     "### Lecture 2: Supervised Learning: Linear Regression\n",
 10 |     "\n",
 11 |     "Instructor: Md Shahidullah Kawsar\n",
 12 |     "<br>Data Scientist, IDARE, Houston, TX, USA\n",
 13 |     "\n",
 14 |     "#### Objectives:\n",
 15 |     "- Supervised Learning: Linear Regression\n",
 16 |     "- train data, test data\n",
 17 |     "- Understanding the equation of a straight line\n",
 18 |     "- feature coefficient (slope, gradient, m)\n",
 19 |     "- bias coeffcient (y-interccept, c)\n",
 20 |     "- domain: x-axis, independent variable\n",
 21 |     "- range: y-axis, dependent variable\n",
 22 |     "- loss function, cost function, objective function, error function\n",
 23 |     "- bias-variance tradeoff, overfitting, underfitting\n",
 24 |     "- ordinary least square method\n",
 25 |     "- gradient descent method\n",
 26 |     "- residual, error, squared error, RMSE - Root Mean Squared Error\n",
 27 |     "\n",
 28 |     "#### References:\n",
 29 |     "[1] A Gentle Introduction to Machine Learning: https://www.youtube.com/watch?v=Gv9_4yMHFhI&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF&ab_channel=StatQuestwithJoshStarmer\n",
 30 |     "<br>[2] Linear Regression, Clearly Explained!!!: https://www.youtube.com/watch?v=nk2CQITm_eo&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF&index=10&ab_channel=StatQuestwithJoshStarmer\n",
 31 |     "<br>[3] Linear Regression scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html\n",
 32 |     "<br>[4] Data Splitting: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html\n",
 33 |     "<br>[5] Mean Squared Error: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html\n",
 34 |     "<br>[6] RMSE calculation: https://www.youtube.com/watch?v=zMFdb__sUpw&ab_channel=KhanAcademy\n",
 35 |     "<br>[7] Regression coefficients: https://statisticsbyjim.com/glossary/regression-coefficient/\n",
 36 |     "<br>[8] Machine Learning Quiz 01: Linear Regression https://kawsar34.medium.com/machine-learning-quiz-01-a2fac2712a55\n",
 37 |     "<br>[9] Linear Regression Assumptions: https://www.statology.org/linear-regression-assumptions/\n",
 38 |     "<br>[10] Constant Variance: https://stats.stackexchange.com/questions/52089/what-does-having-constant-variance-in-a-linear-regression-model-mean\n",
 39 |     "<br>[11] Multiple Regression: https://www.youtube.com/watch?v=zITIFTsivN8&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF&index=11&ab_channel=StatQuestwithJoshStarmer\n",
 40 |     "<br>[12] Linear Regression Simplified - Ordinary Least Square vs Gradient Descent: https://towardsdatascience.com/linear-regression-simplified-ordinary-least-square-vs-gradient-descent-48145de2cf76"
 41 |    ]
 42 |   },
 43 |   {
 44 |    "cell_type": "markdown",
 45 |    "metadata": {},
 46 |    "source": [
 47 |     "#### Terminologies:\n",
 48 |     "\n",
 49 |     "- equation of a straight line: y=mx+c\n",
 50 |     "<br> Straight lines: https://github.com/SKawsar/Data_Visualization_with_Python/blob/main/Lecture_4.ipynb\n",
 51 |     "- feature coefficient (slope, gradient, m)\n",
 52 |     "- bias coeffcient (y-interccept, c)\n",
 53 |     "- domain: x-axis, independent variable\n",
 54 |     "- range: y-axis, dependent variable\n",
 55 |     "- loss function, cost function, objective function, error function\n",
 56 |     "- bias-variance tradeoff, overfitting, underfitting\n",
 57 |     "- ordinary least square method\n",
 58 |     "- gradient descent method\n",
 59 |     "- residual, error, squared error\n",
 60 |     "- train data, test data\n"
 61 |    ]
 62 |   },
 63 |   {
 64 |    "cell_type": "markdown",
 65 |    "metadata": {},
 66 |    "source": [
 67 |     "#### Import required Libraries"
 68 |    ]
 69 |   },
 70 |   {
 71 |    "cell_type": "code",
 72 |    "execution_count": 108,
 73 |    "metadata": {},
 74 |    "outputs": [],
 75 |    "source": [
 76 |     "import pandas as pd\n",
 77 |     "import numpy as np\n",
 78 |     "\n",
 79 |     "from sklearn.model_selection import train_test_split\n",
 80 |     "from sklearn.linear_model import LinearRegression\n",
 81 |     "from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score"
 82 |    ]
 83 |   },
 84 |   {
 85 |    "cell_type": "markdown",
 86 |    "metadata": {},
 87 |    "source": [
 88 |     "#### Load data"
 89 |    ]
 90 |   },
 91 |   {
 92 |    "cell_type": "code",
 93 |    "execution_count": 109,
 94 |    "metadata": {},
 95 |    "outputs": [
 96 |     {
 97 |      "data": {
 98 |       "text/html": [
 99 |        "<div>\n",
100 |        "<style scoped>\n",
101 |        "    .dataframe tbody tr th:only-of-type {\n",
102 |        "        vertical-align: middle;\n",
103 |        "    }\n",
104 |        "\n",
105 |        "    .dataframe tbody tr th {\n",
106 |        "        vertical-align: top;\n",
107 |        "    }\n",
108 |        "\n",
109 |        "    .dataframe thead th {\n",
110 |        "        text-align: right;\n",
111 |        "    }\n",
112 |        "</style>\n",
113 |        "<table border=\"1\" class=\"dataframe\">\n",
114 |        "  <thead>\n",
115 |        "    <tr style=\"text-align: right;\">\n",
116 |        "      <th></th>\n",
117 |        "      <th>model</th>\n",
118 |        "      <th>year</th>\n",
119 |        "      <th>price</th>\n",
120 |        "      <th>transmission</th>\n",
121 |        "      <th>mileage</th>\n",
122 |        "      <th>fuelType</th>\n",
123 |        "      <th>mpg</th>\n",
124 |        "      <th>engineSize</th>\n",
125 |        "    </tr>\n",
126 |        "  </thead>\n",
127 |        "  <tbody>\n",
128 |        "    <tr>\n",
129 |        "      <th>0</th>\n",
130 |        "      <td>T-Roc</td>\n",
131 |        "      <td>2019</td>\n",
132 |        "      <td>25000</td>\n",
133 |        "      <td>Automatic</td>\n",
134 |        "      <td>13904</td>\n",
135 |        "      <td>Diesel</td>\n",
136 |        "      <td>49.6</td>\n",
137 |        "      <td>2.0</td>\n",
138 |        "    </tr>\n",
139 |        "    <tr>\n",
140 |        "      <th>1</th>\n",
141 |        "      <td>T-Roc</td>\n",
142 |        "      <td>2019</td>\n",
143 |        "      <td>26883</td>\n",
144 |        "      <td>Automatic</td>\n",
145 |        "      <td>4562</td>\n",
146 |        "      <td>Diesel</td>\n",
147 |        "      <td>49.6</td>\n",
148 |        "      <td>2.0</td>\n",
149 |        "    </tr>\n",
150 |        "    <tr>\n",
151 |        "      <th>2</th>\n",
152 |        "      <td>T-Roc</td>\n",
153 |        "      <td>2019</td>\n",
154 |        "      <td>20000</td>\n",
155 |        "      <td>Manual</td>\n",
156 |        "      <td>7414</td>\n",
157 |        "      <td>Diesel</td>\n",
158 |        "      <td>50.4</td>\n",
159 |        "      <td>2.0</td>\n",
160 |        "    </tr>\n",
161 |        "    <tr>\n",
162 |        "      <th>3</th>\n",
163 |        "      <td>T-Roc</td>\n",
164 |        "      <td>2019</td>\n",
165 |        "      <td>33492</td>\n",
166 |        "      <td>Automatic</td>\n",
167 |        "      <td>4825</td>\n",
168 |        "      <td>Petrol</td>\n",
169 |        "      <td>32.5</td>\n",
170 |        "      <td>2.0</td>\n",
171 |        "    </tr>\n",
172 |        "    <tr>\n",
173 |        "      <th>4</th>\n",
174 |        "      <td>T-Roc</td>\n",
175 |        "      <td>2019</td>\n",
176 |        "      <td>22900</td>\n",
177 |        "      <td>Semi-Auto</td>\n",
178 |        "      <td>6500</td>\n",
179 |        "      <td>Petrol</td>\n",
180 |        "      <td>39.8</td>\n",
181 |        "      <td>1.5</td>\n",
182 |        "    </tr>\n",
183 |        "  </tbody>\n",
184 |        "</table>\n",
185 |        "</div>"
186 |       ],
187 |       "text/plain": [
188 |        "    model  year  price transmission  mileage fuelType   mpg  engineSize\n",
189 |        "0   T-Roc  2019  25000    Automatic    13904   Diesel  49.6         2.0\n",
190 |        "1   T-Roc  2019  26883    Automatic     4562   Diesel  49.6         2.0\n",
191 |        "2   T-Roc  2019  20000       Manual     7414   Diesel  50.4         2.0\n",
192 |        "3   T-Roc  2019  33492    Automatic     4825   Petrol  32.5         2.0\n",
193 |        "4   T-Roc  2019  22900    Semi-Auto     6500   Petrol  39.8         1.5"
194 |       ]
195 |      },
196 |      "metadata": {},
197 |      "output_type": "display_data"
198 |     },
199 |     {
200 |      "name": "stdout",
201 |      "output_type": "stream",
202 |      "text": [
203 |       "(15157, 8)\n"
204 |      ]
205 |     }
206 |    ],
207 |    "source": [
208 |     "df = pd.read_csv(\"vw.csv\")\n",
209 |     "\n",
210 |     "display(df.head())\n",
211 |     "print(df.shape)"
212 |    ]
213 |   },
214 |   {
215 |    "cell_type": "code",
216 |    "execution_count": 110,
217 |    "metadata": {},
218 |    "outputs": [
219 |     {
220 |      "name": "stdout",
221 |      "output_type": "stream",
222 |      "text": [
223 |       "<class 'pandas.core.frame.DataFrame'>\n",
224 |       "RangeIndex: 15157 entries, 0 to 15156\n",
225 |       "Data columns (total 8 columns):\n",
226 |       " #   Column        Non-Null Count  Dtype  \n",
227 |       "---  ------        --------------  -----  \n",
228 |       " 0   model         15157 non-null  object \n",
229 |       " 1   year          15157 non-null  int64  \n",
230 |       " 2   price         15157 non-null  int64  \n",
231 |       " 3   transmission  15157 non-null  object \n",
232 |       " 4   mileage       15157 non-null  int64  \n",
233 |       " 5   fuelType      15157 non-null  object \n",
234 |       " 6   mpg           15157 non-null  float64\n",
235 |       " 7   engineSize    15157 non-null  float64\n",
236 |       "dtypes: float64(2), int64(3), object(3)\n",
237 |       "memory usage: 947.4+ KB\n"
238 |      ]
239 |     }
240 |    ],
241 |    "source": [
242 |     "df.info()"
243 |    ]
244 |   },
245 |   {
246 |    "cell_type": "code",
247 |    "execution_count": 111,
248 |    "metadata": {},
249 |    "outputs": [
250 |     {
251 |      "data": {
252 |       "text/html": [
253 |        "<div>\n",
254 |        "<style scoped>\n",
255 |        "    .dataframe tbody tr th:only-of-type {\n",
256 |        "        vertical-align: middle;\n",
257 |        "    }\n",
258 |        "\n",
259 |        "    .dataframe tbody tr th {\n",
260 |        "        vertical-align: top;\n",
261 |        "    }\n",
262 |        "\n",
263 |        "    .dataframe thead th {\n",
264 |        "        text-align: right;\n",
265 |        "    }\n",
266 |        "</style>\n",
267 |        "<table border=\"1\" class=\"dataframe\">\n",
268 |        "  <thead>\n",
269 |        "    <tr style=\"text-align: right;\">\n",
270 |        "      <th></th>\n",
271 |        "      <th>year</th>\n",
272 |        "      <th>price</th>\n",
273 |        "      <th>mileage</th>\n",
274 |        "      <th>mpg</th>\n",
275 |        "      <th>engineSize</th>\n",
276 |        "    </tr>\n",
277 |        "  </thead>\n",
278 |        "  <tbody>\n",
279 |        "    <tr>\n",
280 |        "      <th>count</th>\n",
281 |        "      <td>15157.000000</td>\n",
282 |        "      <td>15157.000000</td>\n",
283 |        "      <td>15157.000000</td>\n",
284 |        "      <td>15157.000000</td>\n",
285 |        "      <td>15157.000000</td>\n",
286 |        "    </tr>\n",
287 |        "    <tr>\n",
288 |        "      <th>mean</th>\n",
289 |        "      <td>2017.255789</td>\n",
290 |        "      <td>16838.952365</td>\n",
291 |        "      <td>22092.785644</td>\n",
292 |        "      <td>53.753355</td>\n",
293 |        "      <td>1.600693</td>\n",
294 |        "    </tr>\n",
295 |        "    <tr>\n",
296 |        "      <th>std</th>\n",
297 |        "      <td>2.053059</td>\n",
298 |        "      <td>7755.015206</td>\n",
299 |        "      <td>21148.941635</td>\n",
300 |        "      <td>13.642182</td>\n",
301 |        "      <td>0.461695</td>\n",
302 |        "    </tr>\n",
303 |        "    <tr>\n",
304 |        "      <th>min</th>\n",
305 |        "      <td>2000.000000</td>\n",
306 |        "      <td>899.000000</td>\n",
307 |        "      <td>1.000000</td>\n",
308 |        "      <td>0.300000</td>\n",
309 |        "      <td>0.000000</td>\n",
310 |        "    </tr>\n",
311 |        "    <tr>\n",
312 |        "      <th>25%</th>\n",
313 |        "      <td>2016.000000</td>\n",
314 |        "      <td>10990.000000</td>\n",
315 |        "      <td>5962.000000</td>\n",
316 |        "      <td>46.300000</td>\n",
317 |        "      <td>1.200000</td>\n",
318 |        "    </tr>\n",
319 |        "    <tr>\n",
320 |        "      <th>50%</th>\n",
321 |        "      <td>2017.000000</td>\n",
322 |        "      <td>15497.000000</td>\n",
323 |        "      <td>16393.000000</td>\n",
324 |        "      <td>53.300000</td>\n",
325 |        "      <td>1.600000</td>\n",
326 |        "    </tr>\n",
327 |        "    <tr>\n",
328 |        "      <th>75%</th>\n",
329 |        "      <td>2019.000000</td>\n",
330 |        "      <td>20998.000000</td>\n",
331 |        "      <td>31824.000000</td>\n",
332 |        "      <td>60.100000</td>\n",
333 |        "      <td>2.000000</td>\n",
334 |        "    </tr>\n",
335 |        "    <tr>\n",
336 |        "      <th>max</th>\n",
337 |        "      <td>2020.000000</td>\n",
338 |        "      <td>69994.000000</td>\n",
339 |        "      <td>212000.000000</td>\n",
340 |        "      <td>188.300000</td>\n",
341 |        "      <td>3.200000</td>\n",
342 |        "    </tr>\n",
343 |        "  </tbody>\n",
344 |        "</table>\n",
345 |        "</div>"
346 |       ],
347 |       "text/plain": [
348 |        "               year         price        mileage           mpg    engineSize\n",
349 |        "count  15157.000000  15157.000000   15157.000000  15157.000000  15157.000000\n",
350 |        "mean    2017.255789  16838.952365   22092.785644     53.753355      1.600693\n",
351 |        "std        2.053059   7755.015206   21148.941635     13.642182      0.461695\n",
352 |        "min     2000.000000    899.000000       1.000000      0.300000      0.000000\n",
353 |        "25%     2016.000000  10990.000000    5962.000000     46.300000      1.200000\n",
354 |        "50%     2017.000000  15497.000000   16393.000000     53.300000      1.600000\n",
355 |        "75%     2019.000000  20998.000000   31824.000000     60.100000      2.000000\n",
356 |        "max     2020.000000  69994.000000  212000.000000    188.300000      3.200000"
357 |       ]
358 |      },
359 |      "execution_count": 111,
360 |      "metadata": {},
361 |      "output_type": "execute_result"
362 |     }
363 |    ],
364 |    "source": [
365 |     "df.describe()"
366 |    ]
367 |   },
368 |   {
369 |    "cell_type": "code",
370 |    "execution_count": 112,
371 |    "metadata": {},
372 |    "outputs": [
373 |     {
374 |      "data": {
375 |       "text/plain": [
376 |        "-9600.0"
377 |       ]
378 |      },
379 |      "execution_count": 112,
380 |      "metadata": {},
381 |      "output_type": "execute_result"
382 |     }
383 |    ],
384 |    "source": [
385 |     "120000*-0.08"
386 |    ]
387 |   },
388 |   {
389 |    "cell_type": "code",
390 |    "execution_count": 113,
391 |    "metadata": {},
392 |    "outputs": [
393 |     {
394 |      "data": {
395 |       "text/plain": [
396 |        " Golf               4863\n",
397 |        " Polo               3287\n",
398 |        " Tiguan             1765\n",
399 |        " Passat              915\n",
400 |        " Up                  884\n",
401 |        " T-Roc               733\n",
402 |        " Touareg             363\n",
403 |        " Touran              352\n",
404 |        " T-Cross             300\n",
405 |        " Golf SV             268\n",
406 |        " Sharan              260\n",
407 |        " Arteon              248\n",
408 |        " Scirocco            242\n",
409 |        " Amarok              111\n",
410 |        " Caravelle           101\n",
411 |        " CC                   95\n",
412 |        " Tiguan Allspace      91\n",
413 |        " Beetle               83\n",
414 |        " Shuttle              61\n",
415 |        " Caddy Maxi Life      59\n",
416 |        " Jetta                32\n",
417 |        " California           15\n",
418 |        " Caddy Life            8\n",
419 |        " Eos                   7\n",
420 |        " Caddy                 6\n",
421 |        " Caddy Maxi            4\n",
422 |        " Fox                   4\n",
423 |        "Name: model, dtype: int64"
424 |       ]
425 |      },
426 |      "execution_count": 113,
427 |      "metadata": {},
428 |      "output_type": "execute_result"
429 |     }
430 |    ],
431 |    "source": [
432 |     "df[\"model\"].value_counts()"
433 |    ]
434 |   },
435 |   {
436 |    "cell_type": "code",
437 |    "execution_count": 114,
438 |    "metadata": {},
439 |    "outputs": [
440 |     {
441 |      "data": {
442 |       "text/plain": [
443 |        "Index(['model', 'year', 'price', 'transmission', 'mileage', 'fuelType', 'mpg',\n",
444 |        "       'engineSize'],\n",
445 |        "      dtype='object')"
446 |       ]
447 |      },
448 |      "execution_count": 114,
449 |      "metadata": {},
450 |      "output_type": "execute_result"
451 |     }
452 |    ],
453 |    "source": [
454 |     "df.columns"
455 |    ]
456 |   },
457 |   {
458 |    "cell_type": "code",
459 |    "execution_count": 115,
460 |    "metadata": {},
461 |    "outputs": [
462 |     {
463 |      "data": {
464 |       "text/plain": [
465 |        "Manual       9417\n",
466 |        "Semi-Auto    3780\n",
467 |        "Automatic    1960\n",
468 |        "Name: transmission, dtype: int64"
469 |       ]
470 |      },
471 |      "execution_count": 115,
472 |      "metadata": {},
473 |      "output_type": "execute_result"
474 |     }
475 |    ],
476 |    "source": [
477 |     "df[\"transmission\"].value_counts()"
478 |    ]
479 |   },
480 |   {
481 |    "cell_type": "code",
482 |    "execution_count": 116,
483 |    "metadata": {},
484 |    "outputs": [
485 |     {
486 |      "data": {
487 |       "text/plain": [
488 |        "Petrol    8553\n",
489 |        "Diesel    6372\n",
490 |        "Hybrid     145\n",
491 |        "Other       87\n",
492 |        "Name: fuelType, dtype: int64"
493 |       ]
494 |      },
495 |      "execution_count": 116,
496 |      "metadata": {},
497 |      "output_type": "execute_result"
498 |     }
499 |    ],
500 |    "source": [
501 |     "df[\"fuelType\"].value_counts()"
502 |    ]
503 |   },
504 |   {
505 |    "cell_type": "markdown",
506 |    "metadata": {},
507 |    "source": [
508 |     "#### Correlation plot: \n",
509 |     "https://github.com/SKawsar/Data_Analysis_with_Python/blob/main/Lecture_8.ipynb"
510 |    ]
511 |   },
512 |   {
513 |    "cell_type": "markdown",
514 |    "metadata": {},
515 |    "source": [
516 |     "#### Separating the features and target variable"
517 |    ]
518 |   },
519 |   {
520 |    "cell_type": "code",
521 |    "execution_count": 117,
522 |    "metadata": {},
523 |    "outputs": [
524 |     {
525 |      "name": "stdout",
526 |      "output_type": "stream",
527 |      "text": [
528 |       "Shape of X =  (15157, 3)\n",
529 |       "Shape of y =  (15157, 1)\n"
530 |      ]
531 |     }
532 |    ],
533 |    "source": [
534 |     "features = ['year', 'mpg', 'engineSize']\n",
535 |     "target = ['price']\n",
536 |     "\n",
537 |     "X = df[features]\n",
538 |     "y = df[target]\n",
539 |     "\n",
540 |     "print(\"Shape of X = \", X.shape)\n",
541 |     "print(\"Shape of y = \", y.shape)"
542 |    ]
543 |   },
544 |   {
545 |    "cell_type": "markdown",
546 |    "metadata": {},
547 |    "source": [
548 |     "#### Create train and test set"
549 |    ]
550 |   },
551 |   {
552 |    "cell_type": "code",
553 |    "execution_count": 118,
554 |    "metadata": {},
555 |    "outputs": [
556 |     {
557 |      "name": "stdout",
558 |      "output_type": "stream",
559 |      "text": [
560 |       "(12125, 3) (3032, 3) (12125, 1) (3032, 1)\n"
561 |      ]
562 |     }
563 |    ],
564 |    "source": [
565 |     "X_train, X_test, y_train, y_test = train_test_split(X, \n",
566 |     "                                                    y,\n",
567 |     "                                                    test_size=0.2, \n",
568 |     "                                                    random_state=42)\n",
569 |     "\n",
570 |     "print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
571 |    ]
572 |   },
573 |   {
574 |    "cell_type": "markdown",
575 |    "metadata": {},
576 |    "source": [
577 |     "#### Linear Regression"
578 |    ]
579 |   },
580 |   {
581 |    "cell_type": "code",
582 |    "execution_count": 119,
583 |    "metadata": {},
584 |    "outputs": [],
585 |    "source": [
586 |     "model = LinearRegression()\n",
587 |     "model = model.fit(X_train, y_train)"
588 |    ]
589 |   },
590 |   {
591 |    "cell_type": "markdown",
592 |    "metadata": {},
593 |    "source": [
594 |     "y = m1*x1 + m2*x2 + m3*x3 + m4*x4 + c"
595 |    ]
596 |   },
597 |   {
598 |    "cell_type": "code",
599 |    "execution_count": 120,
600 |    "metadata": {},
601 |    "outputs": [
602 |     {
603 |      "name": "stdout",
604 |      "output_type": "stream",
605 |      "text": [
606 |       "[[2114.02170162 -106.30742038 8759.3771598 ]]\n"
607 |      ]
608 |     }
609 |    ],
610 |    "source": [
611 |     "coefficients = model.coef_\n",
612 |     "print(coefficients)"
613 |    ]
614 |   },
615 |   {
616 |    "cell_type": "code",
617 |    "execution_count": 121,
618 |    "metadata": {},
619 |    "outputs": [
620 |     {
621 |      "name": "stdout",
622 |      "output_type": "stream",
623 |      "text": [
624 |       "[-4255990.43023728]\n"
625 |      ]
626 |     }
627 |    ],
628 |    "source": [
629 |     "c = model.intercept_\n",
630 |     "print(c)"
631 |    ]
632 |   },
633 |   {
634 |    "cell_type": "code",
635 |    "execution_count": 122,
636 |    "metadata": {},
637 |    "outputs": [
638 |     {
639 |      "data": {
640 |       "text/plain": [
641 |        "Index(['year', 'mpg', 'engineSize'], dtype='object')"
642 |       ]
643 |      },
644 |      "execution_count": 122,
645 |      "metadata": {},
646 |      "output_type": "execute_result"
647 |     }
648 |    ],
649 |    "source": [
650 |     "X.columns"
651 |    ]
652 |   },
653 |   {
654 |    "cell_type": "code",
655 |    "execution_count": 123,
656 |    "metadata": {},
657 |    "outputs": [
658 |     {
659 |      "data": {
660 |       "text/html": [
661 |        "<div>\n",
662 |        "<style scoped>\n",
663 |        "    .dataframe tbody tr th:only-of-type {\n",
664 |        "        vertical-align: middle;\n",
665 |        "    }\n",
666 |        "\n",
667 |        "    .dataframe tbody tr th {\n",
668 |        "        vertical-align: top;\n",
669 |        "    }\n",
670 |        "\n",
671 |        "    .dataframe thead th {\n",
672 |        "        text-align: right;\n",
673 |        "    }\n",
674 |        "</style>\n",
675 |        "<table border=\"1\" class=\"dataframe\">\n",
676 |        "  <thead>\n",
677 |        "    <tr style=\"text-align: right;\">\n",
678 |        "      <th></th>\n",
679 |        "      <th>features</th>\n",
680 |        "      <th>coefficients</th>\n",
681 |        "    </tr>\n",
682 |        "  </thead>\n",
683 |        "  <tbody>\n",
684 |        "    <tr>\n",
685 |        "      <th>0</th>\n",
686 |        "      <td>year</td>\n",
687 |        "      <td>2114.021702</td>\n",
688 |        "    </tr>\n",
689 |        "    <tr>\n",
690 |        "      <th>1</th>\n",
691 |        "      <td>mpg</td>\n",
692 |        "      <td>-106.307420</td>\n",
693 |        "    </tr>\n",
694 |        "    <tr>\n",
695 |        "      <th>2</th>\n",
696 |        "      <td>engineSize</td>\n",
697 |        "      <td>8759.377160</td>\n",
698 |        "    </tr>\n",
699 |        "  </tbody>\n",
700 |        "</table>\n",
701 |        "</div>"
702 |       ],
703 |       "text/plain": [
704 |        "     features  coefficients\n",
705 |        "0        year   2114.021702\n",
706 |        "1         mpg   -106.307420\n",
707 |        "2  engineSize   8759.377160"
708 |       ]
709 |      },
710 |      "metadata": {},
711 |      "output_type": "display_data"
712 |     }
713 |    ],
714 |    "source": [
715 |     "coef_df = pd.DataFrame({\"features\": X.columns, \n",
716 |     "                       \"coefficients\": np.squeeze(coefficients)})\n",
717 |     "\n",
718 |     "display(coef_df)"
719 |    ]
720 |   },
721 |   {
722 |    "cell_type": "markdown",
723 |    "metadata": {},
724 |    "source": [
725 |     "- A positive sign indicates that as the feature variable increases, the target variable also increases.\n",
726 |     "- A negative sign indicates that as the feature variable increases, the target variable decreases."
727 |    ]
728 |   },
729 |   {
730 |    "cell_type": "markdown",
731 |    "metadata": {},
732 |    "source": [
733 |     "#### Prediction"
734 |    ]
735 |   },
736 |   {
737 |    "cell_type": "code",
738 |    "execution_count": 124,
739 |    "metadata": {},
740 |    "outputs": [
741 |     {
742 |      "name": "stdout",
743 |      "output_type": "stream",
744 |      "text": [
745 |       "[[13304.86271129]\n",
746 |       " [24646.01422239]\n",
747 |       " [18355.60686375]\n",
748 |       " ...\n",
749 |       " [ 8375.19033353]\n",
750 |       " [ 5771.45345778]\n",
751 |       " [ 6261.1686319 ]]\n"
752 |      ]
753 |     }
754 |    ],
755 |    "source": [
756 |     "y_pred = model.predict(X_test)\n",
757 |     "print(y_pred)"
758 |    ]
759 |   },
760 |   {
761 |    "cell_type": "code",
762 |    "execution_count": 125,
763 |    "metadata": {},
764 |    "outputs": [
765 |     {
766 |      "name": "stdout",
767 |      "output_type": "stream",
768 |      "text": [
769 |       "       price\n",
770 |       "7342   14450\n",
771 |       "10328  23950\n",
772 |       "14992  10495\n",
773 |       "8466    9990\n",
774 |       "10347  21998\n",
775 |       "...      ...\n",
776 |       "8211   17250\n",
777 |       "8401   10450\n",
778 |       "9810   10290\n",
779 |       "7872    7499\n",
780 |       "9399    7290\n",
781 |       "\n",
782 |       "[3032 rows x 1 columns]\n"
783 |      ]
784 |     }
785 |    ],
786 |    "source": [
787 |     "print(y_test)"
788 |    ]
789 |   },
790 |   {
791 |    "cell_type": "code",
792 |    "execution_count": 126,
793 |    "metadata": {},
794 |    "outputs": [],
795 |    "source": [
796 |     "# actual value - predicted value = +"
797 |    ]
798 |   },
799 |   {
800 |    "cell_type": "markdown",
801 |    "metadata": {},
802 |    "source": [
803 |     "#### Prediction Error"
804 |    ]
805 |   },
806 |   {
807 |    "cell_type": "code",
808 |    "execution_count": 127,
809 |    "metadata": {},
810 |    "outputs": [
811 |     {
812 |      "name": "stdout",
813 |      "output_type": "stream",
814 |      "text": [
815 |       "MAE =  2681.5051965262132\n",
816 |       "MSE =  16459163.341451917\n",
817 |       "RMSE =  4056.9894431033363\n",
818 |       "r_squared =  0.7226377192887656\n"
819 |      ]
820 |     }
821 |    ],
822 |    "source": [
823 |     "MAE = mean_absolute_error(y_test, y_pred)\n",
824 |     "print(\"MAE = \", MAE)\n",
825 |     "\n",
826 |     "MSE = mean_squared_error(y_test, y_pred, squared=True)\n",
827 |     "print(\"MSE = \", MSE)\n",
828 |     "\n",
829 |     "RMSE = mean_squared_error(y_test, y_pred, squared=False)\n",
830 |     "print(\"RMSE = \", RMSE)\n",
831 |     "\n",
832 |     "r2 = r2_score(y_test, y_pred)\n",
833 |     "print(\"r_squared = \", r2)"
834 |    ]
835 |   },
836 |   {
837 |    "cell_type": "code",
838 |    "execution_count": 128,
839 |    "metadata": {},
840 |    "outputs": [],
841 |    "source": [
842 |     "# mean_absolute_error, r2_score"
843 |    ]
844 |   },
845 |   {
846 |    "cell_type": "code",
847 |    "execution_count": null,
848 |    "metadata": {},
849 |    "outputs": [],
850 |    "source": []
851 |   }
852 |  ],
853 |  "metadata": {
854 |   "kernelspec": {
855 |    "display_name": "Python 3 (ipykernel)",
856 |    "language": "python",
857 |    "name": "python3"
858 |   },
859 |   "language_info": {
860 |    "codemirror_mode": {
861 |     "name": "ipython",
862 |     "version": 3
863 |    },
864 |    "file_extension": ".py",
865 |    "mimetype": "text/x-python",
866 |    "name": "python",
867 |    "nbconvert_exporter": "python",
868 |    "pygments_lexer": "ipython3",
869 |    "version": "3.8.5"
870 |   }
871 |  },
872 |  "nbformat": 4,
873 |  "nbformat_minor": 4
874 | }
875 | 


--------------------------------------------------------------------------------
/3. Regression accuracy metrics.ipynb:
--------------------------------------------------------------------------------
  1 | {
  2 |  "cells": [
  3 |   {
  4 |    "cell_type": "markdown",
  5 |    "metadata": {},
  6 |    "source": [
  7 |     "# Expedition to Data Science and Machine Learning\n",
  8 |     "## Module 4: Machine Learning with Python\n",
  9 |     "### Lecture 3: Supervised Learning: Linear Regression and Regression accuracy metrics\n",
 10 |     "\n",
 11 |     "Instructor: Md Shahidullah Kawsar\n",
 12 |     "<br>Data Scientist, IDARE, Houston, TX, USA\n",
 13 |     "\n",
 14 |     "#### Objectives:\n",
 15 |     "- Supervised Learning: Linear Regression\n",
 16 |     "- Accuracy metric in Regression problem\n",
 17 |     "- Mean Absolute Error (MAE)\n",
 18 |     "- Mean Absolute Percentage Error (MAPE)\n",
 19 |     "- Mean Squared Error (MSE)\n",
 20 |     "- Root Mean Squared Error (RMSE)\n",
 21 |     "- R-squared or coefficient of determination\n",
 22 |     "- Prediction result evaluation\n",
 23 |     "\n",
 24 |     "#### References:\n",
 25 |     "[1] Accuracy metrics in sklearn: https://scikit-learn.org/stable/modules/model_evaluation.html\n",
 26 |     "<br>[2] Mean Absolute Error (MAE): https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error\n",
 27 |     "<br>[3] Mean Squared Error (MSE) and Root Mean Squared Error (RMSE) https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html#sklearn.metrics.mean_squared_error\n",
 28 |     "<br>[4] R-squared or coefficient of determination: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.r2_score.html#sklearn.metrics.r2_score\n",
 29 |     "<br>[5] MAE, MSE, RMSE, Coefficient of Determination, Adjusted R Squared — Which Metric is Better? https://medium.com/analytics-vidhya/mae-mse-rmse-coefficient-of-determination-adjusted-r-squared-which-metric-is-better-cd0326a5697e"
 30 |    ]
 31 |   },
 32 |   {
 33 |    "cell_type": "markdown",
 34 |    "metadata": {},
 35 |    "source": [
 36 |     "#### Import required Libraries"
 37 |    ]
 38 |   },
 39 |   {
 40 |    "cell_type": "code",
 41 |    "execution_count": 10,
 42 |    "metadata": {},
 43 |    "outputs": [],
 44 |    "source": [
 45 |     "import pandas as pd\n",
 46 |     "import numpy as np"
 47 |    ]
 48 |   },
 49 |   {
 50 |    "cell_type": "markdown",
 51 |    "metadata": {},
 52 |    "source": [
 53 |     "#### Accuracy metrics in Supervised Learning: Regression"
 54 |    ]
 55 |   },
 56 |   {
 57 |    "cell_type": "code",
 58 |    "execution_count": 11,
 59 |    "metadata": {},
 60 |    "outputs": [],
 61 |    "source": [
 62 |     "actual_value = [1,2,3,4,5,6,7,8,9,10]\n",
 63 |     "predicted_value = [1,3,4,5,6,5,6,5,8,9]"
 64 |    ]
 65 |   },
 66 |   {
 67 |    "cell_type": "markdown",
 68 |    "metadata": {},
 69 |    "source": [
 70 |     "**Mean absolute error** represents the average of the absolute difference between the actual and predicted values in the dataset. It measures the average of the residuals in the dataset.\n",
 71 |     "\n",
 72 |     "**Mean Squared Error** represents the average of the squared difference between the original and predicted values in the data set. It measures the variance of the residuals.\n",
 73 |     "\n",
 74 |     "**Root Mean Squared Error** is the square root of Mean Squared error. It measures the standard deviation of residuals.\n",
 75 |     "\n",
 76 |     "**Coefficient of determination or R-squared** represents the proportion of the variance in the dependent variable. It is a scale-free score i.e. irrespective of the values being small or large, the value of R square will be less than one."
 77 |    ]
 78 |   },
 79 |   {
 80 |    "cell_type": "code",
 81 |    "execution_count": 24,
 82 |    "metadata": {},
 83 |    "outputs": [
 84 |     {
 85 |      "data": {
 86 |       "text/html": [
 87 |        "<div>\n",
 88 |        "<style scoped>\n",
 89 |        "    .dataframe tbody tr th:only-of-type {\n",
 90 |        "        vertical-align: middle;\n",
 91 |        "    }\n",
 92 |        "\n",
 93 |        "    .dataframe tbody tr th {\n",
 94 |        "        vertical-align: top;\n",
 95 |        "    }\n",
 96 |        "\n",
 97 |        "    .dataframe thead th {\n",
 98 |        "        text-align: right;\n",
 99 |        "    }\n",
100 |        "</style>\n",
101 |        "<table border=\"1\" class=\"dataframe\">\n",
102 |        "  <thead>\n",
103 |        "    <tr style=\"text-align: right;\">\n",
104 |        "      <th></th>\n",
105 |        "      <th>actual</th>\n",
106 |        "      <th>predicted</th>\n",
107 |        "      <th>dif</th>\n",
108 |        "      <th>abs_error</th>\n",
109 |        "      <th>squared_error</th>\n",
110 |        "      <th>actual_subtract_mean</th>\n",
111 |        "      <th>squared_actual_subtract_mean</th>\n",
112 |        "    </tr>\n",
113 |        "  </thead>\n",
114 |        "  <tbody>\n",
115 |        "    <tr>\n",
116 |        "      <th>0</th>\n",
117 |        "      <td>1</td>\n",
118 |        "      <td>1</td>\n",
119 |        "      <td>0</td>\n",
120 |        "      <td>0</td>\n",
121 |        "      <td>0</td>\n",
122 |        "      <td>-4.5</td>\n",
123 |        "      <td>20.25</td>\n",
124 |        "    </tr>\n",
125 |        "    <tr>\n",
126 |        "      <th>1</th>\n",
127 |        "      <td>2</td>\n",
128 |        "      <td>3</td>\n",
129 |        "      <td>-1</td>\n",
130 |        "      <td>1</td>\n",
131 |        "      <td>1</td>\n",
132 |        "      <td>-3.5</td>\n",
133 |        "      <td>12.25</td>\n",
134 |        "    </tr>\n",
135 |        "    <tr>\n",
136 |        "      <th>2</th>\n",
137 |        "      <td>3</td>\n",
138 |        "      <td>4</td>\n",
139 |        "      <td>-1</td>\n",
140 |        "      <td>1</td>\n",
141 |        "      <td>1</td>\n",
142 |        "      <td>-2.5</td>\n",
143 |        "      <td>6.25</td>\n",
144 |        "    </tr>\n",
145 |        "    <tr>\n",
146 |        "      <th>3</th>\n",
147 |        "      <td>4</td>\n",
148 |        "      <td>5</td>\n",
149 |        "      <td>-1</td>\n",
150 |        "      <td>1</td>\n",
151 |        "      <td>1</td>\n",
152 |        "      <td>-1.5</td>\n",
153 |        "      <td>2.25</td>\n",
154 |        "    </tr>\n",
155 |        "    <tr>\n",
156 |        "      <th>4</th>\n",
157 |        "      <td>5</td>\n",
158 |        "      <td>6</td>\n",
159 |        "      <td>-1</td>\n",
160 |        "      <td>1</td>\n",
161 |        "      <td>1</td>\n",
162 |        "      <td>-0.5</td>\n",
163 |        "      <td>0.25</td>\n",
164 |        "    </tr>\n",
165 |        "    <tr>\n",
166 |        "      <th>5</th>\n",
167 |        "      <td>6</td>\n",
168 |        "      <td>5</td>\n",
169 |        "      <td>1</td>\n",
170 |        "      <td>1</td>\n",
171 |        "      <td>1</td>\n",
172 |        "      <td>0.5</td>\n",
173 |        "      <td>0.25</td>\n",
174 |        "    </tr>\n",
175 |        "    <tr>\n",
176 |        "      <th>6</th>\n",
177 |        "      <td>7</td>\n",
178 |        "      <td>6</td>\n",
179 |        "      <td>1</td>\n",
180 |        "      <td>1</td>\n",
181 |        "      <td>1</td>\n",
182 |        "      <td>1.5</td>\n",
183 |        "      <td>2.25</td>\n",
184 |        "    </tr>\n",
185 |        "    <tr>\n",
186 |        "      <th>7</th>\n",
187 |        "      <td>8</td>\n",
188 |        "      <td>5</td>\n",
189 |        "      <td>3</td>\n",
190 |        "      <td>3</td>\n",
191 |        "      <td>9</td>\n",
192 |        "      <td>2.5</td>\n",
193 |        "      <td>6.25</td>\n",
194 |        "    </tr>\n",
195 |        "    <tr>\n",
196 |        "      <th>8</th>\n",
197 |        "      <td>9</td>\n",
198 |        "      <td>8</td>\n",
199 |        "      <td>1</td>\n",
200 |        "      <td>1</td>\n",
201 |        "      <td>1</td>\n",
202 |        "      <td>3.5</td>\n",
203 |        "      <td>12.25</td>\n",
204 |        "    </tr>\n",
205 |        "    <tr>\n",
206 |        "      <th>9</th>\n",
207 |        "      <td>10</td>\n",
208 |        "      <td>9</td>\n",
209 |        "      <td>1</td>\n",
210 |        "      <td>1</td>\n",
211 |        "      <td>1</td>\n",
212 |        "      <td>4.5</td>\n",
213 |        "      <td>20.25</td>\n",
214 |        "    </tr>\n",
215 |        "  </tbody>\n",
216 |        "</table>\n",
217 |        "</div>"
218 |       ],
219 |       "text/plain": [
220 |        "   actual  predicted  dif  abs_error  squared_error  actual_subtract_mean  \\\n",
221 |        "0       1          1    0          0              0                  -4.5   \n",
222 |        "1       2          3   -1          1              1                  -3.5   \n",
223 |        "2       3          4   -1          1              1                  -2.5   \n",
224 |        "3       4          5   -1          1              1                  -1.5   \n",
225 |        "4       5          6   -1          1              1                  -0.5   \n",
226 |        "5       6          5    1          1              1                   0.5   \n",
227 |        "6       7          6    1          1              1                   1.5   \n",
228 |        "7       8          5    3          3              9                   2.5   \n",
229 |        "8       9          8    1          1              1                   3.5   \n",
230 |        "9      10          9    1          1              1                   4.5   \n",
231 |        "\n",
232 |        "   squared_actual_subtract_mean  \n",
233 |        "0                         20.25  \n",
234 |        "1                         12.25  \n",
235 |        "2                          6.25  \n",
236 |        "3                          2.25  \n",
237 |        "4                          0.25  \n",
238 |        "5                          0.25  \n",
239 |        "6                          2.25  \n",
240 |        "7                          6.25  \n",
241 |        "8                         12.25  \n",
242 |        "9                         20.25  "
243 |       ]
244 |      },
245 |      "metadata": {},
246 |      "output_type": "display_data"
247 |     }
248 |    ],
249 |    "source": [
250 |     "df = pd.DataFrame({\"actual\":actual_value,\n",
251 |     "                   \"predicted\": predicted_value})\n",
252 |     "\n",
253 |     "df[\"dif\"] = df[\"actual\"] - df[\"predicted\"]\n",
254 |     "df[\"abs_error\"] = np.abs(df[\"dif\"])\n",
255 |     "df[\"squared_error\"] = df[\"dif\"]**2\n",
256 |     "\n",
257 |     "df[\"actual_subtract_mean\"] = df[\"actual\"] - df[\"actual\"].mean()\n",
258 |     "df[\"squared_actual_subtract_mean\"] = df[\"actual_subtract_mean\"]**2\n",
259 |     "\n",
260 |     "display(df)"
261 |    ]
262 |   },
263 |   {
264 |    "cell_type": "code",
265 |    "execution_count": 23,
266 |    "metadata": {},
267 |    "outputs": [
268 |     {
269 |      "data": {
270 |       "text/plain": [
271 |        "5.5"
272 |       ]
273 |      },
274 |      "execution_count": 23,
275 |      "metadata": {},
276 |      "output_type": "execute_result"
277 |     }
278 |    ],
279 |    "source": [
280 |     "df[\"actual\"].mean()"
281 |    ]
282 |   },
283 |   {
284 |    "cell_type": "code",
285 |    "execution_count": 27,
286 |    "metadata": {},
287 |    "outputs": [
288 |     {
289 |      "name": "stdout",
290 |      "output_type": "stream",
291 |      "text": [
292 |       "MAE =  1.1\n",
293 |       "MAPE =  21.79\n",
294 |       "MSE =  1.7\n",
295 |       "RMSE =  1.3\n",
296 |       "r_squared =  0.79\n"
297 |      ]
298 |     }
299 |    ],
300 |    "source": [
301 |     "# mean absolute error: lower is better\n",
302 |     "MAE = df[\"abs_error\"].mean()\n",
303 |     "print(\"MAE = \", MAE)\n",
304 |     "\n",
305 |     "# MAPE: Mean Absolute Percentage Error: lower is better\n",
306 |     "MAPE = np.round(np.mean(df[\"abs_error\"]/df[\"actual\"])*100, 2)\n",
307 |     "print(\"MAPE = \", MAPE)\n",
308 |     "\n",
309 |     "# mean squared error: lower is better\n",
310 |     "MSE = df[\"squared_error\"].mean()\n",
311 |     "print(\"MSE = \", MSE)\n",
312 |     "\n",
313 |     "# root mean squared error: lower is better\n",
314 |     "RMSE = np.round(np.sqrt(MSE), 2)\n",
315 |     "print(\"RMSE = \", RMSE)\n",
316 |     "\n",
317 |     "# coefficient of determination == r_squared: greater is better. Max =1, min=-\n",
318 |     "r_squared = np.round(1- df[\"squared_error\"].sum()/df[\"squared_actual_subtract_mean\"].sum(), 2)\n",
319 |     "print(\"r_squared = \", r_squared)"
320 |    ]
321 |   },
322 |   {
323 |    "cell_type": "code",
324 |    "execution_count": null,
325 |    "metadata": {},
326 |    "outputs": [],
327 |    "source": []
328 |   }
329 |  ],
330 |  "metadata": {
331 |   "kernelspec": {
332 |    "display_name": "Python 3 (ipykernel)",
333 |    "language": "python",
334 |    "name": "python3"
335 |   },
336 |   "language_info": {
337 |    "codemirror_mode": {
338 |     "name": "ipython",
339 |     "version": 3
340 |    },
341 |    "file_extension": ".py",
342 |    "mimetype": "text/x-python",
343 |    "name": "python",
344 |    "nbconvert_exporter": "python",
345 |    "pygments_lexer": "ipython3",
346 |    "version": "3.8.5"
347 |   }
348 |  },
349 |  "nbformat": 4,
350 |  "nbformat_minor": 4
351 | }
352 | 


--------------------------------------------------------------------------------
/4. Logistic Regression.ipynb:
--------------------------------------------------------------------------------
   1 | {
   2 |  "cells": [
   3 |   {
   4 |    "cell_type": "markdown",
   5 |    "metadata": {},
   6 |    "source": [
   7 |     "# Expedition to Data Science and Machine Learning\n",
   8 |     "## Module 4: Machine Learning with Python\n",
   9 |     "### Lecture 4: Supervised Learning - Classification: Logistic Regression\n",
  10 |     "\n",
  11 |     "Instructor: Md Shahidullah Kawsar\n",
  12 |     "<br>Data Scientist, IDARE, Houston, TX, USA\n",
  13 |     "\n",
  14 |     "#### Objectives:\n",
  15 |     "- Supervised Learning - Classification: Logistic Regression\n",
  16 |     "\n",
  17 |     "#### References:\n",
  18 |     "[1] Machine Learning Fundamentals: The Confusion Matrix - https://www.youtube.com/watch?v=Kdsp6soqA7o&ab_channel=StatQuestwithJoshStarmer\n",
  19 |     "<br>[2] Machine Learning Fundamentals: Sensitivity and Specificity: https://www.youtube.com/watch?v=sunUKFXMHGk&ab_channel=StatQuestwithJoshStarmer\n",
  20 |     "<br>[3] ROC and AUC, Clearly Explained! https://www.youtube.com/watch?v=4jRBRDbJemM&ab_channel=StatQuestwithJoshStarmer\n",
  21 |     "<br>[4] Machine Learning Fundamentals: Bias and Variance: https://www.youtube.com/watch?v=EuBBz3bI-aA&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF&index=5&ab_channel=StatQuestwithJoshStarmer\n",
  22 |     "<br>[5] StatQuest: Logistic Regression - https://www.youtube.com/watch?v=yIYKR4sgzI8&ab_channel=StatQuestwithJoshStarmer\n",
  23 |     "<br>[6] Logistic Regression Details Pt1: Coefficients: https://www.youtube.com/watch?v=vN5cNN2-HWE&ab_channel=StatQuestwithJoshStarmer\n",
  24 |     "<br>[7] Logistic Regression Details Pt 2: Maximum Likelihood: https://www.youtube.com/watch?v=BfKanl1aSG0&t=163s&ab_channel=StatQuestwithJoshStarmer\n",
  25 |     "<br>[8] Machine Learning Tutorial Python - 8: Logistic Regression (Binary Classification): https://www.youtube.com/watch?v=zM4VZR0px8E&ab_channel=codebasics\n",
  26 |     "<br>[9] Maximum Likelihood, clearly explained!!!: https://www.youtube.com/watch?v=XepXtl9YKwc&ab_channel=StatQuestwithJoshStarmer"
  27 |    ]
  28 |   },
  29 |   {
  30 |    "cell_type": "markdown",
  31 |    "metadata": {},
  32 |    "source": [
  33 |     "#### Import required libraries"
  34 |    ]
  35 |   },
  36 |   {
  37 |    "cell_type": "code",
  38 |    "execution_count": 123,
  39 |    "metadata": {},
  40 |    "outputs": [],
  41 |    "source": [
  42 |     "import pandas as pd\n",
  43 |     "\n",
  44 |     "import numpy as np\n",
  45 |     "\n",
  46 |     "import matplotlib.pyplot as plt\n",
  47 |     "import seaborn as sns\n",
  48 |     "\n",
  49 |     "from sklearn.model_selection import train_test_split\n",
  50 |     "\n",
  51 |     "from sklearn.linear_model import LogisticRegression"
  52 |    ]
  53 |   },
  54 |   {
  55 |    "cell_type": "markdown",
  56 |    "metadata": {},
  57 |    "source": [
  58 |     "#### Load data\n",
  59 |     "Dataset Source: https://learn.datacamp.com/courses/marketing-analytics-predicting-customer-churn-in-python"
  60 |    ]
  61 |   },
  62 |   {
  63 |    "cell_type": "code",
  64 |    "execution_count": 124,
  65 |    "metadata": {},
  66 |    "outputs": [
  67 |     {
  68 |      "data": {
  69 |       "text/html": [
  70 |        "<div>\n",
  71 |        "<style scoped>\n",
  72 |        "    .dataframe tbody tr th:only-of-type {\n",
  73 |        "        vertical-align: middle;\n",
  74 |        "    }\n",
  75 |        "\n",
  76 |        "    .dataframe tbody tr th {\n",
  77 |        "        vertical-align: top;\n",
  78 |        "    }\n",
  79 |        "\n",
  80 |        "    .dataframe thead th {\n",
  81 |        "        text-align: right;\n",
  82 |        "    }\n",
  83 |        "</style>\n",
  84 |        "<table border=\"1\" class=\"dataframe\">\n",
  85 |        "  <thead>\n",
  86 |        "    <tr style=\"text-align: right;\">\n",
  87 |        "      <th></th>\n",
  88 |        "      <th>Account_Length</th>\n",
  89 |        "      <th>Vmail_Message</th>\n",
  90 |        "      <th>Day_Mins</th>\n",
  91 |        "      <th>Eve_Mins</th>\n",
  92 |        "      <th>Night_Mins</th>\n",
  93 |        "      <th>Intl_Mins</th>\n",
  94 |        "      <th>CustServ_Calls</th>\n",
  95 |        "      <th>Churn</th>\n",
  96 |        "      <th>Intl_Plan</th>\n",
  97 |        "      <th>Vmail_Plan</th>\n",
  98 |        "      <th>Day_Calls</th>\n",
  99 |        "      <th>Day_Charge</th>\n",
 100 |        "      <th>Eve_Calls</th>\n",
 101 |        "      <th>Eve_Charge</th>\n",
 102 |        "      <th>Night_Calls</th>\n",
 103 |        "      <th>Night_Charge</th>\n",
 104 |        "      <th>Intl_Calls</th>\n",
 105 |        "      <th>Intl_Charge</th>\n",
 106 |        "      <th>State</th>\n",
 107 |        "      <th>Area_Code</th>\n",
 108 |        "      <th>Phone</th>\n",
 109 |        "    </tr>\n",
 110 |        "  </thead>\n",
 111 |        "  <tbody>\n",
 112 |        "    <tr>\n",
 113 |        "      <th>1648</th>\n",
 114 |        "      <td>102</td>\n",
 115 |        "      <td>0</td>\n",
 116 |        "      <td>174.5</td>\n",
 117 |        "      <td>236.8</td>\n",
 118 |        "      <td>270.4</td>\n",
 119 |        "      <td>8.5</td>\n",
 120 |        "      <td>0</td>\n",
 121 |        "      <td>no</td>\n",
 122 |        "      <td>no</td>\n",
 123 |        "      <td>no</td>\n",
 124 |        "      <td>79</td>\n",
 125 |        "      <td>29.67</td>\n",
 126 |        "      <td>136</td>\n",
 127 |        "      <td>20.13</td>\n",
 128 |        "      <td>110</td>\n",
 129 |        "      <td>12.17</td>\n",
 130 |        "      <td>5</td>\n",
 131 |        "      <td>2.30</td>\n",
 132 |        "      <td>VA</td>\n",
 133 |        "      <td>510</td>\n",
 134 |        "      <td>398-5788</td>\n",
 135 |        "    </tr>\n",
 136 |        "    <tr>\n",
 137 |        "      <th>2706</th>\n",
 138 |        "      <td>125</td>\n",
 139 |        "      <td>0</td>\n",
 140 |        "      <td>206.0</td>\n",
 141 |        "      <td>198.1</td>\n",
 142 |        "      <td>135.9</td>\n",
 143 |        "      <td>13.2</td>\n",
 144 |        "      <td>0</td>\n",
 145 |        "      <td>no</td>\n",
 146 |        "      <td>no</td>\n",
 147 |        "      <td>no</td>\n",
 148 |        "      <td>128</td>\n",
 149 |        "      <td>35.02</td>\n",
 150 |        "      <td>71</td>\n",
 151 |        "      <td>16.84</td>\n",
 152 |        "      <td>116</td>\n",
 153 |        "      <td>6.12</td>\n",
 154 |        "      <td>3</td>\n",
 155 |        "      <td>3.56</td>\n",
 156 |        "      <td>WV</td>\n",
 157 |        "      <td>415</td>\n",
 158 |        "      <td>381-7597</td>\n",
 159 |        "    </tr>\n",
 160 |        "    <tr>\n",
 161 |        "      <th>3198</th>\n",
 162 |        "      <td>53</td>\n",
 163 |        "      <td>32</td>\n",
 164 |        "      <td>131.2</td>\n",
 165 |        "      <td>227.4</td>\n",
 166 |        "      <td>178.9</td>\n",
 167 |        "      <td>12.8</td>\n",
 168 |        "      <td>2</td>\n",
 169 |        "      <td>no</td>\n",
 170 |        "      <td>no</td>\n",
 171 |        "      <td>yes</td>\n",
 172 |        "      <td>63</td>\n",
 173 |        "      <td>22.30</td>\n",
 174 |        "      <td>125</td>\n",
 175 |        "      <td>19.33</td>\n",
 176 |        "      <td>105</td>\n",
 177 |        "      <td>8.05</td>\n",
 178 |        "      <td>2</td>\n",
 179 |        "      <td>3.46</td>\n",
 180 |        "      <td>DE</td>\n",
 181 |        "      <td>415</td>\n",
 182 |        "      <td>416-9723</td>\n",
 183 |        "    </tr>\n",
 184 |        "    <tr>\n",
 185 |        "      <th>306</th>\n",
 186 |        "      <td>113</td>\n",
 187 |        "      <td>0</td>\n",
 188 |        "      <td>272.1</td>\n",
 189 |        "      <td>268.5</td>\n",
 190 |        "      <td>213.8</td>\n",
 191 |        "      <td>8.5</td>\n",
 192 |        "      <td>1</td>\n",
 193 |        "      <td>yes</td>\n",
 194 |        "      <td>no</td>\n",
 195 |        "      <td>no</td>\n",
 196 |        "      <td>111</td>\n",
 197 |        "      <td>46.26</td>\n",
 198 |        "      <td>118</td>\n",
 199 |        "      <td>22.82</td>\n",
 200 |        "      <td>105</td>\n",
 201 |        "      <td>9.62</td>\n",
 202 |        "      <td>10</td>\n",
 203 |        "      <td>2.30</td>\n",
 204 |        "      <td>VT</td>\n",
 205 |        "      <td>415</td>\n",
 206 |        "      <td>419-1714</td>\n",
 207 |        "    </tr>\n",
 208 |        "    <tr>\n",
 209 |        "      <th>2227</th>\n",
 210 |        "      <td>41</td>\n",
 211 |        "      <td>0</td>\n",
 212 |        "      <td>237.8</td>\n",
 213 |        "      <td>223.5</td>\n",
 214 |        "      <td>217.4</td>\n",
 215 |        "      <td>10.2</td>\n",
 216 |        "      <td>2</td>\n",
 217 |        "      <td>no</td>\n",
 218 |        "      <td>no</td>\n",
 219 |        "      <td>no</td>\n",
 220 |        "      <td>92</td>\n",
 221 |        "      <td>40.43</td>\n",
 222 |        "      <td>155</td>\n",
 223 |        "      <td>19.00</td>\n",
 224 |        "      <td>90</td>\n",
 225 |        "      <td>9.78</td>\n",
 226 |        "      <td>6</td>\n",
 227 |        "      <td>2.75</td>\n",
 228 |        "      <td>SC</td>\n",
 229 |        "      <td>408</td>\n",
 230 |        "      <td>417-6906</td>\n",
 231 |        "    </tr>\n",
 232 |        "    <tr>\n",
 233 |        "      <th>2749</th>\n",
 234 |        "      <td>95</td>\n",
 235 |        "      <td>0</td>\n",
 236 |        "      <td>229.9</td>\n",
 237 |        "      <td>202.4</td>\n",
 238 |        "      <td>171.4</td>\n",
 239 |        "      <td>14.2</td>\n",
 240 |        "      <td>1</td>\n",
 241 |        "      <td>no</td>\n",
 242 |        "      <td>no</td>\n",
 243 |        "      <td>no</td>\n",
 244 |        "      <td>116</td>\n",
 245 |        "      <td>39.08</td>\n",
 246 |        "      <td>110</td>\n",
 247 |        "      <td>17.20</td>\n",
 248 |        "      <td>105</td>\n",
 249 |        "      <td>7.71</td>\n",
 250 |        "      <td>6</td>\n",
 251 |        "      <td>3.83</td>\n",
 252 |        "      <td>AL</td>\n",
 253 |        "      <td>415</td>\n",
 254 |        "      <td>350-7273</td>\n",
 255 |        "    </tr>\n",
 256 |        "    <tr>\n",
 257 |        "      <th>2970</th>\n",
 258 |        "      <td>90</td>\n",
 259 |        "      <td>22</td>\n",
 260 |        "      <td>124.5</td>\n",
 261 |        "      <td>231.7</td>\n",
 262 |        "      <td>222.2</td>\n",
 263 |        "      <td>6.4</td>\n",
 264 |        "      <td>1</td>\n",
 265 |        "      <td>no</td>\n",
 266 |        "      <td>no</td>\n",
 267 |        "      <td>yes</td>\n",
 268 |        "      <td>94</td>\n",
 269 |        "      <td>21.17</td>\n",
 270 |        "      <td>90</td>\n",
 271 |        "      <td>19.69</td>\n",
 272 |        "      <td>108</td>\n",
 273 |        "      <td>10.00</td>\n",
 274 |        "      <td>12</td>\n",
 275 |        "      <td>1.73</td>\n",
 276 |        "      <td>ND</td>\n",
 277 |        "      <td>415</td>\n",
 278 |        "      <td>329-8638</td>\n",
 279 |        "    </tr>\n",
 280 |        "    <tr>\n",
 281 |        "      <th>2427</th>\n",
 282 |        "      <td>83</td>\n",
 283 |        "      <td>0</td>\n",
 284 |        "      <td>159.3</td>\n",
 285 |        "      <td>202.3</td>\n",
 286 |        "      <td>229.0</td>\n",
 287 |        "      <td>9.5</td>\n",
 288 |        "      <td>2</td>\n",
 289 |        "      <td>no</td>\n",
 290 |        "      <td>no</td>\n",
 291 |        "      <td>no</td>\n",
 292 |        "      <td>104</td>\n",
 293 |        "      <td>27.08</td>\n",
 294 |        "      <td>98</td>\n",
 295 |        "      <td>17.20</td>\n",
 296 |        "      <td>73</td>\n",
 297 |        "      <td>10.31</td>\n",
 298 |        "      <td>3</td>\n",
 299 |        "      <td>2.57</td>\n",
 300 |        "      <td>ID</td>\n",
 301 |        "      <td>415</td>\n",
 302 |        "      <td>350-4297</td>\n",
 303 |        "    </tr>\n",
 304 |        "    <tr>\n",
 305 |        "      <th>615</th>\n",
 306 |        "      <td>48</td>\n",
 307 |        "      <td>43</td>\n",
 308 |        "      <td>172.0</td>\n",
 309 |        "      <td>200.2</td>\n",
 310 |        "      <td>233.1</td>\n",
 311 |        "      <td>8.0</td>\n",
 312 |        "      <td>1</td>\n",
 313 |        "      <td>no</td>\n",
 314 |        "      <td>no</td>\n",
 315 |        "      <td>yes</td>\n",
 316 |        "      <td>111</td>\n",
 317 |        "      <td>29.24</td>\n",
 318 |        "      <td>64</td>\n",
 319 |        "      <td>17.02</td>\n",
 320 |        "      <td>96</td>\n",
 321 |        "      <td>10.49</td>\n",
 322 |        "      <td>5</td>\n",
 323 |        "      <td>2.16</td>\n",
 324 |        "      <td>UT</td>\n",
 325 |        "      <td>510</td>\n",
 326 |        "      <td>340-3075</td>\n",
 327 |        "    </tr>\n",
 328 |        "    <tr>\n",
 329 |        "      <th>2414</th>\n",
 330 |        "      <td>16</td>\n",
 331 |        "      <td>0</td>\n",
 332 |        "      <td>110.0</td>\n",
 333 |        "      <td>147.3</td>\n",
 334 |        "      <td>190.5</td>\n",
 335 |        "      <td>6.4</td>\n",
 336 |        "      <td>0</td>\n",
 337 |        "      <td>no</td>\n",
 338 |        "      <td>yes</td>\n",
 339 |        "      <td>no</td>\n",
 340 |        "      <td>91</td>\n",
 341 |        "      <td>18.70</td>\n",
 342 |        "      <td>75</td>\n",
 343 |        "      <td>12.52</td>\n",
 344 |        "      <td>73</td>\n",
 345 |        "      <td>8.57</td>\n",
 346 |        "      <td>7</td>\n",
 347 |        "      <td>1.73</td>\n",
 348 |        "      <td>IL</td>\n",
 349 |        "      <td>415</td>\n",
 350 |        "      <td>342-2013</td>\n",
 351 |        "    </tr>\n",
 352 |        "  </tbody>\n",
 353 |        "</table>\n",
 354 |        "</div>"
 355 |       ],
 356 |       "text/plain": [
 357 |        "      Account_Length  Vmail_Message  Day_Mins  Eve_Mins  Night_Mins  \\\n",
 358 |        "1648             102              0     174.5     236.8       270.4   \n",
 359 |        "2706             125              0     206.0     198.1       135.9   \n",
 360 |        "3198              53             32     131.2     227.4       178.9   \n",
 361 |        "306              113              0     272.1     268.5       213.8   \n",
 362 |        "2227              41              0     237.8     223.5       217.4   \n",
 363 |        "2749              95              0     229.9     202.4       171.4   \n",
 364 |        "2970              90             22     124.5     231.7       222.2   \n",
 365 |        "2427              83              0     159.3     202.3       229.0   \n",
 366 |        "615               48             43     172.0     200.2       233.1   \n",
 367 |        "2414              16              0     110.0     147.3       190.5   \n",
 368 |        "\n",
 369 |        "      Intl_Mins  CustServ_Calls Churn Intl_Plan Vmail_Plan  Day_Calls  \\\n",
 370 |        "1648        8.5               0    no        no         no         79   \n",
 371 |        "2706       13.2               0    no        no         no        128   \n",
 372 |        "3198       12.8               2    no        no        yes         63   \n",
 373 |        "306         8.5               1   yes        no         no        111   \n",
 374 |        "2227       10.2               2    no        no         no         92   \n",
 375 |        "2749       14.2               1    no        no         no        116   \n",
 376 |        "2970        6.4               1    no        no        yes         94   \n",
 377 |        "2427        9.5               2    no        no         no        104   \n",
 378 |        "615         8.0               1    no        no        yes        111   \n",
 379 |        "2414        6.4               0    no       yes         no         91   \n",
 380 |        "\n",
 381 |        "      Day_Charge  Eve_Calls  Eve_Charge  Night_Calls  Night_Charge  \\\n",
 382 |        "1648       29.67        136       20.13          110         12.17   \n",
 383 |        "2706       35.02         71       16.84          116          6.12   \n",
 384 |        "3198       22.30        125       19.33          105          8.05   \n",
 385 |        "306        46.26        118       22.82          105          9.62   \n",
 386 |        "2227       40.43        155       19.00           90          9.78   \n",
 387 |        "2749       39.08        110       17.20          105          7.71   \n",
 388 |        "2970       21.17         90       19.69          108         10.00   \n",
 389 |        "2427       27.08         98       17.20           73         10.31   \n",
 390 |        "615        29.24         64       17.02           96         10.49   \n",
 391 |        "2414       18.70         75       12.52           73          8.57   \n",
 392 |        "\n",
 393 |        "      Intl_Calls  Intl_Charge State  Area_Code     Phone  \n",
 394 |        "1648           5         2.30    VA        510  398-5788  \n",
 395 |        "2706           3         3.56    WV        415  381-7597  \n",
 396 |        "3198           2         3.46    DE        415  416-9723  \n",
 397 |        "306           10         2.30    VT        415  419-1714  \n",
 398 |        "2227           6         2.75    SC        408  417-6906  \n",
 399 |        "2749           6         3.83    AL        415  350-7273  \n",
 400 |        "2970          12         1.73    ND        415  329-8638  \n",
 401 |        "2427           3         2.57    ID        415  350-4297  \n",
 402 |        "615            5         2.16    UT        510  340-3075  \n",
 403 |        "2414           7         1.73    IL        415  342-2013  "
 404 |       ]
 405 |      },
 406 |      "metadata": {},
 407 |      "output_type": "display_data"
 408 |     },
 409 |     {
 410 |      "name": "stdout",
 411 |      "output_type": "stream",
 412 |      "text": [
 413 |       "(3333, 21)\n"
 414 |      ]
 415 |     }
 416 |    ],
 417 |    "source": [
 418 |     "df = pd.read_csv(\"Churn.csv\")\n",
 419 |     "\n",
 420 |     "pd.options.display.max_columns = df.shape[1]\n",
 421 |     "\n",
 422 |     "display(df.sample(10))\n",
 423 |     "print(df.shape)"
 424 |    ]
 425 |   },
 426 |   {
 427 |    "cell_type": "code",
 428 |    "execution_count": 125,
 429 |    "metadata": {},
 430 |    "outputs": [
 431 |     {
 432 |      "name": "stdout",
 433 |      "output_type": "stream",
 434 |      "text": [
 435 |       "(3333, 18)\n"
 436 |      ]
 437 |     }
 438 |    ],
 439 |    "source": [
 440 |     "df = df.drop([\"Area_Code\", \"Phone\", \"State\"], axis=1)\n",
 441 |     "\n",
 442 |     "print(df.shape)"
 443 |    ]
 444 |   },
 445 |   {
 446 |    "cell_type": "code",
 447 |    "execution_count": 126,
 448 |    "metadata": {},
 449 |    "outputs": [],
 450 |    "source": [
 451 |     "# abul = sorted(df[\"State\"].unique())\n",
 452 |     "# print(abul)"
 453 |    ]
 454 |   },
 455 |   {
 456 |    "cell_type": "code",
 457 |    "execution_count": 127,
 458 |    "metadata": {},
 459 |    "outputs": [
 460 |     {
 461 |      "name": "stdout",
 462 |      "output_type": "stream",
 463 |      "text": [
 464 |       "<class 'pandas.core.frame.DataFrame'>\n",
 465 |       "RangeIndex: 3333 entries, 0 to 3332\n",
 466 |       "Data columns (total 18 columns):\n",
 467 |       " #   Column          Non-Null Count  Dtype  \n",
 468 |       "---  ------          --------------  -----  \n",
 469 |       " 0   Account_Length  3333 non-null   int64  \n",
 470 |       " 1   Vmail_Message   3333 non-null   int64  \n",
 471 |       " 2   Day_Mins        3333 non-null   float64\n",
 472 |       " 3   Eve_Mins        3333 non-null   float64\n",
 473 |       " 4   Night_Mins      3333 non-null   float64\n",
 474 |       " 5   Intl_Mins       3333 non-null   float64\n",
 475 |       " 6   CustServ_Calls  3333 non-null   int64  \n",
 476 |       " 7   Churn           3333 non-null   object \n",
 477 |       " 8   Intl_Plan       3333 non-null   object \n",
 478 |       " 9   Vmail_Plan      3333 non-null   object \n",
 479 |       " 10  Day_Calls       3333 non-null   int64  \n",
 480 |       " 11  Day_Charge      3333 non-null   float64\n",
 481 |       " 12  Eve_Calls       3333 non-null   int64  \n",
 482 |       " 13  Eve_Charge      3333 non-null   float64\n",
 483 |       " 14  Night_Calls     3333 non-null   int64  \n",
 484 |       " 15  Night_Charge    3333 non-null   float64\n",
 485 |       " 16  Intl_Calls      3333 non-null   int64  \n",
 486 |       " 17  Intl_Charge     3333 non-null   float64\n",
 487 |       "dtypes: float64(8), int64(7), object(3)\n",
 488 |       "memory usage: 468.8+ KB\n"
 489 |      ]
 490 |     }
 491 |    ],
 492 |    "source": [
 493 |     "df.info()"
 494 |    ]
 495 |   },
 496 |   {
 497 |    "cell_type": "markdown",
 498 |    "metadata": {},
 499 |    "source": [
 500 |     "#### Data Preprocessing"
 501 |    ]
 502 |   },
 503 |   {
 504 |    "cell_type": "code",
 505 |    "execution_count": 128,
 506 |    "metadata": {},
 507 |    "outputs": [
 508 |     {
 509 |      "data": {
 510 |       "text/plain": [
 511 |        "no     2850\n",
 512 |        "yes     483\n",
 513 |        "Name: Churn, dtype: int64"
 514 |       ]
 515 |      },
 516 |      "execution_count": 128,
 517 |      "metadata": {},
 518 |      "output_type": "execute_result"
 519 |     }
 520 |    ],
 521 |    "source": [
 522 |     "df[\"Churn\"].value_counts()"
 523 |    ]
 524 |   },
 525 |   {
 526 |    "cell_type": "code",
 527 |    "execution_count": 129,
 528 |    "metadata": {},
 529 |    "outputs": [
 530 |     {
 531 |      "data": {
 532 |       "text/plain": [
 533 |        "no     3010\n",
 534 |        "yes     323\n",
 535 |        "Name: Intl_Plan, dtype: int64"
 536 |       ]
 537 |      },
 538 |      "execution_count": 129,
 539 |      "metadata": {},
 540 |      "output_type": "execute_result"
 541 |     }
 542 |    ],
 543 |    "source": [
 544 |     "df[\"Intl_Plan\"].value_counts()"
 545 |    ]
 546 |   },
 547 |   {
 548 |    "cell_type": "code",
 549 |    "execution_count": 130,
 550 |    "metadata": {},
 551 |    "outputs": [
 552 |     {
 553 |      "data": {
 554 |       "text/plain": [
 555 |        "no     2411\n",
 556 |        "yes     922\n",
 557 |        "Name: Vmail_Plan, dtype: int64"
 558 |       ]
 559 |      },
 560 |      "execution_count": 130,
 561 |      "metadata": {},
 562 |      "output_type": "execute_result"
 563 |     }
 564 |    ],
 565 |    "source": [
 566 |     "df[\"Vmail_Plan\"].value_counts()"
 567 |    ]
 568 |   },
 569 |   {
 570 |    "cell_type": "code",
 571 |    "execution_count": 131,
 572 |    "metadata": {},
 573 |    "outputs": [
 574 |     {
 575 |      "data": {
 576 |       "text/html": [
 577 |        "<div>\n",
 578 |        "<style scoped>\n",
 579 |        "    .dataframe tbody tr th:only-of-type {\n",
 580 |        "        vertical-align: middle;\n",
 581 |        "    }\n",
 582 |        "\n",
 583 |        "    .dataframe tbody tr th {\n",
 584 |        "        vertical-align: top;\n",
 585 |        "    }\n",
 586 |        "\n",
 587 |        "    .dataframe thead th {\n",
 588 |        "        text-align: right;\n",
 589 |        "    }\n",
 590 |        "</style>\n",
 591 |        "<table border=\"1\" class=\"dataframe\">\n",
 592 |        "  <thead>\n",
 593 |        "    <tr style=\"text-align: right;\">\n",
 594 |        "      <th></th>\n",
 595 |        "      <th>Account_Length</th>\n",
 596 |        "      <th>Vmail_Message</th>\n",
 597 |        "      <th>Day_Mins</th>\n",
 598 |        "      <th>Eve_Mins</th>\n",
 599 |        "      <th>Night_Mins</th>\n",
 600 |        "      <th>Intl_Mins</th>\n",
 601 |        "      <th>CustServ_Calls</th>\n",
 602 |        "      <th>Churn</th>\n",
 603 |        "      <th>Intl_Plan</th>\n",
 604 |        "      <th>Vmail_Plan</th>\n",
 605 |        "      <th>Day_Calls</th>\n",
 606 |        "      <th>Day_Charge</th>\n",
 607 |        "      <th>Eve_Calls</th>\n",
 608 |        "      <th>Eve_Charge</th>\n",
 609 |        "      <th>Night_Calls</th>\n",
 610 |        "      <th>Night_Charge</th>\n",
 611 |        "      <th>Intl_Calls</th>\n",
 612 |        "      <th>Intl_Charge</th>\n",
 613 |        "    </tr>\n",
 614 |        "  </thead>\n",
 615 |        "  <tbody>\n",
 616 |        "    <tr>\n",
 617 |        "      <th>0</th>\n",
 618 |        "      <td>128</td>\n",
 619 |        "      <td>25</td>\n",
 620 |        "      <td>265.1</td>\n",
 621 |        "      <td>197.4</td>\n",
 622 |        "      <td>244.7</td>\n",
 623 |        "      <td>10.0</td>\n",
 624 |        "      <td>1</td>\n",
 625 |        "      <td>0</td>\n",
 626 |        "      <td>0</td>\n",
 627 |        "      <td>1</td>\n",
 628 |        "      <td>110</td>\n",
 629 |        "      <td>45.07</td>\n",
 630 |        "      <td>99</td>\n",
 631 |        "      <td>16.78</td>\n",
 632 |        "      <td>91</td>\n",
 633 |        "      <td>11.01</td>\n",
 634 |        "      <td>3</td>\n",
 635 |        "      <td>2.70</td>\n",
 636 |        "    </tr>\n",
 637 |        "    <tr>\n",
 638 |        "      <th>1</th>\n",
 639 |        "      <td>107</td>\n",
 640 |        "      <td>26</td>\n",
 641 |        "      <td>161.6</td>\n",
 642 |        "      <td>195.5</td>\n",
 643 |        "      <td>254.4</td>\n",
 644 |        "      <td>13.7</td>\n",
 645 |        "      <td>1</td>\n",
 646 |        "      <td>0</td>\n",
 647 |        "      <td>0</td>\n",
 648 |        "      <td>1</td>\n",
 649 |        "      <td>123</td>\n",
 650 |        "      <td>27.47</td>\n",
 651 |        "      <td>103</td>\n",
 652 |        "      <td>16.62</td>\n",
 653 |        "      <td>103</td>\n",
 654 |        "      <td>11.45</td>\n",
 655 |        "      <td>3</td>\n",
 656 |        "      <td>3.70</td>\n",
 657 |        "    </tr>\n",
 658 |        "    <tr>\n",
 659 |        "      <th>2</th>\n",
 660 |        "      <td>137</td>\n",
 661 |        "      <td>0</td>\n",
 662 |        "      <td>243.4</td>\n",
 663 |        "      <td>121.2</td>\n",
 664 |        "      <td>162.6</td>\n",
 665 |        "      <td>12.2</td>\n",
 666 |        "      <td>0</td>\n",
 667 |        "      <td>0</td>\n",
 668 |        "      <td>0</td>\n",
 669 |        "      <td>0</td>\n",
 670 |        "      <td>114</td>\n",
 671 |        "      <td>41.38</td>\n",
 672 |        "      <td>110</td>\n",
 673 |        "      <td>10.30</td>\n",
 674 |        "      <td>104</td>\n",
 675 |        "      <td>7.32</td>\n",
 676 |        "      <td>5</td>\n",
 677 |        "      <td>3.29</td>\n",
 678 |        "    </tr>\n",
 679 |        "    <tr>\n",
 680 |        "      <th>3</th>\n",
 681 |        "      <td>84</td>\n",
 682 |        "      <td>0</td>\n",
 683 |        "      <td>299.4</td>\n",
 684 |        "      <td>61.9</td>\n",
 685 |        "      <td>196.9</td>\n",
 686 |        "      <td>6.6</td>\n",
 687 |        "      <td>2</td>\n",
 688 |        "      <td>0</td>\n",
 689 |        "      <td>1</td>\n",
 690 |        "      <td>0</td>\n",
 691 |        "      <td>71</td>\n",
 692 |        "      <td>50.90</td>\n",
 693 |        "      <td>88</td>\n",
 694 |        "      <td>5.26</td>\n",
 695 |        "      <td>89</td>\n",
 696 |        "      <td>8.86</td>\n",
 697 |        "      <td>7</td>\n",
 698 |        "      <td>1.78</td>\n",
 699 |        "    </tr>\n",
 700 |        "    <tr>\n",
 701 |        "      <th>4</th>\n",
 702 |        "      <td>75</td>\n",
 703 |        "      <td>0</td>\n",
 704 |        "      <td>166.7</td>\n",
 705 |        "      <td>148.3</td>\n",
 706 |        "      <td>186.9</td>\n",
 707 |        "      <td>10.1</td>\n",
 708 |        "      <td>3</td>\n",
 709 |        "      <td>0</td>\n",
 710 |        "      <td>1</td>\n",
 711 |        "      <td>0</td>\n",
 712 |        "      <td>113</td>\n",
 713 |        "      <td>28.34</td>\n",
 714 |        "      <td>122</td>\n",
 715 |        "      <td>12.61</td>\n",
 716 |        "      <td>121</td>\n",
 717 |        "      <td>8.41</td>\n",
 718 |        "      <td>3</td>\n",
 719 |        "      <td>2.73</td>\n",
 720 |        "    </tr>\n",
 721 |        "    <tr>\n",
 722 |        "      <th>5</th>\n",
 723 |        "      <td>118</td>\n",
 724 |        "      <td>0</td>\n",
 725 |        "      <td>223.4</td>\n",
 726 |        "      <td>220.6</td>\n",
 727 |        "      <td>203.9</td>\n",
 728 |        "      <td>6.3</td>\n",
 729 |        "      <td>0</td>\n",
 730 |        "      <td>0</td>\n",
 731 |        "      <td>1</td>\n",
 732 |        "      <td>0</td>\n",
 733 |        "      <td>98</td>\n",
 734 |        "      <td>37.98</td>\n",
 735 |        "      <td>101</td>\n",
 736 |        "      <td>18.75</td>\n",
 737 |        "      <td>118</td>\n",
 738 |        "      <td>9.18</td>\n",
 739 |        "      <td>6</td>\n",
 740 |        "      <td>1.70</td>\n",
 741 |        "    </tr>\n",
 742 |        "    <tr>\n",
 743 |        "      <th>6</th>\n",
 744 |        "      <td>121</td>\n",
 745 |        "      <td>24</td>\n",
 746 |        "      <td>218.2</td>\n",
 747 |        "      <td>348.5</td>\n",
 748 |        "      <td>212.6</td>\n",
 749 |        "      <td>7.5</td>\n",
 750 |        "      <td>3</td>\n",
 751 |        "      <td>0</td>\n",
 752 |        "      <td>0</td>\n",
 753 |        "      <td>1</td>\n",
 754 |        "      <td>88</td>\n",
 755 |        "      <td>37.09</td>\n",
 756 |        "      <td>108</td>\n",
 757 |        "      <td>29.62</td>\n",
 758 |        "      <td>118</td>\n",
 759 |        "      <td>9.57</td>\n",
 760 |        "      <td>7</td>\n",
 761 |        "      <td>2.03</td>\n",
 762 |        "    </tr>\n",
 763 |        "    <tr>\n",
 764 |        "      <th>7</th>\n",
 765 |        "      <td>147</td>\n",
 766 |        "      <td>0</td>\n",
 767 |        "      <td>157.0</td>\n",
 768 |        "      <td>103.1</td>\n",
 769 |        "      <td>211.8</td>\n",
 770 |        "      <td>7.1</td>\n",
 771 |        "      <td>0</td>\n",
 772 |        "      <td>0</td>\n",
 773 |        "      <td>1</td>\n",
 774 |        "      <td>0</td>\n",
 775 |        "      <td>79</td>\n",
 776 |        "      <td>26.69</td>\n",
 777 |        "      <td>94</td>\n",
 778 |        "      <td>8.76</td>\n",
 779 |        "      <td>96</td>\n",
 780 |        "      <td>9.53</td>\n",
 781 |        "      <td>6</td>\n",
 782 |        "      <td>1.92</td>\n",
 783 |        "    </tr>\n",
 784 |        "    <tr>\n",
 785 |        "      <th>8</th>\n",
 786 |        "      <td>117</td>\n",
 787 |        "      <td>0</td>\n",
 788 |        "      <td>184.5</td>\n",
 789 |        "      <td>351.6</td>\n",
 790 |        "      <td>215.8</td>\n",
 791 |        "      <td>8.7</td>\n",
 792 |        "      <td>1</td>\n",
 793 |        "      <td>0</td>\n",
 794 |        "      <td>0</td>\n",
 795 |        "      <td>0</td>\n",
 796 |        "      <td>97</td>\n",
 797 |        "      <td>31.37</td>\n",
 798 |        "      <td>80</td>\n",
 799 |        "      <td>29.89</td>\n",
 800 |        "      <td>90</td>\n",
 801 |        "      <td>9.71</td>\n",
 802 |        "      <td>4</td>\n",
 803 |        "      <td>2.35</td>\n",
 804 |        "    </tr>\n",
 805 |        "    <tr>\n",
 806 |        "      <th>9</th>\n",
 807 |        "      <td>141</td>\n",
 808 |        "      <td>37</td>\n",
 809 |        "      <td>258.6</td>\n",
 810 |        "      <td>222.0</td>\n",
 811 |        "      <td>326.4</td>\n",
 812 |        "      <td>11.2</td>\n",
 813 |        "      <td>0</td>\n",
 814 |        "      <td>0</td>\n",
 815 |        "      <td>1</td>\n",
 816 |        "      <td>1</td>\n",
 817 |        "      <td>84</td>\n",
 818 |        "      <td>43.96</td>\n",
 819 |        "      <td>111</td>\n",
 820 |        "      <td>18.87</td>\n",
 821 |        "      <td>97</td>\n",
 822 |        "      <td>14.69</td>\n",
 823 |        "      <td>5</td>\n",
 824 |        "      <td>3.02</td>\n",
 825 |        "    </tr>\n",
 826 |        "  </tbody>\n",
 827 |        "</table>\n",
 828 |        "</div>"
 829 |       ],
 830 |       "text/plain": [
 831 |        "   Account_Length  Vmail_Message  Day_Mins  Eve_Mins  Night_Mins  Intl_Mins  \\\n",
 832 |        "0             128             25     265.1     197.4       244.7       10.0   \n",
 833 |        "1             107             26     161.6     195.5       254.4       13.7   \n",
 834 |        "2             137              0     243.4     121.2       162.6       12.2   \n",
 835 |        "3              84              0     299.4      61.9       196.9        6.6   \n",
 836 |        "4              75              0     166.7     148.3       186.9       10.1   \n",
 837 |        "5             118              0     223.4     220.6       203.9        6.3   \n",
 838 |        "6             121             24     218.2     348.5       212.6        7.5   \n",
 839 |        "7             147              0     157.0     103.1       211.8        7.1   \n",
 840 |        "8             117              0     184.5     351.6       215.8        8.7   \n",
 841 |        "9             141             37     258.6     222.0       326.4       11.2   \n",
 842 |        "\n",
 843 |        "   CustServ_Calls  Churn  Intl_Plan  Vmail_Plan  Day_Calls  Day_Charge  \\\n",
 844 |        "0               1      0          0           1        110       45.07   \n",
 845 |        "1               1      0          0           1        123       27.47   \n",
 846 |        "2               0      0          0           0        114       41.38   \n",
 847 |        "3               2      0          1           0         71       50.90   \n",
 848 |        "4               3      0          1           0        113       28.34   \n",
 849 |        "5               0      0          1           0         98       37.98   \n",
 850 |        "6               3      0          0           1         88       37.09   \n",
 851 |        "7               0      0          1           0         79       26.69   \n",
 852 |        "8               1      0          0           0         97       31.37   \n",
 853 |        "9               0      0          1           1         84       43.96   \n",
 854 |        "\n",
 855 |        "   Eve_Calls  Eve_Charge  Night_Calls  Night_Charge  Intl_Calls  Intl_Charge  \n",
 856 |        "0         99       16.78           91         11.01           3         2.70  \n",
 857 |        "1        103       16.62          103         11.45           3         3.70  \n",
 858 |        "2        110       10.30          104          7.32           5         3.29  \n",
 859 |        "3         88        5.26           89          8.86           7         1.78  \n",
 860 |        "4        122       12.61          121          8.41           3         2.73  \n",
 861 |        "5        101       18.75          118          9.18           6         1.70  \n",
 862 |        "6        108       29.62          118          9.57           7         2.03  \n",
 863 |        "7         94        8.76           96          9.53           6         1.92  \n",
 864 |        "8         80       29.89           90          9.71           4         2.35  \n",
 865 |        "9        111       18.87           97         14.69           5         3.02  "
 866 |       ]
 867 |      },
 868 |      "metadata": {},
 869 |      "output_type": "display_data"
 870 |     },
 871 |     {
 872 |      "name": "stdout",
 873 |      "output_type": "stream",
 874 |      "text": [
 875 |       "<class 'pandas.core.frame.DataFrame'>\n",
 876 |       "RangeIndex: 3333 entries, 0 to 3332\n",
 877 |       "Data columns (total 18 columns):\n",
 878 |       " #   Column          Non-Null Count  Dtype  \n",
 879 |       "---  ------          --------------  -----  \n",
 880 |       " 0   Account_Length  3333 non-null   int64  \n",
 881 |       " 1   Vmail_Message   3333 non-null   int64  \n",
 882 |       " 2   Day_Mins        3333 non-null   float64\n",
 883 |       " 3   Eve_Mins        3333 non-null   float64\n",
 884 |       " 4   Night_Mins      3333 non-null   float64\n",
 885 |       " 5   Intl_Mins       3333 non-null   float64\n",
 886 |       " 6   CustServ_Calls  3333 non-null   int64  \n",
 887 |       " 7   Churn           3333 non-null   int64  \n",
 888 |       " 8   Intl_Plan       3333 non-null   int64  \n",
 889 |       " 9   Vmail_Plan      3333 non-null   int64  \n",
 890 |       " 10  Day_Calls       3333 non-null   int64  \n",
 891 |       " 11  Day_Charge      3333 non-null   float64\n",
 892 |       " 12  Eve_Calls       3333 non-null   int64  \n",
 893 |       " 13  Eve_Charge      3333 non-null   float64\n",
 894 |       " 14  Night_Calls     3333 non-null   int64  \n",
 895 |       " 15  Night_Charge    3333 non-null   float64\n",
 896 |       " 16  Intl_Calls      3333 non-null   int64  \n",
 897 |       " 17  Intl_Charge     3333 non-null   float64\n",
 898 |       "dtypes: float64(8), int64(10)\n",
 899 |       "memory usage: 468.8 KB\n"
 900 |      ]
 901 |     }
 902 |    ],
 903 |    "source": [
 904 |     "df[\"Churn\"] = df[\"Churn\"].replace({\"no\":0, \"yes\": 1})\n",
 905 |     "df[\"Intl_Plan\"] = df[\"Intl_Plan\"].replace({\"no\":0, \"yes\": 1})\n",
 906 |     "df[\"Vmail_Plan\"] = df[\"Vmail_Plan\"].replace({\"no\":0, \"yes\": 1})\n",
 907 |     "\n",
 908 |     "display(df.head(10))\n",
 909 |     "df.info()"
 910 |    ]
 911 |   },
 912 |   {
 913 |    "cell_type": "markdown",
 914 |    "metadata": {},
 915 |    "source": [
 916 |     "#### Target variable visualization"
 917 |    ]
 918 |   },
 919 |   {
 920 |    "cell_type": "code",
 921 |    "execution_count": 132,
 922 |    "metadata": {},
 923 |    "outputs": [
 924 |     {
 925 |      "data": {
 926 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAATIElEQVR4nO3dfZBdd13H8ffHpFBaqCSwiaEP8mCGWtAWWGuhI6Kh2iKaoHQsDrrjVOMDAlVHJzpq0dGZ6jCOD6NIBpB1xEJEMBEQyawCOgi4LUFaak1FSEPXZFukFYqlKV//uCenS9wkt5u952z2vl8zO79zfuece76dub2fnKffSVUhSRLA1/RdgCRp5TAUJEktQ0GS1DIUJEktQ0GS1DIUJEmtkYVCkjclOZzklgV965PsTbK/adctWPZLSe5IcnuS7x5VXZKk4xvlkcKbgSuP6dsBzFTVZmCmmSfJRcA1wDOabf44yZoR1iZJWsTIQqGqPgh87pjurcB0Mz0NbFvQ/9aqeqCq/hO4A7h0VLVJkha3tuP9bayqOYCqmkuyoek/F/jwgvUONn3/T5LtwHaAs88++zkXXnjhCMuVpNXnpptuuruqJhZb1nUoHE8W6Vt0/I2q2gnsBJicnKzZ2dlR1iVJq06SzxxvWdd3Hx1KsgmgaQ83/QeB8xesdx5wV8e1SdLY6zoU9gBTzfQUsHtB/zVJHp3kKcBm4KMd1yZJY29kp4+S3Ai8AHhikoPA9cANwK4k1wIHgKsBqurWJLuATwJHgFdU1UOjqk2StLiRhUJVvew4i7YcZ/3fAn5rVPVIkk7OJ5olSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSS1DQZLUMhQkSa1eQiHJq5PckuTWJNc1feuT7E2yv2nX9VGbJI2zzkMhyTOBHwcuBS4GXpxkM7ADmKmqzcBMMy9J6lAfRwrfCHy4qu6vqiPAB4CXAFuB6WadaWBbD7VJ0ljrIxRuAZ6f5AlJzgJexOD9zBurag6gaTcstnGS7Ulmk8zOz893VrQkjYPOQ6GqbgN+G9gLvBf4OINXcA67/c6qmqyqyYmJiRFVKUnjqZcLzVX1xqp6dlU9H/gcsB84lGQTQNMe7qM2SRpnfd19tKFpLwC+H7gR2ANMNatMAbv7qE2Sxtnanvb7V0meADwIvKKq/jvJDcCuJNcCB4Cre6pNksZWL6FQVd+2SN89wJYeypEkNXyiWZLUMhQkSS1DQZLUMhQkSa2+7j5aEc49/wLuOnhn32VolXrSeefz2TsP9F2G9IiMdSjcdfBOfvD1H+q7DK1Sb/uJ5/VdgvSIefpIktQyFCRJLUNBktQyFCRJLUNBktQyFCRJLUNBktTq630KP5vk1iS3JLkxyZlJ1ifZm2R/067rozZJGmedh0KSc4FXAZNV9UxgDXANsAOYqarNwEwzL0nqUF+nj9YCj0myFjgLuAvYCkw3y6eBbf2UJknjq/NQqKrPAq9l8Ha1OeDeqnofsLGq5pp15oANi22fZHuS2SSz8/PzXZUtSWOhj9NH6xgcFTwFeBJwdpKXD7t9Ve2sqsmqmpyYmBhVmZI0lvo4ffRC4D+rar6qHgTeATwPOJRkE0DTHu6hNkkaa32EwgHgsiRnJQmD9zLfBuwBppp1poDdPdQmSWOt86Gzq+ojSd4O3AwcAT4G7AQeC+xKci2D4Li669okadz18j6FqroeuP6Y7gcYHDVIknriE82SpJahIElqGQqSpJahIElqGQqSpJahIElqGQqSpJahIElqGQqSpJahIElqGQqSpJahIElq9fGSnacn2bfg774k1yVZn2Rvkv1Nu67r2iRp3PXxOs7bq+qSqroEeA5wP/BOYAcwU1WbgZlmXpLUob5PH20B/qOqPsPgFZ3TTf80sK2voiRpXPUdCtcANzbTG6tqDqBpN/RWlSSNqd5CIcmjgO8D/vIRbrc9yWyS2fn5+dEUJ0ljqs8jhauAm6vqUDN/KMkmgKY9vNhGVbWzqiaranJiYqKjUiVpPPQZCi/j4VNHAHuAqWZ6CtjdeUWSNOZ6CYUkZwFXAO9Y0H0DcEWS/c2yG/qoTZLG2do+dlpV9wNPOKbvHgZ3I0mSetL33UeSpBXEUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVLLUJAktQwFSVKrr6GzH5/k7Un+LcltSZ6bZH2SvUn2N+26PmqTpHHW15HC7wPvraoLgYuB24AdwExVbQZmmnlJUoc6D4Uk5wDPB94IUFVfrqrPA1uB6Wa1aWBb17VJ0rjr40jhqcA88KdJPpbkDUnOBjZW1RxA025YbOMk25PMJpmdn5/vrmpJGgN9hMJa4NnA66rqWcAXeQSniqpqZ1VNVtXkxMTEqGqUpLHURygcBA5W1Uea+bczCIlDSTYBNO3hHmqTpLHWeShU1X8BdyZ5etO1BfgksAeYavqmgN1d1yZJ425tT/t9JfCWJI8CPgX8KIOA2pXkWuAAcHVPtUnS2OolFKpqHzC5yKItHZciSVpgqNNHSS4fpk+SdHob9prCHw7ZJ0k6jZ3w9FGS5wLPAyaS/NyCRecAa0ZZmCSpeye7pvAo4LHNeo9b0H8f8NJRFSVJ6scJQ6GqPgB8IMmbq+ozHdUkSerJsHcfPTrJTuDJC7epqu8cRVGSpH4MGwp/CfwJ8AbgodGVI0nq07ChcKSqXjfSSiRJvRv2ltS/SfLTSTY1L8NZn2T9SCuTJHVu2COFo2MS/cKCvmIwDLYkaZUYKhSq6imjLkSS1L9hh7k4K8mvNHcgkWRzkhePtjRJUteGvabwp8CXGTzdDIN3IvzmSCqSJPVm2FB4WlX9DvAgQFV9CchSd5rk00k+kWRfktmmb32SvUn2N+26pX6+JGlphg2FLyd5DIOLyyR5GvDAKe77O6rqkqo6OoT2DmCmqjYDMzyCV3RKkpbHsKFwPfBe4Pwkb2Hwo/2Ly1zLVmC6mZ4Gti3z50uSTmLYu4/2JrkZuIzBaaNXV9Xdp7DfAt6XpIDXV9VOYGNVzTX7m0uyYbENk2wHtgNccMEFp1CCJOlYw9599BIGTzW/u6reBRxJsu0U9nt5VT0buAp4RZLnD7thVe2sqsmqmpyYmDiFEiRJxxr69FFV3Xt0pqo+z+CU0pJU1V1Nexh4J3ApcCjJJoCmPbzUz5ckLc2wobDYekt6v3OSs5M87ug08F3ALcAeHn5yegrYvZTPlyQt3bA/7LNJfhf4IwbXA14J3LTEfW4E3pnk6P7/oqrem+RfgF1JrgUOAFcv8fMlSUs0bCi8EvhV4G3N/PuAX1nKDqvqU8DFi/TfA2xZymdKkpbHSUMhyRpgd1W9sIN6JEk9Ouk1hap6CLg/ydd2UI8kqUfDnj76X+ATSfYCXzzaWVWvGklVkqReDBsK727+JEmr2LBPNE83Yx9dUFW3j7gmSVJPhn2i+XuBfQzGPyLJJUn2jLAuSVIPhn147TUMnjr+PEBV7QN8G5skrTLDhsKRhcNcNGq5i5Ek9WvYC823JPkhYE2SzcCrgA+NrixJUh+GPVJ4JfAMBi/W+QvgXuC6EdUkSerJCY8UkpwJ/CTwDcAngOdW1ZEuCpMkde9kRwrTwCSDQLgKeO3IK5Ik9eZk1xQuqqpvAkjyRuCjoy9JktSXkx0pPHh0YrlPGyVZk+RjSd7VzK9PsjfJ/qZdt5z7kySd3MlC4eIk9zV//wN889HpJPed4r5fDdy2YH4HMFNVm4GZZl6S1KEThkJVramqc5q/x1XV2gXT5yx1p0nOA74HeMOC7q0MrmHQtNuW+vmSpKUZ9pbU5fZ7wC8CX1nQt7Gq5gCadsNiGybZnmQ2yez8/PzIC5WkcdJ5KCR5MXC4qpb0Os+q2llVk1U1OTExsczVSdJ4G/aJ5uV0OfB9SV4EnAmck+TPgUNJNlXVXJJNwOEeapOksdb5kUJV/VJVnVdVTwauAf6+ql4O7AGmmtWmgN1d1yZJ466vawqLuQG4Isl+4IpmXpLUoT5OH7Wq6v3A+5vpe4AtfdYjSeNuJR0pSJJ6ZihIklqGgiSpZShIklqGgiSpZShIklqGgiSpZShIklqGgiSpZShIklqGgiSpZShIklp9vGTnzCQfTfLxJLcm+fWmf32SvUn2N+26rmuTpHHXx5HCA8B3VtXFwCXAlUkuA3YAM1W1GZhp5iVJHerjJTtVVV9oZs9o/grYCkw3/dPAtq5rk6Rx18s1hSRrkuxj8MrNvVX1EWBjVc0BNO2GPmqTpHHWSyhU1UNVdQlwHnBpkmcOu22S7Ulmk8zOz8+PrEZJGke93n1UVZ9n8Oa1K4FDSTYBNO3h42yzs6omq2pyYmKiq1IlaSz0cffRRJLHN9OPAV4I/BuwB5hqVpsCdnddmySNuz7e0bwJmE6yhkEo7aqqdyX5Z2BXkmuBA8DVPdQmSWOt81Coqn8FnrVI/z3Alq7rkSQ9zCeaJUktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEktQ0GS1DIUJEmtPt6ncH6Sf0hyW5Jbk7y66V+fZG+S/U27ruvaJGnc9XGkcAT4+ar6RuAy4BVJLgJ2ADNVtRmYaeYlSR3qPBSqaq6qbm6m/we4DTgX2ApMN6tNA9u6rk2Sxl2v1xSSPJnBC3c+AmysqjkYBAew4TjbbE8ym2R2fn6+s1olaRz0FgpJHgv8FXBdVd037HZVtbOqJqtqcmJiYnQFStIY6iUUkpzBIBDeUlXvaLoPJdnULN8EHO6jNkkaZ33cfRTgjcBtVfW7CxbtAaaa6Slgd9e1SdK4W9vDPi8Hfhj4RJJ9Td8vAzcAu5JcCxwAru6hNkkaa52HQlX9E5DjLN7SZS2SpK/mE82SpJahIElqGQqSpJahIElqGQqSpJahIElq9fGcgjQevmYtg2c1peX3pPPO57N3Hlj2zzUUpFH5yhF+8PUf6rsKrVJv+4nnjeRzPX0kSWoZCpKklqEgSWoZCpKklqEgSWr19ZKdNyU5nOSWBX3rk+xNsr9p1/VRmySNs76OFN4MXHlM3w5gpqo2AzPNvCSpQ72EQlV9EPjcMd1bgelmehrY1mVNkqSVdU1hY1XNATTthsVWSrI9yWyS2fn5+U4LlKTVbiWFwlCqamdVTVbV5MTERN/lSNKqspJC4VCSTQBNe7jneiRp7KykUNgDTDXTU8DuHmuRpLHU1y2pNwL/DDw9ycEk1wI3AFck2Q9c0cxLkjrUyyipVfWy4yza0mkhkqSvspJOH0mSemYoSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqWUoSJJahoIkqbXiQiHJlUluT3JHkh191yNJ42RFhUKSNcAfAVcBFwEvS3JRv1VJ0vhYUaEAXArcUVWfqqovA28FtvZckySNjVRV3zW0krwUuLKqfqyZ/2HgW6vqZxassx3Y3sw+Hbj9FHb5RODuU9heOhG/XxqlU/l+fX1VTSy2oJc3r51AFun7qtSqqp3AzmXZWTJbVZPL8VnSsfx+aZRG9f1aaaePDgLnL5g/D7irp1okaeystFD4F2BzkqckeRRwDbCn55okaWysqNNHVXUkyc8AfwesAd5UVbeOcJfLchpKOg6/XxqlkXy/VtSFZklSv1ba6SNJUo8MBUlSa9WHwsmGzcjAHzTL/zXJs/uoU6enIb5fL0hyb5J9zd+v9VGnTk9J3pTkcJJbjrN82X+/VnUoDDlsxlXA5uZvO/C6TovUaesRDMvyj1V1SfP3G50WqdPdm4ErT7B82X+/VnUoMNywGVuBP6uBDwOPT7Kp60J1WnJYFo1UVX0Q+NwJVln236/VHgrnAncumD/Y9D3SdaTFDPvdeW6Sjyf52yTP6KY0jYll//1aUc8pjMBJh80Ych1pMcN8d25mMM7MF5K8CPhrBof60nJY9t+v1X6kMMywGQ6toaU66Xenqu6rqi800+8BzkjyxO5K1Cq37L9fqz0Uhhk2Yw/wI81V/MuAe6tqrutCdVo66fcrydclSTN9KYP/5+7pvFKtVsv++7WqTx8db9iMJD/ZLP8T4D3Ai4A7gPuBH+2rXp1ehvx+vRT4qSRHgC8B15TDCGhISW4EXgA8MclB4HrgDBjd75fDXEiSWqv99JEk6REwFCRJLUNBktQyFCRJLUNBktQyFKSTaJ41eGuS/0jyySTvSbI9ybv6rk1aboaCdALNg2fvBN5fVU+rqouAXwY2nuLnrupnhHT68ospndh3AA82DwoBUFX7kjwe2JLk7cAzgZuAl1dVJfk0MFlVdyeZBF5bVS9I8hrgScCTgbuT/DtwAfDUpv29qvqD7v7TpP/PIwXpxI7+4C/mWcB1DN6l8FTg8iE+7znA1qr6oWb+QuC7GQzDfX2SM06pWukUGQrS0n20qg5W1VeAfQyOAE5mT1V9acH8u6vqgaq6GzjMKZ6Wkk6VoSCd2K0M/nW/mAcWTD/Ew6djj/Dw/1tnHrPNF4f8DKkXhoJ0Yn8PPDrJjx/tSPItwLefYJtP83CQ/MDoSpOWn6EgnUAzoulLgCuaW1JvBV7Dices/3Xg95P8I4N//UunDUdJlSS1PFKQJLUMBUlSy1CQJLUMBUlSy1CQJLUMBUlSy1CQJLX+D20jWftLfdavAAAAAElFTkSuQmCC\n",
 927 |       "text/plain": [
 928 |        "<Figure size 432x288 with 1 Axes>"
 929 |       ]
 930 |      },
 931 |      "metadata": {
 932 |       "needs_background": "light"
 933 |      },
 934 |      "output_type": "display_data"
 935 |     }
 936 |    ],
 937 |    "source": [
 938 |     "sns.histplot(x=\"Churn\", data=df, binwidth=0.5, stat=\"percent\")\n",
 939 |     "plt.yticks(np.arange(0,101,10))\n",
 940 |     "plt.xticks(np.arange(0,1.1,0.5))\n",
 941 |     "plt.show()"
 942 |    ]
 943 |   },
 944 |   {
 945 |    "cell_type": "markdown",
 946 |    "metadata": {},
 947 |    "source": [
 948 |     "#### Feature and target variable separation"
 949 |    ]
 950 |   },
 951 |   {
 952 |    "cell_type": "code",
 953 |    "execution_count": 133,
 954 |    "metadata": {},
 955 |    "outputs": [
 956 |     {
 957 |      "name": "stdout",
 958 |      "output_type": "stream",
 959 |      "text": [
 960 |       "(3333, 17) (3333, 1)\n"
 961 |      ]
 962 |     }
 963 |    ],
 964 |    "source": [
 965 |     "X = df.drop(\"Churn\", axis=1)\n",
 966 |     "y = df[[\"Churn\"]]\n",
 967 |     "\n",
 968 |     "print(X.shape, y.shape)"
 969 |    ]
 970 |   },
 971 |   {
 972 |    "cell_type": "markdown",
 973 |    "metadata": {},
 974 |    "source": [
 975 |     "#### Train test separation"
 976 |    ]
 977 |   },
 978 |   {
 979 |    "cell_type": "code",
 980 |    "execution_count": 134,
 981 |    "metadata": {},
 982 |    "outputs": [
 983 |     {
 984 |      "name": "stdout",
 985 |      "output_type": "stream",
 986 |      "text": [
 987 |       "(2666, 17) (667, 17) (2666, 1) (667, 1)\n"
 988 |      ]
 989 |     }
 990 |    ],
 991 |    "source": [
 992 |     "X_train, X_test, y_train, y_test = train_test_split(X, \n",
 993 |     "                                                    y,\n",
 994 |     "                                                    test_size=0.2, \n",
 995 |     "                                                    random_state=42,\n",
 996 |     "                                                    stratify=y)\n",
 997 |     "\n",
 998 |     "print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
 999 |    ]
1000 |   },
1001 |   {
1002 |    "cell_type": "code",
1003 |    "execution_count": 135,
1004 |    "metadata": {},
1005 |    "outputs": [
1006 |     {
1007 |      "data": {
1008 |       "text/plain": [
1009 |        "0    85.52138\n",
1010 |        "1    14.47862\n",
1011 |        "Name: Churn, dtype: float64"
1012 |       ]
1013 |      },
1014 |      "execution_count": 135,
1015 |      "metadata": {},
1016 |      "output_type": "execute_result"
1017 |     }
1018 |    ],
1019 |    "source": [
1020 |     "y_train[\"Churn\"].value_counts(normalize=True)*100"
1021 |    ]
1022 |   },
1023 |   {
1024 |    "cell_type": "code",
1025 |    "execution_count": 136,
1026 |    "metadata": {},
1027 |    "outputs": [
1028 |     {
1029 |      "data": {
1030 |       "text/plain": [
1031 |        "0    85.457271\n",
1032 |        "1    14.542729\n",
1033 |        "Name: Churn, dtype: float64"
1034 |       ]
1035 |      },
1036 |      "execution_count": 136,
1037 |      "metadata": {},
1038 |      "output_type": "execute_result"
1039 |     }
1040 |    ],
1041 |    "source": [
1042 |     "y_test[\"Churn\"].value_counts(normalize=True)*100"
1043 |    ]
1044 |   },
1045 |   {
1046 |    "cell_type": "markdown",
1047 |    "metadata": {},
1048 |    "source": [
1049 |     "#### Training: Logistic Regression"
1050 |    ]
1051 |   },
1052 |   {
1053 |    "cell_type": "code",
1054 |    "execution_count": null,
1055 |    "metadata": {},
1056 |    "outputs": [],
1057 |    "source": []
1058 |   },
1059 |   {
1060 |    "cell_type": "code",
1061 |    "execution_count": null,
1062 |    "metadata": {},
1063 |    "outputs": [],
1064 |    "source": []
1065 |   }
1066 |  ],
1067 |  "metadata": {
1068 |   "kernelspec": {
1069 |    "display_name": "Python 3 (ipykernel)",
1070 |    "language": "python",
1071 |    "name": "python3"
1072 |   },
1073 |   "language_info": {
1074 |    "codemirror_mode": {
1075 |     "name": "ipython",
1076 |     "version": 3
1077 |    },
1078 |    "file_extension": ".py",
1079 |    "mimetype": "text/x-python",
1080 |    "name": "python",
1081 |    "nbconvert_exporter": "python",
1082 |    "pygments_lexer": "ipython3",
1083 |    "version": "3.8.5"
1084 |   }
1085 |  },
1086 |  "nbformat": 4,
1087 |  "nbformat_minor": 4
1088 | }
1089 | 


--------------------------------------------------------------------------------
/5. Classification: Accuracy Metrics.ipynb:
--------------------------------------------------------------------------------
   1 | {
   2 |  "cells": [
   3 |   {
   4 |    "cell_type": "markdown",
   5 |    "metadata": {},
   6 |    "source": [
   7 |     "# Lecture 5: Supervised Learning - Classification: Accuracy Metrics\n",
   8 |     "\n",
   9 |     "Instructor: Md Shahidullah Kawsar\n",
  10 |     "<br>Data Scientist, IDARE, Houston, TX, USA\n",
  11 |     "\n",
  12 |     "#### Objectives:\n",
  13 |     "- Supervised Learning - Classification: Logistic Regression\n",
  14 |     "- Confusion Matrix\n",
  15 |     "- Accuracy, Precision, Recall/Sensitivity/True Positive Rate, F1 score, False Positive Rate\n",
  16 |     "- ROC: Receiver Operating Characterisitcs and AUC: Area Under the Curve\n",
  17 |     "- Classification report\n",
  18 |     "\n",
  19 |     "#### References:\n",
  20 |     "[1] Machine Learning Fundamentals: The Confusion Matrix - https://www.youtube.com/watch?v=Kdsp6soqA7o&ab_channel=StatQuestwithJoshStarmer\n",
  21 |     "<br>[2] Machine Learning Fundamentals: Sensitivity and Specificity: https://www.youtube.com/watch?v=sunUKFXMHGk&ab_channel=StatQuestwithJoshStarmer\n",
  22 |     "<br>[3] ROC and AUC, Clearly Explained! https://www.youtube.com/watch?v=4jRBRDbJemM&ab_channel=StatQuestwithJoshStarmer\n",
  23 |     "<br>[4] Machine Learning Fundamentals: Bias and Variance: https://www.youtube.com/watch?v=EuBBz3bI-aA&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF&index=5&ab_channel=StatQuestwithJoshStarmer\n",
  24 |     "<br>[5] StatQuest: Logistic Regression - https://www.youtube.com/watch?v=yIYKR4sgzI8&ab_channel=StatQuestwithJoshStarmer\n",
  25 |     "<br>[6] Logistic Regression Details Pt1: Coefficients: https://www.youtube.com/watch?v=vN5cNN2-HWE&ab_channel=StatQuestwithJoshStarmer\n",
  26 |     "<br>[7] Logistic Regression Details Pt 2: Maximum Likelihood: https://www.youtube.com/watch?v=BfKanl1aSG0&t=163s&ab_channel=StatQuestwithJoshStarmer\n",
  27 |     "<br>[8] Machine Learning Tutorial Python - 8: Logistic Regression (Binary Classification): https://www.youtube.com/watch?v=zM4VZR0px8E&ab_channel=codebasics\n",
  28 |     "<br>[9] Maximum Likelihood, clearly explained!!!: https://www.youtube.com/watch?v=XepXtl9YKwc&ab_channel=StatQuestwithJoshStarmer"
  29 |    ]
  30 |   },
  31 |   {
  32 |    "cell_type": "markdown",
  33 |    "metadata": {},
  34 |    "source": [
  35 |     "#### Import required libraries"
  36 |    ]
  37 |   },
  38 |   {
  39 |    "cell_type": "code",
  40 |    "execution_count": 43,
  41 |    "metadata": {},
  42 |    "outputs": [],
  43 |    "source": [
  44 |     "import pandas as pd\n",
  45 |     "import numpy as np\n",
  46 |     "\n",
  47 |     "pd.options.display.max_columns = 20\n",
  48 |     "# pd.options.display.max_rows = 100\n",
  49 |     "\n",
  50 |     "from sklearn.model_selection import train_test_split\n",
  51 |     "from sklearn.linear_model import LogisticRegression\n",
  52 |     "# from sklearn.tree import DecisionTreeClassifier\n",
  53 |     "# from sklearn.ensemble import RandomForestClassifier\n",
  54 |     "\n",
  55 |     "from sklearn.metrics import accuracy_score, classification_report, confusion_matrix\n",
  56 |     "from sklearn.metrics import roc_curve, roc_auc_score, precision_score, recall_score, f1_score\n",
  57 |     "from sklearn.metrics import plot_confusion_matrix\n",
  58 |     "\n",
  59 |     "import matplotlib.pyplot as plt\n",
  60 |     "import seaborn as sns\n",
  61 |     "\n",
  62 |     "import warnings\n",
  63 |     "warnings.filterwarnings('ignore')"
  64 |    ]
  65 |   },
  66 |   {
  67 |    "cell_type": "markdown",
  68 |    "metadata": {},
  69 |    "source": [
  70 |     "#### Load data\n",
  71 |     "Dataset Source: https://learn.datacamp.com/courses/marketing-analytics-predicting-customer-churn-in-python"
  72 |    ]
  73 |   },
  74 |   {
  75 |    "cell_type": "code",
  76 |    "execution_count": 44,
  77 |    "metadata": {},
  78 |    "outputs": [
  79 |     {
  80 |      "data": {
  81 |       "text/html": [
  82 |        "<div>\n",
  83 |        "<style scoped>\n",
  84 |        "    .dataframe tbody tr th:only-of-type {\n",
  85 |        "        vertical-align: middle;\n",
  86 |        "    }\n",
  87 |        "\n",
  88 |        "    .dataframe tbody tr th {\n",
  89 |        "        vertical-align: top;\n",
  90 |        "    }\n",
  91 |        "\n",
  92 |        "    .dataframe thead th {\n",
  93 |        "        text-align: right;\n",
  94 |        "    }\n",
  95 |        "</style>\n",
  96 |        "<table border=\"1\" class=\"dataframe\">\n",
  97 |        "  <thead>\n",
  98 |        "    <tr style=\"text-align: right;\">\n",
  99 |        "      <th></th>\n",
 100 |        "      <th>Account_Length</th>\n",
 101 |        "      <th>Vmail_Message</th>\n",
 102 |        "      <th>Day_Mins</th>\n",
 103 |        "      <th>Eve_Mins</th>\n",
 104 |        "      <th>Night_Mins</th>\n",
 105 |        "      <th>Intl_Mins</th>\n",
 106 |        "      <th>CustServ_Calls</th>\n",
 107 |        "      <th>Churn</th>\n",
 108 |        "      <th>Intl_Plan</th>\n",
 109 |        "      <th>Vmail_Plan</th>\n",
 110 |        "      <th>...</th>\n",
 111 |        "      <th>Day_Charge</th>\n",
 112 |        "      <th>Eve_Calls</th>\n",
 113 |        "      <th>Eve_Charge</th>\n",
 114 |        "      <th>Night_Calls</th>\n",
 115 |        "      <th>Night_Charge</th>\n",
 116 |        "      <th>Intl_Calls</th>\n",
 117 |        "      <th>Intl_Charge</th>\n",
 118 |        "      <th>State</th>\n",
 119 |        "      <th>Area_Code</th>\n",
 120 |        "      <th>Phone</th>\n",
 121 |        "    </tr>\n",
 122 |        "  </thead>\n",
 123 |        "  <tbody>\n",
 124 |        "    <tr>\n",
 125 |        "      <th>0</th>\n",
 126 |        "      <td>128</td>\n",
 127 |        "      <td>25</td>\n",
 128 |        "      <td>265.1</td>\n",
 129 |        "      <td>197.4</td>\n",
 130 |        "      <td>244.7</td>\n",
 131 |        "      <td>10.0</td>\n",
 132 |        "      <td>1</td>\n",
 133 |        "      <td>no</td>\n",
 134 |        "      <td>no</td>\n",
 135 |        "      <td>yes</td>\n",
 136 |        "      <td>...</td>\n",
 137 |        "      <td>45.07</td>\n",
 138 |        "      <td>99</td>\n",
 139 |        "      <td>16.78</td>\n",
 140 |        "      <td>91</td>\n",
 141 |        "      <td>11.01</td>\n",
 142 |        "      <td>3</td>\n",
 143 |        "      <td>2.70</td>\n",
 144 |        "      <td>KS</td>\n",
 145 |        "      <td>415</td>\n",
 146 |        "      <td>382-4657</td>\n",
 147 |        "    </tr>\n",
 148 |        "    <tr>\n",
 149 |        "      <th>1</th>\n",
 150 |        "      <td>107</td>\n",
 151 |        "      <td>26</td>\n",
 152 |        "      <td>161.6</td>\n",
 153 |        "      <td>195.5</td>\n",
 154 |        "      <td>254.4</td>\n",
 155 |        "      <td>13.7</td>\n",
 156 |        "      <td>1</td>\n",
 157 |        "      <td>no</td>\n",
 158 |        "      <td>no</td>\n",
 159 |        "      <td>yes</td>\n",
 160 |        "      <td>...</td>\n",
 161 |        "      <td>27.47</td>\n",
 162 |        "      <td>103</td>\n",
 163 |        "      <td>16.62</td>\n",
 164 |        "      <td>103</td>\n",
 165 |        "      <td>11.45</td>\n",
 166 |        "      <td>3</td>\n",
 167 |        "      <td>3.70</td>\n",
 168 |        "      <td>OH</td>\n",
 169 |        "      <td>415</td>\n",
 170 |        "      <td>371-7191</td>\n",
 171 |        "    </tr>\n",
 172 |        "    <tr>\n",
 173 |        "      <th>2</th>\n",
 174 |        "      <td>137</td>\n",
 175 |        "      <td>0</td>\n",
 176 |        "      <td>243.4</td>\n",
 177 |        "      <td>121.2</td>\n",
 178 |        "      <td>162.6</td>\n",
 179 |        "      <td>12.2</td>\n",
 180 |        "      <td>0</td>\n",
 181 |        "      <td>no</td>\n",
 182 |        "      <td>no</td>\n",
 183 |        "      <td>no</td>\n",
 184 |        "      <td>...</td>\n",
 185 |        "      <td>41.38</td>\n",
 186 |        "      <td>110</td>\n",
 187 |        "      <td>10.30</td>\n",
 188 |        "      <td>104</td>\n",
 189 |        "      <td>7.32</td>\n",
 190 |        "      <td>5</td>\n",
 191 |        "      <td>3.29</td>\n",
 192 |        "      <td>NJ</td>\n",
 193 |        "      <td>415</td>\n",
 194 |        "      <td>358-1921</td>\n",
 195 |        "    </tr>\n",
 196 |        "    <tr>\n",
 197 |        "      <th>3</th>\n",
 198 |        "      <td>84</td>\n",
 199 |        "      <td>0</td>\n",
 200 |        "      <td>299.4</td>\n",
 201 |        "      <td>61.9</td>\n",
 202 |        "      <td>196.9</td>\n",
 203 |        "      <td>6.6</td>\n",
 204 |        "      <td>2</td>\n",
 205 |        "      <td>no</td>\n",
 206 |        "      <td>yes</td>\n",
 207 |        "      <td>no</td>\n",
 208 |        "      <td>...</td>\n",
 209 |        "      <td>50.90</td>\n",
 210 |        "      <td>88</td>\n",
 211 |        "      <td>5.26</td>\n",
 212 |        "      <td>89</td>\n",
 213 |        "      <td>8.86</td>\n",
 214 |        "      <td>7</td>\n",
 215 |        "      <td>1.78</td>\n",
 216 |        "      <td>OH</td>\n",
 217 |        "      <td>408</td>\n",
 218 |        "      <td>375-9999</td>\n",
 219 |        "    </tr>\n",
 220 |        "    <tr>\n",
 221 |        "      <th>4</th>\n",
 222 |        "      <td>75</td>\n",
 223 |        "      <td>0</td>\n",
 224 |        "      <td>166.7</td>\n",
 225 |        "      <td>148.3</td>\n",
 226 |        "      <td>186.9</td>\n",
 227 |        "      <td>10.1</td>\n",
 228 |        "      <td>3</td>\n",
 229 |        "      <td>no</td>\n",
 230 |        "      <td>yes</td>\n",
 231 |        "      <td>no</td>\n",
 232 |        "      <td>...</td>\n",
 233 |        "      <td>28.34</td>\n",
 234 |        "      <td>122</td>\n",
 235 |        "      <td>12.61</td>\n",
 236 |        "      <td>121</td>\n",
 237 |        "      <td>8.41</td>\n",
 238 |        "      <td>3</td>\n",
 239 |        "      <td>2.73</td>\n",
 240 |        "      <td>OK</td>\n",
 241 |        "      <td>415</td>\n",
 242 |        "      <td>330-6626</td>\n",
 243 |        "    </tr>\n",
 244 |        "    <tr>\n",
 245 |        "      <th>5</th>\n",
 246 |        "      <td>118</td>\n",
 247 |        "      <td>0</td>\n",
 248 |        "      <td>223.4</td>\n",
 249 |        "      <td>220.6</td>\n",
 250 |        "      <td>203.9</td>\n",
 251 |        "      <td>6.3</td>\n",
 252 |        "      <td>0</td>\n",
 253 |        "      <td>no</td>\n",
 254 |        "      <td>yes</td>\n",
 255 |        "      <td>no</td>\n",
 256 |        "      <td>...</td>\n",
 257 |        "      <td>37.98</td>\n",
 258 |        "      <td>101</td>\n",
 259 |        "      <td>18.75</td>\n",
 260 |        "      <td>118</td>\n",
 261 |        "      <td>9.18</td>\n",
 262 |        "      <td>6</td>\n",
 263 |        "      <td>1.70</td>\n",
 264 |        "      <td>AL</td>\n",
 265 |        "      <td>510</td>\n",
 266 |        "      <td>391-8027</td>\n",
 267 |        "    </tr>\n",
 268 |        "    <tr>\n",
 269 |        "      <th>6</th>\n",
 270 |        "      <td>121</td>\n",
 271 |        "      <td>24</td>\n",
 272 |        "      <td>218.2</td>\n",
 273 |        "      <td>348.5</td>\n",
 274 |        "      <td>212.6</td>\n",
 275 |        "      <td>7.5</td>\n",
 276 |        "      <td>3</td>\n",
 277 |        "      <td>no</td>\n",
 278 |        "      <td>no</td>\n",
 279 |        "      <td>yes</td>\n",
 280 |        "      <td>...</td>\n",
 281 |        "      <td>37.09</td>\n",
 282 |        "      <td>108</td>\n",
 283 |        "      <td>29.62</td>\n",
 284 |        "      <td>118</td>\n",
 285 |        "      <td>9.57</td>\n",
 286 |        "      <td>7</td>\n",
 287 |        "      <td>2.03</td>\n",
 288 |        "      <td>MA</td>\n",
 289 |        "      <td>510</td>\n",
 290 |        "      <td>355-9993</td>\n",
 291 |        "    </tr>\n",
 292 |        "    <tr>\n",
 293 |        "      <th>7</th>\n",
 294 |        "      <td>147</td>\n",
 295 |        "      <td>0</td>\n",
 296 |        "      <td>157.0</td>\n",
 297 |        "      <td>103.1</td>\n",
 298 |        "      <td>211.8</td>\n",
 299 |        "      <td>7.1</td>\n",
 300 |        "      <td>0</td>\n",
 301 |        "      <td>no</td>\n",
 302 |        "      <td>yes</td>\n",
 303 |        "      <td>no</td>\n",
 304 |        "      <td>...</td>\n",
 305 |        "      <td>26.69</td>\n",
 306 |        "      <td>94</td>\n",
 307 |        "      <td>8.76</td>\n",
 308 |        "      <td>96</td>\n",
 309 |        "      <td>9.53</td>\n",
 310 |        "      <td>6</td>\n",
 311 |        "      <td>1.92</td>\n",
 312 |        "      <td>MO</td>\n",
 313 |        "      <td>415</td>\n",
 314 |        "      <td>329-9001</td>\n",
 315 |        "    </tr>\n",
 316 |        "    <tr>\n",
 317 |        "      <th>8</th>\n",
 318 |        "      <td>117</td>\n",
 319 |        "      <td>0</td>\n",
 320 |        "      <td>184.5</td>\n",
 321 |        "      <td>351.6</td>\n",
 322 |        "      <td>215.8</td>\n",
 323 |        "      <td>8.7</td>\n",
 324 |        "      <td>1</td>\n",
 325 |        "      <td>no</td>\n",
 326 |        "      <td>no</td>\n",
 327 |        "      <td>no</td>\n",
 328 |        "      <td>...</td>\n",
 329 |        "      <td>31.37</td>\n",
 330 |        "      <td>80</td>\n",
 331 |        "      <td>29.89</td>\n",
 332 |        "      <td>90</td>\n",
 333 |        "      <td>9.71</td>\n",
 334 |        "      <td>4</td>\n",
 335 |        "      <td>2.35</td>\n",
 336 |        "      <td>LA</td>\n",
 337 |        "      <td>408</td>\n",
 338 |        "      <td>335-4719</td>\n",
 339 |        "    </tr>\n",
 340 |        "    <tr>\n",
 341 |        "      <th>9</th>\n",
 342 |        "      <td>141</td>\n",
 343 |        "      <td>37</td>\n",
 344 |        "      <td>258.6</td>\n",
 345 |        "      <td>222.0</td>\n",
 346 |        "      <td>326.4</td>\n",
 347 |        "      <td>11.2</td>\n",
 348 |        "      <td>0</td>\n",
 349 |        "      <td>no</td>\n",
 350 |        "      <td>yes</td>\n",
 351 |        "      <td>yes</td>\n",
 352 |        "      <td>...</td>\n",
 353 |        "      <td>43.96</td>\n",
 354 |        "      <td>111</td>\n",
 355 |        "      <td>18.87</td>\n",
 356 |        "      <td>97</td>\n",
 357 |        "      <td>14.69</td>\n",
 358 |        "      <td>5</td>\n",
 359 |        "      <td>3.02</td>\n",
 360 |        "      <td>WV</td>\n",
 361 |        "      <td>415</td>\n",
 362 |        "      <td>330-8173</td>\n",
 363 |        "    </tr>\n",
 364 |        "  </tbody>\n",
 365 |        "</table>\n",
 366 |        "<p>10 rows × 21 columns</p>\n",
 367 |        "</div>"
 368 |       ],
 369 |       "text/plain": [
 370 |        "   Account_Length  Vmail_Message  Day_Mins  Eve_Mins  Night_Mins  Intl_Mins  \\\n",
 371 |        "0             128             25     265.1     197.4       244.7       10.0   \n",
 372 |        "1             107             26     161.6     195.5       254.4       13.7   \n",
 373 |        "2             137              0     243.4     121.2       162.6       12.2   \n",
 374 |        "3              84              0     299.4      61.9       196.9        6.6   \n",
 375 |        "4              75              0     166.7     148.3       186.9       10.1   \n",
 376 |        "5             118              0     223.4     220.6       203.9        6.3   \n",
 377 |        "6             121             24     218.2     348.5       212.6        7.5   \n",
 378 |        "7             147              0     157.0     103.1       211.8        7.1   \n",
 379 |        "8             117              0     184.5     351.6       215.8        8.7   \n",
 380 |        "9             141             37     258.6     222.0       326.4       11.2   \n",
 381 |        "\n",
 382 |        "   CustServ_Calls Churn Intl_Plan Vmail_Plan  ...  Day_Charge  Eve_Calls  \\\n",
 383 |        "0               1    no        no        yes  ...       45.07         99   \n",
 384 |        "1               1    no        no        yes  ...       27.47        103   \n",
 385 |        "2               0    no        no         no  ...       41.38        110   \n",
 386 |        "3               2    no       yes         no  ...       50.90         88   \n",
 387 |        "4               3    no       yes         no  ...       28.34        122   \n",
 388 |        "5               0    no       yes         no  ...       37.98        101   \n",
 389 |        "6               3    no        no        yes  ...       37.09        108   \n",
 390 |        "7               0    no       yes         no  ...       26.69         94   \n",
 391 |        "8               1    no        no         no  ...       31.37         80   \n",
 392 |        "9               0    no       yes        yes  ...       43.96        111   \n",
 393 |        "\n",
 394 |        "   Eve_Charge  Night_Calls  Night_Charge  Intl_Calls  Intl_Charge  State  \\\n",
 395 |        "0       16.78           91         11.01           3         2.70     KS   \n",
 396 |        "1       16.62          103         11.45           3         3.70     OH   \n",
 397 |        "2       10.30          104          7.32           5         3.29     NJ   \n",
 398 |        "3        5.26           89          8.86           7         1.78     OH   \n",
 399 |        "4       12.61          121          8.41           3         2.73     OK   \n",
 400 |        "5       18.75          118          9.18           6         1.70     AL   \n",
 401 |        "6       29.62          118          9.57           7         2.03     MA   \n",
 402 |        "7        8.76           96          9.53           6         1.92     MO   \n",
 403 |        "8       29.89           90          9.71           4         2.35     LA   \n",
 404 |        "9       18.87           97         14.69           5         3.02     WV   \n",
 405 |        "\n",
 406 |        "  Area_Code     Phone  \n",
 407 |        "0       415  382-4657  \n",
 408 |        "1       415  371-7191  \n",
 409 |        "2       415  358-1921  \n",
 410 |        "3       408  375-9999  \n",
 411 |        "4       415  330-6626  \n",
 412 |        "5       510  391-8027  \n",
 413 |        "6       510  355-9993  \n",
 414 |        "7       415  329-9001  \n",
 415 |        "8       408  335-4719  \n",
 416 |        "9       415  330-8173  \n",
 417 |        "\n",
 418 |        "[10 rows x 21 columns]"
 419 |       ]
 420 |      },
 421 |      "metadata": {},
 422 |      "output_type": "display_data"
 423 |     }
 424 |    ],
 425 |    "source": [
 426 |     "df = pd.read_csv(\"Churn.csv\")\n",
 427 |     "\n",
 428 |     "display(df.head(10))"
 429 |    ]
 430 |   },
 431 |   {
 432 |    "cell_type": "code",
 433 |    "execution_count": 45,
 434 |    "metadata": {},
 435 |    "outputs": [
 436 |     {
 437 |      "name": "stdout",
 438 |      "output_type": "stream",
 439 |      "text": [
 440 |       "(3333, 21)\n"
 441 |      ]
 442 |     }
 443 |    ],
 444 |    "source": [
 445 |     "print(df.shape)"
 446 |    ]
 447 |   },
 448 |   {
 449 |    "cell_type": "code",
 450 |    "execution_count": 46,
 451 |    "metadata": {},
 452 |    "outputs": [
 453 |     {
 454 |      "name": "stdout",
 455 |      "output_type": "stream",
 456 |      "text": [
 457 |       "<class 'pandas.core.frame.DataFrame'>\n",
 458 |       "RangeIndex: 3333 entries, 0 to 3332\n",
 459 |       "Data columns (total 21 columns):\n",
 460 |       " #   Column          Non-Null Count  Dtype  \n",
 461 |       "---  ------          --------------  -----  \n",
 462 |       " 0   Account_Length  3333 non-null   int64  \n",
 463 |       " 1   Vmail_Message   3333 non-null   int64  \n",
 464 |       " 2   Day_Mins        3333 non-null   float64\n",
 465 |       " 3   Eve_Mins        3333 non-null   float64\n",
 466 |       " 4   Night_Mins      3333 non-null   float64\n",
 467 |       " 5   Intl_Mins       3333 non-null   float64\n",
 468 |       " 6   CustServ_Calls  3333 non-null   int64  \n",
 469 |       " 7   Churn           3333 non-null   object \n",
 470 |       " 8   Intl_Plan       3333 non-null   object \n",
 471 |       " 9   Vmail_Plan      3333 non-null   object \n",
 472 |       " 10  Day_Calls       3333 non-null   int64  \n",
 473 |       " 11  Day_Charge      3333 non-null   float64\n",
 474 |       " 12  Eve_Calls       3333 non-null   int64  \n",
 475 |       " 13  Eve_Charge      3333 non-null   float64\n",
 476 |       " 14  Night_Calls     3333 non-null   int64  \n",
 477 |       " 15  Night_Charge    3333 non-null   float64\n",
 478 |       " 16  Intl_Calls      3333 non-null   int64  \n",
 479 |       " 17  Intl_Charge     3333 non-null   float64\n",
 480 |       " 18  State           3333 non-null   object \n",
 481 |       " 19  Area_Code       3333 non-null   int64  \n",
 482 |       " 20  Phone           3333 non-null   object \n",
 483 |       "dtypes: float64(8), int64(8), object(5)\n",
 484 |       "memory usage: 546.9+ KB\n"
 485 |      ]
 486 |     }
 487 |    ],
 488 |    "source": [
 489 |     "df.info()"
 490 |    ]
 491 |   },
 492 |   {
 493 |    "cell_type": "markdown",
 494 |    "metadata": {},
 495 |    "source": [
 496 |     "#### Data Preprocessing"
 497 |    ]
 498 |   },
 499 |   {
 500 |    "cell_type": "code",
 501 |    "execution_count": 47,
 502 |    "metadata": {},
 503 |    "outputs": [
 504 |     {
 505 |      "name": "stdout",
 506 |      "output_type": "stream",
 507 |      "text": [
 508 |       "['no' 'yes']\n",
 509 |       "['no' 'yes']\n",
 510 |       "['yes' 'no']\n"
 511 |      ]
 512 |     }
 513 |    ],
 514 |    "source": [
 515 |     "print(df['Churn'].unique())\n",
 516 |     "print(df['Intl_Plan'].unique())\n",
 517 |     "print(df['Vmail_Plan'].unique())"
 518 |    ]
 519 |   },
 520 |   {
 521 |    "cell_type": "code",
 522 |    "execution_count": 48,
 523 |    "metadata": {},
 524 |    "outputs": [
 525 |     {
 526 |      "data": {
 527 |       "text/html": [
 528 |        "<div>\n",
 529 |        "<style scoped>\n",
 530 |        "    .dataframe tbody tr th:only-of-type {\n",
 531 |        "        vertical-align: middle;\n",
 532 |        "    }\n",
 533 |        "\n",
 534 |        "    .dataframe tbody tr th {\n",
 535 |        "        vertical-align: top;\n",
 536 |        "    }\n",
 537 |        "\n",
 538 |        "    .dataframe thead th {\n",
 539 |        "        text-align: right;\n",
 540 |        "    }\n",
 541 |        "</style>\n",
 542 |        "<table border=\"1\" class=\"dataframe\">\n",
 543 |        "  <thead>\n",
 544 |        "    <tr style=\"text-align: right;\">\n",
 545 |        "      <th></th>\n",
 546 |        "      <th>Account_Length</th>\n",
 547 |        "      <th>Vmail_Message</th>\n",
 548 |        "      <th>Day_Mins</th>\n",
 549 |        "      <th>Eve_Mins</th>\n",
 550 |        "      <th>Night_Mins</th>\n",
 551 |        "      <th>Intl_Mins</th>\n",
 552 |        "      <th>CustServ_Calls</th>\n",
 553 |        "      <th>Churn</th>\n",
 554 |        "      <th>Intl_Plan</th>\n",
 555 |        "      <th>Vmail_Plan</th>\n",
 556 |        "      <th>Day_Calls</th>\n",
 557 |        "      <th>Day_Charge</th>\n",
 558 |        "      <th>Eve_Calls</th>\n",
 559 |        "      <th>Eve_Charge</th>\n",
 560 |        "      <th>Night_Calls</th>\n",
 561 |        "      <th>Night_Charge</th>\n",
 562 |        "      <th>Intl_Calls</th>\n",
 563 |        "      <th>Intl_Charge</th>\n",
 564 |        "    </tr>\n",
 565 |        "  </thead>\n",
 566 |        "  <tbody>\n",
 567 |        "    <tr>\n",
 568 |        "      <th>0</th>\n",
 569 |        "      <td>128</td>\n",
 570 |        "      <td>25</td>\n",
 571 |        "      <td>265.1</td>\n",
 572 |        "      <td>197.4</td>\n",
 573 |        "      <td>244.7</td>\n",
 574 |        "      <td>10.0</td>\n",
 575 |        "      <td>1</td>\n",
 576 |        "      <td>0</td>\n",
 577 |        "      <td>0</td>\n",
 578 |        "      <td>1</td>\n",
 579 |        "      <td>110</td>\n",
 580 |        "      <td>45.07</td>\n",
 581 |        "      <td>99</td>\n",
 582 |        "      <td>16.78</td>\n",
 583 |        "      <td>91</td>\n",
 584 |        "      <td>11.01</td>\n",
 585 |        "      <td>3</td>\n",
 586 |        "      <td>2.70</td>\n",
 587 |        "    </tr>\n",
 588 |        "    <tr>\n",
 589 |        "      <th>1</th>\n",
 590 |        "      <td>107</td>\n",
 591 |        "      <td>26</td>\n",
 592 |        "      <td>161.6</td>\n",
 593 |        "      <td>195.5</td>\n",
 594 |        "      <td>254.4</td>\n",
 595 |        "      <td>13.7</td>\n",
 596 |        "      <td>1</td>\n",
 597 |        "      <td>0</td>\n",
 598 |        "      <td>0</td>\n",
 599 |        "      <td>1</td>\n",
 600 |        "      <td>123</td>\n",
 601 |        "      <td>27.47</td>\n",
 602 |        "      <td>103</td>\n",
 603 |        "      <td>16.62</td>\n",
 604 |        "      <td>103</td>\n",
 605 |        "      <td>11.45</td>\n",
 606 |        "      <td>3</td>\n",
 607 |        "      <td>3.70</td>\n",
 608 |        "    </tr>\n",
 609 |        "    <tr>\n",
 610 |        "      <th>2</th>\n",
 611 |        "      <td>137</td>\n",
 612 |        "      <td>0</td>\n",
 613 |        "      <td>243.4</td>\n",
 614 |        "      <td>121.2</td>\n",
 615 |        "      <td>162.6</td>\n",
 616 |        "      <td>12.2</td>\n",
 617 |        "      <td>0</td>\n",
 618 |        "      <td>0</td>\n",
 619 |        "      <td>0</td>\n",
 620 |        "      <td>0</td>\n",
 621 |        "      <td>114</td>\n",
 622 |        "      <td>41.38</td>\n",
 623 |        "      <td>110</td>\n",
 624 |        "      <td>10.30</td>\n",
 625 |        "      <td>104</td>\n",
 626 |        "      <td>7.32</td>\n",
 627 |        "      <td>5</td>\n",
 628 |        "      <td>3.29</td>\n",
 629 |        "    </tr>\n",
 630 |        "    <tr>\n",
 631 |        "      <th>3</th>\n",
 632 |        "      <td>84</td>\n",
 633 |        "      <td>0</td>\n",
 634 |        "      <td>299.4</td>\n",
 635 |        "      <td>61.9</td>\n",
 636 |        "      <td>196.9</td>\n",
 637 |        "      <td>6.6</td>\n",
 638 |        "      <td>2</td>\n",
 639 |        "      <td>0</td>\n",
 640 |        "      <td>1</td>\n",
 641 |        "      <td>0</td>\n",
 642 |        "      <td>71</td>\n",
 643 |        "      <td>50.90</td>\n",
 644 |        "      <td>88</td>\n",
 645 |        "      <td>5.26</td>\n",
 646 |        "      <td>89</td>\n",
 647 |        "      <td>8.86</td>\n",
 648 |        "      <td>7</td>\n",
 649 |        "      <td>1.78</td>\n",
 650 |        "    </tr>\n",
 651 |        "    <tr>\n",
 652 |        "      <th>4</th>\n",
 653 |        "      <td>75</td>\n",
 654 |        "      <td>0</td>\n",
 655 |        "      <td>166.7</td>\n",
 656 |        "      <td>148.3</td>\n",
 657 |        "      <td>186.9</td>\n",
 658 |        "      <td>10.1</td>\n",
 659 |        "      <td>3</td>\n",
 660 |        "      <td>0</td>\n",
 661 |        "      <td>1</td>\n",
 662 |        "      <td>0</td>\n",
 663 |        "      <td>113</td>\n",
 664 |        "      <td>28.34</td>\n",
 665 |        "      <td>122</td>\n",
 666 |        "      <td>12.61</td>\n",
 667 |        "      <td>121</td>\n",
 668 |        "      <td>8.41</td>\n",
 669 |        "      <td>3</td>\n",
 670 |        "      <td>2.73</td>\n",
 671 |        "    </tr>\n",
 672 |        "    <tr>\n",
 673 |        "      <th>5</th>\n",
 674 |        "      <td>118</td>\n",
 675 |        "      <td>0</td>\n",
 676 |        "      <td>223.4</td>\n",
 677 |        "      <td>220.6</td>\n",
 678 |        "      <td>203.9</td>\n",
 679 |        "      <td>6.3</td>\n",
 680 |        "      <td>0</td>\n",
 681 |        "      <td>0</td>\n",
 682 |        "      <td>1</td>\n",
 683 |        "      <td>0</td>\n",
 684 |        "      <td>98</td>\n",
 685 |        "      <td>37.98</td>\n",
 686 |        "      <td>101</td>\n",
 687 |        "      <td>18.75</td>\n",
 688 |        "      <td>118</td>\n",
 689 |        "      <td>9.18</td>\n",
 690 |        "      <td>6</td>\n",
 691 |        "      <td>1.70</td>\n",
 692 |        "    </tr>\n",
 693 |        "    <tr>\n",
 694 |        "      <th>6</th>\n",
 695 |        "      <td>121</td>\n",
 696 |        "      <td>24</td>\n",
 697 |        "      <td>218.2</td>\n",
 698 |        "      <td>348.5</td>\n",
 699 |        "      <td>212.6</td>\n",
 700 |        "      <td>7.5</td>\n",
 701 |        "      <td>3</td>\n",
 702 |        "      <td>0</td>\n",
 703 |        "      <td>0</td>\n",
 704 |        "      <td>1</td>\n",
 705 |        "      <td>88</td>\n",
 706 |        "      <td>37.09</td>\n",
 707 |        "      <td>108</td>\n",
 708 |        "      <td>29.62</td>\n",
 709 |        "      <td>118</td>\n",
 710 |        "      <td>9.57</td>\n",
 711 |        "      <td>7</td>\n",
 712 |        "      <td>2.03</td>\n",
 713 |        "    </tr>\n",
 714 |        "    <tr>\n",
 715 |        "      <th>7</th>\n",
 716 |        "      <td>147</td>\n",
 717 |        "      <td>0</td>\n",
 718 |        "      <td>157.0</td>\n",
 719 |        "      <td>103.1</td>\n",
 720 |        "      <td>211.8</td>\n",
 721 |        "      <td>7.1</td>\n",
 722 |        "      <td>0</td>\n",
 723 |        "      <td>0</td>\n",
 724 |        "      <td>1</td>\n",
 725 |        "      <td>0</td>\n",
 726 |        "      <td>79</td>\n",
 727 |        "      <td>26.69</td>\n",
 728 |        "      <td>94</td>\n",
 729 |        "      <td>8.76</td>\n",
 730 |        "      <td>96</td>\n",
 731 |        "      <td>9.53</td>\n",
 732 |        "      <td>6</td>\n",
 733 |        "      <td>1.92</td>\n",
 734 |        "    </tr>\n",
 735 |        "    <tr>\n",
 736 |        "      <th>8</th>\n",
 737 |        "      <td>117</td>\n",
 738 |        "      <td>0</td>\n",
 739 |        "      <td>184.5</td>\n",
 740 |        "      <td>351.6</td>\n",
 741 |        "      <td>215.8</td>\n",
 742 |        "      <td>8.7</td>\n",
 743 |        "      <td>1</td>\n",
 744 |        "      <td>0</td>\n",
 745 |        "      <td>0</td>\n",
 746 |        "      <td>0</td>\n",
 747 |        "      <td>97</td>\n",
 748 |        "      <td>31.37</td>\n",
 749 |        "      <td>80</td>\n",
 750 |        "      <td>29.89</td>\n",
 751 |        "      <td>90</td>\n",
 752 |        "      <td>9.71</td>\n",
 753 |        "      <td>4</td>\n",
 754 |        "      <td>2.35</td>\n",
 755 |        "    </tr>\n",
 756 |        "    <tr>\n",
 757 |        "      <th>9</th>\n",
 758 |        "      <td>141</td>\n",
 759 |        "      <td>37</td>\n",
 760 |        "      <td>258.6</td>\n",
 761 |        "      <td>222.0</td>\n",
 762 |        "      <td>326.4</td>\n",
 763 |        "      <td>11.2</td>\n",
 764 |        "      <td>0</td>\n",
 765 |        "      <td>0</td>\n",
 766 |        "      <td>1</td>\n",
 767 |        "      <td>1</td>\n",
 768 |        "      <td>84</td>\n",
 769 |        "      <td>43.96</td>\n",
 770 |        "      <td>111</td>\n",
 771 |        "      <td>18.87</td>\n",
 772 |        "      <td>97</td>\n",
 773 |        "      <td>14.69</td>\n",
 774 |        "      <td>5</td>\n",
 775 |        "      <td>3.02</td>\n",
 776 |        "    </tr>\n",
 777 |        "  </tbody>\n",
 778 |        "</table>\n",
 779 |        "</div>"
 780 |       ],
 781 |       "text/plain": [
 782 |        "   Account_Length  Vmail_Message  Day_Mins  Eve_Mins  Night_Mins  Intl_Mins  \\\n",
 783 |        "0             128             25     265.1     197.4       244.7       10.0   \n",
 784 |        "1             107             26     161.6     195.5       254.4       13.7   \n",
 785 |        "2             137              0     243.4     121.2       162.6       12.2   \n",
 786 |        "3              84              0     299.4      61.9       196.9        6.6   \n",
 787 |        "4              75              0     166.7     148.3       186.9       10.1   \n",
 788 |        "5             118              0     223.4     220.6       203.9        6.3   \n",
 789 |        "6             121             24     218.2     348.5       212.6        7.5   \n",
 790 |        "7             147              0     157.0     103.1       211.8        7.1   \n",
 791 |        "8             117              0     184.5     351.6       215.8        8.7   \n",
 792 |        "9             141             37     258.6     222.0       326.4       11.2   \n",
 793 |        "\n",
 794 |        "   CustServ_Calls  Churn  Intl_Plan  Vmail_Plan  Day_Calls  Day_Charge  \\\n",
 795 |        "0               1      0          0           1        110       45.07   \n",
 796 |        "1               1      0          0           1        123       27.47   \n",
 797 |        "2               0      0          0           0        114       41.38   \n",
 798 |        "3               2      0          1           0         71       50.90   \n",
 799 |        "4               3      0          1           0        113       28.34   \n",
 800 |        "5               0      0          1           0         98       37.98   \n",
 801 |        "6               3      0          0           1         88       37.09   \n",
 802 |        "7               0      0          1           0         79       26.69   \n",
 803 |        "8               1      0          0           0         97       31.37   \n",
 804 |        "9               0      0          1           1         84       43.96   \n",
 805 |        "\n",
 806 |        "   Eve_Calls  Eve_Charge  Night_Calls  Night_Charge  Intl_Calls  Intl_Charge  \n",
 807 |        "0         99       16.78           91         11.01           3         2.70  \n",
 808 |        "1        103       16.62          103         11.45           3         3.70  \n",
 809 |        "2        110       10.30          104          7.32           5         3.29  \n",
 810 |        "3         88        5.26           89          8.86           7         1.78  \n",
 811 |        "4        122       12.61          121          8.41           3         2.73  \n",
 812 |        "5        101       18.75          118          9.18           6         1.70  \n",
 813 |        "6        108       29.62          118          9.57           7         2.03  \n",
 814 |        "7         94        8.76           96          9.53           6         1.92  \n",
 815 |        "8         80       29.89           90          9.71           4         2.35  \n",
 816 |        "9        111       18.87           97         14.69           5         3.02  "
 817 |       ]
 818 |      },
 819 |      "metadata": {},
 820 |      "output_type": "display_data"
 821 |     },
 822 |     {
 823 |      "name": "stdout",
 824 |      "output_type": "stream",
 825 |      "text": [
 826 |       "(3333, 18)\n"
 827 |      ]
 828 |     }
 829 |    ],
 830 |    "source": [
 831 |     "df = df.drop(['State', 'Area_Code', 'Phone'], axis=1)\n",
 832 |     "\n",
 833 |     "df['Churn'] = df['Churn'].replace(({'no':0, 'yes':1}))\n",
 834 |     "df['Intl_Plan'] = df['Intl_Plan'].replace(({'no':0, 'yes':1}))\n",
 835 |     "df['Vmail_Plan'] = df['Vmail_Plan'].replace(({'no':0, 'yes':1}))\n",
 836 |     "\n",
 837 |     "display(df.head(10))\n",
 838 |     "print(df.shape)"
 839 |    ]
 840 |   },
 841 |   {
 842 |    "cell_type": "code",
 843 |    "execution_count": 49,
 844 |    "metadata": {},
 845 |    "outputs": [
 846 |     {
 847 |      "name": "stdout",
 848 |      "output_type": "stream",
 849 |      "text": [
 850 |       "<class 'pandas.core.frame.DataFrame'>\n",
 851 |       "RangeIndex: 3333 entries, 0 to 3332\n",
 852 |       "Data columns (total 18 columns):\n",
 853 |       " #   Column          Non-Null Count  Dtype  \n",
 854 |       "---  ------          --------------  -----  \n",
 855 |       " 0   Account_Length  3333 non-null   int64  \n",
 856 |       " 1   Vmail_Message   3333 non-null   int64  \n",
 857 |       " 2   Day_Mins        3333 non-null   float64\n",
 858 |       " 3   Eve_Mins        3333 non-null   float64\n",
 859 |       " 4   Night_Mins      3333 non-null   float64\n",
 860 |       " 5   Intl_Mins       3333 non-null   float64\n",
 861 |       " 6   CustServ_Calls  3333 non-null   int64  \n",
 862 |       " 7   Churn           3333 non-null   int64  \n",
 863 |       " 8   Intl_Plan       3333 non-null   int64  \n",
 864 |       " 9   Vmail_Plan      3333 non-null   int64  \n",
 865 |       " 10  Day_Calls       3333 non-null   int64  \n",
 866 |       " 11  Day_Charge      3333 non-null   float64\n",
 867 |       " 12  Eve_Calls       3333 non-null   int64  \n",
 868 |       " 13  Eve_Charge      3333 non-null   float64\n",
 869 |       " 14  Night_Calls     3333 non-null   int64  \n",
 870 |       " 15  Night_Charge    3333 non-null   float64\n",
 871 |       " 16  Intl_Calls      3333 non-null   int64  \n",
 872 |       " 17  Intl_Charge     3333 non-null   float64\n",
 873 |       "dtypes: float64(8), int64(10)\n",
 874 |       "memory usage: 468.8 KB\n"
 875 |      ]
 876 |     }
 877 |    ],
 878 |    "source": [
 879 |     "df.info()"
 880 |    ]
 881 |   },
 882 |   {
 883 |    "cell_type": "markdown",
 884 |    "metadata": {},
 885 |    "source": [
 886 |     "#### Target variable "
 887 |    ]
 888 |   },
 889 |   {
 890 |    "cell_type": "code",
 891 |    "execution_count": 50,
 892 |    "metadata": {},
 893 |    "outputs": [
 894 |     {
 895 |      "data": {
 896 |       "text/plain": [
 897 |        "0    2850\n",
 898 |        "1     483\n",
 899 |        "Name: Churn, dtype: int64"
 900 |       ]
 901 |      },
 902 |      "execution_count": 50,
 903 |      "metadata": {},
 904 |      "output_type": "execute_result"
 905 |     }
 906 |    ],
 907 |    "source": [
 908 |     "df['Churn'].value_counts()"
 909 |    ]
 910 |   },
 911 |   {
 912 |    "cell_type": "markdown",
 913 |    "metadata": {},
 914 |    "source": [
 915 |     "#### Feature and target variable separation"
 916 |    ]
 917 |   },
 918 |   {
 919 |    "cell_type": "code",
 920 |    "execution_count": 51,
 921 |    "metadata": {},
 922 |    "outputs": [
 923 |     {
 924 |      "name": "stdout",
 925 |      "output_type": "stream",
 926 |      "text": [
 927 |       "(3333, 17) (3333, 1)\n"
 928 |      ]
 929 |     }
 930 |    ],
 931 |    "source": [
 932 |     "X = df.drop('Churn', axis=1)\n",
 933 |     "y = df[['Churn']]\n",
 934 |     "\n",
 935 |     "print(X.shape, y.shape)"
 936 |    ]
 937 |   },
 938 |   {
 939 |    "cell_type": "markdown",
 940 |    "metadata": {},
 941 |    "source": [
 942 |     "#### Train test separation"
 943 |    ]
 944 |   },
 945 |   {
 946 |    "cell_type": "code",
 947 |    "execution_count": 52,
 948 |    "metadata": {},
 949 |    "outputs": [
 950 |     {
 951 |      "name": "stdout",
 952 |      "output_type": "stream",
 953 |      "text": [
 954 |       "(2333, 17) (1000, 17) (2333, 1) (1000, 1)\n"
 955 |      ]
 956 |     }
 957 |    ],
 958 |    "source": [
 959 |     "X_train, X_test, y_train, y_test = train_test_split(X, \n",
 960 |     "                                                    y, \n",
 961 |     "                                                    test_size=0.3, \n",
 962 |     "                                                    random_state=42, \n",
 963 |     "                                                    stratify=y)\n",
 964 |     "\n",
 965 |     "print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
 966 |    ]
 967 |   },
 968 |   {
 969 |    "cell_type": "code",
 970 |    "execution_count": 66,
 971 |    "metadata": {},
 972 |    "outputs": [
 973 |     {
 974 |      "data": {
 975 |       "text/plain": [
 976 |        "0    855\n",
 977 |        "1    145\n",
 978 |        "Name: Churn, dtype: int64"
 979 |       ]
 980 |      },
 981 |      "execution_count": 66,
 982 |      "metadata": {},
 983 |      "output_type": "execute_result"
 984 |     }
 985 |    ],
 986 |    "source": [
 987 |     "y_test[\"Churn\"].value_counts()"
 988 |    ]
 989 |   },
 990 |   {
 991 |    "cell_type": "markdown",
 992 |    "metadata": {},
 993 |    "source": [
 994 |     "#### Training: Logistic Regression"
 995 |    ]
 996 |   },
 997 |   {
 998 |    "cell_type": "code",
 999 |    "execution_count": 53,
1000 |    "metadata": {},
1001 |    "outputs": [],
1002 |    "source": [
1003 |     "model = LogisticRegression()\n",
1004 |     "model = model.fit(X_train, y_train)"
1005 |    ]
1006 |   },
1007 |   {
1008 |    "cell_type": "markdown",
1009 |    "metadata": {},
1010 |    "source": [
1011 |     "#### Prediction"
1012 |    ]
1013 |   },
1014 |   {
1015 |    "cell_type": "code",
1016 |    "execution_count": 54,
1017 |    "metadata": {},
1018 |    "outputs": [],
1019 |    "source": [
1020 |     "y_pred = model.predict(X_test)\n",
1021 |     "# print(y_pred)"
1022 |    ]
1023 |   },
1024 |   {
1025 |    "cell_type": "code",
1026 |    "execution_count": 55,
1027 |    "metadata": {},
1028 |    "outputs": [],
1029 |    "source": [
1030 |     "# model.predict_proba(X_test)"
1031 |    ]
1032 |   },
1033 |   {
1034 |    "cell_type": "markdown",
1035 |    "metadata": {},
1036 |    "source": [
1037 |     "#### Prediction Comparison with the test data"
1038 |    ]
1039 |   },
1040 |   {
1041 |    "cell_type": "code",
1042 |    "execution_count": 56,
1043 |    "metadata": {},
1044 |    "outputs": [
1045 |     {
1046 |      "data": {
1047 |       "text/html": [
1048 |        "<div>\n",
1049 |        "<style scoped>\n",
1050 |        "    .dataframe tbody tr th:only-of-type {\n",
1051 |        "        vertical-align: middle;\n",
1052 |        "    }\n",
1053 |        "\n",
1054 |        "    .dataframe tbody tr th {\n",
1055 |        "        vertical-align: top;\n",
1056 |        "    }\n",
1057 |        "\n",
1058 |        "    .dataframe thead th {\n",
1059 |        "        text-align: right;\n",
1060 |        "    }\n",
1061 |        "</style>\n",
1062 |        "<table border=\"1\" class=\"dataframe\">\n",
1063 |        "  <thead>\n",
1064 |        "    <tr style=\"text-align: right;\">\n",
1065 |        "      <th></th>\n",
1066 |        "      <th>Churn</th>\n",
1067 |        "      <th>Churn_probability</th>\n",
1068 |        "      <th>Churn_predicted</th>\n",
1069 |        "    </tr>\n",
1070 |        "  </thead>\n",
1071 |        "  <tbody>\n",
1072 |        "    <tr>\n",
1073 |        "      <th>68</th>\n",
1074 |        "      <td>0</td>\n",
1075 |        "      <td>0.35</td>\n",
1076 |        "      <td>0</td>\n",
1077 |        "    </tr>\n",
1078 |        "    <tr>\n",
1079 |        "      <th>1653</th>\n",
1080 |        "      <td>1</td>\n",
1081 |        "      <td>0.24</td>\n",
1082 |        "      <td>0</td>\n",
1083 |        "    </tr>\n",
1084 |        "    <tr>\n",
1085 |        "      <th>1716</th>\n",
1086 |        "      <td>0</td>\n",
1087 |        "      <td>0.12</td>\n",
1088 |        "      <td>0</td>\n",
1089 |        "    </tr>\n",
1090 |        "    <tr>\n",
1091 |        "      <th>3251</th>\n",
1092 |        "      <td>0</td>\n",
1093 |        "      <td>0.16</td>\n",
1094 |        "      <td>0</td>\n",
1095 |        "    </tr>\n",
1096 |        "    <tr>\n",
1097 |        "      <th>2406</th>\n",
1098 |        "      <td>0</td>\n",
1099 |        "      <td>0.10</td>\n",
1100 |        "      <td>0</td>\n",
1101 |        "    </tr>\n",
1102 |        "  </tbody>\n",
1103 |        "</table>\n",
1104 |        "</div>"
1105 |       ],
1106 |       "text/plain": [
1107 |        "      Churn  Churn_probability  Churn_predicted\n",
1108 |        "68        0               0.35                0\n",
1109 |        "1653      1               0.24                0\n",
1110 |        "1716      0               0.12                0\n",
1111 |        "3251      0               0.16                0\n",
1112 |        "2406      0               0.10                0"
1113 |       ]
1114 |      },
1115 |      "metadata": {},
1116 |      "output_type": "display_data"
1117 |     }
1118 |    ],
1119 |    "source": [
1120 |     "y_test[\"Churn_probability\"] = np.round(model.predict_proba(X_test)[:,1], 2)\n",
1121 |     "y_test[\"Churn_predicted\"] = y_pred\n",
1122 |     "\n",
1123 |     "display(y_test.head())"
1124 |    ]
1125 |   },
1126 |   {
1127 |    "cell_type": "code",
1128 |    "execution_count": 57,
1129 |    "metadata": {},
1130 |    "outputs": [
1131 |     {
1132 |      "name": "stdout",
1133 |      "output_type": "stream",
1134 |      "text": [
1135 |       "Shape of X_test =  (1000, 17)\n",
1136 |       "Shape of y_test =  (1000, 3)\n"
1137 |      ]
1138 |     }
1139 |    ],
1140 |    "source": [
1141 |     "print(\"Shape of X_test = \", X_test.shape)\n",
1142 |     "print(\"Shape of y_test = \", y_test.shape)"
1143 |    ]
1144 |   },
1145 |   {
1146 |    "cell_type": "code",
1147 |    "execution_count": 59,
1148 |    "metadata": {},
1149 |    "outputs": [
1150 |     {
1151 |      "name": "stdout",
1152 |      "output_type": "stream",
1153 |      "text": [
1154 |       "(1000, 20)\n"
1155 |      ]
1156 |     },
1157 |     {
1158 |      "data": {
1159 |       "text/html": [
1160 |        "<div>\n",
1161 |        "<style scoped>\n",
1162 |        "    .dataframe tbody tr th:only-of-type {\n",
1163 |        "        vertical-align: middle;\n",
1164 |        "    }\n",
1165 |        "\n",
1166 |        "    .dataframe tbody tr th {\n",
1167 |        "        vertical-align: top;\n",
1168 |        "    }\n",
1169 |        "\n",
1170 |        "    .dataframe thead th {\n",
1171 |        "        text-align: right;\n",
1172 |        "    }\n",
1173 |        "</style>\n",
1174 |        "<table border=\"1\" class=\"dataframe\">\n",
1175 |        "  <thead>\n",
1176 |        "    <tr style=\"text-align: right;\">\n",
1177 |        "      <th></th>\n",
1178 |        "      <th>Account_Length</th>\n",
1179 |        "      <th>Vmail_Message</th>\n",
1180 |        "      <th>Day_Mins</th>\n",
1181 |        "      <th>Eve_Mins</th>\n",
1182 |        "      <th>Night_Mins</th>\n",
1183 |        "      <th>Intl_Mins</th>\n",
1184 |        "      <th>CustServ_Calls</th>\n",
1185 |        "      <th>Intl_Plan</th>\n",
1186 |        "      <th>Vmail_Plan</th>\n",
1187 |        "      <th>Day_Calls</th>\n",
1188 |        "      <th>Day_Charge</th>\n",
1189 |        "      <th>Eve_Calls</th>\n",
1190 |        "      <th>Eve_Charge</th>\n",
1191 |        "      <th>Night_Calls</th>\n",
1192 |        "      <th>Night_Charge</th>\n",
1193 |        "      <th>Intl_Calls</th>\n",
1194 |        "      <th>Intl_Charge</th>\n",
1195 |        "      <th>Churn</th>\n",
1196 |        "      <th>Churn_probability</th>\n",
1197 |        "      <th>Churn_predicted</th>\n",
1198 |        "    </tr>\n",
1199 |        "  </thead>\n",
1200 |        "  <tbody>\n",
1201 |        "    <tr>\n",
1202 |        "      <th>68</th>\n",
1203 |        "      <td>126</td>\n",
1204 |        "      <td>0</td>\n",
1205 |        "      <td>211.6</td>\n",
1206 |        "      <td>216.9</td>\n",
1207 |        "      <td>153.5</td>\n",
1208 |        "      <td>7.8</td>\n",
1209 |        "      <td>1</td>\n",
1210 |        "      <td>0</td>\n",
1211 |        "      <td>0</td>\n",
1212 |        "      <td>70</td>\n",
1213 |        "      <td>35.97</td>\n",
1214 |        "      <td>80</td>\n",
1215 |        "      <td>18.44</td>\n",
1216 |        "      <td>60</td>\n",
1217 |        "      <td>6.91</td>\n",
1218 |        "      <td>1</td>\n",
1219 |        "      <td>2.11</td>\n",
1220 |        "      <td>0</td>\n",
1221 |        "      <td>0.35</td>\n",
1222 |        "      <td>0</td>\n",
1223 |        "    </tr>\n",
1224 |        "    <tr>\n",
1225 |        "      <th>1653</th>\n",
1226 |        "      <td>93</td>\n",
1227 |        "      <td>0</td>\n",
1228 |        "      <td>131.4</td>\n",
1229 |        "      <td>219.7</td>\n",
1230 |        "      <td>155.7</td>\n",
1231 |        "      <td>11.1</td>\n",
1232 |        "      <td>1</td>\n",
1233 |        "      <td>1</td>\n",
1234 |        "      <td>0</td>\n",
1235 |        "      <td>78</td>\n",
1236 |        "      <td>22.34</td>\n",
1237 |        "      <td>106</td>\n",
1238 |        "      <td>18.67</td>\n",
1239 |        "      <td>103</td>\n",
1240 |        "      <td>7.01</td>\n",
1241 |        "      <td>2</td>\n",
1242 |        "      <td>3.00</td>\n",
1243 |        "      <td>1</td>\n",
1244 |        "      <td>0.24</td>\n",
1245 |        "      <td>0</td>\n",
1246 |        "    </tr>\n",
1247 |        "    <tr>\n",
1248 |        "      <th>1716</th>\n",
1249 |        "      <td>36</td>\n",
1250 |        "      <td>25</td>\n",
1251 |        "      <td>152.8</td>\n",
1252 |        "      <td>242.8</td>\n",
1253 |        "      <td>147.4</td>\n",
1254 |        "      <td>9.1</td>\n",
1255 |        "      <td>1</td>\n",
1256 |        "      <td>0</td>\n",
1257 |        "      <td>1</td>\n",
1258 |        "      <td>110</td>\n",
1259 |        "      <td>25.98</td>\n",
1260 |        "      <td>67</td>\n",
1261 |        "      <td>20.64</td>\n",
1262 |        "      <td>74</td>\n",
1263 |        "      <td>6.63</td>\n",
1264 |        "      <td>2</td>\n",
1265 |        "      <td>2.46</td>\n",
1266 |        "      <td>0</td>\n",
1267 |        "      <td>0.12</td>\n",
1268 |        "      <td>0</td>\n",
1269 |        "    </tr>\n",
1270 |        "    <tr>\n",
1271 |        "      <th>3251</th>\n",
1272 |        "      <td>88</td>\n",
1273 |        "      <td>0</td>\n",
1274 |        "      <td>274.6</td>\n",
1275 |        "      <td>161.1</td>\n",
1276 |        "      <td>194.4</td>\n",
1277 |        "      <td>9.2</td>\n",
1278 |        "      <td>2</td>\n",
1279 |        "      <td>0</td>\n",
1280 |        "      <td>0</td>\n",
1281 |        "      <td>105</td>\n",
1282 |        "      <td>46.68</td>\n",
1283 |        "      <td>121</td>\n",
1284 |        "      <td>13.69</td>\n",
1285 |        "      <td>123</td>\n",
1286 |        "      <td>8.75</td>\n",
1287 |        "      <td>4</td>\n",
1288 |        "      <td>2.48</td>\n",
1289 |        "      <td>0</td>\n",
1290 |        "      <td>0.16</td>\n",
1291 |        "      <td>0</td>\n",
1292 |        "    </tr>\n",
1293 |        "    <tr>\n",
1294 |        "      <th>2406</th>\n",
1295 |        "      <td>81</td>\n",
1296 |        "      <td>0</td>\n",
1297 |        "      <td>145.6</td>\n",
1298 |        "      <td>287.9</td>\n",
1299 |        "      <td>181.7</td>\n",
1300 |        "      <td>9.2</td>\n",
1301 |        "      <td>2</td>\n",
1302 |        "      <td>0</td>\n",
1303 |        "      <td>0</td>\n",
1304 |        "      <td>59</td>\n",
1305 |        "      <td>24.75</td>\n",
1306 |        "      <td>131</td>\n",
1307 |        "      <td>24.47</td>\n",
1308 |        "      <td>121</td>\n",
1309 |        "      <td>8.18</td>\n",
1310 |        "      <td>4</td>\n",
1311 |        "      <td>2.48</td>\n",
1312 |        "      <td>0</td>\n",
1313 |        "      <td>0.10</td>\n",
1314 |        "      <td>0</td>\n",
1315 |        "    </tr>\n",
1316 |        "    <tr>\n",
1317 |        "      <th>793</th>\n",
1318 |        "      <td>45</td>\n",
1319 |        "      <td>0</td>\n",
1320 |        "      <td>112.8</td>\n",
1321 |        "      <td>218.8</td>\n",
1322 |        "      <td>240.2</td>\n",
1323 |        "      <td>9.0</td>\n",
1324 |        "      <td>2</td>\n",
1325 |        "      <td>0</td>\n",
1326 |        "      <td>0</td>\n",
1327 |        "      <td>108</td>\n",
1328 |        "      <td>19.18</td>\n",
1329 |        "      <td>120</td>\n",
1330 |        "      <td>18.60</td>\n",
1331 |        "      <td>106</td>\n",
1332 |        "      <td>10.81</td>\n",
1333 |        "      <td>3</td>\n",
1334 |        "      <td>2.43</td>\n",
1335 |        "      <td>0</td>\n",
1336 |        "      <td>0.06</td>\n",
1337 |        "      <td>0</td>\n",
1338 |        "    </tr>\n",
1339 |        "    <tr>\n",
1340 |        "      <th>1029</th>\n",
1341 |        "      <td>116</td>\n",
1342 |        "      <td>0</td>\n",
1343 |        "      <td>201.8</td>\n",
1344 |        "      <td>231.5</td>\n",
1345 |        "      <td>226.1</td>\n",
1346 |        "      <td>16.5</td>\n",
1347 |        "      <td>0</td>\n",
1348 |        "      <td>0</td>\n",
1349 |        "      <td>0</td>\n",
1350 |        "      <td>82</td>\n",
1351 |        "      <td>34.31</td>\n",
1352 |        "      <td>95</td>\n",
1353 |        "      <td>19.68</td>\n",
1354 |        "      <td>130</td>\n",
1355 |        "      <td>10.17</td>\n",
1356 |        "      <td>5</td>\n",
1357 |        "      <td>4.46</td>\n",
1358 |        "      <td>0</td>\n",
1359 |        "      <td>0.08</td>\n",
1360 |        "      <td>0</td>\n",
1361 |        "    </tr>\n",
1362 |        "    <tr>\n",
1363 |        "      <th>1943</th>\n",
1364 |        "      <td>125</td>\n",
1365 |        "      <td>0</td>\n",
1366 |        "      <td>168.6</td>\n",
1367 |        "      <td>175.6</td>\n",
1368 |        "      <td>243.3</td>\n",
1369 |        "      <td>10.9</td>\n",
1370 |        "      <td>0</td>\n",
1371 |        "      <td>0</td>\n",
1372 |        "      <td>0</td>\n",
1373 |        "      <td>99</td>\n",
1374 |        "      <td>28.66</td>\n",
1375 |        "      <td>107</td>\n",
1376 |        "      <td>14.93</td>\n",
1377 |        "      <td>92</td>\n",
1378 |        "      <td>10.95</td>\n",
1379 |        "      <td>7</td>\n",
1380 |        "      <td>2.94</td>\n",
1381 |        "      <td>0</td>\n",
1382 |        "      <td>0.04</td>\n",
1383 |        "      <td>0</td>\n",
1384 |        "    </tr>\n",
1385 |        "    <tr>\n",
1386 |        "      <th>1161</th>\n",
1387 |        "      <td>40</td>\n",
1388 |        "      <td>0</td>\n",
1389 |        "      <td>170.7</td>\n",
1390 |        "      <td>179.1</td>\n",
1391 |        "      <td>281.9</td>\n",
1392 |        "      <td>8.2</td>\n",
1393 |        "      <td>3</td>\n",
1394 |        "      <td>1</td>\n",
1395 |        "      <td>0</td>\n",
1396 |        "      <td>55</td>\n",
1397 |        "      <td>29.02</td>\n",
1398 |        "      <td>108</td>\n",
1399 |        "      <td>15.22</td>\n",
1400 |        "      <td>89</td>\n",
1401 |        "      <td>12.69</td>\n",
1402 |        "      <td>9</td>\n",
1403 |        "      <td>2.21</td>\n",
1404 |        "      <td>0</td>\n",
1405 |        "      <td>0.41</td>\n",
1406 |        "      <td>0</td>\n",
1407 |        "    </tr>\n",
1408 |        "    <tr>\n",
1409 |        "      <th>2837</th>\n",
1410 |        "      <td>75</td>\n",
1411 |        "      <td>0</td>\n",
1412 |        "      <td>203.3</td>\n",
1413 |        "      <td>228.9</td>\n",
1414 |        "      <td>222.2</td>\n",
1415 |        "      <td>14.3</td>\n",
1416 |        "      <td>1</td>\n",
1417 |        "      <td>0</td>\n",
1418 |        "      <td>0</td>\n",
1419 |        "      <td>70</td>\n",
1420 |        "      <td>34.56</td>\n",
1421 |        "      <td>97</td>\n",
1422 |        "      <td>19.46</td>\n",
1423 |        "      <td>118</td>\n",
1424 |        "      <td>10.00</td>\n",
1425 |        "      <td>3</td>\n",
1426 |        "      <td>3.86</td>\n",
1427 |        "      <td>0</td>\n",
1428 |        "      <td>0.19</td>\n",
1429 |        "      <td>0</td>\n",
1430 |        "    </tr>\n",
1431 |        "  </tbody>\n",
1432 |        "</table>\n",
1433 |        "</div>"
1434 |       ],
1435 |       "text/plain": [
1436 |        "      Account_Length  Vmail_Message  Day_Mins  Eve_Mins  Night_Mins  \\\n",
1437 |        "68               126              0     211.6     216.9       153.5   \n",
1438 |        "1653              93              0     131.4     219.7       155.7   \n",
1439 |        "1716              36             25     152.8     242.8       147.4   \n",
1440 |        "3251              88              0     274.6     161.1       194.4   \n",
1441 |        "2406              81              0     145.6     287.9       181.7   \n",
1442 |        "793               45              0     112.8     218.8       240.2   \n",
1443 |        "1029             116              0     201.8     231.5       226.1   \n",
1444 |        "1943             125              0     168.6     175.6       243.3   \n",
1445 |        "1161              40              0     170.7     179.1       281.9   \n",
1446 |        "2837              75              0     203.3     228.9       222.2   \n",
1447 |        "\n",
1448 |        "      Intl_Mins  CustServ_Calls  Intl_Plan  Vmail_Plan  Day_Calls  Day_Charge  \\\n",
1449 |        "68          7.8               1          0           0         70       35.97   \n",
1450 |        "1653       11.1               1          1           0         78       22.34   \n",
1451 |        "1716        9.1               1          0           1        110       25.98   \n",
1452 |        "3251        9.2               2          0           0        105       46.68   \n",
1453 |        "2406        9.2               2          0           0         59       24.75   \n",
1454 |        "793         9.0               2          0           0        108       19.18   \n",
1455 |        "1029       16.5               0          0           0         82       34.31   \n",
1456 |        "1943       10.9               0          0           0         99       28.66   \n",
1457 |        "1161        8.2               3          1           0         55       29.02   \n",
1458 |        "2837       14.3               1          0           0         70       34.56   \n",
1459 |        "\n",
1460 |        "      Eve_Calls  Eve_Charge  Night_Calls  Night_Charge  Intl_Calls  \\\n",
1461 |        "68           80       18.44           60          6.91           1   \n",
1462 |        "1653        106       18.67          103          7.01           2   \n",
1463 |        "1716         67       20.64           74          6.63           2   \n",
1464 |        "3251        121       13.69          123          8.75           4   \n",
1465 |        "2406        131       24.47          121          8.18           4   \n",
1466 |        "793         120       18.60          106         10.81           3   \n",
1467 |        "1029         95       19.68          130         10.17           5   \n",
1468 |        "1943        107       14.93           92         10.95           7   \n",
1469 |        "1161        108       15.22           89         12.69           9   \n",
1470 |        "2837         97       19.46          118         10.00           3   \n",
1471 |        "\n",
1472 |        "      Intl_Charge  Churn  Churn_probability  Churn_predicted  \n",
1473 |        "68           2.11      0               0.35                0  \n",
1474 |        "1653         3.00      1               0.24                0  \n",
1475 |        "1716         2.46      0               0.12                0  \n",
1476 |        "3251         2.48      0               0.16                0  \n",
1477 |        "2406         2.48      0               0.10                0  \n",
1478 |        "793          2.43      0               0.06                0  \n",
1479 |        "1029         4.46      0               0.08                0  \n",
1480 |        "1943         2.94      0               0.04                0  \n",
1481 |        "1161         2.21      0               0.41                0  \n",
1482 |        "2837         3.86      0               0.19                0  "
1483 |       ]
1484 |      },
1485 |      "metadata": {},
1486 |      "output_type": "display_data"
1487 |     }
1488 |    ],
1489 |    "source": [
1490 |     "test = pd.concat([X_test, y_test], axis=1)\n",
1491 |     "\n",
1492 |     "print(test.shape)\n",
1493 |     "display(test.head(10))"
1494 |    ]
1495 |   },
1496 |   {
1497 |    "cell_type": "markdown",
1498 |    "metadata": {},
1499 |    "source": [
1500 |     "#### Confusion Matrix"
1501 |    ]
1502 |   },
1503 |   {
1504 |    "cell_type": "code",
1505 |    "execution_count": 62,
1506 |    "metadata": {},
1507 |    "outputs": [
1508 |     {
1509 |      "data": {
1510 |       "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEGCAYAAABxfL6kAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAkE0lEQVR4nO3de7xd853/8df75E7ikhKCVIIEcYuKqBoq+BHaCp0iWm2oqdKU0VZdRluGZqQdrSnGtKGtKBVxDzUuTWm0LrmVkBBScckkcnMNEkl8fn+s72E7PWfvdU72yb7k/exjPfZaa6/1Xd99dn3y3d/1XZ+vIgIzM6sfDZWugJmZlZcDu5lZnXFgNzOrMw7sZmZ1xoHdzKzOdKx0BdZ36tgt1LlHpathrbDnzp+sdBWslWbMmL40IjZv6/kdNto2YvV7uY6N95bcFxHD2nqtcnBgrzB17kGXHY+tdDWsFf76+JWVroK1UrdOemltzo/VK+iy04hcx6742xWbrc21ysGB3cysFAFSpWuRmwO7mVkeqp1bkg7sZmZ5uMVuZlZPBA0dKl2J3Grnt4WZWaWIrCsmz5KnOOk7kmZJelrSjZK6Suop6QFJz6fXTQuOP0/SXElzJB1WqnwHdjOzkpR1xeRZSpUkbQ2cAQyOiF2BDsAI4FxgUkT0ByalbSQNTO/vAgwDrpJU9OeDA7uZWR5lbLGTdYN3k9QR2ABYAAwHxqX3xwFHpfXhwPiIWBkR84C5wJBihTuwm5nlkb/FvpmkaQXLKYXFRMT/AZcCLwMLgTcj4n5gi4hYmI5ZCPRKp2wNvFJQxPy0r0W+eWpmVpJa0xpfGhGDWywp6zsfDvQD3gBulnRC8Yv/g6ITaTiwm5mVIso5KuYQYF5ELAGQdBvwGWCRpN4RsVBSb2BxOn4+0Kfg/G3Ium5a5K4YM7OSVM4+9peBT0vaQJKAg4FngInAyHTMSODOtD4RGCGpi6R+QH9gSrELuMVuZpZHQ3keUIqIxyXdAswAVgN/A8YC3YEJkk4mC/7HpONnSZoAzE7Hj4qINcWu4cBuZlZK4zj2MomIC4ALmuxeSdZ6b+740cDovOU7sJuZ5eGUAmZm9aS2Ugo4sJuZ5eHsjmZmdSRnuoBq4cBuZpaHW+xmZnXGLXYzs3rSqpQCFefAbmZWSnlTCrQ7B3Yzs5LcYjczqz/uYzczqzNusZuZ1Rm32M3M6ojcx25mVnfU4MBuZlY3BMhdMWZmdUQ0P/NolXJgNzMrSW6xm5nVm1oK7LVzN8DMrIIaGhpyLaVI2lHSEwXLW5LOlNRT0gOSnk+vmxacc56kuZLmSDqsZF3X8rOamdU/tWIpISLmRMSgiBgE7AW8C9wOnAtMioj+wKS0jaSBwAhgF2AYcJWkoolrHNjNzEpQ6mPPs7TSwcDfI+IlYDgwLu0fBxyV1ocD4yNiZUTMA+YCQ4oV6j52M7McWhG0N5M0rWB7bESMbeHYEcCNaX2LiFgIEBELJfVK+7cGHis4Z37a1yIHdjOzHFoR2JdGxOAc5XUGjgTOK3VoM/ui2AnuijEzy6EdumIOB2ZExKK0vUhS73St3sDitH8+0KfgvG2ABcUKdmA3MytFoAblWlrheD7qhgGYCIxM6yOBOwv2j5DURVI/oD8wpVjB7ooxMytBZX5ASdIGwP8DvlmwewwwQdLJwMvAMQARMUvSBGA2sBoYFRFripXvwG5mlkM5A3tEvAt8osm+ZWSjZJo7fjQwOm/5DuxmZnnUzoOnDuxmZiWptlIKOLCbmeXgwG5mVkeEcuWBqRYO7GZmedROg92B3cysJPexm5nVHwd2M7M648BuZlZnWpkuoKIc2K1NTjt+KF896jMQwey5Cxh10fV87+uHccQBu/NBBEtee5tR/349ry59kwOH7MQF3z6Szp068v6q1fzo8jt4eNpzlf4I67X5r77OaRdex+Jlb9EgMfLo/Tj1+KGM/p+7uWfyTBokNu/Zg/++4AR6b75JpatbcW3MtV4xiiia/bH9LizdA3w5It6QtDwiukvqC9wdEbuWofwzyfIgv5vz+BOBwRHx7bW9dms0bNAruux47Lq85FrrvfnG/O/V3+HTx41mxcpV/OY/vs4Dj8zi7gef5O13VgBwynGfZad+vfnumPHsNmAblrz2Nq8ufZOdt+/NLZePYpfP/aDCn6LtXp96ZaWrsNZeXfomi5a+xR479eHtd1Yw9Gs/4fr/PIWtem3CRt27AfCr8Q/x7LyFXHbe8RWu7drr1knT86TSbUmXLfpH7xGX5Tr2pcu/sFbXKoeKDcyMiCMi4o12vMSZwAbtWP7HSFqvfv107NiBrl060aFDAxt07cyrS978MKgDbNitC42Nhqeem8+rS98E4Jm/L6Rr50507rRe/bmqzpabbcweO2WZYHts2JUBfbdk4ZI3PgzqAO+8t7KmWqntrZ1mUGoX7fJfl6SzgRURcbmky4A9IuIgSQcDJ0XECZJeJGshL81R3oHAhcBSYFdgOnBCREQq89L0WaYCp5FlTNsKeFDS0ogY2qS8vYFfABsCK/ko8c5Wku4Ftgduj4iz0/HLI6J7Wv8S8PmIOFHStcBrwJ7ADEmfAN4CBgNbAmdHxC2t++tVv4VL3uSK6yfx1F0Xs2Ll+zz4+LM8+PizAPzgtC8w4nNDeGv5e3zh1Mv/4dwjDxrEzOde4f1Vq9d1ta0FLy9Yxsw589lrl74AXHzVRMb/YQobde/GXb88o7KVqybVEbNzaa8W+2Rg/7Q+GOguqRPwT8DDbSxzT7JW+EBgO2A/SV2Ba4HjImI3suB+WkRcTpaIfmgzQb0zcBPwrxGxB3AI8F56exBwHLAbcJykwuT2LRkAHBIR30vbvck+5+fJ0nD+A0mnSJomaVqsfq+5Q6raxj26ccQBuzFo+AXsfPj5bNC1M8cevjcAP/6fu9j18z/k5nun8Y1jD/jYeTtttyUXnj6c7/zH+EpU25qx/N2VfO2ca7jku//8YWv9h986kll/+DHHDBvM1RMmV7iG1aOWWuztFdinA3tJ6kHWIn6ULMDvT9sD+5SImB8RHwBPAH2BHYF5EdF4J24ccEDzp39oR2BhREwFiIi3IqKx+TgpIt6MiBVkuY+3zVGvm5vkRr4jIj6IiNnAFs2dEBFjI2JwRAxWx27NHVLVDhyyEy8tWMayN5azes0H3PXgkwzZvd/Hjrnl3qkcedCgD7e36rUJv/vpKZx2we948f9K/kizdWDV6jWMPOdqjhk2mC8UfFeNvjRsbyb+6Yl1Xq9qJEFDg3It1aBdAntErAJeBE4CHiEL5kPJujieaWOxKwvW15C1ztvyVxQtzxfY3DVocnzXJue8U6SM6viWy2z+q68xeLd+dOvSCYDP7r0jc+YtYrs+m394zLADdue5F7MZvzbq3o2bLjuVi/57Io/PfKEidbaPiwhOv/gGBvTdklFf+SgF+N9fXvzh+r2TZzKgb7Ntk/VQvtZ6tbTY2/MO1mTgLODrwFPAz4HpUd5hOM8CfSXtEBFzga8Cf07vvQ30IOuXb3rOVpL2joip6VdFqf6QRZJ2BuYAR6ey11vTZ73ExEl/46Hrz2HNmg+YOWc+427/K1f/+ET6b9uLDz4IXnn1Nb57Sdbl8o1jD6Bfn835/r8M4/v/MgyAL377Spa+vrySH2O99tiTL3DTPVMYuMNW7P/lSwD44agjuf7OR3j+pcU0NIg+W/bk5+eNqHBNq0eVxOxc2jOwPwycDzwaEe9IWkHbu2GaFRErJJ0E3JxGpUwFfpneHgv8r6SFhf3sEfG+pOOAKyR1Iwvqh5S41LnA3cArwNNA93J+jlo0Zuw9jBl7z8f2jTznmmaP/dlv7uNnv7lvXVTLctp30PbNDts8dL9dKlCb2lAtrfE8KjaO3TK1OI59fVcP49jXN2s7jr1r7wHRd+QVuY6d85NhJa8laRPgGrJRfkHWszGHbGBHX7Ku7GMj4vV0/HnAyWRdxGdERNGWUu0kGDYzqxBR9punvwDujYidgD3I7j2eSzaAoz8wKW0jaSAwAtgFGAZcJalDscId2M3McihXYJe0EdnovV9D1j2cHtYcTjayj/R6VFofDoyPiJURMQ+YCwwpWtc2fD4zs/WLspuneRZgs8bnVNJySpPStgOWAL+V9DdJ10jaENgiIhYCpNde6fitye7vNZqf9rXIz3WbmZUgWnXzdGmJPvaOwKeA0yPicUm/IHW7FLl8U0VvjrrFbmZWUlnHsc8H5kfE42n7FrJAv0hSb4D0urjg+MKn4Lche7K+RQ7sZmY5tKIrpqiIeBV4RdKOadfBZE+6TwRGpn0jgTvT+kRghKQukvoB/YEpxa7hrhgzs1JSSoEyOh24IeWueoHsKf0GYIKkk4GXgWMAImKWpAlkwX81MKpJGpN/4MBuZlZCK/vYS4qIJ8jyZzV1cDP7iIjRwOi85Tuwm5nlUEMPnjqwm5nlUUspBRzYzcxyqKG47sBuZlaS3GI3M6sronom0cjDgd3MLIcaarA7sJuZ5eGuGDOzepLzqdJq4cBuZlZCuR9Qam8O7GZmOTiwm5nVGY+KMTOrJ+5jNzOrLyJ3rvWq4MBuZpZDDcV1B3YzszwaaiiyO7CbmZWg8k+00a4c2M3McqihuO45T83M8ijjZNZIelHSU5KekDQt7esp6QFJz6fXTQuOP0/SXElzJB1WqvwWW+ySrgCipfcj4oxcn8DMrA60Qxf70IhYWrB9LjApIsZIOjdtnyNpIDAC2AXYCvijpAHF5j0t1hUzrQwVNzOreSIb8tjOhgMHpvVxwEPAOWn/+IhYCcyTNBcYAjzaUkEtBvaIGFe4LWnDiHhnraptZlajytzHHsD9kgL4VUSMBbaIiIUAEbFQUq907NbAYwXnzk/7WlTy5qmkfYFfA92BT0raA/hmRHyr1R/FzKwWqVUTbWzW2G+ejE2Bu9B+EbEgBe8HJD1b7OrN7GuxmxzyjYr5L+AwYCJARDwp6YAc55mZ1QXRqnHsSyNicLEDImJBel0s6XayrpVFknqn1npvYHE6fD7Qp+D0bYAFxcrPNSomIl5psqvFTnszs3ok5VtKl6MNJfVoXAcOBZ4mazyPTIeNBO5M6xOBEZK6SOoH9AemFLtGnhb7K5I+A4SkzsAZwDM5zjMzqxtlzBWzBXB7Kq8j8PuIuFfSVGCCpJOBl4FjACJilqQJwGxgNTCq2IiYxkJLORX4BVln/f8B9wGj2vZ5zMxqT97WeB4R8QKwRzP7lwEHt3DOaGB03muUDOxpnOVX8hZoZlaPOtRQrpiSfeyStpN0l6QlkhZLulPSduuicmZm1aKcT562tzw3T38PTAB6kz31dDNwY3tWysysmmSjYvIt1SBPYFdE/C4iVqflekqMoTQzqys5W+vV0mIvliumZ1p9MOUtGE8W0I8D/rAO6mZmVjWqJGbnUuzm6XSyQN74cb5Z8F4AF7dXpczMqk21tMbzKJYrpt+6rIiZWbUS0KFaOtBzyDXRhqRdgYFA18Z9EXFde1XKzKza1E5Yz5cE7AKyVJIDgXuAw4G/AA7sZrZekGprztM8o2K+RPY01KsRcRLZE1Nd2rVWZmZVply5YtaFPF0x70XEB5JWS9qILOOYH1Ays/VKXdw8LTBN0ibA1WQjZZZTIrOYmVm9qaG4nitXTOOEGr+UdC+wUUTMbN9qmZlVD0n1MSpG0qeKvRcRM9qnSmZm1adeumJ+VuS9AA4qc13WS7sM6MPEBy6tdDXMrIRcsxJViWIPKA1dlxUxM6tWon5a7GZmltRQF7sDu5lZKVJtpRSopW4jM7OKKXc+dkkdJP1N0t1pu6ekByQ9n143LTj2PElzJc2RdFjJuua4uCSdIOlHafuTkobkr76ZWe1rhydP/xV4pmD7XGBSRPQHJqVtJA0ERgC7AMOAqyR1KFZwnhb7VcC+wPFp+23gv1tTezOzWpbNoKRcS67ypG2AzwHXFOweDoxL6+OAowr2j4+IlRExD5gLFG1c5wns+0TEKGAFQES8DnTOVXszszrRkHMBNpM0rWA5pZni/gs4G/igYN8WEbEQIL32Svu3Bl4pOG5+2teiPDdPV6VmfwBI2rxJZczM6l4rulmWRsTglsvR54HFETFd0oF5Lt3MvqLTk+YJ7JcDtwO9JI0my/b4gxznmZnVhTKnFNgPOFLSEWRzXGwk6XpgkaTeEbFQUm+yhIuQtdD7FJy/DbCg2AVKdsVExA1kPxkuARYCR0XEza3+KGZmNaxco2Ii4ryI2CYi+pLdFP1TRJwATARGpsNGAnem9YnACEldJPUD+lMiEWOeiTY+CbwL3FW4LyJeLv0RzMxqX+PN03Y2Bpgg6WTgZeAYgIiYJWkCMBtYDYyKiDXFCsrTFfMHPprUuivQD5hDNvTGzGy90B5xPSIeAh5K68vIJjVq7rjRwOi85eZJ27tb4XbK+vjNvBcwM6t5rXz4qNJanVIgImZI2rs9KmNmVq1UQ9NZ5+lj/27BZgPwKWBJu9XIzKzKCOhYQwlY8rTYexSsrybrc7+1fapjZlad6iZtb3owqXtEfH8d1cfMrOpko2IqXYv8ik2N1zEiVhebIs/MbL3Q+gRfFVWsxT6FrD/9CUkTgZuBdxrfjIjb2rluZmZVYx2MYy+bPH3sPYFlZHOcNo5nD8CB3czWCwI61MnN015pRMzTfBTQGxVNQGNmVl9EQ50Md+wAdKcNmcXMzOpJNpl1pWuRX7HAvjAiLlpnNTEzq1Z19ORpDX0MM7P2VS83T5tNRmNmtr6pm66YiHhtXVbEzKyalXGijXbX6iRgZmbrG5Fvguhq4cBuZlaK6ihXjJmZZWonrDuwm5mVtI6mxiubWuo2MjOrGOVcSpYjdZU0RdKTkmZJ+ve0v6ekByQ9n143LTjnPElzJc2RdFipaziwm5mVJBoa8i05rAQOiog9gEHAMEmfBs4FJkVEf2BS2kbSQGAE2TzTw4CrUkr1Fjmwm5mV0DgqJs9SSmSWp81OaQlgODAu7R8HHJXWhwPjI2JlRMwD5gJDil3Dgd3MLAdJuRZgM0nTCpZTmimrg6QngMXAAxHxOLBFRCwESK+90uFbA68UnD4/7WuRb56ameXQilunSyNicLEDImINMEjSJsDtknZt5aWLJmJ0i93MrBS1qsWeW0S8ATxE1ne+SFJvgPS6OB02H+hTcNo2wIJi5Tqwm5mVIKCDlGspWZa0eWqpI6kbcAjwLDARGJkOGwncmdYnAiMkdZHUD+hPNsNdi9wVY2aWQxlHsfcGxqWRLQ3AhIi4W9KjwARJJwMvA8cARMQsSROA2cBqYFTqymmRA7uZWQ7lej4pImYCezazfxktZNWNiNHA6LzXcGA3MyshG+5YO0+eOrCbmeVQQxkFHNjNzEoTcovdzKx+NI6KqRUO7GZmpchdMWZmdceB3cyszriP3cysjmQTbVS6Fvk5sJuZ5VBLMyg5sJuZ5eCuGKt75196E39+fDY9N+nOxKu/D8B/jr2Lhx6bTaeOHemz1ScYfdZxbNS9G2+89Q5nXnQdT815haMPHcwPTv9ihWtv8199ndMuvI7Fy96iQWLk0ftx6vFDP3z/it/9kR9dfgdzHxjDJzbpXsGaVoda64qpm+yOku4pyJi2PL32lfR0K8p4SFLRPMqWOfrQwYz9j298bN9nPjWAO68+izvGfo++W2/G1TdOAqBzp46cfuIwvn/K5ytRVWtGx44N/PjML/L4zT/k/t+exTW3TObZFxYCWdB/aMqzbLPlpiVKWZ8o9/+qQd0E9og4IuU2rphS8xDWk8G7b8/GPTb42L79Bu9Ixw7Zn2CPnbfl1aVvArBBty7stWs/unTutM7rac3bcrON2WOnLMV3jw27MqDvlixc8gYA5192KxeeflSrc4vXtTSOPc9SDWoisEs6W9IZaf0ySX9K6wdLuj6tvyhps1aW+VSaKXxMwVvHpBnEn5O0fzr2RElXFpx7t6QD0/pySRdJehzYN22PTuU+JmmLtf38tei2+6aw/947VboalsPLC5Yxc8589tqlL/f8eSa9N9+E3QZsU+lqVR3lXKpBTQR2YDKwf1ofDHSX1An4J+Dh1hYm6XCyiWL3STOF/7Tg7Y4RMQQ4E7ggR3EbAk9HxD4R8Ze0/VgqdzLwjaYnSDqlcT7E15YtaW31q94vb/gjHTp04AsHf6rSVbESlr+7kq+dcw2XfPef6dixAz//7X2cd+rnKl2tqlPOiTbWhVoJ7NOBvST1AFYCj5IF+P1pQ2Anm7HktxHxLkBEvFbw3m0F1+ybo6w1wK0F2+8DdxcrIyLGRsTgiBjc8xObt67mVe6O+6fy58ef4afnftk/5avcqtVrGHnO1RwzbDBfOGgQ8+Yv4aUFy9j/y5ew+5E/YsHiN/jsCT9h0dK3Kl3V6lBDTfaaGBUTEaskvQicBDwCzASGAtsDz7ShSNHyZLAr0+saPvr7rObj/wh2LVhf0WQ2k1UR0Vh2YRl17+Gpz3LNTQ9y3c++RbeunStdHSsiIjj94hsY0HdLRn0lm9thlx225vn7P+qV3P3IH/HgdWd7VExSLTdG86iloDMZOAv4OvAU8HNgekEQbY37gR9J+n1EvCupZ5NWe1MvAt+S1ABsDQxpwzXrylmjr2fKzL/zxpvvMPT4i/n21w5l7Pg/sWrVak4+ZywAe+z8SS4880sAHHLCaJa/u4JVq9Yw6ZFZXD3mG+yw7ZaV/AjrtceefIGb7pnCwB22Yv8vXwLAD0cdyaH77VLhmlWvWvoBWkuB/WHgfODRiHhH0gra1g1DRNwraRAwTdL7wD3AvxU55a/APLJ/UJ4GZrTluvXk0vNP+Id9/3z4Pi0e/8frz2/P6lgr7Ttoe16femXRY2ZOvGgd1aY2lCuuS+oDXAdsCXwAjI2IX0jqCdxE1n37InBsRLyezjkPOJmsF+CMiLiv6DXa1uC1ctlt0F4x8Y9/rXQ1rBV6b9K19EFWVbp10vSIaPMzKjvvtmdcN/GhXMcO2W6ToteS1BvoHREz0n3D6WSDOU4EXouIMZLOBTaNiHMkDQRuJOsp2Ar4IzCg2ITWtXLz1MysYqQsV0yepZSIWBgRM9L622T3CbcGhgPj0mHjyII9af/4iFgZEfOAuZToDnZgNzPLoRWDYjZrHM6cllNaLFPqC+wJPA5sERELIQv+QK902NbAKwWnzU/7WlRLfexmZpWTv5N9aZ5uH0ndyYZKnxkRbxUZHtzcG0X70N1iNzMrqby5YtIDlrcCN0RE47Mzi1L/e2M//OK0fz7Qp+D0bYAFxcp3YDczy6FcuWKUNc1/DTwTET8veGsiMDKtjwTuLNg/QlIXSf2A/sCUYtdwV4yZWQmirOPY9wO+Cjwl6Ym079+AMcAESScDLwPHAETELEkTgNlkD0uOKjYiBhzYzcxyKdeTpymnVEuFHdzCOaOB0Xmv4cBuZpaDnzw1M6szNRTXHdjNzEqqosyNeTiwm5nl4OyOZmZ1pNYms3ZgNzPLw4HdzKy+uCvGzKzOeLijmVmdqaG47sBuZpZLDUV2B3YzsxIaJ9qoFQ7sZmY51E5Yd2A3M8unhiK7A7uZWUn5J9GoBg7sZmY51FAXuwO7mVkpZZ5oo905sJuZ5eCuGDOzOlNLLXZPZm1mloNyLiXLkX4jabGkpwv29ZT0gKTn0+umBe+dJ2mupDmSDstTVwd2M7NSlLXY8yw5XAsMa7LvXGBSRPQHJqVtJA0ERgC7pHOuktSh1AUc2M3McilPmz0iJgOvNdk9HBiX1scBRxXsHx8RKyNiHjAXGFLqGu5jNzMroZUTbWwmaVrB9tiIGFvinC0iYiFARCyU1Cvt3xp4rOC4+WlfUQ7sZmY5tOLm6dKIGFyuyzazL0qd5K4YM7MclPN/bbRIUm+A9Lo47Z8P9Ck4bhtgQanCHNjNzPIo17CY5k0ERqb1kcCdBftHSOoiqR/QH5hSqjB3xZiZ5VCuYeySbgQOJOuLnw9cAIwBJkg6GXgZOAYgImZJmgDMBlYDoyJiTalrOLCbmZXQiqGMJUXE8S28dXALx48GRrfmGg7sZmY5qIYePXVgNzPLoXbCugO7mVkuNdRgd2A3MyvNE22YmdUV52M3M6tDDuxmZnXGXTFmZvWkjOPY1wUHdjOzEtYuW8C658BuZpZHDUV2B3Yzsxzcx25mVmdaMdFGxTmwm5nl4cBuZlZf3BVjZlZHau3JU0WUnD7P2pGkJcBLla5HO9gMWFrpSlir1PN3tm1EbN7WkyXdS/b3yWNpRAxr67XKwYHd2oWkaWWc0NfWAX9n9cNznpqZ1RkHdjOzOuPAbu1lbKUrYK3m76xOuI/dzKzOuMVuZlZnHNjNzOqMA/t6SNI9kjZJ68vTa19JT5ep/DMlbdCK40+UdGU5rr2+K8d3K+khSR72WMMc2NdDEXFERLzRjpc4E8gd2NeWJD9BnayD77YkSR0qeX1zYK87ks6WdEZav0zSn9L6wZKuT+svSsr1FJ2kA1ML7hZJz0q6Qcoerk5l/k3SU5J+I6lLuvZWwIOSHmymvL0lPSLpSUlTJPVIb20l6V5Jz0v6acHxywvWvyTp2rR+raSfp2v8JG1fnsp+QdKX2vDnq2rl/m4LynwqfR9jCt46Jn0/z0naPx37sV9Wku6WdGBaXy7pIkmPA/um7dGp3MckbbG2n9/yc2CvP5OB/dP6YKC7pE7APwEPt7HMPcla4QOB7YD9JHUFrgWOi4jdyPIOnRYRlwMLgKERMbSwEEmdgZuAf42IPYBDgPfS24OA44DdgOMk9clRrwHAIRHxvbTdm+xzfh4Y0+JZtaus362kw4GjgH3S9/HTgrc7RsQQsu/9ghzFbQg8HRH7RMRf0vZjqdzJwDdaWz9rOwf2+jMd2Cu1hFcCj5IFgf1pe2CfEhHzI+ID4AmgL7AjMC8inkvHjAMOKFHOjsDCiJgKEBFvRcTq9N6kiHgzIlYAs4Ftc9Tr5ohYU7B9R0R8EBGzgXpsIZb7uz0E+G1EvAsQEa8VvHdbwTX75ihrDXBrwfb7wN2tLMPKxH2TdSYiVkl6ETgJeASYCQwFtgeeaWOxKwvW15D9/6Ytue4EtPTgRHPXoMnxXZuc806RMmooF18+7fDd5vk+Cr+L1Xy8MVj4faxo8o/sqvjoIZnCMmwdcIu9Pk0GzkqvDwOnAk8U/IdWDs8CfSXtkLa/Cvw5rb8N9GjhnK0k7Q0gqUeOG5+LJO0sqQE4ugz1rnXl/G7vB77eOIJJUs8Sx78IDJLUkLrKhrThmrYOOLDXp4fJ+psfjYhFwAra3g3TrNRlchJws6SngA+AX6a3xwL/2/TmaUS8T9aPfoWkJ4EH+MdWeFPnkv2k/xOwsHyfoGaV7buNiHuBicA0SU+Q/YNRzF+BecBTwKXAjLZc19qfUwqYmdUZt9jNzOqMA7uZWZ1xYDczqzMO7GZmdcaB3cyszjiwW9WTtEbSE5KelnRzazJHNlPWtY15ZCRdI2lgkWMPlPSZNlyj2XwtefK4FObGyXmtCyWVGqZo6xkHdqsF70XEoIjYlexR9VML32xrNsGI+JeUfqAlBwKtDuxmlebAbrXmYWCH1Jp+UNLvgackdZD0n5KmSpop6ZsAylwpabakPwC9GgtSQd5xScMkzUjZCCdJ6kv2D8h30q+F/SVtLunWdI2pkvZL535C0v3KMl3+ihzpDCTdIWm6pFmSTmny3s9SXSZJ2jzt215Z9svpkh6WtFNZ/ppWl5y/wWpGSj9wOHBv2jUE2DUi5qXg+GZE7C2pC/BXSfeTZabckSxr5BZkCcZ+06TczYGrgQNSWT0j4jVJvwSWR8Sl6bjfA5dFxF8kfRK4D9iZLPvhXyLiIkmfAz4WqFvw9XSNbsBUSbdGxDKyrIgzIuJ7kn6Uyv422dO8p0bE85L2Aa4CDmrDn9HWAw7sVgu6pUfeIWux/5qsi2RKRMxL+w8FdtdHedg3BvqTZZy8MSWoWqCUw7yJTwOTG8tqkuWw0CHAQOnDBvlGKdPiAcAX07l/kPR6js90hqTG3Dd9Ul2XkaVmuCntvx64TVL39HlvLrh2lxzXsPWUA7vVgvciYlDhjhTgCrM7Cjg9Iu5rctwRtJzBsPDcPLk1GoB9I+K9wp2pLrlzc6TJKQ5JZb0r6SFazpkT6bpvNP0bmLXEfexWL+4DTksTTyBpgKQNybIgjkh98L3J0tw29SjwWUn90rmNWQ6bZqm8n6xbhHTcoLQ6GfhK2nc4sGmJum4MvJ6C+k5kvxgaNQCNvzq+TNbF8xYwT9Ix6RqStEeJa9h6zIHd6sU1ZP3nM5RN3Pwrsl+ktwPPk2Uk/B8+Si38oYhYQtYvflvKOtnYFXIXcHTjzVPgDGBwujk7m49G5/w7cICkGWRdQi+XqOu9QEdJM4GLgccK3nsH2EXSdLI+9IvS/q8AJ6f6zQKG5/ib2HrK2R3NzOqMW+xmZnXGgd3MrM44sJuZ1RkHdjOzOuPAbmZWZxzYzczqjAO7mVmd+f+zJEOQB7swrAAAAABJRU5ErkJggg==\n",
1511 |       "text/plain": [
1512 |        "<Figure size 432x288 with 2 Axes>"
1513 |       ]
1514 |      },
1515 |      "metadata": {
1516 |       "needs_background": "light"
1517 |      },
1518 |      "output_type": "display_data"
1519 |     }
1520 |    ],
1521 |    "source": [
1522 |     "plot_confusion_matrix(model,\n",
1523 |     "                      X_test,\n",
1524 |     "                      y_test[\"Churn\"],\n",
1525 |     "                      display_labels = [\"will not churn\", \"will churn\"],\n",
1526 |     "                      cmap = \"Blues\")\n",
1527 |     "\n",
1528 |     "plt.show()"
1529 |    ]
1530 |   },
1531 |   {
1532 |    "cell_type": "code",
1533 |    "execution_count": 63,
1534 |    "metadata": {},
1535 |    "outputs": [
1536 |     {
1537 |      "name": "stdout",
1538 |      "output_type": "stream",
1539 |      "text": [
1540 |       "[[832  23]\n",
1541 |       " [121  24]]\n"
1542 |      ]
1543 |     }
1544 |    ],
1545 |    "source": [
1546 |     "confusion_matrix_ = confusion_matrix(y_test[\"Churn\"],\n",
1547 |     "                                     y_test[\"Churn_predicted\"])\n",
1548 |     "\n",
1549 |     "print(confusion_matrix_)"
1550 |    ]
1551 |   },
1552 |   {
1553 |    "cell_type": "code",
1554 |    "execution_count": 64,
1555 |    "metadata": {},
1556 |    "outputs": [
1557 |     {
1558 |      "name": "stdout",
1559 |      "output_type": "stream",
1560 |      "text": [
1561 |       "True Negatives =  832\n",
1562 |       "False Negatives =  121\n",
1563 |       "False Positives =  23\n",
1564 |       "True Positives =  24\n"
1565 |      ]
1566 |     }
1567 |    ],
1568 |    "source": [
1569 |     "TN = confusion_matrix_[0,0]\n",
1570 |     "print(\"True Negatives = \", TN)\n",
1571 |     "\n",
1572 |     "FN = confusion_matrix_[1,0]\n",
1573 |     "print(\"False Negatives = \", FN)\n",
1574 |     "\n",
1575 |     "FP = confusion_matrix_[0,1]\n",
1576 |     "print(\"False Positives = \", FP)\n",
1577 |     "\n",
1578 |     "TP = confusion_matrix_[1,1]\n",
1579 |     "print(\"True Positives = \", TP)"
1580 |    ]
1581 |   },
1582 |   {
1583 |    "cell_type": "markdown",
1584 |    "metadata": {},
1585 |    "source": [
1586 |     "#### Accuracy"
1587 |    ]
1588 |   },
1589 |   {
1590 |    "cell_type": "code",
1591 |    "execution_count": 65,
1592 |    "metadata": {},
1593 |    "outputs": [
1594 |     {
1595 |      "name": "stdout",
1596 |      "output_type": "stream",
1597 |      "text": [
1598 |       "0.856\n"
1599 |      ]
1600 |     }
1601 |    ],
1602 |    "source": [
1603 |     "accuracy = (TN+TP)/(TN+TP+FP+FN)\n",
1604 |     "print(accuracy)"
1605 |    ]
1606 |   },
1607 |   {
1608 |    "cell_type": "markdown",
1609 |    "metadata": {},
1610 |    "source": [
1611 |     "#### Precision, Recall/Sensitivity/True Positive Rate, F1 score\n",
1612 |     "\n",
1613 |     "**precision:**\n",
1614 |     "Precision can be seen as a measure of a classifier’s exactness. For each class, it is defined as the ratio of true positives to the sum of true and false positives. Said another way, “for all instances classified positive, what percent was correct?”\n",
1615 |     "\n",
1616 |     "**recall:**\n",
1617 |     "Recall is a measure of the classifier’s completeness; the ability of a classifier to correctly find all positive instances. For each class, it is defined as the ratio of true positives to the sum of true positives and false negatives. Said another way, “for all instances that were actually positive, what percent was classified correctly?”\n",
1618 |     "\n",
1619 |     "**f1 score:**\n",
1620 |     "The F1 score is a weighted harmonic mean of precision and recall such that the best score is 1.0 and the worst is 0.0. Generally speaking, F1 scores are lower than accuracy measures as they embed precision and recall into their computation. As a rule of thumb, the weighted average of F1 should be used to compare classifier models, not global accuracy.\n",
1621 |     "\n",
1622 |     "**support:**\n",
1623 |     "Support is the number of actual occurrences of the class in the specified dataset. Imbalanced support in the training data may indicate structural weaknesses in the reported scores of the classifier and could indicate the need for stratified sampling or rebalancing. Support doesn’t change between models but instead diagnoses the evaluation process."
1624 |    ]
1625 |   },
1626 |   {
1627 |    "cell_type": "code",
1628 |    "execution_count": 73,
1629 |    "metadata": {},
1630 |    "outputs": [
1631 |     {
1632 |      "name": "stdout",
1633 |      "output_type": "stream",
1634 |      "text": [
1635 |       "Calculated __________________________\n",
1636 |       "Precision =  0.51\n",
1637 |       "Recall =  0.17\n",
1638 |       "f1_score =  0.26\n",
1639 |       "\n",
1640 |       "Scikit-Learn ________________________\n",
1641 |       "Precision =  0.51\n",
1642 |       "Recall =  0.17\n",
1643 |       "f1_score =  0.25\n"
1644 |      ]
1645 |     }
1646 |    ],
1647 |    "source": [
1648 |     "print(\"Calculated __________________________\")\n",
1649 |     "precision = np.round(TP/(TP+FP), 2) #predicted class 1\n",
1650 |     "recall = np.round(TP/(TP+FN), 2) #actual class 1\n",
1651 |     "f1_score_ = np.round((2*precision*recall)/(precision+recall), 2)\n",
1652 |     "\n",
1653 |     "print(\"Precision = \", precision)\n",
1654 |     "print(\"Recall = \", recall)\n",
1655 |     "print(\"f1_score = \", f1_score_)\n",
1656 |     "\n",
1657 |     "print(\"\\nScikit-Learn ________________________\")\n",
1658 |     "precision_ = np.round(precision_score(y_test['Churn'], \n",
1659 |     "                                      y_test['Churn_predicted']), 2)\n",
1660 |     "recall_ = np.round(recall_score(y_test['Churn'], \n",
1661 |     "                                y_test['Churn_predicted']), 2)\n",
1662 |     "\n",
1663 |     "f1_score__ = np.round(f1_score(y_test['Churn'], \n",
1664 |     "                               y_test['Churn_predicted']), 2)\n",
1665 |     "\n",
1666 |     "print(\"Precision = \", precision_)\n",
1667 |     "print(\"Recall = \", recall_)\n",
1668 |     "print(\"f1_score = \", f1_score__)"
1669 |    ]
1670 |   },
1671 |   {
1672 |    "cell_type": "markdown",
1673 |    "metadata": {},
1674 |    "source": [
1675 |     "#### Classification report"
1676 |    ]
1677 |   },
1678 |   {
1679 |    "cell_type": "code",
1680 |    "execution_count": 69,
1681 |    "metadata": {},
1682 |    "outputs": [
1683 |     {
1684 |      "name": "stdout",
1685 |      "output_type": "stream",
1686 |      "text": [
1687 |       "              precision    recall  f1-score   support\n",
1688 |       "\n",
1689 |       "           0       0.87      0.97      0.92       855\n",
1690 |       "           1       0.51      0.17      0.25       145\n",
1691 |       "\n",
1692 |       "    accuracy                           0.86      1000\n",
1693 |       "   macro avg       0.69      0.57      0.59      1000\n",
1694 |       "weighted avg       0.82      0.86      0.82      1000\n",
1695 |       "\n"
1696 |      ]
1697 |     }
1698 |    ],
1699 |    "source": [
1700 |     "classification_report_ = classification_report(y_test[\"Churn\"],\n",
1701 |     "                                               y_test[\"Churn_predicted\"])\n",
1702 |     "print(classification_report_)"
1703 |    ]
1704 |   },
1705 |   {
1706 |    "cell_type": "code",
1707 |    "execution_count": null,
1708 |    "metadata": {},
1709 |    "outputs": [],
1710 |    "source": []
1711 |   }
1712 |  ],
1713 |  "metadata": {
1714 |   "kernelspec": {
1715 |    "display_name": "Python 3 (ipykernel)",
1716 |    "language": "python",
1717 |    "name": "python3"
1718 |   },
1719 |   "language_info": {
1720 |    "codemirror_mode": {
1721 |     "name": "ipython",
1722 |     "version": 3
1723 |    },
1724 |    "file_extension": ".py",
1725 |    "mimetype": "text/x-python",
1726 |    "name": "python",
1727 |    "nbconvert_exporter": "python",
1728 |    "pygments_lexer": "ipython3",
1729 |    "version": "3.8.5"
1730 |   }
1731 |  },
1732 |  "nbformat": 4,
1733 |  "nbformat_minor": 4
1734 | }
1735 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | I taught Machine Learning to 250+ students through an online platform. These are the course materials I built for my students (Mostly graduate-level students from the Non-CS background).
 2 | 
 3 | Video Lectures: https://www.youtube.com/playlist?list=PLGjf1T0akmhN1B4YHB5PT42XmJwr9BpS6
 4 | 
 5 | Quiz: https://kawsar34.medium.com/list/machine-learning-interview-quiz-057db29497a4
 6 | 
 7 | #### Lecture 1: Difference between Supervised Learning & Unsupervised Learning
 8 | - Defining Machine Learning: What can ML do?
 9 | - How does ML work?
10 | - Machine Learning terminologies
11 | - Supervised Learning, Unsupervised Learning, Deep Learning
12 | 
13 | #### Lecture 2: Supervised Learning: Linear Regression
14 | - Supervised Learning: Linear Regression
15 | - train data, test data
16 | - Understanding the equation of a straight line
17 | - feature coefficient (slope, gradient, m)
18 | - bias coefficient (y-intercept, c)
19 | - domain: x-axis, independent variable
20 | - range: y-axis, dependent variable
21 | - loss function, cost function, objective function, error function
22 | - bias-variance tradeoff, overfitting, underfitting
23 | - ordinary least square method
24 | - gradient descent method
25 | - residual, error, squared error, RMSE - Root Mean Squared Error
26 | 
27 | #### Lecture 3: Supervised Learning: Linear Regression and Regression accuracy metrics
28 | - Supervised Learning: Linear Regression
29 | - Accuracy metric in Regression problem
30 | - Mean Absolute Error (MAE)
31 | - Mean Absolute Percentage Error (MAPE)
32 | - Mean Squared Error (MSE)
33 | - Root Mean Squared Error (RMSE)
34 | - R-squared or coefficient of determination
35 | - Prediction result evaluation
36 | 
37 | #### Lecture 4: Supervised Learning - Classification: Logistic Regression
38 | - Supervised Learning - Classification: Logistic Regression
39 | 
40 | #### Lecture 5: Supervised Learning - Classification: Accuracy Metrics
41 | - Supervised Learning - Classification: Logistic Regression
42 | - Confusion Matrix
43 | - Accuracy, Precision, Recall/Sensitivity/True Positive Rate, F1 score, False Positive Rate
44 | - ROC: Receiver Operating Characteristics and AUC: Area Under the Curve
45 | - Classification report
46 | 
47 | #### Lecture 6: Supervised Learning - Decision Tree
48 | - Decision Tree Classification and Regression
49 | 
50 | #### Lecture 7: Supervised Learning - Decision Tree, Cross-validation and Grid Search
51 | - Decision Tree Classification
52 | - Cross-Validation
53 | - Grid Search
54 | 
55 | #### Lecture 8: Unsupervised Learning - K-means Clustering 
56 | - Unsupervised Learning
57 | - K-means Clustering
58 | - Elbow method
59 | 


--------------------------------------------------------------------------------
/processed.cleveland.data:
--------------------------------------------------------------------------------
  1 | 63.0,1.0,1.0,145.0,233.0,1.0,2.0,150.0,0.0,2.3,3.0,0.0,6.0,0
  2 | 67.0,1.0,4.0,160.0,286.0,0.0,2.0,108.0,1.0,1.5,2.0,3.0,3.0,2
  3 | 67.0,1.0,4.0,120.0,229.0,0.0,2.0,129.0,1.0,2.6,2.0,2.0,7.0,1
  4 | 37.0,1.0,3.0,130.0,250.0,0.0,0.0,187.0,0.0,3.5,3.0,0.0,3.0,0
  5 | 41.0,0.0,2.0,130.0,204.0,0.0,2.0,172.0,0.0,1.4,1.0,0.0,3.0,0
  6 | 56.0,1.0,2.0,120.0,236.0,0.0,0.0,178.0,0.0,0.8,1.0,0.0,3.0,0
  7 | 62.0,0.0,4.0,140.0,268.0,0.0,2.0,160.0,0.0,3.6,3.0,2.0,3.0,3
  8 | 57.0,0.0,4.0,120.0,354.0,0.0,0.0,163.0,1.0,0.6,1.0,0.0,3.0,0
  9 | 63.0,1.0,4.0,130.0,254.0,0.0,2.0,147.0,0.0,1.4,2.0,1.0,7.0,2
 10 | 53.0,1.0,4.0,140.0,203.0,1.0,2.0,155.0,1.0,3.1,3.0,0.0,7.0,1
 11 | 57.0,1.0,4.0,140.0,192.0,0.0,0.0,148.0,0.0,0.4,2.0,0.0,6.0,0
 12 | 56.0,0.0,2.0,140.0,294.0,0.0,2.0,153.0,0.0,1.3,2.0,0.0,3.0,0
 13 | 56.0,1.0,3.0,130.0,256.0,1.0,2.0,142.0,1.0,0.6,2.0,1.0,6.0,2
 14 | 44.0,1.0,2.0,120.0,263.0,0.0,0.0,173.0,0.0,0.0,1.0,0.0,7.0,0
 15 | 52.0,1.0,3.0,172.0,199.0,1.0,0.0,162.0,0.0,0.5,1.0,0.0,7.0,0
 16 | 57.0,1.0,3.0,150.0,168.0,0.0,0.0,174.0,0.0,1.6,1.0,0.0,3.0,0
 17 | 48.0,1.0,2.0,110.0,229.0,0.0,0.0,168.0,0.0,1.0,3.0,0.0,7.0,1
 18 | 54.0,1.0,4.0,140.0,239.0,0.0,0.0,160.0,0.0,1.2,1.0,0.0,3.0,0
 19 | 48.0,0.0,3.0,130.0,275.0,0.0,0.0,139.0,0.0,0.2,1.0,0.0,3.0,0
 20 | 49.0,1.0,2.0,130.0,266.0,0.0,0.0,171.0,0.0,0.6,1.0,0.0,3.0,0
 21 | 64.0,1.0,1.0,110.0,211.0,0.0,2.0,144.0,1.0,1.8,2.0,0.0,3.0,0
 22 | 58.0,0.0,1.0,150.0,283.0,1.0,2.0,162.0,0.0,1.0,1.0,0.0,3.0,0
 23 | 58.0,1.0,2.0,120.0,284.0,0.0,2.0,160.0,0.0,1.8,2.0,0.0,3.0,1
 24 | 58.0,1.0,3.0,132.0,224.0,0.0,2.0,173.0,0.0,3.2,1.0,2.0,7.0,3
 25 | 60.0,1.0,4.0,130.0,206.0,0.0,2.0,132.0,1.0,2.4,2.0,2.0,7.0,4
 26 | 50.0,0.0,3.0,120.0,219.0,0.0,0.0,158.0,0.0,1.6,2.0,0.0,3.0,0
 27 | 58.0,0.0,3.0,120.0,340.0,0.0,0.0,172.0,0.0,0.0,1.0,0.0,3.0,0
 28 | 66.0,0.0,1.0,150.0,226.0,0.0,0.0,114.0,0.0,2.6,3.0,0.0,3.0,0
 29 | 43.0,1.0,4.0,150.0,247.0,0.0,0.0,171.0,0.0,1.5,1.0,0.0,3.0,0
 30 | 40.0,1.0,4.0,110.0,167.0,0.0,2.0,114.0,1.0,2.0,2.0,0.0,7.0,3
 31 | 69.0,0.0,1.0,140.0,239.0,0.0,0.0,151.0,0.0,1.8,1.0,2.0,3.0,0
 32 | 60.0,1.0,4.0,117.0,230.0,1.0,0.0,160.0,1.0,1.4,1.0,2.0,7.0,2
 33 | 64.0,1.0,3.0,140.0,335.0,0.0,0.0,158.0,0.0,0.0,1.0,0.0,3.0,1
 34 | 59.0,1.0,4.0,135.0,234.0,0.0,0.0,161.0,0.0,0.5,2.0,0.0,7.0,0
 35 | 44.0,1.0,3.0,130.0,233.0,0.0,0.0,179.0,1.0,0.4,1.0,0.0,3.0,0
 36 | 42.0,1.0,4.0,140.0,226.0,0.0,0.0,178.0,0.0,0.0,1.0,0.0,3.0,0
 37 | 43.0,1.0,4.0,120.0,177.0,0.0,2.0,120.0,1.0,2.5,2.0,0.0,7.0,3
 38 | 57.0,1.0,4.0,150.0,276.0,0.0,2.0,112.0,1.0,0.6,2.0,1.0,6.0,1
 39 | 55.0,1.0,4.0,132.0,353.0,0.0,0.0,132.0,1.0,1.2,2.0,1.0,7.0,3
 40 | 61.0,1.0,3.0,150.0,243.0,1.0,0.0,137.0,1.0,1.0,2.0,0.0,3.0,0
 41 | 65.0,0.0,4.0,150.0,225.0,0.0,2.0,114.0,0.0,1.0,2.0,3.0,7.0,4
 42 | 40.0,1.0,1.0,140.0,199.0,0.0,0.0,178.0,1.0,1.4,1.0,0.0,7.0,0
 43 | 71.0,0.0,2.0,160.0,302.0,0.0,0.0,162.0,0.0,0.4,1.0,2.0,3.0,0
 44 | 59.0,1.0,3.0,150.0,212.0,1.0,0.0,157.0,0.0,1.6,1.0,0.0,3.0,0
 45 | 61.0,0.0,4.0,130.0,330.0,0.0,2.0,169.0,0.0,0.0,1.0,0.0,3.0,1
 46 | 58.0,1.0,3.0,112.0,230.0,0.0,2.0,165.0,0.0,2.5,2.0,1.0,7.0,4
 47 | 51.0,1.0,3.0,110.0,175.0,0.0,0.0,123.0,0.0,0.6,1.0,0.0,3.0,0
 48 | 50.0,1.0,4.0,150.0,243.0,0.0,2.0,128.0,0.0,2.6,2.0,0.0,7.0,4
 49 | 65.0,0.0,3.0,140.0,417.0,1.0,2.0,157.0,0.0,0.8,1.0,1.0,3.0,0
 50 | 53.0,1.0,3.0,130.0,197.0,1.0,2.0,152.0,0.0,1.2,3.0,0.0,3.0,0
 51 | 41.0,0.0,2.0,105.0,198.0,0.0,0.0,168.0,0.0,0.0,1.0,1.0,3.0,0
 52 | 65.0,1.0,4.0,120.0,177.0,0.0,0.0,140.0,0.0,0.4,1.0,0.0,7.0,0
 53 | 44.0,1.0,4.0,112.0,290.0,0.0,2.0,153.0,0.0,0.0,1.0,1.0,3.0,2
 54 | 44.0,1.0,2.0,130.0,219.0,0.0,2.0,188.0,0.0,0.0,1.0,0.0,3.0,0
 55 | 60.0,1.0,4.0,130.0,253.0,0.0,0.0,144.0,1.0,1.4,1.0,1.0,7.0,1
 56 | 54.0,1.0,4.0,124.0,266.0,0.0,2.0,109.0,1.0,2.2,2.0,1.0,7.0,1
 57 | 50.0,1.0,3.0,140.0,233.0,0.0,0.0,163.0,0.0,0.6,2.0,1.0,7.0,1
 58 | 41.0,1.0,4.0,110.0,172.0,0.0,2.0,158.0,0.0,0.0,1.0,0.0,7.0,1
 59 | 54.0,1.0,3.0,125.0,273.0,0.0,2.0,152.0,0.0,0.5,3.0,1.0,3.0,0
 60 | 51.0,1.0,1.0,125.0,213.0,0.0,2.0,125.0,1.0,1.4,1.0,1.0,3.0,0
 61 | 51.0,0.0,4.0,130.0,305.0,0.0,0.0,142.0,1.0,1.2,2.0,0.0,7.0,2
 62 | 46.0,0.0,3.0,142.0,177.0,0.0,2.0,160.0,1.0,1.4,3.0,0.0,3.0,0
 63 | 58.0,1.0,4.0,128.0,216.0,0.0,2.0,131.0,1.0,2.2,2.0,3.0,7.0,1
 64 | 54.0,0.0,3.0,135.0,304.0,1.0,0.0,170.0,0.0,0.0,1.0,0.0,3.0,0
 65 | 54.0,1.0,4.0,120.0,188.0,0.0,0.0,113.0,0.0,1.4,2.0,1.0,7.0,2
 66 | 60.0,1.0,4.0,145.0,282.0,0.0,2.0,142.0,1.0,2.8,2.0,2.0,7.0,2
 67 | 60.0,1.0,3.0,140.0,185.0,0.0,2.0,155.0,0.0,3.0,2.0,0.0,3.0,1
 68 | 54.0,1.0,3.0,150.0,232.0,0.0,2.0,165.0,0.0,1.6,1.0,0.0,7.0,0
 69 | 59.0,1.0,4.0,170.0,326.0,0.0,2.0,140.0,1.0,3.4,3.0,0.0,7.0,2
 70 | 46.0,1.0,3.0,150.0,231.0,0.0,0.0,147.0,0.0,3.6,2.0,0.0,3.0,1
 71 | 65.0,0.0,3.0,155.0,269.0,0.0,0.0,148.0,0.0,0.8,1.0,0.0,3.0,0
 72 | 67.0,1.0,4.0,125.0,254.0,1.0,0.0,163.0,0.0,0.2,2.0,2.0,7.0,3
 73 | 62.0,1.0,4.0,120.0,267.0,0.0,0.0,99.0,1.0,1.8,2.0,2.0,7.0,1
 74 | 65.0,1.0,4.0,110.0,248.0,0.0,2.0,158.0,0.0,0.6,1.0,2.0,6.0,1
 75 | 44.0,1.0,4.0,110.0,197.0,0.0,2.0,177.0,0.0,0.0,1.0,1.0,3.0,1
 76 | 65.0,0.0,3.0,160.0,360.0,0.0,2.0,151.0,0.0,0.8,1.0,0.0,3.0,0
 77 | 60.0,1.0,4.0,125.0,258.0,0.0,2.0,141.0,1.0,2.8,2.0,1.0,7.0,1
 78 | 51.0,0.0,3.0,140.0,308.0,0.0,2.0,142.0,0.0,1.5,1.0,1.0,3.0,0
 79 | 48.0,1.0,2.0,130.0,245.0,0.0,2.0,180.0,0.0,0.2,2.0,0.0,3.0,0
 80 | 58.0,1.0,4.0,150.0,270.0,0.0,2.0,111.0,1.0,0.8,1.0,0.0,7.0,3
 81 | 45.0,1.0,4.0,104.0,208.0,0.0,2.0,148.0,1.0,3.0,2.0,0.0,3.0,0
 82 | 53.0,0.0,4.0,130.0,264.0,0.0,2.0,143.0,0.0,0.4,2.0,0.0,3.0,0
 83 | 39.0,1.0,3.0,140.0,321.0,0.0,2.0,182.0,0.0,0.0,1.0,0.0,3.0,0
 84 | 68.0,1.0,3.0,180.0,274.0,1.0,2.0,150.0,1.0,1.6,2.0,0.0,7.0,3
 85 | 52.0,1.0,2.0,120.0,325.0,0.0,0.0,172.0,0.0,0.2,1.0,0.0,3.0,0
 86 | 44.0,1.0,3.0,140.0,235.0,0.0,2.0,180.0,0.0,0.0,1.0,0.0,3.0,0
 87 | 47.0,1.0,3.0,138.0,257.0,0.0,2.0,156.0,0.0,0.0,1.0,0.0,3.0,0
 88 | 53.0,0.0,3.0,128.0,216.0,0.0,2.0,115.0,0.0,0.0,1.0,0.0,?,0
 89 | 53.0,0.0,4.0,138.0,234.0,0.0,2.0,160.0,0.0,0.0,1.0,0.0,3.0,0
 90 | 51.0,0.0,3.0,130.0,256.0,0.0,2.0,149.0,0.0,0.5,1.0,0.0,3.0,0
 91 | 66.0,1.0,4.0,120.0,302.0,0.0,2.0,151.0,0.0,0.4,2.0,0.0,3.0,0
 92 | 62.0,0.0,4.0,160.0,164.0,0.0,2.0,145.0,0.0,6.2,3.0,3.0,7.0,3
 93 | 62.0,1.0,3.0,130.0,231.0,0.0,0.0,146.0,0.0,1.8,2.0,3.0,7.0,0
 94 | 44.0,0.0,3.0,108.0,141.0,0.0,0.0,175.0,0.0,0.6,2.0,0.0,3.0,0
 95 | 63.0,0.0,3.0,135.0,252.0,0.0,2.0,172.0,0.0,0.0,1.0,0.0,3.0,0
 96 | 52.0,1.0,4.0,128.0,255.0,0.0,0.0,161.0,1.0,0.0,1.0,1.0,7.0,1
 97 | 59.0,1.0,4.0,110.0,239.0,0.0,2.0,142.0,1.0,1.2,2.0,1.0,7.0,2
 98 | 60.0,0.0,4.0,150.0,258.0,0.0,2.0,157.0,0.0,2.6,2.0,2.0,7.0,3
 99 | 52.0,1.0,2.0,134.0,201.0,0.0,0.0,158.0,0.0,0.8,1.0,1.0,3.0,0
100 | 48.0,1.0,4.0,122.0,222.0,0.0,2.0,186.0,0.0,0.0,1.0,0.0,3.0,0
101 | 45.0,1.0,4.0,115.0,260.0,0.0,2.0,185.0,0.0,0.0,1.0,0.0,3.0,0
102 | 34.0,1.0,1.0,118.0,182.0,0.0,2.0,174.0,0.0,0.0,1.0,0.0,3.0,0
103 | 57.0,0.0,4.0,128.0,303.0,0.0,2.0,159.0,0.0,0.0,1.0,1.0,3.0,0
104 | 71.0,0.0,3.0,110.0,265.0,1.0,2.0,130.0,0.0,0.0,1.0,1.0,3.0,0
105 | 49.0,1.0,3.0,120.0,188.0,0.0,0.0,139.0,0.0,2.0,2.0,3.0,7.0,3
106 | 54.0,1.0,2.0,108.0,309.0,0.0,0.0,156.0,0.0,0.0,1.0,0.0,7.0,0
107 | 59.0,1.0,4.0,140.0,177.0,0.0,0.0,162.0,1.0,0.0,1.0,1.0,7.0,2
108 | 57.0,1.0,3.0,128.0,229.0,0.0,2.0,150.0,0.0,0.4,2.0,1.0,7.0,1
109 | 61.0,1.0,4.0,120.0,260.0,0.0,0.0,140.0,1.0,3.6,2.0,1.0,7.0,2
110 | 39.0,1.0,4.0,118.0,219.0,0.0,0.0,140.0,0.0,1.2,2.0,0.0,7.0,3
111 | 61.0,0.0,4.0,145.0,307.0,0.0,2.0,146.0,1.0,1.0,2.0,0.0,7.0,1
112 | 56.0,1.0,4.0,125.0,249.0,1.0,2.0,144.0,1.0,1.2,2.0,1.0,3.0,1
113 | 52.0,1.0,1.0,118.0,186.0,0.0,2.0,190.0,0.0,0.0,2.0,0.0,6.0,0
114 | 43.0,0.0,4.0,132.0,341.0,1.0,2.0,136.0,1.0,3.0,2.0,0.0,7.0,2
115 | 62.0,0.0,3.0,130.0,263.0,0.0,0.0,97.0,0.0,1.2,2.0,1.0,7.0,2
116 | 41.0,1.0,2.0,135.0,203.0,0.0,0.0,132.0,0.0,0.0,2.0,0.0,6.0,0
117 | 58.0,1.0,3.0,140.0,211.0,1.0,2.0,165.0,0.0,0.0,1.0,0.0,3.0,0
118 | 35.0,0.0,4.0,138.0,183.0,0.0,0.0,182.0,0.0,1.4,1.0,0.0,3.0,0
119 | 63.0,1.0,4.0,130.0,330.0,1.0,2.0,132.0,1.0,1.8,1.0,3.0,7.0,3
120 | 65.0,1.0,4.0,135.0,254.0,0.0,2.0,127.0,0.0,2.8,2.0,1.0,7.0,2
121 | 48.0,1.0,4.0,130.0,256.0,1.0,2.0,150.0,1.0,0.0,1.0,2.0,7.0,3
122 | 63.0,0.0,4.0,150.0,407.0,0.0,2.0,154.0,0.0,4.0,2.0,3.0,7.0,4
123 | 51.0,1.0,3.0,100.0,222.0,0.0,0.0,143.0,1.0,1.2,2.0,0.0,3.0,0
124 | 55.0,1.0,4.0,140.0,217.0,0.0,0.0,111.0,1.0,5.6,3.0,0.0,7.0,3
125 | 65.0,1.0,1.0,138.0,282.0,1.0,2.0,174.0,0.0,1.4,2.0,1.0,3.0,1
126 | 45.0,0.0,2.0,130.0,234.0,0.0,2.0,175.0,0.0,0.6,2.0,0.0,3.0,0
127 | 56.0,0.0,4.0,200.0,288.0,1.0,2.0,133.0,1.0,4.0,3.0,2.0,7.0,3
128 | 54.0,1.0,4.0,110.0,239.0,0.0,0.0,126.0,1.0,2.8,2.0,1.0,7.0,3
129 | 44.0,1.0,2.0,120.0,220.0,0.0,0.0,170.0,0.0,0.0,1.0,0.0,3.0,0
130 | 62.0,0.0,4.0,124.0,209.0,0.0,0.0,163.0,0.0,0.0,1.0,0.0,3.0,0
131 | 54.0,1.0,3.0,120.0,258.0,0.0,2.0,147.0,0.0,0.4,2.0,0.0,7.0,0
132 | 51.0,1.0,3.0,94.0,227.0,0.0,0.0,154.0,1.0,0.0,1.0,1.0,7.0,0
133 | 29.0,1.0,2.0,130.0,204.0,0.0,2.0,202.0,0.0,0.0,1.0,0.0,3.0,0
134 | 51.0,1.0,4.0,140.0,261.0,0.0,2.0,186.0,1.0,0.0,1.0,0.0,3.0,0
135 | 43.0,0.0,3.0,122.0,213.0,0.0,0.0,165.0,0.0,0.2,2.0,0.0,3.0,0
136 | 55.0,0.0,2.0,135.0,250.0,0.0,2.0,161.0,0.0,1.4,2.0,0.0,3.0,0
137 | 70.0,1.0,4.0,145.0,174.0,0.0,0.0,125.0,1.0,2.6,3.0,0.0,7.0,4
138 | 62.0,1.0,2.0,120.0,281.0,0.0,2.0,103.0,0.0,1.4,2.0,1.0,7.0,3
139 | 35.0,1.0,4.0,120.0,198.0,0.0,0.0,130.0,1.0,1.6,2.0,0.0,7.0,1
140 | 51.0,1.0,3.0,125.0,245.0,1.0,2.0,166.0,0.0,2.4,2.0,0.0,3.0,0
141 | 59.0,1.0,2.0,140.0,221.0,0.0,0.0,164.0,1.0,0.0,1.0,0.0,3.0,0
142 | 59.0,1.0,1.0,170.0,288.0,0.0,2.0,159.0,0.0,0.2,2.0,0.0,7.0,1
143 | 52.0,1.0,2.0,128.0,205.0,1.0,0.0,184.0,0.0,0.0,1.0,0.0,3.0,0
144 | 64.0,1.0,3.0,125.0,309.0,0.0,0.0,131.0,1.0,1.8,2.0,0.0,7.0,1
145 | 58.0,1.0,3.0,105.0,240.0,0.0,2.0,154.0,1.0,0.6,2.0,0.0,7.0,0
146 | 47.0,1.0,3.0,108.0,243.0,0.0,0.0,152.0,0.0,0.0,1.0,0.0,3.0,1
147 | 57.0,1.0,4.0,165.0,289.0,1.0,2.0,124.0,0.0,1.0,2.0,3.0,7.0,4
148 | 41.0,1.0,3.0,112.0,250.0,0.0,0.0,179.0,0.0,0.0,1.0,0.0,3.0,0
149 | 45.0,1.0,2.0,128.0,308.0,0.0,2.0,170.0,0.0,0.0,1.0,0.0,3.0,0
150 | 60.0,0.0,3.0,102.0,318.0,0.0,0.0,160.0,0.0,0.0,1.0,1.0,3.0,0
151 | 52.0,1.0,1.0,152.0,298.0,1.0,0.0,178.0,0.0,1.2,2.0,0.0,7.0,0
152 | 42.0,0.0,4.0,102.0,265.0,0.0,2.0,122.0,0.0,0.6,2.0,0.0,3.0,0
153 | 67.0,0.0,3.0,115.0,564.0,0.0,2.0,160.0,0.0,1.6,2.0,0.0,7.0,0
154 | 55.0,1.0,4.0,160.0,289.0,0.0,2.0,145.0,1.0,0.8,2.0,1.0,7.0,4
155 | 64.0,1.0,4.0,120.0,246.0,0.0,2.0,96.0,1.0,2.2,3.0,1.0,3.0,3
156 | 70.0,1.0,4.0,130.0,322.0,0.0,2.0,109.0,0.0,2.4,2.0,3.0,3.0,1
157 | 51.0,1.0,4.0,140.0,299.0,0.0,0.0,173.0,1.0,1.6,1.0,0.0,7.0,1
158 | 58.0,1.0,4.0,125.0,300.0,0.0,2.0,171.0,0.0,0.0,1.0,2.0,7.0,1
159 | 60.0,1.0,4.0,140.0,293.0,0.0,2.0,170.0,0.0,1.2,2.0,2.0,7.0,2
160 | 68.0,1.0,3.0,118.0,277.0,0.0,0.0,151.0,0.0,1.0,1.0,1.0,7.0,0
161 | 46.0,1.0,2.0,101.0,197.0,1.0,0.0,156.0,0.0,0.0,1.0,0.0,7.0,0
162 | 77.0,1.0,4.0,125.0,304.0,0.0,2.0,162.0,1.0,0.0,1.0,3.0,3.0,4
163 | 54.0,0.0,3.0,110.0,214.0,0.0,0.0,158.0,0.0,1.6,2.0,0.0,3.0,0
164 | 58.0,0.0,4.0,100.0,248.0,0.0,2.0,122.0,0.0,1.0,2.0,0.0,3.0,0
165 | 48.0,1.0,3.0,124.0,255.0,1.0,0.0,175.0,0.0,0.0,1.0,2.0,3.0,0
166 | 57.0,1.0,4.0,132.0,207.0,0.0,0.0,168.0,1.0,0.0,1.0,0.0,7.0,0
167 | 52.0,1.0,3.0,138.0,223.0,0.0,0.0,169.0,0.0,0.0,1.0,?,3.0,0
168 | 54.0,0.0,2.0,132.0,288.0,1.0,2.0,159.0,1.0,0.0,1.0,1.0,3.0,0
169 | 35.0,1.0,4.0,126.0,282.0,0.0,2.0,156.0,1.0,0.0,1.0,0.0,7.0,1
170 | 45.0,0.0,2.0,112.0,160.0,0.0,0.0,138.0,0.0,0.0,2.0,0.0,3.0,0
171 | 70.0,1.0,3.0,160.0,269.0,0.0,0.0,112.0,1.0,2.9,2.0,1.0,7.0,3
172 | 53.0,1.0,4.0,142.0,226.0,0.0,2.0,111.0,1.0,0.0,1.0,0.0,7.0,0
173 | 59.0,0.0,4.0,174.0,249.0,0.0,0.0,143.0,1.0,0.0,2.0,0.0,3.0,1
174 | 62.0,0.0,4.0,140.0,394.0,0.0,2.0,157.0,0.0,1.2,2.0,0.0,3.0,0
175 | 64.0,1.0,4.0,145.0,212.0,0.0,2.0,132.0,0.0,2.0,2.0,2.0,6.0,4
176 | 57.0,1.0,4.0,152.0,274.0,0.0,0.0,88.0,1.0,1.2,2.0,1.0,7.0,1
177 | 52.0,1.0,4.0,108.0,233.0,1.0,0.0,147.0,0.0,0.1,1.0,3.0,7.0,0
178 | 56.0,1.0,4.0,132.0,184.0,0.0,2.0,105.0,1.0,2.1,2.0,1.0,6.0,1
179 | 43.0,1.0,3.0,130.0,315.0,0.0,0.0,162.0,0.0,1.9,1.0,1.0,3.0,0
180 | 53.0,1.0,3.0,130.0,246.0,1.0,2.0,173.0,0.0,0.0,1.0,3.0,3.0,0
181 | 48.0,1.0,4.0,124.0,274.0,0.0,2.0,166.0,0.0,0.5,2.0,0.0,7.0,3
182 | 56.0,0.0,4.0,134.0,409.0,0.0,2.0,150.0,1.0,1.9,2.0,2.0,7.0,2
183 | 42.0,1.0,1.0,148.0,244.0,0.0,2.0,178.0,0.0,0.8,1.0,2.0,3.0,0
184 | 59.0,1.0,1.0,178.0,270.0,0.0,2.0,145.0,0.0,4.2,3.0,0.0,7.0,0
185 | 60.0,0.0,4.0,158.0,305.0,0.0,2.0,161.0,0.0,0.0,1.0,0.0,3.0,1
186 | 63.0,0.0,2.0,140.0,195.0,0.0,0.0,179.0,0.0,0.0,1.0,2.0,3.0,0
187 | 42.0,1.0,3.0,120.0,240.0,1.0,0.0,194.0,0.0,0.8,3.0,0.0,7.0,0
188 | 66.0,1.0,2.0,160.0,246.0,0.0,0.0,120.0,1.0,0.0,2.0,3.0,6.0,2
189 | 54.0,1.0,2.0,192.0,283.0,0.0,2.0,195.0,0.0,0.0,1.0,1.0,7.0,1
190 | 69.0,1.0,3.0,140.0,254.0,0.0,2.0,146.0,0.0,2.0,2.0,3.0,7.0,2
191 | 50.0,1.0,3.0,129.0,196.0,0.0,0.0,163.0,0.0,0.0,1.0,0.0,3.0,0
192 | 51.0,1.0,4.0,140.0,298.0,0.0,0.0,122.0,1.0,4.2,2.0,3.0,7.0,3
193 | 43.0,1.0,4.0,132.0,247.0,1.0,2.0,143.0,1.0,0.1,2.0,?,7.0,1
194 | 62.0,0.0,4.0,138.0,294.0,1.0,0.0,106.0,0.0,1.9,2.0,3.0,3.0,2
195 | 68.0,0.0,3.0,120.0,211.0,0.0,2.0,115.0,0.0,1.5,2.0,0.0,3.0,0
196 | 67.0,1.0,4.0,100.0,299.0,0.0,2.0,125.0,1.0,0.9,2.0,2.0,3.0,3
197 | 69.0,1.0,1.0,160.0,234.0,1.0,2.0,131.0,0.0,0.1,2.0,1.0,3.0,0
198 | 45.0,0.0,4.0,138.0,236.0,0.0,2.0,152.0,1.0,0.2,2.0,0.0,3.0,0
199 | 50.0,0.0,2.0,120.0,244.0,0.0,0.0,162.0,0.0,1.1,1.0,0.0,3.0,0
200 | 59.0,1.0,1.0,160.0,273.0,0.0,2.0,125.0,0.0,0.0,1.0,0.0,3.0,1
201 | 50.0,0.0,4.0,110.0,254.0,0.0,2.0,159.0,0.0,0.0,1.0,0.0,3.0,0
202 | 64.0,0.0,4.0,180.0,325.0,0.0,0.0,154.0,1.0,0.0,1.0,0.0,3.0,0
203 | 57.0,1.0,3.0,150.0,126.0,1.0,0.0,173.0,0.0,0.2,1.0,1.0,7.0,0
204 | 64.0,0.0,3.0,140.0,313.0,0.0,0.0,133.0,0.0,0.2,1.0,0.0,7.0,0
205 | 43.0,1.0,4.0,110.0,211.0,0.0,0.0,161.0,0.0,0.0,1.0,0.0,7.0,0
206 | 45.0,1.0,4.0,142.0,309.0,0.0,2.0,147.0,1.0,0.0,2.0,3.0,7.0,3
207 | 58.0,1.0,4.0,128.0,259.0,0.0,2.0,130.0,1.0,3.0,2.0,2.0,7.0,3
208 | 50.0,1.0,4.0,144.0,200.0,0.0,2.0,126.0,1.0,0.9,2.0,0.0,7.0,3
209 | 55.0,1.0,2.0,130.0,262.0,0.0,0.0,155.0,0.0,0.0,1.0,0.0,3.0,0
210 | 62.0,0.0,4.0,150.0,244.0,0.0,0.0,154.0,1.0,1.4,2.0,0.0,3.0,1
211 | 37.0,0.0,3.0,120.0,215.0,0.0,0.0,170.0,0.0,0.0,1.0,0.0,3.0,0
212 | 38.0,1.0,1.0,120.0,231.0,0.0,0.0,182.0,1.0,3.8,2.0,0.0,7.0,4
213 | 41.0,1.0,3.0,130.0,214.0,0.0,2.0,168.0,0.0,2.0,2.0,0.0,3.0,0
214 | 66.0,0.0,4.0,178.0,228.0,1.0,0.0,165.0,1.0,1.0,2.0,2.0,7.0,3
215 | 52.0,1.0,4.0,112.0,230.0,0.0,0.0,160.0,0.0,0.0,1.0,1.0,3.0,1
216 | 56.0,1.0,1.0,120.0,193.0,0.0,2.0,162.0,0.0,1.9,2.0,0.0,7.0,0
217 | 46.0,0.0,2.0,105.0,204.0,0.0,0.0,172.0,0.0,0.0,1.0,0.0,3.0,0
218 | 46.0,0.0,4.0,138.0,243.0,0.0,2.0,152.0,1.0,0.0,2.0,0.0,3.0,0
219 | 64.0,0.0,4.0,130.0,303.0,0.0,0.0,122.0,0.0,2.0,2.0,2.0,3.0,0
220 | 59.0,1.0,4.0,138.0,271.0,0.0,2.0,182.0,0.0,0.0,1.0,0.0,3.0,0
221 | 41.0,0.0,3.0,112.0,268.0,0.0,2.0,172.0,1.0,0.0,1.0,0.0,3.0,0
222 | 54.0,0.0,3.0,108.0,267.0,0.0,2.0,167.0,0.0,0.0,1.0,0.0,3.0,0
223 | 39.0,0.0,3.0,94.0,199.0,0.0,0.0,179.0,0.0,0.0,1.0,0.0,3.0,0
224 | 53.0,1.0,4.0,123.0,282.0,0.0,0.0,95.0,1.0,2.0,2.0,2.0,7.0,3
225 | 63.0,0.0,4.0,108.0,269.0,0.0,0.0,169.0,1.0,1.8,2.0,2.0,3.0,1
226 | 34.0,0.0,2.0,118.0,210.0,0.0,0.0,192.0,0.0,0.7,1.0,0.0,3.0,0
227 | 47.0,1.0,4.0,112.0,204.0,0.0,0.0,143.0,0.0,0.1,1.0,0.0,3.0,0
228 | 67.0,0.0,3.0,152.0,277.0,0.0,0.0,172.0,0.0,0.0,1.0,1.0,3.0,0
229 | 54.0,1.0,4.0,110.0,206.0,0.0,2.0,108.0,1.0,0.0,2.0,1.0,3.0,3
230 | 66.0,1.0,4.0,112.0,212.0,0.0,2.0,132.0,1.0,0.1,1.0,1.0,3.0,2
231 | 52.0,0.0,3.0,136.0,196.0,0.0,2.0,169.0,0.0,0.1,2.0,0.0,3.0,0
232 | 55.0,0.0,4.0,180.0,327.0,0.0,1.0,117.0,1.0,3.4,2.0,0.0,3.0,2
233 | 49.0,1.0,3.0,118.0,149.0,0.0,2.0,126.0,0.0,0.8,1.0,3.0,3.0,1
234 | 74.0,0.0,2.0,120.0,269.0,0.0,2.0,121.0,1.0,0.2,1.0,1.0,3.0,0
235 | 54.0,0.0,3.0,160.0,201.0,0.0,0.0,163.0,0.0,0.0,1.0,1.0,3.0,0
236 | 54.0,1.0,4.0,122.0,286.0,0.0,2.0,116.0,1.0,3.2,2.0,2.0,3.0,3
237 | 56.0,1.0,4.0,130.0,283.0,1.0,2.0,103.0,1.0,1.6,3.0,0.0,7.0,2
238 | 46.0,1.0,4.0,120.0,249.0,0.0,2.0,144.0,0.0,0.8,1.0,0.0,7.0,1
239 | 49.0,0.0,2.0,134.0,271.0,0.0,0.0,162.0,0.0,0.0,2.0,0.0,3.0,0
240 | 42.0,1.0,2.0,120.0,295.0,0.0,0.0,162.0,0.0,0.0,1.0,0.0,3.0,0
241 | 41.0,1.0,2.0,110.0,235.0,0.0,0.0,153.0,0.0,0.0,1.0,0.0,3.0,0
242 | 41.0,0.0,2.0,126.0,306.0,0.0,0.0,163.0,0.0,0.0,1.0,0.0,3.0,0
243 | 49.0,0.0,4.0,130.0,269.0,0.0,0.0,163.0,0.0,0.0,1.0,0.0,3.0,0
244 | 61.0,1.0,1.0,134.0,234.0,0.0,0.0,145.0,0.0,2.6,2.0,2.0,3.0,2
245 | 60.0,0.0,3.0,120.0,178.0,1.0,0.0,96.0,0.0,0.0,1.0,0.0,3.0,0
246 | 67.0,1.0,4.0,120.0,237.0,0.0,0.0,71.0,0.0,1.0,2.0,0.0,3.0,2
247 | 58.0,1.0,4.0,100.0,234.0,0.0,0.0,156.0,0.0,0.1,1.0,1.0,7.0,2
248 | 47.0,1.0,4.0,110.0,275.0,0.0,2.0,118.0,1.0,1.0,2.0,1.0,3.0,1
249 | 52.0,1.0,4.0,125.0,212.0,0.0,0.0,168.0,0.0,1.0,1.0,2.0,7.0,3
250 | 62.0,1.0,2.0,128.0,208.0,1.0,2.0,140.0,0.0,0.0,1.0,0.0,3.0,0
251 | 57.0,1.0,4.0,110.0,201.0,0.0,0.0,126.0,1.0,1.5,2.0,0.0,6.0,0
252 | 58.0,1.0,4.0,146.0,218.0,0.0,0.0,105.0,0.0,2.0,2.0,1.0,7.0,1
253 | 64.0,1.0,4.0,128.0,263.0,0.0,0.0,105.0,1.0,0.2,2.0,1.0,7.0,0
254 | 51.0,0.0,3.0,120.0,295.0,0.0,2.0,157.0,0.0,0.6,1.0,0.0,3.0,0
255 | 43.0,1.0,4.0,115.0,303.0,0.0,0.0,181.0,0.0,1.2,2.0,0.0,3.0,0
256 | 42.0,0.0,3.0,120.0,209.0,0.0,0.0,173.0,0.0,0.0,2.0,0.0,3.0,0
257 | 67.0,0.0,4.0,106.0,223.0,0.0,0.0,142.0,0.0,0.3,1.0,2.0,3.0,0
258 | 76.0,0.0,3.0,140.0,197.0,0.0,1.0,116.0,0.0,1.1,2.0,0.0,3.0,0
259 | 70.0,1.0,2.0,156.0,245.0,0.0,2.0,143.0,0.0,0.0,1.0,0.0,3.0,0
260 | 57.0,1.0,2.0,124.0,261.0,0.0,0.0,141.0,0.0,0.3,1.0,0.0,7.0,1
261 | 44.0,0.0,3.0,118.0,242.0,0.0,0.0,149.0,0.0,0.3,2.0,1.0,3.0,0
262 | 58.0,0.0,2.0,136.0,319.0,1.0,2.0,152.0,0.0,0.0,1.0,2.0,3.0,3
263 | 60.0,0.0,1.0,150.0,240.0,0.0,0.0,171.0,0.0,0.9,1.0,0.0,3.0,0
264 | 44.0,1.0,3.0,120.0,226.0,0.0,0.0,169.0,0.0,0.0,1.0,0.0,3.0,0
265 | 61.0,1.0,4.0,138.0,166.0,0.0,2.0,125.0,1.0,3.6,2.0,1.0,3.0,4
266 | 42.0,1.0,4.0,136.0,315.0,0.0,0.0,125.0,1.0,1.8,2.0,0.0,6.0,2
267 | 52.0,1.0,4.0,128.0,204.0,1.0,0.0,156.0,1.0,1.0,2.0,0.0,?,2
268 | 59.0,1.0,3.0,126.0,218.0,1.0,0.0,134.0,0.0,2.2,2.0,1.0,6.0,2
269 | 40.0,1.0,4.0,152.0,223.0,0.0,0.0,181.0,0.0,0.0,1.0,0.0,7.0,1
270 | 42.0,1.0,3.0,130.0,180.0,0.0,0.0,150.0,0.0,0.0,1.0,0.0,3.0,0
271 | 61.0,1.0,4.0,140.0,207.0,0.0,2.0,138.0,1.0,1.9,1.0,1.0,7.0,1
272 | 66.0,1.0,4.0,160.0,228.0,0.0,2.0,138.0,0.0,2.3,1.0,0.0,6.0,0
273 | 46.0,1.0,4.0,140.0,311.0,0.0,0.0,120.0,1.0,1.8,2.0,2.0,7.0,2
274 | 71.0,0.0,4.0,112.0,149.0,0.0,0.0,125.0,0.0,1.6,2.0,0.0,3.0,0
275 | 59.0,1.0,1.0,134.0,204.0,0.0,0.0,162.0,0.0,0.8,1.0,2.0,3.0,1
276 | 64.0,1.0,1.0,170.0,227.0,0.0,2.0,155.0,0.0,0.6,2.0,0.0,7.0,0
277 | 66.0,0.0,3.0,146.0,278.0,0.0,2.0,152.0,0.0,0.0,2.0,1.0,3.0,0
278 | 39.0,0.0,3.0,138.0,220.0,0.0,0.0,152.0,0.0,0.0,2.0,0.0,3.0,0
279 | 57.0,1.0,2.0,154.0,232.0,0.0,2.0,164.0,0.0,0.0,1.0,1.0,3.0,1
280 | 58.0,0.0,4.0,130.0,197.0,0.0,0.0,131.0,0.0,0.6,2.0,0.0,3.0,0
281 | 57.0,1.0,4.0,110.0,335.0,0.0,0.0,143.0,1.0,3.0,2.0,1.0,7.0,2
282 | 47.0,1.0,3.0,130.0,253.0,0.0,0.0,179.0,0.0,0.0,1.0,0.0,3.0,0
283 | 55.0,0.0,4.0,128.0,205.0,0.0,1.0,130.0,1.0,2.0,2.0,1.0,7.0,3
284 | 35.0,1.0,2.0,122.0,192.0,0.0,0.0,174.0,0.0,0.0,1.0,0.0,3.0,0
285 | 61.0,1.0,4.0,148.0,203.0,0.0,0.0,161.0,0.0,0.0,1.0,1.0,7.0,2
286 | 58.0,1.0,4.0,114.0,318.0,0.0,1.0,140.0,0.0,4.4,3.0,3.0,6.0,4
287 | 58.0,0.0,4.0,170.0,225.0,1.0,2.0,146.0,1.0,2.8,2.0,2.0,6.0,2
288 | 58.0,1.0,2.0,125.0,220.0,0.0,0.0,144.0,0.0,0.4,2.0,?,7.0,0
289 | 56.0,1.0,2.0,130.0,221.0,0.0,2.0,163.0,0.0,0.0,1.0,0.0,7.0,0
290 | 56.0,1.0,2.0,120.0,240.0,0.0,0.0,169.0,0.0,0.0,3.0,0.0,3.0,0
291 | 67.0,1.0,3.0,152.0,212.0,0.0,2.0,150.0,0.0,0.8,2.0,0.0,7.0,1
292 | 55.0,0.0,2.0,132.0,342.0,0.0,0.0,166.0,0.0,1.2,1.0,0.0,3.0,0
293 | 44.0,1.0,4.0,120.0,169.0,0.0,0.0,144.0,1.0,2.8,3.0,0.0,6.0,2
294 | 63.0,1.0,4.0,140.0,187.0,0.0,2.0,144.0,1.0,4.0,1.0,2.0,7.0,2
295 | 63.0,0.0,4.0,124.0,197.0,0.0,0.0,136.0,1.0,0.0,2.0,0.0,3.0,1
296 | 41.0,1.0,2.0,120.0,157.0,0.0,0.0,182.0,0.0,0.0,1.0,0.0,3.0,0
297 | 59.0,1.0,4.0,164.0,176.0,1.0,2.0,90.0,0.0,1.0,2.0,2.0,6.0,3
298 | 57.0,0.0,4.0,140.0,241.0,0.0,0.0,123.0,1.0,0.2,2.0,0.0,7.0,1
299 | 45.0,1.0,1.0,110.0,264.0,0.0,0.0,132.0,0.0,1.2,2.0,0.0,7.0,1
300 | 68.0,1.0,4.0,144.0,193.0,1.0,0.0,141.0,0.0,3.4,2.0,2.0,7.0,2
301 | 57.0,1.0,4.0,130.0,131.0,0.0,0.0,115.0,1.0,1.2,2.0,1.0,7.0,3
302 | 57.0,0.0,2.0,130.0,236.0,0.0,2.0,174.0,0.0,0.0,2.0,1.0,3.0,1
303 | 38.0,1.0,3.0,138.0,175.0,0.0,0.0,173.0,0.0,0.0,1.0,?,3.0,0
304 | 


--------------------------------------------------------------------------------