├── README.md ├── sagemaker-churn-lab.ipynb └── sagemaker-churn-lab.yaml /README.md: -------------------------------------------------------------------------------- 1 | 2 | # **Predicting Customer Churn with Amazon SageMaker** 3 | 4 | This lab is provided as part of the [AWS Innovate Online Conference](https://aws.amazon.com/events/aws-innovate/machine-learning/). It has been adapted from an [AWS blog post](https://aws.amazon.com/blogs/ai/predicting-customer-churn-with-amazon-machine-learning/). 5 | 6 | ℹ️ You will run this lab in your own AWS account. Please follow directions at the end of the lab to remove resources to minimize costs. 7 | 8 | ## Background 9 | 10 | Losing customers is costly for any business. Identifying unhappy customers early on gives you a chance to offer them incentives to stay. This notebook describes using machine learning (ML) for the automated identification of unhappy customers, also known as customer churn prediction. ML models rarely give perfect predictions though, so this notebook is also about how to incorporate the relative costs of prediction mistakes when determining the financial outcome of using ML. 11 | 12 | We use an example of churn that is familiar to all of us–leaving a mobile phone operator. It seems like you can always find fault with my provider du jour! And if your provider knows that I’m thinking of leaving, it can offer timely incentives–you can always use a phone upgrade or perhaps have a new feature activated–and you might just stick around. Incentives are often much more cost effective than losing and reacquiring a customer. 13 | 14 | ## Setup: Launch AWS CloudFormation Stack 15 | 16 | You will use AWS CloudFormation to deploy Amazon SageMaker in your AWS account. It will be deployed in the **Sydney region**. Please ensure you follow directions at the end of the lab to delete the CloudFormation stack to remove resources. 17 | 18 | **1.** Login to your AWS account. 19 | 20 | **2.** Right-click this link and open in a new browser tab: [Launch Stack into ap-southeast-2 with CloudFormation](https://console.aws.amazon.com/cloudformation/home?region=ap-southeast-2#/stacks/new?stackName=SageMakerChurn&templateURL=https://s3-ap-southeast-2.amazonaws.com/share.jr-class.net/sagemaker-churn-lab.yaml) 21 | 22 | If that fails due to a file not found, then instead: 23 | - Right-click this link and download it to your computer: [sagemaker-churn-lab.yaml](https://raw.githubusercontent.com/aws-john/sagemaker-churn/master/sagemaker-churn-lab.yaml) 24 | - In the CloudFormation management console, **Create stack** and upload the YAML file you just downloaded 25 | 26 | **3.** Click **Next** three times. 27 | 28 | **4.** At the bottom of the page, select "I acknowledge that AWS CloudFormation might create IAM resources". 29 | 30 | **5.** Click **Create stack**. 31 | 32 | This will launch an Amazon SageMaker notebook instance. It will take approximately 3 minutes. 33 | 34 | ## Access SageMaker 35 | 36 | **6.** In the **Services** menu at the top of the page, select **Amazon SageMaker**. 37 | 38 | **7.** In the left navigation pane, click **Notebook instances**. 39 | 40 | A SageMaker notebook instance will be displayed. 41 | 42 | **8.** Click **Open Jupyter** in the right-hand column. 43 | 44 | This will open a new browser tab to the Jupyter interface. 45 | 46 | **9.** Click **lab_notebook.ipynb**. 47 | 48 | This will open the notebook that will be used for the lab. 49 | 50 | **Please follow the instructions in the notebook.** At the conclusion of the lab, please return the page you are currently reading to delete the CloudFormation template. 51 | 52 | ## Important: Clean-up 53 | 54 | When you have completed the lab, you must delete the CloudFormation stack as follows: 55 | 56 | **10.** Return to the AWS console and use the **Services** menu to go to **CloudFormation**. 57 | 58 | **11.** Select the CloudFormation stack (click the circle). 59 | 60 | **12.** Click **Delete**. 61 | 62 | This will delete the stack and will stop charges being incurred in your AWS account. 63 | -------------------------------------------------------------------------------- /sagemaker-churn-lab.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Customer Churn Prediction with XGBoost\n", 8 | "_**Using Gradient Boosted Trees to Predict Mobile Customer Departure**_\n", 9 | "\n", 10 | "---\n", 11 | "\n", 12 | "---\n", 13 | "\n", 14 | "## Contents\n", 15 | "\n", 16 | "1. [Background](#Background)\n", 17 | "1. [Setup](#Setup)\n", 18 | "1. [Data](#Data)\n", 19 | "1. [Train](#Train)\n", 20 | "1. [Host](#Host)\n", 21 | " 1. [Evaluate](#Evaluate)\n", 22 | " 1. [Relative cost of errors](#Relative-cost-of-errors)\n", 23 | "1. [Extensions](#Extensions)\n", 24 | "\n", 25 | "---\n", 26 | "\n", 27 | "## Background\n", 28 | "\n", 29 | "This notebook has been adapted from an [AWS blog post](https://aws.amazon.com/blogs/ai/predicting-customer-churn-with-amazon-machine-learning/).\n", 30 | "\n", 31 | "Losing customers is costly for any business. Identifying unhappy customers early on gives you a chance to offer them incentives to stay. This notebook describes using machine learning (ML) for the automated identification of unhappy customers, also known as customer churn prediction. ML models rarely give perfect predictions though, so this notebook is also about how to incorporate the relative costs of prediction mistakes when determining the financial outcome of using ML.\n", 32 | "\n", 33 | "We use an example of churn that is familiar to all of us–leaving a mobile phone operator. It seems like you can always find fault with my provider du jour! And if your provider knows that I’m thinking of leaving, it can offer timely incentives–you can always use a phone upgrade or perhaps have a new feature activated–and you might just stick around. Incentives are often much more cost effective than losing and reacquiring a customer.\n", 34 | "\n", 35 | "---\n", 36 | "\n", 37 | "## How to use this notebook\n", 38 | "\n", 39 | "The notebook contains pre-written instructions for Amazon SageMaker.\n", 40 | "\n", 41 | "To the left of each block, you will see: `In: [ ]`\n", 42 | "\n", 43 | "This indicates that the block has not yet been run.\n", 44 | "\n", 45 | "To run a block, click within the block and **press Shift+Enter** or click the **Run** button at the top of the page.\n", 46 | "\n", 47 | "The sidebar will change to: `In: [*]`\n", 48 | "\n", 49 | "This indicates that the block is executing. You should wait until the sidebar changes to a number, such as: `In: [1]`\n", 50 | "\n", 51 | "The number indicates that the block has been run, and you can continue to the next block.\n", 52 | "\n", 53 | "Let's start by specifying loading the IAM Role and S3 bucket that will be used.\n" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "metadata": { 60 | "collapsed": true, 61 | "isConfigCell": true 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "# Define IAM role and S3 Bucket\n", 66 | "import boto3\n", 67 | "import re\n", 68 | "from sagemaker import get_execution_role\n", 69 | "\n", 70 | "role = get_execution_role()\n", 71 | "\n", 72 | "s3_client = boto3.client('s3')\n", 73 | "response = s3_client.list_buckets()\n", 74 | "bucket = [bucket['Name'] for bucket in response['Buckets'] if bucket['Name'].startswith('sagemaker-lab')][0]\n", 75 | "prefix = 'sagemaker/DEMO-xgboost-churn'" 76 | ] 77 | }, 78 | { 79 | "cell_type": "markdown", 80 | "metadata": {}, 81 | "source": [ 82 | "Next, we'll import the Python libraries we'll need for the remainder of the exercise." 83 | ] 84 | }, 85 | { 86 | "cell_type": "code", 87 | "execution_count": null, 88 | "metadata": { 89 | "collapsed": true 90 | }, 91 | "outputs": [], 92 | "source": [ 93 | "import pandas as pd\n", 94 | "import numpy as np\n", 95 | "import matplotlib.pyplot as plt\n", 96 | "import io\n", 97 | "import os\n", 98 | "import sys\n", 99 | "import time\n", 100 | "import json\n", 101 | "from IPython.display import display\n", 102 | "from time import strftime, gmtime\n", 103 | "import sagemaker\n", 104 | "from sagemaker.predictor import csv_serializer" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "metadata": {}, 110 | "source": [ 111 | "---\n", 112 | "## Data\n", 113 | "\n", 114 | "Mobile operators have historical records on which customers ultimately ended up churning and which continued using the service. We can use this historical information to construct an ML model of one mobile operator’s churn using a process called training. After training the model, we can pass the profile information of an arbitrary customer (the same profile information that we used to train the model) to the model, and have the model predict whether this customer is going to churn. Of course, we expect the model to make mistakes–after all, predicting the future is tricky business! But I’ll also show how to deal with prediction errors.\n", 115 | "\n", 116 | "The dataset we use is publicly available and was mentioned in the book [Discovering Knowledge in Data](https://www.amazon.com/dp/0470908742/) by Daniel T. Larose. It is attributed by the author to the University of California Irvine Repository of Machine Learning Datasets. Let's download and read that dataset in now:" 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": null, 122 | "metadata": { 123 | "collapsed": true 124 | }, 125 | "outputs": [], 126 | "source": [ 127 | "!wget http://dataminingconsultant.com/DKD2e_data_sets.zip\n", 128 | "!unzip -o DKD2e_data_sets.zip" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": null, 134 | "metadata": { 135 | "collapsed": true 136 | }, 137 | "outputs": [], 138 | "source": [ 139 | "churn = pd.read_csv('./Data sets/churn.txt')\n", 140 | "pd.set_option('display.max_columns', 500)\n", 141 | "churn" 142 | ] 143 | }, 144 | { 145 | "cell_type": "markdown", 146 | "metadata": {}, 147 | "source": [ 148 | "By modern standards, it’s a relatively small dataset, with only 3,333 records, where each record uses 21 attributes to describe the profile of a customer of an unknown US mobile operator. The attributes are:\n", 149 | "\n", 150 | "- `State`: the US state in which the customer resides, indicated by a two-letter abbreviation; for example, OH or NJ\n", 151 | "- `Account Length`: the number of days that this account has been active\n", 152 | "- `Area Code`: the three-digit area code of the corresponding customer’s phone number\n", 153 | "- `Phone`: the remaining seven-digit phone number\n", 154 | "- `Int’l Plan`: whether the customer has an international calling plan: yes/no\n", 155 | "- `VMail Plan`: whether the customer has a voice mail feature: yes/no\n", 156 | "- `VMail Message`: presumably the average number of voice mail messages per month\n", 157 | "- `Day Mins`: the total number of calling minutes used during the day\n", 158 | "- `Day Calls`: the total number of calls placed during the day\n", 159 | "- `Day Charge`: the billed cost of daytime calls\n", 160 | "- `Eve Mins, Eve Calls, Eve Charge`: the billed cost for calls placed during the evening\n", 161 | "- `Night Mins`, `Night Calls`, `Night Charge`: the billed cost for calls placed during nighttime\n", 162 | "- `Intl Mins`, `Intl Calls`, `Intl Charge`: the billed cost for international calls\n", 163 | "- `CustServ Calls`: the number of calls placed to Customer Service\n", 164 | "- `Churn?`: whether the customer left the service: true/false\n", 165 | "\n", 166 | "The last attribute, `Churn?`, is known as the target attribute–the attribute that we want the ML model to predict. Because the target attribute is binary, our model will be performing binary prediction, also known as binary classification.\n", 167 | "\n", 168 | "Let's begin exploring the data:" 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": null, 174 | "metadata": { 175 | "collapsed": true 176 | }, 177 | "outputs": [], 178 | "source": [ 179 | "# Frequency tables for each categorical feature\n", 180 | "for column in churn.select_dtypes(include=['object']).columns:\n", 181 | " display(pd.crosstab(index=churn[column], columns='% observations', normalize='columns'))\n", 182 | "\n", 183 | "# Histograms for each numeric features\n", 184 | "display(churn.describe())\n", 185 | "%matplotlib inline\n", 186 | "hist = churn.hist(bins=30, sharey=True, figsize=(10, 10))" 187 | ] 188 | }, 189 | { 190 | "cell_type": "markdown", 191 | "metadata": {}, 192 | "source": [ 193 | "We can see immediately that:\n", 194 | "- `State` appears to be quite evenly distributed\n", 195 | "- `Phone` takes on too many unique values to be of any practical use. It's possible parsing out the prefix could have some value, but without more context on how these are allocated, we should avoid using it.\n", 196 | "- Only 14% of customers churned, so there is some class imabalance, but nothing extreme.\n", 197 | "- Most of the numeric features are surprisingly nicely distributed, with many showing bell-like gaussianity. `VMail Message` being a notable exception (and `Area Code` showing up as a feature we should convert to non-numeric)." 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": null, 203 | "metadata": { 204 | "collapsed": true 205 | }, 206 | "outputs": [], 207 | "source": [ 208 | "churn = churn.drop('Phone', axis=1)\n", 209 | "churn['Area Code'] = churn['Area Code'].astype(object)" 210 | ] 211 | }, 212 | { 213 | "cell_type": "markdown", 214 | "metadata": {}, 215 | "source": [ 216 | "Next let's look at the relationship between each of the features and our target variable." 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "execution_count": null, 222 | "metadata": { 223 | "collapsed": true 224 | }, 225 | "outputs": [], 226 | "source": [ 227 | "for column in churn.select_dtypes(include=['object']).columns:\n", 228 | " if column != 'Churn?':\n", 229 | " display(pd.crosstab(index=churn[column], columns=churn['Churn?'], normalize='columns'))\n", 230 | "\n", 231 | "for column in churn.select_dtypes(exclude=['object']).columns:\n", 232 | " print(column)\n", 233 | " hist = churn[[column, 'Churn?']].hist(by='Churn?', bins=30)\n", 234 | " plt.show()" 235 | ] 236 | }, 237 | { 238 | "cell_type": "markdown", 239 | "metadata": {}, 240 | "source": [ 241 | "Interestingly we see that churners appear:\n", 242 | "- Fairly evenly distributed geographically\n", 243 | "- More likely to have an international plan\n", 244 | "- Less likely to have a voicemail plan\n", 245 | "- To exhibit some bimodality in daily minutes (either higher or lower than the average for non-churners)\n", 246 | "- To have a larger number of customer service calls (which makes sense as we'd expect customers who experience lots of problems may be more likely to churn)\n", 247 | "\n", 248 | "In addition, we see that churners take on very similar distributions for features like `Day Mins` and `Day Charge`. That's not surprising as we'd expect minutes spent talking to correlate with charges. Let's dig deeper into the relationships between our features." 249 | ] 250 | }, 251 | { 252 | "cell_type": "code", 253 | "execution_count": null, 254 | "metadata": { 255 | "collapsed": true 256 | }, 257 | "outputs": [], 258 | "source": [ 259 | "display(churn.corr())\n", 260 | "pd.plotting.scatter_matrix(churn, figsize=(12, 12))\n", 261 | "plt.show()" 262 | ] 263 | }, 264 | { 265 | "cell_type": "markdown", 266 | "metadata": {}, 267 | "source": [ 268 | "We see several features that essentially have 100% correlation with one another. Including these feature pairs in some machine learning algorithms can create catastrophic problems, while in others it will only introduce minor redundancy and bias. Let's remove one feature from each of the highly correlated pairs: Day Charge from the pair with Day Mins, Night Charge from the pair with Night Mins, Intl Charge from the pair with Intl Mins:" 269 | ] 270 | }, 271 | { 272 | "cell_type": "code", 273 | "execution_count": null, 274 | "metadata": { 275 | "collapsed": true 276 | }, 277 | "outputs": [], 278 | "source": [ 279 | "churn = churn.drop(['Day Charge', 'Eve Charge', 'Night Charge', 'Intl Charge'], axis=1)" 280 | ] 281 | }, 282 | { 283 | "cell_type": "markdown", 284 | "metadata": {}, 285 | "source": [ 286 | "Now that we've cleaned up our dataset, let's determine which algorithm to use. As mentioned above, there appear to be some variables where both high and low (but not intermediate) values are predictive of churn. In order to accommodate this in an algorithm like linear regression, we'd need to generate polynomial (or bucketed) terms. Instead, let's attempt to model this problem using gradient boosted trees. Amazon SageMaker provides an XGBoost container that we can use to train in a managed, distributed setting, and then host as a real-time prediction endpoint. XGBoost uses gradient boosted trees which naturally account for non-linear relationships between features and the target variable, as well as accommodating complex interactions between features.\n", 287 | "\n", 288 | "Amazon SageMaker XGBoost can train on data in either a CSV or LibSVM format. For this example, we'll stick with CSV. It should:\n", 289 | "- Have the predictor variable in the first column\n", 290 | "- Not have a header row\n", 291 | "\n", 292 | "But first, let's convert our categorical features into numeric features." 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": null, 298 | "metadata": { 299 | "collapsed": true 300 | }, 301 | "outputs": [], 302 | "source": [ 303 | "model_data = pd.get_dummies(churn)\n", 304 | "model_data = pd.concat([model_data['Churn?_True.'], model_data.drop(['Churn?_False.', 'Churn?_True.'], axis=1)], axis=1)" 305 | ] 306 | }, 307 | { 308 | "cell_type": "markdown", 309 | "metadata": {}, 310 | "source": [ 311 | "And now let's split the data into training, validation, and test sets. This will help prevent us from overfitting the model, and allow us to test the models accuracy on data it hasn't already seen." 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": null, 317 | "metadata": { 318 | "collapsed": true 319 | }, 320 | "outputs": [], 321 | "source": [ 322 | "train_data, validation_data, test_data = np.split(model_data.sample(frac=1, random_state=1729), [int(0.7 * len(model_data)), int(0.9 * len(model_data))])\n", 323 | "train_data.to_csv('train.csv', header=False, index=False)\n", 324 | "validation_data.to_csv('validation.csv', header=False, index=False)" 325 | ] 326 | }, 327 | { 328 | "cell_type": "markdown", 329 | "metadata": {}, 330 | "source": [ 331 | "Now we'll upload these files to S3." 332 | ] 333 | }, 334 | { 335 | "cell_type": "code", 336 | "execution_count": null, 337 | "metadata": { 338 | "collapsed": true 339 | }, 340 | "outputs": [], 341 | "source": [ 342 | "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'train/train.csv')).upload_file('train.csv')\n", 343 | "boto3.Session().resource('s3').Bucket(bucket).Object(os.path.join(prefix, 'validation/validation.csv')).upload_file('validation.csv')" 344 | ] 345 | }, 346 | { 347 | "cell_type": "markdown", 348 | "metadata": {}, 349 | "source": [ 350 | "---\n", 351 | "## Train\n", 352 | "\n", 353 | "Moving onto training, first we'll need to specify the locations of the XGBoost algorithm containers." 354 | ] 355 | }, 356 | { 357 | "cell_type": "code", 358 | "execution_count": null, 359 | "metadata": { 360 | "collapsed": true 361 | }, 362 | "outputs": [], 363 | "source": [ 364 | "from sagemaker.amazon.amazon_estimator import get_image_uri\n", 365 | "container = get_image_uri(boto3.Session().region_name, 'xgboost')" 366 | ] 367 | }, 368 | { 369 | "cell_type": "markdown", 370 | "metadata": {}, 371 | "source": [ 372 | "Then, because we're training with the CSV file format, we'll create `s3_input`s that our training function can use as a pointer to the files in S3." 373 | ] 374 | }, 375 | { 376 | "cell_type": "code", 377 | "execution_count": null, 378 | "metadata": { 379 | "collapsed": true 380 | }, 381 | "outputs": [], 382 | "source": [ 383 | "s3_input_train = sagemaker.s3_input(s3_data='s3://{}/{}/train'.format(bucket, prefix), content_type='csv')\n", 384 | "s3_input_validation = sagemaker.s3_input(s3_data='s3://{}/{}/validation/'.format(bucket, prefix), content_type='csv')" 385 | ] 386 | }, 387 | { 388 | "cell_type": "markdown", 389 | "metadata": {}, 390 | "source": [ 391 | "Now, we can specify a few parameters like what type of training instances we'd like to use and how many, as well as our XGBoost hyperparameters. A few key hyperparameters are:\n", 392 | "- `max_depth` controls how deep each tree within the algorithm can be built. Deeper trees can lead to better fit, but are more computationally expensive and can lead to overfitting. There is typically some trade-off in model performance that needs to be explored between a large number of shallow trees and a smaller number of deeper trees.\n", 393 | "- `subsample` controls sampling of the training data. This technique can help reduce overfitting, but setting it too low can also starve the model of data.\n", 394 | "- `num_round` controls the number of boosting rounds. This is essentially the subsequent models that are trained using the residuals of previous iterations. Again, more rounds should produce a better fit on the training data, but can be computationally expensive or lead to overfitting.\n", 395 | "- `eta` controls how aggressive each round of boosting is. Larger values lead to more conservative boosting.\n", 396 | "- `gamma` controls how aggressively trees are grown. Larger values lead to more conservative models.\n", 397 | "\n", 398 | "More detail on XGBoost's hyperparmeters can be found on their GitHub [page](https://github.com/dmlc/xgboost/blob/master/doc/parameter.md).\n", 399 | "\n", 400 | "**This step will take approximately 3 minutes.**" 401 | ] 402 | }, 403 | { 404 | "cell_type": "code", 405 | "execution_count": null, 406 | "metadata": { 407 | "collapsed": true 408 | }, 409 | "outputs": [], 410 | "source": [ 411 | "sess = sagemaker.Session()\n", 412 | "\n", 413 | "xgb = sagemaker.estimator.Estimator(container,\n", 414 | " role, \n", 415 | " train_instance_count=1, \n", 416 | " train_instance_type='ml.m4.xlarge',\n", 417 | " output_path='s3://{}/{}/output'.format(bucket, prefix),\n", 418 | " sagemaker_session=sess)\n", 419 | "xgb.set_hyperparameters(max_depth=5,\n", 420 | " eta=0.2,\n", 421 | " gamma=4,\n", 422 | " min_child_weight=6,\n", 423 | " subsample=0.8,\n", 424 | " silent=0,\n", 425 | " objective='binary:logistic',\n", 426 | " num_round=100)\n", 427 | "\n", 428 | "xgb.fit({'train': s3_input_train, 'validation': s3_input_validation}) " 429 | ] 430 | }, 431 | { 432 | "cell_type": "markdown", 433 | "metadata": {}, 434 | "source": [ 435 | "---\n", 436 | "## Host\n", 437 | "\n", 438 | "Now that we've trained the algorithm, let's create a model and deploy it to a hosted endpoint.\n", 439 | "\n", 440 | "**This step will take approximately 5 minutes.**" 441 | ] 442 | }, 443 | { 444 | "cell_type": "code", 445 | "execution_count": null, 446 | "metadata": { 447 | "collapsed": true 448 | }, 449 | "outputs": [], 450 | "source": [ 451 | "xgb_predictor = xgb.deploy(initial_instance_count=1,\n", 452 | " instance_type='ml.m4.xlarge')" 453 | ] 454 | }, 455 | { 456 | "cell_type": "markdown", 457 | "metadata": {}, 458 | "source": [ 459 | "### Evaluate\n", 460 | "\n", 461 | "Now that we have a hosted endpoint running, we can make real-time predictions from our model very easily, simply by making an http POST request. But first, we'll need to setup serializers and deserializers for passing our `test_data` NumPy arrays to the model behind the endpoint." 462 | ] 463 | }, 464 | { 465 | "cell_type": "code", 466 | "execution_count": null, 467 | "metadata": { 468 | "collapsed": true 469 | }, 470 | "outputs": [], 471 | "source": [ 472 | "xgb_predictor.content_type = 'text/csv'\n", 473 | "xgb_predictor.serializer = csv_serializer\n", 474 | "xgb_predictor.deserializer = None" 475 | ] 476 | }, 477 | { 478 | "cell_type": "markdown", 479 | "metadata": {}, 480 | "source": [ 481 | "Now, we'll use a simple function to:\n", 482 | "1. Loop over our test dataset\n", 483 | "1. Split it into mini-batches of rows \n", 484 | "1. Convert those mini-batchs to CSV string payloads\n", 485 | "1. Retrieve mini-batch predictions by invoking the XGBoost endpoint\n", 486 | "1. Collect predictions and convert from the CSV output our model provides into a NumPy array" 487 | ] 488 | }, 489 | { 490 | "cell_type": "code", 491 | "execution_count": null, 492 | "metadata": { 493 | "collapsed": true 494 | }, 495 | "outputs": [], 496 | "source": [ 497 | "def predict(data, rows=500):\n", 498 | " split_array = np.array_split(data, int(data.shape[0] / float(rows) + 1))\n", 499 | " predictions = ''\n", 500 | " for array in split_array:\n", 501 | " predictions = ','.join([predictions, xgb_predictor.predict(array).decode('utf-8')])\n", 502 | "\n", 503 | " return np.fromstring(predictions[1:], sep=',')\n", 504 | "\n", 505 | "predictions = predict(test_data.as_matrix()[:, 1:])" 506 | ] 507 | }, 508 | { 509 | "cell_type": "markdown", 510 | "metadata": {}, 511 | "source": [ 512 | "There are many ways to compare the performance of a machine learning model, but let's start by simply by comparing actual to predicted values. In this case, we're simply predicting whether the customer churned (`1`) or not (`0`), which produces a simple confusion matrix." 513 | ] 514 | }, 515 | { 516 | "cell_type": "code", 517 | "execution_count": null, 518 | "metadata": { 519 | "collapsed": true 520 | }, 521 | "outputs": [], 522 | "source": [ 523 | "pd.crosstab(index=test_data.iloc[:, 0], columns=np.round(predictions), rownames=['actual'], colnames=['predictions'])" 524 | ] 525 | }, 526 | { 527 | "cell_type": "markdown", 528 | "metadata": {}, 529 | "source": [ 530 | "_Note, due to randomized elements of the algorithm, you results may differ slightly._\n", 531 | "\n", 532 | "Of the 48 churners, we've correctly predicted 39 of them (true positives). And, we incorrectly predicted 4 customers would churn who then ended up not doing so (false positives). There are also 9 customers who ended up churning, that we predicted would not (false negatives).\n", 533 | "\n", 534 | "An important point here is that because of the `np.round()` function above we are using a simple threshold (or cutoff) of 0.5. Our predictions from `xgboost` come out as continuous values between 0 and 1 and we force them into the binary classes that we began with. However, because a customer that churns is expected to cost the company more than proactively trying to retain a customer who we think might churn, we should consider adjusting this cutoff. That will almost certainly increase the number of false positives, but it can also be expected to increase the number of true positives and reduce the number of false negatives.\n", 535 | "\n", 536 | "To get a rough intuition here, let's look at the continuous values of our predictions." 537 | ] 538 | }, 539 | { 540 | "cell_type": "code", 541 | "execution_count": null, 542 | "metadata": { 543 | "collapsed": true 544 | }, 545 | "outputs": [], 546 | "source": [ 547 | "plt.hist(predictions)\n", 548 | "plt.show()" 549 | ] 550 | }, 551 | { 552 | "cell_type": "markdown", 553 | "metadata": {}, 554 | "source": [ 555 | "The continuous valued predictions coming from our model tend to skew toward 0 or 1, but there is sufficient mass between 0.1 and 0.9 that adjusting the cutoff should indeed shift a number of customers' predictions. For example..." 556 | ] 557 | }, 558 | { 559 | "cell_type": "code", 560 | "execution_count": null, 561 | "metadata": { 562 | "collapsed": true 563 | }, 564 | "outputs": [], 565 | "source": [ 566 | "pd.crosstab(index=test_data.iloc[:, 0], columns=np.where(predictions > 0.3, 1, 0))" 567 | ] 568 | }, 569 | { 570 | "cell_type": "markdown", 571 | "metadata": {}, 572 | "source": [ 573 | "We can see that changing the cutoff from 0.5 to 0.3 results in 1 more true positives, 3 more false positives, and 1 fewer false negatives. The numbers are small overall here, but that's 6-10% of customers overall that are shifting because of a change to the cutoff. Was this the right decision? We may end up retaining 3 extra customers, but we also unnecessarily incentivized 5 more customers who would have stayed. Determining optimal cutoffs is a key step in properly applying machine learning in a real-world setting. Let's discuss this more broadly and then apply a specific, hypothetical solution for our current problem.\n", 574 | "\n", 575 | "### Relative cost of errors\n", 576 | "\n", 577 | "Any practical binary classification problem is likely to produce a similarly sensitive cutoff. That by itself isn’t a problem. After all, if the scores for two classes are really easy to separate, the problem probably isn’t very hard to begin with and might even be solvable with simple rules instead of ML.\n", 578 | "\n", 579 | "More important, if I put an ML model into production, there are costs associated with the model erroneously assigning false positives and false negatives. I also need to look at similar costs associated with correct predictions of true positives and true negatives. Because the choice of the cutoff affects all four of these statistics, I need to consider the relative costs to the business for each of these four outcomes for each prediction.\n", 580 | "\n", 581 | "#### Assigning costs\n", 582 | "\n", 583 | "What are the costs for our problem of mobile operator churn? The costs, of course, depend on the specific actions that the business takes. Let's make some assumptions here.\n", 584 | "\n", 585 | "First, assign the true negatives the cost of \\$0. Our model essentially correctly identified a happy customer in this case, and we don’t need to do anything.\n", 586 | "\n", 587 | "False negatives are the most problematic, because they incorrectly predict that a churning customer will stay. We lose the customer and will have to pay all the costs of acquiring a replacement customer, including foregone revenue, advertising costs, administrative costs, point of sale costs, and likely a phone hardware subsidy. A quick search on the Internet reveals that such costs typically run in the hundreds of dollars so, for the purposes of this example, let's assume \\$500. This is the cost of false negatives.\n", 588 | "\n", 589 | "Finally, for customers that our model identifies as churning, let's assume a retention incentive in the amount of \\$100. If my provider offered me such a concession, I’d certainly think twice before leaving. This is the cost of both true positive and false positive outcomes. In the case of false positives (the customer is happy, but the model mistakenly predicted churn), we will “waste” the \\$100 concession. We probably could have spent that \\$100 more effectively, but it's possible we increased the loyalty of an already loyal customer, so that’s not so bad." 590 | ] 591 | }, 592 | { 593 | "cell_type": "markdown", 594 | "metadata": {}, 595 | "source": [ 596 | "#### Finding the optimal cutoff\n", 597 | "\n", 598 | "It’s clear that false negatives are substantially more costly than false positives. Instead of optimizing for error based on the number of customers, we should be minimizing a cost function that looks like this:\n", 599 | "\n", 600 | "```txt\n", 601 | "$500 * FN(C) + $0 * TN(C) + $100 * FP(C) + $100 * TP(C)\n", 602 | "```\n", 603 | "\n", 604 | "FN(C) means that the false negative percentage is a function of the cutoff, C, and similar for TN, FP, and TP. We need to find the cutoff, C, where the result of the expression is smallest.\n", 605 | "\n", 606 | "A straightforward way to do this, is to simply run a simulation over a large number of possible cutoffs. We test 100 possible values in the for loop below." 607 | ] 608 | }, 609 | { 610 | "cell_type": "code", 611 | "execution_count": null, 612 | "metadata": { 613 | "collapsed": true 614 | }, 615 | "outputs": [], 616 | "source": [ 617 | "cutoffs = np.arange(0.01, 1, 0.01)\n", 618 | "costs = []\n", 619 | "for c in cutoffs:\n", 620 | " costs.append(np.sum(np.sum(np.array([[0, 100], [500, 100]]) * \n", 621 | " pd.crosstab(index=test_data.iloc[:, 0], \n", 622 | " columns=np.where(predictions > c, 1, 0)))))\n", 623 | "\n", 624 | "costs = np.array(costs)\n", 625 | "plt.plot(cutoffs, costs)\n", 626 | "plt.show()\n", 627 | "print('Cost is minimized near a cutoff of:', cutoffs[np.argmin(costs)], 'for a cost of:', np.min(costs))" 628 | ] 629 | }, 630 | { 631 | "cell_type": "markdown", 632 | "metadata": {}, 633 | "source": [ 634 | "The above chart shows how picking a threshold too low results in costs skyrocketing as all customers are given a retention incentive. Meanwhile, setting the threshold too high results in too many lost customers, which ultimately grows to be nearly as costly. The overall cost can be minimized at \\$8400 by setting the cutoff to 0.46, which is substantially better than the \\$20k+ I would expect to lose by not taking any action." 635 | ] 636 | }, 637 | { 638 | "cell_type": "markdown", 639 | "metadata": {}, 640 | "source": [ 641 | "---\n", 642 | "## Extensions\n", 643 | "\n", 644 | "This notebook showcased how to build a model that predicts whether a customer is likely to churn, and then how to optimally set a threshold that accounts for the cost of true positives, false positives, and false negatives. There are several means of extending it including:\n", 645 | "- Some customers who receive retention incentives will still churn. Including a probability of churning despite receiving an incentive in our cost function would provide a better ROI on our retention programs.\n", 646 | "- Customers who switch to a lower-priced plan or who deactivate a paid feature represent different kinds of churn that could be modeled separately.\n", 647 | "- Modeling the evolution of customer behavior. If usage is dropping and the number of calls placed to Customer Service is increasing, you are more likely to experience churn then if the trend is the opposite. A customer profile should incorporate behavior trends.\n", 648 | "- Actual training data and monetary cost assignments could be more complex.\n", 649 | "- Multiple models for each type of churn could be needed.\n", 650 | "\n", 651 | "Regardless of additional complexity, similar principles described in this notebook are likely apply." 652 | ] 653 | }, 654 | { 655 | "cell_type": "markdown", 656 | "metadata": {}, 657 | "source": [ 658 | "### Clean-up\n", 659 | "\n", 660 | "Please run the cell below. This will remove the hosted endpoint and Amazon S3 bucket to avoid any charges from a stray instance being left on." 661 | ] 662 | }, 663 | { 664 | "cell_type": "code", 665 | "execution_count": null, 666 | "metadata": { 667 | "collapsed": true 668 | }, 669 | "outputs": [], 670 | "source": [ 671 | "sagemaker.Session().delete_endpoint(xgb_predictor.endpoint)\n", 672 | "\n", 673 | "import os\n", 674 | "os.system('aws s3 rb --force s3://' + bucket)" 675 | ] 676 | }, 677 | { 678 | "cell_type": "markdown", 679 | "metadata": {}, 680 | "source": [ 681 | "### Return to the lab instructions\n", 682 | "\n", 683 | "**Important:** Please return to the Lab instructions and follow the steps to delete the CloudFormation stack to prevent further SageMaker costs." 684 | ] 685 | } 686 | ], 687 | "metadata": { 688 | "kernelspec": { 689 | "display_name": "conda_python3", 690 | "language": "python", 691 | "name": "conda_python3" 692 | }, 693 | "language_info": { 694 | "codemirror_mode": { 695 | "name": "ipython", 696 | "version": 3 697 | }, 698 | "file_extension": ".py", 699 | "mimetype": "text/x-python", 700 | "name": "python", 701 | "nbconvert_exporter": "python", 702 | "pygments_lexer": "ipython3", 703 | "version": "3.6.2" 704 | }, 705 | "notice": "Copyright 2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License." 706 | }, 707 | "nbformat": 4, 708 | "nbformat_minor": 2 709 | } -------------------------------------------------------------------------------- /sagemaker-churn-lab.yaml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: 2010-09-09 2 | 3 | Parameters: 4 | SageMakerNotebookInstanceType: 5 | Type: String 6 | Default: ml.m4.xlarge 7 | 8 | Resources: 9 | LabBucket: 10 | Type: AWS::S3::Bucket 11 | DeletionPolicy: Retain 12 | Properties: 13 | BucketName: !Join 14 | - "-" 15 | - - "sagemaker-lab" 16 | - !Select 17 | - 0 18 | - !Split 19 | - "-" 20 | - !Select 21 | - 2 22 | - !Split 23 | - "/" 24 | - !Ref "AWS::StackId" 25 | 26 | SageMakerNotebookInstanceRole: 27 | Type: 'AWS::IAM::Role' 28 | Properties: 29 | AssumeRolePolicyDocument: 30 | Version: 2012-10-17 31 | Statement: 32 | - Effect: Allow 33 | Principal: 34 | Service: 35 | - sagemaker.amazonaws.com 36 | Action: 37 | - 'sts:AssumeRole' 38 | ManagedPolicyArns: 39 | - 'arn:aws:iam::aws:policy/AmazonSageMakerFullAccess' 40 | Path: / 41 | Policies: 42 | - PolicyName: IAMReadOnly 43 | PolicyDocument: 44 | Version: 2012-10-17 45 | Statement: 46 | - Effect: Allow 47 | Action: 48 | - 'iam:List*' 49 | - 'iam:Get*' 50 | Resource: '*' 51 | - PolicyName: DataBucketAccess 52 | PolicyDocument: 53 | Version: 2012-10-17 54 | Statement: 55 | - Effect: Allow 56 | Action: ['s3:GetObject', 57 | 's3:PutObject', 58 | 's3:GetObjectAcl', 59 | 's3:PutObjectAcl', 60 | 's3:ListBucket', 61 | 's3:DeleteObject', 62 | 's3:DeleteBucket'] 63 | Resource: 64 | - arn:aws:s3:::sagemaker-lab* 65 | - arn:aws:s3:::sagemaker-lab*/* 66 | 67 | SageMakerNotebookInstanceLifecycleConfig: 68 | Type: 'AWS::SageMaker::NotebookInstanceLifecycleConfig' 69 | Properties: 70 | OnStart: 71 | - Content: 72 | Fn::Base64: 73 | !Sub | 74 | #!/bin/bash 75 | wget -O /home/ec2-user/SageMaker/lab_notebook.ipynb https://raw.githubusercontent.com/aws-john/sagemaker-churn/master/sagemaker-churn-lab.ipynb 76 | chown ec2-user:ec2-user /home/ec2-user/SageMaker/lab_notebook.ipynb 77 | 78 | SageMakerNotebookInstance: 79 | Type: 'AWS::SageMaker::NotebookInstance' 80 | Properties: 81 | InstanceType: !Ref SageMakerNotebookInstanceType 82 | LifecycleConfigName: !GetAtt 83 | - SageMakerNotebookInstanceLifecycleConfig 84 | - NotebookInstanceLifecycleConfigName 85 | RoleArn: !GetAtt 86 | - SageMakerNotebookInstanceRole 87 | - Arn 88 | 89 | Outputs: 90 | LabBucket: 91 | Value: !Ref LabBucket 92 | --------------------------------------------------------------------------------