├── Images ├── after_normalization.png ├── age_vs_daily_internet_usage.png ├── age_vs_daily_time_spent_on_site.png ├── before_normalization.png ├── bivariate_cats.png ├── bivariate_nums.png ├── bivariate_nums2.png ├── clicked_on_ad.png ├── confusion_matrix_best_model.png ├── corr_cats.png ├── corr_nums.png ├── corr_nums_target.png ├── daily_internet_usage_vs_daily_spent_time_on_site.png ├── data_exploration.png ├── desc_categorical.png ├── desc_numerical.png ├── duplicates.png ├── feature_importance.png ├── feature_importance2.png ├── header.png ├── learning_curve_best_model.png ├── null_values.png ├── null_values_after.png ├── outliers_after.png ├── outliers_before.png ├── uni_cats.png ├── uni_nums.png └── uni_nums2.png ├── Predicting_Ad_Clicks_Classification_by_Using_Machine_Learning.ipynb ├── README.md └── requirements.txt /Images/after_normalization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/after_normalization.png -------------------------------------------------------------------------------- /Images/age_vs_daily_internet_usage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/age_vs_daily_internet_usage.png -------------------------------------------------------------------------------- /Images/age_vs_daily_time_spent_on_site.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/age_vs_daily_time_spent_on_site.png -------------------------------------------------------------------------------- /Images/before_normalization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/before_normalization.png -------------------------------------------------------------------------------- /Images/bivariate_cats.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/bivariate_cats.png -------------------------------------------------------------------------------- /Images/bivariate_nums.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/bivariate_nums.png -------------------------------------------------------------------------------- /Images/bivariate_nums2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/bivariate_nums2.png -------------------------------------------------------------------------------- /Images/clicked_on_ad.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/clicked_on_ad.png -------------------------------------------------------------------------------- /Images/confusion_matrix_best_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/confusion_matrix_best_model.png -------------------------------------------------------------------------------- /Images/corr_cats.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/corr_cats.png -------------------------------------------------------------------------------- /Images/corr_nums.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/corr_nums.png -------------------------------------------------------------------------------- /Images/corr_nums_target.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/corr_nums_target.png -------------------------------------------------------------------------------- /Images/daily_internet_usage_vs_daily_spent_time_on_site.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/daily_internet_usage_vs_daily_spent_time_on_site.png -------------------------------------------------------------------------------- /Images/data_exploration.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/data_exploration.png -------------------------------------------------------------------------------- /Images/desc_categorical.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/desc_categorical.png -------------------------------------------------------------------------------- /Images/desc_numerical.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/desc_numerical.png -------------------------------------------------------------------------------- /Images/duplicates.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/duplicates.png -------------------------------------------------------------------------------- /Images/feature_importance.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/feature_importance.png -------------------------------------------------------------------------------- /Images/feature_importance2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/feature_importance2.png -------------------------------------------------------------------------------- /Images/header.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/header.png -------------------------------------------------------------------------------- /Images/learning_curve_best_model.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/learning_curve_best_model.png -------------------------------------------------------------------------------- /Images/null_values.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/null_values.png -------------------------------------------------------------------------------- /Images/null_values_after.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/null_values_after.png -------------------------------------------------------------------------------- /Images/outliers_after.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/outliers_after.png -------------------------------------------------------------------------------- /Images/outliers_before.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/outliers_before.png -------------------------------------------------------------------------------- /Images/uni_cats.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/uni_cats.png -------------------------------------------------------------------------------- /Images/uni_nums.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/uni_nums.png -------------------------------------------------------------------------------- /Images/uni_nums2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/farrellwahyudi/Predicting-Ad-Clicks-Classification-by-Using-Machine-Learning/2c7db2e2dc82bec2a0b7d609155130a662455100/Images/uni_nums2.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Predicting Ad Clicks: Classification by Using Machine Learning 2 | Predict Ad Clicks 3 | 4 | ## Background 5 | An Indonesian company is interested in evaluating the performance of advertisement on their website. This evaluation is essential for the company as it helps gauge the advertisement’s reach and its ability to engage customers. 6 | 7 | By analyzing historical advertisement data and uncovering insights and trends, this approach can aid the company in defining their marketing objectives. In this specific case, the primary emphasis lies in developing a machine learning classification model that can accurately identify the target customer demographic. 8 | 9 | ## Problem 10 | 1. The company in question currently displays ads for all of its users. This “shotgun” strategy yields them an ad click rate of 50%. 11 | 2. The company spends a nonoptimal amount of resources by displaying ads for all of its users. 12 | 13 | ## Objectives 14 | 1. Create well-fit machine learning models that can reliably predict which users are likely to click on an ad. 15 | 2. Determine the best machine learning model to be implemented in this use case based on; evaluation metrics (Recall and Accuracy), model simplicity, and prediction time. 16 | 3. Identify the factors that most influence users’ likelihood to click on an ad. 17 | 4. Provide recommendations for potential strategies regarding targeted ads to marketing and management teams based on findings from analyzes and modeling. 18 | 5. Calculate the potential impact of model implementation on profit and click rate. 19 | 20 | ## About the Dataset 21 | The dataset was obtained from [Rakamin Academy](https://www.rakamin.com/). 22 | 23 | **Description:** 24 | 25 | - Unnamed: 0 = ID of Customers 26 | - Daily Time Spent on Site = Time spent by the user on a site in minutes 27 | - Age = Customer’s age in terms of years 28 | - Area Income = Average income of geographical area of consumer 29 | - Daily Internet Usage = Average minutes in a day consumer is on the internet 30 | - Male = Gender of the customer 31 | - Timestamp = Time at which user clicked on an Ad or the closed window 32 | - Clicked on Ad = Whether or not the customer clicked on an Ad (Target Variable) 33 | - city = City of the consumer 34 | - province = Province of the consumer 35 | - category = Category of the advertisement 36 | 37 | **Overview:** 38 | 39 | 1. Dataset contains 1000 rows, 10 features and 1 Unnamed: 0 column which is the ID column. 40 | 2. Dataset consists of 3 data types; float64, int64 and object. 41 | 3. Timestamp feature could be changed into datetime data type. 42 | 4. Dataset contains null values in various columns. 43 | 44 | ## Data Analysis 45 | ### Univariate 46 | Univariate Numerical 47 |                                                 KDE Plot of Numerical Features 48 |


49 | Univariate Numerical 50 |                                                     Boxplot of Numerical Features 51 | 52 | **Analysis:** 53 | 54 | - Area Income is the only feature with a slight skew (left-skewed). 55 | - Daily Internet Usage is nearly uniformly distributed. 56 | - While Age and Daily Time Spent on Site is nearly normally distributed. 57 | 58 | ### Bivariate 59 | Bivariate Numerical 60 |     KDE Plot of Numerical Features Between Users That Clicked and Didn’t Click On Ad 61 | 62 |


63 | Bivariate Numerical 64 |               Boxplot of Numerical Features Between Users That Clicked and Didn’t Click On Ad 65 |
66 | 67 | ### Scatterplot 68 | 69 | Scatterplot 70 | Scatterplot 71 | Scatterplot 72 | 73 | **Analysis:** 74 | 75 | - The more time is spent on site by the customer the less likely they will click on an ad. 76 | - The average age of customers that clicked on an ad is 40, while the average for those that didn’t is 31. 77 | - The average area income of customers that clicked on an ad is considerably lower than those that didn’t. 78 | - Similar to time spent, the more the daily internet usage is, the less likely the customer will click on an ad. 79 | - As can be seen the last scatterplot above, there is a quite clear separation between two clusters of data. One cluster is less active and the other more so. Less active customers have a higher tendency to click on an ad compared to more active customers. 80 | 81 | ### Multivariate (Numerical) 82 | Numerical Correlation 83 | 84 | Since the target variable is binary (Clicked or didn’t click), we can’t use the standard Pearson correlation to see the correlation between the numerical features and the target. Hence, the **Point Biserial correlation** was used to analyze the correlation between the target and the numerical features. 85 | 86 | Numerical Correlation with Target 87 | 88 | ### Multivariate (Categorical) 89 | In order to see the correlation between categorical features, I again couldn’t employ the standard Pearson correlation. There are numerous methods out there, but in this study I used **Cramer’s V** to understand the association and by extension the correlation between categorical features. 90 | 91 | Categorical Correlation 92 | 93 | **Analysis:** 94 | 95 | - The perfect correlation coefficient of 1 indicates that city and province are perfectly associated. This makes sense, since if you knew the city you would also know the province. This means that using both features for machine learning modeling is redundant. 96 | - All the numerical features (especially Daily Internet Usage and Daily Time Spent on Site) have high correlation with the target variable. 97 | 98 | ## Data Preprocessing 99 | ### Handling Missing Values 100 | Missing or null values were present in various columns in the data. These null values were imputed using statistically central values such as mean or median accordingly. 101 | 102 | ### Feature Extraction 103 | The Timestamp feature was of “object” data type. This feature was transformed into “date-time” data type in order to have its contents extracted. There are 3 extra features that were extracted from the Timestamp feature, these are: 104 | 105 | 1. Month 106 | 2. Week 107 | 3. Day (Mon-Sun) 108 | 109 | ### Handling Outliers 110 | Outliers were present in the Area Income feature. To help the data in becoming more “model friendly” to linear and distance based models, these outliers were removed from the data (using IQR method). 111 | 112 | **Before:** 113 | 114 | Outliers Before 115 | 116 | **After:** 117 | 118 | Outliers Before 119 | 120 | ### Feature Selection 121 | From the multivariate analysis it can be seen that city and province are redundant. However, both features have high cardinality and don’t correlate too well with the target variable. As a result, both of these features were excluded in the modeling as to prevent the curse of dimensionality. Timestamp was also excluded since its contents have been extracted and it was no longer needed. Unnamed: 0 or ID was obviously excluded since it is an identifier column and is unique for every user. 122 | 123 | ### Feature Encoding 124 | The categorical features were encoded so that machine learning models could read and understand its values. The category feature was One-Hot encoded whilst the rest (Gender and Clicked on Ad) were label encoded. 125 | 126 | ### Dataset Splitting 127 | The data was split into training and testing sets. This is common practice as to make sure that the machine learning models weren’t simply memorizing the answers from the data it was given. The data was split in a 75:25 ratio, 75% training data and 25% testing data. 128 | 129 | ## Modeling 130 | In the modeling phase, an experiment was conducted. Machine learning models were trained and tested with non-normalized/non-standardized version of the data. The results of which were compared to models trained and tested with normalized/standardized version of the data. Hyperparameter tuning was done in both scenarios, as to get the best out of each model. The model and standardization method with the best results will be selected. Six models were experimented with, these are: 131 | 132 | - Logistic Regression 133 | - Decision Tree 134 | - Random Forest 135 | - K-Nearest Neighbors 136 | - Gradient Boosting 137 | - XGBoost 138 | 139 | Target Variable 140 |          Target Variable Class Balance 141 |

142 | Because the target has perfect class balance the primary metric that will be used is Accuracy. Recall will be the secondary metric as to minimize false negatives. 143 | 144 | ### Without Normalization/Standardization 145 | **Results:** 146 | 147 | Before Normalization/Standardization 148 | 149 | **Observation:** 150 | - Decision Tree had the lowest fit time of all the models but the second lowest accuracy overall. 151 | - Gradient Boosting had the highest accuracy and recall scores but XGBoost is not far behind. 152 | - Due to the non-normalized data, distance based algorithms like K-Nearest Neighbours and linear algorithms like Logistic Regression suffered heavily. 153 | - Logistic Regression could not converge properly using newton-cg and as a result had the highest fit time of all the models, even though it probably is the simplest model of them all. 154 | - K-Nearest Neighbours suffered in accuracy and recall scores, with both being by far the lowest of all the models tested. 155 | 156 | ### Using Normalization/Standardization 157 | **Results:** 158 | 159 | After Normalization/Standardization 160 | 161 | **Observation:** 162 | - K-Nearest Neighbours had the highest fit time and elapsed time. 163 | - Gradient Boosting had the highest accuracy and the highest recall (tied with Random Forest), but Logistic Regression in close second, had the highest cross-validated accuracy of all the models tested. 164 | - Random Forest and XGBoost also had nearly identical scores in close third and fourth, although XGBoost had the better fit and elapsed times. 165 | - With normalized data, the previously poor performing distance based and linear models have shone through. 166 | - Logistic Regression's fit and elapsed times had been reduced significantly making it the model with the lowest times. It's scores have also massively improved making it a close second-place model. 167 | - K-Nearest Neighbours also saw massive improvement in its scores, with the model no longer sitting in last place in terms of scores. 168 | 169 | By taking consideration of not only the above metrics but also the simplicity, explainability and fit and elapsed times, the model that will be chosen is the Logistic Regression model with normalization/standardization. This is not only because of the very high scores (especially cross-validated scores) but also the simplicity, explainability and relatively quick fit and elapsed times. 170 | 171 | ### Evaluation of Selected Model (Logistic Regreession) 172 | **Learning Curve:** 173 | 174 | Learning Curve 175 | As can be seen from the above learning curve, the tuned Logistic Regression model with normalization/standardization is well fitted with no overfitting/underfitting. 176 |

177 | 178 | **Confusion Matrix:** 179 | 180 | Confusion Matrix 181 | 182 | From the test set confusion matrix above, from 120 people that clicked on an ad the algorithm correctly classified 116 of them and incorrectly classified 4 of them. Similarly, out of 128 people that did not click on an ad the algorithm correctly classified 124 of them and incorrectly classified only 4. 183 | 184 | Based on the confusion matrix and also the learning curve, it can be seen that Logistic Regression is a more than capable model to be implemented on this dataset. 185 | 186 | **Feature Importance:** 187 | 188 | Since Logistic Regression is such a simple and explainable model, to get the feature importance we can simply look at the coefficients of each feature in the model. 189 | 190 | The coefficients represent the change in the log odds for a one-unit change in the feature variable. Larger absolute values indicate a stronger relationship between the feature and the target variable, while the sign of the coefficients (negative or positive) indicates the direction of the relationship between the two. 191 | 192 | Feature Importance 193 | Feature Importance 194 | 195 | **Analysis:** 196 | 197 | Based on the feature importance charts above, it can clearly be seen that the two features with most effect on the model are Daily Time Spent on Site and Daily Internet Usage. 198 | - The lower the Daily Time Spent on Site the bigger the odds that the customer will click on an ad and vice versa. 199 | - Similarly, the lower the Daily Internet Usage the higher the chances that the customer will click on an ad and vice versa. 200 | - Other Important features include; Area Income and Age. 201 | 202 | ## Business Recommendations 203 | Based on the insights that have been gathered in the EDA as well as the feature importance from the model, the following business recommendations are formulated. 204 | 205 | - **Content Personalization and Targeting:** 206 | Since the lower the Daily Time Spent on Site is the more likely the user is to click on an ad, it’s essential to focus on content personalization and user engagement. Tailor content to keep users engaged but not overloaded. This can be achieved through strategies like recommending relevant content and using user data to customize the user experience. 207 | - **Age-Targeted Advertising:** 208 | Older individuals are more likely to engage with ads. Therefore we can consider creating ad campaigns that are specifically designed to target and appeal to older demographics. This may include promoting products or services relevant to their age group. 209 | - **Income-Level Targeting:** 210 | Users in areas with lower income levels are more likely to click on ads. Therefore we can create ad campaigns that are budget-friendly and appealing to users with lower income. Additionally, consider tailoring the ad messaging to highlight cost-effective solutions. 211 | - **Optimize Ad Placement for Active Internet Users:** 212 | Heavy internet users are less responsive to ads. To improve ad performance, consider optimizing ad placement for users with lower internet usage or finding ways to make ads stand out to this group, such as through eye-catching visuals or unique offers. 213 | 214 | ## Potential Impact Simulation 215 | Target Variable 216 | Using the original dataset’s Clicked on Ad numbers as can be seen above, the business simulation of before and after model implementation are as follows: 217 |

218 | 219 | **Assumption:** 220 | 221 | Cost per Advertisement: Rp.1000 222 | 223 | Revenue per Ad clicked: Rp.4000 224 | **** 225 | **Before model implementation:** 226 | 227 | - **No. Users Advertised**:
228 | Every User = 1000 229 | - **Click Rate**:
230 | 500/1000 = 50% 231 | - **Total Cost**:
232 | No. Users Advertised x Cost per Ad = 1000 x 1000 = Rp.1,000,000 233 | - **Total Revenue**:
234 | Click Rate x No. Users Advertised x Revenue per Ad Clicked = 0.5 x 1000 x 4000 = Rp.2,000,000 235 | - **Total Profit**:
236 | Total Revenue - Total Cost = **Rp.1,000,000** 237 | 238 | **After model implementation:** 239 | 240 | - **No. Users Advertised**:
241 | (Precision x 500) + ((1-Specificity) x 500) = (96.67% x 500) + (0.03125 x 500) = 483 + 16 = 499 242 | - **Click Rate**:
243 | (Precision x 500)/No. Users Advertised = 483/499 = 96.8% 244 | - **Total Cost**:
245 | No. Users Advertised x Cost per Ad = 499 x 1000 = Rp.499,000 246 | - **Total Revenue**:
247 | Click Rate x No. Users Advertised x Revenue per Ad Clicked = 0.968 x 499 x 4000 = Rp.1,932,000 248 | - **Total Profit**:
249 | Total Revenue - Total Cost = 1,932,000 - 499,000 = **Rp.1,433,000** 250 | 251 | **Conclusion:** 252 | 253 | By comparing the profits and click rates of before and after model implementation, we can see that with model implementation click rate is up from **50%** to **96.8%**, and similarly profit is up from **Rp.1,000,000** to **Rp.1,433,000** (a **43.3%** increase). 254 | 255 | Predict Ad Clicks 256 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pandas==1.5.3 2 | numpy==1.23.5 3 | matplotlib==3.7.1 4 | seaborn==0.13.0 5 | scipy==1.11.3 6 | scikit-learn==1.2.2 7 | xgboost==2.0.1 --------------------------------------------------------------------------------