├── 1. Statistical Simulation in R.Rmd ├── 2. Statistical Simulation in Python.ipynb ├── 3. A Practical Guide To AB Tests.ipynb ├── 4. AA Test.ipynb └── README.md /1. Statistical Simulation in R.Rmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Statistical Simulation" 3 | author: "Leihua Ye" 4 | date: "9/23/2020" 5 | output: html_document 6 | --- 7 | 8 | ```{r setup, include=FALSE} 9 | knitr::opts_chunk$set(echo = TRUE) 10 | library(dplyr) 11 | ``` 12 | 13 | # statistical simulation 101 14 | # part i: basics of sampling 15 | - runif 16 | - sample 17 | - sapply 18 | - apply 19 | - applications 20 | 21 | Check the sample function; it is the foundation of statistical modeling 22 | 23 | sample(x, size, replace = FALSE, prob = NULL) 24 | 25 | x: a vector of one or more elements from which to choose, or a positive integer 26 | size: a non-negative integer giving the number of items to choose 27 | replace: should sampling be with replacement? 28 | prob: a vector of probability weights for obtaining the elements of the vector being sampled 29 | 30 | # how to generate a random sample 31 | runif(n,min=minimal_value,max=max_value) 32 | ```{r} 33 | set.seed(2) 34 | runif(25,min=0,max=10) 35 | 36 | runif(25,min=0,max=10) %>% 37 | round(.,digits = 0) 38 | ``` 39 | 40 | 41 | ```{r} 42 | # flip a coin 43 | sample(c('H','T'),size = 10,replace=TRUE) 44 | sample(c(1:6),size=10,replace=TRUE) 45 | ``` 46 | 47 | # the sample() function draws randomly from a specified set of (scalar) objects allowing you to sample from arbitrary distributions of numbers 48 | ```{r} 49 | #1. random permutation of sequence [1,10] 50 | set.seed(2) 51 | sample(10) 52 | ?sample() 53 | ``` 54 | 55 | ```{r} 56 | sample(1:10,4) 57 | ``` 58 | 59 | ```{r} 60 | set.seed(1) 61 | sample(letters,18) 62 | ``` 63 | 64 | ```{r} 65 | # from r documentation 66 | x<- 1:10 67 | #a random permutation 68 | sample(x) 69 | 70 | # resample with replacement 71 | sample(x,replace=TRUE) 72 | ``` 73 | 74 | ```{r} 75 | # random sample of size 10 from sequence [1,5] with equal probabilities 76 | equal_prob_dist = sample(5,10000,prob=rep(0.1,5),replace=T) 77 | hist(equal_prob_dist) 78 | ``` 79 | 80 | ```{r} 81 | #random sample of size 10 from sequence[1,5] with unequal probabilities 82 | unequal_prob_dist = sample(5,10000,prob = c(0.1,0.25,0.4,0.25,0.1),replace=T) 83 | hist(unequal_prob_dist) 84 | ``` 85 | By default, the probability is equal if don't specify it. 86 | 87 | # To sample rows from a data frame or a list, we can sample the indices into an object rather than the elements of the object itself. 取元素对应的indices！ 88 | ```{r} 89 | head(mtcars) 90 | ``` 91 | ```{r} 92 | # create an index vector for the elements/rows 93 | index <- seq_len(nrow(mtcars)) 94 | 95 | #sample from the index vector 96 | set.seed(12) 97 | 98 | #to obtain a random sample of 10 99 | sample_index <- sample(index,10) 100 | 101 | # to show the sampled elements/rows 102 | mtcars[sample_index,] 103 | ``` 104 | 105 | 106 | 107 | # Part 2: Application 108 | # example 1: dies 109 | ```{r} 110 | # use sample() to run 10,000 trials using two fair dies. what is the probability of rolling a 7? 111 | set.seed(1) 112 | die = 1:6 113 | die1 = sample(die,10000,replace = TRUE,prob=NULL) 114 | die2= sample(die,10000,replace=TRUE,prob = NULL) 115 | outcomes = die1+die2 116 | mean(outcomes == 7) 117 | ``` 118 | 119 | ```{r} 120 | # simulate the process 10000 times and check for discrepency 121 | set.seed(1) 122 | for (i in 10000){ 123 | die_1 = sample(die,prob=NULL,replace=TRUE) 124 | die_2 = sample(die,prob=NULL,replace=TRUE) 125 | die_sum = die_1+die_2 126 | print(mean(die_sum==7)) 127 | } 128 | ``` 129 | ```{r} 130 | # check for system time 131 | system.time(for (i in 10000){ 132 | die_1 = sample(die,prob=NULL,replace=TRUE) 133 | die_2 = sample(die,prob=NULL,replace=TRUE) 134 | die_sum = die_1+die_2 135 | print(mean(die_sum==7)) 136 | }) 137 | ``` 138 | 139 | ```{r} 140 | #You have two dies, and what is the probability of rolling a 7? 141 | # use sample() to run 10,000 trials using two fair dies. 142 | set.seed(1) 143 | die = 1:6 144 | die_1 = sample(die,20000,replace=TRUE,prob=rep(1/6,6)) 145 | die_2 = sample(die,20000,replace=TRUE,prob=rep(1/6,6)) 146 | outcomes = die_1+die_2 147 | mean(outcomes==7) 148 | ``` 149 | 150 | ```{r} 151 | # What is the probability of rolling a 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, and 13? 152 | sapply(2:13,function(x) mean(outcomes==x)) 153 | #?sapply() 154 | ``` 155 | # check for sapply() 156 | sapply is a user-friendly version and wrapper of lapply by default returning a vector, matrix or, if simplify = "array", an array if appropriate, by applying simplify2array(). sapply(x, f, simplify = FALSE, USE.NAMES = FALSE) is the same as lapply(x, f). 157 | 158 | 159 | ```{r} 160 | # Use the sample() function to determine the probability of rolling a 7 using three fair six-sided dies. 161 | set.seed(1) 162 | die = 1:6 163 | die1 = sample(die,10000,replace=TRUE, prob = NULL) 164 | die2 = sample(die1,10000,replace=TRUE, prob = NULL) 165 | die3 = sample(die,10000,replace = TRUE, prob= NULL) 166 | outcomes2 = die1+die2+die3 167 | mean(outcomes2==7) 168 | ``` 169 | 170 | ```{r} 171 | # Use the sample() function to determine the probability of rolling a 7 using three fair six-sided dies. 172 | set.seed(1) 173 | die=1:6 174 | die_1 = sample(die, 20000,replace=TRUE,prob=NULL)# by default, equal probability 175 | die_2 = sample(die,20000,replace=TRUE, prob=NULL) 176 | die_3 = sample(die,20000,replace=TRUE, prob=NULL) 177 | die_combn = die_1+die_2+die_3 178 | mean(die_combn==7) 179 | ``` 180 | 181 | ```{r} 182 | # Using three dies, what is the probability of rolling a 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, and 13? 183 | sapply(2:13,function(x) mean(die_combn==x)) 184 | ``` 185 | 186 | # example 2 187 | ```{r} 188 | # do 10k samples and calculate how many observations are included and how many are not included? 189 | set.seed(1) 190 | n=10000 191 | 192 | included_obs = length(unique(sample(1:n, replace = TRUE))) 193 | included_obs 194 | 195 | missing_obs = n-included_obs 196 | missing_obs 197 | ``` 198 | 199 | ```{r} 200 | # from 1:100, do a 10k sample and calculate how many observations are included and how many are not included? 201 | set.seed(1) 202 | n = 100 203 | included_observations = length(unique(sample(1:100,replace=TRUE, prob=NULL))) 204 | included_observations/n 205 | 206 | (n-included_observations)/n 207 | ``` 208 | 209 | #3 example 3 210 | ```{r} 211 | # three ways of generating a m*n matrix with randomly assigned 0/1 212 | 213 | #3.1 for loop 214 | # create an empty matrix 215 | m <- 10 216 | n <- 10 217 | m00 <- matrix(0,m,n) 218 | 219 | for (i in 1:m) { 220 | for (j in 1:n) { 221 | m00[i,j] <-sample(c(0,1),1) 222 | } 223 | } 224 | 225 | m00 226 | ``` 227 | 228 | ```{r} 229 | system.time(for (i in 1:m) { 230 | for (j in 1:n) { 231 | m00[m,n] <-sample(c(0,1),1) 232 | } 233 | } 234 | ) 235 | ``` 236 | 237 | 238 | 239 | ```{r} 240 | #3.2 apply() function 241 | m <-10 242 | n<-10 243 | 244 | m0 <- matrix(0,m,n) 245 | 246 | apply(m0,c(1,2),function(x) sample(c(0,1),1)) 247 | 248 | system.time(apply(m0,c(1,2),function(x) sample(c(0,1),1))) 249 | ?apply() 250 | ``` 251 | apply(): 252 | - Returns a vector or array or list of values obtained by applying a function to margins of an array or matrix. 253 | - apply(x,margin, fun, ...) 254 | - x: an array, or matrix 255 | - margin: a vector giving the subscripts which the function will be applied over. for a matrix 1 indicates rows, and 2 indicates columns. c(1,2) indicates rows and columns. where x has named dimnames, it can be a character vector selecting dimension names 256 | - fun: the function to be applied. 257 | 258 | ```{r} 259 | #3.3 some other methods 260 | #1 generate a bunch of uniformly distributed[0,1) random numbers; round them to the closest integer 261 | m1<-round(matrix(runif(r*c),r,c)) 262 | #m1 263 | system.time(m1<-round(matrix(runif(r*c),r,c))) 264 | ``` 265 | 266 | ```{r} 267 | #generate r*c random numbers following a binomial distribution; 268 | # allow for different probabilities, rahter than 0.5 as in m1 269 | m2 <- matrix(rbinom(r*c,1,0.5),r,c) 270 | #m2 271 | ``` 272 | 273 | ```{r} 274 | system.time(m3<-matrix(round(runif(r*c)),r,c)) 275 | ``` 276 | 277 | ```{r} 278 | m4<-matrix(sample(0:1,r*c,replace=TRUE),r,c) 279 | #m4 280 | ``` 281 | 282 | #4 example 4 283 | Flip a coin 10 times and simulate the process for 10,000 times. Show the distribution of the number of heads shown up. 284 | ```{r} 285 | # create an empty list 286 | total_heads = c() 287 | 288 | # use a for looop to simulate coin-flipping 10 times; repeat it for 10,000 times. 289 | for (i in 1:10000){ 290 | sum_heads = sum(round(runif(10,0,1))) 291 | total_heads = c(total_heads, sum_heads) 292 | } 293 | 294 | hist(total_heads) 295 | ``` 296 | 297 | Reference: 298 | 1. https://bookdown.org/rdpeng/rprogdatascience/simulation.html#random-sampling 299 | 2. R documentation of Sampling 300 | 3. https://crumplab.github.io/programmingforpsych/simulating-and-analyzing-data-in-r.html 301 | -------------------------------------------------------------------------------- /2. Statistical Simulation in Python.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 1. Uniform Distribution" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# Question 1\n", 15 | "- In R or Python, please answer the following question. \n", 16 | "- Condition 1: For a sequence of numbers, (a1,a2,a3,a4,...,an), please write a function that randomly returns each element, ai, with probability ai/∑ai. \n", 17 | "- Condition 2: For example, for a sequence (1,2,3,4), the function returns element with a probability 1/10, and 4 with a probability 4/10.\n", 18 | "- Condition 3: You can use any library, but no random.choice()." 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": 63, 24 | "metadata": {}, 25 | "outputs": [], 26 | "source": [ 27 | "import numpy as np \n", 28 | "\n", 29 | "def weight_func(sequence):\n", 30 | "\n", 31 | " prob = [] # empty list to store probabilities of each element\n", 32 | "\n", 33 | " cum_prob = [] # empty list to store cumulative probabilities of each element \n", 34 | " \n", 35 | " total_sum = sum(sequence) # total sum of the sequence\n", 36 | " \n", 37 | " uniform = np.random.uniform(0,1) # a random value between 0 and 1 \n", 38 | " \n", 39 | " for i in range(len(sequence)): # create iterations for the sequence\n", 40 | " \n", 41 | " prob.append(sequence[i]/total_sum) # append the weighted probability (ai/∑ai) to the list, prob\n", 42 | " \n", 43 | " cum_prob.append(sum(prob)) # append the cumulative sum to the list, cum_prob \n", 44 | " \n", 45 | " if uniform < cum_prob[i]: # if the cumulative sum > uniform (the value generated from the uniform distribution)\n", 46 | " break # end the for looop \n", 47 | " \n", 48 | " return sequence[i] # and return the value at position i." 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": 66, 54 | "metadata": {}, 55 | "outputs": [ 56 | { 57 | "data": { 58 | "text/plain": [ 59 | "1" 60 | ] 61 | }, 62 | "execution_count": 66, 63 | "metadata": {}, 64 | "output_type": "execute_result" 65 | } 66 | ], 67 | "source": [ 68 | "weight_return([1,4,3,2]) # test case" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "---" 76 | ] 77 | }, 78 | { 79 | "cell_type": "markdown", 80 | "metadata": {}, 81 | "source": [ 82 | "# Question 2: Binomial Distribution" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": {}, 88 | "source": [ 89 | "An online shopping website (e.g., Amazon, Alibaba, etc.) wants to test out two versions of banners that will appear on the top of the website. The engineering team assigns the probability of visiting version A at 0.6 and version B 0.4. \n", 90 | "\n", 91 | "After 10,000 visits, there are 6050 visitors being exposed to version A and 3950 people exposed to version B. \n", 92 | "\n", 93 | "What is the probability that there are 6050 cases when the randomization process is correct? \n", 94 | "\n", 95 | "In other words, the probability for version A is indeed 0.6." 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 68, 101 | "metadata": {}, 102 | "outputs": [ 103 | { 104 | "data": { 105 | "text/plain": [ 106 | "0.1498" 107 | ] 108 | }, 109 | "execution_count": 68, 110 | "metadata": {}, 111 | "output_type": "execute_result" 112 | } 113 | ], 114 | "source": [ 115 | "import numpy as np\n", 116 | "\n", 117 | "np.random.seed(123)\n", 118 | "\n", 119 | "def some_funct(number_trials, probability_A):\n", 120 | " \n", 121 | " binomial_dist = np.random.binomial(n = number_trials,p = probability_A,size=10000) # generate a binomial distribution with n = number, p \n", 122 | " \n", 123 | " count = 0 # initialize count\n", 124 | " \n", 125 | " for value in binomial_dist: #iterate over binomial_dist\n", 126 | " \n", 127 | " if value > 6050: # if value>6050, count+1\n", 128 | " \n", 129 | " count += 1\n", 130 | " \n", 131 | " return count/number_trials # return the probability that cases larger than 6050 out of number_trials \n", 132 | "\n", 133 | "some_funct(number_trials=10000, probability_A = 0.6)" 134 | ] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "metadata": {}, 139 | "source": [ 140 | "---" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "metadata": {}, 146 | "source": [ 147 | "# Question 3: Poisson Distribution" 148 | ] 149 | }, 150 | { 151 | "cell_type": "markdown", 152 | "metadata": {}, 153 | "source": [ 154 | "My medium blog has 500 visits per day, and the number of visits follows a Poisson distribution. Out of 1000 times, what is the ratio that there would be more than 510 visits per day? " 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": 165, 160 | "metadata": {}, 161 | "outputs": [], 162 | "source": [ 163 | "import numpy as np \n", 164 | "\n", 165 | "np.random.seed(123)\n", 166 | "\n", 167 | "def Poisson(value_1,value_2): # two arguments\n", 168 | " \n", 169 | " count = 0 # initialize the counter\n", 170 | "\n", 171 | " poisson = np.random.poisson(lam=value_1,size = value_2) # a poisson distribution\n", 172 | "\n", 173 | " for i in poisson: # iteration\n", 174 | " \n", 175 | " if i > 510: # if clause to count numbers\n", 176 | " \n", 177 | " count+=1 \n", 178 | " \n", 179 | " return(count/value_2) # return the value" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 166, 185 | "metadata": {}, 186 | "outputs": [ 187 | { 188 | "data": { 189 | "text/plain": [ 190 | "0.318" 191 | ] 192 | }, 193 | "execution_count": 166, 194 | "metadata": {}, 195 | "output_type": "execute_result" 196 | } 197 | ], 198 | "source": [ 199 | "Poisson(500,1000)" 200 | ] 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "metadata": {}, 205 | "source": [ 206 | "---" 207 | ] 208 | }, 209 | { 210 | "cell_type": "markdown", 211 | "metadata": {}, 212 | "source": [ 213 | "# Question 4: Normal Distribution\n", 214 | "Write a function to generate X samples from a normal distribution and plot the histogram." 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": 21, 220 | "metadata": {}, 221 | "outputs": [], 222 | "source": [ 223 | "import numpy as np \n", 224 | "\n", 225 | "np.random.seed(123)\n", 226 | "\n", 227 | "def normal_func(X):\n", 228 | " \n", 229 | " norm_dist = np.random.normal(loc=10,scale=2,size=100)\n", 230 | " \n", 231 | " result = np.random.choice(norm_dist, X)\n", 232 | " \n", 233 | " return(result)" 234 | ] 235 | }, 236 | { 237 | "cell_type": "code", 238 | "execution_count": 22, 239 | "metadata": {}, 240 | "outputs": [ 241 | { 242 | "data": { 243 | "text/plain": [ 244 | "array([ 7.27305691, 6.98741057, 14.37357218, 14.17422672, 7.57495374,\n", 245 | " 10.39904815, 7.27305691, 7.41182935, 10.565957 , 12.0081078 ])" 246 | ] 247 | }, 248 | "execution_count": 22, 249 | "metadata": {}, 250 | "output_type": "execute_result" 251 | } 252 | ], 253 | "source": [ 254 | "X_samples = normal_func(10)\n", 255 | "X_samples" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": 23, 261 | "metadata": {}, 262 | "outputs": [ 263 | { 264 | "data": { 265 | "text/plain": [ 266 | "(array([5., 0., 0., 0., 2., 0., 1., 0., 0., 2.]),\n", 267 | " array([ 6.98741057, 7.72602673, 8.46464289, 9.20325905, 9.94187521,\n", 268 | " 10.68049138, 11.41910754, 12.1577237 , 12.89633986, 13.63495602,\n", 269 | " 14.37357218]),\n", 270 | " )" 271 | ] 272 | }, 273 | "execution_count": 23, 274 | "metadata": {}, 275 | "output_type": "execute_result" 276 | }, 277 | { 278 | "data": { 279 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAALgElEQVR4nO3cfajdBR3H8c+nzfIhyXRH8+l2JUSS4VMXiwQhtZgPzOwBFBUh6/6TpFHYRAgkAsOo/glqpEzKBwqVSvEJy0xIa7Npm9M0m8+5mZSKYE4//XHO3N25Z7tn8/zO77v5fsHlPpxz7/2w3b332+/8zpxEAIC63tP2AADA1hFqACiOUANAcYQaAIoj1ABQ3PwmvuiCBQsyOTnZxJcGgJ3SihUrXkzSGXRbI6GenJzU8uXLm/jSALBTsv3klm7j1AcAFEeoAaA4Qg0AxRFqACiOUANAcYQaAIob6vI822slvSLpTUkbkkw1OQoAsMm2XEf9qSQvNrYEADAQpz4AoLhhj6gj6Q7bkfTTJEv772B7WtK0JE1MTGz3oMklt2z3574Tay8/tZXvCwBzGfaI+rgkx0g6WdJXbR/ff4ckS5NMJZnqdAY+XR0AsB2GCnWS53qv10m6SdKxTY4CAGwyZ6ht72F7z41vS/qMpFVNDwMAdA1zjno/STfZ3nj/a5Pc1ugqAMDb5gx1kickHTmGLQCAAbg8DwCKI9QAUByhBoDiCDUAFEeoAaA4Qg0AxRFqACiOUANAcYQaAIoj1ABQHKEGgOIINQAUR6gBoDhCDQDFEWoAKI5QA0BxhBoAiiPUAFAcoQaA4gg1ABRHqAGgOEINAMURagAojlADQHGEGgCKI9QAUByhBoDiCDUAFEeoAaA4Qg0AxRFqACiOUANAcUOH2vY823+1fXOTgwAAm9uWI+oLJa1paggAYLChQm37IEmnSvpZs3MAAP2GPaL+kaSLJb21pTvYnra93Pby9evXj2QcAGCIUNs+TdK6JCu2dr8kS5NMJZnqdDojGwgA73bDHFEfJ2mx7bWSrpd0gu1fNLoKAPC2OUOd5JIkByWZlHSmpN8lOafxZQAASVxHDQDlzd+WOye5W9LdjSwBAAzEETUAFEeoAaA4Qg0AxRFqACiOUANAcYQaAIoj1ABQHKEGgOIINQAUR6gBoDhCDQDFEWoAKI5QA0BxhBoAiiPUAFAcoQaA4gg1ABRHqAGgOEINAMURagAojlADQHGEGgCKI9QAUByhBoDiCDUAFEeoAaA4Qg0AxRFqACiOUANAcYQaAIoj1ABQHKEGgOLmDLXtXW3/2faDtlfbvmwcwwAAXfOHuM/rkk5I8qrtXSTda/vWJPc1vA0AoCFCnSSSXu29u0vvJU2OAgBsMtQ5atvzbK+UtE7SnUnub3YWAGCjoUKd5M0kR0k6SNKxthf238f2tO3ltpevX79+1DsB4F1rm676SPIfSXdLWjTgtqVJppJMdTqdEc0DAAxz1UfH9l69t3eTdJKkR5oeBgDoGuaqj/0lXW17nrph/2WSm5udBQDYaJirPh6SdPQYtgAABuCZiQBQHKEGgOIINQAUR6gBoDhCDQDFEWoAKI5QA0BxhBoAiiPUAFAcoQaA4gg1ABRHqAGgOEINAMURagAojlADQHGEGgCKI9QAUByhBoDiCDUAFEeoAaA4Qg0AxRFqACiOUANAcYQaAIoj1ABQHKEGgOIINQAUR6gBoDhCDQDFEWoAKI5QA0BxhBoAiiPUAFDcnKG2fbDt39teY3u17QvHMQwA0DV/iPtskPSNJA/Y3lPSCtt3Jnm44W0AAA1xRJ3k+SQP9N5+RdIaSQc2PQwA0DXMEfXbbE9KOlrS/QNum5Y0LUkTExMjmIad2eSSW1r5vmsvP7WV74vx2tl+voZ+MNH2+yXdIOmiJC/3355kaZKpJFOdTmeUGwHgXW2oUNveRd1IX5PkxmYnAQBmGuaqD0u6UtKaJD9ofhIAYKZhjqiPk3SupBNsr+y9nNLwLgBAz5wPJia5V5LHsAUAMADPTASA4gg1ABRHqAGgOEINAMURagAojlADQHGEGgCKI9QAUByhBoDiCDUAFEeoAaA4Qg0AxRFqACiOUANAcYQaAIoj1ABQHKEGgOIINQAUR6gBoDhCDQDFEWoAKI5QA0BxhBoAiiPUAFAcoQaA4gg1ABRHqAGgOEINAMURagAojlADQHGEGgCKI9QAUNycobZ9le11tleNYxAAYHPDHFEvk7So4R0AgC2YM9RJ7pH00hi2AAAGmD+qL2R7WtK0JE1MTIzqywI7hcklt7T2vddefmpr3xujMbIHE5MsTTKVZKrT6YzqywLAux5XfQBAcYQaAIob5vK86yT9SdJhtp+xfX7zswAAG835YGKSs8YxBAAwGKc+AKA4Qg0AxRFqACiOUANAcYQaAIoj1ABQHKEGgOIINQAUR6gBoDhCDQDFEWoAKI5QA0BxhBoAiiPUAFAcoQaA4gg1ABRHqAGgOEINAMURagAojlADQHGEGgCKI9QAUByhBoDiCDUAFEeoAaA4Qg0AxRFqACiOUANAcYQaAIoj1ABQHKEGgOIINQAUR6gBoLihQm17ke1HbT9ue0nTowAAm8wZatvzJP1Y0smSDpd0lu3Dmx4GAOga5oj6WEmPJ3kiyf8kXS/p9GZnAQA2cpKt38H+gqRFSb7ce/9cSR9PckHf/aYlTffePUzSo9u5aYGkF7fzc8dpR9jJxtFg4+jsCDvb2vjhJJ1BN8wf4pM94GOz6p5kqaSl2zhs9jezlyeZeqdfp2k7wk42jgYbR2dH2Flx4zCnPp6RdPCM9w+S9FwzcwAA/YYJ9V8kHWr7ENvvlXSmpN80OwsAsNGcpz6SbLB9gaTbJc2TdFWS1Q1uesenT8ZkR9jJxtFg4+jsCDvLbZzzwUQAQLt4ZiIAFEeoAaC4MqG2fZjtlTNeXrZ9Udu7+tn+uu3VtlfZvs72rm1v6mf7wt6+1ZV+DW1fZXud7VUzPra37TttP9Z7/cGCG7/Y+7V8y3brl21tYeMVth+x/ZDtm2zvVXDjd3r7Vtq+w/YBbW7sbZq1c8Zt37Qd2wva2DZTmVAneTTJUUmOkvQxSa9JuqnlWZuxfaCkr0maSrJQ3QdXz2x31eZsL5T0FXWfUXqkpNNsH9ruqrctk7So72NLJN2V5FBJd/Xeb9Myzd64StLnJN0z9jWDLdPsjXdKWpjkCEl/l3TJuEf1WabZG69IckTvz/jNkr499lWzLdPsnbJ9sKRPS3pq3IMGKRPqPidK+keSJ9seMsB8SbvZni9pd9W7pvyjku5L8lqSDZL+IOmMljdJkpLcI+mlvg+fLunq3ttXS/rsWEf1GbQxyZok2/tM25HbwsY7er/fknSfus93aM0WNr484909NOCJc+O2hZ9JSfqhpItVYKNUN9RnSrqu7RH9kjwr6fvq/i37vKT/Jrmj3VWzrJJ0vO19bO8u6RRt/oSlavZL8rwk9V7v2/KencGXJN3a9ohBbH/X9tOSzlaNI+pZbC+W9GySB9veslG5UPeeVLNY0q/a3tKvd/70dEmHSDpA0h62z2l31eaSrJH0PXX/KXybpAclbdjqJ2GnYftSdX+/r2l7yyBJLk1ysLr7Lpjr/uPWO7i5VMX+EikXanX/O9UHkrzQ9pABTpL0zyTrk7wh6UZJn2x50yxJrkxyTJLj1f1n3WNtb9qKF2zvL0m91+ta3rPDsn2epNMknZ36T5C4VtLn2x4xwEfUPRB70PZadU8hPWD7Q22Oqhjqs1TwtEfPU5I+YXt321b3XPqaljfNYnvf3usJdR8Eq/rrKXX/O4Lzem+fJ+nXLW7ZYdleJOlbkhYnea3tPYP0Pai9WNIjbW3ZkiR/S7Jvkskkk+r+X0fHJPlX28PKvKj74Ny/JX2g7S1b2XiZuj9gqyT9XNL72t40YOMfJT2s7mmPE9veM2PXdeqe239D3T8A50vaR92rPR7rvd674MYzem+/LukFSbcX3Pi4pKclrey9/KTgxht6f24ekvRbSQdW/Jnsu32tpAVt7+Qp5ABQXMVTHwCAGQg1ABRHqAGgOEINAMURagAojlADQHGEGgCK+z8o/DIzg3EuYQAAAABJRU5ErkJggg==\n", 280 | "text/plain": [ 281 | "
" 282 | ] 283 | }, 284 | "metadata": { 285 | "needs_background": "light" 286 | }, 287 | "output_type": "display_data" 288 | } 289 | ], 290 | "source": [ 291 | "import matplotlib.pyplot as plt\n", 292 | "\n", 293 | "plt.hist(X_samples)" 294 | ] 295 | }, 296 | { 297 | "cell_type": "markdown", 298 | "metadata": {}, 299 | "source": [ 300 | "---" 301 | ] 302 | } 303 | ], 304 | "metadata": { 305 | "kernelspec": { 306 | "display_name": "Python 3", 307 | "language": "python", 308 | "name": "python3" 309 | }, 310 | "language_info": { 311 | "codemirror_mode": { 312 | "name": "ipython", 313 | "version": 3 314 | }, 315 | "file_extension": ".py", 316 | "mimetype": "text/x-python", 317 | "name": "python", 318 | "nbconvert_exporter": "python", 319 | "pygments_lexer": "ipython3", 320 | "version": "3.7.4" 321 | } 322 | }, 323 | "nbformat": 4, 324 | "nbformat_minor": 2 325 | } 326 | -------------------------------------------------------------------------------- /3. A Practical Guide To AB Tests.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Part 1: Power Analysis" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 2, 13 | "metadata": {}, 14 | "outputs": [ 15 | { 16 | "name": "stdout", 17 | "output_type": "stream", 18 | "text": [ 19 | "Sample Size: 1571.000\n" 20 | ] 21 | } 22 | ], 23 | "source": [ 24 | "from statsmodels.stats.power import TTestIndPower\n", 25 | "\n", 26 | "# parameters for power analysis \n", 27 | "# effect: standardized effect size, difference between the two means divided by the standard deviation.\n", 28 | "# effect_size has to be positive.\n", 29 | "\n", 30 | "effect = 0.1\n", 31 | "alpha = 0.05\n", 32 | "power = 0.8\n", 33 | "\n", 34 | "# perform power analysis \n", 35 | "analysis = TTestIndPower()\n", 36 | "result = analysis.solve_power(effect, power = power,nobs1= None, ratio = 1.0, alpha = alpha)\n", 37 | "print('Sample Size: %.3f' % round(result))" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "# Part 2: Data Generation Process Through Statistical Simulation\n", 45 | "Variables to be simulated: \n", 46 | "1. userid\n", 47 | "2. version\n", 48 | "3. minutes of plays \n", 49 | "4. user engagement after 1 day (metric_1)\n", 50 | "5. user engagement after 7 days (metric_2)" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 3, 56 | "metadata": {}, 57 | "outputs": [], 58 | "source": [ 59 | "# variable 1: userid\n", 60 | "user_id_control = list(range(1,1601))# 1600 control\n", 61 | "user_id_treatment = list(range(1601,3350))# 1749 treated" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "----" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": 4, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "#variable 2: version \n", 78 | "import numpy as np\n", 79 | "control_status = ['control']*1600\n", 80 | "treatment_status = ['treatment']*1749 " 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "----" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": 5, 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [ 96 | "# variable 3: minutes of plays, which follows a normal distribution with a μ of 30 minutes and σ² of 10 (central limit theorem)\n", 97 | "\n", 98 | "# for control group\n", 99 | "\n", 100 | "μ_1 = 30\n", 101 | "\n", 102 | "σ_squared_1 = 10\n", 103 | "\n", 104 | "np.random.seed(123)\n", 105 | "\n", 106 | "minutes_control = np.random.normal(loc = μ_1, scale = σ_squared_1, size = 1600)" 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 6, 112 | "metadata": {}, 113 | "outputs": [], 114 | "source": [ 115 | "# for treatment group, which increases the user engagement by \n", 116 | "# according to the formula (μ_1 - μ_2)/σ_squared = 0.1, we obtain μ_2 = 31\n", 117 | "\n", 118 | "μ_2 = 31\n", 119 | "\n", 120 | "σ_squared_2 = 10\n", 121 | "\n", 122 | "np.random.seed(123)\n", 123 | "\n", 124 | "minutes_treat = np.random.normal(loc = μ_2, scale = σ_squared_2, size = 1749)" 125 | ] 126 | }, 127 | { 128 | "cell_type": "markdown", 129 | "metadata": {}, 130 | "source": [ 131 | "----" 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": 7, 137 | "metadata": {}, 138 | "outputs": [], 139 | "source": [ 140 | "# variable 4 user engagement after 1 day (metric_1)\n", 141 | "# after day 1, treatment performs better than control --> to simulate novelty effect\n", 142 | "\n", 143 | "Active_status = [True,False]\n", 144 | "\n", 145 | "# control \n", 146 | "day_1_control = np.random.choice(Active_status, 1600, p=[0.3,0.7])\n", 147 | "\n", 148 | "# treatment\n", 149 | "day_1_treatment = np.random.choice(Active_status, 1749, p=[0.35,0.65])" 150 | ] 151 | }, 152 | { 153 | "cell_type": "markdown", 154 | "metadata": {}, 155 | "source": [ 156 | "----" 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": 8, 162 | "metadata": {}, 163 | "outputs": [], 164 | "source": [ 165 | "# variable 5 user engagement after 7 day (metric_2)\n", 166 | "# after day 7, control > treatment --> the novelty effect diminishes & performance reversed\n", 167 | "\n", 168 | "Active_status = [True,False]\n", 169 | "\n", 170 | "# control \n", 171 | "day_7_control = np.random.choice(Active_status, 1600, p=[0.35,0.65])\n", 172 | "\n", 173 | "# treatment\n", 174 | "day_7_treatment = np.random.choice(Active_status, 1749, p=[0.25,0.75])" 175 | ] 176 | }, 177 | { 178 | "cell_type": "markdown", 179 | "metadata": {}, 180 | "source": [ 181 | "----" 182 | ] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "metadata": {}, 187 | "source": [ 188 | "# construct the control group " 189 | ] 190 | }, 191 | { 192 | "cell_type": "markdown", 193 | "metadata": {}, 194 | "source": [ 195 | "1. user_id_control; \n", 196 | "2. control_status\n", 197 | "3. minutes_control\n", 198 | "4. day_1_control\n", 199 | "5. day_7_control" 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": 10, 205 | "metadata": {}, 206 | "outputs": [ 207 | { 208 | "data": { 209 | "text/html": [ 210 | "
\n", 211 | "\n", 224 | "\n", 225 | " \n", 226 | " \n", 227 | " \n", 228 | " \n", 229 | " \n", 230 | " \n", 231 | " \n", 232 | " \n", 233 | " \n", 234 | " \n", 235 | " \n", 236 | " \n", 237 | " \n", 238 | " \n", 239 | " \n", 240 | " \n", 241 | " \n", 242 | " \n", 243 | " \n", 244 | " \n", 245 | " \n", 246 | " \n", 247 | " \n", 248 | " \n", 249 | " \n", 250 | " \n", 251 | " \n", 252 | " \n", 253 | " \n", 254 | " \n", 255 | " \n", 256 | " \n", 257 | " \n", 258 | " \n", 259 | " \n", 260 | " \n", 261 | " \n", 262 | " \n", 263 | " \n", 264 | " \n", 265 | " \n", 266 | " \n", 267 | " \n", 268 | " \n", 269 | " \n", 270 | " \n", 271 | " \n", 272 | " \n", 273 | " \n", 274 | " \n", 275 | " \n", 276 | " \n", 277 | " \n", 278 | " \n", 279 | " \n", 280 | " \n", 281 | " \n", 282 | " \n", 283 | " \n", 284 | " \n", 285 | " \n", 286 | " \n", 287 | " \n", 288 | " \n", 289 | " \n", 290 | " \n", 291 | " \n", 292 | " \n", 293 | " \n", 294 | " \n", 295 | " \n", 296 | " \n", 297 | " \n", 298 | " \n", 299 | " \n", 300 | " \n", 301 | " \n", 302 | " \n", 303 | " \n", 304 | " \n", 305 | " \n", 306 | " \n", 307 | " \n", 308 | " \n", 309 | " \n", 310 | " \n", 311 | " \n", 312 | " \n", 313 | " \n", 314 | " \n", 315 | " \n", 316 | " \n", 317 | " \n", 318 | " \n", 319 | " \n", 320 | " \n", 321 | " \n", 322 | " \n", 323 | " \n", 324 | " \n", 325 | "
user_idversionminutes_playday_1_activeday_7_active
01control19.143694FalseFalse
12control39.973454FalseFalse
23control32.829785FalseFalse
34control14.937053FalseFalse
45control24.213997TrueTrue
..................
15951596control27.154466TrueTrue
15961597control46.414042TrueTrue
15971598control41.523560FalseFalse
15981599control23.981909FalseFalse
15991600control17.843379FalseFalse
\n", 326 | "

1600 rows × 5 columns

\n", 327 | "
" 328 | ], 329 | "text/plain": [ 330 | " user_id version minutes_play day_1_active day_7_active\n", 331 | "0 1 control 19.143694 False False\n", 332 | "1 2 control 39.973454 False False\n", 333 | "2 3 control 32.829785 False False\n", 334 | "3 4 control 14.937053 False False\n", 335 | "4 5 control 24.213997 True True\n", 336 | "... ... ... ... ... ...\n", 337 | "1595 1596 control 27.154466 True True\n", 338 | "1596 1597 control 46.414042 True True\n", 339 | "1597 1598 control 41.523560 False False\n", 340 | "1598 1599 control 23.981909 False False\n", 341 | "1599 1600 control 17.843379 False False\n", 342 | "\n", 343 | "[1600 rows x 5 columns]" 344 | ] 345 | }, 346 | "execution_count": 10, 347 | "metadata": {}, 348 | "output_type": "execute_result" 349 | } 350 | ], 351 | "source": [ 352 | "# control data\n", 353 | "import pandas as pd\n", 354 | "raw_control = {'user_id':user_id_control,\n", 355 | " 'version':control_status,\n", 356 | " 'minutes_play':minutes_control,\n", 357 | " 'day_1_active':day_1_control,\n", 358 | " 'day_7_active':day_1_control\n", 359 | " }\n", 360 | "\n", 361 | "control_group = pd.DataFrame(data = raw_control)\n", 362 | "control_group" 363 | ] 364 | }, 365 | { 366 | "cell_type": "markdown", 367 | "metadata": {}, 368 | "source": [ 369 | "1. user_id_treatment\n", 370 | "2. treatment_status \n", 371 | "3. minutes_treat\n", 372 | "4. day_1_treatment\n", 373 | "5. day_7_treatment" 374 | ] 375 | }, 376 | { 377 | "cell_type": "code", 378 | "execution_count": 11, 379 | "metadata": {}, 380 | "outputs": [ 381 | { 382 | "data": { 383 | "text/html": [ 384 | "
\n", 385 | "\n", 398 | "\n", 399 | " \n", 400 | " \n", 401 | " \n", 402 | " \n", 403 | " \n", 404 | " \n", 405 | " \n", 406 | " \n", 407 | " \n", 408 | " \n", 409 | " \n", 410 | " \n", 411 | " \n", 412 | " \n", 413 | " \n", 414 | " \n", 415 | " \n", 416 | " \n", 417 | " \n", 418 | " \n", 419 | " \n", 420 | " \n", 421 | " \n", 422 | " \n", 423 | " \n", 424 | " \n", 425 | " \n", 426 | " \n", 427 | " \n", 428 | " \n", 429 | " \n", 430 | " \n", 431 | " \n", 432 | " \n", 433 | " \n", 434 | " \n", 435 | " \n", 436 | " \n", 437 | " \n", 438 | " \n", 439 | " \n", 440 | " \n", 441 | " \n", 442 | " \n", 443 | " \n", 444 | " \n", 445 | " \n", 446 | " \n", 447 | " \n", 448 | " \n", 449 | " \n", 450 | " \n", 451 | " \n", 452 | " \n", 453 | " \n", 454 | " \n", 455 | " \n", 456 | " \n", 457 | " \n", 458 | " \n", 459 | " \n", 460 | " \n", 461 | " \n", 462 | " \n", 463 | " \n", 464 | " \n", 465 | " \n", 466 | " \n", 467 | " \n", 468 | " \n", 469 | " \n", 470 | " \n", 471 | " \n", 472 | " \n", 473 | " \n", 474 | " \n", 475 | " \n", 476 | " \n", 477 | " \n", 478 | " \n", 479 | " \n", 480 | " \n", 481 | " \n", 482 | " \n", 483 | " \n", 484 | " \n", 485 | " \n", 486 | " \n", 487 | " \n", 488 | " \n", 489 | " \n", 490 | " \n", 491 | " \n", 492 | " \n", 493 | " \n", 494 | " \n", 495 | " \n", 496 | " \n", 497 | " \n", 498 | " \n", 499 | "
user_idversionminutes_playday_1_activeday_7_active
01601treatment25.515817FalseFalse
11602treatment41.653829FalseFalse
21603treatment25.760776FalseFalse
31604treatment36.459880TrueFalse
41605treatment25.341199TrueFalse
..................
17443345treatment24.098838TrueFalse
17453346treatment29.683718TrueFalse
17463347treatment34.013900TrueFalse
17473348treatment44.909702FalseFalse
17483349treatment51.464873FalseFalse
\n", 500 | "

1749 rows × 5 columns

\n", 501 | "
" 502 | ], 503 | "text/plain": [ 504 | " user_id version minutes_play day_1_active day_7_active\n", 505 | "0 1601 treatment 25.515817 False False\n", 506 | "1 1602 treatment 41.653829 False False\n", 507 | "2 1603 treatment 25.760776 False False\n", 508 | "3 1604 treatment 36.459880 True False\n", 509 | "4 1605 treatment 25.341199 True False\n", 510 | "... ... ... ... ... ...\n", 511 | "1744 3345 treatment 24.098838 True False\n", 512 | "1745 3346 treatment 29.683718 True False\n", 513 | "1746 3347 treatment 34.013900 True False\n", 514 | "1747 3348 treatment 44.909702 False False\n", 515 | "1748 3349 treatment 51.464873 False False\n", 516 | "\n", 517 | "[1749 rows x 5 columns]" 518 | ] 519 | }, 520 | "execution_count": 11, 521 | "metadata": {}, 522 | "output_type": "execute_result" 523 | } 524 | ], 525 | "source": [ 526 | "# treatment data \n", 527 | "raw_treatment = {'user_id':user_id_treatment,\n", 528 | " 'version':treatment_status,\n", 529 | " 'minutes_play':minutes_treat,\n", 530 | " 'day_1_active':day_1_treatment,\n", 531 | " 'day_7_active':day_7_treatment\n", 532 | " }\n", 533 | "\n", 534 | "treatment_group = pd.DataFrame(data = raw_treatment)\n", 535 | "treatment_group" 536 | ] 537 | }, 538 | { 539 | "cell_type": "code", 540 | "execution_count": 12, 541 | "metadata": {}, 542 | "outputs": [], 543 | "source": [ 544 | "# combine these two datasets\n", 545 | "two_datasets = control_group.append(treatment_group)\n", 546 | "\n", 547 | "# randomize the orders using df.sample(frac=1)\n", 548 | "# The frac keyword argument: specifies the fraction of rows to return in the random sample\n", 549 | "# so frac=1 means return all rows (in random order).\n", 550 | "final_data = two_datasets.sample(frac=1)" 551 | ] 552 | }, 553 | { 554 | "cell_type": "code", 555 | "execution_count": 13, 556 | "metadata": {}, 557 | "outputs": [ 558 | { 559 | "name": "stdout", 560 | "output_type": "stream", 561 | "text": [ 562 | "\n", 563 | "Int64Index: 3349 entries, 1558 to 345\n", 564 | "Data columns (total 5 columns):\n", 565 | "user_id 3349 non-null int64\n", 566 | "version 3349 non-null object\n", 567 | "minutes_play 3349 non-null float64\n", 568 | "day_1_active 3349 non-null bool\n", 569 | "day_7_active 3349 non-null bool\n", 570 | "dtypes: bool(2), float64(1), int64(1), object(1)\n", 571 | "memory usage: 111.2+ KB\n" 572 | ] 573 | } 574 | ], 575 | "source": [ 576 | "final_data.info()" 577 | ] 578 | }, 579 | { 580 | "cell_type": "code", 581 | "execution_count": 29, 582 | "metadata": {}, 583 | "outputs": [ 584 | { 585 | "data": { 586 | "text/html": [ 587 | "
\n", 588 | "\n", 601 | "\n", 602 | " \n", 603 | " \n", 604 | " \n", 605 | " \n", 606 | " \n", 607 | " \n", 608 | " \n", 609 | " \n", 610 | " \n", 611 | " \n", 612 | " \n", 613 | " \n", 614 | " \n", 615 | " \n", 616 | " \n", 617 | " \n", 618 | " \n", 619 | " \n", 620 | " \n", 621 | " \n", 622 | " \n", 623 | " \n", 624 | " \n", 625 | " \n", 626 | " \n", 627 | " \n", 628 | " \n", 629 | " \n", 630 | " \n", 631 | " \n", 632 | " \n", 633 | " \n", 634 | " \n", 635 | " \n", 636 | " \n", 637 | " \n", 638 | " \n", 639 | " \n", 640 | " \n", 641 | " \n", 642 | " \n", 643 | " \n", 644 | " \n", 645 | " \n", 646 | " \n", 647 | " \n", 648 | " \n", 649 | " \n", 650 | " \n", 651 | " \n", 652 | " \n", 653 | " \n", 654 | " \n", 655 | " \n", 656 | " \n", 657 | " \n", 658 | " \n", 659 | " \n", 660 | "
user_idversionminutes_playday_1_activeday_7_activeminutes_play_integers
15583159treatment26.156152FalseTrue26.0
12231224control24.313143FalseFalse24.0
392393control29.013153TrueTrue29.0
808809control27.797075TrueTrue28.0
4852086treatment32.152395FalseFalse32.0
\n", 661 | "
" 662 | ], 663 | "text/plain": [ 664 | " user_id version minutes_play day_1_active day_7_active \\\n", 665 | "1558 3159 treatment 26.156152 False True \n", 666 | "1223 1224 control 24.313143 False False \n", 667 | "392 393 control 29.013153 True True \n", 668 | "808 809 control 27.797075 True True \n", 669 | "485 2086 treatment 32.152395 False False \n", 670 | "\n", 671 | " minutes_play_integers \n", 672 | "1558 26.0 \n", 673 | "1223 24.0 \n", 674 | "392 29.0 \n", 675 | "808 28.0 \n", 676 | "485 32.0 " 677 | ] 678 | }, 679 | "execution_count": 29, 680 | "metadata": {}, 681 | "output_type": "execute_result" 682 | } 683 | ], 684 | "source": [ 685 | "final_data.head()" 686 | ] 687 | }, 688 | { 689 | "cell_type": "markdown", 690 | "metadata": {}, 691 | "source": [ 692 | "# Part 3: After-Test Data Analysis" 693 | ] 694 | }, 695 | { 696 | "cell_type": "markdown", 697 | "metadata": {}, 698 | "source": [ 699 | "### 3.1 Count the Number of Users in Each Version " 700 | ] 701 | }, 702 | { 703 | "cell_type": "code", 704 | "execution_count": 14, 705 | "metadata": {}, 706 | "outputs": [ 707 | { 708 | "data": { 709 | "text/plain": [ 710 | "version\n", 711 | "control 1600\n", 712 | "treatment 1749\n", 713 | "Name: user_id, dtype: int64" 714 | ] 715 | }, 716 | "execution_count": 14, 717 | "metadata": {}, 718 | "output_type": "execute_result" 719 | } 720 | ], 721 | "source": [ 722 | "# calculate the number of users in each version\n", 723 | "final_data.groupby('version')['user_id'].count()" 724 | ] 725 | }, 726 | { 727 | "cell_type": "markdown", 728 | "metadata": {}, 729 | "source": [ 730 | "# the assignment process looks suspicious as more people assigned to the treatment than the control. " 731 | ] 732 | }, 733 | { 734 | "cell_type": "markdown", 735 | "metadata": {}, 736 | "source": [ 737 | "### 3.2 Formally Test for Sample Ratio Mismatch\n", 738 | "- Chi-Square test" 739 | ] 740 | }, 741 | { 742 | "cell_type": "code", 743 | "execution_count": 30, 744 | "metadata": {}, 745 | "outputs": [ 746 | { 747 | "data": { 748 | "text/plain": [ 749 | "Power_divergenceResult(statistic=6.627462686567164, pvalue=0.010041820594939122)" 750 | ] 751 | }, 752 | "execution_count": 30, 753 | "metadata": {}, 754 | "output_type": "execute_result" 755 | } 756 | ], 757 | "source": [ 758 | "from scipy.stats import chisquare \n", 759 | "chisquare([1600,1749],f_exp = [1675,1675])" 760 | ] 761 | }, 762 | { 763 | "cell_type": "markdown", 764 | "metadata": {}, 765 | "source": [ 766 | "Typically, we set the alpha level at 0.001 to test Sample Ratio Mismatch. Since the p value is 0.01, we have to reject the null hypothesis and conclude no evidence of SRM.\n", 767 | "In other words, the treatment assignment works as expected." 768 | ] 769 | }, 770 | { 771 | "cell_type": "code", 772 | "execution_count": 31, 773 | "metadata": {}, 774 | "outputs": [ 775 | { 776 | "data": { 777 | "text/html": [ 778 | "
\n", 779 | "\n", 792 | "\n", 793 | " \n", 794 | " \n", 795 | " \n", 796 | " \n", 797 | " \n", 798 | " \n", 799 | " \n", 800 | " \n", 801 | " \n", 802 | " \n", 803 | " \n", 804 | " \n", 805 | " \n", 806 | " \n", 807 | " \n", 808 | " \n", 809 | " \n", 810 | " \n", 811 | " \n", 812 | " \n", 813 | " \n", 814 | " \n", 815 | " \n", 816 | " \n", 817 | " \n", 818 | " \n", 819 | " \n", 820 | " \n", 821 | " \n", 822 | " \n", 823 | " \n", 824 | " \n", 825 | " \n", 826 | " \n", 827 | " \n", 828 | " \n", 829 | " \n", 830 | " \n", 831 | " \n", 832 | " \n", 833 | " \n", 834 | " \n", 835 | " \n", 836 | " \n", 837 | " \n", 838 | " \n", 839 | " \n", 840 | " \n", 841 | " \n", 842 | " \n", 843 | " \n", 844 | " \n", 845 | " \n", 846 | " \n", 847 | " \n", 848 | " \n", 849 | " \n", 850 | " \n", 851 | "
user_idversionminutes_playday_1_activeday_7_activeminutes_play_integers
15583159treatment26.156152FalseTrue26.0
12231224control24.313143FalseFalse24.0
392393control29.013153TrueTrue29.0
808809control27.797075TrueTrue28.0
4852086treatment32.152395FalseFalse32.0
\n", 852 | "
" 853 | ], 854 | "text/plain": [ 855 | " user_id version minutes_play day_1_active day_7_active \\\n", 856 | "1558 3159 treatment 26.156152 False True \n", 857 | "1223 1224 control 24.313143 False False \n", 858 | "392 393 control 29.013153 True True \n", 859 | "808 809 control 27.797075 True True \n", 860 | "485 2086 treatment 32.152395 False False \n", 861 | "\n", 862 | " minutes_play_integers \n", 863 | "1558 26.0 \n", 864 | "1223 24.0 \n", 865 | "392 29.0 \n", 866 | "808 28.0 \n", 867 | "485 32.0 " 868 | ] 869 | }, 870 | "execution_count": 31, 871 | "metadata": {}, 872 | "output_type": "execute_result" 873 | } 874 | ], 875 | "source": [ 876 | "final_data.head()" 877 | ] 878 | }, 879 | { 880 | "cell_type": "markdown", 881 | "metadata": {}, 882 | "source": [ 883 | "### 3.3 Plot the Distribution of Video Played for Each Group " 884 | ] 885 | }, 886 | { 887 | "cell_type": "code", 888 | "execution_count": 39, 889 | "metadata": {}, 890 | "outputs": [ 891 | { 892 | "data": { 893 | "text/plain": [ 894 | "Text(0, 0.5, 'User Count')" 895 | ] 896 | }, 897 | "execution_count": 39, 898 | "metadata": {}, 899 | "output_type": "execute_result" 900 | }, 901 | { 902 | "data": { 903 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEGCAYAAACD7ClEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAYB0lEQVR4nO3debRlZXnn8e/PYkYQkYtBoSwwaprQCnbhLFFQQ5SIdogBTcSxuu0lTh0NNGs5dFaycFhxaGxNiYpRggOiEowyCIgjUmAxiTjEEkvRKmIrKhgBn/5j72sdLnc4dW6ds2/V/n7Wuuue8+599vuc9wzP2dOzU1VIkvrpHl0HIEnqjklAknrMJCBJPWYSkKQeMwlIUo9t13UAw9hrr71qxYoVXYchSVuVK6644uaqmppvnq0iCaxYsYI1a9Z0HYYkbVWSfH+hedwcJEk9ZhKQpB4zCUhSj5kEJKnHTAKS1GMmAUnqsbElgSTvS7IhybWzTPvrJJVkr3H1L0la2DjXBE4HjpzZmGQ/4MnAjWPsW5I0hLElgaq6FPjpLJPeCrwG8EIGktSxiZ4xnOTpwA+r6qokC827ClgFsHz58pH7XHHip0d+7GKtO+VpnfUtScOY2I7hJLsAJwOvHWb+qlpdVSurauXU1LylLyRJI5rk0UEPBPYHrkqyDtgXuDLJ700wBknSgIltDqqqa4C9p++3iWBlVd08qRgkSXc1zkNEzwS+AjwkyfokLxxXX5Kk0YxtTaCqjltg+opx9S1JGo5nDEtSj5kEJKnHTAKS1GMmAUnqMZOAJPWYSUCSeswkIEk9ZhKQpB4zCUhSj5kEJKnHTAKS1GMmAUnqMZOAJPWYSUCSeswkIEk9ZhKQpB4zCUhSj5kEJKnHTAKS1GMmAUnqsbElgSTvS7IhybUDbW9O8s0kVyf5RJI9xtW/JGlh41wTOB04ckbbBcBBVfVQ4FvASWPsX5K0gLElgaq6FPjpjLbzq+qO9u5XgX3H1b8kaWFd7hN4AfCZuSYmWZVkTZI1GzdunGBYktQfnSSBJCcDdwBnzDVPVa2uqpVVtXJqampywUlSj2w36Q6THA8cBRxRVTXp/iVJm0w0CSQ5Evgb4I+q6tZJ9i1JurtxHiJ6JvAV4CFJ1id5IXAqsBtwQZK1Sd49rv4lSQsb25pAVR03S/N7x9WfJGnzecawJPWYSUCSeswkIEk9ZhKQpB4zCUhSj5kEJKnHTAKS1GMmAUnqMZOAJPWYSUCSeswkIEk9ZhKQpB4zCUhSj5kEJKnHTAKS1GMmAUnqMZOAJPWYSUCSeswkIEk9ZhKQpB4zCUhSj40tCSR5X5INSa4daNszyQVJvt3+v/e4+pckLWycawKnA0fOaDsR+FxVPQj4XHtfktSRsSWBqroU+OmM5qOBD7S3PwA8Y1z9S5IWNul9AvetqpsA2v97zzVjklVJ1iRZs3HjxokFKEl9smR3DFfV6qpaWVUrp6amug5HkrZJk04CP0myD0D7f8OE+5ckDZh0EjgHOL69fTzwqQn3L0kaMM5DRM8EvgI8JMn6JC8ETgGenOTbwJPb+5Kkjmw3rgVX1XFzTDpiXH1KkjbPkt0xLEkaP5OAJPWYSUCSeswkIEk9ZhKQpB4zCUhSj5kEJKnHFkwCSR47TJskaeszzJrA/xmyTZK0lZnzjOEkjwYeA0wledXApN2BZeMOTJI0fvOVjdgBuGc7z24D7bcAx4wzKEnSZMyZBKrq88Dnk5xeVd+fYEySpAkZpoDcjklWAysG56+qw8cVlCRpMoZJAh8D3g2cBtw53nAkSZM0TBK4o6reNfZIJEkTN8whov+S5H8k2SfJntN/Y49MkjR2w6wJTF8O8tUDbQUcsOXDkSRN0oJJoKr2n0QgkqTJWzAJJHnubO1V9U9bPhxJ0iQNszno0IHbO9FcI/hKwCQgSVu5YTYHnTB4P8m9gA8uptMkrwReRLNv4Rrg+VX168UsU5K0+UYpJX0r8KBRO0xyf+BlwMqqOoimDtGxoy5PkjS6YfYJ/AvNL3ZovrD/E/DRLdDvzkluB3YBfrTI5UmSRjDMPoG3DNy+A/h+Va0ftcOq+mGStwA3ArcB51fV+TPnS7IKWAWwfPnyUbuTJM1jwc1BbSG5b9JUEr038JvFdJjk3sDRwP7A/YBdk/zlLP2urqqVVbVyampqMV1KkuYwzJXFngV8Dfhz4FnAZUkWU0r6ScD3qmpjVd0OnE1z3QJJ0oQNsznoZODQqtoAkGQKuBA4a8Q+bwQelWQXms1BRwBrRlyWJGkRhjk66B7TCaD170M+blZVdRlNArmS5vDQewCrR12eJGl0w6wJfDbJecCZ7f2/AD6zmE6r6nXA6xazDEnS4g1zstirk/xX4HFAgNVV9YmxRyZJGrv5LjT/+8B9q+pLVXU2zQ5ckhyW5IFV9d1JBSlJGo/5tu2/DfjFLO23ttMkSVu5+ZLAiqq6emZjVa2hud6wJGkrN18S2GmeaTtv6UAkSZM3XxK4PMmLZzYmeSFwxfhCkiRNynxHB70C+ESS57DpS38lsAPwzHEHJkkavzmTQFX9BHhMkicCB7XNn66qiyYSmSRp7IY5T+Bi4OIJxCJJmrBhzhiWpN9ZceKnO+t73SlP66zvbdXINYAkSVu/eZNAkmVJLpxUMJKkyZo3CVTVncCt7cXlJUnbmGH2CfwauCbJBcCvphur6mVji0qSNBHDJIFPt3+SpG3MMIeIfiDJzsDyqrphAjFJkiZkmGsM/ymwFvhse//gJOeMOzBJ0vgNc4jo64FHAD8DqKq1wP5jjEmSNCHDJIE7qurnM9pqHMFIkiZrmB3D1yZ5NrAsyYOAlwFfHm9YkqRJGGZN4ATgD4H/oLnY/C00FUYlSVu5YY4OuhU4GTg5yTJg16r69WI6TbIHcBpNddICXlBVX1nMMiVJm2+Yo4P+OcnuSXYFrgNuSPLqRfb7duCzVfUHwMOA6xe5PEnSCIbZHHRgVd0CPAP4V2A58Fejdphkd+Aw4L0AVfWbqvrZqMuTJI1umB3D2yfZniYJnFpVtydZzNFBBwAbgfcneRjNVcteXlW/GpwpySpgFcDy5csX0Z00PpZV1tZumDWBfwTWAbsClyZ5AM3O4VFtBzwceFdVHUJTj+jEmTNV1eqqWllVK6emphbRnSRpLgsmgap6R1Xdv6qeWlUF3Ag8cRF9rgfWV9Vl7f2zaJKCJGnC5twclORVM5oKuBn4YlV9b9QOq+rHSX6Q5CFtLaIjgG+MujxJ0ujmWxPYbcbf7sBK4DNJjl1kvycAZyS5GjgY+PtFLk+SNII51wSq6g2ztSfZE7gQ+PConbb1h1aO+nhJ0pax2dcYrqqfAhlDLJKkCdvsJJDkcOD/jSEWSdKEzbdj+BruXi10T+BHwHPHGZQkaTLmO1nsqBn3C/j3mSd1SZK2XvPtGP7+JAORJE3eZu8TkCRtO0wCktRjJgFJ6jGTgCT12DClpLWV6bK8cVcsq9wPXb23t+X3l2sCktRjJgFJ6jGTgCT1mElAknrMJCBJPWYSkKQeMwlIUo+ZBCSpx0wCktRjJgFJ6rHOkkCSZUm+nuTcrmKQpL7rck3g5cD1HfYvSb3XSRJIsi/wNOC0LvqXJDW6WhN4G/Aa4Lcd9S9JooMkkOQoYENVXbHAfKuSrEmyZuPGjROKTpL6pYs1gccCT0+yDvgwcHiSD82cqapWV9XKqlo5NTU16RglqRcmngSq6qSq2reqVgDHAhdV1V9OOg5JkucJSFKvdXp5yaq6BLikyxgkqc9cE5CkHjMJSFKPmQQkqcdMApLUYyYBSeoxk4Ak9ZhJQJJ6zCQgST1mEpCkHuv0jGFpS1lx4qe7DkHaKrkmIEk9ZhKQpB4zCUhSj5kEJKnHTAKS1GMmAUnqMZOAJPWYSUCSeswkIEk9ZhKQpB4zCUhSj008CSTZL8nFSa5Pcl2Sl086BklSo4sCcncA/7OqrkyyG3BFkguq6hsdxCJJvTbxNYGquqmqrmxv/wK4Hrj/pOOQJHVcSjrJCuAQ4LJZpq0CVgEsX758onFtKZY3lrYNXX6W153ytLEuv7Mdw0nuCXwceEVV3TJzelWtrqqVVbVyampq8gFKUg90kgSSbE+TAM6oqrO7iEGS1M3RQQHeC1xfVf8w6f4lSZt0sSbwWOCvgMOTrG3/ntpBHJLUexPfMVxVXwQy6X4lSXfnGcOS1GMmAUnqMZOAJPWYSUCSeswkIEk9ZhKQpB4zCUhSj5kEJKnHTAKS1GOdlpKWNDpLlWtLcE1AknrMJCBJPWYSkKQeMwlIUo+ZBCSpx0wCktRjJgFJ6jGTgCT1mElAknrMJCBJPWYSkKQe6yQJJDkyyQ1JvpPkxC5ikCR1kASSLAPeCfwJcCBwXJIDJx2HJKmbNYFHAN+pqn+rqt8AHwaO7iAOSeq9LkpJ3x/4wcD99cAjZ86UZBWwqr37yyQ3bGY/ewE3jxTh+C3l2GBpx2dso1vK8RnbHPLGeScvFNsDFlp+F0kgs7TV3RqqVgOrR+4kWVNVK0d9/Dgt5dhgacdnbKNbyvEZ22i2RGxdbA5aD+w3cH9f4EcdxCFJvddFErgceFCS/ZPsABwLnNNBHJLUexPfHFRVdyR5KXAesAx4X1VdN4auRt6UNAFLOTZY2vEZ2+iWcnzGNppFx5aqu22OlyT1hGcMS1KPmQQkqce2ySSwlMpSJNkvycVJrk9yXZKXt+17Jrkgybfb//fuMMZlSb6e5Nz2/v5JLmtj+0i7A7+LuPZIclaSb7bj9+glNm6vbF/Ta5OcmWSnrsYuyfuSbEhy7UDbrGOVxjvaz8fVSR7eQWxvbl/Xq5N8IskeA9NOamO7IckfjzO2ueIbmPbXSSrJXu39zseubT+hHZ/rkrxpoH3zx66qtqk/mp3N3wUOAHYArgIO7DCefYCHt7d3A75FUy7jTcCJbfuJwBs7jPFVwD8D57b3Pwoc295+N/CSjuL6APCi9vYOwB5LZdxoTnr8HrDzwJg9r6uxAw4DHg5cO9A261gBTwU+Q3POzqOAyzqI7SnAdu3tNw7EdmD7md0R2L/9LC+bdHxt+340B7B8H9hrCY3dE4ELgR3b+3svZuwm9qGZ1B/waOC8gfsnASd1HddAPJ8CngzcAOzTtu0D3NBRPPsCnwMOB85t39w3D3xA7zKeE4xr9/ZLNjPal8q4TZ/5vifNUXbnAn/c5dgBK2Z8Wcw6VsA/AsfNNt+kYpsx7ZnAGe3tu3xe2y/hR0967Nq2s4CHAesGkkDnY0fzQ+NJs8w30thti5uDZitLcf+OYrmLJCuAQ4DLgPtW1U0A7f+9OwrrbcBrgN+29+8D/Kyq7mjvdzV+BwAbgfe3m6pOS7IrS2TcquqHwFuAG4GbgJ8DV7A0xm7aXGO11D4jL6D5dQ1LJLYkTwd+WFVXzZi0FOJ7MPD4drPj55McupjYtsUkMFRZiklLck/g48ArquqWruMBSHIUsKGqrhhsnmXWLsZvO5rV4HdV1SHAr2g2aSwJ7fb1o2lWu+8H7EpTGXemzt97s1gqrzFJTgbuAM6YbppltonGlmQX4GTgtbNNnqVt0mO3HXBvms1RrwY+miSMGNu2mASWXFmKJNvTJIAzqurstvknSfZpp+8DbOggtMcCT0+yjqaa6+E0awZ7JJk+kbCr8VsPrK+qy9r7Z9EkhaUwbgBPAr5XVRur6nbgbOAxLI2xmzbXWC2Jz0iS44GjgOdUu/1iicT2QJrkflX72dgXuDLJ7y2R+NYDZ1fjazRr8XuNGtu2mASWVFmKNkO/F7i+qv5hYNI5wPHt7eNp9hVMVFWdVFX7VtUKmnG6qKqeA1wMHNNxbD8GfpDkIW3TEcA3WALj1roReFSSXdrXeDq+zsduwFxjdQ7w3PZIl0cBP5/ebDQpSY4E/gZ4elXdOjDpHODYJDsm2R94EPC1ScZWVddU1d5VtaL9bKynObjjxyyBsQM+SfODjSQPpjlo4mZGHbtx73Dp4o9mD/63aPaOn9xxLI+jWSW7Gljb/j2VZtv754Bvt//37DjOJ7Dp6KAD2jfPd4CP0R6F0EFMBwNr2rH7JM0q8JIZN+ANwDeBa4EP0hyV0cnYAWfS7Ju4neZL64VzjRXNZoN3tp+Pa4CVHcT2HZrt19OfiXcPzH9yG9sNwJ90MXYzpq9j047hpTB2OwAfat93VwKHL2bsLBshST22LW4OkiQNySQgST1mEpCkHjMJSFKPmQQkqcdMAtu4JHcmWdtWG7wqyauSbLHXPcnzktxv4P5pSQ7cUsufpb8dk1zYPqe/mBHHmTPm3SvJxvYxs8bVPu7ULRDXE5L8vC1xcX2S1w20n7vY5S/Q9yVJ7nax8c19LdqxqCRHDLQ9s207ZpRlzlj+iiTPHuWxGp+JX15SE3dbVR0MkGRvmmqh9wJeN+wCkiyrqjvnmPw8muOVfwRQVS9aVLQLOwTYfvo5DTgbeEuSXWrTyUfHAOdU1X8A444L4AtVdVRb42jtuL/8FzLia3ENcBzNeQXQnET4u/o5i3x9VwDPpnkPaolwTaBHqmoDsAp4aXvG411+BSc5N8kT2tu/TPK/k1wGPDrJa5NcnqZ2/ur28ccAK4Ez2l/mOw/+Kk1yXJJr2se8caCfXyb5u3bN5KtJ7jsz1jS18D+Zpmb7V5M8tE1iHwIObvt74MBzuwW4FPjTgcUcS3OyDTPien6SbyX5PE3pjOk+p5J8vH2elyd57FyxLDDOv6IpJvfAwfYkj0jy5XZt4ctpz4ZO8oUkBw/M96X2+e6app785e1jjm6n75zkw208HwF2ni2OGc95wTFvfQF4RJLt09S7+n2ak7mGXmaS06fXHKbna2+eQlP4bG2aazEsS3Ndgcvb5/Lf2vn3SXJpO9+1SR4/33hrcUwCPVNV/0bzui9UfXNXmvK1j6yqLwKnVtWhVXUQzZfOUVV1Fs0Zvc+pqoOr6rbpB6fZRPRGmtPbDwYOTfKMgWV/taoeRvPF/eJZ+n8D8PWqeijwv4B/apPYi2h+cR9cVd+d8Zgzab74p/t/ME0Zh99JU0PnDTRf/k+mqcE+7e3AW6vqUODPgNPmimW+gUtyH5riXtfNmPRN4LBqCuK9Fvj7tv00mjWq6TIAO1bV1TRnf17UxvNE4M3tWsZLgFvbeP4O+C/zxdMaZsyhObv9Qpqy2Eczf8mVYZc57UQ2vXZvpTn79eft8zsUeHGacgfPpinBfTBNKee1cy5Ri2YS6KfZqg3OdCdN0btpT0xTuvYami/2P1zg8YcCl1RTYG26SuRh7bTf0NTfh+YX84pZHv84mlIMVNVFwH2S3GuBPs8FHpdkd+BZwFmzbMZ65EBcvwE+MjDtScCpSdbSfPntnmS3zYjl8Um+DpwPnFJVM5PAvYCPpblK1FvZNIYfA45KU2jwBcDpbftTgBPbeC4BdgKW04zjh9p4rqYpq7GQYcZ82odpkunv1qS2wDJn8xSaOjxracqr34em3s3lwPOTvB74z1X1i81crjaD+wR6JskBNF/wG2hK+A7+ENhp4Pavp79Ak+wE/F+aOik/aD+cg/PO2tU8026vTfVK7mT29+Fml8WtqtuSfJbmIiXHAq+ca9Y52u9BcxGO2wYbkwwbyxeq6qh5Qvxb4OKqemaaa0tc0sZ9a5ILaH55P4tmExs0Y/BnVXXDjHjmew5zGWbMaeP5WpKDaPYnfWv2pz/vMn/3vmrHbq5LbAY4oarOu9uE5DDgacAHk7y5quZd+9LoXBPokSRTNJc8PLX98K6j2b5+jyT7AY+Y46HTX/g3t9uJjxmY9guay2bOdBnwR2mO0FlGs7Px85sR7qXAc9q4nwDcXMNdh+FMmstl3hf46hxxPSHJfdpf3n8+MO184KXTdwa2048ay0z3An7Y3n7ejGmnAe8ALq+qn7Zt5wEnTCehJIfMEs9BwLz7KEZ0Es2mr1GsY9MmqqOB7dvbM98r5wEvaV8Hkjy43Q/yAJrrXLyHpgLvWK/j23euCWz7dm5Xt7en+YX2QWC6pPWXaC7heA2bKhLeTVX9LMl72vnW0ayuTzsdeHeS22gupzj9mJuSnESzTT7Av1bV5pRVfj3NVcWuBm5lU0nkhZxPc23i9w78Sh18Lje1azJfoanOeCXNdakBXga8s+1zO5ov2/++iFhmehPwgSSvAi6aEdcVSW4B3j/Q/Lc013e4uk0E62jq779rIJ61jKHUclV9ZuG55vQe4FNJvkZzlNGv2vargTuSXEXzvnk7zSakK9vntxF4Bk1F21cnuR34JfDcRcSiBVhFVFoC2h3ZlwB/UFW/XWB2aYtxc5DUsSTPpdlMdbIJQJPmmoAk9ZhrApLUYyYBSeoxk4Ak9ZhJQJJ6zCQgST32/wET3F5zmoPOLgAAAABJRU5ErkJggg==\n", 904 | "text/plain": [ 905 | "
" 906 | ] 907 | }, 908 | "metadata": { 909 | "needs_background": "light" 910 | }, 911 | "output_type": "display_data" 912 | } 913 | ], 914 | "source": [ 915 | "%matplotlib inline\n", 916 | "\n", 917 | "final_data['minutes_play_integers'] = round(final_data['minutes_play'])\n", 918 | "plot_df = final_data.groupby('minutes_play_integers')['user_id'].count()\n", 919 | "\n", 920 | "# Plot the distribution of players that played 0 to 50 minutes\n", 921 | "ax = plot_df.head(n=50).plot(x=\"minutes_play_integers\", y=\"user_id\", kind=\"hist\")\n", 922 | "ax.set_xlabel(\"Duration of Video Played in Minutes\")\n", 923 | "ax.set_ylabel(\"User Count\")" 924 | ] 925 | }, 926 | { 927 | "cell_type": "markdown", 928 | "metadata": {}, 929 | "source": [ 930 | "# Metric 1: 1-day retention by AB-Group" 931 | ] 932 | }, 933 | { 934 | "cell_type": "code", 935 | "execution_count": 35, 936 | "metadata": {}, 937 | "outputs": [ 938 | { 939 | "data": { 940 | "text/plain": [ 941 | "0.3248730964467005" 942 | ] 943 | }, 944 | "execution_count": 35, 945 | "metadata": {}, 946 | "output_type": "execute_result" 947 | } 948 | ], 949 | "source": [ 950 | "# 1-day retention\n", 951 | "final_data['day_1_active'].mean()" 952 | ] 953 | }, 954 | { 955 | "cell_type": "code", 956 | "execution_count": 36, 957 | "metadata": {}, 958 | "outputs": [ 959 | { 960 | "data": { 961 | "text/plain": [ 962 | "version\n", 963 | "control 0.296875\n", 964 | "treatment 0.350486\n", 965 | "Name: day_1_active, dtype: float64" 966 | ] 967 | }, 968 | "execution_count": 36, 969 | "metadata": {}, 970 | "output_type": "execute_result" 971 | } 972 | ], 973 | "source": [ 974 | "# 1-day retention by group\n", 975 | "final_data.groupby('version')['day_1_active'].mean()" 976 | ] 977 | }, 978 | { 979 | "cell_type": "markdown", 980 | "metadata": {}, 981 | "source": [ 982 | "# Some interesting questions:\n", 983 | "1. Treatment has a higher retention rate (0.35) than the control (0.29). Is the difference significant?\n", 984 | "2. To what extent can we trust the result?\n", 985 | "3. What is the variability of the difference? \n", 986 | "4. In other words, how many times do we obtain more extreme values, if we repeat the process for 100 times?\n", 987 | "--> Solution: Bootstrap (resampling, replication) and check for the variability " 988 | ] 989 | }, 990 | { 991 | "cell_type": "code", 992 | "execution_count": 20, 993 | "metadata": {}, 994 | "outputs": [ 995 | { 996 | "data": { 997 | "text/plain": [ 998 | "" 999 | ] 1000 | }, 1001 | "execution_count": 20, 1002 | "metadata": {}, 1003 | "output_type": "execute_result" 1004 | }, 1005 | { 1006 | "data": { 1007 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxddZ34/9c7+97sbdqkTdIWum+kBSlCsWzqgKBlBlRoVQYVHWa+Mz+/4IxfYVB/oCA4zBcZmBFBQSmCI4jIgIXKIlsL3dMlzd6mzdbsS5Pcz/ePc2+aplluknvuOffe9/PxyOPk3nuW9z29zft+djHGoJRSKrJFOR2AUkop52kyUEoppclAKaWUJgOllFJoMlBKKQXEOB2AP7Kzs01hYaHTYSilVEjZvn17ozEmx599QyIZFBYWsm3bNqfDUEqpkCIiVf7uq9VESimlNBkopZTSZKCUUgob2wxEJAF4A4j3XudZY8wdIvI4cBHQ6t11kzFmh11xKKXcpa+vj9raWnp6epwOJWwkJCSQn59PbGzspM9hZwNyL/AJY0yHiMQCb4nIH72vfcsY86yN11ZKuVRtbS2pqakUFhYiIk6HE/KMMTQ1NVFbW0tRUdGkz2NbNZGxdHgfxnp/dFY8pSJcT08PWVlZmggCRETIysqacknL1jYDEYkWkR1APfCqMeY970s/EJFdIvKAiMSPcuzNIrJNRLY1NDTYGaZSKsg0EQRWIO6nrcnAGDNgjFkB5ANrRGQJ8G1gAbAayARuG+XYR40xJcaYkpwcv8ZMKBX+yrbAu/8BPW1OR6LCTFB6ExljWoCtwBXGmDpvFVIv8HNgTTBiUCrkbX8cnvwsvHwb/OqvwTPgdEQR5bvf/S5/+tOfnA7DNrYlAxHJEZF07++JwCXAfhHJ8z4nwNXAHrtiUCps1JfCS/8b5q6HT90H1e9A6e+djirsGGPweDwjvnbXXXdxySWXBDmi4LGzZJAHvC4iu4APsNoMXgSeEpHdwG4gG/i+jTEoFfqMgT/+b4hLgmsegZIvQ/oc+PAXTkfmWrfddhs//elPBx/feeed/PjHP+bee+9l9erVLFu2jDvuuAOAyspKFi5cyC233MKqVauoqalh06ZNLFmyhKVLl/LAAw8AsGnTJp591uoEuWXLFlauXMnSpUv58pe/TG9vL2BNnXPHHXewatUqli5dyv79+4P8zifPzt5Eu4wxK40xy4wxS4wxd3mf/4QxZqn3uS8O6XGklBrJ/j9AxRuw7p8hJQeiomHpBijfCp2NTkfnStdddx2bN28efPzMM8+Qk5PDoUOHeP/999mxYwfbt2/njTfeAODAgQPceOONfPTRRzQ2NnLkyBH27NnD7t27+dKXvnTauXt6eti0aRObN29m9+7d9Pf38/DDDw++np2dzYcffsjXv/517rvvvuC84QDQEchKuZlnALb8K2SfbZUIfBZdDWYADv6Pc7G52MqVK6mvr+fo0aPs3LmTjIwMdu3axSuvvMLKlStZtWoV+/fv59ChQwDMmTOH8847D4Di4mLKy8v5u7/7O15++WXS0tJOO/eBAwcoKirirLPOAmDjxo2DSQXgs5/9LADnnHMOlZWVQXi3gaHJQCk32/8iNB6EdbdB9JAxojOWQnIulL/uXGwut2HDBp599lk2b97MddddhzGGb3/72+zYsYMdO3ZQVlbGV77yFQCSk5MHj8vIyGDnzp2sW7eOhx56iJtuuum08xoz9nCp+Hirt3x0dDT9/f0Bflf20WSglJu9+zBkFFklgaFEoHidVVU0SoNnpLvuuut4+umnefbZZ9mwYQOXX345jz32GB0dVs30kSNHqK+vP+O4xsZGPB4Pn/vc5/je977Hhx9+eNrrCxYsoLKykrKyMgB++ctfctFFF9n/hmwWEusZKBWRTlRZvYbWf9dqJxhu7sWw+xmo32uVFNRpFi9eTHt7O7NmzSIvL4+8vDxKS0v52Mc+BkBKSgpPPvkk0dGn39sjR47wpS99abBX0d13333a6wkJCfz85z/n2muvpb+/n9WrV/O1r30tOG/KRjJekccNSkpKjC5uE76MMdQ0d5OTGk9i3Kn/mH8pa+T5HUe5bPF01i+c7mCEDnnzfqu94O93Qkbhma+3HYX7F8Kl34O1twY9vMkqLS1l4cKFTocRdka6ryKy3RhT4s/xWjJQjrv9ud1s3lZDanwMX76giM+tyuextyt4/C+ViMAz22t46qZzOX9uttOhBteBl2DWOSMnAoC0mZCzwKoqCqFkoNxJ2wyUo17fX8/mbTVce04+a+dl829bDnHhva/z+F8q2XR+IR9+51JmTkvkgVcPOh1qcPW2w5EPrXaBsRSvg6q/QJ9OB62mRksGylFPvFPJzGkJ/P+fXUpsdBS7alt4v6KZ84qzWDJrGgBfPG8OP3x5P5WNnRRmJ499wnBR/a7VdbTw42PvV7wO3vsPqH0fii4MRmQqTGnJQDmmufMkbx5q5OqVs4iNtj6Ky/LTuenjxYOJAODqlTMBeHnvMUfidETFGxAVCwXnjr1f4QUg0VZVkVJToMlAOebd8iYGPIZLFo3dOJw3LZH5uSm8XRZBo21rt8HMldYUFGOJT4X81XBYxxuoqdFkoBzzbnkTSXHRLB1SChjN2nnZfFDZzMn+COhT7/HAsV0wc4V/+xevg6MfQfcJO6NSYU6TgXLMe+XNlBRmDlYRjaWkMIOePg8Hj7cHITKHNR+Gkx2Qt9y//YvXAQYq3rQxKDVUZWUlv/rVryZ13JIlS2yIaOo0GShHtPX0ceB4O6vnZPi1/7JZ6QDsqm21Myx3qNtpbf1NBvklEJeiU1ME0VjJIJSmoBhKk4FyxP466xv+Ej+qiAAKMhNJT4plV22LnWG5Q90OiI63xhD4IzrWakgu/7O9cYWRX/ziFyxbtozly5dzww03UFVVxfr161m2bBnr16+nuroasKatvvXWWzn//PMpLi4enML69ttv580332TFihU88MADPP7441x77bVceeWVXHbZZRhj+Na3vjU4DfbQGVTdSruWKkeU1lnLNi7MSxtnT4uIsHTWNHZGQsng+D7IOdv6I++voovg4MvQUgPpBfbFFmD/+vu97Dsa2CU8F81M444rF4/6+t69e/nBD37A22+/TXZ2Ns3NzWzcuJEbb7yRjRs38thjj3Hrrbfyu9/9DoC6ujreeust9u/fz1VXXcWGDRu45557uO+++3jxxRcBePzxx3nnnXfYtWsXmZmZPPfcc+zYsYOdO3fS2NjI6tWrufBCd3f91ZKBckRpXRuZyXFMT4v3+5hFM9Moq2+nbyDMG5EbD/pfKvAp9k6UVqGlg/G89tprbNiwgexsa0R7ZmYm77zzDp///OcBuOGGG3jrrbcG97/66quJiopi0aJFHD9+fNTzXnrppWRmZgLw1ltvcf311xMdHc306dO56KKL+OCDD2x8V1OnJQPliH11bSzMS8Va/dQ/Z+Wm0jdgqGrqZF5uqo3ROai3A1prIGfjxI7LXQTJOVZV0cov2hObDcb6Bm8XY8y4n7uhr/umpPYdO5qh02CHwpxvw2nJQAVd/4CHA8faWeRnFZHPWdOtBHDweBgvjtfonXYj++yJHSdiVRVV/NlaJlONav369TzzzDM0NTUB0NzczPnnn8/TTz8NwFNPPcUFF1ww5jlSU1Npbx+9Z9uFF17I5s2bGRgYoKGhgTfeeIM1a9YE7k3YQEsGKugqGjvp7ff43V7gMy83BRE4eLydTy3Nsyk6h/mSQc4EkwFY01HseRYaD0HOWYGNK4wsXryYf/mXf+Giiy4iOjqalStX8uCDD/LlL3+Ze++9l5ycHH7+85+PeY5ly5YRExPD8uXL2bRpExkZp/eKu+aaa3jnnXdYvnw5IsKPfvQjZsyY4eqVz2xLBiKSALwBxHuv86wx5g4RKQKeBjKBD4EbjDEn7YpDuc8+b+PxopkTSwaJcdEUZCRxKJxLBg0HICoGMosnfmyB95vn0Q81GYxj48aNbNx4elXca6+9dsZ+jz/++GmPfQvjxMbGsmXLltNe27Rp0+DvIsK9997Lvffee9o+hYWF7NmzZwqR28fOaqJe4BPGmOXACuAKETkP+CHwgDFmPnAC+IqNMSgX2lfXRlx0FHNzUiZ87FnTU8J74FnjQcicO7GeRD7ZZ1njDY5sD3xcKuzZlgyMxfcVLtb7Y4BPAM96n38CuHqEw1UYK61rZ15uil8jj4ebPz2VisbO8J2WouHA5L/VR0VD3gpNBmpSbG1AFpFoEdkB1AOvAoeBFmOMb4heLTBrlGNvFpFtIrKtoaHBzjBVkJXWtU24vcDnrOkp9HsMlU2dAY7KBfpPQnP5xBuPh5q1Co7thv7ewMWlIoKtycAYM2CMWQHkA2uAkda6G7HrgzHmUWNMiTGmJCcnx84wVRDVt/fQ0N7L4gm2F/jMy7F6FB2uD8N2g+bD1hoGk2k89slbDgMnrUZkpSYgKF1LjTEtwFbgPCBdRHwN1/nA0WDEoNyh1DsNxUQbj32Kc6y+3OWNYVgyGOxWOoXG39xF1ra+dOrxqIhiWzIQkRwRSff+nghcApQCrwMbvLttBJ63KwblPr6pByZbTZQcH8OMtAQON4RhycD3bT5r3uTPkTXP6o3UoMlATYydJYM84HUR2QV8ALxqjHkRuA34RxEpA7KAn9kYg3KZfXVt5GckMi1xEr1lvIpzkilvCMOSQVMZpM6E+In3shoUE2clBC0ZjKqlpYWf/vSnATvfT37yE7q6ugJ2PqeuY2dvol3GmJXGmGXGmCXGmLu8z5cbY9YYY+YZY641xmhLVwTZe7R1wiOPh7OSQUdIDvkfU+MhyJ5CqcAnZ4EmgzGMlgwGBgYmdT5NBkpNUNfJfioaOyfdXuAzNyeFtp5+GjvCaKyiMdB0CLLmT/1cuYvgRCWctP8PVCi6/fbbOXz4MCtWrGD16tVcfPHFfP7zn2fp0qUAPPnkk6xZs4YVK1bw1a9+dTBJfP3rX6ekpITFixdzxx13APDggw9y9OhRLr74Yi6++GIAUlJSuO222zjnnHO45JJLeP/991m3bh3FxcW88MILgJV4vvWtb7F69WqWLVvGI488AsDWrVtZt24dGzZsYMGCBXzhC1/AGDPidQJNp6NQQbP/WDvGwOKZ/q1hMJpi72C18oYOclL9n/XU1ToboacVsgOQDLLnAQZOVMD04E8ENyF/vN3qChtIM5bCJ+8Z9eV77rmHPXv2sGPHDrZu3cqnP/1p9uzZQ1FREaWlpWzevJm3336b2NhYbrnlFp566iluvPFGfvCDH5CZmcnAwADr169n165d3Hrrrdx///28/vrrg7OgdnZ2sm7dOn74wx9yzTXX8J3vfIdXX32Vffv2sXHjRq666ip+9rOfMW3aND744AN6e3tZu3Ytl112GQAfffQRe/fuZebMmaxdu5a33357xOsEmiYDFTS+xuOplgyKs0/1KDq3OGvKcblCU5m1nUrjsY9vKoumw+5PBi6wZs0aioqKANiyZQvbt29n9erVAHR3d5ObmwvAM888w6OPPkp/fz91dXXs27ePZcuWnXG+uLg4rrjiCgCWLl1KfHw8sbGxLF26dHBuoldeeYVdu3YNLpbT2trKoUOHiIuLY82aNeTn5wOwYsUKKisrx504LxA0Gaig2VfXxrTEWGZOS5jSeWalJxIfE0V5OPUoagpATyKfzLnWtvnw1M9ltzG+wQfL8KmnN27cyN13333aPhUVFdx333188MEHZGRksGnTJnp6ekY8X2xs7OAU2FFRUYNTYEdFRQ0uiWmM4d///d+5/PLLTzt269atp02ZHR0dHbRlNLXNQAXNvqNtLMpLm9AaBiOJihKKspM5HE49ihoPWUtdps+e+rkS0qy1DZpCIBk4YKzpp9evX8+zzz5LfX09YE1vXVVVRVtbG8nJyUybNo3jx4/zxz/+0a/zjebyyy/n4Ycfpq+vD4CDBw/S2Tn253ky15kILRmooBjwGPYfa+ML584JyPnm5qSw92gYLYHZVGZV70RFB+Z8mXOtqS3UGbKysli7di1LliwhMTGR6dOnD762aNEivv/973PZZZfh8XiIjY3loYce4rzzzmPlypUsXryY4uJi1q5dO3jMzTffzCc/+Uny8vJ4/fXX/YrhpptuorKyklWrVmGMIScnZ3CZzdFM5joTIaHQPa+kpMRs27bN6TDUFJTVd3DJ/X/mvmuXs+Gc/Cmf78evHOCnWw9TetcVxMWEQQH3/662Rh5f91Rgzve7W6BsC/x/BwJzvgAqLS1l4cKRZqZRUzHSfRWR7caYEn+OD4P/RSoU+NYwmOycRMMV5yQz4DFUN4dBVdFAPzRXBKYnkU9mMXQcs5bRVMoPmgxUUOw92jrpNQxG4jtPWX0YJIOWKvD0BWaMgU+WrxFZq4qUfzQZqKDYd7SN+dNTAlalUzTYvTQMvvn65iQKdMkAXNujKBSqp0NJIO6nJgNlO2PMYE+iQElNiCU3NT485igKZLdSH1/3Uhf2KEpISKCpqUkTQoAYY2hqaiIhYWpdtrU3kbJdQ3svTZ0nA9Ze4OOboyjkNZVBYiYkZQbunPEpkJxrTUvhMvn5+dTW1qKLVgVOQkLC4EC1ydJkoGy3t8438nhq01AMV5yTwh921WGMmfLYBUc1lgW2isgno9CVySA2NnZwxK9yD60mUrbzTUOxIC81oOedm5NCa3cfzZ0hPmFdoCaoG86lyUC5kyYDZbvSujYKMhNJS5j8GgYjCYtVz3raoOP4qd4/gZRZBK211trKSo1Dk4GyXVl9B2flBrZUADA3+9TspSGryYaeRD4ZhYCBlurAn1uFHU0GylYDHkN5YydzcwMzvmCoWRmJxMVEhfYcRb7ePrZUE3nr5bWqSPlBk4GyVU1zFyf7PcwL0GCzoaKjhKKsEO9R1HgIJMqq0gm0jEJre6Ii8OdWYUeTgbJVWb31h9qOkgGEwXrITYesmUpjbFikJ2U6xCRoyUD5RZOBslWZ91v7PBuTQXVzF30DHlvOb7vGQ9YEdXaIitIeRcpvtiUDESkQkddFpFRE9orI33ufv1NEjojIDu/Pp+yKQTmvrN5amnJaYmB7EvkUZ6fQ7zFUNYXger+eAXuTAWgyUH6zc9BZP/BPxpgPRSQV2C4ir3pfe8AYc5+N11Yucbihw5b2Ah9f9VN5Q4dtpQ/btFTDQK/NyaAIKt4EYyCUB+Yp29lWMjDG1BljPvT+3g6UArPsup5yH2MMZfUdzM1NHn/nSQrpsQaNB61tztn2XSOjEPo6obPRvmuosBCUNgMRKQRWAu95n/qmiOwSkcdEJGOUY24WkW0isk3nMAlNJ7r6aO/ppzDLvmSQlhBLdkp8aPYo8iUDu6uJQHsUqXHZngxEJAV4DvgHY0wb8DAwF1gB1AE/Huk4Y8yjxpgSY0xJTk6O3WEqG1Q1Wd/W59iYDCCEexQ1HICk7MBOUDdcpo41UP6xNRmISCxWInjKGPNbAGPMcWPMgDHGA/wnsMbOGJRzqputRt05WUm2XqcwK4mq5hBsQG48ZG8VEVjdVkGTgRqXnb2JBPgZUGqMuX/I83lDdrsG2GNXDMpZ1d4ePrMz7U0GszOTaGjvpfvkgK3XCShjoPGAPdNQDBWbCKl51rKaSo3Bzt5Ea4EbgN0issP73D8D14vICsAAlcBXbYxBOaiquYvpafEkxEbbep3Z3mqomhNdnDU98HMg2aKzEbpPQLbNJQOwehRpyUCNw7ZkYIx5CxipL9tLdl1TuUt1U5ftpQI4VfKobgqhZBCMxmOfjEIo32r/dVRI0xHIyjZVzZ3MzrS38RiGJINQajdoPGBtc4KQDDKLoP0o9HXbfy0VsjQZKFv09A1wvK3X9sZjgIykWFLiY0IsGRyC2CRIm9pShX7xdS/VqazVGDQZKFvUBKknEYCIUJCZFFrJoOEAZM2z5g+y2+BYg0r7r6VCliYDZYuqIPUk8pkTasmg8aD93Up9fOsaaI8iNQZNBsoWvn7/wUoGs7OSqGnuwuMxQbnelJzshNaa4DQeAyRnQ2yylgzUmDQZKFvUNHeREh9DZnJcUK5XkJlEb7+Hho7eoFxvShp9S10GKRmIWI3IOiWFGoMmA2WLqqZOZmcmIUGaKTOkehT5kkGwqolAp7JW49JkoGxR1dwVlMZjnzlDxhq4XuMB71KXxcG7pi8ZmBCoRlOO0GSgAm7AY6ht7mZ2EJPBzPREooTQmKOoqQzS59iz1OVoMgqhvwfajwXvmiqkaDJQAXesrYeTA56gNR4DxMVEkTctkdpQSAbN5ZA1N7jXzNDZS9XYNBmogPNV1cwJwujjoQoyE93fZmCM1cUzmFVEoGMN1Lg0GaiAq272rWMQvJIBQEFGEjUnXJ4Mupqgty34ySB9NiDao0iNSpOBCriqpi5iooS8aQlBve7szCSOt/XS0+fiqayby61tsJNBTBxMy9eSgRqVJgMVcFXNXeRnJBITHdyPV4G3jaL2hIsnZHMqGYBVVaSjkNUoNBmogKtu6hr8wxxMBZmJwKl5kVypudzqVupbgSyYdKyBGoMmAxVw1UEeY+DjS0CubjdoLreqa4LZrdQnoxA666G3I/jXVq6nyUAFVGtXH63dfUHvSQSQkxJPQmyU+0sGTlQRwakeRa01zlxfuZomAxVQVd6eRMEccOYjIhRkuHz2UieTwbQCa9uiyUCdSZOBCijf1NVOVBOBVVVU0+zSBuTuFmvdY98AsGBL9yaDVl3kRp3JtmQgIgUi8rqIlIrIXhH5e+/zmSLyqogc8m4z7IpBBZ/vW3lBhkPJICORmuYujBvn4PFVzzjReAyQMgOiYrVkoEZkZ8mgH/gnY8xC4DzgGyKyCLgd2GKMmQ9s8T5WYaK6qYvslHiS42McuX5BZhLtvf20dvc5cv0x+f4I+6prgi0qymq81jYDNQLbkoExps4Y86H393agFJgFfAZ4wrvbE8DVdsWggq+qudOxKiI41aPIle0GrbXWNt2hZOC7tq6FrEbgVzIQkedE5NMiMqnkISKFwErgPWC6MaYOrIQB5I5yzM0isk1EtjU0NEzmssoB1U1dg9NJO8E3OZ4r2w1aayA6HpKynYth2mytJlIj8veP+8PA54FDInKPiCzw9wIikgI8B/yDMabN3+OMMY8aY0qMMSU5OTn+HqYc1Ns/QF1bjyM9iXzcXTKosappohzst5FeAB3HoD8EVoRTQeXXp9IY8ydjzBeAVUAl8KqI/EVEviQisaMd533tOeApY8xvvU8fF5E87+t5QP1U3oByj5rmbowJ3rrHI/EttenKgWettVYycJKvvcJXZaWUl99fUUQkC9gE3AR8BPwbVnJ4dZT9BfgZUGqMuX/ISy8AG72/bwSen3DUypV8g72cbDOAUz2KXKelxtn2AjjVk0kbkdUwfnX5EJHfAguAXwJX+ur8gc0ism2Uw9YCNwC7RWSH97l/Bu4BnhGRrwDVwLWTDV65S1WTd8CZA6OPh8rPTGLvkVZHYzhDf69VPeNUTyKfdB14pkbmb/+//zLGvDT0CRGJN8b0GmNKRjrAGPMWMNpq6OsnEKMKEVXNXSTFRZOdEudoHLMzk3hl7zEGPIboqNE+gkHWdsTaOl1NlDbLmihPexSpYfytJvr+CM+9E8hAVOirbupidmYSVg2hcwoykugbMBxr63E0jtP46uidLhlEx0JqnlYTqTOMWTIQkRlYYwMSRWQlp77ppwHOVgwr16lu7qIo29kqIhjavbSLWemJDkfjNZgMHC4ZgJWQtJpIDTNeNdHlWI3G+cDQRuB2rPp/pQDweAzVzV2sO9v5bsC+dQ2qm7s4rzjL4Wi8BkcfuyAZpM+GmnedjkK5zJjJwBjzBPCEiHzOGPNckGJSIai+vZfefg+zs5wvGcxMTyRKXLbITWsNpEx3Zh2D4dILYO9vYaAfop2ZNkS5z3jVRF80xjwJFIrIPw5/fViXURXBfD2JnBx97BMbHUV+RhKVTS5LBk63F/hMKwBPP7TXOd/VVbnGeA3Ivq95KUDqCD9KAVZPInB+jIHP3JxkyupdtKKXGwac+QxOZa3tBuqU8aqJHvFu/zU44ahQVdPcRXSUMNMlDbZzc1L4y+EmPB5DlNPdS42xksHZn3Q2Dp9p3oFnLTUwx9lQlHv4O1Hdj0QkTURiRWSLiDSKyBftDk6FjqqmLmamJxAb7Y71kubmptDb7+FIiwsmrOtshP4e91QT6SI3agT+/s+9zDvJ3F8BtcBZwLdsi0qFnKrmLkfWPR7NvNwUAA43uKCqqNVFPYkAYhMhOUe7l6rT+JsMfJPRfQr4tTGm2aZ4VIiqbup0dLbS4ebm+JJBp8ORMCQZuKRkAFYs2maghvA3GfxeRPYDJcAWEckBXDS8UzmpraePE119ruhJ5JOZHEdGUqw7GpHdNODMRxe5UcP4O4X17cDHgBJjTB/QibVimVJUe7twOjl19Ujm5qS4pJqoFmKTIdFFy31PK7DicuNa0coRExlxshBrvMHQY34R4HhUCPItJOOmaiKwksGW/cedDsOqjkkvAIfnbDpN+hyrUbuzAVJGXGxQRRh/exP9ErgPuABY7f0ZcbZSFXmqmnxjDNzTgAxWI3Jjx0lauk46G0hLjbuqiECnslZn8LdkUAIsMkbLlOpM1c2dZCXHkRLvrqkN5uZayelwQwfnzMl0LpDWWpi5wrnrj2TakO6l+ec4G4tyBX8bkPcAM+wMRIWuqqYu11URwakeRY42Ivd1Q1eju3oSgZYM1Bn8/SqXDewTkfeBwZW0jTFX2RKVCinVzV2cM8dFjaNe+RlJxMVEOdu91C3rGAyXMA3ip2mPIjXI32Rwp51BqNB1st/D0ZZuPrtyltOhnCE6SijOdniOIrcNOBsqXccaqFP8SgbGmD+LyBxgvjHmTyKSBETbG5oKBUdauvEYXDF19Ujm5aawq9bB9ZDdOMbAJ302nKhyOgrlEv72Jvpb4FngEe9Ts4Df2RWUCh2DU1e7sM0ArHaDmhNd9PQNOBNAS4215nDaTGeuPxYdhayG8LcB+RvAWqANwBhzCBizc7KIPCYi9SKyZ8hzd4rIERHZ4f351LdCf7oAABlnSURBVGQDV+7gG2PgptHHQ83LTcEYqGh0qN2gtdZaczg6dvx9gy29AHrboLvF6UiUC/ibDHqNMYOdtb0Dz8brZvo4cMUIzz9gjFnh/XnJz+srl6pq6iIxNpqcVBes4DUC34R1jrUbtLpwjIFPum8qa60qUv4ngz+LyD8DiSJyKfAb4PdjHWCMeQPQCe3CXFVTF7MzkxA3ja4doig7GREHZy9106I2w2XNs7aNh5yNQ7mCv8ngdqAB2A18FXgJ+M4kr/lNEdnlrUYatT+iiNwsIttEZFtDQ8MkL6XsVtXU6dr2AoCE2GgKMpKcKRkM9Hunopgd/Gv7I3Ou1Z6hyUDh/0R1HqwG41uMMRuMMf85ydHIDwNzgRVAHfDjMa75qDGmxBhTkpOTM4lLKbt5PIbq5i5XJwOwqoocSQZttdZawxlFwb+2P2ITrDmKGg86HYlygTGTgVjuFJFGYD9wQEQaROS7k7mYMea4MWbAm1z+E1gzmfMod6hv76W33+PabqU+c3OSqWjsZMAT5NlUmiusbaZLkwFA9lmaDBQwfsngH7B6Ea02xmQZYzKBc4G1IvK/JnoxEckb8vAarGkuVIiq9HYrLQyBkkFvv4cjJ4K8BOYJbzJwa8kAIHs+NJWBx6Gut8o1xksGNwLXG2MqfE8YY8qBL3pfG5WI/Bp4BzhbRGpF5CvAj0Rkt4jsAi4GJpxQlHv41jFw03KXIxnsUdTQHtwLN1dAdJw7xxj4ZJ9lTWWt4w0i3ngjkGONMY3DnzTGNIjImB2njTHXj/D0zyYSnHK3quZOYqKEmekJTocypqET1n1iwfTgXfhEhVUnH+XiwfrZZ1nbxkOQUehoKMpZ45UMxpoI3uFJ4pXTKpu6yM9IJCba305pzkhPiiMzOY6Kxq7gXri50t3tBQA5Z1vbhgPOxqEcN17JYLmItI3wvADu/jqobFfd1OX6xmOfouxkKhqD2KPIGKtkMOf84F1zMpIyISkLGjUZRLoxk4ExxsXlW+UkYwyVTZ2sKEh3OhS/FGYl83bZGTWe9ulshJMd7i8ZAOQs0JKB8nvQmVKnaenqo72n3/VjDHyKc5I51tZDZ29/cC4YCj2JfHIXQn2pVZpREUuTgZqUqmZ3rns8mqJsK05fd1jbNZdb21AoGeQutCasazvidCTKQZoM1KS4ferq4Qq9SasyWI3IDQcgKiZESgaLrG19qbNxKEdpMlCTUuUdYzDbpVNXD1eYbcUZtEbkhgPW3D8xccG53lTkLLC29fucjUM5SpOBmpSqpi5mpCWQEBsafQyS4mKYkZYQvO6lDftPddt0u6RMSJkB9fudjkQ5SJOBmpTq5k5mh0gVkU/Qupf29VgNyL5v3KEgd6GWDCKcJgM1KZVNXa5d3Ww0hdnJwVnxrKkMjCd0SgZgtRs0HNA5iiKYJgM1YV0n+2lo76UwOzR6EvkUZydzoquPli6bB883eKtbQq1k0N8NJyqdjkQ5RJOBmjDfuseh0njs4+teanvpoOGAtWiMbyWxUKA9iiKeJgM1Yb6eRKHSrdSnMFhjDRr2W11KY0NoxhZflZYmg4ilyUBNWGWjb4xBaFUTzc5MIkqgoiEIJYNQqiICiE+xZljVRuSIpclATVh5QyfZKXFMSxxzFnPXiYuJIj8jiXI7q4n6T0Lz4dBqPPbJXaQlgwimyUBNWEVj52D9e6gpyk62t5qoudxa9zjUSgYAuQug6ZCV0FTE0WSgJqy8sYPi7BSnw5iUouxkKho6MXZNyjbYkyhESwaefqtkoyKOJgM1Ia3dfTR2nKQ4J3RLBp0nB2ho77XnAg0HADm1glgoyV1obbXdICJpMlAT4uuWGcrVRGBj99KG/ZA+G+JCq6cVAFnzQaK13SBCaTJQE1LeYE3nUJwTutVEYGcyCMGeRD6xCZA1V5NBhLItGYjIYyJSLyJ7hjyXKSKvisgh7zbDrusre1Q0dhIdJSE34MxnZnoicdFRVNjRiDzQbzXAhmJ7gY/OURSx7CwZPA5cMey524Etxpj5wBbvYxVCyhs6KchIJC4mNAuV0VHCnKwke8YanKiEgZOhWzIAqxG5uQJOBml2V+Uatv2PNsa8ATQPe/ozwBPe358Arrbr+soehxs6QraKyMe2CetCcU6i4XIXAgYadU3kSBPsr3fTjTF1AN5t7mg7isjNIrJNRLY1NDQELUA1Oo/HUNkUumMMfIqzk6lq7mLAE+DupYPJIAR7Evnk+HoU6doGkca1ZX1jzKPGmBJjTElOTo7T4Sigrq2Hnj5PyHYr9SnMTuZkv4ejLd2BPXHDAUjLh/jUwJ43mDKLITpO2w0iULCTwXERyQPwbuuDfH01BQePtQMwPzeE/9hxqkdRwEcih9LqZqOJjoHss7VHUQQKdjJ4Adjo/X0j8HyQr6+m4MBxKxmcPT20k0GxHd1LPQPQePDUwK1QlrtQk0EEsrNr6a+Bd4CzRaRWRL4C3ANcKiKHgEu9j1WIOHCsnbxpCUxLCq0J6obLSY0nOS6a8kD2KGqphv6e0C8ZgJUM2mqhp9XpSFQQxdh1YmPM9aO8tN6uayp77T/WzlkhXioAEBEKAz1hXYO3900o9yTyGVzoZj/MPtfZWFTQuLYBWblL/4CHw/UdLJgR+skArHaDsvqOwJ3Q15MoFOckGk7nKIpImgyUXyqbOjk54OHsMEkGC/PSqD3RTWt3X2BO2HAAUvMgMT0w53PStAKIS9F2gwijyUD5Zb+3J1E4VBMBLJ6ZBkBpXVtgThgOPYl8oqKs99KgySCSaDJQfjl4rJ3oKGFebmiPPvZZ5E0G+44GIBkYE9oT1I1EexRFHE0Gyi+lx9opzEoiITba6VACIjc1gZzUePYGIhm01kJfZ/iUDMBqRO5sgA4d/R8pNBkov+yubWXprGlOhxFQi/LS2BeIaqJw6knk42tE1qqiiKHJQI2rvq2HY209LM0Pg8bRIRbPTOPQ8XZ6+wemdiLfH8zsMCsZgFYVRRBNBmpcu49Yg4+W5YdZyWBmGv0ew6HjU+xienwfpEyH5KzABOYGKdMhMUO7l0YQTQZqXLtqW4kSq1olnCyeaSW3PUemONK2fu+pb9LhQsR6T1oyiBiaDNS49hxpZV5uCsnxtg1Yd0RhVhJpCTHsrG2Z/Ek8A1abwfTFgQvMLXw9ikyAp/pWrqTJQI3JGMOuI60snRVe7QVgTUuxvCCdHTVTKBk0l1tzEoVbyQCsZNDbZvWWUmFPk4Ea07G2Hhrae8OuvcBnZUE6B4610XWyf3InOL7X2oZjySBvhbU9+qGzcaig0GSgxvRRtVWFEq7JYMXsdDzG6jo7Kcf3gkSF1xgDnxnLIDoeat53OhIVBJoM1Ji2VZ4gPiZqsLE13Cz3dpfdUTPJdoP6fZA5F2ITAxiVS8TEwaxVUPOe05GoINBkoMa0vfoEywvSiYsJz49KVko8BZmJk08Gx/fC9DBsL/DJXw1Hd0Bfj9ORKJuF5/9wFRDdJwfYe6SVkjkZTodiqxUFGZNLBt0n4EQF5C0PfFBuUXAuePqgbqfTkSibaTJQo9pZ20K/x3BO2CeDdOpaezjWOsFvv0c/srYzVwU+KLcoWGNttaoo7GkyUKPaXnUCgFWzwzsZrC603t97FU0TO/CIt5fNzJUBjshFUnIho0iTQQTQZKBGtb3qBHNzkslIjnM6FFstyksjNT6Gd8ubJ3bg0Y+sxuNwWNBmLAXnQvW7OvgszDmSDESkUkR2i8gOEdnmRAxqbB6PYXvVCUrmZDodiu1ioqMoKcyYXMlgVhhXEfkUroWuxlOzs6qw5GTJ4GJjzApjTImDMahR7Ktro7W7j/Pmhn8yADi3OIvyhk7q2/1sN2g7Cu1Hw7u9wKfw49a28k1n41C20moiNaJ3Dlvfks+fm+1wJMFxbpGV9N6v8LOqqPItazvnfJsicpGMQkjLP/WeVVhyKhkY4BUR2S4iN4+0g4jcLCLbRGRbQ4OuthRsbx9uZG5OMtPTEpwOJSiWzJpGclz0YBIcV8UbkDANZiy1NzA3EIHCC6xkoO0GYcupZLDWGLMK+CTwDRG5cPgOxphHjTElxpiSnJyc4EcYwfoGPLxf0RwxpQKA2OgoPjY3m60HGjD+/MGrfBPmXABR4bEM6LiKPu5tN9jvdCTKJo4kA2PMUe+2HvhvYI0TcaiR7aptoevkAOfPDaPFWvywfmEuR1q6OTjeYjctNXCi0voDGSkKL7C2WlUUtoKeDEQkWURSfb8DlwF7gh2HGt0bBxsRgfOKIysZXHx2LgBb9h8fe8fDr1nbojMKtOEroxCmzbaqx1RYcqJkMB14S0R2Au8DfzDGvOxAHGoUW/YfZ9XsjLAfXzDcjGkJLJmVxpbS+rF33P8HSJ8dnmsYjKX4Qqj4Mwz0OR2JskHQk4ExptwYs9z7s9gY84Ngx6BGd7Slmz1H2rhk4XSnQ3HEpQtn8GH1Cepau0feobcdyrfCgiuthtVIMv9y6GnV0chhSruWqtNsKbWqSC5dFJnJ4DMrZmIMvLDj6Mg7lP0JBnphwaeDG5gbzL0YomLh4P84HYmygSYDdZrndxxlXm4Kc3OSnQ7FEYXZyaycnc5/f3Rk5B1KX4SkLGuKhkgTn2qNRtZkEJY0GahBFY2dbKs6wYZz8pFIqwIZ4pqVs9h/rJ09R4atftbbAQdeggV/BdExzgTntLOugMYD1trPKqxoMlCDnt1eQ5RYfwwj2VXLZ5IQG8VT71Wd/kLp76GvC5Zf70xgbuCrHtvznLNxqIDTZKAA6Ojt58l3q1m/cHrEjDoeTXpSHFevmMV/f3SE1q4hPWd2/trqYjn7PMdic1z6bJh9Pux6RkcjhxlNBgqAp96torW7j29cPM/pUFzhxo8V0tPn4Tfba6wnWmutPvbLr4+8XkTDLf8baDwIdTucjkQFkCYDRUN7Lw+9XsbH52ezoiDM5+b306KZaawpzOQX71Qx4DGw41eAgWV/7XRozlv0GYiOg51POx2JCiBNBoq7Xyqlu2+AO65c7HQorvKltYVUN3fxyu4a+OBnMHc9ZBY7HZbzEjNg4ZWw49dWo7oKC5oMItw7h5v47UdHuPnCYublpjgdjqtctngGRdnJ7PnTL6HjGJz7NadDco9zvwa9rbBLSwfhQpNBBOsb8PDd5/eQn5HINy+e73Q4rhMdJfztBYV8qnUz3amFMO8Sp0Nyj/zV1trP7z2iDclhQpNBBHvsrQoO1Xdw55WLSYyLkKmYJ2hD4jYWR1XxeNx1EKX/XQaJwLlftxqSy7Y4HY0KAP10R6i61m7+bcshLlmYyyUROvXEuPq6ifvzD2hKmsu9R5aw72ib0xG5y+JrYFoBbL1bSwdhQJNBhPr+i6UMeIw2Go9l6z3QXE78lfeSFB/Hg1sOOR2Ru8TEwYXfgiPb4KBOPBzqNBlEoDcPNfCH3XV84+J5FGQmOR2OO9Vug7/8O6y6kZSF67n5wmJe3nuMDyr9XCM5Uqz4PGTOhZe/DSe7nI5GTYEmgwjT0zfAHc/vpTAriZsv1G6SI+pshGduhGmz4NLvAXDTx4uYnhbPXb/fR9+Ax+EAXSQ6Fq78CZyogNe+73Q0ago0GUSYh7cepryxk+9dvYSEWG00PkP/SXj2S9DVBH/zJCRag/CS4mL47l8tZveRVu5/9aDDQbpM0YWw+m/h3Ye8g/NUKIrQqRcj08Hj7Ty89TBXLZ/Jx+fnOB2O+3g88Pwt1rQTV/8H5C0/7eVPL8vjzUMFPLz1MDPTE7nhvDkOBepCV9xt9Sx6/psgUbD8OqcjUhOkJYMI0dnbzy1PfUhaYgz/568ibLlGfxgDL98Ou38D6++AFSPPTHrXZ5awfkEu/+d3e7j7pVL6tcrIEh0L1z1lrXfw31+FN+6zkqsKGZoMIkDXyX6+8sQHlDd08OB1K8lJjXc6JHfxDMDv/x7efwQ+9k244H+NumtcTBQPf/EcvnjebB55o5y/efRdyup1SgbAWvzmC8/Ckg3w2vdg8xeg+4TTUSk/aTIIY8YY3itv4jP/923eq2jm/r9ewfnzsp0Oy11aauCXV8OHT8DH/wku+/64s5LGxUTx/auX8pO/WUFZfQefevBNfvKng7T36ELxxMTD5/4LrvghHHoFHl4L+57XcQghQIwD/0gicgXwb0A08F/GmHvG2r+kpMRs27YtKLGFumOtPbxf2cz7FU28XdZERWMnuanx/Pivl2s7gTHQdgSO7YHju6FuJxx42ari+OSPYNUNEz5lfXsPd76wl5d2H2NaYixXLs/jEwtyWTAjjRlpCURFRfB017XbrRLX8d2QvwYu+AdrpbQo7bgQLCKy3RhT4te+wU4GIhINHAQuBWqBD4DrjTH7RjtGk8HpPB5Dx8l+Wrv6aOjoZe+RVj6qbuGDqmZqmrsBSI6L5tziLNYvzOWzK/PDf7oJY6CvG3rboKfN2nY2QFOZ9dNYBsf3QE/LqWMyCq35htb+vbVoyxTsqm3h0TfK2VJaT3ffAABx0VHMTE9gVkYi+elJzMpIZG5OCgvyUinMSiY6EhLFQL9V6nr7J9BSDSnTrWVD80sga77VWysuGfp7YeAknOyA3nbrB6z1ppOyISEN4lKsfSN9PYkJcHsy+BhwpzHmcu/jbwMYY+4e7ZjJJoMHtxzihZ1H8b3HwXdqTm2Gv2YGXzOnPx52m8Y6zgw5/+nHDjvneOcyZsSYu0724xkWT3ZKHCVzMlldlMmawkwWHf0N0e8/OuRAM+xEE3x82q8BOueEH49x/f4e8PQzosRMyJoH0xfB9CUwYynkLrL+wARY18l+9h5tY/+xdmqbu6ht6ebIiW6OtHTT0N47uF+UQFpiLKkJMUSLICIIgECU7/cQcseVi7lg/hhVkAP9sP9Fa7nMsj9Zy4dOilhVUVGx1jrUUTFDfo/1ljpC7e6N48qfwJzzJ3XoRJKBE11LZwE1Qx7XAucO30lEbgZuBpg9e3Lf2nJT4zl7eqr3hKdtBhd8F0590RjptdOP9T5/xv5nvi7DDvbnmJFiYMj+vn2T4qJJT4olLTGWzKQ4FuSlMis98fRj2nIgd+HpFxt8I5N9PNY+471u8+OYeOuPe7z3JyHNmwTmQlImwZIUF8PqwkxWF555zZ6+AcrqOyita6OqqYvW7j46evvxGIPHnEr+TlTdTlVy/Dglz+gYWHy19eMZsEprJ6qsktrJTuvfLzrOaoSOT7VKARhrAGBXk1XSO9lprZ8w0GslF08fDPR5t97HnoGgvN+gigvO1PJOlAyuBS43xtzkfXwDsMYY83ejHaPVREopNXETKRk40ZuoFigY8jgfOOpAHEoppbycSAYfAPNFpEhE4oDrgBcciEMppZRX0NsMjDH9IvJN4H+wupY+ZozZG+w4lFJKneLI3ETGmJeAl5y4tlJKqTPpCGSllFKaDJRSSmkyUEophSYDpZRSODRR3USJSANQ5XQcDsgGGp0OwiX0Xlj0Ppyi98Iy1n2YY4zxa4bKkEgGkUpEtvk7ejDc6b2w6H04Re+FJVD3QauJlFJKaTJQSimlycDtHnU6ABfRe2HR+3CK3gtLQO6DthkopZTSkoFSSilNBkoppdBk4BgRuUJEDohImYjcPsLr/ygi+0Rkl4hsEZE5Q14bEJEd3p+Qnv7bj/vwNRHZ7X2vb4nIoiGvfdt73AERuTy4kQfeZO+FiBSKSPeQz8R/BD/6wBnvPgzZb4OIGBEpGfJcRH0mhux32r2Y1GfCGKM/Qf7Bmrr7MFAMxAE7gUXD9rkYSPL+/nVg85DXOpx+D0G8D2lDfr8KeNn7+yLv/vFAkfc80U6/J4fuRSGwx+n3EKz74N0vFXgDeBcoidTPxBj3YsKfCS0ZOGMNUGaMKTfGnASeBj4zdAdjzOvGGN+q4e9irQgXbvy5D21DHiYDvh4PnwGeNsb0GmMqgDLv+ULVVO5FOBn3Pnh9D/gR0DPkuYj7THiNdC8mTJOBM2YBNUMe13qfG81XgD8OeZwgIttE5F0RudqOAIPEr/sgIt8QkcNYH/hbJ3JsCJnKvQAoEpGPROTPIvJxe0O11bj3QURWAgXGmBcnemyImcq9gAl+JhxZ3EYhIzw34rc8EfkiUAJcNOTp2caYoyJSDLwmIruNMYdtiNNuft0HY8xDwEMi8nngO8BGf48NIVO5F3VYn4kmETkH+J2ILB5WkggVY94HEYkCHgA2TfTYEDSVezHhz4SWDJxRCxQMeZwPHB2+k4hcAvwLcJUxptf3vDHmqHdbDmwFVtoZrI38ug9DPA34SkITPdbtJn0vvNUiTd7ft2PVM59lU5x2G+8+pAJLgK0iUgmcB7zgbTiNtM/EqPdiUp8JpxtJIvEHq0RWjtXI5WsYWjxsn5Xef8D5w57PAOK9v2cDhxihUSkUfvy8D/OH/H4lsM37+2JObywsJ7QbC6dyL3J87x2rsfEIkOn0e7LrPgzbfyunGk0j7jMxxr2Y8GdCq4kcYIzpF5FvAv+D1WPgMWPMXhG5C+s/+AvAvUAK8BsRAag2xlwFLAQeEREPVsnuHmPMPkfeyBT5eR++6S0h9QEnsKpF8O73DLAP6Ae+YYwZcOSNBMBU7gVwIXCXiPQDA8DXjDHNwX8XU+fnfRjt2Ej8TIxmwp8JnY5CKaWUthkopZTSZKCUUgpNBkoppdBkoJRSCk0GSiml0GSglFIKTQZKKaWA/wd8caeYw7t0uQAAAABJRU5ErkJggg==\n", 1008 | "text/plain": [ 1009 | "
" 1010 | ] 1011 | }, 1012 | "metadata": { 1013 | "needs_background": "light" 1014 | }, 1015 | "output_type": "display_data" 1016 | } 1017 | ], 1018 | "source": [ 1019 | "# solution: bootstrap\n", 1020 | "boot_means = []\n", 1021 | "\n", 1022 | "# run the simulation for 10k times \n", 1023 | "for i in range(10000):\n", 1024 | " #frac=1 means randomize the order of all rows \n", 1025 | " boot_sample = final_data.sample(frac=1,replace=True).groupby('version')['day_1_active'].mean()\n", 1026 | " boot_means.append(boot_sample)\n", 1027 | "\n", 1028 | "# a Pandas DataFrame\n", 1029 | "boot_means = pd.DataFrame(boot_means)\n", 1030 | "\n", 1031 | "# kernel density estimate\n", 1032 | "boot_means.plot(kind = 'kde')" 1033 | ] 1034 | }, 1035 | { 1036 | "cell_type": "code", 1037 | "execution_count": 22, 1038 | "metadata": {}, 1039 | "outputs": [], 1040 | "source": [ 1041 | "# create a new column, diff, which is the difference between the two variants, scaled by the control group\n", 1042 | "boot_means['diff'] = (boot_means['treatment'] - boot_means['control'])/boot_means['control']*100" 1043 | ] 1044 | }, 1045 | { 1046 | "cell_type": "code", 1047 | "execution_count": 23, 1048 | "metadata": {}, 1049 | "outputs": [ 1050 | { 1051 | "data": { 1052 | "text/plain": [ 1053 | "day_1_active 12.603674\n", 1054 | "day_1_active 20.623621\n", 1055 | "day_1_active 17.682652\n", 1056 | "day_1_active 20.093840\n", 1057 | "day_1_active 15.809040\n", 1058 | " ... \n", 1059 | "day_1_active 13.919948\n", 1060 | "day_1_active 17.328947\n", 1061 | "day_1_active 13.024745\n", 1062 | "day_1_active 22.401725\n", 1063 | "day_1_active 19.605505\n", 1064 | "Name: diff, Length: 10000, dtype: float64" 1065 | ] 1066 | }, 1067 | "execution_count": 23, 1068 | "metadata": {}, 1069 | "output_type": "execute_result" 1070 | } 1071 | ], 1072 | "source": [ 1073 | "boot_means['diff']" 1074 | ] 1075 | }, 1076 | { 1077 | "cell_type": "code", 1078 | "execution_count": 24, 1079 | "metadata": {}, 1080 | "outputs": [ 1081 | { 1082 | "data": { 1083 | "text/plain": [ 1084 | "Text(0.5, 0, '% diff in means')" 1085 | ] 1086 | }, 1087 | "execution_count": 24, 1088 | "metadata": {}, 1089 | "output_type": "execute_result" 1090 | }, 1091 | { 1092 | "data": { 1093 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxc9Xnv8c+jfbMkW5J3y5IXwDYGbIwDJTvBLCG4baCBJrc0JaF5paRN07QlSUtTbntv0/aG5N7QNDSkIUnZSkpiWlpIgNAEiLHBBmMbgyxvso2txZYtyVrnuX/MkT0MI0u2dXRm+b5fL7+YOed3zjwaRnrmt5u7IyIikiwv6gBERCQ9KUGIiEhKShAiIpKSEoSIiKSkBCEiIikVRB3AeKmtrfWGhoaowxARySgvvvhim7vXpTqXNQmioaGB9evXRx2GiEhGMbNdI51TE5OIiKSkBCEiIikpQYiISEpKECIikpIShIiIpKQEISIiKSlBiIhISlkzD0IkHbg7L+0+xPqdhxhyZ9GMSt65oJbCfH0Xk8yjBCEyTjp7Bvjsgxt4elvrW47Pryvn2zddRGNteUSRiZwefa0RGQd9g0N88nvrebapnT/74CJevn0VW+64gn/46HIO9Qxw03deoLtvMOowRU6JEoTIOPj6T9/ghZ0d/P1vnM8n3jWPqrJCyooKuHrpDP7xYxeyu6OHf/hZU9RhipwSJQiRM/Tq3k6+9d/NXH/hbK49f+bbzq9snMLVS6fzved30aVahGQQJQiRMzA4FONPf/gKU8qL+LMPLh6x3McvbeRo7yA/3XJgAqMTOTNKECJn4LvP7WTzviP85bVLqCorHLHchfWTmVFVwr+/sm8CoxM5M0oQIqdp25tH+T9PvM5l50zlqnOnn7RsXp5x9dIZPPN6q5qZJGMoQYichu2tXfzOd9cxqaSA//3rSzGzUa95/zlTGRhy1u3omIAIRc6cEoTIKVq3s4MPf/M5egeGuOemi5haWTKm6y6cO5migjye294WcoQi40MT5UROwfPb2/n4d19gZlUp3/34SuprysZ8bUlhPhfWT+bZpvYQIxQZP6pBiIxRe1cft973EnMml/HQpy45peQw7OJ5NWx98whHegdCiFBkfClBiIzRN3+2ncPHBrjro8uprSg+rXssq6/GHV7Z0znO0YmMPyUIkTE41j/E/S/s5trzZ3LWtEmnfZ/z51QDsHHPofEKTSQ0oSYIM7vSzLaZWZOZ3ZbifLGZPRicX2tmDQnnzjOz581ss5ltMrOx9QSKhODJ1w7Q3T/E9RfOPqP7VJUWMq+unI17Do9TZCLhCS1BmFk+cBdwFbAYuNHMkqea3gwccvcFwJ3AV4JrC4AfAJ9y9yXAewE12kpkHn15H1MnFfOOeTVnfK8L5lSzYfdh3H0cIhMJT5g1iJVAk7s3u3s/8ACwOqnMauDe4PHDwGUWH1C+CnjF3V8GcPd2dx8KMVaREQ0OxXi2qZ3LFk0jP2/0+Q6jWTanmvbufloOHRuH6ETCE2aCmAXsSXjeEhxLWcbdB4FOoAY4C3Aze9zMXjKzPwkxTpGT2rL/CF19g1w8b8q43O+82fF+iE171VEt6S3MBJHqq1ZynXqkMgXAO4GPBv/9NTO77G0vYHaLma03s/Wtra3Jp0XGxdrm+Mzni8eheQng7OmTyM8zNu9TgpD0FmaCaAHmJDyfDSSvVHa8TNDvUAV0BMefcfc2d+8BHgOWJ7+Au9/t7ivcfUVdXV0IP4IIvLCzg4aaMqaNccb0aEoK81k4tYJX9x4Zl/uJhCXMBLEOWGhmjWZWBNwArEkqswa4KXh8HfCUx3vuHgfOM7OyIHG8B9gSYqwiI3p1b+fx4anjZcnMKjbv61RHtaS10BJE0KdwK/E/9luBh9x9s5ndYWbXBsXuAWrMrAn4HHBbcO0h4KvEk8xG4CV3/4+wYhUZSUd3P/s7e1k8o3Jc73vurErauvo5eLRvXO8rMp5CXYvJ3R8j3jyUeOz2hMe9wPUjXPsD4kNdRSKzdX+8GWjJzKpxve+5s+L3e3Vv57g1XYmMN82kFjmJLfviCWLRjNOfPZ3KohmVmMHmfeqHkPSlBCFyElv2H2F6ZQk1p7n20kgqigtorCnnVQ11lTSmBCFyEpv3dbJ45vj2PwxbMqtKNQhJa0oQIiMYGIrR3NrN2dPHt3lp2JKZlew9fIxD3f2h3F/kTClBiIyg5dAxBmPOvNryUO5/btDxrVqEpCslCJER7GzrBqAxpASxJGi60oxqSVdKECIjaA45QUwuL2JWdSmvqgYhaUoJQmQEO9q6qCwpYEp5UWivsWRmpWoQkraUIERGsLOth8bacuIr0Ifj3FlV7GjrprtvMLTXEDldShAiI9jR1h1a89KwJTMrcT8xY1sknShBiKTQOzDEvs5jNNZWhPo6w3MslCAkHSlBiKSwq70Hd2ioLQv1daZXllBVWsjWN4+G+joip0MJQiSFHW1dAMwLuQZhZpwzfZJqEJKWlCBEUtjR1gOEX4OA+MJ92948SiymvSEkvShBiKSwo62L2opiJpUUhv5ai2ZMoqd/iN0dPaG/lsipUIIQSWFnW09oS2wkO2d6vKP6tTfVzCTpRQlCJIXmtu4JaV4COGvaJPIMtu5XR7WkFyUIkSRHewdo6+oLfYjrsNKifBpqy9VRLWlHCUIkyc6ggzrsSXKJFk2v5DUNdZU0owQhkqQ5GOI6oQlixiR2d/TQpSU3JI0oQYgk2dHWjRnMrZmYPgiABVPjmxJtP9g1Ya8pMholCJEkO9u6mVlVSklh/oS95oKp8f6O7a1KEJI+Qk0QZnalmW0zsyYzuy3F+WIzezA4v9bMGoLjDWZ2zMw2Bv/+Mcw4RRJNxCJ9yebWlFGQZzSpBiFpJLQEYWb5wF3AVcBi4EYzW5xU7GbgkLsvAO4EvpJwbru7XxD8+1RYcYokcvdIEkRhfh5za8qUICSthFmDWAk0uXuzu/cDDwCrk8qsBu4NHj8MXGZhLr4vMoqO7n6O9A7SMMEJAuLNTGpiknQSZoKYBexJeN4SHEtZxt0HgU6gJjjXaGYbzOwZM3tXqhcws1vMbL2ZrW9tbR3f6CUn7Qi2GZ2oWdSJFkytYFd7DwNDsQl/bZFUwkwQqWoCyauRjVRmP1Dv7suAzwH3mVnl2wq63+3uK9x9RV1d3RkHLDKcIKKoQcyvq2Aw5uxq15pMkh7CTBAtwJyE57OBfSOVMbMCoArocPc+d28HcPcXge3AWSHGKgLEE0RBnjF7cumEv/bwSCb1Q0i6CDNBrAMWmlmjmRUBNwBrksqsAW4KHl8HPOXubmZ1QSc3ZjYPWAg0hxirCAA727upn1JGYf7EjwCfX6ehrpJeCsK6sbsPmtmtwONAPvAdd99sZncA6919DXAP8H0zawI6iCcRgHcDd5jZIDAEfMrdO8KKVWRYc2t3JM1LAOXFBcyoKtFkOUkboSUIAHd/DHgs6djtCY97getTXPdD4IdhxiaSLBZzdrZ3c+mC2shiaKwtpznoBxGJmmZSiwQOHO2ldyA24XMgEs2rK6e5tQt37S4n0VOCEAnsaI1/c48yQTTWVnCkd5BDPQORxSAyTAlCJDDctBNpDSJ47WZ1VEsaUIIQCexo66a4II/plSWRxTCvLkgQ6oeQNKAEIRJobu1iXl0FeXnRrfYyq7qUwnw7PmFPJEpKECKB5rbu49/go1KQn0f9lDI1MUlaUIIQAfoGh9jT0cP8CPsfhs2rq1ANQtKCEoQIsKu9h5jH/zhHbV5tOTvbexiKaairREsJQoQTo4aibmKC+Ciq/sEY+w4fizoUyXFKECLA9mAORDrUIIaH2Wokk0RNCUKE+BpM0yqLqSgOdfWZMRlOUjvUUS0RU4IQIb6C6rza6GsPALUVRUwqLlANQiKnBCE5z92DORDR9z8AmBmNdeUaySSRU4KQnNce7EOdDv0Pw+bVltPcqgQh0VKCkJzXfLyDOj1qEBBftG9f5zF6B4aiDkVymBKE5LzhIa4L0qgG0VhXjnt8hzuRqChBSM5rbuumqCCPmdUTvw/1SIZXdd2hZiaJkBKE5LztB7torCknP8JF+pJpLoSkAyUIyXnpsEhfsvLiAqZVFqujWiKlBCE5rX8wxu6OnrRLEBCvRexo02Q5iY4ShOS03R3xRfHmp1EH9TCt6ipRCzVBmNmVZrbNzJrM7LYU54vN7MHg/Foza0g6X29mXWb2+TDjlNx1YpG+NEwQteUc6hngUHd/1KFIjgotQZhZPnAXcBWwGLjRzBYnFbsZOOTuC4A7ga8knb8T+M+wYhTZnoZzIIapo1qiFmYNYiXQ5O7N7t4PPACsTiqzGrg3ePwwcJmZGYCZ/SrQDGwOMUbJcc2tXdRWFFNZUhh1KG8znCDUzCRRCTNBzAL2JDxvCY6lLOPug0AnUGNm5cCfAn95shcws1vMbL2ZrW9tbR23wCV3pOMIpmFzppRRkGfaflQiE2aCSDWoPHmLrJHK/CVwp7uf9DfD3e929xXuvqKuru40w5Rc1tzalZYd1ACFwf7UqkFIVMJc/L4FmJPwfDawb4QyLWZWAFQBHcA7gOvM7G+BaiBmZr3u/o0Q45Uc09Hdz6GeAeanaQ0Choe6KkFINMJMEOuAhWbWCOwFbgB+M6nMGuAm4HngOuApd3fgXcMFzOzLQJeSg4y3dNpmdCTz6sr5RVMbsZiTl0YzvSU3hNbEFPQp3Ao8DmwFHnL3zWZ2h5ldGxS7h3ifQxPwOeBtQ2FFwnJ8Fdc02SgolcbaCvoGY+zr1P7UMvFC3V/R3R8DHks6dnvC417g+lHu8eVQgpOct72ti8J8Y/bk9FmkL1niSKbZk8sijkZyzZhqEGb2QzP7oJlp5rVkjebWbubWlFOQn74f6+H+Ea3JJFEY62/GN4n3H7xhZn9jZueEGJPIhIiPYErf/geAuknFlBflq6NaIjGmBOHuP3X3jwLLgZ3AT8zsOTP7uJml3wwjkVEMDMXY1d6TlktsJBren3q75kJIBMZctzazGuC3gU8AG4CvE08YPwklMpEQ7enoYTDmxzfmSWdnTZvE1v1HiQ/wE5k4Y+2D+Dfg50AZ8CF3v9bdH3T3zwDp/RVMJIUT+1Cn/8d36awq2rr6OHi0L+pQJMeMdRTTt4MRSceZWbG797n7ihDiEglVc7DPQrr3QUA8QQBsaulk2uKSiKORXDLWJqa/SnHs+fEMRGQiNbd2U1NeRHVZUdShjGrRjErMYNPezqhDkRxz0hqEmU0nvqBeqZkt48TaSZXEm5tEMlJza/ou0pesvLiA+XUVvKoEIRNstCamK4h3TM8Gvppw/CjwxZBiEgnd9tYuPrBoWtRhjNnSWVU829QWdRiSY06aINz9XuBeM/uwu/9wgmISCVVnzwDt3f0ZU4MAOHdWFY9s2MvBI71MrVQ/hEyM0ZqYPubuPwAazOxzyefd/aspLhNJa9vb0neb0ZEMd1S/0tLJB9RRLRNktE7q4a9YFcCkFP9EMk5zGm8zOpJzZ1WSn2ds3HM46lAkh4zWxPSt4L8n3dlNJJM0t3ZRkGfUT8mccRZlRQWcM30SG/YcijoUySFjnSj3t2ZWaWaFZvakmbWZ2cfCDk4kDNtbu6ivKaMwjRfpS2VZfTUv7+lkKKYZ1TIxxvobssrdjwDXEN8F7izgj0OLSiREza3dab0HxEiW10+mq2+QpoNal0kmxlgTxPCCfFcD97t7R0jxiIRqKObsau/JiBnUyZbVTwZgw241M8nEGGuCeNTMXgNWAE+aWR3QG15YIuFoOdRD/1AsozqohzXUlFFdVsiG3eqolokx1uW+bwMuAVa4+wDQDawOMzCRMAyPYJqfQUNch5kZy+ZUq6NaJsypbDm6iPh8iMRrvjfO8YiEanhfhUyaA5FoWf1kfvZ6K0d6B6gs0VYsEq4xJQgz+z4wH9gIDAWHHSUIyTDbW7upLitkSnn6L9KXyrL6atzhlT2dvHNhbdThSJYbaw1iBbDYtWOJZLjm1q6M2CRoJOfPqcYs3lGtBCFhG2sn9avA9FO9uZldaWbbzKzJzG5Lcb7YzB4Mzq81s4bg+Eoz2xj8e9nMfu1UX1sklea27oxtXgKoLClkQV0FGzSjWibAWGsQtcAWM3sBOL6tlbtfO9IFZpYP3AVcTnzuxDozW+PuWxKK3QwccvcFZnYD8BXgI8QT0gp3HzSzGcDLZvaouw+eyg8nkuhI7wCtR/sycgRTomX11fxkywHcHTMb/QKR0zTWBPHl07j3SqDJ3ZsBzOwB4iOfEhPE6oR7Pwx8w8zM3XsSypQQ7+8QOSOZPIIp0bL6yTy0voVd7T00ZHBzmaS/sQ5zfQbYCRQGj9cBL41y2SxgT8LzluBYyjJB7aATqAEws3eY2WZgE/CpVLUHM7vFzNab2frW1tax/CiSw5pbM2eb0ZNZVl8NoOGuErqxrsX0SeLf8L8VHJoF/Gi0y1IcS64JjFjG3de6+xLgIuALZva2NY7d/W53X+HuK+rq6kYJR3Jdc2s3+XlG/ZTMThALp06ivChfE+YkdGPtpP494FLgCIC7vwFMHeWaFmBOwvPZwL6RygTzK6qAtyzj4e5biU/MO3eMsYqk1NzWxZzJpRQVZNYifcny84zz51QrQUjoxvqb0ufu/cNPgj/mo/ULrAMWmlmjmRUBNwBrksqsAW4KHl8HPOXuHlxTELzWXOBs4k1cIqctvg91Zvc/DFtWX83W/Uc41j80emGR0zTWBPGMmX0RKDWzy4F/BR492QVBn8GtwOPAVuAhd99sZneY2fDop3uAGjNrAj4HDA+FfSfxkUsbgUeAT7u7NuSV0zYUc5rbujO+/2HYsjmTGYw5r+7rjDoUyWJjHcV0G/EhqZuA3wUeA7492kXu/lhQNvHY7QmPe4HrU1z3feD7Y4xNZFT7Dh+jfzCWNTWIC4Y7qncf4qKGKRFHI9lqTAnC3WNm9iPgR+6u4UKScY6vwZQlw0JrK4qpn1KmfggJ1UmbmCzuy2bWBrwGbDOzVjO7/WTXiaSbE/tQZ0cNAuL9EEoQEqbR+iA+S3z00kXuXuPuU4B3AJea2R+GHp3IOGlu62JSSQG1FZm5SF8qy+sn8+aRXvZ3Hos6FMlSoyWI3wJudPcdwweCmdEfC86JZIThEUzZtDTF8QlzqkVISEZLEIWpRg8F/RBajF4yxvbWrqwZwTTsnOmVFBfkaQtSCc1oCaL/NM+JpI2uvkEOHOnL+DWYkhUV5LF0VhUvqQYhIRltFNP5ZnYkxXEjvoieSNrbMdxBnSUjmBItq6/m3ud30T8Yy/gZ4pJ+TvqJcvd8d69M8W+Su6uJSTJCU+tRAOZPza4aBMRXdu0fjLF1f6rvcSJnRl85JOu9caCLgjyjoSY7axCA+iEkFEoQkvXeONhFQ215VjbBzKgqZXpliXaYk1Bk32+MSJKmg10szMLmpWGaMCdhUYKQrNY7MMSu9u6sTxC7O3po6+obvbDIKVCCkKy2o62bmMPCaZOiDiU0y+onA7BRtQgZZ0oQktXeOBhfpG/htOytQZw7s4qCPNMWpDLulCAkqzUdOEqeQWMWzoEYVlqUz6IZleqHkHGnBCFZ7Y2DXTTUlFNckB91KKFaVl/Ny3sOMxQbbaNHkbFTgpCs9sbBLhZkcQf1sGX11XT3D/HGwaNRhyJZRAlCslb/YIydbd1Z3f8wbNmceEe1mplkPClBSNZqOtjFYMw5K4tHMA2bW1PG5LJCzaiWcaUEIVlr875OAJbMrIo4kvCZGcvqJ6sGIeNKCUKy1uZ9RygtzM/qEUyJls2p5o2DXXQeG4g6FMkSoSYIM7vSzLaZWZOZ3ZbifLGZPRicX2tmDcHxy83sRTPbFPz3/WHGKdlpy74jLJoxify87NlF7mSWz433Q7zSolqEjI/QEoSZ5QN3AVcBi4EbzWxxUrGbgUPuvgC4E/hKcLwN+JC7LwVuAr4fVpySnWIxZ8v+IznRvDTsvNlVmKmjWsZPmDWIlUCTuze7ez/wALA6qcxq4N7g8cPAZWZm7r7B3fcFxzcDJWZWHGKskmX2HOqhq2+QJTMrow5lwkwqKeSsqZPUUS3jJswEMQvYk/C8JTiWsoy7DwKdQE1SmQ8DG9z9bSuRmdktZrbezNa3traOW+CS+Tbvi2+gk0s1CAhWdt1zGHdNmJMzF2aCSNXwm/ypPWkZM1tCvNnpd1O9gLvf7e4r3H1FXV3daQcq2Wfzvk7y8ywn5kAkWlZfzeGeAXa290QdimSBMBNECzAn4flsYN9IZcysAKgCOoLns4FHgN9y9+0hxilZ6OU9nZw9bRIlhdm9xEay4ZVd1cwk4yHMBLEOWGhmjWZWBNwArEkqs4Z4JzTAdcBT7u5mVg38B/AFd382xBglCw3FnI17DrN8bnXUoUy4BXUVTCou4MVdShBy5kJLEEGfwq3A48BW4CF332xmd5jZtUGxe4AaM2sCPgcMD4W9FVgA/LmZbQz+TQ0rVskurx84SlffIMuDb9O5JC/PWNk4hZ+/0aZ+CDljBWHe3N0fAx5LOnZ7wuNe4PoU1/0V8FdhxibZ66WgeSUXEwTAe8+ZypOvHaS5rZv5dbnVByPjSzOpJeu8tOswU8qLmFtTFnUokXjf2fEBG0+/djDiSCTTKUFI1tmw+xDL66sxy40Z1MlmTy7jrGkVPL1NCULOjBKEZJVD3f00t3UfH82Tq953zlRe2NFBV99g1KFIBlOCkKyydkc7ACsbp0QcSbQuO2caA0OuZiY5I0oQklWe395OaWE+58/OvSGuiS6cO5m6ScU8tml/1KFIBlOCkKzy3PZ2VjRMpqggtz/a+XnGVedO5+ltB+npVzOTnJ7c/i2SrNJ6tI83DnbxK/Nrow4lLVy9dAa9AzGeUjOTnCYlCMkav2yO9z9cMj95vcfcdFHDFGor1Mwkp08JQrLGc9vbmVRcwLk5tMT3yQw3Mz31mpqZ5PQoQUjW+GVzOysbp1CQr4/1sKuWTqd3IMbTr2k5fDl1+k2SrLCno4cdbd1cukD9D4ne0VhDbUWRmpnktChBSFb4WTBr+L1na1+QRPl5xhVL4s1MvQNDUYcjGUYJQrLC09taqZ9SRmNtedShpJ0rlkzn2MAQv3ijLepQJMMoQUjG6x0Y4rntbbzv7LqcXX/pZC6eV8OkkgKe2PJm1KFIhlGCkIz3wo4OegdivPdsbRmSSlFBHu8/Zyo/3XqQwaFY1OFIBlGCkIz39LaDFBXkcfE8zX8YyarF0+no7tdOc3JKlCAk4z2zrZVL5tVQWpRb+0+fivecXUdRQR5PbDkQdSiSQZQgJKPtbOumua1bo5dGUVFcwDsX1PLElje1FamMmRKEZLThjtcPLJoWcSTpb9XiaezpOMbW/UejDkUyhBKEZLTHNx9gycxK5kzJze1FT8UHFk/DDI1mkjFTgpCMdfBoLy/tPsSqxdOjDiUj1FYUs2LuZH6ifggZIyUIyVhPbj2IO6xaoualsVq1eDqb9x2h5VBP1KFIBgg1QZjZlWa2zcyazOy2FOeLzezB4PxaM2sIjteY2dNm1mVm3wgzRslcT2x+kzlTSjln+qSoQ8kYly+OJ1PVImQsQksQZpYP3AVcBSwGbjSzxUnFbgYOufsC4E7gK8HxXuDPgc+HFZ9ktqO9Azzb1M4Vi6dr9vQpaKgt56xpFTyxWQlCRhdmDWIl0OTuze7eDzwArE4qsxq4N3j8MHCZmZm7d7v7L4gnCpG3eeb1VvqHYqxaov6HU7Vq8XRe2NnBoe7+qEORNBdmgpgF7El43hIcS1nG3QeBTmDM02HN7BYzW29m61tbtd59Lnli8wFqyou4cO7kqEPJOKuWTGMo5tqKVEYVZoJIVe9PnqEzljIjcve73X2Fu6+oq9NEqVzROzDEk1sP8IFF08jPU/PSqVo6q4rplSUa7iqjCjNBtABzEp7PBvaNVMbMCoAqoCPEmCQL/GzbQbr7h/jQ+TOjDiUjmRmrlkzjmddbOdavPSJkZGEmiHXAQjNrNLMi4AZgTVKZNcBNwePrgKdc6wDIKB59eT815UVcPG9K1KFkrMsXT6N3IMbP31DTrIwstAQR9CncCjwObAUecvfNZnaHmV0bFLsHqDGzJuBzwPGhsGa2E/gq8Ntm1pJiBJTkoO6+QZ587QBXL52hvafPwMXzaphSXsSPX06u1IucUBDmzd39MeCxpGO3JzzuBa4f4dqGMGOTzPTkawfpHYhxzXkzog4loxXm5/Gh82Zw/7o9dB4boKq0MOqQJA3pK5hklEdf3se0ymIualDz0pn69eWz6R+M8Z+b9kcdiqQpJQjJGIe6+3lmWysfXDqTPI1eOmPnza5ifl05//bS3qhDkTSlBCEZ45ENe+kfinH9itlRh5IVzIzrV8zhhZ0dvPbmkajDkTSkBCEZwd15aP0ezp9dxaIZlVGHkzVuuGgOJYV5/PMvdkYdiqQhJQjJCK+0dPLam0f5jYvmjF5Yxqy6rIgPL5/NIxv30t7VF3U4kmaUICQjfO/5XZQW5mtyXAg+fmkD/YMx/vnZnVGHImlGCULS3oEjvax5eS8fuWgOlSUajjneFkydxDXnzeCeX+zg4BGtjyknKEFI2vvuczsZijm/c2lj1KFkrc+vOpuBoRh//8S2qEORNKIEIWntSO8A//LLXVyxZDr1Ndp3OiwNteXc/M5GHlrfouU35DglCElrdz/TzJHeQX7vfQuiDiXr/eHlZzGvrpw//tdX1NQkgBKEpLHWo33c84sdXHPeDM6dVRV1OFmvpDCf/3fjMjqPDfDJ762nq28w6pAkYkoQkra+8dQb9A/F+KNVZ0cdSs5YMrOKr99wAa/uO8Jv3bOWI70DUYckEVKCkLS0df8RfrB2NzdcNIfG2vKow8kpq5ZM5xs3LmPT3k5+4x+fZ+/hY1GHJBFRgpC0E4s5X3pkE9WlhfzxFao9ROGqpTP4zm9fxN5Dx1j9jWfZuOdw1CFJBJQgJO08sG4PL+0+zBevXkR1WVHU4eSsdy2s498+/SuUFuXxkW89z6PaOyLnKEFIWmlu7eKv/mMLl8yr4deXz4o6nJy3cNokfvTpS1k6q4rP3L+BLz2yST8bNsYAAAuPSURBVNuU5hAlCEkbvQNDfOb+DRQV5PHVj5yPmZb0Tgc1FcXc98mL+d13z+Nf1u5m1dee4ccb9xKLaXfgbKcEIWlhKOZ89oGNbN53hL+77nxmVJVGHZIkKCrI4wtXL+K+T76DiuJC/uCBjVz19Z/z8Ist9A/Gog5PQqIEIZHrHRji1vte4r82v8mfX7OYyxdPizokGcGvzK/lPz7zTr72kQsA+Py/vsy7//Zp7v7v7RzVkNisY+7ZUU1csWKFr1+/Puow5BS4O883t/MXP95MU2sXX7p6EZ9417yow5Ixcneeeb2Vu/+7mee2tzOpuIDffEc9H7+0kelVJVGHJ2NkZi+6+4qU55QgZKLtPXyMJza/ySMb9vJKSyezqkv5X7++lPecVRd1aHKaNrV08q3/3s5jm/aTn2esvmAWt7x7HmdNmxR1aDKKyBKEmV0JfB3IB77t7n+TdL4Y+B5wIdAOfMTddwbnvgDcDAwBv+/uj5/stZQg0pe7s+3AUX665QCPbz7Apr2dAJw9bRL/45K5fHj5bEqL8iOOUsbDno4e7vnFDh5ct4djA0NcMq+GK8+dzuWLpzGzWv1K6SiSBGFm+cDrwOVAC7AOuNHdtySU+TRwnrt/ysxuAH7N3T9iZouB+4GVwEzgp8BZ7j7i+DoliInh7vQPxegdiNE7MMSx/iGODcT/9Qb/jvXHOHysn93tPTS3dfPSrkO0d/cDcMGcaq5YMp0rlkxjXl1FxD+NhOVQdz8/+OUufvzyPpoOdgHQWFvOebOrmF9XwdRJxVSVFlKQn0dBnlGQbxTl51FWVEBpUT5lwb/SonyK8vM0oi1EJ0sQBSG+7kqgyd2bgyAeAFYDWxLKrAa+HDx+GPiGxT8Jq4EH3L0P2GFmTcH9nh/vIF978wi33rcBiP/xG/aWtOlvPzZSWT9e1t92LPnxWO+VeL+3HhvpvqOVffvrvTXG1PHEYs6xgSHGOrqxKD+P2VNKec/ZdVw8r4Z3L6xT23SOmFxexGcuW8hnLlvI9tYuntx6gBd3HWJtcwc/3nhqE+7y84yywniyKCnMJy/IFWbG8bRhHH/8luM54r1n1/GlDy4e9/uGmSBmAXsSnrcA7xipjLsPmlknUBMc/2XStW+bNWVmtwC3ANTX159WkCUF+Zyd2E5qKR8e/wbz1mNjL/vW+1pC2VTXj1J2hBunime0e40U+1teITiUZ0Zpwi9q/HEepYX5FA8/D85XFBcwrbKE/Lxc+1WVZPPrKpifUFvsGxyi9WgfXX2DDA45A0MxBmNO30CMYwND9PQPcqx/iJ6gdtrTPxh/3B+voTrxLzQnvtwkfB3zt345yxXTKsP54hVmgkj1lyH5/9xIZcZyLe5+N3A3xJuYTjVAiG+UctdHl5/OpSJyGooL8pk9WZs/ZYIw50G0AHMSns8GkuuWx8uYWQFQBXSM8VoREQlRmAliHbDQzBrNrAi4AViTVGYNcFPw+DrgKY83gK8BbjCzYjNrBBYCL4QYq4iIJAmtiSnoU7gVeJz4MNfvuPtmM7sDWO/ua4B7gO8HndAdxJMIQbmHiHdoDwK/d7IRTCIiMv40UU5EJIedbJir1mISEZGUlCBERCQlJQgREUlJCUJERFLKmk5qM2sFdkUdR5JaoC3qINKA3ocT9F6coPfihCjfi7nunnIp5axJEOnIzNaPNDogl+h9OEHvxQl6L05I1/dCTUwiIpKSEoSIiKSkBBGuu6MOIE3ofThB78UJei9OSMv3Qn0QIiKSkmoQIiKSkhKEiIikpAQxzszs78zsNTN7xcweMbPqhHNfMLMmM9tmZldEGedEMbMrg5+3ycxuizqeiWRmc8zsaTPbamabzewPguNTzOwnZvZG8N/JUcc6Ecws38w2mNm/B88bzWxt8D48GGwLkPXMrNrMHg7+Tmw1s0vS9TOhBDH+fgKc6+7nAa8DXwAws8XElzNfAlwJ/IOZ5UcW5QQIfr67gKuAxcCNwfuQKwaBP3L3RcDFwO8FP/9twJPuvhB4MnieC/4A2Jrw/CvAncH7cAi4OZKoJt7Xgf9y93OA84m/J2n5mVCCGGfu/oS7DwZPf0l8NzyA1cAD7t7n7juAJmBlFDFOoJVAk7s3u3s/8ADx9yEnuPt+d38peHyU+B+CWcTfg3uDYvcCvxpNhBPHzGYDHwS+HTw34P3Aw0GRXHkfKoF3E98LB3fvd/fDpOlnQgkiXL8D/GfweBawJ+FcS3Asm+Xiz5ySmTUAy4C1wDR33w/xJAJMjS6yCfM14E+AWPC8Bjic8GUqVz4b84BW4J+D5rZvm1k5afqZUII4DWb2UzN7NcW/1QllvkS8ieFfhg+luFW2jzHOxZ/5bcysAvgh8Fl3PxJ1PBPNzK4BDrr7i4mHUxTNhc9GAbAc+Ka7LwO6SZPmpFRC23I0m7n7B0523sxuAq4BLvMTE01agDkJxWYD+8KJMG3k4s/8FmZWSDw5/Iu7/1tw+ICZzXD3/WY2AzgYXYQT4lLgWjO7GigBKonXKKrNrCCoReTKZ6MFaHH3tcHzh4kniLT8TKgGMc7M7ErgT4Fr3b0n4dQa4AYzKzazRmAh8EIUMU6gdcDCYLRKEfFO+jURxzRhgnb2e4Ct7v7VhFNrgJuCxzcBP57o2CaSu3/B3We7ewPxz8BT7v5R4GnguqBY1r8PAO7+JrDHzM4ODl0GbCFNPxOaST3OzKwJKAbag0O/dPdPBee+RLxfYpB4c8N/pr5L9gi+NX4NyAe+4+5/HXFIE8bM3gn8HNjEibb3LxLvh3gIqAd2A9e7e0ckQU4wM3sv8Hl3v8bM5hEfuDAF2AB8zN37ooxvIpjZBcQ764uAZuDjxL+sp91nQglCRERSUhOTiIikpAQhIiIpKUGIiEhKShAiIpKSEoSIiKSkBCFZw8zqzOwXwaz2X004/mMzmznGe3QF/51pZg8nHL8/WKH3D83sHDPbGCyVMD/p+scSV/AVyWQa5ipZw8x+HzhGfGz9f7n7pWb2IWC5u//lGO/R5e4VScemA2vdfW7w/Dag1N3/Ynx/ApH0ohqEZJMBoJT4RMWYmRUAnwX+bqQLglnez5vZOjP7nwnHG8zs1eDpE8DUoNbwF8E9P2FmT6e4304zqw2u32pm/xTsBfGEmZWmKP9dM/tmsG9Es5m9x8y+E1z73YRyq4I4XzKzfw3Wd8LMbg9if9XM7g5mb2NmPzOzr5jZC2b2upm9Kzi+JDi2MagRLTzVN1lyhxKEZJP7gCuA/wK+DHwa+F7SkifJvk584bSLgDdHKHMtsN3dLwhqIv9IfB+D940Sz0LgLndfAhwGPjxCucnEl77+Q+BR4E7i+4YsNbMLzKwW+DPgA+6+HFgPfC649hvufpG7n0s8OV6TcN8Cd19JPKEN13Y+BXzd3S8AVhBfG0gkJSUIyRru3unuH3T3FcBLxP9Y/jD4Fv+wmV2S4rJLgfuDx98f55B2uPvG4PGLQMMI5R4NFnXcBBxw903uHgM2B9dcTHzDpWfNbCPxtXrmBte+z+K7sm0inmSWJNx3eHHAxNd+Hviimf0pMNfdj53ZjyjZTKu5Sra6Hfhr4EbifyDvI74AWqpv/WF1xCWuKzRE/Bv+ycrFkq6JEf8dHQJ+4u43Jl5kZiXAPwAr3H2PmX2Z+GqpyfcdCu6Du99nZmuJb97zuJl9wt2fOtUfTHKDahCSdYJ29Znu/gxQRvwPrfPWP57DniW+wijARycmwlP2S+BSM1sAYGZlZnYWJ36etqBP4rqRbjAsWCCv2d3/L/EVRM8LKWbJAkoQko3+mnibPcSbj36b+B/Zv09R9g+I7xW9DqiakOhOkbu3Ev8Z7jezV4j/LOcEW1X+E/GmqR8RX159NB8BXg2aqs4BvhdK0JIVNMxVRERSUg1CRERSUoIQEZGUlCBERCQlJQgREUlJCUJERFJSghARkZSUIEREJKX/D5P9mbNTjDp2AAAAAElFTkSuQmCC\n", 1094 | "text/plain": [ 1095 | "
" 1096 | ] 1097 | }, 1098 | "metadata": { 1099 | "needs_background": "light" 1100 | }, 1101 | "output_type": "display_data" 1102 | } 1103 | ], 1104 | "source": [ 1105 | "# plot the bootstrap sample difference \n", 1106 | "ax = boot_means['diff'].plot(kind = 'kde')\n", 1107 | "ax.set_xlabel(\"% diff in means\")" 1108 | ] 1109 | }, 1110 | { 1111 | "cell_type": "code", 1112 | "execution_count": 25, 1113 | "metadata": {}, 1114 | "outputs": [ 1115 | { 1116 | "data": { 1117 | "text/html": [ 1118 | "
\n", 1119 | "\n", 1132 | "\n", 1133 | " \n", 1134 | " \n", 1135 | " \n", 1136 | " \n", 1137 | " \n", 1138 | " \n", 1139 | " \n", 1140 | " \n", 1141 | " \n", 1142 | " \n", 1143 | " \n", 1144 | " \n", 1145 | " \n", 1146 | " \n", 1147 | " \n", 1148 | " \n", 1149 | " \n", 1150 | " \n", 1151 | " \n", 1152 | " \n", 1153 | " \n", 1154 | " \n", 1155 | " \n", 1156 | " \n", 1157 | " \n", 1158 | " \n", 1159 | " \n", 1160 | " \n", 1161 | " \n", 1162 | " \n", 1163 | " \n", 1164 | " \n", 1165 | " \n", 1166 | " \n", 1167 | " \n", 1168 | " \n", 1169 | " \n", 1170 | " \n", 1171 | " \n", 1172 | " \n", 1173 | " \n", 1174 | " \n", 1175 | " \n", 1176 | " \n", 1177 | " \n", 1178 | " \n", 1179 | " \n", 1180 | " \n", 1181 | " \n", 1182 | " \n", 1183 | " \n", 1184 | " \n", 1185 | " \n", 1186 | " \n", 1187 | " \n", 1188 | " \n", 1189 | " \n", 1190 | " \n", 1191 | " \n", 1192 | " \n", 1193 | " \n", 1194 | " \n", 1195 | " \n", 1196 | " \n", 1197 | " \n", 1198 | " \n", 1199 | " \n", 1200 | " \n", 1201 | " \n", 1202 | " \n", 1203 | " \n", 1204 | " \n", 1205 | " \n", 1206 | " \n", 1207 | " \n", 1208 | " \n", 1209 | "
versioncontroltreatmentdiff
day_1_active0.3170580.35701912.603674
day_1_active0.2916410.35178820.623621
day_1_active0.2878320.33872817.682652
day_1_active0.2687770.32278520.093840
day_1_active0.3071610.35572015.809040
............
day_1_active0.2996900.34140713.919948
day_1_active0.2941540.34512817.328947
day_1_active0.3141620.35508113.024745
day_1_active0.2924590.35797422.401725
day_1_active0.2937190.35130419.605505
\n", 1210 | "

9996 rows × 3 columns

\n", 1211 | "
" 1212 | ], 1213 | "text/plain": [ 1214 | "version control treatment diff\n", 1215 | "day_1_active 0.317058 0.357019 12.603674\n", 1216 | "day_1_active 0.291641 0.351788 20.623621\n", 1217 | "day_1_active 0.287832 0.338728 17.682652\n", 1218 | "day_1_active 0.268777 0.322785 20.093840\n", 1219 | "day_1_active 0.307161 0.355720 15.809040\n", 1220 | "... ... ... ...\n", 1221 | "day_1_active 0.299690 0.341407 13.919948\n", 1222 | "day_1_active 0.294154 0.345128 17.328947\n", 1223 | "day_1_active 0.314162 0.355081 13.024745\n", 1224 | "day_1_active 0.292459 0.357974 22.401725\n", 1225 | "day_1_active 0.293719 0.351304 19.605505\n", 1226 | "\n", 1227 | "[9996 rows x 3 columns]" 1228 | ] 1229 | }, 1230 | "execution_count": 25, 1231 | "metadata": {}, 1232 | "output_type": "execute_result" 1233 | } 1234 | ], 1235 | "source": [ 1236 | "boot_means[boot_means['diff'] > 0]" 1237 | ] 1238 | }, 1239 | { 1240 | "cell_type": "code", 1241 | "execution_count": 26, 1242 | "metadata": {}, 1243 | "outputs": [ 1244 | { 1245 | "data": { 1246 | "text/plain": [ 1247 | "0.9996" 1248 | ] 1249 | }, 1250 | "execution_count": 26, 1251 | "metadata": {}, 1252 | "output_type": "execute_result" 1253 | } 1254 | ], 1255 | "source": [ 1256 | "# p value \n", 1257 | "p = (boot_means['diff'] >0).sum()/len(boot_means)\n", 1258 | "p" 1259 | ] 1260 | }, 1261 | { 1262 | "cell_type": "markdown", 1263 | "metadata": {}, 1264 | "source": [ 1265 | "# Conclusion 1: treatment has a better performance than the control on 1-day user retention 99.96% of the time." 1266 | ] 1267 | }, 1268 | { 1269 | "cell_type": "markdown", 1270 | "metadata": {}, 1271 | "source": [ 1272 | "# Metric 7: 7-day retention by AB-Group" 1273 | ] 1274 | }, 1275 | { 1276 | "cell_type": "code", 1277 | "execution_count": 37, 1278 | "metadata": {}, 1279 | "outputs": [], 1280 | "source": [ 1281 | "boot_7d = []\n", 1282 | "\n", 1283 | "for i in range(10000):\n", 1284 | " #set frac=1 --> sample all rows\n", 1285 | " boot_mean = final_data.sample(frac=1,replace=True).groupby('version')['day_7_active'].mean() \n", 1286 | " boot_7d.append(boot_mean)\n", 1287 | " \n", 1288 | "boot_7d = pd.DataFrame(boot_7d)\n", 1289 | "\n", 1290 | "boot_7d['diff'] = (boot_7d['treatment'] - boot_7d['control'])/boot_7d['control'] *100" 1291 | ] 1292 | }, 1293 | { 1294 | "cell_type": "code", 1295 | "execution_count": 40, 1296 | "metadata": {}, 1297 | "outputs": [ 1298 | { 1299 | "data": { 1300 | "text/plain": [ 1301 | "Text(0.5, 0, '% diff in means')" 1302 | ] 1303 | }, 1304 | "execution_count": 40, 1305 | "metadata": {}, 1306 | "output_type": "execute_result" 1307 | }, 1308 | { 1309 | "data": { 1310 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxcZ33v8c9Puy3JkizJtiwv8hZvcVbHsRMoSwokBOLQQkmA27SFpi1Nt9zeEgg3pbTclm659JKmDQ1tgEJIExZTAgESoCXEa+LEdmzHsmXHtizLsmxttvbf/eOcSSbKSBrbGp1Zvu/XSy+fOcvMT/JovjrP85znmLsjIiIyUl7UBYiISHpSQIiISEIKCBERSUgBISIiCSkgREQkoYKoC5goNTU13tDQEHUZIiIZZdu2bW3uXptoW9YERENDA1u3bo26DBGRjGJmh0bbpiYmERFJSAEhIiIJKSBERCQhBYSIiCSkgBARkYQUECIikpACQkREEsqa6yBEonKmf5AfvnicI6fOsnhGGW9dNoPCfP3tJZlPASFyAX760gnu/Pp2Tvb0v7Ju2axyvvhrVzG7ckqElYlcOP2ZI3Kefrynld/4ty3Ulhfz9dvXsuvP3sH9H7yCo6fO8uv/uoXuvsGoSxS5ICkNCDO73sz2mlmjmd2VYHuxmX093L7JzBrC9YVm9pCZ7TCz3Wb28VTWKXKumk+f5fe+9hzL68p59Heu4eqF1ZQWF3DDqjr+8UNXsPd4F//vyX1RlylyQVIWEGaWD9wH3ACsAG41sxUjdvswcMrdFwP3Ap8N178PKHb3VcCVwG/FwkMkHdzz7Z0MDTv3f/BKyopf21L7xiW1vO/KOXzx6SYOt5+JqEKRC5fKM4g1QKO7H3D3fuBhYP2IfdYDD4XLjwLXmZkBDpSaWQEwBegHOlNYq0jSnn35FD/a3crvXbeYudOnJtznzrdfxLDDVzaOOg+aSNpLZUDUA4fjHh8J1yXcx90HgQ6gmiAseoBjwMvA37p7+8gXMLPbzWyrmW09ceLExH8HIgnc91Qj00uLuG1dw6j71FVM4fqVs3h4y2F6B4YmrziRCZTKgLAE6zzJfdYAQ8BsYAHwP81s4et2dH/A3Ve7++ra2oTTmYtMqJaOXn68t5Vb18yltHjsQYC3rJlLx9kBfrJXf7xIZkplQBwB5sY9ngM0j7ZP2JxUAbQDHwC+7+4D7t4KPA2sTmGtIkl57NkjDDu878q54+67bmE100uLeHzHsUmoTGTipTIgtgBLzGyBmRUBtwAbRuyzAbgtXH4v8JS7O0Gz0lstUAqsBfaksFaRcbk7j207wpqG6TTUlI67f0F+Hu9YOZMndx+nb1DNTJJ5UhYQYZ/CHcATwG7gEXffZWafNrObwt0eBKrNrBG4E4gNhb0PKAN2EgTNv7r7C6mqVSQZ+1q7OdDWw/rLZyd9zHXLZtLTP8S2Q6dSWJlIaqT0Smp3fxx4fMS6e+KWewmGtI48rjvRepEoPbm7FQg+9JO1dlE1BXnGz/a1cc2imlSVJpISupJaJElP7TnOxfXTmFVRkvQxZcUFXDGviv/e15bCykRSQwEhkoRTPf1sO3SKt57D2UPMG5fUsLO5g/a4+ZpEMoECQiQJzxw4ybDDmy469+HUb1hSgzs8s/9kCioTSR0FhEgSNh04ydSifC6ZU3HOx66qr6CsuIBnDqiZSTKLAkIkCZua2rlyftV53eehID+Pqxqq+LnOICTDKCBExnGqp589LV2sXVh93s+xblE1B070cLyzdwIrE0ktBYTIODY1BdOAXb1g+nk/x7qFwRDXjQd0FiGZQwEhMo5NTScpKczjkjmV5/0cK2ZPo7ykQB3VklEUECLj2HboFJfNraSo4Px/XfLzjKsXTOcZnUFIBlFAiIyhd2CI3cc6uXxe1QU/19qF1Rw6eYbm02cnoDKR1FNAiIxhV3MnA0POZXPPv3kpZt2ioJNbzUySKRQQImN4/vBpgAkJiOWzplE5tVDNTJIxFBAiY9h++DR1FSXMnJb8/EujyYv1Q+gMQjKEAkJkDNsPn56Qs4eYdQurOXr6LIfbz0zYc4qkigJCZBTtPf283H6GSycyIMIpv3UWIZlAASEyionsf4i5aGYZ1aVF6oeQjKCAEBnFc4dPk2fBZHsTxcxYu7CaZ/afJLi7rkj6UkCIjOL5w6e5aGY5pcUTe+PFtYuqaens5UBbz4Q+r8hEU0CIJODu7DjacV7Te4/nLUuDe0o8satlwp9bZCIpIEQSONbRS3tPPxdPYPNSzJyqqVw2t5LHdxyb8OcWmUgKCJEEdjV3ArBy9rSUPP+Nq+rYebSTQyfVzCTpSwEhksCu5g7MYNms1ATEDatmAfCfL+gsQtKXAkIkgV3NnSysKZ3wDuqYOVVTWT2/im89d1SjmSRtKSBEEth1tIOVsye+/yHezZfXs6+1mxePdab0dUTOlwJCZIRTPf00d/SmrP8h5sZVdRTkGd/e3pzS1xE5XwoIkRFe7aBO7RlEVWkRb15ay7e3H2VoWM1Mkn4UECIj7GruAFI3ginezZfXc7yzj02aekPSkAJCZIRdzZ3MriihqrQo5a/1i8tnUlZcwLe2H035a4mcKwWEyAg7mztYkeLmpZiSwnzesXIW39vRQu/A0KS8pkiyFBAicXr6Bmlq6+Hi+tQ3L8XcfPlsuvoGeWpP66S9pkgyFBAicfa0dOKe+g7qeNcsqqG2vJhvPadmJkkvCgiROLuPdQGwvK580l4zP8+46dLZ/HhvK529A5P2uiLjUUCIxNnT0kl5cQH1lVMm9XXftmImA0PORt1pTtKIAkIkzt6WLpbOKsfMJvV1L59XyZTCfJ5ubJvU1xUZiwJCJOTu7AkDYrIVF+Rz9cLp/LcCQtKIAkIkdKyjl67eQZZFEBAAb1hcw4ETPTSfPhvJ64uMpIAQCe1tCTqol9VN3hDXeNcsqgFgo66qljShgBAJ7W4J5mC6aGY0ZxBLZ5VTXlzAsy+fiuT1RUZSQIiE9rZ0MbuihIophZG8fn6ecdm8SrYdOh3J64uMlNKAMLPrzWyvmTWa2V0Jtheb2dfD7ZvMrCFu2yVm9oyZ7TKzHWZWkspaRfa2dEXWvBRz5fwq9rZ00qXrISQNpCwgzCwfuA+4AVgB3GpmK0bs9mHglLsvBu4FPhseWwB8Bfhtd18JvBnQb4ykTP/gMPtPdEcyginelfOrGHbYflhnERK9VJ5BrAEa3f2Au/cDDwPrR+yzHngoXH4UuM6CAehvB15w9+cB3P2ku2smM0mZA23dDAx5ZCOYYi6bW4kZbDukfgiJXioDoh44HPf4SLgu4T7uPgh0ANXARYCb2RNm9qyZ/UmiFzCz281sq5ltPXHixIR/A5I7YiOYoj6DKC8pZFFtGTuPdkRahwikNiASXYo68rZZo+1TALwB+GD473vM7LrX7ej+gLuvdvfVtbW1F1qv5LA9LV0U5hsLa8qiLoVV9RXsPKr7VEv0UhkQR4C5cY/nACNvvvvKPmG/QwXQHq7/qbu3ufsZ4HHgihTWKjlub0sXi2rLKCqIfmDfytnTaOns5URXX9SlSI5L5W/DFmCJmS0wsyLgFmDDiH02ALeFy+8FnnJ3B54ALjGzqWFwvAl4MYW1So7bc6wz8ualmFX1wVTjamaSqKUsIMI+hTsIPux3A4+4+y4z+7SZ3RTu9iBQbWaNwJ3AXeGxp4C/JwiZ7cCz7v7dVNUqua3j7ADNHb0smxXtENeYlQoISRMFqXxyd3+coHkoft09ccu9wPtGOfYrBENdRVLqpePhFBtpcgZRVlzAwppSdiggJGLRN7iKRGxPmoxgindxfYXOICRyCgjJeXuOdVJeUkBdRfpcrH9x/TSaO3o52a2OaomOAkJy3r7WbpbOnPybBI3l4lg/RLOGu0p0FBCS8xpbu1kyM/rrH+KtrAsCYvcxBYRERwEhOe1kdx/tPf0snpE+/Q8AFVMLmV1Rwh4FhERIASE5bV9rNwBLZqTXGQQENy6KdaCLREEBITntlYBIsyYmCIbdNrZ20z84HHUpkqMUEJLTGo93UVZcwKxp6TOCKWZZ3TQGh53GMMREJpsCQnLavtZuFs8oS6sRTDHLw+sy9rSoH0KioYCQnLavtTst+x8AFtSUUlSQp34IiYwCQnLW6TP9nOjqS8v+B4CC/Dwumlmmoa4SGQWE5KzGV0YwpdcQ13jLZmkkk0RHASE5KzaCaXGaNjFBMJLpRFcfbZpyQyKggJCcte94N1MK86mvnBJ1KaNaXhdMQb7nmM4iZPIpICRn7WvtYvGMMvLy0m8EU8wyjWSSCCkgJGc1pvEIppjqsmJqy4vZrTMIiYACQnJSV+8Axzp6WZTmAQFBM5POICQKCgjJSU1tPQAsqs2AgJhVzr7j3QwOacoNmVwKCMlJsYBYWFsacSXjW1ZXTv/Q8Cs1i0wWBYTkpKa2Hsxg3vSpUZcyrmWzgpFML+qCOZlkCgjJSU1tPcyumEJJYX7UpYxrUW0ZhfmmjmqZdAoIyUlNbT0Z0bwEUFSQx6JaTbkhk08BITnH3Wk60cOCmswICIAVddMUEDLpFBCSc9q6++nqG8yogFheN43Wrj5OasoNmURJBYSZPWZmN5qZAkUyXmw0UKYFBKB+CJlUyX7g3w98ANhnZn9lZstSWJNISh2MDXGtSf9rIGKW1wVTbqiZSSZTUgHh7j9y9w8CVwAHgR+a2c/N7NfNrDCVBYpMtANtPRTmG/VV6TtJ30jVZcXMKC9WQMikSrrJyMyqgV8DPgI8B3yOIDB+mJLKRFKkqa2b+dWl5KfxJH2JLK+bpmshZFIVJLOTmX0DWAZ8GXi3ux8LN33dzLamqjiRVGhqy6wRTDHL66bx8/1t9A8OU1Sg7kBJvWTfZf/i7ivc/S9j4WBmxQDuvjpl1YlMsKFh5+DJMxkaEOUMDPkrd8ITSbVkA+IvEqx7ZiILEZkMzafP0j84nJEBseKVkUxqZpLJMWYTk5nNAuqBKWZ2ORBrtJ0GpP8kNiIjZOIQ15gFNaUUFeRp6m+ZNOP1QbyDoGN6DvD3ceu7gE+kqCaRlHllFtcMDIiC/DyWzizXtRAyacYMCHd/CHjIzH7Z3R+bpJpEUqaprYfSonxqy4ujLuW8LK8r58ndrbg7Zpk1Cksyz3hNTB9y968ADWZ258jt7v73CQ4TSVtNbT0sqC3N2A/X5XXTeGTrEU509TFjWknU5UiWG6+TOnYeXgaUJ/gSySjBENfMuYJ6pNiUG7oeQibDeE1M/xz++2eTU45I6vQNDnHk1Bluvrw+6lLO2/JZr87J9OalMyKuRrJdspP1/bWZTTOzQjN70szazOxDqS5OZCIdbj/DsMOCmswdgFcxtZD6yika6iqTItnrIN7u7p3Au4AjwEXA/0pZVSIp0NR2BiCjm5gAVsyexs7mjqjLkByQbEDEJuR7J/A1d29P5iAzu97M9ppZo5ndlWB7sZl9Pdy+ycwaRmyfZ2bdZvbHSdYpMqqmtuAK5AXVmTfENd4l9RUcONFDV+9A1KVIlks2IL5jZnuA1cCTZlYL9I51gJnlA/cBNwArgFvNbMWI3T4MnHL3xcC9wGdHbL8X+F6SNYqMqantDFVTC6mYmtkTEK+aUwHAzqNqZpLUSna677uAdcBqdx8AeoD14xy2Bmh09wPu3g88nOCY9cBD4fKjwHUWjj80s5uBA8CuZGoUGc/BDJ2kb6RV9UFA7Dh6OuJKJNslNZtraDnB9RDxx3xpjP3rgcNxj48AV4+2j7sPmlkHUG1mZ4GPAW8D1LwkE6KprYdrFldHXcYFqy4rpr5yCjt0BiEplux0318GFgHbgaFwtTN2QCS6EsmT3OfPgHvdvXusC5rM7HbgdoB58+aNUYrkurP9Q7R09mZ8/0PMJXMq2HFEZxCSWsmeQawGVrj7yA/4sRwB5sY9ngM0j7LPkfDMpAJoJzjTeK+Z/TVQCQybWa+7fz7+YHd/AHgAYPXq1edSm+SYgyfDSfpqsyMgVs2p4Hs7W+g4M5DxfSqSvpLtpN4JzDrH594CLDGzBWZWBNwCbBixzwbgtnD5vcBTHnijuze4ewPwf4H/MzIcRM5FbJK+hmw5g6ivBNBwV0mpZM8gaoAXzWwz0Bdb6e43jXZA2KdwB/AEkA980d13mdmnga3uvgF4EPiymTUSnDnccp7fh8iYXgmILOikhlc7ql840sG1i2sirkayVbIB8anzeXJ3fxx4fMS6e+KWe4H3jfMc5/XaIvEOtvUwo7yYsuJzGZeRviqmFjK/eqpGMklKJfXb4u4/NbP5wBJ3/5GZTSU4KxDJCE1tPVlz9hCzqr6C515WQEjqJDsX028SXKfwz+GqeuBbqSpKZKIdPNmTNSOYYi6fV8XR02dp6RjzmlWR85ZsJ/XvAtcCnQDuvg/QVJKSETp7B2jr7s+aEUwxq+dXAbDt0KmIK5FslWxA9IVXQwMQDknVsFLJCAezbARTzIrZ0ygpzFNASMokGxA/NbNPAFPM7G3AfwDfSV1ZIhMnNoIpG6bZiFeYn8elcyrZdiipuTNFzlmyAXEXcALYAfwWwcikT6aqKJGJ1NTWgxnMr87c+0CM5sr5Vexq7uRs/9D4O4uco2Qn6xsm6JT+qLu/192/cI5XVYtE5sCJHmZXTKGkMPsG3q1uqGJw2HlB025ICowZEBb4lJm1AXuAvWZ2wszuGes4kXSyr7WbJTMz+yZBo7l8btBRvVX9EJIC451B/CHB6KWr3L3a3acTzJN0rZn9UcqrE7lAQ8POgRPdLK7NzoCoKi1iUW0pWw+qH0Im3ngB8avAre7eFFvh7geAD4XbRNLakVNn6BscztozCIB1i6rZ3NTOwNBw1KVIlhkvIArdvW3kSnc/wau3IRVJW42twW1GF8/I3oC4ZlENPf1D7DiqiftkYo0XEP3nuU0kLbwSELXlEVeSOmsXBjdBemb/yYgrkWwzXkBcamadCb66gFWTUaDIhdjX2k1teXFW3zNhemkRy2aV8/P9rzvZF7kgY07W5+7ZNy5QckpjazdLsrh5KeaaRTX8+6ZD9A4MZeVwXolGshfKiWQcd2d/a3dW9z/EXLOomr7BYc3uKhNKASFZq6Wzl66+wZw4g1izcDp5Bs+omUkmkAJCstaLzZ1AMKldtptWUsiqOZU8rY5qmUAKCMlaLzZ3YgZLZ2V/QEDQzPT84dN09w1GXYpkCQWEZK1dzZ00VJdmzW1Gx3PtohoGh53NTTqLkImhgJCs9eKxTlbU5cbZAwQT9xUV5PF0owJCJoYCQrJSZ+8AL7efyYn+h5iSwnyunFfF043qqJaJoYCQrLTnWBdATp1BAFy7uJo9LV20dfdFXYpkAQWEZKUXm4N5iXLpDALgmsU1gKbdkImhgJCstONoJ9WlRcwoL466lEl1SX0F5cUFmnZDJoQCQrLSloPtXDm/CjOLupRJVZCfx9ULq9VRLRNCASFZ51jHWV5uP8OaBdOjLiUS1y6u5uX2MxxuPxN1KZLhFBCSdTY3BXdXu3pBdcSVROPasB9CzUxyoRQQknU2N7VTVlzA8rrsvQfEWJbMKKO2vFjNTHLBFBCSdTY3Bf0PBfm5+fY2M65ZVM3P95/E3aMuRzJYbv4GSdZq7+lnX2t3zvY/xFy7qIa27j5eOt4ddSmSwRQQklW2HIz1P+R2QFyzOOh/0VXVciEUEJJVNje1U1yQx6o5FVGXEqk5VVOZN30qGw+oH0LOnwJCssrmpnYum1tJcYFuu7l24XQ2NbUzPKx+CDk/CgjJGl29A+xq7sj55qWYdYuq6Tg7wO6WzqhLkQylgJCsse3QKYYd1uTo9Q8jrV0Y/Bw0L5OcLwWEZI0tB9spyDOumF8ZdSlpoa5iCg3VU9l4oD3qUiRDKSAka2xuaufi+gqmFuXGHeSSsXZhNZuaTjKkfgg5DwoIyQq9A0M8f1j9DyOtW1RNV+8gu4+pH0LOnQJCssL2w6fpHxrO+QvkRlI/hFwIBYRkhc1N7ZjB6vkKiHgzp5WwsKZU10PIeUlpQJjZ9Wa218wazeyuBNuLzezr4fZNZtYQrn+bmW0zsx3hv29NZZ2S+TY3tbN0ZjkVUwujLiXtXL2wms1N7QwODUddimSYlAWEmeUD9wE3ACuAW81sxYjdPgyccvfFwL3AZ8P1bcC73X0VcBvw5VTVKZlvYGiYbYdOqf9hFOsWVdPVN8iL6oeQc5TKM4g1QKO7H3D3fuBhYP2IfdYDD4XLjwLXmZm5+3Pu3hyu3wWUmFlu3TtSkraruZOzA0O6/mEUaxcGwal+CDlXqQyIeuBw3OMj4bqE+7j7INABjPwt/2XgOXfvG/kCZna7mW01s60nTpyYsMIls2xuCj74rlpQFXEl6WlGeQmLatUPIeculQGR6GbAIwdjj7mPma0kaHb6rUQv4O4PuPtqd19dW1t73oVKZtvc1M7CmlJmlJdEXUraWreomi0HT6kfQs5JKgPiCDA37vEcoHm0fcysAKgA2sPHc4BvAr/q7vtTWKdksOFhZ3NTu4a3jmPtwmq6+wbZ2ax+CEleKgNiC7DEzBaYWRFwC7BhxD4bCDqhAd4LPOXubmaVwHeBj7v70ymsUTLcnpYuOnsHuapBATEWXQ8h5yNlARH2KdwBPAHsBh5x911m9mkzuync7UGg2swagTuB2FDYO4DFwP82s+3h14xU1SqZa1PY/7B2kTqox1JTVsySGWXqh5BzktJJa9z9ceDxEevuiVvuBd6X4Li/AP4ilbVJdth44CRzp0+hvnJK1KWkvXWLqnls2xEGhoYpzNH7dcu50btEMlas/+FqDW9NytqF1fT0D7HjaEfUpUiGUEBIxtrX2s2pMwO6QC5JsZ+TmpkkWQoIyVixD7pYB6yMrbqsmGWzyvnZvraoS5EMoYCQjLWp6ST1lVOYO31q1KVkjDctrWXLwXa6+wajLkUygAJCMpK7s+lAO1cvVPPSuXjr0hkMDLnOIiQpCgjJSI2t3Zzs6WetOqjPyRXzqygvKeDHe1qjLkUygAJCMpL6H85PYX4ev3BRLT/e24q7bkMqY1NASEb6WWNb2P+g6x/O1VuWzqC1q49dmnZDxqGAkIwzMDTM040n+YWLajFLNN+jjOXNS4OJLX+yV81MMjYFhGSc514+TXffIG+6qCbqUjJSTVkxl86p4Ee7FRAyNgWEZJyfvtRKfp5xzWIFxPl6+8pZbD98mqOnz0ZdiqQxBYRknP96qY0r5lUyrUT3nz5fN66qA+B7O45FXImkMwWEZJTWzl52HO3gTRfpBlEXoqGmlIvrp/FdBYSMQQEhGeWJXS0AXH/xrIgryXzvXFXHcy+f5sipM1GXImlKASEZ5bs7jrFkRhmLZ5RHXUrGizUzPa6zCBmFAkIyRlt3H5ub2rkh/GCTCzO/upRL51Tw2LajumhOElJASMb49vZmhh3edYkCYqK8/6p57D3exfbDp6MuRdKQAkIygrvztc0vc9ncSi6aqealifLuS+uYUpjPw5sPR12KpCEFhGSEbYdO0djazQfWzIu6lKxSXlLIuy+t4zsvNNPVOxB1OZJmFBCSEb74dBPlJQXcqOalCfehtfM50z+kswh5HQWEpL3G1i6+t7OF29Y1UFpcEHU5WeeSOZWsW1jNgz9ron9wOOpyJI0oICTtfe7JRkoK8vmNNyyIupSs9dtvXkRLZy//sU1nEfIqBYSktacb2/jO88385hsXML20KOpystYvLKnhqoYq7v3hS7odqbxCASFpq3dgiE9+aycN1VP56FsWR11OVjMz7r5xBW3d/XzuRy9FXY6kCQWEpK37f7KfprYe/uLmVZQU5kddTta7bG4lt66Zx7/8rOmVO/ZJblNASFraf6Kb+3+yn/WXzeYNSzSt92T55I3LmT99Kr//tec0R5MoICT9uDt3f3MHJYV5fPLGFVGXk1NKiwv45/+xmrMDQ/zav27hVE9/1CVJhBQQkna+8exRNh5o52M3LKO2vDjqcnLO0lnlfOFXV/Ny+xlu/cJGTnb3RV2SREQBIWnlVE8/n3l8N1fMq+TWq3TVdFTWLqzmwdtWc/BkD7c8sJHWzt6oS5IIKCAkrfzl93bTcXaAz7xnFXl5FnU5Oe2NS2r5t19fw9HTZ3n/Axs51qHbk+YaBYSkjc1N7Tyy9QgfecMCltdNi7ocITiT+PKH19DW1cev/PMzHG5Xx3UuUUBIWugfHOYT39xBfeUU/uAXl0RdjsS5cv50vvKRq+k4M8AtD2zkYFtP1CXJJFFASFr4x5800tjazafXr2RqkeZbSjeXzq3kq7+5ljP9g/zS/T/nuZdPRV2STAIFhERu59EOPv9UIzddOpvrls+MuhwZxcX1FTz2O9dQVlzArV/YqFuV5gAFhESqp2+QOx/ZzvTSIj69fmXU5cg4FtaW8Y2PXsPyuml89N+f5Z5v76R3YCjqsiRFFBASmaFh538+8jyNrd383a9cSuVUTcaXCWrKivn67ev4yBsW8KVnDvGef/w5O450RF2WpIACQiIxMDTMnY9s5/u7Wrj7xhW8cUlt1CXJOSgqyOOT71rBg7et5kRXH+vv+xmf/s6LuvI6y5i7R13DhFi9erVv3bo16jIkCXtaOvnYoy/w/JEO/uT6pXz0zZqpNZN1nB3gr7+/h69ufpmphfl84Op5vP+quSyeoXuHZwIz2+buqxNuU0DIZDjZ3cfT+0+yYftRfrS7lYophfzlL63inat0C9Fs8dLxLj7/VCPf3XGMoWFnVX0FN6yaxXXLZnLRzDLMdOFjOoosIMzseuBzQD7wL+7+VyO2FwNfAq4ETgLvd/eD4baPAx8GhoDfd/cnxnotBUT6ONM/yJ6WLnYf6+S5l0+z7dApmsKx8zOnFfMrq+fyG9cuoEo3AMpKJ7r62PB8M9/efpQXwr6J+sopvHlpLZfOqWRZXTmzK6cwfWqRrpZPA5EEhJnlAy8BbwOOAFuAW939xbh9Pgpc4u6/bWa3AO9x9/eb2Qrga8AaYDbwI+Aidx91uIQCYnL0DQ7R3TtId98gXb2DtHX3ceTUWQ6fOsPLJ8+wp6WLgyd7iL2tqkuLuGJ+FVfOr+Kqhioum1tFvjW7FNwAAAoiSURBVD4UckZLRy8/3tvKk7tbeWZ/Gz39r/4KF+QZteXFzCgvpqasmNrw35qyImrKi6ktK6YmXDetpEBnICkyVkCk8oqkNUCjux8Ii3gYWA+8GLfPeuBT4fKjwOcteBesBx529z6gycwaw+d7ZqKL3NPSyR1ffQ4IppmO8dctvGbxlX1fuy62n79u3cjlkc8T/1yvOSbBcyV6zfgtfh41j7aduNfsHxymfyjxTe2L8vOor5rC0pnl3HxZPcvrylleN405VVP0i53DZlWUcOuaedy6Zh7Dw86h9jPsbemkpaOX1q4+jnf20drVS3NHLy8c7aC9p5+h4df/ohQV5FFdWkRBvmEYZpBnhgFmwR3xcvld9ualtdydgqnxUxkQ9UD8HdCPAFePto+7D5pZB1Adrt844tj6kS9gZrcDtwPMm3d+M3+WFOSzdGZcZ5q9fjH+Ay7+TRhb/dp19rp1r31Oe82xoz+nvW7da5dH2Z5w3ev3TVTza0pOcHxxYR5lxQWUlxRQVhx8VZUWMbdqKjPKi9VcIGPKyzMW1JSyoKZ01H2Gh51TZ/o50d1HW1c/bd19nOjqo627j5M9/QwPO8Me/NnkTtxydvSlnq+Z00pS8rypDIhEnxYj/xdH2yeZY3H3B4AHIGhiOtcCARpqSrnvg1ecz6EiMsHy8ozqsmKqy4phVtTVSCqvgzgCzI17PAdoHm0fMysAKoD2JI8VEZEUSmVAbAGWmNkCMysCbgE2jNhnA3BbuPxe4CkPzhU3ALeYWbGZLQCWAJtTWKuIiIyQsiamsE/hDuAJgmGuX3T3XWb2aWCru28AHgS+HHZCtxOECOF+jxB0aA8CvzvWCCYREZl4ulBORCSHjTXMVXMxiYhIQgoIERFJSAEhIiIJKSBERCShrOmkNrMTwKEkdq0B2lJczkRSvamlelMnk2qF3K13vrsnvCFL1gREssxs62g99ulI9aaW6k2dTKoVVG8iamISEZGEFBAiIpJQLgbEA1EXcI5Ub2qp3tTJpFpB9b5OzvVBiIhIcnLxDEJERJKggBARkYRyJiDM7FNmdtTMtodf74zb9nEzazSzvWb2jijrHMnM/tjM3MxqwsdmZv8Q1vuCmaXF3Y7M7M/Derab2Q/MbHa4Pu3qNbO/MbM9YT3fNLPKuG1p914ws/eZ2S4zGzaz1SO2pV29AGZ2fVhTo5ndFXU9I5nZF82s1cx2xq2bbmY/NLN94b9VUdYYz8zmmtmPzWx3+F74g3B9amt295z4Irj39R8nWL8CeB4oBhYA+4H8qOsNa5tLMF36IaAmXPdO4HsEd91bC2yKus6wrmlxy78P/FO61gu8HSgIlz8LfDad3wvAcmAp8BNgddz6dK03P6xlIVAU1rgi6rpG1PgLwBXAzrh1fw3cFS7fFXtfpMMXUAdcES6XAy+F//8prTlnziDGsB542N373L0JaATWRFxTzL3An/Da262uB77kgY1ApZnVRVJdHHfvjHtYyqs1p1297v4Ddx8MH24kuGMhpOl7wd13u/veBJvSsl6CGhrd/YC79wMPE9SaNtz9vwjuQRNvPfBQuPwQcPOkFjUGdz/m7s+Gy13AbqCeFNecawFxR9is8MW4U7F64HDcPkfCdZEys5uAo+7+/IhNaVkvgJl9xswOAx8E7glXp229od8gOMOB9K91pHStN13rGs9Mdz8GwQcyMCPiehIyswbgcmATKa45ZXeUi4KZ/YjEtzq/G7gf+HOCv2z/HPg7gg8HS7D/pIz9HafeTxA0hbzusATrIq/X3b/t7ncDd5vZx4E7gD8lonrHqzXc526COxb+e+ywBPunxc92tMMSrEuHcevpWlfGM7My4DHgD9290yzRj3riZFVAuPsvJrOfmX0B+M/w4RGCtv6YOUDzBJeW0Gj1mtkqgjbl58M3wBzgWTNbQxrWm8BXge8SBEQk9Y5Xq5ndBrwLuM7DBlwy42cbL7J6x5GudY3nuJnVufuxsBm0NeqC4plZIUE4/Lu7fyNcndKac6aJaUS793uA2OiFDcAtZlZsZguAJcDmya4vnrvvcPcZ7t7g7g0Ev3BXuHsLQb2/Go4OWgt0xE4xo2RmS+Ie3gTsCZfTrl4zux74GHCTu5+J25R274VxpGu9W4AlZrbAzIoI7jW/IeKakrEBuC1cvg0Y7cxt0lnwl+KDwG53//u4TamtOere+cn6Ar4M7ABeCH+odXHb7iYYdbEXuCHqWhPUfpBXRzEZcF9Y7w7iRrVEXONjBKH7AvAdoD5d6yXozD0MbA+//imd3wsEf9AcAfqA48AT6VxvWNc7CUba7CdoJou8phH1fQ04BgyEP9sPA9XAk8C+8N/pUdcZV+8bCJrpXoh7374z1TVrqg0REUkoZ5qYRETk3CggREQkIQWEiIgkpIAQEZGEFBAiIpKQAkKyhpnVmtnPzGynmd0ct/7bsdllk3iO7vDf2Wb2aNz6r4XTtPyRmS0LZ619zswWjTj+8fjZYUUymYa5StYws98HzhJMDvd9d7/WzN5NcJHhnyX5HN3uXjZi3SyCWWjnh4/vAqa4+59O7Hcgkl50BiHZZACYQjD99bCZFQB/CPzNaAeEV/s+Y2ZbzOzP49Y3xN0r4AfAjPCs4U/D5/yImf04wfMdNLOa8PjdZvaFcP7+H5jZlAT7/5uZ3R/O9X/AzN4UTia528z+LW6/t4d1Pmtm/xHOyYOZ3RPWvtPMHgivuMXMfmJmnzWzzWb2kpm9MVy/Mly3PTwjWjKyJpEYBYRkk68C7wC+T3D/j48STDV+ZoxjPgfc7+5XAS2j7HMTsN/dLwvPRP4JuNfd3zJOPUuA+9x9JXAa+OVR9qsC3gr8EcFV6PcCK4FVZnaZBTeL+iTwi+5+BbAVuDM89vPufpW7X0wQju+Ke94Cd19DEGixs53fBj7n7pcBqwmuIhZJSAEhWcPdO9z9RndfDTxL8GH5WPhX/KNmti7BYdcSTLsAwXQsE6nJ3beHy9uAhlH2+44Hbb07gOMezMU1DOwKj1lLcHOYp81sO8GcO/PDY99iZpvMbAdByKyMe97YhG7xr/0M8Akz+xgw393PXti3KNksq2ZzFYlzD/AZ4FaCD8ivEkxkluiv/lR1xPXFLQ8R/IU/1n7DI44ZJvgdHQJ+6O63xh9kZiXAPxLMb3XYzD4FlCR43qHweXD3r5rZJuBG4Akz+4i7P3Wu35jkBp1BSNYJ29Vnu/tPgakEH7TOaz88Y54mmG0UghsdpaONwLVmthjAzKaa2UW8+v20hX0S7x3vicxsIXDA3f+BYNLKS1JUs2QBBYRko88QtNlD0Hz0awQfsn+bYN8/AH7XzLYAFZNS3Tly9xME38PXzOwFgu9lmbufBr5A0DT1LYJptsfzfmBn2FS1DPhSSoqWrKBhriIikpDOIEREJCEFhIiIJKSAEBGRhBQQIiKSkAJCREQSUkCIiEhCCggREUno/wMWv4Jno6RwhAAAAABJRU5ErkJggg==\n", 1311 | "text/plain": [ 1312 | "
" 1313 | ] 1314 | }, 1315 | "metadata": { 1316 | "needs_background": "light" 1317 | }, 1318 | "output_type": "display_data" 1319 | } 1320 | ], 1321 | "source": [ 1322 | "# Ploting the bootstrap % difference\n", 1323 | "ax = boot_7d['diff'].plot(kind = 'kde')\n", 1324 | "ax.set_xlabel(\"% diff in means\")" 1325 | ] 1326 | }, 1327 | { 1328 | "cell_type": "code", 1329 | "execution_count": 41, 1330 | "metadata": {}, 1331 | "outputs": [ 1332 | { 1333 | "data": { 1334 | "text/plain": [ 1335 | "0.9983" 1336 | ] 1337 | }, 1338 | "execution_count": 41, 1339 | "metadata": {}, 1340 | "output_type": "execute_result" 1341 | } 1342 | ], 1343 | "source": [ 1344 | "# Calculating the probability that 7-day retention is greater when the gate is at level 30\n", 1345 | "p = (boot_7d['diff']>0).sum()/len(boot_7d)\n", 1346 | "\n", 1347 | "1-p" 1348 | ] 1349 | }, 1350 | { 1351 | "cell_type": "markdown", 1352 | "metadata": {}, 1353 | "source": [ 1354 | "# Conclusion 2: control has a better performance than the treatment on 7-day user retention 99.89% of the time." 1355 | ] 1356 | }, 1357 | { 1358 | "cell_type": "markdown", 1359 | "metadata": {}, 1360 | "source": [ 1361 | "## After double-checking with the true parameters, these two estimates are unbiased estimates." 1362 | ] 1363 | } 1364 | ], 1365 | "metadata": { 1366 | "kernelspec": { 1367 | "display_name": "Python 3", 1368 | "language": "python", 1369 | "name": "python3" 1370 | }, 1371 | "language_info": { 1372 | "codemirror_mode": { 1373 | "name": "ipython", 1374 | "version": 3 1375 | }, 1376 | "file_extension": ".py", 1377 | "mimetype": "text/x-python", 1378 | "name": "python", 1379 | "nbconvert_exporter": "python", 1380 | "pygments_lexer": "ipython3", 1381 | "version": "3.7.4" 1382 | } 1383 | }, 1384 | "nbformat": 4, 1385 | "nbformat_minor": 2 1386 | } 1387 | -------------------------------------------------------------------------------- /4. AA Test.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Part 1: A/A tests for Normal Distribution" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "- split the users into two groups & assign the same treatment to both groups \n", 15 | "- In repeated trials, 5% of the time a given metric should be statistically significant & p < 0.05. \n", 16 | "- conduct t-tests to compute p-values & the distribution of p-values from repeated trials form a uniform distribution " 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "# 1. simulate the hashing process & single iteration" 24 | ] 25 | }, 26 | { 27 | "cell_type": "code", 28 | "execution_count": 4, 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "import numpy as np\n", 33 | "\n", 34 | "# 1: population\n", 35 | "np.random.seed(123)\n", 36 | "\n", 37 | "population = np.random.normal(loc = 100, scale = 5, size = 1000)" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": 5, 43 | "metadata": {}, 44 | "outputs": [ 45 | { 46 | "data": { 47 | "text/plain": [ 48 | "Ttest_indResult(statistic=-0.5697163252872851, pvalue=0.568998328712083)" 49 | ] 50 | }, 51 | "execution_count": 5, 52 | "metadata": {}, 53 | "output_type": "execute_result" 54 | } 55 | ], 56 | "source": [ 57 | "import random\n", 58 | "\n", 59 | "A_1 = []\n", 60 | "\n", 61 | "A_2 = []\n", 62 | "\n", 63 | "for i in population:\n", 64 | " \n", 65 | " hash_val = random.random()\n", 66 | " \n", 67 | " if hash_val <= 0.5:\n", 68 | " A_2.append(i)\n", 69 | " \n", 70 | " else:\n", 71 | " A_1.append(i)\n", 72 | " \n", 73 | "# two sample t test \n", 74 | "from scipy import stats\n", 75 | "\n", 76 | "stats.ttest_ind(A_1,A_2)" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "# 2. run A/A 10,000 times & check False Positive Rate" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": 2, 89 | "metadata": {}, 90 | "outputs": [], 91 | "source": [ 92 | "import random\n", 93 | "from scipy import stats" 94 | ] 95 | }, 96 | { 97 | "cell_type": "markdown", 98 | "metadata": {}, 99 | "source": [ 100 | "### solution 1: eyeball " 101 | ] 102 | }, 103 | { 104 | "cell_type": "code", 105 | "execution_count": 6, 106 | "metadata": {}, 107 | "outputs": [], 108 | "source": [ 109 | "count_5_perc = 0 #False Positives\n", 110 | "count_10_perc = 0\n", 111 | "count_20_perc = 0\n", 112 | "count_30_perc = 0\n", 113 | "count_40_perc = 0\n", 114 | "count_50_perc = 0\n", 115 | "count_60_perc = 0\n", 116 | "count_70_perc = 0\n", 117 | "count_80_perc = 0\n", 118 | "count_90_perc = 0\n", 119 | "\n", 120 | "for i in range(10000):\n", 121 | " \n", 122 | " A_1 = []\n", 123 | " \n", 124 | " A_2 = []\n", 125 | " \n", 126 | " for j in population:\n", 127 | " \n", 128 | " hash_val = random.random()\n", 129 | " \n", 130 | " if hash_val <= 0.5:\n", 131 | " A_2.append(j)\n", 132 | " \n", 133 | " else:\n", 134 | " A_1.append(j)\n", 135 | " \n", 136 | " result = stats.ttest_ind(A_1,A_2)\n", 137 | " \n", 138 | " if result.pvalue <= 0.1:\n", 139 | " count_10_perc+=1\n", 140 | " \n", 141 | " elif 0.1 different distributions \n", 454 | "# big p-value --> same distributions\n", 455 | "import numpy as np\n", 456 | "import scipy\n", 457 | "\n", 458 | "\n", 459 | "dddd = np.random.normal(0,1,1000)\n", 460 | "kstest(dddd,'norm')" 461 | ] 462 | }, 463 | { 464 | "cell_type": "code", 465 | "execution_count": 20, 466 | "metadata": {}, 467 | "outputs": [ 468 | { 469 | "data": { 470 | "text/plain": [ 471 | "KstestResult(statistic=0.4999999999999999, pvalue=0.06558641975308652)" 472 | ] 473 | }, 474 | "execution_count": 20, 475 | "metadata": {}, 476 | "output_type": "execute_result" 477 | } 478 | ], 479 | "source": [ 480 | "from scipy import stats\n", 481 | "x = np.linspace(-25, 17, 6)\n", 482 | "stats.kstest(x, 'norm')" 483 | ] 484 | }, 485 | { 486 | "cell_type": "code", 487 | "execution_count": 24, 488 | "metadata": {}, 489 | "outputs": [ 490 | { 491 | "data": { 492 | "text/plain": [ 493 | "KstestResult(statistic=0.03859901423041939, pvalue=0.09899489774451381)" 494 | ] 495 | }, 496 | "execution_count": 24, 497 | "metadata": {}, 498 | "output_type": "execute_result" 499 | } 500 | ], 501 | "source": [ 502 | "from scipy.stats import kstest\n", 503 | "import numpy as np\n", 504 | "\n", 505 | "x = np.random.normal(0,1,1000)\n", 506 | "test_stat = kstest(x, 'norm')\n", 507 | "test_stat" 508 | ] 509 | }, 510 | { 511 | "cell_type": "markdown", 512 | "metadata": {}, 513 | "source": [ 514 | "---" 515 | ] 516 | }, 517 | { 518 | "cell_type": "markdown", 519 | "metadata": {}, 520 | "source": [ 521 | "# split at 0.2%" 522 | ] 523 | }, 524 | { 525 | "cell_type": "code", 526 | "execution_count": 21, 527 | "metadata": {}, 528 | "outputs": [], 529 | "source": [ 530 | "# solution 2\n", 531 | "\n", 532 | "import numpy as np\n", 533 | "import scipy\n", 534 | "\n", 535 | "p_values = []\n", 536 | "\n", 537 | "for i in range(10000):\n", 538 | " \n", 539 | " A_1 = []\n", 540 | " \n", 541 | " A_2 = []\n", 542 | " \n", 543 | " for j in population:\n", 544 | " \n", 545 | " hash_val = random.random()\n", 546 | " \n", 547 | " if hash_val <= 0.2:\n", 548 | " A_2.append(j)\n", 549 | " \n", 550 | " else:\n", 551 | " A_1.append(j)\n", 552 | " \n", 553 | " result=(stats.ttest_ind(A_1,A_2))\n", 554 | " \n", 555 | " p_values.append(result.pvalue)" 556 | ] 557 | }, 558 | { 559 | "cell_type": "code", 560 | "execution_count": 22, 561 | "metadata": {}, 562 | "outputs": [ 563 | { 564 | "data": { 565 | "text/plain": [ 566 | "KstestResult(statistic=0.009005414094706121, pvalue=0.39198360324923237)" 567 | ] 568 | }, 569 | "execution_count": 22, 570 | "metadata": {}, 571 | "output_type": "execute_result" 572 | } 573 | ], 574 | "source": [ 575 | "import scipy\n", 576 | "scipy.stats.kstest(p_values,\"uniform\")" 577 | ] 578 | }, 579 | { 580 | "cell_type": "markdown", 581 | "metadata": {}, 582 | "source": [ 583 | "----" 584 | ] 585 | }, 586 | { 587 | "cell_type": "markdown", 588 | "metadata": {}, 589 | "source": [ 590 | "# Part 2: A/A tests for Left Skewed Distribution with Heavy Users" 591 | ] 592 | }, 593 | { 594 | "cell_type": "code", 595 | "execution_count": 1, 596 | "metadata": { 597 | "scrolled": true 598 | }, 599 | "outputs": [ 600 | { 601 | "data": { 602 | "text/plain": [ 603 | "
" 604 | ] 605 | }, 606 | "metadata": {}, 607 | "output_type": "display_data" 608 | } 609 | ], 610 | "source": [ 611 | "# generate skewed distribution \n", 612 | "# source: https://stackoverflow.com/questions/24854965/create-random-numbers-with-left-skewed-probability-distribution\n", 613 | "\n", 614 | "from scipy.stats import skewnorm\n", 615 | "import matplotlib.pyplot as plt\n", 616 | "\n", 617 | "numValues = 10000\n", 618 | "maxValue = 1000\n", 619 | "skewness = -50 #Negative values are left skewed, positive values are right skewed.\n", 620 | "\n", 621 | "random_1 = skewnorm.rvs(a = skewness,loc=maxValue, size=numValues) #Skewnorm function\n", 622 | "\n", 623 | "#random_1 = random_1 - min(random_1) #Shift the set so the minimum value is equal to zero.\n", 624 | "#random_1 = random_1 / max(random_1) #Standadize all the vlues between 0 and 1. \n", 625 | "#random_1 = random_1 * maxValue #Multiply the standardized values by the maximum value.\n", 626 | "\n", 627 | "#Plot histogram to check skewness\n", 628 | "plt.hist(random_1,30,density=True, color = 'red', alpha=0.1)\n", 629 | "plt.show()" 630 | ] 631 | }, 632 | { 633 | "cell_type": "code", 634 | "execution_count": 2, 635 | "metadata": {}, 636 | "outputs": [ 637 | { 638 | "data": { 639 | "text/plain": [ 640 | "array([999.9315308 , 999.00985561, 998.7830595 , ..., 999.56253601,\n", 641 | " 999.87125823, 997.82424896])" 642 | ] 643 | }, 644 | "execution_count": 2, 645 | "metadata": {}, 646 | "output_type": "execute_result" 647 | } 648 | ], 649 | "source": [ 650 | "random_1" 651 | ] 652 | }, 653 | { 654 | "cell_type": "code", 655 | "execution_count": 3, 656 | "metadata": { 657 | "scrolled": true 658 | }, 659 | "outputs": [ 660 | { 661 | "data": { 662 | "text/plain": [ 663 | "Ttest_indResult(statistic=0.7640304333604793, pvalue=0.44486713677102585)" 664 | ] 665 | }, 666 | "execution_count": 3, 667 | "metadata": {}, 668 | "output_type": "execute_result" 669 | } 670 | ], 671 | "source": [ 672 | "#2. simulate the hashing process & single iteration\n", 673 | "\n", 674 | "import random\n", 675 | "\n", 676 | "A_1 = []\n", 677 | "\n", 678 | "A_2 = []\n", 679 | "\n", 680 | "for i in random_1:\n", 681 | " \n", 682 | " hash_val = random.random()\n", 683 | " \n", 684 | " if hash_val <= 0.5:\n", 685 | " A_2.append(i)\n", 686 | " \n", 687 | " else:\n", 688 | " A_1.append(i)\n", 689 | " \n", 690 | "# two sample t test \n", 691 | "from scipy import stats\n", 692 | "\n", 693 | "stats.ttest_ind(A_1,A_2)" 694 | ] 695 | }, 696 | { 697 | "cell_type": "code", 698 | "execution_count": 4, 699 | "metadata": {}, 700 | "outputs": [], 701 | "source": [ 702 | "#2. run A/A 10,000 times & check False Positive Rate\n", 703 | "import random\n", 704 | "from scipy import stats" 705 | ] 706 | }, 707 | { 708 | "cell_type": "code", 709 | "execution_count": 5, 710 | "metadata": {}, 711 | "outputs": [], 712 | "source": [ 713 | "count_5_perc = 0 #False Positives\n", 714 | "count_10_perc = 0\n", 715 | "count_20_perc = 0\n", 716 | "count_30_perc = 0\n", 717 | "count_40_perc = 0\n", 718 | "count_50_perc = 0\n", 719 | "count_60_perc = 0\n", 720 | "count_70_perc = 0\n", 721 | "count_80_perc = 0\n", 722 | "count_90_perc = 0\n", 723 | "\n", 724 | "for i in range(10000):\n", 725 | " \n", 726 | " A_1 = []\n", 727 | " \n", 728 | " A_2 = []\n", 729 | " \n", 730 | " for j in random_1:\n", 731 | " \n", 732 | " hash_val = random.random()\n", 733 | " \n", 734 | " if hash_val <= 0.05:\n", 735 | " A_2.append(j)\n", 736 | " \n", 737 | " else:\n", 738 | " A_1.append(j)\n", 739 | " \n", 740 | " result = stats.ttest_ind(A_1,A_2)\n", 741 | " \n", 742 | " if result.pvalue <= 0.1:\n", 743 | " count_10_perc+=1\n", 744 | " \n", 745 | " elif 0.1" 907 | ] 908 | }, 909 | "execution_count": 49, 910 | "metadata": {}, 911 | "output_type": "execute_result" 912 | }, 913 | { 914 | "data": { 915 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3ycV53v8c9vika9WM1Ws2RL7o5LFDvFiUPaJpQ4gTQCS+6S3cC+CLDL3QVy4e5C7l0WWF4EdhPg5kVYEhLHKQRwIMQhvbjK3ZKbXFSs3pul0cyc+8eMjZAlayyP9Ez5vV8vvzTzzJHmp+OZr86c53nOI8YYlFJKRS+b1QUopZSaWhr0SikV5TTolVIqymnQK6VUlNOgV0qpKOewuoDRsrKyTHFxsdVlKKVURNm5c2ebMSZ7rMfCLuiLi4upqKiwugyllIooIlIz3mM6daOUUlFOg14ppaKcBr1SSkU5DXqllIpyGvRKKRXlNOiVUirKadArpVSU06BXSqkop0GvlFJRLuzOjFUq1NZvq52wzb2ri6ahEqWsoSN6pZSKchr0SikV5TTolVIqymnQK6VUlNOdsUoR3A5b0J22KjLpiF4ppaKcBr1SSkU5DXqllIpyGvRKKRXlNOiVUirKadArpVSU06BXSqkop0GvlFJRToNeKaWiXFBnxorIzcCPATvwc2PMd0c97gKeAi4F2oG7jTEnRcQJ/BxYGXiup4wx/x7C+pWatAG3h1f2N1HT3k+/24MgXF2WxZVzs4hz6BhIRY8Jg15E7MBjwI1APbBDRDYaY6pGNLsf6DTGlIrIPcD3gLuBOwGXMWapiCQCVSLyrDHmZKh/EaUuxNGWXn69s57+IS8L81JJcTno6HfzWlUzW4+3c1d5IXOyk60uU6mQCGZEvwqoNsYcBxCRDcA6YGTQrwO+Fbj9IvCoiAhggCQRcQAJgBvoCU3pSk1OZUM3z2yrJTvFxWeuKCYvPeHsYyfa+vnN7lP8amsNn1s7l5mp8RZWqlRoBPP5NB+oG3G/PrBtzDbGGA/QDWTiD/1+oBGoBX5gjOkY/QQi8oCIVIhIRWtr6wX/EkoFq61viBd31lOQkcCDHyr9i5AHKMlK4rNXFRPnsPHk5pP0nB62qFKlQieYoJcxtpkg26wCvEAeUAL8TxGZc05DYx43xpQbY8qzs7ODKEmpC+f2+Fi/rRa7Tbh3VRFO+9gv//TEOD5zRTGn3V6e2VaDz4x+uSsVWYIJ+nqgcMT9AqBhvDaBaZo0oAO4F3jVGDNsjGkBPgDKL7ZopSbj1commnsGuau8kPTEuPO2zU9P4LYVedR1nmZnTec0VajU1Agm6HcAZSJSIiJxwD3AxlFtNgL3BW7fAbxpjDH4p2uuE78k4HLgUGhKVyp4bb1DbD/RzqqSGczLTQnqe5YVpDM7M5HXKps47fZOcYVKTZ0Jgz4w5/4gsAk4CDxvjKkUkYdF5NZAsyeATBGpBr4CfD2w/TEgGTiA/w/Gfxtj9oX4d1BqQq9VNeGw2bhuQU7Q3yMifOySPAbcXt481DyF1Sk1tYI6jt4Y8wrwyqht/zLi9iD+QylHf1/fWNuVmk51HQMcaOjhugU5pMQ7L+h789ITKC/OYMvxdlaXZE5RhUpNLT0rREW9VyubSHI5uLo0a1Lff8PCXGwivFetR4SpyKRBr6La7tpOTrT1c+28bFxO+6R+Rkq8kxVFGeyu7aKtbyjEFSo19TToVVR7cvNJXA4b5bMzLurnrCnNwuMzPLWlJkSVKTV9NOhV1GrpHeQP+xtZOTtj0qP5M7JTXCyYmcLTW2v0CBwVcTToVdRav62WYa/hijmh2Yl6dVk2Hf1uXtxVH5Kfp9R00aBXUcnt8fHMtlqunZ9NVrIrJD+zODORpflpPL2lBqNny6oIokGvotKrlU209g5x35XFIfuZIsI9qwo53NzLvvrukP1cpaaaBr2KSr/eWU9+egJry0K7dtLHluUR77TxXEXdxI2VChMa9CrqtPQO8t7RVm5bkYfNNtZ6e5OXGu/kw0tm8fKeBt0pqyKGBr2KOhv3NOAzcPuKgin5+XddVkjvkIc/Hmickp+vVKgFtQSCUuFo/bbaMbc/8f4J8tMT2H6ig+0nzrn8wUU/pzGGGUlx/Neb1QwO+85pc+/qopA+p1IXS0f0Kqo0dQ/S2D3IiqL0KXsOEaF8dgYn2vrp6HdP2fMoFSoa9Cqq7K7rxCZwScHUBT34lzAG2FffNaXPo1QoaNCrqGGMYX99N2U5KSS7pnZWMiMpjtkzEtmrQa8igAa9ihoN3YN0nR5mSX7qtDzfssJ0mnuGaOw+PS3Pp9RkadCrqFHV0IMA82dOT9AvzU/DJrC3Tk+eUuFNg15FjYONPczOTJryaZszklwOynJS2FffpRcQV2FNg15Fhfa+IZp6BlmcNz2j+TOWFabRdXqY2vaBaX1epS6EBr2KClWNPQAsnDW9Qb9wVipOu+hOWRXWNOhVVKhq7GFWWjwzkuKm9XldDjsLZ6Wy/1Q3Xp9O36jwpEGvIl7fkIfa9oFpH82fsawgnQG3l+qWXkueX6mJaNCriHeosQcDLLIo6Mtyk0lw2tmrSxerMKVBryJeVWMPGYlOZqXFW/L8DpuNJflpVDX04Pacu/aNUlbToFcRbWjYS3VLH4tmpSIS2iWJL8SywjTcXh8HAzuFlQonGvQqoh1p6cPjMyyc5sMqRyvOTCI13qFH36iwpEGvItrBxh4S4+zMnpFkaR02EZYVpHOkuZdOXdFShRkNehWxvD7DoaYeFs5MxR7iK0lNxrLCdHwGXtELkqgwo0GvItaJtn4Gh30ssnja5oxZafFkJ7v43Z4Gq0tR6i9o0KuIVdnQjdMulOYkW10K4L8gybLCNLaf6KChS1e0VOFDg15FJJ/PcLCxh7KcFJz28HkZn7kgyct7dVSvwkf4vEOUugD7T3XTM+gJm2mbMzKTXSwvTNfpGxVWNOhVRHqtqgmbwIKZKVaXco51y/OoauzhSLMuiaDCgwa9ikivVTZTnJVEYtz0rD1/IT56SR4Om/BCRZ3VpSgFaNCrCHS8tY+jgbNhw1F2iovrF+bw0q5TuiSCCgsa9CrivFbVDFi3iFkw7r6skPZ+N28eara6FKU06FXkea2yiSX5qaQnTu/a8xfimrJsclNdPLdDp2+U9TToVURp6Rlkd10Xf7VoptWlnJfDbuOOSwt450grTd2DVpejYpwGvYoorx9swRi4aXF4Bz3AXeWF+Ay6U1ZZLqigF5GbReSwiFSLyNfHeNwlIs8FHt8mIsUjHrtERLaISKWI7BcRaxYNV1FhU2UTszMTmZcbHmfDns/szCTWlGaxfnstw17dKausM2HQi4gdeAy4BVgEfFJEFo1qdj/QaYwpBR4Bvhf4XgfwNPB5Y8xi4FpgOGTVq5jSOzjM5mNt3LQo19K15y/E31xVTGP3IK8eaLK6FBXDghnRrwKqjTHHjTFuYAOwblSbdcCTgdsvAteL/514E7DPGLMXwBjTbozxhqZ0FWvePtzKsNdExLTNGR+an0NxZiK/+OCE1aWoGBZM0OcDIycZ6wPbxmxjjPEA3UAmMA8wIrJJRHaJyFfHegIReUBEKkSkorW19UJ/BxUjXqtqJis5jpVFGVaXEjSbTfibq0rYXdvFrtpOq8tRMSqYoB/rM7IJso0DWAN8KvD1dhG5/pyGxjxujCk3xpRnZ2cHUZKKNUMeL28dauGGhblhsfb8hbjj0gJS4h389wcnrS5Fxahggr4eKBxxvwAYvWLT2TaBefk0oCOw/R1jTJsxZgB4BVh5sUWr2LPlWDt9Qx5uWpxrdSkXLMnl4JOrinhlfyO17QNWl6NiUDBBvwMoE5ESEYkD7gE2jmqzEbgvcPsO4E1jjAE2AZeISGLgD8BaoCo0patY8uqBJpLi7Fw5N8vqUibl/jUlOGzCY29VW12KikETBn1gzv1B/KF9EHjeGFMpIg+LyK2BZk8AmSJSDXwF+HrgezuBH+L/Y7EH2GWM+UPofw0VzTxeH69VNXPdwlzinXary5mU3NR47l1dxIu76nVUr6ZdUEv/GWNewT/tMnLbv4y4PQjcOc73Po3/EEulJmX7yQ46+t18eEnkHG0zlr9fO5f122p59K2jfP+OZVaXo2KInhmrwt6rB5qId9pYOz+yd9TnpMbzqdWz+fWuU9S091tdjoohGvQqrPl8hlcPNHHtvJywXHv+Qn1+7RycduEHrx2xuhQVQzToVVjbVdtJS+8QtyyN7GmbM3JS43ngmrm8vLeBnTV6XL2aHpE/RFJRaf22WgD+sK8Bu01o73Of3RbpPnfNHDZsr+X//L6Kl/7+SmwRdl6Aijw6oldhy2cMBxp6KMtJjtijbcaS5HLwT381nz11Xby8Ty8irqaeBr0KWzXtA3SfHmZZQbrVpYTcHSsLWJyXynf/eIgBt8fqclSU06BXYWtvfRdOu7AwjC8ZOFk2m/CtWxfT2D3IT946ZnU5Kspp0Kuw5PUZDpzqZuGsVOIc0fkyvax4Brctz+Pxd49zsk0Pt1RTJzrfQSriVbf0MuD2RuW0zUgPfXghTrvw8O91ZRA1dTToVVjaW99NgtNOWQRcSepi5KbG8+UbynjzUAtvHGy2uhwVpTToVdg57fZS1dDDkvxUHLbof4n+jytLmJudxMO/r2JwWK/Lo0Iv+t9FKuJsqmzC7fWxrDC6p23OiHPY+Nati6lpH+Dn7x23uhwVhfSEKRV2nq+oY0ZSHCWZSVaXMinBnth17+qis7evLsvmliUzefStam5fWUB+esJUladikI7oVVip6xhg87F2VhZlRMwFwEPlGx9ZCMB3/3jI4kpUtNGgV2HlhZ31iMDKotiYthmpICORv10zh5f3NnDgVLfV5agookGvwobPZ/j1znrWlGaRnhhndTmWeGDtHNITnXx/02GrS1FRRINehY3Nx9o51XWau8oLJ24cpVLjnXzh2lLePdLK5mNtVpejooTujFVh49kdtaQlOLlxUS4v7TpldTmW+esrZvOLD07wvVcPc9el/UHtqxi5Y1ep0XREr8JCS88gmw40ceelBVG1UuVkxDvtfPn6MvbWdVHd0md1OSoKaNCrsPDs9jo8PsOnL59tdSlh4faV+cxMjeedI61Wl6KigAa9styw18f67TWsnZdNcVZkHjsfai6Hnb+9uoTjbf3UdQxYXY6KcDpHryz3elUzzT1D/NttsTWan+jEKrsICU477xxp1U866qLoiF5Z7qktNeSnJ/ChBTlWlxJWXE47l8/J5GBjDy29g1aXoyKYBr2yVFVDD1uOt/Ppy2dj12unnuOKuZnYbcKWY+1Wl6IimAa9stTP3z9OYpyde1fp4YFjSXY5uKQgjd11XbqypZo0DXplmabuQTbuaeDuywpJS3RaXU7YWl2SidvjY09dl9WlqAilQa8s88vNJ/EZw2evKrG6lLBWkJFAfnoCW4+3Y4yxuhwVgTTolSX6hjw8s62GW5bOonBGotXlhDURYXXJDFp6hzjZrodaqgunh1eqaXXmkMIPqtvoHfRQlJEY9PrtseySgnReOdDI1uPtlOi5BuoC6YheTTuvz/DBsTaKMxN1NB+kOIeNFUUZVDX2MOD2WF2OijAa9GraVTZ00zUwzJrSbKtLiSiXFmXg9Rn21uta9erCaNCraWWM4f3qNjKT4lgwK8XqciJKXnoCs9Li2VXTaXUpKsJo0KtpdbJ9gPrO06wpy8IWY5cKDIWVRRmc6jpNU4+eKauCp0GvptX7R1tJjLOzojDD6lIi0rLCdGwCu3VUry6ABr2aNsda+zjY1MvlczKJc+hLbzKSXQ4WzExld10XXp8eU6+Co+82NW1+/t4JHDbh8jmZVpcS0VYWZdA35OFoS6/VpagIoUGvpkVb3xAv7apnRVE6yS49feNizJuZTILTrksiqKBp0Ktp8astNQx5fFxVmmV1KRHPYbOxND+Ng409DHl0oTM1saCCXkRuFpHDIlItIl8f43GXiDwXeHybiBSPerxIRPpE5J9CU7aKJIPDXn61tYbrF+SQkxJvdTlRYXlhOsNeQ1VDj9WlqAgwYdCLiB14DLgFWAR8UkQWjWp2P9BpjCkFHgG+N+rxR4A/Xny5KhJt3NNAR7+b+6/WxctCpSgzkfREJ3vrdfpGTSyYydJVQLUx5jiAiGwA1gFVI9qsA74VuP0i8KiIiDHGiMhtwHGgP2RVq7A01po1xhgefaua3FQXJ1r7ET12PiRsIiwrSOfdI630Dg5bXY4Kc8FM3eQDdSPu1we2jdnGGOMBuoFMEUkCvgZ8++JLVZGotmOAxu5BLp+TqSEfYssL0zHA/lO6JII6v2CCfqx35+gDeMdr823gEWNM33mfQOQBEakQkYrW1tYgSlKRYsvxduKdNpYXpltdStTJTY1nVlq8Hn2jJhRM0NcDhSPuFwAN47UREQeQBnQAq4Hvi8hJ4B+A/yUiD45+AmPM48aYcmNMeXa2LnQVLXoGhzlwqptLizJwOexWlxOVlhemU995mhNtOjOqxhdM0O8AykSkRETigHuAjaPabATuC9y+A3jT+F1tjCk2xhQDPwK+Y4x5NES1qzC340QHPoOeIDWFLilIR4Df7j5ldSkqjE0Y9IE59weBTcBB4HljTKWIPCwitwaaPYF/Tr4a+ApwziGYKrZ4fD62n+hgXm4ymckuq8uJWmkJTkqyk/jdnlN6mUE1rqBOUTTGvAK8Mmrbv4y4PQjcOcHP+NYk6lMRqqqhh94hDx/X0fyUW16Qzku7T7G3vlv3hagx6ZmxakpsOd7OjKQ4ynJ1zfmptiQ/jTiHTadv1Lg06FXINXSdpqZ9gMvnZOqa89Mg3mnnhoU5/H5fAx6vz+pyVBjSoFcht/V4O067cGmRrjk/XdYtz6etz8371W1Wl6LCkAa9CqnTbi9767tYXphOQpweUjldrp2fTWq8g9/tGX3ks1Ia9CrE9tZ3Mew1rCrWnbDTyeWw85FLZrGpsokBt8fqclSY0aBXIVVxsoO8tHjyMxKsLiXm3LY8nwG3lz9VNVtdigozGvQqZE51nqahe5Dy4hlWlxKTLiueQV5avB59o86hQa9CZkdNB067f1VFNf1sNuHW5fm8e7SNtr4hq8tRYUSDXoXEgNvD3rouluSl6U5YC92+Ih+vz/D7vbpTVv2ZBr0Kid/va2TI4+Mynbax1PyZKSzJT+WFnfVWl6LCiAa9ConndtSRnexidmai1aXEvLvKC6ls6OGArlOvAjTo1UU70tzLzppOyosz9OIiYeDWZXnEOWy8qKN6FaBBry7aczvqcNqFFXombFhIT4zjpkW5/HbPKYY8XqvLUWFAg15dlCGPl5d21XPTopkku4JaDFVNg7vKC+kaGOb1qharS1FhQINeXZTXKpvpHBjm7ssKJ26sps1VpVnkpcXzXEXdxI1V1NOgVxdlw45a8tMTWFOaZXUpagS7TbizvJD3jrZS1zFgdTnKYhr0atJOtvXzQXU791xWiM2mO2HDzT2rCrGJsH57rdWlKItp0KtJ27CjDrtNuEunbcLSrLQErl+Qw/M76nSnbIzToFeT4vb4eHFnHdcvyCE3Nd7qctQ4PnX5bNr73Wyq1IXOYpkGvZqU1w8209bn5pOri6wuRZ3H1aVZzM5M5OmtNVaXoiykQa8m5dnt/p2w15RlW12KOg+bTbh3VRHbT3RwuKnX6nKURTTo1QWrbR/gvaNt3H1ZIXbdCRv27iovJN5p478/OGF1KcoiGvTqgm3YUYtN/AGiwl9GUhyfWFnAS7tP6fLFMUqDXl2QYa+P5yvquW5BLjPTdCdspPjsmhLcHh/PbNVDLWORnrOuLsjrVc209Q1x72odzYeT9dsmDvD5uSk8/t5x0hOdOO3jj/Hu1R3sUUdH9OqCrN9ey6y0eNbOy7G6FHWBrirNon/If4EYFVs06FXQ6jp0J2wkm5udxMzUeD441oYxxupy1DTSoFdBe3a77oSNZCLCmtIsmnuGqG7ts7ocNY10jl5NaP22Woa9Pn65+SQLZqby9uFWq0tSk3RJQRqvVjbxQXUbZTkpVpejpomO6FVQ9tV3MeD2csXcTKtLURfBYbdx+ZxMjjT30dwzaHU5appo0KsJGWPYfKyd3FQXc7KSrC5HXaTVJTNw2ITNx9qsLkVNEw16NaGT7QM0dg9y5ZwsvSZsFEhyOVhRlMHu2i56B4etLkdNAw16NaEtx9tJcNpZVphudSkqRK4uzcLrM2w51m51KWoaaNCr86rrGKCqoZvy4gziHPpyiRZZKS4W56Wy9UQ7g8O6Vn2003euOq8n3vcvhHXlXL1UYLS5Zl42g8M+tp/osLoUNcU06NW4OvvdPLejjuWF6aQlOK0uR4VYQUYipdnJfFDdxrDXZ3U5agpp0KtxPb21htPDXtbomvNRa+38bHqHPOyu1WURopkGvRrT4LCXJ7ec5Nr52czUSwVGrTlZSRRkJPDu0Va8Pl0WIVpp0KsxvbCznrY+Nw9cM8fqUtQUEhGuKcumo99NZUO31eWoKRJU0IvIzSJyWESqReTrYzzuEpHnAo9vE5HiwPYbRWSniOwPfL0utOWrqeD2+PjZ28dYWZTOFXP0TNhotygvlaxkF+8cadXFzqLUhEEvInbgMeAWYBHwSRFZNKrZ/UCnMaYUeAT4XmB7G/AxY8xS4D7gV6EqXE2dl3bVc6rrNF+6vkxPkIoBNhHWzsuisXuQoy262Fk0CmZEvwqoNsYcN8a4gQ3AulFt1gFPBm6/CFwvImKM2W2MaQhsrwTiRcQVisLV1Bj2+njs7WqWFaSxdp7uhI0VywJHVr1zRBesi0bBBH0+UDfifn1g25htjDEeoBsY/Zn/E8BuY8w5F60UkQdEpEJEKlpb9YVmpd/uPkVdh47mY43DZmNNaRYn2vrZWdNpdTkqxIIJ+rHe7aMn8s7bRkQW45/O+dxYT2CMedwYU26MKc/O1lGkVdweH//1ZjWL81K5boFeQSrWlBdnkOC089O3j1ldigqxYIK+Hhh5pYkCoGG8NiLiANKAjsD9AuA3wGeMMfoKCmPPVdRR2zHAP900X0fzMcjlsHPF3ExeP9jMkeZeq8tRIRRM0O8AykSkRETigHuAjaPabMS/sxXgDuBNY4wRkXTgD8BDxpgPQlW0Cr3Tbi//9cZRLivO4Nr5+qkqVl05J5MEp52fvaNjsmgyYdAH5twfBDYBB4HnjTGVIvKwiNwaaPYEkCki1cBXgDOHYD4IlAL/W0T2BP7pnEAYenLLSVp6h/jqzQt0NB/DEl0OPrmqiI17GqjvHLC6HBUiEm7HzZaXl5uKigqry4gZ67fVctrt5QevHaZoRiL3XVlsdUnKYtfOz+aa77/Fpy+fzbduXWx1OSpIIrLTGFM+1mN6ZqzircMtDA57uWlxrtWlqDCQl57AbSvy2bCjlva+cw6SUxFIgz7GtfcNseVYOytnZzArLcHqclSY+PzaOQx5/BeEV5FPgz7Gbapswm4Tblyoo3n1Z6U5Kdy0KJcnN5+kb8hjdTnqIjmsLkBZZ8fJDg409HDDwhxSdb15FbB+Wy0Ac7KS2VTZzD+/sJerRy1Vfe/qIitKU5OkI/oY5fUZ/vV3laQlOFlTqodTqnMVzkhkbnYS7x1tw+3RC5NEMg36GPXMthqqGnu4ZclMvRasGtd1C3LpG/Kw46RebjCS6Ts8BrX3DfGDTYe5cm4mS/PTrC5HhbGSrCRKspJ490irXm4wgmnQx6Dvv3qYAbeXh9ct1pOj1ISuX5BDr47qI5oGfYzZdryd5yrquH9NCaU5KVaXoyJASVYSxZmJOqqPYBr0MWRw2MtDv9lPQUYCX76hzOpyVIQQEW5YmEvPoIdtJ3RUH4k06GPIT96q5nhrP9+5fSmJcXpkrQrenOxkSnOSeftwC0PDXqvLURdIgz5GHG7q5afvHOP2Fflco1eOUpNw06JcBtxe3j/WZnUp6gJp0McAt8fHV57fQ2q8k29+ZKHV5agIVZCRyKJZqbx/tI3OfrfV5agLoEEfAx59q5rKhh6+8/GlZCbrJXvV5N24KBe3x8eP3zhqdSnqAuhEbZQ6cxp7fecAP3vnGCsK02nvc5/drtRk5KbGU148g6e31vDpy2dTmpNsdUkqCDqij2JDHi/PV9SR7HLw0UvyrC5HRYkbF+WS4LTznVcOWl2KCpIGfRTbuKeB9j43d5UXkhBnt7ocFSWSXQ6+eH0pbx5q4Z0jrVaXo4KgQR+ldtV0sruui+sW5DAnWz9eq9C678piijMT+fbLlQzq4ZZhT4M+Ch1q6uF3e09RkpXEhxboJXpV6Lkcdr69bgnHW/v5yVvVVpejJqBBH2U6+9383VMVxDvt3F1eiE3XslFTZO28bD6+Ip+fvH2MQ009VpejzkODPop4vD6+sH4Xzd1DfHr1bL2YiJpy3/zoIlITnHzt1/vx+ozV5ahxaNBHCWMM/7qxks3H2vm325dQOCPR6pJUDJiRFMe/fmwRe+u6eEyncMKWBn2U+PEbR3lmWy2fXzuXO8sLrS5HxZBbl+Vx2/I8fvT6EbbromdhSYM+Cjy9tYYfvX6UT6ws4Gs3z7e6HBVjRIT/e/tSimYk8uUNu3V5hDCkQR/hntlWwzd/e4DrFuTw3U8s1QuJKEskuxw8eu9K2vqG+OKzu3Xd+jCjSyBEoDPLGGw+1sbv9zUyPzeFtfOyeaGi3uLKVCxbkp/Gd25fyj+/uI9v/uaADjzCiAZ9BPIZwxsHW3jrcAuLZqVyz6pCHDb9cKasd2d5IXUdA/znm9UUZSbyhQ+VWl2SQoM+4px2e9mwo44Dp7q5dHYGty3Px27TUZMKH/944zzqOk/zH5sO47AJn1s71+qSYp4GfQSpbunjS8/u5mBjD7csmcma0iz9aKwsMdEqqCuLMqhu6ePf/3iIIY+PL15Xqq9VC2nQRwBjDBt21PHwy1XEO2389RWzWTAz1eqylBqX3SbcfVkhZbnJ/PBPR2juGeRfP7aYOIdOMVpBgz7MVbf08c3f7mfr8Q6uKs3kh3ct542DLVaXpdSEbCL84I5lZCe7+H/vHudwUy8/+fRKclLirS4t5mjQh9/BXqMAAAjUSURBVKm2viF++vYxntpykgSnnX+7fQmfvKwIm87HqwhiswkPfXghS/LT+OqL+7j5R+/x7VsX89FLZulUzjTSoA8zzT2D/HLzSZ7cfJLBYS93XFrAV29eQJZeAlBFsI8ty2P+zBT++YW9fPHZ3Wzc28A3PryQ4qwkq0uLCRr0YcAYQ0VNJ89uq+XlfQ14fIaPLJ3FP944j7m6lryKEvNyU/j131/JLz44wSN/OsoNP3yHuy8r5AsfKiUvPcHq8qKaBr1FjDEcae7jlf2N/G7PKU62D5AUZ+dTq2fzN1cVMztTRzoq+jjsNh64Zi63rcjn0TerWb+tlg076rh58Uz++orZrCqeodOTU0CDfpqs31ZL35CHE239HGvt41hLH+39bgQozkrijpUFfHvdYpJc+l+iol9OSjwPr1vCA9fM4aGX9vPGoWb+sL+R9AQnlxSkMX9mKkUzEv/iHJF7VxdZWHFk01SZIkMeL0ea+qhq7KayoYfXKptp6hkEIM5hoyQziTVlWSyalUpKvH/deA15FWsKMhK5Zcksrl+QS1VjN3vrunm/uo13j7adfZ+U5iQzNzsZr8/oyYGTJMaE18UCysvLTUVFhdVlBO0X75+grW+Itj534OsQLT1DtPQOcuY6DHEOG4UZCczNTmZOdjL56Qn6glVqHKfdXo639VHd4v/XHlgNM8FpZ1FeKkvyUlmSn8bCWamUZCXpAClARHYaY8rHfCyYoBeRm4EfA3bg58aY74563AU8BVwKtAN3G2NOBh57CLgf8AJfMsZsOt9zhVvQe7w+mnoGqe88zanO09R3nqa+c4Ca9gGOt/XT1jd0tq0A6YlOslNc5KUlMCs9gby0eDKS4vSSfkpNUueAmxNt/aTEO6g81UNlQzf97j9fkDw31UVJVhIlWckUZCSQmxpPTorr7NfUBGdMDKzOF/QT/ikUETvwGHAjUA/sEJGNxpiqEc3uBzqNMaUicg/wPeBuEVkE3AMsBvKA10VknjFmSi8bb4zBZ8DrM3h9Bo/PF/hq8PkMQx4ffUMe+oc8ga9eOgfctPe5ae8for3PTWvfEA1dp2nsHjznEmk5KS6KZiRy3YJsuk97yEqOIyvZxYykOJx2PfNPqVDKSIwjoygOgLKcFG5dnkd7n5umnkHaA5+iG7oG2VvXzenhsaMl3mkjKc5BksuB2+PDYRdsIthtgsPm/2q3+bedub9gVgpOuy3wT87ejrPbcATux9ltuJw2XA4bLqedeIedeKcNV+BrvNNOvNNOnMOGXQQRzj6vTZi2cwmC+cyzCqg2xhwHEJENwDpgZNCvA74VuP0i8Kj4f4N1wAZjzBBwQkSqAz9vS2jK/7N99V3c+bMtZwN9stISnGQlx5GZ5KJ8dgYFGYnkZyRQkJFAQUYis9LiiXfaz7afaM0PpVRo2UTITnGRnXLuuSW3rcgLTJ36p09beoboHfQw4PYP6gbcXg439Y4YBBoGh714jcHnA4/P4A0MDE+09eP2+vB4DcNe30Xlyvl/H/+SESLCR5bO4pG7l4f8OYIJ+nygbsT9emD1eG2MMR4R6QYyA9u3jvre/NFPICIPAA8E7vaJyOGgqh9fFtB2kT8jmmn/TEz76Py0f85vUv3zI+BH90z6OWeP90AwQT/WZ4vRf9rGaxPM92KMeRx4PIhagiIiFePNVSntn2BoH52f9s/5hVv/BDOhXA+MvNp0AdAwXhsRcQBpQEeQ36uUUmoKBRP0O4AyESkRkTj8O1c3jmqzEbgvcPsO4E3jP5xnI3CPiLhEpAQoA7aHpnSllFLBmHDqJjDn/iCwCf/hlb8wxlSKyMNAhTFmI/AE8KvAztYO/H8MCLR7Hv+OWw/whak+4iYgZNNAUUr7Z2LaR+en/XN+YdU/YXfClFJKqdDSg76VUirKadArpVSUi/igF5F/FJFKETkgIs+KSHxgx/E2ETkqIs8FdiLHJBH5cqBvKkXkHwLbZojInwL98ycRybC6zukkIr8QkRYROTBi25h9In7/KSLVIrJPRFZaV/n0GaeP7gy8jnwiUj6q/UOBPjosIn81/RVPr3H65z9E5FDgdfIbEUkf8Zil/RPRQS8i+cCXgHJjzBL8O4vPLMHwiDGmDOjEv0RDzBGRJcDf4T8beRnwUREpA74OvBHonzcC92PJL4GbR20br09uwX+0WBn+k/p+Ok01Wu2XnNtHB4CPA++O3DhqqZObgZ8Elk6JZr/k3P75E7DEGHMJcAR4CMKjfyI66AMcQELg+P1EoBG4Dv9SDABPArdZVJvVFgJbjTEDxhgP8A5wO/6lKZ4MtIm5/jHGvIv/6LCRxuuTdcBTxm8rkC4is6anUuuM1UfGmIPGmLHOWj+71Ikx5gRwZqmTqDVO/7wWeJ+Bf0WAgsBty/snooPeGHMK+AFQiz/gu4GdQNeIDh9z2YUYcQC4RkQyRSQR+DD+E9hyjTGNAIGvORbWGC7G65OxlgCJ1dfTeLSPzvVZ4I+B25b3T0QHfWAedR1Qgn91zCT8H7VHi8ljSI0xB/FPY/0JeBXYi/98BhW8oJbxiHHaRyOIyDfwv8+eObNpjGbT2j8RHfTADcAJY0yrMWYYeAm4Ev/H6zMng8X0sgvGmCeMMSuNMdfg/6h5FGg+M/0Q+NpiZY1hYrw+0WU8JqZ9FCAi9wEfBT5l/nySkuX9E+lBXwtcLiKJgWWRr8d/Fu5b+JdiAP/SDL+zqD7LiUhO4GsR/h1pz/KXS1bEdP+MMF6fbAQ+Ezj65nKg+8wUjzpLlzrh7AWavgbcaowZGPGQ9f1jjInof8C3gUP456N/BbiAOYGOrAZeAFxW12lh/7yH/4/fXuD6wLZM/EeWHA18nWF1ndPcJ8/i36czjH+0df94fYL/Y/djwDFgP/4jvCz/HSzqo9sDt4eAZmDTiPbfCPTRYeAWq+u3qH+q8c/F7wn8+1m49I8ugaCUUlEu0qdulFJKTUCDXimlopwGvVJKRTkNeqWUinIa9EopFeU06JVSKspp0CulVJT7/2ngBMa6nkqiAAAAAElFTkSuQmCC\n", 916 | "text/plain": [ 917 | "
" 918 | ] 919 | }, 920 | "metadata": { 921 | "needs_background": "light" 922 | }, 923 | "output_type": "display_data" 924 | } 925 | ], 926 | "source": [ 927 | "import numpy as np\n", 928 | "import seaborn as sns\n", 929 | "\n", 930 | "# 1: population\n", 931 | "np.random.seed(123)\n", 932 | "\n", 933 | "regular_user = np.random.normal(loc = 100, scale = 5, size = 1000)\n", 934 | "sns.distplot(regular_user)" 935 | ] 936 | }, 937 | { 938 | "cell_type": "code", 939 | "execution_count": 55, 940 | "metadata": {}, 941 | "outputs": [ 942 | { 943 | "data": { 944 | "text/plain": [ 945 | "" 946 | ] 947 | }, 948 | "execution_count": 55, 949 | "metadata": {}, 950 | "output_type": "execute_result" 951 | }, 952 | { 953 | "data": { 954 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD4CAYAAAAHHSreAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXgc1Zno/+8rtVr7vluSLdmW8Ybxohg7QAKYxSyxyQDBJCHODQmZTJjcJHfmDtz5JTfDb+Z5YDITMvmFkCHADIGAAYcEhTg4LE6AgBfZeJONbXnXYlubJWtf+v390WVopJbVsiVVS3o/z6NHVadOHb1dauntOnXqlKgqxhhjTKAItwMwxhgTfiw5GGOM6ceSgzHGmH4sORhjjOnHkoMxxph+PG4HMBwyMjK0sLDQ7TCMMWZM2bp1a52qZgbbNi6SQ2FhIWVlZW6HYYwxY4qIHB1om3UrGWOM6ceSgzHGmH4sORhjjOnHkoMxxph+LDkYY4zpx5KDMcaYfiw5GGOM6ceSgzHGmH5CSg4islxE9olIhYjcF2R7tIg872zfJCKFTvm1IrJVRHY5368O2GeRU14hIj8REXHK00TkNRE54HxPHZ6XaowxJlSD3iEtIpHAI8C1QCWwRURKVXVPQLW7gUZVnS4iq4CHgDuAOuAzqlotInOB9UCes8+jwD3ARmAdsBz4A3Af8IaqPugkovuAf7jwl2rM6Ht20zFXf/7nL53s6s83Y1coZw6LgQpVPaSqXcAaYGWfOiuBp5zltcAyERFVfV9Vq53yciDGOcvIBZJU9T31P4rul8AtQdp6KqDcGGPMKAklOeQBxwPWK/no03+/OqraAzQB6X3q3Aq8r6qdTv3KAdrMVtUap60aICtYUCJyj4iUiUhZbW1tCC/DGGNMqEJJDhKkrO+Dp89ZR0Tm4O9q+voQ2jwnVX1MVUtUtSQzM+ikgsYYY85TKMmhEigIWM8HqgeqIyIeIBlocNbzgd8AX1LVgwH18wdo86TT7YTz/VSoL8YYY8zwCCU5bAGKRaRIRLzAKqC0T51SYLWzfBvwpqqqiKQAvwfuV9W/nK3sdBedEZElziilLwEvB2lrdUC5McaYUTJocnCuIdyLf6TRXuAFVS0XkQdEZIVT7QkgXUQqgO/iH2GEs9904Hsist35OnsN4RvA40AFcBD/SCWAB4FrReQA/hFSD17oizTGGDM04h8sNLaVlJSoPezHhCMbymrCmYhsVdWSYNvsDmljjDH9WHIwxhjTjyUHY4wx/VhyMMYY048lB2OMMf1YcjDGGNOPJQdjjDH9WHIwxhjTjyUHY4wx/VhyMMYY048lB2OMMf1YcjDGGNOPJQdjjDH9WHIwxhjTjyUHY4wx/VhyMMYY048lB2OMMf2ElBxEZLmI7BORChG5L8j2aBF53tm+SUQKnfJ0EdkgIi0i8tOA+okBjw3dLiJ1IvJjZ9uXRaQ2YNtXh+elGmOMCZVnsAoiEgk8gv95zpXAFhEpVdU9AdXuBhpVdbqIrAIeAu4AOoDvAXOdLwBU9QwwP+BnbAVeCmjveVW997xflTHGmAsSypnDYqBCVQ+pahewBljZp85K4ClneS2wTEREVVtV9R38SSIoESkGsoC3hxy9McaYERFKcsgDjgesVzplQeuoag/QBKSHGMOd+M8UNKDsVhHZKSJrRaQg2E4ico+IlIlIWW1tbYg/yhhjTChCSQ4SpEzPo85AVgHPBaz/DihU1XnA63x0RvLxxlUfU9USVS3JzMwM8UcZY4wJRSjJoRII/PSeD1QPVEdEPEAy0DBYwyJyCeBR1a1ny1S1XlU7ndVfAItCiNEYY8wwCiU5bAGKRaRIRLz4P+mX9qlTCqx2lm8D3uzTTTSQO/n4WQMikhuwugLYG0I7xhhjhtGgo5VUtUdE7gXWA5HAk6paLiIPAGWqWgo8ATwtIhX4zxhWnd1fRI4ASYBXRG4BrgsY6fQ54MY+P/JbIrIC6HHa+vIFvD5jjDHnYdDkAKCq64B1fcq+H7DcAdw+wL6F52h3apCy+4H7Q4nLGGPMyLA7pI0xxvRjycEYY0w/lhyMMcb0Y8nBGGNMP5YcjDHG9GPJwRhjTD+WHIwxxvRjycEYY0w/Id0EZ4wZulPNHeyuauJ4YxtN7d20dvagCgkxHpJjoyjOSqQoI57IiGDzVhrjLksOxgyjU2c6+N2OGn77fhW7qpoA8EQISbFRJER7EKCysZ3y6mbePlBHbFQknyhM5cqLsoiJinQ3eGMCWHIw5gK1dPawfvcJfru9ir9U1OFTmJuXxH03zKSprZvclBg8ER/vwe3q8VFx6gw7Kpt460AdW4+dZvmcHBZOTkHEziSM+yw5GHMeunt9vH2glt+8X81re07Q0e0jPzWWv7lyOrcsmMT0rEQAnt10LOj+Xk8EsyclM3tSMlc0tvHKzhp+va2SysY2bp43ybqajOssORgTos6eXjYdauC1PSf5/a4aGlq7SImL4taF+Xx2QR6LpqSe16f+/NQ47vnUVNaXn+DtA3U0tnVx5ycmE23dTMZFlhyMCcLnU6qb2jlY28qO46cpO9rI1iMNtHb1EhMVwbJZ2dwyP49Pz8jE67nwQX8RItwwN5eM+Ghe3lHFs5uP8aWlhXYGYVxjycGMuIG6VkbLyvmT2FXVRGVjOzWn2znd3k1bVw9tXb20dvbS3t3j/97VS2tXD+1dvZzp6KGr1/dhGzOyE7hlQR5Xz8zisukZI3bx+BNFaYjAS+9X8fL2Kj67IM+uQRhXWHIw49KZjm62HGlkb00z33t5N72+jx5MGO+NJC7aQ5w3kjiv/3tijIfspGjivR5ivZEkxHiYkhbP1Mx4ZuUkkRwXNWqxlxSm0dDaxZ/215KeEM2nZ9gz0s3oCyk5iMhy4D/wPwnucVV9sM/2aOCX+J/3XA/coapHRCQdWAt8AvhvVb03YJ8/AblAu1N0naqeGqit836FZkJp7uhm/e4T7KxsoleVKWlx/M2V01g0JZXC9HhykmPGxJDRa2ZnU9faxWt7TjAtM5781Di3QzITzKDJQUQigUeAa4FKYIuIlAY86hPgbqBRVaeLyCrgIeAOoAP4HjDX+errC6pa1qdsoLaMGZCqsvVoI+t219DTqyyemsaSonQyE6P5/KWT3Q5vyCJE+Oz8PI43tPFCWSV/e/V0oiJtQgMzekJ5ty0GKlT1kKp2AWuAlX3qrASecpbXAstERFS1VVXfwZ8kQhW0rSHsbyaYXp+ydmslL71fRU5SLN+6upjPzJtEZmK026FdkFhvJLcuzKeupZP15SfcDsdMMKEkhzzgeMB6pVMWtI6q9gBNQHoIbf+XiGwXke8FJIDzbctMQN29Pn616SjvHz/NsplZfPWKIjLGeFIIND0rgSVT03n3YD3H6lvdDsdMIKEkh2Cf2vU86vT1BVW9GLjC+bprKG2JyD0iUiYiZbW1tYP8KDMe9fh8PPXeEfadOMOKSyaxbFY2EePwJPP6Odkkxnj4/a4afDrYn5UxwyOU5FAJFASs5wPVA9UREQ+QDDScq1FVrXK+nwGexd99FXJbqvqYqpaoaklmpo3mmIhe2VnDodpWbl2Yz5Kp4/fkMtoTyXWzczje2M7Oyia3wzETRCjJYQtQLCJFIuIFVgGlfeqUAqud5duAN1UH/ogjIh4RyXCWo4Cbgd3n05aZmDYeqmfz4QY+VZzBwimpbocz4hZMTmFSSgzry0/Q1eMbfAdjLtCgycHp978XWA/sBV5Q1XIReUBEVjjVngDSRaQC+C5w39n9ReQI8CPgyyJSKSKzgWhgvYjsBLYDVcAvBmvLGIDKxjZe2VnNRdmJXDcnx+1wRkWECDddPImm9m7+crDO7XDMBBDSfQ6qug5Y16fs+wHLHcDtA+xbOECziwaoP2BbxvT4fLy0rYqEaA+fKykYl9cYBlKUEc+snETeOVDH0qnpY+J+DTN22cBpM6a8tb+OE80drJyfR6x34v1zvGpmFu3dvWw6VO92KGacs+RgxoxTzR1s2HeKi/OSmZWb5HY4rshPjWNGdgJvV9TZtQczoiw5mDHjlZ01eCMj+Mwlk9wOxVVXXZRFW1cvmw/b2YMZOZYczJhQcaqFitoWrp6ZRUL0xJ4vckq6f0LAtw/U0dNrZw9mZFhyMGHPp8r68hOkxEVxaVGa2+GEhU8XZ3Kms+fD51QbM9wsOZiwt7uqiarT7VwzKxuPTT4H+KfVyEyI5t2D9dhtQGYk2F+aCWu9PuW1PSfJTopmfkGK2+GEDRFh6bR0qk63c7yhze1wzDhkycGEtfLqJupbu1g2c3zOm3QhFkxOISYqgndtWKsZAZYcTNhSVd4+UEd6vJfZkybm0NVzifZEUjIljd1VTTS1d7sdjhlnLDmYsHWorpWq0+1cXpxhZw0DWDI1HVXYcuSc81waM2SWHEzYevtALfHRHhZOHv8T652vtHgv07MS2Hq00abzNsPKkoMJSyeaO9h/soWlU9Pt8ZiDKClMo6m9m4pTLW6HYsYR+6szYWnjwXo8EcISu69hULNyEonzRlrXkhlWlhxM2Ons7mV75Wnm5acQN8Hvhg6FJzKChZNT2VvTTEtnj9vhmHHC/vJM2NleeZquHt+w3Q397KZjw9JOOFs0JZV3KurYdrSRT82wJyOaC2dnDiasqCqbDzeQmxxDfmqs2+GMGdlJMUxOi2Pr0Ua7Y9oMC0sOJqxUNrZT09TB4qI0xIavDsnCyanUtnRSfbrD7VDMOBBSchCR5SKyT0QqRKTfYztFJFpEnne2bxKRQqc8XUQ2iEiLiPw0oH6ciPxeRD4QkXIReTBg25dFpFZEtjtfX73wl2nGis2HG/B6Ipifb1NlDNXFeclERgjvH290OxQzDgyaHEQkEngEuAGYDdzpPAc60N1Ao6pOBx4GHnLKO4DvAX8XpOl/U9WZwALgMhG5IWDb86o63/l6fEivyIxZnT297KpqYl5eMtH2CMwhi/VGMjMnkR2VTfT6rGvJXJhQzhwWAxWqekhVu4A1wMo+dVYCTznLa4FlIiKq2qqq7+BPEh9S1TZV3eAsdwHbgPwLeB1mHCivbqar18eiKXbT2/laUJBCa2eP3fNgLlgoySEPOB6wXumUBa2jqj1AE5AeSgAikgJ8BngjoPhWEdkpImtFpGCA/e4RkTIRKautrQ3lR5kwt+1YI2nxXianxbkdypg1IzuR2KhI61oyFyyU5BDsqmDfc9ZQ6vRvWMQDPAf8RFUPOcW/AwpVdR7wOh+dkXy8cdXHVLVEVUsyM23o3lh3uq2Lw7WtLJicYheiL4AnMoKL85PZW9NMZ3ev2+GYMSyU5FAJBH56zweqB6rj/MNPBkK5XfMx4ICq/vhsgarWq2qns/oLYFEI7Zgxbvvx0yiwoMC6lC7UgoIUunuVvSea3Q7FjGGhJIctQLGIFImIF1gFlPapUwqsdpZvA97UQQZbi8g/408i3+5TnhuwugLYG0KMZgxTVbYdO01hejxp8V63wxnzCtLiSIrxsKvKkoM5f4PeIa2qPSJyL7AeiASeVNVyEXkAKFPVUuAJ4GkRqcB/xrDq7P4icgRIArwicgtwHdAM/CPwAbDN6Ub4qTMy6VsisgLocdr68jC9VhOmqk63U9fSyRXFGW6HMi5EiDA3L5nNhxs409FNYkyU2yGZMSik6TNUdR2wrk/Z9wOWO4DbB9i3cIBmg3Ysq+r9wP2hxGXGh52VTURGCHMnJbsdyrhxcV4y7x6s5429p7hlQd/xI8YMzu6QNq7yqbKz8rR/lI3X7m0YLme7ll7ZWeN2KGaMsuRgXHW0vo3mjh7m5dtZw3CKEOHivGTe2l/LmQ57hKgZOksOxlU7K08TFSnMyrFnRA+3uXnJdPX6eH3vSbdDMWOQJQfjml6fsquqiZk5SXg99lYcbgVpceQmx/D7nSfcDsWMQfYXaVxzsLaFtq5eLrEupRERIcINc3N5a38tzda1ZIbIkoNxza7KJqI9EczITnQ7lHHrpnm5dPX6eMO6lswQWXIwruj1KXtqmpmVm4Qn0t6GI2VBQYrTtWSjlszQ2F+lccXhulbau3uZO8kuRI+kiAjhxotzeWt/nXUtmSGx5GBcsbuqCW9kBMXWpTTibrzY37X0+h7rWjKhs+RgRp1PlfKaZmbkJBJlXUojbkFBCpOsa8kMkf1lmlF3tL6N1s4e61IaJRERwg0X5/L2AetaMqGz5GBG3e7qJjwRwkXWpTRqbpibQ1evjw0fnHI7FDNGWHIwo8qnyp7qZoqzE+050aNo4eRUMhOjWV9uN8SZ0FhyMKOqqrGdpvZu61IaZRERwnWzs9nwQS0d9oQ4EwJLDmZU7a5uIlKEmTaX0qhbPjeH9u5e3tpvz1w3g7PkYEaNqrK7qolpWfE2PbcLlkxNJynGw6vWtWRCEFJyEJHlIrJPRCpE5L4g26NF5Hln+yYRKXTK00Vkg4i0iMhP++yzSER2Ofv8RJzHwYlImoi8JiIHnO/2UOFxoqapg8a2bubYQ31cERUZwTWzs3l9z0m6e31uh2PC3KDJQUQigUeAG4DZwJ0iMrtPtbuBRlWdDjwMPOSUdwDfA/4uSNOPAvcAxc7Xcqf8PuANVS0G3nDWzTiwu7oJAWblWpeSW5bPyaG5o4eNh+rdDsWEuVDOHBYDFap6SFW7gDXAyj51VgJPOctrgWUiIqraqqrv4E8SHxKRXCBJVd9TVQV+CdwSpK2nAsrNGFde1UxRRjwJ0SE9ndaMgE/NyCTOG8mru61ryZxbKMkhDzgesF7plAWto6o9QBOQPkiblQO0ma2qNU5bNUBWCDGaMHeyuYPalk7m5FmXkptioiK56qIs1pefpNenbodjwlgoyUGClPV9V4VS50Lq929A5B4RKRORstpaG30R7sqdLqU51qXkuuvn5lDX0sn7xxrdDsWEsVCSQyVQELCeD1QPVEdEPEAy0DBIm/kDtHnS6XY62/0U9JZOVX1MVUtUtSQzMzOEl2HcVF7dzOS0OJJio9wOZcK76qJMvJER1rVkzimU5LAFKBaRIhHxAquA0j51SoHVzvJtwJvOtYSgnO6iMyKyxBml9CXg5SBtrQ4oN2NUfUsnNU0d1qUUJhJjori8OINXy09wjj9TM8ENmhycawj3AuuBvcALqlouIg+IyAqn2hNAuohUAN8lYISRiBwBfgR8WUQqA0Y6fQN4HKgADgJ/cMofBK4VkQPAtc66GcN2VzcDMMfuig4by+fkUNnYTrnzuzGmr5CGjajqOmBdn7LvByx3ALcPsG/hAOVlwNwg5fXAslDiMmNDeXUTeSmxpMZ53Q7FOK6ZnU3ES7C+/ARz7YzOBGF3SJsRVXW6ncrGdptLKcykxXu5tCjdrjuYAVlyMCPq7D8fu94QfpbPzeHAqRYqTrW4HYoJQ5YczIh6dXcNOUkxZCREux2K6eO6OdkANo23CcqSgxkxJ5o6KDvayNw861IKR7nJscwvSLHkYIKy5GBGzB9216CKXfAMY8vn5rCzsomq0+1uh2LCjCUHM2LW7arhouxEshJj3A7FDOD6OTkArLcL06YPSw5mRJxs9ncp3XhxrtuhmHMoyohnZk6iPePB9GPJwYyIP+zydyndNC/H7VDMIK6fk8OWIw3Unul0OxQTRiw5mBHxe6dLaXpWotuhmEEsn5uDKry+96TboZgwYsnBDDvrUhpbZuYkMiU9zm6IMx9jycEMO+tSGltEhOVzcnj3YB1N7d1uh2PChCUHM+zW7TphXUpjzPVzc+juVTZ8EHSGfDMBWXIww+pkcwdbjjZYl9IYMz8/heykaOtaMh+y5GCGlXUpjU0REcL1c3L40/5TtHf1uh2OCQOWHMywWrfrBDOyE6xLaQxaPieHjm4ff95vj901lhzMMDrldCnddPEkt0Mx52FxURopcVE215IBLDmYYfSH3SesS2kM80RGcO2sbF7fe5KuHp/b4RiXhZQcRGS5iOwTkQoRuS/I9mgRed7ZvklECgO23e+U7xOR652yi0Rke8BXs4h829n2AxGpCth24/C8VDPSSndU2yilMW753BzOdPTw3qF6t0MxLhs0OYhIJPAIcAMwG7gz4DnQZ90NNKrqdOBh4CFn39nAKmAOsBz4mYhEquo+VZ2vqvOBRUAb8JuA9h4+u915RKkJc8fq29h6tJGVC6xLaSy7bHoG8d5IG7VkQjpzWAxUqOohVe0C1gAr+9RZCTzlLK8FlomIOOVrVLVTVQ8DFU57gZYBB1X16Pm+COO+l7dXAbDiEksOY1lMVCRXzczitT0n6PWp2+EYF4WSHPKA4wHrlU5Z0Dqq2gM0Aekh7rsKeK5P2b0islNEnhSR1GBBicg9IlImImW1tTa6wk2qym+3V7G4MI381Di3wzEXaPncHOpauth6tNHtUIyLQkkOEqSs70eKgeqcc18R8QIrgBcDtj8KTAPmAzXAvwcLSlUfU9USVS3JzMwcOHoz4sqrmzlY22pdSuPElRdl4fVEWNfSBBdKcqgECgLW84HqgeqIiAdIBhpC2PcGYJuqfjgdpKqeVNVeVfUBv6B/N5QJM799v4qoSOEmuyt6XEiI9vCp4gzWl59A1bqWJqpQksMWoFhEipxP+quA0j51SoHVzvJtwJvqf1eVAquc0UxFQDGwOWC/O+nTpSQigf9hPgvsDvXFmNHX61N+t7OaKy/KIiXO63Y4ZphcPyeHqtPt7K5qdjsU4xLPYBVUtUdE7gXWA5HAk6paLiIPAGWqWgo8ATwtIhX4zxhWOfuWi8gLwB6gB/imqvYCiEgccC3w9T4/8l9FZD7+7qcjQbabMPLWgVpONnfyVwv6XkoyY9k1s7KJjBBeLa/h4nx7BvhENGhyAHCGk67rU/b9gOUO4PYB9v0X4F+ClLfhv2jdt/yuUGIy4WFtWSWpcVEsm5XtdihmGKXGe1kyNY1Xd5/g76+f6XY4xgV2h7Q5b6fbunhtz0lWzs/D67G30nizfE4OB2tbqTh1xu1QjAvsL9qct5e3V9PV6+P2kny3QzEj4Lo5/mlQbNTSxGTJwZy3F7ceZ3ZuEnMmWZ/0eJSdFMPCySm8ahPxTUiWHMx52VvTzO6qZj5nZw3j2vK5OeyuauZ4Q5vboZhRZsnBnJfnNh/D64lg5XwbpTSeLZ/jH1m+bleNy5GY0WbJwQxZa2cPL22r4qaLc0mNt3sbxrPJ6XFcUpBC6Y6+972a8c6Sgxmy0h3VtHT28MUlk90OxYyCFZdMory6mYpTLW6HYkaRJQczJKrKMxuPMjMnkYWTg86JaMaZz8zLRQQ7e5hgLDmYIdlR2UR5dTNfWDIF/6zsZrzLSorhk9PSKd1eZXMtTSCWHMyQPLPxKPHeSD5r02VMKCsumcSR+jZ2VTW5HYoZJZYcTMhqz3RSuqOazy7MIyE6pJlXzDixfE4u3sgIXt5uXUsThSUHE7JnNh6lq8fHVy4rcjsUM8qS46K48qJMXt5eTU+vz+1wzCiw5GBC0tHdyzMbj3LNrCymZia4HY5xwV8tzKeupZO3K+rcDsWMAksOJiQvb6+ivrWLr1xuZw0T1dUzs0iNi+LXWyvdDsWMAksOZlCqyuNvH2Z2bhJLp/abZd1MEF5PBCsumcQf95ykqb3b7XDMCLPkYAa1Yd8pDpxq4e7Li2z46gT3Vwvz6erx2XQaE4AlB3NOqspP3qggPzWWFfMnuR2Ocdm8/GSmZyVY19IEEFJyEJHlIrJPRCpE5L4g26NF5Hln+yYRKQzYdr9Tvk9Erg8oPyIiu0Rku4iUBZSnichrInLA+W634broLxX1bD9+mm9cOY2oSPssMdGJCLcuzKfsaCOH61rdDseMoEH/2kUkEngEuAGYDdwpIrP7VLsbaFTV6cDDwEPOvrPxP096DrAc+JnT3llXqep8VS0JKLsPeENVi4E3nHXjkv/vzQPkJMVw2yKbmtv43bowj8gIYc2WY26HYkZQKB8FFwMVqnpIVbuANcDKPnVWAk85y2uBZeLvnF4JrFHVTlU9DFQ47Z1LYFtPAbeEEKMZAZsPN7DpcANf//RUoj2Rg+9gJoSspBiunpnFr7dW0tVj9zyMV6EkhzzgeMB6pVMWtI6q9gBNQPog+yrwRxHZKiL3BNTJVtUap60aICtYUCJyj4iUiUhZbW1tCC/DDIWq8qPX9pGR4GXVJ2z2VfNxn188mbqWLl7fe9LtUMwICSU5BBue0nf2rYHqnGvfy1R1If7uqm+KyKdCiOWjRlQfU9USVS3JzMwcyq4mBH/eX8vGQw387dXFxHrtrMF83KdmZDIpOYbnNlvX0ngVygQ5lUBBwHo+0HeClbN1KkXEAyQDDefaV1XPfj8lIr/B3930FnBSRHJVtUZEcoFTQ35V5mOe3TS0P2CfKo9sqCAt3ovI0Pc34WMkf3czc5PY8MEpfvqm/70SzOcvtbPOsSqUM4ctQLGIFImIF/8F5tI+dUqB1c7ybcCb6p/btxRY5YxmKgKKgc0iEi8iiQAiEg9cB+wO0tZq4OXze2nmfO2qbKKmqYNrZmXjibARSia4kin+gYRbjjS4HIkZCYOeOahqj4jcC6wHIoEnVbVcRB4AylS1FHgCeFpEKvCfMaxy9i0XkReAPUAP8E1V7RWRbOA3zg1VHuBZVX3V+ZEPAi+IyN3AMeD2YXy9ZhDdvT5e23uS3OQY5uUnux2OCWMpcV5m5iSy5UgDV8/MsqHO40xI8y6r6jpgXZ+y7wcsdzDAP3FV/RfgX/qUHQIuGaB+PbAslLjM8PtLRR0NrV38j8sKibC7oc0glk7LYO+Jw+yqbGLhFLslaTyxVG8+1NTezYZ9p5idm0RxVqLb4ZgxYFpmPFmJ0bx7qM6eEjfOWHIwH3p1dw2qcOPFuW6HYsYIEWHptHSqT3dwrKHN7XDMMLLkYAA4XNfKjsomrijOHHDkiTHBLChIJSYqgncP1rsdihlGlhwM3b0+fvN+JalxUXx6ht0zYobG64mgZEoa5dVNNLZ2uR2OGSaWHAwb9p2irqWLWxbk4fXYW8IM3WXTMxCEd+wpceOG/SeY4Gqa2nlrfy0LJ6fYRWhz3pJjo5g/OYWyow20dPa4HY4ZBpYcJrAen49fb6sk1uvhxjlD3XAAABHuSURBVLl2EdpcmCuKM+jpVd6zaw/jgiWHCezNvaeoPt3BLfMnERcd0i0vxgwoKzGG2ZOS2Hions7uXrfDMRfIksMEdaSulT/vr2XRlFTmTLI7oc3w+PSMTNq7e3nvkJ09jHWWHCagju5eXtx6nNR4LzfbPQ1mGOWnxnFRdiJvHailvcvOHsYySw4TjKry622VNLV3c/uifKKjbDpuM7yunZ1NR7ePdyrsOStjmSWHCea9Q/WUVzdz/ZwcpqTHux2OGYcmpcRycV4yf6mop66l0+1wzHmy5DCBHG9o4w+7TjArJ5HLp2e4HY4Zx66ZlU13r49HNlS4HYo5T5YcJojmjm5+tekoSbEebltUgNiMq2YEZSZGU1KYytPvHaXi1Bm3wzHnwZLDBNDT6+NXG4/S3t3LF5dMscd+mlFx7ewcYr2R/NPv9tiMrWOQJYdxTlV5eXs1xxvbuX1RAbnJsW6HZCaIhGgP3712Bm8fqOOPe066HY4ZopCSg4gsF5F9IlIhIvcF2R4tIs872zeJSGHAtvud8n0icr1TViAiG0Rkr4iUi8j/DKj/AxGpEpHtzteNF/4yJ66f/ekgW481cvXMLObm2f0MZnTdtWQKM7IT+H9f2WNDW8eYQZODiEQCjwA3ALOBO0Vkdp9qdwONqjodeBh4yNl3Nv5Hhs4BlgM/c9rrAf6Xqs4ClgDf7NPmw6o63/n62BPoTOhe3l7FD9fv45L8ZJbNzHI7HDMBeSIjeGDlXCob2/nh+n1uh2OGIJQzh8VAhaoeUtUuYA2wsk+dlcBTzvJaYJn4r3iuBNaoaqeqHgYqgMWqWqOq2wBU9QywF8i78Jdjznr3YB1//+JOLi1K49aF+XYB2rhmydR0vrR0Cv/17mE22Z3TY0YoySEPOB6wXkn/f+Qf1lHVHqAJSA9lX6cLagGwKaD4XhHZKSJPikjQB9OKyD0iUiYiZbW1drNNoPePNfK1p8oozIjjsbtK8NiD343L7rthJpPT4vj7tTtptVlbx4RQ/msE+8jZd+jBQHXOua+IJAC/Br6tqs1O8aPANGA+UAP8e7CgVPUxVS1R1ZLMTHtAzVl7a5pZ/eRmMhKjeebuS0mOi3I7JGOI83r4t9sv4XhjG/+3tNxGL40BoSSHSqAgYD0fqB6ojoh4gGSg4Vz7ikgU/sTwK1V96WwFVT2pqr2q6gN+gb9by4TgUG0Ldz2xifhoD8/cfSlZSTFuh2TMhz5RmMbfXl3M2q2VPLv5mNvhmEGEkhy2AMUiUiQiXvwXmEv71CkFVjvLtwFvqv+jQSmwyhnNVAQUA5ud6xFPAHtV9UeBDYlI4ExwnwV2D/VFTUSVjW188fFNqMIzX72UgrQ4t0Mypp9vLyvmyosy+UFpOduONbodjjmHQZODcw3hXmA9/gvHL6hquYg8ICIrnGpPAOkiUgF8F7jP2bcceAHYA7wKfFNVe4HLgLuAq4MMWf1XEdklIjuBq4DvDNeLHa9qmtr54uObaOns4em7L2VaZoLbIRkTVESE8OM75pObHMtfP72VY/VtbodkBiDjoe+vpKREy8rK3A7DFUfrW/n8LzbR3N7Nf39lMYum9L9+/+wmO4U37vj8pZODlu8/eYbP/ed7JMVE8eJfLyXbukBdISJbVbUk2DYbxjKG7T95htt//h5tXT08+7UlQRODMeFoRnYi//0/FlPf0sldT2yi3mZvDTuWHMaoXZVN3PGf7wHw/NeXcnG+3f1sxpb5BSn8YnUJR+vbuPXRdzlS1+p2SCaAJYcxaMuRBj7/i43EeT28+NdLmZGd6HZIxpyXT07L4NmvLaGpvZu/evRdth61i9ThwpLDGFO6o5ovPr6JzKRo1n5jqT2wx4x5i6ak8tLfXEZijIc7/vM9HtlQQa9v7F8LHessOYwRPp/yo9f2863n3mdefjIvfn2pzbBqxo2ijHhKv3k518/N4Yfr93HnYxvZd8KeA+EmSw5jQHtXL/c+t42fvHGA2xfl88xXLyU9IdrtsIwZVslxUfz0zgX8++2XsO/kGW74j7e4/6VdnGjqcDu0CcnjdgDm3E40dfC1X5axu7qJ/3PjTL52xVSbRM+MWyLCrYvyWTYrix+/foBnNh7lxbLj3Dwvl9WfLGR+QYq9/0eJJYcw9vaBWr7z/Hbau3p5/EslLJuV7XZIxoyKlDgvP1gxh69cVsR/vXuYF7Yc57fbq5mcFsfN83K5amYW8wtSiLJJJUeM3QQXhrp6fPz49f08+ueDFGcl8MjnF1J8ASOS7CY4M9Z1dPeyu6qJXVVNHKxtwafg9UQwOS2OvJTYD79S4qI+dmYx0E14xu9cN8HZmUOY+eBEM//rhR2UVzdzR0kBP1gxx575bCa8mKhISgrTKClMo72rl0N1LVScauF4QxtvH6jl7OCmOG8kuckx5CbHkpMcw/yCFKZnJeD12BnGUFlyCBMd3b38558P8ciGChJjPPz8i4tYPjfH7bCMCTux3kjmTEpmziT/jZ/dvT5ONndQ2dhO9el2apo62Hionh6fsnZrJVGRwvSsRGblJjI7N4lZuUnMzk0iNd7r8isJb5YcwsCf99fyg9JyDte1cvO8XP5pxRwbjWRMiKIiI8hPjSM/9aOZiHt9Sl1LJ1Mz49lbc4a9Nc28c6COl7ZVfVhnamY8JVNSKZmSRklhKkUZ8XaxO4AlBxftrmrioVc/4O0DdRRlxPP03Yu5otgeXGTMhYqMELKTYlg5P4+V8z8qr2/pZG/NGXZVNbH1aAN/3HOSF8oqAchI8HLp1HQ+OS2dpVPTJ3yysOTggvePNfKzPx3ktT0nSY2L4ns3z+aLSyYT7bFrC8aMpPSEaC4vjuby4gxgGj6fcqiuhS1HGtl8uIH3Dtbz+501AGQnRbN0ajqfnJbB0mnpE+4ZKZYcRklnTy+v7j7BrzYeY/ORBpJjo/jWsmK+ekURSTH2KE9j3BAR4b8eMT0rkTsXT0ZVOVzXynuH6nnvYD3vVNTx2+3+B1/mpcSy1DmrWDotnUkp43uGAksOI6jXp5QdaaB0RzXrdtXQ2NbNlPQ4/p+bZrFq8WQSou3wGxNORISpmQlMzUzgC5dOQVWpONXCuwf9yeL1vSdZu9XfDVWYHsfSaekscZJFVuL4eiaF/XcaRj6fcri+lfePneadA7X8eX8tjW3dxEZFsmxWFp8rKeDy6RlEREzcfkxjxhIRoTg7keLsRFZ/shCfT9l7opn3Dtaz8VA9r+yo4bnNxwGYlhnPoimpzkgq/6io+DH8ATCkyEVkOfAfQCTwuKo+2Gd7NPBLYBFQD9yhqkecbfcDdwO9wLdUdf252nSeNb0GSAO2AXepateFvczh19TWzbGGNo42tLLvxBm2Hz/NjuOnae7oASAt3stVF2Vx1cwsls3KIs47dt8kxhi/iAj5cBjtV6+YSq9PKa9u4l0nWby+99SHF7hFoCg9nqmZCRRlxDElPZ6ijHimpMeRlRgT9vdeDPofS0QigUeAa4FKYIuIlKrqnoBqdwONqjpdRFYBDwF3iMhsYBUwB5gEvC4iM5x9BmrzIeBhVV0jIj932n50OF5sX6fbuqg900lHt4/Onl46e5zv3T46e3y0d/fS2NZFY2sXDa3dNDr1j9a3fpgEwD8yYkZ2IjfNm8SCghQuKUihOCvBzhCMGeciI4R5+SnMy0/hrz89DVXlRHMH5VXNlFc3s6emicN1rbx9oJbOHt/H9k2NiyIrMYaspGgyEqJJjo0iMcZDUkwUSbEe4rwevJ4IvJ4IoiMjPlz2eiLwREQQIf4zm/QE74hctwzl4+xioEJVDwGIyBpgJRCYHFYCP3CW1wI/Ff8YsJXAGlXtBA6LSIXTHsHaFJG9wNXA5506TzntjkhyWLPlOA/+4YNB68VGRZIW7yUlLoqMhGjmF6QwOS2OgrQ4JqfFUZQRb3cxG2MQEXKTY8lNjuWa2R/Nhebz+ZPGkfpWjtW3cbK5k1NnOjh1ppNTZzo5VNvKmY5uznT2MNQZjf75lrl8ccmUYX4loSWHPOB4wHolcOlAdVS1R0SagHSnfGOfffOc5WBtpgOnVbUnSP2PEZF7gHuc1RYR2TfI68gA6gap4waLa2gsrqEJx7hGLaYvDK16OB4rGCSuux6Cu86/7QGzSijJIVjfSN/cNlCdgcqDdbadq37/QtXHgMeCbQtGRMoGmmDKTRbX0FhcQxOOcYVjTGBx9RXKFZFKoCBgPR+oHqiOiHiAZKDhHPsOVF4HpDhtDPSzjDHGjLBQksMWoFhEikTEi/8Cc2mfOqXAamf5NuBN9c8FXgqsEpFoZxRSMbB5oDadfTY4beC0+fL5vzxjjDHnY9BuJecawr3AevzDTp9U1XIReQAoU9VS4AngaeeCcwP+f/Y49V7Af/G6B/imqvYCBGvT+ZH/AKwRkX8G3nfaHg4hd0GNMotraCyuoQnHuMIxJrC4PmZcPOzHGGPM8ArvuzCMMca4wpKDMcaYfsZtchCRSBF5X0RecdaLRGSTiBwQkeedC+GjHVOKiKwVkQ9EZK+ILBWRNBF5zYnrNRFJdSGu74hIuYjsFpHnRCTGjeMlIk+KyCkR2R1QFvT4iN9PRKRCRHaKyMJRjuuHzu9xp4j8RkRSArbd78S1T0SuH824Arb9nYioiGQ4664eL6f8b51jUi4i/xpQ7trxEpH5IrJRRLaLSJmILHbKR/N4FYjIBud/QrmI/E+n3N33vqqOyy/gu8CzwCvO+gvAKmf558A3XIjpKeCrzrIXSAH+FbjPKbsPeGiUY8oDDgOxAcfpy24cL+BTwEJgd0BZ0OMD3Aj8Af+9MUuATaMc13WAx1l+KCCu2cAOIBooAg4CkaMVl1NegH+wx1EgI0yO11XA60C0s54VDscL+CNwQ8Ax+pMLxysXWOgsJwL7nePi6nt/XJ45iEg+cBPwuLMu+KflWOtUeQq4ZZRjSsL/5nwCQFW7VPU0/ilGnnIrLocHiBX//SVxQA0uHC9VfQv/aLdAAx2flcAv1W8j/vtjckcrLlX9o350J/9G/PfknI1rjap2quphIHDKmBGPy/Ew8L/5+A2krh4v4BvAg+qfSgdVPRUQl5vHS4EkZzmZj+6rGs3jVaOq25zlM8Be/B/aXH3vj8vkAPwY/x/H2ZmuQp6WYwRNBWqB/3K6ux4XkXggW1VrwP8mAbJGMyhVrQL+DTiGPyk0AVtx/3idNdDxCTati1sxfgX/JzlwOS4RWQFUqeqOPpvcPl4zgCucrso/i8gnwiSubwM/FJHj+P8O7nczLhEpBBYAm3D5vT/ukoOI3AycUtWtgcVBqo72GF4P/lPaR1V1AdCK/1TRVU4/5kr8p/STgHjghiBVw23Mczj8ThGRf8R/D8+vzhYFqTYqcYlIHPCPwPeDbQ5SNprHywOk4u8G+XvgBeeM3u24vgF8R1ULgO/w0X1Vox6XiCQAvwa+rarN56oapGzYYxt3yQG4DFghIkfwPxfiavxnEm5Py1EJVKrqJmd9Lf5kcfLsKaHz/dQA+4+Ua4DDqlqrqt3AS8Ancf94nTXQ8QllWpcRJSKrgZuBL6jTGexyXNPwJ/kdzvs/H9gmIjkux4Xz819yukI24z+rzwiDuFbjf88DvMhHXVqjGpeIROFPDL9S1bPxuPreH3fJQVXvV9V8VS3Ef6f2m6r6BVyelkNVTwDHReQip2gZ/jvHA6cecWO6kGPAEhGJcz7JnY0rXKYxGej4lAJfckZuLAGazp6CjwbxP6zqH4AVqtrWJ95gU8aMOFXdpapZqlrovP8r8V/oPIHLxwv4Lf4Paoj/mS5e/HOpuXa8HNXAp53lq4EDzvKoHS/n7+4JYK+q/ihgk7vv/ZG6Ah8OX8CVfDRaaSr+N10F/k8I0S7EMx8oA3bi/2NJxX895A38b8o3gDQX4von4ANgN/A0/pEjo368gOfwX/foxv+P7e6Bjg/+U+tH8I9u2QWUjHJcFfj7fbc7Xz8PqP+PTlz7cEbCjFZcfbYf4aPRSm4fLy/wjPMe2wZcHQ7HC7gc/zW2Hfj7+Re5cLwux98ttDPg/XSj2+99mz7DGGNMP+OuW8kYY8yFs+RgjDGmH0sOxhhj+rHkYIwxph9LDsYYY/qx5GCMMaYfSw7GGGP6+f8B7EqKh0xaGS8AAAAASUVORK5CYII=\n", 955 | "text/plain": [ 956 | "
" 957 | ] 958 | }, 959 | "metadata": { 960 | "needs_background": "light" 961 | }, 962 | "output_type": "display_data" 963 | } 964 | ], 965 | "source": [ 966 | "# 5% heavy users \n", 967 | "heavy_users = np.random.normal(loc=120,scale = 25,size = 50)\n", 968 | "sns.distplot(heavy_users)" 969 | ] 970 | }, 971 | { 972 | "cell_type": "code", 973 | "execution_count": 56, 974 | "metadata": {}, 975 | "outputs": [], 976 | "source": [ 977 | "total_user = []\n", 978 | "\n", 979 | "for i in heavy_users:\n", 980 | " total_user.append(i)\n", 981 | "for j in regular_user:\n", 982 | " total_user.append(j)" 983 | ] 984 | }, 985 | { 986 | "cell_type": "code", 987 | "execution_count": 60, 988 | "metadata": {}, 989 | "outputs": [ 990 | { 991 | "data": { 992 | "text/plain": [ 993 | "Ttest_indResult(statistic=0.40481427665266323, pvalue=0.6856966445371957)" 994 | ] 995 | }, 996 | "execution_count": 60, 997 | "metadata": {}, 998 | "output_type": "execute_result" 999 | } 1000 | ], 1001 | "source": [ 1002 | "#1. simulate the hashing process & single iteration\n", 1003 | "\n", 1004 | "import random\n", 1005 | "\n", 1006 | "A_1 = []\n", 1007 | "\n", 1008 | "A_2 = []\n", 1009 | "\n", 1010 | "for i in total_user:\n", 1011 | " \n", 1012 | " hash_val = random.random()\n", 1013 | " \n", 1014 | " if hash_val <= 0.5:\n", 1015 | " A_2.append(i)\n", 1016 | " \n", 1017 | " else:\n", 1018 | " A_1.append(i)\n", 1019 | " \n", 1020 | "# two sample t test \n", 1021 | "from scipy import stats\n", 1022 | "\n", 1023 | "stats.ttest_ind(A_1,A_2)" 1024 | ] 1025 | }, 1026 | { 1027 | "cell_type": "code", 1028 | "execution_count": 61, 1029 | "metadata": {}, 1030 | "outputs": [], 1031 | "source": [ 1032 | "# no difference " 1033 | ] 1034 | }, 1035 | { 1036 | "cell_type": "code", 1037 | "execution_count": 72, 1038 | "metadata": {}, 1039 | "outputs": [ 1040 | { 1041 | "data": { 1042 | "text/plain": [ 1043 | "" 1044 | ] 1045 | }, 1046 | "execution_count": 72, 1047 | "metadata": {}, 1048 | "output_type": "execute_result" 1049 | }, 1050 | { 1051 | "data": { 1052 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxc5X3v8c9vFu2rLdmWJcu7jY2xMRgDWYCEkEBLcJO2CZD00jQtN6+EhjS9uaWv9JK8uF1ukjbpbUPa0ISW5DYsoUnqgFNCCNmAgBfwhrGRbVmrbS3Wvs88948Z2WN5JI3tkY505vt+vfTSnDmPRj+Oh6+eec5znmPOOUREZPYLeF2AiIikhwJdRMQnFOgiIj6hQBcR8QkFuoiIT4S8+sVlZWVuyZIlXv16EZFZaefOna3OufJk+zwL9CVLlrBjxw6vfr2IyKxkZsfG26chFxERn1Cgi4j4hAJdRMQnFOgiIj6hQBcR8QkFuoiIT6QU6GZ2s5kdNLMaM7svyf7fN7MWM3st/vWH6S9VREQmMuk8dDMLAg8CNwENwHYz2+qce31M08edc/dMQY0iIpKCVHrom4Ea59wR59wQ8BiwZWrLEhGR85XKlaKVQH3CdgNwdZJ2v21m1wGHgD9xztWPbWBmdwN3A1RXV59/tdPgOy/XTbj/zqtnZt0iIqn00C3Jc2Nvc/RDYIlzbj3wE+CRZC/knHvIObfJObepvDzpUgQiInKBUgn0BmBRwnYV0JTYwDnX5pwbjG/+C3BlesoTEZFUpRLo24GVZrbUzLKA24GtiQ3MrCJh8zbgQPpKFBGRVEw6hu6cGzGze4BngCDwsHNuv5k9AOxwzm0FPmlmtwEjQDvw+1NYs4iIJJHS8rnOuW3AtjHP3Z/w+M+BP09vaSIicj50paiIiE8o0EVEfEKBLiLiEwp0ERGfUKCLiPiEAl1ExCcU6CIiPqFAFxHxCQW6iIhPKNBFRHxCgS4i4hMKdBERn1Cgi4j4hAJdRMQnFOgiIj6hQBcR8QkFuoiITyjQRUR8QoEuIuITCnQREZ9QoIuI+IQCXUTEJxToIiI+oUAXEfEJBbqIiE8o0EVEfEKBLiLiEwp0ERGfUKCLiPiEAl1ExCcU6CIiPqFAFxHxiZQC3cxuNrODZlZjZvdN0O53zMyZ2ab0lSgiIqmYNNDNLAg8CNwCrAXuMLO1SdoVAp8EXk53kSIiMrlUeuibgRrn3BHn3BDwGLAlSbv/DXwRGEhjfSIikqJUAr0SqE/Ybog/d5qZbQQWOeeemuiFzOxuM9thZjtaWlrOu1gRERlfKoFuSZ5zp3eaBYCvAH862Qs55x5yzm1yzm0qLy9PvUoREZlUKoHeACxK2K4CmhK2C4F1wM/MrBa4BtiqE6MiItMrlUDfDqw0s6VmlgXcDmwd3emc63TOlTnnljjnlgC/Bm5zzu2YkopFRCSpSQPdOTcC3AM8AxwAnnDO7TezB8zstqkuUEREUhNKpZFzbhuwbcxz94/T9oaLL0tERM6XrhQVEfEJBbqIiE8o0EVEfEKBLiLiEwp0ERGfUKCLiPiEAl1ExCcU6CIiPqFAFxHxCQW6iIhPKNBFRHxCgS4i4hMKdBERn1Cgi4j4hAJdRMQnFOgiIj6hQBcR8QkFuoiITyjQRUR8QoEuIuITCnQREZ9QoIuI+IQCXUTEJxToF+Dg8W5u+vLP+eNHX6VvaMTrckREAAX6eXtm/3He/7UXaO0Z5Kk9TXzg6y/R3NnvdVkiIgr083G0tZf//u2drJhfyI/uvY6H77qK2tY+fvtrL9I7qJ66iHhLgX4edh07RWF2iEf/6GoWFOfwjkvm8a8fuYqmzgH+7cVar8sTkQynQE/RSCTK/uZObrp0PnlZodPPX7VkDu9aM59//vlhOvuGPaxQRDKdAj1Fb57sYWA4yns3LDxn35++exU9gyN8/ReHPahMRCRGgZ6ivY2d5IaDvG1F2Tn71lQUcduGhfzrC7Wc7B7woDoREQV6SoYjUV5v7mJdZRHhYPJD9ifvWsXgSIRvvXhsmqsTEYlRoKfg4PFuhkaiXFZZMm6bJWX53LB6Hk/sqGckEp3G6kREYhToKdjX1El+doilZfkTtrtjczUnuwd57o2T01SZiMgZKQW6md1sZgfNrMbM7kuy/2NmttfMXjOzX5nZ2vSX6p2GU/0sLcsnGLAJ271jdTnzi7J59JW6aapMROSMSQPdzILAg8AtwFrgjiSB/R3n3GXOucuBLwJfTnulHukfitDeO8TC4pxJ24aCAT64aRE/P9RCw6m+aahOROSM0ORN2AzUOOeOAJjZY8AW4PXRBs65roT2+YBLZ5FeGr2sf2FJLgDfeXni3vcHN1fzj8/X8MT2ej797tVTXp+IyKhUhlwqgfqE7Yb4c2cxs0+Y2WFiPfRPJnshM7vbzHaY2Y6WlpYLqXfaNXWcHeiTqSzJ5fpV5Ty5s4Fo1Dd/10RkFkgl0JMNHJ+TVM65B51zy4E/A/4i2Qs55x5yzm1yzm0qLy8/v0o90tQ5QFFOiILsVD7MxLxvYyVNnQPsOHZqCisTETlbKinVACxK2K4CmiZo/xjwTxdT1EzS1NGfcu8cYkMygyMRwkHjb398kN+6/OwPM3deXZ3uEkVEgNR66NuBlWa21MyygNuBrYkNzGxlwuZvAm+mr0TvDI1EaekePK9AB8gOBVlTUcTehk5GopqTLiLTY9JAd86NAPcAzwAHgCecc/vN7AEzuy3e7B4z229mrwGfBu6asoqn0fGuARykNMNlrMurSugfjlBzsif9hYmIJJHSwLBzbhuwbcxz9yc8vjfNdc0I53tCNNGK+QXkhoPsru/gkgVF6S5NROQculJ0Ak0d/eSGgxTnhs/7Z0OBAOsqi3m9uYuhEQ27iMjUU6BPoLlzgIUlOZhNfIXoeDYsKmY44jhwvGvyxiIiF0mBPo5I1HG8a+CChltGLZmbT1FOiN31HWmsTEQkOQX6ONp6B4lEHQuKzv+E6KiAGRuqSjh0ops+3XNURKaYAn0c7T1DAMwtyL6o19mwqISog31NGnYRkamlQB9Ha28s0Mvysy7qdSqKcygvyGZ3g4ZdRGRqKdDH0dYzSE44QG5W8KJex8zYsKiY2tZeOvqG0lSdiMi5FOjjaO8dYm5+9gXPcEm0oaoER+y+pCIiU0WBPo623iHmFlzccMuouQXZVJXmsqdBgS4iU0eBnsRINMqp3iHmXuT4eaLLKotp7OintrU3ba8pIpJIgZ5ER98wDpibf3EzXBJdVlkMwNN7m9P2miIiiRToSbSdnrKYvh56SV4W1XPy+OHuiVYeFhG5cAr0JNp6BwGYk8YhF4D1VcW8cbybmpPdaX1dERFQoCfV1jNEVihwXncpSsW6hcWYwQ93a9hFRNJPgZ5EW+8gZflZaZmymKgoN8zmJXN4ak8Tzul+oyKSXgr0JNp6hphzkZf8j+fWDQs53NLLG8c17CIi6aVAHyMSdZzqS++UxUS3rFtAwOCpPTo5KiLppUAfo7N/mKhjygK9rCCbt64o46k9zRp2EZG0UqCP0dYTm+FysassTuTW9RUca+tjX6NWYBSR9FGgj9EWX2VxqnroAO+5dAGhgPFDDbuISBop0Mdo7x0iFDAKc9I7ZTFRSV4Wb19ZxtMadhGRNFKgj9HRN0RJXjjtUxbHunX9Qho7+tlVp3XSRSQ9FOhjdPQPU5I3dcMto266dD5ZwYBmu4hI2ijQxzjVN0xJbnjKf09RTpjrV5ezbW8z0aiGXUTk4inQEwwMR+gdHJmWHjrEZruc6Bpke237tPw+EfG3qTvzNws1dvQDUJo3dT3077xcd/rx4EiEcND48rOH2HJ5JQB3Xl09Zb9bRPxNPfQEjadigT5dPfTsUJDVC4rY19hJRMMuInKRFOgJRnvoJVPYQx9rfWUxvUMRjupORiJykRToCRpP9ROw2AnL6bJ6QSFZoQB7GjR9UUQujgI9QWNHP0U5YYKBqZ2DnigcDLC2ooh9TZ0MR6LT9ntFxH8U6AkaT/VP63DLqMsXlTAwHOWgltQVkYugQE/Q2NE/bSdEEy0vL6AwO8Sr9Rp2EZELl1Kgm9nNZnbQzGrM7L4k+z9tZq+b2R4ze87MFqe/1Kk1EolyvGvAkx56MGCsryrm0PFuTsUXBxMROV+TBrqZBYEHgVuAtcAdZrZ2TLNXgU3OufXAk8AX013oVDveNUAk6ijNnf4eOsDG6lIizvHUXt1vVEQuTCo99M1AjXPuiHNuCHgM2JLYwDn3vHOuL775a6AqvWVOvTNz0Ke/hw5QUZzDvMJsvr+rwZPfLyKzXyqBXgnUJ2w3xJ8bz0eBHyXbYWZ3m9kOM9vR0tKSepXT4MwcdG966GbGxupSdtV1aE66iFyQVAI92Ry+pJc1mtmHgU3Al5Ltd8495Jzb5JzbVF5ennqV08DrHjrAxkUlBAPG49vrJ28sIjJGKoHeACxK2K4Czlnz1czeBXwWuM05N5ie8qZPY0c/ZQVZhIPeTfwpyg3zjtXzeHJng+aki8h5SyW9tgMrzWypmWUBtwNbExuY2Ubg68TC/GT6y5x6jR39VJbkel0Gd2xeRGvPIM8dOOF1KSIyy0wa6M65EeAe4BngAPCEc26/mT1gZrfFm30JKAC+a2avmdnWcV5uxmo81U9lqfeBfv2qcuYXZfPoKxp2EZHzk9Lyuc65bcC2Mc/dn/D4XWmua1o552js6OfGNfO8LoVQMMAHNi3iq8/XzJhPDSIyO+hKUaC1Z4jBkeiMCc8PbIqdsnj8lbpJWoqInKEbXHBmymJlaR4t3d6ezx29AcaqeYV881dHmVuQfdaJWt0AQ0TGox46Z6YszpQeOsDbVpbROxRht9Z3EZEUKdCBxo7YRa4z4aToqGVl+SwoyuGFw604p7sZicjkFOjEeuiF2SGKc727qGgsM+OtK8o40TVITUuP1+WIyCygQCc+B30G9c5HbagqpiA7xAs1rV6XIiKzgAIdaDg1M6cHhoIBrl0+l0Mnek6P84uIjEeBzsztoQNcu2wuueEgP9GVoyIyiYwP9K6BYboHRmZkDx0gJxzkulXlHDzRTV2bVmEUkfFlfKCfnrI4Q3voEOul52eHeFa9dBGZgAJ9Bs5BHysrFOD6VeUcbunlRZ0gFZFxKNA7Zn4PHeDqpXMozQvz+R/u19K6IpKUAr2jn6xQgLL8bK9LmVA4GODW9Qs5dKKHR16s9bocEZmBFOjxKYuBQLIbM80slywo5J2XzOPvf/ImJ7oGvC5HRGaYjA/0hlm0RK2Z8bn3rmUoEuWBp173uhwRmWEyPtAbZ+hFReNZPDefT75zBU/vaWbr7nPuBCgiGSyjA31gOEJrz+CMPyE61seuX87G6hL+4vt7Od6poRcRicnoQG/qmPlTFpMJBQN85QOXMxxxfObJ3USjWo1RRDI80GfLlMVklpTl8xe3ruGXb7by0C+PeF2OiMwAGX3Hovr2WKBXz8nzuJLUjd7RCAAH6yqL+eJ/vUFbzxCf/c013hUmIp7L6B76sfZesoIB5hfleF3KBTEz3r+xktK8LB7fXkdrj7e3zxMRb2V0oNe391FVmktwFsxBH09OOMidV1fTNxThU4+9RkTj6SIZK6MDva69j0WzaLhlPBXFubx3w0J+VdPKV39a43U5IuKRjB5Dr2vrY+OiUq/LSItNi0sx4O+fO8SmJaW8dUWZ1yWJyDTL2B56Z98wXQMjLJ47+3voEBtP/8v3rWNFeQH3PvaqlgYQyUAZG+jH2mM3i/DDkMuovKwQX/vQFfQORvjjR19lRKsyimSUjA30uvY+YHZNWUzFyvmF/PX71/HK0Xa+/Owhr8sRkWmUsWPoo4Hupx564hz1q5aU8rWfHaZ3cITVC4oAuPPqaq9KE5FpkLE99Pr2PubmZ1GQ7c+/abeuX0hFcQ5P7Gigo2/I63JEZBpkbKDXtfdR7ZMTosmEgwHu2FxN1DkefaWOkajG00X8LmMD/Vhbn+/Gz8cqK8jm/VdUUX+qnx/v1w2mRfwuIwN9OBKlqaPf94EOcFllMdcum8uvalr5r33HvS5HRKZQSoFuZjeb2UEzqzGz+5Lsv87MdpnZiJn9TvrLTK+mjn6izl8nRCdyy7oFVJXm8pknd1PX1ud1OSIyRSYNdDMLAg8CtwBrgTvMbO2YZnXA7wPfSXeBU8GvUxbHEwoGuOOqagz4+Hd2MjAc8bokEZkCqfTQNwM1zrkjzrkh4DFgS2ID51ytc24PMCvOvGVaoAOU5mfxdx+4nH2NXfzV0we8LkdEpkAqgV4J1CdsN8SfO29mdreZ7TCzHS0tLRfyEmlR195HVjDAglm6bO6FumntfO6+bhnf/vUx3Y9UxIdSCfRka8te0BqtzrmHnHObnHObysvLL+Ql0uJYax9Vc3IJzOJlcy/UZ96zmisXl/Ln/7GHwy09XpcjImmUSqA3AIsStquAWd29q2npYXl5gddleCIcDPDVOzeSHQ7y8f+3i/4hjaeL+EUqgb4dWGlmS80sC7gd2Dq1ZU2doZEota29rJyXmYEOsfXTv/LByzl0spv7/3Of1+WISJpMGujOuRHgHuAZ4ADwhHNuv5k9YGa3AZjZVWbWAPwu8HUz2z+VRV+M2rZeRqKOVfMLvS7FU9evKueP37GC7+5s4Ikd9ZP/gIjMeCktZOKc2wZsG/Pc/QmPtxMbipnx3jwRGzdekcE99FH3vmsVO46d4n/9YB+XVRazpqLI65JE5CL4c2WqCbx5shuzzAz0xNUYR12/qpy9jZ18+Bsv87PP3EBhTtiDykQkHTLu0v83T/RQPSePnHDQ61JmhMKcMLdfVU177xD3fW8vzukm0yKzVeYF+snujD4hmszSsnzevXY+T+9p5hu/POp1OSJygTJqyGU4EuVoay83rpnvdSkzzttXldPQ0c9fbztA/ak+Lllw7ni6bpAhMrNlVA/9WFsvwxGnHnoSATN+98pFVBTn8Pj2eo7rJtMis05GBfroDJeV8zJ7yuJ4skIBfu/aJWQFAzzyYq3udCQyy2RWoJ/sydgZLqkqzg1z11uWMDAc4eEXaukZHPG6JBFJUUYF+qET3VSV5pKbpRkuE1lYkstd1y6hs3+If3vxKH1DCnWR2SCjAr3mZI+GW1K0pCyfOzcv5mTXIA/94gid/cNelyQik8iYQB+JRDnS0svK+RpuSdXqBYXc9ZYldPYP8/VfHKbmpFZnFJnJMibQD53oYSgSZU2S6XgyvuXlBfzh25cxPBLltq/+ih+82uh1SSIyjowJ9J11pwC4cnGpx5XMPpUludzzzpVcurCITz3+Gv/ju7tp6xn0uiwRGSNjAn3XsVOUF2ZTVZrrdSmzUnFumEf/6Bo+8Y7lfP/VRm7425/xjV8e0f1JRWaQjAn0Hcfa2bS4FLPMu0tRuoSCAT7znkv4r3vfzsbqUv7y6QNc8zfP8TfbDnBEdz8S8VxGXPp/smuA+vZ+7rp2idel+MLK+YU88pGreOVoO4+8VMs3fnWUr//iCPOLsllTUcTy8gKq5+QRDp7pL2jZAJGplxGBvis+fn6Fxs/Txsy4etlcrl42l+OdA/xoXzPfeukYPz/Yws8OthAKGIvm5LGsLJ+l5fkMjkTIDmn+v8hUyohA31F7iqxQgEsXaobLVFhQnMNH3rqU7FCQgeEIta29HGnt5UhrDz994yTuDfj2S8e4orqUKxeXsrG6hMsXlTC3INvr0kV8JSMCfWfdKTZUFauHOA1ywkEuqSjikvjdj/qHItS29ZIVCvDy0Tb+6eeHiURja64vnpvHxkUlbKyOhfyaiqKzhmlE5Pz4PtAHhiPsa+zkD9621OtSMlJuVvD0re2WlxcwNBKlsaOf+vY+6k/18dwbJ/nBa02xtuEgG6tLuGrJHDYvncPG6hLysnz/FhVJG9//37KvsZPhiOPKao2fX6xkt7A7X1mhAEvL8llalg+Ac47O/mHq2vs41tbH0dZeXjrchgMC8YXULqss5v73Xkpxrm6PJzIR3wf6y0fbAZ0QnanMjJK8LEryslhfVQLEPlUda+vjSEsP+5o6+Y9djTy9t5ktGyr5vWsXs66y2OOqRWYm3wf6U3ua2VhdQplOwM0aOeEgqxcUsnpBITevW0BjRz/ba9v53qsNPL6jnuXl+Vy/ah7Ly/NPX1egaZEiPg/0mpPdHGju4nPvXet1KXKBzIyq0jyqSvO4ZV0Frxxt54XDrTz8wlGqSnO5flX56TF6kUzn60DfuruZgMFvXlbhdSmSBjnhINetKufa5XN5ta6DX7zZwr+/XEd5QTbZoQBbLq8kK6RZMpK5fPvud87xw91NXLNsLvOKcrwuR9IoHAyweekcPn3TKm6/ahGhoPGZJ/dw/Zee5x+ee5Pmzn6vSxTxhG976Pubujja2svd1y3zuhSZIgEz1leVcFllMZWluXzjl0f58rOH+PufHOKaZXN5z6ULuHHNPKpK87wuVWRa+DbQt+5uIhw0blm3wOtSZIqZGTesnscNq+dR19bHd3fWs21vM5/bup/Pbd1PRXEOG6tLuKK6lI3VpVy6sIicsC4yE//xZaD3DI7wvV2NXLeynJK8LK/LkWmQOEe+ojiXj75tGS3dg7x5spu69j5eOtzGtr3Hgdj89rKCbBYU51BRlMOC4lw+/o7lzCvM1mqcMqv5MtAffL6G1p5B7nnnCq9LEQ+VF2ZTXpjNW5bHtrsGhqlv76Oxo5/jnQPUtfWxp6ETgEdeqqU0L8yaiiLWVBRxyYJC1lQUsWJegXrzMmv4LtBrW3v55i+P8v4rKtmoq0MlQVFOmEsXFnPpwjMXJvUPRTjeNUBzZzzk2/vYXtvOcCS23sxob76iONaTryjOYUFRDoU5Ic2BlxnHd4H+l08fIBw07rv5Eq9LkVkgNyt41lIEAFHnaOsZOh3yx7sGqG3rY3e8Nw+QlxU8He7ZoQBrF8Z681pcTLzkq0D/txeO8pMDJ/izmy/RVEW5YAGz08M166vOPN8/FKG5Kx7ynQM0dw7w8tF2XjjcBkBWMMDK+QWsjQ/brKkoYklZHvMLcwgENDYvU88Xge6c4x+eq+ErPznEu9bM56NaWVGmQG5WkGVlBSwrKzj9XCTquHb5HPY3dfF6cxevN3Xx0zdO8t2dDafbZIcCLJqTx+I5eVSW5jI3P5s5+WHm5GczJz+L0vwwRTlhinPD5GUFdWJWLlhKgW5mNwP/FwgC33DO/Z8x+7OBbwFXAm3AB51ztektNbn9TZ187fnDPL23mfdfUckXf3s9IX3slWkSDBgr5hWyYl4hWy6vBGIdjJbuQQ4cj82wqWvrPb2a5PbadroGRiZ8vaKcEEW5sYAvygnT0TdEblaQ/OwQhTlhCrNDFOXEHv/hdUu1zr+cNmmgm1kQeBC4CWgAtpvZVufc6wnNPgqccs6tMLPbgS8AH5yKglu6B9lVd4q9DZ28crSdV2rbycsKcu+NK7n3xpX6aCvTbqJlhYNmLC0rYOmYXn3f0Ai9gxF6h0boG4owMByhf/R7/GtgOEJH33BseyhC7+AIbszrf+nHBynODTOvMJt5RdnMK8xhXny4aF5R7HFJXpjccJCccJCcUJDscIBQwAiYYcas/kTgnEt4fPa+qHNEnCMSTfLlHCMRR9Q5RqKOaDT2fXT/SDS+L94mEnWEAkYoGCAcNMLBAKGgEQrEtkPBAOHAmefDwdgxDsbzKOpgOBI9/drZocCUzJ5KpYe+Gahxzh0BMLPHgC1AYqBvAT4ff/wk8FUzM+fGHuKL98SOer70zEGCAWPlvAL+582r+dDmxRTnaa1smR2CAYv1tHPO7z0bdY7ewRG6B0boHhime2CEpWX5nOwe5GT3ACe7B3nlaDst3YMMRaIpv64ZGLFzB4H4RsDAOBP0bsyfksT/s8/5n/ysfeMHbuLm2Kg4e9+k/wkzllny+v/qfev40NWL0/77Ugn0SqA+YbsBuHq8Ns65ETPrBOYCrYmNzOxu4O74Zo+ZHbyQokcdAZ4BPnExL3KuMsbUPcOp3qk1m+qdTbVCBtf74S/Ahy/8x8f9S5BKoCf7PDb2b04qbXDOPQQ8lMLv9IyZ7XDObfK6jlSp3qk1m+qdTbWC6p0KqZw9bAAWJWxXAU3jtTGzEFAMtKejQBERSU0qgb4dWGlmS80sC7gd2DqmzVbgrvjj3wF+OhXj5yIiMr5Jh1ziY+L3EBuuDgIPO+f2m9kDwA7n3Fbgm8C3zayGWM/89qkseorN6CGhJFTv1JpN9c6mWkH1pp2pIy0i4g+6AkdExCcU6CIiPpGxgW5mN5vZQTOrMbP7kuzPNrPH4/tfNrMl01/l6VoWmdnzZnbAzPab2b1J2txgZp1m9lr8634vak2op9bM9sZr2ZFkv5nZP8SP7x4zu8KjOlcnHLPXzKzLzD41po2nx9bMHjazk2a2L+G5OWb2rJm9Gf+edK1oM7sr3uZNM7srWZtpqvdLZvZG/N/6+2ZWMs7PTvi+mcZ6P29mjQn/5r8xzs9OmCPTzjmXcV/ETu4eBpYBWcBuYO2YNh8H/jn++HbgcQ/rrQCuiD8uBA4lqfcG4Cmvj21CPbVA2QT7fwP4EbFrGK4BXp4BNQeB48DimXRsgeuAK4B9Cc99Ebgv/vg+4AtJfm4Osevv5gCl8celHtX7biAUf/yFZPWm8r6Zxno/D/yPFN4vE+bIdH9lag/99HIGzrkhYHQ5g0RbgEfij58EbjSPFr1wzjU753bFH3cDB4hdnTubbQG+5WJ+DZSYWYXHNd0IHHbOHfO4jrM4537Budd1JL4/HwF+K8mPvgd41jnX7pw7BTwL3DxlhcYlq9c592Pn3OiqZL8mdj3LjDDO8U1FKjkyrTI10JMtZzA2IM9azgAYXc7AU/Ghn43Ay0l2X2tmu83sR2Z26bQWdi4H/NjMdsaXfBgrlX+D6XY78Og4+2bSsQWY75xrhtgffGBekjYz8RgD/AGxT2fJTPa+mU73xIeIHh5nSGvGHd9MDfS0LWcwncysAPgP4FPOua4xu3cRGyrYAPwj8IPprm+MtzrnrgBuAT5hZteN2T+jjm/8ornbgO8m2T3Tjm2qZtQxBjCzzwIjwL+P02Sy9xmPzwsAAAHZSURBVM10+SdgOXA50Az8XZI2M+74Zmqgz7rlDMwsTCzM/905972x+51zXc65nvjjbUDYzMqmuczEepri308C3yf28TRRKv8G0+kWYJdz7sTYHTPt2MadGB2iin8/maTNjDrG8ZOytwIfcvFB6LFSeN9MC+fcCedcxDkXBf5lnDpm1PGFzA30WbWcQXzs/pvAAefcl8dps2B0jN/MNhP7t22bvirPqiXfzApHHxM7IbZvTLOtwH+Lz3a5BugcHULwyB2MM9wyk45tgsT3513AfyZp8wzwbjMrjQ8ZvDv+3LSz2E1y/gy4zTnXN06bVN4302LM+Zz3jVNHKjkyvbw8I+vlF7FZFoeInaX+bPy5B4i94QByiH38rgFeAZZ5WOvbiH2U2wO8Fv/6DeBjwMfibe4B9hM70/5r4C0e1rssXsfueE2jxzexXiN245TDwF5gk4f15hEL6OKE52bMsSX2h6YZGCbWK/wosfM5zwFvxr/PibfdROyuYqM/+wfx93AN8BEP660hNt48+v4dnUG2ENg20fvGo3q/HX9f7iEW0hVj641vn5MjXn7p0n8REZ/I1CEXERHfUaCLiPiEAl1ExCcU6CIiPqFAFxHxCQW6iIhPKNBFRHzi/wMwqKcdJtquLQAAAABJRU5ErkJggg==\n", 1053 | "text/plain": [ 1054 | "
" 1055 | ] 1056 | }, 1057 | "metadata": { 1058 | "needs_background": "light" 1059 | }, 1060 | "output_type": "display_data" 1061 | } 1062 | ], 1063 | "source": [ 1064 | "exponential = np.random.exponential(scale=2, size=1000)\n", 1065 | "sns.distplot(exponential)" 1066 | ] 1067 | }, 1068 | { 1069 | "cell_type": "code", 1070 | "execution_count": 73, 1071 | "metadata": {}, 1072 | "outputs": [ 1073 | { 1074 | "data": { 1075 | "text/plain": [ 1076 | "Ttest_indResult(statistic=0.1348387526112502, pvalue=0.8927665540225909)" 1077 | ] 1078 | }, 1079 | "execution_count": 73, 1080 | "metadata": {}, 1081 | "output_type": "execute_result" 1082 | } 1083 | ], 1084 | "source": [ 1085 | "#1. simulate the hashing process & single iteration\n", 1086 | "\n", 1087 | "import random\n", 1088 | "\n", 1089 | "A_1 = []\n", 1090 | "\n", 1091 | "A_2 = []\n", 1092 | "\n", 1093 | "for i in exponential:\n", 1094 | " \n", 1095 | " hash_val = random.random()\n", 1096 | " \n", 1097 | " if hash_val <= 0.5:\n", 1098 | " A_2.append(i)\n", 1099 | " \n", 1100 | " else:\n", 1101 | " A_1.append(i)\n", 1102 | " \n", 1103 | "# two sample t test \n", 1104 | "from scipy import stats\n", 1105 | "\n", 1106 | "stats.ttest_ind(A_1,A_2)" 1107 | ] 1108 | }, 1109 | { 1110 | "cell_type": "markdown", 1111 | "metadata": {}, 1112 | "source": [ 1113 | "---" 1114 | ] 1115 | }, 1116 | { 1117 | "cell_type": "markdown", 1118 | "metadata": {}, 1119 | "source": [ 1120 | "# Part 4: Check for Variability " 1121 | ] 1122 | }, 1123 | { 1124 | "cell_type": "code", 1125 | "execution_count": 23, 1126 | "metadata": {}, 1127 | "outputs": [], 1128 | "source": [ 1129 | "# Bootstrap & 70% sample size\n", 1130 | "\n", 1131 | "boot_diffs = []\n", 1132 | "\n", 1133 | "for i in range(10000):\n", 1134 | " boot_sample_1 = np.random.choice(A_1, replace=False, size = int(0.7*len(A_1)))\n", 1135 | " \n", 1136 | " boot_sample_2 = np.random.choice(A_2, replace= False, size = int(0.7*len(A_2)))\n", 1137 | " \n", 1138 | " result = stats.ttest_ind(boot_sample_1,boot_sample_2)\n", 1139 | "\n", 1140 | " boot_diffs.append(result.pvalue)" 1141 | ] 1142 | }, 1143 | { 1144 | "cell_type": "code", 1145 | "execution_count": 25, 1146 | "metadata": {}, 1147 | "outputs": [ 1148 | { 1149 | "data": { 1150 | "text/plain": [ 1151 | "0.0004" 1152 | ] 1153 | }, 1154 | "execution_count": 25, 1155 | "metadata": {}, 1156 | "output_type": "execute_result" 1157 | } 1158 | ], 1159 | "source": [ 1160 | "count= 0 \n", 1161 | "\n", 1162 | "for i in boot_diffs:\n", 1163 | " if i < 0.05:\n", 1164 | " count+=1\n", 1165 | " \n", 1166 | "count/len(boot_diffs)" 1167 | ] 1168 | }, 1169 | { 1170 | "cell_type": "markdown", 1171 | "metadata": {}, 1172 | "source": [ 1173 | "---" 1174 | ] 1175 | }, 1176 | { 1177 | "cell_type": "code", 1178 | "execution_count": 26, 1179 | "metadata": {}, 1180 | "outputs": [], 1181 | "source": [ 1182 | "# Bootstrap & 80% sample size\n", 1183 | "\n", 1184 | "boot_diffs = []\n", 1185 | "\n", 1186 | "for i in range(10000):\n", 1187 | " boot_sample_1 = np.random.choice(A_1, replace=False, size = int(0.8*len(A_1)))\n", 1188 | " \n", 1189 | " boot_sample_2 = np.random.choice(A_2, replace= False, size = int(0.8*len(A_2)))\n", 1190 | " \n", 1191 | " result = stats.ttest_ind(boot_sample_1,boot_sample_2)\n", 1192 | "\n", 1193 | " boot_diffs.append(result.pvalue)" 1194 | ] 1195 | }, 1196 | { 1197 | "cell_type": "code", 1198 | "execution_count": 27, 1199 | "metadata": {}, 1200 | "outputs": [ 1201 | { 1202 | "data": { 1203 | "text/plain": [ 1204 | "0.0001" 1205 | ] 1206 | }, 1207 | "execution_count": 27, 1208 | "metadata": {}, 1209 | "output_type": "execute_result" 1210 | } 1211 | ], 1212 | "source": [ 1213 | "count= 0 \n", 1214 | "\n", 1215 | "for i in boot_diffs:\n", 1216 | " if i < 0.05:\n", 1217 | " count+=1\n", 1218 | " \n", 1219 | "count/len(boot_diffs)" 1220 | ] 1221 | }, 1222 | { 1223 | "cell_type": "markdown", 1224 | "metadata": {}, 1225 | "source": [ 1226 | "---" 1227 | ] 1228 | }, 1229 | { 1230 | "cell_type": "code", 1231 | "execution_count": 28, 1232 | "metadata": {}, 1233 | "outputs": [], 1234 | "source": [ 1235 | "# Bootstrap & 90% sample size\n", 1236 | "\n", 1237 | "boot_diffs = []\n", 1238 | "\n", 1239 | "for i in range(10000):\n", 1240 | " boot_sample_1 = np.random.choice(A_1, replace=False, size = int(0.9*len(A_1)))\n", 1241 | " \n", 1242 | " boot_sample_2 = np.random.choice(A_2, replace= False, size = int(0.9*len(A_2)))\n", 1243 | " \n", 1244 | " result = stats.ttest_ind(boot_sample_1,boot_sample_2)\n", 1245 | "\n", 1246 | " boot_diffs.append(result.pvalue)" 1247 | ] 1248 | }, 1249 | { 1250 | "cell_type": "code", 1251 | "execution_count": 29, 1252 | "metadata": {}, 1253 | "outputs": [ 1254 | { 1255 | "data": { 1256 | "text/plain": [ 1257 | "0.0" 1258 | ] 1259 | }, 1260 | "execution_count": 29, 1261 | "metadata": {}, 1262 | "output_type": "execute_result" 1263 | } 1264 | ], 1265 | "source": [ 1266 | "count= 0 \n", 1267 | "\n", 1268 | "for i in boot_diffs:\n", 1269 | " if i < 0.05:\n", 1270 | " count+=1\n", 1271 | " \n", 1272 | "count/len(boot_diffs)" 1273 | ] 1274 | }, 1275 | { 1276 | "cell_type": "markdown", 1277 | "metadata": {}, 1278 | "source": [ 1279 | "# Conclusion 1: for equal split & 90% sample size, the p value is 0.0101 (False Positive Rate);\n", 1280 | "- Note: set replace = False; otherwise, it returns a high p value" 1281 | ] 1282 | }, 1283 | { 1284 | "cell_type": "markdown", 1285 | "metadata": {}, 1286 | "source": [ 1287 | "---" 1288 | ] 1289 | }, 1290 | { 1291 | "cell_type": "code", 1292 | "execution_count": 93, 1293 | "metadata": {}, 1294 | "outputs": [], 1295 | "source": [ 1296 | "# Bootstrap & 100% sample size\n", 1297 | "\n", 1298 | "boot_diffs_2 = []\n", 1299 | "\n", 1300 | "for i in range(10000):\n", 1301 | " \n", 1302 | " boot_sample_1 = np.random.choice(A_1, replace=False, size = len(A_1))\n", 1303 | " \n", 1304 | " boot_sample_2 = np.random.choice(A_2, replace= False, size = len(A_2))\n", 1305 | " \n", 1306 | " result = stats.ttest_ind(boot_sample_1,boot_sample_2)\n", 1307 | "\n", 1308 | " boot_diffs_2.append(result.pvalue)" 1309 | ] 1310 | }, 1311 | { 1312 | "cell_type": "code", 1313 | "execution_count": 94, 1314 | "metadata": {}, 1315 | "outputs": [ 1316 | { 1317 | "data": { 1318 | "text/plain": [ 1319 | "0.0" 1320 | ] 1321 | }, 1322 | "execution_count": 94, 1323 | "metadata": {}, 1324 | "output_type": "execute_result" 1325 | } 1326 | ], 1327 | "source": [ 1328 | "count= 0 \n", 1329 | "for i in boot_diffs_2:\n", 1330 | " if i < 0.05:\n", 1331 | " count+=1\n", 1332 | "count/len(boot_diffs_2)" 1333 | ] 1334 | }, 1335 | { 1336 | "cell_type": "code", 1337 | "execution_count": 97, 1338 | "metadata": {}, 1339 | "outputs": [ 1340 | { 1341 | "data": { 1342 | "text/plain": [ 1343 | "{0.19427810106851318,\n", 1344 | " 0.19427810106856205,\n", 1345 | " 0.19427810106856214,\n", 1346 | " 0.1942781010685622,\n", 1347 | " 0.19427810106861107,\n", 1348 | " 0.19427810106866003}" 1349 | ] 1350 | }, 1351 | "execution_count": 97, 1352 | "metadata": {}, 1353 | "output_type": "execute_result" 1354 | } 1355 | ], 1356 | "source": [ 1357 | "set(boot_diffs_2)" 1358 | ] 1359 | }, 1360 | { 1361 | "cell_type": "markdown", 1362 | "metadata": {}, 1363 | "source": [ 1364 | "# Conclusion 2: for equal split & 100% sample size, 0 percent of observing different distributions (False Positive Rate)\n", 1365 | "- Note: set replace = False; otherwise, it returns a high p value" 1366 | ] 1367 | }, 1368 | { 1369 | "cell_type": "markdown", 1370 | "metadata": {}, 1371 | "source": [ 1372 | "---" 1373 | ] 1374 | }, 1375 | { 1376 | "cell_type": "markdown", 1377 | "metadata": {}, 1378 | "source": [ 1379 | "# Test for SRM & use a chi-square test" 1380 | ] 1381 | }, 1382 | { 1383 | "cell_type": "code", 1384 | "execution_count": 30, 1385 | "metadata": {}, 1386 | "outputs": [ 1387 | { 1388 | "data": { 1389 | "text/plain": [ 1390 | "Power_divergenceResult(statistic=25.0, pvalue=5.733031437583875e-07)" 1391 | ] 1392 | }, 1393 | "execution_count": 30, 1394 | "metadata": {}, 1395 | "output_type": "execute_result" 1396 | } 1397 | ], 1398 | "source": [ 1399 | "from scipy.stats import chisquare \n", 1400 | "half = int((len(A_1)+len(A_2))/2)\n", 1401 | "chisquare([len(A_1),len(A_2)],f_exp = [half,half])" 1402 | ] 1403 | }, 1404 | { 1405 | "cell_type": "markdown", 1406 | "metadata": {}, 1407 | "source": [ 1408 | "# Conclusion 3: p_value = 0.327 & no SRM" 1409 | ] 1410 | } 1411 | ], 1412 | "metadata": { 1413 | "kernelspec": { 1414 | "display_name": "Python 3", 1415 | "language": "python", 1416 | "name": "python3" 1417 | }, 1418 | "language_info": { 1419 | "codemirror_mode": { 1420 | "name": "ipython", 1421 | "version": 3 1422 | }, 1423 | "file_extension": ".py", 1424 | "mimetype": "text/x-python", 1425 | "name": "python", 1426 | "nbconvert_exporter": "python", 1427 | "pygments_lexer": "ipython3", 1428 | "version": "3.7.4" 1429 | } 1430 | }, 1431 | "nbformat": 4, 1432 | "nbformat_minor": 2 1433 | } 1434 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Statistical Simulation in Python 2 | 3 | # Project Summary 4 | 5 | This repo hosts R/Python code to common statistical distributions. This is an on-going project, and currently it has three parts: 6 | 1. sampling (part 1) 7 | 2. distribution and applications (part 2) 8 | 3. An end-to-end intro to A/B tests 9 | 4. Why A/A Tests? What if we fail A/A tests? 10 | 11 | In part 1, I introduce how to create random numbers, sample with equal and unequal probabilities, their applications. In part 2, I apply these distributions to solve real-life Data Science Interview questions. Part 3 walks through how to conduct an A/B test end-to-end with simulated data. 12 | 13 | The entire write-ups are available here. 14 | 15 | Part 1: https://towardsdatascience.com/statistical-simulation-in-r-part-1-d9cb4dc393c9 16 | 17 | Part 2: https://towardsdatascience.com/statistical-simulation-in-python-part-2-91f71f474f77 18 | 19 | Part 3: https://towardsdatascience.com/a-practical-guide-to-a-b-tests-in-python-66666f5c3b02?source=friends_link&sk=null 20 | 21 | Part 4: https://towardsdatascience.com/an-a-b-test-loses-its-luster-if-a-a-tests-fail-2dd11fa6d241?sk=8d4cebf2d3362704a4121b4518364c36 22 | 23 | ## Installing 24 | 25 | For part 1, please install dplyr in R. 26 | 27 | For part 2, please install numpy in Python. 28 | 29 | For part 3, please install pandas in Python. 30 | 31 | For part 4, please install pandas in Python 32 | 33 | ## About the Author 34 | 35 | Leihua Ye is a Ph.D. Researcher at the UC, Santa Barbara. He has received extensive training in Causal Inference, Research Design, Machine Learning, Big Data, and Machine Learning. 36 | 37 | He receives his B.A. and M.A. from the Uni. of Nottingham. 38 | 39 | ## Contact 40 | 41 | Email: yeleihua@gmail.com 42 | 43 | LinkedIn: www.linkedin.com/in/leihuaye 44 | 45 | Tech Blog: https://leihua-ye.medium.com 46 | --------------------------------------------------------------------------------